- 'css_for_each_child'
- 'css_for_each_descendant_post'
- 'css_for_each_descendant_pre'
- - 'cxl_for_each_cmd'
- 'device_for_each_child_node'
+ - 'displayid_iter_for_each'
- 'dma_fence_chain_for_each'
- 'do_for_each_ftrace_op'
- 'drm_atomic_crtc_for_each_plane'
- 'drm_mm_for_each_node_in_range'
- 'drm_mm_for_each_node_safe'
- 'flow_action_for_each'
+ - 'for_each_acpi_dev_match'
- 'for_each_active_dev_scope'
- 'for_each_active_drhd_unit'
- 'for_each_active_iommu'
- 'for_each_dapm_widgets'
- 'for_each_dev_addr'
- 'for_each_dev_scope'
- - 'for_each_displayid_db'
- 'for_each_dma_cap_mask'
- 'for_each_dpcm_be'
- 'for_each_dpcm_be_rollback'
- 'for_each_dpcm_fe'
- 'for_each_drhd_unit'
- 'for_each_dss_dev'
+ - 'for_each_dtpm_table'
- 'for_each_efi_memory_desc'
- 'for_each_efi_memory_desc_in_map'
- 'for_each_element'
- 'for_each_migratetype_order'
- 'for_each_msi_entry'
- 'for_each_msi_entry_safe'
+ - 'for_each_msi_vector'
- 'for_each_net'
- 'for_each_net_continue_reverse'
- 'for_each_netdev'
- 'for_each_prime_number_from'
- 'for_each_process'
- 'for_each_process_thread'
+ - 'for_each_prop_codec_conf'
+ - 'for_each_prop_dai_codec'
+ - 'for_each_prop_dai_cpu'
+ - 'for_each_prop_dlc_codecs'
+ - 'for_each_prop_dlc_cpus'
+ - 'for_each_prop_dlc_platforms'
- 'for_each_property_of_node'
- 'for_each_registered_fb'
- 'for_each_requested_gpio'
- 'queue_for_each_hw_ctx'
- 'radix_tree_for_each_slot'
- 'radix_tree_for_each_tagged'
+ - 'rb_for_each'
- 'rbtree_postorder_for_each_entry_safe'
- 'rdma_for_each_block'
- 'rdma_for_each_port'
Jeff Layton <jlayton@kernel.org> <jlayton@redhat.com>
Jens Axboe <axboe@suse.de>
Jens Osterkamp <Jens.Osterkamp@de.ibm.com>
+Jernej Skrabec <jernej.skrabec@gmail.com> <jernej.skrabec@siol.net>
Jiri Slaby <jirislaby@kernel.org> <jirislaby@gmail.com>
Jiri Slaby <jirislaby@kernel.org> <jslaby@novell.com>
Jiri Slaby <jirislaby@kernel.org> <jslaby@suse.com>
Mayuresh Janorkar <mayur@ti.com>
Michael Buesch <m@bues.ch>
Michel Dänzer <michel@tungstengraphics.com>
+Michel Lespinasse <michel@lespinasse.org>
+Michel Lespinasse <michel@lespinasse.org> <walken@google.com>
+Michel Lespinasse <michel@lespinasse.org> <walken@zoy.org>
Miguel Ojeda <ojeda@kernel.org> <miguel.ojeda.sandonis@gmail.com>
Mike Rapoport <rppt@kernel.org> <mike@compulab.co.il>
Mike Rapoport <rppt@kernel.org> <mike.rapoport@gmail.com>
What: /sys/class/dax/
Date: May, 2016
KernelVersion: v4.7
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description: Device DAX is the device-centric analogue of Filesystem
DAX (CONFIG_FS_DAX). It allows memory ranges to be
allocated and mapped without need of an intervening file
-This ABI is renamed and moved to a new location /sys/kernel/fadump/registered.¬
+This ABI is renamed and moved to a new location /sys/kernel/fadump/registered.
What: /sys/kernel/fadump_registered
Date: Feb 2012
-This ABI is renamed and moved to a new location /sys/kernel/fadump/release_mem.¬
+This ABI is renamed and moved to a new location /sys/kernel/fadump/release_mem.
What: /sys/kernel/fadump_release_mem
Date: Feb 2012
What: /sys/bus/nd/devices/regionX/nfit/ecc_unit_size
Date: Aug, 2017
KernelVersion: v4.14 (Removed v4.18)
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) Size of a write request to a DIMM that will not incur a
read-modify-write cycle at the memory controller.
What: /sys/bus/nd/devices/nmemX/nfit/serial
Date: Jun, 2015
KernelVersion: v4.2
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) Serial number of the NVDIMM (non-volatile dual in-line
memory module), assigned by the module vendor.
What: /sys/bus/nd/devices/nmemX/nfit/handle
Date: Apr, 2015
KernelVersion: v4.2
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) The address (given by the _ADR object) of the device on its
parent bus of the NVDIMM device containing the NVDIMM region.
What: /sys/bus/nd/devices/nmemX/nfit/device
Date: Apr, 2015
KernelVersion: v4.1
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) Device id for the NVDIMM, assigned by the module vendor.
What: /sys/bus/nd/devices/nmemX/nfit/rev_id
Date: Jun, 2015
KernelVersion: v4.2
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) Revision of the NVDIMM, assigned by the module vendor.
What: /sys/bus/nd/devices/nmemX/nfit/phys_id
Date: Apr, 2015
KernelVersion: v4.2
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) Handle (i.e., instance number) for the SMBIOS (system
management BIOS) Memory Device structure describing the NVDIMM
What: /sys/bus/nd/devices/nmemX/nfit/flags
Date: Jun, 2015
KernelVersion: v4.2
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) The flags in the NFIT memory device sub-structure indicate
the state of the data on the nvdimm relative to its energy
What: /sys/bus/nd/devices/nmemX/nfit/formats
Date: Apr, 2016
KernelVersion: v4.7
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) The interface codes indicate support for persistent memory
mapped directly into system physical address space and / or a
What: /sys/bus/nd/devices/nmemX/nfit/vendor
Date: Apr, 2016
KernelVersion: v4.7
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) Vendor id of the NVDIMM.
What: /sys/bus/nd/devices/nmemX/nfit/dsm_mask
Date: May, 2016
KernelVersion: v4.7
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) The bitmask indicates the supported device specific control
functions relative to the NVDIMM command family supported by the
What: /sys/bus/nd/devices/nmemX/nfit/family
Date: Apr, 2016
KernelVersion: v4.7
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) Displays the NVDIMM family command sets. Values
0, 1, 2 and 3 correspond to NVDIMM_FAMILY_INTEL,
What: /sys/bus/nd/devices/nmemX/nfit/id
Date: Apr, 2016
KernelVersion: v4.7
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) ACPI specification 6.2 section 5.2.25.9, defines an
identifier for an NVDIMM, which refelects the id attribute.
What: /sys/bus/nd/devices/nmemX/nfit/subsystem_vendor
Date: Apr, 2016
KernelVersion: v4.7
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) Sub-system vendor id of the NVDIMM non-volatile memory
subsystem controller.
What: /sys/bus/nd/devices/nmemX/nfit/subsystem_rev_id
Date: Apr, 2016
KernelVersion: v4.7
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) Sub-system revision id of the NVDIMM non-volatile memory subsystem
controller, assigned by the non-volatile memory subsystem
What: /sys/bus/nd/devices/nmemX/nfit/subsystem_device
Date: Apr, 2016
KernelVersion: v4.7
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) Sub-system device id for the NVDIMM non-volatile memory
subsystem controller, assigned by the non-volatile memory
What: /sys/bus/nd/devices/ndbusX/nfit/revision
Date: Jun, 2015
KernelVersion: v4.2
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) ACPI NFIT table revision number.
What: /sys/bus/nd/devices/ndbusX/nfit/scrub
Date: Sep, 2016
KernelVersion: v4.9
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RW) This shows the number of full Address Range Scrubs (ARS)
that have been completed since driver load time. Userspace can
What: /sys/bus/nd/devices/ndbusX/nfit/hw_error_scrub
Date: Sep, 2016
KernelVersion: v4.9
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RW) Provides a way to toggle the behavior between just adding
the address (cache line) where the MCE happened to the poison
What: /sys/bus/nd/devices/ndbusX/nfit/dsm_mask
Date: Jun, 2017
KernelVersion: v4.13
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) The bitmask indicates the supported bus specific control
functions. See the section named 'NVDIMM Root Device _DSMs' in
What: /sys/bus/nd/devices/ndbusX/nfit/firmware_activate_noidle
Date: Apr, 2020
KernelVersion: v5.8
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RW) The Intel platform implementation of firmware activate
support exposes an option let the platform force idle devices in
What: /sys/bus/nd/devices/regionX/nfit/range_index
Date: Jun, 2015
KernelVersion: v4.2
-Contact: linux-nvdimm@lists.01.org
+Contact: nvdimm@lists.linux.dev
Description:
(RO) A unique number provided by the BIOS to identify an address
range. Used by NVDIMM Region Mapping Structure to uniquely refer
What: /sys/bus/nd/devices/nmemX/papr/flags
Date: Apr, 2020
KernelVersion: v5.8
-Contact: linuxppc-dev <linuxppc-dev@lists.ozlabs.org>, linux-nvdimm@lists.01.org,
+Contact: linuxppc-dev <linuxppc-dev@lists.ozlabs.org>, nvdimm@lists.linux.dev,
Description:
(RO) Report flags indicating various states of a
papr-pmem NVDIMM device. Each flag maps to a one or
What: /sys/bus/nd/devices/nmemX/papr/perf_stats
Date: May, 2020
KernelVersion: v5.9
-Contact: linuxppc-dev <linuxppc-dev@lists.ozlabs.org>, linux-nvdimm@lists.01.org,
+Contact: linuxppc-dev <linuxppc-dev@lists.ozlabs.org>, nvdimm@lists.linux.dev,
Description:
(RO) Report various performance stats related to papr-scm NVDIMM
device. Each stat is reported on a new line with each line
--- /dev/null
+What: /sys/devices/platform/soc@X/XXXXXXX.ipa/
+Date: June 2021
+KernelVersion: v5.14
+Contact: Alex Elder <elder@kernel.org>
+Description:
+ The /sys/devices/platform/soc@X/XXXXXXX.ipa/ directory
+ contains read-only attributes exposing information about
+ an IPA device. The X values could vary, but are typically
+ "soc@0/1e40000.ipa".
+
+What: .../XXXXXXX.ipa/version
+Date: June 2021
+KernelVersion: v5.14
+Contact: Alex Elder <elder@kernel.org>
+Description:
+ The .../XXXXXXX.ipa/version file contains the IPA hardware
+ version, as a period-separated set of two or three integers
+ (e.g., "3.5.1" or "4.2").
+
+What: .../XXXXXXX.ipa/feature/
+Date: June 2021
+KernelVersion: v5.14
+Contact: Alex Elder <elder@kernel.org>
+Description:
+ The .../XXXXXXX.ipa/feature/ directory contains a set of
+ attributes describing features implemented by the IPA
+ hardware.
+
+What: .../XXXXXXX.ipa/feature/rx_offload
+Date: June 2021
+KernelVersion: v5.14
+Contact: Alex Elder <elder@kernel.org>
+Description:
+ The .../XXXXXXX.ipa/feature/rx_offload file contains a
+ string indicating the type of receive checksum offload
+ that is supported by the hardware. The possible values
+ are "MAPv4" or "MAPv5".
+
+What: .../XXXXXXX.ipa/feature/tx_offload
+Date: June 2021
+KernelVersion: v5.14
+Contact: Alex Elder <elder@kernel.org>
+Description:
+ The .../XXXXXXX.ipa/feature/tx_offload file contains a
+ string indicating the type of transmit checksum offload
+ that is supported by the hardware. The possible values
+ are "MAPv4" or "MAPv5".
+
+What: .../XXXXXXX.ipa/modem/
+Date: June 2021
+KernelVersion: v5.14
+Contact: Alex Elder <elder@kernel.org>
+Description:
+ The .../XXXXXXX.ipa/modem/ directory contains a set of
+ attributes describing properties of the modem execution
+ environment reachable by the IPA hardware.
+
+What: .../XXXXXXX.ipa/modem/rx_endpoint_id
+Date: June 2021
+KernelVersion: v5.14
+Contact: Alex Elder <elder@kernel.org>
+Description:
+ The .../XXXXXXX.ipa/feature/rx_endpoint_id file contains
+ the AP endpoint ID that receives packets originating from
+ the modem execution environment. The "rx" is from the
+ perspective of the AP; this endpoint is considered an "IPA
+ producer". An endpoint ID is a small unsigned integer.
+
+What: .../XXXXXXX.ipa/modem/tx_endpoint_id
+Date: June 2021
+KernelVersion: v5.14
+Contact: Alex Elder <elder@kernel.org>
+Description:
+ The .../XXXXXXX.ipa/feature/tx_endpoint_id file contains
+ the AP endpoint ID used to transmit packets destined for
+ the modem execution environment. The "tx" is from the
+ perspective of the AP; this endpoint is considered an "IPA
+ consumer". An endpoint ID is a small unsigned integer.
What: /sys/module/*/{coresize,initsize}
Date: Jan 2012
-KernelVersion:»·3.3
+KernelVersion: 3.3
Contact: Kay Sievers <kay.sievers@vrfy.org>
Description: Module size in bytes.
What: /sys/module/*/taint
Date: Jan 2012
-KernelVersion:»·3.3
+KernelVersion: 3.3
Contact: Kay Sievers <kay.sievers@vrfy.org>
Description: Module taint flags:
== =====================
========
The full path to the usermode helper for autoloading kernel modules,
-by default "/sbin/modprobe". This binary is executed when the kernel
-requests a module. For example, if userspace passes an unknown
-filesystem type to mount(), then the kernel will automatically request
-the corresponding filesystem module by executing this usermode helper.
+by default ``CONFIG_MODPROBE_PATH``, which in turn defaults to
+"/sbin/modprobe". This binary is executed when the kernel requests a
+module. For example, if userspace passes an unknown filesystem type
+to mount(), then the kernel will automatically request the
+corresponding filesystem module by executing this usermode helper.
This usermode helper should insert the needed module into the kernel.
This sysctl only affects module autoloading. It has no effect on the
=========================
Writing 1 to this entry will disable unprivileged calls to ``bpf()``;
-once disabled, calling ``bpf()`` without ``CAP_SYS_ADMIN`` will return
-``-EPERM``.
+once disabled, calling ``bpf()`` without ``CAP_SYS_ADMIN`` or ``CAP_BPF``
+will return ``-EPERM``. Once set to 1, this can't be cleared from the
+running kernel anymore.
-Once set, this can't be cleared.
+Writing 2 to this entry will also disable unprivileged calls to ``bpf()``,
+however, an admin can still change this setting later on, if needed, by
+writing 0 or 1 to this entry.
+If ``BPF_UNPRIV_DEFAULT_OFF`` is enabled in the kernel config, then this
+entry will default to 2 instead of 0.
+
+= =============================================================
+0 Unprivileged calls to ``bpf()`` are enabled
+1 Unprivileged calls to ``bpf()`` are disabled without recovery
+2 Unprivileged calls to ``bpf()`` are disabled
+= =============================================================
watchdog
========
-==============
+==============
Data Integrity
==============
:maxdepth: 1
ringbuf
+ llvm_reloc
.. Links:
.. _networking-filter: ../networking/filter.rst
--- /dev/null
+.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
+====================
+BPF LLVM Relocations
+====================
+
+This document describes LLVM BPF backend relocation types.
+
+Relocation Record
+=================
+
+LLVM BPF backend records each relocation with the following 16-byte
+ELF structure::
+
+ typedef struct
+ {
+ Elf64_Addr r_offset; // Offset from the beginning of section.
+ Elf64_Xword r_info; // Relocation type and symbol index.
+ } Elf64_Rel;
+
+For example, for the following code::
+
+ int g1 __attribute__((section("sec")));
+ int g2 __attribute__((section("sec")));
+ static volatile int l1 __attribute__((section("sec")));
+ static volatile int l2 __attribute__((section("sec")));
+ int test() {
+ return g1 + g2 + l1 + l2;
+ }
+
+Compiled with ``clang -target bpf -O2 -c test.c``, the following is
+the code with ``llvm-objdump -dr test.o``::
+
+ 0: 18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
+ 0000000000000000: R_BPF_64_64 g1
+ 2: 61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)
+ 3: 18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 = 0 ll
+ 0000000000000018: R_BPF_64_64 g2
+ 5: 61 20 00 00 00 00 00 00 r0 = *(u32 *)(r2 + 0)
+ 6: 0f 10 00 00 00 00 00 00 r0 += r1
+ 7: 18 01 00 00 08 00 00 00 00 00 00 00 00 00 00 00 r1 = 8 ll
+ 0000000000000038: R_BPF_64_64 sec
+ 9: 61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)
+ 10: 0f 10 00 00 00 00 00 00 r0 += r1
+ 11: 18 01 00 00 0c 00 00 00 00 00 00 00 00 00 00 00 r1 = 12 ll
+ 0000000000000058: R_BPF_64_64 sec
+ 13: 61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)
+ 14: 0f 10 00 00 00 00 00 00 r0 += r1
+ 15: 95 00 00 00 00 00 00 00 exit
+
+There are four relations in the above for four ``LD_imm64`` instructions.
+The following ``llvm-readelf -r test.o`` shows the binary values of the four
+relocations::
+
+ Relocation section '.rel.text' at offset 0x190 contains 4 entries:
+ Offset Info Type Symbol's Value Symbol's Name
+ 0000000000000000 0000000600000001 R_BPF_64_64 0000000000000000 g1
+ 0000000000000018 0000000700000001 R_BPF_64_64 0000000000000004 g2
+ 0000000000000038 0000000400000001 R_BPF_64_64 0000000000000000 sec
+ 0000000000000058 0000000400000001 R_BPF_64_64 0000000000000000 sec
+
+Each relocation is represented by ``Offset`` (8 bytes) and ``Info`` (8 bytes).
+For example, the first relocation corresponds to the first instruction
+(Offset 0x0) and the corresponding ``Info`` indicates the relocation type
+of ``R_BPF_64_64`` (type 1) and the entry in the symbol table (entry 6).
+The following is the symbol table with ``llvm-readelf -s test.o``::
+
+ Symbol table '.symtab' contains 8 entries:
+ Num: Value Size Type Bind Vis Ndx Name
+ 0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
+ 1: 0000000000000000 0 FILE LOCAL DEFAULT ABS test.c
+ 2: 0000000000000008 4 OBJECT LOCAL DEFAULT 4 l1
+ 3: 000000000000000c 4 OBJECT LOCAL DEFAULT 4 l2
+ 4: 0000000000000000 0 SECTION LOCAL DEFAULT 4 sec
+ 5: 0000000000000000 128 FUNC GLOBAL DEFAULT 2 test
+ 6: 0000000000000000 4 OBJECT GLOBAL DEFAULT 4 g1
+ 7: 0000000000000004 4 OBJECT GLOBAL DEFAULT 4 g2
+
+The 6th entry is global variable ``g1`` with value 0.
+
+Similarly, the second relocation is at ``.text`` offset ``0x18``, instruction 3,
+for global variable ``g2`` which has a symbol value 4, the offset
+from the start of ``.data`` section.
+
+The third and fourth relocations refers to static variables ``l1``
+and ``l2``. From ``.rel.text`` section above, it is not clear
+which symbols they really refers to as they both refers to
+symbol table entry 4, symbol ``sec``, which has ``STT_SECTION`` type
+and represents a section. So for static variable or function,
+the section offset is written to the original insn
+buffer, which is called ``A`` (addend). Looking at
+above insn ``7`` and ``11``, they have section offset ``8`` and ``12``.
+From symbol table, we can find that they correspond to entries ``2``
+and ``3`` for ``l1`` and ``l2``.
+
+In general, the ``A`` is 0 for global variables and functions,
+and is the section offset or some computation result based on
+section offset for static variables/functions. The non-section-offset
+case refers to function calls. See below for more details.
+
+Different Relocation Types
+==========================
+
+Six relocation types are supported. The following is an overview and
+``S`` represents the value of the symbol in the symbol table::
+
+ Enum ELF Reloc Type Description BitSize Offset Calculation
+ 0 R_BPF_NONE None
+ 1 R_BPF_64_64 ld_imm64 insn 32 r_offset + 4 S + A
+ 2 R_BPF_64_ABS64 normal data 64 r_offset S + A
+ 3 R_BPF_64_ABS32 normal data 32 r_offset S + A
+ 4 R_BPF_64_NODYLD32 .BTF[.ext] data 32 r_offset S + A
+ 10 R_BPF_64_32 call insn 32 r_offset + 4 (S + A) / 8 - 1
+
+For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
+The actual to-be-relocated data (0 or section offset)
+is stored at ``r_offset + 4`` and the read/write
+data bitsize is 32 (4 bytes). The relocation can be resolved with
+the symbol value plus implicit addend. Note that the ``BitSize`` is 32 which
+means the section offset must be less than or equal to ``UINT32_MAX`` and this
+is enforced by LLVM BPF backend.
+
+In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
+The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
+bitsize is 64 (8 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
+But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
+``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
+is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
+to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
+become unusable by bcc and kernel.
+
+Type ``R_BPF_64_32`` is used for call instruction. The call target section
+offset is stored at ``r_offset + 4`` (32bit) and calculated as
+``(S + A) / 8 - 1``.
+
+Examples
+========
+
+Types ``R_BPF_64_64`` and ``R_BPF_64_32`` are used to resolve ``ld_imm64``
+and ``call`` instructions. For example::
+
+ __attribute__((noinline)) __attribute__((section("sec1")))
+ int gfunc(int a, int b) {
+ return a * b;
+ }
+ static __attribute__((noinline)) __attribute__((section("sec1")))
+ int lfunc(int a, int b) {
+ return a + b;
+ }
+ int global __attribute__((section("sec2")));
+ int test(int a, int b) {
+ return gfunc(a, b) + lfunc(a, b) + global;
+ }
+
+Compiled with ``clang -target bpf -O2 -c test.c``, we will have
+following code with `llvm-objdump -dr test.o``::
+
+ Disassembly of section .text:
+
+ 0000000000000000 <test>:
+ 0: bf 26 00 00 00 00 00 00 r6 = r2
+ 1: bf 17 00 00 00 00 00 00 r7 = r1
+ 2: 85 10 00 00 ff ff ff ff call -1
+ 0000000000000010: R_BPF_64_32 gfunc
+ 3: bf 08 00 00 00 00 00 00 r8 = r0
+ 4: bf 71 00 00 00 00 00 00 r1 = r7
+ 5: bf 62 00 00 00 00 00 00 r2 = r6
+ 6: 85 10 00 00 02 00 00 00 call 2
+ 0000000000000030: R_BPF_64_32 sec1
+ 7: 0f 80 00 00 00 00 00 00 r0 += r8
+ 8: 18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
+ 0000000000000040: R_BPF_64_64 global
+ 10: 61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)
+ 11: 0f 10 00 00 00 00 00 00 r0 += r1
+ 12: 95 00 00 00 00 00 00 00 exit
+
+ Disassembly of section sec1:
+
+ 0000000000000000 <gfunc>:
+ 0: bf 20 00 00 00 00 00 00 r0 = r2
+ 1: 2f 10 00 00 00 00 00 00 r0 *= r1
+ 2: 95 00 00 00 00 00 00 00 exit
+
+ 0000000000000018 <lfunc>:
+ 3: bf 20 00 00 00 00 00 00 r0 = r2
+ 4: 0f 10 00 00 00 00 00 00 r0 += r1
+ 5: 95 00 00 00 00 00 00 00 exit
+
+The first relocation corresponds to ``gfunc(a, b)`` where ``gfunc`` has a value of 0,
+so the ``call`` instruction offset is ``(0 + 0)/8 - 1 = -1``.
+The second relocation corresponds to ``lfunc(a, b)`` where ``lfunc`` has a section
+offset ``0x18``, so the ``call`` instruction offset is ``(0 + 0x18)/8 - 1 = 2``.
+The third relocation corresponds to ld_imm64 of ``global``, which has a section
+offset ``0``.
+
+The following is an example to show how R_BPF_64_ABS64 could be generated::
+
+ int global() { return 0; }
+ struct t { void *g; } gbl = { global };
+
+Compiled with ``clang -target bpf -O2 -g -c test.c``, we will see a
+relocation below in ``.data`` section with command
+``llvm-readelf -r test.o``::
+
+ Relocation section '.rel.data' at offset 0x458 contains 1 entries:
+ Offset Info Type Symbol's Value Symbol's Name
+ 0000000000000000 0000000700000002 R_BPF_64_ABS64 0000000000000000 global
+
+The relocation says the first 8-byte of ``.data`` section should be
+filled with address of ``global`` variable.
+
+With ``llvm-readelf`` output, we can see that dwarf sections have a bunch of
+``R_BPF_64_ABS32`` and ``R_BPF_64_ABS64`` relocations::
+
+ Relocation section '.rel.debug_info' at offset 0x468 contains 13 entries:
+ Offset Info Type Symbol's Value Symbol's Name
+ 0000000000000006 0000000300000003 R_BPF_64_ABS32 0000000000000000 .debug_abbrev
+ 000000000000000c 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
+ 0000000000000012 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
+ 0000000000000016 0000000600000003 R_BPF_64_ABS32 0000000000000000 .debug_line
+ 000000000000001a 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
+ 000000000000001e 0000000200000002 R_BPF_64_ABS64 0000000000000000 .text
+ 000000000000002b 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
+ 0000000000000037 0000000800000002 R_BPF_64_ABS64 0000000000000000 gbl
+ 0000000000000040 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
+ ......
+
+The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations::
+
+ Relocation section '.rel.BTF' at offset 0x538 contains 1 entries:
+ Offset Info Type Symbol's Value Symbol's Name
+ 0000000000000084 0000000800000004 R_BPF_64_NODYLD32 0000000000000000 gbl
+
+ Relocation section '.rel.BTF.ext' at offset 0x548 contains 2 entries:
+ Offset Info Type Symbol's Value Symbol's Name
+ 000000000000002c 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text
+ 0000000000000040 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text
*struct file_operations*::
struct file_operations cdrom_fops = {
- NULL, /∗ lseek ∗/
- block _read , /∗ read—general block-dev read ∗/
- block _write, /∗ write—general block-dev write ∗/
- NULL, /∗ readdir ∗/
- NULL, /∗ select ∗/
- cdrom_ioctl, /∗ ioctl ∗/
- NULL, /∗ mmap ∗/
- cdrom_open, /∗ open ∗/
- cdrom_release, /∗ release ∗/
- NULL, /∗ fsync ∗/
- NULL, /∗ fasync ∗/
- NULL /∗ revalidate ∗/
+ NULL, /* lseek */
+ block _read , /* read--general block-dev read */
+ block _write, /* write--general block-dev write */
+ NULL, /* readdir */
+ NULL, /* select */
+ cdrom_ioctl, /* ioctl */
+ NULL, /* mmap */
+ cdrom_open, /* open */
+ cdrom_release, /* release */
+ NULL, /* fsync */
+ NULL, /* fasync */
+ NULL /* revalidate */
};
Every active CD-ROM device shares this *struct*. The routines
`cdrom.c`, currently contains the following fields::
struct cdrom_device_info {
- const struct cdrom_device_ops * ops; /* device operations for this major */
+ const struct cdrom_device_ops * ops; /* device operations for this major */
struct list_head list; /* linked list of all device_info */
struct gendisk * disk; /* matching block layer disk */
void * handle; /* driver-dependent data */
- int mask; /* mask of capability: disables them */
+ int mask; /* mask of capability: disables them */
int speed; /* maximum speed for reading data */
int capacity; /* number of discs in a jukebox */
In the file `cdrom.c` you will encounter many constructions of the type::
- if (cdo->capability & ∼cdi->mask & CDC _⟨capability⟩) ...
+ if (cdo->capability & ~cdi->mask & CDC _<capability>) ...
There is no *ioctl* to set the mask... The reason is that
I think it is better to control the **behavior** rather than the
maxItems: 2
idt,xtal-load-femtofarads:
- $ref: /schemas/types.yaml#/definitions/uint32
minimum: 9000
maximum: 22760
description: Optional load capacitor for XTAL1 and XTAL2
enum: [ 1800000, 2500000, 3300000 ]
idt,slew-percent:
description: The Slew rate control for CMOS single-ended.
- $ref: /schemas/types.yaml#/definitions/uint32
enum: [ 80, 85, 90, 100 ]
required:
maxItems: 6
$ref: /schemas/types.yaml#/definitions/uint32-array
+ sink-vdos-v1:
+ description: An array of u32 with each entry, a Vendor Defined Message Object (VDO),
+ providing additional information corresponding to the product, the detailed bit
+ definitions and the order of each VDO can be found in
+ "USB Power Delivery Specification Revision 2.0, Version 1.3" chapter 6.4.4.3.1 Discover
+ Identity. User can specify the VDO array via VDO_IDH/_CERT/_PRODUCT/_CABLE/_AMA defined in
+ dt-bindings/usb/pd.h.
+ minItems: 3
+ maxItems: 6
+ $ref: /schemas/types.yaml#/definitions/uint32-array
+
op-sink-microwatt:
description: Sink required operating power in microwatt, if source can't
offer the power, Capability Mismatch is set. Required for power sink and
SNK_READY for non-pd link.
type: boolean
+dependencies:
+ sink-vdos-v1: [ 'sink-vdos' ]
+ sink-vdos: [ 'sink-vdos-v1' ]
+
required:
- compatible
#size-cells = <0>;
adc@48 {
- comatible = "ti,ads7828";
+ compatible = "ti,ads7828";
reg = <0x48>;
vref-supply = <&vref>;
ti,differential-input;
description: |
I2C bus timeout in microseconds
+ fsl,i2c-erratum-a004447:
+ $ref: /schemas/types.yaml#/definitions/flag
+ description: |
+ Indicates the presence of QorIQ erratum A-004447, which
+ says that the standard i2c recovery scheme mechanism does
+ not work and an alternate implementation is needed.
+
required:
- compatible
- reg
st,adc-channel-names:
description: List of single-ended channel names.
- $ref: /schemas/types.yaml#/definitions/string-array
st,filter-order:
description: |
Duration in seconds which the key should be kept pressed for device to
reset automatically. Device with key pressed reset feature can specify
this property.
- $ref: /schemas/types.yaml#/definitions/uint32
additionalProperties: true
this interconnect to send RPMh commands.
qcom,bcm-voter-names:
- $ref: /schemas/types.yaml#/definitions/string-array
description: |
Names for each of the qcom,bcm-voters specified.
In these SoCs it's possible to control LEDs both as GPIOs or by hardware.
However, on some devices there are Serial LEDs (LEDs connected to a 74x164
controller), which can either be controlled by software (exporting the 74x164
-as spi-gpio. See Documentation/devicetree/bindings/gpio/gpio-74x164.txt), or
-by hardware using this driver.
+as spi-gpio. See Documentation/devicetree/bindings/gpio/fairchild,74hc595.yaml),
+or by hardware using this driver.
Some of these Serial LEDs are hardware controlled (e.g. ethernet LEDs) and
exporting the 74x164 as spi-gpio prevents those LEDs to be hardware
controlled, so the only chance to keep them working is by using this driver.
This controller is present on BCM6358 and BCM6368.
In these SoCs there are Serial LEDs (LEDs connected to a 74x164 controller),
which can either be controlled by software (exporting the 74x164 as spi-gpio.
-See Documentation/devicetree/bindings/gpio/gpio-74x164.txt), or
+See Documentation/devicetree/bindings/gpio/fairchild,74hc595.yaml), or
by hardware using this driver.
Required properties:
maxItems: 1
clock-names:
- maxItems: 1
- items:
- - const: fck
+ const: fck
resets:
maxItems: 1
Indicates that the channel acts as primary among the bonded channels.
port:
- type: object
+ $ref: /schemas/graph.yaml#/properties/port
+ unevaluatedProperties: false
description:
- Child port node corresponding to the data input, in accordance with the
- video interface bindings defined in
- Documentation/devicetree/bindings/media/video-interfaces.txt.
- The port node must contain at least one endpoint.
+ Child port node corresponding to the data input. The port node must
+ contain at least one endpoint.
properties:
endpoint:
- type: object
+ $ref: /schemas/graph.yaml#/$defs/endpoint-base
+ unevaluatedProperties: false
properties:
- remote-endpoint:
- description:
- A phandle to the remote tuner endpoint subnode in remote node
- port.
-
sync-active:
+ $ref: /schemas/types.yaml#/definitions/uint32
enum: [0, 1]
description:
Indicates sync signal polarity, 0/1 for low/high respectively.
This property maps to SYNCAC bit in the hardware manual. The
default is 1 (active high).
- additionalProperties: false
-
required:
- compatible
- reg
+++ /dev/null
-* Broadcom iProc MDIO bus controller
-
-Required properties:
-- compatible: should be "brcm,iproc-mdio"
-- reg: address and length of the register set for the MDIO interface
-- #size-cells: must be 1
-- #address-cells: must be 0
-
-Child nodes of this MDIO bus controller node are standard Ethernet PHY device
-nodes as described in Documentation/devicetree/bindings/net/phy.txt
-
-Example:
-
-mdio@18002000 {
- compatible = "brcm,iproc-mdio";
- reg = <0x18002000 0x8>;
- #size-cells = <1>;
- #address-cells = <0>;
-
- enet-gphy@0 {
- reg = <0>;
- };
-};
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/net/brcm,iproc-mdio.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Broadcom iProc MDIO bus controller
+
+maintainers:
+ - Rafał Miłecki <rafal@milecki.pl>
+
+allOf:
+ - $ref: mdio.yaml#
+
+properties:
+ compatible:
+ const: brcm,iproc-mdio
+
+ reg:
+ maxItems: 1
+
+unevaluatedProperties: false
+
+required:
+ - reg
+
+examples:
+ - |
+ mdio@18002000 {
+ compatible = "brcm,iproc-mdio";
+ reg = <0x18002000 0x8>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ ethernet-phy@0 {
+ reg = <0>;
+ };
+ };
+++ /dev/null
-Renesas R-Car CAN controller Device Tree Bindings
--------------------------------------------------
-
-Required properties:
-- compatible: "renesas,can-r8a7742" if CAN controller is a part of R8A7742 SoC.
- "renesas,can-r8a7743" if CAN controller is a part of R8A7743 SoC.
- "renesas,can-r8a7744" if CAN controller is a part of R8A7744 SoC.
- "renesas,can-r8a7745" if CAN controller is a part of R8A7745 SoC.
- "renesas,can-r8a77470" if CAN controller is a part of R8A77470 SoC.
- "renesas,can-r8a774a1" if CAN controller is a part of R8A774A1 SoC.
- "renesas,can-r8a774b1" if CAN controller is a part of R8A774B1 SoC.
- "renesas,can-r8a774c0" if CAN controller is a part of R8A774C0 SoC.
- "renesas,can-r8a774e1" if CAN controller is a part of R8A774E1 SoC.
- "renesas,can-r8a7778" if CAN controller is a part of R8A7778 SoC.
- "renesas,can-r8a7779" if CAN controller is a part of R8A7779 SoC.
- "renesas,can-r8a7790" if CAN controller is a part of R8A7790 SoC.
- "renesas,can-r8a7791" if CAN controller is a part of R8A7791 SoC.
- "renesas,can-r8a7792" if CAN controller is a part of R8A7792 SoC.
- "renesas,can-r8a7793" if CAN controller is a part of R8A7793 SoC.
- "renesas,can-r8a7794" if CAN controller is a part of R8A7794 SoC.
- "renesas,can-r8a7795" if CAN controller is a part of R8A7795 SoC.
- "renesas,can-r8a7796" if CAN controller is a part of R8A77960 SoC.
- "renesas,can-r8a77961" if CAN controller is a part of R8A77961 SoC.
- "renesas,can-r8a77965" if CAN controller is a part of R8A77965 SoC.
- "renesas,can-r8a77990" if CAN controller is a part of R8A77990 SoC.
- "renesas,can-r8a77995" if CAN controller is a part of R8A77995 SoC.
- "renesas,rcar-gen1-can" for a generic R-Car Gen1 compatible device.
- "renesas,rcar-gen2-can" for a generic R-Car Gen2 or RZ/G1
- compatible device.
- "renesas,rcar-gen3-can" for a generic R-Car Gen3 or RZ/G2
- compatible device.
- When compatible with the generic version, nodes must list the
- SoC-specific version corresponding to the platform first
- followed by the generic version.
-
-- reg: physical base address and size of the R-Car CAN register map.
-- interrupts: interrupt specifier for the sole interrupt.
-- clocks: phandles and clock specifiers for 3 CAN clock inputs.
-- clock-names: 3 clock input name strings: "clkp1", "clkp2", and "can_clk".
-- pinctrl-0: pin control group to be used for this controller.
-- pinctrl-names: must be "default".
-
-Required properties for R8A774A1, R8A774B1, R8A774C0, R8A774E1, R8A7795,
-R8A77960, R8A77961, R8A77965, R8A77990, and R8A77995:
-For the denoted SoCs, "clkp2" can be CANFD clock. This is a div6 clock and can
-be used by both CAN and CAN FD controller at the same time. It needs to be
-scaled to maximum frequency if any of these controllers use it. This is done
-using the below properties:
-
-- assigned-clocks: phandle of clkp2(CANFD) clock.
-- assigned-clock-rates: maximum frequency of this clock.
-
-Optional properties:
-- renesas,can-clock-select: R-Car CAN Clock Source Select. Valid values are:
- <0x0> (default) : Peripheral clock (clkp1)
- <0x1> : Peripheral clock (clkp2)
- <0x3> : External input clock
-
-Example
--------
-
-SoC common .dtsi file:
-
- can0: can@e6e80000 {
- compatible = "renesas,can-r8a7791", "renesas,rcar-gen2-can";
- reg = <0 0xe6e80000 0 0x1000>;
- interrupts = <0 186 IRQ_TYPE_LEVEL_HIGH>;
- clocks = <&mstp9_clks R8A7791_CLK_RCAN0>,
- <&cpg_clocks R8A7791_CLK_RCAN>, <&can_clk>;
- clock-names = "clkp1", "clkp2", "can_clk";
- status = "disabled";
- };
-
-Board specific .dts file:
-
-&can0 {
- pinctrl-0 = <&can0_pins>;
- pinctrl-names = "default";
- status = "okay";
-};
+++ /dev/null
-Renesas R-Car CAN FD controller Device Tree Bindings
-----------------------------------------------------
-
-Required properties:
-- compatible: Must contain one or more of the following:
- - "renesas,rcar-gen3-canfd" for R-Car Gen3 and RZ/G2 compatible controllers.
- - "renesas,r8a774a1-canfd" for R8A774A1 (RZ/G2M) compatible controller.
- - "renesas,r8a774b1-canfd" for R8A774B1 (RZ/G2N) compatible controller.
- - "renesas,r8a774c0-canfd" for R8A774C0 (RZ/G2E) compatible controller.
- - "renesas,r8a774e1-canfd" for R8A774E1 (RZ/G2H) compatible controller.
- - "renesas,r8a7795-canfd" for R8A7795 (R-Car H3) compatible controller.
- - "renesas,r8a7796-canfd" for R8A7796 (R-Car M3-W) compatible controller.
- - "renesas,r8a77965-canfd" for R8A77965 (R-Car M3-N) compatible controller.
- - "renesas,r8a77970-canfd" for R8A77970 (R-Car V3M) compatible controller.
- - "renesas,r8a77980-canfd" for R8A77980 (R-Car V3H) compatible controller.
- - "renesas,r8a77990-canfd" for R8A77990 (R-Car E3) compatible controller.
- - "renesas,r8a77995-canfd" for R8A77995 (R-Car D3) compatible controller.
-
- When compatible with the generic version, nodes must list the
- SoC-specific version corresponding to the platform first, followed by the
- family-specific and/or generic versions.
-
-- reg: physical base address and size of the R-Car CAN FD register map.
-- interrupts: interrupt specifiers for the Channel & Global interrupts
-- clocks: phandles and clock specifiers for 3 clock inputs.
-- clock-names: 3 clock input name strings: "fck", "canfd", "can_clk".
-- pinctrl-0: pin control group to be used for this controller.
-- pinctrl-names: must be "default".
-
-Required child nodes:
-The controller supports two channels and each is represented as a child node.
-The name of the child nodes are "channel0" and "channel1" respectively. Each
-child node supports the "status" property only, which is used to
-enable/disable the respective channel.
-
-Required properties for R8A774A1, R8A774B1, R8A774C0, R8A774E1, R8A7795,
-R8A7796, R8A77965, R8A77990, and R8A77995:
-In the denoted SoCs, canfd clock is a div6 clock and can be used by both CAN
-and CAN FD controller at the same time. It needs to be scaled to maximum
-frequency if any of these controllers use it. This is done using the below
-properties:
-
-- assigned-clocks: phandle of canfd clock.
-- assigned-clock-rates: maximum frequency of this clock.
-
-Optional property:
-The controller can operate in either CAN FD only mode (default) or
-Classical CAN only mode. The mode is global to both the channels. In order to
-enable the later, define the following optional property.
- - renesas,no-can-fd: puts the controller in Classical CAN only mode.
-
-Example
--------
-
-SoC common .dtsi file:
-
- canfd: can@e66c0000 {
- compatible = "renesas,r8a7795-canfd",
- "renesas,rcar-gen3-canfd";
- reg = <0 0xe66c0000 0 0x8000>;
- interrupts = <GIC_SPI 29 IRQ_TYPE_LEVEL_HIGH>,
- <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>;
- clocks = <&cpg CPG_MOD 914>,
- <&cpg CPG_CORE R8A7795_CLK_CANFD>,
- <&can_clk>;
- clock-names = "fck", "canfd", "can_clk";
- assigned-clocks = <&cpg CPG_CORE R8A7795_CLK_CANFD>;
- assigned-clock-rates = <40000000>;
- power-domains = <&cpg>;
- status = "disabled";
-
- channel0 {
- status = "disabled";
- };
-
- channel1 {
- status = "disabled";
- };
- };
-
-Board specific .dts file:
-
-E.g. below enables Channel 1 alone in the board in Classical CAN only mode.
-
-&canfd {
- pinctrl-0 = <&canfd1_pins>;
- pinctrl-names = "default";
- renesas,no-can-fd;
- status = "okay";
-
- channel1 {
- status = "okay";
- };
-};
-
-E.g. below enables Channel 0 alone in the board using External clock
-as fCAN clock.
-
-&canfd {
- pinctrl-0 = <&canfd0_pins>, <&can_clk_pins>;
- pinctrl-names = "default";
- status = "okay";
-
- channel0 {
- status = "okay";
- };
-};
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/net/can/renesas,rcar-can.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Renesas R-Car CAN Controller
+
+maintainers:
+ - Sergei Shtylyov <sergei.shtylyov@gmail.com>
+
+properties:
+ compatible:
+ oneOf:
+ - items:
+ - enum:
+ - renesas,can-r8a7778 # R-Car M1-A
+ - renesas,can-r8a7779 # R-Car H1
+ - const: renesas,rcar-gen1-can # R-Car Gen1
+
+ - items:
+ - enum:
+ - renesas,can-r8a7742 # RZ/G1H
+ - renesas,can-r8a7743 # RZ/G1M
+ - renesas,can-r8a7744 # RZ/G1N
+ - renesas,can-r8a7745 # RZ/G1E
+ - renesas,can-r8a77470 # RZ/G1C
+ - renesas,can-r8a7790 # R-Car H2
+ - renesas,can-r8a7791 # R-Car M2-W
+ - renesas,can-r8a7792 # R-Car V2H
+ - renesas,can-r8a7793 # R-Car M2-N
+ - renesas,can-r8a7794 # R-Car E2
+ - const: renesas,rcar-gen2-can # R-Car Gen2 and RZ/G1
+
+ - items:
+ - enum:
+ - renesas,can-r8a774a1 # RZ/G2M
+ - renesas,can-r8a774b1 # RZ/G2N
+ - renesas,can-r8a774c0 # RZ/G2E
+ - renesas,can-r8a774e1 # RZ/G2H
+ - renesas,can-r8a7795 # R-Car H3
+ - renesas,can-r8a7796 # R-Car M3-W
+ - renesas,can-r8a77961 # R-Car M3-W+
+ - renesas,can-r8a77965 # R-Car M3-N
+ - renesas,can-r8a77990 # R-Car E3
+ - renesas,can-r8a77995 # R-Car D3
+ - const: renesas,rcar-gen3-can # R-Car Gen3 and RZ/G2
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ maxItems: 1
+
+ clocks:
+ maxItems: 3
+
+ clock-names:
+ items:
+ - const: clkp1
+ - const: clkp2
+ - const: can_clk
+
+ power-domains:
+ maxItems: 1
+
+ resets:
+ maxItems: 1
+
+ renesas,can-clock-select:
+ $ref: /schemas/types.yaml#/definitions/uint32
+ enum: [ 0, 1, 3 ]
+ default: 0
+ description: |
+ R-Car CAN Clock Source Select. Valid values are:
+ <0x0> (default) : Peripheral clock (clkp1)
+ <0x1> : Peripheral clock (clkp2)
+ <0x3> : External input clock
+
+ assigned-clocks:
+ description:
+ Reference to the clkp2 (CANFD) clock.
+ On R-Car Gen3 and RZ/G2 SoCs, "clkp2" is the CANFD clock. This is a div6
+ clock and can be used by both CAN and CAN FD controllers at the same
+ time. It needs to be scaled to maximum frequency if any of these
+ controllers use it.
+
+ assigned-clock-rates:
+ description: Maximum frequency of the CANFD clock.
+
+required:
+ - compatible
+ - reg
+ - interrupts
+ - clocks
+ - clock-names
+ - power-domains
+
+allOf:
+ - $ref: can-controller.yaml#
+
+ - if:
+ not:
+ properties:
+ compatible:
+ contains:
+ const: renesas,rcar-gen1-can
+ then:
+ required:
+ - resets
+
+ - if:
+ properties:
+ compatible:
+ contains:
+ const: renesas,rcar-gen3-can
+ then:
+ required:
+ - assigned-clocks
+ - assigned-clock-rates
+
+unevaluatedProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/clock/r8a7791-cpg-mssr.h>
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+ #include <dt-bindings/power/r8a7791-sysc.h>
+
+ can0: can@e6e80000 {
+ compatible = "renesas,can-r8a7791", "renesas,rcar-gen2-can";
+ reg = <0xe6e80000 0x1000>;
+ interrupts = <GIC_SPI 186 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&cpg CPG_MOD 916>,
+ <&cpg CPG_CORE R8A7791_CLK_RCAN>, <&can_clk>;
+ clock-names = "clkp1", "clkp2", "can_clk";
+ power-domains = <&sysc R8A7791_PD_ALWAYS_ON>;
+ resets = <&cpg 916>;
+ };
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/net/can/renesas,rcar-canfd.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Renesas R-Car CAN FD Controller
+
+maintainers:
+ - Fabrizio Castro <fabrizio.castro.jz@renesas.com>
+
+allOf:
+ - $ref: can-controller.yaml#
+
+properties:
+ compatible:
+ oneOf:
+ - items:
+ - enum:
+ - renesas,r8a774a1-canfd # RZ/G2M
+ - renesas,r8a774b1-canfd # RZ/G2N
+ - renesas,r8a774c0-canfd # RZ/G2E
+ - renesas,r8a774e1-canfd # RZ/G2H
+ - renesas,r8a7795-canfd # R-Car H3
+ - renesas,r8a7796-canfd # R-Car M3-W
+ - renesas,r8a77965-canfd # R-Car M3-N
+ - renesas,r8a77970-canfd # R-Car V3M
+ - renesas,r8a77980-canfd # R-Car V3H
+ - renesas,r8a77990-canfd # R-Car E3
+ - renesas,r8a77995-canfd # R-Car D3
+ - const: renesas,rcar-gen3-canfd # R-Car Gen3 and RZ/G2
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ items:
+ - description: Channel interrupt
+ - description: Global interrupt
+
+ clocks:
+ maxItems: 3
+
+ clock-names:
+ items:
+ - const: fck
+ - const: canfd
+ - const: can_clk
+
+ power-domains:
+ maxItems: 1
+
+ resets:
+ maxItems: 1
+
+ renesas,no-can-fd:
+ $ref: /schemas/types.yaml#/definitions/flag
+ description:
+ The controller can operate in either CAN FD only mode (default) or
+ Classical CAN only mode. The mode is global to both the channels.
+ Specify this property to put the controller in Classical CAN only mode.
+
+ assigned-clocks:
+ description:
+ Reference to the CANFD clock. The CANFD clock is a div6 clock and can be
+ used by both CAN (if present) and CAN FD controllers at the same time.
+ It needs to be scaled to maximum frequency if any of these controllers
+ use it.
+
+ assigned-clock-rates:
+ description: Maximum frequency of the CANFD clock.
+
+patternProperties:
+ "^channel[01]$":
+ type: object
+ description:
+ The controller supports two channels and each is represented as a child
+ node. Each child node supports the "status" property only, which
+ is used to enable/disable the respective channel.
+
+required:
+ - compatible
+ - reg
+ - interrupts
+ - clocks
+ - clock-names
+ - power-domains
+ - resets
+ - assigned-clocks
+ - assigned-clock-rates
+ - channel0
+ - channel1
+
+unevaluatedProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/clock/r8a7795-cpg-mssr.h>
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+ #include <dt-bindings/power/r8a7795-sysc.h>
+
+ canfd: can@e66c0000 {
+ compatible = "renesas,r8a7795-canfd",
+ "renesas,rcar-gen3-canfd";
+ reg = <0xe66c0000 0x8000>;
+ interrupts = <GIC_SPI 29 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&cpg CPG_MOD 914>,
+ <&cpg CPG_CORE R8A7795_CLK_CANFD>,
+ <&can_clk>;
+ clock-names = "fck", "canfd", "can_clk";
+ assigned-clocks = <&cpg CPG_CORE R8A7795_CLK_CANFD>;
+ assigned-clock-rates = <40000000>;
+ power-domains = <&sysc R8A7795_PD_ALWAYS_ON>;
+ resets = <&cpg 914>;
+
+ channel0 {
+ };
+
+ channel1 {
+ };
+ };
- gpio-controller: Boolean; if defined, MT7530's LED controller will run on
GPIO mode.
- #gpio-cells: Must be 2 if gpio-controller is defined.
+- interrupt-controller: Boolean; Enables the internal interrupt controller.
+
+If interrupt-controller is defined, the following properties are required.
+
+- #interrupt-cells: Must be 1.
+- interrupts: Parent interrupt for the interrupt controller.
See Documentation/devicetree/bindings/net/dsa/dsa.txt for a list of additional
required, optional properties and how the integrated switch subnodes must
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/net/dsa/nxp,sja1105.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: NXP SJA1105 Automotive Ethernet Switch Family Device Tree Bindings
+
+description:
+ The SJA1105 SPI interface requires a CS-to-CLK time (t2 in UM10944.pdf) of at
+ least one half of t_CLK. At an SPI frequency of 1MHz, this means a minimum
+ cs_sck_delay of 500ns. Ensuring that this SPI timing requirement is observed
+ depends on the SPI bus master driver.
+
+allOf:
+ - $ref: "dsa.yaml#"
+
+maintainers:
+ - Vladimir Oltean <vladimir.oltean@nxp.com>
+
+properties:
+ compatible:
+ enum:
+ - nxp,sja1105e
+ - nxp,sja1105t
+ - nxp,sja1105p
+ - nxp,sja1105q
+ - nxp,sja1105r
+ - nxp,sja1105s
+ - nxp,sja1110a
+ - nxp,sja1110b
+ - nxp,sja1110c
+ - nxp,sja1110d
+
+ reg:
+ maxItems: 1
+
+ # Optional container node for the 2 internal MDIO buses of the SJA1110
+ # (one for the internal 100base-T1 PHYs and the other for the single
+ # 100base-TX PHY). The "reg" property does not have physical significance.
+ # The PHY addresses to port correspondence is as follows: for 100base-T1,
+ # port 5 has PHY 1, port 6 has PHY 2 etc, while for 100base-TX, port 1 has
+ # PHY 1.
+ mdios:
+ type: object
+
+ properties:
+ '#address-cells':
+ const: 1
+ '#size-cells':
+ const: 0
+
+ patternProperties:
+ "^mdio@[0-1]$":
+ type: object
+
+ allOf:
+ - $ref: "http://devicetree.org/schemas/net/mdio.yaml#"
+
+ properties:
+ compatible:
+ oneOf:
+ - enum:
+ - nxp,sja1110-base-t1-mdio
+ - nxp,sja1110-base-tx-mdio
+
+ reg:
+ oneOf:
+ - enum:
+ - 0
+ - 1
+
+ required:
+ - compatible
+ - reg
+
+required:
+ - compatible
+ - reg
+
+unevaluatedProperties: false
+
+examples:
+ - |
+ spi {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ ethernet-switch@1 {
+ reg = <0x1>;
+ compatible = "nxp,sja1105t";
+
+ ethernet-ports {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ port@0 {
+ phy-handle = <&rgmii_phy6>;
+ phy-mode = "rgmii-id";
+ reg = <0>;
+ };
+
+ port@1 {
+ phy-handle = <&rgmii_phy3>;
+ phy-mode = "rgmii-id";
+ reg = <1>;
+ };
+
+ port@2 {
+ phy-handle = <&rgmii_phy4>;
+ phy-mode = "rgmii-id";
+ reg = <2>;
+ };
+
+ port@3 {
+ phy-mode = "rgmii-id";
+ reg = <3>;
+ };
+
+ port@4 {
+ ethernet = <&enet2>;
+ phy-mode = "rgmii";
+ reg = <4>;
+
+ fixed-link {
+ speed = <1000>;
+ full-duplex;
+ };
+ };
+ };
+ };
+ };
Required properties:
- compatible: should be one of:
+ "qca,qca8327"
"qca,qca8334"
"qca,qca8337"
mdio-bus each subnode describing a port needs to have a valid phandle
referencing the internal PHY it is connected to. This is because there's no
N:N mapping of port and PHY id.
+To declare the internal mdio-bus configuration, declare a mdio node in the
+switch node and declare the phandle for the port referencing the internal
+PHY is connected to. In this config a internal mdio-bus is registered and
+the mdio MASTER is used as communication.
Don't use mixed external and internal mdio-bus configurations, as this is
not supported by the hardware.
port@1 {
reg = <1>;
label = "lan1";
+ phy-mode = "internal";
+ phy-handle = <&phy_port1>;
};
port@2 {
reg = <2>;
label = "lan2";
+ phy-mode = "internal";
+ phy-handle = <&phy_port2>;
};
port@3 {
reg = <3>;
label = "lan3";
+ phy-mode = "internal";
+ phy-handle = <&phy_port3>;
};
port@4 {
reg = <4>;
label = "lan4";
+ phy-mode = "internal";
+ phy-handle = <&phy_port4>;
};
port@5 {
reg = <5>;
label = "wan";
+ phy-mode = "internal";
+ phy-handle = <&phy_port5>;
+ };
+ };
+
+ mdio {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ phy_port1: phy@0 {
+ reg = <0>;
+ };
+
+ phy_port2: phy@1 {
+ reg = <1>;
+ };
+
+ phy_port3: phy@2 {
+ reg = <2>;
+ };
+
+ phy_port4: phy@3 {
+ reg = <3>;
+ };
+
+ phy_port5: phy@4 {
+ reg = <4>;
};
};
};
+++ /dev/null
-NXP SJA1105 switch driver
-=========================
-
-Required properties:
-
-- compatible:
- Must be one of:
- - "nxp,sja1105e"
- - "nxp,sja1105t"
- - "nxp,sja1105p"
- - "nxp,sja1105q"
- - "nxp,sja1105r"
- - "nxp,sja1105s"
-
- Although the device ID could be detected at runtime, explicit bindings
- are required in order to be able to statically check their validity.
- For example, SGMII can only be specified on port 4 of R and S devices,
- and the non-SGMII devices, while pin-compatible, are not equal in terms
- of support for RGMII internal delays (supported on P/Q/R/S, but not on
- E/T).
-
-Optional properties:
-
-- sja1105,role-mac:
-- sja1105,role-phy:
- Boolean properties that can be assigned under each port node. By
- default (unless otherwise specified) a port is configured as MAC if it
- is driving a PHY (phy-handle is present) or as PHY if it is PHY-less
- (fixed-link specified, presumably because it is connected to a MAC).
- The effect of this property (in either its implicit or explicit form)
- is:
- - In the case of MII or RMII it specifies whether the SJA1105 port is a
- clock source or sink for this interface (not applicable for RGMII
- where there is a Tx and an Rx clock).
- - In the case of RGMII it affects the behavior regarding internal
- delays:
- 1. If sja1105,role-mac is specified, and the phy-mode property is one
- of "rgmii-id", "rgmii-txid" or "rgmii-rxid", then the entity
- designated to apply the delay/clock skew necessary for RGMII
- is the PHY. The SJA1105 MAC does not apply any internal delays.
- 2. If sja1105,role-phy is specified, and the phy-mode property is one
- of the above, the designated entity to apply the internal delays
- is the SJA1105 MAC (if hardware-supported). This is only supported
- by the second-generation (P/Q/R/S) hardware. On a first-generation
- E or T device, it is an error to specify an RGMII phy-mode other
- than "rgmii" for a port that is in fixed-link mode. In that case,
- the clock skew must either be added by the MAC at the other end of
- the fixed-link, or by PCB serpentine traces on the board.
- These properties are required, for example, in the case where SJA1105
- ports are at both ends of a MII/RMII PHY-less setup. One end would need
- to have sja1105,role-mac, while the other sja1105,role-phy.
-
-See Documentation/devicetree/bindings/net/dsa/dsa.txt for the list of standard
-DSA required and optional properties.
-
-Other observations
-------------------
-
-The SJA1105 SPI interface requires a CS-to-CLK time (t2 in UM10944) of at least
-one half of t_CLK. At an SPI frequency of 1MHz, this means a minimum
-cs_sck_delay of 500ns. Ensuring that this SPI timing requirement is observed
-depends on the SPI bus master driver.
-
-Example
--------
-
-Ethernet switch connected via SPI to the host, CPU port wired to enet2:
-
-arch/arm/boot/dts/ls1021a-tsn.dts:
-
-/* SPI controller of the LS1021 */
-&dspi0 {
- sja1105@1 {
- reg = <0x1>;
- #address-cells = <1>;
- #size-cells = <0>;
- compatible = "nxp,sja1105t";
- spi-max-frequency = <4000000>;
- fsl,spi-cs-sck-delay = <1000>;
- fsl,spi-sck-cs-delay = <1000>;
- ports {
- #address-cells = <1>;
- #size-cells = <0>;
- port@0 {
- /* ETH5 written on chassis */
- label = "swp5";
- phy-handle = <&rgmii_phy6>;
- phy-mode = "rgmii-id";
- reg = <0>;
- /* Implicit "sja1105,role-mac;" */
- };
- port@1 {
- /* ETH2 written on chassis */
- label = "swp2";
- phy-handle = <&rgmii_phy3>;
- phy-mode = "rgmii-id";
- reg = <1>;
- /* Implicit "sja1105,role-mac;" */
- };
- port@2 {
- /* ETH3 written on chassis */
- label = "swp3";
- phy-handle = <&rgmii_phy4>;
- phy-mode = "rgmii-id";
- reg = <2>;
- /* Implicit "sja1105,role-mac;" */
- };
- port@3 {
- /* ETH4 written on chassis */
- phy-handle = <&rgmii_phy5>;
- label = "swp4";
- phy-mode = "rgmii-id";
- reg = <3>;
- /* Implicit "sja1105,role-mac;" */
- };
- port@4 {
- /* Internal port connected to eth2 */
- ethernet = <&enet2>;
- phy-mode = "rgmii";
- reg = <4>;
- /* Implicit "sja1105,role-phy;" */
- fixed-link {
- speed = <1000>;
- full-duplex;
- };
- };
- };
- };
-};
-
-/* MDIO controller of the LS1021 */
-&mdio0 {
- /* BCM5464 */
- rgmii_phy3: ethernet-phy@3 {
- reg = <0x3>;
- };
- rgmii_phy4: ethernet-phy@4 {
- reg = <0x4>;
- };
- rgmii_phy5: ethernet-phy@5 {
- reg = <0x5>;
- };
- rgmii_phy6: ethernet-phy@6 {
- reg = <0x6>;
- };
-};
-
-/* Ethernet master port of the LS1021 */
-&enet2 {
- phy-connection-type = "rgmii";
- status = "ok";
- fixed-link {
- speed = <1000>;
- full-duplex;
- };
-};
- tbi
- rev-mii
- rmii
+ - rev-rmii
# RX and TX delays are added by the MAC when required
- rgmii
- 10gbase-kr
- usxgmii
- 10gbase-r
+ - 25gbase-r
phy-mode:
$ref: "#/properties/phy-connection-type"
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/net/ingenic,mac.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Bindings for MAC in Ingenic SoCs
+
+maintainers:
+ - 周琰杰 (Zhou Yanjie) <zhouyanjie@wanyeetech.com>
+
+description:
+ The Ethernet Media Access Controller in Ingenic SoCs.
+
+properties:
+ compatible:
+ enum:
+ - ingenic,jz4775-mac
+ - ingenic,x1000-mac
+ - ingenic,x1600-mac
+ - ingenic,x1830-mac
+ - ingenic,x2000-mac
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ maxItems: 1
+
+ interrupt-names:
+ const: macirq
+
+ clocks:
+ maxItems: 1
+
+ clock-names:
+ const: stmmaceth
+
+ mode-reg:
+ description: An extra syscon register that control ethernet interface and timing delay
+
+ rx-clk-delay-ps:
+ description: RGMII receive clock delay defined in pico seconds
+
+ tx-clk-delay-ps:
+ description: RGMII transmit clock delay defined in pico seconds
+
+required:
+ - compatible
+ - reg
+ - interrupts
+ - interrupt-names
+ - clocks
+ - clock-names
+ - mode-reg
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/clock/x1000-cgu.h>
+
+ mac: ethernet@134b0000 {
+ compatible = "ingenic,x1000-mac";
+ reg = <0x134b0000 0x2000>;
+
+ interrupt-parent = <&intc>;
+ interrupts = <55>;
+ interrupt-names = "macirq";
+
+ clocks = <&cgu X1000_CLK_MAC>;
+ clock-names = "stmmaceth";
+
+ mode-reg = <&mac_phy_ctrl>;
+ };
+...
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/net/microchip,sparx5-switch.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Microchip Sparx5 Ethernet switch controller
+
+maintainers:
+ - Steen Hegelund <steen.hegelund@microchip.com>
+ - Lars Povlsen <lars.povlsen@microchip.com>
+
+description: |
+ The SparX-5 Enterprise Ethernet switch family provides a rich set of
+ Enterprise switching features such as advanced TCAM-based VLAN and
+ QoS processing enabling delivery of differentiated services, and
+ security through TCAM-based frame processing using versatile content
+ aware processor (VCAP).
+
+ IPv4/IPv6 Layer 3 (L3) unicast and multicast routing is supported
+ with up to 18K IPv4/9K IPv6 unicast LPM entries and up to 9K IPv4/3K
+ IPv6 (S,G) multicast groups.
+
+ L3 security features include source guard and reverse path
+ forwarding (uRPF) tasks. Additional L3 features include VRF-Lite and
+ IP tunnels (IP over GRE/IP).
+
+ The SparX-5 switch family targets managed Layer 2 and Layer 3
+ equipment in SMB, SME, and Enterprise where high port count
+ 1G/2.5G/5G/10G switching with 10G/25G aggregation links is required.
+
+properties:
+ $nodename:
+ pattern: "^switch@[0-9a-f]+$"
+
+ compatible:
+ const: microchip,sparx5-switch
+
+ reg:
+ items:
+ - description: cpu target
+ - description: devices target
+ - description: general control block target
+
+ reg-names:
+ items:
+ - const: cpu
+ - const: devices
+ - const: gcb
+
+ interrupts:
+ minItems: 1
+ items:
+ - description: register based extraction
+ - description: frame dma based extraction
+
+ interrupt-names:
+ minItems: 1
+ items:
+ - const: xtr
+ - const: fdma
+
+ resets:
+ items:
+ - description: Reset controller used for switch core reset (soft reset)
+
+ reset-names:
+ items:
+ - const: switch
+
+ mac-address: true
+
+ ethernet-ports:
+ type: object
+ patternProperties:
+ "^port@[0-9a-f]+$":
+ type: object
+
+ properties:
+ '#address-cells':
+ const: 1
+ '#size-cells':
+ const: 0
+
+ reg:
+ description: Switch port number
+
+ phys:
+ maxItems: 1
+ description:
+ phandle of a Ethernet SerDes PHY. This defines which SerDes
+ instance will handle the Ethernet traffic.
+
+ phy-mode:
+ description:
+ This specifies the interface used by the Ethernet SerDes towards
+ the PHY or SFP.
+
+ microchip,bandwidth:
+ description: Specifies bandwidth in Mbit/s allocated to the port.
+ $ref: "/schemas/types.yaml#/definitions/uint32"
+ maximum: 25000
+
+ phy-handle:
+ description:
+ phandle of a Ethernet PHY. This is optional and if provided it
+ points to the cuPHY used by the Ethernet SerDes.
+
+ sfp:
+ description:
+ phandle of an SFP. This is optional and used when not specifying
+ a cuPHY. It points to the SFP node that describes the SFP used by
+ the Ethernet SerDes.
+
+ managed: true
+
+ microchip,sd-sgpio:
+ description:
+ Index of the ports Signal Detect SGPIO in the set of 384 SGPIOs
+ This is optional, and only needed if the default used index is
+ is not correct.
+ $ref: "/schemas/types.yaml#/definitions/uint32"
+ minimum: 0
+ maximum: 383
+
+ required:
+ - reg
+ - phys
+ - phy-mode
+ - microchip,bandwidth
+
+ oneOf:
+ - required:
+ - phy-handle
+ - required:
+ - sfp
+ - managed
+
+required:
+ - compatible
+ - reg
+ - reg-names
+ - interrupts
+ - interrupt-names
+ - resets
+ - reset-names
+ - ethernet-ports
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+ switch: switch@600000000 {
+ compatible = "microchip,sparx5-switch";
+ reg = <0 0x401000>,
+ <0x10004000 0x7fc000>,
+ <0x11010000 0xaf0000>;
+ reg-names = "cpu", "devices", "gcb";
+ interrupts = <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-names = "xtr";
+ resets = <&reset 0>;
+ reset-names = "switch";
+ ethernet-ports {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ port0: port@0 {
+ reg = <0>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 13>;
+ phy-handle = <&phy0>;
+ phy-mode = "qsgmii";
+ };
+ /* ... */
+ /* Then the 25G interfaces */
+ port60: port@60 {
+ reg = <60>;
+ microchip,bandwidth = <25000>;
+ phys = <&serdes 29>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth60>;
+ managed = "in-band-status";
+ microchip,sd-sgpio = <365>;
+ };
+ port61: port@61 {
+ reg = <61>;
+ microchip,bandwidth = <25000>;
+ phys = <&serdes 30>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth61>;
+ managed = "in-band-status";
+ microchip,sd-sgpio = <369>;
+ };
+ port62: port@62 {
+ reg = <62>;
+ microchip,bandwidth = <25000>;
+ phys = <&serdes 31>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth62>;
+ managed = "in-band-status";
+ microchip,sd-sgpio = <373>;
+ };
+ port63: port@63 {
+ reg = <63>;
+ microchip,bandwidth = <25000>;
+ phys = <&serdes 32>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth63>;
+ managed = "in-band-status";
+ microchip,sd-sgpio = <377>;
+ };
+ /* Finally the Management interface */
+ port64: port@64 {
+ reg = <64>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 0>;
+ phy-handle = <&phy64>;
+ phy-mode = "sgmii";
+ mac-address = [ 00 00 00 01 02 03 ];
+ };
+ };
+ };
+
+...
+# vim: set ts=2 sw=2 sts=2 tw=80 et cc=80 ft=yaml :
reg:
maxItems: 1
+ clocks:
+ maxItems: 1
+
wake-gpios:
maxItems: 1
description:
en-gpios = <&gpf1 4 GPIO_ACTIVE_HIGH>;
wake-gpios = <&gpj0 2 GPIO_ACTIVE_HIGH>;
+
+ clocks = <&rpmcc 20>;
};
};
# UART example on Raspberry Pi
properties:
compatible:
enum:
+ - qcom,msm8998-ipa
- qcom,sc7180-ipa
- qcom,sc7280-ipa
- qcom,sdm845-ipa
- description: Whether the IPA clock is enabled (if valid)
qcom,smem-state-names:
- $ref: /schemas/types.yaml#/definitions/string-array
description: The names of the state bits used for SMP2P output
items:
- const: ipa-clock-enabled-valid
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0+
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/net/realtek,rtl82xx.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Realtek RTL82xx PHY
+
+maintainers:
+ - Andrew Lunn <andrew@lunn.ch>
+ - Florian Fainelli <f.fainelli@gmail.com>
+ - Heiner Kallweit <hkallweit1@gmail.com>
+
+description:
+ Bindings for Realtek RTL82xx PHYs
+
+allOf:
+ - $ref: ethernet-phy.yaml#
+
+properties:
+ realtek,clkout-disable:
+ type: boolean
+ description:
+ Disable CLKOUT clock, CLKOUT clock default is enabled after hardware reset.
+
+
+ realtek,aldps-enable:
+ type: boolean
+ description:
+ Enable ALDPS mode, ALDPS mode default is disabled after hardware reset.
+
+unevaluatedProperties: false
+
+examples:
+ - |
+ mdio {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ ethphy1: ethernet-phy@1 {
+ reg = <1>;
+ realtek,clkout-disable;
+ realtek,aldps-enable;
+ };
+ };
- $ref: ethernet-controller.yaml#
maintainers:
- - Sergei Shtylyov <sergei.shtylyov@cogentembedded.com>
+ - Sergei Shtylyov <sergei.shtylyov@gmail.com>
properties:
compatible:
- rockchip,rk3128-gmac
- rockchip,rk3228-gmac
- rockchip,rk3288-gmac
+ - rockchip,rk3308-gmac
- rockchip,rk3328-gmac
- rockchip,rk3366-gmac
- rockchip,rk3368-gmac
- rockchip,rk3399-gmac
+ - rockchip,rk3568-gmac
- rockchip,rv1108-gmac
required:
- compatible
properties:
compatible:
- items:
- - enum:
- - rockchip,px30-gmac
- - rockchip,rk3128-gmac
- - rockchip,rk3228-gmac
- - rockchip,rk3288-gmac
- - rockchip,rk3328-gmac
- - rockchip,rk3366-gmac
- - rockchip,rk3368-gmac
- - rockchip,rk3399-gmac
- - rockchip,rv1108-gmac
+ oneOf:
+ - items:
+ - enum:
+ - rockchip,px30-gmac
+ - rockchip,rk3128-gmac
+ - rockchip,rk3228-gmac
+ - rockchip,rk3288-gmac
+ - rockchip,rk3308-gmac
+ - rockchip,rk3328-gmac
+ - rockchip,rk3366-gmac
+ - rockchip,rk3368-gmac
+ - rockchip,rk3399-gmac
+ - rockchip,rv1108-gmac
+ - items:
+ - enum:
+ - rockchip,rk3568-gmac
+ - const: snps,dwmac-4.20a
clocks:
minItems: 5
- allwinner,sun8i-r40-emac
- allwinner,sun8i-v3s-emac
- allwinner,sun50i-a64-emac
+ - loongson,ls2k-dwmac
+ - loongson,ls7a-dwmac
- amlogic,meson6-dwmac
- amlogic,meson8b-dwmac
- amlogic,meson8m2-dwmac
- amlogic,meson-gxbb-dwmac
- amlogic,meson-axg-dwmac
+ - loongson,ls2k-dwmac
+ - loongson,ls7a-dwmac
+ - ingenic,jz4775-mac
+ - ingenic,x1000-mac
+ - ingenic,x1600-mac
+ - ingenic,x1830-mac
+ - ingenic,x2000-mac
- rockchip,px30-gmac
- rockchip,rk3128-gmac
- rockchip,rk3228-gmac
- allwinner,sun8i-r40-emac
- allwinner,sun8i-v3s-emac
- allwinner,sun50i-a64-emac
+ - ingenic,jz4775-mac
+ - ingenic,x1000-mac
+ - ingenic,x1600-mac
+ - ingenic,x1830-mac
+ - ingenic,x2000-mac
- snps,dwxgmac
- snps,dwxgmac-2.10
- st,spear600-gmac
- allwinner,sun8i-r40-emac
- allwinner,sun8i-v3s-emac
- allwinner,sun50i-a64-emac
+ - loongson,ls2k-dwmac
+ - loongson,ls7a-dwmac
+ - ingenic,jz4775-mac
+ - ingenic,x1000-mac
+ - ingenic,x1600-mac
+ - ingenic,x1830-mac
+ - ingenic,x2000-mac
- snps,dwmac-4.00
- snps,dwmac-4.10a
- snps,dwmac-4.20a
"mediatek,mt8173-efuse" or "mediatek,efuse": for MT8173
"mediatek,mt8192-efuse", "mediatek,efuse": for MT8192
"mediatek,mt8516-efuse", "mediatek,efuse": for MT8516
- "mediatek,mt8192-efuse", "mediatek,efuse": for MT8192
- reg: Should contain registers location and length
= Data cells =
description:
Specifies the Spread Spectrum Clocking mode used. It can be NO_SSC,
EXTERNAL_SSC or INTERNAL_SSC.
- Refer include/dt-bindings/phy/phy-cadence-torrent.h for the constants to be used.
+ Refer include/dt-bindings/phy/phy-cadence.h for the constants to be used.
$ref: /schemas/types.yaml#/definitions/uint32
enum: [0, 1, 2]
default: 0
maxItems: 1
phys:
- $ref: /schemas/types.yaml#/definitions/phandle
+ maxItems: 1
description: phandle to the USB phy
monitored-battery:
rate
sound-dai:
- $ref: /schemas/types.yaml#/definitions/phandle
+ $ref: /schemas/types.yaml#/definitions/phandle-array
description: phandle of the CPU DAI
patternProperties:
properties:
sound-dai:
- $ref: /schemas/types.yaml#/definitions/phandle
+ $ref: /schemas/types.yaml#/definitions/phandle-array
description: phandle of the codec DAI
required:
maxItems: 1
memory-region:
- $ref: /schemas/types.yaml#/definitions/phandle
+ maxItems: 1
description:
phandle to a node describing reserved memory (System RAM memory)
The M core can't access all the DDR memory space on some platform,
mux-controls = <&mux>;
- spi-flash@0 {
+ flash@0 {
compatible = "jedec,spi-nor";
reg = <0>;
spi-max-frequency = <40000000>;
libnvdimm - kernel / libndctl - userspace helper library
-linux-nvdimm@lists.01.org
+nvdimm@lists.linux.dev
Version 13
moxa-smartio
n_gsm
- rocket
serial-iso7816
serial-rs485
USB-Standard Types
==================
-In ``drivers/usb/common/common.c`` and ``drivers/usb/common/debug.c`` you
-will find the USB data types defined in chapter 9 of the USB specification.
-These data types are used throughout USB, and in APIs including this host
-side API, gadget APIs, usb character devices and debugfs interfaces.
+In ``include/uapi/linux/usb/ch9.h`` you will find the USB data types defined
+in chapter 9 of the USB specification. These data types are used throughout
+USB, and in APIs including this host side API, gadget APIs, usb character
+devices and debugfs interfaces. That file is itself included by
+``include/linux/usb/ch9.h``, which also contains declarations of a few
+utility routines for manipulating these data types; the implementations
+are in ``drivers/usb/common/common.c``.
.. kernel-doc:: drivers/usb/common/common.c
:export:
-.. kernel-doc:: drivers/usb/common/debug.c
- :export:
+In addition, some functions useful for creating debugging output are
+defined in ``drivers/usb/common/debug.c``.
+
+.. _usb_header:
Host-Side Data Types and Macros
===============================
- Support POSIX.1e ACLs by using xattrs;
- - Support transparent file compression as an option:
- LZ4 algorithm with 4 KB fixed-sized output compression for high performance.
+ - Support transparent data compression as an option:
+ LZ4 algorithm with the fixed-sized output compression for high performance.
The following git tree provides the file system user-space tools under
development (ex, formatting tool mkfs.erofs):
::
- |-> aligned with 8B
- |-> followed closely
- + meta_blkaddr blocks |-> another slot
- _____________________________________________________________________
- | ... | inode | xattrs | extents | data inline | ... | inode ...
- |________|_______|(optional)|(optional)|__(optional)_|_____|__________
- |-> aligned with the inode slot size
- . .
- . .
- . .
- . .
- . .
- . .
- .____________________________________________________|-> aligned with 4B
- | xattr_ibody_header | shared xattrs | inline xattrs |
- |____________________|_______________|_______________|
- |-> 12 bytes <-|->x * 4 bytes<-| .
- . . .
- . . .
- . . .
- ._______________________________.______________________.
- | id | id | id | id | ... | id | ent | ... | ent| ... |
- |____|____|____|____|______|____|_____|_____|____|_____|
- |-> aligned with 4B
- |-> aligned with 4B
+ |-> aligned with 8B
+ |-> followed closely
+ + meta_blkaddr blocks |-> another slot
+ _____________________________________________________________________
+ | ... | inode | xattrs | extents | data inline | ... | inode ...
+ |________|_______|(optional)|(optional)|__(optional)_|_____|__________
+ |-> aligned with the inode slot size
+ . .
+ . .
+ . .
+ . .
+ . .
+ . .
+ .____________________________________________________|-> aligned with 4B
+ | xattr_ibody_header | shared xattrs | inline xattrs |
+ |____________________|_______________|_______________|
+ |-> 12 bytes <-|->x * 4 bytes<-| .
+ . . .
+ . . .
+ . . .
+ ._______________________________.______________________.
+ | id | id | id | id | ... | id | ent | ... | ent| ... |
+ |____|____|____|____|______|____|_____|_____|____|_____|
+ |-> aligned with 4B
+ |-> aligned with 4B
Inode could be 32 or 64 bytes, which can be distinguished from a common
field which all inode versions have -- i_format::
Each share xattr can also be directly found by the following formula:
xattr offset = xattr_blkaddr * block_size + 4 * xattr_id
- ::
+::
- |-> aligned by 4 bytes
- + xattr_blkaddr blocks |-> aligned with 4 bytes
- _________________________________________________________________________
- | ... | xattr_entry | xattr data | ... | xattr_entry | xattr data ...
- |________|_____________|_____________|_____|______________|_______________
+ |-> aligned by 4 bytes
+ + xattr_blkaddr blocks |-> aligned with 4 bytes
+ _________________________________________________________________________
+ | ... | xattr_entry | xattr data | ... | xattr_entry | xattr data ...
+ |________|_____________|_____________|_____|______________|_______________
Directories
-----------
::
- ___________________________
- / |
- / ______________|________________
- / / | nameoff1 | nameoffN-1
- ____________.______________._______________v________________v__________
- | dirent | dirent | ... | dirent | filename | filename | ... | filename |
- |___.0___|____1___|_____|___N-1__|____0_____|____1_____|_____|___N-1____|
- \ ^
- \ | * could have
- \ | trailing '\0'
- \________________________| nameoff0
-
- Directory block
+ ___________________________
+ / |
+ / ______________|________________
+ / / | nameoff1 | nameoffN-1
+ ____________.______________._______________v________________v__________
+ | dirent | dirent | ... | dirent | filename | filename | ... | filename |
+ |___.0___|____1___|_____|___N-1__|____0_____|____1_____|_____|___N-1____|
+ \ ^
+ \ | * could have
+ \ | trailing '\0'
+ \________________________| nameoff0
+ Directory block
Note that apart from the offset of the first filename, nameoff0 also indicates
the total number of directory entries in this block since it is no need to
introduce another on-disk field at all.
-Compression
------------
-Currently, EROFS supports 4KB fixed-sized output transparent file compression,
-as illustrated below::
-
- |---- Variant-Length Extent ----|-------- VLE --------|----- VLE -----
- clusterofs clusterofs clusterofs
- | | | logical data
- _________v_______________________________v_____________________v_______________
- ... | . | | . | | . | ...
- ____|____.________|_____________|________.____|_____________|__.__________|____
- |-> cluster <-|-> cluster <-|-> cluster <-|-> cluster <-|-> cluster <-|
- size size size size size
- . . . .
- . . . .
- . . . .
- _______._____________._____________._____________._____________________
- ... | | | | ... physical data
- _______|_____________|_____________|_____________|_____________________
- |-> cluster <-|-> cluster <-|-> cluster <-|
- size size size
-
-Currently each on-disk physical cluster can contain 4KB (un)compressed data
-at most. For each logical cluster, there is a corresponding on-disk index to
-describe its cluster type, physical cluster address, etc.
-
-See "struct z_erofs_vle_decompressed_index" in erofs_fs.h for more details.
+Data compression
+----------------
+EROFS implements LZ4 fixed-sized output compression which generates fixed-sized
+compressed data blocks from variable-sized input in contrast to other existing
+fixed-sized input solutions. Relatively higher compression ratios can be gotten
+by using fixed-sized output compression since nowadays popular data compression
+algorithms are mostly LZ77-based and such fixed-sized output approach can be
+benefited from the historical dictionary (aka. sliding window).
+
+In details, original (uncompressed) data is turned into several variable-sized
+extents and in the meanwhile, compressed into physical clusters (pclusters).
+In order to record each variable-sized extent, logical clusters (lclusters) are
+introduced as the basic unit of compress indexes to indicate whether a new
+extent is generated within the range (HEAD) or not (NONHEAD). Lclusters are now
+fixed in block size, as illustrated below::
+
+ |<- variable-sized extent ->|<- VLE ->|
+ clusterofs clusterofs clusterofs
+ | | |
+ _________v_________________________________v_______________________v________
+ ... | . | | . | | . ...
+ ____|____._________|______________|________.___ _|______________|__.________
+ |-> lcluster <-|-> lcluster <-|-> lcluster <-|-> lcluster <-|
+ (HEAD) (NONHEAD) (HEAD) (NONHEAD) .
+ . CBLKCNT . .
+ . . .
+ . . .
+ _______._____________________________.______________._________________
+ ... | | | | ...
+ _______|______________|______________|______________|_________________
+ |-> big pcluster <-|-> pcluster <-|
+
+A physical cluster can be seen as a container of physical compressed blocks
+which contains compressed data. Previously, only lcluster-sized (4KB) pclusters
+were supported. After big pcluster feature is introduced (available since
+Linux v5.13), pcluster can be a multiple of lcluster size.
+
+For each HEAD lcluster, clusterofs is recorded to indicate where a new extent
+starts and blkaddr is used to seek the compressed data. For each NONHEAD
+lcluster, delta0 and delta1 are available instead of blkaddr to indicate the
+distance to its HEAD lcluster and the next HEAD lcluster. A PLAIN lcluster is
+also a HEAD lcluster except that its data is uncompressed. See the comments
+around "struct z_erofs_vle_decompressed_index" in erofs_fs.h for more details.
+
+If big pcluster is enabled, pcluster size in lclusters needs to be recorded as
+well. Let the delta0 of the first NONHEAD lcluster store the compressed block
+count with a special flag as a new called CBLKCNT NONHEAD lcluster. It's easy
+to understand its delta0 is constantly 1, as illustrated below::
+
+ __________________________________________________________
+ | HEAD | NONHEAD | NONHEAD | ... | NONHEAD | HEAD | HEAD |
+ |__:___|_(CBLKCNT)_|_________|_____|_________|__:___|____:_|
+ |<----- a big pcluster (with CBLKCNT) ------>|<-- -->|
+ a lcluster-sized pcluster (without CBLKCNT) ^
+
+If another HEAD follows a HEAD lcluster, there is no room to record CBLKCNT,
+but it's easy to know the size of such pcluster is 1 lcluster as well.
--- /dev/null
+.. SPDX-License-Identifier: GPL-2.0
+
+=========================
+MDIO bus and PHYs in ACPI
+=========================
+
+The PHYs on an MDIO bus [1] are probed and registered using
+fwnode_mdiobus_register_phy().
+
+Later, for connecting these PHYs to their respective MACs, the PHYs registered
+on the MDIO bus have to be referenced.
+
+This document introduces two _DSD properties that are to be used
+for connecting PHYs on the MDIO bus [3] to the MAC layer.
+
+These properties are defined in accordance with the "Device
+Properties UUID For _DSD" [2] document and the
+daffd814-6eba-4d8c-8a91-bc9bbf4aa301 UUID must be used in the Device
+Data Descriptors containing them.
+
+phy-handle
+----------
+For each MAC node, a device property "phy-handle" is used to reference
+the PHY that is registered on an MDIO bus. This is mandatory for
+network interfaces that have PHYs connected to MAC via MDIO bus.
+
+During the MDIO bus driver initialization, PHYs on this bus are probed
+using the _ADR object as shown below and are registered on the MDIO bus.
+
+.. code-block:: none
+
+ Scope(\_SB.MDI0)
+ {
+ Device(PHY1) {
+ Name (_ADR, 0x1)
+ } // end of PHY1
+
+ Device(PHY2) {
+ Name (_ADR, 0x2)
+ } // end of PHY2
+ }
+
+Later, during the MAC driver initialization, the registered PHY devices
+have to be retrieved from the MDIO bus. For this, the MAC driver needs
+references to the previously registered PHYs which are provided
+as device object references (e.g. \_SB.MDI0.PHY1).
+
+phy-mode
+--------
+The "phy-mode" _DSD property is used to describe the connection to
+the PHY. The valid values for "phy-mode" are defined in [4].
+
+managed
+-------
+Optional property, which specifies the PHY management type.
+The valid values for "managed" are defined in [4].
+
+fixed-link
+----------
+The "fixed-link" is described by a data-only subnode of the
+MAC port, which is linked in the _DSD package via
+hierarchical data extension (UUID dbb8e3e6-5886-4ba6-8795-1319f52a966b
+in accordance with [5] "_DSD Implementation Guide" document).
+The subnode should comprise a required property ("speed") and
+possibly the optional ones - complete list of parameters and
+their values are specified in [4].
+
+The following ASL example illustrates the usage of these properties.
+
+DSDT entry for MDIO node
+------------------------
+
+The MDIO bus has an SoC component (MDIO controller) and a platform
+component (PHYs on the MDIO bus).
+
+a) Silicon Component
+This node describes the MDIO controller, MDI0
+---------------------------------------------
+
+.. code-block:: none
+
+ Scope(_SB)
+ {
+ Device(MDI0) {
+ Name(_HID, "NXP0006")
+ Name(_CCA, 1)
+ Name(_UID, 0)
+ Name(_CRS, ResourceTemplate() {
+ Memory32Fixed(ReadWrite, MDI0_BASE, MDI_LEN)
+ Interrupt(ResourceConsumer, Level, ActiveHigh, Shared)
+ {
+ MDI0_IT
+ }
+ }) // end of _CRS for MDI0
+ } // end of MDI0
+ }
+
+b) Platform Component
+The PHY1 and PHY2 nodes represent the PHYs connected to MDIO bus MDI0
+---------------------------------------------------------------------
+
+.. code-block:: none
+
+ Scope(\_SB.MDI0)
+ {
+ Device(PHY1) {
+ Name (_ADR, 0x1)
+ } // end of PHY1
+
+ Device(PHY2) {
+ Name (_ADR, 0x2)
+ } // end of PHY2
+ }
+
+DSDT entries representing MAC nodes
+-----------------------------------
+
+Below are the MAC nodes where PHY nodes are referenced.
+phy-mode and phy-handle are used as explained earlier.
+------------------------------------------------------
+
+.. code-block:: none
+
+ Scope(\_SB.MCE0.PR17)
+ {
+ Name (_DSD, Package () {
+ ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
+ Package () {
+ Package (2) {"phy-mode", "rgmii-id"},
+ Package (2) {"phy-handle", \_SB.MDI0.PHY1}
+ }
+ })
+ }
+
+ Scope(\_SB.MCE0.PR18)
+ {
+ Name (_DSD, Package () {
+ ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
+ Package () {
+ Package (2) {"phy-mode", "rgmii-id"},
+ Package (2) {"phy-handle", \_SB.MDI0.PHY2}}
+ }
+ })
+ }
+
+MAC node example where "managed" property is specified.
+-------------------------------------------------------
+
+.. code-block:: none
+
+ Scope(\_SB.PP21.ETH0)
+ {
+ Name (_DSD, Package () {
+ ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
+ Package () {
+ Package () {"phy-mode", "sgmii"},
+ Package () {"managed", "in-band-status"}
+ }
+ })
+ }
+
+MAC node example with a "fixed-link" subnode.
+---------------------------------------------
+
+.. code-block:: none
+
+ Scope(\_SB.PP21.ETH1)
+ {
+ Name (_DSD, Package () {
+ ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
+ Package () {
+ Package () {"phy-mode", "sgmii"},
+ },
+ ToUUID("dbb8e3e6-5886-4ba6-8795-1319f52a966b"),
+ Package () {
+ Package () {"fixed-link", "LNK0"}
+ }
+ })
+ Name (LNK0, Package(){ // Data-only subnode of port
+ ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
+ Package () {
+ Package () {"speed", 1000},
+ Package () {"full-duplex", 1}
+ }
+ })
+ }
+
+References
+==========
+
+[1] Documentation/networking/phy.rst
+
+[2] https://www.uefi.org/sites/default/files/resources/_DSD-device-properties-UUID.pdf
+
+[3] Documentation/firmware-guide/acpi/DSD-properties-rules.rst
+
+[4] Documentation/devicetree/bindings/net/ethernet-controller.yaml
+
+[5] https://github.com/UEFI/DSD-Guide/blob/main/dsd-guide.pdf
dsd/graph
dsd/data-node-references
dsd/leds
+ dsd/phy
enumeration
osi
method-customizing
The TMP103 is a digital output temperature sensor in a four-ball
wafer chip-scale package (WCSP). The TMP103 is capable of reading
temperatures to a resolution of 1°C. The TMP103 is specified for
-operation over a temperature range of –40°C to +125°C.
+operation over a temperature range of -40°C to +125°C.
Resolution: 8 Bits
-Accuracy: ±1°C Typ (–10°C to +100°C)
+Accuracy: ±1°C Typ (-10°C to +100°C)
The driver provides the common sysfs-interface for temperatures (see
Documentation/hwmon/sysfs-interface.rst under Temperatures).
2. Packet format
================
-a. MAP packet (data / control)
+a. MAP packet v1 (data / control)
-MAP header has the same endianness of the IP packet.
+MAP header fields are in big endian format.
Packet format::
- Bit 0 1 2-7 8 - 15 16 - 31
+ Bit 0 1 2-7 8-15 16-31
Function Command / Data Reserved Pad Multiplexer ID Payload length
- Bit 32 - x
- Function Raw Bytes
+
+ Bit 32-x
+ Function Raw bytes
Command (1)/ Data (0) bit value is to indicate if the packet is a MAP command
-or data packet. Control packet is used for transport level flow control. Data
+or data packet. Command packet is used for transport level flow control. Data
packets are standard IP packets.
-Reserved bits are usually zeroed out and to be ignored by receiver.
+Reserved bits must be zero when sent and ignored when received.
-Padding is number of bytes to be added for 4 byte alignment if required by
-hardware.
+Padding is the number of bytes to be appended to the payload to
+ensure 4 byte alignment.
Multiplexer ID is to indicate the PDN on which data has to be sent.
Payload length includes the padding length but does not include MAP header
length.
-b. MAP packet (command specific)::
+b. Map packet v4 (data / control)
+
+MAP header fields are in big endian format.
+
+Packet format::
+
+ Bit 0 1 2-7 8-15 16-31
+ Function Command / Data Reserved Pad Multiplexer ID Payload length
+
+ Bit 32-(x-33) (x-32)-x
+ Function Raw bytes Checksum offload header
+
+Command (1)/ Data (0) bit value is to indicate if the packet is a MAP command
+or data packet. Command packet is used for transport level flow control. Data
+packets are standard IP packets.
+
+Reserved bits must be zero when sent and ignored when received.
+
+Padding is the number of bytes to be appended to the payload to
+ensure 4 byte alignment.
+
+Multiplexer ID is to indicate the PDN on which data has to be sent.
+
+Payload length includes the padding length but does not include MAP header
+length.
+
+Checksum offload header, has the information about the checksum processing done
+by the hardware.Checksum offload header fields are in big endian format.
+
+Packet format::
+
+ Bit 0-14 15 16-31
+ Function Reserved Valid Checksum start offset
+
+ Bit 31-47 48-64
+ Function Checksum length Checksum value
+
+Reserved bits must be zero when sent and ignored when received.
+
+Valid bit indicates whether the partial checksum is calculated and is valid.
+Set to 1, if its is valid. Set to 0 otherwise.
+
+Padding is the number of bytes to be appended to the payload to
+ensure 4 byte alignment.
+
+Checksum start offset, Indicates the offset in bytes from the beginning of the
+IP header, from which modem computed checksum.
+
+Checksum length is the Length in bytes starting from CKSUM_START_OFFSET,
+over which checksum is computed.
+
+Checksum value, indicates the checksum computed.
+
+c. MAP packet v5 (data / control)
+
+MAP header fields are in big endian format.
+
+Packet format::
+
+ Bit 0 1 2-7 8-15 16-31
+ Function Command / Data Next header Pad Multiplexer ID Payload length
+
+ Bit 32-x
+ Function Raw bytes
+
+Command (1)/ Data (0) bit value is to indicate if the packet is a MAP command
+or data packet. Command packet is used for transport level flow control. Data
+packets are standard IP packets.
+
+Next header is used to indicate the presence of another header, currently is
+limited to checksum header.
+
+Padding is the number of bytes to be appended to the payload to
+ensure 4 byte alignment.
+
+Multiplexer ID is to indicate the PDN on which data has to be sent.
+
+Payload length includes the padding length but does not include MAP header
+length.
+
+d. Checksum offload header v5
+
+Checksum offload header fields are in big endian format.
+
+ Bit 0 - 6 7 8-15 16-31
+ Function Header Type Next Header Checksum Valid Reserved
+
+Header Type is to indicate the type of header, this usually is set to CHECKSUM
+
+Header types
+= ==========================================
+0 Reserved
+1 Reserved
+2 checksum header
+
+Checksum Valid is to indicate whether the header checksum is valid. Value of 1
+implies that checksum is calculated on this packet and is valid, value of 0
+indicates that the calculated packet checksum is invalid.
+
+Reserved bits must be zero when sent and ignored when received.
+
+e. MAP packet v1/v5 (command specific)::
- Bit 0 1 2-7 8 - 15 16 - 31
+ Bit 0 1 2-7 8 - 15 16 - 31
Function Command Reserved Pad Multiplexer ID Payload length
Bit 32 - 39 40 - 45 46 - 47 48 - 63
Function Command name Reserved Command Type Reserved
3 is for error during processing of commands
= ==========================================
-c. Aggregation
+f. Aggregation
Aggregation is multiple MAP packets (can be data or command) delivered to
rmnet in a single linear skb. rmnet will process the individual
features and system architectures.
The ENA device exposes a lightweight management interface with a
-minimal set of memory mapped registers and extendable command set
+minimal set of memory mapped registers and extendible command set
through an Admin Queue.
The driver supports a range of ENA devices, is link-speed independent
-(i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has
-a negotiated and extendable feature set.
+(i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc), and has
+a negotiated and extendible feature set.
Some ENA devices support SR-IOV. This driver is used for both the
SR-IOV Physical Function (PF) and Virtual Function (VF) devices.
interrupt vector per Tx/Rx queue pair, adaptive interrupt moderation,
and CPU cacheline optimized data placement.
-The ENA driver supports industry standard TCP/IP offload features such
-as checksum offload and TCP transmit segmentation offload (TSO).
-Receive-side scaling (RSS) is supported for multi-core scaling.
+The ENA driver supports industry standard TCP/IP offload features such as
+checksum offload. Receive-side scaling (RSS) is supported for multi-core
+scaling.
The ENA driver and its corresponding devices implement health
monitoring mechanisms such as watchdog, enabling the device and driver
Some of the ENA devices support a working mode called Low-latency
Queue (LLQ), which saves several more microseconds.
-
ENA Source Code Directory Structure
===================================
================= ======================================================
ena_com.[ch] Management communication layer. This layer is
- responsible for the handling all the management
- (admin) communication between the device and the
- driver.
+ responsible for the handling all the management
+ (admin) communication between the device and the
+ driver.
ena_eth_com.[ch] Tx/Rx data path.
ena_admin_defs.h Definition of ENA management interface.
ena_eth_io_defs.h Definition of ENA data path interface.
ena_common_defs.h Common definitions for ena_com layer.
ena_regs_defs.h Definition of ENA PCI memory-mapped (MMIO) registers.
ena_netdev.[ch] Main Linux kernel driver.
-ena_syfsfs.[ch] Sysfs files.
ena_ethtool.c ethtool callbacks.
ena_pci_id_tbl.h Supported device IDs.
================= ======================================================
- Asynchronous Event Notification Queue (AENQ)
ENA device MMIO Registers are accessed only during driver
-initialization and are not involved in further normal device
+initialization and are not used during further normal device
operation.
AQ is used for submitting management commands, and the
The events are:
- ==================== ===============
- Group Syndrome
- ==================== ===============
- Link state change **X**
- Fatal error **X**
- Notification Suspend traffic
- Notification Resume traffic
- Keep-Alive **X**
- ==================== ===============
+==================== ===============
+Group Syndrome
+==================== ===============
+Link state change **X**
+Fatal error **X**
+Notification Suspend traffic
+Notification Resume traffic
+Keep-Alive **X**
+==================== ===============
ACQ and AENQ share the same MSI-X vector.
-Keep-Alive is a special mechanism that allows monitoring of the
-device's health. The driver maintains a watchdog (WD) handler which,
-if fired, logs the current state and statistics then resets and
-restarts the ENA device and driver. A Keep-Alive event is delivered by
-the device every second. The driver re-arms the WD upon reception of a
-Keep-Alive event. A missed Keep-Alive event causes the WD handler to
-fire.
+Keep-Alive is a special mechanism that allows monitoring the device's health.
+A Keep-Alive event is delivered by the device every second.
+The driver maintains a watchdog (WD) handler which logs the current state and
+statistics. If the keep-alive events aren't delivered as expected the WD resets
+the device and the driver.
Data Path Interface
===================
+
I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx
SQ correspondingly). Each SQ has a completion queue (CQ) associated
with it.
The ENA driver supports two Queue Operation modes for Tx SQs:
-- Regular mode
+- **Regular mode:**
+ In this mode the Tx SQs reside in the host's memory. The ENA
+ device fetches the ENA Tx descriptors and packet data from host
+ memory.
- * In this mode the Tx SQs reside in the host's memory. The ENA
- device fetches the ENA Tx descriptors and packet data from host
- memory.
+- **Low Latency Queue (LLQ) mode or "push-mode":**
+ In this mode the driver pushes the transmit descriptors and the
+ first 128 bytes of the packet directly to the ENA device memory
+ space. The rest of the packet payload is fetched by the
+ device. For this operation mode, the driver uses a dedicated PCI
+ device memory BAR, which is mapped with write-combine capability.
-- Low Latency Queue (LLQ) mode or "push-mode".
-
- * In this mode the driver pushes the transmit descriptors and the
- first 128 bytes of the packet directly to the ENA device memory
- space. The rest of the packet payload is fetched by the
- device. For this operation mode, the driver uses a dedicated PCI
- device memory BAR, which is mapped with write-combine capability.
+ **Note that** not all ENA devices support LLQ, and this feature is negotiated
+ with the device upon initialization. If the ENA device does not
+ support LLQ mode, the driver falls back to the regular mode.
The Rx SQs support only the regular mode.
-Note: Not all ENA devices support LLQ, and this feature is negotiated
- with the device upon initialization. If the ENA device does not
- support LLQ mode, the driver falls back to the regular mode.
-
The driver supports multi-queue for both Tx and Rx. This has various
benefits:
Interrupt Modes
===============
+
The driver assigns a single MSI-X vector per queue pair (for both Tx
and Rx directions). The driver assigns an additional dedicated MSI-X vector
for management (for ACQ and AENQ).
Interrupt Moderation
====================
+
ENA driver and device can operate in conventional or adaptive interrupt
moderation mode.
-In conventional mode the driver instructs device to postpone interrupt
+**In conventional mode** the driver instructs device to postpone interrupt
posting according to static interrupt delay value. The interrupt delay
-value can be configured through ethtool(8). The following ethtool
-parameters are supported by the driver: tx-usecs, rx-usecs
+value can be configured through `ethtool(8)`. The following `ethtool`
+parameters are supported by the driver: ``tx-usecs``, ``rx-usecs``
-In adaptive interrupt moderation mode the interrupt delay value is
+**In adaptive interrupt** moderation mode the interrupt delay value is
updated by the driver dynamically and adjusted every NAPI cycle
according to the traffic nature.
-Adaptive coalescing can be switched on/off through ethtool(8)
-adaptive_rx on|off parameter.
+Adaptive coalescing can be switched on/off through `ethtool(8)`'s
+:code:`adaptive_rx on|off` parameter.
More information about Adaptive Interrupt Moderation (DIM) can be found in
Documentation/networking/net_dim.rst
and can be configured by the ETHTOOL_STUNABLE command of the
SIOCETHTOOL ioctl.
-SKB
-===
-The driver-allocated SKB for frames received from Rx handling using
-NAPI context. The allocation method depends on the size of the packet.
-If the frame length is larger than rx_copybreak, napi_get_frags()
-is used, otherwise netdev_alloc_skb_ip_align() is used, the buffer
-content is copied (by CPU) to the SKB, and the buffer is recycled.
-
Statistics
==========
-The user can obtain ENA device and driver statistics using ethtool.
+
+The user can obtain ENA device and driver statistics using `ethtool`.
The driver can collect regular or extended statistics (including
per-queue stats) from the device.
MTU
===
+
The driver supports an arbitrarily large MTU with a maximum that is
negotiated with the device. The driver configures MTU using the
SetFeature command (ENA_ADMIN_MTU property). The user can change MTU
-via ip(8) and similar legacy tools.
+via `ip(8)` and similar legacy tools.
Stateless Offloads
==================
+
The ENA driver supports:
-- TSO over IPv4/IPv6
-- TSO with ECN
- IPv4 header checksum offload
- TCP/UDP over IPv4/IPv6 checksum offloads
RSS
===
+
- The ENA device supports RSS that allows flexible Rx traffic
steering.
- Toeplitz and CRC32 hash functions are supported.
function delivered in the Rx CQ descriptor is set in the received
SKB.
- The user can provide a hash key, hash function, and configure the
- indirection table through ethtool(8).
+ indirection table through `ethtool(8)`.
DATA PATH
=========
+
Tx
--
-ena_start_xmit() is called by the stack. This function does the following:
+:code:`ena_start_xmit()` is called by the stack. This function does the following:
-- Maps data buffers (skb->data and frags).
-- Populates ena_buf for the push buffer (if the driver and device are
- in push mode.)
+- Maps data buffers (``skb->data`` and frags).
+- Populates ``ena_buf`` for the push buffer (if the driver and device are
+ in push mode).
- Prepares ENA bufs for the remaining frags.
-- Allocates a new request ID from the empty req_id ring. The request
+- Allocates a new request ID from the empty ``req_id`` ring. The request
ID is the index of the packet in the Tx info. This is used for
- out-of-order TX completions.
+ out-of-order Tx completions.
- Adds the packet to the proper place in the Tx ring.
-- Calls ena_com_prepare_tx(), an ENA communication layer that converts
- the ena_bufs to ENA descriptors (and adds meta ENA descriptors as
- needed.)
+- Calls :code:`ena_com_prepare_tx()`, an ENA communication layer that converts
+ the ``ena_bufs`` to ENA descriptors (and adds meta ENA descriptors as
+ needed).
* This function also copies the ENA descriptors and the push buffer
- to the Device memory space (if in push mode.)
+ to the Device memory space (if in push mode).
-- Writes doorbell to the ENA device.
+- Writes a doorbell to the ENA device.
- When the ENA device finishes sending the packet, a completion
interrupt is raised.
- The interrupt handler schedules NAPI.
-- The ena_clean_tx_irq() function is called. This function handles the
+- The :code:`ena_clean_tx_irq()` function is called. This function handles the
completion descriptors generated by the ENA, with a single
completion descriptor per completed packet.
- * req_id is retrieved from the completion descriptor. The tx_info of
- the packet is retrieved via the req_id. The data buffers are
- unmapped and req_id is returned to the empty req_id ring.
+ * ``req_id`` is retrieved from the completion descriptor. The ``tx_info`` of
+ the packet is retrieved via the ``req_id``. The data buffers are
+ unmapped and ``req_id`` is returned to the empty ``req_id`` ring.
* The function stops when the completion descriptors are completed or
the budget is reached.
- When a packet is received from the ENA device.
- The interrupt handler schedules NAPI.
-- The ena_clean_rx_irq() function is called. This function calls
- ena_rx_pkt(), an ENA communication layer function, which returns the
- number of descriptors used for a new unhandled packet, and zero if
+- The :code:`ena_clean_rx_irq()` function is called. This function calls
+ :code:`ena_com_rx_pkt()`, an ENA communication layer function, which returns the
+ number of descriptors used for a new packet, and zero if
no new packet is found.
-- Then it calls the ena_clean_rx_irq() function.
-- ena_eth_rx_skb() checks packet length:
+- :code:`ena_rx_skb()` checks packet length:
* If the packet is small (len < rx_copybreak), the driver allocates
a SKB for the new packet, and copies the packet payload into the
- In this way the original data buffer is not passed to the stack
and is reused for future Rx packets.
- * Otherwise the function unmaps the Rx buffer, then allocates the
- new SKB structure and hooks the Rx buffer to the SKB frags.
+ * Otherwise the function unmaps the Rx buffer, sets the first
+ descriptor as `skb`'s linear part and the other descriptors as the
+ `skb`'s frags.
- The new SKB is updated with the necessary information (protocol,
- checksum hw verify result, etc.), and then passed to the network
- stack, using the NAPI interface function napi_gro_receive().
+ checksum hw verify result, etc), and then passed to the network
+ stack, using the NAPI interface function :code:`napi_gro_receive()`.
- Transmit and Receive Queues
- See description below
+Descriptor Formats
+------------------
+GVE supports two descriptor formats: GQI and DQO. These two formats have
+entirely different descriptors, which will be described below.
+
Registers
---------
-All registers are MMIO and big endian.
+All registers are MMIO.
The registers are used for initializing and configuring the device as well as
querying device status in response to management interrupts.
+Endianness
+----------
+- Admin Queue messages and registers are all Big Endian.
+- GQI descriptors and datapath registers are Big Endian.
+- DQO descriptors and datapath registers are Little Endian.
+
Admin Queue (AQ)
----------------
The Admin Queue is a PAGE_SIZE memory block, treated as an array of AQ
The handler for these irqs schedule the napi for that block to run
and poll the queues.
-Traffic Queues
---------------
-gVNIC's queues are composed of a descriptor ring and a buffer and are
-assigned to a notification block.
+GQI Traffic Queues
+------------------
+GQI queues are composed of a descriptor ring and a buffer and are assigned to a
+notification block.
The descriptor rings are power-of-two-sized ring buffers consisting of
fixed-size descriptors. They advance their head pointer using a __be32
The buffers for receive rings are put into a data ring that is the same
length as the descriptor ring and the head and tail pointers advance over
the rings together.
+
+DQO Traffic Queues
+------------------
+- Every TX and RX queue is assigned a notification block.
+
+- TX and RX buffers queues, which send descriptors to the device, use MMIO
+ doorbells to notify the device of new descriptors.
+
+- RX and TX completion queues, which receive descriptors from the device, use a
+ "generation bit" to know when a descriptor was populated by the device. The
+ driver initializes all bits with the "current generation". The device will
+ populate received descriptors with the "next generation" which is inverted
+ from the current generation. When the ring wraps, the current/next generation
+ are swapped.
+
+- It's the driver's responsibility to ensure that the RX and TX completion
+ queues are not overrun. This can be accomplished by limiting the number of
+ descriptors posted to HW.
+
+- TX packets have a 16 bit completion_tag and RX buffers have a 16 bit
+ buffer_id. These will be returned on the TX completion and RX queues
+ respectively to let the driver know which packet/buffer was completed.
+
+Transmit
+~~~~~~~~
+A packet's buffers are DMA mapped for the device to access before transmission.
+After the packet was successfully transmitted, the buffers are unmapped.
+
+Receive
+~~~~~~~
+The driver posts fixed sized buffers to HW on the RX buffer queue. The packet
+received on the associated RX queue may span multiple descriptors.
driver. To re-enable ATR, the sideband can be disabled with the ethtool -K
option. For example::
- ethtool –K [adapter] ntuple [off|on]
+ ethtool -K [adapter] ntuple [off|on]
If sideband is re-enabled after ATR is re-enabled, ATR remains enabled until a
TCP-IP flow is added. When all TCP-IP sideband rules are deleted, ATR is
Totals must be equal or less than port speed.
For example: min_rate 1Gbit 3Gbit: Verify bandwidth limit using network
-monitoring tools such as ifstat or sar –n DEV [interval] [number of samples]
+monitoring tools such as `ifstat` or `sar -n DEV [interval] [number of samples]`
2. Enable HW TC offload on interface::
Totals must be equal or less than port speed.
For example: min_rate 1Gbit 3Gbit: Verify bandwidth limit using network
-monitoring tools such as ifstat or sar –n DEV [interval] [number of samples]
+monitoring tools such as ``ifstat`` or ``sar -n DEV [interval] [number of samples]``
NOTE:
Setting up channels via ethtool (ethtool -L) is not supported when the
- `Enabling the driver and kconfig options`_
- `Devlink info`_
- `Devlink parameters`_
+- `Bridge offload`_
- `mlx5 subfunction`_
- `mlx5 function attributes`_
- `Devlink health reporters`_
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
+Bridge offload
+==============
+The mlx5 driver implements support for offloading bridge rules when in switchdev
+mode. Linux bridge FDBs are automatically offloaded when mlx5 switchdev
+representor is attached to bridge.
+
+- Change device to switchdev mode::
+
+ $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
+
+- Attach mlx5 switchdev representor 'enp8s0f0' to bridge netdev 'bridge1'::
+
+ $ ip link set enp8s0f0 master bridge1
+
+VLANs
+-----
+Following bridge VLAN functions are supported by mlx5:
+
+- VLAN filtering (including multiple VLANs per port)::
+
+ $ ip link set bridge1 type bridge vlan_filtering 1
+ $ bridge vlan add dev enp8s0f0 vid 2-3
+
+- VLAN push on bridge ingress::
+
+ $ bridge vlan add dev enp8s0f0 vid 3 pvid
+
+- VLAN pop on bridge egress::
+
+ $ bridge vlan add dev enp8s0f0 vid 3 untagged
+
mlx5 subfunction
================
mlx5 supports subfunction management using devlink port (see :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`) interface.
$ cat /sys/kernel/debug/tracing/trace
...
kworker/u48:7-2221 [009] ...1 1475.387435: mlx5e_rep_neigh_update: netdev: ens1f0 MAC: 24:8a:07:9a:17:9a IPv4: 1.1.1.10 IPv6: ::ffff:1.1.1.10 neigh_connected=1
+
+Bridge offloads tracepoints:
+
+- mlx5_esw_bridge_fdb_entry_init: trace bridge FDB entry offloaded to mlx5::
+
+ $ echo mlx5:mlx5_esw_bridge_fdb_entry_init >> set_event
+ $ cat /sys/kernel/debug/tracing/trace
+ ...
+ kworker/u20:9-2217 [003] ...1 318.582243: mlx5_esw_bridge_fdb_entry_init: net_device=enp8s0f0_0 addr=e4:fd:05:08:00:02 vid=0 flags=0 used=0
+
+- mlx5_esw_bridge_fdb_entry_cleanup: trace bridge FDB entry deleted from mlx5::
+
+ $ echo mlx5:mlx5_esw_bridge_fdb_entry_cleanup >> set_event
+ $ cat /sys/kernel/debug/tracing/trace
+ ...
+ ip-2581 [005] ...1 318.629871: mlx5_esw_bridge_fdb_entry_cleanup: net_device=enp8s0f0_1 addr=e4:fd:05:08:00:03 vid=0 flags=0 used=16
+
+- mlx5_esw_bridge_fdb_entry_refresh: trace bridge FDB entry offload refreshed in
+ mlx5::
+
+ $ echo mlx5:mlx5_esw_bridge_fdb_entry_refresh >> set_event
+ $ cat /sys/kernel/debug/tracing/trace
+ ...
+ kworker/u20:8-3849 [003] ...1 466716: mlx5_esw_bridge_fdb_entry_refresh: net_device=enp8s0f0_0 addr=e4:fd:05:08:00:02 vid=3 flags=0 used=0
+
+- mlx5_esw_bridge_vlan_create: trace bridge VLAN object add on mlx5
+ representor::
+
+ $ echo mlx5:mlx5_esw_bridge_vlan_create >> set_event
+ $ cat /sys/kernel/debug/tracing/trace
+ ...
+ ip-2560 [007] ...1 318.460258: mlx5_esw_bridge_vlan_create: vid=1 flags=6
+
+- mlx5_esw_bridge_vlan_cleanup: trace bridge VLAN object delete from mlx5
+ representor::
+
+ $ echo mlx5:mlx5_esw_bridge_vlan_cleanup >> set_event
+ $ cat /sys/kernel/debug/tracing/trace
+ ...
+ bridge-2582 [007] ...1 318.653496: mlx5_esw_bridge_vlan_cleanup: vid=2 flags=8
+
+- mlx5_esw_bridge_vport_init: trace mlx5 vport assigned with bridge upper
+ device::
+
+ $ echo mlx5:mlx5_esw_bridge_vport_init >> set_event
+ $ cat /sys/kernel/debug/tracing/trace
+ ...
+ ip-2560 [007] ...1 318.458915: mlx5_esw_bridge_vport_init: vport_num=1
+
+- mlx5_esw_bridge_vport_cleanup: trace mlx5 vport removed from bridge upper
+ device::
+
+ $ echo mlx5:mlx5_esw_bridge_vport_cleanup >> set_event
+ $ cat /sys/kernel/debug/tracing/trace
+ ...
+ ip-5387 [000] ...1 573713: mlx5_esw_bridge_vport_cleanup: vport_num=1
qlogic/index
wan/index
wifi/index
+ wwan/index
.. only:: subproject and html
--- /dev/null
+.. SPDX-License-Identifier: GPL-2.0-only
+
+WWAN Device Drivers
+===================
+
+Contents:
+
+.. toctree::
+ :maxdepth: 2
+
+ iosm
+
+.. only:: subproject and html
+
+ Indices
+ =======
+
+ * :ref:`genindex`
--- /dev/null
+.. SPDX-License-Identifier: GPL-2.0-only
+
+.. Copyright (C) 2020-21 Intel Corporation
+
+.. _iosm_driver_doc:
+
+===========================================
+IOSM Driver for Intel M.2 PCIe based Modems
+===========================================
+The IOSM (IPC over Shared Memory) driver is a WWAN PCIe host driver developed
+for linux or chrome platform for data exchange over PCIe interface between
+Host platform & Intel M.2 Modem. The driver exposes interface conforming to the
+MBIM protocol [1]. Any front end application ( eg: Modem Manager) could easily
+manage the MBIM interface to enable data communication towards WWAN.
+
+Basic usage
+===========
+MBIM functions are inactive when unmanaged. The IOSM driver only provides a
+userspace interface MBIM "WWAN PORT" representing MBIM control channel and does
+not play any role in managing the functionality. It is the job of a userspace
+application to detect port enumeration and enable MBIM functionality.
+
+Examples of few such userspace application are:
+- mbimcli (included with the libmbim [2] library), and
+- Modem Manager [3]
+
+Management Applications to carry out below required actions for establishing
+MBIM IP session:
+- open the MBIM control channel
+- configure network connection settings
+- connect to network
+- configure IP network interface
+
+Management application development
+==================================
+The driver and userspace interfaces are described below. The MBIM protocol is
+described in [1] Mobile Broadband Interface Model v1.0 Errata-1.
+
+MBIM control channel userspace ABI
+----------------------------------
+
+/dev/wwan0mbim0 character device
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The driver exposes an MBIM interface to the MBIM function by implementing
+MBIM WWAN Port. The userspace end of the control channel pipe is a
+/dev/wwan0mbim0 character device. Application shall use this interface for
+MBIM protocol communication.
+
+Fragmentation
+~~~~~~~~~~~~~
+The userspace application is responsible for all control message fragmentation
+and defragmentation as per MBIM specification.
+
+/dev/wwan0mbim0 write()
+~~~~~~~~~~~~~~~~~~~~~~~
+The MBIM control messages from the management application must not exceed the
+negotiated control message size.
+
+/dev/wwan0mbim0 read()
+~~~~~~~~~~~~~~~~~~~~~~
+The management application must accept control messages of up the negotiated
+control message size.
+
+MBIM data channel userspace ABI
+-------------------------------
+
+wwan0-X network device
+~~~~~~~~~~~~~~~~~~~~~~
+The IOSM driver exposes IP link interface "wwan0-X" of type "wwan" for IP
+traffic. Iproute network utility is used for creating "wwan0-X" network
+interface and for associating it with MBIM IP session. The Driver supports
+upto 8 IP sessions for simultaneous IP communication.
+
+The userspace management application is responsible for creating new IP link
+prior to establishing MBIM IP session where the SessionId is greater than 0.
+
+For example, creating new IP link for a MBIM IP session with SessionId 1:
+
+ ip link add dev wwan0-1 parentdev-name wwan0 type wwan linkid 1
+
+The driver will automatically map the "wwan0-1" network device to MBIM IP
+session 1.
+
+References
+==========
+[1] "MBIM (Mobile Broadband Interface Model) Errata-1"
+ - https://www.usb.org/document-library/
+
+[2] libmbim - "a glib-based library for talking to WWAN modems and
+ devices which speak the Mobile Interface Broadband Model (MBIM)
+ protocol"
+ - http://www.freedesktop.org/wiki/Software/libmbim/
+
+[3] Modem Manager - "a DBus-activated daemon which controls mobile
+ broadband (2G/3G/4G) devices and connections"
+ - http://www.freedesktop.org/wiki/Software/ModemManager/
A subfunction device is created on the :ref:`Documentation/driver-api/auxiliary_bus.rst <auxiliary_bus>`.
At this point a matching subfunction driver binds to the subfunction's auxiliary device.
+Rate object management
+======================
+
+Devlink provides API to manage tx rates of single devlink port or a group.
+This is done through rate objects, which can be one of the two types:
+
+``leaf``
+ Represents a single devlink port; created/destroyed by the driver. Since leaf
+ have 1to1 mapping to its devlink port, in user space it is referred as
+ ``pci/<bus_addr>/<port_index>``;
+
+``node``
+ Represents a group of rate objects (leafs and/or nodes); created/deleted by
+ request from the userspace; initially empty (no rate objects added). In
+ userspace it is referred as ``pci/<bus_addr>/<node_name>``, where
+ ``node_name`` can be any identifier, except decimal number, to avoid
+ collisions with leafs.
+
+API allows to configure following rate object's parameters:
+
+``tx_share``
+ Minimum TX rate value shared among all other rate objects, or rate objects
+ that parts of the parent group, if it is a part of the same group.
+
+``tx_max``
+ Maximum TX rate value.
+
+``parent``
+ Parent node name. Parent node rate limits are considered as additional limits
+ to all node children limits. ``tx_max`` is an upper limit for children.
+ ``tx_share`` is a total bandwidth distributed among children.
+
+Driver implementations are allowed to support both or either rate object types
+and setting methods of their parameters.
+
Terms and Definitions
=====================
* :doc:`netdevsim`
* :doc:`mlxsw`
+ * :doc:`prestera`
.. _Generic-Packet-Trap-Groups:
qed
ti-cpsw-switch
am65-nuss-cpsw-switch
+ prestera
$ devlink resource set netdevsim/netdevsim0 path /nexthops size 16
$ devlink dev reload netdevsim/netdevsim0
+Rate objects
+============
+
+The ``netdevsim`` driver supports rate objects management, which includes:
+
+- registerging/unregistering leaf rate objects per VF devlink port;
+- creation/deletion node rate objects;
+- setting tx_share and tx_max rate values for any rate object type;
+- setting parent node for any rate object type.
+
+Rate nodes and it's parameters are exposed in ``netdevsim`` debugfs in RO mode.
+For example created rate node with name ``some_group``:
+
+.. code:: shell
+
+ $ ls /sys/kernel/debug/netdevsim/netdevsim0/rate_groups/some_group
+ rate_parent tx_max tx_share
+
+Same parameters are exposed for leaf objects in corresponding ports directories.
+For ex.:
+
+.. code:: shell
+
+ $ ls /sys/kernel/debug/netdevsim/netdevsim0/ports/1
+ dev ethtool rate_parent tx_max tx_share
+
Driver-specific Traps
=====================
--- /dev/null
+.. SPDX-License-Identifier: GPL-2.0
+
+========================
+prestera devlink support
+========================
+
+This document describes the devlink features implemented by the ``prestera``
+device driver.
+
+Driver-specific Traps
+=====================
+
+.. list-table:: List of Driver-specific Traps Registered by ``prestera``
+ :widths: 5 5 90
+
+ * - Name
+ - Type
+ - Description
+.. list-table:: List of Driver-specific Traps Registered by ``prestera``
+ :widths: 5 5 90
+
+ * - Name
+ - Type
+ - Description
+ * - ``arp_bc``
+ - ``trap``
+ - Traps ARP broadcast packets (both requests/responses)
+ * - ``is_is``
+ - ``trap``
+ - Traps IS-IS packets
+ * - ``ospf``
+ - ``trap``
+ - Traps OSPF packets
+ * - ``ip_bc_mac``
+ - ``trap``
+ - Traps IPv4 packets with broadcast DA Mac address
+ * - ``stp``
+ - ``trap``
+ - Traps STP BPDU
+ * - ``lacp``
+ - ``trap``
+ - Traps LACP packets
+ * - ``lldp``
+ - ``trap``
+ - Traps LLDP packets
+ * - ``router_mc``
+ - ``trap``
+ - Traps multicast packets
+ * - ``vrrp``
+ - ``trap``
+ - Traps VRRP packets
+ * - ``dhcp``
+ - ``trap``
+ - Traps DHCP packets
+ * - ``mtu_error``
+ - ``trap``
+ - Traps (exception) packets that exceeded port's MTU
+ * - ``mac_to_me``
+ - ``trap``
+ - Traps packets with switch-port's DA Mac address
+ * - ``ttl_error``
+ - ``trap``
+ - Traps (exception) IPv4 packets whose TTL exceeded
+ * - ``ipv4_options``
+ - ``trap``
+ - Traps (exception) packets due to the malformed IPV4 header options
+ * - ``ip_default_route``
+ - ``trap``
+ - Traps packets that have no specific IP interface (IP to me) and no forwarding prefix
+ * - ``local_route``
+ - ``trap``
+ - Traps packets that have been send to one of switch IP interfaces addresses
+ * - ``ipv4_icmp_redirect``
+ - ``trap``
+ - Traps (exception) IPV4 ICMP redirect packets
+ * - ``arp_response``
+ - ``trap``
+ - Traps ARP replies packets that have switch-port's DA Mac address
+ * - ``acl_code_0``
+ - ``trap``
+ - Traps packets that have ACL priority set to 0 (tc pref 0)
+ * - ``acl_code_1``
+ - ``trap``
+ - Traps packets that have ACL priority set to 1 (tc pref 1)
+ * - ``acl_code_2``
+ - ``trap``
+ - Traps packets that have ACL priority set to 2 (tc pref 2)
+ * - ``acl_code_3``
+ - ``trap``
+ - Traps packets that have ACL priority set to 3 (tc pref 3)
+ * - ``acl_code_4``
+ - ``trap``
+ - Traps packets that have ACL priority set to 4 (tc pref 4)
+ * - ``acl_code_5``
+ - ``trap``
+ - Traps packets that have ACL priority set to 5 (tc pref 5)
+ * - ``acl_code_6``
+ - ``trap``
+ - Traps packets that have ACL priority set to 6 (tc pref 6)
+ * - ``acl_code_7``
+ - ``trap``
+ - Traps packets that have ACL priority set to 7 (tc pref 7)
+ * - ``ipv4_bgp``
+ - ``trap``
+ - Traps IPv4 BGP packets
+ * - ``ssh``
+ - ``trap``
+ - Traps SSH packets
+ * - ``telnet``
+ - ``trap``
+ - Traps Telnet packets
+ * - ``icmp``
+ - ``trap``
+ - Traps ICMP packets
+ * - ``rxdma_drop``
+ - ``drop``
+ - Drops packets (RxDMA) due to the lack of ingress buffers etc.
+ * - ``port_no_vlan``
+ - ``drop``
+ - Drops packets due to faulty-configured network or due to internal bug (config issue).
+ * - ``local_port``
+ - ``drop``
+ - Drops packets whose decision (FDB entry) is to bridge packet back to the incoming port/trunk.
+ * - ``invalid_sa``
+ - ``drop``
+ - Drops packets with multicast source MAC address.
+ * - ``illegal_ip_addr``
+ - ``drop``
+ - Drops packets with illegal SIP/DIP multicast/unicast addresses.
+ * - ``illegal_ipv4_hdr``
+ - ``drop``
+ - Drops packets with illegal IPV4 header.
+ * - ``ip_uc_dip_da_mismatch``
+ - ``drop``
+ - Drops packets with destination MAC being unicast, but destination IP address being multicast.
+ * - ``ip_sip_is_zero``
+ - ``drop``
+ - Drops packets with zero (0) IPV4 source address.
+ * - ``met_red``
+ - ``drop``
+ - Drops non-conforming packets (dropped by Ingress policer, metering drop), e.g. packet rate exceeded configured bandwith.
the tag length might vary (for example packets with PTP timestamps might
require an extended switch tag, or there might be one tag length on TX and a
different one on RX). Either way, the tagging protocol driver must populate the
-``struct dsa_device_ops::overhead`` with the length in octets of the longest
-switch frame header. The DSA framework will automatically adjust the MTU of the
-master interface to accomodate for this extra size in order for DSA user ports
-to support the standard MTU (L2 payload length) of 1500 octets. The ``overhead``
-is also used to request from the network stack, on a best-effort basis, the
-allocation of packets with a ``needed_headroom`` or ``needed_tailroom``
-sufficient such that the act of pushing the switch tag on transmission of a
-packet does not cause it to reallocate due to lack of memory.
+``struct dsa_device_ops::needed_headroom`` and/or ``struct dsa_device_ops::needed_tailroom``
+with the length in octets of the longest switch frame header/trailer. The DSA
+framework will automatically adjust the MTU of the master interface to
+accommodate for this extra size in order for DSA user ports to support the
+standard MTU (L2 payload length) of 1500 octets. The ``needed_headroom`` and
+``needed_tailroom`` properties are also used to request from the network stack,
+on a best-effort basis, the allocation of packets with enough extra space such
+that the act of pushing the switch tag on transmission of a packet does not
+cause it to reallocate due to lack of memory.
Even though applications are not expected to parse DSA-specific frame headers,
the format on the wire of the tagging protocol represents an Application Binary
understand what egress port the packet is for (and not deliver it towards other
ports). Typically this is fulfilled by pushing a frame header. Checking for
insufficient size in the skb headroom or tailroom is unnecessary provided that
-the ``overhead`` and ``tail_tag`` properties were filled out properly, because
-DSA ensures there is enough space before calling this method.
+the ``needed_headroom`` and ``needed_tailroom`` properties were filled out
+properly, because DSA ensures there is enough space before calling this method.
The reception of a packet goes through the tagger's ``rcv`` function. The
passed ``struct sk_buff *skb`` has ``skb->data`` pointing at
Overview
========
-The NXP SJA1105 is a family of 6 devices:
+The NXP SJA1105 is a family of 10 SPI-managed automotive switches:
- SJA1105E: First generation, no TTEthernet
- SJA1105T: First generation, TTEthernet
- SJA1105Q: Second generation, TTEthernet, no SGMII
- SJA1105R: Second generation, no TTEthernet, SGMII
- SJA1105S: Second generation, TTEthernet, SGMII
-
-These are SPI-managed automotive switches, with all ports being gigabit
-capable, and supporting MII/RMII/RGMII and optionally SGMII on one port.
+- SJA1110A: Third generation, TTEthernet, SGMII, integrated 100base-T1 and
+ 100base-TX PHYs
+- SJA1110B: Third generation, TTEthernet, SGMII, 100base-T1, 100base-TX
+- SJA1110C: Third generation, TTEthernet, SGMII, 100base-T1, 100base-TX
+- SJA1110D: Third generation, TTEthernet, SGMII, 100base-T1
Being automotive parts, their configuration interface is geared towards
set-and-forget use, with minimal dynamic interaction at runtime. They
MDIO bus available to Linux within the system (e.g. to the DSA master's MDIO
bus). Link state management then works by the driver manually keeping in sync
(over SPI commands) the MAC link speed with the settings negotiated by the PHY.
+
+By comparison, the SJA1110 supports an MDIO slave access point over which its
+internal 100base-T1 PHYs can be accessed from the host. This is, however, not
+used by the driver, instead the internal 100base-T1 and 100base-TX PHYs are
+accessed through SPI commands, modeled in Linux as virtual MDIO buses.
+
+The microcontroller attached to the SJA1110 port 0 also has an MDIO controller
+operating in master mode, however the driver does not support this either,
+since the microcontroller gets disabled when the Linux driver operates.
+Discrete PHYs connected to the switch ports should have their MDIO interface
+attached to an MDIO controller from the host system and not to the switch,
+similar to SJA1105.
+
+Port compatibility matrix
+-------------------------
+
+The SJA1105 port compatibility matrix is:
+
+===== ============== ============== ==============
+Port SJA1105E/T SJA1105P/Q SJA1105R/S
+===== ============== ============== ==============
+0 xMII xMII xMII
+1 xMII xMII xMII
+2 xMII xMII xMII
+3 xMII xMII xMII
+4 xMII xMII SGMII
+===== ============== ============== ==============
+
+
+The SJA1110 port compatibility matrix is:
+
+===== ============== ============== ============== ==============
+Port SJA1110A SJA1110B SJA1110C SJA1110D
+===== ============== ============== ============== ==============
+0 RevMII (uC) RevMII (uC) RevMII (uC) RevMII (uC)
+1 100base-TX 100base-TX 100base-TX
+ or SGMII SGMII
+2 xMII xMII xMII xMII
+ or SGMII or SGMII
+3 xMII xMII xMII
+ or SGMII or SGMII SGMII
+ or 2500base-X or 2500base-X or 2500base-X
+4 SGMII SGMII SGMII SGMII
+ or 2500base-X or 2500base-X or 2500base-X or 2500base-X
+5 100base-T1 100base-T1 100base-T1 100base-T1
+6 100base-T1 100base-T1 100base-T1 100base-T1
+7 100base-T1 100base-T1 100base-T1 100base-T1
+8 100base-T1 100base-T1 n/a n/a
+9 100base-T1 100base-T1 n/a n/a
+10 100base-T1 n/a n/a n/a
+===== ============== ============== ============== ==============
``ETHTOOL_A_FEC_AUTO`` requests the driver to choose FEC mode based on SFP
module parameters. This does not mean autonegotiation.
-MODULE_EEPROM
-=============
+MODULE_EEPROM_GET
+=================
Fetch module EEPROM data dump.
This interface is designed to allow dumps of at most 1/2 page at once. This
``ETHTOOL_A_MODULE_EEPROM_I2C_ADDRESS`` u8 page I2C address
======================================= ====== ==========================
+If ``ETHTOOL_A_MODULE_EEPROM_BANK`` is not specified, bank 0 is assumed.
+
Kernel response contents:
+---------------------------------------------+--------+---------------------+
| ``ETHTOOL_A_MODULE_EEPROM_HEADER`` | nested | reply header |
+---------------------------------------------+--------+---------------------+
- | ``ETHTOOL_A_MODULE_EEPROM_DATA`` | nested | array of bytes from |
+ | ``ETHTOOL_A_MODULE_EEPROM_DATA`` | binary | array of bytes from |
| | | module EEPROM |
+---------------------------------------------+--------+---------------------+
- 0 - Layer 3
- 1 - Layer 4
- 2 - Layer 3 or inner Layer 3 if present
+ - 3 - Custom multipath hash. Fields used for multipath hash calculation
+ are determined by fib_multipath_hash_fields sysctl
+
+fib_multipath_hash_fields - UNSIGNED INTEGER
+ When fib_multipath_hash_policy is set to 3 (custom multipath hash), the
+ fields used for multipath hash calculation are determined by this
+ sysctl.
+
+ This value is a bitmask which enables various fields for multipath hash
+ calculation.
+
+ Possible fields are:
+
+ ====== ============================
+ 0x0001 Source IP address
+ 0x0002 Destination IP address
+ 0x0004 IP protocol
+ 0x0008 Unused (Flow Label)
+ 0x0010 Source port
+ 0x0020 Destination port
+ 0x0040 Inner source IP address
+ 0x0080 Inner destination IP address
+ 0x0100 Inner IP protocol
+ 0x0200 Inner Flow Label
+ 0x0400 Inner source port
+ 0x0800 Inner destination port
+ ====== ============================
+
+ Default: 0x0007 (source IP, destination IP and IP protocol)
fib_sync_mem - UNSIGNED INTEGER
Amount of dirty memory from fib entries that can be backlogged before
network connections you can set this knob to 2 to enable
unconditionally generation of syncookies.
+tcp_migrate_req - BOOLEAN
+ The incoming connection is tied to a specific listening socket when
+ the initial SYN packet is received during the three-way handshake.
+ When a listener is closed, in-flight request sockets during the
+ handshake and established sockets in the accept queue are aborted.
+
+ If the listener has SO_REUSEPORT enabled, other listeners on the
+ same port should have been able to accept such connections. This
+ option makes it possible to migrate such child sockets to another
+ listener after close() or shutdown().
+
+ The BPF_SK_REUSEPORT_SELECT_OR_MIGRATE type of eBPF program should
+ usually be used to define the policy to pick an alive listener.
+ Otherwise, the kernel will randomly pick an alive listener only if
+ this option is enabled.
+
+ Note that migration between listeners with different settings may
+ crash applications. Let's say migration happens from listener A to
+ B, and only B has TCP_SAVE_SYN enabled. B cannot read SYN data from
+ the requests migrated from A. To avoid such a situation, cancel
+ migration by returning SK_DROP in the type of eBPF program, or
+ disable this option.
+
+ Default: 0
+
tcp_fastopen - INTEGER
Enable TCP Fast Open (RFC7413) to send and accept data in the opening
SYN packet.
- 0 - Layer 3 (source and destination addresses plus flow label)
- 1 - Layer 4 (standard 5-tuple)
- 2 - Layer 3 or inner Layer 3 if present
+ - 3 - Custom multipath hash. Fields used for multipath hash calculation
+ are determined by fib_multipath_hash_fields sysctl
+
+fib_multipath_hash_fields - UNSIGNED INTEGER
+ When fib_multipath_hash_policy is set to 3 (custom multipath hash), the
+ fields used for multipath hash calculation are determined by this
+ sysctl.
+
+ This value is a bitmask which enables various fields for multipath hash
+ calculation.
+
+ Possible fields are:
+
+ ====== ============================
+ 0x0001 Source IP address
+ 0x0002 Destination IP address
+ 0x0004 IP protocol
+ 0x0008 Flow Label
+ 0x0010 Source port
+ 0x0020 Destination port
+ 0x0040 Inner source IP address
+ 0x0080 Inner destination IP address
+ 0x0100 Inner IP protocol
+ 0x0200 Inner Flow Label
+ 0x0400 Inner source port
+ 0x0800 Inner destination port
+ ====== ============================
+
+ Default: 0x0007 (source IP, destination IP and IP protocol)
anycast_src_echo_reply - BOOLEAN
Controls the use of anycast addresses as source addresses for ICMPv6
Default: 0
+plpmtud_probe_interval - INTEGER
+ The time interval (in milliseconds) for the PLPMTUD probe timer,
+ which is configured to expire after this period to receive an
+ acknowledgment to a probe packet. This is also the time interval
+ between the probes for the current pmtu when the probe search
+ is done.
+
+ PLPMTUD will be disabled when 0 is set, and other values for it
+ must be >= 5000.
+
+ Default: 0
+
``/proc/sys/net/core/*``
========================
/proc/sys/net/mptcp/* Variables
===============================
-enabled - INTEGER
+enabled - BOOLEAN
Control whether MPTCP sockets can be created.
- MPTCP sockets can be created if the value is nonzero. This is
- a per-namespace sysctl.
+ MPTCP sockets can be created if the value is 1. This is a
+ per-namespace sysctl.
- Default: 1
+ Default: 1 (enabled)
add_addr_timeout - INTEGER (seconds)
Set the timeout after which an ADD_ADDR control message will be
sysctl.
Default: 120
+
+checksum_enabled - BOOLEAN
+ Control whether DSS checksum can be enabled.
+
+ DSS checksum can be enabled if the value is nonzero. This is a
+ per-namespace sysctl.
+
+ Default: 0
+
+allow_join_initial_addr_port - BOOLEAN
+ Allow peers to send join requests to the IP address and port number used
+ by the initial subflow if the value is 1. This controls a flag that is
+ sent to the peer at connection time, and whether such join requests are
+ accepted or denied.
+
+ Joins to addresses advertised with ADD_ADDR are not affected by this
+ value.
+
+ This is a per-namespace sysctl.
+
+ Default: 1
This extended timeout will be used in case there is an GRE stream
detected.
+
+nf_flowtable_tcp_timeout - INTEGER (seconds)
+ default 30
+
+ Control offload timeout for tcp connections.
+ TCP connections may be offloaded from nf conntrack to nf flow table.
+ Once aged, the connection is returned to nf conntrack with tcp pickup timeout.
+
+nf_flowtable_tcp_pickup - INTEGER (seconds)
+ default 120
+
+ TCP connection timeout after being aged from nf flow table offload.
+
+nf_flowtable_udp_timeout - INTEGER (seconds)
+ default 30
+
+ Control offload timeout for udp connections.
+ UDP connections may be offloaded from nf conntrack to nf flow table.
+ Once aged, the connection is returned to nf conntrack with udp pickup timeout.
+
+nf_flowtable_udp_pickup - INTEGER (seconds)
+ default 30
+
+ UDP connection timeout after being aged from nf flow table offload.
Note: due to legacy usage, some 10GBASE-R usage incorrectly makes
use of this definition.
+``PHY_INTERFACE_MODE_25GBASER``
+ This is the IEEE 802.3 PCS Clause 107 defined 25GBASE-R protocol.
+ The PCS is identical to 10GBASE-R, i.e. 64B/66B encoded
+ running 2.5 as fast, giving a fixed bit rate of 25.78125 Gbaud.
+ Please refer to the IEEE standard for further information.
+
``PHY_INTERFACE_MODE_100BASEX``
This defines IEEE 802.3 Clause 24. The link operates at a fixed data
rate of 125Mpbs using a 4B/5B encoding scheme, resulting in an underlying
scv 0 syscalls will always behave as PPC_FEATURE2_HTM_NOSC.
+ptrace
+------
+When ptracing system calls (PTRACE_SYSCALL), the pt_regs.trap value contains
+the system call type that can be used to distinguish between sc and scv 0
+system calls, and the different register conventions can be accounted for.
+
+If the value of (pt_regs.trap & 0xfff0) is 0xc00 then the system call was
+performed with the sc instruction, if it is 0x3000 then the system call was
+performed with the scv 0 instruction.
+
vsyscall
========
-.. _process_statement_kernel:
+.. _process_statement_kernel:
Linux Kernel Enforcement Statement
----------------------------------
calling set_sched_topology() with this array as the parameter.
The sched-domains debugging infrastructure can be enabled by enabling
-CONFIG_SCHED_DEBUG and adding 'sched_debug_verbose' to your cmdline. If you
+CONFIG_SCHED_DEBUG and adding 'sched_verbose' to your cmdline. If you
forgot to tweak your cmdline, you can also flip the
/sys/kernel/debug/sched/verbose knob. This enables an error checking parse of
the sched domains which should catch most possible errors (described above). It
-=============================
+=============================
Virtual TPM interface for Xen
=============================
-======================================
+======================================
NO_HZ: Reducing Scheduling-Clock Ticks
======================================
+++ /dev/null
-Chinese translated version of Documentation/admin-guide/security-bugs.rst
-
-If you have any comment or update to the content, please contact the
-original document maintainer directly. However, if you have a problem
-communicating in English you can also ask the Chinese maintainer for
-help. Contact the Chinese maintainer if this translation is outdated
-or if there is a problem with the translation.
-
-Chinese maintainer: Harry Wei <harryxiyou@gmail.com>
----------------------------------------------------------------------
-Documentation/admin-guide/security-bugs.rst 的中文翻译
-
-如果想评论或更新本文的内容，请直接联系原文档的维护者。如果你使用英文
-交流有困难的话，也可以向中文版维护者求助。如果本翻译更新不及时或者翻
-译存在问题，请联系中文版维护者。
-
-中文版维护者： 贾威威 Harry Wei <harryxiyou@gmail.com>
-中文版翻译者： 贾威威 Harry Wei <harryxiyou@gmail.com>
-中文版校译者： 贾威威 Harry Wei <harryxiyou@gmail.com>
-
-
-以下为正文
----------------------------------------------------------------------
-Linux内核开发者认为安全非常重要。因此，我们想要知道当一个有关于
-安全的漏洞被发现的时候，并且它可能会被尽快的修复或者公开。请把这个安全
-漏洞报告给Linux内核安全团队。
-
-1) 联系
-
-linux内核安全团队可以通过email<security@kernel.org>来联系。这是
-一组独立的安全工作人员，可以帮助改善漏洞报告并且公布和取消一个修复。安
-全团队有可能会从部分的维护者那里引进额外的帮助来了解并且修复安全漏洞。
-当遇到任何漏洞，所能提供的信息越多就越能诊断和修复。如果你不清楚什么
-是有帮助的信息，那就请重温一下admin-guide/reporting-bugs.rst文件中的概述过程。任
-何攻击性的代码都是非常有用的，未经报告者的同意不会被取消，除非它已经
-被公布于众。
-
-2) 公开
-
-Linux内核安全团队的宗旨就是和漏洞提交者一起处理漏洞的解决方案直
-到公开。我们喜欢尽快地完全公开漏洞。当一个漏洞或者修复还没有被完全地理
-解，解决方案没有通过测试或者供应商协调，可以合理地延迟公开。然而，我们
-期望这些延迟尽可能的短些，是可数的几天，而不是几个星期或者几个月。公开
-日期是通过安全团队和漏洞提供者以及供应商洽谈后的结果。公开时间表是从很
-短（特殊的，它已经被公众所知道）到几个星期。作为一个基本的默认政策，我
-们所期望通知公众的日期是7天的安排。
-
-3) 保密协议
-
-Linux内核安全团队不是一个正式的团体，因此不能加入任何的保密协议。
Each function provides its specific set of attributes, with either read-only
or read-write access. Where applicable they need to be written to as
appropriate.
-Please refer to Documentation/ABI/*/configfs-usb-gadget* for more information.
+Please refer to Documentation/ABI/testing/configfs-usb-gadget for more information.
4. Associating the functions with their configurations
------------------------------------------------------
-================
+================
mtouchusb driver
================
-==========
+==========
USB serial
==========
seccomp notification fd to receive a ``struct seccomp_notif``, which contains
five members: the input length of the structure, a unique-per-filter ``id``,
the ``pid`` of the task which triggered this request (which may be 0 if the
-task is in a pid ns not visible from the listener's pid namespace), a ``flags``
-member which for now only has ``SECCOMP_NOTIF_FLAG_SIGNALED``, representing
-whether or not the notification is a result of a non-fatal signal, and the
-``data`` passed to seccomp. Userspace can then make a decision based on this
-information about what to do, and ``ioctl(SECCOMP_IOCTL_NOTIF_SEND)`` a
-response, indicating what should be returned to userspace. The ``id`` member of
-``struct seccomp_notif_resp`` should be the same ``id`` as in ``struct
-seccomp_notif``.
+task is in a pid ns not visible from the listener's pid namespace). The
+notification also contains the ``data`` passed to seccomp, and a filters flag.
+The structure should be zeroed out prior to calling the ioctl.
+
+Userspace can then make a decision based on this information about what to do,
+and ``ioctl(SECCOMP_IOCTL_NOTIF_SEND)`` a response, indicating what should be
+returned to userspace. The ``id`` member of ``struct seccomp_notif_resp`` should
+be the same ``id`` as in ``struct seccomp_notif``.
It is worth noting that ``struct seccomp_data`` contains the values of register
arguments to the syscall, but does not contain pointers to memory. The task's
[ecx]:
Bits[31:0] Number of encrypted guests supported simultaneously
-If support for SEV is present, MSR 0xc001_0010 (MSR_K8_SYSCFG) and MSR 0xc001_0015
+If support for SEV is present, MSR 0xc001_0010 (MSR_AMD64_SYSCFG) and MSR 0xc001_0015
(MSR_K7_HWCR) can be used to determine if it can be enabled::
0xc001_0010:
4.126 KVM_X86_SET_MSR_FILTER
----------------------------
-:Capability: KVM_X86_SET_MSR_FILTER
+:Capability: KVM_CAP_X86_MSR_FILTER
:Architectures: x86
:Type: vm ioctl
:Parameters: struct kvm_msr_filter
instead get bounced to user space through the KVM_EXIT_X86_RDMSR and
KVM_EXIT_X86_WRMSR exit notifications.
-8.27 KVM_X86_SET_MSR_FILTER
+8.27 KVM_CAP_X86_MSR_FILTER
---------------------------
:Architectures: x86
shadow pages) so role.quadrant takes values in the range 0..3. Each
quadrant maps 1GB virtual address space.
role.access:
- Inherited guest access permissions in the form uwx. Note execute
- permission is positive, not negative.
+ Inherited guest access permissions from the parent ptes in the form uwx.
+ Note execute permission is positive, not negative.
role.invalid:
The page is invalid and should not be used. It is a root page that is
currently pinned (by a cpu hardware register pointing to it); once it is
necessary to inform each VCPU to completely refresh the tables. This
request is used for that.
-KVM_REQ_PENDING_TIMER
+KVM_REQ_UNBLOCK
- This request may be made from a timer handler run on the host on behalf
- of a VCPU. It informs the VCPU thread to inject a timer interrupt.
+ This request informs the vCPU to exit kvm_vcpu_block. It is used for
+ example from timer handlers that run on the host on behalf of a vCPU,
+ or in order to update the interrupt routing and ensure that assigned
+ devices will wake up the vCPU.
KVM_REQ_UNHALT
Here is a sample of slub debug output::
====================================================================
- BUG kmalloc-8: Redzone overwritten
+ BUG kmalloc-8: Right Redzone overwritten
--------------------------------------------------------------------
INFO: 0xc90f6d28-0xc90f6d2b. First byte 0x00 instead of 0xcc
INFO: Object 0xc90f6d20 @offset=3360 fp=0xc90f6d58
INFO: Allocated in get_modalias+0x61/0xf5 age=53 cpu=1 pid=554
- Bytes b4 0xc90f6d10: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
- Object 0xc90f6d20: 31 30 31 39 2e 30 30 35 1019.005
- Redzone 0xc90f6d28: 00 cc cc cc .
- Padding 0xc90f6d50: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
+ Bytes b4 (0xc90f6d10): 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
+ Object (0xc90f6d20): 31 30 31 39 2e 30 30 35 1019.005
+ Redzone (0xc90f6d28): 00 cc cc cc .
+ Padding (0xc90f6d50): 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
[<c010523d>] dump_trace+0x63/0x1eb
[<c01053df>] show_trace_log_lvl+0x1a/0x2f
system physical addresses, not guest physical
addresses)
-If support for SME is present, MSR 0xc00100010 (MSR_K8_SYSCFG) can be used to
+If support for SME is present, MSR 0xc00100010 (MSR_AMD64_SYSCFG) can be used to
determine if SME is enabled and/or to enable memory encryption::
0xc0010010:
The CPU supports SME (determined through CPUID instruction).
- Enabled:
- Supported and bit 23 of MSR_K8_SYSCFG is set.
+ Supported and bit 23 of MSR_AMD64_SYSCFG is set.
- Active:
Supported, Enabled and the Linux kernel is actively applying
SME can also be enabled and activated in the BIOS. If SME is enabled and
activated in the BIOS, then all memory accesses will be encrypted and it will
not be necessary to activate the Linux memory encryption support. If the BIOS
-merely enables SME (sets bit 23 of the MSR_K8_SYSCFG), then Linux can activate
+merely enables SME (sets bit 23 of the MSR_AMD64_SYSCFG), then Linux can activate
memory encryption by default (CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT=y) or
by supplying mem_encrypt=on on the kernel command line. However, if BIOS does
not enable SME, then Linux will not be able to activate memory encryption, even
ARM/Allwinner sunXi SoC support
M: Maxime Ripard <mripard@kernel.org>
M: Chen-Yu Tsai <wens@csie.org>
-R: Jernej Skrabec <jernej.skrabec@siol.net>
+R: Jernej Skrabec <jernej.skrabec@gmail.com>
L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
S: Maintained
T: git git://git.kernel.org/pub/scm/linux/kernel/git/sunxi/linux.git
F: sound/soc/meson/
ARM/Amlogic Meson SoC support
+M: Neil Armstrong <narmstrong@baylibre.com>
M: Kevin Hilman <khilman@baylibre.com>
-R: Neil Armstrong <narmstrong@baylibre.com>
R: Jerome Brunet <jbrunet@baylibre.com>
R: Martin Blumenstingl <martin.blumenstingl@googlemail.com>
L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
S: Maintained
W: http://btrfs.wiki.kernel.org/
Q: http://patchwork.kernel.org/project/linux-btrfs/list/
+C: irc://irc.libera.chat/btrfs
T: git git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git
F: Documentation/filesystems/btrfs.rst
F: fs/btrfs/
F: Documentation/devicetree/bindings/arm/cavium-thunder2.txt
F: arch/arm64/boot/dts/cavium/thunder2-99xx*
+CBS/ETF/TAPRIO QDISCS
+M: Vinicius Costa Gomes <vinicius.gomes@intel.com>
+S: Maintained
+L: netdev@vger.kernel.org
+F: net/sched/sch_cbs.c
+F: net/sched/sch_etf.c
+F: net/sched/sch_taprio.c
+
CC2520 IEEE-802.15.4 RADIO DRIVER
M: Varka Bhadram <varkabhadram@gmail.com>
L: linux-wpan@vger.kernel.org
F: drivers/net/fddi/defza.*
DEINTERLACE DRIVERS FOR ALLWINNER H3
-M: Jernej Skrabec <jernej.skrabec@siol.net>
+M: Jernej Skrabec <jernej.skrabec@gmail.com>
L: linux-media@vger.kernel.org
S: Maintained
T: git git://linuxtv.org/media_tree.git
M: Dan Williams <dan.j.williams@intel.com>
M: Vishal Verma <vishal.l.verma@intel.com>
M: Dave Jiang <dave.jiang@intel.com>
-L: linux-nvdimm@lists.01.org
+L: nvdimm@lists.linux.dev
S: Supported
F: drivers/dax/
DPAA2 ETHERNET DRIVER
M: Ioana Ciornei <ioana.ciornei@nxp.com>
-M: Ioana Radulescu <ruxandra.radulescu@nxp.com>
L: netdev@vger.kernel.org
S: Maintained
F: Documentation/networking/device_drivers/ethernet/freescale/dpaa2/ethernet-driver.rst
DRM DRIVER FOR ALLWINNER DE2 AND DE3 ENGINE
M: Maxime Ripard <mripard@kernel.org>
M: Chen-Yu Tsai <wens@csie.org>
-R: Jernej Skrabec <jernej.skrabec@siol.net>
+R: Jernej Skrabec <jernej.skrabec@gmail.com>
L: dri-devel@lists.freedesktop.org
S: Supported
T: git git://anongit.freedesktop.org/drm/drm-misc
F: drivers/gpu/drm/sun4i/sun8i*
DRM DRIVER FOR ARM PL111 CLCD
-M: Eric Anholt <eric@anholt.net>
+M: Emma Anholt <emma@anholt.net>
S: Supported
T: git git://anongit.freedesktop.org/drm/drm-misc
F: drivers/gpu/drm/pl111/
F: drivers/gpu/drm/tiny/gm12u320.c
DRM DRIVER FOR HX8357D PANELS
-M: Eric Anholt <eric@anholt.net>
+M: Emma Anholt <emma@anholt.net>
S: Maintained
T: git git://anongit.freedesktop.org/drm/drm-misc
F: Documentation/devicetree/bindings/display/himax,hx8357d.txt
M: Robert Foss <robert.foss@linaro.org>
R: Laurent Pinchart <Laurent.pinchart@ideasonboard.com>
R: Jonas Karlman <jonas@kwiboo.se>
-R: Jernej Skrabec <jernej.skrabec@siol.net>
+R: Jernej Skrabec <jernej.skrabec@gmail.com>
S: Maintained
T: git git://anongit.freedesktop.org/drm/drm-misc
F: drivers/gpu/drm/bridge/
F: drivers/gpu/drm/omapdrm/
DRM DRIVERS FOR V3D
-M: Eric Anholt <eric@anholt.net>
+M: Emma Anholt <emma@anholt.net>
S: Supported
T: git git://anongit.freedesktop.org/drm/drm-misc
F: Documentation/devicetree/bindings/gpu/brcm,bcm-v3d.yaml
F: include/uapi/drm/v3d_drm.h
DRM DRIVERS FOR VC4
-M: Eric Anholt <eric@anholt.net>
+M: Emma Anholt <emma@anholt.net>
M: Maxime Ripard <mripard@kernel.org>
S: Supported
T: git git://github.com/anholt/linux
F: Documentation/devicetree/bindings/net/qca,ar803x.yaml
F: Documentation/networking/phy.rst
F: drivers/net/mdio/
+F: drivers/net/mdio/acpi_mdio.c
+F: drivers/net/mdio/fwnode_mdio.c
F: drivers/net/mdio/of_mdio.c
F: drivers/net/pcs/
F: drivers/net/phy/
FANOTIFY
M: Jan Kara <jack@suse.cz>
R: Amir Goldstein <amir73il@gmail.com>
+R: Matthew Bobrowski <repnop@google.com>
L: linux-fsdevel@vger.kernel.org
S: Maintained
F: fs/notify/fanotify/
R: Matthew Wilcox <willy@infradead.org>
R: Jan Kara <jack@suse.cz>
L: linux-fsdevel@vger.kernel.org
-L: linux-nvdimm@lists.01.org
+L: nvdimm@lists.linux.dev
S: Supported
F: fs/dax.c
F: include/linux/dax.h
F: drivers/net/ethernet/intel/
F: drivers/net/ethernet/intel/*/
F: include/linux/avf/virtchnl.h
+F: include/linux/net/intel/iidc.h
INTEL FRAMEBUFFER DRIVER (excluding 810 and 815)
M: Maik Broemme <mbroemme@libmpq.org>
S: Maintained
F: drivers/platform/x86/intel-wmi-thunderbolt.c
+INTEL WWAN IOSM DRIVER
+M: M Chetan Kumar <m.chetan.kumar@intel.com>
+M: Intel Corporation <linuxwwan@intel.com>
+L: netdev@vger.kernel.org
+S: Maintained
+F: drivers/net/wwan/iosm/
+
INTEL(R) TRACE HUB
M: Alexander Shishkin <alexander.shishkin@linux.intel.com>
S: Supported
M: Dan Williams <dan.j.williams@intel.com>
M: Vishal Verma <vishal.l.verma@intel.com>
M: Dave Jiang <dave.jiang@intel.com>
-L: linux-nvdimm@lists.01.org
+L: nvdimm@lists.linux.dev
S: Supported
Q: https://patchwork.kernel.org/project/linux-nvdimm/list/
P: Documentation/nvdimm/maintainer-entry-profile.rst
M: Vishal Verma <vishal.l.verma@intel.com>
M: Dan Williams <dan.j.williams@intel.com>
M: Dave Jiang <dave.jiang@intel.com>
-L: linux-nvdimm@lists.01.org
+L: nvdimm@lists.linux.dev
S: Supported
Q: https://patchwork.kernel.org/project/linux-nvdimm/list/
P: Documentation/nvdimm/maintainer-entry-profile.rst
M: Dan Williams <dan.j.williams@intel.com>
M: Vishal Verma <vishal.l.verma@intel.com>
M: Dave Jiang <dave.jiang@intel.com>
-L: linux-nvdimm@lists.01.org
+L: nvdimm@lists.linux.dev
S: Supported
Q: https://patchwork.kernel.org/project/linux-nvdimm/list/
P: Documentation/nvdimm/maintainer-entry-profile.rst
LIBNVDIMM: DEVICETREE BINDINGS
M: Oliver O'Halloran <oohall@gmail.com>
-L: linux-nvdimm@lists.01.org
+L: nvdimm@lists.linux.dev
S: Supported
Q: https://patchwork.kernel.org/project/linux-nvdimm/list/
F: Documentation/devicetree/bindings/pmem/pmem-region.txt
M: Vishal Verma <vishal.l.verma@intel.com>
M: Dave Jiang <dave.jiang@intel.com>
M: Ira Weiny <ira.weiny@intel.com>
-L: linux-nvdimm@lists.01.org
+L: nvdimm@lists.linux.dev
S: Supported
Q: https://patchwork.kernel.org/project/linux-nvdimm/list/
P: Documentation/nvdimm/maintainer-entry-profile.rst
MICROSOFT SURFACE SYSTEM AGGREGATOR SUBSYSTEM
M: Maximilian Luz <luzmaximilian@gmail.com>
+L: platform-driver-x86@vger.kernel.org
S: Maintained
W: https://github.com/linux-surface/surface-aggregator-module
C: irc://chat.freenode.net/##linux-surface
F: drivers/media/pci/meye/
F: include/uapi/linux/meye.h
+MOTORCOMM PHY DRIVER
+M: Peter Geis <pgwipeout@gmail.com>
+L: netdev@vger.kernel.org
+S: Maintained
+F: drivers/net/phy/motorcomm.c
+
MOXA SMARTIO/INDUSTIO/INTELLIO SERIAL CARD
S: Orphan
F: Documentation/driver-api/serial/moxa-smartio.rst
F: include/linux/mfd/ntxec.h
NETRONOME ETHERNET DRIVERS
-M: Simon Horman <simon.horman@netronome.com>
+M: Simon Horman <simon.horman@corigine.com>
R: Jakub Kicinski <kuba@kernel.org>
-L: oss-drivers@netronome.com
+L: oss-drivers@corigine.com
S: Maintained
F: drivers/net/ethernet/netronome/
M: Jakub Kicinski <kuba@kernel.org>
L: netdev@vger.kernel.org
S: Maintained
-W: http://www.linuxfoundation.org/en/Net
Q: https://patchwork.kernel.org/project/netdevbpf/list/
T: git git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git
T: git git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git
M: Jakub Kicinski <kuba@kernel.org>
L: netdev@vger.kernel.org
S: Maintained
-W: http://www.linuxfoundation.org/en/Net
Q: https://patchwork.kernel.org/project/netdevbpf/list/
B: mailto:netdev@vger.kernel.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git
F: net/ipv4/nexthop.c
NFC SUBSYSTEM
+M: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
+L: linux-nfc@lists.01.org (subscribers-only)
L: netdev@vger.kernel.org
-S: Orphan
+S: Maintained
F: Documentation/devicetree/bindings/net/nfc/
F: drivers/nfc/
F: include/linux/platform_data/nfcmrvl.h
NFC VIRTUAL NCI DEVICE DRIVER
M: Bongsu Jeon <bongsu.jeon@samsung.com>
L: netdev@vger.kernel.org
-L: linux-nfc@lists.01.org (moderated for non-subscribers)
+L: linux-nfc@lists.01.org (subscribers-only)
S: Supported
F: drivers/nfc/virtual_ncidev.c
F: tools/testing/selftests/nci/
L: linux-kernel@vger.kernel.org
S: Maintained
F: drivers/net/dsa/sja1105
+F: drivers/net/pcs/pcs-xpcs-nxp.c
NXP TDA998X DRM DRIVER
M: Russell King <linux@armlinux.org.uk>
F: sound/soc/codecs/tfa9879*
NXP-NCI NFC DRIVER
-M: Clément Perrochaud <clement.perrochaud@effinnov.com>
R: Charles Gorand <charles.gorand@effinnov.com>
-L: linux-nfc@lists.01.org (moderated for non-subscribers)
+L: linux-nfc@lists.01.org (subscribers-only)
S: Supported
F: drivers/nfc/nxp-nci
PCI ENDPOINT SUBSYSTEM
M: Kishon Vijay Abraham I <kishon@ti.com>
M: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
+R: Krzysztof Wilczyński <kw@linux.com>
L: linux-pci@vger.kernel.org
S: Supported
F: Documentation/PCI/endpoint/*
PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS
M: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
R: Rob Herring <robh@kernel.org>
+R: Krzysztof Wilczyński <kw@linux.com>
L: linux-pci@vger.kernel.org
S: Supported
Q: http://patchwork.ozlabs.org/project/linux-pci/list/
M: Dennis Zhou <dennis@kernel.org>
M: Tejun Heo <tj@kernel.org>
M: Christoph Lameter <cl@linux.com>
+L: linux-mm@kvack.org
S: Maintained
T: git git://git.kernel.org/pub/scm/linux/kernel/git/dennis/percpu.git
F: arch/*/include/asm/percpu.h
F: include/linux/percpu*.h
+F: lib/percpu*.c
F: mm/percpu*.c
PER-TASK DELAY ACCOUNTING
F: drivers/net/wireless/intersil/prism54/
PROC FILESYSTEM
-R: Alexey Dobriyan <adobriyan@gmail.com>
L: linux-kernel@vger.kernel.org
L: linux-fsdevel@vger.kernel.org
S: Maintained
F: include/uapi/linux/rpmsg.h
F: samples/rpmsg/
+REMOTE PROCESSOR MESSAGING (RPMSG) WWAN CONTROL DRIVER
+M: Stephan Gerhold <stephan@gerhold.net>
+L: netdev@vger.kernel.org
+L: linux-remoteproc@vger.kernel.org
+S: Maintained
+F: drivers/net/wwan/rpmsg_wwan_ctrl.c
+
RENESAS CLOCK DRIVERS
M: Geert Uytterhoeven <geert+renesas@glider.be>
L: linux-renesas-soc@vger.kernel.org
F: net/rose/
ROTATION DRIVER FOR ALLWINNER A83T
-M: Jernej Skrabec <jernej.skrabec@siol.net>
+M: Jernej Skrabec <jernej.skrabec@gmail.com>
L: linux-media@vger.kernel.org
S: Maintained
T: git git://linuxtv.org/media_tree.git
M: Julian Wiedmann <jwi@linux.ibm.com>
M: Karsten Graul <kgraul@linux.ibm.com>
L: linux-s390@vger.kernel.org
+L: netdev@vger.kernel.org
S: Supported
W: http://www.ibm.com/developerworks/linux/linux390/
F: drivers/s390/net/*iucv*
M: Julian Wiedmann <jwi@linux.ibm.com>
M: Karsten Graul <kgraul@linux.ibm.com>
L: linux-s390@vger.kernel.org
+L: netdev@vger.kernel.org
S: Supported
W: http://www.ibm.com/developerworks/linux/linux390/
F: drivers/s390/net/
SAMSUNG S3FWRN5 NFC DRIVER
M: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
M: Krzysztof Opasiak <k.opasiak@samsung.com>
-L: linux-nfc@lists.01.org (moderated for non-subscribers)
+L: linux-nfc@lists.01.org (subscribers-only)
S: Maintained
F: Documentation/devicetree/bindings/net/nfc/samsung,s3fwrn5.yaml
F: drivers/nfc/s3fwrn5
SHARED MEMORY COMMUNICATIONS (SMC) SOCKETS
M: Karsten Graul <kgraul@linux.ibm.com>
+M: Guvenc Gulce <guvenc@linux.ibm.com>
L: linux-s390@vger.kernel.org
S: Supported
W: http://www.ibm.com/developerworks/linux/linux390/
S: Maintained
F: drivers/i2c/busses/i2c-stm32*
+ST STM32 SPI DRIVER
+M: Alain Volmat <alain.volmat@foss.st.com>
+L: linux-spi@vger.kernel.org
+S: Maintained
+F: drivers/spi/spi-stm32.c
+
ST STPDDC60 DRIVER
M: Daniel Nilsson <daniel.nilsson@flex.com>
L: linux-hwmon@vger.kernel.org
L: netdev@vger.kernel.org
S: Supported
F: drivers/net/pcs/pcs-xpcs.c
+F: drivers/net/pcs/pcs-xpcs.h
F: include/linux/pcs/pcs-xpcs.h
SYNOPSYS DESIGNWARE I2C DRIVER
L: linux-i2c@vger.kernel.org
S: Maintained
F: drivers/i2c/busses/i2c-designware-*
-F: include/linux/platform_data/i2c-designware.h
SYNOPSYS DESIGNWARE MMC/SD/SDIO DRIVER
M: Jaehoon Chung <jh80.chung@samsung.com>
TI TRF7970A NFC DRIVER
M: Mark Greer <mgreer@animalcreek.com>
L: linux-wireless@vger.kernel.org
-L: linux-nfc@lists.01.org (moderated for non-subscribers)
+L: linux-nfc@lists.01.org (subscribers-only)
S: Supported
F: Documentation/devicetree/bindings/net/nfc/trf7970a.txt
F: drivers/nfc/trf7970a.c
F: drivers/usb/host/isp116x*
F: include/linux/usb/isp116x.h
+USB ISP1760 DRIVER
+M: Rui Miguel Silva <rui.silva@linaro.org>
+L: linux-usb@vger.kernel.org
+S: Maintained
+F: drivers/usb/isp1760/*
+F: Documentation/devicetree/bindings/usb/nxp,isp1760.yaml
+
USB LAN78XX ETHERNET DRIVER
M: Woojung Huh <woojung.huh@microchip.com>
M: UNGLinuxDriver@microchip.com
F: include/linux/workqueue.h
F: kernel/workqueue.c
+WWAN DRIVERS
+M: Loic Poulain <loic.poulain@linaro.org>
+M: Sergey Ryazanov <ryazanov.s.a@gmail.com>
+R: Johannes Berg <johannes@sipsolutions.net>
+L: netdev@vger.kernel.org
+S: Maintained
+F: drivers/net/wwan/
+F: include/linux/wwan.h
+F: include/uapi/linux/wwan.h
+
X-POWERS AXP288 PMIC DRIVERS
M: Hans de Goede <hdegoede@redhat.com>
S: Maintained
F: drivers/xen/*swiotlb*
XFS FILESYSTEM
+C: irc://irc.oftc.net/xfs
M: Darrick J. Wong <djwong@kernel.org>
M: linux-xfs@vger.kernel.org
L: linux-xfs@vger.kernel.org
# SPDX-License-Identifier: GPL-2.0
VERSION = 5
-PATCHLEVEL = 12
+PATCHLEVEL = 13
SUBLEVEL = 0
-EXTRAVERSION =
+EXTRAVERSION = -rc6
NAME = Frozen Wasteland
# *DOCUMENTATION*
# Limit inlining across translation units to reduce binary size
KBUILD_LDFLAGS += -mllvm -import-instr-limit=5
+
+# Check for frame size exceeding threshold during prolog/epilog insertion
+# when using lld < 13.0.0.
+ifneq ($(CONFIG_FRAME_WARN),0)
+ifeq ($(shell test $(CONFIG_LLD_VERSION) -lt 130000; echo $$?),0)
+KBUILD_LDFLAGS += -plugin-opt=-warn-stack-size=$(CONFIG_FRAME_WARN)
+endif
+endif
endif
ifdef CONFIG_LTO
#define SO_PREFER_BUSY_POLL 69
#define SO_BUSY_POLL_BUDGET 70
+#define SO_NETNS_COOKIE 71
+
#if !defined(__KERNEL__)
#if __BITS_PER_LONG == 64
550 common process_madvise sys_process_madvise
551 common epoll_pwait2 sys_epoll_pwait2
552 common mount_setattr sys_mount_setattr
-553 common quotactl_path sys_quotactl_path
+# 553 reserved for quotactl_path
554 common landlock_create_ruleset sys_landlock_create_ruleset
555 common landlock_add_rule sys_landlock_add_rule
556 common landlock_restrict_self sys_landlock_restrict_self
ifdef CONFIG_ARC_CURR_IN_REG
-# For a global register defintion, make sure it gets passed to every file
+# For a global register definition, make sure it gets passed to every file
# We had a customer reported bug where some code built in kernel was NOT using
# any kernel headers, and missing the r25 global register
# Can't do unconditionally because of recursive include issues
*
* Technically the lock is also needed for UP (boils down to irq save/restore)
* but we can cheat a bit since cmpxchg() atomic_ops_lock() would cause irqs to
- * be disabled thus can't possibly be interrpted/preempted/clobbered by xchg()
+ * be disabled thus can't possibly be interrupted/preempted/clobbered by xchg()
* Other way around, xchg is one instruction anyways, so can't be interrupted
* as such
*/
/*
* "atomic" variant of xchg()
* REQ: It needs to follow the same serialization rules as other atomic_xxx()
- * Since xchg() doesn't always do that, it would seem that following defintion
+ * Since xchg() doesn't always do that, it would seem that following definition
* is incorrect. But here's the rationale:
* SMP : Even xchg() takes the atomic_ops_lock, so OK.
* LLSC: atomic_ops_lock are not relevant at all (even if SMP, since LLSC
#include <uapi/asm/page.h>
+#ifdef CONFIG_ARC_HAS_PAE40
+
+#define MAX_POSSIBLE_PHYSMEM_BITS 40
+#define PAGE_MASK_PHYS (0xff00000000ull | PAGE_MASK)
+
+#else /* CONFIG_ARC_HAS_PAE40 */
+
+#define MAX_POSSIBLE_PHYSMEM_BITS 32
+#define PAGE_MASK_PHYS PAGE_MASK
+
+#endif /* CONFIG_ARC_HAS_PAE40 */
+
#ifndef __ASSEMBLY__
#define clear_page(paddr) memset((paddr), 0, PAGE_SIZE)
#define ___DEF (_PAGE_PRESENT | _PAGE_CACHEABLE)
/* Set of bits not changed in pte_modify */
-#define _PAGE_CHG_MASK (PAGE_MASK | _PAGE_ACCESSED | _PAGE_DIRTY | _PAGE_SPECIAL)
-
+#define _PAGE_CHG_MASK (PAGE_MASK_PHYS | _PAGE_ACCESSED | _PAGE_DIRTY | \
+ _PAGE_SPECIAL)
/* More Abbrevaited helpers */
#define PAGE_U_NONE __pgprot(___DEF)
#define PAGE_U_R __pgprot(___DEF | _PAGE_READ)
#define PTE_BITS_IN_PD0 (_PAGE_GLOBAL | _PAGE_PRESENT | _PAGE_HW_SZ)
#define PTE_BITS_RWX (_PAGE_EXECUTE | _PAGE_WRITE | _PAGE_READ)
-#ifdef CONFIG_ARC_HAS_PAE40
-#define PTE_BITS_NON_RWX_IN_PD1 (0xff00000000 | PAGE_MASK | _PAGE_CACHEABLE)
-#define MAX_POSSIBLE_PHYSMEM_BITS 40
-#else
-#define PTE_BITS_NON_RWX_IN_PD1 (PAGE_MASK | _PAGE_CACHEABLE)
-#define MAX_POSSIBLE_PHYSMEM_BITS 32
-#endif
+#define PTE_BITS_NON_RWX_IN_PD1 (PAGE_MASK_PHYS | _PAGE_CACHEABLE)
/**************************************************************************
* Mapping of vm_flags (Generic VM) to PTE flags (arch specific)
#define PAGE_MASK (~(PAGE_SIZE-1))
-
#endif /* _UAPI__ASM_ARC_PAGE_H */
*/
struct sigcontext {
struct user_regs_struct regs;
+ struct user_regs_arcv2 v2abi;
};
#endif /* _ASM_ARC_SIGCONTEXT_H */
; Do the Sys Call as we normally would.
; Validate the Sys Call number
- cmp r8, NR_syscalls
+ cmp r8, NR_syscalls - 1
mov.hi r0, -ENOSYS
bhi tracesys_exit
;============ Normal syscall case
; syscall num shd not exceed the total system calls avail
- cmp r8, NR_syscalls
+ cmp r8, NR_syscalls - 1
mov.hi r0, -ENOSYS
bhi .Lret_from_system_call
ptr = &remcomInBuffer[1];
if (kgdb_hex2long(&ptr, &addr))
regs->ret = addr;
+ fallthrough;
case 'D':
case 'k':
int ret;
/*
- * This is only for old cores lacking LLOCK/SCOND, which by defintion
+ * This is only for old cores lacking LLOCK/SCOND, which by definition
* can't possibly be SMP. Thus doesn't need to be SMP safe.
* And this also helps reduce the overhead for serializing in
* the UP case
*/
WARN_ON_ONCE(IS_ENABLED(CONFIG_SMP));
- /* Z indicates to userspace if operation succeded */
+ /* Z indicates to userspace if operation succeeded */
regs->status32 &= ~STATUS_Z_MASK;
ret = access_ok(uaddr, sizeof(*uaddr));
void arch_cpu_idle(void)
{
- /* Re-enable interrupts <= default irq priority before commiting SLEEP */
+ /* Re-enable interrupts <= default irq priority before committing SLEEP */
const unsigned int arg = 0x10 | ARCV2_IRQ_DEF_PRIO;
__asm__ __volatile__(
void arch_cpu_idle(void)
{
- /* sleep, but enable both set E1/E2 (levels of interrutps) before committing */
+ /* sleep, but enable both set E1/E2 (levels of interrupts) before committing */
__asm__ __volatile__("sleep 0x3 \n");
}
unsigned int sigret_magic;
};
+static int save_arcv2_regs(struct sigcontext *mctx, struct pt_regs *regs)
+{
+ int err = 0;
+#ifndef CONFIG_ISA_ARCOMPACT
+ struct user_regs_arcv2 v2abi;
+
+ v2abi.r30 = regs->r30;
+#ifdef CONFIG_ARC_HAS_ACCL_REGS
+ v2abi.r58 = regs->r58;
+ v2abi.r59 = regs->r59;
+#else
+ v2abi.r58 = v2abi.r59 = 0;
+#endif
+ err = __copy_to_user(&mctx->v2abi, &v2abi, sizeof(v2abi));
+#endif
+ return err;
+}
+
+static int restore_arcv2_regs(struct sigcontext *mctx, struct pt_regs *regs)
+{
+ int err = 0;
+#ifndef CONFIG_ISA_ARCOMPACT
+ struct user_regs_arcv2 v2abi;
+
+ err = __copy_from_user(&v2abi, &mctx->v2abi, sizeof(v2abi));
+
+ regs->r30 = v2abi.r30;
+#ifdef CONFIG_ARC_HAS_ACCL_REGS
+ regs->r58 = v2abi.r58;
+ regs->r59 = v2abi.r59;
+#endif
+#endif
+ return err;
+}
+
static int
stash_usr_regs(struct rt_sigframe __user *sf, struct pt_regs *regs,
sigset_t *set)
err = __copy_to_user(&(sf->uc.uc_mcontext.regs.scratch), &uregs.scratch,
sizeof(sf->uc.uc_mcontext.regs.scratch));
+
+ if (is_isa_arcv2())
+ err |= save_arcv2_regs(&(sf->uc.uc_mcontext), regs);
+
err |= __copy_to_user(&sf->uc.uc_sigmask, set, sizeof(sigset_t));
return err ? -EFAULT : 0;
err |= __copy_from_user(&uregs.scratch,
&(sf->uc.uc_mcontext.regs.scratch),
sizeof(sf->uc.uc_mcontext.regs.scratch));
+
+ if (is_isa_arcv2())
+ err |= restore_arcv2_regs(&(sf->uc.uc_mcontext), regs);
+
if (err)
return -EFAULT;
regs->r2 = (unsigned long)&sf->uc;
/*
- * small optim to avoid unconditonally calling do_sigaltstack
+ * small optim to avoid unconditionally calling do_sigaltstack
* in sigreturn path, now that we only have rt_sigreturn
*/
magic = MAGIC_SIGALTSTK;
void do_notify_resume(struct pt_regs *regs)
{
/*
- * ASM glue gaurantees that this is only called when returning to
+ * ASM glue guarantees that this is only called when returning to
* user mode
*/
if (test_thread_flag(TIF_NOTIFY_RESUME))
.init.ramfs : { INIT_RAM_FS }
. = ALIGN(PAGE_SIZE);
- _stext = .;
HEAD_TEXT_SECTION
INIT_TEXT_SECTION(L1_CACHE_BYTES)
.text : {
_text = .;
+ _stext = .;
TEXT_TEXT
SCHED_TEXT
CPUIDLE_TEXT
min_high_pfn = PFN_DOWN(high_mem_start);
max_high_pfn = PFN_DOWN(high_mem_start + high_mem_sz);
- max_zone_pfn[ZONE_HIGHMEM] = min_low_pfn;
+ /*
+ * max_high_pfn should be ok here for both HIGHMEM and HIGHMEM+PAE.
+ * For HIGHMEM without PAE max_high_pfn should be less than
+ * min_low_pfn to guarantee that these two regions don't overlap.
+ * For PAE case highmem is greater than lowmem, so it is natural
+ * to use max_high_pfn.
+ *
+ * In both cases, holes should be handled by pfn_valid().
+ */
+ max_zone_pfn[ZONE_HIGHMEM] = max_high_pfn;
high_memory = (void *)(min_high_pfn << PAGE_SHIFT);
void __iomem *ioremap_prot(phys_addr_t paddr, unsigned long size,
unsigned long flags)
{
+ unsigned int off;
unsigned long vaddr;
struct vm_struct *area;
- phys_addr_t off, end;
+ phys_addr_t end;
pgprot_t prot = __pgprot(flags);
/* Don't allow wraparound, zero size */
/* Mappings have to be page-aligned */
off = paddr & ~PAGE_MASK;
- paddr &= PAGE_MASK;
+ paddr &= PAGE_MASK_PHYS;
size = PAGE_ALIGN(end + 1) - paddr;
/*
pte_t *ptep)
{
unsigned long vaddr = vaddr_unaligned & PAGE_MASK;
- phys_addr_t paddr = pte_val(*ptep) & PAGE_MASK;
+ phys_addr_t paddr = pte_val(*ptep) & PAGE_MASK_PHYS;
struct page *page = pfn_to_page(pte_pfn(*ptep));
create_tlb(vma, vaddr, ptep);
phy-reset-gpios = <&gpio1 25 GPIO_ACTIVE_LOW>;
phy-reset-duration = <20>;
phy-supply = <&sw2_reg>;
- phy-handle = <ðphy0>;
status = "okay";
+ fixed-link {
+ speed = <1000>;
+ full-duplex;
+ };
+
mdio {
#address-cells = <1>;
#size-cells = <0>;
vin-supply = <&sw1_reg>;
};
+®_pu {
+ vin-supply = <&sw1_reg>;
+};
+
+®_vdd1p1 {
+ vin-supply = <&sw2_reg>;
+};
+
+®_vdd2p5 {
+ vin-supply = <&sw2_reg>;
+};
+
&uart1 {
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_uart1>;
compatible = "nxp,pca8574";
reg = <0x3a>;
gpio-controller;
- #gpio-cells = <1>;
+ #gpio-cells = <2>;
};
};
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_usdhc1>;
keep-power-in-suspend;
- tuning-step = <2>;
+ fsl,tuning-step = <2>;
vmmc-supply = <®_3p3v>;
no-1-8-v;
broken-cd;
pinctrl-2 = <&pinctrl_usdhc1_200mhz>;
cd-gpios = <&gpio5 0 GPIO_ACTIVE_LOW>;
bus-width = <4>;
- tuning-step = <2>;
+ fsl,tuning-step = <2>;
vmmc-supply = <®_3p3v>;
wakeup-source;
no-1-8-v;
#ifdef CONFIG_CPU_IDLE
extern int arm_cpuidle_simple_enter(struct cpuidle_device *dev,
struct cpuidle_driver *drv, int index);
+#define __cpuidle_method_section __used __section("__cpuidle_method_of_table")
#else
static inline int arm_cpuidle_simple_enter(struct cpuidle_device *dev,
struct cpuidle_driver *drv, int index) { return -ENODEV; }
+#define __cpuidle_method_section __maybe_unused /* drop silently */
#endif
/* Common ARM WFI state */
#define CPUIDLE_METHOD_OF_DECLARE(name, _method, _ops) \
static const struct of_cpuidle_method __cpuidle_method_of_table_##name \
- __used __section("__cpuidle_method_of_table") \
- = { .method = _method, .ops = _ops }
+ __cpuidle_method_section = { .method = _method, .ops = _ops }
extern int arm_cpuidle_suspend(int index);
#include <linux/suspend.h>
#include <linux/io.h>
+#include "common.h"
#include "hardware.h"
static int mx27_suspend_enter(suspend_state_t state)
bool "Support for WPCM450 BMC (Hermon)"
depends on ARCH_MULTI_V5
select CPU_ARM926T
+ select WPCM450_AIC
select NPCM7XX_TIMER
help
General support for WPCM450 BMC (Hermon).
#ifdef CONFIG_LEDS_TRIGGERS
DEFINE_LED_TRIGGER(ams_delta_camera_led_trigger);
-
-static int ams_delta_camera_power(struct device *dev, int power)
-{
- /*
- * turn on camera LED
- */
- if (power)
- led_trigger_event(ams_delta_camera_led_trigger, LED_FULL);
- else
- led_trigger_event(ams_delta_camera_led_trigger, LED_OFF);
- return 0;
-}
-#else
-#define ams_delta_camera_power NULL
#endif
static struct platform_device ams_delta_audio_device = {
{
if (!IS_BUILTIN(CONFIG_TPS65010))
return -ENOSYS;
-
+
tps65010_config_vregs1(TPS_LDO2_ENABLE | TPS_VLDO2_3_0V |
TPS_LDO1_ENABLE | TPS_VLDO1_3_0V);
BUG_ON(gpio_request(H2_NAND_RB_GPIO_PIN, "NAND ready") < 0);
gpio_direction_input(H2_NAND_RB_GPIO_PIN);
+ gpiod_add_lookup_table(&isp1301_gpiod_table);
+
omap_cfg_reg(L3_1610_FLASH_CS2B_OE);
omap_cfg_reg(M8_1610_FLASH_CS2B_WE);
irq = INT_7XX_WAKE_UP_REQ;
else if (cpu_is_omap16xx())
irq = INT_1610_WAKE_UP_REQ;
- if (request_irq(irq, omap_wakeup_interrupt, 0, "peripheral wakeup",
- NULL))
- pr_err("Failed to request irq %d (peripheral wakeup)\n", irq);
+ else
+ irq = -1;
+
+ if (irq >= 0) {
+ if (request_irq(irq, omap_wakeup_interrupt, 0, "peripheral wakeup", NULL))
+ pr_err("Failed to request irq %d (peripheral wakeup)\n", irq);
+ }
/* Program new power ramp-up time
* (0 for most boards since we don't lower voltage when in deep sleep)
static void n8x0_mmc_callback(void *data, u8 card_mask)
{
+#ifdef CONFIG_MMC_OMAP
int bit, *openp, index;
if (board_is_n800()) {
else
*openp = 0;
-#ifdef CONFIG_MMC_OMAP
omap_mmc_notify_cover_event(mmc_device, index, *openp);
#else
pr_warn("MMC: notify cover event not available\n");
return fpga->irq;
base_irq = platform_get_irq(pdev, 1);
- if (base_irq < 0)
+ if (base_irq < 0) {
base_irq = 0;
+ } else {
+ ret = devm_irq_alloc_descs(&pdev->dev, base_irq, base_irq, CPLDS_NB_IRQ, 0);
+ if (ret < 0)
+ return ret;
+ }
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
fpga->base = devm_ioremap_resource(&pdev->dev, res);
440 common process_madvise sys_process_madvise
441 common epoll_pwait2 sys_epoll_pwait2
442 common mount_setattr sys_mount_setattr
-443 common quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 common landlock_create_ruleset sys_landlock_create_ruleset
445 common landlock_add_rule sys_landlock_add_rule
446 common landlock_restrict_self sys_landlock_restrict_self
return;
}
-int xen_swiotlb_detect(void)
-{
- if (!xen_domain())
- return 0;
- if (xen_feature(XENFEAT_direct_mapped))
- return 1;
- /* legacy case */
- if (!xen_feature(XENFEAT_not_direct_mapped) && xen_initial_domain())
- return 1;
- return 0;
-}
-
static int __init xen_mm_init(void)
{
struct gnttab_cache_flush cflush;
+ int rc;
+
if (!xen_swiotlb_detect())
return 0;
- xen_swiotlb_init();
+
+ rc = xen_swiotlb_init();
+ /* we can work with the default swiotlb */
+ if (rc < 0 && rc != -EEXIST)
+ return rc;
cflush.op = 0;
cflush.a.dev_bus_addr = 0;
# SPDX-License-Identifier: GPL-2.0-only
-obj-y += kernel/ mm/
-obj-$(CONFIG_NET) += net/
+obj-y += kernel/ mm/ net/
obj-$(CONFIG_KVM) += kvm/
obj-$(CONFIG_XEN) += xen/
obj-$(CONFIG_CRYPTO) += crypto/
config ARCH_MESON
bool "Amlogic Platforms"
+ select COMMON_CLK
select MESON_IRQ_GPIO
help
This enables support for the arm64 based Amlogic SoCs
$(if $(CONFIG_COMPAT_VDSO), \
$(Q)$(MAKE) $(build)=arch/arm64/kernel/vdso32 $@)
+archprepare:
+ $(Q)$(MAKE) $(build)=arch/arm64/tools kapi
+
# We use MRPROPER_FILES and CLEAN_FILES now
archclean:
$(Q)$(MAKE) $(clean)=$(boot)
eee-broken-100tx;
qca,clk-out-frequency = <125000000>;
qca,clk-out-strength = <AR803X_STRENGTH_FULL>;
- vddio-supply = <&vddh>;
+ qca,keep-pll-enabled;
+ vddio-supply = <&vddio>;
vddio: vddio-regulator {
regulator-name = "VDDIO";
reg = <0x4>;
eee-broken-1000t;
eee-broken-100tx;
-
qca,clk-out-frequency = <125000000>;
qca,clk-out-strength = <AR803X_STRENGTH_FULL>;
-
- vddio-supply = <&vddh>;
+ qca,keep-pll-enabled;
+ vddio-supply = <&vddio>;
vddio: vddio-regulator {
regulator-name = "VDDIO";
ddr: memory-controller@1080000 {
compatible = "fsl,qoriq-memory-controller";
reg = <0x0 0x1080000 0x0 0x1000>;
- interrupts = <GIC_SPI 144 IRQ_TYPE_LEVEL_HIGH>;
- big-endian;
+ interrupts = <GIC_SPI 17 IRQ_TYPE_LEVEL_HIGH>;
+ little-endian;
};
dcfg: syscon@1e00000 {
pinctrl-0 = <&pinctrl_codec2>;
reg = <0x18>;
#sound-dai-cells = <0>;
- HPVDD-supply = <®_3p3v>;
- SPRVDD-supply = <®_3p3v>;
- SPLVDD-supply = <®_3p3v>;
- AVDD-supply = <®_3p3v>;
- IOVDD-supply = <®_3p3v>;
+ HPVDD-supply = <®_gen_3p3>;
+ SPRVDD-supply = <®_gen_3p3>;
+ SPLVDD-supply = <®_gen_3p3>;
+ AVDD-supply = <®_gen_3p3>;
+ IOVDD-supply = <®_gen_3p3>;
DVDD-supply = <&vgen4_reg>;
reset-gpios = <&gpio3 4 GPIO_ACTIVE_HIGH>;
};
reg_12p0_main: regulator-12p0-main {
compatible = "regulator-fixed";
regulator-name = "12V_MAIN";
- regulator-min-microvolt = <5000000>;
- regulator-max-microvolt = <5000000>;
+ regulator-min-microvolt = <12000000>;
+ regulator-max-microvolt = <12000000>;
regulator-always-on;
};
regulator-always-on;
};
- reg_3p3v: regulator-3p3v {
- compatible = "regulator-fixed";
- vin-supply = <®_3p3_main>;
- regulator-name = "GEN_3V3";
- regulator-min-microvolt = <3300000>;
- regulator-max-microvolt = <3300000>;
- regulator-always-on;
- };
-
reg_usdhc2_vmmc: regulator-vsd-3v3 {
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_reg_usdhc2>;
pinctrl-0 = <&pinctrl_codec1>;
reg = <0x18>;
#sound-dai-cells = <0>;
- HPVDD-supply = <®_3p3v>;
- SPRVDD-supply = <®_3p3v>;
- SPLVDD-supply = <®_3p3v>;
- AVDD-supply = <®_3p3v>;
- IOVDD-supply = <®_3p3v>;
+ HPVDD-supply = <®_gen_3p3>;
+ SPRVDD-supply = <®_gen_3p3>;
+ SPLVDD-supply = <®_gen_3p3>;
+ AVDD-supply = <®_gen_3p3>;
+ IOVDD-supply = <®_gen_3p3>;
DVDD-supply = <&vgen4_reg>;
reset-gpios = <&gpio3 3 GPIO_ACTIVE_LOW>;
};
};
};
- reset@611010008 {
- compatible = "microchip,sparx5-chip-reset";
+ reset: reset-controller@611010008 {
+ compatible = "microchip,sparx5-switch-reset";
reg = <0x6 0x11010008 0x4>;
+ reg-names = "gcb";
+ #reset-cells = <1>;
+ cpu-syscon = <&cpu_ctrl>;
};
uart0: serial@600100000 {
"GPIO_46", "GPIO_47";
function = "emmc";
};
+
+ miim1_pins: miim1-pins {
+ pins = "GPIO_56", "GPIO_57";
+ function = "miim";
+ };
+
+ miim2_pins: miim2-pins {
+ pins = "GPIO_58", "GPIO_59";
+ function = "miim";
+ };
+
+ miim3_pins: miim3-pins {
+ pins = "GPIO_52", "GPIO_53";
+ function = "miim";
+ };
};
sgpio0: gpio@61101036c {
clocks = <&sys_clk>;
pinctrl-0 = <&sgpio0_pins>;
pinctrl-names = "default";
+ resets = <&reset 0>;
+ reset-names = "switch";
reg = <0x6 0x1101036c 0x100>;
sgpio_in0: gpio@0 {
compatible = "microchip,sparx5-sgpio-bank";
gpio-controller;
#gpio-cells = <3>;
ngpios = <96>;
+ interrupts = <GIC_SPI 17 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-controller;
+ #interrupt-cells = <3>;
};
sgpio_out0: gpio@1 {
compatible = "microchip,sparx5-sgpio-bank";
clocks = <&sys_clk>;
pinctrl-0 = <&sgpio1_pins>;
pinctrl-names = "default";
+ resets = <&reset 0>;
+ reset-names = "switch";
reg = <0x6 0x11010484 0x100>;
sgpio_in1: gpio@0 {
compatible = "microchip,sparx5-sgpio-bank";
gpio-controller;
#gpio-cells = <3>;
ngpios = <96>;
+ interrupts = <GIC_SPI 18 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-controller;
+ #interrupt-cells = <3>;
};
sgpio_out1: gpio@1 {
compatible = "microchip,sparx5-sgpio-bank";
clocks = <&sys_clk>;
pinctrl-0 = <&sgpio2_pins>;
pinctrl-names = "default";
+ resets = <&reset 0>;
+ reset-names = "switch";
reg = <0x6 0x1101059c 0x100>;
sgpio_in2: gpio@0 {
reg = <0>;
gpio-controller;
#gpio-cells = <3>;
ngpios = <96>;
+ interrupts = <GIC_SPI 19 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-controller;
+ #interrupt-cells = <3>;
};
sgpio_out2: gpio@1 {
compatible = "microchip,sparx5-sgpio-bank";
#thermal-sensor-cells = <0>;
clocks = <&ahb_clk>;
};
+
+ mdio0: mdio@6110102b0 {
+ compatible = "mscc,ocelot-miim";
+ status = "disabled";
+ #address-cells = <1>;
+ #size-cells = <0>;
+ reg = <0x6 0x110102b0 0x24>;
+ };
+
+ mdio1: mdio@6110102d4 {
+ compatible = "mscc,ocelot-miim";
+ status = "disabled";
+ pinctrl-0 = <&miim1_pins>;
+ pinctrl-names = "default";
+ #address-cells = <1>;
+ #size-cells = <0>;
+ reg = <0x6 0x110102d4 0x24>;
+ };
+
+ mdio2: mdio@6110102f8 {
+ compatible = "mscc,ocelot-miim";
+ status = "disabled";
+ pinctrl-0 = <&miim2_pins>;
+ pinctrl-names = "default";
+ #address-cells = <1>;
+ #size-cells = <0>;
+ reg = <0x6 0x110102d4 0x24>;
+ };
+
+ mdio3: mdio@61101031c {
+ compatible = "mscc,ocelot-miim";
+ status = "disabled";
+ pinctrl-0 = <&miim3_pins>;
+ pinctrl-names = "default";
+ #address-cells = <1>;
+ #size-cells = <0>;
+ reg = <0x6 0x1101031c 0x24>;
+ };
+
+ serdes: serdes@10808000 {
+ compatible = "microchip,sparx5-serdes";
+ #phy-cells = <1>;
+ clocks = <&sys_clk>;
+ reg = <0x6 0x10808000 0x5d0000>;
+ };
+
+ switch: switch@0x600000000 {
+ compatible = "microchip,sparx5-switch";
+ reg = <0x6 0 0x401000>,
+ <0x6 0x10004000 0x7fc000>,
+ <0x6 0x11010000 0xaf0000>;
+ reg-names = "cpu", "dev", "gcb";
+ interrupt-names = "xtr";
+ interrupts = <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>;
+ resets = <&reset 0>;
+ reset-names = "switch";
+ };
};
};
#include "sparx5_pcb_common.dtsi"
/{
- aliases {
- i2c0 = &i2c0;
- i2c100 = &i2c100;
- i2c101 = &i2c101;
- i2c102 = &i2c102;
- i2c103 = &i2c103;
- i2c104 = &i2c104;
- i2c105 = &i2c105;
- i2c106 = &i2c106;
- i2c107 = &i2c107;
- i2c108 = &i2c108;
- i2c109 = &i2c109;
- i2c110 = &i2c110;
- i2c111 = &i2c111;
- i2c112 = &i2c112;
- i2c113 = &i2c113;
- i2c114 = &i2c114;
- i2c115 = &i2c115;
- i2c116 = &i2c116;
- i2c117 = &i2c117;
- i2c118 = &i2c118;
- i2c119 = &i2c119;
- };
-
gpio-restart {
compatible = "gpio-restart";
gpios = <&gpio 37 GPIO_ACTIVE_LOW>;
&spi0 {
status = "okay";
- spi@0 {
- compatible = "spi-mux";
- mux-controls = <&mux>;
- #address-cells = <1>;
- #size-cells = <0>;
- reg = <0>; /* CS0 */
- spi-flash@9 {
- compatible = "jedec,spi-nor";
- spi-max-frequency = <8000000>;
- reg = <0x9>; /* SPI */
- };
+ spi-flash@0 {
+ compatible = "jedec,spi-nor";
+ spi-max-frequency = <8000000>;
+ reg = <0>;
};
};
};
};
+&sgpio0 {
+ status = "okay";
+ microchip,sgpio-port-ranges = <8 15>;
+ gpio@0 {
+ ngpios = <64>;
+ };
+ gpio@1 {
+ ngpios = <64>;
+ };
+};
+
+&sgpio1 {
+ status = "okay";
+ microchip,sgpio-port-ranges = <24 31>;
+ gpio@0 {
+ ngpios = <64>;
+ };
+ gpio@1 {
+ ngpios = <64>;
+ };
+};
+
+&sgpio2 {
+ status = "okay";
+ microchip,sgpio-port-ranges = <0 0>, <11 31>;
+};
+
&gpio {
i2cmux_pins_i: i2cmux-pins-i {
pins = "GPIO_16", "GPIO_17", "GPIO_18", "GPIO_19",
&i2c0_imux {
pinctrl-names =
- "i2c100", "i2c101", "i2c102", "i2c103",
- "i2c104", "i2c105", "i2c106", "i2c107",
- "i2c108", "i2c109", "i2c110", "i2c111", "idle";
+ "i2c_sfp1", "i2c_sfp2", "i2c_sfp3", "i2c_sfp4",
+ "i2c_sfp5", "i2c_sfp6", "i2c_sfp7", "i2c_sfp8",
+ "i2c_sfp9", "i2c_sfp10", "i2c_sfp11", "i2c_sfp12", "idle";
pinctrl-0 = <&i2cmux_0>;
pinctrl-1 = <&i2cmux_1>;
pinctrl-2 = <&i2cmux_2>;
pinctrl-10 = <&i2cmux_10>;
pinctrl-11 = <&i2cmux_11>;
pinctrl-12 = <&i2cmux_pins_i>;
- i2c100: i2c_sfp1 {
+ i2c_sfp1: i2c_sfp1 {
reg = <0x0>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c101: i2c_sfp2 {
+ i2c_sfp2: i2c_sfp2 {
reg = <0x1>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c102: i2c_sfp3 {
+ i2c_sfp3: i2c_sfp3 {
reg = <0x2>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c103: i2c_sfp4 {
+ i2c_sfp4: i2c_sfp4 {
reg = <0x3>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c104: i2c_sfp5 {
+ i2c_sfp5: i2c_sfp5 {
reg = <0x4>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c105: i2c_sfp6 {
+ i2c_sfp6: i2c_sfp6 {
reg = <0x5>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c106: i2c_sfp7 {
+ i2c_sfp7: i2c_sfp7 {
reg = <0x6>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c107: i2c_sfp8 {
+ i2c_sfp8: i2c_sfp8 {
reg = <0x7>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c108: i2c_sfp9 {
+ i2c_sfp9: i2c_sfp9 {
reg = <0x8>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c109: i2c_sfp10 {
+ i2c_sfp10: i2c_sfp10 {
reg = <0x9>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c110: i2c_sfp11 {
+ i2c_sfp11: i2c_sfp11 {
reg = <0xa>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c111: i2c_sfp12 {
+ i2c_sfp12: i2c_sfp12 {
reg = <0xb>;
#address-cells = <1>;
#size-cells = <0>;
&gpio 61 GPIO_ACTIVE_HIGH
&gpio 54 GPIO_ACTIVE_HIGH>;
idle-state = <0x8>;
- i2c112: i2c_sfp13 {
+ i2c_sfp13: i2c_sfp13 {
reg = <0x0>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c113: i2c_sfp14 {
+ i2c_sfp14: i2c_sfp14 {
reg = <0x1>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c114: i2c_sfp15 {
+ i2c_sfp15: i2c_sfp15 {
reg = <0x2>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c115: i2c_sfp16 {
+ i2c_sfp16: i2c_sfp16 {
reg = <0x3>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c116: i2c_sfp17 {
+ i2c_sfp17: i2c_sfp17 {
reg = <0x4>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c117: i2c_sfp18 {
+ i2c_sfp18: i2c_sfp18 {
reg = <0x5>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c118: i2c_sfp19 {
+ i2c_sfp19: i2c_sfp19 {
reg = <0x6>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c119: i2c_sfp20 {
+ i2c_sfp20: i2c_sfp20 {
reg = <0x7>;
#address-cells = <1>;
#size-cells = <0>;
};
};
+
+&mdio3 {
+ status = "ok";
+ phy64: ethernet-phy@64 {
+ reg = <28>;
+ };
+};
+
+&axi {
+ sfp_eth12: sfp-eth12 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp1>;
+ tx-disable-gpios = <&sgpio_out2 11 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 11 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 11 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 12 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth13: sfp-eth13 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp2>;
+ tx-disable-gpios = <&sgpio_out2 12 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 12 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 12 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 13 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth14: sfp-eth14 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp3>;
+ tx-disable-gpios = <&sgpio_out2 13 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 13 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 13 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 14 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth15: sfp-eth15 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp4>;
+ tx-disable-gpios = <&sgpio_out2 14 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 14 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 14 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 15 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth48: sfp-eth48 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp5>;
+ tx-disable-gpios = <&sgpio_out2 15 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 15 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 15 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 16 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth49: sfp-eth49 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp6>;
+ tx-disable-gpios = <&sgpio_out2 16 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 16 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 16 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 17 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth50: sfp-eth50 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp7>;
+ tx-disable-gpios = <&sgpio_out2 17 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 17 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 17 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 18 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth51: sfp-eth51 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp8>;
+ tx-disable-gpios = <&sgpio_out2 18 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 18 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 18 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 19 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth52: sfp-eth52 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp9>;
+ tx-disable-gpios = <&sgpio_out2 19 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 19 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 19 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 20 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth53: sfp-eth53 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp10>;
+ tx-disable-gpios = <&sgpio_out2 20 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 20 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 20 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 21 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth54: sfp-eth54 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp11>;
+ tx-disable-gpios = <&sgpio_out2 21 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 21 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 21 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 22 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth55: sfp-eth55 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp12>;
+ tx-disable-gpios = <&sgpio_out2 22 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 22 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 22 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 23 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth56: sfp-eth56 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp13>;
+ tx-disable-gpios = <&sgpio_out2 23 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 23 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 23 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 24 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth57: sfp-eth57 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp14>;
+ tx-disable-gpios = <&sgpio_out2 24 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 24 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 24 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 25 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth58: sfp-eth58 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp15>;
+ tx-disable-gpios = <&sgpio_out2 25 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 25 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 25 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 26 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth59: sfp-eth59 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp16>;
+ tx-disable-gpios = <&sgpio_out2 26 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 26 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 26 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 27 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth60: sfp-eth60 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp17>;
+ tx-disable-gpios = <&sgpio_out2 27 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 27 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 27 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 28 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth61: sfp-eth61 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp18>;
+ tx-disable-gpios = <&sgpio_out2 28 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 28 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 28 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 29 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth62: sfp-eth62 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp19>;
+ tx-disable-gpios = <&sgpio_out2 29 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 29 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 29 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 30 0 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth63: sfp-eth63 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp20>;
+ tx-disable-gpios = <&sgpio_out2 30 1 GPIO_ACTIVE_LOW>;
+ los-gpios = <&sgpio_in2 30 1 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 30 2 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 31 0 GPIO_ACTIVE_HIGH>;
+ };
+};
+
+&switch {
+ ethernet-ports {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ /* 10G SFPs */
+ port12: port@12 {
+ reg = <12>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 13>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth12>;
+ microchip,sd-sgpio = <301>;
+ managed = "in-band-status";
+ };
+ port13: port@13 {
+ reg = <13>;
+ /* Example: CU SFP, 1G speed */
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 14>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth13>;
+ microchip,sd-sgpio = <305>;
+ managed = "in-band-status";
+ };
+ port14: port@14 {
+ reg = <14>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 15>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth14>;
+ microchip,sd-sgpio = <309>;
+ managed = "in-band-status";
+ };
+ port15: port@15 {
+ reg = <15>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 16>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth15>;
+ microchip,sd-sgpio = <313>;
+ managed = "in-band-status";
+ };
+ port48: port@48 {
+ reg = <48>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 17>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth48>;
+ microchip,sd-sgpio = <317>;
+ managed = "in-band-status";
+ };
+ port49: port@49 {
+ reg = <49>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 18>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth49>;
+ microchip,sd-sgpio = <321>;
+ managed = "in-band-status";
+ };
+ port50: port@50 {
+ reg = <50>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 19>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth50>;
+ microchip,sd-sgpio = <325>;
+ managed = "in-band-status";
+ };
+ port51: port@51 {
+ reg = <51>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 20>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth51>;
+ microchip,sd-sgpio = <329>;
+ managed = "in-band-status";
+ };
+ port52: port@52 {
+ reg = <52>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 21>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth52>;
+ microchip,sd-sgpio = <333>;
+ managed = "in-band-status";
+ };
+ port53: port@53 {
+ reg = <53>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 22>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth53>;
+ microchip,sd-sgpio = <337>;
+ managed = "in-band-status";
+ };
+ port54: port@54 {
+ reg = <54>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 23>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth54>;
+ microchip,sd-sgpio = <341>;
+ managed = "in-band-status";
+ };
+ port55: port@55 {
+ reg = <55>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 24>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth55>;
+ microchip,sd-sgpio = <345>;
+ managed = "in-band-status";
+ };
+ /* 25G SFPs */
+ port56: port@56 {
+ reg = <56>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 25>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth56>;
+ microchip,sd-sgpio = <349>;
+ managed = "in-band-status";
+ };
+ port57: port@57 {
+ reg = <57>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 26>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth57>;
+ microchip,sd-sgpio = <353>;
+ managed = "in-band-status";
+ };
+ port58: port@58 {
+ reg = <58>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 27>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth58>;
+ microchip,sd-sgpio = <357>;
+ managed = "in-band-status";
+ };
+ port59: port@59 {
+ reg = <59>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 28>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth59>;
+ microchip,sd-sgpio = <361>;
+ managed = "in-band-status";
+ };
+ port60: port@60 {
+ reg = <60>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 29>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth60>;
+ microchip,sd-sgpio = <365>;
+ managed = "in-band-status";
+ };
+ port61: port@61 {
+ reg = <61>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 30>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth61>;
+ microchip,sd-sgpio = <369>;
+ managed = "in-band-status";
+ };
+ port62: port@62 {
+ reg = <62>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 31>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth62>;
+ microchip,sd-sgpio = <373>;
+ managed = "in-band-status";
+ };
+ port63: port@63 {
+ reg = <63>;
+ microchip,bandwidth = <10000>;
+ phys = <&serdes 32>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth63>;
+ microchip,sd-sgpio = <377>;
+ managed = "in-band-status";
+ };
+ /* Finally the Management interface */
+ port64: port@64 {
+ reg = <64>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 0>;
+ phy-handle = <&phy64>;
+ phy-mode = "sgmii";
+ };
+ };
+};
#include "sparx5_pcb_common.dtsi"
/{
- aliases {
- i2c0 = &i2c0;
- i2c152 = &i2c152;
- i2c153 = &i2c153;
- i2c154 = &i2c154;
- i2c155 = &i2c155;
- };
-
gpio-restart {
compatible = "gpio-restart";
gpios = <&gpio 37 GPIO_ACTIVE_LOW>;
&spi0 {
status = "okay";
- spi@0 {
- compatible = "spi-mux";
- mux-controls = <&mux>;
- #address-cells = <1>;
- #size-cells = <0>;
- reg = <0>; /* CS0 */
- spi-flash@9 {
- compatible = "jedec,spi-nor";
- spi-max-frequency = <8000000>;
- reg = <0x9>; /* SPI */
- };
+ spi-flash@0 {
+ compatible = "jedec,spi-nor";
+ spi-max-frequency = <8000000>;
+ reg = <0>;
};
};
};
};
+&sgpio2 {
+ status = "okay";
+ microchip,sgpio-port-ranges = <0 0>, <16 18>, <28 31>;
+};
+
&axi {
i2c0_imux: i2c0-imux@0 {
compatible = "i2c-mux-pinctrl";
&i2c0_imux {
pinctrl-names =
- "i2c152", "i2c153", "i2c154", "i2c155",
+ "i2c_sfp1", "i2c_sfp2", "i2c_sfp3", "i2c_sfp4",
"idle";
pinctrl-0 = <&i2cmux_s29>;
pinctrl-1 = <&i2cmux_s30>;
pinctrl-2 = <&i2cmux_s31>;
pinctrl-3 = <&i2cmux_s32>;
pinctrl-4 = <&i2cmux_pins_i>;
- i2c152: i2c_sfp1 {
+ i2c_sfp1: i2c_sfp1 {
reg = <0x0>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c153: i2c_sfp2 {
+ i2c_sfp2: i2c_sfp2 {
reg = <0x1>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c154: i2c_sfp3 {
+ i2c_sfp3: i2c_sfp3 {
reg = <0x2>;
#address-cells = <1>;
#size-cells = <0>;
};
- i2c155: i2c_sfp4 {
+ i2c_sfp4: i2c_sfp4 {
reg = <0x3>;
#address-cells = <1>;
#size-cells = <0>;
};
};
+
+&axi {
+ sfp_eth60: sfp-eth60 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp1>;
+ tx-disable-gpios = <&sgpio_out2 28 0 GPIO_ACTIVE_LOW>;
+ rate-select0-gpios = <&sgpio_out2 28 1 GPIO_ACTIVE_HIGH>;
+ los-gpios = <&sgpio_in2 28 0 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 28 1 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 28 2 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth61: sfp-eth61 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp2>;
+ tx-disable-gpios = <&sgpio_out2 29 0 GPIO_ACTIVE_LOW>;
+ rate-select0-gpios = <&sgpio_out2 29 1 GPIO_ACTIVE_HIGH>;
+ los-gpios = <&sgpio_in2 29 0 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 29 1 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 29 2 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth62: sfp-eth62 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp3>;
+ tx-disable-gpios = <&sgpio_out2 30 0 GPIO_ACTIVE_LOW>;
+ rate-select0-gpios = <&sgpio_out2 30 1 GPIO_ACTIVE_HIGH>;
+ los-gpios = <&sgpio_in2 30 0 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 30 1 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 30 2 GPIO_ACTIVE_HIGH>;
+ };
+ sfp_eth63: sfp-eth63 {
+ compatible = "sff,sfp";
+ i2c-bus = <&i2c_sfp4>;
+ tx-disable-gpios = <&sgpio_out2 31 0 GPIO_ACTIVE_LOW>;
+ rate-select0-gpios = <&sgpio_out2 31 1 GPIO_ACTIVE_HIGH>;
+ los-gpios = <&sgpio_in2 31 0 GPIO_ACTIVE_HIGH>;
+ mod-def0-gpios = <&sgpio_in2 31 1 GPIO_ACTIVE_LOW>;
+ tx-fault-gpios = <&sgpio_in2 31 2 GPIO_ACTIVE_HIGH>;
+ };
+};
+
+&mdio0 {
+ status = "ok";
+ phy0: ethernet-phy@0 {
+ reg = <0>;
+ };
+ phy1: ethernet-phy@1 {
+ reg = <1>;
+ };
+ phy2: ethernet-phy@2 {
+ reg = <2>;
+ };
+ phy3: ethernet-phy@3 {
+ reg = <3>;
+ };
+ phy4: ethernet-phy@4 {
+ reg = <4>;
+ };
+ phy5: ethernet-phy@5 {
+ reg = <5>;
+ };
+ phy6: ethernet-phy@6 {
+ reg = <6>;
+ };
+ phy7: ethernet-phy@7 {
+ reg = <7>;
+ };
+ phy8: ethernet-phy@8 {
+ reg = <8>;
+ };
+ phy9: ethernet-phy@9 {
+ reg = <9>;
+ };
+ phy10: ethernet-phy@10 {
+ reg = <10>;
+ };
+ phy11: ethernet-phy@11 {
+ reg = <11>;
+ };
+ phy12: ethernet-phy@12 {
+ reg = <12>;
+ };
+ phy13: ethernet-phy@13 {
+ reg = <13>;
+ };
+ phy14: ethernet-phy@14 {
+ reg = <14>;
+ };
+ phy15: ethernet-phy@15 {
+ reg = <15>;
+ };
+ phy16: ethernet-phy@16 {
+ reg = <16>;
+ };
+ phy17: ethernet-phy@17 {
+ reg = <17>;
+ };
+ phy18: ethernet-phy@18 {
+ reg = <18>;
+ };
+ phy19: ethernet-phy@19 {
+ reg = <19>;
+ };
+ phy20: ethernet-phy@20 {
+ reg = <20>;
+ };
+ phy21: ethernet-phy@21 {
+ reg = <21>;
+ };
+ phy22: ethernet-phy@22 {
+ reg = <22>;
+ };
+ phy23: ethernet-phy@23 {
+ reg = <23>;
+ };
+};
+
+&mdio1 {
+ status = "ok";
+ phy24: ethernet-phy@24 {
+ reg = <0>;
+ };
+ phy25: ethernet-phy@25 {
+ reg = <1>;
+ };
+ phy26: ethernet-phy@26 {
+ reg = <2>;
+ };
+ phy27: ethernet-phy@27 {
+ reg = <3>;
+ };
+ phy28: ethernet-phy@28 {
+ reg = <4>;
+ };
+ phy29: ethernet-phy@29 {
+ reg = <5>;
+ };
+ phy30: ethernet-phy@30 {
+ reg = <6>;
+ };
+ phy31: ethernet-phy@31 {
+ reg = <7>;
+ };
+ phy32: ethernet-phy@32 {
+ reg = <8>;
+ };
+ phy33: ethernet-phy@33 {
+ reg = <9>;
+ };
+ phy34: ethernet-phy@34 {
+ reg = <10>;
+ };
+ phy35: ethernet-phy@35 {
+ reg = <11>;
+ };
+ phy36: ethernet-phy@36 {
+ reg = <12>;
+ };
+ phy37: ethernet-phy@37 {
+ reg = <13>;
+ };
+ phy38: ethernet-phy@38 {
+ reg = <14>;
+ };
+ phy39: ethernet-phy@39 {
+ reg = <15>;
+ };
+ phy40: ethernet-phy@40 {
+ reg = <16>;
+ };
+ phy41: ethernet-phy@41 {
+ reg = <17>;
+ };
+ phy42: ethernet-phy@42 {
+ reg = <18>;
+ };
+ phy43: ethernet-phy@43 {
+ reg = <19>;
+ };
+ phy44: ethernet-phy@44 {
+ reg = <20>;
+ };
+ phy45: ethernet-phy@45 {
+ reg = <21>;
+ };
+ phy46: ethernet-phy@46 {
+ reg = <22>;
+ };
+ phy47: ethernet-phy@47 {
+ reg = <23>;
+ };
+};
+
+&mdio3 {
+ status = "ok";
+ phy64: ethernet-phy@64 {
+ reg = <28>;
+ };
+};
+
+&switch {
+ ethernet-ports {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ port0: port@0 {
+ reg = <0>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 13>;
+ phy-handle = <&phy0>;
+ phy-mode = "qsgmii";
+ };
+ port1: port@1 {
+ reg = <1>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 13>;
+ phy-handle = <&phy1>;
+ phy-mode = "qsgmii";
+ };
+ port2: port@2 {
+ reg = <2>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 13>;
+ phy-handle = <&phy2>;
+ phy-mode = "qsgmii";
+ };
+ port3: port@3 {
+ reg = <3>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 13>;
+ phy-handle = <&phy3>;
+ phy-mode = "qsgmii";
+ };
+ port4: port@4 {
+ reg = <4>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 14>;
+ phy-handle = <&phy4>;
+ phy-mode = "qsgmii";
+ };
+ port5: port@5 {
+ reg = <5>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 14>;
+ phy-handle = <&phy5>;
+ phy-mode = "qsgmii";
+ };
+ port6: port@6 {
+ reg = <6>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 14>;
+ phy-handle = <&phy6>;
+ phy-mode = "qsgmii";
+ };
+ port7: port@7 {
+ reg = <7>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 14>;
+ phy-handle = <&phy7>;
+ phy-mode = "qsgmii";
+ };
+ port8: port@8 {
+ reg = <8>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 15>;
+ phy-handle = <&phy8>;
+ phy-mode = "qsgmii";
+ };
+ port9: port@9 {
+ reg = <9>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 15>;
+ phy-handle = <&phy9>;
+ phy-mode = "qsgmii";
+ };
+ port10: port@10 {
+ reg = <10>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 15>;
+ phy-handle = <&phy10>;
+ phy-mode = "qsgmii";
+ };
+ port11: port@11 {
+ reg = <11>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 15>;
+ phy-handle = <&phy11>;
+ phy-mode = "qsgmii";
+ };
+ port12: port@12 {
+ reg = <12>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 16>;
+ phy-handle = <&phy12>;
+ phy-mode = "qsgmii";
+ };
+ port13: port@13 {
+ reg = <13>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 16>;
+ phy-handle = <&phy13>;
+ phy-mode = "qsgmii";
+ };
+ port14: port@14 {
+ reg = <14>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 16>;
+ phy-handle = <&phy14>;
+ phy-mode = "qsgmii";
+ };
+ port15: port@15 {
+ reg = <15>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 16>;
+ phy-handle = <&phy15>;
+ phy-mode = "qsgmii";
+ };
+ port16: port@16 {
+ reg = <16>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 17>;
+ phy-handle = <&phy16>;
+ phy-mode = "qsgmii";
+ };
+ port17: port@17 {
+ reg = <17>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 17>;
+ phy-handle = <&phy17>;
+ phy-mode = "qsgmii";
+ };
+ port18: port@18 {
+ reg = <18>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 17>;
+ phy-handle = <&phy18>;
+ phy-mode = "qsgmii";
+ };
+ port19: port@19 {
+ reg = <19>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 17>;
+ phy-handle = <&phy19>;
+ phy-mode = "qsgmii";
+ };
+ port20: port@20 {
+ reg = <20>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 18>;
+ phy-handle = <&phy20>;
+ phy-mode = "qsgmii";
+ };
+ port21: port@21 {
+ reg = <21>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 18>;
+ phy-handle = <&phy21>;
+ phy-mode = "qsgmii";
+ };
+ port22: port@22 {
+ reg = <22>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 18>;
+ phy-handle = <&phy22>;
+ phy-mode = "qsgmii";
+ };
+ port23: port@23 {
+ reg = <23>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 18>;
+ phy-handle = <&phy23>;
+ phy-mode = "qsgmii";
+ };
+ port24: port@24 {
+ reg = <24>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 19>;
+ phy-handle = <&phy24>;
+ phy-mode = "qsgmii";
+ };
+ port25: port@25 {
+ reg = <25>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 19>;
+ phy-handle = <&phy25>;
+ phy-mode = "qsgmii";
+ };
+ port26: port@26 {
+ reg = <26>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 19>;
+ phy-handle = <&phy26>;
+ phy-mode = "qsgmii";
+ };
+ port27: port@27 {
+ reg = <27>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 19>;
+ phy-handle = <&phy27>;
+ phy-mode = "qsgmii";
+ };
+ port28: port@28 {
+ reg = <28>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 20>;
+ phy-handle = <&phy28>;
+ phy-mode = "qsgmii";
+ };
+ port29: port@29 {
+ reg = <29>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 20>;
+ phy-handle = <&phy29>;
+ phy-mode = "qsgmii";
+ };
+ port30: port@30 {
+ reg = <30>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 20>;
+ phy-handle = <&phy30>;
+ phy-mode = "qsgmii";
+ };
+ port31: port@31 {
+ reg = <31>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 20>;
+ phy-handle = <&phy31>;
+ phy-mode = "qsgmii";
+ };
+ port32: port@32 {
+ reg = <32>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 21>;
+ phy-handle = <&phy32>;
+ phy-mode = "qsgmii";
+ };
+ port33: port@33 {
+ reg = <33>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 21>;
+ phy-handle = <&phy33>;
+ phy-mode = "qsgmii";
+ };
+ port34: port@34 {
+ reg = <34>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 21>;
+ phy-handle = <&phy34>;
+ phy-mode = "qsgmii";
+ };
+ port35: port@35 {
+ reg = <35>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 21>;
+ phy-handle = <&phy35>;
+ phy-mode = "qsgmii";
+ };
+ port36: port@36 {
+ reg = <36>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 22>;
+ phy-handle = <&phy36>;
+ phy-mode = "qsgmii";
+ };
+ port37: port@37 {
+ reg = <37>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 22>;
+ phy-handle = <&phy37>;
+ phy-mode = "qsgmii";
+ };
+ port38: port@38 {
+ reg = <38>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 22>;
+ phy-handle = <&phy38>;
+ phy-mode = "qsgmii";
+ };
+ port39: port@39 {
+ reg = <39>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 22>;
+ phy-handle = <&phy39>;
+ phy-mode = "qsgmii";
+ };
+ port40: port@40 {
+ reg = <40>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 23>;
+ phy-handle = <&phy40>;
+ phy-mode = "qsgmii";
+ };
+ port41: port@41 {
+ reg = <41>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 23>;
+ phy-handle = <&phy41>;
+ phy-mode = "qsgmii";
+ };
+ port42: port@42 {
+ reg = <42>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 23>;
+ phy-handle = <&phy42>;
+ phy-mode = "qsgmii";
+ };
+ port43: port@43 {
+ reg = <43>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 23>;
+ phy-handle = <&phy43>;
+ phy-mode = "qsgmii";
+ };
+ port44: port@44 {
+ reg = <44>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 24>;
+ phy-handle = <&phy44>;
+ phy-mode = "qsgmii";
+ };
+ port45: port@45 {
+ reg = <45>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 24>;
+ phy-handle = <&phy45>;
+ phy-mode = "qsgmii";
+ };
+ port46: port@46 {
+ reg = <46>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 24>;
+ phy-handle = <&phy46>;
+ phy-mode = "qsgmii";
+ };
+ port47: port@47 {
+ reg = <47>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 24>;
+ phy-handle = <&phy47>;
+ phy-mode = "qsgmii";
+ };
+ /* Then the 25G interfaces */
+ port60: port@60 {
+ reg = <60>;
+ microchip,bandwidth = <25000>;
+ phys = <&serdes 29>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth60>;
+ managed = "in-band-status";
+ };
+ port61: port@61 {
+ reg = <61>;
+ microchip,bandwidth = <25000>;
+ phys = <&serdes 30>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth61>;
+ managed = "in-band-status";
+ };
+ port62: port@62 {
+ reg = <62>;
+ microchip,bandwidth = <25000>;
+ phys = <&serdes 31>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth62>;
+ managed = "in-band-status";
+ };
+ port63: port@63 {
+ reg = <63>;
+ microchip,bandwidth = <25000>;
+ phys = <&serdes 32>;
+ phy-mode = "10gbase-r";
+ sfp = <&sfp_eth63>;
+ managed = "in-band-status";
+ };
+ /* Finally the Management interface */
+ port64: port@64 {
+ reg = <64>;
+ microchip,bandwidth = <1000>;
+ phys = <&serdes 0>;
+ phy-handle = <&phy64>;
+ phy-mode = "sgmii";
+ };
+ };
+};
ports {
port@0 {
- reg = <0>;
csi20_in: endpoint {
clock-lanes = <0>;
data-lanes = <1 2>;
ports {
port@0 {
- reg = <0>;
csi40_in: endpoint {
clock-lanes = <0>;
data-lanes = <1 2>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
status = "okay";
ports {
- port {
+ port@0 {
csi40_in: endpoint {
clock-lanes = <0>;
data-lanes = <1 2>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
ports {
port@0 {
- reg = <0>;
-
csi40_in: endpoint {
clock-lanes = <0>;
data-lanes = <1 2>;
#address-cells = <1>;
#size-cells = <0>;
+ port@0 {
+ reg = <0>;
+ };
+
port@1 {
#address-cells = <1>;
#size-cells = <0>;
ports {
port@0 {
- reg = <0>;
csi20_in: endpoint {
clock-lanes = <0>;
data-lanes = <1>;
ports {
port@0 {
- reg = <0>;
-
csi40_in: endpoint {
clock-lanes = <0>;
data-lanes = <1 2 3 4>;
status = "disabled";
};
+ gmac: ethernet@ff4e0000 {
+ compatible = "rockchip,rk3308-gmac";
+ reg = <0x0 0xff4e0000 0x0 0x10000>;
+ interrupts = <GIC_SPI 64 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-names = "macirq";
+ clocks = <&cru SCLK_MAC>, <&cru SCLK_MAC_RX_TX>,
+ <&cru SCLK_MAC_RX_TX>, <&cru SCLK_MAC_REF>,
+ <&cru SCLK_MAC>, <&cru ACLK_MAC>,
+ <&cru PCLK_MAC>, <&cru SCLK_MAC_RMII>;
+ clock-names = "stmmaceth", "mac_clk_rx",
+ "mac_clk_tx", "clk_mac_ref",
+ "clk_mac_refout", "aclk_mac",
+ "pclk_mac", "clk_mac_speed";
+ phy-mode = "rmii";
+ pinctrl-names = "default";
+ pinctrl-0 = <&rmii_pins &mac_refclk_12ma>;
+ resets = <&cru SRST_MAC_A>;
+ reset-names = "stmmaceth";
+ rockchip,grf = <&grf>;
+ status = "disabled";
+ };
+
cru: clock-controller@ff500000 {
compatible = "rockchip,rk3308-cru";
reg = <0x0 0xff500000 0x0 0x1000>;
};
};
- dmss: dmss {
+ dmss: bus@48000000 {
compatible = "simple-mfd";
#address-cells = <2>;
#size-cells = <2>;
dma-ranges;
- ranges;
+ ranges = <0x00 0x48000000 0x00 0x48000000 0x00 0x06400000>;
ti,sci-dev-id = <25>;
};
};
- dmsc: dmsc@44043000 {
+ dmsc: system-controller@44043000 {
compatible = "ti,k2g-sci";
ti,host-id = <12>;
mbox-names = "rx", "tx";
#power-domain-cells = <2>;
};
- k3_clks: clocks {
+ k3_clks: clock-controller {
compatible = "ti,k2g-sci-clk";
#clock-cells = <2>;
};
clocks = <&k3_clks 145 0>;
};
- main_gpio_intr: interrupt-controller0 {
+ main_gpio_intr: interrupt-controller@a00000 {
compatible = "ti,sci-intr";
+ reg = <0x00 0x00a00000 0x00 0x800>;
ti,intr-trigger-type = <1>;
interrupt-controller;
interrupt-parent = <&gic500>;
clocks = <&k3_clks 148 0>;
};
- mcu_gpio_intr: interrupt-controller1 {
+ mcu_gpio_intr: interrupt-controller@4210000 {
compatible = "ti,sci-intr";
+ reg = <0x00 0x04210000 0x00 0x200>;
ti,intr-trigger-type = <1>;
interrupt-controller;
interrupt-parent = <&gic500>;
#phy-cells = <0>;
};
- intr_main_gpio: interrupt-controller0 {
+ intr_main_gpio: interrupt-controller@a00000 {
compatible = "ti,sci-intr";
+ reg = <0x0 0x00a00000 0x0 0x400>;
ti,intr-trigger-type = <1>;
interrupt-controller;
interrupt-parent = <&gic500>;
ti,interrupt-ranges = <0 392 32>;
};
- main-navss {
+ main_navss: bus@30800000 {
compatible = "simple-mfd";
#address-cells = <2>;
#size-cells = <2>;
- ranges;
+ ranges = <0x0 0x30800000 0x0 0x30800000 0x0 0xbc00000>;
dma-coherent;
dma-ranges;
ti,sci-dev-id = <118>;
- intr_main_navss: interrupt-controller1 {
+ intr_main_navss: interrupt-controller@310e0000 {
compatible = "ti,sci-intr";
+ reg = <0x0 0x310e0000 0x0 0x2000>;
ti,intr-trigger-type = <4>;
interrupt-controller;
interrupt-parent = <&gic500>;
};
};
- mcu-navss {
+ mcu_navss: bus@28380000 {
compatible = "simple-mfd";
#address-cells = <2>;
#size-cells = <2>;
- ranges;
+ ranges = <0x00 0x28380000 0x00 0x28380000 0x00 0x03880000>;
dma-coherent;
dma-ranges;
*/
&cbass_wakeup {
- dmsc: dmsc {
+ dmsc: system-controller@44083000 {
compatible = "ti,am654-sci";
ti,host-id = <12>;
- #address-cells = <1>;
- #size-cells = <1>;
- ranges;
mbox-names = "rx", "tx";
mboxes= <&secure_proxy_main 11>,
<&secure_proxy_main 13>;
+ reg-names = "debug_messages";
+ reg = <0x44083000 0x1000>;
+
k3_pds: power-controller {
compatible = "ti,sci-pm-domain";
#power-domain-cells = <2>;
};
- k3_clks: clocks {
+ k3_clks: clock-controller {
compatible = "ti,k2g-sci-clk";
#clock-cells = <2>;
};
power-domains = <&k3_pds 115 TI_SCI_PD_EXCLUSIVE>;
};
- intr_wkup_gpio: interrupt-controller2 {
+ intr_wkup_gpio: interrupt-controller@42200000 {
compatible = "ti,sci-intr";
+ reg = <0x42200000 0x200>;
ti,intr-trigger-type = <1>;
interrupt-controller;
interrupt-parent = <&gic500>;
gpios = <&wkup_gpio0 27 GPIO_ACTIVE_LOW>;
};
};
-
- clk_ov5640_fixed: clock {
- compatible = "fixed-clock";
- #clock-cells = <0>;
- clock-frequency = <24000000>;
- };
};
&wkup_pmx0 {
pinctrl-names = "default";
pinctrl-0 = <&main_i2c1_pins_default>;
clock-frequency = <400000>;
-
- ov5640: camera@3c {
- compatible = "ovti,ov5640";
- reg = <0x3c>;
-
- clocks = <&clk_ov5640_fixed>;
- clock-names = "xclk";
-
- port {
- csi2_cam0: endpoint {
- remote-endpoint = <&csi2_phy0>;
- clock-lanes = <0>;
- data-lanes = <1 2>;
- };
- };
- };
-
};
&main_i2c2 {
};
};
-&csi2_0 {
- csi2_phy0: endpoint {
- remote-endpoint = <&csi2_cam0>;
- clock-lanes = <0>;
- data-lanes = <1 2>;
- };
-};
-
&mcu_cpsw {
pinctrl-names = "default";
pinctrl-0 = <&mcu_cpsw_pins_default &mcu_mdio_pins_default>;
};
};
- main_gpio_intr: interrupt-controller0 {
+ main_gpio_intr: interrupt-controller@a00000 {
compatible = "ti,sci-intr";
+ reg = <0x00 0x00a00000 0x00 0x800>;
ti,intr-trigger-type = <1>;
interrupt-controller;
interrupt-parent = <&gic500>;
#size-cells = <2>;
ranges = <0x00 0x30000000 0x00 0x30000000 0x00 0x0c400000>;
ti,sci-dev-id = <199>;
+ dma-coherent;
+ dma-ranges;
- main_navss_intr: interrupt-controller1 {
+ main_navss_intr: interrupt-controller@310e0000 {
compatible = "ti,sci-intr";
+ reg = <0x00 0x310e0000 0x00 0x4000>;
ti,intr-trigger-type = <4>;
interrupt-controller;
interrupt-parent = <&gic500>;
*/
&cbass_mcu_wakeup {
- dmsc: dmsc@44083000 {
+ dmsc: system-controller@44083000 {
compatible = "ti,k2g-sci";
ti,host-id = <12>;
#power-domain-cells = <2>;
};
- k3_clks: clocks {
+ k3_clks: clock-controller {
compatible = "ti,k2g-sci-clk";
#clock-cells = <2>;
};
clock-names = "fclk";
};
- wkup_gpio_intr: interrupt-controller2 {
+ wkup_gpio_intr: interrupt-controller@42200000 {
compatible = "ti,sci-intr";
+ reg = <0x00 0x42200000 0x00 0x400>;
ti,intr-trigger-type = <1>;
interrupt-controller;
interrupt-parent = <&gic500>;
};
};
- main_gpio_intr: interrupt-controller0 {
+ main_gpio_intr: interrupt-controller@a00000 {
compatible = "ti,sci-intr";
+ reg = <0x00 0x00a00000 0x00 0x800>;
ti,intr-trigger-type = <1>;
interrupt-controller;
interrupt-parent = <&gic500>;
ti,interrupt-ranges = <8 392 56>;
};
- main-navss {
+ main_navss: bus@30000000 {
compatible = "simple-mfd";
#address-cells = <2>;
#size-cells = <2>;
- ranges;
+ ranges = <0x00 0x30000000 0x00 0x30000000 0x00 0x0c400000>;
dma-coherent;
dma-ranges;
ti,sci-dev-id = <199>;
- main_navss_intr: interrupt-controller1 {
+ main_navss_intr: interrupt-controller@310e0000 {
compatible = "ti,sci-intr";
+ reg = <0x0 0x310e0000 0x0 0x4000>;
ti,intr-trigger-type = <4>;
interrupt-controller;
interrupt-parent = <&gic500>;
*/
&cbass_mcu_wakeup {
- dmsc: dmsc@44083000 {
+ dmsc: system-controller@44083000 {
compatible = "ti,k2g-sci";
ti,host-id = <12>;
#power-domain-cells = <2>;
};
- k3_clks: clocks {
+ k3_clks: clock-controller {
compatible = "ti,k2g-sci-clk";
#clock-cells = <2>;
};
clock-names = "fclk";
};
- wkup_gpio_intr: interrupt-controller2 {
+ wkup_gpio_intr: interrupt-controller@42200000 {
compatible = "ti,sci-intr";
+ reg = <0x00 0x42200000 0x00 0x400>;
ti,intr-trigger-type = <1>;
interrupt-controller;
interrupt-parent = <&gic500>;
};
};
- mcu-navss {
+ mcu_navss: bus@28380000 {
compatible = "simple-mfd";
#address-cells = <2>;
#size-cells = <2>;
- ranges;
+ ranges = <0x00 0x28380000 0x00 0x28380000 0x00 0x03880000>;
dma-coherent;
dma-ranges;
generic-y += qspinlock.h
generic-y += set_memory.h
generic-y += user.h
+
+generated-y += cpucaps.h
* This insanity brought to you by speculative system register reads,
* out-of-order memory accesses, sequence locks and Thomas Gleixner.
*
- * http://lists.infradead.org/pipermail/linux-arm-kernel/2019-February/631195.html
+ * https://lore.kernel.org/r/alpine.DEB.2.21.1902081950260.1662@nanos.tec.linutronix.de/
*/
#define arch_counter_enforce_ordering(val) do { \
u64 tmp, _val = (val); \
+++ /dev/null
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * arch/arm64/include/asm/cpucaps.h
- *
- * Copyright (C) 2016 ARM Ltd.
- */
-#ifndef __ASM_CPUCAPS_H
-#define __ASM_CPUCAPS_H
-
-#define ARM64_WORKAROUND_CLEAN_CACHE 0
-#define ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE 1
-#define ARM64_WORKAROUND_845719 2
-#define ARM64_HAS_SYSREG_GIC_CPUIF 3
-#define ARM64_HAS_PAN 4
-#define ARM64_HAS_LSE_ATOMICS 5
-#define ARM64_WORKAROUND_CAVIUM_23154 6
-#define ARM64_WORKAROUND_834220 7
-#define ARM64_HAS_NO_HW_PREFETCH 8
-#define ARM64_HAS_VIRT_HOST_EXTN 11
-#define ARM64_WORKAROUND_CAVIUM_27456 12
-#define ARM64_HAS_32BIT_EL0 13
-#define ARM64_SPECTRE_V3A 14
-#define ARM64_HAS_CNP 15
-#define ARM64_HAS_NO_FPSIMD 16
-#define ARM64_WORKAROUND_REPEAT_TLBI 17
-#define ARM64_WORKAROUND_QCOM_FALKOR_E1003 18
-#define ARM64_WORKAROUND_858921 19
-#define ARM64_WORKAROUND_CAVIUM_30115 20
-#define ARM64_HAS_DCPOP 21
-#define ARM64_SVE 22
-#define ARM64_UNMAP_KERNEL_AT_EL0 23
-#define ARM64_SPECTRE_V2 24
-#define ARM64_HAS_RAS_EXTN 25
-#define ARM64_WORKAROUND_843419 26
-#define ARM64_HAS_CACHE_IDC 27
-#define ARM64_HAS_CACHE_DIC 28
-#define ARM64_HW_DBM 29
-#define ARM64_SPECTRE_V4 30
-#define ARM64_MISMATCHED_CACHE_TYPE 31
-#define ARM64_HAS_STAGE2_FWB 32
-#define ARM64_HAS_CRC32 33
-#define ARM64_SSBS 34
-#define ARM64_WORKAROUND_1418040 35
-#define ARM64_HAS_SB 36
-#define ARM64_WORKAROUND_SPECULATIVE_AT 37
-#define ARM64_HAS_ADDRESS_AUTH_ARCH 38
-#define ARM64_HAS_ADDRESS_AUTH_IMP_DEF 39
-#define ARM64_HAS_GENERIC_AUTH_ARCH 40
-#define ARM64_HAS_GENERIC_AUTH_IMP_DEF 41
-#define ARM64_HAS_IRQ_PRIO_MASKING 42
-#define ARM64_HAS_DCPODP 43
-#define ARM64_WORKAROUND_1463225 44
-#define ARM64_WORKAROUND_CAVIUM_TX2_219_TVM 45
-#define ARM64_WORKAROUND_CAVIUM_TX2_219_PRFM 46
-#define ARM64_WORKAROUND_1542419 47
-#define ARM64_HAS_E0PD 48
-#define ARM64_HAS_RNG 49
-#define ARM64_HAS_AMU_EXTN 50
-#define ARM64_HAS_ADDRESS_AUTH 51
-#define ARM64_HAS_GENERIC_AUTH 52
-#define ARM64_HAS_32BIT_EL1 53
-#define ARM64_BTI 54
-#define ARM64_HAS_ARMv8_4_TTL 55
-#define ARM64_HAS_TLB_RANGE 56
-#define ARM64_MTE 57
-#define ARM64_WORKAROUND_1508412 58
-#define ARM64_HAS_LDAPR 59
-#define ARM64_KVM_PROTECTED_MODE 60
-#define ARM64_WORKAROUND_NVIDIA_CARMEL_CNP 61
-#define ARM64_HAS_EPAN 62
-
-#define ARM64_NCAPS 63
-
-#endif /* __ASM_CPUCAPS_H */
#define __KVM_HOST_SMCCC_FUNC___pkvm_cpu_set_vector 18
#define __KVM_HOST_SMCCC_FUNC___pkvm_prot_finalize 19
#define __KVM_HOST_SMCCC_FUNC___pkvm_mark_hyp 20
+#define __KVM_HOST_SMCCC_FUNC___kvm_adjust_pc 21
#ifndef __ASSEMBLY__
extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
+extern void __kvm_adjust_pc(struct kvm_vcpu *vcpu);
+
extern u64 __vgic_v3_get_gic_config(void);
extern u64 __vgic_v3_read_vmcr(void);
extern void __vgic_v3_write_vmcr(u32 vmcr);
vcpu->arch.flags |= KVM_ARM64_INCREMENT_PC;
}
+static inline bool vcpu_has_feature(struct kvm_vcpu *vcpu, int feature)
+{
+ return test_bit(feature, vcpu->arch.features);
+}
+
#endif /* __ARM64_KVM_EMULATE_H__ */
__SYSCALL(__NR_epoll_pwait2, compat_sys_epoll_pwait2)
#define __NR_mount_setattr 442
__SYSCALL(__NR_mount_setattr, sys_mount_setattr)
-#define __NR_quotactl_path 443
-__SYSCALL(__NR_quotactl_path, sys_quotactl_path)
+/* 443 is reserved for quotactl_path */
#define __NR_landlock_create_ruleset 444
__SYSCALL(__NR_landlock_create_ruleset, sys_landlock_create_ruleset)
#define __NR_landlock_add_rule 445
return ret;
}
- if (run->immediate_exit)
- return -EINTR;
-
vcpu_load(vcpu);
+ if (run->immediate_exit) {
+ ret = -EINTR;
+ goto out;
+ }
+
kvm_sigset_activate(vcpu);
ret = 1;
kvm_sigset_deactivate(vcpu);
+out:
+ /*
+ * In the unlikely event that we are returning to userspace
+ * with pending exceptions or PC adjustment, commit these
+ * adjustments in order to give userspace a consistent view of
+ * the vcpu state. Note that this relies on __kvm_adjust_pc()
+ * being preempt-safe on VHE.
+ */
+ if (unlikely(vcpu->arch.flags & (KVM_ARM64_PENDING_EXCEPTION |
+ KVM_ARM64_INCREMENT_PC)))
+ kvm_call_hyp(__kvm_adjust_pc, vcpu);
+
vcpu_put(vcpu);
return ret;
}
*vcpu_pc(vcpu) = vect_offset;
}
-void kvm_inject_exception(struct kvm_vcpu *vcpu)
+static void kvm_inject_exception(struct kvm_vcpu *vcpu)
{
if (vcpu_el1_is_32bit(vcpu)) {
switch (vcpu->arch.flags & KVM_ARM64_EXCEPT_MASK) {
}
}
}
+
+/*
+ * Adjust the guest PC (and potentially exception state) depending on
+ * flags provided by the emulation code.
+ */
+void __kvm_adjust_pc(struct kvm_vcpu *vcpu)
+{
+ if (vcpu->arch.flags & KVM_ARM64_PENDING_EXCEPTION) {
+ kvm_inject_exception(vcpu);
+ vcpu->arch.flags &= ~(KVM_ARM64_PENDING_EXCEPTION |
+ KVM_ARM64_EXCEPT_MASK);
+ } else if (vcpu->arch.flags & KVM_ARM64_INCREMENT_PC) {
+ kvm_skip_instr(vcpu);
+ vcpu->arch.flags &= ~KVM_ARM64_INCREMENT_PC;
+ }
+}
#include <asm/kvm_emulate.h>
#include <asm/kvm_host.h>
-void kvm_inject_exception(struct kvm_vcpu *vcpu);
-
static inline void kvm_skip_instr(struct kvm_vcpu *vcpu)
{
if (vcpu_mode_is_32bit(vcpu)) {
}
/*
- * Adjust the guest PC on entry, depending on flags provided by EL1
- * for the purpose of emulation (MMIO, sysreg) or exception injection.
- */
-static inline void __adjust_pc(struct kvm_vcpu *vcpu)
-{
- if (vcpu->arch.flags & KVM_ARM64_PENDING_EXCEPTION) {
- kvm_inject_exception(vcpu);
- vcpu->arch.flags &= ~(KVM_ARM64_PENDING_EXCEPTION |
- KVM_ARM64_EXCEPT_MASK);
- } else if (vcpu->arch.flags & KVM_ARM64_INCREMENT_PC) {
- kvm_skip_instr(vcpu);
- vcpu->arch.flags &= ~KVM_ARM64_INCREMENT_PC;
- }
-}
-
-/*
* Skip an instruction while host sysregs are live.
* Assumes host is always 64-bit.
*/
cpu_reg(host_ctxt, 1) = __kvm_vcpu_run(kern_hyp_va(vcpu));
}
+static void handle___kvm_adjust_pc(struct kvm_cpu_context *host_ctxt)
+{
+ DECLARE_REG(struct kvm_vcpu *, vcpu, host_ctxt, 1);
+
+ __kvm_adjust_pc(kern_hyp_va(vcpu));
+}
+
static void handle___kvm_flush_vm_context(struct kvm_cpu_context *host_ctxt)
{
__kvm_flush_vm_context();
static const hcall_t host_hcall[] = {
HANDLE_FUNC(__kvm_vcpu_run),
+ HANDLE_FUNC(__kvm_adjust_pc),
HANDLE_FUNC(__kvm_flush_vm_context),
HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa),
HANDLE_FUNC(__kvm_tlb_flush_vmid),
extern unsigned long hyp_nr_cpus;
struct host_kvm host_kvm;
-struct hyp_pool host_s2_mem;
-struct hyp_pool host_s2_dev;
+static struct hyp_pool host_s2_mem;
+static struct hyp_pool host_s2_dev;
/*
* Copies of the host's CPU features registers holding sanitized values.
#include <nvhe/trap_handler.h>
struct hyp_pool hpool;
-struct kvm_pgtable_mm_ops pkvm_pgtable_mm_ops;
unsigned long hyp_nr_cpus;
#define hyp_percpu_size ((unsigned long)__per_cpu_end - \
static void *hyp_pgt_base;
static void *host_s2_mem_pgt_base;
static void *host_s2_dev_pgt_base;
+static struct kvm_pgtable_mm_ops pkvm_pgtable_mm_ops;
static int divide_memory_pool(void *virt, unsigned long size)
{
* Author: Marc Zyngier <marc.zyngier@arm.com>
*/
-#include <hyp/adjust_pc.h>
#include <hyp/switch.h>
#include <hyp/sysreg-sr.h>
*/
__debug_save_host_buffers_nvhe(vcpu);
- __adjust_pc(vcpu);
+ __kvm_adjust_pc(vcpu);
/*
* We must restore the 32-bit state before the sysregs, thanks
* Author: Marc Zyngier <marc.zyngier@arm.com>
*/
-#include <hyp/adjust_pc.h>
#include <hyp/switch.h>
#include <linux/arm-smccc.h>
__load_guest_stage2(vcpu->arch.hw_mmu);
__activate_traps(vcpu);
- __adjust_pc(vcpu);
+ __kvm_adjust_pc(vcpu);
sysreg_restore_guest_state_vhe(guest_ctxt);
__debug_switch_to_guest(vcpu);
bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
{
if (!kvm->arch.mmu.pgt)
- return 0;
+ return false;
__unmap_stage2_range(&kvm->arch.mmu, range->start << PAGE_SHIFT,
(range->end - range->start) << PAGE_SHIFT,
range->may_block);
- return 0;
+ return false;
}
bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
kvm_pfn_t pfn = pte_pfn(range->pte);
if (!kvm->arch.mmu.pgt)
- return 0;
+ return false;
WARN_ON(range->end - range->start != 1);
PAGE_SIZE, __pfn_to_phys(pfn),
KVM_PGTABLE_PROT_R, NULL);
- return 0;
+ return false;
}
bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
pte_t pte;
if (!kvm->arch.mmu.pgt)
- return 0;
+ return false;
WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE);
bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
{
if (!kvm->arch.mmu.pgt)
- return 0;
+ return false;
return kvm_pgtable_stage2_is_young(kvm->arch.mmu.pgt,
range->start << PAGE_SHIFT);
return 0;
}
+static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
+{
+ struct kvm_vcpu *tmp;
+ bool is32bit;
+ int i;
+
+ is32bit = vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
+ if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1) && is32bit)
+ return false;
+
+ /* Check that the vcpus are either all 32bit or all 64bit */
+ kvm_for_each_vcpu(i, tmp, vcpu->kvm) {
+ if (vcpu_has_feature(tmp, KVM_ARM_VCPU_EL1_32BIT) != is32bit)
+ return false;
+ }
+
+ return true;
+}
+
/**
* kvm_reset_vcpu - sets core registers and sys_regs to reset value
* @vcpu: The VCPU pointer
}
}
+ if (!vcpu_allowed_register_width(vcpu)) {
+ ret = -EINVAL;
+ goto out;
+ }
+
switch (vcpu->arch.target) {
default:
if (test_bit(KVM_ARM_VCPU_EL1_32BIT, vcpu->arch.features)) {
- if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1)) {
- ret = -EINVAL;
- goto out;
- }
pstate = VCPU_RESET_PSTATE_SVC;
} else {
pstate = VCPU_RESET_PSTATE_EL1;
struct sys_reg_params *p,
const struct sys_reg_desc *rd)
{
- u64 *dbg_reg = &vcpu->arch.vcpu_debug_state.dbg_bvr[rd->reg];
+ u64 *dbg_reg = &vcpu->arch.vcpu_debug_state.dbg_bvr[rd->CRm];
if (p->is_write)
reg_to_dbg(vcpu, p, rd, dbg_reg);
else
dbg_to_reg(vcpu, p, rd, dbg_reg);
- trace_trap_reg(__func__, rd->reg, p->is_write, *dbg_reg);
+ trace_trap_reg(__func__, rd->CRm, p->is_write, *dbg_reg);
return true;
}
static int set_bvr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
const struct kvm_one_reg *reg, void __user *uaddr)
{
- __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_bvr[rd->reg];
+ __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_bvr[rd->CRm];
if (copy_from_user(r, uaddr, KVM_REG_SIZE(reg->id)) != 0)
return -EFAULT;
static int get_bvr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
const struct kvm_one_reg *reg, void __user *uaddr)
{
- __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_bvr[rd->reg];
+ __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_bvr[rd->CRm];
if (copy_to_user(uaddr, r, KVM_REG_SIZE(reg->id)) != 0)
return -EFAULT;
static void reset_bvr(struct kvm_vcpu *vcpu,
const struct sys_reg_desc *rd)
{
- vcpu->arch.vcpu_debug_state.dbg_bvr[rd->reg] = rd->val;
+ vcpu->arch.vcpu_debug_state.dbg_bvr[rd->CRm] = rd->val;
}
static bool trap_bcr(struct kvm_vcpu *vcpu,
struct sys_reg_params *p,
const struct sys_reg_desc *rd)
{
- u64 *dbg_reg = &vcpu->arch.vcpu_debug_state.dbg_bcr[rd->reg];
+ u64 *dbg_reg = &vcpu->arch.vcpu_debug_state.dbg_bcr[rd->CRm];
if (p->is_write)
reg_to_dbg(vcpu, p, rd, dbg_reg);
else
dbg_to_reg(vcpu, p, rd, dbg_reg);
- trace_trap_reg(__func__, rd->reg, p->is_write, *dbg_reg);
+ trace_trap_reg(__func__, rd->CRm, p->is_write, *dbg_reg);
return true;
}
static int set_bcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
const struct kvm_one_reg *reg, void __user *uaddr)
{
- __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_bcr[rd->reg];
+ __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_bcr[rd->CRm];
if (copy_from_user(r, uaddr, KVM_REG_SIZE(reg->id)) != 0)
return -EFAULT;
static int get_bcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
const struct kvm_one_reg *reg, void __user *uaddr)
{
- __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_bcr[rd->reg];
+ __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_bcr[rd->CRm];
if (copy_to_user(uaddr, r, KVM_REG_SIZE(reg->id)) != 0)
return -EFAULT;
static void reset_bcr(struct kvm_vcpu *vcpu,
const struct sys_reg_desc *rd)
{
- vcpu->arch.vcpu_debug_state.dbg_bcr[rd->reg] = rd->val;
+ vcpu->arch.vcpu_debug_state.dbg_bcr[rd->CRm] = rd->val;
}
static bool trap_wvr(struct kvm_vcpu *vcpu,
struct sys_reg_params *p,
const struct sys_reg_desc *rd)
{
- u64 *dbg_reg = &vcpu->arch.vcpu_debug_state.dbg_wvr[rd->reg];
+ u64 *dbg_reg = &vcpu->arch.vcpu_debug_state.dbg_wvr[rd->CRm];
if (p->is_write)
reg_to_dbg(vcpu, p, rd, dbg_reg);
else
dbg_to_reg(vcpu, p, rd, dbg_reg);
- trace_trap_reg(__func__, rd->reg, p->is_write,
- vcpu->arch.vcpu_debug_state.dbg_wvr[rd->reg]);
+ trace_trap_reg(__func__, rd->CRm, p->is_write,
+ vcpu->arch.vcpu_debug_state.dbg_wvr[rd->CRm]);
return true;
}
static int set_wvr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
const struct kvm_one_reg *reg, void __user *uaddr)
{
- __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_wvr[rd->reg];
+ __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_wvr[rd->CRm];
if (copy_from_user(r, uaddr, KVM_REG_SIZE(reg->id)) != 0)
return -EFAULT;
static int get_wvr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
const struct kvm_one_reg *reg, void __user *uaddr)
{
- __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_wvr[rd->reg];
+ __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_wvr[rd->CRm];
if (copy_to_user(uaddr, r, KVM_REG_SIZE(reg->id)) != 0)
return -EFAULT;
static void reset_wvr(struct kvm_vcpu *vcpu,
const struct sys_reg_desc *rd)
{
- vcpu->arch.vcpu_debug_state.dbg_wvr[rd->reg] = rd->val;
+ vcpu->arch.vcpu_debug_state.dbg_wvr[rd->CRm] = rd->val;
}
static bool trap_wcr(struct kvm_vcpu *vcpu,
struct sys_reg_params *p,
const struct sys_reg_desc *rd)
{
- u64 *dbg_reg = &vcpu->arch.vcpu_debug_state.dbg_wcr[rd->reg];
+ u64 *dbg_reg = &vcpu->arch.vcpu_debug_state.dbg_wcr[rd->CRm];
if (p->is_write)
reg_to_dbg(vcpu, p, rd, dbg_reg);
else
dbg_to_reg(vcpu, p, rd, dbg_reg);
- trace_trap_reg(__func__, rd->reg, p->is_write, *dbg_reg);
+ trace_trap_reg(__func__, rd->CRm, p->is_write, *dbg_reg);
return true;
}
static int set_wcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
const struct kvm_one_reg *reg, void __user *uaddr)
{
- __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_wcr[rd->reg];
+ __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_wcr[rd->CRm];
if (copy_from_user(r, uaddr, KVM_REG_SIZE(reg->id)) != 0)
return -EFAULT;
static int get_wcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
const struct kvm_one_reg *reg, void __user *uaddr)
{
- __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_wcr[rd->reg];
+ __u64 *r = &vcpu->arch.vcpu_debug_state.dbg_wcr[rd->CRm];
if (copy_to_user(uaddr, r, KVM_REG_SIZE(reg->id)) != 0)
return -EFAULT;
static void reset_wcr(struct kvm_vcpu *vcpu,
const struct sys_reg_desc *rd)
{
- vcpu->arch.vcpu_debug_state.dbg_wcr[rd->reg] = rd->val;
+ vcpu->arch.vcpu_debug_state.dbg_wcr[rd->CRm] = rd->val;
}
static void reset_amair_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
{
struct page *page = pte_page(pte);
- if (!test_and_set_bit(PG_dcache_clean, &page->flags))
+ if (!test_bit(PG_dcache_clean, &page->flags)) {
sync_icache_aliases(page_address(page), page_size(page));
+ set_bit(PG_dcache_clean, &page->flags);
+ }
}
EXPORT_SYMBOL_GPL(__sync_icache_dcache);
#include <linux/sizes.h>
#include <asm/tlb.h>
#include <asm/alternative.h>
+#include <asm/xen/swiotlb-xen.h>
/*
* We need to be able to catch inadvertent references to memstart_addr
if (swiotlb_force == SWIOTLB_FORCE ||
max_pfn > PFN_DOWN(arm64_dma_phys_limit))
swiotlb_init(1);
- else
+ else if (!xen_swiotlb_detect())
swiotlb_force = SWIOTLB_NO_FORCE;
set_max_mapnr(max_pfn - PHYS_PFN_OFFSET);
*/
BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end));
- if (rodata_full || crash_mem_map || debug_pagealloc_enabled())
+ if (rodata_full || crash_mem_map || debug_pagealloc_enabled() ||
+ IS_ENABLED(CONFIG_KFENCE))
flags |= NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS;
/*
mov x10, #(SYS_GCR_EL1_RRND | SYS_GCR_EL1_EXCL_MASK)
msr_s SYS_GCR_EL1, x10
+ /*
+ * If GCR_EL1.RRND=1 is implemented the same way as RRND=0, then
+ * RGSR_EL1.SEED must be non-zero for IRG to produce
+ * pseudorandom numbers. As RGSR_EL1 is UNKNOWN out of reset, we
+ * must initialize it.
+ */
+ mrs x10, CNTVCT_EL0
+ ands x10, x10, #SYS_RGSR_EL1_SEED_MASK
+ csinc x10, x10, xzr, ne
+ lsl x10, x10, #SYS_RGSR_EL1_SEED_SHIFT
+ msr_s SYS_RGSR_EL1, x10
+
/* clear any pending tag check faults in TFSR*_EL1 */
msr_s SYS_TFSR_EL1, xzr
msr_s SYS_TFSRE0_EL1, xzr
return !(imm & ~0xfff) || !(imm & ~0xfff000);
}
-/* Stack must be multiples of 16B */
-#define STACK_ALIGN(sz) (((sz) + 15) & ~15)
-
/* Tail call offset to jump into */
#if IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)
#define PROLOGUE_OFFSET 8
emit(A64_BTI_J, ctx);
}
- ctx->stack_size = STACK_ALIGN(prog->aux->stack_depth);
+ /* Stack must be multiples of 16B */
+ ctx->stack_size = round_up(prog->aux->stack_depth, 16);
/* Set up function call stack */
emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
break;
case BPF_ALU | BPF_DIV | BPF_X:
case BPF_ALU64 | BPF_DIV | BPF_X:
+ emit(A64_UDIV(is64, dst, dst, src), ctx);
+ break;
case BPF_ALU | BPF_MOD | BPF_X:
case BPF_ALU64 | BPF_MOD | BPF_X:
- switch (BPF_OP(code)) {
- case BPF_DIV:
- emit(A64_UDIV(is64, dst, dst, src), ctx);
- break;
- case BPF_MOD:
- emit(A64_UDIV(is64, tmp, dst, src), ctx);
- emit(A64_MSUB(is64, dst, dst, tmp, src), ctx);
- break;
- }
+ emit(A64_UDIV(is64, tmp, dst, src), ctx);
+ emit(A64_MSUB(is64, dst, dst, tmp, src), ctx);
break;
case BPF_ALU | BPF_LSH | BPF_X:
case BPF_ALU64 | BPF_LSH | BPF_X:
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0
+
+gen := arch/$(ARCH)/include/generated
+kapi := $(gen)/asm
+
+kapi-hdrs-y := $(kapi)/cpucaps.h
+
+targets += $(addprefix ../../../,$(gen-y) $(kapi-hdrs-y))
+
+PHONY += kapi
+
+kapi: $(kapi-hdrs-y) $(gen-y)
+
+# Create output directory if not already present
+_dummy := $(shell [ -d '$(kapi)' ] || mkdir -p '$(kapi)')
+
+quiet_cmd_gen_cpucaps = GEN $@
+ cmd_gen_cpucaps = mkdir -p $(dir $@) && \
+ $(AWK) -f $(filter-out $(PHONY),$^) > $@
+
+$(kapi)/cpucaps.h: $(src)/gen-cpucaps.awk $(src)/cpucaps FORCE
+ $(call if_changed,gen_cpucaps)
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0
+#
+# Internal CPU capabilities constants, keep this list sorted
+
+BTI
+HAS_32BIT_EL0
+HAS_32BIT_EL1
+HAS_ADDRESS_AUTH
+HAS_ADDRESS_AUTH_ARCH
+HAS_ADDRESS_AUTH_IMP_DEF
+HAS_AMU_EXTN
+HAS_ARMv8_4_TTL
+HAS_CACHE_DIC
+HAS_CACHE_IDC
+HAS_CNP
+HAS_CRC32
+HAS_DCPODP
+HAS_DCPOP
+HAS_E0PD
+HAS_EPAN
+HAS_GENERIC_AUTH
+HAS_GENERIC_AUTH_ARCH
+HAS_GENERIC_AUTH_IMP_DEF
+HAS_IRQ_PRIO_MASKING
+HAS_LDAPR
+HAS_LSE_ATOMICS
+HAS_NO_FPSIMD
+HAS_NO_HW_PREFETCH
+HAS_PAN
+HAS_RAS_EXTN
+HAS_RNG
+HAS_SB
+HAS_STAGE2_FWB
+HAS_SYSREG_GIC_CPUIF
+HAS_TLB_RANGE
+HAS_VIRT_HOST_EXTN
+HW_DBM
+KVM_PROTECTED_MODE
+MISMATCHED_CACHE_TYPE
+MTE
+SPECTRE_V2
+SPECTRE_V3A
+SPECTRE_V4
+SSBS
+SVE
+UNMAP_KERNEL_AT_EL0
+WORKAROUND_834220
+WORKAROUND_843419
+WORKAROUND_845719
+WORKAROUND_858921
+WORKAROUND_1418040
+WORKAROUND_1463225
+WORKAROUND_1508412
+WORKAROUND_1542419
+WORKAROUND_CAVIUM_23154
+WORKAROUND_CAVIUM_27456
+WORKAROUND_CAVIUM_30115
+WORKAROUND_CAVIUM_TX2_219_PRFM
+WORKAROUND_CAVIUM_TX2_219_TVM
+WORKAROUND_CLEAN_CACHE
+WORKAROUND_DEVICE_LOAD_ACQUIRE
+WORKAROUND_NVIDIA_CARMEL_CNP
+WORKAROUND_QCOM_FALKOR_E1003
+WORKAROUND_REPEAT_TLBI
+WORKAROUND_SPECULATIVE_AT
--- /dev/null
+#!/bin/awk -f
+# SPDX-License-Identifier: GPL-2.0
+# gen-cpucaps.awk: arm64 cpucaps header generator
+#
+# Usage: awk -f gen-cpucaps.awk cpucaps.txt
+
+# Log an error and terminate
+function fatal(msg) {
+ print "Error at line " NR ": " msg > "/dev/stderr"
+ exit 1
+}
+
+# skip blank lines and comment lines
+/^$/ { next }
+/^#/ { next }
+
+BEGIN {
+ print "#ifndef __ASM_CPUCAPS_H"
+ print "#define __ASM_CPUCAPS_H"
+ print ""
+ print "/* Generated file - do not edit */"
+ cap_num = 0
+ print ""
+}
+
+/^[vA-Z0-9_]+$/ {
+ printf("#define ARM64_%-30s\t%d\n", $0, cap_num++)
+ next
+}
+
+END {
+ printf("#define ARM64_NCAPS\t\t\t\t%d\n", cap_num)
+ print ""
+ print "#endif /* __ASM_CPUCAPS_H */"
+}
+
+# Any lines not handled by previous rules are unexpected
+{
+ fatal("unhandled statement")
+}
440 common process_madvise sys_process_madvise
441 common epoll_pwait2 sys_epoll_pwait2
442 common mount_setattr sys_mount_setattr
-443 common quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 common landlock_create_ruleset sys_landlock_create_ruleset
445 common landlock_add_rule sys_landlock_add_rule
446 common landlock_restrict_self sys_landlock_restrict_self
BUILD_BUG_ON(offsetof(siginfo_t, si_pkey) != 0x12);
/* _sigfault._perf */
- BUILD_BUG_ON(offsetof(siginfo_t, si_perf) != 0x10);
+ BUILD_BUG_ON(offsetof(siginfo_t, si_perf_data) != 0x10);
+ BUILD_BUG_ON(offsetof(siginfo_t, si_perf_type) != 0x14);
/* _sigpoll */
BUILD_BUG_ON(offsetof(siginfo_t, si_band) != 0x0c);
440 common process_madvise sys_process_madvise
441 common epoll_pwait2 sys_epoll_pwait2
442 common mount_setattr sys_mount_setattr
-443 common quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 common landlock_create_ruleset sys_landlock_create_ruleset
445 common landlock_add_rule sys_landlock_add_rule
446 common landlock_restrict_self sys_landlock_restrict_self
440 common process_madvise sys_process_madvise
441 common epoll_pwait2 sys_epoll_pwait2
442 common mount_setattr sys_mount_setattr
-443 common quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 common landlock_create_ruleset sys_landlock_create_ruleset
445 common landlock_add_rule sys_landlock_add_rule
446 common landlock_restrict_self sys_landlock_restrict_self
#include <asm/reboot.h>
#include <asm/setup.h>
#include <asm/mach-au1x00/au1000.h>
+#include <asm/mach-au1x00/gpio-au1000.h>
#include <prom.h>
const char *get_system_type(void)
ranges = <0x01000000 0x0 0x00000000 0x0 0x18000000 0x0 0x00010000>,
<0x02000000 0x0 0x40000000 0x0 0x40000000 0x0 0x40000000>;
+ gmac@3,0 {
+ compatible = "pci0014,7a03.0",
+ "pci0014,7a03",
+ "pciclass0c0320",
+ "pciclass0c03",
+ "loongson, pci-gmac";
+
+ reg = <0x1800 0x0 0x0 0x0 0x0>;
+ interrupts = <12 IRQ_TYPE_LEVEL_LOW>,
+ <13 IRQ_TYPE_LEVEL_LOW>;
+ interrupt-names = "macirq", "eth_lpi";
+ interrupt-parent = <&liointc0>;
+ phy-mode = "rgmii";
+ mdio {
+ #address-cells = <1>;
+ #size-cells = <0>;
+ compatible = "snps,dwmac-mdio";
+ phy0: ethernet-phy@0 {
+ reg = <0>;
+ };
+ };
+ };
+
+ gmac@3,1 {
+ compatible = "pci0014,7a03.0",
+ "pci0014,7a03",
+ "pciclass0c0320",
+ "pciclass0c03",
+ "loongson, pci-gmac";
+
+ reg = <0x1900 0x0 0x0 0x0 0x0>;
+ interrupts = <14 IRQ_TYPE_LEVEL_LOW>,
+ <15 IRQ_TYPE_LEVEL_LOW>;
+ interrupt-names = "macirq", "eth_lpi";
+ interrupt-parent = <&liointc0>;
+ phy-mode = "rgmii";
+ mdio {
+ #address-cells = <1>;
+ #size-cells = <0>;
+ compatible = "snps,dwmac-mdio";
+ phy1: ethernet-phy@1 {
+ reg = <0>;
+ };
+ };
+ };
+
ehci@4,1 {
compatible = "pci0014,7a14.0",
"pci0014,7a14",
compatible = "pci0014,7a03.0",
"pci0014,7a03",
"pciclass020000",
- "pciclass0200";
+ "pciclass0200",
+ "loongson, pci-gmac";
reg = <0x1800 0x0 0x0 0x0 0x0>;
interrupts = <12 IRQ_TYPE_LEVEL_HIGH>,
compatible = "pci0014,7a03.0",
"pci0014,7a03",
"pciclass020000",
- "pciclass0200";
+ "pciclass0200",
+ "loongson, pci-gmac";
reg = <0x1900 0x0 0x0 0x0 0x0>;
interrupts = <14 IRQ_TYPE_LEVEL_HIGH>,
*
*/
+#ifndef _ASM_MIPS_BOARDS_LAUNCH_H
+#define _ASM_MIPS_BOARDS_LAUNCH_H
+
#ifndef _ASSEMBLER_
struct cpulaunch {
/* Polling period in count cycles for secondary CPU's */
#define LAUNCHPERIOD 10000
+
+#endif /* _ASM_MIPS_BOARDS_LAUNCH_H */
#define SO_PREFER_BUSY_POLL 69
#define SO_BUSY_POLL_BUDGET 70
+#define SO_NETNS_COOKIE 71
+
#if !defined(__KERNEL__)
#if __BITS_PER_LONG == 64
440 n32 process_madvise sys_process_madvise
441 n32 epoll_pwait2 compat_sys_epoll_pwait2
442 n32 mount_setattr sys_mount_setattr
-443 n32 quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 n32 landlock_create_ruleset sys_landlock_create_ruleset
445 n32 landlock_add_rule sys_landlock_add_rule
446 n32 landlock_restrict_self sys_landlock_restrict_self
440 n64 process_madvise sys_process_madvise
441 n64 epoll_pwait2 sys_epoll_pwait2
442 n64 mount_setattr sys_mount_setattr
-443 n64 quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 n64 landlock_create_ruleset sys_landlock_create_ruleset
445 n64 landlock_add_rule sys_landlock_add_rule
446 n64 landlock_restrict_self sys_landlock_restrict_self
440 o32 process_madvise sys_process_madvise
441 o32 epoll_pwait2 sys_epoll_pwait2 compat_sys_epoll_pwait2
442 o32 mount_setattr sys_mount_setattr
-443 o32 quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 o32 landlock_create_ruleset sys_landlock_create_ruleset
445 o32 landlock_add_rule sys_landlock_add_rule
446 o32 landlock_restrict_self sys_landlock_restrict_self
*/
notrace void arch_local_irq_disable(void)
{
- preempt_disable();
+ preempt_disable_notrace();
__asm__ __volatile__(
" .set push \n"
: /* no inputs */
: "memory");
- preempt_enable();
+ preempt_enable_notrace();
}
EXPORT_SYMBOL(arch_local_irq_disable);
{
unsigned long flags;
- preempt_disable();
+ preempt_disable_notrace();
__asm__ __volatile__(
" .set push \n"
: /* no inputs */
: "memory");
- preempt_enable();
+ preempt_enable_notrace();
return flags;
}
{
unsigned long __tmp1;
- preempt_disable();
+ preempt_disable_notrace();
__asm__ __volatile__(
" .set push \n"
: "0" (flags)
: "memory");
- preempt_enable();
+ preempt_enable_notrace();
}
EXPORT_SYMBOL(arch_local_irq_restore);
EXPORT_SYMBOL(_page_cachable_default);
#define PM(p) __pgprot(_page_cachable_default | (p))
-#define PVA(p) PM(_PAGE_VALID | _PAGE_ACCESSED | (p))
static inline void setup_protection_map(void)
{
protection_map[0] = PM(_PAGE_PRESENT | _PAGE_NO_EXEC | _PAGE_NO_READ);
- protection_map[1] = PVA(_PAGE_PRESENT | _PAGE_NO_EXEC);
- protection_map[2] = PVA(_PAGE_PRESENT | _PAGE_NO_EXEC | _PAGE_NO_READ);
- protection_map[3] = PVA(_PAGE_PRESENT | _PAGE_NO_EXEC);
- protection_map[4] = PVA(_PAGE_PRESENT);
- protection_map[5] = PVA(_PAGE_PRESENT);
- protection_map[6] = PVA(_PAGE_PRESENT);
- protection_map[7] = PVA(_PAGE_PRESENT);
+ protection_map[1] = PM(_PAGE_PRESENT | _PAGE_NO_EXEC);
+ protection_map[2] = PM(_PAGE_PRESENT | _PAGE_NO_EXEC | _PAGE_NO_READ);
+ protection_map[3] = PM(_PAGE_PRESENT | _PAGE_NO_EXEC);
+ protection_map[4] = PM(_PAGE_PRESENT);
+ protection_map[5] = PM(_PAGE_PRESENT);
+ protection_map[6] = PM(_PAGE_PRESENT);
+ protection_map[7] = PM(_PAGE_PRESENT);
protection_map[8] = PM(_PAGE_PRESENT | _PAGE_NO_EXEC | _PAGE_NO_READ);
- protection_map[9] = PVA(_PAGE_PRESENT | _PAGE_NO_EXEC);
- protection_map[10] = PVA(_PAGE_PRESENT | _PAGE_NO_EXEC | _PAGE_WRITE |
+ protection_map[9] = PM(_PAGE_PRESENT | _PAGE_NO_EXEC);
+ protection_map[10] = PM(_PAGE_PRESENT | _PAGE_NO_EXEC | _PAGE_WRITE |
_PAGE_NO_READ);
- protection_map[11] = PVA(_PAGE_PRESENT | _PAGE_NO_EXEC | _PAGE_WRITE);
- protection_map[12] = PVA(_PAGE_PRESENT);
- protection_map[13] = PVA(_PAGE_PRESENT);
- protection_map[14] = PVA(_PAGE_PRESENT);
- protection_map[15] = PVA(_PAGE_PRESENT);
+ protection_map[11] = PM(_PAGE_PRESENT | _PAGE_NO_EXEC | _PAGE_WRITE);
+ protection_map[12] = PM(_PAGE_PRESENT);
+ protection_map[13] = PM(_PAGE_PRESENT);
+ protection_map[14] = PM(_PAGE_PRESENT | _PAGE_WRITE);
+ protection_map[15] = PM(_PAGE_PRESENT | _PAGE_WRITE);
}
-#undef _PVA
#undef PM
void cpu_cache_init(void)
#include <linux/io.h>
#include <linux/clk.h>
+#include <linux/export.h>
#include <linux/init.h>
#include <linux/sizes.h>
#include <linux/of_fdt.h>
__iomem void *rt_sysc_membase;
__iomem void *rt_memc_membase;
+EXPORT_SYMBOL_GPL(rt_sysc_membase);
__iomem void *plat_of_remap_node(const char *node)
{
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_BARRIER_H
+#define __ASM_BARRIER_H
+
+#define mb() asm volatile ("l.msync" ::: "memory")
+
+#include <asm-generic/barrier.h>
+
+#endif /* __ASM_BARRIER_H */
pr_cont("%lu.%02lu BogoMIPS (lpj=%lu)\n",
loops_per_jiffy / (500000 / HZ),
(loops_per_jiffy / (5000 / HZ)) % 100, loops_per_jiffy);
+
+ of_node_put(cpu);
}
void __init setup_arch(char **cmdline_p)
/* These mark extents of read-only kernel pages...
* ...from vmlinux.lds.S
*/
- struct memblock_region *region;
v = PAGE_OFFSET;
}
printk(KERN_INFO "%s: Memory: 0x%x-0x%x\n", __func__,
- region->base, region->base + region->size);
+ start, end);
}
}
{
extern void tlb_init(void);
- unsigned long end;
int i;
printk(KERN_INFO "Setting up paging and PTEs.\n");
*/
current_pgd[smp_processor_id()] = init_mm.pgd;
- end = (unsigned long)__va(max_low_pfn * PAGE_SIZE);
-
map_ram();
zone_sizes_init();
#define SO_PREFER_BUSY_POLL 0x4043
#define SO_BUSY_POLL_BUDGET 0x4044
+#define SO_NETNS_COOKIE 0x4045
+
#if !defined(__KERNEL__)
#if __BITS_PER_LONG == 64
440 common process_madvise sys_process_madvise
441 common epoll_pwait2 sys_epoll_pwait2 compat_sys_epoll_pwait2
442 common mount_setattr sys_mount_setattr
-443 common quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 common landlock_create_ruleset sys_landlock_create_ruleset
445 common landlock_add_rule sys_landlock_add_rule
446 common landlock_restrict_self sys_landlock_restrict_self
};
/include/ "pq3-i2c-0.dtsi"
+ i2c@3000 {
+ fsl,i2c-erratum-a004447;
+ };
+
/include/ "pq3-i2c-1.dtsi"
+ i2c@3100 {
+ fsl,i2c-erratum-a004447;
+ };
+
/include/ "pq3-duart-0.dtsi"
/include/ "pq3-espi-0.dtsi"
spi0: spi@7000 {
};
/include/ "qoriq-i2c-0.dtsi"
+ i2c@118000 {
+ fsl,i2c-erratum-a004447;
+ };
+
+ i2c@118100 {
+ fsl,i2c-erratum-a004447;
+ };
+
/include/ "qoriq-i2c-1.dtsi"
+ i2c@119000 {
+ fsl,i2c-erratum-a004447;
+ };
+
+ i2c@119100 {
+ fsl,i2c-erratum-a004447;
+ };
+
/include/ "qoriq-duart-0.dtsi"
/include/ "qoriq-duart-1.dtsi"
/include/ "qoriq-gpio-0.dtsi"
*/
long plpar_hcall_norets(unsigned long opcode, ...);
+/* Variant which does not do hcall tracing */
+long plpar_hcall_norets_notrace(unsigned long opcode, ...);
+
/**
* plpar_hcall: - Make a pseries hypervisor call
* @opcode: The hypervisor call to make.
*/
static inline void interrupt_exit_prepare(struct pt_regs *regs, struct interrupt_state *state)
{
- if (user_mode(regs))
- kuep_unlock();
}
static inline void interrupt_async_enter_prepare(struct pt_regs *regs, struct interrupt_state *state)
local_paca->irq_soft_mask = IRQS_ALL_DISABLED;
local_paca->irq_happened |= PACA_IRQ_HARD_DIS;
+ if (IS_ENABLED(CONFIG_PPC_BOOK3S_64) && !(regs->msr & MSR_PR) &&
+ regs->nip < (unsigned long)__end_interrupts) {
+ // Kernel code running below __end_interrupts is
+ // implicitly soft-masked.
+ regs->softe = IRQS_ALL_DISABLED;
+ }
+
/* Don't do any per-CPU operations until interrupt state is fixed */
if (nmi_disables_ftrace(regs)) {
/* PPC-specific vcpu->requests bit members */
#define KVM_REQ_WATCHDOG KVM_ARCH_REQ(0)
#define KVM_REQ_EPR_EXIT KVM_ARCH_REQ(1)
+#define KVM_REQ_PENDING_TIMER KVM_ARCH_REQ(2)
#include <linux/mmu_notifier.h>
return be32_to_cpu(yield_count);
}
+/*
+ * Spinlock code confers and prods, so don't trace the hcalls because the
+ * tracing code takes spinlocks which can cause recursion deadlocks.
+ *
+ * These calls are made while the lock is not held: the lock slowpath yields if
+ * it can not acquire the lock, and unlock slow path might prod if a waiter has
+ * yielded). So this may not be a problem for simple spin locks because the
+ * tracing does not technically recurse on the lock, but we avoid it anyway.
+ *
+ * However the queued spin lock contended path is more strictly ordered: the
+ * H_CONFER hcall is made after the task has queued itself on the lock, so then
+ * recursing on that lock will cause the task to then queue up again behind the
+ * first instance (or worse: queued spinlocks use tricks that assume a context
+ * never waits on more than one spinlock, so such recursion may cause random
+ * corruption in the lock code).
+ */
static inline void yield_to_preempted(int cpu, u32 yield_count)
{
- plpar_hcall_norets(H_CONFER, get_hard_smp_processor_id(cpu), yield_count);
+ plpar_hcall_norets_notrace(H_CONFER, get_hard_smp_processor_id(cpu), yield_count);
}
static inline void prod_cpu(int cpu)
{
- plpar_hcall_norets(H_PROD, get_hard_smp_processor_id(cpu));
+ plpar_hcall_norets_notrace(H_PROD, get_hard_smp_processor_id(cpu));
}
static inline void yield_to_any(void)
{
- plpar_hcall_norets(H_CONFER, -1, 0);
+ plpar_hcall_norets_notrace(H_CONFER, -1, 0);
}
#else
static inline bool is_shared_processor(void)
static inline long cede_processor(void)
{
- return plpar_hcall_norets(H_CEDE);
+ /*
+ * We cannot call tracepoints inside RCU idle regions which
+ * means we must not trace H_CEDE.
+ */
+ return plpar_hcall_norets_notrace(H_CEDE);
}
static inline long extended_cede_processor(unsigned long latency_hint)
pgd_t *pgdir = init_mm.pgd;
return __find_linux_pte(pgdir, ea, NULL, hshift);
}
+
+/*
+ * Convert a kernel vmap virtual address (vmalloc or ioremap space) to a
+ * physical address, without taking locks. This can be used in real-mode.
+ */
+static inline phys_addr_t ppc_find_vmap_phys(unsigned long addr)
+{
+ pte_t *ptep;
+ phys_addr_t pa;
+ int hugepage_shift;
+
+ /*
+ * init_mm does not free page tables, and does not do THP. It may
+ * have huge pages from huge vmalloc / ioremap etc.
+ */
+ ptep = find_init_mm_pte(addr, &hugepage_shift);
+ if (WARN_ON(!ptep))
+ return 0;
+
+ pa = PFN_PHYS(pte_pfn(*ptep));
+
+ if (!hugepage_shift)
+ hugepage_shift = PAGE_SHIFT;
+
+ pa |= addr & ((1ul << hugepage_shift) - 1);
+
+ return pa;
+}
+
/*
* This is what we should always use. Any other lockless page table lookup needs
* careful audit against THP split.
#ifndef _ASM_POWERPC_PTRACE_H
#define _ASM_POWERPC_PTRACE_H
+#include <linux/err.h>
#include <uapi/asm/ptrace.h>
#include <asm/asm-const.h>
long do_syscall_trace_enter(struct pt_regs *regs);
void do_syscall_trace_leave(struct pt_regs *regs);
-#define kernel_stack_pointer(regs) ((regs)->gpr[1])
-static inline int is_syscall_success(struct pt_regs *regs)
-{
- return !(regs->ccr & 0x10000000);
-}
-
-static inline long regs_return_value(struct pt_regs *regs)
-{
- if (is_syscall_success(regs))
- return regs->gpr[3];
- else
- return -regs->gpr[3];
-}
-
-static inline void regs_set_return_value(struct pt_regs *regs, unsigned long rc)
-{
- regs->gpr[3] = rc;
-}
-
#ifdef __powerpc64__
#define user_mode(regs) ((((regs)->msr) >> MSR_PR_LG) & 0x1)
#else
regs->trap |= 0x1;
}
+#define kernel_stack_pointer(regs) ((regs)->gpr[1])
+static inline int is_syscall_success(struct pt_regs *regs)
+{
+ if (trap_is_scv(regs))
+ return !IS_ERR_VALUE((unsigned long)regs->gpr[3]);
+ else
+ return !(regs->ccr & 0x10000000);
+}
+
+static inline long regs_return_value(struct pt_regs *regs)
+{
+ if (trap_is_scv(regs))
+ return regs->gpr[3];
+
+ if (is_syscall_success(regs))
+ return regs->gpr[3];
+ else
+ return -regs->gpr[3];
+}
+
+static inline void regs_set_return_value(struct pt_regs *regs, unsigned long rc)
+{
+ regs->gpr[3] = rc;
+}
+
#define arch_has_single_step() (1)
#define arch_has_block_step() (true)
#define ARCH_HAS_USER_SINGLE_STEP_REPORT
static inline long syscall_get_error(struct task_struct *task,
struct pt_regs *regs)
{
- /*
- * If the system call failed,
- * regs->gpr[3] contains a positive ERRORCODE.
- */
- return (regs->ccr & 0x10000000UL) ? -regs->gpr[3] : 0;
+ if (trap_is_scv(regs)) {
+ unsigned long error = regs->gpr[3];
+
+ return IS_ERR_VALUE(error) ? error : 0;
+ } else {
+ /*
+ * If the system call failed,
+ * regs->gpr[3] contains a positive ERRORCODE.
+ */
+ return (regs->ccr & 0x10000000UL) ? -regs->gpr[3] : 0;
+ }
}
static inline long syscall_get_return_value(struct task_struct *task,
struct pt_regs *regs,
int error, long val)
{
- /*
- * In the general case it's not obvious that we must deal with CCR
- * here, as the syscall exit path will also do that for us. However
- * there are some places, eg. the signal code, which check ccr to
- * decide if the value in r3 is actually an error.
- */
- if (error) {
- regs->ccr |= 0x10000000L;
- regs->gpr[3] = error;
+ if (trap_is_scv(regs)) {
+ regs->gpr[3] = (long) error ?: val;
} else {
- regs->ccr &= ~0x10000000L;
- regs->gpr[3] = val;
+ /*
+ * In the general case it's not obvious that we must deal with
+ * CCR here, as the syscall exit path will also do that for us.
+ * However there are some places, eg. the signal code, which
+ * check ccr to decide if the value in r3 is actually an error.
+ */
+ if (error) {
+ regs->ccr |= 0x10000000L;
+ regs->gpr[3] = error;
+ } else {
+ regs->ccr &= ~0x10000000L;
+ regs->gpr[3] = val;
+ }
}
}
"2: lwz%X1 %L0, %L1\n" \
EX_TABLE(1b, %l2) \
EX_TABLE(2b, %l2) \
- : "=r" (x) \
+ : "=&r" (x) \
: "m" (*addr) \
: \
: label)
*/
static inline unsigned long eeh_token_to_phys(unsigned long token)
{
- pte_t *ptep;
- unsigned long pa;
- int hugepage_shift;
-
- /*
- * We won't find hugepages here(this is iomem). Hence we are not
- * worried about _PAGE_SPLITTING/collapse. Also we will not hit
- * page table free, because of init_mm.
- */
- ptep = find_init_mm_pte(token, &hugepage_shift);
- if (!ptep)
- return token;
-
- pa = pte_pfn(*ptep);
-
- /* On radix we can do hugepage mappings for io, so handle that */
- if (!hugepage_shift)
- hugepage_shift = PAGE_SHIFT;
-
- pa <<= PAGE_SHIFT;
- pa |= token & ((1ul << hugepage_shift) - 1);
- return pa;
+ return ppc_find_vmap_phys(token);
}
/*
andi. r10,r10,IRQS_DISABLED; /* yes -> go out of line */ \
bne masked_interrupt_book3e_##n
+/*
+ * Additional regs must be re-loaded from paca before EXCEPTION_COMMON* is
+ * called, because that does SAVE_NVGPRS which must see the original register
+ * values, otherwise the scratch values might be restored when exiting the
+ * interrupt.
+ */
#define PROLOG_ADDITION_2REGS_GEN(n) \
std r14,PACA_EXGEN+EX_R14(r13); \
std r15,PACA_EXGEN+EX_R15(r13)
PROLOG_ADDITION_2REGS)
mfspr r14,SPRN_DEAR
mfspr r15,SPRN_ESR
+ std r14,_DAR(r1)
+ std r15,_DSISR(r1)
+ ld r14,PACA_EXGEN+EX_R14(r13)
+ ld r15,PACA_EXGEN+EX_R15(r13)
EXCEPTION_COMMON(0x300)
b storage_fault_common
PROLOG_ADDITION_2REGS)
li r15,0
mr r14,r10
+ std r14,_DAR(r1)
+ std r15,_DSISR(r1)
+ ld r14,PACA_EXGEN+EX_R14(r13)
+ ld r15,PACA_EXGEN+EX_R15(r13)
EXCEPTION_COMMON(0x400)
b storage_fault_common
PROLOG_ADDITION_2REGS)
mfspr r14,SPRN_DEAR
mfspr r15,SPRN_ESR
+ std r14,_DAR(r1)
+ std r15,_DSISR(r1)
+ ld r14,PACA_EXGEN+EX_R14(r13)
+ ld r15,PACA_EXGEN+EX_R15(r13)
EXCEPTION_COMMON(0x600)
b alignment_more /* no room, go out of line */
NORMAL_EXCEPTION_PROLOG(0x700, BOOKE_INTERRUPT_PROGRAM,
PROLOG_ADDITION_1REG)
mfspr r14,SPRN_ESR
- EXCEPTION_COMMON(0x700)
std r14,_DSISR(r1)
- addi r3,r1,STACK_FRAME_OVERHEAD
ld r14,PACA_EXGEN+EX_R14(r13)
+ EXCEPTION_COMMON(0x700)
+ addi r3,r1,STACK_FRAME_OVERHEAD
bl program_check_exception
REST_NVGPRS(r1)
b interrupt_return
* normal exception
*/
mfspr r14,SPRN_DBSR
- EXCEPTION_COMMON_CRIT(0xd00)
std r14,_DSISR(r1)
- addi r3,r1,STACK_FRAME_OVERHEAD
ld r14,PACA_EXCRIT+EX_R14(r13)
ld r15,PACA_EXCRIT+EX_R15(r13)
+ EXCEPTION_COMMON_CRIT(0xd00)
+ addi r3,r1,STACK_FRAME_OVERHEAD
bl DebugException
REST_NVGPRS(r1)
b interrupt_return
* normal exception
*/
mfspr r14,SPRN_DBSR
- EXCEPTION_COMMON_DBG(0xd08)
std r14,_DSISR(r1)
- addi r3,r1,STACK_FRAME_OVERHEAD
ld r14,PACA_EXDBG+EX_R14(r13)
ld r15,PACA_EXDBG+EX_R15(r13)
+ EXCEPTION_COMMON_DBG(0xd08)
+ addi r3,r1,STACK_FRAME_OVERHEAD
bl DebugException
REST_NVGPRS(r1)
b interrupt_return
* original values stashed away in the PACA
*/
storage_fault_common:
- std r14,_DAR(r1)
- std r15,_DSISR(r1)
addi r3,r1,STACK_FRAME_OVERHEAD
- ld r14,PACA_EXGEN+EX_R14(r13)
- ld r15,PACA_EXGEN+EX_R15(r13)
bl do_page_fault
b interrupt_return
* continues here.
*/
alignment_more:
- std r14,_DAR(r1)
- std r15,_DSISR(r1)
addi r3,r1,STACK_FRAME_OVERHEAD
- ld r14,PACA_EXGEN+EX_R14(r13)
- ld r15,PACA_EXGEN+EX_R15(r13)
bl alignment_exception
REST_NVGPRS(r1)
b interrupt_return
syscall_fn f;
kuep_lock();
-#ifdef CONFIG_PPC32
- kuap_save_and_lock(regs);
-#endif
regs->orig_gpr3 = r3;
/* Restore user access locks last */
kuap_user_restore(regs);
+ kuep_unlock();
return ret;
}
#ifdef CONFIG_PPC_INDIRECT_MMIO
struct iowa_bus *iowa_mem_find_bus(const PCI_IO_ADDR addr)
{
- unsigned hugepage_shift;
struct iowa_bus *bus;
int token;
bus = &iowa_busses[token - 1];
else {
unsigned long vaddr, paddr;
- pte_t *ptep;
vaddr = (unsigned long)PCI_FIX_ADDR(addr);
if (vaddr < PHB_IO_BASE || vaddr >= PHB_IO_END)
return NULL;
- /*
- * We won't find huge pages here (iomem). Also can't hit
- * a page table free due to init_mm
- */
- ptep = find_init_mm_pte(vaddr, &hugepage_shift);
- if (ptep == NULL)
- paddr = 0;
- else {
- WARN_ON(hugepage_shift);
- paddr = pte_pfn(*ptep) << PAGE_SHIFT;
- }
+
+ paddr = ppc_find_vmap_phys(vaddr);
+
bus = iowa_pci_find(vaddr, paddr);
if (bus == NULL)
unsigned int order;
unsigned int nio_pages, io_order;
struct page *page;
- size_t size_io = size;
size = PAGE_ALIGN(size);
order = get_order(size);
memset(ret, 0, size);
/* Set up tces to cover the allocated range */
- size_io = IOMMU_PAGE_ALIGN(size_io, tbl);
- nio_pages = size_io >> tbl->it_page_shift;
- io_order = get_iommu_order(size_io, tbl);
+ nio_pages = size >> tbl->it_page_shift;
+ io_order = get_iommu_order(size, tbl);
mapping = iommu_alloc(dev, tbl, ret, nio_pages, DMA_BIDIRECTIONAL,
mask >> tbl->it_page_shift, io_order, 0);
if (mapping == DMA_MAPPING_ERROR) {
void *vaddr, dma_addr_t dma_handle)
{
if (tbl) {
- size_t size_io = IOMMU_PAGE_ALIGN(size, tbl);
- unsigned int nio_pages = size_io >> tbl->it_page_shift;
+ unsigned int nio_pages;
+ size = PAGE_ALIGN(size);
+ nio_pages = size >> tbl->it_page_shift;
iommu_free(tbl, dma_handle, nio_pages);
size = PAGE_ALIGN(size);
free_pages((unsigned long)vaddr, get_order(size));
int ret = 0;
struct kprobe *prev;
struct ppc_inst insn = ppc_inst_read((struct ppc_inst *)p->addr);
- struct ppc_inst prefix = ppc_inst_read((struct ppc_inst *)(p->addr - 1));
if ((unsigned long)p->addr & 0x03) {
printk("Attempt to register kprobe at an unaligned address\n");
} else if (IS_MTMSRD(insn) || IS_RFID(insn) || IS_RFI(insn)) {
printk("Cannot register a kprobe on rfi/rfid or mtmsr[d]\n");
ret = -EINVAL;
- } else if (ppc_inst_prefixed(prefix)) {
+ } else if ((unsigned long)p->addr & ~PAGE_MASK &&
+ ppc_inst_prefixed(ppc_inst_read((struct ppc_inst *)(p->addr - 1)))) {
printk("Cannot register a kprobe on the second word of prefixed instruction\n");
ret = -EINVAL;
}
static int __init ioremap_legacy_serial_console(void)
{
- struct legacy_serial_info *info = &legacy_serial_infos[legacy_serial_console];
- struct plat_serial8250_port *port = &legacy_serial_ports[legacy_serial_console];
+ struct plat_serial8250_port *port;
+ struct legacy_serial_info *info;
void __iomem *vaddr;
if (legacy_serial_console < 0)
return 0;
+ info = &legacy_serial_infos[legacy_serial_console];
+ port = &legacy_serial_ports[legacy_serial_console];
+
if (!info->early_addr)
return 0;
apply_feature_fixups();
setup_feature_keys();
- early_ioremap_setup();
-
/* Initialize the hash table or TLB handling */
early_init_mmu();
+ early_ioremap_setup();
+
/*
* After firmware and early platform setup code has set things up,
* we note the SPR values for configurable control/performance
}
#endif /* CONFIG_PPC_TRANSACTIONAL_MEM */
#else
-#define unsafe_copy_fpr_to_user(to, task, label) do { } while (0)
+#define unsafe_copy_fpr_to_user(to, task, label) do { if (0) goto label;} while (0)
-#define unsafe_copy_fpr_from_user(task, from, label) do { } while (0)
+#define unsafe_copy_fpr_from_user(task, from, label) do { if (0) goto label;} while (0)
static inline unsigned long
copy_fpr_to_user(void __user *to, struct task_struct *task)
440 common process_madvise sys_process_madvise
441 common epoll_pwait2 sys_epoll_pwait2 compat_sys_epoll_pwait2
442 common mount_setattr sys_mount_setattr
-443 common quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 common landlock_create_ruleset sys_landlock_create_ruleset
445 common landlock_add_rule sys_landlock_add_rule
446 common landlock_restrict_self sys_landlock_restrict_self
kvm_unmap_radix(kvm, range->slot, gfn);
} else {
for (gfn = range->start; gfn < range->end; gfn++)
- kvm_unmap_rmapp(kvm, range->slot, range->start);
+ kvm_unmap_rmapp(kvm, range->slot, gfn);
}
return false;
break;
}
cur = ktime_get();
- } while (single_task_running() && ktime_before(cur, stop));
+ } while (kvm_vcpu_can_poll(cur, stop));
spin_lock(&vc->lock);
vc->vcore_state = VCORE_INACTIVE;
mtspr(SPRN_EBBRR, ebb_regs[1]);
mtspr(SPRN_BESCR, ebb_regs[2]);
mtspr(SPRN_TAR, user_tar);
- mtspr(SPRN_FSCR, current->thread.fscr);
}
mtspr(SPRN_VRSAVE, user_vrsave);
#include <asm/pte-walk.h>
/* Translate address of a vmalloc'd thing to a linear map address */
-static void *real_vmalloc_addr(void *x)
+static void *real_vmalloc_addr(void *addr)
{
- unsigned long addr = (unsigned long) x;
- pte_t *p;
- /*
- * assume we don't have huge pages in vmalloc space...
- * So don't worry about THP collapse/split. Called
- * Only in realmode with MSR_EE = 0, hence won't need irq_save/restore.
- */
- p = find_init_mm_pte(addr, NULL);
- if (!p || !pte_present(*p))
- return NULL;
- addr = (pte_pfn(*p) << PAGE_SHIFT) | (addr & ~PAGE_MASK);
- return __va(addr);
+ return __va(ppc_find_vmap_phys((unsigned long)addr));
}
/* Return 1 if we need to do a global tlbie, 0 if we can use tlbiel */
#define STACK_SLOT_UAMOR (SFS-88)
#define STACK_SLOT_DAWR1 (SFS-96)
#define STACK_SLOT_DAWRX1 (SFS-104)
+#define STACK_SLOT_FSCR (SFS-112)
/* the following is used by the P9 short path */
#define STACK_SLOT_NVGPRS (SFS-152) /* 18 gprs */
std r6, STACK_SLOT_DAWR0(r1)
std r7, STACK_SLOT_DAWRX0(r1)
std r8, STACK_SLOT_IAMR(r1)
+ mfspr r5, SPRN_FSCR
+ std r5, STACK_SLOT_FSCR(r1)
END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S)
BEGIN_FTR_SECTION
mfspr r6, SPRN_DAWR1
ld r7, STACK_SLOT_HFSCR(r1)
mtspr SPRN_HFSCR, r7
ALT_FTR_SECTION_END_IFCLR(CPU_FTR_ARCH_300)
+BEGIN_FTR_SECTION
+ ld r5, STACK_SLOT_FSCR(r1)
+ mtspr SPRN_FSCR, r5
+END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S)
/*
* Restore various registers to 0, where non-zero values
* set by the guest could disrupt the host.
#include <linux/string.h>
#include <linux/init.h>
#include <linux/sched/mm.h>
+#include <linux/stop_machine.h>
#include <asm/cputable.h>
#include <asm/code-patching.h>
#include <asm/page.h>
pr_devel("patching dest %lx\n", (unsigned long)dest);
- patch_instruction((struct ppc_inst *)dest, ppc_inst(instrs[0]));
-
- if (types & STF_BARRIER_FALLBACK)
+ // See comment in do_entry_flush_fixups() RE order of patching
+ if (types & STF_BARRIER_FALLBACK) {
+ patch_instruction((struct ppc_inst *)dest, ppc_inst(instrs[0]));
+ patch_instruction((struct ppc_inst *)(dest + 2), ppc_inst(instrs[2]));
patch_branch((struct ppc_inst *)(dest + 1),
- (unsigned long)&stf_barrier_fallback,
- BRANCH_SET_LINK);
- else
- patch_instruction((struct ppc_inst *)(dest + 1),
- ppc_inst(instrs[1]));
-
- patch_instruction((struct ppc_inst *)(dest + 2), ppc_inst(instrs[2]));
+ (unsigned long)&stf_barrier_fallback, BRANCH_SET_LINK);
+ } else {
+ patch_instruction((struct ppc_inst *)(dest + 1), ppc_inst(instrs[1]));
+ patch_instruction((struct ppc_inst *)(dest + 2), ppc_inst(instrs[2]));
+ patch_instruction((struct ppc_inst *)dest, ppc_inst(instrs[0]));
+ }
}
printk(KERN_DEBUG "stf-barrier: patched %d entry locations (%s barrier)\n", i,
: "unknown");
}
+static int __do_stf_barrier_fixups(void *data)
+{
+ enum stf_barrier_type *types = data;
+
+ do_stf_entry_barrier_fixups(*types);
+ do_stf_exit_barrier_fixups(*types);
+
+ return 0;
+}
void do_stf_barrier_fixups(enum stf_barrier_type types)
{
- do_stf_entry_barrier_fixups(types);
- do_stf_exit_barrier_fixups(types);
+ /*
+ * The call to the fallback entry flush, and the fallback/sync-ori exit
+ * flush can not be safely patched in/out while other CPUs are executing
+ * them. So call __do_stf_barrier_fixups() on one CPU while all other CPUs
+ * spin in the stop machine core with interrupts hard disabled.
+ */
+ stop_machine(__do_stf_barrier_fixups, &types, NULL);
}
void do_uaccess_flush_fixups(enum l1d_flush_type types)
: "unknown");
}
-void do_entry_flush_fixups(enum l1d_flush_type types)
+static int __do_entry_flush_fixups(void *data)
{
+ enum l1d_flush_type types = *(enum l1d_flush_type *)data;
unsigned int instrs[3], *dest;
long *start, *end;
int i;
if (types & L1D_FLUSH_MTTRIG)
instrs[i++] = 0x7c12dba6; /* mtspr TRIG2,r0 (SPR #882) */
+ /*
+ * If we're patching in or out the fallback flush we need to be careful about the
+ * order in which we patch instructions. That's because it's possible we could
+ * take a page fault after patching one instruction, so the sequence of
+ * instructions must be safe even in a half patched state.
+ *
+ * To make that work, when patching in the fallback flush we patch in this order:
+ * - the mflr (dest)
+ * - the mtlr (dest + 2)
+ * - the branch (dest + 1)
+ *
+ * That ensures the sequence is safe to execute at any point. In contrast if we
+ * patch the mtlr last, it's possible we could return from the branch and not
+ * restore LR, leading to a crash later.
+ *
+ * When patching out the fallback flush (either with nops or another flush type),
+ * we patch in this order:
+ * - the branch (dest + 1)
+ * - the mtlr (dest + 2)
+ * - the mflr (dest)
+ *
+ * Note we are protected by stop_machine() from other CPUs executing the code in a
+ * semi-patched state.
+ */
+
start = PTRRELOC(&__start___entry_flush_fixup);
end = PTRRELOC(&__stop___entry_flush_fixup);
for (i = 0; start < end; start++, i++) {
pr_devel("patching dest %lx\n", (unsigned long)dest);
- patch_instruction((struct ppc_inst *)dest, ppc_inst(instrs[0]));
-
- if (types == L1D_FLUSH_FALLBACK)
- patch_branch((struct ppc_inst *)(dest + 1), (unsigned long)&entry_flush_fallback,
- BRANCH_SET_LINK);
- else
+ if (types == L1D_FLUSH_FALLBACK) {
+ patch_instruction((struct ppc_inst *)dest, ppc_inst(instrs[0]));
+ patch_instruction((struct ppc_inst *)(dest + 2), ppc_inst(instrs[2]));
+ patch_branch((struct ppc_inst *)(dest + 1),
+ (unsigned long)&entry_flush_fallback, BRANCH_SET_LINK);
+ } else {
patch_instruction((struct ppc_inst *)(dest + 1), ppc_inst(instrs[1]));
-
- patch_instruction((struct ppc_inst *)(dest + 2), ppc_inst(instrs[2]));
+ patch_instruction((struct ppc_inst *)(dest + 2), ppc_inst(instrs[2]));
+ patch_instruction((struct ppc_inst *)dest, ppc_inst(instrs[0]));
+ }
}
start = PTRRELOC(&__start___scv_entry_flush_fixup);
pr_devel("patching dest %lx\n", (unsigned long)dest);
- patch_instruction((struct ppc_inst *)dest, ppc_inst(instrs[0]));
-
- if (types == L1D_FLUSH_FALLBACK)
- patch_branch((struct ppc_inst *)(dest + 1), (unsigned long)&scv_entry_flush_fallback,
- BRANCH_SET_LINK);
- else
+ if (types == L1D_FLUSH_FALLBACK) {
+ patch_instruction((struct ppc_inst *)dest, ppc_inst(instrs[0]));
+ patch_instruction((struct ppc_inst *)(dest + 2), ppc_inst(instrs[2]));
+ patch_branch((struct ppc_inst *)(dest + 1),
+ (unsigned long)&scv_entry_flush_fallback, BRANCH_SET_LINK);
+ } else {
patch_instruction((struct ppc_inst *)(dest + 1), ppc_inst(instrs[1]));
-
- patch_instruction((struct ppc_inst *)(dest + 2), ppc_inst(instrs[2]));
+ patch_instruction((struct ppc_inst *)(dest + 2), ppc_inst(instrs[2]));
+ patch_instruction((struct ppc_inst *)dest, ppc_inst(instrs[0]));
+ }
}
: "ori type" :
(types & L1D_FLUSH_MTTRIG) ? "mttrig type"
: "unknown");
+
+ return 0;
+}
+
+void do_entry_flush_fixups(enum l1d_flush_type types)
+{
+ /*
+ * The call to the fallback flush can not be safely patched in/out while
+ * other CPUs are executing it. So call __do_entry_flush_fixups() on one
+ * CPU while all other CPUs spin in the stop machine core with interrupts
+ * hard disabled.
+ */
+ stop_machine(__do_entry_flush_fixups, &types, NULL);
}
void do_rfi_flush_fixups(enum l1d_flush_type types)
#define HCALL_BRANCH(LABEL)
#endif
+_GLOBAL_TOC(plpar_hcall_norets_notrace)
+ HMT_MEDIUM
+
+ mfcr r0
+ stw r0,8(r1)
+ HVSC /* invoke the hypervisor */
+ lwz r0,8(r1)
+ mtcrf 0xff,r0
+ blr /* return r3 = status */
+
_GLOBAL_TOC(plpar_hcall_norets)
HMT_MEDIUM
#endif
/*
- * Since the tracing code might execute hcalls we need to guard against
- * recursion. One example of this are spinlocks calling H_YIELD on
- * shared processor partitions.
+ * Keep track of hcall tracing depth and prevent recursion. Warn if any is
+ * detected because it may indicate a problem. This will not catch all
+ * problems with tracing code making hcalls, because the tracing might have
+ * been invoked from a non-hcall, so the first hcall could recurse into it
+ * without warning here, but this better than nothing.
+ *
+ * Hcalls with specific problems being traced should use the _notrace
+ * plpar_hcall variants.
*/
static DEFINE_PER_CPU(unsigned int, hcall_trace_depth);
-void __trace_hcall_entry(unsigned long opcode, unsigned long *args)
+notrace void __trace_hcall_entry(unsigned long opcode, unsigned long *args)
{
unsigned long flags;
unsigned int *depth;
- /*
- * We cannot call tracepoints inside RCU idle regions which
- * means we must not trace H_CEDE.
- */
- if (opcode == H_CEDE)
- return;
-
local_irq_save(flags);
depth = this_cpu_ptr(&hcall_trace_depth);
- if (*depth)
+ if (WARN_ON_ONCE(*depth))
goto out;
(*depth)++;
local_irq_restore(flags);
}
-void __trace_hcall_exit(long opcode, long retval, unsigned long *retbuf)
+notrace void __trace_hcall_exit(long opcode, long retval, unsigned long *retbuf)
{
unsigned long flags;
unsigned int *depth;
- if (opcode == H_CEDE)
- return;
-
local_irq_save(flags);
depth = this_cpu_ptr(&hcall_trace_depth);
- if (*depth)
+ if (*depth) /* Don't warn again on the way out */
goto out;
(*depth)++;
select ARCH_OPTIONAL_KERNEL_RWX if ARCH_HAS_STRICT_KERNEL_RWX
select ARCH_OPTIONAL_KERNEL_RWX_DEFAULT
select ARCH_SUPPORTS_HUGETLBFS if MMU
+ select ARCH_USE_MEMTEST
select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU
select ARCH_WANT_FRAME_POINTERS
select ARCH_WANT_HUGE_PMD_SHARE if 64BIT
select GENERIC_TIME_VSYSCALL if MMU && 64BIT
select HANDLE_DOMAIN_IRQ
select HAVE_ARCH_AUDITSYSCALL
- select HAVE_ARCH_JUMP_LABEL
- select HAVE_ARCH_JUMP_LABEL_RELATIVE
+ select HAVE_ARCH_JUMP_LABEL if !XIP_KERNEL
+ select HAVE_ARCH_JUMP_LABEL_RELATIVE if !XIP_KERNEL
select HAVE_ARCH_KASAN if MMU && 64BIT
select HAVE_ARCH_KASAN_VMALLOC if MMU && 64BIT
- select HAVE_ARCH_KGDB
+ select HAVE_ARCH_KGDB if !XIP_KERNEL
select HAVE_ARCH_KGDB_QXFER_PKT
select HAVE_ARCH_MMAP_RND_BITS if MMU
select HAVE_ARCH_SECCOMP_FILTER
select HAVE_GCC_PLUGINS
select HAVE_GENERIC_VDSO if MMU && 64BIT
select HAVE_IRQ_TIME_ACCOUNTING
- select HAVE_KPROBES
- select HAVE_KPROBES_ON_FTRACE
- select HAVE_KRETPROBES
+ select HAVE_KPROBES if !XIP_KERNEL
+ select HAVE_KPROBES_ON_FTRACE if !XIP_KERNEL
+ select HAVE_KRETPROBES if !XIP_KERNEL
select HAVE_PCI
select HAVE_PERF_EVENTS
select HAVE_PERF_REGS
bool "RV64I"
select 64BIT
select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 && GCC_VERSION >= 50000
- select HAVE_DYNAMIC_FTRACE if MMU && $(cc-option,-fpatchable-function-entry=8)
+ select HAVE_DYNAMIC_FTRACE if !XIP_KERNEL && MMU && $(cc-option,-fpatchable-function-entry=8)
select HAVE_DYNAMIC_FTRACE_WITH_REGS if HAVE_DYNAMIC_FTRACE
- select HAVE_FTRACE_MCOUNT_RECORD
+ select HAVE_FTRACE_MCOUNT_RECORD if !XIP_KERNEL
select HAVE_FUNCTION_GRAPH_TRACER
- select HAVE_FUNCTION_TRACER
+ select HAVE_FUNCTION_TRACER if !XIP_KERNEL
select SWIOTLB if MMU
endchoice
KBUILD_LDFLAGS += -melf32lriscv
endif
+ifeq ($(CONFIG_LD_IS_LLD),y)
+ KBUILD_CFLAGS += -mno-relax
+ KBUILD_AFLAGS += -mno-relax
+ifneq ($(LLVM_IAS),1)
+ KBUILD_CFLAGS += -Wa,-mno-relax
+ KBUILD_AFLAGS += -Wa,-mno-relax
+endif
+endif
+
# ISA string setting
riscv-march-$(CONFIG_ARCH_RV32I) := rv32ima
riscv-march-$(CONFIG_ARCH_RV64I) := rv64ima
# SPDX-License-Identifier: GPL-2.0
dtb-$(CONFIG_SOC_MICROCHIP_POLARFIRE) += microchip-mpfs-icicle-kit.dtb
+obj-$(CONFIG_BUILTIN_DTB) += $(addsuffix .o, $(dtb-y))
# SPDX-License-Identifier: GPL-2.0
dtb-$(CONFIG_SOC_SIFIVE) += hifive-unleashed-a00.dtb \
hifive-unmatched-a00.dtb
+obj-$(CONFIG_BUILTIN_DTB) += $(addsuffix .o, $(dtb-y))
-obj-y += errata_cip_453.o
+obj-$(CONFIG_ERRATA_SIFIVE_CIP_453) += errata_cip_453.o
obj-y += errata.o
REG_ASM " " newlen "\n" \
".word " errata_id "\n"
-#define ALT_NEW_CONSTENT(vendor_id, errata_id, enable, new_c) \
+#define ALT_NEW_CONTENT(vendor_id, errata_id, enable, new_c) \
".if " __stringify(enable) " == 1\n" \
".pushsection .alternative, \"a\"\n" \
ALT_ENTRY("886b", "888f", __stringify(vendor_id), __stringify(errata_id), "889f - 888f") \
"886 :\n" \
old_c "\n" \
"887 :\n" \
- ALT_NEW_CONSTENT(vendor_id, errata_id, enable, new_c)
+ ALT_NEW_CONTENT(vendor_id, errata_id, enable, new_c)
#define _ALTERNATIVE_CFG(old_c, new_c, vendor_id, errata_id, CONFIG_k) \
__ALTERNATIVE_CFG(old_c, new_c, vendor_id, errata_id, IS_ENABLED(CONFIG_k))
unsigned long fdt_addr;
};
-const extern unsigned char riscv_kexec_relocate[];
-const extern unsigned int riscv_kexec_relocate_size;
+extern const unsigned char riscv_kexec_relocate[];
+extern const unsigned int riscv_kexec_relocate_size;
typedef void (*riscv_kexec_method)(unsigned long first_ind_entry,
unsigned long jump_addr,
#include <asm/set_memory.h> /* For set_memory_x() */
#include <linux/compiler.h> /* For unreachable() */
#include <linux/cpu.h> /* For cpu_down() */
+#include <linux/reboot.h>
-/**
+/*
* kexec_image_info - Print received image details
*/
static void
}
}
-/**
+/*
* machine_kexec_prepare - Initialize kexec
*
* This function is called from do_kexec_load, when the user has
}
-/**
+/*
* machine_kexec_cleanup - Cleanup any leftovers from
* machine_kexec_prepare
*
#endif
}
-/**
+/*
* machine_crash_shutdown - Prepare to kexec after a kernel crash
*
* This function is called by crash_kexec just before machine_kexec
pr_info("Starting crashdump kernel...\n");
}
-/**
+/*
* machine_kexec - Jump to the loaded kimage
*
* This function is called by kernel_kexec which is called by the
return 0;
}
+#ifdef CONFIG_MMU
void *alloc_insn_page(void)
{
return __vmalloc_node_range(PAGE_SIZE, 1, VMALLOC_START, VMALLOC_END,
VM_FLUSH_RESET_PERMS, NUMA_NO_NODE,
__builtin_return_address(0));
}
+#endif
/* install breakpoint in text */
void __kprobes arch_arm_kprobe(struct kprobe *p)
/* Clean-up any unused pre-allocated resources */
mem_res_sz = (num_resources - res_idx + 1) * sizeof(*mem_res);
- memblock_free((phys_addr_t) mem_res, mem_res_sz);
+ memblock_free(__pa(mem_res), mem_res_sz);
return;
error:
/* Better an empty resource tree than an inconsistent one */
release_child_resources(&iomem_resource);
- memblock_free((phys_addr_t) mem_res, mem_res_sz);
+ memblock_free(__pa(mem_res), mem_res_sz);
}
fp = frame_pointer(regs);
sp = user_stack_pointer(regs);
pc = instruction_pointer(regs);
- } else if (task == NULL || task == current) {
- fp = (unsigned long)__builtin_frame_address(0);
- sp = sp_in_global;
- pc = (unsigned long)walk_stackframe;
+ } else if (task == current) {
+ fp = (unsigned long)__builtin_frame_address(1);
+ sp = (unsigned long)__builtin_frame_address(0);
+ pc = (unsigned long)__builtin_return_address(0);
} else {
/* task blocked in __switch_to */
fp = task->thread.s[0];
return true;
}
-void dump_backtrace(struct pt_regs *regs, struct task_struct *task,
+noinline void dump_backtrace(struct pt_regs *regs, struct task_struct *task,
const char *loglvl)
{
- pr_cont("%sCall Trace:\n", loglvl);
walk_stackframe(task, regs, print_trace_address, (void *)loglvl);
}
void show_stack(struct task_struct *task, unsigned long *sp, const char *loglvl)
{
+ pr_cont("%sCall Trace:\n", loglvl);
dump_backtrace(NULL, task, loglvl);
}
#ifdef CONFIG_STACKTRACE
-void arch_stack_walk(stack_trace_consume_fn consume_entry, void *cookie,
+noinline void arch_stack_walk(stack_trace_consume_fn consume_entry, void *cookie,
struct task_struct *task, struct pt_regs *regs)
{
walk_stackframe(task, regs, consume_entry, cookie);
}
}
+#if defined (CONFIG_XIP_KERNEL) && defined (CONFIG_RISCV_ERRATA_ALTERNATIVE)
+#define __trap_section __section(".xip.traps")
+#else
+#define __trap_section
+#endif
#define DO_ERROR_INFO(name, signo, code, str) \
-asmlinkage __visible void name(struct pt_regs *regs) \
+asmlinkage __visible __trap_section void name(struct pt_regs *regs) \
{ \
do_trap_error(regs, signo, code, regs->epc, "Oops - " str); \
}
int handle_misaligned_load(struct pt_regs *regs);
int handle_misaligned_store(struct pt_regs *regs);
-asmlinkage void do_trap_load_misaligned(struct pt_regs *regs)
+asmlinkage void __trap_section do_trap_load_misaligned(struct pt_regs *regs)
{
if (!handle_misaligned_load(regs))
return;
"Oops - load address misaligned");
}
-asmlinkage void do_trap_store_misaligned(struct pt_regs *regs)
+asmlinkage void __trap_section do_trap_store_misaligned(struct pt_regs *regs)
{
if (!handle_misaligned_store(regs))
return;
return GET_INSN_LENGTH(insn);
}
-asmlinkage __visible void do_trap_break(struct pt_regs *regs)
+asmlinkage __visible __trap_section void do_trap_break(struct pt_regs *regs)
{
#ifdef CONFIG_KPROBES
if (kprobe_single_step_handler(regs))
}
PERCPU_SECTION(L1_CACHE_BYTES)
- . = ALIGN(PAGE_SIZE);
+ . = ALIGN(8);
+ .alternative : {
+ __alt_start = .;
+ *(.alternative)
+ __alt_end = .;
+ }
__init_end = .;
+ . = ALIGN(16);
+ .xip.traps : {
+ __xip_traps_start = .;
+ *(.xip.traps)
+ __xip_traps_end = .;
+ }
+
+ . = ALIGN(PAGE_SIZE);
.sdata : {
__global_pointer$ = . + 0x800;
*(.sdata*)
unsigned long init_data_start = (unsigned long)__init_data_begin;
unsigned long rodata_start = (unsigned long)__start_rodata;
unsigned long data_start = (unsigned long)_data;
- unsigned long max_low = (unsigned long)(__va(PFN_PHYS(max_low_pfn)));
+#if defined(CONFIG_64BIT) && defined(CONFIG_MMU)
+ unsigned long end_va = kernel_virt_addr + load_sz;
+#else
+ unsigned long end_va = (unsigned long)(__va(PFN_PHYS(max_low_pfn)));
+#endif
set_memory_ro(text_start, (init_text_start - text_start) >> PAGE_SHIFT);
set_memory_ro(init_text_start, (init_data_start - init_text_start) >> PAGE_SHIFT);
set_memory_nx(init_data_start, (rodata_start - init_data_start) >> PAGE_SHIFT);
/* rodata section is marked readonly in mark_rodata_ro */
set_memory_nx(rodata_start, (data_start - rodata_start) >> PAGE_SHIFT);
- set_memory_nx(data_start, (max_low - data_start) >> PAGE_SHIFT);
+ set_memory_nx(data_start, (end_va - data_start) >> PAGE_SHIFT);
}
void mark_rodata_ro(void)
* @user0: user defineable value
* @res4: reserved paramater
* @user1: user defineable value
- * @user2: user defineable value
*/
struct qaob {
u64 res0[6];
u16 dcount[QDIO_MAX_ELEMENTS_PER_BUFFER];
u64 user0;
u64 res4[2];
- u64 user1;
- u64 user2;
+ u8 user1[16];
} __attribute__ ((packed, aligned(256)));
/**
440 common process_madvise sys_process_madvise sys_process_madvise
441 common epoll_pwait2 sys_epoll_pwait2 compat_sys_epoll_pwait2
442 common mount_setattr sys_mount_setattr sys_mount_setattr
-443 common quotactl_path sys_quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 common landlock_create_ruleset sys_landlock_create_ruleset sys_landlock_create_ruleset
445 common landlock_add_rule sys_landlock_add_rule sys_landlock_add_rule
446 common landlock_restrict_self sys_landlock_restrict_self sys_landlock_restrict_self
440 common process_madvise sys_process_madvise
441 common epoll_pwait2 sys_epoll_pwait2
442 common mount_setattr sys_mount_setattr
-443 common quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 common landlock_create_ruleset sys_landlock_create_ruleset
445 common landlock_add_rule sys_landlock_add_rule
446 common landlock_restrict_self sys_landlock_restrict_self
BUILD_TRAP_HANDLER(nmi)
{
- unsigned int cpu = smp_processor_id();
TRAP_HANDLER_DECL;
arch_ftrace_nmi_enter();
#define SO_PREFER_BUSY_POLL 0x0048
#define SO_BUSY_POLL_BUDGET 0x0049
+#define SO_NETNS_COOKIE 0x0050
+
#if !defined(__KERNEL__)
440 common process_madvise sys_process_madvise
441 common epoll_pwait2 sys_epoll_pwait2 compat_sys_epoll_pwait2
442 common mount_setattr sys_mount_setattr
-443 common quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 common landlock_create_ruleset sys_landlock_create_ruleset
445 common landlock_add_rule sys_landlock_add_rule
446 common landlock_restrict_self sys_landlock_restrict_self
KBUILD_CFLAGS += $(call cc-option,-maccumulate-outgoing-args,)
endif
-ifdef CONFIG_LTO_CLANG
-KBUILD_LDFLAGS += -plugin-opt=-code-model=kernel \
- -plugin-opt=-stack-alignment=$(if $(CONFIG_X86_32),4,8)
-endif
-
# Workaround for a gcc prelease that unfortunately was shipped in a suse release
KBUILD_CFLAGS += -Wno-sign-compare
#
endif
endif
-KBUILD_LDFLAGS := -m elf_$(UTS_MACHINE)
+KBUILD_LDFLAGS += -m elf_$(UTS_MACHINE)
+
+ifdef CONFIG_LTO_CLANG
+ifeq ($(shell test $(CONFIG_LLD_VERSION) -lt 130000; echo $$?),0)
+KBUILD_LDFLAGS += -plugin-opt=-stack-alignment=$(if $(CONFIG_X86_32),4,8)
+endif
+endif
ifdef CONFIG_X86_NEED_RELOCS
LDFLAGS_vmlinux := --emit-relocs --discard-none
KBUILD_CFLAGS := -m$(BITS) -O2
KBUILD_CFLAGS += -fno-strict-aliasing -fPIE
+KBUILD_CFLAGS += -Wundef
KBUILD_CFLAGS += -DDISABLE_BRANCH_PROFILING
cflags-$(CONFIG_X86_32) := -march=i386
cflags-$(CONFIG_X86_64) := -mcmodel=small -mno-red-zone
KBUILD_CFLAGS += -include $(srctree)/include/linux/hidden.h
KBUILD_CFLAGS += $(CLANG_FLAGS)
-# sev-es.c indirectly inludes inat-table.h which is generated during
+# sev.c indirectly inludes inat-table.h which is generated during
# compilation and stored in $(objtree). Add the directory to the includes so
# that the compiler finds it even with out-of-tree builds (make O=/some/path).
-CFLAGS_sev-es.o += -I$(objtree)/arch/x86/lib/
+CFLAGS_sev.o += -I$(objtree)/arch/x86/lib/
KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__
GCOV_PROFILE := n
vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o
vmlinux-objs-y += $(obj)/mem_encrypt.o
vmlinux-objs-y += $(obj)/pgtable_64.o
- vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/sev-es.o
+ vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/sev.o
endif
vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o
}
}
-#if CONFIG_X86_NEED_RELOCS
+#ifdef CONFIG_X86_NEED_RELOCS
static void handle_relocations(void *output, unsigned long output_len,
unsigned long virt_addr)
{
u64 size;
};
-#if CONFIG_RANDOMIZE_BASE
+#ifdef CONFIG_RANDOMIZE_BASE
/* kaslr.c */
void choose_random_location(unsigned long input,
unsigned long input_size,
#include "misc.h"
#include <asm/pgtable_types.h>
-#include <asm/sev-es.h>
+#include <asm/sev.h>
#include <asm/trapnr.h>
#include <asm/trap_pf.h>
#include <asm/msr-index.h>
#include "../../lib/insn.c"
/* Include code for early handlers */
-#include "../../kernel/sev-es-shared.c"
+#include "../../kernel/sev-shared.c"
static bool early_setup_sev_es(void)
{
440 i386 process_madvise sys_process_madvise
441 i386 epoll_pwait2 sys_epoll_pwait2 compat_sys_epoll_pwait2
442 i386 mount_setattr sys_mount_setattr
-443 i386 quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 i386 landlock_create_ruleset sys_landlock_create_ruleset
445 i386 landlock_add_rule sys_landlock_add_rule
446 i386 landlock_restrict_self sys_landlock_restrict_self
440 common process_madvise sys_process_madvise
441 common epoll_pwait2 sys_epoll_pwait2
442 common mount_setattr sys_mount_setattr
-443 common quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 common landlock_create_ruleset sys_landlock_create_ruleset
445 common landlock_add_rule sys_landlock_add_rule
446 common landlock_restrict_self sys_landlock_restrict_self
#include "../perf_event.h"
#include "iommu.h"
-#define COUNTER_SHIFT 16
-
/* iommu pmu conf masks */
#define GET_CSOURCE(x) ((x)->conf & 0xFFULL)
#define GET_DEVID(x) (((x)->conf >> 8) & 0xFFFFULL)
WARN_ON_ONCE(!(hwc->state & PERF_HES_UPTODATE));
hwc->state = 0;
+ /*
+ * To account for power-gating, which prevents write to
+ * the counter, we need to enable the counter
+ * before setting up counter register.
+ */
+ perf_iommu_enable_event(event);
+
if (flags & PERF_EF_RELOAD) {
- u64 prev_raw_count = local64_read(&hwc->prev_count);
+ u64 count = 0;
struct amd_iommu *iommu = perf_event_2_iommu(event);
+ /*
+ * Since the IOMMU PMU only support counting mode,
+ * the counter always start with value zero.
+ */
amd_iommu_pc_set_reg(iommu, hwc->iommu_bank, hwc->iommu_cntr,
- IOMMU_PC_COUNTER_REG, &prev_raw_count);
+ IOMMU_PC_COUNTER_REG, &count);
}
- perf_iommu_enable_event(event);
perf_event_update_userpage(event);
-
}
static void perf_iommu_read(struct perf_event *event)
{
- u64 count, prev, delta;
+ u64 count;
struct hw_perf_event *hwc = &event->hw;
struct amd_iommu *iommu = perf_event_2_iommu(event);
/* IOMMU pc counter register is only 48 bits */
count &= GENMASK_ULL(47, 0);
- prev = local64_read(&hwc->prev_count);
- if (local64_cmpxchg(&hwc->prev_count, prev, count) != prev)
- return;
-
- /* Handle 48-bit counter overflow */
- delta = (count << COUNTER_SHIFT) - (prev << COUNTER_SHIFT);
- delta >>= COUNTER_SHIFT;
- local64_add(delta, &event->count);
+ /*
+ * Since the counter always start with value zero,
+ * simply just accumulate the count for the event.
+ */
+ local64_add(count, &event->count);
}
static void perf_iommu_stop(struct perf_event *event, int flags)
if (hwc->state & PERF_HES_UPTODATE)
return;
+ /*
+ * To account for power-gating, in which reading the counter would
+ * return zero, we need to read the register before disabling.
+ */
+ perf_iommu_read(event);
+ hwc->state |= PERF_HES_UPTODATE;
+
perf_iommu_disable_event(event);
WARN_ON_ONCE(hwc->state & PERF_HES_STOPPED);
hwc->state |= PERF_HES_STOPPED;
-
- if (hwc->state & PERF_HES_UPTODATE)
- return;
-
- perf_iommu_read(event);
- hwc->state |= PERF_HES_UPTODATE;
}
static int perf_iommu_add(struct perf_event *event, int flags)
if (!atomic_inc_not_zero(&pmc_refcount)) {
mutex_lock(&pmc_reserve_mutex);
if (atomic_read(&pmc_refcount) == 0) {
- if (!reserve_pmc_hardware())
+ if (!reserve_pmc_hardware()) {
err = -EBUSY;
- else
+ } else {
reserve_ds_buffers();
+ reserve_lbr_buffers();
+ }
}
if (!err)
atomic_inc(&pmc_refcount);
* Check all LBT MSR here.
* Disable LBR access if any LBR MSRs can not be accessed.
*/
- if (x86_pmu.lbr_nr && !check_msr(x86_pmu.lbr_tos, 0x3UL))
+ if (x86_pmu.lbr_tos && !check_msr(x86_pmu.lbr_tos, 0x3UL))
x86_pmu.lbr_nr = 0;
for (i = 0; i < x86_pmu.lbr_nr; i++) {
if (!(check_msr(x86_pmu.lbr_from + i, 0xffffUL) &&
void intel_pmu_lbr_add(struct perf_event *event)
{
- struct kmem_cache *kmem_cache = event->pmu->task_ctx_cache;
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!x86_pmu.lbr_nr)
perf_sched_cb_inc(event->ctx->pmu);
if (!cpuc->lbr_users++ && !event->total_time_running)
intel_pmu_lbr_reset();
-
- if (static_cpu_has(X86_FEATURE_ARCH_LBR) &&
- kmem_cache && !cpuc->lbr_xsave &&
- (cpuc->lbr_users != cpuc->lbr_pebs_users))
- cpuc->lbr_xsave = kmem_cache_alloc(kmem_cache, GFP_KERNEL);
}
void release_lbr_buffers(void)
}
}
+void reserve_lbr_buffers(void)
+{
+ struct kmem_cache *kmem_cache;
+ struct cpu_hw_events *cpuc;
+ int cpu;
+
+ if (!static_cpu_has(X86_FEATURE_ARCH_LBR))
+ return;
+
+ for_each_possible_cpu(cpu) {
+ cpuc = per_cpu_ptr(&cpu_hw_events, cpu);
+ kmem_cache = x86_get_pmu(cpu)->task_ctx_cache;
+ if (!kmem_cache || cpuc->lbr_xsave)
+ continue;
+
+ cpuc->lbr_xsave = kmem_cache_alloc_node(kmem_cache, GFP_KERNEL,
+ cpu_to_node(cpu));
+ }
+}
+
void intel_pmu_lbr_del(struct perf_event *event)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
die_id = i;
else
die_id = topology_phys_to_logical_pkg(i);
+ if (die_id < 0)
+ die_id = -ENODEV;
map->pbus_to_dieid[bus] = die_id;
break;
}
i = -1;
if (reverse) {
for (bus = 255; bus >= 0; bus--) {
- if (map->pbus_to_dieid[bus] >= 0)
+ if (map->pbus_to_dieid[bus] != -1)
i = map->pbus_to_dieid[bus];
else
map->pbus_to_dieid[bus] = i;
}
} else {
for (bus = 0; bus <= 255; bus++) {
- if (map->pbus_to_dieid[bus] >= 0)
+ if (map->pbus_to_dieid[bus] != -1)
i = map->pbus_to_dieid[bus];
else
map->pbus_to_dieid[bus] = i;
.perf_ctr = SNR_M2M_PCI_PMON_CTR0,
.event_ctl = SNR_M2M_PCI_PMON_CTL0,
.event_mask = SNBEP_PMON_RAW_EVENT_MASK,
+ .event_mask_ext = SNR_M2M_PCI_PMON_UMASK_EXT,
.box_ctl = SNR_M2M_PCI_PMON_BOX_CTL,
.ops = &snr_m2m_uncore_pci_ops,
- .format_group = &skx_uncore_format_group,
+ .format_group = &snr_m2m_uncore_format_group,
};
static struct attribute *icx_upi_uncore_formats_attr[] = {
void release_lbr_buffers(void);
+void reserve_lbr_buffers(void);
+
extern struct event_constraint bts_constraint;
extern struct event_constraint vlbr_constraint;
{
}
+static inline void reserve_lbr_buffers(void)
+{
+}
+
static inline int intel_pmu_init(void)
{
return 0;
extern int setup_APIC_eilvt(u8 lvt_off, u8 vector, u8 msg_type, u8 mask);
extern void lapic_assign_system_vectors(void);
extern void lapic_assign_legacy_vector(unsigned int isairq, bool replace);
+extern void lapic_update_legacy_vectors(void);
extern void lapic_online(void);
extern void lapic_offline(void);
extern bool apic_needs_pit(void);
/*
* Despite that some emulators terminate on UD2, we use it for WARN().
- *
- * Since various instruction decoders/specs disagree on the encoding of
- * UD0/UD1.
*/
-
-#define ASM_UD0 ".byte 0x0f, 0xff" /* + ModRM (for Intel) */
-#define ASM_UD1 ".byte 0x0f, 0xb9" /* + ModRM */
#define ASM_UD2 ".byte 0x0f, 0x0b"
-
-#define INSN_UD0 0xff0f
#define INSN_UD2 0x0b0f
-
#define LEN_UD2 2
#ifdef CONFIG_GENERIC_BUG
# define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31))
#endif
-#ifdef CONFIG_IOMMU_SUPPORT
-# define DISABLE_ENQCMD 0
-#else
-# define DISABLE_ENQCMD (1 << (X86_FEATURE_ENQCMD & 31))
-#endif
+/* Force disable because it's broken beyond repair */
+#define DISABLE_ENQCMD (1 << (X86_FEATURE_ENQCMD & 31))
#ifdef CONFIG_X86_SGX
# define DISABLE_SGX 0
*/
#define PASID_DISABLED 0
-#ifdef CONFIG_IOMMU_SUPPORT
-/* Update current's PASID MSR/state by mm's PASID. */
-void update_pasid(void);
-#else
static inline void update_pasid(void) { }
-#endif
+
#endif /* _ASM_X86_FPU_API_H */
pkru_val = pk->pkru;
}
__write_pkru(pkru_val);
-
- /*
- * Expensive PASID MSR write will be avoided in update_pasid() because
- * TIF_NEED_FPU_LOAD was set. And the PASID state won't be updated
- * unless it's different from mm->pasid to reduce overhead.
- */
- update_pasid();
}
#endif /* _ASM_X86_FPU_INTERNAL_H */
#endif
/* NMI */
+
+#if defined(CONFIG_X86_64) && IS_ENABLED(CONFIG_KVM_INTEL)
+/*
+ * Special NOIST entry point for VMX which invokes this on the kernel
+ * stack. asm_exc_nmi() requires an IST to work correctly vs. the NMI
+ * 'executing' marker.
+ *
+ * On 32bit this just uses the regular NMI entry point because 32-bit does
+ * not have ISTs.
+ */
+DECLARE_IDTENTRY(X86_TRAP_NMI, exc_nmi_noist);
+#else
+#define asm_exc_nmi_noist asm_exc_nmi
+#endif
+
DECLARE_IDTENTRY_NMI(X86_TRAP_NMI, exc_nmi);
#ifdef CONFIG_XEN_PV
DECLARE_IDTENTRY_RAW(X86_TRAP_NMI, xenpv_exc_nmi);
KVM_X86_OP_NULL(vcpu_blocking)
KVM_X86_OP_NULL(vcpu_unblocking)
KVM_X86_OP_NULL(update_pi_irte)
+KVM_X86_OP_NULL(start_assignment)
KVM_X86_OP_NULL(apicv_post_state_restore)
KVM_X86_OP_NULL(dy_apicv_has_pending_interrupt)
KVM_X86_OP_NULL(set_hv_timer)
#define VALID_PAGE(x) ((x) != INVALID_PAGE)
#define UNMAPPED_GVA (~(gpa_t)0)
+#define INVALID_GPA (~(gpa_t)0)
/* KVM Hugepage definitions for x86 */
#define KVM_MAX_HUGEPAGE_LEVEL PG_LEVEL_1G
#define KVM_NR_DB_REGS 4
+#define DR6_BUS_LOCK (1 << 11)
#define DR6_BD (1 << 13)
#define DR6_BS (1 << 14)
#define DR6_BT (1 << 15)
* DR6_ACTIVE_LOW is also used as the init/reset value for DR6.
*/
#define DR6_ACTIVE_LOW 0xffff0ff0
-#define DR6_VOLATILE 0x0001e00f
+#define DR6_VOLATILE 0x0001e80f
#define DR6_FIXED_1 (DR6_ACTIVE_LOW & ~DR6_VOLATILE)
#define DR7_BP_EN_MASK 0x000000ff
u32 pkru_mask;
u64 *pae_root;
- u64 *lm_root;
+ u64 *pml4_root;
/*
* check zero bits on shadow page table entries, these
int (*update_pi_irte)(struct kvm *kvm, unsigned int host_irq,
uint32_t guest_irq, bool set);
+ void (*start_assignment)(struct kvm *kvm);
void (*apicv_post_state_restore)(struct kvm_vcpu *vcpu);
bool (*dy_apicv_has_pending_interrupt)(struct kvm_vcpu *vcpu);
bool direct_map;
};
+extern u32 __read_mostly kvm_nr_uret_msrs;
extern u64 __read_mostly host_efer;
extern bool __read_mostly allow_smaller_maxphyaddr;
extern struct kvm_x86_ops kvm_x86_ops;
unsigned long ipi_bitmap_high, u32 min,
unsigned long icr, int op_64_bit);
-void kvm_define_user_return_msr(unsigned index, u32 msr);
+int kvm_add_user_return_msr(u32 msr);
+int kvm_find_user_return_msr(u32 msr);
int kvm_set_user_return_msr(unsigned index, u64 val, u64 mask);
+static inline bool kvm_is_supported_user_return_msr(u32 msr)
+{
+ return kvm_find_user_return_msr(msr) >= 0;
+}
+
u64 kvm_scale_tsc(struct kvm_vcpu *vcpu, u64 tsc);
u64 kvm_read_l1_tsc(struct kvm_vcpu *vcpu, u64 host_tsc);
#include <linux/interrupt.h>
#include <uapi/asm/kvm_para.h>
-extern void kvmclock_init(void);
-
#ifdef CONFIG_KVM_GUEST
bool kvm_check_and_clear_guest_paused(void);
#else
}
#ifdef CONFIG_KVM_GUEST
+void kvmclock_init(void);
+void kvmclock_disable(void);
bool kvm_para_available(void);
unsigned int kvm_arch_para_features(void);
unsigned int kvm_arch_para_hints(void);
void kvm_async_pf_task_wait_schedule(u32 token);
void kvm_async_pf_task_wake(u32 token);
u32 kvm_read_and_reset_apf_flags(void);
-void kvm_disable_steal_time(void);
bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token);
DECLARE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
return 0;
}
-static inline void kvm_disable_steal_time(void)
-{
- return;
-}
-
static __always_inline bool kvm_handle_async_pf(struct pt_regs *regs, u32 token)
{
return false;
/* K8 MSRs */
#define MSR_K8_TOP_MEM1 0xc001001a
#define MSR_K8_TOP_MEM2 0xc001001d
-#define MSR_K8_SYSCFG 0xc0010010
-#define MSR_K8_SYSCFG_MEM_ENCRYPT_BIT 23
-#define MSR_K8_SYSCFG_MEM_ENCRYPT BIT_ULL(MSR_K8_SYSCFG_MEM_ENCRYPT_BIT)
+#define MSR_AMD64_SYSCFG 0xc0010010
+#define MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT 23
+#define MSR_AMD64_SYSCFG_MEM_ENCRYPT BIT_ULL(MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT)
#define MSR_K8_INT_PENDING_MSG 0xc0010055
/* C1E active bits in int pending message */
#define K8_INTP_C1E_ACTIVE_MASK 0x18000000
return wrmsr_safe(msr, (u32)val, (u32)(val >> 32));
}
-#define write_tsc(low, high) wrmsr(MSR_IA32_TSC, (low), (high))
-
-#define write_rdtscp_aux(val) wrmsr(MSR_TSC_AUX, (val), 0)
-
struct msr *msrs_alloc(void);
void msrs_free(struct msr *msrs);
int msr_set_bit(u32 msr, u8 bit);
void copy_page(void *to, void *from);
+#ifdef CONFIG_X86_5LEVEL
+/*
+ * User space process size. This is the first address outside the user range.
+ * There are a few constraints that determine this:
+ *
+ * On Intel CPUs, if a SYSCALL instruction is at the highest canonical
+ * address, then that syscall will enter the kernel with a
+ * non-canonical return address, and SYSRET will explode dangerously.
+ * We avoid this particular problem by preventing anything
+ * from being mapped at the maximum canonical address.
+ *
+ * On AMD CPUs in the Ryzen family, there's a nasty bug in which the
+ * CPUs malfunction if they execute code from the highest canonical page.
+ * They'll speculate right off the end of the canonical space, and
+ * bad things happen. This is worked around in the same way as the
+ * Intel problem.
+ *
+ * With page table isolation enabled, we map the LDT in ... [stay tuned]
+ */
+static inline unsigned long task_size_max(void)
+{
+ unsigned long ret;
+
+ alternative_io("movq %[small],%0","movq %[large],%0",
+ X86_FEATURE_LA57,
+ "=r" (ret),
+ [small] "i" ((1ul << 47)-PAGE_SIZE),
+ [large] "i" ((1ul << 56)-PAGE_SIZE));
+
+ return ret;
+}
+#endif /* CONFIG_X86_5LEVEL */
+
#endif /* !__ASSEMBLY__ */
#ifdef CONFIG_X86_VSYSCALL_EMULATION
#ifdef CONFIG_X86_5LEVEL
#define __VIRTUAL_MASK_SHIFT (pgtable_l5_enabled() ? 56 : 47)
+/* See task_size_max() in <asm/page_64.h> */
#else
#define __VIRTUAL_MASK_SHIFT 47
+#define task_size_max() ((_AC(1,UL) << __VIRTUAL_MASK_SHIFT) - PAGE_SIZE)
#endif
-/*
- * User space process size. This is the first address outside the user range.
- * There are a few constraints that determine this:
- *
- * On Intel CPUs, if a SYSCALL instruction is at the highest canonical
- * address, then that syscall will enter the kernel with a
- * non-canonical return address, and SYSRET will explode dangerously.
- * We avoid this particular problem by preventing anything
- * from being mapped at the maximum canonical address.
- *
- * On AMD CPUs in the Ryzen family, there's a nasty bug in which the
- * CPUs malfunction if they execute code from the highest canonical page.
- * They'll speculate right off the end of the canonical space, and
- * bad things happen. This is worked around in the same way as the
- * Intel problem.
- *
- * With page table isolation enabled, we map the LDT in ... [stay tuned]
- */
-#define TASK_SIZE_MAX ((_AC(1,UL) << __VIRTUAL_MASK_SHIFT) - PAGE_SIZE)
-
+#define TASK_SIZE_MAX task_size_max()
#define DEFAULT_MAP_WINDOW ((1UL << 47) - PAGE_SIZE)
/* This decides where the kernel will search for a free chunk of vm
#ifdef CONFIG_CPU_SUP_AMD
extern u32 amd_get_nodes_per_socket(void);
+extern u32 amd_get_highest_perf(void);
#else
static inline u32 amd_get_nodes_per_socket(void) { return 0; }
+static inline u32 amd_get_highest_perf(void) { return 0; }
#endif
static inline uint32_t hypervisor_cpuid_base(const char *sig, uint32_t leaves)
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * AMD SEV header common between the guest and the hypervisor.
+ *
+ * Author: Brijesh Singh <brijesh.singh@amd.com>
+ */
+
+#ifndef __ASM_X86_SEV_COMMON_H
+#define __ASM_X86_SEV_COMMON_H
+
+#define GHCB_MSR_INFO_POS 0
+#define GHCB_MSR_INFO_MASK (BIT_ULL(12) - 1)
+
+#define GHCB_MSR_SEV_INFO_RESP 0x001
+#define GHCB_MSR_SEV_INFO_REQ 0x002
+#define GHCB_MSR_VER_MAX_POS 48
+#define GHCB_MSR_VER_MAX_MASK 0xffff
+#define GHCB_MSR_VER_MIN_POS 32
+#define GHCB_MSR_VER_MIN_MASK 0xffff
+#define GHCB_MSR_CBIT_POS 24
+#define GHCB_MSR_CBIT_MASK 0xff
+#define GHCB_MSR_SEV_INFO(_max, _min, _cbit) \
+ ((((_max) & GHCB_MSR_VER_MAX_MASK) << GHCB_MSR_VER_MAX_POS) | \
+ (((_min) & GHCB_MSR_VER_MIN_MASK) << GHCB_MSR_VER_MIN_POS) | \
+ (((_cbit) & GHCB_MSR_CBIT_MASK) << GHCB_MSR_CBIT_POS) | \
+ GHCB_MSR_SEV_INFO_RESP)
+#define GHCB_MSR_INFO(v) ((v) & 0xfffUL)
+#define GHCB_MSR_PROTO_MAX(v) (((v) >> GHCB_MSR_VER_MAX_POS) & GHCB_MSR_VER_MAX_MASK)
+#define GHCB_MSR_PROTO_MIN(v) (((v) >> GHCB_MSR_VER_MIN_POS) & GHCB_MSR_VER_MIN_MASK)
+
+#define GHCB_MSR_CPUID_REQ 0x004
+#define GHCB_MSR_CPUID_RESP 0x005
+#define GHCB_MSR_CPUID_FUNC_POS 32
+#define GHCB_MSR_CPUID_FUNC_MASK 0xffffffff
+#define GHCB_MSR_CPUID_VALUE_POS 32
+#define GHCB_MSR_CPUID_VALUE_MASK 0xffffffff
+#define GHCB_MSR_CPUID_REG_POS 30
+#define GHCB_MSR_CPUID_REG_MASK 0x3
+#define GHCB_CPUID_REQ_EAX 0
+#define GHCB_CPUID_REQ_EBX 1
+#define GHCB_CPUID_REQ_ECX 2
+#define GHCB_CPUID_REQ_EDX 3
+#define GHCB_CPUID_REQ(fn, reg) \
+ (GHCB_MSR_CPUID_REQ | \
+ (((unsigned long)reg & GHCB_MSR_CPUID_REG_MASK) << GHCB_MSR_CPUID_REG_POS) | \
+ (((unsigned long)fn) << GHCB_MSR_CPUID_FUNC_POS))
+
+#define GHCB_MSR_TERM_REQ 0x100
+#define GHCB_MSR_TERM_REASON_SET_POS 12
+#define GHCB_MSR_TERM_REASON_SET_MASK 0xf
+#define GHCB_MSR_TERM_REASON_POS 16
+#define GHCB_MSR_TERM_REASON_MASK 0xff
+#define GHCB_SEV_TERM_REASON(reason_set, reason_val) \
+ (((((u64)reason_set) & GHCB_MSR_TERM_REASON_SET_MASK) << GHCB_MSR_TERM_REASON_SET_POS) | \
+ ((((u64)reason_val) & GHCB_MSR_TERM_REASON_MASK) << GHCB_MSR_TERM_REASON_POS))
+
+#define GHCB_SEV_ES_REASON_GENERAL_REQUEST 0
+#define GHCB_SEV_ES_REASON_PROTOCOL_UNSUPPORTED 1
+
+#define GHCB_RESP_CODE(v) ((v) & GHCB_MSR_INFO_MASK)
+
+#endif
#include <linux/types.h>
#include <asm/insn.h>
+#include <asm/sev-common.h>
-#define GHCB_SEV_INFO 0x001UL
-#define GHCB_SEV_INFO_REQ 0x002UL
-#define GHCB_INFO(v) ((v) & 0xfffUL)
-#define GHCB_PROTO_MAX(v) (((v) >> 48) & 0xffffUL)
-#define GHCB_PROTO_MIN(v) (((v) >> 32) & 0xffffUL)
-#define GHCB_PROTO_OUR 0x0001UL
-#define GHCB_SEV_CPUID_REQ 0x004UL
-#define GHCB_CPUID_REQ_EAX 0
-#define GHCB_CPUID_REQ_EBX 1
-#define GHCB_CPUID_REQ_ECX 2
-#define GHCB_CPUID_REQ_EDX 3
-#define GHCB_CPUID_REQ(fn, reg) (GHCB_SEV_CPUID_REQ | \
- (((unsigned long)reg & 3) << 30) | \
- (((unsigned long)fn) << 32))
+#define GHCB_PROTO_OUR 0x0001UL
+#define GHCB_PROTOCOL_MAX 1ULL
+#define GHCB_DEFAULT_USAGE 0ULL
-#define GHCB_PROTOCOL_MAX 0x0001UL
-#define GHCB_DEFAULT_USAGE 0x0000UL
-
-#define GHCB_SEV_CPUID_RESP 0x005UL
-#define GHCB_SEV_TERMINATE 0x100UL
-#define GHCB_SEV_TERMINATE_REASON(reason_set, reason_val) \
- (((((u64)reason_set) & 0x7) << 12) | \
- ((((u64)reason_val) & 0xff) << 16))
-#define GHCB_SEV_ES_REASON_GENERAL_REQUEST 0
-#define GHCB_SEV_ES_REASON_PROTOCOL_UNSUPPORTED 1
-
-#define GHCB_SEV_GHCB_RESP_CODE(v) ((v) & 0xfff)
#define VMGEXIT() { asm volatile("rep; vmmcall\n\r"); }
enum es_result {
#define _ASM_X86_THERMAL_H
#ifdef CONFIG_X86_THERMAL_VECTOR
+void therm_lvt_init(void);
void intel_init_thermal(struct cpuinfo_x86 *c);
bool x86_thermal_enabled(void);
void intel_thermal_interrupt(void);
#else
-static inline void intel_init_thermal(struct cpuinfo_x86 *c) { }
+static inline void therm_lvt_init(void) { }
+static inline void intel_init_thermal(struct cpuinfo_x86 *c) { }
#endif
#endif /* _ASM_X86_THERMAL_H */
VDSO_CLOCKMODE_PVCLOCK, \
VDSO_CLOCKMODE_HVCLOCK
+#define HAVE_VDSO_CLOCKMODE_HVCLOCK
+
#endif /* __ASM_VDSO_CLOCKSOURCE_H */
__u16 flags;
} smm;
+ __u16 pad;
+
__u32 flags;
__u64 preemption_timer_deadline;
};
CFLAGS_REMOVE_ftrace.o = -pg
CFLAGS_REMOVE_early_printk.o = -pg
CFLAGS_REMOVE_head64.o = -pg
-CFLAGS_REMOVE_sev-es.o = -pg
+CFLAGS_REMOVE_sev.o = -pg
endif
KASAN_SANITIZE_head$(BITS).o := n
KASAN_SANITIZE_dumpstack_$(BITS).o := n
KASAN_SANITIZE_stacktrace.o := n
KASAN_SANITIZE_paravirt.o := n
-KASAN_SANITIZE_sev-es.o := n
+KASAN_SANITIZE_sev.o := n
# With some compiler versions the generated code results in boot hangs, caused
# by several compilation units. To be safe, disable all instrumentation.
obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o
obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o
-obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev-es.o
+obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev.o
###
# 64 bit specific files
ifeq ($(CONFIG_X86_64),y)
}
/*
+ * optimize_nops_range() - Optimize a sequence of single byte NOPs (0x90)
+ *
+ * @instr: instruction byte stream
+ * @instrlen: length of the above
+ * @off: offset within @instr where the first NOP has been detected
+ *
+ * Return: number of NOPs found (and replaced).
+ */
+static __always_inline int optimize_nops_range(u8 *instr, u8 instrlen, int off)
+{
+ unsigned long flags;
+ int i = off, nnops;
+
+ while (i < instrlen) {
+ if (instr[i] != 0x90)
+ break;
+
+ i++;
+ }
+
+ nnops = i - off;
+
+ if (nnops <= 1)
+ return nnops;
+
+ local_irq_save(flags);
+ add_nops(instr + off, nnops);
+ local_irq_restore(flags);
+
+ DUMP_BYTES(instr, instrlen, "%px: [%d:%d) optimized NOPs: ", instr, off, i);
+
+ return nnops;
+}
+
+/*
* "noinline" to cause control flow change and thus invalidate I$ and
* cause refetch after modification.
*/
static void __init_or_module noinline optimize_nops(struct alt_instr *a, u8 *instr)
{
- unsigned long flags;
struct insn insn;
- int nop, i = 0;
+ int i = 0;
/*
- * Jump over the non-NOP insns, the remaining bytes must be single-byte
- * NOPs, optimize them.
+ * Jump over the non-NOP insns and optimize single-byte NOPs into bigger
+ * ones.
*/
for (;;) {
if (insn_decode_kernel(&insn, &instr[i]))
return;
+ /*
+ * See if this and any potentially following NOPs can be
+ * optimized.
+ */
if (insn.length == 1 && insn.opcode.bytes[0] == 0x90)
- break;
-
- if ((i += insn.length) >= a->instrlen)
- return;
- }
+ i += optimize_nops_range(instr, a->instrlen, i);
+ else
+ i += insn.length;
- for (nop = i; i < a->instrlen; i++) {
- if (WARN_ONCE(instr[i] != 0x90, "Not a NOP at 0x%px\n", &instr[i]))
+ if (i >= a->instrlen)
return;
}
-
- local_irq_save(flags);
- add_nops(instr + nop, i - nop);
- local_irq_restore(flags);
-
- DUMP_BYTES(instr, a->instrlen, "%px: [%d:%d) optimized NOPs: ",
- instr, nop, a->instrlen);
}
/*
end_local_APIC_setup();
irq_remap_enable_fault_handling();
setup_IO_APIC();
+ lapic_update_legacy_vectors();
}
#ifdef CONFIG_UP_LATE_INIT
irq_matrix_assign_system(vector_matrix, ISA_IRQ_VECTOR(irq), replace);
}
+void __init lapic_update_legacy_vectors(void)
+{
+ unsigned int i;
+
+ if (IS_ENABLED(CONFIG_X86_IO_APIC) && nr_ioapics > 0)
+ return;
+
+ /*
+ * If the IO/APIC is disabled via config, kernel command line or
+ * lack of enumeration then all legacy interrupts are routed
+ * through the PIC. Make sure that they are marked as legacy
+ * vectors. PIC_CASCADE_IRQ has already been marked in
+ * lapic_assign_system_vectors().
+ */
+ for (i = 0; i < nr_legacy_irqs(); i++) {
+ if (i != PIC_CASCADE_IR)
+ lapic_assign_legacy_vector(i, true);
+ }
+}
+
void __init lapic_assign_system_vectors(void)
{
unsigned int i, vector = 0;
*/
if (cpu_has(c, X86_FEATURE_SME) || cpu_has(c, X86_FEATURE_SEV)) {
/* Check if memory encryption is enabled */
- rdmsrl(MSR_K8_SYSCFG, msr);
- if (!(msr & MSR_K8_SYSCFG_MEM_ENCRYPT))
+ rdmsrl(MSR_AMD64_SYSCFG, msr);
+ if (!(msr & MSR_AMD64_SYSCFG_MEM_ENCRYPT))
goto clear_all;
/*
break;
}
}
+
+u32 amd_get_highest_perf(void)
+{
+ struct cpuinfo_x86 *c = &boot_cpu_data;
+
+ if (c->x86 == 0x17 && ((c->x86_model >= 0x30 && c->x86_model < 0x40) ||
+ (c->x86_model >= 0x70 && c->x86_model < 0x80)))
+ return 166;
+
+ if (c->x86 == 0x19 && ((c->x86_model >= 0x20 && c->x86_model < 0x30) ||
+ (c->x86_model >= 0x40 && c->x86_model < 0x70)))
+ return 166;
+
+ return 255;
+}
+EXPORT_SYMBOL_GPL(amd_get_highest_perf);
unsigned long cpudata = vdso_encode_cpunode(cpu, early_cpu_to_node(cpu));
struct desc_struct d = { };
- if (boot_cpu_has(X86_FEATURE_RDTSCP))
- write_rdtscp_aux(cpudata);
+ if (boot_cpu_has(X86_FEATURE_RDTSCP) || boot_cpu_has(X86_FEATURE_RDPID))
+ wrmsr(MSR_TSC_AUX, cpudata, 0);
/* Store CPU and node number in limit. */
d.limit0 = cpudata;
if (boot_cpu_data.x86 < 0xf)
return 0;
/* In case some hypervisor doesn't pass SYSCFG through: */
- if (rdmsr_safe(MSR_K8_SYSCFG, &l, &h) < 0)
+ if (rdmsr_safe(MSR_AMD64_SYSCFG, &l, &h) < 0)
return 0;
/*
* Memory between 4GB and top of mem is forced WB by this magic bit.
(boot_cpu_data.x86 >= 0x0f)))
return;
- rdmsr(MSR_K8_SYSCFG, lo, hi);
+ rdmsr(MSR_AMD64_SYSCFG, lo, hi);
if (lo & K8_MTRRFIXRANGE_DRAM_MODIFY) {
pr_err(FW_WARN "MTRR: CPU %u: SYSCFG[MtrrFixDramModEn]"
" not cleared by BIOS, clearing this bit\n",
smp_processor_id());
lo &= ~K8_MTRRFIXRANGE_DRAM_MODIFY;
- mtrr_wrmsr(MSR_K8_SYSCFG, lo, hi);
+ mtrr_wrmsr(MSR_AMD64_SYSCFG, lo, hi);
}
}
case 15:
return msr - MSR_P4_BPU_PERFCTR0;
}
- fallthrough;
+ break;
case X86_VENDOR_ZHAOXIN:
case X86_VENDOR_CENTAUR:
return msr - MSR_ARCH_PERFMON_PERFCTR0;
case 15:
return msr - MSR_P4_BSU_ESCR0;
}
- fallthrough;
+ break;
case X86_VENDOR_ZHAOXIN:
case X86_VENDOR_CENTAUR:
return msr - MSR_ARCH_PERFMON_EVENTSEL0;
static const struct mbm_correction_factor_table {
u32 rmidthreshold;
u64 cf;
-} mbm_cf_table[] __initdata = {
+} mbm_cf_table[] __initconst = {
{7, CF(1.000000)},
{15, CF(1.000000)},
{15, CF(0.969650)},
return 0;
}
#endif /* CONFIG_PROC_PID_ARCH_STATUS */
-
-#ifdef CONFIG_IOMMU_SUPPORT
-void update_pasid(void)
-{
- u64 pasid_state;
- u32 pasid;
-
- if (!cpu_feature_enabled(X86_FEATURE_ENQCMD))
- return;
-
- if (!current->mm)
- return;
-
- pasid = READ_ONCE(current->mm->pasid);
- /* Set the valid bit in the PASID MSR/state only for valid pasid. */
- pasid_state = pasid == PASID_DISABLED ?
- pasid : pasid | MSR_IA32_PASID_VALID;
-
- /*
- * No need to hold fregs_lock() since the task's fpstate won't
- * be changed by others (e.g. ptrace) while the task is being
- * switched to or is in IPI.
- */
- if (!test_thread_flag(TIF_NEED_FPU_LOAD)) {
- /* The MSR is active and can be directly updated. */
- wrmsrl(MSR_IA32_PASID, pasid_state);
- } else {
- struct fpu *fpu = ¤t->thread.fpu;
- struct ia32_pasid_state *ppasid_state;
- struct xregs_state *xsave;
-
- /*
- * The CPU's xstate registers are not currently active. Just
- * update the PASID state in the memory buffer here. The
- * PASID MSR will be loaded when returning to user mode.
- */
- xsave = &fpu->state.xsave;
- xsave->header.xfeatures |= XFEATURE_MASK_PASID;
- ppasid_state = get_xsave_addr(xsave, XFEATURE_PASID);
- /*
- * Since XFEATURE_MASK_PASID is set in xfeatures, ppasid_state
- * won't be NULL and no need to check its value.
- *
- * Only update the task's PASID state when it's different
- * from the mm's pasid.
- */
- if (ppasid_state->pasid != pasid_state) {
- /*
- * Invalid fpregs so that state restoring will pick up
- * the PASID state.
- */
- __fpu_invalidate_fpregs_state(fpu);
- ppasid_state->pasid = pasid_state;
- }
- }
-}
-#endif /* CONFIG_IOMMU_SUPPORT */
#include <asm/realmode.h>
#include <asm/extable.h>
#include <asm/trapnr.h>
-#include <asm/sev-es.h>
+#include <asm/sev.h>
/*
* Manage page tables very early on.
#include <linux/kprobes.h>
#include <linux/nmi.h>
#include <linux/swait.h>
+#include <linux/syscore_ops.h>
#include <asm/timer.h>
#include <asm/cpu.h>
#include <asm/traps.h>
#include <asm/tlb.h>
#include <asm/cpuidle_haltpoll.h>
#include <asm/ptrace.h>
+#include <asm/reboot.h>
#include <asm/svm.h>
DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
wrmsrl(MSR_KVM_ASYNC_PF_EN, pa);
__this_cpu_write(apf_reason.enabled, 1);
- pr_info("KVM setup async PF for cpu %d\n", smp_processor_id());
+ pr_info("setup async PF for cpu %d\n", smp_processor_id());
}
if (kvm_para_has_feature(KVM_FEATURE_PV_EOI)) {
wrmsrl(MSR_KVM_ASYNC_PF_EN, 0);
__this_cpu_write(apf_reason.enabled, 0);
- pr_info("Unregister pv shared memory for cpu %d\n", smp_processor_id());
+ pr_info("disable async PF for cpu %d\n", smp_processor_id());
}
-static void kvm_pv_guest_cpu_reboot(void *unused)
+static void kvm_disable_steal_time(void)
{
- /*
- * We disable PV EOI before we load a new kernel by kexec,
- * since MSR_KVM_PV_EOI_EN stores a pointer into old kernel's memory.
- * New kernel can re-enable when it boots.
- */
- if (kvm_para_has_feature(KVM_FEATURE_PV_EOI))
- wrmsrl(MSR_KVM_PV_EOI_EN, 0);
- kvm_pv_disable_apf();
- kvm_disable_steal_time();
-}
+ if (!has_steal_clock)
+ return;
-static int kvm_pv_reboot_notify(struct notifier_block *nb,
- unsigned long code, void *unused)
-{
- if (code == SYS_RESTART)
- on_each_cpu(kvm_pv_guest_cpu_reboot, NULL, 1);
- return NOTIFY_DONE;
+ wrmsr(MSR_KVM_STEAL_TIME, 0, 0);
}
-static struct notifier_block kvm_pv_reboot_nb = {
- .notifier_call = kvm_pv_reboot_notify,
-};
-
static u64 kvm_steal_clock(int cpu)
{
u64 steal;
return steal;
}
-void kvm_disable_steal_time(void)
-{
- if (!has_steal_clock)
- return;
-
- wrmsr(MSR_KVM_STEAL_TIME, 0, 0);
-}
-
static inline void __set_percpu_decrypted(void *ptr, unsigned long size)
{
early_set_memory_decrypted((unsigned long) ptr, size);
}
}
+static void kvm_guest_cpu_offline(bool shutdown)
+{
+ kvm_disable_steal_time();
+ if (kvm_para_has_feature(KVM_FEATURE_PV_EOI))
+ wrmsrl(MSR_KVM_PV_EOI_EN, 0);
+ kvm_pv_disable_apf();
+ if (!shutdown)
+ apf_task_wake_all();
+ kvmclock_disable();
+}
+
+static int kvm_cpu_online(unsigned int cpu)
+{
+ unsigned long flags;
+
+ local_irq_save(flags);
+ kvm_guest_cpu_init();
+ local_irq_restore(flags);
+ return 0;
+}
+
#ifdef CONFIG_SMP
static DEFINE_PER_CPU(cpumask_var_t, __pv_cpu_mask);
kvm_spinlock_init();
}
-static void kvm_guest_cpu_offline(void)
+static int kvm_cpu_down_prepare(unsigned int cpu)
{
- kvm_disable_steal_time();
- if (kvm_para_has_feature(KVM_FEATURE_PV_EOI))
- wrmsrl(MSR_KVM_PV_EOI_EN, 0);
- kvm_pv_disable_apf();
- apf_task_wake_all();
+ unsigned long flags;
+
+ local_irq_save(flags);
+ kvm_guest_cpu_offline(false);
+ local_irq_restore(flags);
+ return 0;
}
-static int kvm_cpu_online(unsigned int cpu)
+#endif
+
+static int kvm_suspend(void)
{
- local_irq_disable();
- kvm_guest_cpu_init();
- local_irq_enable();
+ kvm_guest_cpu_offline(false);
+
return 0;
}
-static int kvm_cpu_down_prepare(unsigned int cpu)
+static void kvm_resume(void)
{
- local_irq_disable();
- kvm_guest_cpu_offline();
- local_irq_enable();
- return 0;
+ kvm_cpu_online(raw_smp_processor_id());
+}
+
+static struct syscore_ops kvm_syscore_ops = {
+ .suspend = kvm_suspend,
+ .resume = kvm_resume,
+};
+
+static void kvm_pv_guest_cpu_reboot(void *unused)
+{
+ kvm_guest_cpu_offline(true);
+}
+
+static int kvm_pv_reboot_notify(struct notifier_block *nb,
+ unsigned long code, void *unused)
+{
+ if (code == SYS_RESTART)
+ on_each_cpu(kvm_pv_guest_cpu_reboot, NULL, 1);
+ return NOTIFY_DONE;
}
+static struct notifier_block kvm_pv_reboot_nb = {
+ .notifier_call = kvm_pv_reboot_notify,
+};
+
+/*
+ * After a PV feature is registered, the host will keep writing to the
+ * registered memory location. If the guest happens to shutdown, this memory
+ * won't be valid. In cases like kexec, in which you install a new kernel, this
+ * means a random memory location will be kept being written.
+ */
+#ifdef CONFIG_KEXEC_CORE
+static void kvm_crash_shutdown(struct pt_regs *regs)
+{
+ kvm_guest_cpu_offline(true);
+ native_machine_crash_shutdown(regs);
+}
#endif
static void __init kvm_guest_init(void)
kvm_guest_cpu_init();
#endif
+#ifdef CONFIG_KEXEC_CORE
+ machine_ops.crash_shutdown = kvm_crash_shutdown;
+#endif
+
+ register_syscore_ops(&kvm_syscore_ops);
+
/*
* Hard lockup detection is enabled by default. Disable it, as guests
* can get false positives too easily, for example if the host is
#include <asm/hypervisor.h>
#include <asm/mem_encrypt.h>
#include <asm/x86_init.h>
-#include <asm/reboot.h>
#include <asm/kvmclock.h>
static int kvmclock __initdata = 1;
}
#endif
-/*
- * After the clock is registered, the host will keep writing to the
- * registered memory location. If the guest happens to shutdown, this memory
- * won't be valid. In cases like kexec, in which you install a new kernel, this
- * means a random memory location will be kept being written. So before any
- * kind of shutdown from our side, we unregister the clock by writing anything
- * that does not have the 'enable' bit set in the msr
- */
-#ifdef CONFIG_KEXEC_CORE
-static void kvm_crash_shutdown(struct pt_regs *regs)
-{
- native_write_msr(msr_kvm_system_time, 0, 0);
- kvm_disable_steal_time();
- native_machine_crash_shutdown(regs);
-}
-#endif
-
-static void kvm_shutdown(void)
+void kvmclock_disable(void)
{
native_write_msr(msr_kvm_system_time, 0, 0);
- kvm_disable_steal_time();
- native_machine_shutdown();
}
static void __init kvmclock_init_mem(void)
#endif
x86_platform.save_sched_clock_state = kvm_save_sched_clock_state;
x86_platform.restore_sched_clock_state = kvm_restore_sched_clock_state;
- machine_ops.shutdown = kvm_shutdown;
-#ifdef CONFIG_KEXEC_CORE
- machine_ops.crash_shutdown = kvm_crash_shutdown;
-#endif
kvm_get_preset_lpj();
/*
return;
/* SYS_CFG */
- address = MSR_K8_SYSCFG;
+ address = MSR_AMD64_SYSCFG;
rdmsrl(address, val);
/* TOP_MEM2 is not enabled? */
#include <asm/reboot.h>
#include <asm/cache.h>
#include <asm/nospec-branch.h>
-#include <asm/sev-es.h>
+#include <asm/sev.h>
#define CREATE_TRACE_POINTS
#include <trace/events/nmi.h>
mds_user_clear_cpu_buffers();
}
+#if defined(CONFIG_X86_64) && IS_ENABLED(CONFIG_KVM_INTEL)
+DEFINE_IDTENTRY_RAW(exc_nmi_noist)
+{
+ exc_nmi(regs);
+}
+#endif
+#if IS_MODULE(CONFIG_KVM_INTEL)
+EXPORT_SYMBOL_GPL(asm_exc_nmi_noist);
+#endif
+
void stop_nmi(void)
{
ignore_nmis++;
#include <asm/pci-direct.h>
#include <asm/prom.h>
#include <asm/proto.h>
+#include <asm/thermal.h>
#include <asm/unwind.h>
#include <asm/vsyscall.h>
#include <linux/vmalloc.h>
* them from accessing certain memory ranges, namely anything below
* 1M and in the pages listed in bad_pages[] above.
*
- * To avoid these pages being ever accessed by SNB gfx devices
- * reserve all memory below the 1 MB mark and bad_pages that have
- * not already been reserved at boot time.
+ * To avoid these pages being ever accessed by SNB gfx devices reserve
+ * bad_pages that have not already been reserved at boot time.
+ * All memory below the 1 MB mark is anyway reserved later during
+ * setup_arch(), so there is no need to reserve it here.
*/
- memblock_reserve(0, 1<<20);
for (i = 0; i < ARRAY_SIZE(bad_pages); i++) {
if (memblock_reserve(bad_pages[i], PAGE_SIZE))
* The first 4Kb of memory is a BIOS owned area, but generally it is
* not listed as such in the E820 table.
*
- * Reserve the first memory page and typically some additional
- * memory (64KiB by default) since some BIOSes are known to corrupt
- * low memory. See the Kconfig help text for X86_RESERVE_LOW.
+ * Reserve the first 64K of memory since some BIOSes are known to
+ * corrupt low memory. After the real mode trampoline is allocated the
+ * rest of the memory below 640k is reserved.
*
* In addition, make sure page 0 is always reserved because on
* systems with L1TF its contents can be leaked to user processes.
*/
- memblock_reserve(0, ALIGN(reserve_low, PAGE_SIZE));
+ memblock_reserve(0, SZ_64K);
early_reserve_initrd();
reserve_ibft_region();
reserve_bios_regions();
+ trim_snb_memory();
}
/*
(max_pfn_mapped<<PAGE_SHIFT) - 1);
#endif
- reserve_real_mode();
-
/*
- * Reserving memory causing GPU hangs on Sandy Bridge integrated
- * graphics devices should be done after we allocated memory under
- * 1M for the real mode trampoline.
+ * Find free memory for the real mode trampoline and place it
+ * there.
+ * If there is not enough free memory under 1M, on EFI-enabled
+ * systems there will be additional attempt to reclaim the memory
+ * for the real mode trampoline at efi_free_boot_services().
+ *
+ * Unconditionally reserve the entire first 1M of RAM because
+ * BIOSes are know to corrupt low memory and several
+ * hundred kilobytes are not worth complex detection what memory gets
+ * clobbered. Moreover, on machines with SandyBridge graphics or in
+ * setups that use crashkernel the entire 1M is reserved anyway.
*/
- trim_snb_memory();
+ reserve_real_mode();
init_mem_mapping();
x86_init.timers.wallclock_init();
+ /*
+ * This needs to run before setup_local_APIC() which soft-disables the
+ * local APIC temporarily and that masks the thermal LVT interrupt,
+ * leading to softlockups on machines which have configured SMI
+ * interrupt delivery.
+ */
+ therm_lvt_init();
+
mcheck_init();
register_refined_jiffies(CLOCK_TICK_RATE);
static void __noreturn sev_es_terminate(unsigned int reason)
{
- u64 val = GHCB_SEV_TERMINATE;
+ u64 val = GHCB_MSR_TERM_REQ;
/*
* Tell the hypervisor what went wrong - only reason-set 0 is
* currently supported.
*/
- val |= GHCB_SEV_TERMINATE_REASON(0, reason);
+ val |= GHCB_SEV_TERM_REASON(0, reason);
/* Request Guest Termination from Hypvervisor */
sev_es_wr_ghcb_msr(val);
u64 val;
/* Do the GHCB protocol version negotiation */
- sev_es_wr_ghcb_msr(GHCB_SEV_INFO_REQ);
+ sev_es_wr_ghcb_msr(GHCB_MSR_SEV_INFO_REQ);
VMGEXIT();
val = sev_es_rd_ghcb_msr();
- if (GHCB_INFO(val) != GHCB_SEV_INFO)
+ if (GHCB_MSR_INFO(val) != GHCB_MSR_SEV_INFO_RESP)
return false;
- if (GHCB_PROTO_MAX(val) < GHCB_PROTO_OUR ||
- GHCB_PROTO_MIN(val) > GHCB_PROTO_OUR)
+ if (GHCB_MSR_PROTO_MAX(val) < GHCB_PROTO_OUR ||
+ GHCB_MSR_PROTO_MIN(val) > GHCB_PROTO_OUR)
return false;
return true;
static __always_inline void vc_ghcb_invalidate(struct ghcb *ghcb)
{
+ ghcb->save.sw_exit_code = 0;
memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap));
}
sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_EAX));
VMGEXIT();
val = sev_es_rd_ghcb_msr();
- if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ if (GHCB_RESP_CODE(val) != GHCB_MSR_CPUID_RESP)
goto fail;
regs->ax = val >> 32;
sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_EBX));
VMGEXIT();
val = sev_es_rd_ghcb_msr();
- if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ if (GHCB_RESP_CODE(val) != GHCB_MSR_CPUID_RESP)
goto fail;
regs->bx = val >> 32;
sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_ECX));
VMGEXIT();
val = sev_es_rd_ghcb_msr();
- if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ if (GHCB_RESP_CODE(val) != GHCB_MSR_CPUID_RESP)
goto fail;
regs->cx = val >> 32;
sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_EDX));
VMGEXIT();
val = sev_es_rd_ghcb_msr();
- if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ if (GHCB_RESP_CODE(val) != GHCB_MSR_CPUID_RESP)
goto fail;
regs->dx = val >> 32;
#include <asm/cpu_entry_area.h>
#include <asm/stacktrace.h>
-#include <asm/sev-es.h>
+#include <asm/sev.h>
#include <asm/insn-eval.h>
#include <asm/fpu/internal.h>
#include <asm/processor.h>
if (unlikely(data->ghcb_active)) {
/* GHCB is already in use - save its contents */
- if (unlikely(data->backup_ghcb_active))
- return NULL;
+ if (unlikely(data->backup_ghcb_active)) {
+ /*
+ * Backup-GHCB is also already in use. There is no way
+ * to continue here so just kill the machine. To make
+ * panic() work, mark GHCBs inactive so that messages
+ * can be printed out.
+ */
+ data->ghcb_active = false;
+ data->backup_ghcb_active = false;
+
+ panic("Unable to handle #VC exception! GHCB and Backup GHCB are already in use");
+ }
/* Mark backup_ghcb active before writing to it */
data->backup_ghcb_active = true;
return ghcb;
}
-static __always_inline void sev_es_put_ghcb(struct ghcb_state *state)
-{
- struct sev_es_runtime_data *data;
- struct ghcb *ghcb;
-
- data = this_cpu_read(runtime_data);
- ghcb = &data->ghcb_page;
-
- if (state->ghcb) {
- /* Restore GHCB from Backup */
- *ghcb = *state->ghcb;
- data->backup_ghcb_active = false;
- state->ghcb = NULL;
- } else {
- data->ghcb_active = false;
- }
-}
-
/* Needed in vc_early_forward_exception */
void do_early_exception(struct pt_regs *regs, int trapnr);
u16 d2;
u8 d1;
- /* If instruction ran in kernel mode and the I/O buffer is in kernel space */
- if (!user_mode(ctxt->regs) && !access_ok(target, size)) {
- memcpy(dst, buf, size);
- return ES_OK;
- }
-
+ /*
+ * This function uses __put_user() independent of whether kernel or user
+ * memory is accessed. This works fine because __put_user() does no
+ * sanity checks of the pointer being accessed. All that it does is
+ * to report when the access failed.
+ *
+ * Also, this function runs in atomic context, so __put_user() is not
+ * allowed to sleep. The page-fault handler detects that it is running
+ * in atomic context and will not try to take mmap_sem and handle the
+ * fault, so additional pagefault_enable()/disable() calls are not
+ * needed.
+ *
+ * The access can't be done via copy_to_user() here because
+ * vc_write_mem() must not use string instructions to access unsafe
+ * memory. The reason is that MOVS is emulated by the #VC handler by
+ * splitting the move up into a read and a write and taking a nested #VC
+ * exception on whatever of them is the MMIO access. Using string
+ * instructions here would cause infinite nesting.
+ */
switch (size) {
case 1:
memcpy(&d1, buf, 1);
- if (put_user(d1, target))
+ if (__put_user(d1, target))
goto fault;
break;
case 2:
memcpy(&d2, buf, 2);
- if (put_user(d2, target))
+ if (__put_user(d2, target))
goto fault;
break;
case 4:
memcpy(&d4, buf, 4);
- if (put_user(d4, target))
+ if (__put_user(d4, target))
goto fault;
break;
case 8:
memcpy(&d8, buf, 8);
- if (put_user(d8, target))
+ if (__put_user(d8, target))
goto fault;
break;
default:
u16 d2;
u8 d1;
- /* If instruction ran in kernel mode and the I/O buffer is in kernel space */
- if (!user_mode(ctxt->regs) && !access_ok(s, size)) {
- memcpy(buf, src, size);
- return ES_OK;
- }
-
+ /*
+ * This function uses __get_user() independent of whether kernel or user
+ * memory is accessed. This works fine because __get_user() does no
+ * sanity checks of the pointer being accessed. All that it does is
+ * to report when the access failed.
+ *
+ * Also, this function runs in atomic context, so __get_user() is not
+ * allowed to sleep. The page-fault handler detects that it is running
+ * in atomic context and will not try to take mmap_sem and handle the
+ * fault, so additional pagefault_enable()/disable() calls are not
+ * needed.
+ *
+ * The access can't be done via copy_from_user() here because
+ * vc_read_mem() must not use string instructions to access unsafe
+ * memory. The reason is that MOVS is emulated by the #VC handler by
+ * splitting the move up into a read and a write and taking a nested #VC
+ * exception on whatever of them is the MMIO access. Using string
+ * instructions here would cause infinite nesting.
+ */
switch (size) {
case 1:
- if (get_user(d1, s))
+ if (__get_user(d1, s))
goto fault;
memcpy(buf, &d1, 1);
break;
case 2:
- if (get_user(d2, s))
+ if (__get_user(d2, s))
goto fault;
memcpy(buf, &d2, 2);
break;
case 4:
- if (get_user(d4, s))
+ if (__get_user(d4, s))
goto fault;
memcpy(buf, &d4, 4);
break;
case 8:
- if (get_user(d8, s))
+ if (__get_user(d8, s))
goto fault;
memcpy(buf, &d8, 8);
break;
}
/* Include code shared with pre-decompression boot stage */
-#include "sev-es-shared.c"
+#include "sev-shared.c"
+
+static __always_inline void sev_es_put_ghcb(struct ghcb_state *state)
+{
+ struct sev_es_runtime_data *data;
+ struct ghcb *ghcb;
+
+ data = this_cpu_read(runtime_data);
+ ghcb = &data->ghcb_page;
+
+ if (state->ghcb) {
+ /* Restore GHCB from Backup */
+ *ghcb = *state->ghcb;
+ data->backup_ghcb_active = false;
+ state->ghcb = NULL;
+ } else {
+ /*
+ * Invalidate the GHCB so a VMGEXIT instruction issued
+ * from userspace won't appear to be valid.
+ */
+ vc_ghcb_invalidate(ghcb);
+ data->ghcb_active = false;
+ }
+}
void noinstr __sev_es_nmi_complete(void)
{
case X86_TRAP_UD:
exc_invalid_op(ctxt->regs);
break;
+ case X86_TRAP_PF:
+ write_cr2(ctxt->fi.cr2);
+ exc_page_fault(ctxt->regs, error_code);
+ break;
case X86_TRAP_AC:
exc_alignment_check(ctxt->regs, error_code);
break;
*/
DEFINE_IDTENTRY_VC_SAFE_STACK(exc_vmm_communication)
{
- struct sev_es_runtime_data *data = this_cpu_read(runtime_data);
irqentry_state_t irq_state;
struct ghcb_state state;
struct es_em_ctxt ctxt;
*/
ghcb = sev_es_get_ghcb(&state);
- if (!ghcb) {
- /*
- * Mark GHCBs inactive so that panic() is able to print the
- * message.
- */
- data->ghcb_active = false;
- data->backup_ghcb_active = false;
-
- panic("Unable to handle #VC exception! GHCB and Backup GHCB are already in use");
- }
vc_ghcb_invalidate(ghcb);
result = vc_init_em_ctxt(&ctxt, regs, error_code);
BUILD_BUG_ON(offsetof(siginfo_t, si_addr) != 0x10);
BUILD_BUG_ON(offsetof(compat_siginfo_t, si_addr) != 0x0C);
+ BUILD_BUG_ON(offsetof(siginfo_t, si_trapno) != 0x18);
+ BUILD_BUG_ON(offsetof(compat_siginfo_t, si_trapno) != 0x10);
+
BUILD_BUG_ON(offsetof(siginfo_t, si_addr_lsb) != 0x18);
BUILD_BUG_ON(offsetof(compat_siginfo_t, si_addr_lsb) != 0x10);
BUILD_BUG_ON(offsetof(siginfo_t, si_pkey) != 0x20);
BUILD_BUG_ON(offsetof(compat_siginfo_t, si_pkey) != 0x14);
- BUILD_BUG_ON(offsetof(siginfo_t, si_perf) != 0x18);
- BUILD_BUG_ON(offsetof(compat_siginfo_t, si_perf) != 0x10);
+ BUILD_BUG_ON(offsetof(siginfo_t, si_perf_data) != 0x18);
+ BUILD_BUG_ON(offsetof(siginfo_t, si_perf_type) != 0x20);
+ BUILD_BUG_ON(offsetof(compat_siginfo_t, si_perf_data) != 0x10);
+ BUILD_BUG_ON(offsetof(compat_siginfo_t, si_perf_type) != 0x14);
CHECK_CSI_OFFSET(_sigpoll);
CHECK_CSI_SIZE (_sigpoll, 2*sizeof(int));
return true;
}
-#include <asm/cpu_device_id.h>
-#include <asm/intel-family.h>
-
#define X86_MATCH(model) \
X86_MATCH_VENDOR_FAM_MODEL_FEATURE(INTEL, 6, \
INTEL_FAM6_##model, X86_FEATURE_APERFMPERF, NULL)
return false;
}
- highest_perf = perf_caps.highest_perf;
+ highest_perf = amd_get_highest_perf();
nominal_perf = perf_caps.nominal_perf;
if (!highest_perf || !nominal_perf) {
F(AVX512_VPOPCNTDQ) | F(UMIP) | F(AVX512_VBMI2) | F(GFNI) |
F(VAES) | F(VPCLMULQDQ) | F(AVX512_VNNI) | F(AVX512_BITALG) |
F(CLDEMOTE) | F(MOVDIRI) | F(MOVDIR64B) | 0 /*WAITPKG*/ |
- F(SGX_LC)
+ F(SGX_LC) | F(BUS_LOCK_DETECT)
);
/* Set LA57 based on hardware capability. */
if (cpuid_ecx(7) & F(LA57))
F(ACE2) | F(ACE2_EN) | F(PHE) | F(PHE_EN) |
F(PMM) | F(PMM_EN)
);
+
+ /*
+ * Hide RDTSCP and RDPID if either feature is reported as supported but
+ * probing MSR_TSC_AUX failed. This is purely a sanity check and
+ * should never happen, but the guest will likely crash if RDTSCP or
+ * RDPID is misreported, and KVM has botched MSR_TSC_AUX emulation in
+ * the past. For example, the sanity check may fire if this instance of
+ * KVM is running as L1 on top of an older, broken KVM.
+ */
+ if (WARN_ON((kvm_cpu_cap_has(X86_FEATURE_RDTSCP) ||
+ kvm_cpu_cap_has(X86_FEATURE_RDPID)) &&
+ !kvm_is_supported_user_return_msr(MSR_TSC_AUX))) {
+ kvm_cpu_cap_clear(X86_FEATURE_RDTSCP);
+ kvm_cpu_cap_clear(X86_FEATURE_RDPID);
+ }
}
EXPORT_SYMBOL_GPL(kvm_set_cpu_caps);
case 7:
entry->flags |= KVM_CPUID_FLAG_SIGNIFCANT_INDEX;
entry->eax = 0;
- entry->ecx = F(RDPID);
+ if (kvm_cpu_cap_has(X86_FEATURE_RDTSCP))
+ entry->ecx = F(RDPID);
++array->nent;
+ break;
default:
break;
}
* from the register case of group9.
*/
static const struct gprefix pfx_0f_c7_7 = {
- N, N, N, II(DstMem | ModRM | Op3264 | EmulateOnUD, em_rdpid, rdtscp),
+ N, N, N, II(DstMem | ModRM | Op3264 | EmulateOnUD, em_rdpid, rdpid),
};
return rc;
}
-int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len)
+int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int emulation_type)
{
int rc = X86EMUL_CONTINUE;
int mode = ctxt->mode;
ctxt->execute = opcode.u.execute;
- if (unlikely(ctxt->ud) && likely(!(ctxt->d & EmulateOnUD)))
+ if (unlikely(emulation_type & EMULTYPE_TRAP_UD) &&
+ likely(!(ctxt->d & EmulateOnUD)))
return EMULATION_FAILED;
if (unlikely(ctxt->d &
{
struct kvm_hv *hv = to_kvm_hv(kvm);
u64 gfn;
+ int idx;
if (hv->hv_tsc_page_status == HV_TSC_PAGE_BROKEN ||
hv->hv_tsc_page_status == HV_TSC_PAGE_UNSET ||
gfn = hv->hv_tsc_page >> HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT;
hv->tsc_ref.tsc_sequence = 0;
+
+ /*
+ * Take the srcu lock as memslots will be accessed to check the gfn
+ * cache generation against the memslots generation.
+ */
+ idx = srcu_read_lock(&kvm->srcu);
if (kvm_write_guest(kvm, gfn_to_gpa(gfn),
&hv->tsc_ref, sizeof(hv->tsc_ref.tsc_sequence)))
hv->hv_tsc_page_status = HV_TSC_PAGE_BROKEN;
+ srcu_read_unlock(&kvm->srcu, idx);
out_unlock:
mutex_unlock(&hv->hv_lock);
int interruptibility;
bool perm_ok; /* do not check permissions if true */
- bool ud; /* inject an #UD if host doesn't support insn */
bool tf; /* TF value before instruction (after for syscall/sysret) */
bool have_exception;
x86_intercept_clgi,
x86_intercept_skinit,
x86_intercept_rdtscp,
+ x86_intercept_rdpid,
x86_intercept_icebp,
x86_intercept_wbinvd,
x86_intercept_monitor,
#define X86EMUL_MODE_HOST X86EMUL_MODE_PROT64
#endif
-int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len);
+int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int emulation_type);
bool x86_page_table_writing_insn(struct x86_emulate_ctxt *ctxt);
#define EMULATION_FAILED -1
#define EMULATION_OK 0
if (!apic_x2apic_mode(apic))
valid_reg_mask |= APIC_REG_MASK(APIC_ARBPRI);
+ if (alignment + len > 4)
+ return 1;
+
if (offset > 0x3f0 || !(valid_reg_mask & APIC_REG_MASK(offset)))
return 1;
static void cancel_hv_timer(struct kvm_lapic *apic);
+static void cancel_apic_timer(struct kvm_lapic *apic)
+{
+ hrtimer_cancel(&apic->lapic_timer.timer);
+ preempt_disable();
+ if (apic->lapic_timer.hv_timer_in_use)
+ cancel_hv_timer(apic);
+ preempt_enable();
+}
+
static void apic_update_lvtt(struct kvm_lapic *apic)
{
u32 timer_mode = kvm_lapic_get_reg(apic, APIC_LVTT) &
if (apic->lapic_timer.timer_mode != timer_mode) {
if (apic_lvtt_tscdeadline(apic) != (timer_mode ==
APIC_LVT_TIMER_TSCDEADLINE)) {
- hrtimer_cancel(&apic->lapic_timer.timer);
- preempt_disable();
- if (apic->lapic_timer.hv_timer_in_use)
- cancel_hv_timer(apic);
- preempt_enable();
+ cancel_apic_timer(apic);
kvm_lapic_set_reg(apic, APIC_TMICT, 0);
apic->lapic_timer.period = 0;
apic->lapic_timer.tscdeadline = 0;
guest_tsc = kvm_read_l1_tsc(vcpu, rdtsc());
apic->lapic_timer.advance_expire_delta = guest_tsc - tsc_deadline;
+ if (lapic_timer_advance_dynamic) {
+ adjust_lapic_timer_advance(vcpu, apic->lapic_timer.advance_expire_delta);
+ /*
+ * If the timer fired early, reread the TSC to account for the
+ * overhead of the above adjustment to avoid waiting longer
+ * than is necessary.
+ */
+ if (guest_tsc < tsc_deadline)
+ guest_tsc = kvm_read_l1_tsc(vcpu, rdtsc());
+ }
+
if (guest_tsc < tsc_deadline)
__wait_lapic_expire(vcpu, tsc_deadline - guest_tsc);
-
- if (lapic_timer_advance_dynamic)
- adjust_lapic_timer_advance(vcpu, apic->lapic_timer.advance_expire_delta);
}
void kvm_wait_lapic_expire(struct kvm_vcpu *vcpu)
}
atomic_inc(&apic->lapic_timer.pending);
- kvm_make_request(KVM_REQ_PENDING_TIMER, vcpu);
+ kvm_make_request(KVM_REQ_UNBLOCK, vcpu);
if (from_timer_fn)
kvm_vcpu_kick(vcpu);
}
if (!apic->lapic_timer.hv_timer_in_use)
goto out;
WARN_ON(rcuwait_active(&vcpu->wait));
- cancel_hv_timer(apic);
apic_timer_expired(apic, false);
+ cancel_hv_timer(apic);
if (apic_lvtt_period(apic) && apic->lapic_timer.period) {
advance_periodic_target_expiration(apic);
if (apic_lvtt_tscdeadline(apic))
break;
- hrtimer_cancel(&apic->lapic_timer.timer);
+ cancel_apic_timer(apic);
kvm_lapic_set_reg(apic, APIC_TMICT, val);
start_apic_timer(apic);
break;
if (mmu->shadow_root_level == PT64_ROOT_4LEVEL) {
pm_mask |= PT_ACCESSED_MASK | PT_WRITABLE_MASK | PT_USER_MASK;
- if (WARN_ON_ONCE(!mmu->lm_root)) {
+ if (WARN_ON_ONCE(!mmu->pml4_root)) {
r = -EIO;
goto out_unlock;
}
- mmu->lm_root[0] = __pa(mmu->pae_root) | pm_mask;
+ mmu->pml4_root[0] = __pa(mmu->pae_root) | pm_mask;
}
for (i = 0; i < 4; ++i) {
}
if (mmu->shadow_root_level == PT64_ROOT_4LEVEL)
- mmu->root_hpa = __pa(mmu->lm_root);
+ mmu->root_hpa = __pa(mmu->pml4_root);
else
mmu->root_hpa = __pa(mmu->pae_root);
static int mmu_alloc_special_roots(struct kvm_vcpu *vcpu)
{
struct kvm_mmu *mmu = vcpu->arch.mmu;
- u64 *lm_root, *pae_root;
+ u64 *pml4_root, *pae_root;
/*
* When shadowing 32-bit or PAE NPT with 64-bit NPT, the PML4 and PDP
if (WARN_ON_ONCE(mmu->shadow_root_level != PT64_ROOT_4LEVEL))
return -EIO;
- if (mmu->pae_root && mmu->lm_root)
+ if (mmu->pae_root && mmu->pml4_root)
return 0;
/*
* The special roots should always be allocated in concert. Yell and
* bail if KVM ends up in a state where only one of the roots is valid.
*/
- if (WARN_ON_ONCE(!tdp_enabled || mmu->pae_root || mmu->lm_root))
+ if (WARN_ON_ONCE(!tdp_enabled || mmu->pae_root || mmu->pml4_root))
return -EIO;
/*
if (!pae_root)
return -ENOMEM;
- lm_root = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT);
- if (!lm_root) {
+ pml4_root = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT);
+ if (!pml4_root) {
free_page((unsigned long)pae_root);
return -ENOMEM;
}
mmu->pae_root = pae_root;
- mmu->lm_root = lm_root;
+ mmu->pml4_root = pml4_root;
return 0;
}
context->inject_page_fault = kvm_inject_page_fault;
}
+static union kvm_mmu_role kvm_calc_nested_mmu_role(struct kvm_vcpu *vcpu)
+{
+ union kvm_mmu_role role = kvm_calc_shadow_root_page_role_common(vcpu, false);
+
+ /*
+ * Nested MMUs are used only for walking L2's gva->gpa, they never have
+ * shadow pages of their own and so "direct" has no meaning. Set it
+ * to "true" to try to detect bogus usage of the nested MMU.
+ */
+ role.base.direct = true;
+
+ if (!is_paging(vcpu))
+ role.base.level = 0;
+ else if (is_long_mode(vcpu))
+ role.base.level = is_la57_mode(vcpu) ? PT64_ROOT_5LEVEL :
+ PT64_ROOT_4LEVEL;
+ else if (is_pae(vcpu))
+ role.base.level = PT32E_ROOT_LEVEL;
+ else
+ role.base.level = PT32_ROOT_LEVEL;
+
+ return role;
+}
+
static void init_kvm_nested_mmu(struct kvm_vcpu *vcpu)
{
- union kvm_mmu_role new_role = kvm_calc_mmu_role_common(vcpu, false);
+ union kvm_mmu_role new_role = kvm_calc_nested_mmu_role(vcpu);
struct kvm_mmu *g_context = &vcpu->arch.nested_mmu;
if (new_role.as_u64 == g_context->mmu_role.as_u64)
if (!tdp_enabled && mmu->pae_root)
set_memory_encrypted((unsigned long)mmu->pae_root, 1);
free_page((unsigned long)mmu->pae_root);
- free_page((unsigned long)mmu->lm_root);
+ free_page((unsigned long)mmu->pml4_root);
}
static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu)
gpa_t pte_gpa[PT_MAX_FULL_LEVELS];
pt_element_t __user *ptep_user[PT_MAX_FULL_LEVELS];
bool pte_writable[PT_MAX_FULL_LEVELS];
- unsigned pt_access;
- unsigned pte_access;
+ unsigned int pt_access[PT_MAX_FULL_LEVELS];
+ unsigned int pte_access;
gfn_t gfn;
struct x86_exception fault;
};
}
walker->ptes[walker->level - 1] = pte;
+
+ /* Convert to ACC_*_MASK flags for struct guest_walker. */
+ walker->pt_access[walker->level - 1] = FNAME(gpte_access)(pt_access ^ walk_nx_mask);
} while (!is_last_gpte(mmu, walker->level, pte));
pte_pkey = FNAME(gpte_pkeys)(vcpu, pte);
accessed_dirty = have_ad ? pte_access & PT_GUEST_ACCESSED_MASK : 0;
/* Convert to ACC_*_MASK flags for struct guest_walker. */
- walker->pt_access = FNAME(gpte_access)(pt_access ^ walk_nx_mask);
walker->pte_access = FNAME(gpte_access)(pte_access ^ walk_nx_mask);
errcode = permission_fault(vcpu, mmu, walker->pte_access, pte_pkey, access);
if (unlikely(errcode))
}
pgprintk("%s: pte %llx pte_access %x pt_access %x\n",
- __func__, (u64)pte, walker->pte_access, walker->pt_access);
+ __func__, (u64)pte, walker->pte_access,
+ walker->pt_access[walker->level - 1]);
return 1;
error:
bool huge_page_disallowed = exec && nx_huge_page_workaround_enabled;
struct kvm_mmu_page *sp = NULL;
struct kvm_shadow_walk_iterator it;
- unsigned direct_access, access = gw->pt_access;
+ unsigned int direct_access, access;
int top_level, level, req_level, ret;
gfn_t base_gfn = gw->gfn;
sp = NULL;
if (!is_shadow_present_pte(*it.sptep)) {
table_gfn = gw->table_gfn[it.level - 2];
+ access = gw->pt_access[it.level - 2];
sp = kvm_mmu_get_page(vcpu, table_gfn, addr, it.level-1,
false, access);
}
}
/**
- * handle_changed_spte - handle bookkeeping associated with an SPTE change
+ * __handle_changed_spte - handle bookkeeping associated with an SPTE change
* @kvm: kvm instance
* @as_id: the address space of the paging structure the SPTE was a part of
* @gfn: the base GFN that was mapped by the SPTE
trace_kvm_tdp_mmu_spte_changed(as_id, gfn, level, old_spte, new_spte);
+ if (is_large_pte(old_spte) != is_large_pte(new_spte)) {
+ if (is_large_pte(old_spte))
+ atomic64_sub(1, (atomic64_t*)&kvm->stat.lpages);
+ else
+ atomic64_add(1, (atomic64_t*)&kvm->stat.lpages);
+ }
+
/*
* The only times a SPTE should be changed from a non-present to
* non-present state is when an MMIO entry is installed/modified/
}
if (!is_shadow_present_pte(iter.old_spte)) {
+ /*
+ * If SPTE has been forzen by another thread, just
+ * give up and retry, avoiding unnecessary page table
+ * allocation and free.
+ */
+ if (is_removed_spte(iter.old_spte))
+ break;
+
sp = alloc_tdp_mmu_page(vcpu, iter.gfn, iter.level);
child_pt = sp->spt;
}
/*
- * Remove write access from all the SPTEs mapping GFNs [start, end). If
- * skip_4k is set, SPTEs that map 4k pages, will not be write-protected.
- * Returns true if an SPTE has been changed and the TLBs need to be flushed.
+ * Remove write access from all SPTEs at or above min_level that map GFNs
+ * [start, end). Returns true if an SPTE has been changed and the TLBs need to
+ * be flushed.
*/
static bool wrprot_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
gfn_t start, gfn_t end, int min_level)
#include "svm.h"
/* enable / disable AVIC */
-int avic;
-#ifdef CONFIG_X86_LOCAL_APIC
-module_param(avic, int, S_IRUGO);
-#endif
+bool avic;
+module_param(avic, bool, S_IRUGO);
#define SVM_AVIC_DOORBELL 0xc001011b
return &avic_physical_id_table[index];
}
-/**
+/*
* Note:
* AVIC hardware walks the nested page table to check permissions,
* but does not use the SPA address specified in the leaf page
return ret;
}
-/**
+/*
* Note:
* The HW cannot support posting multicast/broadcast
* interrupts to a vCPU. So, we still use legacy interrupt
WRITE_ONCE(*(svm->avic_physical_id_cache), entry);
}
-/**
+/*
* This function is called during VCPU halt/unhalt.
*/
static void avic_set_running(struct kvm_vcpu *vcpu, bool is_run)
nested_svm_copy_common_state(svm->nested.vmcb02.ptr, svm->vmcb01.ptr);
svm_switch_vmcb(svm, &svm->vmcb01);
- WARN_ON_ONCE(svm->vmcb->control.exit_code != SVM_EXIT_VMRUN);
/*
* On vmexit the GIF is set to false and
__free_page(virt_to_page(svm->nested.vmcb02.ptr));
svm->nested.vmcb02.ptr = NULL;
+ /*
+ * When last_vmcb12_gpa matches the current vmcb12 gpa,
+ * some vmcb12 fields are not loaded if they are marked clean
+ * in the vmcb12, since in this case they are up to date already.
+ *
+ * When the vmcb02 is freed, this optimization becomes invalid.
+ */
+ svm->nested.last_vmcb12_gpa = INVALID_GPA;
+
svm->nested.initialized = false;
}
if (is_guest_mode(vcpu)) {
svm->nested.nested_run_pending = 0;
+ svm->nested.vmcb12_gpa = INVALID_GPA;
+
leave_guest_mode(vcpu);
- svm_switch_vmcb(svm, &svm->nested.vmcb02);
+ svm_switch_vmcb(svm, &svm->vmcb01);
nested_svm_uninit_mmu_context(vcpu);
vmcb_mark_all_dirty(svm->vmcb);
* L2 registers if needed are moved from the current VMCB to VMCB02.
*/
+ if (is_guest_mode(vcpu))
+ svm_leave_nested(svm);
+ else
+ svm->nested.vmcb02.ptr->save = svm->vmcb01.ptr->save;
+
+ svm_set_gif(svm, !!(kvm_state->flags & KVM_STATE_NESTED_GIF_SET));
+
svm->nested.nested_run_pending =
!!(kvm_state->flags & KVM_STATE_NESTED_RUN_PENDING);
svm->nested.vmcb12_gpa = kvm_state->hdr.svm.vmcb_pa;
- if (svm->current_vmcb == &svm->vmcb01)
- svm->nested.vmcb02.ptr->save = svm->vmcb01.ptr->save;
svm->vmcb01.ptr->save.es = save->es;
svm->vmcb01.ptr->save.cs = save->cs;
sev->misc_cg = NULL;
}
-static void sev_unbind_asid(struct kvm *kvm, unsigned int handle)
+static void sev_decommission(unsigned int handle)
{
struct sev_data_decommission decommission;
+
+ if (!handle)
+ return;
+
+ decommission.handle = handle;
+ sev_guest_decommission(&decommission, NULL);
+}
+
+static void sev_unbind_asid(struct kvm *kvm, unsigned int handle)
+{
struct sev_data_deactivate deactivate;
if (!handle)
sev_guest_deactivate(&deactivate, NULL);
up_read(&sev_deactivate_lock);
- /* decommission handle */
- decommission.handle = handle;
- sev_guest_decommission(&decommission, NULL);
+ sev_decommission(handle);
}
static int sev_guest_init(struct kvm *kvm, struct kvm_sev_cmd *argp)
/* Bind ASID to this guest */
ret = sev_bind_asid(kvm, start.handle, error);
- if (ret)
+ if (ret) {
+ sev_decommission(start.handle);
goto e_free_session;
+ }
/* return handle to userspace */
params.handle = start.handle;
}
static int __sev_dbg_decrypt_user(struct kvm *kvm, unsigned long paddr,
- unsigned long __user dst_uaddr,
+ void __user *dst_uaddr,
unsigned long dst_paddr,
int size, int *err)
{
if (tpage) {
offset = paddr & 15;
- if (copy_to_user((void __user *)(uintptr_t)dst_uaddr,
- page_address(tpage) + offset, size))
+ if (copy_to_user(dst_uaddr, page_address(tpage) + offset, size))
ret = -EFAULT;
}
}
static int __sev_dbg_encrypt_user(struct kvm *kvm, unsigned long paddr,
- unsigned long __user vaddr,
+ void __user *vaddr,
unsigned long dst_paddr,
- unsigned long __user dst_vaddr,
+ void __user *dst_vaddr,
int size, int *error)
{
struct page *src_tpage = NULL;
int ret, len = size;
/* If source buffer is not aligned then use an intermediate buffer */
- if (!IS_ALIGNED(vaddr, 16)) {
+ if (!IS_ALIGNED((unsigned long)vaddr, 16)) {
src_tpage = alloc_page(GFP_KERNEL);
if (!src_tpage)
return -ENOMEM;
- if (copy_from_user(page_address(src_tpage),
- (void __user *)(uintptr_t)vaddr, size)) {
+ if (copy_from_user(page_address(src_tpage), vaddr, size)) {
__free_page(src_tpage);
return -EFAULT;
}
* - copy the source buffer in an intermediate buffer
* - use the intermediate buffer as source buffer
*/
- if (!IS_ALIGNED(dst_vaddr, 16) || !IS_ALIGNED(size, 16)) {
+ if (!IS_ALIGNED((unsigned long)dst_vaddr, 16) || !IS_ALIGNED(size, 16)) {
int dst_offset;
dst_tpage = alloc_page(GFP_KERNEL);
page_address(src_tpage), size);
else {
if (copy_from_user(page_address(dst_tpage) + dst_offset,
- (void __user *)(uintptr_t)vaddr, size)) {
+ vaddr, size)) {
ret = -EFAULT;
goto e_free;
}
if (dec)
ret = __sev_dbg_decrypt_user(kvm,
__sme_page_pa(src_p[0]) + s_off,
- dst_vaddr,
+ (void __user *)dst_vaddr,
__sme_page_pa(dst_p[0]) + d_off,
len, &argp->error);
else
ret = __sev_dbg_encrypt_user(kvm,
__sme_page_pa(src_p[0]) + s_off,
- vaddr,
+ (void __user *)vaddr,
__sme_page_pa(dst_p[0]) + d_off,
- dst_vaddr,
+ (void __user *)dst_vaddr,
len, &argp->error);
sev_unpin_memory(kvm, src_p, n);
struct sev_data_send_start data;
int ret;
+ memset(&data, 0, sizeof(data));
data.handle = sev->handle;
ret = sev_issue_cmd(kvm, SEV_CMD_SEND_START, &data, &argp->error);
- if (ret < 0)
- return ret;
params->session_len = data.session_len;
if (copy_to_user((void __user *)(uintptr_t)argp->data, params,
struct sev_data_send_update_data data;
int ret;
+ memset(&data, 0, sizeof(data));
data.handle = sev->handle;
ret = sev_issue_cmd(kvm, SEV_CMD_SEND_UPDATE_DATA, &data, &argp->error);
- if (ret < 0)
- return ret;
params->hdr_len = data.hdr_len;
params->trans_len = data.trans_len;
e_source_unlock:
mutex_unlock(&source_kvm->lock);
e_source_put:
- fput(source_kvm_file);
+ if (source_kvm_file)
+ fput(source_kvm_file);
return ret;
}
return -EINVAL;
}
-static void pre_sev_es_run(struct vcpu_svm *svm)
+void sev_es_unmap_ghcb(struct vcpu_svm *svm)
{
if (!svm->ghcb)
return;
struct svm_cpu_data *sd = per_cpu(svm_data, cpu);
int asid = sev_get_asid(svm->vcpu.kvm);
- /* Perform any SEV-ES pre-run actions */
- pre_sev_es_run(svm);
-
/* Assign the asid allocated with this SEV guest */
svm->asid = asid;
* RDTSCP and RDPID are not used in the kernel, specifically to allow KVM to
* defer the restoration of TSC_AUX until the CPU returns to userspace.
*/
-#define TSC_AUX_URET_SLOT 0
+static int tsc_aux_uret_slot __read_mostly = -1;
static const u32 msrpm_ranges[] = {0, 0xc0000000, 0xc0010000};
return 0;
}
+ if (pgtable_l5_enabled()) {
+ pr_info("KVM doesn't yet support 5-level paging on AMD SVM\n");
+ return 0;
+ }
+
return 1;
}
return;
/* If memory encryption is not enabled, use existing mask */
- rdmsrl(MSR_K8_SYSCFG, msr);
- if (!(msr & MSR_K8_SYSCFG_MEM_ENCRYPT))
+ rdmsrl(MSR_AMD64_SYSCFG, msr);
+ if (!(msr & MSR_AMD64_SYSCFG_MEM_ENCRYPT))
return;
enc_bit = cpuid_ebx(0x8000001f) & 0x3f;
kvm_tsc_scaling_ratio_frac_bits = 32;
}
- if (boot_cpu_has(X86_FEATURE_RDTSCP))
- kvm_define_user_return_msr(TSC_AUX_URET_SLOT, MSR_TSC_AUX);
+ tsc_aux_uret_slot = kvm_add_user_return_msr(MSR_TSC_AUX);
/* Check for pause filtering support */
if (!boot_cpu_has(X86_FEATURE_PAUSEFILTER)) {
}
if (avic) {
- if (!npt_enabled ||
- !boot_cpu_has(X86_FEATURE_AVIC) ||
- !IS_ENABLED(CONFIG_X86_LOCAL_APIC)) {
+ if (!npt_enabled || !boot_cpu_has(X86_FEATURE_AVIC)) {
avic = false;
} else {
pr_info("AVIC enabled\n");
return svm->vmcb->control.tsc_offset;
}
-static void svm_check_invpcid(struct vcpu_svm *svm)
+/* Evaluate instruction intercepts that depend on guest CPUID features. */
+static void svm_recalc_instruction_intercepts(struct kvm_vcpu *vcpu,
+ struct vcpu_svm *svm)
{
/*
* Intercept INVPCID if shadow paging is enabled to sync/free shadow
else
svm_clr_intercept(svm, INTERCEPT_INVPCID);
}
+
+ if (kvm_cpu_cap_has(X86_FEATURE_RDTSCP)) {
+ if (guest_cpuid_has(vcpu, X86_FEATURE_RDTSCP))
+ svm_clr_intercept(svm, INTERCEPT_RDTSCP);
+ else
+ svm_set_intercept(svm, INTERCEPT_RDTSCP);
+ }
}
static void init_vmcb(struct kvm_vcpu *vcpu)
svm->current_vmcb->asid_generation = 0;
svm->asid = 0;
- svm->nested.vmcb12_gpa = 0;
- svm->nested.last_vmcb12_gpa = 0;
+ svm->nested.vmcb12_gpa = INVALID_GPA;
+ svm->nested.last_vmcb12_gpa = INVALID_GPA;
vcpu->arch.hflags = 0;
if (!kvm_pause_in_guest(vcpu->kvm)) {
svm_clr_intercept(svm, INTERCEPT_PAUSE);
}
- svm_check_invpcid(svm);
+ svm_recalc_instruction_intercepts(vcpu, svm);
/*
* If the host supports V_SPEC_CTRL then disable the interception
struct vcpu_svm *svm = to_svm(vcpu);
struct svm_cpu_data *sd = per_cpu(svm_data, vcpu->cpu);
+ if (sev_es_guest(vcpu->kvm))
+ sev_es_unmap_ghcb(svm);
+
if (svm->guest_state_loaded)
return;
}
}
- if (static_cpu_has(X86_FEATURE_RDTSCP))
- kvm_set_user_return_msr(TSC_AUX_URET_SLOT, svm->tsc_aux, -1ull);
+ if (likely(tsc_aux_uret_slot >= 0))
+ kvm_set_user_return_msr(tsc_aux_uret_slot, svm->tsc_aux, -1ull);
svm->guest_state_loaded = true;
}
msr_info->data |= (u64)svm->sysenter_esp_hi << 32;
break;
case MSR_TSC_AUX:
- if (!boot_cpu_has(X86_FEATURE_RDTSCP))
- return 1;
- if (!msr_info->host_initiated &&
- !guest_cpuid_has(vcpu, X86_FEATURE_RDTSCP))
- return 1;
msr_info->data = svm->tsc_aux;
break;
/*
svm->sysenter_esp_hi = guest_cpuid_is_intel(vcpu) ? (data >> 32) : 0;
break;
case MSR_TSC_AUX:
- if (!boot_cpu_has(X86_FEATURE_RDTSCP))
- return 1;
-
- if (!msr->host_initiated &&
- !guest_cpuid_has(vcpu, X86_FEATURE_RDTSCP))
- return 1;
-
- /*
- * Per Intel's SDM, bits 63:32 are reserved, but AMD's APM has
- * incomplete and conflicting architectural behavior. Current
- * AMD CPUs completely ignore bits 63:32, i.e. they aren't
- * reserved and always read as zeros. Emulate AMD CPU behavior
- * to avoid explosions if the vCPU is migrated from an AMD host
- * to an Intel host.
- */
- data = (u32)data;
-
/*
* TSC_AUX is usually changed only during boot and never read
* directly. Intercept TSC_AUX instead of exposing it to the
* guest via direct_access_msrs, and switch it via user return.
*/
preempt_disable();
- r = kvm_set_user_return_msr(TSC_AUX_URET_SLOT, data, -1ull);
+ r = kvm_set_user_return_msr(tsc_aux_uret_slot, data, -1ull);
preempt_enable();
if (r)
return 1;
[SVM_EXIT_STGI] = stgi_interception,
[SVM_EXIT_CLGI] = clgi_interception,
[SVM_EXIT_SKINIT] = skinit_interception,
+ [SVM_EXIT_RDTSCP] = kvm_handle_invalid_op,
[SVM_EXIT_WBINVD] = kvm_emulate_wbinvd,
[SVM_EXIT_MONITOR] = kvm_emulate_monitor,
[SVM_EXIT_MWAIT] = kvm_emulate_mwait,
struct vcpu_svm *svm = to_svm(vcpu);
unsigned long vmcb_pa = svm->current_vmcb->pa;
- /*
- * VMENTER enables interrupts (host state), but the kernel state is
- * interrupts disabled when this is invoked. Also tell RCU about
- * it. This is the same logic as for exit_to_user_mode().
- *
- * This ensures that e.g. latency analysis on the host observes
- * guest mode as interrupt enabled.
- *
- * guest_enter_irqoff() informs context tracking about the
- * transition to guest mode and if enabled adjusts RCU state
- * accordingly.
- */
- instrumentation_begin();
- trace_hardirqs_on_prepare();
- lockdep_hardirqs_on_prepare(CALLER_ADDR0);
- instrumentation_end();
-
- guest_enter_irqoff();
- lockdep_hardirqs_on(CALLER_ADDR0);
+ kvm_guest_enter_irqoff();
if (sev_es_guest(vcpu->kvm)) {
__svm_sev_es_vcpu_run(vmcb_pa);
vmload(__sme_page_pa(sd->save_area));
}
- /*
- * VMEXIT disables interrupts (host state), but tracing and lockdep
- * have them in state 'on' as recorded before entering guest mode.
- * Same as enter_from_user_mode().
- *
- * guest_exit_irqoff() restores host context and reinstates RCU if
- * enabled and required.
- *
- * This needs to be done before the below as native_read_msr()
- * contains a tracepoint and x86_spec_ctrl_restore_host() calls
- * into world and some more.
- */
- lockdep_hardirqs_off(CALLER_ADDR0);
- guest_exit_irqoff();
-
- instrumentation_begin();
- trace_hardirqs_off_finish();
- instrumentation_end();
+ kvm_guest_exit_irqoff();
}
static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu)
svm->nrips_enabled = kvm_cpu_cap_has(X86_FEATURE_NRIPS) &&
guest_cpuid_has(vcpu, X86_FEATURE_NRIPS);
- /* Check again if INVPCID interception if required */
- svm_check_invpcid(svm);
+ svm_recalc_instruction_intercepts(vcpu, svm);
/* For sev guests, the memory encryption bit is not reserved in CR3. */
if (sev_guest(vcpu->kvm)) {
#include <linux/bits.h>
#include <asm/svm.h>
+#include <asm/sev-common.h>
#define __sme_page_pa(x) __sme_set(page_to_pfn(x) << PAGE_SHIFT)
#define VMCB_AVIC_APIC_BAR_MASK 0xFFFFFFFFFF000ULL
-extern int avic;
+extern bool avic;
static inline void avic_update_vapic_bar(struct vcpu_svm *svm, u64 data)
{
/* sev.c */
-#define GHCB_VERSION_MAX 1ULL
-#define GHCB_VERSION_MIN 1ULL
-
-#define GHCB_MSR_INFO_POS 0
-#define GHCB_MSR_INFO_MASK (BIT_ULL(12) - 1)
-
-#define GHCB_MSR_SEV_INFO_RESP 0x001
-#define GHCB_MSR_SEV_INFO_REQ 0x002
-#define GHCB_MSR_VER_MAX_POS 48
-#define GHCB_MSR_VER_MAX_MASK 0xffff
-#define GHCB_MSR_VER_MIN_POS 32
-#define GHCB_MSR_VER_MIN_MASK 0xffff
-#define GHCB_MSR_CBIT_POS 24
-#define GHCB_MSR_CBIT_MASK 0xff
-#define GHCB_MSR_SEV_INFO(_max, _min, _cbit) \
- ((((_max) & GHCB_MSR_VER_MAX_MASK) << GHCB_MSR_VER_MAX_POS) | \
- (((_min) & GHCB_MSR_VER_MIN_MASK) << GHCB_MSR_VER_MIN_POS) | \
- (((_cbit) & GHCB_MSR_CBIT_MASK) << GHCB_MSR_CBIT_POS) | \
- GHCB_MSR_SEV_INFO_RESP)
-
-#define GHCB_MSR_CPUID_REQ 0x004
-#define GHCB_MSR_CPUID_RESP 0x005
-#define GHCB_MSR_CPUID_FUNC_POS 32
-#define GHCB_MSR_CPUID_FUNC_MASK 0xffffffff
-#define GHCB_MSR_CPUID_VALUE_POS 32
-#define GHCB_MSR_CPUID_VALUE_MASK 0xffffffff
-#define GHCB_MSR_CPUID_REG_POS 30
-#define GHCB_MSR_CPUID_REG_MASK 0x3
-
-#define GHCB_MSR_TERM_REQ 0x100
-#define GHCB_MSR_TERM_REASON_SET_POS 12
-#define GHCB_MSR_TERM_REASON_SET_MASK 0xf
-#define GHCB_MSR_TERM_REASON_POS 16
-#define GHCB_MSR_TERM_REASON_MASK 0xff
+#define GHCB_VERSION_MAX 1ULL
+#define GHCB_VERSION_MIN 1ULL
+
extern unsigned int max_sev_asid;
void sev_es_create_vcpu(struct vcpu_svm *svm);
void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector);
void sev_es_prepare_guest_switch(struct vcpu_svm *svm, unsigned int cpu);
+void sev_es_unmap_ghcb(struct vcpu_svm *svm);
/* vmenter.S */
TP_ARGS(msg, err),
TP_STRUCT__entry(
- __field(const char *, msg)
+ __string(msg, msg)
__field(u32, err)
),
TP_fast_assign(
- __entry->msg = msg;
+ __assign_str(msg, msg);
__entry->err = err;
),
- TP_printk("%s%s", __entry->msg, !__entry->err ? "" :
+ TP_printk("%s%s", __get_str(msg), !__entry->err ? "" :
__print_symbolic(__entry->err, VMX_VMENTER_INSTRUCTION_ERRORS))
);
static inline bool cpu_has_vmx_posted_intr(void)
{
- return IS_ENABLED(CONFIG_X86_LOCAL_APIC) &&
- vmcs_config.pin_based_exec_ctrl & PIN_BASED_POSTED_INTR;
+ return vmcs_config.pin_based_exec_ctrl & PIN_BASED_POSTED_INTR;
}
static inline bool cpu_has_load_ia32_efer(void)
{
u64 debugctl = 0;
+ if (boot_cpu_has(X86_FEATURE_BUS_LOCK_DETECT))
+ debugctl |= DEBUGCTLMSR_BUS_LOCK_DETECT;
+
if (vmx_get_perf_capabilities() & PMU_CAP_LBR_FMT)
debugctl |= DEBUGCTLMSR_LBR_MASK;
nested_vmx_handle_enlightened_vmptrld(vcpu, false);
if (evmptrld_status == EVMPTRLD_VMFAIL ||
- evmptrld_status == EVMPTRLD_ERROR) {
- pr_debug_ratelimited("%s: enlightened vmptrld failed\n",
- __func__);
- vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
- vcpu->run->internal.suberror =
- KVM_INTERNAL_ERROR_EMULATION;
- vcpu->run->internal.ndata = 0;
+ evmptrld_status == EVMPTRLD_ERROR)
return false;
- }
}
return true;
static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
{
- if (!nested_get_evmcs_page(vcpu))
+ if (!nested_get_evmcs_page(vcpu)) {
+ pr_debug_ratelimited("%s: enlightened vmptrld failed\n",
+ __func__);
+ vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
+ vcpu->run->internal.suberror =
+ KVM_INTERNAL_ERROR_EMULATION;
+ vcpu->run->internal.ndata = 0;
+
return false;
+ }
if (is_guest_mode(vcpu) && !nested_get_vmcs12_pages(vcpu))
return false;
/* Similarly, triple faults in L2 should never escape. */
WARN_ON_ONCE(kvm_check_request(KVM_REQ_TRIPLE_FAULT, vcpu));
- kvm_clear_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
+ if (kvm_check_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu)) {
+ /*
+ * KVM_REQ_GET_NESTED_STATE_PAGES is also used to map
+ * Enlightened VMCS after migration and we still need to
+ * do that when something is forcing L2->L1 exit prior to
+ * the first L2 run.
+ */
+ (void)nested_get_evmcs_page(vcpu);
+ }
/* Service the TLB flush request for L2 before switching to L1. */
if (kvm_check_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu))
/*
+ * Bail out of the block loop if the VM has an assigned
+ * device, but the blocking vCPU didn't reconfigure the
+ * PI.NV to the wakeup vector, i.e. the assigned device
+ * came along after the initial check in pi_pre_block().
+ */
+void vmx_pi_start_assignment(struct kvm *kvm)
+{
+ if (!irq_remapping_cap(IRQ_POSTING_CAP))
+ return;
+
+ kvm_make_all_cpus_request(kvm, KVM_REQ_UNBLOCK);
+}
+
+/*
* pi_update_irte - set IRTE for Posted-Interrupts
*
* @kvm: kvm
bool pi_has_pending_interrupt(struct kvm_vcpu *vcpu);
int pi_update_irte(struct kvm *kvm, unsigned int host_irq, uint32_t guest_irq,
bool set);
+void vmx_pi_start_assignment(struct kvm *kvm);
#endif /* __KVM_X86_VMX_POSTED_INTR_H */
#include <asm/debugreg.h>
#include <asm/desc.h>
#include <asm/fpu/internal.h>
+#include <asm/idtentry.h>
#include <asm/io.h>
#include <asm/irq_remapping.h>
#include <asm/kexec.h>
static unsigned long host_idt_base;
-/*
- * Though SYSCALL is only supported in 64-bit mode on Intel CPUs, kvm
- * will emulate SYSCALL in legacy mode if the vendor string in guest
- * CPUID.0:{EBX,ECX,EDX} is "AuthenticAMD" or "AMDisbetter!" To
- * support this emulation, IA32_STAR must always be included in
- * vmx_uret_msrs_list[], even in i386 builds.
- */
-static const u32 vmx_uret_msrs_list[] = {
-#ifdef CONFIG_X86_64
- MSR_SYSCALL_MASK, MSR_LSTAR, MSR_CSTAR,
-#endif
- MSR_EFER, MSR_TSC_AUX, MSR_STAR,
- MSR_IA32_TSX_CTRL,
-};
-
#if IS_ENABLED(CONFIG_HYPERV)
static bool __read_mostly enlightened_vmcs = true;
module_param(enlightened_vmcs, bool, 0444);
return r;
}
-static inline int __vmx_find_uret_msr(struct vcpu_vmx *vmx, u32 msr)
-{
- int i;
-
- for (i = 0; i < vmx->nr_uret_msrs; ++i)
- if (vmx_uret_msrs_list[vmx->guest_uret_msrs[i].slot] == msr)
- return i;
- return -1;
-}
-
struct vmx_uret_msr *vmx_find_uret_msr(struct vcpu_vmx *vmx, u32 msr)
{
int i;
- i = __vmx_find_uret_msr(vmx, msr);
+ i = kvm_find_user_return_msr(msr);
if (i >= 0)
return &vmx->guest_uret_msrs[i];
return NULL;
static int vmx_set_guest_uret_msr(struct vcpu_vmx *vmx,
struct vmx_uret_msr *msr, u64 data)
{
+ unsigned int slot = msr - vmx->guest_uret_msrs;
int ret = 0;
u64 old_msr_data = msr->data;
msr->data = data;
- if (msr - vmx->guest_uret_msrs < vmx->nr_active_uret_msrs) {
+ if (msr->load_into_hardware) {
preempt_disable();
- ret = kvm_set_user_return_msr(msr->slot, msr->data, msr->mask);
+ ret = kvm_set_user_return_msr(slot, msr->data, msr->mask);
preempt_enable();
if (ret)
msr->data = old_msr_data;
return false;
}
- i = __vmx_find_uret_msr(vmx, MSR_EFER);
+ i = kvm_find_user_return_msr(MSR_EFER);
if (i < 0)
return false;
*/
if (!vmx->guest_uret_msrs_loaded) {
vmx->guest_uret_msrs_loaded = true;
- for (i = 0; i < vmx->nr_active_uret_msrs; ++i)
- kvm_set_user_return_msr(vmx->guest_uret_msrs[i].slot,
+ for (i = 0; i < kvm_nr_uret_msrs; ++i) {
+ if (!vmx->guest_uret_msrs[i].load_into_hardware)
+ continue;
+
+ kvm_set_user_return_msr(i,
vmx->guest_uret_msrs[i].data,
vmx->guest_uret_msrs[i].mask);
-
+ }
}
if (vmx->nested.need_vmcs12_to_shadow_sync)
vmx_clear_hlt(vcpu);
}
-static void vmx_setup_uret_msr(struct vcpu_vmx *vmx, unsigned int msr)
+static void vmx_setup_uret_msr(struct vcpu_vmx *vmx, unsigned int msr,
+ bool load_into_hardware)
{
- struct vmx_uret_msr tmp;
- int from, to;
+ struct vmx_uret_msr *uret_msr;
- from = __vmx_find_uret_msr(vmx, msr);
- if (from < 0)
+ uret_msr = vmx_find_uret_msr(vmx, msr);
+ if (!uret_msr)
return;
- to = vmx->nr_active_uret_msrs++;
- tmp = vmx->guest_uret_msrs[to];
- vmx->guest_uret_msrs[to] = vmx->guest_uret_msrs[from];
- vmx->guest_uret_msrs[from] = tmp;
+ uret_msr->load_into_hardware = load_into_hardware;
}
/*
*/
static void setup_msrs(struct vcpu_vmx *vmx)
{
- vmx->guest_uret_msrs_loaded = false;
- vmx->nr_active_uret_msrs = 0;
#ifdef CONFIG_X86_64
+ bool load_syscall_msrs;
+
/*
* The SYSCALL MSRs are only needed on long mode guests, and only
* when EFER.SCE is set.
*/
- if (is_long_mode(&vmx->vcpu) && (vmx->vcpu.arch.efer & EFER_SCE)) {
- vmx_setup_uret_msr(vmx, MSR_STAR);
- vmx_setup_uret_msr(vmx, MSR_LSTAR);
- vmx_setup_uret_msr(vmx, MSR_SYSCALL_MASK);
- }
+ load_syscall_msrs = is_long_mode(&vmx->vcpu) &&
+ (vmx->vcpu.arch.efer & EFER_SCE);
+
+ vmx_setup_uret_msr(vmx, MSR_STAR, load_syscall_msrs);
+ vmx_setup_uret_msr(vmx, MSR_LSTAR, load_syscall_msrs);
+ vmx_setup_uret_msr(vmx, MSR_SYSCALL_MASK, load_syscall_msrs);
#endif
- if (update_transition_efer(vmx))
- vmx_setup_uret_msr(vmx, MSR_EFER);
+ vmx_setup_uret_msr(vmx, MSR_EFER, update_transition_efer(vmx));
- if (guest_cpuid_has(&vmx->vcpu, X86_FEATURE_RDTSCP))
- vmx_setup_uret_msr(vmx, MSR_TSC_AUX);
+ vmx_setup_uret_msr(vmx, MSR_TSC_AUX,
+ guest_cpuid_has(&vmx->vcpu, X86_FEATURE_RDTSCP) ||
+ guest_cpuid_has(&vmx->vcpu, X86_FEATURE_RDPID));
- vmx_setup_uret_msr(vmx, MSR_IA32_TSX_CTRL);
+ /*
+ * hle=0, rtm=0, tsx_ctrl=1 can be found with some combinations of new
+ * kernel and old userspace. If those guests run on a tsx=off host, do
+ * allow guests to use TSX_CTRL, but don't change the value in hardware
+ * so that TSX remains always disabled.
+ */
+ vmx_setup_uret_msr(vmx, MSR_IA32_TSX_CTRL, boot_cpu_has(X86_FEATURE_RTM));
if (cpu_has_vmx_msr_bitmap())
vmx_update_msr_bitmap(&vmx->vcpu);
+
+ /*
+ * The set of MSRs to load may have changed, reload MSRs before the
+ * next VM-Enter.
+ */
+ vmx->guest_uret_msrs_loaded = false;
}
static u64 vmx_write_l1_tsc_offset(struct kvm_vcpu *vcpu, u64 offset)
else
msr_info->data = vmx->pt_desc.guest.addr_a[index / 2];
break;
- case MSR_TSC_AUX:
- if (!msr_info->host_initiated &&
- !guest_cpuid_has(vcpu, X86_FEATURE_RDTSCP))
- return 1;
- goto find_uret_msr;
case MSR_IA32_DEBUGCTLMSR:
msr_info->data = vmcs_read64(GUEST_IA32_DEBUGCTL);
break;
if (!intel_pmu_lbr_is_enabled(vcpu))
debugctl &= ~DEBUGCTLMSR_LBR_MASK;
+ if (!guest_cpuid_has(vcpu, X86_FEATURE_BUS_LOCK_DETECT))
+ debugctl &= ~DEBUGCTLMSR_BUS_LOCK_DETECT;
+
return debugctl;
}
else
vmx->pt_desc.guest.addr_a[index / 2] = data;
break;
- case MSR_TSC_AUX:
- if (!msr_info->host_initiated &&
- !guest_cpuid_has(vcpu, X86_FEATURE_RDTSCP))
- return 1;
- /* Check reserved bit, higher 32 bits should be zero */
- if ((data >> 32) != 0)
- return 1;
- goto find_uret_msr;
case MSR_IA32_PERF_CAPABILITIES:
if (data && !vcpu_to_pmu(vcpu)->version)
return 1;
xsaves_enabled, false);
}
- vmx_adjust_sec_exec_feature(vmx, &exec_control, rdtscp, RDTSCP);
+ /*
+ * RDPID is also gated by ENABLE_RDTSCP, turn on the control if either
+ * feature is exposed to the guest. This creates a virtualization hole
+ * if both are supported in hardware but only one is exposed to the
+ * guest, but letting the guest execute RDTSCP or RDPID when either one
+ * is advertised is preferable to emulating the advertised instruction
+ * in KVM on #UD, and obviously better than incorrectly injecting #UD.
+ */
+ if (cpu_has_vmx_rdtscp()) {
+ bool rdpid_or_rdtscp_enabled =
+ guest_cpuid_has(vcpu, X86_FEATURE_RDTSCP) ||
+ guest_cpuid_has(vcpu, X86_FEATURE_RDPID);
+
+ vmx_adjust_secondary_exec_control(vmx, &exec_control,
+ SECONDARY_EXEC_ENABLE_RDTSCP,
+ rdpid_or_rdtscp_enabled, false);
+ }
vmx_adjust_sec_exec_feature(vmx, &exec_control, invpcid, INVPCID);
vmx_adjust_sec_exec_exiting(vmx, &exec_control, rdrand, RDRAND);
struct vcpu_vmx *vmx = to_vmx(vcpu);
struct kvm_run *kvm_run = vcpu->run;
u32 intr_info, ex_no, error_code;
- unsigned long cr2, rip, dr6;
+ unsigned long cr2, dr6;
u32 vect_info;
vect_info = vmx->idt_vectoring_info;
vmx->vcpu.arch.event_exit_inst_len =
vmcs_read32(VM_EXIT_INSTRUCTION_LEN);
kvm_run->exit_reason = KVM_EXIT_DEBUG;
- rip = kvm_rip_read(vcpu);
- kvm_run->debug.arch.pc = vmcs_readl(GUEST_CS_BASE) + rip;
+ kvm_run->debug.arch.pc = kvm_get_linear_rip(vcpu);
kvm_run->debug.arch.exception = ex_no;
break;
case AC_VECTOR:
switch (kvm_get_apic_mode(vcpu)) {
case LAPIC_MODE_INVALID:
WARN_ONCE(true, "Invalid local APIC state");
+ break;
case LAPIC_MODE_DISABLED:
break;
case LAPIC_MODE_XAPIC:
void vmx_do_interrupt_nmi_irqoff(unsigned long entry);
-static void handle_interrupt_nmi_irqoff(struct kvm_vcpu *vcpu, u32 intr_info)
+static void handle_interrupt_nmi_irqoff(struct kvm_vcpu *vcpu,
+ unsigned long entry)
{
- unsigned int vector = intr_info & INTR_INFO_VECTOR_MASK;
- gate_desc *desc = (gate_desc *)host_idt_base + vector;
-
kvm_before_interrupt(vcpu);
- vmx_do_interrupt_nmi_irqoff(gate_offset(desc));
+ vmx_do_interrupt_nmi_irqoff(entry);
kvm_after_interrupt(vcpu);
}
static void handle_exception_nmi_irqoff(struct vcpu_vmx *vmx)
{
+ const unsigned long nmi_entry = (unsigned long)asm_exc_nmi_noist;
u32 intr_info = vmx_get_intr_info(&vmx->vcpu);
/* if exit due to PF check for async PF */
kvm_machine_check();
/* We need to handle NMIs before interrupts are enabled */
else if (is_nmi(intr_info))
- handle_interrupt_nmi_irqoff(&vmx->vcpu, intr_info);
+ handle_interrupt_nmi_irqoff(&vmx->vcpu, nmi_entry);
}
static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu)
{
u32 intr_info = vmx_get_intr_info(vcpu);
+ unsigned int vector = intr_info & INTR_INFO_VECTOR_MASK;
+ gate_desc *desc = (gate_desc *)host_idt_base + vector;
if (WARN_ONCE(!is_external_intr(intr_info),
"KVM: unexpected VM-Exit interrupt info: 0x%x", intr_info))
return;
- handle_interrupt_nmi_irqoff(vcpu, intr_info);
+ handle_interrupt_nmi_irqoff(vcpu, gate_offset(desc));
}
static void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu)
static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu,
struct vcpu_vmx *vmx)
{
- /*
- * VMENTER enables interrupts (host state), but the kernel state is
- * interrupts disabled when this is invoked. Also tell RCU about
- * it. This is the same logic as for exit_to_user_mode().
- *
- * This ensures that e.g. latency analysis on the host observes
- * guest mode as interrupt enabled.
- *
- * guest_enter_irqoff() informs context tracking about the
- * transition to guest mode and if enabled adjusts RCU state
- * accordingly.
- */
- instrumentation_begin();
- trace_hardirqs_on_prepare();
- lockdep_hardirqs_on_prepare(CALLER_ADDR0);
- instrumentation_end();
-
- guest_enter_irqoff();
- lockdep_hardirqs_on(CALLER_ADDR0);
+ kvm_guest_enter_irqoff();
/* L1D Flush includes CPU buffer clear to mitigate MDS */
if (static_branch_unlikely(&vmx_l1d_should_flush))
vcpu->arch.cr2 = native_read_cr2();
- /*
- * VMEXIT disables interrupts (host state), but tracing and lockdep
- * have them in state 'on' as recorded before entering guest mode.
- * Same as enter_from_user_mode().
- *
- * guest_exit_irqoff() restores host context and reinstates RCU if
- * enabled and required.
- *
- * This needs to be done before the below as native_read_msr()
- * contains a tracepoint and x86_spec_ctrl_restore_host() calls
- * into world and some more.
- */
- lockdep_hardirqs_off(CALLER_ADDR0);
- guest_exit_irqoff();
-
- instrumentation_begin();
- trace_hardirqs_off_finish();
- instrumentation_end();
+ kvm_guest_exit_irqoff();
}
static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu)
static int vmx_create_vcpu(struct kvm_vcpu *vcpu)
{
+ struct vmx_uret_msr *tsx_ctrl;
struct vcpu_vmx *vmx;
int i, cpu, err;
goto free_vpid;
}
- BUILD_BUG_ON(ARRAY_SIZE(vmx_uret_msrs_list) != MAX_NR_USER_RETURN_MSRS);
-
- for (i = 0; i < ARRAY_SIZE(vmx_uret_msrs_list); ++i) {
- u32 index = vmx_uret_msrs_list[i];
- u32 data_low, data_high;
- int j = vmx->nr_uret_msrs;
-
- if (rdmsr_safe(index, &data_low, &data_high) < 0)
- continue;
- if (wrmsr_safe(index, data_low, data_high) < 0)
- continue;
-
- vmx->guest_uret_msrs[j].slot = i;
- vmx->guest_uret_msrs[j].data = 0;
- switch (index) {
- case MSR_IA32_TSX_CTRL:
- /*
- * TSX_CTRL_CPUID_CLEAR is handled in the CPUID
- * interception. Keep the host value unchanged to avoid
- * changing CPUID bits under the host kernel's feet.
- *
- * hle=0, rtm=0, tsx_ctrl=1 can be found with some
- * combinations of new kernel and old userspace. If
- * those guests run on a tsx=off host, do allow guests
- * to use TSX_CTRL, but do not change the value on the
- * host so that TSX remains always disabled.
- */
- if (boot_cpu_has(X86_FEATURE_RTM))
- vmx->guest_uret_msrs[j].mask = ~(u64)TSX_CTRL_CPUID_CLEAR;
- else
- vmx->guest_uret_msrs[j].mask = 0;
- break;
- default:
- vmx->guest_uret_msrs[j].mask = -1ull;
- break;
- }
- ++vmx->nr_uret_msrs;
+ for (i = 0; i < kvm_nr_uret_msrs; ++i) {
+ vmx->guest_uret_msrs[i].data = 0;
+ vmx->guest_uret_msrs[i].mask = -1ull;
+ }
+ if (boot_cpu_has(X86_FEATURE_RTM)) {
+ /*
+ * TSX_CTRL_CPUID_CLEAR is handled in the CPUID interception.
+ * Keep the host value unchanged to avoid changing CPUID bits
+ * under the host kernel's feet.
+ */
+ tsx_ctrl = vmx_find_uret_msr(vmx, MSR_IA32_TSX_CTRL);
+ if (tsx_ctrl)
+ vmx->guest_uret_msrs[i].mask = ~(u64)TSX_CTRL_CPUID_CLEAR;
}
err = alloc_loaded_vmcs(&vmx->vmcs01);
if (!cpu_has_vmx_xsaves())
kvm_cpu_cap_clear(X86_FEATURE_XSAVES);
- /* CPUID 0x80000001 */
- if (!cpu_has_vmx_rdtscp())
+ /* CPUID 0x80000001 and 0x7 (RDPID) */
+ if (!cpu_has_vmx_rdtscp()) {
kvm_cpu_cap_clear(X86_FEATURE_RDTSCP);
+ kvm_cpu_cap_clear(X86_FEATURE_RDPID);
+ }
if (cpu_has_vmx_waitpkg())
kvm_cpu_cap_check_and_set(X86_FEATURE_WAITPKG);
/*
* RDPID causes #UD if disabled through secondary execution controls.
* Because it is marked as EmulateOnUD, we need to intercept it here.
+ * Note, RDPID is hidden behind ENABLE_RDTSCP.
*/
- case x86_intercept_rdtscp:
+ case x86_intercept_rdpid:
if (!nested_cpu_has2(vmcs12, SECONDARY_EXEC_ENABLE_RDTSCP)) {
exception->vector = UD_VECTOR;
exception->error_code_valid = false;
.nested_ops = &vmx_nested_ops,
.update_pi_irte = pi_update_irte,
+ .start_assignment = vmx_pi_start_assignment,
#ifdef CONFIG_X86_64
.set_hv_timer = vmx_set_hv_timer,
.vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector,
};
+static __init void vmx_setup_user_return_msrs(void)
+{
+
+ /*
+ * Though SYSCALL is only supported in 64-bit mode on Intel CPUs, kvm
+ * will emulate SYSCALL in legacy mode if the vendor string in guest
+ * CPUID.0:{EBX,ECX,EDX} is "AuthenticAMD" or "AMDisbetter!" To
+ * support this emulation, MSR_STAR is included in the list for i386,
+ * but is never loaded into hardware. MSR_CSTAR is also never loaded
+ * into hardware and is here purely for emulation purposes.
+ */
+ const u32 vmx_uret_msrs_list[] = {
+ #ifdef CONFIG_X86_64
+ MSR_SYSCALL_MASK, MSR_LSTAR, MSR_CSTAR,
+ #endif
+ MSR_EFER, MSR_TSC_AUX, MSR_STAR,
+ MSR_IA32_TSX_CTRL,
+ };
+ int i;
+
+ BUILD_BUG_ON(ARRAY_SIZE(vmx_uret_msrs_list) != MAX_NR_USER_RETURN_MSRS);
+
+ for (i = 0; i < ARRAY_SIZE(vmx_uret_msrs_list); ++i)
+ kvm_add_user_return_msr(vmx_uret_msrs_list[i]);
+}
+
static __init int hardware_setup(void)
{
unsigned long host_bndcfgs;
struct desc_ptr dt;
- int r, i, ept_lpage_level;
+ int r, ept_lpage_level;
store_idt(&dt);
host_idt_base = dt.address;
- for (i = 0; i < ARRAY_SIZE(vmx_uret_msrs_list); ++i)
- kvm_define_user_return_msr(i, vmx_uret_msrs_list[i]);
+ vmx_setup_user_return_msrs();
if (setup_vmcs_config(&vmcs_config, &vmx_capability) < 0)
return -EIO;
};
struct vmx_uret_msr {
- unsigned int slot; /* The MSR's slot in kvm_user_return_msrs. */
+ bool load_into_hardware;
u64 data;
u64 mask;
};
u32 idt_vectoring_info;
ulong rflags;
+ /*
+ * User return MSRs are always emulated when enabled in the guest, but
+ * only loaded into hardware when necessary, e.g. SYSCALL #UDs outside
+ * of 64-bit mode or if EFER.SCE=1, thus the SYSCALL MSRs don't need to
+ * be loaded into hardware if those conditions aren't met.
+ * nr_active_uret_msrs tracks the number of MSRs that need to be loaded
+ * into hardware when running the guest. guest_uret_msrs[] is resorted
+ * whenever the number of "active" uret MSRs is modified.
+ */
struct vmx_uret_msr guest_uret_msrs[MAX_NR_USER_RETURN_MSRS];
- int nr_uret_msrs;
int nr_active_uret_msrs;
bool guest_uret_msrs_loaded;
#ifdef CONFIG_X86_64
*/
#define KVM_MAX_NR_USER_RETURN_MSRS 16
-struct kvm_user_return_msrs_global {
- int nr;
- u32 msrs[KVM_MAX_NR_USER_RETURN_MSRS];
-};
-
struct kvm_user_return_msrs {
struct user_return_notifier urn;
bool registered;
} values[KVM_MAX_NR_USER_RETURN_MSRS];
};
-static struct kvm_user_return_msrs_global __read_mostly user_return_msrs_global;
+u32 __read_mostly kvm_nr_uret_msrs;
+EXPORT_SYMBOL_GPL(kvm_nr_uret_msrs);
+static u32 __read_mostly kvm_uret_msrs_list[KVM_MAX_NR_USER_RETURN_MSRS];
static struct kvm_user_return_msrs __percpu *user_return_msrs;
#define KVM_SUPPORTED_XCR0 (XFEATURE_MASK_FP | XFEATURE_MASK_SSE \
user_return_notifier_unregister(urn);
}
local_irq_restore(flags);
- for (slot = 0; slot < user_return_msrs_global.nr; ++slot) {
+ for (slot = 0; slot < kvm_nr_uret_msrs; ++slot) {
values = &msrs->values[slot];
if (values->host != values->curr) {
- wrmsrl(user_return_msrs_global.msrs[slot], values->host);
+ wrmsrl(kvm_uret_msrs_list[slot], values->host);
values->curr = values->host;
}
}
}
-void kvm_define_user_return_msr(unsigned slot, u32 msr)
+static int kvm_probe_user_return_msr(u32 msr)
+{
+ u64 val;
+ int ret;
+
+ preempt_disable();
+ ret = rdmsrl_safe(msr, &val);
+ if (ret)
+ goto out;
+ ret = wrmsrl_safe(msr, val);
+out:
+ preempt_enable();
+ return ret;
+}
+
+int kvm_add_user_return_msr(u32 msr)
{
- BUG_ON(slot >= KVM_MAX_NR_USER_RETURN_MSRS);
- user_return_msrs_global.msrs[slot] = msr;
- if (slot >= user_return_msrs_global.nr)
- user_return_msrs_global.nr = slot + 1;
+ BUG_ON(kvm_nr_uret_msrs >= KVM_MAX_NR_USER_RETURN_MSRS);
+
+ if (kvm_probe_user_return_msr(msr))
+ return -1;
+
+ kvm_uret_msrs_list[kvm_nr_uret_msrs] = msr;
+ return kvm_nr_uret_msrs++;
+}
+EXPORT_SYMBOL_GPL(kvm_add_user_return_msr);
+
+int kvm_find_user_return_msr(u32 msr)
+{
+ int i;
+
+ for (i = 0; i < kvm_nr_uret_msrs; ++i) {
+ if (kvm_uret_msrs_list[i] == msr)
+ return i;
+ }
+ return -1;
}
-EXPORT_SYMBOL_GPL(kvm_define_user_return_msr);
+EXPORT_SYMBOL_GPL(kvm_find_user_return_msr);
static void kvm_user_return_msr_cpu_online(void)
{
u64 value;
int i;
- for (i = 0; i < user_return_msrs_global.nr; ++i) {
- rdmsrl_safe(user_return_msrs_global.msrs[i], &value);
+ for (i = 0; i < kvm_nr_uret_msrs; ++i) {
+ rdmsrl_safe(kvm_uret_msrs_list[i], &value);
msrs->values[i].host = value;
msrs->values[i].curr = value;
}
value = (value & mask) | (msrs->values[slot].host & ~mask);
if (value == msrs->values[slot].curr)
return 0;
- err = wrmsrl_safe(user_return_msrs_global.msrs[slot], value);
+ err = wrmsrl_safe(kvm_uret_msrs_list[slot], value);
if (err)
return 1;
if (!guest_cpuid_has(vcpu, X86_FEATURE_RTM))
fixed |= DR6_RTM;
+
+ if (!guest_cpuid_has(vcpu, X86_FEATURE_BUS_LOCK_DETECT))
+ fixed |= DR6_BUS_LOCK;
return fixed;
}
* invokes 64-bit SYSENTER.
*/
data = get_canonical(data, vcpu_virt_addr_bits(vcpu));
+ break;
+ case MSR_TSC_AUX:
+ if (!kvm_is_supported_user_return_msr(MSR_TSC_AUX))
+ return 1;
+
+ if (!host_initiated &&
+ !guest_cpuid_has(vcpu, X86_FEATURE_RDTSCP) &&
+ !guest_cpuid_has(vcpu, X86_FEATURE_RDPID))
+ return 1;
+
+ /*
+ * Per Intel's SDM, bits 63:32 are reserved, but AMD's APM has
+ * incomplete and conflicting architectural behavior. Current
+ * AMD CPUs completely ignore bits 63:32, i.e. they aren't
+ * reserved and always read as zeros. Enforce Intel's reserved
+ * bits check if and only if the guest CPU is Intel, and clear
+ * the bits in all other cases. This ensures cross-vendor
+ * migration will provide consistent behavior for the guest.
+ */
+ if (guest_cpuid_is_intel(vcpu) && (data >> 32) != 0)
+ return 1;
+
+ data = (u32)data;
+ break;
}
msr.data = data;
if (!host_initiated && !kvm_msr_allowed(vcpu, index, KVM_MSR_FILTER_READ))
return KVM_MSR_RET_FILTERED;
+ switch (index) {
+ case MSR_TSC_AUX:
+ if (!kvm_is_supported_user_return_msr(MSR_TSC_AUX))
+ return 1;
+
+ if (!host_initiated &&
+ !guest_cpuid_has(vcpu, X86_FEATURE_RDTSCP) &&
+ !guest_cpuid_has(vcpu, X86_FEATURE_RDPID))
+ return 1;
+ break;
+ }
+
msr.index = index;
msr.host_initiated = host_initiated;
static void kvm_vcpu_flush_tlb_guest(struct kvm_vcpu *vcpu)
{
++vcpu->stat.tlb_flush;
+
+ if (!tdp_enabled) {
+ /*
+ * A TLB flush on behalf of the guest is equivalent to
+ * INVPCID(all), toggling CR4.PGE, etc., which requires
+ * a forced sync of the shadow page tables. Unload the
+ * entire MMU here and the subsequent load will sync the
+ * shadow page tables, and also flush the TLB.
+ */
+ kvm_mmu_unload(vcpu);
+ return;
+ }
+
static_call(kvm_x86_tlb_flush_guest)(vcpu);
}
* expensive IPIs.
*/
if (guest_pv_has(vcpu, KVM_FEATURE_PV_TLB_FLUSH)) {
+ u8 st_preempted = xchg(&st->preempted, 0);
+
trace_kvm_pv_tlb_flush(vcpu->vcpu_id,
- st->preempted & KVM_VCPU_FLUSH_TLB);
- if (xchg(&st->preempted, 0) & KVM_VCPU_FLUSH_TLB)
+ st_preempted & KVM_VCPU_FLUSH_TLB);
+ if (st_preempted & KVM_VCPU_FLUSH_TLB)
kvm_vcpu_flush_tlb_guest(vcpu);
+ } else {
+ st->preempted = 0;
}
vcpu->arch.st.preempted = 0;
case MSR_IA32_LASTBRANCHTOIP:
case MSR_IA32_LASTINTFROMIP:
case MSR_IA32_LASTINTTOIP:
- case MSR_K8_SYSCFG:
+ case MSR_AMD64_SYSCFG:
case MSR_K8_TSEG_ADDR:
case MSR_K8_TSEG_MASK:
case MSR_VM_HSAVE_PA:
static int kvm_add_msr_filter(struct kvm_x86_msr_filter *msr_filter,
struct kvm_msr_filter_range *user_range)
{
- struct msr_bitmap_range range;
unsigned long *bitmap = NULL;
size_t bitmap_size;
- int r;
if (!user_range->nmsrs)
return 0;
+ if (user_range->flags & ~(KVM_MSR_FILTER_READ | KVM_MSR_FILTER_WRITE))
+ return -EINVAL;
+
+ if (!user_range->flags)
+ return -EINVAL;
+
bitmap_size = BITS_TO_LONGS(user_range->nmsrs) * sizeof(long);
if (!bitmap_size || bitmap_size > KVM_MSR_FILTER_MAX_BITMAP_SIZE)
return -EINVAL;
if (IS_ERR(bitmap))
return PTR_ERR(bitmap);
- range = (struct msr_bitmap_range) {
+ msr_filter->ranges[msr_filter->count] = (struct msr_bitmap_range) {
.flags = user_range->flags,
.base = user_range->base,
.nmsrs = user_range->nmsrs,
.bitmap = bitmap,
};
- if (range.flags & ~(KVM_MSR_FILTER_READ | KVM_MSR_FILTER_WRITE)) {
- r = -EINVAL;
- goto err;
- }
-
- if (!range.flags) {
- r = -EINVAL;
- goto err;
- }
-
- /* Everything ok, add this range identifier. */
- msr_filter->ranges[msr_filter->count] = range;
msr_filter->count++;
-
return 0;
-err:
- kfree(bitmap);
- return r;
}
static int kvm_vm_ioctl_set_msr_filter(struct kvm *kvm, void __user *argp)
continue;
break;
case MSR_TSC_AUX:
- if (!kvm_cpu_cap_has(X86_FEATURE_RDTSCP))
+ if (!kvm_cpu_cap_has(X86_FEATURE_RDTSCP) &&
+ !kvm_cpu_cap_has(X86_FEATURE_RDPID))
continue;
break;
case MSR_IA32_UMWAIT_CONTROL:
static void emulator_set_hflags(struct x86_emulate_ctxt *ctxt, unsigned emul_flags)
{
- emul_to_vcpu(ctxt)->arch.hflags = emul_flags;
+ struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
+
+ vcpu->arch.hflags = emul_flags;
+ kvm_mmu_reset_context(vcpu);
}
static int emulator_pre_leave_smm(struct x86_emulate_ctxt *ctxt,
BUILD_BUG_ON(HF_SMM_MASK != X86EMUL_SMM_MASK);
BUILD_BUG_ON(HF_SMM_INSIDE_NMI_MASK != X86EMUL_SMM_INSIDE_NMI_MASK);
+ ctxt->interruptibility = 0;
+ ctxt->have_exception = false;
+ ctxt->exception.vector = -1;
+ ctxt->perm_ok = false;
+
init_decode_cache(ctxt);
vcpu->arch.emulate_regs_need_sync_from_vcpu = false;
}
kvm_vcpu_check_breakpoint(vcpu, &r))
return r;
- ctxt->interruptibility = 0;
- ctxt->have_exception = false;
- ctxt->exception.vector = -1;
- ctxt->perm_ok = false;
-
- ctxt->ud = emulation_type & EMULTYPE_TRAP_UD;
-
- r = x86_decode_insn(ctxt, insn, insn_len);
+ r = x86_decode_insn(ctxt, insn, insn_len, emulation_type);
trace_kvm_emulate_insn_start(vcpu);
++vcpu->stat.insn_emulation;
static DECLARE_WORK(pvclock_gtod_work, pvclock_gtod_update_fn);
/*
+ * Indirection to move queue_work() out of the tk_core.seq write held
+ * region to prevent possible deadlocks against time accessors which
+ * are invoked with work related locks held.
+ */
+static void pvclock_irq_work_fn(struct irq_work *w)
+{
+ queue_work(system_long_wq, &pvclock_gtod_work);
+}
+
+static DEFINE_IRQ_WORK(pvclock_irq_work, pvclock_irq_work_fn);
+
+/*
* Notification about pvclock gtod data update.
*/
static int pvclock_gtod_notify(struct notifier_block *nb, unsigned long unused,
update_pvclock_gtod(tk);
- /* disable master clock if host does not trust, or does not
- * use, TSC based clocksource.
+ /*
+ * Disable master clock if host does not trust, or does not use,
+ * TSC based clocksource. Delegate queue_work() to irq_work as
+ * this is invoked with tk_core.seq write held.
*/
if (!gtod_is_based_on_tsc(gtod->clock.vclock_mode) &&
atomic_read(&kvm_guest_has_master_clock) != 0)
- queue_work(system_long_wq, &pvclock_gtod_work);
-
+ irq_work_queue(&pvclock_irq_work);
return 0;
}
printk(KERN_ERR "kvm: failed to allocate percpu kvm_user_return_msrs\n");
goto out_free_x86_emulator_cache;
}
+ kvm_nr_uret_msrs = 0;
r = kvm_mmu_module_init();
if (r)
cpuhp_remove_state_nocalls(CPUHP_AP_X86_KVM_CLK_ONLINE);
#ifdef CONFIG_X86_64
pvclock_gtod_unregister_notifier(&pvclock_gtod_notifier);
+ irq_work_sync(&pvclock_irq_work);
+ cancel_work_sync(&pvclock_gtod_work);
#endif
kvm_x86_ops.hardware_enable = NULL;
kvm_mmu_module_exit();
free_percpu(user_return_msrs);
+ kmem_cache_destroy(x86_emulator_cache);
kmem_cache_destroy(x86_fpu_cache);
#ifdef CONFIG_KVM_XEN
static_key_deferred_flush(&kvm_xen_enabled);
vcpu->stat.directed_yield_attempted++;
+ if (single_task_running())
+ goto no_yield;
+
rcu_read_lock();
map = rcu_dereference(vcpu->kvm->arch.apic_map);
local_irq_disable();
kvm_after_interrupt(vcpu);
+ /*
+ * Wait until after servicing IRQs to account guest time so that any
+ * ticks that occurred while running the guest are properly accounted
+ * to the guest. Waiting until IRQs are enabled degrades the accuracy
+ * of accounting via context tracking, but the loss of accuracy is
+ * acceptable for all known use cases.
+ */
+ vtime_account_guest_exit();
+
if (lapic_in_kernel(vcpu)) {
s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta;
if (delta != S64_MIN) {
if (r <= 0)
break;
- kvm_clear_request(KVM_REQ_PENDING_TIMER, vcpu);
+ kvm_clear_request(KVM_REQ_UNBLOCK, vcpu);
if (kvm_cpu_has_pending_timer(vcpu))
kvm_inject_pending_timer_irqs(vcpu);
kvm_update_dr7(vcpu);
if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP)
- vcpu->arch.singlestep_rip = kvm_rip_read(vcpu) +
- get_segment_base(vcpu, VCPU_SREG_CS);
+ vcpu->arch.singlestep_rip = kvm_get_linear_rip(vcpu);
/*
* Trigger an rflags update that will inject or remove the trace
void kvm_arch_start_assignment(struct kvm *kvm)
{
- atomic_inc(&kvm->arch.assigned_device_count);
+ if (atomic_inc_return(&kvm->arch.assigned_device_count) == 1)
+ static_call_cond(kvm_x86_start_assignment)(kvm);
}
EXPORT_SYMBOL_GPL(kvm_arch_start_assignment);
#include "kvm_cache_regs.h"
#include "kvm_emulate.h"
+static __always_inline void kvm_guest_enter_irqoff(void)
+{
+ /*
+ * VMENTER enables interrupts (host state), but the kernel state is
+ * interrupts disabled when this is invoked. Also tell RCU about
+ * it. This is the same logic as for exit_to_user_mode().
+ *
+ * This ensures that e.g. latency analysis on the host observes
+ * guest mode as interrupt enabled.
+ *
+ * guest_enter_irqoff() informs context tracking about the
+ * transition to guest mode and if enabled adjusts RCU state
+ * accordingly.
+ */
+ instrumentation_begin();
+ trace_hardirqs_on_prepare();
+ lockdep_hardirqs_on_prepare(CALLER_ADDR0);
+ instrumentation_end();
+
+ guest_enter_irqoff();
+ lockdep_hardirqs_on(CALLER_ADDR0);
+}
+
+static __always_inline void kvm_guest_exit_irqoff(void)
+{
+ /*
+ * VMEXIT disables interrupts (host state), but tracing and lockdep
+ * have them in state 'on' as recorded before entering guest mode.
+ * Same as enter_from_user_mode().
+ *
+ * context_tracking_guest_exit() restores host context and reinstates
+ * RCU if enabled and required.
+ *
+ * This needs to be done immediately after VM-Exit, before any code
+ * that might contain tracepoints or call out to the greater world,
+ * e.g. before x86_spec_ctrl_restore_host().
+ */
+ lockdep_hardirqs_off(CALLER_ADDR0);
+ context_tracking_guest_exit();
+
+ instrumentation_begin();
+ trace_hardirqs_off_finish();
+ instrumentation_end();
+}
+
#define KVM_NESTED_VMENTER_CONSISTENCY_CHECK(consistency_check) \
({ \
bool failed = (consistency_check); \
#include <xen/xen.h>
#include <asm/fpu/internal.h>
-#include <asm/sev-es.h>
+#include <asm/sev.h>
#include <asm/traps.h>
#include <asm/kdebug.h>
if (si_code == SEGV_PKUERR)
force_sig_pkuerr((void __user *)address, pkey);
-
- force_sig_fault(SIGSEGV, si_code, (void __user *)address);
+ else
+ force_sig_fault(SIGSEGV, si_code, (void __user *)address);
local_irq_disable();
}
#define AMD_SME_BIT BIT(0)
#define AMD_SEV_BIT BIT(1)
- /* Check the SEV MSR whether SEV or SME is enabled */
- sev_status = __rdmsr(MSR_AMD64_SEV);
- feature_mask = (sev_status & MSR_AMD64_SEV_ENABLED) ? AMD_SEV_BIT : AMD_SME_BIT;
-
/*
* Check for the SME/SEV feature:
* CPUID Fn8000_001F[EAX]
eax = 0x8000001f;
ecx = 0;
native_cpuid(&eax, &ebx, &ecx, &edx);
- if (!(eax & feature_mask))
+ /* Check whether SEV or SME is supported */
+ if (!(eax & (AMD_SEV_BIT | AMD_SME_BIT)))
return;
me_mask = 1UL << (ebx & 0x3f);
+ /* Check the SEV MSR whether SEV or SME is enabled */
+ sev_status = __rdmsr(MSR_AMD64_SEV);
+ feature_mask = (sev_status & MSR_AMD64_SEV_ENABLED) ? AMD_SEV_BIT : AMD_SME_BIT;
+
/* Check if memory encryption is enabled */
if (feature_mask == AMD_SME_BIT) {
/*
* No SME if Hypervisor bit is set. This check is here to
* prevent a guest from trying to enable SME. For running as a
- * KVM guest the MSR_K8_SYSCFG will be sufficient, but there
+ * KVM guest the MSR_AMD64_SYSCFG will be sufficient, but there
* might be other hypervisors which emulate that MSR as non-zero
* or even pass it through to the guest.
* A malicious hypervisor can still trick a guest into this
return;
/* For SME, check the SYSCFG MSR */
- msr = __rdmsr(MSR_K8_SYSCFG);
- if (!(msr & MSR_K8_SYSCFG_MEM_ENCRYPT))
+ msr = __rdmsr(MSR_AMD64_SYSCFG);
+ if (!(msr & MSR_AMD64_SYSCFG_MEM_ENCRYPT))
return;
} else {
/* SEV state cannot be controlled by a command line option */
/* need to take out [4G, TOM2) for RAM*/
/* SYS_CFG */
- address = MSR_K8_SYSCFG;
+ address = MSR_AMD64_SYSCFG;
rdmsrl(address, val);
/* TOP_MEM2 is enabled? */
if (val & (1<<21)) {
DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x15b1, pci_amd_enable_64bit_bar);
DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x1601, pci_amd_enable_64bit_bar);
+#define RS690_LOWER_TOP_OF_DRAM2 0x30
+#define RS690_LOWER_TOP_OF_DRAM2_VALID 0x1
+#define RS690_UPPER_TOP_OF_DRAM2 0x31
+#define RS690_HTIU_NB_INDEX 0xA8
+#define RS690_HTIU_NB_INDEX_WR_ENABLE 0x100
+#define RS690_HTIU_NB_DATA 0xAC
+
+/*
+ * Some BIOS implementations support RAM above 4GB, but do not configure the
+ * PCI host to respond to bus master accesses for these addresses. These
+ * implementations set the TOP_OF_DRAM_SLOT1 register correctly, so PCI DMA
+ * works as expected for addresses below 4GB.
+ *
+ * Reference: "AMD RS690 ASIC Family Register Reference Guide" (pg. 2-57)
+ * https://www.amd.com/system/files/TechDocs/43372_rs690_rrg_3.00o.pdf
+ */
+static void rs690_fix_64bit_dma(struct pci_dev *pdev)
+{
+ u32 val = 0;
+ phys_addr_t top_of_dram = __pa(high_memory - 1) + 1;
+
+ if (top_of_dram <= (1ULL << 32))
+ return;
+
+ pci_write_config_dword(pdev, RS690_HTIU_NB_INDEX,
+ RS690_LOWER_TOP_OF_DRAM2);
+ pci_read_config_dword(pdev, RS690_HTIU_NB_DATA, &val);
+
+ if (val)
+ return;
+
+ pci_info(pdev, "Adjusting top of DRAM to %pa for 64-bit DMA support\n", &top_of_dram);
+
+ pci_write_config_dword(pdev, RS690_HTIU_NB_INDEX,
+ RS690_UPPER_TOP_OF_DRAM2 | RS690_HTIU_NB_INDEX_WR_ENABLE);
+ pci_write_config_dword(pdev, RS690_HTIU_NB_DATA, top_of_dram >> 32);
+
+ pci_write_config_dword(pdev, RS690_HTIU_NB_INDEX,
+ RS690_LOWER_TOP_OF_DRAM2 | RS690_HTIU_NB_INDEX_WR_ENABLE);
+ pci_write_config_dword(pdev, RS690_HTIU_NB_DATA,
+ top_of_dram | RS690_LOWER_TOP_OF_DRAM2_VALID);
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x7910, rs690_fix_64bit_dma);
+
#endif
#include <asm/realmode.h>
#include <asm/time.h>
#include <asm/pgalloc.h>
-#include <asm/sev-es.h>
+#include <asm/sev.h>
/*
* We allocate runtime services regions top-down, starting from -4G, i.e.
size -= rm_size;
}
+ /*
+ * Don't free memory under 1M for two reasons:
+ * - BIOS might clobber it
+ * - Crash kernel needs it to be reserved
+ */
+ if (start + size < SZ_1M)
+ continue;
+ if (start < SZ_1M) {
+ size -= (SZ_1M - start);
+ start = SZ_1M;
+ }
+
memblock_free_late(start, size);
}
#include <asm/realmode.h>
#include <asm/tlbflush.h>
#include <asm/crash.h>
-#include <asm/sev-es.h>
+#include <asm/sev.h>
struct real_mode_header *real_mode_header;
u32 *trampoline_cr4_features;
/* Has to be under 1M so we can execute real-mode AP code. */
mem = memblock_find_in_range(0, 1<<20, size, PAGE_SIZE);
- if (!mem) {
+ if (!mem)
pr_info("No sub-1M memory is available for the trampoline\n");
- return;
- }
+ else
+ set_real_mode_mem(mem);
- memblock_reserve(mem, size);
- set_real_mode_mem(mem);
- crash_reserve_low_1M();
+ /*
+ * Unconditionally reserve the entire fisrt 1M, see comment in
+ * setup_arch().
+ */
+ memblock_reserve(0, SZ_1M);
}
static void sme_sev_setup_real_mode(struct trampoline_header *th)
*/
btl $TH_FLAGS_SME_ACTIVE_BIT, pa_tr_flags
jnc .Ldone
- movl $MSR_K8_SYSCFG, %ecx
+ movl $MSR_AMD64_SYSCFG, %ecx
rdmsr
- bts $MSR_K8_SYSCFG_MEM_ENCRYPT_BIT, %eax
+ bts $MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT, %eax
jc .Ldone
/*
/* Get mfn list */
xen_build_dynamic_phys_to_machine();
+ /* Work out if we support NX */
+ get_cpu_cap(&boot_cpu_data);
+ x86_configure_nx();
+
/*
* Set up kernel GDT and segment registers, mainly so that
* -fstack-protector code can be executed.
*/
xen_setup_gdt(0);
- /* Work out if we support NX */
- get_cpu_cap(&boot_cpu_data);
- x86_configure_nx();
-
/* Determine virtual and physical address sizes */
get_cpu_address_sizes(&boot_cpu_data);
440 common process_madvise sys_process_madvise
441 common epoll_pwait2 sys_epoll_pwait2
442 common mount_setattr sys_mount_setattr
-443 common quotactl_path sys_quotactl_path
+# 443 reserved for quotactl_path
444 common landlock_create_ruleset sys_landlock_create_ruleset
445 common landlock_add_rule sys_landlock_add_rule
446 common landlock_restrict_self sys_landlock_restrict_self
return bic->bfqq[is_sync];
}
+static void bfq_put_stable_ref(struct bfq_queue *bfqq);
+
void bic_set_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq, bool is_sync)
{
+ /*
+ * If bfqq != NULL, then a non-stable queue merge between
+ * bic->bfqq and bfqq is happening here. This causes troubles
+ * in the following case: bic->bfqq has also been scheduled
+ * for a possible stable merge with bic->stable_merge_bfqq,
+ * and bic->stable_merge_bfqq == bfqq happens to
+ * hold. Troubles occur because bfqq may then undergo a split,
+ * thereby becoming eligible for a stable merge. Yet, if
+ * bic->stable_merge_bfqq points exactly to bfqq, then bfqq
+ * would be stably merged with itself. To avoid this anomaly,
+ * we cancel the stable merge if
+ * bic->stable_merge_bfqq == bfqq.
+ */
bic->bfqq[is_sync] = bfqq;
+
+ if (bfqq && bic->stable_merge_bfqq == bfqq) {
+ /*
+ * Actually, these same instructions are executed also
+ * in bfq_setup_cooperator, in case of abort or actual
+ * execution of a stable merge. We could avoid
+ * repeating these instructions there too, but if we
+ * did so, we would nest even more complexity in this
+ * function.
+ */
+ bfq_put_stable_ref(bic->stable_merge_bfqq);
+
+ bic->stable_merge_bfqq = NULL;
+ }
}
struct bfq_data *bic_to_bfqd(struct bfq_io_cq *bic)
}
-static bool bfq_bio_merge(struct blk_mq_hw_ctx *hctx, struct bio *bio,
+static bool bfq_bio_merge(struct request_queue *q, struct bio *bio,
unsigned int nr_segs)
{
- struct request_queue *q = hctx->queue;
struct bfq_data *bfqd = q->elevator->elevator_data;
struct request *free = NULL;
/*
static bool idling_boosts_thr_without_issues(struct bfq_data *bfqd,
struct bfq_queue *bfqq);
-static void bfq_put_stable_ref(struct bfq_queue *bfqq);
-
/*
* Attempt to schedule a merge of bfqq with the currently in-service
* queue or with a close queue among the scheduled queues. Return
}
EXPORT_SYMBOL(bio_init);
-unsigned int bio_max_size(struct bio *bio)
-{
- struct block_device *bdev = bio->bi_bdev;
-
- return bdev ? bdev->bd_disk->queue->limits.bio_max_bytes : UINT_MAX;
-}
-
/**
* bio_reset - reinitialize a bio
* @bio: bio to reset
struct bio_vec *bv = &bio->bi_io_vec[bio->bi_vcnt - 1];
if (page_is_mergeable(bv, page, len, off, same_page)) {
- if (bio->bi_iter.bi_size > bio_max_size(bio) - len) {
+ if (bio->bi_iter.bi_size > UINT_MAX - len) {
*same_page = false;
return false;
}
{
unsigned short nr_pages = bio->bi_max_vecs - bio->bi_vcnt;
unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt;
- unsigned int bytes_left = bio_max_size(bio) - bio->bi_iter.bi_size;
struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt;
struct page **pages = (struct page **)bv;
bool same_page = false;
BUILD_BUG_ON(PAGE_PTRS_PER_BVEC < 2);
pages += entries_left * (PAGE_PTRS_PER_BVEC - 1);
- size = iov_iter_get_pages(iter, pages, bytes_left, nr_pages,
- &offset);
+ size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset);
if (unlikely(size <= 0))
return size ? size : -EFAULT;
lockdep_assert_held(&ioc->lock);
- inuse = clamp_t(u32, inuse, 1, active);
+ /*
+ * For an active leaf node, its inuse shouldn't be zero or exceed
+ * @active. An active internal node's inuse is solely determined by the
+ * inuse to active ratio of its children regardless of @inuse.
+ */
+ if (list_empty(&iocg->active_list) && iocg->child_active_sum) {
+ inuse = DIV64_U64_ROUND_UP(active * iocg->child_inuse_sum,
+ iocg->child_active_sum);
+ } else {
+ inuse = clamp_t(u32, inuse, 1, active);
+ }
iocg->last_inuse = iocg->inuse;
if (save)
/* update the level sums */
parent->child_active_sum += (s32)(active - child->active);
parent->child_inuse_sum += (s32)(inuse - child->inuse);
- /* apply the udpates */
+ /* apply the updates */
child->active = active;
child->inuse = inuse;
unsigned int nr_segs)
{
struct elevator_queue *e = q->elevator;
- struct blk_mq_ctx *ctx = blk_mq_get_ctx(q);
- struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, bio->bi_opf, ctx);
+ struct blk_mq_ctx *ctx;
+ struct blk_mq_hw_ctx *hctx;
bool ret = false;
enum hctx_type type;
if (e && e->type->ops.bio_merge)
- return e->type->ops.bio_merge(hctx, bio, nr_segs);
+ return e->type->ops.bio_merge(q, bio, nr_segs);
+ ctx = blk_mq_get_ctx(q);
+ hctx = blk_mq_map_queue(q, bio->bi_opf, ctx);
type = hctx->type;
if (!(hctx->flags & BLK_MQ_F_SHOULD_MERGE) ||
list_empty_careful(&ctx->rq_lists[type]))
/* Bypass scheduler for flush requests */
blk_insert_flush(rq);
blk_mq_run_hw_queue(data.hctx, true);
- } else if (plug && (q->nr_hw_queues == 1 || q->mq_ops->commit_rqs ||
- !blk_queue_nonrot(q))) {
+ } else if (plug && (q->nr_hw_queues == 1 ||
+ blk_mq_is_sbitmap_shared(rq->mq_hctx->flags) ||
+ q->mq_ops->commit_rqs || !blk_queue_nonrot(q))) {
/*
* Use plugging if we have a ->commit_rqs() hook as well, as
* we know the driver uses bd->last in a smart fashion.
/* tags can _not_ be used after returning from blk_mq_exit_queue */
void blk_mq_exit_queue(struct request_queue *q)
{
- struct blk_mq_tag_set *set = q->tag_set;
+ struct blk_mq_tag_set *set = q->tag_set;
- blk_mq_del_queue_tag_set(q);
+ /* Checks hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED. */
blk_mq_exit_hw_queues(q, set, set->nr_hw_queues);
+ /* May clear BLK_MQ_F_TAG_QUEUE_SHARED in hctx->flags. */
+ blk_mq_del_queue_tag_set(q);
}
static int __blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set)
*/
void blk_set_default_limits(struct queue_limits *lim)
{
- lim->bio_max_bytes = UINT_MAX;
lim->max_segments = BLK_MAX_SEGMENTS;
lim->max_discard_segments = 1;
lim->max_integrity_segments = 0;
limits->logical_block_size >> SECTOR_SHIFT);
limits->max_sectors = max_sectors;
- if (check_shl_overflow(max_sectors, SECTOR_SHIFT,
- &limits->bio_max_bytes))
- limits->bio_max_bytes = UINT_MAX;
-
q->backing_dev_info->io_pages = max_sectors >> (PAGE_SHIFT - 9);
}
EXPORT_SYMBOL(blk_queue_max_hw_sectors);
static struct kobject *block_depr;
-DECLARE_RWSEM(bdev_lookup_sem);
-
/* for extended dynamic devt allocation, currently only one major is used */
#define NR_EXT_DEVT (1 << MINORBITS)
static DEFINE_IDA(ext_devt_ida);
blk_integrity_del(disk);
disk_del_events(disk);
- /*
- * Block lookups of the disk until all bdevs are unhashed and the
- * disk is marked as dead (GENHD_FL_UP cleared).
- */
- down_write(&bdev_lookup_sem);
-
mutex_lock(&disk->part0->bd_mutex);
+ disk->flags &= ~GENHD_FL_UP;
blk_drop_partitions(disk);
mutex_unlock(&disk->part0->bd_mutex);
remove_inode_hash(disk->part0->bd_inode);
set_capacity(disk, 0);
- disk->flags &= ~GENHD_FL_UP;
- up_write(&bdev_lookup_sem);
if (!(disk->flags & GENHD_FL_HIDDEN)) {
sysfs_remove_link(&disk_to_dev(disk)->kobj, "bdi");
}
}
-static bool kyber_bio_merge(struct blk_mq_hw_ctx *hctx, struct bio *bio,
+static bool kyber_bio_merge(struct request_queue *q, struct bio *bio,
unsigned int nr_segs)
{
+ struct blk_mq_ctx *ctx = blk_mq_get_ctx(q);
+ struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, bio->bi_opf, ctx);
struct kyber_hctx_data *khd = hctx->sched_data;
- struct blk_mq_ctx *ctx = blk_mq_get_ctx(hctx->queue);
struct kyber_ctx_queue *kcq = &khd->kcqs[ctx->index_hw[hctx->type]];
unsigned int sched_domain = kyber_sched_domain(bio->bi_opf);
struct list_head *rq_list = &kcq->rq_list[sched_domain];
return ELEVATOR_NO_MERGE;
}
-static bool dd_bio_merge(struct blk_mq_hw_ctx *hctx, struct bio *bio,
+static bool dd_bio_merge(struct request_queue *q, struct bio *bio,
unsigned int nr_segs)
{
- struct request_queue *q = hctx->queue;
struct deadline_data *dd = q->elevator->elevator_data;
struct request *free = NULL;
bool ret;
}
/**
- * efi_partition(struct parsed_partitions *state)
+ * efi_partition - scan for GPT partitions
* @state: disk parsed partitions
*
* Description: called from check.c, if the disk contains GPT
if (submit->flags & ASYNC_TX_XOR_DROP_DST) {
src_cnt--;
src_list++;
- src_offs++;
+ if (src_offs)
+ src_offs++;
}
/* wait for any prerequisite operations */
{ "AMDI0010", APD_ADDR(wt_i2c_desc) },
{ "AMD0020", APD_ADDR(cz_uart_desc) },
{ "AMDI0020", APD_ADDR(cz_uart_desc) },
+ { "AMDI0022", APD_ADDR(cz_uart_desc) },
{ "AMD0030", },
{ "AMD0040", APD_ADDR(fch_misc_desc)},
{ "HYGO0010", APD_ADDR(wt_i2c_desc) },
}
break;
+ case ACPI_TYPE_LOCAL_ADDRESS_HANDLER:
+
+ ACPI_DEBUG_PRINT((ACPI_DB_ALLOCATIONS,
+ "***** Address handler %p\n", object));
+
+ acpi_os_delete_mutex(object->address_space.context_mutex);
+ break;
+
default:
break;
if (ACPI_FAILURE(acpi_run_osc(handle, &context)))
return;
- capbuf_ret = context.ret.pointer;
- if (context.ret.length <= OSC_SUPPORT_DWORD) {
- kfree(context.ret.pointer);
- return;
- }
+ kfree(context.ret.pointer);
- /*
- * Now run _OSC again with query flag clear and with the caps
- * supported by both the OS and the platform.
- */
+ /* Now run _OSC again with query flag clear */
capbuf[OSC_QUERY_DWORD] = 0;
- capbuf[OSC_SUPPORT_DWORD] = capbuf_ret[OSC_SUPPORT_DWORD];
- kfree(context.ret.pointer);
if (ACPI_FAILURE(acpi_run_osc(handle, &context)))
return;
capbuf_ret = context.ret.pointer;
- if (context.ret.length > OSC_SUPPORT_DWORD) {
- osc_sb_apei_support_acked =
- capbuf_ret[OSC_SUPPORT_DWORD] & OSC_SB_APEI_SUPPORT;
- osc_pc_lpi_support_confirmed =
- capbuf_ret[OSC_SUPPORT_DWORD] & OSC_SB_PCLPI_SUPPORT;
- osc_sb_native_usb4_support_confirmed =
- capbuf_ret[OSC_SUPPORT_DWORD] & OSC_SB_NATIVE_USB4_SUPPORT;
- }
+ osc_sb_apei_support_acked =
+ capbuf_ret[OSC_SUPPORT_DWORD] & OSC_SB_APEI_SUPPORT;
+ osc_pc_lpi_support_confirmed =
+ capbuf_ret[OSC_SUPPORT_DWORD] & OSC_SB_PCLPI_SUPPORT;
+ osc_sb_native_usb4_support_confirmed =
+ capbuf_ret[OSC_SUPPORT_DWORD] & OSC_SB_NATIVE_USB4_SUPPORT;
kfree(context.ret.pointer);
}
{"PNP0C0B", }, /* Generic ACPI fan */
{"INT3404", }, /* Fan */
{"INTC1044", }, /* Fan for Tiger Lake generation */
+ {"INTC1048", }, /* Fan for Alder Lake generation */
{}
};
struct acpi_device *adev = ACPI_COMPANION(dev);
void acpi_power_resources_list_free(struct list_head *list);
int acpi_extract_power_resources(union acpi_object *package, unsigned int start,
struct list_head *list);
-int acpi_add_power_resource(acpi_handle handle);
+struct acpi_device *acpi_add_power_resource(acpi_handle handle);
void acpi_power_add_remove_device(struct acpi_device *adev, bool add);
int acpi_power_wakeup_list_init(struct list_head *list, int *system_level);
int acpi_device_sleep_wake(struct acpi_device *dev,
int acpi_power_get_inferred_state(struct acpi_device *device, int *state);
int acpi_power_on_resources(struct acpi_device *device, int state);
int acpi_power_transition(struct acpi_device *device, int state);
+void acpi_turn_off_unused_power_resources(bool init);
/* --------------------------------------------------------------------------
Device Power Management
return -1;
}
+static size_t sizeof_spa(struct acpi_nfit_system_address *spa)
+{
+ if (spa->flags & ACPI_NFIT_LOCATION_COOKIE_VALID)
+ return sizeof(*spa);
+ return sizeof(*spa) - 8;
+}
+
static bool add_spa(struct acpi_nfit_desc *acpi_desc,
struct nfit_table_prev *prev,
struct acpi_nfit_system_address *spa)
struct device *dev = acpi_desc->dev;
struct nfit_spa *nfit_spa;
- if (spa->header.length != sizeof(*spa))
+ if (spa->header.length != sizeof_spa(spa))
return false;
list_for_each_entry(nfit_spa, &prev->spas, list) {
- if (memcmp(nfit_spa->spa, spa, sizeof(*spa)) == 0) {
+ if (memcmp(nfit_spa->spa, spa, sizeof_spa(spa)) == 0) {
list_move_tail(&nfit_spa->list, &acpi_desc->spas);
return true;
}
}
- nfit_spa = devm_kzalloc(dev, sizeof(*nfit_spa) + sizeof(*spa),
+ nfit_spa = devm_kzalloc(dev, sizeof(*nfit_spa) + sizeof_spa(spa),
GFP_KERNEL);
if (!nfit_spa)
return false;
INIT_LIST_HEAD(&nfit_spa->list);
- memcpy(nfit_spa->spa, spa, sizeof(*spa));
+ memcpy(nfit_spa->spa, spa, sizeof_spa(spa));
list_add_tail(&nfit_spa->list, &acpi_desc->spas);
dev_dbg(dev, "spa index: %d type: %s\n",
spa->range_index,
u32 system_level;
u32 order;
unsigned int ref_count;
+ unsigned int users;
bool wakeup_enabled;
struct mutex resource_lock;
struct list_head dependents;
for (i = start; i < package->package.count; i++) {
union acpi_object *element = &package->package.elements[i];
+ struct acpi_device *rdev;
acpi_handle rhandle;
if (element->type != ACPI_TYPE_LOCAL_REFERENCE) {
if (acpi_power_resource_is_dup(package, start, i))
continue;
- err = acpi_add_power_resource(rhandle);
- if (err)
+ rdev = acpi_add_power_resource(rhandle);
+ if (!rdev) {
+ err = -ENODEV;
break;
-
+ }
err = acpi_power_resources_list_add(rhandle, list);
if (err)
break;
+
+ to_power_resource(rdev)->users++;
}
if (err)
acpi_power_resources_list_free(list);
mutex_unlock(&power_resource_list_lock);
}
-int acpi_add_power_resource(acpi_handle handle)
+struct acpi_device *acpi_add_power_resource(acpi_handle handle)
{
struct acpi_power_resource *resource;
struct acpi_device *device = NULL;
acpi_bus_get_device(handle, &device);
if (device)
- return 0;
+ return device;
resource = kzalloc(sizeof(*resource), GFP_KERNEL);
if (!resource)
- return -ENOMEM;
+ return NULL;
device = &resource->device;
acpi_init_device_object(device, handle, ACPI_BUS_TYPE_POWER);
acpi_power_add_resource_to_list(resource);
acpi_device_add_finalize(device);
- return 0;
+ return device;
err:
acpi_release_power_resource(&device->dev);
- return result;
+ return NULL;
}
#ifdef CONFIG_ACPI_SLEEP
mutex_unlock(&power_resource_list_lock);
}
+#endif
+
+static void acpi_power_turn_off_if_unused(struct acpi_power_resource *resource,
+ bool init)
+{
+ if (resource->ref_count > 0)
+ return;
+
+ if (init) {
+ if (resource->users > 0)
+ return;
+ } else {
+ int result, state;
+
+ result = acpi_power_get_state(resource->device.handle, &state);
+ if (result || state == ACPI_POWER_RESOURCE_STATE_OFF)
+ return;
+ }
+
+ dev_info(&resource->device.dev, "Turning OFF\n");
+ __acpi_power_off(resource);
+}
-void acpi_turn_off_unused_power_resources(void)
+/**
+ * acpi_turn_off_unused_power_resources - Turn off power resources not in use.
+ * @init: Control switch.
+ *
+ * If @ainit is set, unconditionally turn off all of the ACPI power resources
+ * without any users.
+ *
+ * Otherwise, turn off all ACPI power resources without active references (that
+ * is, the ones that should be "off" at the moment) that are "on".
+ */
+void acpi_turn_off_unused_power_resources(bool init)
{
struct acpi_power_resource *resource;
list_for_each_entry_reverse(resource, &acpi_power_resource_list, list_node) {
mutex_lock(&resource->resource_lock);
- if (!resource->ref_count) {
- dev_info(&resource->device.dev, "Turning OFF\n");
- __acpi_power_off(resource);
- }
+ acpi_power_turn_off_if_unused(resource, init);
mutex_unlock(&resource->resource_lock);
}
mutex_unlock(&power_resource_list_lock);
}
-#endif
result = acpi_device_set_name(device, acpi_device_bus_id);
if (result) {
+ kfree_const(acpi_device_bus_id->bus_id);
kfree(acpi_device_bus_id);
goto err_unlock;
}
}
}
+ acpi_turn_off_unused_power_resources(true);
+
acpi_scan_initialized = true;
out:
*/
static void acpi_pm_end(void)
{
- acpi_turn_off_unused_power_resources();
+ acpi_turn_off_unused_power_resources(false);
acpi_scan_lock_release();
/*
* This is necessary in case acpi_pm_finish() is not called during a
return;
acpi_get_table(ACPI_SIG_FACS, 1, (struct acpi_table_header **)&facs);
- if (facs) {
+ if (facs)
s4_hardware_signature = facs->hardware_signature;
- acpi_put_table((struct acpi_table_header *)facs);
- }
}
#else /* !CONFIG_HIBERNATION */
static inline void acpi_sleep_hibernate_setup(void) {}
extern struct mutex acpi_device_lock;
extern void acpi_resume_power_resources(void);
-extern void acpi_turn_off_unused_power_resources(void);
static inline acpi_status acpi_set_waking_vector(u32 wakeup_address)
{
EXPORT_SYMBOL(acpi_evaluate_integer);
+int acpi_get_local_address(acpi_handle handle, u32 *addr)
+{
+ unsigned long long adr;
+ acpi_status status;
+
+ status = acpi_evaluate_integer(handle, METHOD_NAME__ADR, NULL, &adr);
+ if (ACPI_FAILURE(status))
+ return -ENODATA;
+
+ *addr = (u32)adr;
+ return 0;
+}
+EXPORT_SYMBOL(acpi_get_local_address);
+
acpi_status
acpi_evaluate_reference(acpi_handle handle,
acpi_string pathname,
uint32_t enable;
if (copy_from_user(&enable, ubuf, sizeof(enable))) {
- ret = -EINVAL;
+ ret = -EFAULT;
goto err;
}
binder_inner_proc_lock(proc);
#define ZEPROM_V1_REG PCI_VENDOR_ID /* PCI register */
#define ZEPROM_V2_REG 0x40
-/* Bits in contol register */
+/* Bits in control register */
#define ZEPROM_SK 0x80000000 /* strobe (probably on raising edge) */
#define ZEPROM_CS 0x40000000 /* Chip Select */
fwnode_links_purge_consumers(fwnode);
}
-static void fw_devlink_purge_absent_suppliers(struct fwnode_handle *fwnode)
+void fw_devlink_purge_absent_suppliers(struct fwnode_handle *fwnode)
{
struct fwnode_handle *child;
fwnode_for_each_available_child_node(fwnode, child)
fw_devlink_purge_absent_suppliers(child);
}
+EXPORT_SYMBOL_GPL(fw_devlink_purge_absent_suppliers);
#ifdef CONFIG_SRCU
static DEFINE_MUTEX(device_links_lock);
{
return srcu_read_lock_held(&device_links_srcu);
}
+
+static void device_link_synchronize_removal(void)
+{
+ synchronize_srcu(&device_links_srcu);
+}
+
+static void device_link_remove_from_lists(struct device_link *link)
+{
+ list_del_rcu(&link->s_node);
+ list_del_rcu(&link->c_node);
+}
#else /* !CONFIG_SRCU */
static DECLARE_RWSEM(device_links_lock);
return lockdep_is_held(&device_links_lock);
}
#endif
+
+static inline void device_link_synchronize_removal(void)
+{
+}
+
+static void device_link_remove_from_lists(struct device_link *link)
+{
+ list_del(&link->s_node);
+ list_del(&link->c_node);
+}
#endif /* !CONFIG_SRCU */
static bool device_is_ancestor(struct device *dev, struct device *target)
};
ATTRIBUTE_GROUPS(devlink);
-static void device_link_free(struct device_link *link)
+static void device_link_release_fn(struct work_struct *work)
{
+ struct device_link *link = container_of(work, struct device_link, rm_work);
+
+ /* Ensure that all references to the link object have been dropped. */
+ device_link_synchronize_removal();
+
while (refcount_dec_not_one(&link->rpm_active))
pm_runtime_put(link->supplier);
kfree(link);
}
-#ifdef CONFIG_SRCU
-static void __device_link_free_srcu(struct rcu_head *rhead)
-{
- device_link_free(container_of(rhead, struct device_link, rcu_head));
-}
-
static void devlink_dev_release(struct device *dev)
{
struct device_link *link = to_devlink(dev);
- call_srcu(&device_links_srcu, &link->rcu_head, __device_link_free_srcu);
-}
-#else
-static void devlink_dev_release(struct device *dev)
-{
- device_link_free(to_devlink(dev));
+ INIT_WORK(&link->rm_work, device_link_release_fn);
+ /*
+ * It may take a while to complete this work because of the SRCU
+ * synchronization in device_link_release_fn() and if the consumer or
+ * supplier devices get deleted when it runs, so put it into the "long"
+ * workqueue.
+ */
+ queue_work(system_long_wq, &link->rm_work);
}
-#endif
static struct class devlink_class = {
.name = "devlink",
}
EXPORT_SYMBOL_GPL(device_link_add);
-#ifdef CONFIG_SRCU
static void __device_link_del(struct kref *kref)
{
struct device_link *link = container_of(kref, struct device_link, kref);
pm_runtime_drop_link(link);
- list_del_rcu(&link->s_node);
- list_del_rcu(&link->c_node);
- device_unregister(&link->link_dev);
-}
-#else /* !CONFIG_SRCU */
-static void __device_link_del(struct kref *kref)
-{
- struct device_link *link = container_of(kref, struct device_link, kref);
-
- dev_info(link->consumer, "Dropping the link to %s\n",
- dev_name(link->supplier));
-
- pm_runtime_drop_link(link);
-
- list_del(&link->s_node);
- list_del(&link->c_node);
+ device_link_remove_from_lists(link);
device_unregister(&link->link_dev);
}
-#endif /* !CONFIG_SRCU */
static void device_link_put_kref(struct device_link *link)
{
}
EXPORT_SYMBOL_GPL(device_set_of_node_from_dev);
+void device_set_node(struct device *dev, struct fwnode_handle *fwnode)
+{
+ dev->fwnode = fwnode;
+ dev->of_node = to_of_node(fwnode);
+}
+EXPORT_SYMBOL_GPL(device_set_node);
+
int device_match_name(struct device *dev, const void *name)
{
return sysfs_streq(dev_name(dev), name);
struct zone *zone;
int ret;
- zone = page_zone(pfn_to_page(start_pfn));
-
/*
* Unaccount before offlining, such that unpopulated zone and kthreads
* can properly be torn down in offline_pages().
*/
- if (nr_vmemmap_pages)
+ if (nr_vmemmap_pages) {
+ zone = page_zone(pfn_to_page(start_pfn));
adjust_present_page_count(zone, -nr_vmemmap_pages);
+ }
ret = offline_pages(start_pfn + nr_vmemmap_pages,
nr_pages - nr_vmemmap_pages);
dev->power.request_pending = false;
dev->power.request = RPM_REQ_NONE;
dev->power.deferred_resume = false;
+ dev->power.needs_force_resume = 0;
INIT_WORK(&dev->power.work, pm_runtime_work);
dev->power.timer_expires = 0;
* its parent, but set its status to RPM_SUSPENDED anyway in case this
* function will be called again for it in the meantime.
*/
- if (pm_runtime_need_not_resume(dev))
+ if (pm_runtime_need_not_resume(dev)) {
pm_runtime_set_suspended(dev);
- else
+ } else {
__update_runtime_status(dev, RPM_SUSPENDED);
+ dev->power.needs_force_resume = 1;
+ }
return 0;
int (*callback)(struct device *);
int ret = 0;
- if (!pm_runtime_status_suspended(dev) || pm_runtime_need_not_resume(dev))
+ if (!pm_runtime_status_suspended(dev) || !dev->power.needs_force_resume)
goto out;
/*
pm_runtime_mark_last_busy(dev);
out:
+ dev->power.needs_force_resume = 0;
pm_runtime_enable(dev);
return ret;
}
static int lo_open(struct block_device *bdev, fmode_t mode)
{
- struct loop_device *lo;
+ struct loop_device *lo = bdev->bd_disk->private_data;
int err;
- /*
- * take loop_ctl_mutex to protect lo pointer from race with
- * loop_control_ioctl(LOOP_CTL_REMOVE), however, to reduce contention
- * release it prior to updating lo->lo_refcnt.
- */
- err = mutex_lock_killable(&loop_ctl_mutex);
- if (err)
- return err;
- lo = bdev->bd_disk->private_data;
- if (!lo) {
- mutex_unlock(&loop_ctl_mutex);
- return -ENXIO;
- }
err = mutex_lock_killable(&lo->lo_mutex);
- mutex_unlock(&loop_ctl_mutex);
if (err)
return err;
- atomic_inc(&lo->lo_refcnt);
+ if (lo->lo_state == Lo_deleting)
+ err = -ENXIO;
+ else
+ atomic_inc(&lo->lo_refcnt);
mutex_unlock(&lo->lo_mutex);
- return 0;
+ return err;
}
static void lo_release(struct gendisk *disk, fmode_t mode)
mutex_unlock(&lo->lo_mutex);
break;
}
- lo->lo_disk->private_data = NULL;
+ lo->lo_state = Lo_deleting;
mutex_unlock(&lo->lo_mutex);
idr_remove(&loop_index_idr, lo->lo_number);
loop_remove(lo);
Lo_unbound,
Lo_bound,
Lo_rundown,
+ Lo_deleting,
};
struct loop_func_table;
* config ref and try to destroy the workqueue from inside the work
* queue.
*/
- flush_workqueue(nbd->recv_workq);
+ if (nbd->recv_workq)
+ flush_workqueue(nbd->recv_workq);
if (test_and_clear_bit(NBD_RT_HAS_CONFIG_REF,
&nbd->config->runtime_flags))
nbd_config_put(nbd);
return -EINVAL;
}
mutex_unlock(&nbd_index_mutex);
- if (!refcount_inc_not_zero(&nbd->config_refs)) {
- nbd_put(nbd);
- return 0;
- }
+ if (!refcount_inc_not_zero(&nbd->config_refs))
+ goto put_nbd;
nbd_disconnect_and_put(nbd);
nbd_config_put(nbd);
+put_nbd:
nbd_put(nbd);
return 0;
}
/* Realtek 8822CE Bluetooth devices */
{ USB_DEVICE(0x0bda, 0xb00c), .driver_info = BTUSB_REALTEK |
BTUSB_WIDEBAND_SPEECH },
+ { USB_DEVICE(0x0bda, 0xc822), .driver_info = BTUSB_REALTEK |
+ BTUSB_WIDEBAND_SPEECH },
/* Realtek 8852AE Bluetooth devices */
{ USB_DEVICE(0x0bda, 0xc852), .driver_info = BTUSB_REALTEK |
}
btusb_setup_intel_newgen_get_fw_name(ver, fwname, sizeof(fwname), "sfi");
- err = request_firmware(&fw, fwname, &hdev->dev);
+ err = firmware_request_nowarn(&fw, fwname, &hdev->dev);
if (err < 0) {
+ if (!test_bit(BTUSB_BOOTLOADER, &data->flags)) {
+ /* Firmware has already been loaded */
+ set_bit(BTUSB_FIRMWARE_LOADED, &data->flags);
+ return 0;
+ }
+
bt_dev_err(hdev, "Failed to load Intel firmware file %s (%d)",
fwname, err);
+
return err;
}
err = btusb_setup_intel_new_get_fw_name(ver, params, fwname,
sizeof(fwname), "sfi");
if (err < 0) {
+ if (!test_bit(BTUSB_BOOTLOADER, &data->flags)) {
+ /* Firmware has already been loaded */
+ set_bit(BTUSB_FIRMWARE_LOADED, &data->flags);
+ return 0;
+ }
+
bt_dev_err(hdev, "Unsupported Intel firmware naming");
return -EINVAL;
}
- err = request_firmware(&fw, fwname, &hdev->dev);
+ err = firmware_request_nowarn(&fw, fwname, &hdev->dev);
if (err < 0) {
+ if (!test_bit(BTUSB_BOOTLOADER, &data->flags)) {
+ /* Firmware has already been loaded */
+ set_bit(BTUSB_FIRMWARE_LOADED, &data->flags);
+ return 0;
+ }
+
bt_dev_err(hdev, "Failed to load Intel firmware file %s (%d)",
fwname, err);
return err;
MHI_CHANNEL_CONFIG_DL(5, "DIAG", 32, 1),
MHI_CHANNEL_CONFIG_UL(12, "MBIM", 32, 0),
MHI_CHANNEL_CONFIG_DL(13, "MBIM", 32, 0),
- MHI_CHANNEL_CONFIG_UL(32, "AT", 32, 0),
- MHI_CHANNEL_CONFIG_DL(33, "AT", 32, 0),
+ MHI_CHANNEL_CONFIG_UL(32, "DUN", 32, 0),
+ MHI_CHANNEL_CONFIG_DL(33, "DUN", 32, 0),
MHI_CHANNEL_CONFIG_HW_UL(100, "IP_HW0_MBIM", 128, 2),
MHI_CHANNEL_CONFIG_HW_DL(101, "IP_HW0_MBIM", 128, 3),
};
struct mhi_pci_device *mhi_pdev = pci_get_drvdata(pdev);
struct mhi_controller *mhi_cntrl = &mhi_pdev->mhi_cntrl;
- del_timer(&mhi_pdev->health_check_timer);
+ del_timer_sync(&mhi_pdev->health_check_timer);
cancel_work_sync(&mhi_pdev->recovery_work);
if (test_and_clear_bit(MHI_PCI_DEV_STARTED, &mhi_pdev->status)) {
return ret;
}
+static int __maybe_unused mhi_pci_freeze(struct device *dev)
+{
+ struct mhi_pci_device *mhi_pdev = dev_get_drvdata(dev);
+ struct mhi_controller *mhi_cntrl = &mhi_pdev->mhi_cntrl;
+
+ /* We want to stop all operations, hibernation does not guarantee that
+ * device will be in the same state as before freezing, especially if
+ * the intermediate restore kernel reinitializes MHI device with new
+ * context.
+ */
+ if (test_and_clear_bit(MHI_PCI_DEV_STARTED, &mhi_pdev->status)) {
+ mhi_power_down(mhi_cntrl, false);
+ mhi_unprepare_after_power_down(mhi_cntrl);
+ }
+
+ return 0;
+}
+
+static int __maybe_unused mhi_pci_restore(struct device *dev)
+{
+ struct mhi_pci_device *mhi_pdev = dev_get_drvdata(dev);
+
+ /* Reinitialize the device */
+ queue_work(system_long_wq, &mhi_pdev->recovery_work);
+
+ return 0;
+}
+
static const struct dev_pm_ops mhi_pci_pm_ops = {
SET_RUNTIME_PM_OPS(mhi_pci_runtime_suspend, mhi_pci_runtime_resume, NULL)
- SET_SYSTEM_SLEEP_PM_OPS(mhi_pci_suspend, mhi_pci_resume)
+#ifdef CONFIG_PM_SLEEP
+ .suspend = mhi_pci_suspend,
+ .resume = mhi_pci_resume,
+ .freeze = mhi_pci_freeze,
+ .thaw = mhi_pci_restore,
+ .restore = mhi_pci_restore,
+#endif
};
static struct pci_driver mhi_pci_driver = {
return error;
}
+static int sysc_reinit_module(struct sysc *ddata, bool leave_enabled)
+{
+ struct device *dev = ddata->dev;
+ int error;
+
+ /* Disable target module if it is enabled */
+ if (ddata->enabled) {
+ error = sysc_runtime_suspend(dev);
+ if (error)
+ dev_warn(dev, "reinit suspend failed: %i\n", error);
+ }
+
+ /* Enable target module */
+ error = sysc_runtime_resume(dev);
+ if (error)
+ dev_warn(dev, "reinit resume failed: %i\n", error);
+
+ if (leave_enabled)
+ return error;
+
+ /* Disable target module if no leave_enabled was set */
+ error = sysc_runtime_suspend(dev);
+ if (error)
+ dev_warn(dev, "reinit suspend failed: %i\n", error);
+
+ return error;
+}
+
static int __maybe_unused sysc_noirq_suspend(struct device *dev)
{
struct sysc *ddata;
(SYSC_QUIRK_LEGACY_IDLE | SYSC_QUIRK_NO_IDLE))
return 0;
- return pm_runtime_force_suspend(dev);
+ if (!ddata->enabled)
+ return 0;
+
+ ddata->needs_resume = 1;
+
+ return sysc_runtime_suspend(dev);
}
static int __maybe_unused sysc_noirq_resume(struct device *dev)
{
struct sysc *ddata;
+ int error = 0;
ddata = dev_get_drvdata(dev);
(SYSC_QUIRK_LEGACY_IDLE | SYSC_QUIRK_NO_IDLE))
return 0;
- return pm_runtime_force_resume(dev);
+ if (ddata->cfg.quirks & SYSC_QUIRK_REINIT_ON_RESUME) {
+ error = sysc_reinit_module(ddata, ddata->needs_resume);
+ if (error)
+ dev_warn(dev, "noirq_resume failed: %i\n", error);
+ } else if (ddata->needs_resume) {
+ error = sysc_runtime_resume(dev);
+ if (error)
+ dev_warn(dev, "noirq_resume failed: %i\n", error);
+ }
+
+ ddata->needs_resume = 0;
+
+ return error;
}
static const struct dev_pm_ops sysc_pm_ops = {
SYSC_QUIRK_SWSUP_SIDLE | SYSC_QUIRK_LEGACY_IDLE),
/* Uarts on omap4 and later */
SYSC_QUIRK("uart", 0, 0x50, 0x54, 0x58, 0x50411e03, 0xffff00ff,
- SYSC_QUIRK_SWSUP_SIDLE_ACT | SYSC_QUIRK_LEGACY_IDLE),
+ SYSC_QUIRK_SWSUP_SIDLE | SYSC_QUIRK_LEGACY_IDLE),
SYSC_QUIRK("uart", 0, 0x50, 0x54, 0x58, 0x47422e03, 0xffffffff,
- SYSC_QUIRK_SWSUP_SIDLE_ACT | SYSC_QUIRK_LEGACY_IDLE),
+ SYSC_QUIRK_SWSUP_SIDLE | SYSC_QUIRK_LEGACY_IDLE),
/* Quirks that need to be set based on the module address */
SYSC_QUIRK("mcpdm", 0x40132000, 0, 0x10, -ENODEV, 0x50000800, 0xffffffff,
SYSC_QUIRK_SWSUP_SIDLE | SYSC_QUIRK_SWSUP_MSTANDBY),
SYSC_QUIRK("tptc", 0, 0, -ENODEV, -ENODEV, 0x40007c00, 0xffffffff,
SYSC_QUIRK_SWSUP_SIDLE | SYSC_QUIRK_SWSUP_MSTANDBY),
+ SYSC_QUIRK("sata", 0, 0xfc, 0x1100, -ENODEV, 0x5e412000, 0xffffffff,
+ SYSC_QUIRK_SWSUP_SIDLE | SYSC_QUIRK_SWSUP_MSTANDBY),
SYSC_QUIRK("usb_host_hs", 0, 0, 0x10, 0x14, 0x50700100, 0xffffffff,
SYSC_QUIRK_SWSUP_SIDLE | SYSC_QUIRK_SWSUP_MSTANDBY),
SYSC_QUIRK("usb_host_hs", 0, 0, 0x10, -ENODEV, 0x50700101, 0xffffffff,
SYSC_QUIRK("usb_otg_hs", 0, 0x400, 0x404, 0x408, 0x00000050,
0xffffffff, SYSC_QUIRK_SWSUP_SIDLE | SYSC_QUIRK_SWSUP_MSTANDBY),
SYSC_QUIRK("usb_otg_hs", 0, 0, 0x10, -ENODEV, 0x4ea2080d, 0xffffffff,
- SYSC_QUIRK_SWSUP_SIDLE | SYSC_QUIRK_SWSUP_MSTANDBY),
+ SYSC_QUIRK_SWSUP_SIDLE | SYSC_QUIRK_SWSUP_MSTANDBY |
+ SYSC_QUIRK_REINIT_ON_RESUME),
SYSC_QUIRK("wdt", 0, 0, 0x10, 0x14, 0x502a0500, 0xfffff0f0,
SYSC_MODULE_QUIRK_WDT),
/* PRUSS on am3, am4 and am5 */
SYSC_QUIRK("prcm", 0, 0, -ENODEV, -ENODEV, 0x40000400, 0xffffffff, 0),
SYSC_QUIRK("rfbi", 0x4832a800, 0, 0x10, 0x14, 0x00000010, 0xffffffff, 0),
SYSC_QUIRK("rfbi", 0x58002000, 0, 0x10, 0x14, 0x00000010, 0xffffffff, 0),
- SYSC_QUIRK("sata", 0, 0xfc, 0x1100, -ENODEV, 0x5e412000, 0xffffffff, 0),
SYSC_QUIRK("scm", 0, 0, 0x10, -ENODEV, 0x40000900, 0xffffffff, 0),
SYSC_QUIRK("scm", 0, 0, -ENODEV, -ENODEV, 0x4e8b0100, 0xffffffff, 0),
SYSC_QUIRK("scm", 0, 0, -ENODEV, -ENODEV, 0x4f000100, 0xffffffff, 0),
static int probe_gdrom(struct platform_device *devptr)
{
int err;
+
+ /*
+ * Ensure our "one" device is initialized properly in case of previous
+ * usages of it
+ */
+ memset(&gd, 0, sizeof(gd));
+
/* Start the device */
if (gdrom_execute_diagnostic() != 1) {
pr_warn("ATA Probe for GDROM failed\n");
if (gdrom_major)
unregister_blkdev(gdrom_major, GDROM_DEV_NAME);
unregister_cdrom(gd.cd_info);
+ kfree(gd.cd_info);
+ kfree(gd.toc);
return 0;
}
static int __init init_gdrom(void)
{
int rc;
- gd.toc = NULL;
+
rc = platform_driver_register(&gdrom_driver);
if (rc)
return rc;
{
platform_device_unregister(pd);
platform_driver_unregister(&gdrom_driver);
- kfree(gd.toc);
- kfree(gd.cd_info);
}
module_init(init_gdrom);
hdp->hd_phys_address = fixmem32->address;
hdp->hd_address = ioremap(fixmem32->address,
HPET_RANGE_SIZE);
+ if (!hdp->hd_address)
+ return AE_ERROR;
if (hpet_is_known(hdp)) {
iounmap(hdp->hd_address);
if (nr_commands !=
be32_to_cpup((__be32 *)&buf.data[TPM_HEADER_SIZE + 5])) {
+ rc = -EFAULT;
tpm_buf_destroy(&buf);
goto out;
}
cap_t cap;
int ret;
- /* TPM 2.0 */
- if (chip->flags & TPM_CHIP_FLAG_TPM2)
- return tpm2_get_tpm_pt(chip, 0x100, &cap2, desc);
-
- /* TPM 1.2 */
ret = request_locality(chip, 0);
if (ret < 0)
return ret;
- ret = tpm1_getcap(chip, TPM_CAP_PROP_TIS_TIMEOUT, &cap, desc, 0);
+ if (chip->flags & TPM_CHIP_FLAG_TPM2)
+ ret = tpm2_get_tpm_pt(chip, 0x100, &cap2, desc);
+ else
+ ret = tpm1_getcap(chip, TPM_CAP_PROP_TIS_TIMEOUT, &cap, desc, 0);
release_locality(chip, 0);
if (ret)
return ret;
- /* TPM 1.2 requires self-test on resume. This function actually returns
+ /*
+ * TPM 1.2 requires self-test on resume. This function actually returns
* an error code but for unknown reason it isn't handled.
*/
- if (!(chip->flags & TPM_CHIP_FLAG_TPM2))
+ if (!(chip->flags & TPM_CHIP_FLAG_TPM2)) {
+ ret = request_locality(chip, 0);
+ if (ret < 0)
+ return ret;
+
tpm1_do_selftest(chip);
+ release_locality(chip, 0);
+ }
+
return 0;
}
EXPORT_SYMBOL_GPL(tpm_tis_resume);
struct of_clk_provider *cp;
int ret;
+ if (!np)
+ return 0;
+
cp = kzalloc(sizeof(*cp), GFP_KERNEL);
if (!cp)
return -ENOMEM;
struct of_clk_provider *cp;
int ret;
+ if (!np)
+ return 0;
+
cp = kzalloc(sizeof(*cp), GFP_KERNEL);
if (!cp)
return -ENOMEM;
{
struct of_clk_provider *cp;
+ if (!np)
+ return;
+
mutex_lock(&of_clk_mutex);
list_for_each_entry(cp, &of_clk_providers, link) {
if (cp->node == np) {
hv_set_register(HV_REGISTER_REFERENCE_TSC, tsc_msr);
}
-#ifdef VDSO_CLOCKMODE_HVCLOCK
+#ifdef HAVE_VDSO_CLOCKMODE_HVCLOCK
static int hv_cs_enable(struct clocksource *cs)
{
vclocks_set_used(VDSO_CLOCKMODE_HVCLOCK);
.flags = CLOCK_SOURCE_IS_CONTINUOUS,
.suspend= suspend_hv_clock_tsc,
.resume = resume_hv_clock_tsc,
-#ifdef VDSO_CLOCKMODE_HVCLOCK
+#ifdef HAVE_VDSO_CLOCKMODE_HVCLOCK
.enable = hv_cs_enable,
.vdso_clock_mode = VDSO_CLOCKMODE_HVCLOCK,
#else
If in doubt, say N.
-config ACPI_CPPC_CPUFREQ_FIE
- bool "Frequency Invariance support for CPPC cpufreq driver"
- depends on ACPI_CPPC_CPUFREQ && GENERIC_ARCH_TOPOLOGY
- default y
- help
- This extends frequency invariance support in the CPPC cpufreq driver,
- by using CPPC delivered and reference performance counters.
-
- If in doubt, say N.
-
config ARM_ALLWINNER_SUN50I_CPUFREQ_NVMEM
tristate "Allwinner nvmem based SUN50I CPUFreq driver"
depends on ARCH_SUNXI
return 0;
}
- highest_perf = perf_caps.highest_perf;
+ if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD)
+ highest_perf = amd_get_highest_perf();
+ else
+ highest_perf = perf_caps.highest_perf;
+
nominal_perf = perf_caps.nominal_perf;
if (!highest_perf || !nominal_perf) {
#define pr_fmt(fmt) "CPPC Cpufreq:" fmt
-#include <linux/arch_topology.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/delay.h>
#include <linux/cpu.h>
#include <linux/cpufreq.h>
#include <linux/dmi.h>
-#include <linux/irq_work.h>
-#include <linux/kthread.h>
#include <linux/time.h>
#include <linux/vmalloc.h>
-#include <uapi/linux/sched/types.h>
#include <asm/unaligned.h>
}
};
-#ifdef CONFIG_ACPI_CPPC_CPUFREQ_FIE
-
-/* Frequency invariance support */
-struct cppc_freq_invariance {
- int cpu;
- struct irq_work irq_work;
- struct kthread_work work;
- struct cppc_perf_fb_ctrs prev_perf_fb_ctrs;
- struct cppc_cpudata *cpu_data;
-};
-
-static DEFINE_PER_CPU(struct cppc_freq_invariance, cppc_freq_inv);
-static struct kthread_worker *kworker_fie;
-static bool fie_disabled;
-
-static struct cpufreq_driver cppc_cpufreq_driver;
-static unsigned int hisi_cppc_cpufreq_get_rate(unsigned int cpu);
-static int cppc_perf_from_fbctrs(struct cppc_cpudata *cpu_data,
- struct cppc_perf_fb_ctrs fb_ctrs_t0,
- struct cppc_perf_fb_ctrs fb_ctrs_t1);
-
-/**
- * cppc_scale_freq_workfn - CPPC arch_freq_scale updater for frequency invariance
- * @work: The work item.
- *
- * The CPPC driver register itself with the topology core to provide its own
- * implementation (cppc_scale_freq_tick()) of topology_scale_freq_tick() which
- * gets called by the scheduler on every tick.
- *
- * Note that the arch specific counters have higher priority than CPPC counters,
- * if available, though the CPPC driver doesn't need to have any special
- * handling for that.
- *
- * On an invocation of cppc_scale_freq_tick(), we schedule an irq work (since we
- * reach here from hard-irq context), which then schedules a normal work item
- * and cppc_scale_freq_workfn() updates the per_cpu arch_freq_scale variable
- * based on the counter updates since the last tick.
- */
-static void cppc_scale_freq_workfn(struct kthread_work *work)
-{
- struct cppc_freq_invariance *cppc_fi;
- struct cppc_perf_fb_ctrs fb_ctrs = {0};
- struct cppc_cpudata *cpu_data;
- unsigned long local_freq_scale;
- u64 perf;
-
- cppc_fi = container_of(work, struct cppc_freq_invariance, work);
- cpu_data = cppc_fi->cpu_data;
-
- if (cppc_get_perf_ctrs(cppc_fi->cpu, &fb_ctrs)) {
- pr_warn("%s: failed to read perf counters\n", __func__);
- return;
- }
-
- cppc_fi->prev_perf_fb_ctrs = fb_ctrs;
- perf = cppc_perf_from_fbctrs(cpu_data, cppc_fi->prev_perf_fb_ctrs,
- fb_ctrs);
-
- perf <<= SCHED_CAPACITY_SHIFT;
- local_freq_scale = div64_u64(perf, cpu_data->perf_caps.highest_perf);
- if (WARN_ON(local_freq_scale > 1024))
- local_freq_scale = 1024;
-
- per_cpu(arch_freq_scale, cppc_fi->cpu) = local_freq_scale;
-}
-
-static void cppc_irq_work(struct irq_work *irq_work)
-{
- struct cppc_freq_invariance *cppc_fi;
-
- cppc_fi = container_of(irq_work, struct cppc_freq_invariance, irq_work);
- kthread_queue_work(kworker_fie, &cppc_fi->work);
-}
-
-static void cppc_scale_freq_tick(void)
-{
- struct cppc_freq_invariance *cppc_fi = &per_cpu(cppc_freq_inv, smp_processor_id());
-
- /*
- * cppc_get_perf_ctrs() can potentially sleep, call that from the right
- * context.
- */
- irq_work_queue(&cppc_fi->irq_work);
-}
-
-static struct scale_freq_data cppc_sftd = {
- .source = SCALE_FREQ_SOURCE_CPPC,
- .set_freq_scale = cppc_scale_freq_tick,
-};
-
-static void cppc_freq_invariance_policy_init(struct cpufreq_policy *policy,
- struct cppc_cpudata *cpu_data)
-{
- struct cppc_perf_fb_ctrs fb_ctrs = {0};
- struct cppc_freq_invariance *cppc_fi;
- int i, ret;
-
- if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate)
- return;
-
- if (fie_disabled)
- return;
-
- for_each_cpu(i, policy->cpus) {
- cppc_fi = &per_cpu(cppc_freq_inv, i);
- cppc_fi->cpu = i;
- cppc_fi->cpu_data = cpu_data;
- kthread_init_work(&cppc_fi->work, cppc_scale_freq_workfn);
- init_irq_work(&cppc_fi->irq_work, cppc_irq_work);
-
- ret = cppc_get_perf_ctrs(i, &fb_ctrs);
- if (ret) {
- pr_warn("%s: failed to read perf counters: %d\n",
- __func__, ret);
- fie_disabled = true;
- } else {
- cppc_fi->prev_perf_fb_ctrs = fb_ctrs;
- }
- }
-}
-
-static void __init cppc_freq_invariance_init(void)
-{
- struct sched_attr attr = {
- .size = sizeof(struct sched_attr),
- .sched_policy = SCHED_DEADLINE,
- .sched_nice = 0,
- .sched_priority = 0,
- /*
- * Fake (unused) bandwidth; workaround to "fix"
- * priority inheritance.
- */
- .sched_runtime = 1000000,
- .sched_deadline = 10000000,
- .sched_period = 10000000,
- };
- int ret;
-
- if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate)
- return;
-
- if (fie_disabled)
- return;
-
- kworker_fie = kthread_create_worker(0, "cppc_fie");
- if (IS_ERR(kworker_fie))
- return;
-
- ret = sched_setattr_nocheck(kworker_fie->task, &attr);
- if (ret) {
- pr_warn("%s: failed to set SCHED_DEADLINE: %d\n", __func__,
- ret);
- kthread_destroy_worker(kworker_fie);
- return;
- }
-
- /* Register for freq-invariance */
- topology_set_scale_freq_source(&cppc_sftd, cpu_present_mask);
-}
-
-static void cppc_freq_invariance_exit(void)
-{
- struct cppc_freq_invariance *cppc_fi;
- int i;
-
- if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate)
- return;
-
- if (fie_disabled)
- return;
-
- topology_clear_scale_freq_source(SCALE_FREQ_SOURCE_CPPC, cpu_present_mask);
-
- for_each_possible_cpu(i) {
- cppc_fi = &per_cpu(cppc_freq_inv, i);
- irq_work_sync(&cppc_fi->irq_work);
- }
-
- kthread_destroy_worker(kworker_fie);
- kworker_fie = NULL;
-}
-
-#else
-static inline void
-cppc_freq_invariance_policy_init(struct cpufreq_policy *policy,
- struct cppc_cpudata *cpu_data)
-{
-}
-
-static inline void cppc_freq_invariance_init(void)
-{
-}
-
-static inline void cppc_freq_invariance_exit(void)
-{
-}
-#endif /* CONFIG_ACPI_CPPC_CPUFREQ_FIE */
-
/* Callback function used to retrieve the max frequency from DMI */
static void cppc_find_dmi_mhz(const struct dmi_header *dm, void *private)
{
cpu_data->perf_ctrls.desired_perf = caps->highest_perf;
ret = cppc_set_perf(cpu, &cpu_data->perf_ctrls);
- if (ret) {
+ if (ret)
pr_debug("Err setting perf value:%d on CPU:%d. ret:%d\n",
caps->highest_perf, cpu, ret);
- } else {
- cppc_freq_invariance_policy_init(policy, cpu_data);
- }
return ret;
}
return (u32)t1 - (u32)t0;
}
-static int cppc_perf_from_fbctrs(struct cppc_cpudata *cpu_data,
- struct cppc_perf_fb_ctrs fb_ctrs_t0,
- struct cppc_perf_fb_ctrs fb_ctrs_t1)
+static int cppc_get_rate_from_fbctrs(struct cppc_cpudata *cpu_data,
+ struct cppc_perf_fb_ctrs fb_ctrs_t0,
+ struct cppc_perf_fb_ctrs fb_ctrs_t1)
{
u64 delta_reference, delta_delivered;
- u64 reference_perf;
+ u64 reference_perf, delivered_perf;
reference_perf = fb_ctrs_t0.reference_perf;
delta_delivered = get_delta(fb_ctrs_t1.delivered,
fb_ctrs_t0.delivered);
- /* Check to avoid divide-by zero and invalid delivered_perf */
- if (!delta_reference || !delta_delivered)
- return cpu_data->perf_ctrls.desired_perf;
-
- return (reference_perf * delta_delivered) / delta_reference;
-}
-
-static int cppc_get_rate_from_fbctrs(struct cppc_cpudata *cpu_data,
- struct cppc_perf_fb_ctrs fb_ctrs_t0,
- struct cppc_perf_fb_ctrs fb_ctrs_t1)
-{
- u64 delivered_perf;
-
- delivered_perf = cppc_perf_from_fbctrs(cpu_data, fb_ctrs_t0,
- fb_ctrs_t1);
+ /* Check to avoid divide-by zero */
+ if (delta_reference || delta_delivered)
+ delivered_perf = (reference_perf * delta_delivered) /
+ delta_reference;
+ else
+ delivered_perf = cpu_data->perf_ctrls.desired_perf;
return cppc_cpufreq_perf_to_khz(cpu_data, delivered_perf);
}
static int __init cppc_cpufreq_init(void)
{
- int ret;
-
if ((acpi_disabled) || !acpi_cpc_valid())
return -ENODEV;
cppc_check_hisi_workaround();
- ret = cpufreq_register_driver(&cppc_cpufreq_driver);
- if (!ret)
- cppc_freq_invariance_init();
-
- return ret;
+ return cpufreq_register_driver(&cppc_cpufreq_driver);
}
static inline void free_cpu_data(void)
static void __exit cppc_cpufreq_exit(void)
{
- cppc_freq_invariance_exit();
cpufreq_unregister_driver(&cppc_cpufreq_driver);
free_cpu_data();
{}
};
+static bool intel_pstate_hwp_is_enabled(void)
+{
+ u64 value;
+
+ rdmsrl(MSR_PM_ENABLE, value);
+ return !!(value & 0x1);
+}
+
static int __init intel_pstate_init(void)
{
const struct x86_cpu_id *id;
* Avoid enabling HWP for processors without EPP support,
* because that means incomplete HWP implementation which is a
* corner case and supporting it is generally problematic.
+ *
+ * If HWP is enabled already, though, there is no choice but to
+ * deal with it.
*/
- if (!no_hwp && boot_cpu_has(X86_FEATURE_HWP_EPP)) {
+ if ((!no_hwp && boot_cpu_has(X86_FEATURE_HWP_EPP)) ||
+ intel_pstate_hwp_is_enabled()) {
hwp_active++;
hwp_mode_bdw = id->driver_data;
intel_pstate.attr = hwp_cpufreq_attrs;
err = pci_request_mem_regions(pdev, nitrox_driver_name);
if (err) {
pci_disable_device(pdev);
- dev_err(&pdev->dev, "Failed to request mem regions!\n");
return err;
}
pci_set_master(pdev);
if (dma_buf_is_dynamic(attach->dmabuf)) {
dma_resv_lock(attach->dmabuf->resv, NULL);
- ret = dma_buf_pin(attach);
+ ret = dmabuf->ops->pin(attach);
if (ret)
goto err_unlock;
}
err_unpin:
if (dma_buf_is_dynamic(attach->dmabuf))
- dma_buf_unpin(attach);
+ dmabuf->ops->unpin(attach);
err_unlock:
if (dma_buf_is_dynamic(attach->dmabuf))
__unmap_dma_buf(attach, attach->sgt, attach->dir);
if (dma_buf_is_dynamic(attach->dmabuf)) {
- dma_buf_unpin(attach);
+ dmabuf->ops->unpin(attach);
dma_resv_unlock(attach->dmabuf->resv);
}
}
if (dma_buf_is_dynamic(attach->dmabuf)) {
dma_resv_assert_held(attach->dmabuf->resv);
if (!IS_ENABLED(CONFIG_DMABUF_MOVE_NOTIFY)) {
- r = dma_buf_pin(attach);
+ r = attach->dmabuf->ops->pin(attach);
if (r)
return ERR_PTR(r);
}
if (IS_ERR(sg_table) && dma_buf_is_dynamic(attach->dmabuf) &&
!IS_ENABLED(CONFIG_DMABUF_MOVE_NOTIFY))
- dma_buf_unpin(attach);
+ attach->dmabuf->ops->unpin(attach);
if (!IS_ERR(sg_table) && attach->dmabuf->ops->cache_sgt_mapping) {
attach->sgt = sg_table;
#devices
config ALTERA_MSGDMA
tristate "Altera / Intel mSGDMA Engine"
+ depends on HAS_IOMEM
select DMA_ENGINE
help
Enable support for Altera / Intel mSGDMA controller.
config XILINX_ZYNQMP_DPDMA
tristate "Xilinx DPDMA Engine"
+ depends on HAS_IOMEM && OF
select DMA_ENGINE
select DMA_VIRTUAL_CHANNELS
help
}
if (priv->dpdmai_attr.version.major > DPDMAI_VER_MAJOR) {
+ err = -EINVAL;
dev_err(dev, "DPDMAI major version mismatch\n"
"Found %u.%u, supported version is %u.%u\n",
priv->dpdmai_attr.version.major,
}
if (priv->dpdmai_attr.version.minor > DPDMAI_VER_MINOR) {
+ err = -EINVAL;
dev_err(dev, "DPDMAI minor version mismatch\n"
"Found %u.%u, supported version is %u.%u\n",
priv->dpdmai_attr.version.major,
ppriv->store =
dpaa2_io_store_create(DPAA2_QDMA_STORE_SIZE, dev);
if (!ppriv->store) {
+ err = -ENOMEM;
dev_err(dev, "dpaa2_io_store_create() failed\n");
goto err_store;
}
pasid = iommu_sva_get_pasid(sva);
if (pasid == IOMMU_PASID_INVALID) {
iommu_sva_unbind_device(sva);
+ rc = -EINVAL;
goto failed;
}
return rc;
}
+static void idxd_cleanup_interrupts(struct idxd_device *idxd)
+{
+ struct pci_dev *pdev = idxd->pdev;
+ struct idxd_irq_entry *irq_entry;
+ int i, msixcnt;
+
+ msixcnt = pci_msix_vec_count(pdev);
+ if (msixcnt <= 0)
+ return;
+
+ irq_entry = &idxd->irq_entries[0];
+ free_irq(irq_entry->vector, irq_entry);
+
+ for (i = 1; i < msixcnt; i++) {
+
+ irq_entry = &idxd->irq_entries[i];
+ if (idxd->hw.cmd_cap & BIT(IDXD_CMD_RELEASE_INT_HANDLE))
+ idxd_device_release_int_handle(idxd, idxd->int_handles[i],
+ IDXD_IRQ_MSIX);
+ free_irq(irq_entry->vector, irq_entry);
+ }
+
+ idxd_mask_error_interrupts(idxd);
+ pci_free_irq_vectors(pdev);
+}
+
static int idxd_setup_wqs(struct idxd_device *idxd)
{
struct device *dev = &idxd->pdev->dev;
engine->idxd = idxd;
device_initialize(&engine->conf_dev);
engine->conf_dev.parent = &idxd->conf_dev;
+ engine->conf_dev.bus = &dsa_bus_type;
engine->conf_dev.type = &idxd_engine_device_type;
rc = dev_set_name(&engine->conf_dev, "engine%d.%d", idxd->id, engine->id);
if (rc < 0) {
return rc;
}
+static void idxd_cleanup_internals(struct idxd_device *idxd)
+{
+ int i;
+
+ for (i = 0; i < idxd->max_groups; i++)
+ put_device(&idxd->groups[i]->conf_dev);
+ for (i = 0; i < idxd->max_engines; i++)
+ put_device(&idxd->engines[i]->conf_dev);
+ for (i = 0; i < idxd->max_wqs; i++)
+ put_device(&idxd->wqs[i]->conf_dev);
+ destroy_workqueue(idxd->wq);
+}
+
static int idxd_setup_internals(struct idxd_device *idxd)
{
struct device *dev = &idxd->pdev->dev;
dev_dbg(dev, "Loading RO device config\n");
rc = idxd_device_load_config(idxd);
if (rc < 0)
- goto err;
+ goto err_config;
}
rc = idxd_setup_interrupts(idxd);
if (rc)
- goto err;
+ goto err_config;
dev_dbg(dev, "IDXD interrupt setup complete.\n");
dev_dbg(dev, "IDXD device %d probed successfully\n", idxd->id);
return 0;
+ err_config:
+ idxd_cleanup_internals(idxd);
err:
if (device_pasid_enabled(idxd))
idxd_disable_system_pasid(idxd);
return rc;
}
+static void idxd_cleanup(struct idxd_device *idxd)
+{
+ struct device *dev = &idxd->pdev->dev;
+
+ perfmon_pmu_remove(idxd);
+ idxd_cleanup_interrupts(idxd);
+ idxd_cleanup_internals(idxd);
+ if (device_pasid_enabled(idxd))
+ idxd_disable_system_pasid(idxd);
+ iommu_dev_disable_feature(dev, IOMMU_DEV_FEAT_SVA);
+}
+
static int idxd_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
struct device *dev = &pdev->dev;
rc = idxd_register_devices(idxd);
if (rc) {
dev_err(dev, "IDXD sysfs setup failed\n");
- goto err;
+ goto err_dev_register;
}
idxd->state = IDXD_DEV_CONF_READY;
return 0;
+ err_dev_register:
+ idxd_cleanup(idxd);
err:
pci_iounmap(pdev, idxd->reg_base);
err_iomap:
* If the CPU does not support MOVDIR64B or ENQCMDS, there's no point in
* enumerating the device. We can not utilize it.
*/
- if (!boot_cpu_has(X86_FEATURE_MOVDIR64B)) {
+ if (!cpu_feature_enabled(X86_FEATURE_MOVDIR64B)) {
pr_warn("idxd driver failed to load without MOVDIR64B.\n");
return -ENODEV;
}
- if (!boot_cpu_has(X86_FEATURE_ENQCMD))
+ if (!cpu_feature_enabled(X86_FEATURE_ENQCMD))
pr_warn("Platform does not have ENQCMD(S) support.\n");
else
support_enqcmd = true;
static void __exit idxd_exit_module(void)
{
+ idxd_unregister_driver();
pci_unregister_driver(&idxd_pci_driver);
idxd_cdev_remove();
idxd_unregister_bus_type();
}
/**
- * ipu_irq_map() - map an IPU interrupt source to an IRQ number
+ * ipu_irq_unmap() - unmap an IPU interrupt source
* @source: interrupt source bit position (see ipu_irq_map())
* @return: 0 or negative error code
*/
static void mtk_uart_apdma_desc_free(struct virt_dma_desc *vd)
{
- struct dma_chan *chan = vd->tx.chan;
- struct mtk_chan *c = to_mtk_uart_apdma_chan(chan);
-
- kfree(c->desc);
+ kfree(container_of(vd, struct mtk_uart_apdma_desc, vd));
}
static void mtk_uart_apdma_start_tx(struct mtk_chan *c)
static void mtk_uart_apdma_tx_handler(struct mtk_chan *c)
{
- struct mtk_uart_apdma_desc *d = c->desc;
-
mtk_uart_apdma_write(c, VFF_INT_FLAG, VFF_TX_INT_CLR_B);
mtk_uart_apdma_write(c, VFF_INT_EN, VFF_INT_EN_CLR_B);
mtk_uart_apdma_write(c, VFF_EN, VFF_EN_CLR_B);
-
- list_del(&d->vd.node);
- vchan_cookie_complete(&d->vd);
}
static void mtk_uart_apdma_rx_handler(struct mtk_chan *c)
c->rx_status = d->avail_len - cnt;
mtk_uart_apdma_write(c, VFF_RPT, wg);
+}
- list_del(&d->vd.node);
- vchan_cookie_complete(&d->vd);
+static void mtk_uart_apdma_chan_complete_handler(struct mtk_chan *c)
+{
+ struct mtk_uart_apdma_desc *d = c->desc;
+
+ if (d) {
+ list_del(&d->vd.node);
+ vchan_cookie_complete(&d->vd);
+ c->desc = NULL;
+ }
}
static irqreturn_t mtk_uart_apdma_irq_handler(int irq, void *dev_id)
mtk_uart_apdma_rx_handler(c);
else if (c->dir == DMA_MEM_TO_DEV)
mtk_uart_apdma_tx_handler(c);
+ mtk_uart_apdma_chan_complete_handler(c);
spin_unlock_irqrestore(&c->vc.lock, flags);
return IRQ_HANDLED;
return NULL;
/* Now allocate and setup the descriptor */
- d = kzalloc(sizeof(*d), GFP_ATOMIC);
+ d = kzalloc(sizeof(*d), GFP_NOWAIT);
if (!d)
return NULL;
unsigned long flags;
spin_lock_irqsave(&c->vc.lock, flags);
- if (vchan_issue_pending(&c->vc)) {
+ if (vchan_issue_pending(&c->vc) && !c->desc) {
vd = vchan_next_desc(&c->vc);
c->desc = to_mtk_uart_apdma_desc(&vd->tx);
for (i = 0; i < len / period_len; i++) {
desc = pl330_get_desc(pch);
if (!desc) {
+ unsigned long iflags;
+
dev_err(pch->dmac->ddma.dev, "%s:%d Unable to fetch desc\n",
__func__, __LINE__);
if (!first)
return NULL;
- spin_lock_irqsave(&pl330->pool_lock, flags);
+ spin_lock_irqsave(&pl330->pool_lock, iflags);
while (!list_empty(&first->node)) {
desc = list_entry(first->node.next,
list_move_tail(&first->node, &pl330->desc_pool);
- spin_unlock_irqrestore(&pl330->pool_lock, flags);
+ spin_unlock_irqrestore(&pl330->pool_lock, iflags);
return NULL;
}
config QCOM_HIDMA_MGMT
tristate "Qualcomm Technologies HIDMA Management support"
+ depends on HAS_IOMEM
select DMA_ENGINE
help
Enable support for the Qualcomm Technologies HIDMA Management.
hidma_mgmt_of_populate_channels(child);
}
#endif
- return platform_driver_register(&hidma_mgmt_driver);
+ /*
+ * We do not check for return value here, as it is assumed that
+ * platform_driver_register must not fail. The reason for this is that
+ * the (potential) hidma_mgmt_of_populate_channels calls above are not
+ * cleaned up if it does fail, and to do this work is quite
+ * complicated. In particular, various calls of of_address_to_resource,
+ * of_irq_to_resource, platform_device_register_full, of_dma_configure,
+ * and of_msi_configure which then call other functions and so on, must
+ * be cleaned up - this is not a trivial exercise.
+ *
+ * Currently, this module is not intended to be unloaded, and there is
+ * no module_exit function defined which does the needed cleanup. For
+ * this reason, we have to assume success here.
+ */
+ platform_driver_register(&hidma_mgmt_driver);
+ return 0;
}
module_init(hidma_mgmt_init);
MODULE_LICENSE("GPL v2");
config SF_PDMA
tristate "Sifive PDMA controller driver"
+ depends on HAS_IOMEM
select DMA_ENGINE
select DMA_VIRTUAL_CHANNELS
help
/* Enable runtime PM and initialize the device. */
pm_runtime_enable(&pdev->dev);
- ret = pm_runtime_get_sync(&pdev->dev);
+ ret = pm_runtime_resume_and_get(&pdev->dev);
if (ret < 0) {
dev_err(&pdev->dev, "runtime PM get sync failed (%d)\n", ret);
return ret;
kfree(base->lcla_pool.base_unaligned);
+ if (base->lcpa_base)
+ iounmap(base->lcpa_base);
+
if (base->phy_lcpa)
release_mem_region(base->phy_lcpa,
base->lcpa_size);
return -ENOMEM;
}
- ret = pm_runtime_get_sync(dmadev->ddev.dev);
+ ret = pm_runtime_resume_and_get(dmadev->ddev.dev);
if (ret < 0)
return ret;
u32 ccr, id;
int ret;
- ret = pm_runtime_get_sync(dev);
+ ret = pm_runtime_resume_and_get(dev);
if (ret < 0)
return ret;
#define XILINX_DPDMA_CH_VDO 0x020
#define XILINX_DPDMA_CH_PYLD_SZ 0x024
#define XILINX_DPDMA_CH_DESC_ID 0x028
+#define XILINX_DPDMA_CH_DESC_ID_MASK GENMASK(15, 0)
/* DPDMA descriptor fields */
#define XILINX_DPDMA_DESC_CONTROL_PREEMBLE 0xa5
* will be used, but it should be enough.
*/
list_for_each_entry(sw_desc, &desc->descriptors, node)
- sw_desc->hw.desc_id = desc->vdesc.tx.cookie;
+ sw_desc->hw.desc_id = desc->vdesc.tx.cookie
+ & XILINX_DPDMA_CH_DESC_ID_MASK;
sw_desc = list_first_entry(&desc->descriptors,
struct xilinx_dpdma_sw_desc, node);
if (!chan->running || !pending)
goto out;
- desc_id = dpdma_read(chan->reg, XILINX_DPDMA_CH_DESC_ID);
+ desc_id = dpdma_read(chan->reg, XILINX_DPDMA_CH_DESC_ID)
+ & XILINX_DPDMA_CH_DESC_ID_MASK;
/* If the retrigger raced with vsync, retry at the next frame. */
sw_desc = list_first_entry(&pending->descriptors,
*/
static void xilinx_dpdma_disable_irq(struct xilinx_dpdma_device *xdev)
{
- dpdma_write(xdev->reg, XILINX_DPDMA_IDS, XILINX_DPDMA_INTR_ERR_ALL);
+ dpdma_write(xdev->reg, XILINX_DPDMA_IDS, XILINX_DPDMA_INTR_ALL);
dpdma_write(xdev->reg, XILINX_DPDMA_EIDS, XILINX_DPDMA_EINTR_ALL);
}
return dma_get_slave_channel(&xdev->chan[chan_id]->vchan.chan);
}
+static void dpdma_hw_init(struct xilinx_dpdma_device *xdev)
+{
+ unsigned int i;
+ void __iomem *reg;
+
+ /* Disable all interrupts */
+ xilinx_dpdma_disable_irq(xdev);
+
+ /* Stop all channels */
+ for (i = 0; i < ARRAY_SIZE(xdev->chan); i++) {
+ reg = xdev->reg + XILINX_DPDMA_CH_BASE
+ + XILINX_DPDMA_CH_OFFSET * i;
+ dpdma_clr(reg, XILINX_DPDMA_CH_CNTL, XILINX_DPDMA_CH_CNTL_ENABLE);
+ }
+
+ /* Clear the interrupt status registers */
+ dpdma_write(xdev->reg, XILINX_DPDMA_ISR, XILINX_DPDMA_INTR_ALL);
+ dpdma_write(xdev->reg, XILINX_DPDMA_EISR, XILINX_DPDMA_EINTR_ALL);
+}
+
static int xilinx_dpdma_probe(struct platform_device *pdev)
{
struct xilinx_dpdma_device *xdev;
if (IS_ERR(xdev->reg))
return PTR_ERR(xdev->reg);
+ dpdma_hw_init(xdev);
+
xdev->irq = platform_get_irq(pdev, 0);
if (xdev->irq < 0) {
dev_err(xdev->dev, "failed to get platform irq\n");
struct zynqmp_dma_desc_sw *desc;
int i, ret;
- ret = pm_runtime_get_sync(chan->dev);
+ ret = pm_runtime_resume_and_get(chan->dev);
if (ret < 0)
return ret;
edac_dbg(0, " TOP_MEM: 0x%016llx\n", pvt->top_mem);
/* Check first whether TOP_MEM2 is enabled: */
- rdmsrl(MSR_K8_SYSCFG, msr_val);
+ rdmsrl(MSR_AMD64_SYSCFG, msr_val);
if (msr_val & BIT(21)) {
rdmsrl(MSR_K8_TOP_MEM2, pvt->top_mem2);
edac_dbg(0, " TOP_MEM2: 0x%016llx\n", pvt->top_mem2);
int scmi_notification_init(struct scmi_handle *handle);
void scmi_notification_exit(struct scmi_handle *handle);
-
-struct scmi_protocol_handle;
int scmi_register_protocol_events(const struct scmi_handle *handle, u8 proto_id,
const struct scmi_protocol_handle *ph,
const struct scmi_protocol_events *ee);
ret = scpi_send_message(CMD_GET_CLOCK_VALUE, &le_clk_id,
sizeof(le_clk_id), &rate, sizeof(rate));
+ if (ret)
+ return 0;
- return ret ? ret : le32_to_cpu(rate);
+ return le32_to_cpu(rate);
}
static int scpi_clk_set_val(u16 clk_id, unsigned long rate)
if (!msg || !(mem->validation_bits & CPER_MEM_VALID_MODULE_HANDLE))
return 0;
- n = 0;
- len = CPER_REC_LEN - 1;
+ len = CPER_REC_LEN;
dmi_memdev_name(mem->mem_dev_handle, &bank, &device);
if (bank && device)
n = snprintf(msg, len, "DIMM location: %s %s ", bank, device);
"DIMM location: not present. DMI handle: 0x%.4x ",
mem->mem_dev_handle);
- msg[n] = '\0';
return n;
}
BUILD_BUG_ON(ARRAY_SIZE(target) != ARRAY_SIZE(name));
BUILD_BUG_ON(ARRAY_SIZE(target) != ARRAY_SIZE(dt_params[0].params));
+ if (!fdt)
+ return 0;
+
for (i = 0; i < ARRAY_SIZE(dt_params); i++) {
node = fdt_path_offset(fdt, dt_params[i].path);
if (node < 0)
return 0;
/* Skip any leading slashes */
- while (cmdline[i] == L'/' || cmdline[i] == L'\\')
+ while (i < cmdline_len && (cmdline[i] == L'/' || cmdline[i] == L'\\'))
i++;
while (--result_len > 0 && i < cmdline_len) {
return false;
}
- if (!(in->attribute & (EFI_MEMORY_RO | EFI_MEMORY_XP))) {
- pr_warn("Entry attributes invalid: RO and XP bits both cleared\n");
- return false;
- }
-
if (PAGE_SIZE > EFI_PAGE_SIZE &&
(!PAGE_ALIGNED(in->phys_addr) ||
!PAGE_ALIGNED(in->num_pages << EFI_PAGE_SHIFT))) {
{ .compatible = "cdns,gpio-r1p02" },
{ /* sentinel */ },
};
+MODULE_DEVICE_TABLE(of, cdns_of_ids);
static struct platform_driver cdns_gpio_driver = {
.driver = {
return 0;
}
-static int tegra186_irq_set_affinity(struct irq_data *data,
- const struct cpumask *dest,
- bool force)
-{
- if (data->parent_data)
- return irq_chip_set_affinity_parent(data, dest, force);
-
- return -EINVAL;
-}
-
static void tegra186_gpio_irq(struct irq_desc *desc)
{
struct tegra_gpio *gpio = irq_desc_get_handler_data(desc);
gpio->intc.irq_unmask = tegra186_irq_unmask;
gpio->intc.irq_set_type = tegra186_irq_set_type;
gpio->intc.irq_set_wake = tegra186_irq_set_wake;
- gpio->intc.irq_set_affinity = tegra186_irq_set_affinity;
irq = &gpio->gpio.irq;
irq->chip = &gpio->intc;
#include <linux/slab.h>
#include <linux/of_device.h>
-#define WCD_PIN_MASK(p) BIT(p - 1)
+#define WCD_PIN_MASK(p) BIT(p)
#define WCD_REG_DIR_CTL_OFFSET 0x42
#define WCD_REG_VAL_CTL_OFFSET 0x43
#define WCD934X_NPINS 5
}
/**
- * xgpio_of_probe - Probe method for the GPIO device.
+ * xgpio_probe - Probe method for the GPIO device.
* @pdev: pointer to the platform device
*
* Return:
struct amdgpu_df df;
struct amdgpu_ip_block ip_blocks[AMDGPU_MAX_IP_NUM];
+ uint32_t harvest_ip_mask;
int num_ip_blocks;
struct mutex mn_lock;
DECLARE_HASHTABLE(mn_hash, 7);
mmSDMA0_RLC0_RB_CNTL) - mmSDMA0_RLC0_RB_CNTL;
break;
case 1:
- sdma_engine_reg_base = SOC15_REG_OFFSET(SDMA1, 0,
+ sdma_engine_reg_base = SOC15_REG_OFFSET(SDMA0, 0,
mmSDMA1_RLC0_RB_CNTL) - mmSDMA0_RLC0_RB_CNTL;
break;
case 2:
- sdma_engine_reg_base = SOC15_REG_OFFSET(SDMA2, 0,
- mmSDMA2_RLC0_RB_CNTL) - mmSDMA2_RLC0_RB_CNTL;
+ sdma_engine_reg_base = SOC15_REG_OFFSET(SDMA0, 0,
+ mmSDMA2_RLC0_RB_CNTL) - mmSDMA0_RLC0_RB_CNTL;
break;
case 3:
- sdma_engine_reg_base = SOC15_REG_OFFSET(SDMA3, 0,
- mmSDMA3_RLC0_RB_CNTL) - mmSDMA2_RLC0_RB_CNTL;
+ sdma_engine_reg_base = SOC15_REG_OFFSET(SDMA0, 0,
+ mmSDMA3_RLC0_RB_CNTL) - mmSDMA0_RLC0_RB_CNTL;
break;
}
engine_id, queue_id);
uint32_t i = 0, reg;
#undef HQD_N_REGS
-#define HQD_N_REGS (19+6+7+10)
+#define HQD_N_REGS (19+6+7+12)
*dump = kmalloc(HQD_N_REGS*2*sizeof(uint32_t), GFP_KERNEL);
if (*dump == NULL)
{
struct amdgpu_ctx *ctx;
struct amdgpu_ctx_mgr *mgr;
- unsigned long ras_counter;
if (!fpriv)
return -EINVAL;
if (atomic_read(&ctx->guilty))
out->state.flags |= AMDGPU_CTX_QUERY2_FLAGS_GUILTY;
- /*query ue count*/
- ras_counter = amdgpu_ras_query_error_count(adev, false);
- /*ras counter is monotonic increasing*/
- if (ras_counter != ctx->ras_counter_ue) {
- out->state.flags |= AMDGPU_CTX_QUERY2_FLAGS_RAS_UE;
- ctx->ras_counter_ue = ras_counter;
- }
-
- /*query ce count*/
- ras_counter = amdgpu_ras_query_error_count(adev, true);
- if (ras_counter != ctx->ras_counter_ce) {
- out->state.flags |= AMDGPU_CTX_QUERY2_FLAGS_RAS_CE;
- ctx->ras_counter_ce = ras_counter;
- }
-
mutex_unlock(&mgr->lock);
return 0;
}
if (!ip_block_version)
return -EINVAL;
+ switch (ip_block_version->type) {
+ case AMD_IP_BLOCK_TYPE_VCN:
+ if (adev->harvest_ip_mask & AMD_HARVEST_IP_VCN_MASK)
+ return 0;
+ break;
+ case AMD_IP_BLOCK_TYPE_JPEG:
+ if (adev->harvest_ip_mask & AMD_HARVEST_IP_JPEG_MASK)
+ return 0;
+ break;
+ default:
+ break;
+ }
+
DRM_INFO("add ip block number %d <%s>\n", adev->num_ip_blocks,
ip_block_version->funcs->name);
*/
bool amdgpu_device_has_dc_support(struct amdgpu_device *adev)
{
- if (amdgpu_sriov_vf(adev) || adev->enable_virtual_display)
+ if (amdgpu_sriov_vf(adev) ||
+ adev->enable_virtual_display ||
+ (adev->harvest_ip_mask & AMD_HARVEST_IP_DMU_MASK))
return false;
return amdgpu_device_asic_has_dc_support(adev->asic_type);
}
-
static void amdgpu_device_xgmi_reset_func(struct work_struct *__work)
{
struct amdgpu_device *adev =
adev->vm_manager.vm_pte_funcs = NULL;
adev->vm_manager.vm_pte_num_scheds = 0;
adev->gmc.gmc_funcs = NULL;
+ adev->harvest_ip_mask = 0x0;
adev->fence_context = dma_fence_context_alloc(AMDGPU_MAX_RINGS);
bitmap_zero(adev->gfx.pipe_reserve_bitmap, AMDGPU_MAX_COMPUTE_QUEUES);
/* doorbell bar mapping and doorbell index init*/
amdgpu_device_doorbell_init(adev);
- /* if we have > 1 VGA cards, then disable the amdgpu VGA resources */
- /* this will fail for cards that aren't VGA class devices, just
- * ignore it */
- if ((adev->pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA)
- vga_client_register(adev->pdev, adev, NULL, amdgpu_device_vga_set_decode);
-
- if (amdgpu_device_supports_px(ddev)) {
- px = true;
- vga_switcheroo_register_client(adev->pdev,
- &amdgpu_switcheroo_ops, px);
- vga_switcheroo_init_domain_pm_ops(adev->dev, &adev->vga_pm_domain);
- }
-
if (amdgpu_emu_mode == 1) {
/* post the asic on emulation mode */
emu_soc_asic_init(adev);
if (amdgpu_device_cache_pci_state(adev->pdev))
pci_restore_state(pdev);
+ /* if we have > 1 VGA cards, then disable the amdgpu VGA resources */
+ /* this will fail for cards that aren't VGA class devices, just
+ * ignore it */
+ if ((adev->pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA)
+ vga_client_register(adev->pdev, adev, NULL, amdgpu_device_vga_set_decode);
+
+ if (amdgpu_device_supports_px(ddev)) {
+ px = true;
+ vga_switcheroo_register_client(adev->pdev,
+ &amdgpu_switcheroo_ops, px);
+ vga_switcheroo_init_domain_pm_ops(adev->dev, &adev->vga_pm_domain);
+ }
+
if (adev->gmc.xgmi.pending_reset)
queue_delayed_work(system_wq, &mgpu_info.delayed_reset_work,
msecs_to_jiffies(AMDGPU_RESUME_MS));
failed:
amdgpu_vf_error_trans_all(adev);
- if (px)
- vga_switcheroo_fini_domain_pm_ops(adev->dev);
failed_unmap:
iounmap(adev->rmmio);
r = amdgpu_ib_ring_tests(tmp_adev);
if (r) {
dev_err(tmp_adev->dev, "ib ring test failed (%d).\n", r);
- r = amdgpu_device_ip_suspend(tmp_adev);
need_full_reset = true;
r = -EAGAIN;
goto end;
return -EINVAL;
}
+void amdgpu_discovery_harvest_ip(struct amdgpu_device *adev)
+{
+ struct binary_header *bhdr;
+ struct harvest_table *harvest_info;
+ int i;
+
+ bhdr = (struct binary_header *)adev->mman.discovery_bin;
+ harvest_info = (struct harvest_table *)(adev->mman.discovery_bin +
+ le16_to_cpu(bhdr->table_list[HARVEST_INFO].offset));
+
+ for (i = 0; i < 32; i++) {
+ if (le32_to_cpu(harvest_info->list[i].hw_id) == 0)
+ break;
+
+ switch (le32_to_cpu(harvest_info->list[i].hw_id)) {
+ case VCN_HWID:
+ adev->harvest_ip_mask |= AMD_HARVEST_IP_VCN_MASK;
+ adev->harvest_ip_mask |= AMD_HARVEST_IP_JPEG_MASK;
+ break;
+ case DMU_HWID:
+ adev->harvest_ip_mask |= AMD_HARVEST_IP_DMU_MASK;
+ break;
+ default:
+ break;
+ }
+ }
+}
+
int amdgpu_discovery_get_gfx_info(struct amdgpu_device *adev)
{
struct binary_header *bhdr;
void amdgpu_discovery_fini(struct amdgpu_device *adev);
int amdgpu_discovery_reg_base_init(struct amdgpu_device *adev);
+void amdgpu_discovery_harvest_ip(struct amdgpu_device *adev);
int amdgpu_discovery_get_ip_version(struct amdgpu_device *adev, int hw_id,
int *major, int *minor, int *revision);
int amdgpu_discovery_get_gfx_info(struct amdgpu_device *adev);
return 0;
}
+static void get_block_dimensions(unsigned int block_log2, unsigned int cpp,
+ unsigned int *width, unsigned int *height)
+{
+ unsigned int cpp_log2 = ilog2(cpp);
+ unsigned int pixel_log2 = block_log2 - cpp_log2;
+ unsigned int width_log2 = (pixel_log2 + 1) / 2;
+ unsigned int height_log2 = pixel_log2 - width_log2;
+
+ *width = 1 << width_log2;
+ *height = 1 << height_log2;
+}
+
+static unsigned int get_dcc_block_size(uint64_t modifier, bool rb_aligned,
+ bool pipe_aligned)
+{
+ unsigned int ver = AMD_FMT_MOD_GET(TILE_VERSION, modifier);
+
+ switch (ver) {
+ case AMD_FMT_MOD_TILE_VER_GFX9: {
+ /*
+ * TODO: for pipe aligned we may need to check the alignment of the
+ * total size of the surface, which may need to be bigger than the
+ * natural alignment due to some HW workarounds
+ */
+ return max(10 + (rb_aligned ? (int)AMD_FMT_MOD_GET(RB, modifier) : 0), 12);
+ }
+ case AMD_FMT_MOD_TILE_VER_GFX10:
+ case AMD_FMT_MOD_TILE_VER_GFX10_RBPLUS: {
+ int pipes_log2 = AMD_FMT_MOD_GET(PIPE_XOR_BITS, modifier);
+
+ if (ver == AMD_FMT_MOD_TILE_VER_GFX10_RBPLUS && pipes_log2 > 1 &&
+ AMD_FMT_MOD_GET(PACKERS, modifier) == pipes_log2)
+ ++pipes_log2;
+
+ return max(8 + (pipe_aligned ? pipes_log2 : 0), 12);
+ }
+ default:
+ return 0;
+ }
+}
+
+static int amdgpu_display_verify_plane(struct amdgpu_framebuffer *rfb, int plane,
+ const struct drm_format_info *format,
+ unsigned int block_width, unsigned int block_height,
+ unsigned int block_size_log2)
+{
+ unsigned int width = rfb->base.width /
+ ((plane && plane < format->num_planes) ? format->hsub : 1);
+ unsigned int height = rfb->base.height /
+ ((plane && plane < format->num_planes) ? format->vsub : 1);
+ unsigned int cpp = plane < format->num_planes ? format->cpp[plane] : 1;
+ unsigned int block_pitch = block_width * cpp;
+ unsigned int min_pitch = ALIGN(width * cpp, block_pitch);
+ unsigned int block_size = 1 << block_size_log2;
+ uint64_t size;
+
+ if (rfb->base.pitches[plane] % block_pitch) {
+ drm_dbg_kms(rfb->base.dev,
+ "pitch %d for plane %d is not a multiple of block pitch %d\n",
+ rfb->base.pitches[plane], plane, block_pitch);
+ return -EINVAL;
+ }
+ if (rfb->base.pitches[plane] < min_pitch) {
+ drm_dbg_kms(rfb->base.dev,
+ "pitch %d for plane %d is less than minimum pitch %d\n",
+ rfb->base.pitches[plane], plane, min_pitch);
+ return -EINVAL;
+ }
+
+ /* Force at least natural alignment. */
+ if (rfb->base.offsets[plane] % block_size) {
+ drm_dbg_kms(rfb->base.dev,
+ "offset 0x%x for plane %d is not a multiple of block pitch 0x%x\n",
+ rfb->base.offsets[plane], plane, block_size);
+ return -EINVAL;
+ }
+
+ size = rfb->base.offsets[plane] +
+ (uint64_t)rfb->base.pitches[plane] / block_pitch *
+ block_size * DIV_ROUND_UP(height, block_height);
+
+ if (rfb->base.obj[0]->size < size) {
+ drm_dbg_kms(rfb->base.dev,
+ "BO size 0x%zx is less than 0x%llx required for plane %d\n",
+ rfb->base.obj[0]->size, size, plane);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+
+static int amdgpu_display_verify_sizes(struct amdgpu_framebuffer *rfb)
+{
+ const struct drm_format_info *format_info = drm_format_info(rfb->base.format->format);
+ uint64_t modifier = rfb->base.modifier;
+ int ret;
+ unsigned int i, block_width, block_height, block_size_log2;
+
+ if (!rfb->base.dev->mode_config.allow_fb_modifiers)
+ return 0;
+
+ for (i = 0; i < format_info->num_planes; ++i) {
+ if (modifier == DRM_FORMAT_MOD_LINEAR) {
+ block_width = 256 / format_info->cpp[i];
+ block_height = 1;
+ block_size_log2 = 8;
+ } else {
+ int swizzle = AMD_FMT_MOD_GET(TILE, modifier);
+
+ switch ((swizzle & ~3) + 1) {
+ case DC_SW_256B_S:
+ block_size_log2 = 8;
+ break;
+ case DC_SW_4KB_S:
+ case DC_SW_4KB_S_X:
+ block_size_log2 = 12;
+ break;
+ case DC_SW_64KB_S:
+ case DC_SW_64KB_S_T:
+ case DC_SW_64KB_S_X:
+ block_size_log2 = 16;
+ break;
+ default:
+ drm_dbg_kms(rfb->base.dev,
+ "Swizzle mode with unknown block size: %d\n", swizzle);
+ return -EINVAL;
+ }
+
+ get_block_dimensions(block_size_log2, format_info->cpp[i],
+ &block_width, &block_height);
+ }
+
+ ret = amdgpu_display_verify_plane(rfb, i, format_info,
+ block_width, block_height, block_size_log2);
+ if (ret)
+ return ret;
+ }
+
+ if (AMD_FMT_MOD_GET(DCC, modifier)) {
+ if (AMD_FMT_MOD_GET(DCC_RETILE, modifier)) {
+ block_size_log2 = get_dcc_block_size(modifier, false, false);
+ get_block_dimensions(block_size_log2 + 8, format_info->cpp[0],
+ &block_width, &block_height);
+ ret = amdgpu_display_verify_plane(rfb, i, format_info,
+ block_width, block_height,
+ block_size_log2);
+ if (ret)
+ return ret;
+
+ ++i;
+ block_size_log2 = get_dcc_block_size(modifier, true, true);
+ } else {
+ bool pipe_aligned = AMD_FMT_MOD_GET(DCC_PIPE_ALIGN, modifier);
+
+ block_size_log2 = get_dcc_block_size(modifier, true, pipe_aligned);
+ }
+ get_block_dimensions(block_size_log2 + 8, format_info->cpp[0],
+ &block_width, &block_height);
+ ret = amdgpu_display_verify_plane(rfb, i, format_info,
+ block_width, block_height, block_size_log2);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
static int amdgpu_display_get_fb_info(const struct amdgpu_framebuffer *amdgpu_fb,
uint64_t *tiling_flags, bool *tmz_surface)
{
return 0;
err:
- drm_err(dev, "Failed to init gem fb: %d\n", ret);
+ drm_dbg_kms(dev, "Failed to init gem fb: %d\n", ret);
rfb->base.obj[0] = NULL;
return ret;
}
int ret;
rfb->base.obj[0] = obj;
-
- /* Verify that bo size can fit the fb size. */
- ret = drm_gem_fb_init_with_funcs(dev, &rfb->base, file_priv, mode_cmd,
- &amdgpu_fb_funcs);
+ drm_helper_mode_fill_fb_struct(dev, &rfb->base, mode_cmd);
+ ret = drm_framebuffer_init(dev, &rfb->base, &amdgpu_fb_funcs);
if (ret)
goto err;
/* Verify that the modifier is supported. */
return 0;
err:
- drm_err(dev, "Failed to verify and init gem fb: %d\n", ret);
+ drm_dbg_kms(dev, "Failed to verify and init gem fb: %d\n", ret);
rfb->base.obj[0] = NULL;
return ret;
}
}
}
- for (i = 1; i < rfb->base.format->num_planes; ++i) {
+ ret = amdgpu_display_verify_sizes(rfb);
+ if (ret)
+ return ret;
+
+ for (i = 0; i < rfb->base.format->num_planes; ++i) {
drm_gem_object_get(rfb->base.obj[0]);
- drm_gem_object_put(rfb->base.obj[i]);
rfb->base.obj[i] = rfb->base.obj[0];
}
domains = amdgpu_display_supported_domains(drm_to_adev(dev), bo->flags);
if (obj->import_attach && !(domains & AMDGPU_GEM_DOMAIN_GTT)) {
drm_dbg_kms(dev, "Cannot create framebuffer from imported dma_buf\n");
+ drm_gem_object_put(obj);
return ERR_PTR(-EINVAL);
}
}
}
}
- return r;
+ return 0;
}
int amdgpu_display_resume_helper(struct amdgpu_device *adev)
amdgpu_device_baco_exit(drm_dev);
}
ret = amdgpu_device_resume(drm_dev, false);
+ if (ret)
+ return ret;
+
if (amdgpu_device_supports_px(drm_dev))
drm_dev->switch_power_state = DRM_SWITCH_POWER_ON;
adev->in_runpm = false;
static int amdgpu_fbdev_destroy(struct drm_device *dev, struct amdgpu_fbdev *rfbdev)
{
struct amdgpu_framebuffer *rfb = &rfbdev->rfb;
+ int i;
drm_fb_helper_unregister_fbi(&rfbdev->helper);
if (rfb->base.obj[0]) {
+ for (i = 0; i < rfb->base.format->num_planes; i++)
+ drm_gem_object_put(rfb->base.obj[0]);
amdgpufb_destroy_pinned_object(rfb->base.obj[0]);
rfb->base.obj[0] = NULL;
drm_framebuffer_unregister_private(&rfb->base);
int amdgpu_fru_get_product_info(struct amdgpu_device *adev)
{
unsigned char buff[34];
- int addrptr = 0, size = 0;
+ int addrptr, size;
+ int len;
if (!is_fru_eeprom_supported(adev))
return 0;
/* If algo exists, it means that the i2c_adapter's initialized */
if (!adev->pm.smu_i2c.algo) {
DRM_WARN("Cannot access FRU, EEPROM accessor not initialized");
- return 0;
+ return -ENODEV;
}
/* There's a lot of repetition here. This is due to the FRU having
size = amdgpu_fru_read_eeprom(adev, addrptr, buff);
if (size < 1) {
DRM_ERROR("Failed to read FRU Manufacturer, ret:%d", size);
- return size;
+ return -EINVAL;
}
/* Increment the addrptr by the size of the field, and 1 due to the
size = amdgpu_fru_read_eeprom(adev, addrptr, buff);
if (size < 1) {
DRM_ERROR("Failed to read FRU product name, ret:%d", size);
- return size;
+ return -EINVAL;
}
+ len = size;
/* Product name should only be 32 characters. Any more,
* and something could be wrong. Cap it at 32 to be safe
*/
- if (size > 32) {
+ if (len >= sizeof(adev->product_name)) {
DRM_WARN("FRU Product Number is larger than 32 characters. This is likely a mistake");
- size = 32;
+ len = sizeof(adev->product_name) - 1;
}
/* Start at 2 due to buff using fields 0 and 1 for the address */
- memcpy(adev->product_name, &buff[2], size);
- adev->product_name[size] = '\0';
+ memcpy(adev->product_name, &buff[2], len);
+ adev->product_name[len] = '\0';
addrptr += size + 1;
size = amdgpu_fru_read_eeprom(adev, addrptr, buff);
if (size < 1) {
DRM_ERROR("Failed to read FRU product number, ret:%d", size);
- return size;
+ return -EINVAL;
}
+ len = size;
/* Product number should only be 16 characters. Any more,
* and something could be wrong. Cap it at 16 to be safe
*/
- if (size > 16) {
+ if (len >= sizeof(adev->product_number)) {
DRM_WARN("FRU Product Number is larger than 16 characters. This is likely a mistake");
- size = 16;
+ len = sizeof(adev->product_number) - 1;
}
- memcpy(adev->product_number, &buff[2], size);
- adev->product_number[size] = '\0';
+ memcpy(adev->product_number, &buff[2], len);
+ adev->product_number[len] = '\0';
addrptr += size + 1;
size = amdgpu_fru_read_eeprom(adev, addrptr, buff);
if (size < 1) {
DRM_ERROR("Failed to read FRU product version, ret:%d", size);
- return size;
+ return -EINVAL;
}
addrptr += size + 1;
if (size < 1) {
DRM_ERROR("Failed to read FRU serial number, ret:%d", size);
- return size;
+ return -EINVAL;
}
+ len = size;
/* Serial number should only be 16 characters. Any more,
* and something could be wrong. Cap it at 16 to be safe
*/
- if (size > 16) {
+ if (len >= sizeof(adev->serial)) {
DRM_WARN("FRU Serial Number is larger than 16 characters. This is likely a mistake");
- size = 16;
+ len = sizeof(adev->serial) - 1;
}
- memcpy(adev->serial, &buff[2], size);
- adev->serial[size] = '\0';
+ memcpy(adev->serial, &buff[2], len);
+ adev->serial[len] = '\0';
return 0;
}
/* Check if we have an idle VMID */
i = 0;
list_for_each_entry((*idle), &id_mgr->ids_lru, list) {
- fences[i] = amdgpu_sync_peek_fence(&(*idle)->active, ring);
+ /* Don't use per engine and per process VMID at the same time */
+ struct amdgpu_ring *r = adev->vm_manager.concurrent_flush ?
+ NULL : ring;
+
+ fences[i] = amdgpu_sync_peek_fence(&(*idle)->active, r);
if (!fences[i])
break;
++i;
if (updates && (*id)->flushed_updates &&
updates->context == (*id)->flushed_updates->context &&
!dma_fence_is_later(updates, (*id)->flushed_updates))
- updates = NULL;
+ updates = NULL;
if ((*id)->owner != vm->immediate.fence_context ||
job->vm_pd_addr != (*id)->pd_gpu_addr ||
!dma_fence_is_signaled((*id)->last_flush))) {
struct dma_fence *tmp;
+ /* Don't use per engine and per process VMID at the same time */
+ if (adev->vm_manager.concurrent_flush)
+ ring = NULL;
+
/* to prevent one context starved by another context */
(*id)->pd_gpu_addr = 0;
tmp = amdgpu_sync_peek_fence(&(*id)->active, ring);
if (updates && (!flushed || dma_fence_is_later(updates, flushed)))
needs_flush = true;
- /* Concurrent flushes are only possible starting with Vega10 and
- * are broken on Navi10 and Navi14.
- */
- if (needs_flush && (adev->asic_type < CHIP_VEGA10 ||
- adev->asic_type == CHIP_NAVI10 ||
- adev->asic_type == CHIP_NAVI14))
+ if (needs_flush && !adev->vm_manager.concurrent_flush)
continue;
/* Good, we can use this VMID. Remember this submission as
kfree(ubo->metadata);
}
- kfree(bo);
+ kvfree(bo);
}
/**
BUG_ON(bp->bo_ptr_size < sizeof(struct amdgpu_bo));
*bo_ptr = NULL;
- bo = kzalloc(bp->bo_ptr_size, GFP_KERNEL);
+ bo = kvzalloc(bp->bo_ptr_size, GFP_KERNEL);
if (bo == NULL)
return -ENOMEM;
drm_gem_private_object_init(adev_to_drm(adev), &bo->tbo.base, size);
uint64_t ring_mem_mc_addr;
void *ring_mem_handle;
uint32_t ring_size;
+ uint32_t ring_wptr;
};
/* More registers may will be supported */
*addr += mm_cur->start & ~PAGE_MASK;
num_dw = ALIGN(adev->mman.buffer_funcs->copy_num_dw, 8);
- num_bytes = num_pages * 8;
+ num_bytes = num_pages * 8 * AMDGPU_GPU_PAGES_IN_CPU_PAGE;
r = amdgpu_job_alloc_with_ib(adev, num_dw * 4 + num_bytes,
AMDGPU_IB_POOL_DELAYED, &job);
if (gtt && gtt->userptr) {
amdgpu_ttm_tt_set_user_pages(ttm, NULL);
kfree(ttm->sg);
+ ttm->sg = NULL;
ttm->page_flags &= ~TTM_PAGE_FLAG_SG;
return;
}
{
unsigned i;
+ /* Concurrent flushes are only possible starting with Vega10 and
+ * are broken on Navi10 and Navi14.
+ */
+ adev->vm_manager.concurrent_flush = !(adev->asic_type < CHIP_VEGA10 ||
+ adev->asic_type == CHIP_NAVI10 ||
+ adev->asic_type == CHIP_NAVI14);
amdgpu_vmid_mgr_init(adev);
adev->vm_manager.fence_context =
/* Handling of VMIDs */
struct amdgpu_vmid_mgr id_mgr[AMDGPU_MAX_VMHUBS];
unsigned int first_kfd_vmid;
+ bool concurrent_flush;
/* Handling of VM fences */
u64 fence_context;
#define mmGC_THROTTLE_CTRL_Sienna_Cichlid 0x2030
#define mmGC_THROTTLE_CTRL_Sienna_Cichlid_BASE_IDX 0
+#define mmRLC_SPARE_INT_0_Sienna_Cichlid 0x4ca5
+#define mmRLC_SPARE_INT_0_Sienna_Cichlid_BASE_IDX 1
+
#define GFX_RLCG_GC_WRITE_OLD (0x8 << 28)
#define GFX_RLCG_GC_WRITE (0x0 << 28)
#define GFX_RLCG_GC_READ (0x1 << 28)
SOC15_REG_GOLDEN_VALUE(GC, 0, mmDB_DEBUG, 0xffffffff, 0x20000000),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmDB_DEBUG2, 0xffffffff, 0x00000420),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmDB_DEBUG3, 0xffffffff, 0x00000200),
- SOC15_REG_GOLDEN_VALUE(GC, 0, mmDB_DEBUG4, 0xffffffff, 0x04800000),
+ SOC15_REG_GOLDEN_VALUE(GC, 0, mmDB_DEBUG4, 0xffffffff, 0x04900000),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmDB_DFSM_TILES_IN_FLIGHT, 0x0000ffff, 0x0000003f),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmDB_LAST_OF_BURST_CONFIG, 0xffffffff, 0x03860204),
+ SOC15_REG_GOLDEN_VALUE(GC, 0, mmGB_ADDR_CONFIG, 0x0c1800ff, 0x00000044),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmGCR_GENERAL_CNTL, 0x1ff0ffff, 0x00000500),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmGE_PRIV_CONTROL, 0x00007fff, 0x000001fe),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmGL1_PIPE_STEER, 0xffffffff, 0xe4e4e4e4),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmPA_SC_ENHANCE_2, 0x00000820, 0x00000820),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmPA_SC_LINE_STIPPLE_STATE, 0x0000ff0f, 0x00000000),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmRMI_SPARE, 0xffffffff, 0xffff3101),
+ SOC15_REG_GOLDEN_VALUE(GC, 0, mmSPI_CONFIG_CNTL_1, 0x001f0000, 0x00070104),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmSQ_ALU_CLK_CTRL, 0xffffffff, 0xffffffff),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmSQ_ARB_CONFIG, 0x00000133, 0x00000130),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmSQ_LDS_CLK_CTRL, 0xffffffff, 0xffffffff),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmTA_CNTL_AUX, 0xfff7ffff, 0x01030000),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmTCP_CNTL, 0xffdf80ff, 0x479c0010),
- SOC15_REG_GOLDEN_VALUE(GC, 0, mmUTCL1_CTRL, 0xffffffff, 0x00800000)
+ SOC15_REG_GOLDEN_VALUE(GC, 0, mmUTCL1_CTRL, 0xffffffff, 0x00c00000)
};
static bool gfx_v10_is_rlcg_rw(struct amdgpu_device *adev, u32 offset, uint32_t *flag, bool write)
(adev->reg_offset[GC_HWIP][0][mmSCRATCH_REG0_BASE_IDX] + mmSCRATCH_REG2) * 4;
scratch_reg3 = adev->rmmio +
(adev->reg_offset[GC_HWIP][0][mmSCRATCH_REG1_BASE_IDX] + mmSCRATCH_REG3) * 4;
- spare_int = adev->rmmio +
- (adev->reg_offset[GC_HWIP][0][mmRLC_SPARE_INT_BASE_IDX] + mmRLC_SPARE_INT) * 4;
+
+ if (adev->asic_type >= CHIP_SIENNA_CICHLID) {
+ spare_int = adev->rmmio +
+ (adev->reg_offset[GC_HWIP][0][mmRLC_SPARE_INT_0_Sienna_Cichlid_BASE_IDX]
+ + mmRLC_SPARE_INT_0_Sienna_Cichlid) * 4;
+ } else {
+ spare_int = adev->rmmio +
+ (adev->reg_offset[GC_HWIP][0][mmRLC_SPARE_INT_BASE_IDX] + mmRLC_SPARE_INT) * 4;
+ }
grbm_cntl = adev->reg_offset[GC_HWIP][0][mmGRBM_GFX_CNTL_BASE_IDX] + mmGRBM_GFX_CNTL;
grbm_idx = adev->reg_offset[GC_HWIP][0][mmGRBM_GFX_INDEX_BASE_IDX] + mmGRBM_GFX_INDEX;
if (ring->use_doorbell) {
WREG32_SOC15(GC, 0, mmCP_MEC_DOORBELL_RANGE_LOWER,
(adev->doorbell_index.kiq * 2) << 2);
+ /* If GC has entered CGPG, ringing doorbell > first page doesn't
+ * wakeup GC. Enlarge CP_MEC_DOORBELL_RANGE_UPPER to workaround
+ * this issue.
+ */
WREG32_SOC15(GC, 0, mmCP_MEC_DOORBELL_RANGE_UPPER,
- (adev->doorbell_index.userqueue_end * 2) << 2);
+ (adev->doorbell.size - 4));
}
WREG32_SOC15(GC, 0, mmCP_HQD_PQ_DOORBELL_CONTROL,
if (amdgpu_sriov_vf(adev)) {
gfx_v10_0_cp_gfx_enable(adev, false);
/* Program KIQ position of RLC_CP_SCHEDULERS during destroy */
- tmp = RREG32_SOC15(GC, 0, mmRLC_CP_SCHEDULERS);
- tmp &= 0xffffff00;
- WREG32_SOC15(GC, 0, mmRLC_CP_SCHEDULERS, tmp);
+ if (adev->asic_type >= CHIP_SIENNA_CICHLID) {
+ tmp = RREG32_SOC15(GC, 0, mmRLC_CP_SCHEDULERS_Sienna_Cichlid);
+ tmp &= 0xffffff00;
+ WREG32_SOC15(GC, 0, mmRLC_CP_SCHEDULERS_Sienna_Cichlid, tmp);
+ } else {
+ tmp = RREG32_SOC15(GC, 0, mmRLC_CP_SCHEDULERS);
+ tmp &= 0xffffff00;
+ WREG32_SOC15(GC, 0, mmRLC_CP_SCHEDULERS, tmp);
+ }
return 0;
}
if (ring->use_doorbell) {
WREG32_SOC15(GC, 0, mmCP_MEC_DOORBELL_RANGE_LOWER,
(adev->doorbell_index.kiq * 2) << 2);
+ /* If GC has entered CGPG, ringing doorbell > first page doesn't
+ * wakeup GC. Enlarge CP_MEC_DOORBELL_RANGE_UPPER to workaround
+ * this issue.
+ */
WREG32_SOC15(GC, 0, mmCP_MEC_DOORBELL_RANGE_UPPER,
- (adev->doorbell_index.userqueue_end * 2) << 2);
+ (adev->doorbell.size - 4));
}
WREG32_SOC15_RLC(GC, 0, mmCP_HQD_PQ_DOORBELL_CONTROL,
amdgpu_gfx_rlc_enter_safe_mode(adev);
/* Enable 3D CGCG/CGLS */
- if (enable && (adev->cg_flags & AMD_CG_SUPPORT_GFX_3D_CGCG)) {
+ if (enable) {
/* write cmd to clear cgcg/cgls ov */
def = data = RREG32_SOC15(GC, 0, mmRLC_CGTT_MGCG_OVERRIDE);
/* unset CGCG override */
/* enable 3Dcgcg FSM(0x0000363f) */
def = RREG32_SOC15(GC, 0, mmRLC_CGCG_CGLS_CTRL_3D);
- data = (0x36 << RLC_CGCG_CGLS_CTRL_3D__CGCG_GFX_IDLE_THRESHOLD__SHIFT) |
- RLC_CGCG_CGLS_CTRL_3D__CGCG_EN_MASK;
+ if (adev->cg_flags & AMD_CG_SUPPORT_GFX_3D_CGCG)
+ data = (0x36 << RLC_CGCG_CGLS_CTRL_3D__CGCG_GFX_IDLE_THRESHOLD__SHIFT) |
+ RLC_CGCG_CGLS_CTRL_3D__CGCG_EN_MASK;
+ else
+ data = 0x0 << RLC_CGCG_CGLS_CTRL_3D__CGCG_GFX_IDLE_THRESHOLD__SHIFT;
+
if (adev->cg_flags & AMD_CG_SUPPORT_GFX_3D_CGLS)
data |= (0x000F << RLC_CGCG_CGLS_CTRL_3D__CGLS_REP_COMPANSAT_DELAY__SHIFT) |
RLC_CGCG_CGLS_CTRL_3D__CGLS_EN_MASK;
MODULE_FIRMWARE("amdgpu/polaris11_mc.bin");
MODULE_FIRMWARE("amdgpu/polaris10_mc.bin");
MODULE_FIRMWARE("amdgpu/polaris12_mc.bin");
+MODULE_FIRMWARE("amdgpu/polaris12_32_mc.bin");
MODULE_FIRMWARE("amdgpu/polaris11_k_mc.bin");
MODULE_FIRMWARE("amdgpu/polaris10_k_mc.bin");
MODULE_FIRMWARE("amdgpu/polaris12_k_mc.bin");
chip_name = "polaris10";
break;
case CHIP_POLARIS12:
- if (ASICID_IS_P23(adev->pdev->device, adev->pdev->revision))
+ if (ASICID_IS_P23(adev->pdev->device, adev->pdev->revision)) {
chip_name = "polaris12_k";
- else
- chip_name = "polaris12";
+ } else {
+ WREG32(mmMC_SEQ_IO_DEBUG_INDEX, ixMC_IO_DEBUG_UP_159);
+ /* Polaris12 32bit ASIC needs a special MC firmware */
+ if (RREG32(mmMC_SEQ_IO_DEBUG_DATA) == 0x05b4dc40)
+ chip_name = "polaris12_32";
+ else
+ chip_name = "polaris12";
+ }
break;
case CHIP_FIJI:
case CHIP_CARRIZO:
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ cancel_delayed_work_sync(&adev->vcn.idle_work);
+
if (adev->jpeg.cur_state != AMD_PG_STATE_GATE &&
RREG32_SOC15(JPEG, 0, mmUVD_JRBC_STATUS))
jpeg_v2_0_set_powergating_state(adev, AMD_PG_STATE_GATE);
static int jpeg_v2_5_hw_fini(void *handle)
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
- struct amdgpu_ring *ring;
int i;
+ cancel_delayed_work_sync(&adev->vcn.idle_work);
+
for (i = 0; i < adev->jpeg.num_jpeg_inst; ++i) {
if (adev->jpeg.harvest_config & (1 << i))
continue;
- ring = &adev->jpeg.inst[i].ring_dec;
if (adev->jpeg.cur_state != AMD_PG_STATE_GATE &&
RREG32_SOC15(JPEG, i, mmUVD_JRBC_STATUS))
jpeg_v2_5_set_powergating_state(adev, AMD_PG_STATE_GATE);
-
- ring->sched.ready = false;
}
return 0;
static int jpeg_v3_0_hw_fini(void *handle)
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
- struct amdgpu_ring *ring;
- ring = &adev->jpeg.inst->ring_dec;
+ cancel_delayed_work_sync(&adev->vcn.idle_work);
+
if (adev->jpeg.cur_state != AMD_PG_STATE_GATE &&
RREG32_SOC15(JPEG, 0, mmUVD_JRBC_STATUS))
jpeg_v3_0_set_powergating_state(adev, AMD_PG_STATE_GATE);
- ring->sched.ready = false;
-
return 0;
}
.funcs = &nv_common_ip_funcs,
};
+static bool nv_is_headless_sku(struct pci_dev *pdev)
+{
+ if ((pdev->device == 0x731E &&
+ (pdev->revision == 0xC6 || pdev->revision == 0xC7)) ||
+ (pdev->device == 0x7340 && pdev->revision == 0xC9) ||
+ (pdev->device == 0x7360 && pdev->revision == 0xC7))
+ return true;
+ return false;
+}
+
static int nv_reg_base_init(struct amdgpu_device *adev)
{
int r;
goto legacy_init;
}
+ amdgpu_discovery_harvest_ip(adev);
+ if (nv_is_headless_sku(adev->pdev)) {
+ adev->harvest_ip_mask |= AMD_HARVEST_IP_VCN_MASK;
+ adev->harvest_ip_mask |= AMD_HARVEST_IP_JPEG_MASK;
+ }
+
return 0;
}
adev->virt.ops = &xgpu_nv_virt_ops;
}
-static bool nv_is_headless_sku(struct pci_dev *pdev)
-{
- if ((pdev->device == 0x731E &&
- (pdev->revision == 0xC6 || pdev->revision == 0xC7)) ||
- (pdev->device == 0x7340 && pdev->revision == 0xC9) ||
- (pdev->device == 0x7360 && pdev->revision == 0xC7))
- return true;
- return false;
-}
-
int nv_set_ip_blocks(struct amdgpu_device *adev)
{
int r;
if (adev->firmware.load_type == AMDGPU_FW_LOAD_DIRECT &&
!amdgpu_sriov_vf(adev))
amdgpu_device_ip_block_add(adev, &smu_v11_0_ip_block);
- if (!nv_is_headless_sku(adev->pdev))
- amdgpu_device_ip_block_add(adev, &vcn_v2_0_ip_block);
+ amdgpu_device_ip_block_add(adev, &vcn_v2_0_ip_block);
amdgpu_device_ip_block_add(adev, &jpeg_v2_0_ip_block);
if (adev->enable_mes)
amdgpu_device_ip_block_add(adev, &mes_v10_1_ip_block);
if (adev->firmware.load_type == AMDGPU_FW_LOAD_DIRECT &&
!amdgpu_sriov_vf(adev))
amdgpu_device_ip_block_add(adev, &smu_v11_0_ip_block);
- if (!nv_is_headless_sku(adev->pdev))
- amdgpu_device_ip_block_add(adev, &vcn_v2_0_ip_block);
+ amdgpu_device_ip_block_add(adev, &vcn_v2_0_ip_block);
if (!amdgpu_sriov_vf(adev))
amdgpu_device_ip_block_add(adev, &jpeg_v2_0_ip_block);
break;
amdgpu_device_ip_block_add(adev, &vcn_v3_0_ip_block);
if (!amdgpu_sriov_vf(adev))
amdgpu_device_ip_block_add(adev, &jpeg_v3_0_ip_block);
-
if (adev->enable_mes)
amdgpu_device_ip_block_add(adev, &mes_v10_1_ip_block);
break;
return -EINVAL;
}
+ if (adev->harvest_ip_mask & AMD_HARVEST_IP_VCN_MASK)
+ adev->pg_flags &= ~(AMD_PG_SUPPORT_VCN |
+ AMD_PG_SUPPORT_VCN_DPG |
+ AMD_PG_SUPPORT_JPEG);
+
if (amdgpu_sriov_vf(adev)) {
amdgpu_virt_init_setting(adev);
xgpu_nv_mailbox_set_irq_funcs(adev);
struct amdgpu_device *adev = psp->adev;
if (amdgpu_sriov_vf(adev))
- data = RREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_102);
+ data = psp->km_ring.ring_wptr;
else
data = RREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_67);
if (amdgpu_sriov_vf(adev)) {
WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_102, value);
WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_101, GFX_CTRL_CMD_ID_CONSUME_CMD);
+ psp->km_ring.ring_wptr = value;
} else
WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_67, value);
}
struct amdgpu_device *adev = psp->adev;
if (amdgpu_sriov_vf(adev))
- data = RREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_102);
+ data = psp->km_ring.ring_wptr;
else
data = RREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_67);
return data;
/* send interrupt to PSP for SRIOV ring write pointer update */
WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_101,
GFX_CTRL_CMD_ID_CONSUME_CMD);
+ psp->km_ring.ring_wptr = value;
} else
WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_67, value);
}
static const struct soc15_reg_golden golden_settings_sdma_nv12[] = {
SOC15_REG_GOLDEN_VALUE(GC, 0, mmSDMA0_RLC3_RB_WPTR_POLL_CNTL, 0xfffffff7, 0x00403000),
+ SOC15_REG_GOLDEN_VALUE(GC, 0, mmSDMA0_GB_ADDR_CONFIG, 0x001877ff, 0x00000044),
+ SOC15_REG_GOLDEN_VALUE(GC, 0, mmSDMA0_GB_ADDR_CONFIG_READ, 0x001877ff, 0x00000044),
+ SOC15_REG_GOLDEN_VALUE(GC, 0, mmSDMA1_GB_ADDR_CONFIG, 0x001877ff, 0x00000044),
+ SOC15_REG_GOLDEN_VALUE(GC, 0, mmSDMA1_GB_ADDR_CONFIG_READ, 0x001877ff, 0x00000044),
SOC15_REG_GOLDEN_VALUE(GC, 0, mmSDMA1_RLC3_RB_WPTR_POLL_CNTL, 0xfffffff7, 0x00403000),
};
ib_cntl = REG_SET_FIELD(ib_cntl, SDMA0_GFX_IB_CNTL, IB_ENABLE, 0);
WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_IB_CNTL), ib_cntl);
}
-
- sdma0->sched.ready = false;
- sdma1->sched.ready = false;
- sdma2->sched.ready = false;
- sdma3->sched.ready = false;
}
/**
*codecs = &rv_video_codecs_decode;
return 0;
case CHIP_ARCTURUS:
+ case CHIP_ALDEBARAN:
case CHIP_RENOIR:
if (encode)
*codecs = &vega_video_codecs_encode;
adev->cg_flags = AMD_CG_SUPPORT_GFX_MGCG |
AMD_CG_SUPPORT_GFX_MGLS |
AMD_CG_SUPPORT_GFX_CP_LS |
- AMD_CG_SUPPORT_GFX_3D_CGCG |
AMD_CG_SUPPORT_GFX_3D_CGLS |
AMD_CG_SUPPORT_GFX_CGCG |
AMD_CG_SUPPORT_GFX_CGLS |
AMD_CG_SUPPORT_MC_MGCG |
AMD_CG_SUPPORT_MC_LS |
AMD_CG_SUPPORT_SDMA_MGCG |
- AMD_CG_SUPPORT_SDMA_LS;
+ AMD_CG_SUPPORT_SDMA_LS |
+ AMD_CG_SUPPORT_VCN_MGCG;
adev->pg_flags = AMD_PG_SUPPORT_SDMA |
AMD_PG_SUPPORT_MMHUB |
AMD_CG_SUPPORT_GFX_MGLS |
AMD_CG_SUPPORT_GFX_RLC_LS |
AMD_CG_SUPPORT_GFX_CP_LS |
- AMD_CG_SUPPORT_GFX_3D_CGCG |
AMD_CG_SUPPORT_GFX_3D_CGLS |
AMD_CG_SUPPORT_GFX_CGCG |
AMD_CG_SUPPORT_GFX_CGLS |
error:
dma_fence_put(fence);
+ amdgpu_bo_unpin(bo);
amdgpu_bo_unreserve(bo);
amdgpu_bo_unref(&bo);
return r;
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ cancel_delayed_work_sync(&adev->vcn.idle_work);
+
if ((adev->pg_flags & AMD_PG_SUPPORT_VCN_DPG) ||
- RREG32_SOC15(VCN, 0, mmUVD_STATUS))
+ (adev->vcn.cur_state != AMD_PG_STATE_GATE &&
+ RREG32_SOC15(VCN, 0, mmUVD_STATUS))) {
vcn_v1_0_set_powergating_state(adev, AMD_PG_STATE_GATE);
+ }
return 0;
}
UVD_LMI_STATUS__WRITE_CLEAN_RAW_MASK;
SOC15_WAIT_ON_RREG(UVD, 0, mmUVD_LMI_STATUS, tmp, tmp);
- /* put VCPU into reset */
- WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_SOFT_RESET),
- UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK,
- ~UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK);
+ /* stall UMC channel */
+ WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_CTRL2),
+ UVD_LMI_CTRL2__STALL_ARB_UMC_MASK,
+ ~UVD_LMI_CTRL2__STALL_ARB_UMC_MASK);
tmp = UVD_LMI_STATUS__UMC_READ_CLEAN_RAW_MASK |
UVD_LMI_STATUS__UMC_WRITE_CLEAN_RAW_MASK;
UVD_SOFT_RESET__LMI_SOFT_RESET_MASK,
~UVD_SOFT_RESET__LMI_SOFT_RESET_MASK);
+ /* put VCPU into reset */
+ WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_SOFT_RESET),
+ UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK,
+ ~UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK);
+
WREG32_SOC15(UVD, 0, mmUVD_STATUS, 0);
vcn_v1_0_enable_clock_gating(adev);
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ cancel_delayed_work_sync(&adev->vcn.idle_work);
+
if ((adev->pg_flags & AMD_PG_SUPPORT_VCN_DPG) ||
(adev->vcn.cur_state != AMD_PG_STATE_GATE &&
RREG32_SOC15(VCN, 0, mmUVD_STATUS)))
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
int i;
+ cancel_delayed_work_sync(&adev->vcn.idle_work);
+
for (i = 0; i < adev->vcn.num_vcn_inst; ++i) {
if (adev->vcn.harvest_config & (1 << i))
continue;
static int vcn_v3_0_hw_fini(void *handle)
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
- struct amdgpu_ring *ring;
- int i, j;
+ int i;
+
+ cancel_delayed_work_sync(&adev->vcn.idle_work);
for (i = 0; i < adev->vcn.num_vcn_inst; ++i) {
if (adev->vcn.harvest_config & (1 << i))
continue;
- ring = &adev->vcn.inst[i].ring_dec;
-
if (!amdgpu_sriov_vf(adev)) {
if ((adev->pg_flags & AMD_PG_SUPPORT_VCN_DPG) ||
(adev->vcn.cur_state != AMD_PG_STATE_GATE &&
vcn_v3_0_set_powergating_state(adev, AMD_PG_STATE_GATE);
}
}
- ring->sched.ready = false;
-
- for (j = 0; j < adev->vcn.num_enc_rings; ++j) {
- ring = &adev->vcn.inst[i].ring_enc[j];
- ring->sched.ready = false;
- }
}
return 0;
WREG32_SOC15_DPG_MODE(inst_idx, SOC15_DPG_MODE_OFFSET(
VCN, inst_idx, mmUVD_VCPU_NONCACHE_SIZE0),
AMDGPU_GPU_PAGE_ALIGN(sizeof(struct amdgpu_fw_shared)), 0, indirect);
+
+ /* VCN global tiling registers */
+ WREG32_SOC15_DPG_MODE(0, SOC15_DPG_MODE_OFFSET(
+ UVD, 0, mmUVD_GFX10_ADDR_CONFIG), adev->gfx.config.gb_addr_config, 0, indirect);
}
static void vcn_v3_0_disable_static_power_gating(struct amdgpu_device *adev, int inst)
abm->dmcu_is_running = dmcu->funcs->is_dmcu_initialized(dmcu);
}
- adev->dm.dc->ctx->dmub_srv = dc_dmub_srv_create(adev->dm.dc, dmub_srv);
+ if (!adev->dm.dc->ctx->dmub_srv)
+ adev->dm.dc->ctx->dmub_srv = dc_dmub_srv_create(adev->dm.dc, dmub_srv);
if (!adev->dm.dc->ctx->dmub_srv) {
DRM_ERROR("Couldn't allocate DC DMUB server!\n");
return -ENOMEM;
amdgpu_dm_irq_suspend(adev);
-
dc_set_power_state(dm->dc, DC_ACPI_CM_POWER_STATE_D3);
return 0;
scaling_info->src_rect.x = state->src_x >> 16;
scaling_info->src_rect.y = state->src_y >> 16;
+ /*
+ * For reasons we don't (yet) fully understand a non-zero
+ * src_y coordinate into an NV12 buffer can cause a
+ * system hang. To avoid hangs (and maybe be overly cautious)
+ * let's reject both non-zero src_x and src_y.
+ *
+ * We currently know of only one use-case to reproduce a
+ * scenario with non-zero src_x and src_y for NV12, which
+ * is to gesture the YouTube Android app into full screen
+ * on ChromeOS.
+ */
+ if (state->fb &&
+ state->fb->format->format == DRM_FORMAT_NV12 &&
+ (scaling_info->src_rect.x != 0 ||
+ scaling_info->src_rect.y != 0))
+ return -EINVAL;
+
scaling_info->src_rect.width = state->src_w >> 16;
if (scaling_info->src_rect.width == 0)
return -EINVAL;
struct drm_display_mode saved_mode;
struct drm_display_mode *freesync_mode = NULL;
bool native_mode_found = false;
- bool recalculate_timing = dm_state ? (dm_state->scaling != RMX_OFF) : false;
+ bool recalculate_timing = false;
+ bool scale = dm_state ? (dm_state->scaling != RMX_OFF) : false;
int mode_refresh;
int preferred_refresh = 0;
#if defined(CONFIG_DRM_AMD_DC_DCN)
*/
DRM_DEBUG_DRIVER("No preferred mode found\n");
} else {
- recalculate_timing |= amdgpu_freesync_vid_mode &&
+ recalculate_timing = amdgpu_freesync_vid_mode &&
is_freesync_video_mode(&mode, aconnector);
if (recalculate_timing) {
freesync_mode = get_highest_refresh_rate_mode(aconnector, false);
mode = *freesync_mode;
} else {
decide_crtc_timing_for_drm_display_mode(
- &mode, preferred_mode,
- dm_state ? (dm_state->scaling != RMX_OFF) : false);
- }
+ &mode, preferred_mode, scale);
- preferred_refresh = drm_mode_vrefresh(preferred_mode);
+ preferred_refresh = drm_mode_vrefresh(preferred_mode);
+ }
}
if (recalculate_timing)
* If scaling is enabled and refresh rate didn't change
* we copy the vic and polarities of the old timings
*/
- if (!recalculate_timing || mode_refresh != preferred_refresh)
+ if (!scale || mode_refresh != preferred_refresh)
fill_stream_properties_from_drm_display_mode(
stream, &mode, &aconnector->base, con_state, NULL,
requested_bpc);
if (cursor_scale_w != primary_scale_w ||
cursor_scale_h != primary_scale_h) {
- DRM_DEBUG_ATOMIC("Cursor plane scaling doesn't match primary plane\n");
+ drm_dbg_atomic(crtc->dev, "Cursor plane scaling doesn't match primary plane\n");
return -EINVAL;
}
}
#endif
+static int validate_overlay(struct drm_atomic_state *state)
+{
+ int i;
+ struct drm_plane *plane;
+ struct drm_plane_state *old_plane_state, *new_plane_state;
+ struct drm_plane_state *primary_state, *cursor_state, *overlay_state = NULL;
+
+ /* Check if primary plane is contained inside overlay */
+ for_each_oldnew_plane_in_state_reverse(state, plane, old_plane_state, new_plane_state, i) {
+ if (plane->type == DRM_PLANE_TYPE_OVERLAY) {
+ if (drm_atomic_plane_disabling(plane->state, new_plane_state))
+ return 0;
+
+ overlay_state = new_plane_state;
+ continue;
+ }
+ }
+
+ /* check if we're making changes to the overlay plane */
+ if (!overlay_state)
+ return 0;
+
+ /* check if overlay plane is enabled */
+ if (!overlay_state->crtc)
+ return 0;
+
+ /* find the primary plane for the CRTC that the overlay is enabled on */
+ primary_state = drm_atomic_get_plane_state(state, overlay_state->crtc->primary);
+ if (IS_ERR(primary_state))
+ return PTR_ERR(primary_state);
+
+ /* check if primary plane is enabled */
+ if (!primary_state->crtc)
+ return 0;
+
+ /* check if cursor plane is enabled */
+ cursor_state = drm_atomic_get_plane_state(state, overlay_state->crtc->cursor);
+ if (IS_ERR(cursor_state))
+ return PTR_ERR(cursor_state);
+
+ if (drm_atomic_plane_disabling(plane->state, cursor_state))
+ return 0;
+
+ /* Perform the bounds check to ensure the overlay plane covers the primary */
+ if (primary_state->crtc_x < overlay_state->crtc_x ||
+ primary_state->crtc_y < overlay_state->crtc_y ||
+ primary_state->crtc_x + primary_state->crtc_w > overlay_state->crtc_x + overlay_state->crtc_w ||
+ primary_state->crtc_y + primary_state->crtc_h > overlay_state->crtc_y + overlay_state->crtc_h) {
+ DRM_DEBUG_ATOMIC("Overlay plane is enabled with hardware cursor but does not fully cover primary plane\n");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
/**
* amdgpu_dm_atomic_check() - Atomic check implementation for AMDgpu DM.
* @dev: The DRM device
goto fail;
}
+ ret = validate_overlay(state);
+ if (ret)
+ goto fail;
+
/* Add new/modified planes */
for_each_oldnew_plane_in_state_reverse(state, plane, old_plane_state, new_plane_state, i) {
ret = dm_update_plane_state(dc, state, plane,
if (!aconnector->dc_link)
continue;
- if (!(aconnector->port && &aconnector->mst_port->mst_mgr))
+ if (!aconnector->mst_port)
continue;
link = aconnector->dc_link;
/* File created at /sys/class/drm/card0/device/hdcp_srm*/
hdcp_work[0].attr = data_attr;
+ sysfs_bin_attr_init(&hdcp_work[0].attr);
if (sysfs_create_bin_file(&adev->dev->kobj, &hdcp_work[0].attr))
DRM_WARN("Failed to create device file hdcp_srm");
dc_is_dvi_signal(link->connector_signal)) {
if (prev_sink)
dc_sink_release(prev_sink);
+ link_disconnect_sink(link);
+
+ return false;
+ }
+ /*
+ * Abort detection for DP connectors if we have
+ * no EDID and connector is active converter
+ * as there are no display downstream
+ *
+ */
+ if (dc_is_dp_sst_signal(link->connector_signal) &&
+ (link->dpcd_caps.dongle_type ==
+ DISPLAY_DONGLE_DP_VGA_CONVERTER ||
+ link->dpcd_caps.dongle_type ==
+ DISPLAY_DONGLE_DP_DVI_CONVERTER)) {
+ if (prev_sink)
+ dc_sink_release(prev_sink);
+ link_disconnect_sink(link);
return false;
}
voltage_supported = dcn20_validate_bandwidth_internal(dc, context, false);
dummy_pstate_supported = context->bw_ctx.bw.dcn.clk.p_state_change_support;
- if (voltage_supported && dummy_pstate_supported) {
+ if (voltage_supported && (dummy_pstate_supported || !(context->stream_count))) {
context->bw_ctx.bw.dcn.clk.p_state_change_support = false;
goto restore_dml_state;
}
.fp16 = 16000
},
+ /* 6:1 downscaling ratio: 1000/6 = 166.666 */
.max_downscale_factor = {
- .argb8888 = 600,
- .nv12 = 600,
- .fp16 = 600
+ .argb8888 = 167,
+ .nv12 = 167,
+ .fp16 = 167
}
};
.fp16 = 16000
},
+ /* 6:1 downscaling ratio: 1000/6 = 166.666 */
.max_downscale_factor = {
- .argb8888 = 600,
- .nv12 = 600,
- .fp16 = 600
+ .argb8888 = 167,
+ .nv12 = 167,
+ .fp16 = 167
},
64,
64
.nv12 = 16000,
.fp16 = 16000
},
+ /* 6:1 downscaling ratio: 1000/6 = 166.666 */
.max_downscale_factor = {
- .argb8888 = 600,
- .nv12 = 600,
- .fp16 = 600
+ .argb8888 = 167,
+ .nv12 = 167,
+ .fp16 = 167
},
16,
16
PP_GFX_DCS_MASK = 0x80000,
};
+enum amd_harvest_ip_mask {
+ AMD_HARVEST_IP_VCN_MASK = 0x1,
+ AMD_HARVEST_IP_JPEG_MASK = 0x2,
+ AMD_HARVEST_IP_DMU_MASK = 0x4,
+};
+
enum DC_FEATURE_MASK {
DC_FBC_MASK = 0x1,
DC_MULTI_MON_PP_MCLK_SWITCH_MASK = 0x2,
struct drm_device *ddev = dev_get_drvdata(dev);
struct amdgpu_device *adev = drm_to_adev(ddev);
const struct amd_pm_funcs *pp_funcs = adev->powerplay.pp_funcs;
- struct pp_states_info data;
+ struct pp_states_info data = {0};
enum amd_pm_state_type pm = 0;
int i = 0, ret = 0;
}
}
+ if (DEVICE_ATTR_IS(pp_dpm_dcefclk)) {
+ /* SMU MP1 does not support dcefclk level setting */
+ if (asic_type >= CHIP_NAVI10) {
+ dev_attr->attr.mode &= ~S_IWUGO;
+ dev_attr->store = NULL;
+ }
+ }
+
#undef DEVICE_ATTR_IS
return 0;
break;
case AMD_DPM_FORCED_LEVEL_MANUAL:
data->fine_grain_enabled = 1;
+ break;
case AMD_DPM_FORCED_LEVEL_PROFILE_EXIT:
default:
break;
u32 reg;
int ret;
- table->initialState.levels[0].mclk.vDLL_CNTL =
+ table->initialState.level.mclk.vDLL_CNTL =
cpu_to_be32(si_pi->clock_registers.dll_cntl);
- table->initialState.levels[0].mclk.vMCLK_PWRMGT_CNTL =
+ table->initialState.level.mclk.vMCLK_PWRMGT_CNTL =
cpu_to_be32(si_pi->clock_registers.mclk_pwrmgt_cntl);
- table->initialState.levels[0].mclk.vMPLL_AD_FUNC_CNTL =
+ table->initialState.level.mclk.vMPLL_AD_FUNC_CNTL =
cpu_to_be32(si_pi->clock_registers.mpll_ad_func_cntl);
- table->initialState.levels[0].mclk.vMPLL_DQ_FUNC_CNTL =
+ table->initialState.level.mclk.vMPLL_DQ_FUNC_CNTL =
cpu_to_be32(si_pi->clock_registers.mpll_dq_func_cntl);
- table->initialState.levels[0].mclk.vMPLL_FUNC_CNTL =
+ table->initialState.level.mclk.vMPLL_FUNC_CNTL =
cpu_to_be32(si_pi->clock_registers.mpll_func_cntl);
- table->initialState.levels[0].mclk.vMPLL_FUNC_CNTL_1 =
+ table->initialState.level.mclk.vMPLL_FUNC_CNTL_1 =
cpu_to_be32(si_pi->clock_registers.mpll_func_cntl_1);
- table->initialState.levels[0].mclk.vMPLL_FUNC_CNTL_2 =
+ table->initialState.level.mclk.vMPLL_FUNC_CNTL_2 =
cpu_to_be32(si_pi->clock_registers.mpll_func_cntl_2);
- table->initialState.levels[0].mclk.vMPLL_SS =
+ table->initialState.level.mclk.vMPLL_SS =
cpu_to_be32(si_pi->clock_registers.mpll_ss1);
- table->initialState.levels[0].mclk.vMPLL_SS2 =
+ table->initialState.level.mclk.vMPLL_SS2 =
cpu_to_be32(si_pi->clock_registers.mpll_ss2);
- table->initialState.levels[0].mclk.mclk_value =
+ table->initialState.level.mclk.mclk_value =
cpu_to_be32(initial_state->performance_levels[0].mclk);
- table->initialState.levels[0].sclk.vCG_SPLL_FUNC_CNTL =
+ table->initialState.level.sclk.vCG_SPLL_FUNC_CNTL =
cpu_to_be32(si_pi->clock_registers.cg_spll_func_cntl);
- table->initialState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_2 =
+ table->initialState.level.sclk.vCG_SPLL_FUNC_CNTL_2 =
cpu_to_be32(si_pi->clock_registers.cg_spll_func_cntl_2);
- table->initialState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_3 =
+ table->initialState.level.sclk.vCG_SPLL_FUNC_CNTL_3 =
cpu_to_be32(si_pi->clock_registers.cg_spll_func_cntl_3);
- table->initialState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_4 =
+ table->initialState.level.sclk.vCG_SPLL_FUNC_CNTL_4 =
cpu_to_be32(si_pi->clock_registers.cg_spll_func_cntl_4);
- table->initialState.levels[0].sclk.vCG_SPLL_SPREAD_SPECTRUM =
+ table->initialState.level.sclk.vCG_SPLL_SPREAD_SPECTRUM =
cpu_to_be32(si_pi->clock_registers.cg_spll_spread_spectrum);
- table->initialState.levels[0].sclk.vCG_SPLL_SPREAD_SPECTRUM_2 =
+ table->initialState.level.sclk.vCG_SPLL_SPREAD_SPECTRUM_2 =
cpu_to_be32(si_pi->clock_registers.cg_spll_spread_spectrum_2);
- table->initialState.levels[0].sclk.sclk_value =
+ table->initialState.level.sclk.sclk_value =
cpu_to_be32(initial_state->performance_levels[0].sclk);
- table->initialState.levels[0].arbRefreshState =
+ table->initialState.level.arbRefreshState =
SISLANDS_INITIAL_STATE_ARB_INDEX;
- table->initialState.levels[0].ACIndex = 0;
+ table->initialState.level.ACIndex = 0;
ret = si_populate_voltage_value(adev, &eg_pi->vddc_voltage_table,
initial_state->performance_levels[0].vddc,
- &table->initialState.levels[0].vddc);
+ &table->initialState.level.vddc);
if (!ret) {
u16 std_vddc;
ret = si_get_std_voltage_value(adev,
- &table->initialState.levels[0].vddc,
+ &table->initialState.level.vddc,
&std_vddc);
if (!ret)
si_populate_std_voltage_value(adev, std_vddc,
- table->initialState.levels[0].vddc.index,
- &table->initialState.levels[0].std_vddc);
+ table->initialState.level.vddc.index,
+ &table->initialState.level.std_vddc);
}
if (eg_pi->vddci_control)
si_populate_voltage_value(adev,
&eg_pi->vddci_voltage_table,
initial_state->performance_levels[0].vddci,
- &table->initialState.levels[0].vddci);
+ &table->initialState.level.vddci);
if (si_pi->vddc_phase_shed_control)
si_populate_phase_shedding_value(adev,
initial_state->performance_levels[0].vddc,
initial_state->performance_levels[0].sclk,
initial_state->performance_levels[0].mclk,
- &table->initialState.levels[0].vddc);
+ &table->initialState.level.vddc);
- si_populate_initial_mvdd_value(adev, &table->initialState.levels[0].mvdd);
+ si_populate_initial_mvdd_value(adev, &table->initialState.level.mvdd);
reg = CG_R(0xffff) | CG_L(0);
- table->initialState.levels[0].aT = cpu_to_be32(reg);
- table->initialState.levels[0].bSP = cpu_to_be32(pi->dsp);
- table->initialState.levels[0].gen2PCIE = (u8)si_pi->boot_pcie_gen;
+ table->initialState.level.aT = cpu_to_be32(reg);
+ table->initialState.level.bSP = cpu_to_be32(pi->dsp);
+ table->initialState.level.gen2PCIE = (u8)si_pi->boot_pcie_gen;
if (adev->gmc.vram_type == AMDGPU_VRAM_TYPE_GDDR5) {
- table->initialState.levels[0].strobeMode =
+ table->initialState.level.strobeMode =
si_get_strobe_mode_settings(adev,
initial_state->performance_levels[0].mclk);
if (initial_state->performance_levels[0].mclk > pi->mclk_edc_enable_threshold)
- table->initialState.levels[0].mcFlags = SISLANDS_SMC_MC_EDC_RD_FLAG | SISLANDS_SMC_MC_EDC_WR_FLAG;
+ table->initialState.level.mcFlags = SISLANDS_SMC_MC_EDC_RD_FLAG | SISLANDS_SMC_MC_EDC_WR_FLAG;
else
- table->initialState.levels[0].mcFlags = 0;
+ table->initialState.level.mcFlags = 0;
}
table->initialState.levelCount = 1;
table->initialState.flags |= PPSMC_SWSTATE_FLAG_DC;
- table->initialState.levels[0].dpm2.MaxPS = 0;
- table->initialState.levels[0].dpm2.NearTDPDec = 0;
- table->initialState.levels[0].dpm2.AboveSafeInc = 0;
- table->initialState.levels[0].dpm2.BelowSafeInc = 0;
- table->initialState.levels[0].dpm2.PwrEfficiencyRatio = 0;
+ table->initialState.level.dpm2.MaxPS = 0;
+ table->initialState.level.dpm2.NearTDPDec = 0;
+ table->initialState.level.dpm2.AboveSafeInc = 0;
+ table->initialState.level.dpm2.BelowSafeInc = 0;
+ table->initialState.level.dpm2.PwrEfficiencyRatio = 0;
reg = MIN_POWER_MASK | MAX_POWER_MASK;
- table->initialState.levels[0].SQPowerThrottle = cpu_to_be32(reg);
+ table->initialState.level.SQPowerThrottle = cpu_to_be32(reg);
reg = MAX_POWER_DELTA_MASK | STI_SIZE_MASK | LTI_RATIO_MASK;
- table->initialState.levels[0].SQPowerThrottle_2 = cpu_to_be32(reg);
+ table->initialState.level.SQPowerThrottle_2 = cpu_to_be32(reg);
return 0;
}
if (pi->acpi_vddc) {
ret = si_populate_voltage_value(adev, &eg_pi->vddc_voltage_table,
- pi->acpi_vddc, &table->ACPIState.levels[0].vddc);
+ pi->acpi_vddc, &table->ACPIState.level.vddc);
if (!ret) {
u16 std_vddc;
ret = si_get_std_voltage_value(adev,
- &table->ACPIState.levels[0].vddc, &std_vddc);
+ &table->ACPIState.level.vddc, &std_vddc);
if (!ret)
si_populate_std_voltage_value(adev, std_vddc,
- table->ACPIState.levels[0].vddc.index,
- &table->ACPIState.levels[0].std_vddc);
+ table->ACPIState.level.vddc.index,
+ &table->ACPIState.level.std_vddc);
}
- table->ACPIState.levels[0].gen2PCIE = si_pi->acpi_pcie_gen;
+ table->ACPIState.level.gen2PCIE = si_pi->acpi_pcie_gen;
if (si_pi->vddc_phase_shed_control) {
si_populate_phase_shedding_value(adev,
pi->acpi_vddc,
0,
0,
- &table->ACPIState.levels[0].vddc);
+ &table->ACPIState.level.vddc);
}
} else {
ret = si_populate_voltage_value(adev, &eg_pi->vddc_voltage_table,
- pi->min_vddc_in_table, &table->ACPIState.levels[0].vddc);
+ pi->min_vddc_in_table, &table->ACPIState.level.vddc);
if (!ret) {
u16 std_vddc;
ret = si_get_std_voltage_value(adev,
- &table->ACPIState.levels[0].vddc, &std_vddc);
+ &table->ACPIState.level.vddc, &std_vddc);
if (!ret)
si_populate_std_voltage_value(adev, std_vddc,
- table->ACPIState.levels[0].vddc.index,
- &table->ACPIState.levels[0].std_vddc);
+ table->ACPIState.level.vddc.index,
+ &table->ACPIState.level.std_vddc);
}
- table->ACPIState.levels[0].gen2PCIE =
+ table->ACPIState.level.gen2PCIE =
(u8)amdgpu_get_pcie_gen_support(adev,
si_pi->sys_pcie_mask,
si_pi->boot_pcie_gen,
pi->min_vddc_in_table,
0,
0,
- &table->ACPIState.levels[0].vddc);
+ &table->ACPIState.level.vddc);
}
if (pi->acpi_vddc) {
if (eg_pi->acpi_vddci)
si_populate_voltage_value(adev, &eg_pi->vddci_voltage_table,
eg_pi->acpi_vddci,
- &table->ACPIState.levels[0].vddci);
+ &table->ACPIState.level.vddci);
}
mclk_pwrmgt_cntl |= MRDCK0_RESET | MRDCK1_RESET;
spll_func_cntl_2 &= ~SCLK_MUX_SEL_MASK;
spll_func_cntl_2 |= SCLK_MUX_SEL(4);
- table->ACPIState.levels[0].mclk.vDLL_CNTL =
+ table->ACPIState.level.mclk.vDLL_CNTL =
cpu_to_be32(dll_cntl);
- table->ACPIState.levels[0].mclk.vMCLK_PWRMGT_CNTL =
+ table->ACPIState.level.mclk.vMCLK_PWRMGT_CNTL =
cpu_to_be32(mclk_pwrmgt_cntl);
- table->ACPIState.levels[0].mclk.vMPLL_AD_FUNC_CNTL =
+ table->ACPIState.level.mclk.vMPLL_AD_FUNC_CNTL =
cpu_to_be32(mpll_ad_func_cntl);
- table->ACPIState.levels[0].mclk.vMPLL_DQ_FUNC_CNTL =
+ table->ACPIState.level.mclk.vMPLL_DQ_FUNC_CNTL =
cpu_to_be32(mpll_dq_func_cntl);
- table->ACPIState.levels[0].mclk.vMPLL_FUNC_CNTL =
+ table->ACPIState.level.mclk.vMPLL_FUNC_CNTL =
cpu_to_be32(mpll_func_cntl);
- table->ACPIState.levels[0].mclk.vMPLL_FUNC_CNTL_1 =
+ table->ACPIState.level.mclk.vMPLL_FUNC_CNTL_1 =
cpu_to_be32(mpll_func_cntl_1);
- table->ACPIState.levels[0].mclk.vMPLL_FUNC_CNTL_2 =
+ table->ACPIState.level.mclk.vMPLL_FUNC_CNTL_2 =
cpu_to_be32(mpll_func_cntl_2);
- table->ACPIState.levels[0].mclk.vMPLL_SS =
+ table->ACPIState.level.mclk.vMPLL_SS =
cpu_to_be32(si_pi->clock_registers.mpll_ss1);
- table->ACPIState.levels[0].mclk.vMPLL_SS2 =
+ table->ACPIState.level.mclk.vMPLL_SS2 =
cpu_to_be32(si_pi->clock_registers.mpll_ss2);
- table->ACPIState.levels[0].sclk.vCG_SPLL_FUNC_CNTL =
+ table->ACPIState.level.sclk.vCG_SPLL_FUNC_CNTL =
cpu_to_be32(spll_func_cntl);
- table->ACPIState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_2 =
+ table->ACPIState.level.sclk.vCG_SPLL_FUNC_CNTL_2 =
cpu_to_be32(spll_func_cntl_2);
- table->ACPIState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_3 =
+ table->ACPIState.level.sclk.vCG_SPLL_FUNC_CNTL_3 =
cpu_to_be32(spll_func_cntl_3);
- table->ACPIState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_4 =
+ table->ACPIState.level.sclk.vCG_SPLL_FUNC_CNTL_4 =
cpu_to_be32(spll_func_cntl_4);
- table->ACPIState.levels[0].mclk.mclk_value = 0;
- table->ACPIState.levels[0].sclk.sclk_value = 0;
+ table->ACPIState.level.mclk.mclk_value = 0;
+ table->ACPIState.level.sclk.sclk_value = 0;
- si_populate_mvdd_value(adev, 0, &table->ACPIState.levels[0].mvdd);
+ si_populate_mvdd_value(adev, 0, &table->ACPIState.level.mvdd);
if (eg_pi->dynamic_ac_timing)
- table->ACPIState.levels[0].ACIndex = 0;
+ table->ACPIState.level.ACIndex = 0;
- table->ACPIState.levels[0].dpm2.MaxPS = 0;
- table->ACPIState.levels[0].dpm2.NearTDPDec = 0;
- table->ACPIState.levels[0].dpm2.AboveSafeInc = 0;
- table->ACPIState.levels[0].dpm2.BelowSafeInc = 0;
- table->ACPIState.levels[0].dpm2.PwrEfficiencyRatio = 0;
+ table->ACPIState.level.dpm2.MaxPS = 0;
+ table->ACPIState.level.dpm2.NearTDPDec = 0;
+ table->ACPIState.level.dpm2.AboveSafeInc = 0;
+ table->ACPIState.level.dpm2.BelowSafeInc = 0;
+ table->ACPIState.level.dpm2.PwrEfficiencyRatio = 0;
reg = MIN_POWER_MASK | MAX_POWER_MASK;
- table->ACPIState.levels[0].SQPowerThrottle = cpu_to_be32(reg);
+ table->ACPIState.level.SQPowerThrottle = cpu_to_be32(reg);
reg = MAX_POWER_DELTA_MASK | STI_SIZE_MASK | LTI_RATIO_MASK;
- table->ACPIState.levels[0].SQPowerThrottle_2 = cpu_to_be32(reg);
+ table->ACPIState.level.SQPowerThrottle_2 = cpu_to_be32(reg);
return 0;
}
static int si_populate_ulv_state(struct amdgpu_device *adev,
- SISLANDS_SMC_SWSTATE *state)
+ struct SISLANDS_SMC_SWSTATE_SINGLE *state)
{
struct evergreen_power_info *eg_pi = evergreen_get_pi(adev);
struct si_power_info *si_pi = si_get_pi(adev);
int ret;
ret = si_convert_power_level_to_smc(adev, &ulv->pl,
- &state->levels[0]);
+ &state->level);
if (!ret) {
if (eg_pi->sclk_deep_sleep) {
if (sclk_in_sr <= SCLK_MIN_DEEPSLEEP_FREQ)
- state->levels[0].stateFlags |= PPSMC_STATEFLAG_DEEPSLEEP_BYPASS;
+ state->level.stateFlags |= PPSMC_STATEFLAG_DEEPSLEEP_BYPASS;
else
- state->levels[0].stateFlags |= PPSMC_STATEFLAG_DEEPSLEEP_THROTTLE;
+ state->level.stateFlags |= PPSMC_STATEFLAG_DEEPSLEEP_THROTTLE;
}
if (ulv->one_pcie_lane_in_ulv)
state->flags |= PPSMC_SWSTATE_FLAG_PCIE_X1;
- state->levels[0].arbRefreshState = (u8)(SISLANDS_ULV_STATE_ARB_INDEX);
- state->levels[0].ACIndex = 1;
- state->levels[0].std_vddc = state->levels[0].vddc;
+ state->level.arbRefreshState = (u8)(SISLANDS_ULV_STATE_ARB_INDEX);
+ state->level.ACIndex = 1;
+ state->level.std_vddc = state->level.vddc;
state->levelCount = 1;
state->flags |= PPSMC_SWSTATE_FLAG_DC;
if (ret)
return ret;
- table->driverState = table->initialState;
+ table->driverState.flags = table->initialState.flags;
+ table->driverState.levelCount = table->initialState.levelCount;
+ table->driverState.levels[0] = table->initialState.level;
ret = si_do_program_memory_timing_parameters(adev, amdgpu_boot_state,
SISLANDS_INITIAL_STATE_ARB_INDEX);
if (ulv->supported && ulv->pl.vddc) {
u32 address = si_pi->state_table_start +
offsetof(SISLANDS_SMC_STATETABLE, ULVState);
- SISLANDS_SMC_SWSTATE *smc_state = &si_pi->smc_statetable.ULVState;
- u32 state_size = sizeof(SISLANDS_SMC_SWSTATE);
+ struct SISLANDS_SMC_SWSTATE_SINGLE *smc_state = &si_pi->smc_statetable.ULVState;
+ u32 state_size = sizeof(struct SISLANDS_SMC_SWSTATE_SINGLE);
memset(smc_state, 0, state_size);
typedef struct SISLANDS_SMC_SWSTATE SISLANDS_SMC_SWSTATE;
+struct SISLANDS_SMC_SWSTATE_SINGLE {
+ uint8_t flags;
+ uint8_t levelCount;
+ uint8_t padding2;
+ uint8_t padding3;
+ SISLANDS_SMC_HW_PERFORMANCE_LEVEL level;
+};
+
#define SISLANDS_SMC_VOLTAGEMASK_VDDC 0
#define SISLANDS_SMC_VOLTAGEMASK_MVDD 1
#define SISLANDS_SMC_VOLTAGEMASK_VDDCI 2
struct SISLANDS_SMC_STATETABLE
{
- uint8_t thermalProtectType;
- uint8_t systemFlags;
- uint8_t maxVDDCIndexInPPTable;
- uint8_t extraFlags;
- uint32_t lowSMIO[SISLANDS_MAX_NO_VREG_STEPS];
- SISLANDS_SMC_VOLTAGEMASKTABLE voltageMaskTable;
- SISLANDS_SMC_VOLTAGEMASKTABLE phaseMaskTable;
- PP_SIslands_DPM2Parameters dpm2Params;
- SISLANDS_SMC_SWSTATE initialState;
- SISLANDS_SMC_SWSTATE ACPIState;
- SISLANDS_SMC_SWSTATE ULVState;
- SISLANDS_SMC_SWSTATE driverState;
- SISLANDS_SMC_HW_PERFORMANCE_LEVEL dpmLevels[SISLANDS_MAX_SMC_PERFORMANCE_LEVELS_PER_SWSTATE - 1];
+ uint8_t thermalProtectType;
+ uint8_t systemFlags;
+ uint8_t maxVDDCIndexInPPTable;
+ uint8_t extraFlags;
+ uint32_t lowSMIO[SISLANDS_MAX_NO_VREG_STEPS];
+ SISLANDS_SMC_VOLTAGEMASKTABLE voltageMaskTable;
+ SISLANDS_SMC_VOLTAGEMASKTABLE phaseMaskTable;
+ PP_SIslands_DPM2Parameters dpm2Params;
+ struct SISLANDS_SMC_SWSTATE_SINGLE initialState;
+ struct SISLANDS_SMC_SWSTATE_SINGLE ACPIState;
+ struct SISLANDS_SMC_SWSTATE_SINGLE ULVState;
+ SISLANDS_SMC_SWSTATE driverState;
+ SISLANDS_SMC_HW_PERFORMANCE_LEVEL dpmLevels[SISLANDS_MAX_SMC_PERFORMANCE_LEVELS_PER_SWSTATE];
};
typedef struct SISLANDS_SMC_STATETABLE SISLANDS_SMC_STATETABLE;
case SMU_SOCCLK:
case SMU_MCLK:
case SMU_UCLK:
- case SMU_DCEFCLK:
case SMU_FCLK:
/* There is only 2 levels for fine grained DPM */
if (navi10_is_support_fine_grained_dpm(smu, clk_type)) {
if (ret)
return size;
break;
+ case SMU_DCEFCLK:
+ dev_info(smu->adev->dev,"Setting DCEFCLK min/max dpm level is not supported!\n");
+ break;
+
default:
break;
}
static int navi10_enable_mgpu_fan_boost(struct smu_context *smu)
{
+ struct smu_table_context *table_context = &smu->smu_table;
+ PPTable_t *smc_pptable = table_context->driver_pptable;
struct amdgpu_device *adev = smu->adev;
uint32_t param = 0;
if (adev->asic_type == CHIP_NAVI12)
return 0;
+ /*
+ * Skip the MGpuFanBoost setting for those ASICs
+ * which do not support it
+ */
+ if (!smc_pptable->MGpuFanBoostLimitRpm)
+ return 0;
+
/* Workaround for WS SKU */
if (adev->pdev->device == 0x7312 &&
adev->pdev->revision == 0)
case SMU_SOCCLK:
case SMU_MCLK:
case SMU_UCLK:
- case SMU_DCEFCLK:
case SMU_FCLK:
/* There is only 2 levels for fine grained DPM */
if (sienna_cichlid_is_support_fine_grained_dpm(smu, clk_type)) {
if (ret)
goto forec_level_out;
break;
+ case SMU_DCEFCLK:
+ dev_info(smu->adev->dev,"Setting DCEFCLK min/max dpm level is not supported!\n");
+ break;
default:
break;
}
static int sienna_cichlid_enable_mgpu_fan_boost(struct smu_context *smu)
{
+ struct smu_table_context *table_context = &smu->smu_table;
+ PPTable_t *smc_pptable = table_context->driver_pptable;
+
+ /*
+ * Skip the MGpuFanBoost setting for those ASICs
+ * which do not support it
+ */
+ if (!smc_pptable->MGpuFanBoostLimitRpm)
+ return 0;
+
return smu_cmn_send_smc_msg_with_param(smu,
SMU_MSG_SetMGpuFanBoostLimitRpm,
0,
void drm_master_release(struct drm_file *file_priv)
{
struct drm_device *dev = file_priv->minor->dev;
- struct drm_master *master = file_priv->master;
+ struct drm_master *master;
mutex_lock(&dev->master_mutex);
+ master = file_priv->master;
if (file_priv->magic)
idr_remove(&file_priv->master->magic_map, file_priv->magic);
struct drm_file *file_priv)
{
struct drm_unique *u = data;
- struct drm_master *master = file_priv->master;
+ struct drm_master *master;
- mutex_lock(&master->dev->master_mutex);
+ mutex_lock(&dev->master_mutex);
+ master = file_priv->master;
if (u->unique_len >= master->unique_len) {
if (copy_to_user(u->unique, master->unique, master->unique_len)) {
- mutex_unlock(&master->dev->master_mutex);
+ mutex_unlock(&dev->master_mutex);
return -EFAULT;
}
}
u->unique_len = master->unique_len;
- mutex_unlock(&master->dev->master_mutex);
+ mutex_unlock(&dev->master_mutex);
return 0;
}
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
ctx->addr = devm_ioremap_resource(dev, res);
- if (IS_ERR(ctx->addr)) {
- dev_err(dev, "ioremap failed\n");
+ if (IS_ERR(ctx->addr))
return PTR_ERR(ctx->addr);
- }
ret = decon_conf_irq(ctx, "vsync", decon_irq_handler, 0);
if (ret < 0)
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
dsi->reg_base = devm_ioremap_resource(dev, res);
- if (IS_ERR(dsi->reg_base)) {
- dev_err(dev, "failed to remap io region\n");
+ if (IS_ERR(dsi->reg_base))
return PTR_ERR(dsi->reg_base);
- }
dsi->phy = devm_phy_get(dev, "dsim");
if (IS_ERR(dsi->phy)) {
}
/**
- * shadow_protect_win() - disable updating values from shadow registers at vsync
+ * fimd_shadow_protect_win() - disable updating values from shadow registers at vsync
*
* @ctx: local driver data
* @win: window to protect registers for
select INPUT if ACPI
select ACPI_VIDEO if ACPI
select ACPI_BUTTON if ACPI
- select IO_MAPPING
select SYNC_FILE
select IOSF_MBI
select CRC32
bool "Enable Intel GVT-g graphics virtualization host support"
depends on DRM_I915
depends on 64BIT
- depends on VFIO_MDEV=y || VFIO_MDEV=DRM_I915
default n
help
Choose this option if you want to enable Intel GVT-g graphics
return -EINVAL;
}
-/* Optimize link config in order: max bpp, min lanes, min clock */
-static int
-intel_dp_compute_link_config_fast(struct intel_dp *intel_dp,
- struct intel_crtc_state *pipe_config,
- const struct link_config_limits *limits)
-{
- const struct drm_display_mode *adjusted_mode = &pipe_config->hw.adjusted_mode;
- int bpp, clock, lane_count;
- int mode_rate, link_clock, link_avail;
-
- for (bpp = limits->max_bpp; bpp >= limits->min_bpp; bpp -= 2 * 3) {
- int output_bpp = intel_dp_output_bpp(pipe_config->output_format, bpp);
-
- mode_rate = intel_dp_link_required(adjusted_mode->crtc_clock,
- output_bpp);
-
- for (lane_count = limits->min_lane_count;
- lane_count <= limits->max_lane_count;
- lane_count <<= 1) {
- for (clock = limits->min_clock; clock <= limits->max_clock; clock++) {
- link_clock = intel_dp->common_rates[clock];
- link_avail = intel_dp_max_data_rate(link_clock,
- lane_count);
-
- if (mode_rate <= link_avail) {
- pipe_config->lane_count = lane_count;
- pipe_config->pipe_bpp = bpp;
- pipe_config->port_clock = link_clock;
-
- return 0;
- }
- }
- }
- }
-
- return -EINVAL;
-}
-
static int intel_dp_dsc_compute_bpp(struct intel_dp *intel_dp, u8 dsc_max_bpc)
{
int i, num_bpc;
intel_dp_can_bigjoiner(intel_dp))
pipe_config->bigjoiner = true;
- if (intel_dp_is_edp(intel_dp))
- /*
- * Optimize for fast and narrow. eDP 1.3 section 3.3 and eDP 1.4
- * section A.1: "It is recommended that the minimum number of
- * lanes be used, using the minimum link rate allowed for that
- * lane configuration."
- *
- * Note that we fall back to the max clock and lane count for eDP
- * panels that fail with the fast optimal settings (see
- * intel_dp->use_max_params), in which case the fast vs. wide
- * choice doesn't matter.
- */
- ret = intel_dp_compute_link_config_fast(intel_dp, pipe_config, &limits);
- else
- /* Optimize for slow and wide. */
- ret = intel_dp_compute_link_config_wide(intel_dp, pipe_config, &limits);
+ /*
+ * Optimize for slow and wide for everything, because there are some
+ * eDP 1.3 and 1.4 panels don't work well with fast and narrow.
+ */
+ ret = intel_dp_compute_link_config_wide(intel_dp, pipe_config, &limits);
/* enable compression if the mode doesn't fit available BW */
drm_dbg_kms(&i915->drm, "Force DSC en = %d\n", intel_dp->force_dsc_en);
* -PCON supports SRC_CTL_MODE (VESA DP2.0-HDMI2.1 PCON Spec Draft-1 Sec-7)
* -sink is HDMI2.1
*/
- if (!(intel_dp->dpcd[2] & DP_PCON_SOURCE_CTL_MODE) ||
+ if (!(intel_dp->downstream_ports[2] & DP_PCON_SOURCE_CTL_MODE) ||
!intel_dp_is_hdmi_2_1_sink(intel_dp) ||
intel_dp->frl.is_trained)
return;
return drm_dp_dpcd_write(&intel_dp->aux, DP_PHY_REPEATER_MODE, &val, 1) == 1;
}
-/**
- * intel_dp_init_lttpr_and_dprx_caps - detect LTTPR and DPRX caps, init the LTTPR link training mode
- * @intel_dp: Intel DP struct
- *
- * Read the LTTPR common and DPRX capabilities and switch to non-transparent
- * link training mode if any is detected and read the PHY capabilities for all
- * detected LTTPRs. In case of an LTTPR detection error or if the number of
- * LTTPRs is more than is supported (8), fall back to the no-LTTPR,
- * transparent mode link training mode.
- *
- * Returns:
- * >0 if LTTPRs were detected and the non-transparent LT mode was set. The
- * DPRX capabilities are read out.
- * 0 if no LTTPRs or more than 8 LTTPRs were detected or in case of a
- * detection failure and the transparent LT mode was set. The DPRX
- * capabilities are read out.
- * <0 Reading out the DPRX capabilities failed.
- */
-int intel_dp_init_lttpr_and_dprx_caps(struct intel_dp *intel_dp)
+static int intel_dp_init_lttpr(struct intel_dp *intel_dp)
{
int lttpr_count;
- bool ret;
int i;
- ret = intel_dp_read_lttpr_common_caps(intel_dp);
-
- /* The DPTX shall read the DPRX caps after LTTPR detection. */
- if (drm_dp_read_dpcd_caps(&intel_dp->aux, intel_dp->dpcd)) {
- intel_dp_reset_lttpr_common_caps(intel_dp);
- return -EIO;
- }
-
- if (!ret)
- return 0;
-
- /*
- * The 0xF0000-0xF02FF range is only valid if the DPCD revision is
- * at least 1.4.
- */
- if (intel_dp->dpcd[DP_DPCD_REV] < 0x14) {
- intel_dp_reset_lttpr_common_caps(intel_dp);
+ if (!intel_dp_read_lttpr_common_caps(intel_dp))
return 0;
- }
lttpr_count = drm_dp_lttpr_count(intel_dp->lttpr_common_caps);
/*
return lttpr_count;
}
+
+/**
+ * intel_dp_init_lttpr_and_dprx_caps - detect LTTPR and DPRX caps, init the LTTPR link training mode
+ * @intel_dp: Intel DP struct
+ *
+ * Read the LTTPR common and DPRX capabilities and switch to non-transparent
+ * link training mode if any is detected and read the PHY capabilities for all
+ * detected LTTPRs. In case of an LTTPR detection error or if the number of
+ * LTTPRs is more than is supported (8), fall back to the no-LTTPR,
+ * transparent mode link training mode.
+ *
+ * Returns:
+ * >0 if LTTPRs were detected and the non-transparent LT mode was set. The
+ * DPRX capabilities are read out.
+ * 0 if no LTTPRs or more than 8 LTTPRs were detected or in case of a
+ * detection failure and the transparent LT mode was set. The DPRX
+ * capabilities are read out.
+ * <0 Reading out the DPRX capabilities failed.
+ */
+int intel_dp_init_lttpr_and_dprx_caps(struct intel_dp *intel_dp)
+{
+ int lttpr_count = intel_dp_init_lttpr(intel_dp);
+
+ /* The DPTX shall read the DPRX caps after LTTPR detection. */
+ if (drm_dp_read_dpcd_caps(&intel_dp->aux, intel_dp->dpcd)) {
+ intel_dp_reset_lttpr_common_caps(intel_dp);
+ return -EIO;
+ }
+
+ return lttpr_count;
+}
EXPORT_SYMBOL(intel_dp_init_lttpr_and_dprx_caps);
static u8 dp_voltage_max(u8 preemph)
i830_overlay_clock_gating(dev_priv, true);
}
-static void
+__i915_active_call static void
intel_overlay_last_flip_retire(struct i915_active *active)
{
struct intel_overlay *overlay =
struct i915_ggtt_view view;
if (i915_gem_object_is_tiled(obj))
- chunk = roundup(chunk, tile_row_pages(obj));
+ chunk = roundup(chunk, tile_row_pages(obj) ?: 1);
view.type = I915_GGTT_VIEW_PARTIAL;
view.partial.offset = rounddown(page_offset, chunk);
goto err_unpin;
/* Finally, remap it using the new GTT offset */
- ret = io_mapping_map_user(&ggtt->iomap, area, area->vm_start +
- (vma->ggtt_view.partial.offset << PAGE_SHIFT),
- (ggtt->gmadr.start + vma->node.start) >> PAGE_SHIFT,
- min_t(u64, vma->size, area->vm_end - area->vm_start));
+ ret = remap_io_mapping(area,
+ area->vm_start + (vma->ggtt_view.partial.offset << PAGE_SHIFT),
+ (ggtt->gmadr.start + vma->node.start) >> PAGE_SHIFT,
+ min_t(u64, vma->size, area->vm_end - area->vm_start),
+ &ggtt->iomap);
if (ret)
goto err_fence;
i915->quirks & QUIRK_PIN_SWIZZLED_PAGES) {
GEM_BUG_ON(i915_gem_object_has_tiling_quirk(obj));
i915_gem_object_set_tiling_quirk(obj);
+ GEM_BUG_ON(!list_empty(&obj->mm.link));
+ atomic_inc(&obj->mm.shrink_pin);
shrinkable = false;
}
gen7_emit_pipeline_invalidate(&cmds);
batch_add(&cmds, MI_LOAD_REGISTER_IMM(2));
batch_add(&cmds, i915_mmio_reg_offset(CACHE_MODE_0_GEN7));
- batch_add(&cmds, 0xffff0000);
+ batch_add(&cmds, 0xffff0000 |
+ ((IS_IVB_GT1(i915) || IS_VALLEYVIEW(i915)) ?
+ HIZ_RAW_STALL_OPT_DISABLE :
+ 0));
batch_add(&cmds, i915_mmio_reg_offset(CACHE_MODE_1));
batch_add(&cmds, 0xffff0000 | PIXEL_SUBSPAN_COLLECT_OPT_DISABLE);
gen7_emit_pipeline_invalidate(&cmds);
err = pin_pt_dma(vm, pde->pt.base);
if (err) {
- i915_gem_object_put(pde->pt.base);
free_pd(vm, pde);
return err;
}
* banks of memory are paired and unswizzled on the
* uneven portion, so leave that as unknown.
*/
- if (intel_uncore_read(uncore, C0DRB3) ==
- intel_uncore_read(uncore, C1DRB3)) {
+ if (intel_uncore_read16(uncore, C0DRB3) ==
+ intel_uncore_read16(uncore, C1DRB3)) {
swizzle_x = I915_BIT_6_SWIZZLE_9_10;
swizzle_y = I915_BIT_6_SWIZZLE_9;
}
[INTEL_GVT_HYPERVISOR_KVM] = "KVM",
};
-static struct intel_vgpu_type *
-intel_gvt_find_vgpu_type(struct intel_gvt *gvt, unsigned int type_group_id)
-{
- if (WARN_ON(type_group_id >= gvt->num_types))
- return NULL;
- return &gvt->types[type_group_id];
-}
-
-static ssize_t available_instances_show(struct mdev_type *mtype,
- struct mdev_type_attribute *attr,
- char *buf)
-{
- struct intel_vgpu_type *type;
- unsigned int num = 0;
- void *gvt = kdev_to_i915(mtype_get_parent_dev(mtype))->gvt;
-
- type = intel_gvt_find_vgpu_type(gvt, mtype_get_type_group_id(mtype));
- if (!type)
- num = 0;
- else
- num = type->avail_instance;
-
- return sprintf(buf, "%u\n", num);
-}
-
-static ssize_t device_api_show(struct mdev_type *mtype,
- struct mdev_type_attribute *attr, char *buf)
-{
- return sprintf(buf, "%s\n", VFIO_DEVICE_API_PCI_STRING);
-}
-
-static ssize_t description_show(struct mdev_type *mtype,
- struct mdev_type_attribute *attr, char *buf)
-{
- struct intel_vgpu_type *type;
- void *gvt = kdev_to_i915(mtype_get_parent_dev(mtype))->gvt;
-
- type = intel_gvt_find_vgpu_type(gvt, mtype_get_type_group_id(mtype));
- if (!type)
- return 0;
-
- return sprintf(buf, "low_gm_size: %dMB\nhigh_gm_size: %dMB\n"
- "fence: %d\nresolution: %s\n"
- "weight: %d\n",
- BYTES_TO_MB(type->low_gm_size),
- BYTES_TO_MB(type->high_gm_size),
- type->fence, vgpu_edid_str(type->resolution),
- type->weight);
-}
-
-static MDEV_TYPE_ATTR_RO(available_instances);
-static MDEV_TYPE_ATTR_RO(device_api);
-static MDEV_TYPE_ATTR_RO(description);
-
-static struct attribute *gvt_type_attrs[] = {
- &mdev_type_attr_available_instances.attr,
- &mdev_type_attr_device_api.attr,
- &mdev_type_attr_description.attr,
- NULL,
-};
-
-static struct attribute_group *gvt_vgpu_type_groups[] = {
- [0 ... NR_MAX_INTEL_VGPU_TYPES - 1] = NULL,
-};
-
-static bool intel_get_gvt_attrs(struct attribute_group ***intel_vgpu_type_groups)
-{
- *intel_vgpu_type_groups = gvt_vgpu_type_groups;
- return true;
-}
-
-static int intel_gvt_init_vgpu_type_groups(struct intel_gvt *gvt)
-{
- int i, j;
- struct intel_vgpu_type *type;
- struct attribute_group *group;
-
- for (i = 0; i < gvt->num_types; i++) {
- type = &gvt->types[i];
-
- group = kzalloc(sizeof(struct attribute_group), GFP_KERNEL);
- if (WARN_ON(!group))
- goto unwind;
-
- group->name = type->name;
- group->attrs = gvt_type_attrs;
- gvt_vgpu_type_groups[i] = group;
- }
-
- return 0;
-
-unwind:
- for (j = 0; j < i; j++) {
- group = gvt_vgpu_type_groups[j];
- kfree(group);
- }
-
- return -ENOMEM;
-}
-
-static void intel_gvt_cleanup_vgpu_type_groups(struct intel_gvt *gvt)
-{
- int i;
- struct attribute_group *group;
-
- for (i = 0; i < gvt->num_types; i++) {
- group = gvt_vgpu_type_groups[i];
- gvt_vgpu_type_groups[i] = NULL;
- kfree(group);
- }
-}
-
static const struct intel_gvt_ops intel_gvt_ops = {
.emulate_cfg_read = intel_vgpu_emulate_cfg_read,
.emulate_cfg_write = intel_vgpu_emulate_cfg_write,
.vgpu_reset = intel_gvt_reset_vgpu,
.vgpu_activate = intel_gvt_activate_vgpu,
.vgpu_deactivate = intel_gvt_deactivate_vgpu,
- .gvt_find_vgpu_type = intel_gvt_find_vgpu_type,
- .get_gvt_attrs = intel_get_gvt_attrs,
.vgpu_query_plane = intel_vgpu_query_plane,
.vgpu_get_dmabuf = intel_vgpu_get_dmabuf,
.write_protect_handler = intel_vgpu_page_track_handler,
return;
intel_gvt_destroy_idle_vgpu(gvt->idle_vgpu);
- intel_gvt_cleanup_vgpu_type_groups(gvt);
intel_gvt_clean_vgpu_types(gvt);
intel_gvt_debugfs_clean(gvt);
if (ret)
goto out_clean_thread;
- ret = intel_gvt_init_vgpu_type_groups(gvt);
- if (ret) {
- gvt_err("failed to init vgpu type groups: %d\n", ret);
- goto out_clean_types;
- }
-
vgpu = intel_gvt_create_idle_vgpu(gvt);
if (IS_ERR(vgpu)) {
ret = PTR_ERR(vgpu);
void
intel_gvt_unregister_hypervisor(void)
{
- intel_gvt_hypervisor_host_exit(intel_gvt_host.dev);
+ void *gvt = (void *)kdev_to_i915(intel_gvt_host.dev)->gvt;
+ intel_gvt_hypervisor_host_exit(intel_gvt_host.dev, gvt);
module_put(THIS_MODULE);
}
EXPORT_SYMBOL_GPL(intel_gvt_unregister_hypervisor);
void (*vgpu_reset)(struct intel_vgpu *);
void (*vgpu_activate)(struct intel_vgpu *);
void (*vgpu_deactivate)(struct intel_vgpu *);
- struct intel_vgpu_type *(*gvt_find_vgpu_type)(
- struct intel_gvt *gvt, unsigned int type_group_id);
- bool (*get_gvt_attrs)(struct attribute_group ***intel_vgpu_type_groups);
int (*vgpu_query_plane)(struct intel_vgpu *vgpu, void *);
int (*vgpu_get_dmabuf)(struct intel_vgpu *vgpu, unsigned int);
int (*write_protect_handler)(struct intel_vgpu *, u64, void *,
link_n = vgpu_vreg_t(vgpu, PIPE_LINK_N1(TRANSCODER_A));
/* Get H/V total from transcoder timing */
- htotal = (vgpu_vreg_t(vgpu, HTOTAL(TRANSCODER_A)) >> TRANS_HTOTAL_SHIFT) + 1;
- vtotal = (vgpu_vreg_t(vgpu, VTOTAL(TRANSCODER_A)) >> TRANS_VTOTAL_SHIFT) + 1;
+ htotal = (vgpu_vreg_t(vgpu, HTOTAL(TRANSCODER_A)) >> TRANS_HTOTAL_SHIFT);
+ vtotal = (vgpu_vreg_t(vgpu, VTOTAL(TRANSCODER_A)) >> TRANS_VTOTAL_SHIFT);
if (dp_br && link_n && htotal && vtotal) {
u64 pixel_clk = 0;
pixel_clk *= MSEC_PER_SEC;
/* Calcuate refresh rate by (pixel_clk / (h_total * v_total)) */
- new_rate = DIV64_U64_ROUND_CLOSEST(pixel_clk, div64_u64(mul_u32_u32(htotal, vtotal), MSEC_PER_SEC));
+ new_rate = DIV64_U64_ROUND_CLOSEST(mul_u64_u32_shr(pixel_clk, MSEC_PER_SEC, 0), mul_u32_u32(htotal + 1, vtotal + 1));
if (*old_rate != new_rate)
*old_rate = new_rate;
struct intel_gvt_mpt {
enum hypervisor_type type;
int (*host_init)(struct device *dev, void *gvt, const void *ops);
- void (*host_exit)(struct device *dev);
+ void (*host_exit)(struct device *dev, void *gvt);
int (*attach_vgpu)(void *vgpu, unsigned long *handle);
void (*detach_vgpu)(void *vgpu);
int (*inject_msi)(unsigned long handle, u32 addr, u16 data);
return !!(handle & ~0xff);
}
+static ssize_t available_instances_show(struct mdev_type *mtype,
+ struct mdev_type_attribute *attr,
+ char *buf)
+{
+ struct intel_vgpu_type *type;
+ unsigned int num = 0;
+ struct intel_gvt *gvt = kdev_to_i915(mtype_get_parent_dev(mtype))->gvt;
+
+ type = &gvt->types[mtype_get_type_group_id(mtype)];
+ if (!type)
+ num = 0;
+ else
+ num = type->avail_instance;
+
+ return sprintf(buf, "%u\n", num);
+}
+
+static ssize_t device_api_show(struct mdev_type *mtype,
+ struct mdev_type_attribute *attr, char *buf)
+{
+ return sprintf(buf, "%s\n", VFIO_DEVICE_API_PCI_STRING);
+}
+
+static ssize_t description_show(struct mdev_type *mtype,
+ struct mdev_type_attribute *attr, char *buf)
+{
+ struct intel_vgpu_type *type;
+ struct intel_gvt *gvt = kdev_to_i915(mtype_get_parent_dev(mtype))->gvt;
+
+ type = &gvt->types[mtype_get_type_group_id(mtype)];
+ if (!type)
+ return 0;
+
+ return sprintf(buf, "low_gm_size: %dMB\nhigh_gm_size: %dMB\n"
+ "fence: %d\nresolution: %s\n"
+ "weight: %d\n",
+ BYTES_TO_MB(type->low_gm_size),
+ BYTES_TO_MB(type->high_gm_size),
+ type->fence, vgpu_edid_str(type->resolution),
+ type->weight);
+}
+
+static MDEV_TYPE_ATTR_RO(available_instances);
+static MDEV_TYPE_ATTR_RO(device_api);
+static MDEV_TYPE_ATTR_RO(description);
+
+static struct attribute *gvt_type_attrs[] = {
+ &mdev_type_attr_available_instances.attr,
+ &mdev_type_attr_device_api.attr,
+ &mdev_type_attr_description.attr,
+ NULL,
+};
+
+static struct attribute_group *gvt_vgpu_type_groups[] = {
+ [0 ... NR_MAX_INTEL_VGPU_TYPES - 1] = NULL,
+};
+
+static int intel_gvt_init_vgpu_type_groups(struct intel_gvt *gvt)
+{
+ int i, j;
+ struct intel_vgpu_type *type;
+ struct attribute_group *group;
+
+ for (i = 0; i < gvt->num_types; i++) {
+ type = &gvt->types[i];
+
+ group = kzalloc(sizeof(struct attribute_group), GFP_KERNEL);
+ if (!group)
+ goto unwind;
+
+ group->name = type->name;
+ group->attrs = gvt_type_attrs;
+ gvt_vgpu_type_groups[i] = group;
+ }
+
+ return 0;
+
+unwind:
+ for (j = 0; j < i; j++) {
+ group = gvt_vgpu_type_groups[j];
+ kfree(group);
+ }
+
+ return -ENOMEM;
+}
+
+static void intel_gvt_cleanup_vgpu_type_groups(struct intel_gvt *gvt)
+{
+ int i;
+ struct attribute_group *group;
+
+ for (i = 0; i < gvt->num_types; i++) {
+ group = gvt_vgpu_type_groups[i];
+ gvt_vgpu_type_groups[i] = NULL;
+ kfree(group);
+ }
+}
+
static int kvmgt_guest_init(struct mdev_device *mdev);
static void intel_vgpu_release_work(struct work_struct *work);
static bool kvmgt_guest_exit(struct kvmgt_guest_info *info);
struct intel_vgpu *vgpu = NULL;
struct intel_vgpu_type *type;
struct device *pdev;
- void *gvt;
+ struct intel_gvt *gvt;
int ret;
pdev = mdev_parent_dev(mdev);
gvt = kdev_to_i915(pdev)->gvt;
- type = intel_gvt_ops->gvt_find_vgpu_type(gvt,
- mdev_get_type_group_id(mdev));
+ type = &gvt->types[mdev_get_type_group_id(mdev)];
if (!type) {
ret = -EINVAL;
goto out;
static int kvmgt_host_init(struct device *dev, void *gvt, const void *ops)
{
- struct attribute_group **kvm_vgpu_type_groups;
+ int ret;
+
+ ret = intel_gvt_init_vgpu_type_groups((struct intel_gvt *)gvt);
+ if (ret)
+ return ret;
intel_gvt_ops = ops;
- if (!intel_gvt_ops->get_gvt_attrs(&kvm_vgpu_type_groups))
- return -EFAULT;
- intel_vgpu_ops.supported_type_groups = kvm_vgpu_type_groups;
+ intel_vgpu_ops.supported_type_groups = gvt_vgpu_type_groups;
- return mdev_register_device(dev, &intel_vgpu_ops);
+ ret = mdev_register_device(dev, &intel_vgpu_ops);
+ if (ret)
+ intel_gvt_cleanup_vgpu_type_groups((struct intel_gvt *)gvt);
+
+ return ret;
}
-static void kvmgt_host_exit(struct device *dev)
+static void kvmgt_host_exit(struct device *dev, void *gvt)
{
mdev_unregister_device(dev);
+ intel_gvt_cleanup_vgpu_type_groups((struct intel_gvt *)gvt);
}
static int kvmgt_page_track_add(unsigned long handle, u64 gfn)
/**
* intel_gvt_hypervisor_host_exit - exit GVT-g host side
*/
-static inline void intel_gvt_hypervisor_host_exit(struct device *dev)
+static inline void intel_gvt_hypervisor_host_exit(struct device *dev, void *gvt)
{
/* optional to provide */
if (!intel_gvt_host.mpt->host_exit)
return;
- intel_gvt_host.mpt->host_exit(dev);
+ intel_gvt_host.mpt->host_exit(dev, gvt);
}
/**
return 0;
}
-static void auto_retire(struct i915_active *ref)
+__i915_active_call static void
+auto_retire(struct i915_active *ref)
{
i915_active_put(ref);
}
struct drm_file *file);
/* i915_mm.c */
+int remap_io_mapping(struct vm_area_struct *vma,
+ unsigned long addr, unsigned long pfn, unsigned long size,
+ struct io_mapping *iomap);
int remap_io_sg(struct vm_area_struct *vma,
unsigned long addr, unsigned long size,
struct scatterlist *sgl, resource_size_t iobase);
obj->mm.madv = args->madv;
if (i915_gem_object_has_pages(obj)) {
- struct list_head *list;
+ unsigned long flags;
- if (i915_gem_object_is_shrinkable(obj)) {
- unsigned long flags;
-
- spin_lock_irqsave(&i915->mm.obj_lock, flags);
+ spin_lock_irqsave(&i915->mm.obj_lock, flags);
+ if (!list_empty(&obj->mm.link)) {
+ struct list_head *list;
if (obj->mm.madv != I915_MADV_WILLNEED)
list = &i915->mm.purge_list;
list = &i915->mm.shrink_list;
list_move_tail(&obj->mm.link, list);
- spin_unlock_irqrestore(&i915->mm.obj_lock, flags);
}
+ spin_unlock_irqrestore(&i915->mm.obj_lock, flags);
}
/* if the object is no longer attached, discard its backing storage */
#include "i915_drv.h"
-#define EXPECTED_FLAGS (VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP)
+struct remap_pfn {
+ struct mm_struct *mm;
+ unsigned long pfn;
+ pgprot_t prot;
+
+ struct sgt_iter sgt;
+ resource_size_t iobase;
+};
+
+static int remap_pfn(pte_t *pte, unsigned long addr, void *data)
+{
+ struct remap_pfn *r = data;
+
+ /* Special PTE are not associated with any struct page */
+ set_pte_at(r->mm, addr, pte, pte_mkspecial(pfn_pte(r->pfn, r->prot)));
+ r->pfn++;
+
+ return 0;
+}
#define use_dma(io) ((io) != -1)
+static inline unsigned long sgt_pfn(const struct remap_pfn *r)
+{
+ if (use_dma(r->iobase))
+ return (r->sgt.dma + r->sgt.curr + r->iobase) >> PAGE_SHIFT;
+ else
+ return r->sgt.pfn + (r->sgt.curr >> PAGE_SHIFT);
+}
+
+static int remap_sg(pte_t *pte, unsigned long addr, void *data)
+{
+ struct remap_pfn *r = data;
+
+ if (GEM_WARN_ON(!r->sgt.sgp))
+ return -EINVAL;
+
+ /* Special PTE are not associated with any struct page */
+ set_pte_at(r->mm, addr, pte,
+ pte_mkspecial(pfn_pte(sgt_pfn(r), r->prot)));
+ r->pfn++; /* track insertions in case we need to unwind later */
+
+ r->sgt.curr += PAGE_SIZE;
+ if (r->sgt.curr >= r->sgt.max)
+ r->sgt = __sgt_iter(__sg_next(r->sgt.sgp), use_dma(r->iobase));
+
+ return 0;
+}
+
+/**
+ * remap_io_mapping - remap an IO mapping to userspace
+ * @vma: user vma to map to
+ * @addr: target user address to start at
+ * @pfn: physical address of kernel memory
+ * @size: size of map area
+ * @iomap: the source io_mapping
+ *
+ * Note: this is only safe if the mm semaphore is held when called.
+ */
+int remap_io_mapping(struct vm_area_struct *vma,
+ unsigned long addr, unsigned long pfn, unsigned long size,
+ struct io_mapping *iomap)
+{
+ struct remap_pfn r;
+ int err;
+
+#define EXPECTED_FLAGS (VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP)
+ GEM_BUG_ON((vma->vm_flags & EXPECTED_FLAGS) != EXPECTED_FLAGS);
+
+ /* We rely on prevalidation of the io-mapping to skip track_pfn(). */
+ r.mm = vma->vm_mm;
+ r.pfn = pfn;
+ r.prot = __pgprot((pgprot_val(iomap->prot) & _PAGE_CACHE_MASK) |
+ (pgprot_val(vma->vm_page_prot) & ~_PAGE_CACHE_MASK));
+
+ err = apply_to_page_range(r.mm, addr, size, remap_pfn, &r);
+ if (unlikely(err)) {
+ zap_vma_ptes(vma, addr, (r.pfn - pfn) << PAGE_SHIFT);
+ return err;
+ }
+
+ return 0;
+}
+
/**
* remap_io_sg - remap an IO mapping to userspace
* @vma: user vma to map to
unsigned long addr, unsigned long size,
struct scatterlist *sgl, resource_size_t iobase)
{
- unsigned long pfn, len, remapped = 0;
+ struct remap_pfn r = {
+ .mm = vma->vm_mm,
+ .prot = vma->vm_page_prot,
+ .sgt = __sgt_iter(sgl, use_dma(iobase)),
+ .iobase = iobase,
+ };
int err;
/* We rely on prevalidation of the io-mapping to skip track_pfn(). */
if (!use_dma(iobase))
flush_cache_range(vma, addr, size);
- do {
- if (use_dma(iobase)) {
- if (!sg_dma_len(sgl))
- break;
- pfn = (sg_dma_address(sgl) + iobase) >> PAGE_SHIFT;
- len = sg_dma_len(sgl);
- } else {
- pfn = page_to_pfn(sg_page(sgl));
- len = sgl->length;
- }
-
- err = remap_pfn_range(vma, addr + remapped, pfn, len,
- vma->vm_page_prot);
- if (err)
- break;
- remapped += len;
- } while ((sgl = __sg_next(sgl)));
-
- if (err)
- zap_vma_ptes(vma, addr, remapped);
- return err;
+ err = apply_to_page_range(r.mm, addr, size, remap_sg, &r);
+ if (unlikely(err)) {
+ zap_vma_ptes(vma, addr, r.pfn << PAGE_SHIFT);
+ return err;
+ }
+
+ return 0;
}
for (n = 0; n < smoke[0].ncontexts; n++) {
smoke[0].contexts[n] = live_context(i915, file);
- if (!smoke[0].contexts[n]) {
- ret = -ENOMEM;
+ if (IS_ERR(smoke[0].contexts[n])) {
+ ret = PTR_ERR(smoke[0].contexts[n]);
goto out_contexts;
}
}
* porches and sync.
*/
/* (ps/s) / (pixels/s) = ps/pixels */
- pclk = DIV_ROUND_UP_ULL(1000000000000, mode->clock);
+ pclk = DIV_ROUND_UP_ULL(1000000000000, (mode->clock * 1000));
dev_dbg(d->dev, "picoseconds between two pixels: %llu\n",
pclk);
static void meson_drv_shutdown(struct platform_device *pdev)
{
struct meson_drm *priv = dev_get_drvdata(&pdev->dev);
- struct drm_device *drm = priv->drm;
- DRM_DEBUG_DRIVER("\n");
- drm_kms_helper_poll_fini(drm);
- drm_atomic_helper_shutdown(drm);
+ if (!priv)
+ return;
+
+ drm_kms_helper_poll_fini(priv->drm);
+ drm_atomic_helper_shutdown(priv->drm);
}
static int meson_drv_probe(struct platform_device *pdev)
* GPU registers so we need to add 0x1a800 to the register value on A630
* to get the right value from PM4.
*/
- get_stats_counter(ring, REG_A6XX_GMU_ALWAYS_ON_COUNTER_L + 0x1a800,
+ get_stats_counter(ring, REG_A6XX_CP_ALWAYS_ON_COUNTER_LO,
rbmemptr_stats(ring, index, alwayson_start));
/* Invalidate CCU depth and color */
get_stats_counter(ring, REG_A6XX_RBBM_PERFCTR_CP_0_LO,
rbmemptr_stats(ring, index, cpcycles_end));
- get_stats_counter(ring, REG_A6XX_GMU_ALWAYS_ON_COUNTER_L + 0x1a800,
+ get_stats_counter(ring, REG_A6XX_CP_ALWAYS_ON_COUNTER_LO,
rbmemptr_stats(ring, index, alwayson_end));
/* Write the fence to the scratch register */
OUT_RING(ring, submit->seqno);
trace_msm_gpu_submit_flush(submit,
- gmu_read64(&a6xx_gpu->gmu, REG_A6XX_GMU_ALWAYS_ON_COUNTER_L,
- REG_A6XX_GMU_ALWAYS_ON_COUNTER_H));
+ gpu_read64(gpu, REG_A6XX_CP_ALWAYS_ON_COUNTER_LO,
+ REG_A6XX_CP_ALWAYS_ON_COUNTER_HI));
a6xx_flush(gpu, ring);
}
gpu_write(gpu, REG_A6XX_RBBM_CLOCK_CNTL, state ? clock_cntl_on : 0);
}
+/* For a615, a616, a618, A619, a630, a640 and a680 */
+static const u32 a6xx_protect[] = {
+ A6XX_PROTECT_RDONLY(0x00000, 0x04ff),
+ A6XX_PROTECT_RDONLY(0x00501, 0x0005),
+ A6XX_PROTECT_RDONLY(0x0050b, 0x02f4),
+ A6XX_PROTECT_NORDWR(0x0050e, 0x0000),
+ A6XX_PROTECT_NORDWR(0x00510, 0x0000),
+ A6XX_PROTECT_NORDWR(0x00534, 0x0000),
+ A6XX_PROTECT_NORDWR(0x00800, 0x0082),
+ A6XX_PROTECT_NORDWR(0x008a0, 0x0008),
+ A6XX_PROTECT_NORDWR(0x008ab, 0x0024),
+ A6XX_PROTECT_RDONLY(0x008de, 0x00ae),
+ A6XX_PROTECT_NORDWR(0x00900, 0x004d),
+ A6XX_PROTECT_NORDWR(0x0098d, 0x0272),
+ A6XX_PROTECT_NORDWR(0x00e00, 0x0001),
+ A6XX_PROTECT_NORDWR(0x00e03, 0x000c),
+ A6XX_PROTECT_NORDWR(0x03c00, 0x00c3),
+ A6XX_PROTECT_RDONLY(0x03cc4, 0x1fff),
+ A6XX_PROTECT_NORDWR(0x08630, 0x01cf),
+ A6XX_PROTECT_NORDWR(0x08e00, 0x0000),
+ A6XX_PROTECT_NORDWR(0x08e08, 0x0000),
+ A6XX_PROTECT_NORDWR(0x08e50, 0x001f),
+ A6XX_PROTECT_NORDWR(0x09624, 0x01db),
+ A6XX_PROTECT_NORDWR(0x09e70, 0x0001),
+ A6XX_PROTECT_NORDWR(0x09e78, 0x0187),
+ A6XX_PROTECT_NORDWR(0x0a630, 0x01cf),
+ A6XX_PROTECT_NORDWR(0x0ae02, 0x0000),
+ A6XX_PROTECT_NORDWR(0x0ae50, 0x032f),
+ A6XX_PROTECT_NORDWR(0x0b604, 0x0000),
+ A6XX_PROTECT_NORDWR(0x0be02, 0x0001),
+ A6XX_PROTECT_NORDWR(0x0be20, 0x17df),
+ A6XX_PROTECT_NORDWR(0x0f000, 0x0bff),
+ A6XX_PROTECT_RDONLY(0x0fc00, 0x1fff),
+ A6XX_PROTECT_NORDWR(0x11c00, 0x0000), /* note: infinite range */
+};
+
+/* These are for a620 and a650 */
+static const u32 a650_protect[] = {
+ A6XX_PROTECT_RDONLY(0x00000, 0x04ff),
+ A6XX_PROTECT_RDONLY(0x00501, 0x0005),
+ A6XX_PROTECT_RDONLY(0x0050b, 0x02f4),
+ A6XX_PROTECT_NORDWR(0x0050e, 0x0000),
+ A6XX_PROTECT_NORDWR(0x00510, 0x0000),
+ A6XX_PROTECT_NORDWR(0x00534, 0x0000),
+ A6XX_PROTECT_NORDWR(0x00800, 0x0082),
+ A6XX_PROTECT_NORDWR(0x008a0, 0x0008),
+ A6XX_PROTECT_NORDWR(0x008ab, 0x0024),
+ A6XX_PROTECT_RDONLY(0x008de, 0x00ae),
+ A6XX_PROTECT_NORDWR(0x00900, 0x004d),
+ A6XX_PROTECT_NORDWR(0x0098d, 0x0272),
+ A6XX_PROTECT_NORDWR(0x00e00, 0x0001),
+ A6XX_PROTECT_NORDWR(0x00e03, 0x000c),
+ A6XX_PROTECT_NORDWR(0x03c00, 0x00c3),
+ A6XX_PROTECT_RDONLY(0x03cc4, 0x1fff),
+ A6XX_PROTECT_NORDWR(0x08630, 0x01cf),
+ A6XX_PROTECT_NORDWR(0x08e00, 0x0000),
+ A6XX_PROTECT_NORDWR(0x08e08, 0x0000),
+ A6XX_PROTECT_NORDWR(0x08e50, 0x001f),
+ A6XX_PROTECT_NORDWR(0x08e80, 0x027f),
+ A6XX_PROTECT_NORDWR(0x09624, 0x01db),
+ A6XX_PROTECT_NORDWR(0x09e60, 0x0011),
+ A6XX_PROTECT_NORDWR(0x09e78, 0x0187),
+ A6XX_PROTECT_NORDWR(0x0a630, 0x01cf),
+ A6XX_PROTECT_NORDWR(0x0ae02, 0x0000),
+ A6XX_PROTECT_NORDWR(0x0ae50, 0x032f),
+ A6XX_PROTECT_NORDWR(0x0b604, 0x0000),
+ A6XX_PROTECT_NORDWR(0x0b608, 0x0007),
+ A6XX_PROTECT_NORDWR(0x0be02, 0x0001),
+ A6XX_PROTECT_NORDWR(0x0be20, 0x17df),
+ A6XX_PROTECT_NORDWR(0x0f000, 0x0bff),
+ A6XX_PROTECT_RDONLY(0x0fc00, 0x1fff),
+ A6XX_PROTECT_NORDWR(0x18400, 0x1fff),
+ A6XX_PROTECT_NORDWR(0x1a800, 0x1fff),
+ A6XX_PROTECT_NORDWR(0x1f400, 0x0443),
+ A6XX_PROTECT_RDONLY(0x1f844, 0x007b),
+ A6XX_PROTECT_NORDWR(0x1f887, 0x001b),
+ A6XX_PROTECT_NORDWR(0x1f8c0, 0x0000), /* note: infinite range */
+};
+
+static void a6xx_set_cp_protect(struct msm_gpu *gpu)
+{
+ struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu);
+ const u32 *regs = a6xx_protect;
+ unsigned i, count = ARRAY_SIZE(a6xx_protect), count_max = 32;
+
+ BUILD_BUG_ON(ARRAY_SIZE(a6xx_protect) > 32);
+ BUILD_BUG_ON(ARRAY_SIZE(a650_protect) > 48);
+
+ if (adreno_is_a650(adreno_gpu)) {
+ regs = a650_protect;
+ count = ARRAY_SIZE(a650_protect);
+ count_max = 48;
+ }
+
+ /*
+ * Enable access protection to privileged registers, fault on an access
+ * protect violation and select the last span to protect from the start
+ * address all the way to the end of the register address space
+ */
+ gpu_write(gpu, REG_A6XX_CP_PROTECT_CNTL, BIT(0) | BIT(1) | BIT(3));
+
+ for (i = 0; i < count - 1; i++)
+ gpu_write(gpu, REG_A6XX_CP_PROTECT(i), regs[i]);
+ /* last CP_PROTECT to have "infinite" length on the last entry */
+ gpu_write(gpu, REG_A6XX_CP_PROTECT(count_max - 1), regs[i]);
+}
+
static void a6xx_set_ubwc_config(struct msm_gpu *gpu)
{
struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu);
rgb565_predicator << 11 | amsbc << 4 | lower_bit << 1);
gpu_write(gpu, REG_A6XX_TPL1_NC_MODE_CNTL, lower_bit << 1);
gpu_write(gpu, REG_A6XX_SP_NC_MODE_CNTL,
- uavflagprd_inv >> 4 | lower_bit << 1);
+ uavflagprd_inv << 4 | lower_bit << 1);
gpu_write(gpu, REG_A6XX_UCHE_MODE_CNTL, lower_bit << 21);
}
}
/* Protect registers from the CP */
- gpu_write(gpu, REG_A6XX_CP_PROTECT_CNTL, 0x00000003);
-
- gpu_write(gpu, REG_A6XX_CP_PROTECT(0),
- A6XX_PROTECT_RDONLY(0x600, 0x51));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(1), A6XX_PROTECT_RW(0xae50, 0x2));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(2), A6XX_PROTECT_RW(0x9624, 0x13));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(3), A6XX_PROTECT_RW(0x8630, 0x8));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(4), A6XX_PROTECT_RW(0x9e70, 0x1));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(5), A6XX_PROTECT_RW(0x9e78, 0x187));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(6), A6XX_PROTECT_RW(0xf000, 0x810));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(7),
- A6XX_PROTECT_RDONLY(0xfc00, 0x3));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(8), A6XX_PROTECT_RW(0x50e, 0x0));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(9), A6XX_PROTECT_RDONLY(0x50f, 0x0));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(10), A6XX_PROTECT_RW(0x510, 0x0));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(11),
- A6XX_PROTECT_RDONLY(0x0, 0x4f9));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(12),
- A6XX_PROTECT_RDONLY(0x501, 0xa));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(13),
- A6XX_PROTECT_RDONLY(0x511, 0x44));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(14), A6XX_PROTECT_RW(0xe00, 0xe));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(15), A6XX_PROTECT_RW(0x8e00, 0x0));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(16), A6XX_PROTECT_RW(0x8e50, 0xf));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(17), A6XX_PROTECT_RW(0xbe02, 0x0));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(18),
- A6XX_PROTECT_RW(0xbe20, 0x11f3));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(19), A6XX_PROTECT_RW(0x800, 0x82));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(20), A6XX_PROTECT_RW(0x8a0, 0x8));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(21), A6XX_PROTECT_RW(0x8ab, 0x19));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(22), A6XX_PROTECT_RW(0x900, 0x4d));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(23), A6XX_PROTECT_RW(0x98d, 0x76));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(24),
- A6XX_PROTECT_RDONLY(0x980, 0x4));
- gpu_write(gpu, REG_A6XX_CP_PROTECT(25), A6XX_PROTECT_RW(0xa630, 0x0));
+ a6xx_set_cp_protect(gpu);
/* Enable expanded apriv for targets that support it */
if (gpu->hw_apriv) {
{
struct device_node *phandle;
- a6xx_gpu->llc_mmio = msm_ioremap(pdev, "cx_mem", "gpu_cx");
- if (IS_ERR(a6xx_gpu->llc_mmio))
- return;
-
/*
* There is a different programming path for targets with an mmu500
* attached, so detect if that is the case
of_device_is_compatible(phandle, "arm,mmu-500"));
of_node_put(phandle);
+ if (a6xx_gpu->have_mmu500)
+ a6xx_gpu->llc_mmio = NULL;
+ else
+ a6xx_gpu->llc_mmio = msm_ioremap(pdev, "cx_mem", "gpu_cx");
+
a6xx_gpu->llc_slice = llcc_slice_getd(LLCC_GPU);
a6xx_gpu->htw_llc_slice = llcc_slice_getd(LLCC_GPUHTW);
if (ret)
return ret;
- if (adreno_gpu->base.hw_apriv || a6xx_gpu->has_whereami)
+ if (a6xx_gpu->shadow_bo)
for (i = 0; i < gpu->nr_rings; i++)
a6xx_gpu->shadow[i] = 0;
* REG_CP_PROTECT_REG(n) - this will block both reads and writes for _len
* registers starting at _reg.
*/
-#define A6XX_PROTECT_RW(_reg, _len) \
+#define A6XX_PROTECT_NORDWR(_reg, _len) \
((1 << 31) | \
(((_len) & 0x3FFF) << 18) | ((_reg) & 0x3FFFF))
if (unlikely(!cstate->num_mixers))
return;
- /*
- * For planes without commit update, drm framework will not add
- * those planes to current state since hardware update is not
- * required. However, if those planes were power collapsed since
- * last commit cycle, driver has to restore the hardware state
- * of those planes explicitly here prior to plane flush.
- */
- drm_atomic_crtc_for_each_plane(plane, crtc)
- dpu_plane_restore(plane, state);
-
/* update performance setting before crtc kickoff */
dpu_core_perf_crtc_update(crtc, 1, false);
}
}
-void dpu_plane_restore(struct drm_plane *plane, struct drm_atomic_state *state)
-{
- struct dpu_plane *pdpu;
-
- if (!plane || !plane->state) {
- DPU_ERROR("invalid plane\n");
- return;
- }
-
- pdpu = to_dpu_plane(plane);
-
- DPU_DEBUG_PLANE(pdpu, "\n");
-
- dpu_plane_atomic_update(plane, state);
-}
-
static void dpu_plane_destroy(struct drm_plane *plane)
{
struct dpu_plane *pdpu = plane ? to_dpu_plane(plane) : NULL;
u32 *flush_sspp);
/**
- * dpu_plane_restore - restore hw state if previously power collapsed
- * @plane: Pointer to drm plane structure
- */
-void dpu_plane_restore(struct drm_plane *plane, struct drm_atomic_state *state);
-
-/**
* dpu_plane_flush - final plane operations before commit flush
* @plane: Pointer to drm plane structure
*/
dp_audio_setup_acr(audio);
dp_audio_safe_to_exit_level(audio);
dp_audio_enable(audio, true);
+ dp_display_signal_audio_start(dp_display);
dp_display->audio_enabled = true;
end:
return 0;
}
+void dp_display_signal_audio_start(struct msm_dp *dp_display)
+{
+ struct dp_display_private *dp;
+
+ dp = container_of(dp_display, struct dp_display_private, dp_display);
+
+ reinit_completion(&dp->audio_comp);
+}
+
void dp_display_signal_audio_complete(struct msm_dp *dp_display)
{
struct dp_display_private *dp;
mutex_lock(&dp->event_mutex);
state = dp->hpd_state;
- if (state == ST_CONNECT_PENDING) {
- dp_display_enable(dp, 0);
+ if (state == ST_CONNECT_PENDING)
dp->hpd_state = ST_CONNECTED;
- }
mutex_unlock(&dp->event_mutex);
dp_add_event(dp, EV_DISCONNECT_PENDING_TIMEOUT, 0, DP_TIMEOUT_5_SECOND);
/* signal the disconnect event early to ensure proper teardown */
- reinit_completion(&dp->audio_comp);
dp_display_handle_plugged_change(g_dp_display, false);
dp_catalog_hpd_config_intr(dp->catalog, DP_DP_HPD_PLUG_INT_MASK |
mutex_lock(&dp->event_mutex);
state = dp->hpd_state;
- if (state == ST_DISCONNECT_PENDING) {
- dp_display_disable(dp, 0);
+ if (state == ST_DISCONNECT_PENDING)
dp->hpd_state = ST_DISCONNECTED;
- }
mutex_unlock(&dp->event_mutex);
/* wait only if audio was enabled */
if (dp_display->audio_enabled) {
/* signal the disconnect event */
- reinit_completion(&dp->audio_comp);
dp_display_handle_plugged_change(dp_display, false);
if (!wait_for_completion_timeout(&dp->audio_comp,
HZ * 5))
status = dp_catalog_link_is_connected(dp->catalog);
- if (status)
+ /*
+ * can not declared display is connected unless
+ * HDMI cable is plugged in and sink_count of
+ * dongle become 1
+ */
+ if (status && dp->link->sink_count)
dp->dp_display.is_connected = true;
else
dp->dp_display.is_connected = false;
int dp_display_request_irq(struct msm_dp *dp_display);
bool dp_display_check_video_test(struct msm_dp *dp_display);
int dp_display_get_test_bpp(struct msm_dp *dp_display);
+void dp_display_signal_audio_start(struct msm_dp *dp_display);
void dp_display_signal_audio_complete(struct msm_dp *dp_display);
#endif /* _DP_DISPLAY_H_ */
if (pixel_clk_provider)
*pixel_clk_provider = phy->provided_clocks->hws[DSI_PIXEL_PLL_CLK]->clk;
- return -EINVAL;
+ return 0;
}
void msm_dsi_phy_pll_save_state(struct msm_dsi_phy *phy)
pll_freq += div_u64(tmp64, multiplier);
vco_rate = pll_freq;
+ pll_10nm->vco_current_rate = vco_rate;
DBG("DSI PLL%d returning vco rate = %lu, dec = %x, frac = %x",
pll_10nm->phy->id, (unsigned long)vco_rate, dec, frac);
if (!vco_name)
return -ENOMEM;
+ parent_name = devm_kzalloc(dev, 32, GFP_KERNEL);
+ if (!parent_name)
+ return -ENOMEM;
+
clk_name = devm_kzalloc(dev, 32, GFP_KERNEL);
if (!clk_name)
return -ENOMEM;
pll_freq += div_u64(tmp64, multiplier);
vco_rate = pll_freq;
+ pll_7nm->vco_current_rate = vco_rate;
DBG("DSI PLL%d returning vco rate = %lu, dec = %x, frac = %x",
pll_7nm->phy->id, (unsigned long)vco_rate, dec, frac);
* - 1.7.0 - Add MSM_PARAM_SUSPENDS to access suspend count
*/
#define MSM_VERSION_MAJOR 1
-#define MSM_VERSION_MINOR 6
+#define MSM_VERSION_MINOR 7
#define MSM_VERSION_PATCHLEVEL 0
static const struct drm_mode_config_funcs mode_config_funcs = {
}
p = get_pages(obj);
+
+ if (!IS_ERR(p)) {
+ msm_obj->pin_count++;
+ update_inactive(msm_obj);
+ }
+
msm_gem_unlock(obj);
return p;
}
void msm_gem_put_pages(struct drm_gem_object *obj)
{
- /* when we start tracking the pin count, then do something here */
+ struct msm_gem_object *msm_obj = to_msm_bo(obj);
+
+ msm_gem_lock(obj);
+ msm_obj->pin_count--;
+ GEM_WARN_ON(msm_obj->pin_count < 0);
+ update_inactive(msm_obj);
+ msm_gem_unlock(obj);
}
int msm_gem_mmap_obj(struct drm_gem_object *obj,
ret = -ENOMEM;
goto fail;
}
+
+ update_inactive(msm_obj);
}
return msm_obj->vaddr;
to_msm_bo(obj)->vram_node = &vma->node;
+ /* Call chain get_pages() -> update_inactive() tries to
+ * access msm_obj->mm_list, but it is not initialized yet.
+ * To avoid NULL pointer dereference error, initialize
+ * mm_list to be empty.
+ */
+ INIT_LIST_HEAD(&msm_obj->mm_list);
+
msm_gem_lock(obj);
pages = get_pages(obj);
msm_gem_unlock(obj);
/* imported/exported objects are not purgeable: */
static inline bool is_unpurgeable(struct msm_gem_object *msm_obj)
{
- return msm_obj->base.dma_buf && msm_obj->base.import_attach;
+ return msm_obj->base.import_attach || msm_obj->pin_count;
}
static inline bool is_purgeable(struct msm_gem_object *msm_obj)
static inline bool is_unevictable(struct msm_gem_object *msm_obj)
{
- return is_unpurgeable(msm_obj) || msm_obj->pin_count || msm_obj->vaddr;
+ return is_unpurgeable(msm_obj) || msm_obj->vaddr;
}
static inline void mark_evictable(struct msm_gem_object *msm_obj)
u32 reg;
int ret;
- table->initialState.levels[0].mclk.vMPLL_AD_FUNC_CNTL =
+ table->initialState.level.mclk.vMPLL_AD_FUNC_CNTL =
cpu_to_be32(ni_pi->clock_registers.mpll_ad_func_cntl);
- table->initialState.levels[0].mclk.vMPLL_AD_FUNC_CNTL_2 =
+ table->initialState.level.mclk.vMPLL_AD_FUNC_CNTL_2 =
cpu_to_be32(ni_pi->clock_registers.mpll_ad_func_cntl_2);
- table->initialState.levels[0].mclk.vMPLL_DQ_FUNC_CNTL =
+ table->initialState.level.mclk.vMPLL_DQ_FUNC_CNTL =
cpu_to_be32(ni_pi->clock_registers.mpll_dq_func_cntl);
- table->initialState.levels[0].mclk.vMPLL_DQ_FUNC_CNTL_2 =
+ table->initialState.level.mclk.vMPLL_DQ_FUNC_CNTL_2 =
cpu_to_be32(ni_pi->clock_registers.mpll_dq_func_cntl_2);
- table->initialState.levels[0].mclk.vMCLK_PWRMGT_CNTL =
+ table->initialState.level.mclk.vMCLK_PWRMGT_CNTL =
cpu_to_be32(ni_pi->clock_registers.mclk_pwrmgt_cntl);
- table->initialState.levels[0].mclk.vDLL_CNTL =
+ table->initialState.level.mclk.vDLL_CNTL =
cpu_to_be32(ni_pi->clock_registers.dll_cntl);
- table->initialState.levels[0].mclk.vMPLL_SS =
+ table->initialState.level.mclk.vMPLL_SS =
cpu_to_be32(ni_pi->clock_registers.mpll_ss1);
- table->initialState.levels[0].mclk.vMPLL_SS2 =
+ table->initialState.level.mclk.vMPLL_SS2 =
cpu_to_be32(ni_pi->clock_registers.mpll_ss2);
- table->initialState.levels[0].mclk.mclk_value =
+ table->initialState.level.mclk.mclk_value =
cpu_to_be32(initial_state->performance_levels[0].mclk);
- table->initialState.levels[0].sclk.vCG_SPLL_FUNC_CNTL =
+ table->initialState.level.sclk.vCG_SPLL_FUNC_CNTL =
cpu_to_be32(ni_pi->clock_registers.cg_spll_func_cntl);
- table->initialState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_2 =
+ table->initialState.level.sclk.vCG_SPLL_FUNC_CNTL_2 =
cpu_to_be32(ni_pi->clock_registers.cg_spll_func_cntl_2);
- table->initialState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_3 =
+ table->initialState.level.sclk.vCG_SPLL_FUNC_CNTL_3 =
cpu_to_be32(ni_pi->clock_registers.cg_spll_func_cntl_3);
- table->initialState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_4 =
+ table->initialState.level.sclk.vCG_SPLL_FUNC_CNTL_4 =
cpu_to_be32(ni_pi->clock_registers.cg_spll_func_cntl_4);
- table->initialState.levels[0].sclk.vCG_SPLL_SPREAD_SPECTRUM =
+ table->initialState.level.sclk.vCG_SPLL_SPREAD_SPECTRUM =
cpu_to_be32(ni_pi->clock_registers.cg_spll_spread_spectrum);
- table->initialState.levels[0].sclk.vCG_SPLL_SPREAD_SPECTRUM_2 =
+ table->initialState.level.sclk.vCG_SPLL_SPREAD_SPECTRUM_2 =
cpu_to_be32(ni_pi->clock_registers.cg_spll_spread_spectrum_2);
- table->initialState.levels[0].sclk.sclk_value =
+ table->initialState.level.sclk.sclk_value =
cpu_to_be32(initial_state->performance_levels[0].sclk);
- table->initialState.levels[0].arbRefreshState =
+ table->initialState.level.arbRefreshState =
NISLANDS_INITIAL_STATE_ARB_INDEX;
- table->initialState.levels[0].ACIndex = 0;
+ table->initialState.level.ACIndex = 0;
ret = ni_populate_voltage_value(rdev, &eg_pi->vddc_voltage_table,
initial_state->performance_levels[0].vddc,
- &table->initialState.levels[0].vddc);
+ &table->initialState.level.vddc);
if (!ret) {
u16 std_vddc;
ret = ni_get_std_voltage_value(rdev,
- &table->initialState.levels[0].vddc,
+ &table->initialState.level.vddc,
&std_vddc);
if (!ret)
ni_populate_std_voltage_value(rdev, std_vddc,
- table->initialState.levels[0].vddc.index,
- &table->initialState.levels[0].std_vddc);
+ table->initialState.level.vddc.index,
+ &table->initialState.level.std_vddc);
}
if (eg_pi->vddci_control)
ni_populate_voltage_value(rdev,
&eg_pi->vddci_voltage_table,
initial_state->performance_levels[0].vddci,
- &table->initialState.levels[0].vddci);
+ &table->initialState.level.vddci);
- ni_populate_initial_mvdd_value(rdev, &table->initialState.levels[0].mvdd);
+ ni_populate_initial_mvdd_value(rdev, &table->initialState.level.mvdd);
reg = CG_R(0xffff) | CG_L(0);
- table->initialState.levels[0].aT = cpu_to_be32(reg);
+ table->initialState.level.aT = cpu_to_be32(reg);
- table->initialState.levels[0].bSP = cpu_to_be32(pi->dsp);
+ table->initialState.level.bSP = cpu_to_be32(pi->dsp);
if (pi->boot_in_gen2)
- table->initialState.levels[0].gen2PCIE = 1;
+ table->initialState.level.gen2PCIE = 1;
else
- table->initialState.levels[0].gen2PCIE = 0;
+ table->initialState.level.gen2PCIE = 0;
if (pi->mem_gddr5) {
- table->initialState.levels[0].strobeMode =
+ table->initialState.level.strobeMode =
cypress_get_strobe_mode_settings(rdev,
initial_state->performance_levels[0].mclk);
if (initial_state->performance_levels[0].mclk > pi->mclk_edc_enable_threshold)
- table->initialState.levels[0].mcFlags = NISLANDS_SMC_MC_EDC_RD_FLAG | NISLANDS_SMC_MC_EDC_WR_FLAG;
+ table->initialState.level.mcFlags = NISLANDS_SMC_MC_EDC_RD_FLAG | NISLANDS_SMC_MC_EDC_WR_FLAG;
else
- table->initialState.levels[0].mcFlags = 0;
+ table->initialState.level.mcFlags = 0;
}
table->initialState.levelCount = 1;
table->initialState.flags |= PPSMC_SWSTATE_FLAG_DC;
- table->initialState.levels[0].dpm2.MaxPS = 0;
- table->initialState.levels[0].dpm2.NearTDPDec = 0;
- table->initialState.levels[0].dpm2.AboveSafeInc = 0;
- table->initialState.levels[0].dpm2.BelowSafeInc = 0;
+ table->initialState.level.dpm2.MaxPS = 0;
+ table->initialState.level.dpm2.NearTDPDec = 0;
+ table->initialState.level.dpm2.AboveSafeInc = 0;
+ table->initialState.level.dpm2.BelowSafeInc = 0;
reg = MIN_POWER_MASK | MAX_POWER_MASK;
- table->initialState.levels[0].SQPowerThrottle = cpu_to_be32(reg);
+ table->initialState.level.SQPowerThrottle = cpu_to_be32(reg);
reg = MAX_POWER_DELTA_MASK | STI_SIZE_MASK | LTI_RATIO_MASK;
- table->initialState.levels[0].SQPowerThrottle_2 = cpu_to_be32(reg);
+ table->initialState.level.SQPowerThrottle_2 = cpu_to_be32(reg);
return 0;
}
if (pi->acpi_vddc) {
ret = ni_populate_voltage_value(rdev,
&eg_pi->vddc_voltage_table,
- pi->acpi_vddc, &table->ACPIState.levels[0].vddc);
+ pi->acpi_vddc, &table->ACPIState.level.vddc);
if (!ret) {
u16 std_vddc;
ret = ni_get_std_voltage_value(rdev,
- &table->ACPIState.levels[0].vddc, &std_vddc);
+ &table->ACPIState.level.vddc, &std_vddc);
if (!ret)
ni_populate_std_voltage_value(rdev, std_vddc,
- table->ACPIState.levels[0].vddc.index,
- &table->ACPIState.levels[0].std_vddc);
+ table->ACPIState.level.vddc.index,
+ &table->ACPIState.level.std_vddc);
}
if (pi->pcie_gen2) {
if (pi->acpi_pcie_gen2)
- table->ACPIState.levels[0].gen2PCIE = 1;
+ table->ACPIState.level.gen2PCIE = 1;
else
- table->ACPIState.levels[0].gen2PCIE = 0;
+ table->ACPIState.level.gen2PCIE = 0;
} else {
- table->ACPIState.levels[0].gen2PCIE = 0;
+ table->ACPIState.level.gen2PCIE = 0;
}
} else {
ret = ni_populate_voltage_value(rdev,
&eg_pi->vddc_voltage_table,
pi->min_vddc_in_table,
- &table->ACPIState.levels[0].vddc);
+ &table->ACPIState.level.vddc);
if (!ret) {
u16 std_vddc;
ret = ni_get_std_voltage_value(rdev,
- &table->ACPIState.levels[0].vddc,
+ &table->ACPIState.level.vddc,
&std_vddc);
if (!ret)
ni_populate_std_voltage_value(rdev, std_vddc,
- table->ACPIState.levels[0].vddc.index,
- &table->ACPIState.levels[0].std_vddc);
+ table->ACPIState.level.vddc.index,
+ &table->ACPIState.level.std_vddc);
}
- table->ACPIState.levels[0].gen2PCIE = 0;
+ table->ACPIState.level.gen2PCIE = 0;
}
if (eg_pi->acpi_vddci) {
ni_populate_voltage_value(rdev,
&eg_pi->vddci_voltage_table,
eg_pi->acpi_vddci,
- &table->ACPIState.levels[0].vddci);
+ &table->ACPIState.level.vddci);
}
spll_func_cntl_2 &= ~SCLK_MUX_SEL_MASK;
spll_func_cntl_2 |= SCLK_MUX_SEL(4);
- table->ACPIState.levels[0].mclk.vMPLL_AD_FUNC_CNTL = cpu_to_be32(mpll_ad_func_cntl);
- table->ACPIState.levels[0].mclk.vMPLL_AD_FUNC_CNTL_2 = cpu_to_be32(mpll_ad_func_cntl_2);
- table->ACPIState.levels[0].mclk.vMPLL_DQ_FUNC_CNTL = cpu_to_be32(mpll_dq_func_cntl);
- table->ACPIState.levels[0].mclk.vMPLL_DQ_FUNC_CNTL_2 = cpu_to_be32(mpll_dq_func_cntl_2);
- table->ACPIState.levels[0].mclk.vMCLK_PWRMGT_CNTL = cpu_to_be32(mclk_pwrmgt_cntl);
- table->ACPIState.levels[0].mclk.vDLL_CNTL = cpu_to_be32(dll_cntl);
+ table->ACPIState.level.mclk.vMPLL_AD_FUNC_CNTL = cpu_to_be32(mpll_ad_func_cntl);
+ table->ACPIState.level.mclk.vMPLL_AD_FUNC_CNTL_2 = cpu_to_be32(mpll_ad_func_cntl_2);
+ table->ACPIState.level.mclk.vMPLL_DQ_FUNC_CNTL = cpu_to_be32(mpll_dq_func_cntl);
+ table->ACPIState.level.mclk.vMPLL_DQ_FUNC_CNTL_2 = cpu_to_be32(mpll_dq_func_cntl_2);
+ table->ACPIState.level.mclk.vMCLK_PWRMGT_CNTL = cpu_to_be32(mclk_pwrmgt_cntl);
+ table->ACPIState.level.mclk.vDLL_CNTL = cpu_to_be32(dll_cntl);
- table->ACPIState.levels[0].mclk.mclk_value = 0;
+ table->ACPIState.level.mclk.mclk_value = 0;
- table->ACPIState.levels[0].sclk.vCG_SPLL_FUNC_CNTL = cpu_to_be32(spll_func_cntl);
- table->ACPIState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_2 = cpu_to_be32(spll_func_cntl_2);
- table->ACPIState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_3 = cpu_to_be32(spll_func_cntl_3);
- table->ACPIState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_4 = cpu_to_be32(spll_func_cntl_4);
+ table->ACPIState.level.sclk.vCG_SPLL_FUNC_CNTL = cpu_to_be32(spll_func_cntl);
+ table->ACPIState.level.sclk.vCG_SPLL_FUNC_CNTL_2 = cpu_to_be32(spll_func_cntl_2);
+ table->ACPIState.level.sclk.vCG_SPLL_FUNC_CNTL_3 = cpu_to_be32(spll_func_cntl_3);
+ table->ACPIState.level.sclk.vCG_SPLL_FUNC_CNTL_4 = cpu_to_be32(spll_func_cntl_4);
- table->ACPIState.levels[0].sclk.sclk_value = 0;
+ table->ACPIState.level.sclk.sclk_value = 0;
- ni_populate_mvdd_value(rdev, 0, &table->ACPIState.levels[0].mvdd);
+ ni_populate_mvdd_value(rdev, 0, &table->ACPIState.level.mvdd);
if (eg_pi->dynamic_ac_timing)
- table->ACPIState.levels[0].ACIndex = 1;
+ table->ACPIState.level.ACIndex = 1;
- table->ACPIState.levels[0].dpm2.MaxPS = 0;
- table->ACPIState.levels[0].dpm2.NearTDPDec = 0;
- table->ACPIState.levels[0].dpm2.AboveSafeInc = 0;
- table->ACPIState.levels[0].dpm2.BelowSafeInc = 0;
+ table->ACPIState.level.dpm2.MaxPS = 0;
+ table->ACPIState.level.dpm2.NearTDPDec = 0;
+ table->ACPIState.level.dpm2.AboveSafeInc = 0;
+ table->ACPIState.level.dpm2.BelowSafeInc = 0;
reg = MIN_POWER_MASK | MAX_POWER_MASK;
- table->ACPIState.levels[0].SQPowerThrottle = cpu_to_be32(reg);
+ table->ACPIState.level.SQPowerThrottle = cpu_to_be32(reg);
reg = MAX_POWER_DELTA_MASK | STI_SIZE_MASK | LTI_RATIO_MASK;
- table->ACPIState.levels[0].SQPowerThrottle_2 = cpu_to_be32(reg);
+ table->ACPIState.level.SQPowerThrottle_2 = cpu_to_be32(reg);
return 0;
}
if (ret)
return ret;
- table->driverState = table->initialState;
+ table->driverState.flags = table->initialState.flags;
+ table->driverState.levelCount = table->initialState.levelCount;
+ table->driverState.levels[0] = table->initialState.level;
table->ULVState = table->initialState;
typedef struct NISLANDS_SMC_SWSTATE NISLANDS_SMC_SWSTATE;
+struct NISLANDS_SMC_SWSTATE_SINGLE {
+ uint8_t flags;
+ uint8_t levelCount;
+ uint8_t padding2;
+ uint8_t padding3;
+ NISLANDS_SMC_HW_PERFORMANCE_LEVEL level;
+};
+
#define NISLANDS_SMC_VOLTAGEMASK_VDDC 0
#define NISLANDS_SMC_VOLTAGEMASK_MVDD 1
#define NISLANDS_SMC_VOLTAGEMASK_VDDCI 2
struct NISLANDS_SMC_STATETABLE
{
- uint8_t thermalProtectType;
- uint8_t systemFlags;
- uint8_t maxVDDCIndexInPPTable;
- uint8_t extraFlags;
- uint8_t highSMIO[NISLANDS_MAX_NO_VREG_STEPS];
- uint32_t lowSMIO[NISLANDS_MAX_NO_VREG_STEPS];
- NISLANDS_SMC_VOLTAGEMASKTABLE voltageMaskTable;
- PP_NIslands_DPM2Parameters dpm2Params;
- NISLANDS_SMC_SWSTATE initialState;
- NISLANDS_SMC_SWSTATE ACPIState;
- NISLANDS_SMC_SWSTATE ULVState;
- NISLANDS_SMC_SWSTATE driverState;
- NISLANDS_SMC_HW_PERFORMANCE_LEVEL dpmLevels[NISLANDS_MAX_SMC_PERFORMANCE_LEVELS_PER_SWSTATE - 1];
+ uint8_t thermalProtectType;
+ uint8_t systemFlags;
+ uint8_t maxVDDCIndexInPPTable;
+ uint8_t extraFlags;
+ uint8_t highSMIO[NISLANDS_MAX_NO_VREG_STEPS];
+ uint32_t lowSMIO[NISLANDS_MAX_NO_VREG_STEPS];
+ NISLANDS_SMC_VOLTAGEMASKTABLE voltageMaskTable;
+ PP_NIslands_DPM2Parameters dpm2Params;
+ struct NISLANDS_SMC_SWSTATE_SINGLE initialState;
+ struct NISLANDS_SMC_SWSTATE_SINGLE ACPIState;
+ struct NISLANDS_SMC_SWSTATE_SINGLE ULVState;
+ NISLANDS_SMC_SWSTATE driverState;
+ NISLANDS_SMC_HW_PERFORMANCE_LEVEL dpmLevels[NISLANDS_MAX_SMC_PERFORMANCE_LEVELS_PER_SWSTATE];
};
typedef struct NISLANDS_SMC_STATETABLE NISLANDS_SMC_STATETABLE;
void *priv;
u32 new_active_crtcs;
int new_active_crtc_count;
+ int high_pixelclock_count;
u32 current_active_crtcs;
int current_active_crtc_count;
bool single_display;
return state_index;
/* last mode is usually default, array is low to high */
for (i = 0; i < num_modes; i++) {
- rdev->pm.power_state[state_index].clock_info =
- kcalloc(1, sizeof(struct radeon_pm_clock_info),
- GFP_KERNEL);
+ /* avoid memory leaks from invalid modes or unknown frev. */
+ if (!rdev->pm.power_state[state_index].clock_info) {
+ rdev->pm.power_state[state_index].clock_info =
+ kzalloc(sizeof(struct radeon_pm_clock_info),
+ GFP_KERNEL);
+ }
if (!rdev->pm.power_state[state_index].clock_info)
- return state_index;
+ goto out;
rdev->pm.power_state[state_index].num_clock_modes = 1;
rdev->pm.power_state[state_index].clock_info[0].voltage.type = VOLTAGE_NONE;
switch (frev) {
break;
}
}
+out:
+ /* free any unused clock_info allocation. */
+ if (state_index && state_index < num_modes) {
+ kfree(rdev->pm.power_state[state_index].clock_info);
+ rdev->pm.power_state[state_index].clock_info = NULL;
+ }
+
/* last mode is usually default */
- if (rdev->pm.default_power_state_index == -1) {
+ if (state_index && rdev->pm.default_power_state_index == -1) {
rdev->pm.power_state[state_index - 1].type =
POWER_STATE_TYPE_DEFAULT;
rdev->pm.default_power_state_index = state_index - 1;
rdev->pm.power_state[state_index - 1].default_clock_mode =
&rdev->pm.power_state[state_index - 1].clock_info[0];
- rdev->pm.power_state[state_index].flags &=
+ rdev->pm.power_state[state_index - 1].flags &=
~RADEON_PM_STATE_SINGLE_DISPLAY_ONLY;
- rdev->pm.power_state[state_index].misc = 0;
- rdev->pm.power_state[state_index].misc2 = 0;
+ rdev->pm.power_state[state_index - 1].misc = 0;
+ rdev->pm.power_state[state_index - 1].misc2 = 0;
}
return state_index;
}
p = t / (PAGE_SIZE / RADEON_GPU_PAGE_SIZE);
for (i = 0; i < pages; i++, p++) {
- rdev->gart.pages[p] = pagelist[i];
+ rdev->gart.pages[p] = pagelist ? pagelist[i] :
+ rdev->dummy_page.page;
page_base = dma_addr[i];
for (j = 0; j < (PAGE_SIZE / RADEON_GPU_PAGE_SIZE); j++, t++) {
page_entry = radeon_gart_get_page_entry(page_base, flags);
struct drm_device *ddev = rdev->ddev;
struct drm_crtc *crtc;
struct radeon_crtc *radeon_crtc;
+ struct radeon_connector *radeon_connector;
if (!rdev->pm.dpm_enabled)
return;
/* update active crtc counts */
rdev->pm.dpm.new_active_crtcs = 0;
rdev->pm.dpm.new_active_crtc_count = 0;
+ rdev->pm.dpm.high_pixelclock_count = 0;
if (rdev->num_crtc && rdev->mode_info.mode_config_initialized) {
list_for_each_entry(crtc,
&ddev->mode_config.crtc_list, head) {
if (crtc->enabled) {
rdev->pm.dpm.new_active_crtcs |= (1 << radeon_crtc->crtc_id);
rdev->pm.dpm.new_active_crtc_count++;
+ if (!radeon_crtc->connector)
+ continue;
+
+ radeon_connector = to_radeon_connector(radeon_crtc->connector);
+ if (radeon_connector->pixelclock_for_modeset > 297000)
+ rdev->pm.dpm.high_pixelclock_count++;
}
}
}
if (rdev->uvd.vcpu_bo == NULL)
return -EINVAL;
- memcpy(rdev->uvd.cpu_addr, rdev->uvd_fw->data, rdev->uvd_fw->size);
+ memcpy_toio((void __iomem *)rdev->uvd.cpu_addr, rdev->uvd_fw->data, rdev->uvd_fw->size);
size = radeon_bo_size(rdev->uvd.vcpu_bo);
size -= rdev->uvd_fw->size;
ptr = rdev->uvd.cpu_addr;
ptr += rdev->uvd_fw->size;
- memset(ptr, 0, size);
+ memset_io((void __iomem *)ptr, 0, size);
return 0;
}
(rdev->pdev->device == 0x6605)) {
max_sclk = 75000;
}
+
+ if (rdev->pm.dpm.high_pixelclock_count > 1)
+ disable_sclk_switching = true;
}
if (rps->vce_active) {
u32 reg;
int ret;
- table->initialState.levels[0].mclk.vDLL_CNTL =
+ table->initialState.level.mclk.vDLL_CNTL =
cpu_to_be32(si_pi->clock_registers.dll_cntl);
- table->initialState.levels[0].mclk.vMCLK_PWRMGT_CNTL =
+ table->initialState.level.mclk.vMCLK_PWRMGT_CNTL =
cpu_to_be32(si_pi->clock_registers.mclk_pwrmgt_cntl);
- table->initialState.levels[0].mclk.vMPLL_AD_FUNC_CNTL =
+ table->initialState.level.mclk.vMPLL_AD_FUNC_CNTL =
cpu_to_be32(si_pi->clock_registers.mpll_ad_func_cntl);
- table->initialState.levels[0].mclk.vMPLL_DQ_FUNC_CNTL =
+ table->initialState.level.mclk.vMPLL_DQ_FUNC_CNTL =
cpu_to_be32(si_pi->clock_registers.mpll_dq_func_cntl);
- table->initialState.levels[0].mclk.vMPLL_FUNC_CNTL =
+ table->initialState.level.mclk.vMPLL_FUNC_CNTL =
cpu_to_be32(si_pi->clock_registers.mpll_func_cntl);
- table->initialState.levels[0].mclk.vMPLL_FUNC_CNTL_1 =
+ table->initialState.level.mclk.vMPLL_FUNC_CNTL_1 =
cpu_to_be32(si_pi->clock_registers.mpll_func_cntl_1);
- table->initialState.levels[0].mclk.vMPLL_FUNC_CNTL_2 =
+ table->initialState.level.mclk.vMPLL_FUNC_CNTL_2 =
cpu_to_be32(si_pi->clock_registers.mpll_func_cntl_2);
- table->initialState.levels[0].mclk.vMPLL_SS =
+ table->initialState.level.mclk.vMPLL_SS =
cpu_to_be32(si_pi->clock_registers.mpll_ss1);
- table->initialState.levels[0].mclk.vMPLL_SS2 =
+ table->initialState.level.mclk.vMPLL_SS2 =
cpu_to_be32(si_pi->clock_registers.mpll_ss2);
- table->initialState.levels[0].mclk.mclk_value =
+ table->initialState.level.mclk.mclk_value =
cpu_to_be32(initial_state->performance_levels[0].mclk);
- table->initialState.levels[0].sclk.vCG_SPLL_FUNC_CNTL =
+ table->initialState.level.sclk.vCG_SPLL_FUNC_CNTL =
cpu_to_be32(si_pi->clock_registers.cg_spll_func_cntl);
- table->initialState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_2 =
+ table->initialState.level.sclk.vCG_SPLL_FUNC_CNTL_2 =
cpu_to_be32(si_pi->clock_registers.cg_spll_func_cntl_2);
- table->initialState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_3 =
+ table->initialState.level.sclk.vCG_SPLL_FUNC_CNTL_3 =
cpu_to_be32(si_pi->clock_registers.cg_spll_func_cntl_3);
- table->initialState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_4 =
+ table->initialState.level.sclk.vCG_SPLL_FUNC_CNTL_4 =
cpu_to_be32(si_pi->clock_registers.cg_spll_func_cntl_4);
- table->initialState.levels[0].sclk.vCG_SPLL_SPREAD_SPECTRUM =
+ table->initialState.level.sclk.vCG_SPLL_SPREAD_SPECTRUM =
cpu_to_be32(si_pi->clock_registers.cg_spll_spread_spectrum);
- table->initialState.levels[0].sclk.vCG_SPLL_SPREAD_SPECTRUM_2 =
+ table->initialState.level.sclk.vCG_SPLL_SPREAD_SPECTRUM_2 =
cpu_to_be32(si_pi->clock_registers.cg_spll_spread_spectrum_2);
- table->initialState.levels[0].sclk.sclk_value =
+ table->initialState.level.sclk.sclk_value =
cpu_to_be32(initial_state->performance_levels[0].sclk);
- table->initialState.levels[0].arbRefreshState =
+ table->initialState.level.arbRefreshState =
SISLANDS_INITIAL_STATE_ARB_INDEX;
- table->initialState.levels[0].ACIndex = 0;
+ table->initialState.level.ACIndex = 0;
ret = si_populate_voltage_value(rdev, &eg_pi->vddc_voltage_table,
initial_state->performance_levels[0].vddc,
- &table->initialState.levels[0].vddc);
+ &table->initialState.level.vddc);
if (!ret) {
u16 std_vddc;
ret = si_get_std_voltage_value(rdev,
- &table->initialState.levels[0].vddc,
+ &table->initialState.level.vddc,
&std_vddc);
if (!ret)
si_populate_std_voltage_value(rdev, std_vddc,
- table->initialState.levels[0].vddc.index,
- &table->initialState.levels[0].std_vddc);
+ table->initialState.level.vddc.index,
+ &table->initialState.level.std_vddc);
}
if (eg_pi->vddci_control)
si_populate_voltage_value(rdev,
&eg_pi->vddci_voltage_table,
initial_state->performance_levels[0].vddci,
- &table->initialState.levels[0].vddci);
+ &table->initialState.level.vddci);
if (si_pi->vddc_phase_shed_control)
si_populate_phase_shedding_value(rdev,
initial_state->performance_levels[0].vddc,
initial_state->performance_levels[0].sclk,
initial_state->performance_levels[0].mclk,
- &table->initialState.levels[0].vddc);
+ &table->initialState.level.vddc);
- si_populate_initial_mvdd_value(rdev, &table->initialState.levels[0].mvdd);
+ si_populate_initial_mvdd_value(rdev, &table->initialState.level.mvdd);
reg = CG_R(0xffff) | CG_L(0);
- table->initialState.levels[0].aT = cpu_to_be32(reg);
+ table->initialState.level.aT = cpu_to_be32(reg);
- table->initialState.levels[0].bSP = cpu_to_be32(pi->dsp);
+ table->initialState.level.bSP = cpu_to_be32(pi->dsp);
- table->initialState.levels[0].gen2PCIE = (u8)si_pi->boot_pcie_gen;
+ table->initialState.level.gen2PCIE = (u8)si_pi->boot_pcie_gen;
if (pi->mem_gddr5) {
- table->initialState.levels[0].strobeMode =
+ table->initialState.level.strobeMode =
si_get_strobe_mode_settings(rdev,
initial_state->performance_levels[0].mclk);
if (initial_state->performance_levels[0].mclk > pi->mclk_edc_enable_threshold)
- table->initialState.levels[0].mcFlags = SISLANDS_SMC_MC_EDC_RD_FLAG | SISLANDS_SMC_MC_EDC_WR_FLAG;
+ table->initialState.level.mcFlags = SISLANDS_SMC_MC_EDC_RD_FLAG | SISLANDS_SMC_MC_EDC_WR_FLAG;
else
- table->initialState.levels[0].mcFlags = 0;
+ table->initialState.level.mcFlags = 0;
}
table->initialState.levelCount = 1;
table->initialState.flags |= PPSMC_SWSTATE_FLAG_DC;
- table->initialState.levels[0].dpm2.MaxPS = 0;
- table->initialState.levels[0].dpm2.NearTDPDec = 0;
- table->initialState.levels[0].dpm2.AboveSafeInc = 0;
- table->initialState.levels[0].dpm2.BelowSafeInc = 0;
- table->initialState.levels[0].dpm2.PwrEfficiencyRatio = 0;
+ table->initialState.level.dpm2.MaxPS = 0;
+ table->initialState.level.dpm2.NearTDPDec = 0;
+ table->initialState.level.dpm2.AboveSafeInc = 0;
+ table->initialState.level.dpm2.BelowSafeInc = 0;
+ table->initialState.level.dpm2.PwrEfficiencyRatio = 0;
reg = MIN_POWER_MASK | MAX_POWER_MASK;
- table->initialState.levels[0].SQPowerThrottle = cpu_to_be32(reg);
+ table->initialState.level.SQPowerThrottle = cpu_to_be32(reg);
reg = MAX_POWER_DELTA_MASK | STI_SIZE_MASK | LTI_RATIO_MASK;
- table->initialState.levels[0].SQPowerThrottle_2 = cpu_to_be32(reg);
+ table->initialState.level.SQPowerThrottle_2 = cpu_to_be32(reg);
return 0;
}
if (pi->acpi_vddc) {
ret = si_populate_voltage_value(rdev, &eg_pi->vddc_voltage_table,
- pi->acpi_vddc, &table->ACPIState.levels[0].vddc);
+ pi->acpi_vddc, &table->ACPIState.level.vddc);
if (!ret) {
u16 std_vddc;
ret = si_get_std_voltage_value(rdev,
- &table->ACPIState.levels[0].vddc, &std_vddc);
+ &table->ACPIState.level.vddc, &std_vddc);
if (!ret)
si_populate_std_voltage_value(rdev, std_vddc,
- table->ACPIState.levels[0].vddc.index,
- &table->ACPIState.levels[0].std_vddc);
+ table->ACPIState.level.vddc.index,
+ &table->ACPIState.level.std_vddc);
}
- table->ACPIState.levels[0].gen2PCIE = si_pi->acpi_pcie_gen;
+ table->ACPIState.level.gen2PCIE = si_pi->acpi_pcie_gen;
if (si_pi->vddc_phase_shed_control) {
si_populate_phase_shedding_value(rdev,
pi->acpi_vddc,
0,
0,
- &table->ACPIState.levels[0].vddc);
+ &table->ACPIState.level.vddc);
}
} else {
ret = si_populate_voltage_value(rdev, &eg_pi->vddc_voltage_table,
- pi->min_vddc_in_table, &table->ACPIState.levels[0].vddc);
+ pi->min_vddc_in_table, &table->ACPIState.level.vddc);
if (!ret) {
u16 std_vddc;
ret = si_get_std_voltage_value(rdev,
- &table->ACPIState.levels[0].vddc, &std_vddc);
+ &table->ACPIState.level.vddc, &std_vddc);
if (!ret)
si_populate_std_voltage_value(rdev, std_vddc,
- table->ACPIState.levels[0].vddc.index,
- &table->ACPIState.levels[0].std_vddc);
+ table->ACPIState.level.vddc.index,
+ &table->ACPIState.level.std_vddc);
}
- table->ACPIState.levels[0].gen2PCIE = (u8)r600_get_pcie_gen_support(rdev,
+ table->ACPIState.level.gen2PCIE = (u8)r600_get_pcie_gen_support(rdev,
si_pi->sys_pcie_mask,
si_pi->boot_pcie_gen,
RADEON_PCIE_GEN1);
pi->min_vddc_in_table,
0,
0,
- &table->ACPIState.levels[0].vddc);
+ &table->ACPIState.level.vddc);
}
if (pi->acpi_vddc) {
if (eg_pi->acpi_vddci)
si_populate_voltage_value(rdev, &eg_pi->vddci_voltage_table,
eg_pi->acpi_vddci,
- &table->ACPIState.levels[0].vddci);
+ &table->ACPIState.level.vddci);
}
mclk_pwrmgt_cntl |= MRDCK0_RESET | MRDCK1_RESET;
spll_func_cntl_2 &= ~SCLK_MUX_SEL_MASK;
spll_func_cntl_2 |= SCLK_MUX_SEL(4);
- table->ACPIState.levels[0].mclk.vDLL_CNTL =
+ table->ACPIState.level.mclk.vDLL_CNTL =
cpu_to_be32(dll_cntl);
- table->ACPIState.levels[0].mclk.vMCLK_PWRMGT_CNTL =
+ table->ACPIState.level.mclk.vMCLK_PWRMGT_CNTL =
cpu_to_be32(mclk_pwrmgt_cntl);
- table->ACPIState.levels[0].mclk.vMPLL_AD_FUNC_CNTL =
+ table->ACPIState.level.mclk.vMPLL_AD_FUNC_CNTL =
cpu_to_be32(mpll_ad_func_cntl);
- table->ACPIState.levels[0].mclk.vMPLL_DQ_FUNC_CNTL =
+ table->ACPIState.level.mclk.vMPLL_DQ_FUNC_CNTL =
cpu_to_be32(mpll_dq_func_cntl);
- table->ACPIState.levels[0].mclk.vMPLL_FUNC_CNTL =
+ table->ACPIState.level.mclk.vMPLL_FUNC_CNTL =
cpu_to_be32(mpll_func_cntl);
- table->ACPIState.levels[0].mclk.vMPLL_FUNC_CNTL_1 =
+ table->ACPIState.level.mclk.vMPLL_FUNC_CNTL_1 =
cpu_to_be32(mpll_func_cntl_1);
- table->ACPIState.levels[0].mclk.vMPLL_FUNC_CNTL_2 =
+ table->ACPIState.level.mclk.vMPLL_FUNC_CNTL_2 =
cpu_to_be32(mpll_func_cntl_2);
- table->ACPIState.levels[0].mclk.vMPLL_SS =
+ table->ACPIState.level.mclk.vMPLL_SS =
cpu_to_be32(si_pi->clock_registers.mpll_ss1);
- table->ACPIState.levels[0].mclk.vMPLL_SS2 =
+ table->ACPIState.level.mclk.vMPLL_SS2 =
cpu_to_be32(si_pi->clock_registers.mpll_ss2);
- table->ACPIState.levels[0].sclk.vCG_SPLL_FUNC_CNTL =
+ table->ACPIState.level.sclk.vCG_SPLL_FUNC_CNTL =
cpu_to_be32(spll_func_cntl);
- table->ACPIState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_2 =
+ table->ACPIState.level.sclk.vCG_SPLL_FUNC_CNTL_2 =
cpu_to_be32(spll_func_cntl_2);
- table->ACPIState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_3 =
+ table->ACPIState.level.sclk.vCG_SPLL_FUNC_CNTL_3 =
cpu_to_be32(spll_func_cntl_3);
- table->ACPIState.levels[0].sclk.vCG_SPLL_FUNC_CNTL_4 =
+ table->ACPIState.level.sclk.vCG_SPLL_FUNC_CNTL_4 =
cpu_to_be32(spll_func_cntl_4);
- table->ACPIState.levels[0].mclk.mclk_value = 0;
- table->ACPIState.levels[0].sclk.sclk_value = 0;
+ table->ACPIState.level.mclk.mclk_value = 0;
+ table->ACPIState.level.sclk.sclk_value = 0;
- si_populate_mvdd_value(rdev, 0, &table->ACPIState.levels[0].mvdd);
+ si_populate_mvdd_value(rdev, 0, &table->ACPIState.level.mvdd);
if (eg_pi->dynamic_ac_timing)
- table->ACPIState.levels[0].ACIndex = 0;
+ table->ACPIState.level.ACIndex = 0;
- table->ACPIState.levels[0].dpm2.MaxPS = 0;
- table->ACPIState.levels[0].dpm2.NearTDPDec = 0;
- table->ACPIState.levels[0].dpm2.AboveSafeInc = 0;
- table->ACPIState.levels[0].dpm2.BelowSafeInc = 0;
- table->ACPIState.levels[0].dpm2.PwrEfficiencyRatio = 0;
+ table->ACPIState.level.dpm2.MaxPS = 0;
+ table->ACPIState.level.dpm2.NearTDPDec = 0;
+ table->ACPIState.level.dpm2.AboveSafeInc = 0;
+ table->ACPIState.level.dpm2.BelowSafeInc = 0;
+ table->ACPIState.level.dpm2.PwrEfficiencyRatio = 0;
reg = MIN_POWER_MASK | MAX_POWER_MASK;
- table->ACPIState.levels[0].SQPowerThrottle = cpu_to_be32(reg);
+ table->ACPIState.level.SQPowerThrottle = cpu_to_be32(reg);
reg = MAX_POWER_DELTA_MASK | STI_SIZE_MASK | LTI_RATIO_MASK;
- table->ACPIState.levels[0].SQPowerThrottle_2 = cpu_to_be32(reg);
+ table->ACPIState.level.SQPowerThrottle_2 = cpu_to_be32(reg);
return 0;
}
static int si_populate_ulv_state(struct radeon_device *rdev,
- SISLANDS_SMC_SWSTATE *state)
+ struct SISLANDS_SMC_SWSTATE_SINGLE *state)
{
struct evergreen_power_info *eg_pi = evergreen_get_pi(rdev);
struct si_power_info *si_pi = si_get_pi(rdev);
int ret;
ret = si_convert_power_level_to_smc(rdev, &ulv->pl,
- &state->levels[0]);
+ &state->level);
if (!ret) {
if (eg_pi->sclk_deep_sleep) {
if (sclk_in_sr <= SCLK_MIN_DEEPSLEEP_FREQ)
- state->levels[0].stateFlags |= PPSMC_STATEFLAG_DEEPSLEEP_BYPASS;
+ state->level.stateFlags |= PPSMC_STATEFLAG_DEEPSLEEP_BYPASS;
else
- state->levels[0].stateFlags |= PPSMC_STATEFLAG_DEEPSLEEP_THROTTLE;
+ state->level.stateFlags |= PPSMC_STATEFLAG_DEEPSLEEP_THROTTLE;
}
if (ulv->one_pcie_lane_in_ulv)
state->flags |= PPSMC_SWSTATE_FLAG_PCIE_X1;
- state->levels[0].arbRefreshState = (u8)(SISLANDS_ULV_STATE_ARB_INDEX);
- state->levels[0].ACIndex = 1;
- state->levels[0].std_vddc = state->levels[0].vddc;
+ state->level.arbRefreshState = (u8)(SISLANDS_ULV_STATE_ARB_INDEX);
+ state->level.ACIndex = 1;
+ state->level.std_vddc = state->level.vddc;
state->levelCount = 1;
state->flags |= PPSMC_SWSTATE_FLAG_DC;
if (ret)
return ret;
- table->driverState = table->initialState;
+ table->driverState.flags = table->initialState.flags;
+ table->driverState.levelCount = table->initialState.levelCount;
+ table->driverState.levels[0] = table->initialState.level;
ret = si_do_program_memory_timing_parameters(rdev, radeon_boot_state,
SISLANDS_INITIAL_STATE_ARB_INDEX);
if (ulv->supported && ulv->pl.vddc) {
u32 address = si_pi->state_table_start +
offsetof(SISLANDS_SMC_STATETABLE, ULVState);
- SISLANDS_SMC_SWSTATE *smc_state = &si_pi->smc_statetable.ULVState;
- u32 state_size = sizeof(SISLANDS_SMC_SWSTATE);
+ struct SISLANDS_SMC_SWSTATE_SINGLE *smc_state = &si_pi->smc_statetable.ULVState;
+ u32 state_size = sizeof(struct SISLANDS_SMC_SWSTATE_SINGLE);
memset(smc_state, 0, state_size);
typedef struct SISLANDS_SMC_SWSTATE SISLANDS_SMC_SWSTATE;
+struct SISLANDS_SMC_SWSTATE_SINGLE {
+ uint8_t flags;
+ uint8_t levelCount;
+ uint8_t padding2;
+ uint8_t padding3;
+ SISLANDS_SMC_HW_PERFORMANCE_LEVEL level;
+};
+
#define SISLANDS_SMC_VOLTAGEMASK_VDDC 0
#define SISLANDS_SMC_VOLTAGEMASK_MVDD 1
#define SISLANDS_SMC_VOLTAGEMASK_VDDCI 2
struct SISLANDS_SMC_STATETABLE
{
- uint8_t thermalProtectType;
- uint8_t systemFlags;
- uint8_t maxVDDCIndexInPPTable;
- uint8_t extraFlags;
- uint32_t lowSMIO[SISLANDS_MAX_NO_VREG_STEPS];
- SISLANDS_SMC_VOLTAGEMASKTABLE voltageMaskTable;
- SISLANDS_SMC_VOLTAGEMASKTABLE phaseMaskTable;
- PP_SIslands_DPM2Parameters dpm2Params;
- SISLANDS_SMC_SWSTATE initialState;
- SISLANDS_SMC_SWSTATE ACPIState;
- SISLANDS_SMC_SWSTATE ULVState;
- SISLANDS_SMC_SWSTATE driverState;
- SISLANDS_SMC_HW_PERFORMANCE_LEVEL dpmLevels[SISLANDS_MAX_SMC_PERFORMANCE_LEVELS_PER_SWSTATE - 1];
+ uint8_t thermalProtectType;
+ uint8_t systemFlags;
+ uint8_t maxVDDCIndexInPPTable;
+ uint8_t extraFlags;
+ uint32_t lowSMIO[SISLANDS_MAX_NO_VREG_STEPS];
+ SISLANDS_SMC_VOLTAGEMASKTABLE voltageMaskTable;
+ SISLANDS_SMC_VOLTAGEMASKTABLE phaseMaskTable;
+ PP_SIslands_DPM2Parameters dpm2Params;
+ struct SISLANDS_SMC_SWSTATE_SINGLE initialState;
+ struct SISLANDS_SMC_SWSTATE_SINGLE ACPIState;
+ struct SISLANDS_SMC_SWSTATE_SINGLE ULVState;
+ SISLANDS_SMC_SWSTATE driverState;
+ SISLANDS_SMC_HW_PERFORMANCE_LEVEL dpmLevels[SISLANDS_MAX_SMC_PERFORMANCE_LEVELS_PER_SWSTATE];
};
typedef struct SISLANDS_SMC_STATETABLE SISLANDS_SMC_STATETABLE;
goto err_disable_clk_tmds;
}
- ret = sun8i_hdmi_phy_probe(hdmi, phy_node);
+ ret = sun8i_hdmi_phy_get(hdmi, phy_node);
of_node_put(phy_node);
if (ret) {
dev_err(dev, "Couldn't get the HDMI PHY\n");
cleanup_encoder:
drm_encoder_cleanup(encoder);
- sun8i_hdmi_phy_remove(hdmi);
err_disable_clk_tmds:
clk_disable_unprepare(hdmi->clk_tmds);
err_assert_ctrl_reset:
struct sun8i_dw_hdmi *hdmi = dev_get_drvdata(dev);
dw_hdmi_unbind(hdmi->hdmi);
- sun8i_hdmi_phy_remove(hdmi);
clk_disable_unprepare(hdmi->clk_tmds);
reset_control_assert(hdmi->rst_ctrl);
gpiod_set_value(hdmi->ddc_en, 0);
.of_match_table = sun8i_dw_hdmi_dt_ids,
},
};
-module_platform_driver(sun8i_dw_hdmi_pltfm_driver);
+
+static int __init sun8i_dw_hdmi_init(void)
+{
+ int ret;
+
+ ret = platform_driver_register(&sun8i_dw_hdmi_pltfm_driver);
+ if (ret)
+ return ret;
+
+ ret = platform_driver_register(&sun8i_hdmi_phy_driver);
+ if (ret) {
+ platform_driver_unregister(&sun8i_dw_hdmi_pltfm_driver);
+ return ret;
+ }
+
+ return ret;
+}
+
+static void __exit sun8i_dw_hdmi_exit(void)
+{
+ platform_driver_unregister(&sun8i_dw_hdmi_pltfm_driver);
+ platform_driver_unregister(&sun8i_hdmi_phy_driver);
+}
+
+module_init(sun8i_dw_hdmi_init);
+module_exit(sun8i_dw_hdmi_exit);
MODULE_AUTHOR("Jernej Skrabec <jernej.skrabec@siol.net>");
MODULE_DESCRIPTION("Allwinner DW HDMI bridge");
struct gpio_desc *ddc_en;
};
+extern struct platform_driver sun8i_hdmi_phy_driver;
+
static inline struct sun8i_dw_hdmi *
encoder_to_sun8i_dw_hdmi(struct drm_encoder *encoder)
{
return container_of(encoder, struct sun8i_dw_hdmi, encoder);
}
-int sun8i_hdmi_phy_probe(struct sun8i_dw_hdmi *hdmi, struct device_node *node);
-void sun8i_hdmi_phy_remove(struct sun8i_dw_hdmi *hdmi);
+int sun8i_hdmi_phy_get(struct sun8i_dw_hdmi *hdmi, struct device_node *node);
void sun8i_hdmi_phy_init(struct sun8i_hdmi_phy *phy);
void sun8i_hdmi_phy_set_ops(struct sun8i_hdmi_phy *phy,
#include <linux/delay.h>
#include <linux/of_address.h>
+#include <linux/of_platform.h>
#include "sun8i_dw_hdmi.h"
{ /* sentinel */ }
};
-int sun8i_hdmi_phy_probe(struct sun8i_dw_hdmi *hdmi, struct device_node *node)
+int sun8i_hdmi_phy_get(struct sun8i_dw_hdmi *hdmi, struct device_node *node)
+{
+ struct platform_device *pdev = of_find_device_by_node(node);
+ struct sun8i_hdmi_phy *phy;
+
+ if (!pdev)
+ return -EPROBE_DEFER;
+
+ phy = platform_get_drvdata(pdev);
+ if (!phy)
+ return -EPROBE_DEFER;
+
+ hdmi->phy = phy;
+
+ put_device(&pdev->dev);
+
+ return 0;
+}
+
+static int sun8i_hdmi_phy_probe(struct platform_device *pdev)
{
const struct of_device_id *match;
- struct device *dev = hdmi->dev;
+ struct device *dev = &pdev->dev;
+ struct device_node *node = dev->of_node;
struct sun8i_hdmi_phy *phy;
struct resource res;
void __iomem *regs;
clk_prepare_enable(phy->clk_phy);
}
- hdmi->phy = phy;
+ platform_set_drvdata(pdev, phy);
return 0;
return ret;
}
-void sun8i_hdmi_phy_remove(struct sun8i_dw_hdmi *hdmi)
+static int sun8i_hdmi_phy_remove(struct platform_device *pdev)
{
- struct sun8i_hdmi_phy *phy = hdmi->phy;
+ struct sun8i_hdmi_phy *phy = platform_get_drvdata(pdev);
clk_disable_unprepare(phy->clk_mod);
clk_disable_unprepare(phy->clk_bus);
clk_put(phy->clk_pll1);
clk_put(phy->clk_mod);
clk_put(phy->clk_bus);
+ return 0;
}
+
+struct platform_driver sun8i_hdmi_phy_driver = {
+ .probe = sun8i_hdmi_phy_probe,
+ .remove = sun8i_hdmi_phy_remove,
+ .driver = {
+ .name = "sun8i-hdmi-phy",
+ .of_match_table = sun8i_hdmi_phy_of_table,
+ },
+};
#include "trace.h"
/* XXX move to include/uapi/drm/drm_fourcc.h? */
-#define DRM_FORMAT_MOD_NVIDIA_SECTOR_LAYOUT BIT(22)
+#define DRM_FORMAT_MOD_NVIDIA_SECTOR_LAYOUT BIT_ULL(22)
struct reset_control;
* dGPU sector layout.
*/
if (tegra_plane_state->tiling.sector_layout == TEGRA_BO_SECTOR_LAYOUT_GPU)
- base |= BIT(39);
+ base |= BIT_ULL(39);
#endif
tegra_plane_writel(p, tegra_plane_state->format, DC_WIN_COLOR_DEPTH);
if (err < 0) {
dev_err(sor->dev, "failed to acquire SOR reset: %d\n",
err);
- return err;
+ goto rpm_put;
}
err = reset_control_assert(sor->rst);
if (err < 0) {
dev_err(sor->dev, "failed to assert SOR reset: %d\n",
err);
- return err;
+ goto rpm_put;
}
}
err = clk_prepare_enable(sor->clk);
if (err < 0) {
dev_err(sor->dev, "failed to enable clock: %d\n", err);
- return err;
+ goto rpm_put;
}
usleep_range(1000, 3000);
dev_err(sor->dev, "failed to deassert SOR reset: %d\n",
err);
clk_disable_unprepare(sor->clk);
- return err;
+ goto rpm_put;
}
reset_control_release(sor->rst);
}
return 0;
+
+rpm_put:
+ if (sor->rst)
+ pm_runtime_put(sor->dev);
+
+ return err;
}
static int tegra_sor_exit(struct host1x_client *client)
if (!sor->aux)
return -EPROBE_DEFER;
- if (get_device(&sor->aux->ddc.dev)) {
- if (try_module_get(sor->aux->ddc.owner))
- sor->output.ddc = &sor->aux->ddc;
- else
- put_device(&sor->aux->ddc.dev);
- }
+ if (get_device(sor->aux->dev))
+ sor->output.ddc = &sor->aux->ddc;
}
if (!sor->aux) {
err = tegra_sor_parse_dt(sor);
if (err < 0)
- return err;
+ goto put_aux;
err = tegra_output_probe(&sor->output);
- if (err < 0)
- return dev_err_probe(&pdev->dev, err,
- "failed to probe output\n");
+ if (err < 0) {
+ dev_err_probe(&pdev->dev, err, "failed to probe output\n");
+ goto put_aux;
+ }
if (sor->ops && sor->ops->probe) {
err = sor->ops->probe(sor);
platform_set_drvdata(pdev, sor);
pm_runtime_enable(&pdev->dev);
- INIT_LIST_HEAD(&sor->client.list);
+ host1x_client_init(&sor->client);
sor->client.ops = &sor_client_ops;
sor->client.dev = &pdev->dev;
- err = host1x_client_register(&sor->client);
- if (err < 0) {
- dev_err(&pdev->dev, "failed to register host1x client: %d\n",
- err);
- goto rpm_disable;
- }
-
/*
* On Tegra210 and earlier, provide our own implementation for the
* pad output clock.
sor->index);
if (!name) {
err = -ENOMEM;
- goto unregister;
+ goto uninit;
}
err = host1x_client_resume(&sor->client);
if (err < 0) {
dev_err(sor->dev, "failed to resume: %d\n", err);
- goto unregister;
+ goto uninit;
}
sor->clk_pad = tegra_clk_sor_pad_register(sor, name);
err = PTR_ERR(sor->clk_pad);
dev_err(sor->dev, "failed to register SOR pad clock: %d\n",
err);
- goto unregister;
+ goto uninit;
+ }
+
+ err = __host1x_client_register(&sor->client);
+ if (err < 0) {
+ dev_err(&pdev->dev, "failed to register host1x client: %d\n",
+ err);
+ goto uninit;
}
return 0;
-unregister:
- host1x_client_unregister(&sor->client);
-rpm_disable:
+uninit:
+ host1x_client_exit(&sor->client);
pm_runtime_disable(&pdev->dev);
remove:
+ if (sor->aux)
+ sor->output.ddc = NULL;
+
tegra_output_remove(&sor->output);
+put_aux:
+ if (sor->aux)
+ put_device(sor->aux->dev);
+
return err;
}
pm_runtime_disable(&pdev->dev);
+ if (sor->aux) {
+ put_device(sor->aux->dev);
+ sor->output.ddc = NULL;
+ }
+
tegra_output_remove(&sor->output);
return 0;
if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, NULL))
return -EBUSY;
- if (!ttm_bo_get_unless_zero(bo)) {
+ if (!bo->ttm || !ttm_tt_is_populated(bo->ttm) ||
+ bo->ttm->page_flags & TTM_PAGE_FLAG_SG ||
+ bo->ttm->page_flags & TTM_PAGE_FLAG_SWAPPED ||
+ !ttm_bo_get_unless_zero(bo)) {
if (locked)
dma_resv_unlock(bo->base.resv);
return -EBUSY;
for (j = 0; j < TTM_MAX_BO_PRIORITY; ++j) {
list_for_each_entry(bo, &man->lru[j], lru) {
- uint32_t num_pages;
+ uint32_t num_pages = PFN_UP(bo->base.size);
- if (!bo->ttm ||
- bo->ttm->page_flags & TTM_PAGE_FLAG_SG ||
- bo->ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)
- continue;
-
- num_pages = bo->ttm->num_pages;
ret = ttm_bo_swapout(bo, ctx, gfp_flags);
/* ttm_bo_swapout has dropped the lru_lock */
if (!ret)
if (!old_hvs_state->fifo_state[channel].in_use)
continue;
- ret = drm_crtc_commit_wait(old_hvs_state->fifo_state[i].pending_commit);
+ ret = drm_crtc_commit_wait(old_hvs_state->fifo_state[channel].pending_commit);
if (ret)
drm_err(dev, "Timed out waiting for commit\n");
}
struct drm_encoder *encoder;
};
-static inline struct vc4_vec_connector *
-to_vc4_vec_connector(struct drm_connector *connector)
-{
- return container_of(connector, struct vc4_vec_connector, base);
-}
-
enum vc4_vec_tv_mode_id {
VC4_VEC_TV_MODE_NTSC,
VC4_VEC_TV_MODE_NTSC_J,
EXPORT_SYMBOL(host1x_driver_unregister);
/**
+ * __host1x_client_init() - initialize a host1x client
+ * @client: host1x client
+ * @key: lock class key for the client-specific mutex
+ */
+void __host1x_client_init(struct host1x_client *client, struct lock_class_key *key)
+{
+ INIT_LIST_HEAD(&client->list);
+ __mutex_init(&client->lock, "host1x client lock", key);
+ client->usecount = 0;
+}
+EXPORT_SYMBOL(__host1x_client_init);
+
+/**
+ * host1x_client_exit() - uninitialize a host1x client
+ * @client: host1x client
+ */
+void host1x_client_exit(struct host1x_client *client)
+{
+ mutex_destroy(&client->lock);
+}
+EXPORT_SYMBOL(host1x_client_exit);
+
+/**
* __host1x_client_register() - register a host1x client
* @client: host1x client
* @key: lock class key for the client-specific mutex
* device and call host1x_device_init(), which will in turn call each client's
* &host1x_client_ops.init implementation.
*/
-int __host1x_client_register(struct host1x_client *client,
- struct lock_class_key *key)
+int __host1x_client_register(struct host1x_client *client)
{
struct host1x *host1x;
int err;
- INIT_LIST_HEAD(&client->list);
- __mutex_init(&client->lock, "host1x client lock", key);
- client->usecount = 0;
-
mutex_lock(&devices_lock);
list_for_each_entry(host1x, &devices, list) {
depends on HID
config HID_A4TECH
- tristate "A4 tech mice"
+ tristate "A4TECH mice"
depends on HID
default !EXPERT
help
- Support for A4 tech X5 and WOP-35 / Trust 450L mice.
+ Support for some A4TECH mice with two scroll wheels.
config HID_ACCUTOUCH
tristate "Accutouch touch device"
help
Support for Samsung InfraRed remote control or keyboards.
+config HID_SEMITEK
+ tristate "Semitek USB keyboards"
+ depends on HID
+ help
+ Support for Semitek USB keyboards that are not fully compliant
+ with the HID standard.
+
+ There are many variants, including:
+ - GK61, GK64, GK68, GK84, GK96, etc.
+ - SK61, SK64, SK68, SK84, SK96, etc.
+ - Dierya DK61/DK66
+ - Tronsmart TK09R
+ - Woo-dy
+ - X-Bows Nature/Knight
+
config HID_SONY
tristate "Sony PS2/3/4 accessories"
depends on USB_HID
obj-$(CONFIG_HID_RMI) += hid-rmi.o
obj-$(CONFIG_HID_SAITEK) += hid-saitek.o
obj-$(CONFIG_HID_SAMSUNG) += hid-samsung.o
+obj-$(CONFIG_HID_SEMITEK) += hid-semitek.o
obj-$(CONFIG_HID_SMARTJOYPLUS) += hid-sjoy.o
obj-$(CONFIG_HID_SONY) += hid-sony.o
obj-$(CONFIG_HID_SPEEDLINK) += hid-speedlink.o
sensor_index = req_node->sensor_idx;
report_id = req_node->report_id;
node_type = req_node->report_type;
+ kfree(req_node);
if (node_type == HID_FEATURE_REPORT) {
report_size = get_feature_report(sensor_index, report_id,
int rc, i;
dev = &privdata->pdev->dev;
- cl_data = kzalloc(sizeof(*cl_data), GFP_KERNEL);
+ cl_data = devm_kzalloc(dev, sizeof(*cl_data), GFP_KERNEL);
if (!cl_data)
return -ENOMEM;
rc = -EINVAL;
goto cleanup;
}
- cl_data->feature_report[i] = kzalloc(feature_report_size, GFP_KERNEL);
+ cl_data->feature_report[i] = devm_kzalloc(dev, feature_report_size, GFP_KERNEL);
if (!cl_data->feature_report[i]) {
rc = -ENOMEM;
goto cleanup;
}
- cl_data->input_report[i] = kzalloc(input_report_size, GFP_KERNEL);
+ cl_data->input_report[i] = devm_kzalloc(dev, input_report_size, GFP_KERNEL);
if (!cl_data->input_report[i]) {
rc = -ENOMEM;
goto cleanup;
info.sensor_idx = cl_idx;
info.dma_address = cl_data->sensor_dma_addr[i];
- cl_data->report_descr[i] = kzalloc(cl_data->report_descr_sz[i], GFP_KERNEL);
+ cl_data->report_descr[i] =
+ devm_kzalloc(dev, cl_data->report_descr_sz[i], GFP_KERNEL);
if (!cl_data->report_descr[i]) {
rc = -ENOMEM;
goto cleanup;
cl_data->sensor_virt_addr[i],
cl_data->sensor_dma_addr[i]);
}
- kfree(cl_data->feature_report[i]);
- kfree(cl_data->input_report[i]);
- kfree(cl_data->report_descr[i]);
+ devm_kfree(dev, cl_data->feature_report[i]);
+ devm_kfree(dev, cl_data->input_report[i]);
+ devm_kfree(dev, cl_data->report_descr[i]);
}
- kfree(cl_data);
+ devm_kfree(dev, cl_data);
return rc;
}
cl_data->sensor_dma_addr[i]);
}
}
- kfree(cl_data);
return 0;
}
int i;
for (i = 0; i < cli_data->num_hid_devices; ++i) {
- kfree(cli_data->feature_report[i]);
- kfree(cli_data->input_report[i]);
- kfree(cli_data->report_descr[i]);
if (cli_data->hid_sensor_hubs[i]) {
kfree(cli_data->hid_sensor_hubs[i]->driver_data);
hid_destroy_device(cli_data->hid_sensor_hubs[i]);
.driver_data = A4_2WHEEL_MOUSE_HACK_B8 },
{ HID_USB_DEVICE(USB_VENDOR_ID_A4TECH, USB_DEVICE_ID_A4TECH_RP_649),
.driver_data = A4_2WHEEL_MOUSE_HACK_B8 },
+ { HID_USB_DEVICE(USB_VENDOR_ID_A4TECH, USB_DEVICE_ID_A4TECH_NB_95),
+ .driver_data = A4_2WHEEL_MOUSE_HACK_B8 },
{ }
};
MODULE_DEVICE_TABLE(hid, a4_devices);
#define QUIRK_T100_KEYBOARD BIT(6)
#define QUIRK_T100CHI BIT(7)
#define QUIRK_G752_KEYBOARD BIT(8)
-#define QUIRK_T101HA_DOCK BIT(9)
-#define QUIRK_T90CHI BIT(10)
-#define QUIRK_MEDION_E1239T BIT(11)
-#define QUIRK_ROG_NKEY_KEYBOARD BIT(12)
+#define QUIRK_T90CHI BIT(9)
+#define QUIRK_MEDION_E1239T BIT(10)
+#define QUIRK_ROG_NKEY_KEYBOARD BIT(11)
#define I2C_KEYBOARD_QUIRKS (QUIRK_FIX_NOTEBOOK_REPORT | \
QUIRK_NO_INIT_REPORTS | \
if (drvdata->quirks & QUIRK_MEDION_E1239T)
return asus_e1239t_event(drvdata, data, size);
- if (drvdata->quirks & QUIRK_ROG_NKEY_KEYBOARD) {
+ if (drvdata->quirks & QUIRK_USE_KBD_BACKLIGHT) {
/*
* Skip these report ID, the device emits a continuous stream associated
* with the AURA mode it is in which looks like an 'echo'.
return -1;
}
}
+ if (drvdata->quirks & QUIRK_ROG_NKEY_KEYBOARD) {
+ /*
+ * G713 and G733 send these codes on some keypresses, depending on
+ * the key pressed it can trigger a shutdown event if not caught.
+ */
+ if(data[0] == 0x02 && data[1] == 0x30) {
+ return -1;
+ }
+ }
+
}
return 0;
return ret;
}
- /* use hid-multitouch for T101HA touchpad */
- if (id->driver_data & QUIRK_T101HA_DOCK &&
- hdev->collection->usage == HID_GD_MOUSE)
- return -ENODEV;
-
ret = hid_hw_start(hdev, HID_CONNECT_DEFAULT);
if (ret) {
hid_err(hdev, "Asus hw start failed: %d\n", ret);
{ HID_USB_DEVICE(USB_VENDOR_ID_ASUSTEK,
USB_DEVICE_ID_ASUSTEK_T100TAF_KEYBOARD),
QUIRK_T100_KEYBOARD | QUIRK_NO_CONSUMER_USAGES },
- { HID_USB_DEVICE(USB_VENDOR_ID_ASUSTEK,
- USB_DEVICE_ID_ASUSTEK_T101HA_KEYBOARD), QUIRK_T101HA_DOCK },
{ HID_USB_DEVICE(USB_VENDOR_ID_CHICONY, USB_DEVICE_ID_ASUS_AK1D) },
{ HID_USB_DEVICE(USB_VENDOR_ID_TURBOX, USB_DEVICE_ID_ASUS_MD_5110) },
{ HID_USB_DEVICE(USB_VENDOR_ID_JESS, USB_DEVICE_ID_ASUS_MD_5112) },
USB_DEVICE_ID_ASUSTEK_T100CHI_KEYBOARD), QUIRK_T100CHI },
{ HID_USB_DEVICE(USB_VENDOR_ID_ITE, USB_DEVICE_ID_ITE_MEDION_E1239T),
QUIRK_MEDION_E1239T },
+ /*
+ * Note bind to the HID_GROUP_GENERIC group, so that we only bind to the keyboard
+ * part, while letting hid-multitouch.c handle the touchpad.
+ */
+ { HID_DEVICE(BUS_USB, HID_GROUP_GENERIC,
+ USB_VENDOR_ID_ASUSTEK, USB_DEVICE_ID_ASUSTEK_T101HA_KEYBOARD) },
{ }
};
MODULE_DEVICE_TABLE(hid, asus_devices);
case BUS_I2C:
bus = "I2C";
break;
+ case BUS_VIRTUAL:
+ bus = "VIRTUAL";
+ break;
default:
bus = "<UNKNOWN>";
}
return 0;
}
-
EXPORT_SYMBOL_GPL(hid_check_keys_pressed);
static int __init hid_init(void)
[KEY_APPSELECT] = "AppSelect",
[KEY_SCREENSAVER] = "ScreenSaver",
[KEY_VOICECOMMAND] = "VoiceCommand",
+ [KEY_ASSISTANT] = "Assistant",
+ [KEY_KBD_LAYOUT_NEXT] = "KbdLayoutNext",
+ [KEY_EMOJI_PICKER] = "EmojiPicker",
[KEY_BRIGHTNESS_MIN] = "BrightnessMin",
[KEY_BRIGHTNESS_MAX] = "BrightnessMax",
[KEY_BRIGHTNESS_AUTO] = "BrightnessAuto",
u8 address; /* 7-bit I2C address */
u8 flag; /* I2C transaction condition */
u8 length; /* data payload length */
- u8 data[60]; /* data payload */
+ u8 data[FT260_WR_DATA_MAX]; /* data payload */
} __packed;
struct ft260_i2c_read_request_report {
ret = hid_hw_raw_request(hdev, report_id, buf, len, HID_FEATURE_REPORT,
HID_REQ_GET_REPORT);
- memcpy(data, buf, len);
+ if (likely(ret == len))
+ memcpy(data, buf, len);
+ else if (ret >= 0)
+ ret = -EIO;
kfree(buf);
return ret;
}
ret = ft260_hid_feature_report_get(hdev, FT260_I2C_STATUS,
(u8 *)&report, sizeof(report));
- if (ret < 0) {
+ if (unlikely(ret < 0)) {
hid_err(hdev, "failed to retrieve status: %d\n", ret);
return ret;
}
struct ft260_i2c_write_request_report *rep =
(struct ft260_i2c_write_request_report *)dev->write_buf;
+ if (data_len >= sizeof(rep->data))
+ return -EINVAL;
+
rep->address = addr;
rep->data[0] = cmd;
rep->length = data_len + 1;
ret = ft260_hid_feature_report_get(hdev, FT260_SYSTEM_SETTINGS,
(u8 *)cfg, len);
- if (ret != len) {
+ if (ret < 0) {
hid_err(hdev, "failed to retrieve system status\n");
- if (ret >= 0)
- return -EIO;
+ return ret;
}
return 0;
}
int ret;
ret = ft260_hid_feature_report_get(hdev, id, cfg, len);
- if (ret != len && ret >= 0)
- return -EIO;
+ if (ret < 0)
+ return ret;
return scnprintf(buf, PAGE_SIZE, "%hi\n", *field);
}
int ret;
ret = ft260_hid_feature_report_get(hdev, id, cfg, len);
- if (ret != len && ret >= 0)
- return -EIO;
+ if (ret < 0)
+ return ret;
return scnprintf(buf, PAGE_SIZE, "%hi\n", le16_to_cpu(*field));
}
ret = ft260_hid_feature_report_get(hdev, FT260_CHIP_VERSION,
(u8 *)&version, sizeof(version));
- if (ret != sizeof(version)) {
+ if (ret < 0) {
hid_err(hdev, "failed to retrieve chip version\n");
- if (ret >= 0)
- ret = -EIO;
goto err_hid_close;
}
{ HID_USB_DEVICE(USB_VENDOR_ID_MSI, USB_DEVICE_ID_MSI_GT683R_LED_PANEL) },
{ }
};
+MODULE_DEVICE_TABLE(hid, gt683r_led_id);
static void gt683r_brightness_set(struct led_classdev *led_cdev,
enum led_brightness brightness)
#define USB_DEVICE_ID_A4TECH_WCP32PU 0x0006
#define USB_DEVICE_ID_A4TECH_X5_005D 0x000a
#define USB_DEVICE_ID_A4TECH_RP_649 0x001a
+#define USB_DEVICE_ID_A4TECH_NB_95 0x022b
#define USB_VENDOR_ID_AASHIMA 0x06d6
#define USB_DEVICE_ID_AASHIMA_GAMEPAD 0x0025
#define USB_VENDOR_ID_CORSAIR 0x1b1c
#define USB_DEVICE_ID_CORSAIR_K90 0x1b02
-
-#define USB_VENDOR_ID_CORSAIR 0x1b1c
#define USB_DEVICE_ID_CORSAIR_K70R 0x1b09
#define USB_DEVICE_ID_CORSAIR_K95RGB 0x1b11
#define USB_DEVICE_ID_CORSAIR_M65RGB 0x1b12
#define USB_DEVICE_ID_LENOVO_X1_COVER 0x6085
#define USB_DEVICE_ID_LENOVO_X1_TAB 0x60a3
#define USB_DEVICE_ID_LENOVO_X1_TAB3 0x60b5
+#define USB_DEVICE_ID_LENOVO_OPTICAL_USB_MOUSE_600E 0x600e
#define USB_DEVICE_ID_LENOVO_PIXART_USB_MOUSE_608D 0x608d
#define USB_DEVICE_ID_LENOVO_PIXART_USB_MOUSE_6019 0x6019
#define USB_DEVICE_ID_LENOVO_PIXART_USB_MOUSE_602E 0x602e
#define USB_DEVICE_ID_SAITEK_X52 0x075c
#define USB_DEVICE_ID_SAITEK_X52_2 0x0255
#define USB_DEVICE_ID_SAITEK_X52_PRO 0x0762
+#define USB_DEVICE_ID_SAITEK_X65 0x0b6a
#define USB_VENDOR_ID_SAMSUNG 0x0419
#define USB_DEVICE_ID_SAMSUNG_IR_REMOTE 0x0001
#define USB_DEVICE_ID_SEMICO_USB_KEYKOARD 0x0023
#define USB_DEVICE_ID_SEMICO_USB_KEYKOARD2 0x0027
+#define USB_VENDOR_ID_SEMITEK 0x1ea7
+#define USB_DEVICE_ID_SEMITEK_KEYBOARD 0x0907
+
#define USB_VENDOR_ID_SENNHEISER 0x1395
#define USB_DEVICE_ID_SENNHEISER_BTD500USB 0x002c
#define USB_DEVICE_ID_SYNAPTICS_DELL_K12A 0x2819
#define USB_DEVICE_ID_SYNAPTICS_ACER_SWITCH5_012 0x2968
#define USB_DEVICE_ID_SYNAPTICS_TP_V103 0x5710
+#define USB_DEVICE_ID_SYNAPTICS_DELL_K15A 0x6e21
#define USB_DEVICE_ID_SYNAPTICS_ACER_ONE_S1002 0x73f4
#define USB_DEVICE_ID_SYNAPTICS_ACER_ONE_S1003 0x73f5
#define USB_DEVICE_ID_SYNAPTICS_ACER_SWITCH5 0x81a7
case 0x0cd: map_key_clear(KEY_PLAYPAUSE); break;
case 0x0cf: map_key_clear(KEY_VOICECOMMAND); break;
+
+ case 0x0d9: map_key_clear(KEY_EMOJI_PICKER); break;
+
case 0x0e0: map_abs_clear(ABS_VOLUME); break;
case 0x0e2: map_key_clear(KEY_MUTE); break;
case 0x0e5: map_key_clear(KEY_BASSBOOST); break;
int status;
long flags = (long) data[2];
+ *level = POWER_SUPPLY_CAPACITY_LEVEL_UNKNOWN;
if (flags & 0x80)
switch (flags & 0x07) {
if (id->vendor == USB_VENDOR_ID_APPLE &&
id->product == USB_DEVICE_ID_APPLE_MAGICTRACKPAD2 &&
hdev->type != HID_TYPE_USBMOUSE)
- return 0;
+ return -ENODEV;
msc = devm_kzalloc(&hdev->dev, sizeof(*msc), GFP_KERNEL);
if (msc == NULL) {
static void magicmouse_remove(struct hid_device *hdev)
{
struct magicmouse_sc *msc = hid_get_drvdata(hdev);
- cancel_delayed_work_sync(&msc->work);
+
+ if (msc)
+ cancel_delayed_work_sync(&msc->work);
+
hid_hw_stop(hdev);
}
#define MT_QUIRK_WIN8_PTP_BUTTONS BIT(18)
#define MT_QUIRK_SEPARATE_APP_REPORT BIT(19)
#define MT_QUIRK_FORCE_MULTI_INPUT BIT(20)
+#define MT_QUIRK_DISABLE_WAKEUP BIT(21)
#define MT_INPUTMODE_TOUCHSCREEN 0x02
#define MT_INPUTMODE_TOUCHPAD 0x03
#define MT_CLS_EXPORT_ALL_INPUTS 0x0013
/* reserved 0x0014 */
#define MT_CLS_WIN_8_FORCE_MULTI_INPUT 0x0015
+#define MT_CLS_WIN_8_DISABLE_WAKEUP 0x0016
/* vendor specific classes */
#define MT_CLS_3M 0x0101
MT_QUIRK_WIN8_PTP_BUTTONS |
MT_QUIRK_FORCE_MULTI_INPUT,
.export_all_inputs = true },
+ { .name = MT_CLS_WIN_8_DISABLE_WAKEUP,
+ .quirks = MT_QUIRK_ALWAYS_VALID |
+ MT_QUIRK_IGNORE_DUPLICATES |
+ MT_QUIRK_HOVERING |
+ MT_QUIRK_CONTACT_CNT_ACCURATE |
+ MT_QUIRK_STICKY_FINGERS |
+ MT_QUIRK_WIN8_PTP_BUTTONS |
+ MT_QUIRK_DISABLE_WAKEUP,
+ .export_all_inputs = true },
/*
* vendor specific classes
if (!(HID_MAIN_ITEM_VARIABLE & field->flags))
continue;
- for (n = 0; n < field->report_count; n++) {
- if (field->usage[n].hid == HID_DG_CONTACTID)
- rdata->is_mt_collection = true;
+ if (field->logical == HID_DG_FINGER || td->hdev->group != HID_GROUP_MULTITOUCH_WIN_8) {
+ for (n = 0; n < field->report_count; n++) {
+ if (field->usage[n].hid == HID_DG_CONTACTID) {
+ rdata->is_mt_collection = true;
+ break;
+ }
+ }
}
}
return 1;
case HID_DG_CONFIDENCE:
if ((cls->name == MT_CLS_WIN_8 ||
- cls->name == MT_CLS_WIN_8_FORCE_MULTI_INPUT) &&
+ cls->name == MT_CLS_WIN_8_FORCE_MULTI_INPUT ||
+ cls->name == MT_CLS_WIN_8_DISABLE_WAKEUP) &&
(field->application == HID_DG_TOUCHPAD ||
field->application == HID_DG_TOUCHSCREEN))
app->quirks |= MT_QUIRK_CONFIDENCE;
/* we do not set suffix = "Touchscreen" */
hi->input->name = hdev->name;
break;
- case HID_DG_STYLUS:
- /* force BTN_STYLUS to allow tablet matching in udev */
- __set_bit(BTN_STYLUS, hi->input->keybit);
- break;
case HID_VD_ASUS_CUSTOM_MEDIA_KEYS:
suffix = "Custom Media Keys";
break;
+ case HID_DG_STYLUS:
+ /* force BTN_STYLUS to allow tablet matching in udev */
+ __set_bit(BTN_STYLUS, hi->input->keybit);
+ fallthrough;
case HID_DG_PEN:
suffix = "Stylus";
break;
#ifdef CONFIG_PM
static int mt_suspend(struct hid_device *hdev, pm_message_t state)
{
+ struct mt_device *td = hid_get_drvdata(hdev);
+
/* High latency is desirable for power savings during S3/S0ix */
- mt_set_modes(hdev, HID_LATENCY_HIGH, true, true);
+ if (td->mtclass.quirks & MT_QUIRK_DISABLE_WAKEUP)
+ mt_set_modes(hdev, HID_LATENCY_HIGH, false, false);
+ else
+ mt_set_modes(hdev, HID_LATENCY_HIGH, true, true);
+
return 0;
}
MT_USB_DEVICE(USB_VENDOR_ID_ANTON,
USB_DEVICE_ID_ANTON_TOUCH_PAD) },
+ /* Asus T101HA */
+ { .driver_data = MT_CLS_WIN_8_DISABLE_WAKEUP,
+ HID_DEVICE(BUS_USB, HID_GROUP_MULTITOUCH_WIN_8,
+ USB_VENDOR_ID_ASUSTEK,
+ USB_DEVICE_ID_ASUSTEK_T101HA_KEYBOARD) },
+
/* Asus T304UA */
{ .driver_data = MT_CLS_ASUS,
HID_DEVICE(BUS_USB, HID_GROUP_MULTITOUCH_WIN_8,
{ HID_USB_DEVICE(USB_VENDOR_ID_KYE, USB_DEVICE_ID_KYE_PENSKETCH_M912), HID_QUIRK_MULTI_INPUT },
{ HID_USB_DEVICE(USB_VENDOR_ID_KYE, USB_DEVICE_ID_KYE_EASYPEN_M406XE), HID_QUIRK_MULTI_INPUT },
{ HID_USB_DEVICE(USB_VENDOR_ID_KYE, USB_DEVICE_ID_PIXART_USB_OPTICAL_MOUSE_ID2), HID_QUIRK_ALWAYS_POLL },
+ { HID_USB_DEVICE(USB_VENDOR_ID_LENOVO, USB_DEVICE_ID_LENOVO_OPTICAL_USB_MOUSE_600E), HID_QUIRK_ALWAYS_POLL },
{ HID_USB_DEVICE(USB_VENDOR_ID_LENOVO, USB_DEVICE_ID_LENOVO_PIXART_USB_MOUSE_608D), HID_QUIRK_ALWAYS_POLL },
{ HID_USB_DEVICE(USB_VENDOR_ID_LENOVO, USB_DEVICE_ID_LENOVO_PIXART_USB_MOUSE_6019), HID_QUIRK_ALWAYS_POLL },
{ HID_USB_DEVICE(USB_VENDOR_ID_LENOVO, USB_DEVICE_ID_LENOVO_PIXART_USB_MOUSE_602E), HID_QUIRK_ALWAYS_POLL },
{ HID_USB_DEVICE(USB_VENDOR_ID_SAITEK, USB_DEVICE_ID_SAITEK_X52), HID_QUIRK_INCREMENT_USAGE_ON_DUPLICATE },
{ HID_USB_DEVICE(USB_VENDOR_ID_SAITEK, USB_DEVICE_ID_SAITEK_X52_2), HID_QUIRK_INCREMENT_USAGE_ON_DUPLICATE },
{ HID_USB_DEVICE(USB_VENDOR_ID_SAITEK, USB_DEVICE_ID_SAITEK_X52_PRO), HID_QUIRK_INCREMENT_USAGE_ON_DUPLICATE },
+ { HID_USB_DEVICE(USB_VENDOR_ID_SAITEK, USB_DEVICE_ID_SAITEK_X65), HID_QUIRK_INCREMENT_USAGE_ON_DUPLICATE },
{ HID_USB_DEVICE(USB_VENDOR_ID_SEMICO, USB_DEVICE_ID_SEMICO_USB_KEYKOARD2), HID_QUIRK_NO_INIT_REPORTS },
{ HID_USB_DEVICE(USB_VENDOR_ID_SEMICO, USB_DEVICE_ID_SEMICO_USB_KEYKOARD), HID_QUIRK_NO_INIT_REPORTS },
{ HID_USB_DEVICE(USB_VENDOR_ID_SENNHEISER, USB_DEVICE_ID_SENNHEISER_BTD500USB), HID_QUIRK_NOGET },
{ HID_USB_DEVICE(USB_VENDOR_ID_SYNAPTICS, USB_DEVICE_ID_SYNAPTICS_QUAD_HD), HID_QUIRK_NO_INIT_REPORTS },
{ HID_USB_DEVICE(USB_VENDOR_ID_SYNAPTICS, USB_DEVICE_ID_SYNAPTICS_TP_V103), HID_QUIRK_NO_INIT_REPORTS },
{ HID_USB_DEVICE(USB_VENDOR_ID_SYNAPTICS, USB_DEVICE_ID_SYNAPTICS_DELL_K12A), HID_QUIRK_NO_INIT_REPORTS },
+ { HID_USB_DEVICE(USB_VENDOR_ID_SYNAPTICS, USB_DEVICE_ID_SYNAPTICS_DELL_K15A), HID_QUIRK_NO_INIT_REPORTS },
{ HID_USB_DEVICE(USB_VENDOR_ID_TOPMAX, USB_DEVICE_ID_TOPMAX_COBRAPAD), HID_QUIRK_BADPAD },
{ HID_USB_DEVICE(USB_VENDOR_ID_TOUCHPACK, USB_DEVICE_ID_TOUCHPACK_RTS), HID_QUIRK_MULTI_INPUT },
{ HID_USB_DEVICE(USB_VENDOR_ID_TPV, USB_DEVICE_ID_TPV_OPTICAL_TOUCHSCREEN_8882), HID_QUIRK_NOGET },
{ HID_USB_DEVICE(USB_VENDOR_ID_A4TECH, USB_DEVICE_ID_A4TECH_WCP32PU) },
{ HID_USB_DEVICE(USB_VENDOR_ID_A4TECH, USB_DEVICE_ID_A4TECH_X5_005D) },
{ HID_USB_DEVICE(USB_VENDOR_ID_A4TECH, USB_DEVICE_ID_A4TECH_RP_649) },
+ { HID_USB_DEVICE(USB_VENDOR_ID_A4TECH, USB_DEVICE_ID_A4TECH_NB_95) },
#endif
#if IS_ENABLED(CONFIG_HID_ACCUTOUCH)
{ HID_USB_DEVICE(USB_VENDOR_ID_ELO, USB_DEVICE_ID_ELO_ACCUTOUCH_2216) },
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * HID driver for Semitek keyboards
+ *
+ * Copyright (c) 2021 Benjamin Moody
+ */
+
+#include <linux/device.h>
+#include <linux/hid.h>
+#include <linux/module.h>
+
+#include "hid-ids.h"
+
+static __u8 *semitek_report_fixup(struct hid_device *hdev, __u8 *rdesc,
+ unsigned int *rsize)
+{
+ /* In the report descriptor for interface 2, fix the incorrect
+ description of report ID 0x04 (the report contains a
+ bitmask, not an array of keycodes.) */
+ if (*rsize == 0xcb && rdesc[0x83] == 0x81 && rdesc[0x84] == 0x00) {
+ hid_info(hdev, "fixing up Semitek report descriptor\n");
+ rdesc[0x84] = 0x02;
+ }
+ return rdesc;
+}
+
+static const struct hid_device_id semitek_devices[] = {
+ { HID_USB_DEVICE(USB_VENDOR_ID_SEMITEK, USB_DEVICE_ID_SEMITEK_KEYBOARD) },
+ { }
+};
+MODULE_DEVICE_TABLE(hid, semitek_devices);
+
+static struct hid_driver semitek_driver = {
+ .name = "semitek",
+ .id_table = semitek_devices,
+ .report_fixup = semitek_report_fixup,
+};
+module_hid_driver(semitek_driver);
+
+MODULE_LICENSE("GPL");
struct hid_sensor_custom *sensor_inst = dev_get_drvdata(dev);
int index, field_index, usage;
char name[HID_CUSTOM_NAME_LENGTH];
- int value;
+ int value, ret;
if (sscanf(attr->attr.name, "feature-%x-%x-%s", &index, &usage,
name) == 3) {
report_id = sensor_inst->fields[field_index].attribute.
report_id;
- sensor_hub_set_feature(sensor_inst->hsdev, report_id,
- index, sizeof(value), &value);
+ ret = sensor_hub_set_feature(sensor_inst->hsdev, report_id,
+ index, sizeof(value), &value);
+ if (ret)
+ return ret;
} else
return -EINVAL;
buffer_size = buffer_size / sizeof(__s32);
if (buffer_size) {
for (i = 0; i < buffer_size; ++i) {
- hid_set_field(report->field[field_index], i,
- (__force __s32)cpu_to_le32(*buf32));
+ ret = hid_set_field(report->field[field_index], i,
+ (__force __s32)cpu_to_le32(*buf32));
+ if (ret)
+ goto done_proc;
+
++buf32;
}
}
if (remaining_bytes) {
value = 0;
memcpy(&value, (u8 *)buf32, remaining_bytes);
- hid_set_field(report->field[field_index], i,
- (__force __s32)cpu_to_le32(value));
+ ret = hid_set_field(report->field[field_index], i,
+ (__force __s32)cpu_to_le32(value));
+ if (ret)
+ goto done_proc;
}
hid_hw_request(hsdev->hdev, report, HID_REQ_SET_REPORT);
hid_hw_wait(hsdev->hdev);
}
tm_wheel->change_request = kzalloc(sizeof(struct usb_ctrlrequest), GFP_KERNEL);
- if (!tm_wheel->model_request) {
+ if (!tm_wheel->change_request) {
ret = -ENOMEM;
goto error5;
}
#define I2C_HID_QUIRK_BOGUS_IRQ BIT(4)
#define I2C_HID_QUIRK_RESET_ON_RESUME BIT(5)
#define I2C_HID_QUIRK_BAD_INPUT_SIZE BIT(6)
+#define I2C_HID_QUIRK_NO_WAKEUP_AFTER_RESET BIT(7)
/* flags */
I2C_HID_QUIRK_RESET_ON_RESUME },
{ USB_VENDOR_ID_ITE, I2C_DEVICE_ID_ITE_LENOVO_LEGION_Y720,
I2C_HID_QUIRK_BAD_INPUT_SIZE },
+ /*
+ * Sending the wakeup after reset actually break ELAN touchscreen controller
+ */
+ { USB_VENDOR_ID_ELAN, HID_ANY_ID,
+ I2C_HID_QUIRK_NO_WAKEUP_AFTER_RESET },
{ 0, 0 }
};
}
/* At least some SIS devices need this after reset */
- ret = i2c_hid_set_power(client, I2C_HID_PWR_ON);
+ if (!(ihid->quirks & I2C_HID_QUIRK_NO_WAKEUP_AFTER_RESET))
+ ret = i2c_hid_set_power(client, I2C_HID_PWR_ON);
out_unlock:
mutex_unlock(&ihid->reset_lock);
hid->vendor = le16_to_cpu(ihid->hdesc.wVendorID);
hid->product = le16_to_cpu(ihid->hdesc.wProductID);
- snprintf(hid->name, sizeof(hid->name), "%s %04hX:%04hX",
- client->name, hid->vendor, hid->product);
+ snprintf(hid->name, sizeof(hid->name), "%s %04X:%04X",
+ client->name, (u16)hid->vendor, (u16)hid->product);
strlcpy(hid->phys, dev_name(&client->dev), sizeof(hid->phys));
ihid->quirks = i2c_hid_lookup_quirk(hid->vendor, hid->product);
#define EHL_Ax_DEVICE_ID 0x4BB3
#define TGL_LP_DEVICE_ID 0xA0FC
#define TGL_H_DEVICE_ID 0x43FC
+#define ADL_S_DEVICE_ID 0x7AF8
+#define ADL_P_DEVICE_ID 0x51FC
#define REVISION_ID_CHT_A0 0x6
#define REVISION_ID_CHT_Ax_SI 0x0
{PCI_DEVICE(PCI_VENDOR_ID_INTEL, EHL_Ax_DEVICE_ID)},
{PCI_DEVICE(PCI_VENDOR_ID_INTEL, TGL_LP_DEVICE_ID)},
{PCI_DEVICE(PCI_VENDOR_ID_INTEL, TGL_H_DEVICE_ID)},
+ {PCI_DEVICE(PCI_VENDOR_ID_INTEL, ADL_S_DEVICE_ID)},
+ {PCI_DEVICE(PCI_VENDOR_ID_INTEL, ADL_P_DEVICE_ID)},
{0, }
};
MODULE_DEVICE_TABLE(pci, ish_pci_tbl);
shid->hid->dev.parent = shid->dev;
shid->hid->bus = BUS_HOST;
- shid->hid->vendor = cpu_to_le16(shid->attrs.vendor);
- shid->hid->product = cpu_to_le16(shid->attrs.product);
- shid->hid->version = cpu_to_le16(shid->hid_desc.hid_version);
+ shid->hid->vendor = get_unaligned_le16(&shid->attrs.vendor);
+ shid->hid->product = get_unaligned_le16(&shid->attrs.product);
+ shid->hid->version = get_unaligned_le16(&shid->hid_desc.hid_version);
shid->hid->country = shid->hid_desc.country_code;
snprintf(shid->hid->name, sizeof(shid->hid->name), "Microsoft Surface %04X:%04X",
raw_report = usbhid->ctrl[usbhid->ctrltail].raw_report;
dir = usbhid->ctrl[usbhid->ctrltail].dir;
- len = ((report->size - 1) >> 3) + 1 + (report->id > 0);
+ len = hid_report_len(report);
if (dir == USB_DIR_OUT) {
usbhid->urbctrl->pipe = usb_sndctrlpipe(hid_to_usb_dev(hid), 0);
usbhid->urbctrl->transfer_buffer_length = len;
if (pidff->pool[PID_DEVICE_MANAGED_POOL].value &&
pidff->pool[PID_DEVICE_MANAGED_POOL].value[0] == 0) {
+ error = -EPERM;
hid_notice(hid,
"device does not support device managed pool\n");
goto fail;
reg = ADM9240_REG_IN_MIN(channel);
break;
case hwmon_in_max:
- reg = ADM9240_REG_IN(channel);
+ reg = ADM9240_REG_IN_MAX(channel);
break;
default:
return -EOPNOTSUPP;
return 0444;
default:
return 0;
- };
+ }
}
static umode_t corsairpsu_hwmon_in_is_visible(const struct corsairpsu_data *priv, u32 attr,
break;
default:
break;
- };
+ }
return res;
}
return 0;
}
+#ifdef CONFIG_PM
+static int corsairpsu_resume(struct hid_device *hdev)
+{
+ struct corsairpsu_data *priv = hid_get_drvdata(hdev);
+
+ /* some PSUs turn off the microcontroller during standby, so a reinit is required */
+ return corsairpsu_init(priv);
+}
+#endif
+
static const struct hid_device_id corsairpsu_idtable[] = {
{ HID_USB_DEVICE(0x1b1c, 0x1c03) }, /* Corsair HX550i */
{ HID_USB_DEVICE(0x1b1c, 0x1c04) }, /* Corsair HX650i */
.probe = corsairpsu_probe,
.remove = corsairpsu_remove,
.raw_event = corsairpsu_raw_event,
+#ifdef CONFIG_PM
+ .resume = corsairpsu_resume,
+ .reset_resume = corsairpsu_resume,
+#endif
};
module_hid_driver(corsairpsu_driver);
static umode_t i8k_is_visible(struct kobject *kobj, struct attribute *attr,
int index)
{
- if (disallow_fan_support && index >= 8)
+ if (disallow_fan_support && index >= 20)
return 0;
if (disallow_fan_type_call &&
- (index == 9 || index == 12 || index == 15))
+ (index == 21 || index == 25 || index == 28))
return 0;
if (index >= 0 && index <= 1 &&
!(i8k_hwmon_flags & I8K_HWMON_HAVE_TEMP1))
struct device *dev = &client->dev;
struct device *hwmon_dev;
struct lm80_data *data;
- int rv;
data = devm_kzalloc(dev, sizeof(struct lm80_data), GFP_KERNEL);
if (!data)
lm80_init_client(client);
/* A few vars need to be filled upon startup */
- rv = lm80_read_value(client, LM80_REG_FAN_MIN(1));
- if (rv < 0)
- return rv;
- data->fan[f_min][0] = rv;
- rv = lm80_read_value(client, LM80_REG_FAN_MIN(2));
- if (rv < 0)
- return rv;
- data->fan[f_min][1] = rv;
+ data->fan[f_min][0] = lm80_read_value(client, LM80_REG_FAN_MIN(1));
+ data->fan[f_min][1] = lm80_read_value(client, LM80_REG_FAN_MIN(2));
hwmon_dev = devm_hwmon_device_register_with_groups(dev, client->name,
data, lm80_groups);
fwnode_for_each_available_child_node(fwnode, child) {
ret = fwnode_property_read_u32(child, "reg", &addr);
- if (ret < 0)
+ if (ret < 0) {
+ fwnode_handle_put(child);
return ret;
+ }
- if (addr > 1)
+ if (addr > 1) {
+ fwnode_handle_put(child);
return -EINVAL;
+ }
ret = fwnode_property_read_u32(child, "shunt-resistor-micro-ohms", &val);
if (!ret)
return rc;
/* limit the maximum rate of polling the OCC */
- if (time_after(jiffies, occ->last_update + OCC_UPDATE_FREQUENCY)) {
+ if (time_after(jiffies, occ->next_update)) {
rc = occ_poll(occ);
- occ->last_update = jiffies;
+ occ->next_update = jiffies + OCC_UPDATE_FREQUENCY;
} else {
rc = occ->last_error;
}
return rc;
}
+ occ->next_update = jiffies + OCC_UPDATE_FREQUENCY;
occ_parse_poll_response(occ);
rc = occ_setup_sensor_attrs(occ);
u8 poll_cmd_data; /* to perform OCC poll command */
int (*send_cmd)(struct occ *occ, u8 *cmd);
- unsigned long last_update;
+ unsigned long next_update;
struct mutex lock; /* lock OCC access */
struct device *hwmon;
struct pmbus_driver_info info;
int chip;
int page;
+
+ bool vout_linear_11;
};
#define to_fsp3y_data(x) container_of(x, struct fsp3y_data, info)
case YH5151E_PAGE_12V_LOG:
return YH5151E_PAGE_12V_REAL;
case YH5151E_PAGE_5V_LOG:
- return YH5151E_PAGE_5V_LOG;
+ return YH5151E_PAGE_5V_REAL;
case YH5151E_PAGE_3V3_LOG:
return YH5151E_PAGE_3V3_REAL;
}
static int fsp3y_read_byte_data(struct i2c_client *client, int page, int reg)
{
+ const struct pmbus_driver_info *info = pmbus_get_driver_info(client);
+ struct fsp3y_data *data = to_fsp3y_data(info);
int rv;
+ /*
+ * Inject an exponent for non-compliant YH5151-E.
+ */
+ if (data->vout_linear_11 && reg == PMBUS_VOUT_MODE)
+ return 0x1A;
+
rv = set_page(client, page);
if (rv < 0)
return rv;
static int fsp3y_read_word_data(struct i2c_client *client, int page, int phase, int reg)
{
+ const struct pmbus_driver_info *info = pmbus_get_driver_info(client);
+ struct fsp3y_data *data = to_fsp3y_data(info);
int rv;
/*
if (rv < 0)
return rv;
- return i2c_smbus_read_word_data(client, reg);
+ rv = i2c_smbus_read_word_data(client, reg);
+ if (rv < 0)
+ return rv;
+
+ /*
+ * Handle YH-5151E non-compliant linear11 vout voltage.
+ */
+ if (data->vout_linear_11 && reg == PMBUS_READ_VOUT)
+ rv = sign_extend32(rv, 10) & 0xffff;
+
+ return rv;
}
static struct pmbus_driver_info fsp3y_info[] = {
data->info = fsp3y_info[data->chip];
+ /*
+ * YH-5151E sometimes reports vout in linear11 and sometimes in
+ * linear16. This depends on the exact individual piece of hardware. One
+ * YH-5151E can use linear16 and another might use linear11 instead.
+ *
+ * The format can be recognized by reading VOUT_MODE - if it doesn't
+ * report a valid exponent, then vout uses linear11. Otherwise, the
+ * device is compliant and uses linear16.
+ */
+ data->vout_linear_11 = false;
+ if (data->chip == yh5151e) {
+ rv = i2c_smbus_read_byte_data(client, PMBUS_VOUT_MODE);
+ if (rv < 0)
+ return rv;
+
+ if (rv == 0xFF)
+ data->vout_linear_11 = true;
+ }
+
return pmbus_do_probe(client, &data->info);
}
info->read_word_data = raa_dmpvr2_read_word_data;
break;
case raa_dmpvr2_2rail_nontc:
- info->func[0] &= ~PMBUS_HAVE_TEMP;
- info->func[1] &= ~PMBUS_HAVE_TEMP;
+ info->func[0] &= ~PMBUS_HAVE_TEMP3;
+ info->func[1] &= ~PMBUS_HAVE_TEMP3;
fallthrough;
case raa_dmpvr2_2rail:
info->pages = 2;
dev_err(&client->dev, "Failed to read Manufacturer ID\n");
return ret;
}
- if (ret != 5 || strncmp(buf, "DELTA", 5)) {
+ if (ret != 6 || strncmp(buf, "DELTA", 5)) {
buf[ret] = '\0';
dev_err(dev, "Unsupported Manufacturer ID '%s'\n", buf);
return -ENODEV;
scpi_scale_reading(&value, sensor);
+ /*
+ * Temperature sensor values are treated as signed values based on
+ * observation even though that is not explicitly specified, and
+ * because an unsigned u64 temperature does not really make practical
+ * sense especially when the temperature is below zero degrees Celsius.
+ */
+ if (sensor->info.class == TEMPERATURE)
+ return sprintf(buf, "%lld\n", (s64)value);
+
return sprintf(buf, "%llu\n", value);
}
#define POWER_ENABLE 0x19
#define TPS23861_NUM_PORTS 4
+#define TPS23861_GENERAL_MASK_1 0x17
+#define TPS23861_CURRENT_SHUNT_MASK BIT(0)
+
#define TEMPERATURE_LSB 652 /* 0.652 degrees Celsius */
#define VOLTAGE_LSB 3662 /* 3.662 mV */
#define SHUNT_RESISTOR_DEFAULT 255000 /* 255 mOhm */
-#define CURRENT_LSB_255 62260 /* 62.260 uA */
-#define CURRENT_LSB_250 61039 /* 61.039 uA */
+#define CURRENT_LSB_250 62260 /* 62.260 uA */
+#define CURRENT_LSB_255 61039 /* 61.039 uA */
#define RESISTANCE_LSB 110966 /* 11.0966 Ohm*/
#define RESISTANCE_LSB_LOW 157216 /* 15.7216 Ohm*/
static struct regmap_config tps23861_regmap_config = {
.reg_bits = 8,
.val_bits = 8,
+ .max_register = 0x6f,
};
static int tps23861_read_temp(struct tps23861_data *data, long *val)
else
data->shunt_resistor = SHUNT_RESISTOR_DEFAULT;
+ if (data->shunt_resistor == SHUNT_RESISTOR_DEFAULT)
+ regmap_clear_bits(data->regmap,
+ TPS23861_GENERAL_MASK_1,
+ TPS23861_CURRENT_SHUNT_MASK);
+ else
+ regmap_set_bits(data->regmap,
+ TPS23861_GENERAL_MASK_1,
+ TPS23861_CURRENT_SHUNT_MASK);
+
hwmon_dev = devm_hwmon_device_register_with_info(dev, client->name,
data, &tps23861_chip_info,
NULL);
config I2C_HISI
tristate "HiSilicon I2C controller"
- depends on ARM64 || COMPILE_TEST
+ depends on (ARM64 && ACPI) || COMPILE_TEST
help
Say Y here if you want to have Hisilicon I2C controller support
available on the Kunpeng Server.
// SPDX-License-Identifier: GPL-2.0-only
-/**
+/*
* i2c-ali1563.c - i2c driver for the ALi 1563 Southbridge
*
* Copyright (C) 2004 Patrick Mochel
#define ALTR_I2C_XFER_TIMEOUT (msecs_to_jiffies(250))
/**
- * altr_i2c_dev - I2C device context
+ * struct altr_i2c_dev - I2C device context
* @base: pointer to register struct
* @msg: pointer to current message
* @msg_len: number of bytes transferred in msg
altr_i2c_int_enable(idev, ALTR_I2C_ALL_IRQ, false);
}
-/**
+/*
* altr_i2c_transfer - On the last byte to be transmitted, send
* a Stop bit on the last byte.
*/
writel(data, idev->base + ALTR_I2C_TFR_CMD);
}
-/**
+/*
* altr_i2c_empty_rx_fifo - Fetch data from RX FIFO until end of
* transfer. Send a Stop bit on the last byte.
*/
}
}
-/**
+/*
* altr_i2c_fill_tx_fifo - Fill TX FIFO from current message buffer.
- * @return: Number of bytes left to transfer.
*/
static int altr_i2c_fill_tx_fifo(struct altr_i2c_dev *idev)
{
};
/**
- * enum cdns_i2c_slave_mode - Slave state when I2C is operating in slave mode
+ * enum cdns_i2c_slave_state - Slave state when I2C is operating in slave mode
*
* @CDNS_I2C_SLAVE_STATE_IDLE: I2C slave idle
* @CDNS_I2C_SLAVE_STATE_SEND: I2C slave sending data to master
}
/**
- * i2c_dw_init() - Initialize the designware I2C master hardware
+ * i2c_dw_init_master() - Initialize the designware I2C master hardware
* @dev: device private data
*
* This functions configures and enables the I2C master.
/**
* struct adapter_info - This structure holds the adapter information for the
- PCH i2c controller
+ * PCH i2c controller
* @pch_data: stores a list of i2c_algo_pch_data
* @pch_i2c_suspended: specifies whether the system is suspended or not
* perhaps with more lines and words.
/**
* pch_i2c_writebytes() - write data to I2C bus in normal mode
* @i2c_adap: Pointer to the struct i2c_adapter.
+ * @msgs: Pointer to the i2c message structure.
* @last: specifies whether last message or not.
* In the case of compound mode it will be 1 for last message,
* otherwise 0.
dev_err(&priv->pci_dev->dev, "Transaction timeout\n");
/* try to stop the current command */
dev_dbg(&priv->pci_dev->dev, "Terminating the current operation\n");
- outb_p(inb_p(SMBHSTCNT(priv)) | SMBHSTCNT_KILL,
- SMBHSTCNT(priv));
+ outb_p(SMBHSTCNT_KILL, SMBHSTCNT(priv));
usleep_range(1000, 2000);
- outb_p(inb_p(SMBHSTCNT(priv)) & (~SMBHSTCNT_KILL),
- SMBHSTCNT(priv));
+ outb_p(0, SMBHSTCNT(priv));
/* Check if it worked */
status = inb_p(SMBHSTSTS(priv));
{
struct icy_i2c *i2c;
struct i2c_algo_pcf_data *algo_data;
- struct fwnode_handle *new_fwnode;
struct i2c_board_info ltc2990_info = {
.type = "ltc2990",
.swnode = &icy_ltc2990_node,
#include <linux/clk.h>
#include <linux/io.h>
+#include <linux/iopoll.h>
#include <linux/fsl_devices.h>
#include <linux/i2c.h>
#include <linux/interrupt.h>
#define CCR_MTX 0x10
#define CCR_TXAK 0x08
#define CCR_RSTA 0x04
+#define CCR_RSVD 0x02
#define CSR_MCF 0x80
#define CSR_MAAS 0x40
u32 block;
int rc;
int expect_rxack;
-
+ bool has_errata_A004447;
};
struct mpc_i2c_divider {
}
}
+static int i2c_mpc_wait_sr(struct mpc_i2c *i2c, int mask)
+{
+ void __iomem *addr = i2c->base + MPC_I2C_SR;
+ u8 val;
+
+ return readb_poll_timeout(addr, val, val & mask, 0, 100);
+}
+
+/*
+ * Workaround for Erratum A004447. From the P2040CE Rev Q
+ *
+ * 1. Set up the frequency divider and sampling rate.
+ * 2. I2CCR - a0h
+ * 3. Poll for I2CSR[MBB] to get set.
+ * 4. If I2CSR[MAL] is set (an indication that SDA is stuck low), then go to
+ * step 5. If MAL is not set, then go to step 13.
+ * 5. I2CCR - 00h
+ * 6. I2CCR - 22h
+ * 7. I2CCR - a2h
+ * 8. Poll for I2CSR[MBB] to get set.
+ * 9. Issue read to I2CDR.
+ * 10. Poll for I2CSR[MIF] to be set.
+ * 11. I2CCR - 82h
+ * 12. Workaround complete. Skip the next steps.
+ * 13. Issue read to I2CDR.
+ * 14. Poll for I2CSR[MIF] to be set.
+ * 15. I2CCR - 80h
+ */
+static void mpc_i2c_fixup_A004447(struct mpc_i2c *i2c)
+{
+ int ret;
+ u32 val;
+
+ writeccr(i2c, CCR_MEN | CCR_MSTA);
+ ret = i2c_mpc_wait_sr(i2c, CSR_MBB);
+ if (ret) {
+ dev_err(i2c->dev, "timeout waiting for CSR_MBB\n");
+ return;
+ }
+
+ val = readb(i2c->base + MPC_I2C_SR);
+
+ if (val & CSR_MAL) {
+ writeccr(i2c, 0x00);
+ writeccr(i2c, CCR_MSTA | CCR_RSVD);
+ writeccr(i2c, CCR_MEN | CCR_MSTA | CCR_RSVD);
+ ret = i2c_mpc_wait_sr(i2c, CSR_MBB);
+ if (ret) {
+ dev_err(i2c->dev, "timeout waiting for CSR_MBB\n");
+ return;
+ }
+ val = readb(i2c->base + MPC_I2C_DR);
+ ret = i2c_mpc_wait_sr(i2c, CSR_MIF);
+ if (ret) {
+ dev_err(i2c->dev, "timeout waiting for CSR_MIF\n");
+ return;
+ }
+ writeccr(i2c, CCR_MEN | CCR_RSVD);
+ } else {
+ val = readb(i2c->base + MPC_I2C_DR);
+ ret = i2c_mpc_wait_sr(i2c, CSR_MIF);
+ if (ret) {
+ dev_err(i2c->dev, "timeout waiting for CSR_MIF\n");
+ return;
+ }
+ writeccr(i2c, CCR_MEN);
+ }
+}
+
#if defined(CONFIG_PPC_MPC52xx) || defined(CONFIG_PPC_MPC512x)
static const struct mpc_i2c_divider mpc_i2c_dividers_52xx[] = {
{20, 0x20}, {22, 0x21}, {24, 0x22}, {26, 0x23},
{
struct mpc_i2c *i2c = i2c_get_adapdata(adap);
- mpc_i2c_fixup(i2c);
+ if (i2c->has_errata_A004447)
+ mpc_i2c_fixup_A004447(i2c);
+ else
+ mpc_i2c_fixup(i2c);
return 0;
}
}
dev_info(i2c->dev, "timeout %u us\n", mpc_ops.timeout * 1000000 / HZ);
+ if (of_property_read_bool(op->dev.of_node, "fsl,i2c-erratum-a004447"))
+ i2c->has_errata_A004447 = true;
+
i2c->adap = mpc_ops;
scnprintf(i2c->adap.name, sizeof(i2c->adap.name),
"MPC adapter (%s)", of_node_full_name(op->dev.of_node));
static void mtk_i2c_init_hw(struct mtk_i2c *i2c)
{
u16 control_reg;
+ u16 intr_stat_reg;
+
+ mtk_i2c_writew(i2c, I2C_CHN_CLR_FLAG, OFFSET_START);
+ intr_stat_reg = mtk_i2c_readw(i2c, OFFSET_INTR_STAT);
+ mtk_i2c_writew(i2c, intr_stat_reg, OFFSET_INTR_STAT);
if (i2c->dev_comp->apdma_sync) {
writel(I2C_DMA_WARM_RST, i2c->pdmabase + OFFSET_RST);
* @clk_freq: clock frequency for the operation mode
* @tft: Tx FIFO Threshold in bytes
* @rft: Rx FIFO Threshold in bytes
- * @timeout Slave response timeout (ms)
+ * @timeout: Slave response timeout (ms)
* @sm: speed mode
* @stop: stop condition.
* @xfer_complete: acknowledge completion for a I2C message.
}
/**
- * Process timeout event
+ * ocores_process_timeout() - Process timeout event
* @i2c: ocores I2C device instance
*/
static void ocores_process_timeout(struct ocores_i2c *i2c)
}
/**
- * Wait until something change in a given register
+ * ocores_wait() - Wait until something change in a given register
* @i2c: ocores I2C device instance
* @reg: register to query
* @mask: bitmask to apply on register value
}
/**
- * Wait until is possible to process some data
+ * ocores_poll_wait() - Wait until is possible to process some data
* @i2c: ocores I2C device instance
*
* Used when the device is in polling mode (interrupts disabled).
}
/**
- * It handles an IRQ-less transfer
+ * ocores_process_polling() - It handles an IRQ-less transfer
* @i2c: ocores I2C device instance
*
* Even if IRQ are disabled, the I2C OpenCore IP behavior is exactly the same
/**
* i2c_pnx_start - start a device
* @slave_addr: slave address
- * @adap: pointer to adapter structure
+ * @alg_data: pointer to local driver data structure
*
* Generate a START signal in the desired mode.
*/
/**
* i2c_pnx_stop - stop a device
- * @adap: pointer to I2C adapter structure
+ * @alg_data: pointer to local driver data structure
*
* Generate a STOP signal to terminate the master transaction.
*/
/**
* i2c_pnx_master_xmit - transmit data to slave
- * @adap: pointer to I2C adapter structure
+ * @alg_data: pointer to local driver data structure
*
* Sends one byte of data to the slave
*/
/**
* i2c_pnx_master_rcv - receive data from slave
- * @adap: pointer to I2C adapter structure
+ * @alg_data: pointer to local driver data structure
*
* Reads one byte data from the slave
*/
[GP_IRQ0] = {-EIO, "Unknown I2C err GP_IRQ0"},
[NACK] = {-ENXIO, "NACK: slv unresponsive, check its power/reset-ln"},
[GP_IRQ2] = {-EIO, "Unknown I2C err GP IRQ2"},
- [BUS_PROTO] = {-EPROTO, "Bus proto err, noisy/unepxected start/stop"},
+ [BUS_PROTO] = {-EPROTO, "Bus proto err, noisy/unexpected start/stop"},
[ARB_LOST] = {-EAGAIN, "Bus arbitration lost, clock line undriveable"},
[GP_IRQ5] = {-EIO, "Unknown I2C err GP IRQ5"},
[GENI_OVERRUN] = {-EIO, "Cmd overrun, check GENI cmd-state machine"},
return 0;
}
+static void geni_i2c_shutdown(struct platform_device *pdev)
+{
+ struct geni_i2c_dev *gi2c = platform_get_drvdata(pdev);
+
+ /* Make client i2c transfers start failing */
+ i2c_mark_adapter_suspended(&gi2c->adap);
+}
+
static int __maybe_unused geni_i2c_runtime_suspend(struct device *dev)
{
int ret;
{
struct geni_i2c_dev *gi2c = dev_get_drvdata(dev);
+ i2c_mark_adapter_suspended(&gi2c->adap);
+
if (!gi2c->suspended) {
geni_i2c_runtime_suspend(dev);
pm_runtime_disable(dev);
return 0;
}
+static int __maybe_unused geni_i2c_resume_noirq(struct device *dev)
+{
+ struct geni_i2c_dev *gi2c = dev_get_drvdata(dev);
+
+ i2c_mark_adapter_resumed(&gi2c->adap);
+ return 0;
+}
+
static const struct dev_pm_ops geni_i2c_pm_ops = {
- SET_NOIRQ_SYSTEM_SLEEP_PM_OPS(geni_i2c_suspend_noirq, NULL)
+ SET_NOIRQ_SYSTEM_SLEEP_PM_OPS(geni_i2c_suspend_noirq, geni_i2c_resume_noirq)
SET_RUNTIME_PM_OPS(geni_i2c_runtime_suspend, geni_i2c_runtime_resume,
NULL)
};
static struct platform_driver geni_i2c_driver = {
.probe = geni_i2c_probe,
.remove = geni_i2c_remove,
+ .shutdown = geni_i2c_shutdown,
.driver = {
.name = "geni_i2c",
.pm = &geni_i2c_pm_ops,
* forces us to send a new START
* when we change direction
*/
+ dev_dbg(i2c->dev,
+ "missing START before write->read\n");
s3c24xx_i2c_stop(i2c, -EINVAL);
+ break;
}
goto retry_write;
static const struct of_device_id sh_mobile_i2c_dt_ids[] = {
{ .compatible = "renesas,iic-r8a73a4", .data = &fast_clock_dt_config },
{ .compatible = "renesas,iic-r8a7740", .data = &r8a7740_dt_config },
- { .compatible = "renesas,iic-r8a774c0", .data = &fast_clock_dt_config },
+ { .compatible = "renesas,iic-r8a774c0", .data = &v2_freq_calc_dt_config },
{ .compatible = "renesas,iic-r8a7790", .data = &v2_freq_calc_dt_config },
{ .compatible = "renesas,iic-r8a7791", .data = &v2_freq_calc_dt_config },
{ .compatible = "renesas,iic-r8a7792", .data = &v2_freq_calc_dt_config },
}
/**
- * st_i2c_handle_write() - Handle FIFO enmpty interrupt in case of read
+ * st_i2c_handle_read() - Handle FIFO empty interrupt in case of read
* @i2c_dev: Controller's private data
*/
static void st_i2c_handle_read(struct st_i2c_dev *i2c_dev)
}
/**
- * st_i2c_isr() - Interrupt routine
+ * st_i2c_isr_thread() - Interrupt routine
* @irq: interrupt number
* @data: Controller's private data
*/
}
/**
- * stm32f4_i2c_write_ byte() - Write a byte in the data register
+ * stm32f4_i2c_write_byte() - Write a byte in the data register
* @i2c_dev: Controller's private data
* @byte: Data to write in the register
*/
*out |= SERIALI2C_RECV_LEN;
}
-/**
+/*
* The serialized I2C format is simply the following:
* [addr little-endian][flags little-endian][len little-endian][data if write]
* [addr little-endian][flags little-endian][len little-endian][data if write]
request->xfer.data_size = pos;
}
-/**
+/*
* The data in the BPMP -> CPU direction is composed of sequential blocks for
* those messages that have I2C_M_RD. So, for example, if you have:
*
};
-/**
+/*
* i2c_arbitrator_select - claim the I2C bus
*
* Use the GPIO-based signalling protocol; return -EBUSY if we fail.
return -EBUSY;
}
-/**
+/*
* i2c_arbitrator_deselect - release the I2C bus
*
* Release the I2C bus using the GPIO-based signalling protocol.
config HID_SENSOR_ACCEL_3D
depends on HID_SENSOR_HUB
select IIO_BUFFER
- select IIO_TRIGGERED_BUFFER
select HID_SENSOR_IIO_COMMON
select HID_SENSOR_IIO_TRIGGER
tristate "HID Accelerometers 3D"
if (ret)
goto err;
+ if (channel >= indio_dev->num_channels) {
+ dev_err(indio_dev->dev.parent,
+ "Channel index >= number of channels\n");
+ ret = -EINVAL;
+ goto err;
+ }
+
ret = of_property_read_u32_array(child, "diff-channels",
ain, 2);
if (ret)
return ret;
}
+static void ad7124_reg_disable(void *r)
+{
+ regulator_disable(r);
+}
+
static int ad7124_probe(struct spi_device *spi)
{
const struct ad7124_chip_info *info;
ret = regulator_enable(st->vref[i]);
if (ret)
return ret;
+
+ ret = devm_add_action_or_reset(&spi->dev, ad7124_reg_disable,
+ st->vref[i]);
+ if (ret)
+ return ret;
}
st->mclk = devm_clk_get(&spi->dev, "mclk");
- if (IS_ERR(st->mclk)) {
- ret = PTR_ERR(st->mclk);
- goto error_regulator_disable;
- }
+ if (IS_ERR(st->mclk))
+ return PTR_ERR(st->mclk);
ret = clk_prepare_enable(st->mclk);
if (ret < 0)
- goto error_regulator_disable;
+ return ret;
ret = ad7124_soft_reset(st);
if (ret < 0)
ad_sd_cleanup_buffer_and_trigger(indio_dev);
error_clk_disable_unprepare:
clk_disable_unprepare(st->mclk);
-error_regulator_disable:
- for (i = ARRAY_SIZE(st->vref) - 1; i >= 0; i--) {
- if (!IS_ERR_OR_NULL(st->vref[i]))
- regulator_disable(st->vref[i]);
- }
return ret;
}
{
struct iio_dev *indio_dev = spi_get_drvdata(spi);
struct ad7124_state *st = iio_priv(indio_dev);
- int i;
iio_device_unregister(indio_dev);
ad_sd_cleanup_buffer_and_trigger(indio_dev);
clk_disable_unprepare(st->mclk);
- for (i = ARRAY_SIZE(st->vref) - 1; i >= 0; i--) {
- if (!IS_ERR_OR_NULL(st->vref[i]))
- regulator_disable(st->vref[i]);
- }
-
return 0;
}
{
struct ad7192_state *st;
struct iio_dev *indio_dev;
- int ret, voltage_uv = 0;
+ int ret;
if (!spi->irq) {
dev_err(&spi->dev, "no IRQ?\n");
goto error_disable_avdd;
}
- voltage_uv = regulator_get_voltage(st->avdd);
-
- if (voltage_uv > 0) {
- st->int_vref_mv = voltage_uv / 1000;
- } else {
- ret = voltage_uv;
+ ret = regulator_get_voltage(st->avdd);
+ if (ret < 0) {
dev_err(&spi->dev, "Device tree error, reference voltage undefined\n");
goto error_disable_avdd;
}
+ st->int_vref_mv = ret / 1000;
spi_set_drvdata(spi, indio_dev);
st->chip_info = of_device_get_match_data(&spi->dev);
return 0;
error_disable_clk:
- clk_disable_unprepare(st->mclk);
+ if (st->clock_sel == AD7192_CLK_EXT_MCLK1_2 ||
+ st->clock_sel == AD7192_CLK_EXT_MCLK2)
+ clk_disable_unprepare(st->mclk);
error_remove_trigger:
ad_sd_cleanup_buffer_and_trigger(indio_dev);
error_disable_dvdd:
struct ad7192_state *st = iio_priv(indio_dev);
iio_device_unregister(indio_dev);
- clk_disable_unprepare(st->mclk);
+ if (st->clock_sel == AD7192_CLK_EXT_MCLK1_2 ||
+ st->clock_sel == AD7192_CLK_EXT_MCLK2)
+ clk_disable_unprepare(st->mclk);
ad_sd_cleanup_buffer_and_trigger(indio_dev);
regulator_disable(st->dvdd);
* transfer buffers to live in their own cache lines.
*/
union {
+ struct {
+ __be32 chan;
+ s64 timestamp;
+ } scan;
__be32 d32;
u8 d8[2];
} data ____cacheline_aligned;
mutex_lock(&st->lock);
- ret = spi_read(st->spi, &st->data.d32, 3);
+ ret = spi_read(st->spi, &st->data.scan.chan, 3);
if (ret < 0)
goto err_unlock;
- iio_push_to_buffers_with_timestamp(indio_dev, &st->data.d32,
+ iio_push_to_buffers_with_timestamp(indio_dev, &st->data.scan,
iio_get_time_ns(indio_dev));
iio_trigger_notify_done(indio_dev->trig);
id &= AD7793_ID_MASK;
if (id != st->chip_info->id) {
+ ret = -ENODEV;
dev_err(&st->sd.spi->dev, "device ID query failed\n");
goto out;
}
/*
* DMA (thus cache coherency maintenance) requires the
* transfer buffers to live in their own cache lines.
+ * Ensure rx_buf can be directly used in iio_push_to_buffers_with_timetamp
+ * Length = 8 channels + 4 extra for 8 byte timestamp
*/
- __be16 rx_buf[4] ____cacheline_aligned;
+ __be16 rx_buf[12] ____cacheline_aligned;
__be16 tx_buf[4];
};
tristate "Common module (trigger) for all HID Sensor IIO drivers"
depends on HID_SENSOR_HUB && HID_SENSOR_IIO_COMMON && IIO_BUFFER
select IIO_TRIGGER
+ select IIO_TRIGGERED_BUFFER
help
Say yes here to build trigger support for HID sensors.
Triggers will be send if all requested attributes were read.
device_for_each_child_node(&st->spi->dev, child) {
ret = fwnode_property_read_u32(child, "num", &num);
if (ret)
- return ret;
- if (num >= AD5770R_MAX_CHANNELS)
- return -EINVAL;
+ goto err_child_out;
+ if (num >= AD5770R_MAX_CHANNELS) {
+ ret = -EINVAL;
+ goto err_child_out;
+ }
ret = fwnode_property_read_u32_array(child,
"adi,range-microamp",
tmp, 2);
if (ret)
- return ret;
+ goto err_child_out;
min = tmp[0] / 1000;
max = tmp[1] / 1000;
ret = ad5770r_store_output_range(st, min, max, num);
if (ret)
- return ret;
+ goto err_child_out;
}
+ return 0;
+
+err_child_out:
+ fwnode_handle_put(child);
return ret;
}
config HID_SENSOR_GYRO_3D
depends on HID_SENSOR_HUB
select IIO_BUFFER
- select IIO_TRIGGERED_BUFFER
select HID_SENSOR_IIO_COMMON
select HID_SENSOR_IIO_TRIGGER
tristate "HID Gyroscope 3D"
ret = regmap_field_read(data->regmap_fields[F_TEMP], &temp);
if (ret < 0) {
dev_err(dev, "failed to read temp: %d\n", ret);
+ fxas21002c_pm_put(data);
goto data_unlock;
}
&axis_be, sizeof(axis_be));
if (ret < 0) {
dev_err(dev, "failed to read axis: %d: %d\n", index, ret);
+ fxas21002c_pm_put(data);
goto data_unlock;
}
case IIO_CHAN_INFO_OFFSET:
switch (chan->type) {
case IIO_TEMP:
- /* The temperature scaling is (x+23000)/280 Celsius */
+ /*
+ * The temperature scaling is (x+23000)/280 Celsius
+ * for the "best fit straight line" temperature range
+ * of -30C..85C. The 23000 includes room temperature
+ * offset of +35C, 280 is the precision scale and x is
+ * the 16-bit signed integer reported by hardware.
+ *
+ * Temperature value itself represents temperature of
+ * the sensor die.
+ */
*val = 23000;
return IIO_VAL_INT;
default:
goto out_read_raw_unlock;
}
- *val = be16_to_cpu(raw_val);
+ *val = (s16)be16_to_cpu(raw_val);
ret = IIO_VAL_INT;
goto out_read_raw_unlock;
tristate "HID Environmental humidity sensor"
depends on HID_SENSOR_HUB
select IIO_BUFFER
- select IIO_TRIGGERED_BUFFER
select HID_SENSOR_IIO_COMMON
select HID_SENSOR_IIO_TRIGGER
help
if (!indio_dev->info)
goto out_unlock;
- ret = -EINVAL;
list_for_each_entry(h, &iio_dev_opaque->ioctl_handlers, entry) {
ret = h->ioctl(indio_dev, filp, cmd, arg);
if (ret != IIO_IOCTL_UNHANDLED)
}
if (ret == IIO_IOCTL_UNHANDLED)
- ret = -EINVAL;
+ ret = -ENODEV;
out_unlock:
mutex_unlock(&indio_dev->info_exist_lock);
**/
void iio_device_unregister(struct iio_dev *indio_dev)
{
- struct iio_dev_opaque *iio_dev_opaque = to_iio_dev_opaque(indio_dev);
- struct iio_ioctl_handler *h, *t;
-
cdev_device_del(&indio_dev->chrdev, &indio_dev->dev);
mutex_lock(&indio_dev->info_exist_lock);
indio_dev->info = NULL;
- list_for_each_entry_safe(h, t, &iio_dev_opaque->ioctl_handlers, entry)
- list_del(&h->entry);
-
iio_device_wakeup_eventset(indio_dev);
iio_buffer_wakeup_poll(indio_dev);
config HID_SENSOR_ALS
depends on HID_SENSOR_HUB
select IIO_BUFFER
- select IIO_TRIGGERED_BUFFER
select HID_SENSOR_IIO_COMMON
select HID_SENSOR_IIO_TRIGGER
tristate "HID ALS"
config HID_SENSOR_PROX
depends on HID_SENSOR_HUB
select IIO_BUFFER
- select IIO_TRIGGERED_BUFFER
select HID_SENSOR_IIO_COMMON
select HID_SENSOR_IIO_TRIGGER
tristate "HID PROX"
"gp2ap002", indio_dev);
if (ret) {
dev_err(dev, "unable to request IRQ\n");
- goto out_disable_vio;
+ goto out_put_pm;
}
gp2ap002->irq = client->irq;
return 0;
-out_disable_pm:
+out_put_pm:
pm_runtime_put_noidle(dev);
+out_disable_pm:
pm_runtime_disable(dev);
out_disable_vio:
regulator_disable(gp2ap002->vio);
return lux_val;
}
+ /* Avoid division by zero of lux_value later on */
+ if (lux_val == 0) {
+ dev_err(&chip->client->dev,
+ "%s: lux_val of 0 will produce out of range trim_value\n",
+ __func__);
+ return -ENODATA;
+ }
+
gain_trim_val = (unsigned int)(((chip->als_settings.als_cal_target)
* chip->als_settings.als_gain_trim) / lux_val);
if ((gain_trim_val < 250) || (gain_trim_val > 4000)) {
config HID_SENSOR_MAGNETOMETER_3D
depends on HID_SENSOR_HUB
select IIO_BUFFER
- select IIO_TRIGGERED_BUFFER
select HID_SENSOR_IIO_COMMON
select HID_SENSOR_IIO_TRIGGER
tristate "HID Magenetometer 3D"
config HID_SENSOR_INCLINOMETER_3D
depends on HID_SENSOR_HUB
select IIO_BUFFER
- select IIO_TRIGGERED_BUFFER
select HID_SENSOR_IIO_COMMON
select HID_SENSOR_IIO_TRIGGER
tristate "HID Inclinometer 3D"
config HID_SENSOR_DEVICE_ROTATION
depends on HID_SENSOR_HUB
select IIO_BUFFER
- select IIO_TRIGGERED_BUFFER
select HID_SENSOR_IIO_COMMON
select HID_SENSOR_IIO_TRIGGER
tristate "HID Device Rotation"
config HID_SENSOR_PRESS
depends on HID_SENSOR_HUB
select IIO_BUFFER
- select IIO_TRIGGERED_BUFFER
select HID_SENSOR_IIO_COMMON
select HID_SENSOR_IIO_TRIGGER
tristate "HID PRESS"
ret = lidar_write_control(data, LIDAR_REG_CONTROL_ACQUIRE);
if (ret < 0) {
dev_err(&client->dev, "cannot send start measurement command");
+ pm_runtime_put_noidle(&client->dev);
return ret;
}
tristate "HID Environmental temperature sensor"
depends on HID_SENSOR_HUB
select IIO_BUFFER
- select IIO_TRIGGERED_BUFFER
select HID_SENSOR_IIO_COMMON
select HID_SENSOR_IIO_TRIGGER
help
list_del(&id_priv->list);
cma_dev_put(id_priv->cma_dev);
id_priv->cma_dev = NULL;
+ id_priv->id.device = NULL;
if (id_priv->id.route.addr.dev_addr.sgid_attr) {
rdma_put_gid_attr(id_priv->id.route.addr.dev_addr.sgid_attr);
id_priv->id.route.addr.dev_addr.sgid_attr = NULL;
iw_destroy_cm_id(id_priv->cm_id.iw);
}
cma_leave_mc_groups(id_priv);
+ rdma_restrack_del(&id_priv->res);
cma_release_dev(id_priv);
}
kfree(id_priv->id.route.path_rec);
put_net(id_priv->id.route.addr.dev_addr.net);
- rdma_restrack_del(&id_priv->res);
kfree(id_priv);
}
}
id_priv->backlog = backlog;
- if (id->device) {
+ if (id_priv->cma_dev) {
if (rdma_cap_ib_cm(id->device, 1)) {
ret = cma_ib_listen(id_priv);
if (ret)
goto err_free_attr;
}
+ if (!rdma_is_port_valid(uobj->context->device, cmd.flow_attr.port)) {
+ err = -EINVAL;
+ goto err_uobj;
+ }
+
qp = uobj_get_obj_read(qp, UVERBS_OBJECT_QP, cmd.qp_handle, attrs);
if (!qp) {
err = -EINVAL;
return ret;
uapi_object = uapi_get_object(attrs->ufile->device->uapi, object_id);
- if (!uapi_object)
- return -EINVAL;
+ if (IS_ERR(uapi_object))
+ return PTR_ERR(uapi_object);
handles = gather_objects_handle(attrs->ufile, uapi_object, attrs,
out_len, &total);
if (ret)
return ret;
+ if (!user_entry_size)
+ return -EINVAL;
+
max_entries = uverbs_attr_ptr_get_array_size(
attrs, UVERBS_ATTR_QUERY_GID_TABLE_RESP_ENTRIES,
user_entry_size);
struct i40e_qv_info *iw_qvinfo;
u32 ceq_idx;
u32 i;
- u32 size;
+ size_t size;
if (!ldev->msix_count) {
i40iw_pr_err("No MSI-X vectors\n");
iwdev->msix_count = ldev->msix_count;
size = sizeof(struct i40iw_msix_vector) * iwdev->msix_count;
- size += sizeof(struct i40e_qvlist_info);
- size += sizeof(struct i40e_qv_info) * iwdev->msix_count - 1;
+ size += struct_size(iw_qvlist, qv_info, iwdev->msix_count);
iwdev->iw_msixtbl = kzalloc(size, GFP_KERNEL);
if (!iwdev->iw_msixtbl)
props->cq_caps.max_cq_moderation_count = MLX4_MAX_CQ_COUNT;
props->cq_caps.max_cq_moderation_period = MLX4_MAX_CQ_PERIOD;
- if (!mlx4_is_slave(dev->dev))
- err = mlx4_get_internal_clock_params(dev->dev, &clock_params);
-
if (uhw->outlen >= resp.response_length + sizeof(resp.hca_core_clock_offset)) {
resp.response_length += sizeof(resp.hca_core_clock_offset);
- if (!err && !mlx4_is_slave(dev->dev)) {
+ if (!mlx4_get_internal_clock_params(dev->dev, &clock_params)) {
resp.comp_mask |= MLX4_IB_QUERY_DEV_RESP_MASK_CORE_CLOCK_OFFSET;
resp.hca_core_clock_offset = clock_params.offset % PAGE_SIZE;
}
struct mlx4_dev *dev = (to_mdev(qp->device))->dev;
int is_bonded = mlx4_is_bonded(dev);
- if (!rdma_is_port_valid(qp->device, flow_attr->port))
- return ERR_PTR(-EINVAL);
-
if (flow_attr->flags & ~IB_FLOW_ATTR_FLAGS_DONT_TRAP)
return ERR_PTR(-EOPNOTSUPP);
ib_umem_release(cq->buf.umem);
}
-static void init_cq_frag_buf(struct mlx5_ib_cq *cq,
- struct mlx5_ib_cq_buf *buf)
+static void init_cq_frag_buf(struct mlx5_ib_cq_buf *buf)
{
int i;
void *cqe;
struct mlx5_cqe64 *cqe64;
for (i = 0; i < buf->nent; i++) {
- cqe = get_cqe(cq, i);
+ cqe = mlx5_frag_buf_get_wqe(&buf->fbc, i);
cqe64 = buf->cqe_size == 64 ? cqe : cqe + 64;
cqe64->op_own = MLX5_CQE_INVALID << 4;
}
if (err)
goto err_db;
- init_cq_frag_buf(cq, &cq->buf);
+ init_cq_frag_buf(&cq->buf);
*inlen = MLX5_ST_SZ_BYTES(create_cq_in) +
MLX5_FLD_SZ_BYTES(create_cq_in, pas[0]) *
if (err)
goto ex;
- init_cq_frag_buf(cq, cq->resize_buf);
+ init_cq_frag_buf(cq->resize_buf);
return 0;
case UVERBS_OBJECT_QP:
{
struct mlx5_ib_qp *qp = to_mqp(uobj->object);
- enum ib_qp_type qp_type = qp->ibqp.qp_type;
- if (qp_type == IB_QPT_RAW_PACKET ||
+ if (qp->type == IB_QPT_RAW_PACKET ||
(qp->flags & IB_QP_CREATE_SOURCE_QPN)) {
struct mlx5_ib_raw_packet_qp *raw_packet_qp =
&qp->raw_packet_qp;
sq->tisn) == obj_id);
}
- if (qp_type == MLX5_IB_QPT_DCT)
+ if (qp->type == MLX5_IB_QPT_DCT)
return get_enc_obj_id(MLX5_CMD_OP_CREATE_DCT,
qp->dct.mdct.mqp.qpn) == obj_id;
-
return get_enc_obj_id(MLX5_CMD_OP_CREATE_QP,
qp->ibqp.qp_num) == obj_id;
}
if (err)
return err;
+ if (op >= BITS_PER_TYPE(u32))
+ return -EOPNOTSUPP;
+
if (!(MLX5_CAP_DEV_MEM(dev->mdev, memic_operations) & BIT(op)))
return -EOPNOTSUPP;
struct ib_umem *umem;
unsigned long user_virt;
int refcnt;
+ struct mm_struct *mm;
};
int mlx5_ib_db_map_user(struct mlx5_ib_ucontext *context,
mutex_lock(&context->db_page_mutex);
list_for_each_entry(page, &context->db_page_list, list)
- if (page->user_virt == (virt & PAGE_MASK))
+ if ((current->mm == page->mm) &&
+ (page->user_virt == (virt & PAGE_MASK)))
goto found;
page = kmalloc(sizeof(*page), GFP_KERNEL);
kfree(page);
goto out;
}
+ mmgrab(current->mm);
+ page->mm = current->mm;
list_add(&page->list, &context->db_page_list);
if (!--db->u.user_page->refcnt) {
list_del(&db->u.user_page->list);
+ mmdrop(db->u.user_page->mm);
ib_umem_release(db->u.user_page->umem);
kfree(db->u.user_page);
}
goto free_ucmd;
}
- if (flow_attr->port > dev->num_ports ||
- (flow_attr->flags &
- ~(IB_FLOW_ATTR_FLAGS_DONT_TRAP | IB_FLOW_ATTR_FLAGS_EGRESS))) {
+ if (flow_attr->flags &
+ ~(IB_FLOW_ATTR_FLAGS_DONT_TRAP | IB_FLOW_ATTR_FLAGS_EGRESS)) {
err = -EINVAL;
goto free_ucmd;
}
if (err)
goto end;
+ if (obj->ns_type == MLX5_FLOW_NAMESPACE_FDB &&
+ mlx5_eswitch_mode(dev->mdev) != MLX5_ESWITCH_OFFLOADS) {
+ err = -EINVAL;
+ goto end;
+ }
+
uobj->object = obj;
obj->mdev = dev->mdev;
atomic_set(&obj->usecnt, 0);
u8 ft_type, u8 dv_prt,
void *in, size_t len)
{
+ struct mlx5_pkt_reformat_params reformat_params;
enum mlx5_flow_namespace_type namespace;
u8 prm_prt;
int ret;
if (ret)
return ret;
+ memset(&reformat_params, 0, sizeof(reformat_params));
+ reformat_params.type = prm_prt;
+ reformat_params.size = len;
+ reformat_params.data = in;
maction->flow_action_raw.pkt_reformat =
- mlx5_packet_reformat_alloc(dev->mdev, prm_prt, len,
- in, namespace);
+ mlx5_packet_reformat_alloc(dev->mdev, &reformat_params,
+ namespace);
if (IS_ERR(maction->flow_action_raw.pkt_reformat)) {
ret = PTR_ERR(maction->flow_action_raw.pkt_reformat);
return ret;
if (bound) {
rdma_roce_rescan_device(&dev->ib_dev);
+ mpi->ibdev->ib_active = true;
break;
}
}
ent->xlt = (1 << ent->order) * sizeof(struct mlx5_mtt) /
MLX5_IB_UMR_OCTOWORD;
ent->access_mode = MLX5_MKC_ACCESS_MODE_MTT;
- if ((dev->mdev->profile->mask & MLX5_PROF_MASK_MR_CACHE) &&
+ if ((dev->mdev->profile.mask & MLX5_PROF_MASK_MR_CACHE) &&
!dev->is_rep && mlx5_core_is_pf(dev->mdev) &&
mlx5_ib_can_load_pas_with_umr(dev, 0))
- ent->limit = dev->mdev->profile->mr_cache[i].limit;
+ ent->limit = dev->mdev->profile.mr_cache[i].limit;
else
ent->limit = 0;
spin_lock_irq(&ent->lock);
mlx5r_deref_wait_odp_mkey(&mr->mmkey);
if (ibmr->type == IB_MR_TYPE_INTEGRITY) {
- xa_cmpxchg(&dev->sig_mrs, mlx5_base_mkey(mr->mmkey.key), ibmr,
- NULL, GFP_KERNEL);
+ xa_cmpxchg(&dev->sig_mrs, mlx5_base_mkey(mr->mmkey.key),
+ mr->sig, NULL, GFP_KERNEL);
if (mr->mtt_mr) {
rc = mlx5_ib_dereg_mr(&mr->mtt_mr->ibmr, NULL);
}
eq->irq_nb.notifier_call = mlx5_ib_eq_pf_int;
- param = (struct mlx5_eq_param){
- .irq_index = 0,
+ param = (struct mlx5_eq_param) {
.nent = MLX5_IB_NUM_PF_EQE,
};
param.mask[0] = 1ull << MLX5_EVENT_TYPE_PAGE_FAULT;
+ if (!zalloc_cpumask_var(¶m.affinity, GFP_KERNEL)) {
+ err = -ENOMEM;
+ goto err_wq;
+ }
eq->core = mlx5_eq_create_generic(dev->mdev, ¶m);
+ free_cpumask_var(param.affinity);
if (IS_ERR(eq->core)) {
err = PTR_ERR(eq->core);
goto err_wq;
ret = copy_data(qp->pd, IB_ACCESS_LOCAL_WRITE,
&wqe->dma, payload_addr(pkt),
payload_size(pkt), to_mr_obj, NULL);
- if (ret)
+ if (ret) {
+ wqe->status = IB_WC_LOC_PROT_ERR;
return COMPST_ERROR;
+ }
if (wqe->dma.resid == 0 && (pkt->mask & RXE_END_MASK))
return COMPST_COMP_ACK;
- else
- return COMPST_UPDATE_COMP;
+
+ return COMPST_UPDATE_COMP;
}
static inline enum comp_state do_atomic(struct rxe_qp *qp,
ret = copy_data(qp->pd, IB_ACCESS_LOCAL_WRITE,
&wqe->dma, &atomic_orig,
sizeof(u64), to_mr_obj, NULL);
- if (ret)
+ if (ret) {
+ wqe->status = IB_WC_LOC_PROT_ERR;
return COMPST_ERROR;
- else
- return COMPST_COMP_ACK;
+ }
+
+ return COMPST_COMP_ACK;
}
static void make_send_cqe(struct rxe_qp *qp, struct rxe_send_wqe *wqe,
if (err) {
vfree(qp->sq.queue->buf);
kfree(qp->sq.queue);
+ qp->sq.queue = NULL;
return err;
}
if (err) {
vfree(qp->rq.queue->buf);
kfree(qp->rq.queue);
+ qp->rq.queue = NULL;
return err;
}
}
err2:
rxe_queue_cleanup(qp->sq.queue);
err1:
+ qp->pd = NULL;
+ qp->rcq = NULL;
+ qp->scq = NULL;
+ qp->srq = NULL;
+
if (srq)
rxe_drop_ref(srq);
rxe_drop_ref(scq);
struct siw_ucontext *uctx =
rdma_udata_to_drv_context(udata, struct siw_ucontext,
base_ucontext);
- struct siw_cq *scq = NULL, *rcq = NULL;
unsigned long flags;
int num_sqe, num_rqe, rv = 0;
size_t length;
rv = -EINVAL;
goto err_out;
}
- scq = to_siw_cq(attrs->send_cq);
- rcq = to_siw_cq(attrs->recv_cq);
- if (!scq || (!rcq && !attrs->srq)) {
+ if (!attrs->send_cq || (!attrs->recv_cq && !attrs->srq)) {
siw_dbg(base_dev, "send CQ or receive CQ invalid\n");
rv = -EINVAL;
goto err_out;
else {
/* Zero sized SQ is not supported */
rv = -EINVAL;
- goto err_out;
+ goto err_out_xa;
}
if (num_rqe)
num_rqe = roundup_pow_of_two(num_rqe);
}
}
qp->pd = pd;
- qp->scq = scq;
- qp->rcq = rcq;
+ qp->scq = to_siw_cq(attrs->send_cq);
+ qp->rcq = to_siw_cq(attrs->recv_cq);
if (attrs->srq) {
/*
static struct rtnl_link_ops ipoib_link_ops __read_mostly = {
.kind = "ipoib",
+ .netns_refund = true,
.maxtype = IFLA_IPOIB_MAX,
.policy = ipoib_policy,
.priv_size = sizeof(struct ipoib_dev_priv),
// SPDX-License-Identifier: GPL-2.0
/*
- * Copyright (c) 2020, The Linux Foundation. All rights reserved.
+ * Copyright (c) 2020-2021, The Linux Foundation. All rights reserved.
*/
#include <asm/div64.h>
}
mutex_unlock(&bcm_voter_lock);
+ of_node_put(node);
return voter;
}
EXPORT_SYMBOL_GPL(of_bcm_voter_get);
{ .compatible = "qcom,bcm-voter" },
{ }
};
+MODULE_DEVICE_TABLE(of, bcm_voter_of_match);
static struct platform_driver qcom_icc_bcm_voter_driver = {
.probe = qcom_icc_bcm_voter_probe,
* The msb-bit must be clear on the address. Just set all the
* lower bits.
*/
- address |= 1ull << (msb_diff - 1);
+ address |= (1ull << msb_diff) - 1;
}
/* Clear bits 11:0 */
domain = iommu_get_domain_for_dev(dev);
if (domain->type == IOMMU_DOMAIN_DMA)
iommu_setup_dma_ops(dev, IOVA_START_PFN << PAGE_SHIFT, 0);
+ else
+ set_dma_ops(dev, NULL);
}
static void amd_iommu_release_device(struct device *dev)
err = iommu_device_register(&iommu->iommu, &intel_iommu_ops, NULL);
if (err)
- goto err_unmap;
+ goto err_sysfs;
}
drhd->iommu = iommu;
return 0;
+err_sysfs:
+ iommu_device_sysfs_remove(&iommu->iommu);
err_unmap:
unmap_iommu(iommu);
error_free_seq_id:
struct device *dev,
u32 pasid)
{
- int flags = PASID_FLAG_SUPERVISOR_MODE;
struct dma_pte *pgd = domain->pgd;
int agaw, level;
+ int flags = 0;
/*
* Skip top levels of page tables for iommu which has
if (level != 4 && level != 5)
return -EINVAL;
- flags |= (level == 5) ? PASID_FLAG_FL5LP : 0;
+ if (pasid != PASID_RID2PASID)
+ flags |= PASID_FLAG_SUPERVISOR_MODE;
+ if (level == 5)
+ flags |= PASID_FLAG_FL5LP;
if (domain->domain.type == IOMMU_DOMAIN_UNMANAGED)
flags |= PASID_FLAG_PAGE_SNOOP;
if (!sinfo) {
sinfo = kzalloc(sizeof(*sinfo), GFP_ATOMIC);
+ if (!sinfo)
+ return -ENOMEM;
sinfo->domain = domain;
sinfo->pdev = dev;
list_add(&sinfo->link_phys, &info->subdevices);
* Since it is a second level only translation setup, we should
* set SRE bit as well (addresses are expected to be GPAs).
*/
- pasid_set_sre(pte);
+ if (pasid != PASID_RID2PASID)
+ pasid_set_sre(pte);
pasid_set_present(pte);
pasid_flush_caches(iommu, pte, pasid, did);
{ VIRTIO_ID_IOMMU, VIRTIO_DEV_ANY_ID },
{ 0 },
};
+MODULE_DEVICE_TABLE(virtio, id_table);
static struct virtio_driver virtio_iommu_drv = {
.driver.name = KBUILD_MODNAME,
config APPLE_AIC
bool "Apple Interrupt Controller (AIC)"
depends on ARM64
- default ARCH_APPLE
+ depends on ARCH_APPLE || COMPILE_TEST
help
Support for the Apple Interrupt Controller found on Apple Silicon SoCs,
such as the M1.
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
icu->base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(icu->base)) {
- dev_err(&pdev->dev, "Failed to map icu base address.\n");
+ if (IS_ERR(icu->base))
return PTR_ERR(icu->base);
- }
/*
* Legacy bindings: ICU is one node with one MSI parent: force manually
sei->res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
sei->base = devm_ioremap_resource(sei->dev, sei->res);
- if (IS_ERR(sei->base)) {
- dev_err(sei->dev, "Failed to remap SEI resource\n");
+ if (IS_ERR(sei->base))
return PTR_ERR(sei->base);
- }
/* Retrieve the SEI capabilities with the interrupt ranges */
sei->caps = of_device_get_match_data(&pdev->dev);
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
host_data->base = devm_ioremap_resource(dev, res);
- if (IS_ERR(host_data->base)) {
- dev_err(dev, "Unable to map registers\n");
+ if (IS_ERR(host_data->base))
return PTR_ERR(host_data->base);
- }
for (i = 0; i < drv_data->bank_nr; i++)
stm32_exti_chip_init(host_data, i, np);
static void hfcsusb_stop_endpoint(struct hfcsusb *hw, int channel);
static int hfcsusb_setup_bch(struct bchannel *bch, int protocol);
static void deactivate_bchannel(struct bchannel *bch);
-static void hfcsusb_ph_info(struct hfcsusb *hw);
+static int hfcsusb_ph_info(struct hfcsusb *hw);
/* start next background transfer for control channel */
static void
* send full D/B channel status information
* as MPH_INFORMATION_IND
*/
-static void
+static int
hfcsusb_ph_info(struct hfcsusb *hw)
{
struct ph_info *phi;
phi = kzalloc(struct_size(phi, bch, dch->dev.nrbchan), GFP_ATOMIC);
if (!phi)
- return;
+ return -ENOMEM;
phi->dch.ch.protocol = hw->protocol;
phi->dch.ch.Flags = dch->Flags;
_queue_data(&dch->dev.D, MPH_INFORMATION_IND, MISDN_ID_ANY,
struct_size(phi, bch, dch->dev.nrbchan), phi, GFP_ATOMIC);
kfree(phi);
+
+ return 0;
}
/*
ret = l1_event(dch->l1, hh->prim);
break;
case MPH_INFORMATION_REQ:
- hfcsusb_ph_info(hw);
- ret = 0;
+ ret = hfcsusb_ph_info(hw);
break;
}
hw->name, __func__, cmd);
return -1;
}
- hfcsusb_ph_info(hw);
- return 0;
+ return hfcsusb_ph_info(hw);
}
static int
handle_led(hw, (bch->nr == 1) ? LED_B1_OFF :
LED_B2_OFF);
}
- hfcsusb_ph_info(hw);
- return 0;
+ return hfcsusb_ph_info(hw);
}
static void
release_io(struct inf_hw *hw)
{
if (hw->cfg.mode) {
- if (hw->cfg.p) {
+ if (hw->cfg.mode == AM_MEMIO) {
release_mem_region(hw->cfg.start, hw->cfg.size);
- iounmap(hw->cfg.p);
+ if (hw->cfg.p)
+ iounmap(hw->cfg.p);
} else
release_region(hw->cfg.start, hw->cfg.size);
hw->cfg.mode = AM_NONE;
}
if (hw->addr.mode) {
- if (hw->addr.p) {
+ if (hw->addr.mode == AM_MEMIO) {
release_mem_region(hw->addr.start, hw->addr.size);
- iounmap(hw->addr.p);
+ if (hw->addr.p)
+ iounmap(hw->addr.p);
} else
release_region(hw->addr.start, hw->addr.size);
hw->addr.mode = AM_NONE;
(ulong)hw->cfg.start, (ulong)hw->cfg.size);
return err;
}
- if (hw->ci->cfg_mode == AM_MEMIO)
- hw->cfg.p = ioremap(hw->cfg.start, hw->cfg.size);
hw->cfg.mode = hw->ci->cfg_mode;
+ if (hw->ci->cfg_mode == AM_MEMIO) {
+ hw->cfg.p = ioremap(hw->cfg.start, hw->cfg.size);
+ if (!hw->cfg.p)
+ return -ENOMEM;
+ }
if (debug & DEBUG_HW)
pr_notice("%s: IO cfg %lx (%lu bytes) mode%d\n",
hw->name, (ulong)hw->cfg.start,
(ulong)hw->addr.start, (ulong)hw->addr.size);
return err;
}
+ hw->addr.mode = hw->ci->addr_mode;
if (hw->ci->addr_mode == AM_MEMIO) {
hw->addr.p = ioremap(hw->addr.start, hw->addr.size);
- if (unlikely(!hw->addr.p))
+ if (!hw->addr.p)
return -ENOMEM;
}
- hw->addr.mode = hw->ci->addr_mode;
if (debug & DEBUG_HW)
pr_notice("%s: IO addr %lx (%lu bytes) mode%d\n",
hw->name, (ulong)hw->addr.start,
card->typ = NETJET_S_TJ300;
card->base = pci_resource_start(pdev, 0);
- card->irq = pdev->irq;
pci_set_drvdata(pdev, card);
err = setup_instance(card);
if (err)
#include "dsp.h"
#include "dsp_hwec.h"
-/* uncomment for debugging */
-/*#define PIPELINE_DEBUG*/
-
struct dsp_pipeline_entry {
struct mISDN_dsp_element *elem;
void *p;
}
}
-#ifdef PIPELINE_DEBUG
- printk(KERN_DEBUG "%s: %s registered\n", __func__, elem->name);
-#endif
-
return 0;
err2:
list_for_each_entry_safe(entry, n, &dsp_elements, list)
if (entry->elem == elem) {
device_unregister(&entry->dev);
-#ifdef PIPELINE_DEBUG
- printk(KERN_DEBUG "%s: %s unregistered\n",
- __func__, elem->name);
-#endif
return;
}
printk(KERN_ERR "%s: element %s not in list.\n", __func__, elem->name);
if (IS_ERR(elements_class))
return PTR_ERR(elements_class);
-#ifdef PIPELINE_DEBUG
- printk(KERN_DEBUG "%s: dsp pipeline module initialized\n", __func__);
-#endif
-
dsp_hwec_init();
return 0;
__func__, entry->elem->name);
kfree(entry);
}
-
-#ifdef PIPELINE_DEBUG
- printk(KERN_DEBUG "%s: dsp pipeline module exited\n", __func__);
-#endif
}
int dsp_pipeline_init(struct dsp_pipeline *pipeline)
INIT_LIST_HEAD(&pipeline->list);
-#ifdef PIPELINE_DEBUG
- printk(KERN_DEBUG "%s: dsp pipeline ready\n", __func__);
-#endif
-
return 0;
}
return;
_dsp_pipeline_destroy(pipeline);
-
-#ifdef PIPELINE_DEBUG
- printk(KERN_DEBUG "%s: dsp pipeline destroyed\n", __func__);
-#endif
}
int dsp_pipeline_build(struct dsp_pipeline *pipeline, const char *cfg)
{
- int incomplete = 0, found = 0;
+ int found = 0;
char *dup, *tok, *name, *args;
struct dsp_element_entry *entry, *n;
struct dsp_pipeline_entry *pipeline_entry;
printk(KERN_ERR "%s: failed to add "
"entry to pipeline: %s (out of "
"memory)\n", __func__, elem->name);
- incomplete = 1;
goto _out;
}
pipeline_entry->elem = elem;
if (pipeline_entry->p) {
list_add_tail(&pipeline_entry->
list, &pipeline->list);
-#ifdef PIPELINE_DEBUG
- printk(KERN_DEBUG "%s: created "
- "instance of %s%s%s\n",
- __func__, name, args ?
- " with args " : "", args ?
- args : "");
-#endif
} else {
printk(KERN_ERR "%s: failed "
"to add entry to pipeline: "
"%s (new() returned NULL)\n",
__func__, elem->name);
kfree(pipeline_entry);
- incomplete = 1;
}
}
found = 1;
if (found)
found = 0;
- else {
+ else
printk(KERN_ERR "%s: element not found, skipping: "
"%s\n", __func__, name);
- incomplete = 1;
- }
}
_out:
else
pipeline->inuse = 0;
-#ifdef PIPELINE_DEBUG
- printk(KERN_DEBUG "%s: dsp pipeline built%s: %s\n",
- __func__, incomplete ? " incomplete" : "", cfg);
-#endif
kfree(dup);
return 0;
}
usleep_range(3000, 6000);
ret = lp55xx_read(chip, LP5523_REG_STATUS, &status);
if (ret)
- return ret;
+ goto out;
status &= LP5523_ENG_STATUS_MASK;
if (status != LP5523_ENG_STATUS_MASK) {
/* The rest of this all shows up in sysfs */
unsigned int sequential_cutoff;
- unsigned int readahead;
unsigned int io_disable:1;
unsigned int verify:1;
struct bio *bio, unsigned int sectors)
{
int ret = MAP_CONTINUE;
- unsigned int reada = 0;
struct cached_dev *dc = container_of(s->d, struct cached_dev, disk);
struct bio *miss, *cache_bio;
+ unsigned int size_limit;
s->cache_missed = 1;
goto out_submit;
}
- if (!(bio->bi_opf & REQ_RAHEAD) &&
- !(bio->bi_opf & (REQ_META|REQ_PRIO)) &&
- s->iop.c->gc_stats.in_use < CUTOFF_CACHE_READA)
- reada = min_t(sector_t, dc->readahead >> 9,
- get_capacity(bio->bi_bdev->bd_disk) -
- bio_end_sector(bio));
-
- s->insert_bio_sectors = min(sectors, bio_sectors(bio) + reada);
+ /* Limitation for valid replace key size and cache_bio bvecs number */
+ size_limit = min_t(unsigned int, BIO_MAX_VECS * PAGE_SECTORS,
+ (1 << KEY_SIZE_BITS) - 1);
+ s->insert_bio_sectors = min3(size_limit, sectors, bio_sectors(bio));
s->iop.replace_key = KEY(s->iop.inode,
bio->bi_iter.bi_sector + s->insert_bio_sectors,
s->iop.replace = true;
- miss = bio_next_split(bio, sectors, GFP_NOIO, &s->d->bio_split);
+ miss = bio_next_split(bio, s->insert_bio_sectors, GFP_NOIO,
+ &s->d->bio_split);
/* btree_search_recurse()'s btree iterator is no good anymore */
ret = miss == bio ? MAP_DONE : -EINTR;
if (bch_bio_alloc_pages(cache_bio, __GFP_NOWARN|GFP_NOIO))
goto out_put;
- if (reada)
- bch_mark_cache_readahead(s->iop.c, s->d);
-
s->cache_miss = miss;
s->iop.bio = cache_bio;
bio_get(cache_bio);
read_attribute(cache_bypass_hits);
read_attribute(cache_bypass_misses);
read_attribute(cache_hit_ratio);
-read_attribute(cache_readaheads);
read_attribute(cache_miss_collisions);
read_attribute(bypassed);
DIV_SAFE(var(cache_hits) * 100,
var(cache_hits) + var(cache_misses)));
- var_print(cache_readaheads);
var_print(cache_miss_collisions);
sysfs_hprint(bypassed, var(sectors_bypassed) << 9);
#undef var
&sysfs_cache_bypass_hits,
&sysfs_cache_bypass_misses,
&sysfs_cache_hit_ratio,
- &sysfs_cache_readaheads,
&sysfs_cache_miss_collisions,
&sysfs_bypassed,
NULL
acc->total.cache_misses = 0;
acc->total.cache_bypass_hits = 0;
acc->total.cache_bypass_misses = 0;
- acc->total.cache_readaheads = 0;
acc->total.cache_miss_collisions = 0;
acc->total.sectors_bypassed = 0;
}
scale_stat(&stats->cache_misses);
scale_stat(&stats->cache_bypass_hits);
scale_stat(&stats->cache_bypass_misses);
- scale_stat(&stats->cache_readaheads);
scale_stat(&stats->cache_miss_collisions);
scale_stat(&stats->sectors_bypassed);
}
move_stat(cache_misses);
move_stat(cache_bypass_hits);
move_stat(cache_bypass_misses);
- move_stat(cache_readaheads);
move_stat(cache_miss_collisions);
move_stat(sectors_bypassed);
mark_cache_stats(&c->accounting.collector, hit, bypass);
}
-void bch_mark_cache_readahead(struct cache_set *c, struct bcache_device *d)
-{
- struct cached_dev *dc = container_of(d, struct cached_dev, disk);
-
- atomic_inc(&dc->accounting.collector.cache_readaheads);
- atomic_inc(&c->accounting.collector.cache_readaheads);
-}
-
void bch_mark_cache_miss_collision(struct cache_set *c, struct bcache_device *d)
{
struct cached_dev *dc = container_of(d, struct cached_dev, disk);
atomic_t cache_misses;
atomic_t cache_bypass_hits;
atomic_t cache_bypass_misses;
- atomic_t cache_readaheads;
atomic_t cache_miss_collisions;
atomic_t sectors_bypassed;
};
rw_attribute(discard);
rw_attribute(running);
rw_attribute(label);
-rw_attribute(readahead);
rw_attribute(errors);
rw_attribute(io_error_limit);
rw_attribute(io_error_halflife);
var_printf(partial_stripes_expensive, "%u");
var_hprint(sequential_cutoff);
- var_hprint(readahead);
sysfs_print(running, atomic_read(&dc->running));
sysfs_print(state, states[BDEV_STATE(&dc->sb)]);
sysfs_strtoul_clamp(sequential_cutoff,
dc->sequential_cutoff,
0, UINT_MAX);
- d_strtoi_h(readahead);
if (attr == &sysfs_clear_stats)
bch_cache_accounting_clear(&dc->accounting);
&sysfs_running,
&sysfs_state,
&sysfs_label,
- &sysfs_readahead,
#ifdef CONFIG_BCACHE_DEBUG
&sysfs_verify,
&sysfs_bypass_torture_test,
__u8 magic[8];
__u8 version;
__u8 log2_interleave_sectors;
- __u16 integrity_tag_size;
- __u32 journal_sections;
- __u64 provided_data_sectors; /* userspace uses this value */
- __u32 flags;
+ __le16 integrity_tag_size;
+ __le32 journal_sections;
+ __le64 provided_data_sectors; /* userspace uses this value */
+ __le32 flags;
__u8 log2_sectors_per_block;
__u8 log2_blocks_per_bitmap_bit;
__u8 pad[2];
- __u64 recalc_sector;
+ __le64 recalc_sector;
__u8 pad2[8];
__u8 salt[SALT_SIZE];
};
#define JOURNAL_ENTRY_ROUNDUP 8
-typedef __u64 commit_id_t;
+typedef __le64 commit_id_t;
#define JOURNAL_MAC_PER_SECTOR 8
struct journal_entry {
union {
struct {
- __u32 sector_lo;
- __u32 sector_hi;
+ __le32 sector_lo;
+ __le32 sector_hi;
} s;
- __u64 sector;
+ __le64 sector;
} u;
commit_id_t last_bytes[];
/* __u8 tag[0]; */
}
if (ic->sb->flags & cpu_to_le32(SB_FLAG_FIXED_HMAC)) {
- uint64_t section_le;
+ __le64 section_le;
r = crypto_shash_update(desc, (__u8 *)&ic->sb->salt, SALT_SIZE);
if (unlikely(r < 0)) {
static void integrity_sector_checksum(struct dm_integrity_c *ic, sector_t sector,
const char *data, char *result)
{
- __u64 sector_le = cpu_to_le64(sector);
+ __le64 sector_le = cpu_to_le64(sector);
SHASH_DESC_ON_STACK(req, ic->internal_hash);
int r;
unsigned digest_size;
if (unlikely(dm_integrity_failed(ic)))
goto err;
- if (!ic->discard) {
- io_req.bi_op = REQ_OP_READ;
- io_req.bi_op_flags = 0;
- io_req.mem.type = DM_IO_VMA;
- io_req.mem.ptr.addr = ic->recalc_buffer;
- io_req.notify.fn = NULL;
- io_req.client = ic->io;
- io_loc.bdev = ic->dev->bdev;
- io_loc.sector = get_data_sector(ic, area, offset);
- io_loc.count = n_sectors;
+ io_req.bi_op = REQ_OP_READ;
+ io_req.bi_op_flags = 0;
+ io_req.mem.type = DM_IO_VMA;
+ io_req.mem.ptr.addr = ic->recalc_buffer;
+ io_req.notify.fn = NULL;
+ io_req.client = ic->io;
+ io_loc.bdev = ic->dev->bdev;
+ io_loc.sector = get_data_sector(ic, area, offset);
+ io_loc.count = n_sectors;
- r = dm_io(&io_req, 1, &io_loc, NULL);
- if (unlikely(r)) {
- dm_integrity_io_error(ic, "reading data", r);
- goto err;
- }
+ r = dm_io(&io_req, 1, &io_loc, NULL);
+ if (unlikely(r)) {
+ dm_integrity_io_error(ic, "reading data", r);
+ goto err;
+ }
- t = ic->recalc_tags;
- for (i = 0; i < n_sectors; i += ic->sectors_per_block) {
- integrity_sector_checksum(ic, logical_sector + i, ic->recalc_buffer + (i << SECTOR_SHIFT), t);
- t += ic->tag_size;
- }
- } else {
- t = ic->recalc_tags + (n_sectors >> ic->sb->log2_sectors_per_block) * ic->tag_size;
+ t = ic->recalc_tags;
+ for (i = 0; i < n_sectors; i += ic->sectors_per_block) {
+ integrity_sector_checksum(ic, logical_sector + i, ic->recalc_buffer + (i << SECTOR_SHIFT), t);
+ t += ic->tag_size;
}
metadata_block = get_metadata_sector_and_offset(ic, area, offset, &metadata_offset);
for (i = 0; i < ic->journal_sections; i++) {
struct scatterlist sg;
struct skcipher_request *section_req;
- __u32 section_le = cpu_to_le32(i);
+ __le32 section_le = cpu_to_le32(i);
memset(crypt_iv, 0x00, ivsize);
memset(crypt_data, 0x00, crypt_len);
goto bad;
}
INIT_WORK(&ic->recalc_work, integrity_recalc);
- if (!ic->discard) {
- ic->recalc_buffer = vmalloc(RECALC_SECTORS << SECTOR_SHIFT);
- if (!ic->recalc_buffer) {
- ti->error = "Cannot allocate buffer for recalculating";
- r = -ENOMEM;
- goto bad;
- }
+ ic->recalc_buffer = vmalloc(RECALC_SECTORS << SECTOR_SHIFT);
+ if (!ic->recalc_buffer) {
+ ti->error = "Cannot allocate buffer for recalculating";
+ r = -ENOMEM;
+ goto bad;
}
ic->recalc_tags = kvmalloc_array(RECALC_SECTORS >> ic->sb->log2_sectors_per_block,
ic->tag_size, GFP_KERNEL);
r = -ENOMEM;
goto bad;
}
- if (ic->discard)
- memset(ic->recalc_tags, DISCARD_FILLER,
- (RECALC_SECTORS >> ic->sb->log2_sectors_per_block) * ic->tag_size);
} else {
if (ic->sb->flags & cpu_to_le32(SB_FLAG_RECALCULATING)) {
ti->error = "Recalculate can only be specified with internal_hash";
static struct target_type integrity_target = {
.name = "integrity",
- .version = {1, 9, 0},
+ .version = {1, 10, 0},
.module = THIS_MODULE,
.features = DM_TARGET_SINGLETON | DM_TARGET_INTEGRITY,
.ctr = dm_integrity_ctr,
static uint32_t __minimum_chunk_size(struct origin *o)
{
struct dm_snapshot *snap;
- unsigned chunk_size = 0;
+ unsigned chunk_size = rounddown_pow_of_two(UINT_MAX);
if (o)
list_for_each_entry(snap, &o->snapshots, list)
if (!s->store->chunk_size) {
ti->error = "Chunk size not set";
+ r = -EINVAL;
goto bad_read_metadata;
}
#define DM_VERITY_VERIFY_ERR(s) DM_VERITY_ROOT_HASH_VERIFICATION " " s
static bool require_signatures;
-module_param(require_signatures, bool, false);
+module_param(require_signatures, bool, 0444);
MODULE_PARM_DESC(require_signatures,
"Verify the roothash of dm-verity hash tree");
unsigned int chunk_sectors;
unsigned int bio_sectors = bio_sectors(bio);
- WARN_ON_ONCE(bio->bi_bdev->bd_partno);
-
chunk_sectors = min(conf->chunk_sectors, conf->prev_chunk_sectors);
return chunk_sectors >=
((sector & (chunk_sectors - 1)) + bio_sectors);
// read status reg in order to clear pending irqs
err = sp8870_readreg(state, 0x200);
- if (err)
+ if (err < 0)
return err;
// system controller start
{
struct rcar_drif_sdr *sdr = video_drvdata(file);
- memset(f->fmt.sdr.reserved, 0, sizeof(f->fmt.sdr.reserved));
f->fmt.sdr.pixelformat = sdr->fmt->pixelformat;
f->fmt.sdr.buffersize = sdr->fmt->buffersize;
{
struct sd *sd = (struct sd *) gspca_dev;
struct cam *cam;
- int ret;
sd->mainsFreq = FREQ_DEF == V4L2_CID_POWER_LINE_FREQUENCY_60HZ;
reset_camera_params(gspca_dev);
cam->cam_mode = mode;
cam->nmodes = ARRAY_SIZE(mode);
- ret = goto_low_power(gspca_dev);
- if (ret)
- gspca_err(gspca_dev, "Cannot go to low power mode: %d\n",
- ret);
+ goto_low_power(gspca_dev);
/* Check the firmware version. */
sd->params.version.firmwareVersion = 0;
get_version_information(gspca_dev);
int mt9m111_probe(struct sd *sd)
{
u8 data[2] = {0x00, 0x00};
- int i, rc = 0;
+ int i, err;
struct gspca_dev *gspca_dev = (struct gspca_dev *)sd;
if (force_sensor) {
/* Do the preinit */
for (i = 0; i < ARRAY_SIZE(preinit_mt9m111); i++) {
if (preinit_mt9m111[i][0] == BRIDGE) {
- rc |= m5602_write_bridge(sd,
- preinit_mt9m111[i][1],
- preinit_mt9m111[i][2]);
+ err = m5602_write_bridge(sd,
+ preinit_mt9m111[i][1],
+ preinit_mt9m111[i][2]);
} else {
data[0] = preinit_mt9m111[i][2];
data[1] = preinit_mt9m111[i][3];
- rc |= m5602_write_sensor(sd,
- preinit_mt9m111[i][1], data, 2);
+ err = m5602_write_sensor(sd,
+ preinit_mt9m111[i][1], data, 2);
}
+ if (err < 0)
+ return err;
}
- if (rc < 0)
- return rc;
if (m5602_read_sensor(sd, MT9M111_SC_CHIPVER, data, 2))
return -ENODEV;
int po1030_probe(struct sd *sd)
{
- int rc = 0;
u8 dev_id_h = 0, i;
+ int err;
struct gspca_dev *gspca_dev = (struct gspca_dev *)sd;
if (force_sensor) {
for (i = 0; i < ARRAY_SIZE(preinit_po1030); i++) {
u8 data = preinit_po1030[i][2];
if (preinit_po1030[i][0] == SENSOR)
- rc |= m5602_write_sensor(sd,
- preinit_po1030[i][1], &data, 1);
+ err = m5602_write_sensor(sd, preinit_po1030[i][1],
+ &data, 1);
else
- rc |= m5602_write_bridge(sd, preinit_po1030[i][1],
- data);
+ err = m5602_write_bridge(sd, preinit_po1030[i][1],
+ data);
+ if (err < 0)
+ return err;
}
- if (rc < 0)
- return rc;
if (m5602_read_sensor(sd, PO1030_DEVID_H, &dev_id_h, 1))
return -ENODEV;
pcr->sd30_drive_sel_1v8 = DRIVER_TYPE_B;
pcr->sd30_drive_sel_3v3 = DRIVER_TYPE_D;
pcr->aspm_en = ASPM_L1_EN;
+ pcr->aspm_mode = ASPM_MODE_CFG;
pcr->tx_initial_phase = SET_CLOCK_PHASE(23, 7, 14);
pcr->rx_initial_phase = SET_CLOCK_PHASE(4, 3, 10);
pcr->ic_version = rtl8411_get_ic_version(pcr);
pcr->sd30_drive_sel_1v8 = DRIVER_TYPE_B;
pcr->sd30_drive_sel_3v3 = DRIVER_TYPE_D;
pcr->aspm_en = ASPM_L1_EN;
+ pcr->aspm_mode = ASPM_MODE_CFG;
pcr->tx_initial_phase = SET_CLOCK_PHASE(27, 27, 16);
pcr->rx_initial_phase = SET_CLOCK_PHASE(24, 6, 5);
pcr->sd30_drive_sel_1v8 = CFG_DRIVER_TYPE_B;
pcr->sd30_drive_sel_3v3 = CFG_DRIVER_TYPE_B;
pcr->aspm_en = ASPM_L1_EN;
+ pcr->aspm_mode = ASPM_MODE_CFG;
pcr->tx_initial_phase = SET_CLOCK_PHASE(27, 27, 15);
pcr->rx_initial_phase = SET_CLOCK_PHASE(30, 7, 7);
rts5227_init_params(pcr);
pcr->ops = &rts522a_pcr_ops;
+ pcr->aspm_mode = ASPM_MODE_REG;
pcr->tx_initial_phase = SET_CLOCK_PHASE(20, 20, 11);
pcr->reg_pm_ctrl3 = RTS522A_PM_CTRL3;
pcr->sd30_drive_sel_1v8 = CFG_DRIVER_TYPE_B;
pcr->sd30_drive_sel_3v3 = CFG_DRIVER_TYPE_B;
pcr->aspm_en = ASPM_L1_EN;
+ pcr->aspm_mode = ASPM_MODE_REG;
pcr->tx_initial_phase = SET_CLOCK_PHASE(28, 27, 11);
pcr->rx_initial_phase = SET_CLOCK_PHASE(24, 6, 5);
pcr->sd30_drive_sel_1v8 = DRIVER_TYPE_B;
pcr->sd30_drive_sel_3v3 = DRIVER_TYPE_D;
pcr->aspm_en = ASPM_L1_EN;
+ pcr->aspm_mode = ASPM_MODE_CFG;
pcr->tx_initial_phase = SET_CLOCK_PHASE(27, 27, 15);
pcr->rx_initial_phase = SET_CLOCK_PHASE(30, 6, 6);
pcr->sd30_drive_sel_1v8 = CFG_DRIVER_TYPE_B;
pcr->sd30_drive_sel_3v3 = CFG_DRIVER_TYPE_B;
pcr->aspm_en = ASPM_L1_EN;
+ pcr->aspm_mode = ASPM_MODE_CFG;
pcr->tx_initial_phase = SET_CLOCK_PHASE(1, 29, 16);
pcr->rx_initial_phase = SET_CLOCK_PHASE(24, 6, 5);
void rts524a_init_params(struct rtsx_pcr *pcr)
{
rts5249_init_params(pcr);
+ pcr->aspm_mode = ASPM_MODE_REG;
pcr->tx_initial_phase = SET_CLOCK_PHASE(27, 29, 11);
pcr->option.ltr_l1off_sspwrgate = LTR_L1OFF_SSPWRGATE_5250_DEF;
pcr->option.ltr_l1off_snooze_sspwrgate =
void rts525a_init_params(struct rtsx_pcr *pcr)
{
rts5249_init_params(pcr);
+ pcr->aspm_mode = ASPM_MODE_REG;
pcr->tx_initial_phase = SET_CLOCK_PHASE(25, 29, 11);
pcr->option.ltr_l1off_sspwrgate = LTR_L1OFF_SSPWRGATE_5250_DEF;
pcr->option.ltr_l1off_snooze_sspwrgate =
pcr->sd30_drive_sel_1v8 = CFG_DRIVER_TYPE_B;
pcr->sd30_drive_sel_3v3 = CFG_DRIVER_TYPE_B;
pcr->aspm_en = ASPM_L1_EN;
+ pcr->aspm_mode = ASPM_MODE_REG;
pcr->tx_initial_phase = SET_CLOCK_PHASE(27, 29, 11);
pcr->rx_initial_phase = SET_CLOCK_PHASE(24, 6, 5);
pcr->sd30_drive_sel_1v8 = 0x00;
pcr->sd30_drive_sel_3v3 = 0x00;
pcr->aspm_en = ASPM_L1_EN;
+ pcr->aspm_mode = ASPM_MODE_REG;
pcr->tx_initial_phase = SET_CLOCK_PHASE(27, 27, 11);
pcr->rx_initial_phase = SET_CLOCK_PHASE(24, 6, 5);
if (pcr->aspm_enabled == enable)
return;
- if (pcr->aspm_en & 0x02)
- rtsx_pci_write_register(pcr, ASPM_FORCE_CTL, FORCE_ASPM_CTL0 |
- FORCE_ASPM_CTL1, enable ? 0 : FORCE_ASPM_CTL0 | FORCE_ASPM_CTL1);
- else
- rtsx_pci_write_register(pcr, ASPM_FORCE_CTL, FORCE_ASPM_CTL0 |
- FORCE_ASPM_CTL1, FORCE_ASPM_CTL0 | FORCE_ASPM_CTL1);
+ if (pcr->aspm_mode == ASPM_MODE_CFG) {
+ pcie_capability_clear_and_set_word(pcr->pci, PCI_EXP_LNKCTL,
+ PCI_EXP_LNKCTL_ASPMC,
+ enable ? pcr->aspm_en : 0);
+ } else if (pcr->aspm_mode == ASPM_MODE_REG) {
+ if (pcr->aspm_en & 0x02)
+ rtsx_pci_write_register(pcr, ASPM_FORCE_CTL, FORCE_ASPM_CTL0 |
+ FORCE_ASPM_CTL1, enable ? 0 : FORCE_ASPM_CTL0 | FORCE_ASPM_CTL1);
+ else
+ rtsx_pci_write_register(pcr, ASPM_FORCE_CTL, FORCE_ASPM_CTL0 |
+ FORCE_ASPM_CTL1, FORCE_ASPM_CTL0 | FORCE_ASPM_CTL1);
+ }
if (!enable && (pcr->aspm_en & 0x02))
mdelay(10);
return err;
}
- rtsx_pci_write_register(pcr, ASPM_FORCE_CTL, 0x30, 0x30);
+ if (pcr->aspm_mode == ASPM_MODE_REG)
+ rtsx_pci_write_register(pcr, ASPM_FORCE_CTL, 0x30, 0x30);
/* No CD interrupt if probing driver with card inserted.
* So we need to initialize pcr->card_exist here.
static int rtsx_pci_init_chip(struct rtsx_pcr *pcr)
{
int err;
+ u16 cfg_val;
+ u8 val;
spin_lock_init(&pcr->lock);
mutex_init(&pcr->pcr_mutex);
if (!pcr->slots)
return -ENOMEM;
+ if (pcr->aspm_mode == ASPM_MODE_CFG) {
+ pcie_capability_read_word(pcr->pci, PCI_EXP_LNKCTL, &cfg_val);
+ if (cfg_val & PCI_EXP_LNKCTL_ASPM_L1)
+ pcr->aspm_enabled = true;
+ else
+ pcr->aspm_enabled = false;
+
+ } else if (pcr->aspm_mode == ASPM_MODE_REG) {
+ rtsx_pci_read_register(pcr, ASPM_FORCE_CTL, &val);
+ if (val & FORCE_ASPM_CTL0 && val & FORCE_ASPM_CTL1)
+ pcr->aspm_enabled = false;
+ else
+ pcr->aspm_enabled = true;
+ }
+
if (pcr->ops->fetch_vendor_settings)
pcr->ops->fetch_vendor_settings(pcr);
struct pcr_handle *handle;
u32 base, len;
int ret, i, bar = 0;
- u8 val;
dev_dbg(&(pcidev->dev),
": Realtek PCI-E Card Reader found at %s [%04x:%04x] (rev %x)\n",
pcr->host_cmds_addr = pcr->rtsx_resv_buf_addr;
pcr->host_sg_tbl_ptr = pcr->rtsx_resv_buf + HOST_CMDS_BUF_LEN;
pcr->host_sg_tbl_addr = pcr->rtsx_resv_buf_addr + HOST_CMDS_BUF_LEN;
- rtsx_pci_read_register(pcr, ASPM_FORCE_CTL, &val);
- if (val & FORCE_ASPM_CTL0 && val & FORCE_ASPM_CTL1)
- pcr->aspm_enabled = false;
- else
- pcr->aspm_enabled = true;
pcr->card_inserted = 0;
pcr->card_removed = 0;
INIT_DELAYED_WORK(&pcr->carddet_work, rtsx_pci_card_detect);
at24->nvmem = devm_nvmem_register(dev, &nvmem_config);
if (IS_ERR(at24->nvmem)) {
pm_runtime_disable(dev);
- regulator_disable(at24->vcc_reg);
+ if (!pm_runtime_status_suspended(dev))
+ regulator_disable(at24->vcc_reg);
return PTR_ERR(at24->nvmem);
}
err = at24_read(at24, 0, &test_byte, 1);
if (err) {
pm_runtime_disable(dev);
- regulator_disable(at24->vcc_reg);
+ if (!pm_runtime_status_suspended(dev))
+ regulator_disable(at24->vcc_reg);
return -ENODEV;
}
if (completion_value >= target_value) {
*status = CS_WAIT_STATUS_COMPLETED;
} else {
- timeout -= jiffies_to_usecs(completion_rc);
+ timeout = completion_rc;
goto wait_again;
}
} else {
}
if (err_val & CPU_BOOT_ERR0_SECURITY_NOT_RDY) {
- dev_warn(hdev->dev,
+ dev_err(hdev->dev,
"Device boot warning - security not ready\n");
- /* This is a warning so we don't want it to disable the
- * device
- */
- err_val &= ~CPU_BOOT_ERR0_SECURITY_NOT_RDY;
+ err_exists = true;
}
if (err_val & CPU_BOOT_ERR0_SECURITY_FAIL) {
err_exists = true;
}
- if (err_exists)
+ if (err_exists && ((err_val & ~CPU_BOOT_ERR0_ENABLED) &
+ lower_32_bits(hdev->boot_error_status_mask)))
return -EIO;
return 0;
return rc;
}
-int get_used_pll_index(struct hl_device *hdev, enum pll_index input_pll_index,
+int get_used_pll_index(struct hl_device *hdev, u32 input_pll_index,
enum pll_index *pll_index)
{
struct asic_fixed_properties *prop = &hdev->asic_prop;
u8 pll_byte, pll_bit_off;
bool dynamic_pll;
-
- if (input_pll_index >= PLL_MAX) {
- dev_err(hdev->dev, "PLL index %d is out of range\n",
- input_pll_index);
- return -EINVAL;
- }
+ int fw_pll_idx;
dynamic_pll = prop->fw_security_status_valid &&
(prop->fw_app_security_map & CPU_BOOT_DEV_STS0_DYN_PLL_EN);
if (!dynamic_pll) {
/*
* in case we are working with legacy FW (each asic has unique
- * PLL numbering) extract the legacy numbering
+ * PLL numbering) use the driver based index as they are
+ * aligned with fw legacy numbering
*/
- *pll_index = hdev->legacy_pll_map[input_pll_index];
+ *pll_index = input_pll_index;
return 0;
}
+ /* retrieve a FW compatible PLL index based on
+ * ASIC specific user request
+ */
+ fw_pll_idx = hdev->asic_funcs->map_pll_idx_to_fw_idx(input_pll_index);
+ if (fw_pll_idx < 0) {
+ dev_err(hdev->dev, "Invalid PLL index (%u) error %d\n",
+ input_pll_index, fw_pll_idx);
+ return -EINVAL;
+ }
+
/* PLL map is a u8 array */
- pll_byte = prop->cpucp_info.pll_map[input_pll_index >> 3];
- pll_bit_off = input_pll_index & 0x7;
+ pll_byte = prop->cpucp_info.pll_map[fw_pll_idx >> 3];
+ pll_bit_off = fw_pll_idx & 0x7;
if (!(pll_byte & BIT(pll_bit_off))) {
dev_err(hdev->dev, "PLL index %d is not supported\n",
- input_pll_index);
+ fw_pll_idx);
return -EINVAL;
}
- *pll_index = input_pll_index;
+ *pll_index = fw_pll_idx;
return 0;
}
-int hl_fw_cpucp_pll_info_get(struct hl_device *hdev, enum pll_index pll_index,
+int hl_fw_cpucp_pll_info_get(struct hl_device *hdev, u32 pll_index,
u16 *pll_freq_arr)
{
struct cpucp_packet pkt;
if (rc) {
dev_err(hdev->dev, "Failed to read preboot version\n");
detect_cpu_boot_status(hdev, status);
- fw_read_errors(hdev, boot_err0_reg,
- cpu_security_boot_status_reg);
+
+ /* If we read all FF, then something is totally wrong, no point
+ * of reading specific errors
+ */
+ if (status != -1)
+ fw_read_errors(hdev, boot_err0_reg,
+ cpu_security_boot_status_reg);
return -EIO;
}
* driver is ready to receive asynchronous events. This
* function should be called during the first init and
* after every hard-reset of the device
+ * @get_msi_info: Retrieve asic-specific MSI ID of the f/w async event
+ * @map_pll_idx_to_fw_idx: convert driver specific per asic PLL index to
+ * generic f/w compatible PLL Indexes
*/
struct hl_asic_funcs {
int (*early_init)(struct hl_device *hdev);
u32 block_id, u32 block_size);
void (*enable_events_from_fw)(struct hl_device *hdev);
void (*get_msi_info)(u32 *table);
+ int (*map_pll_idx_to_fw_idx)(u32 pll_idx);
};
* @aggregated_cs_counters: aggregated cs counters among all contexts
* @mmu_priv: device-specific MMU data.
* @mmu_func: device-related MMU functions.
- * @legacy_pll_map: map holding map between dynamic (common) PLL indexes and
- * static (asic specific) PLL indexes.
* @dram_used_mem: current DRAM memory consumption.
* @timeout_jiffies: device CS timeout value.
* @max_power: the max power of the device, as configured by the sysadmin. This
* @clock_gating_mask: is clock gating enabled. bitmask that represents the
* different engines. See debugfs-driver-habanalabs for
* details.
+ * @boot_error_status_mask: contains a mask of the device boot error status.
+ * Each bit represents a different error, according to
+ * the defines in hl_boot_if.h. If the bit is cleared,
+ * the error will be ignored by the driver during
+ * device initialization. Mainly used to debug and
+ * workaround firmware bugs
* @in_reset: is device in reset flow.
* @curr_pll_profile: current PLL profile.
* @card_type: Various ASICs have several card types. This indicates the card
struct hl_mmu_priv mmu_priv;
struct hl_mmu_funcs mmu_func[MMU_NUM_PGT_LOCATIONS];
- enum pll_index *legacy_pll_map;
-
atomic64_t dram_used_mem;
u64 timeout_jiffies;
u64 max_power;
u64 clock_gating_mask;
+ u64 boot_error_status_mask;
atomic_t in_reset;
enum hl_pll_frequency curr_pll_profile;
enum cpucp_card_types card_type;
struct hl_info_pci_counters *counters);
int hl_fw_cpucp_total_energy_get(struct hl_device *hdev,
u64 *total_energy);
-int get_used_pll_index(struct hl_device *hdev, enum pll_index input_pll_index,
+int get_used_pll_index(struct hl_device *hdev, u32 input_pll_index,
enum pll_index *pll_index);
-int hl_fw_cpucp_pll_info_get(struct hl_device *hdev, enum pll_index pll_index,
+int hl_fw_cpucp_pll_info_get(struct hl_device *hdev, u32 pll_index,
u16 *pll_freq_arr);
int hl_fw_cpucp_power_get(struct hl_device *hdev, u64 *power);
int hl_fw_init_cpu(struct hl_device *hdev, u32 cpu_boot_status_reg,
int hl_pci_init(struct hl_device *hdev);
void hl_pci_fini(struct hl_device *hdev);
-long hl_get_frequency(struct hl_device *hdev, enum pll_index pll_index,
+long hl_get_frequency(struct hl_device *hdev, u32 pll_index,
bool curr);
-void hl_set_frequency(struct hl_device *hdev, enum pll_index pll_index,
+void hl_set_frequency(struct hl_device *hdev, u32 pll_index,
u64 freq);
int hl_get_temperature(struct hl_device *hdev,
int sensor_index, u32 attr, long *value);
static int timeout_locked = 30;
static int reset_on_lockup = 1;
static int memory_scrub = 1;
+static ulong boot_error_status_mask = ULONG_MAX;
module_param(timeout_locked, int, 0444);
MODULE_PARM_DESC(timeout_locked,
MODULE_PARM_DESC(memory_scrub,
"Scrub device memory in various states (0 = no, 1 = yes, default yes)");
+module_param(boot_error_status_mask, ulong, 0444);
+MODULE_PARM_DESC(boot_error_status_mask,
+ "Mask of the error status during device CPU boot (If bitX is cleared then error X is masked. Default all 1's)");
+
#define PCI_VENDOR_ID_HABANALABS 0x1da3
#define PCI_IDS_GOYA 0x0001
hdev->major = hl_major;
hdev->reset_on_lockup = reset_on_lockup;
hdev->memory_scrub = memory_scrub;
+ hdev->boot_error_status_mask = boot_error_status_mask;
+
hdev->pldm = 0;
set_driver_behavior_per_device(hdev);
#include <linux/pci.h>
-long hl_get_frequency(struct hl_device *hdev, enum pll_index pll_index,
+long hl_get_frequency(struct hl_device *hdev, u32 pll_index,
bool curr)
{
struct cpucp_packet pkt;
return (long) result;
}
-void hl_set_frequency(struct hl_device *hdev, enum pll_index pll_index,
+void hl_set_frequency(struct hl_device *hdev, u32 pll_index,
u64 freq)
{
struct cpucp_packet pkt;
#define GAUDI_PLL_MAX 10
-/*
- * this enum kept here for compatibility with old FW (in which each asic has
- * unique PLL numbering
- */
-enum gaudi_pll_index {
- GAUDI_CPU_PLL = 0,
- GAUDI_PCI_PLL,
- GAUDI_SRAM_PLL,
- GAUDI_HBM_PLL,
- GAUDI_NIC_PLL,
- GAUDI_DMA_PLL,
- GAUDI_MESH_PLL,
- GAUDI_MME_PLL,
- GAUDI_TPC_PLL,
- GAUDI_IF_PLL,
-};
-
-static enum pll_index gaudi_pll_map[PLL_MAX] = {
- [CPU_PLL] = GAUDI_CPU_PLL,
- [PCI_PLL] = GAUDI_PCI_PLL,
- [SRAM_PLL] = GAUDI_SRAM_PLL,
- [HBM_PLL] = GAUDI_HBM_PLL,
- [NIC_PLL] = GAUDI_NIC_PLL,
- [DMA_PLL] = GAUDI_DMA_PLL,
- [MESH_PLL] = GAUDI_MESH_PLL,
- [MME_PLL] = GAUDI_MME_PLL,
- [TPC_PLL] = GAUDI_TPC_PLL,
- [IF_PLL] = GAUDI_IF_PLL,
-};
-
static const char gaudi_irq_name[GAUDI_MSI_ENTRIES][GAUDI_MAX_STRING_LEN] = {
"gaudi cq 0_0", "gaudi cq 0_1", "gaudi cq 0_2", "gaudi cq 0_3",
"gaudi cq 1_0", "gaudi cq 1_1", "gaudi cq 1_2", "gaudi cq 1_3",
freq = 0;
}
} else {
- rc = hl_fw_cpucp_pll_info_get(hdev, CPU_PLL, pll_freq_arr);
+ rc = hl_fw_cpucp_pll_info_get(hdev, HL_GAUDI_CPU_PLL, pll_freq_arr);
if (rc)
return rc;
hdev->asic_specific = gaudi;
- /* store legacy PLL map */
- hdev->legacy_pll_map = gaudi_pll_map;
-
/* Create DMA pool for small allocations */
hdev->dma_pool = dma_pool_create(dev_name(hdev->dev),
&hdev->pdev->dev, GAUDI_DMA_POOL_BLK_SIZE, 8, 0);
struct hl_cs_job *job;
u32 cb_size, ctl, err_cause;
struct hl_cb *cb;
+ u64 id;
int rc;
cb = hl_cb_kernel_create(hdev, PAGE_SIZE, false);
}
release_cb:
+ id = cb->id;
hl_cb_put(cb);
- hl_cb_destroy(hdev, &hdev->kernel_cb_mgr, cb->id << PAGE_SHIFT);
+ hl_cb_destroy(hdev, &hdev->kernel_cb_mgr, id << PAGE_SHIFT);
return rc;
}
WREG32(mmGIC_DISTRIBUTOR__5_GICD_SETSPI_NSR, GAUDI_EVENT_INTS_REGISTER);
}
+static int gaudi_map_pll_idx_to_fw_idx(u32 pll_idx)
+{
+ switch (pll_idx) {
+ case HL_GAUDI_CPU_PLL: return CPU_PLL;
+ case HL_GAUDI_PCI_PLL: return PCI_PLL;
+ case HL_GAUDI_NIC_PLL: return NIC_PLL;
+ case HL_GAUDI_DMA_PLL: return DMA_PLL;
+ case HL_GAUDI_MESH_PLL: return MESH_PLL;
+ case HL_GAUDI_MME_PLL: return MME_PLL;
+ case HL_GAUDI_TPC_PLL: return TPC_PLL;
+ case HL_GAUDI_IF_PLL: return IF_PLL;
+ case HL_GAUDI_SRAM_PLL: return SRAM_PLL;
+ case HL_GAUDI_HBM_PLL: return HBM_PLL;
+ default: return -EINVAL;
+ }
+}
+
static const struct hl_asic_funcs gaudi_funcs = {
.early_init = gaudi_early_init,
.early_fini = gaudi_early_fini,
.ack_protection_bits_errors = gaudi_ack_protection_bits_errors,
.get_hw_block_id = gaudi_get_hw_block_id,
.hw_block_mmap = gaudi_block_mmap,
- .enable_events_from_fw = gaudi_enable_events_from_fw
+ .enable_events_from_fw = gaudi_enable_events_from_fw,
+ .map_pll_idx_to_fw_idx = gaudi_map_pll_idx_to_fw_idx
};
/**
struct gaudi_device *gaudi = hdev->asic_specific;
if (freq == PLL_LAST)
- hl_set_frequency(hdev, MME_PLL, gaudi->max_freq_value);
+ hl_set_frequency(hdev, HL_GAUDI_MME_PLL, gaudi->max_freq_value);
}
int gaudi_get_clk_rate(struct hl_device *hdev, u32 *cur_clk, u32 *max_clk)
if (!hl_device_operational(hdev, NULL))
return -ENODEV;
- value = hl_get_frequency(hdev, MME_PLL, false);
+ value = hl_get_frequency(hdev, HL_GAUDI_MME_PLL, false);
if (value < 0) {
dev_err(hdev->dev, "Failed to retrieve device max clock %ld\n",
*max_clk = (value / 1000 / 1000);
- value = hl_get_frequency(hdev, MME_PLL, true);
+ value = hl_get_frequency(hdev, HL_GAUDI_MME_PLL, true);
if (value < 0) {
dev_err(hdev->dev,
if (!hl_device_operational(hdev, NULL))
return -ENODEV;
- value = hl_get_frequency(hdev, MME_PLL, false);
+ value = hl_get_frequency(hdev, HL_GAUDI_MME_PLL, false);
gaudi->max_freq_value = value;
gaudi->max_freq_value = value * 1000 * 1000;
- hl_set_frequency(hdev, MME_PLL, gaudi->max_freq_value);
+ hl_set_frequency(hdev, HL_GAUDI_MME_PLL, gaudi->max_freq_value);
fail:
return count;
if (!hl_device_operational(hdev, NULL))
return -ENODEV;
- value = hl_get_frequency(hdev, MME_PLL, true);
+ value = hl_get_frequency(hdev, HL_GAUDI_MME_PLL, true);
return sprintf(buf, "%lu\n", (value / 1000 / 1000));
}
#define IS_MME_IDLE(mme_arch_sts) \
(((mme_arch_sts) & MME_ARCH_IDLE_MASK) == MME_ARCH_IDLE_MASK)
-/*
- * this enum kept here for compatibility with old FW (in which each asic has
- * unique PLL numbering
- */
-enum goya_pll_index {
- GOYA_CPU_PLL = 0,
- GOYA_IC_PLL,
- GOYA_MC_PLL,
- GOYA_MME_PLL,
- GOYA_PCI_PLL,
- GOYA_EMMC_PLL,
- GOYA_TPC_PLL,
-};
-
-static enum pll_index goya_pll_map[PLL_MAX] = {
- [CPU_PLL] = GOYA_CPU_PLL,
- [IC_PLL] = GOYA_IC_PLL,
- [MC_PLL] = GOYA_MC_PLL,
- [MME_PLL] = GOYA_MME_PLL,
- [PCI_PLL] = GOYA_PCI_PLL,
- [EMMC_PLL] = GOYA_EMMC_PLL,
- [TPC_PLL] = GOYA_TPC_PLL,
-};
-
static const char goya_irq_name[GOYA_MSIX_ENTRIES][GOYA_MAX_STRING_LEN] = {
"goya cq 0", "goya cq 1", "goya cq 2", "goya cq 3",
"goya cq 4", "goya cpu eq"
freq = 0;
}
} else {
- rc = hl_fw_cpucp_pll_info_get(hdev, PCI_PLL, pll_freq_arr);
+ rc = hl_fw_cpucp_pll_info_get(hdev, HL_GOYA_PCI_PLL,
+ pll_freq_arr);
if (rc)
return;
hdev->asic_specific = goya;
- /* store legacy PLL map */
- hdev->legacy_pll_map = goya_pll_map;
-
/* Create DMA pool for small allocations */
hdev->dma_pool = dma_pool_create(dev_name(hdev->dev),
&hdev->pdev->dev, GOYA_DMA_POOL_BLK_SIZE, 8, 0);
GOYA_ASYNC_EVENT_ID_INTS_REGISTER);
}
+static int goya_map_pll_idx_to_fw_idx(u32 pll_idx)
+{
+ switch (pll_idx) {
+ case HL_GOYA_CPU_PLL: return CPU_PLL;
+ case HL_GOYA_PCI_PLL: return PCI_PLL;
+ case HL_GOYA_MME_PLL: return MME_PLL;
+ case HL_GOYA_TPC_PLL: return TPC_PLL;
+ case HL_GOYA_IC_PLL: return IC_PLL;
+ case HL_GOYA_MC_PLL: return MC_PLL;
+ case HL_GOYA_EMMC_PLL: return EMMC_PLL;
+ default: return -EINVAL;
+ }
+}
+
static const struct hl_asic_funcs goya_funcs = {
.early_init = goya_early_init,
.early_fini = goya_early_fini,
.ack_protection_bits_errors = goya_ack_protection_bits_errors,
.get_hw_block_id = goya_get_hw_block_id,
.hw_block_mmap = goya_block_mmap,
- .enable_events_from_fw = goya_enable_events_from_fw
+ .enable_events_from_fw = goya_enable_events_from_fw,
+ .map_pll_idx_to_fw_idx = goya_map_pll_idx_to_fw_idx
};
/*
switch (freq) {
case PLL_HIGH:
- hl_set_frequency(hdev, MME_PLL, hdev->high_pll);
- hl_set_frequency(hdev, TPC_PLL, hdev->high_pll);
- hl_set_frequency(hdev, IC_PLL, hdev->high_pll);
+ hl_set_frequency(hdev, HL_GOYA_MME_PLL, hdev->high_pll);
+ hl_set_frequency(hdev, HL_GOYA_TPC_PLL, hdev->high_pll);
+ hl_set_frequency(hdev, HL_GOYA_IC_PLL, hdev->high_pll);
break;
case PLL_LOW:
- hl_set_frequency(hdev, MME_PLL, GOYA_PLL_FREQ_LOW);
- hl_set_frequency(hdev, TPC_PLL, GOYA_PLL_FREQ_LOW);
- hl_set_frequency(hdev, IC_PLL, GOYA_PLL_FREQ_LOW);
+ hl_set_frequency(hdev, HL_GOYA_MME_PLL, GOYA_PLL_FREQ_LOW);
+ hl_set_frequency(hdev, HL_GOYA_TPC_PLL, GOYA_PLL_FREQ_LOW);
+ hl_set_frequency(hdev, HL_GOYA_IC_PLL, GOYA_PLL_FREQ_LOW);
break;
case PLL_LAST:
- hl_set_frequency(hdev, MME_PLL, goya->mme_clk);
- hl_set_frequency(hdev, TPC_PLL, goya->tpc_clk);
- hl_set_frequency(hdev, IC_PLL, goya->ic_clk);
+ hl_set_frequency(hdev, HL_GOYA_MME_PLL, goya->mme_clk);
+ hl_set_frequency(hdev, HL_GOYA_TPC_PLL, goya->tpc_clk);
+ hl_set_frequency(hdev, HL_GOYA_IC_PLL, goya->ic_clk);
break;
default:
dev_err(hdev->dev, "unknown frequency setting\n");
if (!hl_device_operational(hdev, NULL))
return -ENODEV;
- value = hl_get_frequency(hdev, MME_PLL, false);
+ value = hl_get_frequency(hdev, HL_GOYA_MME_PLL, false);
if (value < 0) {
dev_err(hdev->dev, "Failed to retrieve device max clock %ld\n",
*max_clk = (value / 1000 / 1000);
- value = hl_get_frequency(hdev, MME_PLL, true);
+ value = hl_get_frequency(hdev, HL_GOYA_MME_PLL, true);
if (value < 0) {
dev_err(hdev->dev,
if (!hl_device_operational(hdev, NULL))
return -ENODEV;
- value = hl_get_frequency(hdev, MME_PLL, false);
+ value = hl_get_frequency(hdev, HL_GOYA_MME_PLL, false);
if (value < 0)
return value;
goto fail;
}
- hl_set_frequency(hdev, MME_PLL, value);
+ hl_set_frequency(hdev, HL_GOYA_MME_PLL, value);
goya->mme_clk = value;
fail:
if (!hl_device_operational(hdev, NULL))
return -ENODEV;
- value = hl_get_frequency(hdev, TPC_PLL, false);
+ value = hl_get_frequency(hdev, HL_GOYA_TPC_PLL, false);
if (value < 0)
return value;
goto fail;
}
- hl_set_frequency(hdev, TPC_PLL, value);
+ hl_set_frequency(hdev, HL_GOYA_TPC_PLL, value);
goya->tpc_clk = value;
fail:
if (!hl_device_operational(hdev, NULL))
return -ENODEV;
- value = hl_get_frequency(hdev, IC_PLL, false);
+ value = hl_get_frequency(hdev, HL_GOYA_IC_PLL, false);
if (value < 0)
return value;
goto fail;
}
- hl_set_frequency(hdev, IC_PLL, value);
+ hl_set_frequency(hdev, HL_GOYA_IC_PLL, value);
goya->ic_clk = value;
fail:
if (!hl_device_operational(hdev, NULL))
return -ENODEV;
- value = hl_get_frequency(hdev, MME_PLL, true);
+ value = hl_get_frequency(hdev, HL_GOYA_MME_PLL, true);
if (value < 0)
return value;
if (!hl_device_operational(hdev, NULL))
return -ENODEV;
- value = hl_get_frequency(hdev, TPC_PLL, true);
+ value = hl_get_frequency(hdev, HL_GOYA_TPC_PLL, true);
if (value < 0)
return value;
if (!hl_device_operational(hdev, NULL))
return -ENODEV;
- value = hl_get_frequency(hdev, IC_PLL, true);
+ value = hl_get_frequency(hdev, HL_GOYA_IC_PLL, true);
if (value < 0)
return value;
for (i = 0; i < NUM_MIRRORED_REGS; i++) {
temp = i2c_smbus_read_word_data(client, regs_to_copy[i]);
if (temp < 0)
- data->regs[regs_to_copy[i]] = 0;
+ temp = 0;
data->regs[regs_to_copy[i]] = temp >> 8;
}
printk(KERN_INFO a); \
} while (0)
#define v2printk(a...) do { \
- if (verbose > 1) \
+ if (verbose > 1) { \
printk(KERN_INFO a); \
+ } \
touch_nmi_watchdog(); \
} while (0)
#define eprintk(a...) do { \
int regs_size;
u8 *reg_cache;
bool regs_stored;
+ bool init_required;
u8 odr_mask; /* ODR bit mask */
u8 whoami; /* indicates measurement precision */
s16 (*read_data) (struct lis3lv02d *lis3, int reg);
return ret;
}
+ pm_runtime_mark_last_busy(dev->dev);
+ pm_request_autosuspend(dev->dev);
+
list_move_tail(&cb->list, &cl->rd_pending);
return 0;
if (host->dram_access_quirk)
return;
- if (data->blocks > 1) {
+ /* SD_IO_RW_EXTENDED (CMD53) can also use block mode under the hood */
+ if (data->blocks > 1 || mrq->cmd->opcode == SD_IO_RW_EXTENDED) {
/*
* In block mode DMA descriptor format, "length" field indicates
* number of blocks and there is no way to pass DMA size that
for_each_sg(data->sg, sg, data->sg_len, i) {
/* check for 8 byte alignment */
if (sg->offset % 8) {
- WARN_ONCE(1, "unaligned scatterlist buffer\n");
+ dev_warn_once(mmc_dev(mmc),
+ "unaligned sg offset %u, disabling descriptor DMA for transfer\n",
+ sg->offset);
return;
}
}
/* Issue CMD19 twice for each tap */
for (i = 0; i < 2 * priv->tap_num; i++) {
+ int cmd_error;
+
/* Set sampling clock position */
sd_scc_write32(host, priv, SH_MOBILE_SDHI_SCC_TAPSET, i % priv->tap_num);
- if (mmc_send_tuning(mmc, opcode, NULL) == 0)
+ if (mmc_send_tuning(mmc, opcode, &cmd_error) == 0)
set_bit(i, priv->taps);
if (sd_scc_read32(host, priv, SH_MOBILE_SDHI_SCC_SMPCMP) == 0)
set_bit(i, priv->smpcmp);
+
+ if (cmd_error)
+ mmc_abort_tuning(mmc, opcode);
}
ret = renesas_sdhi_select_tuning(host);
{ .soc_id = "r8a7795", .revision = "ES3.*", .data = &sdhi_quirks_bad_taps2367 },
{ .soc_id = "r8a7796", .revision = "ES1.[012]", .data = &sdhi_quirks_4tap_nohs400 },
{ .soc_id = "r8a7796", .revision = "ES1.*", .data = &sdhi_quirks_r8a7796_es13 },
- { .soc_id = "r8a7796", .revision = "ES3.*", .data = &sdhi_quirks_bad_taps1357 },
+ { .soc_id = "r8a77961", .data = &sdhi_quirks_bad_taps1357 },
{ .soc_id = "r8a77965", .data = &sdhi_quirks_r8a77965 },
{ .soc_id = "r8a77980", .data = &sdhi_quirks_nohs400 },
{ .soc_id = "r8a77990", .data = &sdhi_quirks_r8a77990 },
*
* Wait 5ms after set 1.8V signal enable in Host Control 2 register
* to ensure 1.8V signal enable bit is set by GL9750/GL9755.
+ *
+ * ...however, the controller in the NUC10i3FNK4 (a 9755) requires
+ * slightly longer than 5ms before the control register reports that
+ * 1.8V is ready, and far longer still before the card will actually
+ * work reliably.
*/
- usleep_range(5000, 5500);
+ usleep_range(100000, 110000);
}
static void sdhci_gl9750_reset(struct sdhci_host *host, u8 mask)
#include <linux/module.h>
#include <linux/delay.h>
#include <linux/mtd/mtd.h>
+#include <linux/mtd/nand-ecc-sw-hamming.h>
#include <linux/mtd/rawnand.h>
#include <linux/mtd/partitions.h>
#include <linux/iopoll.h>
return 0;
}
+static int cs553x_ecc_correct(struct nand_chip *chip,
+ unsigned char *buf,
+ unsigned char *read_ecc,
+ unsigned char *calc_ecc)
+{
+ return ecc_sw_hamming_correct(buf, read_ecc, calc_ecc,
+ chip->ecc.size, false);
+}
+
static struct cs553x_nand_controller *controllers[4];
static int cs553x_attach_chip(struct nand_chip *chip)
chip->ecc.bytes = 3;
chip->ecc.hwctl = cs_enable_hwecc;
chip->ecc.calculate = cs_calculate_ecc;
- chip->ecc.correct = rawnand_sw_hamming_correct;
+ chip->ecc.correct = cs553x_ecc_correct;
chip->ecc.strength = 1;
return 0;
#include <linux/sched.h>
#include <linux/types.h>
#include <linux/mtd/mtd.h>
+#include <linux/mtd/nand-ecc-sw-hamming.h>
#include <linux/mtd/rawnand.h>
#include <linux/platform_device.h>
#include <linux/of.h>
return 0;
}
+static int fsmc_correct_ecc1(struct nand_chip *chip,
+ unsigned char *buf,
+ unsigned char *read_ecc,
+ unsigned char *calc_ecc)
+{
+ return ecc_sw_hamming_correct(buf, read_ecc, calc_ecc,
+ chip->ecc.size, false);
+}
+
/* Count the number of 0's in buff upto a max of max_bits */
static int count_written_bits(u8 *buff, int size, int max_bits)
{
case NAND_ECC_ENGINE_TYPE_ON_HOST:
dev_info(host->dev, "Using 1-bit HW ECC scheme\n");
nand->ecc.calculate = fsmc_read_hwecc_ecc1;
- nand->ecc.correct = rawnand_sw_hamming_correct;
+ nand->ecc.correct = fsmc_correct_ecc1;
nand->ecc.hwctl = fsmc_enable_hwecc;
nand->ecc.bytes = 3;
nand->ecc.strength = 1;
#include <linux/of.h>
#include <linux/of_gpio.h>
#include <linux/mtd/lpc32xx_slc.h>
+#include <linux/mtd/nand-ecc-sw-hamming.h>
#define LPC32XX_MODNAME "lpc32xx-nand"
}
/*
+ * Corrects the data
+ */
+static int lpc32xx_nand_ecc_correct(struct nand_chip *chip,
+ unsigned char *buf,
+ unsigned char *read_ecc,
+ unsigned char *calc_ecc)
+{
+ return ecc_sw_hamming_correct(buf, read_ecc, calc_ecc,
+ chip->ecc.size, false);
+}
+
+/*
* Read a single byte from NAND device
*/
static uint8_t lpc32xx_nand_read_byte(struct nand_chip *chip)
chip->ecc.write_oob = lpc32xx_nand_write_oob_syndrome;
chip->ecc.read_oob = lpc32xx_nand_read_oob_syndrome;
chip->ecc.calculate = lpc32xx_nand_ecc_calculate;
- chip->ecc.correct = rawnand_sw_hamming_correct;
+ chip->ecc.correct = lpc32xx_nand_ecc_correct;
chip->ecc.hwctl = lpc32xx_nand_ecc_enable;
/*
#include <linux/mtd/ndfc.h>
#include <linux/slab.h>
#include <linux/mtd/mtd.h>
+#include <linux/mtd/nand-ecc-sw-hamming.h>
#include <linux/of_address.h>
#include <linux/of_platform.h>
#include <asm/io.h>
return 0;
}
+static int ndfc_correct_ecc(struct nand_chip *chip,
+ unsigned char *buf,
+ unsigned char *read_ecc,
+ unsigned char *calc_ecc)
+{
+ return ecc_sw_hamming_correct(buf, read_ecc, calc_ecc,
+ chip->ecc.size, false);
+}
+
/*
* Speedups for buffer read/write/verify
*
chip->controller = &ndfc->ndfc_control;
chip->legacy.read_buf = ndfc_read_buf;
chip->legacy.write_buf = ndfc_write_buf;
- chip->ecc.correct = rawnand_sw_hamming_correct;
+ chip->ecc.correct = ndfc_correct_ecc;
chip->ecc.hwctl = ndfc_enable_hwecc;
chip->ecc.calculate = ndfc_calculate_ecc;
chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_ON_HOST;
#include <linux/module.h>
#include <linux/delay.h>
#include <linux/mtd/mtd.h>
+#include <linux/mtd/nand-ecc-sw-hamming.h>
#include <linux/mtd/rawnand.h>
#include <linux/mtd/partitions.h>
#include <linux/mtd/sharpsl.h>
return readb(sharpsl->io + ECCCNTR) != 0;
}
+static int sharpsl_nand_correct_ecc(struct nand_chip *chip,
+ unsigned char *buf,
+ unsigned char *read_ecc,
+ unsigned char *calc_ecc)
+{
+ return ecc_sw_hamming_correct(buf, read_ecc, calc_ecc,
+ chip->ecc.size, false);
+}
+
static int sharpsl_attach_chip(struct nand_chip *chip)
{
if (chip->ecc.engine_type != NAND_ECC_ENGINE_TYPE_ON_HOST)
chip->ecc.strength = 1;
chip->ecc.hwctl = sharpsl_nand_enable_hwecc;
chip->ecc.calculate = sharpsl_nand_calculate_ecc;
- chip->ecc.correct = rawnand_sw_hamming_correct;
+ chip->ecc.correct = sharpsl_nand_correct_ecc;
return 0;
}
#include <linux/interrupt.h>
#include <linux/ioport.h>
#include <linux/mtd/mtd.h>
+#include <linux/mtd/nand-ecc-sw-hamming.h>
#include <linux/mtd/rawnand.h>
#include <linux/mtd/partitions.h>
#include <linux/slab.h>
int r0, r1;
/* assume ecc.size = 512 and ecc.bytes = 6 */
- r0 = rawnand_sw_hamming_correct(chip, buf, read_ecc, calc_ecc);
+ r0 = ecc_sw_hamming_correct(buf, read_ecc, calc_ecc,
+ chip->ecc.size, false);
if (r0 < 0)
return r0;
- r1 = rawnand_sw_hamming_correct(chip, buf + 256, read_ecc + 3,
- calc_ecc + 3);
+ r1 = ecc_sw_hamming_correct(buf + 256, read_ecc + 3, calc_ecc + 3,
+ chip->ecc.size, false);
if (r1 < 0)
return r1;
return r0 + r1;
#include <linux/platform_device.h>
#include <linux/delay.h>
#include <linux/mtd/mtd.h>
+#include <linux/mtd/nand-ecc-sw-hamming.h>
#include <linux/mtd/rawnand.h>
#include <linux/mtd/partitions.h>
#include <linux/io.h>
int stat;
for (eccsize = chip->ecc.size; eccsize > 0; eccsize -= 256) {
- stat = rawnand_sw_hamming_correct(chip, buf, read_ecc,
- calc_ecc);
+ stat = ecc_sw_hamming_correct(buf, read_ecc, calc_ecc,
+ chip->ecc.size, false);
if (stat < 0)
return stat;
corrected += stat;
if (!mtd_node)
return 0;
- ofpart_node = of_get_child_by_name(mtd_node, "partitions");
- if (!ofpart_node && !master->parent) {
- /*
- * We might get here even when ofpart isn't used at all (e.g.,
- * when using another parser), so don't be louder than
- * KERN_DEBUG
- */
- pr_debug("%s: 'partitions' subnode not found on %pOF. Trying to parse direct subnodes as partitions.\n",
- master->name, mtd_node);
+ if (!master->parent) { /* Master */
+ ofpart_node = of_get_child_by_name(mtd_node, "partitions");
+ if (!ofpart_node) {
+ /*
+ * We might get here even when ofpart isn't used at all (e.g.,
+ * when using another parser), so don't be louder than
+ * KERN_DEBUG
+ */
+ pr_debug("%s: 'partitions' subnode not found on %pOF. Trying to parse direct subnodes as partitions.\n",
+ master->name, mtd_node);
+ ofpart_node = mtd_node;
+ dedicated = false;
+ }
+ } else { /* Partition */
ofpart_node = mtd_node;
- dedicated = false;
}
- if (!ofpart_node)
- return 0;
of_id = of_match_node(parse_ofpart_match_table, ofpart_node);
if (dedicated && !of_id) {
will be called geneve.
config BAREUDP
- tristate "Bare UDP Encapsulation"
- depends on INET
- depends on IPV6 || !IPV6
- select NET_UDP_TUNNEL
- select GRO_CELLS
- help
- This adds a bare UDP tunnel module for tunnelling different
- kinds of traffic like MPLS, IP, etc. inside a UDP tunnel.
-
- To compile this driver as a module, choose M here: the module
- will be called bareudp.
+ tristate "Bare UDP Encapsulation"
+ depends on INET
+ depends on IPV6 || !IPV6
+ select NET_UDP_TUNNEL
+ select GRO_CELLS
+ help
+ This adds a bare UDP tunnel module for tunnelling different
+ kinds of traffic like MPLS, IP, etc. inside a UDP tunnel.
+
+ To compile this driver as a module, choose M here: the module
+ will be called bareudp.
config GTP
tristate "GPRS Tunneling Protocol datapath (GTP-U)"
config MHI_NET
tristate "MHI network driver"
depends on MHI_BUS
+ select WWAN
help
This is the network driver for MHI bus. It can be used with
QCOM based WWAN modems (like SDX55). Say Y or M.
break;
}
+ dev->base_addr = ioaddr;
+
/* Reserve any actual interrupt. */
if (dev->irq) {
retval = request_irq(dev->irq, cops_interrupt, 0, dev->name, dev);
goto err_out;
}
- dev->base_addr = ioaddr;
-
lp = netdev_priv(dev);
spin_lock_init(&lp->lock);
if(lp->board == DAYNA)
{
- /* Empty any pending adapter responses. */
+ /* Empty any pending adapter responses. */
while((inb(ioaddr+DAYNA_CARD_STATUS)&DAYNA_TX_READY)==0)
{
outb(0, ioaddr+COPS_CLEAR_INT); /* Clear interrupts. */
- if((inb(ioaddr+DAYNA_CARD_STATUS)&0x03)==DAYNA_RX_REQUEST)
- cops_rx(dev); /* Kick any packets waiting. */
+ if((inb(ioaddr+DAYNA_CARD_STATUS)&0x03)==DAYNA_RX_REQUEST)
+ cops_rx(dev); /* Kick any packets waiting. */
schedule();
}
while(inb(ioaddr+TANG_CARD_STATUS)&TANG_RX_READY)
{
outb(0, ioaddr+COPS_CLEAR_INT); /* Clear interrupt. */
- cops_rx(dev); /* Kick out packets waiting. */
+ cops_rx(dev); /* Kick out packets waiting. */
schedule();
}
/* Not sure what Tangent does if nodeid picked is used. */
if(nodeid == 0) /* Seed. */
- nodeid = jiffies&0xFF; /* Get a random try */
+ nodeid = jiffies&0xFF; /* Get a random try */
outb(2, ioaddr); /* Command length LSB */
outb(0, ioaddr); /* Command length MSB */
outb(LAP_INIT, ioaddr); /* Send LAP_INIT byte */
if(lp->board == DAYNA)
{
- if((inb(ioaddr+DAYNA_CARD_STATUS)&0x03)==DAYNA_RX_REQUEST)
- cops_rx(dev); /* Grab the nodeid put in lp->node_acquire. */
+ if((inb(ioaddr+DAYNA_CARD_STATUS)&0x03)==DAYNA_RX_REQUEST)
+ cops_rx(dev); /* Grab the nodeid put in lp->node_acquire. */
}
if(lp->board == TANGENT)
{
if(inb(ioaddr+TANG_CARD_STATUS)&TANG_RX_READY)
- cops_rx(dev); /* Grab the nodeid put in lp->node_acquire. */
+ cops_rx(dev); /* Grab the nodeid put in lp->node_acquire. */
}
schedule();
}
{
do {
outb(0, ioaddr + COPS_CLEAR_INT);
- status=inb(ioaddr+DAYNA_CARD_STATUS);
- if((status&0x03)==DAYNA_RX_REQUEST)
- cops_rx(dev);
- netif_wake_queue(dev);
+ status=inb(ioaddr+DAYNA_CARD_STATUS);
+ if((status&0x03)==DAYNA_RX_REQUEST)
+ cops_rx(dev);
+ netif_wake_queue(dev);
} while(++boguscount < 20);
}
else
{
do {
- status=inb(ioaddr+TANG_CARD_STATUS);
+ status=inb(ioaddr+TANG_CARD_STATUS);
if(status & TANG_RX_READY)
cops_rx(dev);
if(status & TANG_TX_READY)
if(lp->board==TANGENT)
{
if((inb(ioaddr+TANG_CARD_STATUS)&TANG_TX_READY)==0)
- printk(KERN_WARNING "%s: No TX complete interrupt.\n", dev->name);
+ printk(KERN_WARNING "%s: No TX complete interrupt.\n", dev->name);
}
printk(KERN_WARNING "%s: Transmit timed out.\n", dev->name);
cops_jumpstart(dev); /* Restart the card. */
outb(LAP_WRITE, ioaddr);
if(lp->board == DAYNA) /* Check the transmit buffer again. */
- while((inb(ioaddr+DAYNA_CARD_STATUS)&DAYNA_TX_READY)==0);
+ while((inb(ioaddr+DAYNA_CARD_STATUS)&DAYNA_TX_READY)==0);
outsb(ioaddr, skb->data, skb->len); /* Send out the data. */
printk("%02x ",ltdmacbuf[i]);
printk("\n");
}
+
handlecommand(dev);
- if(0xfa==inb_p(base+6)) {
- /* we timed out, so return */
- goto done;
- }
+
+ if (0xfa == inb_p(base + 6)) {
+ /* we timed out, so return */
+ goto done;
+ }
} else {
/* we don't seem to have a command */
if (!mboxinuse[0]) {
static int __init ltpc_probe_dma(int base, int dma)
{
int want = (dma == 3) ? 2 : (dma == 1) ? 1 : 3;
- unsigned long timeout;
- unsigned long f;
+ unsigned long timeout;
+ unsigned long f;
- if (want & 1) {
+ if (want & 1) {
if (request_dma(1,"ltpc")) {
want &= ~1;
} else {
skb->dev = bareudp->dev;
oiph = skb_network_header(skb);
skb_reset_network_header(skb);
+ skb_reset_mac_header(skb);
if (!IS_ENABLED(CONFIG_IPV6) || family == AF_INET)
err = IP_ECN_decapsulate(oiph, skb);
index = SLAVE_TLB_INFO(slave).head;
while (index != TLB_NULL_INDEX) {
u32 next_index = tx_hash_table[index].next;
+
tlb_init_table_entry(&tx_hash_table[index], save_load);
index = next_index;
}
{
struct slave *tx_slave;
- /* We don't need to disable softirq here, becase
+ /* We don't need to disable softirq here, because
* tlb_choose_channel() is only called by bond_alb_xmit()
* which already has softirq disabled.
*/
client_info->ip_src = arp->ip_src;
client_info->ip_dst = arp->ip_dst;
- /* arp->mac_dst is broadcast for arp reqeusts.
+ /* arp->mac_dst is broadcast for arp requests.
* will be updated with clients actual unicast mac address
* upon receiving an arp reply.
*/
if (!client_info->assigned) {
u32 prev_tbl_head = bond_info->rx_hashtbl_used_head;
+
bond_info->rx_hashtbl_used_head = hash_index;
client_info->used_next = prev_tbl_head;
if (prev_tbl_head != RLB_NULL_INDEX) {
while (index != RLB_NULL_INDEX) {
struct rlb_client_info *entry = &(bond_info->rx_hashtbl[index]);
u32 next_index = entry->src_next;
+
if (entry->ip_src == arp->ip_src &&
!ether_addr_equal_64bits(arp->mac_src, entry->mac_src))
- rlb_delete_table_entry(bond, index);
+ rlb_delete_table_entry(bond, index);
index = next_index;
}
spin_unlock_bh(&bond->mode_lock);
return res;
}
-/************************ exported alb funcions ************************/
+/************************ exported alb functions ************************/
int bond_alb_initialize(struct bonding *bond, int rlb_enabled)
{
bond_for_each_slave_rcu(bond, slave, iter) {
/* If updating current_active, use all currently
- * user mac addreses (!strict_match). Otherwise, only
+ * user mac addresses (!strict_match). Otherwise, only
* use mac of the slave device.
* In RLB mode, we always use strict matches.
*/
{
bonding_debug_root = debugfs_create_dir("bonding", NULL);
- if (!bonding_debug_root) {
+ if (!bonding_debug_root)
pr_warn("Warning: Cannot create bonding directory in debugfs\n");
- }
}
void bond_destroy_debugfs(void)
*/
/* Yes, the mii is overlaid on the ifreq.ifr_ifru */
- strncpy(ifr.ifr_name, slave_dev->name, IFNAMSIZ);
+ strscpy_pad(ifr.ifr_name, slave_dev->name, IFNAMSIZ);
mii = if_mii(&ifr);
if (ioctl(slave_dev, &ifr, SIOCGMIIPHY) == 0) {
mii->reg_num = MII_BMSR;
if (bond_is_lb(bond))
bond_alb_handle_link_change(bond, new_active, BOND_LINK_UP);
} else {
- if (bond_uses_primary(bond)) {
+ if (bond_uses_primary(bond))
slave_info(bond->dev, new_active->dev, "making interface the new active one\n");
- }
}
}
slave->bond = bond;
slave->dev = slave_dev;
+ INIT_DELAYED_WORK(&slave->notify_work, bond_netdev_notify_work);
if (bond_kobj_init(slave))
return NULL;
return NULL;
}
}
- INIT_DELAYED_WORK(&slave->notify_work, bond_netdev_notify_work);
return slave;
}
static void bond_info_query(struct net_device *bond_dev, struct ifbond *info)
{
struct bonding *bond = netdev_priv(bond_dev);
+
bond_fill_ifbond(bond, info);
}
slave_id = prandom_u32();
break;
case 1:
- slave_id = bond->rr_tx_counter;
+ slave_id = this_cpu_inc_return(*bond->rr_tx_counter);
break;
default:
reciprocal_packets_per_slave =
bond->params.reciprocal_packets_per_slave;
- slave_id = reciprocal_divide(bond->rr_tx_counter,
+ slave_id = this_cpu_inc_return(*bond->rr_tx_counter);
+ slave_id = reciprocal_divide(slave_id,
reciprocal_packets_per_slave);
break;
}
- bond->rr_tx_counter++;
return slave_id;
}
static void bond_destructor(struct net_device *bond_dev)
{
struct bonding *bond = netdev_priv(bond_dev);
+
if (bond->wq)
destroy_workqueue(bond->wq);
+
+ if (bond->rr_tx_counter)
+ free_percpu(bond->rr_tx_counter);
}
void bond_setup(struct net_device *bond_dev)
(struct reciprocal_value) { 0 };
}
- if (primary) {
- strncpy(params->primary, primary, IFNAMSIZ);
- params->primary[IFNAMSIZ - 1] = 0;
- }
+ if (primary)
+ strscpy_pad(params->primary, primary, sizeof(params->primary));
memcpy(params->arp_targets, arp_target, sizeof(arp_target));
if (!bond->wq)
return -ENOMEM;
+ if (BOND_MODE(bond) == BOND_MODE_ROUNDROBIN) {
+ bond->rr_tx_counter = alloc_percpu(u32);
+ if (!bond->rr_tx_counter) {
+ destroy_workqueue(bond->wq);
+ bond->wq = NULL;
+ return -ENOMEM;
+ }
+ }
+
spin_lock_init(&bond->stats_lock);
netdev_lockdep_set_classes(bond_dev);
goto nla_put_failure;
if (nla_put_u32(skb, IFLA_BOND_RESEND_IGMP,
- bond->params.resend_igmp))
+ bond->params.resend_igmp))
goto nla_put_failure;
if (nla_put_u8(skb, IFLA_BOND_NUM_PEER_NOTIF,
int __bond_opt_set_notify(struct bonding *bond,
unsigned int option, struct bond_opt_value *val)
{
- int ret = -ENOENT;
+ int ret;
ASSERT_RTNL();
RCU_INIT_POINTER(bond->primary_slave, NULL);
bond_select_active_slave(bond);
}
- strncpy(bond->params.primary, primary, IFNAMSIZ);
- bond->params.primary[IFNAMSIZ - 1] = 0;
+ strscpy_pad(bond->params.primary, primary, IFNAMSIZ);
netdev_dbg(bond->dev, "Recording %s as primary, but it has not been enslaved yet\n",
primary);
/* ARP information */
if (bond->params.arp_interval > 0) {
int printed = 0;
+
seq_printf(seq, "ARP Polling Interval (ms): %d\n",
bond->params.arp_interval);
char *buf)
{
struct bonding *bond = to_bond(d);
+
return sprintf(buf, "%d\n", bond->params.num_peer_notif);
}
static DEVICE_ATTR(num_grat_arp, 0644,
if (BOND_MODE(bond) == BOND_MODE_8023AD) {
struct ad_info ad_info;
+
count = sprintf(buf, "%d\n",
bond_3ad_get_active_agg_info(bond, &ad_info)
? 0 : ad_info.aggregator_id);
if (BOND_MODE(bond) == BOND_MODE_8023AD) {
struct ad_info ad_info;
+
count = sprintf(buf, "%d\n",
bond_3ad_get_active_agg_info(bond, &ad_info)
? 0 : ad_info.ports);
if (BOND_MODE(bond) == BOND_MODE_8023AD && capable(CAP_NET_ADMIN)) {
struct ad_info ad_info;
+
count = sprintf(buf, "%d\n",
bond_3ad_get_active_agg_info(bond, &ad_info)
? 0 : ad_info.actor_key);
if (BOND_MODE(bond) == BOND_MODE_8023AD && capable(CAP_NET_ADMIN)) {
struct ad_info ad_info;
+
count = sprintf(buf, "%d\n",
bond_3ad_get_active_agg_info(bond, &ad_info)
? 0 : ad_info.partner_key);
if (BOND_MODE(bond) == BOND_MODE_8023AD && capable(CAP_NET_ADMIN)) {
struct ad_info ad_info;
+
if (!bond_3ad_get_active_agg_info(bond, &ad_info))
count = sprintf(buf, "%pM\n", ad_info.partner_system);
}
char *buf)
{
struct bonding *bond = to_bond(d);
+
return sprintf(buf, "%d\n", bond->params.tlb_dynamic_lb);
}
static DEVICE_ATTR(tlb_dynamic_lb, 0644,
{
struct ser_device *ser;
- if (WARN_ON(!dev))
- return -EINVAL;
-
ser = netdev_priv(dev);
/* Send flow off once, on high water mark */
rtnl_lock();
result = register_netdevice(dev);
if (result) {
+ tty_kref_put(tty);
rtnl_unlock();
free_netdev(dev);
return -ENODEV;
case 0:
++cfv->stats.rx_napi_complete;
- /* Really out of patckets? (stolen from virtio_net)*/
+ /* Really out of packets? (stolen from virtio_net)*/
napi_complete(napi);
if (unlikely(!vringh_notify_enable_kern(cfv->vr_rx)) &&
napi_schedule_prep(napi)) {
vringh_notify_disable_kern(cfv->vr_rx);
napi_disable(&cfv->napi);
- /* Release any TX buffers on both used and avilable rings */
+ /* Release any TX buffers on both used and available rings */
cfv_release_used_buf(cfv->vq_tx);
spin_lock_irqsave(&cfv->tx_lock, flags);
while ((buf_info = virtqueue_detach_unused_buf(cfv->vq_tx)))
if (unlikely(!buf_info))
goto err;
- /* Make the IP header aligned in tbe buffer */
+ /* Make the IP header aligned in the buffer */
hdr_ofs = cfv->tx_hr + info->hdr_len;
pad_len = hdr_ofs & (IP_HDR_ALIGN - 1);
buf_info->size = cfv->tx_hr + skb->len + cfv->tx_tr + pad_len;
};
#define AT91_IS(_model) \
-static inline int at91_is_sam##_model(const struct at91_priv *priv) \
+static inline int __maybe_unused at91_is_sam##_model(const struct at91_priv *priv) \
{ \
return priv->devtype_data.type == AT91_DEVTYPE_SAM##_model; \
}
#
obj-$(CONFIG_CAN_C_CAN) += c_can.o
+
+c_can-objs :=
+c_can-objs += c_can_ethtool.o
+c_can-objs += c_can_main.o
+
obj-$(CONFIG_CAN_C_CAN_PLATFORM) += c_can_platform.o
obj-$(CONFIG_CAN_C_CAN_PCI) += c_can_pci.o
struct c_can_raminit raminit_sys; /* RAMINIT via syscon regmap */
void (*raminit)(const struct c_can_priv *priv, bool enable);
u32 comm_rcv_high;
- u32 rxmasked;
u32 dlc[];
};
int c_can_power_down(struct net_device *dev);
#endif
+void c_can_set_ethtool_ops(struct net_device *dev);
+
#endif /* C_CAN_H */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright 2021, Dario Binacchi <dariobin@libero.it>
+ */
+
+#include <linux/ethtool.h>
+#include <linux/kernel.h>
+#include <linux/platform_device.h>
+#include <linux/netdevice.h>
+#include <linux/can/dev.h>
+
+#include "c_can.h"
+
+static void c_can_get_drvinfo(struct net_device *netdev,
+ struct ethtool_drvinfo *info)
+{
+ struct c_can_priv *priv = netdev_priv(netdev);
+ struct platform_device *pdev = to_platform_device(priv->device);
+
+ strscpy(info->driver, "c_can", sizeof(info->driver));
+ strscpy(info->bus_info, pdev->name, sizeof(info->bus_info));
+}
+
+static void c_can_get_ringparam(struct net_device *netdev,
+ struct ethtool_ringparam *ring)
+{
+ struct c_can_priv *priv = netdev_priv(netdev);
+
+ ring->rx_max_pending = priv->msg_obj_num;
+ ring->tx_max_pending = priv->msg_obj_num;
+ ring->rx_pending = priv->msg_obj_rx_num;
+ ring->tx_pending = priv->msg_obj_tx_num;
+}
+
+static const struct ethtool_ops c_can_ethtool_ops = {
+ .get_drvinfo = c_can_get_drvinfo,
+ .get_ringparam = c_can_get_ringparam,
+};
+
+void c_can_set_ethtool_ops(struct net_device *netdev)
+{
+ netdev->ethtool_ops = &c_can_ethtool_ops;
+}
/* Clear all internal status */
atomic_set(&priv->tx_active, 0);
- priv->rxmasked = 0;
priv->tx_dir = 0;
/* set bittiming params */
dev->flags |= IFF_ECHO; /* we support local echo */
dev->netdev_ops = &c_can_netdev_ops;
+ c_can_set_ethtool_ops(dev);
err = register_candev(dev);
if (!err)
#define MRAM_CFG_LEN 8
/* Core Release Register (CREL) */
-#define CREL_REL_SHIFT 28
-#define CREL_REL_MASK (0xF << CREL_REL_SHIFT)
-#define CREL_STEP_SHIFT 24
-#define CREL_STEP_MASK (0xF << CREL_STEP_SHIFT)
-#define CREL_SUBSTEP_SHIFT 20
-#define CREL_SUBSTEP_MASK (0xF << CREL_SUBSTEP_SHIFT)
+#define CREL_REL_MASK GENMASK(31, 28)
+#define CREL_STEP_MASK GENMASK(27, 24)
+#define CREL_SUBSTEP_MASK GENMASK(23, 20)
/* Data Bit Timing & Prescaler Register (DBTP) */
#define DBTP_TDC BIT(23)
-#define DBTP_DBRP_SHIFT 16
-#define DBTP_DBRP_MASK (0x1f << DBTP_DBRP_SHIFT)
-#define DBTP_DTSEG1_SHIFT 8
-#define DBTP_DTSEG1_MASK (0x1f << DBTP_DTSEG1_SHIFT)
-#define DBTP_DTSEG2_SHIFT 4
-#define DBTP_DTSEG2_MASK (0xf << DBTP_DTSEG2_SHIFT)
-#define DBTP_DSJW_SHIFT 0
-#define DBTP_DSJW_MASK (0xf << DBTP_DSJW_SHIFT)
+#define DBTP_DBRP_MASK GENMASK(20, 16)
+#define DBTP_DTSEG1_MASK GENMASK(12, 8)
+#define DBTP_DTSEG2_MASK GENMASK(7, 4)
+#define DBTP_DSJW_MASK GENMASK(3, 0)
/* Transmitter Delay Compensation Register (TDCR) */
-#define TDCR_TDCO_SHIFT 8
-#define TDCR_TDCO_MASK (0x7F << TDCR_TDCO_SHIFT)
-#define TDCR_TDCF_SHIFT 0
-#define TDCR_TDCF_MASK (0x7F << TDCR_TDCF_SHIFT)
+#define TDCR_TDCO_MASK GENMASK(14, 8)
+#define TDCR_TDCF_MASK GENMASK(6, 0)
/* Test Register (TEST) */
#define TEST_LBCK BIT(4)
-/* CC Control Register(CCCR) */
-#define CCCR_CMR_MASK 0x3
-#define CCCR_CMR_SHIFT 10
-#define CCCR_CMR_CANFD 0x1
-#define CCCR_CMR_CANFD_BRS 0x2
-#define CCCR_CMR_CAN 0x3
-#define CCCR_CME_MASK 0x3
-#define CCCR_CME_SHIFT 8
-#define CCCR_CME_CAN 0
-#define CCCR_CME_CANFD 0x1
-#define CCCR_CME_CANFD_BRS 0x2
+/* CC Control Register (CCCR) */
#define CCCR_TXP BIT(14)
#define CCCR_TEST BIT(7)
#define CCCR_DAR BIT(6)
#define CCCR_ASM BIT(2)
#define CCCR_CCE BIT(1)
#define CCCR_INIT BIT(0)
-#define CCCR_CANFD 0x10
+/* for version 3.0.x */
+#define CCCR_CMR_MASK GENMASK(11, 10)
+#define CCCR_CMR_CANFD 0x1
+#define CCCR_CMR_CANFD_BRS 0x2
+#define CCCR_CMR_CAN 0x3
+#define CCCR_CME_MASK GENMASK(9, 8)
+#define CCCR_CME_CAN 0
+#define CCCR_CME_CANFD 0x1
+#define CCCR_CME_CANFD_BRS 0x2
/* for version >=3.1.x */
#define CCCR_EFBI BIT(13)
#define CCCR_PXHD BIT(12)
#define CCCR_BRSE BIT(9)
#define CCCR_FDOE BIT(8)
-/* only for version >=3.2.x */
+/* for version >=3.2.x */
#define CCCR_NISO BIT(15)
+/* for version >=3.3.x */
+#define CCCR_WMM BIT(11)
+#define CCCR_UTSU BIT(10)
/* Nominal Bit Timing & Prescaler Register (NBTP) */
-#define NBTP_NSJW_SHIFT 25
-#define NBTP_NSJW_MASK (0x7f << NBTP_NSJW_SHIFT)
-#define NBTP_NBRP_SHIFT 16
-#define NBTP_NBRP_MASK (0x1ff << NBTP_NBRP_SHIFT)
-#define NBTP_NTSEG1_SHIFT 8
-#define NBTP_NTSEG1_MASK (0xff << NBTP_NTSEG1_SHIFT)
-#define NBTP_NTSEG2_SHIFT 0
-#define NBTP_NTSEG2_MASK (0x7f << NBTP_NTSEG2_SHIFT)
+#define NBTP_NSJW_MASK GENMASK(31, 25)
+#define NBTP_NBRP_MASK GENMASK(24, 16)
+#define NBTP_NTSEG1_MASK GENMASK(15, 8)
+#define NBTP_NTSEG2_MASK GENMASK(6, 0)
/* Timestamp Counter Configuration Register (TSCC) */
#define TSCC_TCP_MASK GENMASK(19, 16)
/* Timestamp Counter Value Register (TSCV) */
#define TSCV_TSC_MASK GENMASK(15, 0)
-/* Error Counter Register(ECR) */
+/* Error Counter Register (ECR) */
#define ECR_RP BIT(15)
-#define ECR_REC_SHIFT 8
-#define ECR_REC_MASK (0x7f << ECR_REC_SHIFT)
-#define ECR_TEC_SHIFT 0
-#define ECR_TEC_MASK 0xff
+#define ECR_REC_MASK GENMASK(14, 8)
+#define ECR_TEC_MASK GENMASK(7, 0)
-/* Protocol Status Register(PSR) */
+/* Protocol Status Register (PSR) */
#define PSR_BO BIT(7)
#define PSR_EW BIT(6)
#define PSR_EP BIT(5)
-#define PSR_LEC_MASK 0x7
+#define PSR_LEC_MASK GENMASK(2, 0)
-/* Interrupt Register(IR) */
+/* Interrupt Register (IR) */
#define IR_ALL_INT 0xffffffff
/* Renamed bits for versions > 3.1.x */
IR_BEC | IR_TOO | IR_MRAF | IR_TSW | IR_TEFL | \
IR_RF1L | IR_RF0L)
#define IR_ERR_ALL_30X (IR_ERR_STATE | IR_ERR_BUS_30X)
+
/* Interrupts for version >= 3.1.x */
#define IR_ERR_LEC_31X (IR_PED | IR_PEA)
#define IR_ERR_BUS_31X (IR_ERR_LEC_31X | IR_WDI | IR_ELO | IR_BEU | \
#define ILE_EINT0 BIT(0)
/* Rx FIFO 0/1 Configuration (RXF0C/RXF1C) */
-#define RXFC_FWM_SHIFT 24
-#define RXFC_FWM_MASK (0x7f << RXFC_FWM_SHIFT)
-#define RXFC_FS_SHIFT 16
-#define RXFC_FS_MASK (0x7f << RXFC_FS_SHIFT)
+#define RXFC_FWM_MASK GENMASK(30, 24)
+#define RXFC_FS_MASK GENMASK(22, 16)
/* Rx FIFO 0/1 Status (RXF0S/RXF1S) */
#define RXFS_RFL BIT(25)
#define RXFS_FF BIT(24)
-#define RXFS_FPI_SHIFT 16
-#define RXFS_FPI_MASK 0x3f0000
-#define RXFS_FGI_SHIFT 8
-#define RXFS_FGI_MASK 0x3f00
-#define RXFS_FFL_MASK 0x7f
+#define RXFS_FPI_MASK GENMASK(21, 16)
+#define RXFS_FGI_MASK GENMASK(13, 8)
+#define RXFS_FFL_MASK GENMASK(6, 0)
/* Rx Buffer / FIFO Element Size Configuration (RXESC) */
-#define M_CAN_RXESC_8BYTES 0x0
-#define M_CAN_RXESC_64BYTES 0x777
+#define RXESC_RBDS_MASK GENMASK(10, 8)
+#define RXESC_F1DS_MASK GENMASK(6, 4)
+#define RXESC_F0DS_MASK GENMASK(2, 0)
+#define RXESC_64B 0x7
-/* Tx Buffer Configuration(TXBC) */
-#define TXBC_NDTB_SHIFT 16
-#define TXBC_NDTB_MASK (0x3f << TXBC_NDTB_SHIFT)
-#define TXBC_TFQS_SHIFT 24
-#define TXBC_TFQS_MASK (0x3f << TXBC_TFQS_SHIFT)
+/* Tx Buffer Configuration (TXBC) */
+#define TXBC_TFQS_MASK GENMASK(29, 24)
+#define TXBC_NDTB_MASK GENMASK(21, 16)
/* Tx FIFO/Queue Status (TXFQS) */
#define TXFQS_TFQF BIT(21)
-#define TXFQS_TFQPI_SHIFT 16
-#define TXFQS_TFQPI_MASK (0x1f << TXFQS_TFQPI_SHIFT)
-#define TXFQS_TFGI_SHIFT 8
-#define TXFQS_TFGI_MASK (0x1f << TXFQS_TFGI_SHIFT)
-#define TXFQS_TFFL_SHIFT 0
-#define TXFQS_TFFL_MASK (0x3f << TXFQS_TFFL_SHIFT)
+#define TXFQS_TFQPI_MASK GENMASK(20, 16)
+#define TXFQS_TFGI_MASK GENMASK(12, 8)
+#define TXFQS_TFFL_MASK GENMASK(5, 0)
-/* Tx Buffer Element Size Configuration(TXESC) */
-#define TXESC_TBDS_8BYTES 0x0
-#define TXESC_TBDS_64BYTES 0x7
+/* Tx Buffer Element Size Configuration (TXESC) */
+#define TXESC_TBDS_MASK GENMASK(2, 0)
+#define TXESC_TBDS_64B 0x7
/* Tx Event FIFO Configuration (TXEFC) */
-#define TXEFC_EFS_SHIFT 16
-#define TXEFC_EFS_MASK (0x3f << TXEFC_EFS_SHIFT)
+#define TXEFC_EFS_MASK GENMASK(21, 16)
/* Tx Event FIFO Status (TXEFS) */
#define TXEFS_TEFL BIT(25)
#define TXEFS_EFF BIT(24)
-#define TXEFS_EFGI_SHIFT 8
-#define TXEFS_EFGI_MASK (0x1f << TXEFS_EFGI_SHIFT)
-#define TXEFS_EFFL_SHIFT 0
-#define TXEFS_EFFL_MASK (0x3f << TXEFS_EFFL_SHIFT)
+#define TXEFS_EFGI_MASK GENMASK(12, 8)
+#define TXEFS_EFFL_MASK GENMASK(5, 0)
/* Tx Event FIFO Acknowledge (TXEFA) */
-#define TXEFA_EFAI_SHIFT 0
-#define TXEFA_EFAI_MASK (0x1f << TXEFA_EFAI_SHIFT)
+#define TXEFA_EFAI_MASK GENMASK(4, 0)
/* Message RAM Configuration (in bytes) */
#define SIDF_ELEMENT_SIZE 4
#define TX_BUF_EFC BIT(23)
#define TX_BUF_FDF BIT(21)
#define TX_BUF_BRS BIT(20)
-#define TX_BUF_MM_SHIFT 24
-#define TX_BUF_MM_MASK (0xff << TX_BUF_MM_SHIFT)
+#define TX_BUF_MM_MASK GENMASK(31, 24)
+#define TX_BUF_DLC_MASK GENMASK(19, 16)
/* Tx event FIFO Element */
/* E1 */
-#define TX_EVENT_MM_SHIFT TX_BUF_MM_SHIFT
-#define TX_EVENT_MM_MASK (0xff << TX_EVENT_MM_SHIFT)
+#define TX_EVENT_MM_MASK GENMASK(31, 24)
#define TX_EVENT_TXTS_MASK GENMASK(15, 0)
static inline u32 m_can_read(struct m_can_classdev *cdev, enum m_can_reg reg)
net->stats.tx_errors++;
if (cdev->version > 30)
- putidx = ((m_can_read(cdev, M_CAN_TXFQS) &
- TXFQS_TFQPI_MASK) >> TXFQS_TFQPI_SHIFT);
+ putidx = FIELD_GET(TXFQS_TFQPI_MASK,
+ m_can_read(cdev, M_CAN_TXFQS));
can_free_echo_skb(cdev->net, putidx, NULL);
cdev->tx_skb = NULL;
int i;
/* calculate the fifo get index for where to read data */
- fgi = (rxfs & RXFS_FGI_MASK) >> RXFS_FGI_SHIFT;
+ fgi = FIELD_GET(RXFS_FGI_MASK, rxfs);
dlc = m_can_fifo_read(cdev, fgi, M_CAN_FIFO_DLC);
if (dlc & RX_BUF_FDF)
skb = alloc_canfd_skb(dev, &cf);
unsigned int ecr;
ecr = m_can_read(cdev, M_CAN_ECR);
- bec->rxerr = (ecr & ECR_REC_MASK) >> ECR_REC_SHIFT;
- bec->txerr = (ecr & ECR_TEC_MASK) >> ECR_TEC_SHIFT;
+ bec->rxerr = FIELD_GET(ECR_REC_MASK, ecr);
+ bec->txerr = FIELD_GET(ECR_TEC_MASK, ecr);
return 0;
}
m_can_txefs = m_can_read(cdev, M_CAN_TXEFS);
/* Get Tx Event fifo element count */
- txe_count = (m_can_txefs & TXEFS_EFFL_MASK) >> TXEFS_EFFL_SHIFT;
+ txe_count = FIELD_GET(TXEFS_EFFL_MASK, m_can_txefs);
/* Get and process all sent elements */
for (i = 0; i < txe_count; i++) {
u32 txe, timestamp = 0;
/* retrieve get index */
- fgi = (m_can_read(cdev, M_CAN_TXEFS) & TXEFS_EFGI_MASK) >>
- TXEFS_EFGI_SHIFT;
+ fgi = FIELD_GET(TXEFS_EFGI_MASK, m_can_read(cdev, M_CAN_TXEFS));
/* get message marker, timestamp */
txe = m_can_txe_fifo_read(cdev, fgi, 4);
- msg_mark = (txe & TX_EVENT_MM_MASK) >> TX_EVENT_MM_SHIFT;
+ msg_mark = FIELD_GET(TX_EVENT_MM_MASK, txe);
timestamp = FIELD_GET(TX_EVENT_TXTS_MASK, txe);
/* ack txe element */
- m_can_write(cdev, M_CAN_TXEFA, (TXEFA_EFAI_MASK &
- (fgi << TXEFA_EFAI_SHIFT)));
+ m_can_write(cdev, M_CAN_TXEFA, FIELD_PREP(TXEFA_EFAI_MASK,
+ fgi));
/* update stats */
m_can_tx_update_stats(cdev, msg_mark, timestamp);
sjw = bt->sjw - 1;
tseg1 = bt->prop_seg + bt->phase_seg1 - 1;
tseg2 = bt->phase_seg2 - 1;
- reg_btp = (brp << NBTP_NBRP_SHIFT) | (sjw << NBTP_NSJW_SHIFT) |
- (tseg1 << NBTP_NTSEG1_SHIFT) | (tseg2 << NBTP_NTSEG2_SHIFT);
+ reg_btp = FIELD_PREP(NBTP_NBRP_MASK, brp) |
+ FIELD_PREP(NBTP_NSJW_MASK, sjw) |
+ FIELD_PREP(NBTP_NTSEG1_MASK, tseg1) |
+ FIELD_PREP(NBTP_NTSEG2_MASK, tseg2);
m_can_write(cdev, M_CAN_NBTP, reg_btp);
if (cdev->can.ctrlmode & CAN_CTRLMODE_FD) {
reg_btp |= DBTP_TDC;
m_can_write(cdev, M_CAN_TDCR,
- tdco << TDCR_TDCO_SHIFT);
+ FIELD_PREP(TDCR_TDCO_MASK, tdco));
}
- reg_btp |= (brp << DBTP_DBRP_SHIFT) |
- (sjw << DBTP_DSJW_SHIFT) |
- (tseg1 << DBTP_DTSEG1_SHIFT) |
- (tseg2 << DBTP_DTSEG2_SHIFT);
+ reg_btp = FIELD_PREP(NBTP_NBRP_MASK, brp) |
+ FIELD_PREP(NBTP_NSJW_MASK, sjw) |
+ FIELD_PREP(NBTP_NTSEG1_MASK, tseg1) |
+ FIELD_PREP(NBTP_NTSEG2_MASK, tseg2);
m_can_write(cdev, M_CAN_DBTP, reg_btp);
}
m_can_config_endisable(cdev, true);
/* RX Buffer/FIFO Element Size 64 bytes data field */
- m_can_write(cdev, M_CAN_RXESC, M_CAN_RXESC_64BYTES);
+ m_can_write(cdev, M_CAN_RXESC,
+ FIELD_PREP(RXESC_RBDS_MASK, RXESC_64B) |
+ FIELD_PREP(RXESC_F1DS_MASK, RXESC_64B) |
+ FIELD_PREP(RXESC_F0DS_MASK, RXESC_64B));
/* Accept Non-matching Frames Into FIFO 0 */
m_can_write(cdev, M_CAN_GFC, 0x0);
if (cdev->version == 30) {
/* only support one Tx Buffer currently */
- m_can_write(cdev, M_CAN_TXBC, (1 << TXBC_NDTB_SHIFT) |
+ m_can_write(cdev, M_CAN_TXBC, FIELD_PREP(TXBC_NDTB_MASK, 1) |
cdev->mcfg[MRAM_TXB].off);
} else {
/* TX FIFO is used for newer IP Core versions */
m_can_write(cdev, M_CAN_TXBC,
- (cdev->mcfg[MRAM_TXB].num << TXBC_TFQS_SHIFT) |
- (cdev->mcfg[MRAM_TXB].off));
+ FIELD_PREP(TXBC_TFQS_MASK,
+ cdev->mcfg[MRAM_TXB].num) |
+ cdev->mcfg[MRAM_TXB].off);
}
/* support 64 bytes payload */
- m_can_write(cdev, M_CAN_TXESC, TXESC_TBDS_64BYTES);
+ m_can_write(cdev, M_CAN_TXESC,
+ FIELD_PREP(TXESC_TBDS_MASK, TXESC_TBDS_64B));
/* TX Event FIFO */
if (cdev->version == 30) {
- m_can_write(cdev, M_CAN_TXEFC, (1 << TXEFC_EFS_SHIFT) |
+ m_can_write(cdev, M_CAN_TXEFC,
+ FIELD_PREP(TXEFC_EFS_MASK, 1) |
cdev->mcfg[MRAM_TXE].off);
} else {
/* Full TX Event FIFO is used */
m_can_write(cdev, M_CAN_TXEFC,
- ((cdev->mcfg[MRAM_TXE].num << TXEFC_EFS_SHIFT)
- & TXEFC_EFS_MASK) |
+ FIELD_PREP(TXEFC_EFS_MASK,
+ cdev->mcfg[MRAM_TXE].num) |
cdev->mcfg[MRAM_TXE].off);
}
/* rx fifo configuration, blocking mode, fifo size 1 */
m_can_write(cdev, M_CAN_RXF0C,
- (cdev->mcfg[MRAM_RXF0].num << RXFC_FS_SHIFT) |
+ FIELD_PREP(RXFC_FS_MASK, cdev->mcfg[MRAM_RXF0].num) |
cdev->mcfg[MRAM_RXF0].off);
m_can_write(cdev, M_CAN_RXF1C,
- (cdev->mcfg[MRAM_RXF1].num << RXFC_FS_SHIFT) |
+ FIELD_PREP(RXFC_FS_MASK, cdev->mcfg[MRAM_RXF1].num) |
cdev->mcfg[MRAM_RXF1].off);
cccr = m_can_read(cdev, M_CAN_CCCR);
/* Version 3.0.x */
cccr &= ~(CCCR_TEST | CCCR_MON | CCCR_DAR |
- (CCCR_CMR_MASK << CCCR_CMR_SHIFT) |
- (CCCR_CME_MASK << CCCR_CME_SHIFT));
+ FIELD_PREP(CCCR_CMR_MASK, FIELD_MAX(CCCR_CMR_MASK)) |
+ FIELD_PREP(CCCR_CME_MASK, FIELD_MAX(CCCR_CME_MASK)));
if (cdev->can.ctrlmode & CAN_CTRLMODE_FD)
- cccr |= CCCR_CME_CANFD_BRS << CCCR_CME_SHIFT;
+ cccr |= FIELD_PREP(CCCR_CME_MASK, CCCR_CME_CANFD_BRS);
} else {
/* Version 3.1.x or 3.2.x */
* Example: Version 3.2.1 => rel = 3; step = 2; substep = 1;
*/
crel_reg = m_can_read(cdev, M_CAN_CREL);
- rel = (u8)((crel_reg & CREL_REL_MASK) >> CREL_REL_SHIFT);
- step = (u8)((crel_reg & CREL_STEP_MASK) >> CREL_STEP_SHIFT);
+ rel = (u8)FIELD_GET(CREL_REL_MASK, crel_reg);
+ step = (u8)FIELD_GET(CREL_STEP_MASK, crel_reg);
if (rel == 3) {
/* M_CAN v3.x.y: create return value */
if (cdev->can.ctrlmode & CAN_CTRLMODE_FD) {
cccr = m_can_read(cdev, M_CAN_CCCR);
- cccr &= ~(CCCR_CMR_MASK << CCCR_CMR_SHIFT);
+ cccr &= ~CCCR_CMR_MASK;
if (can_is_canfd_skb(skb)) {
if (cf->flags & CANFD_BRS)
- cccr |= CCCR_CMR_CANFD_BRS <<
- CCCR_CMR_SHIFT;
+ cccr |= FIELD_PREP(CCCR_CMR_MASK,
+ CCCR_CMR_CANFD_BRS);
else
- cccr |= CCCR_CMR_CANFD <<
- CCCR_CMR_SHIFT;
+ cccr |= FIELD_PREP(CCCR_CMR_MASK,
+ CCCR_CMR_CANFD);
} else {
- cccr |= CCCR_CMR_CAN << CCCR_CMR_SHIFT;
+ cccr |= FIELD_PREP(CCCR_CMR_MASK, CCCR_CMR_CAN);
}
m_can_write(cdev, M_CAN_CCCR, cccr);
}
}
/* get put index for frame */
- putidx = ((m_can_read(cdev, M_CAN_TXFQS) & TXFQS_TFQPI_MASK)
- >> TXFQS_TFQPI_SHIFT);
+ putidx = FIELD_GET(TXFQS_TFQPI_MASK,
+ m_can_read(cdev, M_CAN_TXFQS));
/* Write ID Field to FIFO Element */
m_can_fifo_write(cdev, putidx, M_CAN_FIFO_ID, id);
* sending the correct echo frame
*/
m_can_fifo_write(cdev, putidx, M_CAN_FIFO_DLC,
- ((putidx << TX_BUF_MM_SHIFT) &
- TX_BUF_MM_MASK) |
- (can_fd_len2dlc(cf->len) << 16) |
+ FIELD_PREP(TX_BUF_MM_MASK, putidx) |
+ FIELD_PREP(TX_BUF_DLC_MASK,
+ can_fd_len2dlc(cf->len)) |
fdflags | TX_BUF_EFC);
for (i = 0; i < cf->len; i += 4)
cdev->mcfg[MRAM_RXF0].off = cdev->mcfg[MRAM_XIDF].off +
cdev->mcfg[MRAM_XIDF].num * XIDF_ELEMENT_SIZE;
cdev->mcfg[MRAM_RXF0].num = mram_config_vals[3] &
- (RXFC_FS_MASK >> RXFC_FS_SHIFT);
+ FIELD_MAX(RXFC_FS_MASK);
cdev->mcfg[MRAM_RXF1].off = cdev->mcfg[MRAM_RXF0].off +
cdev->mcfg[MRAM_RXF0].num * RXF0_ELEMENT_SIZE;
cdev->mcfg[MRAM_RXF1].num = mram_config_vals[4] &
- (RXFC_FS_MASK >> RXFC_FS_SHIFT);
+ FIELD_MAX(RXFC_FS_MASK);
cdev->mcfg[MRAM_RXB].off = cdev->mcfg[MRAM_RXF1].off +
cdev->mcfg[MRAM_RXF1].num * RXF1_ELEMENT_SIZE;
cdev->mcfg[MRAM_RXB].num = mram_config_vals[5];
cdev->mcfg[MRAM_TXB].off = cdev->mcfg[MRAM_TXE].off +
cdev->mcfg[MRAM_TXE].num * TXE_ELEMENT_SIZE;
cdev->mcfg[MRAM_TXB].num = mram_config_vals[7] &
- (TXBC_NDTB_MASK >> TXBC_NDTB_SHIFT);
+ FIELD_MAX(TXBC_NDTB_MASK);
dev_dbg(cdev->dev,
"sidf 0x%x %d xidf 0x%x %d rxf0 0x%x %d rxf1 0x%x %d rxb 0x%x %d txe 0x%x %d txb 0x%x %d\n",
DPRAM_INFO_BUSSTATE2 : DPRAM_INFO_BUSSTATE]);
/* timestamp */
tmp_u32 = le32_to_cpup((void *)ptr);
- ptr += 4;
ktime = softing_raw2ktime(card, tmp_u32);
++netdev->stats.rx_errors;
ktime = softing_raw2ktime(card, tmp_u32);
if (!(msg.can_id & CAN_RTR_FLAG))
memcpy(&msg.data[0], ptr, 8);
- ptr += 8;
/* update socket */
if (cmd & CMD_ACK) {
/* acknowledge, was tx msg */
CAN_CTRLMODE_BERR_REPORTING;
if (of_id)
- priv->model = (enum hi3110_model)of_id->data;
+ priv->model = (enum hi3110_model)(uintptr_t)of_id->data;
else
priv->model = spi_get_device_id(spi)->driver_data;
priv->net = net;
priv->can.ctrlmode_supported = CAN_CTRLMODE_3_SAMPLES |
CAN_CTRLMODE_LOOPBACK | CAN_CTRLMODE_LISTENONLY;
if (match)
- priv->model = (enum mcp251x_model)match;
+ priv->model = (enum mcp251x_model)(uintptr_t)match;
else
priv->model = spi_get_device_id(spi)->driver_data;
priv->net = net;
return __mcp251xfd_chip_set_mode(priv, mode_req, false);
}
-static inline int
+static inline int __maybe_unused
mcp251xfd_chip_set_mode_nowait(const struct mcp251xfd_priv *priv,
const u8 mode_req)
{
- Scania VCI2 (if you have the Kvaser logo on top)
- Kvaser BlackBird v2
- Kvaser Leaf Pro HS v2
+ - Kvaser Hybrid CAN/LIN
- Kvaser Hybrid 2xCAN/LIN
+ - Kvaser Hybrid Pro CAN/LIN
- Kvaser Hybrid Pro 2xCAN/LIN
- Kvaser Memorator 2xHS v2
- Kvaser Memorator Pro 2xHS v2
#define USB_USBCAN_PRO_2HS_V2_PRODUCT_ID 264
#define USB_MEMO_2HS_PRODUCT_ID 265
#define USB_MEMO_PRO_2HS_V2_PRODUCT_ID 266
-#define USB_HYBRID_CANLIN_PRODUCT_ID 267
+#define USB_HYBRID_2CANLIN_PRODUCT_ID 267
#define USB_ATI_USBCAN_PRO_2HS_V2_PRODUCT_ID 268
#define USB_ATI_MEMO_PRO_2HS_V2_PRODUCT_ID 269
-#define USB_HYBRID_PRO_CANLIN_PRODUCT_ID 270
+#define USB_HYBRID_PRO_2CANLIN_PRODUCT_ID 270
#define USB_U100_PRODUCT_ID 273
#define USB_U100P_PRODUCT_ID 274
#define USB_U100S_PRODUCT_ID 275
#define USB_USBCAN_PRO_4HS_PRODUCT_ID 276
+#define USB_HYBRID_CANLIN_PRODUCT_ID 277
+#define USB_HYBRID_PRO_CANLIN_PRODUCT_ID 278
#define USB_HYDRA_PRODUCT_ID_END \
- USB_USBCAN_PRO_4HS_PRODUCT_ID
+ USB_HYBRID_PRO_CANLIN_PRODUCT_ID
static inline bool kvaser_is_leaf(const struct usb_device_id *id)
{
{ USB_DEVICE(KVASER_VENDOR_ID, USB_USBCAN_PRO_2HS_V2_PRODUCT_ID) },
{ USB_DEVICE(KVASER_VENDOR_ID, USB_MEMO_2HS_PRODUCT_ID) },
{ USB_DEVICE(KVASER_VENDOR_ID, USB_MEMO_PRO_2HS_V2_PRODUCT_ID) },
- { USB_DEVICE(KVASER_VENDOR_ID, USB_HYBRID_CANLIN_PRODUCT_ID) },
+ { USB_DEVICE(KVASER_VENDOR_ID, USB_HYBRID_2CANLIN_PRODUCT_ID) },
{ USB_DEVICE(KVASER_VENDOR_ID, USB_ATI_USBCAN_PRO_2HS_V2_PRODUCT_ID) },
{ USB_DEVICE(KVASER_VENDOR_ID, USB_ATI_MEMO_PRO_2HS_V2_PRODUCT_ID) },
- { USB_DEVICE(KVASER_VENDOR_ID, USB_HYBRID_PRO_CANLIN_PRODUCT_ID) },
+ { USB_DEVICE(KVASER_VENDOR_ID, USB_HYBRID_PRO_2CANLIN_PRODUCT_ID) },
{ USB_DEVICE(KVASER_VENDOR_ID, USB_U100_PRODUCT_ID) },
{ USB_DEVICE(KVASER_VENDOR_ID, USB_U100P_PRODUCT_ID) },
{ USB_DEVICE(KVASER_VENDOR_ID, USB_U100S_PRODUCT_ID) },
{ USB_DEVICE(KVASER_VENDOR_ID, USB_USBCAN_PRO_4HS_PRODUCT_ID) },
+ { USB_DEVICE(KVASER_VENDOR_ID, USB_HYBRID_CANLIN_PRODUCT_ID) },
+ { USB_DEVICE(KVASER_VENDOR_ID, USB_HYBRID_PRO_CANLIN_PRODUCT_ID) },
{ }
};
MODULE_DEVICE_TABLE(usb, kvaser_usb_table);
bool can_ka_first_pass;
bool can_speed_check;
atomic_t free_ctx_cnt;
+ void *rxbuf[MCBA_MAX_RX_URBS];
+ dma_addr_t rxbuf_dma[MCBA_MAX_RX_URBS];
};
/* CAN frame */
for (i = 0; i < MCBA_MAX_RX_URBS; i++) {
struct urb *urb = NULL;
u8 *buf;
+ dma_addr_t buf_dma;
/* create a URB, and a buffer for it */
urb = usb_alloc_urb(0, GFP_KERNEL);
}
buf = usb_alloc_coherent(priv->udev, MCBA_USB_RX_BUFF_SIZE,
- GFP_KERNEL, &urb->transfer_dma);
+ GFP_KERNEL, &buf_dma);
if (!buf) {
netdev_err(netdev, "No memory left for USB buffer\n");
usb_free_urb(urb);
if (err) {
usb_unanchor_urb(urb);
usb_free_coherent(priv->udev, MCBA_USB_RX_BUFF_SIZE,
- buf, urb->transfer_dma);
+ buf, buf_dma);
usb_free_urb(urb);
break;
}
+ priv->rxbuf[i] = buf;
+ priv->rxbuf_dma[i] = buf_dma;
+
/* Drop reference, USB core will take care of freeing it */
usb_free_urb(urb);
}
static void mcba_urb_unlink(struct mcba_priv *priv)
{
+ int i;
+
usb_kill_anchored_urbs(&priv->rx_submitted);
+
+ for (i = 0; i < MCBA_MAX_RX_URBS; ++i)
+ usb_free_coherent(priv->udev, MCBA_USB_RX_BUFF_SIZE,
+ priv->rxbuf[i], priv->rxbuf_dma[i]);
+
usb_kill_anchored_urbs(&priv->tx_submitted);
}
return 0;
}
+static bool b53_vlan_port_needs_forced_tagged(struct dsa_switch *ds, int port)
+{
+ struct b53_device *dev = ds->priv;
+
+ return dev->tag_protocol == DSA_TAG_PROTO_NONE && dsa_is_cpu_port(ds, port);
+}
+
int b53_configure_vlan(struct dsa_switch *ds)
{
struct b53_device *dev = ds->priv;
b53_enable_vlan(dev, -1, dev->vlan_enabled, ds->vlan_filtering);
- b53_for_each_port(dev, i)
+ /* Create an untagged VLAN entry for the default PVID in case
+ * CONFIG_VLAN_8021Q is disabled and there are no calls to
+ * dsa_slave_vlan_rx_add_vid() to create the default VLAN
+ * entry. Do this only when the tagging protocol is not
+ * DSA_TAG_PROTO_NONE
+ */
+ b53_for_each_port(dev, i) {
+ v = &dev->vlans[def_vid];
+ v->members |= BIT(i);
+ if (!b53_vlan_port_needs_forced_tagged(ds, i))
+ v->untag = v->members;
b53_write16(dev, B53_VLAN_PAGE,
B53_VLAN_PORT_DEF_TAG(i), def_vid);
+ }
/* Upon initial call we have not set-up any VLANs, but upon
* system resume, we need to restore all VLAN entries.
unsigned int port;
int ret;
+ /* Request bridge PVID untagged when DSA_TAG_PROTO_NONE is set
+ * which forces the CPU port to be tagged in all VLANs.
+ */
+ ds->untag_bridge_pvid = dev->tag_protocol == DSA_TAG_PROTO_NONE;
+
ret = b53_reset_switch(dev);
if (ret) {
dev_err(ds->dev, "failed to reset switch\n");
untagged = true;
vl->members |= BIT(port);
- if (untagged && !dsa_is_cpu_port(ds, port))
+ if (untagged && !b53_vlan_port_needs_forced_tagged(ds, port))
vl->untag |= BIT(port);
else
vl->untag &= ~BIT(port);
if (pvid == vlan->vid)
pvid = b53_default_pvid(dev);
- if (untagged && !dsa_is_cpu_port(ds, port))
+ if (untagged && !b53_vlan_port_needs_forced_tagged(ds, port))
vl->untag &= ~(BIT(port));
b53_set_vlan_entry(dev, vlan->vid, vl);
dev->priv = priv;
dev->ops = ops;
ds->ops = &b53_switch_ops;
- ds->untag_bridge_pvid = true;
dev->vlan_enabled = true;
/* Let DSA handle the case were multiple bridges span the same switch
* device and different VLAN awareness settings are requested, which
struct b53_srab_priv *priv = dev->priv;
b53_srab_intr_set(priv, false);
- if (dev)
- b53_switch_remove(dev);
+ b53_switch_remove(dev);
return 0;
}
bcm_sf2_sw_mac_link_set(ds, port, interface, true);
if (port != core_readl(priv, CORE_IMP0_PRT_ID)) {
- u32 reg_rgmii_ctrl;
+ u32 reg_rgmii_ctrl = 0;
u32 reg, offset;
- reg_rgmii_ctrl = bcm_sf2_reg_rgmii_cntrl(priv, port);
-
if (priv->type == BCM4908_DEVICE_ID ||
priv->type == BCM7445_DEVICE_ID)
offset = CORE_STS_OVERRIDE_GMIIP_PORT(port);
interface == PHY_INTERFACE_MODE_RGMII_TXID ||
interface == PHY_INTERFACE_MODE_MII ||
interface == PHY_INTERFACE_MODE_REVMII) {
+ reg_rgmii_ctrl = bcm_sf2_reg_rgmii_cntrl(priv, port);
reg = reg_readl(priv, reg_rgmii_ctrl);
reg &= ~(RX_PAUSE_EN | TX_PAUSE_EN);
/* Read table */
for (i = 0; i < hellcreek->fdb_entries; ++i) {
- unsigned char null_addr[ETH_ALEN] = { 0 };
struct hellcreek_fdb_entry entry = { 0 };
/* Read entry */
hellcreek_write(hellcreek, 0x00, HR_FDBRDH);
/* Check valid */
- if (!memcmp(entry.mac, null_addr, ETH_ALEN))
+ if (is_zero_ether_addr(entry.mac))
continue;
/* Check port mask */
* Tristram Ha <Tristram.Ha@microchip.com>
*/
+#include <linux/bitfield.h>
#include <linux/delay.h>
#include <linux/export.h>
#include <linux/gpio.h>
#include <linux/phy.h>
#include <linux/etherdevice.h>
#include <linux/if_bridge.h>
+#include <linux/micrel_phy.h>
#include <net/dsa.h>
#include <net/switchdev.h>
+#include <linux/phylink.h>
#include "ksz_common.h"
#include "ksz8795_reg.h"
u8 restart, speed, ctrl, link;
const u8 *regs = ksz8->regs;
int processed = true;
+ u8 val1, val2;
u16 data = 0;
u8 p = phy;
switch (reg) {
- case PHY_REG_CTRL:
+ case MII_BMCR:
ksz_pread8(dev, p, regs[P_NEG_RESTART_CTRL], &restart);
ksz_pread8(dev, p, regs[P_SPEED_STATUS], &speed);
ksz_pread8(dev, p, regs[P_FORCE_CTRL], &ctrl);
if (restart & PORT_PHY_LOOPBACK)
- data |= PHY_LOOPBACK;
+ data |= BMCR_LOOPBACK;
if (ctrl & PORT_FORCE_100_MBIT)
- data |= PHY_SPEED_100MBIT;
+ data |= BMCR_SPEED100;
if (ksz_is_ksz88x3(dev)) {
if ((ctrl & PORT_AUTO_NEG_ENABLE))
- data |= PHY_AUTO_NEG_ENABLE;
+ data |= BMCR_ANENABLE;
} else {
if (!(ctrl & PORT_AUTO_NEG_DISABLE))
- data |= PHY_AUTO_NEG_ENABLE;
+ data |= BMCR_ANENABLE;
}
if (restart & PORT_POWER_DOWN)
- data |= PHY_POWER_DOWN;
+ data |= BMCR_PDOWN;
if (restart & PORT_AUTO_NEG_RESTART)
- data |= PHY_AUTO_NEG_RESTART;
+ data |= BMCR_ANRESTART;
if (ctrl & PORT_FORCE_FULL_DUPLEX)
- data |= PHY_FULL_DUPLEX;
+ data |= BMCR_FULLDPLX;
if (speed & PORT_HP_MDIX)
- data |= PHY_HP_MDIX;
+ data |= KSZ886X_BMCR_HP_MDIX;
if (restart & PORT_FORCE_MDIX)
- data |= PHY_FORCE_MDIX;
+ data |= KSZ886X_BMCR_FORCE_MDI;
if (restart & PORT_AUTO_MDIX_DISABLE)
- data |= PHY_AUTO_MDIX_DISABLE;
+ data |= KSZ886X_BMCR_DISABLE_AUTO_MDIX;
if (restart & PORT_TX_DISABLE)
- data |= PHY_TRANSMIT_DISABLE;
+ data |= KSZ886X_BMCR_DISABLE_TRANSMIT;
if (restart & PORT_LED_OFF)
- data |= PHY_LED_DISABLE;
+ data |= KSZ886X_BMCR_DISABLE_LED;
break;
- case PHY_REG_STATUS:
+ case MII_BMSR:
ksz_pread8(dev, p, regs[P_LINK_STATUS], &link);
- data = PHY_100BTX_FD_CAPABLE |
- PHY_100BTX_CAPABLE |
- PHY_10BT_FD_CAPABLE |
- PHY_10BT_CAPABLE |
- PHY_AUTO_NEG_CAPABLE;
+ data = BMSR_100FULL |
+ BMSR_100HALF |
+ BMSR_10FULL |
+ BMSR_10HALF |
+ BMSR_ANEGCAPABLE;
if (link & PORT_AUTO_NEG_COMPLETE)
- data |= PHY_AUTO_NEG_ACKNOWLEDGE;
+ data |= BMSR_ANEGCOMPLETE;
if (link & PORT_STAT_LINK_GOOD)
- data |= PHY_LINK_STATUS;
+ data |= BMSR_LSTATUS;
break;
- case PHY_REG_ID_1:
+ case MII_PHYSID1:
data = KSZ8795_ID_HI;
break;
- case PHY_REG_ID_2:
+ case MII_PHYSID2:
if (ksz_is_ksz88x3(dev))
data = KSZ8863_ID_LO;
else
data = KSZ8795_ID_LO;
break;
- case PHY_REG_AUTO_NEGOTIATION:
+ case MII_ADVERTISE:
ksz_pread8(dev, p, regs[P_LOCAL_CTRL], &ctrl);
- data = PHY_AUTO_NEG_802_3;
+ data = ADVERTISE_CSMA;
if (ctrl & PORT_AUTO_NEG_SYM_PAUSE)
- data |= PHY_AUTO_NEG_SYM_PAUSE;
+ data |= ADVERTISE_PAUSE_CAP;
if (ctrl & PORT_AUTO_NEG_100BTX_FD)
- data |= PHY_AUTO_NEG_100BTX_FD;
+ data |= ADVERTISE_100FULL;
if (ctrl & PORT_AUTO_NEG_100BTX)
- data |= PHY_AUTO_NEG_100BTX;
+ data |= ADVERTISE_100HALF;
if (ctrl & PORT_AUTO_NEG_10BT_FD)
- data |= PHY_AUTO_NEG_10BT_FD;
+ data |= ADVERTISE_10FULL;
if (ctrl & PORT_AUTO_NEG_10BT)
- data |= PHY_AUTO_NEG_10BT;
+ data |= ADVERTISE_10HALF;
break;
- case PHY_REG_REMOTE_CAPABILITY:
+ case MII_LPA:
ksz_pread8(dev, p, regs[P_REMOTE_STATUS], &link);
- data = PHY_AUTO_NEG_802_3;
+ data = LPA_SLCT;
if (link & PORT_REMOTE_SYM_PAUSE)
- data |= PHY_AUTO_NEG_SYM_PAUSE;
+ data |= LPA_PAUSE_CAP;
if (link & PORT_REMOTE_100BTX_FD)
- data |= PHY_AUTO_NEG_100BTX_FD;
+ data |= LPA_100FULL;
if (link & PORT_REMOTE_100BTX)
- data |= PHY_AUTO_NEG_100BTX;
+ data |= LPA_100HALF;
if (link & PORT_REMOTE_10BT_FD)
- data |= PHY_AUTO_NEG_10BT_FD;
+ data |= LPA_10FULL;
if (link & PORT_REMOTE_10BT)
- data |= PHY_AUTO_NEG_10BT;
- if (data & ~PHY_AUTO_NEG_802_3)
- data |= PHY_REMOTE_ACKNOWLEDGE_NOT;
+ data |= LPA_10HALF;
+ if (data & ~LPA_SLCT)
+ data |= LPA_LPACK;
+ break;
+ case PHY_REG_LINK_MD:
+ ksz_pread8(dev, p, REG_PORT_LINK_MD_CTRL, &val1);
+ ksz_pread8(dev, p, REG_PORT_LINK_MD_RESULT, &val2);
+ if (val1 & PORT_START_CABLE_DIAG)
+ data |= PHY_START_CABLE_DIAG;
+
+ if (val1 & PORT_CABLE_10M_SHORT)
+ data |= PHY_CABLE_10M_SHORT;
+
+ data |= FIELD_PREP(PHY_CABLE_DIAG_RESULT_M,
+ FIELD_GET(PORT_CABLE_DIAG_RESULT_M, val1));
+
+ data |= FIELD_PREP(PHY_CABLE_FAULT_COUNTER_M,
+ (FIELD_GET(PORT_CABLE_FAULT_COUNTER_H, val1) << 8) |
+ FIELD_GET(PORT_CABLE_FAULT_COUNTER_L, val2));
+ break;
+ case PHY_REG_PHY_CTRL:
+ ksz_pread8(dev, p, regs[P_LINK_STATUS], &link);
+ if (link & PORT_MDIX_STATUS)
+ data |= KSZ886X_CTRL_MDIX_STAT;
break;
default:
processed = false;
u8 p = phy;
switch (reg) {
- case PHY_REG_CTRL:
+ case MII_BMCR:
/* Do not support PHY reset function. */
- if (val & PHY_RESET)
+ if (val & BMCR_RESET)
break;
ksz_pread8(dev, p, regs[P_SPEED_STATUS], &speed);
data = speed;
- if (val & PHY_HP_MDIX)
+ if (val & KSZ886X_BMCR_HP_MDIX)
data |= PORT_HP_MDIX;
else
data &= ~PORT_HP_MDIX;
ksz_pread8(dev, p, regs[P_FORCE_CTRL], &ctrl);
data = ctrl;
if (ksz_is_ksz88x3(dev)) {
- if ((val & PHY_AUTO_NEG_ENABLE))
+ if ((val & BMCR_ANENABLE))
data |= PORT_AUTO_NEG_ENABLE;
else
data &= ~PORT_AUTO_NEG_ENABLE;
} else {
- if (!(val & PHY_AUTO_NEG_ENABLE))
+ if (!(val & BMCR_ANENABLE))
data |= PORT_AUTO_NEG_DISABLE;
else
data &= ~PORT_AUTO_NEG_DISABLE;
data |= PORT_AUTO_NEG_DISABLE;
}
- if (val & PHY_SPEED_100MBIT)
+ if (val & BMCR_SPEED100)
data |= PORT_FORCE_100_MBIT;
else
data &= ~PORT_FORCE_100_MBIT;
- if (val & PHY_FULL_DUPLEX)
+ if (val & BMCR_FULLDPLX)
data |= PORT_FORCE_FULL_DUPLEX;
else
data &= ~PORT_FORCE_FULL_DUPLEX;
ksz_pwrite8(dev, p, regs[P_FORCE_CTRL], data);
ksz_pread8(dev, p, regs[P_NEG_RESTART_CTRL], &restart);
data = restart;
- if (val & PHY_LED_DISABLE)
+ if (val & KSZ886X_BMCR_DISABLE_LED)
data |= PORT_LED_OFF;
else
data &= ~PORT_LED_OFF;
- if (val & PHY_TRANSMIT_DISABLE)
+ if (val & KSZ886X_BMCR_DISABLE_TRANSMIT)
data |= PORT_TX_DISABLE;
else
data &= ~PORT_TX_DISABLE;
- if (val & PHY_AUTO_NEG_RESTART)
+ if (val & BMCR_ANRESTART)
data |= PORT_AUTO_NEG_RESTART;
else
data &= ~(PORT_AUTO_NEG_RESTART);
- if (val & PHY_POWER_DOWN)
+ if (val & BMCR_PDOWN)
data |= PORT_POWER_DOWN;
else
data &= ~PORT_POWER_DOWN;
- if (val & PHY_AUTO_MDIX_DISABLE)
+ if (val & KSZ886X_BMCR_DISABLE_AUTO_MDIX)
data |= PORT_AUTO_MDIX_DISABLE;
else
data &= ~PORT_AUTO_MDIX_DISABLE;
- if (val & PHY_FORCE_MDIX)
+ if (val & KSZ886X_BMCR_FORCE_MDI)
data |= PORT_FORCE_MDIX;
else
data &= ~PORT_FORCE_MDIX;
- if (val & PHY_LOOPBACK)
+ if (val & BMCR_LOOPBACK)
data |= PORT_PHY_LOOPBACK;
else
data &= ~PORT_PHY_LOOPBACK;
if (data != restart)
ksz_pwrite8(dev, p, regs[P_NEG_RESTART_CTRL], data);
break;
- case PHY_REG_AUTO_NEGOTIATION:
+ case MII_ADVERTISE:
ksz_pread8(dev, p, regs[P_LOCAL_CTRL], &ctrl);
data = ctrl;
data &= ~(PORT_AUTO_NEG_SYM_PAUSE |
PORT_AUTO_NEG_100BTX |
PORT_AUTO_NEG_10BT_FD |
PORT_AUTO_NEG_10BT);
- if (val & PHY_AUTO_NEG_SYM_PAUSE)
+ if (val & ADVERTISE_PAUSE_CAP)
data |= PORT_AUTO_NEG_SYM_PAUSE;
- if (val & PHY_AUTO_NEG_100BTX_FD)
+ if (val & ADVERTISE_100FULL)
data |= PORT_AUTO_NEG_100BTX_FD;
- if (val & PHY_AUTO_NEG_100BTX)
+ if (val & ADVERTISE_100HALF)
data |= PORT_AUTO_NEG_100BTX;
- if (val & PHY_AUTO_NEG_10BT_FD)
+ if (val & ADVERTISE_10FULL)
data |= PORT_AUTO_NEG_10BT_FD;
- if (val & PHY_AUTO_NEG_10BT)
+ if (val & ADVERTISE_10HALF)
data |= PORT_AUTO_NEG_10BT;
if (data != ctrl)
ksz_pwrite8(dev, p, regs[P_LOCAL_CTRL], data);
break;
+ case PHY_REG_LINK_MD:
+ if (val & PHY_START_CABLE_DIAG)
+ ksz_port_cfg(dev, p, REG_PORT_LINK_MD_CTRL, PORT_START_CABLE_DIAG, true);
+ break;
default:
break;
}
DSA_TAG_PROTO_KSZ9893 : DSA_TAG_PROTO_KSZ8795;
}
+static u32 ksz8_sw_get_phy_flags(struct dsa_switch *ds, int port)
+{
+ /* Silicon Errata Sheet (DS80000830A):
+ * Port 1 does not work with LinkMD Cable-Testing.
+ * Port 1 does not respond to received PAUSE control frames.
+ */
+ if (!port)
+ return MICREL_KSZ8_P1_ERRATA;
+
+ return 0;
+}
+
static void ksz8_get_strings(struct dsa_switch *ds, int port,
u32 stringset, uint8_t *buf)
{
return 0;
}
+static void ksz8_validate(struct dsa_switch *ds, int port,
+ unsigned long *supported,
+ struct phylink_link_state *state)
+{
+ __ETHTOOL_DECLARE_LINK_MODE_MASK(mask) = { 0, };
+ struct ksz_device *dev = ds->priv;
+
+ if (port == dev->cpu_port) {
+ if (state->interface != PHY_INTERFACE_MODE_RMII &&
+ state->interface != PHY_INTERFACE_MODE_MII &&
+ state->interface != PHY_INTERFACE_MODE_NA)
+ goto unsupported;
+ } else {
+ if (state->interface != PHY_INTERFACE_MODE_INTERNAL &&
+ state->interface != PHY_INTERFACE_MODE_NA)
+ goto unsupported;
+ }
+
+ /* Allow all the expected bits */
+ phylink_set_port_modes(mask);
+ phylink_set(mask, Autoneg);
+
+ /* Silicon Errata Sheet (DS80000830A):
+ * "Port 1 does not respond to received flow control PAUSE frames"
+ * So, disable Pause support on "Port 1" (port == 0) for all ksz88x3
+ * switches.
+ */
+ if (!ksz_is_ksz88x3(dev) || port)
+ phylink_set(mask, Pause);
+
+ /* Asym pause is not supported on KSZ8863 and KSZ8873 */
+ if (!ksz_is_ksz88x3(dev))
+ phylink_set(mask, Asym_Pause);
+
+ /* 10M and 100M are only supported */
+ phylink_set(mask, 10baseT_Half);
+ phylink_set(mask, 10baseT_Full);
+ phylink_set(mask, 100baseT_Half);
+ phylink_set(mask, 100baseT_Full);
+
+ bitmap_and(supported, supported, mask,
+ __ETHTOOL_LINK_MODE_MASK_NBITS);
+ bitmap_and(state->advertising, state->advertising, mask,
+ __ETHTOOL_LINK_MODE_MASK_NBITS);
+
+ return;
+
+unsupported:
+ bitmap_zero(supported, __ETHTOOL_LINK_MODE_MASK_NBITS);
+ dev_err(ds->dev, "Unsupported interface: %s, port: %d\n",
+ phy_modes(state->interface), port);
+}
+
static const struct dsa_switch_ops ksz8_switch_ops = {
.get_tag_protocol = ksz8_get_tag_protocol,
+ .get_phy_flags = ksz8_sw_get_phy_flags,
.setup = ksz8_setup,
.phy_read = ksz_phy_read16,
.phy_write = ksz_phy_write16,
+ .phylink_validate = ksz8_validate,
.phylink_mac_link_down = ksz_mac_link_down,
.port_enable = ksz_enable_port,
.get_strings = ksz8_get_strings,
#define REG_PORT_4_LINK_MD_CTRL 0x4A
#define PORT_CABLE_10M_SHORT BIT(7)
-#define PORT_CABLE_DIAG_RESULT_M 0x3
+#define PORT_CABLE_DIAG_RESULT_M GENMASK(6, 5)
#define PORT_CABLE_DIAG_RESULT_S 5
#define PORT_CABLE_STAT_NORMAL 0
#define PORT_CABLE_STAT_OPEN 1
#define PORT_ACL_FORCE_DLR_MISS BIT(0)
-#ifndef PHY_REG_CTRL
-#define PHY_REG_CTRL 0
-
-#define PHY_RESET BIT(15)
-#define PHY_LOOPBACK BIT(14)
-#define PHY_SPEED_100MBIT BIT(13)
-#define PHY_AUTO_NEG_ENABLE BIT(12)
-#define PHY_POWER_DOWN BIT(11)
-#define PHY_MII_DISABLE BIT(10)
-#define PHY_AUTO_NEG_RESTART BIT(9)
-#define PHY_FULL_DUPLEX BIT(8)
-#define PHY_COLLISION_TEST_NOT BIT(7)
-#define PHY_HP_MDIX BIT(5)
-#define PHY_FORCE_MDIX BIT(4)
-#define PHY_AUTO_MDIX_DISABLE BIT(3)
-#define PHY_REMOTE_FAULT_DISABLE BIT(2)
-#define PHY_TRANSMIT_DISABLE BIT(1)
-#define PHY_LED_DISABLE BIT(0)
-
-#define PHY_REG_STATUS 1
-
-#define PHY_100BT4_CAPABLE BIT(15)
-#define PHY_100BTX_FD_CAPABLE BIT(14)
-#define PHY_100BTX_CAPABLE BIT(13)
-#define PHY_10BT_FD_CAPABLE BIT(12)
-#define PHY_10BT_CAPABLE BIT(11)
-#define PHY_MII_SUPPRESS_CAPABLE_NOT BIT(6)
-#define PHY_AUTO_NEG_ACKNOWLEDGE BIT(5)
-#define PHY_REMOTE_FAULT BIT(4)
-#define PHY_AUTO_NEG_CAPABLE BIT(3)
-#define PHY_LINK_STATUS BIT(2)
-#define PHY_JABBER_DETECT_NOT BIT(1)
-#define PHY_EXTENDED_CAPABILITY BIT(0)
-
-#define PHY_REG_ID_1 2
-#define PHY_REG_ID_2 3
-
-#define PHY_REG_AUTO_NEGOTIATION 4
-
-#define PHY_AUTO_NEG_NEXT_PAGE_NOT BIT(15)
-#define PHY_AUTO_NEG_REMOTE_FAULT_NOT BIT(13)
-#define PHY_AUTO_NEG_SYM_PAUSE BIT(10)
-#define PHY_AUTO_NEG_100BT4 BIT(9)
-#define PHY_AUTO_NEG_100BTX_FD BIT(8)
-#define PHY_AUTO_NEG_100BTX BIT(7)
-#define PHY_AUTO_NEG_10BT_FD BIT(6)
-#define PHY_AUTO_NEG_10BT BIT(5)
-#define PHY_AUTO_NEG_SELECTOR 0x001F
-#define PHY_AUTO_NEG_802_3 0x0001
-
-#define PHY_REG_REMOTE_CAPABILITY 5
-
-#define PHY_REMOTE_NEXT_PAGE_NOT BIT(15)
-#define PHY_REMOTE_ACKNOWLEDGE_NOT BIT(14)
-#define PHY_REMOTE_REMOTE_FAULT_NOT BIT(13)
-#define PHY_REMOTE_SYM_PAUSE BIT(10)
-#define PHY_REMOTE_100BTX_FD BIT(8)
-#define PHY_REMOTE_100BTX BIT(7)
-#define PHY_REMOTE_10BT_FD BIT(6)
-#define PHY_REMOTE_10BT BIT(5)
-#endif
-
#define KSZ8795_ID_HI 0x0022
#define KSZ8795_ID_LO 0x1550
#define KSZ8863_ID_LO 0x1430
#define PHY_REG_LINK_MD 0x1D
#define PHY_START_CABLE_DIAG BIT(15)
+#define PHY_CABLE_DIAG_RESULT_M GENMASK(14, 13)
#define PHY_CABLE_DIAG_RESULT 0x6000
#define PHY_CABLE_STAT_NORMAL 0x0000
#define PHY_CABLE_STAT_OPEN 0x2000
#define PHY_CABLE_STAT_SHORT 0x4000
#define PHY_CABLE_STAT_FAILED 0x6000
#define PHY_CABLE_10M_SHORT BIT(12)
-#define PHY_CABLE_FAULT_COUNTER 0x01FF
+#define PHY_CABLE_FAULT_COUNTER_M GENMASK(8, 0)
#define PHY_REG_PHY_CTRL 0x1F
.num_statics = 16,
.cpu_ports = 0x7F, /* can be configured as cpu port */
.port_cnt = 7, /* total physical port count */
+ .phy_errata_9477 = true,
},
};
#include <linux/mfd/syscon.h>
#include <linux/module.h>
#include <linux/netdevice.h>
+#include <linux/of_irq.h>
#include <linux/of_mdio.h>
#include <linux/of_net.h>
#include <linux/of_platform.h>
mt7530_write(priv, MT7530_MIB_CCR, CCR_MIB_ACTIVATE);
}
-static int mt7530_phy_read(struct dsa_switch *ds, int port, int regnum)
+static int mt7530_phy_read(struct mt7530_priv *priv, int port, int regnum)
{
- struct mt7530_priv *priv = ds->priv;
-
return mdiobus_read_nested(priv->bus, port, regnum);
}
-static int mt7530_phy_write(struct dsa_switch *ds, int port, int regnum,
+static int mt7530_phy_write(struct mt7530_priv *priv, int port, int regnum,
u16 val)
{
- struct mt7530_priv *priv = ds->priv;
-
return mdiobus_write_nested(priv->bus, port, regnum, val);
}
}
static int
-mt7531_ind_phy_read(struct dsa_switch *ds, int port, int regnum)
+mt7531_ind_phy_read(struct mt7530_priv *priv, int port, int regnum)
{
- struct mt7530_priv *priv = ds->priv;
int devad;
int ret;
}
static int
-mt7531_ind_phy_write(struct dsa_switch *ds, int port, int regnum,
+mt7531_ind_phy_write(struct mt7530_priv *priv, int port, int regnum,
u16 data)
{
- struct mt7530_priv *priv = ds->priv;
int devad;
int ret;
return ret;
}
+static int
+mt753x_phy_read(struct mii_bus *bus, int port, int regnum)
+{
+ struct mt7530_priv *priv = bus->priv;
+
+ return priv->info->phy_read(priv, port, regnum);
+}
+
+static int
+mt753x_phy_write(struct mii_bus *bus, int port, int regnum, u16 val)
+{
+ struct mt7530_priv *priv = bus->priv;
+
+ return priv->info->phy_write(priv, port, regnum, val);
+}
+
static void
mt7530_get_strings(struct dsa_switch *ds, int port, u32 stringset,
uint8_t *data)
{
struct mt7530_priv *priv = ds->priv;
- /* The real fabric path would be decided on the membership in the
- * entry of VLAN table. PCR_MATRIX set up here with ALL_MEMBERS
- * means potential VLAN can be consisting of certain subset of all
- * ports.
- */
- mt7530_rmw(priv, MT7530_PCR_P(port),
- PCR_MATRIX_MASK, PCR_MATRIX(MT7530_ALL_MEMBERS));
-
/* Trapped into security mode allows packet forwarding through VLAN
* table lookup. CPU port is set to fallback mode to let untagged
* frames pass through.
}
#endif /* CONFIG_GPIOLIB */
+static irqreturn_t
+mt7530_irq_thread_fn(int irq, void *dev_id)
+{
+ struct mt7530_priv *priv = dev_id;
+ bool handled = false;
+ u32 val;
+ int p;
+
+ mutex_lock_nested(&priv->bus->mdio_lock, MDIO_MUTEX_NESTED);
+ val = mt7530_mii_read(priv, MT7530_SYS_INT_STS);
+ mt7530_mii_write(priv, MT7530_SYS_INT_STS, val);
+ mutex_unlock(&priv->bus->mdio_lock);
+
+ for (p = 0; p < MT7530_NUM_PHYS; p++) {
+ if (BIT(p) & val) {
+ unsigned int irq;
+
+ irq = irq_find_mapping(priv->irq_domain, p);
+ handle_nested_irq(irq);
+ handled = true;
+ }
+ }
+
+ return IRQ_RETVAL(handled);
+}
+
+static void
+mt7530_irq_mask(struct irq_data *d)
+{
+ struct mt7530_priv *priv = irq_data_get_irq_chip_data(d);
+
+ priv->irq_enable &= ~BIT(d->hwirq);
+}
+
+static void
+mt7530_irq_unmask(struct irq_data *d)
+{
+ struct mt7530_priv *priv = irq_data_get_irq_chip_data(d);
+
+ priv->irq_enable |= BIT(d->hwirq);
+}
+
+static void
+mt7530_irq_bus_lock(struct irq_data *d)
+{
+ struct mt7530_priv *priv = irq_data_get_irq_chip_data(d);
+
+ mutex_lock_nested(&priv->bus->mdio_lock, MDIO_MUTEX_NESTED);
+}
+
+static void
+mt7530_irq_bus_sync_unlock(struct irq_data *d)
+{
+ struct mt7530_priv *priv = irq_data_get_irq_chip_data(d);
+
+ mt7530_mii_write(priv, MT7530_SYS_INT_EN, priv->irq_enable);
+ mutex_unlock(&priv->bus->mdio_lock);
+}
+
+static struct irq_chip mt7530_irq_chip = {
+ .name = KBUILD_MODNAME,
+ .irq_mask = mt7530_irq_mask,
+ .irq_unmask = mt7530_irq_unmask,
+ .irq_bus_lock = mt7530_irq_bus_lock,
+ .irq_bus_sync_unlock = mt7530_irq_bus_sync_unlock,
+};
+
+static int
+mt7530_irq_map(struct irq_domain *domain, unsigned int irq,
+ irq_hw_number_t hwirq)
+{
+ irq_set_chip_data(irq, domain->host_data);
+ irq_set_chip_and_handler(irq, &mt7530_irq_chip, handle_simple_irq);
+ irq_set_nested_thread(irq, true);
+ irq_set_noprobe(irq);
+
+ return 0;
+}
+
+static const struct irq_domain_ops mt7530_irq_domain_ops = {
+ .map = mt7530_irq_map,
+ .xlate = irq_domain_xlate_onecell,
+};
+
+static void
+mt7530_setup_mdio_irq(struct mt7530_priv *priv)
+{
+ struct dsa_switch *ds = priv->ds;
+ int p;
+
+ for (p = 0; p < MT7530_NUM_PHYS; p++) {
+ if (BIT(p) & ds->phys_mii_mask) {
+ unsigned int irq;
+
+ irq = irq_create_mapping(priv->irq_domain, p);
+ ds->slave_mii_bus->irq[p] = irq;
+ }
+ }
+}
+
+static int
+mt7530_setup_irq(struct mt7530_priv *priv)
+{
+ struct device *dev = priv->dev;
+ struct device_node *np = dev->of_node;
+ int ret;
+
+ if (!of_property_read_bool(np, "interrupt-controller")) {
+ dev_info(dev, "no interrupt support\n");
+ return 0;
+ }
+
+ priv->irq = of_irq_get(np, 0);
+ if (priv->irq <= 0) {
+ dev_err(dev, "failed to get parent IRQ: %d\n", priv->irq);
+ return priv->irq ? : -EINVAL;
+ }
+
+ priv->irq_domain = irq_domain_add_linear(np, MT7530_NUM_PHYS,
+ &mt7530_irq_domain_ops, priv);
+ if (!priv->irq_domain) {
+ dev_err(dev, "failed to create IRQ domain\n");
+ return -ENOMEM;
+ }
+
+ /* This register must be set for MT7530 to properly fire interrupts */
+ if (priv->id != ID_MT7531)
+ mt7530_set(priv, MT7530_TOP_SIG_CTRL, TOP_SIG_CTRL_NORMAL);
+
+ ret = request_threaded_irq(priv->irq, NULL, mt7530_irq_thread_fn,
+ IRQF_ONESHOT, KBUILD_MODNAME, priv);
+ if (ret) {
+ irq_domain_remove(priv->irq_domain);
+ dev_err(dev, "failed to request IRQ: %d\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+static void
+mt7530_free_mdio_irq(struct mt7530_priv *priv)
+{
+ int p;
+
+ for (p = 0; p < MT7530_NUM_PHYS; p++) {
+ if (BIT(p) & priv->ds->phys_mii_mask) {
+ unsigned int irq;
+
+ irq = irq_find_mapping(priv->irq_domain, p);
+ irq_dispose_mapping(irq);
+ }
+ }
+}
+
+static void
+mt7530_free_irq_common(struct mt7530_priv *priv)
+{
+ free_irq(priv->irq, priv);
+ irq_domain_remove(priv->irq_domain);
+}
+
+static void
+mt7530_free_irq(struct mt7530_priv *priv)
+{
+ mt7530_free_mdio_irq(priv);
+ mt7530_free_irq_common(priv);
+}
+
+static int
+mt7530_setup_mdio(struct mt7530_priv *priv)
+{
+ struct dsa_switch *ds = priv->ds;
+ struct device *dev = priv->dev;
+ struct mii_bus *bus;
+ static int idx;
+ int ret;
+
+ bus = devm_mdiobus_alloc(dev);
+ if (!bus)
+ return -ENOMEM;
+
+ ds->slave_mii_bus = bus;
+ bus->priv = priv;
+ bus->name = KBUILD_MODNAME "-mii";
+ snprintf(bus->id, MII_BUS_ID_SIZE, KBUILD_MODNAME "-%d", idx++);
+ bus->read = mt753x_phy_read;
+ bus->write = mt753x_phy_write;
+ bus->parent = dev;
+ bus->phy_mask = ~ds->phys_mii_mask;
+
+ if (priv->irq)
+ mt7530_setup_mdio_irq(priv);
+
+ ret = mdiobus_register(bus);
+ if (ret) {
+ dev_err(dev, "failed to register MDIO bus: %d\n", ret);
+ if (priv->irq)
+ mt7530_free_mdio_irq(priv);
+ }
+
+ return ret;
+}
+
static int
mt7530_setup(struct dsa_switch *ds)
{
mt753x_setup(struct dsa_switch *ds)
{
struct mt7530_priv *priv = ds->priv;
+ int ret = priv->info->sw_setup(ds);
- return priv->info->sw_setup(ds);
-}
-
-static int
-mt753x_phy_read(struct dsa_switch *ds, int port, int regnum)
-{
- struct mt7530_priv *priv = ds->priv;
+ if (ret)
+ return ret;
- return priv->info->phy_read(ds, port, regnum);
-}
+ ret = mt7530_setup_irq(priv);
+ if (ret)
+ return ret;
-static int
-mt753x_phy_write(struct dsa_switch *ds, int port, int regnum, u16 val)
-{
- struct mt7530_priv *priv = ds->priv;
+ ret = mt7530_setup_mdio(priv);
+ if (ret && priv->irq)
+ mt7530_free_irq_common(priv);
- return priv->info->phy_write(ds, port, regnum, val);
+ return ret;
}
static int mt753x_get_mac_eee(struct dsa_switch *ds, int port,
.get_tag_protocol = mtk_get_tag_protocol,
.setup = mt753x_setup,
.get_strings = mt7530_get_strings,
- .phy_read = mt753x_phy_read,
- .phy_write = mt753x_phy_write,
.get_ethtool_stats = mt7530_get_ethtool_stats,
.get_sset_count = mt7530_get_sset_count,
.set_ageing_time = mt7530_set_ageing_time,
dev_err(priv->dev, "Failed to disable io pwr: %d\n",
ret);
+ if (priv->irq)
+ mt7530_free_irq(priv);
+
dsa_unregister_switch(priv->ds);
mutex_destroy(&priv->reg_mutex);
}
#define __MT7530_H
#define MT7530_NUM_PORTS 7
+#define MT7530_NUM_PHYS 5
#define MT7530_CPU_PORT 6
#define MT7530_NUM_FDB_RECORDS 2048
#define MT7530_ALL_MEMBERS 0xff
#define SYS_CTRL_SW_RST BIT(1)
#define SYS_CTRL_REG_RST BIT(0)
+/* Register for system interrupt */
+#define MT7530_SYS_INT_EN 0x7008
+
+/* Register for system interrupt status */
+#define MT7530_SYS_INT_STS 0x700c
+
/* Register for PHY Indirect Access Control */
#define MT7531_PHY_IAC 0x701C
#define MT7531_PHY_ACS_ST BIT(31)
}
}
+struct mt7530_priv;
+
/* struct mt753x_info - This is the main data structure for holding the specific
* part for each supported device
* @sw_setup: Holding the handler to a device initialization
enum mt753x_id id;
int (*sw_setup)(struct dsa_switch *ds);
- int (*phy_read)(struct dsa_switch *ds, int port, int regnum);
- int (*phy_write)(struct dsa_switch *ds, int port, int regnum, u16 val);
+ int (*phy_read)(struct mt7530_priv *priv, int port, int regnum);
+ int (*phy_write)(struct mt7530_priv *priv, int port, int regnum, u16 val);
int (*pad_setup)(struct dsa_switch *ds, phy_interface_t interface);
int (*cpu_port_config)(struct dsa_switch *ds, int port);
bool (*phy_mode_supported)(struct dsa_switch *ds, int port,
* registers
* @p6_interface Holding the current port 6 interface
* @p5_intf_sel: Holding the current port 5 interface select
+ *
+ * @irq: IRQ number of the switch
+ * @irq_domain: IRQ domain of the switch irq_chip
+ * @irq_enable: IRQ enable bits, synced to SYS_INT_EN
*/
struct mt7530_priv {
struct device *dev;
struct mt7530_port ports[MT7530_NUM_PORTS];
/* protect among processes for registers access*/
struct mutex reg_mutex;
+ int irq;
+ struct irq_domain *irq_domain;
+ u32 irq_enable;
};
struct mt7530_hw_vlan_entry {
ocelot_write_rix(ocelot, 0, ANA_POL_FLOWC, port);
+ ocelot_fields_write(ocelot, port, SYS_PAUSE_CFG_PAUSE_ENA, tx_pause);
+
/* Undo the effects of felix_phylink_mac_link_down:
* enable MAC module
*/
if (taprio->num_entries > VSC9959_TAS_GCL_ENTRY_MAX)
return -ERANGE;
- /* Set port num and disable ALWAYS_GUARD_BAND_SCH_Q, which means set
- * guard band to be implemented for nonschedule queues to schedule
- * queues transition.
+ /* Enable guard band. The switch will schedule frames without taking
+ * their length into account. Thus we'll always need to enable the
+ * guard band which reserves the time of a maximum sized frame at the
+ * end of the time window.
+ *
+ * Although the ALWAYS_GUARD_BAND_SCH_Q bit is global for all ports, we
+ * need to set PORT_NUM, because subsequent writes to PARAM_CFG_REG_n
+ * operate on the port number.
*/
- ocelot_rmw(ocelot,
- QSYS_TAS_PARAM_CFG_CTRL_PORT_NUM(port),
+ ocelot_rmw(ocelot, QSYS_TAS_PARAM_CFG_CTRL_PORT_NUM(port) |
+ QSYS_TAS_PARAM_CFG_CTRL_ALWAYS_GUARD_BAND_SCH_Q,
QSYS_TAS_PARAM_CFG_CTRL_PORT_NUM_M |
QSYS_TAS_PARAM_CFG_CTRL_ALWAYS_GUARD_BAND_SCH_Q,
QSYS_TAS_PARAM_CFG_CTRL);
felix->info = &seville_info_vsc9953;
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!res) {
+ err = -EINVAL;
+ dev_err(&pdev->dev, "Invalid resource\n");
+ goto err_alloc_felix;
+ }
felix->switch_base = res->start;
ds = kzalloc(sizeof(struct dsa_switch), GFP_KERNEL);
#include <linux/netdevice.h>
#include <net/dsa.h>
#include <linux/of_net.h>
+#include <linux/of_mdio.h>
#include <linux/of_platform.h>
#include <linux/if_bridge.h>
#include <linux/mdio.h>
*page = regaddr & 0x3ff;
}
-static u32
-qca8k_mii_read32(struct mii_bus *bus, int phy_id, u32 regnum)
+static int
+qca8k_mii_read32(struct mii_bus *bus, int phy_id, u32 regnum, u32 *val)
{
- u32 val;
int ret;
ret = bus->read(bus, phy_id, regnum);
if (ret >= 0) {
- val = ret;
+ *val = ret;
ret = bus->read(bus, phy_id, regnum + 1);
- val |= ret << 16;
+ *val |= ret << 16;
}
if (ret < 0) {
dev_err_ratelimited(&bus->dev,
"failed to read qca8k 32bit register\n");
+ *val = 0;
return ret;
}
- return val;
+ return 0;
}
static void
"failed to write qca8k 32bit register\n");
}
-static void
+static int
qca8k_set_page(struct mii_bus *bus, u16 page)
{
+ int ret;
+
if (page == qca8k_current_page)
- return;
+ return 0;
- if (bus->write(bus, 0x18, 0, page) < 0)
+ ret = bus->write(bus, 0x18, 0, page);
+ if (ret < 0) {
dev_err_ratelimited(&bus->dev,
"failed to set qca8k page\n");
+ return ret;
+ }
+
qca8k_current_page = page;
+ usleep_range(1000, 2000);
+ return 0;
}
-static u32
-qca8k_read(struct qca8k_priv *priv, u32 reg)
+static int
+qca8k_read(struct qca8k_priv *priv, u32 reg, u32 *val)
{
+ struct mii_bus *bus = priv->bus;
u16 r1, r2, page;
- u32 val;
+ int ret;
qca8k_split_addr(reg, &r1, &r2, &page);
- mutex_lock_nested(&priv->bus->mdio_lock, MDIO_MUTEX_NESTED);
+ mutex_lock_nested(&bus->mdio_lock, MDIO_MUTEX_NESTED);
- qca8k_set_page(priv->bus, page);
- val = qca8k_mii_read32(priv->bus, 0x10 | r2, r1);
+ ret = qca8k_set_page(bus, page);
+ if (ret < 0)
+ goto exit;
- mutex_unlock(&priv->bus->mdio_lock);
+ ret = qca8k_mii_read32(bus, 0x10 | r2, r1, val);
- return val;
+exit:
+ mutex_unlock(&bus->mdio_lock);
+ return ret;
}
-static void
+static int
qca8k_write(struct qca8k_priv *priv, u32 reg, u32 val)
{
+ struct mii_bus *bus = priv->bus;
u16 r1, r2, page;
+ int ret;
qca8k_split_addr(reg, &r1, &r2, &page);
- mutex_lock_nested(&priv->bus->mdio_lock, MDIO_MUTEX_NESTED);
+ mutex_lock_nested(&bus->mdio_lock, MDIO_MUTEX_NESTED);
+
+ ret = qca8k_set_page(bus, page);
+ if (ret < 0)
+ goto exit;
- qca8k_set_page(priv->bus, page);
- qca8k_mii_write32(priv->bus, 0x10 | r2, r1, val);
+ qca8k_mii_write32(bus, 0x10 | r2, r1, val);
- mutex_unlock(&priv->bus->mdio_lock);
+exit:
+ mutex_unlock(&bus->mdio_lock);
+ return ret;
}
-static u32
-qca8k_rmw(struct qca8k_priv *priv, u32 reg, u32 mask, u32 val)
+static int
+qca8k_rmw(struct qca8k_priv *priv, u32 reg, u32 mask, u32 write_val)
{
+ struct mii_bus *bus = priv->bus;
u16 r1, r2, page;
- u32 ret;
+ u32 val;
+ int ret;
qca8k_split_addr(reg, &r1, &r2, &page);
- mutex_lock_nested(&priv->bus->mdio_lock, MDIO_MUTEX_NESTED);
+ mutex_lock_nested(&bus->mdio_lock, MDIO_MUTEX_NESTED);
+
+ ret = qca8k_set_page(bus, page);
+ if (ret < 0)
+ goto exit;
+
+ ret = qca8k_mii_read32(bus, 0x10 | r2, r1, &val);
+ if (ret < 0)
+ goto exit;
- qca8k_set_page(priv->bus, page);
- ret = qca8k_mii_read32(priv->bus, 0x10 | r2, r1);
- ret &= ~mask;
- ret |= val;
- qca8k_mii_write32(priv->bus, 0x10 | r2, r1, ret);
+ val &= ~mask;
+ val |= write_val;
+ qca8k_mii_write32(bus, 0x10 | r2, r1, val);
- mutex_unlock(&priv->bus->mdio_lock);
+exit:
+ mutex_unlock(&bus->mdio_lock);
return ret;
}
-static void
+static int
qca8k_reg_set(struct qca8k_priv *priv, u32 reg, u32 val)
{
- qca8k_rmw(priv, reg, 0, val);
+ return qca8k_rmw(priv, reg, 0, val);
}
-static void
+static int
qca8k_reg_clear(struct qca8k_priv *priv, u32 reg, u32 val)
{
- qca8k_rmw(priv, reg, val, 0);
+ return qca8k_rmw(priv, reg, val, 0);
}
static int
{
struct qca8k_priv *priv = (struct qca8k_priv *)ctx;
- *val = qca8k_read(priv, reg);
-
- return 0;
+ return qca8k_read(priv, reg, val);
}
static int
{
struct qca8k_priv *priv = (struct qca8k_priv *)ctx;
- qca8k_write(priv, reg, val);
-
- return 0;
+ return qca8k_write(priv, reg, val);
}
static const struct regmap_range qca8k_readable_ranges[] = {
static int
qca8k_busy_wait(struct qca8k_priv *priv, u32 reg, u32 mask)
{
- unsigned long timeout;
-
- timeout = jiffies + msecs_to_jiffies(20);
+ int ret, ret1;
+ u32 val;
- /* loop until the busy flag has cleared */
- do {
- u32 val = qca8k_read(priv, reg);
- int busy = val & mask;
+ ret = read_poll_timeout(qca8k_read, ret1, !(val & mask),
+ 0, QCA8K_BUSY_WAIT_TIMEOUT * USEC_PER_MSEC, false,
+ priv, reg, &val);
- if (!busy)
- break;
- cond_resched();
- } while (!time_after_eq(jiffies, timeout));
+ /* Check if qca8k_read has failed for a different reason
+ * before returning -ETIMEDOUT
+ */
+ if (ret < 0 && ret1 < 0)
+ return ret1;
- return time_after_eq(jiffies, timeout);
+ return ret;
}
-static void
+static int
qca8k_fdb_read(struct qca8k_priv *priv, struct qca8k_fdb *fdb)
{
- u32 reg[4];
- int i;
+ u32 reg[4], val;
+ int i, ret;
/* load the ARL table into an array */
- for (i = 0; i < 4; i++)
- reg[i] = qca8k_read(priv, QCA8K_REG_ATU_DATA0 + (i * 4));
+ for (i = 0; i < 4; i++) {
+ ret = qca8k_read(priv, QCA8K_REG_ATU_DATA0 + (i * 4), &val);
+ if (ret < 0)
+ return ret;
+
+ reg[i] = val;
+ }
/* vid - 83:72 */
fdb->vid = (reg[2] >> QCA8K_ATU_VID_S) & QCA8K_ATU_VID_M;
fdb->mac[3] = (reg[0] >> QCA8K_ATU_ADDR3_S) & 0xff;
fdb->mac[4] = (reg[0] >> QCA8K_ATU_ADDR4_S) & 0xff;
fdb->mac[5] = reg[0] & 0xff;
+
+ return 0;
}
static void
qca8k_fdb_access(struct qca8k_priv *priv, enum qca8k_fdb_cmd cmd, int port)
{
u32 reg;
+ int ret;
/* Set the command and FDB index */
reg = QCA8K_ATU_FUNC_BUSY;
}
/* Write the function register triggering the table access */
- qca8k_write(priv, QCA8K_REG_ATU_FUNC, reg);
+ ret = qca8k_write(priv, QCA8K_REG_ATU_FUNC, reg);
+ if (ret)
+ return ret;
/* wait for completion */
- if (qca8k_busy_wait(priv, QCA8K_REG_ATU_FUNC, QCA8K_ATU_FUNC_BUSY))
- return -1;
+ ret = qca8k_busy_wait(priv, QCA8K_REG_ATU_FUNC, QCA8K_ATU_FUNC_BUSY);
+ if (ret)
+ return ret;
/* Check for table full violation when adding an entry */
if (cmd == QCA8K_FDB_LOAD) {
- reg = qca8k_read(priv, QCA8K_REG_ATU_FUNC);
+ ret = qca8k_read(priv, QCA8K_REG_ATU_FUNC, ®);
+ if (ret < 0)
+ return ret;
if (reg & QCA8K_ATU_FUNC_FULL)
return -1;
}
qca8k_fdb_write(priv, fdb->vid, fdb->port_mask, fdb->mac, fdb->aging);
ret = qca8k_fdb_access(priv, QCA8K_FDB_NEXT, port);
- if (ret >= 0)
- qca8k_fdb_read(priv, fdb);
+ if (ret < 0)
+ return ret;
- return ret;
+ return qca8k_fdb_read(priv, fdb);
}
static int
qca8k_vlan_access(struct qca8k_priv *priv, enum qca8k_vlan_cmd cmd, u16 vid)
{
u32 reg;
+ int ret;
/* Set the command and VLAN index */
reg = QCA8K_VTU_FUNC1_BUSY;
reg |= vid << QCA8K_VTU_FUNC1_VID_S;
/* Write the function register triggering the table access */
- qca8k_write(priv, QCA8K_REG_VTU_FUNC1, reg);
+ ret = qca8k_write(priv, QCA8K_REG_VTU_FUNC1, reg);
+ if (ret)
+ return ret;
/* wait for completion */
- if (qca8k_busy_wait(priv, QCA8K_REG_VTU_FUNC1, QCA8K_VTU_FUNC1_BUSY))
- return -ETIMEDOUT;
+ ret = qca8k_busy_wait(priv, QCA8K_REG_VTU_FUNC1, QCA8K_VTU_FUNC1_BUSY);
+ if (ret)
+ return ret;
/* Check for table full violation when adding an entry */
if (cmd == QCA8K_VLAN_LOAD) {
- reg = qca8k_read(priv, QCA8K_REG_VTU_FUNC1);
+ ret = qca8k_read(priv, QCA8K_REG_VTU_FUNC1, ®);
+ if (ret < 0)
+ return ret;
if (reg & QCA8K_VTU_FUNC1_FULL)
return -ENOMEM;
}
if (ret < 0)
goto out;
- reg = qca8k_read(priv, QCA8K_REG_VTU_FUNC0);
+ ret = qca8k_read(priv, QCA8K_REG_VTU_FUNC0, ®);
+ if (ret < 0)
+ goto out;
reg |= QCA8K_VTU_FUNC0_VALID | QCA8K_VTU_FUNC0_IVL_EN;
reg &= ~(QCA8K_VTU_FUNC0_EG_MODE_MASK << QCA8K_VTU_FUNC0_EG_MODE_S(port));
if (untagged)
reg |= QCA8K_VTU_FUNC0_EG_MODE_TAG <<
QCA8K_VTU_FUNC0_EG_MODE_S(port);
- qca8k_write(priv, QCA8K_REG_VTU_FUNC0, reg);
+ ret = qca8k_write(priv, QCA8K_REG_VTU_FUNC0, reg);
+ if (ret)
+ goto out;
ret = qca8k_vlan_access(priv, QCA8K_VLAN_LOAD, vid);
out:
if (ret < 0)
goto out;
- reg = qca8k_read(priv, QCA8K_REG_VTU_FUNC0);
+ ret = qca8k_read(priv, QCA8K_REG_VTU_FUNC0, ®);
+ if (ret < 0)
+ goto out;
reg &= ~(3 << QCA8K_VTU_FUNC0_EG_MODE_S(port));
reg |= QCA8K_VTU_FUNC0_EG_MODE_NOT <<
QCA8K_VTU_FUNC0_EG_MODE_S(port);
if (del) {
ret = qca8k_vlan_access(priv, QCA8K_VLAN_PURGE, vid);
} else {
- qca8k_write(priv, QCA8K_REG_VTU_FUNC0, reg);
+ ret = qca8k_write(priv, QCA8K_REG_VTU_FUNC0, reg);
+ if (ret)
+ goto out;
ret = qca8k_vlan_access(priv, QCA8K_VLAN_LOAD, vid);
}
return ret;
}
-static void
+static int
qca8k_mib_init(struct qca8k_priv *priv)
{
+ int ret;
+
mutex_lock(&priv->reg_mutex);
- qca8k_reg_set(priv, QCA8K_REG_MIB, QCA8K_MIB_FLUSH | QCA8K_MIB_BUSY);
- qca8k_busy_wait(priv, QCA8K_REG_MIB, QCA8K_MIB_BUSY);
- qca8k_reg_set(priv, QCA8K_REG_MIB, QCA8K_MIB_CPU_KEEP);
- qca8k_write(priv, QCA8K_REG_MODULE_EN, QCA8K_MODULE_EN_MIB);
+ ret = qca8k_reg_set(priv, QCA8K_REG_MIB, QCA8K_MIB_FLUSH | QCA8K_MIB_BUSY);
+ if (ret)
+ goto exit;
+
+ ret = qca8k_busy_wait(priv, QCA8K_REG_MIB, QCA8K_MIB_BUSY);
+ if (ret)
+ goto exit;
+
+ ret = qca8k_reg_set(priv, QCA8K_REG_MIB, QCA8K_MIB_CPU_KEEP);
+ if (ret)
+ goto exit;
+
+ ret = qca8k_write(priv, QCA8K_REG_MODULE_EN, QCA8K_MODULE_EN_MIB);
+
+exit:
mutex_unlock(&priv->reg_mutex);
+ return ret;
}
static void
}
static int
-qca8k_mdio_write(struct qca8k_priv *priv, int port, u32 regnum, u16 data)
+qca8k_mdio_busy_wait(struct mii_bus *bus, u32 reg, u32 mask)
{
- u32 phy, val;
+ u16 r1, r2, page;
+ u32 val;
+ int ret, ret1;
+
+ qca8k_split_addr(reg, &r1, &r2, &page);
+
+ ret = read_poll_timeout(qca8k_mii_read32, ret1, !(val & mask), 0,
+ QCA8K_BUSY_WAIT_TIMEOUT * USEC_PER_MSEC, false,
+ bus, 0x10 | r2, r1, &val);
+
+ /* Check if qca8k_read has failed for a different reason
+ * before returnting -ETIMEDOUT
+ */
+ if (ret < 0 && ret1 < 0)
+ return ret1;
+
+ return ret;
+}
+
+static int
+qca8k_mdio_write(struct mii_bus *salve_bus, int phy, int regnum, u16 data)
+{
+ struct qca8k_priv *priv = salve_bus->priv;
+ struct mii_bus *bus = priv->bus;
+ u16 r1, r2, page;
+ u32 val;
+ int ret;
if (regnum >= QCA8K_MDIO_MASTER_MAX_REG)
return -EINVAL;
- /* callee is responsible for not passing bad ports,
- * but we still would like to make spills impossible.
- */
- phy = qca8k_port_to_phy(port) % PHY_MAX_ADDR;
val = QCA8K_MDIO_MASTER_BUSY | QCA8K_MDIO_MASTER_EN |
QCA8K_MDIO_MASTER_WRITE | QCA8K_MDIO_MASTER_PHY_ADDR(phy) |
QCA8K_MDIO_MASTER_REG_ADDR(regnum) |
QCA8K_MDIO_MASTER_DATA(data);
- qca8k_write(priv, QCA8K_MDIO_MASTER_CTRL, val);
+ qca8k_split_addr(QCA8K_MDIO_MASTER_CTRL, &r1, &r2, &page);
+
+ mutex_lock_nested(&bus->mdio_lock, MDIO_MUTEX_NESTED);
+
+ ret = qca8k_set_page(bus, page);
+ if (ret)
+ goto exit;
+
+ qca8k_mii_write32(bus, 0x10 | r2, r1, val);
- return qca8k_busy_wait(priv, QCA8K_MDIO_MASTER_CTRL,
- QCA8K_MDIO_MASTER_BUSY);
+ ret = qca8k_mdio_busy_wait(bus, QCA8K_MDIO_MASTER_CTRL,
+ QCA8K_MDIO_MASTER_BUSY);
+
+exit:
+ /* even if the busy_wait timeouts try to clear the MASTER_EN */
+ qca8k_mii_write32(bus, 0x10 | r2, r1, 0);
+
+ mutex_unlock(&bus->mdio_lock);
+
+ return ret;
}
static int
-qca8k_mdio_read(struct qca8k_priv *priv, int port, u32 regnum)
+qca8k_mdio_read(struct mii_bus *salve_bus, int phy, int regnum)
{
- u32 phy, val;
+ struct qca8k_priv *priv = salve_bus->priv;
+ struct mii_bus *bus = priv->bus;
+ u16 r1, r2, page;
+ u32 val;
+ int ret;
if (regnum >= QCA8K_MDIO_MASTER_MAX_REG)
return -EINVAL;
- /* callee is responsible for not passing bad ports,
- * but we still would like to make spills impossible.
- */
- phy = qca8k_port_to_phy(port) % PHY_MAX_ADDR;
val = QCA8K_MDIO_MASTER_BUSY | QCA8K_MDIO_MASTER_EN |
QCA8K_MDIO_MASTER_READ | QCA8K_MDIO_MASTER_PHY_ADDR(phy) |
QCA8K_MDIO_MASTER_REG_ADDR(regnum);
- qca8k_write(priv, QCA8K_MDIO_MASTER_CTRL, val);
+ qca8k_split_addr(QCA8K_MDIO_MASTER_CTRL, &r1, &r2, &page);
+
+ mutex_lock_nested(&bus->mdio_lock, MDIO_MUTEX_NESTED);
+
+ ret = qca8k_set_page(bus, page);
+ if (ret)
+ goto exit;
+
+ qca8k_mii_write32(bus, 0x10 | r2, r1, val);
- if (qca8k_busy_wait(priv, QCA8K_MDIO_MASTER_CTRL,
- QCA8K_MDIO_MASTER_BUSY))
- return -ETIMEDOUT;
+ ret = qca8k_mdio_busy_wait(bus, QCA8K_MDIO_MASTER_CTRL,
+ QCA8K_MDIO_MASTER_BUSY);
+ if (ret)
+ goto exit;
+
+ ret = qca8k_mii_read32(bus, 0x10 | r2, r1, &val);
+
+exit:
+ /* even if the busy_wait timeouts try to clear the MASTER_EN */
+ qca8k_mii_write32(bus, 0x10 | r2, r1, 0);
- val = (qca8k_read(priv, QCA8K_MDIO_MASTER_CTRL) &
- QCA8K_MDIO_MASTER_DATA_MASK);
+ mutex_unlock(&bus->mdio_lock);
+
+ if (ret >= 0)
+ ret = val & QCA8K_MDIO_MASTER_DATA_MASK;
- return val;
+ return ret;
}
static int
{
struct qca8k_priv *priv = ds->priv;
- return qca8k_mdio_write(priv, port, regnum, data);
+ /* Check if the legacy mapping should be used and the
+ * port is not correctly mapped to the right PHY in the
+ * devicetree
+ */
+ if (priv->legacy_phy_port_mapping)
+ port = qca8k_port_to_phy(port) % PHY_MAX_ADDR;
+
+ return qca8k_mdio_write(priv->bus, port, regnum, data);
}
static int
struct qca8k_priv *priv = ds->priv;
int ret;
- ret = qca8k_mdio_read(priv, port, regnum);
+ /* Check if the legacy mapping should be used and the
+ * port is not correctly mapped to the right PHY in the
+ * devicetree
+ */
+ if (priv->legacy_phy_port_mapping)
+ port = qca8k_port_to_phy(port) % PHY_MAX_ADDR;
+
+ ret = qca8k_mdio_read(priv->bus, port, regnum);
if (ret < 0)
return 0xffff;
}
static int
+qca8k_mdio_register(struct qca8k_priv *priv, struct device_node *mdio)
+{
+ struct dsa_switch *ds = priv->ds;
+ struct mii_bus *bus;
+
+ bus = devm_mdiobus_alloc(ds->dev);
+
+ if (!bus)
+ return -ENOMEM;
+
+ bus->priv = (void *)priv;
+ bus->name = "qca8k slave mii";
+ bus->read = qca8k_mdio_read;
+ bus->write = qca8k_mdio_write;
+ snprintf(bus->id, MII_BUS_ID_SIZE, "qca8k-%d",
+ ds->index);
+
+ bus->parent = ds->dev;
+ bus->phy_mask = ~ds->phys_mii_mask;
+
+ ds->slave_mii_bus = bus;
+
+ return devm_of_mdiobus_register(priv->dev, bus, mdio);
+}
+
+static int
qca8k_setup_mdio_bus(struct qca8k_priv *priv)
{
u32 internal_mdio_mask = 0, external_mdio_mask = 0, reg;
- struct device_node *ports, *port;
+ struct device_node *ports, *port, *mdio;
+ phy_interface_t mode;
int err;
ports = of_get_child_by_name(priv->dev->of_node, "ports");
if (!ports)
+ ports = of_get_child_by_name(priv->dev->of_node, "ethernet-ports");
+
+ if (!ports)
return -EINVAL;
for_each_available_child_of_node(ports, port) {
if (!dsa_is_user_port(priv->ds, reg))
continue;
- if (of_property_read_bool(port, "phy-handle"))
+ of_get_phy_mode(port, &mode);
+
+ if (of_property_read_bool(port, "phy-handle") &&
+ mode != PHY_INTERFACE_MODE_INTERNAL)
external_mdio_mask |= BIT(reg);
else
internal_mdio_mask |= BIT(reg);
* a dt-overlay and driver reload changed the configuration
*/
- qca8k_reg_clear(priv, QCA8K_MDIO_MASTER_CTRL,
- QCA8K_MDIO_MASTER_EN);
- return 0;
+ return qca8k_reg_clear(priv, QCA8K_MDIO_MASTER_CTRL,
+ QCA8K_MDIO_MASTER_EN);
+ }
+
+ /* Check if the devicetree declare the port:phy mapping */
+ mdio = of_get_child_by_name(priv->dev->of_node, "mdio");
+ if (of_device_is_available(mdio)) {
+ err = qca8k_mdio_register(priv, mdio);
+ if (err)
+ of_node_put(mdio);
+
+ return err;
}
+ /* If a mapping can't be found the legacy mapping is used,
+ * using the qca8k_port_to_phy function
+ */
+ priv->legacy_phy_port_mapping = true;
priv->ops.phy_read = qca8k_phy_read;
priv->ops.phy_write = qca8k_phy_write;
+
+ return 0;
+}
+
+static int
+qca8k_setup_of_rgmii_delay(struct qca8k_priv *priv)
+{
+ struct device_node *port_dn;
+ phy_interface_t mode;
+ struct dsa_port *dp;
+ u32 val;
+
+ /* CPU port is already checked */
+ dp = dsa_to_port(priv->ds, 0);
+
+ port_dn = dp->dn;
+
+ /* Check if port 0 is set to the correct type */
+ of_get_phy_mode(port_dn, &mode);
+ if (mode != PHY_INTERFACE_MODE_RGMII_ID &&
+ mode != PHY_INTERFACE_MODE_RGMII_RXID &&
+ mode != PHY_INTERFACE_MODE_RGMII_TXID) {
+ return 0;
+ }
+
+ switch (mode) {
+ case PHY_INTERFACE_MODE_RGMII_ID:
+ case PHY_INTERFACE_MODE_RGMII_RXID:
+ if (of_property_read_u32(port_dn, "rx-internal-delay-ps", &val))
+ val = 2;
+ else
+ /* Switch regs accept value in ns, convert ps to ns */
+ val = val / 1000;
+
+ if (val > QCA8K_MAX_DELAY) {
+ dev_err(priv->dev, "rgmii rx delay is limited to a max value of 3ns, setting to the max value");
+ val = 3;
+ }
+
+ priv->rgmii_rx_delay = val;
+ /* Stop here if we need to check only for rx delay */
+ if (mode != PHY_INTERFACE_MODE_RGMII_ID)
+ break;
+
+ fallthrough;
+ case PHY_INTERFACE_MODE_RGMII_TXID:
+ if (of_property_read_u32(port_dn, "tx-internal-delay-ps", &val))
+ val = 1;
+ else
+ /* Switch regs accept value in ns, convert ps to ns */
+ val = val / 1000;
+
+ if (val > QCA8K_MAX_DELAY) {
+ dev_err(priv->dev, "rgmii tx delay is limited to a max value of 3ns, setting to the max value");
+ val = 3;
+ }
+
+ priv->rgmii_tx_delay = val;
+ break;
+ default:
+ return 0;
+ }
+
return 0;
}
{
struct qca8k_priv *priv = (struct qca8k_priv *)ds->priv;
int ret, i;
+ u32 mask;
/* Make sure that port 0 is the cpu port */
if (!dsa_is_cpu_port(ds, 0)) {
- pr_err("port 0 is not the CPU port\n");
+ dev_err(priv->dev, "port 0 is not the CPU port");
return -EINVAL;
}
priv->regmap = devm_regmap_init(ds->dev, NULL, priv,
&qca8k_regmap_config);
if (IS_ERR(priv->regmap))
- pr_warn("regmap initialization failed");
+ dev_warn(priv->dev, "regmap initialization failed");
ret = qca8k_setup_mdio_bus(priv);
if (ret)
return ret;
+ ret = qca8k_setup_of_rgmii_delay(priv);
+ if (ret)
+ return ret;
+
/* Enable CPU Port */
- qca8k_reg_set(priv, QCA8K_REG_GLOBAL_FW_CTRL0,
- QCA8K_GLOBAL_FW_CTRL0_CPU_PORT_EN);
+ ret = qca8k_reg_set(priv, QCA8K_REG_GLOBAL_FW_CTRL0,
+ QCA8K_GLOBAL_FW_CTRL0_CPU_PORT_EN);
+ if (ret) {
+ dev_err(priv->dev, "failed enabling CPU port");
+ return ret;
+ }
/* Enable MIB counters */
- qca8k_mib_init(priv);
+ ret = qca8k_mib_init(priv);
+ if (ret)
+ dev_warn(priv->dev, "mib init failed");
/* Enable QCA header mode on the cpu port */
- qca8k_write(priv, QCA8K_REG_PORT_HDR_CTRL(QCA8K_CPU_PORT),
- QCA8K_PORT_HDR_CTRL_ALL << QCA8K_PORT_HDR_CTRL_TX_S |
- QCA8K_PORT_HDR_CTRL_ALL << QCA8K_PORT_HDR_CTRL_RX_S);
+ ret = qca8k_write(priv, QCA8K_REG_PORT_HDR_CTRL(QCA8K_CPU_PORT),
+ QCA8K_PORT_HDR_CTRL_ALL << QCA8K_PORT_HDR_CTRL_TX_S |
+ QCA8K_PORT_HDR_CTRL_ALL << QCA8K_PORT_HDR_CTRL_RX_S);
+ if (ret) {
+ dev_err(priv->dev, "failed enabling QCA header mode");
+ return ret;
+ }
/* Disable forwarding by default on all ports */
- for (i = 0; i < QCA8K_NUM_PORTS; i++)
- qca8k_rmw(priv, QCA8K_PORT_LOOKUP_CTRL(i),
- QCA8K_PORT_LOOKUP_MEMBER, 0);
+ for (i = 0; i < QCA8K_NUM_PORTS; i++) {
+ ret = qca8k_rmw(priv, QCA8K_PORT_LOOKUP_CTRL(i),
+ QCA8K_PORT_LOOKUP_MEMBER, 0);
+ if (ret)
+ return ret;
+ }
/* Disable MAC by default on all ports */
for (i = 1; i < QCA8K_NUM_PORTS; i++)
qca8k_port_set_status(priv, i, 0);
/* Forward all unknown frames to CPU port for Linux processing */
- qca8k_write(priv, QCA8K_REG_GLOBAL_FW_CTRL1,
- BIT(0) << QCA8K_GLOBAL_FW_CTRL1_IGMP_DP_S |
- BIT(0) << QCA8K_GLOBAL_FW_CTRL1_BC_DP_S |
- BIT(0) << QCA8K_GLOBAL_FW_CTRL1_MC_DP_S |
- BIT(0) << QCA8K_GLOBAL_FW_CTRL1_UC_DP_S);
+ ret = qca8k_write(priv, QCA8K_REG_GLOBAL_FW_CTRL1,
+ BIT(0) << QCA8K_GLOBAL_FW_CTRL1_IGMP_DP_S |
+ BIT(0) << QCA8K_GLOBAL_FW_CTRL1_BC_DP_S |
+ BIT(0) << QCA8K_GLOBAL_FW_CTRL1_MC_DP_S |
+ BIT(0) << QCA8K_GLOBAL_FW_CTRL1_UC_DP_S);
+ if (ret)
+ return ret;
/* Setup connection between CPU port & user ports */
for (i = 0; i < QCA8K_NUM_PORTS; i++) {
/* CPU port gets connected to all user ports of the switch */
if (dsa_is_cpu_port(ds, i)) {
- qca8k_rmw(priv, QCA8K_PORT_LOOKUP_CTRL(QCA8K_CPU_PORT),
- QCA8K_PORT_LOOKUP_MEMBER, dsa_user_ports(ds));
+ ret = qca8k_rmw(priv, QCA8K_PORT_LOOKUP_CTRL(QCA8K_CPU_PORT),
+ QCA8K_PORT_LOOKUP_MEMBER, dsa_user_ports(ds));
+ if (ret)
+ return ret;
}
/* Individual user ports get connected to CPU port only */
if (dsa_is_user_port(ds, i)) {
int shift = 16 * (i % 2);
- qca8k_rmw(priv, QCA8K_PORT_LOOKUP_CTRL(i),
- QCA8K_PORT_LOOKUP_MEMBER,
- BIT(QCA8K_CPU_PORT));
+ ret = qca8k_rmw(priv, QCA8K_PORT_LOOKUP_CTRL(i),
+ QCA8K_PORT_LOOKUP_MEMBER,
+ BIT(QCA8K_CPU_PORT));
+ if (ret)
+ return ret;
/* Enable ARP Auto-learning by default */
- qca8k_reg_set(priv, QCA8K_PORT_LOOKUP_CTRL(i),
- QCA8K_PORT_LOOKUP_LEARN);
+ ret = qca8k_reg_set(priv, QCA8K_PORT_LOOKUP_CTRL(i),
+ QCA8K_PORT_LOOKUP_LEARN);
+ if (ret)
+ return ret;
/* For port based vlans to work we need to set the
* default egress vid
*/
- qca8k_rmw(priv, QCA8K_EGRESS_VLAN(i),
- 0xfff << shift,
- QCA8K_PORT_VID_DEF << shift);
- qca8k_write(priv, QCA8K_REG_PORT_VLAN_CTRL0(i),
- QCA8K_PORT_VLAN_CVID(QCA8K_PORT_VID_DEF) |
- QCA8K_PORT_VLAN_SVID(QCA8K_PORT_VID_DEF));
+ ret = qca8k_rmw(priv, QCA8K_EGRESS_VLAN(i),
+ 0xfff << shift,
+ QCA8K_PORT_VID_DEF << shift);
+ if (ret)
+ return ret;
+
+ ret = qca8k_write(priv, QCA8K_REG_PORT_VLAN_CTRL0(i),
+ QCA8K_PORT_VLAN_CVID(QCA8K_PORT_VID_DEF) |
+ QCA8K_PORT_VLAN_SVID(QCA8K_PORT_VID_DEF));
+ if (ret)
+ return ret;
}
}
+ /* The port 5 of the qca8337 have some problem in flood condition. The
+ * original legacy driver had some specific buffer and priority settings
+ * for the different port suggested by the QCA switch team. Add this
+ * missing settings to improve switch stability under load condition.
+ * This problem is limited to qca8337 and other qca8k switch are not affected.
+ */
+ if (priv->switch_id == QCA8K_ID_QCA8337) {
+ for (i = 0; i < QCA8K_NUM_PORTS; i++) {
+ switch (i) {
+ /* The 2 CPU port and port 5 requires some different
+ * priority than any other ports.
+ */
+ case 0:
+ case 5:
+ case 6:
+ mask = QCA8K_PORT_HOL_CTRL0_EG_PRI0(0x3) |
+ QCA8K_PORT_HOL_CTRL0_EG_PRI1(0x4) |
+ QCA8K_PORT_HOL_CTRL0_EG_PRI2(0x4) |
+ QCA8K_PORT_HOL_CTRL0_EG_PRI3(0x4) |
+ QCA8K_PORT_HOL_CTRL0_EG_PRI4(0x6) |
+ QCA8K_PORT_HOL_CTRL0_EG_PRI5(0x8) |
+ QCA8K_PORT_HOL_CTRL0_EG_PORT(0x1e);
+ break;
+ default:
+ mask = QCA8K_PORT_HOL_CTRL0_EG_PRI0(0x3) |
+ QCA8K_PORT_HOL_CTRL0_EG_PRI1(0x4) |
+ QCA8K_PORT_HOL_CTRL0_EG_PRI2(0x6) |
+ QCA8K_PORT_HOL_CTRL0_EG_PRI3(0x8) |
+ QCA8K_PORT_HOL_CTRL0_EG_PORT(0x19);
+ }
+ qca8k_write(priv, QCA8K_REG_PORT_HOL_CTRL0(i), mask);
+
+ mask = QCA8K_PORT_HOL_CTRL1_ING(0x6) |
+ QCA8K_PORT_HOL_CTRL1_EG_PRI_BUF_EN |
+ QCA8K_PORT_HOL_CTRL1_EG_PORT_BUF_EN |
+ QCA8K_PORT_HOL_CTRL1_WRED_EN;
+ qca8k_rmw(priv, QCA8K_REG_PORT_HOL_CTRL1(i),
+ QCA8K_PORT_HOL_CTRL1_ING_BUF |
+ QCA8K_PORT_HOL_CTRL1_EG_PRI_BUF_EN |
+ QCA8K_PORT_HOL_CTRL1_EG_PORT_BUF_EN |
+ QCA8K_PORT_HOL_CTRL1_WRED_EN,
+ mask);
+ }
+ }
+
+ /* Special GLOBAL_FC_THRESH value are needed for ar8327 switch */
+ if (priv->switch_id == QCA8K_ID_QCA8327) {
+ mask = QCA8K_GLOBAL_FC_GOL_XON_THRES(288) |
+ QCA8K_GLOBAL_FC_GOL_XOFF_THRES(496);
+ qca8k_rmw(priv, QCA8K_REG_GLOBAL_FC_THRESH,
+ QCA8K_GLOBAL_FC_GOL_XON_THRES_S |
+ QCA8K_GLOBAL_FC_GOL_XOFF_THRES_S,
+ mask);
+ }
+
/* Setup our port MTUs to match power on defaults */
for (i = 0; i < QCA8K_NUM_PORTS; i++)
priv->port_mtu[i] = ETH_FRAME_LEN + ETH_FCS_LEN;
- qca8k_write(priv, QCA8K_MAX_FRAME_SIZE, ETH_FRAME_LEN + ETH_FCS_LEN);
+ ret = qca8k_write(priv, QCA8K_MAX_FRAME_SIZE, ETH_FRAME_LEN + ETH_FCS_LEN);
+ if (ret)
+ dev_warn(priv->dev, "failed setting MTU settings");
/* Flush the FDB table */
qca8k_fdb_flush(priv);
{
struct qca8k_priv *priv = ds->priv;
u32 reg, val;
+ int ret;
switch (port) {
case 0: /* 1st CPU port */
if (state->interface != PHY_INTERFACE_MODE_RGMII &&
state->interface != PHY_INTERFACE_MODE_RGMII_ID &&
+ state->interface != PHY_INTERFACE_MODE_RGMII_TXID &&
+ state->interface != PHY_INTERFACE_MODE_RGMII_RXID &&
state->interface != PHY_INTERFACE_MODE_SGMII)
return;
case 6: /* 2nd CPU port / external PHY */
if (state->interface != PHY_INTERFACE_MODE_RGMII &&
state->interface != PHY_INTERFACE_MODE_RGMII_ID &&
+ state->interface != PHY_INTERFACE_MODE_RGMII_TXID &&
+ state->interface != PHY_INTERFACE_MODE_RGMII_RXID &&
state->interface != PHY_INTERFACE_MODE_SGMII &&
state->interface != PHY_INTERFACE_MODE_1000BASEX)
return;
qca8k_write(priv, reg, QCA8K_PORT_PAD_RGMII_EN);
break;
case PHY_INTERFACE_MODE_RGMII_ID:
+ case PHY_INTERFACE_MODE_RGMII_TXID:
+ case PHY_INTERFACE_MODE_RGMII_RXID:
/* RGMII_ID needs internal delay. This is enabled through
* PORT5_PAD_CTRL for all ports, rather than individual port
* registers
*/
qca8k_write(priv, reg,
QCA8K_PORT_PAD_RGMII_EN |
- QCA8K_PORT_PAD_RGMII_TX_DELAY(QCA8K_MAX_DELAY) |
- QCA8K_PORT_PAD_RGMII_RX_DELAY(QCA8K_MAX_DELAY));
- qca8k_write(priv, QCA8K_REG_PORT5_PAD_CTRL,
+ QCA8K_PORT_PAD_RGMII_TX_DELAY(priv->rgmii_tx_delay) |
+ QCA8K_PORT_PAD_RGMII_RX_DELAY(priv->rgmii_rx_delay) |
+ QCA8K_PORT_PAD_RGMII_TX_DELAY_EN |
QCA8K_PORT_PAD_RGMII_RX_DELAY_EN);
+ /* QCA8337 requires to set rgmii rx delay */
+ if (priv->switch_id == QCA8K_ID_QCA8337)
+ qca8k_write(priv, QCA8K_REG_PORT5_PAD_CTRL,
+ QCA8K_PORT_PAD_RGMII_RX_DELAY_EN);
break;
case PHY_INTERFACE_MODE_SGMII:
case PHY_INTERFACE_MODE_1000BASEX:
qca8k_write(priv, reg, QCA8K_PORT_PAD_SGMII_EN);
/* Enable/disable SerDes auto-negotiation as necessary */
- val = qca8k_read(priv, QCA8K_REG_PWS);
+ ret = qca8k_read(priv, QCA8K_REG_PWS, &val);
+ if (ret)
+ return;
if (phylink_autoneg_inband(mode))
val &= ~QCA8K_PWS_SERDES_AEN_DIS;
else
qca8k_write(priv, QCA8K_REG_PWS, val);
/* Configure the SGMII parameters */
- val = qca8k_read(priv, QCA8K_REG_SGMII_CTRL);
+ ret = qca8k_read(priv, QCA8K_REG_SGMII_CTRL, &val);
+ if (ret)
+ return;
val |= QCA8K_SGMII_EN_PLL | QCA8K_SGMII_EN_RX |
QCA8K_SGMII_EN_TX | QCA8K_SGMII_EN_SD;
if (state->interface != PHY_INTERFACE_MODE_NA &&
state->interface != PHY_INTERFACE_MODE_RGMII &&
state->interface != PHY_INTERFACE_MODE_RGMII_ID &&
+ state->interface != PHY_INTERFACE_MODE_RGMII_TXID &&
+ state->interface != PHY_INTERFACE_MODE_RGMII_RXID &&
state->interface != PHY_INTERFACE_MODE_SGMII)
goto unsupported;
break;
case 5:
/* Internal PHY */
if (state->interface != PHY_INTERFACE_MODE_NA &&
- state->interface != PHY_INTERFACE_MODE_GMII)
+ state->interface != PHY_INTERFACE_MODE_GMII &&
+ state->interface != PHY_INTERFACE_MODE_INTERNAL)
goto unsupported;
break;
case 6: /* 2nd CPU port / external PHY */
if (state->interface != PHY_INTERFACE_MODE_NA &&
state->interface != PHY_INTERFACE_MODE_RGMII &&
state->interface != PHY_INTERFACE_MODE_RGMII_ID &&
+ state->interface != PHY_INTERFACE_MODE_RGMII_TXID &&
+ state->interface != PHY_INTERFACE_MODE_RGMII_RXID &&
state->interface != PHY_INTERFACE_MODE_SGMII &&
state->interface != PHY_INTERFACE_MODE_1000BASEX)
goto unsupported;
{
struct qca8k_priv *priv = ds->priv;
u32 reg;
+ int ret;
- reg = qca8k_read(priv, QCA8K_REG_PORT_STATUS(port));
+ ret = qca8k_read(priv, QCA8K_REG_PORT_STATUS(port), ®);
+ if (ret < 0)
+ return ret;
state->link = !!(reg & QCA8K_PORT_STATUS_LINK_UP);
state->an_complete = state->link;
{
struct qca8k_priv *priv = (struct qca8k_priv *)ds->priv;
const struct qca8k_mib_desc *mib;
- u32 reg, i;
- u64 hi;
+ u32 reg, i, val;
+ u32 hi = 0;
+ int ret;
for (i = 0; i < ARRAY_SIZE(ar8327_mib); i++) {
mib = &ar8327_mib[i];
reg = QCA8K_PORT_MIB_COUNTER(port) + mib->offset;
- data[i] = qca8k_read(priv, reg);
+ ret = qca8k_read(priv, reg, &val);
+ if (ret < 0)
+ continue;
+
if (mib->size == 2) {
- hi = qca8k_read(priv, reg + 4);
- data[i] |= hi << 32;
+ ret = qca8k_read(priv, reg + 4, &hi);
+ if (ret < 0)
+ continue;
}
+
+ data[i] = val;
+ if (mib->size == 2)
+ data[i] |= (u64)hi << 32;
}
}
struct qca8k_priv *priv = (struct qca8k_priv *)ds->priv;
u32 lpi_en = QCA8K_REG_EEE_CTRL_LPI_EN(port);
u32 reg;
+ int ret;
mutex_lock(&priv->reg_mutex);
- reg = qca8k_read(priv, QCA8K_REG_EEE_CTRL);
+ ret = qca8k_read(priv, QCA8K_REG_EEE_CTRL, ®);
+ if (ret < 0)
+ goto exit;
+
if (eee->eee_enabled)
reg |= lpi_en;
else
reg &= ~lpi_en;
- qca8k_write(priv, QCA8K_REG_EEE_CTRL, reg);
- mutex_unlock(&priv->reg_mutex);
+ ret = qca8k_write(priv, QCA8K_REG_EEE_CTRL, reg);
- return 0;
+exit:
+ mutex_unlock(&priv->reg_mutex);
+ return ret;
}
static int
{
struct qca8k_priv *priv = (struct qca8k_priv *)ds->priv;
int port_mask = BIT(QCA8K_CPU_PORT);
- int i;
+ int i, ret;
for (i = 1; i < QCA8K_NUM_PORTS; i++) {
if (dsa_to_port(ds, i)->bridge_dev != br)
/* Add this port to the portvlan mask of the other ports
* in the bridge
*/
- qca8k_reg_set(priv,
- QCA8K_PORT_LOOKUP_CTRL(i),
- BIT(port));
+ ret = qca8k_reg_set(priv,
+ QCA8K_PORT_LOOKUP_CTRL(i),
+ BIT(port));
+ if (ret)
+ return ret;
if (i != port)
port_mask |= BIT(i);
}
+
/* Add all other ports to this ports portvlan mask */
- qca8k_rmw(priv, QCA8K_PORT_LOOKUP_CTRL(port),
- QCA8K_PORT_LOOKUP_MEMBER, port_mask);
+ ret = qca8k_rmw(priv, QCA8K_PORT_LOOKUP_CTRL(port),
+ QCA8K_PORT_LOOKUP_MEMBER, port_mask);
- return 0;
+ return ret;
}
static void
mtu = priv->port_mtu[i];
/* Include L2 header / FCS length */
- qca8k_write(priv, QCA8K_MAX_FRAME_SIZE, mtu + ETH_HLEN + ETH_FCS_LEN);
-
- return 0;
+ return qca8k_write(priv, QCA8K_MAX_FRAME_SIZE, mtu + ETH_HLEN + ETH_FCS_LEN);
}
static int
struct netlink_ext_ack *extack)
{
struct qca8k_priv *priv = ds->priv;
+ int ret;
if (vlan_filtering) {
- qca8k_rmw(priv, QCA8K_PORT_LOOKUP_CTRL(port),
- QCA8K_PORT_LOOKUP_VLAN_MODE,
- QCA8K_PORT_LOOKUP_VLAN_MODE_SECURE);
+ ret = qca8k_rmw(priv, QCA8K_PORT_LOOKUP_CTRL(port),
+ QCA8K_PORT_LOOKUP_VLAN_MODE,
+ QCA8K_PORT_LOOKUP_VLAN_MODE_SECURE);
} else {
- qca8k_rmw(priv, QCA8K_PORT_LOOKUP_CTRL(port),
- QCA8K_PORT_LOOKUP_VLAN_MODE,
- QCA8K_PORT_LOOKUP_VLAN_MODE_NONE);
+ ret = qca8k_rmw(priv, QCA8K_PORT_LOOKUP_CTRL(port),
+ QCA8K_PORT_LOOKUP_VLAN_MODE,
+ QCA8K_PORT_LOOKUP_VLAN_MODE_NONE);
}
- return 0;
+ return ret;
}
static int
bool untagged = vlan->flags & BRIDGE_VLAN_INFO_UNTAGGED;
bool pvid = vlan->flags & BRIDGE_VLAN_INFO_PVID;
struct qca8k_priv *priv = ds->priv;
- int ret = 0;
+ int ret;
ret = qca8k_vlan_add(priv, port, vlan->vid, untagged);
if (ret) {
if (pvid) {
int shift = 16 * (port % 2);
- qca8k_rmw(priv, QCA8K_EGRESS_VLAN(port),
- 0xfff << shift, vlan->vid << shift);
- qca8k_write(priv, QCA8K_REG_PORT_VLAN_CTRL0(port),
- QCA8K_PORT_VLAN_CVID(vlan->vid) |
- QCA8K_PORT_VLAN_SVID(vlan->vid));
+ ret = qca8k_rmw(priv, QCA8K_EGRESS_VLAN(port),
+ 0xfff << shift, vlan->vid << shift);
+ if (ret)
+ return ret;
+
+ ret = qca8k_write(priv, QCA8K_REG_PORT_VLAN_CTRL0(port),
+ QCA8K_PORT_VLAN_CVID(vlan->vid) |
+ QCA8K_PORT_VLAN_SVID(vlan->vid));
}
- return 0;
+ return ret;
}
static int
const struct switchdev_obj_port_vlan *vlan)
{
struct qca8k_priv *priv = ds->priv;
- int ret = 0;
+ int ret;
ret = qca8k_vlan_del(priv, port, vlan->vid);
if (ret)
return ret;
}
+static u32 qca8k_get_phy_flags(struct dsa_switch *ds, int port)
+{
+ struct qca8k_priv *priv = ds->priv;
+
+ /* Communicate to the phy internal driver the switch revision.
+ * Based on the switch revision different values needs to be
+ * set to the dbg and mmd reg on the phy.
+ * The first 2 bit are used to communicate the switch revision
+ * to the phy driver.
+ */
+ if (port > 0 && port < 6)
+ return priv->switch_revision;
+
+ return 0;
+}
+
static enum dsa_tag_protocol
qca8k_get_tag_protocol(struct dsa_switch *ds, int port,
enum dsa_tag_protocol mp)
.phylink_mac_config = qca8k_phylink_mac_config,
.phylink_mac_link_down = qca8k_phylink_mac_link_down,
.phylink_mac_link_up = qca8k_phylink_mac_link_up,
+ .get_phy_flags = qca8k_get_phy_flags,
};
+static int qca8k_read_switch_id(struct qca8k_priv *priv)
+{
+ const struct qca8k_match_data *data;
+ u32 val;
+ u8 id;
+ int ret;
+
+ /* get the switches ID from the compatible */
+ data = of_device_get_match_data(priv->dev);
+ if (!data)
+ return -ENODEV;
+
+ ret = qca8k_read(priv, QCA8K_REG_MASK_CTRL, &val);
+ if (ret < 0)
+ return -ENODEV;
+
+ id = QCA8K_MASK_CTRL_DEVICE_ID(val & QCA8K_MASK_CTRL_DEVICE_ID_MASK);
+ if (id != data->id) {
+ dev_err(priv->dev, "Switch id detected %x but expected %x", id, data->id);
+ return -ENODEV;
+ }
+
+ priv->switch_id = id;
+
+ /* Save revision to communicate to the internal PHY driver */
+ priv->switch_revision = (val & QCA8K_MASK_CTRL_REV_ID_MASK);
+
+ return 0;
+}
+
static int
qca8k_sw_probe(struct mdio_device *mdiodev)
{
struct qca8k_priv *priv;
- u32 id;
+ int ret;
/* allocate the private data struct so that we can probe the switches
* ID register
gpiod_set_value_cansleep(priv->reset_gpio, 0);
}
- /* read the switches ID register */
- id = qca8k_read(priv, QCA8K_REG_MASK_CTRL);
- id >>= QCA8K_MASK_CTRL_ID_S;
- id &= QCA8K_MASK_CTRL_ID_M;
- if (id != QCA8K_ID_QCA8337)
- return -ENODEV;
+ /* Check the detected switch id */
+ ret = qca8k_read_switch_id(priv);
+ if (ret)
+ return ret;
priv->ds = devm_kzalloc(&mdiodev->dev, sizeof(*priv->ds), GFP_KERNEL);
if (!priv->ds)
static SIMPLE_DEV_PM_OPS(qca8k_pm_ops,
qca8k_suspend, qca8k_resume);
+static const struct qca8k_match_data qca832x = {
+ .id = QCA8K_ID_QCA8327,
+};
+
+static const struct qca8k_match_data qca833x = {
+ .id = QCA8K_ID_QCA8337,
+};
+
static const struct of_device_id qca8k_of_match[] = {
- { .compatible = "qca,qca8334" },
- { .compatible = "qca,qca8337" },
+ { .compatible = "qca,qca8327", .data = &qca832x },
+ { .compatible = "qca,qca8334", .data = &qca833x },
+ { .compatible = "qca,qca8337", .data = &qca833x },
{ /* sentinel */ },
};
#define QCA8K_NUM_PORTS 7
#define QCA8K_MAX_MTU 9000
+#define PHY_ID_QCA8327 0x004dd034
+#define QCA8K_ID_QCA8327 0x12
#define PHY_ID_QCA8337 0x004dd036
#define QCA8K_ID_QCA8337 0x13
+#define QCA8K_BUSY_WAIT_TIMEOUT 2000
+
#define QCA8K_NUM_FDB_RECORDS 2048
#define QCA8K_CPU_PORT 0
/* Global control registers */
#define QCA8K_REG_MASK_CTRL 0x000
-#define QCA8K_MASK_CTRL_ID_M 0xff
-#define QCA8K_MASK_CTRL_ID_S 8
+#define QCA8K_MASK_CTRL_REV_ID_MASK GENMASK(7, 0)
+#define QCA8K_MASK_CTRL_REV_ID(x) ((x) >> 0)
+#define QCA8K_MASK_CTRL_DEVICE_ID_MASK GENMASK(15, 8)
+#define QCA8K_MASK_CTRL_DEVICE_ID(x) ((x) >> 8)
#define QCA8K_REG_PORT0_PAD_CTRL 0x004
#define QCA8K_REG_PORT5_PAD_CTRL 0x008
#define QCA8K_REG_PORT6_PAD_CTRL 0x00c
#define QCA8K_PORT_PAD_RGMII_EN BIT(26)
-#define QCA8K_PORT_PAD_RGMII_TX_DELAY(x) \
- ((0x8 + (x & 0x3)) << 22)
-#define QCA8K_PORT_PAD_RGMII_RX_DELAY(x) \
- ((0x10 + (x & 0x3)) << 20)
-#define QCA8K_MAX_DELAY 3
+#define QCA8K_PORT_PAD_RGMII_TX_DELAY(x) ((x) << 22)
+#define QCA8K_PORT_PAD_RGMII_RX_DELAY(x) ((x) << 20)
+#define QCA8K_PORT_PAD_RGMII_TX_DELAY_EN BIT(25)
#define QCA8K_PORT_PAD_RGMII_RX_DELAY_EN BIT(24)
+#define QCA8K_MAX_DELAY 3
#define QCA8K_PORT_PAD_SGMII_EN BIT(7)
#define QCA8K_REG_PWS 0x010
#define QCA8K_PWS_SERDES_AEN_DIS BIT(7)
#define QCA8K_PORT_LOOKUP_STATE GENMASK(18, 16)
#define QCA8K_PORT_LOOKUP_LEARN BIT(20)
+#define QCA8K_REG_GLOBAL_FC_THRESH 0x800
+#define QCA8K_GLOBAL_FC_GOL_XON_THRES(x) ((x) << 16)
+#define QCA8K_GLOBAL_FC_GOL_XON_THRES_S GENMASK(24, 16)
+#define QCA8K_GLOBAL_FC_GOL_XOFF_THRES(x) ((x) << 0)
+#define QCA8K_GLOBAL_FC_GOL_XOFF_THRES_S GENMASK(8, 0)
+
+#define QCA8K_REG_PORT_HOL_CTRL0(_i) (0x970 + (_i) * 0x8)
+#define QCA8K_PORT_HOL_CTRL0_EG_PRI0_BUF GENMASK(3, 0)
+#define QCA8K_PORT_HOL_CTRL0_EG_PRI0(x) ((x) << 0)
+#define QCA8K_PORT_HOL_CTRL0_EG_PRI1_BUF GENMASK(7, 4)
+#define QCA8K_PORT_HOL_CTRL0_EG_PRI1(x) ((x) << 4)
+#define QCA8K_PORT_HOL_CTRL0_EG_PRI2_BUF GENMASK(11, 8)
+#define QCA8K_PORT_HOL_CTRL0_EG_PRI2(x) ((x) << 8)
+#define QCA8K_PORT_HOL_CTRL0_EG_PRI3_BUF GENMASK(15, 12)
+#define QCA8K_PORT_HOL_CTRL0_EG_PRI3(x) ((x) << 12)
+#define QCA8K_PORT_HOL_CTRL0_EG_PRI4_BUF GENMASK(19, 16)
+#define QCA8K_PORT_HOL_CTRL0_EG_PRI4(x) ((x) << 16)
+#define QCA8K_PORT_HOL_CTRL0_EG_PRI5_BUF GENMASK(23, 20)
+#define QCA8K_PORT_HOL_CTRL0_EG_PRI5(x) ((x) << 20)
+#define QCA8K_PORT_HOL_CTRL0_EG_PORT_BUF GENMASK(29, 24)
+#define QCA8K_PORT_HOL_CTRL0_EG_PORT(x) ((x) << 24)
+
+#define QCA8K_REG_PORT_HOL_CTRL1(_i) (0x974 + (_i) * 0x8)
+#define QCA8K_PORT_HOL_CTRL1_ING_BUF GENMASK(3, 0)
+#define QCA8K_PORT_HOL_CTRL1_ING(x) ((x) << 0)
+#define QCA8K_PORT_HOL_CTRL1_EG_PRI_BUF_EN BIT(6)
+#define QCA8K_PORT_HOL_CTRL1_EG_PORT_BUF_EN BIT(7)
+#define QCA8K_PORT_HOL_CTRL1_WRED_EN BIT(8)
+#define QCA8K_PORT_HOL_CTRL1_EG_MIRROR_EN BIT(16)
+
/* Pkt edit registers */
#define QCA8K_EGRESS_VLAN(x) (0x0c70 + (4 * (x / 2)))
int enabled;
};
+struct qca8k_match_data {
+ u8 id;
+};
+
struct qca8k_priv {
+ u8 switch_id;
+ u8 switch_revision;
+ u8 rgmii_tx_delay;
+ u8 rgmii_rx_delay;
+ bool legacy_phy_port_mapping;
struct regmap *regmap;
struct mii_bus *bus;
struct ar8xxx_port_status port_sts[QCA8K_NUM_PORTS];
tristate "NXP SJA1105 Ethernet switch family support"
depends on NET_DSA && SPI
select NET_DSA_TAG_SJA1105
+ select PCS_XPCS
select PACKING
select CRC32
help
- This is the driver for the NXP SJA1105 automotive Ethernet switch
- family. These are 5-port devices and are managed over an SPI
+ This is the driver for the NXP SJA1105 (5-port) and SJA1110 (10-port)
+ automotive Ethernet switch family. These are managed over an SPI
interface. Probing is handled based on OF bindings and so is the
linkage to PHYLINK. The driver supports the following revisions:
- SJA1105E (Gen. 1, No TT-Ethernet)
- SJA1105Q (Gen. 2, No SGMII, TT-Ethernet)
- SJA1105R (Gen. 2, SGMII, No TT-Ethernet)
- SJA1105S (Gen. 2, SGMII, TT-Ethernet)
+ - SJA1110A (Gen. 3, SGMII, TT-Ethernet, 100base-TX PHY, 10 ports)
+ - SJA1110B (Gen. 3, SGMII, TT-Ethernet, 100base-TX PHY, 9 ports)
+ - SJA1110C (Gen. 3, SGMII, TT-Ethernet, 100base-TX PHY, 7 ports)
+ - SJA1110D (Gen. 3, SGMII, TT-Ethernet, no 100base-TX PHY, 7 ports)
config NET_DSA_SJA1105_PTP
bool "Support for the PTP clock on the NXP SJA1105 Ethernet switch"
sja1105-objs := \
sja1105_spi.o \
sja1105_main.o \
+ sja1105_mdio.o \
sja1105_flower.o \
sja1105_ethtool.o \
sja1105_devlink.o \
#include <linux/mutex.h>
#include "sja1105_static_config.h"
-#define SJA1105_NUM_PORTS 5
-#define SJA1105_NUM_TC 8
#define SJA1105ET_FDB_BIN_SIZE 4
/* The hardware value is in multiples of 10 ms.
* The passed parameter is in multiples of 1 ms.
*/
#define SJA1105_AGEING_TIME_MS(ms) ((ms) / 10)
-#define SJA1105_NUM_L2_POLICERS 45
+#define SJA1105_NUM_L2_POLICERS SJA1110_MAX_L2_POLICING_COUNT
typedef enum {
SPI_READ = 0,
#include "sja1105_tas.h"
#include "sja1105_ptp.h"
+enum sja1105_stats_area {
+ MAC,
+ HL1,
+ HL2,
+ ETHER,
+ __MAX_SJA1105_STATS_AREA,
+};
+
/* Keeps the different addresses between E/T and P/Q/R/S */
struct sja1105_regs {
u64 device_id;
u64 rgu;
u64 vl_status;
u64 config;
- u64 sgmii;
u64 rmii_pll1;
u64 ptppinst;
u64 ptppindur;
u64 ptpclkcorp;
u64 ptpsyncts;
u64 ptpschtm;
- u64 ptpegr_ts[SJA1105_NUM_PORTS];
- u64 pad_mii_tx[SJA1105_NUM_PORTS];
- u64 pad_mii_rx[SJA1105_NUM_PORTS];
- u64 pad_mii_id[SJA1105_NUM_PORTS];
- u64 cgu_idiv[SJA1105_NUM_PORTS];
- u64 mii_tx_clk[SJA1105_NUM_PORTS];
- u64 mii_rx_clk[SJA1105_NUM_PORTS];
- u64 mii_ext_tx_clk[SJA1105_NUM_PORTS];
- u64 mii_ext_rx_clk[SJA1105_NUM_PORTS];
- u64 rgmii_tx_clk[SJA1105_NUM_PORTS];
- u64 rmii_ref_clk[SJA1105_NUM_PORTS];
- u64 rmii_ext_tx_clk[SJA1105_NUM_PORTS];
- u64 mac[SJA1105_NUM_PORTS];
- u64 mac_hl1[SJA1105_NUM_PORTS];
- u64 mac_hl2[SJA1105_NUM_PORTS];
- u64 ether_stats[SJA1105_NUM_PORTS];
- u64 qlevel[SJA1105_NUM_PORTS];
+ u64 ptpegr_ts[SJA1105_MAX_NUM_PORTS];
+ u64 pad_mii_tx[SJA1105_MAX_NUM_PORTS];
+ u64 pad_mii_rx[SJA1105_MAX_NUM_PORTS];
+ u64 pad_mii_id[SJA1105_MAX_NUM_PORTS];
+ u64 cgu_idiv[SJA1105_MAX_NUM_PORTS];
+ u64 mii_tx_clk[SJA1105_MAX_NUM_PORTS];
+ u64 mii_rx_clk[SJA1105_MAX_NUM_PORTS];
+ u64 mii_ext_tx_clk[SJA1105_MAX_NUM_PORTS];
+ u64 mii_ext_rx_clk[SJA1105_MAX_NUM_PORTS];
+ u64 rgmii_tx_clk[SJA1105_MAX_NUM_PORTS];
+ u64 rmii_ref_clk[SJA1105_MAX_NUM_PORTS];
+ u64 rmii_ext_tx_clk[SJA1105_MAX_NUM_PORTS];
+ u64 stats[__MAX_SJA1105_STATS_AREA][SJA1105_MAX_NUM_PORTS];
+ u64 mdio_100base_tx;
+ u64 mdio_100base_t1;
+ u64 pcs_base[SJA1105_MAX_NUM_PORTS];
+};
+
+struct sja1105_mdio_private {
+ struct sja1105_private *priv;
+};
+
+enum {
+ SJA1105_SPEED_AUTO,
+ SJA1105_SPEED_10MBPS,
+ SJA1105_SPEED_100MBPS,
+ SJA1105_SPEED_1000MBPS,
+ SJA1105_SPEED_2500MBPS,
+ SJA1105_SPEED_MAX,
+};
+
+enum sja1105_internal_phy_t {
+ SJA1105_NO_PHY = 0,
+ SJA1105_PHY_BASE_TX,
+ SJA1105_PHY_BASE_T1,
};
struct sja1105_info {
*/
int ptpegr_ts_bytes;
int num_cbs_shapers;
+ int max_frame_mem;
+ int num_ports;
+ bool multiple_cascade_ports;
+ enum dsa_tag_protocol tag_proto;
const struct sja1105_dynamic_table_ops *dyn_ops;
const struct sja1105_table_ops *static_ops;
const struct sja1105_regs *regs;
const unsigned char *addr, u16 vid);
void (*ptp_cmd_packing)(u8 *buf, struct sja1105_ptp_cmd *cmd,
enum packing_op op);
+ bool (*rxtstamp)(struct dsa_switch *ds, int port, struct sk_buff *skb);
+ void (*txtstamp)(struct dsa_switch *ds, int port, struct sk_buff *skb);
+ int (*clocking_setup)(struct sja1105_private *priv);
+ int (*pcs_mdio_read)(struct mii_bus *bus, int phy, int reg);
+ int (*pcs_mdio_write)(struct mii_bus *bus, int phy, int reg, u16 val);
+ int (*disable_microcontroller)(struct sja1105_private *priv);
const char *name;
+ bool supports_mii[SJA1105_MAX_NUM_PORTS];
+ bool supports_rmii[SJA1105_MAX_NUM_PORTS];
+ bool supports_rgmii[SJA1105_MAX_NUM_PORTS];
+ bool supports_sgmii[SJA1105_MAX_NUM_PORTS];
+ bool supports_2500basex[SJA1105_MAX_NUM_PORTS];
+ enum sja1105_internal_phy_t internal_phy[SJA1105_MAX_NUM_PORTS];
+ const u64 port_speed[SJA1105_SPEED_MAX];
};
enum sja1105_key_type {
struct sja1105_private {
struct sja1105_static_config static_config;
- bool rgmii_rx_delay[SJA1105_NUM_PORTS];
- bool rgmii_tx_delay[SJA1105_NUM_PORTS];
+ bool rgmii_rx_delay[SJA1105_MAX_NUM_PORTS];
+ bool rgmii_tx_delay[SJA1105_MAX_NUM_PORTS];
+ phy_interface_t phy_mode[SJA1105_MAX_NUM_PORTS];
+ bool fixed_link[SJA1105_MAX_NUM_PORTS];
bool best_effort_vlan_filtering;
unsigned long learn_ena;
unsigned long ucast_egress_floods;
unsigned long bcast_egress_floods;
const struct sja1105_info *info;
+ size_t max_xfer_len;
struct gpio_desc *reset_gpio;
struct spi_device *spidev;
struct dsa_switch *ds;
struct list_head dsa_8021q_vlans;
struct list_head bridge_vlans;
struct sja1105_flow_block flow_block;
- struct sja1105_port ports[SJA1105_NUM_PORTS];
+ struct sja1105_port ports[SJA1105_MAX_NUM_PORTS];
/* Serializes transmission of management frames so that
* the switch doesn't confuse them with one another.
*/
enum sja1105_vlan_state vlan_state;
struct devlink_region **regions;
struct sja1105_cbs_entry *cbs;
+ struct mii_bus *mdio_base_t1;
+ struct mii_bus *mdio_base_tx;
+ struct mii_bus *mdio_pcs;
+ struct dw_xpcs *xpcs[SJA1105_MAX_NUM_PORTS];
struct sja1105_tagger_data tagger_data;
struct sja1105_ptp_data ptp_data;
struct sja1105_tas_data tas_data;
struct netlink_ext_ack *extack);
void sja1105_frame_memory_partitioning(struct sja1105_private *priv);
+/* From sja1105_mdio.c */
+int sja1105_mdiobus_register(struct dsa_switch *ds);
+void sja1105_mdiobus_unregister(struct dsa_switch *ds);
+int sja1105_pcs_mdio_read(struct mii_bus *bus, int phy, int reg);
+int sja1105_pcs_mdio_write(struct mii_bus *bus, int phy, int reg, u16 val);
+int sja1110_pcs_mdio_read(struct mii_bus *bus, int phy, int reg);
+int sja1110_pcs_mdio_write(struct mii_bus *bus, int phy, int reg, u16 val);
+
/* From sja1105_devlink.c */
int sja1105_devlink_setup(struct dsa_switch *ds);
void sja1105_devlink_teardown(struct dsa_switch *ds);
extern const struct sja1105_info sja1105q_info;
extern const struct sja1105_info sja1105r_info;
extern const struct sja1105_info sja1105s_info;
+extern const struct sja1105_info sja1110a_info;
+extern const struct sja1105_info sja1110b_info;
+extern const struct sja1105_info sja1110c_info;
+extern const struct sja1105_info sja1110d_info;
/* From sja1105_clocking.c */
XMII_MODE_SGMII = 3,
} sja1105_phy_interface_t;
-typedef enum {
- SJA1105_SPEED_10MBPS = 3,
- SJA1105_SPEED_100MBPS = 2,
- SJA1105_SPEED_1000MBPS = 1,
- SJA1105_SPEED_AUTO = 0,
-} sja1105_speed_t;
-
int sja1105pqrs_setup_rgmii_delay(const void *ctx, int port);
+int sja1110_setup_rgmii_delay(const void *ctx, int port);
int sja1105_clocking_setup_port(struct sja1105_private *priv, int port);
int sja1105_clocking_setup(struct sja1105_private *priv);
+int sja1110_disable_microcontroller(struct sja1105_private *priv);
/* From sja1105_ethtool.c */
void sja1105_get_ethtool_stats(struct dsa_switch *ds, int port, u64 *data);
SJA1105_S_TAG = 1, /* Outer VLAN header */
};
+enum sja1110_vlan_type {
+ SJA1110_VLAN_INVALID = 0,
+ SJA1110_VLAN_C_TAG = 1, /* Single inner VLAN tag */
+ SJA1110_VLAN_S_TAG = 2, /* Single outer VLAN tag */
+ SJA1110_VLAN_D_TAG = 3, /* Double tagged, use outer tag for lookup */
+};
+
+enum sja1110_shaper_type {
+ SJA1110_LEAKY_BUCKET_SHAPER = 0,
+ SJA1110_CBS_SHAPER = 1,
+};
+
u8 sja1105et_fdb_hash(struct sja1105_private *priv, const u8 *addr, u16 vid);
int sja1105et_fdb_add(struct dsa_switch *ds, int port,
const unsigned char *addr, u16 vid);
#include "sja1105.h"
#define SJA1105_SIZE_CGU_CMD 4
+#define SJA1110_BASE_MCSS_CLK SJA1110_CGU_ADDR(0x70)
+#define SJA1110_BASE_TIMER_CLK SJA1110_CGU_ADDR(0x74)
/* Common structure for CFG_PAD_MIIx_RX and CFG_PAD_MIIx_TX */
struct sja1105_cfg_pad_mii {
u64 pd;
};
+struct sja1110_cgu_outclk {
+ u64 clksrc;
+ u64 autoblock;
+ u64 pd;
+};
+
enum {
CLKSRC_MII0_TX_CLK = 0x00,
CLKSRC_MII0_RX_CLK = 0x01,
struct sja1105_cgu_idiv idiv;
u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0};
+ if (regs->cgu_idiv[port] == SJA1105_RSV_ADDR)
+ return 0;
+
if (enabled && factor != 1 && factor != 10) {
dev_err(dev, "idiv factor must be 1 or 10\n");
return -ERANGE;
u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0};
int clksrc;
+ if (regs->mii_tx_clk[port] == SJA1105_RSV_ADDR)
+ return 0;
+
if (role == XMII_MAC)
clksrc = mac_clk_sources[port];
else
CLKSRC_MII4_RX_CLK,
};
+ if (regs->mii_rx_clk[port] == SJA1105_RSV_ADDR)
+ return 0;
+
/* Payload for packed_buf */
mii_rx_clk.clksrc = clk_sources[port];
mii_rx_clk.autoblock = 1; /* Autoblock clk while changing clksrc */
CLKSRC_IDIV4,
};
+ if (regs->mii_ext_tx_clk[port] == SJA1105_RSV_ADDR)
+ return 0;
+
/* Payload for packed_buf */
mii_ext_tx_clk.clksrc = clk_sources[port];
mii_ext_tx_clk.autoblock = 1; /* Autoblock clk while changing clksrc */
CLKSRC_IDIV4,
};
+ if (regs->mii_ext_rx_clk[port] == SJA1105_RSV_ADDR)
+ return 0;
+
/* Payload for packed_buf */
mii_ext_rx_clk.clksrc = clk_sources[port];
mii_ext_rx_clk.autoblock = 1; /* Autoblock clk while changing clksrc */
}
static int sja1105_cgu_rgmii_tx_clk_config(struct sja1105_private *priv,
- int port, sja1105_speed_t speed)
+ int port, u64 speed)
{
const struct sja1105_regs *regs = priv->info->regs;
struct sja1105_cgu_mii_ctrl txc;
u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0};
int clksrc;
- if (speed == SJA1105_SPEED_1000MBPS) {
+ if (regs->rgmii_tx_clk[port] == SJA1105_RSV_ADDR)
+ return 0;
+
+ if (speed == priv->info->port_speed[SJA1105_SPEED_1000MBPS]) {
clksrc = CLKSRC_PLL0;
} else {
int clk_sources[] = {CLKSRC_IDIV0, CLKSRC_IDIV1, CLKSRC_IDIV2,
struct sja1105_cfg_pad_mii pad_mii_tx = {0};
u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0};
+ if (regs->pad_mii_tx[port] == SJA1105_RSV_ADDR)
+ return 0;
+
/* Payload */
pad_mii_tx.d32_os = 3; /* TXD[3:2] output stage: */
/* high noise/high speed */
struct sja1105_cfg_pad_mii pad_mii_rx = {0};
u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0};
+ if (regs->pad_mii_rx[port] == SJA1105_RSV_ADDR)
+ return 0;
+
/* Payload */
pad_mii_rx.d32_ih = 0; /* RXD[3:2] input stage hysteresis: */
/* non-Schmitt (default) */
sja1105_packing(buf, &cmd->txc_pd, 0, 0, size, op);
}
+static void
+sja1110_cfg_pad_mii_id_packing(void *buf, struct sja1105_cfg_pad_mii_id *cmd,
+ enum packing_op op)
+{
+ const int size = SJA1105_SIZE_CGU_CMD;
+ u64 range = 4;
+
+ /* Fields RXC_RANGE and TXC_RANGE select the input frequency range:
+ * 0 = 2.5MHz
+ * 1 = 25MHz
+ * 2 = 50MHz
+ * 3 = 125MHz
+ * 4 = Automatically determined by port speed.
+ * There's no point in defining a structure different than the one for
+ * SJA1105, so just hardcode the frequency range to automatic, just as
+ * before.
+ */
+ sja1105_packing(buf, &cmd->rxc_stable_ovr, 26, 26, size, op);
+ sja1105_packing(buf, &cmd->rxc_delay, 25, 21, size, op);
+ sja1105_packing(buf, &range, 20, 18, size, op);
+ sja1105_packing(buf, &cmd->rxc_bypass, 17, 17, size, op);
+ sja1105_packing(buf, &cmd->rxc_pd, 16, 16, size, op);
+ sja1105_packing(buf, &cmd->txc_stable_ovr, 10, 10, size, op);
+ sja1105_packing(buf, &cmd->txc_delay, 9, 5, size, op);
+ sja1105_packing(buf, &range, 4, 2, size, op);
+ sja1105_packing(buf, &cmd->txc_bypass, 1, 1, size, op);
+ sja1105_packing(buf, &cmd->txc_pd, 0, 0, size, op);
+}
+
/* Valid range in degrees is an integer between 73.8 and 101.7 */
static u64 sja1105_rgmii_delay(u64 phase)
{
packed_buf, SJA1105_SIZE_CGU_CMD);
}
+int sja1110_setup_rgmii_delay(const void *ctx, int port)
+{
+ const struct sja1105_private *priv = ctx;
+ const struct sja1105_regs *regs = priv->info->regs;
+ struct sja1105_cfg_pad_mii_id pad_mii_id = {0};
+ u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0};
+
+ pad_mii_id.rxc_pd = 1;
+ pad_mii_id.txc_pd = 1;
+
+ if (priv->rgmii_rx_delay[port]) {
+ pad_mii_id.rxc_delay = sja1105_rgmii_delay(90);
+ /* The "BYPASS" bit in SJA1110 is actually a "don't bypass" */
+ pad_mii_id.rxc_bypass = 1;
+ pad_mii_id.rxc_pd = 0;
+ }
+
+ if (priv->rgmii_tx_delay[port]) {
+ pad_mii_id.txc_delay = sja1105_rgmii_delay(90);
+ pad_mii_id.txc_bypass = 1;
+ pad_mii_id.txc_pd = 0;
+ }
+
+ sja1110_cfg_pad_mii_id_packing(packed_buf, &pad_mii_id, PACK);
+
+ return sja1105_xfer_buf(priv, SPI_WRITE, regs->pad_mii_id[port],
+ packed_buf, SJA1105_SIZE_CGU_CMD);
+}
+
static int sja1105_rgmii_clocking_setup(struct sja1105_private *priv, int port,
sja1105_mii_role_t role)
{
struct device *dev = priv->ds->dev;
struct sja1105_mac_config_entry *mac;
- sja1105_speed_t speed;
+ u64 speed;
int rc;
mac = priv->static_config.tables[BLK_IDX_MAC_CONFIG].entries;
speed = mac[port].speed;
- dev_dbg(dev, "Configuring port %d RGMII at speed %dMbps\n",
+ dev_dbg(dev, "Configuring port %d RGMII at speed %lldMbps\n",
port, speed);
- switch (speed) {
- case SJA1105_SPEED_1000MBPS:
+ if (speed == priv->info->port_speed[SJA1105_SPEED_1000MBPS]) {
/* 1000Mbps, IDIV disabled (125 MHz) */
rc = sja1105_cgu_idiv_config(priv, port, false, 1);
- break;
- case SJA1105_SPEED_100MBPS:
+ } else if (speed == priv->info->port_speed[SJA1105_SPEED_100MBPS]) {
/* 100Mbps, IDIV enabled, divide by 1 (25 MHz) */
rc = sja1105_cgu_idiv_config(priv, port, true, 1);
- break;
- case SJA1105_SPEED_10MBPS:
+ } else if (speed == priv->info->port_speed[SJA1105_SPEED_10MBPS]) {
/* 10Mbps, IDIV enabled, divide by 10 (2.5 MHz) */
rc = sja1105_cgu_idiv_config(priv, port, true, 10);
- break;
- case SJA1105_SPEED_AUTO:
+ } else if (speed == priv->info->port_speed[SJA1105_SPEED_AUTO]) {
/* Skip CGU configuration if there is no speed available
* (e.g. link is not established yet)
*/
dev_dbg(dev, "Speed not available, skipping CGU config\n");
return 0;
- default:
+ } else {
rc = -EINVAL;
}
dev_err(dev, "Failed to configure Tx pad registers\n");
return rc;
}
+
if (!priv->info->setup_rgmii_delay)
return 0;
- /* The role has no hardware effect for RGMII. However we use it as
- * a proxy for this interface being a MAC-to-MAC connection, with
- * the RGMII internal delays needing to be applied by us.
- */
- if (role == XMII_MAC)
- return 0;
return priv->info->setup_rgmii_delay(priv, port);
}
CLKSRC_MII4_TX_CLK,
};
+ if (regs->rmii_ref_clk[port] == SJA1105_RSV_ADDR)
+ return 0;
+
/* Payload for packed_buf */
ref_clk.clksrc = clk_sources[port];
ref_clk.autoblock = 1; /* Autoblock clk while changing clksrc */
struct sja1105_cgu_mii_ctrl ext_tx_clk;
u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0};
+ if (regs->rmii_ext_tx_clk[port] == SJA1105_RSV_ADDR)
+ return 0;
+
/* Payload for packed_buf */
ext_tx_clk.clksrc = CLKSRC_PLL1;
ext_tx_clk.autoblock = 1; /* Autoblock clk while changing clksrc */
struct device *dev = priv->ds->dev;
int rc;
+ if (regs->rmii_pll1 == SJA1105_RSV_ADDR)
+ return 0;
+
/* PLL1 must be enabled and output 50 Mhz.
* This is done by writing first 0x0A010941 to
* the PLL_1_C register and then deasserting
int sja1105_clocking_setup(struct sja1105_private *priv)
{
+ struct dsa_switch *ds = priv->ds;
int port, rc;
- for (port = 0; port < SJA1105_NUM_PORTS; port++) {
+ for (port = 0; port < ds->num_ports; port++) {
rc = sja1105_clocking_setup_port(priv, port);
if (rc < 0)
return rc;
}
return 0;
}
+
+static void
+sja1110_cgu_outclk_packing(void *buf, struct sja1110_cgu_outclk *outclk,
+ enum packing_op op)
+{
+ const int size = 4;
+
+ sja1105_packing(buf, &outclk->clksrc, 27, 24, size, op);
+ sja1105_packing(buf, &outclk->autoblock, 11, 11, size, op);
+ sja1105_packing(buf, &outclk->pd, 0, 0, size, op);
+}
+
+int sja1110_disable_microcontroller(struct sja1105_private *priv)
+{
+ u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0};
+ struct sja1110_cgu_outclk outclk_6_c = {
+ .clksrc = 0x3,
+ .pd = true,
+ };
+ struct sja1110_cgu_outclk outclk_7_c = {
+ .clksrc = 0x5,
+ .pd = true,
+ };
+ int rc;
+
+ /* Power down the BASE_TIMER_CLK to disable the watchdog timer */
+ sja1110_cgu_outclk_packing(packed_buf, &outclk_7_c, PACK);
+
+ rc = sja1105_xfer_buf(priv, SPI_WRITE, SJA1110_BASE_TIMER_CLK,
+ packed_buf, SJA1105_SIZE_CGU_CMD);
+ if (rc)
+ return rc;
+
+ /* Power down the BASE_MCSS_CLOCK to gate the microcontroller off */
+ sja1110_cgu_outclk_packing(packed_buf, &outclk_6_c, PACK);
+
+ return sja1105_xfer_buf(priv, SPI_WRITE, SJA1110_BASE_MCSS_CLK,
+ packed_buf, SJA1105_SIZE_CGU_CMD);
+}
* on its ENTRY portion, as a result of a SPI write command.
* Only the TCAM-based FDB table on SJA1105 P/Q/R/S supports
* this.
+ * OP_VALID_ANYWAY: Reading some tables through the dynamic config
+ * interface is possible even if the VALIDENT bit is not
+ * set in the writeback. So don't error out in that case.
* - .max_entry_count: The number of entries, counting from zero, that can be
* reconfigured through the dynamic interface. If a static
* table can be reconfigured at all dynamically, this
#define SJA1105PQRS_SIZE_VL_LOOKUP_DYN_CMD \
(SJA1105_SIZE_DYN_CMD + SJA1105_SIZE_VL_LOOKUP_ENTRY)
+#define SJA1110_SIZE_VL_POLICING_DYN_CMD \
+ (SJA1105_SIZE_DYN_CMD + SJA1105_SIZE_VL_POLICING_ENTRY)
+
#define SJA1105ET_SIZE_MAC_CONFIG_DYN_ENTRY \
SJA1105_SIZE_DYN_CMD
#define SJA1105PQRS_SIZE_L2_LOOKUP_DYN_CMD \
(SJA1105_SIZE_DYN_CMD + SJA1105PQRS_SIZE_L2_LOOKUP_ENTRY)
+#define SJA1110_SIZE_L2_LOOKUP_DYN_CMD \
+ (SJA1105_SIZE_DYN_CMD + SJA1110_SIZE_L2_LOOKUP_ENTRY)
+
#define SJA1105_SIZE_VLAN_LOOKUP_DYN_CMD \
(SJA1105_SIZE_DYN_CMD + 4 + SJA1105_SIZE_VLAN_LOOKUP_ENTRY)
+#define SJA1110_SIZE_VLAN_LOOKUP_DYN_CMD \
+ (SJA1105_SIZE_DYN_CMD + SJA1110_SIZE_VLAN_LOOKUP_ENTRY)
+
#define SJA1105_SIZE_L2_FORWARDING_DYN_CMD \
(SJA1105_SIZE_DYN_CMD + SJA1105_SIZE_L2_FORWARDING_ENTRY)
#define SJA1105PQRS_SIZE_L2_LOOKUP_PARAMS_DYN_CMD \
(SJA1105_SIZE_DYN_CMD + SJA1105PQRS_SIZE_L2_LOOKUP_PARAMS_ENTRY)
+#define SJA1110_SIZE_L2_LOOKUP_PARAMS_DYN_CMD \
+ (SJA1105_SIZE_DYN_CMD + SJA1110_SIZE_L2_LOOKUP_PARAMS_ENTRY)
+
#define SJA1105ET_SIZE_GENERAL_PARAMS_DYN_CMD \
SJA1105_SIZE_DYN_CMD
#define SJA1105PQRS_SIZE_GENERAL_PARAMS_DYN_CMD \
(SJA1105_SIZE_DYN_CMD + SJA1105PQRS_SIZE_GENERAL_PARAMS_ENTRY)
+#define SJA1110_SIZE_GENERAL_PARAMS_DYN_CMD \
+ (SJA1105_SIZE_DYN_CMD + SJA1110_SIZE_GENERAL_PARAMS_ENTRY)
+
#define SJA1105PQRS_SIZE_AVB_PARAMS_DYN_CMD \
(SJA1105_SIZE_DYN_CMD + SJA1105PQRS_SIZE_AVB_PARAMS_ENTRY)
#define SJA1105PQRS_SIZE_CBS_DYN_CMD \
(SJA1105_SIZE_DYN_CMD + SJA1105PQRS_SIZE_CBS_ENTRY)
+#define SJA1110_SIZE_XMII_PARAMS_DYN_CMD \
+ SJA1110_SIZE_XMII_PARAMS_ENTRY
+
+#define SJA1110_SIZE_L2_POLICING_DYN_CMD \
+ (SJA1105_SIZE_DYN_CMD + SJA1105_SIZE_L2_POLICING_ENTRY)
+
+#define SJA1110_SIZE_L2_FORWARDING_PARAMS_DYN_CMD \
+ SJA1105_SIZE_L2_FORWARDING_PARAMS_ENTRY
+
#define SJA1105_MAX_DYN_CMD_SIZE \
- SJA1105PQRS_SIZE_GENERAL_PARAMS_DYN_CMD
+ SJA1110_SIZE_GENERAL_PARAMS_DYN_CMD
struct sja1105_dyn_cmd {
bool search;
SJA1105_HOSTCMD_INVALIDATE = 4,
};
+/* Command and entry overlap */
static void
-sja1105_vl_lookup_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
- enum packing_op op)
+sja1105et_vl_lookup_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
+ enum packing_op op)
{
const int size = SJA1105_SIZE_DYN_CMD;
sja1105_packing(buf, &cmd->index, 9, 0, size, op);
}
+/* Command and entry are separate */
+static void
+sja1105pqrs_vl_lookup_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
+ enum packing_op op)
+{
+ u8 *p = buf + SJA1105_SIZE_VL_LOOKUP_ENTRY;
+ const int size = SJA1105_SIZE_DYN_CMD;
+
+ sja1105_packing(p, &cmd->valid, 31, 31, size, op);
+ sja1105_packing(p, &cmd->errors, 30, 30, size, op);
+ sja1105_packing(p, &cmd->rdwrset, 29, 29, size, op);
+ sja1105_packing(p, &cmd->index, 9, 0, size, op);
+}
+
+static void
+sja1110_vl_lookup_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
+ enum packing_op op)
+{
+ u8 *p = buf + SJA1105PQRS_SIZE_L2_LOOKUP_ENTRY;
+ const int size = SJA1105_SIZE_DYN_CMD;
+
+ sja1105_packing(p, &cmd->valid, 31, 31, size, op);
+ sja1105_packing(p, &cmd->rdwrset, 30, 30, size, op);
+ sja1105_packing(p, &cmd->errors, 29, 29, size, op);
+ sja1105_packing(p, &cmd->index, 11, 0, size, op);
+}
+
static size_t sja1105et_vl_lookup_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
}
static void
+sja1110_vl_policing_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
+ enum packing_op op)
+{
+ u8 *p = buf + SJA1105_SIZE_VL_LOOKUP_ENTRY;
+ const int size = SJA1105_SIZE_DYN_CMD;
+
+ sja1105_packing(p, &cmd->valid, 31, 31, size, op);
+ sja1105_packing(p, &cmd->rdwrset, 30, 30, size, op);
+ sja1105_packing(p, &cmd->index, 11, 0, size, op);
+}
+
+static void
sja1105pqrs_l2_lookup_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
enum packing_op op)
{
return sja1105pqrs_l2_lookup_entry_packing(buf, entry_ptr, op);
}
+static size_t sja1110_dyn_l2_lookup_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1105_l2_lookup_entry *entry = entry_ptr;
+ u8 *cmd = buf + SJA1110_SIZE_L2_LOOKUP_ENTRY;
+ const int size = SJA1105_SIZE_DYN_CMD;
+
+ sja1105_packing(cmd, &entry->lockeds, 28, 28, size, op);
+
+ return sja1110_l2_lookup_entry_packing(buf, entry_ptr, op);
+}
+
static void
sja1105et_l2_lookup_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
enum packing_op op)
SJA1105_SIZE_VLAN_LOOKUP_ENTRY, op);
}
+/* In SJA1110 there is no gap between the command and the data, yay... */
+static void
+sja1110_vlan_lookup_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
+ enum packing_op op)
+{
+ u8 *p = buf + SJA1110_SIZE_VLAN_LOOKUP_ENTRY;
+ const int size = SJA1105_SIZE_DYN_CMD;
+ u64 type_entry = 0;
+
+ sja1105_packing(p, &cmd->valid, 31, 31, size, op);
+ sja1105_packing(p, &cmd->rdwrset, 30, 30, size, op);
+ sja1105_packing(p, &cmd->errors, 29, 29, size, op);
+ /* Hack: treat 'vlanid' field of struct sja1105_vlan_lookup_entry as
+ * cmd->index.
+ */
+ sja1105_packing(buf, &cmd->index, 38, 27,
+ SJA1110_SIZE_VLAN_LOOKUP_ENTRY, op);
+
+ /* But the VALIDENT bit has disappeared, now we are supposed to
+ * invalidate an entry through the TYPE_ENTRY field of the entry..
+ * This is a hack to transform the non-zero quality of the TYPE_ENTRY
+ * field into a VALIDENT bit.
+ */
+ if (op == PACK && !cmd->valident) {
+ sja1105_packing(buf, &type_entry, 40, 39,
+ SJA1110_SIZE_VLAN_LOOKUP_ENTRY, PACK);
+ } else if (op == UNPACK) {
+ sja1105_packing(buf, &type_entry, 40, 39,
+ SJA1110_SIZE_VLAN_LOOKUP_ENTRY, UNPACK);
+ cmd->valident = !!type_entry;
+ }
+}
+
static void
sja1105_l2_forwarding_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
enum packing_op op)
}
static void
+sja1110_l2_forwarding_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
+ enum packing_op op)
+{
+ u8 *p = buf + SJA1105_SIZE_L2_FORWARDING_ENTRY;
+ const int size = SJA1105_SIZE_DYN_CMD;
+
+ sja1105_packing(p, &cmd->valid, 31, 31, size, op);
+ sja1105_packing(p, &cmd->rdwrset, 30, 30, size, op);
+ sja1105_packing(p, &cmd->errors, 29, 29, size, op);
+ sja1105_packing(p, &cmd->index, 4, 0, size, op);
+}
+
+static void
sja1105et_mac_config_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
enum packing_op op)
{
}
static void
+sja1110_mac_config_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
+ enum packing_op op)
+{
+ u8 *p = buf + SJA1105PQRS_SIZE_MAC_CONFIG_ENTRY;
+ const int size = SJA1105_SIZE_DYN_CMD;
+
+ sja1105_packing(p, &cmd->valid, 31, 31, size, op);
+ sja1105_packing(p, &cmd->rdwrset, 30, 30, size, op);
+ sja1105_packing(p, &cmd->errors, 29, 29, size, op);
+ sja1105_packing(p, &cmd->index, 3, 0, size, op);
+}
+
+static void
sja1105et_l2_lookup_params_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
enum packing_op op)
{
}
static void
+sja1110_l2_lookup_params_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
+ enum packing_op op)
+{
+ u8 *p = buf + SJA1110_SIZE_L2_LOOKUP_PARAMS_ENTRY;
+ const int size = SJA1105_SIZE_DYN_CMD;
+
+ sja1105_packing(p, &cmd->valid, 31, 31, size, op);
+ sja1105_packing(p, &cmd->rdwrset, 30, 30, size, op);
+ sja1105_packing(p, &cmd->errors, 29, 29, size, op);
+}
+
+static void
sja1105et_general_params_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
enum packing_op op)
{
}
static void
+sja1110_general_params_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
+ enum packing_op op)
+{
+ u8 *p = buf + SJA1110_SIZE_GENERAL_PARAMS_ENTRY;
+ const int size = SJA1105_SIZE_DYN_CMD;
+
+ sja1105_packing(p, &cmd->valid, 31, 31, size, op);
+ sja1105_packing(p, &cmd->rdwrset, 30, 30, size, op);
+ sja1105_packing(p, &cmd->errors, 29, 29, size, op);
+}
+
+static void
sja1105pqrs_avb_params_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
enum packing_op op)
{
sja1105_packing(p, &cmd->index, 5, 0, size, op);
}
+static void
+sja1110_retagging_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
+ enum packing_op op)
+{
+ u8 *p = buf + SJA1105_SIZE_RETAGGING_ENTRY;
+ const int size = SJA1105_SIZE_DYN_CMD;
+
+ sja1105_packing(p, &cmd->valid, 31, 31, size, op);
+ sja1105_packing(p, &cmd->rdwrset, 30, 30, size, op);
+ sja1105_packing(p, &cmd->errors, 29, 29, size, op);
+ sja1105_packing(p, &cmd->valident, 28, 28, size, op);
+ sja1105_packing(p, &cmd->index, 4, 0, size, op);
+}
+
static void sja1105et_cbs_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
enum packing_op op)
{
sja1105_packing(p, &cmd->index, 3, 0, size, op);
}
+static void sja1110_cbs_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
+ enum packing_op op)
+{
+ u8 *p = buf + SJA1105PQRS_SIZE_CBS_ENTRY;
+ const int size = SJA1105_SIZE_DYN_CMD;
+
+ sja1105_packing(p, &cmd->valid, 31, 31, size, op);
+ sja1105_packing(p, &cmd->rdwrset, 30, 30, size, op);
+ sja1105_packing(p, &cmd->errors, 29, 29, size, op);
+ sja1105_packing(p, &cmd->index, 7, 0, size, op);
+}
+
static size_t sja1105pqrs_cbs_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
return size;
}
+static size_t sja1110_cbs_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ const size_t size = SJA1105PQRS_SIZE_CBS_ENTRY;
+ struct sja1105_cbs_entry *entry = entry_ptr;
+ u64 entry_type = SJA1110_CBS_SHAPER;
+
+ sja1105_packing(buf, &entry_type, 159, 159, size, op);
+ sja1105_packing(buf, &entry->credit_lo, 151, 120, size, op);
+ sja1105_packing(buf, &entry->credit_hi, 119, 88, size, op);
+ sja1105_packing(buf, &entry->send_slope, 87, 56, size, op);
+ sja1105_packing(buf, &entry->idle_slope, 55, 24, size, op);
+ return size;
+}
+
+static void sja1110_dummy_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
+ enum packing_op op)
+{
+}
+
+static void
+sja1110_l2_policing_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd,
+ enum packing_op op)
+{
+ u8 *p = buf + SJA1105_SIZE_L2_POLICING_ENTRY;
+ const int size = SJA1105_SIZE_DYN_CMD;
+
+ sja1105_packing(p, &cmd->valid, 31, 31, size, op);
+ sja1105_packing(p, &cmd->rdwrset, 30, 30, size, op);
+ sja1105_packing(p, &cmd->errors, 29, 29, size, op);
+ sja1105_packing(p, &cmd->index, 6, 0, size, op);
+}
+
#define OP_READ BIT(0)
#define OP_WRITE BIT(1)
#define OP_DEL BIT(2)
#define OP_SEARCH BIT(3)
+#define OP_VALID_ANYWAY BIT(4)
/* SJA1105E/T: First generation */
const struct sja1105_dynamic_table_ops sja1105et_dyn_ops[BLK_IDX_MAX_DYN] = {
[BLK_IDX_VL_LOOKUP] = {
.entry_packing = sja1105et_vl_lookup_entry_packing,
- .cmd_packing = sja1105_vl_lookup_cmd_packing,
+ .cmd_packing = sja1105et_vl_lookup_cmd_packing,
.access = OP_WRITE,
.max_entry_count = SJA1105_MAX_VL_LOOKUP_COUNT,
.packed_size = SJA1105ET_SIZE_VL_LOOKUP_DYN_CMD,
[BLK_IDX_MGMT_ROUTE] = {
.entry_packing = sja1105et_mgmt_route_entry_packing,
.cmd_packing = sja1105et_mgmt_route_cmd_packing,
- .access = (OP_READ | OP_WRITE),
+ .access = (OP_READ | OP_WRITE | OP_VALID_ANYWAY),
.max_entry_count = SJA1105_NUM_PORTS,
.packed_size = SJA1105ET_SIZE_L2_LOOKUP_DYN_CMD,
.addr = 0x20,
const struct sja1105_dynamic_table_ops sja1105pqrs_dyn_ops[BLK_IDX_MAX_DYN] = {
[BLK_IDX_VL_LOOKUP] = {
.entry_packing = sja1105_vl_lookup_entry_packing,
- .cmd_packing = sja1105_vl_lookup_cmd_packing,
+ .cmd_packing = sja1105pqrs_vl_lookup_cmd_packing,
.access = (OP_READ | OP_WRITE),
.max_entry_count = SJA1105_MAX_VL_LOOKUP_COUNT,
.packed_size = SJA1105PQRS_SIZE_VL_LOOKUP_DYN_CMD,
[BLK_IDX_MGMT_ROUTE] = {
.entry_packing = sja1105pqrs_mgmt_route_entry_packing,
.cmd_packing = sja1105pqrs_mgmt_route_cmd_packing,
- .access = (OP_READ | OP_WRITE | OP_DEL | OP_SEARCH),
+ .access = (OP_READ | OP_WRITE | OP_DEL | OP_SEARCH | OP_VALID_ANYWAY),
.max_entry_count = SJA1105_NUM_PORTS,
.packed_size = SJA1105PQRS_SIZE_L2_LOOKUP_DYN_CMD,
.addr = 0x24,
},
};
+/* SJA1110: Third generation */
+const struct sja1105_dynamic_table_ops sja1110_dyn_ops[BLK_IDX_MAX_DYN] = {
+ [BLK_IDX_VL_LOOKUP] = {
+ .entry_packing = sja1110_vl_lookup_entry_packing,
+ .cmd_packing = sja1110_vl_lookup_cmd_packing,
+ .access = (OP_READ | OP_WRITE | OP_VALID_ANYWAY),
+ .max_entry_count = SJA1110_MAX_VL_LOOKUP_COUNT,
+ .packed_size = SJA1105PQRS_SIZE_VL_LOOKUP_DYN_CMD,
+ .addr = SJA1110_SPI_ADDR(0x124),
+ },
+ [BLK_IDX_VL_POLICING] = {
+ .entry_packing = sja1110_vl_policing_entry_packing,
+ .cmd_packing = sja1110_vl_policing_cmd_packing,
+ .access = (OP_READ | OP_WRITE | OP_VALID_ANYWAY),
+ .max_entry_count = SJA1110_MAX_VL_POLICING_COUNT,
+ .packed_size = SJA1110_SIZE_VL_POLICING_DYN_CMD,
+ .addr = SJA1110_SPI_ADDR(0x310),
+ },
+ [BLK_IDX_L2_LOOKUP] = {
+ .entry_packing = sja1110_dyn_l2_lookup_entry_packing,
+ .cmd_packing = sja1105pqrs_l2_lookup_cmd_packing,
+ .access = (OP_READ | OP_WRITE | OP_DEL | OP_SEARCH),
+ .max_entry_count = SJA1105_MAX_L2_LOOKUP_COUNT,
+ .packed_size = SJA1110_SIZE_L2_LOOKUP_DYN_CMD,
+ .addr = SJA1110_SPI_ADDR(0x8c),
+ },
+ [BLK_IDX_VLAN_LOOKUP] = {
+ .entry_packing = sja1110_vlan_lookup_entry_packing,
+ .cmd_packing = sja1110_vlan_lookup_cmd_packing,
+ .access = (OP_READ | OP_WRITE | OP_DEL),
+ .max_entry_count = SJA1105_MAX_VLAN_LOOKUP_COUNT,
+ .packed_size = SJA1110_SIZE_VLAN_LOOKUP_DYN_CMD,
+ .addr = SJA1110_SPI_ADDR(0xb4),
+ },
+ [BLK_IDX_L2_FORWARDING] = {
+ .entry_packing = sja1110_l2_forwarding_entry_packing,
+ .cmd_packing = sja1110_l2_forwarding_cmd_packing,
+ .max_entry_count = SJA1110_MAX_L2_FORWARDING_COUNT,
+ .access = (OP_READ | OP_WRITE | OP_VALID_ANYWAY),
+ .packed_size = SJA1105_SIZE_L2_FORWARDING_DYN_CMD,
+ .addr = SJA1110_SPI_ADDR(0xa8),
+ },
+ [BLK_IDX_MAC_CONFIG] = {
+ .entry_packing = sja1110_mac_config_entry_packing,
+ .cmd_packing = sja1110_mac_config_cmd_packing,
+ .max_entry_count = SJA1110_MAX_MAC_CONFIG_COUNT,
+ .access = (OP_READ | OP_WRITE | OP_VALID_ANYWAY),
+ .packed_size = SJA1105PQRS_SIZE_MAC_CONFIG_DYN_CMD,
+ .addr = SJA1110_SPI_ADDR(0x134),
+ },
+ [BLK_IDX_L2_LOOKUP_PARAMS] = {
+ .entry_packing = sja1110_l2_lookup_params_entry_packing,
+ .cmd_packing = sja1110_l2_lookup_params_cmd_packing,
+ .max_entry_count = SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT,
+ .access = (OP_READ | OP_WRITE | OP_VALID_ANYWAY),
+ .packed_size = SJA1110_SIZE_L2_LOOKUP_PARAMS_DYN_CMD,
+ .addr = SJA1110_SPI_ADDR(0x158),
+ },
+ [BLK_IDX_AVB_PARAMS] = {
+ .entry_packing = sja1105pqrs_avb_params_entry_packing,
+ .cmd_packing = sja1105pqrs_avb_params_cmd_packing,
+ .max_entry_count = SJA1105_MAX_AVB_PARAMS_COUNT,
+ .access = (OP_READ | OP_WRITE | OP_VALID_ANYWAY),
+ .packed_size = SJA1105PQRS_SIZE_AVB_PARAMS_DYN_CMD,
+ .addr = SJA1110_SPI_ADDR(0x2000C),
+ },
+ [BLK_IDX_GENERAL_PARAMS] = {
+ .entry_packing = sja1110_general_params_entry_packing,
+ .cmd_packing = sja1110_general_params_cmd_packing,
+ .max_entry_count = SJA1105_MAX_GENERAL_PARAMS_COUNT,
+ .access = (OP_READ | OP_WRITE | OP_VALID_ANYWAY),
+ .packed_size = SJA1110_SIZE_GENERAL_PARAMS_DYN_CMD,
+ .addr = SJA1110_SPI_ADDR(0xe8),
+ },
+ [BLK_IDX_RETAGGING] = {
+ .entry_packing = sja1110_retagging_entry_packing,
+ .cmd_packing = sja1110_retagging_cmd_packing,
+ .max_entry_count = SJA1105_MAX_RETAGGING_COUNT,
+ .access = (OP_READ | OP_WRITE | OP_DEL),
+ .packed_size = SJA1105_SIZE_RETAGGING_DYN_CMD,
+ .addr = SJA1110_SPI_ADDR(0xdc),
+ },
+ [BLK_IDX_CBS] = {
+ .entry_packing = sja1110_cbs_entry_packing,
+ .cmd_packing = sja1110_cbs_cmd_packing,
+ .max_entry_count = SJA1110_MAX_CBS_COUNT,
+ .access = (OP_READ | OP_WRITE | OP_VALID_ANYWAY),
+ .packed_size = SJA1105PQRS_SIZE_CBS_DYN_CMD,
+ .addr = SJA1110_SPI_ADDR(0xc4),
+ },
+ [BLK_IDX_XMII_PARAMS] = {
+ .entry_packing = sja1110_xmii_params_entry_packing,
+ .cmd_packing = sja1110_dummy_cmd_packing,
+ .max_entry_count = SJA1105_MAX_XMII_PARAMS_COUNT,
+ .access = (OP_READ | OP_VALID_ANYWAY),
+ .packed_size = SJA1110_SIZE_XMII_PARAMS_DYN_CMD,
+ .addr = SJA1110_SPI_ADDR(0x3c),
+ },
+ [BLK_IDX_L2_POLICING] = {
+ .entry_packing = sja1110_l2_policing_entry_packing,
+ .cmd_packing = sja1110_l2_policing_cmd_packing,
+ .max_entry_count = SJA1110_MAX_L2_POLICING_COUNT,
+ .access = (OP_READ | OP_WRITE | OP_VALID_ANYWAY),
+ .packed_size = SJA1110_SIZE_L2_POLICING_DYN_CMD,
+ .addr = SJA1110_SPI_ADDR(0x2fc),
+ },
+ [BLK_IDX_L2_FORWARDING_PARAMS] = {
+ .entry_packing = sja1110_l2_forwarding_params_entry_packing,
+ .cmd_packing = sja1110_dummy_cmd_packing,
+ .max_entry_count = SJA1105_MAX_L2_FORWARDING_PARAMS_COUNT,
+ .access = (OP_READ | OP_VALID_ANYWAY),
+ .packed_size = SJA1110_SIZE_L2_FORWARDING_PARAMS_DYN_CMD,
+ .addr = SJA1110_SPI_ADDR(0x20000),
+ },
+};
+
/* Provides read access to the settings through the dynamic interface
* of the switch.
* @blk_idx is used as key to select from the sja1105_dynamic_table_ops.
cmd = (struct sja1105_dyn_cmd) {0};
ops->cmd_packing(packed_buf, &cmd, UNPACK);
- /* UM10944: [valident] will always be found cleared
- * during a read access with MGMTROUTE set.
- * So don't error out in that case.
- */
- if (!cmd.valident && blk_idx != BLK_IDX_MGMT_ROUTE)
+
+ if (!cmd.valident && !(ops->access & OP_VALID_ANYWAY))
return -ENOENT;
cpu_relax();
} while (cmd.valid && --retries);
extern const struct sja1105_dynamic_table_ops sja1105et_dyn_ops[BLK_IDX_MAX_DYN];
extern const struct sja1105_dynamic_table_ops sja1105pqrs_dyn_ops[BLK_IDX_MAX_DYN];
+extern const struct sja1105_dynamic_table_ops sja1110_dyn_ops[BLK_IDX_MAX_DYN];
#endif
*/
#include "sja1105.h"
-#define SJA1105_SIZE_MAC_AREA (0x02 * 4)
-#define SJA1105_SIZE_HL1_AREA (0x10 * 4)
-#define SJA1105_SIZE_HL2_AREA (0x4 * 4)
-#define SJA1105_SIZE_QLEVEL_AREA (0x8 * 4) /* 0x4 to 0xB */
-#define SJA1105_SIZE_ETHER_AREA (0x17 * 4)
-
-struct sja1105_port_status_mac {
- u64 n_runt;
- u64 n_soferr;
- u64 n_alignerr;
- u64 n_miierr;
- u64 typeerr;
- u64 sizeerr;
- u64 tctimeout;
- u64 priorerr;
- u64 nomaster;
- u64 memov;
- u64 memerr;
- u64 invtyp;
- u64 intcyov;
- u64 domerr;
- u64 pcfbagdrop;
- u64 spcprior;
- u64 ageprior;
- u64 portdrop;
- u64 lendrop;
- u64 bagdrop;
- u64 policeerr;
- u64 drpnona664err;
- u64 spcerr;
- u64 agedrp;
-};
-
-struct sja1105_port_status_hl1 {
- u64 n_n664err;
- u64 n_vlanerr;
- u64 n_unreleased;
- u64 n_sizeerr;
- u64 n_crcerr;
- u64 n_vlnotfound;
- u64 n_ctpolerr;
- u64 n_polerr;
- u64 n_rxfrmsh;
- u64 n_rxfrm;
- u64 n_rxbytesh;
- u64 n_rxbyte;
- u64 n_txfrmsh;
- u64 n_txfrm;
- u64 n_txbytesh;
- u64 n_txbyte;
+enum sja1105_counter_index {
+ __SJA1105_COUNTER_UNUSED,
+ /* MAC */
+ N_RUNT,
+ N_SOFERR,
+ N_ALIGNERR,
+ N_MIIERR,
+ TYPEERR,
+ SIZEERR,
+ TCTIMEOUT,
+ PRIORERR,
+ NOMASTER,
+ MEMOV,
+ MEMERR,
+ INVTYP,
+ INTCYOV,
+ DOMERR,
+ PCFBAGDROP,
+ SPCPRIOR,
+ AGEPRIOR,
+ PORTDROP,
+ LENDROP,
+ BAGDROP,
+ POLICEERR,
+ DRPNONA664ERR,
+ SPCERR,
+ AGEDRP,
+ /* HL1 */
+ N_N664ERR,
+ N_VLANERR,
+ N_UNRELEASED,
+ N_SIZEERR,
+ N_CRCERR,
+ N_VLNOTFOUND,
+ N_CTPOLERR,
+ N_POLERR,
+ N_RXFRM,
+ N_RXBYTE,
+ N_TXFRM,
+ N_TXBYTE,
+ /* HL2 */
+ N_QFULL,
+ N_PART_DROP,
+ N_EGR_DISABLED,
+ N_NOT_REACH,
+ __MAX_SJA1105ET_PORT_COUNTER,
+ /* P/Q/R/S only */
+ /* ETHER */
+ N_DROPS_NOLEARN = __MAX_SJA1105ET_PORT_COUNTER,
+ N_DROPS_NOROUTE,
+ N_DROPS_ILL_DTAG,
+ N_DROPS_DTAG,
+ N_DROPS_SOTAG,
+ N_DROPS_SITAG,
+ N_DROPS_UTAG,
+ N_TX_BYTES_1024_2047,
+ N_TX_BYTES_512_1023,
+ N_TX_BYTES_256_511,
+ N_TX_BYTES_128_255,
+ N_TX_BYTES_65_127,
+ N_TX_BYTES_64,
+ N_TX_MCAST,
+ N_TX_BCAST,
+ N_RX_BYTES_1024_2047,
+ N_RX_BYTES_512_1023,
+ N_RX_BYTES_256_511,
+ N_RX_BYTES_128_255,
+ N_RX_BYTES_65_127,
+ N_RX_BYTES_64,
+ N_RX_MCAST,
+ N_RX_BCAST,
+ __MAX_SJA1105PQRS_PORT_COUNTER,
};
-struct sja1105_port_status_hl2 {
- u64 n_qfull;
- u64 n_part_drop;
- u64 n_egr_disabled;
- u64 n_not_reach;
- u64 qlevel_hwm[8]; /* Only for P/Q/R/S */
- u64 qlevel[8]; /* Only for P/Q/R/S */
+struct sja1105_port_counter {
+ enum sja1105_stats_area area;
+ const char name[ETH_GSTRING_LEN];
+ int offset;
+ int start;
+ int end;
+ bool is_64bit;
};
-struct sja1105_port_status_ether {
- u64 n_drops_nolearn;
- u64 n_drops_noroute;
- u64 n_drops_ill_dtag;
- u64 n_drops_dtag;
- u64 n_drops_sotag;
- u64 n_drops_sitag;
- u64 n_drops_utag;
- u64 n_tx_bytes_1024_2047;
- u64 n_tx_bytes_512_1023;
- u64 n_tx_bytes_256_511;
- u64 n_tx_bytes_128_255;
- u64 n_tx_bytes_65_127;
- u64 n_tx_bytes_64;
- u64 n_tx_mcast;
- u64 n_tx_bcast;
- u64 n_rx_bytes_1024_2047;
- u64 n_rx_bytes_512_1023;
- u64 n_rx_bytes_256_511;
- u64 n_rx_bytes_128_255;
- u64 n_rx_bytes_65_127;
- u64 n_rx_bytes_64;
- u64 n_rx_mcast;
- u64 n_rx_bcast;
-};
-
-struct sja1105_port_status {
- struct sja1105_port_status_mac mac;
- struct sja1105_port_status_hl1 hl1;
- struct sja1105_port_status_hl2 hl2;
- struct sja1105_port_status_ether ether;
+static const struct sja1105_port_counter sja1105_port_counters[] = {
+ /* MAC-Level Diagnostic Counters */
+ [N_RUNT] = {
+ .area = MAC,
+ .name = "n_runt",
+ .offset = 0,
+ .start = 31,
+ .end = 24,
+ },
+ [N_SOFERR] = {
+ .area = MAC,
+ .name = "n_soferr",
+ .offset = 0x0,
+ .start = 23,
+ .end = 16,
+ },
+ [N_ALIGNERR] = {
+ .area = MAC,
+ .name = "n_alignerr",
+ .offset = 0x0,
+ .start = 15,
+ .end = 8,
+ },
+ [N_MIIERR] = {
+ .area = MAC,
+ .name = "n_miierr",
+ .offset = 0x0,
+ .start = 7,
+ .end = 0,
+ },
+ /* MAC-Level Diagnostic Flags */
+ [TYPEERR] = {
+ .area = MAC,
+ .name = "typeerr",
+ .offset = 0x1,
+ .start = 27,
+ .end = 27,
+ },
+ [SIZEERR] = {
+ .area = MAC,
+ .name = "sizeerr",
+ .offset = 0x1,
+ .start = 26,
+ .end = 26,
+ },
+ [TCTIMEOUT] = {
+ .area = MAC,
+ .name = "tctimeout",
+ .offset = 0x1,
+ .start = 25,
+ .end = 25,
+ },
+ [PRIORERR] = {
+ .area = MAC,
+ .name = "priorerr",
+ .offset = 0x1,
+ .start = 24,
+ .end = 24,
+ },
+ [NOMASTER] = {
+ .area = MAC,
+ .name = "nomaster",
+ .offset = 0x1,
+ .start = 23,
+ .end = 23,
+ },
+ [MEMOV] = {
+ .area = MAC,
+ .name = "memov",
+ .offset = 0x1,
+ .start = 22,
+ .end = 22,
+ },
+ [MEMERR] = {
+ .area = MAC,
+ .name = "memerr",
+ .offset = 0x1,
+ .start = 21,
+ .end = 21,
+ },
+ [INVTYP] = {
+ .area = MAC,
+ .name = "invtyp",
+ .offset = 0x1,
+ .start = 19,
+ .end = 19,
+ },
+ [INTCYOV] = {
+ .area = MAC,
+ .name = "intcyov",
+ .offset = 0x1,
+ .start = 18,
+ .end = 18,
+ },
+ [DOMERR] = {
+ .area = MAC,
+ .name = "domerr",
+ .offset = 0x1,
+ .start = 17,
+ .end = 17,
+ },
+ [PCFBAGDROP] = {
+ .area = MAC,
+ .name = "pcfbagdrop",
+ .offset = 0x1,
+ .start = 16,
+ .end = 16,
+ },
+ [SPCPRIOR] = {
+ .area = MAC,
+ .name = "spcprior",
+ .offset = 0x1,
+ .start = 15,
+ .end = 12,
+ },
+ [AGEPRIOR] = {
+ .area = MAC,
+ .name = "ageprior",
+ .offset = 0x1,
+ .start = 11,
+ .end = 8,
+ },
+ [PORTDROP] = {
+ .area = MAC,
+ .name = "portdrop",
+ .offset = 0x1,
+ .start = 6,
+ .end = 6,
+ },
+ [LENDROP] = {
+ .area = MAC,
+ .name = "lendrop",
+ .offset = 0x1,
+ .start = 5,
+ .end = 5,
+ },
+ [BAGDROP] = {
+ .area = MAC,
+ .name = "bagdrop",
+ .offset = 0x1,
+ .start = 4,
+ .end = 4,
+ },
+ [POLICEERR] = {
+ .area = MAC,
+ .name = "policeerr",
+ .offset = 0x1,
+ .start = 3,
+ .end = 3,
+ },
+ [DRPNONA664ERR] = {
+ .area = MAC,
+ .name = "drpnona664err",
+ .offset = 0x1,
+ .start = 2,
+ .end = 2,
+ },
+ [SPCERR] = {
+ .area = MAC,
+ .name = "spcerr",
+ .offset = 0x1,
+ .start = 1,
+ .end = 1,
+ },
+ [AGEDRP] = {
+ .area = MAC,
+ .name = "agedrp",
+ .offset = 0x1,
+ .start = 0,
+ .end = 0,
+ },
+ /* High-Level Diagnostic Counters */
+ [N_N664ERR] = {
+ .area = HL1,
+ .name = "n_n664err",
+ .offset = 0xF,
+ .start = 31,
+ .end = 0,
+ },
+ [N_VLANERR] = {
+ .area = HL1,
+ .name = "n_vlanerr",
+ .offset = 0xE,
+ .start = 31,
+ .end = 0,
+ },
+ [N_UNRELEASED] = {
+ .area = HL1,
+ .name = "n_unreleased",
+ .offset = 0xD,
+ .start = 31,
+ .end = 0,
+ },
+ [N_SIZEERR] = {
+ .area = HL1,
+ .name = "n_sizeerr",
+ .offset = 0xC,
+ .start = 31,
+ .end = 0,
+ },
+ [N_CRCERR] = {
+ .area = HL1,
+ .name = "n_crcerr",
+ .offset = 0xB,
+ .start = 31,
+ .end = 0,
+ },
+ [N_VLNOTFOUND] = {
+ .area = HL1,
+ .name = "n_vlnotfound",
+ .offset = 0xA,
+ .start = 31,
+ .end = 0,
+ },
+ [N_CTPOLERR] = {
+ .area = HL1,
+ .name = "n_ctpolerr",
+ .offset = 0x9,
+ .start = 31,
+ .end = 0,
+ },
+ [N_POLERR] = {
+ .area = HL1,
+ .name = "n_polerr",
+ .offset = 0x8,
+ .start = 31,
+ .end = 0,
+ },
+ [N_RXFRM] = {
+ .area = HL1,
+ .name = "n_rxfrm",
+ .offset = 0x6,
+ .start = 31,
+ .end = 0,
+ .is_64bit = true,
+ },
+ [N_RXBYTE] = {
+ .area = HL1,
+ .name = "n_rxbyte",
+ .offset = 0x4,
+ .start = 31,
+ .end = 0,
+ .is_64bit = true,
+ },
+ [N_TXFRM] = {
+ .area = HL1,
+ .name = "n_txfrm",
+ .offset = 0x2,
+ .start = 31,
+ .end = 0,
+ .is_64bit = true,
+ },
+ [N_TXBYTE] = {
+ .area = HL1,
+ .name = "n_txbyte",
+ .offset = 0x0,
+ .start = 31,
+ .end = 0,
+ .is_64bit = true,
+ },
+ [N_QFULL] = {
+ .area = HL2,
+ .name = "n_qfull",
+ .offset = 0x3,
+ .start = 31,
+ .end = 0,
+ },
+ [N_PART_DROP] = {
+ .area = HL2,
+ .name = "n_part_drop",
+ .offset = 0x2,
+ .start = 31,
+ .end = 0,
+ },
+ [N_EGR_DISABLED] = {
+ .area = HL2,
+ .name = "n_egr_disabled",
+ .offset = 0x1,
+ .start = 31,
+ .end = 0,
+ },
+ [N_NOT_REACH] = {
+ .area = HL2,
+ .name = "n_not_reach",
+ .offset = 0x0,
+ .start = 31,
+ .end = 0,
+ },
+ /* Ether Stats */
+ [N_DROPS_NOLEARN] = {
+ .area = ETHER,
+ .name = "n_drops_nolearn",
+ .offset = 0x16,
+ .start = 31,
+ .end = 0,
+ },
+ [N_DROPS_NOROUTE] = {
+ .area = ETHER,
+ .name = "n_drops_noroute",
+ .offset = 0x15,
+ .start = 31,
+ .end = 0,
+ },
+ [N_DROPS_ILL_DTAG] = {
+ .area = ETHER,
+ .name = "n_drops_ill_dtag",
+ .offset = 0x14,
+ .start = 31,
+ .end = 0,
+ },
+ [N_DROPS_DTAG] = {
+ .area = ETHER,
+ .name = "n_drops_dtag",
+ .offset = 0x13,
+ .start = 31,
+ .end = 0,
+ },
+ [N_DROPS_SOTAG] = {
+ .area = ETHER,
+ .name = "n_drops_sotag",
+ .offset = 0x12,
+ .start = 31,
+ .end = 0,
+ },
+ [N_DROPS_SITAG] = {
+ .area = ETHER,
+ .name = "n_drops_sitag",
+ .offset = 0x11,
+ .start = 31,
+ .end = 0,
+ },
+ [N_DROPS_UTAG] = {
+ .area = ETHER,
+ .name = "n_drops_utag",
+ .offset = 0x10,
+ .start = 31,
+ .end = 0,
+ },
+ [N_TX_BYTES_1024_2047] = {
+ .area = ETHER,
+ .name = "n_tx_bytes_1024_2047",
+ .offset = 0x0F,
+ .start = 31,
+ .end = 0,
+ },
+ [N_TX_BYTES_512_1023] = {
+ .area = ETHER,
+ .name = "n_tx_bytes_512_1023",
+ .offset = 0x0E,
+ .start = 31,
+ .end = 0,
+ },
+ [N_TX_BYTES_256_511] = {
+ .area = ETHER,
+ .name = "n_tx_bytes_256_511",
+ .offset = 0x0D,
+ .start = 31,
+ .end = 0,
+ },
+ [N_TX_BYTES_128_255] = {
+ .area = ETHER,
+ .name = "n_tx_bytes_128_255",
+ .offset = 0x0C,
+ .start = 31,
+ .end = 0,
+ },
+ [N_TX_BYTES_65_127] = {
+ .area = ETHER,
+ .name = "n_tx_bytes_65_127",
+ .offset = 0x0B,
+ .start = 31,
+ .end = 0,
+ },
+ [N_TX_BYTES_64] = {
+ .area = ETHER,
+ .name = "n_tx_bytes_64",
+ .offset = 0x0A,
+ .start = 31,
+ .end = 0,
+ },
+ [N_TX_MCAST] = {
+ .area = ETHER,
+ .name = "n_tx_mcast",
+ .offset = 0x09,
+ .start = 31,
+ .end = 0,
+ },
+ [N_TX_BCAST] = {
+ .area = ETHER,
+ .name = "n_tx_bcast",
+ .offset = 0x08,
+ .start = 31,
+ .end = 0,
+ },
+ [N_RX_BYTES_1024_2047] = {
+ .area = ETHER,
+ .name = "n_rx_bytes_1024_2047",
+ .offset = 0x07,
+ .start = 31,
+ .end = 0,
+ },
+ [N_RX_BYTES_512_1023] = {
+ .area = ETHER,
+ .name = "n_rx_bytes_512_1023",
+ .offset = 0x06,
+ .start = 31,
+ .end = 0,
+ },
+ [N_RX_BYTES_256_511] = {
+ .area = ETHER,
+ .name = "n_rx_bytes_256_511",
+ .offset = 0x05,
+ .start = 31,
+ .end = 0,
+ },
+ [N_RX_BYTES_128_255] = {
+ .area = ETHER,
+ .name = "n_rx_bytes_128_255",
+ .offset = 0x04,
+ .start = 31,
+ .end = 0,
+ },
+ [N_RX_BYTES_65_127] = {
+ .area = ETHER,
+ .name = "n_rx_bytes_65_127",
+ .offset = 0x03,
+ .start = 31,
+ .end = 0,
+ },
+ [N_RX_BYTES_64] = {
+ .area = ETHER,
+ .name = "n_rx_bytes_64",
+ .offset = 0x02,
+ .start = 31,
+ .end = 0,
+ },
+ [N_RX_MCAST] = {
+ .area = ETHER,
+ .name = "n_rx_mcast",
+ .offset = 0x01,
+ .start = 31,
+ .end = 0,
+ },
+ [N_RX_BCAST] = {
+ .area = ETHER,
+ .name = "n_rx_bcast",
+ .offset = 0x00,
+ .start = 31,
+ .end = 0,
+ },
};
-static void
-sja1105_port_status_mac_unpack(void *buf,
- struct sja1105_port_status_mac *status)
-{
- /* Make pointer arithmetic work on 4 bytes */
- u32 *p = buf;
-
- sja1105_unpack(p + 0x0, &status->n_runt, 31, 24, 4);
- sja1105_unpack(p + 0x0, &status->n_soferr, 23, 16, 4);
- sja1105_unpack(p + 0x0, &status->n_alignerr, 15, 8, 4);
- sja1105_unpack(p + 0x0, &status->n_miierr, 7, 0, 4);
- sja1105_unpack(p + 0x1, &status->typeerr, 27, 27, 4);
- sja1105_unpack(p + 0x1, &status->sizeerr, 26, 26, 4);
- sja1105_unpack(p + 0x1, &status->tctimeout, 25, 25, 4);
- sja1105_unpack(p + 0x1, &status->priorerr, 24, 24, 4);
- sja1105_unpack(p + 0x1, &status->nomaster, 23, 23, 4);
- sja1105_unpack(p + 0x1, &status->memov, 22, 22, 4);
- sja1105_unpack(p + 0x1, &status->memerr, 21, 21, 4);
- sja1105_unpack(p + 0x1, &status->invtyp, 19, 19, 4);
- sja1105_unpack(p + 0x1, &status->intcyov, 18, 18, 4);
- sja1105_unpack(p + 0x1, &status->domerr, 17, 17, 4);
- sja1105_unpack(p + 0x1, &status->pcfbagdrop, 16, 16, 4);
- sja1105_unpack(p + 0x1, &status->spcprior, 15, 12, 4);
- sja1105_unpack(p + 0x1, &status->ageprior, 11, 8, 4);
- sja1105_unpack(p + 0x1, &status->portdrop, 6, 6, 4);
- sja1105_unpack(p + 0x1, &status->lendrop, 5, 5, 4);
- sja1105_unpack(p + 0x1, &status->bagdrop, 4, 4, 4);
- sja1105_unpack(p + 0x1, &status->policeerr, 3, 3, 4);
- sja1105_unpack(p + 0x1, &status->drpnona664err, 2, 2, 4);
- sja1105_unpack(p + 0x1, &status->spcerr, 1, 1, 4);
- sja1105_unpack(p + 0x1, &status->agedrp, 0, 0, 4);
-}
-
-static void
-sja1105_port_status_hl1_unpack(void *buf,
- struct sja1105_port_status_hl1 *status)
-{
- /* Make pointer arithmetic work on 4 bytes */
- u32 *p = buf;
-
- sja1105_unpack(p + 0xF, &status->n_n664err, 31, 0, 4);
- sja1105_unpack(p + 0xE, &status->n_vlanerr, 31, 0, 4);
- sja1105_unpack(p + 0xD, &status->n_unreleased, 31, 0, 4);
- sja1105_unpack(p + 0xC, &status->n_sizeerr, 31, 0, 4);
- sja1105_unpack(p + 0xB, &status->n_crcerr, 31, 0, 4);
- sja1105_unpack(p + 0xA, &status->n_vlnotfound, 31, 0, 4);
- sja1105_unpack(p + 0x9, &status->n_ctpolerr, 31, 0, 4);
- sja1105_unpack(p + 0x8, &status->n_polerr, 31, 0, 4);
- sja1105_unpack(p + 0x7, &status->n_rxfrmsh, 31, 0, 4);
- sja1105_unpack(p + 0x6, &status->n_rxfrm, 31, 0, 4);
- sja1105_unpack(p + 0x5, &status->n_rxbytesh, 31, 0, 4);
- sja1105_unpack(p + 0x4, &status->n_rxbyte, 31, 0, 4);
- sja1105_unpack(p + 0x3, &status->n_txfrmsh, 31, 0, 4);
- sja1105_unpack(p + 0x2, &status->n_txfrm, 31, 0, 4);
- sja1105_unpack(p + 0x1, &status->n_txbytesh, 31, 0, 4);
- sja1105_unpack(p + 0x0, &status->n_txbyte, 31, 0, 4);
- status->n_rxfrm += status->n_rxfrmsh << 32;
- status->n_rxbyte += status->n_rxbytesh << 32;
- status->n_txfrm += status->n_txfrmsh << 32;
- status->n_txbyte += status->n_txbytesh << 32;
-}
-
-static void
-sja1105_port_status_hl2_unpack(void *buf,
- struct sja1105_port_status_hl2 *status)
-{
- /* Make pointer arithmetic work on 4 bytes */
- u32 *p = buf;
-
- sja1105_unpack(p + 0x3, &status->n_qfull, 31, 0, 4);
- sja1105_unpack(p + 0x2, &status->n_part_drop, 31, 0, 4);
- sja1105_unpack(p + 0x1, &status->n_egr_disabled, 31, 0, 4);
- sja1105_unpack(p + 0x0, &status->n_not_reach, 31, 0, 4);
-}
-
-static void
-sja1105pqrs_port_status_qlevel_unpack(void *buf,
- struct sja1105_port_status_hl2 *status)
-{
- /* Make pointer arithmetic work on 4 bytes */
- u32 *p = buf;
- int i;
-
- for (i = 0; i < 8; i++) {
- sja1105_unpack(p + i, &status->qlevel_hwm[i], 24, 16, 4);
- sja1105_unpack(p + i, &status->qlevel[i], 8, 0, 4);
- }
-}
-
-static void
-sja1105pqrs_port_status_ether_unpack(void *buf,
- struct sja1105_port_status_ether *status)
-{
- /* Make pointer arithmetic work on 4 bytes */
- u32 *p = buf;
-
- sja1105_unpack(p + 0x16, &status->n_drops_nolearn, 31, 0, 4);
- sja1105_unpack(p + 0x15, &status->n_drops_noroute, 31, 0, 4);
- sja1105_unpack(p + 0x14, &status->n_drops_ill_dtag, 31, 0, 4);
- sja1105_unpack(p + 0x13, &status->n_drops_dtag, 31, 0, 4);
- sja1105_unpack(p + 0x12, &status->n_drops_sotag, 31, 0, 4);
- sja1105_unpack(p + 0x11, &status->n_drops_sitag, 31, 0, 4);
- sja1105_unpack(p + 0x10, &status->n_drops_utag, 31, 0, 4);
- sja1105_unpack(p + 0x0F, &status->n_tx_bytes_1024_2047, 31, 0, 4);
- sja1105_unpack(p + 0x0E, &status->n_tx_bytes_512_1023, 31, 0, 4);
- sja1105_unpack(p + 0x0D, &status->n_tx_bytes_256_511, 31, 0, 4);
- sja1105_unpack(p + 0x0C, &status->n_tx_bytes_128_255, 31, 0, 4);
- sja1105_unpack(p + 0x0B, &status->n_tx_bytes_65_127, 31, 0, 4);
- sja1105_unpack(p + 0x0A, &status->n_tx_bytes_64, 31, 0, 4);
- sja1105_unpack(p + 0x09, &status->n_tx_mcast, 31, 0, 4);
- sja1105_unpack(p + 0x08, &status->n_tx_bcast, 31, 0, 4);
- sja1105_unpack(p + 0x07, &status->n_rx_bytes_1024_2047, 31, 0, 4);
- sja1105_unpack(p + 0x06, &status->n_rx_bytes_512_1023, 31, 0, 4);
- sja1105_unpack(p + 0x05, &status->n_rx_bytes_256_511, 31, 0, 4);
- sja1105_unpack(p + 0x04, &status->n_rx_bytes_128_255, 31, 0, 4);
- sja1105_unpack(p + 0x03, &status->n_rx_bytes_65_127, 31, 0, 4);
- sja1105_unpack(p + 0x02, &status->n_rx_bytes_64, 31, 0, 4);
- sja1105_unpack(p + 0x01, &status->n_rx_mcast, 31, 0, 4);
- sja1105_unpack(p + 0x00, &status->n_rx_bcast, 31, 0, 4);
-}
-
-static int
-sja1105pqrs_port_status_get_ether(struct sja1105_private *priv,
- struct sja1105_port_status_ether *ether,
- int port)
-{
- const struct sja1105_regs *regs = priv->info->regs;
- u8 packed_buf[SJA1105_SIZE_ETHER_AREA] = {0};
- int rc;
-
- /* Ethernet statistics area */
- rc = sja1105_xfer_buf(priv, SPI_READ, regs->ether_stats[port],
- packed_buf, SJA1105_SIZE_ETHER_AREA);
- if (rc < 0)
- return rc;
-
- sja1105pqrs_port_status_ether_unpack(packed_buf, ether);
-
- return 0;
-}
-
-static int sja1105_port_status_get_mac(struct sja1105_private *priv,
- struct sja1105_port_status_mac *status,
- int port)
-{
- const struct sja1105_regs *regs = priv->info->regs;
- u8 packed_buf[SJA1105_SIZE_MAC_AREA] = {0};
- int rc;
-
- /* MAC area */
- rc = sja1105_xfer_buf(priv, SPI_READ, regs->mac[port], packed_buf,
- SJA1105_SIZE_MAC_AREA);
- if (rc < 0)
- return rc;
-
- sja1105_port_status_mac_unpack(packed_buf, status);
-
- return 0;
-}
-
-static int sja1105_port_status_get_hl1(struct sja1105_private *priv,
- struct sja1105_port_status_hl1 *status,
- int port)
-{
- const struct sja1105_regs *regs = priv->info->regs;
- u8 packed_buf[SJA1105_SIZE_HL1_AREA] = {0};
- int rc;
-
- rc = sja1105_xfer_buf(priv, SPI_READ, regs->mac_hl1[port], packed_buf,
- SJA1105_SIZE_HL1_AREA);
- if (rc < 0)
- return rc;
-
- sja1105_port_status_hl1_unpack(packed_buf, status);
-
- return 0;
-}
-
-static int sja1105_port_status_get_hl2(struct sja1105_private *priv,
- struct sja1105_port_status_hl2 *status,
- int port)
+static int sja1105_port_counter_read(struct sja1105_private *priv, int port,
+ enum sja1105_counter_index idx, u64 *ctr)
{
- const struct sja1105_regs *regs = priv->info->regs;
- u8 packed_buf[SJA1105_SIZE_QLEVEL_AREA] = {0};
+ const struct sja1105_port_counter *c = &sja1105_port_counters[idx];
+ size_t size = c->is_64bit ? 8 : 4;
+ u8 buf[8] = {0};
+ u64 regs;
int rc;
- rc = sja1105_xfer_buf(priv, SPI_READ, regs->mac_hl2[port], packed_buf,
- SJA1105_SIZE_HL2_AREA);
- if (rc < 0)
- return rc;
-
- sja1105_port_status_hl2_unpack(packed_buf, status);
-
- /* Code below is strictly P/Q/R/S specific. */
- if (priv->info->device_id == SJA1105E_DEVICE_ID ||
- priv->info->device_id == SJA1105T_DEVICE_ID)
- return 0;
+ regs = priv->info->regs->stats[c->area][port];
- rc = sja1105_xfer_buf(priv, SPI_READ, regs->qlevel[port], packed_buf,
- SJA1105_SIZE_QLEVEL_AREA);
- if (rc < 0)
+ rc = sja1105_xfer_buf(priv, SPI_READ, regs + c->offset, buf, size);
+ if (rc)
return rc;
- sja1105pqrs_port_status_qlevel_unpack(packed_buf, status);
+ sja1105_unpack(buf, ctr, c->start, c->end, size);
return 0;
}
-static int sja1105_port_status_get(struct sja1105_private *priv,
- struct sja1105_port_status *status,
- int port)
-{
- int rc;
-
- rc = sja1105_port_status_get_mac(priv, &status->mac, port);
- if (rc < 0)
- return rc;
- rc = sja1105_port_status_get_hl1(priv, &status->hl1, port);
- if (rc < 0)
- return rc;
- rc = sja1105_port_status_get_hl2(priv, &status->hl2, port);
- if (rc < 0)
- return rc;
-
- if (priv->info->device_id == SJA1105E_DEVICE_ID ||
- priv->info->device_id == SJA1105T_DEVICE_ID)
- return 0;
-
- return sja1105pqrs_port_status_get_ether(priv, &status->ether, port);
-}
-
-static char sja1105_port_stats[][ETH_GSTRING_LEN] = {
- /* MAC-Level Diagnostic Counters */
- "n_runt",
- "n_soferr",
- "n_alignerr",
- "n_miierr",
- /* MAC-Level Diagnostic Flags */
- "typeerr",
- "sizeerr",
- "tctimeout",
- "priorerr",
- "nomaster",
- "memov",
- "memerr",
- "invtyp",
- "intcyov",
- "domerr",
- "pcfbagdrop",
- "spcprior",
- "ageprior",
- "portdrop",
- "lendrop",
- "bagdrop",
- "policeerr",
- "drpnona664err",
- "spcerr",
- "agedrp",
- /* High-Level Diagnostic Counters */
- "n_n664err",
- "n_vlanerr",
- "n_unreleased",
- "n_sizeerr",
- "n_crcerr",
- "n_vlnotfound",
- "n_ctpolerr",
- "n_polerr",
- "n_rxfrm",
- "n_rxbyte",
- "n_txfrm",
- "n_txbyte",
- "n_qfull",
- "n_part_drop",
- "n_egr_disabled",
- "n_not_reach",
-};
-
-static char sja1105pqrs_extra_port_stats[][ETH_GSTRING_LEN] = {
- /* Queue Levels */
- "qlevel_hwm_0",
- "qlevel_hwm_1",
- "qlevel_hwm_2",
- "qlevel_hwm_3",
- "qlevel_hwm_4",
- "qlevel_hwm_5",
- "qlevel_hwm_6",
- "qlevel_hwm_7",
- "qlevel_0",
- "qlevel_1",
- "qlevel_2",
- "qlevel_3",
- "qlevel_4",
- "qlevel_5",
- "qlevel_6",
- "qlevel_7",
- /* Ether Stats */
- "n_drops_nolearn",
- "n_drops_noroute",
- "n_drops_ill_dtag",
- "n_drops_dtag",
- "n_drops_sotag",
- "n_drops_sitag",
- "n_drops_utag",
- "n_tx_bytes_1024_2047",
- "n_tx_bytes_512_1023",
- "n_tx_bytes_256_511",
- "n_tx_bytes_128_255",
- "n_tx_bytes_65_127",
- "n_tx_bytes_64",
- "n_tx_mcast",
- "n_tx_bcast",
- "n_rx_bytes_1024_2047",
- "n_rx_bytes_512_1023",
- "n_rx_bytes_256_511",
- "n_rx_bytes_128_255",
- "n_rx_bytes_65_127",
- "n_rx_bytes_64",
- "n_rx_mcast",
- "n_rx_bcast",
-};
-
void sja1105_get_ethtool_stats(struct dsa_switch *ds, int port, u64 *data)
{
struct sja1105_private *priv = ds->priv;
- struct sja1105_port_status *status;
- int rc, i, k = 0;
-
- status = kzalloc(sizeof(*status), GFP_KERNEL);
- if (!status)
- goto out;
-
- rc = sja1105_port_status_get(priv, status, port);
- if (rc < 0) {
- dev_err(ds->dev, "Failed to read port %d counters: %d\n",
- port, rc);
- goto out;
- }
- memset(data, 0, ARRAY_SIZE(sja1105_port_stats) * sizeof(u64));
- data[k++] = status->mac.n_runt;
- data[k++] = status->mac.n_soferr;
- data[k++] = status->mac.n_alignerr;
- data[k++] = status->mac.n_miierr;
- data[k++] = status->mac.typeerr;
- data[k++] = status->mac.sizeerr;
- data[k++] = status->mac.tctimeout;
- data[k++] = status->mac.priorerr;
- data[k++] = status->mac.nomaster;
- data[k++] = status->mac.memov;
- data[k++] = status->mac.memerr;
- data[k++] = status->mac.invtyp;
- data[k++] = status->mac.intcyov;
- data[k++] = status->mac.domerr;
- data[k++] = status->mac.pcfbagdrop;
- data[k++] = status->mac.spcprior;
- data[k++] = status->mac.ageprior;
- data[k++] = status->mac.portdrop;
- data[k++] = status->mac.lendrop;
- data[k++] = status->mac.bagdrop;
- data[k++] = status->mac.policeerr;
- data[k++] = status->mac.drpnona664err;
- data[k++] = status->mac.spcerr;
- data[k++] = status->mac.agedrp;
- data[k++] = status->hl1.n_n664err;
- data[k++] = status->hl1.n_vlanerr;
- data[k++] = status->hl1.n_unreleased;
- data[k++] = status->hl1.n_sizeerr;
- data[k++] = status->hl1.n_crcerr;
- data[k++] = status->hl1.n_vlnotfound;
- data[k++] = status->hl1.n_ctpolerr;
- data[k++] = status->hl1.n_polerr;
- data[k++] = status->hl1.n_rxfrm;
- data[k++] = status->hl1.n_rxbyte;
- data[k++] = status->hl1.n_txfrm;
- data[k++] = status->hl1.n_txbyte;
- data[k++] = status->hl2.n_qfull;
- data[k++] = status->hl2.n_part_drop;
- data[k++] = status->hl2.n_egr_disabled;
- data[k++] = status->hl2.n_not_reach;
+ enum sja1105_counter_index max_ctr, i;
+ int rc, k = 0;
if (priv->info->device_id == SJA1105E_DEVICE_ID ||
priv->info->device_id == SJA1105T_DEVICE_ID)
- goto out;
-
- memset(data + k, 0, ARRAY_SIZE(sja1105pqrs_extra_port_stats) *
- sizeof(u64));
- for (i = 0; i < 8; i++) {
- data[k++] = status->hl2.qlevel_hwm[i];
- data[k++] = status->hl2.qlevel[i];
+ max_ctr = __MAX_SJA1105ET_PORT_COUNTER;
+ else
+ max_ctr = __MAX_SJA1105PQRS_PORT_COUNTER;
+
+ for (i = 0; i < max_ctr; i++) {
+ rc = sja1105_port_counter_read(priv, port, i, &data[k++]);
+ if (rc) {
+ dev_err(ds->dev,
+ "Failed to read port %d counters: %d\n",
+ port, rc);
+ break;
+ }
}
- data[k++] = status->ether.n_drops_nolearn;
- data[k++] = status->ether.n_drops_noroute;
- data[k++] = status->ether.n_drops_ill_dtag;
- data[k++] = status->ether.n_drops_dtag;
- data[k++] = status->ether.n_drops_sotag;
- data[k++] = status->ether.n_drops_sitag;
- data[k++] = status->ether.n_drops_utag;
- data[k++] = status->ether.n_tx_bytes_1024_2047;
- data[k++] = status->ether.n_tx_bytes_512_1023;
- data[k++] = status->ether.n_tx_bytes_256_511;
- data[k++] = status->ether.n_tx_bytes_128_255;
- data[k++] = status->ether.n_tx_bytes_65_127;
- data[k++] = status->ether.n_tx_bytes_64;
- data[k++] = status->ether.n_tx_mcast;
- data[k++] = status->ether.n_tx_bcast;
- data[k++] = status->ether.n_rx_bytes_1024_2047;
- data[k++] = status->ether.n_rx_bytes_512_1023;
- data[k++] = status->ether.n_rx_bytes_256_511;
- data[k++] = status->ether.n_rx_bytes_128_255;
- data[k++] = status->ether.n_rx_bytes_65_127;
- data[k++] = status->ether.n_rx_bytes_64;
- data[k++] = status->ether.n_rx_mcast;
- data[k++] = status->ether.n_rx_bcast;
-out:
- kfree(status);
}
void sja1105_get_strings(struct dsa_switch *ds, int port,
u32 stringset, u8 *data)
{
struct sja1105_private *priv = ds->priv;
- u8 *p = data;
- int i;
+ enum sja1105_counter_index max_ctr, i;
+ char *p = data;
- switch (stringset) {
- case ETH_SS_STATS:
- for (i = 0; i < ARRAY_SIZE(sja1105_port_stats); i++) {
- strlcpy(p, sja1105_port_stats[i], ETH_GSTRING_LEN);
- p += ETH_GSTRING_LEN;
- }
- if (priv->info->device_id == SJA1105E_DEVICE_ID ||
- priv->info->device_id == SJA1105T_DEVICE_ID)
- return;
- for (i = 0; i < ARRAY_SIZE(sja1105pqrs_extra_port_stats); i++) {
- strlcpy(p, sja1105pqrs_extra_port_stats[i],
- ETH_GSTRING_LEN);
- p += ETH_GSTRING_LEN;
- }
- break;
+ if (stringset != ETH_SS_STATS)
+ return;
+
+ if (priv->info->device_id == SJA1105E_DEVICE_ID ||
+ priv->info->device_id == SJA1105T_DEVICE_ID)
+ max_ctr = __MAX_SJA1105ET_PORT_COUNTER;
+ else
+ max_ctr = __MAX_SJA1105PQRS_PORT_COUNTER;
+
+ for (i = 0; i < max_ctr; i++) {
+ strscpy(p, sja1105_port_counters[i].name, ETH_GSTRING_LEN);
+ p += ETH_GSTRING_LEN;
}
}
int sja1105_get_sset_count(struct dsa_switch *ds, int port, int sset)
{
- int count = ARRAY_SIZE(sja1105_port_stats);
struct sja1105_private *priv = ds->priv;
+ enum sja1105_counter_index max_ctr, i;
+ int sset_count = 0;
if (sset != ETH_SS_STATS)
return -EOPNOTSUPP;
- if (priv->info->device_id == SJA1105PR_DEVICE_ID ||
- priv->info->device_id == SJA1105QS_DEVICE_ID)
- count += ARRAY_SIZE(sja1105pqrs_extra_port_stats);
+ if (priv->info->device_id == SJA1105E_DEVICE_ID ||
+ priv->info->device_id == SJA1105T_DEVICE_ID)
+ max_ctr = __MAX_SJA1105ET_PORT_COUNTER;
+ else
+ max_ctr = __MAX_SJA1105PQRS_PORT_COUNTER;
+
+ for (i = 0; i < max_ctr; i++) {
+ if (!strlen(sja1105_port_counters[i].name))
+ continue;
+
+ sset_count++;
+ }
- return count;
+ return sset_count;
}
{
struct sja1105_rule *rule = sja1105_rule_find(priv, cookie);
struct sja1105_l2_policing_entry *policing;
+ struct dsa_switch *ds = priv->ds;
bool new_rule = false;
unsigned long p;
int rc;
policing = priv->static_config.tables[BLK_IDX_L2_POLICING].entries;
- if (policing[(SJA1105_NUM_PORTS * SJA1105_NUM_TC) + port].sharindx != port) {
+ if (policing[(ds->num_ports * SJA1105_NUM_TC) + port].sharindx != port) {
NL_SET_ERR_MSG_MOD(extack,
"Port already has a broadcast policer");
rc = -EEXIST;
/* Make the broadcast policers of all ports attached to this block
* point to the newly allocated policer
*/
- for_each_set_bit(p, &rule->port_mask, SJA1105_NUM_PORTS) {
- int bcast = (SJA1105_NUM_PORTS * SJA1105_NUM_TC) + p;
+ for_each_set_bit(p, &rule->port_mask, SJA1105_MAX_NUM_PORTS) {
+ int bcast = (ds->num_ports * SJA1105_NUM_TC) + p;
policing[bcast].sharindx = rule->bcast_pol.sharindx;
}
/* Make the policers for traffic class @tc of all ports attached to
* this block point to the newly allocated policer
*/
- for_each_set_bit(p, &rule->port_mask, SJA1105_NUM_PORTS) {
+ for_each_set_bit(p, &rule->port_mask, SJA1105_MAX_NUM_PORTS) {
int index = (p * SJA1105_NUM_TC) + tc;
policing[index].sharindx = rule->tc_pol.sharindx;
policing = priv->static_config.tables[BLK_IDX_L2_POLICING].entries;
if (rule->type == SJA1105_RULE_BCAST_POLICER) {
- int bcast = (SJA1105_NUM_PORTS * SJA1105_NUM_TC) + port;
+ int bcast = (ds->num_ports * SJA1105_NUM_TC) + port;
old_sharindx = policing[bcast].sharindx;
policing[bcast].sharindx = port;
INIT_LIST_HEAD(&priv->flow_block.rules);
- for (port = 0; port < SJA1105_NUM_PORTS; port++)
+ for (port = 0; port < ds->num_ports; port++)
priv->flow_block.l2_policer_used[port] = true;
}
#include <linux/of_net.h>
#include <linux/of_mdio.h>
#include <linux/of_device.h>
+#include <linux/pcs/pcs-xpcs.h>
#include <linux/netdev_features.h>
#include <linux/netdevice.h>
#include <linux/if_bridge.h>
#include <linux/if_ether.h>
#include <linux/dsa/8021q.h>
#include "sja1105.h"
-#include "sja1105_sgmii.h"
#include "sja1105_tas.h"
#define SJA1105_UNKNOWN_MULTICAST 0x010000000000ull
+#define SJA1105_DEFAULT_VLAN (VLAN_N_VID - 1)
static const struct dsa_switch_ops sja1105_switch_ops;
return !!(l2_fwd[from].reach_port & BIT(to));
}
-/* Structure used to temporarily transport device tree
- * settings into sja1105_setup
- */
-struct sja1105_dt_port {
- phy_interface_t phy_mode;
- sja1105_mii_role_t role;
-};
-
static int sja1105_init_mac_settings(struct sja1105_private *priv)
{
struct sja1105_mac_config_entry default_mac = {
/* Always put the MAC speed in automatic mode, where it can be
* adjusted at runtime by PHYLINK.
*/
- .speed = SJA1105_SPEED_AUTO,
+ .speed = priv->info->port_speed[SJA1105_SPEED_AUTO],
/* No static correction for 1-step 1588 events */
.tp_delin = 0,
.tp_delout = 0,
.ingress = false,
};
struct sja1105_mac_config_entry *mac;
+ struct dsa_switch *ds = priv->ds;
struct sja1105_table *table;
int i;
table->entry_count = 0;
}
- table->entries = kcalloc(SJA1105_NUM_PORTS,
+ table->entries = kcalloc(table->ops->max_entry_count,
table->ops->unpacked_entry_size, GFP_KERNEL);
if (!table->entries)
return -ENOMEM;
- table->entry_count = SJA1105_NUM_PORTS;
+ table->entry_count = table->ops->max_entry_count;
mac = table->entries;
- for (i = 0; i < SJA1105_NUM_PORTS; i++) {
+ for (i = 0; i < ds->num_ports; i++) {
mac[i] = default_mac;
if (i == dsa_upstream_port(priv->ds, i)) {
/* STP doesn't get called for CPU port, so we need to
return 0;
}
-static bool sja1105_supports_sgmii(struct sja1105_private *priv, int port)
-{
- if (priv->info->part_no != SJA1105R_PART_NO &&
- priv->info->part_no != SJA1105S_PART_NO)
- return false;
-
- if (port != SJA1105_SGMII_PORT)
- return false;
-
- if (dsa_is_unused_port(priv->ds, port))
- return false;
-
- return true;
-}
-
-static int sja1105_init_mii_settings(struct sja1105_private *priv,
- struct sja1105_dt_port *ports)
+static int sja1105_init_mii_settings(struct sja1105_private *priv)
{
struct device *dev = &priv->spidev->dev;
struct sja1105_xmii_params_entry *mii;
+ struct dsa_switch *ds = priv->ds;
struct sja1105_table *table;
int i;
table->entry_count = 0;
}
- table->entries = kcalloc(SJA1105_MAX_XMII_PARAMS_COUNT,
+ table->entries = kcalloc(table->ops->max_entry_count,
table->ops->unpacked_entry_size, GFP_KERNEL);
if (!table->entries)
return -ENOMEM;
/* Override table based on PHYLINK DT bindings */
- table->entry_count = SJA1105_MAX_XMII_PARAMS_COUNT;
+ table->entry_count = table->ops->max_entry_count;
mii = table->entries;
- for (i = 0; i < SJA1105_NUM_PORTS; i++) {
+ for (i = 0; i < ds->num_ports; i++) {
+ sja1105_mii_role_t role = XMII_MAC;
+
if (dsa_is_unused_port(priv->ds, i))
continue;
- switch (ports[i].phy_mode) {
+ switch (priv->phy_mode[i]) {
+ case PHY_INTERFACE_MODE_INTERNAL:
+ if (priv->info->internal_phy[i] == SJA1105_NO_PHY)
+ goto unsupported;
+
+ mii->xmii_mode[i] = XMII_MODE_MII;
+ if (priv->info->internal_phy[i] == SJA1105_PHY_BASE_TX)
+ mii->special[i] = true;
+
+ break;
+ case PHY_INTERFACE_MODE_REVMII:
+ role = XMII_PHY;
+ fallthrough;
case PHY_INTERFACE_MODE_MII:
+ if (!priv->info->supports_mii[i])
+ goto unsupported;
+
mii->xmii_mode[i] = XMII_MODE_MII;
break;
+ case PHY_INTERFACE_MODE_REVRMII:
+ role = XMII_PHY;
+ fallthrough;
case PHY_INTERFACE_MODE_RMII:
+ if (!priv->info->supports_rmii[i])
+ goto unsupported;
+
mii->xmii_mode[i] = XMII_MODE_RMII;
break;
case PHY_INTERFACE_MODE_RGMII:
case PHY_INTERFACE_MODE_RGMII_ID:
case PHY_INTERFACE_MODE_RGMII_RXID:
case PHY_INTERFACE_MODE_RGMII_TXID:
+ if (!priv->info->supports_rgmii[i])
+ goto unsupported;
+
mii->xmii_mode[i] = XMII_MODE_RGMII;
break;
case PHY_INTERFACE_MODE_SGMII:
- if (!sja1105_supports_sgmii(priv, i))
- return -EINVAL;
+ if (!priv->info->supports_sgmii[i])
+ goto unsupported;
+
mii->xmii_mode[i] = XMII_MODE_SGMII;
+ mii->special[i] = true;
break;
+ case PHY_INTERFACE_MODE_2500BASEX:
+ if (!priv->info->supports_2500basex[i])
+ goto unsupported;
+
+ mii->xmii_mode[i] = XMII_MODE_SGMII;
+ mii->special[i] = true;
+ break;
+unsupported:
default:
- dev_err(dev, "Unsupported PHY mode %s!\n",
- phy_modes(ports[i].phy_mode));
+ dev_err(dev, "Unsupported PHY mode %s on port %d!\n",
+ phy_modes(priv->phy_mode[i]), i);
+ return -EINVAL;
}
- /* Even though the SerDes port is able to drive SGMII autoneg
- * like a PHY would, from the perspective of the XMII tables,
- * the SGMII port should always be put in MAC mode.
- */
- if (ports[i].phy_mode == PHY_INTERFACE_MODE_SGMII)
- mii->phy_mac[i] = XMII_MAC;
- else
- mii->phy_mac[i] = ports[i].role;
+ mii->phy_mac[i] = role;
}
return 0;
}
static int sja1105_init_l2_lookup_params(struct sja1105_private *priv)
{
- struct sja1105_table *table;
- u64 max_fdb_entries = SJA1105_MAX_L2_LOOKUP_COUNT / SJA1105_NUM_PORTS;
struct sja1105_l2_lookup_params_entry default_l2_lookup_params = {
/* Learned FDB entries are forgotten after 300 seconds */
.maxage = SJA1105_AGEING_TIME_MS(300000),
.dyn_tbsz = SJA1105ET_FDB_BIN_SIZE,
/* And the P/Q/R/S equivalent setting: */
.start_dynspc = 0,
- .maxaddrp = {max_fdb_entries, max_fdb_entries, max_fdb_entries,
- max_fdb_entries, max_fdb_entries, },
/* 2^8 + 2^5 + 2^3 + 2^2 + 2^1 + 1 in Koopman notation */
.poly = 0x97,
/* This selects between Independent VLAN Learning (IVL) and
.owr_dyn = true,
.drpnolearn = true,
};
+ struct dsa_switch *ds = priv->ds;
+ int port, num_used_ports = 0;
+ struct sja1105_table *table;
+ u64 max_fdb_entries;
+
+ for (port = 0; port < ds->num_ports; port++)
+ if (!dsa_is_unused_port(ds, port))
+ num_used_ports++;
+
+ max_fdb_entries = SJA1105_MAX_L2_LOOKUP_COUNT / num_used_ports;
+
+ for (port = 0; port < ds->num_ports; port++) {
+ if (dsa_is_unused_port(ds, port))
+ continue;
+
+ default_l2_lookup_params.maxaddrp[port] = max_fdb_entries;
+ }
table = &priv->static_config.tables[BLK_IDX_L2_LOOKUP_PARAMS];
table->entry_count = 0;
}
- table->entries = kcalloc(SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT,
+ table->entries = kcalloc(table->ops->max_entry_count,
table->ops->unpacked_entry_size, GFP_KERNEL);
if (!table->entries)
return -ENOMEM;
- table->entry_count = SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT;
+ table->entry_count = table->ops->max_entry_count;
/* This table only has a single entry */
((struct sja1105_l2_lookup_params_entry *)table->entries)[0] =
return 0;
}
+/* Set up a default VLAN for untagged traffic injected from the CPU
+ * using management routes (e.g. STP, PTP) as opposed to tag_8021q.
+ * All DT-defined ports are members of this VLAN, and there are no
+ * restrictions on forwarding (since the CPU selects the destination).
+ * Frames from this VLAN will always be transmitted as untagged, and
+ * neither the bridge nor the 8021q module cannot create this VLAN ID.
+ */
static int sja1105_init_static_vlan(struct sja1105_private *priv)
{
struct sja1105_table *table;
struct sja1105_vlan_lookup_entry pvid = {
+ .type_entry = SJA1110_VLAN_D_TAG,
.ving_mirr = 0,
.vegr_mirr = 0,
.vmemb_port = 0,
.vlan_bc = 0,
.tag_port = 0,
- .vlanid = 1,
+ .vlanid = SJA1105_DEFAULT_VLAN,
};
struct dsa_switch *ds = priv->ds;
int port;
table = &priv->static_config.tables[BLK_IDX_VLAN_LOOKUP];
- /* The static VLAN table will only contain the initial pvid of 1.
- * All other VLANs are to be configured through dynamic entries,
- * and kept in the static configuration table as backing memory.
- */
if (table->entry_count) {
kfree(table->entries);
table->entry_count = 0;
table->entry_count = 1;
- /* VLAN 1: all DT-defined ports are members; no restrictions on
- * forwarding; always transmit as untagged.
- */
for (port = 0; port < ds->num_ports; port++) {
struct sja1105_bridge_vlan *v;
pvid.vlan_bc |= BIT(port);
pvid.tag_port &= ~BIT(port);
- /* Let traffic that don't need dsa_8021q (e.g. STP, PTP) be
- * transmitted as untagged.
- */
v = kzalloc(sizeof(*v), GFP_KERNEL);
if (!v)
return -ENOMEM;
v->port = port;
- v->vid = 1;
+ v->vid = SJA1105_DEFAULT_VLAN;
v->untagged = true;
if (dsa_is_cpu_port(ds, port))
v->pvid = true;
static int sja1105_init_l2_forwarding(struct sja1105_private *priv)
{
struct sja1105_l2_forwarding_entry *l2fwd;
+ struct dsa_switch *ds = priv->ds;
struct sja1105_table *table;
int i, j;
table->entry_count = 0;
}
- table->entries = kcalloc(SJA1105_MAX_L2_FORWARDING_COUNT,
+ table->entries = kcalloc(table->ops->max_entry_count,
table->ops->unpacked_entry_size, GFP_KERNEL);
if (!table->entries)
return -ENOMEM;
- table->entry_count = SJA1105_MAX_L2_FORWARDING_COUNT;
+ table->entry_count = table->ops->max_entry_count;
l2fwd = table->entries;
/* First 5 entries define the forwarding rules */
- for (i = 0; i < SJA1105_NUM_PORTS; i++) {
+ for (i = 0; i < ds->num_ports; i++) {
unsigned int upstream = dsa_upstream_port(priv->ds, i);
+ if (dsa_is_unused_port(ds, i))
+ continue;
+
for (j = 0; j < SJA1105_NUM_TC; j++)
l2fwd[i].vlan_pmap[j] = j;
l2fwd[upstream].bc_domain |= BIT(i);
l2fwd[upstream].fl_domain |= BIT(i);
}
+
/* Next 8 entries define VLAN PCP mapping from ingress to egress.
* Create a one-to-one mapping.
*/
- for (i = 0; i < SJA1105_NUM_TC; i++)
- for (j = 0; j < SJA1105_NUM_PORTS; j++)
- l2fwd[SJA1105_NUM_PORTS + i].vlan_pmap[j] = i;
+ for (i = 0; i < SJA1105_NUM_TC; i++) {
+ for (j = 0; j < ds->num_ports; j++) {
+ if (dsa_is_unused_port(ds, j))
+ continue;
+
+ l2fwd[ds->num_ports + i].vlan_pmap[j] = i;
+ }
+
+ l2fwd[ds->num_ports + i].type_egrpcp2outputq = true;
+ }
+
+ return 0;
+}
+
+static int sja1110_init_pcp_remapping(struct sja1105_private *priv)
+{
+ struct sja1110_pcp_remapping_entry *pcp_remap;
+ struct dsa_switch *ds = priv->ds;
+ struct sja1105_table *table;
+ int port, tc;
+
+ table = &priv->static_config.tables[BLK_IDX_PCP_REMAPPING];
+
+ /* Nothing to do for SJA1105 */
+ if (!table->ops->max_entry_count)
+ return 0;
+
+ if (table->entry_count) {
+ kfree(table->entries);
+ table->entry_count = 0;
+ }
+
+ table->entries = kcalloc(table->ops->max_entry_count,
+ table->ops->unpacked_entry_size, GFP_KERNEL);
+ if (!table->entries)
+ return -ENOMEM;
+
+ table->entry_count = table->ops->max_entry_count;
+
+ pcp_remap = table->entries;
+
+ /* Repeat the configuration done for vlan_pmap */
+ for (port = 0; port < ds->num_ports; port++) {
+ if (dsa_is_unused_port(ds, port))
+ continue;
+
+ for (tc = 0; tc < SJA1105_NUM_TC; tc++)
+ pcp_remap[port].egrpcp[tc] = tc;
+ }
return 0;
}
static int sja1105_init_l2_forwarding_params(struct sja1105_private *priv)
{
- struct sja1105_l2_forwarding_params_entry default_l2fwd_params = {
- /* Disallow dynamic reconfiguration of vlan_pmap */
- .max_dynp = 0,
- /* Use a single memory partition for all ingress queues */
- .part_spc = { SJA1105_MAX_FRAME_MEMORY, 0, 0, 0, 0, 0, 0, 0 },
- };
+ struct sja1105_l2_forwarding_params_entry *l2fwd_params;
struct sja1105_table *table;
table = &priv->static_config.tables[BLK_IDX_L2_FORWARDING_PARAMS];
table->entry_count = 0;
}
- table->entries = kcalloc(SJA1105_MAX_L2_FORWARDING_PARAMS_COUNT,
+ table->entries = kcalloc(table->ops->max_entry_count,
table->ops->unpacked_entry_size, GFP_KERNEL);
if (!table->entries)
return -ENOMEM;
- table->entry_count = SJA1105_MAX_L2_FORWARDING_PARAMS_COUNT;
+ table->entry_count = table->ops->max_entry_count;
/* This table only has a single entry */
- ((struct sja1105_l2_forwarding_params_entry *)table->entries)[0] =
- default_l2fwd_params;
+ l2fwd_params = table->entries;
+
+ /* Disallow dynamic reconfiguration of vlan_pmap */
+ l2fwd_params->max_dynp = 0;
+ /* Use a single memory partition for all ingress queues */
+ l2fwd_params->part_spc[0] = priv->info->max_frame_mem;
return 0;
}
{
struct sja1105_l2_forwarding_params_entry *l2_fwd_params;
struct sja1105_vl_forwarding_params_entry *vl_fwd_params;
+ int max_mem = priv->info->max_frame_mem;
struct sja1105_table *table;
- int max_mem;
/* VLAN retagging is implemented using a loopback port that consumes
* frame buffers. That leaves less for us.
*/
if (priv->vlan_state == SJA1105_VLAN_BEST_EFFORT)
- max_mem = SJA1105_MAX_FRAME_MEMORY_RETAGGING;
- else
- max_mem = SJA1105_MAX_FRAME_MEMORY;
+ max_mem -= SJA1105_FRAME_MEMORY_RETAGGING_OVERHEAD;
table = &priv->static_config.tables[BLK_IDX_L2_FORWARDING_PARAMS];
l2_fwd_params = table->entries;
vl_fwd_params->partspc[0] = SJA1105_VL_FRAME_MEMORY;
}
+/* SJA1110 TDMACONFIGIDX values:
+ *
+ * | 100 Mbps ports | 1Gbps ports | 2.5Gbps ports | Disabled ports
+ * -----+----------------+---------------+---------------+---------------
+ * 0 | 0, [5:10] | [1:2] | [3:4] | retag
+ * 1 |0, [5:10], retag| [1:2] | [3:4] | -
+ * 2 | 0, [5:10] | [1:3], retag | 4 | -
+ * 3 | 0, [5:10] |[1:2], 4, retag| 3 | -
+ * 4 | 0, 2, [5:10] | 1, retag | [3:4] | -
+ * 5 | 0, 1, [5:10] | 2, retag | [3:4] | -
+ * 14 | 0, [5:10] | [1:4], retag | - | -
+ * 15 | [5:10] | [0:4], retag | - | -
+ */
+static void sja1110_select_tdmaconfigidx(struct sja1105_private *priv)
+{
+ struct sja1105_general_params_entry *general_params;
+ struct sja1105_table *table;
+ bool port_1_is_base_tx;
+ bool port_3_is_2500;
+ bool port_4_is_2500;
+ u64 tdmaconfigidx;
+
+ if (priv->info->device_id != SJA1110_DEVICE_ID)
+ return;
+
+ table = &priv->static_config.tables[BLK_IDX_GENERAL_PARAMS];
+ general_params = table->entries;
+
+ /* All the settings below are "as opposed to SGMII", which is the
+ * other pinmuxing option.
+ */
+ port_1_is_base_tx = priv->phy_mode[1] == PHY_INTERFACE_MODE_INTERNAL;
+ port_3_is_2500 = priv->phy_mode[3] == PHY_INTERFACE_MODE_2500BASEX;
+ port_4_is_2500 = priv->phy_mode[4] == PHY_INTERFACE_MODE_2500BASEX;
+
+ if (port_1_is_base_tx)
+ /* Retagging port will operate at 1 Gbps */
+ tdmaconfigidx = 5;
+ else if (port_3_is_2500 && port_4_is_2500)
+ /* Retagging port will operate at 100 Mbps */
+ tdmaconfigidx = 1;
+ else if (port_3_is_2500)
+ /* Retagging port will operate at 1 Gbps */
+ tdmaconfigidx = 3;
+ else if (port_4_is_2500)
+ /* Retagging port will operate at 1 Gbps */
+ tdmaconfigidx = 2;
+ else
+ /* Retagging port will operate at 1 Gbps */
+ tdmaconfigidx = 14;
+
+ general_params->tdmaconfigidx = tdmaconfigidx;
+}
+
static int sja1105_init_general_params(struct sja1105_private *priv)
{
struct sja1105_general_params_entry default_general_params = {
* receieved on host_port itself would be dropped, except
* by installing a temporary 'management route'
*/
- .host_port = dsa_upstream_port(priv->ds, 0),
+ .host_port = priv->ds->num_ports,
/* Default to an invalid value */
- .mirr_port = SJA1105_NUM_PORTS,
- /* Link-local traffic received on casc_port will be forwarded
- * to host_port without embedding the source port and device ID
- * info in the destination MAC address (presumably because it
- * is a cascaded port and a downstream SJA switch already did
- * that). Default to an invalid port (to disable the feature)
- * and overwrite this if we find any DSA (cascaded) ports.
- */
- .casc_port = SJA1105_NUM_PORTS,
+ .mirr_port = priv->ds->num_ports,
/* No TTEthernet */
.vllupformat = SJA1105_VL_FORMAT_PSFP,
.vlmarker = 0,
*/
.tpid = ETH_P_SJA1105,
.tpid2 = ETH_P_SJA1105,
+ /* Enable the TTEthernet engine on SJA1110 */
+ .tte_en = true,
+ /* Set up the EtherType for control packets on SJA1110 */
+ .header_type = ETH_P_SJA1110,
};
+ struct sja1105_general_params_entry *general_params;
+ struct dsa_switch *ds = priv->ds;
struct sja1105_table *table;
+ int port;
+
+ for (port = 0; port < ds->num_ports; port++) {
+ if (dsa_is_cpu_port(ds, port)) {
+ default_general_params.host_port = port;
+ break;
+ }
+ }
table = &priv->static_config.tables[BLK_IDX_GENERAL_PARAMS];
table->entry_count = 0;
}
- table->entries = kcalloc(SJA1105_MAX_GENERAL_PARAMS_COUNT,
+ table->entries = kcalloc(table->ops->max_entry_count,
table->ops->unpacked_entry_size, GFP_KERNEL);
if (!table->entries)
return -ENOMEM;
- table->entry_count = SJA1105_MAX_GENERAL_PARAMS_COUNT;
+ table->entry_count = table->ops->max_entry_count;
+
+ general_params = table->entries;
/* This table only has a single entry */
- ((struct sja1105_general_params_entry *)table->entries)[0] =
- default_general_params;
+ general_params[0] = default_general_params;
+
+ sja1110_select_tdmaconfigidx(priv);
+
+ /* Link-local traffic received on casc_port will be forwarded
+ * to host_port without embedding the source port and device ID
+ * info in the destination MAC address, and no RX timestamps will be
+ * taken either (presumably because it is a cascaded port and a
+ * downstream SJA switch already did that).
+ * To disable the feature, we need to do different things depending on
+ * switch generation. On SJA1105 we need to set an invalid port, while
+ * on SJA1110 which support multiple cascaded ports, this field is a
+ * bitmask so it must be left zero.
+ */
+ if (!priv->info->multiple_cascade_ports)
+ general_params->casc_port = ds->num_ports;
return 0;
}
table->entry_count = 0;
}
- table->entries = kcalloc(SJA1105_MAX_AVB_PARAMS_COUNT,
+ table->entries = kcalloc(table->ops->max_entry_count,
table->ops->unpacked_entry_size, GFP_KERNEL);
if (!table->entries)
return -ENOMEM;
- table->entry_count = SJA1105_MAX_AVB_PARAMS_COUNT;
+ table->entry_count = table->ops->max_entry_count;
avb = table->entries;
static int sja1105_init_l2_policing(struct sja1105_private *priv)
{
struct sja1105_l2_policing_entry *policing;
+ struct dsa_switch *ds = priv->ds;
struct sja1105_table *table;
int port, tc;
table->entry_count = 0;
}
- table->entries = kcalloc(SJA1105_MAX_L2_POLICING_COUNT,
+ table->entries = kcalloc(table->ops->max_entry_count,
table->ops->unpacked_entry_size, GFP_KERNEL);
if (!table->entries)
return -ENOMEM;
- table->entry_count = SJA1105_MAX_L2_POLICING_COUNT;
+ table->entry_count = table->ops->max_entry_count;
policing = table->entries;
/* Setup shared indices for the matchall policers */
- for (port = 0; port < SJA1105_NUM_PORTS; port++) {
- int bcast = (SJA1105_NUM_PORTS * SJA1105_NUM_TC) + port;
+ for (port = 0; port < ds->num_ports; port++) {
+ int mcast = (ds->num_ports * (SJA1105_NUM_TC + 1)) + port;
+ int bcast = (ds->num_ports * SJA1105_NUM_TC) + port;
for (tc = 0; tc < SJA1105_NUM_TC; tc++)
policing[port * SJA1105_NUM_TC + tc].sharindx = port;
policing[bcast].sharindx = port;
+ /* Only SJA1110 has multicast policers */
+ if (mcast <= table->ops->max_entry_count)
+ policing[mcast].sharindx = port;
}
/* Setup the matchall policer parameters */
- for (port = 0; port < SJA1105_NUM_PORTS; port++) {
+ for (port = 0; port < ds->num_ports; port++) {
int mtu = VLAN_ETH_FRAME_LEN + ETH_FCS_LEN;
if (dsa_is_cpu_port(priv->ds, port))
return 0;
}
-static int sja1105_static_config_load(struct sja1105_private *priv,
- struct sja1105_dt_port *ports)
+static int sja1105_static_config_load(struct sja1105_private *priv)
{
int rc;
rc = sja1105_init_mac_settings(priv);
if (rc < 0)
return rc;
- rc = sja1105_init_mii_settings(priv, ports);
+ rc = sja1105_init_mii_settings(priv);
if (rc < 0)
return rc;
rc = sja1105_init_static_fdb(priv);
rc = sja1105_init_avb_params(priv);
if (rc < 0)
return rc;
+ rc = sja1110_init_pcp_remapping(priv);
+ if (rc < 0)
+ return rc;
/* Send initial configuration to hardware via SPI */
return sja1105_static_config_upload(priv);
}
-static int sja1105_parse_rgmii_delays(struct sja1105_private *priv,
- const struct sja1105_dt_port *ports)
+static int sja1105_parse_rgmii_delays(struct sja1105_private *priv)
{
- int i;
+ struct dsa_switch *ds = priv->ds;
+ int port;
- for (i = 0; i < SJA1105_NUM_PORTS; i++) {
- if (ports[i].role == XMII_MAC)
+ for (port = 0; port < ds->num_ports; port++) {
+ if (!priv->fixed_link[port])
continue;
- if (ports[i].phy_mode == PHY_INTERFACE_MODE_RGMII_RXID ||
- ports[i].phy_mode == PHY_INTERFACE_MODE_RGMII_ID)
- priv->rgmii_rx_delay[i] = true;
+ if (priv->phy_mode[port] == PHY_INTERFACE_MODE_RGMII_RXID ||
+ priv->phy_mode[port] == PHY_INTERFACE_MODE_RGMII_ID)
+ priv->rgmii_rx_delay[port] = true;
- if (ports[i].phy_mode == PHY_INTERFACE_MODE_RGMII_TXID ||
- ports[i].phy_mode == PHY_INTERFACE_MODE_RGMII_ID)
- priv->rgmii_tx_delay[i] = true;
+ if (priv->phy_mode[port] == PHY_INTERFACE_MODE_RGMII_TXID ||
+ priv->phy_mode[port] == PHY_INTERFACE_MODE_RGMII_ID)
+ priv->rgmii_tx_delay[port] = true;
- if ((priv->rgmii_rx_delay[i] || priv->rgmii_tx_delay[i]) &&
- !priv->info->setup_rgmii_delay)
+ if ((priv->rgmii_rx_delay[port] || priv->rgmii_tx_delay[port]) &&
+ !priv->info->setup_rgmii_delay)
return -EINVAL;
}
return 0;
}
static int sja1105_parse_ports_node(struct sja1105_private *priv,
- struct sja1105_dt_port *ports,
struct device_node *ports_node)
{
struct device *dev = &priv->spidev->dev;
of_node_put(child);
return -ENODEV;
}
- ports[index].phy_mode = phy_mode;
phy_node = of_parse_phandle(child, "phy-handle", 0);
if (!phy_node) {
/* phy-handle is missing, but fixed-link isn't.
* So it's a fixed link. Default to PHY role.
*/
- ports[index].role = XMII_PHY;
+ priv->fixed_link[index] = true;
} else {
- /* phy-handle present => put port in MAC role */
- ports[index].role = XMII_MAC;
of_node_put(phy_node);
}
- /* The MAC/PHY role can be overridden with explicit bindings */
- if (of_property_read_bool(child, "sja1105,role-mac"))
- ports[index].role = XMII_MAC;
- else if (of_property_read_bool(child, "sja1105,role-phy"))
- ports[index].role = XMII_PHY;
+ priv->phy_mode[index] = phy_mode;
}
return 0;
}
-static int sja1105_parse_dt(struct sja1105_private *priv,
- struct sja1105_dt_port *ports)
+static int sja1105_parse_dt(struct sja1105_private *priv)
{
struct device *dev = &priv->spidev->dev;
struct device_node *switch_node = dev->of_node;
int rc;
ports_node = of_get_child_by_name(switch_node, "ports");
+ if (!ports_node)
+ ports_node = of_get_child_by_name(switch_node, "ethernet-ports");
if (!ports_node) {
dev_err(dev, "Incorrect bindings: absent \"ports\" node\n");
return -ENODEV;
}
- rc = sja1105_parse_ports_node(priv, ports, ports_node);
+ rc = sja1105_parse_ports_node(priv, ports_node);
of_node_put(ports_node);
return rc;
}
-static int sja1105_sgmii_read(struct sja1105_private *priv, int pcs_reg)
-{
- const struct sja1105_regs *regs = priv->info->regs;
- u32 val;
- int rc;
-
- rc = sja1105_xfer_u32(priv, SPI_READ, regs->sgmii + pcs_reg, &val,
- NULL);
- if (rc < 0)
- return rc;
-
- return val;
-}
-
-static int sja1105_sgmii_write(struct sja1105_private *priv, int pcs_reg,
- u16 pcs_val)
-{
- const struct sja1105_regs *regs = priv->info->regs;
- u32 val = pcs_val;
- int rc;
-
- rc = sja1105_xfer_u32(priv, SPI_WRITE, regs->sgmii + pcs_reg, &val,
- NULL);
- if (rc < 0)
- return rc;
-
- return val;
-}
-
-static void sja1105_sgmii_pcs_config(struct sja1105_private *priv,
- bool an_enabled, bool an_master)
-{
- u16 ac = SJA1105_AC_AUTONEG_MODE_SGMII;
-
- /* DIGITAL_CONTROL_1: Enable vendor-specific MMD1, allow the PHY to
- * stop the clock during LPI mode, make the MAC reconfigure
- * autonomously after PCS autoneg is done, flush the internal FIFOs.
- */
- sja1105_sgmii_write(priv, SJA1105_DC1, SJA1105_DC1_EN_VSMMD1 |
- SJA1105_DC1_CLOCK_STOP_EN |
- SJA1105_DC1_MAC_AUTO_SW |
- SJA1105_DC1_INIT);
- /* DIGITAL_CONTROL_2: No polarity inversion for TX and RX lanes */
- sja1105_sgmii_write(priv, SJA1105_DC2, SJA1105_DC2_TX_POL_INV_DISABLE);
- /* AUTONEG_CONTROL: Use SGMII autoneg */
- if (an_master)
- ac |= SJA1105_AC_PHY_MODE | SJA1105_AC_SGMII_LINK;
- sja1105_sgmii_write(priv, SJA1105_AC, ac);
- /* BASIC_CONTROL: enable in-band AN now, if requested. Otherwise,
- * sja1105_sgmii_pcs_force_speed must be called later for the link
- * to become operational.
- */
- if (an_enabled)
- sja1105_sgmii_write(priv, MII_BMCR,
- BMCR_ANENABLE | BMCR_ANRESTART);
-}
-
-static void sja1105_sgmii_pcs_force_speed(struct sja1105_private *priv,
- int speed)
+/* Convert link speed from SJA1105 to ethtool encoding */
+static int sja1105_port_speed_to_ethtool(struct sja1105_private *priv,
+ u64 speed)
{
- int pcs_speed;
-
- switch (speed) {
- case SPEED_1000:
- pcs_speed = BMCR_SPEED1000;
- break;
- case SPEED_100:
- pcs_speed = BMCR_SPEED100;
- break;
- case SPEED_10:
- pcs_speed = BMCR_SPEED10;
- break;
- default:
- dev_err(priv->ds->dev, "Invalid speed %d\n", speed);
- return;
- }
- sja1105_sgmii_write(priv, MII_BMCR, pcs_speed | BMCR_FULLDPLX);
+ if (speed == priv->info->port_speed[SJA1105_SPEED_10MBPS])
+ return SPEED_10;
+ if (speed == priv->info->port_speed[SJA1105_SPEED_100MBPS])
+ return SPEED_100;
+ if (speed == priv->info->port_speed[SJA1105_SPEED_1000MBPS])
+ return SPEED_1000;
+ if (speed == priv->info->port_speed[SJA1105_SPEED_2500MBPS])
+ return SPEED_2500;
+ return SPEED_UNKNOWN;
}
-/* Convert link speed from SJA1105 to ethtool encoding */
-static int sja1105_speed[] = {
- [SJA1105_SPEED_AUTO] = SPEED_UNKNOWN,
- [SJA1105_SPEED_10MBPS] = SPEED_10,
- [SJA1105_SPEED_100MBPS] = SPEED_100,
- [SJA1105_SPEED_1000MBPS] = SPEED_1000,
-};
-
/* Set link speed in the MAC configuration for a specific port. */
static int sja1105_adjust_port_config(struct sja1105_private *priv, int port,
int speed_mbps)
{
- struct sja1105_xmii_params_entry *mii;
struct sja1105_mac_config_entry *mac;
struct device *dev = priv->ds->dev;
- sja1105_phy_interface_t phy_mode;
- sja1105_speed_t speed;
+ u64 speed;
int rc;
/* On P/Q/R/S, one can read from the device via the MAC reconfiguration
* reasonable approximation for both E/T and P/Q/R/S.
*/
mac = priv->static_config.tables[BLK_IDX_MAC_CONFIG].entries;
- mii = priv->static_config.tables[BLK_IDX_XMII_PARAMS].entries;
switch (speed_mbps) {
case SPEED_UNKNOWN:
* ok for power consumption in case AN will never complete -
* otherwise PHYLINK should come back with a new update.
*/
- speed = SJA1105_SPEED_AUTO;
+ speed = priv->info->port_speed[SJA1105_SPEED_AUTO];
break;
case SPEED_10:
- speed = SJA1105_SPEED_10MBPS;
+ speed = priv->info->port_speed[SJA1105_SPEED_10MBPS];
break;
case SPEED_100:
- speed = SJA1105_SPEED_100MBPS;
+ speed = priv->info->port_speed[SJA1105_SPEED_100MBPS];
break;
case SPEED_1000:
- speed = SJA1105_SPEED_1000MBPS;
+ speed = priv->info->port_speed[SJA1105_SPEED_1000MBPS];
+ break;
+ case SPEED_2500:
+ speed = priv->info->port_speed[SJA1105_SPEED_2500MBPS];
break;
default:
dev_err(dev, "Invalid speed %iMbps\n", speed_mbps);
* Actually for the SGMII port, the MAC is fixed at 1 Gbps and
* we need to configure the PCS only (if even that).
*/
- if (sja1105_supports_sgmii(priv, port))
- mac[port].speed = SJA1105_SPEED_1000MBPS;
+ if (priv->phy_mode[port] == PHY_INTERFACE_MODE_SGMII)
+ mac[port].speed = priv->info->port_speed[SJA1105_SPEED_1000MBPS];
+ else if (priv->phy_mode[port] == PHY_INTERFACE_MODE_2500BASEX)
+ mac[port].speed = priv->info->port_speed[SJA1105_SPEED_2500MBPS];
else
mac[port].speed = speed;
* the clock setup does interrupt the clock signal for a certain time
* which causes trouble for all PHYs relying on this signal.
*/
- phy_mode = mii->xmii_mode[port];
- if (phy_mode != XMII_MODE_RGMII)
+ if (!phy_interface_mode_is_rgmii(priv->phy_mode[port]))
return 0;
return sja1105_clocking_setup_port(priv, port);
static bool sja1105_phy_mode_mismatch(struct sja1105_private *priv, int port,
phy_interface_t interface)
{
- struct sja1105_xmii_params_entry *mii;
- sja1105_phy_interface_t phy_mode;
-
- mii = priv->static_config.tables[BLK_IDX_XMII_PARAMS].entries;
- phy_mode = mii->xmii_mode[port];
-
- switch (interface) {
- case PHY_INTERFACE_MODE_MII:
- return (phy_mode != XMII_MODE_MII);
- case PHY_INTERFACE_MODE_RMII:
- return (phy_mode != XMII_MODE_RMII);
- case PHY_INTERFACE_MODE_RGMII:
- case PHY_INTERFACE_MODE_RGMII_ID:
- case PHY_INTERFACE_MODE_RGMII_RXID:
- case PHY_INTERFACE_MODE_RGMII_TXID:
- return (phy_mode != XMII_MODE_RGMII);
- case PHY_INTERFACE_MODE_SGMII:
- return (phy_mode != XMII_MODE_SGMII);
- default:
- return true;
- }
+ return priv->phy_mode[port] != interface;
}
static void sja1105_mac_config(struct dsa_switch *ds, int port,
unsigned int mode,
const struct phylink_link_state *state)
{
+ struct dsa_port *dp = dsa_to_port(ds, port);
struct sja1105_private *priv = ds->priv;
- bool is_sgmii = sja1105_supports_sgmii(priv, port);
+ struct dw_xpcs *xpcs;
if (sja1105_phy_mode_mismatch(priv, port, state->interface)) {
dev_err(ds->dev, "Changing PHY mode to %s not supported!\n",
return;
}
- if (phylink_autoneg_inband(mode) && !is_sgmii) {
- dev_err(ds->dev, "In-band AN not supported!\n");
- return;
- }
+ xpcs = priv->xpcs[port];
- if (is_sgmii)
- sja1105_sgmii_pcs_config(priv, phylink_autoneg_inband(mode),
- false);
+ if (xpcs)
+ phylink_set_pcs(dp->pl, &xpcs->pcs);
}
static void sja1105_mac_link_down(struct dsa_switch *ds, int port,
sja1105_adjust_port_config(priv, port, speed);
- if (sja1105_supports_sgmii(priv, port) && !phylink_autoneg_inband(mode))
- sja1105_sgmii_pcs_force_speed(priv, speed);
-
sja1105_inhibit_tx(priv, BIT(port), false);
}
if (mii->xmii_mode[port] == XMII_MODE_RGMII ||
mii->xmii_mode[port] == XMII_MODE_SGMII)
phylink_set(mask, 1000baseT_Full);
+ if (priv->info->supports_2500basex[port]) {
+ phylink_set(mask, 2500baseT_Full);
+ phylink_set(mask, 2500baseX_Full);
+ }
bitmap_and(supported, supported, mask, __ETHTOOL_LINK_MODE_MASK_NBITS);
bitmap_and(state->advertising, state->advertising, mask,
__ETHTOOL_LINK_MODE_MASK_NBITS);
}
-static int sja1105_mac_pcs_get_state(struct dsa_switch *ds, int port,
- struct phylink_link_state *state)
-{
- struct sja1105_private *priv = ds->priv;
- int ais;
-
- /* Read the vendor-specific AUTONEG_INTR_STATUS register */
- ais = sja1105_sgmii_read(priv, SJA1105_AIS);
- if (ais < 0)
- return ais;
-
- switch (SJA1105_AIS_SPEED(ais)) {
- case 0:
- state->speed = SPEED_10;
- break;
- case 1:
- state->speed = SPEED_100;
- break;
- case 2:
- state->speed = SPEED_1000;
- break;
- default:
- dev_err(ds->dev, "Invalid SGMII PCS speed %lu\n",
- SJA1105_AIS_SPEED(ais));
- }
- state->duplex = SJA1105_AIS_DUPLEX_MODE(ais);
- state->an_complete = SJA1105_AIS_COMPLETE(ais);
- state->link = SJA1105_AIS_LINK_STATUS(ais);
-
- return 0;
-}
-
static int
sja1105_find_static_fdb_entry(struct sja1105_private *priv, int port,
const struct sja1105_l2_lookup_entry *requested)
l2_fwd = priv->static_config.tables[BLK_IDX_L2_FORWARDING].entries;
- for (i = 0; i < SJA1105_NUM_PORTS; i++) {
+ for (i = 0; i < ds->num_ports; i++) {
/* Add this port to the forwarding matrix of the
* other ports in the same bridge, and viceversa.
*/
{
struct ptp_system_timestamp ptp_sts_before;
struct ptp_system_timestamp ptp_sts_after;
+ int speed_mbps[SJA1105_MAX_NUM_PORTS];
+ u16 bmcr[SJA1105_MAX_NUM_PORTS] = {0};
struct sja1105_mac_config_entry *mac;
- int speed_mbps[SJA1105_NUM_PORTS];
struct dsa_switch *ds = priv->ds;
s64 t1, t2, t3, t4;
s64 t12, t34;
- u16 bmcr = 0;
int rc, i;
s64 now;
* switch wants to see in the static config in order to allow us to
* change it through the dynamic interface later.
*/
- for (i = 0; i < SJA1105_NUM_PORTS; i++) {
- speed_mbps[i] = sja1105_speed[mac[i].speed];
- mac[i].speed = SJA1105_SPEED_AUTO;
- }
+ for (i = 0; i < ds->num_ports; i++) {
+ u32 reg_addr = mdiobus_c45_addr(MDIO_MMD_VEND2, MDIO_CTRL1);
+
+ speed_mbps[i] = sja1105_port_speed_to_ethtool(priv,
+ mac[i].speed);
+ mac[i].speed = priv->info->port_speed[SJA1105_SPEED_AUTO];
- if (sja1105_supports_sgmii(priv, SJA1105_SGMII_PORT))
- bmcr = sja1105_sgmii_read(priv, MII_BMCR);
+ if (priv->xpcs[i])
+ bmcr[i] = mdiobus_read(priv->mdio_pcs, i, reg_addr);
+ }
/* No PTP operations can run right now */
mutex_lock(&priv->ptp_data.lock);
rc = __sja1105_ptp_gettimex(ds, &now, &ptp_sts_before);
- if (rc < 0)
- goto out_unlock_ptp;
+ if (rc < 0) {
+ mutex_unlock(&priv->ptp_data.lock);
+ goto out;
+ }
/* Reset switch and send updated static configuration */
rc = sja1105_static_config_upload(priv);
- if (rc < 0)
- goto out_unlock_ptp;
+ if (rc < 0) {
+ mutex_unlock(&priv->ptp_data.lock);
+ goto out;
+ }
rc = __sja1105_ptp_settime(ds, 0, &ptp_sts_after);
- if (rc < 0)
- goto out_unlock_ptp;
+ if (rc < 0) {
+ mutex_unlock(&priv->ptp_data.lock);
+ goto out;
+ }
t1 = timespec64_to_ns(&ptp_sts_before.pre_ts);
t2 = timespec64_to_ns(&ptp_sts_before.post_ts);
__sja1105_ptp_adjtime(ds, now);
-out_unlock_ptp:
mutex_unlock(&priv->ptp_data.lock);
dev_info(priv->ds->dev,
* For these interfaces there is no dynamic configuration
* needed, since PLLs have same settings at all speeds.
*/
- rc = sja1105_clocking_setup(priv);
- if (rc < 0)
- goto out;
+ if (priv->info->clocking_setup) {
+ rc = priv->info->clocking_setup(priv);
+ if (rc < 0)
+ goto out;
+ }
+
+ for (i = 0; i < ds->num_ports; i++) {
+ struct dw_xpcs *xpcs = priv->xpcs[i];
+ unsigned int mode;
- for (i = 0; i < SJA1105_NUM_PORTS; i++) {
rc = sja1105_adjust_port_config(priv, i, speed_mbps[i]);
if (rc < 0)
goto out;
- }
- if (sja1105_supports_sgmii(priv, SJA1105_SGMII_PORT)) {
- bool an_enabled = !!(bmcr & BMCR_ANENABLE);
+ if (!xpcs)
+ continue;
- sja1105_sgmii_pcs_config(priv, an_enabled, false);
+ if (bmcr[i] & BMCR_ANENABLE)
+ mode = MLO_AN_INBAND;
+ else if (priv->fixed_link[i])
+ mode = MLO_AN_FIXED;
+ else
+ mode = MLO_AN_PHY;
+
+ rc = xpcs_do_config(xpcs, priv->phy_mode[i], mode);
+ if (rc < 0)
+ goto out;
- if (!an_enabled) {
+ if (!phylink_autoneg_inband(mode)) {
int speed = SPEED_UNKNOWN;
- if (bmcr & BMCR_SPEED1000)
+ if (priv->phy_mode[i] == PHY_INTERFACE_MODE_2500BASEX)
+ speed = SPEED_2500;
+ else if (bmcr[i] & BMCR_SPEED1000)
speed = SPEED_1000;
- else if (bmcr & BMCR_SPEED100)
+ else if (bmcr[i] & BMCR_SPEED100)
speed = SPEED_100;
else
speed = SPEED_10;
- sja1105_sgmii_pcs_force_speed(priv, speed);
+ xpcs_link_up(&xpcs->pcs, mode, priv->phy_mode[i],
+ speed, DUPLEX_FULL);
}
}
sja1105_get_tag_protocol(struct dsa_switch *ds, int port,
enum dsa_tag_protocol mp)
{
- return DSA_TAG_PROTO_SJA1105;
+ struct sja1105_private *priv = ds->priv;
+
+ return priv->info->tag_proto;
}
static int sja1105_find_free_subvlan(u16 *subvlan_map, bool pvid)
new_vlan[match].vlan_bc |= BIT(v->port);
if (!v->untagged)
new_vlan[match].tag_port |= BIT(v->port);
+ new_vlan[match].type_entry = SJA1110_VLAN_D_TAG;
}
return 0;
new_vlan[match].vlan_bc |= BIT(v->port);
if (!v->untagged)
new_vlan[match].tag_port |= BIT(v->port);
+ new_vlan[match].type_entry = SJA1110_VLAN_D_TAG;
}
return 0;
new_vlan[match].tag_port |= BIT(v->port);
/* But it's always tagged towards the CPU */
new_vlan[match].tag_port |= BIT(upstream);
+ new_vlan[match].type_entry = SJA1110_VLAN_D_TAG;
/* The Retagging Table generates packet *clones* with
* the new VLAN. This is a very odd hardware quirk
if (!tmp->untagged)
new_vlan[match].tag_port |= BIT(tmp->port);
new_vlan[match].tag_port |= BIT(upstream);
+ new_vlan[match].type_entry = SJA1110_VLAN_D_TAG;
/* Deny egress of @rx_vid towards our front-panel port.
* This will force the switch to drop it, and we'll see
* only the re-retagged packets (having the original,
static int sja1105_build_vlan_table(struct sja1105_private *priv, bool notify)
{
- u16 subvlan_map[SJA1105_NUM_PORTS][DSA_8021Q_N_SUBVLAN];
+ u16 subvlan_map[SJA1105_MAX_NUM_PORTS][DSA_8021Q_N_SUBVLAN];
struct sja1105_retagging_entry *new_retagging;
struct sja1105_vlan_lookup_entry *new_vlan;
struct sja1105_table *table;
bool pvid = flags & BRIDGE_VLAN_INFO_PVID;
struct sja1105_bridge_vlan *v;
- list_for_each_entry(v, vlan_list, list)
- if (v->port == port && v->vid == vid &&
- v->untagged == untagged && v->pvid == pvid)
+ list_for_each_entry(v, vlan_list, list) {
+ if (v->port == port && v->vid == vid) {
/* Already added */
- return 0;
+ if (v->untagged == untagged && v->pvid == pvid)
+ /* Nothing changed */
+ return 0;
+
+ /* It's the same VLAN, but some of the flags changed
+ * and the user did not bother to delete it first.
+ * Update it and trigger sja1105_build_vlan_table.
+ */
+ v->untagged = untagged;
+ v->pvid = pvid;
+ return 1;
+ }
+ }
v = kzalloc(sizeof(*v), GFP_KERNEL);
if (!v) {
*/
static int sja1105_setup(struct dsa_switch *ds)
{
- struct sja1105_dt_port ports[SJA1105_NUM_PORTS];
struct sja1105_private *priv = ds->priv;
int rc;
- rc = sja1105_parse_dt(priv, ports);
+ rc = sja1105_parse_dt(priv);
if (rc < 0) {
dev_err(ds->dev, "Failed to parse DT: %d\n", rc);
return rc;
/* Error out early if internal delays are required through DT
* and we can't apply them.
*/
- rc = sja1105_parse_rgmii_delays(priv, ports);
+ rc = sja1105_parse_rgmii_delays(priv);
if (rc < 0) {
dev_err(ds->dev, "RGMII delay not supported\n");
return rc;
dev_err(ds->dev, "Failed to register PTP clock: %d\n", rc);
return rc;
}
+
+ rc = sja1105_mdiobus_register(ds);
+ if (rc < 0) {
+ dev_err(ds->dev, "Failed to register MDIO bus: %pe\n",
+ ERR_PTR(rc));
+ goto out_ptp_clock_unregister;
+ }
+
+ if (priv->info->disable_microcontroller) {
+ rc = priv->info->disable_microcontroller(priv);
+ if (rc < 0) {
+ dev_err(ds->dev,
+ "Failed to disable microcontroller: %pe\n",
+ ERR_PTR(rc));
+ goto out_mdiobus_unregister;
+ }
+ }
+
/* Create and send configuration down to device */
- rc = sja1105_static_config_load(priv, ports);
+ rc = sja1105_static_config_load(priv);
if (rc < 0) {
dev_err(ds->dev, "Failed to load static config: %d\n", rc);
- return rc;
+ goto out_mdiobus_unregister;
}
+
/* Configure the CGU (PHY link modes and speeds) */
- rc = sja1105_clocking_setup(priv);
- if (rc < 0) {
- dev_err(ds->dev, "Failed to configure MII clocking: %d\n", rc);
- return rc;
+ if (priv->info->clocking_setup) {
+ rc = priv->info->clocking_setup(priv);
+ if (rc < 0) {
+ dev_err(ds->dev,
+ "Failed to configure MII clocking: %pe\n",
+ ERR_PTR(rc));
+ goto out_static_config_free;
+ }
}
+
/* On SJA1105, VLAN filtering per se is always enabled in hardware.
* The only thing we can do to disable it is lie about what the 802.1Q
* EtherType is.
rc = sja1105_devlink_setup(ds);
if (rc < 0)
- return rc;
+ goto out_static_config_free;
/* The DSA/switchdev model brings up switch ports in standalone mode by
* default, and that means vlan_filtering is 0 since they're not under
rtnl_lock();
rc = sja1105_setup_8021q_tagging(ds, true);
rtnl_unlock();
+ if (rc)
+ goto out_devlink_teardown;
+
+ return 0;
+
+out_devlink_teardown:
+ sja1105_devlink_teardown(ds);
+out_mdiobus_unregister:
+ sja1105_mdiobus_unregister(ds);
+out_ptp_clock_unregister:
+ sja1105_ptp_clock_unregister(ds);
+out_static_config_free:
+ sja1105_static_config_free(&priv->static_config);
return rc;
}
struct sja1105_bridge_vlan *v, *n;
int port;
- for (port = 0; port < SJA1105_NUM_PORTS; port++) {
+ for (port = 0; port < ds->num_ports; port++) {
struct sja1105_port *sp = &priv->ports[port];
if (!dsa_is_user_port(ds, port))
{
struct sja1105_general_params_entry *general_params;
struct sja1105_mac_config_entry *mac;
+ struct dsa_switch *ds = priv->ds;
struct sja1105_table *table;
bool already_enabled;
u64 new_mirr_port;
mac = priv->static_config.tables[BLK_IDX_MAC_CONFIG].entries;
- already_enabled = (general_params->mirr_port != SJA1105_NUM_PORTS);
+ already_enabled = (general_params->mirr_port != ds->num_ports);
if (already_enabled && enabled && general_params->mirr_port != to) {
dev_err(priv->ds->dev,
"Delete mirroring rules towards port %llu first\n",
int port;
/* Anybody still referencing mirr_port? */
- for (port = 0; port < SJA1105_NUM_PORTS; port++) {
+ for (port = 0; port < ds->num_ports; port++) {
if (mac[port].ing_mirr || mac[port].egr_mirr) {
keep = true;
break;
}
/* Unset already_enabled for next time */
if (!keep)
- new_mirr_port = SJA1105_NUM_PORTS;
+ new_mirr_port = ds->num_ports;
}
if (new_mirr_port != general_params->mirr_port) {
general_params->mirr_port = new_mirr_port;
.port_change_mtu = sja1105_change_mtu,
.port_max_mtu = sja1105_get_max_mtu,
.phylink_validate = sja1105_phylink_validate,
- .phylink_mac_link_state = sja1105_mac_pcs_get_state,
.phylink_mac_config = sja1105_mac_config,
.phylink_mac_link_up = sja1105_mac_link_up,
.phylink_mac_link_down = sja1105_mac_link_down,
struct sja1105_tagger_data *tagger_data;
struct device *dev = &spi->dev;
struct sja1105_private *priv;
+ size_t max_xfer, max_msg;
struct dsa_switch *ds;
int rc, port;
return rc;
}
+ /* In sja1105_xfer, we send spi_messages composed of two spi_transfers:
+ * a small one for the message header and another one for the current
+ * chunk of the packed buffer.
+ * Check that the restrictions imposed by the SPI controller are
+ * respected: the chunk buffer is smaller than the max transfer size,
+ * and the total length of the chunk plus its message header is smaller
+ * than the max message size.
+ * We do that during probe time since the maximum transfer size is a
+ * runtime invariant.
+ */
+ max_xfer = spi_max_transfer_size(spi);
+ max_msg = spi_max_message_size(spi);
+
+ /* We need to send at least one 64-bit word of SPI payload per message
+ * in order to be able to make useful progress.
+ */
+ if (max_msg < SJA1105_SIZE_SPI_MSG_HEADER + 8) {
+ dev_err(dev, "SPI master cannot send large enough buffers, aborting\n");
+ return -EINVAL;
+ }
+
+ priv->max_xfer_len = SJA1105_SIZE_SPI_MSG_MAXLEN;
+ if (priv->max_xfer_len > max_xfer)
+ priv->max_xfer_len = max_xfer;
+ if (priv->max_xfer_len > max_msg - SJA1105_SIZE_SPI_MSG_HEADER)
+ priv->max_xfer_len = max_msg - SJA1105_SIZE_SPI_MSG_HEADER;
+
priv->info = of_device_get_match_data(dev);
/* Detect hardware device */
return -ENOMEM;
ds->dev = dev;
- ds->num_ports = SJA1105_NUM_PORTS;
+ ds->num_ports = priv->info->num_ports;
ds->ops = &sja1105_switch_ops;
ds->priv = priv;
priv->ds = ds;
priv->cbs = devm_kcalloc(dev, priv->info->num_cbs_shapers,
sizeof(struct sja1105_cbs_entry),
GFP_KERNEL);
- if (!priv->cbs)
- return -ENOMEM;
+ if (!priv->cbs) {
+ rc = -ENOMEM;
+ goto out_unregister_switch;
+ }
}
/* Connections between dsa_port and sja1105_port */
- for (port = 0; port < SJA1105_NUM_PORTS; port++) {
+ for (port = 0; port < ds->num_ports; port++) {
struct sja1105_port *sp = &priv->ports[port];
struct dsa_port *dp = dsa_to_port(ds, port);
struct net_device *slave;
dev_err(ds->dev,
"failed to create deferred xmit thread: %d\n",
rc);
- goto out;
+ goto out_destroy_workers;
}
skb_queue_head_init(&sp->xmit_queue);
sp->xmit_tpid = ETH_P_SJA1105;
}
return 0;
-out:
+
+out_destroy_workers:
while (port-- > 0) {
struct sja1105_port *sp = &priv->ports[port];
kthread_destroy_worker(sp->xmit_worker);
}
+
+out_unregister_switch:
+ dsa_unregister_switch(ds);
+
return rc;
}
{ .compatible = "nxp,sja1105q", .data = &sja1105q_info },
{ .compatible = "nxp,sja1105r", .data = &sja1105r_info },
{ .compatible = "nxp,sja1105s", .data = &sja1105s_info },
+ { .compatible = "nxp,sja1110a", .data = &sja1110a_info },
+ { .compatible = "nxp,sja1110b", .data = &sja1110b_info },
+ { .compatible = "nxp,sja1110c", .data = &sja1110c_info },
+ { .compatible = "nxp,sja1110d", .data = &sja1110d_info },
{ /* sentinel */ },
};
MODULE_DEVICE_TABLE(of, sja1105_dt_ids);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright 2021, NXP Semiconductors
+ */
+#include <linux/pcs/pcs-xpcs.h>
+#include <linux/of_mdio.h>
+#include "sja1105.h"
+
+#define SJA1110_PCS_BANK_REG SJA1110_SPI_ADDR(0x3fc)
+
+int sja1105_pcs_mdio_read(struct mii_bus *bus, int phy, int reg)
+{
+ struct sja1105_mdio_private *mdio_priv = bus->priv;
+ struct sja1105_private *priv = mdio_priv->priv;
+ u64 addr;
+ u32 tmp;
+ u16 mmd;
+ int rc;
+
+ if (!(reg & MII_ADDR_C45))
+ return -EINVAL;
+
+ mmd = (reg >> MII_DEVADDR_C45_SHIFT) & 0x1f;
+ addr = (mmd << 16) | (reg & GENMASK(15, 0));
+
+ if (mmd != MDIO_MMD_VEND1 && mmd != MDIO_MMD_VEND2)
+ return 0xffff;
+
+ if (mmd == MDIO_MMD_VEND2 && (reg & GENMASK(15, 0)) == MII_PHYSID1)
+ return NXP_SJA1105_XPCS_ID >> 16;
+ if (mmd == MDIO_MMD_VEND2 && (reg & GENMASK(15, 0)) == MII_PHYSID2)
+ return NXP_SJA1105_XPCS_ID & GENMASK(15, 0);
+
+ rc = sja1105_xfer_u32(priv, SPI_READ, addr, &tmp, NULL);
+ if (rc < 0)
+ return rc;
+
+ return tmp & 0xffff;
+}
+
+int sja1105_pcs_mdio_write(struct mii_bus *bus, int phy, int reg, u16 val)
+{
+ struct sja1105_mdio_private *mdio_priv = bus->priv;
+ struct sja1105_private *priv = mdio_priv->priv;
+ u64 addr;
+ u32 tmp;
+ u16 mmd;
+
+ if (!(reg & MII_ADDR_C45))
+ return -EINVAL;
+
+ mmd = (reg >> MII_DEVADDR_C45_SHIFT) & 0x1f;
+ addr = (mmd << 16) | (reg & GENMASK(15, 0));
+ tmp = val;
+
+ if (mmd != MDIO_MMD_VEND1 && mmd != MDIO_MMD_VEND2)
+ return -EINVAL;
+
+ return sja1105_xfer_u32(priv, SPI_WRITE, addr, &tmp, NULL);
+}
+
+int sja1110_pcs_mdio_read(struct mii_bus *bus, int phy, int reg)
+{
+ struct sja1105_mdio_private *mdio_priv = bus->priv;
+ struct sja1105_private *priv = mdio_priv->priv;
+ const struct sja1105_regs *regs = priv->info->regs;
+ int offset, bank;
+ u64 addr;
+ u32 tmp;
+ u16 mmd;
+ int rc;
+
+ if (!(reg & MII_ADDR_C45))
+ return -EINVAL;
+
+ if (regs->pcs_base[phy] == SJA1105_RSV_ADDR)
+ return -ENODEV;
+
+ mmd = (reg >> MII_DEVADDR_C45_SHIFT) & 0x1f;
+ addr = (mmd << 16) | (reg & GENMASK(15, 0));
+
+ if (mmd == MDIO_MMD_VEND2 && (reg & GENMASK(15, 0)) == MII_PHYSID1)
+ return NXP_SJA1110_XPCS_ID >> 16;
+ if (mmd == MDIO_MMD_VEND2 && (reg & GENMASK(15, 0)) == MII_PHYSID2)
+ return NXP_SJA1110_XPCS_ID & GENMASK(15, 0);
+
+ bank = addr >> 8;
+ offset = addr & GENMASK(7, 0);
+
+ /* This addressing scheme reserves register 0xff for the bank address
+ * register, so that can never be addressed.
+ */
+ if (WARN_ON(offset == 0xff))
+ return -ENODEV;
+
+ tmp = bank;
+
+ rc = sja1105_xfer_u32(priv, SPI_WRITE,
+ regs->pcs_base[phy] + SJA1110_PCS_BANK_REG,
+ &tmp, NULL);
+ if (rc < 0)
+ return rc;
+
+ rc = sja1105_xfer_u32(priv, SPI_READ, regs->pcs_base[phy] + offset,
+ &tmp, NULL);
+ if (rc < 0)
+ return rc;
+
+ return tmp & 0xffff;
+}
+
+int sja1110_pcs_mdio_write(struct mii_bus *bus, int phy, int reg, u16 val)
+{
+ struct sja1105_mdio_private *mdio_priv = bus->priv;
+ struct sja1105_private *priv = mdio_priv->priv;
+ const struct sja1105_regs *regs = priv->info->regs;
+ int offset, bank;
+ u64 addr;
+ u32 tmp;
+ u16 mmd;
+ int rc;
+
+ if (!(reg & MII_ADDR_C45))
+ return -EINVAL;
+
+ if (regs->pcs_base[phy] == SJA1105_RSV_ADDR)
+ return -ENODEV;
+
+ mmd = (reg >> MII_DEVADDR_C45_SHIFT) & 0x1f;
+ addr = (mmd << 16) | (reg & GENMASK(15, 0));
+
+ bank = addr >> 8;
+ offset = addr & GENMASK(7, 0);
+
+ /* This addressing scheme reserves register 0xff for the bank address
+ * register, so that can never be addressed.
+ */
+ if (WARN_ON(offset == 0xff))
+ return -ENODEV;
+
+ tmp = bank;
+
+ rc = sja1105_xfer_u32(priv, SPI_WRITE,
+ regs->pcs_base[phy] + SJA1110_PCS_BANK_REG,
+ &tmp, NULL);
+ if (rc < 0)
+ return rc;
+
+ tmp = val;
+
+ return sja1105_xfer_u32(priv, SPI_WRITE, regs->pcs_base[phy] + offset,
+ &tmp, NULL);
+}
+
+enum sja1105_mdio_opcode {
+ SJA1105_C45_ADDR = 0,
+ SJA1105_C22 = 1,
+ SJA1105_C45_DATA = 2,
+ SJA1105_C45_DATA_AUTOINC = 3,
+};
+
+static u64 sja1105_base_t1_encode_addr(struct sja1105_private *priv,
+ int phy, enum sja1105_mdio_opcode op,
+ int xad)
+{
+ const struct sja1105_regs *regs = priv->info->regs;
+
+ return regs->mdio_100base_t1 | (phy << 7) | (op << 5) | (xad << 0);
+}
+
+static int sja1105_base_t1_mdio_read(struct mii_bus *bus, int phy, int reg)
+{
+ struct sja1105_mdio_private *mdio_priv = bus->priv;
+ struct sja1105_private *priv = mdio_priv->priv;
+ u64 addr;
+ u32 tmp;
+ int rc;
+
+ if (reg & MII_ADDR_C45) {
+ u16 mmd = (reg >> MII_DEVADDR_C45_SHIFT) & 0x1f;
+
+ addr = sja1105_base_t1_encode_addr(priv, phy, SJA1105_C45_ADDR,
+ mmd);
+
+ tmp = reg & MII_REGADDR_C45_MASK;
+
+ rc = sja1105_xfer_u32(priv, SPI_WRITE, addr, &tmp, NULL);
+ if (rc < 0)
+ return rc;
+
+ addr = sja1105_base_t1_encode_addr(priv, phy, SJA1105_C45_DATA,
+ mmd);
+
+ rc = sja1105_xfer_u32(priv, SPI_READ, addr, &tmp, NULL);
+ if (rc < 0)
+ return rc;
+
+ return tmp & 0xffff;
+ }
+
+ /* Clause 22 read */
+ addr = sja1105_base_t1_encode_addr(priv, phy, SJA1105_C22, reg & 0x1f);
+
+ rc = sja1105_xfer_u32(priv, SPI_READ, addr, &tmp, NULL);
+ if (rc < 0)
+ return rc;
+
+ return tmp & 0xffff;
+}
+
+static int sja1105_base_t1_mdio_write(struct mii_bus *bus, int phy, int reg,
+ u16 val)
+{
+ struct sja1105_mdio_private *mdio_priv = bus->priv;
+ struct sja1105_private *priv = mdio_priv->priv;
+ u64 addr;
+ u32 tmp;
+ int rc;
+
+ if (reg & MII_ADDR_C45) {
+ u16 mmd = (reg >> MII_DEVADDR_C45_SHIFT) & 0x1f;
+
+ addr = sja1105_base_t1_encode_addr(priv, phy, SJA1105_C45_ADDR,
+ mmd);
+
+ tmp = reg & MII_REGADDR_C45_MASK;
+
+ rc = sja1105_xfer_u32(priv, SPI_WRITE, addr, &tmp, NULL);
+ if (rc < 0)
+ return rc;
+
+ addr = sja1105_base_t1_encode_addr(priv, phy, SJA1105_C45_DATA,
+ mmd);
+
+ tmp = val & 0xffff;
+
+ rc = sja1105_xfer_u32(priv, SPI_WRITE, addr, &tmp, NULL);
+ if (rc < 0)
+ return rc;
+
+ return 0;
+ }
+
+ /* Clause 22 write */
+ addr = sja1105_base_t1_encode_addr(priv, phy, SJA1105_C22, reg & 0x1f);
+
+ tmp = val & 0xffff;
+
+ return sja1105_xfer_u32(priv, SPI_WRITE, addr, &tmp, NULL);
+}
+
+static int sja1105_base_tx_mdio_read(struct mii_bus *bus, int phy, int reg)
+{
+ struct sja1105_mdio_private *mdio_priv = bus->priv;
+ struct sja1105_private *priv = mdio_priv->priv;
+ const struct sja1105_regs *regs = priv->info->regs;
+ u32 tmp;
+ int rc;
+
+ rc = sja1105_xfer_u32(priv, SPI_READ, regs->mdio_100base_tx + reg,
+ &tmp, NULL);
+ if (rc < 0)
+ return rc;
+
+ return tmp & 0xffff;
+}
+
+static int sja1105_base_tx_mdio_write(struct mii_bus *bus, int phy, int reg,
+ u16 val)
+{
+ struct sja1105_mdio_private *mdio_priv = bus->priv;
+ struct sja1105_private *priv = mdio_priv->priv;
+ const struct sja1105_regs *regs = priv->info->regs;
+ u32 tmp = val;
+
+ return sja1105_xfer_u32(priv, SPI_WRITE, regs->mdio_100base_tx + reg,
+ &tmp, NULL);
+}
+
+static int sja1105_mdiobus_base_tx_register(struct sja1105_private *priv,
+ struct device_node *mdio_node)
+{
+ struct sja1105_mdio_private *mdio_priv;
+ struct device_node *np;
+ struct mii_bus *bus;
+ int rc = 0;
+
+ np = of_find_compatible_node(mdio_node, NULL,
+ "nxp,sja1110-base-tx-mdio");
+ if (!np)
+ return 0;
+
+ if (!of_device_is_available(np))
+ goto out_put_np;
+
+ bus = mdiobus_alloc_size(sizeof(*mdio_priv));
+ if (!bus) {
+ rc = -ENOMEM;
+ goto out_put_np;
+ }
+
+ bus->name = "SJA1110 100base-TX MDIO bus";
+ snprintf(bus->id, MII_BUS_ID_SIZE, "%s-base-tx",
+ dev_name(priv->ds->dev));
+ bus->read = sja1105_base_tx_mdio_read;
+ bus->write = sja1105_base_tx_mdio_write;
+ bus->parent = priv->ds->dev;
+ mdio_priv = bus->priv;
+ mdio_priv->priv = priv;
+
+ rc = of_mdiobus_register(bus, np);
+ if (rc) {
+ mdiobus_free(bus);
+ goto out_put_np;
+ }
+
+ priv->mdio_base_tx = bus;
+
+out_put_np:
+ of_node_put(np);
+
+ return rc;
+}
+
+static void sja1105_mdiobus_base_tx_unregister(struct sja1105_private *priv)
+{
+ if (!priv->mdio_base_tx)
+ return;
+
+ mdiobus_unregister(priv->mdio_base_tx);
+ mdiobus_free(priv->mdio_base_tx);
+ priv->mdio_base_tx = NULL;
+}
+
+static int sja1105_mdiobus_base_t1_register(struct sja1105_private *priv,
+ struct device_node *mdio_node)
+{
+ struct sja1105_mdio_private *mdio_priv;
+ struct device_node *np;
+ struct mii_bus *bus;
+ int rc = 0;
+
+ np = of_find_compatible_node(mdio_node, NULL,
+ "nxp,sja1110-base-t1-mdio");
+ if (!np)
+ return 0;
+
+ if (!of_device_is_available(np))
+ goto out_put_np;
+
+ bus = mdiobus_alloc_size(sizeof(*mdio_priv));
+ if (!bus) {
+ rc = -ENOMEM;
+ goto out_put_np;
+ }
+
+ bus->name = "SJA1110 100base-T1 MDIO bus";
+ snprintf(bus->id, MII_BUS_ID_SIZE, "%s-base-t1",
+ dev_name(priv->ds->dev));
+ bus->read = sja1105_base_t1_mdio_read;
+ bus->write = sja1105_base_t1_mdio_write;
+ bus->parent = priv->ds->dev;
+ mdio_priv = bus->priv;
+ mdio_priv->priv = priv;
+
+ rc = of_mdiobus_register(bus, np);
+ if (rc) {
+ mdiobus_free(bus);
+ goto out_put_np;
+ }
+
+ priv->mdio_base_t1 = bus;
+
+out_put_np:
+ of_node_put(np);
+
+ return rc;
+}
+
+static void sja1105_mdiobus_base_t1_unregister(struct sja1105_private *priv)
+{
+ if (!priv->mdio_base_t1)
+ return;
+
+ mdiobus_unregister(priv->mdio_base_t1);
+ mdiobus_free(priv->mdio_base_t1);
+ priv->mdio_base_t1 = NULL;
+}
+
+static int sja1105_mdiobus_pcs_register(struct sja1105_private *priv)
+{
+ struct sja1105_mdio_private *mdio_priv;
+ struct dsa_switch *ds = priv->ds;
+ struct mii_bus *bus;
+ int rc = 0;
+ int port;
+
+ if (!priv->info->pcs_mdio_read || !priv->info->pcs_mdio_write)
+ return 0;
+
+ bus = mdiobus_alloc_size(sizeof(*mdio_priv));
+ if (!bus)
+ return -ENOMEM;
+
+ bus->name = "SJA1105 PCS MDIO bus";
+ snprintf(bus->id, MII_BUS_ID_SIZE, "%s-pcs",
+ dev_name(ds->dev));
+ bus->read = priv->info->pcs_mdio_read;
+ bus->write = priv->info->pcs_mdio_write;
+ bus->parent = ds->dev;
+ /* There is no PHY on this MDIO bus => mask out all PHY addresses
+ * from auto probing.
+ */
+ bus->phy_mask = ~0;
+ mdio_priv = bus->priv;
+ mdio_priv->priv = priv;
+
+ rc = mdiobus_register(bus);
+ if (rc) {
+ mdiobus_free(bus);
+ return rc;
+ }
+
+ for (port = 0; port < ds->num_ports; port++) {
+ struct mdio_device *mdiodev;
+ struct dw_xpcs *xpcs;
+
+ if (dsa_is_unused_port(ds, port))
+ continue;
+
+ if (priv->phy_mode[port] != PHY_INTERFACE_MODE_SGMII &&
+ priv->phy_mode[port] != PHY_INTERFACE_MODE_2500BASEX)
+ continue;
+
+ mdiodev = mdio_device_create(bus, port);
+ if (IS_ERR(mdiodev)) {
+ rc = PTR_ERR(mdiodev);
+ goto out_pcs_free;
+ }
+
+ xpcs = xpcs_create(mdiodev, priv->phy_mode[port]);
+ if (IS_ERR(xpcs)) {
+ rc = PTR_ERR(xpcs);
+ goto out_pcs_free;
+ }
+
+ priv->xpcs[port] = xpcs;
+ }
+
+ priv->mdio_pcs = bus;
+
+ return 0;
+
+out_pcs_free:
+ for (port = 0; port < ds->num_ports; port++) {
+ if (!priv->xpcs[port])
+ continue;
+
+ mdio_device_free(priv->xpcs[port]->mdiodev);
+ xpcs_destroy(priv->xpcs[port]);
+ priv->xpcs[port] = NULL;
+ }
+
+ mdiobus_unregister(bus);
+ mdiobus_free(bus);
+
+ return rc;
+}
+
+static void sja1105_mdiobus_pcs_unregister(struct sja1105_private *priv)
+{
+ struct dsa_switch *ds = priv->ds;
+ int port;
+
+ if (!priv->mdio_pcs)
+ return;
+
+ for (port = 0; port < ds->num_ports; port++) {
+ if (!priv->xpcs[port])
+ continue;
+
+ mdio_device_free(priv->xpcs[port]->mdiodev);
+ xpcs_destroy(priv->xpcs[port]);
+ priv->xpcs[port] = NULL;
+ }
+
+ mdiobus_unregister(priv->mdio_pcs);
+ mdiobus_free(priv->mdio_pcs);
+ priv->mdio_pcs = NULL;
+}
+
+int sja1105_mdiobus_register(struct dsa_switch *ds)
+{
+ struct sja1105_private *priv = ds->priv;
+ const struct sja1105_regs *regs = priv->info->regs;
+ struct device_node *switch_node = ds->dev->of_node;
+ struct device_node *mdio_node;
+ int rc;
+
+ rc = sja1105_mdiobus_pcs_register(priv);
+ if (rc)
+ return rc;
+
+ mdio_node = of_get_child_by_name(switch_node, "mdios");
+ if (!mdio_node)
+ return 0;
+
+ if (!of_device_is_available(mdio_node))
+ goto out_put_mdio_node;
+
+ if (regs->mdio_100base_tx != SJA1105_RSV_ADDR) {
+ rc = sja1105_mdiobus_base_tx_register(priv, mdio_node);
+ if (rc)
+ goto err_put_mdio_node;
+ }
+
+ if (regs->mdio_100base_t1 != SJA1105_RSV_ADDR) {
+ rc = sja1105_mdiobus_base_t1_register(priv, mdio_node);
+ if (rc)
+ goto err_free_base_tx_mdiobus;
+ }
+
+out_put_mdio_node:
+ of_node_put(mdio_node);
+
+ return 0;
+
+err_free_base_tx_mdiobus:
+ sja1105_mdiobus_base_tx_unregister(priv);
+err_put_mdio_node:
+ of_node_put(mdio_node);
+ sja1105_mdiobus_pcs_unregister(priv);
+
+ return rc;
+}
+
+void sja1105_mdiobus_unregister(struct dsa_switch *ds)
+{
+ struct sja1105_private *priv = ds->priv;
+
+ sja1105_mdiobus_base_t1_unregister(priv);
+ sja1105_mdiobus_base_tx_unregister(priv);
+ sja1105_mdiobus_pcs_unregister(priv);
+}
priv->tagger_data.stampable_skb = NULL;
}
ptp_cancel_worker_sync(ptp_data->clock);
+ skb_queue_purge(&ptp_data->skb_txtstamp_queue);
skb_queue_purge(&ptp_data->skb_rxtstamp_queue);
return sja1105_static_config_reload(priv, SJA1105_RX_HWTSTAMPING);
*shwt = (struct skb_shared_hwtstamps) {0};
- ts = SJA1105_SKB_CB(skb)->meta_tstamp;
+ ts = SJA1105_SKB_CB(skb)->tstamp;
ts = sja1105_tstamp_reconstruct(ds, ticks, ts);
shwt->hwtstamp = ns_to_ktime(sja1105_ticks_to_ns(ts));
return -1;
}
-/* Called from dsa_skb_defer_rx_timestamp */
-bool sja1105_port_rxtstamp(struct dsa_switch *ds, int port,
- struct sk_buff *skb, unsigned int type)
+bool sja1105_rxtstamp(struct dsa_switch *ds, int port, struct sk_buff *skb)
{
struct sja1105_private *priv = ds->priv;
struct sja1105_ptp_data *ptp_data = &priv->ptp_data;
return true;
}
+bool sja1110_rxtstamp(struct dsa_switch *ds, int port, struct sk_buff *skb)
+{
+ struct skb_shared_hwtstamps *shwt = skb_hwtstamps(skb);
+ u64 ts = SJA1105_SKB_CB(skb)->tstamp;
+
+ *shwt = (struct skb_shared_hwtstamps) {0};
+
+ shwt->hwtstamp = ns_to_ktime(sja1105_ticks_to_ns(ts));
+
+ /* Don't defer */
+ return false;
+}
+
+/* Called from dsa_skb_defer_rx_timestamp */
+bool sja1105_port_rxtstamp(struct dsa_switch *ds, int port,
+ struct sk_buff *skb, unsigned int type)
+{
+ struct sja1105_private *priv = ds->priv;
+
+ return priv->info->rxtstamp(ds, port, skb);
+}
+
+void sja1110_process_meta_tstamp(struct dsa_switch *ds, int port, u8 ts_id,
+ enum sja1110_meta_tstamp dir, u64 tstamp)
+{
+ struct sja1105_private *priv = ds->priv;
+ struct sja1105_ptp_data *ptp_data = &priv->ptp_data;
+ struct sk_buff *skb, *skb_tmp, *skb_match = NULL;
+ struct skb_shared_hwtstamps shwt = {0};
+
+ /* We don't care about RX timestamps on the CPU port */
+ if (dir == SJA1110_META_TSTAMP_RX)
+ return;
+
+ spin_lock(&ptp_data->skb_txtstamp_queue.lock);
+
+ skb_queue_walk_safe(&ptp_data->skb_txtstamp_queue, skb, skb_tmp) {
+ if (SJA1105_SKB_CB(skb)->ts_id != ts_id)
+ continue;
+
+ __skb_unlink(skb, &ptp_data->skb_txtstamp_queue);
+ skb_match = skb;
+
+ break;
+ }
+
+ spin_unlock(&ptp_data->skb_txtstamp_queue.lock);
+
+ if (WARN_ON(!skb_match))
+ return;
+
+ shwt.hwtstamp = ns_to_ktime(sja1105_ticks_to_ns(tstamp));
+ skb_complete_tx_timestamp(skb_match, &shwt);
+}
+EXPORT_SYMBOL_GPL(sja1110_process_meta_tstamp);
+
+/* In addition to cloning the skb which is done by the common
+ * sja1105_port_txtstamp, we need to generate a timestamp ID and save the
+ * packet to the TX timestamping queue.
+ */
+void sja1110_txtstamp(struct dsa_switch *ds, int port, struct sk_buff *skb)
+{
+ struct sk_buff *clone = SJA1105_SKB_CB(skb)->clone;
+ struct sja1105_private *priv = ds->priv;
+ struct sja1105_ptp_data *ptp_data = &priv->ptp_data;
+ struct sja1105_port *sp = &priv->ports[port];
+ u8 ts_id;
+
+ skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
+
+ spin_lock(&sp->data->meta_lock);
+
+ ts_id = sp->data->ts_id;
+ /* Deal automatically with 8-bit wraparound */
+ sp->data->ts_id++;
+
+ SJA1105_SKB_CB(clone)->ts_id = ts_id;
+
+ spin_unlock(&sp->data->meta_lock);
+
+ skb_queue_tail(&ptp_data->skb_txtstamp_queue, clone);
+}
+
/* Called from dsa_skb_tx_timestamp. This callback is just to clone
* the skb and have it available in SJA1105_SKB_CB in the .port_deferred_xmit
* callback, where we will timestamp it synchronously.
return;
SJA1105_SKB_CB(skb)->clone = clone;
+
+ if (priv->info->txtstamp)
+ priv->info->txtstamp(ds, port, skb);
}
static int sja1105_ptp_reset(struct dsa_switch *ds)
.n_per_out = 1,
};
+ /* Only used on SJA1105 */
skb_queue_head_init(&ptp_data->skb_rxtstamp_queue);
+ /* Only used on SJA1110 */
+ skb_queue_head_init(&ptp_data->skb_txtstamp_queue);
spin_lock_init(&tagger_data->meta_lock);
ptp_data->clock = ptp_clock_register(&ptp_data->caps, ds->dev);
del_timer_sync(&ptp_data->extts_timer);
ptp_cancel_worker_sync(ptp_data->clock);
+ skb_queue_purge(&ptp_data->skb_txtstamp_queue);
skb_queue_purge(&ptp_data->skb_rxtstamp_queue);
ptp_clock_unregister(ptp_data->clock);
ptp_data->clock = NULL;
struct sja1105_ptp_data {
struct timer_list extts_timer;
+ /* Used only on SJA1105 to reconstruct partial timestamps */
struct sk_buff_head skb_rxtstamp_queue;
+ /* Used on SJA1110 where meta frames are generated only for
+ * 2-step TX timestamps
+ */
+ struct sk_buff_head skb_txtstamp_queue;
struct ptp_clock_info caps;
struct ptp_clock *clock;
struct sja1105_ptp_cmd cmd;
int sja1105_ptp_commit(struct dsa_switch *ds, struct sja1105_ptp_cmd *cmd,
sja1105_spi_rw_mode_t rw);
+bool sja1105_rxtstamp(struct dsa_switch *ds, int port, struct sk_buff *skb);
+bool sja1110_rxtstamp(struct dsa_switch *ds, int port, struct sk_buff *skb);
+void sja1110_txtstamp(struct dsa_switch *ds, int port, struct sk_buff *skb);
+
#else
struct sja1105_ptp_cmd;
#define sja1105_hwtstamp_set NULL
+#define sja1105_rxtstamp NULL
+#define sja1110_rxtstamp NULL
+#define sja1110_txtstamp NULL
+
#endif /* IS_ENABLED(CONFIG_NET_DSA_SJA1105_PTP) */
#endif /* _SJA1105_PTP_H */
+++ /dev/null
-/* SPDX-License-Identifier: BSD-3-Clause */
-/* Copyright 2020, NXP Semiconductors
- */
-#ifndef _SJA1105_SGMII_H
-#define _SJA1105_SGMII_H
-
-#define SJA1105_SGMII_PORT 4
-
-/* DIGITAL_CONTROL_1 (address 1f8000h) */
-#define SJA1105_DC1 0x8000
-#define SJA1105_DC1_VS_RESET BIT(15)
-#define SJA1105_DC1_REMOTE_LOOPBACK BIT(14)
-#define SJA1105_DC1_EN_VSMMD1 BIT(13)
-#define SJA1105_DC1_POWER_SAVE BIT(11)
-#define SJA1105_DC1_CLOCK_STOP_EN BIT(10)
-#define SJA1105_DC1_MAC_AUTO_SW BIT(9)
-#define SJA1105_DC1_INIT BIT(8)
-#define SJA1105_DC1_TX_DISABLE BIT(4)
-#define SJA1105_DC1_AUTONEG_TIMER_OVRR BIT(3)
-#define SJA1105_DC1_BYP_POWERUP BIT(1)
-#define SJA1105_DC1_PHY_MODE_CONTROL BIT(0)
-
-/* DIGITAL_CONTROL_2 register (address 1f80E1h) */
-#define SJA1105_DC2 0x80e1
-#define SJA1105_DC2_TX_POL_INV_DISABLE BIT(4)
-#define SJA1105_DC2_RX_POL_INV BIT(0)
-
-/* DIGITAL_ERROR_CNT register (address 1f80E2h) */
-#define SJA1105_DEC 0x80e2
-#define SJA1105_DEC_ICG_EC_ENA BIT(4)
-#define SJA1105_DEC_CLEAR_ON_READ BIT(0)
-
-/* AUTONEG_CONTROL register (address 1f8001h) */
-#define SJA1105_AC 0x8001
-#define SJA1105_AC_MII_CONTROL BIT(8)
-#define SJA1105_AC_SGMII_LINK BIT(4)
-#define SJA1105_AC_PHY_MODE BIT(3)
-#define SJA1105_AC_AUTONEG_MODE(x) (((x) << 1) & GENMASK(2, 1))
-#define SJA1105_AC_AUTONEG_MODE_SGMII SJA1105_AC_AUTONEG_MODE(2)
-
-/* AUTONEG_INTR_STATUS register (address 1f8002h) */
-#define SJA1105_AIS 0x8002
-#define SJA1105_AIS_LINK_STATUS(x) (!!((x) & BIT(4)))
-#define SJA1105_AIS_SPEED(x) (((x) & GENMASK(3, 2)) >> 2)
-#define SJA1105_AIS_DUPLEX_MODE(x) (!!((x) & BIT(1)))
-#define SJA1105_AIS_COMPLETE(x) (!!((x) & BIT(0)))
-
-/* DEBUG_CONTROL register (address 1f8005h) */
-#define SJA1105_DC 0x8005
-#define SJA1105_DC_SUPPRESS_LOS BIT(4)
-#define SJA1105_DC_RESTART_SYNC BIT(0)
-
-#endif
#include <linux/packing.h>
#include "sja1105.h"
-#define SJA1105_SIZE_RESET_CMD 4
-#define SJA1105_SIZE_SPI_MSG_HEADER 4
-#define SJA1105_SIZE_SPI_MSG_MAXLEN (64 * 4)
-
struct sja1105_chunk {
u8 *buf;
size_t len;
sja1105_pack(buf, &msg->address, 24, 4, size);
}
-#define sja1105_hdr_xfer(xfers, chunk) \
- ((xfers) + 2 * (chunk))
-#define sja1105_chunk_xfer(xfers, chunk) \
- ((xfers) + 2 * (chunk) + 1)
-#define sja1105_hdr_buf(hdr_bufs, chunk) \
- ((hdr_bufs) + (chunk) * SJA1105_SIZE_SPI_MSG_HEADER)
-
/* If @rw is:
* - SPI_WRITE: creates and sends an SPI write message at absolute
* address reg_addr, taking @len bytes from *buf
sja1105_spi_rw_mode_t rw, u64 reg_addr, u8 *buf,
size_t len, struct ptp_system_timestamp *ptp_sts)
{
- struct sja1105_chunk chunk = {
- .len = min_t(size_t, len, SJA1105_SIZE_SPI_MSG_MAXLEN),
- .reg_addr = reg_addr,
- .buf = buf,
- };
+ u8 hdr_buf[SJA1105_SIZE_SPI_MSG_HEADER] = {0};
struct spi_device *spi = priv->spidev;
- struct spi_transfer *xfers;
+ struct spi_transfer xfers[2] = {0};
+ struct spi_transfer *chunk_xfer;
+ struct spi_transfer *hdr_xfer;
+ struct sja1105_chunk chunk;
int num_chunks;
int rc, i = 0;
- u8 *hdr_bufs;
- num_chunks = DIV_ROUND_UP(len, SJA1105_SIZE_SPI_MSG_MAXLEN);
+ num_chunks = DIV_ROUND_UP(len, priv->max_xfer_len);
- /* One transfer for each message header, one for each message
- * payload (chunk).
- */
- xfers = kcalloc(2 * num_chunks, sizeof(struct spi_transfer),
- GFP_KERNEL);
- if (!xfers)
- return -ENOMEM;
+ chunk.reg_addr = reg_addr;
+ chunk.buf = buf;
+ chunk.len = min_t(size_t, len, priv->max_xfer_len);
- /* Packed buffers for the num_chunks SPI message headers,
- * stored as a contiguous array
- */
- hdr_bufs = kcalloc(num_chunks, SJA1105_SIZE_SPI_MSG_HEADER,
- GFP_KERNEL);
- if (!hdr_bufs) {
- kfree(xfers);
- return -ENOMEM;
- }
+ hdr_xfer = &xfers[0];
+ chunk_xfer = &xfers[1];
for (i = 0; i < num_chunks; i++) {
- struct spi_transfer *chunk_xfer = sja1105_chunk_xfer(xfers, i);
- struct spi_transfer *hdr_xfer = sja1105_hdr_xfer(xfers, i);
- u8 *hdr_buf = sja1105_hdr_buf(hdr_bufs, i);
struct spi_transfer *ptp_sts_xfer;
struct sja1105_spi_message msg;
chunk.buf += chunk.len;
chunk.reg_addr += chunk.len / 4;
chunk.len = min_t(size_t, (ptrdiff_t)(buf + len - chunk.buf),
- SJA1105_SIZE_SPI_MSG_MAXLEN);
+ priv->max_xfer_len);
- /* De-assert the chip select after each chunk. */
- if (chunk.len)
- chunk_xfer->cs_change = 1;
+ rc = spi_sync_transfer(spi, xfers, 2);
+ if (rc < 0) {
+ dev_err(&spi->dev, "SPI transfer failed: %d\n", rc);
+ return rc;
+ }
}
- rc = spi_sync_transfer(spi, xfers, 2 * num_chunks);
- if (rc < 0)
- dev_err(&spi->dev, "SPI transfer failed: %d\n", rc);
-
- kfree(hdr_bufs);
- kfree(xfers);
-
- return rc;
+ return 0;
}
int sja1105_xfer_buf(const struct sja1105_private *priv,
{
struct sja1105_private *priv = ds->priv;
const struct sja1105_regs *regs = priv->info->regs;
- u8 packed_buf[SJA1105_SIZE_RESET_CMD] = {0};
- const int size = SJA1105_SIZE_RESET_CMD;
- u64 cold_rst = 1;
+ u32 cold_reset = BIT(3);
- sja1105_pack(packed_buf, &cold_rst, 3, 3, size);
-
- return sja1105_xfer_buf(priv, SPI_WRITE, regs->rgu, packed_buf,
- SJA1105_SIZE_RESET_CMD);
+ /* Cold reset */
+ return sja1105_xfer_u32(priv, SPI_WRITE, regs->rgu, &cold_reset, NULL);
}
static int sja1105pqrs_reset_cmd(struct dsa_switch *ds)
{
struct sja1105_private *priv = ds->priv;
const struct sja1105_regs *regs = priv->info->regs;
- u8 packed_buf[SJA1105_SIZE_RESET_CMD] = {0};
- const int size = SJA1105_SIZE_RESET_CMD;
- u64 cold_rst = 1;
+ u32 cold_reset = BIT(2);
- sja1105_pack(packed_buf, &cold_rst, 2, 2, size);
+ /* Cold reset */
+ return sja1105_xfer_u32(priv, SPI_WRITE, regs->rgu, &cold_reset, NULL);
+}
- return sja1105_xfer_buf(priv, SPI_WRITE, regs->rgu, packed_buf,
- SJA1105_SIZE_RESET_CMD);
+static int sja1110_reset_cmd(struct dsa_switch *ds)
+{
+ struct sja1105_private *priv = ds->priv;
+ const struct sja1105_regs *regs = priv->info->regs;
+ u32 switch_reset = BIT(20);
+
+ /* Only reset the switch core.
+ * A full cold reset would re-enable the BASE_MCSS_CLOCK PLL which
+ * would turn on the microcontroller, potentially letting it execute
+ * code which could interfere with our configuration.
+ */
+ return sja1105_xfer_u32(priv, SPI_WRITE, regs->rgu, &switch_reset, NULL);
}
int sja1105_inhibit_tx(const struct sja1105_private *priv,
char *final_header_ptr;
int crc_len;
- valid = sja1105_static_config_check_valid(config);
+ valid = sja1105_static_config_check_valid(config,
+ priv->info->max_frame_mem);
if (valid != SJA1105_CONFIG_OK) {
dev_err(&priv->spidev->dev,
sja1105_static_config_error_msg[valid]);
int sja1105_static_config_upload(struct sja1105_private *priv)
{
- unsigned long port_bitmap = GENMASK_ULL(SJA1105_NUM_PORTS - 1, 0);
struct sja1105_static_config *config = &priv->static_config;
const struct sja1105_regs *regs = priv->info->regs;
struct device *dev = &priv->spidev->dev;
+ struct dsa_switch *ds = priv->ds;
struct sja1105_status status;
int rc, retries = RETRIES;
u8 *config_buf;
* Tx on all ports and waiting for current packet to drain.
* Otherwise, the PHY will see an unterminated Ethernet packet.
*/
- rc = sja1105_inhibit_tx(priv, port_bitmap, true);
+ rc = sja1105_inhibit_tx(priv, GENMASK_ULL(ds->num_ports - 1, 0), true);
if (rc < 0) {
dev_err(dev, "Failed to inhibit Tx on ports\n");
rc = -ENXIO;
return rc;
}
-static struct sja1105_regs sja1105et_regs = {
+static const struct sja1105_regs sja1105et_regs = {
.device_id = 0x0,
.prod_id = 0x100BC3,
.status = 0x1,
.pad_mii_rx = {0x100801, 0x100803, 0x100805, 0x100807, 0x100809},
.rmii_pll1 = 0x10000A,
.cgu_idiv = {0x10000B, 0x10000C, 0x10000D, 0x10000E, 0x10000F},
- .mac = {0x200, 0x202, 0x204, 0x206, 0x208},
- .mac_hl1 = {0x400, 0x410, 0x420, 0x430, 0x440},
- .mac_hl2 = {0x600, 0x610, 0x620, 0x630, 0x640},
+ .stats[MAC] = {0x200, 0x202, 0x204, 0x206, 0x208},
+ .stats[HL1] = {0x400, 0x410, 0x420, 0x430, 0x440},
+ .stats[HL2] = {0x600, 0x610, 0x620, 0x630, 0x640},
/* UM10944.pdf, Table 78, CGU Register overview */
.mii_tx_clk = {0x100013, 0x10001A, 0x100021, 0x100028, 0x10002F},
.mii_rx_clk = {0x100014, 0x10001B, 0x100022, 0x100029, 0x100030},
.ptpclkval = 0x18, /* Spans 0x18 to 0x19 */
.ptpclkrate = 0x1A,
.ptpclkcorp = 0x1D,
+ .mdio_100base_tx = SJA1105_RSV_ADDR,
+ .mdio_100base_t1 = SJA1105_RSV_ADDR,
};
-static struct sja1105_regs sja1105pqrs_regs = {
+static const struct sja1105_regs sja1105pqrs_regs = {
.device_id = 0x0,
.prod_id = 0x100BC3,
.status = 0x1,
.pad_mii_tx = {0x100800, 0x100802, 0x100804, 0x100806, 0x100808},
.pad_mii_rx = {0x100801, 0x100803, 0x100805, 0x100807, 0x100809},
.pad_mii_id = {0x100810, 0x100811, 0x100812, 0x100813, 0x100814},
- .sgmii = 0x1F0000,
.rmii_pll1 = 0x10000A,
.cgu_idiv = {0x10000B, 0x10000C, 0x10000D, 0x10000E, 0x10000F},
- .mac = {0x200, 0x202, 0x204, 0x206, 0x208},
- .mac_hl1 = {0x400, 0x410, 0x420, 0x430, 0x440},
- .mac_hl2 = {0x600, 0x610, 0x620, 0x630, 0x640},
- .ether_stats = {0x1400, 0x1418, 0x1430, 0x1448, 0x1460},
+ .stats[MAC] = {0x200, 0x202, 0x204, 0x206, 0x208},
+ .stats[HL1] = {0x400, 0x410, 0x420, 0x430, 0x440},
+ .stats[HL2] = {0x600, 0x610, 0x620, 0x630, 0x640},
+ .stats[ETHER] = {0x1400, 0x1418, 0x1430, 0x1448, 0x1460},
/* UM11040.pdf, Table 114 */
.mii_tx_clk = {0x100013, 0x100019, 0x10001F, 0x100025, 0x10002B},
.mii_rx_clk = {0x100014, 0x10001A, 0x100020, 0x100026, 0x10002C},
.rgmii_tx_clk = {0x100016, 0x10001C, 0x100022, 0x100028, 0x10002E},
.rmii_ref_clk = {0x100015, 0x10001B, 0x100021, 0x100027, 0x10002D},
.rmii_ext_tx_clk = {0x100017, 0x10001D, 0x100023, 0x100029, 0x10002F},
- .qlevel = {0x604, 0x614, 0x624, 0x634, 0x644},
.ptpegr_ts = {0xC0, 0xC4, 0xC8, 0xCC, 0xD0},
.ptpschtm = 0x13, /* Spans 0x13 to 0x14 */
.ptppinst = 0x15,
.ptpclkrate = 0x1B,
.ptpclkcorp = 0x1E,
.ptpsyncts = 0x1F,
+ .mdio_100base_tx = SJA1105_RSV_ADDR,
+ .mdio_100base_t1 = SJA1105_RSV_ADDR,
+};
+
+static const struct sja1105_regs sja1110_regs = {
+ .device_id = SJA1110_SPI_ADDR(0x0),
+ .prod_id = SJA1110_ACU_ADDR(0xf00),
+ .status = SJA1110_SPI_ADDR(0x4),
+ .port_control = SJA1110_SPI_ADDR(0x50), /* actually INHIB_TX */
+ .vl_status = 0x10000,
+ .config = 0x020000,
+ .rgu = SJA1110_RGU_ADDR(0x100), /* Reset Control Register 0 */
+ /* Ports 2 and 3 are capable of xMII, but there isn't anything to
+ * configure in the CGU/ACU for them.
+ */
+ .pad_mii_tx = {SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR},
+ .pad_mii_rx = {SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR},
+ .pad_mii_id = {SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1110_ACU_ADDR(0x18), SJA1110_ACU_ADDR(0x28),
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR},
+ .rmii_pll1 = SJA1105_RSV_ADDR,
+ .cgu_idiv = {SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR},
+ .stats[MAC] = {0x200, 0x202, 0x204, 0x206, 0x208, 0x20a,
+ 0x20c, 0x20e, 0x210, 0x212, 0x214},
+ .stats[HL1] = {0x400, 0x410, 0x420, 0x430, 0x440, 0x450,
+ 0x460, 0x470, 0x480, 0x490, 0x4a0},
+ .stats[HL2] = {0x600, 0x610, 0x620, 0x630, 0x640, 0x650,
+ 0x660, 0x670, 0x680, 0x690, 0x6a0},
+ .stats[ETHER] = {0x1400, 0x1418, 0x1430, 0x1448, 0x1460, 0x1478,
+ 0x1490, 0x14a8, 0x14c0, 0x14d8, 0x14f0},
+ .mii_tx_clk = {SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR},
+ .mii_rx_clk = {SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR},
+ .mii_ext_tx_clk = {SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR},
+ .mii_ext_rx_clk = {SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR},
+ .rgmii_tx_clk = {SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR},
+ .rmii_ref_clk = {SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR},
+ .rmii_ext_tx_clk = {SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR},
+ .ptpschtm = SJA1110_SPI_ADDR(0x54),
+ .ptppinst = SJA1110_SPI_ADDR(0x5c),
+ .ptppindur = SJA1110_SPI_ADDR(0x64),
+ .ptp_control = SJA1110_SPI_ADDR(0x68),
+ .ptpclkval = SJA1110_SPI_ADDR(0x6c),
+ .ptpclkrate = SJA1110_SPI_ADDR(0x74),
+ .ptpclkcorp = SJA1110_SPI_ADDR(0x80),
+ .ptpsyncts = SJA1110_SPI_ADDR(0x84),
+ .mdio_100base_tx = 0x1c2400,
+ .mdio_100base_t1 = 0x1c1000,
+ .pcs_base = {SJA1105_RSV_ADDR, 0x1c1400, 0x1c1800, 0x1c1c00, 0x1c2000,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR,
+ SJA1105_RSV_ADDR, SJA1105_RSV_ADDR, SJA1105_RSV_ADDR},
};
const struct sja1105_info sja1105e_info = {
.static_ops = sja1105e_table_ops,
.dyn_ops = sja1105et_dyn_ops,
.qinq_tpid = ETH_P_8021Q,
+ .tag_proto = DSA_TAG_PROTO_SJA1105,
.can_limit_mcast_flood = false,
.ptp_ts_bits = 24,
.ptpegr_ts_bytes = 4,
+ .max_frame_mem = SJA1105_MAX_FRAME_MEMORY,
+ .num_ports = SJA1105_NUM_PORTS,
.num_cbs_shapers = SJA1105ET_MAX_CBS_COUNT,
.reset_cmd = sja1105et_reset_cmd,
.fdb_add_cmd = sja1105et_fdb_add,
.fdb_del_cmd = sja1105et_fdb_del,
.ptp_cmd_packing = sja1105et_ptp_cmd_packing,
+ .rxtstamp = sja1105_rxtstamp,
+ .clocking_setup = sja1105_clocking_setup,
.regs = &sja1105et_regs,
+ .port_speed = {
+ [SJA1105_SPEED_AUTO] = 0,
+ [SJA1105_SPEED_10MBPS] = 3,
+ [SJA1105_SPEED_100MBPS] = 2,
+ [SJA1105_SPEED_1000MBPS] = 1,
+ [SJA1105_SPEED_2500MBPS] = 0, /* Not supported */
+ },
+ .supports_mii = {true, true, true, true, true},
+ .supports_rmii = {true, true, true, true, true},
+ .supports_rgmii = {true, true, true, true, true},
.name = "SJA1105E",
};
.static_ops = sja1105t_table_ops,
.dyn_ops = sja1105et_dyn_ops,
.qinq_tpid = ETH_P_8021Q,
+ .tag_proto = DSA_TAG_PROTO_SJA1105,
.can_limit_mcast_flood = false,
.ptp_ts_bits = 24,
.ptpegr_ts_bytes = 4,
+ .max_frame_mem = SJA1105_MAX_FRAME_MEMORY,
+ .num_ports = SJA1105_NUM_PORTS,
.num_cbs_shapers = SJA1105ET_MAX_CBS_COUNT,
.reset_cmd = sja1105et_reset_cmd,
.fdb_add_cmd = sja1105et_fdb_add,
.fdb_del_cmd = sja1105et_fdb_del,
.ptp_cmd_packing = sja1105et_ptp_cmd_packing,
+ .rxtstamp = sja1105_rxtstamp,
+ .clocking_setup = sja1105_clocking_setup,
.regs = &sja1105et_regs,
+ .port_speed = {
+ [SJA1105_SPEED_AUTO] = 0,
+ [SJA1105_SPEED_10MBPS] = 3,
+ [SJA1105_SPEED_100MBPS] = 2,
+ [SJA1105_SPEED_1000MBPS] = 1,
+ [SJA1105_SPEED_2500MBPS] = 0, /* Not supported */
+ },
+ .supports_mii = {true, true, true, true, true},
+ .supports_rmii = {true, true, true, true, true},
+ .supports_rgmii = {true, true, true, true, true},
.name = "SJA1105T",
};
.static_ops = sja1105p_table_ops,
.dyn_ops = sja1105pqrs_dyn_ops,
.qinq_tpid = ETH_P_8021AD,
+ .tag_proto = DSA_TAG_PROTO_SJA1105,
.can_limit_mcast_flood = true,
.ptp_ts_bits = 32,
.ptpegr_ts_bytes = 8,
+ .max_frame_mem = SJA1105_MAX_FRAME_MEMORY,
+ .num_ports = SJA1105_NUM_PORTS,
.num_cbs_shapers = SJA1105PQRS_MAX_CBS_COUNT,
.setup_rgmii_delay = sja1105pqrs_setup_rgmii_delay,
.reset_cmd = sja1105pqrs_reset_cmd,
.fdb_add_cmd = sja1105pqrs_fdb_add,
.fdb_del_cmd = sja1105pqrs_fdb_del,
.ptp_cmd_packing = sja1105pqrs_ptp_cmd_packing,
+ .rxtstamp = sja1105_rxtstamp,
+ .clocking_setup = sja1105_clocking_setup,
.regs = &sja1105pqrs_regs,
+ .port_speed = {
+ [SJA1105_SPEED_AUTO] = 0,
+ [SJA1105_SPEED_10MBPS] = 3,
+ [SJA1105_SPEED_100MBPS] = 2,
+ [SJA1105_SPEED_1000MBPS] = 1,
+ [SJA1105_SPEED_2500MBPS] = 0, /* Not supported */
+ },
+ .supports_mii = {true, true, true, true, true},
+ .supports_rmii = {true, true, true, true, true},
+ .supports_rgmii = {true, true, true, true, true},
.name = "SJA1105P",
};
.static_ops = sja1105q_table_ops,
.dyn_ops = sja1105pqrs_dyn_ops,
.qinq_tpid = ETH_P_8021AD,
+ .tag_proto = DSA_TAG_PROTO_SJA1105,
.can_limit_mcast_flood = true,
.ptp_ts_bits = 32,
.ptpegr_ts_bytes = 8,
+ .max_frame_mem = SJA1105_MAX_FRAME_MEMORY,
+ .num_ports = SJA1105_NUM_PORTS,
.num_cbs_shapers = SJA1105PQRS_MAX_CBS_COUNT,
.setup_rgmii_delay = sja1105pqrs_setup_rgmii_delay,
.reset_cmd = sja1105pqrs_reset_cmd,
.fdb_add_cmd = sja1105pqrs_fdb_add,
.fdb_del_cmd = sja1105pqrs_fdb_del,
.ptp_cmd_packing = sja1105pqrs_ptp_cmd_packing,
+ .rxtstamp = sja1105_rxtstamp,
+ .clocking_setup = sja1105_clocking_setup,
.regs = &sja1105pqrs_regs,
+ .port_speed = {
+ [SJA1105_SPEED_AUTO] = 0,
+ [SJA1105_SPEED_10MBPS] = 3,
+ [SJA1105_SPEED_100MBPS] = 2,
+ [SJA1105_SPEED_1000MBPS] = 1,
+ [SJA1105_SPEED_2500MBPS] = 0, /* Not supported */
+ },
+ .supports_mii = {true, true, true, true, true},
+ .supports_rmii = {true, true, true, true, true},
+ .supports_rgmii = {true, true, true, true, true},
.name = "SJA1105Q",
};
.static_ops = sja1105r_table_ops,
.dyn_ops = sja1105pqrs_dyn_ops,
.qinq_tpid = ETH_P_8021AD,
+ .tag_proto = DSA_TAG_PROTO_SJA1105,
.can_limit_mcast_flood = true,
.ptp_ts_bits = 32,
.ptpegr_ts_bytes = 8,
+ .max_frame_mem = SJA1105_MAX_FRAME_MEMORY,
+ .num_ports = SJA1105_NUM_PORTS,
.num_cbs_shapers = SJA1105PQRS_MAX_CBS_COUNT,
.setup_rgmii_delay = sja1105pqrs_setup_rgmii_delay,
.reset_cmd = sja1105pqrs_reset_cmd,
.fdb_add_cmd = sja1105pqrs_fdb_add,
.fdb_del_cmd = sja1105pqrs_fdb_del,
.ptp_cmd_packing = sja1105pqrs_ptp_cmd_packing,
+ .rxtstamp = sja1105_rxtstamp,
+ .clocking_setup = sja1105_clocking_setup,
+ .pcs_mdio_read = sja1105_pcs_mdio_read,
+ .pcs_mdio_write = sja1105_pcs_mdio_write,
.regs = &sja1105pqrs_regs,
+ .port_speed = {
+ [SJA1105_SPEED_AUTO] = 0,
+ [SJA1105_SPEED_10MBPS] = 3,
+ [SJA1105_SPEED_100MBPS] = 2,
+ [SJA1105_SPEED_1000MBPS] = 1,
+ [SJA1105_SPEED_2500MBPS] = 0, /* Not supported */
+ },
+ .supports_mii = {true, true, true, true, true},
+ .supports_rmii = {true, true, true, true, true},
+ .supports_rgmii = {true, true, true, true, true},
+ .supports_sgmii = {false, false, false, false, true},
.name = "SJA1105R",
};
.dyn_ops = sja1105pqrs_dyn_ops,
.regs = &sja1105pqrs_regs,
.qinq_tpid = ETH_P_8021AD,
+ .tag_proto = DSA_TAG_PROTO_SJA1105,
.can_limit_mcast_flood = true,
.ptp_ts_bits = 32,
.ptpegr_ts_bytes = 8,
+ .max_frame_mem = SJA1105_MAX_FRAME_MEMORY,
+ .num_ports = SJA1105_NUM_PORTS,
.num_cbs_shapers = SJA1105PQRS_MAX_CBS_COUNT,
.setup_rgmii_delay = sja1105pqrs_setup_rgmii_delay,
.reset_cmd = sja1105pqrs_reset_cmd,
.fdb_add_cmd = sja1105pqrs_fdb_add,
.fdb_del_cmd = sja1105pqrs_fdb_del,
.ptp_cmd_packing = sja1105pqrs_ptp_cmd_packing,
+ .rxtstamp = sja1105_rxtstamp,
+ .clocking_setup = sja1105_clocking_setup,
+ .pcs_mdio_read = sja1105_pcs_mdio_read,
+ .pcs_mdio_write = sja1105_pcs_mdio_write,
+ .port_speed = {
+ [SJA1105_SPEED_AUTO] = 0,
+ [SJA1105_SPEED_10MBPS] = 3,
+ [SJA1105_SPEED_100MBPS] = 2,
+ [SJA1105_SPEED_1000MBPS] = 1,
+ [SJA1105_SPEED_2500MBPS] = 0, /* Not supported */
+ },
+ .supports_mii = {true, true, true, true, true},
+ .supports_rmii = {true, true, true, true, true},
+ .supports_rgmii = {true, true, true, true, true},
+ .supports_sgmii = {false, false, false, false, true},
.name = "SJA1105S",
};
+
+const struct sja1105_info sja1110a_info = {
+ .device_id = SJA1110_DEVICE_ID,
+ .part_no = SJA1110A_PART_NO,
+ .static_ops = sja1110_table_ops,
+ .dyn_ops = sja1110_dyn_ops,
+ .regs = &sja1110_regs,
+ .qinq_tpid = ETH_P_8021AD,
+ .tag_proto = DSA_TAG_PROTO_SJA1110,
+ .can_limit_mcast_flood = true,
+ .multiple_cascade_ports = true,
+ .ptp_ts_bits = 32,
+ .ptpegr_ts_bytes = 8,
+ .max_frame_mem = SJA1110_MAX_FRAME_MEMORY,
+ .num_ports = SJA1110_NUM_PORTS,
+ .num_cbs_shapers = SJA1110_MAX_CBS_COUNT,
+ .setup_rgmii_delay = sja1110_setup_rgmii_delay,
+ .reset_cmd = sja1110_reset_cmd,
+ .fdb_add_cmd = sja1105pqrs_fdb_add,
+ .fdb_del_cmd = sja1105pqrs_fdb_del,
+ .ptp_cmd_packing = sja1105pqrs_ptp_cmd_packing,
+ .rxtstamp = sja1110_rxtstamp,
+ .txtstamp = sja1110_txtstamp,
+ .disable_microcontroller = sja1110_disable_microcontroller,
+ .pcs_mdio_read = sja1110_pcs_mdio_read,
+ .pcs_mdio_write = sja1110_pcs_mdio_write,
+ .port_speed = {
+ [SJA1105_SPEED_AUTO] = 0,
+ [SJA1105_SPEED_10MBPS] = 4,
+ [SJA1105_SPEED_100MBPS] = 3,
+ [SJA1105_SPEED_1000MBPS] = 2,
+ [SJA1105_SPEED_2500MBPS] = 1,
+ },
+ .supports_mii = {true, true, true, true, false,
+ true, true, true, true, true, true},
+ .supports_rmii = {false, false, true, true, false,
+ false, false, false, false, false, false},
+ .supports_rgmii = {false, false, true, true, false,
+ false, false, false, false, false, false},
+ .supports_sgmii = {false, true, true, true, true,
+ false, false, false, false, false, false},
+ .supports_2500basex = {false, false, false, true, true,
+ false, false, false, false, false, false},
+ .internal_phy = {SJA1105_NO_PHY, SJA1105_PHY_BASE_TX,
+ SJA1105_NO_PHY, SJA1105_NO_PHY,
+ SJA1105_NO_PHY, SJA1105_PHY_BASE_T1,
+ SJA1105_PHY_BASE_T1, SJA1105_PHY_BASE_T1,
+ SJA1105_PHY_BASE_T1, SJA1105_PHY_BASE_T1,
+ SJA1105_PHY_BASE_T1},
+ .name = "SJA1110A",
+};
+
+const struct sja1105_info sja1110b_info = {
+ .device_id = SJA1110_DEVICE_ID,
+ .part_no = SJA1110B_PART_NO,
+ .static_ops = sja1110_table_ops,
+ .dyn_ops = sja1110_dyn_ops,
+ .regs = &sja1110_regs,
+ .qinq_tpid = ETH_P_8021AD,
+ .tag_proto = DSA_TAG_PROTO_SJA1110,
+ .can_limit_mcast_flood = true,
+ .multiple_cascade_ports = true,
+ .ptp_ts_bits = 32,
+ .ptpegr_ts_bytes = 8,
+ .max_frame_mem = SJA1110_MAX_FRAME_MEMORY,
+ .num_ports = SJA1110_NUM_PORTS,
+ .num_cbs_shapers = SJA1110_MAX_CBS_COUNT,
+ .setup_rgmii_delay = sja1110_setup_rgmii_delay,
+ .reset_cmd = sja1110_reset_cmd,
+ .fdb_add_cmd = sja1105pqrs_fdb_add,
+ .fdb_del_cmd = sja1105pqrs_fdb_del,
+ .ptp_cmd_packing = sja1105pqrs_ptp_cmd_packing,
+ .rxtstamp = sja1110_rxtstamp,
+ .txtstamp = sja1110_txtstamp,
+ .disable_microcontroller = sja1110_disable_microcontroller,
+ .pcs_mdio_read = sja1110_pcs_mdio_read,
+ .pcs_mdio_write = sja1110_pcs_mdio_write,
+ .port_speed = {
+ [SJA1105_SPEED_AUTO] = 0,
+ [SJA1105_SPEED_10MBPS] = 4,
+ [SJA1105_SPEED_100MBPS] = 3,
+ [SJA1105_SPEED_1000MBPS] = 2,
+ [SJA1105_SPEED_2500MBPS] = 1,
+ },
+ .supports_mii = {true, true, true, true, false,
+ true, true, true, true, true, false},
+ .supports_rmii = {false, false, true, true, false,
+ false, false, false, false, false, false},
+ .supports_rgmii = {false, false, true, true, false,
+ false, false, false, false, false, false},
+ .supports_sgmii = {false, false, false, true, true,
+ false, false, false, false, false, false},
+ .supports_2500basex = {false, false, false, true, true,
+ false, false, false, false, false, false},
+ .internal_phy = {SJA1105_NO_PHY, SJA1105_PHY_BASE_TX,
+ SJA1105_NO_PHY, SJA1105_NO_PHY,
+ SJA1105_NO_PHY, SJA1105_PHY_BASE_T1,
+ SJA1105_PHY_BASE_T1, SJA1105_PHY_BASE_T1,
+ SJA1105_PHY_BASE_T1, SJA1105_PHY_BASE_T1,
+ SJA1105_NO_PHY},
+ .name = "SJA1110B",
+};
+
+const struct sja1105_info sja1110c_info = {
+ .device_id = SJA1110_DEVICE_ID,
+ .part_no = SJA1110C_PART_NO,
+ .static_ops = sja1110_table_ops,
+ .dyn_ops = sja1110_dyn_ops,
+ .regs = &sja1110_regs,
+ .qinq_tpid = ETH_P_8021AD,
+ .tag_proto = DSA_TAG_PROTO_SJA1110,
+ .can_limit_mcast_flood = true,
+ .multiple_cascade_ports = true,
+ .ptp_ts_bits = 32,
+ .ptpegr_ts_bytes = 8,
+ .max_frame_mem = SJA1110_MAX_FRAME_MEMORY,
+ .num_ports = SJA1110_NUM_PORTS,
+ .num_cbs_shapers = SJA1110_MAX_CBS_COUNT,
+ .setup_rgmii_delay = sja1110_setup_rgmii_delay,
+ .reset_cmd = sja1110_reset_cmd,
+ .fdb_add_cmd = sja1105pqrs_fdb_add,
+ .fdb_del_cmd = sja1105pqrs_fdb_del,
+ .ptp_cmd_packing = sja1105pqrs_ptp_cmd_packing,
+ .rxtstamp = sja1110_rxtstamp,
+ .txtstamp = sja1110_txtstamp,
+ .disable_microcontroller = sja1110_disable_microcontroller,
+ .pcs_mdio_read = sja1110_pcs_mdio_read,
+ .pcs_mdio_write = sja1110_pcs_mdio_write,
+ .port_speed = {
+ [SJA1105_SPEED_AUTO] = 0,
+ [SJA1105_SPEED_10MBPS] = 4,
+ [SJA1105_SPEED_100MBPS] = 3,
+ [SJA1105_SPEED_1000MBPS] = 2,
+ [SJA1105_SPEED_2500MBPS] = 1,
+ },
+ .supports_mii = {true, true, true, true, false,
+ true, true, true, false, false, false},
+ .supports_rmii = {false, false, true, true, false,
+ false, false, false, false, false, false},
+ .supports_rgmii = {false, false, true, true, false,
+ false, false, false, false, false, false},
+ .supports_sgmii = {false, false, false, false, true,
+ false, false, false, false, false, false},
+ .supports_2500basex = {false, false, false, false, true,
+ false, false, false, false, false, false},
+ .internal_phy = {SJA1105_NO_PHY, SJA1105_PHY_BASE_TX,
+ SJA1105_NO_PHY, SJA1105_NO_PHY,
+ SJA1105_NO_PHY, SJA1105_PHY_BASE_T1,
+ SJA1105_PHY_BASE_T1, SJA1105_PHY_BASE_T1,
+ SJA1105_NO_PHY, SJA1105_NO_PHY,
+ SJA1105_NO_PHY},
+ .name = "SJA1110C",
+};
+
+const struct sja1105_info sja1110d_info = {
+ .device_id = SJA1110_DEVICE_ID,
+ .part_no = SJA1110D_PART_NO,
+ .static_ops = sja1110_table_ops,
+ .dyn_ops = sja1110_dyn_ops,
+ .regs = &sja1110_regs,
+ .qinq_tpid = ETH_P_8021AD,
+ .tag_proto = DSA_TAG_PROTO_SJA1110,
+ .can_limit_mcast_flood = true,
+ .multiple_cascade_ports = true,
+ .ptp_ts_bits = 32,
+ .ptpegr_ts_bytes = 8,
+ .max_frame_mem = SJA1110_MAX_FRAME_MEMORY,
+ .num_ports = SJA1110_NUM_PORTS,
+ .num_cbs_shapers = SJA1110_MAX_CBS_COUNT,
+ .setup_rgmii_delay = sja1110_setup_rgmii_delay,
+ .reset_cmd = sja1110_reset_cmd,
+ .fdb_add_cmd = sja1105pqrs_fdb_add,
+ .fdb_del_cmd = sja1105pqrs_fdb_del,
+ .ptp_cmd_packing = sja1105pqrs_ptp_cmd_packing,
+ .rxtstamp = sja1110_rxtstamp,
+ .txtstamp = sja1110_txtstamp,
+ .disable_microcontroller = sja1110_disable_microcontroller,
+ .pcs_mdio_read = sja1110_pcs_mdio_read,
+ .pcs_mdio_write = sja1110_pcs_mdio_write,
+ .port_speed = {
+ [SJA1105_SPEED_AUTO] = 0,
+ [SJA1105_SPEED_10MBPS] = 4,
+ [SJA1105_SPEED_100MBPS] = 3,
+ [SJA1105_SPEED_1000MBPS] = 2,
+ [SJA1105_SPEED_2500MBPS] = 1,
+ },
+ .supports_mii = {true, false, true, false, false,
+ true, true, true, false, false, false},
+ .supports_rmii = {false, false, true, false, false,
+ false, false, false, false, false, false},
+ .supports_rgmii = {false, false, true, false, false,
+ false, false, false, false, false, false},
+ .supports_sgmii = {false, true, true, true, true,
+ false, false, false, false, false, false},
+ .supports_2500basex = {false, false, false, true, true,
+ false, false, false, false, false, false},
+ .internal_phy = {SJA1105_NO_PHY, SJA1105_NO_PHY,
+ SJA1105_NO_PHY, SJA1105_NO_PHY,
+ SJA1105_NO_PHY, SJA1105_PHY_BASE_T1,
+ SJA1105_PHY_BASE_T1, SJA1105_PHY_BASE_T1,
+ SJA1105_NO_PHY, SJA1105_NO_PHY,
+ SJA1105_NO_PHY},
+ .name = "SJA1110D",
+};
return size;
}
+size_t sja1110_general_params_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1105_general_params_entry *entry = entry_ptr;
+ const size_t size = SJA1110_SIZE_GENERAL_PARAMS_ENTRY;
+
+ sja1105_packing(buf, &entry->vllupformat, 447, 447, size, op);
+ sja1105_packing(buf, &entry->mirr_ptacu, 446, 446, size, op);
+ sja1105_packing(buf, &entry->switchid, 445, 442, size, op);
+ sja1105_packing(buf, &entry->hostprio, 441, 439, size, op);
+ sja1105_packing(buf, &entry->mac_fltres1, 438, 391, size, op);
+ sja1105_packing(buf, &entry->mac_fltres0, 390, 343, size, op);
+ sja1105_packing(buf, &entry->mac_flt1, 342, 295, size, op);
+ sja1105_packing(buf, &entry->mac_flt0, 294, 247, size, op);
+ sja1105_packing(buf, &entry->incl_srcpt1, 246, 246, size, op);
+ sja1105_packing(buf, &entry->incl_srcpt0, 245, 245, size, op);
+ sja1105_packing(buf, &entry->send_meta1, 244, 244, size, op);
+ sja1105_packing(buf, &entry->send_meta0, 243, 243, size, op);
+ sja1105_packing(buf, &entry->casc_port, 242, 232, size, op);
+ sja1105_packing(buf, &entry->host_port, 231, 228, size, op);
+ sja1105_packing(buf, &entry->mirr_port, 227, 224, size, op);
+ sja1105_packing(buf, &entry->vlmarker, 223, 192, size, op);
+ sja1105_packing(buf, &entry->vlmask, 191, 160, size, op);
+ sja1105_packing(buf, &entry->tpid2, 159, 144, size, op);
+ sja1105_packing(buf, &entry->ignore2stf, 143, 143, size, op);
+ sja1105_packing(buf, &entry->tpid, 142, 127, size, op);
+ sja1105_packing(buf, &entry->queue_ts, 126, 126, size, op);
+ sja1105_packing(buf, &entry->egrmirrvid, 125, 114, size, op);
+ sja1105_packing(buf, &entry->egrmirrpcp, 113, 111, size, op);
+ sja1105_packing(buf, &entry->egrmirrdei, 110, 110, size, op);
+ sja1105_packing(buf, &entry->replay_port, 109, 106, size, op);
+ sja1105_packing(buf, &entry->tdmaconfigidx, 70, 67, size, op);
+ sja1105_packing(buf, &entry->header_type, 64, 49, size, op);
+ sja1105_packing(buf, &entry->tte_en, 16, 16, size, op);
+ return size;
+}
+
static size_t
sja1105_l2_forwarding_params_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
return size;
}
+size_t sja1110_l2_forwarding_params_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1105_l2_forwarding_params_entry *entry = entry_ptr;
+ const size_t size = SJA1105_SIZE_L2_FORWARDING_PARAMS_ENTRY;
+ int offset, i;
+
+ sja1105_packing(buf, &entry->max_dynp, 95, 93, size, op);
+ for (i = 0, offset = 5; i < 8; i++, offset += 11)
+ sja1105_packing(buf, &entry->part_spc[i],
+ offset + 10, offset + 0, size, op);
+ return size;
+}
+
size_t sja1105_l2_forwarding_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
return size;
}
+size_t sja1110_l2_forwarding_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1105_l2_forwarding_entry *entry = entry_ptr;
+ const size_t size = SJA1105_SIZE_L2_FORWARDING_ENTRY;
+ int offset, i;
+
+ if (entry->type_egrpcp2outputq) {
+ for (i = 0, offset = 31; i < SJA1110_NUM_PORTS;
+ i++, offset += 3) {
+ sja1105_packing(buf, &entry->vlan_pmap[i],
+ offset + 2, offset + 0, size, op);
+ }
+ } else {
+ sja1105_packing(buf, &entry->bc_domain, 63, 53, size, op);
+ sja1105_packing(buf, &entry->reach_port, 52, 42, size, op);
+ sja1105_packing(buf, &entry->fl_domain, 41, 31, size, op);
+ }
+ return size;
+}
+
static size_t
sja1105et_l2_lookup_params_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
return size;
}
+size_t sja1110_l2_lookup_params_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1105_l2_lookup_params_entry *entry = entry_ptr;
+ const size_t size = SJA1110_SIZE_L2_LOOKUP_PARAMS_ENTRY;
+ int offset, i;
+
+ for (i = 0, offset = 70; i < SJA1110_NUM_PORTS; i++, offset += 11)
+ sja1105_packing(buf, &entry->maxaddrp[i],
+ offset + 10, offset + 0, size, op);
+ sja1105_packing(buf, &entry->maxage, 69, 55, size, op);
+ sja1105_packing(buf, &entry->start_dynspc, 54, 45, size, op);
+ sja1105_packing(buf, &entry->drpnolearn, 44, 34, size, op);
+ sja1105_packing(buf, &entry->shared_learn, 33, 33, size, op);
+ sja1105_packing(buf, &entry->no_enf_hostprt, 32, 32, size, op);
+ sja1105_packing(buf, &entry->no_mgmt_learn, 31, 31, size, op);
+ sja1105_packing(buf, &entry->use_static, 30, 30, size, op);
+ sja1105_packing(buf, &entry->owr_dyn, 29, 29, size, op);
+ sja1105_packing(buf, &entry->learn_once, 28, 28, size, op);
+ return size;
+}
+
size_t sja1105et_l2_lookup_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
return size;
}
+size_t sja1110_l2_lookup_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ const size_t size = SJA1110_SIZE_L2_LOOKUP_ENTRY;
+ struct sja1105_l2_lookup_entry *entry = entry_ptr;
+
+ if (entry->lockeds) {
+ sja1105_packing(buf, &entry->trap, 168, 168, size, op);
+ sja1105_packing(buf, &entry->mirrvlan, 167, 156, size, op);
+ sja1105_packing(buf, &entry->takets, 155, 155, size, op);
+ sja1105_packing(buf, &entry->mirr, 154, 154, size, op);
+ sja1105_packing(buf, &entry->retag, 153, 153, size, op);
+ } else {
+ sja1105_packing(buf, &entry->touched, 168, 168, size, op);
+ sja1105_packing(buf, &entry->age, 167, 153, size, op);
+ }
+ sja1105_packing(buf, &entry->mask_iotag, 152, 152, size, op);
+ sja1105_packing(buf, &entry->mask_vlanid, 151, 140, size, op);
+ sja1105_packing(buf, &entry->mask_macaddr, 139, 92, size, op);
+ sja1105_packing(buf, &entry->mask_srcport, 91, 88, size, op);
+ sja1105_packing(buf, &entry->iotag, 87, 87, size, op);
+ sja1105_packing(buf, &entry->vlanid, 86, 75, size, op);
+ sja1105_packing(buf, &entry->macaddr, 74, 27, size, op);
+ sja1105_packing(buf, &entry->srcport, 26, 23, size, op);
+ sja1105_packing(buf, &entry->destports, 22, 12, size, op);
+ sja1105_packing(buf, &entry->enfport, 11, 11, size, op);
+ sja1105_packing(buf, &entry->index, 10, 1, size, op);
+ return size;
+}
+
static size_t sja1105_l2_policing_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
return size;
}
+size_t sja1110_l2_policing_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1105_l2_policing_entry *entry = entry_ptr;
+ const size_t size = SJA1105_SIZE_L2_POLICING_ENTRY;
+
+ sja1105_packing(buf, &entry->sharindx, 63, 57, size, op);
+ sja1105_packing(buf, &entry->smax, 56, 39, size, op);
+ sja1105_packing(buf, &entry->rate, 38, 21, size, op);
+ sja1105_packing(buf, &entry->maxlen, 20, 10, size, op);
+ sja1105_packing(buf, &entry->partition, 9, 7, size, op);
+ return size;
+}
+
static size_t sja1105et_mac_config_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
return size;
}
+size_t sja1110_mac_config_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ const size_t size = SJA1105PQRS_SIZE_MAC_CONFIG_ENTRY;
+ struct sja1105_mac_config_entry *entry = entry_ptr;
+ int offset, i;
+
+ for (i = 0, offset = 104; i < 8; i++, offset += 19) {
+ sja1105_packing(buf, &entry->enabled[i],
+ offset + 0, offset + 0, size, op);
+ sja1105_packing(buf, &entry->base[i],
+ offset + 9, offset + 1, size, op);
+ sja1105_packing(buf, &entry->top[i],
+ offset + 18, offset + 10, size, op);
+ }
+ sja1105_packing(buf, &entry->speed, 98, 96, size, op);
+ sja1105_packing(buf, &entry->tp_delin, 95, 80, size, op);
+ sja1105_packing(buf, &entry->tp_delout, 79, 64, size, op);
+ sja1105_packing(buf, &entry->maxage, 63, 56, size, op);
+ sja1105_packing(buf, &entry->vlanprio, 55, 53, size, op);
+ sja1105_packing(buf, &entry->vlanid, 52, 41, size, op);
+ sja1105_packing(buf, &entry->ing_mirr, 40, 40, size, op);
+ sja1105_packing(buf, &entry->egr_mirr, 39, 39, size, op);
+ sja1105_packing(buf, &entry->drpnona664, 38, 38, size, op);
+ sja1105_packing(buf, &entry->drpdtag, 37, 37, size, op);
+ sja1105_packing(buf, &entry->drpuntag, 34, 34, size, op);
+ sja1105_packing(buf, &entry->retag, 33, 33, size, op);
+ sja1105_packing(buf, &entry->dyn_learn, 32, 32, size, op);
+ sja1105_packing(buf, &entry->egress, 31, 31, size, op);
+ sja1105_packing(buf, &entry->ingress, 30, 30, size, op);
+ sja1105_packing(buf, &entry->ifg, 10, 5, size, op);
+ return size;
+}
+
static size_t
sja1105_schedule_entry_points_params_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
return size;
}
+static size_t
+sja1110_schedule_entry_points_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1105_schedule_entry_points_entry *entry = entry_ptr;
+ const size_t size = SJA1110_SIZE_SCHEDULE_ENTRY_POINTS_ENTRY;
+
+ sja1105_packing(buf, &entry->subschindx, 63, 61, size, op);
+ sja1105_packing(buf, &entry->delta, 60, 43, size, op);
+ sja1105_packing(buf, &entry->address, 42, 31, size, op);
+ return size;
+}
+
static size_t sja1105_schedule_params_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
return size;
}
+static size_t sja1110_schedule_params_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1105_schedule_params_entry *entry = entry_ptr;
+ const size_t size = SJA1105_SIZE_SCHEDULE_PARAMS_ENTRY;
+ int offset, i;
+
+ for (i = 0, offset = 0; i < 8; i++, offset += 12)
+ sja1105_packing(buf, &entry->subscheind[i],
+ offset + 11, offset + 0, size, op);
+ return size;
+}
+
static size_t sja1105_schedule_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
return size;
}
+static size_t sja1110_schedule_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ const size_t size = SJA1110_SIZE_SCHEDULE_ENTRY;
+ struct sja1105_schedule_entry *entry = entry_ptr;
+
+ sja1105_packing(buf, &entry->winstindex, 95, 84, size, op);
+ sja1105_packing(buf, &entry->winend, 83, 83, size, op);
+ sja1105_packing(buf, &entry->winst, 82, 82, size, op);
+ sja1105_packing(buf, &entry->destports, 81, 71, size, op);
+ sja1105_packing(buf, &entry->setvalid, 70, 70, size, op);
+ sja1105_packing(buf, &entry->txen, 69, 69, size, op);
+ sja1105_packing(buf, &entry->resmedia_en, 68, 68, size, op);
+ sja1105_packing(buf, &entry->resmedia, 67, 60, size, op);
+ sja1105_packing(buf, &entry->vlindex, 59, 48, size, op);
+ sja1105_packing(buf, &entry->delta, 47, 30, size, op);
+ return size;
+}
+
static size_t
sja1105_vl_forwarding_params_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
return size;
}
+static size_t
+sja1110_vl_forwarding_params_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1105_vl_forwarding_params_entry *entry = entry_ptr;
+ const size_t size = SJA1105_SIZE_VL_FORWARDING_PARAMS_ENTRY;
+ int offset, i;
+
+ for (i = 0, offset = 8; i < 8; i++, offset += 11)
+ sja1105_packing(buf, &entry->partspc[i],
+ offset + 10, offset + 0, size, op);
+ sja1105_packing(buf, &entry->debugen, 7, 7, size, op);
+ return size;
+}
+
static size_t sja1105_vl_forwarding_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
return size;
}
+static size_t sja1110_vl_forwarding_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1105_vl_forwarding_entry *entry = entry_ptr;
+ const size_t size = SJA1105_SIZE_VL_FORWARDING_ENTRY;
+
+ sja1105_packing(buf, &entry->type, 31, 31, size, op);
+ sja1105_packing(buf, &entry->priority, 30, 28, size, op);
+ sja1105_packing(buf, &entry->partition, 27, 25, size, op);
+ sja1105_packing(buf, &entry->destports, 24, 14, size, op);
+ return size;
+}
+
size_t sja1105_vl_lookup_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
return size;
}
+size_t sja1110_vl_lookup_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1105_vl_lookup_entry *entry = entry_ptr;
+ const size_t size = SJA1105_SIZE_VL_LOOKUP_ENTRY;
+
+ if (entry->format == SJA1105_VL_FORMAT_PSFP) {
+ /* Interpreting vllupformat as 0 */
+ sja1105_packing(buf, &entry->destports,
+ 94, 84, size, op);
+ sja1105_packing(buf, &entry->iscritical,
+ 83, 83, size, op);
+ sja1105_packing(buf, &entry->macaddr,
+ 82, 35, size, op);
+ sja1105_packing(buf, &entry->vlanid,
+ 34, 23, size, op);
+ sja1105_packing(buf, &entry->port,
+ 22, 19, size, op);
+ sja1105_packing(buf, &entry->vlanprior,
+ 18, 16, size, op);
+ } else {
+ /* Interpreting vllupformat as 1 */
+ sja1105_packing(buf, &entry->egrmirr,
+ 94, 84, size, op);
+ sja1105_packing(buf, &entry->ingrmirr,
+ 83, 83, size, op);
+ sja1105_packing(buf, &entry->vlid,
+ 50, 35, size, op);
+ sja1105_packing(buf, &entry->port,
+ 22, 19, size, op);
+ }
+ return size;
+}
+
static size_t sja1105_vl_policing_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
return size;
}
+size_t sja1110_vl_policing_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1105_vl_policing_entry *entry = entry_ptr;
+ const size_t size = SJA1105_SIZE_VL_POLICING_ENTRY;
+
+ sja1105_packing(buf, &entry->type, 63, 63, size, op);
+ sja1105_packing(buf, &entry->maxlen, 62, 52, size, op);
+ sja1105_packing(buf, &entry->sharindx, 51, 40, size, op);
+ if (entry->type == 0) {
+ sja1105_packing(buf, &entry->bag, 41, 28, size, op);
+ sja1105_packing(buf, &entry->jitter, 27, 18, size, op);
+ }
+ return size;
+}
+
size_t sja1105_vlan_lookup_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
return size;
}
+size_t sja1110_vlan_lookup_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1105_vlan_lookup_entry *entry = entry_ptr;
+ const size_t size = SJA1110_SIZE_VLAN_LOOKUP_ENTRY;
+
+ sja1105_packing(buf, &entry->ving_mirr, 95, 85, size, op);
+ sja1105_packing(buf, &entry->vegr_mirr, 84, 74, size, op);
+ sja1105_packing(buf, &entry->vmemb_port, 73, 63, size, op);
+ sja1105_packing(buf, &entry->vlan_bc, 62, 52, size, op);
+ sja1105_packing(buf, &entry->tag_port, 51, 41, size, op);
+ sja1105_packing(buf, &entry->type_entry, 40, 39, size, op);
+ sja1105_packing(buf, &entry->vlanid, 38, 27, size, op);
+ return size;
+}
+
static size_t sja1105_xmii_params_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
return size;
}
+size_t sja1110_xmii_params_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ const size_t size = SJA1110_SIZE_XMII_PARAMS_ENTRY;
+ struct sja1105_xmii_params_entry *entry = entry_ptr;
+ int offset, i;
+
+ for (i = 0, offset = 20; i < SJA1110_NUM_PORTS; i++, offset += 4) {
+ sja1105_packing(buf, &entry->xmii_mode[i],
+ offset + 1, offset + 0, size, op);
+ sja1105_packing(buf, &entry->phy_mac[i],
+ offset + 2, offset + 2, size, op);
+ sja1105_packing(buf, &entry->special[i],
+ offset + 3, offset + 3, size, op);
+ }
+ return size;
+}
+
size_t sja1105_retagging_entry_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
return size;
}
+size_t sja1110_retagging_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1105_retagging_entry *entry = entry_ptr;
+ const size_t size = SJA1105_SIZE_RETAGGING_ENTRY;
+
+ sja1105_packing(buf, &entry->egr_port, 63, 53, size, op);
+ sja1105_packing(buf, &entry->ing_port, 52, 42, size, op);
+ sja1105_packing(buf, &entry->vlan_ing, 41, 30, size, op);
+ sja1105_packing(buf, &entry->vlan_egr, 29, 18, size, op);
+ sja1105_packing(buf, &entry->do_not_learn, 17, 17, size, op);
+ sja1105_packing(buf, &entry->use_dest_ports, 16, 16, size, op);
+ sja1105_packing(buf, &entry->destports, 15, 5, size, op);
+ return size;
+}
+
+static size_t sja1110_pcp_remapping_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op)
+{
+ struct sja1110_pcp_remapping_entry *entry = entry_ptr;
+ const size_t size = SJA1110_SIZE_PCP_REMAPPING_ENTRY;
+ int offset, i;
+
+ for (i = 0, offset = 8; i < SJA1105_NUM_TC; i++, offset += 3)
+ sja1105_packing(buf, &entry->egrpcp[i],
+ offset + 2, offset + 0, size, op);
+
+ return size;
+}
+
size_t sja1105_table_header_packing(void *buf, void *entry_ptr,
enum packing_op op)
{
[BLK_IDX_GENERAL_PARAMS] = BLKID_GENERAL_PARAMS,
[BLK_IDX_RETAGGING] = BLKID_RETAGGING,
[BLK_IDX_XMII_PARAMS] = BLKID_XMII_PARAMS,
+ [BLK_IDX_PCP_REMAPPING] = BLKID_PCP_REMAPPING,
};
const char *sja1105_static_config_error_msg[] = {
};
static sja1105_config_valid_t
-static_config_check_memory_size(const struct sja1105_table *tables)
+static_config_check_memory_size(const struct sja1105_table *tables, int max_mem)
{
const struct sja1105_l2_forwarding_params_entry *l2_fwd_params;
const struct sja1105_vl_forwarding_params_entry *vl_fwd_params;
- int i, max_mem, mem = 0;
+ int i, mem = 0;
l2_fwd_params = tables[BLK_IDX_L2_FORWARDING_PARAMS].entries;
}
if (tables[BLK_IDX_RETAGGING].entry_count)
- max_mem = SJA1105_MAX_FRAME_MEMORY_RETAGGING;
- else
- max_mem = SJA1105_MAX_FRAME_MEMORY;
+ max_mem -= SJA1105_FRAME_MEMORY_RETAGGING_OVERHEAD;
if (mem > max_mem)
return SJA1105_OVERCOMMITTED_FRAME_MEMORY;
}
sja1105_config_valid_t
-sja1105_static_config_check_valid(const struct sja1105_static_config *config)
+sja1105_static_config_check_valid(const struct sja1105_static_config *config,
+ int max_mem)
{
const struct sja1105_table *tables = config->tables;
#define IS_FULL(blk_idx) \
(tables[blk_idx].entry_count == tables[blk_idx].ops->max_entry_count)
if (tables[BLK_IDX_SCHEDULE].entry_count) {
- if (config->device_id != SJA1105T_DEVICE_ID &&
- config->device_id != SJA1105QS_DEVICE_ID)
+ if (!tables[BLK_IDX_SCHEDULE].ops->max_entry_count)
return SJA1105_TTETHERNET_NOT_SUPPORTED;
if (tables[BLK_IDX_SCHEDULE_ENTRY_POINTS].entry_count == 0)
if (!IS_FULL(BLK_IDX_XMII_PARAMS))
return SJA1105_MISSING_XMII_TABLE;
- return static_config_check_memory_size(tables);
+ return static_config_check_memory_size(tables, max_mem);
#undef IS_FULL
}
},
};
+/* SJA1110A: Third generation */
+const struct sja1105_table_ops sja1110_table_ops[BLK_IDX_MAX] = {
+ [BLK_IDX_SCHEDULE] = {
+ .packing = sja1110_schedule_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_schedule_entry),
+ .packed_entry_size = SJA1110_SIZE_SCHEDULE_ENTRY,
+ .max_entry_count = SJA1110_MAX_SCHEDULE_COUNT,
+ },
+ [BLK_IDX_SCHEDULE_ENTRY_POINTS] = {
+ .packing = sja1110_schedule_entry_points_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_schedule_entry_points_entry),
+ .packed_entry_size = SJA1110_SIZE_SCHEDULE_ENTRY_POINTS_ENTRY,
+ .max_entry_count = SJA1105_MAX_SCHEDULE_ENTRY_POINTS_COUNT,
+ },
+ [BLK_IDX_VL_LOOKUP] = {
+ .packing = sja1110_vl_lookup_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_vl_lookup_entry),
+ .packed_entry_size = SJA1105_SIZE_VL_LOOKUP_ENTRY,
+ .max_entry_count = SJA1110_MAX_VL_LOOKUP_COUNT,
+ },
+ [BLK_IDX_VL_POLICING] = {
+ .packing = sja1110_vl_policing_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_vl_policing_entry),
+ .packed_entry_size = SJA1105_SIZE_VL_POLICING_ENTRY,
+ .max_entry_count = SJA1110_MAX_VL_POLICING_COUNT,
+ },
+ [BLK_IDX_VL_FORWARDING] = {
+ .packing = sja1110_vl_forwarding_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_vl_forwarding_entry),
+ .packed_entry_size = SJA1105_SIZE_VL_FORWARDING_ENTRY,
+ .max_entry_count = SJA1110_MAX_VL_FORWARDING_COUNT,
+ },
+ [BLK_IDX_L2_LOOKUP] = {
+ .packing = sja1110_l2_lookup_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_l2_lookup_entry),
+ .packed_entry_size = SJA1110_SIZE_L2_LOOKUP_ENTRY,
+ .max_entry_count = SJA1105_MAX_L2_LOOKUP_COUNT,
+ },
+ [BLK_IDX_L2_POLICING] = {
+ .packing = sja1110_l2_policing_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_l2_policing_entry),
+ .packed_entry_size = SJA1105_SIZE_L2_POLICING_ENTRY,
+ .max_entry_count = SJA1110_MAX_L2_POLICING_COUNT,
+ },
+ [BLK_IDX_VLAN_LOOKUP] = {
+ .packing = sja1110_vlan_lookup_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_vlan_lookup_entry),
+ .packed_entry_size = SJA1110_SIZE_VLAN_LOOKUP_ENTRY,
+ .max_entry_count = SJA1105_MAX_VLAN_LOOKUP_COUNT,
+ },
+ [BLK_IDX_L2_FORWARDING] = {
+ .packing = sja1110_l2_forwarding_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_l2_forwarding_entry),
+ .packed_entry_size = SJA1105_SIZE_L2_FORWARDING_ENTRY,
+ .max_entry_count = SJA1110_MAX_L2_FORWARDING_COUNT,
+ },
+ [BLK_IDX_MAC_CONFIG] = {
+ .packing = sja1110_mac_config_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_mac_config_entry),
+ .packed_entry_size = SJA1105PQRS_SIZE_MAC_CONFIG_ENTRY,
+ .max_entry_count = SJA1110_MAX_MAC_CONFIG_COUNT,
+ },
+ [BLK_IDX_SCHEDULE_PARAMS] = {
+ .packing = sja1110_schedule_params_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_schedule_params_entry),
+ .packed_entry_size = SJA1105_SIZE_SCHEDULE_PARAMS_ENTRY,
+ .max_entry_count = SJA1105_MAX_SCHEDULE_PARAMS_COUNT,
+ },
+ [BLK_IDX_SCHEDULE_ENTRY_POINTS_PARAMS] = {
+ .packing = sja1105_schedule_entry_points_params_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_schedule_entry_points_params_entry),
+ .packed_entry_size = SJA1105_SIZE_SCHEDULE_ENTRY_POINTS_PARAMS_ENTRY,
+ .max_entry_count = SJA1105_MAX_SCHEDULE_ENTRY_POINTS_PARAMS_COUNT,
+ },
+ [BLK_IDX_VL_FORWARDING_PARAMS] = {
+ .packing = sja1110_vl_forwarding_params_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_vl_forwarding_params_entry),
+ .packed_entry_size = SJA1105_SIZE_VL_FORWARDING_PARAMS_ENTRY,
+ .max_entry_count = SJA1105_MAX_VL_FORWARDING_PARAMS_COUNT,
+ },
+ [BLK_IDX_L2_LOOKUP_PARAMS] = {
+ .packing = sja1110_l2_lookup_params_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_l2_lookup_params_entry),
+ .packed_entry_size = SJA1110_SIZE_L2_LOOKUP_PARAMS_ENTRY,
+ .max_entry_count = SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT,
+ },
+ [BLK_IDX_L2_FORWARDING_PARAMS] = {
+ .packing = sja1110_l2_forwarding_params_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_l2_forwarding_params_entry),
+ .packed_entry_size = SJA1105_SIZE_L2_FORWARDING_PARAMS_ENTRY,
+ .max_entry_count = SJA1105_MAX_L2_FORWARDING_PARAMS_COUNT,
+ },
+ [BLK_IDX_AVB_PARAMS] = {
+ .packing = sja1105pqrs_avb_params_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_avb_params_entry),
+ .packed_entry_size = SJA1105PQRS_SIZE_AVB_PARAMS_ENTRY,
+ .max_entry_count = SJA1105_MAX_AVB_PARAMS_COUNT,
+ },
+ [BLK_IDX_GENERAL_PARAMS] = {
+ .packing = sja1110_general_params_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_general_params_entry),
+ .packed_entry_size = SJA1110_SIZE_GENERAL_PARAMS_ENTRY,
+ .max_entry_count = SJA1105_MAX_GENERAL_PARAMS_COUNT,
+ },
+ [BLK_IDX_RETAGGING] = {
+ .packing = sja1110_retagging_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_retagging_entry),
+ .packed_entry_size = SJA1105_SIZE_RETAGGING_ENTRY,
+ .max_entry_count = SJA1105_MAX_RETAGGING_COUNT,
+ },
+ [BLK_IDX_XMII_PARAMS] = {
+ .packing = sja1110_xmii_params_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1105_xmii_params_entry),
+ .packed_entry_size = SJA1110_SIZE_XMII_PARAMS_ENTRY,
+ .max_entry_count = SJA1105_MAX_XMII_PARAMS_COUNT,
+ },
+ [BLK_IDX_PCP_REMAPPING] = {
+ .packing = sja1110_pcp_remapping_entry_packing,
+ .unpacked_entry_size = sizeof(struct sja1110_pcp_remapping_entry),
+ .packed_entry_size = SJA1110_SIZE_PCP_REMAPPING_ENTRY,
+ .max_entry_count = SJA1110_MAX_PCP_REMAPPING_COUNT,
+ },
+};
+
int sja1105_static_config_init(struct sja1105_static_config *config,
const struct sja1105_table_ops *static_ops,
u64 device_id)
#include <linux/types.h>
#include <asm/types.h>
+#define SJA1105_NUM_PORTS 5
+#define SJA1110_NUM_PORTS 11
+#define SJA1105_MAX_NUM_PORTS SJA1110_NUM_PORTS
+#define SJA1105_NUM_TC 8
+
+#define SJA1105_SIZE_SPI_MSG_HEADER 4
+#define SJA1105_SIZE_SPI_MSG_MAXLEN (64 * 4)
#define SJA1105_SIZE_DEVICE_ID 4
#define SJA1105_SIZE_TABLE_HEADER 12
#define SJA1105_SIZE_SCHEDULE_ENTRY 8
+#define SJA1110_SIZE_SCHEDULE_ENTRY 12
#define SJA1105_SIZE_SCHEDULE_ENTRY_POINTS_ENTRY 4
+#define SJA1110_SIZE_SCHEDULE_ENTRY_POINTS_ENTRY 8
#define SJA1105_SIZE_VL_LOOKUP_ENTRY 12
#define SJA1105_SIZE_VL_POLICING_ENTRY 8
#define SJA1105_SIZE_VL_FORWARDING_ENTRY 4
#define SJA1105_SIZE_L2_POLICING_ENTRY 8
#define SJA1105_SIZE_VLAN_LOOKUP_ENTRY 8
+#define SJA1110_SIZE_VLAN_LOOKUP_ENTRY 12
#define SJA1105_SIZE_L2_FORWARDING_ENTRY 8
#define SJA1105_SIZE_L2_FORWARDING_PARAMS_ENTRY 12
#define SJA1105_SIZE_RETAGGING_ENTRY 8
#define SJA1105_SIZE_XMII_PARAMS_ENTRY 4
+#define SJA1110_SIZE_XMII_PARAMS_ENTRY 8
#define SJA1105_SIZE_SCHEDULE_PARAMS_ENTRY 12
#define SJA1105_SIZE_SCHEDULE_ENTRY_POINTS_PARAMS_ENTRY 4
#define SJA1105_SIZE_VL_FORWARDING_PARAMS_ENTRY 12
#define SJA1105ET_SIZE_AVB_PARAMS_ENTRY 12
#define SJA1105ET_SIZE_CBS_ENTRY 16
#define SJA1105PQRS_SIZE_L2_LOOKUP_ENTRY 20
+#define SJA1110_SIZE_L2_LOOKUP_ENTRY 24
#define SJA1105PQRS_SIZE_MAC_CONFIG_ENTRY 32
#define SJA1105PQRS_SIZE_L2_LOOKUP_PARAMS_ENTRY 16
+#define SJA1110_SIZE_L2_LOOKUP_PARAMS_ENTRY 28
#define SJA1105PQRS_SIZE_GENERAL_PARAMS_ENTRY 44
+#define SJA1110_SIZE_GENERAL_PARAMS_ENTRY 56
#define SJA1105PQRS_SIZE_AVB_PARAMS_ENTRY 16
#define SJA1105PQRS_SIZE_CBS_ENTRY 20
+#define SJA1110_SIZE_PCP_REMAPPING_ENTRY 4
/* UM10944.pdf Page 11, Table 2. Configuration Blocks */
enum {
BLKID_GENERAL_PARAMS = 0x11,
BLKID_RETAGGING = 0x12,
BLKID_CBS = 0x13,
+ BLKID_PCP_REMAPPING = 0x1C,
BLKID_XMII_PARAMS = 0x4E,
};
BLK_IDX_RETAGGING,
BLK_IDX_CBS,
BLK_IDX_XMII_PARAMS,
+ BLK_IDX_PCP_REMAPPING,
BLK_IDX_MAX,
/* Fake block indices that are only valid for dynamic access */
BLK_IDX_MGMT_ROUTE,
};
#define SJA1105_MAX_SCHEDULE_COUNT 1024
+#define SJA1110_MAX_SCHEDULE_COUNT 4096
#define SJA1105_MAX_SCHEDULE_ENTRY_POINTS_COUNT 2048
#define SJA1105_MAX_VL_LOOKUP_COUNT 1024
+#define SJA1110_MAX_VL_LOOKUP_COUNT 4096
#define SJA1105_MAX_VL_POLICING_COUNT 1024
+#define SJA1110_MAX_VL_POLICING_COUNT 4096
#define SJA1105_MAX_VL_FORWARDING_COUNT 1024
+#define SJA1110_MAX_VL_FORWARDING_COUNT 4096
#define SJA1105_MAX_L2_LOOKUP_COUNT 1024
#define SJA1105_MAX_L2_POLICING_COUNT 45
+#define SJA1110_MAX_L2_POLICING_COUNT 110
#define SJA1105_MAX_VLAN_LOOKUP_COUNT 4096
#define SJA1105_MAX_L2_FORWARDING_COUNT 13
+#define SJA1110_MAX_L2_FORWARDING_COUNT 19
#define SJA1105_MAX_MAC_CONFIG_COUNT 5
+#define SJA1110_MAX_MAC_CONFIG_COUNT 11
#define SJA1105_MAX_SCHEDULE_PARAMS_COUNT 1
#define SJA1105_MAX_SCHEDULE_ENTRY_POINTS_PARAMS_COUNT 1
#define SJA1105_MAX_VL_FORWARDING_PARAMS_COUNT 1
#define SJA1105_MAX_AVB_PARAMS_COUNT 1
#define SJA1105ET_MAX_CBS_COUNT 10
#define SJA1105PQRS_MAX_CBS_COUNT 16
+#define SJA1110_MAX_CBS_COUNT 80
+#define SJA1110_MAX_PCP_REMAPPING_COUNT 11
#define SJA1105_MAX_FRAME_MEMORY 929
-#define SJA1105_MAX_FRAME_MEMORY_RETAGGING 910
+#define SJA1110_MAX_FRAME_MEMORY 1820
+#define SJA1105_FRAME_MEMORY_RETAGGING_OVERHEAD 19
#define SJA1105_VL_FRAME_MEMORY 100
#define SJA1105E_DEVICE_ID 0x9C00000Cull
#define SJA1105T_DEVICE_ID 0x9E00030Eull
#define SJA1105PR_DEVICE_ID 0xAF00030Eull
#define SJA1105QS_DEVICE_ID 0xAE00030Eull
+#define SJA1110_DEVICE_ID 0xB700030Full
#define SJA1105ET_PART_NO 0x9A83
#define SJA1105P_PART_NO 0x9A84
#define SJA1105Q_PART_NO 0x9A85
#define SJA1105R_PART_NO 0x9A86
#define SJA1105S_PART_NO 0x9A87
+#define SJA1110A_PART_NO 0x1110
+#define SJA1110B_PART_NO 0x1111
+#define SJA1110C_PART_NO 0x1112
+#define SJA1110D_PART_NO 0x1113
+
+#define SJA1110_ACU 0x1c4400
+#define SJA1110_RGU 0x1c6000
+#define SJA1110_CGU 0x1c6400
+
+#define SJA1110_SPI_ADDR(x) ((x) / 4)
+#define SJA1110_ACU_ADDR(x) (SJA1110_ACU + SJA1110_SPI_ADDR(x))
+#define SJA1110_CGU_ADDR(x) (SJA1110_CGU + SJA1110_SPI_ADDR(x))
+#define SJA1110_RGU_ADDR(x) (SJA1110_RGU + SJA1110_SPI_ADDR(x))
+
+#define SJA1105_RSV_ADDR 0xffffffffffffffffull
struct sja1105_schedule_entry {
u64 winstindex;
u64 egrmirrpcp;
u64 egrmirrdei;
u64 replay_port;
+ /* SJA1110 only */
+ u64 tte_en;
+ u64 tdmaconfigidx;
+ u64 header_type;
};
struct sja1105_schedule_entry_points_entry {
u64 vlan_bc;
u64 tag_port;
u64 vlanid;
+ u64 type_entry; /* SJA1110 only */
};
struct sja1105_l2_lookup_entry {
u64 mask_iotag;
u64 mask_vlanid;
u64 mask_macaddr;
+ u64 mask_srcport;
u64 iotag;
+ u64 srcport;
u64 lockeds;
union {
/* LOCKEDS=1: Static FDB entries */
struct {
+ /* TSREG is deprecated in SJA1110, TRAP is supported only
+ * in SJA1110.
+ */
+ u64 trap;
u64 tsreg;
u64 mirrvlan;
u64 takets;
};
struct sja1105_l2_lookup_params_entry {
- u64 maxaddrp[5]; /* P/Q/R/S only */
+ u64 maxaddrp[SJA1105_MAX_NUM_PORTS]; /* P/Q/R/S only */
u64 start_dynspc; /* P/Q/R/S only */
u64 drpnolearn; /* P/Q/R/S only */
u64 use_static; /* P/Q/R/S only */
u64 bc_domain;
u64 reach_port;
u64 fl_domain;
- u64 vlan_pmap[8];
+ /* This is actually max(SJA1105_NUM_TC, SJA1105_MAX_NUM_PORTS) */
+ u64 vlan_pmap[SJA1105_MAX_NUM_PORTS];
+ bool type_egrpcp2outputq;
};
struct sja1105_l2_forwarding_params_entry {
};
struct sja1105_cbs_entry {
- u64 port;
- u64 prio;
+ u64 port; /* Not used for SJA1110 */
+ u64 prio; /* Not used for SJA1110 */
u64 credit_hi;
u64 credit_lo;
u64 send_slope;
};
struct sja1105_xmii_params_entry {
- u64 phy_mac[5];
- u64 xmii_mode[5];
+ u64 phy_mac[SJA1105_MAX_NUM_PORTS];
+ u64 xmii_mode[SJA1105_MAX_NUM_PORTS];
+ /* The SJA1110 insists being a snowflake, and requires SGMII,
+ * 2500base-x and internal MII ports connected to the 100base-TX PHY to
+ * set this bit. We set it unconditionally from the high-level logic,
+ * and only sja1110_xmii_params_entry_packing writes it to the static
+ * config. I have no better name for it than "special".
+ */
+ u64 special[SJA1105_MAX_NUM_PORTS];
+};
+
+struct sja1110_pcp_remapping_entry {
+ u64 egrpcp[SJA1105_NUM_TC];
};
enum {
extern const struct sja1105_table_ops sja1105q_table_ops[BLK_IDX_MAX];
extern const struct sja1105_table_ops sja1105r_table_ops[BLK_IDX_MAX];
extern const struct sja1105_table_ops sja1105s_table_ops[BLK_IDX_MAX];
+extern const struct sja1105_table_ops sja1110_table_ops[BLK_IDX_MAX];
size_t sja1105_table_header_packing(void *buf, void *hdr, enum packing_op op);
void
extern const char *sja1105_static_config_error_msg[];
sja1105_config_valid_t
-sja1105_static_config_check_valid(const struct sja1105_static_config *config);
+sja1105_static_config_check_valid(const struct sja1105_static_config *config,
+ int max_mem);
void
sja1105_static_config_pack(void *buf, struct sja1105_static_config *config);
int sja1105_static_config_init(struct sja1105_static_config *config,
/* Common implementations for the static and dynamic configs */
size_t sja1105pqrs_general_params_entry_packing(void *buf, void *entry_ptr,
enum packing_op op);
+size_t sja1110_general_params_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op);
size_t sja1105pqrs_l2_lookup_params_entry_packing(void *buf, void *entry_ptr,
enum packing_op op);
+size_t sja1110_l2_lookup_params_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op);
size_t sja1105_l2_forwarding_entry_packing(void *buf, void *entry_ptr,
enum packing_op op);
+size_t sja1110_l2_forwarding_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op);
size_t sja1105pqrs_l2_lookup_entry_packing(void *buf, void *entry_ptr,
enum packing_op op);
size_t sja1105et_l2_lookup_entry_packing(void *buf, void *entry_ptr,
enum packing_op op);
+size_t sja1110_l2_lookup_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op);
size_t sja1105_vlan_lookup_entry_packing(void *buf, void *entry_ptr,
enum packing_op op);
+size_t sja1110_vlan_lookup_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op);
size_t sja1105_retagging_entry_packing(void *buf, void *entry_ptr,
enum packing_op op);
+size_t sja1110_retagging_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op);
size_t sja1105pqrs_mac_config_entry_packing(void *buf, void *entry_ptr,
enum packing_op op);
+size_t sja1110_mac_config_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op);
size_t sja1105pqrs_avb_params_entry_packing(void *buf, void *entry_ptr,
enum packing_op op);
size_t sja1105_vl_lookup_entry_packing(void *buf, void *entry_ptr,
enum packing_op op);
+size_t sja1110_vl_lookup_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op);
+size_t sja1110_vl_policing_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op);
+size_t sja1110_xmii_params_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op);
+size_t sja1110_l2_policing_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op);
+size_t sja1110_l2_forwarding_params_entry_packing(void *buf, void *entry_ptr,
+ enum packing_op op);
#endif
tas_data->enabled = false;
- for (port = 0; port < SJA1105_NUM_PORTS; port++) {
+ for (port = 0; port < ds->num_ports; port++) {
const struct tc_taprio_qopt_offload *offload;
offload = tas_data->offload[port];
struct sja1105_tas_data *tas_data = &priv->tas_data;
struct sja1105_gating_config *gating_cfg = &tas_data->gating_cfg;
struct sja1105_schedule_entry *schedule;
+ struct dsa_switch *ds = priv->ds;
struct sja1105_table *table;
int schedule_start_idx;
s64 entry_point_delta;
}
/* Figure out the dimensioning of the problem */
- for (port = 0; port < SJA1105_NUM_PORTS; port++) {
+ for (port = 0; port < ds->num_ports; port++) {
if (tas_data->offload[port]) {
num_entries += tas_data->offload[port]->num_entries;
num_cycles++;
schedule_entry_points_params->clksrc = SJA1105_TAS_CLKSRC_PTP;
schedule_entry_points_params->actsubsch = num_cycles - 1;
- for (port = 0; port < SJA1105_NUM_PORTS; port++) {
+ for (port = 0; port < ds->num_ports; port++) {
const struct tc_taprio_qopt_offload *offload;
/* Relative base time */
s64 rbt;
struct sja1105_gating_config *gating_cfg = &priv->tas_data.gating_cfg;
size_t num_entries = gating_cfg->num_entries;
struct tc_taprio_qopt_offload *dummy;
+ struct dsa_switch *ds = priv->ds;
struct sja1105_gate_entry *e;
bool conflict;
int i = 0;
if (port != -1) {
conflict = sja1105_tas_check_conflicts(priv, port, dummy);
} else {
- for (port = 0; port < SJA1105_NUM_PORTS; port++) {
+ for (port = 0; port < ds->num_ports; port++) {
conflict = sja1105_tas_check_conflicts(priv, port,
dummy);
if (conflict)
}
}
- for (other_port = 0; other_port < SJA1105_NUM_PORTS; other_port++) {
+ for (other_port = 0; other_port < ds->num_ports; other_port++) {
if (other_port == port)
continue;
cancel_work_sync(&priv->tas_data.tas_work);
- for (port = 0; port < SJA1105_NUM_PORTS; port++) {
+ for (port = 0; port < ds->num_ports; port++) {
offload = priv->tas_data.offload[port];
if (!offload)
continue;
};
struct sja1105_tas_data {
- struct tc_taprio_qopt_offload *offload[SJA1105_NUM_PORTS];
+ struct tc_taprio_qopt_offload *offload[SJA1105_MAX_NUM_PORTS];
struct sja1105_gating_config gating_cfg;
enum sja1105_tas_state state;
enum sja1105_ptp_op last_op;
if (rule->type != SJA1105_RULE_VL)
continue;
- for_each_set_bit(port, &rule->port_mask, SJA1105_NUM_PORTS) {
+ for_each_set_bit(port, &rule->port_mask, SJA1105_MAX_NUM_PORTS) {
vl_lookup[k].format = SJA1105_VL_FORMAT_PSFP;
vl_lookup[k].port = port;
vl_lookup[k].macaddr = rule->key.vl.dmac;
XRS700X_MIB(XRS_EARLY_DROP_L, "early_drop", tx_dropped),
};
+static const u8 eth_hsrsup_addr[ETH_ALEN] = {
+ 0x01, 0x15, 0x4e, 0x00, 0x01, 0x00};
+
static void xrs700x_get_strings(struct dsa_switch *ds, int port,
u32 stringset, u8 *data)
{
return 0;
}
+/* Add an inbound policy filter which matches the HSR/PRP supervision MAC
+ * range and forwards to the CPU port without discarding duplicates.
+ * This is required to correctly populate the HSR/PRP node_table.
+ * Leave the policy disabled, it will be enabled as needed.
+ */
+static int xrs700x_port_add_hsrsup_ipf(struct dsa_switch *ds, int port,
+ int fwdport)
+{
+ struct xrs700x *priv = ds->priv;
+ unsigned int val = 0;
+ int i = 0;
+ int ret;
+
+ /* Compare 40 bits of the destination MAC address. */
+ ret = regmap_write(priv->regmap, XRS_ETH_ADDR_CFG(port, 1), 40 << 2);
+ if (ret)
+ return ret;
+
+ /* match HSR/PRP supervision destination 01:15:4e:00:01:XX */
+ for (i = 0; i < sizeof(eth_hsrsup_addr); i += 2) {
+ ret = regmap_write(priv->regmap, XRS_ETH_ADDR_0(port, 1) + i,
+ eth_hsrsup_addr[i] |
+ (eth_hsrsup_addr[i + 1] << 8));
+ if (ret)
+ return ret;
+ }
+
+ /* Mirror HSR/PRP supervision to CPU port */
+ for (i = 0; i < ds->num_ports; i++) {
+ if (dsa_is_cpu_port(ds, i))
+ val |= BIT(i);
+ }
+
+ ret = regmap_write(priv->regmap, XRS_ETH_ADDR_FWD_MIRROR(port, 1), val);
+ if (ret)
+ return ret;
+
+ if (fwdport >= 0)
+ val |= BIT(fwdport);
+
+ /* Allow must be set prevent duplicate discard */
+ ret = regmap_write(priv->regmap, XRS_ETH_ADDR_FWD_ALLOW(port, 1), val);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
static int xrs700x_port_setup(struct dsa_switch *ds, int port)
{
bool cpu_port = dsa_is_cpu_port(ds, port);
struct net_device *slave;
int ret, i, hsr_pair[2];
enum hsr_version ver;
+ bool fwd = false;
ret = hsr_get_version(hsr, &ver);
if (ret)
if (ver == HSR_V1) {
val &= ~BIT(partner->index);
val &= ~BIT(port);
+ fwd = true;
}
val &= ~BIT(dsa_upstream_port(ds, port));
regmap_write(priv->regmap, XRS_PORT_FWD_MASK(partner->index), val);
XRS_PORT_FORWARDING);
regmap_fields_write(priv->ps_forward, port, XRS_PORT_FORWARDING);
+ /* Enable inbound policy which allows HSR/PRP supervision forwarding
+ * to the CPU port without discarding duplicates. Continue to
+ * forward to redundant ports when in HSR mode while discarding
+ * duplicates.
+ */
+ ret = xrs700x_port_add_hsrsup_ipf(ds, partner->index, fwd ? port : -1);
+ if (ret)
+ return ret;
+
+ ret = xrs700x_port_add_hsrsup_ipf(ds, port, fwd ? partner->index : -1);
+ if (ret)
+ return ret;
+
+ regmap_update_bits(priv->regmap,
+ XRS_ETH_ADDR_CFG(partner->index, 1), 1, 1);
+ regmap_update_bits(priv->regmap, XRS_ETH_ADDR_CFG(port, 1), 1, 1);
+
hsr_pair[0] = port;
hsr_pair[1] = partner->index;
for (i = 0; i < ARRAY_SIZE(hsr_pair); i++) {
XRS_PORT_FORWARDING);
regmap_fields_write(priv->ps_forward, port, XRS_PORT_FORWARDING);
+ /* Disable inbound policy added by xrs700x_port_add_hsrsup_ipf()
+ * which allows HSR/PRP supervision forwarding to the CPU port without
+ * discarding duplicates.
+ */
+ regmap_update_bits(priv->regmap,
+ XRS_ETH_ADDR_CFG(partner->index, 1), 1, 0);
+ regmap_update_bits(priv->regmap, XRS_ETH_ADDR_CFG(port, 1), 1, 0);
+
hsr_pair[0] = port;
hsr_pair[1] = partner->index;
for (i = 0; i < ARRAY_SIZE(hsr_pair); i++) {
if (pdev) {
vp->pm_state_valid = 1;
pci_save_state(pdev);
- acpi_set_WOL(dev);
+ acpi_set_WOL(dev);
}
retval = register_netdev(dev);
if (retval == 0)
Paul Gortmaker : tweak ANK's above multicast changes a bit.
Paul Gortmaker : update packet statistics for v2.1.x
Alan Cox : support arbitrary stupid port mappings on the
- 68K Macintosh. Support >16bit I/O spaces
+ 68K Macintosh. Support >16bit I/O spaces
Paul Gortmaker : add kmod support for auto-loading of the 8390
module by all drivers that require it.
Alan Cox : Spinlocking work, added 'BUG_83C690'
long e8390_base;
int interrupts, nr_serviced = 0, i;
struct ei_device *ei_local;
- int handled = 0;
+ int handled = 0;
unsigned long flags;
e8390_base = dev->base_addr;
}
outb_p(E8390_NODMA + E8390_PAGE0, e8390_base + E8390_CMD);
- if(dev->flags&IFF_PROMISC)
- outb_p(E8390_RXCONFIG | 0x58, e8390_base + EN0_RXCR);
+ if(dev->flags&IFF_PROMISC)
+ outb_p(E8390_RXCONFIG | 0x58, e8390_base + EN0_RXCR);
else if (dev->flags & IFF_ALLMULTI || !netdev_mc_empty(dev))
- outb_p(E8390_RXCONFIG | 0x48, e8390_base + EN0_RXCR);
- else
- outb_p(E8390_RXCONFIG | 0x40, e8390_base + EN0_RXCR);
+ outb_p(E8390_RXCONFIG | 0x48, e8390_base + EN0_RXCR);
+ else
+ outb_p(E8390_RXCONFIG | 0x40, e8390_base + EN0_RXCR);
outb_p(E8390_NODMA+E8390_PAGE0+E8390_START, e8390_base+E8390_CMD);
}
PCMCIA_DEVICE_PROD_ID12("ACCTON", "EN2216-PCMCIA-ETHERNET", 0xdfc6b5b2, 0x5542bfff),
PCMCIA_DEVICE_PROD_ID12("Allied Telesis, K.K.", "CentreCOM LA100-PCM-T V2 100/10M LAN PC Card", 0xbb7fbdd7, 0xcd91cc68),
PCMCIA_DEVICE_PROD_ID12("Allied Telesis K.K.", "LA100-PCM V2", 0x36634a66, 0xc6d05997),
- PCMCIA_DEVICE_PROD_ID12("Allied Telesis, K.K.", "CentreCOM LA-PCM_V2", 0xbb7fBdd7, 0x28e299f8),
+ PCMCIA_DEVICE_PROD_ID12("Allied Telesis, K.K.", "CentreCOM LA-PCM_V2", 0xbb7fBdd7, 0x28e299f8),
PCMCIA_DEVICE_PROD_ID12("Allied Telesis K.K.", "LA-PCM V3", 0x36634a66, 0x62241d96),
PCMCIA_DEVICE_PROD_ID12("AmbiCom", "AMB8010", 0x5070a7f9, 0x82f96e96),
PCMCIA_DEVICE_PROD_ID12("AmbiCom", "AMB8610", 0x5070a7f9, 0x86741224),
idev))) {
/* Avoid already found cards from previous calls */
if (pnp_device_attach(idev) < 0)
- continue;
+ continue;
if (pnp_activate_dev(idev) < 0) {
__again:
- pnp_device_detach(idev);
- continue;
+ pnp_device_detach(idev);
+ continue;
}
/* if no io and irq, search for next */
if (!pnp_port_valid(idev, 0) || !pnp_irq_valid(idev, 0))
/* New style probing API */
dev = alloc_ei_netdev();
if (!dev)
- return -ENOMEM;
+ return -ENOMEM;
#ifdef CONFIG_SH_STANDARD_BIOS
sh_bios_get_node_addr (stnic_eadr);
}
}
- if (ACE_IS_TIGON_I(ap)) {
- struct cmd cmd;
- cmd.evt = C_SET_RX_JUMBO_PRD_IDX;
- cmd.code = 0;
- cmd.idx = 0;
- ace_issue_cmd(ap->regs, &cmd);
- } else {
- writel(0, &((ap->regs)->RxJumboPrd));
- wmb();
- }
+ if (ACE_IS_TIGON_I(ap)) {
+ struct cmd cmd;
+ cmd.evt = C_SET_RX_JUMBO_PRD_IDX;
+ cmd.code = 0;
+ cmd.idx = 0;
+ ace_issue_cmd(ap->regs, &cmd);
+ } else {
+ writel(0, &((ap->regs)->RxJumboPrd));
+ wmb();
+ }
ap->jumbo = 0;
ap->rx_jumbo_skbprd = 0;
}
}
- wmb();
- ap->tx_prd = idx;
- ace_set_txprd(regs, ap, idx);
+ wmb();
+ ap->tx_prd = idx;
+ ace_set_txprd(regs, ap, idx);
if (flagsize & BD_FLG_COAL_NOW) {
netif_stop_queue(dev);
};
enum ena_admin_aenq_notification_syndrome {
- ENA_ADMIN_SUSPEND = 0,
- ENA_ADMIN_RESUME = 1,
ENA_ADMIN_UPDATE_HINTS = 2,
};
if (rc)
return rc;
- if (get_resp.u.max_queue_ext.version != ENA_FEATURE_MAX_QUEUE_EXT_VER)
+ if (get_resp.u.max_queue_ext.version !=
+ ENA_FEATURE_MAX_QUEUE_EXT_VER)
return -EINVAL;
memcpy(&get_feat_ctx->max_queue_ext, &get_resp.u.max_queue_ext,
return 0;
/* bounce buffer was used, so write it and get a new one */
- if (pkt_ctrl->idx) {
+ if (likely(pkt_ctrl->idx)) {
rc = ena_com_write_bounce_buffer_to_dev(io_sq,
pkt_ctrl->curr_bounce_buf);
- if (unlikely(rc))
+ if (unlikely(rc)) {
+ netdev_err(ena_com_io_sq_to_ena_dev(io_sq)->net_device,
+ "Failed to write bounce buffer to device\n");
return rc;
+ }
pkt_ctrl->curr_bounce_buf =
ena_com_get_next_bounce_buffer(&io_sq->bounce_buf_ctrl);
if (!pkt_ctrl->descs_left_in_line) {
rc = ena_com_write_bounce_buffer_to_dev(io_sq,
pkt_ctrl->curr_bounce_buf);
- if (unlikely(rc))
+ if (unlikely(rc)) {
+ netdev_err(ena_com_io_sq_to_ena_dev(io_sq)->net_device,
+ "Failed to write bounce buffer to device\n");
return rc;
+ }
pkt_ctrl->curr_bounce_buf =
ena_com_get_next_bounce_buffer(&io_sq->bounce_buf_ctrl);
}
if (unlikely(io_sq->mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_DEV &&
- !buffer_to_push))
+ !buffer_to_push)) {
+ netdev_err(ena_com_io_sq_to_ena_dev(io_sq)->net_device,
+ "Push header wasn't provided in LLQ mode\n");
return -EINVAL;
+ }
rc = ena_com_write_header_to_bounce(io_sq, buffer_to_push, header_len);
if (unlikely(rc))
/* If the caller doesn't want to send packets */
if (unlikely(!num_bufs && !header_len)) {
rc = ena_com_close_bounce_buffer(io_sq);
+ if (rc)
+ netdev_err(ena_com_io_sq_to_ena_dev(io_sq)->net_device,
+ "Failed to write buffers to LLQ\n");
*nb_hw_desc = io_sq->tail - start_tail;
return rc;
}
/* The first desc share the same desc as the header */
if (likely(i != 0)) {
rc = ena_com_sq_update_tail(io_sq);
- if (unlikely(rc))
+ if (unlikely(rc)) {
+ netdev_err(ena_com_io_sq_to_ena_dev(io_sq)->net_device,
+ "Failed to update sq tail\n");
return rc;
+ }
desc = get_sq_desc(io_sq);
if (unlikely(!desc))
desc->len_ctrl |= ENA_ETH_IO_TX_DESC_LAST_MASK;
rc = ena_com_sq_update_tail(io_sq);
- if (unlikely(rc))
+ if (unlikely(rc)) {
+ netdev_err(ena_com_io_sq_to_ena_dev(io_sq)->net_device,
+ "Failed to update sq tail of the last descriptor\n");
return rc;
+ }
rc = ena_com_close_bounce_buffer(io_sq);
{
struct ena_adapter *adapter = netdev_priv(netdev);
- if (sset != ETH_SS_STATS)
- return -EOPNOTSUPP;
+ switch (sset) {
+ case ETH_SS_STATS:
+ return ena_get_sw_stats_count(adapter) +
+ ena_get_hw_stats_count(adapter);
+ }
- return ena_get_sw_stats_count(adapter) + ena_get_hw_stats_count(adapter);
+ return -EOPNOTSUPP;
}
static void ena_queue_strings(struct ena_adapter *adapter, u8 **data)
{
struct ena_adapter *adapter = netdev_priv(netdev);
- if (sset != ETH_SS_STATS)
- return;
-
- ena_get_strings(adapter, data, adapter->eni_stats_supported);
+ switch (sset) {
+ case ETH_SS_STATS:
+ ena_get_strings(adapter, data, adapter->eni_stats_supported);
+ break;
+ }
}
static int ena_get_link_ksettings(struct net_device *netdev,
#define DEFAULT_MSG_ENABLE (NETIF_MSG_DRV | NETIF_MSG_PROBE | NETIF_MSG_IFUP | \
NETIF_MSG_TX_DONE | NETIF_MSG_TX_ERR | NETIF_MSG_RX_ERR)
-static int debug = -1;
-module_param(debug, int, 0);
-MODULE_PARM_DESC(debug, "Debug level (0=none,...,16=all)");
static struct ena_aenq_handlers aenq_handlers;
u64_stats_update_end(syncp);
}
+static void ena_ring_tx_doorbell(struct ena_ring *tx_ring)
+{
+ ena_com_write_sq_doorbell(tx_ring->ena_com_io_sq);
+ ena_increase_stat(&tx_ring->tx_stats.doorbells, 1, &tx_ring->syncp);
+}
+
static void ena_tx_timeout(struct net_device *dev, unsigned int txqueue)
{
struct ena_adapter *adapter = netdev_priv(dev);
netif_dbg(adapter, tx_queued, dev,
"llq tx max burst size of queue %d achieved, writing doorbell to send burst\n",
ring->qid);
- ena_com_write_sq_doorbell(ring->ena_com_io_sq);
+ ena_ring_tx_doorbell(ring);
}
/* prepare the packet's descriptors to dma engine */
int ret;
xdp_ring = ena_napi->xdp_ring;
- xdp_ring->first_interrupt = ena_napi->first_interrupt;
xdp_budget = budget;
xdp_ring->tx_stats.napi_comp += napi_comp_call;
xdp_ring->tx_stats.tx_poll++;
u64_stats_update_end(&xdp_ring->syncp);
+ xdp_ring->tx_stats.last_napi_jiffies = jiffies;
return ret;
}
static int ena_xdp_tx_map_frame(struct ena_ring *xdp_ring,
struct ena_tx_buffer *tx_info,
struct xdp_frame *xdpf,
- void **push_hdr,
- u32 *push_len)
+ struct ena_com_tx_ctx *ena_tx_ctx)
{
struct ena_adapter *adapter = xdp_ring->adapter;
struct ena_com_buf *ena_buf;
- dma_addr_t dma = 0;
+ int push_len = 0;
+ dma_addr_t dma;
+ void *data;
u32 size;
tx_info->xdpf = xdpf;
+ data = tx_info->xdpf->data;
size = tx_info->xdpf->len;
- ena_buf = tx_info->bufs;
- /* llq push buffer */
- *push_len = min_t(u32, size, xdp_ring->tx_max_header_size);
- *push_hdr = tx_info->xdpf->data;
+ if (xdp_ring->tx_mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_DEV) {
+ /* Designate part of the packet for LLQ */
+ push_len = min_t(u32, size, xdp_ring->tx_max_header_size);
+
+ ena_tx_ctx->push_header = data;
- if (size - *push_len > 0) {
+ size -= push_len;
+ data += push_len;
+ }
+
+ ena_tx_ctx->header_len = push_len;
+
+ if (size > 0) {
dma = dma_map_single(xdp_ring->dev,
- *push_hdr + *push_len,
- size - *push_len,
+ data,
+ size,
DMA_TO_DEVICE);
if (unlikely(dma_mapping_error(xdp_ring->dev, dma)))
goto error_report_dma_error;
- tx_info->map_linear_data = 1;
- tx_info->num_of_bufs = 1;
- }
+ tx_info->map_linear_data = 0;
- ena_buf->paddr = dma;
- ena_buf->len = size;
+ ena_buf = tx_info->bufs;
+ ena_buf->paddr = dma;
+ ena_buf->len = size;
+
+ ena_tx_ctx->ena_bufs = ena_buf;
+ ena_tx_ctx->num_bufs = tx_info->num_of_bufs = 1;
+ }
return 0;
&xdp_ring->syncp);
netif_warn(adapter, tx_queued, adapter->netdev, "Failed to map xdp buff\n");
- xdp_return_frame_rx_napi(tx_info->xdpf);
- tx_info->xdpf = NULL;
- tx_info->num_of_bufs = 0;
-
return -EINVAL;
}
struct ena_com_tx_ctx ena_tx_ctx = {};
struct ena_tx_buffer *tx_info;
u16 next_to_use, req_id;
- void *push_hdr;
- u32 push_len;
int rc;
next_to_use = xdp_ring->next_to_use;
tx_info = &xdp_ring->tx_buffer_info[req_id];
tx_info->num_of_bufs = 0;
- rc = ena_xdp_tx_map_frame(xdp_ring, tx_info, xdpf, &push_hdr, &push_len);
+ rc = ena_xdp_tx_map_frame(xdp_ring, tx_info, xdpf, &ena_tx_ctx);
if (unlikely(rc))
return rc;
- ena_tx_ctx.ena_bufs = tx_info->bufs;
- ena_tx_ctx.push_header = push_hdr;
- ena_tx_ctx.num_bufs = tx_info->num_of_bufs;
ena_tx_ctx.req_id = req_id;
- ena_tx_ctx.header_len = push_len;
rc = ena_xmit_common(dev,
xdp_ring,
xdpf->len);
if (rc)
goto error_unmap_dma;
- /* trigger the dma engine. ena_com_write_sq_doorbell()
- * has a mb
+
+ /* trigger the dma engine. ena_ring_tx_doorbell()
+ * calls a memory barrier inside it.
*/
- if (flags & XDP_XMIT_FLUSH) {
- ena_com_write_sq_doorbell(xdp_ring->ena_com_io_sq);
- ena_increase_stat(&xdp_ring->tx_stats.doorbells, 1,
- &xdp_ring->syncp);
- }
+ if (flags & XDP_XMIT_FLUSH)
+ ena_ring_tx_doorbell(xdp_ring);
return rc;
}
/* Ring doorbell to make device aware of the packets */
- if (flags & XDP_XMIT_FLUSH) {
- ena_com_write_sq_doorbell(xdp_ring->ena_com_io_sq);
- ena_increase_stat(&xdp_ring->tx_stats.doorbells, 1,
- &xdp_ring->syncp);
- }
+ if (flags & XDP_XMIT_FLUSH)
+ ena_ring_tx_doorbell(xdp_ring);
spin_unlock(&xdp_ring->xdp_tx_lock);
u32 verdict = XDP_PASS;
struct xdp_frame *xdpf;
u64 *xdp_stat;
- int qid;
rcu_read_lock();
xdp_prog = READ_ONCE(rx_ring->xdp_bpf_prog);
}
/* Find xmit queue */
- qid = rx_ring->qid + rx_ring->adapter->num_io_queues;
- xdp_ring = &rx_ring->adapter->tx_ring[qid];
+ xdp_ring = rx_ring->xdp_ring;
/* The XDP queues are shared between XDP_TX and XDP_REDIRECT */
spin_lock(&xdp_ring->xdp_tx_lock);
rx_ring->rx_headroom = XDP_PACKET_HEADROOM;
} else {
ena_xdp_unregister_rxq_info(rx_ring);
- rx_ring->rx_headroom = 0;
+ rx_ring->rx_headroom = NET_SKB_PAD;
}
}
}
ring->ena_dev = adapter->ena_dev;
ring->per_napi_packets = 0;
ring->cpu = 0;
- ring->first_interrupt = false;
ring->no_interrupt_event_cnt = 0;
u64_stats_init(&ring->syncp);
}
rxr->smoothed_interval =
ena_com_get_nonadaptive_moderation_interval_rx(ena_dev);
rxr->empty_rx_queue = 0;
+ rxr->rx_headroom = NET_SKB_PAD;
adapter->ena_napi[i].dim.mode = DIM_CQ_PERIOD_MODE_START_FROM_EQE;
+ rxr->xdp_ring = &adapter->tx_ring[i + adapter->num_io_queues];
}
}
}
ena_free_rx_resources(adapter, i);
}
-static int ena_alloc_rx_page(struct ena_ring *rx_ring,
- struct ena_rx_buffer *rx_info, gfp_t gfp)
+static struct page *ena_alloc_map_page(struct ena_ring *rx_ring,
+ dma_addr_t *dma)
{
- int headroom = rx_ring->rx_headroom;
- struct ena_com_buf *ena_buf;
struct page *page;
- dma_addr_t dma;
-
- /* restore page offset value in case it has been changed by device */
- rx_info->page_offset = headroom;
-
- /* if previous allocated page is not used */
- if (unlikely(rx_info->page))
- return 0;
- page = alloc_page(gfp);
- if (unlikely(!page)) {
+ /* This would allocate the page on the same NUMA node the executing code
+ * is running on.
+ */
+ page = dev_alloc_page();
+ if (!page) {
ena_increase_stat(&rx_ring->rx_stats.page_alloc_fail, 1,
&rx_ring->syncp);
- return -ENOMEM;
+ return ERR_PTR(-ENOSPC);
}
/* To enable NIC-side port-mirroring, AKA SPAN port,
* we make the buffer readable from the nic as well
*/
- dma = dma_map_page(rx_ring->dev, page, 0, ENA_PAGE_SIZE,
- DMA_BIDIRECTIONAL);
- if (unlikely(dma_mapping_error(rx_ring->dev, dma))) {
+ *dma = dma_map_page(rx_ring->dev, page, 0, ENA_PAGE_SIZE,
+ DMA_BIDIRECTIONAL);
+ if (unlikely(dma_mapping_error(rx_ring->dev, *dma))) {
ena_increase_stat(&rx_ring->rx_stats.dma_mapping_err, 1,
&rx_ring->syncp);
-
__free_page(page);
- return -EIO;
+ return ERR_PTR(-EIO);
}
+
+ return page;
+}
+
+static int ena_alloc_rx_buffer(struct ena_ring *rx_ring,
+ struct ena_rx_buffer *rx_info)
+{
+ int headroom = rx_ring->rx_headroom;
+ struct ena_com_buf *ena_buf;
+ struct page *page;
+ dma_addr_t dma;
+ int tailroom;
+
+ /* restore page offset value in case it has been changed by device */
+ rx_info->page_offset = headroom;
+
+ /* if previous allocated page is not used */
+ if (unlikely(rx_info->page))
+ return 0;
+
+ /* We handle DMA here */
+ page = ena_alloc_map_page(rx_ring, &dma);
+ if (unlikely(IS_ERR(page)))
+ return PTR_ERR(page);
+
netif_dbg(rx_ring->adapter, rx_status, rx_ring->netdev,
"Allocate page %p, rx_info %p\n", page, rx_info);
+ tailroom = SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
+
rx_info->page = page;
ena_buf = &rx_info->ena_buf;
ena_buf->paddr = dma + headroom;
- ena_buf->len = ENA_PAGE_SIZE - headroom;
+ ena_buf->len = ENA_PAGE_SIZE - headroom - tailroom;
return 0;
}
rx_info = &rx_ring->rx_buffer_info[req_id];
- rc = ena_alloc_rx_page(rx_ring, rx_info,
- GFP_ATOMIC | __GFP_COMP);
+ rc = ena_alloc_rx_buffer(rx_ring, rx_info);
if (unlikely(rc < 0)) {
netif_warn(rx_ring->adapter, rx_err, rx_ring->netdev,
"Failed to allocate buffer for rx queue %d\n",
return tx_pkts;
}
-static struct sk_buff *ena_alloc_skb(struct ena_ring *rx_ring, bool frags)
+static struct sk_buff *ena_alloc_skb(struct ena_ring *rx_ring, void *first_frag)
{
struct sk_buff *skb;
- if (frags)
- skb = napi_get_frags(rx_ring->napi);
- else
+ if (!first_frag)
skb = netdev_alloc_skb_ip_align(rx_ring->netdev,
rx_ring->rx_copybreak);
+ else
+ skb = build_skb(first_frag, ENA_PAGE_SIZE);
if (unlikely(!skb)) {
ena_increase_stat(&rx_ring->rx_stats.skb_alloc_fail, 1,
&rx_ring->syncp);
+
netif_dbg(rx_ring->adapter, rx_err, rx_ring->netdev,
- "Failed to allocate skb. frags: %d\n", frags);
+ "Failed to allocate skb. first_frag %s\n",
+ first_frag ? "provided" : "not provided");
return NULL;
}
u32 descs,
u16 *next_to_clean)
{
- struct sk_buff *skb;
struct ena_rx_buffer *rx_info;
u16 len, req_id, buf = 0;
- void *va;
+ struct sk_buff *skb;
+ void *page_addr;
+ u32 page_offset;
+ void *data_addr;
len = ena_bufs[buf].len;
req_id = ena_bufs[buf].req_id;
rx_info, rx_info->page);
/* save virt address of first buffer */
- va = page_address(rx_info->page) + rx_info->page_offset;
+ page_addr = page_address(rx_info->page);
+ page_offset = rx_info->page_offset;
+ data_addr = page_addr + page_offset;
- prefetch(va);
+ prefetch(data_addr);
if (len <= rx_ring->rx_copybreak) {
- skb = ena_alloc_skb(rx_ring, false);
+ skb = ena_alloc_skb(rx_ring, NULL);
if (unlikely(!skb))
return NULL;
dma_unmap_addr(&rx_info->ena_buf, paddr),
len,
DMA_FROM_DEVICE);
- skb_copy_to_linear_data(skb, va, len);
+ skb_copy_to_linear_data(skb, data_addr, len);
dma_sync_single_for_device(rx_ring->dev,
dma_unmap_addr(&rx_info->ena_buf, paddr),
len,
return skb;
}
- skb = ena_alloc_skb(rx_ring, true);
+ ena_unmap_rx_buff(rx_ring, rx_info);
+
+ skb = ena_alloc_skb(rx_ring, page_addr);
if (unlikely(!skb))
return NULL;
- do {
- ena_unmap_rx_buff(rx_ring, rx_info);
-
- skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_info->page,
- rx_info->page_offset, len, ENA_PAGE_SIZE);
+ /* Populate skb's linear part */
+ skb_reserve(skb, page_offset);
+ skb_put(skb, len);
+ skb->protocol = eth_type_trans(skb, rx_ring->netdev);
+ do {
netif_dbg(rx_ring->adapter, rx_status, rx_ring->netdev,
"RX skb updated. len %d. data_len %d\n",
skb->len, skb->data_len);
req_id = ena_bufs[buf].req_id;
rx_info = &rx_ring->rx_buffer_info[req_id];
+
+ ena_unmap_rx_buff(rx_ring, rx_info);
+
+ skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_info->page,
+ rx_info->page_offset, len, ENA_PAGE_SIZE);
+
} while (1);
return skb;
skb_record_rx_queue(skb, rx_ring->qid);
- if (rx_ring->ena_bufs[0].len <= rx_ring->rx_copybreak) {
- total_len += rx_ring->ena_bufs[0].len;
+ if (rx_ring->ena_bufs[0].len <= rx_ring->rx_copybreak)
rx_copybreak_pkt++;
- napi_gro_receive(napi, skb);
- } else {
- total_len += skb->len;
- napi_gro_frags(napi);
- }
+
+ total_len += skb->len;
+
+ napi_gro_receive(napi, skb);
res_budget--;
} while (likely(res_budget));
tx_ring = ena_napi->tx_ring;
rx_ring = ena_napi->rx_ring;
- tx_ring->first_interrupt = ena_napi->first_interrupt;
- rx_ring->first_interrupt = ena_napi->first_interrupt;
-
tx_budget = tx_ring->ring_size / ENA_TX_POLL_BUDGET_DIVIDER;
if (!test_bit(ENA_FLAG_DEV_UP, &tx_ring->adapter->flags) ||
tx_ring->tx_stats.tx_poll++;
u64_stats_update_end(&tx_ring->syncp);
+ tx_ring->tx_stats.last_napi_jiffies = jiffies;
+
return ret;
}
{
struct ena_napi *ena_napi = data;
- ena_napi->first_interrupt = true;
+ /* Used to check HW health */
+ WRITE_ONCE(ena_napi->first_interrupt, true);
WRITE_ONCE(ena_napi->interrupts_masked, true);
smp_wmb(); /* write interrupts_masked before calling napi */
}
}
- if (netif_xmit_stopped(txq) || !netdev_xmit_more()) {
- /* trigger the dma engine. ena_com_write_sq_doorbell()
- * has a mb
+ if (netif_xmit_stopped(txq) || !netdev_xmit_more())
+ /* trigger the dma engine. ena_ring_tx_doorbell()
+ * calls a memory barrier inside it.
*/
- ena_com_write_sq_doorbell(tx_ring->ena_com_io_sq);
- ena_increase_stat(&tx_ring->tx_stats.doorbells, 1,
- &tx_ring->syncp);
- }
+ ena_ring_tx_doorbell(tx_ring);
return NETDEV_TX_OK;
llq_feature_mask = 1 << ENA_ADMIN_LLQ;
if (!(ena_dev->supported_features & llq_feature_mask)) {
- dev_err(&pdev->dev,
+ dev_warn(&pdev->dev,
"LLQ is not supported Fallback to host mode policy.\n");
ena_dev->tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_HOST;
return 0;
static int check_for_rx_interrupt_queue(struct ena_adapter *adapter,
struct ena_ring *rx_ring)
{
- if (likely(rx_ring->first_interrupt))
+ struct ena_napi *ena_napi = container_of(rx_ring->napi, struct ena_napi, napi);
+
+ if (likely(READ_ONCE(ena_napi->first_interrupt)))
return 0;
if (ena_com_cq_empty(rx_ring->ena_com_io_cq))
static int check_missing_comp_in_tx_queue(struct ena_adapter *adapter,
struct ena_ring *tx_ring)
{
+ struct ena_napi *ena_napi = container_of(tx_ring->napi, struct ena_napi, napi);
+ unsigned int time_since_last_napi;
+ unsigned int missing_tx_comp_to;
+ bool is_tx_comp_time_expired;
struct ena_tx_buffer *tx_buf;
unsigned long last_jiffies;
u32 missed_tx = 0;
/* no pending Tx at this location */
continue;
- if (unlikely(!tx_ring->first_interrupt && time_is_before_jiffies(last_jiffies +
- 2 * adapter->missing_tx_completion_to))) {
+ is_tx_comp_time_expired = time_is_before_jiffies(last_jiffies +
+ 2 * adapter->missing_tx_completion_to);
+
+ if (unlikely(!READ_ONCE(ena_napi->first_interrupt) && is_tx_comp_time_expired)) {
/* If after graceful period interrupt is still not
* received, we schedule a reset
*/
return -EIO;
}
- if (unlikely(time_is_before_jiffies(last_jiffies +
- adapter->missing_tx_completion_to))) {
- if (!tx_buf->print_once)
+ is_tx_comp_time_expired = time_is_before_jiffies(last_jiffies +
+ adapter->missing_tx_completion_to);
+
+ if (unlikely(is_tx_comp_time_expired)) {
+ if (!tx_buf->print_once) {
+ time_since_last_napi = jiffies_to_usecs(jiffies - tx_ring->tx_stats.last_napi_jiffies);
+ missing_tx_comp_to = jiffies_to_msecs(adapter->missing_tx_completion_to);
netif_notice(adapter, tx_err, adapter->netdev,
- "Found a Tx that wasn't completed on time, qid %d, index %d.\n",
- tx_ring->qid, i);
+ "Found a Tx that wasn't completed on time, qid %d, index %d. %u usecs have passed since last napi execution. Missing Tx timeout value %u msecs\n",
+ tx_ring->qid, i, time_since_last_napi, missing_tx_comp_to);
+ }
tx_buf->print_once = 1;
missed_tx++;
adapter->ena_dev = ena_dev;
adapter->netdev = netdev;
adapter->pdev = pdev;
- adapter->msg_enable = netif_msg_init(debug, DEFAULT_MSG_ENABLE);
+ adapter->msg_enable = DEFAULT_MSG_ENABLE;
ena_dev->net_device = netdev;
#define ENA_TX_WAKEUP_THRESH (MAX_SKB_FRAGS + 2)
#define ENA_DEFAULT_RX_COPYBREAK (256 - NET_IP_ALIGN)
-/* limit the buffer size to 600 bytes to handle MTU changes from very
- * small to very large, in which case the number of buffers per packet
- * could exceed ENA_PKT_MAX_BUFS
- */
-#define ENA_DEFAULT_MIN_RX_BUFF_ALLOC_SIZE 600
-
#define ENA_MIN_MTU 128
#define ENA_NAME_MAX_LEN 20
};
struct ena_napi {
- struct napi_struct napi ____cacheline_aligned;
+ u8 first_interrupt ____cacheline_aligned;
+ u8 interrupts_masked;
+ struct napi_struct napi;
struct ena_ring *tx_ring;
struct ena_ring *rx_ring;
struct ena_ring *xdp_ring;
- bool first_interrupt;
- bool interrupts_masked;
u32 qid;
struct dim dim;
};
u64 llq_buffer_copy;
u64 missed_tx;
u64 unmask_interrupt;
+ u64 last_napi_jiffies;
};
struct ena_stats_rx {
struct bpf_prog *xdp_bpf_prog;
struct xdp_rxq_info xdp_rxq;
spinlock_t xdp_tx_lock; /* synchronize XDP TX/Redirect traffic */
+ /* Used for rx queues only to point to the xdp tx ring, to
+ * which traffic should be redirected from this rx ring.
+ */
+ struct ena_ring *xdp_ring;
u16 next_to_use;
u16 next_to_clean;
/* The maximum header length the device can handle */
u8 tx_max_header_size;
- bool first_interrupt;
bool disable_meta_caching;
u16 no_interrupt_event_cnt;
ENA_XDP_NO_ENOUGH_QUEUES,
};
-static inline bool ena_xdp_queues_present(struct ena_adapter *adapter)
-{
- return adapter->xdp_first_ring != 0;
-}
-
static inline bool ena_xdp_present(struct ena_adapter *adapter)
{
return !!adapter->xdp_bpf_prog;
Abstract:
- AMD8111 based 10/100 Ethernet Controller Driver.
+ AMD8111 based 10/100 Ethernet Controller Driver.
Environment:
Kernel Mode
Revision History:
- 3.0.0
+ 3.0.0
Initial Revision.
3.0.1
1. Dynamic interrupt coalescing.
Abstract:
- AMD8111 based 10/100 Ethernet Controller driver definitions.
+ AMD8111 based 10/100 Ethernet Controller driver definitions.
Environment:
Kernel Mode
Revision History:
- 3.0.0
+ 3.0.0
Initial Revision.
3.0.1
*/
};
enum coal_mode{
- RX_INTR_COAL,
+ RX_INTR_COAL,
TX_INTR_COAL,
DISABLE_COAL,
ENABLE_COAL,
CHECK_OFFSET(offset);
MEM->tx_head[i].base = offset;
MEM->tx_head[i].flag = TMD1_OWN_HOST;
- MEM->tx_head[i].base_hi = 0;
+ MEM->tx_head[i].base_hi = 0;
MEM->tx_head[i].length = 0;
MEM->tx_head[i].misc = 0;
offset += PKT_BUF_SZ;
dev_kfree_skb(skb);
- return NETDEV_TX_OK;
+ return NETDEV_TX_OK;
}
static void lance_load_multicast(struct net_device *dev)
outw(0x0002, ioaddr+LANCE_ADDR);
/* Only touch autoselect bit. */
outw(inw(ioaddr+LANCE_BUS_IF) | 0x0002, ioaddr+LANCE_BUS_IF);
- }
+ }
if (lance_debug > 1)
printk("%s: lance_open() irq %d dma %d tx/rx rings %#x/%#x init %#x.\n",
* We used to clear the InitDone bit, 0x0100, here but Mark Stockton
* reports that doing so triggers a bug in the '974.
*/
- outw(0x0042, ioaddr+LANCE_DATA);
+ outw(0x0042, ioaddr+LANCE_DATA);
if (lance_debug > 2)
printk("%s: LANCE open after %d ticks, init block %#x csr0 %4.4x.\n",
.vendor_id = ni_vendor,
.cardname = "ni6510",
.config = 0x1,
- },
+ },
{
.id0 = NI65_EB_ID0,
.id1 = NI65_EB_ID1,
.vendor_id = ni_vendor,
.cardname = "ni6510 EtherBlaster",
.config = 0x2,
- },
+ },
{
.id0 = NE2100_ID0,
.id1 = NE2100_ID1,
int __init init_module(void)
{
- dev_ni65 = ni65_probe(-1);
+ dev_ni65 = ni65_probe(-1);
return PTR_ERR_OR_ZERO(dev_ni65);
}
void __exit cleanup_module(void)
{
- unregister_netdev(dev_ni65);
- cleanup_card(dev_ni65);
- free_netdev(dev_ni65);
+ unregister_netdev(dev_ni65);
+ cleanup_card(dev_ni65);
+ free_netdev(dev_ni65);
}
#endif /* MODULE */
if(++ct > 500)
{
pr_err("reset failed, card removed?\n");
- return -1;
+ return -1;
}
udelay(1);
}
ct = 0;
while (mace_read(lp, ioaddr, MACE_IAC) & MACE_IAC_ADDRCHG)
{
- if(++ ct > 500)
- {
+ if(++ ct > 500)
+ {
pr_err("ADDRCHG timeout, card removed?\n");
- return -1;
- }
+ return -1;
+ }
}
/* Set PADR register */
for (i = 0; i < ETH_ALEN; i++)
}
if(mace_init(lp, ioaddr, dev->dev_addr) == -1)
- goto failed;
+ goto failed;
/* The if_port symbol can be set when the module is loaded */
if (if_port <= 2)
struct lance_private {
volatile unsigned short *iobase;
struct lance_memory *mem;
- int new_rx, new_tx; /* The next free ring entry */
+ int new_rx, new_tx; /* The next free ring entry */
int old_tx, old_rx; /* ring entry to be processed */
/* These two must be longs for set_bit() */
long tx_full;
for( i = 0; i < TX_RING_SIZE; i++ ) {
MEM->tx_head[i].base = dvma_vtob(MEM->tx_data[i]);
MEM->tx_head[i].flag = 0;
- MEM->tx_head[i].base_hi =
+ MEM->tx_head[i].base_hi =
(dvma_vtob(MEM->tx_data[i])) >>16;
MEM->tx_head[i].length = 0;
MEM->tx_head[i].misc = 0;
}
AREG = CSR0;
- DPRINTK( 2, ( "%s: lance_start_xmit() called, csr0 %4.4x.\n",
- dev->name, DREG ));
+ DPRINTK( 2, ( "%s: lance_start_xmit() called, csr0 %4.4x.\n",
+ dev->name, DREG ));
#ifdef CONFIG_SUN3X
/* this weirdness doesn't appear on sun3... */
/* Trigger an immediate send poll. */
REGA(CSR0) = CSR0_INEA | CSR0_TDMD | CSR0_STRT;
AREG = CSR0;
- DPRINTK( 2, ( "%s: lance_start_xmit() exiting, csr0 %4.4x.\n",
- dev->name, DREG ));
+ DPRINTK( 2, ( "%s: lance_start_xmit() exiting, csr0 %4.4x.\n",
+ dev->name, DREG ));
dev_kfree_skb(skb);
lp->lock = 0;
config = bmread(dev, RXCFG);
bmwrite(dev, RXCFG, (config & ~RxMACEnable));
config = bmread(dev, TXCFG);
- bmwrite(dev, TXCFG, (config & ~TxMACEnable));
+ bmwrite(dev, TXCFG, (config & ~TxMACEnable));
bmwrite(dev, INTDISABLE, DisableAll); /* disable all intrs */
- /* disable rx and tx dma */
+ /* disable rx and tx dma */
rd->control = cpu_to_le32(DBDMA_CLEAR(RUN|PAUSE|FLUSH|WAKE)); /* clear run bit */
td->control = cpu_to_le32(DBDMA_CLEAR(RUN|PAUSE|FLUSH|WAKE)); /* clear run bit */
- /* free some skb's */
- for (i=0; i<N_RX_RING; i++) {
- if (bp->rx_bufs[i] != NULL) {
- dev_kfree_skb(bp->rx_bufs[i]);
- bp->rx_bufs[i] = NULL;
- }
- }
- for (i = 0; i<N_TX_RING; i++) {
+ /* free some skb's */
+ for (i=0; i<N_RX_RING; i++) {
+ if (bp->rx_bufs[i] != NULL) {
+ dev_kfree_skb(bp->rx_bufs[i]);
+ bp->rx_bufs[i] = NULL;
+ }
+ }
+ for (i = 0; i<N_TX_RING; i++) {
if (bp->tx_bufs[i] != NULL) {
dev_kfree_skb(bp->tx_bufs[i]);
bp->tx_bufs[i] = NULL;
}
}
}
- pmac_call_feature(PMAC_FTR_BMAC_ENABLE, macio_get_of_node(bp->mdev), 0, 0);
+ pmac_call_feature(PMAC_FTR_BMAC_ENABLE, macio_get_of_node(bp->mdev), 0, 0);
return 0;
}
bmac_reset_and_enable(dev);
enable_irq(dev->irq);
- enable_irq(bp->tx_dma_intr);
- enable_irq(bp->rx_dma_intr);
- netif_device_attach(dev);
+ enable_irq(bp->tx_dma_intr);
+ enable_irq(bp->rx_dma_intr);
+ netif_device_attach(dev);
return 0;
}
unregister_netdev(dev);
- free_irq(dev->irq, dev);
+ free_irq(dev->irq, dev);
free_irq(bp->tx_dma_intr, dev);
free_irq(bp->rx_dma_intr, dev);
out_8(&mb->iac, 0);
if (mp->port_aaui)
- out_8(&mb->plscc, PORTSEL_AUI + ENPLSIO);
+ out_8(&mb->plscc, PORTSEL_AUI + ENPLSIO);
else
- out_8(&mb->plscc, PORTSEL_GPSI + ENPLSIO);
+ out_8(&mb->plscc, PORTSEL_GPSI + ENPLSIO);
}
static void __mace_set_address(struct net_device *dev, void *addr)
/* load up the hardware address */
if (mp->chipid == BROKEN_ADDRCHG_REV)
- out_8(&mb->iac, PHYADDR);
+ out_8(&mb->iac, PHYADDR);
else {
- out_8(&mb->iac, ADDRCHG | PHYADDR);
+ out_8(&mb->iac, ADDRCHG | PHYADDR);
while ((in_8(&mb->iac) & ADDRCHG) != 0)
;
}
// SPDX-License-Identifier: GPL-2.0-or-later
-/**
+/*
* emac-rockchip.c - Rockchip EMAC specific glue layer
*
* Copyright (C) 2014 Romain Perier <romain.perier@gmail.com>
if (reconfig_phy) {
err = alx_setup_speed_duplex(hw, hw->adv_cfg, fc);
- if (err)
+ if (err) {
+ mutex_unlock(&alx->mtx);
return err;
+ }
}
/* flow control on mac */
err = register_netdev(netdev);
if (err) {
dev_err(&pdev->dev, "register netdevice failed\n");
- goto out_unlock;
+ goto out_unmap;
}
netdev_info(netdev,
free_netdev(netdev);
out_pci_release:
pci_release_mem_regions(pdev);
+ pci_disable_pcie_error_reporting(pdev);
out_pci_disable:
pci_disable_device(pdev);
return err;
#define AT_MAX_RECEIVE_QUEUE 4
#define AT_DEF_RECEIVE_QUEUE 1
-#define AT_MAX_TRANSMIT_QUEUE 2
+#define AT_MAX_TRANSMIT_QUEUE 4
#define AT_DMA_HI_ADDR_MASK 0xffffffff00000000ULL
#define AT_DMA_LO_ADDR_MASK 0x00000000ffffffffULL
athr_mt,
};
-enum atl1c_trans_queue {
- atl1c_trans_normal = 0,
- atl1c_trans_high = 1
-};
-
struct atl1c_hw_stats {
/* rx */
unsigned long rx_ok; /* The number of good packet received. */
/* transimit packet descriptor (tpd) ring */
struct atl1c_tpd_ring {
+ struct atl1c_adapter *adapter;
void *desc; /* descriptor ring virtual address */
dma_addr_t dma; /* descriptor ring physical address */
+ u16 num;
u16 size; /* descriptor ring length in bytes */
u16 count; /* number of descriptors in the ring */
u16 next_to_use;
atomic_t next_to_clean;
struct atl1c_buffer *buffer_info;
+ struct napi_struct napi;
};
/* receive free descriptor (rfd) ring */
/* receive return descriptor (rrd) ring */
struct atl1c_rrd_ring {
+ struct atl1c_adapter *adapter;
void *desc; /* descriptor ring virtual address */
dma_addr_t dma; /* descriptor ring physical address */
+ u16 num;
u16 size; /* descriptor ring length in bytes */
u16 count; /* number of descriptors in the ring */
u16 next_to_use;
u16 next_to_clean;
+ struct napi_struct napi;
+ struct page *rx_page;
+ unsigned int rx_page_offset;
};
/* board specific private data structure */
struct atl1c_adapter {
struct net_device *netdev;
struct pci_dev *pdev;
- struct napi_struct napi;
- struct napi_struct tx_napi;
- struct page *rx_page;
- unsigned int rx_page_offset;
unsigned int rx_frag_size;
struct atl1c_hw hw;
struct atl1c_hw_stats hw_stats;
struct mii_if_info mii; /* MII interface info */
u16 rx_buffer_len;
+ unsigned int tx_queue_count;
+ unsigned int rx_queue_count;
unsigned long flags;
#define __AT_TESTING 0x0001
/* All Descriptor memory */
struct atl1c_ring_header ring_header;
struct atl1c_tpd_ring tpd_ring[AT_MAX_TRANSMIT_QUEUE];
- struct atl1c_rfd_ring rfd_ring;
- struct atl1c_rrd_ring rrd_ring;
+ struct atl1c_rfd_ring rfd_ring[AT_MAX_RECEIVE_QUEUE];
+ struct atl1c_rrd_ring rrd_ring[AT_MAX_RECEIVE_QUEUE];
u32 bd_number; /* board number;*/
};
#define REG_RX_BASE_ADDR_HI 0x1540
#define REG_TX_BASE_ADDR_HI 0x1544
#define REG_RFD0_HEAD_ADDR_LO 0x1550
+#define REG_RFD1_HEAD_ADDR_LO 0x1554
+#define REG_RFD2_HEAD_ADDR_LO 0x1558
+#define REG_RFD3_HEAD_ADDR_LO 0x155C
#define REG_RFD_RING_SIZE 0x1560
#define RFD_RING_SIZE_MASK 0x0FFF
#define REG_RX_BUF_SIZE 0x1564
#define RX_BUF_SIZE_MASK 0xFFFF
#define REG_RRD0_HEAD_ADDR_LO 0x1568
+#define REG_RRD1_HEAD_ADDR_LO 0x156C
+#define REG_RRD2_HEAD_ADDR_LO 0x1570
+#define REG_RRD3_HEAD_ADDR_LO 0x1574
#define REG_RRD_RING_SIZE 0x1578
#define RRD_RING_SIZE_MASK 0x0FFF
#define REG_TPD_PRI1_ADDR_LO 0x157C
#define REG_TPD_PRI0_ADDR_LO 0x1580
+#define REG_TPD_PRI2_ADDR_LO 0x1F10
+#define REG_TPD_PRI3_ADDR_LO 0x1F14
+
#define REG_TPD_RING_SIZE 0x1584
#define TPD_RING_SIZE_MASK 0xFFFF
/* Mail box */
#define MB_RFDX_PROD_IDX_MASK 0xFFFF
#define REG_MB_RFD0_PROD_IDX 0x15E0
+#define REG_MB_RFD1_PROD_IDX 0x15E4
+#define REG_MB_RFD2_PROD_IDX 0x15E8
+#define REG_MB_RFD3_PROD_IDX 0x15EC
#define REG_TPD_PRI1_PIDX 0x15F0 /* 16bit,hi-tpd producer idx */
#define REG_TPD_PRI0_PIDX 0x15F2 /* 16bit,lo-tpd producer idx */
#define REG_TPD_PRI1_CIDX 0x15F4 /* 16bit,hi-tpd consumer idx */
#define REG_TPD_PRI0_CIDX 0x15F6 /* 16bit,lo-tpd consumer idx */
+#define REG_TPD_PRI3_PIDX 0x1F18
+#define REG_TPD_PRI2_PIDX 0x1F1A
+#define REG_TPD_PRI3_CIDX 0x1F1C
+#define REG_TPD_PRI2_CIDX 0x1F1E
+
#define REG_MB_RFD01_CONS_IDX 0x15F8
#define MB_RFD0_CONS_IDX_MASK 0x0000FFFF
#define MB_RFD1_CONS_IDX_MASK 0xFFFF0000
+#define REG_MB_RFD23_CONS_IDX 0x15FC
+#define MB_RFD2_CONS_IDX_MASK 0x0000FFFF
+#define MB_RFD3_CONS_IDX_MASK 0xFFFF0000
/* Interrupt Status Register */
#define REG_ISR 0x1600
/* GPHY low power state interrupt */
#define ISR_GPHY_LPW 0x00002000
#define ISR_TXQ_TO_RST 0x00004000
-#define ISR_TX_PKT 0x00008000
+#define ISR_TX_PKT_0 0x00008000
#define ISR_RX_PKT_0 0x00010000
#define ISR_RX_PKT_1 0x00020000
#define ISR_RX_PKT_2 0x00040000
#define ISR_NFERR_DETECTED 0x01000000
#define ISR_CERR_DETECTED 0x02000000
#define ISR_PHY_LINKDOWN 0x04000000
+#define ISR_TX_PKT_1 0x10000000
+#define ISR_TX_PKT_2 0x20000000
+#define ISR_TX_PKT_3 0x40000000
#define ISR_DIS_INT 0x80000000
/* Interrupt Mask Register */
ISR_TXQ_TO_RST |\
ISR_DMAW_TO_RST |\
ISR_GPHY |\
- ISR_TX_PKT |\
- ISR_RX_PKT_0 |\
ISR_GPHY_LPW |\
ISR_PHY_LINKDOWN)
+#define ISR_TX_PKT ( \
+ ISR_TX_PKT_0 | \
+ ISR_TX_PKT_1 | \
+ ISR_TX_PKT_2 | \
+ ISR_TX_PKT_3)
+
#define ISR_RX_PKT (\
ISR_RX_PKT_0 |\
ISR_RX_PKT_1 |\
#define REG_MT_VERSION 0x1F0C
#define MT_MAGIC 0xaabb1234
+#define MT_MODE_4Q BIT(0)
#define L1D_MPW_PHYID1 0xD01C /* V7 */
#define L1D_MPW_PHYID2 0xD01D /* V1-V6 */
MODULE_DESCRIPTION("Qualcomm Atheros 100/1000M Ethernet Network Driver");
MODULE_LICENSE("GPL");
+struct atl1c_qregs {
+ u16 tpd_addr_lo;
+ u16 tpd_prod;
+ u16 tpd_cons;
+ u16 rfd_addr_lo;
+ u16 rrd_addr_lo;
+ u16 rfd_prod;
+ u32 tx_isr;
+ u32 rx_isr;
+};
+
+static struct atl1c_qregs atl1c_qregs[AT_MAX_TRANSMIT_QUEUE] = {
+ {
+ REG_TPD_PRI0_ADDR_LO, REG_TPD_PRI0_PIDX, REG_TPD_PRI0_CIDX,
+ REG_RFD0_HEAD_ADDR_LO, REG_RRD0_HEAD_ADDR_LO,
+ REG_MB_RFD0_PROD_IDX, ISR_TX_PKT_0, ISR_RX_PKT_0
+ },
+ {
+ REG_TPD_PRI1_ADDR_LO, REG_TPD_PRI1_PIDX, REG_TPD_PRI1_CIDX,
+ REG_RFD1_HEAD_ADDR_LO, REG_RRD1_HEAD_ADDR_LO,
+ REG_MB_RFD1_PROD_IDX, ISR_TX_PKT_1, ISR_RX_PKT_1
+ },
+ {
+ REG_TPD_PRI2_ADDR_LO, REG_TPD_PRI2_PIDX, REG_TPD_PRI2_CIDX,
+ REG_RFD2_HEAD_ADDR_LO, REG_RRD2_HEAD_ADDR_LO,
+ REG_MB_RFD2_PROD_IDX, ISR_TX_PKT_2, ISR_RX_PKT_2
+ },
+ {
+ REG_TPD_PRI3_ADDR_LO, REG_TPD_PRI3_PIDX, REG_TPD_PRI3_CIDX,
+ REG_RFD3_HEAD_ADDR_LO, REG_RRD3_HEAD_ADDR_LO,
+ REG_MB_RFD3_PROD_IDX, ISR_TX_PKT_3, ISR_RX_PKT_3
+ },
+};
+
static int atl1c_stop_mac(struct atl1c_hw *hw);
static void atl1c_disable_l0s_l1(struct atl1c_hw *hw);
static void atl1c_set_aspm(struct atl1c_hw *hw, u16 link_speed);
static void atl1c_start_mac(struct atl1c_adapter *adapter);
-static void atl1c_clean_rx_irq(struct atl1c_adapter *adapter,
- int *work_done, int work_to_do);
static int atl1c_up(struct atl1c_adapter *adapter);
static void atl1c_down(struct atl1c_adapter *adapter);
static int atl1c_reset_mac(struct atl1c_hw *hw);
static void atl1c_reset_dma_ring(struct atl1c_adapter *adapter);
static int atl1c_configure(struct atl1c_adapter *adapter);
-static int atl1c_alloc_rx_buffer(struct atl1c_adapter *adapter, bool napi_mode);
+static int atl1c_alloc_rx_buffer(struct atl1c_adapter *adapter, u32 queue,
+ bool napi_mode);
static const u32 atl1c_default_msg = NETIF_MSG_DRV | NETIF_MSG_PROBE |
}
/**
- * atl1c_set_mac - Change the Ethernet Address of the NIC
+ * atl1c_set_mac_addr - Change the Ethernet Address of the NIC
* @netdev: network interface device structure
* @p: pointer to an address structure
*
return 0;
}
-static void atl1c_set_mac_type(struct atl1c_hw *hw)
+static enum atl1c_nic_type atl1c_get_mac_type(struct pci_dev *pdev,
+ u8 __iomem *hw_addr)
{
- u32 magic;
- switch (hw->device_id) {
+ switch (pdev->device) {
case PCI_DEVICE_ID_ATTANSIC_L2C:
- hw->nic_type = athr_l2c;
- break;
+ return athr_l2c;
case PCI_DEVICE_ID_ATTANSIC_L1C:
- hw->nic_type = athr_l1c;
- break;
+ return athr_l1c;
case PCI_DEVICE_ID_ATHEROS_L2C_B:
- hw->nic_type = athr_l2c_b;
- break;
+ return athr_l2c_b;
case PCI_DEVICE_ID_ATHEROS_L2C_B2:
- hw->nic_type = athr_l2c_b2;
- break;
+ return athr_l2c_b2;
case PCI_DEVICE_ID_ATHEROS_L1D:
- hw->nic_type = athr_l1d;
- break;
+ return athr_l1d;
case PCI_DEVICE_ID_ATHEROS_L1D_2_0:
- hw->nic_type = athr_l1d_2;
- AT_READ_REG(hw, REG_MT_MAGIC, &magic);
- if (magic == MT_MAGIC)
- hw->nic_type = athr_mt;
- break;
+ if (readl(hw_addr + REG_MT_MAGIC) == MT_MAGIC)
+ return athr_mt;
+ return athr_l1d_2;
default:
- break;
+ return athr_l1c;
}
}
{
u32 link_ctrl_data;
- atl1c_set_mac_type(hw);
AT_READ_REG(hw, REG_LINK_CTRL, &link_ctrl_data);
hw->ctrl_flags = ATL1C_INTR_MODRT_ENABLE |
struct atl1c_hw *hw = &adapter->hw;
struct pci_dev *pdev = adapter->pdev;
u32 revision;
-
+ int i;
adapter->wol = 0;
device_set_wakeup_enable(&pdev->dev, false);
adapter->link_speed = SPEED_0;
adapter->link_duplex = FULL_DUPLEX;
adapter->tpd_ring[0].count = 1024;
- adapter->rfd_ring.count = 512;
+ adapter->rfd_ring[0].count = 512;
hw->vendor_id = pdev->vendor;
hw->device_id = pdev->device;
atl1c_patch_assign(hw);
hw->intr_mask = IMR_NORMAL_MASK;
+ for (i = 0; i < adapter->tx_queue_count; ++i)
+ hw->intr_mask |= atl1c_qregs[i].tx_isr;
+ for (i = 0; i < adapter->rx_queue_count; ++i)
+ hw->intr_mask |= atl1c_qregs[i].rx_isr;
hw->phy_configured = false;
hw->preamble_len = 7;
hw->max_frame_size = adapter->netdev->mtu;
/**
* atl1c_clean_tx_ring - Free Tx-skb
* @adapter: board private structure
- * @type: type of transmit queue
+ * @queue: idx of transmit queue
*/
static void atl1c_clean_tx_ring(struct atl1c_adapter *adapter,
- enum atl1c_trans_queue type)
+ u32 queue)
{
- struct atl1c_tpd_ring *tpd_ring = &adapter->tpd_ring[type];
+ struct atl1c_tpd_ring *tpd_ring = &adapter->tpd_ring[queue];
struct atl1c_buffer *buffer_info;
struct pci_dev *pdev = adapter->pdev;
u16 index, ring_count;
/**
* atl1c_clean_rx_ring - Free rx-reservation skbs
* @adapter: board private structure
+ * @queue: idx of transmit queue
*/
-static void atl1c_clean_rx_ring(struct atl1c_adapter *adapter)
+static void atl1c_clean_rx_ring(struct atl1c_adapter *adapter, u32 queue)
{
- struct atl1c_rfd_ring *rfd_ring = &adapter->rfd_ring;
- struct atl1c_rrd_ring *rrd_ring = &adapter->rrd_ring;
+ struct atl1c_rfd_ring *rfd_ring = &adapter->rfd_ring[queue];
+ struct atl1c_rrd_ring *rrd_ring = &adapter->rrd_ring[queue];
struct atl1c_buffer *buffer_info;
struct pci_dev *pdev = adapter->pdev;
int j;
static void atl1c_init_ring_ptrs(struct atl1c_adapter *adapter)
{
struct atl1c_tpd_ring *tpd_ring = adapter->tpd_ring;
- struct atl1c_rfd_ring *rfd_ring = &adapter->rfd_ring;
- struct atl1c_rrd_ring *rrd_ring = &adapter->rrd_ring;
+ struct atl1c_rfd_ring *rfd_ring = adapter->rfd_ring;
+ struct atl1c_rrd_ring *rrd_ring = adapter->rrd_ring;
struct atl1c_buffer *buffer_info;
int i, j;
- for (i = 0; i < AT_MAX_TRANSMIT_QUEUE; i++) {
+ for (i = 0; i < adapter->tx_queue_count; i++) {
tpd_ring[i].next_to_use = 0;
atomic_set(&tpd_ring[i].next_to_clean, 0);
buffer_info = tpd_ring[i].buffer_info;
for (j = 0; j < tpd_ring->count; j++)
ATL1C_SET_BUFFER_STATE(&buffer_info[i],
- ATL1C_BUFFER_FREE);
- }
- rfd_ring->next_to_use = 0;
- rfd_ring->next_to_clean = 0;
- rrd_ring->next_to_use = 0;
- rrd_ring->next_to_clean = 0;
- for (j = 0; j < rfd_ring->count; j++) {
- buffer_info = &rfd_ring->buffer_info[j];
- ATL1C_SET_BUFFER_STATE(buffer_info, ATL1C_BUFFER_FREE);
+ ATL1C_BUFFER_FREE);
+ }
+ for (i = 0; i < adapter->rx_queue_count; i++) {
+ rfd_ring[i].next_to_use = 0;
+ rfd_ring[i].next_to_clean = 0;
+ rrd_ring[i].next_to_use = 0;
+ rrd_ring[i].next_to_clean = 0;
+ for (j = 0; j < rfd_ring[i].count; j++) {
+ buffer_info = &rfd_ring[i].buffer_info[j];
+ ATL1C_SET_BUFFER_STATE(buffer_info, ATL1C_BUFFER_FREE);
+ }
}
}
static void atl1c_free_ring_resources(struct atl1c_adapter *adapter)
{
struct pci_dev *pdev = adapter->pdev;
+ int i;
dma_free_coherent(&pdev->dev, adapter->ring_header.size,
adapter->ring_header.desc, adapter->ring_header.dma);
adapter->ring_header.desc = NULL;
/* Note: just free tdp_ring.buffer_info,
- * it contain rfd_ring.buffer_info, do not double free */
+ * it contain rfd_ring.buffer_info, do not double free
+ */
if (adapter->tpd_ring[0].buffer_info) {
kfree(adapter->tpd_ring[0].buffer_info);
adapter->tpd_ring[0].buffer_info = NULL;
}
- if (adapter->rx_page) {
- put_page(adapter->rx_page);
- adapter->rx_page = NULL;
+ for (i = 0; i < adapter->rx_queue_count; ++i) {
+ if (adapter->rrd_ring[i].rx_page) {
+ put_page(adapter->rrd_ring[i].rx_page);
+ adapter->rrd_ring[i].rx_page = NULL;
+ }
}
}
/**
- * atl1c_setup_mem_resources - allocate Tx / RX descriptor resources
+ * atl1c_setup_ring_resources - allocate Tx / RX descriptor resources
* @adapter: board private structure
*
* Return 0 on success, negative on failure
{
struct pci_dev *pdev = adapter->pdev;
struct atl1c_tpd_ring *tpd_ring = adapter->tpd_ring;
- struct atl1c_rfd_ring *rfd_ring = &adapter->rfd_ring;
- struct atl1c_rrd_ring *rrd_ring = &adapter->rrd_ring;
+ struct atl1c_rfd_ring *rfd_ring = adapter->rfd_ring;
+ struct atl1c_rrd_ring *rrd_ring = adapter->rrd_ring;
struct atl1c_ring_header *ring_header = &adapter->ring_header;
+ int tqc = adapter->tx_queue_count;
+ int rqc = adapter->rx_queue_count;
int size;
int i;
int count = 0;
- int rx_desc_count = 0;
u32 offset = 0;
- rrd_ring->count = rfd_ring->count;
- for (i = 1; i < AT_MAX_TRANSMIT_QUEUE; i++)
+ /* Even though only one tpd queue is actually used, the "high"
+ * priority tpd queue also gets initialized
+ */
+ if (tqc == 1)
+ tqc = 2;
+
+ for (i = 1; i < tqc; i++)
tpd_ring[i].count = tpd_ring[0].count;
- /* 2 tpd queue, one high priority queue,
- * another normal priority queue */
- size = sizeof(struct atl1c_buffer) * (tpd_ring->count * 2 +
- rfd_ring->count);
+ size = sizeof(struct atl1c_buffer) * (tpd_ring->count * tqc +
+ rfd_ring->count * rqc);
tpd_ring->buffer_info = kzalloc(size, GFP_KERNEL);
if (unlikely(!tpd_ring->buffer_info))
goto err_nomem;
- for (i = 0; i < AT_MAX_TRANSMIT_QUEUE; i++) {
- tpd_ring[i].buffer_info =
- (tpd_ring->buffer_info + count);
+ for (i = 0; i < tqc; i++) {
+ tpd_ring[i].adapter = adapter;
+ tpd_ring[i].num = i;
+ tpd_ring[i].buffer_info = (tpd_ring->buffer_info + count);
count += tpd_ring[i].count;
}
- rfd_ring->buffer_info =
- (tpd_ring->buffer_info + count);
- count += rfd_ring->count;
- rx_desc_count += rfd_ring->count;
+ for (i = 0; i < rqc; i++) {
+ rrd_ring[i].adapter = adapter;
+ rrd_ring[i].num = i;
+ rrd_ring[i].count = rfd_ring[0].count;
+ rfd_ring[i].count = rfd_ring[0].count;
+ rfd_ring[i].buffer_info = (tpd_ring->buffer_info + count);
+ count += rfd_ring->count;
+ }
/*
* real ring DMA buffer
* additional bytes tacked onto the end.
*/
ring_header->size = size =
- sizeof(struct atl1c_tpd_desc) * tpd_ring->count * 2 +
- sizeof(struct atl1c_rx_free_desc) * rx_desc_count +
- sizeof(struct atl1c_recv_ret_status) * rx_desc_count +
+ sizeof(struct atl1c_tpd_desc) * tpd_ring->count * tqc +
+ sizeof(struct atl1c_rx_free_desc) * rfd_ring->count * rqc +
+ sizeof(struct atl1c_recv_ret_status) * rfd_ring->count * rqc +
8 * 4;
ring_header->desc = dma_alloc_coherent(&pdev->dev, ring_header->size,
tpd_ring[0].dma = roundup(ring_header->dma, 8);
offset = tpd_ring[0].dma - ring_header->dma;
- for (i = 0; i < AT_MAX_TRANSMIT_QUEUE; i++) {
+ for (i = 0; i < tqc; i++) {
tpd_ring[i].dma = ring_header->dma + offset;
- tpd_ring[i].desc = (u8 *) ring_header->desc + offset;
+ tpd_ring[i].desc = (u8 *)ring_header->desc + offset;
tpd_ring[i].size =
sizeof(struct atl1c_tpd_desc) * tpd_ring[i].count;
offset += roundup(tpd_ring[i].size, 8);
}
- /* init RFD ring */
- rfd_ring->dma = ring_header->dma + offset;
- rfd_ring->desc = (u8 *) ring_header->desc + offset;
- rfd_ring->size = sizeof(struct atl1c_rx_free_desc) * rfd_ring->count;
- offset += roundup(rfd_ring->size, 8);
+ for (i = 0; i < rqc; i++) {
+ /* init RFD ring */
+ rfd_ring[i].dma = ring_header->dma + offset;
+ rfd_ring[i].desc = (u8 *)ring_header->desc + offset;
+ rfd_ring[i].size = sizeof(struct atl1c_rx_free_desc) *
+ rfd_ring[i].count;
+ offset += roundup(rfd_ring[i].size, 8);
- /* init RRD ring */
- rrd_ring->dma = ring_header->dma + offset;
- rrd_ring->desc = (u8 *) ring_header->desc + offset;
- rrd_ring->size = sizeof(struct atl1c_recv_ret_status) *
- rrd_ring->count;
- offset += roundup(rrd_ring->size, 8);
+ /* init RRD ring */
+ rrd_ring[i].dma = ring_header->dma + offset;
+ rrd_ring[i].desc = (u8 *)ring_header->desc + offset;
+ rrd_ring[i].size = sizeof(struct atl1c_recv_ret_status) *
+ rrd_ring[i].count;
+ offset += roundup(rrd_ring[i].size, 8);
+ }
return 0;
static void atl1c_configure_des_ring(struct atl1c_adapter *adapter)
{
struct atl1c_hw *hw = &adapter->hw;
- struct atl1c_rfd_ring *rfd_ring = &adapter->rfd_ring;
- struct atl1c_rrd_ring *rrd_ring = &adapter->rrd_ring;
- struct atl1c_tpd_ring *tpd_ring = (struct atl1c_tpd_ring *)
- adapter->tpd_ring;
+ struct atl1c_rfd_ring *rfd_ring = adapter->rfd_ring;
+ struct atl1c_rrd_ring *rrd_ring = adapter->rrd_ring;
+ struct atl1c_tpd_ring *tpd_ring = adapter->tpd_ring;
+ int i;
+ int tx_queue_count = adapter->tx_queue_count;
+
+ if (tx_queue_count == 1)
+ tx_queue_count = 2;
/* TPD */
AT_WRITE_REG(hw, REG_TX_BASE_ADDR_HI,
- (u32)((tpd_ring[atl1c_trans_normal].dma &
- AT_DMA_HI_ADDR_MASK) >> 32));
+ (u32)((tpd_ring[0].dma & AT_DMA_HI_ADDR_MASK) >> 32));
/* just enable normal priority TX queue */
- AT_WRITE_REG(hw, REG_TPD_PRI0_ADDR_LO,
- (u32)(tpd_ring[atl1c_trans_normal].dma &
- AT_DMA_LO_ADDR_MASK));
- AT_WRITE_REG(hw, REG_TPD_PRI1_ADDR_LO,
- (u32)(tpd_ring[atl1c_trans_high].dma &
- AT_DMA_LO_ADDR_MASK));
+ for (i = 0; i < tx_queue_count; i++) {
+ AT_WRITE_REG(hw, atl1c_qregs[i].tpd_addr_lo,
+ (u32)(tpd_ring[i].dma & AT_DMA_LO_ADDR_MASK));
+ }
AT_WRITE_REG(hw, REG_TPD_RING_SIZE,
(u32)(tpd_ring[0].count & TPD_RING_SIZE_MASK));
/* RFD */
AT_WRITE_REG(hw, REG_RX_BASE_ADDR_HI,
- (u32)((rfd_ring->dma & AT_DMA_HI_ADDR_MASK) >> 32));
- AT_WRITE_REG(hw, REG_RFD0_HEAD_ADDR_LO,
- (u32)(rfd_ring->dma & AT_DMA_LO_ADDR_MASK));
+ (u32)((rfd_ring->dma & AT_DMA_HI_ADDR_MASK) >> 32));
+ for (i = 0; i < adapter->rx_queue_count; i++) {
+ AT_WRITE_REG(hw, atl1c_qregs[i].rfd_addr_lo,
+ (u32)(rfd_ring[i].dma & AT_DMA_LO_ADDR_MASK));
+ }
AT_WRITE_REG(hw, REG_RFD_RING_SIZE,
rfd_ring->count & RFD_RING_SIZE_MASK);
adapter->rx_buffer_len & RX_BUF_SIZE_MASK);
/* RRD */
- AT_WRITE_REG(hw, REG_RRD0_HEAD_ADDR_LO,
- (u32)(rrd_ring->dma & AT_DMA_LO_ADDR_MASK));
+ for (i = 0; i < adapter->rx_queue_count; i++) {
+ AT_WRITE_REG(hw, atl1c_qregs[i].rrd_addr_lo,
+ (u32)(rrd_ring[i].dma & AT_DMA_LO_ADDR_MASK));
+ }
AT_WRITE_REG(hw, REG_RRD_RING_SIZE,
(rrd_ring->count & RRD_RING_SIZE_MASK));
}
/**
- * atl1c_configure - Configure Transmit&Receive Unit after Reset
+ * atl1c_configure_mac - Configure Transmit&Receive Unit after Reset
* @adapter: board private structure
*
* Configure the Tx /Rx unit of the MAC after a reset.
{
struct net_device *netdev = adapter->netdev;
int num;
+ int i;
+
+ if (adapter->hw.nic_type == athr_mt) {
+ u32 mode;
+
+ AT_READ_REG(&adapter->hw, REG_MT_MODE, &mode);
+ if (adapter->rx_queue_count == 4)
+ mode |= MT_MODE_4Q;
+ else
+ mode &= ~MT_MODE_4Q;
+ AT_WRITE_REG(&adapter->hw, REG_MT_MODE, mode);
+ }
atl1c_init_ring_ptrs(adapter);
atl1c_set_multi(netdev);
atl1c_restore_vlan(adapter);
- num = atl1c_alloc_rx_buffer(adapter, false);
- if (unlikely(num == 0))
- return -ENOMEM;
+ for (i = 0; i < adapter->rx_queue_count; ++i) {
+ num = atl1c_alloc_rx_buffer(adapter, i, false);
+ if (unlikely(num == 0))
+ return -ENOMEM;
+ }
if (atl1c_configure_mac(adapter))
return -EIO;
static int atl1c_clean_tx(struct napi_struct *napi, int budget)
{
- struct atl1c_adapter *adapter =
- container_of(napi, struct atl1c_adapter, tx_napi);
- struct atl1c_tpd_ring *tpd_ring = &adapter->tpd_ring[atl1c_trans_normal];
+ struct atl1c_tpd_ring *tpd_ring =
+ container_of(napi, struct atl1c_tpd_ring, napi);
+ struct atl1c_adapter *adapter = tpd_ring->adapter;
+ struct netdev_queue *txq =
+ netdev_get_tx_queue(napi->dev, tpd_ring->num);
struct atl1c_buffer *buffer_info;
struct pci_dev *pdev = adapter->pdev;
u16 next_to_clean = atomic_read(&tpd_ring->next_to_clean);
unsigned int total_bytes = 0, total_packets = 0;
unsigned long flags;
- AT_READ_REGW(&adapter->hw, REG_TPD_PRI0_CIDX, &hw_next_to_clean);
+ AT_READ_REGW(&adapter->hw, atl1c_qregs[tpd_ring->num].tpd_cons,
+ &hw_next_to_clean);
while (next_to_clean != hw_next_to_clean) {
buffer_info = &tpd_ring->buffer_info[next_to_clean];
atomic_set(&tpd_ring->next_to_clean, next_to_clean);
}
- netdev_completed_queue(adapter->netdev, total_packets, total_bytes);
+ netdev_tx_completed_queue(txq, total_packets, total_bytes);
- if (netif_queue_stopped(adapter->netdev) &&
- netif_carrier_ok(adapter->netdev)) {
- netif_wake_queue(adapter->netdev);
- }
+ if (netif_tx_queue_stopped(txq) && netif_carrier_ok(adapter->netdev))
+ netif_tx_wake_queue(txq);
if (total_packets < budget) {
napi_complete_done(napi, total_packets);
spin_lock_irqsave(&adapter->hw.intr_mask_lock, flags);
- adapter->hw.intr_mask |= ISR_TX_PKT;
+ adapter->hw.intr_mask |= atl1c_qregs[tpd_ring->num].tx_isr;
AT_WRITE_REG(&adapter->hw, REG_IMR, adapter->hw.intr_mask);
spin_unlock_irqrestore(&adapter->hw.intr_mask_lock, flags);
return total_packets;
return budget;
}
+static void atl1c_intr_rx_tx(struct atl1c_adapter *adapter, u32 status)
+{
+ struct atl1c_hw *hw = &adapter->hw;
+ u32 intr_mask;
+ int i;
+
+ spin_lock(&hw->intr_mask_lock);
+ intr_mask = hw->intr_mask;
+ for (i = 0; i < adapter->rx_queue_count; ++i) {
+ if (!(status & atl1c_qregs[i].rx_isr))
+ continue;
+ if (napi_schedule_prep(&adapter->rrd_ring[i].napi)) {
+ intr_mask &= ~atl1c_qregs[i].rx_isr;
+ __napi_schedule(&adapter->rrd_ring[i].napi);
+ }
+ }
+ for (i = 0; i < adapter->tx_queue_count; ++i) {
+ if (!(status & atl1c_qregs[i].tx_isr))
+ continue;
+ if (napi_schedule_prep(&adapter->tpd_ring[i].napi)) {
+ intr_mask &= ~atl1c_qregs[i].tx_isr;
+ __napi_schedule(&adapter->tpd_ring[i].napi);
+ }
+ }
+
+ if (hw->intr_mask != intr_mask) {
+ hw->intr_mask = intr_mask;
+ AT_WRITE_REG(hw, REG_IMR, hw->intr_mask);
+ }
+ spin_unlock(&hw->intr_mask_lock);
+}
+
/**
* atl1c_intr - Interrupt Handler
* @irq: interrupt number
atl1c_clear_phy_int(adapter);
/* Ack ISR */
AT_WRITE_REG(hw, REG_ISR, status | ISR_DIS_INT);
- if (status & ISR_RX_PKT) {
- if (likely(napi_schedule_prep(&adapter->napi))) {
- spin_lock(&hw->intr_mask_lock);
- hw->intr_mask &= ~ISR_RX_PKT;
- AT_WRITE_REG(hw, REG_IMR, hw->intr_mask);
- spin_unlock(&hw->intr_mask_lock);
- __napi_schedule(&adapter->napi);
- }
- }
- if (status & ISR_TX_PKT) {
- if (napi_schedule_prep(&adapter->tx_napi)) {
- spin_lock(&hw->intr_mask_lock);
- hw->intr_mask &= ~ISR_TX_PKT;
- AT_WRITE_REG(hw, REG_IMR, hw->intr_mask);
- spin_unlock(&hw->intr_mask_lock);
- __napi_schedule(&adapter->tx_napi);
- }
- }
+ if (status & (ISR_RX_PKT | ISR_TX_PKT))
+ atl1c_intr_rx_tx(adapter, status);
handled = IRQ_HANDLED;
/* check if PCIE PHY Link down */
}
static struct sk_buff *atl1c_alloc_skb(struct atl1c_adapter *adapter,
- bool napi_mode)
+ u32 queue, bool napi_mode)
{
+ struct atl1c_rrd_ring *rrd_ring = &adapter->rrd_ring[queue];
struct sk_buff *skb;
struct page *page;
if (adapter->rx_frag_size > PAGE_SIZE) {
if (likely(napi_mode))
- return napi_alloc_skb(&adapter->napi,
+ return napi_alloc_skb(&rrd_ring->napi,
adapter->rx_buffer_len);
else
return netdev_alloc_skb_ip_align(adapter->netdev,
adapter->rx_buffer_len);
}
- page = adapter->rx_page;
+ page = rrd_ring->rx_page;
if (!page) {
- adapter->rx_page = page = alloc_page(GFP_ATOMIC);
+ page = alloc_page(GFP_ATOMIC);
if (unlikely(!page))
return NULL;
- adapter->rx_page_offset = 0;
+ rrd_ring->rx_page = page;
+ rrd_ring->rx_page_offset = 0;
}
- skb = build_skb(page_address(page) + adapter->rx_page_offset,
+ skb = build_skb(page_address(page) + rrd_ring->rx_page_offset,
adapter->rx_frag_size);
if (likely(skb)) {
skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN);
- adapter->rx_page_offset += adapter->rx_frag_size;
- if (adapter->rx_page_offset >= PAGE_SIZE)
- adapter->rx_page = NULL;
+ rrd_ring->rx_page_offset += adapter->rx_frag_size;
+ if (rrd_ring->rx_page_offset >= PAGE_SIZE)
+ rrd_ring->rx_page = NULL;
else
get_page(page);
}
return skb;
}
-static int atl1c_alloc_rx_buffer(struct atl1c_adapter *adapter, bool napi_mode)
+static int atl1c_alloc_rx_buffer(struct atl1c_adapter *adapter, u32 queue,
+ bool napi_mode)
{
- struct atl1c_rfd_ring *rfd_ring = &adapter->rfd_ring;
+ struct atl1c_rfd_ring *rfd_ring = &adapter->rfd_ring[queue];
struct pci_dev *pdev = adapter->pdev;
struct atl1c_buffer *buffer_info, *next_info;
struct sk_buff *skb;
while (next_info->flags & ATL1C_BUFFER_FREE) {
rfd_desc = ATL1C_RFD_DESC(rfd_ring, rfd_next_to_use);
- skb = atl1c_alloc_skb(adapter, napi_mode);
+ skb = atl1c_alloc_skb(adapter, queue, napi_mode);
if (unlikely(!skb)) {
if (netif_msg_rx_err(adapter))
dev_warn(&pdev->dev, "alloc rx buffer failed\n");
/* TODO: update mailbox here */
wmb();
rfd_ring->next_to_use = rfd_next_to_use;
- AT_WRITE_REG(&adapter->hw, REG_MB_RFD0_PROD_IDX,
- rfd_ring->next_to_use & MB_RFDX_PROD_IDX_MASK);
+ AT_WRITE_REG(&adapter->hw, atl1c_qregs[queue].rfd_prod,
+ rfd_ring->next_to_use & MB_RFDX_PROD_IDX_MASK);
}
return num_alloc;
rfd_ring->next_to_clean = rfd_index;
}
-static void atl1c_clean_rx_irq(struct atl1c_adapter *adapter,
- int *work_done, int work_to_do)
+/**
+ * atl1c_clean_rx - NAPI Rx polling callback
+ * @napi: napi info
+ * @budget: limit of packets to clean
+ */
+static int atl1c_clean_rx(struct napi_struct *napi, int budget)
{
+ struct atl1c_rrd_ring *rrd_ring =
+ container_of(napi, struct atl1c_rrd_ring, napi);
+ struct atl1c_adapter *adapter = rrd_ring->adapter;
u16 rfd_num, rfd_index;
- u16 count = 0;
u16 length;
struct pci_dev *pdev = adapter->pdev;
struct net_device *netdev = adapter->netdev;
- struct atl1c_rfd_ring *rfd_ring = &adapter->rfd_ring;
- struct atl1c_rrd_ring *rrd_ring = &adapter->rrd_ring;
+ struct atl1c_rfd_ring *rfd_ring = &adapter->rfd_ring[rrd_ring->num];
struct sk_buff *skb;
struct atl1c_recv_ret_status *rrs;
struct atl1c_buffer *buffer_info;
+ int work_done = 0;
+ unsigned long flags;
+
+ /* Keep link state information with original netdev */
+ if (!netif_carrier_ok(adapter->netdev))
+ goto quit_polling;
while (1) {
- if (*work_done >= work_to_do)
+ if (work_done >= budget)
break;
rrs = ATL1C_RRD_DESC(rrd_ring, rrd_ring->next_to_clean);
if (likely(RRS_RXD_IS_VALID(rrs->word3))) {
vlan = le16_to_cpu(vlan);
__vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), vlan);
}
- napi_gro_receive(&adapter->napi, skb);
+ napi_gro_receive(napi, skb);
- (*work_done)++;
- count++;
+ work_done++;
}
- if (count)
- atl1c_alloc_rx_buffer(adapter, true);
-}
-
-/**
- * atl1c_clean - NAPI Rx polling callback
- * @napi: napi info
- * @budget: limit of packets to clean
- */
-static int atl1c_clean(struct napi_struct *napi, int budget)
-{
- struct atl1c_adapter *adapter =
- container_of(napi, struct atl1c_adapter, napi);
- int work_done = 0;
- unsigned long flags;
-
- /* Keep link state information with original netdev */
- if (!netif_carrier_ok(adapter->netdev))
- goto quit_polling;
- /* just enable one RXQ */
- atl1c_clean_rx_irq(adapter, &work_done, budget);
+ if (work_done)
+ atl1c_alloc_rx_buffer(adapter, rrd_ring->num, true);
if (work_done < budget) {
quit_polling:
napi_complete_done(napi, work_done);
spin_lock_irqsave(&adapter->hw.intr_mask_lock, flags);
- adapter->hw.intr_mask |= ISR_RX_PKT;
+ adapter->hw.intr_mask |= atl1c_qregs[rrd_ring->num].rx_isr;
AT_WRITE_REG(&adapter->hw, REG_IMR, adapter->hw.intr_mask);
spin_unlock_irqrestore(&adapter->hw.intr_mask_lock, flags);
}
}
#endif
-static inline u16 atl1c_tpd_avail(struct atl1c_adapter *adapter, enum atl1c_trans_queue type)
+static inline u16 atl1c_tpd_avail(struct atl1c_adapter *adapter, u32 queue)
{
- struct atl1c_tpd_ring *tpd_ring = &adapter->tpd_ring[type];
+ struct atl1c_tpd_ring *tpd_ring = &adapter->tpd_ring[queue];
u16 next_to_use = 0;
u16 next_to_clean = 0;
* there is enough tpd to use
*/
static struct atl1c_tpd_desc *atl1c_get_tpd(struct atl1c_adapter *adapter,
- enum atl1c_trans_queue type)
+ u32 queue)
{
- struct atl1c_tpd_ring *tpd_ring = &adapter->tpd_ring[type];
+ struct atl1c_tpd_ring *tpd_ring = &adapter->tpd_ring[queue];
struct atl1c_tpd_desc *tpd_desc;
u16 next_to_use = 0;
static int atl1c_tso_csum(struct atl1c_adapter *adapter,
struct sk_buff *skb,
struct atl1c_tpd_desc **tpd,
- enum atl1c_trans_queue type)
+ u32 queue)
{
struct pci_dev *pdev = adapter->pdev;
unsigned short offload_type;
*(struct atl1c_tpd_ext_desc **)(tpd);
memset(etpd, 0, sizeof(struct atl1c_tpd_ext_desc));
- *tpd = atl1c_get_tpd(adapter, type);
+ *tpd = atl1c_get_tpd(adapter, queue);
ipv6_hdr(skb)->payload_len = 0;
/* check payload == 0 byte ? */
hdr_len = (skb_transport_offset(skb) + tcp_hdrlen(skb));
static void atl1c_tx_rollback(struct atl1c_adapter *adpt,
struct atl1c_tpd_desc *first_tpd,
- enum atl1c_trans_queue type)
+ u32 queue)
{
- struct atl1c_tpd_ring *tpd_ring = &adpt->tpd_ring[type];
+ struct atl1c_tpd_ring *tpd_ring = &adpt->tpd_ring[queue];
struct atl1c_buffer *buffer_info;
struct atl1c_tpd_desc *tpd;
u16 first_index, index;
}
static int atl1c_tx_map(struct atl1c_adapter *adapter,
- struct sk_buff *skb, struct atl1c_tpd_desc *tpd,
- enum atl1c_trans_queue type)
+ struct sk_buff *skb, struct atl1c_tpd_desc *tpd,
+ u32 queue)
{
struct atl1c_tpd_desc *use_tpd = NULL;
struct atl1c_buffer *buffer_info = NULL;
if (mapped_len == 0)
use_tpd = tpd;
else {
- use_tpd = atl1c_get_tpd(adapter, type);
+ use_tpd = atl1c_get_tpd(adapter, queue);
memcpy(use_tpd, tpd, sizeof(struct atl1c_tpd_desc));
}
buffer_info = atl1c_get_tx_buffer(adapter, use_tpd);
for (f = 0; f < nr_frags; f++) {
skb_frag_t *frag = &skb_shinfo(skb)->frags[f];
- use_tpd = atl1c_get_tpd(adapter, type);
+ use_tpd = atl1c_get_tpd(adapter, queue);
memcpy(use_tpd, tpd, sizeof(struct atl1c_tpd_desc));
buffer_info = atl1c_get_tx_buffer(adapter, use_tpd);
return -1;
}
-static void atl1c_tx_queue(struct atl1c_adapter *adapter,
- enum atl1c_trans_queue type)
+static void atl1c_tx_queue(struct atl1c_adapter *adapter, u32 queue)
{
- struct atl1c_tpd_ring *tpd_ring = &adapter->tpd_ring[type];
- u16 reg;
+ struct atl1c_tpd_ring *tpd_ring = &adapter->tpd_ring[queue];
- reg = type == atl1c_trans_high ? REG_TPD_PRI1_PIDX : REG_TPD_PRI0_PIDX;
- AT_WRITE_REGW(&adapter->hw, reg, tpd_ring->next_to_use);
+ AT_WRITE_REGW(&adapter->hw, atl1c_qregs[queue].tpd_prod,
+ tpd_ring->next_to_use);
}
static netdev_tx_t atl1c_xmit_frame(struct sk_buff *skb,
struct net_device *netdev)
{
struct atl1c_adapter *adapter = netdev_priv(netdev);
- u16 tpd_req;
+ u32 queue = skb_get_queue_mapping(skb);
+ struct netdev_queue *txq = netdev_get_tx_queue(netdev, queue);
struct atl1c_tpd_desc *tpd;
- enum atl1c_trans_queue type = atl1c_trans_normal;
+ u16 tpd_req;
if (test_bit(__AT_DOWN, &adapter->flags)) {
dev_kfree_skb_any(skb);
tpd_req = atl1c_cal_tpd_req(skb);
- if (atl1c_tpd_avail(adapter, type) < tpd_req) {
+ if (atl1c_tpd_avail(adapter, queue) < tpd_req) {
/* no enough descriptor, just stop queue */
- atl1c_tx_queue(adapter, type);
- netif_stop_queue(netdev);
+ atl1c_tx_queue(adapter, queue);
+ netif_tx_stop_queue(txq);
return NETDEV_TX_BUSY;
}
- tpd = atl1c_get_tpd(adapter, type);
+ tpd = atl1c_get_tpd(adapter, queue);
/* do TSO and check sum */
- if (atl1c_tso_csum(adapter, skb, &tpd, type) != 0) {
- atl1c_tx_queue(adapter, type);
+ if (atl1c_tso_csum(adapter, skb, &tpd, queue) != 0) {
+ atl1c_tx_queue(adapter, queue);
dev_kfree_skb_any(skb);
return NETDEV_TX_OK;
}
if (skb_network_offset(skb) != ETH_HLEN)
tpd->word1 |= 1 << TPD_ETH_TYPE_SHIFT; /* Ethernet frame */
- if (atl1c_tx_map(adapter, skb, tpd, type) < 0) {
+ if (atl1c_tx_map(adapter, skb, tpd, queue) < 0) {
netif_info(adapter, tx_done, adapter->netdev,
"tx-skb dropped due to dma error\n");
/* roll back tpd/buffer */
- atl1c_tx_rollback(adapter, tpd, type);
+ atl1c_tx_rollback(adapter, tpd, queue);
dev_kfree_skb_any(skb);
} else {
bool more = netdev_xmit_more();
- if (__netdev_sent_queue(adapter->netdev, skb->len, more))
- atl1c_tx_queue(adapter, type);
+ if (__netdev_tx_sent_queue(txq, skb->len, more))
+ atl1c_tx_queue(adapter, queue);
}
return NETDEV_TX_OK;
static void atl1c_reset_dma_ring(struct atl1c_adapter *adapter)
{
+ int i;
/* release tx-pending skbs and reset tx/rx ring index */
- atl1c_clean_tx_ring(adapter, atl1c_trans_normal);
- atl1c_clean_tx_ring(adapter, atl1c_trans_high);
- atl1c_clean_rx_ring(adapter);
+ for (i = 0; i < adapter->tx_queue_count; ++i)
+ atl1c_clean_tx_ring(adapter, i);
+ for (i = 0; i < adapter->rx_queue_count; ++i)
+ atl1c_clean_rx_ring(adapter, i);
}
static int atl1c_up(struct atl1c_adapter *adapter)
{
struct net_device *netdev = adapter->netdev;
int err;
+ int i;
netif_carrier_off(netdev);
atl1c_check_link_status(adapter);
clear_bit(__AT_DOWN, &adapter->flags);
- napi_enable(&adapter->napi);
- napi_enable(&adapter->tx_napi);
+ for (i = 0; i < adapter->tx_queue_count; ++i)
+ napi_enable(&adapter->tpd_ring[i].napi);
+ for (i = 0; i < adapter->rx_queue_count; ++i)
+ napi_enable(&adapter->rrd_ring[i].napi);
atl1c_irq_enable(adapter);
netif_start_queue(netdev);
return err;
err_up:
- atl1c_clean_rx_ring(adapter);
+ for (i = 0; i < adapter->rx_queue_count; ++i)
+ atl1c_clean_rx_ring(adapter, i);
return err;
}
static void atl1c_down(struct atl1c_adapter *adapter)
{
struct net_device *netdev = adapter->netdev;
+ int i;
atl1c_del_timer(adapter);
adapter->work_event = 0; /* clear all event */
* reschedule our watchdog timer */
set_bit(__AT_DOWN, &adapter->flags);
netif_carrier_off(netdev);
- napi_disable(&adapter->napi);
- napi_disable(&adapter->tx_napi);
+ for (i = 0; i < adapter->tx_queue_count; ++i)
+ napi_disable(&adapter->tpd_ring[i].napi);
+ for (i = 0; i < adapter->rx_queue_count; ++i)
+ napi_disable(&adapter->rrd_ring[i].napi);
atl1c_irq_disable(adapter);
atl1c_free_irq(adapter);
/* disable ASPM if device inactive */
struct net_device *netdev;
struct atl1c_adapter *adapter;
static int cards_found;
-
+ u8 __iomem *hw_addr;
+ enum atl1c_nic_type nic_type;
+ u32 queue_count = 1;
int err = 0;
+ int i;
/* enable device (incl. PCI PM wakeup and hotplug setup) */
err = pci_enable_device_mem(pdev);
pci_set_master(pdev);
- netdev = alloc_etherdev(sizeof(struct atl1c_adapter));
+ hw_addr = pci_ioremap_bar(pdev, 0);
+ if (!hw_addr) {
+ err = -EIO;
+ dev_err(&pdev->dev, "cannot map device registers\n");
+ goto err_ioremap;
+ }
+
+ nic_type = atl1c_get_mac_type(pdev, hw_addr);
+ if (nic_type == athr_mt)
+ queue_count = 4;
+
+ netdev = alloc_etherdev_mq(sizeof(struct atl1c_adapter), queue_count);
if (netdev == NULL) {
err = -ENOMEM;
goto err_alloc_etherdev;
adapter->netdev = netdev;
adapter->pdev = pdev;
adapter->hw.adapter = adapter;
+ adapter->hw.nic_type = nic_type;
adapter->msg_enable = netif_msg_init(-1, atl1c_default_msg);
- adapter->hw.hw_addr = ioremap(pci_resource_start(pdev, 0), pci_resource_len(pdev, 0));
- if (!adapter->hw.hw_addr) {
- err = -EIO;
- dev_err(&pdev->dev, "cannot map device registers\n");
- goto err_ioremap;
- }
+ adapter->hw.hw_addr = hw_addr;
+ adapter->tx_queue_count = queue_count;
+ adapter->rx_queue_count = queue_count;
/* init mii data */
adapter->mii.dev = netdev;
adapter->mii.phy_id_mask = 0x1f;
adapter->mii.reg_num_mask = MDIO_CTRL_REG_MASK;
dev_set_threaded(netdev, true);
- netif_napi_add(netdev, &adapter->napi, atl1c_clean, 64);
- netif_napi_add(netdev, &adapter->tx_napi, atl1c_clean_tx, 64);
+ for (i = 0; i < adapter->rx_queue_count; ++i)
+ netif_napi_add(netdev, &adapter->rrd_ring[i].napi,
+ atl1c_clean_rx, 64);
+ for (i = 0; i < adapter->tx_queue_count; ++i)
+ netif_napi_add(netdev, &adapter->tpd_ring[i].napi,
+ atl1c_clean_tx, 64);
timer_setup(&adapter->phy_config_timer, atl1c_phy_config, 0);
/* setup the private structure */
err = atl1c_sw_init(adapter);
err_reset:
err_register:
err_sw_init:
- iounmap(adapter->hw.hw_addr);
err_init_netdev:
-err_ioremap:
free_netdev(netdev);
err_alloc_etherdev:
+ iounmap(hw_addr);
+err_ioremap:
pci_release_regions(pdev);
err_pci_reg:
err_dma:
}
/**
- * atl1e_set_mac - Change the Ethernet Address of the NIC
+ * atl1e_set_mac_addr - Change the Ethernet Address of the NIC
* @netdev: network interface device structure
* @p: pointer to an address structure
*
}
/**
- * atl1e_setup_mem_resources - allocate Tx / RX descriptor resources
+ * atl1e_setup_ring_resources - allocate Tx / RX descriptor resources
* @adapter: board private structure
*
* Return 0 on success, negative on failure
}
/**
- * atl1_setup_mem_resources - allocate Tx / RX descriptor resources
+ * atl1_setup_ring_resources - allocate Tx / RX descriptor resources
* @adapter: board private structure
*
* Return 0 on success, negative on failure
plen0 = b44_magic_pattern(bp->dev->dev_addr, pwol_pattern, pwol_mask,
B44_ETHIPV4UDP_HLEN);
- bwfilter_table(bp, pwol_pattern, B44_PATTERN_SIZE, B44_PATTERN_BASE);
- bwfilter_table(bp, pwol_mask, B44_PMASK_SIZE, B44_PMASK_BASE);
+ bwfilter_table(bp, pwol_pattern, B44_PATTERN_SIZE, B44_PATTERN_BASE);
+ bwfilter_table(bp, pwol_mask, B44_PMASK_SIZE, B44_PMASK_BASE);
/* Raw ethernet II magic packet pattern - pattern 1 */
memset(pwol_pattern, 0, B44_PATTERN_SIZE);
plen1 = b44_magic_pattern(bp->dev->dev_addr, pwol_pattern, pwol_mask,
ETH_HLEN);
- bwfilter_table(bp, pwol_pattern, B44_PATTERN_SIZE,
+ bwfilter_table(bp, pwol_pattern, B44_PATTERN_SIZE,
B44_PATTERN_BASE + B44_PATTERN_SIZE);
- bwfilter_table(bp, pwol_mask, B44_PMASK_SIZE,
+ bwfilter_table(bp, pwol_mask, B44_PMASK_SIZE,
B44_PMASK_BASE + B44_PMASK_SIZE);
/* Ipv6 magic packet pattern - pattern 2 */
plen2 = b44_magic_pattern(bp->dev->dev_addr, pwol_pattern, pwol_mask,
B44_ETHIPV6UDP_HLEN);
- bwfilter_table(bp, pwol_pattern, B44_PATTERN_SIZE,
+ bwfilter_table(bp, pwol_pattern, B44_PATTERN_SIZE,
B44_PATTERN_BASE + B44_PATTERN_SIZE + B44_PATTERN_SIZE);
- bwfilter_table(bp, pwol_mask, B44_PMASK_SIZE,
+ bwfilter_table(bp, pwol_mask, B44_PMASK_SIZE,
B44_PMASK_BASE + B44_PMASK_SIZE + B44_PMASK_SIZE);
kfree(pwol_pattern);
val = br32(bp, B44_DEVCTRL);
bw32(bp, B44_DEVCTRL, val | DEVCTRL_MPM | DEVCTRL_PFE);
- } else {
- b44_setup_pseudo_magicp(bp);
- }
+ } else {
+ b44_setup_pseudo_magicp(bp);
+ }
b44_setup_wol_pci(bp);
}
__b44_cam_write(bp, zero, i);
bw32(bp, B44_RXCONFIG, val);
- val = br32(bp, B44_CAM_CTRL);
+ val = br32(bp, B44_CAM_CTRL);
bw32(bp, B44_CAM_CTRL, val | CAM_CTRL_ENABLE);
}
}
{
struct device_node *np = pdev->dev.of_node;
struct bgmac *bgmac;
- struct resource *regs;
int ret;
bgmac = bgmac_alloc(&pdev->dev);
if (IS_ERR(bgmac->plat.base))
return PTR_ERR(bgmac->plat.base);
- regs = platform_get_resource_byname(pdev, IORESOURCE_MEM, "idm_base");
- if (regs) {
- bgmac->plat.idm_base = devm_ioremap_resource(&pdev->dev, regs);
- if (IS_ERR(bgmac->plat.idm_base))
- return PTR_ERR(bgmac->plat.idm_base);
+ bgmac->plat.idm_base = devm_platform_ioremap_resource_byname(pdev, "idm_base");
+ if (IS_ERR(bgmac->plat.idm_base))
+ return PTR_ERR(bgmac->plat.idm_base);
+ else
bgmac->feature_flags &= ~BGMAC_FEAT_IDM_MASK;
- }
- regs = platform_get_resource_byname(pdev, IORESOURCE_MEM, "nicpm_base");
- if (regs) {
- bgmac->plat.nicpm_base = devm_ioremap_resource(&pdev->dev,
- regs);
- if (IS_ERR(bgmac->plat.nicpm_base))
- return PTR_ERR(bgmac->plat.nicpm_base);
- }
+ bgmac->plat.nicpm_base = devm_platform_ioremap_resource_byname(pdev, "nicpm_base");
+ if (IS_ERR(bgmac->plat.nicpm_base))
+ return PTR_ERR(bgmac->plat.nicpm_base);
bgmac->read = platform_bgmac_read;
bgmac->write = platform_bgmac_write;
}
if (val1 & BNX2_EMAC_MDIO_COMM_START_BUSY)
- ret = -EBUSY;
+ ret = -EBUSY;
else
ret = 0;
for (i = 0; i < NUM_MC_HASH_REGISTERS; i++) {
BNX2_WR(bp, BNX2_EMAC_MULTICAST_HASH0 + (i * 4),
0xffffffff);
- }
+ }
sort_mode |= BNX2_RPM_SORT_USER0_MC_EN;
}
else {
if (addr == page_end-4) {
cmd_flags = BNX2_NVM_COMMAND_LAST;
- }
+ }
rc = bnx2_nvram_write_dword(bp, addr,
&flash_buffer[i], cmd_flags);
BNX2_WR(bp, PCI_COMMAND, reg);
} else if ((BNX2_CHIP_ID(bp) == BNX2_CHIP_ID_5706_A1) &&
!(bp->flags & BNX2_FLAG_PCIX)) {
-
dev_err(&pdev->dev,
"5706 A1 can only be used in a PCIX bus, aborting\n");
+ rc = -EPERM;
goto err_out_unmap;
}
}
/**
- * bnx2x_get_num_none_def_sbs - return the number of none default SBs
+ * bnx2x_get_num_non_def_sbs - return the number of none default SBs
* @pdev: pci device
* @cnic_cnt: count
*
/*************************** Credit handling **********************************/
/**
- * atomic_add_ifless - add if the result is less than a given value.
+ * __atomic_add_ifless - add if the result is less than a given value.
*
* @v: pointer of type atomic_t
* @a: the amount to add to v...
}
/**
- * atomic_dec_ifmoe - dec if the result is more or equal than a given value.
+ * __atomic_dec_ifmoe - dec if the result is more or equal than a given value.
*
* @v: pointer of type atomic_t
* @a: the amount to dec from v...
goto failed;
/* SR-IOV capability was enabled but there are no VFs*/
- if (iov->total == 0)
+ if (iov->total == 0) {
+ err = -EINVAL;
goto failed;
+ }
iov->nr_virtfn = min_t(u16, iov->total, num_vfs_param);
/* VF side vfpf channel functions */
int bnx2x_vfpf_acquire(struct bnx2x *bp, u8 tx_count, u8 rx_count);
int bnx2x_vfpf_release(struct bnx2x *bp);
-int bnx2x_vfpf_release(struct bnx2x *bp);
int bnx2x_vfpf_init(struct bnx2x *bp);
void bnx2x_vfpf_close_vf(struct bnx2x *bp);
int bnx2x_vfpf_setup_q(struct bnx2x *bp, struct bnx2x_fastpath *fp,
{
return (idx == NETXTREME_C_VF || idx == NETXTREME_E_VF ||
idx == NETXTREME_S_VF || idx == NETXTREME_C_VF_HV ||
- idx == NETXTREME_E_VF_HV || idx == NETXTREME_E_P5_VF);
+ idx == NETXTREME_E_VF_HV || idx == NETXTREME_E_P5_VF ||
+ idx == NETXTREME_E_P5_VF_HV);
}
#define DB_CP_REARM_FLAGS (DB_KEY_CP | DB_IDX_VALID)
static void bnxt_hwrm_set_pg_attr(struct bnxt_ring_mem_info *rmem, u8 *pg_attr,
__le64 *pg_dir)
{
- u8 pg_size = 0;
-
if (!rmem->nr_pages)
return;
- if (BNXT_PAGE_SHIFT == 13)
- pg_size = 1 << 4;
- else if (BNXT_PAGE_SIZE == 16)
- pg_size = 2 << 4;
-
- *pg_attr = pg_size;
+ BNXT_SET_CTX_PAGE_ATTR(*pg_attr);
if (rmem->depth >= 1) {
if (rmem->depth == 2)
*pg_attr |= 2;
entries_sp = ctx->vnic_max_vnic_entries + ctx->qp_max_l2_entries +
2 * (extra_qps + ctx->qp_min_qp1_entries) + min;
entries_sp = roundup(entries_sp, ctx->tqm_entries_multiple);
- entries = ctx->qp_max_l2_entries + extra_qps + ctx->qp_min_qp1_entries;
+ entries = ctx->qp_max_l2_entries + 2 * (extra_qps + ctx->qp_min_qp1_entries);
entries = roundup(entries, ctx->tqm_entries_multiple);
entries = clamp_t(u32, entries, min, ctx->tqm_max_entries_per_ring);
for (i = 0; i < ctx->tqm_fp_rings_count + 1; i++) {
return rc;
}
+static bool bnxt_exthdr_check(struct bnxt *bp, struct sk_buff *skb, int nw_off,
+ u8 **nextp)
+{
+ struct ipv6hdr *ip6h = (struct ipv6hdr *)(skb->data + nw_off);
+ int hdr_count = 0;
+ u8 *nexthdr;
+ int start;
+
+ /* Check that there are at most 2 IPv6 extension headers, no
+ * fragment header, and each is <= 64 bytes.
+ */
+ start = nw_off + sizeof(*ip6h);
+ nexthdr = &ip6h->nexthdr;
+ while (ipv6_ext_hdr(*nexthdr)) {
+ struct ipv6_opt_hdr *hp;
+ int hdrlen;
+
+ if (hdr_count >= 3 || *nexthdr == NEXTHDR_NONE ||
+ *nexthdr == NEXTHDR_FRAGMENT)
+ return false;
+ hp = __skb_header_pointer(NULL, start, sizeof(*hp), skb->data,
+ skb_headlen(skb), NULL);
+ if (!hp)
+ return false;
+ if (*nexthdr == NEXTHDR_AUTH)
+ hdrlen = ipv6_authlen(hp);
+ else
+ hdrlen = ipv6_optlen(hp);
+
+ if (hdrlen > 64)
+ return false;
+ nexthdr = &hp->nexthdr;
+ start += hdrlen;
+ hdr_count++;
+ }
+ if (nextp) {
+ /* Caller will check inner protocol */
+ if (skb->encapsulation) {
+ *nextp = nexthdr;
+ return true;
+ }
+ *nextp = NULL;
+ }
+ /* Only support TCP/UDP for non-tunneled ipv6 and inner ipv6 */
+ return *nexthdr == IPPROTO_TCP || *nexthdr == IPPROTO_UDP;
+}
+
+/* For UDP, we can only handle 1 Vxlan port and 1 Geneve port. */
+static bool bnxt_udp_tunl_check(struct bnxt *bp, struct sk_buff *skb)
+{
+ struct udphdr *uh = udp_hdr(skb);
+ __be16 udp_port = uh->dest;
+
+ if (udp_port != bp->vxlan_port && udp_port != bp->nge_port)
+ return false;
+ if (skb->inner_protocol_type == ENCAP_TYPE_ETHER) {
+ struct ethhdr *eh = inner_eth_hdr(skb);
+
+ switch (eh->h_proto) {
+ case htons(ETH_P_IP):
+ return true;
+ case htons(ETH_P_IPV6):
+ return bnxt_exthdr_check(bp, skb,
+ skb_inner_network_offset(skb),
+ NULL);
+ }
+ }
+ return false;
+}
+
+static bool bnxt_tunl_check(struct bnxt *bp, struct sk_buff *skb, u8 l4_proto)
+{
+ switch (l4_proto) {
+ case IPPROTO_UDP:
+ return bnxt_udp_tunl_check(bp, skb);
+ case IPPROTO_IPIP:
+ return true;
+ case IPPROTO_GRE: {
+ switch (skb->inner_protocol) {
+ default:
+ return false;
+ case htons(ETH_P_IP):
+ return true;
+ case htons(ETH_P_IPV6):
+ fallthrough;
+ }
+ }
+ case IPPROTO_IPV6:
+ /* Check ext headers of inner ipv6 */
+ return bnxt_exthdr_check(bp, skb, skb_inner_network_offset(skb),
+ NULL);
+ }
+ return false;
+}
+
static netdev_features_t bnxt_features_check(struct sk_buff *skb,
struct net_device *dev,
netdev_features_t features)
{
- struct bnxt *bp;
- __be16 udp_port;
- u8 l4_proto = 0;
+ struct bnxt *bp = netdev_priv(dev);
+ u8 *l4_proto;
features = vlan_features_check(skb, features);
- if (!skb->encapsulation)
- return features;
-
switch (vlan_get_protocol(skb)) {
case htons(ETH_P_IP):
- l4_proto = ip_hdr(skb)->protocol;
+ if (!skb->encapsulation)
+ return features;
+ l4_proto = &ip_hdr(skb)->protocol;
+ if (bnxt_tunl_check(bp, skb, *l4_proto))
+ return features;
break;
case htons(ETH_P_IPV6):
- l4_proto = ipv6_hdr(skb)->nexthdr;
+ if (!bnxt_exthdr_check(bp, skb, skb_network_offset(skb),
+ &l4_proto))
+ break;
+ if (!l4_proto || bnxt_tunl_check(bp, skb, *l4_proto))
+ return features;
break;
- default:
- return features;
}
-
- if (l4_proto != IPPROTO_UDP)
- return features;
-
- bp = netdev_priv(dev);
- /* For UDP, we can only handle 1 Vxlan port and 1 Geneve port. */
- udp_port = udp_hdr(skb)->dest;
- if (udp_port == bp->vxlan_port || udp_port == bp->nge_port)
- return features;
return features & ~(NETIF_F_CSUM_MASK | NETIF_F_GSO_MASK);
}
bnxt_hwrm_coal_params_qcaps(bp);
}
+static int bnxt_probe_phy(struct bnxt *bp, bool fw_dflt);
+
static int bnxt_fw_init_one(struct bnxt *bp)
{
int rc;
netdev_err(bp->dev, "Firmware init phase 2 failed\n");
return rc;
}
+ rc = bnxt_probe_phy(bp, false);
+ if (rc)
+ return rc;
rc = bnxt_approve_mac(bp, bp->dev->dev_addr, false);
if (rc)
return rc;
bnxt_hwrm_func_drv_unrgtr(bp);
bnxt_free_hwrm_short_cmd_req(bp);
bnxt_free_hwrm_resources(bp);
+ bnxt_ethtool_free(bp);
kfree(bp->fw_health);
bp->fw_health = NULL;
bnxt_cleanup_pci(bp);
#define BNXT_BACKING_STORE_CFG_LEGACY_LEN 256
+#define BNXT_SET_CTX_PAGE_ATTR(attr) \
+do { \
+ if (BNXT_PAGE_SIZE == 0x2000) \
+ attr = FUNC_BACKING_STORE_CFG_REQ_SRQ_PG_SIZE_PG_8K; \
+ else if (BNXT_PAGE_SIZE == 0x10000) \
+ attr = FUNC_BACKING_STORE_CFG_REQ_QPC_PG_SIZE_PG_64K; \
+ else \
+ attr = FUNC_BACKING_STORE_CFG_REQ_QPC_PG_SIZE_PG_4K; \
+} while (0)
+
struct bnxt_ctx_mem_info {
u32 qp_max_entries;
u16 qp_min_qp1_entries;
MODULE_DESCRIPTION("Broadcom GENET Ethernet controller driver");
MODULE_ALIAS("platform:bcmgenet");
MODULE_LICENSE("GPL");
+MODULE_SOFTDEP("pre: mdio-bcm-unimac");
int id, ret;
pres = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!pres) {
+ dev_err(&pdev->dev, "Invalid resource\n");
+ return -EINVAL;
+ }
memset(&res, 0, sizeof(res));
memset(&ppd, 0, sizeof(ppd));
}
/**
- * bfa_cee_get_attr_isr - CEE ISR for get-stats responses from f/w
+ * bfa_cee_get_stats_isr - CEE ISR for get-stats responses from f/w
*
* @cee: Pointer to the CEE module
* @status: Return status from the f/w
struct gem_stats *hwstat = &bp->hw_stats.gem;
struct net_device_stats *nstat = &bp->dev->stats;
+ if (!netif_running(bp->dev))
+ return nstat;
+
gem_update_stats(bp);
nstat->rx_errors = (hwstat->rx_frame_check_sequence_errors +
struct macb *bp;
int err, val;
- regs = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- mem = devm_ioremap_resource(&pdev->dev, regs);
+ mem = devm_platform_get_and_ioremap_resource(pdev, 0, ®s);
if (IS_ERR(mem))
return PTR_ERR(mem);
// SPDX-License-Identifier: GPL-2.0-only
-/**
+/*
* DOC: Cadence GEM PCI wrapper.
*
* Copyright (C) 2016 Cadence Design Systems - https://www.cadence.com
// SPDX-License-Identifier: GPL-2.0-only
-/**
+/*
* 1588 PTP support for Cadence GEM device.
*
* Copyright (C) 2017 Cadence Design Systems - https://www.cadence.com
}
/**
- * init_xgmac_dma_desc_rings - init the RX/TX descriptor rings
+ * xgmac_dma_desc_rings_init - init the RX/TX descriptor rings
* @dev: net device structure
* Description: this function initializes the DMA RX/TX descriptors
* and allocates the socket buffers.
}
/**
- * xgmac_tx:
+ * xgmac_tx_complete:
* @priv: private driver structure
* Description: it reclaims resources after transmission completes.
*/
}
/**
- * xgmac_release - close entry point of the driver
+ * xgmac_stop - close entry point of the driver
* @dev : device pointer.
* Description:
* This is the stop entry point of the driver.
}
/**
- * xgmac_dvr_remove
+ * xgmac_remove
* @pdev: platform device pointer
* Description: this function resets the TX/RX processes, disables the MAC RX/TX
* changes the link status, releases the DMA descriptor rings,
* @lio: per-network private data
* @start_stop: whether to start or stop
*/
-static void send_rx_ctrl_cmd(struct lio *lio, int start_stop)
+static int send_rx_ctrl_cmd(struct lio *lio, int start_stop)
{
struct octeon_soft_command *sc;
union octnet_cmd *ncmd;
int retval;
if (oct->props[lio->ifidx].rx_on == start_stop)
- return;
+ return 0;
sc = (struct octeon_soft_command *)
octeon_alloc_soft_command(oct, OCTNET_CMD_SIZE,
16, 0);
if (!sc) {
netif_info(lio, rx_err, lio->netdev,
- "Failed to allocate octeon_soft_command\n");
- return;
+ "Failed to allocate octeon_soft_command struct\n");
+ return -ENOMEM;
}
ncmd = (union octnet_cmd *)sc->virtdptr;
if (retval == IQ_SEND_FAILED) {
netif_info(lio, rx_err, lio->netdev, "Failed to send RX Control message\n");
octeon_free_soft_command(oct, sc);
- return;
} else {
/* Sleep on a wait queue till the cond flag indicates that the
* response arrived or timed-out.
*/
retval = wait_for_sc_completion_timeout(oct, sc, 0);
if (retval)
- return;
+ return retval;
oct->props[lio->ifidx].rx_on = start_stop;
WRITE_ONCE(sc->caller_is_done, true);
}
+
+ return retval;
}
/**
struct octeon_device_priv *oct_priv =
(struct octeon_device_priv *)oct->priv;
struct napi_struct *napi, *n;
+ int ret = 0;
if (oct->props[lio->ifidx].napi_enabled == 0) {
tasklet_disable(&oct_priv->droq_tasklet);
netif_info(lio, ifup, lio->netdev, "Interface Open, ready for traffic\n");
/* tell Octeon to start forwarding packets to host */
- send_rx_ctrl_cmd(lio, 1);
+ ret = send_rx_ctrl_cmd(lio, 1);
+ if (ret)
+ return ret;
/* start periodical statistics fetch */
INIT_DELAYED_WORK(&lio->stats_wk.work, lio_fetch_stats);
dev_info(&oct->pci_dev->dev, "%s interface is opened\n",
netdev->name);
- return 0;
+ return ret;
}
/**
struct octeon_device_priv *oct_priv =
(struct octeon_device_priv *)oct->priv;
struct napi_struct *napi, *n;
+ int ret = 0;
ifstate_reset(lio, LIO_IFSTATE_RUNNING);
lio->link_changes++;
/* Tell Octeon that nic interface is down. */
- send_rx_ctrl_cmd(lio, 0);
+ ret = send_rx_ctrl_cmd(lio, 0);
+ if (ret)
+ return ret;
if (OCTEON_CN23XX_PF(oct)) {
if (!oct->msix_on)
dev_info(&oct->pci_dev->dev, "%s interface is stopped\n", netdev->name);
- return 0;
+ return ret;
}
/**
* @lio: per-network private data
* @start_stop: whether to start or stop
*/
-static void send_rx_ctrl_cmd(struct lio *lio, int start_stop)
+static int send_rx_ctrl_cmd(struct lio *lio, int start_stop)
{
struct octeon_device *oct = (struct octeon_device *)lio->oct_dev;
struct octeon_soft_command *sc;
int retval;
if (oct->props[lio->ifidx].rx_on == start_stop)
- return;
+ return 0;
sc = (struct octeon_soft_command *)
octeon_alloc_soft_command(oct, OCTNET_CMD_SIZE,
16, 0);
+ if (!sc) {
+ netif_info(lio, rx_err, lio->netdev,
+ "Failed to allocate octeon_soft_command struct\n");
+ return -ENOMEM;
+ }
ncmd = (union octnet_cmd *)sc->virtdptr;
*/
retval = wait_for_sc_completion_timeout(oct, sc, 0);
if (retval)
- return;
+ return retval;
oct->props[lio->ifidx].rx_on = start_stop;
WRITE_ONCE(sc->caller_is_done, true);
}
+
+ return retval;
}
/**
struct octeon_device_priv *oct_priv =
(struct octeon_device_priv *)oct->priv;
struct napi_struct *napi, *n;
+ int ret = 0;
if (!oct->props[lio->ifidx].napi_enabled) {
tasklet_disable(&oct_priv->droq_tasklet);
(LIQUIDIO_NDEV_STATS_POLL_TIME_MS));
/* tell Octeon to start forwarding packets to host */
- send_rx_ctrl_cmd(lio, 1);
+ ret = send_rx_ctrl_cmd(lio, 1);
+ if (ret)
+ return ret;
dev_info(&oct->pci_dev->dev, "%s interface is opened\n", netdev->name);
- return 0;
+ return ret;
}
/**
struct octeon_device_priv *oct_priv =
(struct octeon_device_priv *)oct->priv;
struct napi_struct *napi, *n;
+ int ret = 0;
/* tell Octeon to stop forwarding packets to host */
- send_rx_ctrl_cmd(lio, 0);
+ ret = send_rx_ctrl_cmd(lio, 0);
+ if (ret)
+ return ret;
netif_info(lio, ifdown, lio->netdev, "Stopping interface!\n");
/* Inform that netif carrier is down */
dev_info(&oct->pci_dev->dev, "%s interface is stopped\n", netdev->name);
- return 0;
+ return ret;
}
/**
struct phy_device *phydev;
int link_changed = 0;
- if (!lmac)
- return;
-
phydev = lmac->phydev;
if (!phydev->link && lmac->last_link)
unsigned int token; /* WR token */
dma_addr_t phys_addr; /* physical address of the ring */
struct sk_buff_head sendq; /* List of backpressured offload packets */
- struct tasklet_struct qresume_tsk; /* restarts the queue */
+ struct work_struct qresume_task; /* restarts the queue */
unsigned int cntxt_id; /* SGE context id for the Tx q */
unsigned long stops; /* # of times q has been stopped */
unsigned long restarts; /* # of queue restarts */
int phy_addr, const struct mdio_ops *mdio_ops);
int t3_aq100x_phy_prep(struct cphy *phy, struct adapter *adapter,
int phy_addr, const struct mdio_ops *mdio_ops);
+
+extern struct workqueue_struct *cxgb3_wq;
#endif /* __CHELSIO_COMMON_H */
free_irq(adap->msix_info[0].vec, adap);
goto irq_err;
}
- } else if ((err = request_irq(adap->pdev->irq,
- t3_intr_handler(adap,
- adap->sge.qs[0].rspq.
- polling),
- (adap->flags & USING_MSI) ?
- 0 : IRQF_SHARED,
- adap->name, adap)))
- goto irq_err;
+ } else {
+ err = request_irq(adap->pdev->irq,
+ t3_intr_handler(adap, adap->sge.qs[0].rspq.polling),
+ (adap->flags & USING_MSI) ? 0 : IRQF_SHARED,
+ adap->name, adap);
+ if (err)
+ goto irq_err;
+ }
enable_all_napi(adap);
t3_sge_start(adap);
nqsets = num_cpus;
if (nqsets < 1 || hwports == 4)
nqsets = 1;
- } else
+ } else {
nqsets = 1;
+ }
for_each_port(adap, i) {
struct port_info *pi = adap2pinfo(adap, i);
/**
- * free_qset - free the resources of an SGE queue set
+ * t3_free_qset - free the resources of an SGE queue set
* @adapter: the adapter owning the queue set
* @q: the queue set
*
}
/**
- * eth_xmit - add a packet to the Ethernet Tx queue
+ * t3_eth_xmit - add a packet to the Ethernet Tx queue
* @skb: the packet
* @dev: the egress net device
*
/**
* restart_ctrlq - restart a suspended control queue
- * @t: pointer to the tasklet associated with this handler
+ * @w: pointer to the work associated with this handler
*
* Resumes transmission on a suspended Tx control queue.
*/
-static void restart_ctrlq(struct tasklet_struct *t)
+static void restart_ctrlq(struct work_struct *w)
{
struct sk_buff *skb;
- struct sge_qset *qs = from_tasklet(qs, t, txq[TXQ_CTRL].qresume_tsk);
+ struct sge_qset *qs = container_of(w, struct sge_qset,
+ txq[TXQ_CTRL].qresume_task);
struct sge_txq *q = &qs->txq[TXQ_CTRL];
spin_lock(&q->lock);
/**
* restart_offloadq - restart a suspended offload queue
- * @t: pointer to the tasklet associated with this handler
+ * @w: pointer to the work associated with this handler
*
* Resumes transmission on a suspended Tx offload queue.
*/
-static void restart_offloadq(struct tasklet_struct *t)
+static void restart_offloadq(struct work_struct *w)
{
struct sk_buff *skb;
- struct sge_qset *qs = from_tasklet(qs, t, txq[TXQ_OFLD].qresume_tsk);
+ struct sge_qset *qs = container_of(w, struct sge_qset,
+ txq[TXQ_OFLD].qresume_task);
struct sge_txq *q = &qs->txq[TXQ_OFLD];
const struct port_info *pi = netdev_priv(qs->netdev);
struct adapter *adap = pi->adapter;
should_restart_tx(&qs->txq[TXQ_OFLD]) &&
test_and_clear_bit(TXQ_OFLD, &qs->txq_stopped)) {
qs->txq[TXQ_OFLD].restarts++;
- tasklet_schedule(&qs->txq[TXQ_OFLD].qresume_tsk);
+
+ /* The work can be quite lengthy so we use driver's own queue */
+ queue_work(cxgb3_wq, &qs->txq[TXQ_OFLD].qresume_task);
}
if (test_bit(TXQ_CTRL, &qs->txq_stopped) &&
should_restart_tx(&qs->txq[TXQ_CTRL]) &&
test_and_clear_bit(TXQ_CTRL, &qs->txq_stopped)) {
qs->txq[TXQ_CTRL].restarts++;
- tasklet_schedule(&qs->txq[TXQ_CTRL].qresume_tsk);
+
+ /* The work can be quite lengthy so we use driver's own queue */
+ queue_work(cxgb3_wq, &qs->txq[TXQ_CTRL].qresume_task);
}
}
skb_queue_head_init(&q->txq[i].sendq);
}
- tasklet_setup(&q->txq[TXQ_OFLD].qresume_tsk, restart_offloadq);
- tasklet_setup(&q->txq[TXQ_CTRL].qresume_tsk, restart_ctrlq);
+ INIT_WORK(&q->txq[TXQ_OFLD].qresume_task, restart_offloadq);
+ INIT_WORK(&q->txq[TXQ_CTRL].qresume_task, restart_ctrlq);
q->fl[0].gen = q->fl[1].gen = 1;
q->fl[0].size = p->fl_size;
*
* Can be invoked from interrupt context e.g. error handler.
*
- * Note that this function cannot disable the restart of tasklets as
+ * Note that this function cannot disable the restart of works as
* it cannot wait if called from interrupt context, however the
- * tasklets will have no effect since the doorbells are disabled. The
+ * works will have no effect since the doorbells are disabled. The
* driver will call tg3_sge_stop() later from process context, at
- * which time the tasklets will be stopped if they are still running.
+ * which time the works will be stopped if they are still running.
*/
void t3_sge_stop_dma(struct adapter *adap)
{
* @adap: the adapter
*
* Called from process context. Disables the DMA engine and any
- * pending queue restart tasklets.
+ * pending queue restart works.
*/
void t3_sge_stop(struct adapter *adap)
{
for (i = 0; i < SGE_QSETS; ++i) {
struct sge_qset *qs = &adap->sge.qs[i];
- tasklet_kill(&qs->txq[TXQ_OFLD].qresume_tsk);
- tasklet_kill(&qs->txq[TXQ_CTRL].qresume_tsk);
+ cancel_work_sync(&qs->txq[TXQ_OFLD].qresume_task);
+ cancel_work_sync(&qs->txq[TXQ_CTRL].qresume_task);
}
}
q->coalesce_usecs = 5;
q->rspq_size = 1024;
q->fl_size = 1024;
- q->jumbo_size = 512;
+ q->jumbo_size = 512;
q->txq_size[TXQ_ETH] = 1024;
q->txq_size[TXQ_OFLD] = 1024;
q->txq_size[TXQ_CTRL] = 256;
if (!list_empty(&ctbl->ce_free_head)) {
ce = list_first_entry(&ctbl->ce_free_head,
struct clip_entry, list);
- list_del(&ce->list);
- INIT_LIST_HEAD(&ce->list);
+ list_del_init(&ce->list);
spin_lock_init(&ce->lock);
refcount_set(&ce->refcnt, 0);
atomic_dec(&ctbl->nfree);
write_lock_bh(&ctbl->lock);
spin_lock_bh(&ce->lock);
if (refcount_dec_and_test(&ce->refcnt)) {
- list_del(&ce->list);
- INIT_LIST_HEAD(&ce->list);
+ list_del_init(&ce->list);
list_add_tail(&ce->list, &ctbl->ce_free_head);
atomic_inc(&ctbl->nfree);
if (v6)
bool persistent, u8 *smt_idx);
int cxgb4_get_msix_idx_from_bmap(struct adapter *adap);
void cxgb4_free_msix_idx_in_bmap(struct adapter *adap, u32 msix_idx);
-int cxgb_open(struct net_device *dev);
-int cxgb_close(struct net_device *dev);
void cxgb4_enable_rx(struct adapter *adap, struct sge_rspq *q);
void cxgb4_quiesce_rx(struct sge_rspq *q);
int cxgb4_port_mirror_alloc(struct net_device *dev);
return ret;
}
- spin_lock_bh(&adap->win0_lock);
+ /* We have to RESET the chip/firmware because we need the
+ * chip in uninitialized state for loading new PHY image.
+ * Otherwise, the running firmware will only store the PHY
+ * image in local RAM which will be lost after next reset.
+ */
+ ret = t4_fw_reset(adap, adap->mbox, PIORSTMODE_F | PIORST_F);
+ if (ret < 0) {
+ dev_err(adap->pdev_dev,
+ "Set FW to RESET for flashing PHY FW failed. ret: %d\n",
+ ret);
+ return ret;
+ }
+
ret = t4_load_phy_fw(adap, MEMWIN_NIC, NULL, data, size);
- spin_unlock_bh(&adap->win0_lock);
- if (ret)
- dev_err(adap->pdev_dev, "Failed to load PHY FW\n");
+ if (ret < 0) {
+ dev_err(adap->pdev_dev, "Failed to load PHY FW. ret: %d\n",
+ ret);
+ return ret;
+ }
- return ret;
+ return 0;
}
static int cxgb4_ethtool_flash_fw(struct net_device *netdev,
u32 ftid)
{
struct tid_info *t = &adap->tids;
- struct filter_entry *f;
- if (ftid < t->nhpftids)
- f = &adap->tids.hpftid_tab[ftid];
- else if (ftid < t->nftids)
- f = &adap->tids.ftid_tab[ftid - t->nhpftids];
- else
- f = lookup_tid(&adap->tids, ftid);
+ if (ftid >= t->hpftid_base && ftid < t->hpftid_base + t->nhpftids)
+ return &t->hpftid_tab[ftid - t->hpftid_base];
- return f;
+ if (ftid >= t->ftid_base && ftid < t->ftid_base + t->nftids)
+ return &t->ftid_tab[ftid - t->ftid_base];
+
+ return lookup_tid(t, ftid);
}
static void cxgb4_fill_filter_rule(struct ethtool_rx_flow_spec *fs,
filter_id = filter_info->loc_array[cmd->fs.location];
f = cxgb4_get_filter_entry(adapter, filter_id);
+ if (f->fs.prio)
+ filter_id -= adapter->tids.hpftid_base;
+ else if (!f->fs.hash)
+ filter_id -= (adapter->tids.ftid_base - adapter->tids.nhpftids);
+
ret = cxgb4_flow_rule_destroy(dev, f->fs.tc_prio, &f->fs, filter_id);
if (ret)
goto err;
filter_info = &adapter->ethtool_filters->port[pi->port_id];
+ if (fs.prio)
+ tid += adapter->tids.hpftid_base;
+ else if (!fs.hash)
+ tid += (adapter->tids.ftid_base - adapter->tids.nhpftids);
+
filter_info->loc_array[cmd->fs.location] = tid;
set_bit(cmd->fs.location, filter_info->bmap);
filter_info->in_use++;
WORD_MASK, f->fs.nat_lip[3] |
f->fs.nat_lip[2] << 8 |
f->fs.nat_lip[1] << 16 |
- (u64)f->fs.nat_lip[0] << 25, 1);
+ (u64)f->fs.nat_lip[0] << 24, 1);
}
}
cxgb4_del_filter(dev, f->tid, &f->fs);
}
- sb = t4_read_reg(adapter, LE_DB_SRVR_START_INDEX_A);
+ sb = adapter->tids.stid_base;
for (i = 0; i < sb; i++) {
f = (struct filter_entry *)adapter->tids.tid_tab[i];
/*
* net_device operations
*/
-int cxgb_open(struct net_device *dev)
+static int cxgb_open(struct net_device *dev)
{
struct port_info *pi = netdev_priv(dev);
struct adapter *adapter = pi->adapter;
return err;
}
-int cxgb_close(struct net_device *dev)
+static int cxgb_close(struct net_device *dev)
{
struct port_info *pi = netdev_priv(dev);
struct adapter *adapter = pi->adapter;
.ndo_set_vf_vlan = cxgb4_mgmt_set_vf_vlan,
.ndo_set_vf_link_state = cxgb4_mgmt_set_vf_link_state,
};
-#endif
static void cxgb4_mgmt_get_drvinfo(struct net_device *dev,
struct ethtool_drvinfo *info)
static const struct ethtool_ops cxgb4_mgmt_ethtool_ops = {
.get_drvinfo = cxgb4_mgmt_get_drvinfo,
};
+#endif
static void notify_fatal_err(struct work_struct *work)
{
/* Load PHY Firmware onto adapter.
*/
- spin_lock_bh(&adap->win0_lock);
ret = t4_load_phy_fw(adap, MEMWIN_NIC, phy_info->phy_fw_version,
(u8 *)phyf->data, phyf->size);
- spin_unlock_bh(&adap->win0_lock);
if (ret < 0)
dev_err(adap->pdev_dev, "PHY Firmware transfer error %d\n",
-ret);
adap->uld[CXGB4_ULD_KTLS].tlsdev_ops->tls_dev_del(netdev, tls_ctx,
direction);
- cxgb4_set_ktls_feature(adap, FW_PARAMS_PARAM_DEV_KTLS_HW_DISABLE);
out_unlock:
+ cxgb4_set_ktls_feature(adap, FW_PARAMS_PARAM_DEV_KTLS_HW_DISABLE);
mutex_unlock(&uld_mutex);
}
}
/**
- * cxgb4_ptp_remove - disable PTP device and stop the overflow check
+ * cxgb4_ptp_stop - disable PTP device and stop the overflow check
* @adapter: board private structure
*
* Stop the PTP support.
if (!ch_flower)
return -ENOENT;
+ rhashtable_remove_fast(&adap->flower_tbl, &ch_flower->node,
+ adap->flower_ht_params);
+
ret = cxgb4_flow_rule_destroy(dev, ch_flower->fs.tc_prio,
&ch_flower->fs, ch_flower->filter_id);
if (ret)
- goto err;
+ netdev_err(dev, "Flow rule destroy failed for tid: %u, ret: %d",
+ ch_flower->filter_id, ret);
- ret = rhashtable_remove_fast(&adap->flower_tbl, &ch_flower->node,
- adap->flower_ht_params);
- if (ret) {
- netdev_err(dev, "Flow remove from rhashtable failed");
- goto err;
- }
kfree_rcu(ch_flower, rcu);
-
-err:
return ret;
}
* down before configuring tc params.
*/
if (netif_running(dev)) {
- cxgb_close(dev);
+ netif_tx_stop_all_queues(dev);
+ netif_carrier_off(dev);
needs_bring_up = true;
}
}
out:
- if (needs_bring_up)
- cxgb_open(dev);
+ if (needs_bring_up) {
+ netif_tx_start_all_queues(dev);
+ netif_carrier_on(dev);
+ }
mutex_unlock(&adap->tc_mqprio->mqprio_mutex);
return ret;
if (!eosw_txq)
return -ENOMEM;
+ if (!(adap->flags & CXGB4_FW_OK)) {
+ /* Don't stall caller when access to FW is lost */
+ complete(&eosw_txq->completion);
+ return -EIO;
+ }
+
skb = alloc_skb(len, GFP_KERNEL);
if (!skb)
return -ENOMEM;
* @addr: the start address to write
* @n: length of data to write in bytes
* @data: the data to write
+ * @byte_oriented: whether to store data as bytes or as words
*
* Writes up to a page of data (256 bytes) to the serial flash starting
* at the given address. All the data must be written to the same page.
+ * If @byte_oriented is set the write data is stored as byte stream
+ * (i.e. matches what on disk), otherwise in big-endian.
*/
static int t4_write_flash(struct adapter *adapter, unsigned int addr,
- unsigned int n, const u8 *data)
+ unsigned int n, const u8 *data, bool byte_oriented)
{
- int ret;
- u32 buf[64];
unsigned int i, c, left, val, offset = addr & 0xff;
+ u32 buf[64];
+ int ret;
if (addr >= adapter->params.sf_size || offset + n > SF_PAGE_SIZE)
return -EINVAL;
(ret = sf1_write(adapter, 4, 1, 1, val)) != 0)
goto unlock;
- for (left = n; left; left -= c) {
+ for (left = n; left; left -= c, data += c) {
c = min(left, 4U);
- for (val = 0, i = 0; i < c; ++i)
- val = (val << 8) + *data++;
+ for (val = 0, i = 0; i < c; ++i) {
+ if (byte_oriented)
+ val = (val << 8) + data[i];
+ else
+ val = (val << 8) + data[c - i - 1];
+ }
ret = sf1_write(adapter, c, c != left, 1, val);
if (ret)
t4_write_reg(adapter, SF_OP_A, 0); /* unlock SF */
/* Read the page to verify the write succeeded */
- ret = t4_read_flash(adapter, addr & ~0xff, ARRAY_SIZE(buf), buf, 1);
+ ret = t4_read_flash(adapter, addr & ~0xff, ARRAY_SIZE(buf), buf,
+ byte_oriented);
if (ret)
return ret;
*/
memcpy(first_page, fw_data, SF_PAGE_SIZE);
((struct fw_hdr *)first_page)->fw_ver = cpu_to_be32(0xffffffff);
- ret = t4_write_flash(adap, fw_start, SF_PAGE_SIZE, first_page);
+ ret = t4_write_flash(adap, fw_start, SF_PAGE_SIZE, first_page, true);
if (ret)
goto out;
for (size -= SF_PAGE_SIZE; size; size -= SF_PAGE_SIZE) {
addr += SF_PAGE_SIZE;
fw_data += SF_PAGE_SIZE;
- ret = t4_write_flash(adap, addr, SF_PAGE_SIZE, fw_data);
+ ret = t4_write_flash(adap, addr, SF_PAGE_SIZE, fw_data, true);
if (ret)
goto out;
}
- ret = t4_write_flash(adap,
- fw_start + offsetof(struct fw_hdr, fw_ver),
- sizeof(hdr->fw_ver), (const u8 *)&hdr->fw_ver);
+ ret = t4_write_flash(adap, fw_start + offsetof(struct fw_hdr, fw_ver),
+ sizeof(hdr->fw_ver), (const u8 *)&hdr->fw_ver,
+ true);
out:
if (ret)
dev_err(adap->pdev_dev, "firmware download failed, error %d\n",
/* Copy the supplied PHY Firmware image to the adapter memory location
* allocated by the adapter firmware.
*/
+ spin_lock_bh(&adap->win0_lock);
ret = t4_memory_rw(adap, win, mtype, maddr,
phy_fw_size, (__be32 *)phy_fw_data,
T4_MEMORY_WRITE);
+ spin_unlock_bh(&adap->win0_lock);
if (ret)
return ret;
}
/**
- * t4_init_cmd - ask FW to initialize the device
+ * t4_early_init - ask FW to initialize the device
* @adap: the adapter
* @mbox: mailbox to use for the FW command
*
int idx, bool sleep_ok)
{
struct fw_vi_mac_exact *p;
- u8 addr[] = {0, 0, 0, 0, 0, 0};
struct fw_vi_mac_cmd c;
int ret = 0;
u32 exact;
p = c.u.exact;
p->valid_to_idx = cpu_to_be16(FW_VI_MAC_CMD_VALID_F |
FW_VI_MAC_CMD_IDX_V(idx));
- memcpy(p->macaddr, addr, sizeof(p->macaddr));
+ eth_zero_addr(p->macaddr);
ret = t4_wr_mbox_meat(adap, adap->mbox, &c, sizeof(c), &c, sleep_ok);
return ret;
}
n = size - i;
else
n = SF_PAGE_SIZE;
- ret = t4_write_flash(adap, addr, n, cfg_data);
+ ret = t4_write_flash(adap, addr, n, cfg_data, true);
if (ret)
goto out;
}
/**
- * t4_set_vf_mac - Set MAC address for the specified VF
+ * t4_set_vf_mac_acl - Set MAC address for the specified VF
* @adapter: The adapter
* @vf: one of the VFs instantiated by the specified PF
* @naddr: the number of MAC addresses
for (size -= SF_PAGE_SIZE; size; size -= SF_PAGE_SIZE) {
addr += SF_PAGE_SIZE;
boot_data += SF_PAGE_SIZE;
- ret = t4_write_flash(adap, addr, SF_PAGE_SIZE, boot_data);
+ ret = t4_write_flash(adap, addr, SF_PAGE_SIZE, boot_data,
+ false);
if (ret)
goto out;
}
ret = t4_write_flash(adap, boot_sector, SF_PAGE_SIZE,
- (const u8 *)header);
+ (const u8 *)header, false);
out:
if (ret)
for (i = 0; i < size; i += SF_PAGE_SIZE) {
n = min_t(u32, size - i, SF_PAGE_SIZE);
- ret = t4_write_flash(adap, addr, n, cfg_data);
+ ret = t4_write_flash(adap, addr, n, cfg_data, false);
if (ret)
goto out;
for (i = 0; i < npad; i++) {
u8 data = 0;
- ret = t4_write_flash(adap, cfg_addr + size + i, 1, &data);
+ ret = t4_write_flash(adap, cfg_addr + size + i, 1, &data,
+ false);
if (ret)
goto out;
}
}
/**
- * check_ring_tx_db - check and potentially ring a TX queue's doorbell
+ * ring_tx_db - check and potentially ring a TX queue's doorbell
* @adapter: the adapter
* @tq: the TX queue
* @n: number of new descriptors to give to HW
}
static int chcr_init_tcb_fields(struct chcr_ktls_info *tx_info);
+static void clear_conn_resources(struct chcr_ktls_info *tx_info);
/*
* chcr_ktls_save_keys: calculate and save crypto keys.
* @tx_info - driver specific tls info.
chcr_get_ktls_tx_context(tls_ctx);
struct chcr_ktls_info *tx_info = tx_ctx->chcr_info;
struct ch_ktls_port_stats_debug *port_stats;
+ struct chcr_ktls_uld_ctx *u_ctx;
if (!tx_info)
return;
+ u_ctx = tx_info->adap->uld[CXGB4_ULD_KTLS].handle;
+ if (u_ctx && u_ctx->detach)
+ return;
/* clear l2t entry */
if (tx_info->l2te)
cxgb4_l2t_release(tx_info->l2te);
if (tx_info->tid != -1) {
cxgb4_remove_tid(&tx_info->adap->tids, tx_info->tx_chan,
tx_info->tid, tx_info->ip_family);
+
+ xa_erase(&u_ctx->tid_list, tx_info->tid);
}
port_stats = &tx_info->adap->ch_ktls_stats.ktls_port[tx_info->port_id];
struct tls_context *tls_ctx = tls_get_ctx(sk);
struct ch_ktls_port_stats_debug *port_stats;
struct chcr_ktls_ofld_ctx_tx *tx_ctx;
+ struct chcr_ktls_uld_ctx *u_ctx;
struct chcr_ktls_info *tx_info;
struct dst_entry *dst;
struct adapter *adap;
adap = pi->adapter;
port_stats = &adap->ch_ktls_stats.ktls_port[pi->port_id];
atomic64_inc(&port_stats->ktls_tx_connection_open);
+ u_ctx = adap->uld[CXGB4_ULD_KTLS].handle;
if (direction == TLS_OFFLOAD_CTX_DIR_RX) {
pr_err("not expecting for RX direction\n");
if (tx_ctx->chcr_info)
goto out;
+ if (u_ctx && u_ctx->detach)
+ goto out;
+
tx_info = kvzalloc(sizeof(*tx_info), GFP_KERNEL);
if (!tx_info)
goto out;
cxgb4_remove_tid(&tx_info->adap->tids, tx_info->tx_chan,
tx_info->tid, tx_info->ip_family);
+ xa_erase(&u_ctx->tid_list, tx_info->tid);
+
put_module:
/* release module refcount */
module_put(THIS_MODULE);
{
const struct cpl_act_open_rpl *p = (void *)input;
struct chcr_ktls_info *tx_info = NULL;
+ struct chcr_ktls_ofld_ctx_tx *tx_ctx;
+ struct chcr_ktls_uld_ctx *u_ctx;
unsigned int atid, tid, status;
+ struct tls_context *tls_ctx;
struct tid_info *t;
+ int ret = 0;
tid = GET_TID(p);
status = AOPEN_STATUS_G(ntohl(p->atid_status));
if (!status) {
tx_info->tid = tid;
cxgb4_insert_tid(t, tx_info, tx_info->tid, tx_info->ip_family);
+ /* Adding tid */
+ tls_ctx = tls_get_ctx(tx_info->sk);
+ tx_ctx = chcr_get_ktls_tx_context(tls_ctx);
+ u_ctx = adap->uld[CXGB4_ULD_KTLS].handle;
+ if (u_ctx) {
+ ret = xa_insert_bh(&u_ctx->tid_list, tid, tx_ctx,
+ GFP_NOWAIT);
+ if (ret < 0) {
+ pr_err("%s: Failed to allocate tid XA entry = %d\n",
+ __func__, tx_info->tid);
+ tx_info->open_state = CH_KTLS_OPEN_FAILURE;
+ goto out;
+ }
+ }
tx_info->open_state = CH_KTLS_OPEN_SUCCESS;
} else {
tx_info->open_state = CH_KTLS_OPEN_FAILURE;
}
+out:
spin_unlock(&tx_info->lock);
complete(&tx_info->completion);
- return 0;
+ return ret;
}
/*
goto out;
}
u_ctx->lldi = *lldi;
+ u_ctx->detach = false;
+ xa_init_flags(&u_ctx->tid_list, XA_FLAGS_LOCK_BH);
out:
return u_ctx;
}
return 0;
}
+static void clear_conn_resources(struct chcr_ktls_info *tx_info)
+{
+ /* clear l2t entry */
+ if (tx_info->l2te)
+ cxgb4_l2t_release(tx_info->l2te);
+
+#if IS_ENABLED(CONFIG_IPV6)
+ /* clear clip entry */
+ if (tx_info->ip_family == AF_INET6)
+ cxgb4_clip_release(tx_info->netdev, (const u32 *)
+ &tx_info->sk->sk_v6_rcv_saddr,
+ 1);
+#endif
+
+ /* clear tid */
+ if (tx_info->tid != -1)
+ cxgb4_remove_tid(&tx_info->adap->tids, tx_info->tx_chan,
+ tx_info->tid, tx_info->ip_family);
+}
+
+static void ch_ktls_reset_all_conn(struct chcr_ktls_uld_ctx *u_ctx)
+{
+ struct ch_ktls_port_stats_debug *port_stats;
+ struct chcr_ktls_ofld_ctx_tx *tx_ctx;
+ struct chcr_ktls_info *tx_info;
+ unsigned long index;
+
+ xa_for_each(&u_ctx->tid_list, index, tx_ctx) {
+ tx_info = tx_ctx->chcr_info;
+ clear_conn_resources(tx_info);
+ port_stats = &tx_info->adap->ch_ktls_stats.ktls_port[tx_info->port_id];
+ atomic64_inc(&port_stats->ktls_tx_connection_close);
+ kvfree(tx_info);
+ tx_ctx->chcr_info = NULL;
+ /* release module refcount */
+ module_put(THIS_MODULE);
+ }
+}
+
static int chcr_ktls_uld_state_change(void *handle, enum cxgb4_state new_state)
{
struct chcr_ktls_uld_ctx *u_ctx = handle;
case CXGB4_STATE_DETACH:
pr_info("%s: Down\n", pci_name(u_ctx->lldi.pdev));
mutex_lock(&dev_mutex);
+ u_ctx->detach = true;
list_del(&u_ctx->entry);
+ ch_ktls_reset_all_conn(u_ctx);
+ xa_destroy(&u_ctx->tid_list);
mutex_unlock(&dev_mutex);
break;
default:
adap = pci_get_drvdata(u_ctx->lldi.pdev);
memset(&adap->ch_ktls_stats, 0, sizeof(adap->ch_ktls_stats));
list_del(&u_ctx->entry);
+ xa_destroy(&u_ctx->tid_list);
kfree(u_ctx);
}
mutex_unlock(&dev_mutex);
struct chcr_ktls_uld_ctx {
struct list_head entry;
struct cxgb4_lld_info lldi;
+ struct xarray tid_list;
+ bool detach;
};
static inline struct chcr_ktls_ofld_ctx_tx *
cerr = put_cmsg(msg, SOL_TLS, TLS_GET_RECORD_TYPE,
sizeof(thdr->type), &thdr->type);
- if (cerr && thdr->type != TLS_RECORD_TYPE_DATA)
- return -EIO;
+ if (cerr && thdr->type != TLS_RECORD_TYPE_DATA) {
+ copied = -EIO;
+ break;
+ }
/* don't send tls header, skip copy */
goto skip_copy;
}
struct device *dev = &pdev->dev;
struct gemini_ethernet *geth;
struct net_device *netdev;
- struct resource *gmacres;
- struct resource *dmares;
struct device *parent;
unsigned int id;
int irq;
port->msg_enable = netif_msg_init(debug, DEFAULT_MSG_ENABLE);
/* DMA memory */
- dmares = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!dmares) {
- dev_err(dev, "no DMA resource\n");
- return -ENODEV;
- }
- port->dma_base = devm_ioremap_resource(dev, dmares);
- if (IS_ERR(port->dma_base))
+ port->dma_base = devm_platform_get_and_ioremap_resource(pdev, 0, NULL);
+ if (IS_ERR(port->dma_base)) {
+ dev_err(dev, "get DMA address failed\n");
return PTR_ERR(port->dma_base);
+ }
/* GMAC config memory */
- gmacres = platform_get_resource(pdev, IORESOURCE_MEM, 1);
- if (!gmacres) {
- dev_err(dev, "no GMAC resource\n");
- return -ENODEV;
- }
- port->gmac_base = devm_ioremap_resource(dev, gmacres);
- if (IS_ERR(port->gmac_base))
+ port->gmac_base = devm_platform_get_and_ioremap_resource(pdev, 1, NULL);
+ if (IS_ERR(port->gmac_base)) {
+ dev_err(dev, "get GMAC address failed\n");
return PTR_ERR(port->gmac_base);
+ }
/* Interrupt */
irq = platform_get_irq(pdev, 0);
if (ret)
goto unprepare;
- netdev_info(netdev,
- "irq %d, DMA @ 0x%pap, GMAC @ 0x%pap\n",
- port->irq, &dmares->start,
- &gmacres->start);
return 0;
unprepare:
struct device *dev = &pdev->dev;
struct gemini_ethernet *geth;
unsigned int retry = 5;
- struct resource *res;
u32 val;
/* Global registers */
geth = devm_kzalloc(dev, sizeof(*geth), GFP_KERNEL);
if (!geth)
return -ENOMEM;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!res)
- return -ENODEV;
- geth->base = devm_ioremap_resource(dev, res);
+ geth->base = devm_platform_get_and_ioremap_resource(pdev, 0, NULL);
if (IS_ERR(geth->base))
return PTR_ERR(geth->base);
geth->dev = dev;
/* The chip only need report frame silently dropped. */
spin_lock_irq(&de->lock);
- if (netif_running(dev) && netif_device_present(dev))
- __de_get_stats(de);
+ if (netif_running(dev) && netif_device_present(dev))
+ __de_get_stats(de);
spin_unlock_irq(&de->lock);
return &dev->stats;
<earl@exis.net>.
Updated the PCI interface to conform with the latest
version. I hope nothing is broken...
- Add TX done interrupt modification from suggestion
+ Add TX done interrupt modification from suggestion
by <Austin.Donnelly@cl.cam.ac.uk>.
Fix is_anc_capable() bug reported by
<Austin.Donnelly@cl.cam.ac.uk>.
spin_lock_irqsave(&lp->lock, flags);
netif_stop_queue(dev);
load_packet(dev, skb->data, TD_IC | TD_LS | TD_FS | skb->len, skb);
- lp->stats.tx_bytes += skb->len;
+ lp->stats.tx_bytes += skb->len;
outl(POLL_DEMAND, DE4X5_TPD);/* Start the TX */
lp->tx_new = (lp->tx_new + 1) % lp->txRingSize;
/* Update stats */
lp->stats.rx_packets++;
- lp->stats.rx_bytes += pkt_len;
+ lp->stats.rx_bytes += pkt_len;
}
}
DMFE_DBUG(0, "dmfe_remove_one()", 0);
- if (dev) {
+ if (dev) {
unregister_netdev(dev);
pci_iounmap(db->pdev, db->ioaddr);
/* CR6 operation mode decision */
if ( !chkmode || (db->chip_id == PCI_DM9132_ID) ||
(db->chip_revision >= 0x30) ) {
- db->cr6_data |= DMFE_TXTH_256;
+ db->cr6_data |= DMFE_TXTH_256;
db->cr0_data = CR0_DEFAULT;
db->dm910x_chk_mode=4; /* Enter the normal mode */
- } else {
+ } else {
db->cr6_data |= CR6_SFT; /* Store & Forward mode */
db->cr0_data = 0;
db->dm910x_chk_mode = 1; /* Enter the check mode */
}
}
- txptr = txptr->next_tx_desc;
+ txptr = txptr->next_tx_desc;
}/* End of while */
/* Update TX remove pointer to next */
void __iomem *ioaddr = db->ioaddr;
u32 tmp_cr8;
unsigned char tmp_cr12;
- unsigned long flags;
+ unsigned long flags;
int link_ok, link_ok_phy;
if (link_ok_phy != link_ok) {
DMFE_DBUG (0, "PHY and chip report different link status", 0);
link_ok = link_ok | link_ok_phy;
- }
+ }
if ( !link_ok && netif_carrier_ok(dev)) {
/* Link Failed */
if (db->chip_id == PCI_DM9009_ID) phy_reg &= 0x61;
}
- /* Write new capability to Phyxcer Reg4 */
+ /* Write new capability to Phyxcer Reg4 */
if ( !(phy_reg & 0x01e0)) {
phy_reg|=db->PHY_reg4;
db->media_mode|=DMFE_AUTO;
}
dmfe_phy_write(db->ioaddr, db->phy_addr, 4, phy_reg, db->chip_id);
- /* Restart Auto-Negotiation */
+ /* Restart Auto-Negotiation */
if ( db->chip_type && (db->chip_id == PCI_DM9102_ID) )
dmfe_phy_write(db->ioaddr, db->phy_addr, 0, 0x1800, db->chip_id);
if ( !db->chip_type )
}
dmfe_phy_write(db->ioaddr,
db->phy_addr, 0, phy_reg, db->chip_id);
- if ( db->chip_type && (db->chip_id == PCI_DM9102_ID) )
+ if ( db->chip_type && (db->chip_id == PCI_DM9102_ID) )
mdelay(20);
dmfe_phy_write(db->ioaddr,
db->phy_addr, 0, phy_reg, db->chip_id);
del_timer_sync(&tp->timer);
pnic2_start_nway(dev);
tp->timer.expires = RUN_AT(3*HZ);
- add_timer(&tp->timer);
+ add_timer(&tp->timer);
}
return;
del_timer_sync(&tp->timer);
pnic2_start_nway(dev);
tp->timer.expires = RUN_AT(3*HZ);
- add_timer(&tp->timer);
+ add_timer(&tp->timer);
}
return;
void pnic2_lnk_change(struct net_device *dev, int csr5);
void pnic2_timer(struct timer_list *t);
void pnic2_start_nway(struct net_device *dev);
-void pnic2_lnk_change(struct net_device *dev, int csr5);
/* eeprom.c */
void tulip_parse_eeprom(struct net_device *dev);
}
}
- txptr = txptr->next_tx_desc;
+ txptr = txptr->next_tx_desc;
}/* End of while */
/* Update TX remove pointer to next */
struct net_device *dev = pci_get_drvdata(db->pdev);
struct uli_phy_ops *phy = &db->phy;
void __iomem *ioaddr = db->ioaddr;
- unsigned long flags;
+ unsigned long flags;
u8 tmp_cr12 = 0;
u32 tmp_cr8;
}
- /* Write new capability to Phyxcer Reg4 */
+ /* Write new capability to Phyxcer Reg4 */
if ( !(phy_reg & 0x01e0)) {
phy_reg|=db->PHY_reg4;
db->media_mode|=ULI526X_AUTO;
}
phy->write(db, db->phy_addr, 4, phy_reg);
- /* Restart Auto-Negotiation */
+ /* Restart Auto-Negotiation */
phy->write(db, db->phy_addr, 0, 0x1200);
udelay(50);
}
/*
* Process op-mode
- AUTO mode : PHY controller in Auto-negotiation Mode
+ AUTO mode : PHY controller in Auto-negotiation Mode
* Force mode: PHY controller in force mode with HUB
* N-way force capability with SWITCH
*/
power management.
support for big endian descriptors
Copyright (C) 2001 Manfred Spraul
- * ethtool support (jgarzik)
+ * ethtool support (jgarzik)
* Replace some MII-related magic numbers with constants (jgarzik)
TODO:
np->cur_rx, np->dirty_rx);
}
- /* Stop the chip's Tx and Rx processes. */
+ /* Stop the chip's Tx and Rx processes. */
spin_lock_irq(&np->lock);
netif_device_detach(dev);
update_csr6(dev, 0);
/* Stop the chip's Tx and Rx processes. */
iowrite16(TxDisable | RxDisable | StatsDisable, ioaddr + MACCtrl1);
- for (i = 2000; i > 0; i--) {
- if ((ioread32(ioaddr + DMACtrl) & 0xc000) == 0)
+ for (i = 2000; i > 0; i--) {
+ if ((ioread32(ioaddr + DMACtrl) & 0xc000) == 0)
break;
mdelay(1);
- }
+ }
- iowrite16(GlobalReset | DMAReset | FIFOReset | NetworkReset,
+ iowrite16(GlobalReset | DMAReset | FIFOReset | NetworkReset,
ioaddr + ASIC_HI_WORD(ASICCtrl));
- for (i = 2000; i > 0; i--) {
+ for (i = 2000; i > 0; i--) {
if ((ioread16(ioaddr + ASIC_HI_WORD(ASICCtrl)) & ResetBusy) == 0)
break;
mdelay(1);
- }
+ }
#ifdef __i386__
if (netif_msg_hw(np)) {
struct ec_bhf_priv *priv = netdev_priv(net_dev);
unregister_netdev(net_dev);
- free_netdev(net_dev);
pci_iounmap(dev, priv->dma_io);
pci_iounmap(dev, priv->io);
+
+ free_netdev(net_dev);
+
pci_release_regions(dev);
pci_clear_master(dev);
pci_disable_device(dev);
unmap_bars:
be_unmap_pci_bars(adapter);
free_netdev:
+ pci_disable_pcie_error_reporting(pdev);
free_netdev(netdev);
rel_reg:
pci_release_regions(pdev);
if (np && of_get_property(np, "use-ncsi", NULL)) {
if (!IS_ENABLED(CONFIG_NET_NCSI)) {
dev_err(&pdev->dev, "NCSI stack not enabled\n");
+ err = -EINVAL;
goto err_phy_connect;
}
dev_info(&pdev->dev, "Using NCSI interface\n");
priv->use_ncsi = true;
priv->ndev = ncsi_register_dev(netdev, ftgmac100_ncsi_handler);
- if (!priv->ndev)
+ if (!priv->ndev) {
+ err = -EINVAL;
goto err_phy_connect;
+ }
} else if (np && of_get_property(np, "phy-handle", NULL)) {
struct phy_device *phy;
&ftgmac100_adjust_link);
if (!phy) {
dev_err(&pdev->dev, "Failed to connect to phy\n");
+ err = -EINVAL;
goto err_phy_connect;
}
};
static const struct chip_info skel_netdrv_tbl[] = {
- { "100/10M Ethernet PCI Adapter", HAS_MII_XCVR },
+ { "100/10M Ethernet PCI Adapter", HAS_MII_XCVR },
{ "100/10M Ethernet PCI Adapter", HAS_CHIP_XCVR },
{ "1000/100/10M Ethernet PCI Adapter", HAS_MII_XCVR },
};
void dpaa2_dbg_add(struct dpaa2_eth_priv *priv)
{
+ struct fsl_mc_device *dpni_dev;
struct dentry *dir;
+ char name[10];
/* Create a directory for the interface */
- dir = debugfs_create_dir(priv->net_dev->name, dpaa2_dbg_root);
+ dpni_dev = to_fsl_mc_device(priv->net_dev->dev.parent);
+ snprintf(name, 10, "dpni.%d", dpni_dev->obj_desc.id);
+ dir = debugfs_create_dir(name, dpaa2_dbg_root);
priv->dbg.dir = dir;
/* per-cpu stats file */
if (dpaa2_eth_is_type_phy(priv)) {
err = dpaa2_mac_connect(mac);
- if (err) {
- netdev_err(priv->net_dev, "Error connecting to the MAC endpoint\n");
+ if (err && err != -EPROBE_DEFER)
+ netdev_err(priv->net_dev, "Error connecting to the MAC endpoint: %pe",
+ ERR_PTR(err));
+ if (err)
goto err_close_mac;
- }
}
return 0;
// SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause)
/* Copyright 2019 NXP */
+#include <linux/acpi.h>
+#include <linux/property.h>
+
#include "dpaa2-eth.h"
#include "dpaa2-mac.h"
return 0;
}
-/* Caller must call of_node_put on the returned value */
-static struct device_node *dpaa2_mac_get_node(u16 dpmac_id)
+static struct fwnode_handle *dpaa2_mac_get_node(struct device *dev,
+ u16 dpmac_id)
{
- struct device_node *dpmacs, *dpmac = NULL;
- u32 id;
+ struct fwnode_handle *fwnode, *parent, *child = NULL;
+ struct device_node *dpmacs = NULL;
int err;
+ u32 id;
- dpmacs = of_find_node_by_name(NULL, "dpmacs");
- if (!dpmacs)
- return NULL;
+ fwnode = dev_fwnode(dev->parent);
+ if (is_of_node(fwnode)) {
+ dpmacs = of_find_node_by_name(NULL, "dpmacs");
+ if (!dpmacs)
+ return NULL;
+ parent = of_fwnode_handle(dpmacs);
+ } else if (is_acpi_node(fwnode)) {
+ parent = fwnode;
+ }
- while ((dpmac = of_get_next_child(dpmacs, dpmac)) != NULL) {
- err = of_property_read_u32(dpmac, "reg", &id);
+ fwnode_for_each_child_node(parent, child) {
+ err = -EINVAL;
+ if (is_acpi_device_node(child))
+ err = acpi_get_local_address(ACPI_HANDLE_FWNODE(child), &id);
+ else if (is_of_node(child))
+ err = of_property_read_u32(to_of_node(child), "reg", &id);
if (err)
continue;
- if (id == dpmac_id)
- break;
- }
+ if (id == dpmac_id) {
+ of_node_put(dpmacs);
+ return child;
+ }
+ }
of_node_put(dpmacs);
-
- return dpmac;
+ return NULL;
}
-static int dpaa2_mac_get_if_mode(struct device_node *node,
+static int dpaa2_mac_get_if_mode(struct fwnode_handle *dpmac_node,
struct dpmac_attr attr)
{
phy_interface_t if_mode;
int err;
- err = of_get_phy_mode(node, &if_mode);
- if (!err)
- return if_mode;
+ err = fwnode_get_phy_mode(dpmac_node);
+ if (err > 0)
+ return err;
err = phy_mode(attr.eth_if, &if_mode);
if (!err)
};
static int dpaa2_pcs_create(struct dpaa2_mac *mac,
- struct device_node *dpmac_node, int id)
+ struct fwnode_handle *dpmac_node,
+ int id)
{
struct mdio_device *mdiodev;
- struct device_node *node;
+ struct fwnode_handle *node;
- node = of_parse_phandle(dpmac_node, "pcs-handle", 0);
- if (!node) {
+ node = fwnode_find_reference(dpmac_node, "pcs-handle", 0);
+ if (IS_ERR(node)) {
/* do not error out on old DTS files */
netdev_warn(mac->net_dev, "pcs-handle node not found\n");
return 0;
}
- if (!of_device_is_available(node)) {
+ if (!fwnode_device_is_available(node)) {
netdev_err(mac->net_dev, "pcs-handle node not available\n");
- of_node_put(node);
+ fwnode_handle_put(node);
return -ENODEV;
}
- mdiodev = of_mdio_find_device(node);
- of_node_put(node);
+ mdiodev = fwnode_mdio_find_device(node);
+ fwnode_handle_put(node);
if (!mdiodev)
return -EPROBE_DEFER;
int dpaa2_mac_connect(struct dpaa2_mac *mac)
{
struct net_device *net_dev = mac->net_dev;
- struct device_node *dpmac_node;
+ struct fwnode_handle *dpmac_node;
struct phylink *phylink;
int err;
mac->if_link_type = mac->attr.link_type;
- dpmac_node = dpaa2_mac_get_node(mac->attr.id);
+ dpmac_node = mac->fw_node;
if (!dpmac_node) {
netdev_err(net_dev, "No dpmac@%d node found.\n", mac->attr.id);
return -ENODEV;
}
err = dpaa2_mac_get_if_mode(dpmac_node, mac->attr);
- if (err < 0) {
- err = -EINVAL;
- goto err_put_node;
- }
+ if (err < 0)
+ return -EINVAL;
mac->if_mode = err;
/* The MAC does not have the capability to add RGMII delays so
* error out if the interface mode requests them and there is no PHY
* to act upon them
*/
- if (of_phy_is_fixed_link(dpmac_node) &&
+ if (of_phy_is_fixed_link(to_of_node(dpmac_node)) &&
(mac->if_mode == PHY_INTERFACE_MODE_RGMII_ID ||
mac->if_mode == PHY_INTERFACE_MODE_RGMII_RXID ||
mac->if_mode == PHY_INTERFACE_MODE_RGMII_TXID)) {
netdev_err(net_dev, "RGMII delay not supported\n");
- err = -EINVAL;
- goto err_put_node;
+ return -EINVAL;
}
if ((mac->attr.link_type == DPMAC_LINK_TYPE_PHY &&
mac->attr.link_type == DPMAC_LINK_TYPE_BACKPLANE) {
err = dpaa2_pcs_create(mac, dpmac_node, mac->attr.id);
if (err)
- goto err_put_node;
+ return err;
}
mac->phylink_config.dev = &net_dev->dev;
mac->phylink_config.type = PHYLINK_NETDEV;
phylink = phylink_create(&mac->phylink_config,
- of_fwnode_handle(dpmac_node), mac->if_mode,
+ dpmac_node, mac->if_mode,
&dpaa2_mac_phylink_ops);
if (IS_ERR(phylink)) {
err = PTR_ERR(phylink);
if (mac->pcs)
phylink_set_pcs(mac->phylink, &mac->pcs->pcs);
- err = phylink_of_phy_connect(mac->phylink, dpmac_node, 0);
+ err = phylink_fwnode_phy_connect(mac->phylink, dpmac_node, 0);
if (err) {
- netdev_err(net_dev, "phylink_of_phy_connect() = %d\n", err);
+ netdev_err(net_dev, "phylink_fwnode_phy_connect() = %d\n", err);
goto err_phylink_destroy;
}
- of_node_put(dpmac_node);
-
return 0;
err_phylink_destroy:
phylink_destroy(mac->phylink);
err_pcs_destroy:
dpaa2_pcs_destroy(mac);
-err_put_node:
- of_node_put(dpmac_node);
return err;
}
goto err_close_dpmac;
}
+ /* Find the device node representing the MAC device and link the device
+ * behind the associated netdev to it.
+ */
+ mac->fw_node = dpaa2_mac_get_node(&mac->mc_dev->dev, mac->attr.id);
+ net_dev->dev.of_node = to_of_node(mac->fw_node);
+
return 0;
err_close_dpmac:
struct fsl_mc_device *dpmac_dev = mac->mc_dev;
dpmac_close(mac->mc_io, 0, dpmac_dev->mc_handle);
+ if (mac->fw_node)
+ fwnode_handle_put(mac->fw_node);
}
static char dpaa2_mac_ethtool_stats[][ETH_GSTRING_LEN] = {
phy_interface_t if_mode;
enum dpmac_link_type if_link_type;
struct lynx_pcs *pcs;
+ struct fwnode_handle *fw_node;
};
bool dpaa2_mac_is_type_fixed(struct fsl_mc_device *dpmac_dev,
static int enetc_ierb_probe(struct platform_device *pdev)
{
struct enetc_ierb *ierb;
- struct resource *res;
void __iomem *regs;
ierb = devm_kzalloc(&pdev->dev, sizeof(*ierb), GFP_KERNEL);
if (!ierb)
return -ENOMEM;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- regs = devm_ioremap_resource(&pdev->dev, res);
+ regs = devm_platform_get_and_ioremap_resource(pdev, 0, NULL);
if (IS_ERR(regs))
return PTR_ERR(regs);
// SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause)
/* Copyright 2017-2019 NXP */
+#include <asm/unaligned.h>
#include <linux/mdio.h>
#include <linux/module.h>
#include <linux/fsl/enetc_mdio.h>
u32 upper = __raw_readl(hw->port + ENETC_PSIPMAR0(si));
u16 lower = __raw_readw(hw->port + ENETC_PSIPMAR1(si));
- *(u32 *)addr = upper;
- *(u16 *)(addr + 4) = lower;
+ put_unaligned_le32(upper, addr);
+ put_unaligned_le16(lower, addr + 4);
}
static void enetc_pf_set_primary_mac_addr(struct enetc_hw *hw, int si,
const u8 *addr)
{
- u32 upper = *(const u32 *)addr;
- u16 lower = *(const u16 *)(addr + 4);
+ u32 upper = get_unaligned_le32(addr);
+ u16 lower = get_unaligned_le16(addr + 4);
__raw_writel(upper, hw->port + ENETC_PSIPMAR0(si));
__raw_writew(lower, hw->port + ENETC_PSIPMAR1(si));
struct streamid_conf *si_conf;
u16 data_size;
dma_addr_t dma;
+ int port;
int err;
+ port = enetc_pf_to_port(priv->si->pdev);
+ if (port < 0)
+ return -EINVAL;
+
if (sid->index >= priv->psfp_cap.max_streamid)
return -EINVAL;
si_conf = &cbd.sid_set;
/* Only one port supported for one entry, set itself */
- si_conf->iports = cpu_to_le32(1 << enetc_pf_to_port(priv->si->pdev));
+ si_conf->iports = cpu_to_le32(1 << port);
si_conf->id_type = 1;
si_conf->oui[2] = 0x0;
si_conf->oui[1] = 0x80;
si_conf->en = 0x80;
si_conf->stream_handle = cpu_to_le32(sid->handle);
- si_conf->iports = cpu_to_le32(1 << enetc_pf_to_port(priv->si->pdev));
+ si_conf->iports = cpu_to_le32(1 << port);
si_conf->id_type = sid->filtertype;
si_conf->oui[2] = 0x0;
si_conf->oui[1] = 0x80;
{
struct enetc_cbd cbd = {.cmd = 0};
struct sfi_conf *sfi_config;
+ int port;
+
+ port = enetc_pf_to_port(priv->si->pdev);
+ if (port < 0)
+ return -EINVAL;
cbd.index = cpu_to_le16(sfi->index);
cbd.cls = BDCR_CMD_STREAM_FILTER;
}
sfi_config->sg_inst_table_index = cpu_to_le16(sfi->gate_id);
- sfi_config->input_ports =
- cpu_to_le32(1 << enetc_pf_to_port(priv->si->pdev));
+ sfi_config->input_ports = cpu_to_le32(1 << port);
/* The priority value which may be matched against the
* frame’s priority value to determine a match for this entry.
{
struct enetc_ndev_priv *priv = netdev_priv(ndev);
struct flow_block_offload *f = type_data;
- int err;
+ int port, err;
err = flow_block_cb_setup_simple(f, &enetc_block_cb_list,
enetc_setup_tc_block_cb,
switch (f->command) {
case FLOW_BLOCK_BIND:
- set_bit(enetc_pf_to_port(priv->si->pdev), &epsfp.dev_bitmap);
+ port = enetc_pf_to_port(priv->si->pdev);
+ if (port < 0)
+ return -EINVAL;
+
+ set_bit(port, &epsfp.dev_bitmap);
break;
case FLOW_BLOCK_UNBIND:
- clear_bit(enetc_pf_to_port(priv->si->pdev), &epsfp.dev_bitmap);
+ port = enetc_pf_to_port(priv->si->pdev);
+ if (port < 0)
+ return -EINVAL;
+
+ clear_bit(port, &epsfp.dev_bitmap);
if (!epsfp.dev_bitmap)
clean_psfp_all();
break;
*/
#define FEC_QUIRK_NO_HARD_RESET (1 << 18)
+/* i.MX6SX ENET IP supports multiple queues (3 queues), use this quirk to
+ * represents this ENET IP.
+ */
+#define FEC_QUIRK_HAS_MULTI_QUEUES (1 << 19)
+
struct bufdesc_prop {
int qid;
/* Address of Rx and Tx buffers */
#define DRIVER_NAME "fec"
+static const u16 fec_enet_vlan_pri_to_queue[8] = {0, 0, 1, 1, 1, 2, 2, 2};
+
/* Pause frame feild and FIFO threshold */
#define FEC_ENET_FCE (1 << 5)
#define FEC_ENET_RSEM_V 0x84
FEC_QUIRK_HAS_VLAN | FEC_QUIRK_HAS_AVB |
FEC_QUIRK_ERR007885 | FEC_QUIRK_BUG_CAPTURE |
FEC_QUIRK_HAS_RACC | FEC_QUIRK_HAS_COALESCE |
- FEC_QUIRK_CLEAR_SETUP_MII,
+ FEC_QUIRK_CLEAR_SETUP_MII | FEC_QUIRK_HAS_MULTI_QUEUES,
};
static const struct fec_devinfo fec_imx6ul_info = {
estatus |= FEC_TX_BD_FTYPE(txq->bd.qid);
if (skb->ip_summed == CHECKSUM_PARTIAL)
estatus |= BD_ENET_TX_PINS | BD_ENET_TX_IINS;
+
ebdp->cbd_bdu = 0;
ebdp->cbd_esc = cpu_to_fec32(estatus);
}
* For i.MX6SX SOC, enet use AXI bus, we use disable MAC
* instead of reset MAC itself.
*/
- if (fep->quirks & FEC_QUIRK_HAS_AVB ||
+ if (fep->quirks & FEC_QUIRK_HAS_MULTI_QUEUES ||
((fep->quirks & FEC_QUIRK_NO_HARD_RESET) && fep->link)) {
writel(0, fep->hwp + FEC_ECNTRL);
} else {
* instead of reset MAC itself.
*/
if (!(fep->wol_flag & FEC_WOL_FLAG_SLEEP_ON)) {
- if (fep->quirks & FEC_QUIRK_HAS_AVB) {
+ if (fep->quirks & FEC_QUIRK_HAS_MULTI_QUEUES) {
writel(0, fep->hwp + FEC_ECNTRL);
} else {
writel(1, fep->hwp + FEC_ECNTRL);
}
/* ------------------------------------------------------------------------- */
-static void fec_get_mac(struct net_device *ndev)
+static int fec_get_mac(struct net_device *ndev)
{
struct fec_enet_private *fep = netdev_priv(ndev);
unsigned char *iap, tmpaddr[ETH_ALEN];
ret = of_get_mac_address(np, tmpaddr);
if (!ret)
iap = tmpaddr;
+ else if (ret == -EPROBE_DEFER)
+ return ret;
}
}
eth_hw_addr_random(ndev);
dev_info(&fep->pdev->dev, "Using random MAC address: %pM\n",
ndev->dev_addr);
- return;
+ return 0;
}
memcpy(ndev->dev_addr, iap, ETH_ALEN);
/* Adjust MAC if using macaddr */
if (iap == macaddr)
ndev->dev_addr[ETH_ALEN-1] = macaddr[ETH_ALEN-1] + fep->dev_id;
+
+ return 0;
}
/* ------------------------------------------------------------------------- */
writel(tx_itr, fep->hwp + FEC_TXIC0);
writel(rx_itr, fep->hwp + FEC_RXIC0);
- if (fep->quirks & FEC_QUIRK_HAS_AVB) {
+ if (fep->quirks & FEC_QUIRK_HAS_MULTI_QUEUES) {
writel(tx_itr, fep->hwp + FEC_TXIC1);
writel(rx_itr, fep->hwp + FEC_RXIC1);
writel(tx_itr, fep->hwp + FEC_TXIC2);
return 0;
}
+static u16 fec_enet_get_raw_vlan_tci(struct sk_buff *skb)
+{
+ struct vlan_ethhdr *vhdr;
+ unsigned short vlan_TCI = 0;
+
+ if (skb->protocol == htons(ETH_P_ALL)) {
+ vhdr = (struct vlan_ethhdr *)(skb->data);
+ vlan_TCI = ntohs(vhdr->h_vlan_TCI);
+ }
+
+ return vlan_TCI;
+}
+
+static u16 fec_enet_select_queue(struct net_device *ndev, struct sk_buff *skb,
+ struct net_device *sb_dev)
+{
+ struct fec_enet_private *fep = netdev_priv(ndev);
+ u16 vlan_tag;
+
+ if (!(fep->quirks & FEC_QUIRK_HAS_AVB))
+ return netdev_pick_tx(ndev, skb, NULL);
+
+ vlan_tag = fec_enet_get_raw_vlan_tci(skb);
+ if (!vlan_tag)
+ return vlan_tag;
+
+ return fec_enet_vlan_pri_to_queue[vlan_tag >> 13];
+}
+
static const struct net_device_ops fec_netdev_ops = {
.ndo_open = fec_enet_open,
.ndo_stop = fec_enet_close,
.ndo_start_xmit = fec_enet_start_xmit,
+ .ndo_select_queue = fec_enet_select_queue,
.ndo_set_rx_mode = set_multicast_list,
.ndo_validate_addr = eth_validate_addr,
.ndo_tx_timeout = fec_timeout,
return ret;
}
- fec_enet_alloc_queue(ndev);
+ ret = fec_enet_alloc_queue(ndev);
+ if (ret)
+ return ret;
bd_size = (fep->total_tx_ring_size + fep->total_rx_ring_size) * dsize;
cbd_base = dmam_alloc_coherent(&fep->pdev->dev, bd_size, &bd_dma,
GFP_KERNEL);
if (!cbd_base) {
- return -ENOMEM;
+ ret = -ENOMEM;
+ goto free_queue_mem;
}
/* Get the Ethernet address */
- fec_get_mac(ndev);
+ ret = fec_get_mac(ndev);
+ if (ret)
+ goto free_queue_mem;
+
/* make sure MAC we just acquired is programmed into the hw */
fec_set_mac_address(ndev, NULL);
fep->csum_flags |= FLAG_RX_CSUM_ENABLED;
}
- if (fep->quirks & FEC_QUIRK_HAS_AVB) {
+ if (fep->quirks & FEC_QUIRK_HAS_MULTI_QUEUES) {
fep->tx_align = 0;
fep->rx_align = 0x3f;
}
fec_enet_update_ethtool_stats(ndev);
return 0;
+
+free_queue_mem:
+ fec_enet_free_queue(ndev);
+ return ret;
}
#ifdef CONFIG_OF
{
struct fec_enet_private *fep =
container_of(cc, struct fec_enet_private, cc);
- const struct platform_device_id *id_entry =
- platform_get_device_id(fep->pdev);
u32 tempval;
tempval = readl(fep->hwp + FEC_ATIME_CTRL);
tempval |= FEC_T_CTRL_CAPTURE;
writel(tempval, fep->hwp + FEC_ATIME_CTRL);
- if (id_entry->driver_data & FEC_QUIRK_BUG_CAPTURE)
+ if (fep->quirks & FEC_QUIRK_BUG_CAPTURE)
udelay(1);
return readl(fep->hwp + FEC_ATIME);
fep->ptp_caps.enable = fec_ptp_enable;
fep->cycle_speed = clk_get_rate(fep->clk_ptp);
+ if (!fep->cycle_speed) {
+ fep->cycle_speed = NSEC_PER_SEC;
+ dev_err(&fep->pdev->dev, "clk_ptp clock rate is zero\n");
+ }
fep->ptp_inc = NSEC_PER_SEC / fep->cycle_speed;
spin_lock_init(&fep->tmreg_lock);
gfar_configure_coalescing(priv, 0xFF, 0xFF);
}
-static struct net_device_stats *gfar_get_stats(struct net_device *dev)
+static void gfar_get_stats64(struct net_device *dev, struct rtnl_link_stats64 *stats)
{
struct gfar_private *priv = netdev_priv(dev);
- unsigned long rx_packets = 0, rx_bytes = 0, rx_dropped = 0;
- unsigned long tx_packets = 0, tx_bytes = 0;
int i;
for (i = 0; i < priv->num_rx_queues; i++) {
- rx_packets += priv->rx_queue[i]->stats.rx_packets;
- rx_bytes += priv->rx_queue[i]->stats.rx_bytes;
- rx_dropped += priv->rx_queue[i]->stats.rx_dropped;
+ stats->rx_packets += priv->rx_queue[i]->stats.rx_packets;
+ stats->rx_bytes += priv->rx_queue[i]->stats.rx_bytes;
+ stats->rx_dropped += priv->rx_queue[i]->stats.rx_dropped;
}
- dev->stats.rx_packets = rx_packets;
- dev->stats.rx_bytes = rx_bytes;
- dev->stats.rx_dropped = rx_dropped;
-
for (i = 0; i < priv->num_tx_queues; i++) {
- tx_bytes += priv->tx_queue[i]->stats.tx_bytes;
- tx_packets += priv->tx_queue[i]->stats.tx_packets;
+ stats->tx_bytes += priv->tx_queue[i]->stats.tx_bytes;
+ stats->tx_packets += priv->tx_queue[i]->stats.tx_packets;
}
- dev->stats.tx_bytes = tx_bytes;
- dev->stats.tx_packets = tx_packets;
+ if (priv->device_flags & FSL_GIANFAR_DEV_HAS_RMON) {
+ struct rmon_mib __iomem *rmon = &priv->gfargrp[0].regs->rmon;
+ unsigned long flags;
+ u32 rdrp, car, car_before;
+ u64 rdrp_offset;
+
+ spin_lock_irqsave(&priv->rmon_overflow.lock, flags);
+ car = gfar_read(&rmon->car1) & CAR1_C1RDR;
+ do {
+ car_before = car;
+ rdrp = gfar_read(&rmon->rdrp);
+ car = gfar_read(&rmon->car1) & CAR1_C1RDR;
+ } while (car != car_before);
+ if (car) {
+ priv->rmon_overflow.rdrp++;
+ gfar_write(&rmon->car1, car);
+ }
+ rdrp_offset = priv->rmon_overflow.rdrp;
+ spin_unlock_irqrestore(&priv->rmon_overflow.lock, flags);
- return &dev->stats;
+ stats->rx_missed_errors = rdrp + (rdrp_offset << 16);
+ }
}
/* Set the appropriate hash bit for the given addr */
for (i = 0; i < priv->num_grps; i++) {
struct gfar __iomem *regs = priv->gfargrp[i].regs;
/* Unmask the interrupts we look for */
- gfar_write(®s->imask, IMASK_DEFAULT);
+ gfar_write(®s->imask,
+ IMASK_DEFAULT | priv->rmon_overflow.imask);
}
}
if (likely(napi_schedule_prep(&grp->napi_rx))) {
spin_lock_irqsave(&grp->grplock, flags);
imask = gfar_read(&grp->regs->imask);
- imask &= IMASK_RX_DISABLED;
+ imask &= IMASK_RX_DISABLED | grp->priv->rmon_overflow.imask;
gfar_write(&grp->regs->imask, imask);
spin_unlock_irqrestore(&grp->grplock, flags);
__napi_schedule(&grp->napi_rx);
if (likely(napi_schedule_prep(&grp->napi_tx))) {
spin_lock_irqsave(&grp->grplock, flags);
imask = gfar_read(&grp->regs->imask);
- imask &= IMASK_TX_DISABLED;
+ imask &= IMASK_TX_DISABLED | grp->priv->rmon_overflow.imask;
gfar_write(&grp->regs->imask, imask);
spin_unlock_irqrestore(&grp->grplock, flags);
__napi_schedule(&grp->napi_tx);
}
netif_dbg(priv, tx_err, dev, "Transmit Error\n");
}
+ if (events & IEVENT_MSRO) {
+ struct rmon_mib __iomem *rmon = ®s->rmon;
+ u32 car;
+
+ spin_lock(&priv->rmon_overflow.lock);
+ car = gfar_read(&rmon->car1) & CAR1_C1RDR;
+ if (car) {
+ priv->rmon_overflow.rdrp++;
+ gfar_write(&rmon->car1, car);
+ }
+ spin_unlock(&priv->rmon_overflow.lock);
+ }
if (events & IEVENT_BSY) {
dev->stats.rx_over_errors++;
atomic64_inc(&priv->extra_stats.rx_bsy);
/* Zero out the rmon mib registers if it has them */
if (priv->device_flags & FSL_GIANFAR_DEV_HAS_RMON) {
- memset_io(&(regs->rmon), 0, sizeof(struct rmon_mib));
+ memset_io(®s->rmon, 0, offsetof(struct rmon_mib, car1));
/* Mask off the CAM interrupts */
gfar_write(®s->rmon.cam1, 0xffffffff);
gfar_write(®s->rmon.cam2, 0xffffffff);
+ /* Clear the CAR registers (w1c style) */
+ gfar_write(®s->rmon.car1, 0xffffffff);
+ gfar_write(®s->rmon.car2, 0xffffffff);
}
/* Initialize ECNTRL */
.ndo_set_rx_mode = gfar_set_multi,
.ndo_tx_timeout = gfar_timeout,
.ndo_do_ioctl = gfar_ioctl,
- .ndo_get_stats = gfar_get_stats,
+ .ndo_get_stats64 = gfar_get_stats64,
.ndo_change_carrier = fixed_phy_change_carrier,
.ndo_set_mac_address = gfar_set_mac_addr,
.ndo_validate_addr = eth_validate_addr,
gfar_hw_init(priv);
+ if (priv->device_flags & FSL_GIANFAR_DEV_HAS_RMON) {
+ struct rmon_mib __iomem *rmon = &priv->gfargrp[0].regs->rmon;
+
+ spin_lock_init(&priv->rmon_overflow.lock);
+ priv->rmon_overflow.imask = IMASK_MSRO;
+ gfar_write(&rmon->cam1, gfar_read(&rmon->cam1) & ~CAM1_M1RDR);
+ }
+
/* Carrier starts down, phylib will bring it up */
netif_carrier_off(dev);
#define RQFPR_PER 0x00000002
#define RQFPR_EER 0x00000001
+/* CAR1 bits */
+#define CAR1_C164 0x80000000
+#define CAR1_C1127 0x40000000
+#define CAR1_C1255 0x20000000
+#define CAR1_C1511 0x10000000
+#define CAR1_C11K 0x08000000
+#define CAR1_C1MAX 0x04000000
+#define CAR1_C1MGV 0x02000000
+#define CAR1_C1REJ 0x00020000
+#define CAR1_C1RBY 0x00010000
+#define CAR1_C1RPK 0x00008000
+#define CAR1_C1RFC 0x00004000
+#define CAR1_C1RMC 0x00002000
+#define CAR1_C1RBC 0x00001000
+#define CAR1_C1RXC 0x00000800
+#define CAR1_C1RXP 0x00000400
+#define CAR1_C1RXU 0x00000200
+#define CAR1_C1RAL 0x00000100
+#define CAR1_C1RFL 0x00000080
+#define CAR1_C1RCD 0x00000040
+#define CAR1_C1RCS 0x00000020
+#define CAR1_C1RUN 0x00000010
+#define CAR1_C1ROV 0x00000008
+#define CAR1_C1RFR 0x00000004
+#define CAR1_C1RJB 0x00000002
+#define CAR1_C1RDR 0x00000001
+
+/* CAM1 bits */
+#define CAM1_M164 0x80000000
+#define CAM1_M1127 0x40000000
+#define CAM1_M1255 0x20000000
+#define CAM1_M1511 0x10000000
+#define CAM1_M11K 0x08000000
+#define CAM1_M1MAX 0x04000000
+#define CAM1_M1MGV 0x02000000
+#define CAM1_M1REJ 0x00020000
+#define CAM1_M1RBY 0x00010000
+#define CAM1_M1RPK 0x00008000
+#define CAM1_M1RFC 0x00004000
+#define CAM1_M1RMC 0x00002000
+#define CAM1_M1RBC 0x00001000
+#define CAM1_M1RXC 0x00000800
+#define CAM1_M1RXP 0x00000400
+#define CAM1_M1RXU 0x00000200
+#define CAM1_M1RAL 0x00000100
+#define CAM1_M1RFL 0x00000080
+#define CAM1_M1RCD 0x00000040
+#define CAM1_M1RCS 0x00000020
+#define CAM1_M1RUN 0x00000010
+#define CAM1_M1ROV 0x00000008
+#define CAM1_M1RFR 0x00000004
+#define CAM1_M1RJB 0x00000002
+#define CAM1_M1RDR 0x00000001
+
/* TxBD status field bits */
#define TXBD_READY 0x8000
#define TXBD_PADCRC 0x4000
u32 cam2; /* 0x.73c - Carry Mask Register Two */
};
+struct rmon_overflow {
+ /* lock for synchronization of the rdrp field of this struct, and
+ * CAR1/CAR2 registers
+ */
+ spinlock_t lock;
+ u32 imask;
+ u64 rdrp;
+};
+
struct gfar_extra_stats {
atomic64_t rx_alloc_err;
atomic64_t rx_large;
* Per TX queue stats
*/
struct tx_q_stats {
- unsigned long tx_packets;
- unsigned long tx_bytes;
+ u64 tx_packets;
+ u64 tx_bytes;
};
/**
* Per RX queue stats
*/
struct rx_q_stats {
- unsigned long rx_packets;
- unsigned long rx_bytes;
- unsigned long rx_dropped;
+ u64 rx_packets;
+ u64 rx_bytes;
+ u64 rx_dropped;
};
struct gfar_rx_buff {
/* Network Statistics */
struct gfar_extra_stats extra_stats;
+ struct rmon_overflow rmon_overflow;
/* PHY stuff */
phy_interface_t interface;
if ((ucc_num < 0) || (ucc_num > 7))
return -ENODEV;
- ug_info = kmalloc(sizeof(*ug_info), GFP_KERNEL);
+ ug_info = kmemdup(&ugeth_primary_info, sizeof(*ug_info), GFP_KERNEL);
if (ug_info == NULL)
return -ENOMEM;
- memcpy(ug_info, &ugeth_primary_info, sizeof(*ug_info));
ug_info->uf_info.ucc_num = ucc_num;
* QorIQ 10G MDIO Controller
*
* Copyright 2012 Freescale Semiconductor, Inc.
+ * Copyright 2021 NXP
*
* Authors: Andy Fleming <afleming@freescale.com>
* Timur Tabi <timur@freescale.com>
* kind, whether express or implied.
*/
-#include <linux/kernel.h>
-#include <linux/slab.h>
+#include <linux/acpi.h>
+#include <linux/acpi_mdio.h>
#include <linux/interrupt.h>
-#include <linux/module.h>
-#include <linux/phy.h>
+#include <linux/kernel.h>
#include <linux/mdio.h>
+#include <linux/module.h>
#include <linux/of_address.h>
-#include <linux/of_platform.h>
#include <linux/of_mdio.h>
+#include <linux/of_platform.h>
+#include <linux/phy.h>
+#include <linux/slab.h>
/* Number of microseconds to wait for a register to respond */
#define TIMEOUT 1000
static int xgmac_mdio_probe(struct platform_device *pdev)
{
- struct device_node *np = pdev->dev.of_node;
- struct mii_bus *bus;
- struct resource *res;
+ struct fwnode_handle *fwnode;
struct mdio_fsl_priv *priv;
+ struct resource *res;
+ struct mii_bus *bus;
int ret;
/* In DPAA-1, MDIO is one of the many FMan sub-devices. The FMan
goto err_ioremap;
}
+ /* For both ACPI and DT cases, endianness of MDIO controller
+ * needs to be specified using "little-endian" property.
+ */
priv->is_little_endian = device_property_read_bool(&pdev->dev,
"little-endian");
priv->has_a011043 = device_property_read_bool(&pdev->dev,
"fsl,erratum-a011043");
- ret = of_mdiobus_register(bus, np);
+ fwnode = pdev->dev.fwnode;
+ if (is_of_node(fwnode))
+ ret = of_mdiobus_register(bus, to_of_node(fwnode));
+ else if (is_acpi_node(fwnode))
+ ret = acpi_mdiobus_register(bus, fwnode);
+ else
+ ret = -EINVAL;
if (ret) {
dev_err(&pdev->dev, "cannot register MDIO bus\n");
goto err_registration;
base = ioremap(link->resource[2]->start, resource_size(link->resource[2]));
if (!base) {
- pcmcia_release_window(link, link->resource[2]);
- return -ENOMEM;
+ pcmcia_release_window(link, link->resource[2]);
+ return -1;
}
pcmcia_map_mem_page(link, link->resource[2], 0);
if (length < ETH_ZLEN)
{
- if (skb_padto(skb, ETH_ZLEN))
- return NETDEV_TX_OK;
- length = ETH_ZLEN;
+ if (skb_padto(skb, ETH_ZLEN))
+ return NETDEV_TX_OK;
+ length = ETH_ZLEN;
}
netif_stop_queue(dev);
config GVE
tristate "Google Virtual NIC (gVNIC) support"
- depends on PCI_MSI
+ depends on (PCI_MSI && (X86 || CPU_LITTLE_ENDIAN))
help
This driver supports Google Virtual NIC (gVNIC)"
# Makefile for the Google virtual Ethernet (gve) driver
obj-$(CONFIG_GVE) += gve.o
-gve-objs := gve_main.o gve_tx.o gve_rx.o gve_ethtool.o gve_adminq.o
+gve-objs := gve_main.o gve_tx.o gve_tx_dqo.o gve_rx.o gve_rx_dqo.o gve_ethtool.o gve_adminq.o gve_utils.o
/* SPDX-License-Identifier: (GPL-2.0 OR MIT)
* Google virtual Ethernet (gve) driver
*
- * Copyright (C) 2015-2019 Google, Inc.
+ * Copyright (C) 2015-2021 Google, Inc.
*/
#ifndef _GVE_H_
#include <linux/netdevice.h>
#include <linux/pci.h>
#include <linux/u64_stats_sync.h>
+
#include "gve_desc.h"
+#include "gve_desc_dqo.h"
#ifndef PCI_VENDOR_ID_GOOGLE
#define PCI_VENDOR_ID_GOOGLE 0x1ae0
#define GVE_DATA_SLOT_ADDR_PAGE_MASK (~(PAGE_SIZE - 1))
+/* PTYPEs are always 10 bits. */
+#define GVE_NUM_PTYPES 1024
+
+#define GVE_RX_BUFFER_SIZE_DQO 2048
+
/* Each slot in the desc ring has a 1:1 mapping to a slot in the data ring */
struct gve_rx_desc_queue {
struct gve_rx_desc *desc_ring; /* the descriptor ring */
struct gve_rx_slot_page_info {
struct page *page;
void *page_address;
- u8 page_offset; /* flipped to second half? */
+ u32 page_offset; /* offset to write to in page */
+ int pagecnt_bias; /* expected pagecnt if only the driver has a ref */
u8 can_flip;
};
struct gve_priv;
-/* An RX ring that contains a power-of-two sized desc and data ring. */
+/* RX buffer queue for posting buffers to HW.
+ * Each RX (completion) queue has a corresponding buffer queue.
+ */
+struct gve_rx_buf_queue_dqo {
+ struct gve_rx_desc_dqo *desc_ring;
+ dma_addr_t bus;
+ u32 head; /* Pointer to start cleaning buffers at. */
+ u32 tail; /* Last posted buffer index + 1 */
+ u32 mask; /* Mask for indices to the size of the ring */
+};
+
+/* RX completion queue to receive packets from HW. */
+struct gve_rx_compl_queue_dqo {
+ struct gve_rx_compl_desc_dqo *desc_ring;
+ dma_addr_t bus;
+
+ /* Number of slots which did not have a buffer posted yet. We should not
+ * post more buffers than the queue size to avoid HW overrunning the
+ * queue.
+ */
+ int num_free_slots;
+
+ /* HW uses a "generation bit" to notify SW of new descriptors. When a
+ * descriptor's generation bit is different from the current generation,
+ * that descriptor is ready to be consumed by SW.
+ */
+ u8 cur_gen_bit;
+
+ /* Pointer into desc_ring where the next completion descriptor will be
+ * received.
+ */
+ u32 head;
+ u32 mask; /* Mask for indices to the size of the ring */
+};
+
+/* Stores state for tracking buffers posted to HW */
+struct gve_rx_buf_state_dqo {
+ /* The page posted to HW. */
+ struct gve_rx_slot_page_info page_info;
+
+ /* The DMA address corresponding to `page_info`. */
+ dma_addr_t addr;
+
+ /* Last offset into the page when it only had a single reference, at
+ * which point every other offset is free to be reused.
+ */
+ u32 last_single_ref_offset;
+
+ /* Linked list index to next element in the list, or -1 if none */
+ s16 next;
+};
+
+/* `head` and `tail` are indices into an array, or -1 if empty. */
+struct gve_index_list {
+ s16 head;
+ s16 tail;
+};
+
+/* Contains datapath state used to represent an RX queue. */
struct gve_rx_ring {
struct gve_priv *gve;
- struct gve_rx_desc_queue desc;
- struct gve_rx_data_queue data;
+ union {
+ /* GQI fields */
+ struct {
+ struct gve_rx_desc_queue desc;
+ struct gve_rx_data_queue data;
+
+ /* threshold for posting new buffs and descs */
+ u32 db_threshold;
+ };
+
+ /* DQO fields. */
+ struct {
+ struct gve_rx_buf_queue_dqo bufq;
+ struct gve_rx_compl_queue_dqo complq;
+
+ struct gve_rx_buf_state_dqo *buf_states;
+ u16 num_buf_states;
+
+ /* Linked list of gve_rx_buf_state_dqo. Index into
+ * buf_states, or -1 if empty.
+ */
+ s16 free_buf_states;
+
+ /* Linked list of gve_rx_buf_state_dqo. Indexes into
+ * buf_states, or -1 if empty.
+ *
+ * This list contains buf_states which are pointing to
+ * valid buffers.
+ *
+ * We use a FIFO here in order to increase the
+ * probability that buffers can be reused by increasing
+ * the time between usages.
+ */
+ struct gve_index_list recycled_buf_states;
+
+ /* Linked list of gve_rx_buf_state_dqo. Indexes into
+ * buf_states, or -1 if empty.
+ *
+ * This list contains buf_states which have buffers
+ * which cannot be reused yet.
+ */
+ struct gve_index_list used_buf_states;
+ } dqo;
+ };
+
u64 rbytes; /* free-running bytes received */
u64 rpackets; /* free-running packets received */
u32 cnt; /* free-running total number of completed packets */
u32 fill_cnt; /* free-running total number of descs and buffs posted */
u32 mask; /* masks the cnt and fill_cnt to the size of the ring */
- u32 db_threshold; /* threshold for posting new buffs and descs */
u64 rx_copybreak_pkt; /* free-running count of copybreak packets */
u64 rx_copied_pkt; /* free-running total number of copied packets */
u64 rx_skb_alloc_fail; /* free-running count of skb alloc fails */
struct gve_queue_resources *q_resources; /* head and tail pointer idx */
dma_addr_t q_resources_bus; /* dma address for the queue resources */
struct u64_stats_sync statss; /* sync stats for 32bit archs */
+
+ /* head and tail of skb chain for the current packet or NULL if none */
+ struct sk_buff *skb_head;
+ struct sk_buff *skb_tail;
};
/* A TX desc ring entry */
struct gve_queue_page_list *qpl; /* QPL mapped into this FIFO */
};
-/* A TX ring that contains a power-of-two sized desc ring and a FIFO buffer */
+/* TX descriptor for DQO format */
+union gve_tx_desc_dqo {
+ struct gve_tx_pkt_desc_dqo pkt;
+ struct gve_tx_tso_context_desc_dqo tso_ctx;
+ struct gve_tx_general_context_desc_dqo general_ctx;
+};
+
+enum gve_packet_state {
+ /* Packet is in free list, available to be allocated.
+ * This should always be zero since state is not explicitly initialized.
+ */
+ GVE_PACKET_STATE_UNALLOCATED,
+ /* Packet is expecting a regular data completion or miss completion */
+ GVE_PACKET_STATE_PENDING_DATA_COMPL,
+ /* Packet has received a miss completion and is expecting a
+ * re-injection completion.
+ */
+ GVE_PACKET_STATE_PENDING_REINJECT_COMPL,
+ /* No valid completion received within the specified timeout. */
+ GVE_PACKET_STATE_TIMED_OUT_COMPL,
+};
+
+struct gve_tx_pending_packet_dqo {
+ struct sk_buff *skb; /* skb for this packet */
+
+ /* 0th element corresponds to the linear portion of `skb`, should be
+ * unmapped with `dma_unmap_single`.
+ *
+ * All others correspond to `skb`'s frags and should be unmapped with
+ * `dma_unmap_page`.
+ */
+ struct gve_tx_dma_buf bufs[MAX_SKB_FRAGS + 1];
+ u16 num_bufs;
+
+ /* Linked list index to next element in the list, or -1 if none */
+ s16 next;
+
+ /* Linked list index to prev element in the list, or -1 if none.
+ * Used for tracking either outstanding miss completions or prematurely
+ * freed packets.
+ */
+ s16 prev;
+
+ /* Identifies the current state of the packet as defined in
+ * `enum gve_packet_state`.
+ */
+ u8 state;
+
+ /* If packet is an outstanding miss completion, then the packet is
+ * freed if the corresponding re-injection completion is not received
+ * before kernel jiffies exceeds timeout_jiffies.
+ */
+ unsigned long timeout_jiffies;
+};
+
+/* Contains datapath state used to represent a TX queue. */
struct gve_tx_ring {
/* Cacheline 0 -- Accessed & dirtied during transmit */
- struct gve_tx_fifo tx_fifo;
- u32 req; /* driver tracked head pointer */
- u32 done; /* driver tracked tail pointer */
+ union {
+ /* GQI fields */
+ struct {
+ struct gve_tx_fifo tx_fifo;
+ u32 req; /* driver tracked head pointer */
+ u32 done; /* driver tracked tail pointer */
+ };
+
+ /* DQO fields. */
+ struct {
+ /* Linked list of gve_tx_pending_packet_dqo. Index into
+ * pending_packets, or -1 if empty.
+ *
+ * This is a consumer list owned by the TX path. When it
+ * runs out, the producer list is stolen from the
+ * completion handling path
+ * (dqo_compl.free_pending_packets).
+ */
+ s16 free_pending_packets;
+
+ /* Cached value of `dqo_compl.hw_tx_head` */
+ u32 head;
+ u32 tail; /* Last posted buffer index + 1 */
+
+ /* Index of the last descriptor with "report event" bit
+ * set.
+ */
+ u32 last_re_idx;
+ } dqo_tx;
+ };
/* Cacheline 1 -- Accessed & dirtied during gve_clean_tx_done */
- __be32 last_nic_done ____cacheline_aligned; /* NIC tail pointer */
+ union {
+ /* GQI fields */
+ struct {
+ /* NIC tail pointer */
+ __be32 last_nic_done;
+ };
+
+ /* DQO fields. */
+ struct {
+ u32 head; /* Last read on compl_desc */
+
+ /* Tracks the current gen bit of compl_q */
+ u8 cur_gen_bit;
+
+ /* Linked list of gve_tx_pending_packet_dqo. Index into
+ * pending_packets, or -1 if empty.
+ *
+ * This is the producer list, owned by the completion
+ * handling path. When the consumer list
+ * (dqo_tx.free_pending_packets) is runs out, this list
+ * will be stolen.
+ */
+ atomic_t free_pending_packets;
+
+ /* Last TX ring index fetched by HW */
+ atomic_t hw_tx_head;
+
+ /* List to track pending packets which received a miss
+ * completion but not a corresponding reinjection.
+ */
+ struct gve_index_list miss_completions;
+
+ /* List to track pending packets that were completed
+ * before receiving a valid completion because they
+ * reached a specified timeout.
+ */
+ struct gve_index_list timed_out_completions;
+ } dqo_compl;
+ } ____cacheline_aligned;
u64 pkt_done; /* free-running - total packets completed */
u64 bytes_done; /* free-running - total bytes completed */
u64 dropped_pkt; /* free-running - total packets dropped */
u64 dma_mapping_error; /* count of dma mapping errors */
/* Cacheline 2 -- Read-mostly fields */
- union gve_tx_desc *desc ____cacheline_aligned;
- struct gve_tx_buffer_state *info; /* Maps 1:1 to a desc */
+ union {
+ /* GQI fields */
+ struct {
+ union gve_tx_desc *desc;
+
+ /* Maps 1:1 to a desc */
+ struct gve_tx_buffer_state *info;
+ };
+
+ /* DQO fields. */
+ struct {
+ union gve_tx_desc_dqo *tx_ring;
+ struct gve_tx_compl_desc *compl_ring;
+
+ struct gve_tx_pending_packet_dqo *pending_packets;
+ s16 num_pending_packets;
+
+ u32 complq_mask; /* complq size is complq_mask + 1 */
+ } dqo;
+ } ____cacheline_aligned;
struct netdev_queue *netdev_txq;
struct gve_queue_resources *q_resources; /* head and tail pointer idx */
struct device *dev;
u32 ntfy_id; /* notification block index */
dma_addr_t bus; /* dma address of the descr ring */
dma_addr_t q_resources_bus; /* dma address of the queue resources */
+ dma_addr_t complq_bus_dqo; /* dma address of the dqo.compl_ring */
struct u64_stats_sync statss; /* sync stats for 32bit archs */
} ____cacheline_aligned;
unsigned long *qpl_id_map; /* bitmap of used qpl ids */
};
+struct gve_options_dqo_rda {
+ u16 tx_comp_ring_entries; /* number of tx_comp descriptors */
+ u16 rx_buff_ring_entries; /* number of rx_buff descriptors */
+};
+
+struct gve_ptype {
+ u8 l3_type; /* `gve_l3_type` in gve_adminq.h */
+ u8 l4_type; /* `gve_l4_type` in gve_adminq.h */
+};
+
+struct gve_ptype_lut {
+ struct gve_ptype ptypes[GVE_NUM_PTYPES];
+};
+
+/* GVE_QUEUE_FORMAT_UNSPECIFIED must be zero since 0 is the default value
+ * when the entire configure_device_resources command is zeroed out and the
+ * queue_format is not specified.
+ */
+enum gve_queue_format {
+ GVE_QUEUE_FORMAT_UNSPECIFIED = 0x0,
+ GVE_GQI_RDA_FORMAT = 0x1,
+ GVE_GQI_QPL_FORMAT = 0x2,
+ GVE_DQO_RDA_FORMAT = 0x3,
+};
+
struct gve_priv {
struct net_device *dev;
struct gve_tx_ring *tx; /* array of tx_cfg.num_queues */
u64 num_registered_pages; /* num pages registered with NIC */
u32 rx_copybreak; /* copy packets smaller than this */
u16 default_num_queues; /* default num queues to set up */
- u8 raw_addressing; /* 1 if this dev supports raw addressing, 0 otherwise */
struct gve_queue_config tx_cfg;
struct gve_queue_config rx_cfg;
u32 adminq_set_driver_parameter_cnt;
u32 adminq_report_stats_cnt;
u32 adminq_report_link_speed_cnt;
+ u32 adminq_get_ptype_map_cnt;
/* Global stats */
u32 interface_up_cnt; /* count of times interface turned up since last reset */
/* Gvnic device link speed from hypervisor. */
u64 link_speed;
+
+ struct gve_options_dqo_rda options_dqo_rda;
+ struct gve_ptype_lut *ptype_lut_dqo;
+
+ /* Must be a power of two. */
+ int data_buffer_size_dqo;
+
+ enum gve_queue_format queue_format;
};
enum gve_service_task_flags_bit {
*/
static inline u32 gve_num_tx_qpls(struct gve_priv *priv)
{
- return priv->raw_addressing ? 0 : priv->tx_cfg.num_queues;
+ if (priv->queue_format != GVE_GQI_QPL_FORMAT)
+ return 0;
+
+ return priv->tx_cfg.num_queues;
}
/* Returns the number of rx queue page lists
*/
static inline u32 gve_num_rx_qpls(struct gve_priv *priv)
{
- return priv->raw_addressing ? 0 : priv->rx_cfg.num_queues;
+ if (priv->queue_format != GVE_GQI_QPL_FORMAT)
+ return 0;
+
+ return priv->rx_cfg.num_queues;
}
/* Returns a pointer to the next available tx qpl in the list of qpls
return DMA_FROM_DEVICE;
}
+static inline bool gve_is_gqi(struct gve_priv *priv)
+{
+ return priv->queue_format == GVE_GQI_RDA_FORMAT ||
+ priv->queue_format == GVE_GQI_QPL_FORMAT;
+}
+
/* buffers */
int gve_alloc_page(struct gve_priv *priv, struct device *dev,
struct page **page, dma_addr_t *dma,
netdev_tx_t gve_tx(struct sk_buff *skb, struct net_device *dev);
bool gve_tx_poll(struct gve_notify_block *block, int budget);
int gve_tx_alloc_rings(struct gve_priv *priv);
-void gve_tx_free_rings(struct gve_priv *priv);
+void gve_tx_free_rings_gqi(struct gve_priv *priv);
__be32 gve_tx_load_event_counter(struct gve_priv *priv,
struct gve_tx_ring *tx);
/* rx handling */
void gve_rx_write_doorbell(struct gve_priv *priv, struct gve_rx_ring *rx);
bool gve_rx_poll(struct gve_notify_block *block, int budget);
int gve_rx_alloc_rings(struct gve_priv *priv);
-void gve_rx_free_rings(struct gve_priv *priv);
+void gve_rx_free_rings_gqi(struct gve_priv *priv);
bool gve_clean_rx_done(struct gve_rx_ring *rx, int budget,
netdev_features_t feat);
/* Reset */
// SPDX-License-Identifier: (GPL-2.0 OR MIT)
/* Google virtual Ethernet (gve) driver
*
- * Copyright (C) 2015-2019 Google, Inc.
+ * Copyright (C) 2015-2021 Google, Inc.
*/
#include <linux/etherdevice.h>
"Expected: length=%d, feature_mask=%x.\n" \
"Actual: length=%d, feature_mask=%x.\n"
+#define GVE_DEVICE_OPTION_TOO_BIG_FMT "Length of %s option larger than expected. Possible older version of guest driver.\n"
+
static
struct gve_device_option *gve_get_next_option(struct gve_device_descriptor *descriptor,
struct gve_device_option *option)
static
void gve_parse_device_option(struct gve_priv *priv,
struct gve_device_descriptor *device_descriptor,
- struct gve_device_option *option)
+ struct gve_device_option *option,
+ struct gve_device_option_gqi_rda **dev_op_gqi_rda,
+ struct gve_device_option_gqi_qpl **dev_op_gqi_qpl,
+ struct gve_device_option_dqo_rda **dev_op_dqo_rda)
{
+ u32 req_feat_mask = be32_to_cpu(option->required_features_mask);
u16 option_length = be16_to_cpu(option->option_length);
u16 option_id = be16_to_cpu(option->option_id);
+ /* If the length or feature mask doesn't match, continue without
+ * enabling the feature.
+ */
switch (option_id) {
- case GVE_DEV_OPT_ID_RAW_ADDRESSING:
- /* If the length or feature mask doesn't match,
- * continue without enabling the feature.
- */
- if (option_length != GVE_DEV_OPT_LEN_RAW_ADDRESSING ||
- option->feat_mask != cpu_to_be32(GVE_DEV_OPT_FEAT_MASK_RAW_ADDRESSING)) {
- dev_warn(&priv->pdev->dev, GVE_DEVICE_OPTION_ERROR_FMT, "Raw Addressing",
- GVE_DEV_OPT_LEN_RAW_ADDRESSING,
- cpu_to_be32(GVE_DEV_OPT_FEAT_MASK_RAW_ADDRESSING),
- option_length, option->feat_mask);
- priv->raw_addressing = 0;
- } else {
- dev_info(&priv->pdev->dev,
- "Raw addressing device option enabled.\n");
- priv->raw_addressing = 1;
+ case GVE_DEV_OPT_ID_GQI_RAW_ADDRESSING:
+ if (option_length != GVE_DEV_OPT_LEN_GQI_RAW_ADDRESSING ||
+ req_feat_mask != GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RAW_ADDRESSING) {
+ dev_warn(&priv->pdev->dev, GVE_DEVICE_OPTION_ERROR_FMT,
+ "Raw Addressing",
+ GVE_DEV_OPT_LEN_GQI_RAW_ADDRESSING,
+ GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RAW_ADDRESSING,
+ option_length, req_feat_mask);
+ break;
+ }
+
+ dev_info(&priv->pdev->dev,
+ "Gqi raw addressing device option enabled.\n");
+ priv->queue_format = GVE_GQI_RDA_FORMAT;
+ break;
+ case GVE_DEV_OPT_ID_GQI_RDA:
+ if (option_length < sizeof(**dev_op_gqi_rda) ||
+ req_feat_mask != GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RDA) {
+ dev_warn(&priv->pdev->dev, GVE_DEVICE_OPTION_ERROR_FMT,
+ "GQI RDA", (int)sizeof(**dev_op_gqi_rda),
+ GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RDA,
+ option_length, req_feat_mask);
+ break;
+ }
+
+ if (option_length > sizeof(**dev_op_gqi_rda)) {
+ dev_warn(&priv->pdev->dev,
+ GVE_DEVICE_OPTION_TOO_BIG_FMT, "GQI RDA");
+ }
+ *dev_op_gqi_rda = (void *)(option + 1);
+ break;
+ case GVE_DEV_OPT_ID_GQI_QPL:
+ if (option_length < sizeof(**dev_op_gqi_qpl) ||
+ req_feat_mask != GVE_DEV_OPT_REQ_FEAT_MASK_GQI_QPL) {
+ dev_warn(&priv->pdev->dev, GVE_DEVICE_OPTION_ERROR_FMT,
+ "GQI QPL", (int)sizeof(**dev_op_gqi_qpl),
+ GVE_DEV_OPT_REQ_FEAT_MASK_GQI_QPL,
+ option_length, req_feat_mask);
+ break;
+ }
+
+ if (option_length > sizeof(**dev_op_gqi_qpl)) {
+ dev_warn(&priv->pdev->dev,
+ GVE_DEVICE_OPTION_TOO_BIG_FMT, "GQI QPL");
+ }
+ *dev_op_gqi_qpl = (void *)(option + 1);
+ break;
+ case GVE_DEV_OPT_ID_DQO_RDA:
+ if (option_length < sizeof(**dev_op_dqo_rda) ||
+ req_feat_mask != GVE_DEV_OPT_REQ_FEAT_MASK_DQO_RDA) {
+ dev_warn(&priv->pdev->dev, GVE_DEVICE_OPTION_ERROR_FMT,
+ "DQO RDA", (int)sizeof(**dev_op_dqo_rda),
+ GVE_DEV_OPT_REQ_FEAT_MASK_DQO_RDA,
+ option_length, req_feat_mask);
+ break;
+ }
+
+ if (option_length > sizeof(**dev_op_dqo_rda)) {
+ dev_warn(&priv->pdev->dev,
+ GVE_DEVICE_OPTION_TOO_BIG_FMT, "DQO RDA");
}
+ *dev_op_dqo_rda = (void *)(option + 1);
break;
default:
/* If we don't recognize the option just continue
}
}
+/* Process all device options for a given describe device call. */
+static int
+gve_process_device_options(struct gve_priv *priv,
+ struct gve_device_descriptor *descriptor,
+ struct gve_device_option_gqi_rda **dev_op_gqi_rda,
+ struct gve_device_option_gqi_qpl **dev_op_gqi_qpl,
+ struct gve_device_option_dqo_rda **dev_op_dqo_rda)
+{
+ const int num_options = be16_to_cpu(descriptor->num_device_options);
+ struct gve_device_option *dev_opt;
+ int i;
+
+ /* The options struct directly follows the device descriptor. */
+ dev_opt = (void *)(descriptor + 1);
+ for (i = 0; i < num_options; i++) {
+ struct gve_device_option *next_opt;
+
+ next_opt = gve_get_next_option(descriptor, dev_opt);
+ if (!next_opt) {
+ dev_err(&priv->dev->dev,
+ "options exceed device_descriptor's total length.\n");
+ return -EINVAL;
+ }
+
+ gve_parse_device_option(priv, descriptor, dev_opt,
+ dev_op_gqi_rda, dev_op_gqi_qpl,
+ dev_op_dqo_rda);
+ dev_opt = next_opt;
+ }
+
+ return 0;
+}
+
int gve_adminq_alloc(struct device *dev, struct gve_priv *priv)
{
priv->adminq = dma_alloc_coherent(dev, PAGE_SIZE,
priv->adminq_set_driver_parameter_cnt = 0;
priv->adminq_report_stats_cnt = 0;
priv->adminq_report_link_speed_cnt = 0;
+ priv->adminq_get_ptype_map_cnt = 0;
/* Setup Admin queue with the device */
iowrite32be(priv->adminq_bus_addr / PAGE_SIZE,
case GVE_ADMINQ_REPORT_LINK_SPEED:
priv->adminq_report_link_speed_cnt++;
break;
+ case GVE_ADMINQ_GET_PTYPE_MAP:
+ priv->adminq_get_ptype_map_cnt++;
+ break;
default:
dev_err(&priv->pdev->dev, "unknown AQ command opcode %d\n", opcode);
}
* The caller is also responsible for making sure there are no commands
* waiting to be executed.
*/
-static int gve_adminq_execute_cmd(struct gve_priv *priv, union gve_adminq_command *cmd_orig)
+static int gve_adminq_execute_cmd(struct gve_priv *priv,
+ union gve_adminq_command *cmd_orig)
{
u32 tail, head;
int err;
.irq_db_stride = cpu_to_be32(sizeof(priv->ntfy_blocks[0])),
.ntfy_blk_msix_base_idx =
cpu_to_be32(GVE_NTFY_BLK_BASE_MSIX_IDX),
+ .queue_format = priv->queue_format,
};
return gve_adminq_execute_cmd(priv, &cmd);
{
struct gve_tx_ring *tx = &priv->tx[queue_index];
union gve_adminq_command cmd;
- u32 qpl_id;
- int err;
- qpl_id = priv->raw_addressing ? GVE_RAW_ADDRESSING_QPL_ID : tx->tx_fifo.qpl->id;
memset(&cmd, 0, sizeof(cmd));
cmd.opcode = cpu_to_be32(GVE_ADMINQ_CREATE_TX_QUEUE);
cmd.create_tx_queue = (struct gve_adminq_create_tx_queue) {
.queue_id = cpu_to_be32(queue_index),
- .reserved = 0,
.queue_resources_addr =
cpu_to_be64(tx->q_resources_bus),
.tx_ring_addr = cpu_to_be64(tx->bus),
- .queue_page_list_id = cpu_to_be32(qpl_id),
.ntfy_id = cpu_to_be32(tx->ntfy_id),
};
- err = gve_adminq_issue_cmd(priv, &cmd);
- if (err)
- return err;
+ if (gve_is_gqi(priv)) {
+ u32 qpl_id = priv->queue_format == GVE_GQI_RDA_FORMAT ?
+ GVE_RAW_ADDRESSING_QPL_ID : tx->tx_fifo.qpl->id;
+
+ cmd.create_tx_queue.queue_page_list_id = cpu_to_be32(qpl_id);
+ } else {
+ cmd.create_tx_queue.tx_ring_size =
+ cpu_to_be16(priv->tx_desc_cnt);
+ cmd.create_tx_queue.tx_comp_ring_addr =
+ cpu_to_be64(tx->complq_bus_dqo);
+ cmd.create_tx_queue.tx_comp_ring_size =
+ cpu_to_be16(priv->options_dqo_rda.tx_comp_ring_entries);
+ }
- return 0;
+ return gve_adminq_issue_cmd(priv, &cmd);
}
int gve_adminq_create_tx_queues(struct gve_priv *priv, u32 num_queues)
{
struct gve_rx_ring *rx = &priv->rx[queue_index];
union gve_adminq_command cmd;
- u32 qpl_id;
- int err;
- qpl_id = priv->raw_addressing ? GVE_RAW_ADDRESSING_QPL_ID : rx->data.qpl->id;
memset(&cmd, 0, sizeof(cmd));
cmd.opcode = cpu_to_be32(GVE_ADMINQ_CREATE_RX_QUEUE);
cmd.create_rx_queue = (struct gve_adminq_create_rx_queue) {
.queue_id = cpu_to_be32(queue_index),
- .index = cpu_to_be32(queue_index),
- .reserved = 0,
.ntfy_id = cpu_to_be32(rx->ntfy_id),
.queue_resources_addr = cpu_to_be64(rx->q_resources_bus),
- .rx_desc_ring_addr = cpu_to_be64(rx->desc.bus),
- .rx_data_ring_addr = cpu_to_be64(rx->data.data_bus),
- .queue_page_list_id = cpu_to_be32(qpl_id),
};
- err = gve_adminq_issue_cmd(priv, &cmd);
- if (err)
- return err;
+ if (gve_is_gqi(priv)) {
+ u32 qpl_id = priv->queue_format == GVE_GQI_RDA_FORMAT ?
+ GVE_RAW_ADDRESSING_QPL_ID : rx->data.qpl->id;
+
+ cmd.create_rx_queue.rx_desc_ring_addr =
+ cpu_to_be64(rx->desc.bus),
+ cmd.create_rx_queue.rx_data_ring_addr =
+ cpu_to_be64(rx->data.data_bus),
+ cmd.create_rx_queue.index = cpu_to_be32(queue_index);
+ cmd.create_rx_queue.queue_page_list_id = cpu_to_be32(qpl_id);
+ } else {
+ cmd.create_rx_queue.rx_ring_size =
+ cpu_to_be16(priv->rx_desc_cnt);
+ cmd.create_rx_queue.rx_desc_ring_addr =
+ cpu_to_be64(rx->dqo.complq.bus);
+ cmd.create_rx_queue.rx_data_ring_addr =
+ cpu_to_be64(rx->dqo.bufq.bus);
+ cmd.create_rx_queue.packet_buffer_size =
+ cpu_to_be16(priv->data_buffer_size_dqo);
+ cmd.create_rx_queue.rx_buff_ring_size =
+ cpu_to_be16(priv->options_dqo_rda.rx_buff_ring_entries);
+ cmd.create_rx_queue.enable_rsc =
+ !!(priv->dev->features & NETIF_F_LRO);
+ }
- return 0;
+ return gve_adminq_issue_cmd(priv, &cmd);
}
int gve_adminq_create_rx_queues(struct gve_priv *priv, u32 num_queues)
return gve_adminq_kick_and_wait(priv);
}
+static int gve_set_desc_cnt(struct gve_priv *priv,
+ struct gve_device_descriptor *descriptor)
+{
+ priv->tx_desc_cnt = be16_to_cpu(descriptor->tx_queue_entries);
+ if (priv->tx_desc_cnt * sizeof(priv->tx->desc[0]) < PAGE_SIZE) {
+ dev_err(&priv->pdev->dev, "Tx desc count %d too low\n",
+ priv->tx_desc_cnt);
+ return -EINVAL;
+ }
+ priv->rx_desc_cnt = be16_to_cpu(descriptor->rx_queue_entries);
+ if (priv->rx_desc_cnt * sizeof(priv->rx->desc.desc_ring[0])
+ < PAGE_SIZE) {
+ dev_err(&priv->pdev->dev, "Rx desc count %d too low\n",
+ priv->rx_desc_cnt);
+ return -EINVAL;
+ }
+ return 0;
+}
+
+static int
+gve_set_desc_cnt_dqo(struct gve_priv *priv,
+ const struct gve_device_descriptor *descriptor,
+ const struct gve_device_option_dqo_rda *dev_op_dqo_rda)
+{
+ priv->tx_desc_cnt = be16_to_cpu(descriptor->tx_queue_entries);
+ priv->options_dqo_rda.tx_comp_ring_entries =
+ be16_to_cpu(dev_op_dqo_rda->tx_comp_ring_entries);
+ priv->rx_desc_cnt = be16_to_cpu(descriptor->rx_queue_entries);
+ priv->options_dqo_rda.rx_buff_ring_entries =
+ be16_to_cpu(dev_op_dqo_rda->rx_buff_ring_entries);
+
+ return 0;
+}
+
int gve_adminq_describe_device(struct gve_priv *priv)
{
+ struct gve_device_option_gqi_rda *dev_op_gqi_rda = NULL;
+ struct gve_device_option_gqi_qpl *dev_op_gqi_qpl = NULL;
+ struct gve_device_option_dqo_rda *dev_op_dqo_rda = NULL;
struct gve_device_descriptor *descriptor;
- struct gve_device_option *dev_opt;
union gve_adminq_command cmd;
dma_addr_t descriptor_bus;
- u16 num_options;
int err = 0;
u8 *mac;
u16 mtu;
- int i;
memset(&cmd, 0, sizeof(cmd));
descriptor = dma_alloc_coherent(&priv->pdev->dev, PAGE_SIZE,
if (err)
goto free_device_descriptor;
- priv->tx_desc_cnt = be16_to_cpu(descriptor->tx_queue_entries);
- if (priv->tx_desc_cnt * sizeof(priv->tx->desc[0]) < PAGE_SIZE) {
- dev_err(&priv->pdev->dev, "Tx desc count %d too low\n", priv->tx_desc_cnt);
- err = -EINVAL;
+ err = gve_process_device_options(priv, descriptor, &dev_op_gqi_rda,
+ &dev_op_gqi_qpl, &dev_op_dqo_rda);
+ if (err)
goto free_device_descriptor;
+
+ /* If the GQI_RAW_ADDRESSING option is not enabled and the queue format
+ * is not set to GqiRda, choose the queue format in a priority order:
+ * DqoRda, GqiRda, GqiQpl. Use GqiQpl as default.
+ */
+ if (priv->queue_format == GVE_GQI_RDA_FORMAT) {
+ dev_info(&priv->pdev->dev,
+ "Driver is running with GQI RDA queue format.\n");
+ } else if (dev_op_dqo_rda) {
+ priv->queue_format = GVE_DQO_RDA_FORMAT;
+ dev_info(&priv->pdev->dev,
+ "Driver is running with DQO RDA queue format.\n");
+ } else if (dev_op_gqi_rda) {
+ priv->queue_format = GVE_GQI_RDA_FORMAT;
+ dev_info(&priv->pdev->dev,
+ "Driver is running with GQI RDA queue format.\n");
+ } else {
+ priv->queue_format = GVE_GQI_QPL_FORMAT;
+ dev_info(&priv->pdev->dev,
+ "Driver is running with GQI QPL queue format.\n");
}
- priv->rx_desc_cnt = be16_to_cpu(descriptor->rx_queue_entries);
- if (priv->rx_desc_cnt * sizeof(priv->rx->desc.desc_ring[0])
- < PAGE_SIZE ||
- priv->rx_desc_cnt * sizeof(priv->rx->data.data_ring[0])
- < PAGE_SIZE) {
- dev_err(&priv->pdev->dev, "Rx desc count %d too low\n", priv->rx_desc_cnt);
- err = -EINVAL;
- goto free_device_descriptor;
+ if (gve_is_gqi(priv)) {
+ err = gve_set_desc_cnt(priv, descriptor);
+ } else {
+ /* DQO supports LRO. */
+ priv->dev->hw_features |= NETIF_F_LRO;
+ err = gve_set_desc_cnt_dqo(priv, descriptor, dev_op_dqo_rda);
}
+ if (err)
+ goto free_device_descriptor;
+
priv->max_registered_pages =
be64_to_cpu(descriptor->max_registered_pages);
mtu = be16_to_cpu(descriptor->mtu);
dev_info(&priv->pdev->dev, "MAC addr: %pM\n", mac);
priv->tx_pages_per_qpl = be16_to_cpu(descriptor->tx_pages_per_qpl);
priv->rx_data_slot_cnt = be16_to_cpu(descriptor->rx_pages_per_qpl);
- if (priv->rx_data_slot_cnt < priv->rx_desc_cnt) {
+
+ if (gve_is_gqi(priv) && priv->rx_data_slot_cnt < priv->rx_desc_cnt) {
dev_err(&priv->pdev->dev, "rx_data_slot_cnt cannot be smaller than rx_desc_cnt, setting rx_desc_cnt down to %d.\n",
priv->rx_data_slot_cnt);
priv->rx_desc_cnt = priv->rx_data_slot_cnt;
}
priv->default_num_queues = be16_to_cpu(descriptor->default_num_queues);
- dev_opt = (void *)(descriptor + 1);
-
- num_options = be16_to_cpu(descriptor->num_device_options);
- for (i = 0; i < num_options; i++) {
- struct gve_device_option *next_opt;
-
- next_opt = gve_get_next_option(descriptor, dev_opt);
- if (!next_opt) {
- dev_err(&priv->dev->dev,
- "options exceed device_descriptor's total length.\n");
- err = -EINVAL;
- goto free_device_descriptor;
- }
-
- gve_parse_device_option(priv, descriptor, dev_opt);
- dev_opt = next_opt;
- }
free_device_descriptor:
- dma_free_coherent(&priv->pdev->dev, sizeof(*descriptor), descriptor,
+ dma_free_coherent(&priv->pdev->dev, PAGE_SIZE, descriptor,
descriptor_bus);
return err;
}
link_speed_region_bus);
return err;
}
+
+int gve_adminq_get_ptype_map_dqo(struct gve_priv *priv,
+ struct gve_ptype_lut *ptype_lut)
+{
+ struct gve_ptype_map *ptype_map;
+ union gve_adminq_command cmd;
+ dma_addr_t ptype_map_bus;
+ int err = 0;
+ int i;
+
+ memset(&cmd, 0, sizeof(cmd));
+ ptype_map = dma_alloc_coherent(&priv->pdev->dev, sizeof(*ptype_map),
+ &ptype_map_bus, GFP_KERNEL);
+ if (!ptype_map)
+ return -ENOMEM;
+
+ cmd.opcode = cpu_to_be32(GVE_ADMINQ_GET_PTYPE_MAP);
+ cmd.get_ptype_map = (struct gve_adminq_get_ptype_map) {
+ .ptype_map_len = cpu_to_be64(sizeof(*ptype_map)),
+ .ptype_map_addr = cpu_to_be64(ptype_map_bus),
+ };
+
+ err = gve_adminq_execute_cmd(priv, &cmd);
+ if (err)
+ goto err;
+
+ /* Populate ptype_lut. */
+ for (i = 0; i < GVE_NUM_PTYPES; i++) {
+ ptype_lut->ptypes[i].l3_type =
+ ptype_map->ptypes[i].l3_type;
+ ptype_lut->ptypes[i].l4_type =
+ ptype_map->ptypes[i].l4_type;
+ }
+err:
+ dma_free_coherent(&priv->pdev->dev, sizeof(*ptype_map), ptype_map,
+ ptype_map_bus);
+ return err;
+}
/* SPDX-License-Identifier: (GPL-2.0 OR MIT)
* Google virtual Ethernet (gve) driver
*
- * Copyright (C) 2015-2019 Google, Inc.
+ * Copyright (C) 2015-2021 Google, Inc.
*/
#ifndef _GVE_ADMINQ_H
GVE_ADMINQ_DECONFIGURE_DEVICE_RESOURCES = 0x9,
GVE_ADMINQ_SET_DRIVER_PARAMETER = 0xB,
GVE_ADMINQ_REPORT_STATS = 0xC,
- GVE_ADMINQ_REPORT_LINK_SPEED = 0xD
+ GVE_ADMINQ_REPORT_LINK_SPEED = 0xD,
+ GVE_ADMINQ_GET_PTYPE_MAP = 0xE,
};
/* Admin queue status codes */
struct gve_device_option {
__be16 option_id;
__be16 option_length;
- __be32 feat_mask;
+ __be32 required_features_mask;
};
static_assert(sizeof(struct gve_device_option) == 8);
-#define GVE_DEV_OPT_ID_RAW_ADDRESSING 0x1
-#define GVE_DEV_OPT_LEN_RAW_ADDRESSING 0x0
-#define GVE_DEV_OPT_FEAT_MASK_RAW_ADDRESSING 0x0
+struct gve_device_option_gqi_rda {
+ __be32 supported_features_mask;
+};
+
+static_assert(sizeof(struct gve_device_option_gqi_rda) == 4);
+
+struct gve_device_option_gqi_qpl {
+ __be32 supported_features_mask;
+};
+
+static_assert(sizeof(struct gve_device_option_gqi_qpl) == 4);
+
+struct gve_device_option_dqo_rda {
+ __be32 supported_features_mask;
+ __be16 tx_comp_ring_entries;
+ __be16 rx_buff_ring_entries;
+};
+
+static_assert(sizeof(struct gve_device_option_dqo_rda) == 8);
+
+/* Terminology:
+ *
+ * RDA - Raw DMA Addressing - Buffers associated with SKBs are directly DMA
+ * mapped and read/updated by the device.
+ *
+ * QPL - Queue Page Lists - Driver uses bounce buffers which are DMA mapped with
+ * the device for read/write and data is copied from/to SKBs.
+ */
+enum gve_dev_opt_id {
+ GVE_DEV_OPT_ID_GQI_RAW_ADDRESSING = 0x1,
+ GVE_DEV_OPT_ID_GQI_RDA = 0x2,
+ GVE_DEV_OPT_ID_GQI_QPL = 0x3,
+ GVE_DEV_OPT_ID_DQO_RDA = 0x4,
+};
+
+enum gve_dev_opt_req_feat_mask {
+ GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RAW_ADDRESSING = 0x0,
+ GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RDA = 0x0,
+ GVE_DEV_OPT_REQ_FEAT_MASK_GQI_QPL = 0x0,
+ GVE_DEV_OPT_REQ_FEAT_MASK_DQO_RDA = 0x0,
+};
+
+#define GVE_DEV_OPT_LEN_GQI_RAW_ADDRESSING 0x0
struct gve_adminq_configure_device_resources {
__be64 counter_array;
__be32 num_irq_dbs;
__be32 irq_db_stride;
__be32 ntfy_blk_msix_base_idx;
+ u8 queue_format;
+ u8 padding[7];
};
-static_assert(sizeof(struct gve_adminq_configure_device_resources) == 32);
+static_assert(sizeof(struct gve_adminq_configure_device_resources) == 40);
struct gve_adminq_register_page_list {
__be32 page_list_id;
__be64 tx_ring_addr;
__be32 queue_page_list_id;
__be32 ntfy_id;
+ __be64 tx_comp_ring_addr;
+ __be16 tx_ring_size;
+ __be16 tx_comp_ring_size;
+ u8 padding[4];
};
-static_assert(sizeof(struct gve_adminq_create_tx_queue) == 32);
+static_assert(sizeof(struct gve_adminq_create_tx_queue) == 48);
struct gve_adminq_create_rx_queue {
__be32 queue_id;
__be64 rx_desc_ring_addr;
__be64 rx_data_ring_addr;
__be32 queue_page_list_id;
- u8 padding[4];
+ __be16 rx_ring_size;
+ __be16 packet_buffer_size;
+ __be16 rx_buff_ring_size;
+ u8 enable_rsc;
+ u8 padding[5];
};
-static_assert(sizeof(struct gve_adminq_create_rx_queue) == 48);
+static_assert(sizeof(struct gve_adminq_create_rx_queue) == 56);
/* Queue resources that are shared with the device */
struct gve_queue_resources {
RX_DROPS_INVALID_CHECKSUM = 68,
};
+enum gve_l3_type {
+ /* Must be zero so zero initialized LUT is unknown. */
+ GVE_L3_TYPE_UNKNOWN = 0,
+ GVE_L3_TYPE_OTHER,
+ GVE_L3_TYPE_IPV4,
+ GVE_L3_TYPE_IPV6,
+};
+
+enum gve_l4_type {
+ /* Must be zero so zero initialized LUT is unknown. */
+ GVE_L4_TYPE_UNKNOWN = 0,
+ GVE_L4_TYPE_OTHER,
+ GVE_L4_TYPE_TCP,
+ GVE_L4_TYPE_UDP,
+ GVE_L4_TYPE_ICMP,
+ GVE_L4_TYPE_SCTP,
+};
+
+/* These are control path types for PTYPE which are the same as the data path
+ * types.
+ */
+struct gve_ptype_entry {
+ u8 l3_type;
+ u8 l4_type;
+};
+
+struct gve_ptype_map {
+ struct gve_ptype_entry ptypes[1 << 10]; /* PTYPES are always 10 bits. */
+};
+
+struct gve_adminq_get_ptype_map {
+ __be64 ptype_map_len;
+ __be64 ptype_map_addr;
+};
+
union gve_adminq_command {
struct {
__be32 opcode;
struct gve_adminq_set_driver_parameter set_driver_param;
struct gve_adminq_report_stats report_stats;
struct gve_adminq_report_link_speed report_link_speed;
+ struct gve_adminq_get_ptype_map get_ptype_map;
};
};
u8 reserved[64];
int gve_adminq_report_stats(struct gve_priv *priv, u64 stats_report_len,
dma_addr_t stats_report_addr, u64 interval);
int gve_adminq_report_link_speed(struct gve_priv *priv);
+
+struct gve_ptype_lut;
+int gve_adminq_get_ptype_map_dqo(struct gve_priv *priv,
+ struct gve_ptype_lut *ptype_lut);
+
#endif /* _GVE_ADMINQ_H */
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0 OR MIT)
+ * Google virtual Ethernet (gve) driver
+ *
+ * Copyright (C) 2015-2021 Google, Inc.
+ */
+
+/* GVE DQO Descriptor formats */
+
+#ifndef _GVE_DESC_DQO_H_
+#define _GVE_DESC_DQO_H_
+
+#include <linux/build_bug.h>
+
+#define GVE_TX_MAX_HDR_SIZE_DQO 255
+#define GVE_TX_MIN_TSO_MSS_DQO 88
+
+#ifndef __LITTLE_ENDIAN_BITFIELD
+#error "Only little endian supported"
+#endif
+
+/* Basic TX descriptor (DTYPE 0x0C) */
+struct gve_tx_pkt_desc_dqo {
+ __le64 buf_addr;
+
+ /* Must be GVE_TX_PKT_DESC_DTYPE_DQO (0xc) */
+ u8 dtype: 5;
+
+ /* Denotes the last descriptor of a packet. */
+ u8 end_of_packet: 1;
+ u8 checksum_offload_enable: 1;
+
+ /* If set, will generate a descriptor completion for this descriptor. */
+ u8 report_event: 1;
+ u8 reserved0;
+ __le16 reserved1;
+
+ /* The TX completion associated with this packet will contain this tag.
+ */
+ __le16 compl_tag;
+ u16 buf_size: 14;
+ u16 reserved2: 2;
+} __packed;
+static_assert(sizeof(struct gve_tx_pkt_desc_dqo) == 16);
+
+#define GVE_TX_PKT_DESC_DTYPE_DQO 0xc
+#define GVE_TX_MAX_BUF_SIZE_DQO ((16 * 1024) - 1)
+
+/* Maximum number of data descriptors allowed per packet, or per-TSO segment. */
+#define GVE_TX_MAX_DATA_DESCS 10
+
+/* Min gap between tail and head to avoid cacheline overlap */
+#define GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP 4
+
+/* "report_event" on TX packet descriptors may only be reported on the last
+ * descriptor of a TX packet, and they must be spaced apart with at least this
+ * value.
+ */
+#define GVE_TX_MIN_RE_INTERVAL 32
+
+struct gve_tx_context_cmd_dtype {
+ u8 dtype: 5;
+ u8 tso: 1;
+ u8 reserved1: 2;
+
+ u8 reserved2;
+};
+
+static_assert(sizeof(struct gve_tx_context_cmd_dtype) == 2);
+
+/* TX Native TSO Context DTYPE (0x05)
+ *
+ * "flex" fields allow the driver to send additional packet context to HW.
+ */
+struct gve_tx_tso_context_desc_dqo {
+ /* The L4 payload bytes that should be segmented. */
+ u32 tso_total_len: 24;
+ u32 flex10: 8;
+
+ /* Max segment size in TSO excluding headers. */
+ u16 mss: 14;
+ u16 reserved: 2;
+
+ u8 header_len; /* Header length to use for TSO offload */
+ u8 flex11;
+ struct gve_tx_context_cmd_dtype cmd_dtype;
+ u8 flex0;
+ u8 flex5;
+ u8 flex6;
+ u8 flex7;
+ u8 flex8;
+ u8 flex9;
+} __packed;
+static_assert(sizeof(struct gve_tx_tso_context_desc_dqo) == 16);
+
+#define GVE_TX_TSO_CTX_DESC_DTYPE_DQO 0x5
+
+/* General context descriptor for sending metadata. */
+struct gve_tx_general_context_desc_dqo {
+ u8 flex4;
+ u8 flex5;
+ u8 flex6;
+ u8 flex7;
+ u8 flex8;
+ u8 flex9;
+ u8 flex10;
+ u8 flex11;
+ struct gve_tx_context_cmd_dtype cmd_dtype;
+ u16 reserved;
+ u8 flex0;
+ u8 flex1;
+ u8 flex2;
+ u8 flex3;
+} __packed;
+static_assert(sizeof(struct gve_tx_general_context_desc_dqo) == 16);
+
+#define GVE_TX_GENERAL_CTX_DESC_DTYPE_DQO 0x4
+
+/* Logical structure of metadata which is packed into context descriptor flex
+ * fields.
+ */
+struct gve_tx_metadata_dqo {
+ union {
+ struct {
+ u8 version;
+
+ /* If `skb->l4_hash` is set, this value should be
+ * derived from `skb->hash`.
+ *
+ * A zero value means no l4_hash was associated with the
+ * skb.
+ */
+ u16 path_hash: 15;
+
+ /* Should be set to 1 if the flow associated with the
+ * skb had a rehash from the TCP stack.
+ */
+ u16 rehash_event: 1;
+ } __packed;
+ u8 bytes[12];
+ };
+} __packed;
+static_assert(sizeof(struct gve_tx_metadata_dqo) == 12);
+
+#define GVE_TX_METADATA_VERSION_DQO 0
+
+/* TX completion descriptor */
+struct gve_tx_compl_desc {
+ /* For types 0-4 this is the TX queue ID associated with this
+ * completion.
+ */
+ u16 id: 11;
+
+ /* See: GVE_COMPL_TYPE_DQO* */
+ u16 type: 3;
+ u16 reserved0: 1;
+
+ /* Flipped by HW to notify the descriptor is populated. */
+ u16 generation: 1;
+ union {
+ /* For descriptor completions, this is the last index fetched
+ * by HW + 1.
+ */
+ __le16 tx_head;
+
+ /* For packet completions, this is the completion tag set on the
+ * TX packet descriptors.
+ */
+ __le16 completion_tag;
+ };
+ __le32 reserved1;
+} __packed;
+static_assert(sizeof(struct gve_tx_compl_desc) == 8);
+
+#define GVE_COMPL_TYPE_DQO_PKT 0x2 /* Packet completion */
+#define GVE_COMPL_TYPE_DQO_DESC 0x4 /* Descriptor completion */
+#define GVE_COMPL_TYPE_DQO_MISS 0x1 /* Miss path completion */
+#define GVE_COMPL_TYPE_DQO_REINJECTION 0x3 /* Re-injection completion */
+
+/* Descriptor to post buffers to HW on buffer queue. */
+struct gve_rx_desc_dqo {
+ __le16 buf_id; /* ID returned in Rx completion descriptor */
+ __le16 reserved0;
+ __le32 reserved1;
+ __le64 buf_addr; /* DMA address of the buffer */
+ __le64 header_buf_addr;
+ __le64 reserved2;
+} __packed;
+static_assert(sizeof(struct gve_rx_desc_dqo) == 32);
+
+/* Descriptor for HW to notify SW of new packets received on RX queue. */
+struct gve_rx_compl_desc_dqo {
+ /* Must be 1 */
+ u8 rxdid: 4;
+ u8 reserved0: 4;
+
+ /* Packet originated from this system rather than the network. */
+ u8 loopback: 1;
+ /* Set when IPv6 packet contains a destination options header or routing
+ * header.
+ */
+ u8 ipv6_ex_add: 1;
+ /* Invalid packet was received. */
+ u8 rx_error: 1;
+ u8 reserved1: 5;
+
+ u16 packet_type: 10;
+ u16 ip_hdr_err: 1;
+ u16 udp_len_err: 1;
+ u16 raw_cs_invalid: 1;
+ u16 reserved2: 3;
+
+ u16 packet_len: 14;
+ /* Flipped by HW to notify the descriptor is populated. */
+ u16 generation: 1;
+ /* Should be zero. */
+ u16 buffer_queue_id: 1;
+
+ u16 header_len: 10;
+ u16 rsc: 1;
+ u16 split_header: 1;
+ u16 reserved3: 4;
+
+ u8 descriptor_done: 1;
+ u8 end_of_packet: 1;
+ u8 header_buffer_overflow: 1;
+ u8 l3_l4_processed: 1;
+ u8 csum_ip_err: 1;
+ u8 csum_l4_err: 1;
+ u8 csum_external_ip_err: 1;
+ u8 csum_external_udp_err: 1;
+
+ u8 status_error1;
+
+ __le16 reserved5;
+ __le16 buf_id; /* Buffer ID which was sent on the buffer queue. */
+
+ union {
+ /* Packet checksum. */
+ __le16 raw_cs;
+ /* Segment length for RSC packets. */
+ __le16 rsc_seg_len;
+ };
+ __le32 hash;
+ __le32 reserved6;
+ __le64 reserved7;
+} __packed;
+
+static_assert(sizeof(struct gve_rx_compl_desc_dqo) == 32);
+
+/* Ringing the doorbell too often can hurt performance.
+ *
+ * HW requires this value to be at least 8.
+ */
+#define GVE_RX_BUF_THRESH_DQO 32
+
+#endif /* _GVE_DESC_DQO_H_ */
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0 OR MIT)
+ * Google virtual Ethernet (gve) driver
+ *
+ * Copyright (C) 2015-2021 Google, Inc.
+ */
+
+#ifndef _GVE_DQO_H_
+#define _GVE_DQO_H_
+
+#include "gve_adminq.h"
+
+#define GVE_ITR_ENABLE_BIT_DQO BIT(0)
+#define GVE_ITR_CLEAR_PBA_BIT_DQO BIT(1)
+#define GVE_ITR_NO_UPDATE_DQO (3 << 3)
+
+#define GVE_ITR_INTERVAL_DQO_SHIFT 5
+#define GVE_ITR_INTERVAL_DQO_MASK ((1 << 12) - 1)
+
+#define GVE_TX_IRQ_RATELIMIT_US_DQO 50
+#define GVE_RX_IRQ_RATELIMIT_US_DQO 20
+
+/* Timeout in seconds to wait for a reinjection completion after receiving
+ * its corresponding miss completion.
+ */
+#define GVE_REINJECT_COMPL_TIMEOUT 1
+
+/* Timeout in seconds to deallocate the completion tag for a packet that was
+ * prematurely freed for not receiving a valid completion. This should be large
+ * enough to rule out the possibility of receiving the corresponding valid
+ * completion after this interval.
+ */
+#define GVE_DEALLOCATE_COMPL_TIMEOUT 60
+
+netdev_tx_t gve_tx_dqo(struct sk_buff *skb, struct net_device *dev);
+bool gve_tx_poll_dqo(struct gve_notify_block *block, bool do_clean);
+int gve_rx_poll_dqo(struct gve_notify_block *block, int budget);
+int gve_tx_alloc_rings_dqo(struct gve_priv *priv);
+void gve_tx_free_rings_dqo(struct gve_priv *priv);
+int gve_rx_alloc_rings_dqo(struct gve_priv *priv);
+void gve_rx_free_rings_dqo(struct gve_priv *priv);
+int gve_clean_tx_done_dqo(struct gve_priv *priv, struct gve_tx_ring *tx,
+ struct napi_struct *napi);
+void gve_rx_post_buffers_dqo(struct gve_rx_ring *rx);
+void gve_rx_write_doorbell_dqo(const struct gve_priv *priv, int queue_idx);
+
+static inline void
+gve_tx_put_doorbell_dqo(const struct gve_priv *priv,
+ const struct gve_queue_resources *q_resources, u32 val)
+{
+ u64 index;
+
+ index = be32_to_cpu(q_resources->db_index);
+ iowrite32(val, &priv->db_bar2[index]);
+}
+
+/* Builds register value to write to DQO IRQ doorbell to enable with specified
+ * ratelimit.
+ */
+static inline u32 gve_set_itr_ratelimit_dqo(u32 ratelimit_us)
+{
+ u32 result = GVE_ITR_ENABLE_BIT_DQO;
+
+ /* Interval has 2us granularity. */
+ ratelimit_us >>= 1;
+
+ ratelimit_us &= GVE_ITR_INTERVAL_DQO_MASK;
+ result |= (ratelimit_us << GVE_ITR_INTERVAL_DQO_SHIFT);
+
+ return result;
+}
+
+static inline void
+gve_write_irq_doorbell_dqo(const struct gve_priv *priv,
+ const struct gve_notify_block *block, u32 val)
+{
+ u32 index = be32_to_cpu(block->irq_db_index);
+
+ iowrite32(val, &priv->db_bar2[index]);
+}
+
+#endif /* _GVE_DQO_H_ */
// SPDX-License-Identifier: (GPL-2.0 OR MIT)
/* Google virtual Ethernet (gve) driver
*
- * Copyright (C) 2015-2019 Google, Inc.
+ * Copyright (C) 2015-2021 Google, Inc.
*/
#include <linux/ethtool.h>
for (ring = 0; ring < priv->tx_cfg.num_queues; ring++) {
struct gve_tx_ring *tx = &priv->tx[ring];
- data[i++] = tx->req;
- data[i++] = tx->done;
+ if (gve_is_gqi(priv)) {
+ data[i++] = tx->req;
+ data[i++] = tx->done;
+ } else {
+ /* DQO doesn't currently support
+ * posted/completed descriptor counts;
+ */
+ data[i++] = 0;
+ data[i++] = 0;
+ }
do {
start =
u64_stats_fetch_begin(&priv->tx[ring].statss);
switch (etuna->id) {
case ETHTOOL_RX_COPYBREAK:
+ {
+ u32 max_copybreak = gve_is_gqi(priv) ?
+ (PAGE_SIZE / 2) : priv->data_buffer_size_dqo;
+
len = *(u32 *)value;
- if (len > PAGE_SIZE / 2)
+ if (len > max_copybreak)
return -EINVAL;
priv->rx_copybreak = len;
return 0;
+ }
default:
return -EOPNOTSUPP;
}
// SPDX-License-Identifier: (GPL-2.0 OR MIT)
/* Google virtual Ethernet (gve) driver
*
- * Copyright (C) 2015-2019 Google, Inc.
+ * Copyright (C) 2015-2021 Google, Inc.
*/
#include <linux/cpumask.h>
#include <linux/workqueue.h>
#include <net/sch_generic.h>
#include "gve.h"
+#include "gve_dqo.h"
#include "gve_adminq.h"
#include "gve_register.h"
const char gve_version_str[] = GVE_VERSION;
static const char gve_version_prefix[] = GVE_VERSION_PREFIX;
+static netdev_tx_t gve_start_xmit(struct sk_buff *skb, struct net_device *dev)
+{
+ struct gve_priv *priv = netdev_priv(dev);
+
+ if (gve_is_gqi(priv))
+ return gve_tx(skb, dev);
+ else
+ return gve_tx_dqo(skb, dev);
+}
+
static void gve_get_stats(struct net_device *dev, struct rtnl_link_stats64 *s)
{
struct gve_priv *priv = netdev_priv(dev);
return IRQ_HANDLED;
}
+static irqreturn_t gve_intr_dqo(int irq, void *arg)
+{
+ struct gve_notify_block *block = arg;
+
+ /* Interrupts are automatically masked */
+ napi_schedule_irqoff(&block->napi);
+ return IRQ_HANDLED;
+}
+
static int gve_napi_poll(struct napi_struct *napi, int budget)
{
struct gve_notify_block *block;
/* Double check we have no extra work.
* Ensure unmask synchronizes with checking for work.
*/
- dma_rmb();
+ mb();
if (block->tx)
reschedule |= gve_tx_poll(block, -1);
if (block->rx)
return 0;
}
+static int gve_napi_poll_dqo(struct napi_struct *napi, int budget)
+{
+ struct gve_notify_block *block =
+ container_of(napi, struct gve_notify_block, napi);
+ struct gve_priv *priv = block->priv;
+ bool reschedule = false;
+ int work_done = 0;
+
+ /* Clear PCI MSI-X Pending Bit Array (PBA)
+ *
+ * This bit is set if an interrupt event occurs while the vector is
+ * masked. If this bit is set and we reenable the interrupt, it will
+ * fire again. Since we're just about to poll the queue state, we don't
+ * need it to fire again.
+ *
+ * Under high softirq load, it's possible that the interrupt condition
+ * is triggered twice before we got the chance to process it.
+ */
+ gve_write_irq_doorbell_dqo(priv, block,
+ GVE_ITR_NO_UPDATE_DQO | GVE_ITR_CLEAR_PBA_BIT_DQO);
+
+ if (block->tx)
+ reschedule |= gve_tx_poll_dqo(block, /*do_clean=*/true);
+
+ if (block->rx) {
+ work_done = gve_rx_poll_dqo(block, budget);
+ reschedule |= work_done == budget;
+ }
+
+ if (reschedule)
+ return budget;
+
+ if (likely(napi_complete_done(napi, work_done))) {
+ /* Enable interrupts again.
+ *
+ * We don't need to repoll afterwards because HW supports the
+ * PCI MSI-X PBA feature.
+ *
+ * Another interrupt would be triggered if a new event came in
+ * since the last one.
+ */
+ gve_write_irq_doorbell_dqo(priv, block,
+ GVE_ITR_NO_UPDATE_DQO | GVE_ITR_ENABLE_BIT_DQO);
+ }
+
+ return work_done;
+}
+
static int gve_alloc_notify_blocks(struct gve_priv *priv)
{
int num_vecs_requested = priv->num_ntfy_blks + 1;
int vecs_left = new_num_ntfy_blks % 2;
priv->num_ntfy_blks = new_num_ntfy_blks;
+ priv->mgmt_msix_idx = priv->num_ntfy_blks;
priv->tx_cfg.max_queues = min_t(int, priv->tx_cfg.max_queues,
vecs_per_type);
priv->rx_cfg.max_queues = min_t(int, priv->rx_cfg.max_queues,
name, i);
block->priv = priv;
err = request_irq(priv->msix_vectors[msix_idx].vector,
- gve_intr, 0, block->name, block);
+ gve_is_gqi(priv) ? gve_intr : gve_intr_dqo,
+ 0, block->name, block);
if (err) {
dev_err(&priv->pdev->dev,
"Failed to receive msix vector %d\n", i);
{
int i;
- /* Free the irqs */
- for (i = 0; i < priv->num_ntfy_blks; i++) {
- struct gve_notify_block *block = &priv->ntfy_blocks[i];
- int msix_idx = i;
+ if (priv->msix_vectors) {
+ /* Free the irqs */
+ for (i = 0; i < priv->num_ntfy_blks; i++) {
+ struct gve_notify_block *block = &priv->ntfy_blocks[i];
+ int msix_idx = i;
- irq_set_affinity_hint(priv->msix_vectors[msix_idx].vector,
- NULL);
- free_irq(priv->msix_vectors[msix_idx].vector, block);
+ irq_set_affinity_hint(priv->msix_vectors[msix_idx].vector,
+ NULL);
+ free_irq(priv->msix_vectors[msix_idx].vector, block);
+ }
+ free_irq(priv->msix_vectors[priv->mgmt_msix_idx].vector, priv);
}
dma_free_coherent(&priv->pdev->dev,
priv->num_ntfy_blks * sizeof(*priv->ntfy_blocks),
priv->ntfy_blocks, priv->ntfy_block_bus);
priv->ntfy_blocks = NULL;
- free_irq(priv->msix_vectors[priv->mgmt_msix_idx].vector, priv);
pci_disable_msix(priv->pdev);
kvfree(priv->msix_vectors);
priv->msix_vectors = NULL;
err = -ENXIO;
goto abort_with_stats_report;
}
+
+ if (priv->queue_format == GVE_DQO_RDA_FORMAT) {
+ priv->ptype_lut_dqo = kvzalloc(sizeof(*priv->ptype_lut_dqo),
+ GFP_KERNEL);
+ if (!priv->ptype_lut_dqo) {
+ err = -ENOMEM;
+ goto abort_with_stats_report;
+ }
+ err = gve_adminq_get_ptype_map_dqo(priv, priv->ptype_lut_dqo);
+ if (err) {
+ dev_err(&priv->pdev->dev,
+ "Failed to get ptype map: err=%d\n", err);
+ goto abort_with_ptype_lut;
+ }
+ }
+
err = gve_adminq_report_stats(priv, priv->stats_report_len,
priv->stats_report_bus,
GVE_STATS_REPORT_TIMER_PERIOD);
"Failed to report stats: err=%d\n", err);
gve_set_device_resources_ok(priv);
return 0;
+
+abort_with_ptype_lut:
+ kvfree(priv->ptype_lut_dqo);
+ priv->ptype_lut_dqo = NULL;
abort_with_stats_report:
gve_free_stats_report(priv);
abort_with_ntfy_blocks:
gve_free_notify_blocks(priv);
abort_with_counter:
gve_free_counter_array(priv);
+
return err;
}
gve_trigger_reset(priv);
}
}
+
+ kvfree(priv->ptype_lut_dqo);
+ priv->ptype_lut_dqo = NULL;
+
gve_free_counter_array(priv);
gve_free_notify_blocks(priv);
gve_free_stats_report(priv);
gve_clear_device_resources_ok(priv);
}
-static void gve_add_napi(struct gve_priv *priv, int ntfy_idx)
+static void gve_add_napi(struct gve_priv *priv, int ntfy_idx,
+ int (*gve_poll)(struct napi_struct *, int))
{
struct gve_notify_block *block = &priv->ntfy_blocks[ntfy_idx];
- netif_napi_add(priv->dev, &block->napi, gve_napi_poll,
+ netif_napi_add(priv->dev, &block->napi, gve_poll,
NAPI_POLL_WEIGHT);
}
netif_dbg(priv, drv, priv->dev, "created %d rx queues\n",
priv->rx_cfg.num_queues);
- /* Rx data ring has been prefilled with packet buffers at queue
- * allocation time.
- * Write the doorbell to provide descriptor slots and packet buffers
- * to the NIC.
- */
- for (i = 0; i < priv->rx_cfg.num_queues; i++)
- gve_rx_write_doorbell(priv, &priv->rx[i]);
+ if (gve_is_gqi(priv)) {
+ /* Rx data ring has been prefilled with packet buffers at queue
+ * allocation time.
+ *
+ * Write the doorbell to provide descriptor slots and packet
+ * buffers to the NIC.
+ */
+ for (i = 0; i < priv->rx_cfg.num_queues; i++)
+ gve_rx_write_doorbell(priv, &priv->rx[i]);
+ } else {
+ for (i = 0; i < priv->rx_cfg.num_queues; i++) {
+ /* Post buffers and ring doorbell. */
+ gve_rx_post_buffers_dqo(&priv->rx[i]);
+ }
+ }
return 0;
}
+static void add_napi_init_sync_stats(struct gve_priv *priv,
+ int (*napi_poll)(struct napi_struct *napi,
+ int budget))
+{
+ int i;
+
+ /* Add tx napi & init sync stats*/
+ for (i = 0; i < priv->tx_cfg.num_queues; i++) {
+ int ntfy_idx = gve_tx_idx_to_ntfy(priv, i);
+
+ u64_stats_init(&priv->tx[i].statss);
+ priv->tx[i].ntfy_id = ntfy_idx;
+ gve_add_napi(priv, ntfy_idx, napi_poll);
+ }
+ /* Add rx napi & init sync stats*/
+ for (i = 0; i < priv->rx_cfg.num_queues; i++) {
+ int ntfy_idx = gve_rx_idx_to_ntfy(priv, i);
+
+ u64_stats_init(&priv->rx[i].statss);
+ priv->rx[i].ntfy_id = ntfy_idx;
+ gve_add_napi(priv, ntfy_idx, napi_poll);
+ }
+}
+
+static void gve_tx_free_rings(struct gve_priv *priv)
+{
+ if (gve_is_gqi(priv)) {
+ gve_tx_free_rings_gqi(priv);
+ } else {
+ gve_tx_free_rings_dqo(priv);
+ }
+}
+
static int gve_alloc_rings(struct gve_priv *priv)
{
- int ntfy_idx;
int err;
- int i;
/* Setup tx rings */
priv->tx = kvzalloc(priv->tx_cfg.num_queues * sizeof(*priv->tx),
GFP_KERNEL);
if (!priv->tx)
return -ENOMEM;
- err = gve_tx_alloc_rings(priv);
+
+ if (gve_is_gqi(priv))
+ err = gve_tx_alloc_rings(priv);
+ else
+ err = gve_tx_alloc_rings_dqo(priv);
if (err)
goto free_tx;
+
/* Setup rx rings */
priv->rx = kvzalloc(priv->rx_cfg.num_queues * sizeof(*priv->rx),
GFP_KERNEL);
err = -ENOMEM;
goto free_tx_queue;
}
- err = gve_rx_alloc_rings(priv);
+
+ if (gve_is_gqi(priv))
+ err = gve_rx_alloc_rings(priv);
+ else
+ err = gve_rx_alloc_rings_dqo(priv);
if (err)
goto free_rx;
- /* Add tx napi & init sync stats*/
- for (i = 0; i < priv->tx_cfg.num_queues; i++) {
- u64_stats_init(&priv->tx[i].statss);
- ntfy_idx = gve_tx_idx_to_ntfy(priv, i);
- gve_add_napi(priv, ntfy_idx);
- }
- /* Add rx napi & init sync stats*/
- for (i = 0; i < priv->rx_cfg.num_queues; i++) {
- u64_stats_init(&priv->rx[i].statss);
- ntfy_idx = gve_rx_idx_to_ntfy(priv, i);
- gve_add_napi(priv, ntfy_idx);
- }
+
+ if (gve_is_gqi(priv))
+ add_napi_init_sync_stats(priv, gve_napi_poll);
+ else
+ add_napi_init_sync_stats(priv, gve_napi_poll_dqo);
return 0;
return 0;
}
+static void gve_rx_free_rings(struct gve_priv *priv)
+{
+ if (gve_is_gqi(priv))
+ gve_rx_free_rings_gqi(priv);
+ else
+ gve_rx_free_rings_dqo(priv);
+}
+
static void gve_free_rings(struct gve_priv *priv)
{
int ntfy_idx;
int err;
/* Raw addressing means no QPLs */
- if (priv->raw_addressing)
+ if (priv->queue_format == GVE_GQI_RDA_FORMAT)
return 0;
priv->qpls = kvzalloc(num_qpls * sizeof(*priv->qpls), GFP_KERNEL);
int i;
/* Raw addressing means no QPLs */
- if (priv->raw_addressing)
+ if (priv->queue_format == GVE_GQI_RDA_FORMAT)
return;
kvfree(priv->qpl_cfg.qpl_id_map);
err = gve_alloc_qpls(priv);
if (err)
return err;
+
err = gve_alloc_rings(priv);
if (err)
goto free_qpls;
err = gve_register_qpls(priv);
if (err)
goto reset;
+
+ if (!gve_is_gqi(priv)) {
+ /* Hard code this for now. This may be tuned in the future for
+ * performance.
+ */
+ priv->data_buffer_size_dqo = GVE_RX_BUFFER_SIZE_DQO;
+ }
err = gve_create_rings(priv);
if (err)
goto reset;
+
gve_set_device_rings_ok(priv);
if (gve_get_report_stats(priv))
struct gve_notify_block *block = &priv->ntfy_blocks[ntfy_idx];
napi_enable(&block->napi);
- iowrite32be(0, gve_irq_doorbell(priv, block));
+ if (gve_is_gqi(priv)) {
+ iowrite32be(0, gve_irq_doorbell(priv, block));
+ } else {
+ u32 val = gve_set_itr_ratelimit_dqo(GVE_TX_IRQ_RATELIMIT_US_DQO);
+
+ gve_write_irq_doorbell_dqo(priv, block, val);
+ }
}
for (idx = 0; idx < priv->rx_cfg.num_queues; idx++) {
int ntfy_idx = gve_rx_idx_to_ntfy(priv, idx);
struct gve_notify_block *block = &priv->ntfy_blocks[ntfy_idx];
napi_enable(&block->napi);
- iowrite32be(0, gve_irq_doorbell(priv, block));
+ if (gve_is_gqi(priv)) {
+ iowrite32be(0, gve_irq_doorbell(priv, block));
+ } else {
+ u32 val = gve_set_itr_ratelimit_dqo(GVE_RX_IRQ_RATELIMIT_US_DQO);
+
+ gve_write_irq_doorbell_dqo(priv, block, val);
+ }
}
gve_set_napi_enabled(priv);
priv->tx_timeo_cnt++;
}
+static int gve_set_features(struct net_device *netdev,
+ netdev_features_t features)
+{
+ const netdev_features_t orig_features = netdev->features;
+ struct gve_priv *priv = netdev_priv(netdev);
+ int err;
+
+ if ((netdev->features & NETIF_F_LRO) != (features & NETIF_F_LRO)) {
+ netdev->features ^= NETIF_F_LRO;
+ if (netif_carrier_ok(netdev)) {
+ /* To make this process as simple as possible we
+ * teardown the device, set the new configuration,
+ * and then bring the device up again.
+ */
+ err = gve_close(netdev);
+ /* We have already tried to reset in close, just fail
+ * at this point.
+ */
+ if (err)
+ goto err;
+
+ err = gve_open(netdev);
+ if (err)
+ goto err;
+ }
+ }
+
+ return 0;
+err:
+ /* Reverts the change on error. */
+ netdev->features = orig_features;
+ netif_err(priv, drv, netdev,
+ "Set features failed! !!! DISABLING ALL QUEUES !!!\n");
+ return err;
+}
+
static const struct net_device_ops gve_netdev_ops = {
- .ndo_start_xmit = gve_tx,
+ .ndo_start_xmit = gve_start_xmit,
.ndo_open = gve_open,
.ndo_stop = gve_close,
.ndo_get_stats64 = gve_get_stats,
.ndo_tx_timeout = gve_tx_timeout,
+ .ndo_set_features = gve_set_features,
};
static void gve_handle_status(struct gve_priv *priv, u32 status)
/* tx stats */
if (priv->tx) {
for (idx = 0; idx < priv->tx_cfg.num_queues; idx++) {
+ u32 last_completion = 0;
+ u32 tx_frames = 0;
+
+ /* DQO doesn't currently support these metrics. */
+ if (gve_is_gqi(priv)) {
+ last_completion = priv->tx[idx].done;
+ tx_frames = priv->tx[idx].req;
+ }
+
do {
start = u64_stats_fetch_begin(&priv->tx[idx].statss);
tx_bytes = priv->tx[idx].bytes_done;
};
stats[stats_idx++] = (struct stats) {
.stat_name = cpu_to_be32(TX_FRAMES_SENT),
- .value = cpu_to_be64(priv->tx[idx].req),
+ .value = cpu_to_be64(tx_frames),
.queue_id = cpu_to_be32(idx),
};
stats[stats_idx++] = (struct stats) {
};
stats[stats_idx++] = (struct stats) {
.stat_name = cpu_to_be32(TX_LAST_COMPLETION_PROCESSED),
- .value = cpu_to_be64(priv->tx[idx].done),
+ .value = cpu_to_be64(last_completion),
.queue_id = cpu_to_be32(idx),
};
}
if (skip_describe_device)
goto setup_device;
- priv->raw_addressing = false;
+ priv->queue_format = GVE_QUEUE_FORMAT_UNSPECIFIED;
/* Get the initial information we need from the device */
err = gve_adminq_describe_device(priv);
if (err) {
"Could not get device information: err=%d\n", err);
goto err;
}
- if (priv->dev->max_mtu > PAGE_SIZE) {
+ if (gve_is_gqi(priv) && priv->dev->max_mtu > PAGE_SIZE) {
priv->dev->max_mtu = PAGE_SIZE;
err = gve_adminq_set_mtu(priv, priv->dev->mtu);
if (err) {
pci_set_drvdata(pdev, dev);
dev->ethtool_ops = &gve_ethtool_ops;
dev->netdev_ops = &gve_netdev_ops;
- /* advertise features */
+
+ /* Set default and supported features.
+ *
+ * Features might be set in other locations as well (such as
+ * `gve_adminq_describe_device`).
+ */
dev->hw_features = NETIF_F_HIGHDMA;
dev->hw_features |= NETIF_F_SG;
dev->hw_features |= NETIF_F_HW_CSUM;
goto abort_with_wq;
dev_info(&pdev->dev, "GVE version %s\n", gve_version_str);
+ dev_info(&pdev->dev, "GVE queue format %d\n", (int)priv->queue_format);
gve_clear_probe_in_progress(priv);
queue_work(priv->gve_wq, &priv->service_task);
return 0;
// SPDX-License-Identifier: (GPL-2.0 OR MIT)
/* Google virtual Ethernet (gve) driver
*
- * Copyright (C) 2015-2019 Google, Inc.
+ * Copyright (C) 2015-2021 Google, Inc.
*/
#include "gve.h"
#include "gve_adminq.h"
+#include "gve_utils.h"
#include <linux/etherdevice.h>
-static void gve_rx_remove_from_block(struct gve_priv *priv, int queue_idx)
-{
- struct gve_notify_block *block =
- &priv->ntfy_blocks[gve_rx_idx_to_ntfy(priv, queue_idx)];
-
- block->rx = NULL;
-}
-
static void gve_rx_free_buffer(struct device *dev,
struct gve_rx_slot_page_info *page_info,
union gve_rx_data_slot *data_slot)
return err;
}
-static void gve_rx_add_to_block(struct gve_priv *priv, int queue_idx)
-{
- u32 ntfy_idx = gve_rx_idx_to_ntfy(priv, queue_idx);
- struct gve_notify_block *block = &priv->ntfy_blocks[ntfy_idx];
- struct gve_rx_ring *rx = &priv->rx[queue_idx];
-
- block->rx = rx;
- rx->ntfy_id = ntfy_idx;
-}
-
static int gve_rx_alloc_ring(struct gve_priv *priv, int idx)
{
struct gve_rx_ring *rx = &priv->rx[idx];
slots = priv->rx_data_slot_cnt;
rx->mask = slots - 1;
- rx->data.raw_addressing = priv->raw_addressing;
+ rx->data.raw_addressing = priv->queue_format == GVE_GQI_RDA_FORMAT;
/* alloc rx data ring */
bytes = sizeof(*rx->data.data_ring) * slots;
return err;
}
-void gve_rx_free_rings(struct gve_priv *priv)
+void gve_rx_free_rings_gqi(struct gve_priv *priv)
{
int i;
return PKT_HASH_TYPE_L2;
}
-static struct sk_buff *gve_rx_copy(struct net_device *dev,
- struct napi_struct *napi,
- struct gve_rx_slot_page_info *page_info,
- u16 len)
-{
- struct sk_buff *skb = napi_alloc_skb(napi, len);
- void *va = page_info->page_address + GVE_RX_PAD +
- (page_info->page_offset ? PAGE_SIZE / 2 : 0);
-
- if (unlikely(!skb))
- return NULL;
-
- __skb_put(skb, len);
-
- skb_copy_to_linear_data(skb, va, len);
-
- skb->protocol = eth_type_trans(skb, dev);
-
- return skb;
-}
-
static struct sk_buff *gve_rx_add_frags(struct napi_struct *napi,
struct gve_rx_slot_page_info *page_info,
u16 len)
return NULL;
skb_add_rx_frag(skb, 0, page_info->page,
- (page_info->page_offset ? PAGE_SIZE / 2 : 0) +
+ page_info->page_offset +
GVE_RX_PAD, len, PAGE_SIZE / 2);
return skb;
const __be64 offset = cpu_to_be64(PAGE_SIZE / 2);
/* "flip" to other packet buffer on this page */
- page_info->page_offset ^= 0x1;
+ page_info->page_offset ^= PAGE_SIZE / 2;
*(slot_addr) ^= offset;
}
gve_rx_flip_buff(page_info, &data_slot->qpl_offset);
}
} else {
- skb = gve_rx_copy(netdev, napi, page_info, len);
+ skb = gve_rx_copy(netdev, napi, page_info, len, GVE_RX_PAD);
if (skb) {
u64_stats_update_begin(&rx->statss);
rx->rx_copied_pkt++;
if (len <= priv->rx_copybreak) {
/* Just copy small packets */
- skb = gve_rx_copy(dev, napi, page_info, len);
+ skb = gve_rx_copy(dev, napi, page_info, len, GVE_RX_PAD);
u64_stats_update_begin(&rx->statss);
rx->rx_copied_pkt++;
rx->rx_copybreak_pkt++;
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR MIT)
+/* Google virtual Ethernet (gve) driver
+ *
+ * Copyright (C) 2015-2021 Google, Inc.
+ */
+
+#include "gve.h"
+#include "gve_dqo.h"
+#include "gve_adminq.h"
+#include "gve_utils.h"
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include <linux/skbuff.h>
+#include <linux/slab.h>
+#include <net/ip6_checksum.h>
+#include <net/ipv6.h>
+#include <net/tcp.h>
+
+static int gve_buf_ref_cnt(struct gve_rx_buf_state_dqo *bs)
+{
+ return page_count(bs->page_info.page) - bs->page_info.pagecnt_bias;
+}
+
+static void gve_free_page_dqo(struct gve_priv *priv,
+ struct gve_rx_buf_state_dqo *bs)
+{
+ page_ref_sub(bs->page_info.page, bs->page_info.pagecnt_bias - 1);
+ gve_free_page(&priv->pdev->dev, bs->page_info.page, bs->addr,
+ DMA_FROM_DEVICE);
+ bs->page_info.page = NULL;
+}
+
+static struct gve_rx_buf_state_dqo *gve_alloc_buf_state(struct gve_rx_ring *rx)
+{
+ struct gve_rx_buf_state_dqo *buf_state;
+ s16 buffer_id;
+
+ buffer_id = rx->dqo.free_buf_states;
+ if (unlikely(buffer_id == -1))
+ return NULL;
+
+ buf_state = &rx->dqo.buf_states[buffer_id];
+
+ /* Remove buf_state from free list */
+ rx->dqo.free_buf_states = buf_state->next;
+
+ /* Point buf_state to itself to mark it as allocated */
+ buf_state->next = buffer_id;
+
+ return buf_state;
+}
+
+static bool gve_buf_state_is_allocated(struct gve_rx_ring *rx,
+ struct gve_rx_buf_state_dqo *buf_state)
+{
+ s16 buffer_id = buf_state - rx->dqo.buf_states;
+
+ return buf_state->next == buffer_id;
+}
+
+static void gve_free_buf_state(struct gve_rx_ring *rx,
+ struct gve_rx_buf_state_dqo *buf_state)
+{
+ s16 buffer_id = buf_state - rx->dqo.buf_states;
+
+ buf_state->next = rx->dqo.free_buf_states;
+ rx->dqo.free_buf_states = buffer_id;
+}
+
+static struct gve_rx_buf_state_dqo *
+gve_dequeue_buf_state(struct gve_rx_ring *rx, struct gve_index_list *list)
+{
+ struct gve_rx_buf_state_dqo *buf_state;
+ s16 buffer_id;
+
+ buffer_id = list->head;
+ if (unlikely(buffer_id == -1))
+ return NULL;
+
+ buf_state = &rx->dqo.buf_states[buffer_id];
+
+ /* Remove buf_state from list */
+ list->head = buf_state->next;
+ if (buf_state->next == -1)
+ list->tail = -1;
+
+ /* Point buf_state to itself to mark it as allocated */
+ buf_state->next = buffer_id;
+
+ return buf_state;
+}
+
+static void gve_enqueue_buf_state(struct gve_rx_ring *rx,
+ struct gve_index_list *list,
+ struct gve_rx_buf_state_dqo *buf_state)
+{
+ s16 buffer_id = buf_state - rx->dqo.buf_states;
+
+ buf_state->next = -1;
+
+ if (list->head == -1) {
+ list->head = buffer_id;
+ list->tail = buffer_id;
+ } else {
+ int tail = list->tail;
+
+ rx->dqo.buf_states[tail].next = buffer_id;
+ list->tail = buffer_id;
+ }
+}
+
+static struct gve_rx_buf_state_dqo *
+gve_get_recycled_buf_state(struct gve_rx_ring *rx)
+{
+ struct gve_rx_buf_state_dqo *buf_state;
+ int i;
+
+ /* Recycled buf states are immediately usable. */
+ buf_state = gve_dequeue_buf_state(rx, &rx->dqo.recycled_buf_states);
+ if (likely(buf_state))
+ return buf_state;
+
+ if (unlikely(rx->dqo.used_buf_states.head == -1))
+ return NULL;
+
+ /* Used buf states are only usable when ref count reaches 0, which means
+ * no SKBs refer to them.
+ *
+ * Search a limited number before giving up.
+ */
+ for (i = 0; i < 5; i++) {
+ buf_state = gve_dequeue_buf_state(rx, &rx->dqo.used_buf_states);
+ if (gve_buf_ref_cnt(buf_state) == 0)
+ return buf_state;
+
+ gve_enqueue_buf_state(rx, &rx->dqo.used_buf_states, buf_state);
+ }
+
+ /* If there are no free buf states discard an entry from
+ * `used_buf_states` so it can be used.
+ */
+ if (unlikely(rx->dqo.free_buf_states == -1)) {
+ buf_state = gve_dequeue_buf_state(rx, &rx->dqo.used_buf_states);
+ if (gve_buf_ref_cnt(buf_state) == 0)
+ return buf_state;
+
+ gve_free_page_dqo(rx->gve, buf_state);
+ gve_free_buf_state(rx, buf_state);
+ }
+
+ return NULL;
+}
+
+static int gve_alloc_page_dqo(struct gve_priv *priv,
+ struct gve_rx_buf_state_dqo *buf_state)
+{
+ int err;
+
+ err = gve_alloc_page(priv, &priv->pdev->dev, &buf_state->page_info.page,
+ &buf_state->addr, DMA_FROM_DEVICE);
+ if (err)
+ return err;
+
+ buf_state->page_info.page_offset = 0;
+ buf_state->page_info.page_address =
+ page_address(buf_state->page_info.page);
+ buf_state->last_single_ref_offset = 0;
+
+ /* The page already has 1 ref. */
+ page_ref_add(buf_state->page_info.page, INT_MAX - 1);
+ buf_state->page_info.pagecnt_bias = INT_MAX;
+
+ return 0;
+}
+
+static void gve_rx_free_ring_dqo(struct gve_priv *priv, int idx)
+{
+ struct gve_rx_ring *rx = &priv->rx[idx];
+ struct device *hdev = &priv->pdev->dev;
+ size_t completion_queue_slots;
+ size_t buffer_queue_slots;
+ size_t size;
+ int i;
+
+ completion_queue_slots = rx->dqo.complq.mask + 1;
+ buffer_queue_slots = rx->dqo.bufq.mask + 1;
+
+ gve_rx_remove_from_block(priv, idx);
+
+ if (rx->q_resources) {
+ dma_free_coherent(hdev, sizeof(*rx->q_resources),
+ rx->q_resources, rx->q_resources_bus);
+ rx->q_resources = NULL;
+ }
+
+ for (i = 0; i < rx->dqo.num_buf_states; i++) {
+ struct gve_rx_buf_state_dqo *bs = &rx->dqo.buf_states[i];
+
+ if (bs->page_info.page)
+ gve_free_page_dqo(priv, bs);
+ }
+
+ if (rx->dqo.bufq.desc_ring) {
+ size = sizeof(rx->dqo.bufq.desc_ring[0]) * buffer_queue_slots;
+ dma_free_coherent(hdev, size, rx->dqo.bufq.desc_ring,
+ rx->dqo.bufq.bus);
+ rx->dqo.bufq.desc_ring = NULL;
+ }
+
+ if (rx->dqo.complq.desc_ring) {
+ size = sizeof(rx->dqo.complq.desc_ring[0]) *
+ completion_queue_slots;
+ dma_free_coherent(hdev, size, rx->dqo.complq.desc_ring,
+ rx->dqo.complq.bus);
+ rx->dqo.complq.desc_ring = NULL;
+ }
+
+ kvfree(rx->dqo.buf_states);
+ rx->dqo.buf_states = NULL;
+
+ netif_dbg(priv, drv, priv->dev, "freed rx ring %d\n", idx);
+}
+
+static int gve_rx_alloc_ring_dqo(struct gve_priv *priv, int idx)
+{
+ struct gve_rx_ring *rx = &priv->rx[idx];
+ struct device *hdev = &priv->pdev->dev;
+ size_t size;
+ int i;
+
+ const u32 buffer_queue_slots =
+ priv->options_dqo_rda.rx_buff_ring_entries;
+ const u32 completion_queue_slots = priv->rx_desc_cnt;
+
+ netif_dbg(priv, drv, priv->dev, "allocating rx ring DQO\n");
+
+ memset(rx, 0, sizeof(*rx));
+ rx->gve = priv;
+ rx->q_num = idx;
+ rx->dqo.bufq.mask = buffer_queue_slots - 1;
+ rx->dqo.complq.num_free_slots = completion_queue_slots;
+ rx->dqo.complq.mask = completion_queue_slots - 1;
+ rx->skb_head = NULL;
+ rx->skb_tail = NULL;
+
+ rx->dqo.num_buf_states = min_t(s16, S16_MAX, buffer_queue_slots * 4);
+ rx->dqo.buf_states = kvcalloc(rx->dqo.num_buf_states,
+ sizeof(rx->dqo.buf_states[0]),
+ GFP_KERNEL);
+ if (!rx->dqo.buf_states)
+ return -ENOMEM;
+
+ /* Set up linked list of buffer IDs */
+ for (i = 0; i < rx->dqo.num_buf_states - 1; i++)
+ rx->dqo.buf_states[i].next = i + 1;
+
+ rx->dqo.buf_states[rx->dqo.num_buf_states - 1].next = -1;
+ rx->dqo.recycled_buf_states.head = -1;
+ rx->dqo.recycled_buf_states.tail = -1;
+ rx->dqo.used_buf_states.head = -1;
+ rx->dqo.used_buf_states.tail = -1;
+
+ /* Allocate RX completion queue */
+ size = sizeof(rx->dqo.complq.desc_ring[0]) *
+ completion_queue_slots;
+ rx->dqo.complq.desc_ring =
+ dma_alloc_coherent(hdev, size, &rx->dqo.complq.bus, GFP_KERNEL);
+ if (!rx->dqo.complq.desc_ring)
+ goto err;
+
+ /* Allocate RX buffer queue */
+ size = sizeof(rx->dqo.bufq.desc_ring[0]) * buffer_queue_slots;
+ rx->dqo.bufq.desc_ring =
+ dma_alloc_coherent(hdev, size, &rx->dqo.bufq.bus, GFP_KERNEL);
+ if (!rx->dqo.bufq.desc_ring)
+ goto err;
+
+ rx->q_resources = dma_alloc_coherent(hdev, sizeof(*rx->q_resources),
+ &rx->q_resources_bus, GFP_KERNEL);
+ if (!rx->q_resources)
+ goto err;
+
+ gve_rx_add_to_block(priv, idx);
+
+ return 0;
+
+err:
+ gve_rx_free_ring_dqo(priv, idx);
+ return -ENOMEM;
+}
+
+void gve_rx_write_doorbell_dqo(const struct gve_priv *priv, int queue_idx)
+{
+ const struct gve_rx_ring *rx = &priv->rx[queue_idx];
+ u64 index = be32_to_cpu(rx->q_resources->db_index);
+
+ iowrite32(rx->dqo.bufq.tail, &priv->db_bar2[index]);
+}
+
+int gve_rx_alloc_rings_dqo(struct gve_priv *priv)
+{
+ int err = 0;
+ int i;
+
+ for (i = 0; i < priv->rx_cfg.num_queues; i++) {
+ err = gve_rx_alloc_ring_dqo(priv, i);
+ if (err) {
+ netif_err(priv, drv, priv->dev,
+ "Failed to alloc rx ring=%d: err=%d\n",
+ i, err);
+ goto err;
+ }
+ }
+
+ return 0;
+
+err:
+ for (i--; i >= 0; i--)
+ gve_rx_free_ring_dqo(priv, i);
+
+ return err;
+}
+
+void gve_rx_free_rings_dqo(struct gve_priv *priv)
+{
+ int i;
+
+ for (i = 0; i < priv->rx_cfg.num_queues; i++)
+ gve_rx_free_ring_dqo(priv, i);
+}
+
+void gve_rx_post_buffers_dqo(struct gve_rx_ring *rx)
+{
+ struct gve_rx_compl_queue_dqo *complq = &rx->dqo.complq;
+ struct gve_rx_buf_queue_dqo *bufq = &rx->dqo.bufq;
+ struct gve_priv *priv = rx->gve;
+ u32 num_avail_slots;
+ u32 num_full_slots;
+ u32 num_posted = 0;
+
+ num_full_slots = (bufq->tail - bufq->head) & bufq->mask;
+ num_avail_slots = bufq->mask - num_full_slots;
+
+ num_avail_slots = min_t(u32, num_avail_slots, complq->num_free_slots);
+ while (num_posted < num_avail_slots) {
+ struct gve_rx_desc_dqo *desc = &bufq->desc_ring[bufq->tail];
+ struct gve_rx_buf_state_dqo *buf_state;
+
+ buf_state = gve_get_recycled_buf_state(rx);
+ if (unlikely(!buf_state)) {
+ buf_state = gve_alloc_buf_state(rx);
+ if (unlikely(!buf_state))
+ break;
+
+ if (unlikely(gve_alloc_page_dqo(priv, buf_state))) {
+ u64_stats_update_begin(&rx->statss);
+ rx->rx_buf_alloc_fail++;
+ u64_stats_update_end(&rx->statss);
+ gve_free_buf_state(rx, buf_state);
+ break;
+ }
+ }
+
+ desc->buf_id = cpu_to_le16(buf_state - rx->dqo.buf_states);
+ desc->buf_addr = cpu_to_le64(buf_state->addr +
+ buf_state->page_info.page_offset);
+
+ bufq->tail = (bufq->tail + 1) & bufq->mask;
+ complq->num_free_slots--;
+ num_posted++;
+
+ if ((bufq->tail & (GVE_RX_BUF_THRESH_DQO - 1)) == 0)
+ gve_rx_write_doorbell_dqo(priv, rx->q_num);
+ }
+
+ rx->fill_cnt += num_posted;
+}
+
+static void gve_try_recycle_buf(struct gve_priv *priv, struct gve_rx_ring *rx,
+ struct gve_rx_buf_state_dqo *buf_state)
+{
+ const int data_buffer_size = priv->data_buffer_size_dqo;
+ int pagecount;
+
+ /* Can't reuse if we only fit one buffer per page */
+ if (data_buffer_size * 2 > PAGE_SIZE)
+ goto mark_used;
+
+ pagecount = gve_buf_ref_cnt(buf_state);
+
+ /* Record the offset when we have a single remaining reference.
+ *
+ * When this happens, we know all of the other offsets of the page are
+ * usable.
+ */
+ if (pagecount == 1) {
+ buf_state->last_single_ref_offset =
+ buf_state->page_info.page_offset;
+ }
+
+ /* Use the next buffer sized chunk in the page. */
+ buf_state->page_info.page_offset += data_buffer_size;
+ buf_state->page_info.page_offset &= (PAGE_SIZE - 1);
+
+ /* If we wrap around to the same offset without ever dropping to 1
+ * reference, then we don't know if this offset was ever freed.
+ */
+ if (buf_state->page_info.page_offset ==
+ buf_state->last_single_ref_offset) {
+ goto mark_used;
+ }
+
+ gve_enqueue_buf_state(rx, &rx->dqo.recycled_buf_states, buf_state);
+ return;
+
+mark_used:
+ gve_enqueue_buf_state(rx, &rx->dqo.used_buf_states, buf_state);
+}
+
+static void gve_rx_skb_csum(struct sk_buff *skb,
+ const struct gve_rx_compl_desc_dqo *desc,
+ struct gve_ptype ptype)
+{
+ skb->ip_summed = CHECKSUM_NONE;
+
+ /* HW did not identify and process L3 and L4 headers. */
+ if (unlikely(!desc->l3_l4_processed))
+ return;
+
+ if (ptype.l3_type == GVE_L3_TYPE_IPV4) {
+ if (unlikely(desc->csum_ip_err || desc->csum_external_ip_err))
+ return;
+ } else if (ptype.l3_type == GVE_L3_TYPE_IPV6) {
+ /* Checksum should be skipped if this flag is set. */
+ if (unlikely(desc->ipv6_ex_add))
+ return;
+ }
+
+ if (unlikely(desc->csum_l4_err))
+ return;
+
+ switch (ptype.l4_type) {
+ case GVE_L4_TYPE_TCP:
+ case GVE_L4_TYPE_UDP:
+ case GVE_L4_TYPE_ICMP:
+ case GVE_L4_TYPE_SCTP:
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
+ break;
+ default:
+ break;
+ }
+}
+
+static void gve_rx_skb_hash(struct sk_buff *skb,
+ const struct gve_rx_compl_desc_dqo *compl_desc,
+ struct gve_ptype ptype)
+{
+ enum pkt_hash_types hash_type = PKT_HASH_TYPE_L2;
+
+ if (ptype.l4_type != GVE_L4_TYPE_UNKNOWN)
+ hash_type = PKT_HASH_TYPE_L4;
+ else if (ptype.l3_type != GVE_L3_TYPE_UNKNOWN)
+ hash_type = PKT_HASH_TYPE_L3;
+
+ skb_set_hash(skb, le32_to_cpu(compl_desc->hash), hash_type);
+}
+
+static void gve_rx_free_skb(struct gve_rx_ring *rx)
+{
+ if (!rx->skb_head)
+ return;
+
+ dev_kfree_skb_any(rx->skb_head);
+ rx->skb_head = NULL;
+ rx->skb_tail = NULL;
+}
+
+/* Chains multi skbs for single rx packet.
+ * Returns 0 if buffer is appended, -1 otherwise.
+ */
+static int gve_rx_append_frags(struct napi_struct *napi,
+ struct gve_rx_buf_state_dqo *buf_state,
+ u16 buf_len, struct gve_rx_ring *rx,
+ struct gve_priv *priv)
+{
+ int num_frags = skb_shinfo(rx->skb_tail)->nr_frags;
+
+ if (unlikely(num_frags == MAX_SKB_FRAGS)) {
+ struct sk_buff *skb;
+
+ skb = napi_alloc_skb(napi, 0);
+ if (!skb)
+ return -1;
+
+ skb_shinfo(rx->skb_tail)->frag_list = skb;
+ rx->skb_tail = skb;
+ num_frags = 0;
+ }
+ if (rx->skb_tail != rx->skb_head) {
+ rx->skb_head->len += buf_len;
+ rx->skb_head->data_len += buf_len;
+ rx->skb_head->truesize += priv->data_buffer_size_dqo;
+ }
+
+ skb_add_rx_frag(rx->skb_tail, num_frags,
+ buf_state->page_info.page,
+ buf_state->page_info.page_offset,
+ buf_len, priv->data_buffer_size_dqo);
+ gve_dec_pagecnt_bias(&buf_state->page_info);
+
+ return 0;
+}
+
+/* Returns 0 if descriptor is completed successfully.
+ * Returns -EINVAL if descriptor is invalid.
+ * Returns -ENOMEM if data cannot be copied to skb.
+ */
+static int gve_rx_dqo(struct napi_struct *napi, struct gve_rx_ring *rx,
+ const struct gve_rx_compl_desc_dqo *compl_desc,
+ int queue_idx)
+{
+ const u16 buffer_id = le16_to_cpu(compl_desc->buf_id);
+ const bool eop = compl_desc->end_of_packet != 0;
+ struct gve_rx_buf_state_dqo *buf_state;
+ struct gve_priv *priv = rx->gve;
+ u16 buf_len;
+
+ if (unlikely(buffer_id > rx->dqo.num_buf_states)) {
+ net_err_ratelimited("%s: Invalid RX buffer_id=%u\n",
+ priv->dev->name, buffer_id);
+ return -EINVAL;
+ }
+ buf_state = &rx->dqo.buf_states[buffer_id];
+ if (unlikely(!gve_buf_state_is_allocated(rx, buf_state))) {
+ net_err_ratelimited("%s: RX buffer_id is not allocated: %u\n",
+ priv->dev->name, buffer_id);
+ return -EINVAL;
+ }
+
+ if (unlikely(compl_desc->rx_error)) {
+ gve_enqueue_buf_state(rx, &rx->dqo.recycled_buf_states,
+ buf_state);
+ return -EINVAL;
+ }
+
+ buf_len = compl_desc->packet_len;
+
+ /* Page might have not been used for awhile and was likely last written
+ * by a different thread.
+ */
+ prefetch(buf_state->page_info.page);
+
+ /* Sync the portion of dma buffer for CPU to read. */
+ dma_sync_single_range_for_cpu(&priv->pdev->dev, buf_state->addr,
+ buf_state->page_info.page_offset,
+ buf_len, DMA_FROM_DEVICE);
+
+ /* Append to current skb if one exists. */
+ if (rx->skb_head) {
+ if (unlikely(gve_rx_append_frags(napi, buf_state, buf_len, rx,
+ priv)) != 0) {
+ goto error;
+ }
+
+ gve_try_recycle_buf(priv, rx, buf_state);
+ return 0;
+ }
+
+ /* Prefetch the payload header. */
+ prefetch((char *)buf_state->addr + buf_state->page_info.page_offset);
+#if L1_CACHE_BYTES < 128
+ prefetch((char *)buf_state->addr + buf_state->page_info.page_offset +
+ L1_CACHE_BYTES);
+#endif
+
+ if (eop && buf_len <= priv->rx_copybreak) {
+ rx->skb_head = gve_rx_copy(priv->dev, napi,
+ &buf_state->page_info, buf_len, 0);
+ if (unlikely(!rx->skb_head))
+ goto error;
+ rx->skb_tail = rx->skb_head;
+
+ u64_stats_update_begin(&rx->statss);
+ rx->rx_copied_pkt++;
+ rx->rx_copybreak_pkt++;
+ u64_stats_update_end(&rx->statss);
+
+ gve_enqueue_buf_state(rx, &rx->dqo.recycled_buf_states,
+ buf_state);
+ return 0;
+ }
+
+ rx->skb_head = napi_get_frags(napi);
+ if (unlikely(!rx->skb_head))
+ goto error;
+ rx->skb_tail = rx->skb_head;
+
+ skb_add_rx_frag(rx->skb_head, 0, buf_state->page_info.page,
+ buf_state->page_info.page_offset, buf_len,
+ priv->data_buffer_size_dqo);
+ gve_dec_pagecnt_bias(&buf_state->page_info);
+
+ gve_try_recycle_buf(priv, rx, buf_state);
+ return 0;
+
+error:
+ gve_enqueue_buf_state(rx, &rx->dqo.recycled_buf_states, buf_state);
+ return -ENOMEM;
+}
+
+static int gve_rx_complete_rsc(struct sk_buff *skb,
+ const struct gve_rx_compl_desc_dqo *desc,
+ struct gve_ptype ptype)
+{
+ struct skb_shared_info *shinfo = skb_shinfo(skb);
+
+ /* Only TCP is supported right now. */
+ if (ptype.l4_type != GVE_L4_TYPE_TCP)
+ return -EINVAL;
+
+ switch (ptype.l3_type) {
+ case GVE_L3_TYPE_IPV4:
+ shinfo->gso_type = SKB_GSO_TCPV4;
+ break;
+ case GVE_L3_TYPE_IPV6:
+ shinfo->gso_type = SKB_GSO_TCPV6;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ shinfo->gso_size = le16_to_cpu(desc->rsc_seg_len);
+ return 0;
+}
+
+/* Returns 0 if skb is completed successfully, -1 otherwise. */
+static int gve_rx_complete_skb(struct gve_rx_ring *rx, struct napi_struct *napi,
+ const struct gve_rx_compl_desc_dqo *desc,
+ netdev_features_t feat)
+{
+ struct gve_ptype ptype =
+ rx->gve->ptype_lut_dqo->ptypes[desc->packet_type];
+ int err;
+
+ skb_record_rx_queue(rx->skb_head, rx->q_num);
+
+ if (feat & NETIF_F_RXHASH)
+ gve_rx_skb_hash(rx->skb_head, desc, ptype);
+
+ if (feat & NETIF_F_RXCSUM)
+ gve_rx_skb_csum(rx->skb_head, desc, ptype);
+
+ /* RSC packets must set gso_size otherwise the TCP stack will complain
+ * that packets are larger than MTU.
+ */
+ if (desc->rsc) {
+ err = gve_rx_complete_rsc(rx->skb_head, desc, ptype);
+ if (err < 0)
+ return err;
+ }
+
+ if (skb_headlen(rx->skb_head) == 0)
+ napi_gro_frags(napi);
+ else
+ napi_gro_receive(napi, rx->skb_head);
+
+ return 0;
+}
+
+int gve_rx_poll_dqo(struct gve_notify_block *block, int budget)
+{
+ struct napi_struct *napi = &block->napi;
+ netdev_features_t feat = napi->dev->features;
+
+ struct gve_rx_ring *rx = block->rx;
+ struct gve_rx_compl_queue_dqo *complq = &rx->dqo.complq;
+
+ u32 work_done = 0;
+ u64 bytes = 0;
+ int err;
+
+ while (work_done < budget) {
+ struct gve_rx_compl_desc_dqo *compl_desc =
+ &complq->desc_ring[complq->head];
+ u32 pkt_bytes;
+
+ /* No more new packets */
+ if (compl_desc->generation == complq->cur_gen_bit)
+ break;
+
+ /* Prefetch the next two descriptors. */
+ prefetch(&complq->desc_ring[(complq->head + 1) & complq->mask]);
+ prefetch(&complq->desc_ring[(complq->head + 2) & complq->mask]);
+
+ /* Do not read data until we own the descriptor */
+ dma_rmb();
+
+ err = gve_rx_dqo(napi, rx, compl_desc, rx->q_num);
+ if (err < 0) {
+ gve_rx_free_skb(rx);
+ u64_stats_update_begin(&rx->statss);
+ if (err == -ENOMEM)
+ rx->rx_skb_alloc_fail++;
+ else if (err == -EINVAL)
+ rx->rx_desc_err_dropped_pkt++;
+ u64_stats_update_end(&rx->statss);
+ }
+
+ complq->head = (complq->head + 1) & complq->mask;
+ complq->num_free_slots++;
+
+ /* When the ring wraps, the generation bit is flipped. */
+ complq->cur_gen_bit ^= (complq->head == 0);
+
+ /* Receiving a completion means we have space to post another
+ * buffer on the buffer queue.
+ */
+ {
+ struct gve_rx_buf_queue_dqo *bufq = &rx->dqo.bufq;
+
+ bufq->head = (bufq->head + 1) & bufq->mask;
+ }
+
+ /* Free running counter of completed descriptors */
+ rx->cnt++;
+
+ if (!rx->skb_head)
+ continue;
+
+ if (!compl_desc->end_of_packet)
+ continue;
+
+ work_done++;
+ pkt_bytes = rx->skb_head->len;
+ /* The ethernet header (first ETH_HLEN bytes) is snipped off
+ * by eth_type_trans.
+ */
+ if (skb_headlen(rx->skb_head))
+ pkt_bytes += ETH_HLEN;
+
+ /* gve_rx_complete_skb() will consume skb if successful */
+ if (gve_rx_complete_skb(rx, napi, compl_desc, feat) != 0) {
+ gve_rx_free_skb(rx);
+ u64_stats_update_begin(&rx->statss);
+ rx->rx_desc_err_dropped_pkt++;
+ u64_stats_update_end(&rx->statss);
+ continue;
+ }
+
+ bytes += pkt_bytes;
+ rx->skb_head = NULL;
+ rx->skb_tail = NULL;
+ }
+
+ gve_rx_post_buffers_dqo(rx);
+
+ u64_stats_update_begin(&rx->statss);
+ rx->rpackets += work_done;
+ rx->rbytes += bytes;
+ u64_stats_update_end(&rx->statss);
+
+ return work_done;
+}
// SPDX-License-Identifier: (GPL-2.0 OR MIT)
/* Google virtual Ethernet (gve) driver
*
- * Copyright (C) 2015-2019 Google, Inc.
+ * Copyright (C) 2015-2021 Google, Inc.
*/
#include "gve.h"
#include "gve_adminq.h"
+#include "gve_utils.h"
#include <linux/ip.h>
#include <linux/tcp.h>
#include <linux/vmalloc.h>
atomic_add(bytes, &fifo->available);
}
-static void gve_tx_remove_from_block(struct gve_priv *priv, int queue_idx)
-{
- struct gve_notify_block *block =
- &priv->ntfy_blocks[gve_tx_idx_to_ntfy(priv, queue_idx)];
-
- block->tx = NULL;
-}
-
static int gve_clean_tx_done(struct gve_priv *priv, struct gve_tx_ring *tx,
u32 to_do, bool try_to_wake);
netif_dbg(priv, drv, priv->dev, "freed tx queue %d\n", idx);
}
-static void gve_tx_add_to_block(struct gve_priv *priv, int queue_idx)
-{
- int ntfy_idx = gve_tx_idx_to_ntfy(priv, queue_idx);
- struct gve_notify_block *block = &priv->ntfy_blocks[ntfy_idx];
- struct gve_tx_ring *tx = &priv->tx[queue_idx];
-
- block->tx = tx;
- tx->ntfy_id = ntfy_idx;
-}
-
static int gve_tx_alloc_ring(struct gve_priv *priv, int idx)
{
struct gve_tx_ring *tx = &priv->tx[idx];
if (!tx->desc)
goto abort_with_info;
- tx->raw_addressing = priv->raw_addressing;
+ tx->raw_addressing = priv->queue_format == GVE_GQI_RDA_FORMAT;
tx->dev = &priv->pdev->dev;
if (!tx->raw_addressing) {
tx->tx_fifo.qpl = gve_assign_tx_qpl(priv);
-
+ if (!tx->tx_fifo.qpl)
+ goto abort_with_desc;
/* map Tx FIFO */
if (gve_tx_fifo_init(priv, &tx->tx_fifo))
- goto abort_with_desc;
+ goto abort_with_qpl;
}
tx->q_resources =
abort_with_fifo:
if (!tx->raw_addressing)
gve_tx_fifo_release(priv, &tx->tx_fifo);
+abort_with_qpl:
+ if (!tx->raw_addressing)
+ gve_unassign_qpl(priv, tx->tx_fifo.qpl->id);
abort_with_desc:
dma_free_coherent(hdev, bytes, tx->desc, tx->bus);
tx->desc = NULL;
return err;
}
-void gve_tx_free_rings(struct gve_priv *priv)
+void gve_tx_free_rings_gqi(struct gve_priv *priv)
{
int i;
struct gve_tx_ring *tx;
int nsegs;
- WARN(skb_get_queue_mapping(skb) > priv->tx_cfg.num_queues,
+ WARN(skb_get_queue_mapping(skb) >= priv->tx_cfg.num_queues,
"skb queue index out of range");
tx = &priv->tx[skb_get_queue_mapping(skb)];
if (unlikely(gve_maybe_stop_tx(tx, skb))) {
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR MIT)
+/* Google virtual Ethernet (gve) driver
+ *
+ * Copyright (C) 2015-2021 Google, Inc.
+ */
+
+#include "gve.h"
+#include "gve_adminq.h"
+#include "gve_utils.h"
+#include "gve_dqo.h"
+#include <linux/tcp.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+
+/* Returns true if a gve_tx_pending_packet_dqo object is available. */
+static bool gve_has_pending_packet(struct gve_tx_ring *tx)
+{
+ /* Check TX path's list. */
+ if (tx->dqo_tx.free_pending_packets != -1)
+ return true;
+
+ /* Check completion handler's list. */
+ if (atomic_read_acquire(&tx->dqo_compl.free_pending_packets) != -1)
+ return true;
+
+ return false;
+}
+
+static struct gve_tx_pending_packet_dqo *
+gve_alloc_pending_packet(struct gve_tx_ring *tx)
+{
+ struct gve_tx_pending_packet_dqo *pending_packet;
+ s16 index;
+
+ index = tx->dqo_tx.free_pending_packets;
+
+ /* No pending_packets available, try to steal the list from the
+ * completion handler.
+ */
+ if (unlikely(index == -1)) {
+ tx->dqo_tx.free_pending_packets =
+ atomic_xchg(&tx->dqo_compl.free_pending_packets, -1);
+ index = tx->dqo_tx.free_pending_packets;
+
+ if (unlikely(index == -1))
+ return NULL;
+ }
+
+ pending_packet = &tx->dqo.pending_packets[index];
+
+ /* Remove pending_packet from free list */
+ tx->dqo_tx.free_pending_packets = pending_packet->next;
+ pending_packet->state = GVE_PACKET_STATE_PENDING_DATA_COMPL;
+
+ return pending_packet;
+}
+
+static void
+gve_free_pending_packet(struct gve_tx_ring *tx,
+ struct gve_tx_pending_packet_dqo *pending_packet)
+{
+ s16 index = pending_packet - tx->dqo.pending_packets;
+
+ pending_packet->state = GVE_PACKET_STATE_UNALLOCATED;
+ while (true) {
+ s16 old_head = atomic_read_acquire(&tx->dqo_compl.free_pending_packets);
+
+ pending_packet->next = old_head;
+ if (atomic_cmpxchg(&tx->dqo_compl.free_pending_packets,
+ old_head, index) == old_head) {
+ break;
+ }
+ }
+}
+
+/* gve_tx_free_desc - Cleans up all pending tx requests and buffers.
+ */
+static void gve_tx_clean_pending_packets(struct gve_tx_ring *tx)
+{
+ int i;
+
+ for (i = 0; i < tx->dqo.num_pending_packets; i++) {
+ struct gve_tx_pending_packet_dqo *cur_state =
+ &tx->dqo.pending_packets[i];
+ int j;
+
+ for (j = 0; j < cur_state->num_bufs; j++) {
+ struct gve_tx_dma_buf *buf = &cur_state->bufs[j];
+
+ if (j == 0) {
+ dma_unmap_single(tx->dev,
+ dma_unmap_addr(buf, dma),
+ dma_unmap_len(buf, len),
+ DMA_TO_DEVICE);
+ } else {
+ dma_unmap_page(tx->dev,
+ dma_unmap_addr(buf, dma),
+ dma_unmap_len(buf, len),
+ DMA_TO_DEVICE);
+ }
+ }
+ if (cur_state->skb) {
+ dev_consume_skb_any(cur_state->skb);
+ cur_state->skb = NULL;
+ }
+ }
+}
+
+static void gve_tx_free_ring_dqo(struct gve_priv *priv, int idx)
+{
+ struct gve_tx_ring *tx = &priv->tx[idx];
+ struct device *hdev = &priv->pdev->dev;
+ size_t bytes;
+
+ gve_tx_remove_from_block(priv, idx);
+
+ if (tx->q_resources) {
+ dma_free_coherent(hdev, sizeof(*tx->q_resources),
+ tx->q_resources, tx->q_resources_bus);
+ tx->q_resources = NULL;
+ }
+
+ if (tx->dqo.compl_ring) {
+ bytes = sizeof(tx->dqo.compl_ring[0]) *
+ (tx->dqo.complq_mask + 1);
+ dma_free_coherent(hdev, bytes, tx->dqo.compl_ring,
+ tx->complq_bus_dqo);
+ tx->dqo.compl_ring = NULL;
+ }
+
+ if (tx->dqo.tx_ring) {
+ bytes = sizeof(tx->dqo.tx_ring[0]) * (tx->mask + 1);
+ dma_free_coherent(hdev, bytes, tx->dqo.tx_ring, tx->bus);
+ tx->dqo.tx_ring = NULL;
+ }
+
+ kvfree(tx->dqo.pending_packets);
+ tx->dqo.pending_packets = NULL;
+
+ netif_dbg(priv, drv, priv->dev, "freed tx queue %d\n", idx);
+}
+
+static int gve_tx_alloc_ring_dqo(struct gve_priv *priv, int idx)
+{
+ struct gve_tx_ring *tx = &priv->tx[idx];
+ struct device *hdev = &priv->pdev->dev;
+ int num_pending_packets;
+ size_t bytes;
+ int i;
+
+ memset(tx, 0, sizeof(*tx));
+ tx->q_num = idx;
+ tx->dev = &priv->pdev->dev;
+ tx->netdev_txq = netdev_get_tx_queue(priv->dev, idx);
+ atomic_set_release(&tx->dqo_compl.hw_tx_head, 0);
+
+ /* Queue sizes must be a power of 2 */
+ tx->mask = priv->tx_desc_cnt - 1;
+ tx->dqo.complq_mask = priv->options_dqo_rda.tx_comp_ring_entries - 1;
+
+ /* The max number of pending packets determines the maximum number of
+ * descriptors which maybe written to the completion queue.
+ *
+ * We must set the number small enough to make sure we never overrun the
+ * completion queue.
+ */
+ num_pending_packets = tx->dqo.complq_mask + 1;
+
+ /* Reserve space for descriptor completions, which will be reported at
+ * most every GVE_TX_MIN_RE_INTERVAL packets.
+ */
+ num_pending_packets -=
+ (tx->dqo.complq_mask + 1) / GVE_TX_MIN_RE_INTERVAL;
+
+ /* Each packet may have at most 2 buffer completions if it receives both
+ * a miss and reinjection completion.
+ */
+ num_pending_packets /= 2;
+
+ tx->dqo.num_pending_packets = min_t(int, num_pending_packets, S16_MAX);
+ tx->dqo.pending_packets = kvcalloc(tx->dqo.num_pending_packets,
+ sizeof(tx->dqo.pending_packets[0]),
+ GFP_KERNEL);
+ if (!tx->dqo.pending_packets)
+ goto err;
+
+ /* Set up linked list of pending packets */
+ for (i = 0; i < tx->dqo.num_pending_packets - 1; i++)
+ tx->dqo.pending_packets[i].next = i + 1;
+
+ tx->dqo.pending_packets[tx->dqo.num_pending_packets - 1].next = -1;
+ atomic_set_release(&tx->dqo_compl.free_pending_packets, -1);
+ tx->dqo_compl.miss_completions.head = -1;
+ tx->dqo_compl.miss_completions.tail = -1;
+ tx->dqo_compl.timed_out_completions.head = -1;
+ tx->dqo_compl.timed_out_completions.tail = -1;
+
+ bytes = sizeof(tx->dqo.tx_ring[0]) * (tx->mask + 1);
+ tx->dqo.tx_ring = dma_alloc_coherent(hdev, bytes, &tx->bus, GFP_KERNEL);
+ if (!tx->dqo.tx_ring)
+ goto err;
+
+ bytes = sizeof(tx->dqo.compl_ring[0]) * (tx->dqo.complq_mask + 1);
+ tx->dqo.compl_ring = dma_alloc_coherent(hdev, bytes,
+ &tx->complq_bus_dqo,
+ GFP_KERNEL);
+ if (!tx->dqo.compl_ring)
+ goto err;
+
+ tx->q_resources = dma_alloc_coherent(hdev, sizeof(*tx->q_resources),
+ &tx->q_resources_bus, GFP_KERNEL);
+ if (!tx->q_resources)
+ goto err;
+
+ gve_tx_add_to_block(priv, idx);
+
+ return 0;
+
+err:
+ gve_tx_free_ring_dqo(priv, idx);
+ return -ENOMEM;
+}
+
+int gve_tx_alloc_rings_dqo(struct gve_priv *priv)
+{
+ int err = 0;
+ int i;
+
+ for (i = 0; i < priv->tx_cfg.num_queues; i++) {
+ err = gve_tx_alloc_ring_dqo(priv, i);
+ if (err) {
+ netif_err(priv, drv, priv->dev,
+ "Failed to alloc tx ring=%d: err=%d\n",
+ i, err);
+ goto err;
+ }
+ }
+
+ return 0;
+
+err:
+ for (i--; i >= 0; i--)
+ gve_tx_free_ring_dqo(priv, i);
+
+ return err;
+}
+
+void gve_tx_free_rings_dqo(struct gve_priv *priv)
+{
+ int i;
+
+ for (i = 0; i < priv->tx_cfg.num_queues; i++) {
+ struct gve_tx_ring *tx = &priv->tx[i];
+
+ gve_clean_tx_done_dqo(priv, tx, /*napi=*/NULL);
+ netdev_tx_reset_queue(tx->netdev_txq);
+ gve_tx_clean_pending_packets(tx);
+
+ gve_tx_free_ring_dqo(priv, i);
+ }
+}
+
+/* Returns the number of slots available in the ring */
+static u32 num_avail_tx_slots(const struct gve_tx_ring *tx)
+{
+ u32 num_used = (tx->dqo_tx.tail - tx->dqo_tx.head) & tx->mask;
+
+ return tx->mask - num_used;
+}
+
+/* Stops the queue if available descriptors is less than 'count'.
+ * Return: 0 if stop is not required.
+ */
+static int gve_maybe_stop_tx_dqo(struct gve_tx_ring *tx, int count)
+{
+ if (likely(gve_has_pending_packet(tx) &&
+ num_avail_tx_slots(tx) >= count))
+ return 0;
+
+ /* Update cached TX head pointer */
+ tx->dqo_tx.head = atomic_read_acquire(&tx->dqo_compl.hw_tx_head);
+
+ if (likely(gve_has_pending_packet(tx) &&
+ num_avail_tx_slots(tx) >= count))
+ return 0;
+
+ /* No space, so stop the queue */
+ tx->stop_queue++;
+ netif_tx_stop_queue(tx->netdev_txq);
+
+ /* Sync with restarting queue in `gve_tx_poll_dqo()` */
+ mb();
+
+ /* After stopping queue, check if we can transmit again in order to
+ * avoid TOCTOU bug.
+ */
+ tx->dqo_tx.head = atomic_read_acquire(&tx->dqo_compl.hw_tx_head);
+
+ if (likely(!gve_has_pending_packet(tx) ||
+ num_avail_tx_slots(tx) < count))
+ return -EBUSY;
+
+ netif_tx_start_queue(tx->netdev_txq);
+ tx->wake_queue++;
+ return 0;
+}
+
+static void gve_extract_tx_metadata_dqo(const struct sk_buff *skb,
+ struct gve_tx_metadata_dqo *metadata)
+{
+ memset(metadata, 0, sizeof(*metadata));
+ metadata->version = GVE_TX_METADATA_VERSION_DQO;
+
+ if (skb->l4_hash) {
+ u16 path_hash = skb->hash ^ (skb->hash >> 16);
+
+ path_hash &= (1 << 15) - 1;
+ if (unlikely(path_hash == 0))
+ path_hash = ~path_hash;
+
+ metadata->path_hash = path_hash;
+ }
+}
+
+static void gve_tx_fill_pkt_desc_dqo(struct gve_tx_ring *tx, u32 *desc_idx,
+ struct sk_buff *skb, u32 len, u64 addr,
+ s16 compl_tag, bool eop, bool is_gso)
+{
+ const bool checksum_offload_en = skb->ip_summed == CHECKSUM_PARTIAL;
+
+ while (len > 0) {
+ struct gve_tx_pkt_desc_dqo *desc =
+ &tx->dqo.tx_ring[*desc_idx].pkt;
+ u32 cur_len = min_t(u32, len, GVE_TX_MAX_BUF_SIZE_DQO);
+ bool cur_eop = eop && cur_len == len;
+
+ *desc = (struct gve_tx_pkt_desc_dqo){
+ .buf_addr = cpu_to_le64(addr),
+ .dtype = GVE_TX_PKT_DESC_DTYPE_DQO,
+ .end_of_packet = cur_eop,
+ .checksum_offload_enable = checksum_offload_en,
+ .compl_tag = cpu_to_le16(compl_tag),
+ .buf_size = cur_len,
+ };
+
+ addr += cur_len;
+ len -= cur_len;
+ *desc_idx = (*desc_idx + 1) & tx->mask;
+ }
+}
+
+/* Validates and prepares `skb` for TSO.
+ *
+ * Returns header length, or < 0 if invalid.
+ */
+static int gve_prep_tso(struct sk_buff *skb)
+{
+ struct tcphdr *tcp;
+ int header_len;
+ u32 paylen;
+ int err;
+
+ /* Note: HW requires MSS (gso_size) to be <= 9728 and the total length
+ * of the TSO to be <= 262143.
+ *
+ * However, we don't validate these because:
+ * - Hypervisor enforces a limit of 9K MTU
+ * - Kernel will not produce a TSO larger than 64k
+ */
+
+ if (unlikely(skb_shinfo(skb)->gso_size < GVE_TX_MIN_TSO_MSS_DQO))
+ return -1;
+
+ /* Needed because we will modify header. */
+ err = skb_cow_head(skb, 0);
+ if (err < 0)
+ return err;
+
+ tcp = tcp_hdr(skb);
+
+ /* Remove payload length from checksum. */
+ paylen = skb->len - skb_transport_offset(skb);
+
+ switch (skb_shinfo(skb)->gso_type) {
+ case SKB_GSO_TCPV4:
+ case SKB_GSO_TCPV6:
+ csum_replace_by_diff(&tcp->check,
+ (__force __wsum)htonl(paylen));
+
+ /* Compute length of segmentation header. */
+ header_len = skb_transport_offset(skb) + tcp_hdrlen(skb);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (unlikely(header_len > GVE_TX_MAX_HDR_SIZE_DQO))
+ return -EINVAL;
+
+ return header_len;
+}
+
+static void gve_tx_fill_tso_ctx_desc(struct gve_tx_tso_context_desc_dqo *desc,
+ const struct sk_buff *skb,
+ const struct gve_tx_metadata_dqo *metadata,
+ int header_len)
+{
+ *desc = (struct gve_tx_tso_context_desc_dqo){
+ .header_len = header_len,
+ .cmd_dtype = {
+ .dtype = GVE_TX_TSO_CTX_DESC_DTYPE_DQO,
+ .tso = 1,
+ },
+ .flex0 = metadata->bytes[0],
+ .flex5 = metadata->bytes[5],
+ .flex6 = metadata->bytes[6],
+ .flex7 = metadata->bytes[7],
+ .flex8 = metadata->bytes[8],
+ .flex9 = metadata->bytes[9],
+ .flex10 = metadata->bytes[10],
+ .flex11 = metadata->bytes[11],
+ };
+ desc->tso_total_len = skb->len - header_len;
+ desc->mss = skb_shinfo(skb)->gso_size;
+}
+
+static void
+gve_tx_fill_general_ctx_desc(struct gve_tx_general_context_desc_dqo *desc,
+ const struct gve_tx_metadata_dqo *metadata)
+{
+ *desc = (struct gve_tx_general_context_desc_dqo){
+ .flex0 = metadata->bytes[0],
+ .flex1 = metadata->bytes[1],
+ .flex2 = metadata->bytes[2],
+ .flex3 = metadata->bytes[3],
+ .flex4 = metadata->bytes[4],
+ .flex5 = metadata->bytes[5],
+ .flex6 = metadata->bytes[6],
+ .flex7 = metadata->bytes[7],
+ .flex8 = metadata->bytes[8],
+ .flex9 = metadata->bytes[9],
+ .flex10 = metadata->bytes[10],
+ .flex11 = metadata->bytes[11],
+ .cmd_dtype = {.dtype = GVE_TX_GENERAL_CTX_DESC_DTYPE_DQO},
+ };
+}
+
+/* Returns 0 on success, or < 0 on error.
+ *
+ * Before this function is called, the caller must ensure
+ * gve_has_pending_packet(tx) returns true.
+ */
+static int gve_tx_add_skb_no_copy_dqo(struct gve_tx_ring *tx,
+ struct sk_buff *skb)
+{
+ const struct skb_shared_info *shinfo = skb_shinfo(skb);
+ const bool is_gso = skb_is_gso(skb);
+ u32 desc_idx = tx->dqo_tx.tail;
+
+ struct gve_tx_pending_packet_dqo *pending_packet;
+ struct gve_tx_metadata_dqo metadata;
+ s16 completion_tag;
+ int i;
+
+ pending_packet = gve_alloc_pending_packet(tx);
+ pending_packet->skb = skb;
+ pending_packet->num_bufs = 0;
+ completion_tag = pending_packet - tx->dqo.pending_packets;
+
+ gve_extract_tx_metadata_dqo(skb, &metadata);
+ if (is_gso) {
+ int header_len = gve_prep_tso(skb);
+
+ if (unlikely(header_len < 0))
+ goto err;
+
+ gve_tx_fill_tso_ctx_desc(&tx->dqo.tx_ring[desc_idx].tso_ctx,
+ skb, &metadata, header_len);
+ desc_idx = (desc_idx + 1) & tx->mask;
+ }
+
+ gve_tx_fill_general_ctx_desc(&tx->dqo.tx_ring[desc_idx].general_ctx,
+ &metadata);
+ desc_idx = (desc_idx + 1) & tx->mask;
+
+ /* Note: HW requires that the size of a non-TSO packet be within the
+ * range of [17, 9728].
+ *
+ * We don't double check because
+ * - We limited `netdev->min_mtu` to ETH_MIN_MTU.
+ * - Hypervisor won't allow MTU larger than 9216.
+ */
+
+ /* Map the linear portion of skb */
+ {
+ struct gve_tx_dma_buf *buf =
+ &pending_packet->bufs[pending_packet->num_bufs];
+ u32 len = skb_headlen(skb);
+ dma_addr_t addr;
+
+ addr = dma_map_single(tx->dev, skb->data, len, DMA_TO_DEVICE);
+ if (unlikely(dma_mapping_error(tx->dev, addr)))
+ goto err;
+
+ dma_unmap_len_set(buf, len, len);
+ dma_unmap_addr_set(buf, dma, addr);
+ ++pending_packet->num_bufs;
+
+ gve_tx_fill_pkt_desc_dqo(tx, &desc_idx, skb, len, addr,
+ completion_tag,
+ /*eop=*/shinfo->nr_frags == 0, is_gso);
+ }
+
+ for (i = 0; i < shinfo->nr_frags; i++) {
+ struct gve_tx_dma_buf *buf =
+ &pending_packet->bufs[pending_packet->num_bufs];
+ const skb_frag_t *frag = &shinfo->frags[i];
+ bool is_eop = i == (shinfo->nr_frags - 1);
+ u32 len = skb_frag_size(frag);
+ dma_addr_t addr;
+
+ addr = skb_frag_dma_map(tx->dev, frag, 0, len, DMA_TO_DEVICE);
+ if (unlikely(dma_mapping_error(tx->dev, addr)))
+ goto err;
+
+ dma_unmap_len_set(buf, len, len);
+ dma_unmap_addr_set(buf, dma, addr);
+ ++pending_packet->num_bufs;
+
+ gve_tx_fill_pkt_desc_dqo(tx, &desc_idx, skb, len, addr,
+ completion_tag, is_eop, is_gso);
+ }
+
+ /* Commit the changes to our state */
+ tx->dqo_tx.tail = desc_idx;
+
+ /* Request a descriptor completion on the last descriptor of the
+ * packet if we are allowed to by the HW enforced interval.
+ */
+ {
+ u32 last_desc_idx = (desc_idx - 1) & tx->mask;
+ u32 last_report_event_interval =
+ (last_desc_idx - tx->dqo_tx.last_re_idx) & tx->mask;
+
+ if (unlikely(last_report_event_interval >=
+ GVE_TX_MIN_RE_INTERVAL)) {
+ tx->dqo.tx_ring[last_desc_idx].pkt.report_event = true;
+ tx->dqo_tx.last_re_idx = last_desc_idx;
+ }
+ }
+
+ return 0;
+
+err:
+ for (i = 0; i < pending_packet->num_bufs; i++) {
+ struct gve_tx_dma_buf *buf = &pending_packet->bufs[i];
+
+ if (i == 0) {
+ dma_unmap_single(tx->dev, dma_unmap_addr(buf, dma),
+ dma_unmap_len(buf, len),
+ DMA_TO_DEVICE);
+ } else {
+ dma_unmap_page(tx->dev, dma_unmap_addr(buf, dma),
+ dma_unmap_len(buf, len), DMA_TO_DEVICE);
+ }
+ }
+
+ pending_packet->skb = NULL;
+ pending_packet->num_bufs = 0;
+ gve_free_pending_packet(tx, pending_packet);
+
+ return -1;
+}
+
+static int gve_num_descs_per_buf(size_t size)
+{
+ return DIV_ROUND_UP(size, GVE_TX_MAX_BUF_SIZE_DQO);
+}
+
+static int gve_num_buffer_descs_needed(const struct sk_buff *skb)
+{
+ const struct skb_shared_info *shinfo = skb_shinfo(skb);
+ int num_descs;
+ int i;
+
+ num_descs = gve_num_descs_per_buf(skb_headlen(skb));
+
+ for (i = 0; i < shinfo->nr_frags; i++) {
+ unsigned int frag_size = skb_frag_size(&shinfo->frags[i]);
+
+ num_descs += gve_num_descs_per_buf(frag_size);
+ }
+
+ return num_descs;
+}
+
+/* Returns true if HW is capable of sending TSO represented by `skb`.
+ *
+ * Each segment must not span more than GVE_TX_MAX_DATA_DESCS buffers.
+ * - The header is counted as one buffer for every single segment.
+ * - A buffer which is split between two segments is counted for both.
+ * - If a buffer contains both header and payload, it is counted as two buffers.
+ */
+static bool gve_can_send_tso(const struct sk_buff *skb)
+{
+ const int header_len = skb_checksum_start_offset(skb) + tcp_hdrlen(skb);
+ const int max_bufs_per_seg = GVE_TX_MAX_DATA_DESCS - 1;
+ const struct skb_shared_info *shinfo = skb_shinfo(skb);
+ const int gso_size = shinfo->gso_size;
+ int cur_seg_num_bufs;
+ int cur_seg_size;
+ int i;
+
+ cur_seg_size = skb_headlen(skb) - header_len;
+ cur_seg_num_bufs = cur_seg_size > 0;
+
+ for (i = 0; i < shinfo->nr_frags; i++) {
+ if (cur_seg_size >= gso_size) {
+ cur_seg_size %= gso_size;
+ cur_seg_num_bufs = cur_seg_size > 0;
+ }
+
+ if (unlikely(++cur_seg_num_bufs > max_bufs_per_seg))
+ return false;
+
+ cur_seg_size += skb_frag_size(&shinfo->frags[i]);
+ }
+
+ return true;
+}
+
+/* Attempt to transmit specified SKB.
+ *
+ * Returns 0 if the SKB was transmitted or dropped.
+ * Returns -1 if there is not currently enough space to transmit the SKB.
+ */
+static int gve_try_tx_skb(struct gve_priv *priv, struct gve_tx_ring *tx,
+ struct sk_buff *skb)
+{
+ int num_buffer_descs;
+ int total_num_descs;
+
+ if (skb_is_gso(skb)) {
+ /* If TSO doesn't meet HW requirements, attempt to linearize the
+ * packet.
+ */
+ if (unlikely(!gve_can_send_tso(skb) &&
+ skb_linearize(skb) < 0)) {
+ net_err_ratelimited("%s: Failed to transmit TSO packet\n",
+ priv->dev->name);
+ goto drop;
+ }
+
+ num_buffer_descs = gve_num_buffer_descs_needed(skb);
+ } else {
+ num_buffer_descs = gve_num_buffer_descs_needed(skb);
+
+ if (unlikely(num_buffer_descs > GVE_TX_MAX_DATA_DESCS)) {
+ if (unlikely(skb_linearize(skb) < 0))
+ goto drop;
+
+ num_buffer_descs = 1;
+ }
+ }
+
+ /* Metadata + (optional TSO) + data descriptors. */
+ total_num_descs = 1 + skb_is_gso(skb) + num_buffer_descs;
+ if (unlikely(gve_maybe_stop_tx_dqo(tx, total_num_descs +
+ GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP))) {
+ return -1;
+ }
+
+ if (unlikely(gve_tx_add_skb_no_copy_dqo(tx, skb) < 0))
+ goto drop;
+
+ netdev_tx_sent_queue(tx->netdev_txq, skb->len);
+ skb_tx_timestamp(skb);
+ return 0;
+
+drop:
+ tx->dropped_pkt++;
+ dev_kfree_skb_any(skb);
+ return 0;
+}
+
+/* Transmit a given skb and ring the doorbell. */
+netdev_tx_t gve_tx_dqo(struct sk_buff *skb, struct net_device *dev)
+{
+ struct gve_priv *priv = netdev_priv(dev);
+ struct gve_tx_ring *tx;
+
+ tx = &priv->tx[skb_get_queue_mapping(skb)];
+ if (unlikely(gve_try_tx_skb(priv, tx, skb) < 0)) {
+ /* We need to ring the txq doorbell -- we have stopped the Tx
+ * queue for want of resources, but prior calls to gve_tx()
+ * may have added descriptors without ringing the doorbell.
+ */
+ gve_tx_put_doorbell_dqo(priv, tx->q_resources, tx->dqo_tx.tail);
+ return NETDEV_TX_BUSY;
+ }
+
+ if (!netif_xmit_stopped(tx->netdev_txq) && netdev_xmit_more())
+ return NETDEV_TX_OK;
+
+ gve_tx_put_doorbell_dqo(priv, tx->q_resources, tx->dqo_tx.tail);
+ return NETDEV_TX_OK;
+}
+
+static void add_to_list(struct gve_tx_ring *tx, struct gve_index_list *list,
+ struct gve_tx_pending_packet_dqo *pending_packet)
+{
+ s16 old_tail, index;
+
+ index = pending_packet - tx->dqo.pending_packets;
+ old_tail = list->tail;
+ list->tail = index;
+ if (old_tail == -1)
+ list->head = index;
+ else
+ tx->dqo.pending_packets[old_tail].next = index;
+
+ pending_packet->next = -1;
+ pending_packet->prev = old_tail;
+}
+
+static void remove_from_list(struct gve_tx_ring *tx,
+ struct gve_index_list *list,
+ struct gve_tx_pending_packet_dqo *pending_packet)
+{
+ s16 prev_index, next_index;
+
+ prev_index = pending_packet->prev;
+ next_index = pending_packet->next;
+
+ if (prev_index == -1) {
+ /* Node is head */
+ list->head = next_index;
+ } else {
+ tx->dqo.pending_packets[prev_index].next = next_index;
+ }
+ if (next_index == -1) {
+ /* Node is tail */
+ list->tail = prev_index;
+ } else {
+ tx->dqo.pending_packets[next_index].prev = prev_index;
+ }
+}
+
+static void gve_unmap_packet(struct device *dev,
+ struct gve_tx_pending_packet_dqo *pending_packet)
+{
+ struct gve_tx_dma_buf *buf;
+ int i;
+
+ /* SKB linear portion is guaranteed to be mapped */
+ buf = &pending_packet->bufs[0];
+ dma_unmap_single(dev, dma_unmap_addr(buf, dma),
+ dma_unmap_len(buf, len), DMA_TO_DEVICE);
+ for (i = 1; i < pending_packet->num_bufs; i++) {
+ buf = &pending_packet->bufs[i];
+ dma_unmap_page(dev, dma_unmap_addr(buf, dma),
+ dma_unmap_len(buf, len), DMA_TO_DEVICE);
+ }
+ pending_packet->num_bufs = 0;
+}
+
+/* Completion types and expected behavior:
+ * No Miss compl + Packet compl = Packet completed normally.
+ * Miss compl + Re-inject compl = Packet completed normally.
+ * No Miss compl + Re-inject compl = Skipped i.e. packet not completed.
+ * Miss compl + Packet compl = Skipped i.e. packet not completed.
+ */
+static void gve_handle_packet_completion(struct gve_priv *priv,
+ struct gve_tx_ring *tx, bool is_napi,
+ u16 compl_tag, u64 *bytes, u64 *pkts,
+ bool is_reinjection)
+{
+ struct gve_tx_pending_packet_dqo *pending_packet;
+
+ if (unlikely(compl_tag >= tx->dqo.num_pending_packets)) {
+ net_err_ratelimited("%s: Invalid TX completion tag: %d\n",
+ priv->dev->name, (int)compl_tag);
+ return;
+ }
+
+ pending_packet = &tx->dqo.pending_packets[compl_tag];
+
+ if (unlikely(is_reinjection)) {
+ if (unlikely(pending_packet->state ==
+ GVE_PACKET_STATE_TIMED_OUT_COMPL)) {
+ net_err_ratelimited("%s: Re-injection completion: %d received after timeout.\n",
+ priv->dev->name, (int)compl_tag);
+ /* Packet was already completed as a result of timeout,
+ * so just remove from list and free pending packet.
+ */
+ remove_from_list(tx,
+ &tx->dqo_compl.timed_out_completions,
+ pending_packet);
+ gve_free_pending_packet(tx, pending_packet);
+ return;
+ }
+ if (unlikely(pending_packet->state !=
+ GVE_PACKET_STATE_PENDING_REINJECT_COMPL)) {
+ /* No outstanding miss completion but packet allocated
+ * implies packet receives a re-injection completion
+ * without a a prior miss completion. Return without
+ * completing the packet.
+ */
+ net_err_ratelimited("%s: Re-injection completion received without corresponding miss completion: %d\n",
+ priv->dev->name, (int)compl_tag);
+ return;
+ }
+ remove_from_list(tx, &tx->dqo_compl.miss_completions,
+ pending_packet);
+ } else {
+ /* Packet is allocated but not a pending data completion. */
+ if (unlikely(pending_packet->state !=
+ GVE_PACKET_STATE_PENDING_DATA_COMPL)) {
+ net_err_ratelimited("%s: No pending data completion: %d\n",
+ priv->dev->name, (int)compl_tag);
+ return;
+ }
+ }
+ gve_unmap_packet(tx->dev, pending_packet);
+
+ *bytes += pending_packet->skb->len;
+ (*pkts)++;
+ napi_consume_skb(pending_packet->skb, is_napi);
+ pending_packet->skb = NULL;
+ gve_free_pending_packet(tx, pending_packet);
+}
+
+static void gve_handle_miss_completion(struct gve_priv *priv,
+ struct gve_tx_ring *tx, u16 compl_tag,
+ u64 *bytes, u64 *pkts)
+{
+ struct gve_tx_pending_packet_dqo *pending_packet;
+
+ if (unlikely(compl_tag >= tx->dqo.num_pending_packets)) {
+ net_err_ratelimited("%s: Invalid TX completion tag: %d\n",
+ priv->dev->name, (int)compl_tag);
+ return;
+ }
+
+ pending_packet = &tx->dqo.pending_packets[compl_tag];
+ if (unlikely(pending_packet->state !=
+ GVE_PACKET_STATE_PENDING_DATA_COMPL)) {
+ net_err_ratelimited("%s: Unexpected packet state: %d for completion tag : %d\n",
+ priv->dev->name, (int)pending_packet->state,
+ (int)compl_tag);
+ return;
+ }
+
+ pending_packet->state = GVE_PACKET_STATE_PENDING_REINJECT_COMPL;
+ /* jiffies can wraparound but time comparisons can handle overflows. */
+ pending_packet->timeout_jiffies =
+ jiffies +
+ msecs_to_jiffies(GVE_REINJECT_COMPL_TIMEOUT *
+ MSEC_PER_SEC);
+ add_to_list(tx, &tx->dqo_compl.miss_completions, pending_packet);
+
+ *bytes += pending_packet->skb->len;
+ (*pkts)++;
+}
+
+static void remove_miss_completions(struct gve_priv *priv,
+ struct gve_tx_ring *tx)
+{
+ struct gve_tx_pending_packet_dqo *pending_packet;
+ s16 next_index;
+
+ next_index = tx->dqo_compl.miss_completions.head;
+ while (next_index != -1) {
+ pending_packet = &tx->dqo.pending_packets[next_index];
+ next_index = pending_packet->next;
+ /* Break early because packets should timeout in order. */
+ if (time_is_after_jiffies(pending_packet->timeout_jiffies))
+ break;
+
+ remove_from_list(tx, &tx->dqo_compl.miss_completions,
+ pending_packet);
+ /* Unmap buffers and free skb but do not unallocate packet i.e.
+ * the completion tag is not freed to ensure that the driver
+ * can take appropriate action if a corresponding valid
+ * completion is received later.
+ */
+ gve_unmap_packet(tx->dev, pending_packet);
+ /* This indicates the packet was dropped. */
+ dev_kfree_skb_any(pending_packet->skb);
+ pending_packet->skb = NULL;
+ tx->dropped_pkt++;
+ net_err_ratelimited("%s: No reinjection completion was received for: %d.\n",
+ priv->dev->name,
+ (int)(pending_packet - tx->dqo.pending_packets));
+
+ pending_packet->state = GVE_PACKET_STATE_TIMED_OUT_COMPL;
+ pending_packet->timeout_jiffies =
+ jiffies +
+ msecs_to_jiffies(GVE_DEALLOCATE_COMPL_TIMEOUT *
+ MSEC_PER_SEC);
+ /* Maintain pending packet in another list so the packet can be
+ * unallocated at a later time.
+ */
+ add_to_list(tx, &tx->dqo_compl.timed_out_completions,
+ pending_packet);
+ }
+}
+
+static void remove_timed_out_completions(struct gve_priv *priv,
+ struct gve_tx_ring *tx)
+{
+ struct gve_tx_pending_packet_dqo *pending_packet;
+ s16 next_index;
+
+ next_index = tx->dqo_compl.timed_out_completions.head;
+ while (next_index != -1) {
+ pending_packet = &tx->dqo.pending_packets[next_index];
+ next_index = pending_packet->next;
+ /* Break early because packets should timeout in order. */
+ if (time_is_after_jiffies(pending_packet->timeout_jiffies))
+ break;
+
+ remove_from_list(tx, &tx->dqo_compl.timed_out_completions,
+ pending_packet);
+ gve_free_pending_packet(tx, pending_packet);
+ }
+}
+
+int gve_clean_tx_done_dqo(struct gve_priv *priv, struct gve_tx_ring *tx,
+ struct napi_struct *napi)
+{
+ u64 reinject_compl_bytes = 0;
+ u64 reinject_compl_pkts = 0;
+ int num_descs_cleaned = 0;
+ u64 miss_compl_bytes = 0;
+ u64 miss_compl_pkts = 0;
+ u64 pkt_compl_bytes = 0;
+ u64 pkt_compl_pkts = 0;
+
+ /* Limit in order to avoid blocking for too long */
+ while (!napi || pkt_compl_pkts < napi->weight) {
+ struct gve_tx_compl_desc *compl_desc =
+ &tx->dqo.compl_ring[tx->dqo_compl.head];
+ u16 type;
+
+ if (compl_desc->generation == tx->dqo_compl.cur_gen_bit)
+ break;
+
+ /* Prefetch the next descriptor. */
+ prefetch(&tx->dqo.compl_ring[(tx->dqo_compl.head + 1) &
+ tx->dqo.complq_mask]);
+
+ /* Do not read data until we own the descriptor */
+ dma_rmb();
+ type = compl_desc->type;
+
+ if (type == GVE_COMPL_TYPE_DQO_DESC) {
+ /* This is the last descriptor fetched by HW plus one */
+ u16 tx_head = le16_to_cpu(compl_desc->tx_head);
+
+ atomic_set_release(&tx->dqo_compl.hw_tx_head, tx_head);
+ } else if (type == GVE_COMPL_TYPE_DQO_PKT) {
+ u16 compl_tag = le16_to_cpu(compl_desc->completion_tag);
+
+ gve_handle_packet_completion(priv, tx, !!napi,
+ compl_tag,
+ &pkt_compl_bytes,
+ &pkt_compl_pkts,
+ /*is_reinjection=*/false);
+ } else if (type == GVE_COMPL_TYPE_DQO_MISS) {
+ u16 compl_tag = le16_to_cpu(compl_desc->completion_tag);
+
+ gve_handle_miss_completion(priv, tx, compl_tag,
+ &miss_compl_bytes,
+ &miss_compl_pkts);
+ } else if (type == GVE_COMPL_TYPE_DQO_REINJECTION) {
+ u16 compl_tag = le16_to_cpu(compl_desc->completion_tag);
+
+ gve_handle_packet_completion(priv, tx, !!napi,
+ compl_tag,
+ &reinject_compl_bytes,
+ &reinject_compl_pkts,
+ /*is_reinjection=*/true);
+ }
+
+ tx->dqo_compl.head =
+ (tx->dqo_compl.head + 1) & tx->dqo.complq_mask;
+ /* Flip the generation bit when we wrap around */
+ tx->dqo_compl.cur_gen_bit ^= tx->dqo_compl.head == 0;
+ num_descs_cleaned++;
+ }
+
+ netdev_tx_completed_queue(tx->netdev_txq,
+ pkt_compl_pkts + miss_compl_pkts,
+ pkt_compl_bytes + miss_compl_bytes);
+
+ remove_miss_completions(priv, tx);
+ remove_timed_out_completions(priv, tx);
+
+ u64_stats_update_begin(&tx->statss);
+ tx->bytes_done += pkt_compl_bytes + reinject_compl_bytes;
+ tx->pkt_done += pkt_compl_pkts + reinject_compl_pkts;
+ u64_stats_update_end(&tx->statss);
+ return num_descs_cleaned;
+}
+
+bool gve_tx_poll_dqo(struct gve_notify_block *block, bool do_clean)
+{
+ struct gve_tx_compl_desc *compl_desc;
+ struct gve_tx_ring *tx = block->tx;
+ struct gve_priv *priv = block->priv;
+
+ if (do_clean) {
+ int num_descs_cleaned = gve_clean_tx_done_dqo(priv, tx,
+ &block->napi);
+
+ /* Sync with queue being stopped in `gve_maybe_stop_tx_dqo()` */
+ mb();
+
+ if (netif_tx_queue_stopped(tx->netdev_txq) &&
+ num_descs_cleaned > 0) {
+ tx->wake_queue++;
+ netif_tx_wake_queue(tx->netdev_txq);
+ }
+ }
+
+ /* Return true if we still have work. */
+ compl_desc = &tx->dqo.compl_ring[tx->dqo_compl.head];
+ return compl_desc->generation != tx->dqo_compl.cur_gen_bit;
+}
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR MIT)
+/* Google virtual Ethernet (gve) driver
+ *
+ * Copyright (C) 2015-2021 Google, Inc.
+ */
+
+#include "gve.h"
+#include "gve_adminq.h"
+#include "gve_utils.h"
+
+void gve_tx_remove_from_block(struct gve_priv *priv, int queue_idx)
+{
+ struct gve_notify_block *block =
+ &priv->ntfy_blocks[gve_tx_idx_to_ntfy(priv, queue_idx)];
+
+ block->tx = NULL;
+}
+
+void gve_tx_add_to_block(struct gve_priv *priv, int queue_idx)
+{
+ int ntfy_idx = gve_tx_idx_to_ntfy(priv, queue_idx);
+ struct gve_notify_block *block = &priv->ntfy_blocks[ntfy_idx];
+ struct gve_tx_ring *tx = &priv->tx[queue_idx];
+
+ block->tx = tx;
+ tx->ntfy_id = ntfy_idx;
+}
+
+void gve_rx_remove_from_block(struct gve_priv *priv, int queue_idx)
+{
+ struct gve_notify_block *block =
+ &priv->ntfy_blocks[gve_rx_idx_to_ntfy(priv, queue_idx)];
+
+ block->rx = NULL;
+}
+
+void gve_rx_add_to_block(struct gve_priv *priv, int queue_idx)
+{
+ u32 ntfy_idx = gve_rx_idx_to_ntfy(priv, queue_idx);
+ struct gve_notify_block *block = &priv->ntfy_blocks[ntfy_idx];
+ struct gve_rx_ring *rx = &priv->rx[queue_idx];
+
+ block->rx = rx;
+ rx->ntfy_id = ntfy_idx;
+}
+
+struct sk_buff *gve_rx_copy(struct net_device *dev, struct napi_struct *napi,
+ struct gve_rx_slot_page_info *page_info, u16 len,
+ u16 pad)
+{
+ struct sk_buff *skb = napi_alloc_skb(napi, len);
+ void *va = page_info->page_address + pad +
+ page_info->page_offset;
+
+ if (unlikely(!skb))
+ return NULL;
+
+ __skb_put(skb, len);
+
+ skb_copy_to_linear_data(skb, va, len);
+
+ skb->protocol = eth_type_trans(skb, dev);
+
+ return skb;
+}
+
+void gve_dec_pagecnt_bias(struct gve_rx_slot_page_info *page_info)
+{
+ page_info->pagecnt_bias--;
+ if (page_info->pagecnt_bias == 0) {
+ int pagecount = page_count(page_info->page);
+
+ /* If we have run out of bias - set it back up to INT_MAX
+ * minus the existing refs.
+ */
+ page_info->pagecnt_bias = INT_MAX - pagecount;
+
+ /* Set pagecount back up to max. */
+ page_ref_add(page_info->page, INT_MAX - pagecount);
+ }
+}
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0 OR MIT)
+ * Google virtual Ethernet (gve) driver
+ *
+ * Copyright (C) 2015-2021 Google, Inc.
+ */
+
+#ifndef _GVE_UTILS_H
+#define _GVE_UTILS_H
+
+#include <linux/etherdevice.h>
+
+#include "gve.h"
+
+void gve_tx_remove_from_block(struct gve_priv *priv, int queue_idx);
+void gve_tx_add_to_block(struct gve_priv *priv, int queue_idx);
+
+void gve_rx_remove_from_block(struct gve_priv *priv, int queue_idx);
+void gve_rx_add_to_block(struct gve_priv *priv, int queue_idx);
+
+struct sk_buff *gve_rx_copy(struct net_device *dev, struct napi_struct *napi,
+ struct gve_rx_slot_page_info *page_info, u16 len,
+ u16 pad);
+
+/* Decrement pagecnt_bias. Set it back to INT_MAX if it reached zero. */
+void gve_dec_pagecnt_bias(struct gve_rx_slot_page_info *page_info);
+
+#endif /* _GVE_UTILS_H */
+
tristate "Hisilicon HNS3 HCLGE Acceleration Engine & Compatibility Layer Support"
default m
depends on PCI_MSI
+ imply PTP_1588_CLOCK
help
This selects the HNS3_HCLGE network acceleration engine & its hardware
compatibility layer. The engine would be used in Hisilicon hip08 family of
default m
depends on 64BIT && PCI
depends on INET
+ select DIMLIB
help
This selects the Ethernet Driver for Hisilicon Network Subsystem 3 for hip08
family of SoCs. This module depends upon HNAE3 driver to access the HNAE3
default:
break;
}
-
- return;
}
static void hns_ae_get_ring_bdnum_limit(struct hnae_queue *queue,
}
/**
- *hns_mac_is_adjust_link - check is need change mac speed and duplex register
+ *hns_mac_need_adjust_link - check is need change mac speed and duplex register
*@mac_cb: mac device
*@speed: phy device speed
*@duplex:phy device duplex
}
/**
- * hns_mac_queue_config_bc_en - set broadcast rx&tx enable
+ * hns_mac_port_config_bc_en - set broadcast rx&tx enable
* @mac_cb: mac device
* @port_num: queue number
* @vlan_id: vlan id`
}
/**
- * hns_mac_set_autoneg - set rx & tx pause parameter
+ * hns_mac_set_pauseparam - set rx & tx pause parameter
* @mac_cb: mac control block
* @rx_en: rx enable or not
* @tx_en: tx enable or not
}
} else if (is_acpi_node(mac_cb->fw_port)) {
ret = hns_mac_register_phy(mac_cb);
- /*
- * Mac can work well if there is phy or not.If the port don't
+ /* Mac can work well if there is phy or not.If the port don't
* connect with phy, the return value will be ignored. Only
* when there is phy but can't find mdio bus, the return value
* will be handled.
}
/**
- * hns_ppe_qid_cfg - config ppe qid
+ * hns_dsaf_ppe_qid_cfg - config ppe qid
* @dsaf_dev: dsa fabric id
* @qid_cfg: value array
*/
}
/**
- * dsaf_tbl_tcam_mcast_cfg - tbl
+ * hns_dsaf_tbl_tcam_mcast_cfg - tbl
* @dsaf_dev: dsa fabric id
* @mcast: addr
*/
}
/**
- * hns_dsaf_tbl_tcam_init - INT
+ * hns_dsaf_comm_init - INT
* @dsaf_dev: dsa fabric id
*/
static void hns_dsaf_comm_init(struct dsaf_device *dsaf_dev)
}
/**
- * dsaf_pfc_unit_cnt - set pfc unit count
+ * hns_dsaf_pfc_unit_cnt - set pfc unit count
* @dsaf_dev: dsa fabric id
* @mac_id: id in use
* @rate: value array
}
/**
- * dsaf_port_work_rate_cfg - fifo
+ * hns_dsaf_port_work_rate_cfg - fifo
* @dsaf_dev: dsa fabric id
* @mac_id: mac contrl block
* @rate_mode: value array
}
/**
- *hns_dsaf_get_sset_count - get dsaf regs count
+ *hns_dsaf_get_regs_count - get dsaf regs count
*return dsaf regs count
*/
int hns_dsaf_get_regs_count(void)
}
/**
- * dsaf_probe - probo dsaf dev
+ * hns_dsaf_probe - probo dsaf dev
* @pdev: dasf platform device
* return 0 - success , negative --fail
*/
}
/**
- * dsaf_remove - remove dsaf dev
+ * hns_dsaf_remove - remove dsaf dev
* @pdev: dasf platform device
*/
static int hns_dsaf_remove(struct platform_device *pdev)
}
static void hns_dsaf_acpi_ledctrl_by_port(struct hns_mac_cb *mac_cb, u8 op_type,
- u32 link, u32 port, u32 act)
+ u32 link, u32 port, u32 act)
{
- union acpi_object *obj;
- union acpi_object obj_args[3], argv4;
+ union acpi_object *obj;
+ union acpi_object obj_args[3], argv4;
- obj_args[0].integer.type = ACPI_TYPE_INTEGER;
- obj_args[0].integer.value = link;
- obj_args[1].integer.type = ACPI_TYPE_INTEGER;
- obj_args[1].integer.value = port;
- obj_args[2].integer.type = ACPI_TYPE_INTEGER;
- obj_args[2].integer.value = act;
+ obj_args[0].integer.type = ACPI_TYPE_INTEGER;
+ obj_args[0].integer.value = link;
+ obj_args[1].integer.type = ACPI_TYPE_INTEGER;
+ obj_args[1].integer.value = port;
+ obj_args[2].integer.type = ACPI_TYPE_INTEGER;
+ obj_args[2].integer.value = act;
- argv4.type = ACPI_TYPE_PACKAGE;
- argv4.package.count = 3;
- argv4.package.elements = obj_args;
+ argv4.type = ACPI_TYPE_PACKAGE;
+ argv4.package.count = 3;
+ argv4.package.elements = obj_args;
- obj = acpi_evaluate_dsm(ACPI_HANDLE(mac_cb->dev),
- &hns_dsaf_acpi_dsm_guid, 0, op_type, &argv4);
- if (!obj) {
- dev_warn(mac_cb->dev, "ledctrl fail, link:%d port:%d act:%d!\n",
- link, port, act);
- return;
- }
+ obj = acpi_evaluate_dsm(ACPI_HANDLE(mac_cb->dev),
+ &hns_dsaf_acpi_dsm_guid, 0, op_type, &argv4);
+ if (!obj) {
+ dev_warn(mac_cb->dev, "ledctrl fail, link:%d port:%d act:%d!\n",
+ link, port, act);
+ return;
+ }
- ACPI_FREE(obj);
+ ACPI_FREE(obj);
}
static void hns_dsaf_acpi_locate_ledctrl_by_port(struct hns_mac_cb *mac_cb,
}
static void hns_cpld_set_led_acpi(struct hns_mac_cb *mac_cb, int link_status,
- u16 speed, int data)
+ u16 speed, int data)
{
- if (!mac_cb) {
- pr_err("cpld_led_set mac_cb is null!\n");
- return;
- }
+ if (!mac_cb) {
+ pr_err("cpld_led_set mac_cb is null!\n");
+ return;
+ }
- hns_dsaf_acpi_ledctrl_by_port(mac_cb, HNS_OP_LED_SET_FUNC,
- link_status, mac_cb->mac_id, data);
+ hns_dsaf_acpi_ledctrl_by_port(mac_cb, HNS_OP_LED_SET_FUNC,
+ link_status, mac_cb->mac_id, data);
}
static void cpld_led_reset(struct hns_mac_cb *mac_cb)
static void cpld_led_reset_acpi(struct hns_mac_cb *mac_cb)
{
- if (!mac_cb) {
- pr_err("cpld_led_reset mac_cb is null!\n");
- return;
- }
+ if (!mac_cb) {
+ pr_err("cpld_led_reset mac_cb is null!\n");
+ return;
+ }
- if (mac_cb->media_type != HNAE_MEDIA_TYPE_FIBER)
- return;
+ if (mac_cb->media_type != HNAE_MEDIA_TYPE_FIBER)
+ return;
- hns_dsaf_acpi_ledctrl_by_port(mac_cb, HNS_OP_LED_SET_FUNC,
- 0, mac_cb->mac_id, 0);
+ hns_dsaf_acpi_ledctrl_by_port(mac_cb, HNS_OP_LED_SET_FUNC,
+ 0, mac_cb->mac_id, 0);
}
static int cpld_set_led_id(struct hns_mac_cb *mac_cb,
}
/**
- * hns_dsaf_srst_chns - reset dsaf channels
+ * hns_dsaf_srst_chns_acpi - reset dsaf channels
* @dsaf_dev: dsaf device struct pointer
* @msk: xbar channels mask value:
* @dereset: false - request reset , true - drop reset
}
/**
- * hns_mac_get_sds_mode - get phy ifterface form serdes mode
+ * hns_mac_get_phy_if - get phy ifterface form serdes mode
* @mac_cb: mac control block
* retuen phy interface
*/
reg = HNS_MAC_HILINK4_REG;
else
reg = HNS_MAC_HILINK3_REG;
- } else{
+ } else {
if (!HNS_DSAF_IS_DEBUG(mac_cb->dsaf_dev) && mac_id <= 3)
reg = HNS_MAC_HILINK4V2_REG;
else
}
/**
- * ppe_init_hw - init ppe
+ * hns_ppe_init_hw - init ppe
* @ppe_cb: ppe device
*/
static void hns_ppe_init_hw(struct hns_ppe_cb *ppe_cb)
}
/**
- * ppe_uninit_hw - uninit ppe
+ * hns_ppe_uninit_hw - uninit ppe
* @ppe_cb: ppe device
*/
static void hns_ppe_uninit_hw(struct hns_ppe_cb *ppe_cb)
}
/**
- * hns_ppe_reset - reinit ppe/rcb hw
+ * hns_ppe_reset_common - reinit ppe/rcb hw
* @dsaf_dev: dasf device
* @ppe_common_index: the index
* return void
}
/**
- * ppe_get_strings - get ppe srting
+ * hns_ppe_get_strings - get ppe srting
* @ppe_cb: ppe device
* @stringset: string set type
* @data: output string
}
/**
- *rcb_get_sset_count - rcb ring regs count
+ *hns_rcb_get_ring_regs_count - rcb ring regs count
*return regs count
*/
int hns_rcb_get_ring_regs_count(void)
}
/**
- * hns_xgmac_tx_lf_rf_insert - insert lf rf control about xgmac
+ * hns_xgmac_lf_rf_insert - insert lf rf control about xgmac
* @mac_drv: mac driver
* @mode: inserf rf or lf
*/
}
/**
- * hns_xgmac__lf_rf_control_init - initial the lf rf control register
+ * hns_xgmac_lf_rf_control_init - initial the lf rf control register
* @mac_drv: mac driver
*/
static void hns_xgmac_lf_rf_control_init(struct mac_driver *mac_drv)
}
/**
- * hns_nic_adp_coalesce - self adapte coalesce according to rx rate
+ * hns_nic_adpt_coalesce - self adapte coalesce according to rx rate
* @ring_data: pointer to hns_nic_ring_data
**/
static void hns_nic_adpt_coalesce(struct hns_nic_ring_data *ring_data)
}
/**
- *hns_nic_set_link_settings - implement ethtool set link ksettings
+ *hns_nic_set_link_ksettings - implement ethtool set link ksettings
*@net_dev: net_device
*@cmd: ethtool_link_ksettings
*retuen 0 - success , negative --fail
}
/**
- * get_ethtool_stats - get detail statistics.
+ * hns_get_ethtool_stats - get detail statistics.
* @netdev: net device
* @stats: statistics info.
* @data: statistics data.
}
/**
- * get_strings: Return a set of strings that describe the requested objects
+ * hns_get_strings: Return a set of strings that describe the requested objects
* @netdev: net device
* @stringset: string set ID.
* @data: objects data.
HCLGE_MBX_API_NEGOTIATE, /* (VF -> PF) negotiate API version */
HCLGE_MBX_GET_QINFO, /* (VF -> PF) get queue config */
HCLGE_MBX_GET_QDEPTH, /* (VF -> PF) get queue depth */
- HCLGE_MBX_GET_TCINFO, /* (VF -> PF) get TC config */
+ HCLGE_MBX_GET_BASIC_INFO, /* (VF -> PF) get basic info */
HCLGE_MBX_GET_RETA, /* (VF -> PF) get RETA */
HCLGE_MBX_GET_RSS_KEY, /* (VF -> PF) get RSS key */
HCLGE_MBX_GET_MAC_ADDR, /* (VF -> PF) get MAC addr */
HCLGE_MBX_VLAN_RX_OFF_CFG, /* set rx side vlan offload */
HCLGE_MBX_PORT_BASE_VLAN_CFG, /* set port based vlan configuration */
HCLGE_MBX_GET_PORT_BASE_VLAN_STATE, /* get port based vlan state */
+ HCLGE_MBX_ENABLE_VLAN_FILTER,
};
enum hclge_mbx_tbl_cfg_subcode {
u8 int_gl_index;
};
+struct hclge_basic_info {
+ u8 hw_tc_map;
+ u8 rsv;
+ u16 mbx_api_version;
+ u32 pf_caps;
+};
+
struct hclgevf_mbx_resp_status {
struct mutex mbx_mutex; /* protects against contending sync cmd resp */
u32 origin_mbx_msg;
HNAE3_DEV_SUPPORT_STASH_B,
HNAE3_DEV_SUPPORT_UDP_TUNNEL_CSUM_B,
HNAE3_DEV_SUPPORT_PAUSE_B,
+ HNAE3_DEV_SUPPORT_RAS_IMP_B,
+ HNAE3_DEV_SUPPORT_RXD_ADV_LAYOUT_B,
+ HNAE3_DEV_SUPPORT_PORT_VLAN_BYPASS_B,
+ HNAE3_DEV_SUPPORT_VLAN_FLTR_MDF_B,
};
#define hnae3_dev_fd_supported(hdev) \
#define hnae3_dev_phy_imp_supported(hdev) \
test_bit(HNAE3_DEV_SUPPORT_PHY_IMP_B, (hdev)->ae_dev->caps)
+#define hnae3_dev_ras_imp_supported(hdev) \
+ test_bit(HNAE3_DEV_SUPPORT_RAS_IMP_B, (hdev)->ae_dev->caps)
+
#define hnae3_dev_tqp_txrx_indep_supported(hdev) \
test_bit(HNAE3_DEV_SUPPORT_TQP_TXRX_INDEP_B, (hdev)->ae_dev->caps)
#define hnae3_ae_dev_tqp_txrx_indep_supported(ae_dev) \
test_bit(HNAE3_DEV_SUPPORT_TQP_TXRX_INDEP_B, (ae_dev)->caps)
+#define hnae3_ae_dev_rxd_adv_layout_supported(ae_dev) \
+ test_bit(HNAE3_DEV_SUPPORT_RXD_ADV_LAYOUT_B, (ae_dev)->caps)
+
+enum HNAE3_PF_CAP_BITS {
+ HNAE3_PF_SUPPORT_VLAN_FLTR_MDF_B = 0,
+};
#define ring_ptr_move_fw(ring, p) \
((ring)->p = ((ring)->p + 1) % (ring)->desc_num)
#define ring_ptr_move_bw(ring, p) \
((ring)->p = ((ring)->p - 1 + (ring)->desc_num) % (ring)->desc_num)
-enum hns_desc_type {
- DESC_TYPE_UNKNOWN,
- DESC_TYPE_SKB,
- DESC_TYPE_FRAGLIST_SKB,
- DESC_TYPE_PAGE,
-};
-
struct hnae3_handle;
struct hnae3_queue {
HNAE3_FUNC_RESET,
HNAE3_GLOBAL_RESET,
HNAE3_IMP_RESET,
- HNAE3_UNKNOWN_RESET,
HNAE3_NONE_RESET,
HNAE3_MAX_RESET,
};
HNAE3_PORT_BASE_VLAN_NOCHANGE,
};
+enum hnae3_dbg_cmd {
+ HNAE3_DBG_CMD_TM_NODES,
+ HNAE3_DBG_CMD_TM_PRI,
+ HNAE3_DBG_CMD_TM_QSET,
+ HNAE3_DBG_CMD_TM_MAP,
+ HNAE3_DBG_CMD_TM_PG,
+ HNAE3_DBG_CMD_TM_PORT,
+ HNAE3_DBG_CMD_TC_SCH_INFO,
+ HNAE3_DBG_CMD_QOS_PAUSE_CFG,
+ HNAE3_DBG_CMD_QOS_PRI_MAP,
+ HNAE3_DBG_CMD_QOS_BUF_CFG,
+ HNAE3_DBG_CMD_DEV_INFO,
+ HNAE3_DBG_CMD_TX_BD,
+ HNAE3_DBG_CMD_RX_BD,
+ HNAE3_DBG_CMD_MAC_UC,
+ HNAE3_DBG_CMD_MAC_MC,
+ HNAE3_DBG_CMD_MNG_TBL,
+ HNAE3_DBG_CMD_LOOPBACK,
+ HNAE3_DBG_CMD_PTP_INFO,
+ HNAE3_DBG_CMD_INTERRUPT_INFO,
+ HNAE3_DBG_CMD_RESET_INFO,
+ HNAE3_DBG_CMD_IMP_INFO,
+ HNAE3_DBG_CMD_NCL_CONFIG,
+ HNAE3_DBG_CMD_REG_BIOS_COMMON,
+ HNAE3_DBG_CMD_REG_SSU,
+ HNAE3_DBG_CMD_REG_IGU_EGU,
+ HNAE3_DBG_CMD_REG_RPU,
+ HNAE3_DBG_CMD_REG_NCSI,
+ HNAE3_DBG_CMD_REG_RTC,
+ HNAE3_DBG_CMD_REG_PPP,
+ HNAE3_DBG_CMD_REG_RCB,
+ HNAE3_DBG_CMD_REG_TQP,
+ HNAE3_DBG_CMD_REG_MAC,
+ HNAE3_DBG_CMD_REG_DCB,
+ HNAE3_DBG_CMD_VLAN_CONFIG,
+ HNAE3_DBG_CMD_QUEUE_MAP,
+ HNAE3_DBG_CMD_RX_QUEUE_INFO,
+ HNAE3_DBG_CMD_TX_QUEUE_INFO,
+ HNAE3_DBG_CMD_FD_TCAM,
+ HNAE3_DBG_CMD_MAC_TNL_STATUS,
+ HNAE3_DBG_CMD_SERV_INFO,
+ HNAE3_DBG_CMD_UNKNOWN,
+};
+
struct hnae3_vector_info {
u8 __iomem *io_addr;
int vector;
* Check if any cls flower rule exist
* dbg_read_cmd
* Execute debugfs read command.
+ * set_tx_hwts_info
+ * Save information for 1588 tx packet
+ * get_rx_hwts
+ * Get 1588 rx hwstamp
+ * get_ts_info
+ * Get phc info
*/
struct hnae3_ae_ops {
int (*init_ae_dev)(struct hnae3_ae_dev *ae_dev);
void (*get_mdix_mode)(struct hnae3_handle *handle,
u8 *tp_mdix_ctrl, u8 *tp_mdix);
- void (*enable_vlan_filter)(struct hnae3_handle *handle, bool enable);
+ int (*enable_vlan_filter)(struct hnae3_handle *handle, bool enable);
int (*set_vlan_filter)(struct hnae3_handle *handle, __be16 proto,
u16 vlan_id, bool is_kill);
int (*set_vf_vlan_filter)(struct hnae3_handle *handle, int vfid,
void (*enable_fd)(struct hnae3_handle *handle, bool enable);
int (*add_arfs_entry)(struct hnae3_handle *handle, u16 queue_id,
u16 flow_id, struct flow_keys *fkeys);
- int (*dbg_run_cmd)(struct hnae3_handle *handle, const char *cmd_buf);
- int (*dbg_read_cmd)(struct hnae3_handle *handle, const char *cmd_buf,
+ int (*dbg_read_cmd)(struct hnae3_handle *handle, enum hnae3_dbg_cmd cmd,
char *buf, int len);
pci_ers_result_t (*handle_hw_ras_error)(struct hnae3_ae_dev *ae_dev);
bool (*get_hw_reset_stat)(struct hnae3_handle *handle);
struct ethtool_link_ksettings *cmd);
int (*set_phy_link_ksettings)(struct hnae3_handle *handle,
const struct ethtool_link_ksettings *cmd);
+ bool (*set_tx_hwts_info)(struct hnae3_handle *handle,
+ struct sk_buff *skb);
+ void (*get_rx_hwts)(struct hnae3_handle *handle, struct sk_buff *skb,
+ u32 nsec, u32 sec);
+ int (*get_ts_info)(struct hnae3_handle *handle,
+ struct ethtool_ts_info *info);
};
struct hnae3_dcb_ops {
u16 rx_buf_len;
u16 num_tx_desc;
u16 num_rx_desc;
+ u32 tx_spare_buf_size;
struct hnae3_tc_info tc_info;
#define HNAE3_BPE BIT(2) /* broadcast promisc enable */
#define HNAE3_OVERFLOW_UPE BIT(3) /* unicast mac vlan overflow */
#define HNAE3_OVERFLOW_MPE BIT(4) /* multicast mac vlan overflow */
-#define HNAE3_VLAN_FLTR BIT(5) /* enable vlan filter */
#define HNAE3_UPE (HNAE3_USER_UPE | HNAE3_OVERFLOW_UPE)
#define HNAE3_MPE (HNAE3_USER_MPE | HNAE3_OVERFLOW_MPE)
#define hnae3_get_bit(origin, shift) \
hnae3_get_field(origin, 0x1 << (shift), shift)
-#define HNAE3_DBG_TM_NODES "tm_nodes"
-#define HNAE3_DBG_TM_PRI "tm_priority"
-#define HNAE3_DBG_TM_QSET "tm_qset"
-
int hnae3_register_ae_dev(struct hnae3_ae_dev *ae_dev);
void hnae3_unregister_ae_dev(struct hnae3_ae_dev *ae_dev);
#include <linux/device.h>
#include "hnae3.h"
+#include "hns3_debugfs.h"
#include "hns3_enet.h"
-#define HNS3_DBG_READ_LEN 65536
-#define HNS3_DBG_WRITE_LEN 1024
-
static struct dentry *hns3_dbgfs_root;
-static int hns3_dbg_queue_info(struct hnae3_handle *h,
- const char *cmd_buf)
+static struct hns3_dbg_dentry_info hns3_dbg_dentry[] = {
+ {
+ .name = "tm"
+ },
+ {
+ .name = "tx_bd_info"
+ },
+ {
+ .name = "rx_bd_info"
+ },
+ {
+ .name = "mac_list"
+ },
+ {
+ .name = "reg"
+ },
+ {
+ .name = "queue"
+ },
+ {
+ .name = "fd"
+ },
+ /* keep common at the bottom and add new directory above */
+ {
+ .name = "common"
+ },
+};
+
+static int hns3_dbg_bd_file_init(struct hnae3_handle *handle, unsigned int cmd);
+static int hns3_dbg_common_file_init(struct hnae3_handle *handle,
+ unsigned int cmd);
+
+static struct hns3_dbg_cmd_info hns3_dbg_cmd[] = {
+ {
+ .name = "tm_nodes",
+ .cmd = HNAE3_DBG_CMD_TM_NODES,
+ .dentry = HNS3_DBG_DENTRY_TM,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "tm_priority",
+ .cmd = HNAE3_DBG_CMD_TM_PRI,
+ .dentry = HNS3_DBG_DENTRY_TM,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "tm_qset",
+ .cmd = HNAE3_DBG_CMD_TM_QSET,
+ .dentry = HNS3_DBG_DENTRY_TM,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "tm_map",
+ .cmd = HNAE3_DBG_CMD_TM_MAP,
+ .dentry = HNS3_DBG_DENTRY_TM,
+ .buf_len = HNS3_DBG_READ_LEN_1MB,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "tm_pg",
+ .cmd = HNAE3_DBG_CMD_TM_PG,
+ .dentry = HNS3_DBG_DENTRY_TM,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "tm_port",
+ .cmd = HNAE3_DBG_CMD_TM_PORT,
+ .dentry = HNS3_DBG_DENTRY_TM,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "tc_sch_info",
+ .cmd = HNAE3_DBG_CMD_TC_SCH_INFO,
+ .dentry = HNS3_DBG_DENTRY_TM,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "qos_pause_cfg",
+ .cmd = HNAE3_DBG_CMD_QOS_PAUSE_CFG,
+ .dentry = HNS3_DBG_DENTRY_TM,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "qos_pri_map",
+ .cmd = HNAE3_DBG_CMD_QOS_PRI_MAP,
+ .dentry = HNS3_DBG_DENTRY_TM,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "qos_buf_cfg",
+ .cmd = HNAE3_DBG_CMD_QOS_BUF_CFG,
+ .dentry = HNS3_DBG_DENTRY_TM,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "dev_info",
+ .cmd = HNAE3_DBG_CMD_DEV_INFO,
+ .dentry = HNS3_DBG_DENTRY_COMMON,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "tx_bd_queue",
+ .cmd = HNAE3_DBG_CMD_TX_BD,
+ .dentry = HNS3_DBG_DENTRY_TX_BD,
+ .buf_len = HNS3_DBG_READ_LEN_4MB,
+ .init = hns3_dbg_bd_file_init,
+ },
+ {
+ .name = "rx_bd_queue",
+ .cmd = HNAE3_DBG_CMD_RX_BD,
+ .dentry = HNS3_DBG_DENTRY_RX_BD,
+ .buf_len = HNS3_DBG_READ_LEN_4MB,
+ .init = hns3_dbg_bd_file_init,
+ },
+ {
+ .name = "uc",
+ .cmd = HNAE3_DBG_CMD_MAC_UC,
+ .dentry = HNS3_DBG_DENTRY_MAC,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "mc",
+ .cmd = HNAE3_DBG_CMD_MAC_MC,
+ .dentry = HNS3_DBG_DENTRY_MAC,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "mng_tbl",
+ .cmd = HNAE3_DBG_CMD_MNG_TBL,
+ .dentry = HNS3_DBG_DENTRY_COMMON,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "loopback",
+ .cmd = HNAE3_DBG_CMD_LOOPBACK,
+ .dentry = HNS3_DBG_DENTRY_COMMON,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "interrupt_info",
+ .cmd = HNAE3_DBG_CMD_INTERRUPT_INFO,
+ .dentry = HNS3_DBG_DENTRY_COMMON,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "reset_info",
+ .cmd = HNAE3_DBG_CMD_RESET_INFO,
+ .dentry = HNS3_DBG_DENTRY_COMMON,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "imp_info",
+ .cmd = HNAE3_DBG_CMD_IMP_INFO,
+ .dentry = HNS3_DBG_DENTRY_COMMON,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "ncl_config",
+ .cmd = HNAE3_DBG_CMD_NCL_CONFIG,
+ .dentry = HNS3_DBG_DENTRY_COMMON,
+ .buf_len = HNS3_DBG_READ_LEN_128KB,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "mac_tnl_status",
+ .cmd = HNAE3_DBG_CMD_MAC_TNL_STATUS,
+ .dentry = HNS3_DBG_DENTRY_COMMON,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "bios_common",
+ .cmd = HNAE3_DBG_CMD_REG_BIOS_COMMON,
+ .dentry = HNS3_DBG_DENTRY_REG,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "ssu",
+ .cmd = HNAE3_DBG_CMD_REG_SSU,
+ .dentry = HNS3_DBG_DENTRY_REG,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "igu_egu",
+ .cmd = HNAE3_DBG_CMD_REG_IGU_EGU,
+ .dentry = HNS3_DBG_DENTRY_REG,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "rpu",
+ .cmd = HNAE3_DBG_CMD_REG_RPU,
+ .dentry = HNS3_DBG_DENTRY_REG,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "ncsi",
+ .cmd = HNAE3_DBG_CMD_REG_NCSI,
+ .dentry = HNS3_DBG_DENTRY_REG,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "rtc",
+ .cmd = HNAE3_DBG_CMD_REG_RTC,
+ .dentry = HNS3_DBG_DENTRY_REG,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "ppp",
+ .cmd = HNAE3_DBG_CMD_REG_PPP,
+ .dentry = HNS3_DBG_DENTRY_REG,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "rcb",
+ .cmd = HNAE3_DBG_CMD_REG_RCB,
+ .dentry = HNS3_DBG_DENTRY_REG,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "tqp",
+ .cmd = HNAE3_DBG_CMD_REG_TQP,
+ .dentry = HNS3_DBG_DENTRY_REG,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "mac",
+ .cmd = HNAE3_DBG_CMD_REG_MAC,
+ .dentry = HNS3_DBG_DENTRY_REG,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "dcb",
+ .cmd = HNAE3_DBG_CMD_REG_DCB,
+ .dentry = HNS3_DBG_DENTRY_REG,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "queue_map",
+ .cmd = HNAE3_DBG_CMD_QUEUE_MAP,
+ .dentry = HNS3_DBG_DENTRY_QUEUE,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "rx_queue_info",
+ .cmd = HNAE3_DBG_CMD_RX_QUEUE_INFO,
+ .dentry = HNS3_DBG_DENTRY_QUEUE,
+ .buf_len = HNS3_DBG_READ_LEN_1MB,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "tx_queue_info",
+ .cmd = HNAE3_DBG_CMD_TX_QUEUE_INFO,
+ .dentry = HNS3_DBG_DENTRY_QUEUE,
+ .buf_len = HNS3_DBG_READ_LEN_1MB,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "fd_tcam",
+ .cmd = HNAE3_DBG_CMD_FD_TCAM,
+ .dentry = HNS3_DBG_DENTRY_FD,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "service_task_info",
+ .cmd = HNAE3_DBG_CMD_SERV_INFO,
+ .dentry = HNS3_DBG_DENTRY_COMMON,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "vlan_config",
+ .cmd = HNAE3_DBG_CMD_VLAN_CONFIG,
+ .dentry = HNS3_DBG_DENTRY_COMMON,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+ {
+ .name = "ptp_info",
+ .cmd = HNAE3_DBG_CMD_PTP_INFO,
+ .dentry = HNS3_DBG_DENTRY_COMMON,
+ .buf_len = HNS3_DBG_READ_LEN,
+ .init = hns3_dbg_common_file_init,
+ },
+};
+
+static struct hns3_dbg_cap_info hns3_dbg_cap[] = {
+ {
+ .name = "support FD",
+ .cap_bit = HNAE3_DEV_SUPPORT_FD_B,
+ }, {
+ .name = "support GRO",
+ .cap_bit = HNAE3_DEV_SUPPORT_GRO_B,
+ }, {
+ .name = "support FEC",
+ .cap_bit = HNAE3_DEV_SUPPORT_FEC_B,
+ }, {
+ .name = "support UDP GSO",
+ .cap_bit = HNAE3_DEV_SUPPORT_UDP_GSO_B,
+ }, {
+ .name = "support PTP",
+ .cap_bit = HNAE3_DEV_SUPPORT_PTP_B,
+ }, {
+ .name = "support INT QL",
+ .cap_bit = HNAE3_DEV_SUPPORT_INT_QL_B,
+ }, {
+ .name = "support HW TX csum",
+ .cap_bit = HNAE3_DEV_SUPPORT_HW_TX_CSUM_B,
+ }, {
+ .name = "support UDP tunnel csum",
+ .cap_bit = HNAE3_DEV_SUPPORT_UDP_TUNNEL_CSUM_B,
+ }, {
+ .name = "support TX push",
+ .cap_bit = HNAE3_DEV_SUPPORT_TX_PUSH_B,
+ }, {
+ .name = "support imp-controlled PHY",
+ .cap_bit = HNAE3_DEV_SUPPORT_PHY_IMP_B,
+ }, {
+ .name = "support imp-controlled RAS",
+ .cap_bit = HNAE3_DEV_SUPPORT_RAS_IMP_B,
+ }, {
+ .name = "support rxd advanced layout",
+ .cap_bit = HNAE3_DEV_SUPPORT_RXD_ADV_LAYOUT_B,
+ }, {
+ .name = "support port vlan bypass",
+ .cap_bit = HNAE3_DEV_SUPPORT_PORT_VLAN_BYPASS_B,
+ }, {
+ .name = "support modify vlan filter state",
+ .cap_bit = HNAE3_DEV_SUPPORT_VLAN_FLTR_MDF_B,
+ }
+};
+
+static void hns3_dbg_fill_content(char *content, u16 len,
+ const struct hns3_dbg_item *items,
+ const char **result, u16 size)
+{
+ char *pos = content;
+ u16 i;
+
+ memset(content, ' ', len);
+ for (i = 0; i < size; i++) {
+ if (result)
+ strncpy(pos, result[i], strlen(result[i]));
+ else
+ strncpy(pos, items[i].name, strlen(items[i].name));
+
+ pos += strlen(items[i].name) + items[i].interval;
+ }
+
+ *pos++ = '\n';
+ *pos++ = '\0';
+}
+
+static const struct hns3_dbg_item tx_spare_info_items[] = {
+ { "QUEUE_ID", 2 },
+ { "COPYBREAK", 2 },
+ { "LEN", 7 },
+ { "NTU", 4 },
+ { "NTC", 4 },
+ { "LTC", 4 },
+ { "DMA", 17 },
+};
+
+static void hns3_dbg_tx_spare_info(struct hns3_enet_ring *ring, char *buf,
+ int len, u32 ring_num, int *pos)
+{
+ char data_str[ARRAY_SIZE(tx_spare_info_items)][HNS3_DBG_DATA_STR_LEN];
+ struct hns3_tx_spare *tx_spare = ring->tx_spare;
+ char *result[ARRAY_SIZE(tx_spare_info_items)];
+ char content[HNS3_DBG_INFO_LEN];
+ u32 i, j;
+
+ if (!tx_spare) {
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "tx spare buffer is not enabled\n");
+ return;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(tx_spare_info_items); i++)
+ result[i] = &data_str[i][0];
+
+ *pos += scnprintf(buf + *pos, len - *pos, "tx spare buffer info\n");
+ hns3_dbg_fill_content(content, sizeof(content), tx_spare_info_items,
+ NULL, ARRAY_SIZE(tx_spare_info_items));
+ *pos += scnprintf(buf + *pos, len - *pos, "%s", content);
+
+ for (i = 0; i < ring_num; i++) {
+ j = 0;
+ sprintf(result[j++], "%8u", i);
+ sprintf(result[j++], "%9u", ring->tx_copybreak);
+ sprintf(result[j++], "%3u", tx_spare->len);
+ sprintf(result[j++], "%3u", tx_spare->next_to_use);
+ sprintf(result[j++], "%3u", tx_spare->next_to_clean);
+ sprintf(result[j++], "%3u", tx_spare->last_to_clean);
+ sprintf(result[j++], "%pad", &tx_spare->dma);
+ hns3_dbg_fill_content(content, sizeof(content),
+ tx_spare_info_items,
+ (const char **)result,
+ ARRAY_SIZE(tx_spare_info_items));
+ *pos += scnprintf(buf + *pos, len - *pos, "%s", content);
+ }
+}
+
+static const struct hns3_dbg_item rx_queue_info_items[] = {
+ { "QUEUE_ID", 2 },
+ { "BD_NUM", 2 },
+ { "BD_LEN", 2 },
+ { "TAIL", 2 },
+ { "HEAD", 2 },
+ { "FBDNUM", 2 },
+ { "PKTNUM", 2 },
+ { "COPYBREAK", 2 },
+ { "RING_EN", 2 },
+ { "RX_RING_EN", 2 },
+ { "BASE_ADDR", 10 },
+};
+
+static void hns3_dump_rx_queue_info(struct hns3_enet_ring *ring,
+ struct hnae3_ae_dev *ae_dev, char **result,
+ u32 index)
{
+ u32 base_add_l, base_add_h;
+ u32 j = 0;
+
+ sprintf(result[j++], "%8u", index);
+
+ sprintf(result[j++], "%6u", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_RX_RING_BD_NUM_REG));
+
+ sprintf(result[j++], "%6u", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_RX_RING_BD_LEN_REG));
+
+ sprintf(result[j++], "%4u", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_RX_RING_TAIL_REG));
+
+ sprintf(result[j++], "%4u", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_RX_RING_HEAD_REG));
+
+ sprintf(result[j++], "%6u", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_RX_RING_FBDNUM_REG));
+
+ sprintf(result[j++], "%6u", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_RX_RING_PKTNUM_RECORD_REG));
+ sprintf(result[j++], "%9u", ring->rx_copybreak);
+
+ sprintf(result[j++], "%7s", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_EN_REG) ? "on" : "off");
+
+ if (hnae3_ae_dev_tqp_txrx_indep_supported(ae_dev))
+ sprintf(result[j++], "%10s", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_RX_EN_REG) ? "on" : "off");
+ else
+ sprintf(result[j++], "%10s", "NA");
+
+ base_add_h = readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_RX_RING_BASEADDR_H_REG);
+ base_add_l = readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_RX_RING_BASEADDR_L_REG);
+ sprintf(result[j++], "0x%08x%08x", base_add_h, base_add_l);
+}
+
+static int hns3_dbg_rx_queue_info(struct hnae3_handle *h,
+ char *buf, int len)
+{
+ char data_str[ARRAY_SIZE(rx_queue_info_items)][HNS3_DBG_DATA_STR_LEN];
struct hnae3_ae_dev *ae_dev = pci_get_drvdata(h->pdev);
+ char *result[ARRAY_SIZE(rx_queue_info_items)];
struct hns3_nic_priv *priv = h->priv;
+ char content[HNS3_DBG_INFO_LEN];
struct hns3_enet_ring *ring;
- u32 base_add_l, base_add_h;
- u32 queue_num, queue_max;
- u32 value, i;
- int cnt;
+ int pos = 0;
+ u32 i;
if (!priv->ring) {
dev_err(&h->pdev->dev, "priv->ring is NULL\n");
return -EFAULT;
}
- queue_max = h->kinfo.num_tqps;
- cnt = kstrtouint(&cmd_buf[11], 0, &queue_num);
- if (cnt)
- queue_num = 0;
- else
- queue_max = queue_num + 1;
+ for (i = 0; i < ARRAY_SIZE(rx_queue_info_items); i++)
+ result[i] = &data_str[i][0];
- dev_info(&h->pdev->dev, "queue info\n");
-
- if (queue_num >= h->kinfo.num_tqps) {
- dev_err(&h->pdev->dev,
- "Queue number(%u) is out of range(0-%u)\n", queue_num,
- h->kinfo.num_tqps - 1);
- return -EINVAL;
- }
-
- for (i = queue_num; i < queue_max; i++) {
+ hns3_dbg_fill_content(content, sizeof(content), rx_queue_info_items,
+ NULL, ARRAY_SIZE(rx_queue_info_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
+ for (i = 0; i < h->kinfo.num_tqps; i++) {
/* Each cycle needs to determine whether the instance is reset,
* to prevent reference to invalid memory. And need to ensure
* that the following code is executed within 100ms.
return -EPERM;
ring = &priv->ring[(u32)(i + h->kinfo.num_tqps)];
- base_add_h = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_RX_RING_BASEADDR_H_REG);
- base_add_l = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_RX_RING_BASEADDR_L_REG);
- dev_info(&h->pdev->dev, "RX(%u) BASE ADD: 0x%08x%08x\n", i,
- base_add_h, base_add_l);
+ hns3_dump_rx_queue_info(ring, ae_dev, result, i);
+ hns3_dbg_fill_content(content, sizeof(content),
+ rx_queue_info_items,
+ (const char **)result,
+ ARRAY_SIZE(rx_queue_info_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
+ }
+
+ return 0;
+}
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_RX_RING_BD_NUM_REG);
- dev_info(&h->pdev->dev, "RX(%u) RING BD NUM: %u\n", i, value);
+static const struct hns3_dbg_item tx_queue_info_items[] = {
+ { "QUEUE_ID", 2 },
+ { "BD_NUM", 2 },
+ { "TC", 2 },
+ { "TAIL", 2 },
+ { "HEAD", 2 },
+ { "FBDNUM", 2 },
+ { "OFFSET", 2 },
+ { "PKTNUM", 2 },
+ { "RING_EN", 2 },
+ { "TX_RING_EN", 2 },
+ { "BASE_ADDR", 10 },
+};
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_RX_RING_BD_LEN_REG);
- dev_info(&h->pdev->dev, "RX(%u) RING BD LEN: %u\n", i, value);
+static void hns3_dump_tx_queue_info(struct hns3_enet_ring *ring,
+ struct hnae3_ae_dev *ae_dev, char **result,
+ u32 index)
+{
+ u32 base_add_l, base_add_h;
+ u32 j = 0;
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_RX_RING_TAIL_REG);
- dev_info(&h->pdev->dev, "RX(%u) RING TAIL: %u\n", i, value);
+ sprintf(result[j++], "%8u", index);
+ sprintf(result[j++], "%6u", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_TX_RING_BD_NUM_REG));
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_RX_RING_HEAD_REG);
- dev_info(&h->pdev->dev, "RX(%u) RING HEAD: %u\n", i, value);
+ sprintf(result[j++], "%2u", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_TX_RING_TC_REG));
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_RX_RING_FBDNUM_REG);
- dev_info(&h->pdev->dev, "RX(%u) RING FBDNUM: %u\n", i, value);
+ sprintf(result[j++], "%4u", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_TX_RING_TAIL_REG));
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_RX_RING_PKTNUM_RECORD_REG);
- dev_info(&h->pdev->dev, "RX(%u) RING PKTNUM: %u\n", i, value);
+ sprintf(result[j++], "%4u", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_TX_RING_HEAD_REG));
- ring = &priv->ring[i];
- base_add_h = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_TX_RING_BASEADDR_H_REG);
- base_add_l = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_TX_RING_BASEADDR_L_REG);
- dev_info(&h->pdev->dev, "TX(%u) BASE ADD: 0x%08x%08x\n", i,
- base_add_h, base_add_l);
-
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_TX_RING_BD_NUM_REG);
- dev_info(&h->pdev->dev, "TX(%u) RING BD NUM: %u\n", i, value);
-
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_TX_RING_TC_REG);
- dev_info(&h->pdev->dev, "TX(%u) RING TC: %u\n", i, value);
-
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_TX_RING_TAIL_REG);
- dev_info(&h->pdev->dev, "TX(%u) RING TAIL: %u\n", i, value);
-
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_TX_RING_HEAD_REG);
- dev_info(&h->pdev->dev, "TX(%u) RING HEAD: %u\n", i, value);
-
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_TX_RING_FBDNUM_REG);
- dev_info(&h->pdev->dev, "TX(%u) RING FBDNUM: %u\n", i, value);
-
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_TX_RING_OFFSET_REG);
- dev_info(&h->pdev->dev, "TX(%u) RING OFFSET: %u\n", i, value);
-
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_TX_RING_PKTNUM_RECORD_REG);
- dev_info(&h->pdev->dev, "TX(%u) RING PKTNUM: %u\n", i, value);
-
- value = readl_relaxed(ring->tqp->io_base + HNS3_RING_EN_REG);
- dev_info(&h->pdev->dev, "TX/RX(%u) RING EN: %s\n", i,
- value ? "enable" : "disable");
-
- if (hnae3_ae_dev_tqp_txrx_indep_supported(ae_dev)) {
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_TX_EN_REG);
- dev_info(&h->pdev->dev, "TX(%u) RING EN: %s\n", i,
- value ? "enable" : "disable");
-
- value = readl_relaxed(ring->tqp->io_base +
- HNS3_RING_RX_EN_REG);
- dev_info(&h->pdev->dev, "RX(%u) RING EN: %s\n", i,
- value ? "enable" : "disable");
- }
+ sprintf(result[j++], "%6u", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_TX_RING_FBDNUM_REG));
+
+ sprintf(result[j++], "%6u", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_TX_RING_OFFSET_REG));
+
+ sprintf(result[j++], "%6u", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_TX_RING_PKTNUM_RECORD_REG));
+
+ sprintf(result[j++], "%7s", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_EN_REG) ? "on" : "off");
+
+ if (hnae3_ae_dev_tqp_txrx_indep_supported(ae_dev))
+ sprintf(result[j++], "%10s", readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_TX_EN_REG) ? "on" : "off");
+ else
+ sprintf(result[j++], "%10s", "NA");
+
+ base_add_h = readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_TX_RING_BASEADDR_H_REG);
+ base_add_l = readl_relaxed(ring->tqp->io_base +
+ HNS3_RING_TX_RING_BASEADDR_L_REG);
+ sprintf(result[j++], "0x%08x%08x", base_add_h, base_add_l);
+}
+
+static int hns3_dbg_tx_queue_info(struct hnae3_handle *h,
+ char *buf, int len)
+{
+ char data_str[ARRAY_SIZE(tx_queue_info_items)][HNS3_DBG_DATA_STR_LEN];
+ struct hnae3_ae_dev *ae_dev = pci_get_drvdata(h->pdev);
+ char *result[ARRAY_SIZE(tx_queue_info_items)];
+ struct hns3_nic_priv *priv = h->priv;
+ char content[HNS3_DBG_INFO_LEN];
+ struct hns3_enet_ring *ring;
+ int pos = 0;
+ u32 i;
+
+ if (!priv->ring) {
+ dev_err(&h->pdev->dev, "priv->ring is NULL\n");
+ return -EFAULT;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(tx_queue_info_items); i++)
+ result[i] = &data_str[i][0];
+
+ hns3_dbg_fill_content(content, sizeof(content), tx_queue_info_items,
+ NULL, ARRAY_SIZE(tx_queue_info_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
+
+ for (i = 0; i < h->kinfo.num_tqps; i++) {
+ /* Each cycle needs to determine whether the instance is reset,
+ * to prevent reference to invalid memory. And need to ensure
+ * that the following code is executed within 100ms.
+ */
+ if (!test_bit(HNS3_NIC_STATE_INITED, &priv->state) ||
+ test_bit(HNS3_NIC_STATE_RESETTING, &priv->state))
+ return -EPERM;
- dev_info(&h->pdev->dev, "\n");
+ ring = &priv->ring[i];
+ hns3_dump_tx_queue_info(ring, ae_dev, result, i);
+ hns3_dbg_fill_content(content, sizeof(content),
+ tx_queue_info_items,
+ (const char **)result,
+ ARRAY_SIZE(tx_queue_info_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
}
+ hns3_dbg_tx_spare_info(ring, buf, len, h->kinfo.num_tqps, &pos);
+
return 0;
}
-static int hns3_dbg_queue_map(struct hnae3_handle *h)
+static const struct hns3_dbg_item queue_map_items[] = {
+ { "local_queue_id", 2 },
+ { "global_queue_id", 2 },
+ { "vector_id", 2 },
+};
+
+static int hns3_dbg_queue_map(struct hnae3_handle *h, char *buf, int len)
{
+ char data_str[ARRAY_SIZE(queue_map_items)][HNS3_DBG_DATA_STR_LEN];
+ char *result[ARRAY_SIZE(queue_map_items)];
struct hns3_nic_priv *priv = h->priv;
- int i;
+ char content[HNS3_DBG_INFO_LEN];
+ int pos = 0;
+ int j;
+ u32 i;
if (!h->ae_algo->ops->get_global_queue_id)
return -EOPNOTSUPP;
- dev_info(&h->pdev->dev, "map info for queue id and vector id\n");
- dev_info(&h->pdev->dev,
- "local queue id | global queue id | vector id\n");
- for (i = 0; i < h->kinfo.num_tqps; i++) {
- u16 global_qid;
+ for (i = 0; i < ARRAY_SIZE(queue_map_items); i++)
+ result[i] = &data_str[i][0];
- global_qid = h->ae_algo->ops->get_global_queue_id(h, i);
+ hns3_dbg_fill_content(content, sizeof(content), queue_map_items,
+ NULL, ARRAY_SIZE(queue_map_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
+ for (i = 0; i < h->kinfo.num_tqps; i++) {
if (!priv->ring || !priv->ring[i].tqp_vector)
continue;
-
- dev_info(&h->pdev->dev,
- " %4d %4u %4d\n",
- i, global_qid, priv->ring[i].tqp_vector->vector_irq);
+ j = 0;
+ sprintf(result[j++], "%u", i);
+ sprintf(result[j++], "%u",
+ h->ae_algo->ops->get_global_queue_id(h, i));
+ sprintf(result[j++], "%u",
+ priv->ring[i].tqp_vector->vector_irq);
+ hns3_dbg_fill_content(content, sizeof(content), queue_map_items,
+ (const char **)result,
+ ARRAY_SIZE(queue_map_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
}
return 0;
}
-static int hns3_dbg_bd_info(struct hnae3_handle *h, const char *cmd_buf)
+static const struct hns3_dbg_item rx_bd_info_items[] = {
+ { "BD_IDX", 3 },
+ { "L234_INFO", 2 },
+ { "PKT_LEN", 3 },
+ { "SIZE", 4 },
+ { "RSS_HASH", 4 },
+ { "FD_ID", 2 },
+ { "VLAN_TAG", 2 },
+ { "O_DM_VLAN_ID_FB", 2 },
+ { "OT_VLAN_TAG", 2 },
+ { "BD_BASE_INFO", 2 },
+ { "PTYPE", 2 },
+ { "HW_CSUM", 2 },
+};
+
+static void hns3_dump_rx_bd_info(struct hns3_nic_priv *priv,
+ struct hns3_desc *desc, char **result, int idx)
{
- struct hns3_nic_priv *priv = h->priv;
- struct hns3_desc *rx_desc, *tx_desc;
- struct device *dev = &h->pdev->dev;
- struct hns3_enet_ring *ring;
- u32 tx_index, rx_index;
- u32 q_num, value;
- dma_addr_t addr;
- u16 mss_hw_csum;
- u32 l234info;
- int cnt;
-
- cnt = sscanf(&cmd_buf[8], "%u %u", &q_num, &tx_index);
- if (cnt == 2) {
- rx_index = tx_index;
- } else if (cnt != 1) {
- dev_err(dev, "bd info: bad command string, cnt=%d\n", cnt);
- return -EINVAL;
+ unsigned int j = 0;
+
+ sprintf(result[j++], "%5d", idx);
+ sprintf(result[j++], "%#x", le32_to_cpu(desc->rx.l234_info));
+ sprintf(result[j++], "%7u", le16_to_cpu(desc->rx.pkt_len));
+ sprintf(result[j++], "%4u", le16_to_cpu(desc->rx.size));
+ sprintf(result[j++], "%#x", le32_to_cpu(desc->rx.rss_hash));
+ sprintf(result[j++], "%5u", le16_to_cpu(desc->rx.fd_id));
+ sprintf(result[j++], "%8u", le16_to_cpu(desc->rx.vlan_tag));
+ sprintf(result[j++], "%15u", le16_to_cpu(desc->rx.o_dm_vlan_id_fb));
+ sprintf(result[j++], "%11u", le16_to_cpu(desc->rx.ot_vlan_tag));
+ sprintf(result[j++], "%#x", le32_to_cpu(desc->rx.bd_base_info));
+ if (test_bit(HNS3_NIC_STATE_RXD_ADV_LAYOUT_ENABLE, &priv->state)) {
+ u32 ol_info = le32_to_cpu(desc->rx.ol_info);
+
+ sprintf(result[j++], "%5lu", hnae3_get_field(ol_info,
+ HNS3_RXD_PTYPE_M,
+ HNS3_RXD_PTYPE_S));
+ sprintf(result[j++], "%7u", le16_to_cpu(desc->csum));
+ } else {
+ sprintf(result[j++], "NA");
+ sprintf(result[j++], "NA");
}
+}
+
+static int hns3_dbg_rx_bd_info(struct hns3_dbg_data *d, char *buf, int len)
+{
+ char data_str[ARRAY_SIZE(rx_bd_info_items)][HNS3_DBG_DATA_STR_LEN];
+ struct hns3_nic_priv *priv = d->handle->priv;
+ char *result[ARRAY_SIZE(rx_bd_info_items)];
+ char content[HNS3_DBG_INFO_LEN];
+ struct hns3_enet_ring *ring;
+ struct hns3_desc *desc;
+ unsigned int i;
+ int pos = 0;
- if (q_num >= h->kinfo.num_tqps) {
- dev_err(dev, "Queue number(%u) is out of range(0-%u)\n", q_num,
- h->kinfo.num_tqps - 1);
+ if (d->qid >= d->handle->kinfo.num_tqps) {
+ dev_err(&d->handle->pdev->dev,
+ "queue%u is not in use\n", d->qid);
return -EINVAL;
}
- ring = &priv->ring[q_num];
- value = readl_relaxed(ring->tqp->io_base + HNS3_RING_TX_RING_TAIL_REG);
- tx_index = (cnt == 1) ? value : tx_index;
+ for (i = 0; i < ARRAY_SIZE(rx_bd_info_items); i++)
+ result[i] = &data_str[i][0];
- if (tx_index >= ring->desc_num) {
- dev_err(dev, "bd index(%u) is out of range(0-%u)\n", tx_index,
- ring->desc_num - 1);
- return -EINVAL;
- }
+ pos += scnprintf(buf + pos, len - pos,
+ "Queue %u rx bd info:\n", d->qid);
+ hns3_dbg_fill_content(content, sizeof(content), rx_bd_info_items,
+ NULL, ARRAY_SIZE(rx_bd_info_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
- tx_desc = &ring->desc[tx_index];
- addr = le64_to_cpu(tx_desc->addr);
- mss_hw_csum = le16_to_cpu(tx_desc->tx.mss_hw_csum);
- dev_info(dev, "TX Queue Num: %u, BD Index: %u\n", q_num, tx_index);
- dev_info(dev, "(TX)addr: %pad\n", &addr);
- dev_info(dev, "(TX)vlan_tag: %u\n", le16_to_cpu(tx_desc->tx.vlan_tag));
- dev_info(dev, "(TX)send_size: %u\n",
- le16_to_cpu(tx_desc->tx.send_size));
-
- if (mss_hw_csum & BIT(HNS3_TXD_HW_CS_B)) {
- u32 offset = le32_to_cpu(tx_desc->tx.ol_type_vlan_len_msec);
- u32 start = le32_to_cpu(tx_desc->tx.type_cs_vlan_tso_len);
-
- dev_info(dev, "(TX)csum start: %u\n",
- hnae3_get_field(start,
- HNS3_TXD_CSUM_START_M,
- HNS3_TXD_CSUM_START_S));
- dev_info(dev, "(TX)csum offset: %u\n",
- hnae3_get_field(offset,
- HNS3_TXD_CSUM_OFFSET_M,
- HNS3_TXD_CSUM_OFFSET_S));
- } else {
- dev_info(dev, "(TX)vlan_tso: %u\n",
- tx_desc->tx.type_cs_vlan_tso);
- dev_info(dev, "(TX)l2_len: %u\n", tx_desc->tx.l2_len);
- dev_info(dev, "(TX)l3_len: %u\n", tx_desc->tx.l3_len);
- dev_info(dev, "(TX)l4_len: %u\n", tx_desc->tx.l4_len);
- dev_info(dev, "(TX)vlan_msec: %u\n",
- tx_desc->tx.ol_type_vlan_msec);
- dev_info(dev, "(TX)ol2_len: %u\n", tx_desc->tx.ol2_len);
- dev_info(dev, "(TX)ol3_len: %u\n", tx_desc->tx.ol3_len);
- dev_info(dev, "(TX)ol4_len: %u\n", tx_desc->tx.ol4_len);
- }
+ ring = &priv->ring[d->qid + d->handle->kinfo.num_tqps];
+ for (i = 0; i < ring->desc_num; i++) {
+ desc = &ring->desc[i];
- dev_info(dev, "(TX)vlan_tag: %u\n",
- le16_to_cpu(tx_desc->tx.outer_vlan_tag));
- dev_info(dev, "(TX)tv: %u\n", le16_to_cpu(tx_desc->tx.tv));
- dev_info(dev, "(TX)paylen_ol4cs: %u\n",
- le32_to_cpu(tx_desc->tx.paylen_ol4cs));
- dev_info(dev, "(TX)vld_ra_ri: %u\n",
- le16_to_cpu(tx_desc->tx.bdtp_fe_sc_vld_ra_ri));
- dev_info(dev, "(TX)mss_hw_csum: %u\n", mss_hw_csum);
-
- ring = &priv->ring[q_num + h->kinfo.num_tqps];
- value = readl_relaxed(ring->tqp->io_base + HNS3_RING_RX_RING_TAIL_REG);
- rx_index = (cnt == 1) ? value : tx_index;
- rx_desc = &ring->desc[rx_index];
-
- addr = le64_to_cpu(rx_desc->addr);
- l234info = le32_to_cpu(rx_desc->rx.l234_info);
- dev_info(dev, "RX Queue Num: %u, BD Index: %u\n", q_num, rx_index);
- dev_info(dev, "(RX)addr: %pad\n", &addr);
- dev_info(dev, "(RX)l234_info: %u\n", l234info);
-
- if (l234info & BIT(HNS3_RXD_L2_CSUM_B)) {
- u32 lo, hi;
-
- lo = hnae3_get_field(l234info, HNS3_RXD_L2_CSUM_L_M,
- HNS3_RXD_L2_CSUM_L_S);
- hi = hnae3_get_field(l234info, HNS3_RXD_L2_CSUM_H_M,
- HNS3_RXD_L2_CSUM_H_S);
- dev_info(dev, "(RX)csum: %u\n", lo | hi << 8);
+ hns3_dump_rx_bd_info(priv, desc, result, i);
+ hns3_dbg_fill_content(content, sizeof(content),
+ rx_bd_info_items, (const char **)result,
+ ARRAY_SIZE(rx_bd_info_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
}
- dev_info(dev, "(RX)pkt_len: %u\n", le16_to_cpu(rx_desc->rx.pkt_len));
- dev_info(dev, "(RX)size: %u\n", le16_to_cpu(rx_desc->rx.size));
- dev_info(dev, "(RX)rss_hash: %u\n", le32_to_cpu(rx_desc->rx.rss_hash));
- dev_info(dev, "(RX)fd_id: %u\n", le16_to_cpu(rx_desc->rx.fd_id));
- dev_info(dev, "(RX)vlan_tag: %u\n", le16_to_cpu(rx_desc->rx.vlan_tag));
- dev_info(dev, "(RX)o_dm_vlan_id_fb: %u\n",
- le16_to_cpu(rx_desc->rx.o_dm_vlan_id_fb));
- dev_info(dev, "(RX)ot_vlan_tag: %u\n",
- le16_to_cpu(rx_desc->rx.ot_vlan_tag));
- dev_info(dev, "(RX)bd_base_info: %u\n",
- le32_to_cpu(rx_desc->rx.bd_base_info));
-
return 0;
}
-static void hns3_dbg_help(struct hnae3_handle *h)
-{
-#define HNS3_DBG_BUF_LEN 256
-
- char printf_buf[HNS3_DBG_BUF_LEN];
-
- dev_info(&h->pdev->dev, "available commands\n");
- dev_info(&h->pdev->dev, "queue info <number>\n");
- dev_info(&h->pdev->dev, "queue map\n");
- dev_info(&h->pdev->dev, "bd info <q_num> <bd index>\n");
- dev_info(&h->pdev->dev, "dev capability\n");
- dev_info(&h->pdev->dev, "dev spec\n");
-
- if (!hns3_is_phys_func(h->pdev))
- return;
-
- dev_info(&h->pdev->dev, "dump fd tcam\n");
- dev_info(&h->pdev->dev, "dump tc\n");
- dev_info(&h->pdev->dev, "dump tm map <q_num>\n");
- dev_info(&h->pdev->dev, "dump tm\n");
- dev_info(&h->pdev->dev, "dump qos pause cfg\n");
- dev_info(&h->pdev->dev, "dump qos pri map\n");
- dev_info(&h->pdev->dev, "dump qos buf cfg\n");
- dev_info(&h->pdev->dev, "dump mng tbl\n");
- dev_info(&h->pdev->dev, "dump reset info\n");
- dev_info(&h->pdev->dev, "dump m7 info\n");
- dev_info(&h->pdev->dev, "dump ncl_config <offset> <length>(in hex)\n");
- dev_info(&h->pdev->dev, "dump mac tnl status\n");
- dev_info(&h->pdev->dev, "dump loopback\n");
- dev_info(&h->pdev->dev, "dump qs shaper [qs id]\n");
- dev_info(&h->pdev->dev, "dump uc mac list <func id>\n");
- dev_info(&h->pdev->dev, "dump mc mac list <func id>\n");
- dev_info(&h->pdev->dev, "dump intr\n");
-
- memset(printf_buf, 0, HNS3_DBG_BUF_LEN);
- strncat(printf_buf, "dump reg [[bios common] [ssu <port_id>]",
- HNS3_DBG_BUF_LEN - 1);
- strncat(printf_buf + strlen(printf_buf),
- " [igu egu <port_id>] [rpu <tc_queue_num>]",
- HNS3_DBG_BUF_LEN - strlen(printf_buf) - 1);
- strncat(printf_buf + strlen(printf_buf),
- " [rtc] [ppp] [rcb] [tqp <queue_num>] [mac]]\n",
- HNS3_DBG_BUF_LEN - strlen(printf_buf) - 1);
- dev_info(&h->pdev->dev, "%s", printf_buf);
-
- memset(printf_buf, 0, HNS3_DBG_BUF_LEN);
- strncat(printf_buf, "dump reg dcb <port_id> <pri_id> <pg_id>",
- HNS3_DBG_BUF_LEN - 1);
- strncat(printf_buf + strlen(printf_buf), " <rq_id> <nq_id> <qset_id>\n",
- HNS3_DBG_BUF_LEN - strlen(printf_buf) - 1);
- dev_info(&h->pdev->dev, "%s", printf_buf);
-}
+static const struct hns3_dbg_item tx_bd_info_items[] = {
+ { "BD_IDX", 5 },
+ { "ADDRESS", 2 },
+ { "VLAN_TAG", 2 },
+ { "SIZE", 2 },
+ { "T_CS_VLAN_TSO", 2 },
+ { "OT_VLAN_TAG", 3 },
+ { "TV", 2 },
+ { "OLT_VLAN_LEN", 2},
+ { "PAYLEN_OL4CS", 2},
+ { "BD_FE_SC_VLD", 2},
+ { "MSS_HW_CSUM", 0},
+};
-static void hns3_dbg_dev_caps(struct hnae3_handle *h)
+static void hns3_dump_tx_bd_info(struct hns3_nic_priv *priv,
+ struct hns3_desc *desc, char **result, int idx)
{
- struct hnae3_ae_dev *ae_dev = pci_get_drvdata(h->pdev);
- unsigned long *caps;
-
- caps = ae_dev->caps;
-
- dev_info(&h->pdev->dev, "support FD: %s\n",
- test_bit(HNAE3_DEV_SUPPORT_FD_B, caps) ? "yes" : "no");
- dev_info(&h->pdev->dev, "support GRO: %s\n",
- test_bit(HNAE3_DEV_SUPPORT_GRO_B, caps) ? "yes" : "no");
- dev_info(&h->pdev->dev, "support FEC: %s\n",
- test_bit(HNAE3_DEV_SUPPORT_FEC_B, caps) ? "yes" : "no");
- dev_info(&h->pdev->dev, "support UDP GSO: %s\n",
- test_bit(HNAE3_DEV_SUPPORT_UDP_GSO_B, caps) ? "yes" : "no");
- dev_info(&h->pdev->dev, "support PTP: %s\n",
- test_bit(HNAE3_DEV_SUPPORT_PTP_B, caps) ? "yes" : "no");
- dev_info(&h->pdev->dev, "support INT QL: %s\n",
- test_bit(HNAE3_DEV_SUPPORT_INT_QL_B, caps) ? "yes" : "no");
- dev_info(&h->pdev->dev, "support HW TX csum: %s\n",
- test_bit(HNAE3_DEV_SUPPORT_HW_TX_CSUM_B, caps) ? "yes" : "no");
- dev_info(&h->pdev->dev, "support UDP tunnel csum: %s\n",
- test_bit(HNAE3_DEV_SUPPORT_UDP_TUNNEL_CSUM_B, caps) ?
- "yes" : "no");
- dev_info(&h->pdev->dev, "support PAUSE: %s\n",
- test_bit(HNAE3_DEV_SUPPORT_PAUSE_B, ae_dev->caps) ?
- "yes" : "no");
- dev_info(&h->pdev->dev, "support imp-controlled PHY: %s\n",
- test_bit(HNAE3_DEV_SUPPORT_PHY_IMP_B, caps) ? "yes" : "no");
+ unsigned int j = 0;
+
+ sprintf(result[j++], "%6d", idx);
+ sprintf(result[j++], "%#llx", le64_to_cpu(desc->addr));
+ sprintf(result[j++], "%5u", le16_to_cpu(desc->tx.vlan_tag));
+ sprintf(result[j++], "%5u", le16_to_cpu(desc->tx.send_size));
+ sprintf(result[j++], "%#x",
+ le32_to_cpu(desc->tx.type_cs_vlan_tso_len));
+ sprintf(result[j++], "%5u", le16_to_cpu(desc->tx.outer_vlan_tag));
+ sprintf(result[j++], "%5u", le16_to_cpu(desc->tx.tv));
+ sprintf(result[j++], "%10u",
+ le32_to_cpu(desc->tx.ol_type_vlan_len_msec));
+ sprintf(result[j++], "%#x", le32_to_cpu(desc->tx.paylen_ol4cs));
+ sprintf(result[j++], "%#x", le16_to_cpu(desc->tx.bdtp_fe_sc_vld_ra_ri));
+ sprintf(result[j++], "%5u", le16_to_cpu(desc->tx.mss_hw_csum));
}
-static void hns3_dbg_dev_specs(struct hnae3_handle *h)
+static int hns3_dbg_tx_bd_info(struct hns3_dbg_data *d, char *buf, int len)
{
- struct hnae3_ae_dev *ae_dev = pci_get_drvdata(h->pdev);
- struct hnae3_dev_specs *dev_specs = &ae_dev->dev_specs;
- struct hnae3_knic_private_info *kinfo = &h->kinfo;
- struct hns3_nic_priv *priv = h->priv;
-
- dev_info(priv->dev, "MAC entry num: %u\n", dev_specs->mac_entry_num);
- dev_info(priv->dev, "MNG entry num: %u\n", dev_specs->mng_entry_num);
- dev_info(priv->dev, "MAX non tso bd num: %u\n",
- dev_specs->max_non_tso_bd_num);
- dev_info(priv->dev, "RSS ind tbl size: %u\n",
- dev_specs->rss_ind_tbl_size);
- dev_info(priv->dev, "RSS key size: %u\n", dev_specs->rss_key_size);
- dev_info(priv->dev, "RSS size: %u\n", kinfo->rss_size);
- dev_info(priv->dev, "Allocated RSS size: %u\n", kinfo->req_rss_size);
- dev_info(priv->dev, "Task queue pairs numbers: %u\n", kinfo->num_tqps);
-
- dev_info(priv->dev, "RX buffer length: %u\n", kinfo->rx_buf_len);
- dev_info(priv->dev, "Desc num per TX queue: %u\n", kinfo->num_tx_desc);
- dev_info(priv->dev, "Desc num per RX queue: %u\n", kinfo->num_rx_desc);
- dev_info(priv->dev, "Total number of enabled TCs: %u\n",
- kinfo->tc_info.num_tc);
- dev_info(priv->dev, "MAX INT QL: %u\n", dev_specs->int_ql_max);
- dev_info(priv->dev, "MAX INT GL: %u\n", dev_specs->max_int_gl);
- dev_info(priv->dev, "MAX frame size: %u\n", dev_specs->max_frm_size);
- dev_info(priv->dev, "MAX TM RATE: %uMbps\n", dev_specs->max_tm_rate);
- dev_info(priv->dev, "MAX QSET number: %u\n", dev_specs->max_qset_num);
-}
+ char data_str[ARRAY_SIZE(tx_bd_info_items)][HNS3_DBG_DATA_STR_LEN];
+ struct hns3_nic_priv *priv = d->handle->priv;
+ char *result[ARRAY_SIZE(tx_bd_info_items)];
+ char content[HNS3_DBG_INFO_LEN];
+ struct hns3_enet_ring *ring;
+ struct hns3_desc *desc;
+ unsigned int i;
+ int pos = 0;
-static ssize_t hns3_dbg_cmd_read(struct file *filp, char __user *buffer,
- size_t count, loff_t *ppos)
-{
- int uncopy_bytes;
- char *buf;
- int len;
+ if (d->qid >= d->handle->kinfo.num_tqps) {
+ dev_err(&d->handle->pdev->dev,
+ "queue%u is not in use\n", d->qid);
+ return -EINVAL;
+ }
- if (*ppos != 0)
- return 0;
+ for (i = 0; i < ARRAY_SIZE(tx_bd_info_items); i++)
+ result[i] = &data_str[i][0];
- if (count < HNS3_DBG_READ_LEN)
- return -ENOSPC;
+ pos += scnprintf(buf + pos, len - pos,
+ "Queue %u tx bd info:\n", d->qid);
+ hns3_dbg_fill_content(content, sizeof(content), tx_bd_info_items,
+ NULL, ARRAY_SIZE(tx_bd_info_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
- buf = kzalloc(HNS3_DBG_READ_LEN, GFP_KERNEL);
- if (!buf)
- return -ENOMEM;
+ ring = &priv->ring[d->qid];
+ for (i = 0; i < ring->desc_num; i++) {
+ desc = &ring->desc[i];
- len = scnprintf(buf, HNS3_DBG_READ_LEN, "%s\n",
- "Please echo help to cmd to get help information");
- uncopy_bytes = copy_to_user(buffer, buf, len);
+ hns3_dump_tx_bd_info(priv, desc, result, i);
+ hns3_dbg_fill_content(content, sizeof(content),
+ tx_bd_info_items, (const char **)result,
+ ARRAY_SIZE(tx_bd_info_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
+ }
- kfree(buf);
+ return 0;
+}
- if (uncopy_bytes)
- return -EFAULT;
+static void
+hns3_dbg_dev_caps(struct hnae3_handle *h, char *buf, int len, int *pos)
+{
+ struct hnae3_ae_dev *ae_dev = pci_get_drvdata(h->pdev);
+ static const char * const str[] = {"no", "yes"};
+ unsigned long *caps = ae_dev->caps;
+ u32 i, state;
+
+ *pos += scnprintf(buf + *pos, len - *pos, "dev capability:\n");
+
+ for (i = 0; i < ARRAY_SIZE(hns3_dbg_cap); i++) {
+ state = test_bit(hns3_dbg_cap[i].cap_bit, caps);
+ *pos += scnprintf(buf + *pos, len - *pos, "%s: %s\n",
+ hns3_dbg_cap[i].name, str[state]);
+ }
- return (*ppos = len);
+ *pos += scnprintf(buf + *pos, len - *pos, "\n");
}
-static int hns3_dbg_check_cmd(struct hnae3_handle *handle, char *cmd_buf)
+static void
+hns3_dbg_dev_specs(struct hnae3_handle *h, char *buf, int len, int *pos)
{
- int ret = 0;
-
- if (strncmp(cmd_buf, "help", 4) == 0)
- hns3_dbg_help(handle);
- else if (strncmp(cmd_buf, "queue info", 10) == 0)
- ret = hns3_dbg_queue_info(handle, cmd_buf);
- else if (strncmp(cmd_buf, "queue map", 9) == 0)
- ret = hns3_dbg_queue_map(handle);
- else if (strncmp(cmd_buf, "bd info", 7) == 0)
- ret = hns3_dbg_bd_info(handle, cmd_buf);
- else if (strncmp(cmd_buf, "dev capability", 14) == 0)
- hns3_dbg_dev_caps(handle);
- else if (strncmp(cmd_buf, "dev spec", 8) == 0)
- hns3_dbg_dev_specs(handle);
- else if (handle->ae_algo->ops->dbg_run_cmd)
- ret = handle->ae_algo->ops->dbg_run_cmd(handle, cmd_buf);
- else
- ret = -EOPNOTSUPP;
+ struct hnae3_ae_dev *ae_dev = pci_get_drvdata(h->pdev);
+ struct hnae3_dev_specs *dev_specs = &ae_dev->dev_specs;
+ struct hnae3_knic_private_info *kinfo = &h->kinfo;
- return ret;
+ *pos += scnprintf(buf + *pos, len - *pos, "dev_spec:\n");
+ *pos += scnprintf(buf + *pos, len - *pos, "MAC entry num: %u\n",
+ dev_specs->mac_entry_num);
+ *pos += scnprintf(buf + *pos, len - *pos, "MNG entry num: %u\n",
+ dev_specs->mng_entry_num);
+ *pos += scnprintf(buf + *pos, len - *pos, "MAX non tso bd num: %u\n",
+ dev_specs->max_non_tso_bd_num);
+ *pos += scnprintf(buf + *pos, len - *pos, "RSS ind tbl size: %u\n",
+ dev_specs->rss_ind_tbl_size);
+ *pos += scnprintf(buf + *pos, len - *pos, "RSS key size: %u\n",
+ dev_specs->rss_key_size);
+ *pos += scnprintf(buf + *pos, len - *pos, "RSS size: %u\n",
+ kinfo->rss_size);
+ *pos += scnprintf(buf + *pos, len - *pos, "Allocated RSS size: %u\n",
+ kinfo->req_rss_size);
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "Task queue pairs numbers: %u\n",
+ kinfo->num_tqps);
+ *pos += scnprintf(buf + *pos, len - *pos, "RX buffer length: %u\n",
+ kinfo->rx_buf_len);
+ *pos += scnprintf(buf + *pos, len - *pos, "Desc num per TX queue: %u\n",
+ kinfo->num_tx_desc);
+ *pos += scnprintf(buf + *pos, len - *pos, "Desc num per RX queue: %u\n",
+ kinfo->num_rx_desc);
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "Total number of enabled TCs: %u\n",
+ kinfo->tc_info.num_tc);
+ *pos += scnprintf(buf + *pos, len - *pos, "MAX INT QL: %u\n",
+ dev_specs->int_ql_max);
+ *pos += scnprintf(buf + *pos, len - *pos, "MAX INT GL: %u\n",
+ dev_specs->max_int_gl);
+ *pos += scnprintf(buf + *pos, len - *pos, "MAX TM RATE: %u\n",
+ dev_specs->max_tm_rate);
+ *pos += scnprintf(buf + *pos, len - *pos, "MAX QSET number: %u\n",
+ dev_specs->max_qset_num);
}
-static ssize_t hns3_dbg_cmd_write(struct file *filp, const char __user *buffer,
- size_t count, loff_t *ppos)
+static int hns3_dbg_dev_info(struct hnae3_handle *h, char *buf, int len)
{
- struct hnae3_handle *handle = filp->private_data;
- struct hns3_nic_priv *priv = handle->priv;
- char *cmd_buf, *cmd_buf_tmp;
- int uncopied_bytes;
- int ret;
+ int pos = 0;
- if (*ppos != 0)
- return 0;
+ hns3_dbg_dev_caps(h, buf, len, &pos);
- /* Judge if the instance is being reset. */
- if (!test_bit(HNS3_NIC_STATE_INITED, &priv->state) ||
- test_bit(HNS3_NIC_STATE_RESETTING, &priv->state))
- return 0;
+ hns3_dbg_dev_specs(h, buf, len, &pos);
- if (count > HNS3_DBG_WRITE_LEN)
- return -ENOSPC;
+ return 0;
+}
- cmd_buf = kzalloc(count + 1, GFP_KERNEL);
- if (!cmd_buf)
- return count;
+static int hns3_dbg_get_cmd_index(struct hnae3_handle *handle,
+ const unsigned char *name, u32 *index)
+{
+ u32 i;
- uncopied_bytes = copy_from_user(cmd_buf, buffer, count);
- if (uncopied_bytes) {
- kfree(cmd_buf);
- return -EFAULT;
+ for (i = 0; i < ARRAY_SIZE(hns3_dbg_cmd); i++) {
+ if (!strncmp(name, hns3_dbg_cmd[i].name,
+ strlen(hns3_dbg_cmd[i].name))) {
+ *index = i;
+ return 0;
+ }
}
- cmd_buf[count] = '\0';
+ dev_err(&handle->pdev->dev, "unknown command(%s)\n", name);
+ return -EINVAL;
+}
- cmd_buf_tmp = strchr(cmd_buf, '\n');
- if (cmd_buf_tmp) {
- *cmd_buf_tmp = '\0';
- count = cmd_buf_tmp - cmd_buf + 1;
- }
+static const struct hns3_dbg_func hns3_dbg_cmd_func[] = {
+ {
+ .cmd = HNAE3_DBG_CMD_QUEUE_MAP,
+ .dbg_dump = hns3_dbg_queue_map,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_DEV_INFO,
+ .dbg_dump = hns3_dbg_dev_info,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_TX_BD,
+ .dbg_dump_bd = hns3_dbg_tx_bd_info,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_RX_BD,
+ .dbg_dump_bd = hns3_dbg_rx_bd_info,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_RX_QUEUE_INFO,
+ .dbg_dump = hns3_dbg_rx_queue_info,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_TX_QUEUE_INFO,
+ .dbg_dump = hns3_dbg_tx_queue_info,
+ },
+};
- ret = hns3_dbg_check_cmd(handle, cmd_buf);
- if (ret)
- hns3_dbg_help(handle);
+static int hns3_dbg_read_cmd(struct hns3_dbg_data *dbg_data,
+ enum hnae3_dbg_cmd cmd, char *buf, int len)
+{
+ const struct hnae3_ae_ops *ops = dbg_data->handle->ae_algo->ops;
+ const struct hns3_dbg_func *cmd_func;
+ u32 i;
+
+ for (i = 0; i < ARRAY_SIZE(hns3_dbg_cmd_func); i++) {
+ if (cmd == hns3_dbg_cmd_func[i].cmd) {
+ cmd_func = &hns3_dbg_cmd_func[i];
+ if (cmd_func->dbg_dump)
+ return cmd_func->dbg_dump(dbg_data->handle, buf,
+ len);
+ else
+ return cmd_func->dbg_dump_bd(dbg_data, buf,
+ len);
+ }
+ }
- kfree(cmd_buf);
- cmd_buf = NULL;
+ if (!ops->dbg_read_cmd)
+ return -EOPNOTSUPP;
- return count;
+ return ops->dbg_read_cmd(dbg_data->handle, cmd, buf, len);
}
static ssize_t hns3_dbg_read(struct file *filp, char __user *buffer,
size_t count, loff_t *ppos)
{
- struct hnae3_handle *handle = filp->private_data;
- const struct hnae3_ae_ops *ops = handle->ae_algo->ops;
+ struct hns3_dbg_data *dbg_data = filp->private_data;
+ struct hnae3_handle *handle = dbg_data->handle;
struct hns3_nic_priv *priv = handle->priv;
- char *cmd_buf, *read_buf;
ssize_t size = 0;
- int ret = 0;
-
- read_buf = kzalloc(HNS3_DBG_READ_LEN, GFP_KERNEL);
- if (!read_buf)
- return -ENOMEM;
+ char **save_buf;
+ char *read_buf;
+ u32 index;
+ int ret;
- cmd_buf = filp->f_path.dentry->d_iname;
+ ret = hns3_dbg_get_cmd_index(handle, filp->f_path.dentry->d_iname,
+ &index);
+ if (ret)
+ return ret;
- if (ops->dbg_read_cmd)
- ret = ops->dbg_read_cmd(handle, cmd_buf, read_buf,
- HNS3_DBG_READ_LEN);
+ save_buf = &hns3_dbg_cmd[index].buf;
- if (ret) {
- dev_info(priv->dev, "unknown command\n");
+ if (!test_bit(HNS3_NIC_STATE_INITED, &priv->state) ||
+ test_bit(HNS3_NIC_STATE_RESETTING, &priv->state)) {
+ ret = -EBUSY;
goto out;
}
+ if (*save_buf) {
+ read_buf = *save_buf;
+ } else {
+ read_buf = kvzalloc(hns3_dbg_cmd[index].buf_len, GFP_KERNEL);
+ if (!read_buf)
+ return -ENOMEM;
+
+ /* save the buffer addr until the last read operation */
+ *save_buf = read_buf;
+ }
+
+ /* get data ready for the first time to read */
+ if (!*ppos) {
+ ret = hns3_dbg_read_cmd(dbg_data, hns3_dbg_cmd[index].cmd,
+ read_buf, hns3_dbg_cmd[index].buf_len);
+ if (ret)
+ goto out;
+ }
+
size = simple_read_from_buffer(buffer, count, ppos, read_buf,
strlen(read_buf));
+ if (size > 0)
+ return size;
out:
- kfree(read_buf);
- return size;
-}
+ /* free the buffer for the last read operation */
+ if (*save_buf) {
+ kvfree(*save_buf);
+ *save_buf = NULL;
+ }
-static const struct file_operations hns3_dbg_cmd_fops = {
- .owner = THIS_MODULE,
- .open = simple_open,
- .read = hns3_dbg_cmd_read,
- .write = hns3_dbg_cmd_write,
-};
+ return ret;
+}
static const struct file_operations hns3_dbg_fops = {
.owner = THIS_MODULE,
.read = hns3_dbg_read,
};
-void hns3_dbg_init(struct hnae3_handle *handle)
+static int hns3_dbg_bd_file_init(struct hnae3_handle *handle, u32 cmd)
+{
+ struct dentry *entry_dir;
+ struct hns3_dbg_data *data;
+ u16 max_queue_num;
+ unsigned int i;
+
+ entry_dir = hns3_dbg_dentry[hns3_dbg_cmd[cmd].dentry].dentry;
+ max_queue_num = hns3_get_max_available_channels(handle);
+ data = devm_kzalloc(&handle->pdev->dev, max_queue_num * sizeof(*data),
+ GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+
+ for (i = 0; i < max_queue_num; i++) {
+ char name[HNS3_DBG_FILE_NAME_LEN];
+
+ data[i].handle = handle;
+ data[i].qid = i;
+ sprintf(name, "%s%u", hns3_dbg_cmd[cmd].name, i);
+ debugfs_create_file(name, 0400, entry_dir, &data[i],
+ &hns3_dbg_fops);
+ }
+
+ return 0;
+}
+
+static int
+hns3_dbg_common_file_init(struct hnae3_handle *handle, u32 cmd)
+{
+ struct hns3_dbg_data *data;
+ struct dentry *entry_dir;
+
+ data = devm_kzalloc(&handle->pdev->dev, sizeof(*data), GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+
+ data->handle = handle;
+ entry_dir = hns3_dbg_dentry[hns3_dbg_cmd[cmd].dentry].dentry;
+ debugfs_create_file(hns3_dbg_cmd[cmd].name, 0400, entry_dir,
+ data, &hns3_dbg_fops);
+
+ return 0;
+}
+
+int hns3_dbg_init(struct hnae3_handle *handle)
{
struct hnae3_ae_dev *ae_dev = pci_get_drvdata(handle->pdev);
const char *name = pci_name(handle->pdev);
- struct dentry *entry_dir;
+ int ret;
+ u32 i;
+
+ hns3_dbg_dentry[HNS3_DBG_DENTRY_COMMON].dentry =
+ debugfs_create_dir(name, hns3_dbgfs_root);
+ handle->hnae3_dbgfs = hns3_dbg_dentry[HNS3_DBG_DENTRY_COMMON].dentry;
+
+ for (i = 0; i < HNS3_DBG_DENTRY_COMMON; i++)
+ hns3_dbg_dentry[i].dentry =
+ debugfs_create_dir(hns3_dbg_dentry[i].name,
+ handle->hnae3_dbgfs);
+
+ for (i = 0; i < ARRAY_SIZE(hns3_dbg_cmd); i++) {
+ if ((hns3_dbg_cmd[i].cmd == HNAE3_DBG_CMD_TM_NODES &&
+ ae_dev->dev_version <= HNAE3_DEVICE_VERSION_V2) ||
+ (hns3_dbg_cmd[i].cmd == HNAE3_DBG_CMD_PTP_INFO &&
+ !test_bit(HNAE3_DEV_SUPPORT_PTP_B, ae_dev->caps)))
+ continue;
- handle->hnae3_dbgfs = debugfs_create_dir(name, hns3_dbgfs_root);
+ if (!hns3_dbg_cmd[i].init) {
+ dev_err(&handle->pdev->dev,
+ "cmd %s lack of init func\n",
+ hns3_dbg_cmd[i].name);
+ ret = -EINVAL;
+ goto out;
+ }
- debugfs_create_file("cmd", 0600, handle->hnae3_dbgfs, handle,
- &hns3_dbg_cmd_fops);
+ ret = hns3_dbg_cmd[i].init(handle, i);
+ if (ret) {
+ dev_err(&handle->pdev->dev, "failed to init cmd %s\n",
+ hns3_dbg_cmd[i].name);
+ goto out;
+ }
+ }
- entry_dir = debugfs_create_dir("tm", handle->hnae3_dbgfs);
- if (ae_dev->dev_version > HNAE3_DEVICE_VERSION_V2)
- debugfs_create_file(HNAE3_DBG_TM_NODES, 0600, entry_dir, handle,
- &hns3_dbg_fops);
- debugfs_create_file(HNAE3_DBG_TM_PRI, 0600, entry_dir, handle,
- &hns3_dbg_fops);
- debugfs_create_file(HNAE3_DBG_TM_QSET, 0600, entry_dir, handle,
- &hns3_dbg_fops);
+ return 0;
+
+out:
+ debugfs_remove_recursive(handle->hnae3_dbgfs);
+ handle->hnae3_dbgfs = NULL;
+ return ret;
}
void hns3_dbg_uninit(struct hnae3_handle *handle)
{
+ u32 i;
+
+ for (i = 0; i < ARRAY_SIZE(hns3_dbg_cmd); i++)
+ if (hns3_dbg_cmd[i].buf) {
+ kvfree(hns3_dbg_cmd[i].buf);
+ hns3_dbg_cmd[i].buf = NULL;
+ }
+
debugfs_remove_recursive(handle->hnae3_dbgfs);
handle->hnae3_dbgfs = NULL;
}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0+ */
+/* Copyright (c) 2021 Hisilicon Limited. */
+
+#ifndef __HNS3_DEBUGFS_H
+#define __HNS3_DEBUGFS_H
+
+#define HNS3_DBG_READ_LEN 65536
+#define HNS3_DBG_READ_LEN_128KB 0x20000
+#define HNS3_DBG_READ_LEN_1MB 0x100000
+#define HNS3_DBG_READ_LEN_4MB 0x400000
+#define HNS3_DBG_WRITE_LEN 1024
+
+#define HNS3_DBG_DATA_STR_LEN 32
+#define HNS3_DBG_INFO_LEN 256
+#define HNS3_DBG_ITEM_NAME_LEN 32
+#define HNS3_DBG_FILE_NAME_LEN 16
+
+struct hns3_dbg_item {
+ char name[HNS3_DBG_ITEM_NAME_LEN];
+ u16 interval; /* blank numbers after the item */
+};
+
+struct hns3_dbg_data {
+ struct hnae3_handle *handle;
+ u16 qid;
+};
+
+enum hns3_dbg_dentry_type {
+ HNS3_DBG_DENTRY_TM,
+ HNS3_DBG_DENTRY_TX_BD,
+ HNS3_DBG_DENTRY_RX_BD,
+ HNS3_DBG_DENTRY_MAC,
+ HNS3_DBG_DENTRY_REG,
+ HNS3_DBG_DENTRY_QUEUE,
+ HNS3_DBG_DENTRY_FD,
+ HNS3_DBG_DENTRY_COMMON,
+};
+
+struct hns3_dbg_dentry_info {
+ const char *name;
+ struct dentry *dentry;
+};
+
+struct hns3_dbg_cmd_info {
+ const char *name;
+ enum hnae3_dbg_cmd cmd;
+ enum hns3_dbg_dentry_type dentry;
+ u32 buf_len;
+ char *buf;
+ int (*init)(struct hnae3_handle *handle, unsigned int cmd);
+};
+
+struct hns3_dbg_func {
+ enum hnae3_dbg_cmd cmd;
+ int (*dbg_dump)(struct hnae3_handle *handle, char *buf, int len);
+ int (*dbg_dump_bd)(struct hns3_dbg_data *data, char *buf, int len);
+};
+
+struct hns3_dbg_cap_info {
+ const char *name;
+ enum HNAE3_DEV_CAP_BITS cap_bit;
+};
+
+#endif
module_param(debug, int, 0);
MODULE_PARM_DESC(debug, " Network interface message level setting");
+static unsigned int tx_spare_buf_size;
+module_param(tx_spare_buf_size, uint, 0400);
+MODULE_PARM_DESC(tx_spare_buf_size, "Size used to allocate tx spare buffer");
+
+static unsigned int tx_sgl = 1;
+module_param(tx_sgl, uint, 0600);
+MODULE_PARM_DESC(tx_sgl, "Minimum number of frags when using dma_map_sg() to optimize the IOMMU mapping");
+
+#define HNS3_SGL_SIZE(nfrag) (sizeof(struct scatterlist) * (nfrag) + \
+ sizeof(struct sg_table))
+#define HNS3_MAX_SGL_SIZE ALIGN(HNS3_SGL_SIZE(HNS3_MAX_TSO_BD_NUM),\
+ dma_get_cache_alignment())
+
#define DEFAULT_MSG_LEVEL (NETIF_MSG_PROBE | NETIF_MSG_LINK | \
NETIF_MSG_IFDOWN | NETIF_MSG_IFUP)
};
MODULE_DEVICE_TABLE(pci, hns3_pci_tbl);
+#define HNS3_RX_PTYPE_ENTRY(ptype, l, s, t) \
+ { ptype, \
+ l, \
+ CHECKSUM_##s, \
+ HNS3_L3_TYPE_##t, \
+ 1 }
+
+#define HNS3_RX_PTYPE_UNUSED_ENTRY(ptype) \
+ { ptype, 0, CHECKSUM_NONE, HNS3_L3_TYPE_PARSE_FAIL, 0 }
+
+static const struct hns3_rx_ptype hns3_rx_ptype_tbl[] = {
+ HNS3_RX_PTYPE_UNUSED_ENTRY(0),
+ HNS3_RX_PTYPE_ENTRY(1, 0, COMPLETE, ARP),
+ HNS3_RX_PTYPE_ENTRY(2, 0, COMPLETE, RARP),
+ HNS3_RX_PTYPE_ENTRY(3, 0, COMPLETE, LLDP),
+ HNS3_RX_PTYPE_ENTRY(4, 0, COMPLETE, PARSE_FAIL),
+ HNS3_RX_PTYPE_ENTRY(5, 0, COMPLETE, PARSE_FAIL),
+ HNS3_RX_PTYPE_ENTRY(6, 0, COMPLETE, PARSE_FAIL),
+ HNS3_RX_PTYPE_ENTRY(7, 0, COMPLETE, CNM),
+ HNS3_RX_PTYPE_ENTRY(8, 0, NONE, PARSE_FAIL),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(9),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(10),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(11),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(12),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(13),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(14),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(15),
+ HNS3_RX_PTYPE_ENTRY(16, 0, COMPLETE, PARSE_FAIL),
+ HNS3_RX_PTYPE_ENTRY(17, 0, COMPLETE, IPV4),
+ HNS3_RX_PTYPE_ENTRY(18, 0, COMPLETE, IPV4),
+ HNS3_RX_PTYPE_ENTRY(19, 0, UNNECESSARY, IPV4),
+ HNS3_RX_PTYPE_ENTRY(20, 0, UNNECESSARY, IPV4),
+ HNS3_RX_PTYPE_ENTRY(21, 0, NONE, IPV4),
+ HNS3_RX_PTYPE_ENTRY(22, 0, UNNECESSARY, IPV4),
+ HNS3_RX_PTYPE_ENTRY(23, 0, NONE, IPV4),
+ HNS3_RX_PTYPE_ENTRY(24, 0, NONE, IPV4),
+ HNS3_RX_PTYPE_ENTRY(25, 0, UNNECESSARY, IPV4),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(26),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(27),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(28),
+ HNS3_RX_PTYPE_ENTRY(29, 0, COMPLETE, PARSE_FAIL),
+ HNS3_RX_PTYPE_ENTRY(30, 0, COMPLETE, PARSE_FAIL),
+ HNS3_RX_PTYPE_ENTRY(31, 0, COMPLETE, IPV4),
+ HNS3_RX_PTYPE_ENTRY(32, 0, COMPLETE, IPV4),
+ HNS3_RX_PTYPE_ENTRY(33, 1, UNNECESSARY, IPV4),
+ HNS3_RX_PTYPE_ENTRY(34, 1, UNNECESSARY, IPV4),
+ HNS3_RX_PTYPE_ENTRY(35, 1, UNNECESSARY, IPV4),
+ HNS3_RX_PTYPE_ENTRY(36, 0, COMPLETE, IPV4),
+ HNS3_RX_PTYPE_ENTRY(37, 0, COMPLETE, IPV4),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(38),
+ HNS3_RX_PTYPE_ENTRY(39, 0, COMPLETE, IPV6),
+ HNS3_RX_PTYPE_ENTRY(40, 0, COMPLETE, IPV6),
+ HNS3_RX_PTYPE_ENTRY(41, 1, UNNECESSARY, IPV6),
+ HNS3_RX_PTYPE_ENTRY(42, 1, UNNECESSARY, IPV6),
+ HNS3_RX_PTYPE_ENTRY(43, 1, UNNECESSARY, IPV6),
+ HNS3_RX_PTYPE_ENTRY(44, 0, COMPLETE, IPV6),
+ HNS3_RX_PTYPE_ENTRY(45, 0, COMPLETE, IPV6),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(46),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(47),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(48),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(49),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(50),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(51),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(52),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(53),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(54),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(55),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(56),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(57),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(58),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(59),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(60),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(61),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(62),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(63),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(64),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(65),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(66),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(67),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(68),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(69),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(70),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(71),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(72),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(73),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(74),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(75),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(76),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(77),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(78),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(79),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(80),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(81),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(82),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(83),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(84),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(85),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(86),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(87),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(88),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(89),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(90),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(91),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(92),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(93),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(94),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(95),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(96),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(97),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(98),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(99),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(100),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(101),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(102),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(103),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(104),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(105),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(106),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(107),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(108),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(109),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(110),
+ HNS3_RX_PTYPE_ENTRY(111, 0, COMPLETE, IPV6),
+ HNS3_RX_PTYPE_ENTRY(112, 0, COMPLETE, IPV6),
+ HNS3_RX_PTYPE_ENTRY(113, 0, UNNECESSARY, IPV6),
+ HNS3_RX_PTYPE_ENTRY(114, 0, UNNECESSARY, IPV6),
+ HNS3_RX_PTYPE_ENTRY(115, 0, NONE, IPV6),
+ HNS3_RX_PTYPE_ENTRY(116, 0, UNNECESSARY, IPV6),
+ HNS3_RX_PTYPE_ENTRY(117, 0, NONE, IPV6),
+ HNS3_RX_PTYPE_ENTRY(118, 0, NONE, IPV6),
+ HNS3_RX_PTYPE_ENTRY(119, 0, UNNECESSARY, IPV6),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(120),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(121),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(122),
+ HNS3_RX_PTYPE_ENTRY(123, 0, COMPLETE, PARSE_FAIL),
+ HNS3_RX_PTYPE_ENTRY(124, 0, COMPLETE, PARSE_FAIL),
+ HNS3_RX_PTYPE_ENTRY(125, 0, COMPLETE, IPV4),
+ HNS3_RX_PTYPE_ENTRY(126, 0, COMPLETE, IPV4),
+ HNS3_RX_PTYPE_ENTRY(127, 1, UNNECESSARY, IPV4),
+ HNS3_RX_PTYPE_ENTRY(128, 1, UNNECESSARY, IPV4),
+ HNS3_RX_PTYPE_ENTRY(129, 1, UNNECESSARY, IPV4),
+ HNS3_RX_PTYPE_ENTRY(130, 0, COMPLETE, IPV4),
+ HNS3_RX_PTYPE_ENTRY(131, 0, COMPLETE, IPV4),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(132),
+ HNS3_RX_PTYPE_ENTRY(133, 0, COMPLETE, IPV6),
+ HNS3_RX_PTYPE_ENTRY(134, 0, COMPLETE, IPV6),
+ HNS3_RX_PTYPE_ENTRY(135, 1, UNNECESSARY, IPV6),
+ HNS3_RX_PTYPE_ENTRY(136, 1, UNNECESSARY, IPV6),
+ HNS3_RX_PTYPE_ENTRY(137, 1, UNNECESSARY, IPV6),
+ HNS3_RX_PTYPE_ENTRY(138, 0, COMPLETE, IPV6),
+ HNS3_RX_PTYPE_ENTRY(139, 0, COMPLETE, IPV6),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(140),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(141),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(142),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(143),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(144),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(145),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(146),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(147),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(148),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(149),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(150),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(151),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(152),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(153),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(154),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(155),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(156),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(157),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(158),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(159),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(160),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(161),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(162),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(163),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(164),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(165),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(166),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(167),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(168),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(169),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(170),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(171),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(172),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(173),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(174),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(175),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(176),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(177),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(178),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(179),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(180),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(181),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(182),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(183),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(184),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(185),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(186),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(187),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(188),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(189),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(190),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(191),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(192),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(193),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(194),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(195),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(196),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(197),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(198),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(199),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(200),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(201),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(202),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(203),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(204),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(205),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(206),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(207),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(208),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(209),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(210),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(211),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(212),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(213),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(214),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(215),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(216),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(217),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(218),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(219),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(220),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(221),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(222),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(223),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(224),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(225),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(226),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(227),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(228),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(229),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(230),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(231),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(232),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(233),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(234),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(235),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(236),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(237),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(238),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(239),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(240),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(241),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(242),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(243),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(244),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(245),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(246),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(247),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(248),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(249),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(250),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(251),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(252),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(253),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(254),
+ HNS3_RX_PTYPE_UNUSED_ENTRY(255),
+};
+
+#define HNS3_INVALID_PTYPE \
+ ARRAY_SIZE(hns3_rx_ptype_tbl)
+
static irqreturn_t hns3_irq_handle(int irq, void *vector)
{
struct hns3_enet_tqp_vector *tqp_vector = vector;
napi_schedule_irqoff(&tqp_vector->napi);
+ tqp_vector->event_cnt++;
return IRQ_HANDLED;
}
disable_irq(tqp_vector->vector_irq);
napi_disable(&tqp_vector->napi);
+ cancel_work_sync(&tqp_vector->rx_group.dim.work);
+ cancel_work_sync(&tqp_vector->tx_group.dim.work);
}
void hns3_set_vector_coalesce_rl(struct hns3_enet_tqp_vector *tqp_vector,
struct hnae3_ae_dev *ae_dev = pci_get_drvdata(priv->ae_handle->pdev);
struct hns3_enet_coalesce *tx_coal = &tqp_vector->tx_group.coal;
struct hns3_enet_coalesce *rx_coal = &tqp_vector->rx_group.coal;
+ struct hns3_enet_coalesce *ptx_coal = &priv->tx_coal;
+ struct hns3_enet_coalesce *prx_coal = &priv->rx_coal;
- /* initialize the configuration for interrupt coalescing.
- * 1. GL (Interrupt Gap Limiter)
- * 2. RL (Interrupt Rate Limiter)
- * 3. QL (Interrupt Quantity Limiter)
- *
- * Default: enable interrupt coalescing self-adaptive and GL
- */
- tx_coal->adapt_enable = 1;
- rx_coal->adapt_enable = 1;
+ tx_coal->adapt_enable = ptx_coal->adapt_enable;
+ rx_coal->adapt_enable = prx_coal->adapt_enable;
- tx_coal->int_gl = HNS3_INT_GL_50K;
- rx_coal->int_gl = HNS3_INT_GL_50K;
+ tx_coal->int_gl = ptx_coal->int_gl;
+ rx_coal->int_gl = prx_coal->int_gl;
- rx_coal->flow_level = HNS3_FLOW_LOW;
- tx_coal->flow_level = HNS3_FLOW_LOW;
+ rx_coal->flow_level = prx_coal->flow_level;
+ tx_coal->flow_level = ptx_coal->flow_level;
/* device version above V3(include V3), GL can configure 1us
* unit, so uses 1us unit.
rx_coal->ql_enable = 1;
tx_coal->int_ql_max = ae_dev->dev_specs.int_ql_max;
rx_coal->int_ql_max = ae_dev->dev_specs.int_ql_max;
- tx_coal->int_ql = HNS3_INT_QL_DEFAULT_CFG;
- rx_coal->int_ql = HNS3_INT_QL_DEFAULT_CFG;
+ tx_coal->int_ql = ptx_coal->int_ql;
+ rx_coal->int_ql = prx_coal->int_ql;
}
}
return 0;
}
-static u16 hns3_get_max_available_channels(struct hnae3_handle *h)
+u16 hns3_get_max_available_channels(struct hnae3_handle *h)
{
u16 alloc_tqps, max_rss_size, rss_size;
{
u8 flags = 0;
- if (netdev->flags & IFF_PROMISC) {
+ if (netdev->flags & IFF_PROMISC)
flags = HNAE3_USER_UPE | HNAE3_USER_MPE | HNAE3_BPE;
- } else {
- flags |= HNAE3_VLAN_FLTR;
- if (netdev->flags & IFF_ALLMULTI)
- flags |= HNAE3_USER_MPE;
- }
+ else if (netdev->flags & IFF_ALLMULTI)
+ flags = HNAE3_USER_MPE;
return flags;
}
ops->request_update_promisc_mode(handle);
}
-void hns3_enable_vlan_filter(struct net_device *netdev, bool enable)
+static u32 hns3_tx_spare_space(struct hns3_enet_ring *ring)
{
- struct hns3_nic_priv *priv = netdev_priv(netdev);
- struct hnae3_handle *h = priv->ae_handle;
- struct hnae3_ae_dev *ae_dev = pci_get_drvdata(h->pdev);
- bool last_state;
+ struct hns3_tx_spare *tx_spare = ring->tx_spare;
+ u32 ntc, ntu;
- if (ae_dev->dev_version >= HNAE3_DEVICE_VERSION_V2 &&
- h->ae_algo->ops->enable_vlan_filter) {
- last_state = h->netdev_flags & HNAE3_VLAN_FLTR ? true : false;
- if (enable != last_state) {
- netdev_info(netdev,
- "%s vlan filter\n",
- enable ? "enable" : "disable");
- h->ae_algo->ops->enable_vlan_filter(h, enable);
+ /* This smp_load_acquire() pairs with smp_store_release() in
+ * hns3_tx_spare_update() called in tx desc cleaning process.
+ */
+ ntc = smp_load_acquire(&tx_spare->last_to_clean);
+ ntu = tx_spare->next_to_use;
+
+ if (ntc > ntu)
+ return ntc - ntu - 1;
+
+ /* The free tx buffer is divided into two part, so pick the
+ * larger one.
+ */
+ return (ntc > (tx_spare->len - ntu) ? ntc :
+ (tx_spare->len - ntu)) - 1;
+}
+
+static void hns3_tx_spare_update(struct hns3_enet_ring *ring)
+{
+ struct hns3_tx_spare *tx_spare = ring->tx_spare;
+
+ if (!tx_spare ||
+ tx_spare->last_to_clean == tx_spare->next_to_clean)
+ return;
+
+ /* This smp_store_release() pairs with smp_load_acquire() in
+ * hns3_tx_spare_space() called in xmit process.
+ */
+ smp_store_release(&tx_spare->last_to_clean,
+ tx_spare->next_to_clean);
+}
+
+static bool hns3_can_use_tx_bounce(struct hns3_enet_ring *ring,
+ struct sk_buff *skb,
+ u32 space)
+{
+ u32 len = skb->len <= ring->tx_copybreak ? skb->len :
+ skb_headlen(skb);
+
+ if (len > ring->tx_copybreak)
+ return false;
+
+ if (ALIGN(len, dma_get_cache_alignment()) > space) {
+ u64_stats_update_begin(&ring->syncp);
+ ring->stats.tx_spare_full++;
+ u64_stats_update_end(&ring->syncp);
+ return false;
+ }
+
+ return true;
+}
+
+static bool hns3_can_use_tx_sgl(struct hns3_enet_ring *ring,
+ struct sk_buff *skb,
+ u32 space)
+{
+ if (skb->len <= ring->tx_copybreak || !tx_sgl ||
+ (!skb_has_frag_list(skb) &&
+ skb_shinfo(skb)->nr_frags < tx_sgl))
+ return false;
+
+ if (space < HNS3_MAX_SGL_SIZE) {
+ u64_stats_update_begin(&ring->syncp);
+ ring->stats.tx_spare_full++;
+ u64_stats_update_end(&ring->syncp);
+ return false;
+ }
+
+ return true;
+}
+
+static void hns3_init_tx_spare_buffer(struct hns3_enet_ring *ring)
+{
+ struct hns3_tx_spare *tx_spare;
+ struct page *page;
+ u32 alloc_size;
+ dma_addr_t dma;
+ int order;
+
+ alloc_size = tx_spare_buf_size ? tx_spare_buf_size :
+ ring->tqp->handle->kinfo.tx_spare_buf_size;
+ if (!alloc_size)
+ return;
+
+ order = get_order(alloc_size);
+ tx_spare = devm_kzalloc(ring_to_dev(ring), sizeof(*tx_spare),
+ GFP_KERNEL);
+ if (!tx_spare) {
+ /* The driver still work without the tx spare buffer */
+ dev_warn(ring_to_dev(ring), "failed to allocate hns3_tx_spare\n");
+ return;
+ }
+
+ page = alloc_pages_node(dev_to_node(ring_to_dev(ring)),
+ GFP_KERNEL, order);
+ if (!page) {
+ dev_warn(ring_to_dev(ring), "failed to allocate tx spare pages\n");
+ devm_kfree(ring_to_dev(ring), tx_spare);
+ return;
+ }
+
+ dma = dma_map_page(ring_to_dev(ring), page, 0,
+ PAGE_SIZE << order, DMA_TO_DEVICE);
+ if (dma_mapping_error(ring_to_dev(ring), dma)) {
+ dev_warn(ring_to_dev(ring), "failed to map pages for tx spare\n");
+ put_page(page);
+ devm_kfree(ring_to_dev(ring), tx_spare);
+ return;
+ }
+
+ tx_spare->dma = dma;
+ tx_spare->buf = page_address(page);
+ tx_spare->len = PAGE_SIZE << order;
+ ring->tx_spare = tx_spare;
+}
+
+/* Use hns3_tx_spare_space() to make sure there is enough buffer
+ * before calling below function to allocate tx buffer.
+ */
+static void *hns3_tx_spare_alloc(struct hns3_enet_ring *ring,
+ unsigned int size, dma_addr_t *dma,
+ u32 *cb_len)
+{
+ struct hns3_tx_spare *tx_spare = ring->tx_spare;
+ u32 ntu = tx_spare->next_to_use;
+
+ size = ALIGN(size, dma_get_cache_alignment());
+ *cb_len = size;
+
+ /* Tx spare buffer wraps back here because the end of
+ * freed tx buffer is not enough.
+ */
+ if (ntu + size > tx_spare->len) {
+ *cb_len += (tx_spare->len - ntu);
+ ntu = 0;
+ }
+
+ tx_spare->next_to_use = ntu + size;
+ if (tx_spare->next_to_use == tx_spare->len)
+ tx_spare->next_to_use = 0;
+
+ *dma = tx_spare->dma + ntu;
+
+ return tx_spare->buf + ntu;
+}
+
+static void hns3_tx_spare_rollback(struct hns3_enet_ring *ring, u32 len)
+{
+ struct hns3_tx_spare *tx_spare = ring->tx_spare;
+
+ if (len > tx_spare->next_to_use) {
+ len -= tx_spare->next_to_use;
+ tx_spare->next_to_use = tx_spare->len - len;
+ } else {
+ tx_spare->next_to_use -= len;
+ }
+}
+
+static void hns3_tx_spare_reclaim_cb(struct hns3_enet_ring *ring,
+ struct hns3_desc_cb *cb)
+{
+ struct hns3_tx_spare *tx_spare = ring->tx_spare;
+ u32 ntc = tx_spare->next_to_clean;
+ u32 len = cb->length;
+
+ tx_spare->next_to_clean += len;
+
+ if (tx_spare->next_to_clean >= tx_spare->len) {
+ tx_spare->next_to_clean -= tx_spare->len;
+
+ if (tx_spare->next_to_clean) {
+ ntc = 0;
+ len = tx_spare->next_to_clean;
}
}
+
+ /* This tx spare buffer is only really reclaimed after calling
+ * hns3_tx_spare_update(), so it is still safe to use the info in
+ * the tx buffer to do the dma sync or sg unmapping after
+ * tx_spare->next_to_clean is moved forword.
+ */
+ if (cb->type & (DESC_TYPE_BOUNCE_HEAD | DESC_TYPE_BOUNCE_ALL)) {
+ dma_addr_t dma = tx_spare->dma + ntc;
+
+ dma_sync_single_for_cpu(ring_to_dev(ring), dma, len,
+ DMA_TO_DEVICE);
+ } else {
+ struct sg_table *sgt = tx_spare->buf + ntc;
+
+ dma_unmap_sg(ring_to_dev(ring), sgt->sgl, sgt->orig_nents,
+ DMA_TO_DEVICE);
+ }
}
static int hns3_set_tso(struct sk_buff *skb, u32 *paylen_fdop_ol4cs,
l4.udp->dest == htons(4790))))
return false;
- skb_checksum_help(skb);
-
return true;
}
/* the stack computes the IP header already,
* driver calculate l4 checksum when not TSO.
*/
- skb_checksum_help(skb);
- return 0;
+ return skb_checksum_help(skb);
}
hns3_set_outer_l2l3l4(skb, ol4_proto, ol_type_vlan_len_msec);
break;
case IPPROTO_UDP:
if (hns3_tunnel_csum_bug(skb))
- break;
+ return skb_checksum_help(skb);
hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L4CS_B, 1);
hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L4T_S,
/* the stack computes the IP header already,
* driver calculate l4 checksum when not TSO.
*/
- skb_checksum_help(skb);
- return 0;
+ return skb_checksum_help(skb);
}
return 0;
return 0;
}
-static int hns3_fill_desc(struct hns3_enet_ring *ring, void *priv,
- unsigned int size, enum hns_desc_type type)
+static int hns3_fill_desc(struct hns3_enet_ring *ring, dma_addr_t dma,
+ unsigned int size)
{
#define HNS3_LIKELY_BD_NUM 1
- struct hns3_desc_cb *desc_cb = &ring->desc_cb[ring->next_to_use];
struct hns3_desc *desc = &ring->desc[ring->next_to_use];
- struct device *dev = ring_to_dev(ring);
- skb_frag_t *frag;
unsigned int frag_buf_num;
int k, sizeoflast;
- dma_addr_t dma;
-
- if (type == DESC_TYPE_FRAGLIST_SKB ||
- type == DESC_TYPE_SKB) {
- struct sk_buff *skb = (struct sk_buff *)priv;
-
- dma = dma_map_single(dev, skb->data, size, DMA_TO_DEVICE);
- } else {
- frag = (skb_frag_t *)priv;
- dma = skb_frag_dma_map(dev, frag, 0, size, DMA_TO_DEVICE);
- }
-
- if (unlikely(dma_mapping_error(dev, dma))) {
- u64_stats_update_begin(&ring->syncp);
- ring->stats.sw_err_cnt++;
- u64_stats_update_end(&ring->syncp);
- return -ENOMEM;
- }
-
- desc_cb->priv = priv;
- desc_cb->length = size;
- desc_cb->dma = dma;
- desc_cb->type = type;
if (likely(size <= HNS3_MAX_BD_SIZE)) {
desc->addr = cpu_to_le64(dma);
return frag_buf_num;
}
+static int hns3_map_and_fill_desc(struct hns3_enet_ring *ring, void *priv,
+ unsigned int type)
+{
+ struct hns3_desc_cb *desc_cb = &ring->desc_cb[ring->next_to_use];
+ struct device *dev = ring_to_dev(ring);
+ unsigned int size;
+ dma_addr_t dma;
+
+ if (type & (DESC_TYPE_FRAGLIST_SKB | DESC_TYPE_SKB)) {
+ struct sk_buff *skb = (struct sk_buff *)priv;
+
+ size = skb_headlen(skb);
+ if (!size)
+ return 0;
+
+ dma = dma_map_single(dev, skb->data, size, DMA_TO_DEVICE);
+ } else if (type & DESC_TYPE_BOUNCE_HEAD) {
+ /* Head data has been filled in hns3_handle_tx_bounce(),
+ * just return 0 here.
+ */
+ return 0;
+ } else {
+ skb_frag_t *frag = (skb_frag_t *)priv;
+
+ size = skb_frag_size(frag);
+ if (!size)
+ return 0;
+
+ dma = skb_frag_dma_map(dev, frag, 0, size, DMA_TO_DEVICE);
+ }
+
+ if (unlikely(dma_mapping_error(dev, dma))) {
+ u64_stats_update_begin(&ring->syncp);
+ ring->stats.sw_err_cnt++;
+ u64_stats_update_end(&ring->syncp);
+ return -ENOMEM;
+ }
+
+ desc_cb->priv = priv;
+ desc_cb->length = size;
+ desc_cb->dma = dma;
+ desc_cb->type = type;
+
+ return hns3_fill_desc(ring, dma, size);
+}
+
static unsigned int hns3_skb_bd_num(struct sk_buff *skb, unsigned int *bd_size,
unsigned int bd_num)
{
for (i = 0; i < ring->desc_num; i++) {
struct hns3_desc *desc = &ring->desc[ring->next_to_use];
+ struct hns3_desc_cb *desc_cb;
memset(desc, 0, sizeof(*desc));
/* rollback one */
ring_ptr_move_bw(ring, next_to_use);
- if (!ring->desc_cb[ring->next_to_use].dma)
+ desc_cb = &ring->desc_cb[ring->next_to_use];
+
+ if (!desc_cb->dma)
continue;
/* unmap the descriptor dma address */
- if (ring->desc_cb[ring->next_to_use].type == DESC_TYPE_SKB ||
- ring->desc_cb[ring->next_to_use].type ==
- DESC_TYPE_FRAGLIST_SKB)
- dma_unmap_single(dev,
- ring->desc_cb[ring->next_to_use].dma,
- ring->desc_cb[ring->next_to_use].length,
- DMA_TO_DEVICE);
- else if (ring->desc_cb[ring->next_to_use].length)
- dma_unmap_page(dev,
- ring->desc_cb[ring->next_to_use].dma,
- ring->desc_cb[ring->next_to_use].length,
+ if (desc_cb->type & (DESC_TYPE_SKB | DESC_TYPE_FRAGLIST_SKB))
+ dma_unmap_single(dev, desc_cb->dma, desc_cb->length,
+ DMA_TO_DEVICE);
+ else if (desc_cb->type &
+ (DESC_TYPE_BOUNCE_HEAD | DESC_TYPE_BOUNCE_ALL))
+ hns3_tx_spare_rollback(ring, desc_cb->length);
+ else if (desc_cb->length)
+ dma_unmap_page(dev, desc_cb->dma, desc_cb->length,
DMA_TO_DEVICE);
- ring->desc_cb[ring->next_to_use].length = 0;
- ring->desc_cb[ring->next_to_use].dma = 0;
- ring->desc_cb[ring->next_to_use].type = DESC_TYPE_UNKNOWN;
+ desc_cb->length = 0;
+ desc_cb->dma = 0;
+ desc_cb->type = DESC_TYPE_UNKNOWN;
}
}
static int hns3_fill_skb_to_desc(struct hns3_enet_ring *ring,
- struct sk_buff *skb, enum hns_desc_type type)
+ struct sk_buff *skb, unsigned int type)
{
- unsigned int size = skb_headlen(skb);
struct sk_buff *frag_skb;
int i, ret, bd_num = 0;
- if (size) {
- ret = hns3_fill_desc(ring, skb, size, type);
- if (unlikely(ret < 0))
- return ret;
+ ret = hns3_map_and_fill_desc(ring, skb, type);
+ if (unlikely(ret < 0))
+ return ret;
- bd_num += ret;
- }
+ bd_num += ret;
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- size = skb_frag_size(frag);
- if (!size)
- continue;
-
- ret = hns3_fill_desc(ring, frag, size, DESC_TYPE_PAGE);
+ ret = hns3_map_and_fill_desc(ring, frag, DESC_TYPE_PAGE);
if (unlikely(ret < 0))
return ret;
WRITE_ONCE(ring->last_to_use, ring->next_to_use);
}
+static void hns3_tsyn(struct net_device *netdev, struct sk_buff *skb,
+ struct hns3_desc *desc)
+{
+ struct hnae3_handle *h = hns3_get_handle(netdev);
+
+ if (!(h->ae_algo->ops->set_tx_hwts_info &&
+ h->ae_algo->ops->set_tx_hwts_info(h, skb)))
+ return;
+
+ desc->tx.bdtp_fe_sc_vld_ra_ri |= cpu_to_le16(BIT(HNS3_TXD_TSYN_B));
+}
+
+static int hns3_handle_tx_bounce(struct hns3_enet_ring *ring,
+ struct sk_buff *skb)
+{
+ struct hns3_desc_cb *desc_cb = &ring->desc_cb[ring->next_to_use];
+ unsigned int type = DESC_TYPE_BOUNCE_HEAD;
+ unsigned int size = skb_headlen(skb);
+ dma_addr_t dma;
+ int bd_num = 0;
+ u32 cb_len;
+ void *buf;
+ int ret;
+
+ if (skb->len <= ring->tx_copybreak) {
+ size = skb->len;
+ type = DESC_TYPE_BOUNCE_ALL;
+ }
+
+ /* hns3_can_use_tx_bounce() is called to ensure the below
+ * function can always return the tx buffer.
+ */
+ buf = hns3_tx_spare_alloc(ring, size, &dma, &cb_len);
+
+ ret = skb_copy_bits(skb, 0, buf, size);
+ if (unlikely(ret < 0)) {
+ hns3_tx_spare_rollback(ring, cb_len);
+ u64_stats_update_begin(&ring->syncp);
+ ring->stats.copy_bits_err++;
+ u64_stats_update_end(&ring->syncp);
+ return ret;
+ }
+
+ desc_cb->priv = skb;
+ desc_cb->length = cb_len;
+ desc_cb->dma = dma;
+ desc_cb->type = type;
+
+ bd_num += hns3_fill_desc(ring, dma, size);
+
+ if (type == DESC_TYPE_BOUNCE_HEAD) {
+ ret = hns3_fill_skb_to_desc(ring, skb,
+ DESC_TYPE_BOUNCE_HEAD);
+ if (unlikely(ret < 0))
+ return ret;
+
+ bd_num += ret;
+ }
+
+ dma_sync_single_for_device(ring_to_dev(ring), dma, size,
+ DMA_TO_DEVICE);
+
+ u64_stats_update_begin(&ring->syncp);
+ ring->stats.tx_bounce++;
+ u64_stats_update_end(&ring->syncp);
+ return bd_num;
+}
+
+static int hns3_handle_tx_sgl(struct hns3_enet_ring *ring,
+ struct sk_buff *skb)
+{
+ struct hns3_desc_cb *desc_cb = &ring->desc_cb[ring->next_to_use];
+ u32 nfrag = skb_shinfo(skb)->nr_frags + 1;
+ struct sg_table *sgt;
+ int i, bd_num = 0;
+ dma_addr_t dma;
+ u32 cb_len;
+ int nents;
+
+ if (skb_has_frag_list(skb))
+ nfrag = HNS3_MAX_TSO_BD_NUM;
+
+ /* hns3_can_use_tx_sgl() is called to ensure the below
+ * function can always return the tx buffer.
+ */
+ sgt = hns3_tx_spare_alloc(ring, HNS3_SGL_SIZE(nfrag),
+ &dma, &cb_len);
+
+ /* scatterlist follows by the sg table */
+ sgt->sgl = (struct scatterlist *)(sgt + 1);
+ sg_init_table(sgt->sgl, nfrag);
+ nents = skb_to_sgvec(skb, sgt->sgl, 0, skb->len);
+ if (unlikely(nents < 0)) {
+ hns3_tx_spare_rollback(ring, cb_len);
+ u64_stats_update_begin(&ring->syncp);
+ ring->stats.skb2sgl_err++;
+ u64_stats_update_end(&ring->syncp);
+ return -ENOMEM;
+ }
+
+ sgt->orig_nents = nents;
+ sgt->nents = dma_map_sg(ring_to_dev(ring), sgt->sgl, sgt->orig_nents,
+ DMA_TO_DEVICE);
+ if (unlikely(!sgt->nents)) {
+ hns3_tx_spare_rollback(ring, cb_len);
+ u64_stats_update_begin(&ring->syncp);
+ ring->stats.map_sg_err++;
+ u64_stats_update_end(&ring->syncp);
+ return -ENOMEM;
+ }
+
+ desc_cb->priv = skb;
+ desc_cb->length = cb_len;
+ desc_cb->dma = dma;
+ desc_cb->type = DESC_TYPE_SGL_SKB;
+
+ for (i = 0; i < sgt->nents; i++)
+ bd_num += hns3_fill_desc(ring, sg_dma_address(sgt->sgl + i),
+ sg_dma_len(sgt->sgl + i));
+
+ u64_stats_update_begin(&ring->syncp);
+ ring->stats.tx_sgl++;
+ u64_stats_update_end(&ring->syncp);
+
+ return bd_num;
+}
+
+static int hns3_handle_desc_filling(struct hns3_enet_ring *ring,
+ struct sk_buff *skb)
+{
+ u32 space;
+
+ if (!ring->tx_spare)
+ goto out;
+
+ space = hns3_tx_spare_space(ring);
+
+ if (hns3_can_use_tx_sgl(ring, skb, space))
+ return hns3_handle_tx_sgl(ring, skb);
+
+ if (hns3_can_use_tx_bounce(ring, skb, space))
+ return hns3_handle_tx_bounce(ring, skb);
+
+out:
+ return hns3_fill_skb_to_desc(ring, skb, DESC_TYPE_SKB);
+}
+
netdev_tx_t hns3_nic_net_xmit(struct sk_buff *skb, struct net_device *netdev)
{
struct hns3_nic_priv *priv = netdev_priv(netdev);
* zero, which is unlikely, and 'ret > 0' means how many tx desc
* need to be notified to the hw.
*/
- ret = hns3_fill_skb_to_desc(ring, skb, DESC_TYPE_SKB);
+ ret = hns3_handle_desc_filling(ring, skb);
if (unlikely(ret <= 0))
goto fill_err;
pre_ntu = ring->next_to_use ? (ring->next_to_use - 1) :
(ring->desc_num - 1);
+
+ if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP))
+ hns3_tsyn(netdev, skb, &ring->desc[pre_ntu]);
+
ring->desc[pre_ntu].tx.bdtp_fe_sc_vld_ra_ri |=
cpu_to_le16(BIT(HNS3_TXD_FE_B));
trace_hns3_tx_desc(ring, pre_ntu);
+ skb_tx_timestamp(skb);
+
/* Complete translate all packets */
dev_queue = netdev_get_tx_queue(netdev, ring->queue_index);
doorbell = __netdev_tx_sent_queue(dev_queue, desc_cb->send_bytes,
return -EINVAL;
}
+ if ((changed & NETIF_F_HW_VLAN_CTAG_FILTER) &&
+ h->ae_algo->ops->enable_vlan_filter) {
+ enable = !!(features & NETIF_F_HW_VLAN_CTAG_FILTER);
+ ret = h->ae_algo->ops->enable_vlan_filter(h, enable);
+ if (ret)
+ return ret;
+ }
+
netdev->features = features;
return 0;
}
tx_drop += ring->stats.tx_tso_err;
tx_drop += ring->stats.over_max_recursion;
tx_drop += ring->stats.hw_limitation;
+ tx_drop += ring->stats.copy_bits_err;
+ tx_drop += ring->stats.skb2sgl_err;
+ tx_drop += ring->stats.map_sg_err;
tx_errors += ring->stats.sw_err_cnt;
tx_errors += ring->stats.tx_vlan_err;
tx_errors += ring->stats.tx_l4_proto_err;
tx_errors += ring->stats.tx_tso_err;
tx_errors += ring->stats.over_max_recursion;
tx_errors += ring->stats.hw_limitation;
+ tx_errors += ring->stats.copy_bits_err;
+ tx_errors += ring->stats.skb2sgl_err;
+ tx_errors += ring->stats.map_sg_err;
} while (u64_stats_fetch_retry_irq(&ring->syncp, start));
/* fetch the rx stats */
netdev->hw_features |= NETIF_F_HW_TC;
netdev->features |= NETIF_F_HW_TC;
}
+
+ if (test_bit(HNAE3_DEV_SUPPORT_VLAN_FLTR_MDF_B, ae_dev->caps))
+ netdev->hw_features |= NETIF_F_HW_VLAN_CTAG_FILTER;
}
static int hns3_alloc_buffer(struct hns3_enet_ring *ring,
static void hns3_free_buffer(struct hns3_enet_ring *ring,
struct hns3_desc_cb *cb, int budget)
{
- if (cb->type == DESC_TYPE_SKB)
+ if (cb->type & (DESC_TYPE_SKB | DESC_TYPE_BOUNCE_HEAD |
+ DESC_TYPE_BOUNCE_ALL | DESC_TYPE_SGL_SKB))
napi_consume_skb(cb->priv, budget);
else if (!HNAE3_IS_TX_RING(ring) && cb->pagecnt_bias)
__page_frag_cache_drain(cb->priv, cb->pagecnt_bias);
static void hns3_unmap_buffer(struct hns3_enet_ring *ring,
struct hns3_desc_cb *cb)
{
- if (cb->type == DESC_TYPE_SKB || cb->type == DESC_TYPE_FRAGLIST_SKB)
+ if (cb->type & (DESC_TYPE_SKB | DESC_TYPE_FRAGLIST_SKB))
dma_unmap_single(ring_to_dev(ring), cb->dma, cb->length,
ring_to_dma_dir(ring));
- else if (cb->length)
+ else if ((cb->type & DESC_TYPE_PAGE) && cb->length)
dma_unmap_page(ring_to_dev(ring), cb->dma, cb->length,
ring_to_dma_dir(ring));
+ else if (cb->type & (DESC_TYPE_BOUNCE_ALL | DESC_TYPE_BOUNCE_HEAD |
+ DESC_TYPE_SGL_SKB))
+ hns3_tx_spare_reclaim_cb(ring, cb);
}
static void hns3_buffer_detach(struct hns3_enet_ring *ring, int i)
desc_cb = &ring->desc_cb[ntc];
- if (desc_cb->type == DESC_TYPE_SKB) {
+ if (desc_cb->type & (DESC_TYPE_SKB | DESC_TYPE_BOUNCE_ALL |
+ DESC_TYPE_BOUNCE_HEAD |
+ DESC_TYPE_SGL_SKB)) {
(*pkts)++;
(*bytes) += desc_cb->send_bytes;
}
* ring_space called by hns3_nic_net_xmit.
*/
smp_store_release(&ring->next_to_clean, ntc);
+
+ hns3_tx_spare_update(ring);
+
return true;
}
static bool hns3_can_reuse_page(struct hns3_desc_cb *cb)
{
- return (page_count(cb->priv) - cb->pagecnt_bias) == 1;
+ return page_count(cb->priv) == cb->pagecnt_bias;
}
static void hns3_nic_reuse_page(struct sk_buff *skb, int i,
struct hns3_desc_cb *desc_cb)
{
struct hns3_desc *desc = &ring->desc[ring->next_to_clean];
+ u32 frag_offset = desc_cb->page_offset + pull_len;
int size = le16_to_cpu(desc->rx.size);
u32 truesize = hns3_buf_size(ring);
+ u32 frag_size = size - pull_len;
+ bool reused;
- desc_cb->pagecnt_bias--;
- skb_add_rx_frag(skb, i, desc_cb->priv, desc_cb->page_offset + pull_len,
- size - pull_len, truesize);
+ /* Avoid re-using remote or pfmem page */
+ if (unlikely(!dev_page_is_reusable(desc_cb->priv)))
+ goto out;
- /* Avoid re-using remote and pfmemalloc pages, or the stack is still
- * using the page when page_offset rollback to zero, flag default
- * unreuse
+ reused = hns3_can_reuse_page(desc_cb);
+
+ /* Rx page can be reused when:
+ * 1. Rx page is only owned by the driver when page_offset
+ * is zero, which means 0 @ truesize will be used by
+ * stack after skb_add_rx_frag() is called, and the rest
+ * of rx page can be reused by driver.
+ * Or
+ * 2. Rx page is only owned by the driver when page_offset
+ * is non-zero, which means page_offset @ truesize will
+ * be used by stack after skb_add_rx_frag() is called,
+ * and 0 @ truesize can be reused by driver.
*/
- if (!dev_page_is_reusable(desc_cb->priv) ||
- (!desc_cb->page_offset && !hns3_can_reuse_page(desc_cb))) {
- __page_frag_cache_drain(desc_cb->priv, desc_cb->pagecnt_bias);
- return;
- }
+ if ((!desc_cb->page_offset && reused) ||
+ ((desc_cb->page_offset + truesize + truesize) <=
+ hns3_page_size(ring) && desc_cb->page_offset)) {
+ desc_cb->page_offset += truesize;
+ desc_cb->reuse_flag = 1;
+ } else if (desc_cb->page_offset && reused) {
+ desc_cb->page_offset = 0;
+ desc_cb->reuse_flag = 1;
+ } else if (frag_size <= ring->rx_copybreak) {
+ void *frag = napi_alloc_frag(frag_size);
- /* Move offset up to the next cache line */
- desc_cb->page_offset += truesize;
+ if (unlikely(!frag)) {
+ u64_stats_update_begin(&ring->syncp);
+ ring->stats.frag_alloc_err++;
+ u64_stats_update_end(&ring->syncp);
+
+ hns3_rl_err(ring_to_netdev(ring),
+ "failed to allocate rx frag\n");
+ goto out;
+ }
- if (desc_cb->page_offset + truesize <= hns3_page_size(ring)) {
- desc_cb->reuse_flag = 1;
- } else if (hns3_can_reuse_page(desc_cb)) {
desc_cb->reuse_flag = 1;
- desc_cb->page_offset = 0;
- } else if (desc_cb->pagecnt_bias) {
- __page_frag_cache_drain(desc_cb->priv, desc_cb->pagecnt_bias);
+ memcpy(frag, desc_cb->buf + frag_offset, frag_size);
+ skb_add_rx_frag(skb, i, virt_to_page(frag),
+ offset_in_page(frag), frag_size, frag_size);
+
+ u64_stats_update_begin(&ring->syncp);
+ ring->stats.frag_alloc++;
+ u64_stats_update_end(&ring->syncp);
return;
}
+out:
+ desc_cb->pagecnt_bias--;
+
if (unlikely(!desc_cb->pagecnt_bias)) {
page_ref_add(desc_cb->priv, USHRT_MAX);
desc_cb->pagecnt_bias = USHRT_MAX;
}
+
+ skb_add_rx_frag(skb, i, desc_cb->priv, frag_offset,
+ frag_size, truesize);
+
+ if (unlikely(!desc_cb->reuse_flag))
+ __page_frag_cache_drain(desc_cb->priv, desc_cb->pagecnt_bias);
}
static int hns3_gro_complete(struct sk_buff *skb, u32 l234info)
return 0;
}
-static void hns3_checksum_complete(struct hns3_enet_ring *ring,
- struct sk_buff *skb, u32 l234info)
+static bool hns3_checksum_complete(struct hns3_enet_ring *ring,
+ struct sk_buff *skb, u32 ptype, u16 csum)
{
- u32 lo, hi;
+ if (ptype == HNS3_INVALID_PTYPE ||
+ hns3_rx_ptype_tbl[ptype].ip_summed != CHECKSUM_COMPLETE)
+ return false;
u64_stats_update_begin(&ring->syncp);
ring->stats.csum_complete++;
u64_stats_update_end(&ring->syncp);
skb->ip_summed = CHECKSUM_COMPLETE;
- lo = hnae3_get_field(l234info, HNS3_RXD_L2_CSUM_L_M,
- HNS3_RXD_L2_CSUM_L_S);
- hi = hnae3_get_field(l234info, HNS3_RXD_L2_CSUM_H_M,
- HNS3_RXD_L2_CSUM_H_S);
- skb->csum = csum_unfold((__force __sum16)(lo | hi << 8));
+ skb->csum = csum_unfold((__force __sum16)csum);
+
+ return true;
}
-static void hns3_rx_checksum(struct hns3_enet_ring *ring, struct sk_buff *skb,
- u32 l234info, u32 bd_base_info, u32 ol_info)
+static void hns3_rx_handle_csum(struct sk_buff *skb, u32 l234info,
+ u32 ol_info, u32 ptype)
{
- struct net_device *netdev = ring_to_netdev(ring);
int l3_type, l4_type;
int ol4_type;
- skb->ip_summed = CHECKSUM_NONE;
-
- skb_checksum_none_assert(skb);
-
- if (!(netdev->features & NETIF_F_RXCSUM))
- return;
-
- if (l234info & BIT(HNS3_RXD_L2_CSUM_B)) {
- hns3_checksum_complete(ring, skb, l234info);
- return;
- }
-
- /* check if hardware has done checksum */
- if (!(bd_base_info & BIT(HNS3_RXD_L3L4P_B)))
- return;
-
- if (unlikely(l234info & (BIT(HNS3_RXD_L3E_B) | BIT(HNS3_RXD_L4E_B) |
- BIT(HNS3_RXD_OL3E_B) |
- BIT(HNS3_RXD_OL4E_B)))) {
- u64_stats_update_begin(&ring->syncp);
- ring->stats.l3l4_csum_err++;
- u64_stats_update_end(&ring->syncp);
+ if (ptype != HNS3_INVALID_PTYPE) {
+ skb->csum_level = hns3_rx_ptype_tbl[ptype].csum_level;
+ skb->ip_summed = hns3_rx_ptype_tbl[ptype].ip_summed;
return;
}
}
}
+static void hns3_rx_checksum(struct hns3_enet_ring *ring, struct sk_buff *skb,
+ u32 l234info, u32 bd_base_info, u32 ol_info,
+ u16 csum)
+{
+ struct net_device *netdev = ring_to_netdev(ring);
+ struct hns3_nic_priv *priv = netdev_priv(netdev);
+ u32 ptype = HNS3_INVALID_PTYPE;
+
+ skb->ip_summed = CHECKSUM_NONE;
+
+ skb_checksum_none_assert(skb);
+
+ if (!(netdev->features & NETIF_F_RXCSUM))
+ return;
+
+ if (test_bit(HNS3_NIC_STATE_RXD_ADV_LAYOUT_ENABLE, &priv->state))
+ ptype = hnae3_get_field(ol_info, HNS3_RXD_PTYPE_M,
+ HNS3_RXD_PTYPE_S);
+
+ if (hns3_checksum_complete(ring, skb, ptype, csum))
+ return;
+
+ /* check if hardware has done checksum */
+ if (!(bd_base_info & BIT(HNS3_RXD_L3L4P_B)))
+ return;
+
+ if (unlikely(l234info & (BIT(HNS3_RXD_L3E_B) | BIT(HNS3_RXD_L4E_B) |
+ BIT(HNS3_RXD_OL3E_B) |
+ BIT(HNS3_RXD_OL4E_B)))) {
+ u64_stats_update_begin(&ring->syncp);
+ ring->stats.l3l4_csum_err++;
+ u64_stats_update_end(&ring->syncp);
+
+ return;
+ }
+
+ hns3_rx_handle_csum(skb, l234info, ol_info, ptype);
+}
+
static void hns3_rx_skb(struct hns3_enet_ring *ring, struct sk_buff *skb)
{
if (skb_has_frag_list(skb))
static int hns3_set_gro_and_checksum(struct hns3_enet_ring *ring,
struct sk_buff *skb, u32 l234info,
- u32 bd_base_info, u32 ol_info)
+ u32 bd_base_info, u32 ol_info, u16 csum)
{
+ struct net_device *netdev = ring_to_netdev(ring);
+ struct hns3_nic_priv *priv = netdev_priv(netdev);
u32 l3_type;
skb_shinfo(skb)->gso_size = hnae3_get_field(bd_base_info,
HNS3_RXD_GRO_SIZE_S);
/* if there is no HW GRO, do not set gro params */
if (!skb_shinfo(skb)->gso_size) {
- hns3_rx_checksum(ring, skb, l234info, bd_base_info, ol_info);
+ hns3_rx_checksum(ring, skb, l234info, bd_base_info, ol_info,
+ csum);
return 0;
}
HNS3_RXD_GRO_COUNT_M,
HNS3_RXD_GRO_COUNT_S);
- l3_type = hnae3_get_field(l234info, HNS3_RXD_L3ID_M, HNS3_RXD_L3ID_S);
+ if (test_bit(HNS3_NIC_STATE_RXD_ADV_LAYOUT_ENABLE, &priv->state)) {
+ u32 ptype = hnae3_get_field(ol_info, HNS3_RXD_PTYPE_M,
+ HNS3_RXD_PTYPE_S);
+
+ l3_type = hns3_rx_ptype_tbl[ptype].l3_type;
+ } else {
+ l3_type = hnae3_get_field(l234info, HNS3_RXD_L3ID_M,
+ HNS3_RXD_L3ID_S);
+ }
+
if (l3_type == HNS3_L3_TYPE_IPV4)
skb_shinfo(skb)->gso_type = SKB_GSO_TCPV4;
else if (l3_type == HNS3_L3_TYPE_IPV6)
struct hns3_desc *desc;
unsigned int len;
int pre_ntc, ret;
+ u16 csum;
/* bdinfo handled below is only valid on the last BD of the
* current packet, and ring->next_to_clean indicates the first
bd_base_info = le32_to_cpu(desc->rx.bd_base_info);
l234info = le32_to_cpu(desc->rx.l234_info);
ol_info = le32_to_cpu(desc->rx.ol_info);
+ csum = le16_to_cpu(desc->csum);
+
+ if (unlikely(bd_base_info & BIT(HNS3_RXD_TS_VLD_B))) {
+ struct hnae3_handle *h = hns3_get_handle(netdev);
+ u32 nsec = le32_to_cpu(desc->ts_nsec);
+ u32 sec = le32_to_cpu(desc->ts_sec);
+
+ if (h->ae_algo->ops->get_rx_hwts)
+ h->ae_algo->ops->get_rx_hwts(h, skb, nsec, sec);
+ }
/* Based on hw strategy, the tag offloaded will be stored at
* ot_vlan_tag in two layer tag case, and stored at vlan_tag
/* This is needed in order to enable forwarding support */
ret = hns3_set_gro_and_checksum(ring, skb, l234info,
- bd_base_info, ol_info);
+ bd_base_info, ol_info, csum);
if (unlikely(ret)) {
u64_stats_update_begin(&ring->syncp);
ring->stats.rx_err_cnt++;
return recv_pkts;
}
-static bool hns3_get_new_flow_lvl(struct hns3_enet_ring_group *ring_group)
+static void hns3_update_rx_int_coalesce(struct hns3_enet_tqp_vector *tqp_vector)
{
-#define HNS3_RX_LOW_BYTE_RATE 10000
-#define HNS3_RX_MID_BYTE_RATE 20000
-#define HNS3_RX_ULTRA_PACKET_RATE 40
-
- enum hns3_flow_level_range new_flow_level;
- struct hns3_enet_tqp_vector *tqp_vector;
- int packets_per_msecs, bytes_per_msecs;
- u32 time_passed_ms;
-
- tqp_vector = ring_group->ring->tqp_vector;
- time_passed_ms =
- jiffies_to_msecs(jiffies - tqp_vector->last_jiffies);
- if (!time_passed_ms)
- return false;
-
- do_div(ring_group->total_packets, time_passed_ms);
- packets_per_msecs = ring_group->total_packets;
-
- do_div(ring_group->total_bytes, time_passed_ms);
- bytes_per_msecs = ring_group->total_bytes;
-
- new_flow_level = ring_group->coal.flow_level;
-
- /* Simple throttlerate management
- * 0-10MB/s lower (50000 ints/s)
- * 10-20MB/s middle (20000 ints/s)
- * 20-1249MB/s high (18000 ints/s)
- * > 40000pps ultra (8000 ints/s)
- */
- switch (new_flow_level) {
- case HNS3_FLOW_LOW:
- if (bytes_per_msecs > HNS3_RX_LOW_BYTE_RATE)
- new_flow_level = HNS3_FLOW_MID;
- break;
- case HNS3_FLOW_MID:
- if (bytes_per_msecs > HNS3_RX_MID_BYTE_RATE)
- new_flow_level = HNS3_FLOW_HIGH;
- else if (bytes_per_msecs <= HNS3_RX_LOW_BYTE_RATE)
- new_flow_level = HNS3_FLOW_LOW;
- break;
- case HNS3_FLOW_HIGH:
- case HNS3_FLOW_ULTRA:
- default:
- if (bytes_per_msecs <= HNS3_RX_MID_BYTE_RATE)
- new_flow_level = HNS3_FLOW_MID;
- break;
- }
-
- if (packets_per_msecs > HNS3_RX_ULTRA_PACKET_RATE &&
- &tqp_vector->rx_group == ring_group)
- new_flow_level = HNS3_FLOW_ULTRA;
-
- ring_group->total_bytes = 0;
- ring_group->total_packets = 0;
- ring_group->coal.flow_level = new_flow_level;
-
- return true;
-}
-
-static bool hns3_get_new_int_gl(struct hns3_enet_ring_group *ring_group)
-{
- struct hns3_enet_tqp_vector *tqp_vector;
- u16 new_int_gl;
-
- if (!ring_group->ring)
- return false;
-
- tqp_vector = ring_group->ring->tqp_vector;
- if (!tqp_vector->last_jiffies)
- return false;
-
- if (ring_group->total_packets == 0) {
- ring_group->coal.int_gl = HNS3_INT_GL_50K;
- ring_group->coal.flow_level = HNS3_FLOW_LOW;
- return true;
- }
-
- if (!hns3_get_new_flow_lvl(ring_group))
- return false;
+ struct hns3_enet_ring_group *rx_group = &tqp_vector->rx_group;
+ struct dim_sample sample = {};
- new_int_gl = ring_group->coal.int_gl;
- switch (ring_group->coal.flow_level) {
- case HNS3_FLOW_LOW:
- new_int_gl = HNS3_INT_GL_50K;
- break;
- case HNS3_FLOW_MID:
- new_int_gl = HNS3_INT_GL_20K;
- break;
- case HNS3_FLOW_HIGH:
- new_int_gl = HNS3_INT_GL_18K;
- break;
- case HNS3_FLOW_ULTRA:
- new_int_gl = HNS3_INT_GL_8K;
- break;
- default:
- break;
- }
+ if (!rx_group->coal.adapt_enable)
+ return;
- if (new_int_gl != ring_group->coal.int_gl) {
- ring_group->coal.int_gl = new_int_gl;
- return true;
- }
- return false;
+ dim_update_sample(tqp_vector->event_cnt, rx_group->total_packets,
+ rx_group->total_bytes, &sample);
+ net_dim(&rx_group->dim, sample);
}
-static void hns3_update_new_int_gl(struct hns3_enet_tqp_vector *tqp_vector)
+static void hns3_update_tx_int_coalesce(struct hns3_enet_tqp_vector *tqp_vector)
{
- struct hns3_enet_ring_group *rx_group = &tqp_vector->rx_group;
struct hns3_enet_ring_group *tx_group = &tqp_vector->tx_group;
- bool rx_update, tx_update;
+ struct dim_sample sample = {};
- /* update param every 1000ms */
- if (time_before(jiffies,
- tqp_vector->last_jiffies + msecs_to_jiffies(1000)))
+ if (!tx_group->coal.adapt_enable)
return;
- if (rx_group->coal.adapt_enable) {
- rx_update = hns3_get_new_int_gl(rx_group);
- if (rx_update)
- hns3_set_vector_coalesce_rx_gl(tqp_vector,
- rx_group->coal.int_gl);
- }
-
- if (tx_group->coal.adapt_enable) {
- tx_update = hns3_get_new_int_gl(tx_group);
- if (tx_update)
- hns3_set_vector_coalesce_tx_gl(tqp_vector,
- tx_group->coal.int_gl);
- }
-
- tqp_vector->last_jiffies = jiffies;
+ dim_update_sample(tqp_vector->event_cnt, tx_group->total_packets,
+ tx_group->total_bytes, &sample);
+ net_dim(&tx_group->dim, sample);
}
static int hns3_nic_common_poll(struct napi_struct *napi, int budget)
if (napi_complete(napi) &&
likely(!test_bit(HNS3_NIC_STATE_DOWN, &priv->state))) {
- hns3_update_new_int_gl(tqp_vector);
+ hns3_update_rx_int_coalesce(tqp_vector);
+ hns3_update_tx_int_coalesce(tqp_vector);
+
hns3_mask_vector_irq(tqp_vector, 1);
}
}
}
+static void hns3_rx_dim_work(struct work_struct *work)
+{
+ struct dim *dim = container_of(work, struct dim, work);
+ struct hns3_enet_ring_group *group = container_of(dim,
+ struct hns3_enet_ring_group, dim);
+ struct hns3_enet_tqp_vector *tqp_vector = group->ring->tqp_vector;
+ struct dim_cq_moder cur_moder =
+ net_dim_get_rx_moderation(dim->mode, dim->profile_ix);
+
+ hns3_set_vector_coalesce_rx_gl(group->ring->tqp_vector, cur_moder.usec);
+ tqp_vector->rx_group.coal.int_gl = cur_moder.usec;
+
+ if (cur_moder.pkts < tqp_vector->rx_group.coal.int_ql_max) {
+ hns3_set_vector_coalesce_rx_ql(tqp_vector, cur_moder.pkts);
+ tqp_vector->rx_group.coal.int_ql = cur_moder.pkts;
+ }
+
+ dim->state = DIM_START_MEASURE;
+}
+
+static void hns3_tx_dim_work(struct work_struct *work)
+{
+ struct dim *dim = container_of(work, struct dim, work);
+ struct hns3_enet_ring_group *group = container_of(dim,
+ struct hns3_enet_ring_group, dim);
+ struct hns3_enet_tqp_vector *tqp_vector = group->ring->tqp_vector;
+ struct dim_cq_moder cur_moder =
+ net_dim_get_tx_moderation(dim->mode, dim->profile_ix);
+
+ hns3_set_vector_coalesce_tx_gl(tqp_vector, cur_moder.usec);
+ tqp_vector->tx_group.coal.int_gl = cur_moder.usec;
+
+ if (cur_moder.pkts < tqp_vector->tx_group.coal.int_ql_max) {
+ hns3_set_vector_coalesce_tx_ql(tqp_vector, cur_moder.pkts);
+ tqp_vector->tx_group.coal.int_ql = cur_moder.pkts;
+ }
+
+ dim->state = DIM_START_MEASURE;
+}
+
+static void hns3_nic_init_dim(struct hns3_enet_tqp_vector *tqp_vector)
+{
+ INIT_WORK(&tqp_vector->rx_group.dim.work, hns3_rx_dim_work);
+ tqp_vector->rx_group.dim.mode = DIM_CQ_PERIOD_MODE_START_FROM_EQE;
+ INIT_WORK(&tqp_vector->tx_group.dim.work, hns3_tx_dim_work);
+ tqp_vector->tx_group.dim.mode = DIM_CQ_PERIOD_MODE_START_FROM_EQE;
+}
+
static int hns3_nic_init_vector_data(struct hns3_nic_priv *priv)
{
struct hnae3_handle *h = priv->ae_handle;
tqp_vector = &priv->tqp_vector[i];
hns3_vector_coalesce_init_hw(tqp_vector, priv);
tqp_vector->num_tqps = 0;
+ hns3_nic_init_dim(tqp_vector);
}
for (i = 0; i < h->kinfo.num_tqps; i++) {
return ret;
}
+static void hns3_nic_init_coal_cfg(struct hns3_nic_priv *priv)
+{
+ struct hnae3_ae_dev *ae_dev = pci_get_drvdata(priv->ae_handle->pdev);
+ struct hns3_enet_coalesce *tx_coal = &priv->tx_coal;
+ struct hns3_enet_coalesce *rx_coal = &priv->rx_coal;
+
+ /* initialize the configuration for interrupt coalescing.
+ * 1. GL (Interrupt Gap Limiter)
+ * 2. RL (Interrupt Rate Limiter)
+ * 3. QL (Interrupt Quantity Limiter)
+ *
+ * Default: enable interrupt coalescing self-adaptive and GL
+ */
+ tx_coal->adapt_enable = 1;
+ rx_coal->adapt_enable = 1;
+
+ tx_coal->int_gl = HNS3_INT_GL_50K;
+ rx_coal->int_gl = HNS3_INT_GL_50K;
+
+ rx_coal->flow_level = HNS3_FLOW_LOW;
+ tx_coal->flow_level = HNS3_FLOW_LOW;
+
+ if (ae_dev->dev_specs.int_ql_max) {
+ tx_coal->int_ql = HNS3_INT_QL_DEFAULT_CFG;
+ rx_coal->int_ql = HNS3_INT_QL_DEFAULT_CFG;
+ }
+}
+
static int hns3_nic_alloc_vector_data(struct hns3_nic_priv *priv)
{
struct hnae3_handle *h = priv->ae_handle;
ring = &priv->ring[q->tqp_index];
desc_num = priv->ae_handle->kinfo.num_tx_desc;
ring->queue_index = q->tqp_index;
+ ring->tx_copybreak = priv->tx_copybreak;
+ ring->last_to_use = 0;
} else {
ring = &priv->ring[q->tqp_index + queue_num];
desc_num = priv->ae_handle->kinfo.num_rx_desc;
ring->queue_index = q->tqp_index;
+ ring->rx_copybreak = priv->rx_copybreak;
}
hnae3_set_bit(ring->flag, HNAE3_RING_TYPE_B, ring_type);
ring->desc_num = desc_num;
ring->next_to_use = 0;
ring->next_to_clean = 0;
- ring->last_to_use = 0;
}
static void hns3_queue_to_ring(struct hnae3_queue *tqp,
ret = hns3_alloc_ring_buffers(ring);
if (ret)
goto out_with_desc;
+ } else {
+ hns3_init_tx_spare_buffer(ring);
}
return 0;
ring->next_to_use = 0;
ring->last_to_use = 0;
ring->pending_buf = 0;
- if (ring->skb) {
+ if (!HNAE3_IS_TX_RING(ring) && ring->skb) {
dev_kfree_skb_any(ring->skb);
ring->skb = NULL;
+ } else if (HNAE3_IS_TX_RING(ring) && ring->tx_spare) {
+ struct hns3_tx_spare *tx_spare = ring->tx_spare;
+
+ dma_unmap_page(ring_to_dev(ring), tx_spare->dma, tx_spare->len,
+ DMA_TO_DEVICE);
+ free_pages((unsigned long)tx_spare->buf,
+ get_order(tx_spare->len));
+ devm_kfree(ring_to_dev(ring), tx_spare);
+ ring->tx_spare = NULL;
}
}
goto out_get_ring_cfg;
}
+ hns3_nic_init_coal_cfg(priv);
+
ret = hns3_nic_alloc_vector_data(priv);
if (ret) {
ret = -ENOMEM;
if (ret)
goto out_init_phy;
- ret = register_netdev(netdev);
- if (ret) {
- dev_err(priv->dev, "probe register netdev fail!\n");
- goto out_reg_netdev_fail;
- }
-
/* the device can work without cpu rmap, only aRFS needs it */
ret = hns3_set_rx_cpu_rmap(netdev);
if (ret)
hns3_dcbnl_setup(handle);
- hns3_dbg_init(handle);
+ ret = hns3_dbg_init(handle);
+ if (ret) {
+ dev_err(priv->dev, "failed to init debugfs, ret = %d\n",
+ ret);
+ goto out_client_start;
+ }
netdev->max_mtu = HNS3_MAX_MTU(ae_dev->dev_specs.max_frm_size);
if (test_bit(HNAE3_DEV_SUPPORT_HW_TX_CSUM_B, ae_dev->caps))
set_bit(HNS3_NIC_STATE_HW_TX_CSUM_ENABLE, &priv->state);
+ if (hnae3_ae_dev_rxd_adv_layout_supported(ae_dev))
+ set_bit(HNS3_NIC_STATE_RXD_ADV_LAYOUT_ENABLE, &priv->state);
+
set_bit(HNS3_NIC_STATE_INITED, &priv->state);
if (ae_dev->dev_version >= HNAE3_DEVICE_VERSION_V3)
set_bit(HNAE3_PFLAG_LIMIT_PROMISC, &handle->supported_pflags);
+ ret = register_netdev(netdev);
+ if (ret) {
+ dev_err(priv->dev, "probe register netdev fail!\n");
+ goto out_reg_netdev_fail;
+ }
+
if (netif_msg_drv(handle))
hns3_info_show(priv);
return ret;
+out_reg_netdev_fail:
+ hns3_dbg_uninit(handle);
out_client_start:
hns3_free_rx_cpu_rmap(netdev);
hns3_nic_uninit_irq(priv);
out_init_irq_fail:
- unregister_netdev(netdev);
-out_reg_netdev_fail:
hns3_uninit_phy(netdev);
out_init_phy:
hns3_uninit_all_ring(priv);
return 0;
}
-static void hns3_store_coal(struct hns3_nic_priv *priv)
-{
- /* ethtool only support setting and querying one coal
- * configuration for now, so save the vector 0' coal
- * configuration here in order to restore it.
- */
- memcpy(&priv->tx_coal, &priv->tqp_vector[0].tx_group.coal,
- sizeof(struct hns3_enet_coalesce));
- memcpy(&priv->rx_coal, &priv->tqp_vector[0].rx_group.coal,
- sizeof(struct hns3_enet_coalesce));
-}
-
-static void hns3_restore_coal(struct hns3_nic_priv *priv)
-{
- u16 vector_num = priv->vector_num;
- int i;
-
- for (i = 0; i < vector_num; i++) {
- memcpy(&priv->tqp_vector[i].tx_group.coal, &priv->tx_coal,
- sizeof(struct hns3_enet_coalesce));
- memcpy(&priv->tqp_vector[i].rx_group.coal, &priv->rx_coal,
- sizeof(struct hns3_enet_coalesce));
- }
-}
-
static int hns3_reset_notify_down_enet(struct hnae3_handle *handle)
{
struct hnae3_knic_private_info *kinfo = &handle->kinfo;
if (ret)
goto err_put_ring;
- hns3_restore_coal(priv);
-
ret = hns3_nic_init_vector_data(priv);
if (ret)
goto err_dealloc_vector;
hns3_nic_uninit_vector_data(priv);
- hns3_store_coal(priv);
-
hns3_nic_dealloc_vector_data(priv);
hns3_uninit_all_ring(priv);
#ifndef __HNS3_ENET_H
#define __HNS3_ENET_H
+#include <linux/dim.h>
#include <linux/if_vlan.h>
#include "hnae3.h"
HNS3_NIC_STATE_SERVICE_SCHED,
HNS3_NIC_STATE2_RESET_REQUESTED,
HNS3_NIC_STATE_HW_TX_CSUM_ENABLE,
+ HNS3_NIC_STATE_RXD_ADV_LAYOUT_ENABLE,
HNS3_NIC_STATE_MAX
};
#define HNS3_RXD_STRP_TAGP_S 13
#define HNS3_RXD_STRP_TAGP_M (0x3 << HNS3_RXD_STRP_TAGP_S)
-#define HNS3_RXD_L2_CSUM_B 15
-#define HNS3_RXD_L2_CSUM_L_S 4
-#define HNS3_RXD_L2_CSUM_L_M (0xff << HNS3_RXD_L2_CSUM_L_S)
-#define HNS3_RXD_L2_CSUM_H_S 24
-#define HNS3_RXD_L2_CSUM_H_M (0xff << HNS3_RXD_L2_CSUM_H_S)
-
#define HNS3_RXD_L2E_B 16
#define HNS3_RXD_L3E_B 17
#define HNS3_RXD_L4E_B 18
#define HNS3_RXD_FBLI_S 14
#define HNS3_RXD_FBLI_M (0x3 << HNS3_RXD_FBLI_S)
+#define HNS3_RXD_PTYPE_S 4
+#define HNS3_RXD_PTYPE_M GENMASK(11, 4)
+
#define HNS3_RXD_BDTYPE_S 0
#define HNS3_RXD_BDTYPE_M (0xf << HNS3_RXD_BDTYPE_S)
#define HNS3_RXD_VLD_B 4
#define HNS3_RXD_LUM_B 9
#define HNS3_RXD_CRCP_B 10
#define HNS3_RXD_L3L4P_B 11
-#define HNS3_RXD_TSIND_S 12
-#define HNS3_RXD_TSIND_M (0x7 << HNS3_RXD_TSIND_S)
+#define HNS3_RXD_TSIDX_S 12
+#define HNS3_RXD_TSIDX_M (0x3 << HNS3_RXD_TSIDX_S)
+#define HNS3_RXD_TS_VLD_B 14
#define HNS3_RXD_LKBK_B 15
#define HNS3_RXD_GRO_SIZE_S 16
#define HNS3_RXD_GRO_SIZE_M (0x3fff << HNS3_RXD_GRO_SIZE_S)
/* hardware spec ring buffer format */
struct __packed hns3_desc {
- __le64 addr;
+ union {
+ __le64 addr;
+ __le16 csum;
+ struct {
+ __le32 ts_nsec;
+ __le32 ts_sec;
+ };
+ };
union {
struct {
__le16 vlan_tag;
};
};
+enum hns3_desc_type {
+ DESC_TYPE_UNKNOWN = 0,
+ DESC_TYPE_SKB = 1 << 0,
+ DESC_TYPE_FRAGLIST_SKB = 1 << 1,
+ DESC_TYPE_PAGE = 1 << 2,
+ DESC_TYPE_BOUNCE_ALL = 1 << 3,
+ DESC_TYPE_BOUNCE_HEAD = 1 << 4,
+ DESC_TYPE_SGL_SKB = 1 << 5,
+};
+
struct hns3_desc_cb {
dma_addr_t dma; /* dma address of this desc */
void *buf; /* cpu addr for a desc */
HNS3_OL4_TYPE_UNKNOWN
};
+struct hns3_rx_ptype {
+ u32 ptype:8;
+ u32 csum_level:2;
+ u32 ip_summed:2;
+ u32 l3_type:4;
+ u32 valid:1;
+};
+
struct ring_stats {
u64 sw_err_cnt;
u64 seg_pkt_cnt;
u64 tx_tso_err;
u64 over_max_recursion;
u64 hw_limitation;
+ u64 tx_bounce;
+ u64 tx_spare_full;
+ u64 copy_bits_err;
+ u64 tx_sgl;
+ u64 skb2sgl_err;
+ u64 map_sg_err;
};
struct {
u64 rx_pkts;
u64 csum_complete;
u64 rx_multicast;
u64 non_reuse_pg;
+ u64 frag_alloc_err;
+ u64 frag_alloc;
};
+ __le16 csum;
};
};
+struct hns3_tx_spare {
+ dma_addr_t dma;
+ void *buf;
+ u32 next_to_use;
+ u32 next_to_clean;
+ u32 last_to_clean;
+ u32 len;
+};
+
struct hns3_enet_ring {
struct hns3_desc *desc; /* dma map address space */
struct hns3_desc_cb *desc_cb;
* next_to_use
*/
int next_to_clean;
- union {
- int last_to_use; /* last idx used by xmit */
- u32 pull_len; /* memcpy len for current rx packet */
- };
- u32 frag_num;
- void *va; /* first buffer address for current packet */
-
u32 flag; /* ring attribute */
int pending_buf;
- struct sk_buff *skb;
- struct sk_buff *tail_skb;
+ union {
+ /* for Tx ring */
+ struct {
+ u32 fd_qb_tx_sample;
+ int last_to_use; /* last idx used by xmit */
+ u32 tx_copybreak;
+ struct hns3_tx_spare *tx_spare;
+ };
+
+ /* for Rx ring */
+ struct {
+ u32 pull_len; /* memcpy len for current rx packet */
+ u32 rx_copybreak;
+ u32 frag_num;
+ /* first buffer address for current packet */
+ unsigned char *va;
+ struct sk_buff *skb;
+ struct sk_buff *tail_skb;
+ };
+ };
} ____cacheline_internodealigned_in_smp;
enum hns3_flow_level_range {
u64 total_packets; /* total packets processed this group */
u16 count;
struct hns3_enet_coalesce coal;
+ struct dim dim;
};
struct hns3_enet_tqp_vector {
char name[HNAE3_INT_NAME_LEN];
- unsigned long last_jiffies;
+ u64 event_cnt;
} ____cacheline_internodealigned_in_smp;
struct hns3_nic_priv {
struct hns3_enet_coalesce tx_coal;
struct hns3_enet_coalesce rx_coal;
+ u32 tx_copybreak;
+ u32 rx_copybreak;
};
union l3_hdr_info {
void hns3_set_vector_coalesce_tx_ql(struct hns3_enet_tqp_vector *tqp_vector,
u32 ql_value);
-void hns3_enable_vlan_filter(struct net_device *netdev, bool enable);
void hns3_request_update_promisc_mode(struct hnae3_handle *handle);
#ifdef CONFIG_HNS3_DCB
static inline void hns3_dcbnl_setup(struct hnae3_handle *handle) {}
#endif
-void hns3_dbg_init(struct hnae3_handle *handle);
+int hns3_dbg_init(struct hnae3_handle *handle);
void hns3_dbg_uninit(struct hnae3_handle *handle);
void hns3_dbg_register_debugfs(const char *debugfs_dir_name);
void hns3_dbg_unregister_debugfs(void);
void hns3_shinfo_pack(struct skb_shared_info *shinfo, __u32 *size);
+u16 hns3_get_max_available_channels(struct hnae3_handle *h);
#endif
HNS3_TQP_STAT("tso_err", tx_tso_err),
HNS3_TQP_STAT("over_max_recursion", over_max_recursion),
HNS3_TQP_STAT("hw_limitation", hw_limitation),
+ HNS3_TQP_STAT("bounce", tx_bounce),
+ HNS3_TQP_STAT("spare_full", tx_spare_full),
+ HNS3_TQP_STAT("copy_bits_err", copy_bits_err),
+ HNS3_TQP_STAT("sgl", tx_sgl),
+ HNS3_TQP_STAT("skb2sgl_err", skb2sgl_err),
+ HNS3_TQP_STAT("map_sg_err", map_sg_err),
};
#define HNS3_TXQ_STATS_COUNT ARRAY_SIZE(hns3_txq_stats)
HNS3_TQP_STAT("csum_complete", csum_complete),
HNS3_TQP_STAT("multicast", rx_multicast),
HNS3_TQP_STAT("non_reuse_pg", non_reuse_pg),
+ HNS3_TQP_STAT("frag_alloc_err", frag_alloc_err),
+ HNS3_TQP_STAT("frag_alloc", frag_alloc),
};
#define HNS3_PRIV_FLAGS_LEN ARRAY_SIZE(hns3_priv_flags)
{
struct hnae3_handle *h = hns3_get_handle(ndev);
struct hnae3_ae_dev *ae_dev = pci_get_drvdata(h->pdev);
- bool vlan_filter_enable;
int ret;
if (!h->ae_algo->ops->set_loopback ||
if (ret || ae_dev->dev_version >= HNAE3_DEVICE_VERSION_V2)
return ret;
- if (en) {
+ if (en)
h->ae_algo->ops->set_promisc_mode(h, true, true);
- } else {
+ else
/* recover promisc mode before loopback test */
hns3_request_update_promisc_mode(h);
- vlan_filter_enable = ndev->flags & IFF_PROMISC ? false : true;
- hns3_enable_vlan_filter(ndev, vlan_filter_enable);
- }
return ret;
}
h->ae_algo->ops->get_channels(h, ch);
}
-static int hns3_get_coalesce_per_queue(struct net_device *netdev, u32 queue,
- struct ethtool_coalesce *cmd)
+static int hns3_get_coalesce(struct net_device *netdev,
+ struct ethtool_coalesce *cmd)
{
- struct hns3_enet_tqp_vector *tx_vector, *rx_vector;
struct hns3_nic_priv *priv = netdev_priv(netdev);
+ struct hns3_enet_coalesce *tx_coal = &priv->tx_coal;
+ struct hns3_enet_coalesce *rx_coal = &priv->rx_coal;
struct hnae3_handle *h = priv->ae_handle;
- u16 queue_num = h->kinfo.num_tqps;
if (hns3_nic_resetting(netdev))
return -EBUSY;
- if (queue >= queue_num) {
- netdev_err(netdev,
- "Invalid queue value %u! Queue max id=%u\n",
- queue, queue_num - 1);
- return -EINVAL;
- }
-
- tx_vector = priv->ring[queue].tqp_vector;
- rx_vector = priv->ring[queue_num + queue].tqp_vector;
-
- cmd->use_adaptive_tx_coalesce =
- tx_vector->tx_group.coal.adapt_enable;
- cmd->use_adaptive_rx_coalesce =
- rx_vector->rx_group.coal.adapt_enable;
+ cmd->use_adaptive_tx_coalesce = tx_coal->adapt_enable;
+ cmd->use_adaptive_rx_coalesce = rx_coal->adapt_enable;
- cmd->tx_coalesce_usecs = tx_vector->tx_group.coal.int_gl;
- cmd->rx_coalesce_usecs = rx_vector->rx_group.coal.int_gl;
+ cmd->tx_coalesce_usecs = tx_coal->int_gl;
+ cmd->rx_coalesce_usecs = rx_coal->int_gl;
cmd->tx_coalesce_usecs_high = h->kinfo.int_rl_setting;
cmd->rx_coalesce_usecs_high = h->kinfo.int_rl_setting;
- cmd->tx_max_coalesced_frames = tx_vector->tx_group.coal.int_ql;
- cmd->rx_max_coalesced_frames = rx_vector->rx_group.coal.int_ql;
+ cmd->tx_max_coalesced_frames = tx_coal->int_ql;
+ cmd->rx_max_coalesced_frames = rx_coal->int_ql;
return 0;
}
-static int hns3_get_coalesce(struct net_device *netdev,
- struct ethtool_coalesce *cmd)
-{
- return hns3_get_coalesce_per_queue(netdev, 0, cmd);
-}
-
static int hns3_check_gl_coalesce_para(struct net_device *netdev,
struct ethtool_coalesce *cmd)
{
return ret;
}
- ret = hns3_check_ql_coalesce_param(netdev, cmd);
- if (ret)
- return ret;
-
- if (cmd->use_adaptive_tx_coalesce == 1 ||
- cmd->use_adaptive_rx_coalesce == 1) {
- netdev_info(netdev,
- "adaptive-tx=%u and adaptive-rx=%u, tx_usecs or rx_usecs will changed dynamically.\n",
- cmd->use_adaptive_tx_coalesce,
- cmd->use_adaptive_rx_coalesce);
- }
-
- return 0;
+ return hns3_check_ql_coalesce_param(netdev, cmd);
}
static void hns3_set_coalesce_per_queue(struct net_device *netdev,
struct ethtool_coalesce *cmd)
{
struct hnae3_handle *h = hns3_get_handle(netdev);
+ struct hns3_nic_priv *priv = netdev_priv(netdev);
+ struct hns3_enet_coalesce *tx_coal = &priv->tx_coal;
+ struct hns3_enet_coalesce *rx_coal = &priv->rx_coal;
u16 queue_num = h->kinfo.num_tqps;
int ret;
int i;
h->kinfo.int_rl_setting =
hns3_rl_round_down(cmd->rx_coalesce_usecs_high);
+ tx_coal->adapt_enable = cmd->use_adaptive_tx_coalesce;
+ rx_coal->adapt_enable = cmd->use_adaptive_rx_coalesce;
+
+ tx_coal->int_gl = cmd->tx_coalesce_usecs;
+ rx_coal->int_gl = cmd->rx_coalesce_usecs;
+
+ tx_coal->int_ql = cmd->tx_max_coalesced_frames;
+ rx_coal->int_ql = cmd->rx_max_coalesced_frames;
+
for (i = 0; i < queue_num; i++)
hns3_set_coalesce_per_queue(netdev, cmd, i);
return 0;
}
+static int hns3_get_tunable(struct net_device *netdev,
+ const struct ethtool_tunable *tuna,
+ void *data)
+{
+ struct hns3_nic_priv *priv = netdev_priv(netdev);
+ int ret = 0;
+
+ switch (tuna->id) {
+ case ETHTOOL_TX_COPYBREAK:
+ /* all the tx rings have the same tx_copybreak */
+ *(u32 *)data = priv->tx_copybreak;
+ break;
+ case ETHTOOL_RX_COPYBREAK:
+ *(u32 *)data = priv->rx_copybreak;
+ break;
+ default:
+ ret = -EOPNOTSUPP;
+ break;
+ }
+
+ return ret;
+}
+
+static int hns3_set_tunable(struct net_device *netdev,
+ const struct ethtool_tunable *tuna,
+ const void *data)
+{
+ struct hns3_nic_priv *priv = netdev_priv(netdev);
+ struct hnae3_handle *h = priv->ae_handle;
+ int i, ret = 0;
+
+ switch (tuna->id) {
+ case ETHTOOL_TX_COPYBREAK:
+ priv->tx_copybreak = *(u32 *)data;
+
+ for (i = 0; i < h->kinfo.num_tqps; i++)
+ priv->ring[i].tx_copybreak = priv->tx_copybreak;
+
+ break;
+ case ETHTOOL_RX_COPYBREAK:
+ priv->rx_copybreak = *(u32 *)data;
+
+ for (i = h->kinfo.num_tqps; i < h->kinfo.num_tqps * 2; i++)
+ priv->ring[i].rx_copybreak = priv->rx_copybreak;
+
+ break;
+ default:
+ ret = -EOPNOTSUPP;
+ break;
+ }
+
+ return ret;
+}
+
#define HNS3_ETHTOOL_COALESCE (ETHTOOL_COALESCE_USECS | \
ETHTOOL_COALESCE_USE_ADAPTIVE | \
ETHTOOL_COALESCE_RX_USECS_HIGH | \
ETHTOOL_COALESCE_TX_USECS_HIGH | \
ETHTOOL_COALESCE_MAX_FRAMES)
+static int hns3_get_ts_info(struct net_device *netdev,
+ struct ethtool_ts_info *info)
+{
+ struct hnae3_handle *handle = hns3_get_handle(netdev);
+
+ if (handle->ae_algo->ops->get_ts_info)
+ return handle->ae_algo->ops->get_ts_info(handle, info);
+
+ return ethtool_op_get_ts_info(netdev, info);
+}
+
static const struct ethtool_ops hns3vf_ethtool_ops = {
.supported_coalesce_params = HNS3_ETHTOOL_COALESCE,
.get_drvinfo = hns3_get_drvinfo,
.set_msglevel = hns3_set_msglevel,
.get_priv_flags = hns3_get_priv_flags,
.set_priv_flags = hns3_set_priv_flags,
+ .get_tunable = hns3_get_tunable,
+ .set_tunable = hns3_set_tunable,
};
static const struct ethtool_ops hns3_ethtool_ops = {
.get_module_eeprom = hns3_get_module_eeprom,
.get_priv_flags = hns3_get_priv_flags,
.set_priv_flags = hns3_set_priv_flags,
+ .get_ts_info = hns3_get_ts_info,
+ .get_tunable = hns3_get_tunable,
+ .set_tunable = hns3_set_tunable,
};
void hns3_ethtool_set_ops(struct net_device *netdev)
ccflags-y += -I $(srctree)/$(src)
obj-$(CONFIG_HNS3_HCLGE) += hclge.o
-hclge-objs = hclge_main.o hclge_cmd.o hclge_mdio.o hclge_tm.o hclge_mbx.o hclge_err.o hclge_debugfs.o
+hclge-objs = hclge_main.o hclge_cmd.o hclge_mdio.o hclge_tm.o hclge_mbx.o hclge_err.o hclge_debugfs.o hclge_ptp.o
hclge-$(CONFIG_HNS3_DCB) += hclge_dcb.o
HCLGE_QUERY_CLEAR_MPF_RAS_INT,
HCLGE_QUERY_CLEAR_PF_RAS_INT,
HCLGE_QUERY_CLEAR_ALL_MPF_MSIX_INT,
- HCLGE_QUERY_CLEAR_ALL_PF_MSIX_INT};
+ HCLGE_QUERY_CLEAR_ALL_PF_MSIX_INT,
+ HCLGE_QUERY_ALL_ERR_INFO};
int i;
for (i = 0; i < ARRAY_SIZE(spec_opcode); i++) {
set_bit(HNAE3_DEV_SUPPORT_PAUSE_B, ae_dev->caps);
if (hnae3_get_bit(caps, HCLGE_CAP_PHY_IMP_B))
set_bit(HNAE3_DEV_SUPPORT_PHY_IMP_B, ae_dev->caps);
+ if (hnae3_get_bit(caps, HCLGE_CAP_RAS_IMP_B))
+ set_bit(HNAE3_DEV_SUPPORT_RAS_IMP_B, ae_dev->caps);
+ if (hnae3_get_bit(caps, HCLGE_CAP_RXD_ADV_LAYOUT_B))
+ set_bit(HNAE3_DEV_SUPPORT_RXD_ADV_LAYOUT_B, ae_dev->caps);
+ if (hnae3_get_bit(caps, HCLGE_CAP_PORT_VLAN_BYPASS_B)) {
+ set_bit(HNAE3_DEV_SUPPORT_PORT_VLAN_BYPASS_B, ae_dev->caps);
+ set_bit(HNAE3_DEV_SUPPORT_VLAN_FLTR_MDF_B, ae_dev->caps);
+ }
}
static __le32 hclge_build_api_caps(void)
struct hclge_desc desc;
u32 compat = 0;
- hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_M7_COMPAT_CFG, false);
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_IMP_COMPAT_CFG, false);
req = (struct hclge_firmware_compat_cmd *)desc.data;
HCLGE_OPC_COMMON_LOOPBACK = 0x0315,
HCLGE_OPC_CONFIG_FEC_MODE = 0x031A,
+ /* PTP commands */
+ HCLGE_OPC_PTP_INT_EN = 0x0501,
+ HCLGE_OPC_PTP_MODE_CFG = 0x0507,
+
/* PFC/Pause commands */
HCLGE_OPC_CFG_MAC_PAUSE_EN = 0x0701,
HCLGE_OPC_CFG_PFC_PAUSE_EN = 0x0702,
HCLGE_OPC_VLAN_FILTER_CTRL = 0x1100,
HCLGE_OPC_VLAN_FILTER_PF_CFG = 0x1101,
HCLGE_OPC_VLAN_FILTER_VF_CFG = 0x1102,
+ HCLGE_OPC_PORT_VLAN_BYPASS = 0x1103,
/* Flow Director commands */
HCLGE_OPC_FD_MODE_CTRL = 0x1200,
/* NCL config command */
HCLGE_OPC_QUERY_NCL_CONFIG = 0x7011,
- /* M7 stats command */
- HCLGE_OPC_M7_STATS_BD = 0x7012,
- HCLGE_OPC_M7_STATS_INFO = 0x7013,
- HCLGE_OPC_M7_COMPAT_CFG = 0x701A,
+ /* IMP stats command */
+ HCLGE_OPC_IMP_STATS_BD = 0x7012,
+ HCLGE_OPC_IMP_STATS_INFO = 0x7013,
+ HCLGE_OPC_IMP_COMPAT_CFG = 0x701A,
/* SFP command */
HCLGE_OPC_GET_SFP_EEPROM = 0x7100,
HCLGE_QUERY_MSIX_INT_STS_BD_NUM = 0x1513,
HCLGE_QUERY_CLEAR_ALL_MPF_MSIX_INT = 0x1514,
HCLGE_QUERY_CLEAR_ALL_PF_MSIX_INT = 0x1515,
+ HCLGE_QUERY_ALL_ERR_BD_NUM = 0x1516,
+ HCLGE_QUERY_ALL_ERR_INFO = 0x1517,
HCLGE_CONFIG_ROCEE_RAS_INT_EN = 0x1580,
HCLGE_QUERY_CLEAR_ROCEE_RAS_INT = 0x1581,
HCLGE_ROCEE_PF_RAS_INT_CMD = 0x1584,
HCLGE_CAP_HW_PAD_B,
HCLGE_CAP_STASH_B,
HCLGE_CAP_UDP_TUNNEL_CSUM_B,
+ HCLGE_CAP_RAS_IMP_B = 12,
HCLGE_CAP_FEC_B = 13,
HCLGE_CAP_PAUSE_B = 14,
+ HCLGE_CAP_RXD_ADV_LAYOUT_B = 15,
+ HCLGE_CAP_PORT_VLAN_BYPASS_B = 17,
};
enum HCLGE_API_CAP_BITS {
#define HCLGE_CFG_SPEED_ABILITY_M GENMASK(7, 0)
#define HCLGE_CFG_SPEED_ABILITY_EXT_S 10
#define HCLGE_CFG_SPEED_ABILITY_EXT_M GENMASK(15, 10)
+#define HCLGE_CFG_VLAN_FLTR_CAP_S 8
+#define HCLGE_CFG_VLAN_FLTR_CAP_M GENMASK(9, 8)
#define HCLGE_CFG_UMV_TBL_SPACE_S 16
#define HCLGE_CFG_UMV_TBL_SPACE_M GENMASK(31, 16)
#define HCLGE_CFG_PF_RSS_SIZE_S 0
#define HCLGE_CFG_PF_RSS_SIZE_M GENMASK(3, 0)
+#define HCLGE_CFG_TX_SPARE_BUF_SIZE_S 4
+#define HCLGE_CFG_TX_SPARE_BUF_SIZE_M GENMASK(15, 4)
#define HCLGE_CFG_CMD_CNT 4
u8 vf_bitmap[HCLGE_MAX_VF_BYTES];
};
+#define HCLGE_INGRESS_BYPASS_B 0
+struct hclge_port_vlan_filter_bypass_cmd {
+ u8 bypass_state;
+ u8 rsv1[3];
+ u8 vf_id;
+ u8 rsv2[19];
+};
+
#define HCLGE_SWITCH_ANTI_SPOOF_B 0U
#define HCLGE_SWITCH_ALW_LPBK_B 1U
#define HCLGE_SWITCH_ALW_LCL_LPBK_B 2U
u8 rsv[12];
};
-struct hclge_get_m7_bd_cmd {
+struct hclge_get_imp_bd_cmd {
__le32 bd_num;
u8 rsv[20];
};
#include <linux/device.h>
#include "hclge_debugfs.h"
+#include "hclge_err.h"
#include "hclge_main.h"
#include "hclge_tm.h"
#include "hnae3.h"
+static const char * const state_str[] = { "off", "on" };
+static const char * const hclge_mac_state_str[] = {
+ "TO_ADD", "TO_DEL", "ACTIVE"
+};
+
static const struct hclge_dbg_reg_type_info hclge_dbg_reg_info[] = {
- { .reg_type = "bios common",
+ { .cmd = HNAE3_DBG_CMD_REG_BIOS_COMMON,
.dfx_msg = &hclge_dbg_bios_common_reg[0],
.reg_msg = { .msg_num = ARRAY_SIZE(hclge_dbg_bios_common_reg),
.offset = HCLGE_DBG_DFX_BIOS_OFFSET,
.cmd = HCLGE_OPC_DFX_BIOS_COMMON_REG } },
- { .reg_type = "ssu",
+ { .cmd = HNAE3_DBG_CMD_REG_SSU,
.dfx_msg = &hclge_dbg_ssu_reg_0[0],
.reg_msg = { .msg_num = ARRAY_SIZE(hclge_dbg_ssu_reg_0),
.offset = HCLGE_DBG_DFX_SSU_0_OFFSET,
.cmd = HCLGE_OPC_DFX_SSU_REG_0 } },
- { .reg_type = "ssu",
+ { .cmd = HNAE3_DBG_CMD_REG_SSU,
.dfx_msg = &hclge_dbg_ssu_reg_1[0],
.reg_msg = { .msg_num = ARRAY_SIZE(hclge_dbg_ssu_reg_1),
.offset = HCLGE_DBG_DFX_SSU_1_OFFSET,
.cmd = HCLGE_OPC_DFX_SSU_REG_1 } },
- { .reg_type = "ssu",
+ { .cmd = HNAE3_DBG_CMD_REG_SSU,
.dfx_msg = &hclge_dbg_ssu_reg_2[0],
.reg_msg = { .msg_num = ARRAY_SIZE(hclge_dbg_ssu_reg_2),
.offset = HCLGE_DBG_DFX_SSU_2_OFFSET,
.cmd = HCLGE_OPC_DFX_SSU_REG_2 } },
- { .reg_type = "igu egu",
+ { .cmd = HNAE3_DBG_CMD_REG_IGU_EGU,
.dfx_msg = &hclge_dbg_igu_egu_reg[0],
.reg_msg = { .msg_num = ARRAY_SIZE(hclge_dbg_igu_egu_reg),
.offset = HCLGE_DBG_DFX_IGU_OFFSET,
.cmd = HCLGE_OPC_DFX_IGU_EGU_REG } },
- { .reg_type = "rpu",
+ { .cmd = HNAE3_DBG_CMD_REG_RPU,
.dfx_msg = &hclge_dbg_rpu_reg_0[0],
.reg_msg = { .msg_num = ARRAY_SIZE(hclge_dbg_rpu_reg_0),
.offset = HCLGE_DBG_DFX_RPU_0_OFFSET,
.cmd = HCLGE_OPC_DFX_RPU_REG_0 } },
- { .reg_type = "rpu",
+ { .cmd = HNAE3_DBG_CMD_REG_RPU,
.dfx_msg = &hclge_dbg_rpu_reg_1[0],
.reg_msg = { .msg_num = ARRAY_SIZE(hclge_dbg_rpu_reg_1),
.offset = HCLGE_DBG_DFX_RPU_1_OFFSET,
.cmd = HCLGE_OPC_DFX_RPU_REG_1 } },
- { .reg_type = "ncsi",
+ { .cmd = HNAE3_DBG_CMD_REG_NCSI,
.dfx_msg = &hclge_dbg_ncsi_reg[0],
.reg_msg = { .msg_num = ARRAY_SIZE(hclge_dbg_ncsi_reg),
.offset = HCLGE_DBG_DFX_NCSI_OFFSET,
.cmd = HCLGE_OPC_DFX_NCSI_REG } },
- { .reg_type = "rtc",
+ { .cmd = HNAE3_DBG_CMD_REG_RTC,
.dfx_msg = &hclge_dbg_rtc_reg[0],
.reg_msg = { .msg_num = ARRAY_SIZE(hclge_dbg_rtc_reg),
.offset = HCLGE_DBG_DFX_RTC_OFFSET,
.cmd = HCLGE_OPC_DFX_RTC_REG } },
- { .reg_type = "ppp",
+ { .cmd = HNAE3_DBG_CMD_REG_PPP,
.dfx_msg = &hclge_dbg_ppp_reg[0],
.reg_msg = { .msg_num = ARRAY_SIZE(hclge_dbg_ppp_reg),
.offset = HCLGE_DBG_DFX_PPP_OFFSET,
.cmd = HCLGE_OPC_DFX_PPP_REG } },
- { .reg_type = "rcb",
+ { .cmd = HNAE3_DBG_CMD_REG_RCB,
.dfx_msg = &hclge_dbg_rcb_reg[0],
.reg_msg = { .msg_num = ARRAY_SIZE(hclge_dbg_rcb_reg),
.offset = HCLGE_DBG_DFX_RCB_OFFSET,
.cmd = HCLGE_OPC_DFX_RCB_REG } },
- { .reg_type = "tqp",
+ { .cmd = HNAE3_DBG_CMD_REG_TQP,
.dfx_msg = &hclge_dbg_tqp_reg[0],
.reg_msg = { .msg_num = ARRAY_SIZE(hclge_dbg_tqp_reg),
.offset = HCLGE_DBG_DFX_TQP_OFFSET,
.cmd = HCLGE_OPC_DFX_TQP_REG } },
};
-static int hclge_dbg_get_dfx_bd_num(struct hclge_dev *hdev, int offset)
+static void hclge_dbg_fill_content(char *content, u16 len,
+ const struct hclge_dbg_item *items,
+ const char **result, u16 size)
+{
+ char *pos = content;
+ u16 i;
+
+ memset(content, ' ', len);
+ for (i = 0; i < size; i++) {
+ if (result)
+ strncpy(pos, result[i], strlen(result[i]));
+ else
+ strncpy(pos, items[i].name, strlen(items[i].name));
+ pos += strlen(items[i].name) + items[i].interval;
+ }
+ *pos++ = '\n';
+ *pos++ = '\0';
+}
+
+static char *hclge_dbg_get_func_id_str(char *buf, u8 id)
+{
+ if (id)
+ sprintf(buf, "vf%u", id - 1);
+ else
+ sprintf(buf, "pf");
+
+ return buf;
+}
+
+static int hclge_dbg_get_dfx_bd_num(struct hclge_dev *hdev, int offset,
+ u32 *bd_num)
{
struct hclge_desc desc[HCLGE_GET_DFX_REG_TYPE_CNT];
int entries_per_desc;
ret = hclge_query_bd_num_cmd_send(hdev, desc);
if (ret) {
dev_err(&hdev->pdev->dev,
- "get dfx bdnum fail, ret = %d\n", ret);
+ "failed to get dfx bd_num, offset = %d, ret = %d\n",
+ offset, ret);
return ret;
}
entries_per_desc = ARRAY_SIZE(desc[0].data);
index = offset % entries_per_desc;
- return le32_to_cpu(desc[offset / entries_per_desc].data[index]);
+
+ *bd_num = le32_to_cpu(desc[offset / entries_per_desc].data[index]);
+ if (!(*bd_num)) {
+ dev_err(&hdev->pdev->dev, "The value of dfx bd_num is 0!\n");
+ return -EINVAL;
+ }
+
+ return 0;
}
static int hclge_dbg_cmd_send(struct hclge_dev *hdev,
return ret;
}
-static void hclge_dbg_dump_reg_common(struct hclge_dev *hdev,
- const struct hclge_dbg_reg_type_info *reg_info,
- const char *cmd_buf)
+static int
+hclge_dbg_dump_reg_tqp(struct hclge_dev *hdev,
+ const struct hclge_dbg_reg_type_info *reg_info,
+ char *buf, int len, int *pos)
{
-#define IDX_OFFSET 1
-
- const char *s = &cmd_buf[strlen(reg_info->reg_type) + IDX_OFFSET];
const struct hclge_dbg_dfx_message *dfx_message = reg_info->dfx_msg;
const struct hclge_dbg_reg_common_msg *reg_msg = ®_info->reg_msg;
struct hclge_desc *desc_src;
+ u32 index, entry, i, cnt;
+ int bd_num, min_num, ret;
struct hclge_desc *desc;
- int entries_per_desc;
- int bd_num, buf_len;
- int index = 0;
- int min_num;
- int ret, i;
- if (*s) {
- ret = kstrtouint(s, 0, &index);
- index = (ret != 0) ? 0 : index;
- }
+ ret = hclge_dbg_get_dfx_bd_num(hdev, reg_msg->offset, &bd_num);
+ if (ret)
+ return ret;
+
+ desc_src = kcalloc(bd_num, sizeof(struct hclge_desc), GFP_KERNEL);
+ if (!desc_src)
+ return -ENOMEM;
+
+ min_num = min_t(int, bd_num * HCLGE_DESC_DATA_LEN, reg_msg->msg_num);
+
+ for (i = 0, cnt = 0; i < min_num; i++, dfx_message++)
+ *pos += scnprintf(buf + *pos, len - *pos, "item%u = %s\n",
+ cnt++, dfx_message->message);
+
+ for (i = 0; i < cnt; i++)
+ *pos += scnprintf(buf + *pos, len - *pos, "item%u\t", i);
+
+ *pos += scnprintf(buf + *pos, len - *pos, "\n");
- bd_num = hclge_dbg_get_dfx_bd_num(hdev, reg_msg->offset);
- if (bd_num <= 0) {
- dev_err(&hdev->pdev->dev, "get cmd(%d) bd num(%d) failed\n",
- reg_msg->offset, bd_num);
- return;
+ for (index = 0; index < hdev->vport[0].alloc_tqps; index++) {
+ dfx_message = reg_info->dfx_msg;
+ desc = desc_src;
+ ret = hclge_dbg_cmd_send(hdev, desc, index, bd_num,
+ reg_msg->cmd);
+ if (ret)
+ break;
+
+ for (i = 0; i < min_num; i++, dfx_message++) {
+ entry = i % HCLGE_DESC_DATA_LEN;
+ if (i > 0 && !entry)
+ desc++;
+
+ *pos += scnprintf(buf + *pos, len - *pos, "%#x\t",
+ le32_to_cpu(desc->data[entry]));
+ }
+ *pos += scnprintf(buf + *pos, len - *pos, "\n");
}
- buf_len = sizeof(struct hclge_desc) * bd_num;
- desc_src = kzalloc(buf_len, GFP_KERNEL);
+ kfree(desc_src);
+ return ret;
+}
+
+static int
+hclge_dbg_dump_reg_common(struct hclge_dev *hdev,
+ const struct hclge_dbg_reg_type_info *reg_info,
+ char *buf, int len, int *pos)
+{
+ const struct hclge_dbg_reg_common_msg *reg_msg = ®_info->reg_msg;
+ const struct hclge_dbg_dfx_message *dfx_message = reg_info->dfx_msg;
+ struct hclge_desc *desc_src;
+ int bd_num, min_num, ret;
+ struct hclge_desc *desc;
+ u32 entry, i;
+
+ ret = hclge_dbg_get_dfx_bd_num(hdev, reg_msg->offset, &bd_num);
+ if (ret)
+ return ret;
+
+ desc_src = kcalloc(bd_num, sizeof(struct hclge_desc), GFP_KERNEL);
if (!desc_src)
- return;
+ return -ENOMEM;
desc = desc_src;
- ret = hclge_dbg_cmd_send(hdev, desc, index, bd_num, reg_msg->cmd);
+
+ ret = hclge_dbg_cmd_send(hdev, desc, 0, bd_num, reg_msg->cmd);
if (ret) {
- kfree(desc_src);
- return;
+ kfree(desc);
+ return ret;
}
- entries_per_desc = ARRAY_SIZE(desc->data);
- min_num = min_t(int, bd_num * entries_per_desc, reg_msg->msg_num);
+ min_num = min_t(int, bd_num * HCLGE_DESC_DATA_LEN, reg_msg->msg_num);
- desc = desc_src;
- for (i = 0; i < min_num; i++) {
- if (i > 0 && (i % entries_per_desc) == 0)
+ for (i = 0; i < min_num; i++, dfx_message++) {
+ entry = i % HCLGE_DESC_DATA_LEN;
+ if (i > 0 && !entry)
desc++;
- if (dfx_message->flag)
- dev_info(&hdev->pdev->dev, "%s: 0x%x\n",
- dfx_message->message,
- le32_to_cpu(desc->data[i % entries_per_desc]));
+ if (!dfx_message->flag)
+ continue;
- dfx_message++;
+ *pos += scnprintf(buf + *pos, len - *pos, "%s: %#x\n",
+ dfx_message->message,
+ le32_to_cpu(desc->data[entry]));
}
kfree(desc_src);
+ return 0;
}
-static void hclge_dbg_dump_mac_enable_status(struct hclge_dev *hdev)
+static int hclge_dbg_dump_mac_enable_status(struct hclge_dev *hdev, char *buf,
+ int len, int *pos)
{
struct hclge_config_mac_mode_cmd *req;
struct hclge_desc desc;
if (ret) {
dev_err(&hdev->pdev->dev,
"failed to dump mac enable status, ret = %d\n", ret);
- return;
+ return ret;
}
req = (struct hclge_config_mac_mode_cmd *)desc.data;
loop_en = le32_to_cpu(req->txrx_pad_fcs_loop_en);
- dev_info(&hdev->pdev->dev, "config_mac_trans_en: %#x\n",
- hnae3_get_bit(loop_en, HCLGE_MAC_TX_EN_B));
- dev_info(&hdev->pdev->dev, "config_mac_rcv_en: %#x\n",
- hnae3_get_bit(loop_en, HCLGE_MAC_RX_EN_B));
- dev_info(&hdev->pdev->dev, "config_pad_trans_en: %#x\n",
- hnae3_get_bit(loop_en, HCLGE_MAC_PAD_TX_B));
- dev_info(&hdev->pdev->dev, "config_pad_rcv_en: %#x\n",
- hnae3_get_bit(loop_en, HCLGE_MAC_PAD_RX_B));
- dev_info(&hdev->pdev->dev, "config_1588_trans_en: %#x\n",
- hnae3_get_bit(loop_en, HCLGE_MAC_1588_TX_B));
- dev_info(&hdev->pdev->dev, "config_1588_rcv_en: %#x\n",
- hnae3_get_bit(loop_en, HCLGE_MAC_1588_RX_B));
- dev_info(&hdev->pdev->dev, "config_mac_app_loop_en: %#x\n",
- hnae3_get_bit(loop_en, HCLGE_MAC_APP_LP_B));
- dev_info(&hdev->pdev->dev, "config_mac_line_loop_en: %#x\n",
- hnae3_get_bit(loop_en, HCLGE_MAC_LINE_LP_B));
- dev_info(&hdev->pdev->dev, "config_mac_fcs_tx_en: %#x\n",
- hnae3_get_bit(loop_en, HCLGE_MAC_FCS_TX_B));
- dev_info(&hdev->pdev->dev, "config_mac_rx_oversize_truncate_en: %#x\n",
- hnae3_get_bit(loop_en, HCLGE_MAC_RX_OVERSIZE_TRUNCATE_B));
- dev_info(&hdev->pdev->dev, "config_mac_rx_fcs_strip_en: %#x\n",
- hnae3_get_bit(loop_en, HCLGE_MAC_RX_FCS_STRIP_B));
- dev_info(&hdev->pdev->dev, "config_mac_rx_fcs_en: %#x\n",
- hnae3_get_bit(loop_en, HCLGE_MAC_RX_FCS_B));
- dev_info(&hdev->pdev->dev, "config_mac_tx_under_min_err_en: %#x\n",
- hnae3_get_bit(loop_en, HCLGE_MAC_TX_UNDER_MIN_ERR_B));
- dev_info(&hdev->pdev->dev, "config_mac_tx_oversize_truncate_en: %#x\n",
- hnae3_get_bit(loop_en, HCLGE_MAC_TX_OVERSIZE_TRUNCATE_B));
+ *pos += scnprintf(buf + *pos, len - *pos, "mac_trans_en: %#x\n",
+ hnae3_get_bit(loop_en, HCLGE_MAC_TX_EN_B));
+ *pos += scnprintf(buf + *pos, len - *pos, "mac_rcv_en: %#x\n",
+ hnae3_get_bit(loop_en, HCLGE_MAC_RX_EN_B));
+ *pos += scnprintf(buf + *pos, len - *pos, "pad_trans_en: %#x\n",
+ hnae3_get_bit(loop_en, HCLGE_MAC_PAD_TX_B));
+ *pos += scnprintf(buf + *pos, len - *pos, "pad_rcv_en: %#x\n",
+ hnae3_get_bit(loop_en, HCLGE_MAC_PAD_RX_B));
+ *pos += scnprintf(buf + *pos, len - *pos, "1588_trans_en: %#x\n",
+ hnae3_get_bit(loop_en, HCLGE_MAC_1588_TX_B));
+ *pos += scnprintf(buf + *pos, len - *pos, "1588_rcv_en: %#x\n",
+ hnae3_get_bit(loop_en, HCLGE_MAC_1588_RX_B));
+ *pos += scnprintf(buf + *pos, len - *pos, "mac_app_loop_en: %#x\n",
+ hnae3_get_bit(loop_en, HCLGE_MAC_APP_LP_B));
+ *pos += scnprintf(buf + *pos, len - *pos, "mac_line_loop_en: %#x\n",
+ hnae3_get_bit(loop_en, HCLGE_MAC_LINE_LP_B));
+ *pos += scnprintf(buf + *pos, len - *pos, "mac_fcs_tx_en: %#x\n",
+ hnae3_get_bit(loop_en, HCLGE_MAC_FCS_TX_B));
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "mac_rx_oversize_truncate_en: %#x\n",
+ hnae3_get_bit(loop_en,
+ HCLGE_MAC_RX_OVERSIZE_TRUNCATE_B));
+ *pos += scnprintf(buf + *pos, len - *pos, "mac_rx_fcs_strip_en: %#x\n",
+ hnae3_get_bit(loop_en, HCLGE_MAC_RX_FCS_STRIP_B));
+ *pos += scnprintf(buf + *pos, len - *pos, "mac_rx_fcs_en: %#x\n",
+ hnae3_get_bit(loop_en, HCLGE_MAC_RX_FCS_B));
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "mac_tx_under_min_err_en: %#x\n",
+ hnae3_get_bit(loop_en, HCLGE_MAC_TX_UNDER_MIN_ERR_B));
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "mac_tx_oversize_truncate_en: %#x\n",
+ hnae3_get_bit(loop_en,
+ HCLGE_MAC_TX_OVERSIZE_TRUNCATE_B));
+
+ return 0;
}
-static void hclge_dbg_dump_mac_frame_size(struct hclge_dev *hdev)
+static int hclge_dbg_dump_mac_frame_size(struct hclge_dev *hdev, char *buf,
+ int len, int *pos)
{
struct hclge_config_max_frm_size_cmd *req;
struct hclge_desc desc;
if (ret) {
dev_err(&hdev->pdev->dev,
"failed to dump mac frame size, ret = %d\n", ret);
- return;
+ return ret;
}
req = (struct hclge_config_max_frm_size_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "max_frame_size: %u\n",
- le16_to_cpu(req->max_frm_size));
- dev_info(&hdev->pdev->dev, "min_frame_size: %u\n", req->min_frm_size);
+ *pos += scnprintf(buf + *pos, len - *pos, "max_frame_size: %u\n",
+ le16_to_cpu(req->max_frm_size));
+ *pos += scnprintf(buf + *pos, len - *pos, "min_frame_size: %u\n",
+ req->min_frm_size);
+
+ return 0;
}
-static void hclge_dbg_dump_mac_speed_duplex(struct hclge_dev *hdev)
+static int hclge_dbg_dump_mac_speed_duplex(struct hclge_dev *hdev, char *buf,
+ int len, int *pos)
{
#define HCLGE_MAC_SPEED_SHIFT 0
#define HCLGE_MAC_SPEED_MASK GENMASK(5, 0)
if (ret) {
dev_err(&hdev->pdev->dev,
"failed to dump mac speed duplex, ret = %d\n", ret);
- return;
+ return ret;
}
req = (struct hclge_config_mac_speed_dup_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "speed: %#lx\n",
- hnae3_get_field(req->speed_dup, HCLGE_MAC_SPEED_MASK,
- HCLGE_MAC_SPEED_SHIFT));
- dev_info(&hdev->pdev->dev, "duplex: %#x\n",
- hnae3_get_bit(req->speed_dup, HCLGE_MAC_DUPLEX_SHIFT));
+ *pos += scnprintf(buf + *pos, len - *pos, "speed: %#lx\n",
+ hnae3_get_field(req->speed_dup, HCLGE_MAC_SPEED_MASK,
+ HCLGE_MAC_SPEED_SHIFT));
+ *pos += scnprintf(buf + *pos, len - *pos, "duplex: %#x\n",
+ hnae3_get_bit(req->speed_dup,
+ HCLGE_MAC_DUPLEX_SHIFT));
+ return 0;
}
-static void hclge_dbg_dump_mac(struct hclge_dev *hdev)
+static int hclge_dbg_dump_mac(struct hclge_dev *hdev, char *buf, int len)
{
- hclge_dbg_dump_mac_enable_status(hdev);
+ int pos = 0;
+ int ret;
+
+ ret = hclge_dbg_dump_mac_enable_status(hdev, buf, len, &pos);
+ if (ret)
+ return ret;
- hclge_dbg_dump_mac_frame_size(hdev);
+ ret = hclge_dbg_dump_mac_frame_size(hdev, buf, len, &pos);
+ if (ret)
+ return ret;
- hclge_dbg_dump_mac_speed_duplex(hdev);
+ return hclge_dbg_dump_mac_speed_duplex(hdev, buf, len, &pos);
}
-static void hclge_dbg_dump_dcb(struct hclge_dev *hdev, const char *cmd_buf)
+static int hclge_dbg_dump_dcb_qset(struct hclge_dev *hdev, char *buf, int len,
+ int *pos)
{
- struct device *dev = &hdev->pdev->dev;
struct hclge_dbg_bitmap_cmd *bitmap;
- enum hclge_opcode_type cmd;
- int rq_id, pri_id, qset_id;
- int port_id, nq_id, pg_id;
- struct hclge_desc desc[2];
-
- int cnt, ret;
-
- cnt = sscanf(cmd_buf, "%i %i %i %i %i %i",
- &port_id, &pri_id, &pg_id, &rq_id, &nq_id, &qset_id);
- if (cnt != 6) {
- dev_err(&hdev->pdev->dev,
- "dump dcb: bad command parameter, cnt=%d\n", cnt);
- return;
- }
+ struct hclge_desc desc;
+ u16 qset_id, qset_num;
+ int ret;
- cmd = HCLGE_OPC_QSET_DFX_STS;
- ret = hclge_dbg_cmd_send(hdev, desc, qset_id, 1, cmd);
+ ret = hclge_tm_get_qset_num(hdev, &qset_num);
if (ret)
- goto err_dcb_cmd_send;
+ return ret;
- bitmap = (struct hclge_dbg_bitmap_cmd *)&desc[0].data[1];
- dev_info(dev, "roce_qset_mask: 0x%x\n", bitmap->bit0);
- dev_info(dev, "nic_qs_mask: 0x%x\n", bitmap->bit1);
- dev_info(dev, "qs_shaping_pass: 0x%x\n", bitmap->bit2);
- dev_info(dev, "qs_bp_sts: 0x%x\n", bitmap->bit3);
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "qset_id roce_qset_mask nic_qset_mask qset_shaping_pass qset_bp_status\n");
+ for (qset_id = 0; qset_id < qset_num; qset_id++) {
+ ret = hclge_dbg_cmd_send(hdev, &desc, qset_id, 1,
+ HCLGE_OPC_QSET_DFX_STS);
+ if (ret)
+ return ret;
- cmd = HCLGE_OPC_PRI_DFX_STS;
- ret = hclge_dbg_cmd_send(hdev, desc, pri_id, 1, cmd);
- if (ret)
- goto err_dcb_cmd_send;
+ bitmap = (struct hclge_dbg_bitmap_cmd *)&desc.data[1];
- bitmap = (struct hclge_dbg_bitmap_cmd *)&desc[0].data[1];
- dev_info(dev, "pri_mask: 0x%x\n", bitmap->bit0);
- dev_info(dev, "pri_cshaping_pass: 0x%x\n", bitmap->bit1);
- dev_info(dev, "pri_pshaping_pass: 0x%x\n", bitmap->bit2);
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "%04u %#x %#x %#x %#x\n",
+ qset_id, bitmap->bit0, bitmap->bit1,
+ bitmap->bit2, bitmap->bit3);
+ }
- cmd = HCLGE_OPC_PG_DFX_STS;
- ret = hclge_dbg_cmd_send(hdev, desc, pg_id, 1, cmd);
- if (ret)
- goto err_dcb_cmd_send;
+ return 0;
+}
- bitmap = (struct hclge_dbg_bitmap_cmd *)&desc[0].data[1];
- dev_info(dev, "pg_mask: 0x%x\n", bitmap->bit0);
- dev_info(dev, "pg_cshaping_pass: 0x%x\n", bitmap->bit1);
- dev_info(dev, "pg_pshaping_pass: 0x%x\n", bitmap->bit2);
+static int hclge_dbg_dump_dcb_pri(struct hclge_dev *hdev, char *buf, int len,
+ int *pos)
+{
+ struct hclge_dbg_bitmap_cmd *bitmap;
+ struct hclge_desc desc;
+ u8 pri_id, pri_num;
+ int ret;
- cmd = HCLGE_OPC_PORT_DFX_STS;
- ret = hclge_dbg_cmd_send(hdev, desc, port_id, 1, cmd);
+ ret = hclge_tm_get_pri_num(hdev, &pri_num);
if (ret)
- goto err_dcb_cmd_send;
+ return ret;
- bitmap = (struct hclge_dbg_bitmap_cmd *)&desc[0].data[1];
- dev_info(dev, "port_mask: 0x%x\n", bitmap->bit0);
- dev_info(dev, "port_shaping_pass: 0x%x\n", bitmap->bit1);
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "pri_id pri_mask pri_cshaping_pass pri_pshaping_pass\n");
+ for (pri_id = 0; pri_id < pri_num; pri_id++) {
+ ret = hclge_dbg_cmd_send(hdev, &desc, pri_id, 1,
+ HCLGE_OPC_PRI_DFX_STS);
+ if (ret)
+ return ret;
- cmd = HCLGE_OPC_SCH_NQ_CNT;
- ret = hclge_dbg_cmd_send(hdev, desc, nq_id, 1, cmd);
- if (ret)
- goto err_dcb_cmd_send;
+ bitmap = (struct hclge_dbg_bitmap_cmd *)&desc.data[1];
- dev_info(dev, "sch_nq_cnt: 0x%x\n", le32_to_cpu(desc[0].data[1]));
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "%03u %#x %#x %#x\n",
+ pri_id, bitmap->bit0, bitmap->bit1,
+ bitmap->bit2);
+ }
- cmd = HCLGE_OPC_SCH_RQ_CNT;
- ret = hclge_dbg_cmd_send(hdev, desc, nq_id, 1, cmd);
- if (ret)
- goto err_dcb_cmd_send;
+ return 0;
+}
- dev_info(dev, "sch_rq_cnt: 0x%x\n", le32_to_cpu(desc[0].data[1]));
+static int hclge_dbg_dump_dcb_pg(struct hclge_dev *hdev, char *buf, int len,
+ int *pos)
+{
+ struct hclge_dbg_bitmap_cmd *bitmap;
+ struct hclge_desc desc;
+ u8 pg_id;
+ int ret;
- cmd = HCLGE_OPC_TM_INTERNAL_STS;
- ret = hclge_dbg_cmd_send(hdev, desc, 0, 2, cmd);
- if (ret)
- goto err_dcb_cmd_send;
-
- dev_info(dev, "pri_bp: 0x%x\n", le32_to_cpu(desc[0].data[1]));
- dev_info(dev, "fifo_dfx_info: 0x%x\n", le32_to_cpu(desc[0].data[2]));
- dev_info(dev, "sch_roce_fifo_afull_gap: 0x%x\n",
- le32_to_cpu(desc[0].data[3]));
- dev_info(dev, "tx_private_waterline: 0x%x\n",
- le32_to_cpu(desc[0].data[4]));
- dev_info(dev, "tm_bypass_en: 0x%x\n", le32_to_cpu(desc[0].data[5]));
- dev_info(dev, "SSU_TM_BYPASS_EN: 0x%x\n", le32_to_cpu(desc[1].data[0]));
- dev_info(dev, "SSU_RESERVE_CFG: 0x%x\n", le32_to_cpu(desc[1].data[1]));
-
- cmd = HCLGE_OPC_TM_INTERNAL_CNT;
- ret = hclge_dbg_cmd_send(hdev, desc, port_id, 1, cmd);
- if (ret)
- goto err_dcb_cmd_send;
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "pg_id pg_mask pg_cshaping_pass pg_pshaping_pass\n");
+ for (pg_id = 0; pg_id < hdev->tm_info.num_pg; pg_id++) {
+ ret = hclge_dbg_cmd_send(hdev, &desc, pg_id, 1,
+ HCLGE_OPC_PG_DFX_STS);
+ if (ret)
+ return ret;
- dev_info(dev, "SCH_NIC_NUM: 0x%x\n", le32_to_cpu(desc[0].data[1]));
- dev_info(dev, "SCH_ROCE_NUM: 0x%x\n", le32_to_cpu(desc[0].data[2]));
+ bitmap = (struct hclge_dbg_bitmap_cmd *)&desc.data[1];
- cmd = HCLGE_OPC_TM_INTERNAL_STS_1;
- ret = hclge_dbg_cmd_send(hdev, desc, port_id, 1, cmd);
- if (ret)
- goto err_dcb_cmd_send;
-
- dev_info(dev, "TC_MAP_SEL: 0x%x\n", le32_to_cpu(desc[0].data[1]));
- dev_info(dev, "IGU_PFC_PRI_EN: 0x%x\n", le32_to_cpu(desc[0].data[2]));
- dev_info(dev, "MAC_PFC_PRI_EN: 0x%x\n", le32_to_cpu(desc[0].data[3]));
- dev_info(dev, "IGU_PRI_MAP_TC_CFG: 0x%x\n",
- le32_to_cpu(desc[0].data[4]));
- dev_info(dev, "IGU_TX_PRI_MAP_TC_CFG: 0x%x\n",
- le32_to_cpu(desc[0].data[5]));
- return;
-
-err_dcb_cmd_send:
- dev_err(&hdev->pdev->dev,
- "failed to dump dcb dfx, cmd = %#x, ret = %d\n",
- cmd, ret);
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "%03u %#x %#x %#x\n",
+ pg_id, bitmap->bit0, bitmap->bit1,
+ bitmap->bit2);
+ }
+
+ return 0;
}
-static void hclge_dbg_dump_reg_cmd(struct hclge_dev *hdev, const char *cmd_buf)
+static int hclge_dbg_dump_dcb_queue(struct hclge_dev *hdev, char *buf, int len,
+ int *pos)
{
- const struct hclge_dbg_reg_type_info *reg_info;
- bool has_dump = false;
- int i;
+ struct hclge_desc desc;
+ u16 nq_id;
+ int ret;
- for (i = 0; i < ARRAY_SIZE(hclge_dbg_reg_info); i++) {
- reg_info = &hclge_dbg_reg_info[i];
- if (!strncmp(cmd_buf, reg_info->reg_type,
- strlen(reg_info->reg_type))) {
- hclge_dbg_dump_reg_common(hdev, reg_info, cmd_buf);
- has_dump = true;
- }
- }
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "nq_id sch_nic_queue_cnt sch_roce_queue_cnt\n");
+ for (nq_id = 0; nq_id < hdev->num_tqps; nq_id++) {
+ ret = hclge_dbg_cmd_send(hdev, &desc, nq_id, 1,
+ HCLGE_OPC_SCH_NQ_CNT);
+ if (ret)
+ return ret;
- if (strncmp(cmd_buf, "mac", strlen("mac")) == 0) {
- hclge_dbg_dump_mac(hdev);
- has_dump = true;
- }
+ *pos += scnprintf(buf + *pos, len - *pos, "%04u %#x",
+ nq_id, le32_to_cpu(desc.data[1]));
- if (strncmp(cmd_buf, "dcb", 3) == 0) {
- hclge_dbg_dump_dcb(hdev, &cmd_buf[sizeof("dcb")]);
- has_dump = true;
- }
+ ret = hclge_dbg_cmd_send(hdev, &desc, nq_id, 1,
+ HCLGE_OPC_SCH_RQ_CNT);
+ if (ret)
+ return ret;
- if (!has_dump) {
- dev_info(&hdev->pdev->dev, "unknown command\n");
- return;
+ *pos += scnprintf(buf + *pos, len - *pos,
+ " %#x\n",
+ le32_to_cpu(desc.data[1]));
}
-}
-static void hclge_print_tc_info(struct hclge_dev *hdev, bool flag, int index)
-{
- if (flag)
- dev_info(&hdev->pdev->dev, "tc(%d): no sp mode weight: %u\n",
- index, hdev->tm_info.pg_info[0].tc_dwrr[index]);
- else
- dev_info(&hdev->pdev->dev, "tc(%d): sp mode\n", index);
+ return 0;
}
-static void hclge_dbg_dump_tc(struct hclge_dev *hdev)
+static int hclge_dbg_dump_dcb_port(struct hclge_dev *hdev, char *buf, int len,
+ int *pos)
{
- struct hclge_ets_tc_weight_cmd *ets_weight;
+ struct hclge_dbg_bitmap_cmd *bitmap;
struct hclge_desc desc;
- int i, ret;
-
- if (!hnae3_dev_dcb_supported(hdev)) {
- dev_info(&hdev->pdev->dev,
- "Only DCB-supported dev supports tc\n");
- return;
- }
-
- hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_ETS_TC_WEIGHT, true);
+ u8 port_id = 0;
+ int ret;
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret) {
- dev_err(&hdev->pdev->dev, "dump tc fail, ret = %d\n", ret);
- return;
- }
+ ret = hclge_dbg_cmd_send(hdev, &desc, port_id, 1,
+ HCLGE_OPC_PORT_DFX_STS);
+ if (ret)
+ return ret;
- ets_weight = (struct hclge_ets_tc_weight_cmd *)desc.data;
+ bitmap = (struct hclge_dbg_bitmap_cmd *)&desc.data[1];
- dev_info(&hdev->pdev->dev, "dump tc: %u tc enabled\n",
- hdev->tm_info.num_tc);
- dev_info(&hdev->pdev->dev, "weight_offset: %u\n",
- ets_weight->weight_offset);
+ *pos += scnprintf(buf + *pos, len - *pos, "port_mask: %#x\n",
+ bitmap->bit0);
+ *pos += scnprintf(buf + *pos, len - *pos, "port_shaping_pass: %#x\n",
+ bitmap->bit1);
- for (i = 0; i < HNAE3_MAX_TC; i++)
- hclge_print_tc_info(hdev, ets_weight->tc_weight[i], i);
+ return 0;
}
-static void hclge_dbg_dump_tm_pg(struct hclge_dev *hdev)
+static int hclge_dbg_dump_dcb_tm(struct hclge_dev *hdev, char *buf, int len,
+ int *pos)
{
- struct hclge_port_shapping_cmd *port_shap_cfg_cmd;
- struct hclge_bp_to_qs_map_cmd *bp_to_qs_map_cmd;
- struct hclge_pg_shapping_cmd *pg_shap_cfg_cmd;
- enum hclge_opcode_type cmd;
- struct hclge_desc desc;
+ struct hclge_desc desc[2];
+ u8 port_id = 0;
int ret;
- cmd = HCLGE_OPC_TM_PG_C_SHAPPING;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ ret = hclge_dbg_cmd_send(hdev, desc, port_id, 1,
+ HCLGE_OPC_TM_INTERNAL_CNT);
if (ret)
- goto err_tm_pg_cmd_send;
+ return ret;
- pg_shap_cfg_cmd = (struct hclge_pg_shapping_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "PG_C pg_id: %u\n", pg_shap_cfg_cmd->pg_id);
- dev_info(&hdev->pdev->dev, "PG_C pg_shapping: 0x%x\n",
- le32_to_cpu(pg_shap_cfg_cmd->pg_shapping_para));
+ *pos += scnprintf(buf + *pos, len - *pos, "SCH_NIC_NUM: %#x\n",
+ le32_to_cpu(desc[0].data[1]));
+ *pos += scnprintf(buf + *pos, len - *pos, "SCH_ROCE_NUM: %#x\n",
+ le32_to_cpu(desc[0].data[2]));
- cmd = HCLGE_OPC_TM_PG_P_SHAPPING;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ ret = hclge_dbg_cmd_send(hdev, desc, port_id, 2,
+ HCLGE_OPC_TM_INTERNAL_STS);
if (ret)
- goto err_tm_pg_cmd_send;
-
- pg_shap_cfg_cmd = (struct hclge_pg_shapping_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "PG_P pg_id: %u\n", pg_shap_cfg_cmd->pg_id);
- dev_info(&hdev->pdev->dev, "PG_P pg_shapping: 0x%x\n",
- le32_to_cpu(pg_shap_cfg_cmd->pg_shapping_para));
- dev_info(&hdev->pdev->dev, "PG_P flag: %#x\n", pg_shap_cfg_cmd->flag);
- dev_info(&hdev->pdev->dev, "PG_P pg_rate: %u(Mbps)\n",
- le32_to_cpu(pg_shap_cfg_cmd->pg_rate));
-
- cmd = HCLGE_OPC_TM_PORT_SHAPPING;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ return ret;
+
+ *pos += scnprintf(buf + *pos, len - *pos, "pri_bp: %#x\n",
+ le32_to_cpu(desc[0].data[1]));
+ *pos += scnprintf(buf + *pos, len - *pos, "fifo_dfx_info: %#x\n",
+ le32_to_cpu(desc[0].data[2]));
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "sch_roce_fifo_afull_gap: %#x\n",
+ le32_to_cpu(desc[0].data[3]));
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "tx_private_waterline: %#x\n",
+ le32_to_cpu(desc[0].data[4]));
+ *pos += scnprintf(buf + *pos, len - *pos, "tm_bypass_en: %#x\n",
+ le32_to_cpu(desc[0].data[5]));
+ *pos += scnprintf(buf + *pos, len - *pos, "SSU_TM_BYPASS_EN: %#x\n",
+ le32_to_cpu(desc[1].data[0]));
+ *pos += scnprintf(buf + *pos, len - *pos, "SSU_RESERVE_CFG: %#x\n",
+ le32_to_cpu(desc[1].data[1]));
+
+ if (hdev->hw.mac.media_type == HNAE3_MEDIA_TYPE_COPPER)
+ return 0;
+
+ ret = hclge_dbg_cmd_send(hdev, desc, port_id, 1,
+ HCLGE_OPC_TM_INTERNAL_STS_1);
if (ret)
- goto err_tm_pg_cmd_send;
+ return ret;
- port_shap_cfg_cmd = (struct hclge_port_shapping_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "PORT port_shapping: 0x%x\n",
- le32_to_cpu(port_shap_cfg_cmd->port_shapping_para));
- dev_info(&hdev->pdev->dev, "PORT flag: %#x\n", port_shap_cfg_cmd->flag);
- dev_info(&hdev->pdev->dev, "PORT port_rate: %u(Mbps)\n",
- le32_to_cpu(port_shap_cfg_cmd->port_rate));
+ *pos += scnprintf(buf + *pos, len - *pos, "TC_MAP_SEL: %#x\n",
+ le32_to_cpu(desc[0].data[1]));
+ *pos += scnprintf(buf + *pos, len - *pos, "IGU_PFC_PRI_EN: %#x\n",
+ le32_to_cpu(desc[0].data[2]));
+ *pos += scnprintf(buf + *pos, len - *pos, "MAC_PFC_PRI_EN: %#x\n",
+ le32_to_cpu(desc[0].data[3]));
+ *pos += scnprintf(buf + *pos, len - *pos, "IGU_PRI_MAP_TC_CFG: %#x\n",
+ le32_to_cpu(desc[0].data[4]));
+ *pos += scnprintf(buf + *pos, len - *pos,
+ "IGU_TX_PRI_MAP_TC_CFG: %#x\n",
+ le32_to_cpu(desc[0].data[5]));
- cmd = HCLGE_OPC_TM_PG_SCH_MODE_CFG;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ return 0;
+}
+
+static int hclge_dbg_dump_dcb(struct hclge_dev *hdev, char *buf, int len)
+{
+ int pos = 0;
+ int ret;
+
+ ret = hclge_dbg_dump_dcb_qset(hdev, buf, len, &pos);
if (ret)
- goto err_tm_pg_cmd_send;
+ return ret;
- dev_info(&hdev->pdev->dev, "PG_SCH pg_id: %u\n",
- le32_to_cpu(desc.data[0]));
+ ret = hclge_dbg_dump_dcb_pri(hdev, buf, len, &pos);
+ if (ret)
+ return ret;
- cmd = HCLGE_OPC_TM_PRI_SCH_MODE_CFG;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ ret = hclge_dbg_dump_dcb_pg(hdev, buf, len, &pos);
if (ret)
- goto err_tm_pg_cmd_send;
+ return ret;
- dev_info(&hdev->pdev->dev, "PRI_SCH pri_id: %u\n",
- le32_to_cpu(desc.data[0]));
+ ret = hclge_dbg_dump_dcb_queue(hdev, buf, len, &pos);
+ if (ret)
+ return ret;
- cmd = HCLGE_OPC_TM_QS_SCH_MODE_CFG;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ ret = hclge_dbg_dump_dcb_port(hdev, buf, len, &pos);
if (ret)
- goto err_tm_pg_cmd_send;
+ return ret;
- dev_info(&hdev->pdev->dev, "QS_SCH qs_id: %u\n",
- le32_to_cpu(desc.data[0]));
+ return hclge_dbg_dump_dcb_tm(hdev, buf, len, &pos);
+}
- if (!hnae3_dev_dcb_supported(hdev)) {
- dev_info(&hdev->pdev->dev,
- "Only DCB-supported dev supports tm mapping\n");
- return;
+static int hclge_dbg_dump_reg_cmd(struct hclge_dev *hdev,
+ enum hnae3_dbg_cmd cmd, char *buf, int len)
+{
+ const struct hclge_dbg_reg_type_info *reg_info;
+ int pos = 0, ret = 0;
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(hclge_dbg_reg_info); i++) {
+ reg_info = &hclge_dbg_reg_info[i];
+ if (cmd == reg_info->cmd) {
+ if (cmd == HNAE3_DBG_CMD_REG_TQP)
+ return hclge_dbg_dump_reg_tqp(hdev, reg_info,
+ buf, len, &pos);
+
+ ret = hclge_dbg_dump_reg_common(hdev, reg_info, buf,
+ len, &pos);
+ if (ret)
+ break;
+ }
}
- cmd = HCLGE_OPC_TM_BP_TO_QSET_MAPPING;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
- goto err_tm_pg_cmd_send;
-
- bp_to_qs_map_cmd = (struct hclge_bp_to_qs_map_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "BP_TO_QSET tc_id: %u\n",
- bp_to_qs_map_cmd->tc_id);
- dev_info(&hdev->pdev->dev, "BP_TO_QSET qs_group_id: 0x%x\n",
- bp_to_qs_map_cmd->qs_group_id);
- dev_info(&hdev->pdev->dev, "BP_TO_QSET qs_bit_map: 0x%x\n",
- le32_to_cpu(bp_to_qs_map_cmd->qs_bit_map));
- return;
-
-err_tm_pg_cmd_send:
- dev_err(&hdev->pdev->dev, "dump tm_pg fail(0x%x), ret = %d\n",
- cmd, ret);
+ return ret;
}
-static void hclge_dbg_dump_tm(struct hclge_dev *hdev)
+static int hclge_dbg_dump_tc(struct hclge_dev *hdev, char *buf, int len)
{
- struct hclge_priority_weight_cmd *priority_weight;
- struct hclge_pg_to_pri_link_cmd *pg_to_pri_map;
- struct hclge_qs_to_pri_link_cmd *qs_to_pri_map;
- struct hclge_nq_to_qs_link_cmd *nq_to_qs_map;
- struct hclge_pri_shapping_cmd *shap_cfg_cmd;
- struct hclge_pg_weight_cmd *pg_weight;
- struct hclge_qs_weight_cmd *qs_weight;
- enum hclge_opcode_type cmd;
+ struct hclge_ets_tc_weight_cmd *ets_weight;
struct hclge_desc desc;
+ char *sch_mode_str;
+ int pos = 0;
int ret;
+ u8 i;
+
+ if (!hnae3_dev_dcb_supported(hdev)) {
+ dev_err(&hdev->pdev->dev,
+ "Only DCB-supported dev supports tc\n");
+ return -EOPNOTSUPP;
+ }
- cmd = HCLGE_OPC_TM_PG_TO_PRI_LINK;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_ETS_TC_WEIGHT, true);
ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
- goto err_tm_cmd_send;
+ if (ret) {
+ dev_err(&hdev->pdev->dev, "failed to get tc weight, ret = %d\n",
+ ret);
+ return ret;
+ }
- pg_to_pri_map = (struct hclge_pg_to_pri_link_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "dump tm\n");
- dev_info(&hdev->pdev->dev, "PG_TO_PRI gp_id: %u\n",
- pg_to_pri_map->pg_id);
- dev_info(&hdev->pdev->dev, "PG_TO_PRI map: 0x%x\n",
- pg_to_pri_map->pri_bit_map);
+ ets_weight = (struct hclge_ets_tc_weight_cmd *)desc.data;
- cmd = HCLGE_OPC_TM_QS_TO_PRI_LINK;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
- goto err_tm_cmd_send;
-
- qs_to_pri_map = (struct hclge_qs_to_pri_link_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "QS_TO_PRI qs_id: %u\n",
- le16_to_cpu(qs_to_pri_map->qs_id));
- dev_info(&hdev->pdev->dev, "QS_TO_PRI priority: %u\n",
- qs_to_pri_map->priority);
- dev_info(&hdev->pdev->dev, "QS_TO_PRI link_vld: %u\n",
- qs_to_pri_map->link_vld);
-
- cmd = HCLGE_OPC_TM_NQ_TO_QS_LINK;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
- goto err_tm_cmd_send;
+ pos += scnprintf(buf + pos, len - pos, "enabled tc number: %u\n",
+ hdev->tm_info.num_tc);
+ pos += scnprintf(buf + pos, len - pos, "weight_offset: %u\n",
+ ets_weight->weight_offset);
+
+ pos += scnprintf(buf + pos, len - pos, "TC MODE WEIGHT\n");
+ for (i = 0; i < HNAE3_MAX_TC; i++) {
+ sch_mode_str = ets_weight->tc_weight[i] ? "dwrr" : "sp";
+ pos += scnprintf(buf + pos, len - pos, "%u %4s %3u\n",
+ i, sch_mode_str,
+ hdev->tm_info.pg_info[0].tc_dwrr[i]);
+ }
- nq_to_qs_map = (struct hclge_nq_to_qs_link_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "NQ_TO_QS nq_id: %u\n",
- le16_to_cpu(nq_to_qs_map->nq_id));
- dev_info(&hdev->pdev->dev, "NQ_TO_QS qset_id: 0x%x\n",
- le16_to_cpu(nq_to_qs_map->qset_id));
+ return 0;
+}
- cmd = HCLGE_OPC_TM_PG_WEIGHT;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
- goto err_tm_cmd_send;
+static const struct hclge_dbg_item tm_pg_items[] = {
+ { "ID", 2 },
+ { "PRI_MAP", 2 },
+ { "MODE", 2 },
+ { "DWRR", 2 },
+ { "C_IR_B", 2 },
+ { "C_IR_U", 2 },
+ { "C_IR_S", 2 },
+ { "C_BS_B", 2 },
+ { "C_BS_S", 2 },
+ { "C_FLAG", 2 },
+ { "C_RATE(Mbps)", 2 },
+ { "P_IR_B", 2 },
+ { "P_IR_U", 2 },
+ { "P_IR_S", 2 },
+ { "P_BS_B", 2 },
+ { "P_BS_S", 2 },
+ { "P_FLAG", 2 },
+ { "P_RATE(Mbps)", 0 }
+};
- pg_weight = (struct hclge_pg_weight_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "PG pg_id: %u\n", pg_weight->pg_id);
- dev_info(&hdev->pdev->dev, "PG dwrr: %u\n", pg_weight->dwrr);
+static void hclge_dbg_fill_shaper_content(struct hclge_tm_shaper_para *para,
+ char **result, u8 *index)
+{
+ sprintf(result[(*index)++], "%3u", para->ir_b);
+ sprintf(result[(*index)++], "%3u", para->ir_u);
+ sprintf(result[(*index)++], "%3u", para->ir_s);
+ sprintf(result[(*index)++], "%3u", para->bs_b);
+ sprintf(result[(*index)++], "%3u", para->bs_s);
+ sprintf(result[(*index)++], "%3u", para->flag);
+ sprintf(result[(*index)++], "%6u", para->rate);
+}
- cmd = HCLGE_OPC_TM_QS_WEIGHT;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
- goto err_tm_cmd_send;
+static int hclge_dbg_dump_tm_pg(struct hclge_dev *hdev, char *buf, int len)
+{
+ char data_str[ARRAY_SIZE(tm_pg_items)][HCLGE_DBG_DATA_STR_LEN];
+ struct hclge_tm_shaper_para c_shaper_para, p_shaper_para;
+ char *result[ARRAY_SIZE(tm_pg_items)], *sch_mode_str;
+ u8 pg_id, sch_mode, weight, pri_bit_map, i, j;
+ char content[HCLGE_DBG_TM_INFO_LEN];
+ int pos = 0;
+ int ret;
- qs_weight = (struct hclge_qs_weight_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "QS qs_id: %u\n",
- le16_to_cpu(qs_weight->qs_id));
- dev_info(&hdev->pdev->dev, "QS dwrr: %u\n", qs_weight->dwrr);
+ for (i = 0; i < ARRAY_SIZE(tm_pg_items); i++)
+ result[i] = &data_str[i][0];
- cmd = HCLGE_OPC_TM_PRI_WEIGHT;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
- goto err_tm_cmd_send;
+ hclge_dbg_fill_content(content, sizeof(content), tm_pg_items,
+ NULL, ARRAY_SIZE(tm_pg_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
- priority_weight = (struct hclge_priority_weight_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "PRI pri_id: %u\n", priority_weight->pri_id);
- dev_info(&hdev->pdev->dev, "PRI dwrr: %u\n", priority_weight->dwrr);
+ for (pg_id = 0; pg_id < hdev->tm_info.num_pg; pg_id++) {
+ ret = hclge_tm_get_pg_to_pri_map(hdev, pg_id, &pri_bit_map);
+ if (ret)
+ return ret;
- cmd = HCLGE_OPC_TM_PRI_C_SHAPPING;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
- goto err_tm_cmd_send;
-
- shap_cfg_cmd = (struct hclge_pri_shapping_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "PRI_C pri_id: %u\n", shap_cfg_cmd->pri_id);
- dev_info(&hdev->pdev->dev, "PRI_C pri_shapping: 0x%x\n",
- le32_to_cpu(shap_cfg_cmd->pri_shapping_para));
- dev_info(&hdev->pdev->dev, "PRI_C flag: %#x\n", shap_cfg_cmd->flag);
- dev_info(&hdev->pdev->dev, "PRI_C pri_rate: %u(Mbps)\n",
- le32_to_cpu(shap_cfg_cmd->pri_rate));
-
- cmd = HCLGE_OPC_TM_PRI_P_SHAPPING;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
- goto err_tm_cmd_send;
+ ret = hclge_tm_get_pg_sch_mode(hdev, pg_id, &sch_mode);
+ if (ret)
+ return ret;
- shap_cfg_cmd = (struct hclge_pri_shapping_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "PRI_P pri_id: %u\n", shap_cfg_cmd->pri_id);
- dev_info(&hdev->pdev->dev, "PRI_P pri_shapping: 0x%x\n",
- le32_to_cpu(shap_cfg_cmd->pri_shapping_para));
- dev_info(&hdev->pdev->dev, "PRI_P flag: %#x\n", shap_cfg_cmd->flag);
- dev_info(&hdev->pdev->dev, "PRI_P pri_rate: %u(Mbps)\n",
- le32_to_cpu(shap_cfg_cmd->pri_rate));
+ ret = hclge_tm_get_pg_weight(hdev, pg_id, &weight);
+ if (ret)
+ return ret;
- hclge_dbg_dump_tm_pg(hdev);
+ ret = hclge_tm_get_pg_shaper(hdev, pg_id,
+ HCLGE_OPC_TM_PG_C_SHAPPING,
+ &c_shaper_para);
+ if (ret)
+ return ret;
+
+ ret = hclge_tm_get_pg_shaper(hdev, pg_id,
+ HCLGE_OPC_TM_PG_P_SHAPPING,
+ &p_shaper_para);
+ if (ret)
+ return ret;
- return;
+ sch_mode_str = sch_mode & HCLGE_TM_TX_SCHD_DWRR_MSK ? "dwrr" :
+ "sp";
+
+ j = 0;
+ sprintf(result[j++], "%02u", pg_id);
+ sprintf(result[j++], "0x%02x", pri_bit_map);
+ sprintf(result[j++], "%4s", sch_mode_str);
+ sprintf(result[j++], "%3u", weight);
+ hclge_dbg_fill_shaper_content(&c_shaper_para, result, &j);
+ hclge_dbg_fill_shaper_content(&p_shaper_para, result, &j);
+
+ hclge_dbg_fill_content(content, sizeof(content), tm_pg_items,
+ (const char **)result,
+ ARRAY_SIZE(tm_pg_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
+ }
-err_tm_cmd_send:
- dev_err(&hdev->pdev->dev, "dump tm fail(0x%x), ret = %d\n",
- cmd, ret);
+ return 0;
}
-static void hclge_dbg_dump_tm_map(struct hclge_dev *hdev,
- const char *cmd_buf)
+static int hclge_dbg_dump_tm_port(struct hclge_dev *hdev, char *buf, int len)
{
- struct hclge_bp_to_qs_map_cmd *bp_to_qs_map_cmd;
- struct hclge_nq_to_qs_link_cmd *nq_to_qs_map;
- u32 qset_mapping[HCLGE_BP_EXT_GRP_NUM];
- struct hclge_qs_to_pri_link_cmd *map;
- struct hclge_tqp_tx_queue_tc_cmd *tc;
- u16 group_id, queue_id, qset_id;
- enum hclge_opcode_type cmd;
- u8 grp_num, pri_id, tc_id;
- struct hclge_desc desc;
- u16 qs_id_l;
- u16 qs_id_h;
+ struct hclge_tm_shaper_para shaper_para;
+ int pos = 0;
int ret;
- u32 i;
-
- ret = kstrtou16(cmd_buf, 0, &queue_id);
- queue_id = (ret != 0) ? 0 : queue_id;
- cmd = HCLGE_OPC_TM_NQ_TO_QS_LINK;
- nq_to_qs_map = (struct hclge_nq_to_qs_link_cmd *)desc.data;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- nq_to_qs_map->nq_id = cpu_to_le16(queue_id);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
- goto err_tm_map_cmd_send;
- qset_id = le16_to_cpu(nq_to_qs_map->qset_id);
-
- /* convert qset_id to the following format, drop the vld bit
- * | qs_id_h | vld | qs_id_l |
- * qset_id: | 15 ~ 11 | 10 | 9 ~ 0 |
- * \ \ / /
- * \ \ / /
- * qset_id: | 15 | 14 ~ 10 | 9 ~ 0 |
- */
- qs_id_l = hnae3_get_field(qset_id, HCLGE_TM_QS_ID_L_MSK,
- HCLGE_TM_QS_ID_L_S);
- qs_id_h = hnae3_get_field(qset_id, HCLGE_TM_QS_ID_H_EXT_MSK,
- HCLGE_TM_QS_ID_H_EXT_S);
- qset_id = 0;
- hnae3_set_field(qset_id, HCLGE_TM_QS_ID_L_MSK, HCLGE_TM_QS_ID_L_S,
- qs_id_l);
- hnae3_set_field(qset_id, HCLGE_TM_QS_ID_H_MSK, HCLGE_TM_QS_ID_H_S,
- qs_id_h);
-
- cmd = HCLGE_OPC_TM_QS_TO_PRI_LINK;
- map = (struct hclge_qs_to_pri_link_cmd *)desc.data;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- map->qs_id = cpu_to_le16(qset_id);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ ret = hclge_tm_get_port_shaper(hdev, &shaper_para);
if (ret)
- goto err_tm_map_cmd_send;
- pri_id = map->priority;
+ return ret;
- cmd = HCLGE_OPC_TQP_TX_QUEUE_TC;
- tc = (struct hclge_tqp_tx_queue_tc_cmd *)desc.data;
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- tc->queue_id = cpu_to_le16(queue_id);
- ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
- goto err_tm_map_cmd_send;
- tc_id = tc->tc_id & 0x7;
+ pos += scnprintf(buf + pos, len - pos,
+ "IR_B IR_U IR_S BS_B BS_S FLAG RATE(Mbps)\n");
+ pos += scnprintf(buf + pos, len - pos,
+ "%3u %3u %3u %3u %3u %1u %6u\n",
+ shaper_para.ir_b, shaper_para.ir_u, shaper_para.ir_s,
+ shaper_para.bs_b, shaper_para.bs_s, shaper_para.flag,
+ shaper_para.rate);
- dev_info(&hdev->pdev->dev, "queue_id | qset_id | pri_id | tc_id\n");
- dev_info(&hdev->pdev->dev, "%04u | %04u | %02u | %02u\n",
- queue_id, qset_id, pri_id, tc_id);
+ return 0;
+}
- if (!hnae3_dev_dcb_supported(hdev)) {
- dev_info(&hdev->pdev->dev,
- "Only DCB-supported dev supports tm mapping\n");
- return;
- }
+static int hclge_dbg_dump_tm_bp_qset_map(struct hclge_dev *hdev, u8 tc_id,
+ char *buf, int len)
+{
+ u32 qset_mapping[HCLGE_BP_EXT_GRP_NUM];
+ struct hclge_bp_to_qs_map_cmd *map;
+ struct hclge_desc desc;
+ int pos = 0;
+ u8 group_id;
+ u8 grp_num;
+ u16 i = 0;
+ int ret;
grp_num = hdev->num_tqps <= HCLGE_TQP_MAX_SIZE_DEV_V2 ?
HCLGE_BP_GRP_NUM : HCLGE_BP_EXT_GRP_NUM;
- cmd = HCLGE_OPC_TM_BP_TO_QSET_MAPPING;
- bp_to_qs_map_cmd = (struct hclge_bp_to_qs_map_cmd *)desc.data;
+ map = (struct hclge_bp_to_qs_map_cmd *)desc.data;
for (group_id = 0; group_id < grp_num; group_id++) {
- hclge_cmd_setup_basic_desc(&desc, cmd, true);
- bp_to_qs_map_cmd->tc_id = tc_id;
- bp_to_qs_map_cmd->qs_group_id = group_id;
+ hclge_cmd_setup_basic_desc(&desc,
+ HCLGE_OPC_TM_BP_TO_QSET_MAPPING,
+ true);
+ map->tc_id = tc_id;
+ map->qs_group_id = group_id;
ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
- goto err_tm_map_cmd_send;
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to get bp to qset map, ret = %d\n",
+ ret);
+ return ret;
+ }
- qset_mapping[group_id] =
- le32_to_cpu(bp_to_qs_map_cmd->qs_bit_map);
+ qset_mapping[group_id] = le32_to_cpu(map->qs_bit_map);
}
- dev_info(&hdev->pdev->dev, "index | tm bp qset maping:\n");
-
- i = 0;
+ pos += scnprintf(buf + pos, len - pos, "INDEX | TM BP QSET MAPPING:\n");
for (group_id = 0; group_id < grp_num / 8; group_id++) {
- dev_info(&hdev->pdev->dev,
+ pos += scnprintf(buf + pos, len - pos,
"%04d | %08x:%08x:%08x:%08x:%08x:%08x:%08x:%08x\n",
- group_id * 256, qset_mapping[(u32)(i + 7)],
- qset_mapping[(u32)(i + 6)], qset_mapping[(u32)(i + 5)],
- qset_mapping[(u32)(i + 4)], qset_mapping[(u32)(i + 3)],
- qset_mapping[(u32)(i + 2)], qset_mapping[(u32)(i + 1)],
+ group_id * 256, qset_mapping[i + 7],
+ qset_mapping[i + 6], qset_mapping[i + 5],
+ qset_mapping[i + 4], qset_mapping[i + 3],
+ qset_mapping[i + 2], qset_mapping[i + 1],
qset_mapping[i]);
i += 8;
}
- return;
+ return pos;
+}
+
+static int hclge_dbg_dump_tm_map(struct hclge_dev *hdev, char *buf, int len)
+{
+ u16 queue_id;
+ u16 qset_id;
+ u8 link_vld;
+ int pos = 0;
+ u8 pri_id;
+ u8 tc_id;
+ int ret;
+
+ for (queue_id = 0; queue_id < hdev->num_tqps; queue_id++) {
+ ret = hclge_tm_get_q_to_qs_map(hdev, queue_id, &qset_id);
+ if (ret)
+ return ret;
+
+ ret = hclge_tm_get_qset_map_pri(hdev, qset_id, &pri_id,
+ &link_vld);
+ if (ret)
+ return ret;
+
+ ret = hclge_tm_get_q_to_tc(hdev, queue_id, &tc_id);
+ if (ret)
+ return ret;
+
+ pos += scnprintf(buf + pos, len - pos,
+ "QUEUE_ID QSET_ID PRI_ID TC_ID\n");
+ pos += scnprintf(buf + pos, len - pos,
+ "%04u %4u %3u %2u\n",
+ queue_id, qset_id, pri_id, tc_id);
+
+ if (!hnae3_dev_dcb_supported(hdev))
+ continue;
+
+ ret = hclge_dbg_dump_tm_bp_qset_map(hdev, tc_id, buf + pos,
+ len - pos);
+ if (ret < 0)
+ return ret;
+ pos += ret;
+
+ pos += scnprintf(buf + pos, len - pos, "\n");
+ }
-err_tm_map_cmd_send:
- dev_err(&hdev->pdev->dev, "dump tqp map fail(0x%x), ret = %d\n",
- cmd, ret);
+ return 0;
}
static int hclge_dbg_dump_tm_nodes(struct hclge_dev *hdev, char *buf, int len)
static int hclge_dbg_dump_tm_pri(struct hclge_dev *hdev, char *buf, int len)
{
- struct hclge_pri_shaper_para c_shaper_para;
- struct hclge_pri_shaper_para p_shaper_para;
+ struct hclge_tm_shaper_para c_shaper_para;
+ struct hclge_tm_shaper_para p_shaper_para;
u8 pri_num, sch_mode, weight;
char *sch_mode_str;
int pos = 0;
return 0;
}
+static const struct hclge_dbg_item tm_qset_items[] = {
+ { "ID", 4 },
+ { "MAP_PRI", 2 },
+ { "LINK_VLD", 2 },
+ { "MODE", 2 },
+ { "DWRR", 2 },
+ { "IR_B", 2 },
+ { "IR_U", 2 },
+ { "IR_S", 2 },
+ { "BS_B", 2 },
+ { "BS_S", 2 },
+ { "FLAG", 2 },
+ { "RATE(Mbps)", 0 }
+};
+
static int hclge_dbg_dump_tm_qset(struct hclge_dev *hdev, char *buf, int len)
{
+ char data_str[ARRAY_SIZE(tm_qset_items)][HCLGE_DBG_DATA_STR_LEN];
+ char *result[ARRAY_SIZE(tm_qset_items)], *sch_mode_str;
u8 priority, link_vld, sch_mode, weight;
- char *sch_mode_str;
+ struct hclge_tm_shaper_para shaper_para;
+ char content[HCLGE_DBG_TM_INFO_LEN];
+ u16 qset_num, i;
int ret, pos;
- u16 qset_num;
- u16 i;
+ u8 j;
ret = hclge_tm_get_qset_num(hdev, &qset_num);
if (ret)
return ret;
- pos = scnprintf(buf, len, "ID MAP_PRI LINK_VLD MODE DWRR\n");
+ for (i = 0; i < ARRAY_SIZE(tm_qset_items); i++)
+ result[i] = &data_str[i][0];
+
+ hclge_dbg_fill_content(content, sizeof(content), tm_qset_items,
+ NULL, ARRAY_SIZE(tm_qset_items));
+ pos = scnprintf(buf, len, "%s", content);
for (i = 0; i < qset_num; i++) {
ret = hclge_tm_get_qset_map_pri(hdev, i, &priority, &link_vld);
if (ret)
return ret;
+ ret = hclge_tm_get_qset_shaper(hdev, i, &shaper_para);
+ if (ret)
+ return ret;
+
sch_mode_str = sch_mode & HCLGE_TM_TX_SCHD_DWRR_MSK ? "dwrr" :
"sp";
- pos += scnprintf(buf + pos, len - pos,
- "%04u %4u %1u %4s %3u\n",
- i, priority, link_vld, sch_mode_str, weight);
+
+ j = 0;
+ sprintf(result[j++], "%04u", i);
+ sprintf(result[j++], "%4u", priority);
+ sprintf(result[j++], "%4u", link_vld);
+ sprintf(result[j++], "%4s", sch_mode_str);
+ sprintf(result[j++], "%3u", weight);
+ hclge_dbg_fill_shaper_content(&shaper_para, result, &j);
+
+ hclge_dbg_fill_content(content, sizeof(content), tm_qset_items,
+ (const char **)result,
+ ARRAY_SIZE(tm_qset_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
}
return 0;
}
-static void hclge_dbg_dump_qos_pause_cfg(struct hclge_dev *hdev)
+static int hclge_dbg_dump_qos_pause_cfg(struct hclge_dev *hdev, char *buf,
+ int len)
{
struct hclge_cfg_pause_param_cmd *pause_param;
struct hclge_desc desc;
+ int pos = 0;
int ret;
hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_CFG_MAC_PARA, true);
-
ret = hclge_cmd_send(&hdev->hw, &desc, 1);
if (ret) {
- dev_err(&hdev->pdev->dev, "dump checksum fail, ret = %d\n",
- ret);
- return;
+ dev_err(&hdev->pdev->dev,
+ "failed to dump qos pause, ret = %d\n", ret);
+ return ret;
}
pause_param = (struct hclge_cfg_pause_param_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "dump qos pause cfg\n");
- dev_info(&hdev->pdev->dev, "pause_trans_gap: 0x%x\n",
- pause_param->pause_trans_gap);
- dev_info(&hdev->pdev->dev, "pause_trans_time: 0x%x\n",
- le16_to_cpu(pause_param->pause_trans_time));
+
+ pos += scnprintf(buf + pos, len - pos, "pause_trans_gap: 0x%x\n",
+ pause_param->pause_trans_gap);
+ pos += scnprintf(buf + pos, len - pos, "pause_trans_time: 0x%x\n",
+ le16_to_cpu(pause_param->pause_trans_time));
+ return 0;
}
-static void hclge_dbg_dump_qos_pri_map(struct hclge_dev *hdev)
+static int hclge_dbg_dump_qos_pri_map(struct hclge_dev *hdev, char *buf,
+ int len)
{
+#define HCLGE_DBG_TC_MASK 0x0F
+#define HCLGE_DBG_TC_BIT_WIDTH 4
+
struct hclge_qos_pri_map_cmd *pri_map;
struct hclge_desc desc;
+ int pos = 0;
+ u8 *pri_tc;
+ u8 tc, i;
int ret;
hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_PRI_TO_TC_MAPPING, true);
-
ret = hclge_cmd_send(&hdev->hw, &desc, 1);
if (ret) {
dev_err(&hdev->pdev->dev,
- "dump qos pri map fail, ret = %d\n", ret);
- return;
+ "failed to dump qos pri map, ret = %d\n", ret);
+ return ret;
}
pri_map = (struct hclge_qos_pri_map_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "dump qos pri map\n");
- dev_info(&hdev->pdev->dev, "vlan_to_pri: 0x%x\n", pri_map->vlan_pri);
- dev_info(&hdev->pdev->dev, "pri_0_to_tc: 0x%x\n", pri_map->pri0_tc);
- dev_info(&hdev->pdev->dev, "pri_1_to_tc: 0x%x\n", pri_map->pri1_tc);
- dev_info(&hdev->pdev->dev, "pri_2_to_tc: 0x%x\n", pri_map->pri2_tc);
- dev_info(&hdev->pdev->dev, "pri_3_to_tc: 0x%x\n", pri_map->pri3_tc);
- dev_info(&hdev->pdev->dev, "pri_4_to_tc: 0x%x\n", pri_map->pri4_tc);
- dev_info(&hdev->pdev->dev, "pri_5_to_tc: 0x%x\n", pri_map->pri5_tc);
- dev_info(&hdev->pdev->dev, "pri_6_to_tc: 0x%x\n", pri_map->pri6_tc);
- dev_info(&hdev->pdev->dev, "pri_7_to_tc: 0x%x\n", pri_map->pri7_tc);
+
+ pos += scnprintf(buf + pos, len - pos, "vlan_to_pri: 0x%x\n",
+ pri_map->vlan_pri);
+ pos += scnprintf(buf + pos, len - pos, "PRI TC\n");
+
+ pri_tc = (u8 *)pri_map;
+ for (i = 0; i < HNAE3_MAX_TC; i++) {
+ tc = pri_tc[i >> 1] >> ((i & 1) * HCLGE_DBG_TC_BIT_WIDTH);
+ tc &= HCLGE_DBG_TC_MASK;
+ pos += scnprintf(buf + pos, len - pos, "%u %u\n", i, tc);
+ }
+
+ return 0;
}
-static int hclge_dbg_dump_tx_buf_cfg(struct hclge_dev *hdev)
+static int hclge_dbg_dump_tx_buf_cfg(struct hclge_dev *hdev, char *buf, int len)
{
struct hclge_tx_buff_alloc_cmd *tx_buf_cmd;
struct hclge_desc desc;
+ int pos = 0;
int i, ret;
hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_TX_BUFF_ALLOC, true);
ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to dump tx buf, ret = %d\n", ret);
return ret;
+ }
- dev_info(&hdev->pdev->dev, "dump qos buf cfg\n");
tx_buf_cmd = (struct hclge_tx_buff_alloc_cmd *)desc.data;
for (i = 0; i < HCLGE_MAX_TC_NUM; i++)
- dev_info(&hdev->pdev->dev, "tx_packet_buf_tc_%d: 0x%x\n", i,
- le16_to_cpu(tx_buf_cmd->tx_pkt_buff[i]));
+ pos += scnprintf(buf + pos, len - pos,
+ "tx_packet_buf_tc_%d: 0x%x\n", i,
+ le16_to_cpu(tx_buf_cmd->tx_pkt_buff[i]));
- return 0;
+ return pos;
}
-static int hclge_dbg_dump_rx_priv_buf_cfg(struct hclge_dev *hdev)
+static int hclge_dbg_dump_rx_priv_buf_cfg(struct hclge_dev *hdev, char *buf,
+ int len)
{
struct hclge_rx_priv_buff_cmd *rx_buf_cmd;
struct hclge_desc desc;
+ int pos = 0;
int i, ret;
hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_RX_PRIV_BUFF_ALLOC, true);
ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to dump rx priv buf, ret = %d\n", ret);
return ret;
+ }
+
+ pos += scnprintf(buf + pos, len - pos, "\n");
- dev_info(&hdev->pdev->dev, "\n");
rx_buf_cmd = (struct hclge_rx_priv_buff_cmd *)desc.data;
for (i = 0; i < HCLGE_MAX_TC_NUM; i++)
- dev_info(&hdev->pdev->dev, "rx_packet_buf_tc_%d: 0x%x\n", i,
- le16_to_cpu(rx_buf_cmd->buf_num[i]));
+ pos += scnprintf(buf + pos, len - pos,
+ "rx_packet_buf_tc_%d: 0x%x\n", i,
+ le16_to_cpu(rx_buf_cmd->buf_num[i]));
- dev_info(&hdev->pdev->dev, "rx_share_buf: 0x%x\n",
- le16_to_cpu(rx_buf_cmd->shared_buf));
+ pos += scnprintf(buf + pos, len - pos, "rx_share_buf: 0x%x\n",
+ le16_to_cpu(rx_buf_cmd->shared_buf));
- return 0;
+ return pos;
}
-static int hclge_dbg_dump_rx_common_wl_cfg(struct hclge_dev *hdev)
+static int hclge_dbg_dump_rx_common_wl_cfg(struct hclge_dev *hdev, char *buf,
+ int len)
{
struct hclge_rx_com_wl *rx_com_wl;
struct hclge_desc desc;
+ int pos = 0;
int ret;
hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_RX_COM_WL_ALLOC, true);
ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to dump rx common wl, ret = %d\n", ret);
return ret;
+ }
rx_com_wl = (struct hclge_rx_com_wl *)desc.data;
- dev_info(&hdev->pdev->dev, "\n");
- dev_info(&hdev->pdev->dev, "rx_com_wl: high: 0x%x, low: 0x%x\n",
- le16_to_cpu(rx_com_wl->com_wl.high),
- le16_to_cpu(rx_com_wl->com_wl.low));
+ pos += scnprintf(buf + pos, len - pos, "\n");
+ pos += scnprintf(buf + pos, len - pos,
+ "rx_com_wl: high: 0x%x, low: 0x%x\n",
+ le16_to_cpu(rx_com_wl->com_wl.high),
+ le16_to_cpu(rx_com_wl->com_wl.low));
- return 0;
+ return pos;
}
-static int hclge_dbg_dump_rx_global_pkt_cnt(struct hclge_dev *hdev)
+static int hclge_dbg_dump_rx_global_pkt_cnt(struct hclge_dev *hdev, char *buf,
+ int len)
{
struct hclge_rx_com_wl *rx_packet_cnt;
struct hclge_desc desc;
+ int pos = 0;
int ret;
hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_RX_GBL_PKT_CNT, true);
ret = hclge_cmd_send(&hdev->hw, &desc, 1);
- if (ret)
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to dump rx global pkt cnt, ret = %d\n", ret);
return ret;
+ }
rx_packet_cnt = (struct hclge_rx_com_wl *)desc.data;
- dev_info(&hdev->pdev->dev,
- "rx_global_packet_cnt: high: 0x%x, low: 0x%x\n",
- le16_to_cpu(rx_packet_cnt->com_wl.high),
- le16_to_cpu(rx_packet_cnt->com_wl.low));
+ pos += scnprintf(buf + pos, len - pos,
+ "rx_global_packet_cnt: high: 0x%x, low: 0x%x\n",
+ le16_to_cpu(rx_packet_cnt->com_wl.high),
+ le16_to_cpu(rx_packet_cnt->com_wl.low));
- return 0;
+ return pos;
}
-static int hclge_dbg_dump_rx_priv_wl_buf_cfg(struct hclge_dev *hdev)
+static int hclge_dbg_dump_rx_priv_wl_buf_cfg(struct hclge_dev *hdev, char *buf,
+ int len)
{
struct hclge_rx_priv_wl_buf *rx_priv_wl;
struct hclge_desc desc[2];
+ int pos = 0;
int i, ret;
hclge_cmd_setup_basic_desc(&desc[0], HCLGE_OPC_RX_PRIV_WL_ALLOC, true);
desc[0].flag |= cpu_to_le16(HCLGE_CMD_FLAG_NEXT);
hclge_cmd_setup_basic_desc(&desc[1], HCLGE_OPC_RX_PRIV_WL_ALLOC, true);
ret = hclge_cmd_send(&hdev->hw, desc, 2);
- if (ret)
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to dump rx priv wl buf, ret = %d\n", ret);
return ret;
+ }
rx_priv_wl = (struct hclge_rx_priv_wl_buf *)desc[0].data;
for (i = 0; i < HCLGE_TC_NUM_ONE_DESC; i++)
- dev_info(&hdev->pdev->dev,
+ pos += scnprintf(buf + pos, len - pos,
"rx_priv_wl_tc_%d: high: 0x%x, low: 0x%x\n", i,
le16_to_cpu(rx_priv_wl->tc_wl[i].high),
le16_to_cpu(rx_priv_wl->tc_wl[i].low));
rx_priv_wl = (struct hclge_rx_priv_wl_buf *)desc[1].data;
for (i = 0; i < HCLGE_TC_NUM_ONE_DESC; i++)
- dev_info(&hdev->pdev->dev,
+ pos += scnprintf(buf + pos, len - pos,
"rx_priv_wl_tc_%d: high: 0x%x, low: 0x%x\n",
i + HCLGE_TC_NUM_ONE_DESC,
le16_to_cpu(rx_priv_wl->tc_wl[i].high),
le16_to_cpu(rx_priv_wl->tc_wl[i].low));
- return 0;
+ return pos;
}
-static int hclge_dbg_dump_rx_common_threshold_cfg(struct hclge_dev *hdev)
+static int hclge_dbg_dump_rx_common_threshold_cfg(struct hclge_dev *hdev,
+ char *buf, int len)
{
struct hclge_rx_com_thrd *rx_com_thrd;
struct hclge_desc desc[2];
+ int pos = 0;
int i, ret;
hclge_cmd_setup_basic_desc(&desc[0], HCLGE_OPC_RX_COM_THRD_ALLOC, true);
desc[0].flag |= cpu_to_le16(HCLGE_CMD_FLAG_NEXT);
hclge_cmd_setup_basic_desc(&desc[1], HCLGE_OPC_RX_COM_THRD_ALLOC, true);
ret = hclge_cmd_send(&hdev->hw, desc, 2);
- if (ret)
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to dump rx common threshold, ret = %d\n", ret);
return ret;
+ }
- dev_info(&hdev->pdev->dev, "\n");
+ pos += scnprintf(buf + pos, len - pos, "\n");
rx_com_thrd = (struct hclge_rx_com_thrd *)desc[0].data;
for (i = 0; i < HCLGE_TC_NUM_ONE_DESC; i++)
- dev_info(&hdev->pdev->dev,
+ pos += scnprintf(buf + pos, len - pos,
"rx_com_thrd_tc_%d: high: 0x%x, low: 0x%x\n", i,
le16_to_cpu(rx_com_thrd->com_thrd[i].high),
le16_to_cpu(rx_com_thrd->com_thrd[i].low));
rx_com_thrd = (struct hclge_rx_com_thrd *)desc[1].data;
for (i = 0; i < HCLGE_TC_NUM_ONE_DESC; i++)
- dev_info(&hdev->pdev->dev,
+ pos += scnprintf(buf + pos, len - pos,
"rx_com_thrd_tc_%d: high: 0x%x, low: 0x%x\n",
i + HCLGE_TC_NUM_ONE_DESC,
le16_to_cpu(rx_com_thrd->com_thrd[i].high),
le16_to_cpu(rx_com_thrd->com_thrd[i].low));
- return 0;
+ return pos;
}
-static void hclge_dbg_dump_qos_buf_cfg(struct hclge_dev *hdev)
+static int hclge_dbg_dump_qos_buf_cfg(struct hclge_dev *hdev, char *buf,
+ int len)
{
- enum hclge_opcode_type cmd;
+ int pos = 0;
int ret;
- cmd = HCLGE_OPC_TX_BUFF_ALLOC;
- ret = hclge_dbg_dump_tx_buf_cfg(hdev);
- if (ret)
- goto err_qos_cmd_send;
-
- cmd = HCLGE_OPC_RX_PRIV_BUFF_ALLOC;
- ret = hclge_dbg_dump_rx_priv_buf_cfg(hdev);
- if (ret)
- goto err_qos_cmd_send;
+ ret = hclge_dbg_dump_tx_buf_cfg(hdev, buf + pos, len - pos);
+ if (ret < 0)
+ return ret;
+ pos += ret;
- cmd = HCLGE_OPC_RX_COM_WL_ALLOC;
- ret = hclge_dbg_dump_rx_common_wl_cfg(hdev);
- if (ret)
- goto err_qos_cmd_send;
+ ret = hclge_dbg_dump_rx_priv_buf_cfg(hdev, buf + pos, len - pos);
+ if (ret < 0)
+ return ret;
+ pos += ret;
- cmd = HCLGE_OPC_RX_GBL_PKT_CNT;
- ret = hclge_dbg_dump_rx_global_pkt_cnt(hdev);
- if (ret)
- goto err_qos_cmd_send;
+ ret = hclge_dbg_dump_rx_common_wl_cfg(hdev, buf + pos, len - pos);
+ if (ret < 0)
+ return ret;
+ pos += ret;
- dev_info(&hdev->pdev->dev, "\n");
- if (!hnae3_dev_dcb_supported(hdev)) {
- dev_info(&hdev->pdev->dev,
- "Only DCB-supported dev supports rx priv wl\n");
- return;
- }
+ ret = hclge_dbg_dump_rx_global_pkt_cnt(hdev, buf + pos, len - pos);
+ if (ret < 0)
+ return ret;
+ pos += ret;
- cmd = HCLGE_OPC_RX_PRIV_WL_ALLOC;
- ret = hclge_dbg_dump_rx_priv_wl_buf_cfg(hdev);
- if (ret)
- goto err_qos_cmd_send;
+ pos += scnprintf(buf + pos, len - pos, "\n");
+ if (!hnae3_dev_dcb_supported(hdev))
+ return 0;
- cmd = HCLGE_OPC_RX_COM_THRD_ALLOC;
- ret = hclge_dbg_dump_rx_common_threshold_cfg(hdev);
- if (ret)
- goto err_qos_cmd_send;
+ ret = hclge_dbg_dump_rx_priv_wl_buf_cfg(hdev, buf + pos, len - pos);
+ if (ret < 0)
+ return ret;
+ pos += ret;
- return;
+ ret = hclge_dbg_dump_rx_common_threshold_cfg(hdev, buf + pos,
+ len - pos);
+ if (ret < 0)
+ return ret;
-err_qos_cmd_send:
- dev_err(&hdev->pdev->dev,
- "dump qos buf cfg fail(0x%x), ret = %d\n", cmd, ret);
+ return 0;
}
-static void hclge_dbg_dump_mng_table(struct hclge_dev *hdev)
+static int hclge_dbg_dump_mng_table(struct hclge_dev *hdev, char *buf, int len)
{
struct hclge_mac_ethertype_idx_rd_cmd *req0;
- char printf_buf[HCLGE_DBG_BUF_LEN];
struct hclge_desc desc;
u32 msg_egress_port;
+ int pos = 0;
int ret, i;
- dev_info(&hdev->pdev->dev, "mng tab:\n");
- memset(printf_buf, 0, HCLGE_DBG_BUF_LEN);
- strncat(printf_buf,
- "entry|mac_addr |mask|ether|mask|vlan|mask",
- HCLGE_DBG_BUF_LEN - 1);
- strncat(printf_buf + strlen(printf_buf),
- "|i_map|i_dir|e_type|pf_id|vf_id|q_id|drop\n",
- HCLGE_DBG_BUF_LEN - strlen(printf_buf) - 1);
-
- dev_info(&hdev->pdev->dev, "%s", printf_buf);
+ pos += scnprintf(buf + pos, len - pos,
+ "entry mac_addr mask ether ");
+ pos += scnprintf(buf + pos, len - pos,
+ "mask vlan mask i_map i_dir e_type ");
+ pos += scnprintf(buf + pos, len - pos, "pf_id vf_id q_id drop\n");
for (i = 0; i < HCLGE_DBG_MNG_TBL_MAX; i++) {
hclge_cmd_setup_basic_desc(&desc, HCLGE_MAC_ETHERTYPE_IDX_RD,
ret = hclge_cmd_send(&hdev->hw, &desc, 1);
if (ret) {
dev_err(&hdev->pdev->dev,
- "call hclge_cmd_send fail, ret = %d\n", ret);
- return;
+ "failed to dump manage table, ret = %d\n", ret);
+ return ret;
}
if (!req0->resp_code)
continue;
- memset(printf_buf, 0, HCLGE_DBG_BUF_LEN);
- snprintf(printf_buf, HCLGE_DBG_BUF_LEN,
- "%02u |%02x:%02x:%02x:%02x:%02x:%02x|",
- le16_to_cpu(req0->index),
- req0->mac_addr[0], req0->mac_addr[1],
- req0->mac_addr[2], req0->mac_addr[3],
- req0->mac_addr[4], req0->mac_addr[5]);
-
- snprintf(printf_buf + strlen(printf_buf),
- HCLGE_DBG_BUF_LEN - strlen(printf_buf),
- "%x |%04x |%x |%04x|%x |%02x |%02x |",
- !!(req0->flags & HCLGE_DBG_MNG_MAC_MASK_B),
- le16_to_cpu(req0->ethter_type),
- !!(req0->flags & HCLGE_DBG_MNG_ETHER_MASK_B),
- le16_to_cpu(req0->vlan_tag) & HCLGE_DBG_MNG_VLAN_TAG,
- !!(req0->flags & HCLGE_DBG_MNG_VLAN_MASK_B),
- req0->i_port_bitmap, req0->i_port_direction);
+ pos += scnprintf(buf + pos, len - pos, "%02u %pM ",
+ le16_to_cpu(req0->index), req0->mac_addr);
+
+ pos += scnprintf(buf + pos, len - pos,
+ "%x %04x %x %04x ",
+ !!(req0->flags & HCLGE_DBG_MNG_MAC_MASK_B),
+ le16_to_cpu(req0->ethter_type),
+ !!(req0->flags & HCLGE_DBG_MNG_ETHER_MASK_B),
+ le16_to_cpu(req0->vlan_tag) &
+ HCLGE_DBG_MNG_VLAN_TAG);
+
+ pos += scnprintf(buf + pos, len - pos,
+ "%x %02x %02x ",
+ !!(req0->flags & HCLGE_DBG_MNG_VLAN_MASK_B),
+ req0->i_port_bitmap, req0->i_port_direction);
msg_egress_port = le16_to_cpu(req0->egress_port);
- snprintf(printf_buf + strlen(printf_buf),
- HCLGE_DBG_BUF_LEN - strlen(printf_buf),
- "%x |%x |%02x |%04x|%x\n",
- !!(msg_egress_port & HCLGE_DBG_MNG_E_TYPE_B),
- msg_egress_port & HCLGE_DBG_MNG_PF_ID,
- (msg_egress_port >> 3) & HCLGE_DBG_MNG_VF_ID,
- le16_to_cpu(req0->egress_queue),
- !!(msg_egress_port & HCLGE_DBG_MNG_DROP_B));
-
- dev_info(&hdev->pdev->dev, "%s", printf_buf);
+ pos += scnprintf(buf + pos, len - pos,
+ "%x %x %02x %04x %x\n",
+ !!(msg_egress_port & HCLGE_DBG_MNG_E_TYPE_B),
+ msg_egress_port & HCLGE_DBG_MNG_PF_ID,
+ (msg_egress_port >> 3) & HCLGE_DBG_MNG_VF_ID,
+ le16_to_cpu(req0->egress_queue),
+ !!(msg_egress_port & HCLGE_DBG_MNG_DROP_B));
}
+
+ return 0;
}
-static int hclge_dbg_fd_tcam_read(struct hclge_dev *hdev, u8 stage,
- bool sel_x, u32 loc)
+#define HCLGE_DBG_TCAM_BUF_SIZE 256
+
+static int hclge_dbg_fd_tcam_read(struct hclge_dev *hdev, bool sel_x,
+ char *tcam_buf,
+ struct hclge_dbg_tcam_msg tcam_msg)
{
struct hclge_fd_tcam_config_1_cmd *req1;
struct hclge_fd_tcam_config_2_cmd *req2;
struct hclge_fd_tcam_config_3_cmd *req3;
struct hclge_desc desc[3];
+ int pos = 0;
int ret, i;
u32 *req;
req2 = (struct hclge_fd_tcam_config_2_cmd *)desc[1].data;
req3 = (struct hclge_fd_tcam_config_3_cmd *)desc[2].data;
- req1->stage = stage;
+ req1->stage = tcam_msg.stage;
req1->xy_sel = sel_x ? 1 : 0;
- req1->index = cpu_to_le32(loc);
+ req1->index = cpu_to_le32(tcam_msg.loc);
ret = hclge_cmd_send(&hdev->hw, desc, 3);
if (ret)
return ret;
- dev_info(&hdev->pdev->dev, " read result tcam key %s(%u):\n",
- sel_x ? "x" : "y", loc);
+ pos += scnprintf(tcam_buf + pos, HCLGE_DBG_TCAM_BUF_SIZE - pos,
+ "read result tcam key %s(%u):\n", sel_x ? "x" : "y",
+ tcam_msg.loc);
/* tcam_data0 ~ tcam_data1 */
req = (u32 *)req1->tcam_data;
for (i = 0; i < 2; i++)
- dev_info(&hdev->pdev->dev, "%08x\n", *req++);
+ pos += scnprintf(tcam_buf + pos, HCLGE_DBG_TCAM_BUF_SIZE - pos,
+ "%08x\n", *req++);
/* tcam_data2 ~ tcam_data7 */
req = (u32 *)req2->tcam_data;
for (i = 0; i < 6; i++)
- dev_info(&hdev->pdev->dev, "%08x\n", *req++);
+ pos += scnprintf(tcam_buf + pos, HCLGE_DBG_TCAM_BUF_SIZE - pos,
+ "%08x\n", *req++);
/* tcam_data8 ~ tcam_data12 */
req = (u32 *)req3->tcam_data;
for (i = 0; i < 5; i++)
- dev_info(&hdev->pdev->dev, "%08x\n", *req++);
+ pos += scnprintf(tcam_buf + pos, HCLGE_DBG_TCAM_BUF_SIZE - pos,
+ "%08x\n", *req++);
return ret;
}
}
spin_unlock_bh(&hdev->fd_rule_lock);
- if (cnt != hdev->hclge_fd_rule_num)
+ if (cnt != hdev->hclge_fd_rule_num || cnt == 0)
return -EINVAL;
return cnt;
}
-static void hclge_dbg_fd_tcam(struct hclge_dev *hdev)
+static int hclge_dbg_dump_fd_tcam(struct hclge_dev *hdev, char *buf, int len)
{
+ u32 rule_num = hdev->fd_cfg.rule_num[HCLGE_FD_STAGE_1];
+ struct hclge_dbg_tcam_msg tcam_msg;
int i, ret, rule_cnt;
u16 *rule_locs;
+ char *tcam_buf;
+ int pos = 0;
if (!hnae3_dev_fd_supported(hdev)) {
dev_err(&hdev->pdev->dev,
"Only FD-supported dev supports dump fd tcam\n");
- return;
+ return -EOPNOTSUPP;
}
- if (!hdev->hclge_fd_rule_num ||
- !hdev->fd_cfg.rule_num[HCLGE_FD_STAGE_1])
- return;
+ if (!hdev->hclge_fd_rule_num || !rule_num)
+ return 0;
- rule_locs = kcalloc(hdev->fd_cfg.rule_num[HCLGE_FD_STAGE_1],
- sizeof(u16), GFP_KERNEL);
+ rule_locs = kcalloc(rule_num, sizeof(u16), GFP_KERNEL);
if (!rule_locs)
- return;
+ return -ENOMEM;
+
+ tcam_buf = kzalloc(HCLGE_DBG_TCAM_BUF_SIZE, GFP_KERNEL);
+ if (!tcam_buf) {
+ kfree(rule_locs);
+ return -ENOMEM;
+ }
rule_cnt = hclge_dbg_get_rules_location(hdev, rule_locs);
- if (rule_cnt <= 0) {
+ if (rule_cnt < 0) {
+ ret = rule_cnt;
dev_err(&hdev->pdev->dev,
- "failed to get rule number, ret = %d\n", rule_cnt);
- kfree(rule_locs);
- return;
+ "failed to get rule number, ret = %d\n", ret);
+ goto out;
}
+ ret = 0;
for (i = 0; i < rule_cnt; i++) {
- ret = hclge_dbg_fd_tcam_read(hdev, 0, true, rule_locs[i]);
+ tcam_msg.stage = HCLGE_FD_STAGE_1;
+ tcam_msg.loc = rule_locs[i];
+
+ ret = hclge_dbg_fd_tcam_read(hdev, true, tcam_buf, tcam_msg);
if (ret) {
dev_err(&hdev->pdev->dev,
"failed to get fd tcam key x, ret = %d\n", ret);
- kfree(rule_locs);
- return;
+ goto out;
}
- ret = hclge_dbg_fd_tcam_read(hdev, 0, false, rule_locs[i]);
+ pos += scnprintf(buf + pos, len - pos, "%s", tcam_buf);
+
+ ret = hclge_dbg_fd_tcam_read(hdev, false, tcam_buf, tcam_msg);
if (ret) {
dev_err(&hdev->pdev->dev,
"failed to get fd tcam key y, ret = %d\n", ret);
- kfree(rule_locs);
- return;
+ goto out;
}
+
+ pos += scnprintf(buf + pos, len - pos, "%s", tcam_buf);
}
+out:
+ kfree(tcam_buf);
kfree(rule_locs);
+ return ret;
+}
+
+int hclge_dbg_dump_rst_info(struct hclge_dev *hdev, char *buf, int len)
+{
+ int pos = 0;
+
+ pos += scnprintf(buf + pos, len - pos, "PF reset count: %u\n",
+ hdev->rst_stats.pf_rst_cnt);
+ pos += scnprintf(buf + pos, len - pos, "FLR reset count: %u\n",
+ hdev->rst_stats.flr_rst_cnt);
+ pos += scnprintf(buf + pos, len - pos, "GLOBAL reset count: %u\n",
+ hdev->rst_stats.global_rst_cnt);
+ pos += scnprintf(buf + pos, len - pos, "IMP reset count: %u\n",
+ hdev->rst_stats.imp_rst_cnt);
+ pos += scnprintf(buf + pos, len - pos, "reset done count: %u\n",
+ hdev->rst_stats.reset_done_cnt);
+ pos += scnprintf(buf + pos, len - pos, "HW reset done count: %u\n",
+ hdev->rst_stats.hw_reset_done_cnt);
+ pos += scnprintf(buf + pos, len - pos, "reset count: %u\n",
+ hdev->rst_stats.reset_cnt);
+ pos += scnprintf(buf + pos, len - pos, "reset fail count: %u\n",
+ hdev->rst_stats.reset_fail_cnt);
+ pos += scnprintf(buf + pos, len - pos,
+ "vector0 interrupt enable status: 0x%x\n",
+ hclge_read_dev(&hdev->hw, HCLGE_MISC_VECTOR_REG_BASE));
+ pos += scnprintf(buf + pos, len - pos, "reset interrupt source: 0x%x\n",
+ hclge_read_dev(&hdev->hw, HCLGE_MISC_RESET_STS_REG));
+ pos += scnprintf(buf + pos, len - pos, "reset interrupt status: 0x%x\n",
+ hclge_read_dev(&hdev->hw, HCLGE_MISC_VECTOR_INT_STS));
+ pos += scnprintf(buf + pos, len - pos, "RAS interrupt status: 0x%x\n",
+ hclge_read_dev(&hdev->hw,
+ HCLGE_RAS_PF_OTHER_INT_STS_REG));
+ pos += scnprintf(buf + pos, len - pos, "hardware reset status: 0x%x\n",
+ hclge_read_dev(&hdev->hw, HCLGE_GLOBAL_RESET_REG));
+ pos += scnprintf(buf + pos, len - pos, "handshake status: 0x%x\n",
+ hclge_read_dev(&hdev->hw, HCLGE_NIC_CSQ_DEPTH_REG));
+ pos += scnprintf(buf + pos, len - pos, "function reset status: 0x%x\n",
+ hclge_read_dev(&hdev->hw, HCLGE_FUN_RST_ING));
+ pos += scnprintf(buf + pos, len - pos, "hdev state: 0x%lx\n",
+ hdev->state);
+
+ return 0;
}
-void hclge_dbg_dump_rst_info(struct hclge_dev *hdev)
+static int hclge_dbg_dump_serv_info(struct hclge_dev *hdev, char *buf, int len)
{
- dev_info(&hdev->pdev->dev, "PF reset count: %u\n",
- hdev->rst_stats.pf_rst_cnt);
- dev_info(&hdev->pdev->dev, "FLR reset count: %u\n",
- hdev->rst_stats.flr_rst_cnt);
- dev_info(&hdev->pdev->dev, "GLOBAL reset count: %u\n",
- hdev->rst_stats.global_rst_cnt);
- dev_info(&hdev->pdev->dev, "IMP reset count: %u\n",
- hdev->rst_stats.imp_rst_cnt);
- dev_info(&hdev->pdev->dev, "reset done count: %u\n",
- hdev->rst_stats.reset_done_cnt);
- dev_info(&hdev->pdev->dev, "HW reset done count: %u\n",
- hdev->rst_stats.hw_reset_done_cnt);
- dev_info(&hdev->pdev->dev, "reset count: %u\n",
- hdev->rst_stats.reset_cnt);
- dev_info(&hdev->pdev->dev, "reset fail count: %u\n",
- hdev->rst_stats.reset_fail_cnt);
- dev_info(&hdev->pdev->dev, "vector0 interrupt enable status: 0x%x\n",
- hclge_read_dev(&hdev->hw, HCLGE_MISC_VECTOR_REG_BASE));
- dev_info(&hdev->pdev->dev, "reset interrupt source: 0x%x\n",
- hclge_read_dev(&hdev->hw, HCLGE_MISC_RESET_STS_REG));
- dev_info(&hdev->pdev->dev, "reset interrupt status: 0x%x\n",
- hclge_read_dev(&hdev->hw, HCLGE_MISC_VECTOR_INT_STS));
- dev_info(&hdev->pdev->dev, "hardware reset status: 0x%x\n",
- hclge_read_dev(&hdev->hw, HCLGE_GLOBAL_RESET_REG));
- dev_info(&hdev->pdev->dev, "handshake status: 0x%x\n",
- hclge_read_dev(&hdev->hw, HCLGE_NIC_CSQ_DEPTH_REG));
- dev_info(&hdev->pdev->dev, "function reset status: 0x%x\n",
- hclge_read_dev(&hdev->hw, HCLGE_FUN_RST_ING));
- dev_info(&hdev->pdev->dev, "hdev state: 0x%lx\n", hdev->state);
+ unsigned long rem_nsec;
+ int pos = 0;
+ u64 lc;
+
+ lc = local_clock();
+ rem_nsec = do_div(lc, HCLGE_BILLION_NANO_SECONDS);
+
+ pos += scnprintf(buf + pos, len - pos, "local_clock: [%5lu.%06lu]\n",
+ (unsigned long)lc, rem_nsec / 1000);
+ pos += scnprintf(buf + pos, len - pos, "delta: %u(ms)\n",
+ jiffies_to_msecs(jiffies - hdev->last_serv_processed));
+ pos += scnprintf(buf + pos, len - pos,
+ "last_service_task_processed: %lu(jiffies)\n",
+ hdev->last_serv_processed);
+ pos += scnprintf(buf + pos, len - pos, "last_service_task_cnt: %lu\n",
+ hdev->serv_processed_cnt);
+
+ return 0;
}
-static void hclge_dbg_dump_serv_info(struct hclge_dev *hdev)
+static int hclge_dbg_dump_interrupt(struct hclge_dev *hdev, char *buf, int len)
{
- dev_info(&hdev->pdev->dev, "last_serv_processed: %lu\n",
- hdev->last_serv_processed);
- dev_info(&hdev->pdev->dev, "last_serv_cnt: %lu\n",
- hdev->serv_processed_cnt);
+ int pos = 0;
+
+ pos += scnprintf(buf + pos, len - pos, "num_nic_msi: %u\n",
+ hdev->num_nic_msi);
+ pos += scnprintf(buf + pos, len - pos, "num_roce_msi: %u\n",
+ hdev->num_roce_msi);
+ pos += scnprintf(buf + pos, len - pos, "num_msi_used: %u\n",
+ hdev->num_msi_used);
+ pos += scnprintf(buf + pos, len - pos, "num_msi_left: %u\n",
+ hdev->num_msi_left);
+
+ return 0;
}
-static void hclge_dbg_dump_interrupt(struct hclge_dev *hdev)
+static void hclge_dbg_imp_info_data_print(struct hclge_desc *desc_src,
+ char *buf, int len, u32 bd_num)
{
- dev_info(&hdev->pdev->dev, "num_nic_msi: %u\n", hdev->num_nic_msi);
- dev_info(&hdev->pdev->dev, "num_roce_msi: %u\n", hdev->num_roce_msi);
- dev_info(&hdev->pdev->dev, "num_msi_used: %u\n", hdev->num_msi_used);
- dev_info(&hdev->pdev->dev, "num_msi_left: %u\n", hdev->num_msi_left);
+#define HCLGE_DBG_IMP_INFO_PRINT_OFFSET 0x2
+
+ struct hclge_desc *desc_index = desc_src;
+ u32 offset = 0;
+ int pos = 0;
+ u32 i, j;
+
+ pos += scnprintf(buf + pos, len - pos, "offset | data\n");
+
+ for (i = 0; i < bd_num; i++) {
+ j = 0;
+ while (j < HCLGE_DESC_DATA_LEN - 1) {
+ pos += scnprintf(buf + pos, len - pos, "0x%04x | ",
+ offset);
+ pos += scnprintf(buf + pos, len - pos, "0x%08x ",
+ le32_to_cpu(desc_index->data[j++]));
+ pos += scnprintf(buf + pos, len - pos, "0x%08x\n",
+ le32_to_cpu(desc_index->data[j++]));
+ offset += sizeof(u32) * HCLGE_DBG_IMP_INFO_PRINT_OFFSET;
+ }
+ desc_index++;
+ }
}
-static void hclge_dbg_get_m7_stats_info(struct hclge_dev *hdev)
+static int
+hclge_dbg_get_imp_stats_info(struct hclge_dev *hdev, char *buf, int len)
{
- struct hclge_desc *desc_src, *desc_tmp;
- struct hclge_get_m7_bd_cmd *req;
+ struct hclge_get_imp_bd_cmd *req;
+ struct hclge_desc *desc_src;
struct hclge_desc desc;
- u32 bd_num, buf_len;
- int ret, i;
+ u32 bd_num;
+ int ret;
- hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_M7_STATS_BD, true);
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_IMP_STATS_BD, true);
- req = (struct hclge_get_m7_bd_cmd *)desc.data;
+ req = (struct hclge_get_imp_bd_cmd *)desc.data;
ret = hclge_cmd_send(&hdev->hw, &desc, 1);
if (ret) {
dev_err(&hdev->pdev->dev,
- "get firmware statistics bd number failed, ret = %d\n",
+ "failed to get imp statistics bd number, ret = %d\n",
ret);
- return;
+ return ret;
}
bd_num = le32_to_cpu(req->bd_num);
- buf_len = sizeof(struct hclge_desc) * bd_num;
- desc_src = kzalloc(buf_len, GFP_KERNEL);
+ desc_src = kcalloc(bd_num, sizeof(struct hclge_desc), GFP_KERNEL);
if (!desc_src)
- return;
+ return -ENOMEM;
- desc_tmp = desc_src;
- ret = hclge_dbg_cmd_send(hdev, desc_tmp, 0, bd_num,
- HCLGE_OPC_M7_STATS_INFO);
+ ret = hclge_dbg_cmd_send(hdev, desc_src, 0, bd_num,
+ HCLGE_OPC_IMP_STATS_INFO);
if (ret) {
kfree(desc_src);
dev_err(&hdev->pdev->dev,
- "get firmware statistics failed, ret = %d\n", ret);
- return;
+ "failed to get imp statistics, ret = %d\n", ret);
+ return ret;
}
- for (i = 0; i < bd_num; i++) {
- dev_info(&hdev->pdev->dev, "0x%08x 0x%08x 0x%08x\n",
- le32_to_cpu(desc_tmp->data[0]),
- le32_to_cpu(desc_tmp->data[1]),
- le32_to_cpu(desc_tmp->data[2]));
- dev_info(&hdev->pdev->dev, "0x%08x 0x%08x 0x%08x\n",
- le32_to_cpu(desc_tmp->data[3]),
- le32_to_cpu(desc_tmp->data[4]),
- le32_to_cpu(desc_tmp->data[5]));
-
- desc_tmp++;
- }
+ hclge_dbg_imp_info_data_print(desc_src, buf, len, bd_num);
kfree(desc_src);
+
+ return 0;
}
#define HCLGE_CMD_NCL_CONFIG_BD_NUM 5
+#define HCLGE_MAX_NCL_CONFIG_LENGTH 16384
-static void hclge_ncl_config_data_print(struct hclge_dev *hdev,
- struct hclge_desc *desc, int *offset,
- int *length)
+static void hclge_ncl_config_data_print(struct hclge_desc *desc, int *index,
+ char *buf, int *len, int *pos)
{
#define HCLGE_CMD_DATA_NUM 6
- int i;
- int j;
+ int offset = HCLGE_MAX_NCL_CONFIG_LENGTH - *index;
+ int i, j;
for (i = 0; i < HCLGE_CMD_NCL_CONFIG_BD_NUM; i++) {
for (j = 0; j < HCLGE_CMD_DATA_NUM; j++) {
if (i == 0 && j == 0)
continue;
- dev_info(&hdev->pdev->dev, "0x%04x | 0x%08x\n",
- *offset,
- le32_to_cpu(desc[i].data[j]));
- *offset += sizeof(u32);
- *length -= sizeof(u32);
- if (*length <= 0)
+ *pos += scnprintf(buf + *pos, *len - *pos,
+ "0x%04x | 0x%08x\n", offset,
+ le32_to_cpu(desc[i].data[j]));
+
+ offset += sizeof(u32);
+ *index -= sizeof(u32);
+
+ if (*index <= 0)
return;
}
}
}
-/* hclge_dbg_dump_ncl_config: print specified range of NCL_CONFIG file
- * @hdev: pointer to struct hclge_dev
- * @cmd_buf: string that contains offset and length
- */
-static void hclge_dbg_dump_ncl_config(struct hclge_dev *hdev,
- const char *cmd_buf)
+static int
+hclge_dbg_dump_ncl_config(struct hclge_dev *hdev, char *buf, int len)
{
-#define HCLGE_MAX_NCL_CONFIG_OFFSET 4096
#define HCLGE_NCL_CONFIG_LENGTH_IN_EACH_CMD (20 + 24 * 4)
-#define HCLGE_NCL_CONFIG_PARAM_NUM 2
struct hclge_desc desc[HCLGE_CMD_NCL_CONFIG_BD_NUM];
int bd_num = HCLGE_CMD_NCL_CONFIG_BD_NUM;
- int offset;
- int length;
- int data0;
+ int index = HCLGE_MAX_NCL_CONFIG_LENGTH;
+ int pos = 0;
+ u32 data0;
int ret;
- ret = sscanf(cmd_buf, "%x %x", &offset, &length);
- if (ret != HCLGE_NCL_CONFIG_PARAM_NUM) {
- dev_err(&hdev->pdev->dev,
- "Too few parameters, num = %d.\n", ret);
- return;
- }
-
- if (offset < 0 || offset >= HCLGE_MAX_NCL_CONFIG_OFFSET ||
- length <= 0 || length > HCLGE_MAX_NCL_CONFIG_OFFSET - offset) {
- dev_err(&hdev->pdev->dev,
- "Invalid input, offset = %d, length = %d.\n",
- offset, length);
- return;
- }
+ pos += scnprintf(buf + pos, len - pos, "offset | data\n");
- dev_info(&hdev->pdev->dev, "offset | data\n");
-
- while (length > 0) {
- data0 = offset;
- if (length >= HCLGE_NCL_CONFIG_LENGTH_IN_EACH_CMD)
+ while (index > 0) {
+ data0 = HCLGE_MAX_NCL_CONFIG_LENGTH - index;
+ if (index >= HCLGE_NCL_CONFIG_LENGTH_IN_EACH_CMD)
data0 |= HCLGE_NCL_CONFIG_LENGTH_IN_EACH_CMD << 16;
else
- data0 |= length << 16;
+ data0 |= (u32)index << 16;
ret = hclge_dbg_cmd_send(hdev, desc, data0, bd_num,
HCLGE_OPC_QUERY_NCL_CONFIG);
if (ret)
- return;
+ return ret;
- hclge_ncl_config_data_print(hdev, desc, &offset, &length);
+ hclge_ncl_config_data_print(desc, &index, buf, &len, &pos);
}
+
+ return 0;
}
-static void hclge_dbg_dump_loopback(struct hclge_dev *hdev)
+static int hclge_dbg_dump_loopback(struct hclge_dev *hdev, char *buf, int len)
{
struct phy_device *phydev = hdev->hw.mac.phydev;
struct hclge_config_mac_mode_cmd *req_app;
struct hclge_common_lb_cmd *req_common;
struct hclge_desc desc;
u8 loopback_en;
+ int pos = 0;
int ret;
req_app = (struct hclge_config_mac_mode_cmd *)desc.data;
req_common = (struct hclge_common_lb_cmd *)desc.data;
- dev_info(&hdev->pdev->dev, "mac id: %u\n", hdev->hw.mac.mac_id);
+ pos += scnprintf(buf + pos, len - pos, "mac id: %u\n",
+ hdev->hw.mac.mac_id);
hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_CONFIG_MAC_MODE, true);
ret = hclge_cmd_send(&hdev->hw, &desc, 1);
if (ret) {
dev_err(&hdev->pdev->dev,
"failed to dump app loopback status, ret = %d\n", ret);
- return;
+ return ret;
}
loopback_en = hnae3_get_bit(le32_to_cpu(req_app->txrx_pad_fcs_loop_en),
HCLGE_MAC_APP_LP_B);
- dev_info(&hdev->pdev->dev, "app loopback: %s\n",
- loopback_en ? "on" : "off");
+ pos += scnprintf(buf + pos, len - pos, "app loopback: %s\n",
+ state_str[loopback_en]);
hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_COMMON_LOOPBACK, true);
ret = hclge_cmd_send(&hdev->hw, &desc, 1);
dev_err(&hdev->pdev->dev,
"failed to dump common loopback status, ret = %d\n",
ret);
- return;
+ return ret;
}
loopback_en = req_common->enable & HCLGE_CMD_SERDES_SERIAL_INNER_LOOP_B;
- dev_info(&hdev->pdev->dev, "serdes serial loopback: %s\n",
- loopback_en ? "on" : "off");
+ pos += scnprintf(buf + pos, len - pos, "serdes serial loopback: %s\n",
+ state_str[loopback_en]);
loopback_en = req_common->enable &
- HCLGE_CMD_SERDES_PARALLEL_INNER_LOOP_B;
- dev_info(&hdev->pdev->dev, "serdes parallel loopback: %s\n",
- loopback_en ? "on" : "off");
+ HCLGE_CMD_SERDES_PARALLEL_INNER_LOOP_B ? 1 : 0;
+ pos += scnprintf(buf + pos, len - pos, "serdes parallel loopback: %s\n",
+ state_str[loopback_en]);
if (phydev) {
- dev_info(&hdev->pdev->dev, "phy loopback: %s\n",
- phydev->loopback_enabled ? "on" : "off");
+ loopback_en = phydev->loopback_enabled;
+ pos += scnprintf(buf + pos, len - pos, "phy loopback: %s\n",
+ state_str[loopback_en]);
} else if (hnae3_dev_phy_imp_supported(hdev)) {
loopback_en = req_common->enable &
HCLGE_CMD_GE_PHY_INNER_LOOP_B;
- dev_info(&hdev->pdev->dev, "phy loopback: %s\n",
- loopback_en ? "on" : "off");
+ pos += scnprintf(buf + pos, len - pos, "phy loopback: %s\n",
+ state_str[loopback_en]);
}
+
+ return 0;
}
/* hclge_dbg_dump_mac_tnl_status: print message about mac tnl interrupt
* @hdev: pointer to struct hclge_dev
*/
-static void hclge_dbg_dump_mac_tnl_status(struct hclge_dev *hdev)
+static int
+hclge_dbg_dump_mac_tnl_status(struct hclge_dev *hdev, char *buf, int len)
{
-#define HCLGE_BILLION_NANO_SECONDS 1000000000
-
struct hclge_mac_tnl_stats stats;
unsigned long rem_nsec;
+ int pos = 0;
- dev_info(&hdev->pdev->dev, "Recently generated mac tnl interruption:\n");
+ pos += scnprintf(buf + pos, len - pos,
+ "Recently generated mac tnl interruption:\n");
while (kfifo_get(&hdev->mac_tnl_log, &stats)) {
rem_nsec = do_div(stats.time, HCLGE_BILLION_NANO_SECONDS);
- dev_info(&hdev->pdev->dev, "[%07lu.%03lu] status = 0x%x\n",
- (unsigned long)stats.time, rem_nsec / 1000,
- stats.status);
+
+ pos += scnprintf(buf + pos, len - pos,
+ "[%07lu.%03lu] status = 0x%x\n",
+ (unsigned long)stats.time, rem_nsec / 1000,
+ stats.status);
+ }
+
+ return 0;
+}
+
+
+static const struct hclge_dbg_item mac_list_items[] = {
+ { "FUNC_ID", 2 },
+ { "MAC_ADDR", 12 },
+ { "STATE", 2 },
+};
+
+static void hclge_dbg_dump_mac_list(struct hclge_dev *hdev, char *buf, int len,
+ bool is_unicast)
+{
+ char data_str[ARRAY_SIZE(mac_list_items)][HCLGE_DBG_DATA_STR_LEN];
+ char content[HCLGE_DBG_INFO_LEN], str_id[HCLGE_DBG_ID_LEN];
+ char *result[ARRAY_SIZE(mac_list_items)];
+ struct hclge_mac_node *mac_node, *tmp;
+ struct hclge_vport *vport;
+ struct list_head *list;
+ u32 func_id;
+ int pos = 0;
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(mac_list_items); i++)
+ result[i] = &data_str[i][0];
+
+ pos += scnprintf(buf + pos, len - pos, "%s MAC_LIST:\n",
+ is_unicast ? "UC" : "MC");
+ hclge_dbg_fill_content(content, sizeof(content), mac_list_items,
+ NULL, ARRAY_SIZE(mac_list_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
+
+ for (func_id = 0; func_id < hdev->num_alloc_vport; func_id++) {
+ vport = &hdev->vport[func_id];
+ list = is_unicast ? &vport->uc_mac_list : &vport->mc_mac_list;
+ spin_lock_bh(&vport->mac_list_lock);
+ list_for_each_entry_safe(mac_node, tmp, list, node) {
+ i = 0;
+ result[i++] = hclge_dbg_get_func_id_str(str_id,
+ func_id);
+ sprintf(result[i++], "%pM", mac_node->mac_addr);
+ sprintf(result[i++], "%5s",
+ hclge_mac_state_str[mac_node->state]);
+ hclge_dbg_fill_content(content, sizeof(content),
+ mac_list_items,
+ (const char **)result,
+ ARRAY_SIZE(mac_list_items));
+ pos += scnprintf(buf + pos, len - pos, "%s", content);
+ }
+ spin_unlock_bh(&vport->mac_list_lock);
}
}
-static void hclge_dbg_dump_qs_shaper_single(struct hclge_dev *hdev, u16 qsid)
+static int hclge_get_vlan_rx_offload_cfg(struct hclge_dev *hdev, u8 vf_id,
+ struct hclge_dbg_vlan_cfg *vlan_cfg)
{
- struct hclge_qs_shapping_cmd *shap_cfg_cmd;
- u8 ir_u, ir_b, ir_s, bs_b, bs_s;
+ struct hclge_vport_vtag_rx_cfg_cmd *req;
struct hclge_desc desc;
- u32 shapping_para;
- u32 rate;
+ u16 bmap_index;
+ u8 rx_cfg;
int ret;
- hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_QCN_SHAPPING_CFG, true);
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_VLAN_PORT_RX_CFG, true);
- shap_cfg_cmd = (struct hclge_qs_shapping_cmd *)desc.data;
- shap_cfg_cmd->qs_id = cpu_to_le16(qsid);
+ req = (struct hclge_vport_vtag_rx_cfg_cmd *)desc.data;
+ req->vf_offset = vf_id / HCLGE_VF_NUM_PER_CMD;
+ bmap_index = vf_id % HCLGE_VF_NUM_PER_CMD / HCLGE_VF_NUM_PER_BYTE;
+ req->vf_bitmap[bmap_index] = 1U << (vf_id % HCLGE_VF_NUM_PER_BYTE);
ret = hclge_cmd_send(&hdev->hw, &desc, 1);
if (ret) {
dev_err(&hdev->pdev->dev,
- "qs%u failed to get tx_rate, ret=%d\n",
- qsid, ret);
- return;
+ "failed to get vport%u rxvlan cfg, ret = %d\n",
+ vf_id, ret);
+ return ret;
}
- shapping_para = le32_to_cpu(shap_cfg_cmd->qs_shapping_para);
- ir_b = hclge_tm_get_field(shapping_para, IR_B);
- ir_u = hclge_tm_get_field(shapping_para, IR_U);
- ir_s = hclge_tm_get_field(shapping_para, IR_S);
- bs_b = hclge_tm_get_field(shapping_para, BS_B);
- bs_s = hclge_tm_get_field(shapping_para, BS_S);
- rate = le32_to_cpu(shap_cfg_cmd->qs_rate);
-
- dev_info(&hdev->pdev->dev,
- "qs%u ir_b:%u, ir_u:%u, ir_s:%u, bs_b:%u, bs_s:%u, flag:%#x, rate:%u(Mbps)\n",
- qsid, ir_b, ir_u, ir_s, bs_b, bs_s, shap_cfg_cmd->flag, rate);
+ rx_cfg = req->vport_vlan_cfg;
+ vlan_cfg->strip_tag1 = hnae3_get_bit(rx_cfg, HCLGE_REM_TAG1_EN_B);
+ vlan_cfg->strip_tag2 = hnae3_get_bit(rx_cfg, HCLGE_REM_TAG2_EN_B);
+ vlan_cfg->drop_tag1 = hnae3_get_bit(rx_cfg, HCLGE_DISCARD_TAG1_EN_B);
+ vlan_cfg->drop_tag2 = hnae3_get_bit(rx_cfg, HCLGE_DISCARD_TAG2_EN_B);
+ vlan_cfg->pri_only1 = hnae3_get_bit(rx_cfg, HCLGE_SHOW_TAG1_EN_B);
+ vlan_cfg->pri_only2 = hnae3_get_bit(rx_cfg, HCLGE_SHOW_TAG2_EN_B);
+
+ return 0;
}
-static void hclge_dbg_dump_qs_shaper_all(struct hclge_dev *hdev)
+static int hclge_get_vlan_tx_offload_cfg(struct hclge_dev *hdev, u8 vf_id,
+ struct hclge_dbg_vlan_cfg *vlan_cfg)
{
- struct hnae3_knic_private_info *kinfo;
- struct hclge_vport *vport;
- int vport_id, i;
+ struct hclge_vport_vtag_tx_cfg_cmd *req;
+ struct hclge_desc desc;
+ u16 bmap_index;
+ u8 tx_cfg;
+ int ret;
+
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_VLAN_PORT_TX_CFG, true);
+ req = (struct hclge_vport_vtag_tx_cfg_cmd *)desc.data;
+ req->vf_offset = vf_id / HCLGE_VF_NUM_PER_CMD;
+ bmap_index = vf_id % HCLGE_VF_NUM_PER_CMD / HCLGE_VF_NUM_PER_BYTE;
+ req->vf_bitmap[bmap_index] = 1U << (vf_id % HCLGE_VF_NUM_PER_BYTE);
+
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to get vport%u txvlan cfg, ret = %d\n",
+ vf_id, ret);
+ return ret;
+ }
- for (vport_id = 0; vport_id <= pci_num_vf(hdev->pdev); vport_id++) {
- vport = &hdev->vport[vport_id];
- kinfo = &vport->nic.kinfo;
+ tx_cfg = req->vport_vlan_cfg;
+ vlan_cfg->pvid = le16_to_cpu(req->def_vlan_tag1);
- dev_info(&hdev->pdev->dev, "qs cfg of vport%d:\n", vport_id);
+ vlan_cfg->accept_tag1 = hnae3_get_bit(tx_cfg, HCLGE_ACCEPT_TAG1_B);
+ vlan_cfg->accept_tag2 = hnae3_get_bit(tx_cfg, HCLGE_ACCEPT_TAG2_B);
+ vlan_cfg->accept_untag1 = hnae3_get_bit(tx_cfg, HCLGE_ACCEPT_UNTAG1_B);
+ vlan_cfg->accept_untag2 = hnae3_get_bit(tx_cfg, HCLGE_ACCEPT_UNTAG2_B);
+ vlan_cfg->insert_tag1 = hnae3_get_bit(tx_cfg, HCLGE_PORT_INS_TAG1_EN_B);
+ vlan_cfg->insert_tag2 = hnae3_get_bit(tx_cfg, HCLGE_PORT_INS_TAG2_EN_B);
+ vlan_cfg->shift_tag = hnae3_get_bit(tx_cfg, HCLGE_TAG_SHIFT_MODE_EN_B);
- for (i = 0; i < kinfo->tc_info.num_tc; i++) {
- u16 qsid = vport->qs_offset + i;
+ return 0;
+}
- hclge_dbg_dump_qs_shaper_single(hdev, qsid);
- }
- }
+static int hclge_get_vlan_filter_config_cmd(struct hclge_dev *hdev,
+ u8 vlan_type, u8 vf_id,
+ struct hclge_desc *desc)
+{
+ struct hclge_vlan_filter_ctrl_cmd *req;
+ int ret;
+
+ hclge_cmd_setup_basic_desc(desc, HCLGE_OPC_VLAN_FILTER_CTRL, true);
+ req = (struct hclge_vlan_filter_ctrl_cmd *)desc->data;
+ req->vlan_type = vlan_type;
+ req->vf_id = vf_id;
+
+ ret = hclge_cmd_send(&hdev->hw, desc, 1);
+ if (ret)
+ dev_err(&hdev->pdev->dev,
+ "failed to get vport%u vlan filter config, ret = %d.\n",
+ vf_id, ret);
+
+ return ret;
}
-static void hclge_dbg_dump_qs_shaper(struct hclge_dev *hdev,
- const char *cmd_buf)
+static int hclge_get_vlan_filter_state(struct hclge_dev *hdev, u8 vlan_type,
+ u8 vf_id, u8 *vlan_fe)
{
- u16 qsid;
+ struct hclge_vlan_filter_ctrl_cmd *req;
+ struct hclge_desc desc;
int ret;
- ret = kstrtou16(cmd_buf, 0, &qsid);
- if (ret) {
- hclge_dbg_dump_qs_shaper_all(hdev);
- return;
- }
+ ret = hclge_get_vlan_filter_config_cmd(hdev, vlan_type, vf_id, &desc);
+ if (ret)
+ return ret;
- if (qsid >= hdev->ae_dev->dev_specs.max_qset_num) {
- dev_err(&hdev->pdev->dev, "qsid(%u) out of range[0-%u]\n",
- qsid, hdev->ae_dev->dev_specs.max_qset_num - 1);
- return;
- }
+ req = (struct hclge_vlan_filter_ctrl_cmd *)desc.data;
+ *vlan_fe = req->vlan_fe;
- hclge_dbg_dump_qs_shaper_single(hdev, qsid);
+ return 0;
}
-static int hclge_dbg_dump_mac_list(struct hclge_dev *hdev, const char *cmd_buf,
- bool is_unicast)
+static int hclge_get_port_vlan_filter_bypass_state(struct hclge_dev *hdev,
+ u8 vf_id, u8 *bypass_en)
{
- struct hclge_mac_node *mac_node, *tmp;
- struct hclge_vport *vport;
- struct list_head *list;
- u32 func_id;
+ struct hclge_port_vlan_filter_bypass_cmd *req;
+ struct hclge_desc desc;
int ret;
- ret = kstrtouint(cmd_buf, 0, &func_id);
- if (ret < 0) {
+ if (!test_bit(HNAE3_DEV_SUPPORT_PORT_VLAN_BYPASS_B, hdev->ae_dev->caps))
+ return 0;
+
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_PORT_VLAN_BYPASS, true);
+ req = (struct hclge_port_vlan_filter_bypass_cmd *)desc.data;
+ req->vf_id = vf_id;
+
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ if (ret) {
dev_err(&hdev->pdev->dev,
- "dump mac list: bad command string, ret = %d\n", ret);
- return -EINVAL;
+ "failed to get vport%u port vlan filter bypass state, ret = %d.\n",
+ vf_id, ret);
+ return ret;
}
- if (func_id >= hdev->num_alloc_vport) {
- dev_err(&hdev->pdev->dev,
- "function id(%u) is out of range(0-%u)\n", func_id,
- hdev->num_alloc_vport - 1);
- return -EINVAL;
+ *bypass_en = hnae3_get_bit(req->bypass_state, HCLGE_INGRESS_BYPASS_B);
+
+ return 0;
+}
+
+static const struct hclge_dbg_item vlan_filter_items[] = {
+ { "FUNC_ID", 2 },
+ { "I_VF_VLAN_FILTER", 2 },
+ { "E_VF_VLAN_FILTER", 2 },
+ { "PORT_VLAN_FILTER_BYPASS", 0 }
+};
+
+static const struct hclge_dbg_item vlan_offload_items[] = {
+ { "FUNC_ID", 2 },
+ { "PVID", 4 },
+ { "ACCEPT_TAG1", 2 },
+ { "ACCEPT_TAG2", 2 },
+ { "ACCEPT_UNTAG1", 2 },
+ { "ACCEPT_UNTAG2", 2 },
+ { "INSERT_TAG1", 2 },
+ { "INSERT_TAG2", 2 },
+ { "SHIFT_TAG", 2 },
+ { "STRIP_TAG1", 2 },
+ { "STRIP_TAG2", 2 },
+ { "DROP_TAG1", 2 },
+ { "DROP_TAG2", 2 },
+ { "PRI_ONLY_TAG1", 2 },
+ { "PRI_ONLY_TAG2", 0 }
+};
+
+static int hclge_dbg_dump_vlan_filter_config(struct hclge_dev *hdev, char *buf,
+ int len, int *pos)
+{
+ char content[HCLGE_DBG_VLAN_FLTR_INFO_LEN], str_id[HCLGE_DBG_ID_LEN];
+ const char *result[ARRAY_SIZE(vlan_filter_items)];
+ u8 i, j, vlan_fe, bypass, ingress, egress;
+ u8 func_num = pci_num_vf(hdev->pdev) + 1; /* pf and enabled vf num */
+ int ret;
+
+ ret = hclge_get_vlan_filter_state(hdev, HCLGE_FILTER_TYPE_PORT, 0,
+ &vlan_fe);
+ if (ret)
+ return ret;
+ ingress = vlan_fe & HCLGE_FILTER_FE_NIC_INGRESS_B;
+ egress = vlan_fe & HCLGE_FILTER_FE_NIC_EGRESS_B ? 1 : 0;
+
+ *pos += scnprintf(buf, len, "I_PORT_VLAN_FILTER: %s\n",
+ state_str[ingress]);
+ *pos += scnprintf(buf + *pos, len - *pos, "E_PORT_VLAN_FILTER: %s\n",
+ state_str[egress]);
+
+ hclge_dbg_fill_content(content, sizeof(content), vlan_filter_items,
+ NULL, ARRAY_SIZE(vlan_filter_items));
+ *pos += scnprintf(buf + *pos, len - *pos, "%s", content);
+
+ for (i = 0; i < func_num; i++) {
+ ret = hclge_get_vlan_filter_state(hdev, HCLGE_FILTER_TYPE_VF, i,
+ &vlan_fe);
+ if (ret)
+ return ret;
+
+ ingress = vlan_fe & HCLGE_FILTER_FE_NIC_INGRESS_B;
+ egress = vlan_fe & HCLGE_FILTER_FE_NIC_EGRESS_B ? 1 : 0;
+ ret = hclge_get_port_vlan_filter_bypass_state(hdev, i, &bypass);
+ if (ret)
+ return ret;
+ j = 0;
+ result[j++] = hclge_dbg_get_func_id_str(str_id, i);
+ result[j++] = state_str[ingress];
+ result[j++] = state_str[egress];
+ result[j++] =
+ test_bit(HNAE3_DEV_SUPPORT_PORT_VLAN_BYPASS_B,
+ hdev->ae_dev->caps) ? state_str[bypass] : "NA";
+ hclge_dbg_fill_content(content, sizeof(content),
+ vlan_filter_items, result,
+ ARRAY_SIZE(vlan_filter_items));
+ *pos += scnprintf(buf + *pos, len - *pos, "%s", content);
}
+ *pos += scnprintf(buf + *pos, len - *pos, "\n");
+
+ return 0;
+}
- vport = &hdev->vport[func_id];
+static int hclge_dbg_dump_vlan_offload_config(struct hclge_dev *hdev, char *buf,
+ int len, int *pos)
+{
+ char str_id[HCLGE_DBG_ID_LEN], str_pvid[HCLGE_DBG_ID_LEN];
+ const char *result[ARRAY_SIZE(vlan_offload_items)];
+ char content[HCLGE_DBG_VLAN_OFFLOAD_INFO_LEN];
+ u8 func_num = pci_num_vf(hdev->pdev) + 1; /* pf and enabled vf num */
+ struct hclge_dbg_vlan_cfg vlan_cfg;
+ int ret;
+ u8 i, j;
- list = is_unicast ? &vport->uc_mac_list : &vport->mc_mac_list;
+ hclge_dbg_fill_content(content, sizeof(content), vlan_offload_items,
+ NULL, ARRAY_SIZE(vlan_offload_items));
+ *pos += scnprintf(buf + *pos, len - *pos, "%s", content);
- dev_info(&hdev->pdev->dev, "vport %u %s mac list:\n",
- func_id, is_unicast ? "uc" : "mc");
- dev_info(&hdev->pdev->dev, "mac address state\n");
+ for (i = 0; i < func_num; i++) {
+ ret = hclge_get_vlan_tx_offload_cfg(hdev, i, &vlan_cfg);
+ if (ret)
+ return ret;
- spin_lock_bh(&vport->mac_list_lock);
+ ret = hclge_get_vlan_rx_offload_cfg(hdev, i, &vlan_cfg);
+ if (ret)
+ return ret;
- list_for_each_entry_safe(mac_node, tmp, list, node) {
- dev_info(&hdev->pdev->dev, "%pM %d\n",
- mac_node->mac_addr, mac_node->state);
+ sprintf(str_pvid, "%u", vlan_cfg.pvid);
+ j = 0;
+ result[j++] = hclge_dbg_get_func_id_str(str_id, i);
+ result[j++] = str_pvid;
+ result[j++] = state_str[vlan_cfg.accept_tag1];
+ result[j++] = state_str[vlan_cfg.accept_tag2];
+ result[j++] = state_str[vlan_cfg.accept_untag1];
+ result[j++] = state_str[vlan_cfg.accept_untag2];
+ result[j++] = state_str[vlan_cfg.insert_tag1];
+ result[j++] = state_str[vlan_cfg.insert_tag2];
+ result[j++] = state_str[vlan_cfg.shift_tag];
+ result[j++] = state_str[vlan_cfg.strip_tag1];
+ result[j++] = state_str[vlan_cfg.strip_tag2];
+ result[j++] = state_str[vlan_cfg.drop_tag1];
+ result[j++] = state_str[vlan_cfg.drop_tag2];
+ result[j++] = state_str[vlan_cfg.pri_only1];
+ result[j++] = state_str[vlan_cfg.pri_only2];
+
+ hclge_dbg_fill_content(content, sizeof(content),
+ vlan_offload_items, result,
+ ARRAY_SIZE(vlan_offload_items));
+ *pos += scnprintf(buf + *pos, len - *pos, "%s", content);
}
- spin_unlock_bh(&vport->mac_list_lock);
+ return 0;
+}
+
+static int hclge_dbg_dump_vlan_config(struct hclge_dev *hdev, char *buf,
+ int len)
+{
+ int pos = 0;
+ int ret;
+
+ ret = hclge_dbg_dump_vlan_filter_config(hdev, buf, len, &pos);
+ if (ret)
+ return ret;
+
+ return hclge_dbg_dump_vlan_offload_config(hdev, buf, len, &pos);
+}
+
+static int hclge_dbg_dump_ptp_info(struct hclge_dev *hdev, char *buf, int len)
+{
+ struct hclge_ptp *ptp = hdev->ptp;
+ u32 sw_cfg = ptp->ptp_cfg;
+ unsigned int tx_start;
+ unsigned int last_rx;
+ int pos = 0;
+ u32 hw_cfg;
+ int ret;
+
+ pos += scnprintf(buf + pos, len - pos, "phc %s's debug info:\n",
+ ptp->info.name);
+ pos += scnprintf(buf + pos, len - pos, "ptp enable: %s\n",
+ test_bit(HCLGE_PTP_FLAG_EN, &ptp->flags) ?
+ "yes" : "no");
+ pos += scnprintf(buf + pos, len - pos, "ptp tx enable: %s\n",
+ test_bit(HCLGE_PTP_FLAG_TX_EN, &ptp->flags) ?
+ "yes" : "no");
+ pos += scnprintf(buf + pos, len - pos, "ptp rx enable: %s\n",
+ test_bit(HCLGE_PTP_FLAG_RX_EN, &ptp->flags) ?
+ "yes" : "no");
+
+ last_rx = jiffies_to_msecs(ptp->last_rx);
+ pos += scnprintf(buf + pos, len - pos, "last rx time: %lu.%lu\n",
+ last_rx / MSEC_PER_SEC, last_rx % MSEC_PER_SEC);
+ pos += scnprintf(buf + pos, len - pos, "rx count: %lu\n", ptp->rx_cnt);
+
+ tx_start = jiffies_to_msecs(ptp->tx_start);
+ pos += scnprintf(buf + pos, len - pos, "last tx start time: %lu.%lu\n",
+ tx_start / MSEC_PER_SEC, tx_start % MSEC_PER_SEC);
+ pos += scnprintf(buf + pos, len - pos, "tx count: %lu\n", ptp->tx_cnt);
+ pos += scnprintf(buf + pos, len - pos, "tx skipped count: %lu\n",
+ ptp->tx_skipped);
+ pos += scnprintf(buf + pos, len - pos, "tx timeout count: %lu\n",
+ ptp->tx_timeout);
+ pos += scnprintf(buf + pos, len - pos, "last tx seqid: %u\n",
+ ptp->last_tx_seqid);
+
+ ret = hclge_ptp_cfg_qry(hdev, &hw_cfg);
+ if (ret)
+ return ret;
+
+ pos += scnprintf(buf + pos, len - pos, "sw_cfg: %#x, hw_cfg: %#x\n",
+ sw_cfg, hw_cfg);
+
+ pos += scnprintf(buf + pos, len - pos, "tx type: %d, rx filter: %d\n",
+ ptp->ts_cfg.tx_type, ptp->ts_cfg.rx_filter);
return 0;
}
-int hclge_dbg_run_cmd(struct hnae3_handle *handle, const char *cmd_buf)
+static int hclge_dbg_dump_mac_uc(struct hclge_dev *hdev, char *buf, int len)
{
-#define DUMP_REG "dump reg"
-#define DUMP_TM_MAP "dump tm map"
-#define DUMP_LOOPBACK "dump loopback"
-#define DUMP_INTERRUPT "dump intr"
+ hclge_dbg_dump_mac_list(hdev, buf, len, true);
- struct hclge_vport *vport = hclge_get_vport(handle);
- struct hclge_dev *hdev = vport->back;
+ return 0;
+}
- if (strncmp(cmd_buf, "dump fd tcam", 12) == 0) {
- hclge_dbg_fd_tcam(hdev);
- } else if (strncmp(cmd_buf, "dump tc", 7) == 0) {
- hclge_dbg_dump_tc(hdev);
- } else if (strncmp(cmd_buf, DUMP_TM_MAP, strlen(DUMP_TM_MAP)) == 0) {
- hclge_dbg_dump_tm_map(hdev, &cmd_buf[sizeof(DUMP_TM_MAP)]);
- } else if (strncmp(cmd_buf, "dump tm", 7) == 0) {
- hclge_dbg_dump_tm(hdev);
- } else if (strncmp(cmd_buf, "dump qos pause cfg", 18) == 0) {
- hclge_dbg_dump_qos_pause_cfg(hdev);
- } else if (strncmp(cmd_buf, "dump qos pri map", 16) == 0) {
- hclge_dbg_dump_qos_pri_map(hdev);
- } else if (strncmp(cmd_buf, "dump qos buf cfg", 16) == 0) {
- hclge_dbg_dump_qos_buf_cfg(hdev);
- } else if (strncmp(cmd_buf, "dump mng tbl", 12) == 0) {
- hclge_dbg_dump_mng_table(hdev);
- } else if (strncmp(cmd_buf, DUMP_REG, strlen(DUMP_REG)) == 0) {
- hclge_dbg_dump_reg_cmd(hdev, &cmd_buf[sizeof(DUMP_REG)]);
- } else if (strncmp(cmd_buf, "dump reset info", 15) == 0) {
- hclge_dbg_dump_rst_info(hdev);
- } else if (strncmp(cmd_buf, "dump serv info", 14) == 0) {
- hclge_dbg_dump_serv_info(hdev);
- } else if (strncmp(cmd_buf, "dump m7 info", 12) == 0) {
- hclge_dbg_get_m7_stats_info(hdev);
- } else if (strncmp(cmd_buf, "dump ncl_config", 15) == 0) {
- hclge_dbg_dump_ncl_config(hdev,
- &cmd_buf[sizeof("dump ncl_config")]);
- } else if (strncmp(cmd_buf, "dump mac tnl status", 19) == 0) {
- hclge_dbg_dump_mac_tnl_status(hdev);
- } else if (strncmp(cmd_buf, DUMP_LOOPBACK,
- strlen(DUMP_LOOPBACK)) == 0) {
- hclge_dbg_dump_loopback(hdev);
- } else if (strncmp(cmd_buf, "dump qs shaper", 14) == 0) {
- hclge_dbg_dump_qs_shaper(hdev,
- &cmd_buf[sizeof("dump qs shaper")]);
- } else if (strncmp(cmd_buf, "dump uc mac list", 16) == 0) {
- hclge_dbg_dump_mac_list(hdev,
- &cmd_buf[sizeof("dump uc mac list")],
- true);
- } else if (strncmp(cmd_buf, "dump mc mac list", 16) == 0) {
- hclge_dbg_dump_mac_list(hdev,
- &cmd_buf[sizeof("dump mc mac list")],
- false);
- } else if (strncmp(cmd_buf, DUMP_INTERRUPT,
- strlen(DUMP_INTERRUPT)) == 0) {
- hclge_dbg_dump_interrupt(hdev);
- } else {
- dev_info(&hdev->pdev->dev, "unknown command\n");
- return -EINVAL;
- }
+static int hclge_dbg_dump_mac_mc(struct hclge_dev *hdev, char *buf, int len)
+{
+ hclge_dbg_dump_mac_list(hdev, buf, len, false);
return 0;
}
-int hclge_dbg_read_cmd(struct hnae3_handle *handle, const char *cmd_buf,
+static const struct hclge_dbg_func hclge_dbg_cmd_func[] = {
+ {
+ .cmd = HNAE3_DBG_CMD_TM_NODES,
+ .dbg_dump = hclge_dbg_dump_tm_nodes,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_TM_PRI,
+ .dbg_dump = hclge_dbg_dump_tm_pri,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_TM_QSET,
+ .dbg_dump = hclge_dbg_dump_tm_qset,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_TM_MAP,
+ .dbg_dump = hclge_dbg_dump_tm_map,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_TM_PG,
+ .dbg_dump = hclge_dbg_dump_tm_pg,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_TM_PORT,
+ .dbg_dump = hclge_dbg_dump_tm_port,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_TC_SCH_INFO,
+ .dbg_dump = hclge_dbg_dump_tc,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_QOS_PAUSE_CFG,
+ .dbg_dump = hclge_dbg_dump_qos_pause_cfg,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_QOS_PRI_MAP,
+ .dbg_dump = hclge_dbg_dump_qos_pri_map,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_QOS_BUF_CFG,
+ .dbg_dump = hclge_dbg_dump_qos_buf_cfg,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_MAC_UC,
+ .dbg_dump = hclge_dbg_dump_mac_uc,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_MAC_MC,
+ .dbg_dump = hclge_dbg_dump_mac_mc,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_MNG_TBL,
+ .dbg_dump = hclge_dbg_dump_mng_table,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_LOOPBACK,
+ .dbg_dump = hclge_dbg_dump_loopback,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_PTP_INFO,
+ .dbg_dump = hclge_dbg_dump_ptp_info,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_INTERRUPT_INFO,
+ .dbg_dump = hclge_dbg_dump_interrupt,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_RESET_INFO,
+ .dbg_dump = hclge_dbg_dump_rst_info,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_IMP_INFO,
+ .dbg_dump = hclge_dbg_get_imp_stats_info,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_NCL_CONFIG,
+ .dbg_dump = hclge_dbg_dump_ncl_config,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_REG_BIOS_COMMON,
+ .dbg_dump_reg = hclge_dbg_dump_reg_cmd,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_REG_SSU,
+ .dbg_dump_reg = hclge_dbg_dump_reg_cmd,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_REG_IGU_EGU,
+ .dbg_dump_reg = hclge_dbg_dump_reg_cmd,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_REG_RPU,
+ .dbg_dump_reg = hclge_dbg_dump_reg_cmd,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_REG_NCSI,
+ .dbg_dump_reg = hclge_dbg_dump_reg_cmd,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_REG_RTC,
+ .dbg_dump_reg = hclge_dbg_dump_reg_cmd,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_REG_PPP,
+ .dbg_dump_reg = hclge_dbg_dump_reg_cmd,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_REG_RCB,
+ .dbg_dump_reg = hclge_dbg_dump_reg_cmd,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_REG_TQP,
+ .dbg_dump_reg = hclge_dbg_dump_reg_cmd,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_REG_MAC,
+ .dbg_dump = hclge_dbg_dump_mac,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_REG_DCB,
+ .dbg_dump = hclge_dbg_dump_dcb,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_FD_TCAM,
+ .dbg_dump = hclge_dbg_dump_fd_tcam,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_MAC_TNL_STATUS,
+ .dbg_dump = hclge_dbg_dump_mac_tnl_status,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_SERV_INFO,
+ .dbg_dump = hclge_dbg_dump_serv_info,
+ },
+ {
+ .cmd = HNAE3_DBG_CMD_VLAN_CONFIG,
+ .dbg_dump = hclge_dbg_dump_vlan_config,
+ },
+};
+
+int hclge_dbg_read_cmd(struct hnae3_handle *handle, enum hnae3_dbg_cmd cmd,
char *buf, int len)
{
struct hclge_vport *vport = hclge_get_vport(handle);
+ const struct hclge_dbg_func *cmd_func;
struct hclge_dev *hdev = vport->back;
+ u32 i;
- if (strncmp(cmd_buf, HNAE3_DBG_TM_NODES,
- strlen(HNAE3_DBG_TM_NODES)) == 0)
- return hclge_dbg_dump_tm_nodes(hdev, buf, len);
- else if (strncmp(cmd_buf, HNAE3_DBG_TM_PRI,
- strlen(HNAE3_DBG_TM_PRI)) == 0)
- return hclge_dbg_dump_tm_pri(hdev, buf, len);
- else if (strncmp(cmd_buf, HNAE3_DBG_TM_QSET,
- strlen(HNAE3_DBG_TM_QSET)) == 0)
- return hclge_dbg_dump_tm_qset(hdev, buf, len);
+ for (i = 0; i < ARRAY_SIZE(hclge_dbg_cmd_func); i++) {
+ if (cmd == hclge_dbg_cmd_func[i].cmd) {
+ cmd_func = &hclge_dbg_cmd_func[i];
+ if (cmd_func->dbg_dump)
+ return cmd_func->dbg_dump(hdev, buf, len);
+ else
+ return cmd_func->dbg_dump_reg(hdev, cmd, buf,
+ len);
+ }
+ }
+ dev_err(&hdev->pdev->dev, "invalid command(%d)\n", cmd);
return -EINVAL;
}
#include <linux/etherdevice.h>
#include "hclge_cmd.h"
-#define HCLGE_DBG_BUF_LEN 256
#define HCLGE_DBG_MNG_TBL_MAX 64
#define HCLGE_DBG_MNG_VLAN_MASK_B BIT(0)
enum hclge_opcode_type cmd;
};
+struct hclge_dbg_tcam_msg {
+ u8 stage;
+ u32 loc;
+};
+
#define HCLGE_DBG_MAX_DFX_MSG_LEN 60
struct hclge_dbg_dfx_message {
int flag;
#define HCLGE_DBG_MAC_REG_TYPE_LEN 32
struct hclge_dbg_reg_type_info {
- const char *reg_type;
+ enum hnae3_dbg_cmd cmd;
const struct hclge_dbg_dfx_message *dfx_msg;
struct hclge_dbg_reg_common_msg reg_msg;
};
+struct hclge_dbg_func {
+ enum hnae3_dbg_cmd cmd;
+ int (*dbg_dump)(struct hclge_dev *hdev, char *buf, int len);
+ int (*dbg_dump_reg)(struct hclge_dev *hdev, enum hnae3_dbg_cmd cmd,
+ char *buf, int len);
+};
+
static const struct hclge_dbg_dfx_message hclge_dbg_bios_common_reg[] = {
{false, "Reserved"},
{true, "BP_CPU_STATE"},
{true, "RCB_CFG_TX_RING_EBDNUM"},
};
+#define HCLGE_DBG_INFO_LEN 256
+#define HCLGE_DBG_VLAN_FLTR_INFO_LEN 256
+#define HCLGE_DBG_VLAN_OFFLOAD_INFO_LEN 512
+#define HCLGE_DBG_ID_LEN 16
+#define HCLGE_DBG_ITEM_NAME_LEN 32
+#define HCLGE_DBG_DATA_STR_LEN 32
+#define HCLGE_DBG_TM_INFO_LEN 256
+
+#define HCLGE_BILLION_NANO_SECONDS 1000000000
+
+struct hclge_dbg_item {
+ char name[HCLGE_DBG_ITEM_NAME_LEN];
+ u16 interval; /* blank numbers after the item */
+};
+
+struct hclge_dbg_vlan_cfg {
+ u16 pvid;
+ u8 accept_tag1;
+ u8 accept_tag2;
+ u8 accept_untag1;
+ u8 accept_untag2;
+ u8 insert_tag1;
+ u8 insert_tag2;
+ u8 shift_tag;
+ u8 strip_tag1;
+ u8 strip_tag2;
+ u8 drop_tag1;
+ u8 drop_tag2;
+ u8 pri_only1;
+ u8 pri_only2;
+};
+
#endif
{ /* sentinel */ }
};
+static const struct hclge_hw_module_id hclge_hw_module_id_st[] = {
+ {
+ .module_id = MODULE_NONE,
+ .msg = "MODULE_NONE"
+ }, {
+ .module_id = MODULE_BIOS_COMMON,
+ .msg = "MODULE_BIOS_COMMON"
+ }, {
+ .module_id = MODULE_GE,
+ .msg = "MODULE_GE"
+ }, {
+ .module_id = MODULE_IGU_EGU,
+ .msg = "MODULE_IGU_EGU"
+ }, {
+ .module_id = MODULE_LGE,
+ .msg = "MODULE_LGE"
+ }, {
+ .module_id = MODULE_NCSI,
+ .msg = "MODULE_NCSI"
+ }, {
+ .module_id = MODULE_PPP,
+ .msg = "MODULE_PPP"
+ }, {
+ .module_id = MODULE_QCN,
+ .msg = "MODULE_QCN"
+ }, {
+ .module_id = MODULE_RCB_RX,
+ .msg = "MODULE_RCB_RX"
+ }, {
+ .module_id = MODULE_RTC,
+ .msg = "MODULE_RTC"
+ }, {
+ .module_id = MODULE_SSU,
+ .msg = "MODULE_SSU"
+ }, {
+ .module_id = MODULE_TM,
+ .msg = "MODULE_TM"
+ }, {
+ .module_id = MODULE_RCB_TX,
+ .msg = "MODULE_RCB_TX"
+ }, {
+ .module_id = MODULE_TXDMA,
+ .msg = "MODULE_TXDMA"
+ }, {
+ .module_id = MODULE_MASTER,
+ .msg = "MODULE_MASTER"
+ }, {
+ .module_id = MODULE_ROCEE_TOP,
+ .msg = "MODULE_ROCEE_TOP"
+ }, {
+ .module_id = MODULE_ROCEE_TIMER,
+ .msg = "MODULE_ROCEE_TIMER"
+ }, {
+ .module_id = MODULE_ROCEE_MDB,
+ .msg = "MODULE_ROCEE_MDB"
+ }, {
+ .module_id = MODULE_ROCEE_TSP,
+ .msg = "MODULE_ROCEE_TSP"
+ }, {
+ .module_id = MODULE_ROCEE_TRP,
+ .msg = "MODULE_ROCEE_TRP"
+ }, {
+ .module_id = MODULE_ROCEE_SCC,
+ .msg = "MODULE_ROCEE_SCC"
+ }, {
+ .module_id = MODULE_ROCEE_CAEP,
+ .msg = "MODULE_ROCEE_CAEP"
+ }, {
+ .module_id = MODULE_ROCEE_GEN_AC,
+ .msg = "MODULE_ROCEE_GEN_AC"
+ }, {
+ .module_id = MODULE_ROCEE_QMM,
+ .msg = "MODULE_ROCEE_QMM"
+ }, {
+ .module_id = MODULE_ROCEE_LSAN,
+ .msg = "MODULE_ROCEE_LSAN"
+ }
+};
+
+static const struct hclge_hw_type_id hclge_hw_type_id_st[] = {
+ {
+ .type_id = NONE_ERROR,
+ .msg = "none_error"
+ }, {
+ .type_id = FIFO_ERROR,
+ .msg = "fifo_error"
+ }, {
+ .type_id = MEMORY_ERROR,
+ .msg = "memory_error"
+ }, {
+ .type_id = POISON_ERROR,
+ .msg = "poison_error"
+ }, {
+ .type_id = MSIX_ECC_ERROR,
+ .msg = "msix_ecc_error"
+ }, {
+ .type_id = TQP_INT_ECC_ERROR,
+ .msg = "tqp_int_ecc_error"
+ }, {
+ .type_id = PF_ABNORMAL_INT_ERROR,
+ .msg = "pf_abnormal_int_error"
+ }, {
+ .type_id = MPF_ABNORMAL_INT_ERROR,
+ .msg = "mpf_abnormal_int_error"
+ }, {
+ .type_id = COMMON_ERROR,
+ .msg = "common_error"
+ }, {
+ .type_id = PORT_ERROR,
+ .msg = "port_error"
+ }, {
+ .type_id = ETS_ERROR,
+ .msg = "ets_error"
+ }, {
+ .type_id = NCSI_ERROR,
+ .msg = "ncsi_error"
+ }, {
+ .type_id = GLB_ERROR,
+ .msg = "glb_error"
+ }, {
+ .type_id = ROCEE_NORMAL_ERR,
+ .msg = "rocee_normal_error"
+ }, {
+ .type_id = ROCEE_OVF_ERR,
+ .msg = "rocee_ovf_error"
+ }
+};
+
static void hclge_log_error(struct device *dev, char *reg,
const struct hclge_hw_error *err,
u32 err_sts, unsigned long *reset_requests)
{ /* sentinel */ }
};
+static void hclge_config_all_msix_error(struct hclge_dev *hdev, bool enable)
+{
+ u32 reg_val;
+
+ reg_val = hclge_read_dev(&hdev->hw, HCLGE_PF_OTHER_INT_REG);
+
+ if (enable)
+ reg_val |= BIT(HCLGE_VECTOR0_ALL_MSIX_ERR_B);
+ else
+ reg_val &= ~BIT(HCLGE_VECTOR0_ALL_MSIX_ERR_B);
+
+ hclge_write_dev(&hdev->hw, HCLGE_PF_OTHER_INT_REG, reg_val);
+}
+
int hclge_config_nic_hw_error(struct hclge_dev *hdev, bool state)
{
const struct hclge_hw_blk *module = hw_blk;
int ret = 0;
+ hclge_config_all_msix_error(hdev, state);
+
while (module->name) {
if (module->config_err_int) {
ret = module->config_err_int(hdev, state);
static int hclge_handle_all_hw_msix_error(struct hclge_dev *hdev,
unsigned long *reset_requests)
{
- struct hclge_mac_tnl_stats mac_tnl_stats;
- struct device *dev = &hdev->pdev->dev;
u32 mpf_bd_num, pf_bd_num, bd_num;
struct hclge_desc *desc;
- u32 status;
int ret;
/* query the number of bds for the MSIx int status */
if (ret)
goto msi_error;
+ ret = hclge_handle_mac_tnl(hdev);
+
+msi_error:
+ kfree(desc);
+out:
+ return ret;
+}
+
+int hclge_handle_hw_msix_error(struct hclge_dev *hdev,
+ unsigned long *reset_requests)
+{
+ struct device *dev = &hdev->pdev->dev;
+
+ if (!test_bit(HCLGE_STATE_SERVICE_INITED, &hdev->state)) {
+ dev_err(dev,
+ "failed to handle msix error during dev init\n");
+ return -EAGAIN;
+ }
+
+ return hclge_handle_all_hw_msix_error(hdev, reset_requests);
+}
+
+int hclge_handle_mac_tnl(struct hclge_dev *hdev)
+{
+ struct hclge_mac_tnl_stats mac_tnl_stats;
+ struct device *dev = &hdev->pdev->dev;
+ struct hclge_desc desc;
+ u32 status;
+ int ret;
+
/* query and clear mac tnl interruptions */
- hclge_cmd_setup_basic_desc(&desc[0], HCLGE_OPC_QUERY_MAC_TNL_INT,
- true);
- ret = hclge_cmd_send(&hdev->hw, &desc[0], 1);
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_QUERY_MAC_TNL_INT, true);
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
if (ret) {
- dev_err(dev, "query mac tnl int cmd failed (%d)\n", ret);
- goto msi_error;
+ dev_err(dev, "failed to query mac tnl int, ret = %d.\n", ret);
+ return ret;
}
- status = le32_to_cpu(desc->data[0]);
+ status = le32_to_cpu(desc.data[0]);
if (status) {
/* When mac tnl interrupt occurs, we record current time and
* register status here in a fifo, then clear the status. So
kfifo_put(&hdev->mac_tnl_log, mac_tnl_stats);
ret = hclge_clear_mac_tnl_int(hdev);
if (ret)
- dev_err(dev, "clear mac tnl int failed (%d)\n", ret);
+ dev_err(dev, "failed to clear mac tnl int, ret = %d.\n",
+ ret);
}
-msi_error:
- kfree(desc);
-out:
return ret;
}
-int hclge_handle_hw_msix_error(struct hclge_dev *hdev,
- unsigned long *reset_requests)
-{
- struct device *dev = &hdev->pdev->dev;
-
- if (!test_bit(HCLGE_STATE_SERVICE_INITED, &hdev->state)) {
- dev_err(dev,
- "Can't handle - MSIx error reported during dev init\n");
- return 0;
- }
-
- return hclge_handle_all_hw_msix_error(hdev, reset_requests);
-}
-
void hclge_handle_all_hns_hw_errors(struct hnae3_ae_dev *ae_dev)
{
-#define HCLGE_DESC_NO_DATA_LEN 8
-
struct hclge_dev *hdev = ae_dev->priv;
struct device *dev = &hdev->pdev->dev;
u32 mpf_bd_num, pf_bd_num, bd_num;
msi_error:
kfree(desc);
}
+
+bool hclge_find_error_source(struct hclge_dev *hdev)
+{
+ u32 msix_src_flag, hw_err_src_flag;
+
+ msix_src_flag = hclge_read_dev(&hdev->hw, HCLGE_MISC_VECTOR_INT_STS) &
+ HCLGE_VECTOR0_REG_MSIX_MASK;
+
+ hw_err_src_flag = hclge_read_dev(&hdev->hw,
+ HCLGE_RAS_PF_OTHER_INT_STS_REG) &
+ HCLGE_RAS_REG_ERR_MASK;
+
+ return msix_src_flag || hw_err_src_flag;
+}
+
+void hclge_handle_occurred_error(struct hclge_dev *hdev)
+{
+ struct hnae3_ae_dev *ae_dev = pci_get_drvdata(hdev->pdev);
+
+ if (hclge_find_error_source(hdev))
+ hclge_handle_error_info_log(ae_dev);
+}
+
+static void
+hclge_handle_error_type_reg_log(struct device *dev,
+ struct hclge_mod_err_info *mod_info,
+ struct hclge_type_reg_err_info *type_reg_info)
+{
+#define HCLGE_ERR_TYPE_MASK 0x7F
+#define HCLGE_ERR_TYPE_IS_RAS_OFFSET 7
+
+ u8 mod_id, total_module, type_id, total_type, i, is_ras;
+ u8 index_module = MODULE_NONE;
+ u8 index_type = NONE_ERROR;
+
+ mod_id = mod_info->mod_id;
+ type_id = type_reg_info->type_id & HCLGE_ERR_TYPE_MASK;
+ is_ras = type_reg_info->type_id >> HCLGE_ERR_TYPE_IS_RAS_OFFSET;
+
+ total_module = ARRAY_SIZE(hclge_hw_module_id_st);
+ total_type = ARRAY_SIZE(hclge_hw_type_id_st);
+
+ for (i = 0; i < total_module; i++) {
+ if (mod_id == hclge_hw_module_id_st[i].module_id) {
+ index_module = i;
+ break;
+ }
+ }
+
+ for (i = 0; i < total_type; i++) {
+ if (type_id == hclge_hw_type_id_st[i].type_id) {
+ index_type = i;
+ break;
+ }
+ }
+
+ if (index_module != MODULE_NONE && index_type != NONE_ERROR)
+ dev_err(dev,
+ "found %s %s, is %s error.\n",
+ hclge_hw_module_id_st[index_module].msg,
+ hclge_hw_type_id_st[index_type].msg,
+ is_ras ? "ras" : "msix");
+ else
+ dev_err(dev,
+ "unknown module[%u] or type[%u].\n", mod_id, type_id);
+
+ dev_err(dev, "reg_value:\n");
+ for (i = 0; i < type_reg_info->reg_num; i++)
+ dev_err(dev, "0x%08x\n", type_reg_info->hclge_reg[i]);
+}
+
+static void hclge_handle_error_module_log(struct hnae3_ae_dev *ae_dev,
+ const u32 *buf, u32 buf_size)
+{
+ struct hclge_type_reg_err_info *type_reg_info;
+ struct hclge_dev *hdev = ae_dev->priv;
+ struct device *dev = &hdev->pdev->dev;
+ struct hclge_mod_err_info *mod_info;
+ struct hclge_sum_err_info *sum_info;
+ u8 mod_num, err_num, i;
+ u32 offset = 0;
+
+ sum_info = (struct hclge_sum_err_info *)&buf[offset++];
+ if (sum_info->reset_type &&
+ sum_info->reset_type != HNAE3_NONE_RESET)
+ set_bit(sum_info->reset_type, &ae_dev->hw_err_reset_req);
+ mod_num = sum_info->mod_num;
+
+ while (mod_num--) {
+ if (offset >= buf_size) {
+ dev_err(dev, "The offset(%u) exceeds buf's size(%u).\n",
+ offset, buf_size);
+ return;
+ }
+ mod_info = (struct hclge_mod_err_info *)&buf[offset++];
+ err_num = mod_info->err_num;
+
+ for (i = 0; i < err_num; i++) {
+ if (offset >= buf_size) {
+ dev_err(dev,
+ "The offset(%u) exceeds buf size(%u).\n",
+ offset, buf_size);
+ return;
+ }
+
+ type_reg_info = (struct hclge_type_reg_err_info *)
+ &buf[offset++];
+ hclge_handle_error_type_reg_log(dev, mod_info,
+ type_reg_info);
+
+ offset += type_reg_info->reg_num;
+ }
+ }
+}
+
+static int hclge_query_all_err_bd_num(struct hclge_dev *hdev, u32 *bd_num)
+{
+ struct device *dev = &hdev->pdev->dev;
+ struct hclge_desc desc_bd;
+ int ret;
+
+ hclge_cmd_setup_basic_desc(&desc_bd, HCLGE_QUERY_ALL_ERR_BD_NUM, true);
+ ret = hclge_cmd_send(&hdev->hw, &desc_bd, 1);
+ if (ret) {
+ dev_err(dev, "failed to query error bd_num, ret = %d.\n", ret);
+ return ret;
+ }
+
+ *bd_num = le32_to_cpu(desc_bd.data[0]);
+ if (!(*bd_num)) {
+ dev_err(dev, "The value of bd_num is 0!\n");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int hclge_query_all_err_info(struct hclge_dev *hdev,
+ struct hclge_desc *desc, u32 bd_num)
+{
+ struct device *dev = &hdev->pdev->dev;
+ int ret;
+
+ hclge_cmd_setup_basic_desc(desc, HCLGE_QUERY_ALL_ERR_INFO, true);
+ ret = hclge_cmd_send(&hdev->hw, desc, bd_num);
+ if (ret)
+ dev_err(dev, "failed to query error info, ret = %d.\n", ret);
+
+ return ret;
+}
+
+int hclge_handle_error_info_log(struct hnae3_ae_dev *ae_dev)
+{
+ u32 bd_num, desc_len, buf_len, buf_size, i;
+ struct hclge_dev *hdev = ae_dev->priv;
+ struct hclge_desc *desc;
+ __le32 *desc_data;
+ u32 *buf;
+ int ret;
+
+ ret = hclge_query_all_err_bd_num(hdev, &bd_num);
+ if (ret)
+ goto out;
+
+ desc_len = bd_num * sizeof(struct hclge_desc);
+ desc = kzalloc(desc_len, GFP_KERNEL);
+ if (!desc) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ ret = hclge_query_all_err_info(hdev, desc, bd_num);
+ if (ret)
+ goto err_desc;
+
+ buf_len = bd_num * sizeof(struct hclge_desc) - HCLGE_DESC_NO_DATA_LEN;
+ buf_size = buf_len / sizeof(u32);
+
+ desc_data = kzalloc(buf_len, GFP_KERNEL);
+ if (!desc_data) {
+ ret = -ENOMEM;
+ goto err_desc;
+ }
+
+ buf = kzalloc(buf_len, GFP_KERNEL);
+ if (!buf) {
+ ret = -ENOMEM;
+ goto err_buf_alloc;
+ }
+
+ memcpy(desc_data, &desc[0].data[0], buf_len);
+ for (i = 0; i < buf_size; i++)
+ buf[i] = le32_to_cpu(desc_data[i]);
+
+ hclge_handle_error_module_log(ae_dev, buf, buf_size);
+ kfree(buf);
+
+err_buf_alloc:
+ kfree(desc_data);
+err_desc:
+ kfree(desc);
+out:
+ return ret;
+}
#define HCLGE_RAS_PF_OTHER_INT_STS_REG 0x20B00
#define HCLGE_RAS_REG_NFE_MASK 0xFF00
#define HCLGE_RAS_REG_ROCEE_ERR_MASK 0x3000000
+#define HCLGE_RAS_REG_ERR_MASK \
+ (HCLGE_RAS_REG_NFE_MASK | HCLGE_RAS_REG_ROCEE_ERR_MASK)
#define HCLGE_VECTOR0_REG_MSIX_MASK 0x1FF00
#define HCLGE_ROCEE_OVF_ERR_INT_MASK 0x10000
#define HCLGE_ROCEE_OVF_ERR_TYPE_MASK 0x3F
+#define HCLGE_DESC_DATA_MAX 8
+#define HCLGE_REG_NUM_MAX 256
+#define HCLGE_DESC_NO_DATA_LEN 8
+
enum hclge_err_int_type {
HCLGE_ERR_INT_MSIX = 0,
HCLGE_ERR_INT_RAS_CE = 1,
HCLGE_ERR_INT_RAS_FE = 3,
};
+enum hclge_mod_name_list {
+ MODULE_NONE = 0,
+ MODULE_BIOS_COMMON = 1,
+ MODULE_GE = 2,
+ MODULE_IGU_EGU = 3,
+ MODULE_LGE = 4,
+ MODULE_NCSI = 5,
+ MODULE_PPP = 6,
+ MODULE_QCN = 7,
+ MODULE_RCB_RX = 8,
+ MODULE_RTC = 9,
+ MODULE_SSU = 10,
+ MODULE_TM = 11,
+ MODULE_RCB_TX = 12,
+ MODULE_TXDMA = 13,
+ MODULE_MASTER = 14,
+ /* add new MODULE NAME for NIC here in order */
+ MODULE_ROCEE_TOP = 40,
+ MODULE_ROCEE_TIMER = 41,
+ MODULE_ROCEE_MDB = 42,
+ MODULE_ROCEE_TSP = 43,
+ MODULE_ROCEE_TRP = 44,
+ MODULE_ROCEE_SCC = 45,
+ MODULE_ROCEE_CAEP = 46,
+ MODULE_ROCEE_GEN_AC = 47,
+ MODULE_ROCEE_QMM = 48,
+ MODULE_ROCEE_LSAN = 49,
+ /* add new MODULE NAME for RoCEE here in order */
+};
+
+enum hclge_err_type_list {
+ NONE_ERROR = 0,
+ FIFO_ERROR = 1,
+ MEMORY_ERROR = 2,
+ POISON_ERROR = 3,
+ MSIX_ECC_ERROR = 4,
+ TQP_INT_ECC_ERROR = 5,
+ PF_ABNORMAL_INT_ERROR = 6,
+ MPF_ABNORMAL_INT_ERROR = 7,
+ COMMON_ERROR = 8,
+ PORT_ERROR = 9,
+ ETS_ERROR = 10,
+ NCSI_ERROR = 11,
+ GLB_ERROR = 12,
+ /* add new ERROR TYPE for NIC here in order */
+ ROCEE_NORMAL_ERR = 40,
+ ROCEE_OVF_ERR = 41,
+ /* add new ERROR TYPE for ROCEE here in order */
+};
+
struct hclge_hw_blk {
u32 msk;
const char *name;
enum hnae3_reset_type reset_level;
};
+struct hclge_hw_module_id {
+ enum hclge_mod_name_list module_id;
+ const char *msg;
+};
+
+struct hclge_hw_type_id {
+ enum hclge_err_type_list type_id;
+ const char *msg;
+};
+
+struct hclge_sum_err_info {
+ u8 reset_type;
+ u8 mod_num;
+ u8 rsv[2];
+};
+
+struct hclge_mod_err_info {
+ u8 mod_id;
+ u8 err_num;
+ u8 rsv[2];
+};
+
+struct hclge_type_reg_err_info {
+ u8 type_id;
+ u8 reg_num;
+ u8 rsv[2];
+ u32 hclge_reg[HCLGE_REG_NUM_MAX];
+};
+
int hclge_config_mac_tnl_int(struct hclge_dev *hdev, bool en);
int hclge_config_nic_hw_error(struct hclge_dev *hdev, bool state);
int hclge_config_rocee_ras_interrupt(struct hclge_dev *hdev, bool en);
void hclge_handle_all_hns_hw_errors(struct hnae3_ae_dev *ae_dev);
+bool hclge_find_error_source(struct hclge_dev *hdev);
+void hclge_handle_occurred_error(struct hclge_dev *hdev);
pci_ers_result_t hclge_handle_hw_ras_error(struct hnae3_ae_dev *ae_dev);
int hclge_handle_hw_msix_error(struct hclge_dev *hdev,
unsigned long *reset_requests);
+int hclge_handle_error_info_log(struct hnae3_ae_dev *ae_dev);
+int hclge_handle_mac_tnl(struct hclge_dev *hdev);
#endif
static void hclge_parse_cfg(struct hclge_cfg *cfg, struct hclge_desc *desc)
{
+#define HCLGE_TX_SPARE_SIZE_UNIT 4096
#define SPEED_ABILITY_EXT_SHIFT 8
struct hclge_cfg_param_cmd *req;
HCLGE_CFG_SPEED_ABILITY_EXT_S);
cfg->speed_ability |= speed_ability_ext << SPEED_ABILITY_EXT_SHIFT;
+ cfg->vlan_fliter_cap = hnae3_get_field(__le32_to_cpu(req->param[1]),
+ HCLGE_CFG_VLAN_FLTR_CAP_M,
+ HCLGE_CFG_VLAN_FLTR_CAP_S);
+
cfg->umv_space = hnae3_get_field(__le32_to_cpu(req->param[1]),
HCLGE_CFG_UMV_TBL_SPACE_M,
HCLGE_CFG_UMV_TBL_SPACE_S);
cfg->pf_rss_size_max = cfg->pf_rss_size_max ?
1U << cfg->pf_rss_size_max :
cfg->vf_rss_size_max;
+
+ /* The unit of the tx spare buffer size queried from configuration
+ * file is HCLGE_TX_SPARE_SIZE_UNIT(4096) bytes, so a conversion is
+ * needed here.
+ */
+ cfg->tx_spare_buf_size = hnae3_get_field(__le32_to_cpu(req->param[2]),
+ HCLGE_CFG_TX_SPARE_BUF_SIZE_M,
+ HCLGE_CFG_TX_SPARE_BUF_SIZE_S);
+ cfg->tx_spare_buf_size *= HCLGE_TX_SPARE_SIZE_UNIT;
}
/* hclge_get_cfg: query the static parameter from flash
static int hclge_configure(struct hclge_dev *hdev)
{
+ struct hnae3_ae_dev *ae_dev = pci_get_drvdata(hdev->pdev);
struct hclge_cfg cfg;
unsigned int i;
int ret;
hdev->tc_max = cfg.tc_num;
hdev->tm_info.hw_pfc_map = 0;
hdev->wanted_umv_size = cfg.umv_space;
+ hdev->tx_spare_buf_size = cfg.tx_spare_buf_size;
+ if (cfg.vlan_fliter_cap == HCLGE_VLAN_FLTR_CAN_MDF)
+ set_bit(HNAE3_DEV_SUPPORT_VLAN_FLTR_MDF_B, ae_dev->caps);
if (hnae3_dev_fd_supported(hdev)) {
hdev->fd_en = true;
kinfo->num_rx_desc = num_rx_desc;
kinfo->rx_buf_len = hdev->rx_buf_len;
+ kinfo->tx_spare_buf_size = hdev->tx_spare_buf_size;
kinfo->tqp = devm_kcalloc(&hdev->pdev->dev, num_tqps,
sizeof(struct hnae3_queue *), GFP_KERNEL);
vport->mps = HCLGE_MAC_DEFAULT_FRAME;
vport->port_base_vlan_cfg.state = HNAE3_PORT_BASE_VLAN_DISABLE;
vport->rxvlan_cfg.rx_vlan_offload_en = true;
+ vport->req_vlan_fltr_en = true;
INIT_LIST_HEAD(&vport->vlan_list);
INIT_LIST_HEAD(&vport->uc_mac_list);
INIT_LIST_HEAD(&vport->mc_mac_list);
hclge_wq, &hdev->service_task, 0);
}
+static void hclge_errhand_task_schedule(struct hclge_dev *hdev)
+{
+ if (!test_bit(HCLGE_STATE_REMOVING, &hdev->state) &&
+ !test_and_set_bit(HCLGE_STATE_ERR_SERVICE_SCHED, &hdev->state))
+ mod_delayed_work_on(cpumask_first(&hdev->affinity_mask),
+ hclge_wq, &hdev->service_task, 0);
+}
+
void hclge_task_schedule(struct hclge_dev *hdev, unsigned long delay_time)
{
if (!test_bit(HCLGE_STATE_REMOVING, &hdev->state) &&
static u32 hclge_check_event_cause(struct hclge_dev *hdev, u32 *clearval)
{
- u32 cmdq_src_reg, msix_src_reg;
+ u32 cmdq_src_reg, msix_src_reg, hw_err_src_reg;
/* fetch the events from their corresponding regs */
cmdq_src_reg = hclge_read_dev(&hdev->hw, HCLGE_VECTOR0_CMDQ_SRC_REG);
msix_src_reg = hclge_read_dev(&hdev->hw, HCLGE_MISC_VECTOR_INT_STS);
+ hw_err_src_reg = hclge_read_dev(&hdev->hw,
+ HCLGE_RAS_PF_OTHER_INT_STS_REG);
/* Assumption: If by any chance reset and mailbox events are reported
* together then we will only process reset event in this go and will
return HCLGE_VECTOR0_EVENT_RST;
}
- /* check for vector0 msix event source */
- if (msix_src_reg & HCLGE_VECTOR0_REG_MSIX_MASK) {
- *clearval = msix_src_reg;
+ /* check for vector0 msix event and hardware error event source */
+ if (msix_src_reg & HCLGE_VECTOR0_REG_MSIX_MASK ||
+ hw_err_src_reg & HCLGE_RAS_REG_ERR_MASK)
return HCLGE_VECTOR0_EVENT_ERR;
+
+ /* check for vector0 ptp event source */
+ if (BIT(HCLGE_VECTOR0_REG_PTP_INT_B) & msix_src_reg) {
+ *clearval = msix_src_reg;
+ return HCLGE_VECTOR0_EVENT_PTP;
}
/* check for vector0 mailbox(=CMDQ RX) event source */
/* print other vector0 event source */
dev_info(&hdev->pdev->dev,
- "CMDQ INT status:0x%x, other INT status:0x%x\n",
- cmdq_src_reg, msix_src_reg);
- *clearval = msix_src_reg;
+ "INT status: CMDQ(%#x) HW errors(%#x) other(%#x)\n",
+ cmdq_src_reg, hw_err_src_reg, msix_src_reg);
return HCLGE_VECTOR0_EVENT_OTHER;
}
u32 regclr)
{
switch (event_type) {
+ case HCLGE_VECTOR0_EVENT_PTP:
case HCLGE_VECTOR0_EVENT_RST:
hclge_write_dev(&hdev->hw, HCLGE_MISC_RESET_STS_REG, regclr);
break;
static irqreturn_t hclge_misc_irq_handle(int irq, void *data)
{
struct hclge_dev *hdev = data;
+ unsigned long flags;
u32 clearval = 0;
u32 event_cause;
/* vector 0 interrupt is shared with reset and mailbox source events.*/
switch (event_cause) {
case HCLGE_VECTOR0_EVENT_ERR:
- /* we do not know what type of reset is required now. This could
- * only be decided after we fetch the type of errors which
- * caused this event. Therefore, we will do below for now:
- * 1. Assert HNAE3_UNKNOWN_RESET type of reset. This means we
- * have defered type of reset to be used.
- * 2. Schedule the reset service task.
- * 3. When service task receives HNAE3_UNKNOWN_RESET type it
- * will fetch the correct type of reset. This would be done
- * by first decoding the types of errors.
- */
- set_bit(HNAE3_UNKNOWN_RESET, &hdev->reset_request);
- fallthrough;
+ hclge_errhand_task_schedule(hdev);
+ break;
case HCLGE_VECTOR0_EVENT_RST:
hclge_reset_task_schedule(hdev);
break;
+ case HCLGE_VECTOR0_EVENT_PTP:
+ spin_lock_irqsave(&hdev->ptp->lock, flags);
+ hclge_ptp_clean_tx_hwts(hdev);
+ spin_unlock_irqrestore(&hdev->ptp->lock, flags);
+ break;
case HCLGE_VECTOR0_EVENT_MBX:
/* If we are here then,
* 1. Either we are not handling any mbx task and we are not
hclge_clear_event_cause(hdev, event_cause, clearval);
- /* Enable interrupt if it is not cause by reset. And when
- * clearval equal to 0, it means interrupt status may be
- * cleared by hardware before driver reads status register.
- * For this case, vector0 interrupt also should be enabled.
- */
- if (!clearval ||
- event_cause == HCLGE_VECTOR0_EVENT_MBX) {
+ /* Enable interrupt if it is not caused by reset event or error event */
+ if (event_cause == HCLGE_VECTOR0_EVENT_PTP ||
+ event_cause == HCLGE_VECTOR0_EVENT_MBX ||
+ event_cause == HCLGE_VECTOR0_EVENT_OTHER)
hclge_enable_vector(&hdev->misc_vector, true);
- }
return IRQ_HANDLED;
}
enum hnae3_reset_type rst_level = HNAE3_NONE_RESET;
struct hclge_dev *hdev = ae_dev->priv;
- /* first, resolve any unknown reset type to the known type(s) */
- if (test_bit(HNAE3_UNKNOWN_RESET, addr)) {
- u32 msix_sts_reg = hclge_read_dev(&hdev->hw,
- HCLGE_MISC_VECTOR_INT_STS);
- /* we will intentionally ignore any errors from this function
- * as we will end up in *some* reset request in any case
- */
- if (hclge_handle_hw_msix_error(hdev, addr))
- dev_info(&hdev->pdev->dev, "received msix interrupt 0x%x\n",
- msix_sts_reg);
-
- clear_bit(HNAE3_UNKNOWN_RESET, addr);
- /* We defered the clearing of the error event which caused
- * interrupt since it was not posssible to do that in
- * interrupt context (and this is the reason we introduced
- * new UNKNOWN reset type). Now, the errors have been
- * handled and cleared in hardware we can safely enable
- * interrupts. This is an exception to the norm.
- */
- hclge_enable_vector(&hdev->misc_vector, true);
- }
-
/* return the highest priority reset level amongst all */
if (test_bit(HNAE3_IMP_RESET, addr)) {
rst_level = HNAE3_IMP_RESET;
return ret;
}
+static void hclge_show_rst_info(struct hclge_dev *hdev)
+{
+ char *buf;
+
+ buf = kzalloc(HCLGE_DBG_RESET_INFO_LEN, GFP_KERNEL);
+ if (!buf)
+ return;
+
+ hclge_dbg_dump_rst_info(hdev, buf, HCLGE_DBG_RESET_INFO_LEN);
+
+ dev_info(&hdev->pdev->dev, "dump reset info:\n%s", buf);
+
+ kfree(buf);
+}
+
static bool hclge_reset_err_handle(struct hclge_dev *hdev)
{
#define MAX_RESET_FAIL_CNT 5
dev_err(&hdev->pdev->dev, "Reset fail!\n");
- hclge_dbg_dump_rst_info(hdev);
+ hclge_show_rst_info(hdev);
set_bit(HCLGE_STATE_RST_FAIL, &hdev->state);
hdev->reset_type = HNAE3_NONE_RESET;
}
+static void hclge_handle_err_reset_request(struct hclge_dev *hdev)
+{
+ struct hnae3_ae_dev *ae_dev = pci_get_drvdata(hdev->pdev);
+ enum hnae3_reset_type reset_type;
+
+ if (ae_dev->hw_err_reset_req) {
+ reset_type = hclge_get_reset_level(ae_dev,
+ &ae_dev->hw_err_reset_req);
+ hclge_set_def_reset_request(ae_dev, reset_type);
+ }
+
+ if (hdev->default_reset_request && ae_dev->ops->reset_event)
+ ae_dev->ops->reset_event(hdev->pdev, NULL);
+
+ /* enable interrupt after error handling complete */
+ hclge_enable_vector(&hdev->misc_vector, true);
+}
+
+static void hclge_handle_err_recovery(struct hclge_dev *hdev)
+{
+ struct hnae3_ae_dev *ae_dev = pci_get_drvdata(hdev->pdev);
+
+ ae_dev->hw_err_reset_req = 0;
+
+ if (hclge_find_error_source(hdev)) {
+ hclge_handle_error_info_log(ae_dev);
+ hclge_handle_mac_tnl(hdev);
+ }
+
+ hclge_handle_err_reset_request(hdev);
+}
+
+static void hclge_misc_err_recovery(struct hclge_dev *hdev)
+{
+ struct hnae3_ae_dev *ae_dev = pci_get_drvdata(hdev->pdev);
+ struct device *dev = &hdev->pdev->dev;
+ u32 msix_sts_reg;
+
+ msix_sts_reg = hclge_read_dev(&hdev->hw, HCLGE_MISC_VECTOR_INT_STS);
+ if (msix_sts_reg & HCLGE_VECTOR0_REG_MSIX_MASK) {
+ if (hclge_handle_hw_msix_error
+ (hdev, &hdev->default_reset_request))
+ dev_info(dev, "received msix interrupt 0x%x\n",
+ msix_sts_reg);
+ }
+
+ hclge_handle_hw_ras_error(ae_dev);
+
+ hclge_handle_err_reset_request(hdev);
+}
+
+static void hclge_errhand_service_task(struct hclge_dev *hdev)
+{
+ if (!test_and_clear_bit(HCLGE_STATE_ERR_SERVICE_SCHED, &hdev->state))
+ return;
+
+ if (hnae3_dev_ras_imp_supported(hdev))
+ hclge_handle_err_recovery(hdev);
+ else
+ hclge_misc_err_recovery(hdev);
+}
+
static void hclge_reset_service_task(struct hclge_dev *hdev)
{
if (!test_and_clear_bit(HCLGE_STATE_RST_SERVICE_SCHED, &hdev->state))
hclge_task_schedule(hdev, delta);
}
+static void hclge_ptp_service_task(struct hclge_dev *hdev)
+{
+ unsigned long flags;
+
+ if (!test_bit(HCLGE_STATE_PTP_EN, &hdev->state) ||
+ !test_bit(HCLGE_STATE_PTP_TX_HANDLING, &hdev->state) ||
+ !time_is_before_jiffies(hdev->ptp->tx_start + HZ))
+ return;
+
+ /* to prevent concurrence with the irq handler */
+ spin_lock_irqsave(&hdev->ptp->lock, flags);
+
+ /* check HCLGE_STATE_PTP_TX_HANDLING here again, since the irq
+ * handler may handle it just before spin_lock_irqsave().
+ */
+ if (test_bit(HCLGE_STATE_PTP_TX_HANDLING, &hdev->state))
+ hclge_ptp_clean_tx_hwts(hdev);
+
+ spin_unlock_irqrestore(&hdev->ptp->lock, flags);
+}
+
static void hclge_service_task(struct work_struct *work)
{
struct hclge_dev *hdev =
container_of(work, struct hclge_dev, service_task.work);
+ hclge_errhand_service_task(hdev);
hclge_reset_service_task(hdev);
+ hclge_ptp_service_task(hdev);
hclge_mailbox_service_task(hdev);
hclge_periodic_service_task(hdev);
- /* Handle reset and mbx again in case periodical task delays the
- * handling by calling hclge_task_schedule() in
+ /* Handle error recovery, reset and mbx again in case periodical task
+ * delays the handling by calling hclge_task_schedule() in
* hclge_periodic_service_task().
*/
+ hclge_errhand_service_task(hdev);
hclge_reset_service_task(hdev);
hclge_mailbox_service_task(hdev);
}
static void hclge_request_update_promisc_mode(struct hnae3_handle *handle)
{
struct hclge_vport *vport = hclge_get_vport(handle);
- struct hclge_dev *hdev = vport->back;
- set_bit(HCLGE_STATE_PROMISC_CHANGED, &hdev->state);
+ set_bit(HCLGE_VPORT_STATE_PROMISC_CHANGE, &vport->state);
}
static void hclge_sync_fd_state(struct hclge_dev *hdev)
struct hclge_dev *hdev = vport->back;
set_bit(HCLGE_VPORT_STATE_ALIVE, &vport->state);
+ set_bit(HCLGE_VPORT_STATE_PROMISC_CHANGE, &vport->state);
vport->last_active_jiffies = jiffies;
if (test_bit(vport->vport_id, hdev->vport_config_block)) {
kfree(mac_node);
} else if (mac_node->state == HCLGE_MAC_ACTIVE) {
mac_node->state = HCLGE_MAC_TO_DEL;
- list_del(&mac_node->node);
- list_add_tail(&mac_node->node, mac_list);
+ list_move_tail(&mac_node->node, mac_list);
} else {
list_del(&mac_node->node);
kfree(mac_node);
list_del(&mac_node->node);
kfree(mac_node);
} else {
- list_del(&mac_node->node);
- list_add_tail(&mac_node->node, mac_list);
+ list_move_tail(&mac_node->node, mac_list);
}
}
}
list_for_each_entry_safe(mac_node, tmp, list, node) {
switch (mac_node->state) {
case HCLGE_MAC_TO_DEL:
- list_del(&mac_node->node);
- list_add_tail(&mac_node->node, &tmp_del_list);
+ list_move_tail(&mac_node->node, &tmp_del_list);
break;
case HCLGE_MAC_TO_ADD:
new_node = kzalloc(sizeof(*new_node), GFP_ATOMIC);
switch (mac_cfg->state) {
case HCLGE_MAC_TO_DEL:
case HCLGE_MAC_ACTIVE:
- list_del(&mac_cfg->node);
- list_add_tail(&mac_cfg->node, tmp_del_list);
+ list_move_tail(&mac_cfg->node, tmp_del_list);
break;
case HCLGE_MAC_TO_ADD:
if (is_del_list) {
switch (mac_node->state) {
case HCLGE_MAC_TO_DEL:
case HCLGE_MAC_ACTIVE:
- list_del(&mac_node->node);
- list_add_tail(&mac_node->node, &tmp_del_list);
+ list_move_tail(&mac_node->node, &tmp_del_list);
break;
case HCLGE_MAC_TO_ADD:
list_del(&mac_node->node);
struct hclge_vport *vport = hclge_get_vport(handle);
struct hclge_dev *hdev = vport->back;
- if (!hdev->hw.mac.phydev)
- return hclge_mii_ioctl(hdev, ifr, cmd);
+ switch (cmd) {
+ case SIOCGHWTSTAMP:
+ return hclge_ptp_get_cfg(hdev, ifr);
+ case SIOCSHWTSTAMP:
+ return hclge_ptp_set_cfg(hdev, ifr);
+ default:
+ if (!hdev->hw.mac.phydev)
+ return hclge_mii_ioctl(hdev, ifr, cmd);
+ }
return phy_mii_ioctl(hdev->hw.mac.phydev, ifr, cmd);
}
+static int hclge_set_port_vlan_filter_bypass(struct hclge_dev *hdev, u8 vf_id,
+ bool bypass_en)
+{
+ struct hclge_port_vlan_filter_bypass_cmd *req;
+ struct hclge_desc desc;
+ int ret;
+
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_PORT_VLAN_BYPASS, false);
+ req = (struct hclge_port_vlan_filter_bypass_cmd *)desc.data;
+ req->vf_id = vf_id;
+ hnae3_set_bit(req->bypass_state, HCLGE_INGRESS_BYPASS_B,
+ bypass_en ? 1 : 0);
+
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ if (ret)
+ dev_err(&hdev->pdev->dev,
+ "failed to set vport%u port vlan filter bypass state, ret = %d.\n",
+ vf_id, ret);
+
+ return ret;
+}
+
static int hclge_set_vlan_filter_ctrl(struct hclge_dev *hdev, u8 vlan_type,
u8 fe_type, bool filter_en, u8 vf_id)
{
return ret;
}
-#define HCLGE_FILTER_TYPE_VF 0
-#define HCLGE_FILTER_TYPE_PORT 1
-#define HCLGE_FILTER_FE_EGRESS_V1_B BIT(0)
-#define HCLGE_FILTER_FE_NIC_INGRESS_B BIT(0)
-#define HCLGE_FILTER_FE_NIC_EGRESS_B BIT(1)
-#define HCLGE_FILTER_FE_ROCE_INGRESS_B BIT(2)
-#define HCLGE_FILTER_FE_ROCE_EGRESS_B BIT(3)
-#define HCLGE_FILTER_FE_EGRESS (HCLGE_FILTER_FE_NIC_EGRESS_B \
- | HCLGE_FILTER_FE_ROCE_EGRESS_B)
-#define HCLGE_FILTER_FE_INGRESS (HCLGE_FILTER_FE_NIC_INGRESS_B \
- | HCLGE_FILTER_FE_ROCE_INGRESS_B)
+static int hclge_set_vport_vlan_filter(struct hclge_vport *vport, bool enable)
+{
+ struct hclge_dev *hdev = vport->back;
+ struct hnae3_ae_dev *ae_dev = hdev->ae_dev;
+ int ret;
+
+ if (hdev->ae_dev->dev_version < HNAE3_DEVICE_VERSION_V2)
+ return hclge_set_vlan_filter_ctrl(hdev, HCLGE_FILTER_TYPE_VF,
+ HCLGE_FILTER_FE_EGRESS_V1_B,
+ enable, vport->vport_id);
+
+ ret = hclge_set_vlan_filter_ctrl(hdev, HCLGE_FILTER_TYPE_VF,
+ HCLGE_FILTER_FE_EGRESS, enable,
+ vport->vport_id);
+ if (ret)
+ return ret;
+
+ if (test_bit(HNAE3_DEV_SUPPORT_PORT_VLAN_BYPASS_B, ae_dev->caps))
+ ret = hclge_set_port_vlan_filter_bypass(hdev, vport->vport_id,
+ !enable);
+ else if (!vport->vport_id)
+ ret = hclge_set_vlan_filter_ctrl(hdev, HCLGE_FILTER_TYPE_PORT,
+ HCLGE_FILTER_FE_INGRESS,
+ enable, 0);
+
+ return ret;
+}
-static void hclge_enable_vlan_filter(struct hnae3_handle *handle, bool enable)
+static bool hclge_need_enable_vport_vlan_filter(struct hclge_vport *vport)
{
- struct hclge_vport *vport = hclge_get_vport(handle);
+ struct hnae3_handle *handle = &vport->nic;
+ struct hclge_vport_vlan_cfg *vlan, *tmp;
struct hclge_dev *hdev = vport->back;
- if (hdev->ae_dev->dev_version >= HNAE3_DEVICE_VERSION_V2) {
- hclge_set_vlan_filter_ctrl(hdev, HCLGE_FILTER_TYPE_VF,
- HCLGE_FILTER_FE_EGRESS, enable, 0);
- hclge_set_vlan_filter_ctrl(hdev, HCLGE_FILTER_TYPE_PORT,
- HCLGE_FILTER_FE_INGRESS, enable, 0);
- } else {
- hclge_set_vlan_filter_ctrl(hdev, HCLGE_FILTER_TYPE_VF,
- HCLGE_FILTER_FE_EGRESS_V1_B, enable,
- 0);
+ if (vport->vport_id) {
+ if (vport->port_base_vlan_cfg.state !=
+ HNAE3_PORT_BASE_VLAN_DISABLE)
+ return true;
+
+ if (vport->vf_info.trusted && vport->vf_info.request_uc_en)
+ return false;
+ } else if (handle->netdev_flags & HNAE3_USER_UPE) {
+ return false;
}
- if (enable)
- handle->netdev_flags |= HNAE3_VLAN_FLTR;
- else
- handle->netdev_flags &= ~HNAE3_VLAN_FLTR;
+
+ if (!vport->req_vlan_fltr_en)
+ return false;
+
+ /* compatible with former device, always enable vlan filter */
+ if (!test_bit(HNAE3_DEV_SUPPORT_VLAN_FLTR_MDF_B, hdev->ae_dev->caps))
+ return true;
+
+ list_for_each_entry_safe(vlan, tmp, &vport->vlan_list, node)
+ if (vlan->vlan_id != 0)
+ return true;
+
+ return false;
+}
+
+int hclge_enable_vport_vlan_filter(struct hclge_vport *vport, bool request_en)
+{
+ struct hclge_dev *hdev = vport->back;
+ bool need_en;
+ int ret;
+
+ mutex_lock(&hdev->vport_lock);
+
+ vport->req_vlan_fltr_en = request_en;
+
+ need_en = hclge_need_enable_vport_vlan_filter(vport);
+ if (need_en == vport->cur_vlan_fltr_en) {
+ mutex_unlock(&hdev->vport_lock);
+ return 0;
+ }
+
+ ret = hclge_set_vport_vlan_filter(vport, need_en);
+ if (ret) {
+ mutex_unlock(&hdev->vport_lock);
+ return ret;
+ }
+
+ vport->cur_vlan_fltr_en = need_en;
+
+ mutex_unlock(&hdev->vport_lock);
+
+ return 0;
+}
+
+static int hclge_enable_vlan_filter(struct hnae3_handle *handle, bool enable)
+{
+ struct hclge_vport *vport = hclge_get_vport(handle);
+
+ return hclge_enable_vport_vlan_filter(vport, enable);
}
static int hclge_set_vf_vlan_filter_cmd(struct hclge_dev *hdev, u16 vfid,
static int hclge_vlan_offload_cfg(struct hclge_vport *vport,
u16 port_base_vlan_state,
- u16 vlan_tag)
+ u16 vlan_tag, u8 qos)
{
int ret;
vport->txvlan_cfg.accept_tag1 =
ae_dev->dev_version >= HNAE3_DEVICE_VERSION_V3;
vport->txvlan_cfg.insert_tag1_en = true;
- vport->txvlan_cfg.default_tag1 = vlan_tag;
+ vport->txvlan_cfg.default_tag1 = (qos << VLAN_PRIO_SHIFT) |
+ vlan_tag;
}
vport->txvlan_cfg.accept_untag1 = true;
vport->vport_id);
if (ret)
return ret;
+ vport->cur_vlan_fltr_en = true;
}
ret = hclge_set_vlan_filter_ctrl(hdev, HCLGE_FILTER_TYPE_PORT,
return ret;
}
- handle->netdev_flags |= HNAE3_VLAN_FLTR;
-
hdev->vlan_type_cfg.rx_in_fst_vlan_type = HCLGE_DEF_VLAN_TYPE;
hdev->vlan_type_cfg.rx_in_sec_vlan_type = HCLGE_DEF_VLAN_TYPE;
hdev->vlan_type_cfg.rx_ot_fst_vlan_type = HCLGE_DEF_VLAN_TYPE;
for (i = 0; i < hdev->num_alloc_vport; i++) {
u16 vlan_tag;
+ u8 qos;
vport = &hdev->vport[i];
vlan_tag = vport->port_base_vlan_cfg.vlan_info.vlan_tag;
+ qos = vport->port_base_vlan_cfg.vlan_info.qos;
ret = hclge_vlan_offload_cfg(vport,
vport->port_base_vlan_cfg.state,
- vlan_tag);
+ vlan_tag, qos);
if (ret)
return ret;
}
hclge_restore_mac_table_common(vport);
hclge_restore_vport_vlan_table(vport);
- set_bit(HCLGE_STATE_PROMISC_CHANGED, &hdev->state);
set_bit(HCLGE_STATE_FD_USER_DEF_CHANGED, &hdev->state);
hclge_restore_fd_entries(handle);
}
return hclge_set_vlan_rx_offload_cfg(vport);
}
+static void hclge_set_vport_vlan_fltr_change(struct hclge_vport *vport)
+{
+ struct hclge_dev *hdev = vport->back;
+
+ if (test_bit(HNAE3_DEV_SUPPORT_VLAN_FLTR_MDF_B, hdev->ae_dev->caps))
+ set_bit(HCLGE_VPORT_STATE_VLAN_FLTR_CHANGE, &vport->state);
+}
+
static int hclge_update_vlan_filter_entries(struct hclge_vport *vport,
u16 port_base_vlan_state,
struct hclge_vlan_info *new_info,
if (port_base_vlan_state == HNAE3_PORT_BASE_VLAN_ENABLE) {
hclge_rm_vport_all_vlan_table(vport, false);
+ /* force clear VLAN 0 */
+ ret = hclge_set_vf_vlan_common(hdev, vport->vport_id, true, 0);
+ if (ret)
+ return ret;
return hclge_set_vlan_filter_hw(hdev,
htons(new_info->vlan_proto),
vport->vport_id,
false);
}
+ /* force add VLAN 0 */
+ ret = hclge_set_vf_vlan_common(hdev, vport->vport_id, false, 0);
+ if (ret)
+ return ret;
+
ret = hclge_set_vlan_filter_hw(hdev, htons(old_info->vlan_proto),
vport->vport_id, old_info->vlan_tag,
true);
return hclge_add_vport_all_vlan_table(vport);
}
+static bool hclge_need_update_vlan_filter(const struct hclge_vlan_info *new_cfg,
+ const struct hclge_vlan_info *old_cfg)
+{
+ if (new_cfg->vlan_tag != old_cfg->vlan_tag)
+ return true;
+
+ if (new_cfg->vlan_tag == 0 && (new_cfg->qos == 0 || old_cfg->qos == 0))
+ return true;
+
+ return false;
+}
+
int hclge_update_port_base_vlan_cfg(struct hclge_vport *vport, u16 state,
struct hclge_vlan_info *vlan_info)
{
old_vlan_info = &vport->port_base_vlan_cfg.vlan_info;
- ret = hclge_vlan_offload_cfg(vport, state, vlan_info->vlan_tag);
+ ret = hclge_vlan_offload_cfg(vport, state, vlan_info->vlan_tag,
+ vlan_info->qos);
if (ret)
return ret;
+ if (!hclge_need_update_vlan_filter(vlan_info, old_vlan_info))
+ goto out;
+
if (state == HNAE3_PORT_BASE_VLAN_MODIFY) {
/* add new VLAN tag */
ret = hclge_set_vlan_filter_hw(hdev,
return ret;
/* remove old VLAN tag */
- ret = hclge_set_vlan_filter_hw(hdev,
- htons(old_vlan_info->vlan_proto),
- vport->vport_id,
- old_vlan_info->vlan_tag,
- true);
- if (ret)
+ if (old_vlan_info->vlan_tag == 0)
+ ret = hclge_set_vf_vlan_common(hdev, vport->vport_id,
+ true, 0);
+ else
+ ret = hclge_set_vlan_filter_hw(hdev,
+ htons(ETH_P_8021Q),
+ vport->vport_id,
+ old_vlan_info->vlan_tag,
+ true);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to clear vport%u port base vlan %u, ret = %d.\n",
+ vport->vport_id, old_vlan_info->vlan_tag, ret);
return ret;
+ }
- goto update;
+ goto out;
}
ret = hclge_update_vlan_filter_entries(vport, state, vlan_info,
if (ret)
return ret;
- /* update state only when disable/enable port based VLAN */
+out:
vport->port_base_vlan_cfg.state = state;
if (state == HNAE3_PORT_BASE_VLAN_DISABLE)
nic->port_base_vlan_state = HNAE3_PORT_BASE_VLAN_DISABLE;
else
nic->port_base_vlan_state = HNAE3_PORT_BASE_VLAN_ENABLE;
-update:
- vport->port_base_vlan_cfg.vlan_info.vlan_tag = vlan_info->vlan_tag;
- vport->port_base_vlan_cfg.vlan_info.qos = vlan_info->qos;
- vport->port_base_vlan_cfg.vlan_info.vlan_proto = vlan_info->vlan_proto;
+ vport->port_base_vlan_cfg.vlan_info = *vlan_info;
+ hclge_set_vport_vlan_fltr_change(vport);
return 0;
}
static u16 hclge_get_port_base_vlan_state(struct hclge_vport *vport,
enum hnae3_port_base_vlan_state state,
- u16 vlan)
+ u16 vlan, u8 qos)
{
if (state == HNAE3_PORT_BASE_VLAN_DISABLE) {
- if (!vlan)
+ if (!vlan && !qos)
return HNAE3_PORT_BASE_VLAN_NOCHANGE;
- else
- return HNAE3_PORT_BASE_VLAN_ENABLE;
- } else {
- if (!vlan)
- return HNAE3_PORT_BASE_VLAN_DISABLE;
- else if (vport->port_base_vlan_cfg.vlan_info.vlan_tag == vlan)
- return HNAE3_PORT_BASE_VLAN_NOCHANGE;
- else
- return HNAE3_PORT_BASE_VLAN_MODIFY;
+
+ return HNAE3_PORT_BASE_VLAN_ENABLE;
}
+
+ if (!vlan && !qos)
+ return HNAE3_PORT_BASE_VLAN_DISABLE;
+
+ if (vport->port_base_vlan_cfg.vlan_info.vlan_tag == vlan &&
+ vport->port_base_vlan_cfg.vlan_info.qos == qos)
+ return HNAE3_PORT_BASE_VLAN_NOCHANGE;
+
+ return HNAE3_PORT_BASE_VLAN_MODIFY;
}
static int hclge_set_vf_vlan_filter(struct hnae3_handle *handle, int vfid,
state = hclge_get_port_base_vlan_state(vport,
vport->port_base_vlan_cfg.state,
- vlan);
+ vlan, qos);
if (state == HNAE3_PORT_BASE_VLAN_NOCHANGE)
return 0;
test_bit(HCLGE_VPORT_STATE_ALIVE, &vport->state))
hclge_push_vf_port_base_vlan_info(&hdev->vport[0],
vport->vport_id, state,
- vlan, qos,
- ntohs(proto));
+ &vlan_info);
return 0;
}
*/
set_bit(vlan_id, vport->vlan_del_fail_bmap);
}
+
+ hclge_set_vport_vlan_fltr_change(vport);
+
return ret;
}
+static void hclge_sync_vlan_fltr_state(struct hclge_dev *hdev)
+{
+ struct hclge_vport *vport;
+ int ret;
+ u16 i;
+
+ for (i = 0; i < hdev->num_alloc_vport; i++) {
+ vport = &hdev->vport[i];
+ if (!test_and_clear_bit(HCLGE_VPORT_STATE_VLAN_FLTR_CHANGE,
+ &vport->state))
+ continue;
+
+ ret = hclge_enable_vport_vlan_filter(vport,
+ vport->req_vlan_fltr_en);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to sync vlan filter state for vport%u, ret = %d\n",
+ vport->vport_id, ret);
+ set_bit(HCLGE_VPORT_STATE_VLAN_FLTR_CHANGE,
+ &vport->state);
+ return;
+ }
+ }
+}
+
static void hclge_sync_vlan_filter(struct hclge_dev *hdev)
{
#define HCLGE_MAX_SYNC_COUNT 60
clear_bit(vlan_id, vport->vlan_del_fail_bmap);
hclge_rm_vport_vlan_table(vport, vlan_id, false);
+ hclge_set_vport_vlan_fltr_change(vport);
sync_cnt++;
if (sync_cnt >= HCLGE_MAX_SYNC_COUNT)
VLAN_N_VID);
}
}
+
+ hclge_sync_vlan_fltr_state(hdev);
}
static int hclge_set_mac_mtu(struct hclge_dev *hdev, int new_mps)
hdev->flag & HCLGE_FLAG_DCB_ENABLE ? "enable" : "disable");
dev_info(dev, "MQPRIO %s\n",
hdev->flag & HCLGE_FLAG_MQPRIO_ENABLE ? "enable" : "disable");
+ dev_info(dev, "Default tx spare buffer size: %u\n",
+ hdev->tx_spare_buf_size);
dev_info(dev, "PF info end.\n");
}
}
}
+static void hclge_init_rxd_adv_layout(struct hclge_dev *hdev)
+{
+ if (hnae3_ae_dev_rxd_adv_layout_supported(hdev->ae_dev))
+ hclge_write_dev(&hdev->hw, HCLGE_RXD_ADV_LAYOUT_EN_REG, 1);
+}
+
+static void hclge_uninit_rxd_adv_layout(struct hclge_dev *hdev)
+{
+ if (hnae3_ae_dev_rxd_adv_layout_supported(hdev->ae_dev))
+ hclge_write_dev(&hdev->hw, HCLGE_RXD_ADV_LAYOUT_EN_REG, 0);
+}
+
static int hclge_init_ae_dev(struct hnae3_ae_dev *ae_dev)
{
struct pci_dev *pdev = ae_dev->pdev;
goto err_mdiobus_unreg;
}
+ ret = hclge_ptp_init(hdev);
+ if (ret)
+ goto err_mdiobus_unreg;
+
INIT_KFIFO(hdev->mac_tnl_log);
hclge_dcb_ops_set(hdev);
hclge_clear_resetting_state(hdev);
/* Log and clear the hw errors those already occurred */
- hclge_handle_all_hns_hw_errors(ae_dev);
+ if (hnae3_dev_ras_imp_supported(hdev))
+ hclge_handle_occurred_error(hdev);
+ else
+ hclge_handle_all_hns_hw_errors(ae_dev);
/* request delayed reset for the error recovery because an immediate
* global reset on a PF affecting pending initialization of other PFs
mod_timer(&hdev->reset_timer, jiffies + HCLGE_RESET_INTERVAL);
}
+ hclge_init_rxd_adv_layout(hdev);
+
/* Enable MISC vector(vector0) */
hclge_enable_vector(&hdev->misc_vector, true);
{
struct hclge_vport *vport = hclge_get_vport(handle);
struct hclge_dev *hdev = vport->back;
- struct hnae3_ae_dev *ae_dev = hdev->ae_dev;
u32 new_trusted = enable ? 1 : 0;
- bool en_bc_pmc;
- int ret;
vport = hclge_get_vf_vport(hdev, vf);
if (!vport)
if (vport->vf_info.trusted == new_trusted)
return 0;
- /* Disable promisc mode for VF if it is not trusted any more. */
- if (!enable && vport->vf_info.promisc_enable) {
- en_bc_pmc = ae_dev->dev_version >= HNAE3_DEVICE_VERSION_V2;
- ret = hclge_set_vport_promisc_mode(vport, false, false,
- en_bc_pmc);
- if (ret)
- return ret;
- vport->vf_info.promisc_enable = 0;
- hclge_inform_vf_promisc_info(vport);
- }
-
vport->vf_info.trusted = new_trusted;
+ set_bit(HCLGE_VPORT_STATE_PROMISC_CHANGE, &vport->state);
+ hclge_task_schedule(hdev, 0);
return 0;
}
return ret;
}
+ ret = hclge_ptp_init(hdev);
+ if (ret)
+ return ret;
+
/* Log and clear the hw errors those already occurred */
- hclge_handle_all_hns_hw_errors(ae_dev);
+ if (hnae3_dev_ras_imp_supported(hdev))
+ hclge_handle_occurred_error(hdev);
+ else
+ hclge_handle_all_hns_hw_errors(ae_dev);
/* Re-enable the hw error interrupts because
* the interrupts get disabled on global reset.
if (ret)
return ret;
+ hclge_init_rxd_adv_layout(hdev);
+
dev_info(&pdev->dev, "Reset done, %s driver initialization finished.\n",
HCLGE_DRIVER_NAME);
hclge_clear_vf_vlan(hdev);
hclge_misc_affinity_teardown(hdev);
hclge_state_uninit(hdev);
+ hclge_ptp_uninit(hdev);
+ hclge_uninit_rxd_adv_layout(hdev);
hclge_uninit_mac_table(hdev);
hclge_del_all_fd_entries(hdev);
struct hnae3_handle *handle = &vport->nic;
u8 tmp_flags;
int ret;
+ u16 i;
if (vport->last_promisc_flags != vport->overflow_promisc_flags) {
- set_bit(HCLGE_STATE_PROMISC_CHANGED, &hdev->state);
+ set_bit(HCLGE_VPORT_STATE_PROMISC_CHANGE, &vport->state);
vport->last_promisc_flags = vport->overflow_promisc_flags;
}
- if (test_bit(HCLGE_STATE_PROMISC_CHANGED, &hdev->state)) {
+ if (test_bit(HCLGE_VPORT_STATE_PROMISC_CHANGE, &vport->state)) {
tmp_flags = handle->netdev_flags | vport->last_promisc_flags;
ret = hclge_set_promisc_mode(handle, tmp_flags & HNAE3_UPE,
tmp_flags & HNAE3_MPE);
if (!ret) {
- clear_bit(HCLGE_STATE_PROMISC_CHANGED, &hdev->state);
- hclge_enable_vlan_filter(handle,
- tmp_flags & HNAE3_VLAN_FLTR);
+ clear_bit(HCLGE_VPORT_STATE_PROMISC_CHANGE,
+ &vport->state);
+ set_bit(HCLGE_VPORT_STATE_VLAN_FLTR_CHANGE,
+ &vport->state);
+ }
+ }
+
+ for (i = 1; i < hdev->num_alloc_vport; i++) {
+ bool uc_en = false;
+ bool mc_en = false;
+ bool bc_en;
+
+ vport = &hdev->vport[i];
+
+ if (!test_and_clear_bit(HCLGE_VPORT_STATE_PROMISC_CHANGE,
+ &vport->state))
+ continue;
+
+ if (vport->vf_info.trusted) {
+ uc_en = vport->vf_info.request_uc_en > 0;
+ mc_en = vport->vf_info.request_mc_en > 0;
+ }
+ bc_en = vport->vf_info.request_bc_en > 0;
+
+ ret = hclge_cmd_set_promisc_mode(hdev, vport->vport_id, uc_en,
+ mc_en, bc_en);
+ if (ret) {
+ set_bit(HCLGE_VPORT_STATE_PROMISC_CHANGE,
+ &vport->state);
+ return;
}
+ hclge_set_vport_vlan_fltr_change(vport);
}
}
.get_fd_all_rules = hclge_get_all_rules,
.enable_fd = hclge_enable_fd,
.add_arfs_entry = hclge_add_fd_entry_by_arfs,
- .dbg_run_cmd = hclge_dbg_run_cmd,
.dbg_read_cmd = hclge_dbg_read_cmd,
.handle_hw_ras_error = hclge_handle_hw_ras_error,
.get_hw_reset_stat = hclge_get_hw_reset_stat,
.cls_flower_active = hclge_is_cls_flower_active,
.get_phy_link_ksettings = hclge_get_phy_link_ksettings,
.set_phy_link_ksettings = hclge_set_phy_link_ksettings,
+ .set_tx_hwts_info = hclge_ptp_set_tx_info,
+ .get_rx_hwts = hclge_ptp_get_rx_hwts,
+ .get_ts_info = hclge_ptp_get_ts_info,
};
static struct hnae3_ae_algo ae_algo = {
#include <linux/kfifo.h>
#include "hclge_cmd.h"
+#include "hclge_ptp.h"
#include "hnae3.h"
#define HCLGE_MOD_VERSION "1.0"
/* bar registers for common func */
#define HCLGE_VECTOR0_OTER_EN_REG 0x20600
#define HCLGE_GRO_EN_REG 0x28000
+#define HCLGE_RXD_ADV_LAYOUT_EN_REG 0x28008
/* bar registers for rcb */
#define HCLGE_RING_RX_ADDR_L_REG 0x80000
#define HCLGE_MAX_QSET_NUM 1024
+#define HCLGE_DBG_RESET_INFO_LEN 1024
+
enum HLCGE_PORT_TYPE {
HOST_PORT,
NETWORK_PORT
#define HCLGE_FUN_RST_ING_B 0
/* Vector0 register bits define */
+#define HCLGE_VECTOR0_REG_PTP_INT_B 0
#define HCLGE_VECTOR0_GLOBALRESET_INT_B 5
#define HCLGE_VECTOR0_CORERESET_INT_B 6
#define HCLGE_VECTOR0_IMPRESET_INT_B 7
#define HCLGE_VECTOR0_IMP_RESET_INT_B 1
#define HCLGE_VECTOR0_IMP_CMDQ_ERR_B 4U
#define HCLGE_VECTOR0_IMP_RD_POISON_B 5U
+#define HCLGE_VECTOR0_ALL_MSIX_ERR_B 6U
#define HCLGE_MAC_DEFAULT_FRAME \
(ETH_HLEN + ETH_FCS_LEN + 2 * VLAN_HLEN + ETH_DATA_LEN)
HCLGE_STATE_RST_HANDLING,
HCLGE_STATE_MBX_SERVICE_SCHED,
HCLGE_STATE_MBX_HANDLING,
+ HCLGE_STATE_ERR_SERVICE_SCHED,
HCLGE_STATE_STATISTICS_UPDATING,
HCLGE_STATE_CMD_DISABLE,
HCLGE_STATE_LINK_UPDATING,
- HCLGE_STATE_PROMISC_CHANGED,
HCLGE_STATE_RST_FAIL,
HCLGE_STATE_FD_TBL_CHANGED,
HCLGE_STATE_FD_CLEAR_ALL,
HCLGE_STATE_FD_USER_DEF_CHANGED,
+ HCLGE_STATE_PTP_EN,
+ HCLGE_STATE_PTP_TX_HANDLING,
HCLGE_STATE_MAX
};
HCLGE_VECTOR0_EVENT_RST,
HCLGE_VECTOR0_EVENT_MBX,
HCLGE_VECTOR0_EVENT_ERR,
+ HCLGE_VECTOR0_EVENT_PTP,
HCLGE_VECTOR0_EVENT_OTHER,
};
HCLGE_FC_DEFAULT
};
+#define HCLGE_FILTER_TYPE_VF 0
+#define HCLGE_FILTER_TYPE_PORT 1
+#define HCLGE_FILTER_FE_EGRESS_V1_B BIT(0)
+#define HCLGE_FILTER_FE_NIC_INGRESS_B BIT(0)
+#define HCLGE_FILTER_FE_NIC_EGRESS_B BIT(1)
+#define HCLGE_FILTER_FE_ROCE_INGRESS_B BIT(2)
+#define HCLGE_FILTER_FE_ROCE_EGRESS_B BIT(3)
+#define HCLGE_FILTER_FE_EGRESS (HCLGE_FILTER_FE_NIC_EGRESS_B \
+ | HCLGE_FILTER_FE_ROCE_EGRESS_B)
+#define HCLGE_FILTER_FE_INGRESS (HCLGE_FILTER_FE_NIC_INGRESS_B \
+ | HCLGE_FILTER_FE_ROCE_INGRESS_B)
+
+enum hclge_vlan_fltr_cap {
+ HCLGE_VLAN_FLTR_DEF,
+ HCLGE_VLAN_FLTR_CAN_MDF,
+};
enum hclge_link_fail_code {
HCLGE_LF_NORMAL,
HCLGE_LF_REF_CLOCK_LOST,
struct hclge_cfg {
u8 tc_num;
+ u8 vlan_fliter_cap;
u16 tqp_desc_num;
u16 rx_buf_len;
u16 vf_rss_size_max;
u8 mac_addr[ETH_ALEN];
u8 default_speed;
u32 numa_node_map;
+ u32 tx_spare_buf_size;
u16 speed_ability;
u16 umv_space;
};
struct hclge_vf_vlan_cfg {
u8 mbx_cmd;
u8 subcode;
- u8 is_kill;
- u16 vlan;
- u16 proto;
+ union {
+ struct {
+ u8 is_kill;
+ u16 vlan;
+ u16 proto;
+ };
+ u8 enable;
+ };
};
#pragma pack()
u16 alloc_rss_size; /* Allocated RSS task queue */
u16 vf_rss_size_max; /* HW defined VF max RSS task queue */
u16 pf_rss_size_max; /* HW defined PF max RSS task queue */
+ u32 tx_spare_buf_size; /* HW defined TX spare buffer size */
u16 fdir_pf_filter_count; /* Num of guaranteed filters for this PF */
u16 num_alloc_vport; /* Num vports this driver supports */
/* affinity mask and notify for misc interrupt */
cpumask_t affinity_mask;
struct irq_affinity_notify affinity_notify;
+ struct hclge_ptp *ptp;
};
/* VPort level vlan tag configuration for TX direction */
enum HCLGE_VPORT_STATE {
HCLGE_VPORT_STATE_ALIVE,
HCLGE_VPORT_STATE_MAC_TBL_CHANGE,
+ HCLGE_VPORT_STATE_PROMISC_CHANGE,
+ HCLGE_VPORT_STATE_VLAN_FLTR_CHANGE,
HCLGE_VPORT_STATE_MAX
};
u32 spoofchk;
u32 max_tx_rate;
u32 trusted;
- u16 promisc_enable;
+ u8 request_uc_en;
+ u8 request_mc_en;
+ u8 request_bc_en;
};
struct hclge_vport {
u32 bw_limit; /* VSI BW Limit (0 = disabled) */
u8 dwrr;
+ bool req_vlan_fltr_en;
+ bool cur_vlan_fltr_en;
unsigned long vlan_del_fail_bmap[BITS_TO_LONGS(VLAN_N_VID)];
struct hclge_port_base_vlan_config port_base_vlan_cfg;
struct hclge_tx_vtag_cfg txvlan_cfg;
int hclge_vport_start(struct hclge_vport *vport);
void hclge_vport_stop(struct hclge_vport *vport);
int hclge_set_vport_mtu(struct hclge_vport *vport, int new_mtu);
-int hclge_dbg_run_cmd(struct hnae3_handle *handle, const char *cmd_buf);
-int hclge_dbg_read_cmd(struct hnae3_handle *handle, const char *cmd_buf,
+int hclge_dbg_read_cmd(struct hnae3_handle *handle, enum hnae3_dbg_cmd cmd,
char *buf, int len);
u16 hclge_covert_handle_qid_global(struct hnae3_handle *handle, u16 queue_id);
int hclge_notify_client(struct hclge_dev *hdev,
int hclge_update_port_base_vlan_cfg(struct hclge_vport *vport, u16 state,
struct hclge_vlan_info *vlan_info);
int hclge_push_vf_port_base_vlan_info(struct hclge_vport *vport, u8 vfid,
- u16 state, u16 vlan_tag, u16 qos,
- u16 vlan_proto);
+ u16 state,
+ struct hclge_vlan_info *vlan_info);
void hclge_task_schedule(struct hclge_dev *hdev, unsigned long delay_time);
int hclge_query_bd_num_cmd_send(struct hclge_dev *hdev,
struct hclge_desc *desc);
void hclge_report_hw_error(struct hclge_dev *hdev,
enum hnae3_hw_error_type type);
void hclge_inform_vf_promisc_info(struct hclge_vport *vport);
-void hclge_dbg_dump_rst_info(struct hclge_dev *hdev);
+int hclge_dbg_dump_rst_info(struct hclge_dev *hdev, char *buf, int len);
int hclge_push_vf_link_status(struct hclge_vport *vport);
+int hclge_enable_vport_vlan_filter(struct hclge_vport *vport, bool request_en);
#endif
return ret;
}
-static int hclge_set_vf_promisc_mode(struct hclge_vport *vport,
- struct hclge_mbx_vf_to_pf_cmd *req)
+static void hclge_set_vf_promisc_mode(struct hclge_vport *vport,
+ struct hclge_mbx_vf_to_pf_cmd *req)
{
- bool en_bc = req->msg.en_bc ? true : false;
- bool en_uc = req->msg.en_uc ? true : false;
- bool en_mc = req->msg.en_mc ? true : false;
struct hnae3_handle *handle = &vport->nic;
- int ret;
+ struct hclge_dev *hdev = vport->back;
- if (!vport->vf_info.trusted) {
- en_uc = false;
- en_mc = false;
- }
+ vport->vf_info.request_uc_en = req->msg.en_uc;
+ vport->vf_info.request_mc_en = req->msg.en_mc;
+ vport->vf_info.request_bc_en = req->msg.en_bc;
if (req->msg.en_limit_promisc)
set_bit(HNAE3_PFLAG_LIMIT_PROMISC, &handle->priv_flags);
clear_bit(HNAE3_PFLAG_LIMIT_PROMISC,
&handle->priv_flags);
- ret = hclge_set_vport_promisc_mode(vport, en_uc, en_mc, en_bc);
-
- vport->vf_info.promisc_enable = (en_uc || en_mc) ? 1 : 0;
-
- return ret;
-}
-
-void hclge_inform_vf_promisc_info(struct hclge_vport *vport)
-{
- u8 dest_vfid = (u8)vport->vport_id;
- u8 msg_data[2];
-
- memcpy(&msg_data[0], &vport->vf_info.promisc_enable, sizeof(u16));
-
- hclge_send_mbx_msg(vport, msg_data, sizeof(msg_data),
- HCLGE_MBX_PUSH_PROMISC_INFO, dest_vfid);
+ set_bit(HCLGE_VPORT_STATE_PROMISC_CHANGE, &vport->state);
+ hclge_task_schedule(hdev, 0);
}
static int hclge_set_vf_uc_mac_addr(struct hclge_vport *vport,
}
int hclge_push_vf_port_base_vlan_info(struct hclge_vport *vport, u8 vfid,
- u16 state, u16 vlan_tag, u16 qos,
- u16 vlan_proto)
+ u16 state,
+ struct hclge_vlan_info *vlan_info)
{
#define MSG_DATA_SIZE 8
u8 msg_data[MSG_DATA_SIZE];
memcpy(&msg_data[0], &state, sizeof(u16));
- memcpy(&msg_data[2], &vlan_proto, sizeof(u16));
- memcpy(&msg_data[4], &qos, sizeof(u16));
- memcpy(&msg_data[6], &vlan_tag, sizeof(u16));
+ memcpy(&msg_data[2], &vlan_info->vlan_proto, sizeof(u16));
+ memcpy(&msg_data[4], &vlan_info->qos, sizeof(u16));
+ memcpy(&msg_data[6], &vlan_info->vlan_tag, sizeof(u16));
return hclge_send_mbx_msg(vport, msg_data, sizeof(msg_data),
HCLGE_MBX_PUSH_VLAN_INFO, vfid);
#define HCLGE_MBX_VLAN_STATE_OFFSET 0
#define HCLGE_MBX_VLAN_INFO_OFFSET 2
+ struct hnae3_handle *handle = &vport->nic;
+ struct hclge_dev *hdev = vport->back;
struct hclge_vf_vlan_cfg *msg_cmd;
- int status = 0;
msg_cmd = (struct hclge_vf_vlan_cfg *)&mbx_req->msg;
- if (msg_cmd->subcode == HCLGE_MBX_VLAN_FILTER) {
- struct hnae3_handle *handle = &vport->nic;
- u16 vlan, proto;
- bool is_kill;
-
- is_kill = !!msg_cmd->is_kill;
- vlan = msg_cmd->vlan;
- proto = msg_cmd->proto;
- status = hclge_set_vlan_filter(handle, cpu_to_be16(proto),
- vlan, is_kill);
- } else if (msg_cmd->subcode == HCLGE_MBX_VLAN_RX_OFF_CFG) {
- struct hnae3_handle *handle = &vport->nic;
- bool en = msg_cmd->is_kill ? true : false;
-
- status = hclge_en_hw_strip_rxvtag(handle, en);
- } else if (msg_cmd->subcode == HCLGE_MBX_PORT_BASE_VLAN_CFG) {
- struct hclge_vlan_info *vlan_info;
- u16 *state;
-
- state = (u16 *)&mbx_req->msg.data[HCLGE_MBX_VLAN_STATE_OFFSET];
- vlan_info = (struct hclge_vlan_info *)
- &mbx_req->msg.data[HCLGE_MBX_VLAN_INFO_OFFSET];
- status = hclge_update_port_base_vlan_cfg(vport, *state,
- vlan_info);
- } else if (msg_cmd->subcode == HCLGE_MBX_GET_PORT_BASE_VLAN_STATE) {
- struct hnae3_ae_dev *ae_dev = pci_get_drvdata(vport->nic.pdev);
+ switch (msg_cmd->subcode) {
+ case HCLGE_MBX_VLAN_FILTER:
+ return hclge_set_vlan_filter(handle,
+ cpu_to_be16(msg_cmd->proto),
+ msg_cmd->vlan, msg_cmd->is_kill);
+ case HCLGE_MBX_VLAN_RX_OFF_CFG:
+ return hclge_en_hw_strip_rxvtag(handle, msg_cmd->enable);
+ case HCLGE_MBX_GET_PORT_BASE_VLAN_STATE:
/* vf does not need to know about the port based VLAN state
* on device HNAE3_DEVICE_VERSION_V3. So always return disable
* on device HNAE3_DEVICE_VERSION_V3 if vf queries the port
* based VLAN state.
*/
resp_msg->data[0] =
- ae_dev->dev_version >= HNAE3_DEVICE_VERSION_V3 ?
+ hdev->ae_dev->dev_version >= HNAE3_DEVICE_VERSION_V3 ?
HNAE3_PORT_BASE_VLAN_DISABLE :
vport->port_base_vlan_cfg.state;
resp_msg->len = sizeof(u8);
+ return 0;
+ case HCLGE_MBX_ENABLE_VLAN_FILTER:
+ return hclge_enable_vport_vlan_filter(vport, msg_cmd->enable);
+ default:
+ return 0;
}
-
- return status;
}
static int hclge_set_vf_alive(struct hclge_vport *vport,
return ret;
}
-static void hclge_get_vf_tcinfo(struct hclge_vport *vport,
- struct hclge_respond_to_vf_msg *resp_msg)
+static void hclge_get_basic_info(struct hclge_vport *vport,
+ struct hclge_respond_to_vf_msg *resp_msg)
{
struct hnae3_knic_private_info *kinfo = &vport->nic.kinfo;
+ struct hnae3_ae_dev *ae_dev = vport->back->ae_dev;
+ struct hclge_basic_info *basic_info;
unsigned int i;
+ basic_info = (struct hclge_basic_info *)resp_msg->data;
for (i = 0; i < kinfo->tc_info.num_tc; i++)
- resp_msg->data[0] |= BIT(i);
+ basic_info->hw_tc_map |= BIT(i);
- resp_msg->len = sizeof(u8);
+ if (test_bit(HNAE3_DEV_SUPPORT_VLAN_FLTR_MDF_B, ae_dev->caps))
+ hnae3_set_bit(basic_info->pf_caps,
+ HNAE3_PF_SUPPORT_VLAN_FLTR_MDF_B, 1);
+
+ resp_msg->len = HCLGE_MBX_MAX_RESP_DATA_SIZE;
}
static void hclge_get_vf_queue_info(struct hclge_vport *vport,
unsigned int flag;
int ret = 0;
- memset(&resp_msg, 0, sizeof(resp_msg));
/* handle all the mailbox requests in the queue */
while (!hclge_cmd_crq_empty(&hdev->hw)) {
if (test_bit(HCLGE_STATE_CMD_DISABLE, &hdev->state)) {
trace_hclge_pf_mbx_get(hdev, req);
+ /* clear the resp_msg before processing every mailbox message */
+ memset(&resp_msg, 0, sizeof(resp_msg));
+
switch (req->msg.code) {
case HCLGE_MBX_MAP_RING_TO_VECTOR:
ret = hclge_map_unmap_ring_to_vf_vector(vport, true,
req);
break;
case HCLGE_MBX_SET_PROMISC_MODE:
- ret = hclge_set_vf_promisc_mode(vport, req);
- if (ret)
- dev_err(&hdev->pdev->dev,
- "PF fail(%d) to set VF promisc mode\n",
- ret);
+ hclge_set_vf_promisc_mode(vport, req);
break;
case HCLGE_MBX_SET_UNICAST:
ret = hclge_set_vf_uc_mac_addr(vport, req);
case HCLGE_MBX_GET_QDEPTH:
hclge_get_vf_queue_depth(vport, &resp_msg);
break;
- case HCLGE_MBX_GET_TCINFO:
- hclge_get_vf_tcinfo(vport, &resp_msg);
+ case HCLGE_MBX_GET_BASIC_INFO:
+ hclge_get_basic_info(vport, &resp_msg);
break;
case HCLGE_MBX_GET_LINK_STATUS:
ret = hclge_push_vf_link_status(vport);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+// Copyright (c) 2021 Hisilicon Limited.
+
+#include <linux/skbuff.h>
+#include "hclge_main.h"
+#include "hnae3.h"
+
+static int hclge_ptp_adjfreq(struct ptp_clock_info *ptp, s32 ppb)
+{
+ struct hclge_dev *hdev = hclge_ptp_get_hdev(ptp);
+ u64 adj_val, adj_base, diff;
+ unsigned long flags;
+ bool is_neg = false;
+ u32 quo, numerator;
+
+ if (ppb < 0) {
+ ppb = -ppb;
+ is_neg = true;
+ }
+
+ adj_base = HCLGE_PTP_CYCLE_ADJ_BASE * HCLGE_PTP_CYCLE_ADJ_UNIT;
+ adj_val = adj_base * ppb;
+ diff = div_u64(adj_val, 1000000000ULL);
+
+ if (is_neg)
+ adj_val = adj_base - diff;
+ else
+ adj_val = adj_base + diff;
+
+ /* This clock cycle is defined by three part: quotient, numerator
+ * and denominator. For example, 2.5ns, the quotient is 2,
+ * denominator is fixed to HCLGE_PTP_CYCLE_ADJ_UNIT, and numerator
+ * is 0.5 * HCLGE_PTP_CYCLE_ADJ_UNIT.
+ */
+ quo = div_u64_rem(adj_val, HCLGE_PTP_CYCLE_ADJ_UNIT, &numerator);
+
+ spin_lock_irqsave(&hdev->ptp->lock, flags);
+ writel(quo, hdev->ptp->io_base + HCLGE_PTP_CYCLE_QUO_REG);
+ writel(numerator, hdev->ptp->io_base + HCLGE_PTP_CYCLE_NUM_REG);
+ writel(HCLGE_PTP_CYCLE_ADJ_UNIT,
+ hdev->ptp->io_base + HCLGE_PTP_CYCLE_DEN_REG);
+ writel(HCLGE_PTP_CYCLE_ADJ_EN,
+ hdev->ptp->io_base + HCLGE_PTP_CYCLE_CFG_REG);
+ spin_unlock_irqrestore(&hdev->ptp->lock, flags);
+
+ return 0;
+}
+
+bool hclge_ptp_set_tx_info(struct hnae3_handle *handle, struct sk_buff *skb)
+{
+ struct hclge_vport *vport = hclge_get_vport(handle);
+ struct hclge_dev *hdev = vport->back;
+ struct hclge_ptp *ptp = hdev->ptp;
+
+ if (!test_bit(HCLGE_PTP_FLAG_TX_EN, &ptp->flags) ||
+ test_and_set_bit(HCLGE_STATE_PTP_TX_HANDLING, &hdev->state)) {
+ ptp->tx_skipped++;
+ return false;
+ }
+
+ ptp->tx_start = jiffies;
+ ptp->tx_skb = skb_get(skb);
+ ptp->tx_cnt++;
+
+ return true;
+}
+
+void hclge_ptp_clean_tx_hwts(struct hclge_dev *hdev)
+{
+ struct sk_buff *skb = hdev->ptp->tx_skb;
+ struct skb_shared_hwtstamps hwts;
+ u32 hi, lo;
+ u64 ns;
+
+ ns = readl(hdev->ptp->io_base + HCLGE_PTP_TX_TS_NSEC_REG) &
+ HCLGE_PTP_TX_TS_NSEC_MASK;
+ lo = readl(hdev->ptp->io_base + HCLGE_PTP_TX_TS_SEC_L_REG);
+ hi = readl(hdev->ptp->io_base + HCLGE_PTP_TX_TS_SEC_H_REG) &
+ HCLGE_PTP_TX_TS_SEC_H_MASK;
+ hdev->ptp->last_tx_seqid = readl(hdev->ptp->io_base +
+ HCLGE_PTP_TX_TS_SEQID_REG);
+
+ if (skb) {
+ hdev->ptp->tx_skb = NULL;
+ hdev->ptp->tx_cleaned++;
+
+ ns += (((u64)hi) << 32 | lo) * NSEC_PER_SEC;
+ hwts.hwtstamp = ns_to_ktime(ns);
+ skb_tstamp_tx(skb, &hwts);
+ dev_kfree_skb_any(skb);
+ }
+
+ clear_bit(HCLGE_STATE_PTP_TX_HANDLING, &hdev->state);
+}
+
+void hclge_ptp_get_rx_hwts(struct hnae3_handle *handle, struct sk_buff *skb,
+ u32 nsec, u32 sec)
+{
+ struct hclge_vport *vport = hclge_get_vport(handle);
+ struct hclge_dev *hdev = vport->back;
+ unsigned long flags;
+ u64 ns = nsec;
+ u32 sec_h;
+
+ if (!test_bit(HCLGE_PTP_FLAG_RX_EN, &hdev->ptp->flags))
+ return;
+
+ /* Since the BD does not have enough space for the higher 16 bits of
+ * second, and this part will not change frequently, so read it
+ * from register.
+ */
+ spin_lock_irqsave(&hdev->ptp->lock, flags);
+ sec_h = readl(hdev->ptp->io_base + HCLGE_PTP_CUR_TIME_SEC_H_REG);
+ spin_unlock_irqrestore(&hdev->ptp->lock, flags);
+
+ ns += (((u64)sec_h) << HCLGE_PTP_SEC_H_OFFSET | sec) * NSEC_PER_SEC;
+ skb_hwtstamps(skb)->hwtstamp = ns_to_ktime(ns);
+ hdev->ptp->last_rx = jiffies;
+ hdev->ptp->rx_cnt++;
+}
+
+static int hclge_ptp_gettimex(struct ptp_clock_info *ptp, struct timespec64 *ts,
+ struct ptp_system_timestamp *sts)
+{
+ struct hclge_dev *hdev = hclge_ptp_get_hdev(ptp);
+ unsigned long flags;
+ u32 hi, lo;
+ u64 ns;
+
+ spin_lock_irqsave(&hdev->ptp->lock, flags);
+ ns = readl(hdev->ptp->io_base + HCLGE_PTP_CUR_TIME_NSEC_REG);
+ hi = readl(hdev->ptp->io_base + HCLGE_PTP_CUR_TIME_SEC_H_REG);
+ lo = readl(hdev->ptp->io_base + HCLGE_PTP_CUR_TIME_SEC_L_REG);
+ spin_unlock_irqrestore(&hdev->ptp->lock, flags);
+
+ ns += (((u64)hi) << HCLGE_PTP_SEC_H_OFFSET | lo) * NSEC_PER_SEC;
+ *ts = ns_to_timespec64(ns);
+
+ return 0;
+}
+
+static int hclge_ptp_settime(struct ptp_clock_info *ptp,
+ const struct timespec64 *ts)
+{
+ struct hclge_dev *hdev = hclge_ptp_get_hdev(ptp);
+ unsigned long flags;
+
+ spin_lock_irqsave(&hdev->ptp->lock, flags);
+ writel(ts->tv_nsec, hdev->ptp->io_base + HCLGE_PTP_TIME_NSEC_REG);
+ writel(ts->tv_sec >> HCLGE_PTP_SEC_H_OFFSET,
+ hdev->ptp->io_base + HCLGE_PTP_TIME_SEC_H_REG);
+ writel(ts->tv_sec & HCLGE_PTP_SEC_L_MASK,
+ hdev->ptp->io_base + HCLGE_PTP_TIME_SEC_L_REG);
+ /* synchronize the time of phc */
+ writel(HCLGE_PTP_TIME_SYNC_EN,
+ hdev->ptp->io_base + HCLGE_PTP_TIME_SYNC_REG);
+ spin_unlock_irqrestore(&hdev->ptp->lock, flags);
+
+ return 0;
+}
+
+static int hclge_ptp_adjtime(struct ptp_clock_info *ptp, s64 delta)
+{
+ struct hclge_dev *hdev = hclge_ptp_get_hdev(ptp);
+ unsigned long flags;
+ bool is_neg = false;
+ u32 adj_val = 0;
+
+ if (delta < 0) {
+ adj_val |= HCLGE_PTP_TIME_NSEC_NEG;
+ delta = -delta;
+ is_neg = true;
+ }
+
+ if (delta > HCLGE_PTP_TIME_NSEC_MASK) {
+ struct timespec64 ts;
+ s64 ns;
+
+ hclge_ptp_gettimex(ptp, &ts, NULL);
+ ns = timespec64_to_ns(&ts);
+ ns = is_neg ? ns - delta : ns + delta;
+ ts = ns_to_timespec64(ns);
+ return hclge_ptp_settime(ptp, &ts);
+ }
+
+ adj_val |= delta & HCLGE_PTP_TIME_NSEC_MASK;
+
+ spin_lock_irqsave(&hdev->ptp->lock, flags);
+ writel(adj_val, hdev->ptp->io_base + HCLGE_PTP_TIME_NSEC_REG);
+ writel(HCLGE_PTP_TIME_ADJ_EN,
+ hdev->ptp->io_base + HCLGE_PTP_TIME_ADJ_REG);
+ spin_unlock_irqrestore(&hdev->ptp->lock, flags);
+
+ return 0;
+}
+
+int hclge_ptp_get_cfg(struct hclge_dev *hdev, struct ifreq *ifr)
+{
+ if (!test_bit(HCLGE_STATE_PTP_EN, &hdev->state))
+ return -EOPNOTSUPP;
+
+ return copy_to_user(ifr->ifr_data, &hdev->ptp->ts_cfg,
+ sizeof(struct hwtstamp_config)) ? -EFAULT : 0;
+}
+
+static int hclge_ptp_int_en(struct hclge_dev *hdev, bool en)
+{
+ struct hclge_ptp_int_cmd *req;
+ struct hclge_desc desc;
+ int ret;
+
+ req = (struct hclge_ptp_int_cmd *)desc.data;
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_PTP_INT_EN, false);
+ req->int_en = en ? 1 : 0;
+
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ if (ret)
+ dev_err(&hdev->pdev->dev,
+ "failed to %s ptp interrupt, ret = %d\n",
+ en ? "enable" : "disable", ret);
+
+ return ret;
+}
+
+int hclge_ptp_cfg_qry(struct hclge_dev *hdev, u32 *cfg)
+{
+ struct hclge_ptp_cfg_cmd *req;
+ struct hclge_desc desc;
+ int ret;
+
+ req = (struct hclge_ptp_cfg_cmd *)desc.data;
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_PTP_MODE_CFG, true);
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to query ptp config, ret = %d\n", ret);
+ return ret;
+ }
+
+ *cfg = le32_to_cpu(req->cfg);
+
+ return 0;
+}
+
+static int hclge_ptp_cfg(struct hclge_dev *hdev, u32 cfg)
+{
+ struct hclge_ptp_cfg_cmd *req;
+ struct hclge_desc desc;
+ int ret;
+
+ req = (struct hclge_ptp_cfg_cmd *)desc.data;
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_PTP_MODE_CFG, false);
+ req->cfg = cpu_to_le32(cfg);
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ if (ret)
+ dev_err(&hdev->pdev->dev,
+ "failed to config ptp, ret = %d\n", ret);
+
+ return ret;
+}
+
+static int hclge_ptp_set_tx_mode(struct hwtstamp_config *cfg,
+ unsigned long *flags, u32 *ptp_cfg)
+{
+ switch (cfg->tx_type) {
+ case HWTSTAMP_TX_OFF:
+ clear_bit(HCLGE_PTP_FLAG_TX_EN, flags);
+ break;
+ case HWTSTAMP_TX_ON:
+ set_bit(HCLGE_PTP_FLAG_TX_EN, flags);
+ *ptp_cfg |= HCLGE_PTP_TX_EN_B;
+ break;
+ default:
+ return -ERANGE;
+ }
+
+ return 0;
+}
+
+static int hclge_ptp_set_rx_mode(struct hwtstamp_config *cfg,
+ unsigned long *flags, u32 *ptp_cfg)
+{
+ int rx_filter = cfg->rx_filter;
+
+ switch (cfg->rx_filter) {
+ case HWTSTAMP_FILTER_NONE:
+ clear_bit(HCLGE_PTP_FLAG_RX_EN, flags);
+ break;
+ case HWTSTAMP_FILTER_PTP_V1_L4_SYNC:
+ case HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ:
+ case HWTSTAMP_FILTER_PTP_V1_L4_EVENT:
+ set_bit(HCLGE_PTP_FLAG_RX_EN, flags);
+ *ptp_cfg |= HCLGE_PTP_RX_EN_B;
+ *ptp_cfg |= HCLGE_PTP_UDP_FULL_TYPE << HCLGE_PTP_UDP_EN_SHIFT;
+ rx_filter = HWTSTAMP_FILTER_PTP_V1_L4_EVENT;
+ break;
+ case HWTSTAMP_FILTER_PTP_V2_EVENT:
+ case HWTSTAMP_FILTER_PTP_V2_L4_EVENT:
+ case HWTSTAMP_FILTER_PTP_V2_SYNC:
+ case HWTSTAMP_FILTER_PTP_V2_L4_SYNC:
+ case HWTSTAMP_FILTER_PTP_V2_DELAY_REQ:
+ case HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ:
+ case HWTSTAMP_FILTER_PTP_V2_L2_EVENT:
+ case HWTSTAMP_FILTER_PTP_V2_L2_SYNC:
+ case HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ:
+ set_bit(HCLGE_PTP_FLAG_RX_EN, flags);
+ *ptp_cfg |= HCLGE_PTP_RX_EN_B;
+ *ptp_cfg |= HCLGE_PTP_UDP_FULL_TYPE << HCLGE_PTP_UDP_EN_SHIFT;
+ *ptp_cfg |= HCLGE_PTP_MSG1_V2_DEFAULT << HCLGE_PTP_MSG1_SHIFT;
+ *ptp_cfg |= HCLGE_PTP_MSG0_V2_EVENT << HCLGE_PTP_MSG0_SHIFT;
+ *ptp_cfg |= HCLGE_PTP_MSG_TYPE_V2 << HCLGE_PTP_MSG_TYPE_SHIFT;
+ rx_filter = HWTSTAMP_FILTER_PTP_V2_EVENT;
+ break;
+ case HWTSTAMP_FILTER_ALL:
+ default:
+ return -ERANGE;
+ }
+
+ cfg->rx_filter = rx_filter;
+
+ return 0;
+}
+
+static int hclge_ptp_set_ts_mode(struct hclge_dev *hdev,
+ struct hwtstamp_config *cfg)
+{
+ unsigned long flags = hdev->ptp->flags;
+ u32 ptp_cfg = 0;
+ int ret;
+
+ if (test_bit(HCLGE_PTP_FLAG_EN, &hdev->ptp->flags))
+ ptp_cfg |= HCLGE_PTP_EN_B;
+
+ ret = hclge_ptp_set_tx_mode(cfg, &flags, &ptp_cfg);
+ if (ret)
+ return ret;
+
+ ret = hclge_ptp_set_rx_mode(cfg, &flags, &ptp_cfg);
+ if (ret)
+ return ret;
+
+ ret = hclge_ptp_cfg(hdev, ptp_cfg);
+ if (ret)
+ return ret;
+
+ hdev->ptp->flags = flags;
+ hdev->ptp->ptp_cfg = ptp_cfg;
+
+ return 0;
+}
+
+int hclge_ptp_set_cfg(struct hclge_dev *hdev, struct ifreq *ifr)
+{
+ struct hwtstamp_config cfg;
+ int ret;
+
+ if (!test_bit(HCLGE_STATE_PTP_EN, &hdev->state)) {
+ dev_err(&hdev->pdev->dev, "phc is unsupported\n");
+ return -EOPNOTSUPP;
+ }
+
+ if (copy_from_user(&cfg, ifr->ifr_data, sizeof(cfg)))
+ return -EFAULT;
+
+ ret = hclge_ptp_set_ts_mode(hdev, &cfg);
+ if (ret)
+ return ret;
+
+ hdev->ptp->ts_cfg = cfg;
+
+ return copy_to_user(ifr->ifr_data, &cfg, sizeof(cfg)) ? -EFAULT : 0;
+}
+
+int hclge_ptp_get_ts_info(struct hnae3_handle *handle,
+ struct ethtool_ts_info *info)
+{
+ struct hclge_vport *vport = hclge_get_vport(handle);
+ struct hclge_dev *hdev = vport->back;
+
+ if (!test_bit(HCLGE_STATE_PTP_EN, &hdev->state)) {
+ dev_err(&hdev->pdev->dev, "phc is unsupported\n");
+ return -EOPNOTSUPP;
+ }
+
+ info->so_timestamping = SOF_TIMESTAMPING_TX_SOFTWARE |
+ SOF_TIMESTAMPING_RX_SOFTWARE |
+ SOF_TIMESTAMPING_SOFTWARE |
+ SOF_TIMESTAMPING_TX_HARDWARE |
+ SOF_TIMESTAMPING_RX_HARDWARE |
+ SOF_TIMESTAMPING_RAW_HARDWARE;
+
+ if (hdev->ptp->clock)
+ info->phc_index = ptp_clock_index(hdev->ptp->clock);
+ else
+ info->phc_index = -1;
+
+ info->tx_types = BIT(HWTSTAMP_TX_OFF) | BIT(HWTSTAMP_TX_ON);
+
+ info->rx_filters = BIT(HWTSTAMP_FILTER_NONE) |
+ BIT(HWTSTAMP_FILTER_PTP_V2_L2_EVENT) |
+ BIT(HWTSTAMP_FILTER_PTP_V2_L2_SYNC) |
+ BIT(HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ);
+
+ info->rx_filters |= BIT(HWTSTAMP_FILTER_PTP_V1_L4_SYNC) |
+ BIT(HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ) |
+ BIT(HWTSTAMP_FILTER_PTP_V2_EVENT) |
+ BIT(HWTSTAMP_FILTER_PTP_V2_L4_EVENT) |
+ BIT(HWTSTAMP_FILTER_PTP_V2_SYNC) |
+ BIT(HWTSTAMP_FILTER_PTP_V2_L4_SYNC) |
+ BIT(HWTSTAMP_FILTER_PTP_V2_DELAY_REQ) |
+ BIT(HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ);
+
+ return 0;
+}
+
+static int hclge_ptp_create_clock(struct hclge_dev *hdev)
+{
+ struct hclge_ptp *ptp;
+
+ ptp = devm_kzalloc(&hdev->pdev->dev, sizeof(*ptp), GFP_KERNEL);
+ if (!ptp)
+ return -ENOMEM;
+
+ ptp->hdev = hdev;
+ snprintf(ptp->info.name, sizeof(ptp->info.name), "%s",
+ HCLGE_DRIVER_NAME);
+ ptp->info.owner = THIS_MODULE;
+ ptp->info.max_adj = HCLGE_PTP_CYCLE_ADJ_MAX;
+ ptp->info.n_ext_ts = 0;
+ ptp->info.pps = 0;
+ ptp->info.adjfreq = hclge_ptp_adjfreq;
+ ptp->info.adjtime = hclge_ptp_adjtime;
+ ptp->info.gettimex64 = hclge_ptp_gettimex;
+ ptp->info.settime64 = hclge_ptp_settime;
+
+ ptp->info.n_alarm = 0;
+ ptp->clock = ptp_clock_register(&ptp->info, &hdev->pdev->dev);
+ if (IS_ERR(ptp->clock)) {
+ dev_err(&hdev->pdev->dev,
+ "%d failed to register ptp clock, ret = %ld\n",
+ ptp->info.n_alarm, PTR_ERR(ptp->clock));
+ return -ENODEV;
+ } else if (!ptp->clock) {
+ dev_err(&hdev->pdev->dev, "failed to register ptp clock\n");
+ return -ENODEV;
+ }
+
+ spin_lock_init(&ptp->lock);
+ ptp->io_base = hdev->hw.io_base + HCLGE_PTP_REG_OFFSET;
+ ptp->ts_cfg.rx_filter = HWTSTAMP_FILTER_NONE;
+ ptp->ts_cfg.tx_type = HWTSTAMP_TX_OFF;
+ hdev->ptp = ptp;
+
+ return 0;
+}
+
+static void hclge_ptp_destroy_clock(struct hclge_dev *hdev)
+{
+ ptp_clock_unregister(hdev->ptp->clock);
+ hdev->ptp->clock = NULL;
+ devm_kfree(&hdev->pdev->dev, hdev->ptp);
+ hdev->ptp = NULL;
+}
+
+int hclge_ptp_init(struct hclge_dev *hdev)
+{
+ struct hnae3_ae_dev *ae_dev = pci_get_drvdata(hdev->pdev);
+ struct timespec64 ts;
+ int ret;
+
+ if (!test_bit(HNAE3_DEV_SUPPORT_PTP_B, ae_dev->caps))
+ return 0;
+
+ if (!hdev->ptp) {
+ ret = hclge_ptp_create_clock(hdev);
+ if (ret)
+ return ret;
+ }
+
+ ret = hclge_ptp_int_en(hdev, true);
+ if (ret)
+ goto out;
+
+ set_bit(HCLGE_PTP_FLAG_EN, &hdev->ptp->flags);
+ ret = hclge_ptp_adjfreq(&hdev->ptp->info, 0);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to init freq, ret = %d\n", ret);
+ goto out;
+ }
+
+ ret = hclge_ptp_set_ts_mode(hdev, &hdev->ptp->ts_cfg);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to init ts mode, ret = %d\n", ret);
+ goto out;
+ }
+
+ ktime_get_real_ts64(&ts);
+ ret = hclge_ptp_settime(&hdev->ptp->info, &ts);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to init ts time, ret = %d\n", ret);
+ goto out;
+ }
+
+ set_bit(HCLGE_STATE_PTP_EN, &hdev->state);
+ dev_info(&hdev->pdev->dev, "phc initializes ok!\n");
+
+ return 0;
+
+out:
+ hclge_ptp_destroy_clock(hdev);
+
+ return ret;
+}
+
+void hclge_ptp_uninit(struct hclge_dev *hdev)
+{
+ struct hclge_ptp *ptp = hdev->ptp;
+
+ if (!ptp)
+ return;
+
+ hclge_ptp_int_en(hdev, false);
+ clear_bit(HCLGE_STATE_PTP_EN, &hdev->state);
+ clear_bit(HCLGE_PTP_FLAG_EN, &ptp->flags);
+ ptp->ts_cfg.rx_filter = HWTSTAMP_FILTER_NONE;
+ ptp->ts_cfg.tx_type = HWTSTAMP_TX_OFF;
+
+ if (hclge_ptp_set_ts_mode(hdev, &ptp->ts_cfg))
+ dev_err(&hdev->pdev->dev, "failed to disable phc\n");
+
+ if (ptp->tx_skb) {
+ struct sk_buff *skb = ptp->tx_skb;
+
+ ptp->tx_skb = NULL;
+ dev_kfree_skb_any(skb);
+ }
+
+ hclge_ptp_destroy_clock(hdev);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0+ */
+// Copyright (c) 2021 Hisilicon Limited.
+
+#ifndef __HCLGE_PTP_H
+#define __HCLGE_PTP_H
+
+#include <linux/ptp_clock_kernel.h>
+#include <linux/net_tstamp.h>
+#include <linux/types.h>
+
+#define HCLGE_PTP_REG_OFFSET 0x29000
+
+#define HCLGE_PTP_TX_TS_SEQID_REG 0x0
+#define HCLGE_PTP_TX_TS_NSEC_REG 0x4
+#define HCLGE_PTP_TX_TS_NSEC_MASK GENMASK(29, 0)
+#define HCLGE_PTP_TX_TS_SEC_L_REG 0x8
+#define HCLGE_PTP_TX_TS_SEC_H_REG 0xC
+#define HCLGE_PTP_TX_TS_SEC_H_MASK GENMASK(15, 0)
+#define HCLGE_PTP_TX_TS_CNT_REG 0x30
+
+#define HCLGE_PTP_TIME_SEC_H_REG 0x50
+#define HCLGE_PTP_TIME_SEC_H_MASK GENMASK(15, 0)
+#define HCLGE_PTP_TIME_SEC_L_REG 0x54
+#define HCLGE_PTP_TIME_NSEC_REG 0x58
+#define HCLGE_PTP_TIME_NSEC_MASK GENMASK(29, 0)
+#define HCLGE_PTP_TIME_NSEC_NEG BIT(31)
+#define HCLGE_PTP_TIME_SYNC_REG 0x5C
+#define HCLGE_PTP_TIME_SYNC_EN BIT(0)
+#define HCLGE_PTP_TIME_ADJ_REG 0x60
+#define HCLGE_PTP_TIME_ADJ_EN BIT(0)
+#define HCLGE_PTP_CYCLE_QUO_REG 0x64
+#define HCLGE_PTP_CYCLE_DEN_REG 0x68
+#define HCLGE_PTP_CYCLE_NUM_REG 0x6C
+#define HCLGE_PTP_CYCLE_CFG_REG 0x70
+#define HCLGE_PTP_CYCLE_ADJ_EN BIT(0)
+#define HCLGE_PTP_CUR_TIME_SEC_H_REG 0x74
+#define HCLGE_PTP_CUR_TIME_SEC_L_REG 0x78
+#define HCLGE_PTP_CUR_TIME_NSEC_REG 0x7C
+
+#define HCLGE_PTP_CYCLE_ADJ_BASE 2
+#define HCLGE_PTP_CYCLE_ADJ_MAX 500000000
+#define HCLGE_PTP_CYCLE_ADJ_UNIT 100000000
+#define HCLGE_PTP_SEC_H_OFFSET 32u
+#define HCLGE_PTP_SEC_L_MASK GENMASK(31, 0)
+
+#define HCLGE_PTP_FLAG_EN 0
+#define HCLGE_PTP_FLAG_TX_EN 1
+#define HCLGE_PTP_FLAG_RX_EN 2
+
+struct hclge_ptp {
+ struct hclge_dev *hdev;
+ struct ptp_clock *clock;
+ struct sk_buff *tx_skb;
+ unsigned long flags;
+ void __iomem *io_base;
+ struct ptp_clock_info info;
+ struct hwtstamp_config ts_cfg;
+ spinlock_t lock; /* protects ptp registers */
+ u32 ptp_cfg;
+ u32 last_tx_seqid;
+ unsigned long tx_start;
+ unsigned long tx_cnt;
+ unsigned long tx_skipped;
+ unsigned long tx_cleaned;
+ unsigned long last_rx;
+ unsigned long rx_cnt;
+ unsigned long tx_timeout;
+};
+
+struct hclge_ptp_int_cmd {
+#define HCLGE_PTP_INT_EN_B BIT(0)
+
+ u8 int_en;
+ u8 rsvd[23];
+};
+
+enum hclge_ptp_udp_type {
+ HCLGE_PTP_UDP_NOT_TYPE,
+ HCLGE_PTP_UDP_P13F_TYPE,
+ HCLGE_PTP_UDP_P140_TYPE,
+ HCLGE_PTP_UDP_FULL_TYPE,
+};
+
+enum hclge_ptp_msg_type {
+ HCLGE_PTP_MSG_TYPE_V2_L2,
+ HCLGE_PTP_MSG_TYPE_V2,
+ HCLGE_PTP_MSG_TYPE_V2_EVENT,
+};
+
+enum hclge_ptp_msg0_type {
+ HCLGE_PTP_MSG0_V2_DELAY_REQ = 1,
+ HCLGE_PTP_MSG0_V2_PDELAY_REQ,
+ HCLGE_PTP_MSG0_V2_DELAY_RESP,
+ HCLGE_PTP_MSG0_V2_EVENT = 0xF,
+};
+
+#define HCLGE_PTP_MSG1_V2_DEFAULT 1
+
+struct hclge_ptp_cfg_cmd {
+#define HCLGE_PTP_EN_B BIT(0)
+#define HCLGE_PTP_TX_EN_B BIT(1)
+#define HCLGE_PTP_RX_EN_B BIT(2)
+#define HCLGE_PTP_UDP_EN_SHIFT 3
+#define HCLGE_PTP_UDP_EN_MASK GENMASK(4, 3)
+#define HCLGE_PTP_MSG_TYPE_SHIFT 8
+#define HCLGE_PTP_MSG_TYPE_MASK GENMASK(9, 8)
+#define HCLGE_PTP_MSG1_SHIFT 16
+#define HCLGE_PTP_MSG1_MASK GENMASK(19, 16)
+#define HCLGE_PTP_MSG0_SHIFT 24
+#define HCLGE_PTP_MSG0_MASK GENMASK(27, 24)
+
+ __le32 cfg;
+ u8 rsvd[20];
+};
+
+static inline struct hclge_dev *hclge_ptp_get_hdev(struct ptp_clock_info *info)
+{
+ struct hclge_ptp *ptp = container_of(info, struct hclge_ptp, info);
+
+ return ptp->hdev;
+}
+
+bool hclge_ptp_set_tx_info(struct hnae3_handle *handle, struct sk_buff *skb);
+void hclge_ptp_clean_tx_hwts(struct hclge_dev *dev);
+void hclge_ptp_get_rx_hwts(struct hnae3_handle *handle, struct sk_buff *skb,
+ u32 nsec, u32 sec);
+int hclge_ptp_get_cfg(struct hclge_dev *hdev, struct ifreq *ifr);
+int hclge_ptp_set_cfg(struct hclge_dev *hdev, struct ifreq *ifr);
+int hclge_ptp_init(struct hclge_dev *hdev);
+void hclge_ptp_uninit(struct hclge_dev *hdev);
+int hclge_ptp_get_ts_info(struct hnae3_handle *handle,
+ struct ethtool_ts_info *info);
+int hclge_ptp_cfg_qry(struct hclge_dev *hdev, u32 *cfg);
+#endif
return 0;
}
+int hclge_tm_get_qset_shaper(struct hclge_dev *hdev, u16 qset_id,
+ struct hclge_tm_shaper_para *para)
+{
+ struct hclge_qs_shapping_cmd *shap_cfg_cmd;
+ struct hclge_desc desc;
+ u32 shapping_para;
+ int ret;
+
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_QCN_SHAPPING_CFG, true);
+ shap_cfg_cmd = (struct hclge_qs_shapping_cmd *)desc.data;
+ shap_cfg_cmd->qs_id = cpu_to_le16(qset_id);
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to get qset %u shaper, ret = %d\n", qset_id,
+ ret);
+ return ret;
+ }
+
+ shapping_para = le32_to_cpu(shap_cfg_cmd->qs_shapping_para);
+ para->ir_b = hclge_tm_get_field(shapping_para, IR_B);
+ para->ir_u = hclge_tm_get_field(shapping_para, IR_U);
+ para->ir_s = hclge_tm_get_field(shapping_para, IR_S);
+ para->bs_b = hclge_tm_get_field(shapping_para, BS_B);
+ para->bs_s = hclge_tm_get_field(shapping_para, BS_S);
+ para->flag = shap_cfg_cmd->flag;
+ para->rate = le32_to_cpu(shap_cfg_cmd->qs_rate);
+ return 0;
+}
+
int hclge_tm_get_pri_sch_mode(struct hclge_dev *hdev, u8 pri_id, u8 *mode)
{
struct hclge_pri_sch_mode_cfg_cmd *pri_sch_mode;
int hclge_tm_get_pri_shaper(struct hclge_dev *hdev, u8 pri_id,
enum hclge_opcode_type cmd,
- struct hclge_pri_shaper_para *para)
+ struct hclge_tm_shaper_para *para)
{
struct hclge_pri_shapping_cmd *shap_cfg_cmd;
struct hclge_desc desc;
para->rate = le32_to_cpu(shap_cfg_cmd->pri_rate);
return 0;
}
+
+int hclge_tm_get_q_to_qs_map(struct hclge_dev *hdev, u16 q_id, u16 *qset_id)
+{
+ struct hclge_nq_to_qs_link_cmd *map;
+ struct hclge_desc desc;
+ u16 qs_id_l;
+ u16 qs_id_h;
+ int ret;
+
+ map = (struct hclge_nq_to_qs_link_cmd *)desc.data;
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_TM_NQ_TO_QS_LINK, true);
+ map->nq_id = cpu_to_le16(q_id);
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to get queue to qset map, ret = %d\n", ret);
+ return ret;
+ }
+ *qset_id = le16_to_cpu(map->qset_id);
+
+ /* convert qset_id to the following format, drop the vld bit
+ * | qs_id_h | vld | qs_id_l |
+ * qset_id: | 15 ~ 11 | 10 | 9 ~ 0 |
+ * \ \ / /
+ * \ \ / /
+ * qset_id: | 15 | 14 ~ 10 | 9 ~ 0 |
+ */
+ qs_id_l = hnae3_get_field(*qset_id, HCLGE_TM_QS_ID_L_MSK,
+ HCLGE_TM_QS_ID_L_S);
+ qs_id_h = hnae3_get_field(*qset_id, HCLGE_TM_QS_ID_H_EXT_MSK,
+ HCLGE_TM_QS_ID_H_EXT_S);
+ *qset_id = 0;
+ hnae3_set_field(*qset_id, HCLGE_TM_QS_ID_L_MSK, HCLGE_TM_QS_ID_L_S,
+ qs_id_l);
+ hnae3_set_field(*qset_id, HCLGE_TM_QS_ID_H_MSK, HCLGE_TM_QS_ID_H_S,
+ qs_id_h);
+ return 0;
+}
+
+int hclge_tm_get_q_to_tc(struct hclge_dev *hdev, u16 q_id, u8 *tc_id)
+{
+#define HCLGE_TM_TC_MASK 0x7
+
+ struct hclge_tqp_tx_queue_tc_cmd *tc;
+ struct hclge_desc desc;
+ int ret;
+
+ tc = (struct hclge_tqp_tx_queue_tc_cmd *)desc.data;
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_TQP_TX_QUEUE_TC, true);
+ tc->queue_id = cpu_to_le16(q_id);
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to get queue to tc map, ret = %d\n", ret);
+ return ret;
+ }
+
+ *tc_id = tc->tc_id & HCLGE_TM_TC_MASK;
+ return 0;
+}
+
+int hclge_tm_get_pg_to_pri_map(struct hclge_dev *hdev, u8 pg_id,
+ u8 *pri_bit_map)
+{
+ struct hclge_pg_to_pri_link_cmd *map;
+ struct hclge_desc desc;
+ int ret;
+
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_TM_PG_TO_PRI_LINK, true);
+ map = (struct hclge_pg_to_pri_link_cmd *)desc.data;
+ map->pg_id = pg_id;
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to get pg to pri map, ret = %d\n", ret);
+ return ret;
+ }
+
+ *pri_bit_map = map->pri_bit_map;
+ return 0;
+}
+
+int hclge_tm_get_pg_weight(struct hclge_dev *hdev, u8 pg_id, u8 *weight)
+{
+ struct hclge_pg_weight_cmd *pg_weight_cmd;
+ struct hclge_desc desc;
+ int ret;
+
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_TM_PG_WEIGHT, true);
+ pg_weight_cmd = (struct hclge_pg_weight_cmd *)desc.data;
+ pg_weight_cmd->pg_id = pg_id;
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to get pg weight, ret = %d\n", ret);
+ return ret;
+ }
+
+ *weight = pg_weight_cmd->dwrr;
+ return 0;
+}
+
+int hclge_tm_get_pg_sch_mode(struct hclge_dev *hdev, u8 pg_id, u8 *mode)
+{
+ struct hclge_desc desc;
+ int ret;
+
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_TM_PG_SCH_MODE_CFG, true);
+ desc.data[0] = cpu_to_le32(pg_id);
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to get pg sch mode, ret = %d\n", ret);
+ return ret;
+ }
+
+ *mode = (u8)le32_to_cpu(desc.data[1]);
+ return 0;
+}
+
+int hclge_tm_get_pg_shaper(struct hclge_dev *hdev, u8 pg_id,
+ enum hclge_opcode_type cmd,
+ struct hclge_tm_shaper_para *para)
+{
+ struct hclge_pg_shapping_cmd *shap_cfg_cmd;
+ struct hclge_desc desc;
+ u32 shapping_para;
+ int ret;
+
+ if (cmd != HCLGE_OPC_TM_PG_C_SHAPPING &&
+ cmd != HCLGE_OPC_TM_PG_P_SHAPPING)
+ return -EINVAL;
+
+ hclge_cmd_setup_basic_desc(&desc, cmd, true);
+ shap_cfg_cmd = (struct hclge_pg_shapping_cmd *)desc.data;
+ shap_cfg_cmd->pg_id = pg_id;
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to get pg shaper(%#x), ret = %d\n",
+ cmd, ret);
+ return ret;
+ }
+
+ shapping_para = le32_to_cpu(shap_cfg_cmd->pg_shapping_para);
+ para->ir_b = hclge_tm_get_field(shapping_para, IR_B);
+ para->ir_u = hclge_tm_get_field(shapping_para, IR_U);
+ para->ir_s = hclge_tm_get_field(shapping_para, IR_S);
+ para->bs_b = hclge_tm_get_field(shapping_para, BS_B);
+ para->bs_s = hclge_tm_get_field(shapping_para, BS_S);
+ para->flag = shap_cfg_cmd->flag;
+ para->rate = le32_to_cpu(shap_cfg_cmd->pg_rate);
+ return 0;
+}
+
+int hclge_tm_get_port_shaper(struct hclge_dev *hdev,
+ struct hclge_tm_shaper_para *para)
+{
+ struct hclge_port_shapping_cmd *port_shap_cfg_cmd;
+ struct hclge_desc desc;
+ u32 shapping_para;
+ int ret;
+
+ hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_TM_PORT_SHAPPING, true);
+ ret = hclge_cmd_send(&hdev->hw, &desc, 1);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to get port shaper, ret = %d\n", ret);
+ return ret;
+ }
+
+ port_shap_cfg_cmd = (struct hclge_port_shapping_cmd *)desc.data;
+ shapping_para = le32_to_cpu(port_shap_cfg_cmd->port_shapping_para);
+ para->ir_b = hclge_tm_get_field(shapping_para, IR_B);
+ para->ir_u = hclge_tm_get_field(shapping_para, IR_U);
+ para->ir_s = hclge_tm_get_field(shapping_para, IR_S);
+ para->bs_b = hclge_tm_get_field(shapping_para, BS_B);
+ para->bs_s = hclge_tm_get_field(shapping_para, BS_S);
+ para->flag = port_shap_cfg_cmd->flag;
+ para->rate = le32_to_cpu(port_shap_cfg_cmd->port_rate);
+
+ return 0;
+}
__le16 queue_num;
};
-struct hclge_pri_shaper_para {
+struct hclge_tm_shaper_para {
+ u32 rate;
u8 ir_b;
u8 ir_u;
u8 ir_s;
u8 bs_b;
u8 bs_s;
u8 flag;
- u32 rate;
};
#define hclge_tm_set_field(dest, string, val) \
u8 *link_vld);
int hclge_tm_get_qset_sch_mode(struct hclge_dev *hdev, u16 qset_id, u8 *mode);
int hclge_tm_get_qset_weight(struct hclge_dev *hdev, u16 qset_id, u8 *weight);
+int hclge_tm_get_qset_shaper(struct hclge_dev *hdev, u16 qset_id,
+ struct hclge_tm_shaper_para *para);
int hclge_tm_get_pri_sch_mode(struct hclge_dev *hdev, u8 pri_id, u8 *mode);
int hclge_tm_get_pri_weight(struct hclge_dev *hdev, u8 pri_id, u8 *weight);
int hclge_tm_get_pri_shaper(struct hclge_dev *hdev, u8 pri_id,
enum hclge_opcode_type cmd,
- struct hclge_pri_shaper_para *para);
+ struct hclge_tm_shaper_para *para);
+int hclge_tm_get_q_to_qs_map(struct hclge_dev *hdev, u16 q_id, u16 *qset_id);
+int hclge_tm_get_q_to_tc(struct hclge_dev *hdev, u16 q_id, u8 *tc_id);
+int hclge_tm_get_pg_to_pri_map(struct hclge_dev *hdev, u8 pg_id,
+ u8 *pri_bit_map);
+int hclge_tm_get_pg_weight(struct hclge_dev *hdev, u8 pg_id, u8 *weight);
+int hclge_tm_get_pg_sch_mode(struct hclge_dev *hdev, u8 pg_id, u8 *mode);
+int hclge_tm_get_pg_shaper(struct hclge_dev *hdev, u8 pg_id,
+ enum hclge_opcode_type cmd,
+ struct hclge_tm_shaper_para *para);
+int hclge_tm_get_port_shaper(struct hclge_dev *hdev,
+ struct hclge_tm_shaper_para *para);
#endif
set_bit(HNAE3_DEV_SUPPORT_HW_TX_CSUM_B, ae_dev->caps);
if (hnae3_get_bit(caps, HCLGEVF_CAP_UDP_TUNNEL_CSUM_B))
set_bit(HNAE3_DEV_SUPPORT_UDP_TUNNEL_CSUM_B, ae_dev->caps);
+ if (hnae3_get_bit(caps, HCLGEVF_CAP_RXD_ADV_LAYOUT_B))
+ set_bit(HNAE3_DEV_SUPPORT_RXD_ADV_LAYOUT_B, ae_dev->caps);
}
static __le32 hclgevf_build_api_caps(void)
HCLGEVF_CAP_HW_PAD_B,
HCLGEVF_CAP_STASH_B,
HCLGEVF_CAP_UDP_TUNNEL_CSUM_B,
+ HCLGEVF_CAP_RXD_ADV_LAYOUT_B = 15,
};
enum HCLGEVF_API_CAP_BITS {
}
}
-static int hclgevf_get_tc_info(struct hclgevf_dev *hdev)
+static int hclgevf_get_basic_info(struct hclgevf_dev *hdev)
{
+ struct hnae3_ae_dev *ae_dev = hdev->ae_dev;
+ u8 resp_msg[HCLGE_MBX_MAX_RESP_DATA_SIZE];
+ struct hclge_basic_info *basic_info;
struct hclge_vf_to_pf_msg send_msg;
- u8 resp_msg;
+ unsigned long caps;
int status;
- hclgevf_build_send_msg(&send_msg, HCLGE_MBX_GET_TCINFO, 0);
- status = hclgevf_send_mbx_msg(hdev, &send_msg, true, &resp_msg,
+ hclgevf_build_send_msg(&send_msg, HCLGE_MBX_GET_BASIC_INFO, 0);
+ status = hclgevf_send_mbx_msg(hdev, &send_msg, true, resp_msg,
sizeof(resp_msg));
if (status) {
dev_err(&hdev->pdev->dev,
- "VF request to get TC info from PF failed %d",
- status);
+ "failed to get basic info from pf, ret = %d", status);
return status;
}
- hdev->hw_tc_map = resp_msg;
+ basic_info = (struct hclge_basic_info *)resp_msg;
+
+ hdev->hw_tc_map = basic_info->hw_tc_map;
+ hdev->mbx_api_version = basic_info->mbx_api_version;
+ caps = basic_info->pf_caps;
+ if (test_bit(HNAE3_PF_SUPPORT_VLAN_FLTR_MDF_B, &caps))
+ set_bit(HNAE3_DEV_SUPPORT_VLAN_FLTR_MDF_B, ae_dev->caps);
return 0;
}
kfree(mac_node);
} else if (mac_node->state == HCLGEVF_MAC_ACTIVE) {
mac_node->state = HCLGEVF_MAC_TO_DEL;
- list_del(&mac_node->node);
- list_add_tail(&mac_node->node, mac_list);
+ list_move_tail(&mac_node->node, mac_list);
} else {
list_del(&mac_node->node);
kfree(mac_node);
list_del(&mac_node->node);
kfree(mac_node);
} else {
- list_del(&mac_node->node);
- list_add_tail(&mac_node->node, mac_list);
+ list_move_tail(&mac_node->node, mac_list);
}
}
}
list_for_each_entry_safe(mac_node, tmp, list, node) {
switch (mac_node->state) {
case HCLGEVF_MAC_TO_DEL:
- list_del(&mac_node->node);
- list_add_tail(&mac_node->node, &tmp_del_list);
+ list_move_tail(&mac_node->node, &tmp_del_list);
break;
case HCLGEVF_MAC_TO_ADD:
new_node = kzalloc(sizeof(*new_node), GFP_ATOMIC);
spin_unlock_bh(&hdev->mac_table.mac_list_lock);
}
+static int hclgevf_enable_vlan_filter(struct hnae3_handle *handle, bool enable)
+{
+ struct hclgevf_dev *hdev = hclgevf_ae_get_hdev(handle);
+ struct hnae3_ae_dev *ae_dev = hdev->ae_dev;
+ struct hclge_vf_to_pf_msg send_msg;
+
+ if (!test_bit(HNAE3_DEV_SUPPORT_VLAN_FLTR_MDF_B, ae_dev->caps))
+ return -EOPNOTSUPP;
+
+ hclgevf_build_send_msg(&send_msg, HCLGE_MBX_SET_VLAN,
+ HCLGE_MBX_ENABLE_VLAN_FILTER);
+ send_msg.data[0] = enable ? 1 : 0;
+
+ return hclgevf_send_mbx_msg(hdev, &send_msg, true, NULL, 0);
+}
+
static int hclgevf_set_vlan_filter(struct hnae3_handle *handle,
__be16 proto, u16 vlan_id,
bool is_kill)
{
int ret;
+ ret = hclgevf_get_basic_info(hdev);
+ if (ret)
+ return ret;
+
/* get current port based vlan state from PF */
ret = hclgevf_get_port_base_vlan_filter_state(hdev);
if (ret)
if (ret)
return ret;
- ret = hclgevf_get_pf_media_type(hdev);
- if (ret)
- return ret;
-
- /* get tc configuration from PF */
- return hclgevf_get_tc_info(hdev);
+ return hclgevf_get_pf_media_type(hdev);
}
static int hclgevf_alloc_hdev(struct hnae3_ae_dev *ae_dev)
return hclgevf_send_mbx_msg(hdev, &send_msg, false, NULL, 0);
}
+static void hclgevf_init_rxd_adv_layout(struct hclgevf_dev *hdev)
+{
+ if (hnae3_ae_dev_rxd_adv_layout_supported(hdev->ae_dev))
+ hclgevf_write_dev(&hdev->hw, HCLGEVF_RXD_ADV_LAYOUT_EN_REG, 1);
+}
+
+static void hclgevf_uninit_rxd_adv_layout(struct hclgevf_dev *hdev)
+{
+ if (hnae3_ae_dev_rxd_adv_layout_supported(hdev->ae_dev))
+ hclgevf_write_dev(&hdev->hw, HCLGEVF_RXD_ADV_LAYOUT_EN_REG, 0);
+}
+
static int hclgevf_reset_hdev(struct hclgevf_dev *hdev)
{
struct pci_dev *pdev = hdev->pdev;
set_bit(HCLGEVF_STATE_PROMISC_CHANGED, &hdev->state);
+ hclgevf_init_rxd_adv_layout(hdev);
+
dev_info(&hdev->pdev->dev, "Reset done\n");
return 0;
goto err_config;
}
+ hclgevf_init_rxd_adv_layout(hdev);
+
hdev->last_reset_time = jiffies;
dev_info(&hdev->pdev->dev, "finished initializing %s driver\n",
HCLGEVF_DRIVER_NAME);
struct hclge_vf_to_pf_msg send_msg;
hclgevf_state_uninit(hdev);
+ hclgevf_uninit_rxd_adv_layout(hdev);
hclgevf_build_send_msg(&send_msg, HCLGE_MBX_VF_UNINIT, 0);
hclgevf_send_mbx_msg(hdev, &send_msg, false, NULL, 0);
.get_tc_size = hclgevf_get_tc_size,
.get_fw_version = hclgevf_get_fw_version,
.set_vlan_filter = hclgevf_set_vlan_filter,
+ .enable_vlan_filter = hclgevf_enable_vlan_filter,
.enable_hw_strip_rxvtag = hclgevf_en_hw_strip_rxvtag,
.reset_event = hclgevf_reset_event,
.set_default_reset_request = hclgevf_set_def_reset_request,
/* bar registers for common func */
#define HCLGEVF_GRO_EN_REG 0x28000
+#define HCLGEVF_RXD_ADV_LAYOUT_EN_REG 0x28008
/* bar registers for rcb */
#define HCLGEVF_RING_RX_ADDR_L_REG 0x80000
struct semaphore reset_sem; /* protect reset process */
u32 fw_version;
+ u16 mbx_api_version;
u16 num_tqps; /* num task queue pairs of this VF */
u16 alloc_rss_size; /* allocated RSS task queue */
}
/**
- * cmdq_arm_ceq_handler - cmdq completion event handler for sync command
+ * cmdq_sync_cmd_handler - cmdq completion event handler for sync command
* @cmdq: the cmdq of the command
* @cons_idx: the consumer index to update the error code for
* @errcode: the error code
};
/**
- * get_capability - convert device capabilities to NIC capabilities
+ * parse_capability - convert device capabilities to NIC capabilities
* @hwdev: the HW device to set and convert device capabilities for
* @dev_cap: device capabilities from FW
*
}
/**
- * get_cap_from_fw - get device capabilities from FW
+ * get_capability - get device capabilities from FW
* @pfhwdev: the PF HW device to get capabilities for
*
* Return 0 - Success, negative - Failure
}
/**
- * clear_io_resource - set the IO resources as not active in the NIC
+ * clear_io_resources - set the IO resources as not active in the NIC
* @hwdev: the NIC HW device
*
* Return 0 - Success, negative - Failure
}
/**
- * hinic_hwdev_get_sq - get RQ
+ * hinic_hwdev_get_rq - get RQ
* @hwdev: the NIC HW device
* @i: the position of the RQ
*
}
/**
- * dma_attr_table_init - initialize the default dma attributes
+ * dma_attr_init - initialize the default dma attributes
* @hwif: the HW interface of a pci function device
**/
static void dma_attr_init(struct hinic_hwif *hwif)
}
/**
- * hinic_put_wqe - release the ci for new wqes
+ * hinic_rq_put_wqe - release the ci for new wqes
* @rq: recv queue
* @cons_idx: consumer index of the wqe
* @wqe_size: the size of the wqe
}
/**
- * create_txqs - Create the Logical Rx Queues of specific NIC device
+ * create_rxqs - Create the Logical Rx Queues of specific NIC device
* @nic_dev: the specific NIC device
*
* Return 0 - Success, negative - Failure
}
/**
- * free_txqs - Free the Logical Rx Queues of specific NIC device
+ * free_rxqs - Free the Logical Rx Queues of specific NIC device
* @nic_dev: the specific NIC device
**/
static void free_rxqs(struct hinic_dev *nic_dev)
}
/**
- * free_all_rx_skbs - free all skbs in tx queue
+ * free_all_tx_skbs - free all skbs in tx queue
* @txq: tx queue
**/
static void free_all_tx_skbs(struct hinic_txq *txq)
u16 dummy16 = 0;
cb0 = (void *)get_zeroed_page(GFP_KERNEL);
- if (!cb0) {
- ret = -ENOMEM;
- goto out;
- }
+ if (!cb0)
+ return -ENOMEM;
for (i = 0; i < (port->num_def_qps); i++) {
struct ehea_port_res *pr = &port->port_res[i];
cb0);
if (hret != H_SUCCESS) {
netdev_err(dev, "query_ehea_qp failed (1)\n");
+ ret = -EFAULT;
goto out;
}
&dummy64, &dummy16, &dummy16);
if (hret != H_SUCCESS) {
netdev_err(dev, "modify_ehea_qp failed (1)\n");
+ ret = -EFAULT;
goto out;
}
cb0);
if (hret != H_SUCCESS) {
netdev_err(dev, "query_ehea_qp failed (2)\n");
+ ret = -EFAULT;
goto out;
}
return ret;
}
-static ssize_t ehea_show_port_id(struct device *dev,
- struct device_attribute *attr, char *buf)
+static ssize_t log_port_id_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
{
struct ehea_port *port = container_of(dev, struct ehea_port, ofdev.dev);
return sprintf(buf, "%d", port->logical_port_id);
}
-static DEVICE_ATTR(log_port_id, 0444, ehea_show_port_id, NULL);
+static DEVICE_ATTR_RO(log_port_id);
static void logical_port_release(struct device *dev)
{
return NULL;
}
-static ssize_t ehea_probe_port(struct device *dev,
+static ssize_t probe_port_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
return (ssize_t) count;
}
-static ssize_t ehea_remove_port(struct device *dev,
- struct device_attribute *attr,
- const char *buf, size_t count)
+static ssize_t remove_port_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct ehea_adapter *adapter = dev_get_drvdata(dev);
struct ehea_port *port;
return (ssize_t) count;
}
-static DEVICE_ATTR(probe_port, 0200, NULL, ehea_probe_port);
-static DEVICE_ATTR(remove_port, 0200, NULL, ehea_remove_port);
+static DEVICE_ATTR_WO(probe_port);
+static DEVICE_ATTR_WO(remove_port);
static int ehea_create_device_sysfs(struct platform_device *dev)
{
struct {
u32 rsvd1;
u32 revid;
- u32 rsvd2[2];
+ u32 rsvd2[2];
u32 iaht1; /* Reset, R */
u32 iaht2; /* Reset, R */
u32 iaht3; /* Reset, R */
iph_proto = iph6->nexthdr;
}
- /* In OVS environment, when a flow is not cached, specifically for a
- * new TCP connection, the first packet information is passed up
+ /* When CSO is enabled the TCP checksum may have be set to NULL by
+ * the sender given that we zeroed out TCP checksum field in
+ * transmit path (refer ibmveth_start_xmit routine). In this case set
+ * up CHECKSUM_PARTIAL. If the packet is forwarded, the checksum will
+ * then be recalculated by the destination NIC (CSO must be enabled
+ * on the destination NIC).
+ *
+ * In an OVS environment, when a flow is not cached, specifically for a
+ * new TCP connection, the first packet information is passed up to
* the user space for finding a flow. During this process, OVS computes
* checksum on the first packet when CHECKSUM_PARTIAL flag is set.
*
- * Given that we zeroed out TCP checksum field in transmit path
- * (refer ibmveth_start_xmit routine) as we set "no checksum bit",
- * OVS computed checksum will be incorrect w/o TCP pseudo checksum
- * in the packet. This leads to OVS dropping the packet and hence
- * TCP retransmissions are seen.
- *
- * So, re-compute TCP pseudo header checksum.
+ * So, re-compute TCP pseudo header checksum when configured for
+ * trunk mode.
*/
- if (iph_proto == IPPROTO_TCP && adapter->is_active_trunk) {
+ if (iph_proto == IPPROTO_TCP) {
struct tcphdr *tcph = (struct tcphdr *)(skb->data + iphlen);
-
- tcphdrlen = skb->len - iphlen;
-
- /* Recompute TCP pseudo header checksum */
- if (skb_proto == ETH_P_IP)
- tcph->check = ~csum_tcpudp_magic(iph->saddr,
+ if (tcph->check == 0x0000) {
+ /* Recompute TCP pseudo header checksum */
+ if (adapter->is_active_trunk) {
+ tcphdrlen = skb->len - iphlen;
+ if (skb_proto == ETH_P_IP)
+ tcph->check =
+ ~csum_tcpudp_magic(iph->saddr,
iph->daddr, tcphdrlen, iph_proto, 0);
- else if (skb_proto == ETH_P_IPV6)
- tcph->check = ~csum_ipv6_magic(&iph6->saddr,
+ else if (skb_proto == ETH_P_IPV6)
+ tcph->check =
+ ~csum_ipv6_magic(&iph6->saddr,
&iph6->daddr, tcphdrlen, iph_proto, 0);
-
- /* Setup SKB fields for checksum offload */
- skb_partial_csum_set(skb, iphlen,
- offsetof(struct tcphdr, check));
- skb_reset_network_header(skb);
+ }
+ /* Setup SKB fields for checksum offload */
+ skb_partial_csum_set(skb, iphlen,
+ offsetof(struct tcphdr, check));
+ skb_reset_network_header(skb);
+ }
}
}
struct ibmveth_buff_pool *pool = container_of(kobj,
struct ibmveth_buff_pool,
kobj);
- struct net_device *netdev = dev_get_drvdata(
- container_of(kobj->parent, struct device, kobj));
+ struct net_device *netdev = dev_get_drvdata(kobj_to_dev(kobj->parent));
struct ibmveth_adapter *adapter = netdev_priv(netdev);
long value = simple_strtol(buf, NULL, 10);
long rc;
struct ibmvnic_sub_crq_queue *);
static int ibmvnic_poll(struct napi_struct *napi, int data);
static void send_query_map(struct ibmvnic_adapter *adapter);
-static int send_request_map(struct ibmvnic_adapter *, dma_addr_t, __be32, u8);
+static int send_request_map(struct ibmvnic_adapter *, dma_addr_t, u32, u8);
static int send_request_unmap(struct ibmvnic_adapter *, u8);
static int send_login(struct ibmvnic_adapter *adapter);
static void send_query_cap(struct ibmvnic_adapter *adapter);
{"internal_mac_rx_errors", IBMVNIC_STAT_OFF(internal_mac_rx_errors)},
};
+static int send_crq_init_complete(struct ibmvnic_adapter *adapter)
+{
+ union ibmvnic_crq crq;
+
+ memset(&crq, 0, sizeof(crq));
+ crq.generic.first = IBMVNIC_CRQ_INIT_CMD;
+ crq.generic.cmd = IBMVNIC_CRQ_INIT_COMPLETE;
+
+ return ibmvnic_send_crq(adapter, &crq);
+}
+
+static int send_version_xchg(struct ibmvnic_adapter *adapter)
+{
+ union ibmvnic_crq crq;
+
+ memset(&crq, 0, sizeof(crq));
+ crq.version_exchange.first = IBMVNIC_CRQ_CMD;
+ crq.version_exchange.cmd = VERSION_EXCHANGE;
+ crq.version_exchange.version = cpu_to_be16(ibmvnic_version);
+
+ return ibmvnic_send_crq(adapter, &crq);
+}
+
static long h_reg_sub_crq(unsigned long unit_address, unsigned long token,
unsigned long length, unsigned long *number,
unsigned long *irq)
return "REMOVING";
case VNIC_REMOVED:
return "REMOVED";
- default:
- return "UNKNOWN";
+ case VNIC_DOWN:
+ return "DOWN";
}
+ return "UNKNOWN";
}
static int ibmvnic_login(struct net_device *netdev)
/**
* build_hdr_descs_arr - build a header descriptor array
- * @txbuff: tx buffer
+ * @skb: tx socket buffer
+ * @indir_arr: indirect array
* @num_entries: number of descriptors to be sent
* @hdr_field: bit field determining which headers will be sent
*
return "TIMEOUT";
case VNIC_RESET_CHANGE_PARAM:
return "CHANGE_PARAM";
- default:
- return "UNKNOWN";
+ case VNIC_RESET_PASSIVE_INIT:
+ return "PASSIVE_INIT";
}
+ return "UNKNOWN";
}
/*
goto out;
}
- /* If the adapter was in PROBE state prior to the reset,
+ /* If the adapter was in PROBE or DOWN state prior to the reset,
* exit here.
*/
- if (reset_state == VNIC_PROBED) {
+ if (reset_state == VNIC_PROBED || reset_state == VNIC_DOWN) {
rc = 0;
goto out;
}
if (rc)
goto out;
- /* If the adapter was in PROBE state prior to the reset,
+ /* If the adapter was in PROBE or DOWN state prior to the reset,
* exit here.
*/
- if (reset_state == VNIC_PROBED)
+ if (reset_state == VNIC_PROBED || reset_state == VNIC_DOWN)
goto out;
rc = ibmvnic_login(netdev);
return rwi;
}
+/**
+ * do_passive_init - complete probing when partner device is detected.
+ * @adapter: ibmvnic_adapter struct
+ *
+ * If the ibmvnic device does not have a partner device to communicate with at boot
+ * and that partner device comes online at a later time, this function is called
+ * to complete the initialization process of ibmvnic device.
+ * Caller is expected to hold rtnl_lock().
+ *
+ * Returns non-zero if sub-CRQs are not initialized properly leaving the device
+ * in the down state.
+ * Returns 0 upon success and the device is in PROBED state.
+ */
+
+static int do_passive_init(struct ibmvnic_adapter *adapter)
+{
+ unsigned long timeout = msecs_to_jiffies(30000);
+ struct net_device *netdev = adapter->netdev;
+ struct device *dev = &adapter->vdev->dev;
+ int rc;
+
+ netdev_dbg(netdev, "Partner device found, probing.\n");
+
+ adapter->state = VNIC_PROBING;
+ reinit_completion(&adapter->init_done);
+ adapter->init_done_rc = 0;
+ adapter->crq.active = true;
+
+ rc = send_crq_init_complete(adapter);
+ if (rc)
+ goto out;
+
+ rc = send_version_xchg(adapter);
+ if (rc)
+ netdev_dbg(adapter->netdev, "send_version_xchg failed, rc=%d\n", rc);
+
+ if (!wait_for_completion_timeout(&adapter->init_done, timeout)) {
+ dev_err(dev, "Initialization sequence timed out\n");
+ rc = -ETIMEDOUT;
+ goto out;
+ }
+
+ rc = init_sub_crqs(adapter);
+ if (rc) {
+ dev_err(dev, "Initialization of sub crqs failed, rc=%d\n", rc);
+ goto out;
+ }
+
+ rc = init_sub_crq_irqs(adapter);
+ if (rc) {
+ dev_err(dev, "Failed to initialize sub crq irqs\n, rc=%d", rc);
+ goto init_failed;
+ }
+
+ netdev->mtu = adapter->req_mtu - ETH_HLEN;
+ netdev->min_mtu = adapter->min_mtu - ETH_HLEN;
+ netdev->max_mtu = adapter->max_mtu - ETH_HLEN;
+
+ adapter->state = VNIC_PROBED;
+ netdev_dbg(netdev, "Probed successfully. Waiting for signal from partner device.\n");
+
+ return 0;
+
+init_failed:
+ release_sub_crqs(adapter, 1);
+out:
+ adapter->state = VNIC_DOWN;
+ return rc;
+}
+
static void __ibmvnic_reset(struct work_struct *work)
{
struct ibmvnic_rwi *rwi;
}
spin_unlock_irqrestore(&adapter->state_lock, flags);
- if (adapter->force_reset_recovery) {
+ if (rwi->reset_reason == VNIC_RESET_PASSIVE_INIT) {
+ rtnl_lock();
+ rc = do_passive_init(adapter);
+ rtnl_unlock();
+ if (!rc)
+ netif_carrier_on(adapter->netdev);
+ } else if (adapter->force_reset_recovery) {
/* Since we are doing a hard reset now, clear the
* failover_pending flag so we don't ignore any
* future MOBILITY or other resets.
goto err;
}
- list_for_each(entry, &adapter->rwi_list) {
- tmp = list_entry(entry, struct ibmvnic_rwi, list);
+ list_for_each_entry(tmp, &adapter->rwi_list, list) {
if (tmp->reset_reason == reason) {
netdev_dbg(netdev, "Skipping matching reset, reason=%s\n",
reset_reason_to_string(reason));
return 0;
}
-static int send_version_xchg(struct ibmvnic_adapter *adapter)
-{
- union ibmvnic_crq crq;
-
- memset(&crq, 0, sizeof(crq));
- crq.version_exchange.first = IBMVNIC_CRQ_CMD;
- crq.version_exchange.cmd = VERSION_EXCHANGE;
- crq.version_exchange.version = cpu_to_be16(ibmvnic_version);
-
- return ibmvnic_send_crq(adapter, &crq);
-}
-
struct vnic_login_client_data {
u8 type;
__be16 len;
vlcd->type = 1;
len = strlen(os_name) + 1;
vlcd->len = cpu_to_be16(len);
- strncpy(vlcd->name, os_name, len);
+ strscpy(vlcd->name, os_name, len);
vlcd = (struct vnic_login_client_data *)(vlcd->name + len);
/* Type 2 - LPAR name */
vlcd->type = 2;
len = strlen(utsname()->nodename) + 1;
vlcd->len = cpu_to_be16(len);
- strncpy(vlcd->name, utsname()->nodename, len);
+ strscpy(vlcd->name, utsname()->nodename, len);
vlcd = (struct vnic_login_client_data *)(vlcd->name + len);
/* Type 3 - device name */
vlcd->type = 3;
len = strlen(adapter->netdev->name) + 1;
vlcd->len = cpu_to_be16(len);
- strncpy(vlcd->name, adapter->netdev->name, len);
+ strscpy(vlcd->name, adapter->netdev->name, len);
}
static int send_login(struct ibmvnic_adapter *adapter)
complete:
if (adapter->fw_version[0] == '\0')
- strncpy((char *)adapter->fw_version, "N/A", 3 * sizeof(char));
+ strscpy((char *)adapter->fw_version, "N/A", sizeof(adapter->fw_version));
complete(&adapter->fw_done);
}
complete(&adapter->init_done);
adapter->init_done_rc = -EIO;
}
- rc = ibmvnic_reset(adapter, VNIC_RESET_FAILOVER);
+
+ if (adapter->state == VNIC_DOWN)
+ rc = ibmvnic_reset(adapter, VNIC_RESET_PASSIVE_INIT);
+ else
+ rc = ibmvnic_reset(adapter, VNIC_RESET_FAILOVER);
+
if (rc && rc != -EBUSY) {
/* We were unable to schedule the failover
* reset either because the adapter was still
struct ibmvnic_adapter *adapter;
struct net_device *netdev;
unsigned char *mac_addr_p;
+ bool init_success;
int rc;
dev_dbg(&dev->dev, "entering ibmvnic_probe for UA 0x%x\n",
init_completion(&adapter->stats_done);
clear_bit(0, &adapter->resetting);
+ init_success = false;
do {
rc = init_crq_queue(adapter);
if (rc) {
}
rc = ibmvnic_reset_init(adapter, false);
- if (rc && rc != EAGAIN)
- goto ibmvnic_init_fail;
} while (rc == EAGAIN);
+ /* We are ignoring the error from ibmvnic_reset_init() assuming that the
+ * partner is not ready. CRQ is not active. When the partner becomes
+ * ready, we will do the passive init reset.
+ */
+
+ if (!rc)
+ init_success = true;
+
rc = init_stats_buffers(adapter);
if (rc)
goto ibmvnic_init_fail;
if (rc)
goto ibmvnic_stats_fail;
- netdev->mtu = adapter->req_mtu - ETH_HLEN;
- netdev->min_mtu = adapter->min_mtu - ETH_HLEN;
- netdev->max_mtu = adapter->max_mtu - ETH_HLEN;
-
rc = device_create_file(&dev->dev, &dev_attr_failover);
if (rc)
goto ibmvnic_dev_file_err;
}
dev_info(&dev->dev, "ibmvnic registered\n");
- adapter->state = VNIC_PROBED;
+ if (init_success) {
+ adapter->state = VNIC_PROBED;
+ netdev->mtu = adapter->req_mtu - ETH_HLEN;
+ netdev->min_mtu = adapter->min_mtu - ETH_HLEN;
+ netdev->max_mtu = adapter->max_mtu - ETH_HLEN;
+ } else {
+ adapter->state = VNIC_DOWN;
+ }
adapter->wait_for_reset = false;
adapter->last_reset_time = jiffies;
VNIC_CLOSING,
VNIC_CLOSED,
VNIC_REMOVING,
- VNIC_REMOVED};
+ VNIC_REMOVED,
+ VNIC_DOWN};
enum ibmvnic_reset_reason {VNIC_RESET_FAILOVER = 1,
VNIC_RESET_MOBILITY,
VNIC_RESET_FATAL,
VNIC_RESET_NON_FATAL,
VNIC_RESET_TIMEOUT,
- VNIC_RESET_CHANGE_PARAM};
+ VNIC_RESET_CHANGE_PARAM,
+ VNIC_RESET_PASSIVE_INIT};
struct ibmvnic_rwi {
enum ibmvnic_reset_reason reset_reason;
tristate "Intel(R) Ethernet Controller XL710 Family support"
imply PTP_1588_CLOCK
depends on PCI
+ select AUXILIARY_BUS
help
This driver supports Intel(R) Ethernet Controller XL710 Family of
devices. For more information on how to identify your adapter, go
tristate "Intel(R) Ethernet Connection E800 Series Support"
default n
depends on PCI_MSI
+ select AUXILIARY_BUS
select DIMLIB
select NET_DEVLINK
select PLDMFW
+ imply PTP_1588_CLOCK
help
This driver supports Intel(R) Ethernet Connection E800 Series of
devices. For more information on how to identify your adapter, go
u8 phy_type;
int without_mii;
- phy_type = (nic->eeprom[eeprom_phy_iface] >> 8) & 0x0f;
+ phy_type = (le16_to_cpu(nic->eeprom[eeprom_phy_iface]) >> 8) & 0x0f;
switch (phy_type) {
case NoSuchPhy: /* Non-MII PHY; UNTESTED! */
mdio_write(netdev, nic->mii.phy_id, MII_BMCR, bmcr);
} else if ((nic->mac >= mac_82550_D102) || ((nic->flags & ich) &&
(mdio_read(netdev, nic->mii.phy_id, MII_TPISTATUS) & 0x8000) &&
- (nic->eeprom[eeprom_cnfg_mdix] & eeprom_mdix_enabled))) {
+ (le16_to_cpu(nic->eeprom[eeprom_cnfg_mdix]) & eeprom_mdix_enabled))) {
/* enable/disable MDI/MDI-X auto-switching. */
mdio_write(netdev, nic->mii.phy_id, MII_NCONFIG,
nic->mii.force_media ? 0 : NCONFIG_AUTO_SWITCH);
{
/* ASF can be enabled from eeprom */
return (nic->pdev->device >= 0x1050) && (nic->pdev->device <= 0x1057) &&
- (nic->eeprom[eeprom_config_asf] & eeprom_asf) &&
- !(nic->eeprom[eeprom_config_asf] & eeprom_gcl) &&
- ((nic->eeprom[eeprom_smbus_addr] & 0xFF) != 0xFE);
+ (le16_to_cpu(nic->eeprom[eeprom_config_asf]) & eeprom_asf) &&
+ !(le16_to_cpu(nic->eeprom[eeprom_config_asf]) & eeprom_gcl) &&
+ ((le16_to_cpu(nic->eeprom[eeprom_smbus_addr]) & 0xFF) != 0xFE);
}
static int e100_up(struct nic *nic)
/* Wol magic packet can be enabled from eeprom */
if ((nic->mac >= mac_82558_D101_A4) &&
- (nic->eeprom[eeprom_id] & eeprom_id_wol)) {
+ (le16_to_cpu(nic->eeprom[eeprom_id]) & eeprom_id_wol)) {
nic->flags |= wol_magic;
device_set_wakeup_enable(&pdev->dev, true);
}
memcpy(ptr, bytes, eeprom->len);
for (i = 0; i < last_word - first_word + 1; i++)
- eeprom_buff[i] = cpu_to_le16(eeprom_buff[i]);
+ cpu_to_le16s(&eeprom_buff[i]);
ret_val = e1000_write_eeprom(hw, first_word,
last_word - first_word + 1, eeprom_buff);
* turn it on. For compatibility with a TBI link
* partner, we will store bad packets. Some
* frames have an additional byte on the end and
- * will look like CRC errors to to the hardware.
+ * will look like CRC errors to the hardware.
*/
if (!hw->tbi_compatibility_on) {
hw->tbi_compatibility_on = true;
* e1000_shift_in_mdi_bits - Shifts data bits in from the PHY
* @hw: Struct containing variables accessed by shared code
*
- * Bits are shifted in in MSB to LSB order.
+ * Bits are shifted in MSB to LSB order.
*/
static u16 e1000_shift_in_mdi_bits(struct e1000_hw *hw)
{
if (!test_and_set_bit(__E1000_DISABLED, &adapter->flags))
pci_disable_device(pdev);
- /* Request a slot slot reset. */
+ /* Request a slot reset. */
return PCI_ERS_RESULT_NEED_RESET;
}
* @hw: pointer to the HW structure
*
* ICH8 use the PCI Express bus, but does not contain a PCI Express Capability
- * register, so the the bus width is hard coded.
+ * register, so the bus width is hard coded.
**/
static s32 e1000_get_bus_info_ich8lan(struct e1000_hw *hw)
{
pci_disable_device(pdev);
- /* Request a slot slot reset. */
+ /* Request a slot reset. */
return PCI_ERS_RESULT_NEED_RESET;
}
* @data: pointer to the data to be read or written
* @read: determines if operation is read or write
*
- * Reads the PHY register at offset and stores the retreived information
+ * Reads the PHY register at offset and stores the retrieved information
* in data. Assumes semaphore already acquired. Note that the procedure
* to access these regs uses the address port and data port to read/write.
* These accesses done with PHY address 2 and without using pages.
struct fm10k_hw *hw = &interface->hw;
struct fm10k_mbx_info *mbx = &hw->mbx;
u32 eicr;
- s32 err = 0;
/* unmask any set bits related to this interrupt */
eicr = fm10k_read_reg(hw, FM10K_EICR);
/* service mailboxes */
if (fm10k_mbx_trylock(interface)) {
- err = mbx->ops.process(hw, mbx);
+ s32 err = mbx->ops.process(hw, mbx);
+
+ if (err == FM10K_ERR_RESET_REQUESTED)
+ set_bit(FM10K_FLAG_RESET_REQUESTED, interface->flags);
+
/* handle VFLRE events */
fm10k_iov_event(interface);
fm10k_mbx_unlock(interface);
}
- if (err == FM10K_ERR_RESET_REQUESTED)
- set_bit(FM10K_FLAG_RESET_REQUESTED, interface->flags);
-
/* if switch toggled state we should reset GLORTs */
if (eicr & FM10K_EICR_SWITCHNOTREADY) {
/* force link down for at least 4 seconds */
struct i40e_vsi *vsi;
};
+extern struct ida i40e_client_ida;
+
/* struct that defines an interrupt vector */
struct i40e_q_vector {
struct i40e_vsi *vsi;
static struct i40e_client *registered_client;
static LIST_HEAD(i40e_devices);
static DEFINE_MUTEX(i40e_device_mutex);
+DEFINE_IDA(i40e_client_ida);
static int i40e_client_virtchnl_send(struct i40e_info *ldev,
struct i40e_client *client,
cdev->lan_info.msix_entries = &pf->msix_entries[pf->iwarp_base_vector];
}
+static void i40e_auxiliary_dev_release(struct device *dev)
+{
+ struct i40e_auxiliary_device *i40e_aux_dev =
+ container_of(dev, struct i40e_auxiliary_device, aux_dev.dev);
+
+ ida_free(&i40e_client_ida, i40e_aux_dev->aux_dev.id);
+ kfree(i40e_aux_dev);
+}
+
+static int i40e_register_auxiliary_dev(struct i40e_info *ldev, const char *name)
+{
+ struct i40e_auxiliary_device *i40e_aux_dev;
+ struct pci_dev *pdev = ldev->pcidev;
+ struct auxiliary_device *aux_dev;
+ int ret;
+
+ i40e_aux_dev = kzalloc(sizeof(*i40e_aux_dev), GFP_KERNEL);
+ if (!i40e_aux_dev)
+ return -ENOMEM;
+
+ i40e_aux_dev->ldev = ldev;
+
+ aux_dev = &i40e_aux_dev->aux_dev;
+ aux_dev->name = name;
+ aux_dev->dev.parent = &pdev->dev;
+ aux_dev->dev.release = i40e_auxiliary_dev_release;
+ ldev->aux_dev = aux_dev;
+
+ ret = ida_alloc(&i40e_client_ida, GFP_KERNEL);
+ if (ret < 0) {
+ kfree(i40e_aux_dev);
+ return ret;
+ }
+ aux_dev->id = ret;
+
+ ret = auxiliary_device_init(aux_dev);
+ if (ret < 0) {
+ ida_free(&i40e_client_ida, aux_dev->id);
+ kfree(i40e_aux_dev);
+ return ret;
+ }
+
+ ret = auxiliary_device_add(aux_dev);
+ if (ret) {
+ auxiliary_device_uninit(aux_dev);
+ return ret;
+ }
+
+ return ret;
+}
+
/**
* i40e_client_add_instance - add a client instance struct to the instance list
* @pf: pointer to the board struct
struct netdev_hw_addr *mac = NULL;
struct i40e_vsi *vsi = pf->vsi[pf->lan_vsi];
- if (!registered_client || pf->cinst)
- return;
-
cdev = kzalloc(sizeof(*cdev), GFP_KERNEL);
if (!cdev)
return;
cdev->lan_info.fw_build = pf->hw.aq.fw_build;
set_bit(__I40E_CLIENT_INSTANCE_NONE, &cdev->state);
- if (i40e_client_get_params(vsi, &cdev->lan_info.params)) {
- kfree(cdev);
- cdev = NULL;
- return;
- }
+ if (i40e_client_get_params(vsi, &cdev->lan_info.params))
+ goto free_cdev;
mac = list_first_entry(&cdev->lan_info.netdev->dev_addrs.list,
struct netdev_hw_addr, list);
cdev->client = registered_client;
pf->cinst = cdev;
- i40e_client_update_msix_info(pf);
+ cdev->lan_info.msix_count = pf->num_iwarp_msix;
+ cdev->lan_info.msix_entries = &pf->msix_entries[pf->iwarp_base_vector];
+
+ if (i40e_register_auxiliary_dev(&cdev->lan_info, "iwarp"))
+ goto free_cdev;
+
+ return;
+
+free_cdev:
+ kfree(cdev);
+ pf->cinst = NULL;
}
/**
**/
void i40e_client_subtask(struct i40e_pf *pf)
{
- struct i40e_client *client = registered_client;
+ struct i40e_client *client;
struct i40e_client_instance *cdev;
struct i40e_vsi *vsi = pf->vsi[pf->lan_vsi];
int ret = 0;
test_bit(__I40E_CONFIG_BUSY, pf->state))
return;
- if (!client || !cdev)
+ if (!cdev || !cdev->client)
return;
+ client = cdev->client;
+
/* Here we handle client opens. If the client is down, and
* the netdev is registered, then open the client.
*/
pf->hw.pf_id, pf->hw.bus.bus_id,
pf->hw.bus.device, pf->hw.bus.func);
- /* If a client has already been registered, we need to add an instance
- * of it to our new LAN device.
- */
- if (registered_client)
- i40e_client_add_instance(pf);
+ i40e_client_add_instance(pf);
- /* Since in some cases register may have happened before a device gets
- * added, we can schedule a subtask to go initiate the clients if
- * they can be launched at probe time.
- */
set_bit(__I40E_CLIENT_SERVICE_REQUESTED, pf->state);
i40e_service_event_schedule(pf);
**/
int i40e_lan_del_device(struct i40e_pf *pf)
{
+ struct auxiliary_device *aux_dev = pf->cinst->lan_info.aux_dev;
struct i40e_device *ldev, *tmp;
int ret = -ENODEV;
+ auxiliary_device_delete(aux_dev);
+ auxiliary_device_uninit(aux_dev);
+
/* First, remove any client instance. */
i40e_client_del_instance(pf);
u32 v_idx, i, reg_idx, reg;
ldev->qvlist_info = kzalloc(struct_size(ldev->qvlist_info, qv_info,
- qvlist_info->num_vectors - 1), GFP_KERNEL);
+ qvlist_info->num_vectors), GFP_KERNEL);
if (!ldev->qvlist_info)
return -ENOMEM;
ldev->qvlist_info->num_vectors = qvlist_info->num_vectors;
return err;
}
+void i40e_client_device_register(struct i40e_info *ldev, struct i40e_client *client)
+{
+ struct i40e_pf *pf = ldev->pf;
+
+ pf->cinst->client = client;
+ set_bit(__I40E_CLIENT_SERVICE_REQUESTED, pf->state);
+ i40e_service_event_schedule(pf);
+}
+EXPORT_SYMBOL_GPL(i40e_client_device_register);
+
+void i40e_client_device_unregister(struct i40e_info *ldev)
+{
+ struct i40e_pf *pf = ldev->pf;
+ struct i40e_client_instance *cdev = pf->cinst;
+
+ if (!cdev)
+ return;
+
+ while (test_and_set_bit(__I40E_SERVICE_SCHED, pf->state))
+ usleep_range(500, 1000);
+
+ if (test_bit(__I40E_CLIENT_INSTANCE_OPENED, &cdev->state)) {
+ cdev->client->ops->close(&cdev->lan_info, cdev->client, false);
+ clear_bit(__I40E_CLIENT_INSTANCE_OPENED, &cdev->state);
+ i40e_client_release_qvlist(&cdev->lan_info);
+ }
+
+ pf->cinst->client = NULL;
+ clear_bit(__I40E_SERVICE_SCHED, pf->state);
+}
+EXPORT_SYMBOL_GPL(i40e_client_device_unregister);
+
+/* Retain these legacy global registration/unregistration calls till i40iw is
+ * removed from the kernel. The irdma unified driver does not use these
+ * exported symbols.
+ */
/**
* i40e_register_client - Register a i40e client driver with the L2 driver
* @client: pointer to the i40e_client struct
* ENDIF
*/
-/* macro to make the table lines short */
+/* macro to make the table lines short, use explicit indexing with [PTYPE] */
#define I40E_PTT(PTYPE, OUTER_IP, OUTER_IP_VER, OUTER_FRAG, T, TE, TEF, I, PL)\
- { PTYPE, \
+ [PTYPE] = { \
1, \
I40E_RX_PTYPE_OUTER_##OUTER_IP, \
I40E_RX_PTYPE_OUTER_##OUTER_IP_VER, \
I40E_RX_PTYPE_INNER_PROT_##I, \
I40E_RX_PTYPE_PAYLOAD_LAYER_##PL }
-#define I40E_PTT_UNUSED_ENTRY(PTYPE) \
- { PTYPE, 0, 0, 0, 0, 0, 0, 0, 0, 0 }
+#define I40E_PTT_UNUSED_ENTRY(PTYPE) [PTYPE] = { 0, 0, 0, 0, 0, 0, 0, 0, 0 }
/* shorter macros makes the table fit but are terse */
#define I40E_RX_PTYPE_NOF I40E_RX_PTYPE_NOT_FRAG
#define I40E_RX_PTYPE_FRG I40E_RX_PTYPE_FRAG
#define I40E_RX_PTYPE_INNER_PROT_TS I40E_RX_PTYPE_INNER_PROT_TIMESYNC
-/* Lookup table mapping the HW PTYPE to the bit field for decoding */
-struct i40e_rx_ptype_decoded i40e_ptype_lookup[] = {
+/* Lookup table mapping in the 8-bit HW PTYPE to the bit field for decoding */
+struct i40e_rx_ptype_decoded i40e_ptype_lookup[BIT(8)] = {
/* L2 Packet types */
I40E_PTT_UNUSED_ENTRY(0),
I40E_PTT(1, L2, NONE, NOF, NONE, NONE, NOF, NONE, PAY2),
I40E_PTT(153, IP, IPV6, NOF, IP_GRENAT_MAC_VLAN, IPV6, NOF, ICMP, PAY4),
/* unused entries */
- I40E_PTT_UNUSED_ENTRY(154),
- I40E_PTT_UNUSED_ENTRY(155),
- I40E_PTT_UNUSED_ENTRY(156),
- I40E_PTT_UNUSED_ENTRY(157),
- I40E_PTT_UNUSED_ENTRY(158),
- I40E_PTT_UNUSED_ENTRY(159),
-
- I40E_PTT_UNUSED_ENTRY(160),
- I40E_PTT_UNUSED_ENTRY(161),
- I40E_PTT_UNUSED_ENTRY(162),
- I40E_PTT_UNUSED_ENTRY(163),
- I40E_PTT_UNUSED_ENTRY(164),
- I40E_PTT_UNUSED_ENTRY(165),
- I40E_PTT_UNUSED_ENTRY(166),
- I40E_PTT_UNUSED_ENTRY(167),
- I40E_PTT_UNUSED_ENTRY(168),
- I40E_PTT_UNUSED_ENTRY(169),
-
- I40E_PTT_UNUSED_ENTRY(170),
- I40E_PTT_UNUSED_ENTRY(171),
- I40E_PTT_UNUSED_ENTRY(172),
- I40E_PTT_UNUSED_ENTRY(173),
- I40E_PTT_UNUSED_ENTRY(174),
- I40E_PTT_UNUSED_ENTRY(175),
- I40E_PTT_UNUSED_ENTRY(176),
- I40E_PTT_UNUSED_ENTRY(177),
- I40E_PTT_UNUSED_ENTRY(178),
- I40E_PTT_UNUSED_ENTRY(179),
-
- I40E_PTT_UNUSED_ENTRY(180),
- I40E_PTT_UNUSED_ENTRY(181),
- I40E_PTT_UNUSED_ENTRY(182),
- I40E_PTT_UNUSED_ENTRY(183),
- I40E_PTT_UNUSED_ENTRY(184),
- I40E_PTT_UNUSED_ENTRY(185),
- I40E_PTT_UNUSED_ENTRY(186),
- I40E_PTT_UNUSED_ENTRY(187),
- I40E_PTT_UNUSED_ENTRY(188),
- I40E_PTT_UNUSED_ENTRY(189),
-
- I40E_PTT_UNUSED_ENTRY(190),
- I40E_PTT_UNUSED_ENTRY(191),
- I40E_PTT_UNUSED_ENTRY(192),
- I40E_PTT_UNUSED_ENTRY(193),
- I40E_PTT_UNUSED_ENTRY(194),
- I40E_PTT_UNUSED_ENTRY(195),
- I40E_PTT_UNUSED_ENTRY(196),
- I40E_PTT_UNUSED_ENTRY(197),
- I40E_PTT_UNUSED_ENTRY(198),
- I40E_PTT_UNUSED_ENTRY(199),
-
- I40E_PTT_UNUSED_ENTRY(200),
- I40E_PTT_UNUSED_ENTRY(201),
- I40E_PTT_UNUSED_ENTRY(202),
- I40E_PTT_UNUSED_ENTRY(203),
- I40E_PTT_UNUSED_ENTRY(204),
- I40E_PTT_UNUSED_ENTRY(205),
- I40E_PTT_UNUSED_ENTRY(206),
- I40E_PTT_UNUSED_ENTRY(207),
- I40E_PTT_UNUSED_ENTRY(208),
- I40E_PTT_UNUSED_ENTRY(209),
-
- I40E_PTT_UNUSED_ENTRY(210),
- I40E_PTT_UNUSED_ENTRY(211),
- I40E_PTT_UNUSED_ENTRY(212),
- I40E_PTT_UNUSED_ENTRY(213),
- I40E_PTT_UNUSED_ENTRY(214),
- I40E_PTT_UNUSED_ENTRY(215),
- I40E_PTT_UNUSED_ENTRY(216),
- I40E_PTT_UNUSED_ENTRY(217),
- I40E_PTT_UNUSED_ENTRY(218),
- I40E_PTT_UNUSED_ENTRY(219),
-
- I40E_PTT_UNUSED_ENTRY(220),
- I40E_PTT_UNUSED_ENTRY(221),
- I40E_PTT_UNUSED_ENTRY(222),
- I40E_PTT_UNUSED_ENTRY(223),
- I40E_PTT_UNUSED_ENTRY(224),
- I40E_PTT_UNUSED_ENTRY(225),
- I40E_PTT_UNUSED_ENTRY(226),
- I40E_PTT_UNUSED_ENTRY(227),
- I40E_PTT_UNUSED_ENTRY(228),
- I40E_PTT_UNUSED_ENTRY(229),
-
- I40E_PTT_UNUSED_ENTRY(230),
- I40E_PTT_UNUSED_ENTRY(231),
- I40E_PTT_UNUSED_ENTRY(232),
- I40E_PTT_UNUSED_ENTRY(233),
- I40E_PTT_UNUSED_ENTRY(234),
- I40E_PTT_UNUSED_ENTRY(235),
- I40E_PTT_UNUSED_ENTRY(236),
- I40E_PTT_UNUSED_ENTRY(237),
- I40E_PTT_UNUSED_ENTRY(238),
- I40E_PTT_UNUSED_ENTRY(239),
-
- I40E_PTT_UNUSED_ENTRY(240),
- I40E_PTT_UNUSED_ENTRY(241),
- I40E_PTT_UNUSED_ENTRY(242),
- I40E_PTT_UNUSED_ENTRY(243),
- I40E_PTT_UNUSED_ENTRY(244),
- I40E_PTT_UNUSED_ENTRY(245),
- I40E_PTT_UNUSED_ENTRY(246),
- I40E_PTT_UNUSED_ENTRY(247),
- I40E_PTT_UNUSED_ENTRY(248),
- I40E_PTT_UNUSED_ENTRY(249),
-
- I40E_PTT_UNUSED_ENTRY(250),
- I40E_PTT_UNUSED_ENTRY(251),
- I40E_PTT_UNUSED_ENTRY(252),
- I40E_PTT_UNUSED_ENTRY(253),
- I40E_PTT_UNUSED_ENTRY(254),
- I40E_PTT_UNUSED_ENTRY(255)
+ [154 ... 255] = { 0, 0, 0, 0, 0, 0, 0, 0, 0 }
};
/**
{
pci_unregister_driver(&i40e_driver);
destroy_workqueue(i40e_wq);
+ ida_destroy(&i40e_client_ida);
i40e_dbg_exit();
}
module_exit(i40e_exit_module);
case XDP_TX:
xdp_ring = rx_ring->vsi->xdp_rings[rx_ring->queue_index];
result = i40e_xmit_xdp_tx_ring(xdp, xdp_ring);
+ if (result == I40E_XDP_CONSUMED)
+ goto out_failure;
break;
case XDP_REDIRECT:
err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
- result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED;
+ if (err)
+ goto out_failure;
+ result = I40E_XDP_REDIR;
break;
default:
bpf_warn_invalid_xdp_action(act);
fallthrough;
case XDP_ABORTED:
+out_failure:
trace_xdp_exception(rx_ring->netdev, xdp_prog, act);
fallthrough; /* handle aborts by dropping packet */
case XDP_DROP:
};
struct i40e_rx_ptype_decoded {
- u32 ptype:8;
u32 known:1;
u32 outer_ip:1;
u32 outer_ip_ver:1;
if (likely(act == XDP_REDIRECT)) {
err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
- result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED;
+ if (err)
+ goto out_failure;
rcu_read_unlock();
- return result;
+ return I40E_XDP_REDIR;
}
switch (act) {
case XDP_TX:
xdp_ring = rx_ring->vsi->xdp_rings[rx_ring->queue_index];
result = i40e_xmit_xdp_tx_ring(xdp, xdp_ring);
+ if (result == I40E_XDP_CONSUMED)
+ goto out_failure;
break;
default:
bpf_warn_invalid_xdp_action(act);
fallthrough;
case XDP_ABORTED:
+out_failure:
trace_xdp_exception(rx_ring->netdev, xdp_prog, act);
fallthrough; /* handle aborts by dropping packet */
case XDP_DROP:
* ENDIF
*/
-/* macro to make the table lines short */
+/* macro to make the table lines short, use explicit indexing with [PTYPE] */
#define IAVF_PTT(PTYPE, OUTER_IP, OUTER_IP_VER, OUTER_FRAG, T, TE, TEF, I, PL)\
- { PTYPE, \
+ [PTYPE] = { \
1, \
IAVF_RX_PTYPE_OUTER_##OUTER_IP, \
IAVF_RX_PTYPE_OUTER_##OUTER_IP_VER, \
IAVF_RX_PTYPE_INNER_PROT_##I, \
IAVF_RX_PTYPE_PAYLOAD_LAYER_##PL }
-#define IAVF_PTT_UNUSED_ENTRY(PTYPE) \
- { PTYPE, 0, 0, 0, 0, 0, 0, 0, 0, 0 }
+#define IAVF_PTT_UNUSED_ENTRY(PTYPE) [PTYPE] = { 0, 0, 0, 0, 0, 0, 0, 0, 0 }
/* shorter macros makes the table fit but are terse */
#define IAVF_RX_PTYPE_NOF IAVF_RX_PTYPE_NOT_FRAG
#define IAVF_RX_PTYPE_FRG IAVF_RX_PTYPE_FRAG
#define IAVF_RX_PTYPE_INNER_PROT_TS IAVF_RX_PTYPE_INNER_PROT_TIMESYNC
-/* Lookup table mapping the HW PTYPE to the bit field for decoding */
-struct iavf_rx_ptype_decoded iavf_ptype_lookup[] = {
+/* Lookup table mapping the 8-bit HW PTYPE to the bit field for decoding */
+struct iavf_rx_ptype_decoded iavf_ptype_lookup[BIT(8)] = {
/* L2 Packet types */
IAVF_PTT_UNUSED_ENTRY(0),
IAVF_PTT(1, L2, NONE, NOF, NONE, NONE, NOF, NONE, PAY2),
IAVF_PTT(153, IP, IPV6, NOF, IP_GRENAT_MAC_VLAN, IPV6, NOF, ICMP, PAY4),
/* unused entries */
- IAVF_PTT_UNUSED_ENTRY(154),
- IAVF_PTT_UNUSED_ENTRY(155),
- IAVF_PTT_UNUSED_ENTRY(156),
- IAVF_PTT_UNUSED_ENTRY(157),
- IAVF_PTT_UNUSED_ENTRY(158),
- IAVF_PTT_UNUSED_ENTRY(159),
-
- IAVF_PTT_UNUSED_ENTRY(160),
- IAVF_PTT_UNUSED_ENTRY(161),
- IAVF_PTT_UNUSED_ENTRY(162),
- IAVF_PTT_UNUSED_ENTRY(163),
- IAVF_PTT_UNUSED_ENTRY(164),
- IAVF_PTT_UNUSED_ENTRY(165),
- IAVF_PTT_UNUSED_ENTRY(166),
- IAVF_PTT_UNUSED_ENTRY(167),
- IAVF_PTT_UNUSED_ENTRY(168),
- IAVF_PTT_UNUSED_ENTRY(169),
-
- IAVF_PTT_UNUSED_ENTRY(170),
- IAVF_PTT_UNUSED_ENTRY(171),
- IAVF_PTT_UNUSED_ENTRY(172),
- IAVF_PTT_UNUSED_ENTRY(173),
- IAVF_PTT_UNUSED_ENTRY(174),
- IAVF_PTT_UNUSED_ENTRY(175),
- IAVF_PTT_UNUSED_ENTRY(176),
- IAVF_PTT_UNUSED_ENTRY(177),
- IAVF_PTT_UNUSED_ENTRY(178),
- IAVF_PTT_UNUSED_ENTRY(179),
-
- IAVF_PTT_UNUSED_ENTRY(180),
- IAVF_PTT_UNUSED_ENTRY(181),
- IAVF_PTT_UNUSED_ENTRY(182),
- IAVF_PTT_UNUSED_ENTRY(183),
- IAVF_PTT_UNUSED_ENTRY(184),
- IAVF_PTT_UNUSED_ENTRY(185),
- IAVF_PTT_UNUSED_ENTRY(186),
- IAVF_PTT_UNUSED_ENTRY(187),
- IAVF_PTT_UNUSED_ENTRY(188),
- IAVF_PTT_UNUSED_ENTRY(189),
-
- IAVF_PTT_UNUSED_ENTRY(190),
- IAVF_PTT_UNUSED_ENTRY(191),
- IAVF_PTT_UNUSED_ENTRY(192),
- IAVF_PTT_UNUSED_ENTRY(193),
- IAVF_PTT_UNUSED_ENTRY(194),
- IAVF_PTT_UNUSED_ENTRY(195),
- IAVF_PTT_UNUSED_ENTRY(196),
- IAVF_PTT_UNUSED_ENTRY(197),
- IAVF_PTT_UNUSED_ENTRY(198),
- IAVF_PTT_UNUSED_ENTRY(199),
-
- IAVF_PTT_UNUSED_ENTRY(200),
- IAVF_PTT_UNUSED_ENTRY(201),
- IAVF_PTT_UNUSED_ENTRY(202),
- IAVF_PTT_UNUSED_ENTRY(203),
- IAVF_PTT_UNUSED_ENTRY(204),
- IAVF_PTT_UNUSED_ENTRY(205),
- IAVF_PTT_UNUSED_ENTRY(206),
- IAVF_PTT_UNUSED_ENTRY(207),
- IAVF_PTT_UNUSED_ENTRY(208),
- IAVF_PTT_UNUSED_ENTRY(209),
-
- IAVF_PTT_UNUSED_ENTRY(210),
- IAVF_PTT_UNUSED_ENTRY(211),
- IAVF_PTT_UNUSED_ENTRY(212),
- IAVF_PTT_UNUSED_ENTRY(213),
- IAVF_PTT_UNUSED_ENTRY(214),
- IAVF_PTT_UNUSED_ENTRY(215),
- IAVF_PTT_UNUSED_ENTRY(216),
- IAVF_PTT_UNUSED_ENTRY(217),
- IAVF_PTT_UNUSED_ENTRY(218),
- IAVF_PTT_UNUSED_ENTRY(219),
-
- IAVF_PTT_UNUSED_ENTRY(220),
- IAVF_PTT_UNUSED_ENTRY(221),
- IAVF_PTT_UNUSED_ENTRY(222),
- IAVF_PTT_UNUSED_ENTRY(223),
- IAVF_PTT_UNUSED_ENTRY(224),
- IAVF_PTT_UNUSED_ENTRY(225),
- IAVF_PTT_UNUSED_ENTRY(226),
- IAVF_PTT_UNUSED_ENTRY(227),
- IAVF_PTT_UNUSED_ENTRY(228),
- IAVF_PTT_UNUSED_ENTRY(229),
-
- IAVF_PTT_UNUSED_ENTRY(230),
- IAVF_PTT_UNUSED_ENTRY(231),
- IAVF_PTT_UNUSED_ENTRY(232),
- IAVF_PTT_UNUSED_ENTRY(233),
- IAVF_PTT_UNUSED_ENTRY(234),
- IAVF_PTT_UNUSED_ENTRY(235),
- IAVF_PTT_UNUSED_ENTRY(236),
- IAVF_PTT_UNUSED_ENTRY(237),
- IAVF_PTT_UNUSED_ENTRY(238),
- IAVF_PTT_UNUSED_ENTRY(239),
-
- IAVF_PTT_UNUSED_ENTRY(240),
- IAVF_PTT_UNUSED_ENTRY(241),
- IAVF_PTT_UNUSED_ENTRY(242),
- IAVF_PTT_UNUSED_ENTRY(243),
- IAVF_PTT_UNUSED_ENTRY(244),
- IAVF_PTT_UNUSED_ENTRY(245),
- IAVF_PTT_UNUSED_ENTRY(246),
- IAVF_PTT_UNUSED_ENTRY(247),
- IAVF_PTT_UNUSED_ENTRY(248),
- IAVF_PTT_UNUSED_ENTRY(249),
-
- IAVF_PTT_UNUSED_ENTRY(250),
- IAVF_PTT_UNUSED_ENTRY(251),
- IAVF_PTT_UNUSED_ENTRY(252),
- IAVF_PTT_UNUSED_ENTRY(253),
- IAVF_PTT_UNUSED_ENTRY(254),
- IAVF_PTT_UNUSED_ENTRY(255)
+ [154 ... 255] = { 0, 0, 0, 0, 0, 0, 0, 0, 0 }
};
/**
};
struct iavf_rx_ptype_decoded {
- u32 ptype:8;
u32 known:1;
u32 outer_ip:1;
u32 outer_ip_ver:1;
ice_ethtool_fdir.o \
ice_flex_pipe.o \
ice_flow.o \
+ ice_idc.o \
ice_devlink.o \
ice_fw_update.o \
ice_lag.o \
ice_ethtool.o
ice-$(CONFIG_PCI_IOV) += ice_virtchnl_allowlist.o
ice-$(CONFIG_PCI_IOV) += ice_virtchnl_pf.o ice_sriov.o ice_virtchnl_fdir.o
+ice-$(CONFIG_PTP_1588_CLOCK) += ice_ptp.o ice_ptp_hw.o
ice-$(CONFIG_DCB) += ice_dcb.o ice_dcb_nl.o ice_dcb_lib.o
ice-$(CONFIG_RFS_ACCEL) += ice_arfs.o
ice-$(CONFIG_XDP_SOCKETS) += ice_xsk.o
#include <linux/if_bridge.h>
#include <linux/ctype.h>
#include <linux/bpf.h>
+#include <linux/auxiliary_bus.h>
#include <linux/avf/virtchnl.h>
#include <linux/cpu_rmap.h>
#include <linux/dim.h>
#include "ice_switch.h"
#include "ice_common.h"
#include "ice_sched.h"
+#include "ice_idc_int.h"
#include "ice_virtchnl_pf.h"
#include "ice_sriov.h"
+#include "ice_ptp.h"
#include "ice_fdir.h"
#include "ice_xsk.h"
#include "ice_arfs.h"
#define ICE_DFLT_TRAFFIC_CLASS BIT(0)
#define ICE_INT_NAME_STR_LEN (IFNAMSIZ + 16)
-#define ICE_AQ_LEN 64
+#define ICE_AQ_LEN 192
#define ICE_MBXSQ_LEN 64
+#define ICE_SBQ_LEN 64
#define ICE_MIN_LAN_TXRX_MSIX 1
#define ICE_MIN_LAN_OICR_MSIX 1
#define ICE_MIN_MSIX (ICE_MIN_LAN_TXRX_MSIX + ICE_MIN_LAN_OICR_MSIX)
#define ICE_FDIR_MSIX 2
+#define ICE_RDMA_NUM_AEQ_MSIX 4
+#define ICE_MIN_RDMA_MSIX 2
#define ICE_NO_VSI 0xffff
#define ICE_VSI_MAP_CONTIG 0
#define ICE_VSI_MAP_SCATTER 1
#define ICE_MAX_LG_RSS_QS 256
#define ICE_RES_VALID_BIT 0x8000
#define ICE_RES_MISC_VEC_ID (ICE_RES_VALID_BIT - 1)
+#define ICE_RES_RDMA_VEC_ID (ICE_RES_MISC_VEC_ID - 1)
/* All VF control VSIs share the same IRQ, so assign a unique ID for them */
-#define ICE_RES_VF_CTRL_VEC_ID (ICE_RES_MISC_VEC_ID - 1)
+#define ICE_RES_VF_CTRL_VEC_ID (ICE_RES_RDMA_VEC_ID - 1)
#define ICE_INVAL_Q_INDEX 0xffff
#define ICE_INVAL_VFID 256
ICE_NEEDS_RESTART,
ICE_PREPARED_FOR_RESET, /* set by driver when prepared */
ICE_RESET_OICR_RECV, /* set by driver after rcv reset OICR */
- ICE_PFR_REQ, /* set by driver and peers */
- ICE_CORER_REQ, /* set by driver and peers */
- ICE_GLOBR_REQ, /* set by driver and peers */
+ ICE_PFR_REQ, /* set by driver */
+ ICE_CORER_REQ, /* set by driver */
+ ICE_GLOBR_REQ, /* set by driver */
ICE_CORER_RECV, /* set by OICR handler */
ICE_GLOBR_RECV, /* set by OICR handler */
ICE_EMPR_RECV, /* set by OICR handler */
ICE_STATE_NOMINAL_CHECK_BITS,
ICE_ADMINQ_EVENT_PENDING,
ICE_MAILBOXQ_EVENT_PENDING,
+ ICE_SIDEBANDQ_EVENT_PENDING,
ICE_MDD_EVENT_PENDING,
ICE_VFLR_EVENT_PENDING,
ICE_FLTR_OVERFLOW_PROMISC,
u16 req_rxq; /* User requested Rx queues */
u16 num_rx_desc;
u16 num_tx_desc;
+ u16 qset_handle[ICE_MAX_TRAFFIC_CLASS];
struct ice_tc_cfg tc_cfg;
struct bpf_prog *xdp_prog;
struct ice_ring **xdp_rings; /* XDP ring array */
+ unsigned long *af_xdp_zc_qps; /* tracks AF_XDP ZC enabled qps */
u16 num_xdp_txq; /* Used XDP queues */
u8 xdp_mapping_mode; /* ICE_MAP_MODE_[CONTIG|SCATTER] */
enum ice_pf_flags {
ICE_FLAG_FLTR_SYNC,
+ ICE_FLAG_RDMA_ENA,
ICE_FLAG_RSS_ENA,
ICE_FLAG_SRIOV_ENA,
ICE_FLAG_SRIOV_CAPABLE,
ICE_FLAG_DCB_CAPABLE,
ICE_FLAG_DCB_ENA,
ICE_FLAG_FD_ENA,
+ ICE_FLAG_PTP_SUPPORTED, /* PTP is supported by NVM */
+ ICE_FLAG_PTP, /* PTP is enabled by software */
+ ICE_FLAG_AUX_ENA,
ICE_FLAG_ADV_FEATURES,
ICE_FLAG_LINK_DOWN_ON_CLOSE_ENA,
ICE_FLAG_TOTAL_PORT_SHUTDOWN_ENA,
ICE_FLAG_NO_MEDIA,
ICE_FLAG_FW_LLDP_AGENT,
+ ICE_FLAG_MOD_POWER_UNSUPPORTED,
ICE_FLAG_ETHTOOL_CTXT, /* set when ethtool holds RTNL lock */
ICE_FLAG_LEGACY_RX,
ICE_FLAG_VF_TRUE_PROMISC_ENA,
struct mutex sw_mutex; /* lock for protecting VSI alloc flow */
struct mutex tc_mutex; /* lock to protect TC changes */
u32 msg_enable;
+ struct ice_ptp ptp;
+ u16 num_rdma_msix; /* Total MSIX vectors for RDMA driver */
+ u16 rdma_base_vector;
/* spinlock to protect the AdminQ wait list */
spinlock_t aq_wait_lock;
struct hlist_head aq_wait_list;
wait_queue_head_t aq_wait_queue;
+ wait_queue_head_t reset_wait_queue;
+
u32 hw_csum_rx_error;
u16 oicr_idx; /* Other interrupt cause MSIX vector index */
u16 num_avail_sw_msix; /* remaining MSIX SW vectors left unclaimed */
unsigned long tx_timeout_last_recovery;
u32 tx_timeout_recovery_level;
char int_name[ICE_INT_NAME_STR_LEN];
+ struct auxiliary_device *adev;
+ int aux_idx;
u32 sw_int_count;
__le64 nvm_phy_type_lo; /* NVM PHY type low */
*/
static inline struct xsk_buff_pool *ice_xsk_pool(struct ice_ring *ring)
{
+ struct ice_vsi *vsi = ring->vsi;
u16 qid = ring->q_index;
if (ice_ring_is_xdp(ring))
- qid -= ring->vsi->num_xdp_txq;
+ qid -= vsi->num_xdp_txq;
- if (!ice_is_xdp_ena_vsi(ring->vsi))
+ if (!ice_is_xdp_ena_vsi(vsi) || !test_bit(qid, vsi->af_xdp_zc_qps))
return NULL;
- return xsk_get_pool_from_qid(ring->vsi->netdev, qid);
+ return xsk_get_pool_from_qid(vsi->netdev, qid);
}
/**
void ice_fill_rss_lut(u8 *lut, u16 rss_table_size, u16 rss_size);
int ice_schedule_reset(struct ice_pf *pf, enum ice_reset_req reset);
void ice_print_link_msg(struct ice_vsi *vsi, bool isup);
+int ice_plug_aux_dev(struct ice_pf *pf);
+void ice_unplug_aux_dev(struct ice_pf *pf);
+int ice_init_rdma(struct ice_pf *pf);
const char *ice_stat_str(enum ice_status stat_err);
const char *ice_aq_str(enum ice_aq_err aq_err);
bool ice_is_wol_supported(struct ice_hw *hw);
int ice_stop(struct net_device *netdev);
void ice_service_task_schedule(struct ice_pf *pf);
+/**
+ * ice_set_rdma_cap - enable RDMA support
+ * @pf: PF struct
+ */
+static inline void ice_set_rdma_cap(struct ice_pf *pf)
+{
+ if (pf->hw.func_caps.common_cap.rdma && pf->num_rdma_msix) {
+ set_bit(ICE_FLAG_RDMA_ENA, pf->flags);
+ ice_plug_aux_dev(pf);
+ }
+}
+
+/**
+ * ice_clear_rdma_cap - disable RDMA support
+ * @pf: PF struct
+ */
+static inline void ice_clear_rdma_cap(struct ice_pf *pf)
+{
+ ice_unplug_aux_dev(pf);
+ clear_bit(ICE_FLAG_RDMA_ENA, pf->flags);
+}
#endif /* _ICE_H_ */
#define ICE_AQC_CAPS_TXQS 0x0042
#define ICE_AQC_CAPS_MSIX 0x0043
#define ICE_AQC_CAPS_FD 0x0045
+#define ICE_AQC_CAPS_1588 0x0046
#define ICE_AQC_CAPS_MAX_MTU 0x0047
#define ICE_AQC_CAPS_NVM_VER 0x0048
#define ICE_AQC_CAPS_PENDING_NVM_VER 0x0049
#define ICE_AQC_CAPS_PENDING_OROM_VER 0x004B
#define ICE_AQC_CAPS_NET_VER 0x004C
#define ICE_AQC_CAPS_PENDING_NET_VER 0x004D
+#define ICE_AQC_CAPS_RDMA 0x0051
#define ICE_AQC_CAPS_NVM_MGMT 0x0080
u8 major_ver;
#define ICE_AQ_LINK_TOPO_UNDRUTIL_PRT BIT(5)
#define ICE_AQ_LINK_TOPO_UNDRUTIL_MEDIA BIT(6)
#define ICE_AQ_LINK_TOPO_UNSUPP_MEDIA BIT(7)
- u8 reserved1;
+ u8 link_cfg_err;
+#define ICE_AQ_LINK_MODULE_POWER_UNSUPPORTED BIT(5)
+#define ICE_AQ_LINK_INVAL_MAX_POWER_LIMIT BIT(7)
u8 link_info;
#define ICE_AQ_LINK_UP BIT(0) /* Link Status */
#define ICE_AQ_LINK_FAULT BIT(1)
#define ICE_AQ_CFG_PACING_TYPE_FIXED ICE_AQ_CFG_PACING_TYPE_M
/* External Device Power Ability */
u8 power_desc;
-#define ICE_AQ_PWR_CLASS_M 0x3
+#define ICE_AQ_PWR_CLASS_M 0x3F
#define ICE_AQ_LINK_PWR_BASET_LOW_HIGH 0
#define ICE_AQ_LINK_PWR_BASET_HIGH 1
#define ICE_AQ_LINK_PWR_QSFP_CLASS_1 0
__le32 addr_low;
};
+/* Sideband Control Interface Commands */
+/* Neighbor Device Request (indirect 0x0C00); also used for the response. */
+struct ice_aqc_neigh_dev_req {
+ __le16 sb_data_len;
+ u8 reserved[6];
+ __le32 addr_high;
+ __le32 addr_low;
+};
+
/* Add Tx LAN Queues (indirect 0x0C30) */
struct ice_aqc_add_txqs {
u8 num_qgrps;
__le16 q_id[];
} __packed;
+/* Add Tx RDMA Queue Set (indirect 0x0C33) */
+struct ice_aqc_add_rdma_qset {
+ u8 num_qset_grps;
+ u8 reserved[7];
+ __le32 addr_high;
+ __le32 addr_low;
+};
+
+/* This is the descriptor of each Qset entry for the Add Tx RDMA Queue Set
+ * command (0x0C33). Only used within struct ice_aqc_add_rdma_qset.
+ */
+struct ice_aqc_add_tx_rdma_qset_entry {
+ __le16 tx_qset_id;
+ u8 rsvd[2];
+ __le32 qset_teid;
+ struct ice_aqc_txsched_elem info;
+};
+
+/* The format of the command buffer for Add Tx RDMA Queue Set(0x0C33)
+ * is an array of the following structs. Please note that the length of
+ * each struct ice_aqc_add_rdma_qset is variable due to the variable
+ * number of queues in each group!
+ */
+struct ice_aqc_add_rdma_qset_data {
+ __le32 parent_teid;
+ __le16 num_qsets;
+ u8 rsvd[2];
+ struct ice_aqc_add_tx_rdma_qset_entry rdma_qsets[];
+};
+
/* Configure Firmware Logging Command (indirect 0xFF09)
* Logging Information Read Response (indirect 0xFF10)
* Note: The 0xFF10 command has no input parameters.
struct ice_aqc_get_pkg_info pkg_info[];
};
+/* Driver Shared Parameters (direct, 0x0C90) */
+struct ice_aqc_driver_shared_params {
+ u8 set_or_get_op;
+#define ICE_AQC_DRIVER_PARAM_OP_MASK BIT(0)
+#define ICE_AQC_DRIVER_PARAM_SET 0
+#define ICE_AQC_DRIVER_PARAM_GET 1
+ u8 param_indx;
+#define ICE_AQC_DRIVER_PARAM_MAX_IDX 15
+ u8 rsvd[2];
+ __le32 param_val;
+ __le32 addr_high;
+ __le32 addr_low;
+};
+
+enum ice_aqc_driver_params {
+ /* OS clock index for PTP timer Domain 0 */
+ ICE_AQC_DRIVER_PARAM_CLK_IDX_TMR0 = 0,
+ /* OS clock index for PTP timer Domain 1 */
+ ICE_AQC_DRIVER_PARAM_CLK_IDX_TMR1,
+
+ /* Add new parameters above */
+ ICE_AQC_DRIVER_PARAM_MAX = 16,
+};
+
/* Lan Queue Overflow Event (direct, 0x1001) */
struct ice_aqc_event_lan_overflow {
__le32 prtdcb_ruptq;
struct ice_aqc_lldp_filter_ctrl lldp_filter_ctrl;
struct ice_aqc_get_set_rss_lut get_set_rss_lut;
struct ice_aqc_get_set_rss_key get_set_rss_key;
+ struct ice_aqc_neigh_dev_req neigh_dev;
struct ice_aqc_add_txqs add_txqs;
struct ice_aqc_dis_txqs dis_txqs;
+ struct ice_aqc_add_rdma_qset add_rdma_qset;
struct ice_aqc_add_get_update_free_vsi vsi_cmd;
struct ice_aqc_add_update_free_vsi_resp add_update_free_vsi_res;
struct ice_aqc_fw_logging fw_logging;
struct ice_aqc_get_clear_fw_log get_clear_fw_log;
struct ice_aqc_download_pkg download_pkg;
+ struct ice_aqc_driver_shared_params drv_shared_params;
struct ice_aqc_set_mac_lb set_mac_lb;
struct ice_aqc_alloc_free_res_cmd sw_res_ctrl;
struct ice_aqc_set_mac_cfg set_mac_cfg;
ice_aqc_opc_get_rss_key = 0x0B04,
ice_aqc_opc_get_rss_lut = 0x0B05,
+ /* Sideband Control Interface commands */
+ ice_aqc_opc_neighbour_device_request = 0x0C00,
+
/* Tx queue handling commands/events */
ice_aqc_opc_add_txqs = 0x0C30,
ice_aqc_opc_dis_txqs = 0x0C31,
+ ice_aqc_opc_add_rdma_qset = 0x0C33,
/* package commands */
ice_aqc_opc_download_pkg = 0x0C40,
ice_aqc_opc_update_pkg = 0x0C42,
ice_aqc_opc_get_pkg_info_list = 0x0C43,
+ ice_aqc_opc_driver_shared_params = 0x0C90,
+
/* Standalone Commands/Events */
ice_aqc_opc_event_lan_overflow = 0x1001,
ice_is_arfs_using_perfect_flow(struct ice_hw *hw,
enum ice_fltr_ptype flow_type);
#else
-#define ice_sync_arfs_fltrs(pf) do {} while (0)
-#define ice_init_arfs(vsi) do {} while (0)
-#define ice_clear_arfs(vsi) do {} while (0)
-#define ice_remove_arfs(pf) do {} while (0)
-#define ice_free_cpu_rx_rmap(vsi) do {} while (0)
-#define ice_rebuild_arfs(pf) do {} while (0)
+static inline void ice_clear_arfs(struct ice_vsi *vsi) { }
+static inline void ice_free_cpu_rx_rmap(struct ice_vsi *vsi) { }
+static inline void ice_init_arfs(struct ice_vsi *vsi) { }
+static inline void ice_sync_arfs_fltrs(struct ice_pf *pf) { }
+static inline void ice_remove_arfs(struct ice_pf *pf) { }
+static inline void ice_rebuild_arfs(struct ice_pf *pf) { }
static inline int ice_set_cpu_rx_rmap(struct ice_vsi __always_unused *vsi)
{
/* make sure the context is associated with the right VSI */
tlan_ctx->src_vsi = ice_get_hw_vsi_num(hw, vsi->idx);
+ /* Restrict Tx timestamps to the PF VSI */
+ switch (vsi->type) {
+ case ICE_VSI_PF:
+ tlan_ctx->tsyn_ena = 1;
+ break;
+ default:
+ break;
+ }
+
tlan_ctx->tso_ena = ICE_TX_LEGACY;
tlan_ctx->tso_qnum = pf_q;
*
* Configure the Rx descriptor ring in RLAN context.
*/
-int ice_setup_rx_ctx(struct ice_ring *ring)
+static int ice_setup_rx_ctx(struct ice_ring *ring)
{
- struct device *dev = ice_pf_to_dev(ring->vsi->back);
int chain_len = ICE_MAX_CHAINED_RX_BUFS;
- u16 num_bufs = ICE_DESC_UNUSED(ring);
struct ice_vsi *vsi = ring->vsi;
u32 rxdid = ICE_RXDID_FLEX_NIC;
struct ice_rlan_ctx rlan_ctx;
/* clear the context structure first */
memset(&rlan_ctx, 0, sizeof(rlan_ctx));
- ring->rx_buf_len = vsi->rx_buf_len;
-
- if (ring->vsi->type == ICE_VSI_PF) {
- if (!xdp_rxq_info_is_reg(&ring->xdp_rxq))
- /* coverity[check_return] */
- xdp_rxq_info_reg(&ring->xdp_rxq, ring->netdev,
- ring->q_index, ring->q_vector->napi.napi_id);
-
- ring->xsk_pool = ice_xsk_pool(ring);
- if (ring->xsk_pool) {
- xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);
-
- ring->rx_buf_len =
- xsk_pool_get_rx_frame_size(ring->xsk_pool);
- /* For AF_XDP ZC, we disallow packets to span on
- * multiple buffers, thus letting us skip that
- * handling in the fast-path.
- */
- chain_len = 1;
- err = xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
- MEM_TYPE_XSK_BUFF_POOL,
- NULL);
- if (err)
- return err;
- xsk_pool_set_rxq_info(ring->xsk_pool, &ring->xdp_rxq);
-
- dev_info(dev, "Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring %d\n",
- ring->q_index);
- } else {
- if (!xdp_rxq_info_is_reg(&ring->xdp_rxq))
- /* coverity[check_return] */
- xdp_rxq_info_reg(&ring->xdp_rxq,
- ring->netdev,
- ring->q_index, ring->q_vector->napi.napi_id);
-
- err = xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
- MEM_TYPE_PAGE_SHARED,
- NULL);
- if (err)
- return err;
- }
- }
/* Receive Queue Base Address.
* Indicates the starting address of the descriptor queue defined in
* 128 Byte units.
*/
rlan_ctx.showiv = 0;
+ /* For AF_XDP ZC, we disallow packets to span on
+ * multiple buffers, thus letting us skip that
+ * handling in the fast-path.
+ */
+ if (ring->xsk_pool)
+ chain_len = 1;
/* Max packet size for this queue - must not be set to a larger value
* than 5 x DBUF
*/
* of same priority
*/
if (vsi->type != ICE_VSI_VF)
- ice_write_qrxflxp_cntxt(hw, pf_q, rxdid, 0x3);
+ ice_write_qrxflxp_cntxt(hw, pf_q, rxdid, 0x3, true);
else
- ice_write_qrxflxp_cntxt(hw, pf_q, ICE_RXDID_LEGACY_1, 0x3);
+ ice_write_qrxflxp_cntxt(hw, pf_q, ICE_RXDID_LEGACY_1, 0x3,
+ false);
/* Absolute queue number out of 2K needs to be passed */
err = ice_write_rxq_ctx(hw, &rlan_ctx, pf_q);
if (err) {
- dev_err(dev, "Failed to set LAN Rx queue context for absolute Rx queue %d error: %d\n",
+ dev_err(ice_pf_to_dev(vsi->back), "Failed to set LAN Rx queue context for absolute Rx queue %d error: %d\n",
pf_q, err);
return -EIO;
}
ring->tail = hw->hw_addr + QRX_TAIL(pf_q);
writel(0, ring->tail);
+ return 0;
+}
+
+/**
+ * ice_vsi_cfg_rxq - Configure an Rx queue
+ * @ring: the ring being configured
+ *
+ * Return 0 on success and a negative value on error.
+ */
+int ice_vsi_cfg_rxq(struct ice_ring *ring)
+{
+ struct device *dev = ice_pf_to_dev(ring->vsi->back);
+ u16 num_bufs = ICE_DESC_UNUSED(ring);
+ int err;
+
+ ring->rx_buf_len = ring->vsi->rx_buf_len;
+
+ if (ring->vsi->type == ICE_VSI_PF) {
+ if (!xdp_rxq_info_is_reg(&ring->xdp_rxq))
+ /* coverity[check_return] */
+ xdp_rxq_info_reg(&ring->xdp_rxq, ring->netdev,
+ ring->q_index, ring->q_vector->napi.napi_id);
+
+ ring->xsk_pool = ice_xsk_pool(ring);
+ if (ring->xsk_pool) {
+ xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);
+
+ ring->rx_buf_len =
+ xsk_pool_get_rx_frame_size(ring->xsk_pool);
+ err = xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
+ MEM_TYPE_XSK_BUFF_POOL,
+ NULL);
+ if (err)
+ return err;
+ xsk_pool_set_rxq_info(ring->xsk_pool, &ring->xdp_rxq);
+
+ dev_info(dev, "Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring %d\n",
+ ring->q_index);
+ } else {
+ if (!xdp_rxq_info_is_reg(&ring->xdp_rxq))
+ /* coverity[check_return] */
+ xdp_rxq_info_reg(&ring->xdp_rxq,
+ ring->netdev,
+ ring->q_index, ring->q_vector->napi.napi_id);
+
+ err = xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
+ MEM_TYPE_PAGE_SHARED,
+ NULL);
+ if (err)
+ return err;
+ }
+ }
+
+ err = ice_setup_rx_ctx(ring);
+ if (err) {
+ dev_err(dev, "ice_setup_rx_ctx failed for RxQ %d, err %d\n",
+ ring->q_index, err);
+ return err;
+ }
+
if (ring->xsk_pool) {
bool ok;
}
ok = ice_alloc_rx_bufs_zc(ring, num_bufs);
- if (!ok)
+ if (!ok) {
+ u16 pf_q = ring->vsi->rxq_map[ring->q_index];
+
dev_info(dev, "Failed to allocate some buffers on XSK buffer pool enabled Rx ring %d (pf_q %d)\n",
ring->q_index, pf_q);
+ }
+
return 0;
}
#include "ice.h"
-int ice_setup_rx_ctx(struct ice_ring *ring);
+int ice_vsi_cfg_rxq(struct ice_ring *ring);
int __ice_vsi_get_qs(struct ice_qs_cfg *qs_cfg);
int
ice_vsi_ctrl_one_rx_ring(struct ice_vsi *vsi, bool ena, u16 rxq_idx, bool wait);
/* Copyright (c) 2018, Intel Corporation. */
#include "ice_common.h"
+#include "ice_lib.h"
#include "ice_sched.h"
#include "ice_adminq_cmd.h"
#include "ice_flow.h"
}
/**
+ * ice_is_e810
+ * @hw: pointer to the hardware structure
+ *
+ * returns true if the device is E810 based, false if not.
+ */
+bool ice_is_e810(struct ice_hw *hw)
+{
+ return hw->mac_type == ICE_MAC_E810;
+}
+
+/**
* ice_clear_pf_cfg - Clear PF configuration
* @hw: pointer to the hardware structure
*
li->phy_type_high = le64_to_cpu(link_data.phy_type_high);
*hw_media_type = ice_get_media_type(pi);
li->link_info = link_data.link_info;
+ li->link_cfg_err = link_data.link_cfg_err;
li->an_info = link_data.an_info;
li->ext_info = link_data.ext_info;
li->max_frame_size = le16_to_cpu(link_data.max_frame_size);
(unsigned long long)li->phy_type_high);
ice_debug(hw, ICE_DBG_LINK, " media_type = 0x%x\n", *hw_media_type);
ice_debug(hw, ICE_DBG_LINK, " link_info = 0x%x\n", li->link_info);
+ ice_debug(hw, ICE_DBG_LINK, " link_cfg_err = 0x%x\n", li->link_cfg_err);
ice_debug(hw, ICE_DBG_LINK, " an_info = 0x%x\n", li->an_info);
ice_debug(hw, ICE_DBG_LINK, " ext_info = 0x%x\n", li->ext_info);
ice_debug(hw, ICE_DBG_LINK, " fec_info = 0x%x\n", li->fec_info);
GLNVM_ULD_POR_DONE_1_M |\
GLNVM_ULD_PCIER_DONE_2_M)
- uld_mask = ICE_RESET_DONE_MASK;
+ uld_mask = ICE_RESET_DONE_MASK | (hw->func_caps.common_cap.rdma ?
+ GLNVM_ULD_PE_DONE_M : 0);
/* Device is Active; check Global Reset processes are done */
for (cnt = 0; cnt < ICE_PF_RESET_WAIT_COUNT; cnt++) {
{ 0 }
};
+/* Sideband Queue command wrappers */
+
+/**
+ * ice_sbq_send_cmd - send Sideband Queue command to Sideband Queue
+ * @hw: pointer to the HW struct
+ * @desc: descriptor describing the command
+ * @buf: buffer to use for indirect commands (NULL for direct commands)
+ * @buf_size: size of buffer for indirect commands (0 for direct commands)
+ * @cd: pointer to command details structure
+ */
+static int
+ice_sbq_send_cmd(struct ice_hw *hw, struct ice_sbq_cmd_desc *desc,
+ void *buf, u16 buf_size, struct ice_sq_cd *cd)
+{
+ return ice_status_to_errno(ice_sq_send_cmd(hw, ice_get_sbq(hw),
+ (struct ice_aq_desc *)desc,
+ buf, buf_size, cd));
+}
+
+/**
+ * ice_sbq_rw_reg - Fill Sideband Queue command
+ * @hw: pointer to the HW struct
+ * @in: message info to be filled in descriptor
+ */
+int ice_sbq_rw_reg(struct ice_hw *hw, struct ice_sbq_msg_input *in)
+{
+ struct ice_sbq_cmd_desc desc = {0};
+ struct ice_sbq_msg_req msg = {0};
+ u16 msg_len;
+ int status;
+
+ msg_len = sizeof(msg);
+
+ msg.dest_dev = in->dest_dev;
+ msg.opcode = in->opcode;
+ msg.flags = ICE_SBQ_MSG_FLAGS;
+ msg.sbe_fbe = ICE_SBQ_MSG_SBE_FBE;
+ msg.msg_addr_low = cpu_to_le16(in->msg_addr_low);
+ msg.msg_addr_high = cpu_to_le32(in->msg_addr_high);
+
+ if (in->opcode)
+ msg.data = cpu_to_le32(in->data);
+ else
+ /* data read comes back in completion, so shorten the struct by
+ * sizeof(msg.data)
+ */
+ msg_len -= sizeof(msg.data);
+
+ desc.flags = cpu_to_le16(ICE_AQ_FLAG_RD);
+ desc.opcode = cpu_to_le16(ice_sbq_opc_neigh_dev_req);
+ desc.param0.cmd_len = cpu_to_le16(msg_len);
+ status = ice_sbq_send_cmd(hw, &desc, &msg, msg_len, NULL);
+ if (!status && !in->opcode)
+ in->data = le32_to_cpu
+ (((struct ice_sbq_msg_cmpl *)&msg)->data);
+ return status;
+}
+
/* FW Admin Queue command wrappers */
/* Software lock/mutex that is meant to be held while the Global Config Lock
ice_debug(hw, ICE_DBG_INIT, "%s: nvm_unified_update = %d\n", prefix,
caps->nvm_unified_update);
break;
+ case ICE_AQC_CAPS_RDMA:
+ caps->rdma = (number == 1);
+ ice_debug(hw, ICE_DBG_INIT, "%s: rdma = %d\n", prefix, caps->rdma);
+ break;
case ICE_AQC_CAPS_MAX_MTU:
caps->max_mtu = number;
ice_debug(hw, ICE_DBG_INIT, "%s: max_mtu = %d\n",
caps->maxtc = 4;
ice_debug(hw, ICE_DBG_INIT, "reducing maxtc to %d (based on #ports)\n",
caps->maxtc);
+ if (caps->rdma) {
+ ice_debug(hw, ICE_DBG_INIT, "forcing RDMA off\n");
+ caps->rdma = 0;
+ }
+
+ /* print message only when processing device capabilities
+ * during initialization.
+ */
+ if (caps == &hw->dev_caps.common_cap)
+ dev_info(ice_hw_to_dev(hw), "RDMA functionality is not available with the current device configuration.\n");
}
}
}
/**
+ * ice_parse_1588_func_caps - Parse ICE_AQC_CAPS_1588 function caps
+ * @hw: pointer to the HW struct
+ * @func_p: pointer to function capabilities structure
+ * @cap: pointer to the capability element to parse
+ *
+ * Extract function capabilities for ICE_AQC_CAPS_1588.
+ */
+static void
+ice_parse_1588_func_caps(struct ice_hw *hw, struct ice_hw_func_caps *func_p,
+ struct ice_aqc_list_caps_elem *cap)
+{
+ struct ice_ts_func_info *info = &func_p->ts_func_info;
+ u32 number = le32_to_cpu(cap->number);
+
+ info->ena = ((number & ICE_TS_FUNC_ENA_M) != 0);
+ func_p->common_cap.ieee_1588 = info->ena;
+
+ info->src_tmr_owned = ((number & ICE_TS_SRC_TMR_OWND_M) != 0);
+ info->tmr_ena = ((number & ICE_TS_TMR_ENA_M) != 0);
+ info->tmr_index_owned = ((number & ICE_TS_TMR_IDX_OWND_M) != 0);
+ info->tmr_index_assoc = ((number & ICE_TS_TMR_IDX_ASSOC_M) != 0);
+
+ info->clk_freq = (number & ICE_TS_CLK_FREQ_M) >> ICE_TS_CLK_FREQ_S;
+ info->clk_src = ((number & ICE_TS_CLK_SRC_M) != 0);
+
+ ice_debug(hw, ICE_DBG_INIT, "func caps: ieee_1588 = %u\n",
+ func_p->common_cap.ieee_1588);
+ ice_debug(hw, ICE_DBG_INIT, "func caps: src_tmr_owned = %u\n",
+ info->src_tmr_owned);
+ ice_debug(hw, ICE_DBG_INIT, "func caps: tmr_ena = %u\n",
+ info->tmr_ena);
+ ice_debug(hw, ICE_DBG_INIT, "func caps: tmr_index_owned = %u\n",
+ info->tmr_index_owned);
+ ice_debug(hw, ICE_DBG_INIT, "func caps: tmr_index_assoc = %u\n",
+ info->tmr_index_assoc);
+ ice_debug(hw, ICE_DBG_INIT, "func caps: clk_freq = %u\n",
+ info->clk_freq);
+ ice_debug(hw, ICE_DBG_INIT, "func caps: clk_src = %u\n",
+ info->clk_src);
+}
+
+/**
* ice_parse_fdir_func_caps - Parse ICE_AQC_CAPS_FD function caps
* @hw: pointer to the HW struct
* @func_p: pointer to function capabilities structure
case ICE_AQC_CAPS_VSI:
ice_parse_vsi_func_caps(hw, func_p, &cap_resp[i]);
break;
+ case ICE_AQC_CAPS_1588:
+ ice_parse_1588_func_caps(hw, func_p, &cap_resp[i]);
+ break;
case ICE_AQC_CAPS_FD:
ice_parse_fdir_func_caps(hw, func_p);
break;
}
/**
+ * ice_parse_1588_dev_caps - Parse ICE_AQC_CAPS_1588 device caps
+ * @hw: pointer to the HW struct
+ * @dev_p: pointer to device capabilities structure
+ * @cap: capability element to parse
+ *
+ * Parse ICE_AQC_CAPS_1588 for device capabilities.
+ */
+static void
+ice_parse_1588_dev_caps(struct ice_hw *hw, struct ice_hw_dev_caps *dev_p,
+ struct ice_aqc_list_caps_elem *cap)
+{
+ struct ice_ts_dev_info *info = &dev_p->ts_dev_info;
+ u32 logical_id = le32_to_cpu(cap->logical_id);
+ u32 phys_id = le32_to_cpu(cap->phys_id);
+ u32 number = le32_to_cpu(cap->number);
+
+ info->ena = ((number & ICE_TS_DEV_ENA_M) != 0);
+ dev_p->common_cap.ieee_1588 = info->ena;
+
+ info->tmr0_owner = number & ICE_TS_TMR0_OWNR_M;
+ info->tmr0_owned = ((number & ICE_TS_TMR0_OWND_M) != 0);
+ info->tmr0_ena = ((number & ICE_TS_TMR0_ENA_M) != 0);
+
+ info->tmr1_owner = (number & ICE_TS_TMR1_OWNR_M) >> ICE_TS_TMR1_OWNR_S;
+ info->tmr1_owned = ((number & ICE_TS_TMR1_OWND_M) != 0);
+ info->tmr1_ena = ((number & ICE_TS_TMR1_ENA_M) != 0);
+
+ info->ena_ports = logical_id;
+ info->tmr_own_map = phys_id;
+
+ ice_debug(hw, ICE_DBG_INIT, "dev caps: ieee_1588 = %u\n",
+ dev_p->common_cap.ieee_1588);
+ ice_debug(hw, ICE_DBG_INIT, "dev caps: tmr0_owner = %u\n",
+ info->tmr0_owner);
+ ice_debug(hw, ICE_DBG_INIT, "dev caps: tmr0_owned = %u\n",
+ info->tmr0_owned);
+ ice_debug(hw, ICE_DBG_INIT, "dev caps: tmr0_ena = %u\n",
+ info->tmr0_ena);
+ ice_debug(hw, ICE_DBG_INIT, "dev caps: tmr1_owner = %u\n",
+ info->tmr1_owner);
+ ice_debug(hw, ICE_DBG_INIT, "dev caps: tmr1_owned = %u\n",
+ info->tmr1_owned);
+ ice_debug(hw, ICE_DBG_INIT, "dev caps: tmr1_ena = %u\n",
+ info->tmr1_ena);
+ ice_debug(hw, ICE_DBG_INIT, "dev caps: ieee_1588 ena_ports = %u\n",
+ info->ena_ports);
+ ice_debug(hw, ICE_DBG_INIT, "dev caps: tmr_own_map = %u\n",
+ info->tmr_own_map);
+}
+
+/**
* ice_parse_fdir_dev_caps - Parse ICE_AQC_CAPS_FD device caps
* @hw: pointer to the HW struct
* @dev_p: pointer to device capabilities structure
case ICE_AQC_CAPS_VSI:
ice_parse_vsi_dev_caps(hw, dev_p, &cap_resp[i]);
break;
+ case ICE_AQC_CAPS_1588:
+ ice_parse_1588_dev_caps(hw, dev_p, &cap_resp[i]);
+ break;
case ICE_AQC_CAPS_FD:
ice_parse_fdir_dev_caps(hw, dev_p, &cap_resp[i]);
break;
return status;
}
+/**
+ * ice_aq_add_rdma_qsets
+ * @hw: pointer to the hardware structure
+ * @num_qset_grps: Number of RDMA Qset groups
+ * @qset_list: list of Qset groups to be added
+ * @buf_size: size of buffer for indirect command
+ * @cd: pointer to command details structure or NULL
+ *
+ * Add Tx RDMA Qsets (0x0C33)
+ */
+static int
+ice_aq_add_rdma_qsets(struct ice_hw *hw, u8 num_qset_grps,
+ struct ice_aqc_add_rdma_qset_data *qset_list,
+ u16 buf_size, struct ice_sq_cd *cd)
+{
+ struct ice_aqc_add_rdma_qset_data *list;
+ struct ice_aqc_add_rdma_qset *cmd;
+ struct ice_aq_desc desc;
+ u16 i, sum_size = 0;
+
+ cmd = &desc.params.add_rdma_qset;
+
+ ice_fill_dflt_direct_cmd_desc(&desc, ice_aqc_opc_add_rdma_qset);
+
+ if (num_qset_grps > ICE_LAN_TXQ_MAX_QGRPS)
+ return -EINVAL;
+
+ for (i = 0, list = qset_list; i < num_qset_grps; i++) {
+ u16 num_qsets = le16_to_cpu(list->num_qsets);
+
+ sum_size += struct_size(list, rdma_qsets, num_qsets);
+ list = (struct ice_aqc_add_rdma_qset_data *)(list->rdma_qsets +
+ num_qsets);
+ }
+
+ if (buf_size != sum_size)
+ return -EINVAL;
+
+ desc.flags |= cpu_to_le16(ICE_AQ_FLAG_RD);
+
+ cmd->num_qset_grps = num_qset_grps;
+
+ return ice_status_to_errno(ice_aq_send_cmd(hw, &desc, qset_list,
+ buf_size, cd));
+}
+
/* End of FW Admin Queue command wrappers */
/**
}
/**
+ * ice_cfg_vsi_rdma - configure the VSI RDMA queues
+ * @pi: port information structure
+ * @vsi_handle: software VSI handle
+ * @tc_bitmap: TC bitmap
+ * @max_rdmaqs: max RDMA queues array per TC
+ *
+ * This function adds/updates the VSI RDMA queues per TC.
+ */
+int
+ice_cfg_vsi_rdma(struct ice_port_info *pi, u16 vsi_handle, u16 tc_bitmap,
+ u16 *max_rdmaqs)
+{
+ return ice_status_to_errno(ice_cfg_vsi_qs(pi, vsi_handle, tc_bitmap,
+ max_rdmaqs,
+ ICE_SCHED_NODE_OWNER_RDMA));
+}
+
+/**
+ * ice_ena_vsi_rdma_qset
+ * @pi: port information structure
+ * @vsi_handle: software VSI handle
+ * @tc: TC number
+ * @rdma_qset: pointer to RDMA Qset
+ * @num_qsets: number of RDMA Qsets
+ * @qset_teid: pointer to Qset node TEIDs
+ *
+ * This function adds RDMA Qset
+ */
+int
+ice_ena_vsi_rdma_qset(struct ice_port_info *pi, u16 vsi_handle, u8 tc,
+ u16 *rdma_qset, u16 num_qsets, u32 *qset_teid)
+{
+ struct ice_aqc_txsched_elem_data node = { 0 };
+ struct ice_aqc_add_rdma_qset_data *buf;
+ struct ice_sched_node *parent;
+ enum ice_status status;
+ struct ice_hw *hw;
+ u16 i, buf_size;
+ int ret;
+
+ if (!pi || pi->port_state != ICE_SCHED_PORT_STATE_READY)
+ return -EIO;
+ hw = pi->hw;
+
+ if (!ice_is_vsi_valid(hw, vsi_handle))
+ return -EINVAL;
+
+ buf_size = struct_size(buf, rdma_qsets, num_qsets);
+ buf = kzalloc(buf_size, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+ mutex_lock(&pi->sched_lock);
+
+ parent = ice_sched_get_free_qparent(pi, vsi_handle, tc,
+ ICE_SCHED_NODE_OWNER_RDMA);
+ if (!parent) {
+ ret = -EINVAL;
+ goto rdma_error_exit;
+ }
+ buf->parent_teid = parent->info.node_teid;
+ node.parent_teid = parent->info.node_teid;
+
+ buf->num_qsets = cpu_to_le16(num_qsets);
+ for (i = 0; i < num_qsets; i++) {
+ buf->rdma_qsets[i].tx_qset_id = cpu_to_le16(rdma_qset[i]);
+ buf->rdma_qsets[i].info.valid_sections =
+ ICE_AQC_ELEM_VALID_GENERIC | ICE_AQC_ELEM_VALID_CIR |
+ ICE_AQC_ELEM_VALID_EIR;
+ buf->rdma_qsets[i].info.generic = 0;
+ buf->rdma_qsets[i].info.cir_bw.bw_profile_idx =
+ cpu_to_le16(ICE_SCHED_DFLT_RL_PROF_ID);
+ buf->rdma_qsets[i].info.cir_bw.bw_alloc =
+ cpu_to_le16(ICE_SCHED_DFLT_BW_WT);
+ buf->rdma_qsets[i].info.eir_bw.bw_profile_idx =
+ cpu_to_le16(ICE_SCHED_DFLT_RL_PROF_ID);
+ buf->rdma_qsets[i].info.eir_bw.bw_alloc =
+ cpu_to_le16(ICE_SCHED_DFLT_BW_WT);
+ }
+ ret = ice_aq_add_rdma_qsets(hw, 1, buf, buf_size, NULL);
+ if (ret) {
+ ice_debug(hw, ICE_DBG_RDMA, "add RDMA qset failed\n");
+ goto rdma_error_exit;
+ }
+ node.data.elem_type = ICE_AQC_ELEM_TYPE_LEAF;
+ for (i = 0; i < num_qsets; i++) {
+ node.node_teid = buf->rdma_qsets[i].qset_teid;
+ status = ice_sched_add_node(pi, hw->num_tx_sched_layers - 1,
+ &node);
+ if (status) {
+ ret = ice_status_to_errno(status);
+ break;
+ }
+ qset_teid[i] = le32_to_cpu(node.node_teid);
+ }
+rdma_error_exit:
+ mutex_unlock(&pi->sched_lock);
+ kfree(buf);
+ return ret;
+}
+
+/**
+ * ice_dis_vsi_rdma_qset - free RDMA resources
+ * @pi: port_info struct
+ * @count: number of RDMA Qsets to free
+ * @qset_teid: TEID of Qset node
+ * @q_id: list of queue IDs being disabled
+ */
+int
+ice_dis_vsi_rdma_qset(struct ice_port_info *pi, u16 count, u32 *qset_teid,
+ u16 *q_id)
+{
+ struct ice_aqc_dis_txq_item *qg_list;
+ enum ice_status status = 0;
+ struct ice_hw *hw;
+ u16 qg_size;
+ int i;
+
+ if (!pi || pi->port_state != ICE_SCHED_PORT_STATE_READY)
+ return -EIO;
+
+ hw = pi->hw;
+
+ qg_size = struct_size(qg_list, q_id, 1);
+ qg_list = kzalloc(qg_size, GFP_KERNEL);
+ if (!qg_list)
+ return -ENOMEM;
+
+ mutex_lock(&pi->sched_lock);
+
+ for (i = 0; i < count; i++) {
+ struct ice_sched_node *node;
+
+ node = ice_sched_find_node_by_teid(pi->root, qset_teid[i]);
+ if (!node)
+ continue;
+
+ qg_list->parent_teid = node->info.parent_teid;
+ qg_list->num_qs = 1;
+ qg_list->q_id[0] =
+ cpu_to_le16(q_id[i] |
+ ICE_AQC_Q_DIS_BUF_ELEM_TYPE_RDMA_QSET);
+
+ status = ice_aq_dis_lan_txq(hw, 1, qg_list, qg_size,
+ ICE_NO_RESET, 0, NULL);
+ if (status)
+ break;
+
+ ice_free_sched_node(pi, node);
+ }
+
+ mutex_unlock(&pi->sched_lock);
+ kfree(qg_list);
+ return ice_status_to_errno(status);
+}
+
+/**
* ice_replay_pre_init - replay pre initialization
* @hw: pointer to the HW struct
*
}
/**
+ * ice_aq_set_driver_param - Set driver parameter to share via firmware
+ * @hw: pointer to the HW struct
+ * @idx: parameter index to set
+ * @value: the value to set the parameter to
+ * @cd: pointer to command details structure or NULL
+ *
+ * Set the value of one of the software defined parameters. All PFs connected
+ * to this device can read the value using ice_aq_get_driver_param.
+ *
+ * Note that firmware provides no synchronization or locking, and will not
+ * save the parameter value during a device reset. It is expected that
+ * a single PF will write the parameter value, while all other PFs will only
+ * read it.
+ */
+int
+ice_aq_set_driver_param(struct ice_hw *hw, enum ice_aqc_driver_params idx,
+ u32 value, struct ice_sq_cd *cd)
+{
+ struct ice_aqc_driver_shared_params *cmd;
+ struct ice_aq_desc desc;
+
+ if (idx >= ICE_AQC_DRIVER_PARAM_MAX)
+ return -EIO;
+
+ cmd = &desc.params.drv_shared_params;
+
+ ice_fill_dflt_direct_cmd_desc(&desc, ice_aqc_opc_driver_shared_params);
+
+ cmd->set_or_get_op = ICE_AQC_DRIVER_PARAM_SET;
+ cmd->param_indx = idx;
+ cmd->param_val = cpu_to_le32(value);
+
+ return ice_status_to_errno(ice_aq_send_cmd(hw, &desc, NULL, 0, cd));
+}
+
+/**
+ * ice_aq_get_driver_param - Get driver parameter shared via firmware
+ * @hw: pointer to the HW struct
+ * @idx: parameter index to set
+ * @value: storage to return the shared parameter
+ * @cd: pointer to command details structure or NULL
+ *
+ * Get the value of one of the software defined parameters.
+ *
+ * Note that firmware provides no synchronization or locking. It is expected
+ * that only a single PF will write a given parameter.
+ */
+int
+ice_aq_get_driver_param(struct ice_hw *hw, enum ice_aqc_driver_params idx,
+ u32 *value, struct ice_sq_cd *cd)
+{
+ struct ice_aqc_driver_shared_params *cmd;
+ struct ice_aq_desc desc;
+ enum ice_status status;
+
+ if (idx >= ICE_AQC_DRIVER_PARAM_MAX)
+ return -EIO;
+
+ cmd = &desc.params.drv_shared_params;
+
+ ice_fill_dflt_direct_cmd_desc(&desc, ice_aqc_opc_driver_shared_params);
+
+ cmd->set_or_get_op = ICE_AQC_DRIVER_PARAM_GET;
+ cmd->param_indx = idx;
+
+ status = ice_aq_send_cmd(hw, &desc, NULL, 0, cd);
+ if (status)
+ return ice_status_to_errno(status);
+
+ *value = le32_to_cpu(cmd->param_val);
+
+ return 0;
+}
+
+/**
* ice_fw_supports_link_override
* @hw: pointer to the hardware structure
*
ice_aq_alloc_free_res(struct ice_hw *hw, u16 num_entries,
struct ice_aqc_alloc_free_res_elem *buf, u16 buf_size,
enum ice_adminq_opc opc, struct ice_sq_cd *cd);
+bool ice_is_sbq_supported(struct ice_hw *hw);
+struct ice_ctl_q_info *ice_get_sbq(struct ice_hw *hw);
enum ice_status
ice_sq_send_cmd(struct ice_hw *hw, struct ice_ctl_q_info *cq,
struct ice_aq_desc *desc, void *buf, u16 buf_size,
enum ice_status
ice_aq_manage_mac_write(struct ice_hw *hw, const u8 *mac_addr, u8 flags,
struct ice_sq_cd *cd);
+bool ice_is_e810(struct ice_hw *hw);
enum ice_status ice_clear_pf_cfg(struct ice_hw *hw);
enum ice_status
ice_aq_set_phy_cfg(struct ice_hw *hw, struct ice_port_info *pi,
u16 mem_addr, u8 page, u8 set_page, u8 *data, u8 length,
bool write, struct ice_sq_cd *cd);
+int
+ice_cfg_vsi_rdma(struct ice_port_info *pi, u16 vsi_handle, u16 tc_bitmap,
+ u16 *max_rdmaqs);
+int
+ice_ena_vsi_rdma_qset(struct ice_port_info *pi, u16 vsi_handle, u8 tc,
+ u16 *rdma_qset, u16 num_qsets, u32 *qset_teid);
+int
+ice_dis_vsi_rdma_qset(struct ice_port_info *pi, u16 count, u32 *qset_teid,
+ u16 *q_id);
enum ice_status
ice_dis_vsi_txq(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u8 num_queues,
u16 *q_handle, u16 *q_ids, u32 *q_teids,
void ice_output_fw_log(struct ice_hw *hw, struct ice_aq_desc *desc, void *buf);
struct ice_q_ctx *
ice_get_lan_q_ctx(struct ice_hw *hw, u16 vsi_handle, u8 tc, u16 q_handle);
+int ice_sbq_rw_reg(struct ice_hw *hw, struct ice_sbq_msg_input *in);
void
ice_stat_update40(struct ice_hw *hw, u32 reg, bool prev_stat_loaded,
u64 *prev_stat, u64 *cur_stat);
enum ice_status
ice_sched_query_elem(struct ice_hw *hw, u32 node_teid,
struct ice_aqc_txsched_elem_data *buf);
+int
+ice_aq_set_driver_param(struct ice_hw *hw, enum ice_aqc_driver_params idx,
+ u32 value, struct ice_sq_cd *cd);
+int
+ice_aq_get_driver_param(struct ice_hw *hw, enum ice_aqc_driver_params idx,
+ u32 *value, struct ice_sq_cd *cd);
enum ice_status
ice_aq_set_lldp_mib(struct ice_hw *hw, u8 mib_type, void *buf, u16 buf_size,
struct ice_sq_cd *cd);
}
/**
+ * ice_sb_init_regs - Initialize Sideband registers
+ * @hw: pointer to the hardware structure
+ *
+ * This assumes the alloc_sq and alloc_rq functions have already been called
+ */
+static void ice_sb_init_regs(struct ice_hw *hw)
+{
+ struct ice_ctl_q_info *cq = &hw->sbq;
+
+ ICE_CQ_INIT_REGS(cq, PF_SB);
+}
+
+/**
* ice_check_sq_alive
* @hw: pointer to the HW struct
* @cq: pointer to the specific Control queue
ice_adminq_init_regs(hw);
cq = &hw->adminq;
break;
+ case ICE_CTL_Q_SB:
+ ice_sb_init_regs(hw);
+ cq = &hw->sbq;
+ break;
case ICE_CTL_Q_MAILBOX:
ice_mailbox_init_regs(hw);
cq = &hw->mailboxq;
}
/**
+ * ice_is_sbq_supported - is the sideband queue supported
+ * @hw: pointer to the hardware structure
+ *
+ * Returns true if the sideband control queue interface is
+ * supported for the device, false otherwise
+ */
+bool ice_is_sbq_supported(struct ice_hw *hw)
+{
+ /* The device sideband queue is only supported on devices with the
+ * generic MAC type.
+ */
+ return hw->mac_type == ICE_MAC_GENERIC;
+}
+
+/**
+ * ice_get_sbq - returns the right control queue to use for sideband
+ * @hw: pointer to the hardware structure
+ */
+struct ice_ctl_q_info *ice_get_sbq(struct ice_hw *hw)
+{
+ if (ice_is_sbq_supported(hw))
+ return &hw->sbq;
+ return &hw->adminq;
+}
+
+/**
* ice_shutdown_ctrlq - shutdown routine for any control queue
* @hw: pointer to the hardware structure
* @q_type: specific Control queue type
if (ice_check_sq_alive(hw, cq))
ice_aq_q_shutdown(hw, true);
break;
+ case ICE_CTL_Q_SB:
+ cq = &hw->sbq;
+ break;
case ICE_CTL_Q_MAILBOX:
cq = &hw->mailboxq;
break;
{
/* Shutdown FW admin queue */
ice_shutdown_ctrlq(hw, ICE_CTL_Q_ADMIN);
+ /* Shutdown PHY Sideband */
+ if (ice_is_sbq_supported(hw))
+ ice_shutdown_ctrlq(hw, ICE_CTL_Q_SB);
/* Shutdown PF-VF Mailbox */
ice_shutdown_ctrlq(hw, ICE_CTL_Q_MAILBOX);
}
if (status)
return status;
+ /* sideband control queue (SBQ) interface is not supported on some
+ * devices. Initialize if supported, else fallback to the admin queue
+ * interface
+ */
+ if (ice_is_sbq_supported(hw)) {
+ status = ice_init_ctrlq(hw, ICE_CTL_Q_SB);
+ if (status)
+ return status;
+ }
/* Init Mailbox queue */
return ice_init_ctrlq(hw, ICE_CTL_Q_MAILBOX);
}
enum ice_status ice_create_all_ctrlq(struct ice_hw *hw)
{
ice_init_ctrlq_locks(&hw->adminq);
+ if (ice_is_sbq_supported(hw))
+ ice_init_ctrlq_locks(&hw->sbq);
ice_init_ctrlq_locks(&hw->mailboxq);
return ice_init_all_ctrlq(hw);
ice_shutdown_all_ctrlq(hw);
ice_destroy_ctrlq_locks(&hw->adminq);
+ if (ice_is_sbq_supported(hw))
+ ice_destroy_ctrlq_locks(&hw->sbq);
ice_destroy_ctrlq_locks(&hw->mailboxq);
}
/* Maximum buffer lengths for all control queue types */
#define ICE_AQ_MAX_BUF_LEN 4096
#define ICE_MBXQ_MAX_BUF_LEN 4096
+#define ICE_SBQ_MAX_BUF_LEN 512
#define ICE_CTL_Q_DESC(R, i) \
(&(((struct ice_aq_desc *)((R).desc_buf.va))[i]))
ICE_CTL_Q_UNKNOWN = 0,
ICE_CTL_Q_ADMIN,
ICE_CTL_Q_MAILBOX,
+ ICE_CTL_Q_SB,
};
/* Control Queue timeout settings - max delay 1s */
struct ice_dcbx_cfg *old_cfg, *curr_cfg;
struct device *dev = ice_pf_to_dev(pf);
int ret = ICE_DCB_NO_HW_CHG;
+ struct iidc_event *event;
struct ice_vsi *pf_vsi;
curr_cfg = &pf->hw.port_info->qos_cfg.local_dcbx_cfg;
goto free_cfg;
}
+ /* Notify AUX drivers about impending change to TCs */
+ event = kzalloc(sizeof(*event), GFP_KERNEL);
+ if (!event) {
+ ret = -ENOMEM;
+ goto free_cfg;
+ }
+
+ set_bit(IIDC_EVENT_BEFORE_TC_CHANGE, event->type);
+ ice_send_event_to_aux(pf, event);
+ kfree(event);
+
/* avoid race conditions by holding the lock while disabling and
* re-enabling the VSI
*/
void ice_pf_dcb_recfg(struct ice_pf *pf)
{
struct ice_dcbx_cfg *dcbcfg = &pf->hw.port_info->qos_cfg.local_dcbx_cfg;
+ struct iidc_event *event;
u8 tc_map = 0;
int v, ret;
if (vsi->type == ICE_VSI_PF)
ice_dcbnl_set_all(vsi);
}
+ /* Notify the AUX drivers that TC change is finished */
+ event = kzalloc(sizeof(*event), GFP_KERNEL);
+ if (!event)
+ return;
+
+ set_bit(IIDC_EVENT_AFTER_TC_CHANGE, event->type);
+ ice_send_event_to_aux(pf, event);
+ kfree(event);
}
/**
test_bit(ICE_FLAG_DCB_ENA, pf->flags));
}
#else
-#define ice_dcb_rebuild(pf) do {} while (0)
+static inline void ice_dcb_rebuild(struct ice_pf *pf) { }
static inline u8 ice_dcb_get_ena_tc(struct ice_dcbx_cfg __always_unused *dcbcfg)
{
return false;
}
-#define ice_update_dcb_stats(pf) do {} while (0)
-#define ice_pf_dcb_recfg(pf) do {} while (0)
-#define ice_vsi_cfg_dcb_rings(vsi) do {} while (0)
-#define ice_dcb_process_lldp_set_mib_change(pf, event) do {} while (0)
-#define ice_set_cgd_num(tlan_ctx, ring) do {} while (0)
-#define ice_vsi_cfg_netdev_tc(vsi, ena_tc) do {} while (0)
+static inline void ice_pf_dcb_recfg(struct ice_pf *pf) { }
+static inline void ice_vsi_cfg_dcb_rings(struct ice_vsi *vsi) { }
+static inline void ice_update_dcb_stats(struct ice_pf *pf) { }
+static inline void
+ice_dcb_process_lldp_set_mib_change(struct ice_pf *pf, struct ice_rq_event_info *event) { }
+static inline void ice_vsi_cfg_netdev_tc(struct ice_vsi *vsi, u8 ena_tc) { }
+static inline void ice_set_cgd_num(struct ice_tlan_ctx *tlan_ctx, struct ice_ring *ring) { }
#endif /* CONFIG_DCB */
#endif /* _ICE_DCB_LIB_H_ */
ice_dcbnl_flush_apps(struct ice_pf *pf, struct ice_dcbx_cfg *old_cfg,
struct ice_dcbx_cfg *new_cfg);
#else
-#define ice_dcbnl_setup(vsi) do {} while (0)
-#define ice_dcbnl_set_all(vsi) do {} while (0)
-#define ice_dcbnl_flush_apps(pf, old_cfg, new_cfg) do {} while (0)
+static inline void ice_dcbnl_setup(struct ice_vsi *vsi) { }
+static inline void ice_dcbnl_set_all(struct ice_vsi *vsi) { }
+static inline void
+ice_dcbnl_flush_apps(struct ice_pf *pf, struct ice_dcbx_cfg *old_cfg,
+ struct ice_dcbx_cfg *new_cfg) { }
#endif /* CONFIG_DCB */
-
#endif /* _ICE_DCB_NL_H_ */
size_t i;
int err;
+ err = ice_wait_for_reset(pf, 10 * HZ);
+ if (err) {
+ NL_SET_ERR_MSG_MOD(extack, "Device is busy resetting");
+ return err;
+ }
+
ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
if (!ctx)
return -ENOMEM;
/* discover capabilities first */
status = ice_discover_dev_caps(hw, &ctx->dev_caps);
if (status) {
+ dev_dbg(dev, "Failed to discover device capabilities, status %s aq_err %s\n",
+ ice_stat_str(status), ice_aq_str(hw->adminq.sq_last_status));
+ NL_SET_ERR_MSG_MOD(extack, "Unable to discover device capabilities");
err = -EIO;
goto out_free_ctx;
}
ice_ethtool_advertise_link_mode(ICE_AQ_LINK_SPEED_100GB,
100000baseKR4_Full);
}
-
- /* Autoneg PHY types */
- if (phy_types_low & ICE_PHY_TYPE_LOW_100BASE_TX ||
- phy_types_low & ICE_PHY_TYPE_LOW_1000BASE_T ||
- phy_types_low & ICE_PHY_TYPE_LOW_1000BASE_KX ||
- phy_types_low & ICE_PHY_TYPE_LOW_2500BASE_T ||
- phy_types_low & ICE_PHY_TYPE_LOW_2500BASE_KX ||
- phy_types_low & ICE_PHY_TYPE_LOW_5GBASE_T ||
- phy_types_low & ICE_PHY_TYPE_LOW_5GBASE_KR ||
- phy_types_low & ICE_PHY_TYPE_LOW_10GBASE_T ||
- phy_types_low & ICE_PHY_TYPE_LOW_10GBASE_KR_CR1 ||
- phy_types_low & ICE_PHY_TYPE_LOW_25GBASE_T ||
- phy_types_low & ICE_PHY_TYPE_LOW_25GBASE_CR ||
- phy_types_low & ICE_PHY_TYPE_LOW_25GBASE_CR_S ||
- phy_types_low & ICE_PHY_TYPE_LOW_25GBASE_CR1 ||
- phy_types_low & ICE_PHY_TYPE_LOW_25GBASE_KR ||
- phy_types_low & ICE_PHY_TYPE_LOW_25GBASE_KR_S ||
- phy_types_low & ICE_PHY_TYPE_LOW_25GBASE_KR1 ||
- phy_types_low & ICE_PHY_TYPE_LOW_40GBASE_CR4 ||
- phy_types_low & ICE_PHY_TYPE_LOW_40GBASE_KR4) {
- ethtool_link_ksettings_add_link_mode(ks, supported,
- Autoneg);
- ethtool_link_ksettings_add_link_mode(ks, advertising,
- Autoneg);
- }
- if (phy_types_low & ICE_PHY_TYPE_LOW_50GBASE_CR2 ||
- phy_types_low & ICE_PHY_TYPE_LOW_50GBASE_KR2 ||
- phy_types_low & ICE_PHY_TYPE_LOW_50GBASE_CP ||
- phy_types_low & ICE_PHY_TYPE_LOW_50GBASE_KR_PAM4) {
- ethtool_link_ksettings_add_link_mode(ks, supported,
- Autoneg);
- ethtool_link_ksettings_add_link_mode(ks, advertising,
- Autoneg);
- }
- if (phy_types_low & ICE_PHY_TYPE_LOW_100GBASE_CR4 ||
- phy_types_low & ICE_PHY_TYPE_LOW_100GBASE_KR4 ||
- phy_types_low & ICE_PHY_TYPE_LOW_100GBASE_KR_PAM4 ||
- phy_types_low & ICE_PHY_TYPE_LOW_100GBASE_CP2) {
- ethtool_link_ksettings_add_link_mode(ks, supported,
- Autoneg);
- ethtool_link_ksettings_add_link_mode(ks, advertising,
- Autoneg);
- }
}
#define TEST_SET_BITS_TIMEOUT 50
ks->base.port = PORT_TP;
break;
case ICE_MEDIA_BACKPLANE:
- ethtool_link_ksettings_add_link_mode(ks, supported, Autoneg);
ethtool_link_ksettings_add_link_mode(ks, supported, Backplane);
- ethtool_link_ksettings_add_link_mode(ks, advertising, Autoneg);
ethtool_link_ksettings_add_link_mode(ks, advertising,
Backplane);
ks->base.port = PORT_NONE;
if (caps->link_fec_options & ICE_AQC_PHY_FEC_25G_RS_CLAUSE91_EN)
ethtool_link_ksettings_add_link_mode(ks, supported, FEC_RS);
+ /* Set supported and advertised autoneg */
+ if (ice_is_phy_caps_an_enabled(caps)) {
+ ethtool_link_ksettings_add_link_mode(ks, supported, Autoneg);
+ ethtool_link_ksettings_add_link_mode(ks, advertising, Autoneg);
+ }
+
done:
kfree(caps);
return err;
return 0;
}
+static int
+ice_get_ts_info(struct net_device *dev, struct ethtool_ts_info *info)
+{
+ struct ice_pf *pf = ice_netdev_to_pf(dev);
+
+ /* only report timestamping if PTP is enabled */
+ if (!test_bit(ICE_FLAG_PTP, pf->flags))
+ return ethtool_op_get_ts_info(dev, info);
+
+ info->so_timestamping = SOF_TIMESTAMPING_TX_SOFTWARE |
+ SOF_TIMESTAMPING_RX_SOFTWARE |
+ SOF_TIMESTAMPING_SOFTWARE |
+ SOF_TIMESTAMPING_TX_HARDWARE |
+ SOF_TIMESTAMPING_RX_HARDWARE |
+ SOF_TIMESTAMPING_RAW_HARDWARE;
+
+ info->phc_index = ice_get_ptp_clock_index(pf);
+
+ info->tx_types = BIT(HWTSTAMP_TX_OFF) | BIT(HWTSTAMP_TX_ON);
+
+ info->rx_filters = BIT(HWTSTAMP_FILTER_NONE) | BIT(HWTSTAMP_FILTER_ALL);
+
+ return 0;
+}
+
/**
* ice_get_max_txq - return the maximum number of Tx queues for in a PF
* @pf: PF structure
ice_get_rc_coalesce(struct ethtool_coalesce *ec, enum ice_container_type c_type,
struct ice_ring_container *rc)
{
- struct ice_pf *pf;
-
if (!rc->ring)
return -EINVAL;
- pf = rc->ring->vsi->back;
-
switch (c_type) {
case ICE_RX_CONTAINER:
ec->use_adaptive_rx_coalesce = ITR_IS_DYNAMIC(rc);
ec->tx_coalesce_usecs = rc->itr_setting;
break;
default:
- dev_dbg(ice_pf_to_dev(pf), "Invalid c_type %d\n", c_type);
+ dev_dbg(ice_pf_to_dev(rc->ring->vsi->back), "Invalid c_type %d\n", c_type);
return -EINVAL;
}
.set_rxfh = ice_set_rxfh,
.get_channels = ice_get_channels,
.set_channels = ice_set_channels,
- .get_ts_info = ethtool_op_get_ts_info,
+ .get_ts_info = ice_get_ts_info,
.get_per_queue_coalesce = ice_get_per_q_coalesce,
.set_per_queue_coalesce = ice_set_per_q_coalesce,
.get_fecparam = ice_get_fecparam,
}
err = pldmfw_flash_image(&priv.context, fw);
+ if (err == -ENOENT) {
+ dev_err(dev, "Firmware image has no record matching this device\n");
+ NL_SET_ERR_MSG_MOD(extack, "Firmware image has no record matching this device");
+ } else if (err) {
+ /* Do not set a generic extended ACK message here. A more
+ * specific message may already have been set by one of our
+ * ops.
+ */
+ dev_err(dev, "Failed to flash PLDM image, err %d", err);
+ }
ice_release_nvm(hw);
#define PF_FW_ATQLEN_ATQOVFL_M BIT(29)
#define PF_FW_ATQLEN_ATQCRIT_M BIT(30)
#define VF_MBX_ARQLEN(_VF) (0x0022BC00 + ((_VF) * 4))
+#define VF_MBX_ATQLEN(_VF) (0x0022A800 + ((_VF) * 4))
#define PF_FW_ATQLEN_ATQENABLE_M BIT(31)
#define PF_FW_ATQT 0x00080400
#define PF_MBX_ARQBAH 0x0022E400
#define PF_MBX_ATQLEN_ATQCRIT_M BIT(30)
#define PF_MBX_ATQLEN_ATQENABLE_M BIT(31)
#define PF_MBX_ATQT 0x0022E300
+#define PF_SB_ARQBAH 0x0022FF00
+#define PF_SB_ARQBAH_ARQBAH_S 0
+#define PF_SB_ARQBAH_ARQBAH_M ICE_M(0xFFFFFFFF, 0)
+#define PF_SB_ARQBAL 0x0022FE80
+#define PF_SB_ARQBAL_ARQBAL_LSB_S 0
+#define PF_SB_ARQBAL_ARQBAL_LSB_M ICE_M(0x3F, 0)
+#define PF_SB_ARQBAL_ARQBAL_S 6
+#define PF_SB_ARQBAL_ARQBAL_M ICE_M(0x3FFFFFF, 6)
+#define PF_SB_ARQH 0x00230000
+#define PF_SB_ARQH_ARQH_S 0
+#define PF_SB_ARQH_ARQH_M ICE_M(0x3FF, 0)
+#define PF_SB_ARQLEN 0x0022FF80
+#define PF_SB_ARQLEN_ARQLEN_S 0
+#define PF_SB_ARQLEN_ARQLEN_M ICE_M(0x3FF, 0)
+#define PF_SB_ARQLEN_ARQVFE_S 28
+#define PF_SB_ARQLEN_ARQVFE_M BIT(28)
+#define PF_SB_ARQLEN_ARQOVFL_S 29
+#define PF_SB_ARQLEN_ARQOVFL_M BIT(29)
+#define PF_SB_ARQLEN_ARQCRIT_S 30
+#define PF_SB_ARQLEN_ARQCRIT_M BIT(30)
+#define PF_SB_ARQLEN_ARQENABLE_S 31
+#define PF_SB_ARQLEN_ARQENABLE_M BIT(31)
+#define PF_SB_ARQT 0x00230080
+#define PF_SB_ARQT_ARQT_S 0
+#define PF_SB_ARQT_ARQT_M ICE_M(0x3FF, 0)
+#define PF_SB_ATQBAH 0x0022FC80
+#define PF_SB_ATQBAH_ATQBAH_S 0
+#define PF_SB_ATQBAH_ATQBAH_M ICE_M(0xFFFFFFFF, 0)
+#define PF_SB_ATQBAL 0x0022FC00
+#define PF_SB_ATQBAL_ATQBAL_S 6
+#define PF_SB_ATQBAL_ATQBAL_M ICE_M(0x3FFFFFF, 6)
+#define PF_SB_ATQH 0x0022FD80
+#define PF_SB_ATQH_ATQH_S 0
+#define PF_SB_ATQH_ATQH_M ICE_M(0x3FF, 0)
+#define PF_SB_ATQLEN 0x0022FD00
+#define PF_SB_ATQLEN_ATQLEN_S 0
+#define PF_SB_ATQLEN_ATQLEN_M ICE_M(0x3FF, 0)
+#define PF_SB_ATQLEN_ATQVFE_S 28
+#define PF_SB_ATQLEN_ATQVFE_M BIT(28)
+#define PF_SB_ATQLEN_ATQOVFL_S 29
+#define PF_SB_ATQLEN_ATQOVFL_M BIT(29)
+#define PF_SB_ATQLEN_ATQCRIT_S 30
+#define PF_SB_ATQLEN_ATQCRIT_M BIT(30)
+#define PF_SB_ATQLEN_ATQENABLE_S 31
+#define PF_SB_ATQLEN_ATQENABLE_M BIT(31)
+#define PF_SB_ATQT 0x0022FE00
+#define PF_SB_ATQT_ATQT_S 0
+#define PF_SB_ATQT_ATQT_M ICE_M(0x3FF, 0)
#define PRTDCB_GENC 0x00083000
#define PRTDCB_GENC_PFCLDA_S 16
#define PRTDCB_GENC_PFCLDA_M ICE_M(0xFFFF, 16)
#define GLGEN_CLKSTAT_SRC_PSM_CLK_SRC_S 4
#define GLGEN_CLKSTAT_SRC_PSM_CLK_SRC_M ICE_M(0x3, 4)
#define GLGEN_CLKSTAT_SRC 0x000B826C
+#define GLGEN_GPIO_CTL(_i) (0x000880C8 + ((_i) * 4))
+#define GLGEN_GPIO_CTL_PIN_DIR_M BIT(4)
+#define GLGEN_GPIO_CTL_PIN_FUNC_S 8
+#define GLGEN_GPIO_CTL_PIN_FUNC_M ICE_M(0xF, 8)
#define GLGEN_RSTAT 0x000B8188
#define GLGEN_RSTAT_DEVSTATE_M ICE_M(0x3, 0)
#define GLGEN_RSTCTL 0x000B8180
#define VPGEN_VFRSTAT_VFRD_M BIT(0)
#define VPGEN_VFRTRIG(_VF) (0x00090000 + ((_VF) * 4))
#define VPGEN_VFRTRIG_VFSWR_M BIT(0)
-#define PFHMC_ERRORDATA 0x00520500
-#define PFHMC_ERRORINFO 0x00520400
#define GLINT_CTL 0x0016CC54
#define GLINT_CTL_DIS_AUTOMASK_M BIT(0)
#define GLINT_CTL_ITR_GRAN_200_S 16
#define PFINT_MBX_CTL_ITR_INDX_M ICE_M(0x3, 11)
#define PFINT_MBX_CTL_CAUSE_ENA_M BIT(30)
#define PFINT_OICR 0x0016CA00
+#define PFINT_OICR_TSYN_TX_M BIT(11)
+#define PFINT_OICR_TSYN_EVNT_M BIT(12)
#define PFINT_OICR_ECC_ERR_M BIT(16)
#define PFINT_OICR_MAL_DETECT_M BIT(19)
#define PFINT_OICR_GRST_M BIT(20)
#define PFINT_OICR_PCI_EXCEPTION_M BIT(21)
#define PFINT_OICR_HMC_ERR_M BIT(26)
+#define PFINT_OICR_PE_PUSH_M BIT(27)
#define PFINT_OICR_PE_CRITERR_M BIT(28)
#define PFINT_OICR_VFLR_M BIT(29)
#define PFINT_OICR_SWINT_M BIT(31)
#define PFINT_OICR_CTL_ITR_INDX_M ICE_M(0x3, 11)
#define PFINT_OICR_CTL_CAUSE_ENA_M BIT(30)
#define PFINT_OICR_ENA 0x0016C900
+#define PFINT_SB_CTL 0x0016B600
+#define PFINT_SB_CTL_MSIX_INDX_M ICE_M(0x7FF, 0)
+#define PFINT_SB_CTL_CAUSE_ENA_M BIT(30)
#define QINT_RQCTL(_QRX) (0x00150000 + ((_QRX) * 4))
#define QINT_RQCTL_MSIX_INDX_S 0
#define QINT_RQCTL_MSIX_INDX_M ICE_M(0x7FF, 0)
#define GLV_UPRCL(_i) (0x003B2000 + ((_i) * 8))
#define GLV_UPTCL(_i) (0x0030A000 + ((_i) * 8))
#define PRTRPB_RDPC 0x000AC260
+#define GLTSYN_AUX_IN_0(_i) (0x000889D8 + ((_i) * 4))
+#define GLTSYN_AUX_IN_0_INT_ENA_M BIT(4)
+#define GLTSYN_AUX_OUT_0(_i) (0x00088998 + ((_i) * 4))
+#define GLTSYN_AUX_OUT_0_OUT_ENA_M BIT(0)
+#define GLTSYN_AUX_OUT_0_OUTMOD_M ICE_M(0x3, 1)
+#define GLTSYN_CLKO_0(_i) (0x000889B8 + ((_i) * 4))
+#define GLTSYN_CMD 0x00088810
+#define GLTSYN_CMD_SYNC 0x00088814
+#define GLTSYN_ENA(_i) (0x00088808 + ((_i) * 4))
+#define GLTSYN_ENA_TSYN_ENA_M BIT(0)
+#define GLTSYN_EVNT_H_0(_i) (0x00088970 + ((_i) * 4))
+#define GLTSYN_EVNT_L_0(_i) (0x00088968 + ((_i) * 4))
+#define GLTSYN_INCVAL_H(_i) (0x00088920 + ((_i) * 4))
+#define GLTSYN_INCVAL_L(_i) (0x00088918 + ((_i) * 4))
+#define GLTSYN_SHADJ_H(_i) (0x00088910 + ((_i) * 4))
+#define GLTSYN_SHADJ_L(_i) (0x00088908 + ((_i) * 4))
+#define GLTSYN_SHTIME_0(_i) (0x000888E0 + ((_i) * 4))
+#define GLTSYN_SHTIME_H(_i) (0x000888F0 + ((_i) * 4))
+#define GLTSYN_SHTIME_L(_i) (0x000888E8 + ((_i) * 4))
+#define GLTSYN_STAT(_i) (0x000888C0 + ((_i) * 4))
+#define GLTSYN_STAT_EVENT0_M BIT(0)
+#define GLTSYN_STAT_EVENT1_M BIT(1)
+#define GLTSYN_STAT_EVENT2_M BIT(2)
+#define GLTSYN_SYNC_DLAY 0x00088818
+#define GLTSYN_TGT_H_0(_i) (0x00088930 + ((_i) * 4))
+#define GLTSYN_TGT_L_0(_i) (0x00088928 + ((_i) * 4))
+#define GLTSYN_TIME_H(_i) (0x000888D8 + ((_i) * 4))
+#define GLTSYN_TIME_L(_i) (0x000888D0 + ((_i) * 4))
+#define PFTSYN_SEM 0x00088880
+#define PFTSYN_SEM_BUSY_M BIT(0)
#define VSIQF_FD_CNT(_VSI) (0x00464000 + ((_VSI) * 4))
#define VSIQF_FD_CNT_FD_GCNT_S 0
#define VSIQF_FD_CNT_FD_GCNT_M ICE_M(0x3FFF, 0)
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (C) 2021, Intel Corporation. */
+
+/* Inter-Driver Communication */
+#include "ice.h"
+#include "ice_lib.h"
+#include "ice_dcb_lib.h"
+
+/**
+ * ice_get_auxiliary_drv - retrieve iidc_auxiliary_drv struct
+ * @pf: pointer to PF struct
+ *
+ * This function has to be called with a device_lock on the
+ * pf->adev.dev to avoid race conditions.
+ */
+static struct iidc_auxiliary_drv *ice_get_auxiliary_drv(struct ice_pf *pf)
+{
+ struct auxiliary_device *adev;
+
+ adev = pf->adev;
+ if (!adev || !adev->dev.driver)
+ return NULL;
+
+ return container_of(adev->dev.driver, struct iidc_auxiliary_drv,
+ adrv.driver);
+}
+
+/**
+ * ice_send_event_to_aux - send event to RDMA AUX driver
+ * @pf: pointer to PF struct
+ * @event: event struct
+ */
+void ice_send_event_to_aux(struct ice_pf *pf, struct iidc_event *event)
+{
+ struct iidc_auxiliary_drv *iadrv;
+
+ if (!pf->adev)
+ return;
+
+ device_lock(&pf->adev->dev);
+ iadrv = ice_get_auxiliary_drv(pf);
+ if (iadrv && iadrv->event_handler)
+ iadrv->event_handler(pf, event);
+ device_unlock(&pf->adev->dev);
+}
+
+/**
+ * ice_find_vsi - Find the VSI from VSI ID
+ * @pf: The PF pointer to search in
+ * @vsi_num: The VSI ID to search for
+ */
+static struct ice_vsi *ice_find_vsi(struct ice_pf *pf, u16 vsi_num)
+{
+ int i;
+
+ ice_for_each_vsi(pf, i)
+ if (pf->vsi[i] && pf->vsi[i]->vsi_num == vsi_num)
+ return pf->vsi[i];
+ return NULL;
+}
+
+/**
+ * ice_add_rdma_qset - Add Leaf Node for RDMA Qset
+ * @pf: PF struct
+ * @qset: Resource to be allocated
+ */
+int ice_add_rdma_qset(struct ice_pf *pf, struct iidc_rdma_qset_params *qset)
+{
+ u16 max_rdmaqs[ICE_MAX_TRAFFIC_CLASS];
+ struct ice_vsi *vsi;
+ struct device *dev;
+ u32 qset_teid;
+ u16 qs_handle;
+ int status;
+ int i;
+
+ if (WARN_ON(!pf || !qset))
+ return -EINVAL;
+
+ dev = ice_pf_to_dev(pf);
+
+ if (!test_bit(ICE_FLAG_RDMA_ENA, pf->flags))
+ return -EINVAL;
+
+ vsi = ice_get_main_vsi(pf);
+ if (!vsi) {
+ dev_err(dev, "RDMA QSet invalid VSI\n");
+ return -EINVAL;
+ }
+
+ ice_for_each_traffic_class(i)
+ max_rdmaqs[i] = 0;
+
+ max_rdmaqs[qset->tc]++;
+ qs_handle = qset->qs_handle;
+
+ status = ice_cfg_vsi_rdma(vsi->port_info, vsi->idx, vsi->tc_cfg.ena_tc,
+ max_rdmaqs);
+ if (status) {
+ dev_err(dev, "Failed VSI RDMA Qset config\n");
+ return status;
+ }
+
+ status = ice_ena_vsi_rdma_qset(vsi->port_info, vsi->idx, qset->tc,
+ &qs_handle, 1, &qset_teid);
+ if (status) {
+ dev_err(dev, "Failed VSI RDMA Qset enable\n");
+ return status;
+ }
+ vsi->qset_handle[qset->tc] = qset->qs_handle;
+ qset->teid = qset_teid;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(ice_add_rdma_qset);
+
+/**
+ * ice_del_rdma_qset - Delete leaf node for RDMA Qset
+ * @pf: PF struct
+ * @qset: Resource to be freed
+ */
+int ice_del_rdma_qset(struct ice_pf *pf, struct iidc_rdma_qset_params *qset)
+{
+ struct ice_vsi *vsi;
+ u32 teid;
+ u16 q_id;
+
+ if (WARN_ON(!pf || !qset))
+ return -EINVAL;
+
+ vsi = ice_find_vsi(pf, qset->vport_id);
+ if (!vsi) {
+ dev_err(ice_pf_to_dev(pf), "RDMA Invalid VSI\n");
+ return -EINVAL;
+ }
+
+ q_id = qset->qs_handle;
+ teid = qset->teid;
+
+ vsi->qset_handle[qset->tc] = 0;
+
+ return ice_dis_vsi_rdma_qset(vsi->port_info, 1, &teid, &q_id);
+}
+EXPORT_SYMBOL_GPL(ice_del_rdma_qset);
+
+/**
+ * ice_rdma_request_reset - accept request from RDMA to perform a reset
+ * @pf: struct for PF
+ * @reset_type: type of reset
+ */
+int ice_rdma_request_reset(struct ice_pf *pf, enum iidc_reset_type reset_type)
+{
+ enum ice_reset_req reset;
+
+ if (WARN_ON(!pf))
+ return -EINVAL;
+
+ switch (reset_type) {
+ case IIDC_PFR:
+ reset = ICE_RESET_PFR;
+ break;
+ case IIDC_CORER:
+ reset = ICE_RESET_CORER;
+ break;
+ case IIDC_GLOBR:
+ reset = ICE_RESET_GLOBR;
+ break;
+ default:
+ dev_err(ice_pf_to_dev(pf), "incorrect reset request\n");
+ return -EINVAL;
+ }
+
+ return ice_schedule_reset(pf, reset);
+}
+EXPORT_SYMBOL_GPL(ice_rdma_request_reset);
+
+/**
+ * ice_rdma_update_vsi_filter - update main VSI filters for RDMA
+ * @pf: pointer to struct for PF
+ * @vsi_id: VSI HW idx to update filter on
+ * @enable: bool whether to enable or disable filters
+ */
+int ice_rdma_update_vsi_filter(struct ice_pf *pf, u16 vsi_id, bool enable)
+{
+ struct ice_vsi *vsi;
+ int status;
+
+ if (WARN_ON(!pf))
+ return -EINVAL;
+
+ vsi = ice_find_vsi(pf, vsi_id);
+ if (!vsi)
+ return -EINVAL;
+
+ status = ice_cfg_rdma_fltr(&pf->hw, vsi->idx, enable);
+ if (status) {
+ dev_err(ice_pf_to_dev(pf), "Failed to %sable RDMA filtering\n",
+ enable ? "en" : "dis");
+ } else {
+ if (enable)
+ vsi->info.q_opt_flags |= ICE_AQ_VSI_Q_OPT_PE_FLTR_EN;
+ else
+ vsi->info.q_opt_flags &= ~ICE_AQ_VSI_Q_OPT_PE_FLTR_EN;
+ }
+
+ return status;
+}
+EXPORT_SYMBOL_GPL(ice_rdma_update_vsi_filter);
+
+/**
+ * ice_get_qos_params - parse QoS params for RDMA consumption
+ * @pf: pointer to PF struct
+ * @qos: set of QoS values
+ */
+void ice_get_qos_params(struct ice_pf *pf, struct iidc_qos_params *qos)
+{
+ struct ice_dcbx_cfg *dcbx_cfg;
+ unsigned int i;
+ u32 up2tc;
+
+ dcbx_cfg = &pf->hw.port_info->qos_cfg.local_dcbx_cfg;
+ up2tc = rd32(&pf->hw, PRTDCB_TUP2TC);
+
+ qos->num_tc = ice_dcb_get_num_tc(dcbx_cfg);
+ for (i = 0; i < IIDC_MAX_USER_PRIORITY; i++)
+ qos->up2tc[i] = (up2tc >> (i * 3)) & 0x7;
+
+ for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++)
+ qos->tc_info[i].rel_bw = dcbx_cfg->etscfg.tcbwtable[i];
+}
+EXPORT_SYMBOL_GPL(ice_get_qos_params);
+
+/**
+ * ice_reserve_rdma_qvector - Reserve vector resources for RDMA driver
+ * @pf: board private structure to initialize
+ */
+static int ice_reserve_rdma_qvector(struct ice_pf *pf)
+{
+ if (test_bit(ICE_FLAG_RDMA_ENA, pf->flags)) {
+ int index;
+
+ index = ice_get_res(pf, pf->irq_tracker, pf->num_rdma_msix,
+ ICE_RES_RDMA_VEC_ID);
+ if (index < 0)
+ return index;
+ pf->num_avail_sw_msix -= pf->num_rdma_msix;
+ pf->rdma_base_vector = (u16)index;
+ }
+ return 0;
+}
+
+/**
+ * ice_adev_release - function to be mapped to AUX dev's release op
+ * @dev: pointer to device to free
+ */
+static void ice_adev_release(struct device *dev)
+{
+ struct iidc_auxiliary_dev *iadev;
+
+ iadev = container_of(dev, struct iidc_auxiliary_dev, adev.dev);
+ kfree(iadev);
+}
+
+/**
+ * ice_plug_aux_dev - allocate and register AUX device
+ * @pf: pointer to pf struct
+ */
+int ice_plug_aux_dev(struct ice_pf *pf)
+{
+ struct iidc_auxiliary_dev *iadev;
+ struct auxiliary_device *adev;
+ int ret;
+
+ iadev = kzalloc(sizeof(*iadev), GFP_KERNEL);
+ if (!iadev)
+ return -ENOMEM;
+
+ adev = &iadev->adev;
+ pf->adev = adev;
+ iadev->pf = pf;
+
+ adev->id = pf->aux_idx;
+ adev->dev.release = ice_adev_release;
+ adev->dev.parent = &pf->pdev->dev;
+ adev->name = IIDC_RDMA_ROCE_NAME;
+
+ ret = auxiliary_device_init(adev);
+ if (ret) {
+ pf->adev = NULL;
+ kfree(iadev);
+ return ret;
+ }
+
+ ret = auxiliary_device_add(adev);
+ if (ret) {
+ pf->adev = NULL;
+ auxiliary_device_uninit(adev);
+ return ret;
+ }
+
+ return 0;
+}
+
+/* ice_unplug_aux_dev - unregister and free AUX device
+ * @pf: pointer to pf struct
+ */
+void ice_unplug_aux_dev(struct ice_pf *pf)
+{
+ if (!pf->adev)
+ return;
+
+ auxiliary_device_delete(pf->adev);
+ auxiliary_device_uninit(pf->adev);
+ pf->adev = NULL;
+}
+
+/**
+ * ice_init_rdma - initializes PF for RDMA use
+ * @pf: ptr to ice_pf
+ */
+int ice_init_rdma(struct ice_pf *pf)
+{
+ struct device *dev = &pf->pdev->dev;
+ int ret;
+
+ /* Reserve vector resources */
+ ret = ice_reserve_rdma_qvector(pf);
+ if (ret < 0) {
+ dev_err(dev, "failed to reserve vectors for RDMA\n");
+ return ret;
+ }
+
+ return ice_plug_aux_dev(pf);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2021, Intel Corporation. */
+
+#ifndef _ICE_IDC_INT_H_
+#define _ICE_IDC_INT_H_
+
+#include <linux/net/intel/iidc.h>
+#include "ice.h"
+
+struct ice_pf;
+
+void ice_send_event_to_aux(struct ice_pf *pf, struct iidc_event *event);
+
+#endif /* !_ICE_IDC_INT_H_ */
}
ice_clear_sriov_cap(pf);
+ ice_clear_rdma_cap(pf);
lag->bonded = true;
lag->role = ICE_LAG_UNSET;
}
ice_set_sriov_cap(pf);
+ ice_set_rdma_cap(pf);
lag->bonded = false;
lag->role = ICE_LAG_NONE;
}
#define ICE_FXD_FLTR_WB_QW1_FAIL_PROF_YES 0x1ULL
struct ice_rx_ptype_decoded {
- u32 ptype:10;
u32 known:1;
u32 outer_ip:1;
u32 outer_ip_ver:2;
u8 int_q_state; /* width not needed - internal - DO NOT WRITE!!! */
};
-/* macro to make the table lines short */
+/* The ice_ptype_lkup table is used to convert from the 10-bit ptype in the
+ * hardware to a bit-field that can be used by SW to more easily determine the
+ * packet type.
+ *
+ * Macros are used to shorten the table lines and make this table human
+ * readable.
+ *
+ * We store the PTYPE in the top byte of the bit field - this is just so that
+ * we can check that the table doesn't have a row missing, as the index into
+ * the table should be the PTYPE.
+ *
+ * Typical work flow:
+ *
+ * IF NOT ice_ptype_lkup[ptype].known
+ * THEN
+ * Packet is unknown
+ * ELSE IF ice_ptype_lkup[ptype].outer_ip == ICE_RX_PTYPE_OUTER_IP
+ * Use the rest of the fields to look at the tunnels, inner protocols, etc
+ * ELSE
+ * Use the enum ice_rx_l2_ptype to decode the packet type
+ * ENDIF
+ */
+
+/* macro to make the table lines short, use explicit indexing with [PTYPE] */
#define ICE_PTT(PTYPE, OUTER_IP, OUTER_IP_VER, OUTER_FRAG, T, TE, TEF, I, PL)\
- { PTYPE, \
+ [PTYPE] = { \
1, \
ICE_RX_PTYPE_OUTER_##OUTER_IP, \
ICE_RX_PTYPE_OUTER_##OUTER_IP_VER, \
ICE_RX_PTYPE_INNER_PROT_##I, \
ICE_RX_PTYPE_PAYLOAD_LAYER_##PL }
-#define ICE_PTT_UNUSED_ENTRY(PTYPE) { PTYPE, 0, 0, 0, 0, 0, 0, 0, 0, 0 }
+#define ICE_PTT_UNUSED_ENTRY(PTYPE) [PTYPE] = { 0, 0, 0, 0, 0, 0, 0, 0, 0 }
/* shorter macros makes the table fit but are terse */
#define ICE_RX_PTYPE_NOF ICE_RX_PTYPE_NOT_FRAG
#define ICE_RX_PTYPE_FRG ICE_RX_PTYPE_FRAG
-/* Lookup table mapping the HW PTYPE to the bit field for decoding */
-static const struct ice_rx_ptype_decoded ice_ptype_lkup[] = {
+/* Lookup table mapping in the 10-bit HW PTYPE to the bit field for decoding */
+static const struct ice_rx_ptype_decoded ice_ptype_lkup[BIT(10)] = {
/* L2 Packet types */
ICE_PTT_UNUSED_ENTRY(0),
ICE_PTT(1, L2, NONE, NOF, NONE, NONE, NOF, NONE, PAY2),
- ICE_PTT(2, L2, NONE, NOF, NONE, NONE, NOF, NONE, NONE),
+ ICE_PTT_UNUSED_ENTRY(2),
ICE_PTT_UNUSED_ENTRY(3),
ICE_PTT_UNUSED_ENTRY(4),
ICE_PTT_UNUSED_ENTRY(5),
/* Non Tunneled IPv6 */
ICE_PTT(88, IP, IPV6, FRG, NONE, NONE, NOF, NONE, PAY3),
ICE_PTT(89, IP, IPV6, NOF, NONE, NONE, NOF, NONE, PAY3),
- ICE_PTT(90, IP, IPV6, NOF, NONE, NONE, NOF, UDP, PAY3),
+ ICE_PTT(90, IP, IPV6, NOF, NONE, NONE, NOF, UDP, PAY4),
ICE_PTT_UNUSED_ENTRY(91),
ICE_PTT(92, IP, IPV6, NOF, NONE, NONE, NOF, TCP, PAY4),
ICE_PTT(93, IP, IPV6, NOF, NONE, NONE, NOF, SCTP, PAY4),
ICE_PTT(153, IP, IPV6, NOF, IP_GRENAT_MAC_VLAN, IPV6, NOF, ICMP, PAY4),
/* unused entries */
- ICE_PTT_UNUSED_ENTRY(154),
- ICE_PTT_UNUSED_ENTRY(155),
- ICE_PTT_UNUSED_ENTRY(156),
- ICE_PTT_UNUSED_ENTRY(157),
- ICE_PTT_UNUSED_ENTRY(158),
- ICE_PTT_UNUSED_ENTRY(159),
-
- ICE_PTT_UNUSED_ENTRY(160),
- ICE_PTT_UNUSED_ENTRY(161),
- ICE_PTT_UNUSED_ENTRY(162),
- ICE_PTT_UNUSED_ENTRY(163),
- ICE_PTT_UNUSED_ENTRY(164),
- ICE_PTT_UNUSED_ENTRY(165),
- ICE_PTT_UNUSED_ENTRY(166),
- ICE_PTT_UNUSED_ENTRY(167),
- ICE_PTT_UNUSED_ENTRY(168),
- ICE_PTT_UNUSED_ENTRY(169),
-
- ICE_PTT_UNUSED_ENTRY(170),
- ICE_PTT_UNUSED_ENTRY(171),
- ICE_PTT_UNUSED_ENTRY(172),
- ICE_PTT_UNUSED_ENTRY(173),
- ICE_PTT_UNUSED_ENTRY(174),
- ICE_PTT_UNUSED_ENTRY(175),
- ICE_PTT_UNUSED_ENTRY(176),
- ICE_PTT_UNUSED_ENTRY(177),
- ICE_PTT_UNUSED_ENTRY(178),
- ICE_PTT_UNUSED_ENTRY(179),
-
- ICE_PTT_UNUSED_ENTRY(180),
- ICE_PTT_UNUSED_ENTRY(181),
- ICE_PTT_UNUSED_ENTRY(182),
- ICE_PTT_UNUSED_ENTRY(183),
- ICE_PTT_UNUSED_ENTRY(184),
- ICE_PTT_UNUSED_ENTRY(185),
- ICE_PTT_UNUSED_ENTRY(186),
- ICE_PTT_UNUSED_ENTRY(187),
- ICE_PTT_UNUSED_ENTRY(188),
- ICE_PTT_UNUSED_ENTRY(189),
-
- ICE_PTT_UNUSED_ENTRY(190),
- ICE_PTT_UNUSED_ENTRY(191),
- ICE_PTT_UNUSED_ENTRY(192),
- ICE_PTT_UNUSED_ENTRY(193),
- ICE_PTT_UNUSED_ENTRY(194),
- ICE_PTT_UNUSED_ENTRY(195),
- ICE_PTT_UNUSED_ENTRY(196),
- ICE_PTT_UNUSED_ENTRY(197),
- ICE_PTT_UNUSED_ENTRY(198),
- ICE_PTT_UNUSED_ENTRY(199),
-
- ICE_PTT_UNUSED_ENTRY(200),
- ICE_PTT_UNUSED_ENTRY(201),
- ICE_PTT_UNUSED_ENTRY(202),
- ICE_PTT_UNUSED_ENTRY(203),
- ICE_PTT_UNUSED_ENTRY(204),
- ICE_PTT_UNUSED_ENTRY(205),
- ICE_PTT_UNUSED_ENTRY(206),
- ICE_PTT_UNUSED_ENTRY(207),
- ICE_PTT_UNUSED_ENTRY(208),
- ICE_PTT_UNUSED_ENTRY(209),
-
- ICE_PTT_UNUSED_ENTRY(210),
- ICE_PTT_UNUSED_ENTRY(211),
- ICE_PTT_UNUSED_ENTRY(212),
- ICE_PTT_UNUSED_ENTRY(213),
- ICE_PTT_UNUSED_ENTRY(214),
- ICE_PTT_UNUSED_ENTRY(215),
- ICE_PTT_UNUSED_ENTRY(216),
- ICE_PTT_UNUSED_ENTRY(217),
- ICE_PTT_UNUSED_ENTRY(218),
- ICE_PTT_UNUSED_ENTRY(219),
-
- ICE_PTT_UNUSED_ENTRY(220),
- ICE_PTT_UNUSED_ENTRY(221),
- ICE_PTT_UNUSED_ENTRY(222),
- ICE_PTT_UNUSED_ENTRY(223),
- ICE_PTT_UNUSED_ENTRY(224),
- ICE_PTT_UNUSED_ENTRY(225),
- ICE_PTT_UNUSED_ENTRY(226),
- ICE_PTT_UNUSED_ENTRY(227),
- ICE_PTT_UNUSED_ENTRY(228),
- ICE_PTT_UNUSED_ENTRY(229),
-
- ICE_PTT_UNUSED_ENTRY(230),
- ICE_PTT_UNUSED_ENTRY(231),
- ICE_PTT_UNUSED_ENTRY(232),
- ICE_PTT_UNUSED_ENTRY(233),
- ICE_PTT_UNUSED_ENTRY(234),
- ICE_PTT_UNUSED_ENTRY(235),
- ICE_PTT_UNUSED_ENTRY(236),
- ICE_PTT_UNUSED_ENTRY(237),
- ICE_PTT_UNUSED_ENTRY(238),
- ICE_PTT_UNUSED_ENTRY(239),
-
- ICE_PTT_UNUSED_ENTRY(240),
- ICE_PTT_UNUSED_ENTRY(241),
- ICE_PTT_UNUSED_ENTRY(242),
- ICE_PTT_UNUSED_ENTRY(243),
- ICE_PTT_UNUSED_ENTRY(244),
- ICE_PTT_UNUSED_ENTRY(245),
- ICE_PTT_UNUSED_ENTRY(246),
- ICE_PTT_UNUSED_ENTRY(247),
- ICE_PTT_UNUSED_ENTRY(248),
- ICE_PTT_UNUSED_ENTRY(249),
-
- ICE_PTT_UNUSED_ENTRY(250),
- ICE_PTT_UNUSED_ENTRY(251),
- ICE_PTT_UNUSED_ENTRY(252),
- ICE_PTT_UNUSED_ENTRY(253),
- ICE_PTT_UNUSED_ENTRY(254),
- ICE_PTT_UNUSED_ENTRY(255),
+ [154 ... 1023] = { 0, 0, 0, 0, 0, 0, 0, 0, 0 }
};
static inline struct ice_rx_ptype_decoded ice_decode_rx_desc_ptype(u16 ptype)
if (!vsi->q_vectors)
goto err_vectors;
+ vsi->af_xdp_zc_qps = bitmap_zalloc(max_t(int, vsi->alloc_txq, vsi->alloc_rxq), GFP_KERNEL);
+ if (!vsi->af_xdp_zc_qps)
+ goto err_zc_qps;
+
return 0;
+err_zc_qps:
+ devm_kfree(dev, vsi->q_vectors);
err_vectors:
devm_kfree(dev, vsi->rxq_map);
err_rxq_map:
switch (vsi->type) {
case ICE_VSI_PF:
- vsi->alloc_txq = min3(pf->num_lan_msix,
- ice_get_avail_txq_count(pf),
- (u16)num_online_cpus());
if (vsi->req_txq) {
vsi->alloc_txq = vsi->req_txq;
vsi->num_txq = vsi->req_txq;
+ } else {
+ vsi->alloc_txq = min3(pf->num_lan_msix,
+ ice_get_avail_txq_count(pf),
+ (u16)num_online_cpus());
}
pf->num_lan_tx = vsi->alloc_txq;
if (!test_bit(ICE_FLAG_RSS_ENA, pf->flags)) {
vsi->alloc_rxq = 1;
} else {
- vsi->alloc_rxq = min3(pf->num_lan_msix,
- ice_get_avail_rxq_count(pf),
- (u16)num_online_cpus());
if (vsi->req_rxq) {
vsi->alloc_rxq = vsi->req_rxq;
vsi->num_rxq = vsi->req_rxq;
+ } else {
+ vsi->alloc_rxq = min3(pf->num_lan_msix,
+ ice_get_avail_rxq_count(pf),
+ (u16)num_online_cpus());
}
}
break;
case ICE_VSI_VF:
vf = &pf->vf[vsi->vf_id];
+ if (vf->num_req_qs)
+ vf->num_vf_qs = vf->num_req_qs;
vsi->alloc_txq = vf->num_vf_qs;
vsi->alloc_rxq = vf->num_vf_qs;
/* pf->num_msix_per_vf includes (VF miscellaneous vector +
dev = ice_pf_to_dev(pf);
+ if (vsi->af_xdp_zc_qps) {
+ bitmap_free(vsi->af_xdp_zc_qps);
+ vsi->af_xdp_zc_qps = NULL;
+ }
/* free the ring and vector containers */
if (vsi->q_vectors) {
devm_kfree(dev, vsi->q_vectors);
}
/**
+ * ice_is_aux_ena
+ * @pf: pointer to the PF struct
+ *
+ * returns true if AUX devices/drivers are supported, false otherwise
+ */
+bool ice_is_aux_ena(struct ice_pf *pf)
+{
+ return test_bit(ICE_FLAG_AUX_ENA, pf->flags);
+}
+
+/**
* ice_vsi_clean_rss_flow_fld - Delete RSS configuration
* @vsi: the VSI being cleaned up
*
num_q_vectors = vsi->num_q_vectors;
/* reserve slots from OS requested IRQs */
if (vsi->type == ICE_VSI_CTRL && vsi->vf_id != ICE_INVAL_VFID) {
- struct ice_vf *vf;
int i;
ice_for_each_vf(pf, i) {
- vf = &pf->vf[i];
+ struct ice_vf *vf = &pf->vf[i];
+
if (i != vsi->vf_id && vf->ctrl_vsi_idx != ICE_NO_VSI) {
base = pf->vsi[vf->ctrl_vsi_idx]->base_vector;
break;
ring->reg_idx = vsi->txq_map[i];
ring->ring_active = false;
ring->vsi = vsi;
+ ring->tx_tstamps = &pf->ptp.port.tx;
ring->dev = dev;
ring->count = vsi->num_tx_desc;
WRITE_ONCE(vsi->tx_rings[i], ring);
* @pf_q: index of the Rx queue in the PF's queue space
* @rxdid: flexible descriptor RXDID
* @prio: priority for the RXDID for this queue
+ * @ena_ts: true to enable timestamp and false to disable timestamp
*/
void
-ice_write_qrxflxp_cntxt(struct ice_hw *hw, u16 pf_q, u32 rxdid, u32 prio)
+ice_write_qrxflxp_cntxt(struct ice_hw *hw, u16 pf_q, u32 rxdid, u32 prio,
+ bool ena_ts)
{
int regval = rd32(hw, QRXFLXP_CNTXT(pf_q));
regval |= (prio << QRXFLXP_CNTXT_RXDID_PRIO_S) &
QRXFLXP_CNTXT_RXDID_PRIO_M;
+ if (ena_ts)
+ /* Enable TimeSync on this queue */
+ regval |= QRXFLXP_CNTXT_TS_M;
+
wr32(hw, QRXFLXP_CNTXT(pf_q), regval);
}
+int ice_vsi_cfg_single_rxq(struct ice_vsi *vsi, u16 q_idx)
+{
+ if (q_idx >= vsi->num_rxq)
+ return -EINVAL;
+
+ return ice_vsi_cfg_rxq(vsi->rx_rings[q_idx]);
+}
+
+int ice_vsi_cfg_single_txq(struct ice_vsi *vsi, struct ice_ring **tx_rings, u16 q_idx)
+{
+ struct ice_aqc_add_tx_qgrp *qg_buf;
+ int err;
+
+ if (q_idx >= vsi->alloc_txq || !tx_rings || !tx_rings[q_idx])
+ return -EINVAL;
+
+ qg_buf = kzalloc(struct_size(qg_buf, txqs, 1), GFP_KERNEL);
+ if (!qg_buf)
+ return -ENOMEM;
+
+ qg_buf->num_txqs = 1;
+
+ err = ice_vsi_cfg_txq(vsi, tx_rings[q_idx], qg_buf);
+ kfree(qg_buf);
+ return err;
+}
+
/**
* ice_vsi_cfg_rxqs - Configure the VSI for Rx
* @vsi: the VSI being configured
ice_vsi_cfg_frame_size(vsi);
setup_rings:
/* set up individual rings */
- for (i = 0; i < vsi->num_rxq; i++) {
- int err;
+ ice_for_each_rxq(vsi, i) {
+ int err = ice_vsi_cfg_rxq(vsi->rx_rings[i]);
- err = ice_setup_rx_ctx(vsi->rx_rings[i]);
- if (err) {
- dev_err(ice_pf_to_dev(vsi->back), "ice_setup_rx_ctx failed for RxQ %d, err %d\n",
- i, err);
+ if (err)
return err;
- }
}
return 0;
* ice_vsi_cfg_txqs - Configure the VSI for Tx
* @vsi: the VSI being configured
* @rings: Tx ring array to be configured
+ * @count: number of Tx ring array elements
*
* Return 0 on success and a negative value on error
* Configure the Tx VSI for operation.
*/
static int
-ice_vsi_cfg_txqs(struct ice_vsi *vsi, struct ice_ring **rings)
+ice_vsi_cfg_txqs(struct ice_vsi *vsi, struct ice_ring **rings, u16 count)
{
struct ice_aqc_add_tx_qgrp *qg_buf;
u16 q_idx = 0;
qg_buf->num_txqs = 1;
- for (q_idx = 0; q_idx < vsi->num_txq; q_idx++) {
+ for (q_idx = 0; q_idx < count; q_idx++) {
err = ice_vsi_cfg_txq(vsi, rings[q_idx], qg_buf);
if (err)
goto err_cfg_txqs;
*/
int ice_vsi_cfg_lan_txqs(struct ice_vsi *vsi)
{
- return ice_vsi_cfg_txqs(vsi, vsi->tx_rings);
+ return ice_vsi_cfg_txqs(vsi, vsi->tx_rings, vsi->num_txq);
}
/**
int ret;
int i;
- ret = ice_vsi_cfg_txqs(vsi, vsi->xdp_rings);
+ ret = ice_vsi_cfg_txqs(vsi, vsi->xdp_rings, vsi->num_xdp_txq);
if (ret)
return ret;
* @rst_src: reset source
* @rel_vmvf_num: Relative ID of VF/VM
* @rings: Tx ring array to be stopped
+ * @count: number of Tx ring array elements
*/
static int
ice_vsi_stop_tx_rings(struct ice_vsi *vsi, enum ice_disq_rst_src rst_src,
- u16 rel_vmvf_num, struct ice_ring **rings)
+ u16 rel_vmvf_num, struct ice_ring **rings, u16 count)
{
u16 q_idx;
if (vsi->num_txq > ICE_LAN_TXQ_MAX_QDIS)
return -EINVAL;
- for (q_idx = 0; q_idx < vsi->num_txq; q_idx++) {
+ for (q_idx = 0; q_idx < count; q_idx++) {
struct ice_txq_meta txq_meta = { };
int status;
ice_vsi_stop_lan_tx_rings(struct ice_vsi *vsi, enum ice_disq_rst_src rst_src,
u16 rel_vmvf_num)
{
- return ice_vsi_stop_tx_rings(vsi, rst_src, rel_vmvf_num, vsi->tx_rings);
+ return ice_vsi_stop_tx_rings(vsi, rst_src, rel_vmvf_num, vsi->tx_rings, vsi->num_txq);
}
/**
*/
int ice_vsi_stop_xdp_tx_rings(struct ice_vsi *vsi)
{
- return ice_vsi_stop_tx_rings(vsi, ICE_NO_RESET, 0, vsi->xdp_rings);
+ return ice_vsi_stop_tx_rings(vsi, ICE_NO_RESET, 0, vsi->xdp_rings, vsi->num_xdp_txq);
}
/**
}
if (status)
- dev_err(dev, "Fail %s %s LLDP rule on VSI %i error: %s\n",
+ dev_dbg(dev, "Fail %s %s LLDP rule on VSI %i error: %s\n",
create ? "adding" : "removing", tx ? "TX" : "RX",
vsi->vsi_num, ice_stat_str(status));
}
* cleared in the same manner.
*/
if (vsi->type == ICE_VSI_CTRL && vsi->vf_id != ICE_INVAL_VFID) {
- struct ice_vf *vf;
int i;
ice_for_each_vf(pf, i) {
- vf = &pf->vf[i];
+ struct ice_vf *vf = &pf->vf[i];
+
if (i != vsi->vf_id && vf->ctrl_vsi_idx != ICE_NO_VSI)
break;
}
test_bit(ICE_GLOBR_REQ, state);
}
+/**
+ * ice_wait_for_reset - Wait for driver to finish reset and rebuild
+ * @pf: pointer to the PF structure
+ * @timeout: length of time to wait, in jiffies
+ *
+ * Wait (sleep) for a short time until the driver finishes cleaning up from
+ * a device reset. The caller must be able to sleep. Use this to delay
+ * operations that could fail while the driver is cleaning up after a device
+ * reset.
+ *
+ * Returns 0 on success, -EBUSY if the reset is not finished within the
+ * timeout, and -ERESTARTSYS if the thread was interrupted.
+ */
+int ice_wait_for_reset(struct ice_pf *pf, unsigned long timeout)
+{
+ long ret;
+
+ ret = wait_event_interruptible_timeout(pf->reset_wait_queue,
+ !ice_is_reset_in_progress(pf->state),
+ timeout);
+ if (ret < 0)
+ return ret;
+ else if (!ret)
+ return -EBUSY;
+ else
+ return 0;
+}
+
#ifdef CONFIG_DCB
/**
* ice_vsi_update_q_map - update our copy of the VSI info with new queue map
case ICE_ERR_DOES_NOT_EXIST:
return -ENOENT;
case ICE_ERR_OUT_OF_RANGE:
- return -ENOTTY;
+ case ICE_ERR_AQ_ERROR:
+ case ICE_ERR_AQ_TIMEOUT:
+ case ICE_ERR_AQ_EMPTY:
+ case ICE_ERR_AQ_FW_CRITICAL:
+ return -EIO;
case ICE_ERR_PARAM:
+ case ICE_ERR_INVAL_SIZE:
return -EINVAL;
case ICE_ERR_NO_MEMORY:
return -ENOMEM;
case ICE_ERR_MAX_LIMIT:
return -EAGAIN;
+ case ICE_ERR_RESET_ONGOING:
+ return -EBUSY;
+ case ICE_ERR_AQ_FULL:
+ return -ENOSPC;
default:
return -EINVAL;
}
void ice_update_eth_stats(struct ice_vsi *vsi);
+int ice_vsi_cfg_single_rxq(struct ice_vsi *vsi, u16 q_idx);
+
+int ice_vsi_cfg_single_txq(struct ice_vsi *vsi, struct ice_ring **tx_rings, u16 q_idx);
+
int ice_vsi_cfg_rxqs(struct ice_vsi *vsi);
int ice_vsi_cfg_lan_txqs(struct ice_vsi *vsi);
int ice_vsi_rebuild(struct ice_vsi *vsi, bool init_vsi);
bool ice_is_reset_in_progress(unsigned long *state);
+int ice_wait_for_reset(struct ice_pf *pf, unsigned long timeout);
void
-ice_write_qrxflxp_cntxt(struct ice_hw *hw, u16 pf_q, u32 rxdid, u32 prio);
+ice_write_qrxflxp_cntxt(struct ice_hw *hw, u16 pf_q, u32 rxdid, u32 prio,
+ bool ena_ts);
void ice_vsi_dis_irq(struct ice_vsi *vsi);
ice_vsi_cfg_mac_fltr(struct ice_vsi *vsi, const u8 *macaddr, bool set);
bool ice_is_safe_mode(struct ice_pf *pf);
-
+bool ice_is_aux_ena(struct ice_pf *pf);
bool ice_is_dflt_vsi_in_use(struct ice_sw *sw);
bool ice_is_vsi_dflt_vsi(struct ice_sw *sw, struct ice_vsi *vsi);
#include "ice_dcb_lib.h"
#include "ice_dcb_nl.h"
#include "ice_devlink.h"
+/* Including ice_trace.h with CREATE_TRACE_POINTS defined will generate the
+ * ice tracepoint functions. This must be done exactly once across the
+ * ice driver.
+ */
+#define CREATE_TRACE_POINTS
+#include "ice_trace.h"
#define DRV_SUMMARY "Intel(R) Ethernet Connection E800 Series Linux Driver"
static const char ice_driver_string[] = DRV_SUMMARY;
MODULE_PARM_DESC(debug, "netif level (0=none,...,16=all)");
#endif /* !CONFIG_DYNAMIC_DEBUG */
+static DEFINE_IDA(ice_aux_ida);
+
static struct workqueue_struct *ice_wq;
static const struct net_device_ops ice_netdev_safe_mode_ops;
static const struct net_device_ops ice_netdev_ops;
if (test_bit(ICE_PREPARED_FOR_RESET, pf->state))
return;
+ ice_unplug_aux_dev(pf);
+
/* Notify VFs of impending reset */
if (ice_check_sq_alive(hw, &hw->mailboxq))
ice_vc_notify_reset(pf);
/* disable the VSIs and their queues that are not already DOWN */
ice_pf_dis_all_vsi(pf, false);
+ if (test_bit(ICE_FLAG_PTP_SUPPORTED, pf->flags))
+ ice_ptp_release(pf);
+
if (hw->port_info)
ice_sched_clear_port(hw->port_info);
clear_bit(ICE_PFR_REQ, pf->state);
clear_bit(ICE_CORER_REQ, pf->state);
clear_bit(ICE_GLOBR_REQ, pf->state);
+ wake_up(&pf->reset_wait_queue);
return;
}
ice_rebuild(pf, reset_type);
clear_bit(ICE_PREPARED_FOR_RESET, pf->state);
clear_bit(ICE_PFR_REQ, pf->state);
+ wake_up(&pf->reset_wait_queue);
ice_reset_all_vfs(pf, true);
}
}
clear_bit(ICE_PFR_REQ, pf->state);
clear_bit(ICE_CORER_REQ, pf->state);
clear_bit(ICE_GLOBR_REQ, pf->state);
+ wake_up(&pf->reset_wait_queue);
ice_reset_all_vfs(pf, true);
}
}
/**
+ * ice_check_module_power
+ * @pf: pointer to PF struct
+ * @link_cfg_err: bitmap from the link info structure
+ *
+ * check module power level returned by a previous call to aq_get_link_info
+ * and print error messages if module power level is not supported
+ */
+static void ice_check_module_power(struct ice_pf *pf, u8 link_cfg_err)
+{
+ /* if module power level is supported, clear the flag */
+ if (!(link_cfg_err & (ICE_AQ_LINK_INVAL_MAX_POWER_LIMIT |
+ ICE_AQ_LINK_MODULE_POWER_UNSUPPORTED))) {
+ clear_bit(ICE_FLAG_MOD_POWER_UNSUPPORTED, pf->flags);
+ return;
+ }
+
+ /* if ICE_FLAG_MOD_POWER_UNSUPPORTED was previously set and the
+ * above block didn't clear this bit, there's nothing to do
+ */
+ if (test_bit(ICE_FLAG_MOD_POWER_UNSUPPORTED, pf->flags))
+ return;
+
+ if (link_cfg_err & ICE_AQ_LINK_INVAL_MAX_POWER_LIMIT) {
+ dev_err(ice_pf_to_dev(pf), "The installed module is incompatible with the device's NVM image. Cannot start link\n");
+ set_bit(ICE_FLAG_MOD_POWER_UNSUPPORTED, pf->flags);
+ } else if (link_cfg_err & ICE_AQ_LINK_MODULE_POWER_UNSUPPORTED) {
+ dev_err(ice_pf_to_dev(pf), "The module's power requirements exceed the device's power supply. Cannot start link\n");
+ set_bit(ICE_FLAG_MOD_POWER_UNSUPPORTED, pf->flags);
+ }
+}
+
+/**
* ice_link_event - process the link event
* @pf: PF that the link event is associated with
* @pi: port_info for the port that the link event is associated with
pi->lport, ice_stat_str(status),
ice_aq_str(pi->hw->adminq.sq_last_status));
+ ice_check_module_power(pf, pi->phy.link_info.link_cfg_err);
+
/* Check if the link state is up after updating link info, and treat
* this event as an UP event since the link is actually UP now.
*/
cq = &hw->adminq;
qtype = "Admin";
break;
+ case ICE_CTL_Q_SB:
+ cq = &hw->sbq;
+ qtype = "Sideband";
+ break;
case ICE_CTL_Q_MAILBOX:
cq = &hw->mailboxq;
qtype = "Mailbox";
}
/**
+ * ice_clean_sbq_subtask - clean the Sideband Queue rings
+ * @pf: board private structure
+ */
+static void ice_clean_sbq_subtask(struct ice_pf *pf)
+{
+ struct ice_hw *hw = &pf->hw;
+
+ /* Nothing to do here if sideband queue is not supported */
+ if (!ice_is_sbq_supported(hw)) {
+ clear_bit(ICE_SIDEBANDQ_EVENT_PENDING, pf->state);
+ return;
+ }
+
+ if (!test_bit(ICE_SIDEBANDQ_EVENT_PENDING, pf->state))
+ return;
+
+ if (__ice_clean_ctrlq(pf, ICE_CTL_Q_SB))
+ return;
+
+ clear_bit(ICE_SIDEBANDQ_EVENT_PENDING, pf->state);
+
+ if (ice_ctrlq_pending(hw, &hw->sbq))
+ __ice_clean_ctrlq(pf, ICE_CTL_Q_SB);
+
+ ice_flush(hw);
+}
+
+/**
* ice_service_task_schedule - schedule the service task to wake up
* @pf: board private structure
*
if (err)
return;
+ ice_check_module_power(pf, pi->phy.link_info.link_cfg_err);
+
if (pi->phy.link_info.link_info & ICE_AQ_MEDIA_AVAILABLE) {
if (!test_bit(ICE_PHY_INIT_COMPLETE, pf->state))
ice_init_phy_user_cfg(pi);
ice_process_vflr_event(pf);
ice_clean_mailboxq_subtask(pf);
+ ice_clean_sbq_subtask(pf);
ice_sync_arfs_fltrs(pf);
ice_flush_fdir_ctx(pf);
test_bit(ICE_VFLR_EVENT_PENDING, pf->state) ||
test_bit(ICE_MAILBOXQ_EVENT_PENDING, pf->state) ||
test_bit(ICE_FD_VF_FLUSH_CTX, pf->state) ||
+ test_bit(ICE_SIDEBANDQ_EVENT_PENDING, pf->state) ||
test_bit(ICE_ADMINQ_EVENT_PENDING, pf->state))
mod_timer(&pf->serv_tmr, jiffies);
}
hw->mailboxq.num_sq_entries = ICE_MBXSQ_LEN;
hw->mailboxq.rq_buf_size = ICE_MBXQ_MAX_BUF_LEN;
hw->mailboxq.sq_buf_size = ICE_MBXQ_MAX_BUF_LEN;
+ hw->sbq.num_rq_entries = ICE_SBQ_LEN;
+ hw->sbq.num_sq_entries = ICE_SBQ_LEN;
+ hw->sbq.rq_buf_size = ICE_SBQ_MAX_BUF_LEN;
+ hw->sbq.sq_buf_size = ICE_SBQ_MAX_BUF_LEN;
}
/**
return -EBUSY;
}
+ ice_unplug_aux_dev(pf);
+
switch (reset) {
case ICE_RESET_PFR:
set_bit(ICE_PFR_REQ, pf->state);
}
/**
+ * ice_xdp_safe_mode - XDP handler for safe mode
+ * @dev: netdevice
+ * @xdp: XDP command
+ */
+static int ice_xdp_safe_mode(struct net_device __always_unused *dev,
+ struct netdev_bpf *xdp)
+{
+ NL_SET_ERR_MSG_MOD(xdp->extack,
+ "Please provide working DDP firmware package in order to use XDP\n"
+ "Refer to Documentation/networking/device_drivers/ethernet/intel/ice.rst");
+ return -EOPNOTSUPP;
+}
+
+/**
* ice_xdp - implements XDP handler
* @dev: netdevice
* @xdp: XDP command
PFINT_OICR_PCI_EXCEPTION_M |
PFINT_OICR_VFLR_M |
PFINT_OICR_HMC_ERR_M |
+ PFINT_OICR_PE_PUSH_M |
PFINT_OICR_PE_CRITERR_M);
wr32(hw, PFINT_OICR_ENA, val);
dev = ice_pf_to_dev(pf);
set_bit(ICE_ADMINQ_EVENT_PENDING, pf->state);
set_bit(ICE_MAILBOXQ_EVENT_PENDING, pf->state);
+ set_bit(ICE_SIDEBANDQ_EVENT_PENDING, pf->state);
oicr = rd32(hw, PFINT_OICR);
ena_mask = rd32(hw, PFINT_OICR_ENA);
/* If a reset cycle isn't already in progress, we set a bit in
* pf->state so that the service task can start a reset/rebuild.
- * We also make note of which reset happened so that peer
- * devices/drivers can be informed.
*/
if (!test_and_set_bit(ICE_RESET_OICR_RECV, pf->state)) {
if (reset == ICE_RESET_CORER)
}
}
- if (oicr & PFINT_OICR_HMC_ERR_M) {
- ena_mask &= ~PFINT_OICR_HMC_ERR_M;
- dev_dbg(dev, "HMC Error interrupt - info 0x%x, data 0x%x\n",
- rd32(hw, PFHMC_ERRORINFO),
- rd32(hw, PFHMC_ERRORDATA));
+ if (oicr & PFINT_OICR_TSYN_TX_M) {
+ ena_mask &= ~PFINT_OICR_TSYN_TX_M;
+ ice_ptp_process_ts(pf);
+ }
+
+ if (oicr & PFINT_OICR_TSYN_EVNT_M) {
+ u8 tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned;
+ u32 gltsyn_stat = rd32(hw, GLTSYN_STAT(tmr_idx));
+
+ /* Save EVENTs from GTSYN register */
+ pf->ptp.ext_ts_irq |= gltsyn_stat & (GLTSYN_STAT_EVENT0_M |
+ GLTSYN_STAT_EVENT1_M |
+ GLTSYN_STAT_EVENT2_M);
+ ena_mask &= ~PFINT_OICR_TSYN_EVNT_M;
+ kthread_queue_work(pf->ptp.kworker, &pf->ptp.extts_work);
+ }
+
+#define ICE_AUX_CRIT_ERR (PFINT_OICR_PE_CRITERR_M | PFINT_OICR_HMC_ERR_M | PFINT_OICR_PE_PUSH_M)
+ if (oicr & ICE_AUX_CRIT_ERR) {
+ struct iidc_event *event;
+
+ ena_mask &= ~ICE_AUX_CRIT_ERR;
+ event = kzalloc(sizeof(*event), GFP_KERNEL);
+ if (event) {
+ set_bit(IIDC_EVENT_CRIT_ERR, event->type);
+ /* report the entire OICR value to AUX driver */
+ event->reg = oicr;
+ ice_send_event_to_aux(pf, event);
+ kfree(event);
+ }
}
/* Report any remaining unexpected interrupts */
/* If a critical error is pending there is no choice but to
* reset the device.
*/
- if (oicr & (PFINT_OICR_PE_CRITERR_M |
- PFINT_OICR_PCI_EXCEPTION_M |
+ if (oicr & (PFINT_OICR_PCI_EXCEPTION_M |
PFINT_OICR_ECC_ERR_M)) {
set_bit(ICE_PFR_REQ, pf->state);
ice_service_task_schedule(pf);
wr32(hw, PFINT_MBX_CTL,
rd32(hw, PFINT_MBX_CTL) & ~PFINT_MBX_CTL_CAUSE_ENA_M);
+ wr32(hw, PFINT_SB_CTL,
+ rd32(hw, PFINT_SB_CTL) & ~PFINT_SB_CTL_CAUSE_ENA_M);
+
/* disable Control queue Interrupt causes */
wr32(hw, PFINT_OICR_CTL,
rd32(hw, PFINT_OICR_CTL) & ~PFINT_OICR_CTL_CAUSE_ENA_M);
PFINT_MBX_CTL_CAUSE_ENA_M);
wr32(hw, PFINT_MBX_CTL, val);
+ /* This enables Sideband queue Interrupt causes */
+ val = ((reg_idx & PFINT_SB_CTL_MSIX_INDX_M) |
+ PFINT_SB_CTL_CAUSE_ENA_M);
+ wr32(hw, PFINT_SB_CTL, val);
+
ice_flush(hw);
}
*/
static int ice_cfg_netdev(struct ice_vsi *vsi)
{
- struct ice_pf *pf = vsi->back;
struct ice_netdev_priv *np;
struct net_device *netdev;
u8 mac_addr[ETH_ALEN];
ice_set_ops(netdev);
if (vsi->type == ICE_VSI_PF) {
- SET_NETDEV_DEV(netdev, ice_pf_to_dev(pf));
+ SET_NETDEV_DEV(netdev, ice_pf_to_dev(vsi->back));
ether_addr_copy(mac_addr, vsi->port_info->mac.perm_addr);
ether_addr_copy(netdev->dev_addr, mac_addr);
ether_addr_copy(netdev->perm_addr, mac_addr);
bitmap_free(pf->avail_rxqs);
pf->avail_rxqs = NULL;
}
+
+ if (pf->ptp.clock)
+ ptp_clock_unregister(pf->ptp.clock);
}
/**
{
struct ice_hw_func_caps *func_caps = &pf->hw.func_caps;
+ clear_bit(ICE_FLAG_RDMA_ENA, pf->flags);
+ clear_bit(ICE_FLAG_AUX_ENA, pf->flags);
+ if (func_caps->common_cap.rdma) {
+ set_bit(ICE_FLAG_RDMA_ENA, pf->flags);
+ set_bit(ICE_FLAG_AUX_ENA, pf->flags);
+ }
clear_bit(ICE_FLAG_DCB_CAPABLE, pf->flags);
if (func_caps->common_cap.dcb)
set_bit(ICE_FLAG_DCB_CAPABLE, pf->flags);
func_caps->fd_fltr_best_effort);
}
+ clear_bit(ICE_FLAG_PTP_SUPPORTED, pf->flags);
+ if (func_caps->common_cap.ieee_1588)
+ set_bit(ICE_FLAG_PTP_SUPPORTED, pf->flags);
+
pf->max_pf_txqs = func_caps->common_cap.num_txq;
pf->max_pf_rxqs = func_caps->common_cap.num_rxq;
}
spin_lock_init(&pf->aq_wait_lock);
init_waitqueue_head(&pf->aq_wait_queue);
+ init_waitqueue_head(&pf->reset_wait_queue);
+
/* setup service timer and periodic service task */
timer_setup(&pf->serv_tmr, ice_service_timer, 0);
pf->serv_tmr_period = HZ;
*/
static int ice_ena_msix_range(struct ice_pf *pf)
{
- int v_left, v_actual, v_other, v_budget = 0;
+ int num_cpus, v_left, v_actual, v_other, v_budget = 0;
struct device *dev = ice_pf_to_dev(pf);
int needed, err, i;
v_left = pf->hw.func_caps.common_cap.num_msix_vectors;
+ num_cpus = num_online_cpus();
/* reserve for LAN miscellaneous handler */
needed = ICE_MIN_LAN_OICR_MSIX;
v_other = v_budget;
/* reserve vectors for LAN traffic */
- needed = min_t(int, num_online_cpus(), v_left);
+ needed = num_cpus;
if (v_left < needed)
goto no_hw_vecs_left_err;
pf->num_lan_msix = needed;
v_budget += needed;
v_left -= needed;
+ /* reserve vectors for RDMA auxiliary driver */
+ if (test_bit(ICE_FLAG_RDMA_ENA, pf->flags)) {
+ needed = num_cpus + ICE_RDMA_NUM_AEQ_MSIX;
+ if (v_left < needed)
+ goto no_hw_vecs_left_err;
+ pf->num_rdma_msix = needed;
+ v_budget += needed;
+ v_left -= needed;
+ }
+
pf->msix_entries = devm_kcalloc(dev, v_budget,
sizeof(*pf->msix_entries), GFP_KERNEL);
if (!pf->msix_entries) {
err = -ERANGE;
goto msix_err;
} else {
- int v_traffic = v_actual - v_other;
+ int v_remain = v_actual - v_other;
+ int v_rdma = 0, v_min_rdma = 0;
+
+ if (test_bit(ICE_FLAG_RDMA_ENA, pf->flags)) {
+ /* Need at least 1 interrupt in addition to
+ * AEQ MSIX
+ */
+ v_rdma = ICE_RDMA_NUM_AEQ_MSIX + 1;
+ v_min_rdma = ICE_MIN_RDMA_MSIX;
+ }
if (v_actual == ICE_MIN_MSIX ||
- v_traffic < ICE_MIN_LAN_TXRX_MSIX)
+ v_remain < ICE_MIN_LAN_TXRX_MSIX + v_min_rdma) {
+ dev_warn(dev, "Not enough MSI-X vectors to support RDMA.\n");
+ clear_bit(ICE_FLAG_RDMA_ENA, pf->flags);
+
+ pf->num_rdma_msix = 0;
pf->num_lan_msix = ICE_MIN_LAN_TXRX_MSIX;
- else
- pf->num_lan_msix = v_traffic;
+ } else if ((v_remain < ICE_MIN_LAN_TXRX_MSIX + v_rdma) ||
+ (v_remain - v_rdma < v_rdma)) {
+ /* Support minimum RDMA and give remaining
+ * vectors to LAN MSIX
+ */
+ pf->num_rdma_msix = v_min_rdma;
+ pf->num_lan_msix = v_remain - v_min_rdma;
+ } else {
+ /* Split remaining MSIX with RDMA after
+ * accounting for AEQ MSIX
+ */
+ pf->num_rdma_msix = (v_remain - ICE_RDMA_NUM_AEQ_MSIX) / 2 +
+ ICE_RDMA_NUM_AEQ_MSIX;
+ pf->num_lan_msix = v_remain - pf->num_rdma_msix;
+ }
dev_notice(dev, "Enabled %d MSI-X vectors for LAN traffic.\n",
pf->num_lan_msix);
+
+ if (test_bit(ICE_FLAG_RDMA_ENA, pf->flags))
+ dev_notice(dev, "Enabled %d MSI-X vectors for RDMA.\n",
+ pf->num_rdma_msix);
}
}
needed, v_left);
err = -ERANGE;
exit_err:
+ pf->num_rdma_msix = 0;
pf->num_lan_msix = 0;
return err;
}
ice_init_link_dflt_override(pf->hw.port_info);
+ ice_check_module_power(pf, pf->hw.port_info->phy.link_info.link_cfg_err);
+
/* if media available, initialize PHY settings */
if (pf->hw.port_info->phy.link_info.link_info &
ICE_AQ_MEDIA_AVAILABLE) {
}
/* initialize DDP driven features */
+ if (test_bit(ICE_FLAG_PTP_SUPPORTED, pf->flags))
+ ice_ptp_init(pf);
/* Note: Flow director init failure is non-fatal to load */
if (ice_init_fdir(pf))
/* ready to go, so clear down state bit */
clear_bit(ICE_DOWN, pf->state);
+ if (ice_is_aux_ena(pf)) {
+ pf->aux_idx = ida_alloc(&ice_aux_ida, GFP_KERNEL);
+ if (pf->aux_idx < 0) {
+ dev_err(dev, "Failed to allocate device ID for AUX driver\n");
+ err = -ENOMEM;
+ goto err_netdev_reg;
+ }
+
+ err = ice_init_rdma(pf);
+ if (err) {
+ dev_err(dev, "Failed to initialize RDMA: %d\n", err);
+ err = -EIO;
+ goto err_init_aux_unroll;
+ }
+ } else {
+ dev_warn(dev, "RDMA is not supported on this device\n");
+ }
+
return 0;
+err_init_aux_unroll:
+ pf->adev = NULL;
+ ida_free(&ice_aux_ida, pf->aux_idx);
err_netdev_reg:
err_send_version_unroll:
ice_vsi_release_all(pf);
ice_free_vfs(pf);
}
- set_bit(ICE_DOWN, pf->state);
ice_service_task_stop(pf);
ice_aq_cancel_waiting_tasks(pf);
+ ice_unplug_aux_dev(pf);
+ ida_free(&ice_aux_ida, pf->aux_idx);
+ set_bit(ICE_DOWN, pf->state);
mutex_destroy(&(&pf->hw)->fdir_fltr_lock);
ice_deinit_lag(pf);
+ if (test_bit(ICE_FLAG_PTP_SUPPORTED, pf->flags))
+ ice_ptp_release(pf);
if (!ice_is_safe_mode(pf))
ice_remove_arfs(pf);
ice_setup_mc_magic_wake(pf);
*/
disabled = ice_service_task_stop(pf);
+ ice_unplug_aux_dev(pf);
+
/* Already suspended?, then there is nothing to do */
if (test_and_set_bit(ICE_SUSPENDED, pf->state)) {
if (!disabled)
itr = tx_profile[dim->profile_ix].itr;
intrl = tx_profile[dim->profile_ix].intrl;
+ ice_trace(tx_dim_work, q_vector, dim);
ice_write_itr(rc, itr);
ice_write_intrl(q_vector, intrl);
itr = rx_profile[dim->profile_ix].itr;
intrl = rx_profile[dim->profile_ix].intrl;
+ ice_trace(rx_dim_work, q_vector, dim);
ice_write_itr(rc, itr);
ice_write_intrl(q_vector, intrl);
static void ice_update_vsi_ring_stats(struct ice_vsi *vsi)
{
struct rtnl_link_stats64 *vsi_stats = &vsi->net_stats;
- struct ice_ring *ring;
u64 pkts, bytes;
int i;
/* update Rx rings counters */
ice_for_each_rxq(vsi, i) {
- ring = READ_ONCE(vsi->rx_rings[i]);
+ struct ice_ring *ring = READ_ONCE(vsi->rx_rings[i]);
+
ice_fetch_u64_stats_per_ring(ring, &pkts, &bytes);
vsi_stats->rx_packets += pkts;
vsi_stats->rx_bytes += bytes;
ice_clear_pxe_mode(hw);
+ ret = ice_init_nvm(hw);
+ if (ret) {
+ dev_err(dev, "ice_init_nvm failed %s\n", ice_stat_str(ret));
+ goto err_init_ctrlq;
+ }
+
ret = ice_get_caps(hw);
if (ret) {
dev_err(dev, "ice_get_caps failed %s\n", ice_stat_str(ret));
if (test_bit(ICE_FLAG_DCB_ENA, pf->flags))
ice_dcb_rebuild(pf);
+ /* If the PF previously had enabled PTP, PTP init needs to happen before
+ * the VSI rebuild. If not, this causes the PTP link status events to
+ * fail.
+ */
+ if (test_bit(ICE_FLAG_PTP_SUPPORTED, pf->flags))
+ ice_ptp_init(pf);
+
/* rebuild PF VSI */
err = ice_vsi_rebuild_by_type(pf, ICE_VSI_PF);
if (err) {
/* if we get here, reset flow is successful */
clear_bit(ICE_RESET_FAILED, pf->state);
+
+ ice_plug_aux_dev(pf);
return;
err_vsi_rebuild:
struct ice_netdev_priv *np = netdev_priv(netdev);
struct ice_vsi *vsi = np->vsi;
struct ice_pf *pf = vsi->back;
+ struct iidc_event *event;
u8 count = 0;
+ int err = 0;
if (new_mtu == (int)netdev->mtu) {
netdev_warn(netdev, "MTU is already %u\n", netdev->mtu);
return -EBUSY;
}
+ event = kzalloc(sizeof(*event), GFP_KERNEL);
+ if (!event)
+ return -ENOMEM;
+
+ set_bit(IIDC_EVENT_BEFORE_MTU_CHANGE, event->type);
+ ice_send_event_to_aux(pf, event);
+ clear_bit(IIDC_EVENT_BEFORE_MTU_CHANGE, event->type);
+
netdev->mtu = (unsigned int)new_mtu;
/* if VSI is up, bring it down and then back up */
if (!test_and_set_bit(ICE_VSI_DOWN, vsi->state)) {
- int err;
-
err = ice_down(vsi);
if (err) {
netdev_err(netdev, "change MTU if_down err %d\n", err);
- return err;
+ goto event_after;
}
err = ice_up(vsi);
if (err) {
netdev_err(netdev, "change MTU if_up err %d\n", err);
- return err;
+ goto event_after;
}
}
netdev_dbg(netdev, "changed MTU to %d\n", new_mtu);
- return 0;
+event_after:
+ set_bit(IIDC_EVENT_AFTER_MTU_CHANGE, event->type);
+ ice_send_event_to_aux(pf, event);
+ kfree(event);
+
+ return err;
+}
+
+/**
+ * ice_do_ioctl - Access the hwtstamp interface
+ * @netdev: network interface device structure
+ * @ifr: interface request data
+ * @cmd: ioctl command
+ */
+static int ice_do_ioctl(struct net_device *netdev, struct ifreq *ifr, int cmd)
+{
+ struct ice_netdev_priv *np = netdev_priv(netdev);
+ struct ice_pf *pf = np->vsi->back;
+
+ switch (cmd) {
+ case SIOCGHWTSTAMP:
+ return ice_ptp_get_ts_config(pf, ifr);
+ case SIOCSHWTSTAMP:
+ return ice_ptp_set_ts_config(pf, ifr);
+ default:
+ return -EOPNOTSUPP;
+ }
}
/**
return -EIO;
}
+ ice_check_module_power(pf, pi->phy.link_info.link_cfg_err);
+
/* Set PHY if there is media, otherwise, turn off PHY */
if (pi->phy.link_info.link_info & ICE_AQ_MEDIA_AVAILABLE) {
clear_bit(ICE_FLAG_NO_MEDIA, pf->flags);
.ndo_change_mtu = ice_change_mtu,
.ndo_get_stats64 = ice_get_stats64,
.ndo_tx_timeout = ice_tx_timeout,
+ .ndo_bpf = ice_xdp_safe_mode,
};
static const struct net_device_ops ice_netdev_ops = {
.ndo_change_mtu = ice_change_mtu,
.ndo_get_stats64 = ice_get_stats64,
.ndo_set_tx_maxrate = ice_set_tx_maxrate,
+ .ndo_do_ioctl = ice_do_ioctl,
.ndo_set_vf_spoofchk = ice_set_vf_spoofchk,
.ndo_set_vf_mac = ice_set_vf_mac,
.ndo_get_vf_config = ice_get_vf_cfg,
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (C) 2021, Intel Corporation. */
+
+#include "ice.h"
+#include "ice_lib.h"
+
+#define E810_OUT_PROP_DELAY_NS 1
+
+/**
+ * ice_set_tx_tstamp - Enable or disable Tx timestamping
+ * @pf: The PF pointer to search in
+ * @on: bool value for whether timestamps are enabled or disabled
+ */
+static void ice_set_tx_tstamp(struct ice_pf *pf, bool on)
+{
+ struct ice_vsi *vsi;
+ u32 val;
+ u16 i;
+
+ vsi = ice_get_main_vsi(pf);
+ if (!vsi)
+ return;
+
+ /* Set the timestamp enable flag for all the Tx rings */
+ ice_for_each_rxq(vsi, i) {
+ if (!vsi->tx_rings[i])
+ continue;
+ vsi->tx_rings[i]->ptp_tx = on;
+ }
+
+ /* Configure the Tx timestamp interrupt */
+ val = rd32(&pf->hw, PFINT_OICR_ENA);
+ if (on)
+ val |= PFINT_OICR_TSYN_TX_M;
+ else
+ val &= ~PFINT_OICR_TSYN_TX_M;
+ wr32(&pf->hw, PFINT_OICR_ENA, val);
+}
+
+/**
+ * ice_set_rx_tstamp - Enable or disable Rx timestamping
+ * @pf: The PF pointer to search in
+ * @on: bool value for whether timestamps are enabled or disabled
+ */
+static void ice_set_rx_tstamp(struct ice_pf *pf, bool on)
+{
+ struct ice_vsi *vsi;
+ u16 i;
+
+ vsi = ice_get_main_vsi(pf);
+ if (!vsi)
+ return;
+
+ /* Set the timestamp flag for all the Rx rings */
+ ice_for_each_rxq(vsi, i) {
+ if (!vsi->rx_rings[i])
+ continue;
+ vsi->rx_rings[i]->ptp_rx = on;
+ }
+}
+
+/**
+ * ice_ptp_cfg_timestamp - Configure timestamp for init/deinit
+ * @pf: Board private structure
+ * @ena: bool value to enable or disable time stamp
+ *
+ * This function will configure timestamping during PTP initialization
+ * and deinitialization
+ */
+static void ice_ptp_cfg_timestamp(struct ice_pf *pf, bool ena)
+{
+ ice_set_tx_tstamp(pf, ena);
+ ice_set_rx_tstamp(pf, ena);
+
+ if (ena) {
+ pf->ptp.tstamp_config.rx_filter = HWTSTAMP_FILTER_ALL;
+ pf->ptp.tstamp_config.tx_type = HWTSTAMP_TX_ON;
+ } else {
+ pf->ptp.tstamp_config.rx_filter = HWTSTAMP_FILTER_NONE;
+ pf->ptp.tstamp_config.tx_type = HWTSTAMP_TX_OFF;
+ }
+}
+
+/**
+ * ice_get_ptp_clock_index - Get the PTP clock index
+ * @pf: the PF pointer
+ *
+ * Determine the clock index of the PTP clock associated with this device. If
+ * this is the PF controlling the clock, just use the local access to the
+ * clock device pointer.
+ *
+ * Otherwise, read from the driver shared parameters to determine the clock
+ * index value.
+ *
+ * Returns: the index of the PTP clock associated with this device, or -1 if
+ * there is no associated clock.
+ */
+int ice_get_ptp_clock_index(struct ice_pf *pf)
+{
+ struct device *dev = ice_pf_to_dev(pf);
+ enum ice_aqc_driver_params param_idx;
+ struct ice_hw *hw = &pf->hw;
+ u8 tmr_idx;
+ u32 value;
+ int err;
+
+ /* Use the ptp_clock structure if we're the main PF */
+ if (pf->ptp.clock)
+ return ptp_clock_index(pf->ptp.clock);
+
+ tmr_idx = hw->func_caps.ts_func_info.tmr_index_assoc;
+ if (!tmr_idx)
+ param_idx = ICE_AQC_DRIVER_PARAM_CLK_IDX_TMR0;
+ else
+ param_idx = ICE_AQC_DRIVER_PARAM_CLK_IDX_TMR1;
+
+ err = ice_aq_get_driver_param(hw, param_idx, &value, NULL);
+ if (err) {
+ dev_err(dev, "Failed to read PTP clock index parameter, err %d aq_err %s\n",
+ err, ice_aq_str(hw->adminq.sq_last_status));
+ return -1;
+ }
+
+ /* The PTP clock index is an integer, and will be between 0 and
+ * INT_MAX. The highest bit of the driver shared parameter is used to
+ * indicate whether or not the currently stored clock index is valid.
+ */
+ if (!(value & PTP_SHARED_CLK_IDX_VALID))
+ return -1;
+
+ return value & ~PTP_SHARED_CLK_IDX_VALID;
+}
+
+/**
+ * ice_set_ptp_clock_index - Set the PTP clock index
+ * @pf: the PF pointer
+ *
+ * Set the PTP clock index for this device into the shared driver parameters,
+ * so that other PFs associated with this device can read it.
+ *
+ * If the PF is unable to store the clock index, it will log an error, but
+ * will continue operating PTP.
+ */
+static void ice_set_ptp_clock_index(struct ice_pf *pf)
+{
+ struct device *dev = ice_pf_to_dev(pf);
+ enum ice_aqc_driver_params param_idx;
+ struct ice_hw *hw = &pf->hw;
+ u8 tmr_idx;
+ u32 value;
+ int err;
+
+ if (!pf->ptp.clock)
+ return;
+
+ tmr_idx = hw->func_caps.ts_func_info.tmr_index_assoc;
+ if (!tmr_idx)
+ param_idx = ICE_AQC_DRIVER_PARAM_CLK_IDX_TMR0;
+ else
+ param_idx = ICE_AQC_DRIVER_PARAM_CLK_IDX_TMR1;
+
+ value = (u32)ptp_clock_index(pf->ptp.clock);
+ if (value > INT_MAX) {
+ dev_err(dev, "PTP Clock index is too large to store\n");
+ return;
+ }
+ value |= PTP_SHARED_CLK_IDX_VALID;
+
+ err = ice_aq_set_driver_param(hw, param_idx, value, NULL);
+ if (err) {
+ dev_err(dev, "Failed to set PTP clock index parameter, err %d aq_err %s\n",
+ err, ice_aq_str(hw->adminq.sq_last_status));
+ }
+}
+
+/**
+ * ice_clear_ptp_clock_index - Clear the PTP clock index
+ * @pf: the PF pointer
+ *
+ * Clear the PTP clock index for this device. Must be called when
+ * unregistering the PTP clock, in order to ensure other PFs stop reporting
+ * a clock object that no longer exists.
+ */
+static void ice_clear_ptp_clock_index(struct ice_pf *pf)
+{
+ struct device *dev = ice_pf_to_dev(pf);
+ enum ice_aqc_driver_params param_idx;
+ struct ice_hw *hw = &pf->hw;
+ u8 tmr_idx;
+ int err;
+
+ /* Do not clear the index if we don't own the timer */
+ if (!hw->func_caps.ts_func_info.src_tmr_owned)
+ return;
+
+ tmr_idx = hw->func_caps.ts_func_info.tmr_index_assoc;
+ if (!tmr_idx)
+ param_idx = ICE_AQC_DRIVER_PARAM_CLK_IDX_TMR0;
+ else
+ param_idx = ICE_AQC_DRIVER_PARAM_CLK_IDX_TMR1;
+
+ err = ice_aq_set_driver_param(hw, param_idx, 0, NULL);
+ if (err) {
+ dev_dbg(dev, "Failed to clear PTP clock index parameter, err %d aq_err %s\n",
+ err, ice_aq_str(hw->adminq.sq_last_status));
+ }
+}
+
+/**
+ * ice_ptp_read_src_clk_reg - Read the source clock register
+ * @pf: Board private structure
+ * @sts: Optional parameter for holding a pair of system timestamps from
+ * the system clock. Will be ignored if NULL is given.
+ */
+static u64
+ice_ptp_read_src_clk_reg(struct ice_pf *pf, struct ptp_system_timestamp *sts)
+{
+ struct ice_hw *hw = &pf->hw;
+ u32 hi, lo, lo2;
+ u8 tmr_idx;
+
+ tmr_idx = ice_get_ptp_src_clock_index(hw);
+ /* Read the system timestamp pre PHC read */
+ ptp_read_system_prets(sts);
+
+ lo = rd32(hw, GLTSYN_TIME_L(tmr_idx));
+
+ /* Read the system timestamp post PHC read */
+ ptp_read_system_postts(sts);
+
+ hi = rd32(hw, GLTSYN_TIME_H(tmr_idx));
+ lo2 = rd32(hw, GLTSYN_TIME_L(tmr_idx));
+
+ if (lo2 < lo) {
+ /* if TIME_L rolled over read TIME_L again and update
+ * system timestamps
+ */
+ ptp_read_system_prets(sts);
+ lo = rd32(hw, GLTSYN_TIME_L(tmr_idx));
+ ptp_read_system_postts(sts);
+ hi = rd32(hw, GLTSYN_TIME_H(tmr_idx));
+ }
+
+ return ((u64)hi << 32) | lo;
+}
+
+/**
+ * ice_ptp_update_cached_phctime - Update the cached PHC time values
+ * @pf: Board specific private structure
+ *
+ * This function updates the system time values which are cached in the PF
+ * structure and the Rx rings.
+ *
+ * This function must be called periodically to ensure that the cached value
+ * is never more than 2 seconds old. It must also be called whenever the PHC
+ * time has been changed.
+ */
+static void ice_ptp_update_cached_phctime(struct ice_pf *pf)
+{
+ u64 systime;
+ int i;
+
+ /* Read the current PHC time */
+ systime = ice_ptp_read_src_clk_reg(pf, NULL);
+
+ /* Update the cached PHC time stored in the PF structure */
+ WRITE_ONCE(pf->ptp.cached_phc_time, systime);
+
+ ice_for_each_vsi(pf, i) {
+ struct ice_vsi *vsi = pf->vsi[i];
+ int j;
+
+ if (!vsi)
+ continue;
+
+ if (vsi->type != ICE_VSI_PF)
+ continue;
+
+ ice_for_each_rxq(vsi, j) {
+ if (!vsi->rx_rings[j])
+ continue;
+ WRITE_ONCE(vsi->rx_rings[j]->cached_phctime, systime);
+ }
+ }
+}
+
+/**
+ * ice_ptp_extend_32b_ts - Convert a 32b nanoseconds timestamp to 64b
+ * @cached_phc_time: recently cached copy of PHC time
+ * @in_tstamp: Ingress/egress 32b nanoseconds timestamp value
+ *
+ * Hardware captures timestamps which contain only 32 bits of nominal
+ * nanoseconds, as opposed to the 64bit timestamps that the stack expects.
+ * Note that the captured timestamp values may be 40 bits, but the lower
+ * 8 bits are sub-nanoseconds and generally discarded.
+ *
+ * Extend the 32bit nanosecond timestamp using the following algorithm and
+ * assumptions:
+ *
+ * 1) have a recently cached copy of the PHC time
+ * 2) assume that the in_tstamp was captured 2^31 nanoseconds (~2.1
+ * seconds) before or after the PHC time was captured.
+ * 3) calculate the delta between the cached time and the timestamp
+ * 4) if the delta is smaller than 2^31 nanoseconds, then the timestamp was
+ * captured after the PHC time. In this case, the full timestamp is just
+ * the cached PHC time plus the delta.
+ * 5) otherwise, if the delta is larger than 2^31 nanoseconds, then the
+ * timestamp was captured *before* the PHC time, i.e. because the PHC
+ * cache was updated after the timestamp was captured by hardware. In this
+ * case, the full timestamp is the cached time minus the inverse delta.
+ *
+ * This algorithm works even if the PHC time was updated after a Tx timestamp
+ * was requested, but before the Tx timestamp event was reported from
+ * hardware.
+ *
+ * This calculation primarily relies on keeping the cached PHC time up to
+ * date. If the timestamp was captured more than 2^31 nanoseconds after the
+ * PHC time, it is possible that the lower 32bits of PHC time have
+ * overflowed more than once, and we might generate an incorrect timestamp.
+ *
+ * This is prevented by (a) periodically updating the cached PHC time once
+ * a second, and (b) discarding any Tx timestamp packet if it has waited for
+ * a timestamp for more than one second.
+ */
+static u64 ice_ptp_extend_32b_ts(u64 cached_phc_time, u32 in_tstamp)
+{
+ u32 delta, phc_time_lo;
+ u64 ns;
+
+ /* Extract the lower 32 bits of the PHC time */
+ phc_time_lo = (u32)cached_phc_time;
+
+ /* Calculate the delta between the lower 32bits of the cached PHC
+ * time and the in_tstamp value
+ */
+ delta = (in_tstamp - phc_time_lo);
+
+ /* Do not assume that the in_tstamp is always more recent than the
+ * cached PHC time. If the delta is large, it indicates that the
+ * in_tstamp was taken in the past, and should be converted
+ * forward.
+ */
+ if (delta > (U32_MAX / 2)) {
+ /* reverse the delta calculation here */
+ delta = (phc_time_lo - in_tstamp);
+ ns = cached_phc_time - delta;
+ } else {
+ ns = cached_phc_time + delta;
+ }
+
+ return ns;
+}
+
+/**
+ * ice_ptp_extend_40b_ts - Convert a 40b timestamp to 64b nanoseconds
+ * @pf: Board private structure
+ * @in_tstamp: Ingress/egress 40b timestamp value
+ *
+ * The Tx and Rx timestamps are 40 bits wide, including 32 bits of nominal
+ * nanoseconds, 7 bits of sub-nanoseconds, and a valid bit.
+ *
+ * *--------------------------------------------------------------*
+ * | 32 bits of nanoseconds | 7 high bits of sub ns underflow | v |
+ * *--------------------------------------------------------------*
+ *
+ * The low bit is an indicator of whether the timestamp is valid. The next
+ * 7 bits are a capture of the upper 7 bits of the sub-nanosecond underflow,
+ * and the remaining 32 bits are the lower 32 bits of the PHC timer.
+ *
+ * It is assumed that the caller verifies the timestamp is valid prior to
+ * calling this function.
+ *
+ * Extract the 32bit nominal nanoseconds and extend them. Use the cached PHC
+ * time stored in the device private PTP structure as the basis for timestamp
+ * extension.
+ *
+ * See ice_ptp_extend_32b_ts for a detailed explanation of the extension
+ * algorithm.
+ */
+static u64 ice_ptp_extend_40b_ts(struct ice_pf *pf, u64 in_tstamp)
+{
+ const u64 mask = GENMASK_ULL(31, 0);
+
+ return ice_ptp_extend_32b_ts(pf->ptp.cached_phc_time,
+ (in_tstamp >> 8) & mask);
+}
+
+/**
+ * ice_ptp_read_time - Read the time from the device
+ * @pf: Board private structure
+ * @ts: timespec structure to hold the current time value
+ * @sts: Optional parameter for holding a pair of system timestamps from
+ * the system clock. Will be ignored if NULL is given.
+ *
+ * This function reads the source clock registers and stores them in a timespec.
+ * However, since the registers are 64 bits of nanoseconds, we must convert the
+ * result to a timespec before we can return.
+ */
+static void
+ice_ptp_read_time(struct ice_pf *pf, struct timespec64 *ts,
+ struct ptp_system_timestamp *sts)
+{
+ u64 time_ns = ice_ptp_read_src_clk_reg(pf, sts);
+
+ *ts = ns_to_timespec64(time_ns);
+}
+
+/**
+ * ice_ptp_write_init - Set PHC time to provided value
+ * @pf: Board private structure
+ * @ts: timespec structure that holds the new time value
+ *
+ * Set the PHC time to the specified time provided in the timespec.
+ */
+static int ice_ptp_write_init(struct ice_pf *pf, struct timespec64 *ts)
+{
+ u64 ns = timespec64_to_ns(ts);
+ struct ice_hw *hw = &pf->hw;
+
+ return ice_ptp_init_time(hw, ns);
+}
+
+/**
+ * ice_ptp_write_adj - Adjust PHC clock time atomically
+ * @pf: Board private structure
+ * @adj: Adjustment in nanoseconds
+ *
+ * Perform an atomic adjustment of the PHC time by the specified number of
+ * nanoseconds.
+ */
+static int ice_ptp_write_adj(struct ice_pf *pf, s32 adj)
+{
+ struct ice_hw *hw = &pf->hw;
+
+ return ice_ptp_adj_clock(hw, adj);
+}
+
+/**
+ * ice_ptp_adjfine - Adjust clock increment rate
+ * @info: the driver's PTP info structure
+ * @scaled_ppm: Parts per million with 16-bit fractional field
+ *
+ * Adjust the frequency of the clock by the indicated scaled ppm from the
+ * base frequency.
+ */
+static int ice_ptp_adjfine(struct ptp_clock_info *info, long scaled_ppm)
+{
+ struct ice_pf *pf = ptp_info_to_pf(info);
+ u64 freq, divisor = 1000000ULL;
+ struct ice_hw *hw = &pf->hw;
+ s64 incval, diff;
+ int neg_adj = 0;
+ int err;
+
+ incval = ICE_PTP_NOMINAL_INCVAL_E810;
+
+ if (scaled_ppm < 0) {
+ neg_adj = 1;
+ scaled_ppm = -scaled_ppm;
+ }
+
+ while ((u64)scaled_ppm > div_u64(U64_MAX, incval)) {
+ /* handle overflow by scaling down the scaled_ppm and
+ * the divisor, losing some precision
+ */
+ scaled_ppm >>= 2;
+ divisor >>= 2;
+ }
+
+ freq = (incval * (u64)scaled_ppm) >> 16;
+ diff = div_u64(freq, divisor);
+
+ if (neg_adj)
+ incval -= diff;
+ else
+ incval += diff;
+
+ err = ice_ptp_write_incval_locked(hw, incval);
+ if (err) {
+ dev_err(ice_pf_to_dev(pf), "PTP failed to set incval, err %d\n",
+ err);
+ return -EIO;
+ }
+
+ return 0;
+}
+
+/**
+ * ice_ptp_extts_work - Workqueue task function
+ * @work: external timestamp work structure
+ *
+ * Service for PTP external clock event
+ */
+static void ice_ptp_extts_work(struct kthread_work *work)
+{
+ struct ice_ptp *ptp = container_of(work, struct ice_ptp, extts_work);
+ struct ice_pf *pf = container_of(ptp, struct ice_pf, ptp);
+ struct ptp_clock_event event;
+ struct ice_hw *hw = &pf->hw;
+ u8 chan, tmr_idx;
+ u32 hi, lo;
+
+ tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned;
+ /* Event time is captured by one of the two matched registers
+ * GLTSYN_EVNT_L: 32 LSB of sampled time event
+ * GLTSYN_EVNT_H: 32 MSB of sampled time event
+ * Event is defined in GLTSYN_EVNT_0 register
+ */
+ for (chan = 0; chan < GLTSYN_EVNT_H_IDX_MAX; chan++) {
+ /* Check if channel is enabled */
+ if (pf->ptp.ext_ts_irq & (1 << chan)) {
+ lo = rd32(hw, GLTSYN_EVNT_L(chan, tmr_idx));
+ hi = rd32(hw, GLTSYN_EVNT_H(chan, tmr_idx));
+ event.timestamp = (((u64)hi) << 32) | lo;
+ event.type = PTP_CLOCK_EXTTS;
+ event.index = chan;
+
+ /* Fire event */
+ ptp_clock_event(pf->ptp.clock, &event);
+ pf->ptp.ext_ts_irq &= ~(1 << chan);
+ }
+ }
+}
+
+/**
+ * ice_ptp_cfg_extts - Configure EXTTS pin and channel
+ * @pf: Board private structure
+ * @ena: true to enable; false to disable
+ * @chan: GPIO channel (0-3)
+ * @gpio_pin: GPIO pin
+ * @extts_flags: request flags from the ptp_extts_request.flags
+ */
+static int
+ice_ptp_cfg_extts(struct ice_pf *pf, bool ena, unsigned int chan, u32 gpio_pin,
+ unsigned int extts_flags)
+{
+ u32 func, aux_reg, gpio_reg, irq_reg;
+ struct ice_hw *hw = &pf->hw;
+ u8 tmr_idx;
+
+ if (chan > (unsigned int)pf->ptp.info.n_ext_ts)
+ return -EINVAL;
+
+ tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned;
+
+ irq_reg = rd32(hw, PFINT_OICR_ENA);
+
+ if (ena) {
+ /* Enable the interrupt */
+ irq_reg |= PFINT_OICR_TSYN_EVNT_M;
+ aux_reg = GLTSYN_AUX_IN_0_INT_ENA_M;
+
+#define GLTSYN_AUX_IN_0_EVNTLVL_RISING_EDGE BIT(0)
+#define GLTSYN_AUX_IN_0_EVNTLVL_FALLING_EDGE BIT(1)
+
+ /* set event level to requested edge */
+ if (extts_flags & PTP_FALLING_EDGE)
+ aux_reg |= GLTSYN_AUX_IN_0_EVNTLVL_FALLING_EDGE;
+ if (extts_flags & PTP_RISING_EDGE)
+ aux_reg |= GLTSYN_AUX_IN_0_EVNTLVL_RISING_EDGE;
+
+ /* Write GPIO CTL reg.
+ * 0x1 is input sampled by EVENT register(channel)
+ * + num_in_channels * tmr_idx
+ */
+ func = 1 + chan + (tmr_idx * 3);
+ gpio_reg = ((func << GLGEN_GPIO_CTL_PIN_FUNC_S) &
+ GLGEN_GPIO_CTL_PIN_FUNC_M);
+ pf->ptp.ext_ts_chan |= (1 << chan);
+ } else {
+ /* clear the values we set to reset defaults */
+ aux_reg = 0;
+ gpio_reg = 0;
+ pf->ptp.ext_ts_chan &= ~(1 << chan);
+ if (!pf->ptp.ext_ts_chan)
+ irq_reg &= ~PFINT_OICR_TSYN_EVNT_M;
+ }
+
+ wr32(hw, PFINT_OICR_ENA, irq_reg);
+ wr32(hw, GLTSYN_AUX_IN(chan, tmr_idx), aux_reg);
+ wr32(hw, GLGEN_GPIO_CTL(gpio_pin), gpio_reg);
+
+ return 0;
+}
+
+/**
+ * ice_ptp_cfg_clkout - Configure clock to generate periodic wave
+ * @pf: Board private structure
+ * @chan: GPIO channel (0-3)
+ * @config: desired periodic clk configuration. NULL will disable channel
+ * @store: If set to true the values will be stored
+ *
+ * Configure the internal clock generator modules to generate the clock wave of
+ * specified period.
+ */
+static int ice_ptp_cfg_clkout(struct ice_pf *pf, unsigned int chan,
+ struct ice_perout_channel *config, bool store)
+{
+ u64 current_time, period, start_time, phase;
+ struct ice_hw *hw = &pf->hw;
+ u32 func, val, gpio_pin;
+ u8 tmr_idx;
+
+ tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned;
+
+ /* 0. Reset mode & out_en in AUX_OUT */
+ wr32(hw, GLTSYN_AUX_OUT(chan, tmr_idx), 0);
+
+ /* If we're disabling the output, clear out CLKO and TGT and keep
+ * output level low
+ */
+ if (!config || !config->ena) {
+ wr32(hw, GLTSYN_CLKO(chan, tmr_idx), 0);
+ wr32(hw, GLTSYN_TGT_L(chan, tmr_idx), 0);
+ wr32(hw, GLTSYN_TGT_H(chan, tmr_idx), 0);
+
+ val = GLGEN_GPIO_CTL_PIN_DIR_M;
+ gpio_pin = pf->ptp.perout_channels[chan].gpio_pin;
+ wr32(hw, GLGEN_GPIO_CTL(gpio_pin), val);
+
+ /* Store the value if requested */
+ if (store)
+ memset(&pf->ptp.perout_channels[chan], 0,
+ sizeof(struct ice_perout_channel));
+
+ return 0;
+ }
+ period = config->period;
+ start_time = config->start_time;
+ div64_u64_rem(start_time, period, &phase);
+ gpio_pin = config->gpio_pin;
+
+ /* 1. Write clkout with half of required period value */
+ if (period & 0x1) {
+ dev_err(ice_pf_to_dev(pf), "CLK Period must be an even value\n");
+ goto err;
+ }
+
+ period >>= 1;
+
+ /* For proper operation, the GLTSYN_CLKO must be larger than clock tick
+ */
+#define MIN_PULSE 3
+ if (period <= MIN_PULSE || period > U32_MAX) {
+ dev_err(ice_pf_to_dev(pf), "CLK Period must be > %d && < 2^33",
+ MIN_PULSE * 2);
+ goto err;
+ }
+
+ wr32(hw, GLTSYN_CLKO(chan, tmr_idx), lower_32_bits(period));
+
+ /* Allow time for programming before start_time is hit */
+ current_time = ice_ptp_read_src_clk_reg(pf, NULL);
+
+ /* if start time is in the past start the timer at the nearest second
+ * maintaining phase
+ */
+ if (start_time < current_time)
+ start_time = div64_u64(current_time + NSEC_PER_MSEC - 1,
+ NSEC_PER_SEC) * NSEC_PER_SEC + phase;
+
+ start_time -= E810_OUT_PROP_DELAY_NS;
+
+ /* 2. Write TARGET time */
+ wr32(hw, GLTSYN_TGT_L(chan, tmr_idx), lower_32_bits(start_time));
+ wr32(hw, GLTSYN_TGT_H(chan, tmr_idx), upper_32_bits(start_time));
+
+ /* 3. Write AUX_OUT register */
+ val = GLTSYN_AUX_OUT_0_OUT_ENA_M | GLTSYN_AUX_OUT_0_OUTMOD_M;
+ wr32(hw, GLTSYN_AUX_OUT(chan, tmr_idx), val);
+
+ /* 4. write GPIO CTL reg */
+ func = 8 + chan + (tmr_idx * 4);
+ val = GLGEN_GPIO_CTL_PIN_DIR_M |
+ ((func << GLGEN_GPIO_CTL_PIN_FUNC_S) & GLGEN_GPIO_CTL_PIN_FUNC_M);
+ wr32(hw, GLGEN_GPIO_CTL(gpio_pin), val);
+
+ /* Store the value if requested */
+ if (store) {
+ memcpy(&pf->ptp.perout_channels[chan], config,
+ sizeof(struct ice_perout_channel));
+ pf->ptp.perout_channels[chan].start_time = phase;
+ }
+
+ return 0;
+err:
+ dev_err(ice_pf_to_dev(pf), "PTP failed to cfg per_clk\n");
+ return -EFAULT;
+}
+
+/**
+ * ice_ptp_gpio_enable_e810 - Enable/disable ancillary features of PHC
+ * @info: the driver's PTP info structure
+ * @rq: The requested feature to change
+ * @on: Enable/disable flag
+ */
+static int
+ice_ptp_gpio_enable_e810(struct ptp_clock_info *info,
+ struct ptp_clock_request *rq, int on)
+{
+ struct ice_pf *pf = ptp_info_to_pf(info);
+ struct ice_perout_channel clk_cfg = {0};
+ unsigned int chan;
+ u32 gpio_pin;
+ int err;
+
+ switch (rq->type) {
+ case PTP_CLK_REQ_PEROUT:
+ chan = rq->perout.index;
+ if (chan == PPS_CLK_GEN_CHAN)
+ clk_cfg.gpio_pin = PPS_PIN_INDEX;
+ else
+ clk_cfg.gpio_pin = chan;
+
+ clk_cfg.period = ((rq->perout.period.sec * NSEC_PER_SEC) +
+ rq->perout.period.nsec);
+ clk_cfg.start_time = ((rq->perout.start.sec * NSEC_PER_SEC) +
+ rq->perout.start.nsec);
+ clk_cfg.ena = !!on;
+
+ err = ice_ptp_cfg_clkout(pf, chan, &clk_cfg, true);
+ break;
+ case PTP_CLK_REQ_EXTTS:
+ chan = rq->extts.index;
+ gpio_pin = chan;
+
+ err = ice_ptp_cfg_extts(pf, !!on, chan, gpio_pin,
+ rq->extts.flags);
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ return err;
+}
+
+/**
+ * ice_ptp_gettimex64 - Get the time of the clock
+ * @info: the driver's PTP info structure
+ * @ts: timespec64 structure to hold the current time value
+ * @sts: Optional parameter for holding a pair of system timestamps from
+ * the system clock. Will be ignored if NULL is given.
+ *
+ * Read the device clock and return the correct value on ns, after converting it
+ * into a timespec struct.
+ */
+static int
+ice_ptp_gettimex64(struct ptp_clock_info *info, struct timespec64 *ts,
+ struct ptp_system_timestamp *sts)
+{
+ struct ice_pf *pf = ptp_info_to_pf(info);
+ struct ice_hw *hw = &pf->hw;
+
+ if (!ice_ptp_lock(hw)) {
+ dev_err(ice_pf_to_dev(pf), "PTP failed to get time\n");
+ return -EBUSY;
+ }
+
+ ice_ptp_read_time(pf, ts, sts);
+ ice_ptp_unlock(hw);
+
+ return 0;
+}
+
+/**
+ * ice_ptp_settime64 - Set the time of the clock
+ * @info: the driver's PTP info structure
+ * @ts: timespec64 structure that holds the new time value
+ *
+ * Set the device clock to the user input value. The conversion from timespec
+ * to ns happens in the write function.
+ */
+static int
+ice_ptp_settime64(struct ptp_clock_info *info, const struct timespec64 *ts)
+{
+ struct ice_pf *pf = ptp_info_to_pf(info);
+ struct timespec64 ts64 = *ts;
+ struct ice_hw *hw = &pf->hw;
+ int err;
+
+ if (!ice_ptp_lock(hw)) {
+ err = -EBUSY;
+ goto exit;
+ }
+
+ err = ice_ptp_write_init(pf, &ts64);
+ ice_ptp_unlock(hw);
+
+ if (!err)
+ ice_ptp_update_cached_phctime(pf);
+
+exit:
+ if (err) {
+ dev_err(ice_pf_to_dev(pf), "PTP failed to set time %d\n", err);
+ return err;
+ }
+
+ return 0;
+}
+
+/**
+ * ice_ptp_adjtime_nonatomic - Do a non-atomic clock adjustment
+ * @info: the driver's PTP info structure
+ * @delta: Offset in nanoseconds to adjust the time by
+ */
+static int ice_ptp_adjtime_nonatomic(struct ptp_clock_info *info, s64 delta)
+{
+ struct timespec64 now, then;
+
+ then = ns_to_timespec64(delta);
+ ice_ptp_gettimex64(info, &now, NULL);
+ now = timespec64_add(now, then);
+
+ return ice_ptp_settime64(info, (const struct timespec64 *)&now);
+}
+
+/**
+ * ice_ptp_adjtime - Adjust the time of the clock by the indicated delta
+ * @info: the driver's PTP info structure
+ * @delta: Offset in nanoseconds to adjust the time by
+ */
+static int ice_ptp_adjtime(struct ptp_clock_info *info, s64 delta)
+{
+ struct ice_pf *pf = ptp_info_to_pf(info);
+ struct ice_hw *hw = &pf->hw;
+ struct device *dev;
+ int err;
+
+ dev = ice_pf_to_dev(pf);
+
+ /* Hardware only supports atomic adjustments using signed 32-bit
+ * integers. For any adjustment outside this range, perform
+ * a non-atomic get->adjust->set flow.
+ */
+ if (delta > S32_MAX || delta < S32_MIN) {
+ dev_dbg(dev, "delta = %lld, adjtime non-atomic\n", delta);
+ return ice_ptp_adjtime_nonatomic(info, delta);
+ }
+
+ if (!ice_ptp_lock(hw)) {
+ dev_err(dev, "PTP failed to acquire semaphore in adjtime\n");
+ return -EBUSY;
+ }
+
+ err = ice_ptp_write_adj(pf, delta);
+
+ ice_ptp_unlock(hw);
+
+ if (err) {
+ dev_err(dev, "PTP failed to adjust time, err %d\n", err);
+ return err;
+ }
+
+ ice_ptp_update_cached_phctime(pf);
+
+ return 0;
+}
+
+/**
+ * ice_ptp_get_ts_config - ioctl interface to read the timestamping config
+ * @pf: Board private structure
+ * @ifr: ioctl data
+ *
+ * Copy the timestamping config to user buffer
+ */
+int ice_ptp_get_ts_config(struct ice_pf *pf, struct ifreq *ifr)
+{
+ struct hwtstamp_config *config;
+
+ if (!test_bit(ICE_FLAG_PTP, pf->flags))
+ return -EIO;
+
+ config = &pf->ptp.tstamp_config;
+
+ return copy_to_user(ifr->ifr_data, config, sizeof(*config)) ?
+ -EFAULT : 0;
+}
+
+/**
+ * ice_ptp_set_timestamp_mode - Setup driver for requested timestamp mode
+ * @pf: Board private structure
+ * @config: hwtstamp settings requested or saved
+ */
+static int
+ice_ptp_set_timestamp_mode(struct ice_pf *pf, struct hwtstamp_config *config)
+{
+ /* Reserved for future extensions. */
+ if (config->flags)
+ return -EINVAL;
+
+ switch (config->tx_type) {
+ case HWTSTAMP_TX_OFF:
+ ice_set_tx_tstamp(pf, false);
+ break;
+ case HWTSTAMP_TX_ON:
+ ice_set_tx_tstamp(pf, true);
+ break;
+ default:
+ return -ERANGE;
+ }
+
+ switch (config->rx_filter) {
+ case HWTSTAMP_FILTER_NONE:
+ ice_set_rx_tstamp(pf, false);
+ break;
+ case HWTSTAMP_FILTER_PTP_V1_L4_EVENT:
+ case HWTSTAMP_FILTER_PTP_V1_L4_SYNC:
+ case HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ:
+ case HWTSTAMP_FILTER_PTP_V2_EVENT:
+ case HWTSTAMP_FILTER_PTP_V2_L2_EVENT:
+ case HWTSTAMP_FILTER_PTP_V2_L4_EVENT:
+ case HWTSTAMP_FILTER_PTP_V2_SYNC:
+ case HWTSTAMP_FILTER_PTP_V2_L2_SYNC:
+ case HWTSTAMP_FILTER_PTP_V2_L4_SYNC:
+ case HWTSTAMP_FILTER_PTP_V2_DELAY_REQ:
+ case HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ:
+ case HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ:
+ case HWTSTAMP_FILTER_NTP_ALL:
+ case HWTSTAMP_FILTER_ALL:
+ config->rx_filter = HWTSTAMP_FILTER_ALL;
+ ice_set_rx_tstamp(pf, true);
+ break;
+ default:
+ return -ERANGE;
+ }
+
+ return 0;
+}
+
+/**
+ * ice_ptp_set_ts_config - ioctl interface to control the timestamping
+ * @pf: Board private structure
+ * @ifr: ioctl data
+ *
+ * Get the user config and store it
+ */
+int ice_ptp_set_ts_config(struct ice_pf *pf, struct ifreq *ifr)
+{
+ struct hwtstamp_config config;
+ int err;
+
+ if (!test_bit(ICE_FLAG_PTP, pf->flags))
+ return -EAGAIN;
+
+ if (copy_from_user(&config, ifr->ifr_data, sizeof(config)))
+ return -EFAULT;
+
+ err = ice_ptp_set_timestamp_mode(pf, &config);
+ if (err)
+ return err;
+
+ /* Save these settings for future reference */
+ pf->ptp.tstamp_config = config;
+
+ return copy_to_user(ifr->ifr_data, &config, sizeof(config)) ?
+ -EFAULT : 0;
+}
+
+/**
+ * ice_ptp_rx_hwtstamp - Check for an Rx timestamp
+ * @rx_ring: Ring to get the VSI info
+ * @rx_desc: Receive descriptor
+ * @skb: Particular skb to send timestamp with
+ *
+ * The driver receives a notification in the receive descriptor with timestamp.
+ * The timestamp is in ns, so we must convert the result first.
+ */
+void
+ice_ptp_rx_hwtstamp(struct ice_ring *rx_ring,
+ union ice_32b_rx_flex_desc *rx_desc, struct sk_buff *skb)
+{
+ u32 ts_high;
+ u64 ts_ns;
+
+ /* Populate timesync data into skb */
+ if (rx_desc->wb.time_stamp_low & ICE_PTP_TS_VALID) {
+ struct skb_shared_hwtstamps *hwtstamps;
+
+ /* Use ice_ptp_extend_32b_ts directly, using the ring-specific
+ * cached PHC value, rather than accessing the PF. This also
+ * allows us to simply pass the upper 32bits of nanoseconds
+ * directly. Calling ice_ptp_extend_40b_ts is unnecessary as
+ * it would just discard these bits itself.
+ */
+ ts_high = le32_to_cpu(rx_desc->wb.flex_ts.ts_high);
+ ts_ns = ice_ptp_extend_32b_ts(rx_ring->cached_phctime, ts_high);
+
+ hwtstamps = skb_hwtstamps(skb);
+ memset(hwtstamps, 0, sizeof(*hwtstamps));
+ hwtstamps->hwtstamp = ns_to_ktime(ts_ns);
+ }
+}
+
+/**
+ * ice_ptp_setup_pins_e810 - Setup PTP pins in sysfs
+ * @info: PTP clock capabilities
+ */
+static void ice_ptp_setup_pins_e810(struct ptp_clock_info *info)
+{
+ info->n_per_out = E810_N_PER_OUT;
+ info->n_ext_ts = E810_N_EXT_TS;
+}
+
+/**
+ * ice_ptp_set_funcs_e810 - Set specialized functions for E810 support
+ * @pf: Board private structure
+ * @info: PTP info to fill
+ *
+ * Assign functions to the PTP capabiltiies structure for E810 devices.
+ * Functions which operate across all device families should be set directly
+ * in ice_ptp_set_caps. Only add functions here which are distinct for e810
+ * devices.
+ */
+static void
+ice_ptp_set_funcs_e810(struct ice_pf *pf, struct ptp_clock_info *info)
+{
+ info->enable = ice_ptp_gpio_enable_e810;
+
+ ice_ptp_setup_pins_e810(info);
+}
+
+/**
+ * ice_ptp_set_caps - Set PTP capabilities
+ * @pf: Board private structure
+ */
+static void ice_ptp_set_caps(struct ice_pf *pf)
+{
+ struct ptp_clock_info *info = &pf->ptp.info;
+ struct device *dev = ice_pf_to_dev(pf);
+
+ snprintf(info->name, sizeof(info->name) - 1, "%s-%s-clk",
+ dev_driver_string(dev), dev_name(dev));
+ info->owner = THIS_MODULE;
+ info->max_adj = 999999999;
+ info->adjtime = ice_ptp_adjtime;
+ info->adjfine = ice_ptp_adjfine;
+ info->gettimex64 = ice_ptp_gettimex64;
+ info->settime64 = ice_ptp_settime64;
+
+ ice_ptp_set_funcs_e810(pf, info);
+}
+
+/**
+ * ice_ptp_create_clock - Create PTP clock device for userspace
+ * @pf: Board private structure
+ *
+ * This function creates a new PTP clock device. It only creates one if we
+ * don't already have one. Will return error if it can't create one, but success
+ * if we already have a device. Should be used by ice_ptp_init to create clock
+ * initially, and prevent global resets from creating new clock devices.
+ */
+static long ice_ptp_create_clock(struct ice_pf *pf)
+{
+ struct ptp_clock_info *info;
+ struct ptp_clock *clock;
+ struct device *dev;
+
+ /* No need to create a clock device if we already have one */
+ if (pf->ptp.clock)
+ return 0;
+
+ ice_ptp_set_caps(pf);
+
+ info = &pf->ptp.info;
+ dev = ice_pf_to_dev(pf);
+
+ /* Allocate memory for kernel pins interface */
+ if (info->n_pins) {
+ info->pin_config = devm_kcalloc(dev, info->n_pins,
+ sizeof(*info->pin_config),
+ GFP_KERNEL);
+ if (!info->pin_config) {
+ info->n_pins = 0;
+ return -ENOMEM;
+ }
+ }
+
+ /* Attempt to register the clock before enabling the hardware. */
+ clock = ptp_clock_register(info, dev);
+ if (IS_ERR(clock))
+ return PTR_ERR(clock);
+
+ pf->ptp.clock = clock;
+
+ return 0;
+}
+
+/**
+ * ice_ptp_tx_tstamp_work - Process Tx timestamps for a port
+ * @work: pointer to the kthread_work struct
+ *
+ * Process timestamps captured by the PHY associated with this port. To do
+ * this, loop over each index with a waiting skb.
+ *
+ * If a given index has a valid timestamp, perform the following steps:
+ *
+ * 1) copy the timestamp out of the PHY register
+ * 4) clear the timestamp valid bit in the PHY register
+ * 5) unlock the index by clearing the associated in_use bit.
+ * 2) extend the 40b timestamp value to get a 64bit timestamp
+ * 3) send that timestamp to the stack
+ *
+ * After looping, if we still have waiting SKBs, then re-queue the work. This
+ * may cause us effectively poll even when not strictly necessary. We do this
+ * because it's possible a new timestamp was requested around the same time as
+ * the interrupt. In some cases hardware might not interrupt us again when the
+ * timestamp is captured.
+ *
+ * Note that we only take the tracking lock when clearing the bit and when
+ * checking if we need to re-queue this task. The only place where bits can be
+ * set is the hard xmit routine where an SKB has a request flag set. The only
+ * places where we clear bits are this work function, or the periodic cleanup
+ * thread. If the cleanup thread clears a bit we're processing we catch it
+ * when we lock to clear the bit and then grab the SKB pointer. If a Tx thread
+ * starts a new timestamp, we might not begin processing it right away but we
+ * will notice it at the end when we re-queue the work item. If a Tx thread
+ * starts a new timestamp just after this function exits without re-queuing,
+ * the interrupt when the timestamp finishes should trigger. Avoiding holding
+ * the lock for the entire function is important in order to ensure that Tx
+ * threads do not get blocked while waiting for the lock.
+ */
+static void ice_ptp_tx_tstamp_work(struct kthread_work *work)
+{
+ struct ice_ptp_port *ptp_port;
+ struct ice_ptp_tx *tx;
+ struct ice_pf *pf;
+ struct ice_hw *hw;
+ u8 idx;
+
+ tx = container_of(work, struct ice_ptp_tx, work);
+ if (!tx->init)
+ return;
+
+ ptp_port = container_of(tx, struct ice_ptp_port, tx);
+ pf = ptp_port_to_pf(ptp_port);
+ hw = &pf->hw;
+
+ for_each_set_bit(idx, tx->in_use, tx->len) {
+ struct skb_shared_hwtstamps shhwtstamps = {};
+ u8 phy_idx = idx + tx->quad_offset;
+ u64 raw_tstamp, tstamp;
+ struct sk_buff *skb;
+ int err;
+
+ err = ice_read_phy_tstamp(hw, tx->quad, phy_idx,
+ &raw_tstamp);
+ if (err)
+ continue;
+
+ /* Check if the timestamp is valid */
+ if (!(raw_tstamp & ICE_PTP_TS_VALID))
+ continue;
+
+ /* clear the timestamp register, so that it won't show valid
+ * again when re-used.
+ */
+ ice_clear_phy_tstamp(hw, tx->quad, phy_idx);
+
+ /* The timestamp is valid, so we'll go ahead and clear this
+ * index and then send the timestamp up to the stack.
+ */
+ spin_lock(&tx->lock);
+ clear_bit(idx, tx->in_use);
+ skb = tx->tstamps[idx].skb;
+ tx->tstamps[idx].skb = NULL;
+ spin_unlock(&tx->lock);
+
+ /* it's (unlikely but) possible we raced with the cleanup
+ * thread for discarding old timestamp requests.
+ */
+ if (!skb)
+ continue;
+
+ /* Extend the timestamp using cached PHC time */
+ tstamp = ice_ptp_extend_40b_ts(pf, raw_tstamp);
+ shhwtstamps.hwtstamp = ns_to_ktime(tstamp);
+
+ skb_tstamp_tx(skb, &shhwtstamps);
+ dev_kfree_skb_any(skb);
+ }
+
+ /* Check if we still have work to do. If so, re-queue this task to
+ * poll for remaining timestamps.
+ */
+ spin_lock(&tx->lock);
+ if (!bitmap_empty(tx->in_use, tx->len))
+ kthread_queue_work(pf->ptp.kworker, &tx->work);
+ spin_unlock(&tx->lock);
+}
+
+/**
+ * ice_ptp_request_ts - Request an available Tx timestamp index
+ * @tx: the PTP Tx timestamp tracker to request from
+ * @skb: the SKB to associate with this timestamp request
+ */
+s8 ice_ptp_request_ts(struct ice_ptp_tx *tx, struct sk_buff *skb)
+{
+ u8 idx;
+
+ /* Check if this tracker is initialized */
+ if (!tx->init)
+ return -1;
+
+ spin_lock(&tx->lock);
+ /* Find and set the first available index */
+ idx = find_first_zero_bit(tx->in_use, tx->len);
+ if (idx < tx->len) {
+ /* We got a valid index that no other thread could have set. Store
+ * a reference to the skb and the start time to allow discarding old
+ * requests.
+ */
+ set_bit(idx, tx->in_use);
+ tx->tstamps[idx].start = jiffies;
+ tx->tstamps[idx].skb = skb_get(skb);
+ skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
+ }
+
+ spin_unlock(&tx->lock);
+
+ /* return the appropriate PHY timestamp register index, -1 if no
+ * indexes were available.
+ */
+ if (idx >= tx->len)
+ return -1;
+ else
+ return idx + tx->quad_offset;
+}
+
+/**
+ * ice_ptp_process_ts - Spawn kthread work to handle timestamps
+ * @pf: Board private structure
+ *
+ * Queue work required to process the PTP Tx timestamps outside of interrupt
+ * context.
+ */
+void ice_ptp_process_ts(struct ice_pf *pf)
+{
+ if (pf->ptp.port.tx.init)
+ kthread_queue_work(pf->ptp.kworker, &pf->ptp.port.tx.work);
+}
+
+/**
+ * ice_ptp_alloc_tx_tracker - Initialize tracking for Tx timestamps
+ * @tx: Tx tracking structure to initialize
+ *
+ * Assumes that the length has already been initialized. Do not call directly,
+ * use the ice_ptp_init_tx_e822 or ice_ptp_init_tx_e810 instead.
+ */
+static int
+ice_ptp_alloc_tx_tracker(struct ice_ptp_tx *tx)
+{
+ tx->tstamps = kcalloc(tx->len, sizeof(*tx->tstamps), GFP_KERNEL);
+ if (!tx->tstamps)
+ return -ENOMEM;
+
+ tx->in_use = bitmap_zalloc(tx->len, GFP_KERNEL);
+ if (!tx->in_use) {
+ kfree(tx->tstamps);
+ tx->tstamps = NULL;
+ return -ENOMEM;
+ }
+
+ spin_lock_init(&tx->lock);
+ kthread_init_work(&tx->work, ice_ptp_tx_tstamp_work);
+
+ tx->init = 1;
+
+ return 0;
+}
+
+/**
+ * ice_ptp_flush_tx_tracker - Flush any remaining timestamps from the tracker
+ * @pf: Board private structure
+ * @tx: the tracker to flush
+ */
+static void
+ice_ptp_flush_tx_tracker(struct ice_pf *pf, struct ice_ptp_tx *tx)
+{
+ u8 idx;
+
+ for (idx = 0; idx < tx->len; idx++) {
+ u8 phy_idx = idx + tx->quad_offset;
+
+ /* Clear any potential residual timestamp in the PHY block */
+ if (!pf->hw.reset_ongoing)
+ ice_clear_phy_tstamp(&pf->hw, tx->quad, phy_idx);
+
+ if (tx->tstamps[idx].skb) {
+ dev_kfree_skb_any(tx->tstamps[idx].skb);
+ tx->tstamps[idx].skb = NULL;
+ }
+ }
+}
+
+/**
+ * ice_ptp_release_tx_tracker - Release allocated memory for Tx tracker
+ * @pf: Board private structure
+ * @tx: Tx tracking structure to release
+ *
+ * Free memory associated with the Tx timestamp tracker.
+ */
+static void
+ice_ptp_release_tx_tracker(struct ice_pf *pf, struct ice_ptp_tx *tx)
+{
+ tx->init = 0;
+
+ kthread_cancel_work_sync(&tx->work);
+
+ ice_ptp_flush_tx_tracker(pf, tx);
+
+ kfree(tx->tstamps);
+ tx->tstamps = NULL;
+
+ kfree(tx->in_use);
+ tx->in_use = NULL;
+
+ tx->len = 0;
+}
+
+/**
+ * ice_ptp_init_tx_e810 - Initialize tracking for Tx timestamps
+ * @pf: Board private structure
+ * @tx: the Tx tracking structure to initialize
+ *
+ * Initialize the Tx timestamp tracker for this PF. For E810 devices, each
+ * port has its own block of timestamps, independent of the other ports.
+ */
+static int
+ice_ptp_init_tx_e810(struct ice_pf *pf, struct ice_ptp_tx *tx)
+{
+ tx->quad = pf->hw.port_info->lport;
+ tx->quad_offset = 0;
+ tx->len = INDEX_PER_QUAD;
+
+ return ice_ptp_alloc_tx_tracker(tx);
+}
+
+/**
+ * ice_ptp_tx_tstamp_cleanup - Cleanup old timestamp requests that got dropped
+ * @tx: PTP Tx tracker to clean up
+ *
+ * Loop through the Tx timestamp requests and see if any of them have been
+ * waiting for a long time. Discard any SKBs that have been waiting for more
+ * than 2 seconds. This is long enough to be reasonably sure that the
+ * timestamp will never be captured. This might happen if the packet gets
+ * discarded before it reaches the PHY timestamping block.
+ */
+static void ice_ptp_tx_tstamp_cleanup(struct ice_ptp_tx *tx)
+{
+ u8 idx;
+
+ if (!tx->init)
+ return;
+
+ for_each_set_bit(idx, tx->in_use, tx->len) {
+ struct sk_buff *skb;
+
+ /* Check if this SKB has been waiting for too long */
+ if (time_is_after_jiffies(tx->tstamps[idx].start + 2 * HZ))
+ continue;
+
+ spin_lock(&tx->lock);
+ skb = tx->tstamps[idx].skb;
+ tx->tstamps[idx].skb = NULL;
+ clear_bit(idx, tx->in_use);
+ spin_unlock(&tx->lock);
+
+ /* Free the SKB after we've cleared the bit */
+ dev_kfree_skb_any(skb);
+ }
+}
+
+static void ice_ptp_periodic_work(struct kthread_work *work)
+{
+ struct ice_ptp *ptp = container_of(work, struct ice_ptp, work.work);
+ struct ice_pf *pf = container_of(ptp, struct ice_pf, ptp);
+
+ if (!test_bit(ICE_FLAG_PTP, pf->flags))
+ return;
+
+ ice_ptp_update_cached_phctime(pf);
+
+ ice_ptp_tx_tstamp_cleanup(&pf->ptp.port.tx);
+
+ /* Run twice a second */
+ kthread_queue_delayed_work(ptp->kworker, &ptp->work,
+ msecs_to_jiffies(500));
+}
+
+/**
+ * ice_ptp_init_owner - Initialize PTP_1588_CLOCK device
+ * @pf: Board private structure
+ *
+ * Setup and initialize a PTP clock device that represents the device hardware
+ * clock. Save the clock index for other functions connected to the same
+ * hardware resource.
+ */
+static int ice_ptp_init_owner(struct ice_pf *pf)
+{
+ struct device *dev = ice_pf_to_dev(pf);
+ struct ice_hw *hw = &pf->hw;
+ struct timespec64 ts;
+ u8 src_idx;
+ int err;
+
+ wr32(hw, GLTSYN_SYNC_DLAY, 0);
+
+ /* Clear some HW residue and enable source clock */
+ src_idx = hw->func_caps.ts_func_info.tmr_index_owned;
+
+ /* Enable source clocks */
+ wr32(hw, GLTSYN_ENA(src_idx), GLTSYN_ENA_TSYN_ENA_M);
+
+ /* Enable PHY time sync */
+ err = ice_ptp_init_phy_e810(hw);
+ if (err)
+ goto err_exit;
+
+ /* Clear event status indications for auxiliary pins */
+ (void)rd32(hw, GLTSYN_STAT(src_idx));
+
+ /* Acquire the global hardware lock */
+ if (!ice_ptp_lock(hw)) {
+ err = -EBUSY;
+ goto err_exit;
+ }
+
+ /* Write the increment time value to PHY and LAN */
+ err = ice_ptp_write_incval(hw, ICE_PTP_NOMINAL_INCVAL_E810);
+ if (err) {
+ ice_ptp_unlock(hw);
+ goto err_exit;
+ }
+
+ ts = ktime_to_timespec64(ktime_get_real());
+ /* Write the initial Time value to PHY and LAN */
+ err = ice_ptp_write_init(pf, &ts);
+ if (err) {
+ ice_ptp_unlock(hw);
+ goto err_exit;
+ }
+
+ /* Release the global hardware lock */
+ ice_ptp_unlock(hw);
+
+ /* Ensure we have a clock device */
+ err = ice_ptp_create_clock(pf);
+ if (err)
+ goto err_clk;
+
+ /* Store the PTP clock index for other PFs */
+ ice_set_ptp_clock_index(pf);
+
+ return 0;
+
+err_clk:
+ pf->ptp.clock = NULL;
+err_exit:
+ dev_err(dev, "PTP failed to register clock, err %d\n", err);
+
+ return err;
+}
+
+/**
+ * ice_ptp_init - Initialize the PTP support after device probe or reset
+ * @pf: Board private structure
+ *
+ * This function sets device up for PTP support. The first time it is run, it
+ * will create a clock device. It does not create a clock device if one
+ * already exists. It also reconfigures the device after a reset.
+ */
+void ice_ptp_init(struct ice_pf *pf)
+{
+ struct device *dev = ice_pf_to_dev(pf);
+ struct kthread_worker *kworker;
+ struct ice_hw *hw = &pf->hw;
+ int err;
+
+ /* PTP is currently only supported on E810 devices */
+ if (!ice_is_e810(hw))
+ return;
+
+ /* Check if this PF owns the source timer */
+ if (hw->func_caps.ts_func_info.src_tmr_owned) {
+ err = ice_ptp_init_owner(pf);
+ if (err)
+ return;
+ }
+
+ /* Disable timestamping for both Tx and Rx */
+ ice_ptp_cfg_timestamp(pf, false);
+
+ /* Initialize the PTP port Tx timestamp tracker */
+ ice_ptp_init_tx_e810(pf, &pf->ptp.port.tx);
+
+ /* Initialize work functions */
+ kthread_init_delayed_work(&pf->ptp.work, ice_ptp_periodic_work);
+ kthread_init_work(&pf->ptp.extts_work, ice_ptp_extts_work);
+
+ /* Allocate a kworker for handling work required for the ports
+ * connected to the PTP hardware clock.
+ */
+ kworker = kthread_create_worker(0, "ice-ptp-%s", dev_name(dev));
+ if (IS_ERR(kworker)) {
+ err = PTR_ERR(kworker);
+ goto err_kworker;
+ }
+ pf->ptp.kworker = kworker;
+
+ set_bit(ICE_FLAG_PTP, pf->flags);
+
+ /* Start periodic work going */
+ kthread_queue_delayed_work(pf->ptp.kworker, &pf->ptp.work, 0);
+
+ dev_info(dev, "PTP init successful\n");
+ return;
+
+err_kworker:
+ /* If we registered a PTP clock, release it */
+ if (pf->ptp.clock) {
+ ptp_clock_unregister(pf->ptp.clock);
+ pf->ptp.clock = NULL;
+ }
+ dev_err(dev, "PTP failed %d\n", err);
+}
+
+/**
+ * ice_ptp_release - Disable the driver/HW support and unregister the clock
+ * @pf: Board private structure
+ *
+ * This function handles the cleanup work required from the initialization by
+ * clearing out the important information and unregistering the clock
+ */
+void ice_ptp_release(struct ice_pf *pf)
+{
+ /* Disable timestamping for both Tx and Rx */
+ ice_ptp_cfg_timestamp(pf, false);
+
+ ice_ptp_release_tx_tracker(pf, &pf->ptp.port.tx);
+
+ clear_bit(ICE_FLAG_PTP, pf->flags);
+
+ kthread_cancel_delayed_work_sync(&pf->ptp.work);
+
+ if (pf->ptp.kworker) {
+ kthread_destroy_worker(pf->ptp.kworker);
+ pf->ptp.kworker = NULL;
+ }
+
+ if (!pf->ptp.clock)
+ return;
+
+ ice_clear_ptp_clock_index(pf);
+ ptp_clock_unregister(pf->ptp.clock);
+ pf->ptp.clock = NULL;
+
+ dev_info(ice_pf_to_dev(pf), "Removed PTP clock\n");
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2021, Intel Corporation. */
+
+#ifndef _ICE_PTP_H_
+#define _ICE_PTP_H_
+
+#include <linux/ptp_clock_kernel.h>
+#include <linux/kthread.h>
+
+#include "ice_ptp_hw.h"
+
+enum ice_ptp_pin {
+ GPIO_20 = 0,
+ GPIO_21,
+ GPIO_22,
+ GPIO_23,
+ NUM_ICE_PTP_PIN
+};
+
+struct ice_perout_channel {
+ bool ena;
+ u32 gpio_pin;
+ u64 period;
+ u64 start_time;
+};
+
+/* The ice hardware captures Tx hardware timestamps in the PHY. The timestamp
+ * is stored in a buffer of registers. Depending on the specific hardware,
+ * this buffer might be shared across multiple PHY ports.
+ *
+ * On transmit of a packet to be timestamped, software is responsible for
+ * selecting an open index. Hardware makes no attempt to lock or prevent
+ * re-use of an index for multiple packets.
+ *
+ * To handle this, timestamp indexes must be tracked by software to ensure
+ * that an index is not re-used for multiple transmitted packets. The
+ * structures and functions declared in this file track the available Tx
+ * register indexes, as well as provide storage for the SKB pointers.
+ *
+ * To allow multiple ports to access the shared register block independently,
+ * the blocks are split up so that indexes are assigned to each port based on
+ * hardware logical port number.
+ */
+
+/**
+ * struct ice_tx_tstamp - Tracking for a single Tx timestamp
+ * @skb: pointer to the SKB for this timestamp request
+ * @start: jiffies when the timestamp was first requested
+ *
+ * This structure tracks a single timestamp request. The SKB pointer is
+ * provided when initiating a request. The start time is used to ensure that
+ * we discard old requests that were not fulfilled within a 2 second time
+ * window.
+ */
+struct ice_tx_tstamp {
+ struct sk_buff *skb;
+ unsigned long start;
+};
+
+/**
+ * struct ice_ptp_tx - Tracking structure for all Tx timestamp requests on a port
+ * @work: work function to handle processing of Tx timestamps
+ * @lock: lock to prevent concurrent write to in_use bitmap
+ * @tstamps: array of len to store outstanding requests
+ * @in_use: bitmap of len to indicate which slots are in use
+ * @quad: which quad the timestamps are captured in
+ * @quad_offset: offset into timestamp block of the quad to get the real index
+ * @len: length of the tstamps and in_use fields.
+ * @init: if true, the tracker is initialized;
+ */
+struct ice_ptp_tx {
+ struct kthread_work work;
+ spinlock_t lock; /* lock protecting in_use bitmap */
+ struct ice_tx_tstamp *tstamps;
+ unsigned long *in_use;
+ u8 quad;
+ u8 quad_offset;
+ u8 len;
+ u8 init;
+};
+
+/* Quad and port information for initializing timestamp blocks */
+#define INDEX_PER_QUAD 64
+#define INDEX_PER_PORT (INDEX_PER_QUAD / ICE_PORTS_PER_QUAD)
+
+/**
+ * struct ice_ptp_port - data used to initialize an external port for PTP
+ *
+ * This structure contains PTP data related to the external ports. Currently
+ * it is used for tracking the Tx timestamps of a port. In the future this
+ * structure will also hold information for the E822 port initialization
+ * logic.
+ *
+ * @tx: Tx timestamp tracking for this port
+ */
+struct ice_ptp_port {
+ struct ice_ptp_tx tx;
+};
+
+#define GLTSYN_TGT_H_IDX_MAX 4
+
+/**
+ * struct ice_ptp - data used for integrating with CONFIG_PTP_1588_CLOCK
+ * @port: data for the PHY port initialization procedure
+ * @work: delayed work function for periodic tasks
+ * @extts_work: work function for handling external Tx timestamps
+ * @cached_phc_time: a cached copy of the PHC time for timestamp extension
+ * @ext_ts_chan: the external timestamp channel in use
+ * @ext_ts_irq: the external timestamp IRQ in use
+ * @kworker: kwork thread for handling periodic work
+ * @perout_channels: periodic output data
+ * @info: structure defining PTP hardware capabilities
+ * @clock: pointer to registered PTP clock device
+ * @tstamp_config: hardware timestamping configuration
+ */
+struct ice_ptp {
+ struct ice_ptp_port port;
+ struct kthread_delayed_work work;
+ struct kthread_work extts_work;
+ u64 cached_phc_time;
+ u8 ext_ts_chan;
+ u8 ext_ts_irq;
+ struct kthread_worker *kworker;
+ struct ice_perout_channel perout_channels[GLTSYN_TGT_H_IDX_MAX];
+ struct ptp_clock_info info;
+ struct ptp_clock *clock;
+ struct hwtstamp_config tstamp_config;
+};
+
+#define __ptp_port_to_ptp(p) \
+ container_of((p), struct ice_ptp, port)
+#define ptp_port_to_pf(p) \
+ container_of(__ptp_port_to_ptp((p)), struct ice_pf, ptp)
+
+#define __ptp_info_to_ptp(i) \
+ container_of((i), struct ice_ptp, info)
+#define ptp_info_to_pf(i) \
+ container_of(__ptp_info_to_ptp((i)), struct ice_pf, ptp)
+
+#define PTP_SHARED_CLK_IDX_VALID BIT(31)
+#define ICE_PTP_TS_VALID BIT(0)
+
+/* Per-channel register definitions */
+#define GLTSYN_AUX_OUT(_chan, _idx) (GLTSYN_AUX_OUT_0(_idx) + ((_chan) * 8))
+#define GLTSYN_AUX_IN(_chan, _idx) (GLTSYN_AUX_IN_0(_idx) + ((_chan) * 8))
+#define GLTSYN_CLKO(_chan, _idx) (GLTSYN_CLKO_0(_idx) + ((_chan) * 8))
+#define GLTSYN_TGT_L(_chan, _idx) (GLTSYN_TGT_L_0(_idx) + ((_chan) * 16))
+#define GLTSYN_TGT_H(_chan, _idx) (GLTSYN_TGT_H_0(_idx) + ((_chan) * 16))
+#define GLTSYN_EVNT_L(_chan, _idx) (GLTSYN_EVNT_L_0(_idx) + ((_chan) * 16))
+#define GLTSYN_EVNT_H(_chan, _idx) (GLTSYN_EVNT_H_0(_idx) + ((_chan) * 16))
+#define GLTSYN_EVNT_H_IDX_MAX 3
+
+/* Pin definitions for PTP PPS out */
+#define PPS_CLK_GEN_CHAN 3
+#define PPS_CLK_SRC_CHAN 2
+#define PPS_PIN_INDEX 5
+#define TIME_SYNC_PIN_INDEX 4
+#define E810_N_EXT_TS 3
+#define E810_N_PER_OUT 4
+
+#if IS_ENABLED(CONFIG_PTP_1588_CLOCK)
+struct ice_pf;
+int ice_ptp_set_ts_config(struct ice_pf *pf, struct ifreq *ifr);
+int ice_ptp_get_ts_config(struct ice_pf *pf, struct ifreq *ifr);
+int ice_get_ptp_clock_index(struct ice_pf *pf);
+
+s8 ice_ptp_request_ts(struct ice_ptp_tx *tx, struct sk_buff *skb);
+void ice_ptp_process_ts(struct ice_pf *pf);
+
+void
+ice_ptp_rx_hwtstamp(struct ice_ring *rx_ring,
+ union ice_32b_rx_flex_desc *rx_desc, struct sk_buff *skb);
+void ice_ptp_init(struct ice_pf *pf);
+void ice_ptp_release(struct ice_pf *pf);
+#else /* IS_ENABLED(CONFIG_PTP_1588_CLOCK) */
+static inline int ice_ptp_set_ts_config(struct ice_pf *pf, struct ifreq *ifr)
+{
+ return -EOPNOTSUPP;
+}
+
+static inline int ice_ptp_get_ts_config(struct ice_pf *pf, struct ifreq *ifr)
+{
+ return -EOPNOTSUPP;
+}
+
+static inline int ice_get_ptp_clock_index(struct ice_pf *pf)
+{
+ return -1;
+}
+
+static inline s8
+ice_ptp_request_ts(struct ice_ptp_tx *tx, struct sk_buff *skb)
+{
+ return -1;
+}
+
+static inline void ice_ptp_process_ts(struct ice_pf *pf) { }
+static inline void
+ice_ptp_rx_hwtstamp(struct ice_ring *rx_ring,
+ union ice_32b_rx_flex_desc *rx_desc, struct sk_buff *skb) { }
+static inline void ice_ptp_init(struct ice_pf *pf) { }
+static inline void ice_ptp_release(struct ice_pf *pf) { }
+#endif /* IS_ENABLED(CONFIG_PTP_1588_CLOCK) */
+#endif /* _ICE_PTP_H_ */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (C) 2021, Intel Corporation. */
+
+#include "ice_common.h"
+#include "ice_ptp_hw.h"
+
+/* Low level functions for interacting with and managing the device clock used
+ * for the Precision Time Protocol.
+ *
+ * The ice hardware represents the current time using three registers:
+ *
+ * GLTSYN_TIME_H GLTSYN_TIME_L GLTSYN_TIME_R
+ * +---------------+ +---------------+ +---------------+
+ * | 32 bits | | 32 bits | | 32 bits |
+ * +---------------+ +---------------+ +---------------+
+ *
+ * The registers are incremented every clock tick using a 40bit increment
+ * value defined over two registers:
+ *
+ * GLTSYN_INCVAL_H GLTSYN_INCVAL_L
+ * +---------------+ +---------------+
+ * | 8 bit s | | 32 bits |
+ * +---------------+ +---------------+
+ *
+ * The increment value is added to the GLSTYN_TIME_R and GLSTYN_TIME_L
+ * registers every clock source tick. Depending on the specific device
+ * configuration, the clock source frequency could be one of a number of
+ * values.
+ *
+ * For E810 devices, the increment frequency is 812.5 MHz
+ *
+ * The hardware captures timestamps in the PHY for incoming packets, and for
+ * outgoing packets on request. To support this, the PHY maintains a timer
+ * that matches the lower 64 bits of the global source timer.
+ *
+ * In order to ensure that the PHY timers and the source timer are equivalent,
+ * shadow registers are used to prepare the desired initial values. A special
+ * sync command is issued to trigger copying from the shadow registers into
+ * the appropriate source and PHY registers simultaneously.
+ */
+
+/**
+ * ice_get_ptp_src_clock_index - determine source clock index
+ * @hw: pointer to HW struct
+ *
+ * Determine the source clock index currently in use, based on device
+ * capabilities reported during initialization.
+ */
+u8 ice_get_ptp_src_clock_index(struct ice_hw *hw)
+{
+ return hw->func_caps.ts_func_info.tmr_index_assoc;
+}
+
+/* E810 functions
+ *
+ * The following functions operate on the E810 series devices which use
+ * a separate external PHY.
+ */
+
+/**
+ * ice_read_phy_reg_e810 - Read register from external PHY on E810
+ * @hw: pointer to the HW struct
+ * @addr: the address to read from
+ * @val: On return, the value read from the PHY
+ *
+ * Read a register from the external PHY on the E810 device.
+ */
+static int ice_read_phy_reg_e810(struct ice_hw *hw, u32 addr, u32 *val)
+{
+ struct ice_sbq_msg_input msg = {0};
+ int status;
+
+ msg.msg_addr_low = lower_16_bits(addr);
+ msg.msg_addr_high = upper_16_bits(addr);
+ msg.opcode = ice_sbq_msg_rd;
+ msg.dest_dev = rmn_0;
+
+ status = ice_sbq_rw_reg(hw, &msg);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to send message to PHY, status %d\n",
+ status);
+ return status;
+ }
+
+ *val = msg.data;
+
+ return 0;
+}
+
+/**
+ * ice_write_phy_reg_e810 - Write register on external PHY on E810
+ * @hw: pointer to the HW struct
+ * @addr: the address to writem to
+ * @val: the value to write to the PHY
+ *
+ * Write a value to a register of the external PHY on the E810 device.
+ */
+static int ice_write_phy_reg_e810(struct ice_hw *hw, u32 addr, u32 val)
+{
+ struct ice_sbq_msg_input msg = {0};
+ int status;
+
+ msg.msg_addr_low = lower_16_bits(addr);
+ msg.msg_addr_high = upper_16_bits(addr);
+ msg.opcode = ice_sbq_msg_wr;
+ msg.dest_dev = rmn_0;
+ msg.data = val;
+
+ status = ice_sbq_rw_reg(hw, &msg);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to send message to PHY, status %d\n",
+ status);
+ return status;
+ }
+
+ return 0;
+}
+
+/**
+ * ice_read_phy_tstamp_e810 - Read a PHY timestamp out of the external PHY
+ * @hw: pointer to the HW struct
+ * @lport: the lport to read from
+ * @idx: the timestamp index to read
+ * @tstamp: on return, the 40bit timestamp value
+ *
+ * Read a 40bit timestamp value out of the timestamp block of the external PHY
+ * on the E810 device.
+ */
+static int
+ice_read_phy_tstamp_e810(struct ice_hw *hw, u8 lport, u8 idx, u64 *tstamp)
+{
+ u32 lo_addr, hi_addr, lo, hi;
+ int status;
+
+ lo_addr = TS_EXT(LOW_TX_MEMORY_BANK_START, lport, idx);
+ hi_addr = TS_EXT(HIGH_TX_MEMORY_BANK_START, lport, idx);
+
+ status = ice_read_phy_reg_e810(hw, lo_addr, &lo);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to read low PTP timestamp register, status %d\n",
+ status);
+ return status;
+ }
+
+ status = ice_read_phy_reg_e810(hw, hi_addr, &hi);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to read high PTP timestamp register, status %d\n",
+ status);
+ return status;
+ }
+
+ /* For E810 devices, the timestamp is reported with the lower 32 bits
+ * in the low register, and the upper 8 bits in the high register.
+ */
+ *tstamp = ((u64)hi) << TS_HIGH_S | ((u64)lo & TS_LOW_M);
+
+ return 0;
+}
+
+/**
+ * ice_clear_phy_tstamp_e810 - Clear a timestamp from the external PHY
+ * @hw: pointer to the HW struct
+ * @lport: the lport to read from
+ * @idx: the timestamp index to reset
+ *
+ * Clear a timestamp, resetting its valid bit, from the timestamp block of the
+ * external PHY on the E810 device.
+ */
+static int ice_clear_phy_tstamp_e810(struct ice_hw *hw, u8 lport, u8 idx)
+{
+ u32 lo_addr, hi_addr;
+ int status;
+
+ lo_addr = TS_EXT(LOW_TX_MEMORY_BANK_START, lport, idx);
+ hi_addr = TS_EXT(HIGH_TX_MEMORY_BANK_START, lport, idx);
+
+ status = ice_write_phy_reg_e810(hw, lo_addr, 0);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to clear low PTP timestamp register, status %d\n",
+ status);
+ return status;
+ }
+
+ status = ice_write_phy_reg_e810(hw, hi_addr, 0);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to clear high PTP timestamp register, status %d\n",
+ status);
+ return status;
+ }
+
+ return 0;
+}
+
+/**
+ * ice_ptp_init_phy_e810 - Enable PTP function on the external PHY
+ * @hw: pointer to HW struct
+ *
+ * Enable the timesync PTP functionality for the external PHY connected to
+ * this function.
+ */
+int ice_ptp_init_phy_e810(struct ice_hw *hw)
+{
+ int status;
+ u8 tmr_idx;
+
+ tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned;
+ status = ice_write_phy_reg_e810(hw, ETH_GLTSYN_ENA(tmr_idx),
+ GLTSYN_ENA_TSYN_ENA_M);
+ if (status)
+ ice_debug(hw, ICE_DBG_PTP, "PTP failed in ena_phy_time_syn %d\n",
+ status);
+
+ return status;
+}
+
+/**
+ * ice_ptp_prep_phy_time_e810 - Prepare PHY port with initial time
+ * @hw: Board private structure
+ * @time: Time to initialize the PHY port clock to
+ *
+ * Program the PHY port ETH_GLTSYN_SHTIME registers in preparation setting the
+ * initial clock time. The time will not actually be programmed until the
+ * driver issues an INIT_TIME command.
+ *
+ * The time value is the upper 32 bits of the PHY timer, usually in units of
+ * nominal nanoseconds.
+ */
+static int ice_ptp_prep_phy_time_e810(struct ice_hw *hw, u32 time)
+{
+ int status;
+ u8 tmr_idx;
+
+ tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned;
+ status = ice_write_phy_reg_e810(hw, ETH_GLTSYN_SHTIME_0(tmr_idx), 0);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to write SHTIME_0, status %d\n",
+ status);
+ return status;
+ }
+
+ status = ice_write_phy_reg_e810(hw, ETH_GLTSYN_SHTIME_L(tmr_idx), time);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to write SHTIME_L, status %d\n",
+ status);
+ return status;
+ }
+
+ return 0;
+}
+
+/**
+ * ice_ptp_prep_phy_adj_e810 - Prep PHY port for a time adjustment
+ * @hw: pointer to HW struct
+ * @adj: adjustment value to program
+ *
+ * Prepare the PHY port for an atomic adjustment by programming the PHY
+ * ETH_GLTSYN_SHADJ_L and ETH_GLTSYN_SHADJ_H registers. The actual adjustment
+ * is completed by issuing an ADJ_TIME sync command.
+ *
+ * The adjustment value only contains the portion used for the upper 32bits of
+ * the PHY timer, usually in units of nominal nanoseconds. Negative
+ * adjustments are supported using 2s complement arithmetic.
+ */
+static int ice_ptp_prep_phy_adj_e810(struct ice_hw *hw, s32 adj)
+{
+ int status;
+ u8 tmr_idx;
+
+ tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned;
+
+ /* Adjustments are represented as signed 2's complement values in
+ * nanoseconds. Sub-nanosecond adjustment is not supported.
+ */
+ status = ice_write_phy_reg_e810(hw, ETH_GLTSYN_SHADJ_L(tmr_idx), 0);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to write adj to PHY SHADJ_L, status %d\n",
+ status);
+ return status;
+ }
+
+ status = ice_write_phy_reg_e810(hw, ETH_GLTSYN_SHADJ_H(tmr_idx), adj);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to write adj to PHY SHADJ_H, status %d\n",
+ status);
+ return status;
+ }
+
+ return 0;
+}
+
+/**
+ * ice_ptp_prep_phy_incval_e810 - Prep PHY port increment value change
+ * @hw: pointer to HW struct
+ * @incval: The new 40bit increment value to prepare
+ *
+ * Prepare the PHY port for a new increment value by programming the PHY
+ * ETH_GLTSYN_SHADJ_L and ETH_GLTSYN_SHADJ_H registers. The actual change is
+ * completed by issuing an INIT_INCVAL command.
+ */
+static int ice_ptp_prep_phy_incval_e810(struct ice_hw *hw, u64 incval)
+{
+ u32 high, low;
+ int status;
+ u8 tmr_idx;
+
+ tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned;
+ low = lower_32_bits(incval);
+ high = upper_32_bits(incval);
+
+ status = ice_write_phy_reg_e810(hw, ETH_GLTSYN_SHADJ_L(tmr_idx), low);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to write incval to PHY SHADJ_L, status %d\n",
+ status);
+ return status;
+ }
+
+ status = ice_write_phy_reg_e810(hw, ETH_GLTSYN_SHADJ_H(tmr_idx), high);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to write incval PHY SHADJ_H, status %d\n",
+ status);
+ return status;
+ }
+
+ return 0;
+}
+
+/**
+ * ice_ptp_port_cmd_e810 - Prepare all external PHYs for a timer command
+ * @hw: pointer to HW struct
+ * @cmd: Command to be sent to the port
+ *
+ * Prepare the external PHYs connected to this device for a timer sync
+ * command.
+ */
+static int ice_ptp_port_cmd_e810(struct ice_hw *hw, enum ice_ptp_tmr_cmd cmd)
+{
+ u32 cmd_val, val;
+ int status;
+
+ switch (cmd) {
+ case INIT_TIME:
+ cmd_val = GLTSYN_CMD_INIT_TIME;
+ break;
+ case INIT_INCVAL:
+ cmd_val = GLTSYN_CMD_INIT_INCVAL;
+ break;
+ case ADJ_TIME:
+ cmd_val = GLTSYN_CMD_ADJ_TIME;
+ break;
+ case READ_TIME:
+ cmd_val = GLTSYN_CMD_READ_TIME;
+ break;
+ case ADJ_TIME_AT_TIME:
+ cmd_val = GLTSYN_CMD_ADJ_INIT_TIME;
+ break;
+ }
+
+ /* Read, modify, write */
+ status = ice_read_phy_reg_e810(hw, ETH_GLTSYN_CMD, &val);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to read GLTSYN_CMD, status %d\n", status);
+ return status;
+ }
+
+ /* Modify necessary bits only and perform write */
+ val &= ~TS_CMD_MASK_E810;
+ val |= cmd_val;
+
+ status = ice_write_phy_reg_e810(hw, ETH_GLTSYN_CMD, val);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to write back GLTSYN_CMD, status %d\n", status);
+ return status;
+ }
+
+ return 0;
+}
+
+/* Device agnostic functions
+ *
+ * The following functions implement useful behavior to hide the differences
+ * between E810 and other devices. They call the device-specific
+ * implementations where necessary.
+ *
+ * Currently, the driver only supports E810, but future work will enable
+ * support for E822-based devices.
+ */
+
+/**
+ * ice_ptp_lock - Acquire PTP global semaphore register lock
+ * @hw: pointer to the HW struct
+ *
+ * Acquire the global PTP hardware semaphore lock. Returns true if the lock
+ * was acquired, false otherwise.
+ *
+ * The PFTSYN_SEM register sets the busy bit on read, returning the previous
+ * value. If software sees the busy bit cleared, this means that this function
+ * acquired the lock (and the busy bit is now set). If software sees the busy
+ * bit set, it means that another function acquired the lock.
+ *
+ * Software must clear the busy bit with a write to release the lock for other
+ * functions when done.
+ */
+bool ice_ptp_lock(struct ice_hw *hw)
+{
+ u32 hw_lock;
+ int i;
+
+#define MAX_TRIES 5
+
+ for (i = 0; i < MAX_TRIES; i++) {
+ hw_lock = rd32(hw, PFTSYN_SEM + (PFTSYN_SEM_BYTES * hw->pf_id));
+ hw_lock = hw_lock & PFTSYN_SEM_BUSY_M;
+ if (!hw_lock)
+ break;
+
+ /* Somebody is holding the lock */
+ usleep_range(10000, 20000);
+ }
+
+ return !hw_lock;
+}
+
+/**
+ * ice_ptp_unlock - Release PTP global semaphore register lock
+ * @hw: pointer to the HW struct
+ *
+ * Release the global PTP hardware semaphore lock. This is done by writing to
+ * the PFTSYN_SEM register.
+ */
+void ice_ptp_unlock(struct ice_hw *hw)
+{
+ wr32(hw, PFTSYN_SEM + (PFTSYN_SEM_BYTES * hw->pf_id), 0);
+}
+
+/**
+ * ice_ptp_src_cmd - Prepare source timer for a timer command
+ * @hw: pointer to HW structure
+ * @cmd: Timer command
+ *
+ * Prepare the source timer for an upcoming timer sync command.
+ */
+static void ice_ptp_src_cmd(struct ice_hw *hw, enum ice_ptp_tmr_cmd cmd)
+{
+ u32 cmd_val;
+ u8 tmr_idx;
+
+ tmr_idx = ice_get_ptp_src_clock_index(hw);
+ cmd_val = tmr_idx << SEL_CPK_SRC;
+
+ switch (cmd) {
+ case INIT_TIME:
+ cmd_val |= GLTSYN_CMD_INIT_TIME;
+ break;
+ case INIT_INCVAL:
+ cmd_val |= GLTSYN_CMD_INIT_INCVAL;
+ break;
+ case ADJ_TIME:
+ cmd_val |= GLTSYN_CMD_ADJ_TIME;
+ break;
+ case ADJ_TIME_AT_TIME:
+ cmd_val |= GLTSYN_CMD_ADJ_INIT_TIME;
+ break;
+ case READ_TIME:
+ cmd_val |= GLTSYN_CMD_READ_TIME;
+ break;
+ }
+
+ wr32(hw, GLTSYN_CMD, cmd_val);
+}
+
+/**
+ * ice_ptp_tmr_cmd - Prepare and trigger a timer sync command
+ * @hw: pointer to HW struct
+ * @cmd: the command to issue
+ *
+ * Prepare the source timer and PHY timers and then trigger the requested
+ * command. This causes the shadow registers previously written in preparation
+ * for the command to be synchronously applied to both the source and PHY
+ * timers.
+ */
+static int ice_ptp_tmr_cmd(struct ice_hw *hw, enum ice_ptp_tmr_cmd cmd)
+{
+ int status;
+
+ /* First, prepare the source timer */
+ ice_ptp_src_cmd(hw, cmd);
+
+ /* Next, prepare the ports */
+ status = ice_ptp_port_cmd_e810(hw, cmd);
+ if (status) {
+ ice_debug(hw, ICE_DBG_PTP, "Failed to prepare PHY ports for timer command %u, status %d\n",
+ cmd, status);
+ return status;
+ }
+
+ /* Write the sync command register to drive both source and PHY timer commands
+ * synchronously
+ */
+ wr32(hw, GLTSYN_CMD_SYNC, SYNC_EXEC_CMD);
+
+ return 0;
+}
+
+/**
+ * ice_ptp_init_time - Initialize device time to provided value
+ * @hw: pointer to HW struct
+ * @time: 64bits of time (GLTSYN_TIME_L and GLTSYN_TIME_H)
+ *
+ * Initialize the device to the specified time provided. This requires a three
+ * step process:
+ *
+ * 1) write the new init time to the source timer shadow registers
+ * 2) write the new init time to the PHY timer shadow registers
+ * 3) issue an init_time timer command to synchronously switch both the source
+ * and port timers to the new init time value at the next clock cycle.
+ */
+int ice_ptp_init_time(struct ice_hw *hw, u64 time)
+{
+ int status;
+ u8 tmr_idx;
+
+ tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned;
+
+ /* Source timers */
+ wr32(hw, GLTSYN_SHTIME_L(tmr_idx), lower_32_bits(time));
+ wr32(hw, GLTSYN_SHTIME_H(tmr_idx), upper_32_bits(time));
+ wr32(hw, GLTSYN_SHTIME_0(tmr_idx), 0);
+
+ /* PHY timers */
+ /* Fill Rx and Tx ports and send msg to PHY */
+ status = ice_ptp_prep_phy_time_e810(hw, time & 0xFFFFFFFF);
+ if (status)
+ return status;
+
+ return ice_ptp_tmr_cmd(hw, INIT_TIME);
+}
+
+/**
+ * ice_ptp_write_incval - Program PHC with new increment value
+ * @hw: pointer to HW struct
+ * @incval: Source timer increment value per clock cycle
+ *
+ * Program the PHC with a new increment value. This requires a three-step
+ * process:
+ *
+ * 1) Write the increment value to the source timer shadow registers
+ * 2) Write the increment value to the PHY timer shadow registers
+ * 3) Issue an INIT_INCVAL timer command to synchronously switch both the
+ * source and port timers to the new increment value at the next clock
+ * cycle.
+ */
+int ice_ptp_write_incval(struct ice_hw *hw, u64 incval)
+{
+ int status;
+ u8 tmr_idx;
+
+ tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned;
+
+ /* Shadow Adjust */
+ wr32(hw, GLTSYN_SHADJ_L(tmr_idx), lower_32_bits(incval));
+ wr32(hw, GLTSYN_SHADJ_H(tmr_idx), upper_32_bits(incval));
+
+ status = ice_ptp_prep_phy_incval_e810(hw, incval);
+ if (status)
+ return status;
+
+ return ice_ptp_tmr_cmd(hw, INIT_INCVAL);
+}
+
+/**
+ * ice_ptp_write_incval_locked - Program new incval while holding semaphore
+ * @hw: pointer to HW struct
+ * @incval: Source timer increment value per clock cycle
+ *
+ * Program a new PHC incval while holding the PTP semaphore.
+ */
+int ice_ptp_write_incval_locked(struct ice_hw *hw, u64 incval)
+{
+ int status;
+
+ if (!ice_ptp_lock(hw))
+ return -EBUSY;
+
+ status = ice_ptp_write_incval(hw, incval);
+
+ ice_ptp_unlock(hw);
+
+ return status;
+}
+
+/**
+ * ice_ptp_adj_clock - Adjust PHC clock time atomically
+ * @hw: pointer to HW struct
+ * @adj: Adjustment in nanoseconds
+ *
+ * Perform an atomic adjustment of the PHC time by the specified number of
+ * nanoseconds. This requires a three-step process:
+ *
+ * 1) Write the adjustment to the source timer shadow registers
+ * 2) Write the adjustment to the PHY timer shadow registers
+ * 3) Issue an ADJ_TIME timer command to synchronously apply the adjustment to
+ * both the source and port timers at the next clock cycle.
+ */
+int ice_ptp_adj_clock(struct ice_hw *hw, s32 adj)
+{
+ int status;
+ u8 tmr_idx;
+
+ tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned;
+
+ /* Write the desired clock adjustment into the GLTSYN_SHADJ register.
+ * For an ADJ_TIME command, this set of registers represents the value
+ * to add to the clock time. It supports subtraction by interpreting
+ * the value as a 2's complement integer.
+ */
+ wr32(hw, GLTSYN_SHADJ_L(tmr_idx), 0);
+ wr32(hw, GLTSYN_SHADJ_H(tmr_idx), adj);
+
+ status = ice_ptp_prep_phy_adj_e810(hw, adj);
+ if (status)
+ return status;
+
+ return ice_ptp_tmr_cmd(hw, ADJ_TIME);
+}
+
+/**
+ * ice_read_phy_tstamp - Read a PHY timestamp from the timestamo block
+ * @hw: pointer to the HW struct
+ * @block: the block to read from
+ * @idx: the timestamp index to read
+ * @tstamp: on return, the 40bit timestamp value
+ *
+ * Read a 40bit timestamp value out of the timestamp block.
+ */
+int ice_read_phy_tstamp(struct ice_hw *hw, u8 block, u8 idx, u64 *tstamp)
+{
+ return ice_read_phy_tstamp_e810(hw, block, idx, tstamp);
+}
+
+/**
+ * ice_clear_phy_tstamp - Clear a timestamp from the timestamp block
+ * @hw: pointer to the HW struct
+ * @block: the block to read from
+ * @idx: the timestamp index to reset
+ *
+ * Clear a timestamp, resetting its valid bit, from the timestamp block.
+ */
+int ice_clear_phy_tstamp(struct ice_hw *hw, u8 block, u8 idx)
+{
+ return ice_clear_phy_tstamp_e810(hw, block, idx);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2021, Intel Corporation. */
+
+#ifndef _ICE_PTP_HW_H_
+#define _ICE_PTP_HW_H_
+
+enum ice_ptp_tmr_cmd {
+ INIT_TIME,
+ INIT_INCVAL,
+ ADJ_TIME,
+ ADJ_TIME_AT_TIME,
+ READ_TIME
+};
+
+/* Increment value to generate nanoseconds in the GLTSYN_TIME_L register for
+ * the E810 devices. Based off of a PLL with an 812.5 MHz frequency.
+ */
+#define ICE_PTP_NOMINAL_INCVAL_E810 0x13b13b13bULL
+
+/* Device agnostic functions */
+u8 ice_get_ptp_src_clock_index(struct ice_hw *hw);
+bool ice_ptp_lock(struct ice_hw *hw);
+void ice_ptp_unlock(struct ice_hw *hw);
+int ice_ptp_init_time(struct ice_hw *hw, u64 time);
+int ice_ptp_write_incval(struct ice_hw *hw, u64 incval);
+int ice_ptp_write_incval_locked(struct ice_hw *hw, u64 incval);
+int ice_ptp_adj_clock(struct ice_hw *hw, s32 adj);
+int ice_read_phy_tstamp(struct ice_hw *hw, u8 block, u8 idx, u64 *tstamp);
+int ice_clear_phy_tstamp(struct ice_hw *hw, u8 block, u8 idx);
+
+/* E810 family functions */
+int ice_ptp_init_phy_e810(struct ice_hw *hw);
+
+#define PFTSYN_SEM_BYTES 4
+
+/* PHY timer commands */
+#define SEL_CPK_SRC 8
+
+/* Time Sync command Definitions */
+#define GLTSYN_CMD_INIT_TIME BIT(0)
+#define GLTSYN_CMD_INIT_INCVAL BIT(1)
+#define GLTSYN_CMD_ADJ_TIME BIT(2)
+#define GLTSYN_CMD_ADJ_INIT_TIME (BIT(2) | BIT(3))
+#define GLTSYN_CMD_READ_TIME BIT(7)
+
+#define TS_CMD_MASK_E810 0xFF
+#define SYNC_EXEC_CMD 0x3
+
+/* E810 timesync enable register */
+#define ETH_GLTSYN_ENA(_i) (0x03000348 + ((_i) * 4))
+
+/* E810 shadow init time registers */
+#define ETH_GLTSYN_SHTIME_0(i) (0x03000368 + ((i) * 32))
+#define ETH_GLTSYN_SHTIME_L(i) (0x0300036C + ((i) * 32))
+
+/* E810 shadow time adjust registers */
+#define ETH_GLTSYN_SHADJ_L(_i) (0x03000378 + ((_i) * 32))
+#define ETH_GLTSYN_SHADJ_H(_i) (0x0300037C + ((_i) * 32))
+
+/* E810 timer command register */
+#define ETH_GLTSYN_CMD 0x03000344
+
+/* Source timer incval macros */
+#define INCVAL_HIGH_M 0xFF
+
+/* Timestamp block macros */
+#define TS_LOW_M 0xFFFFFFFF
+#define TS_HIGH_S 32
+
+#define BYTES_PER_IDX_ADDR_L_U 8
+
+/* External PHY timestamp address */
+#define TS_EXT(a, port, idx) ((a) + (0x1000 * (port)) + \
+ ((idx) * BYTES_PER_IDX_ADDR_L_U))
+
+#define LOW_TX_MEMORY_BANK_START 0x03090000
+#define HIGH_TX_MEMORY_BANK_START 0x03090004
+
+#endif /* _ICE_PTP_HW_H_ */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2021, Intel Corporation. */
+
+#ifndef _ICE_SBQ_CMD_H_
+#define _ICE_SBQ_CMD_H_
+
+/* This header file defines the Sideband Queue commands, error codes and
+ * descriptor format. It is shared between Firmware and Software.
+ */
+
+/* Sideband Queue command structure and opcodes */
+enum ice_sbq_opc {
+ /* Sideband Queue commands */
+ ice_sbq_opc_neigh_dev_req = 0x0C00,
+ ice_sbq_opc_neigh_dev_ev = 0x0C01
+};
+
+/* Sideband Queue descriptor. Indirect command
+ * and non posted
+ */
+struct ice_sbq_cmd_desc {
+ __le16 flags;
+ __le16 opcode;
+ __le16 datalen;
+ __le16 cmd_retval;
+
+ /* Opaque message data */
+ __le32 cookie_high;
+ __le32 cookie_low;
+
+ union {
+ __le16 cmd_len;
+ __le16 cmpl_len;
+ } param0;
+
+ u8 reserved[6];
+ __le32 addr_high;
+ __le32 addr_low;
+};
+
+struct ice_sbq_evt_desc {
+ __le16 flags;
+ __le16 opcode;
+ __le16 datalen;
+ __le16 cmd_retval;
+ u8 data[24];
+};
+
+enum ice_sbq_msg_dev {
+ rmn_0 = 0x02,
+ rmn_1 = 0x03,
+ rmn_2 = 0x04,
+ cgu = 0x06
+};
+
+enum ice_sbq_msg_opcode {
+ ice_sbq_msg_rd = 0x00,
+ ice_sbq_msg_wr = 0x01
+};
+
+#define ICE_SBQ_MSG_FLAGS 0x40
+#define ICE_SBQ_MSG_SBE_FBE 0x0F
+
+struct ice_sbq_msg_req {
+ u8 dest_dev;
+ u8 src_dev;
+ u8 opcode;
+ u8 flags;
+ u8 sbe_fbe;
+ u8 func_id;
+ __le16 msg_addr_low;
+ __le32 msg_addr_high;
+ __le32 data;
+};
+
+struct ice_sbq_msg_cmpl {
+ u8 dest_dev;
+ u8 src_dev;
+ u8 opcode;
+ u8 flags;
+ __le32 data;
+};
+
+/* Internal struct */
+struct ice_sbq_msg_input {
+ u8 dest_dev;
+ u8 opcode;
+ u16 msg_addr_low;
+ u32 msg_addr_high;
+ u32 data;
+};
+#endif /* _ICE_SBQ_CMD_H_ */
}
/**
+ * ice_alloc_rdma_q_ctx - allocate RDMA queue contexts for the given VSI and TC
+ * @hw: pointer to the HW struct
+ * @vsi_handle: VSI handle
+ * @tc: TC number
+ * @new_numqs: number of queues
+ */
+static enum ice_status
+ice_alloc_rdma_q_ctx(struct ice_hw *hw, u16 vsi_handle, u8 tc, u16 new_numqs)
+{
+ struct ice_vsi_ctx *vsi_ctx;
+ struct ice_q_ctx *q_ctx;
+
+ vsi_ctx = ice_get_vsi_ctx(hw, vsi_handle);
+ if (!vsi_ctx)
+ return ICE_ERR_PARAM;
+ /* allocate RDMA queue contexts */
+ if (!vsi_ctx->rdma_q_ctx[tc]) {
+ vsi_ctx->rdma_q_ctx[tc] = devm_kcalloc(ice_hw_to_dev(hw),
+ new_numqs,
+ sizeof(*q_ctx),
+ GFP_KERNEL);
+ if (!vsi_ctx->rdma_q_ctx[tc])
+ return ICE_ERR_NO_MEMORY;
+ vsi_ctx->num_rdma_q_entries[tc] = new_numqs;
+ return 0;
+ }
+ /* num queues are increased, update the queue contexts */
+ if (new_numqs > vsi_ctx->num_rdma_q_entries[tc]) {
+ u16 prev_num = vsi_ctx->num_rdma_q_entries[tc];
+
+ q_ctx = devm_kcalloc(ice_hw_to_dev(hw), new_numqs,
+ sizeof(*q_ctx), GFP_KERNEL);
+ if (!q_ctx)
+ return ICE_ERR_NO_MEMORY;
+ memcpy(q_ctx, vsi_ctx->rdma_q_ctx[tc],
+ prev_num * sizeof(*q_ctx));
+ devm_kfree(ice_hw_to_dev(hw), vsi_ctx->rdma_q_ctx[tc]);
+ vsi_ctx->rdma_q_ctx[tc] = q_ctx;
+ vsi_ctx->num_rdma_q_entries[tc] = new_numqs;
+ }
+ return 0;
+}
+
+/**
* ice_aq_rl_profile - performs a rate limiting task
* @hw: pointer to the HW struct
* @opcode: opcode for add, query, or remove profile(s)
if (!vsi_ctx)
return ICE_ERR_PARAM;
- prev_numqs = vsi_ctx->sched.max_lanq[tc];
+ if (owner == ICE_SCHED_NODE_OWNER_LAN)
+ prev_numqs = vsi_ctx->sched.max_lanq[tc];
+ else
+ prev_numqs = vsi_ctx->sched.max_rdmaq[tc];
/* num queues are not changed or less than the previous number */
if (new_numqs <= prev_numqs)
return status;
- status = ice_alloc_lan_q_ctx(hw, vsi_handle, tc, new_numqs);
- if (status)
- return status;
+ if (owner == ICE_SCHED_NODE_OWNER_LAN) {
+ status = ice_alloc_lan_q_ctx(hw, vsi_handle, tc, new_numqs);
+ if (status)
+ return status;
+ } else {
+ status = ice_alloc_rdma_q_ctx(hw, vsi_handle, tc, new_numqs);
+ if (status)
+ return status;
+ }
if (new_numqs)
ice_sched_calc_vsi_child_nodes(hw, new_numqs, new_num_nodes);
new_num_nodes, owner);
if (status)
return status;
- vsi_ctx->sched.max_lanq[tc] = new_numqs;
+ if (owner == ICE_SCHED_NODE_OWNER_LAN)
+ vsi_ctx->sched.max_lanq[tc] = new_numqs;
+ else
+ vsi_ctx->sched.max_rdmaq[tc] = new_numqs;
return 0;
}
* recreate the child nodes all the time in these cases.
*/
vsi_ctx->sched.max_lanq[tc] = 0;
+ vsi_ctx->sched.max_rdmaq[tc] = 0;
}
/* update the VSI child nodes */
}
if (owner == ICE_SCHED_NODE_OWNER_LAN)
vsi_ctx->sched.max_lanq[i] = 0;
+ else
+ vsi_ctx->sched.max_rdmaq[i] = 0;
}
status = 0;
ice_sched_assoc_vsi_to_agg(struct ice_port_info *pi, u32 agg_id,
u16 vsi_handle, unsigned long *tc_bitmap)
{
- struct ice_sched_agg_vsi_info *agg_vsi_info;
- struct ice_sched_agg_info *agg_info;
+ struct ice_sched_agg_vsi_info *agg_vsi_info, *old_agg_vsi_info = NULL;
+ struct ice_sched_agg_info *agg_info, *old_agg_info;
enum ice_status status = 0;
struct ice_hw *hw = pi->hw;
u8 tc;
agg_info = ice_get_agg_info(hw, agg_id);
if (!agg_info)
return ICE_ERR_PARAM;
+ /* If the VSI is already part of another aggregator then update
+ * its VSI info list
+ */
+ old_agg_info = ice_get_vsi_agg_info(hw, vsi_handle);
+ if (old_agg_info && old_agg_info != agg_info) {
+ struct ice_sched_agg_vsi_info *vtmp;
+
+ list_for_each_entry_safe(old_agg_vsi_info, vtmp,
+ &old_agg_info->agg_vsi_list,
+ list_entry)
+ if (old_agg_vsi_info->vsi_handle == vsi_handle)
+ break;
+ }
+
/* check if entry already exist */
agg_vsi_info = ice_get_agg_vsi_info(agg_info, vsi_handle);
if (!agg_vsi_info) {
break;
set_bit(tc, agg_vsi_info->tc_bitmap);
+ if (old_agg_vsi_info)
+ clear_bit(tc, old_agg_vsi_info->tc_bitmap);
+ }
+ if (old_agg_vsi_info && !old_agg_vsi_info->tc_bitmap[0]) {
+ list_del(&old_agg_vsi_info->list_entry);
+ devm_kfree(ice_hw_to_dev(pi->hw), old_agg_vsi_info);
}
return status;
}
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2018, Intel Corporation. */
+#include "ice_lib.h"
#include "ice_switch.h"
#define ICE_ETH_DA_OFFSET 0
devm_kfree(ice_hw_to_dev(hw), vsi->lan_q_ctx[i]);
vsi->lan_q_ctx[i] = NULL;
}
+ if (vsi->rdma_q_ctx[i]) {
+ devm_kfree(ice_hw_to_dev(hw), vsi->rdma_q_ctx[i]);
+ vsi->rdma_q_ctx[i] = NULL;
+ }
}
}
}
/**
+ * ice_cfg_rdma_fltr - enable/disable RDMA filtering on VSI
+ * @hw: pointer to HW struct
+ * @vsi_handle: VSI SW index
+ * @enable: boolean for enable/disable
+ */
+int
+ice_cfg_rdma_fltr(struct ice_hw *hw, u16 vsi_handle, bool enable)
+{
+ struct ice_vsi_ctx *ctx;
+
+ ctx = ice_get_vsi_ctx(hw, vsi_handle);
+ if (!ctx)
+ return -EIO;
+
+ if (enable)
+ ctx->info.q_opt_flags |= ICE_AQ_VSI_Q_OPT_PE_FLTR_EN;
+ else
+ ctx->info.q_opt_flags &= ~ICE_AQ_VSI_Q_OPT_PE_FLTR_EN;
+
+ return ice_status_to_errno(ice_update_vsi(hw, vsi_handle, ctx, NULL));
+}
+
+/**
* ice_aq_alloc_free_vsi_list
* @hw: pointer to the HW struct
* @vsi_list_id: VSI list ID returned or used for lookup
u8 vf_num;
u16 num_lan_q_entries[ICE_MAX_TRAFFIC_CLASS];
struct ice_q_ctx *lan_q_ctx[ICE_MAX_TRAFFIC_CLASS];
+ u16 num_rdma_q_entries[ICE_MAX_TRAFFIC_CLASS];
+ struct ice_q_ctx *rdma_q_ctx[ICE_MAX_TRAFFIC_CLASS];
};
enum ice_sw_fwd_act_type {
ice_add_eth_mac(struct ice_hw *hw, struct list_head *em_list);
enum ice_status
ice_remove_eth_mac(struct ice_hw *hw, struct list_head *em_list);
+int
+ice_cfg_rdma_fltr(struct ice_hw *hw, u16 vsi_handle, bool enable);
void ice_remove_vsi_fltr(struct ice_hw *hw, u16 vsi_handle);
enum ice_status
ice_add_vlan(struct ice_hw *hw, struct list_head *m_list);
enum ice_status ice_init_def_sw_recp(struct ice_hw *hw);
u16 ice_get_hw_vsi_num(struct ice_hw *hw, u16 vsi_handle);
-bool ice_is_vsi_valid(struct ice_hw *hw, u16 vsi_handle);
enum ice_status ice_replay_vsi_all_fltr(struct ice_hw *hw, u16 vsi_handle);
void ice_rm_all_sw_replay_rule_info(struct ice_hw *hw);
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2021 Intel Corporation. */
+
+/* Modeled on trace-events-sample.h */
+
+/* The trace subsystem name for ice will be "ice".
+ *
+ * This file is named ice_trace.h.
+ *
+ * Since this include file's name is different from the trace
+ * subsystem name, we'll have to define TRACE_INCLUDE_FILE at the end
+ * of this file.
+ */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM ice
+
+/* See trace-events-sample.h for a detailed description of why this
+ * guard clause is different from most normal include files.
+ */
+#if !defined(_ICE_TRACE_H_) || defined(TRACE_HEADER_MULTI_READ)
+#define _ICE_TRACE_H_
+
+#include <linux/tracepoint.h>
+
+/* ice_trace() macro enables shared code to refer to trace points
+ * like:
+ *
+ * trace_ice_example(args...)
+ *
+ * ... as:
+ *
+ * ice_trace(example, args...)
+ *
+ * ... to resolve to the PF version of the tracepoint without
+ * ifdefs, and to allow tracepoints to be disabled entirely at build
+ * time.
+ *
+ * Trace point should always be referred to in the driver via this
+ * macro.
+ *
+ * Similarly, ice_trace_enabled(trace_name) wraps references to
+ * trace_ice_<trace_name>_enabled() functions.
+ * @trace_name: name of tracepoint
+ */
+#define _ICE_TRACE_NAME(trace_name) (trace_##ice##_##trace_name)
+#define ICE_TRACE_NAME(trace_name) _ICE_TRACE_NAME(trace_name)
+
+#define ice_trace(trace_name, args...) ICE_TRACE_NAME(trace_name)(args)
+
+#define ice_trace_enabled(trace_name) ICE_TRACE_NAME(trace_name##_enabled)()
+
+/* This is for events common to PF. Corresponding versions will be named
+ * trace_ice_*. The ice_trace() macro above will select the right trace point
+ * name for the driver.
+ */
+
+/* Begin tracepoints */
+
+/* Global tracepoints */
+
+/* Events related to DIM, q_vectors and ring containers */
+DECLARE_EVENT_CLASS(ice_rx_dim_template,
+ TP_PROTO(struct ice_q_vector *q_vector, struct dim *dim),
+ TP_ARGS(q_vector, dim),
+ TP_STRUCT__entry(__field(struct ice_q_vector *, q_vector)
+ __field(struct dim *, dim)
+ __string(devname, q_vector->rx.ring->netdev->name)),
+
+ TP_fast_assign(__entry->q_vector = q_vector;
+ __entry->dim = dim;
+ __assign_str(devname, q_vector->rx.ring->netdev->name);),
+
+ TP_printk("netdev: %s Rx-Q: %d dim-state: %d dim-profile: %d dim-tune: %d dim-st-right: %d dim-st-left: %d dim-tired: %d",
+ __get_str(devname),
+ __entry->q_vector->rx.ring->q_index,
+ __entry->dim->state,
+ __entry->dim->profile_ix,
+ __entry->dim->tune_state,
+ __entry->dim->steps_right,
+ __entry->dim->steps_left,
+ __entry->dim->tired)
+);
+
+DEFINE_EVENT(ice_rx_dim_template, ice_rx_dim_work,
+ TP_PROTO(struct ice_q_vector *q_vector, struct dim *dim),
+ TP_ARGS(q_vector, dim)
+);
+
+DECLARE_EVENT_CLASS(ice_tx_dim_template,
+ TP_PROTO(struct ice_q_vector *q_vector, struct dim *dim),
+ TP_ARGS(q_vector, dim),
+ TP_STRUCT__entry(__field(struct ice_q_vector *, q_vector)
+ __field(struct dim *, dim)
+ __string(devname, q_vector->tx.ring->netdev->name)),
+
+ TP_fast_assign(__entry->q_vector = q_vector;
+ __entry->dim = dim;
+ __assign_str(devname, q_vector->tx.ring->netdev->name);),
+
+ TP_printk("netdev: %s Tx-Q: %d dim-state: %d dim-profile: %d dim-tune: %d dim-st-right: %d dim-st-left: %d dim-tired: %d",
+ __get_str(devname),
+ __entry->q_vector->tx.ring->q_index,
+ __entry->dim->state,
+ __entry->dim->profile_ix,
+ __entry->dim->tune_state,
+ __entry->dim->steps_right,
+ __entry->dim->steps_left,
+ __entry->dim->tired)
+);
+
+DEFINE_EVENT(ice_tx_dim_template, ice_tx_dim_work,
+ TP_PROTO(struct ice_q_vector *q_vector, struct dim *dim),
+ TP_ARGS(q_vector, dim)
+);
+
+/* Events related to a vsi & ring */
+DECLARE_EVENT_CLASS(ice_tx_template,
+ TP_PROTO(struct ice_ring *ring, struct ice_tx_desc *desc,
+ struct ice_tx_buf *buf),
+
+ TP_ARGS(ring, desc, buf),
+ TP_STRUCT__entry(__field(void *, ring)
+ __field(void *, desc)
+ __field(void *, buf)
+ __string(devname, ring->netdev->name)),
+
+ TP_fast_assign(__entry->ring = ring;
+ __entry->desc = desc;
+ __entry->buf = buf;
+ __assign_str(devname, ring->netdev->name);),
+
+ TP_printk("netdev: %s ring: %pK desc: %pK buf %pK", __get_str(devname),
+ __entry->ring, __entry->desc, __entry->buf)
+);
+
+#define DEFINE_TX_TEMPLATE_OP_EVENT(name) \
+DEFINE_EVENT(ice_tx_template, name, \
+ TP_PROTO(struct ice_ring *ring, \
+ struct ice_tx_desc *desc, \
+ struct ice_tx_buf *buf), \
+ TP_ARGS(ring, desc, buf))
+
+DEFINE_TX_TEMPLATE_OP_EVENT(ice_clean_tx_irq);
+DEFINE_TX_TEMPLATE_OP_EVENT(ice_clean_tx_irq_unmap);
+DEFINE_TX_TEMPLATE_OP_EVENT(ice_clean_tx_irq_unmap_eop);
+
+DECLARE_EVENT_CLASS(ice_rx_template,
+ TP_PROTO(struct ice_ring *ring, union ice_32b_rx_flex_desc *desc),
+
+ TP_ARGS(ring, desc),
+
+ TP_STRUCT__entry(__field(void *, ring)
+ __field(void *, desc)
+ __string(devname, ring->netdev->name)),
+
+ TP_fast_assign(__entry->ring = ring;
+ __entry->desc = desc;
+ __assign_str(devname, ring->netdev->name);),
+
+ TP_printk("netdev: %s ring: %pK desc: %pK", __get_str(devname),
+ __entry->ring, __entry->desc)
+);
+DEFINE_EVENT(ice_rx_template, ice_clean_rx_irq,
+ TP_PROTO(struct ice_ring *ring, union ice_32b_rx_flex_desc *desc),
+ TP_ARGS(ring, desc)
+);
+
+DECLARE_EVENT_CLASS(ice_rx_indicate_template,
+ TP_PROTO(struct ice_ring *ring, union ice_32b_rx_flex_desc *desc,
+ struct sk_buff *skb),
+
+ TP_ARGS(ring, desc, skb),
+
+ TP_STRUCT__entry(__field(void *, ring)
+ __field(void *, desc)
+ __field(void *, skb)
+ __string(devname, ring->netdev->name)),
+
+ TP_fast_assign(__entry->ring = ring;
+ __entry->desc = desc;
+ __entry->skb = skb;
+ __assign_str(devname, ring->netdev->name);),
+
+ TP_printk("netdev: %s ring: %pK desc: %pK skb %pK", __get_str(devname),
+ __entry->ring, __entry->desc, __entry->skb)
+);
+
+DEFINE_EVENT(ice_rx_indicate_template, ice_clean_rx_irq_indicate,
+ TP_PROTO(struct ice_ring *ring, union ice_32b_rx_flex_desc *desc,
+ struct sk_buff *skb),
+ TP_ARGS(ring, desc, skb)
+);
+
+DECLARE_EVENT_CLASS(ice_xmit_template,
+ TP_PROTO(struct ice_ring *ring, struct sk_buff *skb),
+
+ TP_ARGS(ring, skb),
+
+ TP_STRUCT__entry(__field(void *, ring)
+ __field(void *, skb)
+ __string(devname, ring->netdev->name)),
+
+ TP_fast_assign(__entry->ring = ring;
+ __entry->skb = skb;
+ __assign_str(devname, ring->netdev->name);),
+
+ TP_printk("netdev: %s skb: %pK ring: %pK", __get_str(devname),
+ __entry->skb, __entry->ring)
+);
+
+#define DEFINE_XMIT_TEMPLATE_OP_EVENT(name) \
+DEFINE_EVENT(ice_xmit_template, name, \
+ TP_PROTO(struct ice_ring *ring, struct sk_buff *skb), \
+ TP_ARGS(ring, skb))
+
+DEFINE_XMIT_TEMPLATE_OP_EVENT(ice_xmit_frame_ring);
+DEFINE_XMIT_TEMPLATE_OP_EVENT(ice_xmit_frame_ring_drop);
+
+/* End tracepoints */
+
+#endif /* _ICE_TRACE_H_ */
+/* This must be outside ifdef _ICE_TRACE_H */
+
+/* This trace include file is not located in the .../include/trace
+ * with the kernel tracepoint definitions, because we're a loadable
+ * module.
+ */
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE ../../drivers/net/ethernet/intel/ice/ice_trace
+#include <trace/define_trace.h>
#include "ice_txrx_lib.h"
#include "ice_lib.h"
#include "ice.h"
+#include "ice_trace.h"
#include "ice_dcb_lib.h"
#include "ice_xsk.h"
smp_rmb(); /* prevent any other reads prior to eop_desc */
+ ice_trace(clean_tx_irq, tx_ring, tx_desc, tx_buf);
/* if the descriptor isn't done, no work yet to do */
if (!(eop_desc->cmd_type_offset_bsz &
cpu_to_le64(ICE_TX_DESC_DTYPE_DESC_DONE)))
/* unmap remaining buffers */
while (tx_desc != eop_desc) {
+ ice_trace(clean_tx_irq_unmap, tx_ring, tx_desc, tx_buf);
tx_buf++;
tx_desc++;
i++;
dma_unmap_len_set(tx_buf, len, 0);
}
}
+ ice_trace(clean_tx_irq_unmap_eop, tx_ring, tx_desc, tx_buf);
/* move us one more past the eop_desc for start of next pkt */
tx_buf++;
struct bpf_prog *xdp_prog)
{
struct ice_ring *xdp_ring;
- int err;
+ int err, result;
u32 act;
act = bpf_prog_run_xdp(xdp_prog, xdp);
return ICE_XDP_PASS;
case XDP_TX:
xdp_ring = rx_ring->vsi->xdp_rings[smp_processor_id()];
- return ice_xmit_xdp_buff(xdp, xdp_ring);
+ result = ice_xmit_xdp_buff(xdp, xdp_ring);
+ if (result == ICE_XDP_CONSUMED)
+ goto out_failure;
+ return result;
case XDP_REDIRECT:
err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
- return !err ? ICE_XDP_REDIR : ICE_XDP_CONSUMED;
+ if (err)
+ goto out_failure;
+ return ICE_XDP_REDIR;
default:
bpf_warn_invalid_xdp_action(act);
fallthrough;
case XDP_ABORTED:
+out_failure:
trace_xdp_exception(rx_ring->netdev, xdp_prog, act);
fallthrough;
case XDP_DROP:
u16 stat_err_bits;
int rx_buf_pgcnt;
u16 vlan_tag = 0;
- u8 rx_ptype;
+ u16 rx_ptype;
/* get the Rx desc from Rx ring based on 'next_to_clean' */
rx_desc = ICE_RX_DESC(rx_ring, rx_ring->next_to_clean);
*/
dma_rmb();
+ ice_trace(clean_rx_irq, rx_ring, rx_desc);
if (rx_desc->wb.rxdid == FDIR_DESC_RXDID || !rx_ring->netdev) {
struct ice_vsi *ctrl_vsi = rx_ring->vsi;
ice_process_skb_fields(rx_ring, rx_desc, skb, rx_ptype);
+ ice_trace(clean_rx_irq_indicate, rx_ring, rx_desc, skb);
/* send completed skb up the stack */
ice_receive_skb(rx_ring, skb, vlan_tag);
skb = NULL;
}
/**
+ * ice_tstamp - set up context descriptor for hardware timestamp
+ * @tx_ring: pointer to the Tx ring to send buffer on
+ * @skb: pointer to the SKB we're sending
+ * @first: Tx buffer
+ * @off: Tx offload parameters
+ */
+static void
+ice_tstamp(struct ice_ring *tx_ring, struct sk_buff *skb,
+ struct ice_tx_buf *first, struct ice_tx_offload_params *off)
+{
+ s8 idx;
+
+ /* only timestamp the outbound packet if the user has requested it */
+ if (likely(!(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)))
+ return;
+
+ if (!tx_ring->ptp_tx)
+ return;
+
+ /* Tx timestamps cannot be sampled when doing TSO */
+ if (first->tx_flags & ICE_TX_FLAGS_TSO)
+ return;
+
+ /* Grab an open timestamp slot */
+ idx = ice_ptp_request_ts(tx_ring->tx_tstamps, skb);
+ if (idx < 0)
+ return;
+
+ off->cd_qw1 |= (u64)(ICE_TX_DESC_DTYPE_CTX |
+ (ICE_TX_CTX_DESC_TSYN << ICE_TXD_CTX_QW1_CMD_S) |
+ ((u64)idx << ICE_TXD_CTX_QW1_TSO_LEN_S));
+ first->tx_flags |= ICE_TX_FLAGS_TSYN;
+}
+
+/**
* ice_xmit_frame_ring - Sends buffer on Tx ring
* @skb: send buffer
* @tx_ring: ring to send buffer on
struct ice_tx_offload_params offload = { 0 };
struct ice_vsi *vsi = tx_ring->vsi;
struct ice_tx_buf *first;
+ struct ethhdr *eth;
unsigned int count;
int tso, csum;
+ ice_trace(xmit_frame_ring, tx_ring, skb);
+
count = ice_xmit_desc_count(skb);
if (ice_chk_linearize(skb, count)) {
if (__skb_linearize(skb))
goto out_drop;
/* allow CONTROL frames egress from main VSI if FW LLDP disabled */
- if (unlikely(skb->priority == TC_PRIO_CONTROL &&
+ eth = (struct ethhdr *)skb_mac_header(skb);
+ if (unlikely((skb->priority == TC_PRIO_CONTROL ||
+ eth->h_proto == htons(ETH_P_LLDP)) &&
vsi->type == ICE_VSI_PF &&
vsi->port_info->qos_cfg.is_sw_lldp))
offload.cd_qw1 |= (u64)(ICE_TX_DESC_DTYPE_CTX |
ICE_TX_CTX_DESC_SWTCH_UPLINK <<
ICE_TXD_CTX_QW1_CMD_S);
+ ice_tstamp(tx_ring, skb, first, &offload);
+
if (offload.cd_qw1 & ICE_TX_DESC_DTYPE_CTX) {
struct ice_tx_ctx_desc *cdesc;
u16 i = tx_ring->next_to_use;
return NETDEV_TX_OK;
out_drop:
+ ice_trace(xmit_frame_ring_drop, tx_ring, skb);
dev_kfree_skb_any(skb);
return NETDEV_TX_OK;
}
* freed instead of returned like skb packets.
*/
#define ICE_TX_FLAGS_DUMMY_PKT BIT(3)
+#define ICE_TX_FLAGS_TSYN BIT(4)
#define ICE_TX_FLAGS_IPV4 BIT(5)
#define ICE_TX_FLAGS_IPV6 BIT(6)
#define ICE_TX_FLAGS_TUNNEL BIT(7)
u32 txq_teid; /* Added Tx queue TEID */
u16 rx_buf_len;
u8 dcb_tc; /* Traffic class of ring */
+ struct ice_ptp_tx *tx_tstamps;
+ u64 cached_phctime;
+ u8 ptp_rx:1;
+ u8 ptp_tx:1;
} ____cacheline_internodealigned_in_smp;
static inline bool ice_ring_uses_build_skb(struct ice_ring *ring)
* ice_ptype_to_htype - get a hash type
* @ptype: the ptype value from the descriptor
*
- * Returns a hash type to be used by skb_set_hash
+ * Returns appropriate hash type (such as PKT_HASH_TYPE_L2/L3/L4) to be used by
+ * skb_set_hash based on PTYPE as parsed by HW Rx pipeline and is part of
+ * Rx desc.
*/
-static enum pkt_hash_types ice_ptype_to_htype(u8 __always_unused ptype)
+static enum pkt_hash_types ice_ptype_to_htype(u16 ptype)
{
+ struct ice_rx_ptype_decoded decoded = ice_decode_rx_desc_ptype(ptype);
+
+ if (!decoded.known)
+ return PKT_HASH_TYPE_NONE;
+ if (decoded.payload_layer == ICE_RX_PTYPE_PAYLOAD_LAYER_PAY4)
+ return PKT_HASH_TYPE_L4;
+ if (decoded.payload_layer == ICE_RX_PTYPE_PAYLOAD_LAYER_PAY3)
+ return PKT_HASH_TYPE_L3;
+ if (decoded.outer_ip == ICE_RX_PTYPE_OUTER_L2)
+ return PKT_HASH_TYPE_L2;
+
return PKT_HASH_TYPE_NONE;
}
*/
static void
ice_rx_hash(struct ice_ring *rx_ring, union ice_32b_rx_flex_desc *rx_desc,
- struct sk_buff *skb, u8 rx_ptype)
+ struct sk_buff *skb, u16 rx_ptype)
{
struct ice_32b_rx_flex_desc_nic *nic_mdid;
u32 hash;
*/
static void
ice_rx_csum(struct ice_ring *ring, struct sk_buff *skb,
- union ice_32b_rx_flex_desc *rx_desc, u8 ptype)
+ union ice_32b_rx_flex_desc *rx_desc, u16 ptype)
{
struct ice_rx_ptype_decoded decoded;
u16 rx_status0, rx_status1;
void
ice_process_skb_fields(struct ice_ring *rx_ring,
union ice_32b_rx_flex_desc *rx_desc,
- struct sk_buff *skb, u8 ptype)
+ struct sk_buff *skb, u16 ptype)
{
ice_rx_hash(rx_ring, rx_desc, skb, ptype);
skb->protocol = eth_type_trans(skb, rx_ring->netdev);
ice_rx_csum(rx_ring, skb, rx_desc, ptype);
+
+ if (rx_ring->ptp_rx)
+ ice_ptp_rx_hwtstamp(rx_ring, rx_desc, skb);
}
/**
void
ice_process_skb_fields(struct ice_ring *rx_ring,
union ice_32b_rx_flex_desc *rx_desc,
- struct sk_buff *skb, u8 ptype);
+ struct sk_buff *skb, u16 ptype);
void
ice_receive_skb(struct ice_ring *rx_ring, struct sk_buff *skb, u16 vlan_tag);
#endif /* !_ICE_TXRX_LIB_H_ */
#include "ice_lan_tx_rx.h"
#include "ice_flex_type.h"
#include "ice_protocol_type.h"
+#include "ice_sbq_cmd.h"
static inline bool ice_is_tc_ena(unsigned long bitmap, u8 tc)
{
#define ICE_DBG_FLOW BIT_ULL(9)
#define ICE_DBG_SW BIT_ULL(13)
#define ICE_DBG_SCHED BIT_ULL(14)
+#define ICE_DBG_RDMA BIT_ULL(15)
#define ICE_DBG_PKG BIT_ULL(16)
#define ICE_DBG_RES BIT_ULL(17)
+#define ICE_DBG_PTP BIT_ULL(19)
#define ICE_DBG_AQ_MSG BIT_ULL(24)
#define ICE_DBG_AQ_DESC BIT_ULL(25)
#define ICE_DBG_AQ_DESC_BUF BIT_ULL(26)
/* FW update timeout definitions are in milliseconds */
#define ICE_NVM_TIMEOUT 180000
#define ICE_CHANGE_LOCK_TIMEOUT 1000
-#define ICE_GLOBAL_CFG_LOCK_TIMEOUT 3000
+#define ICE_GLOBAL_CFG_LOCK_TIMEOUT 5000
enum ice_aq_res_access_type {
ICE_RES_READ = 1,
u16 max_frame_size;
u16 link_speed;
u16 req_speeds;
+ u8 link_cfg_err;
u8 lse_ena; /* Link Status Event notification */
u8 link_info;
u8 an_info;
u8 rss_table_entry_width; /* RSS Entry width in bits */
u8 dcb;
+ u8 ieee_1588;
+ u8 rdma;
bool nvm_update_pending_nvm;
bool nvm_update_pending_orom;
#define ICE_NVM_MGMT_UNIFIED_UPD_SUPPORT BIT(3)
};
+/* IEEE 1588 TIME_SYNC specific info */
+/* Function specific definitions */
+#define ICE_TS_FUNC_ENA_M BIT(0)
+#define ICE_TS_SRC_TMR_OWND_M BIT(1)
+#define ICE_TS_TMR_ENA_M BIT(2)
+#define ICE_TS_TMR_IDX_OWND_S 4
+#define ICE_TS_TMR_IDX_OWND_M BIT(4)
+#define ICE_TS_CLK_FREQ_S 16
+#define ICE_TS_CLK_FREQ_M ICE_M(0x7, ICE_TS_CLK_FREQ_S)
+#define ICE_TS_CLK_SRC_S 20
+#define ICE_TS_CLK_SRC_M BIT(20)
+#define ICE_TS_TMR_IDX_ASSOC_S 24
+#define ICE_TS_TMR_IDX_ASSOC_M BIT(24)
+
+struct ice_ts_func_info {
+ /* Function specific info */
+ u32 clk_freq;
+ u8 clk_src;
+ u8 tmr_index_assoc;
+ u8 ena;
+ u8 tmr_index_owned;
+ u8 src_tmr_owned;
+ u8 tmr_ena;
+};
+
+/* Device specific definitions */
+#define ICE_TS_TMR0_OWNR_M 0x7
+#define ICE_TS_TMR0_OWND_M BIT(3)
+#define ICE_TS_TMR1_OWNR_S 4
+#define ICE_TS_TMR1_OWNR_M ICE_M(0x7, ICE_TS_TMR1_OWNR_S)
+#define ICE_TS_TMR1_OWND_M BIT(7)
+#define ICE_TS_DEV_ENA_M BIT(24)
+#define ICE_TS_TMR0_ENA_M BIT(25)
+#define ICE_TS_TMR1_ENA_M BIT(26)
+
+struct ice_ts_dev_info {
+ /* Device specific info */
+ u32 ena_ports;
+ u32 tmr_own_map;
+ u32 tmr0_owner;
+ u32 tmr1_owner;
+ u8 tmr0_owned;
+ u8 tmr1_owned;
+ u8 ena;
+ u8 tmr0_ena;
+ u8 tmr1_ena;
+};
+
/* Function specific capabilities */
struct ice_hw_func_caps {
struct ice_hw_common_caps common_cap;
u32 guar_num_vsi;
u32 fd_fltr_guar; /* Number of filters guaranteed */
u32 fd_fltr_best_effort; /* Number of best effort filters */
+ struct ice_ts_func_info ts_func_info;
};
/* Device wide capabilities */
u32 num_vfs_exposed; /* Total number of VFs exposed */
u32 num_vsi_allocd_to_host; /* Excluding EMP VSI */
u32 num_flow_director_fltr; /* Number of FD filters available */
+ struct ice_ts_dev_info ts_dev_info;
u32 num_funcs;
};
u8 tc_num;
u8 owner;
#define ICE_SCHED_NODE_OWNER_LAN 0
+#define ICE_SCHED_NODE_OWNER_RDMA 2
};
/* Access Macros for Tx Sched Elements data */
struct ice_sched_node *ag_node[ICE_MAX_TRAFFIC_CLASS];
struct list_head list_entry;
u16 max_lanq[ICE_MAX_TRAFFIC_CLASS];
+ u16 max_rdmaq[ICE_MAX_TRAFFIC_CLASS];
};
/* driver defines the policy */
/* Control Queue info */
struct ice_ctl_q_info adminq;
+ struct ice_ctl_q_info sbq;
struct ice_ctl_q_info mailboxq;
u8 api_branch; /* API branch version */
u8 ucast_shared; /* true if VSIs can share unicast addr */
+#define ICE_PHY_PER_NAC 1
+#define ICE_MAX_QUAD 2
+#define ICE_NUM_QUAD_TYPE 2
+#define ICE_PORTS_PER_QUAD 4
+#define ICE_PHY_0_LAST_QUAD 1
+#define ICE_PORTS_PER_PHY 8
+#define ICE_NUM_EXTERNAL_PORTS ICE_PORTS_PER_PHY
+
/* Active package version (currently active) */
struct ice_pkg_ver active_pkg_ver;
u32 active_track_id;
*/
clear_bit(ICE_VF_STATE_INIT, vf->vf_states);
- /* VF_MBX_ARQLEN is cleared by PFR, so the driver needs to clear it
- * in the case of VFR. If this is done for PFR, it can mess up VF
- * resets because the VF driver may already have started cleanup
- * by the time we get here.
+ /* VF_MBX_ARQLEN and VF_MBX_ATQLEN are cleared by PFR, so the driver
+ * needs to clear them in the case of VFR/VFLR. If this is done for
+ * PFR, it can mess up VF resets because the VF driver may already
+ * have started cleanup by the time we get here.
*/
- if (!is_pfr)
+ if (!is_pfr) {
wr32(hw, VF_MBX_ARQLEN(vf->vf_id), 0);
+ wr32(hw, VF_MBX_ATQLEN(vf->vf_id), 0);
+ }
/* In the case of a VFLR, the HW has already reset the VF and we
* just need to clean up, so don't hit the VFRTRIG register.
vf->num_mac++;
- if (is_valid_ether_addr(vf->dflt_lan_addr.addr)) {
- status = ice_fltr_add_mac(vsi, vf->dflt_lan_addr.addr,
+ if (is_valid_ether_addr(vf->hw_lan_addr.addr)) {
+ status = ice_fltr_add_mac(vsi, vf->hw_lan_addr.addr,
ICE_FWD_TO_VSI);
if (status) {
dev_err(dev, "failed to add default unicast MAC filter %pM for VF %u, error %s\n",
- &vf->dflt_lan_addr.addr[0], vf->vf_id,
+ &vf->hw_lan_addr.addr[0], vf->vf_id,
ice_stat_str(status));
return ice_status_to_errno(status);
}
vf->num_mac++;
+
+ ether_addr_copy(vf->dev_lan_addr.addr, vf->hw_lan_addr.addr);
}
return 0;
else
promisc_m = ICE_UCAST_PROMISC_BITS;
- vsi = ice_get_vf_vsi(vf);
if (ice_vf_set_vsi_promisc(vf, vsi, promisc_m, true))
dev_err(dev, "disabling promiscuous mode failed\n");
}
ice_vf_ctrl_vsi_release(vf);
ice_vf_pre_vsi_rebuild(vf);
- ice_vf_rebuild_vsi_with_release(vf);
+
+ if (ice_vf_rebuild_vsi_with_release(vf)) {
+ dev_err(dev, "Failed to release and setup the VF%u's VSI\n", vf->vf_id);
+ return false;
+ }
+
ice_vf_post_vsi_rebuild(vf);
/* if the VF has been reset allow it to come up again */
vfres->vsi_res[0].vsi_type = VIRTCHNL_VSI_SRIOV;
vfres->vsi_res[0].num_queue_pairs = vsi->num_txq;
ether_addr_copy(vfres->vsi_res[0].default_mac_addr,
- vf->dflt_lan_addr.addr);
+ vf->hw_lan_addr.addr);
/* match guest capabilities */
vf->driver_caps = vfres->vf_cap_flags;
struct virtchnl_vsi_queue_config_info *qci =
(struct virtchnl_vsi_queue_config_info *)msg;
struct virtchnl_queue_pair_info *qpi;
- u16 num_rxq = 0, num_txq = 0;
struct ice_pf *pf = vf->pf;
struct ice_vsi *vsi;
- int i;
+ int i, q_idx;
if (!test_bit(ICE_VF_STATE_ACTIVE, vf->vf_states)) {
v_ret = VIRTCHNL_STATUS_ERR_PARAM;
v_ret = VIRTCHNL_STATUS_ERR_PARAM;
goto error_param;
}
+
+ q_idx = qpi->rxq.queue_id;
+
+ /* make sure selected "q_idx" is in valid range of queues
+ * for selected "vsi"
+ */
+ if (q_idx >= vsi->alloc_txq || q_idx >= vsi->alloc_rxq) {
+ v_ret = VIRTCHNL_STATUS_ERR_PARAM;
+ goto error_param;
+ }
+
/* copy Tx queue info from VF into VSI */
if (qpi->txq.ring_len > 0) {
- num_txq++;
vsi->tx_rings[i]->dma = qpi->txq.dma_ring_addr;
vsi->tx_rings[i]->count = qpi->txq.ring_len;
+ if (ice_vsi_cfg_single_txq(vsi, vsi->tx_rings, q_idx)) {
+ v_ret = VIRTCHNL_STATUS_ERR_PARAM;
+ goto error_param;
+ }
}
/* copy Rx queue info from VF into VSI */
if (qpi->rxq.ring_len > 0) {
u16 max_frame_size = ice_vc_get_max_frame_size(vf);
- num_rxq++;
vsi->rx_rings[i]->dma = qpi->rxq.dma_ring_addr;
vsi->rx_rings[i]->count = qpi->rxq.ring_len;
v_ret = VIRTCHNL_STATUS_ERR_PARAM;
goto error_param;
}
- }
- vsi->max_frame = qpi->rxq.max_pkt_size;
- /* add space for the port VLAN since the VF driver is not
- * expected to account for it in the MTU calculation
- */
- if (vf->port_vlan_info)
- vsi->max_frame += VLAN_HLEN;
- }
-
- /* VF can request to configure less than allocated queues or default
- * allocated queues. So update the VSI with new number
- */
- vsi->num_txq = num_txq;
- vsi->num_rxq = num_rxq;
- /* All queues of VF VSI are in TC 0 */
- vsi->tc_cfg.tc_info[0].qcount_tx = num_txq;
- vsi->tc_cfg.tc_info[0].qcount_rx = num_rxq;
+ vsi->max_frame = qpi->rxq.max_pkt_size;
+ /* add space for the port VLAN since the VF driver is not
+ * expected to account for it in the MTU calculation
+ */
+ if (vf->port_vlan_info)
+ vsi->max_frame += VLAN_HLEN;
- if (ice_vsi_cfg_lan_txqs(vsi) || ice_vsi_cfg_rxqs(vsi))
- v_ret = VIRTCHNL_STATUS_ERR_ADMIN_QUEUE_ERROR;
+ if (ice_vsi_cfg_single_rxq(vsi, q_idx)) {
+ v_ret = VIRTCHNL_STATUS_ERR_PARAM;
+ goto error_param;
+ }
+ }
+ }
error_param:
/* send the response to the VF */
}
/**
+ * ice_vc_ether_addr_type - get type of virtchnl_ether_addr
+ * @vc_ether_addr: used to extract the type
+ */
+static u8
+ice_vc_ether_addr_type(struct virtchnl_ether_addr *vc_ether_addr)
+{
+ return (vc_ether_addr->type & VIRTCHNL_ETHER_ADDR_TYPE_MASK);
+}
+
+/**
+ * ice_is_vc_addr_legacy - check if the MAC address is from an older VF
+ * @vc_ether_addr: VIRTCHNL structure that contains MAC and type
+ */
+static bool
+ice_is_vc_addr_legacy(struct virtchnl_ether_addr *vc_ether_addr)
+{
+ u8 type = ice_vc_ether_addr_type(vc_ether_addr);
+
+ return (type == VIRTCHNL_ETHER_ADDR_LEGACY);
+}
+
+/**
+ * ice_is_vc_addr_primary - check if the MAC address is the VF's primary MAC
+ * @vc_ether_addr: VIRTCHNL structure that contains MAC and type
+ *
+ * This function should only be called when the MAC address in
+ * virtchnl_ether_addr is a valid unicast MAC
+ */
+static bool
+ice_is_vc_addr_primary(struct virtchnl_ether_addr __maybe_unused *vc_ether_addr)
+{
+ u8 type = ice_vc_ether_addr_type(vc_ether_addr);
+
+ return (type == VIRTCHNL_ETHER_ADDR_PRIMARY);
+}
+
+/**
+ * ice_vfhw_mac_add - update the VF's cached hardware MAC if allowed
+ * @vf: VF to update
+ * @vc_ether_addr: structure from VIRTCHNL with MAC to add
+ */
+static void
+ice_vfhw_mac_add(struct ice_vf *vf, struct virtchnl_ether_addr *vc_ether_addr)
+{
+ u8 *mac_addr = vc_ether_addr->addr;
+
+ if (!is_valid_ether_addr(mac_addr))
+ return;
+
+ /* only allow legacy VF drivers to set the device and hardware MAC if it
+ * is zero and allow new VF drivers to set the hardware MAC if the type
+ * was correctly specified over VIRTCHNL
+ */
+ if ((ice_is_vc_addr_legacy(vc_ether_addr) &&
+ is_zero_ether_addr(vf->hw_lan_addr.addr)) ||
+ ice_is_vc_addr_primary(vc_ether_addr)) {
+ ether_addr_copy(vf->dev_lan_addr.addr, mac_addr);
+ ether_addr_copy(vf->hw_lan_addr.addr, mac_addr);
+ }
+
+ /* hardware and device MACs are already set, but its possible that the
+ * VF driver sent the VIRTCHNL_OP_ADD_ETH_ADDR message before the
+ * VIRTCHNL_OP_DEL_ETH_ADDR when trying to update its MAC, so save it
+ * away for the legacy VF driver case as it will be updated in the
+ * delete flow for this case
+ */
+ if (ice_is_vc_addr_legacy(vc_ether_addr)) {
+ ether_addr_copy(vf->legacy_last_added_umac.addr,
+ mac_addr);
+ vf->legacy_last_added_umac.time_modified = jiffies;
+ }
+}
+
+/**
* ice_vc_add_mac_addr - attempt to add the MAC address passed in
* @vf: pointer to the VF info
* @vsi: pointer to the VF's VSI
- * @mac_addr: MAC address to add
+ * @vc_ether_addr: VIRTCHNL MAC address structure used to add MAC
*/
static int
-ice_vc_add_mac_addr(struct ice_vf *vf, struct ice_vsi *vsi, u8 *mac_addr)
+ice_vc_add_mac_addr(struct ice_vf *vf, struct ice_vsi *vsi,
+ struct virtchnl_ether_addr *vc_ether_addr)
{
struct device *dev = ice_pf_to_dev(vf->pf);
+ u8 *mac_addr = vc_ether_addr->addr;
enum ice_status status;
- /* default unicast MAC already added */
- if (ether_addr_equal(mac_addr, vf->dflt_lan_addr.addr))
+ /* device MAC already added */
+ if (ether_addr_equal(mac_addr, vf->dev_lan_addr.addr))
return 0;
if (is_unicast_ether_addr(mac_addr) && !ice_can_vf_change_mac(vf)) {
return -EIO;
}
- /* Set the default LAN address to the latest unicast MAC address added
- * by the VF. The default LAN address is reported by the PF via
- * ndo_get_vf_config.
- */
- if (is_unicast_ether_addr(mac_addr))
- ether_addr_copy(vf->dflt_lan_addr.addr, mac_addr);
+ ice_vfhw_mac_add(vf, vc_ether_addr);
vf->num_mac++;
}
/**
+ * ice_is_legacy_umac_expired - check if last added legacy unicast MAC expired
+ * @last_added_umac: structure used to check expiration
+ */
+static bool ice_is_legacy_umac_expired(struct ice_time_mac *last_added_umac)
+{
+#define ICE_LEGACY_VF_MAC_CHANGE_EXPIRE_TIME msecs_to_jiffies(3000)
+ return time_is_before_jiffies(last_added_umac->time_modified +
+ ICE_LEGACY_VF_MAC_CHANGE_EXPIRE_TIME);
+}
+
+/**
+ * ice_vfhw_mac_del - update the VF's cached hardware MAC if allowed
+ * @vf: VF to update
+ * @vc_ether_addr: structure from VIRTCHNL with MAC to delete
+ */
+static void
+ice_vfhw_mac_del(struct ice_vf *vf, struct virtchnl_ether_addr *vc_ether_addr)
+{
+ u8 *mac_addr = vc_ether_addr->addr;
+
+ if (!is_valid_ether_addr(mac_addr) ||
+ !ether_addr_equal(vf->dev_lan_addr.addr, mac_addr))
+ return;
+
+ /* allow the device MAC to be repopulated in the add flow and don't
+ * clear the hardware MAC (i.e. hw_lan_addr.addr) here as that is meant
+ * to be persistent on VM reboot and across driver unload/load, which
+ * won't work if we clear the hardware MAC here
+ */
+ eth_zero_addr(vf->dev_lan_addr.addr);
+
+ /* only update cached hardware MAC for legacy VF drivers on delete
+ * because we cannot guarantee order/type of MAC from the VF driver
+ */
+ if (ice_is_vc_addr_legacy(vc_ether_addr) &&
+ !ice_is_legacy_umac_expired(&vf->legacy_last_added_umac)) {
+ ether_addr_copy(vf->dev_lan_addr.addr,
+ vf->legacy_last_added_umac.addr);
+ ether_addr_copy(vf->hw_lan_addr.addr,
+ vf->legacy_last_added_umac.addr);
+ }
+}
+
+/**
* ice_vc_del_mac_addr - attempt to delete the MAC address passed in
* @vf: pointer to the VF info
* @vsi: pointer to the VF's VSI
- * @mac_addr: MAC address to delete
+ * @vc_ether_addr: VIRTCHNL MAC address structure used to delete MAC
*/
static int
-ice_vc_del_mac_addr(struct ice_vf *vf, struct ice_vsi *vsi, u8 *mac_addr)
+ice_vc_del_mac_addr(struct ice_vf *vf, struct ice_vsi *vsi,
+ struct virtchnl_ether_addr *vc_ether_addr)
{
struct device *dev = ice_pf_to_dev(vf->pf);
+ u8 *mac_addr = vc_ether_addr->addr;
enum ice_status status;
if (!ice_can_vf_change_mac(vf) &&
- ether_addr_equal(mac_addr, vf->dflt_lan_addr.addr))
+ ether_addr_equal(vf->dev_lan_addr.addr, mac_addr))
return 0;
status = ice_fltr_remove_mac(vsi, mac_addr, ICE_FWD_TO_VSI);
return -EIO;
}
- if (ether_addr_equal(mac_addr, vf->dflt_lan_addr.addr))
- eth_zero_addr(vf->dflt_lan_addr.addr);
+ ice_vfhw_mac_del(vf, vc_ether_addr);
vf->num_mac--;
ice_vc_handle_mac_addr_msg(struct ice_vf *vf, u8 *msg, bool set)
{
int (*ice_vc_cfg_mac)
- (struct ice_vf *vf, struct ice_vsi *vsi, u8 *mac_addr);
+ (struct ice_vf *vf, struct ice_vsi *vsi,
+ struct virtchnl_ether_addr *virtchnl_ether_addr);
enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS;
struct virtchnl_ether_addr_list *al =
(struct virtchnl_ether_addr_list *)msg;
is_zero_ether_addr(mac_addr))
continue;
- result = ice_vc_cfg_mac(vf, vsi, mac_addr);
+ result = ice_vc_cfg_mac(vf, vsi, &al->list[i]);
if (result == -EEXIST || result == -ENOENT) {
continue;
} else if (result) {
return -EBUSY;
ivi->vf = vf_id;
- ether_addr_copy(ivi->mac, vf->dflt_lan_addr.addr);
+ ether_addr_copy(ivi->mac, vf->hw_lan_addr.addr);
/* VF configuration for VLAN and applicable QoS */
ivi->vlan = vf->port_vlan_info & VLAN_VID_MASK;
vf = &pf->vf[vf_id];
/* nothing left to do, unicast MAC already set */
- if (ether_addr_equal(vf->dflt_lan_addr.addr, mac))
+ if (ether_addr_equal(vf->dev_lan_addr.addr, mac) &&
+ ether_addr_equal(vf->hw_lan_addr.addr, mac))
return 0;
ret = ice_check_vf_ready_for_cfg(vf);
/* VF is notified of its new MAC via the PF's response to the
* VIRTCHNL_OP_GET_VF_RESOURCES message after the VF has been reset
*/
- ether_addr_copy(vf->dflt_lan_addr.addr, mac);
+ ether_addr_copy(vf->dev_lan_addr.addr, mac);
+ ether_addr_copy(vf->hw_lan_addr.addr, mac);
if (is_zero_ether_addr(mac)) {
/* VF will send VIRTCHNL_OP_ADD_ETH_ADDR message with its MAC */
vf->pf_set_mac = false;
dev_info(dev, "%d Rx Malicious Driver Detection events detected on PF %d VF %d MAC %pM. mdd-auto-reset-vfs=%s\n",
vf->mdd_rx_events.count, pf->hw.pf_id, vf->vf_id,
- vf->dflt_lan_addr.addr,
+ vf->dev_lan_addr.addr,
test_bit(ICE_FLAG_MDD_AUTO_RESET_VF, pf->flags)
? "on" : "off");
}
dev_info(dev, "%d Tx Malicious Driver Detection events detected on PF %d VF %d MAC %pM.\n",
vf->mdd_tx_events.count, hw->pf_id, i,
- vf->dflt_lan_addr.addr);
+ vf->dev_lan_addr.addr);
}
}
}
if (pf_vsi)
dev_warn(dev, "VF MAC %pM on PF MAC %pM is generating asynchronous messages and may be overflowing the PF message queue. Please see the Adapter User Guide for more information\n",
- &vf->dflt_lan_addr.addr[0],
+ &vf->dev_lan_addr.addr[0],
pf_vsi->netdev->dev_addr);
}
ICE_VIRTCHNL_VF_CAP_PRIVILEGE,
};
+struct ice_time_mac {
+ unsigned long time_modified;
+ u8 addr[ETH_ALEN];
+};
+
/* VF MDD events print structure */
struct ice_mdd_vf_events {
u16 count; /* total count of Rx|Tx events */
struct ice_sw *vf_sw_id; /* switch ID the VF VSIs connect to */
struct virtchnl_version_info vf_ver;
u32 driver_caps; /* reported by VF driver */
- struct virtchnl_ether_addr dflt_lan_addr;
+ struct virtchnl_ether_addr dev_lan_addr;
+ struct virtchnl_ether_addr hw_lan_addr;
+ struct ice_time_mac legacy_last_added_umac;
DECLARE_BITMAP(txq_ena, ICE_MAX_RSS_QS_PER_VF);
DECLARE_BITMAP(rxq_ena, ICE_MAX_RSS_QS_PER_VF);
u16 port_vlan_info; /* Port VLAN ID and QoS */
enum virtchnl_status_code v_retval, u8 *msg, u16 msglen);
bool ice_vc_isvalid_vsi_id(struct ice_vf *vf, u16 vsi_id);
#else /* CONFIG_PCI_IOV */
-#define ice_process_vflr_event(pf) do {} while (0)
-#define ice_free_vfs(pf) do {} while (0)
-#define ice_vc_process_vf_msg(pf, event) do {} while (0)
-#define ice_vc_notify_link_state(pf) do {} while (0)
-#define ice_vc_notify_reset(pf) do {} while (0)
-#define ice_set_vf_state_qs_dis(vf) do {} while (0)
-#define ice_vf_lan_overflow_event(pf, event) do {} while (0)
-#define ice_print_vfs_mdd_events(pf) do {} while (0)
-#define ice_print_vf_rx_mdd_event(vf) do {} while (0)
-#define ice_restore_all_vfs_msi_state(pdev) do {} while (0)
+static inline void ice_process_vflr_event(struct ice_pf *pf) { }
+static inline void ice_free_vfs(struct ice_pf *pf) { }
+static inline
+void ice_vc_process_vf_msg(struct ice_pf *pf, struct ice_rq_event_info *event) { }
+static inline void ice_vc_notify_link_state(struct ice_pf *pf) { }
+static inline void ice_vc_notify_reset(struct ice_pf *pf) { }
+static inline void ice_set_vf_state_qs_dis(struct ice_vf *vf) { }
+static inline
+void ice_vf_lan_overflow_event(struct ice_pf *pf, struct ice_rq_event_info *event) { }
+static inline void ice_print_vfs_mdd_events(struct ice_pf *pf) { }
+static inline void ice_print_vf_rx_mdd_event(struct ice_vf *vf) { }
+static inline void ice_restore_all_vfs_msi_state(struct pci_dev *pdev) { }
static inline bool
ice_is_malicious_vf(struct ice_pf __always_unused *pf,
xdp_ring->xsk_pool = ice_xsk_pool(xdp_ring);
}
- err = ice_setup_rx_ctx(rx_ring);
+ err = ice_vsi_cfg_rxq(rx_ring);
if (err)
goto free_buf;
if (!pool)
return -EINVAL;
+ clear_bit(qid, vsi->af_xdp_zc_qps);
xsk_pool_dma_unmap(pool, ICE_RX_DMA_ATTR);
return 0;
if (err)
return err;
+ set_bit(qid, vsi->af_xdp_zc_qps);
+
return 0;
}
if (likely(act == XDP_REDIRECT)) {
err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
- result = !err ? ICE_XDP_REDIR : ICE_XDP_CONSUMED;
+ if (err)
+ goto out_failure;
rcu_read_unlock();
- return result;
+ return ICE_XDP_REDIR;
}
switch (act) {
case XDP_TX:
xdp_ring = rx_ring->vsi->xdp_rings[rx_ring->q_index];
result = ice_xmit_xdp_buff(xdp, xdp_ring);
+ if (result == ICE_XDP_CONSUMED)
+ goto out_failure;
break;
default:
bpf_warn_invalid_xdp_action(act);
fallthrough;
case XDP_ABORTED:
+out_failure:
trace_xdp_exception(rx_ring->netdev, xdp_prog, act);
fallthrough;
case XDP_DROP:
struct sk_buff *skb;
u16 stat_err_bits;
u16 vlan_tag = 0;
- u8 rx_ptype;
+ u16 rx_ptype;
rx_desc = ICE_RX_DESC(rx_ring, rx_ring->next_to_clean);
return -EOPNOTSUPP;
}
-#define ice_xsk_clean_rx_ring(rx_ring) do {} while (0)
-#define ice_xsk_clean_xdp_ring(xdp_ring) do {} while (0)
+static inline void ice_xsk_clean_rx_ring(struct ice_ring *rx_ring) { }
+static inline void ice_xsk_clean_xdp_ring(struct ice_ring *xdp_ring) { }
#endif /* CONFIG_XDP_SOCKETS */
#endif /* !_ICE_XSK_H_ */
return ret_val;
}
+#ifdef CONFIG_IGB_HWMON
static const u8 e1000_emc_temp_data[4] = {
E1000_EMC_INTERNAL_DATA,
E1000_EMC_DIODE1_DATA,
E1000_EMC_DIODE3_THERM_LIMIT
};
-#ifdef CONFIG_IGB_HWMON
/**
* igb_get_thermal_sensor_data_generic - Gathers thermal sensor data
* @hw: pointer to hardware structure
void igb_ptp_tx_hang(struct igb_adapter *adapter);
void igb_ptp_rx_rgtstamp(struct igb_q_vector *q_vector, struct sk_buff *skb);
int igb_ptp_rx_pktstamp(struct igb_q_vector *q_vector, void *va,
- struct sk_buff *skb);
+ ktime_t *timestamp);
int igb_ptp_set_ts_config(struct net_device *netdev, struct ifreq *ifr);
int igb_ptp_get_ts_config(struct net_device *netdev, struct ifreq *ifr);
void igb_set_flag_queue_pairs(struct igb_adapter *, const u32);
memcpy(ptr, bytes, eeprom->len);
for (i = 0; i < last_word - first_word + 1; i++)
- eeprom_buff[i] = cpu_to_le16(eeprom_buff[i]);
+ cpu_to_le16s(&eeprom_buff[i]);
ret_val = hw->nvm.ops.write(hw, first_word,
last_word - first_word + 1, eeprom_buff);
struct igb_reg_info *reginfo;
struct igb_ring *tx_ring;
union e1000_adv_tx_desc *tx_desc;
- struct my_u0 { u64 a; u64 b; } *u0;
+ struct my_u0 { __le64 a; __le64 b; } *u0;
struct igb_ring *rx_ring;
union e1000_adv_rx_desc *rx_desc;
u32 staterr;
}
input->filter.match_flags |= IGB_FILTER_FLAG_VLAN_TCI;
- input->filter.vlan_tci = match.key->vlan_priority;
+ input->filter.vlan_tci =
+ (__force __be16)match.key->vlan_priority;
}
}
cmd_type |= len | IGB_TXD_DCMD;
tx_desc->read.cmd_type_len = cpu_to_le32(cmd_type);
- olinfo_status = cpu_to_le32(len << E1000_ADVTXD_PAYLEN_SHIFT);
+ olinfo_status = len << E1000_ADVTXD_PAYLEN_SHIFT;
/* 82575 requires a unique index per ring */
if (test_bit(IGB_RING_FLAG_TX_CTX_IDX, &tx_ring->flags))
olinfo_status |= tx_ring->reg_idx << 4;
- tx_desc->read.olinfo_status = olinfo_status;
+ tx_desc->read.olinfo_status = cpu_to_le32(olinfo_status);
netdev_tx_sent_queue(txring_txq(tx_ring), tx_buffer->bytecount);
static struct sk_buff *igb_construct_skb(struct igb_ring *rx_ring,
struct igb_rx_buffer *rx_buffer,
struct xdp_buff *xdp,
- union e1000_adv_rx_desc *rx_desc)
+ ktime_t timestamp)
{
#if (PAGE_SIZE < 8192)
unsigned int truesize = igb_rx_pg_size(rx_ring) / 2;
if (unlikely(!skb))
return NULL;
- if (unlikely(igb_test_staterr(rx_desc, E1000_RXDADV_STAT_TSIP))) {
- if (!igb_ptp_rx_pktstamp(rx_ring->q_vector, xdp->data, skb)) {
- xdp->data += IGB_TS_HDR_LEN;
- size -= IGB_TS_HDR_LEN;
- }
- }
+ if (timestamp)
+ skb_hwtstamps(skb)->hwtstamp = timestamp;
/* Determine available headroom for copy */
headlen = size;
static struct sk_buff *igb_build_skb(struct igb_ring *rx_ring,
struct igb_rx_buffer *rx_buffer,
struct xdp_buff *xdp,
- union e1000_adv_rx_desc *rx_desc)
+ ktime_t timestamp)
{
#if (PAGE_SIZE < 8192)
unsigned int truesize = igb_rx_pg_size(rx_ring) / 2;
if (metasize)
skb_metadata_set(skb, metasize);
- /* pull timestamp out of packet data */
- if (igb_test_staterr(rx_desc, E1000_RXDADV_STAT_TSIP)) {
- if (!igb_ptp_rx_pktstamp(rx_ring->q_vector, skb->data, skb))
- __skb_pull(skb, IGB_TS_HDR_LEN);
- }
+ if (timestamp)
+ skb_hwtstamps(skb)->hwtstamp = timestamp;
/* update buffer offset */
#if (PAGE_SIZE < 8192)
break;
case XDP_TX:
result = igb_xdp_xmit_back(adapter, xdp);
+ if (result == IGB_XDP_CONSUMED)
+ goto out_failure;
break;
case XDP_REDIRECT:
err = xdp_do_redirect(adapter->netdev, xdp, xdp_prog);
- if (!err)
- result = IGB_XDP_REDIR;
- else
- result = IGB_XDP_CONSUMED;
+ if (err)
+ goto out_failure;
+ result = IGB_XDP_REDIR;
break;
default:
bpf_warn_invalid_xdp_action(act);
fallthrough;
case XDP_ABORTED:
+out_failure:
trace_xdp_exception(rx_ring->netdev, xdp_prog, act);
fallthrough;
case XDP_DROP:
if (igb_test_staterr(rx_desc, E1000_RXDEXT_STATERR_LB) &&
test_bit(IGB_RING_FLAG_RX_LB_VLAN_BSWAP, &rx_ring->flags))
- vid = be16_to_cpu(rx_desc->wb.upper.vlan);
+ vid = be16_to_cpu((__force __be16)rx_desc->wb.upper.vlan);
else
vid = le16_to_cpu(rx_desc->wb.upper.vlan);
while (likely(total_packets < budget)) {
union e1000_adv_rx_desc *rx_desc;
struct igb_rx_buffer *rx_buffer;
+ ktime_t timestamp = 0;
+ int pkt_offset = 0;
unsigned int size;
+ void *pktbuf;
/* return some buffers to hardware, one at a time is too slow */
if (cleaned_count >= IGB_RX_BUFFER_WRITE) {
dma_rmb();
rx_buffer = igb_get_rx_buffer(rx_ring, size, &rx_buf_pgcnt);
+ pktbuf = page_address(rx_buffer->page) + rx_buffer->page_offset;
+
+ /* pull rx packet timestamp if available and valid */
+ if (igb_test_staterr(rx_desc, E1000_RXDADV_STAT_TSIP)) {
+ int ts_hdr_len;
+
+ ts_hdr_len = igb_ptp_rx_pktstamp(rx_ring->q_vector,
+ pktbuf, ×tamp);
+
+ pkt_offset += ts_hdr_len;
+ size -= ts_hdr_len;
+ }
/* retrieve a buffer from the ring */
if (!skb) {
- unsigned int offset = igb_rx_offset(rx_ring);
- unsigned char *hard_start;
+ unsigned char *hard_start = pktbuf - igb_rx_offset(rx_ring);
+ unsigned int offset = pkt_offset + igb_rx_offset(rx_ring);
- hard_start = page_address(rx_buffer->page) +
- rx_buffer->page_offset - offset;
xdp_prepare_buff(&xdp, hard_start, offset, size, true);
#if (PAGE_SIZE > 4096)
/* At larger PAGE_SIZE, frame_sz depend on len size */
} else if (skb)
igb_add_rx_frag(rx_ring, rx_buffer, skb, size);
else if (ring_uses_build_skb(rx_ring))
- skb = igb_build_skb(rx_ring, rx_buffer, &xdp, rx_desc);
+ skb = igb_build_skb(rx_ring, rx_buffer, &xdp,
+ timestamp);
else
skb = igb_construct_skb(rx_ring, rx_buffer,
- &xdp, rx_desc);
+ &xdp, timestamp);
/* exit if we failed to retrieve a buffer */
if (!skb) {
dev_kfree_skb_any(skb);
}
-#define IGB_RET_PTP_DISABLED 1
-#define IGB_RET_PTP_INVALID 2
-
/**
* igb_ptp_rx_pktstamp - retrieve Rx per packet timestamp
* @q_vector: Pointer to interrupt specific structure
* @va: Pointer to address containing Rx buffer
- * @skb: Buffer containing timestamp and packet
+ * @timestamp: Pointer where timestamp will be stored
*
* This function is meant to retrieve a timestamp from the first buffer of an
* incoming frame. The value is stored in little endian format starting on
* byte 8
*
- * Returns: 0 if success, nonzero if failure
+ * Returns: The timestamp header length or 0 if not available
**/
int igb_ptp_rx_pktstamp(struct igb_q_vector *q_vector, void *va,
- struct sk_buff *skb)
+ ktime_t *timestamp)
{
struct igb_adapter *adapter = q_vector->adapter;
+ struct skb_shared_hwtstamps ts;
__le64 *regval = (__le64 *)va;
int adjust = 0;
if (!(adapter->ptp_flags & IGB_PTP_ENABLED))
- return IGB_RET_PTP_DISABLED;
+ return 0;
/* The timestamp is recorded in little endian format.
* DWORD: 0 1 2 3
/* check reserved dwords are zero, be/le doesn't matter for zero */
if (regval[0])
- return IGB_RET_PTP_INVALID;
+ return 0;
- igb_ptp_systim_to_hwtstamp(adapter, skb_hwtstamps(skb),
- le64_to_cpu(regval[1]));
+ igb_ptp_systim_to_hwtstamp(adapter, &ts, le64_to_cpu(regval[1]));
/* adjust timestamp for the RX latency based on link speed */
if (adapter->hw.mac.type == e1000_i210) {
break;
}
}
- skb_hwtstamps(skb)->hwtstamp =
- ktime_sub_ns(skb_hwtstamps(skb)->hwtstamp, adjust);
- return 0;
+ *timestamp = ktime_sub_ns(ts.hwtstamp, adjust);
+
+ return IGB_TS_HDR_LEN;
}
/**
| E1000_FTQF_MASK); /* mask all inputs */
ftqf &= ~E1000_FTQF_MASK_PROTO_BP; /* enable protocol check */
- wr32(E1000_IMIR(3), htons(PTP_EV_PORT));
+ wr32(E1000_IMIR(3), (__force unsigned int)htons(PTP_EV_PORT));
wr32(E1000_IMIREXT(3),
(E1000_IMIREXT_SIZE_BP | E1000_IMIREXT_CTRL_BP));
if (hw->mac.type == e1000_82576) {
/* enable source port check */
- wr32(E1000_SPQF(3), htons(PTP_EV_PORT));
+ wr32(E1000_SPQF(3), (__force unsigned int)htons(PTP_EV_PORT));
ftqf &= ~E1000_FTQF_MASK_SOURCE_PORT_BP;
}
wr32(E1000_FTQF(3), ftqf);
static void igbvf_receive_skb(struct igbvf_adapter *adapter,
struct net_device *netdev,
struct sk_buff *skb,
- u32 status, u16 vlan)
+ u32 status, __le16 vlan)
{
u16 vid;
if (status & E1000_RXD_STAT_VP) {
if ((adapter->flags & IGBVF_FLAG_RX_LB_VLAN_BSWAP) &&
(status & E1000_RXDEXT_STATERR_LB))
- vid = be16_to_cpu(vlan) & E1000_RXD_SPC_VLAN_MASK;
+ vid = be16_to_cpu((__force __be16)vlan) & E1000_RXD_SPC_VLAN_MASK;
else
vid = le16_to_cpu(vlan) & E1000_RXD_SPC_VLAN_MASK;
if (test_bit(vid, adapter->active_vlans))
/* remove payload length from inner checksum */
paylen = skb->len - l4_offset;
- csum_replace_by_diff(&l4.tcp->check, htonl(paylen));
+ csum_replace_by_diff(&l4.tcp->check, (__force __wsum)htonl(paylen));
/* MSS L4LEN IDX */
mss_l4len_idx = (*hdr_len - l4_offset) << E1000_ADVTXD_L4LEN_SHIFT;
/* Receive Descriptor - Advanced */
union e1000_adv_rx_desc {
struct {
- u64 pkt_addr; /* Packet buffer address */
- u64 hdr_addr; /* Header buffer address */
+ __le64 pkt_addr; /* Packet buffer address */
+ __le64 hdr_addr; /* Header buffer address */
} read;
struct {
struct {
union {
- u32 data;
+ __le32 data;
struct {
- u16 pkt_info; /* RSS/Packet type */
+ __le16 pkt_info; /* RSS/Packet type */
/* Split Header, hdr buffer length */
- u16 hdr_info;
+ __le16 hdr_info;
} hs_rss;
} lo_dword;
union {
- u32 rss; /* RSS Hash */
+ __le32 rss; /* RSS Hash */
struct {
- u16 ip_id; /* IP id */
- u16 csum; /* Packet Checksum */
+ __le16 ip_id; /* IP id */
+ __le16 csum; /* Packet Checksum */
} csum_ip;
} hi_dword;
} lower;
struct {
- u32 status_error; /* ext status/error */
- u16 length; /* Packet length */
- u16 vlan; /* VLAN tag */
+ __le32 status_error; /* ext status/error */
+ __le16 length; /* Packet length */
+ __le16 vlan; /* VLAN tag */
} upper;
} wb; /* writeback */
};
/* Transmit Descriptor - Advanced */
union e1000_adv_tx_desc {
struct {
- u64 buffer_addr; /* Address of descriptor's data buf */
- u32 cmd_type_len;
- u32 olinfo_status;
+ __le64 buffer_addr; /* Address of descriptor's data buf */
+ __le32 cmd_type_len;
+ __le32 olinfo_status;
} read;
struct {
- u64 rsvd; /* Reserved */
- u32 nxtseq_seed;
- u32 status;
+ __le64 rsvd; /* Reserved */
+ __le32 nxtseq_seed;
+ __le32 status;
} wb;
};
/* Context descriptors */
struct e1000_adv_tx_context_desc {
- u32 vlan_macip_lens;
- u32 seqnum_seed;
- u32 type_tucmd_mlhl;
- u32 mss_l4len_idx;
+ __le32 vlan_macip_lens;
+ __le32 seqnum_seed;
+ __le32 type_tucmd_mlhl;
+ __le32 mss_l4len_idx;
};
#define E1000_ADVTXD_MACLEN_SHIFT 9 /* Adv ctxt desc mac len shift */
};
struct xdp_rxq_info xdp_rxq;
+ struct xsk_buff_pool *xsk_pool;
} ____cacheline_internodealigned_in_smp;
/* Board specific private data structure */
void igc_reset(struct igc_adapter *adapter);
int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx);
void igc_update_stats(struct igc_adapter *adapter);
+void igc_disable_rx_ring(struct igc_ring *ring);
+void igc_enable_rx_ring(struct igc_ring *ring);
+void igc_disable_tx_ring(struct igc_ring *ring);
+void igc_enable_tx_ring(struct igc_ring *ring);
+int igc_xsk_wakeup(struct net_device *dev, u32 queue_id, u32 flags);
/* igc_dump declarations */
void igc_rings_dump(struct igc_adapter *adapter);
/* VLAN info */
#define IGC_TX_FLAGS_VLAN_MASK 0xffff0000
+#define IGC_TX_FLAGS_VLAN_SHIFT 16
/* igc_test_staterr - tests bits within Rx descriptor status and error fields */
static inline __le32 igc_test_staterr(union igc_adv_rx_desc *rx_desc,
/* olinfo flags */
IGC_TX_FLAGS_IPV4 = 0x10,
IGC_TX_FLAGS_CSUM = 0x20,
-
- IGC_TX_FLAGS_XDP = 0x100,
};
enum igc_boards {
#define TXD_USE_COUNT(S) DIV_ROUND_UP((S), IGC_MAX_DATA_PER_TXD)
#define DESC_NEEDED (MAX_SKB_FRAGS + 4)
+enum igc_tx_buffer_type {
+ IGC_TX_BUFFER_TYPE_SKB,
+ IGC_TX_BUFFER_TYPE_XDP,
+ IGC_TX_BUFFER_TYPE_XSK,
+};
+
/* wrapper around a pointer to a socket buffer,
* so a DMA handle can be stored along with the buffer
*/
struct igc_tx_buffer {
union igc_adv_tx_desc *next_to_watch;
unsigned long time_stamp;
+ enum igc_tx_buffer_type type;
union {
struct sk_buff *skb;
struct xdp_frame *xdpf;
};
struct igc_rx_buffer {
- dma_addr_t dma;
- struct page *page;
+ union {
+ struct {
+ dma_addr_t dma;
+ struct page *page;
#if (BITS_PER_LONG > 32) || (PAGE_SIZE >= 65536)
- __u32 page_offset;
+ __u32 page_offset;
#else
- __u16 page_offset;
+ __u16 page_offset;
#endif
- __u16 pagecnt_bias;
+ __u16 pagecnt_bias;
+ };
+ struct xdp_buff *xdp;
+ };
};
struct igc_q_vector {
IGC_RING_FLAG_RX_SCTP_CSUM,
IGC_RING_FLAG_RX_LB_VLAN_BSWAP,
IGC_RING_FLAG_TX_CTX_IDX,
- IGC_RING_FLAG_TX_DETECT_HANG
+ IGC_RING_FLAG_TX_DETECT_HANG,
+ IGC_RING_FLAG_AF_XDP_ZC,
};
#define ring_uses_large_buffer(ring) \
/* Additional Transmit Descriptor Control definitions */
#define IGC_TXDCTL_QUEUE_ENABLE 0x02000000 /* Ena specific Tx Queue */
+#define IGC_TXDCTL_SWFLUSH 0x04000000 /* Transmit Software Flush */
/* Additional Receive Descriptor Control definitions */
#define IGC_RXDCTL_QUEUE_ENABLE 0x02000000 /* Ena specific Rx Queue */
+#define IGC_RXDCTL_SWFLUSH 0x04000000 /* Receive Software Flush */
/* SRRCTL bit definitions */
#define IGC_SRRCTL_BSIZEPKT_SHIFT 10 /* Shift _right_ */
#define IGC_CTRL_SLU 0x00000040 /* Set link up (Force Link) */
#define IGC_CTRL_FRCSPD 0x00000800 /* Force Speed */
#define IGC_CTRL_FRCDPX 0x00001000 /* Force Duplex */
+#define IGC_CTRL_VME 0x40000000 /* IEEE VLAN mode enable */
#define IGC_CTRL_RFCE 0x08000000 /* Receive Flow Control enable */
#define IGC_CTRL_TFCE 0x10000000 /* Transmit flow control enable */
-#define IGC_CTRL_SDP0_DIR 0x00400000 /* SDP0 Data direction */
-#define IGC_CTRL_SDP1_DIR 0x00800000 /* SDP1 Data direction */
+#define IGC_CTRL_SDP0_DIR 0x00400000 /* SDP0 Data direction */
+#define IGC_CTRL_SDP1_DIR 0x00800000 /* SDP1 Data direction */
/* As per the EAS the maximum supported size is 9.5KB (9728 bytes) */
#define MAX_JUMBO_FRAME_SIZE 0x2600
#define NWAY_LPAR_ASM_DIR 0x0800 /* LP Asymmetric Pause Direction bit */
/* 1000BASE-T Control Register */
-#define CR_1000T_ASYM_PAUSE 0x0080 /* Advertise asymmetric pause bit */
#define CR_1000T_HD_CAPS 0x0100 /* Advertise 1000T HD capability */
#define CR_1000T_FD_CAPS 0x0200 /* Advertise 1000T FD capability */
#define IGC_RXD_STAT_IXSM 0x04 /* Ignore checksum */
#define IGC_RXD_STAT_UDPCS 0x10 /* UDP xsum calculated */
#define IGC_RXD_STAT_TCPCS 0x20 /* TCP xsum calculated */
+#define IGC_RXD_STAT_VP 0x08 /* IEEE VLAN Packet */
+
+#define IGC_RXDEXT_STATERR_LB 0x00040000
/* Advanced Receive Descriptor bit definitions */
#define IGC_RXDADV_STAT_TSIP 0x08000 /* timestamp in packet */
void igc_rings_dump(struct igc_adapter *adapter)
{
struct net_device *netdev = adapter->netdev;
- struct my_u0 { u64 a; u64 b; } *u0;
+ struct my_u0 { __le64 a; __le64 b; } *u0;
union igc_adv_tx_desc *tx_desc;
union igc_adv_rx_desc *rx_desc;
struct igc_ring *tx_ring;
memcpy(ptr, bytes, eeprom->len);
for (i = 0; i < last_word - first_word + 1; i++)
- eeprom_buff[i] = cpu_to_le16(eeprom_buff[i]);
+ cpu_to_le16s(&eeprom_buff[i]);
ret_val = hw->nvm.ops.write(hw, first_word,
last_word - first_word + 1, eeprom_buff);
IGC_TEST_LEN * ETH_GSTRING_LEN);
break;
case ETH_SS_STATS:
- for (i = 0; i < IGC_GLOBAL_STATS_LEN; i++) {
- memcpy(p, igc_gstrings_stats[i].stat_string,
- ETH_GSTRING_LEN);
- p += ETH_GSTRING_LEN;
- }
- for (i = 0; i < IGC_NETDEV_STATS_LEN; i++) {
- memcpy(p, igc_gstrings_net_stats[i].stat_string,
- ETH_GSTRING_LEN);
- p += ETH_GSTRING_LEN;
- }
+ for (i = 0; i < IGC_GLOBAL_STATS_LEN; i++)
+ ethtool_sprintf(&p, igc_gstrings_stats[i].stat_string);
+ for (i = 0; i < IGC_NETDEV_STATS_LEN; i++)
+ ethtool_sprintf(&p,
+ igc_gstrings_net_stats[i].stat_string);
for (i = 0; i < adapter->num_tx_queues; i++) {
- sprintf(p, "tx_queue_%u_packets", i);
- p += ETH_GSTRING_LEN;
- sprintf(p, "tx_queue_%u_bytes", i);
- p += ETH_GSTRING_LEN;
- sprintf(p, "tx_queue_%u_restart", i);
- p += ETH_GSTRING_LEN;
+ ethtool_sprintf(&p, "tx_queue_%u_packets", i);
+ ethtool_sprintf(&p, "tx_queue_%u_bytes", i);
+ ethtool_sprintf(&p, "tx_queue_%u_restart", i);
}
for (i = 0; i < adapter->num_rx_queues; i++) {
- sprintf(p, "rx_queue_%u_packets", i);
- p += ETH_GSTRING_LEN;
- sprintf(p, "rx_queue_%u_bytes", i);
- p += ETH_GSTRING_LEN;
- sprintf(p, "rx_queue_%u_drops", i);
- p += ETH_GSTRING_LEN;
- sprintf(p, "rx_queue_%u_csum_err", i);
- p += ETH_GSTRING_LEN;
- sprintf(p, "rx_queue_%u_alloc_failed", i);
- p += ETH_GSTRING_LEN;
+ ethtool_sprintf(&p, "rx_queue_%u_packets", i);
+ ethtool_sprintf(&p, "rx_queue_%u_bytes", i);
+ ethtool_sprintf(&p, "rx_queue_%u_drops", i);
+ ethtool_sprintf(&p, "rx_queue_%u_csum_err", i);
+ ethtool_sprintf(&p, "rx_queue_%u_alloc_failed", i);
}
/* BUG_ON(p - data != IGC_STATS_LEN * ETH_GSTRING_LEN); */
break;
#include <linux/pm_runtime.h>
#include <net/pkt_sched.h>
#include <linux/bpf_trace.h>
-
+#include <net/xdp_sock_drv.h>
#include <net/ipv6.h>
#include "igc.h"
if (!netif_running(adapter->netdev))
igc_power_down_phy_copper_base(&adapter->hw);
+ /* Enable HW to recognize an 802.1Q VLAN Ethernet packet */
+ wr32(IGC_VET, ETH_P_8021Q);
+
/* Re-enable PTP, where applicable. */
igc_ptp_reset(adapter);
ctrl_ext | IGC_CTRL_EXT_DRV_LOAD);
}
+static void igc_unmap_tx_buffer(struct device *dev, struct igc_tx_buffer *buf)
+{
+ dma_unmap_single(dev, dma_unmap_addr(buf, dma),
+ dma_unmap_len(buf, len), DMA_TO_DEVICE);
+
+ dma_unmap_len_set(buf, len, 0);
+}
+
/**
* igc_clean_tx_ring - Free Tx Buffers
* @tx_ring: ring to be cleaned
{
u16 i = tx_ring->next_to_clean;
struct igc_tx_buffer *tx_buffer = &tx_ring->tx_buffer_info[i];
+ u32 xsk_frames = 0;
while (i != tx_ring->next_to_use) {
union igc_adv_tx_desc *eop_desc, *tx_desc;
- if (tx_buffer->tx_flags & IGC_TX_FLAGS_XDP)
+ switch (tx_buffer->type) {
+ case IGC_TX_BUFFER_TYPE_XSK:
+ xsk_frames++;
+ break;
+ case IGC_TX_BUFFER_TYPE_XDP:
xdp_return_frame(tx_buffer->xdpf);
- else
+ igc_unmap_tx_buffer(tx_ring->dev, tx_buffer);
+ break;
+ case IGC_TX_BUFFER_TYPE_SKB:
dev_kfree_skb_any(tx_buffer->skb);
-
- /* unmap skb header data */
- dma_unmap_single(tx_ring->dev,
- dma_unmap_addr(tx_buffer, dma),
- dma_unmap_len(tx_buffer, len),
- DMA_TO_DEVICE);
+ igc_unmap_tx_buffer(tx_ring->dev, tx_buffer);
+ break;
+ default:
+ netdev_warn_once(tx_ring->netdev, "Unknown Tx buffer type\n");
+ break;
+ }
/* check for eop_desc to determine the end of the packet */
eop_desc = tx_buffer->next_to_watch;
/* unmap any remaining paged data */
if (dma_unmap_len(tx_buffer, len))
- dma_unmap_page(tx_ring->dev,
- dma_unmap_addr(tx_buffer, dma),
- dma_unmap_len(tx_buffer, len),
- DMA_TO_DEVICE);
+ igc_unmap_tx_buffer(tx_ring->dev, tx_buffer);
}
/* move us one more past the eop_desc for start of next pkt */
}
}
+ if (tx_ring->xsk_pool && xsk_frames)
+ xsk_tx_completed(tx_ring->xsk_pool, xsk_frames);
+
/* reset BQL for queue */
netdev_tx_reset_queue(txring_txq(tx_ring));
return err;
}
-/**
- * igc_clean_rx_ring - Free Rx Buffers per Queue
- * @rx_ring: ring to free buffers from
- */
-static void igc_clean_rx_ring(struct igc_ring *rx_ring)
+static void igc_clean_rx_ring_page_shared(struct igc_ring *rx_ring)
{
u16 i = rx_ring->next_to_clean;
if (i == rx_ring->count)
i = 0;
}
+}
- clear_ring_uses_large_buffer(rx_ring);
+static void igc_clean_rx_ring_xsk_pool(struct igc_ring *ring)
+{
+ struct igc_rx_buffer *bi;
+ u16 i;
- rx_ring->next_to_alloc = 0;
- rx_ring->next_to_clean = 0;
- rx_ring->next_to_use = 0;
+ for (i = 0; i < ring->count; i++) {
+ bi = &ring->rx_buffer_info[i];
+ if (!bi->xdp)
+ continue;
+
+ xsk_buff_free(bi->xdp);
+ bi->xdp = NULL;
+ }
+}
+
+/**
+ * igc_clean_rx_ring - Free Rx Buffers per Queue
+ * @ring: ring to free buffers from
+ */
+static void igc_clean_rx_ring(struct igc_ring *ring)
+{
+ if (ring->xsk_pool)
+ igc_clean_rx_ring_xsk_pool(ring);
+ else
+ igc_clean_rx_ring_page_shared(ring);
+
+ clear_ring_uses_large_buffer(ring);
+
+ ring->next_to_alloc = 0;
+ ring->next_to_clean = 0;
+ ring->next_to_use = 0;
}
/**
{
igc_clean_rx_ring(rx_ring);
- igc_xdp_unregister_rxq_info(rx_ring);
+ xdp_rxq_info_unreg(&rx_ring->xdp_rxq);
vfree(rx_ring->rx_buffer_info);
rx_ring->rx_buffer_info = NULL;
{
struct net_device *ndev = rx_ring->netdev;
struct device *dev = rx_ring->dev;
+ u8 index = rx_ring->queue_index;
int size, desc_len, res;
- res = igc_xdp_register_rxq_info(rx_ring);
- if (res < 0)
+ res = xdp_rxq_info_reg(&rx_ring->xdp_rxq, ndev, index,
+ rx_ring->q_vector->napi.napi_id);
+ if (res < 0) {
+ netdev_err(ndev, "Failed to register xdp_rxq index %u\n",
+ index);
return res;
+ }
size = sizeof(struct igc_rx_buffer) * rx_ring->count;
rx_ring->rx_buffer_info = vzalloc(size);
return 0;
err:
- igc_xdp_unregister_rxq_info(rx_ring);
+ xdp_rxq_info_unreg(&rx_ring->xdp_rxq);
vfree(rx_ring->rx_buffer_info);
rx_ring->rx_buffer_info = NULL;
netdev_err(ndev, "Unable to allocate memory for Rx descriptor ring\n");
return err;
}
-static bool igc_xdp_is_enabled(struct igc_adapter *adapter)
+static struct xsk_buff_pool *igc_get_xsk_pool(struct igc_adapter *adapter,
+ struct igc_ring *ring)
{
- return !!adapter->xdp_prog;
+ if (!igc_xdp_is_enabled(adapter) ||
+ !test_bit(IGC_RING_FLAG_AF_XDP_ZC, &ring->flags))
+ return NULL;
+
+ return xsk_get_pool_from_qid(ring->netdev, ring->queue_index);
}
/**
int reg_idx = ring->reg_idx;
u32 srrctl = 0, rxdctl = 0;
u64 rdba = ring->dma;
+ u32 buf_size;
+
+ xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);
+ ring->xsk_pool = igc_get_xsk_pool(adapter, ring);
+ if (ring->xsk_pool) {
+ WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
+ MEM_TYPE_XSK_BUFF_POOL,
+ NULL));
+ xsk_pool_set_rxq_info(ring->xsk_pool, &ring->xdp_rxq);
+ } else {
+ WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
+ MEM_TYPE_PAGE_SHARED,
+ NULL));
+ }
if (igc_xdp_is_enabled(adapter))
set_ring_uses_large_buffer(ring);
ring->next_to_clean = 0;
ring->next_to_use = 0;
- /* set descriptor configuration */
- srrctl = IGC_RX_HDR_LEN << IGC_SRRCTL_BSIZEHDRSIZE_SHIFT;
- if (ring_uses_large_buffer(ring))
- srrctl |= IGC_RXBUFFER_3072 >> IGC_SRRCTL_BSIZEPKT_SHIFT;
+ if (ring->xsk_pool)
+ buf_size = xsk_pool_get_rx_frame_size(ring->xsk_pool);
+ else if (ring_uses_large_buffer(ring))
+ buf_size = IGC_RXBUFFER_3072;
else
- srrctl |= IGC_RXBUFFER_2048 >> IGC_SRRCTL_BSIZEPKT_SHIFT;
+ buf_size = IGC_RXBUFFER_2048;
+
+ srrctl = IGC_RX_HDR_LEN << IGC_SRRCTL_BSIZEHDRSIZE_SHIFT;
+ srrctl |= buf_size >> IGC_SRRCTL_BSIZEPKT_SHIFT;
srrctl |= IGC_SRRCTL_DESCTYPE_ADV_ONEBUF;
wr32(IGC_SRRCTL(reg_idx), srrctl);
u64 tdba = ring->dma;
u32 txdctl = 0;
+ ring->xsk_pool = igc_get_xsk_pool(adapter, ring);
+
/* disable the queue */
wr32(IGC_TXDCTL(reg_idx), 0);
wrfl();
((u32)((_input) & (_flag)) * ((_result) / (_flag))) : \
((u32)((_input) & (_flag)) / ((_flag) / (_result))))
-static u32 igc_tx_cmd_type(u32 tx_flags)
+static u32 igc_tx_cmd_type(struct sk_buff *skb, u32 tx_flags)
{
/* set type for advanced descriptor with frame checksum insertion */
u32 cmd_type = IGC_ADVTXD_DTYP_DATA |
IGC_ADVTXD_DCMD_DEXT |
IGC_ADVTXD_DCMD_IFCS;
+ /* set HW vlan bit if vlan is present */
+ cmd_type |= IGC_SET_FLAG(tx_flags, IGC_TX_FLAGS_VLAN,
+ IGC_ADVTXD_DCMD_VLE);
+
/* set segmentation bits for TSO */
cmd_type |= IGC_SET_FLAG(tx_flags, IGC_TX_FLAGS_TSO,
(IGC_ADVTXD_DCMD_TSE));
cmd_type |= IGC_SET_FLAG(tx_flags, IGC_TX_FLAGS_TSTAMP,
(IGC_ADVTXD_MAC_TSTAMP));
+ /* insert frame checksum */
+ cmd_type ^= IGC_SET_FLAG(skb->no_fcs, 1, IGC_ADVTXD_DCMD_IFCS);
+
return cmd_type;
}
u16 i = tx_ring->next_to_use;
unsigned int data_len, size;
dma_addr_t dma;
- u32 cmd_type = igc_tx_cmd_type(tx_flags);
+ u32 cmd_type;
+ cmd_type = igc_tx_cmd_type(skb, tx_flags);
tx_desc = IGC_TX_DESC(tx_ring, i);
igc_tx_olinfo_status(tx_ring, tx_desc, tx_flags, skb->len - hdr_len);
/* clear dma mappings for failed tx_buffer_info map */
while (tx_buffer != first) {
if (dma_unmap_len(tx_buffer, len))
- dma_unmap_page(tx_ring->dev,
- dma_unmap_addr(tx_buffer, dma),
- dma_unmap_len(tx_buffer, len),
- DMA_TO_DEVICE);
- dma_unmap_len_set(tx_buffer, len, 0);
+ igc_unmap_tx_buffer(tx_ring->dev, tx_buffer);
if (i-- == 0)
i += tx_ring->count;
}
if (dma_unmap_len(tx_buffer, len))
- dma_unmap_single(tx_ring->dev,
- dma_unmap_addr(tx_buffer, dma),
- dma_unmap_len(tx_buffer, len),
- DMA_TO_DEVICE);
- dma_unmap_len_set(tx_buffer, len, 0);
+ igc_unmap_tx_buffer(tx_ring->dev, tx_buffer);
dev_kfree_skb_any(tx_buffer->skb);
tx_buffer->skb = NULL;
/* record the location of the first descriptor for this packet */
first = &tx_ring->tx_buffer_info[tx_ring->next_to_use];
+ first->type = IGC_TX_BUFFER_TYPE_SKB;
first->skb = skb;
first->bytecount = skb->len;
first->gso_segs = 1;
}
}
+ if (skb_vlan_tag_present(skb)) {
+ tx_flags |= IGC_TX_FLAGS_VLAN;
+ tx_flags |= (skb_vlan_tag_get(skb) << IGC_TX_FLAGS_VLAN_SHIFT);
+ }
+
/* record initial flags and protocol */
first->tx_flags = tx_flags;
first->protocol = protocol;
PKT_HASH_TYPE_L3);
}
+static void igc_rx_vlan(struct igc_ring *rx_ring,
+ union igc_adv_rx_desc *rx_desc,
+ struct sk_buff *skb)
+{
+ struct net_device *dev = rx_ring->netdev;
+ u16 vid;
+
+ if ((dev->features & NETIF_F_HW_VLAN_CTAG_RX) &&
+ igc_test_staterr(rx_desc, IGC_RXD_STAT_VP)) {
+ if (igc_test_staterr(rx_desc, IGC_RXDEXT_STATERR_LB) &&
+ test_bit(IGC_RING_FLAG_RX_LB_VLAN_BSWAP, &rx_ring->flags))
+ vid = be16_to_cpu((__force __be16)rx_desc->wb.upper.vlan);
+ else
+ vid = le16_to_cpu(rx_desc->wb.upper.vlan);
+
+ __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), vid);
+ }
+}
+
/**
* igc_process_skb_fields - Populate skb header fields from Rx descriptor
* @rx_ring: rx descriptor ring packet is being transacted on
igc_rx_checksum(rx_ring, rx_desc, skb);
+ igc_rx_vlan(rx_ring, rx_desc, skb);
+
skb_record_rx_queue(skb, rx_ring->queue_index);
skb->protocol = eth_type_trans(skb, rx_ring->netdev);
}
+static void igc_vlan_mode(struct net_device *netdev, netdev_features_t features)
+{
+ bool enable = !!(features & NETIF_F_HW_VLAN_CTAG_RX);
+ struct igc_adapter *adapter = netdev_priv(netdev);
+ struct igc_hw *hw = &adapter->hw;
+ u32 ctrl;
+
+ ctrl = rd32(IGC_CTRL);
+
+ if (enable) {
+ /* enable VLAN tag insert/strip */
+ ctrl |= IGC_CTRL_VME;
+ } else {
+ /* disable VLAN tag insert/strip */
+ ctrl &= ~IGC_CTRL_VME;
+ }
+ wr32(IGC_CTRL, ctrl);
+}
+
+static void igc_restore_vlan(struct igc_adapter *adapter)
+{
+ igc_vlan_mode(adapter->netdev, adapter->netdev->features);
+}
+
static struct igc_rx_buffer *igc_get_rx_buffer(struct igc_ring *rx_ring,
const unsigned int size,
int *rx_buffer_pgcnt)
}
}
+static bool igc_alloc_rx_buffers_zc(struct igc_ring *ring, u16 count)
+{
+ union igc_adv_rx_desc *desc;
+ u16 i = ring->next_to_use;
+ struct igc_rx_buffer *bi;
+ dma_addr_t dma;
+ bool ok = true;
+
+ if (!count)
+ return ok;
+
+ desc = IGC_RX_DESC(ring, i);
+ bi = &ring->rx_buffer_info[i];
+ i -= ring->count;
+
+ do {
+ bi->xdp = xsk_buff_alloc(ring->xsk_pool);
+ if (!bi->xdp) {
+ ok = false;
+ break;
+ }
+
+ dma = xsk_buff_xdp_get_dma(bi->xdp);
+ desc->read.pkt_addr = cpu_to_le64(dma);
+
+ desc++;
+ bi++;
+ i++;
+ if (unlikely(!i)) {
+ desc = IGC_RX_DESC(ring, 0);
+ bi = ring->rx_buffer_info;
+ i -= ring->count;
+ }
+
+ /* Clear the length for the next_to_use descriptor. */
+ desc->wb.upper.length = 0;
+
+ count--;
+ } while (count);
+
+ i += ring->count;
+
+ if (ring->next_to_use != i) {
+ ring->next_to_use = i;
+
+ /* Force memory writes to complete before letting h/w
+ * know there are new descriptors to fetch. (Only
+ * applicable for weak-ordered memory model archs,
+ * such as IA-64).
+ */
+ wmb();
+ writel(i, ring->tail);
+ }
+
+ return ok;
+}
+
static int igc_xdp_init_tx_buffer(struct igc_tx_buffer *buffer,
struct xdp_frame *xdpf,
struct igc_ring *ring)
return -ENOMEM;
}
+ buffer->type = IGC_TX_BUFFER_TYPE_XDP;
buffer->xdpf = xdpf;
- buffer->tx_flags = IGC_TX_FLAGS_XDP;
buffer->protocol = 0;
buffer->bytecount = xdpf->len;
buffer->gso_segs = 1;
return res;
}
-static struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
- struct xdp_buff *xdp)
+/* This function assumes rcu_read_lock() is held by the caller. */
+static int __igc_xdp_run_prog(struct igc_adapter *adapter,
+ struct bpf_prog *prog,
+ struct xdp_buff *xdp)
{
- struct bpf_prog *prog;
- int res;
- u32 act;
-
- rcu_read_lock();
+ u32 act = bpf_prog_run_xdp(prog, xdp);
- prog = READ_ONCE(adapter->xdp_prog);
- if (!prog) {
- res = IGC_XDP_PASS;
- goto unlock;
- }
-
- act = bpf_prog_run_xdp(prog, xdp);
switch (act) {
case XDP_PASS:
- res = IGC_XDP_PASS;
- break;
+ return IGC_XDP_PASS;
case XDP_TX:
if (igc_xdp_xmit_back(adapter, xdp) < 0)
- res = IGC_XDP_CONSUMED;
- else
- res = IGC_XDP_TX;
- break;
+ goto out_failure;
+ return IGC_XDP_TX;
case XDP_REDIRECT:
if (xdp_do_redirect(adapter->netdev, xdp, prog) < 0)
- res = IGC_XDP_CONSUMED;
- else
- res = IGC_XDP_REDIRECT;
+ goto out_failure;
+ return IGC_XDP_REDIRECT;
break;
default:
bpf_warn_invalid_xdp_action(act);
fallthrough;
case XDP_ABORTED:
+out_failure:
trace_xdp_exception(adapter->netdev, prog, act);
fallthrough;
case XDP_DROP:
- res = IGC_XDP_CONSUMED;
- break;
+ return IGC_XDP_CONSUMED;
}
+}
+
+static struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
+ struct xdp_buff *xdp)
+{
+ struct bpf_prog *prog;
+ int res;
+
+ rcu_read_lock();
+
+ prog = READ_ONCE(adapter->xdp_prog);
+ if (!prog) {
+ res = IGC_XDP_PASS;
+ goto unlock;
+ }
+
+ res = __igc_xdp_run_prog(adapter, prog, xdp);
unlock:
rcu_read_unlock();
xdp_do_flush();
}
+static void igc_update_rx_stats(struct igc_q_vector *q_vector,
+ unsigned int packets, unsigned int bytes)
+{
+ struct igc_ring *ring = q_vector->rx.ring;
+
+ u64_stats_update_begin(&ring->rx_syncp);
+ ring->rx_stats.packets += packets;
+ ring->rx_stats.bytes += bytes;
+ u64_stats_update_end(&ring->rx_syncp);
+
+ q_vector->rx.total_packets += packets;
+ q_vector->rx.total_bytes += bytes;
+}
+
static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
{
unsigned int total_bytes = 0, total_packets = 0;
}
if (!skb) {
- xdp.data = pktbuf + pkt_offset;
- xdp.data_end = xdp.data + size;
- xdp.data_hard_start = pktbuf - igc_rx_offset(rx_ring);
- xdp_set_data_meta_invalid(&xdp);
- xdp.frame_sz = truesize;
- xdp.rxq = &rx_ring->xdp_rxq;
+ xdp_init_buff(&xdp, truesize, &rx_ring->xdp_rxq);
+ xdp_prepare_buff(&xdp, pktbuf - igc_rx_offset(rx_ring),
+ igc_rx_offset(rx_ring) + pkt_offset, size, false);
skb = igc_xdp_run_prog(adapter, &xdp);
}
/* place incomplete frames back on ring for completion */
rx_ring->skb = skb;
- u64_stats_update_begin(&rx_ring->rx_syncp);
- rx_ring->rx_stats.packets += total_packets;
- rx_ring->rx_stats.bytes += total_bytes;
- u64_stats_update_end(&rx_ring->rx_syncp);
- q_vector->rx.total_packets += total_packets;
- q_vector->rx.total_bytes += total_bytes;
+ igc_update_rx_stats(q_vector, total_packets, total_bytes);
if (cleaned_count)
igc_alloc_rx_buffers(rx_ring, cleaned_count);
return total_packets;
}
+static struct sk_buff *igc_construct_skb_zc(struct igc_ring *ring,
+ struct xdp_buff *xdp)
+{
+ unsigned int metasize = xdp->data - xdp->data_meta;
+ unsigned int datasize = xdp->data_end - xdp->data;
+ unsigned int totalsize = metasize + datasize;
+ struct sk_buff *skb;
+
+ skb = __napi_alloc_skb(&ring->q_vector->napi,
+ xdp->data_end - xdp->data_hard_start,
+ GFP_ATOMIC | __GFP_NOWARN);
+ if (unlikely(!skb))
+ return NULL;
+
+ skb_reserve(skb, xdp->data_meta - xdp->data_hard_start);
+ memcpy(__skb_put(skb, totalsize), xdp->data_meta, totalsize);
+ if (metasize)
+ skb_metadata_set(skb, metasize);
+
+ return skb;
+}
+
+static void igc_dispatch_skb_zc(struct igc_q_vector *q_vector,
+ union igc_adv_rx_desc *desc,
+ struct xdp_buff *xdp,
+ ktime_t timestamp)
+{
+ struct igc_ring *ring = q_vector->rx.ring;
+ struct sk_buff *skb;
+
+ skb = igc_construct_skb_zc(ring, xdp);
+ if (!skb) {
+ ring->rx_stats.alloc_failed++;
+ return;
+ }
+
+ if (timestamp)
+ skb_hwtstamps(skb)->hwtstamp = timestamp;
+
+ if (igc_cleanup_headers(ring, desc, skb))
+ return;
+
+ igc_process_skb_fields(ring, desc, skb);
+ napi_gro_receive(&q_vector->napi, skb);
+}
+
+static int igc_clean_rx_irq_zc(struct igc_q_vector *q_vector, const int budget)
+{
+ struct igc_adapter *adapter = q_vector->adapter;
+ struct igc_ring *ring = q_vector->rx.ring;
+ u16 cleaned_count = igc_desc_unused(ring);
+ int total_bytes = 0, total_packets = 0;
+ u16 ntc = ring->next_to_clean;
+ struct bpf_prog *prog;
+ bool failure = false;
+ int xdp_status = 0;
+
+ rcu_read_lock();
+
+ prog = READ_ONCE(adapter->xdp_prog);
+
+ while (likely(total_packets < budget)) {
+ union igc_adv_rx_desc *desc;
+ struct igc_rx_buffer *bi;
+ ktime_t timestamp = 0;
+ unsigned int size;
+ int res;
+
+ desc = IGC_RX_DESC(ring, ntc);
+ size = le16_to_cpu(desc->wb.upper.length);
+ if (!size)
+ break;
+
+ /* This memory barrier is needed to keep us from reading
+ * any other fields out of the rx_desc until we know the
+ * descriptor has been written back
+ */
+ dma_rmb();
+
+ bi = &ring->rx_buffer_info[ntc];
+
+ if (igc_test_staterr(desc, IGC_RXDADV_STAT_TSIP)) {
+ timestamp = igc_ptp_rx_pktstamp(q_vector->adapter,
+ bi->xdp->data);
+
+ bi->xdp->data += IGC_TS_HDR_LEN;
+
+ /* HW timestamp has been copied into local variable. Metadata
+ * length when XDP program is called should be 0.
+ */
+ bi->xdp->data_meta += IGC_TS_HDR_LEN;
+ size -= IGC_TS_HDR_LEN;
+ }
+
+ bi->xdp->data_end = bi->xdp->data + size;
+ xsk_buff_dma_sync_for_cpu(bi->xdp, ring->xsk_pool);
+
+ res = __igc_xdp_run_prog(adapter, prog, bi->xdp);
+ switch (res) {
+ case IGC_XDP_PASS:
+ igc_dispatch_skb_zc(q_vector, desc, bi->xdp, timestamp);
+ fallthrough;
+ case IGC_XDP_CONSUMED:
+ xsk_buff_free(bi->xdp);
+ break;
+ case IGC_XDP_TX:
+ case IGC_XDP_REDIRECT:
+ xdp_status |= res;
+ break;
+ }
+
+ bi->xdp = NULL;
+ total_bytes += size;
+ total_packets++;
+ cleaned_count++;
+ ntc++;
+ if (ntc == ring->count)
+ ntc = 0;
+ }
+
+ ring->next_to_clean = ntc;
+ rcu_read_unlock();
+
+ if (cleaned_count >= IGC_RX_BUFFER_WRITE)
+ failure = !igc_alloc_rx_buffers_zc(ring, cleaned_count);
+
+ if (xdp_status)
+ igc_finalize_xdp(adapter, xdp_status);
+
+ igc_update_rx_stats(q_vector, total_packets, total_bytes);
+
+ if (xsk_uses_need_wakeup(ring->xsk_pool)) {
+ if (failure || ring->next_to_clean == ring->next_to_use)
+ xsk_set_rx_need_wakeup(ring->xsk_pool);
+ else
+ xsk_clear_rx_need_wakeup(ring->xsk_pool);
+ return total_packets;
+ }
+
+ return failure ? budget : total_packets;
+}
+
+static void igc_update_tx_stats(struct igc_q_vector *q_vector,
+ unsigned int packets, unsigned int bytes)
+{
+ struct igc_ring *ring = q_vector->tx.ring;
+
+ u64_stats_update_begin(&ring->tx_syncp);
+ ring->tx_stats.bytes += bytes;
+ ring->tx_stats.packets += packets;
+ u64_stats_update_end(&ring->tx_syncp);
+
+ q_vector->tx.total_bytes += bytes;
+ q_vector->tx.total_packets += packets;
+}
+
+static void igc_xdp_xmit_zc(struct igc_ring *ring)
+{
+ struct xsk_buff_pool *pool = ring->xsk_pool;
+ struct netdev_queue *nq = txring_txq(ring);
+ union igc_adv_tx_desc *tx_desc = NULL;
+ int cpu = smp_processor_id();
+ u16 ntu = ring->next_to_use;
+ struct xdp_desc xdp_desc;
+ u16 budget;
+
+ if (!netif_carrier_ok(ring->netdev))
+ return;
+
+ __netif_tx_lock(nq, cpu);
+
+ budget = igc_desc_unused(ring);
+
+ while (xsk_tx_peek_desc(pool, &xdp_desc) && budget--) {
+ u32 cmd_type, olinfo_status;
+ struct igc_tx_buffer *bi;
+ dma_addr_t dma;
+
+ cmd_type = IGC_ADVTXD_DTYP_DATA | IGC_ADVTXD_DCMD_DEXT |
+ IGC_ADVTXD_DCMD_IFCS | IGC_TXD_DCMD |
+ xdp_desc.len;
+ olinfo_status = xdp_desc.len << IGC_ADVTXD_PAYLEN_SHIFT;
+
+ dma = xsk_buff_raw_get_dma(pool, xdp_desc.addr);
+ xsk_buff_raw_dma_sync_for_device(pool, dma, xdp_desc.len);
+
+ tx_desc = IGC_TX_DESC(ring, ntu);
+ tx_desc->read.cmd_type_len = cpu_to_le32(cmd_type);
+ tx_desc->read.olinfo_status = cpu_to_le32(olinfo_status);
+ tx_desc->read.buffer_addr = cpu_to_le64(dma);
+
+ bi = &ring->tx_buffer_info[ntu];
+ bi->type = IGC_TX_BUFFER_TYPE_XSK;
+ bi->protocol = 0;
+ bi->bytecount = xdp_desc.len;
+ bi->gso_segs = 1;
+ bi->time_stamp = jiffies;
+ bi->next_to_watch = tx_desc;
+
+ netdev_tx_sent_queue(txring_txq(ring), xdp_desc.len);
+
+ ntu++;
+ if (ntu == ring->count)
+ ntu = 0;
+ }
+
+ ring->next_to_use = ntu;
+ if (tx_desc) {
+ igc_flush_tx_descriptors(ring);
+ xsk_tx_release(pool);
+ }
+
+ __netif_tx_unlock(nq);
+}
+
/**
* igc_clean_tx_irq - Reclaim resources after transmit completes
* @q_vector: pointer to q_vector containing needed info
unsigned int i = tx_ring->next_to_clean;
struct igc_tx_buffer *tx_buffer;
union igc_adv_tx_desc *tx_desc;
+ u32 xsk_frames = 0;
if (test_bit(__IGC_DOWN, &adapter->state))
return true;
total_bytes += tx_buffer->bytecount;
total_packets += tx_buffer->gso_segs;
- if (tx_buffer->tx_flags & IGC_TX_FLAGS_XDP)
+ switch (tx_buffer->type) {
+ case IGC_TX_BUFFER_TYPE_XSK:
+ xsk_frames++;
+ break;
+ case IGC_TX_BUFFER_TYPE_XDP:
xdp_return_frame(tx_buffer->xdpf);
- else
+ igc_unmap_tx_buffer(tx_ring->dev, tx_buffer);
+ break;
+ case IGC_TX_BUFFER_TYPE_SKB:
napi_consume_skb(tx_buffer->skb, napi_budget);
-
- /* unmap skb header data */
- dma_unmap_single(tx_ring->dev,
- dma_unmap_addr(tx_buffer, dma),
- dma_unmap_len(tx_buffer, len),
- DMA_TO_DEVICE);
-
- /* clear tx_buffer data */
- dma_unmap_len_set(tx_buffer, len, 0);
+ igc_unmap_tx_buffer(tx_ring->dev, tx_buffer);
+ break;
+ default:
+ netdev_warn_once(tx_ring->netdev, "Unknown Tx buffer type\n");
+ break;
+ }
/* clear last DMA location and unmap remaining buffers */
while (tx_desc != eop_desc) {
}
/* unmap any remaining paged data */
- if (dma_unmap_len(tx_buffer, len)) {
- dma_unmap_page(tx_ring->dev,
- dma_unmap_addr(tx_buffer, dma),
- dma_unmap_len(tx_buffer, len),
- DMA_TO_DEVICE);
- dma_unmap_len_set(tx_buffer, len, 0);
- }
+ if (dma_unmap_len(tx_buffer, len))
+ igc_unmap_tx_buffer(tx_ring->dev, tx_buffer);
}
/* move us one more past the eop_desc for start of next pkt */
i += tx_ring->count;
tx_ring->next_to_clean = i;
- u64_stats_update_begin(&tx_ring->tx_syncp);
- tx_ring->tx_stats.bytes += total_bytes;
- tx_ring->tx_stats.packets += total_packets;
- u64_stats_update_end(&tx_ring->tx_syncp);
- q_vector->tx.total_bytes += total_bytes;
- q_vector->tx.total_packets += total_packets;
+
+ igc_update_tx_stats(q_vector, total_packets, total_bytes);
+
+ if (tx_ring->xsk_pool) {
+ if (xsk_frames)
+ xsk_tx_completed(tx_ring->xsk_pool, xsk_frames);
+ if (xsk_uses_need_wakeup(tx_ring->xsk_pool))
+ xsk_set_tx_need_wakeup(tx_ring->xsk_pool);
+ igc_xdp_xmit_zc(tx_ring);
+ }
if (test_bit(IGC_RING_FLAG_TX_DETECT_HANG, &tx_ring->flags)) {
struct igc_hw *hw = &adapter->hw;
igc_get_hw_control(adapter);
igc_set_rx_mode(netdev);
+ igc_restore_vlan(adapter);
+
igc_setup_tctl(adapter);
igc_setup_mrqc(adapter);
igc_setup_rctl(adapter);
for (i = 0; i < adapter->num_rx_queues; i++) {
struct igc_ring *ring = adapter->rx_ring[i];
- igc_alloc_rx_buffers(ring, igc_desc_unused(ring));
+ if (ring->xsk_pool)
+ igc_alloc_rx_buffers_zc(ring, igc_desc_unused(ring));
+ else
+ igc_alloc_rx_buffers(ring, igc_desc_unused(ring));
}
}
struct igc_q_vector *q_vector = container_of(napi,
struct igc_q_vector,
napi);
+ struct igc_ring *rx_ring = q_vector->rx.ring;
bool clean_complete = true;
int work_done = 0;
if (q_vector->tx.ring)
clean_complete = igc_clean_tx_irq(q_vector, budget);
- if (q_vector->rx.ring) {
- int cleaned = igc_clean_rx_irq(q_vector, budget);
+ if (rx_ring) {
+ int cleaned = rx_ring->xsk_pool ?
+ igc_clean_rx_irq_zc(q_vector, budget) :
+ igc_clean_rx_irq(q_vector, budget);
work_done += cleaned;
if (cleaned >= budget)
netdev_features_t changed = netdev->features ^ features;
struct igc_adapter *adapter = netdev_priv(netdev);
+ if (changed & NETIF_F_HW_VLAN_CTAG_RX)
+ igc_vlan_mode(netdev, features);
+
/* Add VLAN support */
if (!(changed & (NETIF_F_RXALL | NETIF_F_NTUPLE)))
return 0;
switch (bpf->command) {
case XDP_SETUP_PROG:
return igc_xdp_set_prog(adapter, bpf->prog, bpf->extack);
+ case XDP_SETUP_XSK_POOL:
+ return igc_xdp_setup_pool(adapter, bpf->xsk.pool,
+ bpf->xsk.queue_id);
default:
return -EOPNOTSUPP;
}
return num_frames - drops;
}
+static void igc_trigger_rxtxq_interrupt(struct igc_adapter *adapter,
+ struct igc_q_vector *q_vector)
+{
+ struct igc_hw *hw = &adapter->hw;
+ u32 eics = 0;
+
+ eics |= q_vector->eims_value;
+ wr32(IGC_EICS, eics);
+}
+
+int igc_xsk_wakeup(struct net_device *dev, u32 queue_id, u32 flags)
+{
+ struct igc_adapter *adapter = netdev_priv(dev);
+ struct igc_q_vector *q_vector;
+ struct igc_ring *ring;
+
+ if (test_bit(__IGC_DOWN, &adapter->state))
+ return -ENETDOWN;
+
+ if (!igc_xdp_is_enabled(adapter))
+ return -ENXIO;
+
+ if (queue_id >= adapter->num_rx_queues)
+ return -EINVAL;
+
+ ring = adapter->rx_ring[queue_id];
+
+ if (!ring->xsk_pool)
+ return -ENXIO;
+
+ q_vector = adapter->q_vector[queue_id];
+ if (!napi_if_scheduled_mark_missed(&q_vector->napi))
+ igc_trigger_rxtxq_interrupt(adapter, q_vector);
+
+ return 0;
+}
+
static const struct net_device_ops igc_netdev_ops = {
.ndo_open = igc_open,
.ndo_stop = igc_close,
.ndo_setup_tc = igc_setup_tc,
.ndo_bpf = igc_bpf,
.ndo_xdp_xmit = igc_xdp_xmit,
+ .ndo_xsk_wakeup = igc_xsk_wakeup,
};
/* PCIe configuration access */
/* copy netdev features into list of user selectable features */
netdev->hw_features |= NETIF_F_NTUPLE;
+ netdev->hw_features |= NETIF_F_HW_VLAN_CTAG_TX;
+ netdev->hw_features |= NETIF_F_HW_VLAN_CTAG_RX;
netdev->hw_features |= netdev->features;
if (pci_using_dac)
netdev->features |= NETIF_F_HIGHDMA;
+ netdev->vlan_features |= netdev->features;
+
/* MTU range: 68 - 9216 */
netdev->min_mtu = ETH_MIN_MTU;
netdev->max_mtu = MAX_STD_JUMBO_FRAME_SIZE;
return adapter->netdev;
}
+static void igc_disable_rx_ring_hw(struct igc_ring *ring)
+{
+ struct igc_hw *hw = &ring->q_vector->adapter->hw;
+ u8 idx = ring->reg_idx;
+ u32 rxdctl;
+
+ rxdctl = rd32(IGC_RXDCTL(idx));
+ rxdctl &= ~IGC_RXDCTL_QUEUE_ENABLE;
+ rxdctl |= IGC_RXDCTL_SWFLUSH;
+ wr32(IGC_RXDCTL(idx), rxdctl);
+}
+
+void igc_disable_rx_ring(struct igc_ring *ring)
+{
+ igc_disable_rx_ring_hw(ring);
+ igc_clean_rx_ring(ring);
+}
+
+void igc_enable_rx_ring(struct igc_ring *ring)
+{
+ struct igc_adapter *adapter = ring->q_vector->adapter;
+
+ igc_configure_rx_ring(adapter, ring);
+
+ if (ring->xsk_pool)
+ igc_alloc_rx_buffers_zc(ring, igc_desc_unused(ring));
+ else
+ igc_alloc_rx_buffers(ring, igc_desc_unused(ring));
+}
+
+static void igc_disable_tx_ring_hw(struct igc_ring *ring)
+{
+ struct igc_hw *hw = &ring->q_vector->adapter->hw;
+ u8 idx = ring->reg_idx;
+ u32 txdctl;
+
+ txdctl = rd32(IGC_TXDCTL(idx));
+ txdctl &= ~IGC_TXDCTL_QUEUE_ENABLE;
+ txdctl |= IGC_TXDCTL_SWFLUSH;
+ wr32(IGC_TXDCTL(idx), txdctl);
+}
+
+void igc_disable_tx_ring(struct igc_ring *ring)
+{
+ igc_disable_tx_ring_hw(ring);
+ igc_clean_tx_ring(ring);
+}
+
+void igc_enable_tx_ring(struct igc_ring *ring)
+{
+ struct igc_adapter *adapter = ring->q_vector->adapter;
+
+ igc_configure_tx_ring(adapter, ring);
+}
+
/**
* igc_init_module - Driver Registration Routine
*
#define IGC_EECD 0x00010 /* EEPROM/Flash Control - RW */
#define IGC_CTRL_EXT 0x00018 /* Extended Device Control - RW */
#define IGC_MDIC 0x00020 /* MDI Control - RW */
-#define IGC_MDICNFG 0x00E04 /* MDC/MDIO Configuration - RW */
#define IGC_CONNSW 0x00034 /* Copper/Fiber switch control - RW */
+#define IGC_VET 0x00038 /* VLAN Ether Type - RW */
#define IGC_I225_PHPM 0x00E14 /* I225 PHY Power Management */
#define IGC_GPHY_VERSION 0x0001E /* I225 gPHY Firmware Version */
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2020, Intel Corporation. */
+#include <net/xdp_sock_drv.h>
+
#include "igc.h"
#include "igc_xdp.h"
return 0;
}
-int igc_xdp_register_rxq_info(struct igc_ring *ring)
+static int igc_xdp_enable_pool(struct igc_adapter *adapter,
+ struct xsk_buff_pool *pool, u16 queue_id)
{
- struct net_device *dev = ring->netdev;
+ struct net_device *ndev = adapter->netdev;
+ struct device *dev = &adapter->pdev->dev;
+ struct igc_ring *rx_ring, *tx_ring;
+ struct napi_struct *napi;
+ bool needs_reset;
+ u32 frame_size;
int err;
- err = xdp_rxq_info_reg(&ring->xdp_rxq, dev, ring->queue_index, 0);
- if (err) {
- netdev_err(dev, "Failed to register xdp rxq info\n");
- return err;
+ if (queue_id >= adapter->num_rx_queues ||
+ queue_id >= adapter->num_tx_queues)
+ return -EINVAL;
+
+ frame_size = xsk_pool_get_rx_frame_size(pool);
+ if (frame_size < ETH_FRAME_LEN + VLAN_HLEN * 2) {
+ /* When XDP is enabled, the driver doesn't support frames that
+ * span over multiple buffers. To avoid that, we check if xsk
+ * frame size is big enough to fit the max ethernet frame size
+ * + vlan double tagging.
+ */
+ return -EOPNOTSUPP;
}
- err = xdp_rxq_info_reg_mem_model(&ring->xdp_rxq, MEM_TYPE_PAGE_SHARED,
- NULL);
+ err = xsk_pool_dma_map(pool, dev, IGC_RX_DMA_ATTR);
if (err) {
- netdev_err(dev, "Failed to register xdp rxq mem model\n");
- xdp_rxq_info_unreg(&ring->xdp_rxq);
+ netdev_err(ndev, "Failed to map xsk pool\n");
return err;
}
+ needs_reset = netif_running(adapter->netdev) && igc_xdp_is_enabled(adapter);
+
+ rx_ring = adapter->rx_ring[queue_id];
+ tx_ring = adapter->tx_ring[queue_id];
+ /* Rx and Tx rings share the same napi context. */
+ napi = &rx_ring->q_vector->napi;
+
+ if (needs_reset) {
+ igc_disable_rx_ring(rx_ring);
+ igc_disable_tx_ring(tx_ring);
+ napi_disable(napi);
+ }
+
+ set_bit(IGC_RING_FLAG_AF_XDP_ZC, &rx_ring->flags);
+ set_bit(IGC_RING_FLAG_AF_XDP_ZC, &tx_ring->flags);
+
+ if (needs_reset) {
+ napi_enable(napi);
+ igc_enable_rx_ring(rx_ring);
+ igc_enable_tx_ring(tx_ring);
+
+ err = igc_xsk_wakeup(ndev, queue_id, XDP_WAKEUP_RX);
+ if (err) {
+ xsk_pool_dma_unmap(pool, IGC_RX_DMA_ATTR);
+ return err;
+ }
+ }
+
+ return 0;
+}
+
+static int igc_xdp_disable_pool(struct igc_adapter *adapter, u16 queue_id)
+{
+ struct igc_ring *rx_ring, *tx_ring;
+ struct xsk_buff_pool *pool;
+ struct napi_struct *napi;
+ bool needs_reset;
+
+ if (queue_id >= adapter->num_rx_queues ||
+ queue_id >= adapter->num_tx_queues)
+ return -EINVAL;
+
+ pool = xsk_get_pool_from_qid(adapter->netdev, queue_id);
+ if (!pool)
+ return -EINVAL;
+
+ needs_reset = netif_running(adapter->netdev) && igc_xdp_is_enabled(adapter);
+
+ rx_ring = adapter->rx_ring[queue_id];
+ tx_ring = adapter->tx_ring[queue_id];
+ /* Rx and Tx rings share the same napi context. */
+ napi = &rx_ring->q_vector->napi;
+
+ if (needs_reset) {
+ igc_disable_rx_ring(rx_ring);
+ igc_disable_tx_ring(tx_ring);
+ napi_disable(napi);
+ }
+
+ xsk_pool_dma_unmap(pool, IGC_RX_DMA_ATTR);
+ clear_bit(IGC_RING_FLAG_AF_XDP_ZC, &rx_ring->flags);
+ clear_bit(IGC_RING_FLAG_AF_XDP_ZC, &tx_ring->flags);
+
+ if (needs_reset) {
+ napi_enable(napi);
+ igc_enable_rx_ring(rx_ring);
+ igc_enable_tx_ring(tx_ring);
+ }
+
return 0;
}
-void igc_xdp_unregister_rxq_info(struct igc_ring *ring)
+int igc_xdp_setup_pool(struct igc_adapter *adapter, struct xsk_buff_pool *pool,
+ u16 queue_id)
{
- xdp_rxq_info_unreg(&ring->xdp_rxq);
+ return pool ? igc_xdp_enable_pool(adapter, pool, queue_id) :
+ igc_xdp_disable_pool(adapter, queue_id);
}
int igc_xdp_set_prog(struct igc_adapter *adapter, struct bpf_prog *prog,
struct netlink_ext_ack *extack);
+int igc_xdp_setup_pool(struct igc_adapter *adapter, struct xsk_buff_pool *pool,
+ u16 queue_id);
-int igc_xdp_register_rxq_info(struct igc_ring *ring);
-void igc_xdp_unregister_rxq_info(struct igc_ring *ring);
+static inline bool igc_xdp_is_enabled(struct igc_adapter *adapter)
+{
+ return !!adapter->xdp_prog;
+}
#endif /* _IGC_XDP_H_ */
#define IXGBE_WRITE_REG_BE32(a, reg, value) \
IXGBE_WRITE_REG((a), (reg), IXGBE_STORE_AS_BE32(ntohl(value)))
-#define IXGBE_STORE_AS_BE16(_value) \
- ntohs(((u16)(_value) >> 8) | ((u16)(_value) << 8))
+#define IXGBE_STORE_AS_BE16(_value) __swab16(ntohs((_value)))
s32 ixgbe_fdir_set_input_mask_82599(struct ixgbe_hw *hw,
union ixgbe_atr_input *input_mask)
IXGBE_WRITE_REG_BE32(hw, IXGBE_FDIRIPDA, input->formatted.dst_ip[0]);
/* record source and destination port (little-endian)*/
- fdirport = ntohs(input->formatted.dst_port);
+ fdirport = be16_to_cpu(input->formatted.dst_port);
fdirport <<= IXGBE_FDIRPORT_DESTINATION_SHIFT;
- fdirport |= ntohs(input->formatted.src_port);
+ fdirport |= be16_to_cpu(input->formatted.src_port);
IXGBE_WRITE_REG(hw, IXGBE_FDIRPORT, fdirport);
/* record vlan (little-endian) and flex_bytes(big-endian) */
- fdirvlan = IXGBE_STORE_AS_BE16((__force u16)input->formatted.flex_bytes);
+ fdirvlan = IXGBE_STORE_AS_BE16(input->formatted.flex_bytes);
fdirvlan <<= IXGBE_FDIRVLAN_FLEX_SHIFT;
fdirvlan |= ntohs(input->formatted.vlan_id);
IXGBE_WRITE_REG(hw, IXGBE_FDIRVLAN, fdirvlan);
bool return_data)
{
u32 hdr_size = sizeof(struct ixgbe_hic_hdr);
- union {
- struct ixgbe_hic_hdr hdr;
- u32 u32arr[1];
- } *bp = buffer;
+ struct ixgbe_hic_hdr *hdr = buffer;
+ u32 *u32arr = buffer;
u16 buf_len, dword_len;
s32 status;
u32 bi;
/* first pull in the header so we know the buffer length */
for (bi = 0; bi < dword_len; bi++) {
- bp->u32arr[bi] = IXGBE_READ_REG_ARRAY(hw, IXGBE_FLEX_MNG, bi);
- le32_to_cpus(&bp->u32arr[bi]);
+ u32arr[bi] = IXGBE_READ_REG_ARRAY(hw, IXGBE_FLEX_MNG, bi);
+ le32_to_cpus(&u32arr[bi]);
}
/* If there is any thing in data position pull it in */
- buf_len = bp->hdr.buf_len;
+ buf_len = hdr->buf_len;
if (!buf_len)
goto rel_out;
/* Pull in the rest of the buffer (bi is where we left off) */
for (; bi <= dword_len; bi++) {
- bp->u32arr[bi] = IXGBE_READ_REG_ARRAY(hw, IXGBE_FLEX_MNG, bi);
- le32_to_cpus(&bp->u32arr[bi]);
+ u32arr[bi] = IXGBE_READ_REG_ARRAY(hw, IXGBE_FLEX_MNG, bi);
+ le32_to_cpus(&u32arr[bi]);
}
rel_out:
continue;
reg = IXGBE_READ_REG(hw, MIPAF_ARR(3, i));
- if (reg == xs->id.daddr.a4)
+ if (reg == (__force u32)xs->id.daddr.a4)
return 1;
}
}
if ((bmcipval & BMCIP_MASK) == BMCIP_V4) {
reg = IXGBE_READ_REG(hw, IXGBE_BMCIP(3));
- if (reg == xs->id.daddr.a4)
+ if (reg == (__force u32)xs->id.daddr.a4)
return 1;
}
for (j = 0; j < 4; j++) {
reg = IXGBE_READ_REG(hw, MIPAF_ARR(i, j));
- if (reg != xs->id.daddr.a6[j])
+ if (reg != (__force u32)xs->id.daddr.a6[j])
break;
}
if (j == 4) /* did we match all 4 words? */
if ((bmcipval & BMCIP_MASK) == BMCIP_V6) {
for (j = 0; j < 4; j++) {
reg = IXGBE_READ_REG(hw, IXGBE_BMCIP(j));
- if (reg != xs->id.daddr.a6[j])
+ if (reg != (__force u32)xs->id.daddr.a6[j])
break;
}
if (j == 4) /* did we match all 4 words? */
break;
case XDP_TX:
xdpf = xdp_convert_buff_to_frame(xdp);
- if (unlikely(!xdpf)) {
- result = IXGBE_XDP_CONSUMED;
- break;
- }
+ if (unlikely(!xdpf))
+ goto out_failure;
result = ixgbe_xmit_xdp_ring(adapter, xdpf);
+ if (result == IXGBE_XDP_CONSUMED)
+ goto out_failure;
break;
case XDP_REDIRECT:
err = xdp_do_redirect(adapter->netdev, xdp, xdp_prog);
- if (!err)
- result = IXGBE_XDP_REDIR;
- else
- result = IXGBE_XDP_CONSUMED;
+ if (err)
+ goto out_failure;
+ result = IXGBE_XDP_REDIR;
break;
default:
bpf_warn_invalid_xdp_action(act);
fallthrough;
case XDP_ABORTED:
+out_failure:
trace_xdp_exception(rx_ring->netdev, xdp_prog, act);
fallthrough; /* handle aborts by dropping packet */
case XDP_DROP:
return err;
}
-static s32 ixgbe_set_vf_lpe(struct ixgbe_adapter *adapter, u32 *msgbuf, u32 vf)
+static int ixgbe_set_vf_lpe(struct ixgbe_adapter *adapter, u32 max_frame, u32 vf)
{
struct ixgbe_hw *hw = &adapter->hw;
- int max_frame = msgbuf[1];
u32 max_frs;
+ if (max_frame < ETH_MIN_MTU || max_frame > IXGBE_MAX_JUMBO_FRAME_SIZE) {
+ e_err(drv, "VF max_frame %d out of range\n", max_frame);
+ return -EINVAL;
+ }
+
/*
* For 82599EB we have to keep all PFs and VFs operating with
* the same max_frame value in order to avoid sending an oversize
}
}
- /* MTU < 68 is an error and causes problems on some kernels */
- if (max_frame > IXGBE_MAX_JUMBO_FRAME_SIZE) {
- e_err(drv, "VF max_frame %d out of range\n", max_frame);
- return -EINVAL;
- }
-
/* pull current max frame size from hardware */
max_frs = IXGBE_READ_REG(hw, IXGBE_MAXFRS);
max_frs &= IXGBE_MHADD_MFS_MASK;
retval = ixgbe_set_vf_vlan_msg(adapter, msgbuf, vf);
break;
case IXGBE_VF_SET_LPE:
- retval = ixgbe_set_vf_lpe(adapter, msgbuf, vf);
+ retval = ixgbe_set_vf_lpe(adapter, msgbuf[1], vf);
break;
case IXGBE_VF_SET_MACVLAN:
retval = ixgbe_set_vf_macvlan_msg(adapter, msgbuf, vf);
if (likely(act == XDP_REDIRECT)) {
err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
- result = !err ? IXGBE_XDP_REDIR : IXGBE_XDP_CONSUMED;
+ if (err)
+ goto out_failure;
rcu_read_unlock();
- return result;
+ return IXGBE_XDP_REDIR;
}
switch (act) {
break;
case XDP_TX:
xdpf = xdp_convert_buff_to_frame(xdp);
- if (unlikely(!xdpf)) {
- result = IXGBE_XDP_CONSUMED;
- break;
- }
+ if (unlikely(!xdpf))
+ goto out_failure;
result = ixgbe_xmit_xdp_ring(adapter, xdpf);
+ if (result == IXGBE_XDP_CONSUMED)
+ goto out_failure;
break;
default:
bpf_warn_invalid_xdp_action(act);
fallthrough;
case XDP_ABORTED:
+out_failure:
trace_xdp_exception(rx_ring->netdev, xdp_prog, act);
fallthrough; /* handle aborts by dropping packet */
case XDP_DROP:
case XDP_TX:
xdp_ring = adapter->xdp_ring[rx_ring->queue_index];
result = ixgbevf_xmit_xdp_ring(xdp_ring, xdp);
+ if (result == IXGBEVF_XDP_CONSUMED)
+ goto out_failure;
break;
default:
bpf_warn_invalid_xdp_action(act);
fallthrough;
case XDP_ABORTED:
+out_failure:
trace_xdp_exception(rx_ring->netdev, xdp_prog, act);
fallthrough; /* handle aborts by dropping packet */
case XDP_DROP:
/* remove payload length from inner checksum */
paylen = skb->len - l4_offset;
- csum_replace_by_diff(&l4.tcp->check, htonl(paylen));
+ csum_replace_by_diff(&l4.tcp->check, (__force __wsum)htonl(paylen));
/* update gso size and bytecount with header size */
first->gso_segs = skb_shinfo(skb)->gso_segs;
lp->tx_irq = platform_get_irq_byname(pdev, "tx");
p = devm_platform_ioremap_resource_byname(pdev, "emac");
- if (!p) {
+ if (IS_ERR(p)) {
printk(KERN_ERR DRV_NAME ": cannot remap registers\n");
- return -ENOMEM;
+ return PTR_ERR(p);
}
lp->eth_regs = p;
p = devm_platform_ioremap_resource_byname(pdev, "dma_rx");
- if (!p) {
+ if (IS_ERR(p)) {
printk(KERN_ERR DRV_NAME ": cannot remap Rx DMA registers\n");
- return -ENOMEM;
+ return PTR_ERR(p);
}
lp->rx_dma_regs = p;
p = devm_platform_ioremap_resource_byname(pdev, "dma_tx");
- if (!p) {
+ if (IS_ERR(p)) {
printk(KERN_ERR DRV_NAME ": cannot remap Tx DMA registers\n");
- return -ENOMEM;
+ return PTR_ERR(p);
}
lp->tx_dma_regs = p;
static int xrx200_alloc_skb(struct xrx200_chan *ch)
{
+ struct sk_buff *skb = ch->skb[ch->dma.desc];
+ dma_addr_t mapping;
int ret = 0;
ch->skb[ch->dma.desc] = netdev_alloc_skb_ip_align(ch->priv->net_dev,
goto skip;
}
- ch->dma.desc_base[ch->dma.desc].addr = dma_map_single(ch->priv->dev,
- ch->skb[ch->dma.desc]->data, XRX200_DMA_DATA_LEN,
- DMA_FROM_DEVICE);
- if (unlikely(dma_mapping_error(ch->priv->dev,
- ch->dma.desc_base[ch->dma.desc].addr))) {
+ mapping = dma_map_single(ch->priv->dev, ch->skb[ch->dma.desc]->data,
+ XRX200_DMA_DATA_LEN, DMA_FROM_DEVICE);
+ if (unlikely(dma_mapping_error(ch->priv->dev, mapping))) {
dev_kfree_skb_any(ch->skb[ch->dma.desc]);
+ ch->skb[ch->dma.desc] = skb;
ret = -ENOMEM;
goto skip;
}
+ ch->dma.desc_base[ch->dma.desc].addr = mapping;
+ /* Make sure the address is written before we give it to HW */
+ wmb();
skip:
ch->dma.desc_base[ch->dma.desc].ctl =
LTQ_DMA_OWN | LTQ_DMA_RX_OFFSET(NET_IP_ALIGN) |
ch->dma.desc %= LTQ_DESC_NUM;
if (ret) {
+ net_dev->stats.rx_dropped++;
netdev_err(net_dev, "failed to allocate new rx buffer\n");
return ret;
}
struct xrx200_chan *ch = ptr;
if (napi_schedule_prep(&ch->napi)) {
- __napi_schedule(&ch->napi);
ltq_dma_disable_irq(&ch->dma);
+ __napi_schedule(&ch->napi);
}
ltq_dma_ack_irq(&ch->dma);
{
struct device *dev = &pdev->dev;
struct device_node *np = dev->of_node;
- struct resource *res;
struct xrx200_priv *priv;
struct net_device *net_dev;
int err;
net_dev->max_mtu = XRX200_DMA_DATA_LEN;
/* load the memory ranges */
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!res) {
- dev_err(dev, "failed to get resources\n");
- return -ENOENT;
- }
-
- priv->pmac_reg = devm_ioremap_resource(dev, res);
+ priv->pmac_reg = devm_platform_get_and_ioremap_resource(pdev, 0, NULL);
if (IS_ERR(priv->pmac_reg))
return PTR_ERR(priv->pmac_reg);
* warranty of any kind, whether express or implied.
*/
+#include <linux/acpi.h>
+#include <linux/acpi_mdio.h>
#include <linux/clk.h>
#include <linux/delay.h>
#include <linux/interrupt.h>
struct orion_mdio_dev *dev;
int i, ret;
- type = (enum orion_mdio_bus_type)of_device_get_match_data(&pdev->dev);
+ type = (enum orion_mdio_bus_type)device_get_match_data(&pdev->dev);
r = platform_get_resource(pdev, IORESOURCE_MEM, 0);
if (!r) {
goto out_mdio;
}
- ret = of_mdiobus_register(bus, pdev->dev.of_node);
+ /* For the platforms not supporting DT/ACPI fall-back
+ * to mdiobus_register via of_mdiobus_register.
+ */
+ if (is_acpi_node(pdev->dev.fwnode))
+ ret = acpi_mdiobus_register(bus, pdev->dev.fwnode);
+ else
+ ret = of_mdiobus_register(bus, pdev->dev.of_node);
if (ret < 0) {
dev_err(&pdev->dev, "Cannot register MDIO bus (%d)\n", ret);
goto out_mdio;
};
MODULE_DEVICE_TABLE(of, orion_mdio_match);
+static const struct acpi_device_id orion_mdio_acpi_match[] = {
+ { "MRVL0100", BUS_TYPE_SMI },
+ { "MRVL0101", BUS_TYPE_XSMI },
+ { },
+};
+MODULE_DEVICE_TABLE(acpi, orion_mdio_acpi_match);
+
static struct platform_driver orion_mdio_driver = {
.probe = orion_mdio_probe,
.remove = orion_mdio_remove,
.driver = {
.name = "orion-mdio",
.of_match_table = orion_mdio_match,
+ .acpi_match_table = ACPI_PTR(orion_mdio_acpi_match),
},
};
}
/* Handle RX checksum offload based on the descriptor's status */
-static void mvneta_rx_csum(struct mvneta_port *pp, u32 status,
- struct sk_buff *skb)
+static int mvneta_rx_csum(struct mvneta_port *pp, u32 status)
{
if ((pp->dev->features & NETIF_F_RXCSUM) &&
(status & MVNETA_RXD_L3_IP4) &&
- (status & MVNETA_RXD_L4_CSUM_OK)) {
- skb->csum = 0;
- skb->ip_summed = CHECKSUM_UNNECESSARY;
- return;
- }
+ (status & MVNETA_RXD_L4_CSUM_OK))
+ return CHECKSUM_UNNECESSARY;
- skb->ip_summed = CHECKSUM_NONE;
+ return CHECKSUM_NONE;
}
/* Return tx queue pointer (find last set bit) according to <cause> returned
}
static struct sk_buff *
-mvneta_swbm_build_skb(struct mvneta_port *pp, struct mvneta_rx_queue *rxq,
+mvneta_swbm_build_skb(struct mvneta_port *pp, struct page_pool *pool,
struct xdp_buff *xdp, u32 desc_status)
{
struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp);
if (!skb)
return ERR_PTR(-ENOMEM);
- page_pool_release_page(rxq->page_pool, virt_to_page(xdp->data));
+ skb_mark_for_recycle(skb, virt_to_page(xdp->data), pool);
skb_reserve(skb, xdp->data - xdp->data_hard_start);
skb_put(skb, xdp->data_end - xdp->data);
- mvneta_rx_csum(pp, desc_status, skb);
+ skb->ip_summed = mvneta_rx_csum(pp, desc_status);
for (i = 0; i < num_frags; i++) {
skb_frag_t *frag = &sinfo->frags[i];
skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
skb_frag_page(frag), skb_frag_off(frag),
skb_frag_size(frag), PAGE_SIZE);
- page_pool_release_page(rxq->page_pool, skb_frag_page(frag));
+ /* We don't need to reset pp_recycle here. It's already set, so
+ * just mark fragments for recycling.
+ */
+ page_pool_store_mem_info(skb_frag_page(frag), pool);
}
return skb;
mvneta_run_xdp(pp, rxq, xdp_prog, &xdp_buf, frame_sz, &ps))
goto next;
- skb = mvneta_swbm_build_skb(pp, rxq, &xdp_buf, desc_status);
+ skb = mvneta_swbm_build_skb(pp, rxq->page_pool, &xdp_buf, desc_status);
if (IS_ERR(skb)) {
struct mvneta_pcpu_stats *stats = this_cpu_ptr(pp->stats);
rx_bytes);
skb->protocol = eth_type_trans(skb, dev);
- mvneta_rx_csum(pp, rx_status, skb);
+ skb->ip_summed = mvneta_rx_csum(pp, rx_status);
napi_gro_receive(napi, skb);
rcvd_pkts++;
skb_put(skb, rx_bytes);
skb->protocol = eth_type_trans(skb, dev);
-
- mvneta_rx_csum(pp, rx_status, skb);
+ skb->ip_summed = mvneta_rx_csum(pp, rx_status);
napi_gro_receive(napi, skb);
}
#define MVPP2_DESC_DMA_MASK DMA_BIT_MASK(40)
+/* Buffer header info bits */
+#define MVPP2_B_HDR_INFO_MC_ID_MASK 0xfff
+#define MVPP2_B_HDR_INFO_MC_ID(info) ((info) & MVPP2_B_HDR_INFO_MC_ID_MASK)
+#define MVPP2_B_HDR_INFO_LAST_OFFS 12
+#define MVPP2_B_HDR_INFO_LAST_MASK BIT(12)
+#define MVPP2_B_HDR_INFO_IS_LAST(info) \
+ (((info) & MVPP2_B_HDR_INFO_LAST_MASK) >> MVPP2_B_HDR_INFO_LAST_OFFS)
+
struct mvpp2_tai;
/* Definitions */
u32 indir[MVPP22_RSS_TABLE_ENTRIES];
};
+struct mvpp2_buff_hdr {
+ __le32 next_phys_addr;
+ __le32 next_dma_addr;
+ __le16 byte_count;
+ __le16 info;
+ __le16 reserved1; /* bm_qset (for future use, BM) */
+ u8 next_phys_addr_high;
+ u8 next_dma_addr_high;
+ __le16 reserved2;
+ __le16 reserved3;
+ __le16 reserved4;
+ __le16 reserved5;
+};
+
/* Shared Packet Processor resources */
struct mvpp2 {
/* Shared registers' base addresses */
/* Firmware node associated to the port */
struct fwnode_handle *fwnode;
- /* Is a PHY always connected to the port */
- bool has_phy;
-
/* Per-port registers' base address */
void __iomem *base;
void __iomem *stats_base;
}
/* Handle RX checksum offload */
-static void mvpp2_rx_csum(struct mvpp2_port *port, u32 status,
- struct sk_buff *skb)
+static int mvpp2_rx_csum(struct mvpp2_port *port, u32 status)
{
if (((status & MVPP2_RXD_L3_IP4) &&
!(status & MVPP2_RXD_IP4_HEADER_ERR)) ||
(status & MVPP2_RXD_L3_IP6))
if (((status & MVPP2_RXD_L4_UDP) ||
(status & MVPP2_RXD_L4_TCP)) &&
- (status & MVPP2_RXD_L4_CSUM_OK)) {
- skb->csum = 0;
- skb->ip_summed = CHECKSUM_UNNECESSARY;
- return;
- }
+ (status & MVPP2_RXD_L4_CSUM_OK))
+ return CHECKSUM_UNNECESSARY;
- skb->ip_summed = CHECKSUM_NONE;
+ return CHECKSUM_NONE;
}
/* Allocate a new skb and add it to BM pool */
return ret;
}
+static void mvpp2_buff_hdr_pool_put(struct mvpp2_port *port, struct mvpp2_rx_desc *rx_desc,
+ int pool, u32 rx_status)
+{
+ phys_addr_t phys_addr, phys_addr_next;
+ dma_addr_t dma_addr, dma_addr_next;
+ struct mvpp2_buff_hdr *buff_hdr;
+
+ phys_addr = mvpp2_rxdesc_dma_addr_get(port, rx_desc);
+ dma_addr = mvpp2_rxdesc_cookie_get(port, rx_desc);
+
+ do {
+ buff_hdr = (struct mvpp2_buff_hdr *)phys_to_virt(phys_addr);
+
+ phys_addr_next = le32_to_cpu(buff_hdr->next_phys_addr);
+ dma_addr_next = le32_to_cpu(buff_hdr->next_dma_addr);
+
+ if (port->priv->hw_version >= MVPP22) {
+ phys_addr_next |= ((u64)buff_hdr->next_phys_addr_high << 32);
+ dma_addr_next |= ((u64)buff_hdr->next_dma_addr_high << 32);
+ }
+
+ mvpp2_bm_pool_put(port, pool, dma_addr, phys_addr);
+
+ phys_addr = phys_addr_next;
+ dma_addr = dma_addr_next;
+
+ } while (!MVPP2_B_HDR_INFO_IS_LAST(le16_to_cpu(buff_hdr->info)));
+}
+
/* Main rx processing */
static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
int rx_todo, struct mvpp2_rx_queue *rxq)
phys_addr_t phys_addr;
u32 rx_status, timestamp;
int pool, rx_bytes, err, ret;
+ struct page *page;
void *data;
+ phys_addr = mvpp2_rxdesc_cookie_get(port, rx_desc);
+ data = (void *)phys_to_virt(phys_addr);
+ page = virt_to_page(data);
+ prefetch(page);
+
rx_done++;
rx_status = mvpp2_rxdesc_status_get(port, rx_desc);
rx_bytes = mvpp2_rxdesc_size_get(port, rx_desc);
rx_bytes -= MVPP2_MH_SIZE;
dma_addr = mvpp2_rxdesc_dma_addr_get(port, rx_desc);
- phys_addr = mvpp2_rxdesc_cookie_get(port, rx_desc);
- data = (void *)phys_to_virt(phys_addr);
pool = (rx_status & MVPP2_RXD_BM_POOL_ID_MASK) >>
MVPP2_RXD_BM_POOL_ID_OFFS;
bm_pool = &port->priv->bm_pools[pool];
- /* In case of an error, release the requested buffer pointer
- * to the Buffer Manager. This request process is controlled
- * by the hardware, and the information about the buffer is
- * comprised by the RX descriptor.
- */
- if (rx_status & MVPP2_RXD_ERR_SUMMARY)
- goto err_drop_frame;
-
if (port->priv->percpu_pools) {
pp = port->priv->page_pool[pool];
dma_dir = page_pool_get_dma_dir(pp);
rx_bytes + MVPP2_MH_SIZE,
dma_dir);
+ /* Buffer header not supported */
+ if (rx_status & MVPP2_RXD_BUF_HDR)
+ goto err_drop_frame;
+
+ /* In case of an error, release the requested buffer pointer
+ * to the Buffer Manager. This request process is controlled
+ * by the hardware, and the information about the buffer is
+ * comprised by the RX descriptor.
+ */
+ if (rx_status & MVPP2_RXD_ERR_SUMMARY)
+ goto err_drop_frame;
+
/* Prefetch header */
- prefetch(data);
+ prefetch(data + MVPP2_MH_SIZE + MVPP2_SKB_HEADROOM);
if (bm_pool->frag_size > PAGE_SIZE)
frag_size = 0;
}
if (pp)
- page_pool_release_page(pp, virt_to_page(data));
+ skb_mark_for_recycle(skb, page, pp);
else
dma_unmap_single_attrs(dev->dev.parent, dma_addr,
bm_pool->buf_size, DMA_FROM_DEVICE,
skb_reserve(skb, MVPP2_MH_SIZE + MVPP2_SKB_HEADROOM);
skb_put(skb, rx_bytes);
+ skb->ip_summed = mvpp2_rx_csum(port, rx_status);
skb->protocol = eth_type_trans(skb, dev);
- mvpp2_rx_csum(port, rx_status, skb);
napi_gro_receive(napi, skb);
continue;
dev->stats.rx_errors++;
mvpp2_rx_error(port, rx_desc);
/* Return the buffer to the pool */
- mvpp2_bm_pool_put(port, pool, dma_addr, phys_addr);
+ if (rx_status & MVPP2_RXD_BUF_HDR)
+ mvpp2_buff_hdr_pool_put(port, rx_desc, pool, rx_status);
+ else
+ mvpp2_bm_pool_put(port, pool, dma_addr, phys_addr);
}
rcu_read_unlock();
goto err_cleanup_txqs;
}
- /* Phylink isn't supported yet in ACPI mode */
- if (port->of_node) {
- err = phylink_of_phy_connect(port->phylink, port->of_node, 0);
+ if (port->phylink) {
+ err = phylink_fwnode_phy_connect(port->phylink, port->fwnode, 0);
if (err) {
netdev_err(port->dev, "could not attach PHY (%d)\n",
err);
SPEED_UNKNOWN, DUPLEX_UNKNOWN, false, false);
}
+/* In order to ensure backward compatibility for ACPI, check if the port
+ * firmware node comprises the necessary description allowing to use phylink.
+ */
+static bool mvpp2_use_acpi_compat_mode(struct fwnode_handle *port_fwnode)
+{
+ if (!is_acpi_node(port_fwnode))
+ return false;
+
+ return (!fwnode_property_present(port_fwnode, "phy-handle") &&
+ !fwnode_property_present(port_fwnode, "managed") &&
+ !fwnode_get_named_child_node(port_fwnode, "fixed-link"));
+}
+
/* Ports initialization */
static int mvpp2_port_probe(struct platform_device *pdev,
struct fwnode_handle *port_fwnode,
port = netdev_priv(dev);
port->dev = dev;
port->fwnode = port_fwnode;
- port->has_phy = !!of_find_property(port_node, "phy", NULL);
port->ntxqs = ntxqs;
port->nrxqs = nrxqs;
port->priv = priv;
dev->max_mtu = MVPP2_BM_JUMBO_PKT_SIZE;
dev->dev.of_node = port_node;
- /* Phylink isn't used w/ ACPI as of now */
- if (port_node) {
+ if (!mvpp2_use_acpi_compat_mode(port_fwnode)) {
port->phylink_config.dev = &dev->dev;
port->phylink_config.type = PHYLINK_NETDEV;
}
port->phylink = phylink;
} else {
+ dev_warn(&pdev->dev, "Use link irqs for port#%d. FW update required\n", port->id);
port->phylink = NULL;
}
return PTR_ERR(priv->lms_base);
} else {
res = platform_get_resource(pdev, IORESOURCE_MEM, 1);
+ if (!res) {
+ dev_err(&pdev->dev, "Invalid resource\n");
+ return -EINVAL;
+ }
if (has_acpi_companion(&pdev->dev)) {
/* In case the MDIO memory region is declared in
* the ACPI, it can already appear as 'in-use'
#define SDP_CHANNELS 256
+/* The mask is to extract lower 10-bits of channel number
+ * which CPT will pass to X2P.
+ */
+#define NIX_CHAN_CPT_X2P_MASK (0x3ffull)
+
/* NIX LSO format indices.
* As of now TSO is the only one using, so statically assigning indices.
*/
M(VF_FLR, 0x006, vf_flr, msg_req, msg_rsp) \
M(PTP_OP, 0x007, ptp_op, ptp_req, ptp_rsp) \
M(GET_HW_CAP, 0x008, get_hw_cap, msg_req, get_hw_cap_rsp) \
+M(SET_VF_PERM, 0x00b, set_vf_perm, set_vf_perm, msg_rsp) \
/* CGX mbox IDs (range 0x200 - 0x3FF) */ \
M(CGX_START_RXTX, 0x200, cgx_start_rxtx, msg_req, msg_rsp) \
M(CGX_STOP_RXTX, 0x201, cgx_stop_rxtx, msg_req, msg_rsp) \
M(NIX_GET_MAC_ADDR, 0x8018, nix_get_mac_addr, msg_req, nix_get_mac_addr_rsp) \
M(NIX_CN10K_AQ_ENQ, 0x8019, nix_cn10k_aq_enq, nix_cn10k_aq_enq_req, \
nix_cn10k_aq_enq_rsp) \
-M(NIX_GET_HW_INFO, 0x801a, nix_get_hw_info, msg_req, nix_hw_info)
+M(NIX_GET_HW_INFO, 0x801c, nix_get_hw_info, msg_req, nix_hw_info) \
+M(NIX_BANDPROF_ALLOC, 0x801d, nix_bandprof_alloc, nix_bandprof_alloc_req, \
+ nix_bandprof_alloc_rsp) \
+M(NIX_BANDPROF_FREE, 0x801e, nix_bandprof_free, nix_bandprof_free_req, \
+ msg_rsp)
/* Messages initiated by AF (range 0xC00 - 0xDFF) */
#define MBOX_UP_CGX_MESSAGES \
NIX_AF_INVAL_SSO_PF_FUNC = -420,
NIX_AF_ERR_TX_VTAG_NOSPC = -421,
NIX_AF_ERR_RX_VTAG_INUSE = -422,
- NIX_AF_ERR_NPC_KEY_NOT_SUPP = -423,
+ NIX_AF_ERR_PTP_CONFIG_FAIL = -423,
+ NIX_AF_ERR_NPC_KEY_NOT_SUPP = -424,
+ NIX_AF_ERR_INVALID_NIXBLK = -425,
+ NIX_AF_ERR_INVALID_BANDPROF = -426,
+ NIX_AF_ERR_IPOLICER_NOTSUPP = -427,
+ NIX_AF_ERR_BANDPROF_INVAL_REQ = -428,
};
/* For NIX RX vtag action */
struct nix_cq_ctx_s cq;
struct nix_rsse_s rss;
struct nix_rx_mce_s mce;
+ struct nix_bandprof_s prof;
};
union {
struct nix_cn10k_rq_ctx_s rq_mask;
struct nix_cq_ctx_s cq_mask;
struct nix_rsse_s rss_mask;
struct nix_rx_mce_s mce_mask;
+ struct nix_bandprof_s prof_mask;
};
};
struct nix_cq_ctx_s cq;
struct nix_rsse_s rss;
struct nix_rx_mce_s mce;
+ struct nix_bandprof_s prof;
};
};
struct nix_cq_ctx_s cq;
struct nix_rsse_s rss;
struct nix_rx_mce_s mce;
+ u64 prof;
};
union {
struct nix_rq_ctx_s rq_mask;
struct nix_cq_ctx_s cq_mask;
struct nix_rsse_s rss_mask;
struct nix_rx_mce_s mce_mask;
+ u64 prof_mask;
};
};
struct nix_cq_ctx_s cq;
struct nix_rsse_s rss;
struct nix_rx_mce_s mce;
+ struct nix_bandprof_s prof;
};
};
#define NIX_RX_MODE_UCAST BIT(0)
#define NIX_RX_MODE_PROMISC BIT(1)
#define NIX_RX_MODE_ALLMULTI BIT(2)
+#define NIX_RX_MODE_USE_MCE BIT(3)
u16 mode;
};
u16 min_mtu;
};
+struct nix_bandprof_alloc_req {
+ struct mbox_msghdr hdr;
+ /* Count of profiles needed per layer */
+ u16 prof_count[BAND_PROF_NUM_LAYERS];
+};
+
+struct nix_bandprof_alloc_rsp {
+ struct mbox_msghdr hdr;
+ u16 prof_count[BAND_PROF_NUM_LAYERS];
+
+ /* There is no need to allocate morethan 1 bandwidth profile
+ * per RQ of a PF_FUNC's NIXLF. So limit the maximum
+ * profiles to 64 per PF_FUNC.
+ */
+#define MAX_BANDPROF_PER_PFFUNC 64
+ u16 prof_idx[BAND_PROF_NUM_LAYERS][MAX_BANDPROF_PER_PFFUNC];
+};
+
+struct nix_bandprof_free_req {
+ struct mbox_msghdr hdr;
+ u8 free_all;
+ u16 prof_count[BAND_PROF_NUM_LAYERS];
+ u16 prof_idx[BAND_PROF_NUM_LAYERS][MAX_BANDPROF_PER_PFFUNC];
+};
+
/* NPC mbox message structs */
#define NPC_MCAM_ENTRY_INVALID 0xFFFF
u64 clk;
};
+struct set_vf_perm {
+ struct mbox_msghdr hdr;
+ u16 vf;
+#define RESET_VF_PERM BIT_ULL(0)
+#define VF_TRUSTED BIT_ULL(1)
+ u64 flags;
+};
+
/* CPT mailbox error codes
* Range 901 - 1000.
*/
NPC_LT_LA_IH_2_ETHER,
NPC_LT_LA_HIGIG2_ETHER,
NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ NPC_LT_LA_CH_LEN_90B_ETHER,
+ NPC_LT_LA_CPT_HDR,
+ NPC_LT_LA_CUSTOM_L2_24B_ETHER,
NPC_LT_LA_CUSTOM0 = 0xE,
NPC_LT_LA_CUSTOM1 = 0xF,
};
NPC_LT_LB_CTAG,
NPC_LT_LB_STAG_QINQ,
NPC_LT_LB_BTAG,
- NPC_LT_LB_ITAG,
+ NPC_LT_LB_PPPOE,
NPC_LT_LB_DSA,
NPC_LT_LB_DSA_VLAN,
NPC_LT_LB_EDSA,
NPC_LT_LB_EXDSA,
NPC_LT_LB_EXDSA_VLAN,
NPC_LT_LB_FDSA,
+ NPC_LT_LB_VLAN_EXDSA,
NPC_LT_LB_CUSTOM0 = 0xE,
NPC_LT_LB_CUSTOM1 = 0xF,
};
NPC_LT_LC_NSH,
NPC_LT_LC_PTP,
NPC_LT_LC_FCOE,
+ NPC_LT_LC_NGIO,
NPC_LT_LC_CUSTOM0 = 0xE,
NPC_LT_LC_CUSTOM1 = 0xF,
};
* Ethernet interfaces, LBK interfaces, etc.
*/
enum npc_pkind_type {
- NPC_TX_DEF_PKIND = 63ULL, /* NIX-TX PKIND */
+ NPC_RX_VLAN_EXDSA_PKIND = 56ULL,
+ NPC_RX_CHLEN24B_PKIND = 57ULL,
+ NPC_RX_CPT_HDR_PKIND,
+ NPC_RX_CHLEN90B_PKIND,
+ NPC_TX_HIGIG_PKIND,
+ NPC_RX_HIGIG_PKIND,
+ NPC_RX_EDSA_PKIND,
+ NPC_TX_DEF_PKIND, /* NIX-TX PKIND */
};
/* list of known and supported fields in packet header and
u16 dp1_mask;
u16 dp2;
u16 dp2_mask;
-};
+} __packed;
struct npc_kpu_profile_action {
u8 errlev;
u8 mask;
u8 right;
u8 shift;
-};
+} __packed;
struct npc_kpu_profile {
int cam_entries;
int action_entries;
- const struct npc_kpu_profile_cam *cam;
- const struct npc_kpu_profile_action *action;
+ struct npc_kpu_profile_cam *cam;
+ struct npc_kpu_profile_action *action;
};
/* NPC KPU register formats */
/* NPC MCAM reserved entry index per nixlf */
#define NIXLF_UCAST_ENTRY 0
#define NIXLF_BCAST_ENTRY 1
-#define NIXLF_PROMISC_ENTRY 2
+#define NIXLF_ALLMULTI_ENTRY 2
+#define NIXLF_PROMISC_ENTRY 3
+
+struct npc_coalesced_kpu_prfl {
+#define NPC_SIGN 0x00666f727063706e
+#define NPC_PRFL_NAME "npc_prfls_array"
+#define NPC_NAME_LEN 32
+ __le64 signature; /* "npcprof\0" (8 bytes/ASCII characters) */
+ u8 name[NPC_NAME_LEN]; /* KPU Profile name */
+ u64 version; /* KPU firmware/profile version */
+ u8 num_prfl; /* No of NPC profiles. */
+ u16 prfl_sz[0];
+};
struct npc_mcam_kex {
/* MKEX Profle Header */
u64 intf_ld_flags[NPC_MAX_INTF][NPC_MAX_LD][NPC_MAX_LFL];
} __packed;
+struct npc_kpu_fwdata {
+ int entries;
+ /* What follows is:
+ * struct npc_kpu_profile_cam[entries];
+ * struct npc_kpu_profile_action[entries];
+ */
+ u8 data[0];
+} __packed;
+
struct npc_lt_def {
u8 ltype_mask;
u8 ltype_match;
u8 spi_nz;
};
+struct npc_lt_def_apad {
+ u8 ltype_mask;
+ u8 ltype_match;
+ u8 lid;
+ u8 valid;
+} __packed;
+
+struct npc_lt_def_color {
+ u8 ltype_mask;
+ u8 ltype_match;
+ u8 lid;
+ u8 noffset;
+ u8 offset;
+} __packed;
+
+struct npc_lt_def_et {
+ u8 ltype_mask;
+ u8 ltype_match;
+ u8 lid;
+ u8 valid;
+ u8 offset;
+} __packed;
+
struct npc_lt_def_cfg {
struct npc_lt_def rx_ol2;
struct npc_lt_def rx_oip4;
struct npc_lt_def pck_oip4;
struct npc_lt_def pck_oip6;
struct npc_lt_def pck_iip4;
-};
+ struct npc_lt_def_apad rx_apad0;
+ struct npc_lt_def_apad rx_apad1;
+ struct npc_lt_def_color ovlan;
+ struct npc_lt_def_color ivlan;
+ struct npc_lt_def_color rx_gen0_color;
+ struct npc_lt_def_color rx_gen1_color;
+ struct npc_lt_def_et rx_et[2];
+} __packed;
+
+/* Loadable KPU profile firmware data */
+struct npc_kpu_profile_fwdata {
+#define KPU_SIGN 0x00666f727075706b
+#define KPU_NAME_LEN 32
+/** Maximum number of custom KPU entries supported by the built-in profile. */
+#define KPU_MAX_CST_ENT 2
+ /* KPU Profle Header */
+ __le64 signature; /* "kpuprof\0" (8 bytes/ASCII characters) */
+ u8 name[KPU_NAME_LEN]; /* KPU Profile name */
+ __le64 version; /* KPU profile version */
+ u8 kpus;
+ u8 reserved[7];
+
+ /* Default MKEX profile to be used with this KPU profile. May be
+ * overridden with mkex_profile module parameter. Format is same as for
+ * the MKEX profile to streamline processing.
+ */
+ struct npc_mcam_kex mkex;
+ /* LTYPE values for specific HW offloaded protocols. */
+ struct npc_lt_def_cfg lt_def;
+ /* Dynamically sized data:
+ * Custom KPU CAM and ACTION configuration entries.
+ * struct npc_kpu_fwdata kpu[kpus];
+ */
+ u8 data[0];
+} __packed;
struct rvu_npc_mcam_rule {
struct flow_msg packet;
#ifndef NPC_PROFILE_H
#define NPC_PROFILE_H
-#define NPC_KPU_PROFILE_VER 0x0000000100050000
+#define NPC_KPU_PROFILE_VER 0x0000000100060000
+#define NPC_KPU_VER_MAJ(ver) ((u16)(((ver) >> 32) & 0xFFFF))
+#define NPC_KPU_VER_MIN(ver) ((u16)(((ver) >> 16) & 0xFFFF))
+#define NPC_KPU_VER_PATCH(ver) ((u16)((ver) & 0xFFFF))
#define NPC_IH_W 0x8000
#define NPC_IH_UTAG 0x2000
#define NPC_ETYPE_IP6 0x86dd
#define NPC_ETYPE_ARP 0x0806
#define NPC_ETYPE_RARP 0x8035
+#define NPC_ETYPE_NGIO 0x8842
#define NPC_ETYPE_MPLSU 0x8847
#define NPC_ETYPE_MPLSM 0x8848
#define NPC_ETYPE_ETAG 0x893f
#define NPC_ETYPE_PPP 0x880b
#define NPC_ETYPE_NSH 0x894f
#define NPC_ETYPE_DSA 0xdada
+#define NPC_ETYPE_PPPOE 0x8864
+
+#define NPC_PPP_IP 0x0021
+#define NPC_PPP_IP6 0x0057
#define NPC_IPNH_HOP 0
#define NPC_IPNH_ICMP 1
#define NPC_DSA_EDSA 0x8000
#define NPC_DSA_FDSA 0xc000
-#define NPC_KEXOF_DMAC 8
-#define MKEX_SIGN 0x19bbfdbd15f /* strtoull of "mkexprof" with base:36 */
+#define NPC_KEXOF_DMAC 9
+#define MKEX_SIGN 0x19bbfdbd15f
#define KEX_LD_CFG(bytesm1, hdr_ofs, ena, flags_ena, key_ofs) \
(((bytesm1) << 16) | ((hdr_ofs) << 8) | ((ena) << 7) | \
((flags_ena) << 6) | ((key_ofs) & 0x3F))
/* Rx parse key extract nibble enable */
#define NPC_PARSE_NIBBLE_INTF_RX (NPC_PARSE_NIBBLE_CHAN | \
+ NPC_PARSE_NIBBLE_ERRCODE | \
NPC_PARSE_NIBBLE_LA_LTYPE | \
NPC_PARSE_NIBBLE_LB_LTYPE | \
NPC_PARSE_NIBBLE_LC_LTYPE | \
NPC_S_KPU1_EXDSA,
NPC_S_KPU1_HIGIG2,
NPC_S_KPU1_IH_NIX_HIGIG2,
+ NPC_S_KPU1_CUSTOM_L2_90B,
+ NPC_S_KPU1_CPT_HDR,
+ NPC_S_KPU1_CUSTOM_L2_24B,
+ NPC_S_KPU1_VLAN_EXDSA,
NPC_S_KPU2_CTAG,
NPC_S_KPU2_CTAG2,
NPC_S_KPU2_SBTAG,
NPC_S_KPU2_QINQ,
NPC_S_KPU2_ETAG,
- NPC_S_KPU2_ITAG,
NPC_S_KPU2_PREHEADER,
NPC_S_KPU2_EXDSA,
+ NPC_S_KPU2_NGIO,
NPC_S_KPU3_CTAG,
NPC_S_KPU3_STAG,
NPC_S_KPU3_QINQ,
- NPC_S_KPU3_ITAG,
NPC_S_KPU3_CTAG_C,
NPC_S_KPU3_STAG_C,
NPC_S_KPU3_QINQ_C,
NPC_S_KPU3_DSA,
+ NPC_S_KPU3_VLAN_EXDSA,
NPC_S_KPU4_MPLS,
NPC_S_KPU4_NSH,
NPC_S_KPU4_FDSA,
+ NPC_S_KPU4_VLAN_EXDSA,
+ NPC_S_KPU4_PPPOE,
NPC_S_KPU5_IP,
NPC_S_KPU5_IP6,
NPC_S_KPU5_ARP,
NPC_S_KPU5_MPLS,
NPC_S_KPU5_MPLS_PL,
NPC_S_KPU5_NSH,
+ NPC_S_KPU5_CPT_IP,
+ NPC_S_KPU5_CPT_IP6,
NPC_S_KPU6_IP6_EXT,
NPC_S_KPU6_IP6_HOP_DEST,
NPC_S_KPU6_IP6_ROUT,
NPC_S_KPU6_IP6_FRAG,
+ NPC_S_KPU6_IP6_CPT_FRAG,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST,
+ NPC_S_KPU6_IP6_CPT_ROUT,
NPC_S_KPU7_IP6_EXT,
NPC_S_KPU7_IP6_ROUT,
NPC_S_KPU7_IP6_FRAG,
+ NPC_S_KPU7_CPT_IP6_FRAG,
NPC_S_KPU8_TCP,
NPC_S_KPU8_UDP,
NPC_S_KPU8_SCTP,
NPC_F_LA_L_UNK_ETYPE = 1,
NPC_F_LA_L_WITH_VLAN,
NPC_F_LA_L_WITH_ETAG,
- NPC_F_LA_L_WITH_ITAG,
NPC_F_LA_L_WITH_MPLS,
NPC_F_LA_L_WITH_NSH,
};
NPC_ERRLEV_ENUM_LAST = 16,
};
-static const struct npc_kpu_profile_action ikpu_action_entries[] = {
+#define NPC_KPU_NOP_CAM \
+ { \
+ NPC_S_NA, 0xff, \
+ 0x0000, \
+ 0x0000, \
+ 0x0000, \
+ 0x0000, \
+ 0x0000, \
+ 0x0000, \
+ }
+
+#define NPC_KPU_NOP_ACTION \
+ { \
+ NPC_ERRLEV_RE, NPC_EC_NOERR, \
+ 0, 0, 0, 0, 0, \
+ NPC_S_NA, 0, 0, \
+ NPC_LID_LA, NPC_LT_NA, \
+ 0, \
+ 0, 0, 0, 0, \
+ }
+
+static struct npc_kpu_profile_action ikpu_action_entries[] = {
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
12, 16, 20, 0, 0,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
12, 16, 20, 0, 0,
- NPC_S_KPU1_ETHER, 0, 0,
+ NPC_S_KPU1_VLAN_EXDSA, 0, 0,
NPC_LID_LA, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 12, 16, 20, 0, 0,
- NPC_S_KPU1_ETHER, 0, 0,
+ 36, 40, 44, 0, 0,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0, 0,
NPC_LID_LA, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 12, 16, 20, 0, 0,
- NPC_S_KPU1_ETHER, 0, 0,
+ 40, 54, 58, 0, 0,
+ NPC_S_KPU1_CPT_HDR, 0, 0,
NPC_LID_LA, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 12, 16, 20, 0, 0,
- NPC_S_KPU1_ETHER, 0, 0,
+ 102, 106, 110, 0, 0,
+ NPC_S_KPU1_CUSTOM_L2_90B, 0, 0,
NPC_LID_LA, NPC_LT_NA,
0,
0, 0, 0, 0,
},
};
-static const struct npc_kpu_profile_cam kpu1_cam_entries[] = {
+static struct npc_kpu_profile_cam kpu1_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
{
NPC_S_KPU1_ETHER, 0xff,
NPC_ETYPE_IP,
NPC_S_KPU1_ETHER, 0xff,
NPC_ETYPE_CTAG,
0xffff,
+ NPC_ETYPE_NGIO,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU1_ETHER, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
NPC_ETYPE_CTAG,
0xffff,
0x0000,
},
{
NPC_S_KPU1_ETHER, 0xff,
- NPC_ETYPE_ITAG,
+ NPC_ETYPE_MPLSU,
0xffff,
0x0000,
0x0000,
},
{
NPC_S_KPU1_ETHER, 0xff,
- NPC_ETYPE_MPLSU,
+ NPC_ETYPE_MPLSM,
0xffff,
0x0000,
0x0000,
},
{
NPC_S_KPU1_ETHER, 0xff,
- NPC_ETYPE_MPLSM,
+ NPC_ETYPE_NSH,
0xffff,
0x0000,
0x0000,
},
{
NPC_S_KPU1_ETHER, 0xff,
- NPC_ETYPE_NSH,
+ NPC_ETYPE_DSA,
0xffff,
0x0000,
0x0000,
},
{
NPC_S_KPU1_ETHER, 0xff,
- NPC_ETYPE_DSA,
+ NPC_ETYPE_PPPOE,
0xffff,
0x0000,
0x0000,
},
{
NPC_S_KPU1_IH_NIX, 0xff,
- NPC_ETYPE_ITAG,
- 0xffff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU1_IH_NIX, 0xff,
NPC_ETYPE_MPLSU,
0xffff,
0x0000,
},
{
NPC_S_KPU1_IH, 0xff,
- NPC_IH_W|NPC_IH_UTAG,
- NPC_IH_W|NPC_IH_UTAG,
+ NPC_IH_W | NPC_IH_UTAG,
+ NPC_IH_W | NPC_IH_UTAG,
0x0000,
0x0000,
0x0000,
{
NPC_S_KPU1_IH, 0xff,
NPC_IH_W,
- NPC_IH_W|NPC_IH_UTAG,
+ NPC_IH_W | NPC_IH_UTAG,
0x0000,
0x0000,
0x0000,
{
NPC_S_KPU1_IH, 0xff,
0x0000,
- NPC_IH_W|NPC_IH_UTAG,
+ NPC_IH_W | NPC_IH_UTAG,
0x0000,
0x0000,
0x0000,
},
{
NPC_S_KPU1_HIGIG2, 0xff,
- NPC_ETYPE_ITAG,
- 0xffff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU1_HIGIG2, 0xff,
NPC_ETYPE_MPLSU,
0xffff,
0x0000,
},
{
NPC_S_KPU1_IH_NIX_HIGIG2, 0xff,
- NPC_ETYPE_ITAG,
+ NPC_ETYPE_MPLSU,
0xffff,
0x0000,
0x0000,
},
{
NPC_S_KPU1_IH_NIX_HIGIG2, 0xff,
- NPC_ETYPE_MPLSU,
+ NPC_ETYPE_MPLSM,
0xffff,
0x0000,
0x0000,
},
{
NPC_S_KPU1_IH_NIX_HIGIG2, 0xff,
- NPC_ETYPE_MPLSM,
+ NPC_ETYPE_NSH,
0xffff,
0x0000,
0x0000,
},
{
NPC_S_KPU1_IH_NIX_HIGIG2, 0xff,
- NPC_ETYPE_NSH,
- 0xffff,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU1_IH_NIX_HIGIG2, 0xff,
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ 0x0000,
0x0000,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ 0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_NA, 0X00,
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
+ NPC_ETYPE_ARP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
+ NPC_ETYPE_RARP,
+ 0xffff,
0x0000,
0x0000,
0x0000,
0x0000,
},
-};
-
-static const struct npc_kpu_profile_cam kpu2_cam_entries[] = {
{
- NPC_S_KPU2_CTAG, 0xff,
- NPC_ETYPE_IP,
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
+ NPC_ETYPE_PTP,
0xffff,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_CTAG, 0xff,
- NPC_ETYPE_IP6,
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
+ NPC_ETYPE_FCOE,
0xffff,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_CTAG, 0xff,
- NPC_ETYPE_ARP,
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
+ NPC_ETYPE_CTAG,
0xffff,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_CTAG, 0xff,
- NPC_ETYPE_RARP,
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
+ NPC_ETYPE_SBTAG,
0xffff,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_CTAG, 0xff,
- NPC_ETYPE_PTP,
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
+ NPC_ETYPE_QINQ,
0xffff,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_CTAG, 0xff,
- NPC_ETYPE_FCOE,
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
+ NPC_ETYPE_ETAG,
0xffff,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_CTAG, 0xff,
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
NPC_ETYPE_MPLSU,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_CTAG, 0xff,
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
NPC_ETYPE_MPLSM,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_CTAG, 0xff,
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
NPC_ETYPE_NSH,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_CTAG, 0xff,
+ NPC_S_KPU1_CUSTOM_L2_90B, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU1_CPT_HDR, 0xff,
+ 0x0000,
0xffff,
NPC_ETYPE_IP,
0xffff,
0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU1_CPT_HDR, 0xff,
+ 0x0000,
0xffff,
NPC_ETYPE_IP6,
0xffff,
0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU1_CPT_HDR, 0xff,
+ 0x0000,
0xffff,
- NPC_ETYPE_ARP,
+ NPC_ETYPE_CTAG,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU1_CPT_HDR, 0xff,
+ 0x0000,
0xffff,
- NPC_ETYPE_RARP,
+ NPC_ETYPE_QINQ,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
- NPC_ETYPE_PTP,
+ NPC_S_KPU1_CPT_HDR, 0xff,
+ 0x0000,
0xffff,
0x0000,
0x0000,
+ NPC_ETYPE_IP,
+ 0xffff,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
- NPC_ETYPE_FCOE,
+ NPC_S_KPU1_CPT_HDR, 0xff,
+ 0x0000,
0xffff,
0x0000,
0x0000,
+ NPC_ETYPE_IP6,
+ 0xffff,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
- NPC_ETYPE_MPLSU,
+ NPC_S_KPU1_CPT_HDR, 0xff,
+ 0x0000,
0xffff,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU2_SBTAG, 0xff,
NPC_ETYPE_CTAG,
0xffff,
- NPC_ETYPE_MPLSM,
- 0xffff,
- 0x0000,
- 0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
- NPC_ETYPE_NSH,
+ NPC_S_KPU1_CPT_HDR, 0xff,
+ 0x0000,
0xffff,
0x0000,
0x0000,
+ NPC_ETYPE_QINQ,
+ 0xffff,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
+ NPC_S_KPU1_CPT_HDR, 0xff,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
+ NPC_ETYPE_IP,
0xffff,
0x0000,
0x0000,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
- NPC_ETYPE_SBTAG,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
+ NPC_ETYPE_IP6,
0xffff,
0x0000,
0x0000,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_ITAG,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
+ NPC_ETYPE_ARP,
0xffff,
0x0000,
0x0000,
- NPC_ETYPE_IP,
- 0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_ITAG,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
+ NPC_ETYPE_RARP,
0xffff,
0x0000,
0x0000,
- NPC_ETYPE_IP6,
- 0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_ITAG,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
+ NPC_ETYPE_PTP,
0xffff,
0x0000,
0x0000,
- NPC_ETYPE_ARP,
- 0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_ITAG,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
+ NPC_ETYPE_FCOE,
0xffff,
0x0000,
0x0000,
- NPC_ETYPE_RARP,
- 0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_ITAG,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_CTAG,
0xffff,
0x0000,
0x0000,
- NPC_ETYPE_PTP,
- 0xffff,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_ITAG,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
+ NPC_ETYPE_CTAG,
0xffff,
0x0000,
0x0000,
- NPC_ETYPE_FCOE,
- 0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_ITAG,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
+ NPC_ETYPE_SBTAG,
0xffff,
0x0000,
0x0000,
- NPC_ETYPE_MPLSU,
- 0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_ITAG,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
+ NPC_ETYPE_QINQ,
0xffff,
0x0000,
0x0000,
- NPC_ETYPE_MPLSM,
- 0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_ITAG,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
+ NPC_ETYPE_ETAG,
0xffff,
0x0000,
0x0000,
- NPC_ETYPE_NSH,
- 0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_ITAG,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
+ NPC_ETYPE_MPLSU,
0xffff,
0x0000,
0x0000,
- NPC_ETYPE_SBTAG,
- 0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_ITAG,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
+ NPC_ETYPE_MPLSM,
0xffff,
0x0000,
0x0000,
- NPC_ETYPE_CTAG,
- 0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
- NPC_ETYPE_ITAG,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
+ NPC_ETYPE_NSH,
0xffff,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_SBTAG, 0xff,
+ NPC_S_KPU1_CUSTOM_L2_24B, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_QINQ, 0xff,
+ NPC_S_KPU1_VLAN_EXDSA, 0xff,
NPC_ETYPE_CTAG,
0xffff,
- NPC_ETYPE_IP,
- 0xffff,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_QINQ, 0xff,
- NPC_ETYPE_CTAG,
+ NPC_S_NA, 0X00,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+};
+
+static struct npc_kpu_profile_cam kpu2_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
+ {
+ NPC_S_KPU2_CTAG, 0xff,
+ NPC_ETYPE_IP,
0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_CTAG, 0xff,
NPC_ETYPE_IP6,
0xffff,
0x0000,
0x0000,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_QINQ, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
+ NPC_S_KPU2_CTAG, 0xff,
NPC_ETYPE_ARP,
0xffff,
0x0000,
0x0000,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_QINQ, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
+ NPC_S_KPU2_CTAG, 0xff,
NPC_ETYPE_RARP,
0xffff,
0x0000,
0x0000,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_QINQ, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
+ NPC_S_KPU2_CTAG, 0xff,
NPC_ETYPE_PTP,
0xffff,
0x0000,
0x0000,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_QINQ, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
+ NPC_S_KPU2_CTAG, 0xff,
NPC_ETYPE_FCOE,
0xffff,
0x0000,
0x0000,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_QINQ, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
+ NPC_S_KPU2_CTAG, 0xff,
NPC_ETYPE_MPLSU,
0xffff,
0x0000,
0x0000,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_QINQ, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
+ NPC_S_KPU2_CTAG, 0xff,
NPC_ETYPE_MPLSM,
0xffff,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU2_QINQ, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
- NPC_ETYPE_NSH,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_QINQ, 0xff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU2_CTAG, 0xff,
+ NPC_ETYPE_NSH,
0xffff,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_QINQ, 0xff,
- NPC_ETYPE_QINQ,
- 0xffff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU2_CTAG, 0xff,
+ NPC_ETYPE_PPPOE,
0xffff,
0x0000,
0x0000,
+ NPC_PPP_IP,
+ 0xffff,
},
{
- NPC_S_KPU2_QINQ, 0xff,
- NPC_ETYPE_QINQ,
- 0xffff,
- NPC_ETYPE_QINQ,
+ NPC_S_KPU2_CTAG, 0xff,
+ NPC_ETYPE_PPPOE,
0xffff,
0x0000,
0x0000,
+ NPC_PPP_IP6,
+ 0xffff,
},
{
- NPC_S_KPU2_QINQ, 0xff,
+ NPC_S_KPU2_CTAG, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
NPC_ETYPE_IP,
0xffff,
0x0000,
0x0000,
- 0x0000,
- 0x0000,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
NPC_ETYPE_IP6,
0xffff,
0x0000,
0x0000,
- 0x0000,
- 0x0000,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
NPC_ETYPE_ARP,
0xffff,
0x0000,
0x0000,
- 0x0000,
- 0x0000,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
NPC_ETYPE_RARP,
0xffff,
0x0000,
0x0000,
- 0x0000,
- 0x0000,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
NPC_ETYPE_PTP,
0xffff,
0x0000,
0x0000,
- 0x0000,
- 0x0000,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
NPC_ETYPE_FCOE,
0xffff,
0x0000,
0x0000,
- 0x0000,
- 0x0000,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
NPC_ETYPE_MPLSU,
0xffff,
0x0000,
0x0000,
- 0x0000,
- 0x0000,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
NPC_ETYPE_MPLSM,
0xffff,
0x0000,
0x0000,
- 0x0000,
- 0x0000,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
NPC_ETYPE_NSH,
0xffff,
0x0000,
0x0000,
- 0x0000,
- 0x0000,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
NPC_ETYPE_CTAG,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
NPC_ETYPE_SBTAG,
0xffff,
- NPC_ETYPE_ITAG,
+ NPC_ETYPE_CTAG,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
NPC_ETYPE_SBTAG,
0xffff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU2_ETAG, 0xff,
- NPC_ETYPE_QINQ,
+ NPC_ETYPE_SBTAG,
0xffff,
0x0000,
0x0000,
- 0x0000,
- 0x0000,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
NPC_ETYPE_ITAG,
0xffff,
0x0000,
0xffff,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
NPC_ETYPE_ITAG,
0xffff,
0x0000,
0xffff,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
NPC_ETYPE_ITAG,
0xffff,
0x0000,
0xffff,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
NPC_ETYPE_ITAG,
0xffff,
0x0000,
0x0000,
- NPC_ETYPE_SBTAG,
+ NPC_ETYPE_RARP,
0xffff,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
NPC_ETYPE_ITAG,
0xffff,
0x0000,
0x0000,
- NPC_ETYPE_CTAG,
+ NPC_ETYPE_PTP,
0xffff,
},
{
- NPC_S_KPU2_ETAG, 0xff,
+ NPC_S_KPU2_SBTAG, 0xff,
NPC_ETYPE_ITAG,
0xffff,
0x0000,
0x0000,
- 0x0000,
- 0x0000,
+ NPC_ETYPE_FCOE,
+ 0xffff,
},
{
- NPC_S_KPU2_ETAG, 0xff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_ITAG,
+ 0xffff,
0x0000,
0x0000,
+ NPC_ETYPE_MPLSU,
+ 0xffff,
},
{
- NPC_S_KPU2_ITAG, 0xff,
- NPC_ETYPE_IP,
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_ITAG,
0xffff,
0x0000,
0x0000,
+ NPC_ETYPE_MPLSM,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_ITAG,
+ 0xffff,
0x0000,
0x0000,
+ NPC_ETYPE_NSH,
+ 0xffff,
},
{
- NPC_S_KPU2_ITAG, 0xff,
- NPC_ETYPE_IP6,
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_ITAG,
0xffff,
0x0000,
0x0000,
+ NPC_ETYPE_SBTAG,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_ITAG,
+ 0xffff,
0x0000,
0x0000,
+ NPC_ETYPE_CTAG,
+ 0xffff,
},
{
- NPC_S_KPU2_ITAG, 0xff,
- NPC_ETYPE_ARP,
+ NPC_S_KPU2_SBTAG, 0xff,
+ NPC_ETYPE_ITAG,
0xffff,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_ITAG, 0xff,
- NPC_ETYPE_RARP,
- 0xffff,
+ NPC_S_KPU2_SBTAG, 0xff,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
+ NPC_S_KPU2_QINQ, 0xff,
NPC_ETYPE_CTAG,
0xffff,
NPC_ETYPE_IP,
0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
+ NPC_S_KPU2_QINQ, 0xff,
NPC_ETYPE_CTAG,
0xffff,
NPC_ETYPE_IP6,
0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
+ NPC_S_KPU2_QINQ, 0xff,
NPC_ETYPE_CTAG,
0xffff,
NPC_ETYPE_ARP,
0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
+ NPC_S_KPU2_QINQ, 0xff,
NPC_ETYPE_CTAG,
0xffff,
+ NPC_ETYPE_RARP,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
+ NPC_S_KPU2_QINQ, 0xff,
+ NPC_ETYPE_CTAG,
0xffff,
- NPC_ETYPE_IP,
+ NPC_ETYPE_PTP,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
+ NPC_S_KPU2_QINQ, 0xff,
+ NPC_ETYPE_CTAG,
0xffff,
- NPC_ETYPE_IP6,
+ NPC_ETYPE_FCOE,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
+ NPC_S_KPU2_QINQ, 0xff,
+ NPC_ETYPE_CTAG,
0xffff,
- NPC_ETYPE_ARP,
+ NPC_ETYPE_MPLSU,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
+ NPC_S_KPU2_QINQ, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_MPLSM,
0xffff,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU2_QINQ, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_NSH,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_ITAG, 0xff,
+ NPC_S_KPU2_QINQ, 0xff,
NPC_ETYPE_CTAG,
0xffff,
- NPC_ETYPE_IP,
- 0xffff,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_ITAG, 0xff,
+ NPC_S_KPU2_QINQ, 0xff,
+ NPC_ETYPE_QINQ,
+ 0xffff,
NPC_ETYPE_CTAG,
0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_QINQ, 0xff,
+ NPC_ETYPE_QINQ,
+ 0xffff,
+ NPC_ETYPE_QINQ,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_QINQ, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
NPC_ETYPE_IP6,
0xffff,
0x0000,
0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_ARP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_RARP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_PTP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_FCOE,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_MPLSU,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_MPLSM,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_NSH,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU2_ITAG, 0xff,
+ NPC_S_KPU2_ETAG, 0xff,
NPC_ETYPE_CTAG,
0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_SBTAG,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_QINQ,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_ITAG,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ NPC_ETYPE_IP,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_ITAG,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_ITAG,
+ 0xffff,
+ 0x0000,
+ 0x0000,
NPC_ETYPE_ARP,
0xffff,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_ITAG,
+ 0xffff,
0x0000,
0x0000,
+ NPC_ETYPE_SBTAG,
+ 0xffff,
},
{
- NPC_S_KPU2_ITAG, 0xff,
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_ITAG,
+ 0xffff,
+ 0x0000,
+ 0x0000,
NPC_ETYPE_CTAG,
0xffff,
+ },
+ {
+ NPC_S_KPU2_ETAG, 0xff,
+ NPC_ETYPE_ITAG,
+ 0xffff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU2_ITAG, 0xff,
+ NPC_S_KPU2_ETAG, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
+ NPC_S_KPU2_NGIO, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
NPC_S_NA, 0X00,
0x0000,
0x0000,
},
};
-static const struct npc_kpu_profile_cam kpu3_cam_entries[] = {
+static struct npc_kpu_profile_cam kpu3_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
{
NPC_S_KPU3_CTAG, 0xff,
NPC_ETYPE_IP,
0x0000,
},
{
- NPC_S_KPU3_ITAG, 0xff,
+ NPC_S_KPU3_CTAG_C, 0xff,
NPC_ETYPE_IP,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU3_ITAG, 0xff,
+ NPC_S_KPU3_CTAG_C, 0xff,
NPC_ETYPE_IP6,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU3_ITAG, 0xff,
+ NPC_S_KPU3_CTAG_C, 0xff,
NPC_ETYPE_ARP,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU3_ITAG, 0xff,
+ NPC_S_KPU3_CTAG_C, 0xff,
NPC_ETYPE_RARP,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU3_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
- NPC_ETYPE_CTAG,
- 0xffff,
- NPC_ETYPE_IP,
- 0xffff,
- },
- {
- NPC_S_KPU3_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
- NPC_ETYPE_CTAG,
- 0xffff,
- NPC_ETYPE_IP6,
- 0xffff,
- },
- {
- NPC_S_KPU3_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
- NPC_ETYPE_CTAG,
- 0xffff,
- NPC_ETYPE_ARP,
+ NPC_S_KPU3_CTAG_C, 0xff,
+ NPC_ETYPE_PTP,
0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU3_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
- NPC_ETYPE_IP,
+ NPC_S_KPU3_CTAG_C, 0xff,
+ NPC_ETYPE_FCOE,
0xffff,
0x0000,
0x0000,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU3_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
- NPC_ETYPE_IP6,
+ NPC_S_KPU3_CTAG_C, 0xff,
+ NPC_ETYPE_MPLSU,
0xffff,
0x0000,
0x0000,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU3_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
- NPC_ETYPE_ARP,
+ NPC_S_KPU3_CTAG_C, 0xff,
+ NPC_ETYPE_MPLSM,
0xffff,
0x0000,
0x0000,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU3_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU3_CTAG_C, 0xff,
+ NPC_ETYPE_NSH,
0xffff,
0x0000,
0x0000,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU3_ITAG, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
+ NPC_S_KPU3_CTAG_C, 0xff,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU3_ITAG, 0xff,
+ NPC_S_KPU3_STAG_C, 0xff,
NPC_ETYPE_CTAG,
0xffff,
NPC_ETYPE_IP,
0x0000,
},
{
- NPC_S_KPU3_ITAG, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
- NPC_ETYPE_IP6,
- 0xffff,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_ITAG, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
- NPC_ETYPE_ARP,
- 0xffff,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_ITAG, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_ITAG, 0xff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_CTAG_C, 0xff,
- NPC_ETYPE_IP,
- 0xffff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_CTAG_C, 0xff,
- NPC_ETYPE_IP6,
- 0xffff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_CTAG_C, 0xff,
- NPC_ETYPE_ARP,
- 0xffff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_CTAG_C, 0xff,
- NPC_ETYPE_RARP,
- 0xffff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_CTAG_C, 0xff,
- NPC_ETYPE_PTP,
- 0xffff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_CTAG_C, 0xff,
- NPC_ETYPE_FCOE,
- 0xffff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_CTAG_C, 0xff,
- NPC_ETYPE_MPLSU,
- 0xffff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_CTAG_C, 0xff,
- NPC_ETYPE_MPLSM,
- 0xffff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_CTAG_C, 0xff,
- NPC_ETYPE_NSH,
- 0xffff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_CTAG_C, 0xff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_STAG_C, 0xff,
- NPC_ETYPE_CTAG,
- 0xffff,
- NPC_ETYPE_IP,
- 0xffff,
- 0x0000,
- 0x0000,
- },
- {
- NPC_S_KPU3_STAG_C, 0xff,
+ NPC_S_KPU3_STAG_C, 0xff,
NPC_ETYPE_CTAG,
0xffff,
NPC_ETYPE_IP6,
0x0000,
},
{
+ NPC_S_KPU3_VLAN_EXDSA, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
NPC_S_NA, 0X00,
0x0000,
0x0000,
},
};
-static const struct npc_kpu_profile_cam kpu4_cam_entries[] = {
+static struct npc_kpu_profile_cam kpu4_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
{
NPC_S_KPU4_MPLS, 0xff,
NPC_MPLS_S,
},
{
NPC_S_KPU4_FDSA, 0xff,
- 0x0000,
- NPC_DSA_FDSA,
- 0x0000,
- 0x0000,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_IP,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_NA, 0X00,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
+ NPC_S_KPU4_FDSA, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_IP6,
+ 0xffff,
0x0000,
0x0000,
},
-};
-
-static const struct npc_kpu_profile_cam kpu5_cam_entries[] = {
{
- NPC_S_KPU5_IP, 0xff,
- 0x0000,
- NPC_IP_TTL_MASK,
- 0x0000,
- 0x0000,
+ NPC_S_KPU4_FDSA, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_ARP,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU5_IP, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU4_FDSA, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_RARP,
+ 0xffff,
0x0000,
0x0000,
- 0x0001,
- NPC_IP_HDR_FRAGOFF,
},
{
- NPC_S_KPU5_IP, 0xff,
- NPC_IPNH_TCP,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ NPC_S_KPU4_FDSA, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_PTP,
+ 0xffff,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
- },
- {
- NPC_S_KPU5_IP, 0xff,
- NPC_IPNH_UDP,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
},
{
- NPC_S_KPU5_IP, 0xff,
- NPC_IPNH_SCTP,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ NPC_S_KPU4_FDSA, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_FCOE,
+ 0xffff,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
- },
- {
- NPC_S_KPU5_IP, 0xff,
- NPC_IPNH_ICMP,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
},
{
- NPC_S_KPU5_IP, 0xff,
- NPC_IPNH_IGMP,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ NPC_S_KPU4_FDSA, 0xff,
+ NPC_ETYPE_PPPOE,
+ 0xffff,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
- },
- {
- NPC_S_KPU5_IP, 0xff,
- NPC_IPNH_ESP,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
+ NPC_PPP_IP,
+ 0xffff,
},
{
- NPC_S_KPU5_IP, 0xff,
- NPC_IPNH_AH,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ NPC_S_KPU4_FDSA, 0xff,
+ NPC_ETYPE_PPPOE,
+ 0xffff,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
- },
- {
- NPC_S_KPU5_IP, 0xff,
- NPC_IPNH_GRE,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
+ NPC_PPP_IP6,
+ 0xffff,
},
{
- NPC_S_KPU5_IP, 0xff,
- NPC_IPNH_IP,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ NPC_S_KPU4_FDSA, 0xff,
+ 0x0000,
+ NPC_DSA_FDSA,
+ 0x0000,
+ 0x0000,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
- },
- {
- NPC_S_KPU5_IP, 0xff,
- NPC_IPNH_IP6,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
},
{
- NPC_S_KPU5_IP, 0xff,
- NPC_IPNH_MPLS,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ NPC_S_KPU4_VLAN_EXDSA, 0xff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
},
{
- NPC_S_KPU5_IP, 0xff,
+ NPC_S_KPU4_VLAN_EXDSA, 0xff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ 0x0000,
0x0000,
0x0000,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
},
{
- NPC_S_KPU5_IP, 0xff,
+ NPC_S_KPU4_VLAN_EXDSA, 0xff,
+ NPC_ETYPE_ARP,
+ 0xffff,
0x0000,
0x0000,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU4_VLAN_EXDSA, 0xff,
+ NPC_ETYPE_RARP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU4_VLAN_EXDSA, 0xff,
+ NPC_ETYPE_PTP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU4_VLAN_EXDSA, 0xff,
+ NPC_ETYPE_FCOE,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU4_VLAN_EXDSA, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU4_PPPOE, 0xff,
+ NPC_PPP_IP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU4_PPPOE, 0xff,
+ NPC_PPP_IP6,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_NA, 0X00,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+};
+
+static struct npc_kpu_profile_cam kpu5_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
+ {
+ NPC_S_KPU5_IP, 0xff,
+ 0x0000,
+ NPC_IP_TTL_MASK,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU5_IP, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0001,
+ NPC_IP_HDR_FRAGOFF,
+ },
+ {
+ NPC_S_KPU5_IP, 0xff,
+ NPC_IPNH_TCP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
+ },
+ {
+ NPC_S_KPU5_IP, 0xff,
+ NPC_IPNH_UDP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
+ },
+ {
+ NPC_S_KPU5_IP, 0xff,
+ NPC_IPNH_SCTP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
+ },
+ {
+ NPC_S_KPU5_IP, 0xff,
+ NPC_IPNH_ICMP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
+ },
+ {
+ NPC_S_KPU5_IP, 0xff,
+ NPC_IPNH_IGMP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
+ },
+ {
+ NPC_S_KPU5_IP, 0xff,
+ NPC_IPNH_ESP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
+ },
+ {
+ NPC_S_KPU5_IP, 0xff,
+ NPC_IPNH_AH,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
+ },
+ {
+ NPC_S_KPU5_IP, 0xff,
+ NPC_IPNH_GRE,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
+ },
+ {
+ NPC_S_KPU5_IP, 0xff,
+ NPC_IPNH_IP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
+ },
+ {
+ NPC_S_KPU5_IP, 0xff,
+ NPC_IPNH_IP6,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
+ },
+ {
+ NPC_S_KPU5_IP, 0xff,
+ NPC_IPNH_MPLS,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
+ },
+ {
+ NPC_S_KPU5_IP, 0xff,
+ 0x0000,
+ 0x0000,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
+ },
+ {
+ NPC_S_KPU5_IP, 0xff,
+ 0x0000,
+ 0x0000,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
0x0000,
0x0000,
},
NPC_IP_VER_4,
NPC_IP_VER_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
},
{
NPC_S_KPU5_IP, 0xff,
NPC_IP_VER_4,
NPC_IP_VER_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
},
{
NPC_S_KPU5_IP, 0xff,
NPC_IP_VER_4,
NPC_IP_VER_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
},
{
NPC_S_KPU5_IP, 0xff,
NPC_IP_VER_4,
NPC_IP_VER_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
},
{
NPC_S_KPU5_IP, 0xff,
NPC_IP_VER_4,
NPC_IP_VER_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
},
{
NPC_S_KPU5_IP, 0xff,
NPC_IP_VER_4,
NPC_IP_VER_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
},
{
NPC_S_KPU5_IP, 0xff,
NPC_IP_VER_4,
NPC_IP_VER_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
},
{
NPC_S_KPU5_IP, 0xff,
NPC_IP_VER_4,
NPC_IP_VER_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
},
{
NPC_S_KPU5_IP, 0xff,
NPC_IP_VER_4,
NPC_IP_VER_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
},
{
NPC_S_KPU5_IP, 0xff,
NPC_IP_VER_4,
NPC_IP_VER_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
},
{
NPC_S_KPU5_IP, 0xff,
NPC_IP_VER_4,
NPC_IP_VER_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
},
{
NPC_S_KPU5_IP, 0xff,
NPC_IP_VER_4,
NPC_IP_VER_MASK,
0x0000,
- NPC_IP_HDR_MF|NPC_IP_HDR_FRAGOFF,
+ NPC_IP_HDR_MF | NPC_IP_HDR_FRAGOFF,
},
{
NPC_S_KPU5_IP, 0xff,
0x0000,
},
{
- NPC_S_NA, 0X00,
- 0x0000,
+ NPC_S_KPU5_CPT_IP, 0xff,
0x0000,
+ NPC_IP_TTL_MASK,
0x0000,
0x0000,
0x0000,
0x0000,
},
-};
-
-static const struct npc_kpu_profile_cam kpu6_cam_entries[] = {
{
- NPC_S_KPU6_IP6_EXT, 0xff,
+ NPC_S_KPU5_CPT_IP, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
+ 0x0001,
+ NPC_IP_HDR_FRAGOFF,
+ },
+ {
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_TCP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_FRAG, 0xff,
- NPC_IPNH_TCP << 8,
- 0xff00,
- 0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_UDP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_FRAG, 0xff,
- NPC_IPNH_UDP << 8,
- 0xff00,
- 0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_SCTP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_FRAG, 0xff,
- NPC_IPNH_SCTP << 8,
- 0xff00,
- 0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_ICMP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_FRAG, 0xff,
- NPC_IPNH_ICMP << 8,
- 0xff00,
- 0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_IGMP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_FRAG, 0xff,
- NPC_IPNH_ICMP6 << 8,
- 0xff00,
- 0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_ESP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_FRAG, 0xff,
- NPC_IPNH_ESP << 8,
- 0xff00,
- 0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_AH,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_FRAG, 0xff,
- NPC_IPNH_AH << 8,
- 0xff00,
- 0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_GRE,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_FRAG, 0xff,
- NPC_IPNH_GRE << 8,
- 0xff00,
- 0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_IP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_FRAG, 0xff,
- NPC_IPNH_IP6 << 8,
- 0xff00,
- 0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_IP6,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_FRAG, 0xff,
- NPC_IPNH_MPLS << 8,
- 0xff00,
- 0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_MPLS,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_FRAG, 0xff,
+ NPC_S_KPU5_CPT_IP, 0xff,
0x0000,
0x0000,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_TCP,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_HOP_DEST, 0xff,
- NPC_IPNH_TCP << 8,
- 0xff00,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_UDP,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_SCTP,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_HOP_DEST, 0xff,
- NPC_IPNH_UDP << 8,
- 0xff00,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_ICMP,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_IGMP,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_HOP_DEST, 0xff,
- NPC_IPNH_SCTP << 8,
- 0xff00,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_ESP,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_AH,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_HOP_DEST, 0xff,
- NPC_IPNH_ICMP << 8,
- 0xff00,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_GRE,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_IP,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_HOP_DEST, 0xff,
- NPC_IPNH_ICMP6 << 8,
- 0xff00,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_IP6,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU5_CPT_IP, 0xff,
+ NPC_IPNH_MPLS,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_HOP_DEST, 0xff,
- NPC_IPNH_ESP << 8,
- 0xff00,
+ NPC_S_KPU5_CPT_IP, 0xff,
0x0000,
0x0000,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_HOP_DEST, 0xff,
- NPC_IPNH_AH << 8,
- 0xff00,
+ NPC_S_KPU5_CPT_IP, 0xff,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_HOP_DEST, 0xff,
- NPC_IPNH_GRE << 8,
- 0xff00,
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ 0x0000,
+ NPC_IP6_HOP_MASK,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_HOP_DEST, 0xff,
- NPC_IPNH_IP6 << 8,
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_TCP << 8,
0xff00,
- 0x0000,
- 0x0000,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_HOP_DEST, 0xff,
- NPC_IPNH_MPLS << 8,
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_UDP << 8,
0xff00,
- 0x0000,
- 0x0000,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_HOP_DEST, 0xff,
- NPC_IPNH_ROUT << 8,
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_SCTP << 8,
0xff00,
- 0x0000,
- 0x0000,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_HOP_DEST, 0xff,
- NPC_IPNH_FRAG << 8,
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_ICMP << 8,
0xff00,
- 0x0000,
- 0x0000,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_HOP_DEST, 0xff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_ICMP6 << 8,
+ 0xff00,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_ROUT, 0xff,
- NPC_IPNH_TCP << 8,
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_GRE << 8,
0xff00,
- 0x0000,
- 0x0000,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_ROUT, 0xff,
- NPC_IPNH_UDP << 8,
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_IP6 << 8,
0xff00,
- 0x0000,
- 0x0000,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_ROUT, 0xff,
- NPC_IPNH_SCTP << 8,
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_MPLS << 8,
0xff00,
- 0x0000,
- 0x0000,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_ROUT, 0xff,
- NPC_IPNH_ICMP << 8,
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_HOP << 8,
0xff00,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_DEST << 8,
+ 0xff00,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_ROUT, 0xff,
- NPC_IPNH_ICMP6 << 8,
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_ROUT << 8,
0xff00,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_FRAG << 8,
+ 0xff00,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_ROUT, 0xff,
+ NPC_S_KPU5_CPT_IP6, 0xff,
NPC_IPNH_ESP << 8,
0xff00,
- 0x0000,
- 0x0000,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_ROUT, 0xff,
+ NPC_S_KPU5_CPT_IP6, 0xff,
NPC_IPNH_AH << 8,
0xff00,
- 0x0000,
- 0x0000,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_ROUT, 0xff,
- NPC_IPNH_GRE << 8,
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_MOBILITY << 8,
0xff00,
- 0x0000,
- 0x0000,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_ROUT, 0xff,
- NPC_IPNH_IP6 << 8,
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_HOSTID << 8,
0xff00,
- 0x0000,
- 0x0000,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_ROUT, 0xff,
- NPC_IPNH_MPLS << 8,
+ NPC_S_KPU5_CPT_IP6, 0xff,
+ NPC_IPNH_SHIM6 << 8,
0xff00,
- 0x0000,
- 0x0000,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_ROUT, 0xff,
- NPC_IPNH_FRAG << 8,
- 0xff00,
+ NPC_S_KPU5_CPT_IP6, 0xff,
0x0000,
0x0000,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU6_IP6_ROUT, 0xff,
+ NPC_S_KPU5_CPT_IP6, 0xff,
0x0000,
0x0000,
0x0000,
},
};
-static const struct npc_kpu_profile_cam kpu7_cam_entries[] = {
+static struct npc_kpu_profile_cam kpu6_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
{
- NPC_S_KPU7_IP6_EXT, 0xff,
+ NPC_S_KPU6_IP6_EXT, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_S_KPU6_IP6_FRAG, 0xff,
NPC_IPNH_TCP << 8,
0xff00,
0x0000,
- 0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_S_KPU6_IP6_FRAG, 0xff,
NPC_IPNH_UDP << 8,
0xff00,
0x0000,
- 0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_S_KPU6_IP6_FRAG, 0xff,
NPC_IPNH_SCTP << 8,
0xff00,
0x0000,
- 0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_S_KPU6_IP6_FRAG, 0xff,
NPC_IPNH_ICMP << 8,
0xff00,
0x0000,
- 0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_S_KPU6_IP6_FRAG, 0xff,
NPC_IPNH_ICMP6 << 8,
0xff00,
0x0000,
- 0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_S_KPU6_IP6_FRAG, 0xff,
NPC_IPNH_ESP << 8,
0xff00,
0x0000,
- 0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_S_KPU6_IP6_FRAG, 0xff,
NPC_IPNH_AH << 8,
0xff00,
0x0000,
- 0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_S_KPU6_IP6_FRAG, 0xff,
NPC_IPNH_GRE << 8,
0xff00,
0x0000,
- 0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_S_KPU6_IP6_FRAG, 0xff,
NPC_IPNH_IP6 << 8,
0xff00,
0x0000,
- 0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_S_KPU6_IP6_FRAG, 0xff,
NPC_IPNH_MPLS << 8,
0xff00,
0x0000,
- 0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_S_KPU6_IP6_FRAG, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_S_KPU6_IP6_HOP_DEST, 0xff,
NPC_IPNH_TCP << 8,
0xff00,
0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_S_KPU6_IP6_HOP_DEST, 0xff,
NPC_IPNH_UDP << 8,
0xff00,
0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_S_KPU6_IP6_HOP_DEST, 0xff,
NPC_IPNH_SCTP << 8,
0xff00,
0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_S_KPU6_IP6_HOP_DEST, 0xff,
NPC_IPNH_ICMP << 8,
0xff00,
0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_S_KPU6_IP6_HOP_DEST, 0xff,
NPC_IPNH_ICMP6 << 8,
0xff00,
0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_S_KPU6_IP6_HOP_DEST, 0xff,
NPC_IPNH_ESP << 8,
0xff00,
0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_S_KPU6_IP6_HOP_DEST, 0xff,
NPC_IPNH_AH << 8,
0xff00,
0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_S_KPU6_IP6_HOP_DEST, 0xff,
NPC_IPNH_GRE << 8,
0xff00,
0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_S_KPU6_IP6_HOP_DEST, 0xff,
NPC_IPNH_IP6 << 8,
0xff00,
0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_S_KPU6_IP6_HOP_DEST, 0xff,
NPC_IPNH_MPLS << 8,
0xff00,
0x0000,
- NPC_IP6_FRAG_FRAGOFF,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU7_IP6_FRAG, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU6_IP6_HOP_DEST, 0xff,
+ NPC_IPNH_ROUT << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_NA, 0X00,
- 0x0000,
- 0x0000,
+ NPC_S_KPU6_IP6_HOP_DEST, 0xff,
+ NPC_IPNH_FRAG << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
-};
-
-static const struct npc_kpu_profile_cam kpu8_cam_entries[] = {
{
- NPC_S_KPU8_TCP, 0xff,
+ NPC_S_KPU6_IP6_HOP_DEST, 0xff,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
- NPC_TCP_FLAGS_FIN,
- NPC_TCP_FLAGS_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_TCP, 0xff,
- 0x0000,
+ NPC_S_KPU6_IP6_ROUT, 0xff,
+ NPC_IPNH_TCP << 8,
+ 0xff00,
0x0000,
0x0000,
- NPC_TCP_FLAGS_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_TCP, 0xff,
+ NPC_S_KPU6_IP6_ROUT, 0xff,
+ NPC_IPNH_UDP << 8,
+ 0xff00,
0x0000,
0x0000,
- NPC_TCP_FLAGS_RST|NPC_TCP_FLAGS_FIN,
- NPC_TCP_FLAGS_RST|NPC_TCP_FLAGS_FIN,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_TCP, 0xff,
+ NPC_S_KPU6_IP6_ROUT, 0xff,
+ NPC_IPNH_SCTP << 8,
+ 0xff00,
0x0000,
0x0000,
- NPC_TCP_FLAGS_URG|NPC_TCP_FLAGS_SYN,
- NPC_TCP_FLAGS_URG|NPC_TCP_FLAGS_SYN,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_TCP, 0xff,
+ NPC_S_KPU6_IP6_ROUT, 0xff,
+ NPC_IPNH_ICMP << 8,
+ 0xff00,
0x0000,
0x0000,
- NPC_TCP_FLAGS_RST|NPC_TCP_FLAGS_SYN,
- NPC_TCP_FLAGS_RST|NPC_TCP_FLAGS_SYN,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_TCP, 0xff,
+ NPC_S_KPU6_IP6_ROUT, 0xff,
+ NPC_IPNH_ICMP6 << 8,
+ 0xff00,
0x0000,
0x0000,
- NPC_TCP_FLAGS_SYN|NPC_TCP_FLAGS_FIN,
- NPC_TCP_FLAGS_SYN|NPC_TCP_FLAGS_FIN,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_TCP, 0xff,
- NPC_TCP_PORT_HTTP,
- 0xffff,
- NPC_TCP_DATA_OFFSET_5,
- NPC_TCP_DATA_OFFSET_MASK,
+ NPC_S_KPU6_IP6_ROUT, 0xff,
+ NPC_IPNH_ESP << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_TCP, 0xff,
- NPC_TCP_PORT_HTTPS,
- 0xffff,
- NPC_TCP_DATA_OFFSET_5,
- NPC_TCP_DATA_OFFSET_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_TCP, 0xff,
- NPC_TCP_PORT_PPTP,
- 0xffff,
- NPC_TCP_DATA_OFFSET_5,
- NPC_TCP_DATA_OFFSET_MASK,
+ NPC_S_KPU6_IP6_ROUT, 0xff,
+ NPC_IPNH_AH << 8,
+ 0xff00,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_TCP, 0xff,
+ NPC_S_KPU6_IP6_ROUT, 0xff,
+ NPC_IPNH_GRE << 8,
+ 0xff00,
0x0000,
0x0000,
- NPC_TCP_DATA_OFFSET_5,
- NPC_TCP_DATA_OFFSET_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_TCP, 0xff,
- NPC_TCP_PORT_HTTP,
- 0xffff,
+ NPC_S_KPU6_IP6_ROUT, 0xff,
+ NPC_IPNH_IP6 << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_TCP, 0xff,
- NPC_TCP_PORT_HTTPS,
- 0xffff,
+ NPC_S_KPU6_IP6_ROUT, 0xff,
+ NPC_IPNH_MPLS << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_TCP, 0xff,
- NPC_TCP_PORT_PPTP,
- 0xffff,
+ NPC_S_KPU6_IP6_ROUT, 0xff,
+ NPC_IPNH_FRAG << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_TCP, 0xff,
+ NPC_S_KPU6_IP6_ROUT, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_UDP, 0xff,
- NPC_UDP_PORT_VXLAN,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_FRAG, 0xff,
+ NPC_IPNH_TCP << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_UDP, 0xff,
- NPC_UDP_PORT_VXLANGPE,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_FRAG, 0xff,
+ NPC_IPNH_UDP << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_UDP, 0xff,
- NPC_UDP_PORT_GENEVE,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_FRAG, 0xff,
+ NPC_IPNH_SCTP << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_UDP, 0xff,
- NPC_UDP_PORT_GTPC,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_FRAG, 0xff,
+ NPC_IPNH_ICMP << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_UDP, 0xff,
- NPC_UDP_PORT_GTPU,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_FRAG, 0xff,
+ NPC_IPNH_ICMP6 << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_UDP, 0xff,
- NPC_UDP_PORT_PTP_E,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_FRAG, 0xff,
+ NPC_IPNH_ESP << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_UDP, 0xff,
- NPC_UDP_PORT_PTP_G,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_FRAG, 0xff,
+ NPC_IPNH_AH << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_UDP, 0xff,
- NPC_UDP_PORT_MPLS,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_FRAG, 0xff,
+ NPC_IPNH_GRE << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_UDP, 0xff,
- NPC_UDP_PORT_ESP,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_FRAG, 0xff,
+ NPC_IPNH_IP6 << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_UDP, 0xff,
+ NPC_S_KPU6_IP6_CPT_FRAG, 0xff,
+ NPC_IPNH_MPLS << 8,
+ 0xff00,
0x0000,
0x0000,
- NPC_UDP_PORT_ESP,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_UDP, 0xff,
+ NPC_S_KPU6_IP6_CPT_FRAG, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_SCTP, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 0xff,
+ NPC_IPNH_TCP << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_ICMP, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 0xff,
+ NPC_IPNH_UDP << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_IGMP, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 0xff,
+ NPC_IPNH_SCTP << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_ICMP6, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 0xff,
+ NPC_IPNH_ICMP << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_AH, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 0xff,
+ NPC_IPNH_ICMP6 << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_TRANS_ETH_BR,
- 0xffff,
- NPC_GRE_F_KEY,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 0xff,
+ NPC_IPNH_ESP << 8,
+ 0xff00,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_TRANS_ETH_BR,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 0xff,
+ NPC_IPNH_AH << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSU,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 0xff,
+ NPC_IPNH_GRE << 8,
+ 0xff00,
+ 0x0000,
0x0000,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSU,
- 0xffff,
- NPC_GRE_F_CSUM,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 0xff,
+ NPC_IPNH_IP6 << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSU,
- 0xffff,
- NPC_GRE_F_KEY,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSU,
- 0xffff,
- NPC_GRE_F_SEQ,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 0xff,
+ NPC_IPNH_MPLS << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSU,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_KEY,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSU,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_SEQ,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 0xff,
+ NPC_IPNH_ROUT << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSU,
- 0xffff,
- NPC_GRE_F_KEY|NPC_GRE_F_SEQ,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSU,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_KEY|NPC_GRE_F_SEQ,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 0xff,
+ NPC_IPNH_FRAG << 8,
+ 0xff00,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSM,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
0x0000,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSM,
- 0xffff,
- NPC_GRE_F_CSUM,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_ROUT, 0xff,
+ NPC_IPNH_TCP << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSM,
- 0xffff,
- NPC_GRE_F_KEY,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSM,
- 0xffff,
- NPC_GRE_F_SEQ,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_ROUT, 0xff,
+ NPC_IPNH_UDP << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSM,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_KEY,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSM,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_SEQ,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_ROUT, 0xff,
+ NPC_IPNH_SCTP << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSM,
- 0xffff,
- NPC_GRE_F_KEY|NPC_GRE_F_SEQ,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_MPLSM,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_KEY|NPC_GRE_F_SEQ,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_ROUT, 0xff,
+ NPC_IPNH_ICMP << 8,
+ 0xff00,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_NSH,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_ROUT, 0xff,
+ NPC_IPNH_ICMP6 << 8,
+ 0xff00,
+ 0x0000,
0x0000,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_NSH,
- 0xffff,
- NPC_GRE_F_CSUM,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_ROUT, 0xff,
+ NPC_IPNH_ESP << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_NSH,
- 0xffff,
- NPC_GRE_F_KEY,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_NSH,
- 0xffff,
- NPC_GRE_F_SEQ,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_ROUT, 0xff,
+ NPC_IPNH_AH << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_NSH,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_KEY,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_NSH,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_SEQ,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_ROUT, 0xff,
+ NPC_IPNH_GRE << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_NSH,
- 0xffff,
- NPC_GRE_F_KEY|NPC_GRE_F_SEQ,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_NSH,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_KEY|NPC_GRE_F_SEQ,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_ROUT, 0xff,
+ NPC_IPNH_IP6 << 8,
+ 0xff00,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_ROUT, 0xff,
+ NPC_IPNH_MPLS << 8,
+ 0xff00,
+ 0x0000,
0x0000,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP,
- 0xffff,
- NPC_GRE_F_CSUM,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_ROUT, 0xff,
+ NPC_IPNH_FRAG << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP,
- 0xffff,
- NPC_GRE_F_KEY,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP,
- 0xffff,
- NPC_GRE_F_SEQ,
- 0xffff,
+ NPC_S_KPU6_IP6_CPT_ROUT, 0xff,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_KEY,
- 0xffff,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_SEQ,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP,
- 0xffff,
- NPC_GRE_F_KEY|NPC_GRE_F_SEQ,
- 0xffff,
+ NPC_S_NA, 0X00,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_KEY|NPC_GRE_F_SEQ,
- 0xffff,
0x0000,
0x0000,
},
+};
+
+static struct npc_kpu_profile_cam kpu7_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP6,
- 0xffff,
+ NPC_S_KPU7_IP6_EXT, 0xff,
+ 0x0000,
0x0000,
- 0xffff,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP6,
- 0xffff,
- NPC_GRE_F_CSUM,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP6,
- 0xffff,
- NPC_GRE_F_KEY,
- 0xffff,
+ NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_IPNH_TCP << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP6,
- 0xffff,
- NPC_GRE_F_SEQ,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP6,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_KEY,
- 0xffff,
+ NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_IPNH_UDP << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP6,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_SEQ,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP6,
- 0xffff,
- NPC_GRE_F_KEY|NPC_GRE_F_SEQ,
- 0xffff,
+ NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_IPNH_SCTP << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_IP6,
- 0xffff,
- NPC_GRE_F_CSUM|NPC_GRE_F_KEY|NPC_GRE_F_SEQ,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
+ NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_IPNH_ICMP << 8,
+ 0xff00,
+ 0x0000,
0x0000,
- 0xffff,
- NPC_GRE_F_ROUTE,
- 0x4fff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
+ NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_IPNH_ICMP6 << 8,
+ 0xff00,
0x0000,
- 0xffff,
0x0000,
- 0x4fff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
+ NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_IPNH_ESP << 8,
+ 0xff00,
0x0000,
- 0xffff,
0x0000,
- 0x0003,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_PPP,
- 0xffff,
- NPC_GRE_F_KEY|NPC_GRE_VER_1,
- 0xffff,
+ NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_IPNH_AH << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_PPP,
- 0xffff,
- NPC_GRE_F_KEY|NPC_GRE_F_SEQ|NPC_GRE_VER_1,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_PPP,
- 0xffff,
- NPC_GRE_F_KEY|NPC_GRE_F_ACK|NPC_GRE_VER_1,
- 0xffff,
+ NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_IPNH_GRE << 8,
+ 0xff00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU8_GRE, 0xff,
- NPC_ETYPE_PPP,
- 0xffff,
- NPC_GRE_F_KEY|NPC_GRE_F_SEQ|NPC_GRE_F_ACK|NPC_GRE_VER_1,
- 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
- 0x0000,
- 0xffff,
- 0x2001,
- 0xef7f,
+ NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_IPNH_IP6 << 8,
+ 0xff00,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU8_GRE, 0xff,
+ NPC_S_KPU7_IP6_ROUT, 0xff,
+ NPC_IPNH_MPLS << 8,
+ 0xff00,
+ 0x0000,
0x0000,
- 0xffff,
- 0x0001,
- 0x0003,
0x0000,
0x0000,
},
{
- NPC_S_NA, 0X00,
+ NPC_S_KPU7_IP6_ROUT, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
0x0000,
},
-};
-
-static const struct npc_kpu_profile_cam kpu9_cam_entries[] = {
{
- NPC_S_KPU9_TU_MPLS_IN_GRE, 0xff,
- NPC_MPLS_S,
- NPC_MPLS_S,
- 0x0000,
+ NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_IPNH_TCP << 8,
+ 0xff00,
0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_TU_MPLS_IN_GRE, 0xff,
+ NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_IPNH_UDP << 8,
+ 0xff00,
0x0000,
- NPC_MPLS_S,
- NPC_MPLS_S,
- NPC_MPLS_S,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_TU_MPLS_IN_GRE, 0xff,
+ NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_IPNH_SCTP << 8,
+ 0xff00,
+ 0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
- NPC_MPLS_S,
0x0000,
- NPC_MPLS_S,
- NPC_MPLS_S,
- NPC_MPLS_S,
},
{
- NPC_S_KPU9_TU_MPLS_IN_GRE, 0xff,
+ NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_IPNH_ICMP << 8,
+ 0xff00,
0x0000,
- NPC_MPLS_S,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
- NPC_MPLS_S,
0x0000,
- NPC_MPLS_S,
},
{
- NPC_S_KPU9_TU_MPLS_IN_NSH, 0xff,
- NPC_MPLS_S,
- NPC_MPLS_S,
- 0x0000,
+ NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_IPNH_ICMP6 << 8,
+ 0xff00,
0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_TU_MPLS_IN_NSH, 0xff,
+ NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_IPNH_ESP << 8,
+ 0xff00,
0x0000,
- NPC_MPLS_S,
- NPC_MPLS_S,
- NPC_MPLS_S,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_TU_MPLS_IN_NSH, 0xff,
+ NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_IPNH_AH << 8,
+ 0xff00,
+ 0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
- NPC_MPLS_S,
0x0000,
- NPC_MPLS_S,
- NPC_MPLS_S,
- NPC_MPLS_S,
},
{
- NPC_S_KPU9_TU_MPLS_IN_NSH, 0xff,
+ NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_IPNH_GRE << 8,
+ 0xff00,
0x0000,
- NPC_MPLS_S,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
- NPC_MPLS_S,
0x0000,
- NPC_MPLS_S,
},
{
- NPC_S_KPU9_TU_MPLS_IN_IP, 0xff,
- NPC_MPLS_S,
- NPC_MPLS_S,
- 0x0000,
+ NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_IPNH_IP6 << 8,
+ 0xff00,
0x0000,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_TU_MPLS_IN_IP, 0xff,
+ NPC_S_KPU7_IP6_FRAG, 0xff,
+ NPC_IPNH_MPLS << 8,
+ 0xff00,
0x0000,
- NPC_MPLS_S,
- NPC_MPLS_S,
- NPC_MPLS_S,
+ NPC_IP6_FRAG_FRAGOFF,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_TU_MPLS_IN_IP, 0xff,
+ NPC_S_KPU7_IP6_FRAG, 0xff,
+ 0x0000,
0x0000,
- NPC_MPLS_S,
0x0000,
- NPC_MPLS_S,
- NPC_MPLS_S,
- NPC_MPLS_S,
- },
- {
- NPC_S_KPU9_TU_MPLS_IN_IP, 0xff,
0x0000,
- NPC_MPLS_S,
0x0000,
- NPC_MPLS_S,
0x0000,
- NPC_MPLS_S,
},
{
- NPC_S_KPU9_TU_NSH_IN_GRE, 0xff,
- NPC_NSH_NP_IP,
- NPC_NSH_NP_MASK,
+ NPC_S_KPU7_CPT_IP6_FRAG, 0xff,
+ NPC_IPNH_TCP << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_TU_NSH_IN_GRE, 0xff,
- NPC_NSH_NP_IP6,
- NPC_NSH_NP_MASK,
+ NPC_S_KPU7_CPT_IP6_FRAG, 0xff,
+ NPC_IPNH_UDP << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_TU_NSH_IN_GRE, 0xff,
- NPC_NSH_NP_ETH,
- NPC_NSH_NP_MASK,
+ NPC_S_KPU7_CPT_IP6_FRAG, 0xff,
+ NPC_IPNH_SCTP << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_TU_NSH_IN_GRE, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU7_CPT_IP6_FRAG, 0xff,
+ NPC_IPNH_ICMP << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_VXLAN, 0xff,
+ NPC_S_KPU7_CPT_IP6_FRAG, 0xff,
+ NPC_IPNH_ICMP6 << 8,
+ 0xff00,
+ 0x0000,
0x0000,
0x0000,
- NPC_VXLAN_I,
- NPC_VXLAN_I,
0x0000,
- 0xffff,
},
{
- NPC_S_KPU9_VXLAN, 0xff,
+ NPC_S_KPU7_CPT_IP6_FRAG, 0xff,
+ NPC_IPNH_ESP << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
- 0xffff,
0x0000,
- 0xffff,
},
{
- NPC_S_KPU9_VXLAN, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU7_CPT_IP6_FRAG, 0xff,
+ NPC_IPNH_AH << 8,
+ 0xff00,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_VXLANGPE, 0xff,
+ NPC_S_KPU7_CPT_IP6_FRAG, 0xff,
+ NPC_IPNH_GRE << 8,
+ 0xff00,
0x0000,
0x0000,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_NP_IP,
- NPC_VXLANGPE_NP_MASK,
- },
- {
- NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_NP_IP6,
- NPC_VXLANGPE_NP_MASK,
},
{
- NPC_S_KPU9_VXLANGPE, 0xff,
+ NPC_S_KPU7_CPT_IP6_FRAG, 0xff,
+ NPC_IPNH_IP6 << 8,
+ 0xff00,
0x0000,
0x0000,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_NP_ETH,
- NPC_VXLANGPE_NP_MASK,
- },
- {
- NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_NP_NSH,
- NPC_VXLANGPE_NP_MASK,
},
{
- NPC_S_KPU9_VXLANGPE, 0xff,
+ NPC_S_KPU7_CPT_IP6_FRAG, 0xff,
+ NPC_IPNH_MPLS << 8,
+ 0xff00,
0x0000,
0x0000,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_NP_MPLS,
- NPC_VXLANGPE_NP_MASK,
- },
- {
- NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
- NPC_VXLANGPE_P,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_NP_IP,
- NPC_VXLANGPE_NP_MASK,
},
{
- NPC_S_KPU9_VXLANGPE, 0xff,
+ NPC_S_KPU7_CPT_IP6_FRAG, 0xff,
0x0000,
0x0000,
- NPC_VXLANGPE_P,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_NP_IP6,
- NPC_VXLANGPE_NP_MASK,
- },
- {
- NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
- NPC_VXLANGPE_P,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_NP_ETH,
- NPC_VXLANGPE_NP_MASK,
- },
- {
- NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
- NPC_VXLANGPE_P,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_NP_NSH,
- NPC_VXLANGPE_NP_MASK,
},
{
- NPC_S_KPU9_VXLANGPE, 0xff,
+ NPC_S_NA, 0X00,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
- NPC_VXLANGPE_P,
- NPC_VXLANGPE_P | NPC_VXLANGPE_I,
- NPC_VXLANGPE_NP_MPLS,
- NPC_VXLANGPE_NP_MASK,
},
+};
+
+static struct npc_kpu_profile_cam kpu8_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
{
- NPC_S_KPU9_VXLANGPE, 0xff,
+ NPC_S_KPU8_TCP, 0xff,
0x0000,
0x0000,
- NPC_VXLANGPE_P,
- NPC_VXLANGPE_P,
+ NPC_TCP_FLAGS_FIN,
+ NPC_TCP_FLAGS_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_VXLANGPE, 0xff,
+ NPC_S_KPU8_TCP, 0xff,
0x0000,
0x0000,
0x0000,
- NPC_VXLANGPE_P,
+ NPC_TCP_FLAGS_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_GENEVE, 0xff,
+ NPC_S_KPU8_TCP, 0xff,
0x0000,
0x0000,
+ NPC_TCP_FLAGS_RST | NPC_TCP_FLAGS_FIN,
+ NPC_TCP_FLAGS_RST | NPC_TCP_FLAGS_FIN,
+ 0x0000,
0x0000,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_ETYPE_TRANS_ETH_BR,
- 0xffff,
},
{
- NPC_S_KPU9_GENEVE, 0xff,
+ NPC_S_KPU8_TCP, 0xff,
0x0000,
0x0000,
- NPC_GENEVE_F_OAM,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_ETYPE_TRANS_ETH_BR,
- 0xffff,
- },
- {
- NPC_S_KPU9_GENEVE, 0xff,
+ NPC_TCP_FLAGS_URG | NPC_TCP_FLAGS_SYN,
+ NPC_TCP_FLAGS_URG | NPC_TCP_FLAGS_SYN,
0x0000,
0x0000,
- NPC_GENEVE_F_CRI_OPT,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_ETYPE_TRANS_ETH_BR,
- 0xffff,
},
{
- NPC_S_KPU9_GENEVE, 0xff,
- 0x0000,
+ NPC_S_KPU8_TCP, 0xff,
0x0000,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_ETYPE_TRANS_ETH_BR,
- 0xffff,
- },
- {
- NPC_S_KPU9_GENEVE, 0xff,
0x0000,
+ NPC_TCP_FLAGS_RST | NPC_TCP_FLAGS_SYN,
+ NPC_TCP_FLAGS_RST | NPC_TCP_FLAGS_SYN,
0x0000,
0x0000,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_ETYPE_IP,
- 0xffff,
},
{
- NPC_S_KPU9_GENEVE, 0xff,
+ NPC_S_KPU8_TCP, 0xff,
0x0000,
0x0000,
- NPC_GENEVE_F_OAM,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_ETYPE_IP,
- 0xffff,
- },
- {
- NPC_S_KPU9_GENEVE, 0xff,
+ NPC_TCP_FLAGS_SYN | NPC_TCP_FLAGS_FIN,
+ NPC_TCP_FLAGS_SYN | NPC_TCP_FLAGS_FIN,
0x0000,
0x0000,
- NPC_GENEVE_F_CRI_OPT,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_ETYPE_IP,
- 0xffff,
},
{
- NPC_S_KPU9_GENEVE, 0xff,
- 0x0000,
- 0x0000,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_ETYPE_IP,
+ NPC_S_KPU8_TCP, 0xff,
+ NPC_TCP_PORT_HTTP,
0xffff,
- },
- {
- NPC_S_KPU9_GENEVE, 0xff,
- 0x0000,
+ NPC_TCP_DATA_OFFSET_5,
+ NPC_TCP_DATA_OFFSET_MASK,
0x0000,
0x0000,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_ETYPE_IP6,
- 0xffff,
},
{
- NPC_S_KPU9_GENEVE, 0xff,
- 0x0000,
- 0x0000,
- NPC_GENEVE_F_OAM,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_ETYPE_IP6,
+ NPC_S_KPU8_TCP, 0xff,
+ NPC_TCP_PORT_HTTPS,
0xffff,
- },
- {
- NPC_S_KPU9_GENEVE, 0xff,
+ NPC_TCP_DATA_OFFSET_5,
+ NPC_TCP_DATA_OFFSET_MASK,
0x0000,
0x0000,
- NPC_GENEVE_F_CRI_OPT,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_ETYPE_IP6,
- 0xffff,
},
{
- NPC_S_KPU9_GENEVE, 0xff,
+ NPC_S_KPU8_TCP, 0xff,
+ NPC_TCP_PORT_PPTP,
+ 0xffff,
+ NPC_TCP_DATA_OFFSET_5,
+ NPC_TCP_DATA_OFFSET_MASK,
0x0000,
0x0000,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
- NPC_ETYPE_IP6,
- 0xffff,
},
{
- NPC_S_KPU9_GTPC, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU8_TCP, 0xff,
0x0000,
0x0000,
+ NPC_TCP_DATA_OFFSET_5,
+ NPC_TCP_DATA_OFFSET_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_GTPU, 0xff,
+ NPC_S_KPU8_TCP, 0xff,
+ NPC_TCP_PORT_HTTP,
+ 0xffff,
0x0000,
0x0000,
- NPC_GTP_PT_GTP | NPC_GTP_VER1 | NPC_GTP_MT_G_PDU,
- NPC_GTP_PT_MASK | NPC_GTP_VER_MASK | NPC_GTP_MT_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_GTPU, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU8_TCP, 0xff,
+ NPC_TCP_PORT_HTTPS,
+ 0xffff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_TU_MPLS_IN_UDP, 0xff,
- NPC_MPLS_S,
- NPC_MPLS_S,
+ NPC_S_KPU8_TCP, 0xff,
+ NPC_TCP_PORT_PPTP,
+ 0xffff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU9_TU_MPLS_IN_UDP, 0xff,
- 0x0000,
- NPC_MPLS_S,
- NPC_MPLS_S,
- NPC_MPLS_S,
- 0x0000,
+ NPC_S_KPU8_TCP, 0xff,
0x0000,
- },
- {
- NPC_S_KPU9_TU_MPLS_IN_UDP, 0xff,
0x0000,
- NPC_MPLS_S,
0x0000,
- NPC_MPLS_S,
- NPC_MPLS_S,
- NPC_MPLS_S,
- },
- {
- NPC_S_KPU9_TU_MPLS_IN_UDP, 0xff,
0x0000,
- NPC_MPLS_S,
0x0000,
- NPC_MPLS_S,
0x0000,
- NPC_MPLS_S,
},
{
- NPC_S_KPU9_ESP, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU8_UDP, 0xff,
+ NPC_UDP_PORT_VXLAN,
+ 0xffff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_NA, 0X00,
- 0x0000,
- 0x0000,
+ NPC_S_KPU8_UDP, 0xff,
+ NPC_UDP_PORT_VXLANGPE,
+ 0xffff,
0x0000,
0x0000,
0x0000,
0x0000,
},
-};
-
-static const struct npc_kpu_profile_cam kpu10_cam_entries[] = {
{
- NPC_S_KPU10_TU_MPLS, 0xff,
- NPC_MPLS_S,
- NPC_MPLS_S,
- NPC_IP_VER_4,
- NPC_IP_VER_MASK,
+ NPC_S_KPU8_UDP, 0xff,
+ NPC_UDP_PORT_GENEVE,
+ 0xffff,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU10_TU_MPLS, 0xff,
- NPC_MPLS_S,
- NPC_MPLS_S,
- NPC_IP_VER_6,
- NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU10_TU_MPLS, 0xff,
- NPC_MPLS_S,
- NPC_MPLS_S,
- 0x0000,
+ NPC_S_KPU8_UDP, 0xff,
+ NPC_UDP_PORT_GTPC,
0xffff,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU10_TU_MPLS, 0xff,
- NPC_MPLS_S,
- NPC_MPLS_S,
- 0x0000,
- 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU10_TU_MPLS, 0xff,
- 0x0000,
- NPC_MPLS_S,
+ NPC_S_KPU8_UDP, 0xff,
+ NPC_UDP_PORT_GTPU,
+ 0xffff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU10_TU_MPLS_PL, 0xff,
- NPC_IP_VER_4,
- NPC_IP_VER_MASK,
+ NPC_S_KPU8_UDP, 0xff,
+ NPC_UDP_PORT_PTP_E,
+ 0xffff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU10_TU_MPLS_PL, 0xff,
- NPC_IP_VER_6,
- NPC_IP_VER_MASK,
+ NPC_S_KPU8_UDP, 0xff,
+ NPC_UDP_PORT_PTP_G,
+ 0xffff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU10_TU_MPLS_PL, 0xff,
- 0x0000,
+ NPC_S_KPU8_UDP, 0xff,
+ NPC_UDP_PORT_MPLS,
0xffff,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU10_TU_MPLS_PL, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU8_UDP, 0xff,
+ NPC_UDP_PORT_ESP,
+ 0xffff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU10_TU_MPLS_IN_VXLANGPE, 0xff,
- NPC_MPLS_S,
- NPC_MPLS_S,
+ NPC_S_KPU8_UDP, 0xff,
0x0000,
0x0000,
+ NPC_UDP_PORT_ESP,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU10_TU_MPLS_IN_VXLANGPE, 0xff,
+ NPC_S_KPU8_UDP, 0xff,
+ 0x0000,
0x0000,
- NPC_MPLS_S,
- NPC_MPLS_S,
- NPC_MPLS_S,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU10_TU_MPLS_IN_VXLANGPE, 0xff,
0x0000,
- NPC_MPLS_S,
0x0000,
- NPC_MPLS_S,
- NPC_MPLS_S,
- NPC_MPLS_S,
},
{
- NPC_S_KPU10_TU_MPLS_IN_VXLANGPE, 0xff,
- 0x0000,
- NPC_MPLS_S,
+ NPC_S_KPU8_SCTP, 0xff,
0x0000,
- NPC_MPLS_S,
0x0000,
- NPC_MPLS_S,
- },
- {
- NPC_S_KPU10_TU_NSH_IN_VXLANGPE, 0xff,
- NPC_NSH_NP_IP,
- NPC_NSH_NP_MASK,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU10_TU_NSH_IN_VXLANGPE, 0xff,
- NPC_NSH_NP_IP6,
- NPC_NSH_NP_MASK,
+ NPC_S_KPU8_ICMP, 0xff,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU10_TU_NSH_IN_VXLANGPE, 0xff,
- NPC_NSH_NP_ETH,
- NPC_NSH_NP_MASK,
+ NPC_S_KPU8_IGMP, 0xff,
+ 0x0000,
+ 0x0000,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU10_TU_NSH_IN_VXLANGPE, 0xff,
+ NPC_S_KPU8_ICMP6, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_NA, 0X00,
+ NPC_S_KPU8_AH, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
0x0000,
},
-};
-
-static const struct npc_kpu_profile_cam kpu11_cam_entries[] = {
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_IP,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_TRANS_ETH_BR,
+ 0xffff,
+ NPC_GRE_F_KEY,
0xffff,
- 0x0000,
- 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_IP6,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_TRANS_ETH_BR,
0xffff,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_ARP,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSU,
0xffff,
0x0000,
- 0x0000,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSU,
0xffff,
- NPC_ETYPE_IP,
+ NPC_GRE_F_CSUM,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSU,
0xffff,
- NPC_ETYPE_IP6,
+ NPC_GRE_F_KEY,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSU,
0xffff,
- NPC_ETYPE_ARP,
+ NPC_GRE_F_SEQ,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSU,
+ 0xffff,
+ NPC_GRE_F_CSUM | NPC_GRE_F_KEY,
0xffff,
- 0x0000,
- 0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSU,
0xffff,
- NPC_ETYPE_IP,
+ NPC_GRE_F_CSUM | NPC_GRE_F_SEQ,
0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSU,
0xffff,
- NPC_ETYPE_IP6,
+ NPC_GRE_F_KEY | NPC_GRE_F_SEQ,
0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_SBTAG,
- 0xffff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSU,
0xffff,
- NPC_ETYPE_ARP,
+ NPC_GRE_F_CSUM | NPC_GRE_F_KEY | NPC_GRE_F_SEQ,
0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_SBTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSM,
0xffff,
- NPC_ETYPE_CTAG,
+ 0x0000,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_SBTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSM,
0xffff,
- NPC_ETYPE_IP,
+ NPC_GRE_F_CSUM,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_SBTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSM,
0xffff,
- NPC_ETYPE_IP6,
+ NPC_GRE_F_KEY,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_SBTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSM,
0xffff,
- NPC_ETYPE_ARP,
+ NPC_GRE_F_SEQ,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_SBTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSM,
+ 0xffff,
+ NPC_GRE_F_CSUM | NPC_GRE_F_KEY,
0xffff,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSM,
+ 0xffff,
+ NPC_GRE_F_CSUM | NPC_GRE_F_SEQ,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_QINQ,
- 0xffff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSM,
0xffff,
- NPC_ETYPE_IP,
+ NPC_GRE_F_KEY | NPC_GRE_F_SEQ,
0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_QINQ,
- 0xffff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_MPLSM,
0xffff,
- NPC_ETYPE_IP6,
+ NPC_GRE_F_CSUM | NPC_GRE_F_KEY | NPC_GRE_F_SEQ,
0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_QINQ,
- 0xffff,
- NPC_ETYPE_CTAG,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_NSH,
0xffff,
- NPC_ETYPE_ARP,
+ 0x0000,
0xffff,
+ 0x0000,
+ 0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_QINQ,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_NSH,
0xffff,
- NPC_ETYPE_CTAG,
+ NPC_GRE_F_CSUM,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_QINQ,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_NSH,
0xffff,
- NPC_ETYPE_IP,
+ NPC_GRE_F_KEY,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_QINQ,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_NSH,
0xffff,
- NPC_ETYPE_IP6,
+ NPC_GRE_F_SEQ,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_QINQ,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_NSH,
0xffff,
- NPC_ETYPE_ARP,
+ NPC_GRE_F_CSUM | NPC_GRE_F_KEY,
0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- NPC_ETYPE_QINQ,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_NSH,
+ 0xffff,
+ NPC_GRE_F_CSUM | NPC_GRE_F_SEQ,
0xffff,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_NSH,
+ 0xffff,
+ NPC_GRE_F_KEY | NPC_GRE_F_SEQ,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER, 0xff,
- 0x0000,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_NSH,
+ 0xffff,
+ NPC_GRE_F_CSUM | NPC_GRE_F_KEY | NPC_GRE_F_SEQ,
+ 0xffff,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP,
+ 0xffff,
0x0000,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_PPP, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ NPC_GRE_F_CSUM,
+ 0xffff,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ NPC_GRE_F_KEY,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_MPLS, 0xff,
- NPC_MPLS_S,
- NPC_MPLS_S,
- NPC_IP_VER_4,
- NPC_IP_VER_MASK,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ NPC_GRE_F_SEQ,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_MPLS, 0xff,
- NPC_MPLS_S,
- NPC_MPLS_S,
- NPC_IP_VER_6,
- NPC_IP_VER_MASK,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ NPC_GRE_F_CSUM | NPC_GRE_F_KEY,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_MPLS, 0xff,
- NPC_MPLS_S,
- NPC_MPLS_S,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ NPC_GRE_F_CSUM | NPC_GRE_F_SEQ,
+ 0xffff,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ NPC_GRE_F_KEY | NPC_GRE_F_SEQ,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_MPLS, 0xff,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ NPC_GRE_F_CSUM | NPC_GRE_F_KEY | NPC_GRE_F_SEQ,
+ 0xffff,
0x0000,
- NPC_MPLS_S,
0x0000,
+ },
+ {
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP6,
+ 0xffff,
0x0000,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_MPLS_PL, 0xff,
- NPC_IP_VER_4,
- NPC_IP_VER_MASK,
- 0x0000,
- 0x0000,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ NPC_GRE_F_CSUM,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_MPLS_PL, 0xff,
- NPC_IP_VER_6,
- NPC_IP_VER_MASK,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ NPC_GRE_F_KEY,
+ 0xffff,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ NPC_GRE_F_SEQ,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_MPLS_PL, 0xff,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ NPC_GRE_F_CSUM | NPC_GRE_F_KEY,
+ 0xffff,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ NPC_GRE_F_CSUM | NPC_GRE_F_SEQ,
+ 0xffff,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ NPC_GRE_F_KEY | NPC_GRE_F_SEQ,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU11_TU_ETHER_IN_NSH, 0xff,
- 0x0000,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ NPC_GRE_F_CSUM | NPC_GRE_F_KEY | NPC_GRE_F_SEQ,
+ 0xffff,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU8_GRE, 0xff,
0x0000,
+ 0xffff,
+ NPC_GRE_F_ROUTE,
+ 0x4fff,
0x0000,
0x0000,
},
{
- NPC_S_NA, 0X00,
- 0x0000,
- 0x0000,
+ NPC_S_KPU8_GRE, 0xff,
0x0000,
+ 0xffff,
0x0000,
+ 0x4fff,
0x0000,
0x0000,
},
-};
-
-static const struct npc_kpu_profile_cam kpu12_cam_entries[] = {
{
- NPC_S_KPU12_TU_IP, 0xff,
- NPC_IPNH_TCP,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ NPC_S_KPU8_GRE, 0xff,
0x0000,
+ 0xffff,
0x0000,
- },
- {
- NPC_S_KPU12_TU_IP, 0xff,
- NPC_IPNH_UDP,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ 0x0003,
0x0000,
0x0000,
},
{
- NPC_S_KPU12_TU_IP, 0xff,
- NPC_IPNH_SCTP,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_PPP,
+ 0xffff,
+ NPC_GRE_F_KEY | NPC_GRE_VER_1,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU12_TU_IP, 0xff,
- NPC_IPNH_ICMP,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_PPP,
+ 0xffff,
+ NPC_GRE_F_KEY | NPC_GRE_F_SEQ | NPC_GRE_VER_1,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU12_TU_IP, 0xff,
- NPC_IPNH_IGMP,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_PPP,
+ 0xffff,
+ NPC_GRE_F_KEY | NPC_GRE_F_ACK | NPC_GRE_VER_1,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU12_TU_IP, 0xff,
- NPC_IPNH_ESP,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ NPC_S_KPU8_GRE, 0xff,
+ NPC_ETYPE_PPP,
+ 0xffff,
+ NPC_GRE_F_KEY | NPC_GRE_F_SEQ | NPC_GRE_F_ACK | NPC_GRE_VER_1,
+ 0xffff,
0x0000,
0x0000,
},
{
- NPC_S_KPU12_TU_IP, 0xff,
- NPC_IPNH_AH,
- 0x00ff,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ NPC_S_KPU8_GRE, 0xff,
+ 0x0000,
+ 0xffff,
+ 0x2001,
+ 0xef7f,
0x0000,
0x0000,
},
{
- NPC_S_KPU12_TU_IP, 0xff,
- 0x0000,
+ NPC_S_KPU8_GRE, 0xff,
0x0000,
- NPC_IP_VER_4|NPC_IP_HDR_LEN_5,
- NPC_IP_VER_MASK|NPC_IP_HDR_LEN_MASK,
+ 0xffff,
+ 0x0001,
+ 0x0003,
0x0000,
0x0000,
},
{
- NPC_S_KPU12_TU_IP, 0xff,
- NPC_IPNH_TCP,
- 0x00ff,
- NPC_IP_VER_4,
- NPC_IP_VER_MASK,
+ NPC_S_NA, 0X00,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU12_TU_IP, 0xff,
- NPC_IPNH_UDP,
- 0x00ff,
- NPC_IP_VER_4,
- NPC_IP_VER_MASK,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU12_TU_IP, 0xff,
- NPC_IPNH_SCTP,
- 0x00ff,
- NPC_IP_VER_4,
- NPC_IP_VER_MASK,
0x0000,
0x0000,
},
+};
+
+static struct npc_kpu_profile_cam kpu9_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
{
- NPC_S_KPU12_TU_IP, 0xff,
- NPC_IPNH_ICMP,
- 0x00ff,
- NPC_IP_VER_4,
- NPC_IP_VER_MASK,
+ NPC_S_KPU9_TU_MPLS_IN_GRE, 0xff,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU12_TU_IP, 0xff,
- NPC_IPNH_IGMP,
- 0x00ff,
- NPC_IP_VER_4,
- NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU12_TU_IP, 0xff,
- NPC_IPNH_ESP,
- 0x00ff,
- NPC_IP_VER_4,
- NPC_IP_VER_MASK,
+ NPC_S_KPU9_TU_MPLS_IN_GRE, 0xff,
+ 0x0000,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
0x0000,
0x0000,
},
{
- NPC_S_KPU12_TU_IP, 0xff,
- NPC_IPNH_AH,
- 0x00ff,
- NPC_IP_VER_4,
- NPC_IP_VER_MASK,
+ NPC_S_KPU9_TU_MPLS_IN_GRE, 0xff,
0x0000,
+ NPC_MPLS_S,
0x0000,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
},
{
- NPC_S_KPU12_TU_IP, 0xff,
- 0x0000,
+ NPC_S_KPU9_TU_MPLS_IN_GRE, 0xff,
0x0000,
- NPC_IP_VER_4,
- NPC_IP_VER_MASK,
+ NPC_MPLS_S,
0x0000,
+ NPC_MPLS_S,
0x0000,
+ NPC_MPLS_S,
},
{
- NPC_S_KPU12_TU_IP, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU9_TU_MPLS_IN_NSH, 0xff,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU12_TU_ARP, 0xff,
- 0x0000,
- 0x0000,
- 0x0000,
+ NPC_S_KPU9_TU_MPLS_IN_NSH, 0xff,
0x0000,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
0x0000,
0x0000,
},
{
- NPC_S_KPU12_TU_IP6, 0xff,
- NPC_IPNH_TCP << 8,
- 0xff00,
- NPC_IP_VER_6,
- NPC_IP_VER_MASK,
+ NPC_S_KPU9_TU_MPLS_IN_NSH, 0xff,
0x0000,
+ NPC_MPLS_S,
0x0000,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
},
{
- NPC_S_KPU12_TU_IP6, 0xff,
- NPC_IPNH_UDP << 8,
- 0xff00,
- NPC_IP_VER_6,
- NPC_IP_VER_MASK,
+ NPC_S_KPU9_TU_MPLS_IN_NSH, 0xff,
0x0000,
+ NPC_MPLS_S,
+ 0x0000,
+ NPC_MPLS_S,
0x0000,
+ NPC_MPLS_S,
},
{
- NPC_S_KPU12_TU_IP6, 0xff,
- NPC_IPNH_SCTP << 8,
- 0xff00,
- NPC_IP_VER_6,
- NPC_IP_VER_MASK,
+ NPC_S_KPU9_TU_MPLS_IN_IP, 0xff,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU12_TU_IP6, 0xff,
- NPC_IPNH_ICMP << 8,
- 0xff00,
- NPC_IP_VER_6,
- NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU12_TU_IP6, 0xff,
- NPC_IPNH_ICMP6 << 8,
- 0xff00,
- NPC_IP_VER_6,
- NPC_IP_VER_MASK,
+ NPC_S_KPU9_TU_MPLS_IN_IP, 0xff,
+ 0x0000,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
0x0000,
0x0000,
},
{
- NPC_S_KPU12_TU_IP6, 0xff,
- NPC_IPNH_ESP << 8,
- 0xff00,
- NPC_IP_VER_6,
- NPC_IP_VER_MASK,
+ NPC_S_KPU9_TU_MPLS_IN_IP, 0xff,
0x0000,
+ NPC_MPLS_S,
0x0000,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
},
{
- NPC_S_KPU12_TU_IP6, 0xff,
- NPC_IPNH_AH << 8,
- 0xff00,
- NPC_IP_VER_6,
- NPC_IP_VER_MASK,
+ NPC_S_KPU9_TU_MPLS_IN_IP, 0xff,
+ 0x0000,
+ NPC_MPLS_S,
0x0000,
+ NPC_MPLS_S,
0x0000,
+ NPC_MPLS_S,
},
{
- NPC_S_KPU12_TU_IP6, 0xff,
+ NPC_S_KPU9_TU_NSH_IN_GRE, 0xff,
+ NPC_NSH_NP_IP,
+ NPC_NSH_NP_MASK,
0x0000,
0x0000,
- NPC_IP_VER_6,
- NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU12_TU_IP6, 0xff,
+ NPC_S_KPU9_TU_NSH_IN_GRE, 0xff,
+ NPC_NSH_NP_IP6,
+ NPC_NSH_NP_MASK,
+ 0x0000,
0x0000,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU9_TU_NSH_IN_GRE, 0xff,
+ NPC_NSH_NP_ETH,
+ NPC_NSH_NP_MASK,
+ 0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_NA, 0X00,
+ NPC_S_KPU9_TU_NSH_IN_GRE, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
0x0000,
},
-};
-
-static const struct npc_kpu_profile_cam kpu13_cam_entries[] = {
{
- NPC_S_KPU13_TU_IP6_EXT, 0xff,
+ NPC_S_KPU9_VXLAN, 0xff,
0x0000,
0x0000,
+ NPC_VXLAN_I,
+ NPC_VXLAN_I,
+ 0x0000,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU9_VXLAN, 0xff,
0x0000,
0x0000,
0x0000,
+ 0xffff,
0x0000,
+ 0xffff,
},
-};
-
-static const struct npc_kpu_profile_cam kpu14_cam_entries[] = {
{
- NPC_S_KPU14_TU_IP6_EXT, 0xff,
+ NPC_S_KPU9_VXLAN, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
0x0000,
},
-};
-
-static const struct npc_kpu_profile_cam kpu15_cam_entries[] = {
{
- NPC_S_KPU15_TU_TCP, 0xff,
+ NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
- NPC_TCP_FLAGS_FIN,
- NPC_TCP_FLAGS_MASK,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_NP_IP,
+ NPC_VXLANGPE_NP_MASK,
+ },
+ {
+ NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_NP_IP6,
+ NPC_VXLANGPE_NP_MASK,
},
{
- NPC_S_KPU15_TU_TCP, 0xff,
- 0x0000,
+ NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
- NPC_TCP_FLAGS_MASK,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_NP_ETH,
+ NPC_VXLANGPE_NP_MASK,
+ },
+ {
+ NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_NP_NSH,
+ NPC_VXLANGPE_NP_MASK,
},
{
- NPC_S_KPU15_TU_TCP, 0xff,
+ NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
- NPC_TCP_FLAGS_RST|NPC_TCP_FLAGS_FIN,
- NPC_TCP_FLAGS_RST|NPC_TCP_FLAGS_FIN,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_NP_MPLS,
+ NPC_VXLANGPE_NP_MASK,
+ },
+ {
+ NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
+ NPC_VXLANGPE_P,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_NP_IP,
+ NPC_VXLANGPE_NP_MASK,
},
{
- NPC_S_KPU15_TU_TCP, 0xff,
+ NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
- NPC_TCP_FLAGS_URG|NPC_TCP_FLAGS_SYN,
- NPC_TCP_FLAGS_URG|NPC_TCP_FLAGS_SYN,
+ NPC_VXLANGPE_P,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_NP_IP6,
+ NPC_VXLANGPE_NP_MASK,
+ },
+ {
+ NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
+ NPC_VXLANGPE_P,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_NP_ETH,
+ NPC_VXLANGPE_NP_MASK,
},
{
- NPC_S_KPU15_TU_TCP, 0xff,
+ NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
- NPC_TCP_FLAGS_RST|NPC_TCP_FLAGS_SYN,
- NPC_TCP_FLAGS_RST|NPC_TCP_FLAGS_SYN,
+ NPC_VXLANGPE_P,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_NP_NSH,
+ NPC_VXLANGPE_NP_MASK,
+ },
+ {
+ NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
+ NPC_VXLANGPE_P,
+ NPC_VXLANGPE_P | NPC_VXLANGPE_I,
+ NPC_VXLANGPE_NP_MPLS,
+ NPC_VXLANGPE_NP_MASK,
},
{
- NPC_S_KPU15_TU_TCP, 0xff,
+ NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
- NPC_TCP_FLAGS_SYN|NPC_TCP_FLAGS_FIN,
- NPC_TCP_FLAGS_SYN|NPC_TCP_FLAGS_FIN,
+ NPC_VXLANGPE_P,
+ NPC_VXLANGPE_P,
0x0000,
0x0000,
},
{
- NPC_S_KPU15_TU_TCP, 0xff,
- NPC_TCP_PORT_HTTP,
- 0xffff,
- NPC_TCP_DATA_OFFSET_5,
- NPC_TCP_DATA_OFFSET_MASK,
+ NPC_S_KPU9_VXLANGPE, 0xff,
0x0000,
0x0000,
- },
- {
- NPC_S_KPU15_TU_TCP, 0xff,
- NPC_TCP_PORT_HTTPS,
- 0xffff,
- NPC_TCP_DATA_OFFSET_5,
- NPC_TCP_DATA_OFFSET_MASK,
+ 0x0000,
+ NPC_VXLANGPE_P,
0x0000,
0x0000,
},
{
- NPC_S_KPU15_TU_TCP, 0xff,
- NPC_TCP_PORT_PPTP,
- 0xffff,
- NPC_TCP_DATA_OFFSET_5,
- NPC_TCP_DATA_OFFSET_MASK,
+ NPC_S_KPU9_GENEVE, 0xff,
+ 0x0000,
0x0000,
0x0000,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_ETYPE_TRANS_ETH_BR,
+ 0xffff,
},
{
- NPC_S_KPU15_TU_TCP, 0xff,
- 0x0000,
- 0x0000,
- NPC_TCP_DATA_OFFSET_5,
- NPC_TCP_DATA_OFFSET_MASK,
+ NPC_S_KPU9_GENEVE, 0xff,
0x0000,
0x0000,
+ NPC_GENEVE_F_OAM,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_ETYPE_TRANS_ETH_BR,
+ 0xffff,
},
{
- NPC_S_KPU15_TU_TCP, 0xff,
- NPC_TCP_PORT_HTTP,
- 0xffff,
+ NPC_S_KPU9_GENEVE, 0xff,
0x0000,
0x0000,
+ NPC_GENEVE_F_CRI_OPT,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_ETYPE_TRANS_ETH_BR,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU9_GENEVE, 0xff,
0x0000,
0x0000,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_ETYPE_TRANS_ETH_BR,
+ 0xffff,
},
{
- NPC_S_KPU15_TU_TCP, 0xff,
- NPC_TCP_PORT_HTTPS,
- 0xffff,
- 0x0000,
+ NPC_S_KPU9_GENEVE, 0xff,
0x0000,
0x0000,
0x0000,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_ETYPE_IP,
+ 0xffff,
},
{
- NPC_S_KPU15_TU_TCP, 0xff,
- NPC_TCP_PORT_PPTP,
- 0xffff,
+ NPC_S_KPU9_GENEVE, 0xff,
0x0000,
0x0000,
+ NPC_GENEVE_F_OAM,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_ETYPE_IP,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU9_GENEVE, 0xff,
0x0000,
0x0000,
+ NPC_GENEVE_F_CRI_OPT,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_ETYPE_IP,
+ 0xffff,
},
{
- NPC_S_KPU15_TU_TCP, 0xff,
- 0x0000,
+ NPC_S_KPU9_GENEVE, 0xff,
0x0000,
0x0000,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_ETYPE_IP,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU9_GENEVE, 0xff,
0x0000,
0x0000,
0x0000,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_ETYPE_IP6,
+ 0xffff,
},
{
- NPC_S_KPU15_TU_UDP, 0xff,
+ NPC_S_KPU9_GENEVE, 0xff,
0x0000,
0x0000,
+ NPC_GENEVE_F_OAM,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU9_GENEVE, 0xff,
0x0000,
0x0000,
+ NPC_GENEVE_F_CRI_OPT,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU9_GENEVE, 0xff,
0x0000,
0x0000,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_GENEVE_F_OAM | NPC_GENEVE_F_CRI_OPT,
+ NPC_ETYPE_IP6,
+ 0xffff,
},
{
- NPC_S_KPU15_TU_SCTP, 0xff,
+ NPC_S_KPU9_GTPC, 0xff,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU15_TU_ICMP, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU9_GTPU, 0xff,
0x0000,
0x0000,
+ NPC_GTP_PT_GTP | NPC_GTP_VER1 | NPC_GTP_MT_G_PDU,
+ NPC_GTP_PT_MASK | NPC_GTP_VER_MASK | NPC_GTP_MT_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU15_TU_IGMP, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU9_GTPU, 0xff,
0x0000,
0x0000,
+ NPC_GTP_PT_GTP | NPC_GTP_VER1,
+ NPC_GTP_PT_MASK | NPC_GTP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU15_TU_ICMP6, 0xff,
+ NPC_S_KPU9_TU_MPLS_IN_UDP, 0xff,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
0x0000,
0x0000,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU9_TU_MPLS_IN_UDP, 0xff,
+ 0x0000,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
0x0000,
0x0000,
},
{
- NPC_S_KPU15_TU_ESP, 0xff,
- 0x0000,
+ NPC_S_KPU9_TU_MPLS_IN_UDP, 0xff,
0x0000,
+ NPC_MPLS_S,
0x0000,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ },
+ {
+ NPC_S_KPU9_TU_MPLS_IN_UDP, 0xff,
0x0000,
+ NPC_MPLS_S,
0x0000,
+ NPC_MPLS_S,
0x0000,
+ NPC_MPLS_S,
},
{
- NPC_S_KPU15_TU_AH, 0xff,
+ NPC_S_KPU9_ESP, 0xff,
0x0000,
0x0000,
0x0000,
},
};
-static const struct npc_kpu_profile_cam kpu16_cam_entries[] = {
+static struct npc_kpu_profile_cam kpu10_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
{
- NPC_S_KPU16_TCP_DATA, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU10_TU_MPLS, 0xff,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU10_TU_MPLS, 0xff,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
},
{
- NPC_S_KPU16_HTTP_DATA, 0xff,
+ NPC_S_KPU10_TU_MPLS, 0xff,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ 0x0000,
+ 0xffff,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU10_TU_MPLS, 0xff,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU16_HTTPS_DATA, 0xff,
- 0x0000,
+ NPC_S_KPU10_TU_MPLS, 0xff,
0x0000,
+ NPC_MPLS_S,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU16_PPTP_DATA, 0xff,
- 0x0000,
- 0x0000,
+ NPC_S_KPU10_TU_MPLS_PL, 0xff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
0x0000,
0x0000,
},
{
- NPC_S_KPU16_UDP_DATA, 0xff,
+ NPC_S_KPU10_TU_MPLS_PL, 0xff,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
0x0000,
0x0000,
0x0000,
0x0000,
+ },
+ {
+ NPC_S_KPU10_TU_MPLS_PL, 0xff,
+ 0x0000,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU10_TU_MPLS_PL, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU10_TU_MPLS_IN_VXLANGPE, 0xff,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU10_TU_MPLS_IN_VXLANGPE, 0xff,
+ 0x0000,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU10_TU_MPLS_IN_VXLANGPE, 0xff,
+ 0x0000,
+ NPC_MPLS_S,
+ 0x0000,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ },
+ {
+ NPC_S_KPU10_TU_MPLS_IN_VXLANGPE, 0xff,
+ 0x0000,
+ NPC_MPLS_S,
+ 0x0000,
+ NPC_MPLS_S,
+ 0x0000,
+ NPC_MPLS_S,
+ },
+ {
+ NPC_S_KPU10_TU_NSH_IN_VXLANGPE, 0xff,
+ NPC_NSH_NP_IP,
+ NPC_NSH_NP_MASK,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU10_TU_NSH_IN_VXLANGPE, 0xff,
+ NPC_NSH_NP_IP6,
+ NPC_NSH_NP_MASK,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU10_TU_NSH_IN_VXLANGPE, 0xff,
+ NPC_NSH_NP_ETH,
+ NPC_NSH_NP_MASK,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU10_TU_NSH_IN_VXLANGPE, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_NA, 0X00,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+};
+
+static struct npc_kpu_profile_cam kpu11_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_ARP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_ARP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_SBTAG,
+ 0xffff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_SBTAG,
+ 0xffff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_SBTAG,
+ 0xffff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_ARP,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_SBTAG,
+ 0xffff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_SBTAG,
+ 0xffff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_SBTAG,
+ 0xffff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_SBTAG,
+ 0xffff,
+ NPC_ETYPE_ARP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_SBTAG,
+ 0xffff,
0x0000,
0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_QINQ,
+ 0xffff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_QINQ,
+ 0xffff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_QINQ,
+ 0xffff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ NPC_ETYPE_ARP,
+ 0xffff,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_QINQ,
+ 0xffff,
+ NPC_ETYPE_CTAG,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_QINQ,
+ 0xffff,
+ NPC_ETYPE_IP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_QINQ,
+ 0xffff,
+ NPC_ETYPE_IP6,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_QINQ,
+ 0xffff,
+ NPC_ETYPE_ARP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ NPC_ETYPE_QINQ,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_PPP, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_MPLS, 0xff,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_MPLS, 0xff,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_MPLS, 0xff,
+ NPC_MPLS_S,
+ NPC_MPLS_S,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_MPLS, 0xff,
+ 0x0000,
+ NPC_MPLS_S,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_MPLS_PL, 0xff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_MPLS_PL, 0xff,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_MPLS_PL, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU11_TU_ETHER_IN_NSH, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_NA, 0X00,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+};
+
+static struct npc_kpu_profile_cam kpu12_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ NPC_IPNH_TCP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ NPC_IPNH_UDP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ NPC_IPNH_SCTP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ NPC_IPNH_ICMP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ NPC_IPNH_IGMP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ NPC_IPNH_ESP,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ NPC_IPNH_AH,
+ 0x00ff,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ 0x0000,
+ 0x0000,
+ NPC_IP_VER_4 | NPC_IP_HDR_LEN_5,
+ NPC_IP_VER_MASK | NPC_IP_HDR_LEN_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ NPC_IPNH_TCP,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ NPC_IPNH_UDP,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ NPC_IPNH_SCTP,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ NPC_IPNH_ICMP,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ NPC_IPNH_IGMP,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ NPC_IPNH_ESP,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ NPC_IPNH_AH,
+ 0x00ff,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ 0x0000,
+ 0x0000,
+ NPC_IP_VER_4,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_ARP, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP6, 0xff,
+ NPC_IPNH_TCP << 8,
+ 0xff00,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP6, 0xff,
+ NPC_IPNH_UDP << 8,
+ 0xff00,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP6, 0xff,
+ NPC_IPNH_SCTP << 8,
+ 0xff00,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP6, 0xff,
+ NPC_IPNH_ICMP << 8,
+ 0xff00,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP6, 0xff,
+ NPC_IPNH_ICMP6 << 8,
+ 0xff00,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP6, 0xff,
+ NPC_IPNH_ESP << 8,
+ 0xff00,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP6, 0xff,
+ NPC_IPNH_AH << 8,
+ 0xff00,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP6, 0xff,
+ 0x0000,
+ 0x0000,
+ NPC_IP_VER_6,
+ NPC_IP_VER_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU12_TU_IP6, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_NA, 0X00,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+};
+
+static struct npc_kpu_profile_cam kpu13_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
+ {
+ NPC_S_KPU13_TU_IP6_EXT, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+};
+
+static struct npc_kpu_profile_cam kpu14_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
+ {
+ NPC_S_KPU14_TU_IP6_EXT, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+};
+
+static struct npc_kpu_profile_cam kpu15_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
+ {
+ NPC_S_KPU15_TU_TCP, 0xff,
+ 0x0000,
+ 0x0000,
+ NPC_TCP_FLAGS_FIN,
+ NPC_TCP_FLAGS_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_TCP, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ NPC_TCP_FLAGS_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_TCP, 0xff,
+ 0x0000,
+ 0x0000,
+ NPC_TCP_FLAGS_RST | NPC_TCP_FLAGS_FIN,
+ NPC_TCP_FLAGS_RST | NPC_TCP_FLAGS_FIN,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_TCP, 0xff,
+ 0x0000,
+ 0x0000,
+ NPC_TCP_FLAGS_URG | NPC_TCP_FLAGS_SYN,
+ NPC_TCP_FLAGS_URG | NPC_TCP_FLAGS_SYN,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_TCP, 0xff,
+ 0x0000,
+ 0x0000,
+ NPC_TCP_FLAGS_RST | NPC_TCP_FLAGS_SYN,
+ NPC_TCP_FLAGS_RST | NPC_TCP_FLAGS_SYN,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_TCP, 0xff,
+ 0x0000,
+ 0x0000,
+ NPC_TCP_FLAGS_SYN | NPC_TCP_FLAGS_FIN,
+ NPC_TCP_FLAGS_SYN | NPC_TCP_FLAGS_FIN,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_TCP, 0xff,
+ NPC_TCP_PORT_HTTP,
+ 0xffff,
+ NPC_TCP_DATA_OFFSET_5,
+ NPC_TCP_DATA_OFFSET_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_TCP, 0xff,
+ NPC_TCP_PORT_HTTPS,
+ 0xffff,
+ NPC_TCP_DATA_OFFSET_5,
+ NPC_TCP_DATA_OFFSET_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_TCP, 0xff,
+ NPC_TCP_PORT_PPTP,
+ 0xffff,
+ NPC_TCP_DATA_OFFSET_5,
+ NPC_TCP_DATA_OFFSET_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_TCP, 0xff,
+ 0x0000,
+ 0x0000,
+ NPC_TCP_DATA_OFFSET_5,
+ NPC_TCP_DATA_OFFSET_MASK,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_TCP, 0xff,
+ NPC_TCP_PORT_HTTP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_TCP, 0xff,
+ NPC_TCP_PORT_HTTPS,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_TCP, 0xff,
+ NPC_TCP_PORT_PPTP,
+ 0xffff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_TCP, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_UDP, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_SCTP, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_ICMP, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_IGMP, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_ICMP6, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_ESP, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU15_TU_AH, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_NA, 0X00,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+};
+
+static struct npc_kpu_profile_cam kpu16_cam_entries[] = {
+ NPC_KPU_NOP_CAM,
+ NPC_KPU_NOP_CAM,
+ {
+ NPC_S_KPU16_TCP_DATA, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU16_HTTP_DATA, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU16_HTTPS_DATA, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU16_PPTP_DATA, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU16_UDP_DATA, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+ {
+ NPC_S_KPU16_UDP_PTP, 0xff,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ 0x0000,
+ },
+};
+
+static struct npc_kpu_profile_action kpu1_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 0, 6, 3, 0,
+ NPC_S_KPU5_IP, 14, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 6, 0, 0, 3, 0,
+ NPC_S_KPU5_IP6, 14, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_ARP, 14, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_RARP, 14, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_PTP, 14, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_FCOE, 14, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 12, 0, 0, 0,
+ NPC_S_KPU2_NGIO, 12, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 12, 0, 0, 0,
+ NPC_S_KPU2_CTAG2, 12, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 12, 0, 0,
+ NPC_S_KPU2_CTAG, 12, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 22, 0, 0,
+ NPC_S_KPU2_SBTAG, 12, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU2_QINQ, 12, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 12, 26, 0, 0,
+ NPC_S_KPU2_ETAG, 12, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_ETAG,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 2, 0,
+ NPC_S_KPU4_MPLS, 14, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_F_LA_L_WITH_MPLS,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 2, 0,
+ NPC_S_KPU4_MPLS, 14, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_F_LA_L_WITH_MPLS,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 0, 0, 2, 0,
+ NPC_S_KPU4_NSH, 14, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_F_LA_L_WITH_NSH,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 12, 0, 1, 0,
+ NPC_S_KPU3_DSA, 12, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 12, 0, 2, 0,
+ NPC_S_KPU4_PPPOE, 12, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LA, NPC_LT_LA_8023,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LA, NPC_LT_LA_8023,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_F_LA_L_UNK_ETYPE,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 0, 6, 3, 0,
+ NPC_S_KPU5_IP, 22, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 6, 0, 0, 3, 0,
+ NPC_S_KPU5_IP6, 22, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_ARP, 22, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_RARP, 22, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_PTP, 22, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_FCOE, 22, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 12, 0, 0, 0,
+ NPC_S_KPU2_CTAG2, 20, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_TAG
+ | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU2_CTAG, 20, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_TAG
+ | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 22, 0, 0,
+ NPC_S_KPU2_SBTAG, 20, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_TAG
+ | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU2_QINQ, 20, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_TAG
+ | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 12, 26, 0, 0,
+ NPC_S_KPU2_ETAG, 20, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_TAG
+ | NPC_F_LA_L_WITH_ETAG,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 2, 0,
+ NPC_S_KPU4_MPLS, 22, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_L_WITH_MPLS,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 2, 0,
+ NPC_S_KPU4_MPLS, 22, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_L_WITH_MPLS,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 0, 0, 2, 0,
+ NPC_S_KPU4_NSH, 22, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_L_WITH_NSH,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_L_UNK_ETYPE,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 12, 14, 16, 0, 0,
+ NPC_S_KPU2_PREHEADER, 8, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_8_ETHER,
+ 0,
+ 1, 0xff, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 12, 14, 16, 0, 0,
+ NPC_S_KPU2_PREHEADER, 4, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_4_ETHER,
+ 0,
+ 1, 0xff, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 12, 14, 16, 0, 0,
+ NPC_S_KPU2_PREHEADER, 2, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_2_ETHER,
+ 0,
+ 1, 0xff, 0, 0,
+ },
+ {
+ NPC_ERRLEV_LA, NPC_EC_IH_LENGTH,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_F_LA_L_UNK_ETYPE,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 16, 0, 0,
+ NPC_S_KPU2_EXDSA, 12, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 12, 2, 0,
+ NPC_S_KPU4_FDSA, 12, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_LA, NPC_EC_EDSA_UNK,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LA, NPC_LT_LA_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 0, 6, 3, 0,
+ NPC_S_KPU5_IP, 30, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 6, 0, 0, 3, 0,
+ NPC_S_KPU5_IP6, 30, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_ARP, 30, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_RARP, 30, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_PTP, 30, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_FCOE, 30, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 12, 0, 0, 0,
+ NPC_S_KPU2_CTAG2, 28, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_U_HAS_TAG
+ | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU2_CTAG, 28, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_U_HAS_TAG
+ | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 22, 0, 0,
+ NPC_S_KPU2_SBTAG, 28, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_U_HAS_TAG
+ | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU2_QINQ, 28, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_U_HAS_TAG
+ | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 12, 26, 0, 0,
+ NPC_S_KPU2_ETAG, 28, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_U_HAS_TAG
+ | NPC_F_LA_L_WITH_ETAG,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 2, 0,
+ NPC_S_KPU4_MPLS, 30, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_L_WITH_MPLS,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 2, 0,
+ NPC_S_KPU4_MPLS, 30, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_L_WITH_MPLS,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 0, 0, 2, 0,
+ NPC_S_KPU4_NSH, 30, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_L_WITH_NSH,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_L_UNK_ETYPE,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 0, 6, 3, 0,
+ NPC_S_KPU5_IP, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 6, 0, 0, 3, 0,
+ NPC_S_KPU5_IP6, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_ARP, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_RARP, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_PTP, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_FCOE, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 12, 0, 0, 0,
+ NPC_S_KPU2_CTAG2, 36, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
+ | NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU2_CTAG, 36, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
+ | NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 22, 0, 0,
+ NPC_S_KPU2_SBTAG, 36, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
+ | NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU2_QINQ, 36, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
+ | NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 12, 26, 0, 0,
+ NPC_S_KPU2_ETAG, 36, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
+ | NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_ETAG,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 2, 0,
+ NPC_S_KPU4_MPLS, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
+ | NPC_F_LA_L_WITH_MPLS,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 2, 0,
+ NPC_S_KPU4_MPLS, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
+ | NPC_F_LA_L_WITH_MPLS,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 0, 0, 2, 0,
+ NPC_S_KPU4_NSH, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
+ | NPC_F_LA_L_WITH_NSH,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
+ NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
+ | NPC_F_LA_L_UNK_ETYPE,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 0, 6, 3, 0,
+ NPC_S_KPU5_IP, 104, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 6, 0, 0, 3, 0,
+ NPC_S_KPU5_IP6, 104, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_ARP, 104, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_RARP, 104, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_PTP, 104, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU5_FCOE, 104, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 12, 0, 0, 0,
+ NPC_S_KPU2_CTAG2, 102, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU2_CTAG, 102, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 22, 0, 0,
+ NPC_S_KPU2_SBTAG, 102, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU2_QINQ, 102, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 12, 26, 0, 0,
+ NPC_S_KPU2_ETAG, 102, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_ETAG,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 2, 0,
+ NPC_S_KPU4_MPLS, 104, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ NPC_F_LA_L_WITH_MPLS,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 2, 0,
+ NPC_S_KPU4_MPLS, 104, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ NPC_F_LA_L_WITH_MPLS,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 0, 0, 2, 0,
+ NPC_S_KPU4_NSH, 104, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ NPC_F_LA_L_WITH_NSH,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_90B_ETHER,
+ NPC_F_LA_L_UNK_ETYPE,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 0, 6, 3, 0,
+ NPC_S_KPU5_CPT_IP, 56, 1,
+ NPC_LID_LA, NPC_LT_LA_CPT_HDR,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 6, 0, 0, 3, 0,
+ NPC_S_KPU5_CPT_IP6, 56, 1,
+ NPC_LID_LA, NPC_LT_LA_CPT_HDR,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU2_CTAG, 54, 1,
+ NPC_LID_LA, NPC_LT_LA_CPT_HDR,
+ NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU2_QINQ, 54, 1,
+ NPC_LID_LA, NPC_LT_LA_CPT_HDR,
+ NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 0, 6, 3, 0,
+ NPC_S_KPU5_CPT_IP, 60, 1,
+ NPC_LID_LA, NPC_LT_LA_CPT_HDR,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 6, 0, 0, 3, 0,
+ NPC_S_KPU5_CPT_IP6, 60, 1,
+ NPC_LID_LA, NPC_LT_LA_CPT_HDR,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU2_CTAG, 58, 1,
+ NPC_LID_LA, NPC_LT_LA_CPT_HDR,
+ NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
},
{
- NPC_S_KPU16_UDP_PTP, 0xff,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
- 0x0000,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU2_QINQ, 58, 1,
+ NPC_LID_LA, NPC_LT_LA_CPT_HDR,
+ NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LA, NPC_LT_LA_CPT_HDR,
+ NPC_F_LA_L_UNK_ETYPE,
+ 0, 0, 0, 0,
},
-};
-
-static const struct npc_kpu_profile_action kpu1_action_entries[] = {
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
8, 0, 6, 3, 0,
- NPC_S_KPU5_IP, 14, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_S_KPU5_IP, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
6, 0, 0, 3, 0,
- NPC_S_KPU5_IP6, 14, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_S_KPU5_IP6, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 3, 0,
- NPC_S_KPU5_ARP, 14, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_S_KPU5_ARP, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 3, 0,
- NPC_S_KPU5_RARP, 14, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_S_KPU5_RARP, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 3, 0,
- NPC_S_KPU5_PTP, 14, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_S_KPU5_PTP, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 3, 0,
- NPC_S_KPU5_FCOE, 14, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_S_KPU5_FCOE, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
8, 12, 0, 0, 0,
- NPC_S_KPU2_CTAG2, 12, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_S_KPU2_CTAG2, 36, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
4, 8, 0, 0, 0,
- NPC_S_KPU2_CTAG, 12, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_S_KPU2_CTAG, 36, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
4, 8, 22, 0, 0,
- NPC_S_KPU2_SBTAG, 12, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_S_KPU2_SBTAG, 36, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
4, 8, 0, 0, 0,
- NPC_S_KPU2_QINQ, 12, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_S_KPU2_QINQ, 36, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
8, 12, 26, 0, 0,
- NPC_S_KPU2_ETAG, 12, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_S_KPU2_ETAG, 36, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_ETAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 18, 22, 26, 0, 0,
- NPC_S_KPU2_ITAG, 12, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
- NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_ITAG,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_RE, NPC_EC_NOERR,
2, 6, 10, 2, 0,
- NPC_S_KPU4_MPLS, 14, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_S_KPU4_MPLS, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
NPC_F_LA_L_WITH_MPLS,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
2, 6, 10, 2, 0,
- NPC_S_KPU4_MPLS, 14, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_S_KPU4_MPLS, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
NPC_F_LA_L_WITH_MPLS,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
2, 0, 0, 2, 0,
- NPC_S_KPU4_NSH, 14, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
+ NPC_S_KPU4_NSH, 38, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
NPC_F_LA_L_WITH_NSH,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 12, 0, 1, 0,
- NPC_S_KPU3_DSA, 12, 1,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LA, NPC_LT_LA_CUSTOM_L2_24B_ETHER,
+ NPC_F_LA_L_UNK_ETYPE,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 12, 0, 0, 1, 0,
+ NPC_S_KPU3_VLAN_EXDSA, 12, 1,
NPC_LID_LA, NPC_LT_LA_ETHER,
0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
+ NPC_ERRLEV_LA, NPC_EC_L2_K1,
0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LA, NPC_LT_LA_8023,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LA, NPC_LT_NA,
0,
0, 0, 0, 0,
},
+};
+
+static struct npc_kpu_profile_action kpu2_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LA, NPC_LT_LA_8023,
+ 8, 0, 6, 2, 0,
+ NPC_S_KPU5_IP, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
- NPC_F_LA_L_UNK_ETYPE,
+ 6, 0, 0, 2, 0,
+ NPC_S_KPU5_IP6, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 3, 0,
- NPC_S_KPU5_IP, 22, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_ARP, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 3, 0,
- NPC_S_KPU5_IP6, 22, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_RARP, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 3, 0,
- NPC_S_KPU5_ARP, 22, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_PTP, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 3, 0,
- NPC_S_KPU5_RARP, 22, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_FCOE, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 3, 0,
- NPC_S_KPU5_PTP, 22, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX,
+ 2, 6, 10, 1, 0,
+ NPC_S_KPU4_MPLS, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 3, 0,
- NPC_S_KPU5_FCOE, 22, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX,
+ 2, 6, 10, 1, 0,
+ NPC_S_KPU4_MPLS, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 12, 0, 0, 0,
- NPC_S_KPU2_CTAG2, 20, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_TAG
- | NPC_F_LA_L_WITH_VLAN,
+ 2, 0, 0, 1, 0,
+ NPC_S_KPU4_NSH, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 4, 8, 0, 0, 0,
- NPC_S_KPU2_CTAG, 20, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_TAG
- | NPC_F_LA_L_WITH_VLAN,
+ 8, 0, 6, 2, 0,
+ NPC_S_KPU5_IP, 14, 1,
+ NPC_LID_LB, NPC_LT_LB_PPPOE,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 4, 8, 22, 0, 0,
- NPC_S_KPU2_SBTAG, 20, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_TAG
- | NPC_F_LA_L_WITH_VLAN,
+ 6, 0, 0, 2, 0,
+ NPC_S_KPU5_IP6, 14, 1,
+ NPC_LID_LB, NPC_LT_LB_PPPOE,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 4, 8, 0, 0, 0,
- NPC_S_KPU2_QINQ, 20, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_TAG
- | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ NPC_F_LB_U_UNK_ETYPE,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 12, 26, 0, 0,
- NPC_S_KPU2_ETAG, 20, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_TAG
- | NPC_F_LA_L_WITH_ETAG,
+ 8, 0, 6, 2, 0,
+ NPC_S_KPU5_IP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 18, 22, 26, 0, 0,
- NPC_S_KPU2_ITAG, 20, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_TAG
- | NPC_F_LA_L_WITH_ITAG,
+ 6, 0, 0, 2, 0,
+ NPC_S_KPU5_IP6, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 2, 0,
- NPC_S_KPU4_MPLS, 22, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_L_WITH_MPLS,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_ARP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 2, 0,
- NPC_S_KPU4_MPLS, 22, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_L_WITH_MPLS,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_RARP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 2, 0,
- NPC_S_KPU4_NSH, 22, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_L_WITH_NSH,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_PTP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_L_UNK_ETYPE,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_FCOE, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 12, 14, 16, 0, 0,
- NPC_S_KPU2_PREHEADER, 8, 1,
- NPC_LID_LA, NPC_LT_LA_IH_8_ETHER,
- 0,
- 1, 0xff, 0, 0,
+ 2, 6, 10, 1, 0,
+ NPC_S_KPU4_MPLS, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
+ 0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 12, 14, 16, 0, 0,
- NPC_S_KPU2_PREHEADER, 4, 1,
- NPC_LID_LA, NPC_LT_LA_IH_4_ETHER,
- 0,
- 1, 0xff, 0, 0,
+ 2, 6, 10, 1, 0,
+ NPC_S_KPU4_MPLS, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
+ 0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 12, 14, 16, 0, 0,
- NPC_S_KPU2_PREHEADER, 2, 1,
- NPC_LID_LA, NPC_LT_LA_IH_2_ETHER,
- 0,
- 1, 0xff, 0, 0,
+ 2, 0, 0, 1, 0,
+ NPC_S_KPU4_NSH, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
+ 0, 0, 0, 0,
},
{
- NPC_ERRLEV_LA, NPC_EC_IH_LENGTH,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 0, 1,
NPC_S_NA, 0, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
- NPC_F_LA_L_UNK_ETYPE,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG_UNK,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 4, 8, 16, 0, 0,
- NPC_S_KPU2_EXDSA, 12, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
- 0,
+ 2, 6, 0, 0, 0,
+ NPC_S_KPU3_CTAG, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_STAG_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 4, 8, 16, 2, 0,
- NPC_S_KPU4_FDSA, 12, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
- 0,
+ 2, 6, 0, 0, 0,
+ NPC_S_KPU3_STAG, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_STAG_STAG,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_LA, NPC_EC_EDSA_UNK,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LA, NPC_LT_LA_ETHER,
- 0,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 0, 6, 2, 0,
+ NPC_S_KPU5_IP, 24, 1,
+ NPC_LID_LB, NPC_LT_LB_BTAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 3, 0,
- NPC_S_KPU5_IP, 30, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2,
+ 6, 0, 0, 2, 0,
+ NPC_S_KPU5_IP6, 24, 1,
+ NPC_LID_LB, NPC_LT_LB_BTAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 3, 0,
- NPC_S_KPU5_IP6, 30, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_ARP, 24, 1,
+ NPC_LID_LB, NPC_LT_LB_BTAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 3, 0,
- NPC_S_KPU5_ARP, 30, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_RARP, 24, 1,
+ NPC_LID_LB, NPC_LT_LB_BTAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 3, 0,
- NPC_S_KPU5_RARP, 30, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_PTP, 24, 1,
+ NPC_LID_LB, NPC_LT_LB_BTAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 3, 0,
- NPC_S_KPU5_PTP, 30, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_FCOE, 24, 1,
+ NPC_LID_LB, NPC_LT_LB_BTAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 3, 0,
- NPC_S_KPU5_FCOE, 30, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2,
+ 2, 6, 10, 1, 0,
+ NPC_S_KPU4_MPLS, 24, 1,
+ NPC_LID_LB, NPC_LT_LB_BTAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 12, 0, 0, 0,
- NPC_S_KPU2_CTAG2, 28, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_U_HAS_TAG
- | NPC_F_LA_L_WITH_VLAN,
+ 2, 6, 10, 1, 0,
+ NPC_S_KPU4_MPLS, 24, 1,
+ NPC_LID_LB, NPC_LT_LB_BTAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 4, 8, 0, 0, 0,
- NPC_S_KPU2_CTAG, 28, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_U_HAS_TAG
- | NPC_F_LA_L_WITH_VLAN,
+ 2, 0, 0, 1, 0,
+ NPC_S_KPU4_NSH, 24, 1,
+ NPC_LID_LB, NPC_LT_LB_BTAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 4, 8, 22, 0, 0,
- NPC_S_KPU2_SBTAG, 28, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_U_HAS_TAG
- | NPC_F_LA_L_WITH_VLAN,
+ 2, 0, 0, 0, 0,
+ NPC_S_KPU3_STAG, 24, 1,
+ NPC_LID_LB, NPC_LT_LB_BTAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG_STAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 4, 8, 0, 0, 0,
- NPC_S_KPU2_QINQ, 28, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_U_HAS_TAG
- | NPC_F_LA_L_WITH_VLAN,
+ 2, 0, 0, 0, 0,
+ NPC_S_KPU3_CTAG, 24, 1,
+ NPC_LID_LB, NPC_LT_LB_BTAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 12, 26, 0, 0,
- NPC_S_KPU2_ETAG, 28, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_U_HAS_TAG
- | NPC_F_LA_L_WITH_ETAG,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LB, NPC_LT_LB_BTAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG_UNK,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 18, 22, 26, 0, 0,
- NPC_S_KPU2_ITAG, 28, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_U_HAS_TAG
- | NPC_F_LA_L_WITH_ITAG,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_UNK_ETYPE,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 2, 0,
- NPC_S_KPU4_MPLS, 30, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_L_WITH_MPLS,
+ 8, 0, 6, 2, 0,
+ NPC_S_KPU5_IP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 2, 0,
- NPC_S_KPU4_MPLS, 30, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_L_WITH_MPLS,
+ 6, 0, 0, 2, 0,
+ NPC_S_KPU5_IP6, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 2, 0,
- NPC_S_KPU4_NSH, 30, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_L_WITH_NSH,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_ARP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LA, NPC_LT_LA_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_HIGIG2 | NPC_F_LA_L_UNK_ETYPE,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_RARP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 3, 0,
- NPC_S_KPU5_IP, 38, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_PTP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 3, 0,
- NPC_S_KPU5_IP6, 38, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_FCOE, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 3, 0,
- NPC_S_KPU5_ARP, 38, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2,
+ 2, 6, 10, 1, 0,
+ NPC_S_KPU4_MPLS, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 3, 0,
- NPC_S_KPU5_RARP, 38, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2,
+ 2, 6, 10, 1, 0,
+ NPC_S_KPU4_MPLS, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 3, 0,
- NPC_S_KPU5_PTP, 38, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2,
+ 2, 0, 0, 1, 0,
+ NPC_S_KPU4_NSH, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 3, 0,
- NPC_S_KPU5_FCOE, 38, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG_UNK,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 12, 0, 0, 0,
- NPC_S_KPU2_CTAG2, 36, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
- | NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 2, 6, 0, 0, 0,
+ NPC_S_KPU3_CTAG, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_QINQ_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 4, 8, 0, 0, 0,
- NPC_S_KPU2_CTAG, 36, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
- | NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 2, 6, 0, 0, 0,
+ NPC_S_KPU3_QINQ, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_QINQ_QINQ,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 4, 8, 22, 0, 0,
- NPC_S_KPU2_SBTAG, 36, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
- | NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_F_LB_U_UNK_ETYPE,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 4, 8, 0, 0, 0,
- NPC_S_KPU2_QINQ, 36, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
- | NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_VLAN,
+ 8, 0, 6, 2, 0,
+ NPC_S_KPU5_IP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 12, 26, 0, 0,
- NPC_S_KPU2_ETAG, 36, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
- | NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_ETAG,
+ 6, 0, 0, 2, 0,
+ NPC_S_KPU5_IP6, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 18, 22, 26, 0, 0,
- NPC_S_KPU2_ITAG, 36, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
- | NPC_F_LA_U_HAS_TAG | NPC_F_LA_L_WITH_ITAG,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_ARP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_RARP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 2, 0,
- NPC_S_KPU4_MPLS, 38, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
- | NPC_F_LA_L_WITH_MPLS,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_PTP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 2, 0,
- NPC_S_KPU4_MPLS, 38, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
- | NPC_F_LA_L_WITH_MPLS,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_FCOE, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 2, 0,
- NPC_S_KPU4_NSH, 38, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
- | NPC_F_LA_L_WITH_NSH,
+ 2, 6, 10, 1, 0,
+ NPC_S_KPU4_MPLS, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ 1,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LA, NPC_LT_LA_IH_NIX_HIGIG2_ETHER,
- NPC_F_LA_U_HAS_IH_NIX | NPC_F_LA_U_HAS_HIGIG2
- | NPC_F_LA_L_UNK_ETYPE,
+ 2, 6, 10, 1, 0,
+ NPC_S_KPU4_MPLS, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ 2,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_LA, NPC_EC_L2_K1,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
- NPC_LID_LA, NPC_LT_NA,
- 0,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 1, 0,
+ NPC_S_KPU4_NSH, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ 2,
0, 0, 0, 0,
},
-};
-
-static const struct npc_kpu_profile_action kpu2_action_entries[] = {
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 2, 0,
- NPC_S_KPU5_IP, 6, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
- 0,
+ 2, 0, 0, 0, 0,
+ NPC_S_KPU3_CTAG, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 2, 0,
- NPC_S_KPU5_IP6, 6, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
- 0,
+ 2, 6, 0, 0, 0,
+ NPC_S_KPU3_STAG, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_STAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_ARP, 6, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
- 0,
+ 2, 6, 0, 0, 0,
+ NPC_S_KPU3_QINQ, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_QINQ,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_RARP, 6, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
- 0,
+ 8, 0, 6, 2, 0,
+ NPC_S_KPU5_IP, 28, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_PTP, 6, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
- 0,
+ 6, 0, 0, 2, 0,
+ NPC_S_KPU5_IP6, 28, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 2, 0,
- NPC_S_KPU5_FCOE, 6, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
- 0,
+ NPC_S_KPU5_ARP, 28, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 1, 0,
- NPC_S_KPU4_MPLS, 6, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
- 0,
+ 2, 0, 0, 0, 0,
+ NPC_S_KPU3_STAG, 28, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG_STAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 1, 0,
- NPC_S_KPU4_MPLS, 6, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
- 0,
+ 2, 0, 0, 0, 0,
+ NPC_S_KPU3_CTAG, 28, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG_CTAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 1, 0,
- NPC_S_KPU4_NSH, 6, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
- 0,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_ITAG_UNK,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 0, 1,
NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
+ NPC_LID_LB, NPC_LT_LB_ETAG,
NPC_F_LB_U_UNK_ETYPE,
0, 0, 0, 0,
},
8, 0, 6, 2, 0,
NPC_S_KPU5_IP, 10, 1,
NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_CTAG,
+ 0,
0, 0, 0, 0,
},
{
6, 0, 0, 2, 0,
NPC_S_KPU5_IP6, 10, 1,
NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_CTAG,
+ 0,
0, 0, 0, 0,
},
{
0, 0, 0, 2, 0,
NPC_S_KPU5_ARP, 10, 1,
NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_CTAG,
+ 0,
0, 0, 0, 0,
},
{
0, 0, 0, 2, 0,
NPC_S_KPU5_RARP, 10, 1,
NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_CTAG,
+ 0,
0, 0, 0, 0,
},
{
0, 0, 0, 2, 0,
NPC_S_KPU5_PTP, 10, 1,
NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_CTAG,
+ 0,
0, 0, 0, 0,
},
{
0, 0, 0, 2, 0,
NPC_S_KPU5_FCOE, 10, 1,
NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_CTAG,
+ 0,
0, 0, 0, 0,
},
{
2, 6, 10, 1, 0,
NPC_S_KPU4_MPLS, 10, 1,
NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_CTAG,
+ 0,
0, 0, 0, 0,
},
{
2, 6, 10, 1, 0,
NPC_S_KPU4_MPLS, 10, 1,
NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_CTAG,
+ 0,
0, 0, 0, 0,
},
{
2, 0, 0, 1, 0,
NPC_S_KPU4_NSH, 10, 1,
NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_CTAG,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_CTAG_UNK,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
2, 6, 0, 0, 0,
- NPC_S_KPU3_CTAG, 10, 1,
+ NPC_S_KPU3_QINQ, 10, 1,
NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_STAG_CTAG,
+ NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_QINQ_QINQ,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 0, 0, 0,
- NPC_S_KPU3_STAG, 10, 1,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_STAG_STAG,
+ NPC_F_LB_U_UNK_ETYPE,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
8, 0, 6, 2, 0,
- NPC_S_KPU5_IP, 24, 1,
- NPC_LID_LB, NPC_LT_LB_BTAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG,
+ NPC_S_KPU5_IP, 14, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
6, 0, 0, 2, 0,
- NPC_S_KPU5_IP6, 24, 1,
- NPC_LID_LB, NPC_LT_LB_BTAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG,
+ NPC_S_KPU5_IP6, 14, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 2, 0,
- NPC_S_KPU5_ARP, 24, 1,
- NPC_LID_LB, NPC_LT_LB_BTAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG,
+ NPC_S_KPU5_ARP, 14, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 2, 0,
- NPC_S_KPU5_RARP, 24, 1,
- NPC_LID_LB, NPC_LT_LB_BTAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG,
+ NPC_S_KPU5_RARP, 14, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 2, 0,
- NPC_S_KPU5_PTP, 24, 1,
- NPC_LID_LB, NPC_LT_LB_BTAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG,
+ NPC_S_KPU5_PTP, 14, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 2, 0,
- NPC_S_KPU5_FCOE, 24, 1,
- NPC_LID_LB, NPC_LT_LB_BTAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 1, 0,
- NPC_S_KPU4_MPLS, 24, 1,
- NPC_LID_LB, NPC_LT_LB_BTAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 1, 0,
- NPC_S_KPU4_MPLS, 24, 1,
- NPC_LID_LB, NPC_LT_LB_BTAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 1, 0,
- NPC_S_KPU4_NSH, 24, 1,
- NPC_LID_LB, NPC_LT_LB_BTAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 0, 0,
- NPC_S_KPU3_STAG, 24, 1,
- NPC_LID_LB, NPC_LT_LB_BTAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG_STAG,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 0, 0,
- NPC_S_KPU3_CTAG, 24, 1,
- NPC_LID_LB, NPC_LT_LB_BTAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG_CTAG,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_BTAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG_UNK,
+ NPC_S_KPU5_FCOE, 14, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_UNK_ETYPE,
+ 2, 6, 0, 0, 0,
+ NPC_S_KPU3_CTAG_C, 14, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 2, 0,
- NPC_S_KPU5_IP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
+ 2, 6, 20, 0, 0,
+ NPC_S_KPU3_STAG_C, 14, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 2, 0,
- NPC_S_KPU5_IP6, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
+ 2, 6, 0, 0, 0,
+ NPC_S_KPU3_QINQ_C, 14, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_ARP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
+ 2, 6, 10, 1, 0,
+ NPC_S_KPU4_MPLS, 14, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_RARP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
+ 2, 6, 10, 1, 0,
+ NPC_S_KPU4_MPLS, 14, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_PTP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 0, 0, 1, 0,
+ NPC_S_KPU4_NSH, 14, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_FCOE, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
+ 8, 0, 6, 2, 0,
+ NPC_S_KPU5_IP, 18, 1,
+ NPC_LID_LB, NPC_LT_LB_EDSA,
+ NPC_F_LB_L_EDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 1, 0,
- NPC_S_KPU4_MPLS, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
+ 6, 0, 0, 2, 0,
+ NPC_S_KPU5_IP6, 18, 1,
+ NPC_LID_LB, NPC_LT_LB_EDSA,
+ NPC_F_LB_L_EDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 1, 0,
- NPC_S_KPU4_MPLS, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_ARP, 18, 1,
+ NPC_LID_LB, NPC_LT_LB_EDSA,
+ NPC_F_LB_L_EDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 1, 0,
- NPC_S_KPU4_NSH, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG,
+ 8, 0, 6, 2, 0,
+ NPC_S_KPU5_RARP, 18, 1,
+ NPC_LID_LB, NPC_LT_LB_EDSA,
+ NPC_F_LB_L_EDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_CTAG_UNK,
+ 6, 0, 0, 2, 0,
+ NPC_S_KPU5_PTP, 18, 1,
+ NPC_LID_LB, NPC_LT_LB_EDSA,
+ NPC_F_LB_L_EDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 0, 0, 0,
- NPC_S_KPU3_CTAG, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_QINQ_CTAG,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU5_FCOE, 18, 1,
+ NPC_LID_LB, NPC_LT_LB_EDSA,
+ NPC_F_LB_L_EDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 0, 0, 0,
- NPC_S_KPU3_QINQ, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_QINQ_QINQ,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU3_CTAG, 16, 1,
+ NPC_LID_LB, NPC_LT_LB_EDSA_VLAN,
+ NPC_F_LB_L_EDSA_VLAN,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 0, 1,
NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_UNK_ETYPE,
+ NPC_LID_LB, NPC_LT_LB_EDSA,
+ NPC_F_LB_U_UNK_ETYPE | NPC_F_LB_L_EDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
8, 0, 6, 2, 0,
NPC_S_KPU5_IP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- 0,
+ NPC_LID_LB, NPC_LT_LB_EXDSA,
+ NPC_F_LB_L_EXDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
6, 0, 0, 2, 0,
NPC_S_KPU5_IP6, 10, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- 0,
+ NPC_LID_LB, NPC_LT_LB_EXDSA,
+ NPC_F_LB_L_EXDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 2, 0,
NPC_S_KPU5_ARP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- 0,
+ NPC_LID_LB, NPC_LT_LB_EXDSA,
+ NPC_F_LB_L_EXDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
+ 8, 0, 6, 2, 0,
NPC_S_KPU5_RARP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- 0,
+ NPC_LID_LB, NPC_LT_LB_EXDSA,
+ NPC_F_LB_L_EXDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
+ 6, 0, 0, 2, 0,
NPC_S_KPU5_PTP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- 0,
+ NPC_LID_LB, NPC_LT_LB_EXDSA,
+ NPC_F_LB_L_EXDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 2, 0,
NPC_S_KPU5_FCOE, 10, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- 0,
+ NPC_LID_LB, NPC_LT_LB_EXDSA,
+ NPC_F_LB_L_EXDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 1, 0,
- NPC_S_KPU4_MPLS, 10, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- 1,
+ 4, 8, 0, 0, 0,
+ NPC_S_KPU3_CTAG, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_EXDSA_VLAN,
+ NPC_F_LB_L_EXDSA_VLAN,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 1, 0,
- NPC_S_KPU4_MPLS, 10, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- 2,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LB, NPC_LT_LB_EXDSA,
+ NPC_F_LB_U_UNK_ETYPE | NPC_F_LB_L_EXDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 1, 0,
- NPC_S_KPU4_NSH, 10, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- 2,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_NGIO,
+ 0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 0, 0,
- NPC_S_KPU3_CTAG, 10, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_CTAG,
+ NPC_ERRLEV_LB, NPC_EC_L2_K3,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
+};
+
+static struct npc_kpu_profile_action kpu3_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 16, 20, 24, 0, 0,
- NPC_S_KPU3_ITAG, 14, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_BTAG_ITAG,
+ 8, 0, 6, 1, 0,
+ NPC_S_KPU5_IP, 6, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 0, 0, 0,
- NPC_S_KPU3_STAG, 10, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_STAG,
+ 6, 0, 0, 1, 0,
+ NPC_S_KPU5_IP6, 6, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 0, 0, 0,
- NPC_S_KPU3_QINQ, 10, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_QINQ,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_ARP, 6, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 2, 0,
- NPC_S_KPU5_IP, 28, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_RARP, 6, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 2, 0,
- NPC_S_KPU5_IP6, 28, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_PTP, 6, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_ARP, 28, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_FCOE, 6, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 0, 0,
- NPC_S_KPU3_STAG, 28, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG_STAG,
+ 2, 6, 10, 0, 0,
+ NPC_S_KPU4_MPLS, 6, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 0, 0,
- NPC_S_KPU3_CTAG, 28, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG_CTAG,
+ 2, 6, 10, 0, 0,
+ NPC_S_KPU4_MPLS, 6, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_ITAG_UNK,
+ 2, 0, 0, 0, 0,
+ NPC_S_KPU4_NSH, 6, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
+ NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_ETAG,
- NPC_F_LB_U_UNK_ETYPE,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 2, 0,
- NPC_S_KPU5_IP, 20, 1,
- NPC_LID_LB, NPC_LT_LB_ITAG,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 2, 0,
- NPC_S_KPU5_IP6, 20, 1,
- NPC_LID_LB, NPC_LT_LB_ITAG,
+ 8, 0, 6, 1, 0,
+ NPC_S_KPU5_IP, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_ARP, 20, 1,
- NPC_LID_LB, NPC_LT_LB_ITAG,
+ 6, 0, 0, 1, 0,
+ NPC_S_KPU5_IP6, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_RARP, 20, 1,
- NPC_LID_LB, NPC_LT_LB_ITAG,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_ARP, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 2, 0,
- NPC_S_KPU5_IP, 28, 1,
- NPC_LID_LB, NPC_LT_LB_ITAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_STAG_CTAG,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 2, 0,
- NPC_S_KPU5_IP6, 28, 1,
- NPC_LID_LB, NPC_LT_LB_ITAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_STAG_CTAG,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_RARP, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_ARP, 28, 1,
- NPC_LID_LB, NPC_LT_LB_ITAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_STAG_CTAG,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_PTP, 8, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 2, 0,
- NPC_S_KPU5_IP, 24, 1,
- NPC_LID_LB, NPC_LT_LB_ITAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_STAG,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_FCOE, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 2, 0,
- NPC_S_KPU5_IP6, 24, 1,
- NPC_LID_LB, NPC_LT_LB_ITAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_STAG,
+ 2, 6, 10, 0, 0,
+ NPC_S_KPU4_MPLS, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_ARP, 24, 1,
- NPC_LID_LB, NPC_LT_LB_ITAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_STAG,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ 2, 6, 10, 0, 0,
+ NPC_S_KPU4_MPLS, 8, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 2, 0,
- NPC_S_KPU5_IP, 24, 1,
- NPC_LID_LB, NPC_LT_LB_ITAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_CTAG,
+ 2, 0, 0, 0, 0,
+ NPC_S_KPU4_NSH, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 2, 0,
- NPC_S_KPU5_IP6, 24, 1,
- NPC_LID_LB, NPC_LT_LB_ITAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_CTAG,
+ 8, 0, 6, 1, 0,
+ NPC_S_KPU5_IP, 4, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_ARP, 24, 1,
- NPC_LID_LB, NPC_LT_LB_ITAG,
- NPC_F_LB_U_MORE_TAG|NPC_F_LB_L_WITH_CTAG,
+ 6, 0, 0, 1, 0,
+ NPC_S_KPU5_IP6, 4, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_ARP, 4, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_RARP, 4, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 2, 0,
- NPC_S_KPU5_IP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 2, 6, 10, 0, 0,
+ NPC_S_KPU4_MPLS, 4, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 2, 0,
- NPC_S_KPU5_IP6, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 2, 6, 10, 0, 0,
+ NPC_S_KPU4_MPLS, 4, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_ARP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 2, 0, 0, 0, 0,
+ NPC_S_KPU4_NSH, 4, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_RARP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_PTP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 8, 0, 6, 1, 0,
+ NPC_S_KPU5_IP, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_FCOE, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 6, 0, 0, 1, 0,
+ NPC_S_KPU5_IP6, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 1, 0,
- NPC_S_KPU4_MPLS, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_ARP, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 1, 0,
- NPC_S_KPU4_MPLS, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_RARP, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 1, 0,
- NPC_S_KPU4_NSH, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_PTP, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 0, 0, 0,
- NPC_S_KPU3_QINQ, 10, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_MORE_TAG | NPC_F_LB_L_WITH_QINQ_QINQ,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_FCOE, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- NPC_F_LB_U_UNK_ETYPE,
+ 2, 6, 10, 0, 0,
+ NPC_S_KPU4_MPLS, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 2, 0,
- NPC_S_KPU5_IP, 14, 0,
+ 2, 6, 10, 0, 0,
+ NPC_S_KPU4_MPLS, 8, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 2, 0,
- NPC_S_KPU5_IP6, 14, 0,
+ 2, 0, 0, 0, 0,
+ NPC_S_KPU4_NSH, 8, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_ARP, 14, 0,
+ 8, 0, 6, 1, 0,
+ NPC_S_KPU5_IP, 4, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_RARP, 14, 0,
+ 6, 0, 0, 1, 0,
+ NPC_S_KPU5_IP6, 4, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_PTP, 14, 0,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_ARP, 4, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_FCOE, 14, 0,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_RARP, 4, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 0, 0, 0,
- NPC_S_KPU3_CTAG_C, 14, 0,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_PTP, 4, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 20, 0, 0,
- NPC_S_KPU3_STAG_C, 14, 0,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_FCOE, 4, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 0, 0, 0,
- NPC_S_KPU3_QINQ_C, 14, 0,
+ 2, 6, 10, 0, 0,
+ NPC_S_KPU4_MPLS, 4, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 1, 0,
- NPC_S_KPU4_MPLS, 14, 0,
+ 2, 6, 10, 0, 0,
+ NPC_S_KPU4_MPLS, 4, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 1, 0,
- NPC_S_KPU4_MPLS, 14, 0,
+ 2, 0, 0, 0, 0,
+ NPC_S_KPU4_NSH, 4, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 1, 0,
- NPC_S_KPU4_NSH, 14, 0,
+ NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 2, 0,
- NPC_S_KPU5_IP, 18, 1,
- NPC_LID_LB, NPC_LT_LB_EDSA,
- NPC_F_LB_L_EDSA,
+ 8, 0, 6, 1, 0,
+ NPC_S_KPU5_IP, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 2, 0,
- NPC_S_KPU5_IP6, 18, 1,
- NPC_LID_LB, NPC_LT_LB_EDSA,
- NPC_F_LB_L_EDSA,
+ 6, 0, 0, 1, 0,
+ NPC_S_KPU5_IP6, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_ARP, 18, 1,
- NPC_LID_LB, NPC_LT_LB_EDSA,
- NPC_F_LB_L_EDSA,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_ARP, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 2, 0,
- NPC_S_KPU5_RARP, 18, 1,
- NPC_LID_LB, NPC_LT_LB_EDSA,
- NPC_F_LB_L_EDSA,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_RARP, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 2, 0,
- NPC_S_KPU5_PTP, 18, 1,
- NPC_LID_LB, NPC_LT_LB_EDSA,
- NPC_F_LB_L_EDSA,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_PTP, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_FCOE, 18, 1,
- NPC_LID_LB, NPC_LT_LB_EDSA,
- NPC_F_LB_L_EDSA,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_FCOE, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 4, 8, 0, 0, 0,
- NPC_S_KPU3_CTAG, 16, 1,
- NPC_LID_LB, NPC_LT_LB_EDSA_VLAN,
- NPC_F_LB_L_EDSA_VLAN,
+ 2, 6, 10, 0, 0,
+ NPC_S_KPU4_MPLS, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_EDSA,
- NPC_F_LB_U_UNK_ETYPE | NPC_F_LB_L_EDSA,
+ 2, 6, 10, 0, 0,
+ NPC_S_KPU4_MPLS, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 2, 0,
- NPC_S_KPU5_IP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_EXDSA,
- NPC_F_LB_L_EXDSA,
+ 2, 0, 0, 0, 0,
+ NPC_S_KPU4_NSH, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 2, 0,
- NPC_S_KPU5_IP6, 10, 1,
- NPC_LID_LB, NPC_LT_LB_EXDSA,
- NPC_F_LB_L_EXDSA,
+ NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_ARP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_EXDSA,
- NPC_F_LB_L_EXDSA,
+ 8, 0, 6, 1, 0,
+ NPC_S_KPU5_IP, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 2, 0,
- NPC_S_KPU5_RARP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_EXDSA,
- NPC_F_LB_L_EXDSA,
+ 6, 0, 0, 1, 0,
+ NPC_S_KPU5_IP6, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 2, 0,
- NPC_S_KPU5_PTP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_EXDSA,
- NPC_F_LB_L_EXDSA,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_ARP, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_FCOE, 10, 1,
- NPC_LID_LB, NPC_LT_LB_EXDSA,
- NPC_F_LB_L_EXDSA,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_RARP, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 4, 8, 0, 0, 0,
- NPC_S_KPU3_CTAG, 8, 1,
- NPC_LID_LB, NPC_LT_LB_EXDSA_VLAN,
- NPC_F_LB_L_EXDSA_VLAN,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_PTP, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_EXDSA,
- NPC_F_LB_U_UNK_ETYPE | NPC_F_LB_L_EXDSA,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_FCOE, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_LB, NPC_EC_L2_K3,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 0, 0,
+ NPC_S_KPU4_MPLS, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
-};
-
-static const struct npc_kpu_profile_action kpu3_action_entries[] = {
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 2, 6, 10, 0, 0,
+ NPC_S_KPU4_MPLS, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 2, 0, 0, 0, 0,
+ NPC_S_KPU4_NSH, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 8, 0, 6, 1, 0,
+ NPC_S_KPU5_IP, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_RARP, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 6, 0, 0, 1, 0,
+ NPC_S_KPU5_IP6, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 1, 0,
- NPC_S_KPU5_PTP, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU5_ARP, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 1, 0,
- NPC_S_KPU5_FCOE, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU5_RARP, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU4_MPLS, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU4_MPLS, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
2, 0, 0, 0, 0,
- NPC_S_KPU4_NSH, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU4_NSH, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU5_IP, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU5_IP6, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU5_ARP, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 1, 0,
- NPC_S_KPU5_RARP, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU5_RARP, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 1, 0,
- NPC_S_KPU5_PTP, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU5_PTP, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 1, 0,
- NPC_S_KPU5_FCOE, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU5_FCOE, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU4_MPLS, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU4_MPLS, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
2, 0, 0, 0, 0,
- NPC_S_KPU4_NSH, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU4_NSH, 8, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU5_IP, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU5_IP6, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU5_ARP, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 1, 0,
- NPC_S_KPU5_RARP, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU5_RARP, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_PTP, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_FCOE, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU4_MPLS, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU4_MPLS, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
2, 0, 0, 0, 0,
- NPC_S_KPU4_NSH, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_KPU4_NSH, 4, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ NPC_S_KPU5_IP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_DSA,
+ NPC_F_LB_L_DSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ NPC_S_KPU5_IP6, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_DSA,
+ NPC_F_LB_L_DSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ NPC_S_KPU5_ARP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_DSA,
+ NPC_F_LB_L_DSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 1, 0,
- NPC_S_KPU5_RARP, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ NPC_S_KPU5_RARP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_DSA,
+ NPC_F_LB_L_DSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 1, 0,
- NPC_S_KPU5_PTP, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ NPC_S_KPU5_PTP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_DSA,
+ NPC_F_LB_L_DSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 1, 0,
- NPC_S_KPU5_FCOE, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ NPC_S_KPU5_FCOE, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_DSA,
+ NPC_F_LB_L_DSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 8, 0, 6, 1, 0,
+ NPC_S_KPU5_IP, 14, 1,
+ NPC_LID_LB, NPC_LT_LB_DSA_VLAN,
+ NPC_F_LB_L_DSA_VLAN,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 6, 0, 0, 1, 0,
+ NPC_S_KPU5_IP6, 14, 1,
+ NPC_LID_LB, NPC_LT_LB_DSA_VLAN,
+ NPC_F_LB_L_DSA_VLAN,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 0, 0,
- NPC_S_KPU4_NSH, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_ARP, 14, 1,
+ NPC_LID_LB, NPC_LT_LB_DSA_VLAN,
+ NPC_F_LB_L_DSA_VLAN,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_RARP, 14, 1,
+ NPC_LID_LB, NPC_LT_LB_DSA_VLAN,
+ NPC_F_LB_L_DSA_VLAN,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU5_PTP, 14, 1,
+ NPC_LID_LB, NPC_LT_LB_DSA_VLAN,
+ NPC_F_LB_L_DSA_VLAN,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ NPC_S_KPU5_FCOE, 14, 1,
+ NPC_LID_LB, NPC_LT_LB_DSA_VLAN,
+ NPC_F_LB_L_DSA_VLAN,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_RARP, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LB, NPC_LT_LB_DSA_VLAN,
+ NPC_F_LB_U_UNK_ETYPE | NPC_F_LB_L_DSA,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LB, NPC_LT_LB_DSA,
+ NPC_F_LB_U_UNK_ETYPE | NPC_F_LB_L_DSA_VLAN,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_PTP, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU4_VLAN_EXDSA, 12, 1,
+ NPC_LID_LB, NPC_LT_LB_VLAN_EXDSA,
0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_FCOE, 4, 0,
+ NPC_ERRLEV_LB, NPC_EC_L2_K3,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
+};
+
+static struct npc_kpu_profile_action kpu4_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU5_MPLS_PL, 4, 1,
+ NPC_LID_LC, NPC_LT_LC_MPLS,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU5_MPLS_PL, 8, 1,
+ NPC_LID_LC, NPC_LT_LC_MPLS,
+ NPC_F_LC_L_MPLS_2_LABELS,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 0, 0,
- NPC_S_KPU4_NSH, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU5_MPLS_PL, 12, 1,
+ NPC_LID_LC, NPC_LT_LC_MPLS,
+ NPC_F_LC_L_MPLS_3_LABELS,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 4, 0, 0, 0,
+ NPC_S_KPU5_MPLS, 12, 1,
+ NPC_LID_LC, NPC_LT_LC_MPLS,
+ NPC_F_LC_L_MPLS_4_LABELS,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 2, 0,
- NPC_S_KPU5_IP, 18, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 8, 0, 6, 7, 0,
+ NPC_S_KPU12_TU_IP, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_NSH,
0,
- 0, 0, 0, 0,
+ 1, 0x3f, 0, 2,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 2, 0,
- NPC_S_KPU5_IP6, 18, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 6, 0, 0, 7, 0,
+ NPC_S_KPU12_TU_IP6, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_NSH,
0,
- 0, 0, 0, 0,
+ 1, 0x3f, 0, 2,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_ARP, 18, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 12, 16, 20, 6, 0,
+ NPC_S_KPU11_TU_ETHER, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_NSH,
0,
- 0, 0, 0, 0,
+ 1, 0x3f, 0, 2,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 2, 0,
- NPC_S_KPU5_RARP, 18, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 0, 0, 0, 4, 0,
+ NPC_S_KPU9_TU_MPLS_IN_NSH, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_NSH,
0,
- 0, 0, 0, 0,
+ 1, 0x3f, 0, 2,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 26, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_ERRLEV_LC, NPC_EC_NSH_UNK,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_NSH,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 26, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 8, 0, 6, 0, 0,
+ NPC_S_KPU5_IP, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_FDSA,
+ NPC_F_LB_L_FDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 26, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 6, 0, 0, 0, 0,
+ NPC_S_KPU5_IP6, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_FDSA,
+ NPC_F_LB_L_FDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 22, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU5_ARP, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_FDSA,
+ NPC_F_LB_L_FDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 22, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 8, 0, 6, 0, 0,
+ NPC_S_KPU5_RARP, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_FDSA,
+ NPC_F_LB_L_FDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 22, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 6, 0, 0, 0, 0,
+ NPC_S_KPU5_PTP, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_FDSA,
+ NPC_F_LB_L_FDSA,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU5_FCOE, 6, 1,
+ NPC_LID_LB, NPC_LT_LB_FDSA,
+ NPC_F_LB_L_FDSA,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 8, 0, 6, 0, 0,
+ NPC_S_KPU5_IP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_FDSA,
+ NPC_F_LB_L_FDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 22, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 6, 0, 0, 0, 0,
+ NPC_S_KPU5_IP6, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_FDSA,
+ NPC_F_LB_L_FDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 22, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU5_ARP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_FDSA,
+ NPC_F_LB_L_FDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 22, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ 8, 0, 6, 0, 0,
+ NPC_S_KPU5_RARP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_FDSA,
+ NPC_F_LB_L_FDSA,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 6, 0, 0, 0, 0,
+ NPC_S_KPU5_PTP, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_FDSA,
+ NPC_F_LB_L_FDSA,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
- NPC_LID_LB, NPC_LT_NA,
- 0,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU5_FCOE, 10, 1,
+ NPC_LID_LB, NPC_LT_LB_FDSA,
+ NPC_F_LB_L_FDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 4, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
+ 8, 0, 6, 0, 0,
+ NPC_S_KPU5_IP, 14, 1,
+ NPC_LID_LB, NPC_LT_LB_PPPOE,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 4, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
+ 6, 0, 0, 0, 0,
+ NPC_S_KPU5_IP6, 14, 1,
+ NPC_LID_LB, NPC_LT_LB_PPPOE,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 4, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
- 0,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LB, NPC_LT_LB_FDSA,
+ NPC_F_LB_U_UNK_ETYPE | NPC_F_LB_L_FDSA,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_RARP, 4, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
+ 8, 0, 6, 0, 0,
+ NPC_S_KPU5_IP, 2, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_PTP, 4, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
+ 6, 0, 0, 0, 0,
+ NPC_S_KPU5_IP6, 2, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_FCOE, 4, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU5_ARP, 2, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 4, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
+ 8, 0, 6, 0, 0,
+ NPC_S_KPU5_RARP, 2, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 4, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
+ 6, 0, 0, 0, 0,
+ NPC_S_KPU5_PTP, 2, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 0, 0,
- NPC_S_KPU4_NSH, 4, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU5_FCOE, 2, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_CTAG,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 8, 0, 6, 0, 0,
+ NPC_S_KPU5_IP, 10, 0,
+ NPC_LID_LB, NPC_LT_LB_PPPOE,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 6, 0, 0, 0, 0,
+ NPC_S_KPU5_IP6, 10, 0,
+ NPC_LID_LB, NPC_LT_LB_PPPOE,
0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_ERRLEV_LB, NPC_EC_L2_K4,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+};
+
+static struct npc_kpu_profile_action kpu5_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
+ {
+ NPC_ERRLEV_LC, NPC_EC_IP_TTL_0,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_RARP, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- 0,
+ NPC_ERRLEV_LC, NPC_EC_IP_FRAG_OFFSET_1,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
+ NPC_F_LC_U_IP_FRAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_PTP, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 2, 12, 0, 2, 0,
+ NPC_S_KPU8_TCP, 20, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_FCOE, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 2, 0, 0, 2, 0,
+ NPC_S_KPU8_UDP, 20, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU8_SCTP, 20, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU8_ICMP, 20, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 0, 0,
- NPC_S_KPU4_NSH, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU8_IGMP, 20, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU9_ESP, 20, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU8_AH, 20, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 2, 0, 0, 2, 0,
+ NPC_S_KPU8_GRE, 20, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_RARP, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- 0,
+ 8, 0, 6, 6, 0,
+ NPC_S_KPU12_TU_IP, 20, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
+ NPC_F_LC_L_IP_IN_IP,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- 0,
+ 6, 0, 0, 6, 0,
+ NPC_S_KPU12_TU_IP6, 20, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
+ NPC_F_LC_L_6TO4,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- 0,
+ 2, 6, 10, 3, 0,
+ NPC_S_KPU9_TU_MPLS_IN_IP, 20, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
+ NPC_F_LC_L_MPLS_IN_IP,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 0, 0,
- NPC_S_KPU4_NSH, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- 0,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
0, 0, 0, 0, 1,
NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- 0,
+ NPC_LID_LC, NPC_LT_LC_IP,
+ NPC_F_LC_U_UNK_PROTO,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- 0,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
+ NPC_F_LC_U_IP_FRAG,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 2, 12, 0, 2, 0,
+ NPC_S_KPU8_TCP, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP_OPT,
0,
- 0, 0, 0, 0,
+ 0, 0xf, 0, 2,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 2, 8, 10, 2, 0,
+ NPC_S_KPU8_UDP, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP_OPT,
0,
- 0, 0, 0, 0,
+ 0, 0xf, 0, 2,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_RARP, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU8_SCTP, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP_OPT,
0,
- 0, 0, 0, 0,
+ 0, 0xf, 0, 2,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_PTP, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU8_ICMP, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP_OPT,
0,
- 0, 0, 0, 0,
+ 0, 0xf, 0, 2,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_FCOE, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU8_IGMP, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP_OPT,
0,
- 0, 0, 0, 0,
+ 0, 0xf, 0, 2,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU9_ESP, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP_OPT,
0,
- 0, 0, 0, 0,
+ 0, 0xf, 0, 2,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU8_AH, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP_OPT,
0,
- 0, 0, 0, 0,
+ 0, 0xf, 0, 2,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 0, 0,
- NPC_S_KPU4_NSH, 8, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 2, 0, 0, 2, 0,
+ NPC_S_KPU8_GRE, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP_OPT,
0,
- 0, 0, 0, 0,
+ 0, 0xf, 0, 2,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- 0,
- 0, 0, 0, 0,
+ 8, 0, 6, 6, 0,
+ NPC_S_KPU12_TU_IP, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP_OPT,
+ NPC_F_LC_L_IP_IN_IP,
+ 0, 0xf, 0, 2,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- 0,
- 0, 0, 0, 0,
+ 6, 0, 0, 6, 0,
+ NPC_S_KPU12_TU_IP6, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP_OPT,
+ NPC_F_LC_L_6TO4,
+ 0, 0xf, 0, 2,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- 0,
- 0, 0, 0, 0,
+ 2, 6, 10, 3, 0,
+ NPC_S_KPU9_TU_MPLS_IN_IP, 20, 1,
+ NPC_LID_LC, NPC_LT_LC_IP_OPT,
+ NPC_F_LC_L_MPLS_IN_IP,
+ 0, 0xf, 0, 2,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_RARP, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- 0,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP_OPT,
+ NPC_F_LC_U_UNK_PROTO,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_PTP, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- 0,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP_OPT,
+ NPC_F_LC_U_IP_FRAG,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_FCOE, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_ERRLEV_LC, NPC_EC_IP_VER,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_ARP,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 6, 10, 0, 0,
- NPC_S_KPU4_MPLS, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_RARP,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 0, 0, 0, 0,
- NPC_S_KPU4_NSH, 4, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
- 0,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_LB, NPC_EC_L2_K3_ETYPE_UNK,
0, 0, 0, 0, 1,
NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_STAG_QINQ,
+ NPC_LID_LC, NPC_LT_LC_PTP,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_DSA,
- NPC_F_LB_L_DSA,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_FCOE,
+ 0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 10, 1,
- NPC_LID_LB, NPC_LT_LB_DSA,
- NPC_F_LB_L_DSA,
+ NPC_ERRLEV_LC, NPC_EC_IP6_HOP_0,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_DSA,
- NPC_F_LB_L_DSA,
+ 2, 12, 0, 2, 0,
+ NPC_S_KPU8_TCP, 40, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_RARP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_DSA,
- NPC_F_LB_L_DSA,
+ 2, 0, 0, 2, 0,
+ NPC_S_KPU8_UDP, 40, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_PTP, 10, 1,
- NPC_LID_LB, NPC_LT_LB_DSA,
- NPC_F_LB_L_DSA,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU8_SCTP, 40, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_FCOE, 10, 1,
- NPC_LID_LB, NPC_LT_LB_DSA,
- NPC_F_LB_L_DSA,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU8_ICMP, 40, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 1, 0,
- NPC_S_KPU5_IP, 14, 1,
- NPC_LID_LB, NPC_LT_LB_DSA_VLAN,
- NPC_F_LB_L_DSA_VLAN,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU8_ICMP6, 40, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 1, 0,
- NPC_S_KPU5_IP6, 14, 1,
- NPC_LID_LB, NPC_LT_LB_DSA_VLAN,
- NPC_F_LB_L_DSA_VLAN,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU8_GRE, 40, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_ARP, 14, 1,
- NPC_LID_LB, NPC_LT_LB_DSA_VLAN,
- NPC_F_LB_L_DSA_VLAN,
+ 6, 0, 0, 6, 0,
+ NPC_S_KPU12_TU_IP6, 40, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6,
+ NPC_F_LC_L_IP6_TUN_IP6,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_RARP, 14, 1,
- NPC_LID_LB, NPC_LT_LB_DSA_VLAN,
- NPC_F_LB_L_DSA_VLAN,
+ 2, 6, 10, 3, 0,
+ NPC_S_KPU9_TU_MPLS_IN_IP, 40, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6,
+ NPC_F_LC_L_IP6_MPLS_IN_IP,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_PTP, 14, 1,
- NPC_LID_LB, NPC_LT_LB_DSA_VLAN,
- NPC_F_LB_L_DSA_VLAN,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU6_IP6_HOP_DEST, 40, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6_EXT,
+ NPC_F_LC_L_EXT_HOP,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 1, 0,
- NPC_S_KPU5_FCOE, 14, 1,
- NPC_LID_LB, NPC_LT_LB_DSA_VLAN,
- NPC_F_LB_L_DSA_VLAN,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU6_IP6_HOP_DEST, 40, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6_EXT,
+ NPC_F_LC_L_EXT_DEST,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_DSA_VLAN,
- NPC_F_LB_U_UNK_ETYPE | NPC_F_LB_L_DSA,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU6_IP6_ROUT, 40, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6_EXT,
+ NPC_F_LC_L_EXT_ROUT,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_DSA,
- NPC_F_LB_U_UNK_ETYPE | NPC_F_LB_L_DSA_VLAN,
+ 0, 2, 0, 0, 0,
+ NPC_S_KPU6_IP6_FRAG, 40, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6_EXT,
+ NPC_F_LC_U_IP6_FRAG,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_LB, NPC_EC_L2_K3,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 3, 0,
+ NPC_S_KPU9_ESP, 40, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6_EXT,
0,
0, 0, 0, 0,
},
-};
-
-static const struct npc_kpu_profile_action kpu4_action_entries[] = {
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 0,
- NPC_S_KPU5_MPLS_PL, 4, 1,
- NPC_LID_LC, NPC_LT_LC_MPLS,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU8_AH, 40, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6_EXT,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 0,
- NPC_S_KPU5_MPLS_PL, 8, 1,
- NPC_LID_LC, NPC_LT_LC_MPLS,
- NPC_F_LC_L_MPLS_2_LABELS,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6_EXT,
+ NPC_F_LC_L_EXT_MOBILITY,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 0,
- NPC_S_KPU5_MPLS_PL, 12, 1,
- NPC_LID_LC, NPC_LT_LC_MPLS,
- NPC_F_LC_L_MPLS_3_LABELS,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6_EXT,
+ NPC_F_LC_L_EXT_HOSTID,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 2, 4, 0, 0, 0,
- NPC_S_KPU5_MPLS, 12, 1,
- NPC_LID_LC, NPC_LT_LC_MPLS,
- NPC_F_LC_L_MPLS_4_LABELS,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6_EXT,
+ NPC_F_LC_L_EXT_SHIM6,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 7, 0,
- NPC_S_KPU12_TU_IP, 0, 1,
- NPC_LID_LC, NPC_LT_LC_NSH,
- 0,
- 1, 0x3f, 0, 2,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6,
+ NPC_F_LC_U_UNK_PROTO,
+ 0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 7, 0,
- NPC_S_KPU12_TU_IP6, 0, 1,
- NPC_LID_LC, NPC_LT_LC_NSH,
+ NPC_ERRLEV_LC, NPC_EC_IP6_VER,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6,
0,
- 1, 0x3f, 0, 2,
+ 0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 12, 16, 20, 6, 0,
- NPC_S_KPU11_TU_ETHER, 0, 1,
- NPC_LID_LC, NPC_LT_LC_NSH,
+ 8, 0, 6, 6, 0,
+ NPC_S_KPU12_TU_IP, 4, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
- 1, 0x3f, 0, 2,
+ 0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 4, 0,
- NPC_S_KPU9_TU_MPLS_IN_NSH, 0, 1,
- NPC_LID_LC, NPC_LT_LC_NSH,
- 0,
- 1, 0x3f, 0, 2,
- },
- {
- NPC_ERRLEV_LC, NPC_EC_NSH_UNK,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LC, NPC_LT_LC_NSH,
+ 6, 0, 0, 6, 0,
+ NPC_S_KPU12_TU_IP6, 4, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 0, 0,
- NPC_S_KPU5_IP, 6, 1,
- NPC_LID_LB, NPC_LT_LB_FDSA,
- NPC_F_LB_L_FDSA,
+ 12, 16, 20, 5, 0,
+ NPC_S_KPU11_TU_ETHER, 8, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 0, 0,
- NPC_S_KPU5_IP6, 6, 1,
- NPC_LID_LB, NPC_LT_LB_FDSA,
- NPC_F_LB_L_FDSA,
+ 12, 16, 20, 5, 0,
+ NPC_S_KPU11_TU_ETHER, 4, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 0,
- NPC_S_KPU5_ARP, 6, 1,
- NPC_LID_LB, NPC_LT_LB_FDSA,
- NPC_F_LB_L_FDSA,
+ NPC_ERRLEV_LB, NPC_EC_MPLS_2MANY,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 0, 0,
- NPC_S_KPU5_RARP, 6, 1,
- NPC_LID_LB, NPC_LT_LB_FDSA,
- NPC_F_LB_L_FDSA,
+ 8, 0, 6, 6, 0,
+ NPC_S_KPU12_TU_IP, 0, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 0, 0,
- NPC_S_KPU5_PTP, 6, 1,
- NPC_LID_LB, NPC_LT_LB_FDSA,
- NPC_F_LB_L_FDSA,
+ 6, 0, 0, 6, 0,
+ NPC_S_KPU12_TU_IP6, 0, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 0,
- NPC_S_KPU5_FCOE, 6, 1,
- NPC_LID_LB, NPC_LT_LB_FDSA,
- NPC_F_LB_L_FDSA,
+ 12, 16, 20, 5, 0,
+ NPC_S_KPU11_TU_ETHER, 4, 0,
+ NPC_LID_LB, NPC_LT_NA,
+ 0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LB, NPC_LT_LB_FDSA,
- NPC_F_LB_U_UNK_ETYPE | NPC_F_LB_L_FDSA,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_LB, NPC_EC_L2_K4,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
- NPC_LID_LC, NPC_LT_NA,
+ 12, 16, 20, 5, 0,
+ NPC_S_KPU11_TU_ETHER, 0, 0,
+ NPC_LID_LB, NPC_LT_NA,
0,
0, 0, 0, 0,
},
-};
-
-static const struct npc_kpu_profile_action kpu5_action_entries[] = {
{
NPC_ERRLEV_LC, NPC_EC_IP_TTL_0,
0, 0, 0, 0, 1,
0, 0, 0, 0, 1,
NPC_S_NA, 0, 1,
NPC_LID_LC, NPC_LT_LC_IP,
- NPC_F_LC_U_UNK_PROTO,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LC, NPC_LT_LC_IP,
- NPC_F_LC_U_IP_FRAG,
+ NPC_F_LC_U_UNK_PROTO,
0, 0, 0, 0,
},
{
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LC, NPC_LT_LC_IP_OPT,
- NPC_F_LC_U_IP_FRAG,
- 0, 0, 0, 0,
- },
- {
NPC_ERRLEV_LC, NPC_EC_IP_VER,
0, 0, 0, 0, 1,
NPC_S_NA, 0, 1,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LC, NPC_LT_LC_ARP,
- 0,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LC, NPC_LT_LC_RARP,
- 0,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LC, NPC_LT_LC_PTP,
- 0,
- 0, 0, 0, 0,
- },
- {
- NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LC, NPC_LT_LC_FCOE,
- 0,
- 0, 0, 0, 0,
- },
- {
NPC_ERRLEV_LC, NPC_EC_IP6_HOP_0,
0, 0, 0, 0, 1,
NPC_S_NA, 0, 1,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 0, 0,
- NPC_S_KPU6_IP6_HOP_DEST, 40, 1,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 40, 1,
NPC_LID_LC, NPC_LT_LC_IP6_EXT,
NPC_F_LC_L_EXT_HOP,
0, 0, 0, 0,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 0, 0,
- NPC_S_KPU6_IP6_HOP_DEST, 40, 1,
+ NPC_S_KPU6_IP6_CPT_HOP_DEST, 40, 1,
NPC_LID_LC, NPC_LT_LC_IP6_EXT,
NPC_F_LC_L_EXT_DEST,
0, 0, 0, 0,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 0, 0,
- NPC_S_KPU6_IP6_ROUT, 40, 1,
+ NPC_S_KPU6_IP6_CPT_ROUT, 40, 1,
NPC_LID_LC, NPC_LT_LC_IP6_EXT,
NPC_F_LC_L_EXT_ROUT,
0, 0, 0, 0,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 2, 0, 0, 0,
- NPC_S_KPU6_IP6_FRAG, 40, 1,
+ NPC_S_KPU6_IP6_CPT_FRAG, 40, 1,
NPC_LID_LC, NPC_LT_LC_IP6_EXT,
NPC_F_LC_U_IP6_FRAG,
0, 0, 0, 0,
0, 0, 0, 0,
},
{
- NPC_ERRLEV_LC, NPC_EC_IP6_VER,
+ NPC_ERRLEV_LC, NPC_EC_IP6_VER,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 1,
+ NPC_LID_LC, NPC_LT_LC_IP6,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_LC, NPC_EC_UNK,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+};
+
+static struct npc_kpu_profile_action kpu6_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 12, 0, 1, 0,
+ NPC_S_KPU8_TCP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 1, 0xff, 0, 3,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 8, 10, 1, 0,
+ NPC_S_KPU8_UDP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 1, 0xff, 0, 3,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_SCTP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 1, 0xff, 0, 3,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_ICMP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 1, 0xff, 0, 3,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_ICMP6, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 1, 0xff, 0, 3,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU9_ESP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 1, 0xff, 0, 3,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_AH, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 1, 0xff, 0, 3,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_GRE, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 1, 0xff, 0, 3,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 6, 0, 0, 5, 0,
+ NPC_S_KPU12_TU_IP6, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 1, 0xff, 0, 3,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 2, 0,
+ NPC_S_KPU9_TU_MPLS_IN_IP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 1, 0xff, 0, 3,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU7_IP6_ROUT, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 1, 0xff, 0, 3,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 2, 0, 0, 0,
+ NPC_S_KPU7_IP6_FRAG, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 1, 0xff, 0, 3,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
- NPC_LID_LC, NPC_LT_LC_IP6,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 6, 0,
- NPC_S_KPU12_TU_IP, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 2, 12, 0, 1, 0,
+ NPC_S_KPU8_TCP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
- 0, 0, 0, 0,
+ 1, 0xff, 0, 3,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 6, 0,
- NPC_S_KPU12_TU_IP6, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 2, 8, 10, 1, 0,
+ NPC_S_KPU8_UDP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
- 0, 0, 0, 0,
+ 1, 0xff, 0, 3,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 12, 16, 20, 5, 0,
- NPC_S_KPU11_TU_ETHER, 8, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_SCTP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
- 0, 0, 0, 0,
+ 1, 0xff, 0, 3,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 12, 16, 20, 5, 0,
- NPC_S_KPU11_TU_ETHER, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_ICMP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
- 0, 0, 0, 0,
+ 1, 0xff, 0, 3,
},
{
- NPC_ERRLEV_LB, NPC_EC_MPLS_2MANY,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
- NPC_LID_LB, NPC_LT_NA,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_ICMP6, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
- 0, 0, 0, 0,
+ 1, 0xff, 0, 3,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 8, 0, 6, 6, 0,
- NPC_S_KPU12_TU_IP, 0, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU9_ESP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
- 0, 0, 0, 0,
+ 1, 0xff, 0, 3,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 6, 0, 0, 6, 0,
- NPC_S_KPU12_TU_IP6, 0, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_AH, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
- 0, 0, 0, 0,
+ 1, 0xff, 0, 3,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 12, 16, 20, 5, 0,
- NPC_S_KPU11_TU_ETHER, 4, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_GRE, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
- 0, 0, 0, 0,
+ 1, 0xff, 0, 3,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 12, 16, 20, 5, 0,
- NPC_S_KPU11_TU_ETHER, 0, 0,
- NPC_LID_LB, NPC_LT_NA,
+ 6, 0, 0, 5, 0,
+ NPC_S_KPU12_TU_IP6, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
0,
- 0, 0, 0, 0,
+ 1, 0xff, 0, 3,
},
{
- NPC_ERRLEV_LC, NPC_EC_UNK,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 2, 0,
+ NPC_S_KPU9_TU_MPLS_IN_IP, 8, 0,
NPC_LID_LC, NPC_LT_NA,
0,
- 0, 0, 0, 0,
+ 1, 0xff, 0, 3,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 2, 0, 0, 0,
+ NPC_S_KPU7_IP6_FRAG, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 1, 0xff, 0, 3,
},
-};
-
-static const struct npc_kpu_profile_action kpu6_action_entries[] = {
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 0, 1,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ 2, 12, 0, 1, 0,
+ NPC_S_KPU8_TCP, 8, 0,
NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ 2, 8, 10, 1, 0,
+ NPC_S_KPU8_UDP, 8, 0,
NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_SCTP, 8, 0,
NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_ICMP, 8, 0,
NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_ICMP6, 8, 0,
NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ 0, 0, 0, 2, 0,
+ NPC_S_KPU9_ESP, 8, 0,
NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_AH, 8, 0,
NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU8_GRE, 8, 0,
NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ 6, 0, 0, 5, 0,
+ NPC_S_KPU12_TU_IP6, 8, 0,
NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 0,
+ 2, 6, 10, 2, 0,
+ NPC_S_KPU9_TU_MPLS_IN_IP, 8, 0,
NPC_LID_LC, NPC_LT_NA,
0,
0, 0, 0, 0,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 2, 0, 0, 0,
- NPC_S_KPU7_IP6_FRAG, 8, 0,
+ NPC_S_KPU7_CPT_IP6_FRAG, 8, 0,
NPC_LID_LC, NPC_LT_NA,
0,
1, 0xff, 0, 3,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 2, 0, 0, 0,
- NPC_S_KPU7_IP6_FRAG, 8, 0,
+ NPC_S_KPU7_CPT_IP6_FRAG, 8, 0,
NPC_LID_LC, NPC_LT_NA,
0,
1, 0xff, 0, 3,
},
};
-static const struct npc_kpu_profile_action kpu7_action_entries[] = {
+static struct npc_kpu_profile_action kpu7_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 0, 1,
0, 0, 0, 0,
},
{
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 12, 0, 0, 0,
+ NPC_S_KPU8_TCP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 8, 10, 0, 0,
+ NPC_S_KPU8_UDP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU8_SCTP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU8_ICMP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU8_ICMP6, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 1, 0,
+ NPC_S_KPU9_ESP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU8_AH, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 0,
+ NPC_S_KPU8_GRE, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 6, 0, 0, 4, 0,
+ NPC_S_KPU12_TU_IP6, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 2, 6, 10, 1, 0,
+ NPC_S_KPU9_TU_MPLS_IN_IP, 8, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
+ NPC_ERRLEV_RE, NPC_EC_NOERR,
+ 0, 0, 0, 0, 1,
+ NPC_S_NA, 0, 0,
+ NPC_LID_LC, NPC_LT_NA,
+ 0,
+ 0, 0, 0, 0,
+ },
+ {
NPC_ERRLEV_LC, NPC_EC_UNK,
0, 0, 0, 0, 1,
NPC_S_NA, 0, 0,
},
};
-static const struct npc_kpu_profile_action kpu8_action_entries[] = {
+static struct npc_kpu_profile_action kpu8_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
{
NPC_ERRLEV_LD, NPC_EC_TCP_FLAGS_FIN_ONLY,
0, 0, 0, 0, 1,
},
};
-static const struct npc_kpu_profile_action kpu9_action_entries[] = {
+static struct npc_kpu_profile_action kpu9_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 0, 0,
},
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
- 0, 0, 0, 0, 1,
- NPC_S_NA, 0, 1,
+ 8, 0, 6, 2, 0,
+ NPC_S_KPU12_TU_IP, 8, 1,
NPC_LID_LE, NPC_LT_LE_GTPU,
- NPC_F_LE_L_GTPU_UNK,
+ 0,
0, 0, 0, 0,
},
{
},
};
-static const struct npc_kpu_profile_action kpu10_action_entries[] = {
+static struct npc_kpu_profile_action kpu10_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
8, 0, 6, 1, 0,
},
};
-static const struct npc_kpu_profile_action kpu11_action_entries[] = {
+static struct npc_kpu_profile_action kpu11_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
8, 0, 6, 0, 0,
},
};
-static const struct npc_kpu_profile_action kpu12_action_entries[] = {
+static struct npc_kpu_profile_action kpu12_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
2, 12, 0, 2, 0,
},
};
-static const struct npc_kpu_profile_action kpu13_action_entries[] = {
+static struct npc_kpu_profile_action kpu13_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 0, 1,
},
};
-static const struct npc_kpu_profile_action kpu14_action_entries[] = {
+static struct npc_kpu_profile_action kpu14_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 0, 1,
},
};
-static const struct npc_kpu_profile_action kpu15_action_entries[] = {
+static struct npc_kpu_profile_action kpu15_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
{
NPC_ERRLEV_LG, NPC_EC_TCP_FLAGS_FIN_ONLY,
0, 0, 0, 0, 1,
},
};
-static const struct npc_kpu_profile_action kpu16_action_entries[] = {
+static struct npc_kpu_profile_action kpu16_action_entries[] = {
+ NPC_KPU_NOP_ACTION,
+ NPC_KPU_NOP_ACTION,
{
NPC_ERRLEV_RE, NPC_EC_NOERR,
0, 0, 0, 0, 1,
},
};
-static const struct npc_kpu_profile npc_kpu_profiles[] = {
+static struct npc_kpu_profile npc_kpu_profiles[] = {
{
ARRAY_SIZE(kpu1_cam_entries),
ARRAY_SIZE(kpu1_action_entries),
},
};
-static const struct npc_lt_def_cfg npc_lt_defaults = {
+static struct npc_lt_def_cfg npc_lt_defaults = {
.rx_ol2 = {
.lid = NPC_LID_LA,
.ltype_match = NPC_LT_LA_ETHER,
.ltype_mask = 0x0F,
},
+ .ovlan = {
+ .lid = NPC_LID_LB,
+ .ltype_match = NPC_LT_LB_CTAG,
+ .ltype_mask = 0x0F,
+ },
+ .ivlan = {
+ .lid = NPC_LID_LB,
+ .ltype_match = NPC_LT_LB_STAG_QINQ,
+ .ltype_mask = 0x0F,
+ },
.rx_oip4 = {
.lid = NPC_LID_LC,
.ltype_match = NPC_LT_LC_IP,
.ltype_match = NPC_LT_LG_TU_IP,
.ltype_mask = 0x0F,
},
+ .rx_apad0 = {
+ .valid = 0,
+ .lid = NPC_LID_LC,
+ .ltype_match = NPC_LT_LC_IP6,
+ .ltype_mask = 0x0F,
+ },
+ .rx_apad1 = {
+ .valid = 0,
+ .lid = NPC_LID_LC,
+ .ltype_match = NPC_LT_LC_IP6,
+ .ltype_mask = 0x0F,
+ },
+ .rx_et = {
+ {
+ .lid = NPC_LID_LB,
+ .ltype_match = NPC_LT_NA,
+ .ltype_mask = 0x0,
+ },
+ {
+ .lid = NPC_LID_LB,
+ .ltype_match = NPC_LT_NA,
+ .ltype_mask = 0x0,
+ },
+ },
};
static struct npc_mcam_kex npc_mkex_default = {
.name = "default",
.kpu_version = NPC_KPU_PROFILE_VER,
.keyx_cfg = {
- /* nibble: LA..LE (ltype only) + channel */
+ /* nibble: LA..LE (ltype only) + Error code + Channel */
[NIX_INTF_RX] = ((u64)NPC_MCAM_KEY_X2 << 32) | NPC_PARSE_NIBBLE_INTF_RX,
/* nibble: LA..LE (ltype only) */
[NIX_INTF_TX] = ((u64)NPC_MCAM_KEY_X2 << 32) | NPC_PARSE_NIBBLE_INTF_TX,
[NPC_LID_LA] = {
/* Layer A: Ethernet: */
[NPC_LT_LA_ETHER] = {
- /* DMAC: 6 bytes, KW1[47:0] */
+ /* DMAC: 6 bytes, KW1[55:8] */
KEX_LD_CFG(0x05, 0x0, 0x1, 0x0, NPC_KEXOF_DMAC),
- /* Ethertype: 2 bytes, KW0[47:32] */
- KEX_LD_CFG(0x01, 0xc, 0x1, 0x0, 0x4),
+ /* Ethertype: 2 bytes, KW0[55:40] */
+ KEX_LD_CFG(0x01, 0xc, 0x1, 0x0, 0x5),
+ },
+ /* Layer A: HiGig2: */
+ [NPC_LT_LA_HIGIG2_ETHER] = {
+ /* Classification: 2 bytes, KW1[23:8] */
+ KEX_LD_CFG(0x01, 0x8, 0x1, 0x0, NPC_KEXOF_DMAC),
+ /* VID: 2 bytes, KW1[39:24] */
+ KEX_LD_CFG(0x01, 0xc, 0x1, 0x0,
+ NPC_KEXOF_DMAC + 2),
},
},
[NPC_LID_LB] = {
/* Layer B: Single VLAN (CTAG) */
- /* CTAG VLAN[2..3] + Ethertype, 4 bytes, KW0[63:32] */
[NPC_LT_LB_CTAG] = {
- KEX_LD_CFG(0x03, 0x2, 0x1, 0x0, 0x4),
+ /* CTAG VLAN: 2 bytes, KW1[7:0], KW0[63:56] */
+ KEX_LD_CFG(0x01, 0x2, 0x1, 0x0, 0x7),
+ /* Ethertype: 2 bytes, KW0[55:40] */
+ KEX_LD_CFG(0x01, 0x4, 0x1, 0x0, 0x5),
},
/* Layer B: Stacked VLAN (STAG|QinQ) */
[NPC_LT_LB_STAG_QINQ] = {
- /* Outer VLAN: 2 bytes, KW0[63:48] */
- KEX_LD_CFG(0x01, 0x2, 0x1, 0x0, 0x6),
- /* Ethertype: 2 bytes, KW0[47:32] */
- KEX_LD_CFG(0x01, 0x8, 0x1, 0x0, 0x4),
+ /* Outer VLAN: 2 bytes, KW1[7:0], KW0[63:56] */
+ KEX_LD_CFG(0x01, 0x2, 0x1, 0x0, 0x7),
+ /* Ethertype: 2 bytes, KW0[55:40] */
+ KEX_LD_CFG(0x01, 0x8, 0x1, 0x0, 0x5),
},
[NPC_LT_LB_FDSA] = {
- /* SWITCH PORT: 1 byte, KW0[63:48] */
- KEX_LD_CFG(0x0, 0x1, 0x1, 0x0, 0x6),
- /* Ethertype: 2 bytes, KW0[47:32] */
- KEX_LD_CFG(0x01, 0x4, 0x1, 0x0, 0x4),
+ /* SWITCH PORT: 1 byte, KW0[63:56] */
+ KEX_LD_CFG(0x0, 0x1, 0x1, 0x0, 0x7),
+ /* Ethertype: 2 bytes, KW0[55:40] */
+ KEX_LD_CFG(0x01, 0x4, 0x1, 0x0, 0x5),
},
},
[NPC_LID_LC] = {
/* DMAC: 6 bytes, KW1[63:16] */
KEX_LD_CFG(0x05, 0x8, 0x1, 0x0, 0xa),
},
+ /* Layer A: HiGig2: */
+ [NPC_LT_LA_IH_NIX_HIGIG2_ETHER] = {
+ /* PF_FUNC: 2B , KW0 [47:32] */
+ KEX_LD_CFG(0x01, 0x0, 0x1, 0x0, 0x4),
+ /* VID: 2 bytes, KW1[31:16] */
+ KEX_LD_CFG(0x01, 0x10, 0x1, 0x0, 0xa),
+ },
},
[NPC_LID_LB] = {
/* Layer B: Single VLAN (CTAG) */
module_param(mkex_profile, charp, 0000);
MODULE_PARM_DESC(mkex_profile, "MKEX profile name string");
+static char *kpu_profile; /* KPU profile name */
+module_param(kpu_profile, charp, 0000);
+MODULE_PARM_DESC(kpu_profile, "KPU profile name string");
+
static void rvu_setup_hw_capabilities(struct rvu *rvu)
{
struct rvu_hwinfo *hw = rvu->hw;
return (rsrc->max - used);
}
+bool is_rsrc_free(struct rsrc_bmap *rsrc, int id)
+{
+ if (!rsrc->bmap)
+ return false;
+
+ return !test_bit(id, rsrc->bmap);
+}
+
int rvu_alloc_bitmap(struct rsrc_bmap *rsrc)
{
rsrc->bmap = kcalloc(BITS_TO_LONGS(rsrc->max),
return 0;
}
+int rvu_mbox_handler_set_vf_perm(struct rvu *rvu, struct set_vf_perm *req,
+ struct msg_rsp *rsp)
+{
+ struct rvu_hwinfo *hw = rvu->hw;
+ u16 pcifunc = req->hdr.pcifunc;
+ struct rvu_pfvf *pfvf;
+ int blkaddr, nixlf;
+ u16 target;
+
+ /* Only PF can add VF permissions */
+ if ((pcifunc & RVU_PFVF_FUNC_MASK) || is_afvf(pcifunc))
+ return -EOPNOTSUPP;
+
+ target = (pcifunc & ~RVU_PFVF_FUNC_MASK) | (req->vf + 1);
+ pfvf = rvu_get_pfvf(rvu, target);
+
+ if (req->flags & RESET_VF_PERM) {
+ pfvf->flags &= RVU_CLEAR_VF_PERM;
+ } else if (test_bit(PF_SET_VF_TRUSTED, &pfvf->flags) ^
+ (req->flags & VF_TRUSTED)) {
+ change_bit(PF_SET_VF_TRUSTED, &pfvf->flags);
+ /* disable multicast and promisc entries */
+ if (!test_bit(PF_SET_VF_TRUSTED, &pfvf->flags)) {
+ blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NIX, target);
+ if (blkaddr < 0)
+ return 0;
+ nixlf = rvu_get_lf(rvu, &hw->block[blkaddr],
+ target, 0);
+ if (nixlf < 0)
+ return 0;
+ npc_enadis_default_mce_entry(rvu, target, nixlf,
+ NIXLF_ALLMULTI_ENTRY,
+ false);
+ npc_enadis_default_mce_entry(rvu, target, nixlf,
+ NIXLF_PROMISC_ENTRY,
+ false);
+ }
+ }
+
+ return 0;
+}
+
static int rvu_process_mbox_msg(struct otx2_mbox *mbox, int devid,
struct mbox_msghdr *req)
{
strscpy(rvu->mkex_pfl_name,
mkex_profile ? mkex_profile : default_pfl_name, MKEX_NAME_LEN);
+ strscpy(rvu->kpu_pfl_name,
+ kpu_profile ? kpu_profile : default_pfl_name, KPU_NAME_LEN);
}
static int rvu_probe(struct pci_dev *pdev, const struct pci_device_id *id)
u16 maxlen;
u16 minlen;
- u8 pf_set_vf_cfg;
u8 mac_addr[ETH_ALEN]; /* MAC address of this PF/VF */
u8 default_mac[ETH_ALEN]; /* MAC address from FWdata */
- /* Broadcast pkt replication info */
+ /* Broadcast/Multicast/Promisc pkt replication info */
u16 bcast_mce_idx;
+ u16 mcast_mce_idx;
+ u16 promisc_mce_idx;
struct nix_mce_list bcast_mce_list;
+ struct nix_mce_list mcast_mce_list;
+ struct nix_mce_list promisc_mce_list;
+ bool use_mce_list;
struct rvu_npc_mcam_rule *def_ucast_rule;
u8 nix_blkaddr; /* BLKADDR_NIX0/1 assigned to this PF */
u8 nix_rx_intf; /* NIX0_RX/NIX1_RX interface to NPC */
u8 nix_tx_intf; /* NIX0_TX/NIX1_TX interface to NPC */
+ unsigned long flags;
};
+enum rvu_pfvf_flags {
+ NIXLF_INITIALIZED = 0,
+ PF_SET_VF_MAC,
+ PF_SET_VF_CFG,
+ PF_SET_VF_TRUSTED,
+};
+
+#define RVU_CLEAR_VF_PERM ~GENMASK(PF_SET_VF_TRUSTED, PF_SET_VF_MAC)
+
struct nix_txsch {
struct rsrc_bmap schq;
u8 lvl;
struct mutex rsrc_lock; /* Serialize resource alloc/free */
};
+struct nix_ipolicer {
+ struct rsrc_bmap band_prof;
+ u16 *pfvf_map;
+ u16 *match_id;
+ u16 *ref_count;
+};
+
struct nix_hw {
int blkaddr;
struct rvu *rvu;
struct nix_mark_format mark_format;
struct nix_lso lso;
struct nix_txvlan txvlan;
+ struct nix_ipolicer *ipolicer;
};
/* RVU block's capabilities or functionality,
bool nix_rx_multicast; /* Rx packet replication support */
bool per_pf_mbox_regs; /* PF mbox specified in per PF registers ? */
bool programmable_chans; /* Channels programmable ? */
+ bool ipolicer;
};
struct rvu_hwinfo {
const struct npc_kpu_profile_action *ikpu; /* array[pkinds] */
const struct npc_kpu_profile *kpu; /* array[kpus] */
struct npc_mcam_kex *mkex;
+ bool custom;
size_t pkinds;
size_t kpus;
};
struct mutex cgx_cfg_lock; /* serialize cgx configuration */
char mkex_pfl_name[MKEX_NAME_LEN]; /* Configured MKEX profile name */
+ char kpu_pfl_name[KPU_NAME_LEN]; /* Configured KPU profile name */
/* Firmware data */
struct rvu_fwdata *fwdata;
+ void *kpu_fwdata;
+ size_t kpu_fwdata_sz;
+ void __iomem *kpu_prfl_addr;
/* NPC KPU data */
struct npc_kpu_profile_adapter kpu;
/* Function Prototypes
* RVU
*/
-static inline int is_afvf(u16 pcifunc)
+static inline bool is_afvf(u16 pcifunc)
{
return !(pcifunc & ~RVU_PFVF_FUNC_MASK);
}
+static inline bool is_vf(u16 pcifunc)
+{
+ return !!(pcifunc & RVU_PFVF_FUNC_MASK);
+}
+
/* check if PF_FUNC is AF */
static inline bool is_pffunc_af(u16 pcifunc)
{
int rvu_alloc_bitmap(struct rsrc_bmap *rsrc);
int rvu_alloc_rsrc(struct rsrc_bmap *rsrc);
void rvu_free_rsrc(struct rsrc_bmap *rsrc, int id);
+bool is_rsrc_free(struct rsrc_bmap *rsrc, int id);
int rvu_rsrc_free_count(struct rsrc_bmap *rsrc);
int rvu_alloc_rsrc_contig(struct rsrc_bmap *rsrc, int nrsrc);
bool rvu_rsrc_check_contig(struct rsrc_bmap *rsrc, int nrsrc);
*lmac_id = (map & 0xF);
}
+static inline bool is_cgx_vf(struct rvu *rvu, u16 pcifunc)
+{
+ return ((pcifunc & RVU_PFVF_FUNC_MASK) &&
+ is_pf_cgxmapped(rvu, rvu_get_pf(pcifunc)));
+}
+
#define M(_name, _id, fn_name, req, rsp) \
int rvu_mbox_handler_ ## fn_name(struct rvu *, struct req *, struct rsp *);
MBOX_MESSAGES
int rvu_get_nixlf_count(struct rvu *rvu);
void rvu_nix_lf_teardown(struct rvu *rvu, u16 pcifunc, int blkaddr, int npalf);
int nix_get_nixlf(struct rvu *rvu, u16 pcifunc, int *nixlf, int *nix_blkaddr);
-int nix_update_bcast_mce_list(struct rvu *rvu, u16 pcifunc, bool add);
+int nix_update_mce_list(struct rvu *rvu, u16 pcifunc,
+ struct nix_mce_list *mce_list,
+ int mce_idx, int mcam_index, bool add);
+void nix_get_mce_list(struct rvu *rvu, u16 pcifunc, int type,
+ struct nix_mce_list **mce_list, int *mce_idx);
struct nix_hw *get_nix_hw(struct rvu_hwinfo *hw, int blkaddr);
int rvu_get_next_nix_blkaddr(struct rvu *rvu, int blkaddr);
void rvu_nix_reset_mac(struct rvu_pfvf *pfvf, int pcifunc);
+int nix_get_struct_ptrs(struct rvu *rvu, u16 pcifunc,
+ struct nix_hw **nix_hw, int *blkaddr);
+int rvu_nix_setup_ratelimit_aggr(struct rvu *rvu, u16 pcifunc,
+ u16 rq_idx, u16 match_id);
+int nix_aq_context_read(struct rvu *rvu, struct nix_hw *nix_hw,
+ struct nix_cn10k_aq_enq_req *aq_req,
+ struct nix_cn10k_aq_enq_rsp *aq_rsp,
+ u16 pcifunc, u8 ctype, u32 qidx);
/* NPC APIs */
int rvu_npc_init(struct rvu *rvu);
void rvu_npc_install_ucast_entry(struct rvu *rvu, u16 pcifunc,
int nixlf, u64 chan, u8 *mac_addr);
void rvu_npc_install_promisc_entry(struct rvu *rvu, u16 pcifunc,
- int nixlf, u64 chan, u8 chan_cnt,
- bool allmulti);
-void rvu_npc_disable_promisc_entry(struct rvu *rvu, u16 pcifunc, int nixlf);
-void rvu_npc_enable_promisc_entry(struct rvu *rvu, u16 pcifunc, int nixlf);
+ int nixlf, u64 chan, u8 chan_cnt);
+void rvu_npc_enable_promisc_entry(struct rvu *rvu, u16 pcifunc, int nixlf,
+ bool enable);
void rvu_npc_install_bcast_match_entry(struct rvu *rvu, u16 pcifunc,
int nixlf, u64 chan);
-void rvu_npc_enable_bcast_entry(struct rvu *rvu, u16 pcifunc, bool enable);
+void rvu_npc_enable_bcast_entry(struct rvu *rvu, u16 pcifunc, int nixlf,
+ bool enable);
+void rvu_npc_install_allmulti_entry(struct rvu *rvu, u16 pcifunc, int nixlf,
+ u64 chan);
+void rvu_npc_enable_allmulti_entry(struct rvu *rvu, u16 pcifunc, int nixlf,
+ bool enable);
+void npc_enadis_default_mce_entry(struct rvu *rvu, u16 pcifunc,
+ int nixlf, int type, bool enable);
void rvu_npc_disable_mcam_entries(struct rvu *rvu, u16 pcifunc, int nixlf);
void rvu_npc_free_mcam_entries(struct rvu *rvu, u16 pcifunc, int nixlf);
void rvu_npc_disable_default_entries(struct rvu *rvu, u16 pcifunc, int nixlf);
RVU_DEBUG_SEQ_FOPS(nix_qsize, nix_qsize_display, nix_qsize_write);
+static void print_band_prof_ctx(struct seq_file *m,
+ struct nix_bandprof_s *prof)
+{
+ char *str;
+
+ switch (prof->pc_mode) {
+ case NIX_RX_PC_MODE_VLAN:
+ str = "VLAN";
+ break;
+ case NIX_RX_PC_MODE_DSCP:
+ str = "DSCP";
+ break;
+ case NIX_RX_PC_MODE_GEN:
+ str = "Generic";
+ break;
+ case NIX_RX_PC_MODE_RSVD:
+ str = "Reserved";
+ break;
+ }
+ seq_printf(m, "W0: pc_mode\t\t%s\n", str);
+ str = (prof->icolor == 3) ? "Color blind" :
+ (prof->icolor == 0) ? "Green" :
+ (prof->icolor == 1) ? "Yellow" : "Red";
+ seq_printf(m, "W0: icolor\t\t%s\n", str);
+ seq_printf(m, "W0: tnl_ena\t\t%d\n", prof->tnl_ena);
+ seq_printf(m, "W0: peir_exponent\t%d\n", prof->peir_exponent);
+ seq_printf(m, "W0: pebs_exponent\t%d\n", prof->pebs_exponent);
+ seq_printf(m, "W0: cir_exponent\t%d\n", prof->cir_exponent);
+ seq_printf(m, "W0: cbs_exponent\t%d\n", prof->cbs_exponent);
+ seq_printf(m, "W0: peir_mantissa\t%d\n", prof->peir_mantissa);
+ seq_printf(m, "W0: pebs_mantissa\t%d\n", prof->pebs_mantissa);
+ seq_printf(m, "W0: cir_mantissa\t%d\n", prof->cir_mantissa);
+
+ seq_printf(m, "W1: cbs_mantissa\t%d\n", prof->cbs_mantissa);
+ str = (prof->lmode == 0) ? "byte" : "packet";
+ seq_printf(m, "W1: lmode\t\t%s\n", str);
+ seq_printf(m, "W1: l_select\t\t%d\n", prof->l_sellect);
+ seq_printf(m, "W1: rdiv\t\t%d\n", prof->rdiv);
+ seq_printf(m, "W1: adjust_exponent\t%d\n", prof->adjust_exponent);
+ seq_printf(m, "W1: adjust_mantissa\t%d\n", prof->adjust_mantissa);
+ str = (prof->gc_action == 0) ? "PASS" :
+ (prof->gc_action == 1) ? "DROP" : "RED";
+ seq_printf(m, "W1: gc_action\t\t%s\n", str);
+ str = (prof->yc_action == 0) ? "PASS" :
+ (prof->yc_action == 1) ? "DROP" : "RED";
+ seq_printf(m, "W1: yc_action\t\t%s\n", str);
+ str = (prof->rc_action == 0) ? "PASS" :
+ (prof->rc_action == 1) ? "DROP" : "RED";
+ seq_printf(m, "W1: rc_action\t\t%s\n", str);
+ seq_printf(m, "W1: meter_algo\t\t%d\n", prof->meter_algo);
+ seq_printf(m, "W1: band_prof_id\t%d\n", prof->band_prof_id);
+ seq_printf(m, "W1: hl_en\t\t%d\n", prof->hl_en);
+
+ seq_printf(m, "W2: ts\t\t\t%lld\n", (u64)prof->ts);
+ seq_printf(m, "W3: pe_accum\t\t%d\n", prof->pe_accum);
+ seq_printf(m, "W3: c_accum\t\t%d\n", prof->c_accum);
+ seq_printf(m, "W4: green_pkt_pass\t%lld\n",
+ (u64)prof->green_pkt_pass);
+ seq_printf(m, "W5: yellow_pkt_pass\t%lld\n",
+ (u64)prof->yellow_pkt_pass);
+ seq_printf(m, "W6: red_pkt_pass\t%lld\n", (u64)prof->red_pkt_pass);
+ seq_printf(m, "W7: green_octs_pass\t%lld\n",
+ (u64)prof->green_octs_pass);
+ seq_printf(m, "W8: yellow_octs_pass\t%lld\n",
+ (u64)prof->yellow_octs_pass);
+ seq_printf(m, "W9: red_octs_pass\t%lld\n", (u64)prof->red_octs_pass);
+ seq_printf(m, "W10: green_pkt_drop\t%lld\n",
+ (u64)prof->green_pkt_drop);
+ seq_printf(m, "W11: yellow_pkt_drop\t%lld\n",
+ (u64)prof->yellow_pkt_drop);
+ seq_printf(m, "W12: red_pkt_drop\t%lld\n", (u64)prof->red_pkt_drop);
+ seq_printf(m, "W13: green_octs_drop\t%lld\n",
+ (u64)prof->green_octs_drop);
+ seq_printf(m, "W14: yellow_octs_drop\t%lld\n",
+ (u64)prof->yellow_octs_drop);
+ seq_printf(m, "W15: red_octs_drop\t%lld\n", (u64)prof->red_octs_drop);
+ seq_puts(m, "==============================\n");
+}
+
+static int rvu_dbg_nix_band_prof_ctx_display(struct seq_file *m, void *unused)
+{
+ struct nix_hw *nix_hw = m->private;
+ struct nix_cn10k_aq_enq_req aq_req;
+ struct nix_cn10k_aq_enq_rsp aq_rsp;
+ struct rvu *rvu = nix_hw->rvu;
+ struct nix_ipolicer *ipolicer;
+ int layer, prof_idx, idx, rc;
+ u16 pcifunc;
+ char *str;
+
+ for (layer = 0; layer < BAND_PROF_NUM_LAYERS; layer++) {
+ if (layer == BAND_PROF_INVAL_LAYER)
+ continue;
+ str = (layer == BAND_PROF_LEAF_LAYER) ? "Leaf" :
+ (layer == BAND_PROF_MID_LAYER) ? "Mid" : "Top";
+
+ seq_printf(m, "\n%s bandwidth profiles\n", str);
+ seq_puts(m, "=======================\n");
+
+ ipolicer = &nix_hw->ipolicer[layer];
+
+ for (idx = 0; idx < ipolicer->band_prof.max; idx++) {
+ if (is_rsrc_free(&ipolicer->band_prof, idx))
+ continue;
+
+ prof_idx = (idx & 0x3FFF) | (layer << 14);
+ rc = nix_aq_context_read(rvu, nix_hw, &aq_req, &aq_rsp,
+ 0x00, NIX_AQ_CTYPE_BANDPROF,
+ prof_idx);
+ if (rc) {
+ dev_err(rvu->dev,
+ "%s: Failed to fetch context of %s profile %d, err %d\n",
+ __func__, str, idx, rc);
+ return 0;
+ }
+ seq_printf(m, "\n%s bandwidth profile:: %d\n", str, idx);
+ pcifunc = ipolicer->pfvf_map[idx];
+ if (!(pcifunc & RVU_PFVF_FUNC_MASK))
+ seq_printf(m, "Allocated to :: PF %d\n",
+ rvu_get_pf(pcifunc));
+ else
+ seq_printf(m, "Allocated to :: PF %d VF %d\n",
+ rvu_get_pf(pcifunc),
+ (pcifunc & RVU_PFVF_FUNC_MASK) - 1);
+ print_band_prof_ctx(m, &aq_rsp.prof);
+ }
+ }
+ return 0;
+}
+
+RVU_DEBUG_SEQ_FOPS(nix_band_prof_ctx, nix_band_prof_ctx_display, NULL);
+
+static int rvu_dbg_nix_band_prof_rsrc_display(struct seq_file *m, void *unused)
+{
+ struct nix_hw *nix_hw = m->private;
+ struct nix_ipolicer *ipolicer;
+ int layer;
+ char *str;
+
+ seq_puts(m, "\nBandwidth profile resource free count\n");
+ seq_puts(m, "=====================================\n");
+ for (layer = 0; layer < BAND_PROF_NUM_LAYERS; layer++) {
+ if (layer == BAND_PROF_INVAL_LAYER)
+ continue;
+ str = (layer == BAND_PROF_LEAF_LAYER) ? "Leaf" :
+ (layer == BAND_PROF_MID_LAYER) ? "Mid " : "Top ";
+
+ ipolicer = &nix_hw->ipolicer[layer];
+ seq_printf(m, "%s :: Max: %4d Free: %4d\n", str,
+ ipolicer->band_prof.max,
+ rvu_rsrc_free_count(&ipolicer->band_prof));
+ }
+ seq_puts(m, "=====================================\n");
+
+ return 0;
+}
+
+RVU_DEBUG_SEQ_FOPS(nix_band_prof_rsrc, nix_band_prof_rsrc_display, NULL);
+
static void rvu_dbg_nix_init(struct rvu *rvu, int blkaddr)
{
struct nix_hw *nix_hw;
&rvu_dbg_nix_ndc_rx_hits_miss_fops);
debugfs_create_file("qsize", 0600, rvu->rvu_dbg.nix, rvu,
&rvu_dbg_nix_qsize_fops);
+ debugfs_create_file("ingress_policer_ctx", 0600, rvu->rvu_dbg.nix, nix_hw,
+ &rvu_dbg_nix_band_prof_ctx_fops);
+ debugfs_create_file("ingress_policer_rsrc", 0600, rvu->rvu_dbg.nix, nix_hw,
+ &rvu_dbg_nix_band_prof_rsrc_fops);
}
static void rvu_dbg_npa_init(struct rvu *rvu)
struct rvu *rvu = s->private;
struct npc_mcam *mcam;
int pf, vf = -1;
+ bool enabled;
int blkaddr;
u16 target;
u64 hits;
}
rvu_dbg_npc_mcam_show_action(s, iter);
- seq_printf(s, "\tenabled: %s\n", iter->enable ? "yes" : "no");
+
+ enabled = is_mcam_entry_enabled(rvu, mcam, blkaddr, iter->entry);
+ seq_printf(s, "\tenabled: %s\n", enabled ? "yes" : "no");
if (!iter->has_cntr)
continue;
static void nix_free_tx_vtag_entries(struct rvu *rvu, u16 pcifunc);
static int rvu_nix_get_bpid(struct rvu *rvu, struct nix_bp_cfg_req *req,
int type, int chan_id);
+static int nix_update_mce_rule(struct rvu *rvu, u16 pcifunc,
+ int type, bool add);
+static int nix_setup_ipolicers(struct rvu *rvu,
+ struct nix_hw *nix_hw, int blkaddr);
+static void nix_ipolicer_freemem(struct nix_hw *nix_hw);
+static int nix_verify_bandprof(struct nix_cn10k_aq_enq_req *req,
+ struct nix_hw *nix_hw, u16 pcifunc);
+static int nix_free_all_bandprof(struct rvu *rvu, u16 pcifunc);
+static void nix_clear_ratelimit_aggr(struct rvu *rvu, struct nix_hw *nix_hw,
+ u32 leaf_prof);
enum mc_tbl_sz {
MC_TBL_SZ_256,
return 0;
}
+int nix_get_struct_ptrs(struct rvu *rvu, u16 pcifunc,
+ struct nix_hw **nix_hw, int *blkaddr)
+{
+ struct rvu_pfvf *pfvf;
+
+ pfvf = rvu_get_pfvf(rvu, pcifunc);
+ *blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NIX, pcifunc);
+ if (!pfvf->nixlf || *blkaddr < 0)
+ return NIX_AF_ERR_AF_LF_INVALID;
+
+ *nix_hw = get_nix_hw(rvu->hw, *blkaddr);
+ if (!*nix_hw)
+ return NIX_AF_ERR_INVALID_NIXBLK;
+ return 0;
+}
+
static void nix_mce_list_init(struct nix_mce_list *list, int max)
{
INIT_HLIST_HEAD(&list->head);
pfvf->tx_chan_cnt = 1;
rvu_npc_install_promisc_entry(rvu, pcifunc, nixlf,
pfvf->rx_chan_base,
- pfvf->rx_chan_cnt, false);
+ pfvf->rx_chan_cnt);
break;
}
pfvf->rx_chan_base, pfvf->mac_addr);
/* Add this PF_FUNC to bcast pkt replication list */
- err = nix_update_bcast_mce_list(rvu, pcifunc, true);
+ err = nix_update_mce_rule(rvu, pcifunc, NIXLF_BCAST_ENTRY, true);
if (err) {
dev_err(rvu->dev,
"Bcast list, failed to enable PF_FUNC 0x%x\n",
pcifunc);
return err;
}
-
+ /* Install MCAM rule matching Ethernet broadcast mac address */
rvu_npc_install_bcast_match_entry(rvu, pcifunc,
nixlf, pfvf->rx_chan_base);
+
pfvf->maxlen = NIC_HW_MIN_FRS;
pfvf->minlen = NIC_HW_MIN_FRS;
pfvf->minlen = 0;
/* Remove this PF_FUNC from bcast pkt replication list */
- err = nix_update_bcast_mce_list(rvu, pcifunc, false);
+ err = nix_update_mce_rule(rvu, pcifunc, NIXLF_BCAST_ENTRY, false);
if (err) {
dev_err(rvu->dev,
"Bcast list, failed to disable PF_FUNC 0x%x\n",
pfvf = rvu_get_pfvf(rvu, pcifunc);
nixlf = rvu_get_lf(rvu, block, pcifunc, 0);
- /* Skip NIXLF check for broadcast MCE entry init */
- if (!(!rsp && req->ctype == NIX_AQ_CTYPE_MCE)) {
+ /* Skip NIXLF check for broadcast MCE entry and bandwidth profile
+ * operations done by AF itself.
+ */
+ if (!((!rsp && req->ctype == NIX_AQ_CTYPE_MCE) ||
+ (req->ctype == NIX_AQ_CTYPE_BANDPROF && !pcifunc))) {
if (!pfvf->nixlf || nixlf < 0)
return NIX_AF_ERR_AF_LF_INVALID;
}
if (rsp)
rc = NIX_AF_ERR_AQ_ENQUEUE;
break;
+ case NIX_AQ_CTYPE_BANDPROF:
+ if (nix_verify_bandprof((struct nix_cn10k_aq_enq_req *)req,
+ nix_hw, pcifunc))
+ rc = NIX_AF_ERR_INVALID_BANDPROF;
+ break;
default:
rc = NIX_AF_ERR_AQ_ENQUEUE;
}
else if (req->ctype == NIX_AQ_CTYPE_MCE)
memcpy(mask, &req->mce_mask,
sizeof(struct nix_rx_mce_s));
+ else if (req->ctype == NIX_AQ_CTYPE_BANDPROF)
+ memcpy(mask, &req->prof_mask,
+ sizeof(struct nix_bandprof_s));
fallthrough;
case NIX_AQ_INSTOP_INIT:
if (req->ctype == NIX_AQ_CTYPE_RQ)
memcpy(ctx, &req->rss, sizeof(struct nix_rsse_s));
else if (req->ctype == NIX_AQ_CTYPE_MCE)
memcpy(ctx, &req->mce, sizeof(struct nix_rx_mce_s));
+ else if (req->ctype == NIX_AQ_CTYPE_BANDPROF)
+ memcpy(ctx, &req->prof, sizeof(struct nix_bandprof_s));
break;
case NIX_AQ_INSTOP_NOP:
case NIX_AQ_INSTOP_READ:
else if (req->ctype == NIX_AQ_CTYPE_MCE)
memcpy(&rsp->mce, ctx,
sizeof(struct nix_rx_mce_s));
+ else if (req->ctype == NIX_AQ_CTYPE_BANDPROF)
+ memcpy(&rsp->prof, ctx,
+ sizeof(struct nix_bandprof_s));
}
}
aq_req.op = op;
aq_req.qidx = mce;
- /* Forward bcast pkts to RQ0, RSS not needed */
- aq_req.mce.op = 0;
+ /* Use RSS with RSS index 0 */
+ aq_req.mce.op = 1;
aq_req.mce.index = 0;
aq_req.mce.eol = eol;
aq_req.mce.pf_func = pcifunc;
return 0;
}
-static int nix_update_mce_list(struct nix_mce_list *mce_list,
- u16 pcifunc, bool add)
+static int nix_update_mce_list_entry(struct nix_mce_list *mce_list,
+ u16 pcifunc, bool add)
{
struct mce *mce, *tail = NULL;
bool delete = false;
if (mce->pcifunc == pcifunc && !add) {
delete = true;
break;
+ } else if (mce->pcifunc == pcifunc && add) {
+ /* entry already exists */
+ return 0;
}
tail = mce;
}
return 0;
}
-int nix_update_bcast_mce_list(struct rvu *rvu, u16 pcifunc, bool add)
+int nix_update_mce_list(struct rvu *rvu, u16 pcifunc,
+ struct nix_mce_list *mce_list,
+ int mce_idx, int mcam_index, bool add)
{
- int err = 0, idx, next_idx, last_idx;
- struct nix_mce_list *mce_list;
+ int err = 0, idx, next_idx, last_idx, blkaddr, npc_blkaddr;
+ struct npc_mcam *mcam = &rvu->hw->mcam;
struct nix_mcast *mcast;
struct nix_hw *nix_hw;
- struct rvu_pfvf *pfvf;
struct mce *mce;
- int blkaddr;
-
- /* Broadcast pkt replication is not needed for AF's VFs, hence skip */
- if (is_afvf(pcifunc))
- return 0;
-
- blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NIX, pcifunc);
- if (blkaddr < 0)
- return 0;
- nix_hw = get_nix_hw(rvu->hw, blkaddr);
- if (!nix_hw)
- return 0;
-
- mcast = &nix_hw->mcast;
+ if (!mce_list)
+ return -EINVAL;
/* Get this PF/VF func's MCE index */
- pfvf = rvu_get_pfvf(rvu, pcifunc & ~RVU_PFVF_FUNC_MASK);
- idx = pfvf->bcast_mce_idx + (pcifunc & RVU_PFVF_FUNC_MASK);
+ idx = mce_idx + (pcifunc & RVU_PFVF_FUNC_MASK);
- mce_list = &pfvf->bcast_mce_list;
- if (idx > (pfvf->bcast_mce_idx + mce_list->max)) {
+ if (idx > (mce_idx + mce_list->max)) {
dev_err(rvu->dev,
"%s: Idx %d > max MCE idx %d, for PF%d bcast list\n",
__func__, idx, mce_list->max,
return -EINVAL;
}
+ err = nix_get_struct_ptrs(rvu, pcifunc, &nix_hw, &blkaddr);
+ if (err)
+ return err;
+
+ mcast = &nix_hw->mcast;
mutex_lock(&mcast->mce_lock);
- err = nix_update_mce_list(mce_list, pcifunc, add);
+ err = nix_update_mce_list_entry(mce_list, pcifunc, add);
if (err)
goto end;
/* Disable MCAM entry in NPC */
if (!mce_list->count) {
- rvu_npc_enable_bcast_entry(rvu, pcifunc, false);
+ npc_blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NPC, 0);
+ npc_enable_mcam_entry(rvu, mcam, npc_blkaddr, mcam_index, false);
goto end;
}
/* Dump the updated list to HW */
- idx = pfvf->bcast_mce_idx;
+ idx = mce_idx;
last_idx = idx + mce_list->count - 1;
hlist_for_each_entry(mce, &mce_list->head, node) {
if (idx > last_idx)
return err;
}
-static int nix_setup_bcast_tables(struct rvu *rvu, struct nix_hw *nix_hw)
+void nix_get_mce_list(struct rvu *rvu, u16 pcifunc, int type,
+ struct nix_mce_list **mce_list, int *mce_idx)
+{
+ struct rvu_hwinfo *hw = rvu->hw;
+ struct rvu_pfvf *pfvf;
+
+ if (!hw->cap.nix_rx_multicast ||
+ !is_pf_cgxmapped(rvu, rvu_get_pf(pcifunc & ~RVU_PFVF_FUNC_MASK))) {
+ *mce_list = NULL;
+ *mce_idx = 0;
+ return;
+ }
+
+ /* Get this PF/VF func's MCE index */
+ pfvf = rvu_get_pfvf(rvu, pcifunc & ~RVU_PFVF_FUNC_MASK);
+
+ if (type == NIXLF_BCAST_ENTRY) {
+ *mce_list = &pfvf->bcast_mce_list;
+ *mce_idx = pfvf->bcast_mce_idx;
+ } else if (type == NIXLF_ALLMULTI_ENTRY) {
+ *mce_list = &pfvf->mcast_mce_list;
+ *mce_idx = pfvf->mcast_mce_idx;
+ } else if (type == NIXLF_PROMISC_ENTRY) {
+ *mce_list = &pfvf->promisc_mce_list;
+ *mce_idx = pfvf->promisc_mce_idx;
+ } else {
+ *mce_list = NULL;
+ *mce_idx = 0;
+ }
+}
+
+static int nix_update_mce_rule(struct rvu *rvu, u16 pcifunc,
+ int type, bool add)
+{
+ int err = 0, nixlf, blkaddr, mcam_index, mce_idx;
+ struct npc_mcam *mcam = &rvu->hw->mcam;
+ struct rvu_hwinfo *hw = rvu->hw;
+ struct nix_mce_list *mce_list;
+
+ /* skip multicast pkt replication for AF's VFs */
+ if (is_afvf(pcifunc))
+ return 0;
+
+ if (!hw->cap.nix_rx_multicast)
+ return 0;
+
+ blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NIX, pcifunc);
+ if (blkaddr < 0)
+ return -EINVAL;
+
+ nixlf = rvu_get_lf(rvu, &hw->block[blkaddr], pcifunc, 0);
+ if (nixlf < 0)
+ return -EINVAL;
+
+ nix_get_mce_list(rvu, pcifunc, type, &mce_list, &mce_idx);
+
+ mcam_index = npc_get_nixlf_mcam_index(mcam,
+ pcifunc & ~RVU_PFVF_FUNC_MASK,
+ nixlf, type);
+ err = nix_update_mce_list(rvu, pcifunc, mce_list,
+ mce_idx, mcam_index, add);
+ return err;
+}
+
+static int nix_setup_mce_tables(struct rvu *rvu, struct nix_hw *nix_hw)
{
struct nix_mcast *mcast = &nix_hw->mcast;
int err, pf, numvfs, idx;
if (pfvf->nix_blkaddr != nix_hw->blkaddr)
continue;
- /* Save the start MCE */
+ /* save start idx of broadcast mce list */
pfvf->bcast_mce_idx = nix_alloc_mce_list(mcast, numvfs + 1);
-
nix_mce_list_init(&pfvf->bcast_mce_list, numvfs + 1);
+ /* save start idx of multicast mce list */
+ pfvf->mcast_mce_idx = nix_alloc_mce_list(mcast, numvfs + 1);
+ nix_mce_list_init(&pfvf->mcast_mce_list, numvfs + 1);
+
+ /* save the start idx of promisc mce list */
+ pfvf->promisc_mce_idx = nix_alloc_mce_list(mcast, numvfs + 1);
+ nix_mce_list_init(&pfvf->promisc_mce_list, numvfs + 1);
+
for (idx = 0; idx < (numvfs + 1); idx++) {
/* idx-0 is for PF, followed by VFs */
pcifunc = (pf << RVU_PFVF_PF_SHIFT);
pcifunc, 0, true);
if (err)
return err;
+
+ /* add dummy entries to multicast mce list */
+ err = nix_blk_setup_mce(rvu, nix_hw,
+ pfvf->mcast_mce_idx + idx,
+ NIX_AQ_INSTOP_INIT,
+ pcifunc, 0, true);
+ if (err)
+ return err;
+
+ /* add dummy entries to promisc mce list */
+ err = nix_blk_setup_mce(rvu, nix_hw,
+ pfvf->promisc_mce_idx + idx,
+ NIX_AQ_INSTOP_INIT,
+ pcifunc, 0, true);
+ if (err)
+ return err;
}
}
return 0;
mutex_init(&mcast->mce_lock);
- return nix_setup_bcast_tables(rvu, nix_hw);
+ return nix_setup_mce_tables(rvu, nix_hw);
}
static int nix_setup_txvlan(struct rvu *rvu, struct nix_hw *nix_hw)
pfvf = rvu_get_pfvf(rvu, pcifunc);
- /* VF can't overwrite admin(PF) changes */
- if (from_vf && pfvf->pf_set_vf_cfg)
+ /* untrusted VF can't overwrite admin(PF) changes */
+ if (!test_bit(PF_SET_VF_TRUSTED, &pfvf->flags) &&
+ (from_vf && test_bit(PF_SET_VF_MAC, &pfvf->flags))) {
+ dev_warn(rvu->dev,
+ "MAC address set by admin(PF) cannot be overwritten by untrusted VF");
return -EPERM;
+ }
ether_addr_copy(pfvf->mac_addr, req->mac_addr);
rvu_npc_install_ucast_entry(rvu, pcifunc, nixlf,
pfvf->rx_chan_base, req->mac_addr);
+ if (test_bit(PF_SET_VF_TRUSTED, &pfvf->flags) && from_vf)
+ ether_addr_copy(pfvf->default_mac, req->mac_addr);
+
return 0;
}
int rvu_mbox_handler_nix_set_rx_mode(struct rvu *rvu, struct nix_rx_mode *req,
struct msg_rsp *rsp)
{
- bool allmulti = false, disable_promisc = false;
+ bool allmulti, promisc, nix_rx_multicast;
u16 pcifunc = req->hdr.pcifunc;
- int blkaddr, nixlf, err;
struct rvu_pfvf *pfvf;
+ int nixlf, err;
- err = nix_get_nixlf(rvu, pcifunc, &nixlf, &blkaddr);
+ pfvf = rvu_get_pfvf(rvu, pcifunc);
+ promisc = req->mode & NIX_RX_MODE_PROMISC ? true : false;
+ allmulti = req->mode & NIX_RX_MODE_ALLMULTI ? true : false;
+ pfvf->use_mce_list = req->mode & NIX_RX_MODE_USE_MCE ? true : false;
+
+ nix_rx_multicast = rvu->hw->cap.nix_rx_multicast & pfvf->use_mce_list;
+
+ if (is_vf(pcifunc) && !nix_rx_multicast &&
+ (promisc || allmulti)) {
+ dev_warn_ratelimited(rvu->dev,
+ "VF promisc/multicast not supported\n");
+ return 0;
+ }
+
+ /* untrusted VF can't configure promisc/allmulti */
+ if (is_vf(pcifunc) && !test_bit(PF_SET_VF_TRUSTED, &pfvf->flags) &&
+ (promisc || allmulti))
+ return 0;
+
+ err = nix_get_nixlf(rvu, pcifunc, &nixlf, NULL);
if (err)
return err;
- pfvf = rvu_get_pfvf(rvu, pcifunc);
+ if (nix_rx_multicast) {
+ /* add/del this PF_FUNC to/from mcast pkt replication list */
+ err = nix_update_mce_rule(rvu, pcifunc, NIXLF_ALLMULTI_ENTRY,
+ allmulti);
+ if (err) {
+ dev_err(rvu->dev,
+ "Failed to update pcifunc 0x%x to multicast list\n",
+ pcifunc);
+ return err;
+ }
- if (req->mode & NIX_RX_MODE_PROMISC)
- allmulti = false;
- else if (req->mode & NIX_RX_MODE_ALLMULTI)
- allmulti = true;
- else
- disable_promisc = true;
+ /* add/del this PF_FUNC to/from promisc pkt replication list */
+ err = nix_update_mce_rule(rvu, pcifunc, NIXLF_PROMISC_ENTRY,
+ promisc);
+ if (err) {
+ dev_err(rvu->dev,
+ "Failed to update pcifunc 0x%x to promisc list\n",
+ pcifunc);
+ return err;
+ }
+ }
- if (disable_promisc)
- rvu_npc_disable_promisc_entry(rvu, pcifunc, nixlf);
- else
+ /* install/uninstall allmulti entry */
+ if (allmulti) {
+ rvu_npc_install_allmulti_entry(rvu, pcifunc, nixlf,
+ pfvf->rx_chan_base);
+ } else {
+ if (!nix_rx_multicast)
+ rvu_npc_enable_allmulti_entry(rvu, pcifunc, nixlf, false);
+ }
+
+ /* install/uninstall promisc entry */
+ if (promisc) {
rvu_npc_install_promisc_entry(rvu, pcifunc, nixlf,
pfvf->rx_chan_base,
- pfvf->rx_chan_cnt, allmulti);
+ pfvf->rx_chan_cnt);
+ } else {
+ if (!nix_rx_multicast)
+ rvu_npc_enable_promisc_entry(rvu, pcifunc, nixlf, false);
+ }
+
return 0;
}
if (err)
return err;
+ err = nix_setup_ipolicers(rvu, nix_hw, blkaddr);
+ if (err)
+ return err;
+
err = nix_af_mark_format_setup(rvu, nix_hw, blkaddr);
if (err)
return err;
(ltdefs->rx_isctp.lid << 8) | (ltdefs->rx_isctp.ltype_match << 4) |
ltdefs->rx_isctp.ltype_mask);
+ if (!is_rvu_otx2(rvu)) {
+ /* Enable APAD calculation for other protocols
+ * matching APAD0 and APAD1 lt def registers.
+ */
+ rvu_write64(rvu, blkaddr, NIX_AF_RX_DEF_CST_APAD0,
+ (ltdefs->rx_apad0.valid << 11) |
+ (ltdefs->rx_apad0.lid << 8) |
+ (ltdefs->rx_apad0.ltype_match << 4) |
+ ltdefs->rx_apad0.ltype_mask);
+ rvu_write64(rvu, blkaddr, NIX_AF_RX_DEF_CST_APAD1,
+ (ltdefs->rx_apad1.valid << 11) |
+ (ltdefs->rx_apad1.lid << 8) |
+ (ltdefs->rx_apad1.ltype_match << 4) |
+ ltdefs->rx_apad1.ltype_mask);
+
+ /* Receive ethertype defination register defines layer
+ * information in NPC_RESULT_S to identify the Ethertype
+ * location in L2 header. Used for Ethertype overwriting
+ * in inline IPsec flow.
+ */
+ rvu_write64(rvu, blkaddr, NIX_AF_RX_DEF_ET(0),
+ (ltdefs->rx_et[0].offset << 12) |
+ (ltdefs->rx_et[0].valid << 11) |
+ (ltdefs->rx_et[0].lid << 8) |
+ (ltdefs->rx_et[0].ltype_match << 4) |
+ ltdefs->rx_et[0].ltype_mask);
+ rvu_write64(rvu, blkaddr, NIX_AF_RX_DEF_ET(1),
+ (ltdefs->rx_et[1].offset << 12) |
+ (ltdefs->rx_et[1].valid << 11) |
+ (ltdefs->rx_et[1].lid << 8) |
+ (ltdefs->rx_et[1].ltype_match << 4) |
+ ltdefs->rx_et[1].ltype_mask);
+ }
+
err = nix_rx_flowkey_alg_cfg(rvu, blkaddr);
if (err)
return err;
kfree(txsch->schq.bmap);
}
+ nix_ipolicer_freemem(nix_hw);
+
vlan = &nix_hw->txvlan;
kfree(vlan->rsrc.bmap);
mutex_destroy(&vlan->rsrc_lock);
struct msg_rsp *rsp)
{
u16 pcifunc = req->hdr.pcifunc;
+ struct rvu_pfvf *pfvf;
int nixlf, err;
err = nix_get_nixlf(rvu, pcifunc, &nixlf, NULL);
npc_mcam_enable_flows(rvu, pcifunc);
+ pfvf = rvu_get_pfvf(rvu, pcifunc);
+ set_bit(NIXLF_INITIALIZED, &pfvf->flags);
+
return rvu_cgx_start_stop_io(rvu, pcifunc, true);
}
struct msg_rsp *rsp)
{
u16 pcifunc = req->hdr.pcifunc;
+ struct rvu_pfvf *pfvf;
int nixlf, err;
err = nix_get_nixlf(rvu, pcifunc, &nixlf, NULL);
rvu_npc_disable_mcam_entries(rvu, pcifunc, nixlf);
+ pfvf = rvu_get_pfvf(rvu, pcifunc);
+ clear_bit(NIXLF_INITIALIZED, &pfvf->flags);
+
return rvu_cgx_start_stop_io(rvu, pcifunc, false);
}
nix_rx_sync(rvu, blkaddr);
nix_txschq_free(rvu, pcifunc);
+ clear_bit(NIXLF_INITIALIZED, &pfvf->flags);
+
rvu_cgx_start_stop_io(rvu, pcifunc, false);
if (pfvf->sq_ctx) {
}
nix_ctx_free(rvu, pfvf);
+
+ nix_free_all_bandprof(rvu, pcifunc);
}
#define NIX_AF_LFX_TX_CFG_PTP_EN BIT_ULL(32)
if (from_vf)
ether_addr_copy(pfvf->mac_addr, pfvf->default_mac);
}
+
+/* NIX ingress policers or bandwidth profiles APIs */
+static void nix_config_rx_pkt_policer_precolor(struct rvu *rvu, int blkaddr)
+{
+ struct npc_lt_def_cfg defs, *ltdefs;
+
+ ltdefs = &defs;
+ memcpy(ltdefs, rvu->kpu.lt_def, sizeof(struct npc_lt_def_cfg));
+
+ /* Extract PCP and DEI fields from outer VLAN from byte offset
+ * 2 from the start of LB_PTR (ie TAG).
+ * VLAN0 is Outer VLAN and VLAN1 is Inner VLAN. Inner VLAN
+ * fields are considered when 'Tunnel enable' is set in profile.
+ */
+ rvu_write64(rvu, blkaddr, NIX_AF_RX_DEF_VLAN0_PCP_DEI,
+ (2UL << 12) | (ltdefs->ovlan.lid << 8) |
+ (ltdefs->ovlan.ltype_match << 4) |
+ ltdefs->ovlan.ltype_mask);
+ rvu_write64(rvu, blkaddr, NIX_AF_RX_DEF_VLAN1_PCP_DEI,
+ (2UL << 12) | (ltdefs->ivlan.lid << 8) |
+ (ltdefs->ivlan.ltype_match << 4) |
+ ltdefs->ivlan.ltype_mask);
+
+ /* DSCP field in outer and tunneled IPv4 packets */
+ rvu_write64(rvu, blkaddr, NIX_AF_RX_DEF_OIP4_DSCP,
+ (1UL << 12) | (ltdefs->rx_oip4.lid << 8) |
+ (ltdefs->rx_oip4.ltype_match << 4) |
+ ltdefs->rx_oip4.ltype_mask);
+ rvu_write64(rvu, blkaddr, NIX_AF_RX_DEF_IIP4_DSCP,
+ (1UL << 12) | (ltdefs->rx_iip4.lid << 8) |
+ (ltdefs->rx_iip4.ltype_match << 4) |
+ ltdefs->rx_iip4.ltype_mask);
+
+ /* DSCP field (traffic class) in outer and tunneled IPv6 packets */
+ rvu_write64(rvu, blkaddr, NIX_AF_RX_DEF_OIP6_DSCP,
+ (1UL << 11) | (ltdefs->rx_oip6.lid << 8) |
+ (ltdefs->rx_oip6.ltype_match << 4) |
+ ltdefs->rx_oip6.ltype_mask);
+ rvu_write64(rvu, blkaddr, NIX_AF_RX_DEF_IIP6_DSCP,
+ (1UL << 11) | (ltdefs->rx_iip6.lid << 8) |
+ (ltdefs->rx_iip6.ltype_match << 4) |
+ ltdefs->rx_iip6.ltype_mask);
+}
+
+static int nix_init_policer_context(struct rvu *rvu, struct nix_hw *nix_hw,
+ int layer, int prof_idx)
+{
+ struct nix_cn10k_aq_enq_req aq_req;
+ int rc;
+
+ memset(&aq_req, 0, sizeof(struct nix_cn10k_aq_enq_req));
+
+ aq_req.qidx = (prof_idx & 0x3FFF) | (layer << 14);
+ aq_req.ctype = NIX_AQ_CTYPE_BANDPROF;
+ aq_req.op = NIX_AQ_INSTOP_INIT;
+
+ /* Context is all zeros, submit to AQ */
+ rc = rvu_nix_blk_aq_enq_inst(rvu, nix_hw,
+ (struct nix_aq_enq_req *)&aq_req, NULL);
+ if (rc)
+ dev_err(rvu->dev, "Failed to INIT bandwidth profile layer %d profile %d\n",
+ layer, prof_idx);
+ return rc;
+}
+
+static int nix_setup_ipolicers(struct rvu *rvu,
+ struct nix_hw *nix_hw, int blkaddr)
+{
+ struct rvu_hwinfo *hw = rvu->hw;
+ struct nix_ipolicer *ipolicer;
+ int err, layer, prof_idx;
+ u64 cfg;
+
+ cfg = rvu_read64(rvu, blkaddr, NIX_AF_CONST);
+ if (!(cfg & BIT_ULL(61))) {
+ hw->cap.ipolicer = false;
+ return 0;
+ }
+
+ hw->cap.ipolicer = true;
+ nix_hw->ipolicer = devm_kcalloc(rvu->dev, BAND_PROF_NUM_LAYERS,
+ sizeof(*ipolicer), GFP_KERNEL);
+ if (!nix_hw->ipolicer)
+ return -ENOMEM;
+
+ cfg = rvu_read64(rvu, blkaddr, NIX_AF_PL_CONST);
+
+ for (layer = 0; layer < BAND_PROF_NUM_LAYERS; layer++) {
+ ipolicer = &nix_hw->ipolicer[layer];
+ switch (layer) {
+ case BAND_PROF_LEAF_LAYER:
+ ipolicer->band_prof.max = cfg & 0XFFFF;
+ break;
+ case BAND_PROF_MID_LAYER:
+ ipolicer->band_prof.max = (cfg >> 16) & 0XFFFF;
+ break;
+ case BAND_PROF_TOP_LAYER:
+ ipolicer->band_prof.max = (cfg >> 32) & 0XFFFF;
+ break;
+ }
+
+ if (!ipolicer->band_prof.max)
+ continue;
+
+ err = rvu_alloc_bitmap(&ipolicer->band_prof);
+ if (err)
+ return err;
+
+ ipolicer->pfvf_map = devm_kcalloc(rvu->dev,
+ ipolicer->band_prof.max,
+ sizeof(u16), GFP_KERNEL);
+ if (!ipolicer->pfvf_map)
+ return -ENOMEM;
+
+ ipolicer->match_id = devm_kcalloc(rvu->dev,
+ ipolicer->band_prof.max,
+ sizeof(u16), GFP_KERNEL);
+ if (!ipolicer->match_id)
+ return -ENOMEM;
+
+ for (prof_idx = 0;
+ prof_idx < ipolicer->band_prof.max; prof_idx++) {
+ /* Set AF as current owner for INIT ops to succeed */
+ ipolicer->pfvf_map[prof_idx] = 0x00;
+
+ /* There is no enable bit in the profile context,
+ * so no context disable. So let's INIT them here
+ * so that PF/VF later on have to just do WRITE to
+ * setup policer rates and config.
+ */
+ err = nix_init_policer_context(rvu, nix_hw,
+ layer, prof_idx);
+ if (err)
+ return err;
+ }
+
+ /* Allocate memory for maintaining ref_counts for MID level
+ * profiles, this will be needed for leaf layer profiles'
+ * aggregation.
+ */
+ if (layer != BAND_PROF_MID_LAYER)
+ continue;
+
+ ipolicer->ref_count = devm_kcalloc(rvu->dev,
+ ipolicer->band_prof.max,
+ sizeof(u16), GFP_KERNEL);
+ }
+
+ /* Set policer timeunit to 2us ie (19 + 1) * 100 nsec = 2us */
+ rvu_write64(rvu, blkaddr, NIX_AF_PL_TS, 19);
+
+ nix_config_rx_pkt_policer_precolor(rvu, blkaddr);
+
+ return 0;
+}
+
+static void nix_ipolicer_freemem(struct nix_hw *nix_hw)
+{
+ struct nix_ipolicer *ipolicer;
+ int layer;
+
+ for (layer = 0; layer < BAND_PROF_NUM_LAYERS; layer++) {
+ ipolicer = &nix_hw->ipolicer[layer];
+
+ if (!ipolicer->band_prof.max)
+ continue;
+
+ kfree(ipolicer->band_prof.bmap);
+ }
+}
+
+static int nix_verify_bandprof(struct nix_cn10k_aq_enq_req *req,
+ struct nix_hw *nix_hw, u16 pcifunc)
+{
+ struct nix_ipolicer *ipolicer;
+ int layer, hi_layer, prof_idx;
+
+ /* Bits [15:14] in profile index represent layer */
+ layer = (req->qidx >> 14) & 0x03;
+ prof_idx = req->qidx & 0x3FFF;
+
+ ipolicer = &nix_hw->ipolicer[layer];
+ if (prof_idx >= ipolicer->band_prof.max)
+ return -EINVAL;
+
+ /* Check if the profile is allocated to the requesting PCIFUNC or not
+ * with the exception of AF. AF is allowed to read and update contexts.
+ */
+ if (pcifunc && ipolicer->pfvf_map[prof_idx] != pcifunc)
+ return -EINVAL;
+
+ /* If this profile is linked to higher layer profile then check
+ * if that profile is also allocated to the requesting PCIFUNC
+ * or not.
+ */
+ if (!req->prof.hl_en)
+ return 0;
+
+ /* Leaf layer profile can link only to mid layer and
+ * mid layer to top layer.
+ */
+ if (layer == BAND_PROF_LEAF_LAYER)
+ hi_layer = BAND_PROF_MID_LAYER;
+ else if (layer == BAND_PROF_MID_LAYER)
+ hi_layer = BAND_PROF_TOP_LAYER;
+ else
+ return -EINVAL;
+
+ ipolicer = &nix_hw->ipolicer[hi_layer];
+ prof_idx = req->prof.band_prof_id;
+ if (prof_idx >= ipolicer->band_prof.max ||
+ ipolicer->pfvf_map[prof_idx] != pcifunc)
+ return -EINVAL;
+
+ return 0;
+}
+
+int rvu_mbox_handler_nix_bandprof_alloc(struct rvu *rvu,
+ struct nix_bandprof_alloc_req *req,
+ struct nix_bandprof_alloc_rsp *rsp)
+{
+ int blkaddr, layer, prof, idx, err;
+ u16 pcifunc = req->hdr.pcifunc;
+ struct nix_ipolicer *ipolicer;
+ struct nix_hw *nix_hw;
+
+ if (!rvu->hw->cap.ipolicer)
+ return NIX_AF_ERR_IPOLICER_NOTSUPP;
+
+ err = nix_get_struct_ptrs(rvu, pcifunc, &nix_hw, &blkaddr);
+ if (err)
+ return err;
+
+ mutex_lock(&rvu->rsrc_lock);
+ for (layer = 0; layer < BAND_PROF_NUM_LAYERS; layer++) {
+ if (layer == BAND_PROF_INVAL_LAYER)
+ continue;
+ if (!req->prof_count[layer])
+ continue;
+
+ ipolicer = &nix_hw->ipolicer[layer];
+ for (idx = 0; idx < req->prof_count[layer]; idx++) {
+ /* Allocate a max of 'MAX_BANDPROF_PER_PFFUNC' profiles */
+ if (idx == MAX_BANDPROF_PER_PFFUNC)
+ break;
+
+ prof = rvu_alloc_rsrc(&ipolicer->band_prof);
+ if (prof < 0)
+ break;
+ rsp->prof_count[layer]++;
+ rsp->prof_idx[layer][idx] = prof;
+ ipolicer->pfvf_map[prof] = pcifunc;
+ }
+ }
+ mutex_unlock(&rvu->rsrc_lock);
+ return 0;
+}
+
+static int nix_free_all_bandprof(struct rvu *rvu, u16 pcifunc)
+{
+ int blkaddr, layer, prof_idx, err;
+ struct nix_ipolicer *ipolicer;
+ struct nix_hw *nix_hw;
+
+ if (!rvu->hw->cap.ipolicer)
+ return NIX_AF_ERR_IPOLICER_NOTSUPP;
+
+ err = nix_get_struct_ptrs(rvu, pcifunc, &nix_hw, &blkaddr);
+ if (err)
+ return err;
+
+ mutex_lock(&rvu->rsrc_lock);
+ /* Free all the profiles allocated to the PCIFUNC */
+ for (layer = 0; layer < BAND_PROF_NUM_LAYERS; layer++) {
+ if (layer == BAND_PROF_INVAL_LAYER)
+ continue;
+ ipolicer = &nix_hw->ipolicer[layer];
+
+ for (prof_idx = 0; prof_idx < ipolicer->band_prof.max; prof_idx++) {
+ if (ipolicer->pfvf_map[prof_idx] != pcifunc)
+ continue;
+
+ /* Clear ratelimit aggregation, if any */
+ if (layer == BAND_PROF_LEAF_LAYER &&
+ ipolicer->match_id[prof_idx])
+ nix_clear_ratelimit_aggr(rvu, nix_hw, prof_idx);
+
+ ipolicer->pfvf_map[prof_idx] = 0x00;
+ ipolicer->match_id[prof_idx] = 0;
+ rvu_free_rsrc(&ipolicer->band_prof, prof_idx);
+ }
+ }
+ mutex_unlock(&rvu->rsrc_lock);
+ return 0;
+}
+
+int rvu_mbox_handler_nix_bandprof_free(struct rvu *rvu,
+ struct nix_bandprof_free_req *req,
+ struct msg_rsp *rsp)
+{
+ int blkaddr, layer, prof_idx, idx, err;
+ u16 pcifunc = req->hdr.pcifunc;
+ struct nix_ipolicer *ipolicer;
+ struct nix_hw *nix_hw;
+
+ if (req->free_all)
+ return nix_free_all_bandprof(rvu, pcifunc);
+
+ if (!rvu->hw->cap.ipolicer)
+ return NIX_AF_ERR_IPOLICER_NOTSUPP;
+
+ err = nix_get_struct_ptrs(rvu, pcifunc, &nix_hw, &blkaddr);
+ if (err)
+ return err;
+
+ mutex_lock(&rvu->rsrc_lock);
+ /* Free the requested profile indices */
+ for (layer = 0; layer < BAND_PROF_NUM_LAYERS; layer++) {
+ if (layer == BAND_PROF_INVAL_LAYER)
+ continue;
+ if (!req->prof_count[layer])
+ continue;
+
+ ipolicer = &nix_hw->ipolicer[layer];
+ for (idx = 0; idx < req->prof_count[layer]; idx++) {
+ prof_idx = req->prof_idx[layer][idx];
+ if (prof_idx >= ipolicer->band_prof.max ||
+ ipolicer->pfvf_map[prof_idx] != pcifunc)
+ continue;
+
+ /* Clear ratelimit aggregation, if any */
+ if (layer == BAND_PROF_LEAF_LAYER &&
+ ipolicer->match_id[prof_idx])
+ nix_clear_ratelimit_aggr(rvu, nix_hw, prof_idx);
+
+ ipolicer->pfvf_map[prof_idx] = 0x00;
+ ipolicer->match_id[prof_idx] = 0;
+ rvu_free_rsrc(&ipolicer->band_prof, prof_idx);
+ if (idx == MAX_BANDPROF_PER_PFFUNC)
+ break;
+ }
+ }
+ mutex_unlock(&rvu->rsrc_lock);
+ return 0;
+}
+
+int nix_aq_context_read(struct rvu *rvu, struct nix_hw *nix_hw,
+ struct nix_cn10k_aq_enq_req *aq_req,
+ struct nix_cn10k_aq_enq_rsp *aq_rsp,
+ u16 pcifunc, u8 ctype, u32 qidx)
+{
+ memset(aq_req, 0, sizeof(struct nix_cn10k_aq_enq_req));
+ aq_req->hdr.pcifunc = pcifunc;
+ aq_req->ctype = ctype;
+ aq_req->op = NIX_AQ_INSTOP_READ;
+ aq_req->qidx = qidx;
+
+ return rvu_nix_blk_aq_enq_inst(rvu, nix_hw,
+ (struct nix_aq_enq_req *)aq_req,
+ (struct nix_aq_enq_rsp *)aq_rsp);
+}
+
+static int nix_ipolicer_map_leaf_midprofs(struct rvu *rvu,
+ struct nix_hw *nix_hw,
+ struct nix_cn10k_aq_enq_req *aq_req,
+ struct nix_cn10k_aq_enq_rsp *aq_rsp,
+ u32 leaf_prof, u16 mid_prof)
+{
+ memset(aq_req, 0, sizeof(struct nix_cn10k_aq_enq_req));
+ aq_req->hdr.pcifunc = 0x00;
+ aq_req->ctype = NIX_AQ_CTYPE_BANDPROF;
+ aq_req->op = NIX_AQ_INSTOP_WRITE;
+ aq_req->qidx = leaf_prof;
+
+ aq_req->prof.band_prof_id = mid_prof;
+ aq_req->prof_mask.band_prof_id = GENMASK(6, 0);
+ aq_req->prof.hl_en = 1;
+ aq_req->prof_mask.hl_en = 1;
+
+ return rvu_nix_blk_aq_enq_inst(rvu, nix_hw,
+ (struct nix_aq_enq_req *)aq_req,
+ (struct nix_aq_enq_rsp *)aq_rsp);
+}
+
+int rvu_nix_setup_ratelimit_aggr(struct rvu *rvu, u16 pcifunc,
+ u16 rq_idx, u16 match_id)
+{
+ int leaf_prof, mid_prof, leaf_match;
+ struct nix_cn10k_aq_enq_req aq_req;
+ struct nix_cn10k_aq_enq_rsp aq_rsp;
+ struct nix_ipolicer *ipolicer;
+ struct nix_hw *nix_hw;
+ int blkaddr, idx, rc;
+
+ if (!rvu->hw->cap.ipolicer)
+ return 0;
+
+ rc = nix_get_struct_ptrs(rvu, pcifunc, &nix_hw, &blkaddr);
+ if (rc)
+ return rc;
+
+ /* Fetch the RQ's context to see if policing is enabled */
+ rc = nix_aq_context_read(rvu, nix_hw, &aq_req, &aq_rsp, pcifunc,
+ NIX_AQ_CTYPE_RQ, rq_idx);
+ if (rc) {
+ dev_err(rvu->dev,
+ "%s: Failed to fetch RQ%d context of PFFUNC 0x%x\n",
+ __func__, rq_idx, pcifunc);
+ return rc;
+ }
+
+ if (!aq_rsp.rq.policer_ena)
+ return 0;
+
+ /* Get the bandwidth profile ID mapped to this RQ */
+ leaf_prof = aq_rsp.rq.band_prof_id;
+
+ ipolicer = &nix_hw->ipolicer[BAND_PROF_LEAF_LAYER];
+ ipolicer->match_id[leaf_prof] = match_id;
+
+ /* Check if any other leaf profile is marked with same match_id */
+ for (idx = 0; idx < ipolicer->band_prof.max; idx++) {
+ if (idx == leaf_prof)
+ continue;
+ if (ipolicer->match_id[idx] != match_id)
+ continue;
+
+ leaf_match = idx;
+ break;
+ }
+
+ if (idx == ipolicer->band_prof.max)
+ return 0;
+
+ /* Fetch the matching profile's context to check if it's already
+ * mapped to a mid level profile.
+ */
+ rc = nix_aq_context_read(rvu, nix_hw, &aq_req, &aq_rsp, 0x00,
+ NIX_AQ_CTYPE_BANDPROF, leaf_match);
+ if (rc) {
+ dev_err(rvu->dev,
+ "%s: Failed to fetch context of leaf profile %d\n",
+ __func__, leaf_match);
+ return rc;
+ }
+
+ ipolicer = &nix_hw->ipolicer[BAND_PROF_MID_LAYER];
+ if (aq_rsp.prof.hl_en) {
+ /* Get Mid layer prof index and map leaf_prof index
+ * also such that flows that are being steered
+ * to different RQs and marked with same match_id
+ * are rate limited in a aggregate fashion
+ */
+ mid_prof = aq_rsp.prof.band_prof_id;
+ rc = nix_ipolicer_map_leaf_midprofs(rvu, nix_hw,
+ &aq_req, &aq_rsp,
+ leaf_prof, mid_prof);
+ if (rc) {
+ dev_err(rvu->dev,
+ "%s: Failed to map leaf(%d) and mid(%d) profiles\n",
+ __func__, leaf_prof, mid_prof);
+ goto exit;
+ }
+
+ mutex_lock(&rvu->rsrc_lock);
+ ipolicer->ref_count[mid_prof]++;
+ mutex_unlock(&rvu->rsrc_lock);
+ goto exit;
+ }
+
+ /* Allocate a mid layer profile and
+ * map both 'leaf_prof' and 'leaf_match' profiles to it.
+ */
+ mutex_lock(&rvu->rsrc_lock);
+ mid_prof = rvu_alloc_rsrc(&ipolicer->band_prof);
+ if (mid_prof < 0) {
+ dev_err(rvu->dev,
+ "%s: Unable to allocate mid layer profile\n", __func__);
+ mutex_unlock(&rvu->rsrc_lock);
+ goto exit;
+ }
+ mutex_unlock(&rvu->rsrc_lock);
+ ipolicer->pfvf_map[mid_prof] = 0x00;
+ ipolicer->ref_count[mid_prof] = 0;
+
+ /* Initialize mid layer profile same as 'leaf_prof' */
+ rc = nix_aq_context_read(rvu, nix_hw, &aq_req, &aq_rsp, 0x00,
+ NIX_AQ_CTYPE_BANDPROF, leaf_prof);
+ if (rc) {
+ dev_err(rvu->dev,
+ "%s: Failed to fetch context of leaf profile %d\n",
+ __func__, leaf_prof);
+ goto exit;
+ }
+
+ memset(&aq_req, 0, sizeof(struct nix_cn10k_aq_enq_req));
+ aq_req.hdr.pcifunc = 0x00;
+ aq_req.qidx = (mid_prof & 0x3FFF) | (BAND_PROF_MID_LAYER << 14);
+ aq_req.ctype = NIX_AQ_CTYPE_BANDPROF;
+ aq_req.op = NIX_AQ_INSTOP_WRITE;
+ memcpy(&aq_req.prof, &aq_rsp.prof, sizeof(struct nix_bandprof_s));
+ /* Clear higher layer enable bit in the mid profile, just in case */
+ aq_req.prof.hl_en = 0;
+ aq_req.prof_mask.hl_en = 1;
+
+ rc = rvu_nix_blk_aq_enq_inst(rvu, nix_hw,
+ (struct nix_aq_enq_req *)&aq_req, NULL);
+ if (rc) {
+ dev_err(rvu->dev,
+ "%s: Failed to INIT context of mid layer profile %d\n",
+ __func__, mid_prof);
+ goto exit;
+ }
+
+ /* Map both leaf profiles to this mid layer profile */
+ rc = nix_ipolicer_map_leaf_midprofs(rvu, nix_hw,
+ &aq_req, &aq_rsp,
+ leaf_prof, mid_prof);
+ if (rc) {
+ dev_err(rvu->dev,
+ "%s: Failed to map leaf(%d) and mid(%d) profiles\n",
+ __func__, leaf_prof, mid_prof);
+ goto exit;
+ }
+
+ mutex_lock(&rvu->rsrc_lock);
+ ipolicer->ref_count[mid_prof]++;
+ mutex_unlock(&rvu->rsrc_lock);
+
+ rc = nix_ipolicer_map_leaf_midprofs(rvu, nix_hw,
+ &aq_req, &aq_rsp,
+ leaf_match, mid_prof);
+ if (rc) {
+ dev_err(rvu->dev,
+ "%s: Failed to map leaf(%d) and mid(%d) profiles\n",
+ __func__, leaf_match, mid_prof);
+ ipolicer->ref_count[mid_prof]--;
+ goto exit;
+ }
+
+ mutex_lock(&rvu->rsrc_lock);
+ ipolicer->ref_count[mid_prof]++;
+ mutex_unlock(&rvu->rsrc_lock);
+
+exit:
+ return rc;
+}
+
+/* Called with mutex rsrc_lock */
+static void nix_clear_ratelimit_aggr(struct rvu *rvu, struct nix_hw *nix_hw,
+ u32 leaf_prof)
+{
+ struct nix_cn10k_aq_enq_req aq_req;
+ struct nix_cn10k_aq_enq_rsp aq_rsp;
+ struct nix_ipolicer *ipolicer;
+ u16 mid_prof;
+ int rc;
+
+ mutex_unlock(&rvu->rsrc_lock);
+
+ rc = nix_aq_context_read(rvu, nix_hw, &aq_req, &aq_rsp, 0x00,
+ NIX_AQ_CTYPE_BANDPROF, leaf_prof);
+
+ mutex_lock(&rvu->rsrc_lock);
+ if (rc) {
+ dev_err(rvu->dev,
+ "%s: Failed to fetch context of leaf profile %d\n",
+ __func__, leaf_prof);
+ return;
+ }
+
+ if (!aq_rsp.prof.hl_en)
+ return;
+
+ mid_prof = aq_rsp.prof.band_prof_id;
+ ipolicer = &nix_hw->ipolicer[BAND_PROF_MID_LAYER];
+ ipolicer->ref_count[mid_prof]--;
+ /* If ref_count is zero, free mid layer profile */
+ if (!ipolicer->ref_count[mid_prof]) {
+ ipolicer->pfvf_map[mid_prof] = 0x00;
+ rvu_free_rsrc(&ipolicer->band_prof, mid_prof);
+ }
+}
#include "cgx.h"
#include "npc_profile.h"
-#define RSVD_MCAM_ENTRIES_PER_PF 2 /* Bcast & Promisc */
+#define RSVD_MCAM_ENTRIES_PER_PF 3 /* Broadcast, Promisc and AllMulticast */
#define RSVD_MCAM_ENTRIES_PER_NIXLF 1 /* Ucast for LFs */
#define NPC_PARSE_RESULT_DMAC_OFFSET 8
#define NPC_KEX_CHAN_MASK 0xFFFULL
#define NPC_KEX_PF_FUNC_MASK 0xFFFFULL
+#define ALIGN_8B_CEIL(__a) (((__a) + 7) & (-8))
+
static const char def_pfl_name[] = "default";
static void npc_mcam_free_all_entries(struct rvu *rvu, struct npc_mcam *mcam,
*/
if (type == NIXLF_BCAST_ENTRY)
return index;
- else if (type == NIXLF_PROMISC_ENTRY)
+ else if (type == NIXLF_ALLMULTI_ENTRY)
return index + 1;
+ else if (type == NIXLF_PROMISC_ENTRY)
+ return index + 2;
}
return npc_get_ucast_mcam_index(mcam, pcifunc, nixlf);
}
}
-static void npc_get_default_entry_action(struct rvu *rvu, struct npc_mcam *mcam,
- int blkaddr, int index,
- struct mcam_entry *entry)
+static u64 npc_get_default_entry_action(struct rvu *rvu, struct npc_mcam *mcam,
+ int blkaddr, u16 pf_func)
+{
+ int bank, nixlf, index;
+
+ /* get ucast entry rule entry index */
+ nix_get_nixlf(rvu, pf_func, &nixlf, NULL);
+ index = npc_get_nixlf_mcam_index(mcam, pf_func, nixlf,
+ NIXLF_UCAST_ENTRY);
+ bank = npc_get_bank(mcam, index);
+ index &= (mcam->banksize - 1);
+
+ return rvu_read64(rvu, blkaddr,
+ NPC_AF_MCAMEX_BANKX_ACTION(index, bank));
+}
+
+static void npc_fixup_vf_rule(struct rvu *rvu, struct npc_mcam *mcam,
+ int blkaddr, int index, struct mcam_entry *entry,
+ bool *enable)
{
u16 owner, target_func;
struct rvu_pfvf *pfvf;
- int bank, nixlf;
u64 rx_action;
owner = mcam->entry2pfvf_map[index];
target_func = (entry->action >> 4) & 0xffff;
- /* return incase target is PF or LBK or rule owner is not PF */
+ /* do nothing when target is LBK/PF or owner is not PF */
if (is_afvf(target_func) || (owner & RVU_PFVF_FUNC_MASK) ||
!(target_func & RVU_PFVF_FUNC_MASK))
return;
+ /* save entry2target_pffunc */
pfvf = rvu_get_pfvf(rvu, target_func);
mcam->entry2target_pffunc[index] = target_func;
- /* return if nixlf is not attached or initialized */
- if (!is_nixlf_attached(rvu, target_func) || !pfvf->def_ucast_rule)
- return;
- /* get VF ucast entry rule */
- nix_get_nixlf(rvu, target_func, &nixlf, NULL);
- index = npc_get_nixlf_mcam_index(mcam, target_func,
- nixlf, NIXLF_UCAST_ENTRY);
- bank = npc_get_bank(mcam, index);
- index &= (mcam->banksize - 1);
+ /* don't enable rule when nixlf not attached or initialized */
+ if (!(is_nixlf_attached(rvu, target_func) &&
+ test_bit(NIXLF_INITIALIZED, &pfvf->flags)))
+ *enable = false;
- rx_action = rvu_read64(rvu, blkaddr,
- NPC_AF_MCAMEX_BANKX_ACTION(index, bank));
+ /* copy VF default entry action to the VF mcam entry */
+ rx_action = npc_get_default_entry_action(rvu, mcam, blkaddr,
+ target_func);
if (rx_action)
entry->action = rx_action;
}
NPC_AF_MCAMEX_BANKX_CAMX_W1(index, bank, 0), cam0);
}
- /* copy VF default entry action to the VF mcam entry */
+ /* PF installing VF rule */
if (intf == NIX_INTF_RX && actindex < mcam->bmap_entries)
- npc_get_default_entry_action(rvu, mcam, blkaddr, actindex,
- entry);
+ npc_fixup_vf_rule(rvu, mcam, blkaddr, index, entry, &enable);
/* Set 'action' */
rvu_write64(rvu, blkaddr,
}
void rvu_npc_install_promisc_entry(struct rvu *rvu, u16 pcifunc,
- int nixlf, u64 chan, u8 chan_cnt,
- bool allmulti)
+ int nixlf, u64 chan, u8 chan_cnt)
{
struct rvu_pfvf *pfvf = rvu_get_pfvf(rvu, pcifunc);
struct npc_install_flow_req req = { 0 };
struct npc_install_flow_rsp rsp = { 0 };
struct npc_mcam *mcam = &rvu->hw->mcam;
+ struct rvu_hwinfo *hw = rvu->hw;
int blkaddr, ucast_idx, index;
- u8 mac_addr[ETH_ALEN] = { 0 };
struct nix_rx_action action;
u64 relaxed_mask;
- /* Only PF or AF VF can add a promiscuous entry */
- if ((pcifunc & RVU_PFVF_FUNC_MASK) && !is_afvf(pcifunc))
+ if (!hw->cap.nix_rx_multicast && is_cgx_vf(rvu, pcifunc))
return;
blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NPC, 0);
if (blkaddr < 0)
return;
- *(u64 *)&action = 0x00;
index = npc_get_nixlf_mcam_index(mcam, pcifunc,
nixlf, NIXLF_PROMISC_ENTRY);
+ if (is_cgx_vf(rvu, pcifunc))
+ index = npc_get_nixlf_mcam_index(mcam,
+ pcifunc & ~RVU_PFVF_FUNC_MASK,
+ nixlf, NIXLF_PROMISC_ENTRY);
+
/* If the corresponding PF's ucast action is RSS,
* use the same action for promisc also
*/
nixlf, NIXLF_UCAST_ENTRY);
if (is_mcam_entry_enabled(rvu, mcam, blkaddr, ucast_idx))
*(u64 *)&action = npc_get_mcam_action(rvu, mcam,
- blkaddr, ucast_idx);
+ blkaddr, ucast_idx);
if (action.op != NIX_RX_ACTIONOP_RSS) {
*(u64 *)&action = 0x00;
action.op = NIX_RX_ACTIONOP_UCAST;
- action.pf_func = pcifunc;
}
- if (allmulti) {
- mac_addr[0] = 0x01; /* LSB bit of 1st byte in DMAC */
- ether_addr_copy(req.packet.dmac, mac_addr);
- ether_addr_copy(req.mask.dmac, mac_addr);
- req.features = BIT_ULL(NPC_DMAC);
+ /* RX_ACTION set to MCAST for CGX PF's */
+ if (hw->cap.nix_rx_multicast && pfvf->use_mce_list &&
+ is_pf_cgxmapped(rvu, rvu_get_pf(pcifunc))) {
+ *(u64 *)&action = 0x00;
+ action.op = NIX_RX_ACTIONOP_MCAST;
+ pfvf = rvu_get_pfvf(rvu, pcifunc & ~RVU_PFVF_FUNC_MASK);
+ action.index = pfvf->promisc_mce_idx;
}
req.chan_mask = 0xFFFU;
rvu_mbox_handler_npc_install_flow(rvu, &req, &rsp);
}
-static void npc_enadis_promisc_entry(struct rvu *rvu, u16 pcifunc,
- int nixlf, bool enable)
+void rvu_npc_enable_promisc_entry(struct rvu *rvu, u16 pcifunc,
+ int nixlf, bool enable)
{
struct npc_mcam *mcam = &rvu->hw->mcam;
int blkaddr, index;
if (blkaddr < 0)
return;
- /* Only PF's have a promiscuous entry */
- if (pcifunc & RVU_PFVF_FUNC_MASK)
- return;
+ /* Get 'pcifunc' of PF device */
+ pcifunc = pcifunc & ~RVU_PFVF_FUNC_MASK;
index = npc_get_nixlf_mcam_index(mcam, pcifunc,
nixlf, NIXLF_PROMISC_ENTRY);
npc_enable_mcam_entry(rvu, mcam, blkaddr, index, enable);
}
-void rvu_npc_disable_promisc_entry(struct rvu *rvu, u16 pcifunc, int nixlf)
-{
- npc_enadis_promisc_entry(rvu, pcifunc, nixlf, false);
-}
-
-void rvu_npc_enable_promisc_entry(struct rvu *rvu, u16 pcifunc, int nixlf)
-{
- npc_enadis_promisc_entry(rvu, pcifunc, nixlf, true);
-}
-
void rvu_npc_install_bcast_match_entry(struct rvu *rvu, u16 pcifunc,
int nixlf, u64 chan)
{
struct npc_mcam *mcam = &rvu->hw->mcam;
struct rvu_hwinfo *hw = rvu->hw;
int blkaddr, index;
- u32 req_index = 0;
- u8 op;
blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NPC, 0);
if (blkaddr < 0)
/* If pkt replication is not supported,
* then only PF is allowed to add a bcast match entry.
*/
- if (!hw->cap.nix_rx_multicast && pcifunc & RVU_PFVF_FUNC_MASK)
+ if (!hw->cap.nix_rx_multicast && is_vf(pcifunc))
return;
/* Get 'pcifunc' of PF device */
* so install entry with UCAST action, so that PF
* receives all broadcast packets.
*/
- op = NIX_RX_ACTIONOP_UCAST;
+ req.op = NIX_RX_ACTIONOP_UCAST;
} else {
- op = NIX_RX_ACTIONOP_MCAST;
- req_index = pfvf->bcast_mce_idx;
+ req.op = NIX_RX_ACTIONOP_MCAST;
+ req.index = pfvf->bcast_mce_idx;
}
eth_broadcast_addr((u8 *)&req.packet.dmac);
req.channel = chan;
req.intf = pfvf->nix_rx_intf;
req.entry = index;
- req.op = op;
req.hdr.pcifunc = 0; /* AF is requester */
req.vf = pcifunc;
- req.index = req_index;
rvu_mbox_handler_npc_install_flow(rvu, &req, &rsp);
}
-void rvu_npc_enable_bcast_entry(struct rvu *rvu, u16 pcifunc, bool enable)
+void rvu_npc_enable_bcast_entry(struct rvu *rvu, u16 pcifunc, int nixlf,
+ bool enable)
{
struct npc_mcam *mcam = &rvu->hw->mcam;
int blkaddr, index;
/* Get 'pcifunc' of PF device */
pcifunc = pcifunc & ~RVU_PFVF_FUNC_MASK;
- index = npc_get_nixlf_mcam_index(mcam, pcifunc, 0, NIXLF_BCAST_ENTRY);
+ index = npc_get_nixlf_mcam_index(mcam, pcifunc, nixlf,
+ NIXLF_BCAST_ENTRY);
+ npc_enable_mcam_entry(rvu, mcam, blkaddr, index, enable);
+}
+
+void rvu_npc_install_allmulti_entry(struct rvu *rvu, u16 pcifunc, int nixlf,
+ u64 chan)
+{
+ struct npc_install_flow_req req = { 0 };
+ struct npc_install_flow_rsp rsp = { 0 };
+ struct npc_mcam *mcam = &rvu->hw->mcam;
+ struct rvu_hwinfo *hw = rvu->hw;
+ int blkaddr, ucast_idx, index;
+ u8 mac_addr[ETH_ALEN] = { 0 };
+ struct nix_rx_action action;
+ struct rvu_pfvf *pfvf;
+ u16 vf_func;
+
+ /* Only CGX PF/VF can add allmulticast entry */
+ if (is_afvf(pcifunc))
+ return;
+
+ blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NPC, 0);
+ if (blkaddr < 0)
+ return;
+
+ /* Get 'pcifunc' of PF device */
+ vf_func = pcifunc & RVU_PFVF_FUNC_MASK;
+ pcifunc = pcifunc & ~RVU_PFVF_FUNC_MASK;
+ pfvf = rvu_get_pfvf(rvu, pcifunc);
+ index = npc_get_nixlf_mcam_index(mcam, pcifunc,
+ nixlf, NIXLF_ALLMULTI_ENTRY);
+
+ /* If the corresponding PF's ucast action is RSS,
+ * use the same action for multicast entry also
+ */
+ ucast_idx = npc_get_nixlf_mcam_index(mcam, pcifunc,
+ nixlf, NIXLF_UCAST_ENTRY);
+ if (is_mcam_entry_enabled(rvu, mcam, blkaddr, ucast_idx))
+ *(u64 *)&action = npc_get_mcam_action(rvu, mcam,
+ blkaddr, ucast_idx);
+
+ if (action.op != NIX_RX_ACTIONOP_RSS) {
+ *(u64 *)&action = 0x00;
+ action.op = NIX_RX_ACTIONOP_UCAST;
+ action.pf_func = pcifunc;
+ }
+
+ /* RX_ACTION set to MCAST for CGX PF's */
+ if (hw->cap.nix_rx_multicast && pfvf->use_mce_list) {
+ *(u64 *)&action = 0x00;
+ action.op = NIX_RX_ACTIONOP_MCAST;
+ action.index = pfvf->mcast_mce_idx;
+ }
+
+ mac_addr[0] = 0x01; /* LSB bit of 1st byte in DMAC */
+ ether_addr_copy(req.packet.dmac, mac_addr);
+ ether_addr_copy(req.mask.dmac, mac_addr);
+ req.features = BIT_ULL(NPC_DMAC);
+
+ /* For cn10k the upper two bits of the channel number are
+ * cpt channel number. with masking out these bits in the
+ * mcam entry, same entry used for NIX will allow packets
+ * received from cpt for parsing.
+ */
+ if (!is_rvu_otx2(rvu))
+ req.chan_mask = NIX_CHAN_CPT_X2P_MASK;
+ else
+ req.chan_mask = 0xFFFU;
+
+ req.channel = chan;
+ req.intf = pfvf->nix_rx_intf;
+ req.entry = index;
+ req.op = action.op;
+ req.hdr.pcifunc = 0; /* AF is requester */
+ req.vf = pcifunc | vf_func;
+ req.index = action.index;
+ req.match_id = action.match_id;
+ req.flow_key_alg = action.flow_key_alg;
+
+ rvu_mbox_handler_npc_install_flow(rvu, &req, &rsp);
+}
+
+void rvu_npc_enable_allmulti_entry(struct rvu *rvu, u16 pcifunc, int nixlf,
+ bool enable)
+{
+ struct npc_mcam *mcam = &rvu->hw->mcam;
+ int blkaddr, index;
+
+ blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NPC, 0);
+ if (blkaddr < 0)
+ return;
+
+ /* Get 'pcifunc' of PF device */
+ pcifunc = pcifunc & ~RVU_PFVF_FUNC_MASK;
+
+ index = npc_get_nixlf_mcam_index(mcam, pcifunc, nixlf,
+ NIXLF_ALLMULTI_ENTRY);
npc_enable_mcam_entry(rvu, mcam, blkaddr, index, enable);
}
int group, int alg_idx, int mcam_index)
{
struct npc_mcam *mcam = &rvu->hw->mcam;
+ struct rvu_hwinfo *hw = rvu->hw;
struct nix_rx_action action;
int blkaddr, index, bank;
struct rvu_pfvf *pfvf;
/* If PF's promiscuous entry is enabled,
* Set RSS action for that entry as well
*/
- if (is_mcam_entry_enabled(rvu, mcam, blkaddr, index)) {
+ if ((!hw->cap.nix_rx_multicast || !pfvf->use_mce_list) &&
+ is_mcam_entry_enabled(rvu, mcam, blkaddr, index)) {
bank = npc_get_bank(mcam, index);
index &= (mcam->banksize - 1);
}
}
+void npc_enadis_default_mce_entry(struct rvu *rvu, u16 pcifunc,
+ int nixlf, int type, bool enable)
+{
+ struct npc_mcam *mcam = &rvu->hw->mcam;
+ struct rvu_hwinfo *hw = rvu->hw;
+ struct nix_mce_list *mce_list;
+ int index, blkaddr, mce_idx;
+ struct rvu_pfvf *pfvf;
+
+ blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NPC, 0);
+ if (blkaddr < 0)
+ return;
+
+ index = npc_get_nixlf_mcam_index(mcam, pcifunc & ~RVU_PFVF_FUNC_MASK,
+ nixlf, type);
+
+ /* disable MCAM entry when packet replication is not supported by hw */
+ if (!hw->cap.nix_rx_multicast && !is_vf(pcifunc)) {
+ npc_enable_mcam_entry(rvu, mcam, blkaddr, index, enable);
+ return;
+ }
+
+ /* return incase mce list is not enabled */
+ pfvf = rvu_get_pfvf(rvu, pcifunc & ~RVU_PFVF_FUNC_MASK);
+ if (hw->cap.nix_rx_multicast && is_vf(pcifunc) &&
+ type != NIXLF_BCAST_ENTRY && !pfvf->use_mce_list)
+ return;
+
+ nix_get_mce_list(rvu, pcifunc, type, &mce_list, &mce_idx);
+
+ nix_update_mce_list(rvu, pcifunc, mce_list,
+ mce_idx, index, enable);
+ if (enable)
+ npc_enable_mcam_entry(rvu, mcam, blkaddr, index, enable);
+}
+
static void npc_enadis_default_entries(struct rvu *rvu, u16 pcifunc,
int nixlf, bool enable)
{
struct npc_mcam *mcam = &rvu->hw->mcam;
- struct nix_rx_action action;
- int index, bank, blkaddr;
+ int index, blkaddr;
blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NPC, 0);
if (blkaddr < 0)
nixlf, NIXLF_UCAST_ENTRY);
npc_enable_mcam_entry(rvu, mcam, blkaddr, index, enable);
- /* For PF, ena/dis promisc and bcast MCAM match entries.
- * For VFs add/delete from bcast list when RX multicast
- * feature is present.
+ /* Nothing to do for VFs, on platforms where pkt replication
+ * is not supported
*/
- if (pcifunc & RVU_PFVF_FUNC_MASK && !rvu->hw->cap.nix_rx_multicast)
+ if ((pcifunc & RVU_PFVF_FUNC_MASK) && !rvu->hw->cap.nix_rx_multicast)
return;
- /* For bcast, enable/disable only if it's action is not
- * packet replication, incase if action is replication
- * then this PF/VF's nixlf is removed from bcast replication
- * list.
- */
- index = npc_get_nixlf_mcam_index(mcam, pcifunc & ~RVU_PFVF_FUNC_MASK,
- nixlf, NIXLF_BCAST_ENTRY);
- bank = npc_get_bank(mcam, index);
- *(u64 *)&action = rvu_read64(rvu, blkaddr,
- NPC_AF_MCAMEX_BANKX_ACTION(index & (mcam->banksize - 1), bank));
-
- /* VFs will not have BCAST entry */
- if (action.op != NIX_RX_ACTIONOP_MCAST &&
- !(pcifunc & RVU_PFVF_FUNC_MASK)) {
- npc_enable_mcam_entry(rvu, mcam,
- blkaddr, index, enable);
- } else {
- nix_update_bcast_mce_list(rvu, pcifunc, enable);
- /* Enable PF's BCAST entry for packet replication */
- rvu_npc_enable_bcast_entry(rvu, pcifunc, enable);
- }
-
- if (enable)
- rvu_npc_enable_promisc_entry(rvu, pcifunc, nixlf);
- else
- rvu_npc_disable_promisc_entry(rvu, pcifunc, nixlf);
+ /* add/delete pf_func to broadcast MCE list */
+ npc_enadis_default_mce_entry(rvu, pcifunc, nixlf,
+ NIXLF_BCAST_ENTRY, enable);
}
void rvu_npc_disable_default_entries(struct rvu *rvu, u16 pcifunc, int nixlf)
{
npc_enadis_default_entries(rvu, pcifunc, nixlf, false);
+
+ /* Delete multicast and promisc MCAM entries */
+ npc_enadis_default_mce_entry(rvu, pcifunc, nixlf,
+ NIXLF_ALLMULTI_ENTRY, false);
+ npc_enadis_default_mce_entry(rvu, pcifunc, nixlf,
+ NIXLF_PROMISC_ENTRY, false);
}
void rvu_npc_enable_default_entries(struct rvu *rvu, u16 pcifunc, int nixlf)
{
+ /* Enables only broadcast match entry. Promisc/Allmulti are enabled
+ * in set_rx_mode mbox handler.
+ */
npc_enadis_default_entries(rvu, pcifunc, nixlf, true);
}
/* Disable MCAM entries directing traffic to this 'pcifunc' */
list_for_each_entry_safe(rule, tmp, &mcam->mcam_rules, list) {
if (is_npc_intf_rx(rule->intf) &&
- rule->rx_action.pf_func == pcifunc) {
+ rule->rx_action.pf_func == pcifunc &&
+ rule->rx_action.op != NIX_RX_ACTIONOP_MCAST) {
npc_enable_mcam_entry(rvu, mcam, blkaddr,
rule->entry, false);
rule->enable = false;
}
}
+static int npc_fwdb_prfl_img_map(struct rvu *rvu, void __iomem **prfl_img_addr,
+ u64 *size)
+{
+ u64 prfl_addr, prfl_sz;
+
+ if (!rvu->fwdata)
+ return -EINVAL;
+
+ prfl_addr = rvu->fwdata->mcam_addr;
+ prfl_sz = rvu->fwdata->mcam_sz;
+
+ if (!prfl_addr || !prfl_sz)
+ return -EINVAL;
+
+ *prfl_img_addr = ioremap_wc(prfl_addr, prfl_sz);
+ if (!(*prfl_img_addr))
+ return -ENOMEM;
+
+ *size = prfl_sz;
+
+ return 0;
+}
+
+/* strtoull of "mkexprof" with base:36 */
#define MKEX_END_SIGN 0xdeadbeef
static void npc_load_mkex_profile(struct rvu *rvu, int blkaddr,
{
struct device *dev = &rvu->pdev->dev;
struct npc_mcam_kex *mcam_kex;
- void *mkex_prfl_addr = NULL;
- u64 prfl_addr, prfl_sz;
+ void __iomem *mkex_prfl_addr = NULL;
+ u64 prfl_sz;
+ int ret;
/* If user not selected mkex profile */
- if (!strncmp(mkex_profile, def_pfl_name, MKEX_NAME_LEN))
- goto program_mkex;
-
- if (!rvu->fwdata)
- goto program_mkex;
- prfl_addr = rvu->fwdata->mcam_addr;
- prfl_sz = rvu->fwdata->mcam_sz;
-
- if (!prfl_addr || !prfl_sz)
+ if (rvu->kpu_fwdata_sz ||
+ !strncmp(mkex_profile, def_pfl_name, MKEX_NAME_LEN))
goto program_mkex;
- mkex_prfl_addr = memremap(prfl_addr, prfl_sz, MEMREMAP_WC);
- if (!mkex_prfl_addr)
+ /* Setting up the mapping for mkex profile image */
+ ret = npc_fwdb_prfl_img_map(rvu, &mkex_prfl_addr, &prfl_sz);
+ if (ret < 0)
goto program_mkex;
- mcam_kex = (struct npc_mcam_kex *)mkex_prfl_addr;
+ mcam_kex = (struct npc_mcam_kex __force *)mkex_prfl_addr;
while (((s64)prfl_sz > 0) && (mcam_kex->mkex_sign != MKEX_END_SIGN)) {
/* Compare with mkex mod_param name string */
/* Program selected mkex profile */
npc_program_mkex_profile(rvu, blkaddr, rvu->kpu.mkex);
if (mkex_prfl_addr)
- memunmap(mkex_prfl_addr);
+ iounmap(mkex_prfl_addr);
}
static void npc_config_kpuaction(struct rvu *rvu, int blkaddr,
const struct npc_kpu_profile *profile)
{
int entry, num_entries, max_entries;
+ u64 entry_mask;
if (profile->cam_entries != profile->action_entries) {
dev_err(rvu->dev,
/* Enable all programmed entries */
num_entries = min_t(int, profile->action_entries, profile->cam_entries);
+ entry_mask = enable_mask(num_entries);
+ /* Disable first KPU_MAX_CST_ENT entries for built-in profile */
+ if (!rvu->kpu.custom)
+ entry_mask |= GENMASK_ULL(KPU_MAX_CST_ENT - 1, 0);
rvu_write64(rvu, blkaddr,
- NPC_AF_KPUX_ENTRY_DISX(kpu, 0), enable_mask(num_entries));
+ NPC_AF_KPUX_ENTRY_DISX(kpu, 0), entry_mask);
if (num_entries > 64) {
rvu_write64(rvu, blkaddr,
NPC_AF_KPUX_ENTRY_DISX(kpu, 1),
static int npc_prepare_default_kpu(struct npc_kpu_profile_adapter *profile)
{
+ profile->custom = 0;
profile->name = def_pfl_name;
profile->version = NPC_KPU_PROFILE_VER;
profile->ikpu = ikpu_action_entries;
return 0;
}
+static int npc_apply_custom_kpu(struct rvu *rvu,
+ struct npc_kpu_profile_adapter *profile)
+{
+ size_t hdr_sz = sizeof(struct npc_kpu_profile_fwdata), offset = 0;
+ struct npc_kpu_profile_fwdata *fw = rvu->kpu_fwdata;
+ struct npc_kpu_profile_action *action;
+ struct npc_kpu_profile_cam *cam;
+ struct npc_kpu_fwdata *fw_kpu;
+ int entries;
+ u16 kpu, entry;
+
+ if (rvu->kpu_fwdata_sz < hdr_sz) {
+ dev_warn(rvu->dev, "Invalid KPU profile size\n");
+ return -EINVAL;
+ }
+ if (le64_to_cpu(fw->signature) != KPU_SIGN) {
+ dev_warn(rvu->dev, "Invalid KPU profile signature %llx\n",
+ fw->signature);
+ return -EINVAL;
+ }
+ /* Verify if the using known profile structure */
+ if (NPC_KPU_VER_MAJ(profile->version) >
+ NPC_KPU_VER_MAJ(NPC_KPU_PROFILE_VER)) {
+ dev_warn(rvu->dev, "Not supported Major version: %d > %d\n",
+ NPC_KPU_VER_MAJ(profile->version),
+ NPC_KPU_VER_MAJ(NPC_KPU_PROFILE_VER));
+ return -EINVAL;
+ }
+ /* Verify if profile is aligned with the required kernel changes */
+ if (NPC_KPU_VER_MIN(profile->version) <
+ NPC_KPU_VER_MIN(NPC_KPU_PROFILE_VER)) {
+ dev_warn(rvu->dev,
+ "Invalid KPU profile version: %d.%d.%d expected version <= %d.%d.%d\n",
+ NPC_KPU_VER_MAJ(profile->version),
+ NPC_KPU_VER_MIN(profile->version),
+ NPC_KPU_VER_PATCH(profile->version),
+ NPC_KPU_VER_MAJ(NPC_KPU_PROFILE_VER),
+ NPC_KPU_VER_MIN(NPC_KPU_PROFILE_VER),
+ NPC_KPU_VER_PATCH(NPC_KPU_PROFILE_VER));
+ return -EINVAL;
+ }
+ /* Verify if profile fits the HW */
+ if (fw->kpus > profile->kpus) {
+ dev_warn(rvu->dev, "Not enough KPUs: %d > %ld\n", fw->kpus,
+ profile->kpus);
+ return -EINVAL;
+ }
+
+ profile->custom = 1;
+ profile->name = fw->name;
+ profile->version = le64_to_cpu(fw->version);
+ profile->mkex = &fw->mkex;
+ profile->lt_def = &fw->lt_def;
+
+ for (kpu = 0; kpu < fw->kpus; kpu++) {
+ fw_kpu = (struct npc_kpu_fwdata *)(fw->data + offset);
+ if (fw_kpu->entries > KPU_MAX_CST_ENT)
+ dev_warn(rvu->dev,
+ "Too many custom entries on KPU%d: %d > %d\n",
+ kpu, fw_kpu->entries, KPU_MAX_CST_ENT);
+ entries = min(fw_kpu->entries, KPU_MAX_CST_ENT);
+ cam = (struct npc_kpu_profile_cam *)fw_kpu->data;
+ offset += sizeof(*fw_kpu) + fw_kpu->entries * sizeof(*cam);
+ action = (struct npc_kpu_profile_action *)(fw->data + offset);
+ offset += fw_kpu->entries * sizeof(*action);
+ if (rvu->kpu_fwdata_sz < hdr_sz + offset) {
+ dev_warn(rvu->dev,
+ "Profile size mismatch on KPU%i parsing.\n",
+ kpu + 1);
+ return -EINVAL;
+ }
+ for (entry = 0; entry < entries; entry++) {
+ profile->kpu[kpu].cam[entry] = cam[entry];
+ profile->kpu[kpu].action[entry] = action[entry];
+ }
+ }
+
+ return 0;
+}
+
+static int npc_load_kpu_prfl_img(struct rvu *rvu, void __iomem *prfl_addr,
+ u64 prfl_sz, const char *kpu_profile)
+{
+ struct npc_kpu_profile_fwdata *kpu_data = NULL;
+ int rc = -EINVAL;
+
+ kpu_data = (struct npc_kpu_profile_fwdata __force *)prfl_addr;
+ if (le64_to_cpu(kpu_data->signature) == KPU_SIGN &&
+ !strncmp(kpu_data->name, kpu_profile, KPU_NAME_LEN)) {
+ dev_info(rvu->dev, "Loading KPU profile from firmware db: %s\n",
+ kpu_profile);
+ rvu->kpu_fwdata = kpu_data;
+ rvu->kpu_fwdata_sz = prfl_sz;
+ rvu->kpu_prfl_addr = prfl_addr;
+ rc = 0;
+ }
+
+ return rc;
+}
+
+static int npc_fwdb_detect_load_prfl_img(struct rvu *rvu, uint64_t prfl_sz,
+ const char *kpu_profile)
+{
+ struct npc_coalesced_kpu_prfl *img_data = NULL;
+ int i = 0, rc = -EINVAL;
+ void __iomem *kpu_prfl_addr;
+ u16 offset;
+
+ img_data = (struct npc_coalesced_kpu_prfl __force *)rvu->kpu_prfl_addr;
+ if (le64_to_cpu(img_data->signature) == KPU_SIGN &&
+ !strncmp(img_data->name, kpu_profile, KPU_NAME_LEN)) {
+ /* Loaded profile is a single KPU profile. */
+ rc = npc_load_kpu_prfl_img(rvu, rvu->kpu_prfl_addr,
+ prfl_sz, kpu_profile);
+ goto done;
+ }
+
+ /* Loaded profile is coalesced image, offset of first KPU profile.*/
+ offset = offsetof(struct npc_coalesced_kpu_prfl, prfl_sz) +
+ (img_data->num_prfl * sizeof(uint16_t));
+ /* Check if mapped image is coalesced image. */
+ while (i < img_data->num_prfl) {
+ /* Profile image offsets are rounded up to next 8 multiple.*/
+ offset = ALIGN_8B_CEIL(offset);
+ kpu_prfl_addr = (void __iomem *)((uintptr_t)rvu->kpu_prfl_addr +
+ offset);
+ rc = npc_load_kpu_prfl_img(rvu, kpu_prfl_addr,
+ img_data->prfl_sz[i], kpu_profile);
+ if (!rc)
+ break;
+ /* Calculating offset of profile image based on profile size.*/
+ offset += img_data->prfl_sz[i];
+ i++;
+ }
+done:
+ return rc;
+}
+
+static int npc_load_kpu_profile_fwdb(struct rvu *rvu, const char *kpu_profile)
+{
+ int ret = -EINVAL;
+ u64 prfl_sz;
+
+ /* Setting up the mapping for NPC profile image */
+ ret = npc_fwdb_prfl_img_map(rvu, &rvu->kpu_prfl_addr, &prfl_sz);
+ if (ret < 0)
+ goto done;
+
+ /* Detect if profile is coalesced or single KPU profile and load */
+ ret = npc_fwdb_detect_load_prfl_img(rvu, prfl_sz, kpu_profile);
+ if (ret == 0)
+ goto done;
+
+ /* Cleaning up if KPU profile image from fwdata is not valid. */
+ if (rvu->kpu_prfl_addr) {
+ iounmap(rvu->kpu_prfl_addr);
+ rvu->kpu_prfl_addr = NULL;
+ rvu->kpu_fwdata_sz = 0;
+ rvu->kpu_fwdata = NULL;
+ }
+
+done:
+ return ret;
+}
+
static void npc_load_kpu_profile(struct rvu *rvu)
{
struct npc_kpu_profile_adapter *profile = &rvu->kpu;
+ const char *kpu_profile = rvu->kpu_pfl_name;
+ const struct firmware *fw = NULL;
+ bool retry_fwdb = false;
+
+ /* If user not specified profile customization */
+ if (!strncmp(kpu_profile, def_pfl_name, KPU_NAME_LEN))
+ goto revert_to_default;
+ /* First prepare default KPU, then we'll customize top entries. */
+ npc_prepare_default_kpu(profile);
+ /* Order of preceedence for load loading NPC profile (high to low)
+ * Firmware binary in filesystem.
+ * Firmware database method.
+ * Default KPU profile.
+ */
+ if (!request_firmware(&fw, kpu_profile, rvu->dev)) {
+ dev_info(rvu->dev, "Loading KPU profile from firmware: %s\n",
+ kpu_profile);
+ rvu->kpu_fwdata = kzalloc(fw->size, GFP_KERNEL);
+ if (rvu->kpu_fwdata) {
+ memcpy(rvu->kpu_fwdata, fw->data, fw->size);
+ rvu->kpu_fwdata_sz = fw->size;
+ }
+ release_firmware(fw);
+ retry_fwdb = true;
+ goto program_kpu;
+ }
+
+load_image_fwdb:
+ /* Loading the KPU profile using firmware database */
+ if (npc_load_kpu_profile_fwdb(rvu, kpu_profile))
+ goto revert_to_default;
+
+program_kpu:
+ /* Apply profile customization if firmware was loaded. */
+ if (!rvu->kpu_fwdata_sz || npc_apply_custom_kpu(rvu, profile)) {
+ /* If image from firmware filesystem fails to load or invalid
+ * retry with firmware database method.
+ */
+ if (rvu->kpu_fwdata || rvu->kpu_fwdata_sz) {
+ /* Loading image from firmware database failed. */
+ if (rvu->kpu_prfl_addr) {
+ iounmap(rvu->kpu_prfl_addr);
+ rvu->kpu_prfl_addr = NULL;
+ } else {
+ kfree(rvu->kpu_fwdata);
+ }
+ rvu->kpu_fwdata = NULL;
+ rvu->kpu_fwdata_sz = 0;
+ if (retry_fwdb) {
+ retry_fwdb = false;
+ goto load_image_fwdb;
+ }
+ }
+
+ dev_warn(rvu->dev,
+ "Can't load KPU profile %s. Using default.\n",
+ kpu_profile);
+ kfree(rvu->kpu_fwdata);
+ rvu->kpu_fwdata = NULL;
+ goto revert_to_default;
+ }
+
+ dev_info(rvu->dev, "Using custom profile '%s', version %d.%d.%d\n",
+ profile->name, NPC_KPU_VER_MAJ(profile->version),
+ NPC_KPU_VER_MIN(profile->version),
+ NPC_KPU_VER_PATCH(profile->version));
+
+ return;
+
+revert_to_default:
npc_prepare_default_kpu(profile);
}
kfree(pkind->rsrc.bmap);
kfree(mcam->counters.bmap);
+ if (rvu->kpu_prfl_addr)
+ iounmap(rvu->kpu_prfl_addr);
+ else
+ kfree(rvu->kpu_fwdata);
mutex_destroy(&mcam->lock);
}
rsp->free_count = 0;
/* Check if ref_entry is within range */
- if (req->priority && req->ref_entry >= mcam->bmap_entries)
+ if (req->priority && req->ref_entry >= mcam->bmap_entries) {
+ dev_err(rvu->dev, "%s: reference entry %d is out of range\n",
+ __func__, req->ref_entry);
return NPC_MCAM_INVALID_REQ;
+ }
/* ref_entry can't be '0' if requested priority is high.
* Can't be last entry if requested priority is low.
/* Since list of allocated indices needs to be sent to requester,
* max number of non-contiguous entries per mbox msg is limited.
*/
- if (!req->contig && req->count > NPC_MAX_NONCONTIG_ENTRIES)
+ if (!req->contig && req->count > NPC_MAX_NONCONTIG_ENTRIES) {
+ dev_err(rvu->dev,
+ "%s: %d Non-contiguous MCAM entries requested is more than max (%d) allowed\n",
+ __func__, req->count, NPC_MAX_NONCONTIG_ENTRIES);
return NPC_MCAM_INVALID_REQ;
+ }
/* Alloc request from PFFUNC with no NIXLF attached should be denied */
if (!is_nixlf_attached(rvu, pcifunc))
static bool npc_is_same(struct npc_key_field *input,
struct npc_key_field *field)
{
- int ret;
-
- ret = memcmp(&input->layer_mdata, &field->layer_mdata,
- sizeof(struct npc_layer_mdata));
- return ret == 0;
+ return memcmp(&input->layer_mdata, &field->layer_mdata,
+ sizeof(struct npc_layer_mdata)) == 0;
}
static void npc_set_layer_mdata(struct npc_mcam *mcam, enum key_fields type,
if (pf_set_vfs_mac) {
ether_addr_copy(pfvf->default_mac, req->packet.dmac);
ether_addr_copy(pfvf->mac_addr, req->packet.dmac);
+ set_bit(PF_SET_VF_MAC, &pfvf->flags);
}
- if (pfvf->pf_set_vf_cfg && req->vtag0_type == NIX_AF_LFX_RX_VTAG_TYPE7)
+ if (test_bit(PF_SET_VF_CFG, &pfvf->flags) &&
+ req->vtag0_type == NIX_AF_LFX_RX_VTAG_TYPE7)
rule->vfvlan_cfg = true;
+ if (is_npc_intf_rx(req->intf) && req->match_id &&
+ (req->op == NIX_RX_ACTIONOP_UCAST || req->op == NIX_RX_ACTIONOP_RSS))
+ return rvu_nix_setup_ratelimit_aggr(rvu, req->hdr.pcifunc,
+ req->index, req->match_id);
+
return 0;
}
/* PF installing for its VF */
if (req->hdr.pcifunc && !from_vf && req->vf)
- pfvf->pf_set_vf_cfg = 1;
+ set_bit(PF_SET_VF_CFG, &pfvf->flags);
/* update req destination mac addr */
if ((req->features & BIT_ULL(NPC_DMAC)) && is_npc_intf_rx(req->intf) &&
}
err = nix_get_nixlf(rvu, target, &nixlf, NULL);
+ if (err && is_npc_intf_rx(req->intf) && !pf_set_vfs_mac)
+ return -EINVAL;
- /* If interface is uninitialized then do not enable entry */
- if (err || (!req->default_rule && !pfvf->def_ucast_rule))
+ /* don't enable rule when nixlf not attached or initialized */
+ if (!(is_nixlf_attached(rvu, target) &&
+ test_bit(NIXLF_INITIALIZED, &pfvf->flags)))
enable = false;
/* Packets reaching NPC in Tx path implies that a
if (from_vf && !enable)
return -EINVAL;
+ /* PF sets VF mac & VF NIXLF is not attached, update the mac addr */
+ if (pf_set_vfs_mac && !enable) {
+ ether_addr_copy(pfvf->default_mac, req->packet.dmac);
+ ether_addr_copy(pfvf->mac_addr, req->packet.dmac);
+ set_bit(PF_SET_VF_MAC, &pfvf->flags);
+ return 0;
+ }
+
/* If message is from VF then its flow should not overlap with
* reserved unicast flow.
*/
#define NIX_AF_SQ_CONST (0x0040)
#define NIX_AF_CQ_CONST (0x0048)
#define NIX_AF_RQ_CONST (0x0050)
+#define NIX_AF_PL_CONST (0x0058)
#define NIX_AF_PSE_CONST (0x0060)
#define NIX_AF_TL1_CONST (0x0070)
#define NIX_AF_TL2_CONST (0x0078)
#define NIX_AF_LSO_CFG (0x00A8)
#define NIX_AF_BLK_RST (0x00B0)
#define NIX_AF_TX_TSTMP_CFG (0x00C0)
+#define NIX_AF_PL_TS (0x00C8)
#define NIX_AF_RX_CFG (0x00D0)
#define NIX_AF_AVG_DELAY (0x00E0)
#define NIX_AF_CINT_DELAY (0x00F0)
#define NIX_AF_RVU_INT_ENA_W1S (0x01D0)
#define NIX_AF_RVU_INT_ENA_W1C (0x01D8)
#define NIX_AF_TCP_TIMER (0x01E0)
-#define NIX_AF_RX_WQE_TAG_CTL (0x01F0)
+#define NIX_AF_RX_DEF_ET(a) (0x01F0ull | (uint64_t)(a) << 3)
#define NIX_AF_RX_DEF_OL2 (0x0200)
#define NIX_AF_RX_DEF_OIP4 (0x0210)
#define NIX_AF_RX_DEF_IIP4 (0x0220)
+#define NIX_AF_RX_DEF_VLAN0_PCP_DEI (0x0228)
#define NIX_AF_RX_DEF_OIP6 (0x0230)
+#define NIX_AF_RX_DEF_VLAN1_PCP_DEI (0x0238)
#define NIX_AF_RX_DEF_IIP6 (0x0240)
#define NIX_AF_RX_DEF_OTCP (0x0250)
#define NIX_AF_RX_DEF_ITCP (0x0260)
#define NIX_AF_RX_DEF_OUDP (0x0270)
#define NIX_AF_RX_DEF_IUDP (0x0280)
#define NIX_AF_RX_DEF_OSCTP (0x0290)
+#define NIX_AF_RX_DEF_CST_APAD0 (0x0298)
#define NIX_AF_RX_DEF_ISCTP (0x02A0)
#define NIX_AF_RX_DEF_IPSECX (0x02B0)
+#define NIX_AF_RX_DEF_CST_APAD1 (0x02A8)
+#define NIX_AF_RX_DEF_IIP4_DSCP (0x02E0)
+#define NIX_AF_RX_DEF_OIP4_DSCP (0x02E8)
+#define NIX_AF_RX_DEF_IIP6_DSCP (0x02F0)
+#define NIX_AF_RX_DEF_OIP6_DSCP (0x02F8)
#define NIX_AF_RX_IPSEC_GEN_CFG (0x0300)
#define NIX_AF_RX_CPTX_INST_ADDR (0x0310)
#define NIX_AF_NDC_TX_SYNC (0x03F0)
NIX_AQ_CTYPE_MCE = 0x3,
NIX_AQ_CTYPE_RSS = 0x4,
NIX_AQ_CTYPE_DYNO = 0x5,
- NIX_AQ_CTYPE_BAND_PROF = 0x6,
+ NIX_AQ_CTYPE_BANDPROF = 0x6,
};
/* NIX admin queue instruction opcodes */
uint64_t next : 16;
};
+enum nix_band_prof_layers {
+ BAND_PROF_LEAF_LAYER = 0,
+ BAND_PROF_INVAL_LAYER = 1,
+ BAND_PROF_MID_LAYER = 2,
+ BAND_PROF_TOP_LAYER = 3,
+ BAND_PROF_NUM_LAYERS = 4,
+};
+
+enum NIX_RX_BAND_PROF_ACTIONRESULT_E {
+ NIX_RX_BAND_PROF_ACTIONRESULT_PASS = 0x0,
+ NIX_RX_BAND_PROF_ACTIONRESULT_DROP = 0x1,
+ NIX_RX_BAND_PROF_ACTIONRESULT_RED = 0x2,
+};
+
+enum nix_band_prof_pc_mode {
+ NIX_RX_PC_MODE_VLAN = 0,
+ NIX_RX_PC_MODE_DSCP = 1,
+ NIX_RX_PC_MODE_GEN = 2,
+ NIX_RX_PC_MODE_RSVD = 3,
+};
+
+/* NIX ingress policer bandwidth profile structure */
+struct nix_bandprof_s {
+ uint64_t pc_mode : 2; /* W0 */
+ uint64_t icolor : 2;
+ uint64_t tnl_ena : 1;
+ uint64_t reserved_5_7 : 3;
+ uint64_t peir_exponent : 5;
+ uint64_t reserved_13_15 : 3;
+ uint64_t pebs_exponent : 5;
+ uint64_t reserved_21_23 : 3;
+ uint64_t cir_exponent : 5;
+ uint64_t reserved_29_31 : 3;
+ uint64_t cbs_exponent : 5;
+ uint64_t reserved_37_39 : 3;
+ uint64_t peir_mantissa : 8;
+ uint64_t pebs_mantissa : 8;
+ uint64_t cir_mantissa : 8;
+ uint64_t cbs_mantissa : 8; /* W1 */
+ uint64_t lmode : 1;
+ uint64_t l_sellect : 3;
+ uint64_t rdiv : 4;
+ uint64_t adjust_exponent : 5;
+ uint64_t reserved_85_86 : 2;
+ uint64_t adjust_mantissa : 9;
+ uint64_t gc_action : 2;
+ uint64_t yc_action : 2;
+ uint64_t rc_action : 2;
+ uint64_t meter_algo : 2;
+ uint64_t band_prof_id : 7;
+ uint64_t reserved_111_118 : 8;
+ uint64_t hl_en : 1;
+ uint64_t reserved_120_127 : 8;
+ uint64_t ts : 48; /* W2 */
+ uint64_t reserved_176_191 : 16;
+ uint64_t pe_accum : 32; /* W3 */
+ uint64_t c_accum : 32;
+ uint64_t green_pkt_pass : 48; /* W4 */
+ uint64_t reserved_304_319 : 16;
+ uint64_t yellow_pkt_pass : 48; /* W5 */
+ uint64_t reserved_368_383 : 16;
+ uint64_t red_pkt_pass : 48; /* W6 */
+ uint64_t reserved_432_447 : 16;
+ uint64_t green_octs_pass : 48; /* W7 */
+ uint64_t reserved_496_511 : 16;
+ uint64_t yellow_octs_pass : 48; /* W8 */
+ uint64_t reserved_560_575 : 16;
+ uint64_t red_octs_pass : 48; /* W9 */
+ uint64_t reserved_624_639 : 16;
+ uint64_t green_pkt_drop : 48; /* W10 */
+ uint64_t reserved_688_703 : 16;
+ uint64_t yellow_pkt_drop : 48; /* W11 */
+ uint64_t reserved_752_767 : 16;
+ uint64_t red_pkt_drop : 48; /* W12 */
+ uint64_t reserved_816_831 : 16;
+ uint64_t green_octs_drop : 48; /* W13 */
+ uint64_t reserved_880_895 : 16;
+ uint64_t yellow_octs_drop : 48; /* W14 */
+ uint64_t reserved_944_959 : 16;
+ uint64_t red_octs_drop : 48; /* W15 */
+ uint64_t reserved_1008_1023 : 16;
+};
+
enum nix_lsoalg {
NIX_LSOALG_NOP,
NIX_LSOALG_ADD_SEGNUM,
sq->head++;
sq->head &= (sq->sqe_cnt - 1);
}
+
+int cn10k_free_all_ipolicers(struct otx2_nic *pfvf)
+{
+ struct nix_bandprof_free_req *req;
+ int rc;
+
+ if (is_dev_otx2(pfvf->pdev))
+ return 0;
+
+ mutex_lock(&pfvf->mbox.lock);
+
+ req = otx2_mbox_alloc_msg_nix_bandprof_free(&pfvf->mbox);
+ if (!req) {
+ rc = -ENOMEM;
+ goto out;
+ }
+
+ /* Free all bandwidth profiles allocated */
+ req->free_all = true;
+
+ rc = otx2_sync_mbox_msg(&pfvf->mbox);
+out:
+ mutex_unlock(&pfvf->mbox.lock);
+ return rc;
+}
+
+int cn10k_alloc_leaf_profile(struct otx2_nic *pfvf, u16 *leaf)
+{
+ struct nix_bandprof_alloc_req *req;
+ struct nix_bandprof_alloc_rsp *rsp;
+ int rc;
+
+ req = otx2_mbox_alloc_msg_nix_bandprof_alloc(&pfvf->mbox);
+ if (!req)
+ return -ENOMEM;
+
+ req->prof_count[BAND_PROF_LEAF_LAYER] = 1;
+
+ rc = otx2_sync_mbox_msg(&pfvf->mbox);
+ if (rc)
+ goto out;
+
+ rsp = (struct nix_bandprof_alloc_rsp *)
+ otx2_mbox_get_rsp(&pfvf->mbox.mbox, 0, &req->hdr);
+ if (!rsp->prof_count[BAND_PROF_LEAF_LAYER]) {
+ rc = -EIO;
+ goto out;
+ }
+
+ *leaf = rsp->prof_idx[BAND_PROF_LEAF_LAYER][0];
+out:
+ if (rc) {
+ dev_warn(pfvf->dev,
+ "Failed to allocate ingress bandwidth policer\n");
+ }
+
+ return rc;
+}
+
+int cn10k_alloc_matchall_ipolicer(struct otx2_nic *pfvf)
+{
+ struct otx2_hw *hw = &pfvf->hw;
+ int ret;
+
+ mutex_lock(&pfvf->mbox.lock);
+
+ ret = cn10k_alloc_leaf_profile(pfvf, &hw->matchall_ipolicer);
+
+ mutex_unlock(&pfvf->mbox.lock);
+
+ return ret;
+}
+
+#define POLICER_TIMESTAMP 1 /* 1 second */
+#define MAX_RATE_EXP 22 /* Valid rate exponent range: 0 - 22 */
+
+static void cn10k_get_ingress_burst_cfg(u32 burst, u32 *burst_exp,
+ u32 *burst_mantissa)
+{
+ int tmp;
+
+ /* Burst is calculated as
+ * (1+[BURST_MANTISSA]/256)*2^[BURST_EXPONENT]
+ * This is the upper limit on number tokens (bytes) that
+ * can be accumulated in the bucket.
+ */
+ *burst_exp = ilog2(burst);
+ if (burst < 256) {
+ /* No float: can't express mantissa in this case */
+ *burst_mantissa = 0;
+ return;
+ }
+
+ if (*burst_exp > MAX_RATE_EXP)
+ *burst_exp = MAX_RATE_EXP;
+
+ /* Calculate mantissa
+ * Find remaining bytes 'burst - 2^burst_exp'
+ * mantissa = (remaining bytes) / 2^ (burst_exp - 8)
+ */
+ tmp = burst - rounddown_pow_of_two(burst);
+ *burst_mantissa = tmp / (1UL << (*burst_exp - 8));
+}
+
+static void cn10k_get_ingress_rate_cfg(u64 rate, u32 *rate_exp,
+ u32 *rate_mantissa, u32 *rdiv)
+{
+ u32 div = 0;
+ u32 exp = 0;
+ u64 tmp;
+
+ /* Figure out mantissa, exponent and divider from given max pkt rate
+ *
+ * To achieve desired rate HW adds
+ * (1+[RATE_MANTISSA]/256)*2^[RATE_EXPONENT] tokens (bytes) at every
+ * policer timeunit * 2^rdiv ie 2 * 2^rdiv usecs, to the token bucket.
+ * Here policer timeunit is 2 usecs and rate is in bits per sec.
+ * Since floating point cannot be used below algorithm uses 1000000
+ * scale factor to support rates upto 100Gbps.
+ */
+ tmp = rate * 32 * 2;
+ if (tmp < 256000000) {
+ while (tmp < 256000000) {
+ tmp = tmp * 2;
+ div++;
+ }
+ } else {
+ for (exp = 0; tmp >= 512000000 && exp <= MAX_RATE_EXP; exp++)
+ tmp = tmp / 2;
+
+ if (exp > MAX_RATE_EXP)
+ exp = MAX_RATE_EXP;
+ }
+
+ *rate_mantissa = (tmp - 256000000) / 1000000;
+ *rate_exp = exp;
+ *rdiv = div;
+}
+
+int cn10k_map_unmap_rq_policer(struct otx2_nic *pfvf, int rq_idx,
+ u16 policer, bool map)
+{
+ struct nix_cn10k_aq_enq_req *aq;
+
+ aq = otx2_mbox_alloc_msg_nix_cn10k_aq_enq(&pfvf->mbox);
+ if (!aq)
+ return -ENOMEM;
+
+ /* Enable policing and set the bandwidth profile (policer) index */
+ if (map)
+ aq->rq.policer_ena = 1;
+ else
+ aq->rq.policer_ena = 0;
+ aq->rq_mask.policer_ena = 1;
+
+ aq->rq.band_prof_id = policer;
+ aq->rq_mask.band_prof_id = GENMASK(9, 0);
+
+ /* Fill AQ info */
+ aq->qidx = rq_idx;
+ aq->ctype = NIX_AQ_CTYPE_RQ;
+ aq->op = NIX_AQ_INSTOP_WRITE;
+
+ return otx2_sync_mbox_msg(&pfvf->mbox);
+}
+
+int cn10k_free_leaf_profile(struct otx2_nic *pfvf, u16 leaf)
+{
+ struct nix_bandprof_free_req *req;
+
+ req = otx2_mbox_alloc_msg_nix_bandprof_free(&pfvf->mbox);
+ if (!req)
+ return -ENOMEM;
+
+ req->prof_count[BAND_PROF_LEAF_LAYER] = 1;
+ req->prof_idx[BAND_PROF_LEAF_LAYER][0] = leaf;
+
+ return otx2_sync_mbox_msg(&pfvf->mbox);
+}
+
+int cn10k_free_matchall_ipolicer(struct otx2_nic *pfvf)
+{
+ struct otx2_hw *hw = &pfvf->hw;
+ int qidx, rc;
+
+ mutex_lock(&pfvf->mbox.lock);
+
+ /* Remove RQ's policer mapping */
+ for (qidx = 0; qidx < hw->rx_queues; qidx++)
+ cn10k_map_unmap_rq_policer(pfvf, qidx,
+ hw->matchall_ipolicer, false);
+
+ rc = cn10k_free_leaf_profile(pfvf, hw->matchall_ipolicer);
+
+ mutex_unlock(&pfvf->mbox.lock);
+ return rc;
+}
+
+int cn10k_set_ipolicer_rate(struct otx2_nic *pfvf, u16 profile,
+ u32 burst, u64 rate, bool pps)
+{
+ struct nix_cn10k_aq_enq_req *aq;
+ u32 burst_exp, burst_mantissa;
+ u32 rate_exp, rate_mantissa;
+ u32 rdiv;
+
+ /* Get exponent and mantissa values for the desired rate */
+ cn10k_get_ingress_burst_cfg(burst, &burst_exp, &burst_mantissa);
+ cn10k_get_ingress_rate_cfg(rate, &rate_exp, &rate_mantissa, &rdiv);
+
+ /* Init bandwidth profile */
+ aq = otx2_mbox_alloc_msg_nix_cn10k_aq_enq(&pfvf->mbox);
+ if (!aq)
+ return -ENOMEM;
+
+ /* Set initial color mode to blind */
+ aq->prof.icolor = 0x03;
+ aq->prof_mask.icolor = 0x03;
+
+ /* Set rate and burst values */
+ aq->prof.cir_exponent = rate_exp;
+ aq->prof_mask.cir_exponent = 0x1F;
+
+ aq->prof.cir_mantissa = rate_mantissa;
+ aq->prof_mask.cir_mantissa = 0xFF;
+
+ aq->prof.cbs_exponent = burst_exp;
+ aq->prof_mask.cbs_exponent = 0x1F;
+
+ aq->prof.cbs_mantissa = burst_mantissa;
+ aq->prof_mask.cbs_mantissa = 0xFF;
+
+ aq->prof.rdiv = rdiv;
+ aq->prof_mask.rdiv = 0xF;
+
+ if (pps) {
+ /* The amount of decremented tokens is calculated according to
+ * the following equation:
+ * max([ LMODE ? 0 : (packet_length - LXPTR)] +
+ * ([ADJUST_MANTISSA]/256 - 1) * 2^[ADJUST_EXPONENT],
+ * 1/256)
+ * if LMODE is 1 then rate limiting will be based on
+ * PPS otherwise bps.
+ * The aim of the ADJUST value is to specify a token cost per
+ * packet in contrary to the packet length that specifies a
+ * cost per byte. To rate limit based on PPS adjust mantissa
+ * is set as 384 and exponent as 1 so that number of tokens
+ * decremented becomes 1 i.e, 1 token per packeet.
+ */
+ aq->prof.adjust_exponent = 1;
+ aq->prof_mask.adjust_exponent = 0x1F;
+
+ aq->prof.adjust_mantissa = 384;
+ aq->prof_mask.adjust_mantissa = 0x1FF;
+
+ aq->prof.lmode = 0x1;
+ aq->prof_mask.lmode = 0x1;
+ }
+
+ /* Two rate three color marker
+ * With PEIR/EIR set to zero, color will be either green or red
+ */
+ aq->prof.meter_algo = 2;
+ aq->prof_mask.meter_algo = 0x3;
+
+ aq->prof.rc_action = NIX_RX_BAND_PROF_ACTIONRESULT_DROP;
+ aq->prof_mask.rc_action = 0x3;
+
+ aq->prof.yc_action = NIX_RX_BAND_PROF_ACTIONRESULT_PASS;
+ aq->prof_mask.yc_action = 0x3;
+
+ aq->prof.gc_action = NIX_RX_BAND_PROF_ACTIONRESULT_PASS;
+ aq->prof_mask.gc_action = 0x3;
+
+ /* Setting exponent value as 24 and mantissa as 0 configures
+ * the bucket with zero values making bucket unused. Peak
+ * information rate and Excess information rate buckets are
+ * unused here.
+ */
+ aq->prof.peir_exponent = 24;
+ aq->prof_mask.peir_exponent = 0x1F;
+
+ aq->prof.peir_mantissa = 0;
+ aq->prof_mask.peir_mantissa = 0xFF;
+
+ aq->prof.pebs_exponent = 24;
+ aq->prof_mask.pebs_exponent = 0x1F;
+
+ aq->prof.pebs_mantissa = 0;
+ aq->prof_mask.pebs_mantissa = 0xFF;
+
+ /* Fill AQ info */
+ aq->qidx = profile;
+ aq->ctype = NIX_AQ_CTYPE_BANDPROF;
+ aq->op = NIX_AQ_INSTOP_WRITE;
+
+ return otx2_sync_mbox_msg(&pfvf->mbox);
+}
+
+int cn10k_set_matchall_ipolicer_rate(struct otx2_nic *pfvf,
+ u32 burst, u64 rate)
+{
+ struct otx2_hw *hw = &pfvf->hw;
+ int qidx, rc;
+
+ mutex_lock(&pfvf->mbox.lock);
+
+ rc = cn10k_set_ipolicer_rate(pfvf, hw->matchall_ipolicer, burst,
+ rate, false);
+ if (rc)
+ goto out;
+
+ for (qidx = 0; qidx < hw->rx_queues; qidx++) {
+ rc = cn10k_map_unmap_rq_policer(pfvf, qidx,
+ hw->matchall_ipolicer, true);
+ if (rc)
+ break;
+ }
+
+out:
+ mutex_unlock(&pfvf->mbox.lock);
+ return rc;
+}
int cn10k_sq_aq_init(void *dev, u16 qidx, u16 sqb_aura);
int cn10k_pf_lmtst_init(struct otx2_nic *pf);
int cn10k_vf_lmtst_init(struct otx2_nic *vf);
+int cn10k_free_all_ipolicers(struct otx2_nic *pfvf);
+int cn10k_alloc_matchall_ipolicer(struct otx2_nic *pfvf);
+int cn10k_free_matchall_ipolicer(struct otx2_nic *pfvf);
+int cn10k_set_matchall_ipolicer_rate(struct otx2_nic *pfvf,
+ u32 burst, u64 rate);
+int cn10k_map_unmap_rq_policer(struct otx2_nic *pfvf, int rq_idx,
+ u16 policer, bool map);
+int cn10k_alloc_leaf_profile(struct otx2_nic *pfvf, u16 *leaf);
+int cn10k_set_ipolicer_rate(struct otx2_nic *pfvf, u16 profile,
+ u32 burst, u64 rate, bool pps);
+int cn10k_free_leaf_profile(struct otx2_nic *pfvf, u16 leaf);
#endif /* CN10K_H */
/* NIX */
u16 txschq_list[NIX_TXSCH_LVL_CNT][MAX_TXSCHQ_PER_FUNC];
+ u16 matchall_ipolicer;
/* HW settings, coalescing etc */
u16 rx_chan_base;
u64 *nix_lmt_base;
};
+enum vfperm {
+ OTX2_RESET_VF_PERM,
+ OTX2_TRUSTED_VF,
+};
+
struct otx2_vf_config {
struct otx2_nic *pf;
struct delayed_work link_event_work;
u8 mac[ETH_ALEN];
u16 vlan;
int tx_vtag_idx;
+ bool trusted;
};
struct flr_work {
struct otx2_flow_config {
u16 entry[NPC_MAX_NONCONTIG_ENTRIES];
- u32 nr_flows;
-#define OTX2_MAX_NTUPLE_FLOWS 32
-#define OTX2_MAX_UNICAST_FLOWS 8
-#define OTX2_MAX_VLAN_FLOWS 1
-#define OTX2_MAX_TC_FLOWS OTX2_MAX_NTUPLE_FLOWS
-#define OTX2_MCAM_COUNT (OTX2_MAX_NTUPLE_FLOWS + \
+ u16 *flow_ent;
+ u16 *def_ent;
+ u16 nr_flows;
+#define OTX2_DEFAULT_FLOWCOUNT 16
+#define OTX2_MAX_UNICAST_FLOWS 8
+#define OTX2_MAX_VLAN_FLOWS 1
+#define OTX2_MAX_TC_FLOWS OTX2_DEFAULT_FLOWCOUNT
+#define OTX2_MCAM_COUNT (OTX2_DEFAULT_FLOWCOUNT + \
OTX2_MAX_UNICAST_FLOWS + \
OTX2_MAX_VLAN_FLOWS)
- u32 ntuple_offset;
- u32 unicast_offset;
- u32 rx_vlan_offset;
- u32 vf_vlan_offset;
-#define OTX2_PER_VF_VLAN_FLOWS 2 /* rx+tx per VF */
+ u16 ntuple_offset;
+ u16 unicast_offset;
+ u16 rx_vlan_offset;
+ u16 vf_vlan_offset;
+#define OTX2_PER_VF_VLAN_FLOWS 2 /* Rx + Tx per VF */
#define OTX2_VF_VLAN_RX_INDEX 0
#define OTX2_VF_VLAN_TX_INDEX 1
- u32 tc_flower_offset;
- u32 ntuple_max_flows;
- u32 tc_max_flows;
+ u16 tc_flower_offset;
+ u16 ntuple_max_flows;
+ u16 tc_max_flows;
struct list_head flow_list;
};
#define OTX2_FLAG_TX_PAUSE_ENABLED BIT_ULL(10)
#define OTX2_FLAG_TC_FLOWER_SUPPORT BIT_ULL(11)
#define OTX2_FLAG_TC_MATCHALL_EGRESS_ENABLED BIT_ULL(12)
+#define OTX2_FLAG_TC_MATCHALL_INGRESS_ENABLED BIT_ULL(13)
u64 flags;
struct otx2_qset qset;
struct otx2_flow_config *flow_cfg;
struct otx2_tc_info tc_info;
+ unsigned long rq_bmap;
};
static inline bool is_otx2_lbkvf(struct pci_dev *pdev)
if (!channel->rx_count || !channel->tx_count)
return -EINVAL;
+ if (bitmap_weight(&pfvf->rq_bmap, pfvf->hw.rx_queues) > 1) {
+ netdev_err(dev,
+ "Receive queues are in use by TC police action\n");
+ return -EINVAL;
+ }
+
if (if_up)
dev->netdev_ops->ndo_stop(dev);
if (hfunc != ETH_RSS_HASH_NO_CHANGE && hfunc != ETH_RSS_HASH_TOP)
return -EOPNOTSUPP;
+ if (*rss_context != ETH_RXFH_CONTEXT_ALLOC &&
+ *rss_context >= MAX_RSS_GROUPS)
+ return -EINVAL;
+
rss = &pfvf->hw.rss_info;
if (!rss->enable) {
int vf;
};
+static void otx2_clear_ntuple_flow_info(struct otx2_nic *pfvf, struct otx2_flow_config *flow_cfg)
+{
+ devm_kfree(pfvf->dev, flow_cfg->flow_ent);
+ flow_cfg->flow_ent = NULL;
+ flow_cfg->ntuple_max_flows = 0;
+ flow_cfg->tc_max_flows = 0;
+}
+
+static int otx2_free_ntuple_mcam_entries(struct otx2_nic *pfvf)
+{
+ struct otx2_flow_config *flow_cfg = pfvf->flow_cfg;
+ struct npc_mcam_free_entry_req *req;
+ int ent, err;
+
+ if (!flow_cfg->ntuple_max_flows)
+ return 0;
+
+ mutex_lock(&pfvf->mbox.lock);
+ for (ent = 0; ent < flow_cfg->ntuple_max_flows; ent++) {
+ req = otx2_mbox_alloc_msg_npc_mcam_free_entry(&pfvf->mbox);
+ if (!req)
+ break;
+
+ req->entry = flow_cfg->flow_ent[ent];
+
+ /* Send message to AF to free MCAM entries */
+ err = otx2_sync_mbox_msg(&pfvf->mbox);
+ if (err)
+ break;
+ }
+ mutex_unlock(&pfvf->mbox.lock);
+ otx2_clear_ntuple_flow_info(pfvf, flow_cfg);
+ return 0;
+}
+
+static int otx2_alloc_ntuple_mcam_entries(struct otx2_nic *pfvf, u16 count)
+{
+ struct otx2_flow_config *flow_cfg = pfvf->flow_cfg;
+ struct npc_mcam_alloc_entry_req *req;
+ struct npc_mcam_alloc_entry_rsp *rsp;
+ int ent, allocated = 0;
+
+ /* Free current ones and allocate new ones with requested count */
+ otx2_free_ntuple_mcam_entries(pfvf);
+
+ if (!count)
+ return 0;
+
+ flow_cfg->flow_ent = devm_kmalloc_array(pfvf->dev, count,
+ sizeof(u16), GFP_KERNEL);
+ if (!flow_cfg->flow_ent)
+ return -ENOMEM;
+
+ mutex_lock(&pfvf->mbox.lock);
+
+ /* In a single request a max of NPC_MAX_NONCONTIG_ENTRIES MCAM entries
+ * can only be allocated.
+ */
+ while (allocated < count) {
+ req = otx2_mbox_alloc_msg_npc_mcam_alloc_entry(&pfvf->mbox);
+ if (!req)
+ goto exit;
+
+ req->contig = false;
+ req->count = (count - allocated) > NPC_MAX_NONCONTIG_ENTRIES ?
+ NPC_MAX_NONCONTIG_ENTRIES : count - allocated;
+ req->priority = NPC_MCAM_HIGHER_PRIO;
+ req->ref_entry = flow_cfg->def_ent[0];
+
+ /* Send message to AF */
+ if (otx2_sync_mbox_msg(&pfvf->mbox))
+ goto exit;
+
+ rsp = (struct npc_mcam_alloc_entry_rsp *)otx2_mbox_get_rsp
+ (&pfvf->mbox.mbox, 0, &req->hdr);
+
+ for (ent = 0; ent < rsp->count; ent++)
+ flow_cfg->flow_ent[ent + allocated] = rsp->entry_list[ent];
+
+ allocated += rsp->count;
+
+ /* If this request is not fulfilled, no need to send
+ * further requests.
+ */
+ if (rsp->count != req->count)
+ break;
+ }
+
+exit:
+ mutex_unlock(&pfvf->mbox.lock);
+
+ flow_cfg->ntuple_offset = 0;
+ flow_cfg->ntuple_max_flows = allocated;
+ flow_cfg->tc_max_flows = allocated;
+
+ if (allocated != count)
+ netdev_info(pfvf->netdev,
+ "Unable to allocate %d MCAM entries for ntuple, got %d\n",
+ count, allocated);
+
+ return allocated;
+}
+
int otx2_alloc_mcam_entries(struct otx2_nic *pfvf)
{
struct otx2_flow_config *flow_cfg = pfvf->flow_cfg;
struct npc_mcam_alloc_entry_req *req;
struct npc_mcam_alloc_entry_rsp *rsp;
int vf_vlan_max_flows;
- int i;
+ int ent, count;
+
+ vf_vlan_max_flows = pfvf->total_vfs * OTX2_PER_VF_VLAN_FLOWS;
+ count = OTX2_MAX_UNICAST_FLOWS +
+ OTX2_MAX_VLAN_FLOWS + vf_vlan_max_flows;
+
+ flow_cfg->def_ent = devm_kmalloc_array(pfvf->dev, count,
+ sizeof(u16), GFP_KERNEL);
+ if (!flow_cfg->def_ent)
+ return -ENOMEM;
mutex_lock(&pfvf->mbox.lock);
return -ENOMEM;
}
- vf_vlan_max_flows = pfvf->total_vfs * OTX2_PER_VF_VLAN_FLOWS;
req->contig = false;
- req->count = OTX2_MCAM_COUNT + vf_vlan_max_flows;
+ req->count = count;
/* Send message to AF */
if (otx2_sync_mbox_msg(&pfvf->mbox)) {
if (rsp->count != req->count) {
netdev_info(pfvf->netdev,
- "Unable to allocate %d MCAM entries, got %d\n",
- req->count, rsp->count);
- /* support only ntuples here */
- flow_cfg->ntuple_max_flows = rsp->count;
- flow_cfg->ntuple_offset = 0;
- pfvf->flags |= OTX2_FLAG_NTUPLE_SUPPORT;
- flow_cfg->tc_max_flows = flow_cfg->ntuple_max_flows;
- pfvf->flags |= OTX2_FLAG_TC_FLOWER_SUPPORT;
- } else {
- flow_cfg->vf_vlan_offset = 0;
- flow_cfg->ntuple_offset = flow_cfg->vf_vlan_offset +
- vf_vlan_max_flows;
- flow_cfg->tc_flower_offset = flow_cfg->ntuple_offset;
- flow_cfg->unicast_offset = flow_cfg->ntuple_offset +
- OTX2_MAX_NTUPLE_FLOWS;
- flow_cfg->rx_vlan_offset = flow_cfg->unicast_offset +
- OTX2_MAX_UNICAST_FLOWS;
- pfvf->flags |= OTX2_FLAG_NTUPLE_SUPPORT;
- pfvf->flags |= OTX2_FLAG_UCAST_FLTR_SUPPORT;
- pfvf->flags |= OTX2_FLAG_RX_VLAN_SUPPORT;
- pfvf->flags |= OTX2_FLAG_VF_VLAN_SUPPORT;
- pfvf->flags |= OTX2_FLAG_TC_FLOWER_SUPPORT;
- }
-
- for (i = 0; i < rsp->count; i++)
- flow_cfg->entry[i] = rsp->entry_list[i];
+ "Unable to allocate MCAM entries for ucast, vlan and vf_vlan\n");
+ mutex_unlock(&pfvf->mbox.lock);
+ devm_kfree(pfvf->dev, flow_cfg->def_ent);
+ return 0;
+ }
- pfvf->flags |= OTX2_FLAG_MCAM_ENTRIES_ALLOC;
+ for (ent = 0; ent < rsp->count; ent++)
+ flow_cfg->def_ent[ent] = rsp->entry_list[ent];
+
+ flow_cfg->vf_vlan_offset = 0;
+ flow_cfg->unicast_offset = vf_vlan_max_flows;
+ flow_cfg->rx_vlan_offset = flow_cfg->unicast_offset +
+ OTX2_MAX_UNICAST_FLOWS;
+ pfvf->flags |= OTX2_FLAG_UCAST_FLTR_SUPPORT;
+ pfvf->flags |= OTX2_FLAG_RX_VLAN_SUPPORT;
+ pfvf->flags |= OTX2_FLAG_VF_VLAN_SUPPORT;
+ pfvf->flags |= OTX2_FLAG_MCAM_ENTRIES_ALLOC;
mutex_unlock(&pfvf->mbox.lock);
+ /* Allocate entries for Ntuple filters */
+ count = otx2_alloc_ntuple_mcam_entries(pfvf, OTX2_DEFAULT_FLOWCOUNT);
+ if (count <= 0) {
+ otx2_clear_ntuple_flow_info(pfvf, flow_cfg);
+ return 0;
+ }
+
+ pfvf->flags |= OTX2_FLAG_NTUPLE_SUPPORT;
+ pfvf->flags |= OTX2_FLAG_TC_FLOWER_SUPPORT;
+
return 0;
}
INIT_LIST_HEAD(&pf->flow_cfg->flow_list);
- pf->flow_cfg->ntuple_max_flows = OTX2_MAX_NTUPLE_FLOWS;
- pf->flow_cfg->tc_max_flows = pf->flow_cfg->ntuple_max_flows;
-
err = otx2_alloc_mcam_entries(pf);
if (err)
return err;
+ /* Check if MCAM entries are allocate or not */
+ if (!(pf->flags & OTX2_FLAG_UCAST_FLTR_SUPPORT))
+ return 0;
+
pf->mac_table = devm_kzalloc(pf->dev, sizeof(struct otx2_mac_table)
* OTX2_MAX_UNICAST_FLOWS, GFP_KERNEL);
if (!pf->mac_table)
ether_addr_copy(pf->mac_table[i].addr, mac);
pf->mac_table[i].inuse = true;
pf->mac_table[i].mcam_entry =
- flow_cfg->entry[i + flow_cfg->unicast_offset];
+ flow_cfg->def_ent[i + flow_cfg->unicast_offset];
req->entry = pf->mac_table[i].mcam_entry;
break;
}
req->features |= BIT_ULL(NPC_IPPROTO_AH);
else
req->features |= BIT_ULL(NPC_IPPROTO_ESP);
+ break;
default:
break;
}
if (!flow)
return -ENOMEM;
flow->location = fsp->location;
- flow->entry = flow_cfg->entry[flow_cfg->ntuple_offset +
- flow->location];
+ flow->entry = flow_cfg->flow_ent[flow->location];
new = true;
}
/* struct copy */
return -ENOMEM;
}
- req->start = flow_cfg->entry[flow_cfg->ntuple_offset];
- req->end = flow_cfg->entry[flow_cfg->ntuple_offset +
- flow_cfg->ntuple_max_flows - 1];
+ req->start = flow_cfg->flow_ent[0];
+ req->end = flow_cfg->flow_ent[flow_cfg->ntuple_max_flows - 1];
err = otx2_sync_mbox_msg(&pfvf->mbox);
mutex_unlock(&pfvf->mbox.lock);
return -ENOMEM;
}
- req->entry = flow_cfg->entry[flow_cfg->rx_vlan_offset];
+ req->entry = flow_cfg->def_ent[flow_cfg->rx_vlan_offset];
req->intf = NIX_INTF_RX;
ether_addr_copy(req->packet.dmac, pfvf->netdev->dev_addr);
eth_broadcast_addr((u8 *)&req->mask.dmac);
return -ENOMEM;
}
- req->entry = flow_cfg->entry[flow_cfg->rx_vlan_offset];
+ req->entry = flow_cfg->def_ent[flow_cfg->rx_vlan_offset];
/* Send message to AF */
err = otx2_sync_mbox_msg(&pfvf->mbox);
mutex_unlock(&pfvf->mbox.lock);
MODULE_LICENSE("GPL v2");
MODULE_DEVICE_TABLE(pci, otx2_pf_id_table);
+static void otx2_vf_link_event_task(struct work_struct *work);
+
enum {
TYPE_PFAF,
TYPE_PFVF,
otx2_free_cq_res(pf);
+ /* Free all ingress bandwidth profiles allocated */
+ cn10k_free_all_ipolicers(pf);
+
mutex_lock(&mbox->lock);
/* Reset NIX LF */
free_req = otx2_mbox_alloc_msg_nix_lf_free(mbox);
if (promisc)
req->mode |= NIX_RX_MODE_PROMISC;
- else if (netdev->flags & (IFF_ALLMULTI | IFF_MULTICAST))
+ if (netdev->flags & (IFF_ALLMULTI | IFF_MULTICAST))
req->mode |= NIX_RX_MODE_ALLMULTI;
+ req->mode |= NIX_RX_MODE_USE_MCE;
+
otx2_sync_mbox_msg(&pf->mbox);
mutex_unlock(&pf->mbox.lock);
}
if (!netif_running(netdev))
return -EAGAIN;
- if (vf >= pci_num_vf(pdev))
+ if (vf >= pf->total_vfs)
return -EINVAL;
if (!is_valid_ether_addr(mac))
ret = otx2_do_set_vf_mac(pf, vf, mac);
if (ret == 0)
- dev_info(&pdev->dev, "Reload VF driver to apply the changes\n");
+ dev_info(&pdev->dev,
+ "Load/Reload VF driver\n");
return ret;
}
}
idx = ((vf * OTX2_PER_VF_VLAN_FLOWS) + OTX2_VF_VLAN_RX_INDEX);
del_req->entry =
- flow_cfg->entry[flow_cfg->vf_vlan_offset + idx];
+ flow_cfg->def_ent[flow_cfg->vf_vlan_offset + idx];
err = otx2_sync_mbox_msg(&pf->mbox);
if (err)
goto out;
}
idx = ((vf * OTX2_PER_VF_VLAN_FLOWS) + OTX2_VF_VLAN_TX_INDEX);
del_req->entry =
- flow_cfg->entry[flow_cfg->vf_vlan_offset + idx];
+ flow_cfg->def_ent[flow_cfg->vf_vlan_offset + idx];
err = otx2_sync_mbox_msg(&pf->mbox);
goto out;
}
idx = ((vf * OTX2_PER_VF_VLAN_FLOWS) + OTX2_VF_VLAN_RX_INDEX);
- req->entry = flow_cfg->entry[flow_cfg->vf_vlan_offset + idx];
+ req->entry = flow_cfg->def_ent[flow_cfg->vf_vlan_offset + idx];
req->packet.vlan_tci = htons(vlan);
req->mask.vlan_tci = htons(VLAN_VID_MASK);
/* af fills the destination mac addr */
eth_zero_addr((u8 *)&req->mask.dmac);
idx = ((vf * OTX2_PER_VF_VLAN_FLOWS) + OTX2_VF_VLAN_TX_INDEX);
- req->entry = flow_cfg->entry[flow_cfg->vf_vlan_offset + idx];
+ req->entry = flow_cfg->def_ent[flow_cfg->vf_vlan_offset + idx];
req->features = BIT_ULL(NPC_DMAC);
req->channel = pf->hw.tx_chan_base;
req->intf = NIX_INTF_TX;
ivi->vf = vf;
ether_addr_copy(ivi->mac, config->mac);
ivi->vlan = config->vlan;
+ ivi->trusted = config->trusted;
return 0;
}
+static int otx2_set_vf_permissions(struct otx2_nic *pf, int vf,
+ int req_perm)
+{
+ struct set_vf_perm *req;
+ int rc;
+
+ mutex_lock(&pf->mbox.lock);
+ req = otx2_mbox_alloc_msg_set_vf_perm(&pf->mbox);
+ if (!req) {
+ rc = -ENOMEM;
+ goto out;
+ }
+
+ /* Let AF reset VF permissions as sriov is disabled */
+ if (req_perm == OTX2_RESET_VF_PERM) {
+ req->flags |= RESET_VF_PERM;
+ } else if (req_perm == OTX2_TRUSTED_VF) {
+ if (pf->vf_configs[vf].trusted)
+ req->flags |= VF_TRUSTED;
+ }
+
+ req->vf = vf;
+ rc = otx2_sync_mbox_msg(&pf->mbox);
+out:
+ mutex_unlock(&pf->mbox.lock);
+ return rc;
+}
+
+static int otx2_ndo_set_vf_trust(struct net_device *netdev, int vf,
+ bool enable)
+{
+ struct otx2_nic *pf = netdev_priv(netdev);
+ struct pci_dev *pdev = pf->pdev;
+ int rc;
+
+ if (vf >= pci_num_vf(pdev))
+ return -EINVAL;
+
+ if (pf->vf_configs[vf].trusted == enable)
+ return 0;
+
+ pf->vf_configs[vf].trusted = enable;
+ rc = otx2_set_vf_permissions(pf, vf, OTX2_TRUSTED_VF);
+
+ if (rc)
+ pf->vf_configs[vf].trusted = !enable;
+ else
+ netdev_info(pf->netdev, "VF %d is %strusted\n",
+ vf, enable ? "" : "not ");
+ return rc;
+}
+
static const struct net_device_ops otx2_netdev_ops = {
.ndo_open = otx2_open,
.ndo_stop = otx2_stop,
.ndo_set_vf_vlan = otx2_set_vf_vlan,
.ndo_get_vf_config = otx2_get_vf_config,
.ndo_setup_tc = otx2_setup_tc,
+ .ndo_set_vf_trust = otx2_ndo_set_vf_trust,
};
static int otx2_wq_init(struct otx2_nic *pf)
return otx2_register_mbox_intr(pf, false);
}
+static int otx2_sriov_vfcfg_init(struct otx2_nic *pf)
+{
+ int i;
+
+ pf->vf_configs = devm_kcalloc(pf->dev, pf->total_vfs,
+ sizeof(struct otx2_vf_config),
+ GFP_KERNEL);
+ if (!pf->vf_configs)
+ return -ENOMEM;
+
+ for (i = 0; i < pf->total_vfs; i++) {
+ pf->vf_configs[i].pf = pf;
+ pf->vf_configs[i].intf_down = true;
+ pf->vf_configs[i].trusted = false;
+ INIT_DELAYED_WORK(&pf->vf_configs[i].link_event_work,
+ otx2_vf_link_event_task);
+ }
+
+ return 0;
+}
+
+static void otx2_sriov_vfcfg_cleanup(struct otx2_nic *pf)
+{
+ int i;
+
+ if (!pf->vf_configs)
+ return;
+
+ for (i = 0; i < pf->total_vfs; i++) {
+ cancel_delayed_work_sync(&pf->vf_configs[i].link_event_work);
+ otx2_set_vf_permissions(pf, i, OTX2_RESET_VF_PERM);
+ }
+}
+
static int otx2_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
struct device *dev = &pdev->dev;
if (err)
goto err_mcam_flow_del;
+ /* Initialize SR-IOV resources */
+ err = otx2_sriov_vfcfg_init(pf);
+ if (err)
+ goto err_pf_sriov_init;
+
/* Enable link notifications */
otx2_cgx_config_linkevents(pf, true);
return 0;
+err_pf_sriov_init:
+ otx2_shutdown_tc(pf);
err_mcam_flow_del:
otx2_mcam_flow_del(pf);
err_unreg_netdev:
{
struct net_device *netdev = pci_get_drvdata(pdev);
struct otx2_nic *pf = netdev_priv(netdev);
- int ret, i;
+ int ret;
/* Init PF <=> VF mailbox stuff */
ret = otx2_pfvf_mbox_init(pf, numvfs);
if (ret)
goto free_mbox;
- pf->vf_configs = kcalloc(numvfs, sizeof(struct otx2_vf_config),
- GFP_KERNEL);
- if (!pf->vf_configs) {
- ret = -ENOMEM;
- goto free_intr;
- }
-
- for (i = 0; i < numvfs; i++) {
- pf->vf_configs[i].pf = pf;
- pf->vf_configs[i].intf_down = true;
- INIT_DELAYED_WORK(&pf->vf_configs[i].link_event_work,
- otx2_vf_link_event_task);
- }
-
ret = otx2_pf_flr_init(pf, numvfs);
if (ret)
- goto free_configs;
+ goto free_intr;
ret = otx2_register_flr_me_intr(pf, numvfs);
if (ret)
otx2_disable_flr_me_intr(pf);
free_flr:
otx2_flr_wq_destroy(pf);
-free_configs:
- kfree(pf->vf_configs);
free_intr:
otx2_disable_pfvf_mbox_intr(pf, numvfs);
free_mbox:
struct net_device *netdev = pci_get_drvdata(pdev);
struct otx2_nic *pf = netdev_priv(netdev);
int numvfs = pci_num_vf(pdev);
- int i;
if (!numvfs)
return 0;
pci_disable_sriov(pdev);
- for (i = 0; i < pci_num_vf(pdev); i++)
- cancel_delayed_work_sync(&pf->vf_configs[i].link_event_work);
- kfree(pf->vf_configs);
-
otx2_disable_flr_me_intr(pf);
otx2_flr_wq_destroy(pf);
otx2_disable_pfvf_mbox_intr(pf, numvfs);
unregister_netdev(netdev);
otx2_sriov_disable(pf->pdev);
+ otx2_sriov_vfcfg_cleanup(pf);
if (pf->otx2_wq)
destroy_workqueue(pf->otx2_wq);
#include <net/tc_act/tc_vlan.h>
#include <net/ipv6.h>
+#include "cn10k.h"
#include "otx2_common.h"
/* Egress rate limiting definitions */
struct otx2_tc_flow {
struct rhash_head node;
unsigned long cookie;
- u16 entry;
unsigned int bitpos;
struct rcu_head rcu;
struct otx2_tc_flow_stats stats;
spinlock_t lock; /* lock for stats */
+ u16 rq;
+ u16 entry;
+ u16 leaf_profile;
+ bool is_act_police;
};
static void otx2_get_egress_burst_cfg(u32 burst, u32 *burst_exp,
return err;
}
+static int otx2_tc_act_set_police(struct otx2_nic *nic,
+ struct otx2_tc_flow *node,
+ struct flow_cls_offload *f,
+ u64 rate, u32 burst, u32 mark,
+ struct npc_install_flow_req *req, bool pps)
+{
+ struct netlink_ext_ack *extack = f->common.extack;
+ struct otx2_hw *hw = &nic->hw;
+ int rq_idx, rc;
+
+ rq_idx = find_first_zero_bit(&nic->rq_bmap, hw->rx_queues);
+ if (rq_idx >= hw->rx_queues) {
+ NL_SET_ERR_MSG_MOD(extack, "Police action rules exceeded");
+ return -EINVAL;
+ }
+
+ mutex_lock(&nic->mbox.lock);
+
+ rc = cn10k_alloc_leaf_profile(nic, &node->leaf_profile);
+ if (rc) {
+ mutex_unlock(&nic->mbox.lock);
+ return rc;
+ }
+
+ rc = cn10k_set_ipolicer_rate(nic, node->leaf_profile, burst, rate, pps);
+ if (rc)
+ goto free_leaf;
+
+ rc = cn10k_map_unmap_rq_policer(nic, rq_idx, node->leaf_profile, true);
+ if (rc)
+ goto free_leaf;
+
+ mutex_unlock(&nic->mbox.lock);
+
+ req->match_id = mark & 0xFFFFULL;
+ req->index = rq_idx;
+ req->op = NIX_RX_ACTIONOP_UCAST;
+ set_bit(rq_idx, &nic->rq_bmap);
+ node->is_act_police = true;
+ node->rq = rq_idx;
+
+ return 0;
+
+free_leaf:
+ if (cn10k_free_leaf_profile(nic, node->leaf_profile))
+ netdev_err(nic->netdev,
+ "Unable to free leaf bandwidth profile(%d)\n",
+ node->leaf_profile);
+ mutex_unlock(&nic->mbox.lock);
+ return rc;
+}
+
static int otx2_tc_parse_actions(struct otx2_nic *nic,
struct flow_action *flow_action,
- struct npc_install_flow_req *req)
+ struct npc_install_flow_req *req,
+ struct flow_cls_offload *f,
+ struct otx2_tc_flow *node)
{
+ struct netlink_ext_ack *extack = f->common.extack;
struct flow_action_entry *act;
struct net_device *target;
struct otx2_nic *priv;
+ u32 burst, mark = 0;
+ u8 nr_police = 0;
+ bool pps;
+ u64 rate;
int i;
if (!flow_action_has_entries(flow_action)) {
- netdev_info(nic->netdev, "no tc actions specified");
+ NL_SET_ERR_MSG_MOD(extack, "no tc actions specified");
return -EINVAL;
}
priv = netdev_priv(target);
/* npc_install_flow_req doesn't support passing a target pcifunc */
if (rvu_get_pf(nic->pcifunc) != rvu_get_pf(priv->pcifunc)) {
- netdev_info(nic->netdev,
- "can't redirect to other pf/vf\n");
+ NL_SET_ERR_MSG_MOD(extack,
+ "can't redirect to other pf/vf");
return -EOPNOTSUPP;
}
req->vf = priv->pcifunc & RVU_PFVF_FUNC_MASK;
/* use RX_VTAG_TYPE7 which is initialized to strip vlan tag */
req->vtag0_type = NIX_AF_LFX_RX_VTAG_TYPE7;
break;
+ case FLOW_ACTION_POLICE:
+ /* Ingress ratelimiting is not supported on OcteonTx2 */
+ if (is_dev_otx2(nic->pdev)) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "Ingress policing not supported on this platform");
+ return -EOPNOTSUPP;
+ }
+
+ if (act->police.rate_bytes_ps > 0) {
+ rate = act->police.rate_bytes_ps * 8;
+ burst = act->police.burst;
+ } else if (act->police.rate_pkt_ps > 0) {
+ /* The algorithm used to calculate rate
+ * mantissa, exponent values for a given token
+ * rate (token can be byte or packet) requires
+ * token rate to be mutiplied by 8.
+ */
+ rate = act->police.rate_pkt_ps * 8;
+ burst = act->police.burst_pkt;
+ pps = true;
+ }
+ nr_police++;
+ break;
+ case FLOW_ACTION_MARK:
+ mark = act->mark;
+ break;
default:
return -EOPNOTSUPP;
}
}
+ if (nr_police > 1) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "rate limit police offload requires a single action");
+ return -EOPNOTSUPP;
+ }
+
+ if (nr_police)
+ return otx2_tc_act_set_police(nic, node, f, rate, burst,
+ mark, req, pps);
+
return 0;
}
-static int otx2_tc_prepare_flow(struct otx2_nic *nic,
+static int otx2_tc_prepare_flow(struct otx2_nic *nic, struct otx2_tc_flow *node,
struct flow_cls_offload *f,
struct npc_install_flow_req *req)
{
+ struct netlink_ext_ack *extack = f->common.extack;
struct flow_msg *flow_spec = &req->packet;
struct flow_msg *flow_mask = &req->mask;
struct flow_dissector *dissector;
flow_rule_match_eth_addrs(rule, &match);
if (!is_zero_ether_addr(match.mask->src)) {
- netdev_err(nic->netdev, "src mac match not supported\n");
+ NL_SET_ERR_MSG_MOD(extack, "src mac match not supported");
return -EOPNOTSUPP;
}
flow_rule_match_ip(rule, &match);
if ((ntohs(flow_spec->etype) != ETH_P_IP) &&
match.mask->tos) {
- netdev_err(nic->netdev, "tos not supported\n");
+ NL_SET_ERR_MSG_MOD(extack, "tos not supported");
return -EOPNOTSUPP;
}
if (match.mask->ttl) {
- netdev_err(nic->netdev, "ttl not supported\n");
+ NL_SET_ERR_MSG_MOD(extack, "ttl not supported");
return -EOPNOTSUPP;
}
flow_spec->tos = match.key->tos;
if (ipv6_addr_loopback(&match.key->dst) ||
ipv6_addr_loopback(&match.key->src)) {
- netdev_err(nic->netdev,
- "Flow matching on IPv6 loopback addr is not supported\n");
+ NL_SET_ERR_MSG_MOD(extack,
+ "Flow matching IPv6 loopback addr not supported");
return -EOPNOTSUPP;
}
req->features |= BIT_ULL(NPC_SPORT_SCTP);
}
- return otx2_tc_parse_actions(nic, &rule->action, req);
+ return otx2_tc_parse_actions(nic, &rule->action, req, f, node);
}
static int otx2_del_mcam_flow_entry(struct otx2_nic *nic, u16 entry)
{
struct otx2_tc_info *tc_info = &nic->tc_info;
struct otx2_tc_flow *flow_node;
+ int err;
flow_node = rhashtable_lookup_fast(&tc_info->flow_table,
&tc_flow_cmd->cookie,
return -EINVAL;
}
+ if (flow_node->is_act_police) {
+ mutex_lock(&nic->mbox.lock);
+
+ err = cn10k_map_unmap_rq_policer(nic, flow_node->rq,
+ flow_node->leaf_profile, false);
+ if (err)
+ netdev_err(nic->netdev,
+ "Unmapping RQ %d & profile %d failed\n",
+ flow_node->rq, flow_node->leaf_profile);
+
+ err = cn10k_free_leaf_profile(nic, flow_node->leaf_profile);
+ if (err)
+ netdev_err(nic->netdev,
+ "Unable to free leaf bandwidth profile(%d)\n",
+ flow_node->leaf_profile);
+
+ __clear_bit(flow_node->rq, &nic->rq_bmap);
+
+ mutex_unlock(&nic->mbox.lock);
+ }
+
otx2_del_mcam_flow_entry(nic, flow_node->entry);
WARN_ON(rhashtable_remove_fast(&nic->tc_info.flow_table,
static int otx2_tc_add_flow(struct otx2_nic *nic,
struct flow_cls_offload *tc_flow_cmd)
{
+ struct netlink_ext_ack *extack = tc_flow_cmd->common.extack;
struct otx2_tc_info *tc_info = &nic->tc_info;
struct otx2_tc_flow *new_node, *old_node;
- struct npc_install_flow_req *req;
- int rc;
+ struct npc_install_flow_req *req, dummy;
+ int rc, err;
if (!(nic->flags & OTX2_FLAG_TC_FLOWER_SUPPORT))
return -ENOMEM;
+ if (bitmap_full(tc_info->tc_entries_bitmap, nic->flow_cfg->tc_max_flows)) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "Not enough MCAM space to add the flow");
+ return -ENOMEM;
+ }
+
/* allocate memory for the new flow and it's node */
new_node = kzalloc(sizeof(*new_node), GFP_KERNEL);
if (!new_node)
spin_lock_init(&new_node->lock);
new_node->cookie = tc_flow_cmd->cookie;
- mutex_lock(&nic->mbox.lock);
- req = otx2_mbox_alloc_msg_npc_install_flow(&nic->mbox);
- if (!req) {
- mutex_unlock(&nic->mbox.lock);
- return -ENOMEM;
- }
+ memset(&dummy, 0, sizeof(struct npc_install_flow_req));
- rc = otx2_tc_prepare_flow(nic, tc_flow_cmd, req);
+ rc = otx2_tc_prepare_flow(nic, new_node, tc_flow_cmd, &dummy);
if (rc) {
- otx2_mbox_reset(&nic->mbox.mbox, 0);
- mutex_unlock(&nic->mbox.lock);
+ kfree_rcu(new_node, rcu);
return rc;
}
if (old_node)
otx2_tc_del_flow(nic, tc_flow_cmd);
- if (bitmap_full(tc_info->tc_entries_bitmap, nic->flow_cfg->tc_max_flows)) {
- netdev_err(nic->netdev, "Not enough MCAM space to add the flow\n");
- otx2_mbox_reset(&nic->mbox.mbox, 0);
+ mutex_lock(&nic->mbox.lock);
+ req = otx2_mbox_alloc_msg_npc_install_flow(&nic->mbox);
+ if (!req) {
mutex_unlock(&nic->mbox.lock);
- return -ENOMEM;
+ rc = -ENOMEM;
+ goto free_leaf;
}
+ memcpy(&dummy.hdr, &req->hdr, sizeof(struct mbox_msghdr));
+ memcpy(req, &dummy, sizeof(struct npc_install_flow_req));
+
new_node->bitpos = find_first_zero_bit(tc_info->tc_entries_bitmap,
nic->flow_cfg->tc_max_flows);
req->channel = nic->hw.rx_chan_base;
- req->entry = nic->flow_cfg->entry[nic->flow_cfg->tc_flower_offset +
- nic->flow_cfg->tc_max_flows - new_node->bitpos];
+ req->entry = nic->flow_cfg->flow_ent[nic->flow_cfg->tc_flower_offset +
+ nic->flow_cfg->tc_max_flows - new_node->bitpos];
req->intf = NIX_INTF_RX;
req->set_cntr = 1;
new_node->entry = req->entry;
/* Send message to AF */
rc = otx2_sync_mbox_msg(&nic->mbox);
if (rc) {
- netdev_err(nic->netdev, "Failed to install MCAM flow entry\n");
+ NL_SET_ERR_MSG_MOD(extack, "Failed to install MCAM flow entry");
mutex_unlock(&nic->mbox.lock);
- goto out;
+ kfree_rcu(new_node, rcu);
+ goto free_leaf;
}
mutex_unlock(&nic->mbox.lock);
if (rc) {
otx2_del_mcam_flow_entry(nic, req->entry);
kfree_rcu(new_node, rcu);
- goto out;
+ goto free_leaf;
}
set_bit(new_node->bitpos, tc_info->tc_entries_bitmap);
tc_info->num_entries++;
-out:
+
+ return 0;
+
+free_leaf:
+ if (new_node->is_act_police) {
+ mutex_lock(&nic->mbox.lock);
+
+ err = cn10k_map_unmap_rq_policer(nic, new_node->rq,
+ new_node->leaf_profile, false);
+ if (err)
+ netdev_err(nic->netdev,
+ "Unmapping RQ %d & profile %d failed\n",
+ new_node->rq, new_node->leaf_profile);
+ err = cn10k_free_leaf_profile(nic, new_node->leaf_profile);
+ if (err)
+ netdev_err(nic->netdev,
+ "Unable to free leaf bandwidth profile(%d)\n",
+ new_node->leaf_profile);
+
+ __clear_bit(new_node->rq, &nic->rq_bmap);
+
+ mutex_unlock(&nic->mbox.lock);
+ }
+
return rc;
}
}
}
+static int otx2_tc_ingress_matchall_install(struct otx2_nic *nic,
+ struct tc_cls_matchall_offload *cls)
+{
+ struct netlink_ext_ack *extack = cls->common.extack;
+ struct flow_action *actions = &cls->rule->action;
+ struct flow_action_entry *entry;
+ u64 rate;
+ int err;
+
+ err = otx2_tc_validate_flow(nic, actions, extack);
+ if (err)
+ return err;
+
+ if (nic->flags & OTX2_FLAG_TC_MATCHALL_INGRESS_ENABLED) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "Only one ingress MATCHALL ratelimitter can be offloaded");
+ return -ENOMEM;
+ }
+
+ entry = &cls->rule->action.entries[0];
+ switch (entry->id) {
+ case FLOW_ACTION_POLICE:
+ /* Ingress ratelimiting is not supported on OcteonTx2 */
+ if (is_dev_otx2(nic->pdev)) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "Ingress policing not supported on this platform");
+ return -EOPNOTSUPP;
+ }
+
+ err = cn10k_alloc_matchall_ipolicer(nic);
+ if (err)
+ return err;
+
+ /* Convert to bits per second */
+ rate = entry->police.rate_bytes_ps * 8;
+ err = cn10k_set_matchall_ipolicer_rate(nic, entry->police.burst, rate);
+ if (err)
+ return err;
+ nic->flags |= OTX2_FLAG_TC_MATCHALL_INGRESS_ENABLED;
+ break;
+ default:
+ NL_SET_ERR_MSG_MOD(extack,
+ "Only police action supported with Ingress MATCHALL offload");
+ return -EOPNOTSUPP;
+ }
+
+ return 0;
+}
+
+static int otx2_tc_ingress_matchall_delete(struct otx2_nic *nic,
+ struct tc_cls_matchall_offload *cls)
+{
+ struct netlink_ext_ack *extack = cls->common.extack;
+ int err;
+
+ if (nic->flags & OTX2_FLAG_INTF_DOWN) {
+ NL_SET_ERR_MSG_MOD(extack, "Interface not initialized");
+ return -EINVAL;
+ }
+
+ err = cn10k_free_matchall_ipolicer(nic);
+ nic->flags &= ~OTX2_FLAG_TC_MATCHALL_INGRESS_ENABLED;
+ return err;
+}
+
+static int otx2_setup_tc_ingress_matchall(struct otx2_nic *nic,
+ struct tc_cls_matchall_offload *cls_matchall)
+{
+ switch (cls_matchall->command) {
+ case TC_CLSMATCHALL_REPLACE:
+ return otx2_tc_ingress_matchall_install(nic, cls_matchall);
+ case TC_CLSMATCHALL_DESTROY:
+ return otx2_tc_ingress_matchall_delete(nic, cls_matchall);
+ case TC_CLSMATCHALL_STATS:
+ default:
+ break;
+ }
+
+ return -EOPNOTSUPP;
+}
+
static int otx2_setup_tc_block_ingress_cb(enum tc_setup_type type,
void *type_data, void *cb_priv)
{
switch (type) {
case TC_SETUP_CLSFLOWER:
return otx2_setup_tc_cls_flower(nic, type_data);
+ case TC_SETUP_CLSMATCHALL:
+ return otx2_setup_tc_ingress_matchall(nic, type_data);
default:
break;
}
{
struct otx2_tc_info *tc = &nic->tc_info;
+ /* Exclude receive queue 0 being used for police action */
+ set_bit(0, &nic->rq_bmap);
+
tc->flow_ht_params = tc_flow_ht_params;
return rhashtable_init(&tc->flow_table, &tc->flow_ht_params);
}
return NETDEV_TX_OK;
}
+static void otx2vf_set_rx_mode(struct net_device *netdev)
+{
+ struct otx2_nic *vf = netdev_priv(netdev);
+
+ queue_work(vf->otx2_wq, &vf->rx_mode_work);
+}
+
+static void otx2vf_do_set_rx_mode(struct work_struct *work)
+{
+ struct otx2_nic *vf = container_of(work, struct otx2_nic, rx_mode_work);
+ struct net_device *netdev = vf->netdev;
+ unsigned int flags = netdev->flags;
+ struct nix_rx_mode *req;
+
+ mutex_lock(&vf->mbox.lock);
+
+ req = otx2_mbox_alloc_msg_nix_set_rx_mode(&vf->mbox);
+ if (!req) {
+ mutex_unlock(&vf->mbox.lock);
+ return;
+ }
+
+ req->mode = NIX_RX_MODE_UCAST;
+
+ if (flags & IFF_PROMISC)
+ req->mode |= NIX_RX_MODE_PROMISC;
+ if (flags & (IFF_ALLMULTI | IFF_MULTICAST))
+ req->mode |= NIX_RX_MODE_ALLMULTI;
+
+ req->mode |= NIX_RX_MODE_USE_MCE;
+
+ otx2_sync_mbox_msg(&vf->mbox);
+
+ mutex_unlock(&vf->mbox.lock);
+}
+
static int otx2vf_change_mtu(struct net_device *netdev, int new_mtu)
{
bool if_up = netif_running(netdev);
.ndo_open = otx2vf_open,
.ndo_stop = otx2vf_stop,
.ndo_start_xmit = otx2vf_xmit,
+ .ndo_set_rx_mode = otx2vf_set_rx_mode,
.ndo_set_mac_address = otx2_set_mac_address,
.ndo_change_mtu = otx2vf_change_mtu,
.ndo_get_stats64 = otx2_get_stats64,
.ndo_tx_timeout = otx2_tx_timeout,
};
+static int otx2_wq_init(struct otx2_nic *vf)
+{
+ vf->otx2_wq = create_singlethread_workqueue("otx2vf_wq");
+ if (!vf->otx2_wq)
+ return -ENOMEM;
+
+ INIT_WORK(&vf->rx_mode_work, otx2vf_do_set_rx_mode);
+ INIT_WORK(&vf->reset_task, otx2vf_reset_task);
+ return 0;
+}
+
static int otx2vf_realloc_msix_vectors(struct otx2_nic *vf)
{
struct otx2_hw *hw = &vf->hw;
netdev->min_mtu = OTX2_MIN_MTU;
netdev->max_mtu = otx2_get_max_mtu(vf);
- INIT_WORK(&vf->reset_task, otx2vf_reset_task);
-
/* To distinguish, for LBK VFs set netdev name explicitly */
if (is_otx2_lbkvf(vf->pdev)) {
int n;
goto err_detach_rsrc;
}
+ err = otx2_wq_init(vf);
+ if (err)
+ goto err_unreg_netdev;
+
otx2vf_set_ethtool_ops(netdev);
/* Enable pause frames by default */
return 0;
+err_unreg_netdev:
+ unregister_netdev(netdev);
err_detach_rsrc:
if (hw->lmt_base)
iounmap(hw->lmt_base);
cancel_work_sync(&vf->reset_task);
unregister_netdev(netdev);
+ if (vf->otx2_wq)
+ destroy_workqueue(vf->otx2_wq);
otx2vf_disable_mbox_intr(vf);
otx2_detach_resources(&vf->mbox);
obj-$(CONFIG_PRESTERA) += prestera.o
prestera-objs := prestera_main.o prestera_hw.o prestera_dsa.o \
prestera_rxtx.o prestera_devlink.o prestera_ethtool.o \
- prestera_switchdev.o
+ prestera_switchdev.o prestera_acl.o prestera_flow.o \
+ prestera_flower.o prestera_span.o
obj-$(CONFIG_PRESTERA_PCI) += prestera_pci.o
u8 transceiver;
};
+struct prestera_lag {
+ struct net_device *dev;
+ struct list_head members;
+ u16 member_count;
+ u16 lag_id;
+};
+
+struct prestera_flow_block;
+
struct prestera_port {
struct net_device *dev;
struct prestera_switch *sw;
+ struct prestera_flow_block *flow_block;
struct devlink_port dl_port;
+ struct list_head lag_member;
+ struct prestera_lag *lag;
u32 id;
u32 hw_id;
u32 dev_id;
} data;
};
+enum prestera_fdb_entry_type {
+ PRESTERA_FDB_ENTRY_TYPE_REG_PORT,
+ PRESTERA_FDB_ENTRY_TYPE_LAG,
+ PRESTERA_FDB_ENTRY_TYPE_MAX
+};
+
enum prestera_fdb_event_id {
PRESTERA_FDB_EVENT_UNSPEC,
PRESTERA_FDB_EVENT_LEARNED,
};
struct prestera_fdb_event {
- u32 port_id;
+ enum prestera_fdb_entry_type type;
+ union {
+ u32 port_id;
+ u16 lag_id;
+ } dest;
u32 vid;
union {
u8 mac[ETH_ALEN];
};
struct prestera_switchdev;
+struct prestera_span;
struct prestera_rxtx;
+struct prestera_trap_data;
+struct prestera_acl;
struct prestera_switch {
struct prestera_device *dev;
struct prestera_switchdev *swdev;
struct prestera_rxtx *rxtx;
+ struct prestera_acl *acl;
+ struct prestera_span *span;
struct list_head event_handlers;
struct notifier_block netdev_nb;
+ struct prestera_trap_data *trap_data;
char base_mac[ETH_ALEN];
struct list_head port_list;
rwlock_t port_list_lock;
u32 mtu_min;
u32 mtu_max;
u8 id;
+ struct prestera_lag *lags;
+ u8 lag_member_max;
+ u8 lag_max;
};
struct prestera_rxtx_params {
bool prestera_netdev_check(const struct net_device *dev);
+bool prestera_port_is_lag_member(const struct prestera_port *port);
+
+struct prestera_lag *prestera_lag_by_id(struct prestera_switch *sw, u16 id);
+
+u16 prestera_port_lag_id(const struct prestera_port *port);
+
#endif /* _PRESTERA_H_ */
--- /dev/null
+// SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0
+/* Copyright (c) 2020 Marvell International Ltd. All rights reserved */
+
+#include <linux/rhashtable.h>
+
+#include "prestera.h"
+#include "prestera_hw.h"
+#include "prestera_acl.h"
+#include "prestera_span.h"
+
+struct prestera_acl {
+ struct prestera_switch *sw;
+ struct list_head rules;
+};
+
+struct prestera_acl_ruleset {
+ struct rhashtable rule_ht;
+ struct prestera_switch *sw;
+ u16 id;
+};
+
+struct prestera_acl_rule {
+ struct rhash_head ht_node;
+ struct list_head list;
+ struct list_head match_list;
+ struct list_head action_list;
+ struct prestera_flow_block *block;
+ unsigned long cookie;
+ u32 priority;
+ u8 n_actions;
+ u8 n_matches;
+ u32 id;
+};
+
+static const struct rhashtable_params prestera_acl_rule_ht_params = {
+ .key_len = sizeof(unsigned long),
+ .key_offset = offsetof(struct prestera_acl_rule, cookie),
+ .head_offset = offsetof(struct prestera_acl_rule, ht_node),
+ .automatic_shrinking = true,
+};
+
+static struct prestera_acl_ruleset *
+prestera_acl_ruleset_create(struct prestera_switch *sw)
+{
+ struct prestera_acl_ruleset *ruleset;
+ int err;
+
+ ruleset = kzalloc(sizeof(*ruleset), GFP_KERNEL);
+ if (!ruleset)
+ return ERR_PTR(-ENOMEM);
+
+ err = rhashtable_init(&ruleset->rule_ht, &prestera_acl_rule_ht_params);
+ if (err)
+ goto err_rhashtable_init;
+
+ err = prestera_hw_acl_ruleset_create(sw, &ruleset->id);
+ if (err)
+ goto err_ruleset_create;
+
+ ruleset->sw = sw;
+
+ return ruleset;
+
+err_ruleset_create:
+ rhashtable_destroy(&ruleset->rule_ht);
+err_rhashtable_init:
+ kfree(ruleset);
+ return ERR_PTR(err);
+}
+
+static void prestera_acl_ruleset_destroy(struct prestera_acl_ruleset *ruleset)
+{
+ prestera_hw_acl_ruleset_del(ruleset->sw, ruleset->id);
+ rhashtable_destroy(&ruleset->rule_ht);
+ kfree(ruleset);
+}
+
+struct prestera_flow_block *
+prestera_acl_block_create(struct prestera_switch *sw, struct net *net)
+{
+ struct prestera_flow_block *block;
+
+ block = kzalloc(sizeof(*block), GFP_KERNEL);
+ if (!block)
+ return NULL;
+ INIT_LIST_HEAD(&block->binding_list);
+ block->net = net;
+ block->sw = sw;
+
+ block->ruleset = prestera_acl_ruleset_create(sw);
+ if (IS_ERR(block->ruleset)) {
+ kfree(block);
+ return NULL;
+ }
+
+ return block;
+}
+
+void prestera_acl_block_destroy(struct prestera_flow_block *block)
+{
+ prestera_acl_ruleset_destroy(block->ruleset);
+ WARN_ON(!list_empty(&block->binding_list));
+ kfree(block);
+}
+
+static struct prestera_flow_block_binding *
+prestera_acl_block_lookup(struct prestera_flow_block *block,
+ struct prestera_port *port)
+{
+ struct prestera_flow_block_binding *binding;
+
+ list_for_each_entry(binding, &block->binding_list, list)
+ if (binding->port == port)
+ return binding;
+
+ return NULL;
+}
+
+int prestera_acl_block_bind(struct prestera_flow_block *block,
+ struct prestera_port *port)
+{
+ struct prestera_flow_block_binding *binding;
+ int err;
+
+ if (WARN_ON(prestera_acl_block_lookup(block, port)))
+ return -EEXIST;
+
+ binding = kzalloc(sizeof(*binding), GFP_KERNEL);
+ if (!binding)
+ return -ENOMEM;
+ binding->span_id = PRESTERA_SPAN_INVALID_ID;
+ binding->port = port;
+
+ err = prestera_hw_acl_port_bind(port, block->ruleset->id);
+ if (err)
+ goto err_rules_bind;
+
+ list_add(&binding->list, &block->binding_list);
+ return 0;
+
+err_rules_bind:
+ kfree(binding);
+ return err;
+}
+
+int prestera_acl_block_unbind(struct prestera_flow_block *block,
+ struct prestera_port *port)
+{
+ struct prestera_flow_block_binding *binding;
+
+ binding = prestera_acl_block_lookup(block, port);
+ if (!binding)
+ return -ENOENT;
+
+ list_del(&binding->list);
+
+ prestera_hw_acl_port_unbind(port, block->ruleset->id);
+
+ kfree(binding);
+ return 0;
+}
+
+struct prestera_acl_ruleset *
+prestera_acl_block_ruleset_get(struct prestera_flow_block *block)
+{
+ return block->ruleset;
+}
+
+u16 prestera_acl_rule_ruleset_id_get(const struct prestera_acl_rule *rule)
+{
+ return rule->block->ruleset->id;
+}
+
+struct net *prestera_acl_block_net(struct prestera_flow_block *block)
+{
+ return block->net;
+}
+
+struct prestera_switch *prestera_acl_block_sw(struct prestera_flow_block *block)
+{
+ return block->sw;
+}
+
+struct prestera_acl_rule *
+prestera_acl_rule_lookup(struct prestera_acl_ruleset *ruleset,
+ unsigned long cookie)
+{
+ return rhashtable_lookup_fast(&ruleset->rule_ht, &cookie,
+ prestera_acl_rule_ht_params);
+}
+
+struct prestera_acl_rule *
+prestera_acl_rule_create(struct prestera_flow_block *block,
+ unsigned long cookie)
+{
+ struct prestera_acl_rule *rule;
+
+ rule = kzalloc(sizeof(*rule), GFP_KERNEL);
+ if (!rule)
+ return ERR_PTR(-ENOMEM);
+
+ INIT_LIST_HEAD(&rule->match_list);
+ INIT_LIST_HEAD(&rule->action_list);
+ rule->cookie = cookie;
+ rule->block = block;
+
+ return rule;
+}
+
+struct list_head *
+prestera_acl_rule_match_list_get(struct prestera_acl_rule *rule)
+{
+ return &rule->match_list;
+}
+
+struct list_head *
+prestera_acl_rule_action_list_get(struct prestera_acl_rule *rule)
+{
+ return &rule->action_list;
+}
+
+int prestera_acl_rule_action_add(struct prestera_acl_rule *rule,
+ struct prestera_acl_rule_action_entry *entry)
+{
+ struct prestera_acl_rule_action_entry *a_entry;
+
+ a_entry = kmalloc(sizeof(*a_entry), GFP_KERNEL);
+ if (!a_entry)
+ return -ENOMEM;
+
+ memcpy(a_entry, entry, sizeof(*entry));
+ list_add(&a_entry->list, &rule->action_list);
+
+ rule->n_actions++;
+ return 0;
+}
+
+u8 prestera_acl_rule_action_len(struct prestera_acl_rule *rule)
+{
+ return rule->n_actions;
+}
+
+u32 prestera_acl_rule_priority_get(struct prestera_acl_rule *rule)
+{
+ return rule->priority;
+}
+
+void prestera_acl_rule_priority_set(struct prestera_acl_rule *rule,
+ u32 priority)
+{
+ rule->priority = priority;
+}
+
+int prestera_acl_rule_match_add(struct prestera_acl_rule *rule,
+ struct prestera_acl_rule_match_entry *entry)
+{
+ struct prestera_acl_rule_match_entry *m_entry;
+
+ m_entry = kmalloc(sizeof(*m_entry), GFP_KERNEL);
+ if (!m_entry)
+ return -ENOMEM;
+
+ memcpy(m_entry, entry, sizeof(*entry));
+ list_add(&m_entry->list, &rule->match_list);
+
+ rule->n_matches++;
+ return 0;
+}
+
+u8 prestera_acl_rule_match_len(struct prestera_acl_rule *rule)
+{
+ return rule->n_matches;
+}
+
+void prestera_acl_rule_destroy(struct prestera_acl_rule *rule)
+{
+ struct prestera_acl_rule_action_entry *a_entry;
+ struct prestera_acl_rule_match_entry *m_entry;
+ struct list_head *pos, *n;
+
+ list_for_each_safe(pos, n, &rule->match_list) {
+ m_entry = list_entry(pos, typeof(*m_entry), list);
+ list_del(pos);
+ kfree(m_entry);
+ }
+
+ list_for_each_safe(pos, n, &rule->action_list) {
+ a_entry = list_entry(pos, typeof(*a_entry), list);
+ list_del(pos);
+ kfree(a_entry);
+ }
+
+ kfree(rule);
+}
+
+int prestera_acl_rule_add(struct prestera_switch *sw,
+ struct prestera_acl_rule *rule)
+{
+ u32 rule_id;
+ int err;
+
+ /* try to add rule to hash table first */
+ err = rhashtable_insert_fast(&rule->block->ruleset->rule_ht,
+ &rule->ht_node,
+ prestera_acl_rule_ht_params);
+ if (err)
+ return err;
+
+ /* add rule to hw */
+ err = prestera_hw_acl_rule_add(sw, rule, &rule_id);
+ if (err)
+ goto err_rule_add;
+
+ rule->id = rule_id;
+
+ list_add_tail(&rule->list, &sw->acl->rules);
+
+ return 0;
+
+err_rule_add:
+ rhashtable_remove_fast(&rule->block->ruleset->rule_ht, &rule->ht_node,
+ prestera_acl_rule_ht_params);
+ return err;
+}
+
+void prestera_acl_rule_del(struct prestera_switch *sw,
+ struct prestera_acl_rule *rule)
+{
+ rhashtable_remove_fast(&rule->block->ruleset->rule_ht, &rule->ht_node,
+ prestera_acl_rule_ht_params);
+ list_del(&rule->list);
+ prestera_hw_acl_rule_del(sw, rule->id);
+}
+
+int prestera_acl_rule_get_stats(struct prestera_switch *sw,
+ struct prestera_acl_rule *rule,
+ u64 *packets, u64 *bytes, u64 *last_use)
+{
+ u64 current_packets;
+ u64 current_bytes;
+ int err;
+
+ err = prestera_hw_acl_rule_stats_get(sw, rule->id, ¤t_packets,
+ ¤t_bytes);
+ if (err)
+ return err;
+
+ *packets = current_packets;
+ *bytes = current_bytes;
+ *last_use = jiffies;
+
+ return 0;
+}
+
+int prestera_acl_init(struct prestera_switch *sw)
+{
+ struct prestera_acl *acl;
+
+ acl = kzalloc(sizeof(*acl), GFP_KERNEL);
+ if (!acl)
+ return -ENOMEM;
+
+ INIT_LIST_HEAD(&acl->rules);
+ sw->acl = acl;
+ acl->sw = sw;
+
+ return 0;
+}
+
+void prestera_acl_fini(struct prestera_switch *sw)
+{
+ struct prestera_acl *acl = sw->acl;
+
+ WARN_ON(!list_empty(&acl->rules));
+ kfree(acl);
+}
--- /dev/null
+/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 */
+/* Copyright (c) 2020 Marvell International Ltd. All rights reserved. */
+
+#ifndef _PRESTERA_ACL_H_
+#define _PRESTERA_ACL_H_
+
+enum prestera_acl_rule_match_entry_type {
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ETH_TYPE = 1,
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ETH_DMAC,
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ETH_SMAC,
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_IP_PROTO,
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_PORT,
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_IP_SRC,
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_IP_DST,
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_L4_PORT_SRC,
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_L4_PORT_DST,
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_L4_PORT_RANGE_SRC,
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_L4_PORT_RANGE_DST,
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_VLAN_ID,
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_VLAN_TPID,
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ICMP_TYPE,
+ PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ICMP_CODE
+};
+
+enum prestera_acl_rule_action {
+ PRESTERA_ACL_RULE_ACTION_ACCEPT,
+ PRESTERA_ACL_RULE_ACTION_DROP,
+ PRESTERA_ACL_RULE_ACTION_TRAP
+};
+
+struct prestera_switch;
+struct prestera_port;
+struct prestera_acl_rule;
+struct prestera_acl_ruleset;
+
+struct prestera_flow_block_binding {
+ struct list_head list;
+ struct prestera_port *port;
+ int span_id;
+};
+
+struct prestera_flow_block {
+ struct list_head binding_list;
+ struct prestera_switch *sw;
+ struct net *net;
+ struct prestera_acl_ruleset *ruleset;
+ struct flow_block_cb *block_cb;
+};
+
+struct prestera_acl_rule_action_entry {
+ struct list_head list;
+ enum prestera_acl_rule_action id;
+};
+
+struct prestera_acl_rule_match_entry {
+ struct list_head list;
+ enum prestera_acl_rule_match_entry_type type;
+ union {
+ struct {
+ u8 key;
+ u8 mask;
+ } u8;
+ struct {
+ u16 key;
+ u16 mask;
+ } u16;
+ struct {
+ u32 key;
+ u32 mask;
+ } u32;
+ struct {
+ u64 key;
+ u64 mask;
+ } u64;
+ struct {
+ u8 key[ETH_ALEN];
+ u8 mask[ETH_ALEN];
+ } mac;
+ } keymask;
+};
+
+int prestera_acl_init(struct prestera_switch *sw);
+void prestera_acl_fini(struct prestera_switch *sw);
+struct prestera_flow_block *
+prestera_acl_block_create(struct prestera_switch *sw, struct net *net);
+void prestera_acl_block_destroy(struct prestera_flow_block *block);
+struct net *prestera_acl_block_net(struct prestera_flow_block *block);
+struct prestera_switch *prestera_acl_block_sw(struct prestera_flow_block *block);
+int prestera_acl_block_bind(struct prestera_flow_block *block,
+ struct prestera_port *port);
+int prestera_acl_block_unbind(struct prestera_flow_block *block,
+ struct prestera_port *port);
+struct prestera_acl_ruleset *
+prestera_acl_block_ruleset_get(struct prestera_flow_block *block);
+struct prestera_acl_rule *
+prestera_acl_rule_create(struct prestera_flow_block *block,
+ unsigned long cookie);
+u32 prestera_acl_rule_priority_get(struct prestera_acl_rule *rule);
+void prestera_acl_rule_priority_set(struct prestera_acl_rule *rule,
+ u32 priority);
+u16 prestera_acl_rule_ruleset_id_get(const struct prestera_acl_rule *rule);
+struct list_head *
+prestera_acl_rule_action_list_get(struct prestera_acl_rule *rule);
+u8 prestera_acl_rule_action_len(struct prestera_acl_rule *rule);
+u8 prestera_acl_rule_match_len(struct prestera_acl_rule *rule);
+int prestera_acl_rule_action_add(struct prestera_acl_rule *rule,
+ struct prestera_acl_rule_action_entry *entry);
+struct list_head *
+prestera_acl_rule_match_list_get(struct prestera_acl_rule *rule);
+int prestera_acl_rule_match_add(struct prestera_acl_rule *rule,
+ struct prestera_acl_rule_match_entry *entry);
+void prestera_acl_rule_destroy(struct prestera_acl_rule *rule);
+struct prestera_acl_rule *
+prestera_acl_rule_lookup(struct prestera_acl_ruleset *ruleset,
+ unsigned long cookie);
+int prestera_acl_rule_add(struct prestera_switch *sw,
+ struct prestera_acl_rule *rule);
+void prestera_acl_rule_del(struct prestera_switch *sw,
+ struct prestera_acl_rule *rule);
+int prestera_acl_rule_get_stats(struct prestera_switch *sw,
+ struct prestera_acl_rule *rule,
+ u64 *packets, u64 *bytes, u64 *last_use);
+
+#endif /* _PRESTERA_ACL_H_ */
#include <net/devlink.h>
#include "prestera_devlink.h"
+#include "prestera_hw.h"
+
+/* All driver-specific traps must be documented in
+ * Documentation/networking/devlink/prestera.rst
+ */
+enum {
+ DEVLINK_PRESTERA_TRAP_ID_BASE = DEVLINK_TRAP_GENERIC_ID_MAX,
+ DEVLINK_PRESTERA_TRAP_ID_ARP_BC,
+ DEVLINK_PRESTERA_TRAP_ID_IS_IS,
+ DEVLINK_PRESTERA_TRAP_ID_OSPF,
+ DEVLINK_PRESTERA_TRAP_ID_IP_BC_MAC,
+ DEVLINK_PRESTERA_TRAP_ID_ROUTER_MC,
+ DEVLINK_PRESTERA_TRAP_ID_VRRP,
+ DEVLINK_PRESTERA_TRAP_ID_DHCP,
+ DEVLINK_PRESTERA_TRAP_ID_MAC_TO_ME,
+ DEVLINK_PRESTERA_TRAP_ID_IPV4_OPTIONS,
+ DEVLINK_PRESTERA_TRAP_ID_IP_DEFAULT_ROUTE,
+ DEVLINK_PRESTERA_TRAP_ID_IP_TO_ME,
+ DEVLINK_PRESTERA_TRAP_ID_IPV4_ICMP_REDIRECT,
+ DEVLINK_PRESTERA_TRAP_ID_ACL_CODE_0,
+ DEVLINK_PRESTERA_TRAP_ID_ACL_CODE_1,
+ DEVLINK_PRESTERA_TRAP_ID_ACL_CODE_2,
+ DEVLINK_PRESTERA_TRAP_ID_ACL_CODE_3,
+ DEVLINK_PRESTERA_TRAP_ID_ACL_CODE_4,
+ DEVLINK_PRESTERA_TRAP_ID_ACL_CODE_5,
+ DEVLINK_PRESTERA_TRAP_ID_ACL_CODE_6,
+ DEVLINK_PRESTERA_TRAP_ID_ACL_CODE_7,
+ DEVLINK_PRESTERA_TRAP_ID_BGP,
+ DEVLINK_PRESTERA_TRAP_ID_SSH,
+ DEVLINK_PRESTERA_TRAP_ID_TELNET,
+ DEVLINK_PRESTERA_TRAP_ID_ICMP,
+ DEVLINK_PRESTERA_TRAP_ID_MET_RED,
+ DEVLINK_PRESTERA_TRAP_ID_IP_SIP_IS_ZERO,
+ DEVLINK_PRESTERA_TRAP_ID_IP_UC_DIP_DA_MISMATCH,
+ DEVLINK_PRESTERA_TRAP_ID_ILLEGAL_IPV4_HDR,
+ DEVLINK_PRESTERA_TRAP_ID_ILLEGAL_IP_ADDR,
+ DEVLINK_PRESTERA_TRAP_ID_INVALID_SA,
+ DEVLINK_PRESTERA_TRAP_ID_LOCAL_PORT,
+ DEVLINK_PRESTERA_TRAP_ID_PORT_NO_VLAN,
+ DEVLINK_PRESTERA_TRAP_ID_RXDMA_DROP,
+};
+
+#define DEVLINK_PRESTERA_TRAP_NAME_ARP_BC \
+ "arp_bc"
+#define DEVLINK_PRESTERA_TRAP_NAME_IS_IS \
+ "is_is"
+#define DEVLINK_PRESTERA_TRAP_NAME_OSPF \
+ "ospf"
+#define DEVLINK_PRESTERA_TRAP_NAME_IP_BC_MAC \
+ "ip_bc_mac"
+#define DEVLINK_PRESTERA_TRAP_NAME_ROUTER_MC \
+ "router_mc"
+#define DEVLINK_PRESTERA_TRAP_NAME_VRRP \
+ "vrrp"
+#define DEVLINK_PRESTERA_TRAP_NAME_DHCP \
+ "dhcp"
+#define DEVLINK_PRESTERA_TRAP_NAME_MAC_TO_ME \
+ "mac_to_me"
+#define DEVLINK_PRESTERA_TRAP_NAME_IPV4_OPTIONS \
+ "ipv4_options"
+#define DEVLINK_PRESTERA_TRAP_NAME_IP_DEFAULT_ROUTE \
+ "ip_default_route"
+#define DEVLINK_PRESTERA_TRAP_NAME_IP_TO_ME \
+ "ip_to_me"
+#define DEVLINK_PRESTERA_TRAP_NAME_IPV4_ICMP_REDIRECT \
+ "ipv4_icmp_redirect"
+#define DEVLINK_PRESTERA_TRAP_NAME_ACL_CODE_0 \
+ "acl_code_0"
+#define DEVLINK_PRESTERA_TRAP_NAME_ACL_CODE_1 \
+ "acl_code_1"
+#define DEVLINK_PRESTERA_TRAP_NAME_ACL_CODE_2 \
+ "acl_code_2"
+#define DEVLINK_PRESTERA_TRAP_NAME_ACL_CODE_3 \
+ "acl_code_3"
+#define DEVLINK_PRESTERA_TRAP_NAME_ACL_CODE_4 \
+ "acl_code_4"
+#define DEVLINK_PRESTERA_TRAP_NAME_ACL_CODE_5 \
+ "acl_code_5"
+#define DEVLINK_PRESTERA_TRAP_NAME_ACL_CODE_6 \
+ "acl_code_6"
+#define DEVLINK_PRESTERA_TRAP_NAME_ACL_CODE_7 \
+ "acl_code_7"
+#define DEVLINK_PRESTERA_TRAP_NAME_BGP \
+ "bgp"
+#define DEVLINK_PRESTERA_TRAP_NAME_SSH \
+ "ssh"
+#define DEVLINK_PRESTERA_TRAP_NAME_TELNET \
+ "telnet"
+#define DEVLINK_PRESTERA_TRAP_NAME_ICMP \
+ "icmp"
+#define DEVLINK_PRESTERA_TRAP_NAME_RXDMA_DROP \
+ "rxdma_drop"
+#define DEVLINK_PRESTERA_TRAP_NAME_PORT_NO_VLAN \
+ "port_no_vlan"
+#define DEVLINK_PRESTERA_TRAP_NAME_LOCAL_PORT \
+ "local_port"
+#define DEVLINK_PRESTERA_TRAP_NAME_INVALID_SA \
+ "invalid_sa"
+#define DEVLINK_PRESTERA_TRAP_NAME_ILLEGAL_IP_ADDR \
+ "illegal_ip_addr"
+#define DEVLINK_PRESTERA_TRAP_NAME_ILLEGAL_IPV4_HDR \
+ "illegal_ipv4_hdr"
+#define DEVLINK_PRESTERA_TRAP_NAME_IP_UC_DIP_DA_MISMATCH \
+ "ip_uc_dip_da_mismatch"
+#define DEVLINK_PRESTERA_TRAP_NAME_IP_SIP_IS_ZERO \
+ "ip_sip_is_zero"
+#define DEVLINK_PRESTERA_TRAP_NAME_MET_RED \
+ "met_red"
+
+struct prestera_trap {
+ struct devlink_trap trap;
+ u8 cpu_code;
+};
+
+struct prestera_trap_item {
+ enum devlink_trap_action action;
+ void *trap_ctx;
+};
+
+struct prestera_trap_data {
+ struct prestera_switch *sw;
+ struct prestera_trap_item *trap_items_arr;
+ u32 traps_count;
+};
+
+#define PRESTERA_TRAP_METADATA DEVLINK_TRAP_METADATA_TYPE_F_IN_PORT
+
+#define PRESTERA_TRAP_CONTROL(_id, _group_id, _action) \
+ DEVLINK_TRAP_GENERIC(CONTROL, _action, _id, \
+ DEVLINK_TRAP_GROUP_GENERIC_ID_##_group_id, \
+ PRESTERA_TRAP_METADATA)
+
+#define PRESTERA_TRAP_DRIVER_CONTROL(_id, _group_id) \
+ DEVLINK_TRAP_DRIVER(CONTROL, TRAP, DEVLINK_PRESTERA_TRAP_ID_##_id, \
+ DEVLINK_PRESTERA_TRAP_NAME_##_id, \
+ DEVLINK_TRAP_GROUP_GENERIC_ID_##_group_id, \
+ PRESTERA_TRAP_METADATA)
+
+#define PRESTERA_TRAP_EXCEPTION(_id, _group_id) \
+ DEVLINK_TRAP_GENERIC(EXCEPTION, TRAP, _id, \
+ DEVLINK_TRAP_GROUP_GENERIC_ID_##_group_id, \
+ PRESTERA_TRAP_METADATA)
+
+#define PRESTERA_TRAP_DRIVER_EXCEPTION(_id, _group_id) \
+ DEVLINK_TRAP_DRIVER(EXCEPTION, TRAP, DEVLINK_PRESTERA_TRAP_ID_##_id, \
+ DEVLINK_PRESTERA_TRAP_NAME_##_id, \
+ DEVLINK_TRAP_GROUP_GENERIC_ID_##_group_id, \
+ PRESTERA_TRAP_METADATA)
+
+#define PRESTERA_TRAP_DRIVER_DROP(_id, _group_id) \
+ DEVLINK_TRAP_DRIVER(DROP, DROP, DEVLINK_PRESTERA_TRAP_ID_##_id, \
+ DEVLINK_PRESTERA_TRAP_NAME_##_id, \
+ DEVLINK_TRAP_GROUP_GENERIC_ID_##_group_id, \
+ PRESTERA_TRAP_METADATA)
+
+static const struct devlink_trap_group prestera_trap_groups_arr[] = {
+ /* No policer is associated with following groups (policerid == 0)*/
+ DEVLINK_TRAP_GROUP_GENERIC(L2_DROPS, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(L3_DROPS, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(L3_EXCEPTIONS, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(NEIGH_DISCOVERY, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(ACL_TRAP, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(ACL_DROPS, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(ACL_SAMPLE, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(OSPF, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(STP, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(LACP, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(LLDP, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(VRRP, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(DHCP, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(BGP, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(LOCAL_DELIVERY, 0),
+ DEVLINK_TRAP_GROUP_GENERIC(BUFFER_DROPS, 0),
+};
+
+/* Initialize trap list, as well as associate CPU code with them. */
+static struct prestera_trap prestera_trap_items_arr[] = {
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(ARP_BC, NEIGH_DISCOVERY),
+ .cpu_code = 5,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(IS_IS, LOCAL_DELIVERY),
+ .cpu_code = 13,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(OSPF, OSPF),
+ .cpu_code = 16,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(IP_BC_MAC, LOCAL_DELIVERY),
+ .cpu_code = 19,
+ },
+ {
+ .trap = PRESTERA_TRAP_CONTROL(STP, STP, TRAP),
+ .cpu_code = 26,
+ },
+ {
+ .trap = PRESTERA_TRAP_CONTROL(LACP, LACP, TRAP),
+ .cpu_code = 27,
+ },
+ {
+ .trap = PRESTERA_TRAP_CONTROL(LLDP, LLDP, TRAP),
+ .cpu_code = 28,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(ROUTER_MC, LOCAL_DELIVERY),
+ .cpu_code = 29,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(VRRP, VRRP),
+ .cpu_code = 30,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(DHCP, DHCP),
+ .cpu_code = 33,
+ },
+ {
+ .trap = PRESTERA_TRAP_EXCEPTION(MTU_ERROR, L3_EXCEPTIONS),
+ .cpu_code = 63,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(MAC_TO_ME, LOCAL_DELIVERY),
+ .cpu_code = 65,
+ },
+ {
+ .trap = PRESTERA_TRAP_EXCEPTION(TTL_ERROR, L3_EXCEPTIONS),
+ .cpu_code = 133,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_EXCEPTION(IPV4_OPTIONS,
+ L3_EXCEPTIONS),
+ .cpu_code = 141,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(IP_DEFAULT_ROUTE,
+ LOCAL_DELIVERY),
+ .cpu_code = 160,
+ },
+ {
+ .trap = PRESTERA_TRAP_CONTROL(LOCAL_ROUTE, LOCAL_DELIVERY,
+ TRAP),
+ .cpu_code = 161,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_EXCEPTION(IPV4_ICMP_REDIRECT,
+ L3_EXCEPTIONS),
+ .cpu_code = 180,
+ },
+ {
+ .trap = PRESTERA_TRAP_CONTROL(ARP_RESPONSE, NEIGH_DISCOVERY,
+ TRAP),
+ .cpu_code = 188,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(ACL_CODE_0, ACL_TRAP),
+ .cpu_code = 192,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(ACL_CODE_1, ACL_TRAP),
+ .cpu_code = 193,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(ACL_CODE_2, ACL_TRAP),
+ .cpu_code = 194,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(ACL_CODE_3, ACL_TRAP),
+ .cpu_code = 195,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(ACL_CODE_4, ACL_TRAP),
+ .cpu_code = 196,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(ACL_CODE_5, ACL_TRAP),
+ .cpu_code = 197,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(ACL_CODE_6, ACL_TRAP),
+ .cpu_code = 198,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(ACL_CODE_7, ACL_TRAP),
+ .cpu_code = 199,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(BGP, BGP),
+ .cpu_code = 206,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(SSH, LOCAL_DELIVERY),
+ .cpu_code = 207,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(TELNET, LOCAL_DELIVERY),
+ .cpu_code = 208,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_CONTROL(ICMP, LOCAL_DELIVERY),
+ .cpu_code = 209,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_DROP(RXDMA_DROP, BUFFER_DROPS),
+ .cpu_code = 37,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_DROP(PORT_NO_VLAN, L2_DROPS),
+ .cpu_code = 39,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_DROP(LOCAL_PORT, L2_DROPS),
+ .cpu_code = 56,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_DROP(INVALID_SA, L2_DROPS),
+ .cpu_code = 60,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_DROP(ILLEGAL_IP_ADDR, L3_DROPS),
+ .cpu_code = 136,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_DROP(ILLEGAL_IPV4_HDR, L3_DROPS),
+ .cpu_code = 137,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_DROP(IP_UC_DIP_DA_MISMATCH,
+ L3_DROPS),
+ .cpu_code = 138,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_DROP(IP_SIP_IS_ZERO, L3_DROPS),
+ .cpu_code = 145,
+ },
+ {
+ .trap = PRESTERA_TRAP_DRIVER_DROP(MET_RED, BUFFER_DROPS),
+ .cpu_code = 185,
+ },
+};
+
+static void prestera_devlink_traps_fini(struct prestera_switch *sw);
+
+static int prestera_drop_counter_get(struct devlink *devlink,
+ const struct devlink_trap *trap,
+ u64 *p_drops);
static int prestera_dl_info_get(struct devlink *dl,
struct devlink_info_req *req,
buf);
}
+static int prestera_trap_init(struct devlink *devlink,
+ const struct devlink_trap *trap, void *trap_ctx);
+
+static int prestera_trap_action_set(struct devlink *devlink,
+ const struct devlink_trap *trap,
+ enum devlink_trap_action action,
+ struct netlink_ext_ack *extack);
+
+static int prestera_devlink_traps_register(struct prestera_switch *sw);
+
static const struct devlink_ops prestera_dl_ops = {
.info_get = prestera_dl_info_get,
+ .trap_init = prestera_trap_init,
+ .trap_action_set = prestera_trap_action_set,
+ .trap_drop_counter_get = prestera_drop_counter_get,
};
struct prestera_switch *prestera_devlink_alloc(void)
int err;
err = devlink_register(dl, sw->dev->dev);
- if (err)
+ if (err) {
dev_err(prestera_dev(sw), "devlink_register failed: %d\n", err);
+ return err;
+ }
- return err;
+ err = prestera_devlink_traps_register(sw);
+ if (err) {
+ devlink_unregister(dl);
+ dev_err(sw->dev->dev, "devlink_traps_register failed: %d\n",
+ err);
+ return err;
+ }
+
+ return 0;
}
void prestera_devlink_unregister(struct prestera_switch *sw)
{
+ struct prestera_trap_data *trap_data = sw->trap_data;
struct devlink *dl = priv_to_devlink(sw);
+ prestera_devlink_traps_fini(sw);
devlink_unregister(dl);
+
+ kfree(trap_data->trap_items_arr);
+ kfree(trap_data);
}
int prestera_devlink_port_register(struct prestera_port *port)
return &port->dl_port;
}
+
+static int prestera_devlink_traps_register(struct prestera_switch *sw)
+{
+ const u32 groups_count = ARRAY_SIZE(prestera_trap_groups_arr);
+ const u32 traps_count = ARRAY_SIZE(prestera_trap_items_arr);
+ struct devlink *devlink = priv_to_devlink(sw);
+ struct prestera_trap_data *trap_data;
+ struct prestera_trap *prestera_trap;
+ int err, i;
+
+ trap_data = kzalloc(sizeof(*trap_data), GFP_KERNEL);
+ if (!trap_data)
+ return -ENOMEM;
+
+ trap_data->trap_items_arr = kcalloc(traps_count,
+ sizeof(struct prestera_trap_item),
+ GFP_KERNEL);
+ if (!trap_data->trap_items_arr) {
+ err = -ENOMEM;
+ goto err_trap_items_alloc;
+ }
+
+ trap_data->sw = sw;
+ trap_data->traps_count = traps_count;
+ sw->trap_data = trap_data;
+
+ err = devlink_trap_groups_register(devlink, prestera_trap_groups_arr,
+ groups_count);
+ if (err)
+ goto err_groups_register;
+
+ for (i = 0; i < traps_count; i++) {
+ prestera_trap = &prestera_trap_items_arr[i];
+ err = devlink_traps_register(devlink, &prestera_trap->trap, 1,
+ sw);
+ if (err)
+ goto err_trap_register;
+ }
+
+ return 0;
+
+err_trap_register:
+ for (i--; i >= 0; i--) {
+ prestera_trap = &prestera_trap_items_arr[i];
+ devlink_traps_unregister(devlink, &prestera_trap->trap, 1);
+ }
+err_groups_register:
+ kfree(trap_data->trap_items_arr);
+err_trap_items_alloc:
+ kfree(trap_data);
+ return err;
+}
+
+static struct prestera_trap_item *
+prestera_get_trap_item_by_cpu_code(struct prestera_switch *sw, u8 cpu_code)
+{
+ struct prestera_trap_data *trap_data = sw->trap_data;
+ struct prestera_trap *prestera_trap;
+ int i;
+
+ for (i = 0; i < trap_data->traps_count; i++) {
+ prestera_trap = &prestera_trap_items_arr[i];
+ if (cpu_code == prestera_trap->cpu_code)
+ return &trap_data->trap_items_arr[i];
+ }
+
+ return NULL;
+}
+
+void prestera_devlink_trap_report(struct prestera_port *port,
+ struct sk_buff *skb, u8 cpu_code)
+{
+ struct prestera_trap_item *trap_item;
+ struct devlink *devlink;
+
+ devlink = port->dl_port.devlink;
+
+ trap_item = prestera_get_trap_item_by_cpu_code(port->sw, cpu_code);
+ if (unlikely(!trap_item))
+ return;
+
+ devlink_trap_report(devlink, skb, trap_item->trap_ctx,
+ &port->dl_port, NULL);
+}
+
+static struct prestera_trap_item *
+prestera_devlink_trap_item_lookup(struct prestera_switch *sw, u16 trap_id)
+{
+ struct prestera_trap_data *trap_data = sw->trap_data;
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(prestera_trap_items_arr); i++) {
+ if (prestera_trap_items_arr[i].trap.id == trap_id)
+ return &trap_data->trap_items_arr[i];
+ }
+
+ return NULL;
+}
+
+static int prestera_trap_init(struct devlink *devlink,
+ const struct devlink_trap *trap, void *trap_ctx)
+{
+ struct prestera_switch *sw = devlink_priv(devlink);
+ struct prestera_trap_item *trap_item;
+
+ trap_item = prestera_devlink_trap_item_lookup(sw, trap->id);
+ if (WARN_ON(!trap_item))
+ return -EINVAL;
+
+ trap_item->trap_ctx = trap_ctx;
+ trap_item->action = trap->init_action;
+
+ return 0;
+}
+
+static int prestera_trap_action_set(struct devlink *devlink,
+ const struct devlink_trap *trap,
+ enum devlink_trap_action action,
+ struct netlink_ext_ack *extack)
+{
+ /* Currently, driver does not support trap action altering */
+ return -EOPNOTSUPP;
+}
+
+static int prestera_drop_counter_get(struct devlink *devlink,
+ const struct devlink_trap *trap,
+ u64 *p_drops)
+{
+ struct prestera_switch *sw = devlink_priv(devlink);
+ enum prestera_hw_cpu_code_cnt_t cpu_code_type =
+ PRESTERA_HW_CPU_CODE_CNT_TYPE_DROP;
+ struct prestera_trap *prestera_trap =
+ container_of(trap, struct prestera_trap, trap);
+
+ return prestera_hw_cpu_code_counters_get(sw, prestera_trap->cpu_code,
+ cpu_code_type, p_drops);
+}
+
+static void prestera_devlink_traps_fini(struct prestera_switch *sw)
+{
+ struct devlink *dl = priv_to_devlink(sw);
+ const struct devlink_trap *trap;
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(prestera_trap_items_arr); ++i) {
+ trap = &prestera_trap_items_arr[i].trap;
+ devlink_traps_unregister(dl, trap, 1);
+ }
+
+ devlink_trap_groups_unregister(dl, prestera_trap_groups_arr,
+ ARRAY_SIZE(prestera_trap_groups_arr));
+}
struct devlink_port *prestera_devlink_get_port(struct net_device *dev);
+void prestera_devlink_trap_report(struct prestera_port *port,
+ struct sk_buff *skb, u8 cpu_code);
+
#endif /* _PRESTERA_DEVLINK_H_ */
#define PRESTERA_DSA_W1_EXT_BIT BIT(31)
#define PRESTERA_DSA_W1_CFI_BIT BIT(30)
#define PRESTERA_DSA_W1_PORT_NUM GENMASK(11, 10)
+#define PRESTERA_DSA_W1_MASK_CPU_CODE GENMASK(7, 0)
#define PRESTERA_DSA_W2_EXT_BIT BIT(31)
#define PRESTERA_DSA_W2_PORT_NUM BIT(20)
(FIELD_GET(PRESTERA_DSA_W1_PORT_NUM, words[1]) << 5) |
(FIELD_GET(PRESTERA_DSA_W2_PORT_NUM, words[2]) << 7);
+ dsa->cpu_code = FIELD_GET(PRESTERA_DSA_W1_MASK_CPU_CODE, words[1]);
+
return 0;
}
struct prestera_dsa_vlan vlan;
u32 hw_dev_num;
u32 port_num;
+ u8 cpu_code;
};
int prestera_dsa_parse(struct prestera_dsa *dsa, const u8 *dsa_buf);
--- /dev/null
+// SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0
+/* Copyright (c) 2020 Marvell International Ltd. All rights reserved */
+
+#include <linux/kernel.h>
+#include <linux/list.h>
+
+#include "prestera.h"
+#include "prestera_acl.h"
+#include "prestera_flow.h"
+#include "prestera_span.h"
+#include "prestera_flower.h"
+
+static LIST_HEAD(prestera_block_cb_list);
+
+static int prestera_flow_block_mall_cb(struct prestera_flow_block *block,
+ struct tc_cls_matchall_offload *f)
+{
+ switch (f->command) {
+ case TC_CLSMATCHALL_REPLACE:
+ return prestera_span_replace(block, f);
+ case TC_CLSMATCHALL_DESTROY:
+ prestera_span_destroy(block);
+ return 0;
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+
+static int prestera_flow_block_flower_cb(struct prestera_flow_block *block,
+ struct flow_cls_offload *f)
+{
+ if (f->common.chain_index != 0)
+ return -EOPNOTSUPP;
+
+ switch (f->command) {
+ case FLOW_CLS_REPLACE:
+ return prestera_flower_replace(block, f);
+ case FLOW_CLS_DESTROY:
+ prestera_flower_destroy(block, f);
+ return 0;
+ case FLOW_CLS_STATS:
+ return prestera_flower_stats(block, f);
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+
+static int prestera_flow_block_cb(enum tc_setup_type type,
+ void *type_data, void *cb_priv)
+{
+ struct prestera_flow_block *block = cb_priv;
+
+ switch (type) {
+ case TC_SETUP_CLSFLOWER:
+ return prestera_flow_block_flower_cb(block, type_data);
+ case TC_SETUP_CLSMATCHALL:
+ return prestera_flow_block_mall_cb(block, type_data);
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+
+static void prestera_flow_block_release(void *cb_priv)
+{
+ struct prestera_flow_block *block = cb_priv;
+
+ prestera_acl_block_destroy(block);
+}
+
+static struct prestera_flow_block *
+prestera_flow_block_get(struct prestera_switch *sw,
+ struct flow_block_offload *f,
+ bool *register_block)
+{
+ struct prestera_flow_block *block;
+ struct flow_block_cb *block_cb;
+
+ block_cb = flow_block_cb_lookup(f->block,
+ prestera_flow_block_cb, sw);
+ if (!block_cb) {
+ block = prestera_acl_block_create(sw, f->net);
+ if (!block)
+ return ERR_PTR(-ENOMEM);
+
+ block_cb = flow_block_cb_alloc(prestera_flow_block_cb,
+ sw, block,
+ prestera_flow_block_release);
+ if (IS_ERR(block_cb)) {
+ prestera_acl_block_destroy(block);
+ return ERR_CAST(block_cb);
+ }
+
+ block->block_cb = block_cb;
+ *register_block = true;
+ } else {
+ block = flow_block_cb_priv(block_cb);
+ *register_block = false;
+ }
+
+ flow_block_cb_incref(block_cb);
+
+ return block;
+}
+
+static void prestera_flow_block_put(struct prestera_flow_block *block)
+{
+ struct flow_block_cb *block_cb = block->block_cb;
+
+ if (flow_block_cb_decref(block_cb))
+ return;
+
+ flow_block_cb_free(block_cb);
+ prestera_acl_block_destroy(block);
+}
+
+static int prestera_setup_flow_block_bind(struct prestera_port *port,
+ struct flow_block_offload *f)
+{
+ struct prestera_switch *sw = port->sw;
+ struct prestera_flow_block *block;
+ struct flow_block_cb *block_cb;
+ bool register_block;
+ int err;
+
+ block = prestera_flow_block_get(sw, f, ®ister_block);
+ if (IS_ERR(block))
+ return PTR_ERR(block);
+
+ block_cb = block->block_cb;
+
+ err = prestera_acl_block_bind(block, port);
+ if (err)
+ goto err_block_bind;
+
+ if (register_block) {
+ flow_block_cb_add(block_cb, f);
+ list_add_tail(&block_cb->driver_list, &prestera_block_cb_list);
+ }
+
+ port->flow_block = block;
+ return 0;
+
+err_block_bind:
+ prestera_flow_block_put(block);
+
+ return err;
+}
+
+static void prestera_setup_flow_block_unbind(struct prestera_port *port,
+ struct flow_block_offload *f)
+{
+ struct prestera_switch *sw = port->sw;
+ struct prestera_flow_block *block;
+ struct flow_block_cb *block_cb;
+ int err;
+
+ block_cb = flow_block_cb_lookup(f->block, prestera_flow_block_cb, sw);
+ if (!block_cb)
+ return;
+
+ block = flow_block_cb_priv(block_cb);
+
+ prestera_span_destroy(block);
+
+ err = prestera_acl_block_unbind(block, port);
+ if (err)
+ goto error;
+
+ if (!flow_block_cb_decref(block_cb)) {
+ flow_block_cb_remove(block_cb, f);
+ list_del(&block_cb->driver_list);
+ }
+error:
+ port->flow_block = NULL;
+}
+
+int prestera_flow_block_setup(struct prestera_port *port,
+ struct flow_block_offload *f)
+{
+ if (f->binder_type != FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS)
+ return -EOPNOTSUPP;
+
+ f->driver_block_list = &prestera_block_cb_list;
+
+ switch (f->command) {
+ case FLOW_BLOCK_BIND:
+ return prestera_setup_flow_block_bind(port, f);
+ case FLOW_BLOCK_UNBIND:
+ prestera_setup_flow_block_unbind(port, f);
+ return 0;
+ default:
+ return -EOPNOTSUPP;
+ }
+}
--- /dev/null
+/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 */
+/* Copyright (c) 2020 Marvell International Ltd. All rights reserved. */
+
+#ifndef _PRESTERA_FLOW_H_
+#define _PRESTERA_FLOW_H_
+
+#include <net/flow_offload.h>
+
+struct prestera_port;
+
+int prestera_flow_block_setup(struct prestera_port *port,
+ struct flow_block_offload *f);
+
+#endif /* _PRESTERA_FLOW_H_ */
--- /dev/null
+// SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0
+/* Copyright (c) 2020 Marvell International Ltd. All rights reserved */
+
+#include "prestera.h"
+#include "prestera_acl.h"
+#include "prestera_flower.h"
+
+static int prestera_flower_parse_actions(struct prestera_flow_block *block,
+ struct prestera_acl_rule *rule,
+ struct flow_action *flow_action,
+ struct netlink_ext_ack *extack)
+{
+ struct prestera_acl_rule_action_entry a_entry;
+ const struct flow_action_entry *act;
+ int err, i;
+
+ if (!flow_action_has_entries(flow_action))
+ return 0;
+
+ flow_action_for_each(i, act, flow_action) {
+ memset(&a_entry, 0, sizeof(a_entry));
+
+ switch (act->id) {
+ case FLOW_ACTION_ACCEPT:
+ a_entry.id = PRESTERA_ACL_RULE_ACTION_ACCEPT;
+ break;
+ case FLOW_ACTION_DROP:
+ a_entry.id = PRESTERA_ACL_RULE_ACTION_DROP;
+ break;
+ case FLOW_ACTION_TRAP:
+ a_entry.id = PRESTERA_ACL_RULE_ACTION_TRAP;
+ break;
+ default:
+ NL_SET_ERR_MSG_MOD(extack, "Unsupported action");
+ pr_err("Unsupported action\n");
+ return -EOPNOTSUPP;
+ }
+
+ err = prestera_acl_rule_action_add(rule, &a_entry);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+static int prestera_flower_parse_meta(struct prestera_acl_rule *rule,
+ struct flow_cls_offload *f,
+ struct prestera_flow_block *block)
+{
+ struct flow_rule *f_rule = flow_cls_offload_flow_rule(f);
+ struct prestera_acl_rule_match_entry m_entry = {0};
+ struct net_device *ingress_dev;
+ struct flow_match_meta match;
+ struct prestera_port *port;
+
+ flow_rule_match_meta(f_rule, &match);
+ if (match.mask->ingress_ifindex != 0xFFFFFFFF) {
+ NL_SET_ERR_MSG_MOD(f->common.extack,
+ "Unsupported ingress ifindex mask");
+ return -EINVAL;
+ }
+
+ ingress_dev = __dev_get_by_index(prestera_acl_block_net(block),
+ match.key->ingress_ifindex);
+ if (!ingress_dev) {
+ NL_SET_ERR_MSG_MOD(f->common.extack,
+ "Can't find specified ingress port to match on");
+ return -EINVAL;
+ }
+
+ if (!prestera_netdev_check(ingress_dev)) {
+ NL_SET_ERR_MSG_MOD(f->common.extack,
+ "Can't match on switchdev ingress port");
+ return -EINVAL;
+ }
+ port = netdev_priv(ingress_dev);
+
+ m_entry.type = PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_PORT;
+ m_entry.keymask.u64.key = port->hw_id | ((u64)port->dev_id << 32);
+ m_entry.keymask.u64.mask = ~(u64)0;
+
+ return prestera_acl_rule_match_add(rule, &m_entry);
+}
+
+static int prestera_flower_parse(struct prestera_flow_block *block,
+ struct prestera_acl_rule *rule,
+ struct flow_cls_offload *f)
+{
+ struct flow_rule *f_rule = flow_cls_offload_flow_rule(f);
+ struct flow_dissector *dissector = f_rule->match.dissector;
+ struct prestera_acl_rule_match_entry m_entry;
+ u16 n_proto_mask = 0;
+ u16 n_proto_key = 0;
+ u16 addr_type = 0;
+ u8 ip_proto = 0;
+ int err;
+
+ if (dissector->used_keys &
+ ~(BIT(FLOW_DISSECTOR_KEY_META) |
+ BIT(FLOW_DISSECTOR_KEY_CONTROL) |
+ BIT(FLOW_DISSECTOR_KEY_BASIC) |
+ BIT(FLOW_DISSECTOR_KEY_ETH_ADDRS) |
+ BIT(FLOW_DISSECTOR_KEY_IPV4_ADDRS) |
+ BIT(FLOW_DISSECTOR_KEY_IPV6_ADDRS) |
+ BIT(FLOW_DISSECTOR_KEY_ICMP) |
+ BIT(FLOW_DISSECTOR_KEY_PORTS) |
+ BIT(FLOW_DISSECTOR_KEY_VLAN))) {
+ NL_SET_ERR_MSG_MOD(f->common.extack, "Unsupported key");
+ return -EOPNOTSUPP;
+ }
+
+ prestera_acl_rule_priority_set(rule, f->common.prio);
+
+ if (flow_rule_match_key(f_rule, FLOW_DISSECTOR_KEY_META)) {
+ err = prestera_flower_parse_meta(rule, f, block);
+ if (err)
+ return err;
+ }
+
+ if (flow_rule_match_key(f_rule, FLOW_DISSECTOR_KEY_CONTROL)) {
+ struct flow_match_control match;
+
+ flow_rule_match_control(f_rule, &match);
+ addr_type = match.key->addr_type;
+ }
+
+ if (flow_rule_match_key(f_rule, FLOW_DISSECTOR_KEY_BASIC)) {
+ struct flow_match_basic match;
+
+ flow_rule_match_basic(f_rule, &match);
+ n_proto_key = ntohs(match.key->n_proto);
+ n_proto_mask = ntohs(match.mask->n_proto);
+
+ if (n_proto_key == ETH_P_ALL) {
+ n_proto_key = 0;
+ n_proto_mask = 0;
+ }
+
+ /* add eth type key,mask */
+ memset(&m_entry, 0, sizeof(m_entry));
+ m_entry.type = PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ETH_TYPE;
+ m_entry.keymask.u16.key = n_proto_key;
+ m_entry.keymask.u16.mask = n_proto_mask;
+ err = prestera_acl_rule_match_add(rule, &m_entry);
+ if (err)
+ return err;
+
+ /* add ip proto key,mask */
+ memset(&m_entry, 0, sizeof(m_entry));
+ m_entry.type = PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_IP_PROTO;
+ m_entry.keymask.u8.key = match.key->ip_proto;
+ m_entry.keymask.u8.mask = match.mask->ip_proto;
+ err = prestera_acl_rule_match_add(rule, &m_entry);
+ if (err)
+ return err;
+
+ ip_proto = match.key->ip_proto;
+ }
+
+ if (flow_rule_match_key(f_rule, FLOW_DISSECTOR_KEY_ETH_ADDRS)) {
+ struct flow_match_eth_addrs match;
+
+ flow_rule_match_eth_addrs(f_rule, &match);
+
+ /* add ethernet dst key,mask */
+ memset(&m_entry, 0, sizeof(m_entry));
+ m_entry.type = PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ETH_DMAC;
+ memcpy(&m_entry.keymask.mac.key,
+ &match.key->dst, sizeof(match.key->dst));
+ memcpy(&m_entry.keymask.mac.mask,
+ &match.mask->dst, sizeof(match.mask->dst));
+ err = prestera_acl_rule_match_add(rule, &m_entry);
+ if (err)
+ return err;
+
+ /* add ethernet src key,mask */
+ memset(&m_entry, 0, sizeof(m_entry));
+ m_entry.type = PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ETH_SMAC;
+ memcpy(&m_entry.keymask.mac.key,
+ &match.key->src, sizeof(match.key->src));
+ memcpy(&m_entry.keymask.mac.mask,
+ &match.mask->src, sizeof(match.mask->src));
+ err = prestera_acl_rule_match_add(rule, &m_entry);
+ if (err)
+ return err;
+ }
+
+ if (addr_type == FLOW_DISSECTOR_KEY_IPV4_ADDRS) {
+ struct flow_match_ipv4_addrs match;
+
+ flow_rule_match_ipv4_addrs(f_rule, &match);
+
+ memset(&m_entry, 0, sizeof(m_entry));
+ m_entry.type = PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_IP_SRC;
+ memcpy(&m_entry.keymask.u32.key,
+ &match.key->src, sizeof(match.key->src));
+ memcpy(&m_entry.keymask.u32.mask,
+ &match.mask->src, sizeof(match.mask->src));
+ err = prestera_acl_rule_match_add(rule, &m_entry);
+ if (err)
+ return err;
+
+ memset(&m_entry, 0, sizeof(m_entry));
+ m_entry.type = PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_IP_DST;
+ memcpy(&m_entry.keymask.u32.key,
+ &match.key->dst, sizeof(match.key->dst));
+ memcpy(&m_entry.keymask.u32.mask,
+ &match.mask->dst, sizeof(match.mask->dst));
+ err = prestera_acl_rule_match_add(rule, &m_entry);
+ if (err)
+ return err;
+ }
+
+ if (flow_rule_match_key(f_rule, FLOW_DISSECTOR_KEY_PORTS)) {
+ struct flow_match_ports match;
+
+ if (ip_proto != IPPROTO_TCP && ip_proto != IPPROTO_UDP) {
+ NL_SET_ERR_MSG_MOD
+ (f->common.extack,
+ "Only UDP and TCP keys are supported");
+ return -EINVAL;
+ }
+
+ flow_rule_match_ports(f_rule, &match);
+
+ memset(&m_entry, 0, sizeof(m_entry));
+ m_entry.type = PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_L4_PORT_SRC;
+ m_entry.keymask.u16.key = ntohs(match.key->src);
+ m_entry.keymask.u16.mask = ntohs(match.mask->src);
+ err = prestera_acl_rule_match_add(rule, &m_entry);
+ if (err)
+ return err;
+
+ memset(&m_entry, 0, sizeof(m_entry));
+ m_entry.type = PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_L4_PORT_DST;
+ m_entry.keymask.u16.key = ntohs(match.key->dst);
+ m_entry.keymask.u16.mask = ntohs(match.mask->dst);
+ err = prestera_acl_rule_match_add(rule, &m_entry);
+ if (err)
+ return err;
+ }
+
+ if (flow_rule_match_key(f_rule, FLOW_DISSECTOR_KEY_VLAN)) {
+ struct flow_match_vlan match;
+
+ flow_rule_match_vlan(f_rule, &match);
+
+ if (match.mask->vlan_id != 0) {
+ memset(&m_entry, 0, sizeof(m_entry));
+ m_entry.type = PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_VLAN_ID;
+ m_entry.keymask.u16.key = match.key->vlan_id;
+ m_entry.keymask.u16.mask = match.mask->vlan_id;
+ err = prestera_acl_rule_match_add(rule, &m_entry);
+ if (err)
+ return err;
+ }
+
+ memset(&m_entry, 0, sizeof(m_entry));
+ m_entry.type = PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_VLAN_TPID;
+ m_entry.keymask.u16.key = ntohs(match.key->vlan_tpid);
+ m_entry.keymask.u16.mask = ntohs(match.mask->vlan_tpid);
+ err = prestera_acl_rule_match_add(rule, &m_entry);
+ if (err)
+ return err;
+ }
+
+ if (flow_rule_match_key(f_rule, FLOW_DISSECTOR_KEY_ICMP)) {
+ struct flow_match_icmp match;
+
+ flow_rule_match_icmp(f_rule, &match);
+
+ memset(&m_entry, 0, sizeof(m_entry));
+ m_entry.type = PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ICMP_TYPE;
+ m_entry.keymask.u8.key = match.key->type;
+ m_entry.keymask.u8.mask = match.mask->type;
+ err = prestera_acl_rule_match_add(rule, &m_entry);
+ if (err)
+ return err;
+
+ memset(&m_entry, 0, sizeof(m_entry));
+ m_entry.type = PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ICMP_CODE;
+ m_entry.keymask.u8.key = match.key->code;
+ m_entry.keymask.u8.mask = match.mask->code;
+ err = prestera_acl_rule_match_add(rule, &m_entry);
+ if (err)
+ return err;
+ }
+
+ return prestera_flower_parse_actions(block, rule,
+ &f->rule->action,
+ f->common.extack);
+}
+
+int prestera_flower_replace(struct prestera_flow_block *block,
+ struct flow_cls_offload *f)
+{
+ struct prestera_switch *sw = prestera_acl_block_sw(block);
+ struct prestera_acl_rule *rule;
+ int err;
+
+ rule = prestera_acl_rule_create(block, f->cookie);
+ if (IS_ERR(rule))
+ return PTR_ERR(rule);
+
+ err = prestera_flower_parse(block, rule, f);
+ if (err)
+ goto err_flower_parse;
+
+ err = prestera_acl_rule_add(sw, rule);
+ if (err)
+ goto err_rule_add;
+
+ return 0;
+
+err_rule_add:
+err_flower_parse:
+ prestera_acl_rule_destroy(rule);
+ return err;
+}
+
+void prestera_flower_destroy(struct prestera_flow_block *block,
+ struct flow_cls_offload *f)
+{
+ struct prestera_acl_rule *rule;
+ struct prestera_switch *sw;
+
+ rule = prestera_acl_rule_lookup(prestera_acl_block_ruleset_get(block),
+ f->cookie);
+ if (rule) {
+ sw = prestera_acl_block_sw(block);
+ prestera_acl_rule_del(sw, rule);
+ prestera_acl_rule_destroy(rule);
+ }
+}
+
+int prestera_flower_stats(struct prestera_flow_block *block,
+ struct flow_cls_offload *f)
+{
+ struct prestera_switch *sw = prestera_acl_block_sw(block);
+ struct prestera_acl_rule *rule;
+ u64 packets;
+ u64 lastuse;
+ u64 bytes;
+ int err;
+
+ rule = prestera_acl_rule_lookup(prestera_acl_block_ruleset_get(block),
+ f->cookie);
+ if (!rule)
+ return -EINVAL;
+
+ err = prestera_acl_rule_get_stats(sw, rule, &packets, &bytes, &lastuse);
+ if (err)
+ return err;
+
+ flow_stats_update(&f->stats, bytes, packets, 0, lastuse,
+ FLOW_ACTION_HW_STATS_IMMEDIATE);
+ return 0;
+}
--- /dev/null
+/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 */
+/* Copyright (c) 2020 Marvell International Ltd. All rights reserved. */
+
+#ifndef _PRESTERA_FLOWER_H_
+#define _PRESTERA_FLOWER_H_
+
+#include <net/pkt_cls.h>
+
+struct prestera_flow_block;
+
+int prestera_flower_replace(struct prestera_flow_block *block,
+ struct flow_cls_offload *f);
+void prestera_flower_destroy(struct prestera_flow_block *block,
+ struct flow_cls_offload *f);
+int prestera_flower_stats(struct prestera_flow_block *block,
+ struct flow_cls_offload *f);
+
+#endif /* _PRESTERA_FLOWER_H_ */
/* Copyright (c) 2019-2020 Marvell International Ltd. All rights reserved */
#include <linux/etherdevice.h>
+#include <linux/if_bridge.h>
#include <linux/ethtool.h>
#include <linux/list.h>
#include "prestera.h"
#include "prestera_hw.h"
+#include "prestera_acl.h"
#define PRESTERA_SWITCH_INIT_TIMEOUT_MS (30 * 1000)
PRESTERA_CMD_TYPE_BRIDGE_PORT_ADD = 0x402,
PRESTERA_CMD_TYPE_BRIDGE_PORT_DELETE = 0x403,
+ PRESTERA_CMD_TYPE_ACL_RULE_ADD = 0x500,
+ PRESTERA_CMD_TYPE_ACL_RULE_DELETE = 0x501,
+ PRESTERA_CMD_TYPE_ACL_RULE_STATS_GET = 0x510,
+ PRESTERA_CMD_TYPE_ACL_RULESET_CREATE = 0x520,
+ PRESTERA_CMD_TYPE_ACL_RULESET_DELETE = 0x521,
+ PRESTERA_CMD_TYPE_ACL_PORT_BIND = 0x530,
+ PRESTERA_CMD_TYPE_ACL_PORT_UNBIND = 0x531,
+
PRESTERA_CMD_TYPE_RXTX_INIT = 0x800,
PRESTERA_CMD_TYPE_RXTX_PORT_INIT = 0x801,
+ PRESTERA_CMD_TYPE_LAG_MEMBER_ADD = 0x900,
+ PRESTERA_CMD_TYPE_LAG_MEMBER_DELETE = 0x901,
+ PRESTERA_CMD_TYPE_LAG_MEMBER_ENABLE = 0x902,
+ PRESTERA_CMD_TYPE_LAG_MEMBER_DISABLE = 0x903,
+
PRESTERA_CMD_TYPE_STP_PORT_SET = 0x1000,
+ PRESTERA_CMD_TYPE_SPAN_GET = 0x1100,
+ PRESTERA_CMD_TYPE_SPAN_BIND = 0x1101,
+ PRESTERA_CMD_TYPE_SPAN_UNBIND = 0x1102,
+ PRESTERA_CMD_TYPE_SPAN_RELEASE = 0x1103,
+
+ PRESTERA_CMD_TYPE_CPU_CODE_COUNTERS_GET = 0x2000,
+
PRESTERA_CMD_TYPE_ACK = 0x10000,
PRESTERA_CMD_TYPE_MAX
};
};
enum {
+ PRESTERA_PORT_FLOOD_TYPE_UC = 0,
+ PRESTERA_PORT_FLOOD_TYPE_MC = 1,
+};
+
+enum {
PRESTERA_PORT_GOOD_OCTETS_RCV_CNT,
PRESTERA_PORT_BAD_OCTETS_RCV_CNT,
PRESTERA_PORT_MAC_TRANSMIT_ERR_CNT,
PRESTERA_FC_SYMM_ASYMM,
};
+enum {
+ PRESTERA_HW_FDB_ENTRY_TYPE_REG_PORT = 0,
+ PRESTERA_HW_FDB_ENTRY_TYPE_LAG = 1,
+ PRESTERA_HW_FDB_ENTRY_TYPE_MAX = 2,
+};
+
struct prestera_fw_event_handler {
struct list_head list;
struct rcu_head rcu;
u32 port_count;
u32 mtu_max;
u8 switch_id;
+ u8 lag_max;
+ u8 lag_member_max;
};
struct prestera_msg_port_autoneg_param {
u8 admin_mode;
};
+struct prestera_msg_port_flood_param {
+ u8 type;
+ u8 enable;
+};
+
union prestera_msg_port_param {
u8 admin_state;
u8 oper_state;
struct prestera_msg_port_mdix_param mdix;
struct prestera_msg_port_autoneg_param autoneg;
struct prestera_msg_port_cap_param cap;
+ struct prestera_msg_port_flood_param flood_ext;
};
struct prestera_msg_port_attr_req {
struct prestera_msg_fdb_req {
struct prestera_msg_cmd cmd;
u8 dest_type;
- u32 port;
- u32 dev;
+ union {
+ struct {
+ u32 port;
+ u32 dev;
+ };
+ u16 lag_id;
+ } dest;
u8 mac[ETH_ALEN];
u16 vid;
u8 dynamic;
u16 bridge;
};
+struct prestera_msg_acl_action {
+ u32 id;
+};
+
+struct prestera_msg_acl_match {
+ u32 type;
+ union {
+ struct {
+ u8 key;
+ u8 mask;
+ } u8;
+ struct {
+ u16 key;
+ u16 mask;
+ } u16;
+ struct {
+ u32 key;
+ u32 mask;
+ } u32;
+ struct {
+ u64 key;
+ u64 mask;
+ } u64;
+ struct {
+ u8 key[ETH_ALEN];
+ u8 mask[ETH_ALEN];
+ } mac;
+ } __packed keymask;
+};
+
+struct prestera_msg_acl_rule_req {
+ struct prestera_msg_cmd cmd;
+ u32 id;
+ u32 priority;
+ u16 ruleset_id;
+ u8 n_actions;
+ u8 n_matches;
+};
+
+struct prestera_msg_acl_rule_resp {
+ struct prestera_msg_ret ret;
+ u32 id;
+};
+
+struct prestera_msg_acl_rule_stats_resp {
+ struct prestera_msg_ret ret;
+ u64 packets;
+ u64 bytes;
+};
+
+struct prestera_msg_acl_ruleset_bind_req {
+ struct prestera_msg_cmd cmd;
+ u32 port;
+ u32 dev;
+ u16 ruleset_id;
+};
+
+struct prestera_msg_acl_ruleset_req {
+ struct prestera_msg_cmd cmd;
+ u16 id;
+};
+
+struct prestera_msg_acl_ruleset_resp {
+ struct prestera_msg_ret ret;
+ u16 id;
+};
+
+struct prestera_msg_span_req {
+ struct prestera_msg_cmd cmd;
+ u32 port;
+ u32 dev;
+ u8 id;
+} __packed __aligned(4);
+
+struct prestera_msg_span_resp {
+ struct prestera_msg_ret ret;
+ u8 id;
+} __packed __aligned(4);
+
struct prestera_msg_stp_req {
struct prestera_msg_cmd cmd;
u32 port;
u32 dev;
};
+struct prestera_msg_lag_req {
+ struct prestera_msg_cmd cmd;
+ u32 port;
+ u32 dev;
+ u16 lag_id;
+};
+
+struct prestera_msg_cpu_code_counter_req {
+ struct prestera_msg_cmd cmd;
+ u8 counter_type;
+ u8 code;
+};
+
+struct mvsw_msg_cpu_code_counter_ret {
+ struct prestera_msg_ret ret;
+ u64 packet_count;
+};
+
struct prestera_msg_event {
u16 type;
u16 id;
struct prestera_msg_event_fdb {
struct prestera_msg_event id;
u8 dest_type;
- u32 port_id;
+ union {
+ u32 port_id;
+ u16 lag_id;
+ } dest;
u32 vid;
union prestera_msg_event_fdb_param param;
};
{
struct prestera_msg_event_fdb *hw_evt = msg;
- evt->fdb_evt.port_id = hw_evt->port_id;
+ switch (hw_evt->dest_type) {
+ case PRESTERA_HW_FDB_ENTRY_TYPE_REG_PORT:
+ evt->fdb_evt.type = PRESTERA_FDB_ENTRY_TYPE_REG_PORT;
+ evt->fdb_evt.dest.port_id = hw_evt->dest.port_id;
+ break;
+ case PRESTERA_HW_FDB_ENTRY_TYPE_LAG:
+ evt->fdb_evt.type = PRESTERA_FDB_ENTRY_TYPE_LAG;
+ evt->fdb_evt.dest.lag_id = hw_evt->dest.lag_id;
+ break;
+ default:
+ return -EINVAL;
+ }
+
evt->fdb_evt.vid = hw_evt->vid;
ether_addr_copy(evt->fdb_evt.data.mac, hw_evt->param.mac);
sw->mtu_min = PRESTERA_MIN_MTU;
sw->mtu_max = resp.mtu_max;
sw->id = resp.switch_id;
+ sw->lag_member_max = resp.lag_member_max;
+ sw->lag_max = resp.lag_max;
return 0;
}
return 0;
}
+int prestera_hw_acl_ruleset_create(struct prestera_switch *sw, u16 *ruleset_id)
+{
+ struct prestera_msg_acl_ruleset_resp resp;
+ struct prestera_msg_acl_ruleset_req req;
+ int err;
+
+ err = prestera_cmd_ret(sw, PRESTERA_CMD_TYPE_ACL_RULESET_CREATE,
+ &req.cmd, sizeof(req), &resp.ret, sizeof(resp));
+ if (err)
+ return err;
+
+ *ruleset_id = resp.id;
+
+ return 0;
+}
+
+int prestera_hw_acl_ruleset_del(struct prestera_switch *sw, u16 ruleset_id)
+{
+ struct prestera_msg_acl_ruleset_req req = {
+ .id = ruleset_id,
+ };
+
+ return prestera_cmd(sw, PRESTERA_CMD_TYPE_ACL_RULESET_DELETE,
+ &req.cmd, sizeof(req));
+}
+
+static int prestera_hw_acl_actions_put(struct prestera_msg_acl_action *action,
+ struct prestera_acl_rule *rule)
+{
+ struct list_head *a_list = prestera_acl_rule_action_list_get(rule);
+ struct prestera_acl_rule_action_entry *a_entry;
+ int i = 0;
+
+ list_for_each_entry(a_entry, a_list, list) {
+ action[i].id = a_entry->id;
+
+ switch (a_entry->id) {
+ case PRESTERA_ACL_RULE_ACTION_ACCEPT:
+ case PRESTERA_ACL_RULE_ACTION_DROP:
+ case PRESTERA_ACL_RULE_ACTION_TRAP:
+ /* just rule action id, no specific data */
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
+static int prestera_hw_acl_matches_put(struct prestera_msg_acl_match *match,
+ struct prestera_acl_rule *rule)
+{
+ struct list_head *m_list = prestera_acl_rule_match_list_get(rule);
+ struct prestera_acl_rule_match_entry *m_entry;
+ int i = 0;
+
+ list_for_each_entry(m_entry, m_list, list) {
+ match[i].type = m_entry->type;
+
+ switch (m_entry->type) {
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ETH_TYPE:
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_L4_PORT_SRC:
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_L4_PORT_DST:
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_VLAN_ID:
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_VLAN_TPID:
+ match[i].keymask.u16.key = m_entry->keymask.u16.key;
+ match[i].keymask.u16.mask = m_entry->keymask.u16.mask;
+ break;
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ICMP_TYPE:
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ICMP_CODE:
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_IP_PROTO:
+ match[i].keymask.u8.key = m_entry->keymask.u8.key;
+ match[i].keymask.u8.mask = m_entry->keymask.u8.mask;
+ break;
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ETH_SMAC:
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_ETH_DMAC:
+ memcpy(match[i].keymask.mac.key,
+ m_entry->keymask.mac.key,
+ sizeof(match[i].keymask.mac.key));
+ memcpy(match[i].keymask.mac.mask,
+ m_entry->keymask.mac.mask,
+ sizeof(match[i].keymask.mac.mask));
+ break;
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_IP_SRC:
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_IP_DST:
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_L4_PORT_RANGE_SRC:
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_L4_PORT_RANGE_DST:
+ match[i].keymask.u32.key = m_entry->keymask.u32.key;
+ match[i].keymask.u32.mask = m_entry->keymask.u32.mask;
+ break;
+ case PRESTERA_ACL_RULE_MATCH_ENTRY_TYPE_PORT:
+ match[i].keymask.u64.key = m_entry->keymask.u64.key;
+ match[i].keymask.u64.mask = m_entry->keymask.u64.mask;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
+int prestera_hw_acl_rule_add(struct prestera_switch *sw,
+ struct prestera_acl_rule *rule,
+ u32 *rule_id)
+{
+ struct prestera_msg_acl_action *actions;
+ struct prestera_msg_acl_match *matches;
+ struct prestera_msg_acl_rule_resp resp;
+ struct prestera_msg_acl_rule_req *req;
+ u8 n_actions;
+ u8 n_matches;
+ void *buff;
+ u32 size;
+ int err;
+
+ n_actions = prestera_acl_rule_action_len(rule);
+ n_matches = prestera_acl_rule_match_len(rule);
+
+ size = sizeof(*req) + sizeof(*actions) * n_actions +
+ sizeof(*matches) * n_matches;
+
+ buff = kzalloc(size, GFP_KERNEL);
+ if (!buff)
+ return -ENOMEM;
+
+ req = buff;
+ actions = buff + sizeof(*req);
+ matches = buff + sizeof(*req) + sizeof(*actions) * n_actions;
+
+ /* put acl actions into the message */
+ err = prestera_hw_acl_actions_put(actions, rule);
+ if (err)
+ goto free_buff;
+
+ /* put acl matches into the message */
+ err = prestera_hw_acl_matches_put(matches, rule);
+ if (err)
+ goto free_buff;
+
+ req->ruleset_id = prestera_acl_rule_ruleset_id_get(rule);
+ req->priority = prestera_acl_rule_priority_get(rule);
+ req->n_actions = prestera_acl_rule_action_len(rule);
+ req->n_matches = prestera_acl_rule_match_len(rule);
+
+ err = prestera_cmd_ret(sw, PRESTERA_CMD_TYPE_ACL_RULE_ADD,
+ &req->cmd, size, &resp.ret, sizeof(resp));
+ if (err)
+ goto free_buff;
+
+ *rule_id = resp.id;
+free_buff:
+ kfree(buff);
+ return err;
+}
+
+int prestera_hw_acl_rule_del(struct prestera_switch *sw, u32 rule_id)
+{
+ struct prestera_msg_acl_rule_req req = {
+ .id = rule_id
+ };
+
+ return prestera_cmd(sw, PRESTERA_CMD_TYPE_ACL_RULE_DELETE,
+ &req.cmd, sizeof(req));
+}
+
+int prestera_hw_acl_rule_stats_get(struct prestera_switch *sw, u32 rule_id,
+ u64 *packets, u64 *bytes)
+{
+ struct prestera_msg_acl_rule_stats_resp resp;
+ struct prestera_msg_acl_rule_req req = {
+ .id = rule_id
+ };
+ int err;
+
+ err = prestera_cmd_ret(sw, PRESTERA_CMD_TYPE_ACL_RULE_STATS_GET,
+ &req.cmd, sizeof(req), &resp.ret, sizeof(resp));
+ if (err)
+ return err;
+
+ *packets = resp.packets;
+ *bytes = resp.bytes;
+
+ return 0;
+}
+
+int prestera_hw_acl_port_bind(const struct prestera_port *port, u16 ruleset_id)
+{
+ struct prestera_msg_acl_ruleset_bind_req req = {
+ .port = port->hw_id,
+ .dev = port->dev_id,
+ .ruleset_id = ruleset_id,
+ };
+
+ return prestera_cmd(port->sw, PRESTERA_CMD_TYPE_ACL_PORT_BIND,
+ &req.cmd, sizeof(req));
+}
+
+int prestera_hw_acl_port_unbind(const struct prestera_port *port,
+ u16 ruleset_id)
+{
+ struct prestera_msg_acl_ruleset_bind_req req = {
+ .port = port->hw_id,
+ .dev = port->dev_id,
+ .ruleset_id = ruleset_id,
+ };
+
+ return prestera_cmd(port->sw, PRESTERA_CMD_TYPE_ACL_PORT_UNBIND,
+ &req.cmd, sizeof(req));
+}
+
+int prestera_hw_span_get(const struct prestera_port *port, u8 *span_id)
+{
+ struct prestera_msg_span_resp resp;
+ struct prestera_msg_span_req req = {
+ .port = port->hw_id,
+ .dev = port->dev_id,
+ };
+ int err;
+
+ err = prestera_cmd_ret(port->sw, PRESTERA_CMD_TYPE_SPAN_GET,
+ &req.cmd, sizeof(req), &resp.ret, sizeof(resp));
+ if (err)
+ return err;
+
+ *span_id = resp.id;
+
+ return 0;
+}
+
+int prestera_hw_span_bind(const struct prestera_port *port, u8 span_id)
+{
+ struct prestera_msg_span_req req = {
+ .port = port->hw_id,
+ .dev = port->dev_id,
+ .id = span_id,
+ };
+
+ return prestera_cmd(port->sw, PRESTERA_CMD_TYPE_SPAN_BIND,
+ &req.cmd, sizeof(req));
+}
+
+int prestera_hw_span_unbind(const struct prestera_port *port)
+{
+ struct prestera_msg_span_req req = {
+ .port = port->hw_id,
+ .dev = port->dev_id,
+ };
+
+ return prestera_cmd(port->sw, PRESTERA_CMD_TYPE_SPAN_UNBIND,
+ &req.cmd, sizeof(req));
+}
+
+int prestera_hw_span_release(struct prestera_switch *sw, u8 span_id)
+{
+ struct prestera_msg_span_req req = {
+ .id = span_id
+ };
+
+ return prestera_cmd(sw, PRESTERA_CMD_TYPE_SPAN_RELEASE,
+ &req.cmd, sizeof(req));
+}
+
int prestera_hw_port_type_get(const struct prestera_port *port, u8 *type)
{
struct prestera_msg_port_attr_req req = {
&req.cmd, sizeof(req));
}
-int prestera_hw_port_flood_set(struct prestera_port *port, bool flood)
+static int prestera_hw_port_uc_flood_set(struct prestera_port *port, bool flood)
+{
+ struct prestera_msg_port_attr_req req = {
+ .attr = PRESTERA_CMD_PORT_ATTR_FLOOD,
+ .port = port->hw_id,
+ .dev = port->dev_id,
+ .param = {
+ .flood_ext = {
+ .type = PRESTERA_PORT_FLOOD_TYPE_UC,
+ .enable = flood,
+ }
+ }
+ };
+
+ return prestera_cmd(port->sw, PRESTERA_CMD_TYPE_PORT_ATTR_SET,
+ &req.cmd, sizeof(req));
+}
+
+static int prestera_hw_port_mc_flood_set(struct prestera_port *port, bool flood)
+{
+ struct prestera_msg_port_attr_req req = {
+ .attr = PRESTERA_CMD_PORT_ATTR_FLOOD,
+ .port = port->hw_id,
+ .dev = port->dev_id,
+ .param = {
+ .flood_ext = {
+ .type = PRESTERA_PORT_FLOOD_TYPE_MC,
+ .enable = flood,
+ }
+ }
+ };
+
+ return prestera_cmd(port->sw, PRESTERA_CMD_TYPE_PORT_ATTR_SET,
+ &req.cmd, sizeof(req));
+}
+
+static int prestera_hw_port_flood_set_v2(struct prestera_port *port, bool flood)
{
struct prestera_msg_port_attr_req req = {
.attr = PRESTERA_CMD_PORT_ATTR_FLOOD,
&req.cmd, sizeof(req));
}
+int prestera_hw_port_flood_set(struct prestera_port *port, unsigned long mask,
+ unsigned long val)
+{
+ int err;
+
+ if (port->sw->dev->fw_rev.maj <= 2) {
+ if (!(mask & BR_FLOOD))
+ return 0;
+
+ return prestera_hw_port_flood_set_v2(port, val & BR_FLOOD);
+ }
+
+ if (mask & BR_FLOOD) {
+ err = prestera_hw_port_uc_flood_set(port, val & BR_FLOOD);
+ if (err)
+ goto err_uc_flood;
+ }
+
+ if (mask & BR_MCAST_FLOOD) {
+ err = prestera_hw_port_mc_flood_set(port, val & BR_MCAST_FLOOD);
+ if (err)
+ goto err_mc_flood;
+ }
+
+ return 0;
+
+err_mc_flood:
+ prestera_hw_port_mc_flood_set(port, 0);
+err_uc_flood:
+ if (mask & BR_FLOOD)
+ prestera_hw_port_uc_flood_set(port, 0);
+
+ return err;
+}
+
int prestera_hw_vlan_create(struct prestera_switch *sw, u16 vid)
{
struct prestera_msg_vlan_req req = {
u16 vid, bool dynamic)
{
struct prestera_msg_fdb_req req = {
- .port = port->hw_id,
- .dev = port->dev_id,
+ .dest = {
+ .dev = port->dev_id,
+ .port = port->hw_id,
+ },
.vid = vid,
.dynamic = dynamic,
};
u16 vid)
{
struct prestera_msg_fdb_req req = {
- .port = port->hw_id,
- .dev = port->dev_id,
+ .dest = {
+ .dev = port->dev_id,
+ .port = port->hw_id,
+ },
.vid = vid,
};
&req.cmd, sizeof(req));
}
+int prestera_hw_lag_fdb_add(struct prestera_switch *sw, u16 lag_id,
+ const unsigned char *mac, u16 vid, bool dynamic)
+{
+ struct prestera_msg_fdb_req req = {
+ .dest_type = PRESTERA_HW_FDB_ENTRY_TYPE_LAG,
+ .dest = {
+ .lag_id = lag_id,
+ },
+ .vid = vid,
+ .dynamic = dynamic,
+ };
+
+ ether_addr_copy(req.mac, mac);
+
+ return prestera_cmd(sw, PRESTERA_CMD_TYPE_FDB_ADD,
+ &req.cmd, sizeof(req));
+}
+
+int prestera_hw_lag_fdb_del(struct prestera_switch *sw, u16 lag_id,
+ const unsigned char *mac, u16 vid)
+{
+ struct prestera_msg_fdb_req req = {
+ .dest_type = PRESTERA_HW_FDB_ENTRY_TYPE_LAG,
+ .dest = {
+ .lag_id = lag_id,
+ },
+ .vid = vid,
+ };
+
+ ether_addr_copy(req.mac, mac);
+
+ return prestera_cmd(sw, PRESTERA_CMD_TYPE_FDB_DELETE,
+ &req.cmd, sizeof(req));
+}
+
int prestera_hw_fdb_flush_port(struct prestera_port *port, u32 mode)
{
struct prestera_msg_fdb_req req = {
- .port = port->hw_id,
- .dev = port->dev_id,
+ .dest = {
+ .dev = port->dev_id,
+ .port = port->hw_id,
+ },
.flush_mode = mode,
};
u32 mode)
{
struct prestera_msg_fdb_req req = {
- .port = port->hw_id,
- .dev = port->dev_id,
+ .dest = {
+ .dev = port->dev_id,
+ .port = port->hw_id,
+ },
.vid = vid,
.flush_mode = mode,
};
&req.cmd, sizeof(req));
}
+int prestera_hw_fdb_flush_lag(struct prestera_switch *sw, u16 lag_id,
+ u32 mode)
+{
+ struct prestera_msg_fdb_req req = {
+ .dest_type = PRESTERA_HW_FDB_ENTRY_TYPE_LAG,
+ .dest = {
+ .lag_id = lag_id,
+ },
+ .flush_mode = mode,
+ };
+
+ return prestera_cmd(sw, PRESTERA_CMD_TYPE_FDB_FLUSH_PORT,
+ &req.cmd, sizeof(req));
+}
+
+int prestera_hw_fdb_flush_lag_vlan(struct prestera_switch *sw,
+ u16 lag_id, u16 vid, u32 mode)
+{
+ struct prestera_msg_fdb_req req = {
+ .dest_type = PRESTERA_HW_FDB_ENTRY_TYPE_LAG,
+ .dest = {
+ .lag_id = lag_id,
+ },
+ .vid = vid,
+ .flush_mode = mode,
+ };
+
+ return prestera_cmd(sw, PRESTERA_CMD_TYPE_FDB_FLUSH_PORT_VLAN,
+ &req.cmd, sizeof(req));
+}
+
int prestera_hw_bridge_create(struct prestera_switch *sw, u16 *bridge_id)
{
struct prestera_msg_bridge_resp resp;
&req.cmd, sizeof(req));
}
+int prestera_hw_lag_member_add(struct prestera_port *port, u16 lag_id)
+{
+ struct prestera_msg_lag_req req = {
+ .port = port->hw_id,
+ .dev = port->dev_id,
+ .lag_id = lag_id,
+ };
+
+ return prestera_cmd(port->sw, PRESTERA_CMD_TYPE_LAG_MEMBER_ADD,
+ &req.cmd, sizeof(req));
+}
+
+int prestera_hw_lag_member_del(struct prestera_port *port, u16 lag_id)
+{
+ struct prestera_msg_lag_req req = {
+ .port = port->hw_id,
+ .dev = port->dev_id,
+ .lag_id = lag_id,
+ };
+
+ return prestera_cmd(port->sw, PRESTERA_CMD_TYPE_LAG_MEMBER_DELETE,
+ &req.cmd, sizeof(req));
+}
+
+int prestera_hw_lag_member_enable(struct prestera_port *port, u16 lag_id,
+ bool enable)
+{
+ struct prestera_msg_lag_req req = {
+ .port = port->hw_id,
+ .dev = port->dev_id,
+ .lag_id = lag_id,
+ };
+ u32 cmd;
+
+ cmd = enable ? PRESTERA_CMD_TYPE_LAG_MEMBER_ENABLE :
+ PRESTERA_CMD_TYPE_LAG_MEMBER_DISABLE;
+
+ return prestera_cmd(port->sw, cmd, &req.cmd, sizeof(req));
+}
+
+int
+prestera_hw_cpu_code_counters_get(struct prestera_switch *sw, u8 code,
+ enum prestera_hw_cpu_code_cnt_t counter_type,
+ u64 *packet_count)
+{
+ struct prestera_msg_cpu_code_counter_req req = {
+ .counter_type = counter_type,
+ .code = code,
+ };
+ struct mvsw_msg_cpu_code_counter_ret resp;
+ int err;
+
+ err = prestera_cmd_ret(sw, PRESTERA_CMD_TYPE_CPU_CODE_COUNTERS_GET,
+ &req.cmd, sizeof(req), &resp.ret, sizeof(resp));
+ if (err)
+ return err;
+
+ *packet_count = resp.packet_count;
+
+ return 0;
+}
+
int prestera_hw_event_handler_register(struct prestera_switch *sw,
enum prestera_event_type type,
prestera_event_cb_t fn,
PRESTERA_STP_FORWARD,
};
+enum prestera_hw_cpu_code_cnt_t {
+ PRESTERA_HW_CPU_CODE_CNT_TYPE_DROP = 0,
+ PRESTERA_HW_CPU_CODE_CNT_TYPE_TRAP = 1,
+};
+
struct prestera_switch;
struct prestera_port;
struct prestera_port_stats;
struct prestera_port_caps;
enum prestera_event_type;
struct prestera_event;
+struct prestera_acl_rule;
typedef void (*prestera_event_cb_t)
(struct prestera_switch *sw, struct prestera_event *evt, void *arg);
int prestera_hw_port_mdix_set(const struct prestera_port *port, u8 mode);
int prestera_hw_port_speed_get(const struct prestera_port *port, u32 *speed);
int prestera_hw_port_learning_set(struct prestera_port *port, bool enable);
-int prestera_hw_port_flood_set(struct prestera_port *port, bool flood);
+int prestera_hw_port_flood_set(struct prestera_port *port, unsigned long mask,
+ unsigned long val);
int prestera_hw_port_accept_frm_type(struct prestera_port *port,
enum prestera_accept_frm_type type);
/* Vlan API */
int prestera_hw_bridge_port_add(struct prestera_port *port, u16 bridge_id);
int prestera_hw_bridge_port_delete(struct prestera_port *port, u16 bridge_id);
+/* ACL API */
+int prestera_hw_acl_ruleset_create(struct prestera_switch *sw,
+ u16 *ruleset_id);
+int prestera_hw_acl_ruleset_del(struct prestera_switch *sw,
+ u16 ruleset_id);
+int prestera_hw_acl_rule_add(struct prestera_switch *sw,
+ struct prestera_acl_rule *rule,
+ u32 *rule_id);
+int prestera_hw_acl_rule_del(struct prestera_switch *sw, u32 rule_id);
+int prestera_hw_acl_rule_stats_get(struct prestera_switch *sw,
+ u32 rule_id, u64 *packets, u64 *bytes);
+int prestera_hw_acl_port_bind(const struct prestera_port *port,
+ u16 ruleset_id);
+int prestera_hw_acl_port_unbind(const struct prestera_port *port,
+ u16 ruleset_id);
+
+/* SPAN API */
+int prestera_hw_span_get(const struct prestera_port *port, u8 *span_id);
+int prestera_hw_span_bind(const struct prestera_port *port, u8 span_id);
+int prestera_hw_span_unbind(const struct prestera_port *port);
+int prestera_hw_span_release(struct prestera_switch *sw, u8 span_id);
+
/* Event handlers */
int prestera_hw_event_handler_register(struct prestera_switch *sw,
enum prestera_event_type type,
struct prestera_rxtx_params *params);
int prestera_hw_rxtx_port_init(struct prestera_port *port);
+/* LAG API */
+int prestera_hw_lag_member_add(struct prestera_port *port, u16 lag_id);
+int prestera_hw_lag_member_del(struct prestera_port *port, u16 lag_id);
+int prestera_hw_lag_member_enable(struct prestera_port *port, u16 lag_id,
+ bool enable);
+int prestera_hw_lag_fdb_add(struct prestera_switch *sw, u16 lag_id,
+ const unsigned char *mac, u16 vid, bool dynamic);
+int prestera_hw_lag_fdb_del(struct prestera_switch *sw, u16 lag_id,
+ const unsigned char *mac, u16 vid);
+int prestera_hw_fdb_flush_lag(struct prestera_switch *sw, u16 lag_id,
+ u32 mode);
+int prestera_hw_fdb_flush_lag_vlan(struct prestera_switch *sw,
+ u16 lag_id, u16 vid, u32 mode);
+
+/* HW trap/drop counters API */
+int
+prestera_hw_cpu_code_counters_get(struct prestera_switch *sw, u8 code,
+ enum prestera_hw_cpu_code_cnt_t counter_type,
+ u64 *packet_count);
+
#endif /* _PRESTERA_HW_H_ */
#include <linux/netdev_features.h>
#include <linux/of.h>
#include <linux/of_net.h>
+#include <linux/if_vlan.h>
#include "prestera.h"
#include "prestera_hw.h"
+#include "prestera_acl.h"
+#include "prestera_flow.h"
+#include "prestera_span.h"
#include "prestera_rxtx.h"
#include "prestera_devlink.h"
#include "prestera_ethtool.h"
msecs_to_jiffies(PRESTERA_STATS_DELAY_MS));
}
+static int prestera_port_setup_tc(struct net_device *dev,
+ enum tc_setup_type type,
+ void *type_data)
+{
+ struct prestera_port *port = netdev_priv(dev);
+
+ switch (type) {
+ case TC_SETUP_BLOCK:
+ return prestera_flow_block_setup(port, type_data);
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+
static const struct net_device_ops prestera_netdev_ops = {
.ndo_open = prestera_port_open,
.ndo_stop = prestera_port_close,
.ndo_start_xmit = prestera_port_xmit,
+ .ndo_setup_tc = prestera_port_setup_tc,
.ndo_change_mtu = prestera_port_change_mtu,
.ndo_get_stats64 = prestera_port_get_stats64,
.ndo_set_mac_address = prestera_port_set_mac_address,
INIT_LIST_HEAD(&port->vlans_list);
port->pvid = PRESTERA_DEFAULT_VID;
+ port->lag = NULL;
port->dev = dev;
port->id = id;
port->sw = sw;
if (err)
goto err_dl_port_register;
- dev->features |= NETIF_F_NETNS_LOCAL;
+ dev->features |= NETIF_F_NETNS_LOCAL | NETIF_F_HW_TC;
dev->netdev_ops = &prestera_netdev_ops;
dev->ethtool_ops = &prestera_ethtool_ops;
return prestera_hw_switch_mac_set(sw, sw->base_mac);
}
+struct prestera_lag *prestera_lag_by_id(struct prestera_switch *sw, u16 id)
+{
+ return id < sw->lag_max ? &sw->lags[id] : NULL;
+}
+
+static struct prestera_lag *prestera_lag_by_dev(struct prestera_switch *sw,
+ struct net_device *dev)
+{
+ struct prestera_lag *lag;
+ u16 id;
+
+ for (id = 0; id < sw->lag_max; id++) {
+ lag = &sw->lags[id];
+ if (lag->dev == dev)
+ return lag;
+ }
+
+ return NULL;
+}
+
+static struct prestera_lag *prestera_lag_create(struct prestera_switch *sw,
+ struct net_device *lag_dev)
+{
+ struct prestera_lag *lag = NULL;
+ u16 id;
+
+ for (id = 0; id < sw->lag_max; id++) {
+ lag = &sw->lags[id];
+ if (!lag->dev)
+ break;
+ }
+ if (lag) {
+ INIT_LIST_HEAD(&lag->members);
+ lag->dev = lag_dev;
+ }
+
+ return lag;
+}
+
+static void prestera_lag_destroy(struct prestera_switch *sw,
+ struct prestera_lag *lag)
+{
+ WARN_ON(!list_empty(&lag->members));
+ lag->member_count = 0;
+ lag->dev = NULL;
+}
+
+static int prestera_lag_port_add(struct prestera_port *port,
+ struct net_device *lag_dev)
+{
+ struct prestera_switch *sw = port->sw;
+ struct prestera_lag *lag;
+ int err;
+
+ lag = prestera_lag_by_dev(sw, lag_dev);
+ if (!lag) {
+ lag = prestera_lag_create(sw, lag_dev);
+ if (!lag)
+ return -ENOSPC;
+ }
+
+ if (lag->member_count >= sw->lag_member_max)
+ return -ENOSPC;
+
+ err = prestera_hw_lag_member_add(port, lag->lag_id);
+ if (err) {
+ if (!lag->member_count)
+ prestera_lag_destroy(sw, lag);
+ return err;
+ }
+
+ list_add(&port->lag_member, &lag->members);
+ lag->member_count++;
+ port->lag = lag;
+
+ return 0;
+}
+
+static int prestera_lag_port_del(struct prestera_port *port)
+{
+ struct prestera_switch *sw = port->sw;
+ struct prestera_lag *lag = port->lag;
+ int err;
+
+ if (!lag || !lag->member_count)
+ return -EINVAL;
+
+ err = prestera_hw_lag_member_del(port, lag->lag_id);
+ if (err)
+ return err;
+
+ list_del(&port->lag_member);
+ lag->member_count--;
+ port->lag = NULL;
+
+ if (netif_is_bridge_port(lag->dev)) {
+ struct net_device *br_dev;
+
+ br_dev = netdev_master_upper_dev_get(lag->dev);
+
+ prestera_bridge_port_leave(br_dev, port);
+ }
+
+ if (!lag->member_count)
+ prestera_lag_destroy(sw, lag);
+
+ return 0;
+}
+
+bool prestera_port_is_lag_member(const struct prestera_port *port)
+{
+ return !!port->lag;
+}
+
+u16 prestera_port_lag_id(const struct prestera_port *port)
+{
+ return port->lag->lag_id;
+}
+
+static int prestera_lag_init(struct prestera_switch *sw)
+{
+ u16 id;
+
+ sw->lags = kcalloc(sw->lag_max, sizeof(*sw->lags), GFP_KERNEL);
+ if (!sw->lags)
+ return -ENOMEM;
+
+ for (id = 0; id < sw->lag_max; id++)
+ sw->lags[id].lag_id = id;
+
+ return 0;
+}
+
+static void prestera_lag_fini(struct prestera_switch *sw)
+{
+ u8 idx;
+
+ for (idx = 0; idx < sw->lag_max; idx++)
+ WARN_ON(sw->lags[idx].member_count);
+
+ kfree(sw->lags);
+}
+
bool prestera_netdev_check(const struct net_device *dev)
{
return dev->netdev_ops == &prestera_netdev_ops;
return port;
}
-static int prestera_netdev_port_event(struct net_device *dev,
+static int prestera_netdev_port_lower_event(struct net_device *dev,
+ unsigned long event, void *ptr)
+{
+ struct netdev_notifier_changelowerstate_info *info = ptr;
+ struct netdev_lag_lower_state_info *lower_state_info;
+ struct prestera_port *port = netdev_priv(dev);
+ bool enabled;
+
+ if (!netif_is_lag_port(dev))
+ return 0;
+ if (!prestera_port_is_lag_member(port))
+ return 0;
+
+ lower_state_info = info->lower_state_info;
+ enabled = lower_state_info->link_up && lower_state_info->tx_enabled;
+
+ return prestera_hw_lag_member_enable(port, port->lag->lag_id, enabled);
+}
+
+static bool prestera_lag_master_check(struct net_device *lag_dev,
+ struct netdev_lag_upper_info *info,
+ struct netlink_ext_ack *ext_ack)
+{
+ if (info->tx_type != NETDEV_LAG_TX_TYPE_HASH) {
+ NL_SET_ERR_MSG_MOD(ext_ack, "Unsupported LAG Tx type");
+ return false;
+ }
+
+ return true;
+}
+
+static int prestera_netdev_port_event(struct net_device *lower,
+ struct net_device *dev,
unsigned long event, void *ptr)
{
+ struct netdev_notifier_changeupper_info *info = ptr;
+ struct prestera_port *port = netdev_priv(dev);
+ struct netlink_ext_ack *extack;
+ struct net_device *upper;
+
+ extack = netdev_notifier_info_to_extack(&info->info);
+ upper = info->upper_dev;
+
switch (event) {
case NETDEV_PRECHANGEUPPER:
+ if (!netif_is_bridge_master(upper) &&
+ !netif_is_lag_master(upper)) {
+ NL_SET_ERR_MSG_MOD(extack, "Unknown upper device type");
+ return -EINVAL;
+ }
+
+ if (!info->linking)
+ break;
+
+ if (netdev_has_any_upper_dev(upper)) {
+ NL_SET_ERR_MSG_MOD(extack, "Upper device is already enslaved");
+ return -EINVAL;
+ }
+
+ if (netif_is_lag_master(upper) &&
+ !prestera_lag_master_check(upper, info->upper_info, extack))
+ return -EOPNOTSUPP;
+ if (netif_is_lag_master(upper) && vlan_uses_dev(dev)) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "Master device is a LAG master and port has a VLAN");
+ return -EINVAL;
+ }
+ if (netif_is_lag_port(dev) && is_vlan_dev(upper) &&
+ !netif_is_lag_master(vlan_dev_real_dev(upper))) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "Can not put a VLAN on a LAG port");
+ return -EINVAL;
+ }
+ break;
+
case NETDEV_CHANGEUPPER:
- return prestera_bridge_port_event(dev, event, ptr);
- default:
- return 0;
+ if (netif_is_bridge_master(upper)) {
+ if (info->linking)
+ return prestera_bridge_port_join(upper, port);
+ else
+ prestera_bridge_port_leave(upper, port);
+ } else if (netif_is_lag_master(upper)) {
+ if (info->linking)
+ return prestera_lag_port_add(port, upper);
+ else
+ prestera_lag_port_del(port);
+ }
+ break;
+
+ case NETDEV_CHANGELOWERSTATE:
+ return prestera_netdev_port_lower_event(dev, event, ptr);
}
+
+ return 0;
+}
+
+static int prestera_netdevice_lag_event(struct net_device *lag_dev,
+ unsigned long event, void *ptr)
+{
+ struct net_device *dev;
+ struct list_head *iter;
+ int err;
+
+ netdev_for_each_lower_dev(lag_dev, dev, iter) {
+ if (prestera_netdev_check(dev)) {
+ err = prestera_netdev_port_event(lag_dev, dev, event,
+ ptr);
+ if (err)
+ return err;
+ }
+ }
+
+ return 0;
}
static int prestera_netdev_event_handler(struct notifier_block *nb,
int err = 0;
if (prestera_netdev_check(dev))
- err = prestera_netdev_port_event(dev, event, ptr);
+ err = prestera_netdev_port_event(dev, dev, event, ptr);
+ else if (netif_is_lag_master(dev))
+ err = prestera_netdevice_lag_event(dev, event, ptr);
return notifier_from_errno(err);
}
if (err)
goto err_handlers_register;
+ err = prestera_acl_init(sw);
+ if (err)
+ goto err_acl_init;
+
+ err = prestera_span_init(sw);
+ if (err)
+ goto err_span_init;
+
err = prestera_devlink_register(sw);
if (err)
goto err_dl_register;
+ err = prestera_lag_init(sw);
+ if (err)
+ goto err_lag_init;
+
err = prestera_create_ports(sw);
if (err)
goto err_ports_create;
return 0;
err_ports_create:
+ prestera_lag_fini(sw);
+err_lag_init:
prestera_devlink_unregister(sw);
err_dl_register:
+ prestera_span_fini(sw);
+err_span_init:
+ prestera_acl_fini(sw);
+err_acl_init:
prestera_event_handlers_unregister(sw);
err_handlers_register:
prestera_rxtx_switch_fini(sw);
static void prestera_switch_fini(struct prestera_switch *sw)
{
prestera_destroy_ports(sw);
+ prestera_lag_fini(sw);
prestera_devlink_unregister(sw);
+ prestera_span_fini(sw);
+ prestera_acl_fini(sw);
prestera_event_handlers_unregister(sw);
prestera_rxtx_switch_fini(sw);
prestera_switchdev_fini(sw);
// SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0
/* Copyright (c) 2019-2020 Marvell International Ltd. All rights reserved */
+#include <linux/bitfield.h>
#include <linux/circ_buf.h>
#include <linux/device.h>
#include <linux/firmware.h>
#define PRESTERA_MSG_MAX_SIZE 1500
-#define PRESTERA_SUPP_FW_MAJ_VER 2
+#define PRESTERA_SUPP_FW_MAJ_VER 3
#define PRESTERA_SUPP_FW_MIN_VER 0
+#define PRESTERA_PREV_FW_MAJ_VER 2
+#define PRESTERA_PREV_FW_MIN_VER 0
+
#define PRESTERA_FW_PATH_FMT "mrvl/prestera/mvsw_prestera_fw-v%u.%u.img"
#define PRESTERA_FW_HDR_MAGIC 0x351D9D06
/* PRESTERA_CMD_RCV_CTL_REG flags */
#define PRESTERA_CMD_F_REPL_SENT BIT(0)
+#define PRESTERA_FW_EVT_CTL_STATUS_MASK GENMASK(1, 0)
+
+#define PRESTERA_FW_EVT_CTL_STATUS_ON 0
+#define PRESTERA_FW_EVT_CTL_STATUS_OFF 1
+
#define PRESTERA_EVTQ_REG_OFFSET(q, f) \
(PRESTERA_FW_REG_OFFSET(evtq_list) + \
(q) * sizeof(struct prestera_fw_evtq_regs) + \
};
struct prestera_fw {
+ struct prestera_fw_rev rev_supp;
+ const struct firmware *bin;
struct workqueue_struct *wq;
struct prestera_device dev;
u8 __iomem *ldr_regs;
return PRESTERA_EVT_QNUM_MAX;
}
+static void prestera_fw_evt_ctl_status_set(struct prestera_fw *fw, u32 val)
+{
+ u32 status = prestera_fw_read(fw, PRESTERA_FW_STATUS_REG);
+
+ u32p_replace_bits(&status, val, PRESTERA_FW_EVT_CTL_STATUS_MASK);
+
+ prestera_fw_write(fw, PRESTERA_FW_STATUS_REG, status);
+}
+
static void prestera_fw_evt_work_fn(struct work_struct *work)
{
struct prestera_fw *fw;
fw = container_of(work, struct prestera_fw, evt_work);
msg = fw->evt_msg;
+ prestera_fw_evt_ctl_status_set(fw, PRESTERA_FW_EVT_CTL_STATUS_OFF);
+
while ((qid = prestera_fw_evtq_pick(fw)) < PRESTERA_EVT_QNUM_MAX) {
u32 idx;
u32 len;
if (fw->dev.recv_msg)
fw->dev.recv_msg(&fw->dev, msg, len);
}
+
+ prestera_fw_evt_ctl_status_set(fw, PRESTERA_FW_EVT_CTL_STATUS_ON);
}
static int prestera_fw_wait_reg32(struct prestera_fw *fw, u32 reg, u32 cmp,
static int prestera_fw_rev_check(struct prestera_fw *fw)
{
struct prestera_fw_rev *rev = &fw->dev.fw_rev;
- u16 maj_supp = PRESTERA_SUPP_FW_MAJ_VER;
- u16 min_supp = PRESTERA_SUPP_FW_MIN_VER;
- if (rev->maj == maj_supp && rev->min >= min_supp)
+ if (rev->maj == fw->rev_supp.maj && rev->min >= fw->rev_supp.min)
return 0;
dev_err(fw->dev.dev, "Driver supports FW version only '%u.%u.x'",
- PRESTERA_SUPP_FW_MAJ_VER, PRESTERA_SUPP_FW_MIN_VER);
+ fw->rev_supp.maj, fw->rev_supp.min);
return -EINVAL;
}
-static int prestera_fw_hdr_parse(struct prestera_fw *fw,
- const struct firmware *img)
+static int prestera_fw_hdr_parse(struct prestera_fw *fw)
{
- struct prestera_fw_header *hdr = (struct prestera_fw_header *)img->data;
struct prestera_fw_rev *rev = &fw->dev.fw_rev;
+ struct prestera_fw_header *hdr;
u32 magic;
+ hdr = (struct prestera_fw_header *)fw->bin->data;
+
magic = be32_to_cpu(hdr->magic_number);
if (magic != PRESTERA_FW_HDR_MAGIC) {
dev_err(fw->dev.dev, "FW img hdr magic is invalid");
return prestera_fw_rev_check(fw);
}
+static int prestera_fw_get(struct prestera_fw *fw)
+{
+ int ver_maj = PRESTERA_SUPP_FW_MAJ_VER;
+ int ver_min = PRESTERA_SUPP_FW_MIN_VER;
+ char fw_path[128];
+ int err;
+
+pick_fw_ver:
+ snprintf(fw_path, sizeof(fw_path), PRESTERA_FW_PATH_FMT,
+ ver_maj, ver_min);
+
+ err = request_firmware_direct(&fw->bin, fw_path, fw->dev.dev);
+ if (err) {
+ if (ver_maj == PRESTERA_SUPP_FW_MAJ_VER) {
+ ver_maj = PRESTERA_PREV_FW_MAJ_VER;
+ ver_min = PRESTERA_PREV_FW_MIN_VER;
+
+ dev_warn(fw->dev.dev,
+ "missing latest %s firmware, fall-back to previous %u.%u version\n",
+ fw_path, ver_maj, ver_min);
+
+ goto pick_fw_ver;
+ } else {
+ dev_err(fw->dev.dev, "failed to request previous firmware: %s\n",
+ fw_path);
+ return err;
+ }
+ }
+
+ dev_info(fw->dev.dev, "Loading %s ...", fw_path);
+
+ fw->rev_supp.maj = ver_maj;
+ fw->rev_supp.min = ver_min;
+ fw->rev_supp.sub = 0;
+
+ return 0;
+}
+
+static void prestera_fw_put(struct prestera_fw *fw)
+{
+ release_firmware(fw->bin);
+}
+
static int prestera_fw_load(struct prestera_fw *fw)
{
size_t hlen = sizeof(struct prestera_fw_header);
- const struct firmware *f;
- char fw_path[128];
int err;
err = prestera_ldr_wait_reg32(fw, PRESTERA_LDR_READY_REG,
fw->ldr_wr_idx = 0;
- snprintf(fw_path, sizeof(fw_path), PRESTERA_FW_PATH_FMT,
- PRESTERA_SUPP_FW_MAJ_VER, PRESTERA_SUPP_FW_MIN_VER);
-
- err = request_firmware_direct(&f, fw_path, fw->dev.dev);
- if (err) {
- dev_err(fw->dev.dev, "failed to request firmware file\n");
+ err = prestera_fw_get(fw);
+ if (err)
return err;
- }
- err = prestera_fw_hdr_parse(fw, f);
+ err = prestera_fw_hdr_parse(fw);
if (err) {
dev_err(fw->dev.dev, "FW image header is invalid\n");
goto out_release;
}
- prestera_ldr_write(fw, PRESTERA_LDR_IMG_SIZE_REG, f->size - hlen);
+ prestera_ldr_write(fw, PRESTERA_LDR_IMG_SIZE_REG, fw->bin->size - hlen);
prestera_ldr_write(fw, PRESTERA_LDR_CTL_REG, PRESTERA_LDR_CTL_DL_START);
- dev_info(fw->dev.dev, "Loading %s ...", fw_path);
-
- err = prestera_ldr_fw_send(fw, f->data + hlen, f->size - hlen);
+ err = prestera_ldr_fw_send(fw, fw->bin->data + hlen,
+ fw->bin->size - hlen);
out_release:
- release_firmware(f);
+ prestera_fw_put(fw);
return err;
}
#include "prestera.h"
#include "prestera_hw.h"
#include "prestera_rxtx.h"
+#include "prestera_devlink.h"
#define PRESTERA_SDMA_WAIT_MUL 10
static int prestera_rxtx_process_skb(struct prestera_sdma *sdma,
struct sk_buff *skb)
{
- const struct prestera_port *port;
+ struct prestera_port *port;
struct prestera_dsa dsa;
u32 hw_port, dev_id;
+ u8 cpu_code;
int err;
skb_pull(skb, ETH_HLEN);
__vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), tci);
}
+ cpu_code = dsa.cpu_code;
+ prestera_devlink_trap_report(port, skb, cpu_code);
+
return 0;
}
--- /dev/null
+// SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0
+/* Copyright (c) 2020 Marvell International Ltd. All rights reserved */
+
+#include <linux/kernel.h>
+#include <linux/list.h>
+
+#include "prestera.h"
+#include "prestera_hw.h"
+#include "prestera_acl.h"
+#include "prestera_span.h"
+
+struct prestera_span_entry {
+ struct list_head list;
+ struct prestera_port *port;
+ refcount_t ref_count;
+ u8 id;
+};
+
+struct prestera_span {
+ struct prestera_switch *sw;
+ struct list_head entries;
+};
+
+static struct prestera_span_entry *
+prestera_span_entry_create(struct prestera_port *port, u8 span_id)
+{
+ struct prestera_span_entry *entry;
+
+ entry = kzalloc(sizeof(*entry), GFP_KERNEL);
+ if (!entry)
+ return ERR_PTR(-ENOMEM);
+
+ refcount_set(&entry->ref_count, 1);
+ entry->port = port;
+ entry->id = span_id;
+ list_add_tail(&entry->list, &port->sw->span->entries);
+
+ return entry;
+}
+
+static void prestera_span_entry_del(struct prestera_span_entry *entry)
+{
+ list_del(&entry->list);
+ kfree(entry);
+}
+
+static struct prestera_span_entry *
+prestera_span_entry_find_by_id(struct prestera_span *span, u8 span_id)
+{
+ struct prestera_span_entry *entry;
+
+ list_for_each_entry(entry, &span->entries, list) {
+ if (entry->id == span_id)
+ return entry;
+ }
+
+ return NULL;
+}
+
+static struct prestera_span_entry *
+prestera_span_entry_find_by_port(struct prestera_span *span,
+ struct prestera_port *port)
+{
+ struct prestera_span_entry *entry;
+
+ list_for_each_entry(entry, &span->entries, list) {
+ if (entry->port == port)
+ return entry;
+ }
+
+ return NULL;
+}
+
+static int prestera_span_get(struct prestera_port *port, u8 *span_id)
+{
+ u8 new_span_id;
+ struct prestera_switch *sw = port->sw;
+ struct prestera_span_entry *entry;
+ int err;
+
+ entry = prestera_span_entry_find_by_port(sw->span, port);
+ if (entry) {
+ refcount_inc(&entry->ref_count);
+ *span_id = entry->id;
+ return 0;
+ }
+
+ err = prestera_hw_span_get(port, &new_span_id);
+ if (err)
+ return err;
+
+ entry = prestera_span_entry_create(port, new_span_id);
+ if (IS_ERR(entry)) {
+ prestera_hw_span_release(sw, new_span_id);
+ return PTR_ERR(entry);
+ }
+
+ *span_id = new_span_id;
+ return 0;
+}
+
+static int prestera_span_put(struct prestera_switch *sw, u8 span_id)
+{
+ struct prestera_span_entry *entry;
+ int err;
+
+ entry = prestera_span_entry_find_by_id(sw->span, span_id);
+ if (!entry)
+ return false;
+
+ if (!refcount_dec_and_test(&entry->ref_count))
+ return 0;
+
+ err = prestera_hw_span_release(sw, span_id);
+ if (err)
+ return err;
+
+ prestera_span_entry_del(entry);
+ return 0;
+}
+
+static int prestera_span_rule_add(struct prestera_flow_block_binding *binding,
+ struct prestera_port *to_port)
+{
+ struct prestera_switch *sw = binding->port->sw;
+ u8 span_id;
+ int err;
+
+ if (binding->span_id != PRESTERA_SPAN_INVALID_ID)
+ /* port already in mirroring */
+ return -EEXIST;
+
+ err = prestera_span_get(to_port, &span_id);
+ if (err)
+ return err;
+
+ err = prestera_hw_span_bind(binding->port, span_id);
+ if (err) {
+ prestera_span_put(sw, span_id);
+ return err;
+ }
+
+ binding->span_id = span_id;
+ return 0;
+}
+
+static int prestera_span_rule_del(struct prestera_flow_block_binding *binding)
+{
+ int err;
+
+ err = prestera_hw_span_unbind(binding->port);
+ if (err)
+ return err;
+
+ err = prestera_span_put(binding->port->sw, binding->span_id);
+ if (err)
+ return err;
+
+ binding->span_id = PRESTERA_SPAN_INVALID_ID;
+ return 0;
+}
+
+int prestera_span_replace(struct prestera_flow_block *block,
+ struct tc_cls_matchall_offload *f)
+{
+ struct prestera_flow_block_binding *binding;
+ __be16 protocol = f->common.protocol;
+ struct flow_action_entry *act;
+ struct prestera_port *port;
+ int err;
+
+ if (!flow_offload_has_one_action(&f->rule->action)) {
+ NL_SET_ERR_MSG(f->common.extack,
+ "Only singular actions are supported");
+ return -EOPNOTSUPP;
+ }
+
+ act = &f->rule->action.entries[0];
+
+ if (!prestera_netdev_check(act->dev)) {
+ NL_SET_ERR_MSG(f->common.extack,
+ "Only Marvell Prestera port is supported");
+ return -EINVAL;
+ }
+ if (!tc_cls_can_offload_and_chain0(act->dev, &f->common))
+ return -EOPNOTSUPP;
+ if (act->id != FLOW_ACTION_MIRRED)
+ return -EOPNOTSUPP;
+ if (protocol != htons(ETH_P_ALL))
+ return -EOPNOTSUPP;
+
+ port = netdev_priv(act->dev);
+
+ list_for_each_entry(binding, &block->binding_list, list) {
+ err = prestera_span_rule_add(binding, port);
+ if (err)
+ goto rollback;
+ }
+
+ return 0;
+
+rollback:
+ list_for_each_entry_continue_reverse(binding,
+ &block->binding_list, list)
+ prestera_span_rule_del(binding);
+ return err;
+}
+
+void prestera_span_destroy(struct prestera_flow_block *block)
+{
+ struct prestera_flow_block_binding *binding;
+
+ list_for_each_entry(binding, &block->binding_list, list)
+ prestera_span_rule_del(binding);
+}
+
+int prestera_span_init(struct prestera_switch *sw)
+{
+ struct prestera_span *span;
+
+ span = kzalloc(sizeof(*span), GFP_KERNEL);
+ if (!span)
+ return -ENOMEM;
+
+ INIT_LIST_HEAD(&span->entries);
+
+ sw->span = span;
+ span->sw = sw;
+
+ return 0;
+}
+
+void prestera_span_fini(struct prestera_switch *sw)
+{
+ struct prestera_span *span = sw->span;
+
+ WARN_ON(!list_empty(&span->entries));
+ kfree(span);
+}
--- /dev/null
+/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 */
+/* Copyright (c) 2019-2020 Marvell International Ltd. All rights reserved. */
+
+#ifndef _PRESTERA_SPAN_H_
+#define _PRESTERA_SPAN_H_
+
+#include <net/pkt_cls.h>
+
+#define PRESTERA_SPAN_INVALID_ID -1
+
+struct prestera_switch;
+struct prestera_flow_block;
+
+int prestera_span_init(struct prestera_switch *sw);
+void prestera_span_fini(struct prestera_switch *sw);
+int prestera_span_replace(struct prestera_flow_block *block,
+ struct tc_cls_matchall_offload *f);
+void prestera_span_destroy(struct prestera_flow_block *block);
+
+#endif /* _PRESTERA_SPAN_H_ */
return ERR_PTR(err);
}
+static int prestera_fdb_add(struct prestera_port *port,
+ const unsigned char *mac, u16 vid, bool dynamic)
+{
+ if (prestera_port_is_lag_member(port))
+ return prestera_hw_lag_fdb_add(port->sw, prestera_port_lag_id(port),
+ mac, vid, dynamic);
+
+ return prestera_hw_fdb_add(port, mac, vid, dynamic);
+}
+
+static int prestera_fdb_del(struct prestera_port *port,
+ const unsigned char *mac, u16 vid)
+{
+ if (prestera_port_is_lag_member(port))
+ return prestera_hw_lag_fdb_del(port->sw, prestera_port_lag_id(port),
+ mac, vid);
+ else
+ return prestera_hw_fdb_del(port, mac, vid);
+}
+
+static int prestera_fdb_flush_port_vlan(struct prestera_port *port, u16 vid,
+ u32 mode)
+{
+ if (prestera_port_is_lag_member(port))
+ return prestera_hw_fdb_flush_lag_vlan(port->sw, prestera_port_lag_id(port),
+ vid, mode);
+ else
+ return prestera_hw_fdb_flush_port_vlan(port, vid, mode);
+}
+
+static int prestera_fdb_flush_port(struct prestera_port *port, u32 mode)
+{
+ if (prestera_port_is_lag_member(port))
+ return prestera_hw_fdb_flush_lag(port->sw, prestera_port_lag_id(port),
+ mode);
+ else
+ return prestera_hw_fdb_flush_port(port, mode);
+}
+
static void
prestera_port_vlan_bridge_leave(struct prestera_port_vlan *port_vlan)
{
last_port = port_count == 1;
if (last_vlan)
- prestera_hw_fdb_flush_port(port, fdb_flush_mode);
+ prestera_fdb_flush_port(port, fdb_flush_mode);
else if (last_port)
prestera_hw_fdb_flush_vlan(port->sw, vid, fdb_flush_mode);
else
- prestera_hw_fdb_flush_port_vlan(port, vid, fdb_flush_mode);
+ prestera_fdb_flush_port_vlan(port, vid, fdb_flush_mode);
list_del(&port_vlan->br_vlan_head);
prestera_bridge_vlan_put(br_vlan);
return NULL;
}
+static int prestera_match_upper_bridge_dev(struct net_device *dev,
+ struct netdev_nested_priv *priv)
+{
+ if (netif_is_bridge_master(dev))
+ priv->data = dev;
+
+ return 0;
+}
+
+static struct net_device *prestera_get_upper_bridge_dev(struct net_device *dev)
+{
+ struct netdev_nested_priv priv = { };
+
+ netdev_walk_all_upper_dev_rcu(dev, prestera_match_upper_bridge_dev,
+ &priv);
+ return priv.data;
+}
+
static struct prestera_bridge_port *
prestera_bridge_port_by_dev(struct prestera_switchdev *swdev,
struct net_device *dev)
{
- struct net_device *br_dev = netdev_master_upper_dev_get(dev);
+ struct net_device *br_dev = prestera_get_upper_bridge_dev(dev);
struct prestera_bridge *bridge;
if (!br_dev)
if (err)
return err;
- err = prestera_hw_port_flood_set(port, br_port->flags & BR_FLOOD);
+ err = prestera_hw_port_flood_set(port, BR_FLOOD | BR_MCAST_FLOOD,
+ br_port->flags);
if (err)
goto err_port_flood_set;
return 0;
err_port_learning_set:
- prestera_hw_port_flood_set(port, false);
err_port_flood_set:
prestera_hw_bridge_port_delete(port, bridge->bridge_id);
return err;
}
-static int prestera_port_bridge_join(struct prestera_port *port,
- struct net_device *upper)
+int prestera_bridge_port_join(struct net_device *br_dev,
+ struct prestera_port *port)
{
struct prestera_switchdev *swdev = port->sw->swdev;
struct prestera_bridge_port *br_port;
struct prestera_bridge *bridge;
int err;
- bridge = prestera_bridge_by_dev(swdev, upper);
+ bridge = prestera_bridge_by_dev(swdev, br_dev);
if (!bridge) {
- bridge = prestera_bridge_create(swdev, upper);
+ bridge = prestera_bridge_create(swdev, br_dev);
if (IS_ERR(bridge))
return PTR_ERR(bridge);
}
return prestera_hw_vlan_port_stp_set(port, vid, hw_state);
}
-static void prestera_port_bridge_leave(struct prestera_port *port,
- struct net_device *upper)
+void prestera_bridge_port_leave(struct net_device *br_dev,
+ struct prestera_port *port)
{
struct prestera_switchdev *swdev = port->sw->swdev;
struct prestera_bridge_port *br_port;
struct prestera_bridge *bridge;
- bridge = prestera_bridge_by_dev(swdev, upper);
+ bridge = prestera_bridge_by_dev(swdev, br_dev);
if (!bridge)
return;
prestera_bridge_1d_port_leave(br_port);
prestera_hw_port_learning_set(port, false);
- prestera_hw_port_flood_set(port, false);
+ prestera_hw_port_flood_set(port, BR_FLOOD | BR_MCAST_FLOOD, 0);
prestera_port_vid_stp_set(port, PRESTERA_VID_ALL, BR_STATE_FORWARDING);
prestera_bridge_port_put(br_port);
}
-int prestera_bridge_port_event(struct net_device *dev, unsigned long event,
- void *ptr)
-{
- struct netdev_notifier_changeupper_info *info = ptr;
- struct netlink_ext_ack *extack;
- struct prestera_port *port;
- struct net_device *upper;
- int err;
-
- extack = netdev_notifier_info_to_extack(&info->info);
- port = netdev_priv(dev);
- upper = info->upper_dev;
-
- switch (event) {
- case NETDEV_PRECHANGEUPPER:
- if (!netif_is_bridge_master(upper)) {
- NL_SET_ERR_MSG_MOD(extack, "Unknown upper device type");
- return -EINVAL;
- }
-
- if (!info->linking)
- break;
-
- if (netdev_has_any_upper_dev(upper)) {
- NL_SET_ERR_MSG_MOD(extack, "Upper device is already enslaved");
- return -EINVAL;
- }
- break;
-
- case NETDEV_CHANGEUPPER:
- if (!netif_is_bridge_master(upper))
- break;
-
- if (info->linking) {
- err = prestera_port_bridge_join(port, upper);
- if (err)
- return err;
- } else {
- prestera_port_bridge_leave(port, upper);
- }
- break;
- }
-
- return 0;
-}
-
static int prestera_port_attr_br_flags_set(struct prestera_port *port,
struct net_device *dev,
struct switchdev_brport_flags flags)
if (!br_port)
return 0;
- if (flags.mask & BR_FLOOD) {
- err = prestera_hw_port_flood_set(port, flags.val & BR_FLOOD);
- if (err)
- return err;
- }
+ err = prestera_hw_port_flood_set(port, flags.mask, flags.val);
+ if (err)
+ return err;
if (flags.mask & BR_LEARNING) {
err = prestera_hw_port_learning_set(port,
vid = bridge->bridge_id;
if (adding)
- err = prestera_hw_fdb_add(port, fdb_info->addr, vid, false);
+ err = prestera_fdb_add(port, fdb_info->addr, vid, false);
else
- err = prestera_hw_fdb_del(port, fdb_info->addr, vid);
+ err = prestera_fdb_del(port, fdb_info->addr, vid);
return err;
}
if (port_vlan->br_port)
return 0;
- err = prestera_hw_port_flood_set(port, br_port->flags & BR_FLOOD);
+ err = prestera_hw_port_flood_set(port, BR_FLOOD | BR_MCAST_FLOOD,
+ br_port->flags);
if (err)
return err;
{
bool flag_untagged = vlan->flags & BRIDGE_VLAN_INFO_UNTAGGED;
bool flag_pvid = vlan->flags & BRIDGE_VLAN_INFO_PVID;
- struct net_device *dev = vlan->obj.orig_dev;
+ struct net_device *orig_dev = vlan->obj.orig_dev;
struct prestera_bridge_port *br_port;
struct prestera_switch *sw = port->sw;
struct prestera_bridge *bridge;
- if (netif_is_bridge_master(dev))
+ if (netif_is_bridge_master(orig_dev))
return 0;
- br_port = prestera_bridge_port_by_dev(sw->swdev, dev);
+ br_port = prestera_bridge_port_by_dev(sw->swdev, port->dev);
if (WARN_ON(!br_port))
return -EINVAL;
static int prestera_port_vlans_del(struct prestera_port *port,
const struct switchdev_obj_port_vlan *vlan)
{
- struct net_device *dev = vlan->obj.orig_dev;
+ struct net_device *orig_dev = vlan->obj.orig_dev;
struct prestera_bridge_port *br_port;
struct prestera_switch *sw = port->sw;
- if (netif_is_bridge_master(dev))
+ if (netif_is_bridge_master(orig_dev))
return -EOPNOTSUPP;
- br_port = prestera_bridge_port_by_dev(sw->swdev, dev);
+ br_port = prestera_bridge_port_by_dev(sw->swdev, port->dev);
if (WARN_ON(!br_port))
return -EINVAL;
struct prestera_event *evt, void *arg)
{
struct switchdev_notifier_fdb_info info;
+ struct net_device *dev = NULL;
struct prestera_port *port;
+ struct prestera_lag *lag;
- port = prestera_find_port(sw, evt->fdb_evt.port_id);
- if (!port)
+ switch (evt->fdb_evt.type) {
+ case PRESTERA_FDB_ENTRY_TYPE_REG_PORT:
+ port = prestera_find_port(sw, evt->fdb_evt.dest.port_id);
+ if (port)
+ dev = port->dev;
+ break;
+ case PRESTERA_FDB_ENTRY_TYPE_LAG:
+ lag = prestera_lag_by_id(sw, evt->fdb_evt.dest.lag_id);
+ if (lag)
+ dev = lag->dev;
+ break;
+ default:
+ return;
+ }
+
+ if (!dev)
return;
info.addr = evt->fdb_evt.data.mac;
switch (evt->id) {
case PRESTERA_FDB_EVENT_LEARNED:
call_switchdev_notifiers(SWITCHDEV_FDB_ADD_TO_BRIDGE,
- port->dev, &info.info, NULL);
+ dev, &info.info, NULL);
break;
case PRESTERA_FDB_EVENT_AGED:
call_switchdev_notifiers(SWITCHDEV_FDB_DEL_TO_BRIDGE,
- port->dev, &info.info, NULL);
+ dev, &info.info, NULL);
break;
}
int prestera_switchdev_init(struct prestera_switch *sw);
void prestera_switchdev_fini(struct prestera_switch *sw);
-int prestera_bridge_port_event(struct net_device *dev, unsigned long event,
- void *ptr);
+int prestera_bridge_port_join(struct net_device *br_dev,
+ struct prestera_port *port);
+
+void prestera_bridge_port_leave(struct net_device *br_dev,
+ struct prestera_port *port);
#endif /* _PRESTERA_SWITCHDEV_H_ */
struct net_device *dev = platform_get_drvdata(pdev);
struct pxa168_eth_private *pep = netdev_priv(dev);
+ cancel_work_sync(&pep->tx_timeout_task);
if (pep->htpr) {
dma_free_coherent(pep->dev->dev.parent, HASH_ADDR_TABLE_SIZE,
pep->htpr, pep->htpr_dma);
clk_disable_unprepare(pep->clk);
mdiobus_unregister(pep->smi_bus);
mdiobus_free(pep->smi_bus);
- cancel_work_sync(&pep->tx_timeout_task);
unregister_netdev(dev);
free_netdev(dev);
return 0;
CHIP_ID_YUKON_LP = 0xb2, /* Chip ID for YUKON-LP */
CHIP_ID_YUKON_XL = 0xb3, /* Chip ID for YUKON-2 XL */
CHIP_ID_YUKON_EC = 0xb6, /* Chip ID for YUKON-2 EC */
- CHIP_ID_YUKON_FE = 0xb7, /* Chip ID for YUKON-2 FE */
+ CHIP_ID_YUKON_FE = 0xb7, /* Chip ID for YUKON-2 FE */
CHIP_REV_YU_LITE_A1 = 3, /* Chip Rev. for YUKON-Lite A1,A2 */
CHIP_REV_YU_LITE_A3 = 7, /* Chip Rev. for YUKON-Lite A3 */
adv |= fiber_fc_adv[sky2->flow_mode];
} else {
reg |= GM_GPCR_AU_FCT_DIS;
- reg |= gm_fc_disable[sky2->flow_mode];
+ reg |= gm_fc_disable[sky2->flow_mode];
/* Forward pause packets to GMAC? */
if (sky2->flow_mode & FC_RX)
tx_init(sky2);
/*
- * On dual port PCI-X card, there is an problem where status
+ * On dual port PCI-X card, there is an problem where status
* can be received out of order due to split transactions
*/
if (otherdev && netif_running(otherdev) &&
- (cap = pci_find_capability(hw->pdev, PCI_CAP_ID_PCIX))) {
- u16 cmd;
+ (cap = pci_find_capability(hw->pdev, PCI_CAP_ID_PCIX))) {
+ u16 cmd;
cmd = sky2_pci_read16(hw, cap + PCI_X_CMD);
- cmd &= ~PCI_X_CMD_MAX_SPLIT;
- sky2_pci_write16(hw, cap + PCI_X_CMD, cmd);
+ cmd &= ~PCI_X_CMD_MAX_SPLIT;
+ sky2_pci_write16(hw, cap + PCI_X_CMD, cmd);
}
sky2_mac_init(hw, port);
u16 mss;
u8 ctrl;
- if (unlikely(tx_avail(sky2) < tx_le_req(skb)))
- return NETDEV_TX_BUSY;
+ if (unlikely(tx_avail(sky2) < tx_le_req(skb)))
+ return NETDEV_TX_BUSY;
len = skb_headlen(skb);
mapping = dma_map_single(&hw->pdev->dev, skb->data, len,
if (!(hw->flags & SKY2_HW_NEW_LE))
mss += ETH_HLEN + ip_hdrlen(skb) + tcp_hdrlen(skb);
- if (mss != sky2->tx_last_mss) {
+ if (mss != sky2->tx_last_mss) {
le = get_tx_le(sky2, &slot);
- le->addr = cpu_to_le32(mss);
+ le->addr = cpu_to_le32(mss);
if (hw->flags & SKY2_HW_NEW_LE)
le->opcode = OP_MSS | HW_OWNER;
/* Handle TCP checksum offload */
if (skb->ip_summed == CHECKSUM_PARTIAL) {
/* On Yukon EX (some versions) encoding change. */
- if (hw->flags & SKY2_HW_AUTO_TX_SUM)
- ctrl |= CALSUM; /* auto checksum */
+ if (hw->flags & SKY2_HW_AUTO_TX_SUM)
+ ctrl |= CALSUM; /* auto checksum */
else {
const unsigned offset = skb_transport_offset(skb);
u32 tcpsum;
if (length == 0) {
/* don't need this page */
- __skb_frag_unref(frag);
+ __skb_frag_unref(frag, false);
--skb_shinfo(skb)->nr_frags;
} else {
size = min(length, (unsigned) PAGE_SIZE);
static struct sk_buff *sky2_receive(struct net_device *dev,
u16 length, u32 status)
{
- struct sky2_port *sky2 = netdev_priv(dev);
+ struct sky2_port *sky2 = netdev_priv(dev);
struct rx_ring_info *re = sky2->rx_ring + sky2->rx_next;
struct sk_buff *skb = NULL;
u16 count = (status & GMR_FS_LEN) >> 16;
if (!disable_msi && pci_enable_msi(pdev) == 0) {
err = sky2_test_msi(hw);
if (err) {
- pci_disable_msi(pdev);
+ pci_disable_msi(pdev);
if (err != -EOPNOTSUPP)
goto err_out_free_netdev;
}
- }
+ }
netif_napi_add(dev, &hw->napi, sky2_poll, NAPI_WEIGHT);
CHIP_ID_YUKON_EC_U = 0xb4, /* YUKON-2 EC Ultra */
CHIP_ID_YUKON_EX = 0xb5, /* YUKON-2 Extreme */
CHIP_ID_YUKON_EC = 0xb6, /* YUKON-2 EC */
- CHIP_ID_YUKON_FE = 0xb7, /* YUKON-2 FE */
- CHIP_ID_YUKON_FE_P = 0xb8, /* YUKON-2 FE+ */
+ CHIP_ID_YUKON_FE = 0xb7, /* YUKON-2 FE */
+ CHIP_ID_YUKON_FE_P = 0xb8, /* YUKON-2 FE+ */
CHIP_ID_YUKON_SUPR = 0xb9, /* YUKON-2 Supreme */
CHIP_ID_YUKON_UL_2 = 0xba, /* YUKON-2 Ultra 2 */
CHIP_ID_YUKON_OPT = 0xbc, /* YUKON-2 Optima */
#define SKY2_FLAG_AUTO_SPEED 0x0002
#define SKY2_FLAG_AUTO_PAUSE 0x0004
- enum flow_control flow_mode;
- enum flow_control flow_status;
+ enum flow_control flow_mode;
+ enum flow_control flow_status;
#ifdef CONFIG_SKY2_DEBUG
struct dentry *debugfs;
void mtk_stats_update_mac(struct mtk_mac *mac)
{
struct mtk_hw_stats *hw_stats = mac->hw_stats;
- unsigned int base = MTK_GDM1_TX_GBCNT;
- u64 stats;
-
- base += hw_stats->reg_offset;
+ struct mtk_eth *eth = mac->hw;
u64_stats_update_begin(&hw_stats->syncp);
- hw_stats->rx_bytes += mtk_r32(mac->hw, base);
- stats = mtk_r32(mac->hw, base + 0x04);
- if (stats)
- hw_stats->rx_bytes += (stats << 32);
- hw_stats->rx_packets += mtk_r32(mac->hw, base + 0x08);
- hw_stats->rx_overflow += mtk_r32(mac->hw, base + 0x10);
- hw_stats->rx_fcs_errors += mtk_r32(mac->hw, base + 0x14);
- hw_stats->rx_short_errors += mtk_r32(mac->hw, base + 0x18);
- hw_stats->rx_long_errors += mtk_r32(mac->hw, base + 0x1c);
- hw_stats->rx_checksum_errors += mtk_r32(mac->hw, base + 0x20);
- hw_stats->rx_flow_control_packets +=
- mtk_r32(mac->hw, base + 0x24);
- hw_stats->tx_skip += mtk_r32(mac->hw, base + 0x28);
- hw_stats->tx_collisions += mtk_r32(mac->hw, base + 0x2c);
- hw_stats->tx_bytes += mtk_r32(mac->hw, base + 0x30);
- stats = mtk_r32(mac->hw, base + 0x34);
- if (stats)
- hw_stats->tx_bytes += (stats << 32);
- hw_stats->tx_packets += mtk_r32(mac->hw, base + 0x38);
+ if (MTK_HAS_CAPS(eth->soc->caps, MTK_SOC_MT7628)) {
+ hw_stats->tx_packets += mtk_r32(mac->hw, MT7628_SDM_TPCNT);
+ hw_stats->tx_bytes += mtk_r32(mac->hw, MT7628_SDM_TBCNT);
+ hw_stats->rx_packets += mtk_r32(mac->hw, MT7628_SDM_RPCNT);
+ hw_stats->rx_bytes += mtk_r32(mac->hw, MT7628_SDM_RBCNT);
+ hw_stats->rx_checksum_errors +=
+ mtk_r32(mac->hw, MT7628_SDM_CS_ERR);
+ } else {
+ unsigned int offs = hw_stats->reg_offset;
+ u64 stats;
+
+ hw_stats->rx_bytes += mtk_r32(mac->hw,
+ MTK_GDM1_RX_GBCNT_L + offs);
+ stats = mtk_r32(mac->hw, MTK_GDM1_RX_GBCNT_H + offs);
+ if (stats)
+ hw_stats->rx_bytes += (stats << 32);
+ hw_stats->rx_packets +=
+ mtk_r32(mac->hw, MTK_GDM1_RX_GPCNT + offs);
+ hw_stats->rx_overflow +=
+ mtk_r32(mac->hw, MTK_GDM1_RX_OERCNT + offs);
+ hw_stats->rx_fcs_errors +=
+ mtk_r32(mac->hw, MTK_GDM1_RX_FERCNT + offs);
+ hw_stats->rx_short_errors +=
+ mtk_r32(mac->hw, MTK_GDM1_RX_SERCNT + offs);
+ hw_stats->rx_long_errors +=
+ mtk_r32(mac->hw, MTK_GDM1_RX_LENCNT + offs);
+ hw_stats->rx_checksum_errors +=
+ mtk_r32(mac->hw, MTK_GDM1_RX_CERCNT + offs);
+ hw_stats->rx_flow_control_packets +=
+ mtk_r32(mac->hw, MTK_GDM1_RX_FCCNT + offs);
+ hw_stats->tx_skip +=
+ mtk_r32(mac->hw, MTK_GDM1_TX_SKIPCNT + offs);
+ hw_stats->tx_collisions +=
+ mtk_r32(mac->hw, MTK_GDM1_TX_COLCNT + offs);
+ hw_stats->tx_bytes +=
+ mtk_r32(mac->hw, MTK_GDM1_TX_GBCNT_L + offs);
+ stats = mtk_r32(mac->hw, MTK_GDM1_TX_GBCNT_H + offs);
+ if (stats)
+ hw_stats->tx_bytes += (stats << 32);
+ hw_stats->tx_packets +=
+ mtk_r32(mac->hw, MTK_GDM1_TX_GPCNT + offs);
+ }
+
u64_stats_update_end(&hw_stats->syncp);
}
val |= cur << MTK_PDMA_DELAY_RX_PINT_SHIFT;
mtk_w32(eth, val, MTK_PDMA_DELAY_INT);
- mtk_w32(eth, val, MTK_QDMA_DELAY_INT);
+ if (MTK_HAS_CAPS(eth->soc->caps, MTK_QDMA))
+ mtk_w32(eth, val, MTK_QDMA_DELAY_INT);
spin_unlock_bh(ð->dim_lock);
val |= cur << MTK_PDMA_DELAY_TX_PINT_SHIFT;
mtk_w32(eth, val, MTK_PDMA_DELAY_INT);
- mtk_w32(eth, val, MTK_QDMA_DELAY_INT);
+ if (MTK_HAS_CAPS(eth->soc->caps, MTK_QDMA))
+ mtk_w32(eth, val, MTK_QDMA_DELAY_INT);
spin_unlock_bh(ð->dim_lock);
goto err_disable_pm;
}
+ /* set interrupt delays based on current Net DIM sample */
+ mtk_dim_rx(ð->rx_dim.work);
+ mtk_dim_tx(ð->tx_dim.work);
+
/* disable delay and normal interrupt */
mtk_tx_irq_disable(eth, ~0);
mtk_rx_irq_disable(eth, ~0);
/* QDMA FQ Free Page Buffer Length Register */
#define MTK_QDMA_FQ_BLEN 0x1B2C
-/* GMA1 Received Good Byte Count Register */
-#define MTK_GDM1_TX_GBCNT 0x2400
+/* GMA1 counter / statics register */
+#define MTK_GDM1_RX_GBCNT_L 0x2400
+#define MTK_GDM1_RX_GBCNT_H 0x2404
+#define MTK_GDM1_RX_GPCNT 0x2408
+#define MTK_GDM1_RX_OERCNT 0x2410
+#define MTK_GDM1_RX_FERCNT 0x2414
+#define MTK_GDM1_RX_SERCNT 0x2418
+#define MTK_GDM1_RX_LENCNT 0x241c
+#define MTK_GDM1_RX_CERCNT 0x2420
+#define MTK_GDM1_RX_FCCNT 0x2424
+#define MTK_GDM1_TX_SKIPCNT 0x2428
+#define MTK_GDM1_TX_COLCNT 0x242c
+#define MTK_GDM1_TX_GBCNT_L 0x2430
+#define MTK_GDM1_TX_GBCNT_H 0x2434
+#define MTK_GDM1_TX_GPCNT 0x2438
#define MTK_STAT_OFFSET 0x40
/* QDMA descriptor txd4 */
#define MT7628_SDM_MAC_ADRL (MT7628_SDM_OFFSET + 0x0c)
#define MT7628_SDM_MAC_ADRH (MT7628_SDM_OFFSET + 0x10)
+/* Counter / stat register */
+#define MT7628_SDM_TPCNT (MT7628_SDM_OFFSET + 0x100)
+#define MT7628_SDM_TBCNT (MT7628_SDM_OFFSET + 0x104)
+#define MT7628_SDM_RPCNT (MT7628_SDM_OFFSET + 0x108)
+#define MT7628_SDM_RBCNT (MT7628_SDM_OFFSET + 0x10c)
+#define MT7628_SDM_CS_ERR (MT7628_SDM_OFFSET + 0x110)
+
struct mtk_rx_dma {
unsigned int rxd1;
unsigned int rxd2;
source "drivers/net/ethernet/mellanox/mlx5/core/Kconfig"
source "drivers/net/ethernet/mellanox/mlxsw/Kconfig"
source "drivers/net/ethernet/mellanox/mlxfw/Kconfig"
+source "drivers/net/ethernet/mellanox/mlxbf_gige/Kconfig"
endif # NET_VENDOR_MELLANOX
obj-$(CONFIG_MLX5_CORE) += mlx5/core/
obj-$(CONFIG_MLXSW_CORE) += mlxsw/
obj-$(CONFIG_MLXFW) += mlxfw/
+obj-$(CONFIG_MLXBF_GIGE) += mlxbf_gige/
return ret;
}
-#define MLX4_EEPROM_PAGE_LEN 256
-
static int mlx4_en_get_module_info(struct net_device *dev,
struct ethtool_modinfo *modinfo)
{
break;
case MLX4_MODULE_ID_SFP:
modinfo->type = ETH_MODULE_SFF_8472;
- modinfo->eeprom_len = MLX4_EEPROM_PAGE_LEN;
+ modinfo->eeprom_len = ETH_MODULE_SFF_8472_LEN;
break;
default:
return -EINVAL;
fail:
while (nr > 0) {
nr--;
- __skb_frag_unref(skb_shinfo(skb)->frags + nr);
+ __skb_frag_unref(skb_shinfo(skb)->frags + nr, false);
}
return 0;
}
#define QUERY_DEV_CAP_MAD_DEMUX_OFFSET 0xb0
#define QUERY_DEV_CAP_DMFS_HIGH_RATE_QPN_BASE_OFFSET 0xa8
#define QUERY_DEV_CAP_DMFS_HIGH_RATE_QPN_RANGE_OFFSET 0xac
+#define QUERY_DEV_CAP_MAP_CLOCK_TO_USER 0xc1
#define QUERY_DEV_CAP_QP_RATE_LIMIT_NUM_OFFSET 0xcc
#define QUERY_DEV_CAP_QP_RATE_LIMIT_MAX_OFFSET 0xd0
#define QUERY_DEV_CAP_QP_RATE_LIMIT_MIN_OFFSET 0xd2
if (mlx4_is_mfunc(dev))
disable_unsupported_roce_caps(outbox);
+ MLX4_GET(field, outbox, QUERY_DEV_CAP_MAP_CLOCK_TO_USER);
+ dev_cap->map_clock_to_user = field & 0x80;
MLX4_GET(field, outbox, QUERY_DEV_CAP_RSVD_QP_OFFSET);
dev_cap->reserved_qps = 1 << (field & 0xf);
MLX4_GET(field, outbox, QUERY_DEV_CAP_MAX_QP_OFFSET);
u32 health_buffer_addrs;
struct mlx4_port_cap port_cap[MLX4_MAX_PORTS + 1];
bool wol_port[MLX4_MAX_PORTS + 1];
+ bool map_clock_to_user;
};
struct mlx4_func_cap {
}
}
+ dev->caps.map_clock_to_user = dev_cap->map_clock_to_user;
dev->caps.uar_page_size = PAGE_SIZE;
dev->caps.num_uars = dev_cap->uar_size / PAGE_SIZE;
dev->caps.local_ca_ack_delay = dev_cap->local_ca_ack_delay;
if (mlx4_is_slave(dev))
return -EOPNOTSUPP;
+ if (!dev->caps.map_clock_to_user) {
+ mlx4_dbg(dev, "Map clock to user is not supported.\n");
+ return -EOPNOTSUPP;
+ }
+
if (!params)
return -EINVAL;
#define I2C_ADDR_LOW 0x50
#define I2C_ADDR_HIGH 0x51
#define I2C_PAGE_SIZE 256
+#define I2C_HIGH_PAGE_SIZE 128
/* Module Info Data */
struct mlx4_cable_info {
return "Unknown Error";
}
+static int mlx4_get_module_id(struct mlx4_dev *dev, u8 port, u8 *module_id)
+{
+ struct mlx4_cmd_mailbox *inbox, *outbox;
+ struct mlx4_mad_ifc *inmad, *outmad;
+ struct mlx4_cable_info *cable_info;
+ int ret;
+
+ inbox = mlx4_alloc_cmd_mailbox(dev);
+ if (IS_ERR(inbox))
+ return PTR_ERR(inbox);
+
+ outbox = mlx4_alloc_cmd_mailbox(dev);
+ if (IS_ERR(outbox)) {
+ mlx4_free_cmd_mailbox(dev, inbox);
+ return PTR_ERR(outbox);
+ }
+
+ inmad = (struct mlx4_mad_ifc *)(inbox->buf);
+ outmad = (struct mlx4_mad_ifc *)(outbox->buf);
+
+ inmad->method = 0x1; /* Get */
+ inmad->class_version = 0x1;
+ inmad->mgmt_class = 0x1;
+ inmad->base_version = 0x1;
+ inmad->attr_id = cpu_to_be16(0xFF60); /* Module Info */
+
+ cable_info = (struct mlx4_cable_info *)inmad->data;
+ cable_info->dev_mem_address = 0;
+ cable_info->page_num = 0;
+ cable_info->i2c_addr = I2C_ADDR_LOW;
+ cable_info->size = cpu_to_be16(1);
+
+ ret = mlx4_cmd_box(dev, inbox->dma, outbox->dma, port, 3,
+ MLX4_CMD_MAD_IFC, MLX4_CMD_TIME_CLASS_C,
+ MLX4_CMD_NATIVE);
+ if (ret)
+ goto out;
+
+ if (be16_to_cpu(outmad->status)) {
+ /* Mad returned with bad status */
+ ret = be16_to_cpu(outmad->status);
+ mlx4_warn(dev,
+ "MLX4_CMD_MAD_IFC Get Module ID attr(%x) port(%d) i2c_addr(%x) offset(%d) size(%d): Response Mad Status(%x) - %s\n",
+ 0xFF60, port, I2C_ADDR_LOW, 0, 1, ret,
+ cable_info_mad_err_str(ret));
+ ret = -ret;
+ goto out;
+ }
+ cable_info = (struct mlx4_cable_info *)outmad->data;
+ *module_id = cable_info->data[0];
+out:
+ mlx4_free_cmd_mailbox(dev, inbox);
+ mlx4_free_cmd_mailbox(dev, outbox);
+ return ret;
+}
+
+static void mlx4_sfp_eeprom_params_set(u8 *i2c_addr, u8 *page_num, u16 *offset)
+{
+ *i2c_addr = I2C_ADDR_LOW;
+ *page_num = 0;
+
+ if (*offset < I2C_PAGE_SIZE)
+ return;
+
+ *i2c_addr = I2C_ADDR_HIGH;
+ *offset -= I2C_PAGE_SIZE;
+}
+
+static void mlx4_qsfp_eeprom_params_set(u8 *i2c_addr, u8 *page_num, u16 *offset)
+{
+ /* Offsets 0-255 belong to page 0.
+ * Offsets 256-639 belong to pages 01, 02, 03.
+ * For example, offset 400 is page 02: 1 + (400 - 256) / 128 = 2
+ */
+ if (*offset < I2C_PAGE_SIZE)
+ *page_num = 0;
+ else
+ *page_num = 1 + (*offset - I2C_PAGE_SIZE) / I2C_HIGH_PAGE_SIZE;
+ *i2c_addr = I2C_ADDR_LOW;
+ *offset -= *page_num * I2C_HIGH_PAGE_SIZE;
+}
+
/**
* mlx4_get_module_info - Read cable module eeprom data
* @dev: mlx4_dev.
struct mlx4_cmd_mailbox *inbox, *outbox;
struct mlx4_mad_ifc *inmad, *outmad;
struct mlx4_cable_info *cable_info;
- u16 i2c_addr;
+ u8 module_id, i2c_addr, page_num;
int ret;
if (size > MODULE_INFO_MAX_READ)
size = MODULE_INFO_MAX_READ;
+ ret = mlx4_get_module_id(dev, port, &module_id);
+ if (ret)
+ return ret;
+
+ switch (module_id) {
+ case MLX4_MODULE_ID_SFP:
+ mlx4_sfp_eeprom_params_set(&i2c_addr, &page_num, &offset);
+ break;
+ case MLX4_MODULE_ID_QSFP:
+ case MLX4_MODULE_ID_QSFP_PLUS:
+ case MLX4_MODULE_ID_QSFP28:
+ mlx4_qsfp_eeprom_params_set(&i2c_addr, &page_num, &offset);
+ break;
+ default:
+ mlx4_err(dev, "Module ID not recognized: %#x\n", module_id);
+ return -EINVAL;
+ }
+
inbox = mlx4_alloc_cmd_mailbox(dev);
if (IS_ERR(inbox))
return PTR_ERR(inbox);
*/
size -= offset + size - I2C_PAGE_SIZE;
- i2c_addr = I2C_ADDR_LOW;
-
cable_info = (struct mlx4_cable_info *)inmad->data;
cable_info->dev_mem_address = cpu_to_be16(offset);
- cable_info->page_num = 0;
+ cable_info->page_num = page_num;
cable_info->i2c_addr = i2c_addr;
cable_info->size = cpu_to_be16(size);
depends on MLXFW || !MLXFW
depends on PTP_1588_CLOCK || !PTP_1588_CLOCK
depends on PCI_HYPERV_INTERFACE || !PCI_HYPERV_INTERFACE
- default n
help
Core driver for low level functionality of the ConnectX-4 and
Connect-IB cards by Mellanox Technologies.
depends on NETDEVICES && ETHERNET && INET && PCI && MLX5_CORE
select PAGE_POOL
select DIMLIB
- default n
help
Ethernet support in Mellanox Technologies ConnectX-4 NIC.
Legacy SRIOV mode (L2 mac vlan steering based).
Switchdev mode (eswitch offloads).
+config MLX5_BRIDGE
+ bool
+ depends on MLX5_ESWITCH && BRIDGE
+ default y
+ help
+ mlx5 ConnectX offloads support for Ethernet Bridging (BRIDGE).
+ Enable adding representors of mlx5 uplink and VF ports to Bridge and
+ offloading rules for traffic between such ports. Supports VLANs (trunk and
+ access modes).
+
config MLX5_CLS_ACT
bool "MLX5 TC classifier action support"
depends on MLX5_ESWITCH && NET_CLS_ACT
config MLX5_CORE_IPOIB
bool "Mellanox 5th generation network adapters (connectX series) IPoIB offloads support"
depends on MLX5_CORE_EN
- default n
help
MLX5 IPoIB offloads & acceleration support.
bool "Mellanox Technologies IPsec Innova support"
depends on MLX5_CORE
depends on MLX5_FPGA
- default n
help
Build IPsec support for the Innova family of network cards by Mellanox
Technologies. Innova network cards are comprised of a ConnectX chip
depends on XFRM_OFFLOAD
depends on INET_ESP_OFFLOAD || INET6_ESP_OFFLOAD
select MLX5_ACCEL
- default n
help
Build IPsec support for the Connect-X family of network cards by Mellanox
Technologies.
depends on XFRM_OFFLOAD
depends on INET_ESP_OFFLOAD || INET6_ESP_OFFLOAD
depends on MLX5_FPGA_IPSEC || MLX5_IPSEC
- default n
help
Build support for IPsec cryptography-offload acceleration in the NIC.
Note: Support for hardware with this capability needs to be selected
depends on MLX5_CORE_EN
depends on MLX5_FPGA
select MLX5_EN_TLS
- default n
help
Build TLS support for the Innova family of network cards by Mellanox
Technologies. Innova network cards are comprised of a ConnectX chip
depends on MLX5_CORE_EN
select MLX5_ACCEL
select MLX5_EN_TLS
- default n
help
Build TLS support for the Connect-X family of network cards by Mellanox
Technologies.
config MLX5_SF
bool "Mellanox Technologies subfunction device support using auxiliary device"
depends on MLX5_CORE && MLX5_CORE_EN
- default n
help
Build support for subfuction device in the NIC. A Mellanox subfunction
device can support RDMA, netdevice and vdpa device.
mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \
health.o mcg.o cq.o alloc.o port.o mr.o pd.o \
transobj.o vport.o sriov.o fs_cmd.o fs_core.o pci_irq.o \
- fs_counters.o rl.o lag.o dev.o events.o wq.o lib/gid.o \
+ fs_counters.o fs_ft_pool.o rl.o lag.o dev.o events.o wq.o lib/gid.o \
lib/devcom.o lib/pci_vsc.o lib/dm.o diag/fs_tracepoint.o \
diag/fw_tracer.o diag/crdump.o devlink.o diag/rsc_dump.o \
fw_reset.o qos.o
esw/acl/ingress_lgcy.o esw/acl/ingress_ofld.o \
esw/devlink_port.o esw/vporttbl.o
mlx5_core-$(CONFIG_MLX5_TC_SAMPLE) += esw/sample.o
+mlx5_core-$(CONFIG_MLX5_BRIDGE) += esw/bridge.o en/rep/bridge.o
mlx5_core-$(CONFIG_MLX5_MPFS) += lib/mpfs.o
mlx5_core-$(CONFIG_VXLAN) += lib/vxlan.o
int ret = 0, i;
mutex_lock(&mlx5_intf_mutex);
+ priv->flags &= ~MLX5_PRIV_FLAGS_DETACH;
for (i = 0; i < ARRAY_SIZE(mlx5_adev_devices); i++) {
if (!priv->adev[i]) {
bool is_supported = false;
}
} else {
adev = &priv->adev[i]->adev;
+
+ /* Pay attention that this is not PCI driver that
+ * mlx5_core_dev is connected, but auxiliary driver.
+ *
+ * Here we can race of module unload with devlink
+ * reload, but we don't need to take extra lock because
+ * we are holding global mlx5_intf_mutex.
+ */
+ if (!adev->dev.driver)
+ continue;
adrv = to_auxiliary_drv(adev->dev.driver);
if (adrv->resume)
continue;
adev = &priv->adev[i]->adev;
+ /* Auxiliary driver was unbind manually through sysfs */
+ if (!adev->dev.driver)
+ goto skip_suspend;
+
adrv = to_auxiliary_drv(adev->dev.driver);
if (adrv->suspend) {
continue;
}
+skip_suspend:
del_adev(&priv->adev[i]->adev);
priv->adev[i] = NULL;
}
+ priv->flags |= MLX5_PRIV_FLAGS_DETACH;
mutex_unlock(&mlx5_intf_mutex);
}
struct mlx5_priv *priv = &dev->priv;
lockdep_assert_held(&mlx5_intf_mutex);
+ if (priv->flags & MLX5_PRIV_FLAGS_DETACH)
+ return 0;
delete_drivers(dev);
if (priv->flags & MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV)
err = devlink_info_version_running_put(req, "fw.version", version_str);
if (err)
return err;
+ err = devlink_info_version_running_put(req,
+ DEVLINK_INFO_VERSION_GENERIC_FW,
+ version_str);
+ if (err)
+ return err;
/* no pending version, return running (stored) version */
if (stored_fw == 0)
err = devlink_info_version_stored_put(req, "fw.version", version_str);
if (err)
return err;
-
- return 0;
+ return devlink_info_version_stored_put(req,
+ DEVLINK_INFO_VERSION_GENERIC_FW,
+ version_str);
}
static int mlx5_devlink_reload_fw_activate(struct devlink *devlink, struct netlink_ext_ack *extack)
struct mlx5e_xsk_param *xsk, int node,
struct mlx5e_rq *rq);
int mlx5e_wait_for_min_rx_wqes(struct mlx5e_rq *rq, int wait_time);
-void mlx5e_deactivate_rq(struct mlx5e_rq *rq);
void mlx5e_close_rq(struct mlx5e_rq *rq);
int mlx5e_create_rq(struct mlx5e_rq *rq, struct mlx5e_rq_param *param);
void mlx5e_destroy_rq(struct mlx5e_rq *rq);
return priv->netdev->num_rx_queues / max_t(u8, profile->rq_groups, 1);
}
+static inline bool
+mlx5e_tx_mpwqe_supported(struct mlx5_core_dev *mdev)
+{
+ return !is_kdump_kernel() &&
+ MLX5_CAP_ETH(mdev, enhanced_multi_pkt_send_wqe);
+}
+
int mlx5e_priv_init(struct mlx5e_priv *priv,
struct net_device *netdev,
struct mlx5_core_dev *mdev);
struct mlx5e_priv *priv = netdev_priv(dev);
struct devlink_port *port;
+ if (!netif_device_present(dev))
+ return NULL;
port = mlx5e_devlink_get_dl_port(priv);
if (port->registered)
return port;
static struct dim_cq_moder mlx5e_get_def_tx_moderation(u8 cq_period_mode)
{
- struct dim_cq_moder moder;
+ struct dim_cq_moder moder = {};
moder.cq_period_mode = cq_period_mode;
moder.pkts = MLX5E_PARAMS_DEFAULT_TX_CQ_MODERATION_PKTS;
static struct dim_cq_moder mlx5e_get_def_rx_moderation(u8 cq_period_mode)
{
- struct dim_cq_moder moder;
+ struct dim_cq_moder moder = {};
moder.cq_period_mode = cq_period_mode;
moder.pkts = MLX5E_PARAMS_DEFAULT_RX_CQ_MODERATION_PKTS;
static u8 mlx5e_build_async_icosq_log_wq_sz(struct mlx5_core_dev *mdev)
{
- if (mlx5_accel_is_ktls_rx(mdev))
+ if (mlx5e_accel_is_ktls_rx(mdev))
return MLX5E_PARAMS_DEFAULT_LOG_SQ_SIZE;
return MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE;
mlx5e_build_sq_param_common(mdev, param);
param->stop_room = mlx5e_stop_room_for_wqe(1); /* for XSK NOP */
- param->is_tls = mlx5_accel_is_ktls_rx(mdev);
+ param->is_tls = mlx5e_accel_is_ktls_rx(mdev);
if (param->is_tls)
param->stop_room += mlx5e_stop_room_for_wqe(1); /* for TLS RX resync NOP */
MLX5_SET(sqc, sqc, reg_umr, MLX5_CAP_ETH(mdev, reg_umr_sq));
// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
// Copyright (c) 2020 Mellanox Technologies
-#include <linux/ptp_classify.h>
#include "en/ptp.h"
#include "en/txrx.h"
#include "en/params.h"
#include "en.h"
#include "en_stats.h"
+#include <linux/ptp_classify.h>
struct mlx5e_ptpsq {
struct mlx5e_txqsq txqsq;
DECLARE_BITMAP(state, MLX5E_PTP_STATE_NUM_STATES);
};
+static inline bool mlx5e_use_ptpsq(struct sk_buff *skb)
+{
+ struct flow_keys fk;
+
+ if (!(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP))
+ return false;
+
+ if (!skb_flow_dissect_flow_keys(skb, &fk, 0))
+ return false;
+
+ if (fk.basic.n_proto == htons(ETH_P_1588))
+ return true;
+
+ if (fk.basic.n_proto != htons(ETH_P_IP) &&
+ fk.basic.n_proto != htons(ETH_P_IPV6))
+ return false;
+
+ return (fk.basic.ip_proto == IPPROTO_UDP &&
+ fk.ports.dst == htons(PTP_EV_PORT));
+}
+
int mlx5e_ptp_open(struct mlx5e_priv *priv, struct mlx5e_params *params,
u8 lag_port, struct mlx5e_ptp **cp);
void mlx5e_ptp_close(struct mlx5e_ptp *c);
rpriv = priv->ppriv;
fwd_vport_num = rpriv->rep->vport;
lag_dev = netdev_master_upper_dev_get(netdev);
+ if (!lag_dev)
+ return;
netdev_dbg(netdev, "lag_dev(%s)'s slave vport(%d) is txable(%d)\n",
lag_dev->name, fwd_vport_num, net_lag_port_dev_txable(netdev));
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2021 Mellanox Technologies. */
+
+#include <linux/netdevice.h>
+#include <linux/if_bridge.h>
+#include <net/netevent.h>
+#include <net/switchdev.h>
+#include "bridge.h"
+#include "esw/bridge.h"
+#include "en_rep.h"
+
+#define MLX5_ESW_BRIDGE_UPDATE_INTERVAL 1000
+
+struct mlx5_bridge_switchdev_fdb_work {
+ struct work_struct work;
+ struct switchdev_notifier_fdb_info fdb_info;
+ struct net_device *dev;
+ bool add;
+};
+
+static int mlx5_esw_bridge_port_changeupper(struct notifier_block *nb, void *ptr)
+{
+ struct mlx5_esw_bridge_offloads *br_offloads = container_of(nb,
+ struct mlx5_esw_bridge_offloads,
+ netdev_nb);
+ struct net_device *dev = netdev_notifier_info_to_dev(ptr);
+ struct netdev_notifier_changeupper_info *info = ptr;
+ struct netlink_ext_ack *extack;
+ struct mlx5e_rep_priv *rpriv;
+ struct mlx5_eswitch *esw;
+ struct mlx5_vport *vport;
+ struct net_device *upper;
+ struct mlx5e_priv *priv;
+ u16 vport_num;
+
+ if (!mlx5e_eswitch_rep(dev))
+ return 0;
+
+ upper = info->upper_dev;
+ if (!netif_is_bridge_master(upper))
+ return 0;
+
+ esw = br_offloads->esw;
+ priv = netdev_priv(dev);
+ if (esw != priv->mdev->priv.eswitch)
+ return 0;
+
+ rpriv = priv->ppriv;
+ vport_num = rpriv->rep->vport;
+ vport = mlx5_eswitch_get_vport(esw, vport_num);
+ if (IS_ERR(vport))
+ return PTR_ERR(vport);
+
+ extack = netdev_notifier_info_to_extack(&info->info);
+
+ return info->linking ?
+ mlx5_esw_bridge_vport_link(upper->ifindex, br_offloads, vport, extack) :
+ mlx5_esw_bridge_vport_unlink(upper->ifindex, br_offloads, vport, extack);
+}
+
+static int mlx5_esw_bridge_switchdev_port_event(struct notifier_block *nb,
+ unsigned long event, void *ptr)
+{
+ int err = 0;
+
+ switch (event) {
+ case NETDEV_PRECHANGEUPPER:
+ break;
+
+ case NETDEV_CHANGEUPPER:
+ err = mlx5_esw_bridge_port_changeupper(nb, ptr);
+ break;
+ }
+
+ return notifier_from_errno(err);
+}
+
+static int mlx5_esw_bridge_port_obj_add(struct net_device *dev,
+ const struct switchdev_obj *obj,
+ struct netlink_ext_ack *extack)
+{
+ const struct switchdev_obj_port_vlan *vlan;
+ struct mlx5e_rep_priv *rpriv;
+ struct mlx5_eswitch *esw;
+ struct mlx5_vport *vport;
+ struct mlx5e_priv *priv;
+ u16 vport_num;
+ int err = 0;
+
+ priv = netdev_priv(dev);
+ rpriv = priv->ppriv;
+ vport_num = rpriv->rep->vport;
+ esw = priv->mdev->priv.eswitch;
+ vport = mlx5_eswitch_get_vport(esw, vport_num);
+ if (IS_ERR(vport))
+ return PTR_ERR(vport);
+
+ switch (obj->id) {
+ case SWITCHDEV_OBJ_ID_PORT_VLAN:
+ vlan = SWITCHDEV_OBJ_PORT_VLAN(obj);
+ err = mlx5_esw_bridge_port_vlan_add(vlan->vid, vlan->flags, esw, vport, extack);
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+ return err;
+}
+
+static int mlx5_esw_bridge_port_obj_del(struct net_device *dev,
+ const struct switchdev_obj *obj)
+{
+ const struct switchdev_obj_port_vlan *vlan;
+ struct mlx5e_rep_priv *rpriv;
+ struct mlx5_eswitch *esw;
+ struct mlx5_vport *vport;
+ struct mlx5e_priv *priv;
+ u16 vport_num;
+
+ priv = netdev_priv(dev);
+ rpriv = priv->ppriv;
+ vport_num = rpriv->rep->vport;
+ esw = priv->mdev->priv.eswitch;
+ vport = mlx5_eswitch_get_vport(esw, vport_num);
+ if (IS_ERR(vport))
+ return PTR_ERR(vport);
+
+ switch (obj->id) {
+ case SWITCHDEV_OBJ_ID_PORT_VLAN:
+ vlan = SWITCHDEV_OBJ_PORT_VLAN(obj);
+ mlx5_esw_bridge_port_vlan_del(vlan->vid, esw, vport);
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+ return 0;
+}
+
+static int mlx5_esw_bridge_port_obj_attr_set(struct net_device *dev,
+ const struct switchdev_attr *attr,
+ struct netlink_ext_ack *extack)
+{
+ struct mlx5e_rep_priv *rpriv;
+ struct mlx5_eswitch *esw;
+ struct mlx5_vport *vport;
+ struct mlx5e_priv *priv;
+ u16 vport_num;
+ int err = 0;
+
+ priv = netdev_priv(dev);
+ rpriv = priv->ppriv;
+ vport_num = rpriv->rep->vport;
+ esw = priv->mdev->priv.eswitch;
+ vport = mlx5_eswitch_get_vport(esw, vport_num);
+ if (IS_ERR(vport))
+ return PTR_ERR(vport);
+
+ switch (attr->id) {
+ case SWITCHDEV_ATTR_ID_PORT_PRE_BRIDGE_FLAGS:
+ if (attr->u.brport_flags.mask & ~(BR_LEARNING | BR_FLOOD | BR_MCAST_FLOOD)) {
+ NL_SET_ERR_MSG_MOD(extack, "Flag is not supported");
+ err = -EINVAL;
+ }
+ break;
+ case SWITCHDEV_ATTR_ID_PORT_BRIDGE_FLAGS:
+ break;
+ case SWITCHDEV_ATTR_ID_BRIDGE_AGEING_TIME:
+ err = mlx5_esw_bridge_ageing_time_set(attr->u.ageing_time, esw, vport);
+ break;
+ case SWITCHDEV_ATTR_ID_BRIDGE_VLAN_FILTERING:
+ err = mlx5_esw_bridge_vlan_filtering_set(attr->u.vlan_filtering, esw, vport);
+ break;
+ default:
+ err = -EOPNOTSUPP;
+ }
+
+ return err;
+}
+
+static int mlx5_esw_bridge_event_blocking(struct notifier_block *unused,
+ unsigned long event, void *ptr)
+{
+ struct net_device *dev = switchdev_notifier_info_to_dev(ptr);
+ int err;
+
+ switch (event) {
+ case SWITCHDEV_PORT_OBJ_ADD:
+ err = switchdev_handle_port_obj_add(dev, ptr,
+ mlx5e_eswitch_rep,
+ mlx5_esw_bridge_port_obj_add);
+ break;
+ case SWITCHDEV_PORT_OBJ_DEL:
+ err = switchdev_handle_port_obj_del(dev, ptr,
+ mlx5e_eswitch_rep,
+ mlx5_esw_bridge_port_obj_del);
+ break;
+ case SWITCHDEV_PORT_ATTR_SET:
+ err = switchdev_handle_port_attr_set(dev, ptr,
+ mlx5e_eswitch_rep,
+ mlx5_esw_bridge_port_obj_attr_set);
+ break;
+ default:
+ err = 0;
+ }
+
+ return notifier_from_errno(err);
+}
+
+static void
+mlx5_esw_bridge_cleanup_switchdev_fdb_work(struct mlx5_bridge_switchdev_fdb_work *fdb_work)
+{
+ dev_put(fdb_work->dev);
+ kfree(fdb_work->fdb_info.addr);
+ kfree(fdb_work);
+}
+
+static void mlx5_esw_bridge_switchdev_fdb_event_work(struct work_struct *work)
+{
+ struct mlx5_bridge_switchdev_fdb_work *fdb_work =
+ container_of(work, struct mlx5_bridge_switchdev_fdb_work, work);
+ struct switchdev_notifier_fdb_info *fdb_info =
+ &fdb_work->fdb_info;
+ struct net_device *dev = fdb_work->dev;
+ struct mlx5e_rep_priv *rpriv;
+ struct mlx5_eswitch *esw;
+ struct mlx5_vport *vport;
+ struct mlx5e_priv *priv;
+ u16 vport_num;
+
+ rtnl_lock();
+
+ priv = netdev_priv(dev);
+ rpriv = priv->ppriv;
+ vport_num = rpriv->rep->vport;
+ esw = priv->mdev->priv.eswitch;
+ vport = mlx5_eswitch_get_vport(esw, vport_num);
+ if (IS_ERR(vport))
+ goto out;
+
+ if (fdb_work->add)
+ mlx5_esw_bridge_fdb_create(dev, esw, vport, fdb_info);
+ else
+ mlx5_esw_bridge_fdb_remove(dev, esw, vport, fdb_info);
+
+out:
+ rtnl_unlock();
+ mlx5_esw_bridge_cleanup_switchdev_fdb_work(fdb_work);
+}
+
+static struct mlx5_bridge_switchdev_fdb_work *
+mlx5_esw_bridge_init_switchdev_fdb_work(struct net_device *dev, bool add,
+ struct switchdev_notifier_fdb_info *fdb_info)
+{
+ struct mlx5_bridge_switchdev_fdb_work *work;
+ u8 *addr;
+
+ work = kzalloc(sizeof(*work), GFP_ATOMIC);
+ if (!work)
+ return ERR_PTR(-ENOMEM);
+
+ INIT_WORK(&work->work, mlx5_esw_bridge_switchdev_fdb_event_work);
+ memcpy(&work->fdb_info, fdb_info, sizeof(work->fdb_info));
+
+ addr = kzalloc(ETH_ALEN, GFP_ATOMIC);
+ if (!addr) {
+ kfree(work);
+ return ERR_PTR(-ENOMEM);
+ }
+ ether_addr_copy(addr, fdb_info->addr);
+ work->fdb_info.addr = addr;
+
+ dev_hold(dev);
+ work->dev = dev;
+ work->add = add;
+ return work;
+}
+
+static int mlx5_esw_bridge_switchdev_event(struct notifier_block *nb,
+ unsigned long event, void *ptr)
+{
+ struct mlx5_esw_bridge_offloads *br_offloads = container_of(nb,
+ struct mlx5_esw_bridge_offloads,
+ nb);
+ struct net_device *dev = switchdev_notifier_info_to_dev(ptr);
+ struct switchdev_notifier_fdb_info *fdb_info;
+ struct mlx5_bridge_switchdev_fdb_work *work;
+ struct switchdev_notifier_info *info = ptr;
+ struct net_device *upper;
+ struct mlx5e_priv *priv;
+
+ if (!mlx5e_eswitch_rep(dev))
+ return NOTIFY_DONE;
+ priv = netdev_priv(dev);
+ if (priv->mdev->priv.eswitch != br_offloads->esw)
+ return NOTIFY_DONE;
+
+ if (event == SWITCHDEV_PORT_ATTR_SET) {
+ int err = switchdev_handle_port_attr_set(dev, ptr,
+ mlx5e_eswitch_rep,
+ mlx5_esw_bridge_port_obj_attr_set);
+ return notifier_from_errno(err);
+ }
+
+ upper = netdev_master_upper_dev_get_rcu(dev);
+ if (!upper)
+ return NOTIFY_DONE;
+ if (!netif_is_bridge_master(upper))
+ return NOTIFY_DONE;
+
+ switch (event) {
+ case SWITCHDEV_FDB_ADD_TO_DEVICE:
+ case SWITCHDEV_FDB_DEL_TO_DEVICE:
+ fdb_info = container_of(info,
+ struct switchdev_notifier_fdb_info,
+ info);
+
+ work = mlx5_esw_bridge_init_switchdev_fdb_work(dev,
+ event == SWITCHDEV_FDB_ADD_TO_DEVICE,
+ fdb_info);
+ if (IS_ERR(work)) {
+ WARN_ONCE(1, "Failed to init switchdev work, err=%ld",
+ PTR_ERR(work));
+ return notifier_from_errno(PTR_ERR(work));
+ }
+
+ queue_work(br_offloads->wq, &work->work);
+ break;
+ default:
+ break;
+ }
+ return NOTIFY_DONE;
+}
+
+static void mlx5_esw_bridge_update_work(struct work_struct *work)
+{
+ struct mlx5_esw_bridge_offloads *br_offloads = container_of(work,
+ struct mlx5_esw_bridge_offloads,
+ update_work.work);
+
+ rtnl_lock();
+ mlx5_esw_bridge_update(br_offloads);
+ rtnl_unlock();
+
+ queue_delayed_work(br_offloads->wq, &br_offloads->update_work,
+ msecs_to_jiffies(MLX5_ESW_BRIDGE_UPDATE_INTERVAL));
+}
+
+void mlx5e_rep_bridge_init(struct mlx5e_priv *priv)
+{
+ struct mlx5_esw_bridge_offloads *br_offloads;
+ struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5_eswitch *esw =
+ mdev->priv.eswitch;
+ int err;
+
+ rtnl_lock();
+ br_offloads = mlx5_esw_bridge_init(esw);
+ rtnl_unlock();
+ if (IS_ERR(br_offloads)) {
+ esw_warn(mdev, "Failed to init esw bridge (err=%ld)\n", PTR_ERR(br_offloads));
+ return;
+ }
+
+ br_offloads->wq = alloc_ordered_workqueue("mlx5_bridge_wq", 0);
+ if (!br_offloads->wq) {
+ esw_warn(mdev, "Failed to allocate bridge offloads workqueue\n");
+ goto err_alloc_wq;
+ }
+ INIT_DELAYED_WORK(&br_offloads->update_work, mlx5_esw_bridge_update_work);
+ queue_delayed_work(br_offloads->wq, &br_offloads->update_work,
+ msecs_to_jiffies(MLX5_ESW_BRIDGE_UPDATE_INTERVAL));
+
+ br_offloads->nb.notifier_call = mlx5_esw_bridge_switchdev_event;
+ err = register_switchdev_notifier(&br_offloads->nb);
+ if (err) {
+ esw_warn(mdev, "Failed to register switchdev notifier (err=%d)\n", err);
+ goto err_register_swdev;
+ }
+
+ br_offloads->nb_blk.notifier_call = mlx5_esw_bridge_event_blocking;
+ err = register_switchdev_blocking_notifier(&br_offloads->nb_blk);
+ if (err) {
+ esw_warn(mdev, "Failed to register blocking switchdev notifier (err=%d)\n", err);
+ goto err_register_swdev_blk;
+ }
+
+ br_offloads->netdev_nb.notifier_call = mlx5_esw_bridge_switchdev_port_event;
+ err = register_netdevice_notifier(&br_offloads->netdev_nb);
+ if (err) {
+ esw_warn(mdev, "Failed to register bridge offloads netdevice notifier (err=%d)\n",
+ err);
+ goto err_register_netdev;
+ }
+ return;
+
+err_register_netdev:
+ unregister_switchdev_blocking_notifier(&br_offloads->nb_blk);
+err_register_swdev_blk:
+ unregister_switchdev_notifier(&br_offloads->nb);
+err_register_swdev:
+ destroy_workqueue(br_offloads->wq);
+err_alloc_wq:
+ mlx5_esw_bridge_cleanup(esw);
+}
+
+void mlx5e_rep_bridge_cleanup(struct mlx5e_priv *priv)
+{
+ struct mlx5_esw_bridge_offloads *br_offloads;
+ struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5_eswitch *esw =
+ mdev->priv.eswitch;
+
+ br_offloads = esw->br_offloads;
+ if (!br_offloads)
+ return;
+
+ unregister_netdevice_notifier(&br_offloads->netdev_nb);
+ unregister_switchdev_blocking_notifier(&br_offloads->nb_blk);
+ unregister_switchdev_notifier(&br_offloads->nb);
+ cancel_delayed_work(&br_offloads->update_work);
+ destroy_workqueue(br_offloads->wq);
+ rtnl_lock();
+ mlx5_esw_bridge_cleanup(esw);
+ rtnl_unlock();
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2021 Mellanox Technologies. */
+
+#ifndef __MLX5_EN_REP_BRIDGE__
+#define __MLX5_EN_REP_BRIDGE__
+
+#include "en.h"
+
+#if IS_ENABLED(CONFIG_MLX5_BRIDGE)
+
+void mlx5e_rep_bridge_init(struct mlx5e_priv *priv);
+void mlx5e_rep_bridge_cleanup(struct mlx5e_priv *priv);
+
+#else /* CONFIG_MLX5_BRIDGE */
+
+static inline void mlx5e_rep_bridge_init(struct mlx5e_priv *priv) {}
+static inline void mlx5e_rep_bridge_cleanup(struct mlx5e_priv *priv) {}
+
+#endif /* CONFIG_MLX5_BRIDGE */
+
+#endif /* __MLX5_EN_REP_BRIDGE__ */
work);
struct mlx5e_neigh_hash_entry *nhe = update_work->nhe;
struct neighbour *n = update_work->n;
+ struct mlx5e_encap_entry *e = NULL;
bool neigh_connected, same_dev;
- struct mlx5e_encap_entry *e;
unsigned char ha[ETH_ALEN];
- struct mlx5e_priv *priv;
u8 nud_state, dead;
rtnl_lock();
if (!same_dev)
goto out;
- list_for_each_entry(e, &nhe->encap_list, encap_list) {
- if (!mlx5e_encap_take(e))
- continue;
+ /* mlx5e_get_next_init_encap() releases previous encap before returning
+ * the next one.
+ */
+ while ((e = mlx5e_get_next_init_encap(nhe, e)) != NULL)
+ mlx5e_rep_update_flows(netdev_priv(e->out_dev), e, neigh_connected, ha);
- priv = netdev_priv(e->out_dev);
- mlx5e_rep_update_flows(priv, e, neigh_connected, ha);
- mlx5e_encap_put(priv, e);
- }
out:
rtnl_unlock();
mlx5e_release_neigh_update_work(update_work);
ASSERT_RTNL();
- /* wait for encap to be fully initialized */
- wait_for_completion(&e->res_ready);
-
mutex_lock(&esw->offloads.encap_tbl_lock);
encap_connected = !!(e->flags & MLX5_ENCAP_ENTRY_VALID);
- if (e->compl_result < 0 || (encap_connected == neigh_connected &&
- ether_addr_equal(e->h_dest, ha)))
+ if (encap_connected == neigh_connected && ether_addr_equal(e->h_dest, ha))
goto unlock;
mlx5e_take_all_encap_flows(e, &flow_list);
struct mlx5e_tc_update_priv *tc_priv)
{
struct mlx5e_priv *priv = netdev_priv(skb->dev);
- u32 tunnel_id = reg_c1 >> ESW_TUN_OFFSET;
+ u32 tunnel_id = (reg_c1 >> ESW_TUN_OFFSET) & TUNNEL_ID_MASK;
if (chain) {
struct mlx5_rep_uplink_priv *uplink_priv;
struct mlx5_eswitch *esw;
u32 zone_restore_id;
- tc_skb_ext = skb_ext_add(skb, TC_SKB_EXT);
+ tc_skb_ext = tc_skb_ext_alloc(skb);
if (!tc_skb_ext) {
WARN_ON(1);
return false;
#include "en_tc.h"
#include "en_rep.h"
-#define MLX5_CT_ZONE_BITS (mlx5e_tc_attr_to_reg_mappings[ZONE_TO_REG].mlen * 8)
+#define MLX5_CT_ZONE_BITS (mlx5e_tc_attr_to_reg_mappings[ZONE_TO_REG].mlen)
#define MLX5_CT_ZONE_MASK GENMASK(MLX5_CT_ZONE_BITS - 1, 0)
#define MLX5_CT_STATE_ESTABLISHED_BIT BIT(1)
#define MLX5_CT_STATE_TRK_BIT BIT(2)
#define MLX5_CT_STATE_RELATED_BIT BIT(5)
#define MLX5_CT_STATE_INVALID_BIT BIT(6)
-#define MLX5_FTE_ID_BITS (mlx5e_tc_attr_to_reg_mappings[FTEID_TO_REG].mlen * 8)
+#define MLX5_FTE_ID_BITS (mlx5e_tc_attr_to_reg_mappings[FTEID_TO_REG].mlen)
#define MLX5_FTE_ID_MAX GENMASK(MLX5_FTE_ID_BITS - 1, 0)
#define MLX5_FTE_ID_MASK MLX5_FTE_ID_MAX
-#define MLX5_CT_LABELS_BITS (mlx5e_tc_attr_to_reg_mappings[LABELS_TO_REG].mlen * 8)
+#define MLX5_CT_LABELS_BITS (mlx5e_tc_attr_to_reg_mappings[LABELS_TO_REG].mlen)
#define MLX5_CT_LABELS_MASK GENMASK(MLX5_CT_LABELS_BITS - 1, 0)
#define ct_dbg(fmt, args...)\
unsigned long flags;
};
+static void
+mlx5_tc_ct_entry_destroy_mod_hdr(struct mlx5_tc_ct_priv *ct_priv,
+ struct mlx5_flow_attr *attr,
+ struct mlx5e_mod_hdr_handle *mh);
+
static const struct rhashtable_params cts_ht_params = {
.head_offset = offsetof(struct mlx5_ct_entry, node),
.key_offset = offsetof(struct mlx5_ct_entry, cookie),
ct_dbg("Deleting ct entry rule in zone %d", entry->tuple.zone);
mlx5_tc_rule_delete(netdev_priv(ct_priv->netdev), zone_rule->rule, attr);
- mlx5e_mod_hdr_detach(ct_priv->dev,
- ct_priv->mod_hdr_tbl, zone_rule->mh);
+ mlx5_tc_ct_entry_destroy_mod_hdr(ct_priv, zone_rule->attr, zone_rule->mh);
mlx5_put_label_mapping(ct_priv, attr->ct_attr.ct_labels_id);
kfree(attr);
}
if (err)
goto err_mapping;
- *mh = mlx5e_mod_hdr_attach(ct_priv->dev,
- ct_priv->mod_hdr_tbl,
- ct_priv->ns_type,
- &mod_acts);
- if (IS_ERR(*mh)) {
- err = PTR_ERR(*mh);
- goto err_mapping;
+ if (nat) {
+ attr->modify_hdr = mlx5_modify_header_alloc(ct_priv->dev, ct_priv->ns_type,
+ mod_acts.num_actions,
+ mod_acts.actions);
+ if (IS_ERR(attr->modify_hdr)) {
+ err = PTR_ERR(attr->modify_hdr);
+ goto err_mapping;
+ }
+
+ *mh = NULL;
+ } else {
+ *mh = mlx5e_mod_hdr_attach(ct_priv->dev,
+ ct_priv->mod_hdr_tbl,
+ ct_priv->ns_type,
+ &mod_acts);
+ if (IS_ERR(*mh)) {
+ err = PTR_ERR(*mh);
+ goto err_mapping;
+ }
+ attr->modify_hdr = mlx5e_mod_hdr_get(*mh);
}
- attr->modify_hdr = mlx5e_mod_hdr_get(*mh);
dealloc_mod_hdr_actions(&mod_acts);
return 0;
return err;
}
+static void
+mlx5_tc_ct_entry_destroy_mod_hdr(struct mlx5_tc_ct_priv *ct_priv,
+ struct mlx5_flow_attr *attr,
+ struct mlx5e_mod_hdr_handle *mh)
+{
+ if (mh)
+ mlx5e_mod_hdr_detach(ct_priv->dev, ct_priv->mod_hdr_tbl, mh);
+ else
+ mlx5_modify_header_dealloc(ct_priv->dev, attr->modify_hdr);
+}
+
static int
mlx5_tc_ct_entry_add_rule(struct mlx5_tc_ct_priv *ct_priv,
struct flow_rule *flow_rule,
return 0;
err_rule:
- mlx5e_mod_hdr_detach(ct_priv->dev,
- ct_priv->mod_hdr_tbl, zone_rule->mh);
+ mlx5_tc_ct_entry_destroy_mod_hdr(ct_priv, zone_rule->attr, zone_rule->mh);
mlx5_put_label_mapping(ct_priv, attr->ct_attr.ct_labels_id);
err_mod_hdr:
kfree(attr);
}
if (rev_entry && refcount_inc_not_zero(&rev_entry->counter->refcount)) {
- ct_dbg("Using shared counter entry=0x%p rev=0x%p\n", entry, rev_entry);
+ ct_dbg("Using shared counter entry=0x%p rev=0x%p", entry, rev_entry);
shared_counter = rev_entry->counter;
spin_unlock_bh(&ct_priv->ht_lock);
#define zone_to_reg_ct {\
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_2,\
.moffset = 0,\
- .mlen = 2,\
+ .mlen = 16,\
.soffset = MLX5_BYTE_OFF(fte_match_param,\
- misc_parameters_2.metadata_reg_c_2) + 2,\
+ misc_parameters_2.metadata_reg_c_2),\
}
#define ctstate_to_reg_ct {\
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_2,\
- .moffset = 2,\
- .mlen = 2,\
+ .moffset = 16,\
+ .mlen = 16,\
.soffset = MLX5_BYTE_OFF(fte_match_param,\
misc_parameters_2.metadata_reg_c_2),\
}
#define mark_to_reg_ct {\
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_3,\
.moffset = 0,\
- .mlen = 4,\
+ .mlen = 32,\
.soffset = MLX5_BYTE_OFF(fte_match_param,\
misc_parameters_2.metadata_reg_c_3),\
}
#define labels_to_reg_ct {\
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_4,\
.moffset = 0,\
- .mlen = 4,\
+ .mlen = 32,\
.soffset = MLX5_BYTE_OFF(fte_match_param,\
misc_parameters_2.metadata_reg_c_4),\
}
#define fteid_to_reg_ct {\
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_5,\
.moffset = 0,\
- .mlen = 4,\
+ .mlen = 32,\
.soffset = MLX5_BYTE_OFF(fte_match_param,\
misc_parameters_2.metadata_reg_c_5),\
}
#define zone_restore_to_reg_ct {\
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_1,\
.moffset = 0,\
- .mlen = (ESW_ZONE_ID_BITS / 8),\
+ .mlen = ESW_ZONE_ID_BITS,\
.soffset = MLX5_BYTE_OFF(fte_match_param,\
- misc_parameters_2.metadata_reg_c_1) + 3,\
+ misc_parameters_2.metadata_reg_c_1),\
}
#define nic_zone_restore_to_reg_ct {\
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_B,\
- .moffset = 2,\
- .mlen = (ESW_ZONE_ID_BITS / 8),\
+ .moffset = 16,\
+ .mlen = ESW_ZONE_ID_BITS,\
}
#define REG_MAPPING_MLEN(reg) (mlx5e_tc_attr_to_reg_mappings[reg].mlen)
#define REG_MAPPING_MOFFSET(reg) (mlx5e_tc_attr_to_reg_mappings[reg].moffset)
-#define REG_MAPPING_SHIFT(reg) (REG_MAPPING_MOFFSET(reg) * 8)
#if IS_ENABLED(CONFIG_MLX5_TC_CT)
{
int max_encap_size = MLX5_CAP_ESW(priv->mdev, max_encap_header_size);
const struct ip_tunnel_key *tun_key = &e->tun_info->key;
+ struct mlx5_pkt_reformat_params reformat_params;
struct mlx5e_neigh m_neigh = {};
TC_TUN_ROUTE_ATTR_INIT(attr);
int ipv4_encap_size;
*/
goto release_neigh;
}
- e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev,
- e->reformat_type,
- ipv4_encap_size, encap_header,
+
+ memset(&reformat_params, 0, sizeof(reformat_params));
+ reformat_params.type = e->reformat_type;
+ reformat_params.size = ipv4_encap_size;
+ reformat_params.data = encap_header;
+ e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev, &reformat_params,
MLX5_FLOW_NAMESPACE_FDB);
if (IS_ERR(e->pkt_reformat)) {
err = PTR_ERR(e->pkt_reformat);
{
int max_encap_size = MLX5_CAP_ESW(priv->mdev, max_encap_header_size);
const struct ip_tunnel_key *tun_key = &e->tun_info->key;
+ struct mlx5_pkt_reformat_params reformat_params;
TC_TUN_ROUTE_ATTR_INIT(attr);
int ipv4_encap_size;
char *encap_header;
*/
goto release_neigh;
}
- e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev,
- e->reformat_type,
- ipv4_encap_size, encap_header,
+
+ memset(&reformat_params, 0, sizeof(reformat_params));
+ reformat_params.type = e->reformat_type;
+ reformat_params.size = ipv4_encap_size;
+ reformat_params.data = encap_header;
+ e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev, &reformat_params,
MLX5_FLOW_NAMESPACE_FDB);
if (IS_ERR(e->pkt_reformat)) {
err = PTR_ERR(e->pkt_reformat);
{
int max_encap_size = MLX5_CAP_ESW(priv->mdev, max_encap_header_size);
const struct ip_tunnel_key *tun_key = &e->tun_info->key;
+ struct mlx5_pkt_reformat_params reformat_params;
struct mlx5e_neigh m_neigh = {};
TC_TUN_ROUTE_ATTR_INIT(attr);
struct ipv6hdr *ip6h;
goto release_neigh;
}
- e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev,
- e->reformat_type,
- ipv6_encap_size, encap_header,
+ memset(&reformat_params, 0, sizeof(reformat_params));
+ reformat_params.type = e->reformat_type;
+ reformat_params.size = ipv6_encap_size;
+ reformat_params.data = encap_header;
+ e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev, &reformat_params,
MLX5_FLOW_NAMESPACE_FDB);
if (IS_ERR(e->pkt_reformat)) {
err = PTR_ERR(e->pkt_reformat);
{
int max_encap_size = MLX5_CAP_ESW(priv->mdev, max_encap_header_size);
const struct ip_tunnel_key *tun_key = &e->tun_info->key;
+ struct mlx5_pkt_reformat_params reformat_params;
TC_TUN_ROUTE_ATTR_INIT(attr);
struct ipv6hdr *ip6h;
int ipv6_encap_size;
goto release_neigh;
}
- e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev,
- e->reformat_type,
- ipv6_encap_size, encap_header,
+ memset(&reformat_params, 0, sizeof(reformat_params));
+ reformat_params.type = e->reformat_type;
+ reformat_params.size = ipv6_encap_size;
+ reformat_params.data = encap_header;
+ e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev, &reformat_params,
MLX5_FLOW_NAMESPACE_FDB);
if (IS_ERR(e->pkt_reformat)) {
err = PTR_ERR(e->pkt_reformat);
struct list_head *flow_list)
{
struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
+ struct mlx5_pkt_reformat_params reformat_params;
struct mlx5_esw_flow_attr *esw_attr;
struct mlx5_flow_handle *rule;
struct mlx5_flow_attr *attr;
if (e->flags & MLX5_ENCAP_ENTRY_NO_ROUTE)
return;
+ memset(&reformat_params, 0, sizeof(reformat_params));
+ reformat_params.type = e->reformat_type;
+ reformat_params.size = e->encap_size;
+ reformat_params.data = e->encap_header;
e->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev,
- e->reformat_type,
- e->encap_size, e->encap_header,
+ &reformat_params,
MLX5_FLOW_NAMESPACE_FDB);
if (IS_ERR(e->pkt_reformat)) {
mlx5_core_warn(priv->mdev, "Failed to offload cached encapsulation header, %lu\n",
mlx5e_take_tmp_flow(flow, flow_list, 0);
}
+typedef bool (match_cb)(struct mlx5e_encap_entry *);
+
static struct mlx5e_encap_entry *
-mlx5e_get_next_valid_encap(struct mlx5e_neigh_hash_entry *nhe,
- struct mlx5e_encap_entry *e)
+mlx5e_get_next_matching_encap(struct mlx5e_neigh_hash_entry *nhe,
+ struct mlx5e_encap_entry *e,
+ match_cb match)
{
struct mlx5e_encap_entry *next = NULL;
/* wait for encap to be fully initialized */
wait_for_completion(&next->res_ready);
/* continue searching if encap entry is not in valid state after completion */
- if (!(next->flags & MLX5_ENCAP_ENTRY_VALID)) {
+ if (!match(next)) {
e = next;
goto retry;
}
return next;
}
+static bool mlx5e_encap_valid(struct mlx5e_encap_entry *e)
+{
+ return e->flags & MLX5_ENCAP_ENTRY_VALID;
+}
+
+static struct mlx5e_encap_entry *
+mlx5e_get_next_valid_encap(struct mlx5e_neigh_hash_entry *nhe,
+ struct mlx5e_encap_entry *e)
+{
+ return mlx5e_get_next_matching_encap(nhe, e, mlx5e_encap_valid);
+}
+
+static bool mlx5e_encap_initialized(struct mlx5e_encap_entry *e)
+{
+ return e->compl_result >= 0;
+}
+
+struct mlx5e_encap_entry *
+mlx5e_get_next_init_encap(struct mlx5e_neigh_hash_entry *nhe,
+ struct mlx5e_encap_entry *e)
+{
+ return mlx5e_get_next_matching_encap(nhe, e, mlx5e_encap_initialized);
+}
+
void mlx5e_tc_update_neigh_used_value(struct mlx5e_neigh_hash_entry *nhe)
{
struct mlx5e_neigh *m_neigh = &nhe->m_neigh;
{
struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
struct mlx5_esw_flow_attr *attr = flow->attr->esw_attr;
+ struct mlx5_pkt_reformat_params reformat_params;
struct mlx5e_tc_flow_parse_attr *parse_attr;
struct mlx5e_decap_entry *d;
struct mlx5e_decap_key key;
hash_add_rcu(esw->offloads.decap_tbl, &d->hlist, hash_key);
mutex_unlock(&esw->offloads.decap_tbl_lock);
+ memset(&reformat_params, 0, sizeof(reformat_params));
+ reformat_params.type = MLX5_REFORMAT_TYPE_L3_TUNNEL_TO_L2;
+ reformat_params.size = sizeof(parse_attr->eth);
+ reformat_params.data = &parse_attr->eth;
d->pkt_reformat = mlx5_packet_reformat_alloc(priv->mdev,
- MLX5_REFORMAT_TYPE_L3_TUNNEL_TO_L2,
- sizeof(parse_attr->eth),
- &parse_attr->eth,
+ &reformat_params,
MLX5_FLOW_NAMESPACE_FDB);
if (IS_ERR(d->pkt_reformat)) {
err = PTR_ERR(d->pkt_reformat);
fen_info = container_of(info, struct fib_entry_notifier_info, info);
fib_dev = fib_info_nh(fen_info->fi, 0)->fib_nh_dev;
- if (fib_dev->netdev_ops != &mlx5e_netdev_ops ||
+ if (!fib_dev || fib_dev->netdev_ops != &mlx5e_netdev_ops ||
fen_info->dst_len != 32)
return NULL;
/* Part of the eseg touched by TX offloads */
#define MLX5E_ACCEL_ESEG_LEN offsetof(struct mlx5_wqe_eth_seg, mss)
-static inline bool mlx5e_accel_tx_eseg(struct mlx5e_priv *priv,
+static inline void mlx5e_accel_tx_eseg(struct mlx5e_priv *priv,
struct sk_buff *skb,
struct mlx5_wqe_eth_seg *eseg, u16 ihs)
{
if (skb->encapsulation && skb->ip_summed == CHECKSUM_PARTIAL)
mlx5e_tx_tunnel_accel(skb, eseg, ihs);
#endif
-
- return true;
}
static inline void mlx5e_accel_tx_finish(struct mlx5e_txqsq *sq,
struct mlx5_core_dev *mdev = priv->mdev;
struct net_device *netdev = priv->netdev;
- if (!priv->ipsec)
- return;
-
if (!(mlx5_accel_ipsec_device_caps(mdev) & MLX5_ACCEL_IPSEC_CAP_ESP) ||
!MLX5_CAP_ETH(mdev, swp)) {
mlx5_core_dbg(mdev, "mlx5e: ESP and SWP offload not supported\n");
struct mlx5_wqe_eth_seg *eseg, u8 mode,
struct xfrm_offload *xo)
{
- struct mlx5e_swp_spec swp_spec = {};
-
/* Tunnel Mode:
* SWP: OutL3 InL3 InL4
* Pkt: MAC IP ESP IP L4
* SWP: OutL3 InL4
* InL3
* Pkt: MAC IP ESP L4
+ *
+ * Tunnel(VXLAN TCP/UDP) over Transport Mode
+ * SWP: OutL3 InL3 InL4
+ * Pkt: MAC IP ESP UDP VXLAN IP L4
*/
- swp_spec.l3_proto = skb->protocol;
- swp_spec.is_tun = mode == XFRM_MODE_TUNNEL;
- if (swp_spec.is_tun) {
- if (xo->proto == IPPROTO_IPV6) {
- swp_spec.tun_l3_proto = htons(ETH_P_IPV6);
- swp_spec.tun_l4_proto = inner_ipv6_hdr(skb)->nexthdr;
- } else {
- swp_spec.tun_l3_proto = htons(ETH_P_IP);
- swp_spec.tun_l4_proto = inner_ip_hdr(skb)->protocol;
- }
- } else {
- swp_spec.tun_l3_proto = skb->protocol;
- swp_spec.tun_l4_proto = xo->proto;
+
+ /* Shared settings */
+ eseg->swp_outer_l3_offset = skb_network_offset(skb) / 2;
+ if (skb->protocol == htons(ETH_P_IPV6))
+ eseg->swp_flags |= MLX5_ETH_WQE_SWP_OUTER_L3_IPV6;
+
+ /* Tunnel mode */
+ if (mode == XFRM_MODE_TUNNEL) {
+ eseg->swp_inner_l3_offset = skb_inner_network_offset(skb) / 2;
+ eseg->swp_inner_l4_offset = skb_inner_transport_offset(skb) / 2;
+ if (xo->proto == IPPROTO_IPV6)
+ eseg->swp_flags |= MLX5_ETH_WQE_SWP_INNER_L3_IPV6;
+ if (inner_ip_hdr(skb)->protocol == IPPROTO_UDP)
+ eseg->swp_flags |= MLX5_ETH_WQE_SWP_INNER_L4_UDP;
+ return;
+ }
+
+ /* Transport mode */
+ if (mode != XFRM_MODE_TRANSPORT)
+ return;
+
+ if (!xo->inner_ipproto) {
+ eseg->swp_inner_l3_offset = skb_network_offset(skb) / 2;
+ eseg->swp_inner_l4_offset = skb_inner_transport_offset(skb) / 2;
+ if (skb->protocol == htons(ETH_P_IPV6))
+ eseg->swp_flags |= MLX5_ETH_WQE_SWP_INNER_L3_IPV6;
+ if (xo->proto == IPPROTO_UDP)
+ eseg->swp_flags |= MLX5_ETH_WQE_SWP_INNER_L4_UDP;
+ return;
+ }
+
+ /* Tunnel(VXLAN TCP/UDP) over Transport Mode */
+ switch (xo->inner_ipproto) {
+ case IPPROTO_UDP:
+ eseg->swp_flags |= MLX5_ETH_WQE_SWP_INNER_L4_UDP;
+ fallthrough;
+ case IPPROTO_TCP:
+ eseg->swp_inner_l3_offset = skb_inner_network_offset(skb) / 2;
+ eseg->swp_inner_l4_offset = (skb->csum_start + skb->head - skb->data) / 2;
+ if (skb->protocol == htons(ETH_P_IPV6))
+ eseg->swp_flags |= MLX5_ETH_WQE_SWP_INNER_L3_IPV6;
+ break;
+ default:
+ break;
}
- mlx5e_set_eseg_swp(skb, eseg, &swp_spec);
+ return;
}
void mlx5e_ipsec_set_iv_esn(struct sk_buff *skb, struct xfrm_state *x,
void mlx5e_ipsec_tx_build_eseg(struct mlx5e_priv *priv, struct sk_buff *skb,
struct mlx5_wqe_eth_seg *eseg);
-static inline bool mlx5e_ipsec_feature_check(struct sk_buff *skb, struct net_device *netdev,
- netdev_features_t features)
+static inline netdev_features_t
+mlx5e_ipsec_feature_check(struct sk_buff *skb, netdev_features_t features)
{
+ struct xfrm_offload *xo = xfrm_offload(skb);
struct sec_path *sp = skb_sec_path(skb);
- if (sp && sp->len) {
+ if (sp && sp->len && xo) {
struct xfrm_state *x = sp->xvec[0];
- if (x && x->xso.offload_handle)
- return true;
+ if (!x || !x->xso.offload_handle)
+ goto out_disable;
+
+ if (xo->inner_ipproto) {
+ /* Cannot support tunnel packet over IPsec tunnel mode
+ * because we cannot offload three IP header csum
+ */
+ if (x->props.mode == XFRM_MODE_TUNNEL)
+ goto out_disable;
+
+ /* Only support UDP or TCP L4 checksum */
+ if (xo->inner_ipproto != IPPROTO_UDP &&
+ xo->inner_ipproto != IPPROTO_TCP)
+ goto out_disable;
+ }
+
+ return features;
+
}
- return false;
+
+ /* Disable CSUM and GSO for software IPsec */
+out_disable:
+ return features & ~(NETIF_F_CSUM_MASK | NETIF_F_GSO_MASK);
}
#else
}
static inline bool mlx5_ipsec_is_rx_flow(struct mlx5_cqe64 *cqe) { return false; }
-static inline bool mlx5e_ipsec_feature_check(struct sk_buff *skb, struct net_device *netdev,
- netdev_features_t features) { return false; }
+static inline netdev_features_t
+mlx5e_ipsec_feature_check(struct sk_buff *skb, netdev_features_t features)
+{ return features & ~(NETIF_F_CSUM_MASK | NETIF_F_GSO_MASK); }
#endif /* CONFIG_MLX5_EN_IPSEC */
#endif /* __MLX5E_IPSEC_RXTX_H__ */
struct net_device *netdev = priv->netdev;
struct mlx5_core_dev *mdev = priv->mdev;
- if (mlx5_accel_is_ktls_tx(mdev)) {
+ if (!mlx5e_accel_is_ktls_tx(mdev) && !mlx5e_accel_is_ktls_rx(mdev))
+ return;
+
+ if (mlx5e_accel_is_ktls_tx(mdev)) {
netdev->hw_features |= NETIF_F_HW_TLS_TX;
netdev->features |= NETIF_F_HW_TLS_TX;
}
- if (mlx5_accel_is_ktls_rx(mdev))
+ if (mlx5e_accel_is_ktls_rx(mdev))
netdev->hw_features |= NETIF_F_HW_TLS_RX;
netdev->tlsdev_ops = &mlx5e_ktls_ops;
{
int err;
- if (!mlx5_accel_is_ktls_rx(priv->mdev))
+ if (!mlx5e_accel_is_ktls_rx(priv->mdev))
return 0;
priv->tls->rx_wq = create_singlethread_workqueue("mlx5e_tls_rx");
void mlx5e_ktls_cleanup_rx(struct mlx5e_priv *priv)
{
- if (!mlx5_accel_is_ktls_rx(priv->mdev))
+ if (!mlx5e_accel_is_ktls_rx(priv->mdev))
return;
if (priv->netdev->features & NETIF_F_HW_TLS_RX)
struct mlx5e_ktls_resync_resp *
mlx5e_ktls_rx_resync_create_resp_list(void);
void mlx5e_ktls_rx_resync_destroy_resp_list(struct mlx5e_ktls_resync_resp *resp_list);
+
+static inline bool mlx5e_accel_is_ktls_tx(struct mlx5_core_dev *mdev)
+{
+ return !is_kdump_kernel() &&
+ mlx5_accel_is_ktls_tx(mdev);
+}
+
+static inline bool mlx5e_accel_is_ktls_rx(struct mlx5_core_dev *mdev)
+{
+ return !is_kdump_kernel() &&
+ mlx5_accel_is_ktls_rx(mdev);
+}
+
+static inline bool mlx5e_accel_is_ktls_device(struct mlx5_core_dev *mdev)
+{
+ return !is_kdump_kernel() &&
+ mlx5_accel_is_ktls_device(mdev);
+}
+
#else
static inline void mlx5e_ktls_build_netdev(struct mlx5e_priv *priv)
static inline void
mlx5e_ktls_rx_resync_destroy_resp_list(struct mlx5e_ktls_resync_resp *resp_list) {}
+
+static inline bool mlx5e_accel_is_ktls_tx(struct mlx5_core_dev *mdev) { return false; }
+static inline bool mlx5e_accel_is_ktls_rx(struct mlx5_core_dev *mdev) { return false; }
+static inline bool mlx5e_accel_is_ktls_device(struct mlx5_core_dev *mdev) { return false; }
+
#endif
#endif /* __MLX5E_TLS_H__ */
return nfrags + DIV_ROUND_UP(sync_len, MLX5E_SW2HW_MTU(params, params->sw_mtu));
}
-u16 mlx5e_ktls_get_stop_room(struct mlx5e_params *params)
+u16 mlx5e_ktls_get_stop_room(struct mlx5_core_dev *mdev, struct mlx5e_params *params)
{
u16 num_dumps, stop_room = 0;
+ if (!mlx5e_accel_is_ktls_tx(mdev))
+ return 0;
+
num_dumps = mlx5e_ktls_dumps_num_wqes(params, MAX_SKB_FRAGS, TLS_MAX_PAYLOAD_SIZE);
stop_room += mlx5e_stop_room_for_wqe(MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS);
u32 tls_tisn;
};
-u16 mlx5e_ktls_get_stop_room(struct mlx5e_params *params);
+u16 mlx5e_ktls_get_stop_room(struct mlx5_core_dev *mdev, struct mlx5e_params *params);
bool mlx5e_ktls_handle_tx_skb(struct tls_context *tls_ctx, struct mlx5e_txqsq *sq,
struct sk_buff *skb, int datalen,
struct net_device *netdev = priv->netdev;
u32 caps;
- if (mlx5_accel_is_ktls_device(priv->mdev)) {
+ if (mlx5e_accel_is_ktls_device(priv->mdev)) {
mlx5e_ktls_build_netdev(priv);
return;
}
/* FPGA */
- if (!mlx5_accel_is_tls_device(priv->mdev))
+ if (!mlx5e_accel_is_tls_device(priv->mdev))
return;
caps = mlx5_accel_tls_device_caps(priv->mdev);
{
struct mlx5e_tls *tls;
- if (!mlx5_accel_is_tls_device(priv->mdev))
+ if (!mlx5e_accel_is_tls_device(priv->mdev))
return 0;
tls = kzalloc(sizeof(*tls), GFP_KERNEL);
int mlx5e_tls_get_strings(struct mlx5e_priv *priv, uint8_t *data);
int mlx5e_tls_get_stats(struct mlx5e_priv *priv, u64 *data);
+static inline bool mlx5e_accel_is_tls_device(struct mlx5_core_dev *mdev)
+{
+ return !is_kdump_kernel() &&
+ mlx5_accel_is_tls_device(mdev);
+}
+
#else
static inline void mlx5e_tls_build_netdev(struct mlx5e_priv *priv)
{
- if (mlx5_accel_is_ktls_device(priv->mdev))
+ if (!is_kdump_kernel() &&
+ mlx5_accel_is_ktls_device(priv->mdev))
mlx5e_ktls_build_netdev(priv);
}
static inline int mlx5e_tls_get_count(struct mlx5e_priv *priv) { return 0; }
static inline int mlx5e_tls_get_strings(struct mlx5e_priv *priv, uint8_t *data) { return 0; }
static inline int mlx5e_tls_get_stats(struct mlx5e_priv *priv, u64 *data) { return 0; }
+static inline bool mlx5e_accel_is_tls_device(struct mlx5_core_dev *mdev) { return false; }
#endif
if (WARN_ON_ONCE(tls_ctx->netdev != netdev))
goto err_out;
- if (mlx5_accel_is_ktls_tx(sq->mdev))
+ if (mlx5e_accel_is_ktls_tx(sq->mdev))
return mlx5e_ktls_handle_tx_skb(tls_ctx, sq, skb, datalen, state);
/* FPGA */
u16 mlx5e_tls_get_stop_room(struct mlx5_core_dev *mdev, struct mlx5e_params *params)
{
- if (!mlx5_accel_is_tls_device(mdev))
+ if (!mlx5e_accel_is_tls_device(mdev))
return 0;
- if (mlx5_accel_is_ktls_device(mdev))
- return mlx5e_ktls_get_stop_room(params);
+ if (mlx5e_accel_is_ktls_device(mdev))
+ return mlx5e_ktls_get_stop_room(mdev, params);
/* FPGA */
/* Resync SKB. */
{
if (!priv->tls)
return NULL;
- if (mlx5_accel_is_ktls_device(priv->mdev))
+ if (mlx5e_accel_is_ktls_device(priv->mdev))
return mlx5e_ktls_sw_stats_desc;
return mlx5e_tls_sw_stats_desc;
}
{
if (!priv->tls)
return 0;
- if (mlx5_accel_is_ktls_device(priv->mdev))
+ if (mlx5e_accel_is_ktls_device(priv->mdev))
return ARRAY_SIZE(mlx5e_ktls_sw_stats_desc);
return ARRAY_SIZE(mlx5e_tls_sw_stats_desc);
}
int mlx5e_arfs_create_tables(struct mlx5e_priv *priv)
{
- int err = 0;
+ int err = -ENOMEM;
int i;
if (!(priv->netdev->hw_features & NETIF_F_NTUPLE))
{
struct mlx5e_priv *priv = netdev_priv(netdev);
struct mlx5_core_dev *mdev = priv->mdev;
+ unsigned long fec_bitmap;
u16 fec_policy = 0;
int mode;
int err;
- if (bitmap_weight((unsigned long *)&fecparam->fec,
- ETHTOOL_FEC_LLRS_BIT + 1) > 1)
+ bitmap_from_arr32(&fec_bitmap, &fecparam->fec, sizeof(fecparam->fec) * BITS_PER_BYTE);
+ if (bitmap_weight(&fec_bitmap, ETHTOOL_FEC_LLRS_BIT + 1) > 1)
return -EOPNOTSUPP;
for (mode = 0; mode < ARRAY_SIZE(pplm_fec_2_ethtool); mode++) {
if (curr_val == new_val)
return 0;
+ if (new_val && !priv->profile->rx_ptp_support &&
+ priv->tstamp.rx_filter != HWTSTAMP_FILTER_NONE) {
+ netdev_err(priv->netdev,
+ "Profile doesn't support enabling of CQE compression while hardware time-stamping is enabled.\n");
+ return -EINVAL;
+ }
+
new_params = priv->channels.params;
MLX5E_SET_PFLAG(&new_params, MLX5E_PFLAG_RX_CQE_COMPRESS, new_val);
if (priv->tstamp.rx_filter != HWTSTAMP_FILTER_NONE)
struct mlx5_core_dev *mdev = priv->mdev;
struct mlx5e_params new_params;
- if (enable && !MLX5_CAP_ETH(mdev, enhanced_multi_pkt_send_wqe))
+ if (enable && !mlx5e_tx_mpwqe_supported(mdev))
return -EOPNOTSUPP;
new_params = priv->channels.params;
#include <linux/ipv6.h>
#include <linux/tcp.h>
#include <linux/mlx5/fs.h>
+#include <linux/mlx5/mpfs.h>
#include "en.h"
#include "en_rep.h"
#include "lib/mpfs.h"
{
struct mlx5_core_dev *mdev = priv->mdev;
u8 port_state;
+ bool up;
port_state = mlx5_query_vport_state(mdev,
MLX5_VPORT_STATE_OP_MOD_VNIC_VPORT,
0);
- if (port_state == VPORT_STATE_UP) {
+ up = port_state == VPORT_STATE_UP;
+ if (up == netif_carrier_ok(priv->netdev))
+ netif_carrier_event(priv->netdev);
+ if (up) {
netdev_info(priv->netdev, "Link up\n");
netif_carrier_on(priv->netdev);
} else {
if (err)
goto err_destroy_rq;
- if (mlx5e_is_tls_on(rq->priv) && !mlx5_accel_is_ktls_device(mdev))
+ if (mlx5e_is_tls_on(rq->priv) && !mlx5e_accel_is_ktls_device(mdev))
__set_bit(MLX5E_RQ_STATE_FPGA_TLS, &rq->state); /* must be FPGA */
if (MLX5_CAP_ETH(mdev, cqe_checksum_full))
void mlx5e_activate_rq(struct mlx5e_rq *rq)
{
set_bit(MLX5E_RQ_STATE_ENABLED, &rq->state);
- if (rq->icosq)
+ if (rq->icosq) {
mlx5e_trigger_irq(rq->icosq);
- else
+ } else {
+ local_bh_disable();
napi_schedule(rq->cq.napi);
+ local_bh_enable();
+ }
}
void mlx5e_deactivate_rq(struct mlx5e_rq *rq)
int err;
old_num_txqs = netdev->real_num_tx_queues;
- old_ntc = netdev->num_tc;
+ old_ntc = netdev->num_tc ? : 1;
nch = priv->channels.params.num_channels;
ntc = priv->channels.params.num_tc;
num_rxqs = nch * priv->profile->rq_groups;
- if (priv->channels.params.ptp_rx)
- num_rxqs++;
mlx5e_netdev_set_tcs(netdev, nch, ntc);
netdev_warn(netdev, "Disabling rxhash, not supported when CQE compress is active\n");
}
+ if (mlx5e_is_uplink_rep(priv)) {
+ features &= ~NETIF_F_HW_TLS_RX;
+ if (netdev->features & NETIF_F_HW_TLS_RX)
+ netdev_warn(netdev, "Disabling hw_tls_rx, not supported in switchdev mode\n");
+
+ features &= ~NETIF_F_HW_TLS_TX;
+ if (netdev->features & NETIF_F_HW_TLS_TX)
+ netdev_warn(netdev, "Disabling hw_tls_tx, not supported in switchdev mode\n");
+ }
+
mutex_unlock(&priv->state_lock);
return features;
return mlx5e_ptp_rx_manage_fs(priv, set);
}
-int mlx5e_hwstamp_set(struct mlx5e_priv *priv, struct ifreq *ifr)
+static int mlx5e_hwstamp_config_no_ptp_rx(struct mlx5e_priv *priv, bool rx_filter)
+{
+ bool rx_cqe_compress_def = priv->channels.params.rx_cqe_compress_def;
+ int err;
+
+ if (!rx_filter)
+ /* Reset CQE compression to Admin default */
+ return mlx5e_modify_rx_cqe_compression_locked(priv, rx_cqe_compress_def);
+
+ if (!MLX5E_GET_PFLAG(&priv->channels.params, MLX5E_PFLAG_RX_CQE_COMPRESS))
+ return 0;
+
+ /* Disable CQE compression */
+ netdev_warn(priv->netdev, "Disabling RX cqe compression\n");
+ err = mlx5e_modify_rx_cqe_compression_locked(priv, false);
+ if (err)
+ netdev_err(priv->netdev, "Failed disabling cqe compression err=%d\n", err);
+
+ return err;
+}
+
+static int mlx5e_hwstamp_config_ptp_rx(struct mlx5e_priv *priv, bool ptp_rx)
{
struct mlx5e_params new_params;
+
+ if (ptp_rx == priv->channels.params.ptp_rx)
+ return 0;
+
+ new_params = priv->channels.params;
+ new_params.ptp_rx = ptp_rx;
+ return mlx5e_safe_switch_params(priv, &new_params, mlx5e_ptp_rx_manage_fs_ctx,
+ &new_params.ptp_rx, true);
+}
+
+int mlx5e_hwstamp_set(struct mlx5e_priv *priv, struct ifreq *ifr)
+{
struct hwtstamp_config config;
bool rx_cqe_compress_def;
+ bool ptp_rx;
int err;
if (!MLX5_CAP_GEN(priv->mdev, device_frequency_khz) ||
}
mutex_lock(&priv->state_lock);
- new_params = priv->channels.params;
rx_cqe_compress_def = priv->channels.params.rx_cqe_compress_def;
/* RX HW timestamp */
switch (config.rx_filter) {
case HWTSTAMP_FILTER_NONE:
- new_params.ptp_rx = false;
+ ptp_rx = false;
break;
case HWTSTAMP_FILTER_ALL:
case HWTSTAMP_FILTER_SOME:
case HWTSTAMP_FILTER_PTP_V2_SYNC:
case HWTSTAMP_FILTER_PTP_V2_DELAY_REQ:
case HWTSTAMP_FILTER_NTP_ALL:
- new_params.ptp_rx = rx_cqe_compress_def;
config.rx_filter = HWTSTAMP_FILTER_ALL;
+ /* ptp_rx is set if both HW TS is set and CQE
+ * compression is set
+ */
+ ptp_rx = rx_cqe_compress_def;
break;
default:
- mutex_unlock(&priv->state_lock);
- return -ERANGE;
+ err = -ERANGE;
+ goto err_unlock;
}
- if (new_params.ptp_rx == priv->channels.params.ptp_rx)
- goto out;
+ if (!priv->profile->rx_ptp_support)
+ err = mlx5e_hwstamp_config_no_ptp_rx(priv,
+ config.rx_filter != HWTSTAMP_FILTER_NONE);
+ else
+ err = mlx5e_hwstamp_config_ptp_rx(priv, ptp_rx);
+ if (err)
+ goto err_unlock;
- err = mlx5e_safe_switch_params(priv, &new_params, mlx5e_ptp_rx_manage_fs_ctx,
- &new_params.ptp_rx, true);
- if (err) {
- mutex_unlock(&priv->state_lock);
- return err;
- }
-out:
memcpy(&priv->tstamp, &config, sizeof(config));
mutex_unlock(&priv->state_lock);
return copy_to_user(ifr->ifr_data, &config,
sizeof(config)) ? -EFAULT : 0;
+err_unlock:
+ mutex_unlock(&priv->state_lock);
+ return err;
}
int mlx5e_hwstamp_get(struct mlx5e_priv *priv, struct ifreq *ifr)
if (port == GENEVE_UDP_PORT && mlx5_geneve_tx_allowed(priv->mdev))
return features;
#endif
+ break;
+#ifdef CONFIG_MLX5_EN_IPSEC
+ case IPPROTO_ESP:
+ return mlx5e_ipsec_feature_check(skb, features);
+#endif
}
out:
features = vlan_features_check(skb, features);
features = vxlan_features_check(skb, features);
- if (mlx5e_ipsec_feature_check(skb, netdev, features))
- return features;
-
/* Validate if the tunneled packet is being offloaded by HW */
if (skb->encapsulation &&
(features & NETIF_F_CSUM_MASK || features & NETIF_F_GSO_MASK))
params->log_sq_size = is_kdump_kernel() ?
MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE :
MLX5E_PARAMS_DEFAULT_LOG_SQ_SIZE;
- MLX5E_SET_PFLAG(params, MLX5E_PFLAG_SKB_TX_MPWQE,
- MLX5_CAP_ETH(mdev, enhanced_multi_pkt_send_wqe));
+ MLX5E_SET_PFLAG(params, MLX5E_PFLAG_SKB_TX_MPWQE, mlx5e_tx_mpwqe_supported(mdev));
/* XDP SQ */
- MLX5E_SET_PFLAG(params, MLX5E_PFLAG_XDP_TX_MPWQE,
- MLX5_CAP_ETH(mdev, enhanced_multi_pkt_send_wqe));
+ MLX5E_SET_PFLAG(params, MLX5E_PFLAG_XDP_TX_MPWQE, mlx5e_tx_mpwqe_supported(mdev));
/* set CQE compression */
params->rx_cqe_compress_def = false;
}
if (mlx5_vxlan_allowed(mdev->vxlan) || mlx5_geneve_tx_allowed(mdev)) {
- netdev->hw_features |= NETIF_F_GSO_UDP_TUNNEL |
- NETIF_F_GSO_UDP_TUNNEL_CSUM;
- netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL |
- NETIF_F_GSO_UDP_TUNNEL_CSUM;
- netdev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
- netdev->vlan_features |= NETIF_F_GSO_UDP_TUNNEL |
- NETIF_F_GSO_UDP_TUNNEL_CSUM;
+ netdev->hw_features |= NETIF_F_GSO_UDP_TUNNEL;
+ netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
+ netdev->vlan_features |= NETIF_F_GSO_UDP_TUNNEL;
}
if (mlx5e_tunnel_proto_supported_tx(mdev, IPPROTO_GRE)) {
- netdev->hw_features |= NETIF_F_GSO_GRE |
- NETIF_F_GSO_GRE_CSUM;
- netdev->hw_enc_features |= NETIF_F_GSO_GRE |
- NETIF_F_GSO_GRE_CSUM;
- netdev->gso_partial_features |= NETIF_F_GSO_GRE |
- NETIF_F_GSO_GRE_CSUM;
+ netdev->hw_features |= NETIF_F_GSO_GRE;
+ netdev->hw_enc_features |= NETIF_F_GSO_GRE;
+ netdev->gso_partial_features |= NETIF_F_GSO_GRE;
}
if (mlx5e_tunnel_proto_supported_tx(mdev, IPPROTO_IPIP)) {
mlx5e_set_netdev_mtu_boundaries(priv);
mlx5e_set_dev_port_mtu(priv);
- mlx5_lag_add(mdev, netdev);
+ mlx5_lag_add_netdev(mdev, netdev);
mlx5e_enable_async_events(priv);
mlx5e_enable_blocking_events(priv);
priv->en_trap = NULL;
}
mlx5e_disable_async_events(priv);
- mlx5_lag_remove(mdev);
+ mlx5_lag_remove_netdev(mdev, priv->netdev);
mlx5_vxlan_reset_to_default(mdev->vxlan);
}
rtnl_unlock();
}
+static void mlx5e_reset_channels(struct net_device *netdev)
+{
+ netdev_reset_tc(netdev);
+}
+
int mlx5e_attach_netdev(struct mlx5e_priv *priv)
{
const bool take_rtnl = priv->netdev->reg_state == NETREG_REGISTERED;
profile->cleanup_tx(priv);
out:
+ mlx5e_reset_channels(priv->netdev);
set_bit(MLX5E_STATE_DESTROYING, &priv->state);
cancel_work_sync(&priv->update_stats_work);
return err;
profile->cleanup_rx(priv);
profile->cleanup_tx(priv);
+ mlx5e_reset_channels(priv->netdev);
cancel_work_sync(&priv->update_stats_work);
}
#include "en_tc.h"
#include "en/rep/tc.h"
#include "en/rep/neigh.h"
+#include "en/rep/bridge.h"
#include "en/devlink.h"
#include "fs_core.h"
#include "lib/mlx5.h"
.ndo_change_carrier = mlx5e_rep_change_carrier,
};
-bool mlx5e_eswitch_uplink_rep(struct net_device *netdev)
+bool mlx5e_eswitch_uplink_rep(const struct net_device *netdev)
{
return netdev->netdev_ops == &mlx5e_netdev_ops &&
mlx5e_is_uplink_rep(netdev_priv(netdev));
}
-bool mlx5e_eswitch_vf_rep(struct net_device *netdev)
+bool mlx5e_eswitch_vf_rep(const struct net_device *netdev)
{
return netdev->netdev_ops == &mlx5e_netdev_ops_rep;
}
if (MLX5_CAP_GEN(mdev, uplink_follow))
mlx5_modify_vport_admin_state(mdev, MLX5_VPORT_STATE_OP_MOD_UPLINK,
0, 0, MLX5_VPORT_ADMIN_STATE_AUTO);
- mlx5_lag_add(mdev, netdev);
+ mlx5_lag_add_netdev(mdev, netdev);
priv->events_nb.notifier_call = uplink_rep_async_event;
mlx5_notifier_register(mdev, &priv->events_nb);
mlx5e_dcbnl_initialize(priv);
mlx5e_dcbnl_init_app(priv);
mlx5e_rep_neigh_init(rpriv);
+ mlx5e_rep_bridge_init(priv);
netdev->wanted_features |= NETIF_F_HW_TC;
netif_device_detach(priv->netdev);
rtnl_unlock();
+ mlx5e_rep_bridge_cleanup(priv);
mlx5e_rep_neigh_cleanup(rpriv);
mlx5e_dcbnl_delete_app(priv);
mlx5_notifier_unregister(mdev, &priv->events_nb);
mlx5e_rep_tc_disable(priv);
- mlx5_lag_remove(mdev);
+ mlx5_lag_remove_netdev(mdev, priv->netdev);
}
static MLX5E_DEFINE_STATS_GRP(sw_rep, 0);
void mlx5e_rep_queue_neigh_stats_work(struct mlx5e_priv *priv);
-bool mlx5e_eswitch_vf_rep(struct net_device *netdev);
-bool mlx5e_eswitch_uplink_rep(struct net_device *netdev);
-static inline bool mlx5e_eswitch_rep(struct net_device *netdev)
+bool mlx5e_eswitch_vf_rep(const struct net_device *netdev);
+bool mlx5e_eswitch_uplink_rep(const struct net_device *netdev);
+static inline bool mlx5e_eswitch_rep(const struct net_device *netdev)
{
return mlx5e_eswitch_vf_rep(netdev) ||
mlx5e_eswitch_uplink_rep(netdev);
if (mlx5_wq_cyc_missing(wq) < wqe_bulk)
return false;
+ if (rq->page_pool)
+ page_pool_nid_changed(rq->page_pool, numa_mem_id());
+
do {
u16 head = mlx5_wq_cyc_get_head(wq);
if (likely(missing < UMR_WQE_BULK))
return false;
+ if (rq->page_pool)
+ page_pool_nid_changed(rq->page_pool, numa_mem_id());
+
head = rq->mpwqe.actual_wq_head;
i = missing;
do {
if (rep->vlan && skb_vlan_tag_present(skb))
skb_vlan_pop(skb);
- if (!mlx5e_rep_tc_update_skb(cqe, skb, &tc_priv)) {
+ if (unlikely(!mlx5_ipsec_is_rx_flow(cqe) &&
+ !mlx5e_rep_tc_update_skb(cqe, skb, &tc_priv))) {
dev_kfree_skb_any(skb);
goto free_wqe;
}
mlx5e_complete_rx_cqe(rq, cqe, cqe_bcnt, skb);
- if (!mlx5e_rep_tc_update_skb(cqe, skb, &tc_priv)) {
+ if (unlikely(!mlx5_ipsec_is_rx_flow(cqe) &&
+ !mlx5e_rep_tc_update_skb(cqe, skb, &tc_priv))) {
dev_kfree_skb_any(skb);
goto mpwrq_cqe_out;
}
if (unlikely(!test_bit(MLX5E_RQ_STATE_ENABLED, &rq->state)))
return 0;
- if (rq->page_pool)
- page_pool_nid_changed(rq->page_pool, numa_mem_id());
-
if (rq->cqd.left) {
work_done += mlx5e_decompress_cqes_cont(rq, cqwq, 0, budget);
- if (rq->cqd.left || work_done >= budget)
+ if (work_done >= budget)
goto out;
}
[CHAIN_TO_REG] = {
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_0,
.moffset = 0,
- .mlen = 2,
+ .mlen = 16,
},
[VPORT_TO_REG] = {
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_0,
- .moffset = 2,
- .mlen = 2,
+ .moffset = 16,
+ .mlen = 16,
},
[TUNNEL_TO_REG] = {
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_1,
- .moffset = 1,
- .mlen = ((ESW_TUN_OPTS_BITS + ESW_TUN_ID_BITS) / 8),
+ .moffset = 8,
+ .mlen = ESW_TUN_OPTS_BITS + ESW_TUN_ID_BITS,
.soffset = MLX5_BYTE_OFF(fte_match_param,
misc_parameters_2.metadata_reg_c_1),
},
[NIC_CHAIN_TO_REG] = {
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_B,
.moffset = 0,
- .mlen = 2,
+ .mlen = 16,
},
[NIC_ZONE_RESTORE_TO_REG] = nic_zone_restore_to_reg_ct,
};
void
mlx5e_tc_match_to_reg_match(struct mlx5_flow_spec *spec,
enum mlx5e_tc_attr_to_reg type,
- u32 data,
+ u32 val,
u32 mask)
{
+ void *headers_c = spec->match_criteria, *headers_v = spec->match_value, *fmask, *fval;
int soffset = mlx5e_tc_attr_to_reg_mappings[type].soffset;
+ int moffset = mlx5e_tc_attr_to_reg_mappings[type].moffset;
int match_len = mlx5e_tc_attr_to_reg_mappings[type].mlen;
- void *headers_c = spec->match_criteria;
- void *headers_v = spec->match_value;
- void *fmask, *fval;
+ u32 max_mask = GENMASK(match_len - 1, 0);
+ __be32 curr_mask_be, curr_val_be;
+ u32 curr_mask, curr_val;
fmask = headers_c + soffset;
fval = headers_v + soffset;
- mask = (__force u32)(cpu_to_be32(mask)) >> (32 - (match_len * 8));
- data = (__force u32)(cpu_to_be32(data)) >> (32 - (match_len * 8));
+ memcpy(&curr_mask_be, fmask, 4);
+ memcpy(&curr_val_be, fval, 4);
+
+ curr_mask = be32_to_cpu(curr_mask_be);
+ curr_val = be32_to_cpu(curr_val_be);
+
+ //move to correct offset
+ WARN_ON(mask > max_mask);
+ mask <<= moffset;
+ val <<= moffset;
+ max_mask <<= moffset;
+
+ //zero val and mask
+ curr_mask &= ~max_mask;
+ curr_val &= ~max_mask;
+
+ //add current to mask
+ curr_mask |= mask;
+ curr_val |= val;
+
+ //back to be32 and write
+ curr_mask_be = cpu_to_be32(curr_mask);
+ curr_val_be = cpu_to_be32(curr_val);
- memcpy(fmask, &mask, match_len);
- memcpy(fval, &data, match_len);
+ memcpy(fmask, &curr_mask_be, 4);
+ memcpy(fval, &curr_val_be, 4);
spec->match_criteria_enable |= MLX5_MATCH_MISC_PARAMETERS_2;
}
void
mlx5e_tc_match_to_reg_get_match(struct mlx5_flow_spec *spec,
enum mlx5e_tc_attr_to_reg type,
- u32 *data,
+ u32 *val,
u32 *mask)
{
+ void *headers_c = spec->match_criteria, *headers_v = spec->match_value, *fmask, *fval;
int soffset = mlx5e_tc_attr_to_reg_mappings[type].soffset;
+ int moffset = mlx5e_tc_attr_to_reg_mappings[type].moffset;
int match_len = mlx5e_tc_attr_to_reg_mappings[type].mlen;
- void *headers_c = spec->match_criteria;
- void *headers_v = spec->match_value;
- void *fmask, *fval;
+ u32 max_mask = GENMASK(match_len - 1, 0);
+ __be32 curr_mask_be, curr_val_be;
+ u32 curr_mask, curr_val;
fmask = headers_c + soffset;
fval = headers_v + soffset;
- memcpy(mask, fmask, match_len);
- memcpy(data, fval, match_len);
+ memcpy(&curr_mask_be, fmask, 4);
+ memcpy(&curr_val_be, fval, 4);
+
+ curr_mask = be32_to_cpu(curr_mask_be);
+ curr_val = be32_to_cpu(curr_val_be);
- *mask = be32_to_cpu((__force __be32)(*mask << (32 - (match_len * 8))));
- *data = be32_to_cpu((__force __be32)(*data << (32 - (match_len * 8))));
+ *mask = (curr_mask >> moffset) & max_mask;
+ *val = (curr_val >> moffset) & max_mask;
}
int
(mod_hdr_acts->num_actions * MLX5_MH_ACT_SZ);
/* Firmware has 5bit length field and 0 means 32bits */
- if (mlen == 4)
+ if (mlen == 32)
mlen = 0;
MLX5_SET(set_action_in, modact, action_type, MLX5_ACTION_TYPE_SET);
MLX5_SET(set_action_in, modact, field, mfield);
- MLX5_SET(set_action_in, modact, offset, moffset * 8);
- MLX5_SET(set_action_in, modact, length, mlen * 8);
+ MLX5_SET(set_action_in, modact, offset, moffset);
+ MLX5_SET(set_action_in, modact, length, mlen);
MLX5_SET(set_action_in, modact, data, data);
err = mod_hdr_acts->num_actions;
mod_hdr_acts->num_actions++;
modact = mod_hdr_acts->actions + (act_id * MLX5_MH_ACT_SZ);
/* Firmware has 5bit length field and 0 means 32bits */
- if (mlen == 4)
+ if (mlen == 32)
mlen = 0;
MLX5_SET(set_action_in, modact, action_type, MLX5_ACTION_TYPE_SET);
MLX5_SET(set_action_in, modact, field, mfield);
- MLX5_SET(set_action_in, modact, offset, moffset * 8);
- MLX5_SET(set_action_in, modact, length, mlen * 8);
+ MLX5_SET(set_action_in, modact, offset, moffset);
+ MLX5_SET(set_action_in, modact, length, mlen);
MLX5_SET(set_action_in, modact, data, data);
}
struct netlink_ext_ack *extack)
{
struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
- struct net_device *out_dev, *encap_dev = NULL;
struct mlx5e_tc_flow_parse_attr *parse_attr;
struct mlx5_flow_attr *attr = flow->attr;
bool vf_tun = false, encap_valid = true;
+ struct net_device *encap_dev = NULL;
struct mlx5_esw_flow_attr *esw_attr;
struct mlx5_fc *counter = NULL;
struct mlx5e_rep_priv *rpriv;
esw_attr = attr->esw_attr;
for (out_index = 0; out_index < MLX5_MAX_FLOW_FWD_VPORTS; out_index++) {
+ struct net_device *out_dev;
int mirred_ifindex;
if (!(esw_attr->dests[out_index].flags & MLX5_ESW_DEST_ENCAP))
continue;
mirred_ifindex = parse_attr->mirred_ifindex[out_index];
- out_dev = __dev_get_by_index(dev_net(priv->netdev),
- mirred_ifindex);
+ out_dev = dev_get_by_index(dev_net(priv->netdev), mirred_ifindex);
+ if (!out_dev) {
+ NL_SET_ERR_MSG_MOD(extack, "Requested mirred device not found");
+ err = -ENODEV;
+ goto err_out;
+ }
err = mlx5e_attach_encap(priv, flow, out_dev, out_index,
extack, &encap_dev, &encap_valid);
+ dev_put(out_dev);
if (err)
goto err_out;
esw_attr->dests[out_index].mdev = out_priv->mdev;
}
+ if (vf_tun && esw_attr->out_count > 1) {
+ NL_SET_ERR_MSG_MOD(extack, "VF tunnel encap with mirroring is not supported");
+ err = -EOPNOTSUPP;
+ goto err_out;
+ }
+
err = mlx5_eswitch_add_vlan_action(esw, attr);
if (err)
goto err_out;
misc_parameters_3);
struct flow_rule *rule = flow_cls_offload_flow_rule(f);
struct flow_dissector *dissector = rule->match.dissector;
+ enum fs_flow_table_type fs_type;
u16 addr_type = 0;
u8 ip_proto = 0;
u8 *match_level;
int err;
+ fs_type = mlx5e_is_eswitch_flow(flow) ? FS_FT_FDB : FS_FT_NIC_RX;
match_level = outer_match_level;
if (dissector->used_keys &
if (match.mask->vlan_id ||
match.mask->vlan_priority ||
match.mask->vlan_tpid) {
+ if (!MLX5_CAP_FLOWTABLE_TYPE(priv->mdev, ft_field_support.outer_second_vid,
+ fs_type)) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "Matching on CVLAN is not supported");
+ return -EOPNOTSUPP;
+ }
+
if (match.key->vlan_tpid == htons(ETH_P_8021AD)) {
MLX5_SET(fte_match_set_misc, misc_c,
outer_second_svlan_tag, 1);
if (err)
return err;
- *out_dev = dev_get_by_index_rcu(dev_net(vlan_dev),
- dev_get_iflink(vlan_dev));
+ rcu_read_lock();
+ *out_dev = dev_get_by_index_rcu(dev_net(vlan_dev), dev_get_iflink(vlan_dev));
+ rcu_read_unlock();
+ if (!*out_dev)
+ return -ENODEV;
+
if (is_vlan_dev(*out_dev))
err = add_vlan_push_action(priv, attr, out_dev, action);
list_for_each_entry_safe(hpe, tmp, &init_wait_list, dead_peer_wait_list) {
wait_for_completion(&hpe->res_ready);
if (!IS_ERR_OR_NULL(hpe->hp) && hpe->peer_vhca_id == peer_vhca_id)
- hpe->hp->pair->peer_gone = true;
+ mlx5_core_hairpin_clear_dead_peer(hpe->hp->pair);
mlx5e_hairpin_put(priv, hpe);
}
if (mapped_obj.type == MLX5_MAPPED_OBJ_CHAIN) {
chain = mapped_obj.chain;
- tc_skb_ext = skb_ext_add(skb, TC_SKB_EXT);
+ tc_skb_ext = tc_skb_ext_alloc(skb);
if (WARN_ON(!tc_skb_ext))
return false;
tc_skb_ext->chain = chain;
- zone_restore_id = (reg_b >> REG_MAPPING_SHIFT(NIC_ZONE_RESTORE_TO_REG)) &
+ zone_restore_id = (reg_b >> REG_MAPPING_MOFFSET(NIC_ZONE_RESTORE_TO_REG)) &
ESW_ZONE_ID_MASK;
if (!mlx5e_tc_ct_restore_flow(tc->ct, skb,
*/
#define TUNNEL_INFO_BITS 12
#define TUNNEL_INFO_BITS_MASK GENMASK(TUNNEL_INFO_BITS - 1, 0)
-#define ENC_OPTS_BITS 12
+#define ENC_OPTS_BITS 11
#define ENC_OPTS_BITS_MASK GENMASK(ENC_OPTS_BITS - 1, 0)
#define TUNNEL_ID_BITS (TUNNEL_INFO_BITS + ENC_OPTS_BITS)
#define TUNNEL_ID_MASK GENMASK(TUNNEL_ID_BITS - 1, 0)
void mlx5e_put_flow_list(struct mlx5e_priv *priv, struct list_head *flow_list);
struct mlx5e_neigh_hash_entry;
+struct mlx5e_encap_entry *
+mlx5e_get_next_init_encap(struct mlx5e_neigh_hash_entry *nhe,
+ struct mlx5e_encap_entry *e);
void mlx5e_tc_update_neigh_used_value(struct mlx5e_neigh_hash_entry *nhe);
void mlx5e_tc_reoffload_flows_work(struct work_struct *work);
struct mlx5e_tc_attr_to_reg_mapping {
int mfield; /* rewrite field */
- int moffset; /* offset of mfield */
- int mlen; /* bytes to rewrite/match */
+ int moffset; /* bit offset of mfield */
+ int mlen; /* bits to rewrite/match */
- int soffset; /* offset of spec for match */
+ int soffset; /* byte offset of spec for match */
};
extern struct mlx5e_tc_attr_to_reg_mapping mlx5e_tc_attr_to_reg_mappings[];
#include <linux/tcp.h>
#include <linux/if_vlan.h>
-#include <linux/ptp_classify.h>
#include <net/geneve.h>
#include <net/dsfield.h>
#include "en.h"
}
#endif
-static bool mlx5e_use_ptpsq(struct sk_buff *skb)
-{
- struct flow_keys fk;
-
- if (!skb_flow_dissect_flow_keys(skb, &fk, 0))
- return false;
-
- if (fk.basic.n_proto == htons(ETH_P_1588))
- return true;
-
- if (fk.basic.n_proto != htons(ETH_P_IP) &&
- fk.basic.n_proto != htons(ETH_P_IPV6))
- return false;
-
- return (fk.basic.ip_proto == IPPROTO_UDP &&
- fk.ports.dst == htons(PTP_EV_PORT));
-}
-
static u16 mlx5e_select_ptpsq(struct net_device *dev, struct sk_buff *skb)
{
struct mlx5e_priv *priv = netdev_priv(dev);
}
ptp_channel = READ_ONCE(priv->channels.ptp);
- if (unlikely(ptp_channel) &&
- test_bit(MLX5E_PTP_STATE_TX, ptp_channel->state) &&
- mlx5e_use_ptpsq(skb))
+ if (unlikely(ptp_channel &&
+ test_bit(MLX5E_PTP_STATE_TX, ptp_channel->state) &&
+ mlx5e_use_ptpsq(skb)))
return mlx5e_select_ptpsq(dev, skb);
txq_ix = netdev_pick_tx(dev, skb, NULL);
mlx5e_tx_mpwqe_session_complete(sq);
}
-static bool mlx5e_txwqe_build_eseg(struct mlx5e_priv *priv, struct mlx5e_txqsq *sq,
+static void mlx5e_txwqe_build_eseg(struct mlx5e_priv *priv, struct mlx5e_txqsq *sq,
struct sk_buff *skb, struct mlx5e_accel_tx_state *accel,
struct mlx5_wqe_eth_seg *eseg, u16 ihs)
{
- if (unlikely(!mlx5e_accel_tx_eseg(priv, skb, eseg, ihs)))
- return false;
-
+ mlx5e_accel_tx_eseg(priv, skb, eseg, ihs);
mlx5e_txwqe_build_eseg_csum(sq, skb, accel, eseg);
-
- return true;
}
netdev_tx_t mlx5e_xmit(struct sk_buff *skb, struct net_device *dev)
if (mlx5e_tx_skb_supports_mpwqe(skb, &attr)) {
struct mlx5_wqe_eth_seg eseg = {};
- if (unlikely(!mlx5e_txwqe_build_eseg(priv, sq, skb, &accel, &eseg,
- attr.ihs)))
- return NETDEV_TX_OK;
-
+ mlx5e_txwqe_build_eseg(priv, sq, skb, &accel, &eseg, attr.ihs);
mlx5e_sq_xmit_mpwqe(sq, skb, &eseg, netdev_xmit_more());
return NETDEV_TX_OK;
}
/* May update the WQE, but may not post other WQEs. */
mlx5e_accel_tx_finish(sq, wqe, &accel,
(struct mlx5_wqe_inline_seg *)(wqe->data + wqe_attr.ds_cnt_inl));
- if (unlikely(!mlx5e_txwqe_build_eseg(priv, sq, skb, &accel, &wqe->eth, attr.ihs)))
- return NETDEV_TX_OK;
-
+ mlx5e_txwqe_build_eseg(priv, sq, skb, &accel, &wqe->eth, attr.ihs);
mlx5e_sq_xmit_wqe(sq, skb, &attr, &wqe_attr, wqe, pi, netdev_xmit_more());
return NETDEV_TX_OK;
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
/*
- * Copyright (c) 2013-2015, Mellanox Technologies. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
+ * Copyright (c) 2013-2021, Mellanox Technologies inc. All rights reserved.
*/
#include <linux/interrupt.h>
#include "eswitch.h"
#include "lib/clock.h"
#include "diag/fw_tracer.h"
+#include "mlx5_irq.h"
enum {
MLX5_EQE_OWNER_INIT_VAL = 0x1,
struct mutex lock; /* sync async eqs creations */
int num_comp_eqs;
struct mlx5_irq_table *irq_table;
+#ifdef CONFIG_RFS_ACCEL
+ struct cpu_rmap *rmap;
+#endif
};
#define MLX5_ASYNC_EVENT_MASK ((1ull << MLX5_EVENT_TYPE_PATH_MIG) | \
eqe = next_eqe_sw(eq);
if (!eqe)
- return 0;
+ goto out;
do {
struct mlx5_core_cq *cq;
++eq->cons_index;
} while ((++num_eqes < MLX5_EQ_POLLING_BUDGET) && (eqe = next_eqe_sw(eq)));
+
+out:
eq_update_ci(eq, 1);
if (cqn != -1)
++eq->cons_index;
} while ((++num_eqes < MLX5_EQ_POLLING_BUDGET) && (eqe = next_eqe_sw(eq)));
- eq_update_ci(eq, 1);
out:
+ eq_update_ci(eq, 1);
mlx5_eq_async_int_unlock(eq_async, recovery, &flags);
return unlikely(recovery) ? num_eqes : 0;
u32 out[MLX5_ST_SZ_DW(create_eq_out)] = {0};
u8 log_eq_stride = ilog2(MLX5_EQE_SIZE);
struct mlx5_priv *priv = &dev->priv;
- u8 vecidx = param->irq_index;
+ u16 vecidx = param->irq_index;
__be64 *pas;
void *eqc;
int inlen;
mlx5_init_fbc(eq->frag_buf.frags, log_eq_stride, log_eq_size, &eq->fbc);
init_eq_buf(eq);
+ eq->irq = mlx5_irq_request(dev, vecidx, param->affinity);
+ if (IS_ERR(eq->irq)) {
+ err = PTR_ERR(eq->irq);
+ goto err_buf;
+ }
+
+ vecidx = mlx5_irq_get_index(eq->irq);
inlen = MLX5_ST_SZ_BYTES(create_eq_in) +
MLX5_FLD_SZ_BYTES(create_eq_in, pas[0]) * eq->frag_buf.npages;
in = kvzalloc(inlen, GFP_KERNEL);
if (!in) {
err = -ENOMEM;
- goto err_buf;
+ goto err_irq;
}
pas = (__be64 *)MLX5_ADDR_OF(create_eq_in, in, pas);
err_in:
kvfree(in);
+err_irq:
+ mlx5_irq_release(eq->irq);
err_buf:
mlx5_frag_buf_free(dev, &eq->frag_buf);
return err;
int mlx5_eq_enable(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
struct notifier_block *nb)
{
- struct mlx5_eq_table *eq_table = dev->priv.eq_table;
int err;
- err = mlx5_irq_attach_nb(eq_table->irq_table, eq->vecidx, nb);
+ err = mlx5_irq_attach_nb(eq->irq, nb);
if (!err)
eq_update_ci(eq, 1);
void mlx5_eq_disable(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
struct notifier_block *nb)
{
- struct mlx5_eq_table *eq_table = dev->priv.eq_table;
-
- mlx5_irq_detach_nb(eq_table->irq_table, eq->vecidx, nb);
+ mlx5_irq_detach_nb(eq->irq, nb);
}
EXPORT_SYMBOL(mlx5_eq_disable);
if (err)
mlx5_core_warn(dev, "failed to destroy a previously created eq: eqn %d\n",
eq->eqn);
- synchronize_irq(eq->irqn);
+ mlx5_irq_release(eq->irq);
mlx5_frag_buf_free(dev, &eq->frag_buf);
-
return err;
}
int err;
mutex_lock(&eq_table->lock);
- /* Async EQs must share irq index 0 */
- if (param->irq_index != 0) {
- err = -EINVAL;
- goto unlock;
- }
-
err = create_map_eq(dev, eq, param);
-unlock:
mutex_unlock(&eq_table->lock);
return err;
}
eq->irq_nb.notifier_call = mlx5_eq_async_int;
spin_lock_init(&eq->lock);
+ if (!zalloc_cpumask_var(¶m->affinity, GFP_KERNEL))
+ return -ENOMEM;
err = create_async_eq(dev, &eq->core, param);
+ free_cpumask_var(param->affinity);
if (err) {
mlx5_core_warn(dev, "failed to create %s EQ %d\n", name, err);
return err;
mlx5_eq_notifier_register(dev, &table->cq_err_nb);
param = (struct mlx5_eq_param) {
- .irq_index = 0,
.nent = MLX5_NUM_CMD_EQE,
.mask[0] = 1ull << MLX5_EVENT_TYPE_CMD,
};
mlx5_cmd_allowed_opcode(dev, CMD_ALLOWED_OPCODE_ALL);
param = (struct mlx5_eq_param) {
- .irq_index = 0,
.nent = MLX5_NUM_ASYNC_EQE,
};
goto err2;
param = (struct mlx5_eq_param) {
- .irq_index = 0,
.nent = /* TODO: sriov max_vf + */ 1,
.mask[0] = 1ull << MLX5_EVENT_TYPE_PAGE_REQUEST,
};
struct mlx5_eq *eq = kvzalloc(sizeof(*eq), GFP_KERNEL);
int err;
+ if (!cpumask_available(param->affinity))
+ return ERR_PTR(-EINVAL);
+
if (!eq)
return ERR_PTR(-ENOMEM);
.irq_index = vecidx,
.nent = nent,
};
- err = create_map_eq(dev, &eq->core, ¶m);
- if (err) {
- kfree(eq);
- goto clean;
+
+ if (!zalloc_cpumask_var(¶m.affinity, GFP_KERNEL)) {
+ err = -ENOMEM;
+ goto clean_eq;
}
+ cpumask_set_cpu(cpumask_local_spread(i, dev->priv.numa_node),
+ param.affinity);
+ err = create_map_eq(dev, &eq->core, ¶m);
+ free_cpumask_var(param.affinity);
+ if (err)
+ goto clean_eq;
err = mlx5_eq_enable(dev, &eq->core, &eq->irq_nb);
if (err) {
destroy_unmap_eq(dev, &eq->core);
- kfree(eq);
- goto clean;
+ goto clean_eq;
}
mlx5_core_dbg(dev, "allocated completion EQN %d\n", eq->core.eqn);
}
return 0;
-
+clean_eq:
+ kfree(eq);
clean:
destroy_comp_eqs(dev);
return err;
struct cpumask *
mlx5_comp_irq_get_affinity_mask(struct mlx5_core_dev *dev, int vector)
{
- int vecidx = vector + MLX5_IRQ_VEC_COMP_BASE;
+ struct mlx5_eq_table *table = dev->priv.eq_table;
+ struct mlx5_eq_comp *eq, *n;
+ int i = 0;
+
+ list_for_each_entry_safe(eq, n, &table->comp_eqs_list, list) {
+ if (i++ == vector)
+ break;
+ }
- return mlx5_irq_get_affinity_mask(dev->priv.eq_table->irq_table,
- vecidx);
+ return mlx5_irq_get_affinity_mask(eq->core.irq);
}
EXPORT_SYMBOL(mlx5_comp_irq_get_affinity_mask);
#ifdef CONFIG_RFS_ACCEL
struct cpu_rmap *mlx5_eq_table_get_rmap(struct mlx5_core_dev *dev)
{
- return mlx5_irq_get_rmap(dev->priv.eq_table->irq_table);
+ return dev->priv.eq_table->rmap;
}
#endif
return ERR_PTR(-ENOENT);
}
+static void clear_rmap(struct mlx5_core_dev *dev)
+{
+#ifdef CONFIG_RFS_ACCEL
+ struct mlx5_eq_table *eq_table = dev->priv.eq_table;
+
+ free_irq_cpu_rmap(eq_table->rmap);
+#endif
+}
+
+static int set_rmap(struct mlx5_core_dev *mdev)
+{
+ int err = 0;
+#ifdef CONFIG_RFS_ACCEL
+ struct mlx5_eq_table *eq_table = mdev->priv.eq_table;
+ int vecidx;
+
+ eq_table->rmap = alloc_irq_cpu_rmap(eq_table->num_comp_eqs);
+ if (!eq_table->rmap) {
+ err = -ENOMEM;
+ mlx5_core_err(mdev, "Failed to allocate cpu_rmap. err %d", err);
+ goto err_out;
+ }
+
+ vecidx = MLX5_IRQ_VEC_COMP_BASE;
+ for (; vecidx < eq_table->num_comp_eqs + MLX5_IRQ_VEC_COMP_BASE;
+ vecidx++) {
+ err = irq_cpu_rmap_add(eq_table->rmap,
+ pci_irq_vector(mdev->pdev, vecidx));
+ if (err) {
+ mlx5_core_err(mdev, "irq_cpu_rmap_add failed. err %d",
+ err);
+ goto err_irq_cpu_rmap_add;
+ }
+ }
+ return 0;
+
+err_irq_cpu_rmap_add:
+ clear_rmap(mdev);
+err_out:
+#endif
+ return err;
+}
+
/* This function should only be called after mlx5_cmd_force_teardown_hca */
void mlx5_core_eq_free_irqs(struct mlx5_core_dev *dev)
{
struct mlx5_eq_table *table = dev->priv.eq_table;
mutex_lock(&table->lock); /* sync with create/destroy_async_eq */
+ if (!mlx5_core_is_sf(dev))
+ clear_rmap(dev);
mlx5_irq_table_destroy(dev);
mutex_unlock(&table->lock);
}
int num_eqs = MLX5_CAP_GEN(dev, max_num_eqs) ?
MLX5_CAP_GEN(dev, max_num_eqs) :
1 << MLX5_CAP_GEN(dev, log_max_eq);
+ int max_eqs_sf;
int err;
eq_table->num_comp_eqs =
min_t(int,
- mlx5_irq_get_num_comp(eq_table->irq_table),
+ mlx5_irq_table_get_num_comp(eq_table->irq_table),
num_eqs - MLX5_MAX_ASYNC_EQS);
+ if (mlx5_core_is_sf(dev)) {
+ max_eqs_sf = min_t(int, MLX5_COMP_EQS_PER_SF,
+ mlx5_irq_table_get_sfs_vec(eq_table->irq_table));
+ eq_table->num_comp_eqs = min_t(int, eq_table->num_comp_eqs,
+ max_eqs_sf);
+ }
err = create_async_eqs(dev);
if (err) {
goto err_async_eqs;
}
+ if (!mlx5_core_is_sf(dev)) {
+ /* rmap is a mapping between irq number and queue number.
+ * each irq can be assign only to a single rmap.
+ * since SFs share IRQs, rmap mapping cannot function correctly
+ * for irqs that are shared for different core/netdev RX rings.
+ * Hence we don't allow netdev rmap for SFs
+ */
+ err = set_rmap(dev);
+ if (err)
+ goto err_rmap;
+ }
+
err = create_comp_eqs(dev);
if (err) {
mlx5_core_err(dev, "Failed to create completion EQs\n");
return 0;
err_comp_eqs:
+ if (!mlx5_core_is_sf(dev))
+ clear_rmap(dev);
+err_rmap:
destroy_async_eqs(dev);
err_async_eqs:
return err;
void mlx5_eq_table_destroy(struct mlx5_core_dev *dev)
{
+ if (!mlx5_core_is_sf(dev))
+ clear_rmap(dev);
destroy_comp_eqs(dev);
destroy_async_eqs(dev);
}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2021 Mellanox Technologies. */
+
+#include <linux/list.h>
+#include <linux/notifier.h>
+#include <net/netevent.h>
+#include <net/switchdev.h>
+#include "bridge.h"
+#include "eswitch.h"
+#include "bridge_priv.h"
+#define CREATE_TRACE_POINTS
+#include "diag/bridge_tracepoint.h"
+
+#define MLX5_ESW_BRIDGE_INGRESS_TABLE_SIZE 64000
+#define MLX5_ESW_BRIDGE_INGRESS_TABLE_VLAN_GRP_IDX_FROM 0
+#define MLX5_ESW_BRIDGE_INGRESS_TABLE_VLAN_GRP_IDX_TO (MLX5_ESW_BRIDGE_INGRESS_TABLE_SIZE / 4 - 1)
+#define MLX5_ESW_BRIDGE_INGRESS_TABLE_FILTER_GRP_IDX_FROM \
+ (MLX5_ESW_BRIDGE_INGRESS_TABLE_VLAN_GRP_IDX_TO + 1)
+#define MLX5_ESW_BRIDGE_INGRESS_TABLE_FILTER_GRP_IDX_TO \
+ (MLX5_ESW_BRIDGE_INGRESS_TABLE_SIZE / 2 - 1)
+#define MLX5_ESW_BRIDGE_INGRESS_TABLE_MAC_GRP_IDX_FROM \
+ (MLX5_ESW_BRIDGE_INGRESS_TABLE_FILTER_GRP_IDX_TO + 1)
+#define MLX5_ESW_BRIDGE_INGRESS_TABLE_MAC_GRP_IDX_TO (MLX5_ESW_BRIDGE_INGRESS_TABLE_SIZE - 1)
+
+#define MLX5_ESW_BRIDGE_EGRESS_TABLE_SIZE 64000
+#define MLX5_ESW_BRIDGE_EGRESS_TABLE_VLAN_GRP_IDX_FROM 0
+#define MLX5_ESW_BRIDGE_EGRESS_TABLE_VLAN_GRP_IDX_TO (MLX5_ESW_BRIDGE_EGRESS_TABLE_SIZE / 2 - 1)
+#define MLX5_ESW_BRIDGE_EGRESS_TABLE_MAC_GRP_IDX_FROM \
+ (MLX5_ESW_BRIDGE_EGRESS_TABLE_VLAN_GRP_IDX_TO + 1)
+#define MLX5_ESW_BRIDGE_EGRESS_TABLE_MAC_GRP_IDX_TO (MLX5_ESW_BRIDGE_EGRESS_TABLE_SIZE - 1)
+
+#define MLX5_ESW_BRIDGE_SKIP_TABLE_SIZE 0
+
+enum {
+ MLX5_ESW_BRIDGE_LEVEL_INGRESS_TABLE,
+ MLX5_ESW_BRIDGE_LEVEL_EGRESS_TABLE,
+ MLX5_ESW_BRIDGE_LEVEL_SKIP_TABLE,
+};
+
+static const struct rhashtable_params fdb_ht_params = {
+ .key_offset = offsetof(struct mlx5_esw_bridge_fdb_entry, key),
+ .key_len = sizeof(struct mlx5_esw_bridge_fdb_key),
+ .head_offset = offsetof(struct mlx5_esw_bridge_fdb_entry, ht_node),
+ .automatic_shrinking = true,
+};
+
+enum {
+ MLX5_ESW_BRIDGE_VLAN_FILTERING_FLAG = BIT(0),
+};
+
+struct mlx5_esw_bridge {
+ int ifindex;
+ int refcnt;
+ struct list_head list;
+ struct mlx5_esw_bridge_offloads *br_offloads;
+
+ struct list_head fdb_list;
+ struct rhashtable fdb_ht;
+ struct xarray vports;
+
+ struct mlx5_flow_table *egress_ft;
+ struct mlx5_flow_group *egress_vlan_fg;
+ struct mlx5_flow_group *egress_mac_fg;
+ unsigned long ageing_time;
+ u32 flags;
+};
+
+static void
+mlx5_esw_bridge_fdb_offload_notify(struct net_device *dev, const unsigned char *addr, u16 vid,
+ unsigned long val)
+{
+ struct switchdev_notifier_fdb_info send_info;
+
+ send_info.addr = addr;
+ send_info.vid = vid;
+ send_info.offloaded = true;
+ call_switchdev_notifiers(val, dev, &send_info.info, NULL);
+}
+
+static struct mlx5_flow_table *
+mlx5_esw_bridge_table_create(int max_fte, u32 level, struct mlx5_eswitch *esw)
+{
+ struct mlx5_flow_table_attr ft_attr = {};
+ struct mlx5_core_dev *dev = esw->dev;
+ struct mlx5_flow_namespace *ns;
+ struct mlx5_flow_table *fdb;
+
+ ns = mlx5_get_flow_namespace(dev, MLX5_FLOW_NAMESPACE_FDB);
+ if (!ns) {
+ esw_warn(dev, "Failed to get FDB namespace\n");
+ return ERR_PTR(-ENOENT);
+ }
+
+ ft_attr.flags = MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT;
+ ft_attr.max_fte = max_fte;
+ ft_attr.level = level;
+ ft_attr.prio = FDB_BR_OFFLOAD;
+ fdb = mlx5_create_flow_table(ns, &ft_attr);
+ if (IS_ERR(fdb))
+ esw_warn(dev, "Failed to create bridge FDB Table (err=%ld)\n", PTR_ERR(fdb));
+
+ return fdb;
+}
+
+static struct mlx5_flow_group *
+mlx5_esw_bridge_ingress_vlan_fg_create(struct mlx5_eswitch *esw, struct mlx5_flow_table *ingress_ft)
+{
+ int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
+ struct mlx5_flow_group *fg;
+ u32 *in, *match;
+
+ in = kvzalloc(inlen, GFP_KERNEL);
+ if (!in)
+ return ERR_PTR(-ENOMEM);
+
+ MLX5_SET(create_flow_group_in, in, match_criteria_enable,
+ MLX5_MATCH_OUTER_HEADERS | MLX5_MATCH_MISC_PARAMETERS_2);
+ match = MLX5_ADDR_OF(create_flow_group_in, in, match_criteria);
+
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.smac_47_16);
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.smac_15_0);
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.cvlan_tag);
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.first_vid);
+
+ MLX5_SET(fte_match_param, match, misc_parameters_2.metadata_reg_c_0,
+ mlx5_eswitch_get_vport_metadata_mask());
+
+ MLX5_SET(create_flow_group_in, in, start_flow_index,
+ MLX5_ESW_BRIDGE_INGRESS_TABLE_VLAN_GRP_IDX_FROM);
+ MLX5_SET(create_flow_group_in, in, end_flow_index,
+ MLX5_ESW_BRIDGE_INGRESS_TABLE_VLAN_GRP_IDX_TO);
+
+ fg = mlx5_create_flow_group(ingress_ft, in);
+ kvfree(in);
+ if (IS_ERR(fg))
+ esw_warn(esw->dev,
+ "Failed to create VLAN flow group for bridge ingress table (err=%ld)\n",
+ PTR_ERR(fg));
+
+ return fg;
+}
+
+static struct mlx5_flow_group *
+mlx5_esw_bridge_ingress_filter_fg_create(struct mlx5_eswitch *esw,
+ struct mlx5_flow_table *ingress_ft)
+{
+ int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
+ struct mlx5_flow_group *fg;
+ u32 *in, *match;
+
+ in = kvzalloc(inlen, GFP_KERNEL);
+ if (!in)
+ return ERR_PTR(-ENOMEM);
+
+ MLX5_SET(create_flow_group_in, in, match_criteria_enable,
+ MLX5_MATCH_OUTER_HEADERS | MLX5_MATCH_MISC_PARAMETERS_2);
+ match = MLX5_ADDR_OF(create_flow_group_in, in, match_criteria);
+
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.smac_47_16);
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.smac_15_0);
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.cvlan_tag);
+
+ MLX5_SET(fte_match_param, match, misc_parameters_2.metadata_reg_c_0,
+ mlx5_eswitch_get_vport_metadata_mask());
+
+ MLX5_SET(create_flow_group_in, in, start_flow_index,
+ MLX5_ESW_BRIDGE_INGRESS_TABLE_FILTER_GRP_IDX_FROM);
+ MLX5_SET(create_flow_group_in, in, end_flow_index,
+ MLX5_ESW_BRIDGE_INGRESS_TABLE_FILTER_GRP_IDX_TO);
+
+ fg = mlx5_create_flow_group(ingress_ft, in);
+ if (IS_ERR(fg))
+ esw_warn(esw->dev,
+ "Failed to create bridge ingress table VLAN filter flow group (err=%ld)\n",
+ PTR_ERR(fg));
+
+ kvfree(in);
+ return fg;
+}
+
+static struct mlx5_flow_group *
+mlx5_esw_bridge_ingress_mac_fg_create(struct mlx5_eswitch *esw, struct mlx5_flow_table *ingress_ft)
+{
+ int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
+ struct mlx5_flow_group *fg;
+ u32 *in, *match;
+
+ in = kvzalloc(inlen, GFP_KERNEL);
+ if (!in)
+ return ERR_PTR(-ENOMEM);
+
+ MLX5_SET(create_flow_group_in, in, match_criteria_enable,
+ MLX5_MATCH_OUTER_HEADERS | MLX5_MATCH_MISC_PARAMETERS_2);
+ match = MLX5_ADDR_OF(create_flow_group_in, in, match_criteria);
+
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.smac_47_16);
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.smac_15_0);
+
+ MLX5_SET(fte_match_param, match, misc_parameters_2.metadata_reg_c_0,
+ mlx5_eswitch_get_vport_metadata_mask());
+
+ MLX5_SET(create_flow_group_in, in, start_flow_index,
+ MLX5_ESW_BRIDGE_INGRESS_TABLE_MAC_GRP_IDX_FROM);
+ MLX5_SET(create_flow_group_in, in, end_flow_index,
+ MLX5_ESW_BRIDGE_INGRESS_TABLE_MAC_GRP_IDX_TO);
+
+ fg = mlx5_create_flow_group(ingress_ft, in);
+ if (IS_ERR(fg))
+ esw_warn(esw->dev,
+ "Failed to create MAC flow group for bridge ingress table (err=%ld)\n",
+ PTR_ERR(fg));
+
+ kvfree(in);
+ return fg;
+}
+
+static struct mlx5_flow_group *
+mlx5_esw_bridge_egress_vlan_fg_create(struct mlx5_eswitch *esw, struct mlx5_flow_table *egress_ft)
+{
+ int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
+ struct mlx5_flow_group *fg;
+ u32 *in, *match;
+
+ in = kvzalloc(inlen, GFP_KERNEL);
+ if (!in)
+ return ERR_PTR(-ENOMEM);
+
+ MLX5_SET(create_flow_group_in, in, match_criteria_enable, MLX5_MATCH_OUTER_HEADERS);
+ match = MLX5_ADDR_OF(create_flow_group_in, in, match_criteria);
+
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.dmac_47_16);
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.dmac_15_0);
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.cvlan_tag);
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.first_vid);
+
+ MLX5_SET(create_flow_group_in, in, start_flow_index,
+ MLX5_ESW_BRIDGE_EGRESS_TABLE_VLAN_GRP_IDX_FROM);
+ MLX5_SET(create_flow_group_in, in, end_flow_index,
+ MLX5_ESW_BRIDGE_EGRESS_TABLE_VLAN_GRP_IDX_TO);
+
+ fg = mlx5_create_flow_group(egress_ft, in);
+ if (IS_ERR(fg))
+ esw_warn(esw->dev,
+ "Failed to create VLAN flow group for bridge egress table (err=%ld)\n",
+ PTR_ERR(fg));
+ kvfree(in);
+ return fg;
+}
+
+static struct mlx5_flow_group *
+mlx5_esw_bridge_egress_mac_fg_create(struct mlx5_eswitch *esw, struct mlx5_flow_table *egress_ft)
+{
+ int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
+ struct mlx5_flow_group *fg;
+ u32 *in, *match;
+
+ in = kvzalloc(inlen, GFP_KERNEL);
+ if (!in)
+ return ERR_PTR(-ENOMEM);
+
+ MLX5_SET(create_flow_group_in, in, match_criteria_enable, MLX5_MATCH_OUTER_HEADERS);
+ match = MLX5_ADDR_OF(create_flow_group_in, in, match_criteria);
+
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.dmac_47_16);
+ MLX5_SET_TO_ONES(fte_match_param, match, outer_headers.dmac_15_0);
+
+ MLX5_SET(create_flow_group_in, in, start_flow_index,
+ MLX5_ESW_BRIDGE_EGRESS_TABLE_MAC_GRP_IDX_FROM);
+ MLX5_SET(create_flow_group_in, in, end_flow_index,
+ MLX5_ESW_BRIDGE_EGRESS_TABLE_MAC_GRP_IDX_TO);
+
+ fg = mlx5_create_flow_group(egress_ft, in);
+ if (IS_ERR(fg))
+ esw_warn(esw->dev,
+ "Failed to create bridge egress table MAC flow group (err=%ld)\n",
+ PTR_ERR(fg));
+ kvfree(in);
+ return fg;
+}
+
+static int
+mlx5_esw_bridge_ingress_table_init(struct mlx5_esw_bridge_offloads *br_offloads)
+{
+ struct mlx5_flow_group *mac_fg, *filter_fg, *vlan_fg;
+ struct mlx5_flow_table *ingress_ft, *skip_ft;
+ int err;
+
+ if (!mlx5_eswitch_vport_match_metadata_enabled(br_offloads->esw))
+ return -EOPNOTSUPP;
+
+ ingress_ft = mlx5_esw_bridge_table_create(MLX5_ESW_BRIDGE_INGRESS_TABLE_SIZE,
+ MLX5_ESW_BRIDGE_LEVEL_INGRESS_TABLE,
+ br_offloads->esw);
+ if (IS_ERR(ingress_ft))
+ return PTR_ERR(ingress_ft);
+
+ skip_ft = mlx5_esw_bridge_table_create(MLX5_ESW_BRIDGE_SKIP_TABLE_SIZE,
+ MLX5_ESW_BRIDGE_LEVEL_SKIP_TABLE,
+ br_offloads->esw);
+ if (IS_ERR(skip_ft)) {
+ err = PTR_ERR(skip_ft);
+ goto err_skip_tbl;
+ }
+
+ vlan_fg = mlx5_esw_bridge_ingress_vlan_fg_create(br_offloads->esw, ingress_ft);
+ if (IS_ERR(vlan_fg)) {
+ err = PTR_ERR(vlan_fg);
+ goto err_vlan_fg;
+ }
+
+ filter_fg = mlx5_esw_bridge_ingress_filter_fg_create(br_offloads->esw, ingress_ft);
+ if (IS_ERR(filter_fg)) {
+ err = PTR_ERR(filter_fg);
+ goto err_filter_fg;
+ }
+
+ mac_fg = mlx5_esw_bridge_ingress_mac_fg_create(br_offloads->esw, ingress_ft);
+ if (IS_ERR(mac_fg)) {
+ err = PTR_ERR(mac_fg);
+ goto err_mac_fg;
+ }
+
+ br_offloads->ingress_ft = ingress_ft;
+ br_offloads->skip_ft = skip_ft;
+ br_offloads->ingress_vlan_fg = vlan_fg;
+ br_offloads->ingress_filter_fg = filter_fg;
+ br_offloads->ingress_mac_fg = mac_fg;
+ return 0;
+
+err_mac_fg:
+ mlx5_destroy_flow_group(filter_fg);
+err_filter_fg:
+ mlx5_destroy_flow_group(vlan_fg);
+err_vlan_fg:
+ mlx5_destroy_flow_table(skip_ft);
+err_skip_tbl:
+ mlx5_destroy_flow_table(ingress_ft);
+ return err;
+}
+
+static void
+mlx5_esw_bridge_ingress_table_cleanup(struct mlx5_esw_bridge_offloads *br_offloads)
+{
+ mlx5_destroy_flow_group(br_offloads->ingress_mac_fg);
+ br_offloads->ingress_mac_fg = NULL;
+ mlx5_destroy_flow_group(br_offloads->ingress_filter_fg);
+ br_offloads->ingress_filter_fg = NULL;
+ mlx5_destroy_flow_group(br_offloads->ingress_vlan_fg);
+ br_offloads->ingress_vlan_fg = NULL;
+ mlx5_destroy_flow_table(br_offloads->skip_ft);
+ br_offloads->skip_ft = NULL;
+ mlx5_destroy_flow_table(br_offloads->ingress_ft);
+ br_offloads->ingress_ft = NULL;
+}
+
+static int
+mlx5_esw_bridge_egress_table_init(struct mlx5_esw_bridge_offloads *br_offloads,
+ struct mlx5_esw_bridge *bridge)
+{
+ struct mlx5_flow_group *mac_fg, *vlan_fg;
+ struct mlx5_flow_table *egress_ft;
+ int err;
+
+ egress_ft = mlx5_esw_bridge_table_create(MLX5_ESW_BRIDGE_EGRESS_TABLE_SIZE,
+ MLX5_ESW_BRIDGE_LEVEL_EGRESS_TABLE,
+ br_offloads->esw);
+ if (IS_ERR(egress_ft))
+ return PTR_ERR(egress_ft);
+
+ vlan_fg = mlx5_esw_bridge_egress_vlan_fg_create(br_offloads->esw, egress_ft);
+ if (IS_ERR(vlan_fg)) {
+ err = PTR_ERR(vlan_fg);
+ goto err_vlan_fg;
+ }
+
+ mac_fg = mlx5_esw_bridge_egress_mac_fg_create(br_offloads->esw, egress_ft);
+ if (IS_ERR(mac_fg)) {
+ err = PTR_ERR(mac_fg);
+ goto err_mac_fg;
+ }
+
+ bridge->egress_ft = egress_ft;
+ bridge->egress_vlan_fg = vlan_fg;
+ bridge->egress_mac_fg = mac_fg;
+ return 0;
+
+err_mac_fg:
+ mlx5_destroy_flow_group(vlan_fg);
+err_vlan_fg:
+ mlx5_destroy_flow_table(egress_ft);
+ return err;
+}
+
+static void
+mlx5_esw_bridge_egress_table_cleanup(struct mlx5_esw_bridge *bridge)
+{
+ mlx5_destroy_flow_group(bridge->egress_mac_fg);
+ mlx5_destroy_flow_group(bridge->egress_vlan_fg);
+ mlx5_destroy_flow_table(bridge->egress_ft);
+}
+
+static struct mlx5_flow_handle *
+mlx5_esw_bridge_ingress_flow_create(u16 vport_num, const unsigned char *addr,
+ struct mlx5_esw_bridge_vlan *vlan, u32 counter_id,
+ struct mlx5_esw_bridge *bridge)
+{
+ struct mlx5_esw_bridge_offloads *br_offloads = bridge->br_offloads;
+ struct mlx5_flow_act flow_act = {
+ .action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST | MLX5_FLOW_CONTEXT_ACTION_COUNT,
+ .flags = FLOW_ACT_NO_APPEND,
+ };
+ struct mlx5_flow_destination dests[2] = {};
+ struct mlx5_flow_spec *rule_spec;
+ struct mlx5_flow_handle *handle;
+ u8 *smac_v, *smac_c;
+
+ rule_spec = kvzalloc(sizeof(*rule_spec), GFP_KERNEL);
+ if (!rule_spec)
+ return ERR_PTR(-ENOMEM);
+
+ rule_spec->match_criteria_enable = MLX5_MATCH_OUTER_HEADERS | MLX5_MATCH_MISC_PARAMETERS_2;
+
+ smac_v = MLX5_ADDR_OF(fte_match_param, rule_spec->match_value,
+ outer_headers.smac_47_16);
+ ether_addr_copy(smac_v, addr);
+ smac_c = MLX5_ADDR_OF(fte_match_param, rule_spec->match_criteria,
+ outer_headers.smac_47_16);
+ eth_broadcast_addr(smac_c);
+
+ MLX5_SET(fte_match_param, rule_spec->match_criteria,
+ misc_parameters_2.metadata_reg_c_0, mlx5_eswitch_get_vport_metadata_mask());
+ MLX5_SET(fte_match_param, rule_spec->match_value, misc_parameters_2.metadata_reg_c_0,
+ mlx5_eswitch_get_vport_metadata_for_match(br_offloads->esw, vport_num));
+
+ if (vlan && vlan->pkt_reformat_push) {
+ flow_act.action |= MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT;
+ flow_act.pkt_reformat = vlan->pkt_reformat_push;
+ } else if (vlan) {
+ MLX5_SET_TO_ONES(fte_match_param, rule_spec->match_criteria,
+ outer_headers.cvlan_tag);
+ MLX5_SET_TO_ONES(fte_match_param, rule_spec->match_value,
+ outer_headers.cvlan_tag);
+ MLX5_SET_TO_ONES(fte_match_param, rule_spec->match_criteria,
+ outer_headers.first_vid);
+ MLX5_SET(fte_match_param, rule_spec->match_value, outer_headers.first_vid,
+ vlan->vid);
+ }
+
+ dests[0].type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
+ dests[0].ft = bridge->egress_ft;
+ dests[1].type = MLX5_FLOW_DESTINATION_TYPE_COUNTER;
+ dests[1].counter_id = counter_id;
+
+ handle = mlx5_add_flow_rules(br_offloads->ingress_ft, rule_spec, &flow_act, dests,
+ ARRAY_SIZE(dests));
+
+ kvfree(rule_spec);
+ return handle;
+}
+
+static struct mlx5_flow_handle *
+mlx5_esw_bridge_ingress_filter_flow_create(u16 vport_num, const unsigned char *addr,
+ struct mlx5_esw_bridge *bridge)
+{
+ struct mlx5_esw_bridge_offloads *br_offloads = bridge->br_offloads;
+ struct mlx5_flow_destination dest = {
+ .type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE,
+ .ft = br_offloads->skip_ft,
+ };
+ struct mlx5_flow_act flow_act = {
+ .action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
+ .flags = FLOW_ACT_NO_APPEND,
+ };
+ struct mlx5_flow_spec *rule_spec;
+ struct mlx5_flow_handle *handle;
+ u8 *smac_v, *smac_c;
+
+ rule_spec = kvzalloc(sizeof(*rule_spec), GFP_KERNEL);
+ if (!rule_spec)
+ return ERR_PTR(-ENOMEM);
+
+ rule_spec->match_criteria_enable = MLX5_MATCH_OUTER_HEADERS | MLX5_MATCH_MISC_PARAMETERS_2;
+
+ smac_v = MLX5_ADDR_OF(fte_match_param, rule_spec->match_value,
+ outer_headers.smac_47_16);
+ ether_addr_copy(smac_v, addr);
+ smac_c = MLX5_ADDR_OF(fte_match_param, rule_spec->match_criteria,
+ outer_headers.smac_47_16);
+ eth_broadcast_addr(smac_c);
+
+ MLX5_SET(fte_match_param, rule_spec->match_criteria,
+ misc_parameters_2.metadata_reg_c_0, mlx5_eswitch_get_vport_metadata_mask());
+ MLX5_SET(fte_match_param, rule_spec->match_value, misc_parameters_2.metadata_reg_c_0,
+ mlx5_eswitch_get_vport_metadata_for_match(br_offloads->esw, vport_num));
+
+ MLX5_SET_TO_ONES(fte_match_param, rule_spec->match_criteria,
+ outer_headers.cvlan_tag);
+ MLX5_SET_TO_ONES(fte_match_param, rule_spec->match_value,
+ outer_headers.cvlan_tag);
+
+ handle = mlx5_add_flow_rules(br_offloads->ingress_ft, rule_spec, &flow_act, &dest, 1);
+
+ kvfree(rule_spec);
+ return handle;
+}
+
+static struct mlx5_flow_handle *
+mlx5_esw_bridge_egress_flow_create(u16 vport_num, const unsigned char *addr,
+ struct mlx5_esw_bridge_vlan *vlan,
+ struct mlx5_esw_bridge *bridge)
+{
+ struct mlx5_flow_destination dest = {
+ .type = MLX5_FLOW_DESTINATION_TYPE_VPORT,
+ .vport.num = vport_num,
+ };
+ struct mlx5_flow_act flow_act = {
+ .action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
+ .flags = FLOW_ACT_NO_APPEND,
+ };
+ struct mlx5_flow_spec *rule_spec;
+ struct mlx5_flow_handle *handle;
+ u8 *dmac_v, *dmac_c;
+
+ rule_spec = kvzalloc(sizeof(*rule_spec), GFP_KERNEL);
+ if (!rule_spec)
+ return ERR_PTR(-ENOMEM);
+
+ rule_spec->match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
+
+ dmac_v = MLX5_ADDR_OF(fte_match_param, rule_spec->match_value,
+ outer_headers.dmac_47_16);
+ ether_addr_copy(dmac_v, addr);
+ dmac_c = MLX5_ADDR_OF(fte_match_param, rule_spec->match_criteria,
+ outer_headers.dmac_47_16);
+ eth_broadcast_addr(dmac_c);
+
+ if (vlan) {
+ if (vlan->pkt_reformat_pop) {
+ flow_act.action |= MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT;
+ flow_act.pkt_reformat = vlan->pkt_reformat_pop;
+ }
+
+ MLX5_SET_TO_ONES(fte_match_param, rule_spec->match_criteria,
+ outer_headers.cvlan_tag);
+ MLX5_SET_TO_ONES(fte_match_param, rule_spec->match_value,
+ outer_headers.cvlan_tag);
+ MLX5_SET_TO_ONES(fte_match_param, rule_spec->match_criteria,
+ outer_headers.first_vid);
+ MLX5_SET(fte_match_param, rule_spec->match_value, outer_headers.first_vid,
+ vlan->vid);
+ }
+
+ handle = mlx5_add_flow_rules(bridge->egress_ft, rule_spec, &flow_act, &dest, 1);
+
+ kvfree(rule_spec);
+ return handle;
+}
+
+static struct mlx5_esw_bridge *mlx5_esw_bridge_create(int ifindex,
+ struct mlx5_esw_bridge_offloads *br_offloads)
+{
+ struct mlx5_esw_bridge *bridge;
+ int err;
+
+ bridge = kvzalloc(sizeof(*bridge), GFP_KERNEL);
+ if (!bridge)
+ return ERR_PTR(-ENOMEM);
+
+ bridge->br_offloads = br_offloads;
+ err = mlx5_esw_bridge_egress_table_init(br_offloads, bridge);
+ if (err)
+ goto err_egress_tbl;
+
+ err = rhashtable_init(&bridge->fdb_ht, &fdb_ht_params);
+ if (err)
+ goto err_fdb_ht;
+
+ INIT_LIST_HEAD(&bridge->fdb_list);
+ xa_init(&bridge->vports);
+ bridge->ifindex = ifindex;
+ bridge->refcnt = 1;
+ bridge->ageing_time = BR_DEFAULT_AGEING_TIME;
+ list_add(&bridge->list, &br_offloads->bridges);
+
+ return bridge;
+
+err_fdb_ht:
+ mlx5_esw_bridge_egress_table_cleanup(bridge);
+err_egress_tbl:
+ kvfree(bridge);
+ return ERR_PTR(err);
+}
+
+static void mlx5_esw_bridge_get(struct mlx5_esw_bridge *bridge)
+{
+ bridge->refcnt++;
+}
+
+static void mlx5_esw_bridge_put(struct mlx5_esw_bridge_offloads *br_offloads,
+ struct mlx5_esw_bridge *bridge)
+{
+ if (--bridge->refcnt)
+ return;
+
+ mlx5_esw_bridge_egress_table_cleanup(bridge);
+ WARN_ON(!xa_empty(&bridge->vports));
+ list_del(&bridge->list);
+ rhashtable_destroy(&bridge->fdb_ht);
+ kvfree(bridge);
+
+ if (list_empty(&br_offloads->bridges))
+ mlx5_esw_bridge_ingress_table_cleanup(br_offloads);
+}
+
+static struct mlx5_esw_bridge *
+mlx5_esw_bridge_lookup(int ifindex, struct mlx5_esw_bridge_offloads *br_offloads)
+{
+ struct mlx5_esw_bridge *bridge;
+
+ ASSERT_RTNL();
+
+ list_for_each_entry(bridge, &br_offloads->bridges, list) {
+ if (bridge->ifindex == ifindex) {
+ mlx5_esw_bridge_get(bridge);
+ return bridge;
+ }
+ }
+
+ if (!br_offloads->ingress_ft) {
+ int err = mlx5_esw_bridge_ingress_table_init(br_offloads);
+
+ if (err)
+ return ERR_PTR(err);
+ }
+
+ bridge = mlx5_esw_bridge_create(ifindex, br_offloads);
+ if (IS_ERR(bridge) && list_empty(&br_offloads->bridges))
+ mlx5_esw_bridge_ingress_table_cleanup(br_offloads);
+ return bridge;
+}
+
+static int mlx5_esw_bridge_port_insert(struct mlx5_esw_bridge_port *port,
+ struct mlx5_esw_bridge *bridge)
+{
+ return xa_insert(&bridge->vports, port->vport_num, port, GFP_KERNEL);
+}
+
+static struct mlx5_esw_bridge_port *
+mlx5_esw_bridge_port_lookup(u16 vport_num, struct mlx5_esw_bridge *bridge)
+{
+ return xa_load(&bridge->vports, vport_num);
+}
+
+static void mlx5_esw_bridge_port_erase(struct mlx5_esw_bridge_port *port,
+ struct mlx5_esw_bridge *bridge)
+{
+ xa_erase(&bridge->vports, port->vport_num);
+}
+
+static void mlx5_esw_bridge_fdb_entry_refresh(unsigned long lastuse,
+ struct mlx5_esw_bridge_fdb_entry *entry)
+{
+ trace_mlx5_esw_bridge_fdb_entry_refresh(entry);
+
+ entry->lastuse = lastuse;
+ mlx5_esw_bridge_fdb_offload_notify(entry->dev, entry->key.addr,
+ entry->key.vid,
+ SWITCHDEV_FDB_ADD_TO_BRIDGE);
+}
+
+static void
+mlx5_esw_bridge_fdb_entry_cleanup(struct mlx5_esw_bridge_fdb_entry *entry,
+ struct mlx5_esw_bridge *bridge)
+{
+ trace_mlx5_esw_bridge_fdb_entry_cleanup(entry);
+
+ rhashtable_remove_fast(&bridge->fdb_ht, &entry->ht_node, fdb_ht_params);
+ mlx5_del_flow_rules(entry->egress_handle);
+ if (entry->filter_handle)
+ mlx5_del_flow_rules(entry->filter_handle);
+ mlx5_del_flow_rules(entry->ingress_handle);
+ mlx5_fc_destroy(bridge->br_offloads->esw->dev, entry->ingress_counter);
+ list_del(&entry->vlan_list);
+ list_del(&entry->list);
+ kvfree(entry);
+}
+
+static void mlx5_esw_bridge_fdb_flush(struct mlx5_esw_bridge *bridge)
+{
+ struct mlx5_esw_bridge_fdb_entry *entry, *tmp;
+
+ list_for_each_entry_safe(entry, tmp, &bridge->fdb_list, list) {
+ if (!(entry->flags & MLX5_ESW_BRIDGE_FLAG_ADDED_BY_USER))
+ mlx5_esw_bridge_fdb_offload_notify(entry->dev, entry->key.addr,
+ entry->key.vid,
+ SWITCHDEV_FDB_DEL_TO_BRIDGE);
+ mlx5_esw_bridge_fdb_entry_cleanup(entry, bridge);
+ }
+}
+
+static struct mlx5_esw_bridge_vlan *
+mlx5_esw_bridge_vlan_lookup(u16 vid, struct mlx5_esw_bridge_port *port)
+{
+ return xa_load(&port->vlans, vid);
+}
+
+static int
+mlx5_esw_bridge_vlan_push_create(struct mlx5_esw_bridge_vlan *vlan, struct mlx5_eswitch *esw)
+{
+ struct {
+ __be16 h_vlan_proto;
+ __be16 h_vlan_TCI;
+ } vlan_hdr = { htons(ETH_P_8021Q), htons(vlan->vid) };
+ struct mlx5_pkt_reformat_params reformat_params = {};
+ struct mlx5_pkt_reformat *pkt_reformat;
+
+ if (!BIT(MLX5_CAP_ESW_FLOWTABLE_FDB(esw->dev, reformat_insert)) ||
+ MLX5_CAP_GEN_2(esw->dev, max_reformat_insert_size) < sizeof(vlan_hdr) ||
+ MLX5_CAP_GEN_2(esw->dev, max_reformat_insert_offset) <
+ offsetof(struct vlan_ethhdr, h_vlan_proto)) {
+ esw_warn(esw->dev, "Packet reformat INSERT_HEADER is not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ reformat_params.type = MLX5_REFORMAT_TYPE_INSERT_HDR;
+ reformat_params.param_0 = MLX5_REFORMAT_CONTEXT_ANCHOR_MAC_START;
+ reformat_params.param_1 = offsetof(struct vlan_ethhdr, h_vlan_proto);
+ reformat_params.size = sizeof(vlan_hdr);
+ reformat_params.data = &vlan_hdr;
+ pkt_reformat = mlx5_packet_reformat_alloc(esw->dev,
+ &reformat_params,
+ MLX5_FLOW_NAMESPACE_FDB);
+ if (IS_ERR(pkt_reformat)) {
+ esw_warn(esw->dev, "Failed to alloc packet reformat INSERT_HEADER (err=%ld)\n",
+ PTR_ERR(pkt_reformat));
+ return PTR_ERR(pkt_reformat);
+ }
+
+ vlan->pkt_reformat_push = pkt_reformat;
+ return 0;
+}
+
+static void
+mlx5_esw_bridge_vlan_push_cleanup(struct mlx5_esw_bridge_vlan *vlan, struct mlx5_eswitch *esw)
+{
+ mlx5_packet_reformat_dealloc(esw->dev, vlan->pkt_reformat_push);
+ vlan->pkt_reformat_push = NULL;
+}
+
+static int
+mlx5_esw_bridge_vlan_pop_create(struct mlx5_esw_bridge_vlan *vlan, struct mlx5_eswitch *esw)
+{
+ struct mlx5_pkt_reformat_params reformat_params = {};
+ struct mlx5_pkt_reformat *pkt_reformat;
+
+ if (!BIT(MLX5_CAP_ESW_FLOWTABLE_FDB(esw->dev, reformat_remove)) ||
+ MLX5_CAP_GEN_2(esw->dev, max_reformat_remove_size) < sizeof(struct vlan_hdr) ||
+ MLX5_CAP_GEN_2(esw->dev, max_reformat_remove_offset) <
+ offsetof(struct vlan_ethhdr, h_vlan_proto)) {
+ esw_warn(esw->dev, "Packet reformat REMOVE_HEADER is not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ reformat_params.type = MLX5_REFORMAT_TYPE_REMOVE_HDR;
+ reformat_params.param_0 = MLX5_REFORMAT_CONTEXT_ANCHOR_MAC_START;
+ reformat_params.param_1 = offsetof(struct vlan_ethhdr, h_vlan_proto);
+ reformat_params.size = sizeof(struct vlan_hdr);
+ pkt_reformat = mlx5_packet_reformat_alloc(esw->dev,
+ &reformat_params,
+ MLX5_FLOW_NAMESPACE_FDB);
+ if (IS_ERR(pkt_reformat)) {
+ esw_warn(esw->dev, "Failed to alloc packet reformat REMOVE_HEADER (err=%ld)\n",
+ PTR_ERR(pkt_reformat));
+ return PTR_ERR(pkt_reformat);
+ }
+
+ vlan->pkt_reformat_pop = pkt_reformat;
+ return 0;
+}
+
+static void
+mlx5_esw_bridge_vlan_pop_cleanup(struct mlx5_esw_bridge_vlan *vlan, struct mlx5_eswitch *esw)
+{
+ mlx5_packet_reformat_dealloc(esw->dev, vlan->pkt_reformat_pop);
+ vlan->pkt_reformat_pop = NULL;
+}
+
+static struct mlx5_esw_bridge_vlan *
+mlx5_esw_bridge_vlan_create(u16 vid, u16 flags, struct mlx5_esw_bridge_port *port,
+ struct mlx5_eswitch *esw)
+{
+ struct mlx5_esw_bridge_vlan *vlan;
+ int err;
+
+ vlan = kvzalloc(sizeof(*vlan), GFP_KERNEL);
+ if (!vlan)
+ return ERR_PTR(-ENOMEM);
+
+ vlan->vid = vid;
+ vlan->flags = flags;
+ INIT_LIST_HEAD(&vlan->fdb_list);
+
+ if (flags & BRIDGE_VLAN_INFO_PVID) {
+ err = mlx5_esw_bridge_vlan_push_create(vlan, esw);
+ if (err)
+ goto err_vlan_push;
+ }
+ if (flags & BRIDGE_VLAN_INFO_UNTAGGED) {
+ err = mlx5_esw_bridge_vlan_pop_create(vlan, esw);
+ if (err)
+ goto err_vlan_pop;
+ }
+
+ err = xa_insert(&port->vlans, vid, vlan, GFP_KERNEL);
+ if (err)
+ goto err_xa_insert;
+
+ trace_mlx5_esw_bridge_vlan_create(vlan);
+ return vlan;
+
+err_xa_insert:
+ if (vlan->pkt_reformat_pop)
+ mlx5_esw_bridge_vlan_pop_cleanup(vlan, esw);
+err_vlan_pop:
+ if (vlan->pkt_reformat_push)
+ mlx5_esw_bridge_vlan_push_cleanup(vlan, esw);
+err_vlan_push:
+ kvfree(vlan);
+ return ERR_PTR(err);
+}
+
+static void mlx5_esw_bridge_vlan_erase(struct mlx5_esw_bridge_port *port,
+ struct mlx5_esw_bridge_vlan *vlan)
+{
+ xa_erase(&port->vlans, vlan->vid);
+}
+
+static void mlx5_esw_bridge_vlan_flush(struct mlx5_esw_bridge_vlan *vlan,
+ struct mlx5_esw_bridge *bridge)
+{
+ struct mlx5_esw_bridge_fdb_entry *entry, *tmp;
+
+ list_for_each_entry_safe(entry, tmp, &vlan->fdb_list, vlan_list) {
+ if (!(entry->flags & MLX5_ESW_BRIDGE_FLAG_ADDED_BY_USER))
+ mlx5_esw_bridge_fdb_offload_notify(entry->dev, entry->key.addr,
+ entry->key.vid,
+ SWITCHDEV_FDB_DEL_TO_BRIDGE);
+ mlx5_esw_bridge_fdb_entry_cleanup(entry, bridge);
+ }
+
+ if (vlan->pkt_reformat_pop)
+ mlx5_esw_bridge_vlan_pop_cleanup(vlan, bridge->br_offloads->esw);
+ if (vlan->pkt_reformat_push)
+ mlx5_esw_bridge_vlan_push_cleanup(vlan, bridge->br_offloads->esw);
+}
+
+static void mlx5_esw_bridge_vlan_cleanup(struct mlx5_esw_bridge_port *port,
+ struct mlx5_esw_bridge_vlan *vlan,
+ struct mlx5_esw_bridge *bridge)
+{
+ trace_mlx5_esw_bridge_vlan_cleanup(vlan);
+ mlx5_esw_bridge_vlan_flush(vlan, bridge);
+ mlx5_esw_bridge_vlan_erase(port, vlan);
+ kvfree(vlan);
+}
+
+static void mlx5_esw_bridge_port_vlans_flush(struct mlx5_esw_bridge_port *port,
+ struct mlx5_esw_bridge *bridge)
+{
+ struct mlx5_esw_bridge_vlan *vlan;
+ unsigned long index;
+
+ xa_for_each(&port->vlans, index, vlan)
+ mlx5_esw_bridge_vlan_cleanup(port, vlan, bridge);
+}
+
+static struct mlx5_esw_bridge_vlan *
+mlx5_esw_bridge_port_vlan_lookup(u16 vid, u16 vport_num, struct mlx5_esw_bridge *bridge,
+ struct mlx5_eswitch *esw)
+{
+ struct mlx5_esw_bridge_port *port;
+ struct mlx5_esw_bridge_vlan *vlan;
+
+ port = mlx5_esw_bridge_port_lookup(vport_num, bridge);
+ if (!port) {
+ /* FDB is added asynchronously on wq while port might have been deleted
+ * concurrently. Report on 'info' logging level and skip the FDB offload.
+ */
+ esw_info(esw->dev, "Failed to lookup bridge port (vport=%u)\n", vport_num);
+ return ERR_PTR(-EINVAL);
+ }
+
+ vlan = mlx5_esw_bridge_vlan_lookup(vid, port);
+ if (!vlan) {
+ /* FDB is added asynchronously on wq while vlan might have been deleted
+ * concurrently. Report on 'info' logging level and skip the FDB offload.
+ */
+ esw_info(esw->dev, "Failed to lookup bridge port vlan metadata (vport=%u)\n",
+ vport_num);
+ return ERR_PTR(-EINVAL);
+ }
+
+ return vlan;
+}
+
+static struct mlx5_esw_bridge_fdb_entry *
+mlx5_esw_bridge_fdb_entry_init(struct net_device *dev, u16 vport_num, const unsigned char *addr,
+ u16 vid, bool added_by_user, struct mlx5_eswitch *esw,
+ struct mlx5_esw_bridge *bridge)
+{
+ struct mlx5_esw_bridge_vlan *vlan = NULL;
+ struct mlx5_esw_bridge_fdb_entry *entry;
+ struct mlx5_flow_handle *handle;
+ struct mlx5_fc *counter;
+ struct mlx5e_priv *priv;
+ int err;
+
+ if (bridge->flags & MLX5_ESW_BRIDGE_VLAN_FILTERING_FLAG && vid) {
+ vlan = mlx5_esw_bridge_port_vlan_lookup(vid, vport_num, bridge, esw);
+ if (IS_ERR(vlan))
+ return ERR_CAST(vlan);
+ }
+
+ priv = netdev_priv(dev);
+ entry = kvzalloc(sizeof(*entry), GFP_KERNEL);
+ if (!entry)
+ return ERR_PTR(-ENOMEM);
+
+ ether_addr_copy(entry->key.addr, addr);
+ entry->key.vid = vid;
+ entry->dev = dev;
+ entry->vport_num = vport_num;
+ entry->lastuse = jiffies;
+ if (added_by_user)
+ entry->flags |= MLX5_ESW_BRIDGE_FLAG_ADDED_BY_USER;
+
+ counter = mlx5_fc_create(priv->mdev, true);
+ if (IS_ERR(counter)) {
+ err = PTR_ERR(counter);
+ goto err_ingress_fc_create;
+ }
+ entry->ingress_counter = counter;
+
+ handle = mlx5_esw_bridge_ingress_flow_create(vport_num, addr, vlan, mlx5_fc_id(counter),
+ bridge);
+ if (IS_ERR(handle)) {
+ err = PTR_ERR(handle);
+ esw_warn(esw->dev, "Failed to create ingress flow(vport=%u,err=%d)\n",
+ vport_num, err);
+ goto err_ingress_flow_create;
+ }
+ entry->ingress_handle = handle;
+
+ if (bridge->flags & MLX5_ESW_BRIDGE_VLAN_FILTERING_FLAG) {
+ handle = mlx5_esw_bridge_ingress_filter_flow_create(vport_num, addr, bridge);
+ if (IS_ERR(handle)) {
+ err = PTR_ERR(handle);
+ esw_warn(esw->dev, "Failed to create ingress filter(vport=%u,err=%d)\n",
+ vport_num, err);
+ goto err_ingress_filter_flow_create;
+ }
+ entry->filter_handle = handle;
+ }
+
+ handle = mlx5_esw_bridge_egress_flow_create(vport_num, addr, vlan, bridge);
+ if (IS_ERR(handle)) {
+ err = PTR_ERR(handle);
+ esw_warn(esw->dev, "Failed to create egress flow(vport=%u,err=%d)\n",
+ vport_num, err);
+ goto err_egress_flow_create;
+ }
+ entry->egress_handle = handle;
+
+ err = rhashtable_insert_fast(&bridge->fdb_ht, &entry->ht_node, fdb_ht_params);
+ if (err) {
+ esw_warn(esw->dev, "Failed to insert FDB flow(vport=%u,err=%d)\n", vport_num, err);
+ goto err_ht_init;
+ }
+
+ if (vlan)
+ list_add(&entry->vlan_list, &vlan->fdb_list);
+ else
+ INIT_LIST_HEAD(&entry->vlan_list);
+ list_add(&entry->list, &bridge->fdb_list);
+
+ trace_mlx5_esw_bridge_fdb_entry_init(entry);
+ return entry;
+
+err_ht_init:
+ mlx5_del_flow_rules(entry->egress_handle);
+err_egress_flow_create:
+ if (entry->filter_handle)
+ mlx5_del_flow_rules(entry->filter_handle);
+err_ingress_filter_flow_create:
+ mlx5_del_flow_rules(entry->ingress_handle);
+err_ingress_flow_create:
+ mlx5_fc_destroy(priv->mdev, entry->ingress_counter);
+err_ingress_fc_create:
+ kvfree(entry);
+ return ERR_PTR(err);
+}
+
+int mlx5_esw_bridge_ageing_time_set(unsigned long ageing_time, struct mlx5_eswitch *esw,
+ struct mlx5_vport *vport)
+{
+ if (!vport->bridge)
+ return -EINVAL;
+
+ vport->bridge->ageing_time = ageing_time;
+ return 0;
+}
+
+int mlx5_esw_bridge_vlan_filtering_set(bool enable, struct mlx5_eswitch *esw,
+ struct mlx5_vport *vport)
+{
+ struct mlx5_esw_bridge *bridge;
+ bool filtering;
+
+ if (!vport->bridge)
+ return -EINVAL;
+
+ bridge = vport->bridge;
+ filtering = bridge->flags & MLX5_ESW_BRIDGE_VLAN_FILTERING_FLAG;
+ if (filtering == enable)
+ return 0;
+
+ mlx5_esw_bridge_fdb_flush(bridge);
+ if (enable)
+ bridge->flags |= MLX5_ESW_BRIDGE_VLAN_FILTERING_FLAG;
+ else
+ bridge->flags &= ~MLX5_ESW_BRIDGE_VLAN_FILTERING_FLAG;
+
+ return 0;
+}
+
+static int mlx5_esw_bridge_vport_init(struct mlx5_esw_bridge_offloads *br_offloads,
+ struct mlx5_esw_bridge *bridge,
+ struct mlx5_vport *vport)
+{
+ struct mlx5_eswitch *esw = br_offloads->esw;
+ struct mlx5_esw_bridge_port *port;
+ int err;
+
+ port = kvzalloc(sizeof(*port), GFP_KERNEL);
+ if (!port) {
+ err = -ENOMEM;
+ goto err_port_alloc;
+ }
+
+ port->vport_num = vport->vport;
+ xa_init(&port->vlans);
+ err = mlx5_esw_bridge_port_insert(port, bridge);
+ if (err) {
+ esw_warn(esw->dev, "Failed to insert port metadata (vport=%u,err=%d)\n",
+ vport->vport, err);
+ goto err_port_insert;
+ }
+ trace_mlx5_esw_bridge_vport_init(port);
+
+ vport->bridge = bridge;
+ return 0;
+
+err_port_insert:
+ kvfree(port);
+err_port_alloc:
+ mlx5_esw_bridge_put(br_offloads, bridge);
+ return err;
+}
+
+static int mlx5_esw_bridge_vport_cleanup(struct mlx5_esw_bridge_offloads *br_offloads,
+ struct mlx5_vport *vport)
+{
+ struct mlx5_esw_bridge *bridge = vport->bridge;
+ struct mlx5_esw_bridge_fdb_entry *entry, *tmp;
+ struct mlx5_esw_bridge_port *port;
+
+ list_for_each_entry_safe(entry, tmp, &bridge->fdb_list, list)
+ if (entry->vport_num == vport->vport)
+ mlx5_esw_bridge_fdb_entry_cleanup(entry, bridge);
+
+ port = mlx5_esw_bridge_port_lookup(vport->vport, bridge);
+ if (!port) {
+ WARN(1, "Vport %u metadata not found on bridge", vport->vport);
+ return -EINVAL;
+ }
+
+ trace_mlx5_esw_bridge_vport_cleanup(port);
+ mlx5_esw_bridge_port_vlans_flush(port, bridge);
+ mlx5_esw_bridge_port_erase(port, bridge);
+ kvfree(port);
+ mlx5_esw_bridge_put(br_offloads, bridge);
+ vport->bridge = NULL;
+ return 0;
+}
+
+int mlx5_esw_bridge_vport_link(int ifindex, struct mlx5_esw_bridge_offloads *br_offloads,
+ struct mlx5_vport *vport, struct netlink_ext_ack *extack)
+{
+ struct mlx5_esw_bridge *bridge;
+ int err;
+
+ WARN_ON(vport->bridge);
+
+ bridge = mlx5_esw_bridge_lookup(ifindex, br_offloads);
+ if (IS_ERR(bridge)) {
+ NL_SET_ERR_MSG_MOD(extack, "Error checking for existing bridge with same ifindex");
+ return PTR_ERR(bridge);
+ }
+
+ err = mlx5_esw_bridge_vport_init(br_offloads, bridge, vport);
+ if (err)
+ NL_SET_ERR_MSG_MOD(extack, "Error initializing port");
+ return err;
+}
+
+int mlx5_esw_bridge_vport_unlink(int ifindex, struct mlx5_esw_bridge_offloads *br_offloads,
+ struct mlx5_vport *vport, struct netlink_ext_ack *extack)
+{
+ struct mlx5_esw_bridge *bridge = vport->bridge;
+ int err;
+
+ if (!bridge) {
+ NL_SET_ERR_MSG_MOD(extack, "Port is not attached to any bridge");
+ return -EINVAL;
+ }
+ if (bridge->ifindex != ifindex) {
+ NL_SET_ERR_MSG_MOD(extack, "Port is attached to another bridge");
+ return -EINVAL;
+ }
+
+ err = mlx5_esw_bridge_vport_cleanup(br_offloads, vport);
+ if (err)
+ NL_SET_ERR_MSG_MOD(extack, "Port cleanup failed");
+ return err;
+}
+
+int mlx5_esw_bridge_port_vlan_add(u16 vid, u16 flags, struct mlx5_eswitch *esw,
+ struct mlx5_vport *vport, struct netlink_ext_ack *extack)
+{
+ struct mlx5_esw_bridge_port *port;
+ struct mlx5_esw_bridge_vlan *vlan;
+
+ port = mlx5_esw_bridge_port_lookup(vport->vport, vport->bridge);
+ if (!port)
+ return -EINVAL;
+
+ vlan = mlx5_esw_bridge_vlan_lookup(vid, port);
+ if (vlan) {
+ if (vlan->flags == flags)
+ return 0;
+ mlx5_esw_bridge_vlan_cleanup(port, vlan, vport->bridge);
+ }
+
+ vlan = mlx5_esw_bridge_vlan_create(vid, flags, port, esw);
+ if (IS_ERR(vlan)) {
+ NL_SET_ERR_MSG_MOD(extack, "Failed to create VLAN entry");
+ return PTR_ERR(vlan);
+ }
+ return 0;
+}
+
+void mlx5_esw_bridge_port_vlan_del(u16 vid, struct mlx5_eswitch *esw, struct mlx5_vport *vport)
+{
+ struct mlx5_esw_bridge_port *port;
+ struct mlx5_esw_bridge_vlan *vlan;
+
+ port = mlx5_esw_bridge_port_lookup(vport->vport, vport->bridge);
+ if (!port)
+ return;
+
+ vlan = mlx5_esw_bridge_vlan_lookup(vid, port);
+ if (!vlan)
+ return;
+ mlx5_esw_bridge_vlan_cleanup(port, vlan, vport->bridge);
+}
+
+void mlx5_esw_bridge_fdb_create(struct net_device *dev, struct mlx5_eswitch *esw,
+ struct mlx5_vport *vport,
+ struct switchdev_notifier_fdb_info *fdb_info)
+{
+ struct mlx5_esw_bridge *bridge = vport->bridge;
+ struct mlx5_esw_bridge_fdb_entry *entry;
+ u16 vport_num = vport->vport;
+
+ if (!bridge) {
+ esw_info(esw->dev, "Vport is not assigned to bridge (vport=%u)\n", vport_num);
+ return;
+ }
+
+ entry = mlx5_esw_bridge_fdb_entry_init(dev, vport_num, fdb_info->addr, fdb_info->vid,
+ fdb_info->added_by_user, esw, bridge);
+ if (IS_ERR(entry))
+ return;
+
+ if (entry->flags & MLX5_ESW_BRIDGE_FLAG_ADDED_BY_USER)
+ mlx5_esw_bridge_fdb_offload_notify(dev, entry->key.addr, entry->key.vid,
+ SWITCHDEV_FDB_OFFLOADED);
+ else
+ /* Take over dynamic entries to prevent kernel bridge from aging them out. */
+ mlx5_esw_bridge_fdb_offload_notify(dev, entry->key.addr, entry->key.vid,
+ SWITCHDEV_FDB_ADD_TO_BRIDGE);
+}
+
+void mlx5_esw_bridge_fdb_remove(struct net_device *dev, struct mlx5_eswitch *esw,
+ struct mlx5_vport *vport,
+ struct switchdev_notifier_fdb_info *fdb_info)
+{
+ struct mlx5_esw_bridge *bridge = vport->bridge;
+ struct mlx5_esw_bridge_fdb_entry *entry;
+ struct mlx5_esw_bridge_fdb_key key;
+ u16 vport_num = vport->vport;
+
+ if (!bridge) {
+ esw_warn(esw->dev, "Vport is not assigned to bridge (vport=%u)\n", vport_num);
+ return;
+ }
+
+ ether_addr_copy(key.addr, fdb_info->addr);
+ key.vid = fdb_info->vid;
+ entry = rhashtable_lookup_fast(&bridge->fdb_ht, &key, fdb_ht_params);
+ if (!entry) {
+ esw_warn(esw->dev,
+ "FDB entry with specified key not found (MAC=%pM,vid=%u,vport=%u)\n",
+ key.addr, key.vid, vport_num);
+ return;
+ }
+
+ if (!(entry->flags & MLX5_ESW_BRIDGE_FLAG_ADDED_BY_USER))
+ mlx5_esw_bridge_fdb_offload_notify(dev, entry->key.addr, entry->key.vid,
+ SWITCHDEV_FDB_DEL_TO_BRIDGE);
+ mlx5_esw_bridge_fdb_entry_cleanup(entry, bridge);
+}
+
+void mlx5_esw_bridge_update(struct mlx5_esw_bridge_offloads *br_offloads)
+{
+ struct mlx5_esw_bridge_fdb_entry *entry, *tmp;
+ struct mlx5_esw_bridge *bridge;
+
+ list_for_each_entry(bridge, &br_offloads->bridges, list) {
+ list_for_each_entry_safe(entry, tmp, &bridge->fdb_list, list) {
+ unsigned long lastuse =
+ (unsigned long)mlx5_fc_query_lastuse(entry->ingress_counter);
+
+ if (entry->flags & MLX5_ESW_BRIDGE_FLAG_ADDED_BY_USER)
+ continue;
+
+ if (time_after(lastuse, entry->lastuse)) {
+ mlx5_esw_bridge_fdb_entry_refresh(lastuse, entry);
+ } else if (time_is_before_jiffies(entry->lastuse + bridge->ageing_time)) {
+ mlx5_esw_bridge_fdb_offload_notify(entry->dev, entry->key.addr,
+ entry->key.vid,
+ SWITCHDEV_FDB_DEL_TO_BRIDGE);
+ mlx5_esw_bridge_fdb_entry_cleanup(entry, bridge);
+ }
+ }
+ }
+}
+
+static void mlx5_esw_bridge_flush(struct mlx5_esw_bridge_offloads *br_offloads)
+{
+ struct mlx5_eswitch *esw = br_offloads->esw;
+ struct mlx5_vport *vport;
+ unsigned long i;
+
+ mlx5_esw_for_each_vport(esw, i, vport)
+ if (vport->bridge)
+ mlx5_esw_bridge_vport_cleanup(br_offloads, vport);
+
+ WARN_ONCE(!list_empty(&br_offloads->bridges),
+ "Cleaning up bridge offloads while still having bridges attached\n");
+}
+
+struct mlx5_esw_bridge_offloads *mlx5_esw_bridge_init(struct mlx5_eswitch *esw)
+{
+ struct mlx5_esw_bridge_offloads *br_offloads;
+
+ br_offloads = kvzalloc(sizeof(*br_offloads), GFP_KERNEL);
+ if (!br_offloads)
+ return ERR_PTR(-ENOMEM);
+
+ INIT_LIST_HEAD(&br_offloads->bridges);
+ br_offloads->esw = esw;
+ esw->br_offloads = br_offloads;
+
+ return br_offloads;
+}
+
+void mlx5_esw_bridge_cleanup(struct mlx5_eswitch *esw)
+{
+ struct mlx5_esw_bridge_offloads *br_offloads = esw->br_offloads;
+
+ if (!br_offloads)
+ return;
+
+ mlx5_esw_bridge_flush(br_offloads);
+
+ esw->br_offloads = NULL;
+ kvfree(br_offloads);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2021 Mellanox Technologies. */
+
+#ifndef __MLX5_ESW_BRIDGE_H__
+#define __MLX5_ESW_BRIDGE_H__
+
+#include <linux/notifier.h>
+#include <linux/list.h>
+#include <linux/workqueue.h>
+#include "eswitch.h"
+
+struct mlx5_flow_table;
+struct mlx5_flow_group;
+
+struct mlx5_esw_bridge_offloads {
+ struct mlx5_eswitch *esw;
+ struct list_head bridges;
+ struct notifier_block netdev_nb;
+ struct notifier_block nb_blk;
+ struct notifier_block nb;
+ struct workqueue_struct *wq;
+ struct delayed_work update_work;
+
+ struct mlx5_flow_table *ingress_ft;
+ struct mlx5_flow_group *ingress_vlan_fg;
+ struct mlx5_flow_group *ingress_filter_fg;
+ struct mlx5_flow_group *ingress_mac_fg;
+
+ struct mlx5_flow_table *skip_ft;
+};
+
+struct mlx5_esw_bridge_offloads *mlx5_esw_bridge_init(struct mlx5_eswitch *esw);
+void mlx5_esw_bridge_cleanup(struct mlx5_eswitch *esw);
+int mlx5_esw_bridge_vport_link(int ifindex, struct mlx5_esw_bridge_offloads *br_offloads,
+ struct mlx5_vport *vport, struct netlink_ext_ack *extack);
+int mlx5_esw_bridge_vport_unlink(int ifindex, struct mlx5_esw_bridge_offloads *br_offloads,
+ struct mlx5_vport *vport, struct netlink_ext_ack *extack);
+void mlx5_esw_bridge_fdb_create(struct net_device *dev, struct mlx5_eswitch *esw,
+ struct mlx5_vport *vport,
+ struct switchdev_notifier_fdb_info *fdb_info);
+void mlx5_esw_bridge_fdb_remove(struct net_device *dev, struct mlx5_eswitch *esw,
+ struct mlx5_vport *vport,
+ struct switchdev_notifier_fdb_info *fdb_info);
+void mlx5_esw_bridge_update(struct mlx5_esw_bridge_offloads *br_offloads);
+int mlx5_esw_bridge_ageing_time_set(unsigned long ageing_time, struct mlx5_eswitch *esw,
+ struct mlx5_vport *vport);
+int mlx5_esw_bridge_vlan_filtering_set(bool enable, struct mlx5_eswitch *esw,
+ struct mlx5_vport *vport);
+int mlx5_esw_bridge_port_vlan_add(u16 vid, u16 flags, struct mlx5_eswitch *esw,
+ struct mlx5_vport *vport, struct netlink_ext_ack *extack);
+void mlx5_esw_bridge_port_vlan_del(u16 vid, struct mlx5_eswitch *esw, struct mlx5_vport *vport);
+
+#endif /* __MLX5_ESW_BRIDGE_H__ */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2021 Mellanox Technologies. */
+
+#ifndef _MLX5_ESW_BRIDGE_PRIVATE_
+#define _MLX5_ESW_BRIDGE_PRIVATE_
+
+#include <linux/netdevice.h>
+#include <linux/if_bridge.h>
+#include <linux/if_vlan.h>
+#include <linux/if_ether.h>
+#include <linux/rhashtable.h>
+#include <linux/xarray.h>
+#include "fs_core.h"
+
+struct mlx5_esw_bridge_fdb_key {
+ unsigned char addr[ETH_ALEN];
+ u16 vid;
+};
+
+enum {
+ MLX5_ESW_BRIDGE_FLAG_ADDED_BY_USER = BIT(0),
+};
+
+struct mlx5_esw_bridge_fdb_entry {
+ struct mlx5_esw_bridge_fdb_key key;
+ struct rhash_head ht_node;
+ struct net_device *dev;
+ struct list_head list;
+ struct list_head vlan_list;
+ u16 vport_num;
+ u16 flags;
+
+ struct mlx5_flow_handle *ingress_handle;
+ struct mlx5_fc *ingress_counter;
+ unsigned long lastuse;
+ struct mlx5_flow_handle *egress_handle;
+ struct mlx5_flow_handle *filter_handle;
+};
+
+struct mlx5_esw_bridge_vlan {
+ u16 vid;
+ u16 flags;
+ struct list_head fdb_list;
+ struct mlx5_pkt_reformat *pkt_reformat_push;
+ struct mlx5_pkt_reformat *pkt_reformat_pop;
+};
+
+struct mlx5_esw_bridge_port {
+ u16 vport_num;
+ struct xarray vlans;
+};
+
+#endif /* _MLX5_ESW_BRIDGE_PRIVATE_ */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2021 Mellanox Technologies. */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM mlx5
+
+#if !defined(_MLX5_ESW_BRIDGE_TRACEPOINT_) || defined(TRACE_HEADER_MULTI_READ)
+#define _MLX5_ESW_BRIDGE_TRACEPOINT_
+
+#include <linux/tracepoint.h>
+#include "../bridge_priv.h"
+
+DECLARE_EVENT_CLASS(mlx5_esw_bridge_fdb_template,
+ TP_PROTO(const struct mlx5_esw_bridge_fdb_entry *fdb),
+ TP_ARGS(fdb),
+ TP_STRUCT__entry(
+ __array(char, dev_name, IFNAMSIZ)
+ __array(unsigned char, addr, ETH_ALEN)
+ __field(u16, vid)
+ __field(u16, flags)
+ __field(unsigned int, used)
+ ),
+ TP_fast_assign(
+ strncpy(__entry->dev_name,
+ netdev_name(fdb->dev),
+ IFNAMSIZ);
+ memcpy(__entry->addr, fdb->key.addr, ETH_ALEN);
+ __entry->vid = fdb->key.vid;
+ __entry->flags = fdb->flags;
+ __entry->used = jiffies_to_msecs(jiffies - fdb->lastuse)
+ ),
+ TP_printk("net_device=%s addr=%pM vid=%hu flags=%hx used=%u",
+ __entry->dev_name,
+ __entry->addr,
+ __entry->vid,
+ __entry->flags,
+ __entry->used / 1000)
+ );
+
+DEFINE_EVENT(mlx5_esw_bridge_fdb_template,
+ mlx5_esw_bridge_fdb_entry_init,
+ TP_PROTO(const struct mlx5_esw_bridge_fdb_entry *fdb),
+ TP_ARGS(fdb)
+ );
+DEFINE_EVENT(mlx5_esw_bridge_fdb_template,
+ mlx5_esw_bridge_fdb_entry_refresh,
+ TP_PROTO(const struct mlx5_esw_bridge_fdb_entry *fdb),
+ TP_ARGS(fdb)
+ );
+DEFINE_EVENT(mlx5_esw_bridge_fdb_template,
+ mlx5_esw_bridge_fdb_entry_cleanup,
+ TP_PROTO(const struct mlx5_esw_bridge_fdb_entry *fdb),
+ TP_ARGS(fdb)
+ );
+
+DECLARE_EVENT_CLASS(mlx5_esw_bridge_vlan_template,
+ TP_PROTO(const struct mlx5_esw_bridge_vlan *vlan),
+ TP_ARGS(vlan),
+ TP_STRUCT__entry(
+ __field(u16, vid)
+ __field(u16, flags)
+ ),
+ TP_fast_assign(
+ __entry->vid = vlan->vid;
+ __entry->flags = vlan->flags;
+ ),
+ TP_printk("vid=%hu flags=%hx",
+ __entry->vid,
+ __entry->flags)
+ );
+
+DEFINE_EVENT(mlx5_esw_bridge_vlan_template,
+ mlx5_esw_bridge_vlan_create,
+ TP_PROTO(const struct mlx5_esw_bridge_vlan *vlan),
+ TP_ARGS(vlan)
+ );
+DEFINE_EVENT(mlx5_esw_bridge_vlan_template,
+ mlx5_esw_bridge_vlan_cleanup,
+ TP_PROTO(const struct mlx5_esw_bridge_vlan *vlan),
+ TP_ARGS(vlan)
+ );
+
+DECLARE_EVENT_CLASS(mlx5_esw_bridge_port_template,
+ TP_PROTO(const struct mlx5_esw_bridge_port *port),
+ TP_ARGS(port),
+ TP_STRUCT__entry(
+ __field(u16, vport_num)
+ ),
+ TP_fast_assign(
+ __entry->vport_num = port->vport_num;
+ ),
+ TP_printk("vport_num=%hu", __entry->vport_num)
+ );
+
+DEFINE_EVENT(mlx5_esw_bridge_port_template,
+ mlx5_esw_bridge_vport_init,
+ TP_PROTO(const struct mlx5_esw_bridge_port *port),
+ TP_ARGS(port)
+ );
+DEFINE_EVENT(mlx5_esw_bridge_port_template,
+ mlx5_esw_bridge_vport_cleanup,
+ TP_PROTO(const struct mlx5_esw_bridge_port *port),
+ TP_ARGS(port)
+ );
+
+#endif
+
+/* This part must be outside protection */
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH esw/diag
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE bridge_tracepoint
+#include <trace/define_trace.h>
#include <linux/mlx5/mlx5_ifc.h>
#include <linux/mlx5/vport.h>
#include <linux/mlx5/fs.h>
+#include <linux/mlx5/mpfs.h>
#include "esw/acl/lgcy.h"
#include "esw/legacy.h"
#include "mlx5_core.h"
goto err_vhca_mapping;
}
+ /* External controller host PF has factory programmed MAC.
+ * Read it from the device.
+ */
+ if (mlx5_core_is_ecpf(esw->dev) && vport_num == MLX5_VPORT_PF)
+ mlx5_query_nic_vport_mac_address(esw->dev, vport_num, true, vport->info.mac);
+
esw_vport_change_handle_locked(vport);
esw->enabled_vports++;
MLX5_VPORT_PROMISC_CHANGE = BIT(3),
};
+struct mlx5_esw_bridge;
+
struct mlx5_vport {
struct mlx5_core_dev *dev;
struct hlist_head uc_list[MLX5_L2_ADDR_HASH_SIZE];
enum mlx5_eswitch_vport_event enabled_events;
int index;
struct devlink_port *dl_port;
+ struct mlx5_esw_bridge *bridge;
};
struct mlx5_esw_indir_table;
struct offloads_fdb {
struct mlx5_flow_namespace *ns;
+ struct mlx5_flow_table *tc_miss_table;
struct mlx5_flow_table *slow_fdb;
struct mlx5_flow_group *send_to_vport_grp;
struct mlx5_flow_group *send_to_vport_meta_grp;
MLX5_ESWITCH_REG_C1_LOOPBACK_ENABLED = BIT(1),
};
+struct mlx5_esw_bridge_offloads;
+
struct mlx5_eswitch {
struct mlx5_core_dev *dev;
struct mlx5_nb nb;
u32 root_tsar_id;
} qos;
+ struct mlx5_esw_bridge_offloads *br_offloads;
struct mlx5_esw_offload offloads;
int mode;
u16 manager_vport;
struct mlx5_fs_chains *chains,
int i)
{
- flow_act->flags |= FLOW_ACT_IGNORE_FLOW_LEVEL;
+ if (mlx5_chains_ignore_flow_level_supported(chains))
+ flow_act->flags |= FLOW_ACT_IGNORE_FLOW_LEVEL;
dest[i].type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
dest[i].ft = mlx5_chains_get_tc_end_ft(chains);
}
}
esw->fdb_table.offloads.slow_fdb = fdb;
- err = esw_chains_create(esw, fdb);
+ /* Create empty TC-miss managed table. This allows plugging in following
+ * priorities without directly exposing their level 0 table to
+ * eswitch_offloads and passing it as miss_fdb to following call to
+ * esw_chains_create().
+ */
+ memset(&ft_attr, 0, sizeof(ft_attr));
+ ft_attr.prio = FDB_TC_MISS;
+ esw->fdb_table.offloads.tc_miss_table = mlx5_create_flow_table(root_ns, &ft_attr);
+ if (IS_ERR(esw->fdb_table.offloads.tc_miss_table)) {
+ err = PTR_ERR(esw->fdb_table.offloads.tc_miss_table);
+ esw_warn(dev, "Failed to create TC miss FDB Table err %d\n", err);
+ goto tc_miss_table_err;
+ }
+
+ err = esw_chains_create(esw, esw->fdb_table.offloads.tc_miss_table);
if (err) {
esw_warn(dev, "Failed to open fdb chains err(%d)\n", err);
goto fdb_chains_err;
send_vport_err:
esw_chains_destroy(esw, esw_chains(esw));
fdb_chains_err:
+ mlx5_destroy_flow_table(esw->fdb_table.offloads.tc_miss_table);
+tc_miss_table_err:
mlx5_destroy_flow_table(esw->fdb_table.offloads.slow_fdb);
slow_fdb_err:
/* Holds true only as long as DMFS is the default */
esw_chains_destroy(esw, esw_chains(esw));
+ mlx5_destroy_flow_table(esw->fdb_table.offloads.tc_miss_table);
mlx5_destroy_flow_table(esw->fdb_table.offloads.slow_fdb);
/* Holds true only as long as DMFS is the default */
mlx5_flow_namespace_set_mode(esw->fdb_table.offloads.ns,
{
struct mlx5_flow_table_attr ft_attr = {};
struct mlx5_flow_namespace *root_ns;
- int err;
+ int err, err2;
root_ns = mlx5_get_flow_namespace(dev, MLX5_FLOW_NAMESPACE_FDB);
if (!root_ns) {
/* As this is the terminating action then the termination table is the
* same prio as the slow path
*/
- ft_attr.flags = MLX5_FLOW_TABLE_TERMINATION |
+ ft_attr.flags = MLX5_FLOW_TABLE_TERMINATION | MLX5_FLOW_TABLE_UNMANAGED |
MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT;
- ft_attr.prio = FDB_SLOW_PATH;
+ ft_attr.prio = FDB_TC_OFFLOAD;
ft_attr.max_fte = 1;
+ ft_attr.level = 1;
ft_attr.autogroup.max_num_groups = 1;
tt->termtbl = mlx5_create_auto_grouped_flow_table(root_ns, &ft_attr);
if (IS_ERR(tt->termtbl)) {
- esw_warn(dev, "Failed to create termination table (error %d)\n",
- IS_ERR(tt->termtbl));
- return -EOPNOTSUPP;
+ err = PTR_ERR(tt->termtbl);
+ esw_warn(dev, "Failed to create termination table, err %pe\n", tt->termtbl);
+ return err;
}
tt->rule = mlx5_add_flow_rules(tt->termtbl, NULL, flow_act,
&tt->dest, 1);
if (IS_ERR(tt->rule)) {
- esw_warn(dev, "Failed to create termination table rule (error %d)\n",
- IS_ERR(tt->rule));
+ err = PTR_ERR(tt->rule);
+ esw_warn(dev, "Failed to create termination table rule, err %pe\n", tt->rule);
goto add_flow_err;
}
return 0;
add_flow_err:
- err = mlx5_destroy_flow_table(tt->termtbl);
- if (err)
- esw_warn(dev, "Failed to destroy termination table\n");
+ err2 = mlx5_destroy_flow_table(tt->termtbl);
+ if (err2)
+ esw_warn(dev, "Failed to destroy termination table, err %d\n", err2);
- return -EOPNOTSUPP;
+ return err;
}
static struct mlx5_termtbl_handle *
}
}
-static bool mlx5_eswitch_termtbl_is_encap_reformat(struct mlx5_pkt_reformat *rt)
-{
- switch (rt->reformat_type) {
- case MLX5_REFORMAT_TYPE_L2_TO_VXLAN:
- case MLX5_REFORMAT_TYPE_L2_TO_NVGRE:
- case MLX5_REFORMAT_TYPE_L2_TO_L2_TUNNEL:
- case MLX5_REFORMAT_TYPE_L2_TO_L3_TUNNEL:
- return true;
- default:
- return false;
- }
-}
-
static void
mlx5_eswitch_termtbl_actions_move(struct mlx5_flow_act *src,
struct mlx5_flow_act *dst)
memset(&src->vlan[1], 0, sizeof(src->vlan[1]));
}
}
-
- if (src->action & MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT &&
- mlx5_eswitch_termtbl_is_encap_reformat(src->pkt_reformat)) {
- src->action &= ~MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT;
- dst->action |= MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT;
- dst->pkt_reformat = src->pkt_reformat;
- src->pkt_reformat = NULL;
- }
}
static bool mlx5_eswitch_offload_is_uplink_port(const struct mlx5_eswitch *esw,
int i;
if (!MLX5_CAP_ESW_FLOWTABLE_FDB(esw->dev, termination_table) ||
+ !MLX5_CAP_ESW_FLOWTABLE_FDB(esw->dev, ignore_flow_level) ||
attr->flags & MLX5_ESW_ATTR_FLAG_SLOW_PATH ||
!mlx5_eswitch_offload_is_uplink_port(esw, spec))
return false;
if (dest[i].type != MLX5_FLOW_DESTINATION_TYPE_VPORT)
continue;
+ if (attr->dests[num_vport_dests].flags & MLX5_ESW_DEST_ENCAP) {
+ term_tbl_act.action |= MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT;
+ term_tbl_act.pkt_reformat = attr->dests[num_vport_dests].pkt_reformat;
+ } else {
+ term_tbl_act.action &= ~MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT;
+ term_tbl_act.pkt_reformat = NULL;
+ }
+
/* get the terminating table for the action list */
tt = mlx5_eswitch_termtbl_get_create(esw, &term_tbl_act,
&dest[i], attr);
if (IS_ERR(tt)) {
- esw_warn(esw->dev, "Failed to get termination table (error %d)\n",
- IS_ERR(tt));
+ esw_warn(esw->dev, "Failed to get termination table, err %pe\n", tt);
goto revert_changes;
}
attr->dests[num_vport_dests].termtbl = tt;
goto revert_changes;
/* create the FTE */
+ flow_act->action &= ~MLX5_FLOW_CONTEXT_ACTION_PACKET_REFORMAT;
+ flow_act->pkt_reformat = NULL;
+ flow_act->flags |= FLOW_ACT_IGNORE_FLOW_LEVEL;
rule = mlx5_add_flow_rules(fdb, spec, flow_act, dest, num_dest);
if (IS_ERR(rule))
goto revert_changes;
#include "fs_core.h"
#include "fs_cmd.h"
+#include "fs_ft_pool.h"
#include "mlx5_core.h"
#include "eswitch.h"
static int mlx5_cmd_stub_create_flow_table(struct mlx5_flow_root_namespace *ns,
struct mlx5_flow_table *ft,
- unsigned int log_size,
+ unsigned int size,
struct mlx5_flow_table *next_ft)
{
+ ft->max_fte = size ? roundup_pow_of_two(size) : 1;
+
return 0;
}
}
static int mlx5_cmd_stub_packet_reformat_alloc(struct mlx5_flow_root_namespace *ns,
- int reformat_type,
- size_t size,
- void *reformat_data,
+ struct mlx5_pkt_reformat_params *params,
enum mlx5_flow_namespace_type namespace,
struct mlx5_pkt_reformat *pkt_reformat)
{
static int mlx5_cmd_create_flow_table(struct mlx5_flow_root_namespace *ns,
struct mlx5_flow_table *ft,
- unsigned int log_size,
+ unsigned int size,
struct mlx5_flow_table *next_ft)
{
int en_encap = !!(ft->flags & MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT);
struct mlx5_core_dev *dev = ns->dev;
int err;
+ if (size != POOL_NEXT_SIZE)
+ size = roundup_pow_of_two(size);
+ size = mlx5_ft_pool_get_avail_sz(dev, ft->type, size);
+ if (!size)
+ return -ENOSPC;
+
MLX5_SET(create_flow_table_in, in, opcode,
MLX5_CMD_OP_CREATE_FLOW_TABLE);
MLX5_SET(create_flow_table_in, in, table_type, ft->type);
MLX5_SET(create_flow_table_in, in, flow_table_context.level, ft->level);
- MLX5_SET(create_flow_table_in, in, flow_table_context.log_size, log_size);
+ MLX5_SET(create_flow_table_in, in, flow_table_context.log_size, size ? ilog2(size) : 0);
MLX5_SET(create_flow_table_in, in, vport_number, ft->vport);
MLX5_SET(create_flow_table_in, in, other_vport,
!!(ft->flags & MLX5_FLOW_TABLE_OTHER_VPORT));
}
err = mlx5_cmd_exec_inout(dev, create_flow_table, in, out);
- if (!err)
+ if (!err) {
ft->id = MLX5_GET(create_flow_table_out, out,
table_id);
+ ft->max_fte = size;
+ } else {
+ mlx5_ft_pool_put_sz(ns->dev, size);
+ }
+
return err;
}
{
u32 in[MLX5_ST_SZ_DW(destroy_flow_table_in)] = {};
struct mlx5_core_dev *dev = ns->dev;
+ int err;
MLX5_SET(destroy_flow_table_in, in, opcode,
MLX5_CMD_OP_DESTROY_FLOW_TABLE);
MLX5_SET(destroy_flow_table_in, in, other_vport,
!!(ft->flags & MLX5_FLOW_TABLE_OTHER_VPORT));
- return mlx5_cmd_exec_in(dev, destroy_flow_table, in);
+ err = mlx5_cmd_exec_in(dev, destroy_flow_table, in);
+ if (!err)
+ mlx5_ft_pool_put_sz(ns->dev, ft->max_fte);
+
+ return err;
}
static int mlx5_cmd_modify_flow_table(struct mlx5_flow_root_namespace *ns,
}
static int mlx5_cmd_packet_reformat_alloc(struct mlx5_flow_root_namespace *ns,
- int reformat_type,
- size_t size,
- void *reformat_data,
+ struct mlx5_pkt_reformat_params *params,
enum mlx5_flow_namespace_type namespace,
struct mlx5_pkt_reformat *pkt_reformat)
{
else
max_encap_size = MLX5_CAP_FLOWTABLE(dev, max_encap_header_size);
- if (size > max_encap_size) {
+ if (params->size > max_encap_size) {
mlx5_core_warn(dev, "encap size %zd too big, max supported is %d\n",
- size, max_encap_size);
+ params->size, max_encap_size);
return -EINVAL;
}
- in = kzalloc(MLX5_ST_SZ_BYTES(alloc_packet_reformat_context_in) + size,
- GFP_KERNEL);
+ in = kzalloc(MLX5_ST_SZ_BYTES(alloc_packet_reformat_context_in) +
+ params->size, GFP_KERNEL);
if (!in)
return -ENOMEM;
reformat = MLX5_ADDR_OF(packet_reformat_context_in,
packet_reformat_context_in,
reformat_data);
- inlen = reformat - (void *)in + size;
+ inlen = reformat - (void *)in + params->size;
MLX5_SET(alloc_packet_reformat_context_in, in, opcode,
MLX5_CMD_OP_ALLOC_PACKET_REFORMAT_CONTEXT);
MLX5_SET(packet_reformat_context_in, packet_reformat_context_in,
- reformat_data_size, size);
+ reformat_data_size, params->size);
+ MLX5_SET(packet_reformat_context_in, packet_reformat_context_in,
+ reformat_type, params->type);
+ MLX5_SET(packet_reformat_context_in, packet_reformat_context_in,
+ reformat_param_0, params->param_0);
MLX5_SET(packet_reformat_context_in, packet_reformat_context_in,
- reformat_type, reformat_type);
- memcpy(reformat, reformat_data, size);
+ reformat_param_1, params->param_1);
+ if (params->data && params->size)
+ memcpy(reformat, params->data, params->size);
err = mlx5_cmd_exec(dev, in, inlen, out, sizeof(out));
struct mlx5_flow_cmds {
int (*create_flow_table)(struct mlx5_flow_root_namespace *ns,
struct mlx5_flow_table *ft,
- unsigned int log_size,
+ unsigned int size,
struct mlx5_flow_table *next_ft);
int (*destroy_flow_table)(struct mlx5_flow_root_namespace *ns,
struct mlx5_flow_table *ft);
bool disconnect);
int (*packet_reformat_alloc)(struct mlx5_flow_root_namespace *ns,
- int reformat_type,
- size_t size,
- void *reformat_data,
+ struct mlx5_pkt_reformat_params *params,
enum mlx5_flow_namespace_type namespace,
struct mlx5_pkt_reformat *pkt_reformat);
#include "mlx5_core.h"
#include "fs_core.h"
#include "fs_cmd.h"
+#include "fs_ft_pool.h"
#include "diag/fs_tracepoint.h"
#include "accel/ipsec.h"
#include "fpga/ipsec.h"
return fg;
}
-static struct mlx5_flow_table *alloc_flow_table(int level, u16 vport, int max_fte,
+static struct mlx5_flow_table *alloc_flow_table(int level, u16 vport,
enum fs_flow_table_type table_type,
enum fs_flow_table_op_mod op_mod,
u32 flags)
ft->op_mod = op_mod;
ft->type = table_type;
ft->vport = vport;
- ft->max_fte = max_fte;
ft->flags = flags;
INIT_LIST_HEAD(&ft->fwd_rules);
mutex_init(&ft->lock);
struct mlx5_flow_table *next_ft;
struct fs_prio *fs_prio = NULL;
struct mlx5_flow_table *ft;
- int log_table_sz;
int err;
if (!root) {
*/
ft = alloc_flow_table(ft_attr->level,
vport,
- ft_attr->max_fte ? roundup_pow_of_two(ft_attr->max_fte) : 0,
root->table_type,
op_mod, ft_attr->flags);
if (IS_ERR(ft)) {
}
tree_init_node(&ft->node, del_hw_flow_table, del_sw_flow_table);
- log_table_sz = ft->max_fte ? ilog2(ft->max_fte) : 0;
next_ft = unmanaged ? ft_attr->next_ft :
find_next_chained_ft(fs_prio);
ft->def_miss_action = ns->def_miss_action;
ft->ns = ns;
- err = root->cmds->create_flow_table(root, ft, log_table_sz, next_ft);
+ err = root->cmds->create_flow_table(root, ft, ft_attr->max_fte, next_ft);
if (err)
goto free_ft;
ft_attr.level = level;
ft_attr.prio = prio;
+ ft_attr.max_fte = 1;
+
return __mlx5_create_flow_table(ns, &ft_attr, FS_FT_OP_MOD_LAG_DEMUX, 0);
}
EXPORT_SYMBOL(mlx5_create_lag_demux_flow_table);
+#define MAX_FLOW_GROUP_SIZE BIT(24)
struct mlx5_flow_table*
mlx5_create_auto_grouped_flow_table(struct mlx5_flow_namespace *ns,
struct mlx5_flow_table_attr *ft_attr)
{
int num_reserved_entries = ft_attr->autogroup.num_reserved_entries;
- int autogroups_max_fte = ft_attr->max_fte - num_reserved_entries;
int max_num_groups = ft_attr->autogroup.max_num_groups;
struct mlx5_flow_table *ft;
-
- if (max_num_groups > autogroups_max_fte)
- return ERR_PTR(-EINVAL);
- if (num_reserved_entries > ft_attr->max_fte)
- return ERR_PTR(-EINVAL);
+ int autogroups_max_fte;
ft = mlx5_create_flow_table(ns, ft_attr);
if (IS_ERR(ft))
return ft;
+ autogroups_max_fte = ft->max_fte - num_reserved_entries;
+ if (max_num_groups > autogroups_max_fte)
+ goto err_validate;
+ if (num_reserved_entries > ft->max_fte)
+ goto err_validate;
+
+ /* Align the number of groups according to the largest group size */
+ if (autogroups_max_fte / (max_num_groups + 1) > MAX_FLOW_GROUP_SIZE)
+ max_num_groups = (autogroups_max_fte / MAX_FLOW_GROUP_SIZE) - 1;
+
ft->autogroup.active = true;
ft->autogroup.required_groups = max_num_groups;
ft->autogroup.max_fte = autogroups_max_fte;
ft->autogroup.group_size = autogroups_max_fte / (max_num_groups + 1);
return ft;
+
+err_validate:
+ mlx5_destroy_flow_table(ft);
+ return ERR_PTR(-ENOSPC);
}
EXPORT_SYMBOL(mlx5_create_auto_grouped_flow_table);
mlx5_cleanup_fc_stats(dev);
kmem_cache_destroy(steering->ftes_cache);
kmem_cache_destroy(steering->fgs_cache);
+ mlx5_ft_pool_destroy(dev);
kfree(steering);
}
if (err)
goto out_err;
+ maj_prio = fs_create_prio(&steering->fdb_root_ns->ns, FDB_TC_MISS, 1);
+ if (IS_ERR(maj_prio)) {
+ err = PTR_ERR(maj_prio);
+ goto out_err;
+ }
+
+ maj_prio = fs_create_prio(&steering->fdb_root_ns->ns, FDB_BR_OFFLOAD, 3);
+ if (IS_ERR(maj_prio)) {
+ err = PTR_ERR(maj_prio);
+ goto out_err;
+ }
+
maj_prio = fs_create_prio(&steering->fdb_root_ns->ns, FDB_SLOW_PATH, 1);
if (IS_ERR(maj_prio)) {
err = PTR_ERR(maj_prio);
if (err)
return err;
+ err = mlx5_ft_pool_init(dev);
+ if (err)
+ return err;
+
steering = kzalloc(sizeof(*steering), GFP_KERNEL);
- if (!steering)
- return -ENOMEM;
+ if (!steering) {
+ err = -ENOMEM;
+ goto err;
+ }
+
steering->dev = dev;
dev->priv.steering = steering;
EXPORT_SYMBOL(mlx5_modify_header_dealloc);
struct mlx5_pkt_reformat *mlx5_packet_reformat_alloc(struct mlx5_core_dev *dev,
- int reformat_type,
- size_t size,
- void *reformat_data,
+ struct mlx5_pkt_reformat_params *params,
enum mlx5_flow_namespace_type ns_type)
{
struct mlx5_pkt_reformat *pkt_reformat;
return ERR_PTR(-ENOMEM);
pkt_reformat->ns_type = ns_type;
- pkt_reformat->reformat_type = reformat_type;
- err = root->cmds->packet_reformat_alloc(root, reformat_type, size,
- reformat_data, ns_type,
+ pkt_reformat->reformat_type = params->type;
+ err = root->cmds->packet_reformat_alloc(root, params, ns_type,
pkt_reformat);
if (err) {
kfree(pkt_reformat);
#define MLX5_CAP_FLOWTABLE_TYPE(mdev, cap, type) ( \
(type == FS_FT_NIC_RX) ? MLX5_CAP_FLOWTABLE_NIC_RX(mdev, cap) : \
+ (type == FS_FT_NIC_TX) ? MLX5_CAP_FLOWTABLE_NIC_TX(mdev, cap) : \
(type == FS_FT_ESW_EGRESS_ACL) ? MLX5_CAP_ESW_EGRESS_ACL(mdev, cap) : \
(type == FS_FT_ESW_INGRESS_ACL) ? MLX5_CAP_ESW_INGRESS_ACL(mdev, cap) : \
(type == FS_FT_FDB) ? MLX5_CAP_ESW_FLOWTABLE_FDB(mdev, cap) : \
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2021 Mellanox Technologies. */
+
+#include "fs_ft_pool.h"
+
+/* Firmware currently has 4 pool of 4 sizes that it supports (FT_POOLS),
+ * and a virtual memory region of 16M (MLX5_FT_SIZE), this region is duplicated
+ * for each flow table pool. We can allocate up to 16M of each pool,
+ * and we keep track of how much we used via mlx5_ft_pool_get_avail_sz.
+ * Firmware doesn't report any of this for now.
+ * ESW_POOL is expected to be sorted from large to small and match firmware
+ * pools.
+ */
+#define FT_SIZE (16 * 1024 * 1024)
+static const unsigned int FT_POOLS[] = { 4 * 1024 * 1024,
+ 1 * 1024 * 1024,
+ 64 * 1024,
+ 128,
+ 1 /* size for termination tables */ };
+struct mlx5_ft_pool {
+ int ft_left[ARRAY_SIZE(FT_POOLS)];
+};
+
+int mlx5_ft_pool_init(struct mlx5_core_dev *dev)
+{
+ struct mlx5_ft_pool *ft_pool;
+ int i;
+
+ ft_pool = kzalloc(sizeof(*ft_pool), GFP_KERNEL);
+ if (!ft_pool)
+ return -ENOMEM;
+
+ for (i = ARRAY_SIZE(FT_POOLS) - 1; i >= 0; i--)
+ ft_pool->ft_left[i] = FT_SIZE / FT_POOLS[i];
+
+ dev->priv.ft_pool = ft_pool;
+ return 0;
+}
+
+void mlx5_ft_pool_destroy(struct mlx5_core_dev *dev)
+{
+ kfree(dev->priv.ft_pool);
+}
+
+int
+mlx5_ft_pool_get_avail_sz(struct mlx5_core_dev *dev, enum fs_flow_table_type table_type,
+ int desired_size)
+{
+ u32 max_ft_size = 1 << MLX5_CAP_FLOWTABLE_TYPE(dev, log_max_ft_size, table_type);
+ int i, found_i = -1;
+
+ for (i = ARRAY_SIZE(FT_POOLS) - 1; i >= 0; i--) {
+ if (dev->priv.ft_pool->ft_left[i] && FT_POOLS[i] >= desired_size &&
+ FT_POOLS[i] <= max_ft_size) {
+ found_i = i;
+ if (desired_size != POOL_NEXT_SIZE)
+ break;
+ }
+ }
+
+ if (found_i != -1) {
+ --dev->priv.ft_pool->ft_left[found_i];
+ return FT_POOLS[found_i];
+ }
+
+ return 0;
+}
+
+void
+mlx5_ft_pool_put_sz(struct mlx5_core_dev *dev, int sz)
+{
+ int i;
+
+ if (!sz)
+ return;
+
+ for (i = ARRAY_SIZE(FT_POOLS) - 1; i >= 0; i--) {
+ if (sz == FT_POOLS[i]) {
+ ++dev->priv.ft_pool->ft_left[i];
+ return;
+ }
+ }
+
+ WARN_ONCE(1, "Couldn't find size %d in flow table size pool", sz);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2021 Mellanox Technologies. */
+
+#ifndef __MLX5_FS_FT_POOL_H__
+#define __MLX5_FS_FT_POOL_H__
+
+#include <linux/mlx5/driver.h>
+#include "fs_core.h"
+
+#define POOL_NEXT_SIZE 0
+
+int mlx5_ft_pool_init(struct mlx5_core_dev *dev);
+void mlx5_ft_pool_destroy(struct mlx5_core_dev *dev);
+
+int
+mlx5_ft_pool_get_avail_sz(struct mlx5_core_dev *dev, enum fs_flow_table_type table_type,
+ int desired_size);
+void
+mlx5_ft_pool_put_sz(struct mlx5_core_dev *dev, int sz);
+
+#endif /* __MLX5_FS_FT_POOL_H__ */
if (err)
return err;
+ if (MLX5_CAP_GEN(dev, hca_cap_2)) {
+ err = mlx5_core_get_caps(dev, MLX5_CAP_GENERAL_2);
+ if (err)
+ return err;
+ }
+
if (MLX5_CAP_GEN(dev, eth_net_offloads)) {
err = mlx5_core_get_caps(dev, MLX5_CAP_ETHERNET_OFFLOADS);
if (err)
reset_abort_work);
struct mlx5_core_dev *dev = fw_reset->dev;
+ if (!test_bit(MLX5_FW_RESET_FLAGS_RESET_REQUESTED, &fw_reset->reset_flags))
+ return;
+
mlx5_sync_reset_clear_reset_requested(dev, true);
mlx5_core_warn(dev, "PCI Sync FW Update Reset Aborted.\n");
}
MLX5_PTYS_RATE_FDR = 1 << 4,
MLX5_PTYS_RATE_EDR = 1 << 5,
MLX5_PTYS_RATE_HDR = 1 << 6,
+ MLX5_PTYS_RATE_NDR = 1 << 7,
};
static inline int mlx5_ptys_rate_enum_to_int(enum mlx5_ptys_rate rate)
case MLX5_PTYS_RATE_FDR: return 14000;
case MLX5_PTYS_RATE_EDR: return 25000;
case MLX5_PTYS_RATE_HDR: return 50000;
+ case MLX5_PTYS_RATE_NDR: return 100000;
default: return -1;
}
}
}
EXPORT_SYMBOL(mlx5_cmd_destroy_vport_lag);
+static int mlx5_lag_netdev_event(struct notifier_block *this,
+ unsigned long event, void *ptr);
+static void mlx5_do_bond_work(struct work_struct *work);
+
+static void mlx5_ldev_free(struct kref *ref)
+{
+ struct mlx5_lag *ldev = container_of(ref, struct mlx5_lag, ref);
+
+ if (ldev->nb.notifier_call)
+ unregister_netdevice_notifier_net(&init_net, &ldev->nb);
+ mlx5_lag_mp_cleanup(ldev);
+ cancel_delayed_work_sync(&ldev->bond_work);
+ destroy_workqueue(ldev->wq);
+ kfree(ldev);
+}
+
+static void mlx5_ldev_put(struct mlx5_lag *ldev)
+{
+ kref_put(&ldev->ref, mlx5_ldev_free);
+}
+
+static void mlx5_ldev_get(struct mlx5_lag *ldev)
+{
+ kref_get(&ldev->ref);
+}
+
+static struct mlx5_lag *mlx5_lag_dev_alloc(struct mlx5_core_dev *dev)
+{
+ struct mlx5_lag *ldev;
+ int err;
+
+ ldev = kzalloc(sizeof(*ldev), GFP_KERNEL);
+ if (!ldev)
+ return NULL;
+
+ ldev->wq = create_singlethread_workqueue("mlx5_lag");
+ if (!ldev->wq) {
+ kfree(ldev);
+ return NULL;
+ }
+
+ kref_init(&ldev->ref);
+ INIT_DELAYED_WORK(&ldev->bond_work, mlx5_do_bond_work);
+
+ ldev->nb.notifier_call = mlx5_lag_netdev_event;
+ if (register_netdevice_notifier_net(&init_net, &ldev->nb)) {
+ ldev->nb.notifier_call = NULL;
+ mlx5_core_err(dev, "Failed to register LAG netdev notifier\n");
+ }
+
+ err = mlx5_lag_mp_init(ldev);
+ if (err)
+ mlx5_core_err(dev, "Failed to init multipath lag err=%d\n",
+ err);
+
+ return ldev;
+}
+
int mlx5_lag_dev_get_netdev_idx(struct mlx5_lag *ldev,
struct net_device *ndev)
{
static void mlx5_infer_tx_affinity_mapping(struct lag_tracker *tracker,
u8 *port1, u8 *port2)
{
+ bool p1en;
+ bool p2en;
+
+ p1en = tracker->netdev_state[MLX5_LAG_P1].tx_enabled &&
+ tracker->netdev_state[MLX5_LAG_P1].link_up;
+
+ p2en = tracker->netdev_state[MLX5_LAG_P2].tx_enabled &&
+ tracker->netdev_state[MLX5_LAG_P2].link_up;
+
*port1 = 1;
*port2 = 2;
- if (!tracker->netdev_state[MLX5_LAG_P1].tx_enabled ||
- !tracker->netdev_state[MLX5_LAG_P1].link_up) {
- *port1 = 2;
+ if ((!p1en && !p2en) || (p1en && p2en))
return;
- }
- if (!tracker->netdev_state[MLX5_LAG_P2].tx_enabled ||
- !tracker->netdev_state[MLX5_LAG_P2].link_up)
+ if (p1en)
*port2 = 1;
+ else
+ *port1 = 2;
}
void mlx5_modify_lag(struct mlx5_lag *ldev,
if (!ldev->pf[i].dev)
continue;
+ if (ldev->pf[i].dev->priv.flags &
+ MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV)
+ continue;
+
ldev->pf[i].dev->priv.flags &= ~MLX5_PRIV_FLAGS_DISABLE_IB_ADEV;
mlx5_rescan_drivers_locked(ldev->pf[i].dev);
}
}
}
+static void mlx5_disable_lag(struct mlx5_lag *ldev)
+{
+ struct mlx5_core_dev *dev0 = ldev->pf[MLX5_LAG_P1].dev;
+ struct mlx5_core_dev *dev1 = ldev->pf[MLX5_LAG_P2].dev;
+ bool roce_lag;
+ int err;
+
+ roce_lag = __mlx5_lag_is_roce(ldev);
+
+ if (roce_lag) {
+ if (!(dev0->priv.flags & MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV)) {
+ dev0->priv.flags |= MLX5_PRIV_FLAGS_DISABLE_IB_ADEV;
+ mlx5_rescan_drivers_locked(dev0);
+ }
+ mlx5_nic_vport_disable_roce(dev1);
+ }
+
+ err = mlx5_deactivate_lag(ldev);
+ if (err)
+ return;
+
+ if (roce_lag)
+ mlx5_lag_add_devices(ldev);
+}
+
static void mlx5_do_bond(struct mlx5_lag *ldev)
{
struct mlx5_core_dev *dev0 = ldev->pf[MLX5_LAG_P1].dev;
if (!mlx5_lag_is_ready(ldev))
return;
- spin_lock(&lag_lock);
tracker = ldev->tracker;
- spin_unlock(&lag_lock);
do_bond = tracker.is_bonded && mlx5_lag_check_prereq(ldev);
!mlx5_sriov_is_enabled(dev1);
#ifdef CONFIG_MLX5_ESWITCH
- roce_lag &= dev0->priv.eswitch->mode == MLX5_ESWITCH_NONE &&
- dev1->priv.eswitch->mode == MLX5_ESWITCH_NONE;
+ roce_lag = roce_lag &&
+ dev0->priv.eswitch->mode == MLX5_ESWITCH_NONE &&
+ dev1->priv.eswitch->mode == MLX5_ESWITCH_NONE;
#endif
if (roce_lag)
} else if (do_bond && __mlx5_lag_is_active(ldev)) {
mlx5_modify_lag(ldev, &tracker);
} else if (!do_bond && __mlx5_lag_is_active(ldev)) {
- roce_lag = __mlx5_lag_is_roce(ldev);
-
- if (roce_lag) {
- dev0->priv.flags |= MLX5_PRIV_FLAGS_DISABLE_IB_ADEV;
- mlx5_rescan_drivers_locked(dev0);
- mlx5_nic_vport_disable_roce(dev1);
- }
-
- err = mlx5_deactivate_lag(ldev);
- if (err)
- return;
-
- if (roce_lag)
- mlx5_lag_add_devices(ldev);
+ mlx5_disable_lag(ldev);
}
}
break;
}
- spin_lock(&lag_lock);
ldev->tracker = tracker;
- spin_unlock(&lag_lock);
if (changed)
mlx5_queue_bond_work(ldev, 0);
return NOTIFY_DONE;
}
-static struct mlx5_lag *mlx5_lag_dev_alloc(void)
+static void mlx5_ldev_add_netdev(struct mlx5_lag *ldev,
+ struct mlx5_core_dev *dev,
+ struct net_device *netdev)
{
- struct mlx5_lag *ldev;
-
- ldev = kzalloc(sizeof(*ldev), GFP_KERNEL);
- if (!ldev)
- return NULL;
-
- ldev->wq = create_singlethread_workqueue("mlx5_lag");
- if (!ldev->wq) {
- kfree(ldev);
- return NULL;
- }
+ unsigned int fn = PCI_FUNC(dev->pdev->devfn);
- INIT_DELAYED_WORK(&ldev->bond_work, mlx5_do_bond_work);
+ if (fn >= MLX5_MAX_PORTS)
+ return;
- return ldev;
+ spin_lock(&lag_lock);
+ ldev->pf[fn].netdev = netdev;
+ ldev->tracker.netdev_state[fn].link_up = 0;
+ ldev->tracker.netdev_state[fn].tx_enabled = 0;
+ spin_unlock(&lag_lock);
}
-static void mlx5_lag_dev_free(struct mlx5_lag *ldev)
+static void mlx5_ldev_remove_netdev(struct mlx5_lag *ldev,
+ struct net_device *netdev)
{
- destroy_workqueue(ldev->wq);
- kfree(ldev);
+ int i;
+
+ spin_lock(&lag_lock);
+ for (i = 0; i < MLX5_MAX_PORTS; i++) {
+ if (ldev->pf[i].netdev == netdev) {
+ ldev->pf[i].netdev = NULL;
+ break;
+ }
+ }
+ spin_unlock(&lag_lock);
}
-static int mlx5_lag_dev_add_pf(struct mlx5_lag *ldev,
- struct mlx5_core_dev *dev,
- struct net_device *netdev)
+static void mlx5_ldev_add_mdev(struct mlx5_lag *ldev,
+ struct mlx5_core_dev *dev)
{
unsigned int fn = PCI_FUNC(dev->pdev->devfn);
if (fn >= MLX5_MAX_PORTS)
- return -EPERM;
-
- spin_lock(&lag_lock);
- ldev->pf[fn].dev = dev;
- ldev->pf[fn].netdev = netdev;
- ldev->tracker.netdev_state[fn].link_up = 0;
- ldev->tracker.netdev_state[fn].tx_enabled = 0;
+ return;
+ ldev->pf[fn].dev = dev;
dev->priv.lag = ldev;
-
- spin_unlock(&lag_lock);
-
- return fn;
}
-static void mlx5_lag_dev_remove_pf(struct mlx5_lag *ldev,
- struct mlx5_core_dev *dev)
+/* Must be called with intf_mutex held */
+static void mlx5_ldev_remove_mdev(struct mlx5_lag *ldev,
+ struct mlx5_core_dev *dev)
{
int i;
if (i == MLX5_MAX_PORTS)
return;
- spin_lock(&lag_lock);
- memset(&ldev->pf[i], 0, sizeof(*ldev->pf));
-
+ ldev->pf[i].dev = NULL;
dev->priv.lag = NULL;
- spin_unlock(&lag_lock);
}
/* Must be called with intf_mutex held */
-void mlx5_lag_add(struct mlx5_core_dev *dev, struct net_device *netdev)
+static void __mlx5_lag_dev_add_mdev(struct mlx5_core_dev *dev)
{
struct mlx5_lag *ldev = NULL;
struct mlx5_core_dev *tmp_dev;
- int i, err;
if (!MLX5_CAP_GEN(dev, vport_group_manager) ||
!MLX5_CAP_GEN(dev, lag_master) ||
ldev = tmp_dev->priv.lag;
if (!ldev) {
- ldev = mlx5_lag_dev_alloc();
+ ldev = mlx5_lag_dev_alloc(dev);
if (!ldev) {
mlx5_core_err(dev, "Failed to alloc lag dev\n");
return;
}
+ } else {
+ mlx5_ldev_get(ldev);
}
- if (mlx5_lag_dev_add_pf(ldev, dev, netdev) < 0)
- return;
+ mlx5_ldev_add_mdev(ldev, dev);
- for (i = 0; i < MLX5_MAX_PORTS; i++)
- if (!ldev->pf[i].dev)
- break;
+ return;
+}
- if (i >= MLX5_MAX_PORTS)
- ldev->flags |= MLX5_LAG_FLAG_READY;
+void mlx5_lag_remove_mdev(struct mlx5_core_dev *dev)
+{
+ struct mlx5_lag *ldev;
- if (!ldev->nb.notifier_call) {
- ldev->nb.notifier_call = mlx5_lag_netdev_event;
- if (register_netdevice_notifier_net(&init_net, &ldev->nb)) {
- ldev->nb.notifier_call = NULL;
- mlx5_core_err(dev, "Failed to register LAG netdev notifier\n");
- }
- }
+ ldev = mlx5_lag_dev(dev);
+ if (!ldev)
+ return;
- err = mlx5_lag_mp_init(ldev);
- if (err)
- mlx5_core_err(dev, "Failed to init multipath lag err=%d\n",
- err);
+ mlx5_dev_list_lock();
+ mlx5_ldev_remove_mdev(ldev, dev);
+ mlx5_dev_list_unlock();
+ mlx5_ldev_put(ldev);
+}
+
+void mlx5_lag_add_mdev(struct mlx5_core_dev *dev)
+{
+ mlx5_dev_list_lock();
+ __mlx5_lag_dev_add_mdev(dev);
+ mlx5_dev_list_unlock();
}
/* Must be called with intf_mutex held */
-void mlx5_lag_remove(struct mlx5_core_dev *dev)
+void mlx5_lag_remove_netdev(struct mlx5_core_dev *dev,
+ struct net_device *netdev)
{
struct mlx5_lag *ldev;
- int i;
- ldev = mlx5_lag_dev_get(dev);
+ ldev = mlx5_lag_dev(dev);
if (!ldev)
return;
if (__mlx5_lag_is_active(ldev))
- mlx5_deactivate_lag(ldev);
-
- mlx5_lag_dev_remove_pf(ldev, dev);
+ mlx5_disable_lag(ldev);
+ mlx5_ldev_remove_netdev(ldev, netdev);
ldev->flags &= ~MLX5_LAG_FLAG_READY;
+}
+
+/* Must be called with intf_mutex held */
+void mlx5_lag_add_netdev(struct mlx5_core_dev *dev,
+ struct net_device *netdev)
+{
+ struct mlx5_lag *ldev;
+ int i;
+
+ ldev = mlx5_lag_dev(dev);
+ if (!ldev)
+ return;
+
+ mlx5_ldev_add_netdev(ldev, dev, netdev);
for (i = 0; i < MLX5_MAX_PORTS; i++)
- if (ldev->pf[i].dev)
+ if (!ldev->pf[i].dev)
break;
- if (i == MLX5_MAX_PORTS) {
- if (ldev->nb.notifier_call) {
- unregister_netdevice_notifier_net(&init_net, &ldev->nb);
- ldev->nb.notifier_call = NULL;
- }
- mlx5_lag_mp_cleanup(ldev);
- cancel_delayed_work_sync(&ldev->bond_work);
- mlx5_lag_dev_free(ldev);
- }
+ if (i >= MLX5_MAX_PORTS)
+ ldev->flags |= MLX5_LAG_FLAG_READY;
}
bool mlx5_lag_is_roce(struct mlx5_core_dev *dev)
bool res;
spin_lock(&lag_lock);
- ldev = mlx5_lag_dev_get(dev);
+ ldev = mlx5_lag_dev(dev);
res = ldev && __mlx5_lag_is_roce(ldev);
spin_unlock(&lag_lock);
bool res;
spin_lock(&lag_lock);
- ldev = mlx5_lag_dev_get(dev);
+ ldev = mlx5_lag_dev(dev);
res = ldev && __mlx5_lag_is_active(ldev);
spin_unlock(&lag_lock);
bool res;
spin_lock(&lag_lock);
- ldev = mlx5_lag_dev_get(dev);
+ ldev = mlx5_lag_dev(dev);
res = ldev && __mlx5_lag_is_sriov(ldev);
spin_unlock(&lag_lock);
struct mlx5_lag *ldev;
mlx5_dev_list_lock();
- ldev = mlx5_lag_dev_get(dev);
+ ldev = mlx5_lag_dev(dev);
if (!ldev)
goto unlock;
struct mlx5_lag *ldev;
spin_lock(&lag_lock);
- ldev = mlx5_lag_dev_get(dev);
+ ldev = mlx5_lag_dev(dev);
if (!(ldev && __mlx5_lag_is_roce(ldev)))
goto unlock;
u8 port = 0;
spin_lock(&lag_lock);
- ldev = mlx5_lag_dev_get(dev);
+ ldev = mlx5_lag_dev(dev);
if (!(ldev && __mlx5_lag_is_roce(ldev)))
goto unlock;
memset(values, 0, sizeof(*values) * num_counters);
spin_lock(&lag_lock);
- ldev = mlx5_lag_dev_get(dev);
+ ldev = mlx5_lag_dev(dev);
if (ldev && __mlx5_lag_is_active(ldev)) {
num_ports = MLX5_MAX_PORTS;
mdev[MLX5_LAG_P1] = ldev->pf[MLX5_LAG_P1].dev;
struct mlx5_lag {
u8 flags;
u8 v2p_map[MLX5_MAX_PORTS];
+ struct kref ref;
struct lag_func pf[MLX5_MAX_PORTS];
struct lag_tracker tracker;
struct workqueue_struct *wq;
};
static inline struct mlx5_lag *
-mlx5_lag_dev_get(struct mlx5_core_dev *dev)
+mlx5_lag_dev(struct mlx5_core_dev *dev)
{
return dev->priv.lag;
}
struct mlx5_lag *ldev;
bool res;
- ldev = mlx5_lag_dev_get(dev);
+ ldev = mlx5_lag_dev(dev);
res = ldev && __mlx5_lag_is_multipath(ldev);
return res;
struct lag_mp *mp = &ldev->lag_mp;
int err;
+ /* always clear mfi, as it might become stale when a route delete event
+ * has been missed
+ */
+ mp->mfi = NULL;
+
if (mp->fib_nb.notifier_call)
return 0;
unregister_fib_notifier(&init_net, &mp->fib_nb);
destroy_workqueue(mp->wq);
mp->fib_nb.notifier_call = NULL;
+ mp->mfi = NULL;
}
/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
-/* Copyright (c) 2018 Mellanox Technologies */
+/* Copyright (c) 2018-2021, Mellanox Technologies inc. All rights reserved. */
#ifndef __LIB_MLX5_EQ_H__
#define __LIB_MLX5_EQ_H__
unsigned int irqn;
u8 eqn;
struct mlx5_rsc_debug *dbg;
+ struct mlx5_irq *irq;
};
struct mlx5_eq_async {
#include <linux/mlx5/fs.h>
#include "lib/fs_chains.h"
+#include "fs_ft_pool.h"
#include "en/mapping.h"
#include "fs_core.h"
#include "en_tc.h"
#define chains_lock(chains) ((chains)->lock)
#define chains_ht(chains) ((chains)->chains_ht)
#define prios_ht(chains) ((chains)->prios_ht)
-#define ft_pool_left(chains) ((chains)->ft_left)
#define tc_default_ft(chains) ((chains)->tc_default_ft)
#define tc_end_ft(chains) ((chains)->tc_end_ft)
#define ns_to_chains_fs_prio(ns) ((ns) == MLX5_FLOW_NAMESPACE_FDB ? \
FDB_TC_OFFLOAD : MLX5E_TC_PRIO)
-
-/* Firmware currently has 4 pool of 4 sizes that it supports (FT_POOLS),
- * and a virtual memory region of 16M (MLX5_FT_SIZE), this region is duplicated
- * for each flow table pool. We can allocate up to 16M of each pool,
- * and we keep track of how much we used via get_next_avail_sz_from_pool.
- * Firmware doesn't report any of this for now.
- * ESW_POOL is expected to be sorted from large to small and match firmware
- * pools.
- */
-#define FT_SIZE (16 * 1024 * 1024)
-static const unsigned int FT_POOLS[] = { 4 * 1024 * 1024,
- 1 * 1024 * 1024,
- 64 * 1024,
- 128 };
#define FT_TBL_SZ (64 * 1024)
struct mlx5_fs_chains {
enum mlx5_flow_namespace_type ns;
u32 group_num;
u32 flags;
-
- int ft_left[ARRAY_SIZE(FT_POOLS)];
};
struct fs_chain {
return chains->flags & MLX5_CHAINS_AND_PRIOS_SUPPORTED;
}
-static bool mlx5_chains_ignore_flow_level_supported(struct mlx5_fs_chains *chains)
+bool mlx5_chains_ignore_flow_level_supported(struct mlx5_fs_chains *chains)
{
return chains->flags & MLX5_CHAINS_IGNORE_FLOW_LEVEL_SUPPORTED;
}
tc_end_ft(chains) = ft;
}
-#define POOL_NEXT_SIZE 0
-static int
-mlx5_chains_get_avail_sz_from_pool(struct mlx5_fs_chains *chains,
- int desired_size)
-{
- int i, found_i = -1;
-
- for (i = ARRAY_SIZE(FT_POOLS) - 1; i >= 0; i--) {
- if (ft_pool_left(chains)[i] && FT_POOLS[i] > desired_size) {
- found_i = i;
- if (desired_size != POOL_NEXT_SIZE)
- break;
- }
- }
-
- if (found_i != -1) {
- --ft_pool_left(chains)[found_i];
- return FT_POOLS[found_i];
- }
-
- return 0;
-}
-
-static void
-mlx5_chains_put_sz_to_pool(struct mlx5_fs_chains *chains, int sz)
-{
- int i;
-
- for (i = ARRAY_SIZE(FT_POOLS) - 1; i >= 0; i--) {
- if (sz == FT_POOLS[i]) {
- ++ft_pool_left(chains)[i];
- return;
- }
- }
-
- WARN_ONCE(1, "Couldn't find size %d in flow table size pool", sz);
-}
-
-static void
-mlx5_chains_init_sz_pool(struct mlx5_fs_chains *chains, u32 ft_max)
-{
- int i;
-
- for (i = ARRAY_SIZE(FT_POOLS) - 1; i >= 0; i--)
- ft_pool_left(chains)[i] =
- FT_POOLS[i] <= ft_max ? FT_SIZE / FT_POOLS[i] : 0;
-}
-
static struct mlx5_flow_table *
mlx5_chains_create_table(struct mlx5_fs_chains *chains,
u32 chain, u32 prio, u32 level)
ft_attr.flags |= (MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT |
MLX5_FLOW_TABLE_TUNNEL_EN_DECAP);
- sz = (chain == mlx5_chains_get_nf_ft_chain(chains)) ?
- mlx5_chains_get_avail_sz_from_pool(chains, FT_TBL_SZ) :
- mlx5_chains_get_avail_sz_from_pool(chains, POOL_NEXT_SIZE);
- if (!sz)
- return ERR_PTR(-ENOSPC);
+ sz = (chain == mlx5_chains_get_nf_ft_chain(chains)) ? FT_TBL_SZ : POOL_NEXT_SIZE;
ft_attr.max_fte = sz;
/* We use tc_default_ft(chains) as the table's next_ft till
if (IS_ERR(ft)) {
mlx5_core_warn(chains->dev, "Failed to create chains table err %d (chain: %d, prio: %d, level: %d, size: %d)\n",
(int)PTR_ERR(ft), chain, prio, level, sz);
- mlx5_chains_put_sz_to_pool(chains, sz);
return ft;
}
return ft;
}
-static void
-mlx5_chains_destroy_table(struct mlx5_fs_chains *chains,
- struct mlx5_flow_table *ft)
-{
- mlx5_chains_put_sz_to_pool(chains, ft->max_fte);
- mlx5_destroy_flow_table(ft);
-}
-
static int
create_chain_restore(struct fs_chain *chain)
{
MLX5_SET(set_action_in, modact, field,
mlx5e_tc_attr_to_reg_mappings[chain_to_reg].mfield);
MLX5_SET(set_action_in, modact, offset,
- mlx5e_tc_attr_to_reg_mappings[chain_to_reg].moffset * 8);
+ mlx5e_tc_attr_to_reg_mappings[chain_to_reg].moffset);
MLX5_SET(set_action_in, modact, length,
- mlx5e_tc_attr_to_reg_mappings[chain_to_reg].mlen * 8);
+ mlx5e_tc_attr_to_reg_mappings[chain_to_reg].mlen == 32 ?
+ 0 : mlx5e_tc_attr_to_reg_mappings[chain_to_reg].mlen);
MLX5_SET(set_action_in, modact, data, chain->id);
mod_hdr = mlx5_modify_header_alloc(chains->dev, chains->ns,
1, modact);
err_miss_rule:
mlx5_destroy_flow_group(miss_group);
err_group:
- mlx5_chains_destroy_table(chains, ft);
+ mlx5_destroy_flow_table(ft);
err_create:
err_alloc:
kvfree(prio_s);
prio_params);
mlx5_del_flow_rules(prio->miss_rule);
mlx5_destroy_flow_group(prio->miss_group);
- mlx5_chains_destroy_table(chains, prio->ft);
+ mlx5_destroy_flow_table(prio->ft);
mlx5_chains_put_chain(chain);
kvfree(prio);
}
mlx5_chains_destroy_global_table(struct mlx5_fs_chains *chains,
struct mlx5_flow_table *ft)
{
- mlx5_chains_destroy_table(chains, ft);
+ mlx5_destroy_flow_table(ft);
}
static struct mlx5_fs_chains *
mlx5_chains_get_chain_range(chains_priv),
mlx5_chains_get_prio_range(chains_priv));
- mlx5_chains_init_sz_pool(chains_priv, attr->max_ft_sz);
-
err = rhashtable_init(&chains_ht(chains_priv), &chain_params);
if (err)
goto init_chains_ht_err;
bool
mlx5_chains_prios_supported(struct mlx5_fs_chains *chains);
+bool mlx5_chains_ignore_flow_level_supported(struct mlx5_fs_chains *chains);
bool
mlx5_chains_backwards_supported(struct mlx5_fs_chains *chains);
u32
#else /* CONFIG_MLX5_CLS_ACT */
+static inline bool
+mlx5_chains_ignore_flow_level_supported(struct mlx5_fs_chains *chains)
+{ return false; }
+
static inline struct mlx5_flow_table *
mlx5_chains_get_table(struct mlx5_fs_chains *chains, u32 chain, u32 prio,
u32 level) { return ERR_PTR(-EOPNOTSUPP); }
#include <linux/etherdevice.h>
#include <linux/mlx5/driver.h>
#include <linux/mlx5/mlx5_ifc.h>
+#include <linux/mlx5/mpfs.h>
#include <linux/mlx5/eswitch.h>
#include "mlx5_core.h"
#include "lib/mpfs.h"
mutex_unlock(&mpfs->lock);
return err;
}
+EXPORT_SYMBOL(mlx5_mpfs_add_mac);
int mlx5_mpfs_del_mac(struct mlx5_core_dev *dev, u8 *mac)
{
mutex_unlock(&mpfs->lock);
return err;
}
+EXPORT_SYMBOL(mlx5_mpfs_del_mac);
#ifdef CONFIG_MLX5_MPFS
int mlx5_mpfs_init(struct mlx5_core_dev *dev);
void mlx5_mpfs_cleanup(struct mlx5_core_dev *dev);
-int mlx5_mpfs_add_mac(struct mlx5_core_dev *dev, u8 *mac);
-int mlx5_mpfs_del_mac(struct mlx5_core_dev *dev, u8 *mac);
#else /* #ifndef CONFIG_MLX5_MPFS */
static inline int mlx5_mpfs_init(struct mlx5_core_dev *dev) { return 0; }
static inline void mlx5_mpfs_cleanup(struct mlx5_core_dev *dev) {}
-static inline int mlx5_mpfs_add_mac(struct mlx5_core_dev *dev, u8 *mac) { return 0; }
-static inline int mlx5_mpfs_del_mac(struct mlx5_core_dev *dev, u8 *mac) { return 0; }
#endif
+
#endif
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2021 Mellanox Technologies Ltd */
+
+#ifndef __LIB_MLX5_SF_H__
+#define __LIB_MLX5_SF_H__
+
+#include <linux/mlx5/driver.h>
+
+static inline u16 mlx5_sf_start_function_id(const struct mlx5_core_dev *dev)
+{
+ return MLX5_CAP_GEN(dev, sf_base_id);
+}
+
+#ifdef CONFIG_MLX5_SF
+
+static inline bool mlx5_sf_supported(const struct mlx5_core_dev *dev)
+{
+ return MLX5_CAP_GEN(dev, sf);
+}
+
+static inline u16 mlx5_sf_max_functions(const struct mlx5_core_dev *dev)
+{
+ if (!mlx5_sf_supported(dev))
+ return 0;
+ if (MLX5_CAP_GEN(dev, max_num_sf))
+ return MLX5_CAP_GEN(dev, max_num_sf);
+ else
+ return 1 << MLX5_CAP_GEN(dev, log_max_sf);
+}
+
+#else
+
+static inline bool mlx5_sf_supported(const struct mlx5_core_dev *dev)
+{
+ return false;
+}
+
+static inline u16 mlx5_sf_max_functions(const struct mlx5_core_dev *dev)
+{
+ return 0;
+}
+
+#endif
+
+#endif
#include "sf/vhca_event.h"
#include "sf/dev/dev.h"
#include "sf/sf.h"
+#include "mlx5_irq.h"
MODULE_AUTHOR("Eli Cohen <eli@mellanox.com>");
MODULE_DESCRIPTION("Mellanox 5th generation network adapters (ConnectX series) core driver");
static int handle_hca_cap(struct mlx5_core_dev *dev, void *set_ctx)
{
- struct mlx5_profile *prof = dev->profile;
+ struct mlx5_profile *prof = &dev->profile;
void *set_hca_cap;
int err;
to_fw_pkey_sz(dev, 128));
/* Check log_max_qp from HCA caps to set in current profile */
- if (MLX5_CAP_GEN_MAX(dev, log_max_qp) < profile[prof_sel].log_max_qp) {
+ if (MLX5_CAP_GEN_MAX(dev, log_max_qp) < prof->log_max_qp) {
mlx5_core_warn(dev, "log_max_qp value in current profile is %d, changing it to HCA capability limit (%d)\n",
- profile[prof_sel].log_max_qp,
+ prof->log_max_qp,
MLX5_CAP_GEN_MAX(dev, log_max_qp));
- profile[prof_sel].log_max_qp = MLX5_CAP_GEN_MAX(dev, log_max_qp);
+ prof->log_max_qp = MLX5_CAP_GEN_MAX(dev, log_max_qp);
}
if (prof->mask & MLX5_PROF_MASK_QP_SIZE)
MLX5_SET(cmd_hca_cap, set_hca_cap, log_max_qp,
err = mlx5_core_set_hca_defaults(dev);
if (err) {
mlx5_core_err(dev, "Failed to set hca defaults\n");
- goto err_sriov;
+ goto err_set_hca;
}
mlx5_vhca_event_start(dev);
}
mlx5_sf_dev_table_create(dev);
+ mlx5_lag_add_mdev(dev);
return 0;
mlx5_sf_hw_table_destroy(dev);
err_vhca:
mlx5_vhca_event_stop(dev);
+err_set_hca:
mlx5_cleanup_fs(dev);
err_fs:
mlx5_accel_tls_cleanup(dev);
static void mlx5_unload(struct mlx5_core_dev *dev)
{
+ mlx5_lag_remove_mdev(dev);
mlx5_sf_dev_table_destroy(dev);
mlx5_sriov_detach(dev);
mlx5_ec_cleanup(dev);
struct mlx5_priv *priv = &dev->priv;
int err;
- dev->profile = &profile[profile_idx];
-
+ memcpy(&dev->profile, &profile[profile_idx], sizeof(dev->profile));
INIT_LIST_HEAD(&priv->ctx_list);
spin_lock_init(&priv->ctx_lock);
mutex_init(&dev->intf_state_mutex);
int mlx5_query_qcam_reg(struct mlx5_core_dev *mdev, u32 *qcam,
u8 feature_group, u8 access_reg_group);
-void mlx5_lag_add(struct mlx5_core_dev *dev, struct net_device *netdev);
-void mlx5_lag_remove(struct mlx5_core_dev *dev);
-
-int mlx5_irq_table_init(struct mlx5_core_dev *dev);
-void mlx5_irq_table_cleanup(struct mlx5_core_dev *dev);
-int mlx5_irq_table_create(struct mlx5_core_dev *dev);
-void mlx5_irq_table_destroy(struct mlx5_core_dev *dev);
-int mlx5_irq_attach_nb(struct mlx5_irq_table *irq_table, int vecidx,
- struct notifier_block *nb);
-int mlx5_irq_detach_nb(struct mlx5_irq_table *irq_table, int vecidx,
- struct notifier_block *nb);
-
-int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int devfn,
- int msix_vec_count);
-int mlx5_get_default_msix_vec_count(struct mlx5_core_dev *dev, int num_vfs);
-
-struct cpumask *
-mlx5_irq_get_affinity_mask(struct mlx5_irq_table *irq_table, int vecidx);
-struct cpu_rmap *mlx5_irq_get_rmap(struct mlx5_irq_table *table);
-int mlx5_irq_get_num_comp(struct mlx5_irq_table *table);
-struct mlx5_irq_table *mlx5_irq_table_get(struct mlx5_core_dev *dev);
+void mlx5_lag_add_netdev(struct mlx5_core_dev *dev, struct net_device *netdev);
+void mlx5_lag_remove_netdev(struct mlx5_core_dev *dev, struct net_device *netdev);
+void mlx5_lag_add_mdev(struct mlx5_core_dev *dev);
+void mlx5_lag_remove_mdev(struct mlx5_core_dev *dev);
int mlx5_events_init(struct mlx5_core_dev *dev);
void mlx5_events_cleanup(struct mlx5_core_dev *dev);
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2021 Mellanox Technologies. */
+
+#ifndef __MLX5_IRQ_H__
+#define __MLX5_IRQ_H__
+
+#include <linux/mlx5/driver.h>
+
+#define MLX5_COMP_EQS_PER_SF 8
+
+#define MLX5_IRQ_EQ_CTRL (0)
+
+struct mlx5_irq;
+
+int mlx5_irq_table_init(struct mlx5_core_dev *dev);
+void mlx5_irq_table_cleanup(struct mlx5_core_dev *dev);
+int mlx5_irq_table_create(struct mlx5_core_dev *dev);
+void mlx5_irq_table_destroy(struct mlx5_core_dev *dev);
+int mlx5_irq_table_get_num_comp(struct mlx5_irq_table *table);
+int mlx5_irq_table_get_sfs_vec(struct mlx5_irq_table *table);
+struct mlx5_irq_table *mlx5_irq_table_get(struct mlx5_core_dev *dev);
+
+int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int devfn,
+ int msix_vec_count);
+int mlx5_get_default_msix_vec_count(struct mlx5_core_dev *dev, int num_vfs);
+
+struct mlx5_irq *mlx5_irq_request(struct mlx5_core_dev *dev, u16 vecidx,
+ struct cpumask *affinity);
+void mlx5_irq_release(struct mlx5_irq *irq);
+int mlx5_irq_attach_nb(struct mlx5_irq *irq, struct notifier_block *nb);
+int mlx5_irq_detach_nb(struct mlx5_irq *irq, struct notifier_block *nb);
+struct cpumask *mlx5_irq_get_affinity_mask(struct mlx5_irq *irq);
+int mlx5_irq_get_index(struct mlx5_irq *irq);
+
+#endif /* __MLX5_IRQ_H__ */
mkey_index = MLX5_GET(create_mkey_out, lout, mkey_index);
mkey->iova = MLX5_GET64(mkc, mkc, start_addr);
mkey->size = MLX5_GET64(mkc, mkc, len);
- mkey->key |= mlx5_idx_to_mkey(mkey_index);
+ mkey->key = (u32)mlx5_mkey_variant(mkey->key) | mlx5_idx_to_mkey(mkey_index);
mkey->pd = MLX5_GET(mkc, mkc, pd);
init_waitqueue_head(&mkey->wait);
#include <linux/module.h>
#include <linux/mlx5/driver.h>
#include "mlx5_core.h"
+#include "mlx5_irq.h"
+#include "lib/sf.h"
#ifdef CONFIG_RFS_ACCEL
#include <linux/cpu_rmap.h>
#endif
#define MLX5_MAX_IRQ_NAME (32)
+/* max irq_index is 255. three chars */
+#define MLX5_MAX_IRQ_IDX_CHARS (3)
+
+#define MLX5_SFS_PER_CTRL_IRQ 64
+#define MLX5_IRQ_CTRL_SF_MAX 8
+/* min num of vectores for SFs to be enabled */
+#define MLX5_IRQ_VEC_COMP_BASE_SF 2
+
+#define MLX5_EQ_SHARE_IRQ_MAX_COMP (8)
+#define MLX5_EQ_SHARE_IRQ_MAX_CTRL (UINT_MAX)
+#define MLX5_EQ_SHARE_IRQ_MIN_COMP (1)
+#define MLX5_EQ_SHARE_IRQ_MIN_CTRL (4)
+#define MLX5_EQ_REFS_PER_IRQ (2)
struct mlx5_irq {
+ u32 index;
struct atomic_notifier_head nh;
cpumask_var_t mask;
char name[MLX5_MAX_IRQ_NAME];
+ struct kref kref;
+ int irqn;
+ struct mlx5_irq_pool *pool;
};
-struct mlx5_irq_table {
- struct mlx5_irq *irq;
- int nvec;
-#ifdef CONFIG_RFS_ACCEL
- struct cpu_rmap *rmap;
-#endif
+struct mlx5_irq_pool {
+ char name[MLX5_MAX_IRQ_NAME - MLX5_MAX_IRQ_IDX_CHARS];
+ struct xa_limit xa_num_irqs;
+ struct mutex lock; /* sync IRQs creations */
+ struct xarray irqs;
+ u32 max_threshold;
+ u32 min_threshold;
+ struct mlx5_core_dev *dev;
};
-int mlx5_irq_table_init(struct mlx5_core_dev *dev)
-{
- struct mlx5_irq_table *irq_table;
-
- if (mlx5_core_is_sf(dev))
- return 0;
-
- irq_table = kvzalloc(sizeof(*irq_table), GFP_KERNEL);
- if (!irq_table)
- return -ENOMEM;
-
- dev->priv.irq_table = irq_table;
- return 0;
-}
-
-void mlx5_irq_table_cleanup(struct mlx5_core_dev *dev)
-{
- if (mlx5_core_is_sf(dev))
- return;
-
- kvfree(dev->priv.irq_table);
-}
-
-int mlx5_irq_get_num_comp(struct mlx5_irq_table *table)
-{
- return table->nvec - MLX5_IRQ_VEC_COMP_BASE;
-}
-
-static struct mlx5_irq *mlx5_irq_get(struct mlx5_core_dev *dev, int vecidx)
-{
- struct mlx5_irq_table *irq_table = dev->priv.irq_table;
-
- return &irq_table->irq[vecidx];
-}
+struct mlx5_irq_table {
+ struct mlx5_irq_pool *pf_pool;
+ struct mlx5_irq_pool *sf_ctrl_pool;
+ struct mlx5_irq_pool *sf_comp_pool;
+};
/**
* mlx5_get_default_msix_vec_count - Get the default number of MSI-X vectors
int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int function_id,
int msix_vec_count)
{
- int sz = MLX5_ST_SZ_BYTES(set_hca_cap_in);
+ int query_sz = MLX5_ST_SZ_BYTES(query_hca_cap_out);
+ int set_sz = MLX5_ST_SZ_BYTES(set_hca_cap_in);
+ void *hca_cap = NULL, *query_cap = NULL, *cap;
int num_vf_msix, min_msix, max_msix;
- void *hca_cap, *cap;
int ret;
num_vf_msix = MLX5_CAP_GEN_MAX(dev, num_total_dynamic_vf_msix);
if (msix_vec_count > max_msix)
return -EOVERFLOW;
- hca_cap = kzalloc(sz, GFP_KERNEL);
- if (!hca_cap)
- return -ENOMEM;
+ query_cap = kzalloc(query_sz, GFP_KERNEL);
+ hca_cap = kzalloc(set_sz, GFP_KERNEL);
+ if (!hca_cap || !query_cap) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ ret = mlx5_vport_get_other_func_cap(dev, function_id, query_cap);
+ if (ret)
+ goto out;
cap = MLX5_ADDR_OF(set_hca_cap_in, hca_cap, capability);
+ memcpy(cap, MLX5_ADDR_OF(query_hca_cap_out, query_cap, capability),
+ MLX5_UN_SZ_BYTES(hca_cap_union));
MLX5_SET(cmd_hca_cap, cap, dynamic_msix_table_size, msix_vec_count);
MLX5_SET(set_hca_cap_in, hca_cap, opcode, MLX5_CMD_OP_SET_HCA_CAP);
MLX5_SET(set_hca_cap_in, hca_cap, op_mod,
MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1);
ret = mlx5_cmd_exec_in(dev, set_hca_cap, hca_cap);
+out:
kfree(hca_cap);
+ kfree(query_cap);
return ret;
}
-int mlx5_irq_attach_nb(struct mlx5_irq_table *irq_table, int vecidx,
- struct notifier_block *nb)
+static void irq_release(struct kref *kref)
{
- struct mlx5_irq *irq;
+ struct mlx5_irq *irq = container_of(kref, struct mlx5_irq, kref);
+ struct mlx5_irq_pool *pool = irq->pool;
- irq = &irq_table->irq[vecidx];
- return atomic_notifier_chain_register(&irq->nh, nb);
+ xa_erase(&pool->irqs, irq->index);
+ /* free_irq requires that affinity and rmap will be cleared
+ * before calling it. This is why there is asymmetry with set_rmap
+ * which should be called after alloc_irq but before request_irq.
+ */
+ irq_set_affinity_hint(irq->irqn, NULL);
+ free_cpumask_var(irq->mask);
+ free_irq(irq->irqn, &irq->nh);
+ kfree(irq);
}
-int mlx5_irq_detach_nb(struct mlx5_irq_table *irq_table, int vecidx,
- struct notifier_block *nb)
+static void irq_put(struct mlx5_irq *irq)
{
- struct mlx5_irq *irq;
+ struct mlx5_irq_pool *pool = irq->pool;
- irq = &irq_table->irq[vecidx];
- return atomic_notifier_chain_unregister(&irq->nh, nb);
+ mutex_lock(&pool->lock);
+ kref_put(&irq->kref, irq_release);
+ mutex_unlock(&pool->lock);
}
-static irqreturn_t mlx5_irq_int_handler(int irq, void *nh)
+static irqreturn_t irq_int_handler(int irq, void *nh)
{
atomic_notifier_call_chain(nh, 0, NULL);
return IRQ_HANDLED;
}
+static void irq_sf_set_name(struct mlx5_irq_pool *pool, char *name, int vecidx)
+{
+ snprintf(name, MLX5_MAX_IRQ_NAME, "%s%d", pool->name, vecidx);
+}
+
static void irq_set_name(char *name, int vecidx)
{
if (vecidx == 0) {
- snprintf(name, MLX5_MAX_IRQ_NAME, "mlx5_async");
+ snprintf(name, MLX5_MAX_IRQ_NAME, "mlx5_async%d", vecidx);
return;
}
vecidx - MLX5_IRQ_VEC_COMP_BASE);
}
-static int request_irqs(struct mlx5_core_dev *dev, int nvec)
+static struct mlx5_irq *irq_request(struct mlx5_irq_pool *pool, int i)
{
+ struct mlx5_core_dev *dev = pool->dev;
char name[MLX5_MAX_IRQ_NAME];
+ struct mlx5_irq *irq;
int err;
- int i;
-
- for (i = 0; i < nvec; i++) {
- struct mlx5_irq *irq = mlx5_irq_get(dev, i);
- int irqn = pci_irq_vector(dev->pdev, i);
+ irq = kzalloc(sizeof(*irq), GFP_KERNEL);
+ if (!irq)
+ return ERR_PTR(-ENOMEM);
+ irq->irqn = pci_irq_vector(dev->pdev, i);
+ if (!pool->name[0])
irq_set_name(name, i);
- ATOMIC_INIT_NOTIFIER_HEAD(&irq->nh);
- snprintf(irq->name, MLX5_MAX_IRQ_NAME,
- "%s@pci:%s", name, pci_name(dev->pdev));
- err = request_irq(irqn, mlx5_irq_int_handler, 0, irq->name,
- &irq->nh);
- if (err) {
- mlx5_core_err(dev, "Failed to request irq\n");
- goto err_request_irq;
- }
+ else
+ irq_sf_set_name(pool, name, i);
+ ATOMIC_INIT_NOTIFIER_HEAD(&irq->nh);
+ snprintf(irq->name, MLX5_MAX_IRQ_NAME,
+ "%s@pci:%s", name, pci_name(dev->pdev));
+ err = request_irq(irq->irqn, irq_int_handler, 0, irq->name,
+ &irq->nh);
+ if (err) {
+ mlx5_core_err(dev, "Failed to request irq. err = %d\n", err);
+ goto err_req_irq;
}
- return 0;
+ if (!zalloc_cpumask_var(&irq->mask, GFP_KERNEL)) {
+ mlx5_core_warn(dev, "zalloc_cpumask_var failed\n");
+ err = -ENOMEM;
+ goto err_cpumask;
+ }
+ kref_init(&irq->kref);
+ irq->index = i;
+ err = xa_err(xa_store(&pool->irqs, irq->index, irq, GFP_KERNEL));
+ if (err) {
+ mlx5_core_err(dev, "Failed to alloc xa entry for irq(%u). err = %d\n",
+ irq->index, err);
+ goto err_xa;
+ }
+ irq->pool = pool;
+ return irq;
+err_xa:
+ free_cpumask_var(irq->mask);
+err_cpumask:
+ free_irq(irq->irqn, &irq->nh);
+err_req_irq:
+ kfree(irq);
+ return ERR_PTR(err);
+}
-err_request_irq:
- while (i--) {
- struct mlx5_irq *irq = mlx5_irq_get(dev, i);
- int irqn = pci_irq_vector(dev->pdev, i);
+int mlx5_irq_attach_nb(struct mlx5_irq *irq, struct notifier_block *nb)
+{
+ int err;
- free_irq(irqn, &irq->nh);
- }
- return err;
+ err = kref_get_unless_zero(&irq->kref);
+ if (WARN_ON_ONCE(!err))
+ /* Something very bad happens here, we are enabling EQ
+ * on non-existing IRQ.
+ */
+ return -ENOENT;
+ err = atomic_notifier_chain_register(&irq->nh, nb);
+ if (err)
+ irq_put(irq);
+ return err;
}
-static void irq_clear_rmap(struct mlx5_core_dev *dev)
+int mlx5_irq_detach_nb(struct mlx5_irq *irq, struct notifier_block *nb)
{
-#ifdef CONFIG_RFS_ACCEL
- struct mlx5_irq_table *irq_table = dev->priv.irq_table;
+ irq_put(irq);
+ return atomic_notifier_chain_unregister(&irq->nh, nb);
+}
- free_irq_cpu_rmap(irq_table->rmap);
-#endif
+struct cpumask *mlx5_irq_get_affinity_mask(struct mlx5_irq *irq)
+{
+ return irq->mask;
}
-static int irq_set_rmap(struct mlx5_core_dev *mdev)
+int mlx5_irq_get_index(struct mlx5_irq *irq)
{
- int err = 0;
-#ifdef CONFIG_RFS_ACCEL
- struct mlx5_irq_table *irq_table = mdev->priv.irq_table;
- int num_affinity_vec;
- int vecidx;
+ return irq->index;
+}
- num_affinity_vec = mlx5_irq_get_num_comp(irq_table);
- irq_table->rmap = alloc_irq_cpu_rmap(num_affinity_vec);
- if (!irq_table->rmap) {
- err = -ENOMEM;
- mlx5_core_err(mdev, "Failed to allocate cpu_rmap. err %d", err);
- goto err_out;
+/* irq_pool API */
+
+/* creating an irq from irq_pool */
+static struct mlx5_irq *irq_pool_create_irq(struct mlx5_irq_pool *pool,
+ struct cpumask *affinity)
+{
+ struct mlx5_irq *irq;
+ u32 irq_index;
+ int err;
+
+ err = xa_alloc(&pool->irqs, &irq_index, NULL, pool->xa_num_irqs,
+ GFP_KERNEL);
+ if (err)
+ return ERR_PTR(err);
+ irq = irq_request(pool, irq_index);
+ if (IS_ERR(irq))
+ return irq;
+ cpumask_copy(irq->mask, affinity);
+ irq_set_affinity_hint(irq->irqn, irq->mask);
+ return irq;
+}
+
+/* looking for the irq with the smallest refcount and the same affinity */
+static struct mlx5_irq *irq_pool_find_least_loaded(struct mlx5_irq_pool *pool,
+ struct cpumask *affinity)
+{
+ int start = pool->xa_num_irqs.min;
+ int end = pool->xa_num_irqs.max;
+ struct mlx5_irq *irq = NULL;
+ struct mlx5_irq *iter;
+ unsigned long index;
+
+ lockdep_assert_held(&pool->lock);
+ xa_for_each_range(&pool->irqs, index, iter, start, end) {
+ if (!cpumask_equal(iter->mask, affinity))
+ continue;
+ if (kref_read(&iter->kref) < pool->min_threshold)
+ return iter;
+ if (!irq || kref_read(&iter->kref) <
+ kref_read(&irq->kref))
+ irq = iter;
}
+ return irq;
+}
- vecidx = MLX5_IRQ_VEC_COMP_BASE;
- for (; vecidx < irq_table->nvec; vecidx++) {
- err = irq_cpu_rmap_add(irq_table->rmap,
- pci_irq_vector(mdev->pdev, vecidx));
- if (err) {
- mlx5_core_err(mdev, "irq_cpu_rmap_add failed. err %d",
- err);
- goto err_irq_cpu_rmap_add;
+/* requesting an irq from a given pool according to given affinity */
+static struct mlx5_irq *irq_pool_request_affinity(struct mlx5_irq_pool *pool,
+ struct cpumask *affinity)
+{
+ struct mlx5_irq *least_loaded_irq, *new_irq;
+
+ mutex_lock(&pool->lock);
+ least_loaded_irq = irq_pool_find_least_loaded(pool, affinity);
+ if (least_loaded_irq &&
+ kref_read(&least_loaded_irq->kref) < pool->min_threshold)
+ goto out;
+ new_irq = irq_pool_create_irq(pool, affinity);
+ if (IS_ERR(new_irq)) {
+ if (!least_loaded_irq) {
+ mlx5_core_err(pool->dev, "Didn't find IRQ for cpu = %u\n",
+ cpumask_first(affinity));
+ mutex_unlock(&pool->lock);
+ return new_irq;
}
+ /* We failed to create a new IRQ for the requested affinity,
+ * sharing existing IRQ.
+ */
+ goto out;
}
- return 0;
+ least_loaded_irq = new_irq;
+ goto unlock;
+out:
+ kref_get(&least_loaded_irq->kref);
+ if (kref_read(&least_loaded_irq->kref) > pool->max_threshold)
+ mlx5_core_dbg(pool->dev, "IRQ %u overloaded, pool_name: %s, %u EQs on this irq\n",
+ least_loaded_irq->irqn, pool->name,
+ kref_read(&least_loaded_irq->kref) / MLX5_EQ_REFS_PER_IRQ);
+unlock:
+ mutex_unlock(&pool->lock);
+ return least_loaded_irq;
+}
-err_irq_cpu_rmap_add:
- irq_clear_rmap(mdev);
-err_out:
-#endif
- return err;
+/* requesting an irq from a given pool according to given index */
+static struct mlx5_irq *
+irq_pool_request_vector(struct mlx5_irq_pool *pool, int vecidx,
+ struct cpumask *affinity)
+{
+ struct mlx5_irq *irq;
+
+ mutex_lock(&pool->lock);
+ irq = xa_load(&pool->irqs, vecidx);
+ if (irq) {
+ kref_get(&irq->kref);
+ goto unlock;
+ }
+ irq = irq_request(pool, vecidx);
+ if (IS_ERR(irq) || !affinity)
+ goto unlock;
+ cpumask_copy(irq->mask, affinity);
+ irq_set_affinity_hint(irq->irqn, irq->mask);
+unlock:
+ mutex_unlock(&pool->lock);
+ return irq;
}
-/* Completion IRQ vectors */
+static struct mlx5_irq_pool *find_sf_irq_pool(struct mlx5_irq_table *irq_table,
+ int i, struct cpumask *affinity)
+{
+ if (cpumask_empty(affinity) && i == MLX5_IRQ_EQ_CTRL)
+ return irq_table->sf_ctrl_pool;
+ return irq_table->sf_comp_pool;
+}
-static int set_comp_irq_affinity_hint(struct mlx5_core_dev *mdev, int i)
+/**
+ * mlx5_irq_release - release an IRQ back to the system.
+ * @irq: irq to be released.
+ */
+void mlx5_irq_release(struct mlx5_irq *irq)
{
- int vecidx = MLX5_IRQ_VEC_COMP_BASE + i;
+ synchronize_irq(irq->irqn);
+ irq_put(irq);
+}
+
+/**
+ * mlx5_irq_request - request an IRQ for mlx5 device.
+ * @dev: mlx5 device that requesting the IRQ.
+ * @vecidx: vector index of the IRQ. This argument is ignore if affinity is
+ * provided.
+ * @affinity: cpumask requested for this IRQ.
+ *
+ * This function returns a pointer to IRQ, or ERR_PTR in case of error.
+ */
+struct mlx5_irq *mlx5_irq_request(struct mlx5_core_dev *dev, u16 vecidx,
+ struct cpumask *affinity)
+{
+ struct mlx5_irq_table *irq_table = mlx5_irq_table_get(dev);
+ struct mlx5_irq_pool *pool;
struct mlx5_irq *irq;
- int irqn;
- irq = mlx5_irq_get(mdev, vecidx);
- irqn = pci_irq_vector(mdev->pdev, vecidx);
- if (!zalloc_cpumask_var(&irq->mask, GFP_KERNEL)) {
- mlx5_core_warn(mdev, "zalloc_cpumask_var failed");
- return -ENOMEM;
+ if (mlx5_core_is_sf(dev)) {
+ pool = find_sf_irq_pool(irq_table, vecidx, affinity);
+ if (!pool)
+ /* we don't have IRQs for SFs, using the PF IRQs */
+ goto pf_irq;
+ if (cpumask_empty(affinity) && !strcmp(pool->name, "mlx5_sf_comp"))
+ /* In case an SF user request IRQ with vecidx */
+ irq = irq_pool_request_vector(pool, vecidx, NULL);
+ else
+ irq = irq_pool_request_affinity(pool, affinity);
+ goto out;
}
+pf_irq:
+ pool = irq_table->pf_pool;
+ irq = irq_pool_request_vector(pool, vecidx, affinity);
+out:
+ if (IS_ERR(irq))
+ return irq;
+ mlx5_core_dbg(dev, "irq %u mapped to cpu %*pbl, %u EQs on this irq\n",
+ irq->irqn, cpumask_pr_args(affinity),
+ kref_read(&irq->kref) / MLX5_EQ_REFS_PER_IRQ);
+ return irq;
+}
- cpumask_set_cpu(cpumask_local_spread(i, mdev->priv.numa_node),
- irq->mask);
- if (IS_ENABLED(CONFIG_SMP) &&
- irq_set_affinity_hint(irqn, irq->mask))
- mlx5_core_warn(mdev, "irq_set_affinity_hint failed, irq 0x%.4x",
- irqn);
-
- return 0;
+static struct mlx5_irq_pool *
+irq_pool_alloc(struct mlx5_core_dev *dev, int start, int size, char *name,
+ u32 min_threshold, u32 max_threshold)
+{
+ struct mlx5_irq_pool *pool = kvzalloc(sizeof(*pool), GFP_KERNEL);
+
+ if (!pool)
+ return ERR_PTR(-ENOMEM);
+ pool->dev = dev;
+ xa_init_flags(&pool->irqs, XA_FLAGS_ALLOC);
+ pool->xa_num_irqs.min = start;
+ pool->xa_num_irqs.max = start + size - 1;
+ if (name)
+ snprintf(pool->name, MLX5_MAX_IRQ_NAME - MLX5_MAX_IRQ_IDX_CHARS,
+ name);
+ pool->min_threshold = min_threshold * MLX5_EQ_REFS_PER_IRQ;
+ pool->max_threshold = max_threshold * MLX5_EQ_REFS_PER_IRQ;
+ mutex_init(&pool->lock);
+ mlx5_core_dbg(dev, "pool->name = %s, pool->size = %d, pool->start = %d",
+ name, size, start);
+ return pool;
}
-static void clear_comp_irq_affinity_hint(struct mlx5_core_dev *mdev, int i)
+static void irq_pool_free(struct mlx5_irq_pool *pool)
{
- int vecidx = MLX5_IRQ_VEC_COMP_BASE + i;
struct mlx5_irq *irq;
- int irqn;
+ unsigned long index;
- irq = mlx5_irq_get(mdev, vecidx);
- irqn = pci_irq_vector(mdev->pdev, vecidx);
- irq_set_affinity_hint(irqn, NULL);
- free_cpumask_var(irq->mask);
+ xa_for_each(&pool->irqs, index, irq)
+ irq_release(&irq->kref);
+ xa_destroy(&pool->irqs);
+ kvfree(pool);
}
-static int set_comp_irq_affinity_hints(struct mlx5_core_dev *mdev)
+static int irq_pools_init(struct mlx5_core_dev *dev, int sf_vec, int pf_vec)
{
- int nvec = mlx5_irq_get_num_comp(mdev->priv.irq_table);
+ struct mlx5_irq_table *table = dev->priv.irq_table;
+ int num_sf_ctrl_by_msix;
+ int num_sf_ctrl_by_sfs;
+ int num_sf_ctrl;
int err;
- int i;
- for (i = 0; i < nvec; i++) {
- err = set_comp_irq_affinity_hint(mdev, i);
- if (err)
- goto err_out;
+ /* init pf_pool */
+ table->pf_pool = irq_pool_alloc(dev, 0, pf_vec, NULL,
+ MLX5_EQ_SHARE_IRQ_MIN_COMP,
+ MLX5_EQ_SHARE_IRQ_MAX_COMP);
+ if (IS_ERR(table->pf_pool))
+ return PTR_ERR(table->pf_pool);
+ if (!mlx5_sf_max_functions(dev))
+ return 0;
+ if (sf_vec < MLX5_IRQ_VEC_COMP_BASE_SF) {
+ mlx5_core_err(dev, "Not enough IRQs for SFs. SF may run at lower performance\n");
+ return 0;
}
+ /* init sf_ctrl_pool */
+ num_sf_ctrl_by_msix = DIV_ROUND_UP(sf_vec, MLX5_COMP_EQS_PER_SF);
+ num_sf_ctrl_by_sfs = DIV_ROUND_UP(mlx5_sf_max_functions(dev),
+ MLX5_SFS_PER_CTRL_IRQ);
+ num_sf_ctrl = min_t(int, num_sf_ctrl_by_msix, num_sf_ctrl_by_sfs);
+ num_sf_ctrl = min_t(int, MLX5_IRQ_CTRL_SF_MAX, num_sf_ctrl);
+ table->sf_ctrl_pool = irq_pool_alloc(dev, pf_vec, num_sf_ctrl,
+ "mlx5_sf_ctrl",
+ MLX5_EQ_SHARE_IRQ_MIN_CTRL,
+ MLX5_EQ_SHARE_IRQ_MAX_CTRL);
+ if (IS_ERR(table->sf_ctrl_pool)) {
+ err = PTR_ERR(table->sf_ctrl_pool);
+ goto err_pf;
+ }
+ /* init sf_comp_pool */
+ table->sf_comp_pool = irq_pool_alloc(dev, pf_vec + num_sf_ctrl,
+ sf_vec - num_sf_ctrl, "mlx5_sf_comp",
+ MLX5_EQ_SHARE_IRQ_MIN_COMP,
+ MLX5_EQ_SHARE_IRQ_MAX_COMP);
+ if (IS_ERR(table->sf_comp_pool)) {
+ err = PTR_ERR(table->sf_comp_pool);
+ goto err_sf_ctrl;
+ }
return 0;
-
-err_out:
- for (i--; i >= 0; i--)
- clear_comp_irq_affinity_hint(mdev, i);
-
+err_sf_ctrl:
+ irq_pool_free(table->sf_ctrl_pool);
+err_pf:
+ irq_pool_free(table->pf_pool);
return err;
}
-static void clear_comp_irqs_affinity_hints(struct mlx5_core_dev *mdev)
+static void irq_pools_destroy(struct mlx5_irq_table *table)
{
- int nvec = mlx5_irq_get_num_comp(mdev->priv.irq_table);
- int i;
-
- for (i = 0; i < nvec; i++)
- clear_comp_irq_affinity_hint(mdev, i);
+ if (table->sf_ctrl_pool) {
+ irq_pool_free(table->sf_comp_pool);
+ irq_pool_free(table->sf_ctrl_pool);
+ }
+ irq_pool_free(table->pf_pool);
}
-struct cpumask *
-mlx5_irq_get_affinity_mask(struct mlx5_irq_table *irq_table, int vecidx)
+/* irq_table API */
+
+int mlx5_irq_table_init(struct mlx5_core_dev *dev)
{
- return irq_table->irq[vecidx].mask;
+ struct mlx5_irq_table *irq_table;
+
+ if (mlx5_core_is_sf(dev))
+ return 0;
+
+ irq_table = kvzalloc(sizeof(*irq_table), GFP_KERNEL);
+ if (!irq_table)
+ return -ENOMEM;
+
+ dev->priv.irq_table = irq_table;
+ return 0;
}
-#ifdef CONFIG_RFS_ACCEL
-struct cpu_rmap *mlx5_irq_get_rmap(struct mlx5_irq_table *irq_table)
+void mlx5_irq_table_cleanup(struct mlx5_core_dev *dev)
{
- return irq_table->rmap;
+ if (mlx5_core_is_sf(dev))
+ return;
+
+ kvfree(dev->priv.irq_table);
}
-#endif
-static void unrequest_irqs(struct mlx5_core_dev *dev)
+int mlx5_irq_table_get_num_comp(struct mlx5_irq_table *table)
{
- struct mlx5_irq_table *table = dev->priv.irq_table;
- int i;
-
- for (i = 0; i < table->nvec; i++)
- free_irq(pci_irq_vector(dev->pdev, i),
- &mlx5_irq_get(dev, i)->nh);
+ return table->pf_pool->xa_num_irqs.max - table->pf_pool->xa_num_irqs.min;
}
int mlx5_irq_table_create(struct mlx5_core_dev *dev)
{
- struct mlx5_priv *priv = &dev->priv;
- struct mlx5_irq_table *table = priv->irq_table;
int num_eqs = MLX5_CAP_GEN(dev, max_num_eqs) ?
MLX5_CAP_GEN(dev, max_num_eqs) :
1 << MLX5_CAP_GEN(dev, log_max_eq);
- int nvec;
+ int total_vec;
+ int pf_vec;
int err;
if (mlx5_core_is_sf(dev))
return 0;
- nvec = MLX5_CAP_GEN(dev, num_ports) * num_online_cpus() +
- MLX5_IRQ_VEC_COMP_BASE;
- nvec = min_t(int, nvec, num_eqs);
- if (nvec <= MLX5_IRQ_VEC_COMP_BASE)
- return -ENOMEM;
-
- table->irq = kcalloc(nvec, sizeof(*table->irq), GFP_KERNEL);
- if (!table->irq)
+ pf_vec = MLX5_CAP_GEN(dev, num_ports) * num_online_cpus() +
+ MLX5_IRQ_VEC_COMP_BASE;
+ pf_vec = min_t(int, pf_vec, num_eqs);
+ if (pf_vec <= MLX5_IRQ_VEC_COMP_BASE)
return -ENOMEM;
- nvec = pci_alloc_irq_vectors(dev->pdev, MLX5_IRQ_VEC_COMP_BASE + 1,
- nvec, PCI_IRQ_MSIX);
- if (nvec < 0) {
- err = nvec;
- goto err_free_irq;
- }
-
- table->nvec = nvec;
+ total_vec = pf_vec;
+ if (mlx5_sf_max_functions(dev))
+ total_vec += MLX5_IRQ_CTRL_SF_MAX +
+ MLX5_COMP_EQS_PER_SF * mlx5_sf_max_functions(dev);
- err = irq_set_rmap(dev);
- if (err)
- goto err_set_rmap;
+ total_vec = pci_alloc_irq_vectors(dev->pdev, MLX5_IRQ_VEC_COMP_BASE + 1,
+ total_vec, PCI_IRQ_MSIX);
+ if (total_vec < 0)
+ return total_vec;
+ pf_vec = min(pf_vec, total_vec);
- err = request_irqs(dev, nvec);
+ err = irq_pools_init(dev, total_vec - pf_vec, pf_vec);
if (err)
- goto err_request_irqs;
-
- err = set_comp_irq_affinity_hints(dev);
- if (err) {
- mlx5_core_err(dev, "Failed to alloc affinity hint cpumask\n");
- goto err_set_affinity;
- }
-
- return 0;
+ pci_free_irq_vectors(dev->pdev);
-err_set_affinity:
- unrequest_irqs(dev);
-err_request_irqs:
- irq_clear_rmap(dev);
-err_set_rmap:
- pci_free_irq_vectors(dev->pdev);
-err_free_irq:
- kfree(table->irq);
return err;
}
void mlx5_irq_table_destroy(struct mlx5_core_dev *dev)
{
struct mlx5_irq_table *table = dev->priv.irq_table;
- int i;
if (mlx5_core_is_sf(dev))
return;
- /* free_irq requires that affinity and rmap will be cleared
- * before calling it. This is why there is asymmetry with set_rmap
- * which should be called after alloc_irq but before request_irq.
+ /* There are cases where IRQs still will be in used when we reaching
+ * to here. Hence, making sure all the irqs are realeased.
*/
- irq_clear_rmap(dev);
- clear_comp_irqs_affinity_hints(dev);
- for (i = 0; i < table->nvec; i++)
- free_irq(pci_irq_vector(dev->pdev, i),
- &mlx5_irq_get(dev, i)->nh);
+ irq_pools_destroy(table);
pci_free_irq_vectors(dev->pdev);
- kfree(table->irq);
+}
+
+int mlx5_irq_table_get_sfs_vec(struct mlx5_irq_table *table)
+{
+ if (table->sf_comp_pool)
+ return table->sf_comp_pool->xa_num_irqs.max -
+ table->sf_comp_pool->xa_num_irqs.min + 1;
+ else
+ return mlx5_irq_table_get_num_comp(table);
}
struct mlx5_irq_table *mlx5_irq_table_get(struct mlx5_core_dev *dev)
{
int err;
+ if (!MLX5_CAP_GEN(dev, roce))
+ return;
+
err = mlx5_nic_vport_enable_roce(dev);
if (err) {
mlx5_core_err(dev, "Failed to enable RoCE: %d\n", err);
sf_index = event->function_id - base_id;
sf_dev = xa_load(&table->devices, sf_index);
switch (event->new_vhca_state) {
+ case MLX5_VHCA_STATE_INVALID:
case MLX5_VHCA_STATE_ALLOCATED:
if (sf_dev)
mlx5_sf_dev_del(table->dev, sf_dev, sf_index);
switch (hw_state) {
case MLX5_VHCA_STATE_ACTIVE:
case MLX5_VHCA_STATE_IN_USE:
- case MLX5_VHCA_STATE_TEARDOWN_REQUEST:
return DEVLINK_PORT_FN_STATE_ACTIVE;
case MLX5_VHCA_STATE_INVALID:
case MLX5_VHCA_STATE_ALLOCATED:
+ case MLX5_VHCA_STATE_TEARDOWN_REQUEST:
default:
return DEVLINK_PORT_FN_STATE_INACTIVE;
}
return err;
}
-static int mlx5_sf_activate(struct mlx5_core_dev *dev, struct mlx5_sf *sf)
+static int mlx5_sf_activate(struct mlx5_core_dev *dev, struct mlx5_sf *sf,
+ struct netlink_ext_ack *extack)
{
int err;
if (mlx5_sf_is_active(sf))
return 0;
- if (sf->hw_state != MLX5_VHCA_STATE_ALLOCATED)
- return -EINVAL;
+ if (sf->hw_state != MLX5_VHCA_STATE_ALLOCATED) {
+ NL_SET_ERR_MSG_MOD(extack, "SF is inactivated but it is still attached");
+ return -EBUSY;
+ }
err = mlx5_cmd_sf_enable_hca(dev, sf->hw_fn_id);
if (err)
static int mlx5_sf_state_set(struct mlx5_core_dev *dev, struct mlx5_sf_table *table,
struct mlx5_sf *sf,
- enum devlink_port_fn_state state)
+ enum devlink_port_fn_state state,
+ struct netlink_ext_ack *extack)
{
int err = 0;
if (state == mlx5_sf_to_devlink_state(sf->hw_state))
goto out;
if (state == DEVLINK_PORT_FN_STATE_ACTIVE)
- err = mlx5_sf_activate(dev, sf);
+ err = mlx5_sf_activate(dev, sf, extack);
else if (state == DEVLINK_PORT_FN_STATE_INACTIVE)
err = mlx5_sf_deactivate(dev, sf);
else
goto out;
}
- err = mlx5_sf_state_set(dev, table, sf, state);
+ err = mlx5_sf_state_set(dev, table, sf, state, extack);
out:
mlx5_sf_table_put(table);
return err;
#include "sf.h"
#include "mlx5_ifc_vhca_event.h"
#include "ecpf.h"
-#include "vhca_event.h"
#include "mlx5_core.h"
#include "eswitch.h"
#define __MLX5_SF_H__
#include <linux/mlx5/driver.h>
-
-static inline u16 mlx5_sf_start_function_id(const struct mlx5_core_dev *dev)
-{
- return MLX5_CAP_GEN(dev, sf_base_id);
-}
-
-#ifdef CONFIG_MLX5_SF
-
-static inline bool mlx5_sf_supported(const struct mlx5_core_dev *dev)
-{
- return MLX5_CAP_GEN(dev, sf);
-}
-
-static inline u16 mlx5_sf_max_functions(const struct mlx5_core_dev *dev)
-{
- if (!mlx5_sf_supported(dev))
- return 0;
- if (MLX5_CAP_GEN(dev, max_num_sf))
- return MLX5_CAP_GEN(dev, max_num_sf);
- else
- return 1 << MLX5_CAP_GEN(dev, log_max_sf);
-}
-
-#else
-
-static inline bool mlx5_sf_supported(const struct mlx5_core_dev *dev)
-{
- return false;
-}
-
-static inline u16 mlx5_sf_max_functions(const struct mlx5_core_dev *dev)
-{
- return 0;
-}
-
-#endif
+#include "lib/sf.h"
#ifdef CONFIG_MLX5_SF_MANAGER
#include <linux/mlx5/driver.h>
#include <linux/mlx5/vport.h>
#include "mlx5_core.h"
+#include "mlx5_irq.h"
#include "eswitch.h"
static int sriov_restore_guids(struct mlx5_core_dev *dev, int vf)
/* Copyright (c) 2019 Mellanox Technologies. */
#include "dr_types.h"
+#include "dr_ste.h"
enum dr_action_domain {
DR_ACTION_DOMAIN_NIC_INGRESS,
enum dr_action_valid_state {
DR_ACTION_STATE_ERR,
DR_ACTION_STATE_NO_ACTION,
- DR_ACTION_STATE_REFORMAT,
+ DR_ACTION_STATE_ENCAP,
+ DR_ACTION_STATE_DECAP,
DR_ACTION_STATE_MODIFY_HDR,
DR_ACTION_STATE_MODIFY_VLAN,
DR_ACTION_STATE_NON_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_TAG] = DR_ACTION_STATE_NON_TERM,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
- [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_REFORMAT,
- [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_DECAP,
+ [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_DECAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
[DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
[DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
},
- [DR_ACTION_STATE_REFORMAT] = {
+ [DR_ACTION_STATE_DECAP] = {
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_QP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
- [DR_ACTION_TYP_TAG] = DR_ACTION_STATE_REFORMAT,
- [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_TAG] = DR_ACTION_STATE_DECAP,
+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_DECAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
[DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
[DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
},
+ [DR_ACTION_STATE_ENCAP] = {
+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
+ [DR_ACTION_TYP_QP] = DR_ACTION_STATE_TERM,
+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
+ [DR_ACTION_TYP_TAG] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_ENCAP,
+ },
[DR_ACTION_STATE_MODIFY_HDR] = {
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_QP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_TAG] = DR_ACTION_STATE_MODIFY_HDR,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_HDR,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
},
[DR_ACTION_STATE_MODIFY_VLAN] = {
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_VLAN,
[DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
[DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
},
[DR_ACTION_STATE_NON_TERM] = {
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_TAG] = DR_ACTION_STATE_NON_TERM,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
- [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_REFORMAT,
- [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_DECAP,
+ [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_DECAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
[DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
[DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
},
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
- [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
- [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
[DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
[DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
},
- [DR_ACTION_STATE_REFORMAT] = {
+ [DR_ACTION_STATE_ENCAP] = {
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
- [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_ENCAP,
},
[DR_ACTION_STATE_MODIFY_HDR] = {
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_HDR,
- [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
- [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
[DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
},
[DR_ACTION_STATE_MODIFY_VLAN] = {
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_VLAN,
[DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
- [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
- [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
},
[DR_ACTION_STATE_NON_TERM] = {
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
- [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
- [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
[DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
[DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
},
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
- [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_REFORMAT,
- [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_DECAP,
+ [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_DECAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
[DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
[DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
[DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
},
- [DR_ACTION_STATE_REFORMAT] = {
+ [DR_ACTION_STATE_DECAP] = {
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
- [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_DECAP,
[DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
[DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
[DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
+ },
+ [DR_ACTION_STATE_ENCAP] = {
+ [DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
+ [DR_ACTION_TYP_QP] = DR_ACTION_STATE_TERM,
+ [DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
+ [DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_ENCAP,
},
[DR_ACTION_STATE_MODIFY_HDR] = {
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_HDR,
[DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
},
[DR_ACTION_STATE_MODIFY_VLAN] = {
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_VLAN,
[DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
},
[DR_ACTION_STATE_NON_TERM] = {
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
- [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_REFORMAT,
- [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_TNL_L2_TO_L2] = DR_ACTION_STATE_DECAP,
+ [DR_ACTION_TYP_TNL_L3_TO_L2] = DR_ACTION_STATE_DECAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
[DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
[DR_ACTION_TYP_POP_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
[DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
[DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
- [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
- [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
[DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
[DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
},
- [DR_ACTION_STATE_REFORMAT] = {
+ [DR_ACTION_STATE_ENCAP] = {
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
- [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_CTR] = DR_ACTION_STATE_ENCAP,
[DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
},
[DR_ACTION_STATE_MODIFY_HDR] = {
[DR_ACTION_TYP_DROP] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_HDR,
- [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
- [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
[DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
[DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
},
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_MODIFY_VLAN,
- [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
- [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
[DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
},
[DR_ACTION_STATE_NON_TERM] = {
[DR_ACTION_TYP_FT] = DR_ACTION_STATE_TERM,
[DR_ACTION_TYP_CTR] = DR_ACTION_STATE_NON_TERM,
[DR_ACTION_TYP_MODIFY_HDR] = DR_ACTION_STATE_MODIFY_HDR,
- [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_REFORMAT,
- [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_REFORMAT,
+ [DR_ACTION_TYP_L2_TO_TNL_L2] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_L2_TO_TNL_L3] = DR_ACTION_STATE_ENCAP,
+ [DR_ACTION_TYP_INSERT_HDR] = DR_ACTION_STATE_ENCAP,
[DR_ACTION_TYP_PUSH_VLAN] = DR_ACTION_STATE_MODIFY_VLAN,
[DR_ACTION_TYP_VPORT] = DR_ACTION_STATE_TERM,
},
case DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L3:
*action_type = DR_ACTION_TYP_L2_TO_TNL_L3;
break;
+ case DR_ACTION_REFORMAT_TYP_INSERT_HDR:
+ *action_type = DR_ACTION_TYP_INSERT_HDR;
+ break;
default:
return -EINVAL;
}
break;
case DR_ACTION_TYP_L2_TO_TNL_L2:
case DR_ACTION_TYP_L2_TO_TNL_L3:
- attr.reformat_size = action->reformat->reformat_size;
- attr.reformat_id = action->reformat->reformat_id;
+ if (rx_rule &&
+ !(dmn->ste_ctx->actions_caps & DR_STE_CTX_ACTION_CAP_RX_ENCAP)) {
+ mlx5dr_info(dmn, "Device doesn't support Encap on RX\n");
+ goto out_invalid_arg;
+ }
+ attr.reformat.size = action->reformat->size;
+ attr.reformat.id = action->reformat->id;
break;
case DR_ACTION_TYP_VPORT:
attr.hit_gvmi = action->vport->caps->vhca_gvmi;
attr.vlans.headers[attr.vlans.count++] = action->push_vlan->vlan_hdr;
break;
+ case DR_ACTION_TYP_INSERT_HDR:
+ attr.reformat.size = action->reformat->size;
+ attr.reformat.id = action->reformat->id;
+ attr.reformat.param_0 = action->reformat->param_0;
+ attr.reformat.param_1 = action->reformat->param_1;
+ break;
default:
goto out_invalid_arg;
}
[DR_ACTION_TYP_MODIFY_HDR] = sizeof(struct mlx5dr_action_rewrite),
[DR_ACTION_TYP_VPORT] = sizeof(struct mlx5dr_action_vport),
[DR_ACTION_TYP_PUSH_VLAN] = sizeof(struct mlx5dr_action_push_vlan),
+ [DR_ACTION_TYP_INSERT_HDR] = sizeof(struct mlx5dr_action_reformat),
};
static struct mlx5dr_action *
if (reformat_action) {
reformat_req = true;
hw_dests[i].vport.reformat_id =
- reformat_action->reformat->reformat_id;
+ reformat_action->reformat->id;
ref_actions[num_of_ref++] = reformat_action;
hw_dests[i].vport.flags |= MLX5_FLOW_DEST_VPORT_REFORMAT_ID;
}
static int
dr_action_verify_reformat_params(enum mlx5dr_action_type reformat_type,
struct mlx5dr_domain *dmn,
+ u8 reformat_param_0,
+ u8 reformat_param_1,
size_t data_sz,
void *data)
{
- if ((!data && data_sz) || (data && !data_sz) || reformat_type >
- DR_ACTION_TYP_L2_TO_TNL_L3) {
+ if ((!data && data_sz) || (data && !data_sz) ||
+ ((reformat_param_0 || reformat_param_1) &&
+ reformat_type != DR_ACTION_TYP_INSERT_HDR) ||
+ reformat_type > DR_ACTION_TYP_INSERT_HDR) {
mlx5dr_dbg(dmn, "Invalid reformat parameter!\n");
goto out_err;
}
static int
dr_action_create_reformat_action(struct mlx5dr_domain *dmn,
+ u8 reformat_param_0, u8 reformat_param_1,
size_t data_sz, void *data,
struct mlx5dr_action *action)
{
else
rt = MLX5_REFORMAT_TYPE_L2_TO_L3_TUNNEL;
- ret = mlx5dr_cmd_create_reformat_ctx(dmn->mdev, rt, data_sz, data,
+ ret = mlx5dr_cmd_create_reformat_ctx(dmn->mdev, rt, 0, 0,
+ data_sz, data,
&reformat_id);
if (ret)
return ret;
- action->reformat->reformat_id = reformat_id;
- action->reformat->reformat_size = data_sz;
+ action->reformat->id = reformat_id;
+ action->reformat->size = data_sz;
return 0;
}
case DR_ACTION_TYP_TNL_L2_TO_L2:
}
return 0;
}
+ case DR_ACTION_TYP_INSERT_HDR:
+ {
+ ret = mlx5dr_cmd_create_reformat_ctx(dmn->mdev,
+ MLX5_REFORMAT_TYPE_INSERT_HDR,
+ reformat_param_0,
+ reformat_param_1,
+ data_sz, data,
+ &reformat_id);
+ if (ret)
+ return ret;
+
+ action->reformat->id = reformat_id;
+ action->reformat->size = data_sz;
+ action->reformat->param_0 = reformat_param_0;
+ action->reformat->param_1 = reformat_param_1;
+ return 0;
+ }
default:
mlx5dr_info(dmn, "Reformat type is not supported %d\n", action->action_type);
return -EINVAL;
struct mlx5dr_action *
mlx5dr_action_create_packet_reformat(struct mlx5dr_domain *dmn,
enum mlx5dr_action_reformat_type reformat_type,
+ u8 reformat_param_0,
+ u8 reformat_param_1,
size_t data_sz,
void *data)
{
goto dec_ref;
}
- ret = dr_action_verify_reformat_params(action_type, dmn, data_sz, data);
+ ret = dr_action_verify_reformat_params(action_type, dmn,
+ reformat_param_0, reformat_param_1,
+ data_sz, data);
if (ret)
goto dec_ref;
action->reformat->dmn = dmn;
ret = dr_action_create_reformat_action(dmn,
+ reformat_param_0,
+ reformat_param_1,
data_sz,
data,
action);
break;
case DR_ACTION_TYP_L2_TO_TNL_L2:
case DR_ACTION_TYP_L2_TO_TNL_L3:
+ case DR_ACTION_TYP_INSERT_HDR:
mlx5dr_cmd_destroy_reformat_ctx((action->reformat->dmn)->mdev,
- action->reformat->reformat_id);
+ action->reformat->id);
refcount_dec(&action->reformat->dmn->refcount);
break;
case DR_ACTION_TYP_MODIFY_HDR:
int mlx5dr_cmd_create_reformat_ctx(struct mlx5_core_dev *mdev,
enum mlx5_reformat_ctx_type rt,
+ u8 reformat_param_0,
+ u8 reformat_param_1,
size_t reformat_size,
void *reformat_data,
u32 *reformat_id)
pdata = MLX5_ADDR_OF(packet_reformat_context_in, prctx, reformat_data);
MLX5_SET(packet_reformat_context_in, prctx, reformat_type, rt);
+ MLX5_SET(packet_reformat_context_in, prctx, reformat_param_0, reformat_param_0);
+ MLX5_SET(packet_reformat_context_in, prctx, reformat_param_1, reformat_param_1);
MLX5_SET(packet_reformat_context_in, prctx, reformat_data_size, reformat_size);
- memcpy(pdata, reformat_data, reformat_size);
+ if (reformat_data && reformat_size)
+ memcpy(pdata, reformat_data, reformat_size);
err = mlx5_cmd_exec(mdev, in, inlen, out, sizeof(out));
if (err)
int ret;
ft_attr.table_type = MLX5_FLOW_TABLE_TYPE_FDB;
- ft_attr.level = dmn->info.caps.max_ft_level - 2;
+ ft_attr.level = min_t(int, dmn->info.caps.max_ft_level - 2,
+ MLX5_FT_MAX_MULTIPATH_LEVEL);
ft_attr.reformat_en = reformat_req;
ft_attr.decap_en = reformat_req;
u16 (*get_byte_mask)(u8 *hw_ste_p);
/* Actions */
+ u32 actions_caps;
void (*set_actions_rx)(struct mlx5dr_domain *dmn,
u8 *action_type_set,
u8 *hw_ste_arr,
attr->gvmi);
dr_ste_v0_set_tx_encap(last_ste,
- attr->reformat_id,
- attr->reformat_size,
+ attr->reformat.id,
+ attr->reformat.size,
action_type_set[DR_ACTION_TYP_L2_TO_TNL_L3]);
/* Whenever prio_tag_required enabled, we can be sure that the
* previous table (ACL) already push vlan to our packet,
.get_byte_mask = &dr_ste_v0_get_byte_mask,
/* Actions */
+ .actions_caps = DR_STE_CTX_ACTION_CAP_NONE,
.set_actions_rx = &dr_ste_v0_set_actions_rx,
.set_actions_tx = &dr_ste_v0_set_actions_tx,
.modify_field_arr_sz = ARRAY_SIZE(dr_ste_v0_action_modify_field_arr),
DR_STE_V1_ACTION_MDFY_FLD_IPV6_SRC_OUT_3 = 0x4f,
DR_STE_V1_ACTION_MDFY_FLD_TCP_MISC_0 = 0x5e,
DR_STE_V1_ACTION_MDFY_FLD_TCP_MISC_1 = 0x5f,
+ DR_STE_V1_ACTION_MDFY_FLD_CFG_HDR_0_0 = 0x6f,
+ DR_STE_V1_ACTION_MDFY_FLD_CFG_HDR_0_1 = 0x70,
DR_STE_V1_ACTION_MDFY_FLD_METADATA_2_CQE = 0x7b,
DR_STE_V1_ACTION_MDFY_FLD_GNRL_PURPOSE = 0x7c,
DR_STE_V1_ACTION_MDFY_FLD_REGISTER_2 = 0x8c,
[MLX5_ACTION_IN_FIELD_OUT_FIRST_VID] = {
.hw_field = DR_STE_V1_ACTION_MDFY_FLD_L2_OUT_2, .start = 0, .end = 15,
},
+ [MLX5_ACTION_IN_FIELD_OUT_EMD_31_0] = {
+ .hw_field = DR_STE_V1_ACTION_MDFY_FLD_CFG_HDR_0_1, .start = 0, .end = 31,
+ },
+ [MLX5_ACTION_IN_FIELD_OUT_EMD_47_32] = {
+ .hw_field = DR_STE_V1_ACTION_MDFY_FLD_CFG_HDR_0_0, .start = 0, .end = 15,
+ },
};
static void dr_ste_v1_set_entry_type(u8 *hw_ste_p, u8 entry_type)
MLX5_SET(ste_match_bwc_v1, hw_ste_p, reparse, 1);
}
-static void dr_ste_v1_set_tx_encap(u8 *hw_ste_p, u8 *d_action,
- u32 reformat_id, int size)
+static void dr_ste_v1_set_encap(u8 *hw_ste_p, u8 *d_action,
+ u32 reformat_id, int size)
{
MLX5_SET(ste_double_action_insert_with_ptr_v1, d_action, action_id,
DR_STE_V1_ACTION_ID_INSERT_POINTER);
dr_ste_v1_set_reparse(hw_ste_p);
}
+static void dr_ste_v1_set_insert_hdr(u8 *hw_ste_p, u8 *d_action,
+ u32 reformat_id,
+ u8 anchor, u8 offset,
+ int size)
+{
+ MLX5_SET(ste_double_action_insert_with_ptr_v1, d_action,
+ action_id, DR_STE_V1_ACTION_ID_INSERT_POINTER);
+ MLX5_SET(ste_double_action_insert_with_ptr_v1, d_action, start_anchor, anchor);
+
+ /* The hardware expects here size and offset in words (2 byte) */
+ MLX5_SET(ste_double_action_insert_with_ptr_v1, d_action, size, size / 2);
+ MLX5_SET(ste_double_action_insert_with_ptr_v1, d_action, start_offset, offset / 2);
+
+ MLX5_SET(ste_double_action_insert_with_ptr_v1, d_action, pointer, reformat_id);
+ MLX5_SET(ste_double_action_insert_with_ptr_v1, d_action, attributes,
+ DR_STE_V1_ACTION_INSERT_PTR_ATTR_NONE);
+
+ dr_ste_v1_set_reparse(hw_ste_p);
+}
+
static void dr_ste_v1_set_tx_push_vlan(u8 *hw_ste_p, u8 *d_action,
u32 vlan_hdr)
{
dr_ste_v1_set_reparse(hw_ste_p);
}
-static void dr_ste_v1_set_tx_encap_l3(u8 *hw_ste_p,
- u8 *frst_s_action,
- u8 *scnd_d_action,
- u32 reformat_id,
- int size)
+static void dr_ste_v1_set_encap_l3(u8 *hw_ste_p,
+ u8 *frst_s_action,
+ u8 *scnd_d_action,
+ u32 reformat_id,
+ int size)
{
/* Remove L2 headers */
MLX5_SET(ste_single_action_remove_header_v1, frst_s_action, action_id,
action_sz = DR_STE_ACTION_TRIPLE_SZ;
allow_encap = true;
}
- dr_ste_v1_set_tx_encap(last_ste, action,
- attr->reformat_id,
- attr->reformat_size);
+ dr_ste_v1_set_encap(last_ste, action,
+ attr->reformat.id,
+ attr->reformat.size);
action_sz -= DR_STE_ACTION_DOUBLE_SZ;
action += DR_STE_ACTION_DOUBLE_SZ;
} else if (action_type_set[DR_ACTION_TYP_L2_TO_TNL_L3]) {
action_sz = DR_STE_ACTION_TRIPLE_SZ;
d_action = action + DR_STE_ACTION_SINGLE_SZ;
- dr_ste_v1_set_tx_encap_l3(last_ste,
- action, d_action,
- attr->reformat_id,
- attr->reformat_size);
+ dr_ste_v1_set_encap_l3(last_ste,
+ action, d_action,
+ attr->reformat.id,
+ attr->reformat.size);
action_sz -= DR_STE_ACTION_TRIPLE_SZ;
action += DR_STE_ACTION_TRIPLE_SZ;
+ } else if (action_type_set[DR_ACTION_TYP_INSERT_HDR]) {
+ if (!allow_encap || action_sz < DR_STE_ACTION_DOUBLE_SZ) {
+ dr_ste_v1_arr_init_next_match(&last_ste, added_stes, attr->gvmi);
+ action = MLX5_ADDR_OF(ste_mask_and_match_v1, last_ste, action);
+ action_sz = DR_STE_ACTION_TRIPLE_SZ;
+ }
+ dr_ste_v1_set_insert_hdr(last_ste, action,
+ attr->reformat.id,
+ attr->reformat.param_0,
+ attr->reformat.param_1,
+ attr->reformat.size);
+ action_sz -= DR_STE_ACTION_DOUBLE_SZ;
+ action += DR_STE_ACTION_DOUBLE_SZ;
}
dr_ste_v1_set_hit_gvmi(last_ste, attr->hit_gvmi);
}
if (action_type_set[DR_ACTION_TYP_CTR]) {
- /* Counter action set after decap to exclude decaped header */
+ /* Counter action set after decap and before insert_hdr
+ * to exclude decaped / encaped header respectively.
+ */
if (!allow_ctr) {
dr_ste_v1_arr_init_next_match(&last_ste, added_stes, attr->gvmi);
action = MLX5_ADDR_OF(ste_mask_and_match_v1, last_ste, action);
dr_ste_v1_set_counter_id(last_ste, attr->ctr_id);
}
+ if (action_type_set[DR_ACTION_TYP_L2_TO_TNL_L2]) {
+ if (action_sz < DR_STE_ACTION_DOUBLE_SZ) {
+ dr_ste_v1_arr_init_next_match(&last_ste, added_stes, attr->gvmi);
+ action = MLX5_ADDR_OF(ste_mask_and_match_v1, last_ste, action);
+ action_sz = DR_STE_ACTION_TRIPLE_SZ;
+ }
+ dr_ste_v1_set_encap(last_ste, action,
+ attr->reformat.id,
+ attr->reformat.size);
+ action_sz -= DR_STE_ACTION_DOUBLE_SZ;
+ action += DR_STE_ACTION_DOUBLE_SZ;
+ allow_modify_hdr = false;
+ } else if (action_type_set[DR_ACTION_TYP_L2_TO_TNL_L3]) {
+ u8 *d_action;
+
+ if (action_sz < DR_STE_ACTION_TRIPLE_SZ) {
+ dr_ste_v1_arr_init_next_match(&last_ste, added_stes, attr->gvmi);
+ action = MLX5_ADDR_OF(ste_mask_and_match_v1, last_ste, action);
+ action_sz = DR_STE_ACTION_TRIPLE_SZ;
+ }
+
+ d_action = action + DR_STE_ACTION_SINGLE_SZ;
+
+ dr_ste_v1_set_encap_l3(last_ste,
+ action, d_action,
+ attr->reformat.id,
+ attr->reformat.size);
+ action_sz -= DR_STE_ACTION_TRIPLE_SZ;
+ allow_modify_hdr = false;
+ } else if (action_type_set[DR_ACTION_TYP_INSERT_HDR]) {
+ /* Modify header, decap, and encap must use different STEs */
+ if (!allow_modify_hdr || action_sz < DR_STE_ACTION_DOUBLE_SZ) {
+ dr_ste_v1_arr_init_next_match(&last_ste, added_stes, attr->gvmi);
+ action = MLX5_ADDR_OF(ste_mask_and_match_v1, last_ste, action);
+ action_sz = DR_STE_ACTION_TRIPLE_SZ;
+ }
+ dr_ste_v1_set_insert_hdr(last_ste, action,
+ attr->reformat.id,
+ attr->reformat.param_0,
+ attr->reformat.param_1,
+ attr->reformat.size);
+ action_sz -= DR_STE_ACTION_DOUBLE_SZ;
+ action += DR_STE_ACTION_DOUBLE_SZ;
+ allow_modify_hdr = false;
+ }
+
dr_ste_v1_set_hit_gvmi(last_ste, attr->hit_gvmi);
dr_ste_v1_set_hit_addr(last_ste, attr->final_icm_addr, 1);
}
if (hw_action_sz / DR_STE_ACTION_DOUBLE_SZ < DR_STE_DECAP_L3_ACTION_NUM)
return -EINVAL;
- memcpy(padded_data, data, data_sz);
+ inline_data_sz =
+ MLX5_FLD_SZ_BYTES(ste_double_action_insert_with_inline_v1, inline_data);
+
+ /* Add an alignment padding */
+ memcpy(padded_data + data_sz % inline_data_sz, data, data_sz);
/* Remove L2L3 outer headers */
MLX5_SET(ste_single_action_remove_header_v1, hw_action, action_id,
hw_action += DR_STE_ACTION_DOUBLE_SZ;
used_actions++; /* Remove and NOP are a single double action */
- inline_data_sz =
- MLX5_FLD_SZ_BYTES(ste_double_action_insert_with_inline_v1, inline_data);
+ /* Point to the last dword of the header */
+ data_ptr += (data_sz / inline_data_sz) * inline_data_sz;
- /* Add the new header inline + 2 extra bytes */
+ /* Add the new header using inline action 4Byte at a time, the header
+ * is added in reversed order to the beginning of the packet to avoid
+ * incorrect parsing by the HW. Since header is 14B or 18B an extra
+ * two bytes are padded and later removed.
+ */
for (i = 0; i < data_sz / inline_data_sz + 1; i++) {
void *addr_inline;
MLX5_SET(ste_double_action_insert_with_inline_v1, hw_action, action_id,
DR_STE_V1_ACTION_ID_INSERT_INLINE);
/* The hardware expects here offset to words (2 bytes) */
- MLX5_SET(ste_double_action_insert_with_inline_v1, hw_action, start_offset,
- i * 2);
+ MLX5_SET(ste_double_action_insert_with_inline_v1, hw_action, start_offset, 0);
/* Copy bytes one by one to avoid endianness problem */
addr_inline = MLX5_ADDR_OF(ste_double_action_insert_with_inline_v1,
hw_action, inline_data);
- memcpy(addr_inline, data_ptr, inline_data_sz);
+ memcpy(addr_inline, data_ptr - i * inline_data_sz, inline_data_sz);
hw_action += DR_STE_ACTION_DOUBLE_SZ;
- data_ptr += inline_data_sz;
used_actions++;
}
- /* Remove 2 extra bytes */
+ /* Remove first 2 extra bytes */
MLX5_SET(ste_single_action_remove_header_size_v1, hw_action, action_id,
DR_STE_V1_ACTION_ID_REMOVE_BY_SIZE);
- MLX5_SET(ste_single_action_remove_header_size_v1, hw_action, start_offset, data_sz / 2);
+ MLX5_SET(ste_single_action_remove_header_size_v1, hw_action, start_offset, 0);
/* The hardware expects here size in words (2 bytes) */
MLX5_SET(ste_single_action_remove_header_size_v1, hw_action, remove_size, 1);
used_actions++;
.set_byte_mask = &dr_ste_v1_set_byte_mask,
.get_byte_mask = &dr_ste_v1_get_byte_mask,
/* Actions */
+ .actions_caps = DR_STE_CTX_ACTION_CAP_RX_ENCAP,
.set_actions_rx = &dr_ste_v1_set_actions_rx,
.set_actions_tx = &dr_ste_v1_set_actions_tx,
.modify_field_arr_sz = ARRAY_SIZE(dr_ste_v1_action_modify_field_arr),
DR_STE_SIZE_REDUCED = DR_STE_SIZE - DR_STE_SIZE_MASK,
};
+enum mlx5dr_ste_ctx_action_cap {
+ DR_STE_CTX_ACTION_CAP_NONE = 0,
+ DR_STE_CTX_ACTION_CAP_RX_ENCAP = 1 << 0,
+};
+
enum {
DR_MODIFY_ACTION_SIZE = 8,
};
DR_ACTION_TYP_VPORT,
DR_ACTION_TYP_POP_VLAN,
DR_ACTION_TYP_PUSH_VLAN,
+ DR_ACTION_TYP_INSERT_HDR,
DR_ACTION_TYP_MAX,
};
u32 ctr_id;
u16 gvmi;
u16 hit_gvmi;
- u32 reformat_id;
- u32 reformat_size;
+ struct {
+ u32 id;
+ u32 size;
+ u8 param_0;
+ u8 param_1;
+ } reformat;
struct {
int count;
u32 headers[MLX5DR_MAX_VLANS];
struct mlx5dr_action_reformat {
struct mlx5dr_domain *dmn;
- u32 reformat_id;
- u32 reformat_size;
+ u32 id;
+ u32 size;
+ u8 param_0;
+ u8 param_1;
};
struct mlx5dr_action_dest_tbl {
struct mlx5dr_cmd_query_flow_table_details *output);
int mlx5dr_cmd_create_reformat_ctx(struct mlx5_core_dev *mdev,
enum mlx5_reformat_ctx_type rt,
+ u8 reformat_param_0,
+ u8 reformat_param_1,
size_t reformat_size,
void *reformat_data,
u32 *reformat_id);
u32 tx_head;
void *buf;
u32 buf_size;
- struct ib_wc wc[MAX_SEND_CQE];
u8 sync_buff[MIN_READ_SYNC];
struct mlx5dr_mr *sync_mr;
spinlock_t lock; /* Protect the data path of the send ring */
static int mlx5_cmd_dr_create_flow_table(struct mlx5_flow_root_namespace *ns,
struct mlx5_flow_table *ft,
- unsigned int log_size,
+ unsigned int size,
struct mlx5_flow_table *next_ft)
{
struct mlx5dr_table *tbl;
if (mlx5_dr_is_fw_table(ft->flags))
return mlx5_fs_cmd_get_fw_cmds()->create_flow_table(ns, ft,
- log_size,
+ size,
next_ft);
flags = ft->flags;
/* turn off encap/decap if not supported for sw-str by fw */
}
}
+ ft->max_fte = INT_MAX;
+
return 0;
}
DR_ACTION_REFORMAT_TYP_TNL_L2_TO_L2;
tmp_action = mlx5dr_action_create_packet_reformat(domain,
- decap_type, 0,
+ decap_type,
+ 0, 0, 0,
NULL);
if (!tmp_action) {
err = -ENOMEM;
}
static int mlx5_cmd_dr_packet_reformat_alloc(struct mlx5_flow_root_namespace *ns,
- int reformat_type,
- size_t size,
- void *reformat_data,
+ struct mlx5_pkt_reformat_params *params,
enum mlx5_flow_namespace_type namespace,
struct mlx5_pkt_reformat *pkt_reformat)
{
struct mlx5dr_action *action;
int dr_reformat;
- switch (reformat_type) {
+ switch (params->type) {
case MLX5_REFORMAT_TYPE_L2_TO_VXLAN:
case MLX5_REFORMAT_TYPE_L2_TO_NVGRE:
case MLX5_REFORMAT_TYPE_L2_TO_L2_TUNNEL:
case MLX5_REFORMAT_TYPE_L2_TO_L3_TUNNEL:
dr_reformat = DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L3;
break;
+ case MLX5_REFORMAT_TYPE_INSERT_HDR:
+ dr_reformat = DR_ACTION_REFORMAT_TYP_INSERT_HDR;
+ break;
default:
mlx5_core_err(ns->dev, "Packet-reformat not supported(%d)\n",
- reformat_type);
+ params->type);
return -EOPNOTSUPP;
}
action = mlx5dr_action_create_packet_reformat(dr_domain,
dr_reformat,
- size,
- reformat_data);
+ params->param_0,
+ params->param_1,
+ params->size,
+ params->data);
if (!action) {
mlx5_core_err(ns->dev, "Failed allocating packet-reformat action\n");
return -EINVAL;
DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L2,
DR_ACTION_REFORMAT_TYP_TNL_L3_TO_L2,
DR_ACTION_REFORMAT_TYP_L2_TO_TNL_L3,
+ DR_ACTION_REFORMAT_TYP_INSERT_HDR,
};
struct mlx5dr_match_parameters {
struct mlx5dr_action *
mlx5dr_action_create_packet_reformat(struct mlx5dr_domain *dmn,
enum mlx5dr_action_reformat_type reformat_type,
+ u8 reformat_param_0,
+ u8 reformat_param_1,
size_t data_sz,
void *data);
static inline bool
mlx5dr_is_supported(struct mlx5_core_dev *dev)
{
- return MLX5_CAP_ESW_FLOWTABLE_FDB(dev, sw_owner) ||
- (MLX5_CAP_ESW_FLOWTABLE_FDB(dev, sw_owner_v2) &&
- (MLX5_CAP_GEN(dev, steering_format_version) <=
- MLX5_STEERING_FORMAT_CONNECTX_6DX));
+ return MLX5_CAP_GEN(dev, roce) &&
+ (MLX5_CAP_ESW_FLOWTABLE_FDB(dev, sw_owner) ||
+ (MLX5_CAP_ESW_FLOWTABLE_FDB(dev, sw_owner_v2) &&
+ (MLX5_CAP_GEN(dev, steering_format_version) <=
+ MLX5_STEERING_FORMAT_CONNECTX_6DX)));
}
/* buddy functions & structure */
return err;
}
+static void mlx5_hairpin_unpair_peer_sq(struct mlx5_hairpin *hp)
+{
+ int i;
+
+ for (i = 0; i < hp->num_channels; i++)
+ mlx5_hairpin_modify_sq(hp->peer_mdev, hp->sqn[i], MLX5_SQC_STATE_RDY,
+ MLX5_SQC_STATE_RST, 0, 0);
+}
+
static void mlx5_hairpin_unpair_queues(struct mlx5_hairpin *hp)
{
int i;
for (i = 0; i < hp->num_channels; i++)
mlx5_hairpin_modify_rq(hp->func_mdev, hp->rqn[i], MLX5_RQC_STATE_RDY,
MLX5_RQC_STATE_RST, 0, 0);
-
/* unset peer SQs */
- if (hp->peer_gone)
- return;
- for (i = 0; i < hp->num_channels; i++)
- mlx5_hairpin_modify_sq(hp->peer_mdev, hp->sqn[i], MLX5_SQC_STATE_RDY,
- MLX5_SQC_STATE_RST, 0, 0);
+ if (!hp->peer_gone)
+ mlx5_hairpin_unpair_peer_sq(hp);
}
struct mlx5_hairpin *
mlx5_hairpin_destroy_queues(hp);
kfree(hp);
}
+
+void mlx5_core_hairpin_clear_dead_peer(struct mlx5_hairpin *hp)
+{
+ int i;
+
+ mlx5_hairpin_unpair_peer_sq(hp);
+
+ /* destroy peer SQ */
+ for (i = 0; i < hp->num_channels; i++)
+ mlx5_core_destroy_sq(hp->peer_mdev, hp->sqn[i]);
+
+ hp->peer_gone = true;
+}
void *in;
int err;
- if (!vport)
- return -EINVAL;
if (!MLX5_CAP_GEN(mdev, vport_group_manager))
return -EACCES;
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause
+#
+# Mellanox GigE driver configuration
+#
+
+config MLXBF_GIGE
+ tristate "Mellanox Technologies BlueField Gigabit Ethernet support"
+ depends on (ARM64 && ACPI) || COMPILE_TEST
+ select PHYLIB
+ help
+ The second generation BlueField SoC from Mellanox Technologies
+ supports an out-of-band Gigabit Ethernet management port to the
+ Arm subsystem.
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause
+
+obj-$(CONFIG_MLXBF_GIGE) += mlxbf_gige.o
+
+mlxbf_gige-y := mlxbf_gige_ethtool.o \
+ mlxbf_gige_gpio.o \
+ mlxbf_gige_intr.o \
+ mlxbf_gige_main.o \
+ mlxbf_gige_mdio.o \
+ mlxbf_gige_rx.o \
+ mlxbf_gige_tx.o
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause */
+
+/* Header file for Gigabit Ethernet driver for Mellanox BlueField SoC
+ * - this file contains software data structures and any chip-specific
+ * data structures (e.g. TX WQE format) that are memory resident.
+ *
+ * Copyright (C) 2020-2021 NVIDIA CORPORATION & AFFILIATES
+ */
+
+#ifndef __MLXBF_GIGE_H__
+#define __MLXBF_GIGE_H__
+
+#include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/irqreturn.h>
+#include <linux/netdevice.h>
+#include <linux/irq.h>
+
+/* The silicon design supports a maximum RX ring size of
+ * 32K entries. Based on current testing this maximum size
+ * is not required to be supported. Instead the RX ring
+ * will be capped at a realistic value of 1024 entries.
+ */
+#define MLXBF_GIGE_MIN_RXQ_SZ 32
+#define MLXBF_GIGE_MAX_RXQ_SZ 1024
+#define MLXBF_GIGE_DEFAULT_RXQ_SZ 128
+
+#define MLXBF_GIGE_MIN_TXQ_SZ 4
+#define MLXBF_GIGE_MAX_TXQ_SZ 256
+#define MLXBF_GIGE_DEFAULT_TXQ_SZ 128
+
+#define MLXBF_GIGE_DEFAULT_BUF_SZ 2048
+
+#define MLXBF_GIGE_DMA_PAGE_SZ 4096
+#define MLXBF_GIGE_DMA_PAGE_SHIFT 12
+
+/* There are four individual MAC RX filters. Currently
+ * two of them are being used: one for the broadcast MAC
+ * (index 0) and one for local MAC (index 1)
+ */
+#define MLXBF_GIGE_BCAST_MAC_FILTER_IDX 0
+#define MLXBF_GIGE_LOCAL_MAC_FILTER_IDX 1
+
+/* Define for broadcast MAC literal */
+#define BCAST_MAC_ADDR 0xFFFFFFFFFFFF
+
+/* There are three individual interrupts:
+ * 1) Errors, "OOB" interrupt line
+ * 2) Receive Packet, "OOB_LLU" interrupt line
+ * 3) LLU and PLU Events, "OOB_PLU" interrupt line
+ */
+#define MLXBF_GIGE_ERROR_INTR_IDX 0
+#define MLXBF_GIGE_RECEIVE_PKT_INTR_IDX 1
+#define MLXBF_GIGE_LLU_PLU_INTR_IDX 2
+#define MLXBF_GIGE_PHY_INT_N 3
+
+#define MLXBF_GIGE_MDIO_DEFAULT_PHY_ADDR 0x3
+
+#define MLXBF_GIGE_DEFAULT_PHY_INT_GPIO 12
+
+struct mlxbf_gige_stats {
+ u64 hw_access_errors;
+ u64 tx_invalid_checksums;
+ u64 tx_small_frames;
+ u64 tx_index_errors;
+ u64 sw_config_errors;
+ u64 sw_access_errors;
+ u64 rx_truncate_errors;
+ u64 rx_mac_errors;
+ u64 rx_din_dropped_pkts;
+ u64 tx_fifo_full;
+ u64 rx_filter_passed_pkts;
+ u64 rx_filter_discard_pkts;
+};
+
+struct mlxbf_gige {
+ void __iomem *base;
+ void __iomem *llu_base;
+ void __iomem *plu_base;
+ struct device *dev;
+ struct net_device *netdev;
+ struct platform_device *pdev;
+ void __iomem *mdio_io;
+ struct mii_bus *mdiobus;
+ void __iomem *gpio_io;
+ struct irq_domain *irqdomain;
+ u32 phy_int_gpio_mask;
+ spinlock_t lock; /* for packet processing indices */
+ spinlock_t gpio_lock; /* for GPIO bus access */
+ u16 rx_q_entries;
+ u16 tx_q_entries;
+ u64 *tx_wqe_base;
+ dma_addr_t tx_wqe_base_dma;
+ u64 *tx_wqe_next;
+ u64 *tx_cc;
+ dma_addr_t tx_cc_dma;
+ dma_addr_t *rx_wqe_base;
+ dma_addr_t rx_wqe_base_dma;
+ u64 *rx_cqe_base;
+ dma_addr_t rx_cqe_base_dma;
+ u16 tx_pi;
+ u16 prev_tx_ci;
+ u64 error_intr_count;
+ u64 rx_intr_count;
+ u64 llu_plu_intr_count;
+ struct sk_buff *rx_skb[MLXBF_GIGE_MAX_RXQ_SZ];
+ struct sk_buff *tx_skb[MLXBF_GIGE_MAX_TXQ_SZ];
+ int error_irq;
+ int rx_irq;
+ int llu_plu_irq;
+ int phy_irq;
+ int hw_phy_irq;
+ bool promisc_enabled;
+ u8 valid_polarity;
+ struct napi_struct napi;
+ struct mlxbf_gige_stats stats;
+};
+
+/* Rx Work Queue Element definitions */
+#define MLXBF_GIGE_RX_WQE_SZ 8
+
+/* Rx Completion Queue Element definitions */
+#define MLXBF_GIGE_RX_CQE_SZ 8
+#define MLXBF_GIGE_RX_CQE_PKT_LEN_MASK GENMASK(10, 0)
+#define MLXBF_GIGE_RX_CQE_VALID_MASK GENMASK(11, 11)
+#define MLXBF_GIGE_RX_CQE_PKT_STATUS_MASK GENMASK(15, 12)
+#define MLXBF_GIGE_RX_CQE_PKT_STATUS_MAC_ERR GENMASK(12, 12)
+#define MLXBF_GIGE_RX_CQE_PKT_STATUS_TRUNCATED GENMASK(13, 13)
+#define MLXBF_GIGE_RX_CQE_CHKSUM_MASK GENMASK(31, 16)
+
+/* Tx Work Queue Element definitions */
+#define MLXBF_GIGE_TX_WQE_SZ_QWORDS 2
+#define MLXBF_GIGE_TX_WQE_SZ 16
+#define MLXBF_GIGE_TX_WQE_PKT_LEN_MASK GENMASK(10, 0)
+#define MLXBF_GIGE_TX_WQE_UPDATE_MASK GENMASK(31, 31)
+#define MLXBF_GIGE_TX_WQE_CHKSUM_LEN_MASK GENMASK(42, 32)
+#define MLXBF_GIGE_TX_WQE_CHKSUM_START_MASK GENMASK(55, 48)
+#define MLXBF_GIGE_TX_WQE_CHKSUM_OFFSET_MASK GENMASK(63, 56)
+
+/* Macro to return packet length of specified TX WQE */
+#define MLXBF_GIGE_TX_WQE_PKT_LEN(tx_wqe_addr) \
+ (*((tx_wqe_addr) + 1) & MLXBF_GIGE_TX_WQE_PKT_LEN_MASK)
+
+/* Tx Completion Count */
+#define MLXBF_GIGE_TX_CC_SZ 8
+
+/* List of resources in ACPI table */
+enum mlxbf_gige_res {
+ MLXBF_GIGE_RES_MAC,
+ MLXBF_GIGE_RES_MDIO9,
+ MLXBF_GIGE_RES_GPIO0,
+ MLXBF_GIGE_RES_LLU,
+ MLXBF_GIGE_RES_PLU
+};
+
+/* Version of register data returned by mlxbf_gige_get_regs() */
+#define MLXBF_GIGE_REGS_VERSION 1
+
+int mlxbf_gige_mdio_probe(struct platform_device *pdev,
+ struct mlxbf_gige *priv);
+void mlxbf_gige_mdio_remove(struct mlxbf_gige *priv);
+irqreturn_t mlxbf_gige_mdio_handle_phy_interrupt(int irq, void *dev_id);
+void mlxbf_gige_mdio_enable_phy_int(struct mlxbf_gige *priv);
+
+void mlxbf_gige_set_mac_rx_filter(struct mlxbf_gige *priv,
+ unsigned int index, u64 dmac);
+void mlxbf_gige_get_mac_rx_filter(struct mlxbf_gige *priv,
+ unsigned int index, u64 *dmac);
+void mlxbf_gige_enable_promisc(struct mlxbf_gige *priv);
+void mlxbf_gige_disable_promisc(struct mlxbf_gige *priv);
+int mlxbf_gige_rx_init(struct mlxbf_gige *priv);
+void mlxbf_gige_rx_deinit(struct mlxbf_gige *priv);
+int mlxbf_gige_tx_init(struct mlxbf_gige *priv);
+void mlxbf_gige_tx_deinit(struct mlxbf_gige *priv);
+bool mlxbf_gige_handle_tx_complete(struct mlxbf_gige *priv);
+netdev_tx_t mlxbf_gige_start_xmit(struct sk_buff *skb,
+ struct net_device *netdev);
+struct sk_buff *mlxbf_gige_alloc_skb(struct mlxbf_gige *priv,
+ unsigned int map_len,
+ dma_addr_t *buf_dma,
+ enum dma_data_direction dir);
+int mlxbf_gige_request_irqs(struct mlxbf_gige *priv);
+void mlxbf_gige_free_irqs(struct mlxbf_gige *priv);
+int mlxbf_gige_poll(struct napi_struct *napi, int budget);
+extern const struct ethtool_ops mlxbf_gige_ethtool_ops;
+void mlxbf_gige_update_tx_wqe_next(struct mlxbf_gige *priv);
+
+int mlxbf_gige_gpio_init(struct platform_device *pdev, struct mlxbf_gige *priv);
+void mlxbf_gige_gpio_free(struct mlxbf_gige *priv);
+
+#endif /* !defined(__MLXBF_GIGE_H__) */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause
+
+/* Ethtool support for Mellanox Gigabit Ethernet driver
+ *
+ * Copyright (C) 2020-2021 NVIDIA CORPORATION & AFFILIATES
+ */
+
+#include <linux/phy.h>
+
+#include "mlxbf_gige.h"
+#include "mlxbf_gige_regs.h"
+
+/* Start of struct ethtool_ops functions */
+static int mlxbf_gige_get_regs_len(struct net_device *netdev)
+{
+ return MLXBF_GIGE_MMIO_REG_SZ;
+}
+
+static void mlxbf_gige_get_regs(struct net_device *netdev,
+ struct ethtool_regs *regs, void *p)
+{
+ struct mlxbf_gige *priv = netdev_priv(netdev);
+
+ regs->version = MLXBF_GIGE_REGS_VERSION;
+
+ /* Read entire MMIO register space and store results
+ * into the provided buffer. Each 64-bit word is converted
+ * to big-endian to make the output more readable.
+ *
+ * NOTE: by design, a read to an offset without an existing
+ * register will be acknowledged and return zero.
+ */
+ memcpy_fromio(p, priv->base, MLXBF_GIGE_MMIO_REG_SZ);
+}
+
+static void mlxbf_gige_get_ringparam(struct net_device *netdev,
+ struct ethtool_ringparam *ering)
+{
+ struct mlxbf_gige *priv = netdev_priv(netdev);
+
+ ering->rx_max_pending = MLXBF_GIGE_MAX_RXQ_SZ;
+ ering->tx_max_pending = MLXBF_GIGE_MAX_TXQ_SZ;
+ ering->rx_pending = priv->rx_q_entries;
+ ering->tx_pending = priv->tx_q_entries;
+}
+
+static const struct {
+ const char string[ETH_GSTRING_LEN];
+} mlxbf_gige_ethtool_stats_keys[] = {
+ { "hw_access_errors" },
+ { "tx_invalid_checksums" },
+ { "tx_small_frames" },
+ { "tx_index_errors" },
+ { "sw_config_errors" },
+ { "sw_access_errors" },
+ { "rx_truncate_errors" },
+ { "rx_mac_errors" },
+ { "rx_din_dropped_pkts" },
+ { "tx_fifo_full" },
+ { "rx_filter_passed_pkts" },
+ { "rx_filter_discard_pkts" },
+};
+
+static int mlxbf_gige_get_sset_count(struct net_device *netdev, int stringset)
+{
+ if (stringset != ETH_SS_STATS)
+ return -EOPNOTSUPP;
+ return ARRAY_SIZE(mlxbf_gige_ethtool_stats_keys);
+}
+
+static void mlxbf_gige_get_strings(struct net_device *netdev, u32 stringset,
+ u8 *buf)
+{
+ if (stringset != ETH_SS_STATS)
+ return;
+ memcpy(buf, &mlxbf_gige_ethtool_stats_keys,
+ sizeof(mlxbf_gige_ethtool_stats_keys));
+}
+
+static void mlxbf_gige_get_ethtool_stats(struct net_device *netdev,
+ struct ethtool_stats *estats,
+ u64 *data)
+{
+ struct mlxbf_gige *priv = netdev_priv(netdev);
+
+ /* Fill data array with interface statistics
+ *
+ * NOTE: the data writes must be in
+ * sync with the strings shown in
+ * the mlxbf_gige_ethtool_stats_keys[] array
+ *
+ * NOTE2: certain statistics below are zeroed upon
+ * port disable, so the calculation below
+ * must include the "cached" value of the stat
+ * plus the value read directly from hardware.
+ * Cached statistics are currently:
+ * rx_din_dropped_pkts
+ * rx_filter_passed_pkts
+ * rx_filter_discard_pkts
+ */
+ *data++ = priv->stats.hw_access_errors;
+ *data++ = priv->stats.tx_invalid_checksums;
+ *data++ = priv->stats.tx_small_frames;
+ *data++ = priv->stats.tx_index_errors;
+ *data++ = priv->stats.sw_config_errors;
+ *data++ = priv->stats.sw_access_errors;
+ *data++ = priv->stats.rx_truncate_errors;
+ *data++ = priv->stats.rx_mac_errors;
+ *data++ = (priv->stats.rx_din_dropped_pkts +
+ readq(priv->base + MLXBF_GIGE_RX_DIN_DROP_COUNTER));
+ *data++ = priv->stats.tx_fifo_full;
+ *data++ = (priv->stats.rx_filter_passed_pkts +
+ readq(priv->base + MLXBF_GIGE_RX_PASS_COUNTER_ALL));
+ *data++ = (priv->stats.rx_filter_discard_pkts +
+ readq(priv->base + MLXBF_GIGE_RX_DISC_COUNTER_ALL));
+}
+
+static void mlxbf_gige_get_pauseparam(struct net_device *netdev,
+ struct ethtool_pauseparam *pause)
+{
+ pause->autoneg = AUTONEG_DISABLE;
+ pause->rx_pause = 1;
+ pause->tx_pause = 1;
+}
+
+const struct ethtool_ops mlxbf_gige_ethtool_ops = {
+ .get_link = ethtool_op_get_link,
+ .get_ringparam = mlxbf_gige_get_ringparam,
+ .get_regs_len = mlxbf_gige_get_regs_len,
+ .get_regs = mlxbf_gige_get_regs,
+ .get_strings = mlxbf_gige_get_strings,
+ .get_sset_count = mlxbf_gige_get_sset_count,
+ .get_ethtool_stats = mlxbf_gige_get_ethtool_stats,
+ .nway_reset = phy_ethtool_nway_reset,
+ .get_pauseparam = mlxbf_gige_get_pauseparam,
+ .get_link_ksettings = phy_ethtool_get_link_ksettings,
+};
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause
+
+/* Initialize and handle GPIO interrupt triggered by INT_N PHY signal.
+ * This GPIO interrupt triggers the PHY state machine to bring the link
+ * up/down.
+ *
+ * Copyright (C) 2021 NVIDIA CORPORATION & AFFILIATES
+ */
+
+#include <linux/acpi.h>
+#include <linux/bitfield.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/gpio/driver.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/irq.h>
+#include <linux/irqdomain.h>
+#include <linux/irqreturn.h>
+#include <linux/platform_device.h>
+#include <linux/property.h>
+
+#include "mlxbf_gige.h"
+#include "mlxbf_gige_regs.h"
+
+#define MLXBF_GIGE_GPIO_CAUSE_FALL_EN 0x48
+#define MLXBF_GIGE_GPIO_CAUSE_OR_CAUSE_EVTEN0 0x80
+#define MLXBF_GIGE_GPIO_CAUSE_OR_EVTEN0 0x94
+#define MLXBF_GIGE_GPIO_CAUSE_OR_CLRCAUSE 0x98
+
+static void mlxbf_gige_gpio_enable(struct mlxbf_gige *priv)
+{
+ unsigned long flags;
+ u32 val;
+
+ spin_lock_irqsave(&priv->gpio_lock, flags);
+ val = readl(priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_CLRCAUSE);
+ val |= priv->phy_int_gpio_mask;
+ writel(val, priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_CLRCAUSE);
+
+ /* The INT_N interrupt level is active low.
+ * So enable cause fall bit to detect when GPIO
+ * state goes low.
+ */
+ val = readl(priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_FALL_EN);
+ val |= priv->phy_int_gpio_mask;
+ writel(val, priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_FALL_EN);
+
+ /* Enable PHY interrupt by setting the priority level */
+ val = readl(priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_EVTEN0);
+ val |= priv->phy_int_gpio_mask;
+ writel(val, priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_EVTEN0);
+ spin_unlock_irqrestore(&priv->gpio_lock, flags);
+}
+
+static void mlxbf_gige_gpio_disable(struct mlxbf_gige *priv)
+{
+ unsigned long flags;
+ u32 val;
+
+ spin_lock_irqsave(&priv->gpio_lock, flags);
+ val = readl(priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_EVTEN0);
+ val &= ~priv->phy_int_gpio_mask;
+ writel(val, priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_EVTEN0);
+ spin_unlock_irqrestore(&priv->gpio_lock, flags);
+}
+
+static irqreturn_t mlxbf_gige_gpio_handler(int irq, void *ptr)
+{
+ struct mlxbf_gige *priv;
+ u32 val;
+
+ priv = ptr;
+
+ /* Check if this interrupt is from PHY device.
+ * Return if it is not.
+ */
+ val = readl(priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_CAUSE_EVTEN0);
+ if (!(val & priv->phy_int_gpio_mask))
+ return IRQ_NONE;
+
+ /* Clear interrupt when done, otherwise, no further interrupt
+ * will be triggered.
+ */
+ val = readl(priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_CLRCAUSE);
+ val |= priv->phy_int_gpio_mask;
+ writel(val, priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_CLRCAUSE);
+
+ generic_handle_irq(priv->phy_irq);
+
+ return IRQ_HANDLED;
+}
+
+static void mlxbf_gige_gpio_mask(struct irq_data *irqd)
+{
+ struct mlxbf_gige *priv = irq_data_get_irq_chip_data(irqd);
+
+ mlxbf_gige_gpio_disable(priv);
+}
+
+static void mlxbf_gige_gpio_unmask(struct irq_data *irqd)
+{
+ struct mlxbf_gige *priv = irq_data_get_irq_chip_data(irqd);
+
+ mlxbf_gige_gpio_enable(priv);
+}
+
+static struct irq_chip mlxbf_gige_gpio_chip = {
+ .name = "mlxbf_gige_phy",
+ .irq_mask = mlxbf_gige_gpio_mask,
+ .irq_unmask = mlxbf_gige_gpio_unmask,
+};
+
+static int mlxbf_gige_gpio_domain_map(struct irq_domain *d,
+ unsigned int irq,
+ irq_hw_number_t hwirq)
+{
+ irq_set_chip_data(irq, d->host_data);
+ irq_set_chip_and_handler(irq, &mlxbf_gige_gpio_chip, handle_simple_irq);
+ irq_set_noprobe(irq);
+
+ return 0;
+}
+
+static const struct irq_domain_ops mlxbf_gige_gpio_domain_ops = {
+ .map = mlxbf_gige_gpio_domain_map,
+ .xlate = irq_domain_xlate_twocell,
+};
+
+#ifdef CONFIG_ACPI
+static int mlxbf_gige_gpio_resources(struct acpi_resource *ares,
+ void *data)
+{
+ struct acpi_resource_gpio *gpio;
+ u32 *phy_int_gpio = data;
+
+ if (ares->type == ACPI_RESOURCE_TYPE_GPIO) {
+ gpio = &ares->data.gpio;
+ *phy_int_gpio = gpio->pin_table[0];
+ }
+
+ return 1;
+}
+#endif
+
+void mlxbf_gige_gpio_free(struct mlxbf_gige *priv)
+{
+ irq_dispose_mapping(priv->phy_irq);
+ irq_domain_remove(priv->irqdomain);
+}
+
+int mlxbf_gige_gpio_init(struct platform_device *pdev,
+ struct mlxbf_gige *priv)
+{
+ struct device *dev = &pdev->dev;
+ struct resource *res;
+ u32 phy_int_gpio = 0;
+ int ret;
+
+ LIST_HEAD(resources);
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, MLXBF_GIGE_RES_GPIO0);
+ if (!res)
+ return -ENODEV;
+
+ priv->gpio_io = devm_ioremap(dev, res->start, resource_size(res));
+ if (!priv->gpio_io)
+ return -ENOMEM;
+
+#ifdef CONFIG_ACPI
+ ret = acpi_dev_get_resources(ACPI_COMPANION(dev),
+ &resources, mlxbf_gige_gpio_resources,
+ &phy_int_gpio);
+ acpi_dev_free_resource_list(&resources);
+ if (ret < 0 || !phy_int_gpio) {
+ dev_err(dev, "Error retrieving the gpio phy pin");
+ return -EINVAL;
+ }
+#endif
+
+ priv->phy_int_gpio_mask = BIT(phy_int_gpio);
+
+ mlxbf_gige_gpio_disable(priv);
+
+ priv->hw_phy_irq = platform_get_irq(pdev, MLXBF_GIGE_PHY_INT_N);
+
+ priv->irqdomain = irq_domain_add_simple(NULL, 1, 0,
+ &mlxbf_gige_gpio_domain_ops,
+ priv);
+ if (!priv->irqdomain) {
+ dev_err(dev, "Failed to add IRQ domain\n");
+ return -ENOMEM;
+ }
+
+ priv->phy_irq = irq_create_mapping(priv->irqdomain, 0);
+ if (!priv->phy_irq) {
+ irq_domain_remove(priv->irqdomain);
+ priv->irqdomain = NULL;
+ dev_err(dev, "Error mapping PHY IRQ\n");
+ return -EINVAL;
+ }
+
+ ret = devm_request_irq(dev, priv->hw_phy_irq, mlxbf_gige_gpio_handler,
+ IRQF_ONESHOT | IRQF_SHARED, "mlxbf_gige_phy", priv);
+ if (ret) {
+ dev_err(dev, "Failed to request PHY IRQ");
+ mlxbf_gige_gpio_free(priv);
+ return ret;
+ }
+
+ return ret;
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause
+
+/* Interrupt related logic for Mellanox Gigabit Ethernet driver
+ *
+ * Copyright (C) 2020-2021 NVIDIA CORPORATION & AFFILIATES
+ */
+
+#include <linux/interrupt.h>
+
+#include "mlxbf_gige.h"
+#include "mlxbf_gige_regs.h"
+
+static irqreturn_t mlxbf_gige_error_intr(int irq, void *dev_id)
+{
+ struct mlxbf_gige *priv;
+ u64 int_status;
+
+ priv = dev_id;
+
+ priv->error_intr_count++;
+
+ int_status = readq(priv->base + MLXBF_GIGE_INT_STATUS);
+
+ if (int_status & MLXBF_GIGE_INT_STATUS_HW_ACCESS_ERROR)
+ priv->stats.hw_access_errors++;
+
+ if (int_status & MLXBF_GIGE_INT_STATUS_TX_CHECKSUM_INPUTS) {
+ priv->stats.tx_invalid_checksums++;
+ /* This error condition is latched into MLXBF_GIGE_INT_STATUS
+ * when the GigE silicon operates on the offending
+ * TX WQE. The write to MLXBF_GIGE_INT_STATUS at the bottom
+ * of this routine clears this error condition.
+ */
+ }
+
+ if (int_status & MLXBF_GIGE_INT_STATUS_TX_SMALL_FRAME_SIZE) {
+ priv->stats.tx_small_frames++;
+ /* This condition happens when the networking stack invokes
+ * this driver's "start_xmit()" method with a packet whose
+ * size < 60 bytes. The GigE silicon will automatically pad
+ * this small frame up to a minimum-sized frame before it is
+ * sent. The "tx_small_frame" condition is latched into the
+ * MLXBF_GIGE_INT_STATUS register when the GigE silicon
+ * operates on the offending TX WQE. The write to
+ * MLXBF_GIGE_INT_STATUS at the bottom of this routine
+ * clears this condition.
+ */
+ }
+
+ if (int_status & MLXBF_GIGE_INT_STATUS_TX_PI_CI_EXCEED_WQ_SIZE)
+ priv->stats.tx_index_errors++;
+
+ if (int_status & MLXBF_GIGE_INT_STATUS_SW_CONFIG_ERROR)
+ priv->stats.sw_config_errors++;
+
+ if (int_status & MLXBF_GIGE_INT_STATUS_SW_ACCESS_ERROR)
+ priv->stats.sw_access_errors++;
+
+ /* Clear all error interrupts by writing '1' back to
+ * all the asserted bits in INT_STATUS. Do not write
+ * '1' back to 'receive packet' bit, since that is
+ * managed separately.
+ */
+
+ int_status &= ~MLXBF_GIGE_INT_STATUS_RX_RECEIVE_PACKET;
+
+ writeq(int_status, priv->base + MLXBF_GIGE_INT_STATUS);
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t mlxbf_gige_rx_intr(int irq, void *dev_id)
+{
+ struct mlxbf_gige *priv;
+
+ priv = dev_id;
+
+ priv->rx_intr_count++;
+
+ /* NOTE: GigE silicon automatically disables "packet rx" interrupt by
+ * setting MLXBF_GIGE_INT_MASK bit0 upon triggering the interrupt
+ * to the ARM cores. Software needs to re-enable "packet rx"
+ * interrupts by clearing MLXBF_GIGE_INT_MASK bit0.
+ */
+
+ napi_schedule(&priv->napi);
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t mlxbf_gige_llu_plu_intr(int irq, void *dev_id)
+{
+ struct mlxbf_gige *priv;
+
+ priv = dev_id;
+ priv->llu_plu_intr_count++;
+
+ return IRQ_HANDLED;
+}
+
+int mlxbf_gige_request_irqs(struct mlxbf_gige *priv)
+{
+ int err;
+
+ err = request_irq(priv->error_irq, mlxbf_gige_error_intr, 0,
+ "mlxbf_gige_error", priv);
+ if (err) {
+ dev_err(priv->dev, "Request error_irq failure\n");
+ return err;
+ }
+
+ err = request_irq(priv->rx_irq, mlxbf_gige_rx_intr, 0,
+ "mlxbf_gige_rx", priv);
+ if (err) {
+ dev_err(priv->dev, "Request rx_irq failure\n");
+ goto free_error_irq;
+ }
+
+ err = request_irq(priv->llu_plu_irq, mlxbf_gige_llu_plu_intr, 0,
+ "mlxbf_gige_llu_plu", priv);
+ if (err) {
+ dev_err(priv->dev, "Request llu_plu_irq failure\n");
+ goto free_rx_irq;
+ }
+
+ return 0;
+
+free_rx_irq:
+ free_irq(priv->rx_irq, priv);
+
+free_error_irq:
+ free_irq(priv->error_irq, priv);
+
+ return err;
+}
+
+void mlxbf_gige_free_irqs(struct mlxbf_gige *priv)
+{
+ free_irq(priv->error_irq, priv);
+ free_irq(priv->rx_irq, priv);
+ free_irq(priv->llu_plu_irq, priv);
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause
+
+/* Gigabit Ethernet driver for Mellanox BlueField SoC
+ *
+ * Copyright (C) 2020-2021 NVIDIA CORPORATION & AFFILIATES
+ */
+
+#include <linux/acpi.h>
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/etherdevice.h>
+#include <linux/interrupt.h>
+#include <linux/iopoll.h>
+#include <linux/module.h>
+#include <linux/phy.h>
+#include <linux/platform_device.h>
+#include <linux/skbuff.h>
+
+#include "mlxbf_gige.h"
+#include "mlxbf_gige_regs.h"
+
+#define DRV_NAME "mlxbf_gige"
+
+/* Allocate SKB whose payload pointer aligns with the Bluefield
+ * hardware DMA limitation, i.e. DMA operation can't cross
+ * a 4KB boundary. A maximum packet size of 2KB is assumed in the
+ * alignment formula. The alignment logic overallocates an SKB,
+ * and then adjusts the headroom so that the SKB data pointer is
+ * naturally aligned to a 2KB boundary.
+ */
+struct sk_buff *mlxbf_gige_alloc_skb(struct mlxbf_gige *priv,
+ unsigned int map_len,
+ dma_addr_t *buf_dma,
+ enum dma_data_direction dir)
+{
+ struct sk_buff *skb;
+ u64 addr, offset;
+
+ /* Overallocate the SKB so that any headroom adjustment (to
+ * provide 2KB natural alignment) does not exceed payload area
+ */
+ skb = netdev_alloc_skb(priv->netdev, MLXBF_GIGE_DEFAULT_BUF_SZ * 2);
+ if (!skb)
+ return NULL;
+
+ /* Adjust the headroom so that skb->data is naturally aligned to
+ * a 2KB boundary, which is the maximum packet size supported.
+ */
+ addr = (long)skb->data;
+ offset = (addr + MLXBF_GIGE_DEFAULT_BUF_SZ - 1) &
+ ~(MLXBF_GIGE_DEFAULT_BUF_SZ - 1);
+ offset -= addr;
+ if (offset)
+ skb_reserve(skb, offset);
+
+ /* Return streaming DMA mapping to caller */
+ *buf_dma = dma_map_single(priv->dev, skb->data, map_len, dir);
+ if (dma_mapping_error(priv->dev, *buf_dma)) {
+ dev_kfree_skb(skb);
+ *buf_dma = (dma_addr_t)0;
+ return NULL;
+ }
+
+ return skb;
+}
+
+static void mlxbf_gige_initial_mac(struct mlxbf_gige *priv)
+{
+ u8 mac[ETH_ALEN];
+ u64 local_mac;
+
+ memset(mac, 0, ETH_ALEN);
+ mlxbf_gige_get_mac_rx_filter(priv, MLXBF_GIGE_LOCAL_MAC_FILTER_IDX,
+ &local_mac);
+ u64_to_ether_addr(local_mac, mac);
+
+ if (is_valid_ether_addr(mac)) {
+ ether_addr_copy(priv->netdev->dev_addr, mac);
+ } else {
+ /* Provide a random MAC if for some reason the device has
+ * not been configured with a valid MAC address already.
+ */
+ eth_hw_addr_random(priv->netdev);
+ }
+
+ local_mac = ether_addr_to_u64(priv->netdev->dev_addr);
+ mlxbf_gige_set_mac_rx_filter(priv, MLXBF_GIGE_LOCAL_MAC_FILTER_IDX,
+ local_mac);
+}
+
+static void mlxbf_gige_cache_stats(struct mlxbf_gige *priv)
+{
+ struct mlxbf_gige_stats *p;
+
+ /* Cache stats that will be cleared by clean port operation */
+ p = &priv->stats;
+ p->rx_din_dropped_pkts += readq(priv->base +
+ MLXBF_GIGE_RX_DIN_DROP_COUNTER);
+ p->rx_filter_passed_pkts += readq(priv->base +
+ MLXBF_GIGE_RX_PASS_COUNTER_ALL);
+ p->rx_filter_discard_pkts += readq(priv->base +
+ MLXBF_GIGE_RX_DISC_COUNTER_ALL);
+}
+
+static int mlxbf_gige_clean_port(struct mlxbf_gige *priv)
+{
+ u64 control;
+ u64 temp;
+ int err;
+
+ /* Set the CLEAN_PORT_EN bit to trigger SW reset */
+ control = readq(priv->base + MLXBF_GIGE_CONTROL);
+ control |= MLXBF_GIGE_CONTROL_CLEAN_PORT_EN;
+ writeq(control, priv->base + MLXBF_GIGE_CONTROL);
+
+ /* Ensure completion of "clean port" write before polling status */
+ mb();
+
+ err = readq_poll_timeout_atomic(priv->base + MLXBF_GIGE_STATUS, temp,
+ (temp & MLXBF_GIGE_STATUS_READY),
+ 100, 100000);
+
+ /* Clear the CLEAN_PORT_EN bit at end of this loop */
+ control = readq(priv->base + MLXBF_GIGE_CONTROL);
+ control &= ~MLXBF_GIGE_CONTROL_CLEAN_PORT_EN;
+ writeq(control, priv->base + MLXBF_GIGE_CONTROL);
+
+ return err;
+}
+
+static int mlxbf_gige_open(struct net_device *netdev)
+{
+ struct mlxbf_gige *priv = netdev_priv(netdev);
+ struct phy_device *phydev = netdev->phydev;
+ u64 int_en;
+ int err;
+
+ err = mlxbf_gige_request_irqs(priv);
+ if (err)
+ return err;
+ mlxbf_gige_cache_stats(priv);
+ err = mlxbf_gige_clean_port(priv);
+ if (err)
+ goto free_irqs;
+ err = mlxbf_gige_rx_init(priv);
+ if (err)
+ goto free_irqs;
+ err = mlxbf_gige_tx_init(priv);
+ if (err)
+ goto rx_deinit;
+
+ phy_start(phydev);
+
+ netif_napi_add(netdev, &priv->napi, mlxbf_gige_poll, NAPI_POLL_WEIGHT);
+ napi_enable(&priv->napi);
+ netif_start_queue(netdev);
+
+ /* Set bits in INT_EN that we care about */
+ int_en = MLXBF_GIGE_INT_EN_HW_ACCESS_ERROR |
+ MLXBF_GIGE_INT_EN_TX_CHECKSUM_INPUTS |
+ MLXBF_GIGE_INT_EN_TX_SMALL_FRAME_SIZE |
+ MLXBF_GIGE_INT_EN_TX_PI_CI_EXCEED_WQ_SIZE |
+ MLXBF_GIGE_INT_EN_SW_CONFIG_ERROR |
+ MLXBF_GIGE_INT_EN_SW_ACCESS_ERROR |
+ MLXBF_GIGE_INT_EN_RX_RECEIVE_PACKET;
+
+ /* Ensure completion of all initialization before enabling interrupts */
+ mb();
+
+ writeq(int_en, priv->base + MLXBF_GIGE_INT_EN);
+
+ return 0;
+
+rx_deinit:
+ mlxbf_gige_rx_deinit(priv);
+
+free_irqs:
+ mlxbf_gige_free_irqs(priv);
+ return err;
+}
+
+static int mlxbf_gige_stop(struct net_device *netdev)
+{
+ struct mlxbf_gige *priv = netdev_priv(netdev);
+
+ writeq(0, priv->base + MLXBF_GIGE_INT_EN);
+ netif_stop_queue(netdev);
+ napi_disable(&priv->napi);
+ netif_napi_del(&priv->napi);
+ mlxbf_gige_free_irqs(priv);
+
+ phy_stop(netdev->phydev);
+
+ mlxbf_gige_rx_deinit(priv);
+ mlxbf_gige_tx_deinit(priv);
+ mlxbf_gige_cache_stats(priv);
+ mlxbf_gige_clean_port(priv);
+
+ return 0;
+}
+
+static int mlxbf_gige_do_ioctl(struct net_device *netdev,
+ struct ifreq *ifr, int cmd)
+{
+ if (!(netif_running(netdev)))
+ return -EINVAL;
+
+ return phy_mii_ioctl(netdev->phydev, ifr, cmd);
+}
+
+static void mlxbf_gige_set_rx_mode(struct net_device *netdev)
+{
+ struct mlxbf_gige *priv = netdev_priv(netdev);
+ bool new_promisc_enabled;
+
+ new_promisc_enabled = netdev->flags & IFF_PROMISC;
+
+ /* Only write to the hardware registers if the new setting
+ * of promiscuous mode is different from the current one.
+ */
+ if (new_promisc_enabled != priv->promisc_enabled) {
+ priv->promisc_enabled = new_promisc_enabled;
+
+ if (new_promisc_enabled)
+ mlxbf_gige_enable_promisc(priv);
+ else
+ mlxbf_gige_disable_promisc(priv);
+ }
+}
+
+static void mlxbf_gige_get_stats64(struct net_device *netdev,
+ struct rtnl_link_stats64 *stats)
+{
+ struct mlxbf_gige *priv = netdev_priv(netdev);
+
+ netdev_stats_to_stats64(stats, &netdev->stats);
+
+ stats->rx_length_errors = priv->stats.rx_truncate_errors;
+ stats->rx_fifo_errors = priv->stats.rx_din_dropped_pkts +
+ readq(priv->base + MLXBF_GIGE_RX_DIN_DROP_COUNTER);
+ stats->rx_crc_errors = priv->stats.rx_mac_errors;
+ stats->rx_errors = stats->rx_length_errors +
+ stats->rx_fifo_errors +
+ stats->rx_crc_errors;
+
+ stats->tx_fifo_errors = priv->stats.tx_fifo_full;
+ stats->tx_errors = stats->tx_fifo_errors;
+}
+
+static const struct net_device_ops mlxbf_gige_netdev_ops = {
+ .ndo_open = mlxbf_gige_open,
+ .ndo_stop = mlxbf_gige_stop,
+ .ndo_start_xmit = mlxbf_gige_start_xmit,
+ .ndo_set_mac_address = eth_mac_addr,
+ .ndo_validate_addr = eth_validate_addr,
+ .ndo_do_ioctl = mlxbf_gige_do_ioctl,
+ .ndo_set_rx_mode = mlxbf_gige_set_rx_mode,
+ .ndo_get_stats64 = mlxbf_gige_get_stats64,
+};
+
+static void mlxbf_gige_adjust_link(struct net_device *netdev)
+{
+ struct phy_device *phydev = netdev->phydev;
+
+ phy_print_status(phydev);
+}
+
+static int mlxbf_gige_probe(struct platform_device *pdev)
+{
+ struct phy_device *phydev;
+ struct net_device *netdev;
+ struct resource *mac_res;
+ struct resource *llu_res;
+ struct resource *plu_res;
+ struct mlxbf_gige *priv;
+ void __iomem *llu_base;
+ void __iomem *plu_base;
+ void __iomem *base;
+ u64 control;
+ int addr;
+ int err;
+
+ mac_res = platform_get_resource(pdev, IORESOURCE_MEM, MLXBF_GIGE_RES_MAC);
+ if (!mac_res)
+ return -ENXIO;
+
+ base = devm_ioremap_resource(&pdev->dev, mac_res);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
+
+ llu_res = platform_get_resource(pdev, IORESOURCE_MEM, MLXBF_GIGE_RES_LLU);
+ if (!llu_res)
+ return -ENXIO;
+
+ llu_base = devm_ioremap_resource(&pdev->dev, llu_res);
+ if (IS_ERR(llu_base))
+ return PTR_ERR(llu_base);
+
+ plu_res = platform_get_resource(pdev, IORESOURCE_MEM, MLXBF_GIGE_RES_PLU);
+ if (!plu_res)
+ return -ENXIO;
+
+ plu_base = devm_ioremap_resource(&pdev->dev, plu_res);
+ if (IS_ERR(plu_base))
+ return PTR_ERR(plu_base);
+
+ /* Perform general init of GigE block */
+ control = readq(base + MLXBF_GIGE_CONTROL);
+ control |= MLXBF_GIGE_CONTROL_PORT_EN;
+ writeq(control, base + MLXBF_GIGE_CONTROL);
+
+ netdev = devm_alloc_etherdev(&pdev->dev, sizeof(*priv));
+ if (!netdev)
+ return -ENOMEM;
+
+ SET_NETDEV_DEV(netdev, &pdev->dev);
+ netdev->netdev_ops = &mlxbf_gige_netdev_ops;
+ netdev->ethtool_ops = &mlxbf_gige_ethtool_ops;
+ priv = netdev_priv(netdev);
+ priv->netdev = netdev;
+
+ platform_set_drvdata(pdev, priv);
+ priv->dev = &pdev->dev;
+ priv->pdev = pdev;
+
+ spin_lock_init(&priv->lock);
+ spin_lock_init(&priv->gpio_lock);
+
+ /* Attach MDIO device */
+ err = mlxbf_gige_mdio_probe(pdev, priv);
+ if (err)
+ return err;
+
+ err = mlxbf_gige_gpio_init(pdev, priv);
+ if (err) {
+ dev_err(&pdev->dev, "PHY IRQ initialization failed\n");
+ mlxbf_gige_mdio_remove(priv);
+ return -ENODEV;
+ }
+
+ priv->base = base;
+ priv->llu_base = llu_base;
+ priv->plu_base = plu_base;
+
+ priv->rx_q_entries = MLXBF_GIGE_DEFAULT_RXQ_SZ;
+ priv->tx_q_entries = MLXBF_GIGE_DEFAULT_TXQ_SZ;
+
+ /* Write initial MAC address to hardware */
+ mlxbf_gige_initial_mac(priv);
+
+ err = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
+ if (err) {
+ dev_err(&pdev->dev, "DMA configuration failed: 0x%x\n", err);
+ goto out;
+ }
+
+ priv->error_irq = platform_get_irq(pdev, MLXBF_GIGE_ERROR_INTR_IDX);
+ priv->rx_irq = platform_get_irq(pdev, MLXBF_GIGE_RECEIVE_PKT_INTR_IDX);
+ priv->llu_plu_irq = platform_get_irq(pdev, MLXBF_GIGE_LLU_PLU_INTR_IDX);
+
+ phydev = phy_find_first(priv->mdiobus);
+ if (!phydev) {
+ err = -ENODEV;
+ goto out;
+ }
+
+ addr = phydev->mdio.addr;
+ priv->mdiobus->irq[addr] = priv->phy_irq;
+ phydev->irq = priv->phy_irq;
+
+ err = phy_connect_direct(netdev, phydev,
+ mlxbf_gige_adjust_link,
+ PHY_INTERFACE_MODE_GMII);
+ if (err) {
+ dev_err(&pdev->dev, "Could not attach to PHY\n");
+ goto out;
+ }
+
+ /* MAC only supports 1000T full duplex mode */
+ phy_remove_link_mode(phydev, ETHTOOL_LINK_MODE_1000baseT_Half_BIT);
+ phy_remove_link_mode(phydev, ETHTOOL_LINK_MODE_100baseT_Full_BIT);
+ phy_remove_link_mode(phydev, ETHTOOL_LINK_MODE_100baseT_Half_BIT);
+ phy_remove_link_mode(phydev, ETHTOOL_LINK_MODE_10baseT_Full_BIT);
+ phy_remove_link_mode(phydev, ETHTOOL_LINK_MODE_10baseT_Half_BIT);
+
+ /* Only symmetric pause with flow control enabled is supported so no
+ * need to negotiate pause.
+ */
+ linkmode_clear_bit(ETHTOOL_LINK_MODE_Pause_BIT, phydev->advertising);
+ linkmode_clear_bit(ETHTOOL_LINK_MODE_Asym_Pause_BIT, phydev->advertising);
+
+ /* Display information about attached PHY device */
+ phy_attached_info(phydev);
+
+ err = register_netdev(netdev);
+ if (err) {
+ dev_err(&pdev->dev, "Failed to register netdev\n");
+ phy_disconnect(phydev);
+ goto out;
+ }
+
+ return 0;
+
+out:
+ mlxbf_gige_gpio_free(priv);
+ mlxbf_gige_mdio_remove(priv);
+ return err;
+}
+
+static int mlxbf_gige_remove(struct platform_device *pdev)
+{
+ struct mlxbf_gige *priv = platform_get_drvdata(pdev);
+
+ unregister_netdev(priv->netdev);
+ phy_disconnect(priv->netdev->phydev);
+ mlxbf_gige_gpio_free(priv);
+ mlxbf_gige_mdio_remove(priv);
+ platform_set_drvdata(pdev, NULL);
+
+ return 0;
+}
+
+static void mlxbf_gige_shutdown(struct platform_device *pdev)
+{
+ struct mlxbf_gige *priv = platform_get_drvdata(pdev);
+
+ writeq(0, priv->base + MLXBF_GIGE_INT_EN);
+ mlxbf_gige_clean_port(priv);
+}
+
+static const struct acpi_device_id __maybe_unused mlxbf_gige_acpi_match[] = {
+ { "MLNXBF17", 0 },
+ {},
+};
+MODULE_DEVICE_TABLE(acpi, mlxbf_gige_acpi_match);
+
+static struct platform_driver mlxbf_gige_driver = {
+ .probe = mlxbf_gige_probe,
+ .remove = mlxbf_gige_remove,
+ .shutdown = mlxbf_gige_shutdown,
+ .driver = {
+ .name = DRV_NAME,
+ .acpi_match_table = ACPI_PTR(mlxbf_gige_acpi_match),
+ },
+};
+
+module_platform_driver(mlxbf_gige_driver);
+
+MODULE_DESCRIPTION("Mellanox BlueField SoC Gigabit Ethernet Driver");
+MODULE_AUTHOR("David Thompson <davthompson@nvidia.com>");
+MODULE_AUTHOR("Asmaa Mnebhi <asmaa@nvidia.com>");
+MODULE_LICENSE("Dual BSD/GPL");
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause
+
+/* MDIO support for Mellanox Gigabit Ethernet driver
+ *
+ * Copyright (C) 2020-2021 NVIDIA CORPORATION & AFFILIATES
+ */
+
+#include <linux/acpi.h>
+#include <linux/bitfield.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/io.h>
+#include <linux/iopoll.h>
+#include <linux/ioport.h>
+#include <linux/irqreturn.h>
+#include <linux/jiffies.h>
+#include <linux/module.h>
+#include <linux/mod_devicetable.h>
+#include <linux/phy.h>
+#include <linux/platform_device.h>
+#include <linux/property.h>
+
+#include "mlxbf_gige.h"
+
+#define MLXBF_GIGE_MDIO_GW_OFFSET 0x0
+#define MLXBF_GIGE_MDIO_CFG_OFFSET 0x4
+
+/* Support clause 22 */
+#define MLXBF_GIGE_MDIO_CL22_ST1 0x1
+#define MLXBF_GIGE_MDIO_CL22_WRITE 0x1
+#define MLXBF_GIGE_MDIO_CL22_READ 0x2
+
+/* Busy bit is set by software and cleared by hardware */
+#define MLXBF_GIGE_MDIO_SET_BUSY 0x1
+
+/* MDIO GW register bits */
+#define MLXBF_GIGE_MDIO_GW_AD_MASK GENMASK(15, 0)
+#define MLXBF_GIGE_MDIO_GW_DEVAD_MASK GENMASK(20, 16)
+#define MLXBF_GIGE_MDIO_GW_PARTAD_MASK GENMASK(25, 21)
+#define MLXBF_GIGE_MDIO_GW_OPCODE_MASK GENMASK(27, 26)
+#define MLXBF_GIGE_MDIO_GW_ST1_MASK GENMASK(28, 28)
+#define MLXBF_GIGE_MDIO_GW_BUSY_MASK GENMASK(30, 30)
+
+/* MDIO config register bits */
+#define MLXBF_GIGE_MDIO_CFG_MDIO_MODE_MASK GENMASK(1, 0)
+#define MLXBF_GIGE_MDIO_CFG_MDIO3_3_MASK GENMASK(2, 2)
+#define MLXBF_GIGE_MDIO_CFG_MDIO_FULL_DRIVE_MASK GENMASK(4, 4)
+#define MLXBF_GIGE_MDIO_CFG_MDC_PERIOD_MASK GENMASK(15, 8)
+#define MLXBF_GIGE_MDIO_CFG_MDIO_IN_SAMP_MASK GENMASK(23, 16)
+#define MLXBF_GIGE_MDIO_CFG_MDIO_OUT_SAMP_MASK GENMASK(31, 24)
+
+/* Formula for encoding the MDIO period. The encoded value is
+ * passed to the MDIO config register.
+ *
+ * mdc_clk = 2*(val + 1)*i1clk
+ *
+ * 400 ns = 2*(val + 1)*(((1/430)*1000) ns)
+ *
+ * val = (((400 * 430 / 1000) / 2) - 1)
+ */
+#define MLXBF_GIGE_I1CLK_MHZ 430
+#define MLXBF_GIGE_MDC_CLK_NS 400
+
+#define MLXBF_GIGE_MDIO_PERIOD (((MLXBF_GIGE_MDC_CLK_NS * MLXBF_GIGE_I1CLK_MHZ / 1000) / 2) - 1)
+
+#define MLXBF_GIGE_MDIO_CFG_VAL (FIELD_PREP(MLXBF_GIGE_MDIO_CFG_MDIO_MODE_MASK, 1) | \
+ FIELD_PREP(MLXBF_GIGE_MDIO_CFG_MDIO3_3_MASK, 1) | \
+ FIELD_PREP(MLXBF_GIGE_MDIO_CFG_MDIO_FULL_DRIVE_MASK, 1) | \
+ FIELD_PREP(MLXBF_GIGE_MDIO_CFG_MDC_PERIOD_MASK, \
+ MLXBF_GIGE_MDIO_PERIOD) | \
+ FIELD_PREP(MLXBF_GIGE_MDIO_CFG_MDIO_IN_SAMP_MASK, 6) | \
+ FIELD_PREP(MLXBF_GIGE_MDIO_CFG_MDIO_OUT_SAMP_MASK, 13))
+
+static u32 mlxbf_gige_mdio_create_cmd(u16 data, int phy_add,
+ int phy_reg, u32 opcode)
+{
+ u32 gw_reg = 0;
+
+ gw_reg |= FIELD_PREP(MLXBF_GIGE_MDIO_GW_AD_MASK, data);
+ gw_reg |= FIELD_PREP(MLXBF_GIGE_MDIO_GW_DEVAD_MASK, phy_reg);
+ gw_reg |= FIELD_PREP(MLXBF_GIGE_MDIO_GW_PARTAD_MASK, phy_add);
+ gw_reg |= FIELD_PREP(MLXBF_GIGE_MDIO_GW_OPCODE_MASK, opcode);
+ gw_reg |= FIELD_PREP(MLXBF_GIGE_MDIO_GW_ST1_MASK,
+ MLXBF_GIGE_MDIO_CL22_ST1);
+ gw_reg |= FIELD_PREP(MLXBF_GIGE_MDIO_GW_BUSY_MASK,
+ MLXBF_GIGE_MDIO_SET_BUSY);
+
+ return gw_reg;
+}
+
+static int mlxbf_gige_mdio_read(struct mii_bus *bus, int phy_add, int phy_reg)
+{
+ struct mlxbf_gige *priv = bus->priv;
+ u32 cmd;
+ int ret;
+ u32 val;
+
+ if (phy_reg & MII_ADDR_C45)
+ return -EOPNOTSUPP;
+
+ /* Send mdio read request */
+ cmd = mlxbf_gige_mdio_create_cmd(0, phy_add, phy_reg, MLXBF_GIGE_MDIO_CL22_READ);
+
+ writel(cmd, priv->mdio_io + MLXBF_GIGE_MDIO_GW_OFFSET);
+
+ ret = readl_poll_timeout_atomic(priv->mdio_io + MLXBF_GIGE_MDIO_GW_OFFSET,
+ val, !(val & MLXBF_GIGE_MDIO_GW_BUSY_MASK), 100, 1000000);
+
+ if (ret) {
+ writel(0, priv->mdio_io + MLXBF_GIGE_MDIO_GW_OFFSET);
+ return ret;
+ }
+
+ ret = readl(priv->mdio_io + MLXBF_GIGE_MDIO_GW_OFFSET);
+ /* Only return ad bits of the gw register */
+ ret &= MLXBF_GIGE_MDIO_GW_AD_MASK;
+
+ return ret;
+}
+
+static int mlxbf_gige_mdio_write(struct mii_bus *bus, int phy_add,
+ int phy_reg, u16 val)
+{
+ struct mlxbf_gige *priv = bus->priv;
+ u32 cmd;
+ int ret;
+ u32 temp;
+
+ if (phy_reg & MII_ADDR_C45)
+ return -EOPNOTSUPP;
+
+ /* Send mdio write request */
+ cmd = mlxbf_gige_mdio_create_cmd(val, phy_add, phy_reg,
+ MLXBF_GIGE_MDIO_CL22_WRITE);
+ writel(cmd, priv->mdio_io + MLXBF_GIGE_MDIO_GW_OFFSET);
+
+ /* If the poll timed out, drop the request */
+ ret = readl_poll_timeout_atomic(priv->mdio_io + MLXBF_GIGE_MDIO_GW_OFFSET,
+ temp, !(temp & MLXBF_GIGE_MDIO_GW_BUSY_MASK), 100, 1000000);
+
+ return ret;
+}
+
+int mlxbf_gige_mdio_probe(struct platform_device *pdev, struct mlxbf_gige *priv)
+{
+ struct device *dev = &pdev->dev;
+ struct resource *res;
+ int ret;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, MLXBF_GIGE_RES_MDIO9);
+ if (!res)
+ return -ENODEV;
+
+ priv->mdio_io = devm_ioremap_resource(dev, res);
+ if (IS_ERR(priv->mdio_io))
+ return PTR_ERR(priv->mdio_io);
+
+ /* Configure mdio parameters */
+ writel(MLXBF_GIGE_MDIO_CFG_VAL,
+ priv->mdio_io + MLXBF_GIGE_MDIO_CFG_OFFSET);
+
+ priv->mdiobus = devm_mdiobus_alloc(dev);
+ if (!priv->mdiobus) {
+ dev_err(dev, "Failed to alloc MDIO bus\n");
+ return -ENOMEM;
+ }
+
+ priv->mdiobus->name = "mlxbf-mdio";
+ priv->mdiobus->read = mlxbf_gige_mdio_read;
+ priv->mdiobus->write = mlxbf_gige_mdio_write;
+ priv->mdiobus->parent = dev;
+ priv->mdiobus->priv = priv;
+ snprintf(priv->mdiobus->id, MII_BUS_ID_SIZE, "%s",
+ dev_name(dev));
+
+ ret = mdiobus_register(priv->mdiobus);
+ if (ret)
+ dev_err(dev, "Failed to register MDIO bus\n");
+
+ return ret;
+}
+
+void mlxbf_gige_mdio_remove(struct mlxbf_gige *priv)
+{
+ mdiobus_unregister(priv->mdiobus);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause */
+
+/* Header file for Mellanox BlueField GigE register defines
+ *
+ * Copyright (C) 2020-2021 NVIDIA CORPORATION & AFFILIATES
+ */
+
+#ifndef __MLXBF_GIGE_REGS_H__
+#define __MLXBF_GIGE_REGS_H__
+
+#define MLXBF_GIGE_STATUS 0x0010
+#define MLXBF_GIGE_STATUS_READY BIT(0)
+#define MLXBF_GIGE_INT_STATUS 0x0028
+#define MLXBF_GIGE_INT_STATUS_RX_RECEIVE_PACKET BIT(0)
+#define MLXBF_GIGE_INT_STATUS_RX_MAC_ERROR BIT(1)
+#define MLXBF_GIGE_INT_STATUS_RX_TRN_ERROR BIT(2)
+#define MLXBF_GIGE_INT_STATUS_SW_ACCESS_ERROR BIT(3)
+#define MLXBF_GIGE_INT_STATUS_SW_CONFIG_ERROR BIT(4)
+#define MLXBF_GIGE_INT_STATUS_TX_PI_CI_EXCEED_WQ_SIZE BIT(5)
+#define MLXBF_GIGE_INT_STATUS_TX_SMALL_FRAME_SIZE BIT(6)
+#define MLXBF_GIGE_INT_STATUS_TX_CHECKSUM_INPUTS BIT(7)
+#define MLXBF_GIGE_INT_STATUS_HW_ACCESS_ERROR BIT(8)
+#define MLXBF_GIGE_INT_EN 0x0030
+#define MLXBF_GIGE_INT_EN_RX_RECEIVE_PACKET BIT(0)
+#define MLXBF_GIGE_INT_EN_RX_MAC_ERROR BIT(1)
+#define MLXBF_GIGE_INT_EN_RX_TRN_ERROR BIT(2)
+#define MLXBF_GIGE_INT_EN_SW_ACCESS_ERROR BIT(3)
+#define MLXBF_GIGE_INT_EN_SW_CONFIG_ERROR BIT(4)
+#define MLXBF_GIGE_INT_EN_TX_PI_CI_EXCEED_WQ_SIZE BIT(5)
+#define MLXBF_GIGE_INT_EN_TX_SMALL_FRAME_SIZE BIT(6)
+#define MLXBF_GIGE_INT_EN_TX_CHECKSUM_INPUTS BIT(7)
+#define MLXBF_GIGE_INT_EN_HW_ACCESS_ERROR BIT(8)
+#define MLXBF_GIGE_INT_MASK 0x0038
+#define MLXBF_GIGE_INT_MASK_RX_RECEIVE_PACKET BIT(0)
+#define MLXBF_GIGE_CONTROL 0x0040
+#define MLXBF_GIGE_CONTROL_PORT_EN BIT(0)
+#define MLXBF_GIGE_CONTROL_MAC_ID_RANGE_EN BIT(1)
+#define MLXBF_GIGE_CONTROL_EN_SPECIFIC_MAC BIT(4)
+#define MLXBF_GIGE_CONTROL_CLEAN_PORT_EN BIT(31)
+#define MLXBF_GIGE_RX_WQ_BASE 0x0200
+#define MLXBF_GIGE_RX_WQE_SIZE_LOG2 0x0208
+#define MLXBF_GIGE_RX_WQE_SIZE_LOG2_RESET_VAL 7
+#define MLXBF_GIGE_RX_CQ_BASE 0x0210
+#define MLXBF_GIGE_TX_WQ_BASE 0x0218
+#define MLXBF_GIGE_TX_WQ_SIZE_LOG2 0x0220
+#define MLXBF_GIGE_TX_WQ_SIZE_LOG2_RESET_VAL 7
+#define MLXBF_GIGE_TX_CI_UPDATE_ADDRESS 0x0228
+#define MLXBF_GIGE_RX_WQE_PI 0x0230
+#define MLXBF_GIGE_TX_PRODUCER_INDEX 0x0238
+#define MLXBF_GIGE_RX_MAC_FILTER 0x0240
+#define MLXBF_GIGE_RX_MAC_FILTER_STRIDE 0x0008
+#define MLXBF_GIGE_RX_DIN_DROP_COUNTER 0x0260
+#define MLXBF_GIGE_TX_CONSUMER_INDEX 0x0310
+#define MLXBF_GIGE_TX_CONTROL 0x0318
+#define MLXBF_GIGE_TX_CONTROL_GRACEFUL_STOP BIT(0)
+#define MLXBF_GIGE_TX_STATUS 0x0388
+#define MLXBF_GIGE_TX_STATUS_DATA_FIFO_FULL BIT(1)
+#define MLXBF_GIGE_RX_MAC_FILTER_DMAC_RANGE_START 0x0520
+#define MLXBF_GIGE_RX_MAC_FILTER_DMAC_RANGE_END 0x0528
+#define MLXBF_GIGE_RX_MAC_FILTER_COUNT_DISC 0x0540
+#define MLXBF_GIGE_RX_MAC_FILTER_COUNT_DISC_EN BIT(0)
+#define MLXBF_GIGE_RX_MAC_FILTER_COUNT_PASS 0x0548
+#define MLXBF_GIGE_RX_MAC_FILTER_COUNT_PASS_EN BIT(0)
+#define MLXBF_GIGE_RX_PASS_COUNTER_ALL 0x0550
+#define MLXBF_GIGE_RX_DISC_COUNTER_ALL 0x0560
+#define MLXBF_GIGE_RX 0x0578
+#define MLXBF_GIGE_RX_STRIP_CRC_EN BIT(1)
+#define MLXBF_GIGE_RX_DMA 0x0580
+#define MLXBF_GIGE_RX_DMA_EN BIT(0)
+#define MLXBF_GIGE_RX_CQE_PACKET_CI 0x05b0
+#define MLXBF_GIGE_MAC_CFG 0x05e8
+
+/* NOTE: MLXBF_GIGE_MAC_CFG is the last defined register offset,
+ * so use that plus size of single register to derive total size
+ */
+#define MLXBF_GIGE_MMIO_REG_SZ (MLXBF_GIGE_MAC_CFG + 8)
+
+#endif /* !defined(__MLXBF_GIGE_REGS_H__) */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause
+
+/* Packet receive logic for Mellanox Gigabit Ethernet driver
+ *
+ * Copyright (C) 2020-2021 NVIDIA CORPORATION & AFFILIATES
+ */
+
+#include <linux/etherdevice.h>
+#include <linux/skbuff.h>
+
+#include "mlxbf_gige.h"
+#include "mlxbf_gige_regs.h"
+
+void mlxbf_gige_set_mac_rx_filter(struct mlxbf_gige *priv,
+ unsigned int index, u64 dmac)
+{
+ void __iomem *base = priv->base;
+ u64 control;
+
+ /* Write destination MAC to specified MAC RX filter */
+ writeq(dmac, base + MLXBF_GIGE_RX_MAC_FILTER +
+ (index * MLXBF_GIGE_RX_MAC_FILTER_STRIDE));
+
+ /* Enable MAC receive filter mask for specified index */
+ control = readq(base + MLXBF_GIGE_CONTROL);
+ control |= (MLXBF_GIGE_CONTROL_EN_SPECIFIC_MAC << index);
+ writeq(control, base + MLXBF_GIGE_CONTROL);
+}
+
+void mlxbf_gige_get_mac_rx_filter(struct mlxbf_gige *priv,
+ unsigned int index, u64 *dmac)
+{
+ void __iomem *base = priv->base;
+
+ /* Read destination MAC from specified MAC RX filter */
+ *dmac = readq(base + MLXBF_GIGE_RX_MAC_FILTER +
+ (index * MLXBF_GIGE_RX_MAC_FILTER_STRIDE));
+}
+
+void mlxbf_gige_enable_promisc(struct mlxbf_gige *priv)
+{
+ void __iomem *base = priv->base;
+ u64 control;
+ u64 end_mac;
+
+ /* Enable MAC_ID_RANGE match functionality */
+ control = readq(base + MLXBF_GIGE_CONTROL);
+ control |= MLXBF_GIGE_CONTROL_MAC_ID_RANGE_EN;
+ writeq(control, base + MLXBF_GIGE_CONTROL);
+
+ /* Set start of destination MAC range check to 0 */
+ writeq(0, base + MLXBF_GIGE_RX_MAC_FILTER_DMAC_RANGE_START);
+
+ /* Set end of destination MAC range check to all FFs */
+ end_mac = BCAST_MAC_ADDR;
+ writeq(end_mac, base + MLXBF_GIGE_RX_MAC_FILTER_DMAC_RANGE_END);
+}
+
+void mlxbf_gige_disable_promisc(struct mlxbf_gige *priv)
+{
+ void __iomem *base = priv->base;
+ u64 control;
+
+ /* Disable MAC_ID_RANGE match functionality */
+ control = readq(base + MLXBF_GIGE_CONTROL);
+ control &= ~MLXBF_GIGE_CONTROL_MAC_ID_RANGE_EN;
+ writeq(control, base + MLXBF_GIGE_CONTROL);
+
+ /* NOTE: no need to change DMAC_RANGE_START or END;
+ * those values are ignored since MAC_ID_RANGE_EN=0
+ */
+}
+
+/* Receive Initialization
+ * 1) Configures RX MAC filters via MMIO registers
+ * 2) Allocates RX WQE array using coherent DMA mapping
+ * 3) Initializes each element of RX WQE array with a receive
+ * buffer pointer (also using coherent DMA mapping)
+ * 4) Allocates RX CQE array using coherent DMA mapping
+ * 5) Completes other misc receive initialization
+ */
+int mlxbf_gige_rx_init(struct mlxbf_gige *priv)
+{
+ size_t wq_size, cq_size;
+ dma_addr_t *rx_wqe_ptr;
+ dma_addr_t rx_buf_dma;
+ u64 data;
+ int i, j;
+
+ /* Configure MAC RX filter #0 to allow RX of broadcast pkts */
+ mlxbf_gige_set_mac_rx_filter(priv, MLXBF_GIGE_BCAST_MAC_FILTER_IDX,
+ BCAST_MAC_ADDR);
+
+ wq_size = MLXBF_GIGE_RX_WQE_SZ * priv->rx_q_entries;
+ priv->rx_wqe_base = dma_alloc_coherent(priv->dev, wq_size,
+ &priv->rx_wqe_base_dma,
+ GFP_KERNEL);
+ if (!priv->rx_wqe_base)
+ return -ENOMEM;
+
+ /* Initialize 'rx_wqe_ptr' to point to first RX WQE in array
+ * Each RX WQE is simply a receive buffer pointer, so walk
+ * the entire array, allocating a 2KB buffer for each element
+ */
+ rx_wqe_ptr = priv->rx_wqe_base;
+
+ for (i = 0; i < priv->rx_q_entries; i++) {
+ priv->rx_skb[i] = mlxbf_gige_alloc_skb(priv, MLXBF_GIGE_DEFAULT_BUF_SZ,
+ &rx_buf_dma, DMA_FROM_DEVICE);
+ if (!priv->rx_skb[i])
+ goto free_wqe_and_skb;
+ *rx_wqe_ptr++ = rx_buf_dma;
+ }
+
+ /* Write RX WQE base address into MMIO reg */
+ writeq(priv->rx_wqe_base_dma, priv->base + MLXBF_GIGE_RX_WQ_BASE);
+
+ cq_size = MLXBF_GIGE_RX_CQE_SZ * priv->rx_q_entries;
+ priv->rx_cqe_base = dma_alloc_coherent(priv->dev, cq_size,
+ &priv->rx_cqe_base_dma,
+ GFP_KERNEL);
+ if (!priv->rx_cqe_base)
+ goto free_wqe_and_skb;
+
+ for (i = 0; i < priv->rx_q_entries; i++)
+ priv->rx_cqe_base[i] |= MLXBF_GIGE_RX_CQE_VALID_MASK;
+
+ /* Write RX CQE base address into MMIO reg */
+ writeq(priv->rx_cqe_base_dma, priv->base + MLXBF_GIGE_RX_CQ_BASE);
+
+ /* Write RX_WQE_PI with current number of replenished buffers */
+ writeq(priv->rx_q_entries, priv->base + MLXBF_GIGE_RX_WQE_PI);
+
+ /* Enable removal of CRC during RX */
+ data = readq(priv->base + MLXBF_GIGE_RX);
+ data |= MLXBF_GIGE_RX_STRIP_CRC_EN;
+ writeq(data, priv->base + MLXBF_GIGE_RX);
+
+ /* Enable RX MAC filter pass and discard counters */
+ writeq(MLXBF_GIGE_RX_MAC_FILTER_COUNT_DISC_EN,
+ priv->base + MLXBF_GIGE_RX_MAC_FILTER_COUNT_DISC);
+ writeq(MLXBF_GIGE_RX_MAC_FILTER_COUNT_PASS_EN,
+ priv->base + MLXBF_GIGE_RX_MAC_FILTER_COUNT_PASS);
+
+ /* Clear MLXBF_GIGE_INT_MASK 'receive pkt' bit to
+ * indicate readiness to receive interrupts
+ */
+ data = readq(priv->base + MLXBF_GIGE_INT_MASK);
+ data &= ~MLXBF_GIGE_INT_MASK_RX_RECEIVE_PACKET;
+ writeq(data, priv->base + MLXBF_GIGE_INT_MASK);
+
+ /* Enable RX DMA to write new packets to memory */
+ data = readq(priv->base + MLXBF_GIGE_RX_DMA);
+ data |= MLXBF_GIGE_RX_DMA_EN;
+ writeq(data, priv->base + MLXBF_GIGE_RX_DMA);
+
+ writeq(ilog2(priv->rx_q_entries),
+ priv->base + MLXBF_GIGE_RX_WQE_SIZE_LOG2);
+
+ return 0;
+
+free_wqe_and_skb:
+ rx_wqe_ptr = priv->rx_wqe_base;
+ for (j = 0; j < i; j++) {
+ dma_unmap_single(priv->dev, *rx_wqe_ptr,
+ MLXBF_GIGE_DEFAULT_BUF_SZ, DMA_FROM_DEVICE);
+ dev_kfree_skb(priv->rx_skb[j]);
+ rx_wqe_ptr++;
+ }
+ dma_free_coherent(priv->dev, wq_size,
+ priv->rx_wqe_base, priv->rx_wqe_base_dma);
+ return -ENOMEM;
+}
+
+/* Receive Deinitialization
+ * This routine will free allocations done by mlxbf_gige_rx_init(),
+ * namely the RX WQE and RX CQE arrays, as well as all RX buffers
+ */
+void mlxbf_gige_rx_deinit(struct mlxbf_gige *priv)
+{
+ dma_addr_t *rx_wqe_ptr;
+ size_t size;
+ u64 data;
+ int i;
+
+ /* Disable RX DMA to prevent packet transfers to memory */
+ data = readq(priv->base + MLXBF_GIGE_RX_DMA);
+ data &= ~MLXBF_GIGE_RX_DMA_EN;
+ writeq(data, priv->base + MLXBF_GIGE_RX_DMA);
+
+ rx_wqe_ptr = priv->rx_wqe_base;
+
+ for (i = 0; i < priv->rx_q_entries; i++) {
+ dma_unmap_single(priv->dev, *rx_wqe_ptr, MLXBF_GIGE_DEFAULT_BUF_SZ,
+ DMA_FROM_DEVICE);
+ dev_kfree_skb(priv->rx_skb[i]);
+ rx_wqe_ptr++;
+ }
+
+ size = MLXBF_GIGE_RX_WQE_SZ * priv->rx_q_entries;
+ dma_free_coherent(priv->dev, size,
+ priv->rx_wqe_base, priv->rx_wqe_base_dma);
+
+ size = MLXBF_GIGE_RX_CQE_SZ * priv->rx_q_entries;
+ dma_free_coherent(priv->dev, size,
+ priv->rx_cqe_base, priv->rx_cqe_base_dma);
+
+ priv->rx_wqe_base = NULL;
+ priv->rx_wqe_base_dma = 0;
+ priv->rx_cqe_base = NULL;
+ priv->rx_cqe_base_dma = 0;
+ writeq(0, priv->base + MLXBF_GIGE_RX_WQ_BASE);
+ writeq(0, priv->base + MLXBF_GIGE_RX_CQ_BASE);
+}
+
+static bool mlxbf_gige_rx_packet(struct mlxbf_gige *priv, int *rx_pkts)
+{
+ struct net_device *netdev = priv->netdev;
+ struct sk_buff *skb = NULL, *rx_skb;
+ u16 rx_pi_rem, rx_ci_rem;
+ dma_addr_t *rx_wqe_addr;
+ dma_addr_t rx_buf_dma;
+ u64 *rx_cqe_addr;
+ u64 datalen;
+ u64 rx_cqe;
+ u16 rx_ci;
+ u16 rx_pi;
+
+ /* Index into RX buffer array is rx_pi w/wrap based on RX_CQE_SIZE */
+ rx_pi = readq(priv->base + MLXBF_GIGE_RX_WQE_PI);
+ rx_pi_rem = rx_pi % priv->rx_q_entries;
+
+ rx_wqe_addr = priv->rx_wqe_base + rx_pi_rem;
+ rx_cqe_addr = priv->rx_cqe_base + rx_pi_rem;
+ rx_cqe = *rx_cqe_addr;
+
+ if ((!!(rx_cqe & MLXBF_GIGE_RX_CQE_VALID_MASK)) != priv->valid_polarity)
+ return false;
+
+ if ((rx_cqe & MLXBF_GIGE_RX_CQE_PKT_STATUS_MASK) == 0) {
+ /* Packet is OK, increment stats */
+ datalen = rx_cqe & MLXBF_GIGE_RX_CQE_PKT_LEN_MASK;
+ netdev->stats.rx_packets++;
+ netdev->stats.rx_bytes += datalen;
+
+ skb = priv->rx_skb[rx_pi_rem];
+
+ skb_put(skb, datalen);
+
+ skb->ip_summed = CHECKSUM_NONE; /* device did not checksum packet */
+
+ skb->protocol = eth_type_trans(skb, netdev);
+
+ /* Alloc another RX SKB for this same index */
+ rx_skb = mlxbf_gige_alloc_skb(priv, MLXBF_GIGE_DEFAULT_BUF_SZ,
+ &rx_buf_dma, DMA_FROM_DEVICE);
+ if (!rx_skb)
+ return false;
+ priv->rx_skb[rx_pi_rem] = rx_skb;
+ dma_unmap_single(priv->dev, *rx_wqe_addr,
+ MLXBF_GIGE_DEFAULT_BUF_SZ, DMA_FROM_DEVICE);
+ *rx_wqe_addr = rx_buf_dma;
+ } else if (rx_cqe & MLXBF_GIGE_RX_CQE_PKT_STATUS_MAC_ERR) {
+ priv->stats.rx_mac_errors++;
+ } else if (rx_cqe & MLXBF_GIGE_RX_CQE_PKT_STATUS_TRUNCATED) {
+ priv->stats.rx_truncate_errors++;
+ }
+
+ /* Let hardware know we've replenished one buffer */
+ rx_pi++;
+
+ /* Ensure completion of all writes before notifying HW of replenish */
+ wmb();
+ writeq(rx_pi, priv->base + MLXBF_GIGE_RX_WQE_PI);
+
+ (*rx_pkts)++;
+
+ rx_pi_rem = rx_pi % priv->rx_q_entries;
+ if (rx_pi_rem == 0)
+ priv->valid_polarity ^= 1;
+ rx_ci = readq(priv->base + MLXBF_GIGE_RX_CQE_PACKET_CI);
+ rx_ci_rem = rx_ci % priv->rx_q_entries;
+
+ if (skb)
+ netif_receive_skb(skb);
+
+ return rx_pi_rem != rx_ci_rem;
+}
+
+/* Driver poll() function called by NAPI infrastructure */
+int mlxbf_gige_poll(struct napi_struct *napi, int budget)
+{
+ struct mlxbf_gige *priv;
+ bool remaining_pkts;
+ int work_done = 0;
+ u64 data;
+
+ priv = container_of(napi, struct mlxbf_gige, napi);
+
+ mlxbf_gige_handle_tx_complete(priv);
+
+ do {
+ remaining_pkts = mlxbf_gige_rx_packet(priv, &work_done);
+ } while (remaining_pkts && work_done < budget);
+
+ /* If amount of work done < budget, turn off NAPI polling
+ * via napi_complete_done(napi, work_done) and then
+ * re-enable interrupts.
+ */
+ if (work_done < budget && napi_complete_done(napi, work_done)) {
+ /* Clear MLXBF_GIGE_INT_MASK 'receive pkt' bit to
+ * indicate receive readiness
+ */
+ data = readq(priv->base + MLXBF_GIGE_INT_MASK);
+ data &= ~MLXBF_GIGE_INT_MASK_RX_RECEIVE_PACKET;
+ writeq(data, priv->base + MLXBF_GIGE_INT_MASK);
+ }
+
+ return work_done;
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause
+
+/* Packet transmit logic for Mellanox Gigabit Ethernet driver
+ *
+ * Copyright (C) 2020-2021 NVIDIA CORPORATION & AFFILIATES
+ */
+
+#include <linux/skbuff.h>
+
+#include "mlxbf_gige.h"
+#include "mlxbf_gige_regs.h"
+
+/* Transmit Initialization
+ * 1) Allocates TX WQE array using coherent DMA mapping
+ * 2) Allocates TX completion counter using coherent DMA mapping
+ */
+int mlxbf_gige_tx_init(struct mlxbf_gige *priv)
+{
+ size_t size;
+
+ size = MLXBF_GIGE_TX_WQE_SZ * priv->tx_q_entries;
+ priv->tx_wqe_base = dma_alloc_coherent(priv->dev, size,
+ &priv->tx_wqe_base_dma,
+ GFP_KERNEL);
+ if (!priv->tx_wqe_base)
+ return -ENOMEM;
+
+ priv->tx_wqe_next = priv->tx_wqe_base;
+
+ /* Write TX WQE base address into MMIO reg */
+ writeq(priv->tx_wqe_base_dma, priv->base + MLXBF_GIGE_TX_WQ_BASE);
+
+ /* Allocate address for TX completion count */
+ priv->tx_cc = dma_alloc_coherent(priv->dev, MLXBF_GIGE_TX_CC_SZ,
+ &priv->tx_cc_dma, GFP_KERNEL);
+ if (!priv->tx_cc) {
+ dma_free_coherent(priv->dev, size,
+ priv->tx_wqe_base, priv->tx_wqe_base_dma);
+ return -ENOMEM;
+ }
+
+ /* Write TX CC base address into MMIO reg */
+ writeq(priv->tx_cc_dma, priv->base + MLXBF_GIGE_TX_CI_UPDATE_ADDRESS);
+
+ writeq(ilog2(priv->tx_q_entries),
+ priv->base + MLXBF_GIGE_TX_WQ_SIZE_LOG2);
+
+ priv->prev_tx_ci = 0;
+ priv->tx_pi = 0;
+
+ return 0;
+}
+
+/* Transmit Deinitialization
+ * This routine will free allocations done by mlxbf_gige_tx_init(),
+ * namely the TX WQE array and the TX completion counter
+ */
+void mlxbf_gige_tx_deinit(struct mlxbf_gige *priv)
+{
+ u64 *tx_wqe_addr;
+ size_t size;
+ int i;
+
+ tx_wqe_addr = priv->tx_wqe_base;
+
+ for (i = 0; i < priv->tx_q_entries; i++) {
+ if (priv->tx_skb[i]) {
+ dma_unmap_single(priv->dev, *tx_wqe_addr,
+ priv->tx_skb[i]->len, DMA_TO_DEVICE);
+ dev_kfree_skb(priv->tx_skb[i]);
+ priv->tx_skb[i] = NULL;
+ }
+ tx_wqe_addr += 2;
+ }
+
+ size = MLXBF_GIGE_TX_WQE_SZ * priv->tx_q_entries;
+ dma_free_coherent(priv->dev, size,
+ priv->tx_wqe_base, priv->tx_wqe_base_dma);
+
+ dma_free_coherent(priv->dev, MLXBF_GIGE_TX_CC_SZ,
+ priv->tx_cc, priv->tx_cc_dma);
+
+ priv->tx_wqe_base = NULL;
+ priv->tx_wqe_base_dma = 0;
+ priv->tx_cc = NULL;
+ priv->tx_cc_dma = 0;
+ priv->tx_wqe_next = NULL;
+ writeq(0, priv->base + MLXBF_GIGE_TX_WQ_BASE);
+ writeq(0, priv->base + MLXBF_GIGE_TX_CI_UPDATE_ADDRESS);
+}
+
+/* Function that returns status of TX ring:
+ * 0: TX ring is full, i.e. there are no
+ * available un-used entries in TX ring.
+ * non-null: TX ring is not full, i.e. there are
+ * some available entries in TX ring.
+ * The non-null value is a measure of
+ * how many TX entries are available, but
+ * it is not the exact number of available
+ * entries (see below).
+ *
+ * The algorithm makes the assumption that if
+ * (prev_tx_ci == tx_pi) then the TX ring is empty.
+ * An empty ring actually has (tx_q_entries-1)
+ * entries, which allows the algorithm to differentiate
+ * the case of an empty ring vs. a full ring.
+ */
+static u16 mlxbf_gige_tx_buffs_avail(struct mlxbf_gige *priv)
+{
+ unsigned long flags;
+ u16 avail;
+
+ spin_lock_irqsave(&priv->lock, flags);
+
+ if (priv->prev_tx_ci == priv->tx_pi)
+ avail = priv->tx_q_entries - 1;
+ else
+ avail = ((priv->tx_q_entries + priv->prev_tx_ci - priv->tx_pi)
+ % priv->tx_q_entries) - 1;
+
+ spin_unlock_irqrestore(&priv->lock, flags);
+
+ return avail;
+}
+
+bool mlxbf_gige_handle_tx_complete(struct mlxbf_gige *priv)
+{
+ struct net_device_stats *stats;
+ u16 tx_wqe_index;
+ u64 *tx_wqe_addr;
+ u64 tx_status;
+ u16 tx_ci;
+
+ tx_status = readq(priv->base + MLXBF_GIGE_TX_STATUS);
+ if (tx_status & MLXBF_GIGE_TX_STATUS_DATA_FIFO_FULL)
+ priv->stats.tx_fifo_full++;
+ tx_ci = readq(priv->base + MLXBF_GIGE_TX_CONSUMER_INDEX);
+ stats = &priv->netdev->stats;
+
+ /* Transmit completion logic needs to loop until the completion
+ * index (in SW) equals TX consumer index (from HW). These
+ * parameters are unsigned 16-bit values and the wrap case needs
+ * to be supported, that is TX consumer index wrapped from 0xFFFF
+ * to 0 while TX completion index is still < 0xFFFF.
+ */
+ for (; priv->prev_tx_ci != tx_ci; priv->prev_tx_ci++) {
+ tx_wqe_index = priv->prev_tx_ci % priv->tx_q_entries;
+ /* Each TX WQE is 16 bytes. The 8 MSB store the 2KB TX
+ * buffer address and the 8 LSB contain information
+ * about the TX WQE.
+ */
+ tx_wqe_addr = priv->tx_wqe_base +
+ (tx_wqe_index * MLXBF_GIGE_TX_WQE_SZ_QWORDS);
+
+ stats->tx_packets++;
+ stats->tx_bytes += MLXBF_GIGE_TX_WQE_PKT_LEN(tx_wqe_addr);
+
+ dma_unmap_single(priv->dev, *tx_wqe_addr,
+ priv->tx_skb[tx_wqe_index]->len, DMA_TO_DEVICE);
+ dev_consume_skb_any(priv->tx_skb[tx_wqe_index]);
+ priv->tx_skb[tx_wqe_index] = NULL;
+
+ /* Ensure completion of updates across all cores */
+ mb();
+ }
+
+ /* Since the TX ring was likely just drained, check if TX queue
+ * had previously been stopped and now that there are TX buffers
+ * available the TX queue can be awakened.
+ */
+ if (netif_queue_stopped(priv->netdev) &&
+ mlxbf_gige_tx_buffs_avail(priv))
+ netif_wake_queue(priv->netdev);
+
+ return true;
+}
+
+/* Function to advance the tx_wqe_next pointer to next TX WQE */
+void mlxbf_gige_update_tx_wqe_next(struct mlxbf_gige *priv)
+{
+ /* Advance tx_wqe_next pointer */
+ priv->tx_wqe_next += MLXBF_GIGE_TX_WQE_SZ_QWORDS;
+
+ /* Check if 'next' pointer is beyond end of TX ring */
+ /* If so, set 'next' back to 'base' pointer of ring */
+ if (priv->tx_wqe_next == (priv->tx_wqe_base +
+ (priv->tx_q_entries * MLXBF_GIGE_TX_WQE_SZ_QWORDS)))
+ priv->tx_wqe_next = priv->tx_wqe_base;
+}
+
+netdev_tx_t mlxbf_gige_start_xmit(struct sk_buff *skb,
+ struct net_device *netdev)
+{
+ struct mlxbf_gige *priv = netdev_priv(netdev);
+ long buff_addr, start_dma_page, end_dma_page;
+ struct sk_buff *tx_skb;
+ dma_addr_t tx_buf_dma;
+ unsigned long flags;
+ u64 *tx_wqe_addr;
+ u64 word2;
+
+ /* If needed, linearize TX SKB as hardware DMA expects this */
+ if (skb->len > MLXBF_GIGE_DEFAULT_BUF_SZ || skb_linearize(skb)) {
+ dev_kfree_skb(skb);
+ netdev->stats.tx_dropped++;
+ return NETDEV_TX_OK;
+ }
+
+ buff_addr = (long)skb->data;
+ start_dma_page = buff_addr >> MLXBF_GIGE_DMA_PAGE_SHIFT;
+ end_dma_page = (buff_addr + skb->len - 1) >> MLXBF_GIGE_DMA_PAGE_SHIFT;
+
+ /* Verify that payload pointer and data length of SKB to be
+ * transmitted does not violate the hardware DMA limitation.
+ */
+ if (start_dma_page != end_dma_page) {
+ /* DMA operation would fail as-is, alloc new aligned SKB */
+ tx_skb = mlxbf_gige_alloc_skb(priv, skb->len,
+ &tx_buf_dma, DMA_TO_DEVICE);
+ if (!tx_skb) {
+ /* Free original skb, could not alloc new aligned SKB */
+ dev_kfree_skb(skb);
+ netdev->stats.tx_dropped++;
+ return NETDEV_TX_OK;
+ }
+
+ skb_put_data(tx_skb, skb->data, skb->len);
+
+ /* Free the original SKB */
+ dev_kfree_skb(skb);
+ } else {
+ tx_skb = skb;
+ tx_buf_dma = dma_map_single(priv->dev, skb->data,
+ skb->len, DMA_TO_DEVICE);
+ if (dma_mapping_error(priv->dev, tx_buf_dma)) {
+ dev_kfree_skb(skb);
+ netdev->stats.tx_dropped++;
+ return NETDEV_TX_OK;
+ }
+ }
+
+ /* Get address of TX WQE */
+ tx_wqe_addr = priv->tx_wqe_next;
+
+ mlxbf_gige_update_tx_wqe_next(priv);
+
+ /* Put PA of buffer address into first 64-bit word of TX WQE */
+ *tx_wqe_addr = tx_buf_dma;
+
+ /* Set TX WQE pkt_len appropriately
+ * NOTE: GigE silicon will automatically pad up to
+ * minimum packet length if needed.
+ */
+ word2 = tx_skb->len & MLXBF_GIGE_TX_WQE_PKT_LEN_MASK;
+
+ /* Write entire 2nd word of TX WQE */
+ *(tx_wqe_addr + 1) = word2;
+
+ spin_lock_irqsave(&priv->lock, flags);
+ priv->tx_skb[priv->tx_pi % priv->tx_q_entries] = tx_skb;
+ priv->tx_pi++;
+ spin_unlock_irqrestore(&priv->lock, flags);
+
+ if (!netdev_xmit_more()) {
+ /* Create memory barrier before write to TX PI */
+ wmb();
+ writeq(priv->tx_pi, priv->base + MLXBF_GIGE_TX_PRODUCER_INDEX);
+ }
+
+ /* Check if the last TX entry was just used */
+ if (!mlxbf_gige_tx_buffs_avail(priv)) {
+ /* TX ring is full, inform stack */
+ netif_stop_queue(netdev);
+
+ /* Since there is no separate "TX complete" interrupt, need
+ * to explicitly schedule NAPI poll. This will trigger logic
+ * which processes TX completions, and will hopefully drain
+ * the TX ring allowing the TX queue to be awakened.
+ */
+ napi_schedule(&priv->napi);
+ }
+
+ return NETDEV_TX_OK;
+}
To compile this driver as a module, choose M here: the
module will be called mlxsw_i2c.
-config MLXSW_SWITCHIB
- tristate "Mellanox Technologies SwitchIB and SwitchIB-2 support"
- depends on MLXSW_CORE && MLXSW_PCI && NET_SWITCHDEV
- default m
- help
- This driver supports Mellanox Technologies SwitchIB and SwitchIB-2
- Infiniband Switch ASICs.
-
- To compile this driver as a module, choose M here: the
- module will be called mlxsw_switchib.
-
-config MLXSW_SWITCHX2
- tristate "Mellanox Technologies SwitchX-2 support"
- depends on MLXSW_CORE && MLXSW_PCI && NET_SWITCHDEV
- default m
- help
- This driver supports Mellanox Technologies SwitchX-2 Ethernet
- Switch ASICs.
-
- To compile this driver as a module, choose M here: the
- module will be called mlxsw_switchx2.
-
config MLXSW_SPECTRUM
tristate "Mellanox Technologies Spectrum family support"
depends on MLXSW_CORE && MLXSW_PCI && NET_SWITCHDEV && VLAN_8021Q
mlxsw_pci-objs := pci.o
obj-$(CONFIG_MLXSW_I2C) += mlxsw_i2c.o
mlxsw_i2c-objs := i2c.o
-obj-$(CONFIG_MLXSW_SWITCHIB) += mlxsw_switchib.o
-mlxsw_switchib-objs := switchib.o
-obj-$(CONFIG_MLXSW_SWITCHX2) += mlxsw_switchx2.o
-mlxsw_switchx2-objs := switchx2.o
obj-$(CONFIG_MLXSW_SPECTRUM) += mlxsw_spectrum.o
mlxsw_spectrum-objs := spectrum.o spectrum_buffers.o \
spectrum_switchdev.o spectrum_router.o \
struct sk_buff *skb;
int err;
- skb = skb_copy(trans->tx_skb, GFP_KERNEL);
+ skb = skb_clone(trans->tx_skb, GFP_KERNEL);
if (!skb)
return -ENOMEM;
if (err)
return err;
- err = devlink_info_version_fixed_put(req, "fw.psid", fw_info_psid);
+ err = devlink_info_version_fixed_put(req,
+ DEVLINK_INFO_VERSION_GENERIC_FW_PSID,
+ fw_info_psid);
if (err)
return err;
if (err)
return err;
- return 0;
+ return devlink_info_version_running_put(req,
+ DEVLINK_INFO_VERSION_GENERIC_FW,
+ buf);
}
static int
#include <linux/kernel.h>
#include <linux/err.h>
+#include <linux/ethtool.h>
#include <linux/sfp.h>
#include "core.h"
static int mlxsw_env_validate_cable_ident(struct mlxsw_core *core, int id,
bool *qsfp, bool *cmis)
{
- char eeprom_tmp[MLXSW_REG_MCIA_EEPROM_SIZE];
char mcia_pl[MLXSW_REG_MCIA_LEN];
+ char *eeprom_tmp;
u8 ident;
int err;
err = mlxsw_reg_query(core, MLXSW_REG(mcia), mcia_pl);
if (err)
return err;
- mlxsw_reg_mcia_eeprom_memcpy_from(mcia_pl, eeprom_tmp);
+ eeprom_tmp = mlxsw_reg_mcia_eeprom_data(mcia_pl);
ident = eeprom_tmp[0];
*cmis = false;
switch (ident) {
u16 offset, u16 size, void *data,
bool qsfp, unsigned int *p_read_size)
{
- char eeprom_tmp[MLXSW_REG_MCIA_EEPROM_SIZE];
char mcia_pl[MLXSW_REG_MCIA_LEN];
+ char *eeprom_tmp;
u16 i2c_addr;
u8 page = 0;
int status;
if (status)
return -EIO;
- mlxsw_reg_mcia_eeprom_memcpy_from(mcia_pl, eeprom_tmp);
+ eeprom_tmp = mlxsw_reg_mcia_eeprom_data(mcia_pl);
memcpy(data, eeprom_tmp, size);
*p_read_size = size;
int mlxsw_env_module_temp_thresholds_get(struct mlxsw_core *core, int module,
int off, int *temp)
{
- char eeprom_tmp[MLXSW_REG_MCIA_EEPROM_SIZE];
+ unsigned int module_temp, module_crit, module_emerg;
union {
u8 buf[MLXSW_REG_MCIA_TH_ITEM_SIZE];
u16 temp;
} temp_thresh;
char mcia_pl[MLXSW_REG_MCIA_LEN] = {0};
char mtmp_pl[MLXSW_REG_MTMP_LEN];
- unsigned int module_temp;
+ char *eeprom_tmp;
bool qsfp, cmis;
int page;
int err;
err = mlxsw_reg_query(core, MLXSW_REG(mtmp), mtmp_pl);
if (err)
return err;
- mlxsw_reg_mtmp_unpack(mtmp_pl, &module_temp, NULL, NULL);
+ mlxsw_reg_mtmp_unpack(mtmp_pl, &module_temp, NULL, &module_crit,
+ &module_emerg, NULL);
if (!module_temp) {
*temp = 0;
return 0;
}
+ /* Validate if threshold reading is available through MTMP register,
+ * otherwise fallback to read through MCIA.
+ */
+ if (module_emerg) {
+ *temp = off == SFP_TEMP_HIGH_WARN ? module_crit : module_emerg;
+ return 0;
+ }
+
/* Read Free Side Device Temperature Thresholds from page 03h
* (MSB at lower byte address).
* Bytes:
if (err)
return err;
- mlxsw_reg_mcia_eeprom_memcpy_from(mcia_pl, eeprom_tmp);
+ eeprom_tmp = mlxsw_reg_mcia_eeprom_data(mcia_pl);
memcpy(temp_thresh.buf, eeprom_tmp, MLXSW_REG_MCIA_TH_ITEM_SIZE);
*temp = temp_thresh.temp * 1000;
}
EXPORT_SYMBOL(mlxsw_env_get_module_eeprom);
+static int mlxsw_env_mcia_status_process(const char *mcia_pl,
+ struct netlink_ext_ack *extack)
+{
+ u8 status = mlxsw_reg_mcia_status_get(mcia_pl);
+
+ switch (status) {
+ case MLXSW_REG_MCIA_STATUS_GOOD:
+ return 0;
+ case MLXSW_REG_MCIA_STATUS_NO_EEPROM_MODULE:
+ NL_SET_ERR_MSG_MOD(extack, "No response from module's EEPROM");
+ return -EIO;
+ case MLXSW_REG_MCIA_STATUS_MODULE_NOT_SUPPORTED:
+ NL_SET_ERR_MSG_MOD(extack, "Module type not supported by the device");
+ return -EOPNOTSUPP;
+ case MLXSW_REG_MCIA_STATUS_MODULE_NOT_CONNECTED:
+ NL_SET_ERR_MSG_MOD(extack, "No module present indication");
+ return -EIO;
+ case MLXSW_REG_MCIA_STATUS_I2C_ERROR:
+ NL_SET_ERR_MSG_MOD(extack, "Error occurred while trying to access module's EEPROM using I2C");
+ return -EIO;
+ case MLXSW_REG_MCIA_STATUS_MODULE_DISABLED:
+ NL_SET_ERR_MSG_MOD(extack, "Module is disabled");
+ return -EIO;
+ default:
+ NL_SET_ERR_MSG_MOD(extack, "Unknown error");
+ return -EIO;
+ }
+}
+
+int
+mlxsw_env_get_module_eeprom_by_page(struct mlxsw_core *mlxsw_core, u8 module,
+ const struct ethtool_module_eeprom *page,
+ struct netlink_ext_ack *extack)
+{
+ u32 bytes_read = 0;
+ u16 device_addr;
+
+ /* Offset cannot be larger than 2 * ETH_MODULE_EEPROM_PAGE_LEN */
+ device_addr = page->offset;
+
+ while (bytes_read < page->length) {
+ char mcia_pl[MLXSW_REG_MCIA_LEN];
+ char *eeprom_tmp;
+ u8 size;
+ int err;
+
+ size = min_t(u8, page->length - bytes_read,
+ MLXSW_REG_MCIA_EEPROM_SIZE);
+
+ mlxsw_reg_mcia_pack(mcia_pl, module, 0, page->page,
+ device_addr + bytes_read, size,
+ page->i2c_address);
+ mlxsw_reg_mcia_bank_number_set(mcia_pl, page->bank);
+
+ err = mlxsw_reg_query(mlxsw_core, MLXSW_REG(mcia), mcia_pl);
+ if (err) {
+ NL_SET_ERR_MSG_MOD(extack, "Failed to access module's EEPROM");
+ return err;
+ }
+
+ err = mlxsw_env_mcia_status_process(mcia_pl, extack);
+ if (err)
+ return err;
+
+ eeprom_tmp = mlxsw_reg_mcia_eeprom_data(mcia_pl);
+ memcpy(page->data + bytes_read, eeprom_tmp, size);
+ bytes_read += size;
+ }
+
+ return bytes_read;
+}
+EXPORT_SYMBOL(mlxsw_env_get_module_eeprom_by_page);
+
static int mlxsw_env_module_has_temp_sensor(struct mlxsw_core *mlxsw_core,
u8 module,
bool *p_has_temp_sensor)
#ifndef _MLXSW_CORE_ENV_H
#define _MLXSW_CORE_ENV_H
+#include <linux/ethtool.h>
+
struct ethtool_modinfo;
struct ethtool_eeprom;
struct ethtool_eeprom *ee, u8 *data);
int
+mlxsw_env_get_module_eeprom_by_page(struct mlxsw_core *mlxsw_core, u8 module,
+ const struct ethtool_module_eeprom *page,
+ struct netlink_ext_ack *extack);
+
+int
mlxsw_env_module_overheat_counter_get(struct mlxsw_core *mlxsw_core, u8 module,
u64 *p_counter);
int mlxsw_env_init(struct mlxsw_core *core, struct mlxsw_env **p_env);
dev_err(mlxsw_hwmon->bus_info->dev, "Failed to query temp sensor\n");
return err;
}
- mlxsw_reg_mtmp_unpack(mtmp_pl, &temp, NULL, NULL);
+ mlxsw_reg_mtmp_unpack(mtmp_pl, &temp, NULL, NULL, NULL, NULL);
return sprintf(buf, "%d\n", temp);
}
dev_err(mlxsw_hwmon->bus_info->dev, "Failed to query temp sensor\n");
return err;
}
- mlxsw_reg_mtmp_unpack(mtmp_pl, NULL, &temp_max, NULL);
+ mlxsw_reg_mtmp_unpack(mtmp_pl, NULL, &temp_max, NULL, NULL, NULL);
return sprintf(buf, "%d\n", temp_max);
}
dev_err(dev, "Failed to query module temperature\n");
return err;
}
- mlxsw_reg_mtmp_unpack(mtmp_pl, p_temp, NULL, NULL);
+ mlxsw_reg_mtmp_unpack(mtmp_pl, p_temp, NULL, NULL, NULL, NULL);
return 0;
}
static int
mlxsw_thermal_module_trips_update(struct device *dev, struct mlxsw_core *core,
- struct mlxsw_thermal_module *tz)
+ struct mlxsw_thermal_module *tz,
+ int crit_temp, int emerg_temp)
{
- int crit_temp, emerg_temp;
int err;
- err = mlxsw_env_module_temp_thresholds_get(core, tz->module,
- SFP_TEMP_HIGH_WARN,
- &crit_temp);
- if (err)
- return err;
+ /* Do not try to query temperature thresholds directly from the module's
+ * EEPROM if we got valid thresholds from MTMP.
+ */
+ if (!emerg_temp || !crit_temp) {
+ err = mlxsw_env_module_temp_thresholds_get(core, tz->module,
+ SFP_TEMP_HIGH_WARN,
+ &crit_temp);
+ if (err)
+ return err;
- err = mlxsw_env_module_temp_thresholds_get(core, tz->module,
- SFP_TEMP_HIGH_ALARM,
- &emerg_temp);
- if (err)
- return err;
+ err = mlxsw_env_module_temp_thresholds_get(core, tz->module,
+ SFP_TEMP_HIGH_ALARM,
+ &emerg_temp);
+ if (err)
+ return err;
+ }
if (crit_temp > emerg_temp) {
dev_warn(dev, "%s : Critical threshold %d is above emergency threshold %d\n",
dev_err(dev, "Failed to query temp sensor\n");
return err;
}
- mlxsw_reg_mtmp_unpack(mtmp_pl, &temp, NULL, NULL);
+ mlxsw_reg_mtmp_unpack(mtmp_pl, &temp, NULL, NULL, NULL, NULL);
if (temp > 0)
mlxsw_thermal_tz_score_update(thermal, tzdev, thermal->trips,
temp);
return err;
}
-static int mlxsw_thermal_module_temp_get(struct thermal_zone_device *tzdev,
- int *p_temp)
+static void
+mlxsw_thermal_module_temp_and_thresholds_get(struct mlxsw_core *core,
+ u16 sensor_index, int *p_temp,
+ int *p_crit_temp,
+ int *p_emerg_temp)
{
- struct mlxsw_thermal_module *tz = tzdev->devdata;
- struct mlxsw_thermal *thermal = tz->parent;
- struct device *dev = thermal->bus_info->dev;
char mtmp_pl[MLXSW_REG_MTMP_LEN];
- int temp;
int err;
- /* Read module temperature. */
- mlxsw_reg_mtmp_pack(mtmp_pl, MLXSW_REG_MTMP_MODULE_INDEX_MIN +
- tz->module, false, false);
- err = mlxsw_reg_query(thermal->core, MLXSW_REG(mtmp), mtmp_pl);
+ /* Read module temperature and thresholds. */
+ mlxsw_reg_mtmp_pack(mtmp_pl, sensor_index, false, false);
+ err = mlxsw_reg_query(core, MLXSW_REG(mtmp), mtmp_pl);
if (err) {
- /* Do not return error - in case of broken module's sensor
- * it will cause error message flooding.
+ /* Set temperature and thresholds to zero to avoid passing
+ * uninitialized data back to the caller.
*/
- temp = 0;
- *p_temp = (int) temp;
- return 0;
+ *p_temp = 0;
+ *p_crit_temp = 0;
+ *p_emerg_temp = 0;
+
+ return;
}
- mlxsw_reg_mtmp_unpack(mtmp_pl, &temp, NULL, NULL);
+ mlxsw_reg_mtmp_unpack(mtmp_pl, p_temp, NULL, p_crit_temp, p_emerg_temp,
+ NULL);
+}
+
+static int mlxsw_thermal_module_temp_get(struct thermal_zone_device *tzdev,
+ int *p_temp)
+{
+ struct mlxsw_thermal_module *tz = tzdev->devdata;
+ struct mlxsw_thermal *thermal = tz->parent;
+ int temp, crit_temp, emerg_temp;
+ struct device *dev;
+ u16 sensor_index;
+ int err;
+
+ dev = thermal->bus_info->dev;
+ sensor_index = MLXSW_REG_MTMP_MODULE_INDEX_MIN + tz->module;
+
+ /* Read module temperature and thresholds. */
+ mlxsw_thermal_module_temp_and_thresholds_get(thermal->core,
+ sensor_index, &temp,
+ &crit_temp, &emerg_temp);
*p_temp = temp;
if (!temp)
return 0;
/* Update trip points. */
- err = mlxsw_thermal_module_trips_update(dev, thermal->core, tz);
+ err = mlxsw_thermal_module_trips_update(dev, thermal->core, tz,
+ crit_temp, emerg_temp);
if (!err && temp > 0)
mlxsw_thermal_tz_score_update(thermal, tzdev, tz->trips, temp);
if (err)
return err;
- mlxsw_reg_mtmp_unpack(mtmp_pl, &temp, NULL, NULL);
+ mlxsw_reg_mtmp_unpack(mtmp_pl, &temp, NULL, NULL, NULL, NULL);
if (temp > 0)
mlxsw_thermal_tz_score_update(thermal, tzdev, tz->trips, temp);
MLXSW_THERMAL_TRIP_MASK,
module_tz,
&mlxsw_thermal_module_ops,
- NULL, 0, 0);
+ NULL, 0,
+ module_tz->parent->polling_delay);
if (IS_ERR(module_tz->tzdev)) {
err = PTR_ERR(module_tz->tzdev);
return err;
struct mlxsw_thermal *thermal, u8 module)
{
struct mlxsw_thermal_module *module_tz;
+ int dummy_temp, crit_temp, emerg_temp;
+ u16 sensor_index;
+ sensor_index = MLXSW_REG_MTMP_MODULE_INDEX_MIN + module;
module_tz = &thermal->tz_module_arr[module];
/* Skip if parent is already set (case of port split). */
if (module_tz->parent)
sizeof(thermal->trips));
/* Initialize all trip point. */
mlxsw_thermal_module_trips_reset(module_tz);
+ /* Read module temperature and thresholds. */
+ mlxsw_thermal_module_temp_and_thresholds_get(core, sensor_index, &dummy_temp,
+ &crit_temp, &emerg_temp);
/* Update trip point according to the module data. */
- return mlxsw_thermal_module_trips_update(dev, core, module_tz);
+ return mlxsw_thermal_module_trips_update(dev, core, module_tz,
+ crit_temp, emerg_temp);
}
static void mlxsw_thermal_module_fini(struct mlxsw_thermal_module *module_tz)
MLXSW_THERMAL_TRIP_MASK,
gearbox_tz,
&mlxsw_thermal_gearbox_ops,
- NULL, 0, 0);
+ NULL, 0,
+ gearbox_tz->parent->polling_delay);
if (IS_ERR(gearbox_tz->tzdev))
return PTR_ERR(gearbox_tz->tzdev);
+++ /dev/null
-/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 */
-/* Copyright (c) 2016-2018 Mellanox Technologies. All rights reserved */
-
-#ifndef _MLXSW_IB_H
-#define _MLXSW_IB_H
-
-#define MLXSW_IB_DEFAULT_MTU 4096
-
-#endif /* _MLXSW_IB_H */
ee, data);
}
+static int
+mlxsw_m_get_module_eeprom_by_page(struct net_device *netdev,
+ const struct ethtool_module_eeprom *page,
+ struct netlink_ext_ack *extack)
+{
+ struct mlxsw_m_port *mlxsw_m_port = netdev_priv(netdev);
+ struct mlxsw_core *core = mlxsw_m_port->mlxsw_m->core;
+
+ return mlxsw_env_get_module_eeprom_by_page(core, mlxsw_m_port->module,
+ page, extack);
+}
+
static const struct ethtool_ops mlxsw_m_port_ethtool_ops = {
.get_drvinfo = mlxsw_m_module_get_drvinfo,
.get_module_info = mlxsw_m_get_module_info,
.get_module_eeprom = mlxsw_m_get_module_eeprom,
+ .get_module_eeprom_by_page = mlxsw_m_get_module_eeprom_by_page,
};
static int
static int mlxsw_m_port_module_map(struct mlxsw_m *mlxsw_m, u8 local_port,
u8 *last_module)
{
+ unsigned int max_ports = mlxsw_core_max_ports(mlxsw_m->core);
u8 module, width;
int err;
if (module == *last_module)
return 0;
*last_module = module;
+
+ if (WARN_ON_ONCE(module >= max_ports))
+ return -EINVAL;
mlxsw_m->module_to_port[module] = ++mlxsw_m->max_ports;
return 0;
unsigned long end;
u32 val;
- if (id->device == PCI_DEVICE_ID_MELLANOX_SWITCHX2) {
- msleep(MLXSW_PCI_SW_RESET_TIMEOUT_MSECS);
- return 0;
- }
-
/* We must wait for the HW to become responsive. */
msleep(MLXSW_PCI_SW_RESET_WAIT_MSECS);
#include <linux/pci.h>
-#define PCI_DEVICE_ID_MELLANOX_SWITCHX2 0xc738
#define PCI_DEVICE_ID_MELLANOX_SPECTRUM 0xcb84
#define PCI_DEVICE_ID_MELLANOX_SPECTRUM2 0xcf6c
#define PCI_DEVICE_ID_MELLANOX_SPECTRUM3 0xcf70
-#define PCI_DEVICE_ID_MELLANOX_SWITCHIB 0xcb20
-#define PCI_DEVICE_ID_MELLANOX_SWITCHIB2 0xcf08
#if IS_ENABLED(CONFIG_MLXSW_PCI)
#define MLXSW_REG_QEEC_HIGHEST_SHAPER_BS 25
#define MLXSW_REG_QEEC_LOWEST_SHAPER_BS_SP1 5
#define MLXSW_REG_QEEC_LOWEST_SHAPER_BS_SP2 11
-#define MLXSW_REG_QEEC_LOWEST_SHAPER_BS_SP3 5
+#define MLXSW_REG_QEEC_LOWEST_SHAPER_BS_SP3 11
static inline void mlxsw_reg_qeec_pack(char *payload, u8 local_port,
enum mlxsw_reg_qeec_hr hr, u8 index,
MLXSW_REG_RECR2_TCP_UDP_EN_IPV4 = 7,
/* Enable TCP/UDP header fields if packet is IPv6 */
MLXSW_REG_RECR2_TCP_UDP_EN_IPV6 = 8,
+
+ __MLXSW_REG_RECR2_HEADER_CNT,
};
/* reg_recr2_outer_header_enables
MLXSW_REG_RECR2_TCP_UDP_SPORT = 74,
/* TCP/UDP Destination Port */
MLXSW_REG_RECR2_TCP_UDP_DPORT = 75,
+
+ __MLXSW_REG_RECR2_FIELD_CNT,
};
/* reg_recr2_outer_header_fields_enable
*/
MLXSW_ITEM_BIT_ARRAY(reg, recr2, outer_header_fields_enable, 0x14, 0x14, 1);
-static inline void mlxsw_reg_recr2_ipv4_sip_enable(char *payload)
-{
- int i;
-
- for (i = MLXSW_REG_RECR2_IPV4_SIP0; i <= MLXSW_REG_RECR2_IPV4_SIP3; i++)
- mlxsw_reg_recr2_outer_header_fields_enable_set(payload, i,
- true);
-}
-
-static inline void mlxsw_reg_recr2_ipv4_dip_enable(char *payload)
-{
- int i;
-
- for (i = MLXSW_REG_RECR2_IPV4_DIP0; i <= MLXSW_REG_RECR2_IPV4_DIP3; i++)
- mlxsw_reg_recr2_outer_header_fields_enable_set(payload, i,
- true);
-}
-
-static inline void mlxsw_reg_recr2_ipv6_sip_enable(char *payload)
-{
- int i = MLXSW_REG_RECR2_IPV6_SIP0_7;
-
- mlxsw_reg_recr2_outer_header_fields_enable_set(payload, i, true);
-
- i = MLXSW_REG_RECR2_IPV6_SIP8;
- for (; i <= MLXSW_REG_RECR2_IPV6_SIP15; i++)
- mlxsw_reg_recr2_outer_header_fields_enable_set(payload, i,
- true);
-}
-
-static inline void mlxsw_reg_recr2_ipv6_dip_enable(char *payload)
-{
- int i = MLXSW_REG_RECR2_IPV6_DIP0_7;
-
- mlxsw_reg_recr2_outer_header_fields_enable_set(payload, i, true);
+/* reg_recr2_inner_header_enables
+ * Bit mask where each bit enables a specific inner layer to be included in the
+ * hash calculation. Same values as reg_recr2_outer_header_enables.
+ * Access: RW
+ */
+MLXSW_ITEM_BIT_ARRAY(reg, recr2, inner_header_enables, 0x2C, 0x04, 1);
- i = MLXSW_REG_RECR2_IPV6_DIP8;
- for (; i <= MLXSW_REG_RECR2_IPV6_DIP15; i++)
- mlxsw_reg_recr2_outer_header_fields_enable_set(payload, i,
- true);
-}
+enum {
+ /* Inner IPv4 Source IP */
+ MLXSW_REG_RECR2_INNER_IPV4_SIP0 = 3,
+ MLXSW_REG_RECR2_INNER_IPV4_SIP3 = 6,
+ /* Inner IPv4 Destination IP */
+ MLXSW_REG_RECR2_INNER_IPV4_DIP0 = 7,
+ MLXSW_REG_RECR2_INNER_IPV4_DIP3 = 10,
+ /* Inner IP Protocol */
+ MLXSW_REG_RECR2_INNER_IPV4_PROTOCOL = 11,
+ /* Inner IPv6 Source IP */
+ MLXSW_REG_RECR2_INNER_IPV6_SIP0_7 = 12,
+ MLXSW_REG_RECR2_INNER_IPV6_SIP8 = 20,
+ MLXSW_REG_RECR2_INNER_IPV6_SIP15 = 27,
+ /* Inner IPv6 Destination IP */
+ MLXSW_REG_RECR2_INNER_IPV6_DIP0_7 = 28,
+ MLXSW_REG_RECR2_INNER_IPV6_DIP8 = 36,
+ MLXSW_REG_RECR2_INNER_IPV6_DIP15 = 43,
+ /* Inner IPv6 Next Header */
+ MLXSW_REG_RECR2_INNER_IPV6_NEXT_HEADER = 44,
+ /* Inner IPv6 Flow Label */
+ MLXSW_REG_RECR2_INNER_IPV6_FLOW_LABEL = 45,
+ /* Inner TCP/UDP Source Port */
+ MLXSW_REG_RECR2_INNER_TCP_UDP_SPORT = 46,
+ /* Inner TCP/UDP Destination Port */
+ MLXSW_REG_RECR2_INNER_TCP_UDP_DPORT = 47,
+
+ __MLXSW_REG_RECR2_INNER_FIELD_CNT,
+};
+
+/* reg_recr2_inner_header_fields_enable
+ * Inner packet fields to enable for ECMP hash subject to inner_header_enables.
+ * Access: RW
+ */
+MLXSW_ITEM_BIT_ARRAY(reg, recr2, inner_header_fields_enable, 0x30, 0x08, 1);
static inline void mlxsw_reg_recr2_pack(char *payload, u32 seed)
{
((s16)((GENMASK(15, 0) + (v_) + 1) \
* 125)); })
+/* reg_mtmp_max_operational_temperature
+ * The highest temperature in the nominal operational range. Reading is in
+ * 0.125 Celsius degrees units.
+ * In case of module this is SFF critical temperature threshold.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mtmp, max_operational_temperature, 0x04, 16, 16);
+
/* reg_mtmp_temperature
* Temperature reading from the sensor. Reading is in 0.125 Celsius
* degrees units.
}
static inline void mlxsw_reg_mtmp_unpack(char *payload, int *p_temp,
- int *p_max_temp, char *sensor_name)
+ int *p_max_temp, int *p_temp_hi,
+ int *p_max_oper_temp,
+ char *sensor_name)
{
s16 temp;
temp = mlxsw_reg_mtmp_max_temperature_get(payload);
*p_max_temp = MLXSW_REG_MTMP_TEMP_TO_MC(temp);
}
+ if (p_temp_hi) {
+ temp = mlxsw_reg_mtmp_temperature_threshold_hi_get(payload);
+ *p_temp_hi = MLXSW_REG_MTMP_TEMP_TO_MC(temp);
+ }
+ if (p_max_oper_temp) {
+ temp = mlxsw_reg_mtmp_max_operational_temperature_get(payload);
+ *p_max_oper_temp = MLXSW_REG_MTMP_TEMP_TO_MC(temp);
+ }
if (sensor_name)
mlxsw_reg_mtmp_sensor_name_memcpy_from(payload, sensor_name);
}
*/
MLXSW_ITEM32(reg, mcia, module, 0x00, 16, 8);
+enum {
+ MLXSW_REG_MCIA_STATUS_GOOD = 0,
+ /* No response from module's EEPROM. */
+ MLXSW_REG_MCIA_STATUS_NO_EEPROM_MODULE = 1,
+ /* Module type not supported by the device. */
+ MLXSW_REG_MCIA_STATUS_MODULE_NOT_SUPPORTED = 2,
+ /* No module present indication. */
+ MLXSW_REG_MCIA_STATUS_MODULE_NOT_CONNECTED = 3,
+ /* Error occurred while trying to access module's EEPROM using I2C. */
+ MLXSW_REG_MCIA_STATUS_I2C_ERROR = 9,
+ /* Module is disabled. */
+ MLXSW_REG_MCIA_STATUS_MODULE_DISABLED = 16,
+};
+
/* reg_mcia_status
* Module status.
* Access: RO
*/
MLXSW_ITEM32(reg, mcia, device_address, 0x04, 0, 16);
+/* reg_mcia_bank_number
+ * Bank number.
+ * Access: Index
+ */
+MLXSW_ITEM32(reg, mcia, bank_number, 0x08, 16, 8);
+
/* reg_mcia_size
* Number of bytes to read/write (up to 48 bytes).
* Access: RW
struct mlxsw_sp *mlxsw_sp = priv;
struct mlxsw_sp_port *mlxsw_sp_port;
enum mlxsw_reg_pude_oper_status status;
+ unsigned int max_ports;
u8 local_port;
+ max_ports = mlxsw_core_max_ports(mlxsw_sp->core);
local_port = mlxsw_reg_pude_local_port_get(pude_pl);
+
+ if (WARN_ON_ONCE(local_port >= max_ports))
+ return;
mlxsw_sp_port = mlxsw_sp->ports[local_port];
if (!mlxsw_sp_port)
return;
static u32 mlxsw_sp_hdroom_int_buf_size_get(struct mlxsw_sp *mlxsw_sp, int mtu, u32 speed)
{
- u32 buffsize = mlxsw_sp->sb_ops->int_buf_size_get(speed, mtu);
+ u32 buffsize = mlxsw_sp->sb_ops->int_buf_size_get(mtu, speed);
return mlxsw_sp_bytes_cells(mlxsw_sp, buffsize) + 1;
}
int i;
/* Internal buffer. */
- reserve_cells = mlxsw_sp_hdroom_int_buf_size_get(mlxsw_sp, mlxsw_sp_port->max_speed,
- mlxsw_sp_port->max_mtu);
+ reserve_cells = mlxsw_sp_hdroom_int_buf_size_get(mlxsw_sp, mlxsw_sp_port->max_mtu,
+ mlxsw_sp_port->max_speed);
reserve_cells = mlxsw_sp_port_headroom_8x_adjust(mlxsw_sp_port, reserve_cells);
hdroom->int_buf.reserve_cells = reserve_cells;
}
static int
+mlxsw_sp_get_module_eeprom_by_page(struct net_device *dev,
+ const struct ethtool_module_eeprom *page,
+ struct netlink_ext_ack *extack)
+{
+ struct mlxsw_sp_port *mlxsw_sp_port = netdev_priv(dev);
+ struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp;
+ u8 module = mlxsw_sp_port->mapping.module;
+
+ return mlxsw_env_get_module_eeprom_by_page(mlxsw_sp->core, module, page,
+ extack);
+}
+
+static int
mlxsw_sp_get_ts_info(struct net_device *netdev, struct ethtool_ts_info *info)
{
struct mlxsw_sp_port *mlxsw_sp_port = netdev_priv(netdev);
.set_link_ksettings = mlxsw_sp_port_set_link_ksettings,
.get_module_info = mlxsw_sp_get_module_info,
.get_module_eeprom = mlxsw_sp_get_module_eeprom,
+ .get_module_eeprom_by_page = mlxsw_sp_get_module_eeprom_by_page,
.get_ts_info = mlxsw_sp_get_ts_info,
.get_eth_phy_stats = mlxsw_sp_get_eth_phy_stats,
.get_eth_mac_stats = mlxsw_sp_get_eth_mac_stats,
u8 domain_number, u16 sequence_id,
u64 timestamp)
{
+ unsigned int max_ports = mlxsw_core_max_ports(mlxsw_sp->core);
struct mlxsw_sp_port *mlxsw_sp_port;
struct mlxsw_sp1_ptp_key key;
u8 types;
+ if (WARN_ON_ONCE(local_port >= max_ports))
+ return;
mlxsw_sp_port = mlxsw_sp->ports[local_port];
if (!mlxsw_sp_port)
return;
u8 band, u32 child_handle)
{
struct mlxsw_sp_qdisc *old_qdisc;
+ u32 parent;
if (band < mlxsw_sp_qdisc->num_classes &&
mlxsw_sp_qdisc->qdiscs[band].handle == child_handle)
if (old_qdisc)
mlxsw_sp_qdisc_destroy(mlxsw_sp_port, old_qdisc);
- mlxsw_sp_qdisc = mlxsw_sp_qdisc->ops->find_class(mlxsw_sp_qdisc, band);
+ parent = TC_H_MAKE(mlxsw_sp_qdisc->handle, band + 1);
+ mlxsw_sp_qdisc = mlxsw_sp_qdisc->ops->find_class(mlxsw_sp_qdisc,
+ parent);
if (!WARN_ON(!mlxsw_sp_qdisc))
mlxsw_sp_qdisc_destroy(mlxsw_sp_port, mlxsw_sp_qdisc);
char *rauhtd_pl,
int ent_index)
{
+ u64 max_rifs = MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_RIFS);
struct net_device *dev;
struct neighbour *n;
__be32 dipn;
mlxsw_reg_rauhtd_ent_ipv4_unpack(rauhtd_pl, ent_index, &rif, &dip);
+ if (WARN_ON_ONCE(rif >= max_rifs))
+ return;
if (!mlxsw_sp->router->rifs[rif]) {
dev_err_ratelimited(mlxsw_sp->bus_info->dev, "Incorrect RIF in neighbour entry\n");
return;
bool offload_change = false;
u32 adj_index;
bool old_adj_index_valid;
- int i, err2, err = 0;
u32 old_adj_index;
+ int i, err2, err;
if (!nhgi->gateway)
return mlxsw_sp_nexthop_fib_entries_update(mlxsw_sp, nh_grp);
return 0;
}
mlxsw_sp_nexthop_group_normalize(nhgi);
- if (!nhgi->sum_norm_weight)
+ if (!nhgi->sum_norm_weight) {
/* No neigh of this group is connected so we just set
* the trap and let everthing flow through kernel.
*/
+ err = 0;
goto set_trap;
+ }
ecmp_size = nhgi->sum_norm_weight;
err = mlxsw_sp_fix_adj_grp_size(mlxsw_sp, &ecmp_size);
struct mlxsw_sp_nexthop_key key;
struct mlxsw_sp_nexthop *nh;
- if (mlxsw_sp->router->aborted)
- return;
-
key.fib_nh = fib_nh;
nh = mlxsw_sp_nexthop_lookup(mlxsw_sp, key);
if (!nh)
ipv6_addr_equal((const struct in6_addr *) &nh->gw_addr,
&rt->fib6_nh->fib_nh_gw6))
return nh;
- continue;
}
return NULL;
struct mlxsw_sp_fib_node *fib_node;
int err;
- if (mlxsw_sp->router->aborted)
- return 0;
-
if (fen_info->fi->nh &&
!mlxsw_sp_nexthop_obj_group_lookup(mlxsw_sp, fen_info->fi->nh->id))
return 0;
struct mlxsw_sp_fib_node *fib_node;
int err;
- if (mlxsw_sp->router->aborted)
- return 0;
-
fib4_entry = mlxsw_sp_fib4_entry_lookup(mlxsw_sp, fen_info);
if (!fib4_entry)
return 0;
struct fib6_info *rt = rt_arr[0];
int err;
- if (mlxsw_sp->router->aborted)
- return 0;
-
if (rt->fib6_src.plen)
return -EINVAL;
struct fib6_info *rt = rt_arr[0];
int err;
- if (mlxsw_sp->router->aborted)
- return 0;
-
if (rt->fib6_src.plen)
return -EINVAL;
struct fib6_info *rt = rt_arr[0];
int err;
- if (mlxsw_sp->router->aborted)
- return 0;
-
if (mlxsw_sp_fib6_rt_should_ignore(rt))
return 0;
return err;
}
-static int __mlxsw_sp_router_set_abort_trap(struct mlxsw_sp *mlxsw_sp,
- enum mlxsw_sp_l3proto proto,
- u8 tree_id)
-{
- const struct mlxsw_sp_router_ll_ops *ll_ops = mlxsw_sp->router->proto_ll_ops[proto];
- enum mlxsw_reg_ralxx_protocol ralxx_proto =
- (enum mlxsw_reg_ralxx_protocol) proto;
- struct mlxsw_sp_fib_entry_priv *priv;
- char xralta_pl[MLXSW_REG_XRALTA_LEN];
- char xralst_pl[MLXSW_REG_XRALST_LEN];
- int i, err;
-
- mlxsw_reg_xralta_pack(xralta_pl, true, ralxx_proto, tree_id);
- err = ll_ops->ralta_write(mlxsw_sp, xralta_pl);
- if (err)
- return err;
-
- mlxsw_reg_xralst_pack(xralst_pl, 0xff, tree_id);
- err = ll_ops->ralst_write(mlxsw_sp, xralst_pl);
- if (err)
- return err;
-
- for (i = 0; i < MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_VRS); i++) {
- struct mlxsw_sp_fib_entry_op_ctx *op_ctx = mlxsw_sp->router->ll_op_ctx;
- struct mlxsw_sp_vr *vr = &mlxsw_sp->router->vrs[i];
- char xraltb_pl[MLXSW_REG_XRALTB_LEN];
-
- mlxsw_sp_fib_entry_op_ctx_clear(op_ctx);
- mlxsw_reg_xraltb_pack(xraltb_pl, vr->id, ralxx_proto, tree_id);
- err = ll_ops->raltb_write(mlxsw_sp, xraltb_pl);
- if (err)
- return err;
-
- priv = mlxsw_sp_fib_entry_priv_create(ll_ops);
- if (IS_ERR(priv))
- return PTR_ERR(priv);
-
- ll_ops->fib_entry_pack(op_ctx, proto, MLXSW_SP_FIB_ENTRY_OP_WRITE,
- vr->id, 0, NULL, priv);
- ll_ops->fib_entry_act_ip2me_pack(op_ctx);
- err = ll_ops->fib_entry_commit(mlxsw_sp, op_ctx, NULL);
- mlxsw_sp_fib_entry_priv_put(priv);
- if (err)
- return err;
- }
-
- return 0;
-}
-
static struct mlxsw_sp_mr_table *
mlxsw_sp_router_fibmr_family_to_table(struct mlxsw_sp_vr *vr, int family)
{
struct mlxsw_sp_mr_table *mrt;
struct mlxsw_sp_vr *vr;
- if (mlxsw_sp->router->aborted)
- return 0;
-
vr = mlxsw_sp_vr_get(mlxsw_sp, men_info->tb_id, NULL);
if (IS_ERR(vr))
return PTR_ERR(vr);
struct mlxsw_sp_mr_table *mrt;
struct mlxsw_sp_vr *vr;
- if (mlxsw_sp->router->aborted)
- return;
-
vr = mlxsw_sp_vr_find(mlxsw_sp, men_info->tb_id);
if (WARN_ON(!vr))
return;
struct mlxsw_sp_rif *rif;
struct mlxsw_sp_vr *vr;
- if (mlxsw_sp->router->aborted)
- return 0;
-
vr = mlxsw_sp_vr_get(mlxsw_sp, ven_info->tb_id, NULL);
if (IS_ERR(vr))
return PTR_ERR(vr);
struct mlxsw_sp_mr_table *mrt;
struct mlxsw_sp_vr *vr;
- if (mlxsw_sp->router->aborted)
- return;
-
vr = mlxsw_sp_vr_find(mlxsw_sp, ven_info->tb_id);
if (WARN_ON(!vr))
return;
mlxsw_sp_vr_put(mlxsw_sp, vr);
}
-static int mlxsw_sp_router_set_abort_trap(struct mlxsw_sp *mlxsw_sp)
-{
- enum mlxsw_sp_l3proto proto = MLXSW_SP_L3_PROTO_IPV4;
- int err;
-
- err = __mlxsw_sp_router_set_abort_trap(mlxsw_sp, proto,
- MLXSW_SP_LPM_TREE_MIN);
- if (err)
- return err;
-
- /* The multicast router code does not need an abort trap as by default,
- * packets that don't match any routes are trapped to the CPU.
- */
-
- proto = MLXSW_SP_L3_PROTO_IPV6;
- return __mlxsw_sp_router_set_abort_trap(mlxsw_sp, proto,
- MLXSW_SP_LPM_TREE_MIN + 1);
-}
-
static void mlxsw_sp_fib4_node_flush(struct mlxsw_sp *mlxsw_sp,
struct mlxsw_sp_fib_node *fib_node)
{
mlxsw_sp->router->adj_discard_index_valid = false;
}
-static void mlxsw_sp_router_fib_abort(struct mlxsw_sp *mlxsw_sp)
-{
- int err;
-
- if (mlxsw_sp->router->aborted)
- return;
- dev_warn(mlxsw_sp->bus_info->dev, "FIB abort triggered. Note that FIB entries are no longer being offloaded to this device.\n");
- mlxsw_sp_router_fib_flush(mlxsw_sp);
- mlxsw_sp->router->aborted = true;
- err = mlxsw_sp_router_set_abort_trap(mlxsw_sp);
- if (err)
- dev_warn(mlxsw_sp->bus_info->dev, "Failed to set abort trap.\n");
-}
-
struct mlxsw_sp_fib6_event {
struct fib6_info **rt_arr;
unsigned int nrt6;
err = mlxsw_sp_router_fib4_replace(mlxsw_sp, op_ctx, &fib_event->fen_info);
if (err) {
mlxsw_sp_fib_entry_op_ctx_priv_put_all(op_ctx);
- mlxsw_sp_router_fib_abort(mlxsw_sp);
+ dev_warn(mlxsw_sp->bus_info->dev, "FIB replace failed.\n");
mlxsw_sp_fib4_offload_failed_flag_set(mlxsw_sp,
&fib_event->fen_info);
}
fib_event->fib6_event.nrt6);
if (err) {
mlxsw_sp_fib_entry_op_ctx_priv_put_all(op_ctx);
- mlxsw_sp_router_fib_abort(mlxsw_sp);
+ dev_warn(mlxsw_sp->bus_info->dev, "FIB replace failed.\n");
mlxsw_sp_fib6_offload_failed_flag_set(mlxsw_sp,
fib6_event->rt_arr,
fib6_event->nrt6);
fib_event->fib6_event.nrt6);
if (err) {
mlxsw_sp_fib_entry_op_ctx_priv_put_all(op_ctx);
- mlxsw_sp_router_fib_abort(mlxsw_sp);
+ dev_warn(mlxsw_sp->bus_info->dev, "FIB append failed.\n");
mlxsw_sp_fib6_offload_failed_flag_set(mlxsw_sp,
fib6_event->rt_arr,
fib6_event->nrt6);
err = mlxsw_sp_router_fibmr_add(mlxsw_sp, &fib_event->men_info, replace);
if (err)
- mlxsw_sp_router_fib_abort(mlxsw_sp);
+ dev_warn(mlxsw_sp->bus_info->dev, "MR entry add failed.\n");
mr_cache_put(fib_event->men_info.mfc);
break;
case FIB_EVENT_ENTRY_DEL:
err = mlxsw_sp_router_fibmr_vif_add(mlxsw_sp,
&fib_event->ven_info);
if (err)
- mlxsw_sp_router_fib_abort(mlxsw_sp);
+ dev_warn(mlxsw_sp->bus_info->dev, "MR VIF add failed.\n");
dev_put(fib_event->ven_info.dev);
break;
case FIB_EVENT_VIF_DEL:
if (event == FIB_EVENT_RULE_DEL)
return 0;
- if (mlxsw_sp->router->aborted)
- return 0;
-
fr_info = container_of(info, struct fib_rule_notifier_info, info);
rule = fr_info->rule;
case FIB_EVENT_ENTRY_ADD:
case FIB_EVENT_ENTRY_REPLACE:
case FIB_EVENT_ENTRY_APPEND:
- if (router->aborted) {
- NL_SET_ERR_MSG_MOD(info->extack, "FIB offload was aborted. Not configuring route");
- return notifier_from_errno(-EINVAL);
- }
if (info->family == AF_INET) {
struct fib_entry_notifier_info *fen_info = ptr;
}
#ifdef CONFIG_IP_ROUTE_MULTIPATH
-static void mlxsw_sp_mp_hash_header_set(char *recr2_pl, int header)
+struct mlxsw_sp_mp_hash_config {
+ DECLARE_BITMAP(headers, __MLXSW_REG_RECR2_HEADER_CNT);
+ DECLARE_BITMAP(fields, __MLXSW_REG_RECR2_FIELD_CNT);
+ DECLARE_BITMAP(inner_headers, __MLXSW_REG_RECR2_HEADER_CNT);
+ DECLARE_BITMAP(inner_fields, __MLXSW_REG_RECR2_INNER_FIELD_CNT);
+};
+
+#define MLXSW_SP_MP_HASH_HEADER_SET(_headers, _header) \
+ bitmap_set(_headers, MLXSW_REG_RECR2_##_header, 1)
+
+#define MLXSW_SP_MP_HASH_FIELD_SET(_fields, _field) \
+ bitmap_set(_fields, MLXSW_REG_RECR2_##_field, 1)
+
+#define MLXSW_SP_MP_HASH_FIELD_RANGE_SET(_fields, _field, _nr) \
+ bitmap_set(_fields, MLXSW_REG_RECR2_##_field, _nr)
+
+static void mlxsw_sp_mp_hash_inner_l3(struct mlxsw_sp_mp_hash_config *config)
{
- mlxsw_reg_recr2_outer_header_enables_set(recr2_pl, header, true);
+ unsigned long *inner_headers = config->inner_headers;
+ unsigned long *inner_fields = config->inner_fields;
+
+ /* IPv4 inner */
+ MLXSW_SP_MP_HASH_HEADER_SET(inner_headers, IPV4_EN_NOT_TCP_NOT_UDP);
+ MLXSW_SP_MP_HASH_HEADER_SET(inner_headers, IPV4_EN_TCP_UDP);
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(inner_fields, INNER_IPV4_SIP0, 4);
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(inner_fields, INNER_IPV4_DIP0, 4);
+ /* IPv6 inner */
+ MLXSW_SP_MP_HASH_HEADER_SET(inner_headers, IPV6_EN_NOT_TCP_NOT_UDP);
+ MLXSW_SP_MP_HASH_HEADER_SET(inner_headers, IPV6_EN_TCP_UDP);
+ MLXSW_SP_MP_HASH_FIELD_SET(inner_fields, INNER_IPV6_SIP0_7);
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(inner_fields, INNER_IPV6_SIP8, 8);
+ MLXSW_SP_MP_HASH_FIELD_SET(inner_fields, INNER_IPV6_DIP0_7);
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(inner_fields, INNER_IPV6_DIP8, 8);
+ MLXSW_SP_MP_HASH_FIELD_SET(inner_fields, INNER_IPV6_NEXT_HEADER);
+ MLXSW_SP_MP_HASH_FIELD_SET(inner_fields, INNER_IPV6_FLOW_LABEL);
}
-static void mlxsw_sp_mp_hash_field_set(char *recr2_pl, int field)
+static void mlxsw_sp_mp4_hash_outer_addr(struct mlxsw_sp_mp_hash_config *config)
{
- mlxsw_reg_recr2_outer_header_fields_enable_set(recr2_pl, field, true);
+ unsigned long *headers = config->headers;
+ unsigned long *fields = config->fields;
+
+ MLXSW_SP_MP_HASH_HEADER_SET(headers, IPV4_EN_NOT_TCP_NOT_UDP);
+ MLXSW_SP_MP_HASH_HEADER_SET(headers, IPV4_EN_TCP_UDP);
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(fields, IPV4_SIP0, 4);
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(fields, IPV4_DIP0, 4);
}
-static void mlxsw_sp_mp4_hash_init(struct mlxsw_sp *mlxsw_sp, char *recr2_pl)
+static void
+mlxsw_sp_mp_hash_inner_custom(struct mlxsw_sp_mp_hash_config *config,
+ u32 hash_fields)
+{
+ unsigned long *inner_headers = config->inner_headers;
+ unsigned long *inner_fields = config->inner_fields;
+
+ /* IPv4 Inner */
+ MLXSW_SP_MP_HASH_HEADER_SET(inner_headers, IPV4_EN_NOT_TCP_NOT_UDP);
+ MLXSW_SP_MP_HASH_HEADER_SET(inner_headers, IPV4_EN_TCP_UDP);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_SRC_IP)
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(inner_fields, INNER_IPV4_SIP0, 4);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_DST_IP)
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(inner_fields, INNER_IPV4_DIP0, 4);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_IP_PROTO)
+ MLXSW_SP_MP_HASH_FIELD_SET(inner_fields, INNER_IPV4_PROTOCOL);
+ /* IPv6 inner */
+ MLXSW_SP_MP_HASH_HEADER_SET(inner_headers, IPV6_EN_NOT_TCP_NOT_UDP);
+ MLXSW_SP_MP_HASH_HEADER_SET(inner_headers, IPV6_EN_TCP_UDP);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_SRC_IP) {
+ MLXSW_SP_MP_HASH_FIELD_SET(inner_fields, INNER_IPV6_SIP0_7);
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(inner_fields, INNER_IPV6_SIP8, 8);
+ }
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_DST_IP) {
+ MLXSW_SP_MP_HASH_FIELD_SET(inner_fields, INNER_IPV6_DIP0_7);
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(inner_fields, INNER_IPV6_DIP8, 8);
+ }
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_IP_PROTO)
+ MLXSW_SP_MP_HASH_FIELD_SET(inner_fields, INNER_IPV6_NEXT_HEADER);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_FLOWLABEL)
+ MLXSW_SP_MP_HASH_FIELD_SET(inner_fields, INNER_IPV6_FLOW_LABEL);
+ /* L4 inner */
+ MLXSW_SP_MP_HASH_HEADER_SET(inner_headers, TCP_UDP_EN_IPV4);
+ MLXSW_SP_MP_HASH_HEADER_SET(inner_headers, TCP_UDP_EN_IPV6);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_SRC_PORT)
+ MLXSW_SP_MP_HASH_FIELD_SET(inner_fields, INNER_TCP_UDP_SPORT);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_DST_PORT)
+ MLXSW_SP_MP_HASH_FIELD_SET(inner_fields, INNER_TCP_UDP_DPORT);
+}
+
+static void mlxsw_sp_mp4_hash_init(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_mp_hash_config *config)
{
struct net *net = mlxsw_sp_net(mlxsw_sp);
- bool only_l3 = !net->ipv4.sysctl_fib_multipath_hash_policy;
-
- mlxsw_sp_mp_hash_header_set(recr2_pl,
- MLXSW_REG_RECR2_IPV4_EN_NOT_TCP_NOT_UDP);
- mlxsw_sp_mp_hash_header_set(recr2_pl, MLXSW_REG_RECR2_IPV4_EN_TCP_UDP);
- mlxsw_reg_recr2_ipv4_sip_enable(recr2_pl);
- mlxsw_reg_recr2_ipv4_dip_enable(recr2_pl);
- if (only_l3)
- return;
- mlxsw_sp_mp_hash_header_set(recr2_pl, MLXSW_REG_RECR2_TCP_UDP_EN_IPV4);
- mlxsw_sp_mp_hash_field_set(recr2_pl, MLXSW_REG_RECR2_IPV4_PROTOCOL);
- mlxsw_sp_mp_hash_field_set(recr2_pl, MLXSW_REG_RECR2_TCP_UDP_SPORT);
- mlxsw_sp_mp_hash_field_set(recr2_pl, MLXSW_REG_RECR2_TCP_UDP_DPORT);
+ unsigned long *headers = config->headers;
+ unsigned long *fields = config->fields;
+ u32 hash_fields;
+
+ switch (net->ipv4.sysctl_fib_multipath_hash_policy) {
+ case 0:
+ mlxsw_sp_mp4_hash_outer_addr(config);
+ break;
+ case 1:
+ mlxsw_sp_mp4_hash_outer_addr(config);
+ MLXSW_SP_MP_HASH_HEADER_SET(headers, TCP_UDP_EN_IPV4);
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, IPV4_PROTOCOL);
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, TCP_UDP_SPORT);
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, TCP_UDP_DPORT);
+ break;
+ case 2:
+ /* Outer */
+ mlxsw_sp_mp4_hash_outer_addr(config);
+ /* Inner */
+ mlxsw_sp_mp_hash_inner_l3(config);
+ break;
+ case 3:
+ hash_fields = net->ipv4.sysctl_fib_multipath_hash_fields;
+ /* Outer */
+ MLXSW_SP_MP_HASH_HEADER_SET(headers, IPV4_EN_NOT_TCP_NOT_UDP);
+ MLXSW_SP_MP_HASH_HEADER_SET(headers, IPV4_EN_TCP_UDP);
+ MLXSW_SP_MP_HASH_HEADER_SET(headers, TCP_UDP_EN_IPV4);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_SRC_IP)
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(fields, IPV4_SIP0, 4);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_DST_IP)
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(fields, IPV4_DIP0, 4);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_IP_PROTO)
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, IPV4_PROTOCOL);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_SRC_PORT)
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, TCP_UDP_SPORT);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_DST_PORT)
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, TCP_UDP_DPORT);
+ /* Inner */
+ mlxsw_sp_mp_hash_inner_custom(config, hash_fields);
+ break;
+ }
}
-static void mlxsw_sp_mp6_hash_init(struct mlxsw_sp *mlxsw_sp, char *recr2_pl)
+static void mlxsw_sp_mp6_hash_outer_addr(struct mlxsw_sp_mp_hash_config *config)
{
- bool only_l3 = !ip6_multipath_hash_policy(mlxsw_sp_net(mlxsw_sp));
+ unsigned long *headers = config->headers;
+ unsigned long *fields = config->fields;
- mlxsw_sp_mp_hash_header_set(recr2_pl,
- MLXSW_REG_RECR2_IPV6_EN_NOT_TCP_NOT_UDP);
- mlxsw_sp_mp_hash_header_set(recr2_pl, MLXSW_REG_RECR2_IPV6_EN_TCP_UDP);
- mlxsw_reg_recr2_ipv6_sip_enable(recr2_pl);
- mlxsw_reg_recr2_ipv6_dip_enable(recr2_pl);
- mlxsw_sp_mp_hash_field_set(recr2_pl, MLXSW_REG_RECR2_IPV6_NEXT_HEADER);
- if (only_l3) {
- mlxsw_sp_mp_hash_field_set(recr2_pl,
- MLXSW_REG_RECR2_IPV6_FLOW_LABEL);
- } else {
- mlxsw_sp_mp_hash_header_set(recr2_pl,
- MLXSW_REG_RECR2_TCP_UDP_EN_IPV6);
- mlxsw_sp_mp_hash_field_set(recr2_pl,
- MLXSW_REG_RECR2_TCP_UDP_SPORT);
- mlxsw_sp_mp_hash_field_set(recr2_pl,
- MLXSW_REG_RECR2_TCP_UDP_DPORT);
+ MLXSW_SP_MP_HASH_HEADER_SET(headers, IPV6_EN_NOT_TCP_NOT_UDP);
+ MLXSW_SP_MP_HASH_HEADER_SET(headers, IPV6_EN_TCP_UDP);
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, IPV6_SIP0_7);
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(fields, IPV6_SIP8, 8);
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, IPV6_DIP0_7);
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(fields, IPV6_DIP8, 8);
+}
+
+static void mlxsw_sp_mp6_hash_init(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_mp_hash_config *config)
+{
+ u32 hash_fields = ip6_multipath_hash_fields(mlxsw_sp_net(mlxsw_sp));
+ unsigned long *headers = config->headers;
+ unsigned long *fields = config->fields;
+
+ switch (ip6_multipath_hash_policy(mlxsw_sp_net(mlxsw_sp))) {
+ case 0:
+ mlxsw_sp_mp6_hash_outer_addr(config);
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, IPV6_NEXT_HEADER);
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, IPV6_FLOW_LABEL);
+ break;
+ case 1:
+ mlxsw_sp_mp6_hash_outer_addr(config);
+ MLXSW_SP_MP_HASH_HEADER_SET(headers, TCP_UDP_EN_IPV6);
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, IPV6_NEXT_HEADER);
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, TCP_UDP_SPORT);
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, TCP_UDP_DPORT);
+ break;
+ case 2:
+ /* Outer */
+ mlxsw_sp_mp6_hash_outer_addr(config);
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, IPV6_NEXT_HEADER);
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, IPV6_FLOW_LABEL);
+ /* Inner */
+ mlxsw_sp_mp_hash_inner_l3(config);
+ break;
+ case 3:
+ /* Outer */
+ MLXSW_SP_MP_HASH_HEADER_SET(headers, IPV6_EN_NOT_TCP_NOT_UDP);
+ MLXSW_SP_MP_HASH_HEADER_SET(headers, IPV6_EN_TCP_UDP);
+ MLXSW_SP_MP_HASH_HEADER_SET(headers, TCP_UDP_EN_IPV6);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_SRC_IP) {
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, IPV6_SIP0_7);
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(fields, IPV6_SIP8, 8);
+ }
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_DST_IP) {
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, IPV6_DIP0_7);
+ MLXSW_SP_MP_HASH_FIELD_RANGE_SET(fields, IPV6_DIP8, 8);
+ }
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_IP_PROTO)
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, IPV6_NEXT_HEADER);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_FLOWLABEL)
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, IPV6_FLOW_LABEL);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_SRC_PORT)
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, TCP_UDP_SPORT);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_DST_PORT)
+ MLXSW_SP_MP_HASH_FIELD_SET(fields, TCP_UDP_DPORT);
+ /* Inner */
+ mlxsw_sp_mp_hash_inner_custom(config, hash_fields);
+ break;
}
}
static int mlxsw_sp_mp_hash_init(struct mlxsw_sp *mlxsw_sp)
{
+ struct mlxsw_sp_mp_hash_config config = {};
char recr2_pl[MLXSW_REG_RECR2_LEN];
+ unsigned long bit;
u32 seed;
seed = jhash(mlxsw_sp->base_mac, sizeof(mlxsw_sp->base_mac), 0);
mlxsw_reg_recr2_pack(recr2_pl, seed);
- mlxsw_sp_mp4_hash_init(mlxsw_sp, recr2_pl);
- mlxsw_sp_mp6_hash_init(mlxsw_sp, recr2_pl);
+ mlxsw_sp_mp4_hash_init(mlxsw_sp, &config);
+ mlxsw_sp_mp6_hash_init(mlxsw_sp, &config);
+
+ for_each_set_bit(bit, config.headers, __MLXSW_REG_RECR2_HEADER_CNT)
+ mlxsw_reg_recr2_outer_header_enables_set(recr2_pl, bit, 1);
+ for_each_set_bit(bit, config.fields, __MLXSW_REG_RECR2_FIELD_CNT)
+ mlxsw_reg_recr2_outer_header_fields_enable_set(recr2_pl, bit, 1);
+ for_each_set_bit(bit, config.inner_headers, __MLXSW_REG_RECR2_HEADER_CNT)
+ mlxsw_reg_recr2_inner_header_enables_set(recr2_pl, bit, 1);
+ for_each_set_bit(bit, config.inner_fields, __MLXSW_REG_RECR2_INNER_FIELD_CNT)
+ mlxsw_reg_recr2_inner_header_fields_enable_set(recr2_pl, bit, 1);
return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(recr2), recr2_pl);
}
#define MLXSW_SP_UNRESOLVED_NH_PROBE_INTERVAL 5000 /* ms */
struct list_head nexthop_neighs_list;
struct list_head ipip_list;
- bool aborted;
struct notifier_block nexthop_nb;
struct notifier_block fib_nb;
struct notifier_block netevent_nb;
char *sfn_pl, int rec_index,
bool adding)
{
+ unsigned int max_ports = mlxsw_core_max_ports(mlxsw_sp->core);
struct mlxsw_sp_port_vlan *mlxsw_sp_port_vlan;
struct mlxsw_sp_bridge_device *bridge_device;
struct mlxsw_sp_bridge_port *bridge_port;
int err;
mlxsw_reg_sfn_mac_unpack(sfn_pl, rec_index, mac, &fid, &local_port);
+
+ if (WARN_ON_ONCE(local_port >= max_ports))
+ return;
mlxsw_sp_port = mlxsw_sp->ports[local_port];
if (!mlxsw_sp_port) {
dev_err_ratelimited(mlxsw_sp->bus_info->dev, "Incorrect local port in FDB notification\n");
+++ /dev/null
-// SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0
-/* Copyright (c) 2016-2018 Mellanox Technologies. All rights reserved */
-
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/types.h>
-#include <linux/pci.h>
-#include <linux/netdevice.h>
-#include <linux/etherdevice.h>
-#include <linux/slab.h>
-#include <linux/device.h>
-#include <linux/skbuff.h>
-#include <linux/if_vlan.h>
-#include <net/switchdev.h>
-
-#include "pci.h"
-#include "core.h"
-#include "reg.h"
-#include "port.h"
-#include "trap.h"
-#include "txheader.h"
-#include "ib.h"
-
-static const char mlxsw_sib_driver_name[] = "mlxsw_switchib";
-static const char mlxsw_sib2_driver_name[] = "mlxsw_switchib2";
-
-struct mlxsw_sib_port;
-
-struct mlxsw_sib {
- struct mlxsw_sib_port **ports;
- struct mlxsw_core *core;
- const struct mlxsw_bus_info *bus_info;
- u8 hw_id[ETH_ALEN];
-};
-
-struct mlxsw_sib_port {
- struct mlxsw_sib *mlxsw_sib;
- u8 local_port;
- struct {
- u8 module;
- } mapping;
-};
-
-/* tx_v1_hdr_version
- * Tx header version.
- * Must be set to 1.
- */
-MLXSW_ITEM32(tx_v1, hdr, version, 0x00, 28, 4);
-
-/* tx_v1_hdr_ctl
- * Packet control type.
- * 0 - Ethernet control (e.g. EMADs, LACP)
- * 1 - Ethernet data
- */
-MLXSW_ITEM32(tx_v1, hdr, ctl, 0x00, 26, 2);
-
-/* tx_v1_hdr_proto
- * Packet protocol type. Must be set to 1 (Ethernet).
- */
-MLXSW_ITEM32(tx_v1, hdr, proto, 0x00, 21, 3);
-
-/* tx_v1_hdr_swid
- * Switch partition ID. Must be set to 0.
- */
-MLXSW_ITEM32(tx_v1, hdr, swid, 0x00, 12, 3);
-
-/* tx_v1_hdr_control_tclass
- * Indicates if the packet should use the control TClass and not one
- * of the data TClasses.
- */
-MLXSW_ITEM32(tx_v1, hdr, control_tclass, 0x00, 6, 1);
-
-/* tx_v1_hdr_port_mid
- * Destination local port for unicast packets.
- * Destination multicast ID for multicast packets.
- *
- * Control packets are directed to a specific egress port, while data
- * packets are transmitted through the CPU port (0) into the switch partition,
- * where forwarding rules are applied.
- */
-MLXSW_ITEM32(tx_v1, hdr, port_mid, 0x04, 16, 16);
-
-/* tx_v1_hdr_type
- * 0 - Data packets
- * 6 - Control packets
- */
-MLXSW_ITEM32(tx_v1, hdr, type, 0x0C, 0, 4);
-
-static void
-mlxsw_sib_tx_v1_hdr_construct(struct sk_buff *skb,
- const struct mlxsw_tx_info *tx_info)
-{
- char *txhdr = skb_push(skb, MLXSW_TXHDR_LEN);
-
- memset(txhdr, 0, MLXSW_TXHDR_LEN);
-
- mlxsw_tx_v1_hdr_version_set(txhdr, MLXSW_TXHDR_VERSION_1);
- mlxsw_tx_v1_hdr_ctl_set(txhdr, MLXSW_TXHDR_ETH_CTL);
- mlxsw_tx_v1_hdr_proto_set(txhdr, MLXSW_TXHDR_PROTO_ETH);
- mlxsw_tx_v1_hdr_swid_set(txhdr, 0);
- mlxsw_tx_v1_hdr_control_tclass_set(txhdr, 1);
- mlxsw_tx_v1_hdr_port_mid_set(txhdr, tx_info->local_port);
- mlxsw_tx_v1_hdr_type_set(txhdr, MLXSW_TXHDR_TYPE_CONTROL);
-}
-
-static int mlxsw_sib_hw_id_get(struct mlxsw_sib *mlxsw_sib)
-{
- char spad_pl[MLXSW_REG_SPAD_LEN] = {0};
- int err;
-
- err = mlxsw_reg_query(mlxsw_sib->core, MLXSW_REG(spad), spad_pl);
- if (err)
- return err;
- mlxsw_reg_spad_base_mac_memcpy_from(spad_pl, mlxsw_sib->hw_id);
- return 0;
-}
-
-static int
-mlxsw_sib_port_admin_status_set(struct mlxsw_sib_port *mlxsw_sib_port,
- bool is_up)
-{
- struct mlxsw_sib *mlxsw_sib = mlxsw_sib_port->mlxsw_sib;
- char paos_pl[MLXSW_REG_PAOS_LEN];
-
- mlxsw_reg_paos_pack(paos_pl, mlxsw_sib_port->local_port,
- is_up ? MLXSW_PORT_ADMIN_STATUS_UP :
- MLXSW_PORT_ADMIN_STATUS_DOWN);
- return mlxsw_reg_write(mlxsw_sib->core, MLXSW_REG(paos), paos_pl);
-}
-
-static int mlxsw_sib_port_mtu_set(struct mlxsw_sib_port *mlxsw_sib_port,
- u16 mtu)
-{
- struct mlxsw_sib *mlxsw_sib = mlxsw_sib_port->mlxsw_sib;
- char pmtu_pl[MLXSW_REG_PMTU_LEN];
- int max_mtu;
- int err;
-
- mlxsw_reg_pmtu_pack(pmtu_pl, mlxsw_sib_port->local_port, 0);
- err = mlxsw_reg_query(mlxsw_sib->core, MLXSW_REG(pmtu), pmtu_pl);
- if (err)
- return err;
- max_mtu = mlxsw_reg_pmtu_max_mtu_get(pmtu_pl);
-
- if (mtu > max_mtu)
- return -EINVAL;
-
- mlxsw_reg_pmtu_pack(pmtu_pl, mlxsw_sib_port->local_port, mtu);
- return mlxsw_reg_write(mlxsw_sib->core, MLXSW_REG(pmtu), pmtu_pl);
-}
-
-static int mlxsw_sib_port_set(struct mlxsw_sib_port *mlxsw_sib_port, u8 port)
-{
- struct mlxsw_sib *mlxsw_sib = mlxsw_sib_port->mlxsw_sib;
- char plib_pl[MLXSW_REG_PLIB_LEN] = {0};
- int err;
-
- mlxsw_reg_plib_local_port_set(plib_pl, mlxsw_sib_port->local_port);
- mlxsw_reg_plib_ib_port_set(plib_pl, port);
- err = mlxsw_reg_write(mlxsw_sib->core, MLXSW_REG(plib), plib_pl);
- return err;
-}
-
-static int mlxsw_sib_port_swid_set(struct mlxsw_sib_port *mlxsw_sib_port,
- u8 swid)
-{
- struct mlxsw_sib *mlxsw_sib = mlxsw_sib_port->mlxsw_sib;
- char pspa_pl[MLXSW_REG_PSPA_LEN];
-
- mlxsw_reg_pspa_pack(pspa_pl, swid, mlxsw_sib_port->local_port);
- return mlxsw_reg_write(mlxsw_sib->core, MLXSW_REG(pspa), pspa_pl);
-}
-
-static int mlxsw_sib_port_module_info_get(struct mlxsw_sib *mlxsw_sib,
- u8 local_port, u8 *p_module,
- u8 *p_width)
-{
- char pmlp_pl[MLXSW_REG_PMLP_LEN];
- int err;
-
- mlxsw_reg_pmlp_pack(pmlp_pl, local_port);
- err = mlxsw_reg_query(mlxsw_sib->core, MLXSW_REG(pmlp), pmlp_pl);
- if (err)
- return err;
- *p_module = mlxsw_reg_pmlp_module_get(pmlp_pl, 0);
- *p_width = mlxsw_reg_pmlp_width_get(pmlp_pl);
- return 0;
-}
-
-static int mlxsw_sib_port_speed_set(struct mlxsw_sib_port *mlxsw_sib_port,
- u16 speed, u16 width)
-{
- struct mlxsw_sib *mlxsw_sib = mlxsw_sib_port->mlxsw_sib;
- char ptys_pl[MLXSW_REG_PTYS_LEN];
-
- mlxsw_reg_ptys_ib_pack(ptys_pl, mlxsw_sib_port->local_port, speed,
- width);
- return mlxsw_reg_write(mlxsw_sib->core, MLXSW_REG(ptys), ptys_pl);
-}
-
-static bool mlxsw_sib_port_created(struct mlxsw_sib *mlxsw_sib, u8 local_port)
-{
- return mlxsw_sib->ports[local_port] != NULL;
-}
-
-static int __mlxsw_sib_port_create(struct mlxsw_sib *mlxsw_sib, u8 local_port,
- u8 module, u8 width)
-{
- struct mlxsw_sib_port *mlxsw_sib_port;
- int err;
-
- mlxsw_sib_port = kzalloc(sizeof(*mlxsw_sib_port), GFP_KERNEL);
- if (!mlxsw_sib_port)
- return -ENOMEM;
- mlxsw_sib_port->mlxsw_sib = mlxsw_sib;
- mlxsw_sib_port->local_port = local_port;
- mlxsw_sib_port->mapping.module = module;
-
- err = mlxsw_sib_port_swid_set(mlxsw_sib_port, 0);
- if (err) {
- dev_err(mlxsw_sib->bus_info->dev, "Port %d: Failed to set SWID\n",
- mlxsw_sib_port->local_port);
- goto err_port_swid_set;
- }
-
- /* Expose the IB port number as it's front panel name */
- err = mlxsw_sib_port_set(mlxsw_sib_port, module + 1);
- if (err) {
- dev_err(mlxsw_sib->bus_info->dev, "Port %d: Failed to set IB port\n",
- mlxsw_sib_port->local_port);
- goto err_port_ib_set;
- }
-
- /* Supports all speeds from SDR to FDR (bitmask) and support bus width
- * of 1x, 2x and 4x (3 bits bitmask)
- */
- err = mlxsw_sib_port_speed_set(mlxsw_sib_port,
- MLXSW_REG_PTYS_IB_SPEED_EDR - 1,
- BIT(3) - 1);
- if (err) {
- dev_err(mlxsw_sib->bus_info->dev, "Port %d: Failed to set speed\n",
- mlxsw_sib_port->local_port);
- goto err_port_speed_set;
- }
-
- /* Change to the maximum MTU the device supports, the SMA will take
- * care of the active MTU
- */
- err = mlxsw_sib_port_mtu_set(mlxsw_sib_port, MLXSW_IB_DEFAULT_MTU);
- if (err) {
- dev_err(mlxsw_sib->bus_info->dev, "Port %d: Failed to set MTU\n",
- mlxsw_sib_port->local_port);
- goto err_port_mtu_set;
- }
-
- err = mlxsw_sib_port_admin_status_set(mlxsw_sib_port, true);
- if (err) {
- dev_err(mlxsw_sib->bus_info->dev, "Port %d: Failed to change admin state to UP\n",
- mlxsw_sib_port->local_port);
- goto err_port_admin_set;
- }
-
- mlxsw_core_port_ib_set(mlxsw_sib->core, mlxsw_sib_port->local_port,
- mlxsw_sib_port);
- mlxsw_sib->ports[local_port] = mlxsw_sib_port;
- return 0;
-
-err_port_admin_set:
-err_port_mtu_set:
-err_port_speed_set:
-err_port_ib_set:
- mlxsw_sib_port_swid_set(mlxsw_sib_port, MLXSW_PORT_SWID_DISABLED_PORT);
-err_port_swid_set:
- kfree(mlxsw_sib_port);
- return err;
-}
-
-static int mlxsw_sib_port_create(struct mlxsw_sib *mlxsw_sib, u8 local_port,
- u8 module, u8 width)
-{
- int err;
-
- err = mlxsw_core_port_init(mlxsw_sib->core, local_port,
- module + 1, false, 0, false, 0,
- mlxsw_sib->hw_id, sizeof(mlxsw_sib->hw_id));
- if (err) {
- dev_err(mlxsw_sib->bus_info->dev, "Port %d: Failed to init core port\n",
- local_port);
- return err;
- }
- err = __mlxsw_sib_port_create(mlxsw_sib, local_port, module, width);
- if (err)
- goto err_port_create;
-
- return 0;
-
-err_port_create:
- mlxsw_core_port_fini(mlxsw_sib->core, local_port);
- return err;
-}
-
-static void __mlxsw_sib_port_remove(struct mlxsw_sib *mlxsw_sib, u8 local_port)
-{
- struct mlxsw_sib_port *mlxsw_sib_port = mlxsw_sib->ports[local_port];
-
- mlxsw_core_port_clear(mlxsw_sib->core, local_port, mlxsw_sib);
- mlxsw_sib->ports[local_port] = NULL;
- mlxsw_sib_port_admin_status_set(mlxsw_sib_port, false);
- mlxsw_sib_port_swid_set(mlxsw_sib_port, MLXSW_PORT_SWID_DISABLED_PORT);
- kfree(mlxsw_sib_port);
-}
-
-static void mlxsw_sib_port_remove(struct mlxsw_sib *mlxsw_sib, u8 local_port)
-{
- __mlxsw_sib_port_remove(mlxsw_sib, local_port);
- mlxsw_core_port_fini(mlxsw_sib->core, local_port);
-}
-
-static void mlxsw_sib_ports_remove(struct mlxsw_sib *mlxsw_sib)
-{
- int i;
-
- for (i = 1; i < MLXSW_PORT_MAX_IB_PORTS; i++)
- if (mlxsw_sib_port_created(mlxsw_sib, i))
- mlxsw_sib_port_remove(mlxsw_sib, i);
- kfree(mlxsw_sib->ports);
-}
-
-static int mlxsw_sib_ports_create(struct mlxsw_sib *mlxsw_sib)
-{
- size_t alloc_size;
- u8 module, width;
- int i;
- int err;
-
- alloc_size = sizeof(struct mlxsw_sib_port *) * MLXSW_PORT_MAX_IB_PORTS;
- mlxsw_sib->ports = kzalloc(alloc_size, GFP_KERNEL);
- if (!mlxsw_sib->ports)
- return -ENOMEM;
-
- for (i = 1; i < MLXSW_PORT_MAX_IB_PORTS; i++) {
- err = mlxsw_sib_port_module_info_get(mlxsw_sib, i, &module,
- &width);
- if (err)
- goto err_port_module_info_get;
- if (!width)
- continue;
- err = mlxsw_sib_port_create(mlxsw_sib, i, module, width);
- if (err)
- goto err_port_create;
- }
- return 0;
-
-err_port_create:
-err_port_module_info_get:
- for (i--; i >= 1; i--)
- if (mlxsw_sib_port_created(mlxsw_sib, i))
- mlxsw_sib_port_remove(mlxsw_sib, i);
- kfree(mlxsw_sib->ports);
- return err;
-}
-
-static void
-mlxsw_sib_pude_ib_event_func(struct mlxsw_sib_port *mlxsw_sib_port,
- enum mlxsw_reg_pude_oper_status status)
-{
- if (status == MLXSW_PORT_OPER_STATUS_UP)
- pr_info("ib link for port %d - up\n",
- mlxsw_sib_port->mapping.module + 1);
- else
- pr_info("ib link for port %d - down\n",
- mlxsw_sib_port->mapping.module + 1);
-}
-
-static void mlxsw_sib_pude_event_func(const struct mlxsw_reg_info *reg,
- char *pude_pl, void *priv)
-{
- struct mlxsw_sib *mlxsw_sib = priv;
- struct mlxsw_sib_port *mlxsw_sib_port;
- enum mlxsw_reg_pude_oper_status status;
- u8 local_port;
-
- local_port = mlxsw_reg_pude_local_port_get(pude_pl);
- mlxsw_sib_port = mlxsw_sib->ports[local_port];
- if (!mlxsw_sib_port) {
- dev_warn(mlxsw_sib->bus_info->dev, "Port %d: Link event received for non-existent port\n",
- local_port);
- return;
- }
-
- status = mlxsw_reg_pude_oper_status_get(pude_pl);
- mlxsw_sib_pude_ib_event_func(mlxsw_sib_port, status);
-}
-
-static const struct mlxsw_listener mlxsw_sib_listener[] = {
- MLXSW_EVENTL(mlxsw_sib_pude_event_func, PUDE, EMAD),
-};
-
-static int mlxsw_sib_taps_init(struct mlxsw_sib *mlxsw_sib)
-{
- int i;
- int err;
-
- for (i = 0; i < ARRAY_SIZE(mlxsw_sib_listener); i++) {
- err = mlxsw_core_trap_register(mlxsw_sib->core,
- &mlxsw_sib_listener[i],
- mlxsw_sib);
- if (err)
- goto err_rx_listener_register;
- }
-
- return 0;
-
-err_rx_listener_register:
- for (i--; i >= 0; i--) {
- mlxsw_core_trap_unregister(mlxsw_sib->core,
- &mlxsw_sib_listener[i],
- mlxsw_sib);
- }
-
- return err;
-}
-
-static void mlxsw_sib_traps_fini(struct mlxsw_sib *mlxsw_sib)
-{
- int i;
-
- for (i = 0; i < ARRAY_SIZE(mlxsw_sib_listener); i++) {
- mlxsw_core_trap_unregister(mlxsw_sib->core,
- &mlxsw_sib_listener[i], mlxsw_sib);
- }
-}
-
-static int mlxsw_sib_basic_trap_groups_set(struct mlxsw_core *mlxsw_core)
-{
- char htgt_pl[MLXSW_REG_HTGT_LEN];
-
- mlxsw_reg_htgt_pack(htgt_pl, MLXSW_REG_HTGT_TRAP_GROUP_EMAD,
- MLXSW_REG_HTGT_INVALID_POLICER,
- MLXSW_REG_HTGT_DEFAULT_PRIORITY,
- MLXSW_REG_HTGT_DEFAULT_TC);
- mlxsw_reg_htgt_swid_set(htgt_pl, MLXSW_PORT_SWID_ALL_SWIDS);
- mlxsw_reg_htgt_local_path_rdq_set(htgt_pl,
- MLXSW_REG_HTGT_LOCAL_PATH_RDQ_SIB_EMAD);
- return mlxsw_reg_write(mlxsw_core, MLXSW_REG(htgt), htgt_pl);
-}
-
-static int mlxsw_sib_init(struct mlxsw_core *mlxsw_core,
- const struct mlxsw_bus_info *mlxsw_bus_info,
- struct netlink_ext_ack *extack)
-{
- struct mlxsw_sib *mlxsw_sib = mlxsw_core_driver_priv(mlxsw_core);
- int err;
-
- mlxsw_sib->core = mlxsw_core;
- mlxsw_sib->bus_info = mlxsw_bus_info;
-
- err = mlxsw_sib_hw_id_get(mlxsw_sib);
- if (err) {
- dev_err(mlxsw_sib->bus_info->dev, "Failed to get switch HW ID\n");
- return err;
- }
-
- err = mlxsw_sib_ports_create(mlxsw_sib);
- if (err) {
- dev_err(mlxsw_sib->bus_info->dev, "Failed to create ports\n");
- return err;
- }
-
- err = mlxsw_sib_taps_init(mlxsw_sib);
- if (err) {
- dev_err(mlxsw_sib->bus_info->dev, "Failed to set traps\n");
- goto err_traps_init_err;
- }
-
- return 0;
-
-err_traps_init_err:
- mlxsw_sib_ports_remove(mlxsw_sib);
- return err;
-}
-
-static void mlxsw_sib_fini(struct mlxsw_core *mlxsw_core)
-{
- struct mlxsw_sib *mlxsw_sib = mlxsw_core_driver_priv(mlxsw_core);
-
- mlxsw_sib_traps_fini(mlxsw_sib);
- mlxsw_sib_ports_remove(mlxsw_sib);
-}
-
-static const struct mlxsw_config_profile mlxsw_sib_config_profile = {
- .used_max_system_port = 1,
- .max_system_port = 48000,
- .used_max_ib_mc = 1,
- .max_ib_mc = 27,
- .used_max_pkey = 1,
- .max_pkey = 32,
- .swid_config = {
- {
- .used_type = 1,
- .type = MLXSW_PORT_SWID_TYPE_IB,
- }
- },
-};
-
-static struct mlxsw_driver mlxsw_sib_driver = {
- .kind = mlxsw_sib_driver_name,
- .priv_size = sizeof(struct mlxsw_sib),
- .init = mlxsw_sib_init,
- .fini = mlxsw_sib_fini,
- .basic_trap_groups_set = mlxsw_sib_basic_trap_groups_set,
- .txhdr_construct = mlxsw_sib_tx_v1_hdr_construct,
- .txhdr_len = MLXSW_TXHDR_LEN,
- .profile = &mlxsw_sib_config_profile,
-};
-
-static struct mlxsw_driver mlxsw_sib2_driver = {
- .kind = mlxsw_sib2_driver_name,
- .priv_size = sizeof(struct mlxsw_sib),
- .init = mlxsw_sib_init,
- .fini = mlxsw_sib_fini,
- .basic_trap_groups_set = mlxsw_sib_basic_trap_groups_set,
- .txhdr_construct = mlxsw_sib_tx_v1_hdr_construct,
- .txhdr_len = MLXSW_TXHDR_LEN,
- .profile = &mlxsw_sib_config_profile,
-};
-
-static const struct pci_device_id mlxsw_sib_pci_id_table[] = {
- {PCI_VDEVICE(MELLANOX, PCI_DEVICE_ID_MELLANOX_SWITCHIB), 0},
- {0, },
-};
-
-static struct pci_driver mlxsw_sib_pci_driver = {
- .name = mlxsw_sib_driver_name,
- .id_table = mlxsw_sib_pci_id_table,
-};
-
-static const struct pci_device_id mlxsw_sib2_pci_id_table[] = {
- {PCI_VDEVICE(MELLANOX, PCI_DEVICE_ID_MELLANOX_SWITCHIB2), 0},
- {0, },
-};
-
-static struct pci_driver mlxsw_sib2_pci_driver = {
- .name = mlxsw_sib2_driver_name,
- .id_table = mlxsw_sib2_pci_id_table,
-};
-
-static int __init mlxsw_sib_module_init(void)
-{
- int err;
-
- err = mlxsw_core_driver_register(&mlxsw_sib_driver);
- if (err)
- return err;
-
- err = mlxsw_core_driver_register(&mlxsw_sib2_driver);
- if (err)
- goto err_sib2_driver_register;
-
- err = mlxsw_pci_driver_register(&mlxsw_sib_pci_driver);
- if (err)
- goto err_sib_pci_driver_register;
-
- err = mlxsw_pci_driver_register(&mlxsw_sib2_pci_driver);
- if (err)
- goto err_sib2_pci_driver_register;
-
- return 0;
-
-err_sib2_pci_driver_register:
- mlxsw_pci_driver_unregister(&mlxsw_sib_pci_driver);
-err_sib_pci_driver_register:
- mlxsw_core_driver_unregister(&mlxsw_sib2_driver);
-err_sib2_driver_register:
- mlxsw_core_driver_unregister(&mlxsw_sib_driver);
- return err;
-}
-
-static void __exit mlxsw_sib_module_exit(void)
-{
- mlxsw_pci_driver_unregister(&mlxsw_sib2_pci_driver);
- mlxsw_pci_driver_unregister(&mlxsw_sib_pci_driver);
- mlxsw_core_driver_unregister(&mlxsw_sib2_driver);
- mlxsw_core_driver_unregister(&mlxsw_sib_driver);
-}
-
-module_init(mlxsw_sib_module_init);
-module_exit(mlxsw_sib_module_exit);
-
-MODULE_LICENSE("Dual BSD/GPL");
-MODULE_AUTHOR("Elad Raz <eladr@@mellanox.com>");
-MODULE_DESCRIPTION("Mellanox SwitchIB and SwitchIB-2 driver");
-MODULE_ALIAS("mlxsw_switchib2");
-MODULE_DEVICE_TABLE(pci, mlxsw_sib_pci_id_table);
-MODULE_DEVICE_TABLE(pci, mlxsw_sib2_pci_id_table);
+++ /dev/null
-// SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0
-/* Copyright (c) 2015-2018 Mellanox Technologies. All rights reserved */
-
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/types.h>
-#include <linux/pci.h>
-#include <linux/netdevice.h>
-#include <linux/ethtool.h>
-#include <linux/etherdevice.h>
-#include <linux/slab.h>
-#include <linux/device.h>
-#include <linux/skbuff.h>
-#include <linux/if_vlan.h>
-
-#include "pci.h"
-#include "core.h"
-#include "reg.h"
-#include "port.h"
-#include "trap.h"
-#include "txheader.h"
-#include "ib.h"
-
-static const char mlxsw_sx_driver_name[] = "mlxsw_switchx2";
-static const char mlxsw_sx_driver_version[] = "1.0";
-
-struct mlxsw_sx_port;
-
-struct mlxsw_sx {
- struct mlxsw_sx_port **ports;
- struct mlxsw_core *core;
- const struct mlxsw_bus_info *bus_info;
- u8 hw_id[ETH_ALEN];
-};
-
-struct mlxsw_sx_port_pcpu_stats {
- u64 rx_packets;
- u64 rx_bytes;
- u64 tx_packets;
- u64 tx_bytes;
- struct u64_stats_sync syncp;
- u32 tx_dropped;
-};
-
-struct mlxsw_sx_port {
- struct net_device *dev;
- struct mlxsw_sx_port_pcpu_stats __percpu *pcpu_stats;
- struct mlxsw_sx *mlxsw_sx;
- u8 local_port;
- struct {
- u8 module;
- } mapping;
-};
-
-/* tx_hdr_version
- * Tx header version.
- * Must be set to 0.
- */
-MLXSW_ITEM32(tx, hdr, version, 0x00, 28, 4);
-
-/* tx_hdr_ctl
- * Packet control type.
- * 0 - Ethernet control (e.g. EMADs, LACP)
- * 1 - Ethernet data
- */
-MLXSW_ITEM32(tx, hdr, ctl, 0x00, 26, 2);
-
-/* tx_hdr_proto
- * Packet protocol type. Must be set to 1 (Ethernet).
- */
-MLXSW_ITEM32(tx, hdr, proto, 0x00, 21, 3);
-
-/* tx_hdr_etclass
- * Egress TClass to be used on the egress device on the egress port.
- * The MSB is specified in the 'ctclass3' field.
- * Range is 0-15, where 15 is the highest priority.
- */
-MLXSW_ITEM32(tx, hdr, etclass, 0x00, 18, 3);
-
-/* tx_hdr_swid
- * Switch partition ID.
- */
-MLXSW_ITEM32(tx, hdr, swid, 0x00, 12, 3);
-
-/* tx_hdr_port_mid
- * Destination local port for unicast packets.
- * Destination multicast ID for multicast packets.
- *
- * Control packets are directed to a specific egress port, while data
- * packets are transmitted through the CPU port (0) into the switch partition,
- * where forwarding rules are applied.
- */
-MLXSW_ITEM32(tx, hdr, port_mid, 0x04, 16, 16);
-
-/* tx_hdr_ctclass3
- * See field 'etclass'.
- */
-MLXSW_ITEM32(tx, hdr, ctclass3, 0x04, 14, 1);
-
-/* tx_hdr_rdq
- * RDQ for control packets sent to remote CPU.
- * Must be set to 0x1F for EMADs, otherwise 0.
- */
-MLXSW_ITEM32(tx, hdr, rdq, 0x04, 9, 5);
-
-/* tx_hdr_cpu_sig
- * Signature control for packets going to CPU. Must be set to 0.
- */
-MLXSW_ITEM32(tx, hdr, cpu_sig, 0x04, 0, 9);
-
-/* tx_hdr_sig
- * Stacking protocl signature. Must be set to 0xE0E0.
- */
-MLXSW_ITEM32(tx, hdr, sig, 0x0C, 16, 16);
-
-/* tx_hdr_stclass
- * Stacking TClass.
- */
-MLXSW_ITEM32(tx, hdr, stclass, 0x0C, 13, 3);
-
-/* tx_hdr_emad
- * EMAD bit. Must be set for EMADs.
- */
-MLXSW_ITEM32(tx, hdr, emad, 0x0C, 5, 1);
-
-/* tx_hdr_type
- * 0 - Data packets
- * 6 - Control packets
- */
-MLXSW_ITEM32(tx, hdr, type, 0x0C, 0, 4);
-
-static void mlxsw_sx_txhdr_construct(struct sk_buff *skb,
- const struct mlxsw_tx_info *tx_info)
-{
- char *txhdr = skb_push(skb, MLXSW_TXHDR_LEN);
- bool is_emad = tx_info->is_emad;
-
- memset(txhdr, 0, MLXSW_TXHDR_LEN);
-
- /* We currently set default values for the egress tclass (QoS). */
- mlxsw_tx_hdr_version_set(txhdr, MLXSW_TXHDR_VERSION_0);
- mlxsw_tx_hdr_ctl_set(txhdr, MLXSW_TXHDR_ETH_CTL);
- mlxsw_tx_hdr_proto_set(txhdr, MLXSW_TXHDR_PROTO_ETH);
- mlxsw_tx_hdr_etclass_set(txhdr, is_emad ? MLXSW_TXHDR_ETCLASS_6 :
- MLXSW_TXHDR_ETCLASS_5);
- mlxsw_tx_hdr_swid_set(txhdr, 0);
- mlxsw_tx_hdr_port_mid_set(txhdr, tx_info->local_port);
- mlxsw_tx_hdr_ctclass3_set(txhdr, MLXSW_TXHDR_CTCLASS3);
- mlxsw_tx_hdr_rdq_set(txhdr, is_emad ? MLXSW_TXHDR_RDQ_EMAD :
- MLXSW_TXHDR_RDQ_OTHER);
- mlxsw_tx_hdr_cpu_sig_set(txhdr, MLXSW_TXHDR_CPU_SIG);
- mlxsw_tx_hdr_sig_set(txhdr, MLXSW_TXHDR_SIG);
- mlxsw_tx_hdr_stclass_set(txhdr, MLXSW_TXHDR_STCLASS_NONE);
- mlxsw_tx_hdr_emad_set(txhdr, is_emad ? MLXSW_TXHDR_EMAD :
- MLXSW_TXHDR_NOT_EMAD);
- mlxsw_tx_hdr_type_set(txhdr, MLXSW_TXHDR_TYPE_CONTROL);
-}
-
-static int mlxsw_sx_port_admin_status_set(struct mlxsw_sx_port *mlxsw_sx_port,
- bool is_up)
-{
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- char paos_pl[MLXSW_REG_PAOS_LEN];
-
- mlxsw_reg_paos_pack(paos_pl, mlxsw_sx_port->local_port,
- is_up ? MLXSW_PORT_ADMIN_STATUS_UP :
- MLXSW_PORT_ADMIN_STATUS_DOWN);
- return mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(paos), paos_pl);
-}
-
-static int mlxsw_sx_port_oper_status_get(struct mlxsw_sx_port *mlxsw_sx_port,
- bool *p_is_up)
-{
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- char paos_pl[MLXSW_REG_PAOS_LEN];
- u8 oper_status;
- int err;
-
- mlxsw_reg_paos_pack(paos_pl, mlxsw_sx_port->local_port, 0);
- err = mlxsw_reg_query(mlxsw_sx->core, MLXSW_REG(paos), paos_pl);
- if (err)
- return err;
- oper_status = mlxsw_reg_paos_oper_status_get(paos_pl);
- *p_is_up = oper_status == MLXSW_PORT_ADMIN_STATUS_UP;
- return 0;
-}
-
-static int __mlxsw_sx_port_mtu_set(struct mlxsw_sx_port *mlxsw_sx_port,
- u16 mtu)
-{
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- char pmtu_pl[MLXSW_REG_PMTU_LEN];
- int max_mtu;
- int err;
-
- mlxsw_reg_pmtu_pack(pmtu_pl, mlxsw_sx_port->local_port, 0);
- err = mlxsw_reg_query(mlxsw_sx->core, MLXSW_REG(pmtu), pmtu_pl);
- if (err)
- return err;
- max_mtu = mlxsw_reg_pmtu_max_mtu_get(pmtu_pl);
-
- if (mtu > max_mtu)
- return -EINVAL;
-
- mlxsw_reg_pmtu_pack(pmtu_pl, mlxsw_sx_port->local_port, mtu);
- return mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(pmtu), pmtu_pl);
-}
-
-static int mlxsw_sx_port_mtu_eth_set(struct mlxsw_sx_port *mlxsw_sx_port,
- u16 mtu)
-{
- mtu += MLXSW_TXHDR_LEN + ETH_HLEN;
- return __mlxsw_sx_port_mtu_set(mlxsw_sx_port, mtu);
-}
-
-static int mlxsw_sx_port_mtu_ib_set(struct mlxsw_sx_port *mlxsw_sx_port,
- u16 mtu)
-{
- return __mlxsw_sx_port_mtu_set(mlxsw_sx_port, mtu);
-}
-
-static int mlxsw_sx_port_ib_port_set(struct mlxsw_sx_port *mlxsw_sx_port,
- u8 ib_port)
-{
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- char plib_pl[MLXSW_REG_PLIB_LEN] = {0};
- int err;
-
- mlxsw_reg_plib_local_port_set(plib_pl, mlxsw_sx_port->local_port);
- mlxsw_reg_plib_ib_port_set(plib_pl, ib_port);
- err = mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(plib), plib_pl);
- return err;
-}
-
-static int mlxsw_sx_port_swid_set(struct mlxsw_sx_port *mlxsw_sx_port, u8 swid)
-{
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- char pspa_pl[MLXSW_REG_PSPA_LEN];
-
- mlxsw_reg_pspa_pack(pspa_pl, swid, mlxsw_sx_port->local_port);
- return mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(pspa), pspa_pl);
-}
-
-static int
-mlxsw_sx_port_system_port_mapping_set(struct mlxsw_sx_port *mlxsw_sx_port)
-{
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- char sspr_pl[MLXSW_REG_SSPR_LEN];
-
- mlxsw_reg_sspr_pack(sspr_pl, mlxsw_sx_port->local_port);
- return mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(sspr), sspr_pl);
-}
-
-static int mlxsw_sx_port_module_info_get(struct mlxsw_sx *mlxsw_sx,
- u8 local_port, u8 *p_module,
- u8 *p_width)
-{
- char pmlp_pl[MLXSW_REG_PMLP_LEN];
- int err;
-
- mlxsw_reg_pmlp_pack(pmlp_pl, local_port);
- err = mlxsw_reg_query(mlxsw_sx->core, MLXSW_REG(pmlp), pmlp_pl);
- if (err)
- return err;
- *p_module = mlxsw_reg_pmlp_module_get(pmlp_pl, 0);
- *p_width = mlxsw_reg_pmlp_width_get(pmlp_pl);
- return 0;
-}
-
-static int mlxsw_sx_port_open(struct net_device *dev)
-{
- struct mlxsw_sx_port *mlxsw_sx_port = netdev_priv(dev);
- int err;
-
- err = mlxsw_sx_port_admin_status_set(mlxsw_sx_port, true);
- if (err)
- return err;
- netif_start_queue(dev);
- return 0;
-}
-
-static int mlxsw_sx_port_stop(struct net_device *dev)
-{
- struct mlxsw_sx_port *mlxsw_sx_port = netdev_priv(dev);
-
- netif_stop_queue(dev);
- return mlxsw_sx_port_admin_status_set(mlxsw_sx_port, false);
-}
-
-static netdev_tx_t mlxsw_sx_port_xmit(struct sk_buff *skb,
- struct net_device *dev)
-{
- struct mlxsw_sx_port *mlxsw_sx_port = netdev_priv(dev);
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- struct mlxsw_sx_port_pcpu_stats *pcpu_stats;
- const struct mlxsw_tx_info tx_info = {
- .local_port = mlxsw_sx_port->local_port,
- .is_emad = false,
- };
- u64 len;
- int err;
-
- if (skb_cow_head(skb, MLXSW_TXHDR_LEN)) {
- this_cpu_inc(mlxsw_sx_port->pcpu_stats->tx_dropped);
- dev_kfree_skb_any(skb);
- return NETDEV_TX_OK;
- }
-
- memset(skb->cb, 0, sizeof(struct mlxsw_skb_cb));
-
- if (mlxsw_core_skb_transmit_busy(mlxsw_sx->core, &tx_info))
- return NETDEV_TX_BUSY;
-
- mlxsw_sx_txhdr_construct(skb, &tx_info);
- /* TX header is consumed by HW on the way so we shouldn't count its
- * bytes as being sent.
- */
- len = skb->len - MLXSW_TXHDR_LEN;
- /* Due to a race we might fail here because of a full queue. In that
- * unlikely case we simply drop the packet.
- */
- err = mlxsw_core_skb_transmit(mlxsw_sx->core, skb, &tx_info);
-
- if (!err) {
- pcpu_stats = this_cpu_ptr(mlxsw_sx_port->pcpu_stats);
- u64_stats_update_begin(&pcpu_stats->syncp);
- pcpu_stats->tx_packets++;
- pcpu_stats->tx_bytes += len;
- u64_stats_update_end(&pcpu_stats->syncp);
- } else {
- this_cpu_inc(mlxsw_sx_port->pcpu_stats->tx_dropped);
- dev_kfree_skb_any(skb);
- }
- return NETDEV_TX_OK;
-}
-
-static int mlxsw_sx_port_change_mtu(struct net_device *dev, int mtu)
-{
- struct mlxsw_sx_port *mlxsw_sx_port = netdev_priv(dev);
- int err;
-
- err = mlxsw_sx_port_mtu_eth_set(mlxsw_sx_port, mtu);
- if (err)
- return err;
- dev->mtu = mtu;
- return 0;
-}
-
-static void
-mlxsw_sx_port_get_stats64(struct net_device *dev,
- struct rtnl_link_stats64 *stats)
-{
- struct mlxsw_sx_port *mlxsw_sx_port = netdev_priv(dev);
- struct mlxsw_sx_port_pcpu_stats *p;
- u64 rx_packets, rx_bytes, tx_packets, tx_bytes;
- u32 tx_dropped = 0;
- unsigned int start;
- int i;
-
- for_each_possible_cpu(i) {
- p = per_cpu_ptr(mlxsw_sx_port->pcpu_stats, i);
- do {
- start = u64_stats_fetch_begin_irq(&p->syncp);
- rx_packets = p->rx_packets;
- rx_bytes = p->rx_bytes;
- tx_packets = p->tx_packets;
- tx_bytes = p->tx_bytes;
- } while (u64_stats_fetch_retry_irq(&p->syncp, start));
-
- stats->rx_packets += rx_packets;
- stats->rx_bytes += rx_bytes;
- stats->tx_packets += tx_packets;
- stats->tx_bytes += tx_bytes;
- /* tx_dropped is u32, updated without syncp protection. */
- tx_dropped += p->tx_dropped;
- }
- stats->tx_dropped = tx_dropped;
-}
-
-static struct devlink_port *
-mlxsw_sx_port_get_devlink_port(struct net_device *dev)
-{
- struct mlxsw_sx_port *mlxsw_sx_port = netdev_priv(dev);
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
-
- return mlxsw_core_port_devlink_port_get(mlxsw_sx->core,
- mlxsw_sx_port->local_port);
-}
-
-static const struct net_device_ops mlxsw_sx_port_netdev_ops = {
- .ndo_open = mlxsw_sx_port_open,
- .ndo_stop = mlxsw_sx_port_stop,
- .ndo_start_xmit = mlxsw_sx_port_xmit,
- .ndo_change_mtu = mlxsw_sx_port_change_mtu,
- .ndo_get_stats64 = mlxsw_sx_port_get_stats64,
- .ndo_get_devlink_port = mlxsw_sx_port_get_devlink_port,
-};
-
-static void mlxsw_sx_port_get_drvinfo(struct net_device *dev,
- struct ethtool_drvinfo *drvinfo)
-{
- struct mlxsw_sx_port *mlxsw_sx_port = netdev_priv(dev);
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
-
- strlcpy(drvinfo->driver, mlxsw_sx_driver_name, sizeof(drvinfo->driver));
- strlcpy(drvinfo->version, mlxsw_sx_driver_version,
- sizeof(drvinfo->version));
- snprintf(drvinfo->fw_version, sizeof(drvinfo->fw_version),
- "%d.%d.%d",
- mlxsw_sx->bus_info->fw_rev.major,
- mlxsw_sx->bus_info->fw_rev.minor,
- mlxsw_sx->bus_info->fw_rev.subminor);
- strlcpy(drvinfo->bus_info, mlxsw_sx->bus_info->device_name,
- sizeof(drvinfo->bus_info));
-}
-
-struct mlxsw_sx_port_hw_stats {
- char str[ETH_GSTRING_LEN];
- u64 (*getter)(const char *payload);
-};
-
-static const struct mlxsw_sx_port_hw_stats mlxsw_sx_port_hw_stats[] = {
- {
- .str = "a_frames_transmitted_ok",
- .getter = mlxsw_reg_ppcnt_a_frames_transmitted_ok_get,
- },
- {
- .str = "a_frames_received_ok",
- .getter = mlxsw_reg_ppcnt_a_frames_received_ok_get,
- },
- {
- .str = "a_frame_check_sequence_errors",
- .getter = mlxsw_reg_ppcnt_a_frame_check_sequence_errors_get,
- },
- {
- .str = "a_alignment_errors",
- .getter = mlxsw_reg_ppcnt_a_alignment_errors_get,
- },
- {
- .str = "a_octets_transmitted_ok",
- .getter = mlxsw_reg_ppcnt_a_octets_transmitted_ok_get,
- },
- {
- .str = "a_octets_received_ok",
- .getter = mlxsw_reg_ppcnt_a_octets_received_ok_get,
- },
- {
- .str = "a_multicast_frames_xmitted_ok",
- .getter = mlxsw_reg_ppcnt_a_multicast_frames_xmitted_ok_get,
- },
- {
- .str = "a_broadcast_frames_xmitted_ok",
- .getter = mlxsw_reg_ppcnt_a_broadcast_frames_xmitted_ok_get,
- },
- {
- .str = "a_multicast_frames_received_ok",
- .getter = mlxsw_reg_ppcnt_a_multicast_frames_received_ok_get,
- },
- {
- .str = "a_broadcast_frames_received_ok",
- .getter = mlxsw_reg_ppcnt_a_broadcast_frames_received_ok_get,
- },
- {
- .str = "a_in_range_length_errors",
- .getter = mlxsw_reg_ppcnt_a_in_range_length_errors_get,
- },
- {
- .str = "a_out_of_range_length_field",
- .getter = mlxsw_reg_ppcnt_a_out_of_range_length_field_get,
- },
- {
- .str = "a_frame_too_long_errors",
- .getter = mlxsw_reg_ppcnt_a_frame_too_long_errors_get,
- },
- {
- .str = "a_symbol_error_during_carrier",
- .getter = mlxsw_reg_ppcnt_a_symbol_error_during_carrier_get,
- },
- {
- .str = "a_mac_control_frames_transmitted",
- .getter = mlxsw_reg_ppcnt_a_mac_control_frames_transmitted_get,
- },
- {
- .str = "a_mac_control_frames_received",
- .getter = mlxsw_reg_ppcnt_a_mac_control_frames_received_get,
- },
- {
- .str = "a_unsupported_opcodes_received",
- .getter = mlxsw_reg_ppcnt_a_unsupported_opcodes_received_get,
- },
- {
- .str = "a_pause_mac_ctrl_frames_received",
- .getter = mlxsw_reg_ppcnt_a_pause_mac_ctrl_frames_received_get,
- },
- {
- .str = "a_pause_mac_ctrl_frames_xmitted",
- .getter = mlxsw_reg_ppcnt_a_pause_mac_ctrl_frames_transmitted_get,
- },
-};
-
-#define MLXSW_SX_PORT_HW_STATS_LEN ARRAY_SIZE(mlxsw_sx_port_hw_stats)
-
-static void mlxsw_sx_port_get_strings(struct net_device *dev,
- u32 stringset, u8 *data)
-{
- u8 *p = data;
- int i;
-
- switch (stringset) {
- case ETH_SS_STATS:
- for (i = 0; i < MLXSW_SX_PORT_HW_STATS_LEN; i++) {
- memcpy(p, mlxsw_sx_port_hw_stats[i].str,
- ETH_GSTRING_LEN);
- p += ETH_GSTRING_LEN;
- }
- break;
- }
-}
-
-static void mlxsw_sx_port_get_stats(struct net_device *dev,
- struct ethtool_stats *stats, u64 *data)
-{
- struct mlxsw_sx_port *mlxsw_sx_port = netdev_priv(dev);
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- char ppcnt_pl[MLXSW_REG_PPCNT_LEN];
- int i;
- int err;
-
- mlxsw_reg_ppcnt_pack(ppcnt_pl, mlxsw_sx_port->local_port,
- MLXSW_REG_PPCNT_IEEE_8023_CNT, 0);
- err = mlxsw_reg_query(mlxsw_sx->core, MLXSW_REG(ppcnt), ppcnt_pl);
- for (i = 0; i < MLXSW_SX_PORT_HW_STATS_LEN; i++)
- data[i] = !err ? mlxsw_sx_port_hw_stats[i].getter(ppcnt_pl) : 0;
-}
-
-static int mlxsw_sx_port_get_sset_count(struct net_device *dev, int sset)
-{
- switch (sset) {
- case ETH_SS_STATS:
- return MLXSW_SX_PORT_HW_STATS_LEN;
- default:
- return -EOPNOTSUPP;
- }
-}
-
-struct mlxsw_sx_port_link_mode {
- u32 mask;
- u32 supported;
- u32 advertised;
- u32 speed;
-};
-
-static const struct mlxsw_sx_port_link_mode mlxsw_sx_port_link_mode[] = {
- {
- .mask = MLXSW_REG_PTYS_ETH_SPEED_SGMII |
- MLXSW_REG_PTYS_ETH_SPEED_1000BASE_KX,
- .supported = SUPPORTED_1000baseKX_Full,
- .advertised = ADVERTISED_1000baseKX_Full,
- .speed = 1000,
- },
- {
- .mask = MLXSW_REG_PTYS_ETH_SPEED_10GBASE_CX4 |
- MLXSW_REG_PTYS_ETH_SPEED_10GBASE_KX4,
- .supported = SUPPORTED_10000baseKX4_Full,
- .advertised = ADVERTISED_10000baseKX4_Full,
- .speed = 10000,
- },
- {
- .mask = MLXSW_REG_PTYS_ETH_SPEED_10GBASE_KR |
- MLXSW_REG_PTYS_ETH_SPEED_10GBASE_CR |
- MLXSW_REG_PTYS_ETH_SPEED_10GBASE_SR |
- MLXSW_REG_PTYS_ETH_SPEED_10GBASE_ER_LR,
- .supported = SUPPORTED_10000baseKR_Full,
- .advertised = ADVERTISED_10000baseKR_Full,
- .speed = 10000,
- },
- {
- .mask = MLXSW_REG_PTYS_ETH_SPEED_40GBASE_CR4,
- .supported = SUPPORTED_40000baseCR4_Full,
- .advertised = ADVERTISED_40000baseCR4_Full,
- .speed = 40000,
- },
- {
- .mask = MLXSW_REG_PTYS_ETH_SPEED_40GBASE_KR4,
- .supported = SUPPORTED_40000baseKR4_Full,
- .advertised = ADVERTISED_40000baseKR4_Full,
- .speed = 40000,
- },
- {
- .mask = MLXSW_REG_PTYS_ETH_SPEED_40GBASE_SR4,
- .supported = SUPPORTED_40000baseSR4_Full,
- .advertised = ADVERTISED_40000baseSR4_Full,
- .speed = 40000,
- },
- {
- .mask = MLXSW_REG_PTYS_ETH_SPEED_40GBASE_LR4_ER4,
- .supported = SUPPORTED_40000baseLR4_Full,
- .advertised = ADVERTISED_40000baseLR4_Full,
- .speed = 40000,
- },
- {
- .mask = MLXSW_REG_PTYS_ETH_SPEED_25GBASE_CR |
- MLXSW_REG_PTYS_ETH_SPEED_25GBASE_KR |
- MLXSW_REG_PTYS_ETH_SPEED_25GBASE_SR,
- .speed = 25000,
- },
- {
- .mask = MLXSW_REG_PTYS_ETH_SPEED_50GBASE_KR4 |
- MLXSW_REG_PTYS_ETH_SPEED_50GBASE_CR2 |
- MLXSW_REG_PTYS_ETH_SPEED_50GBASE_KR2,
- .speed = 50000,
- },
- {
- .mask = MLXSW_REG_PTYS_ETH_SPEED_100GBASE_CR4 |
- MLXSW_REG_PTYS_ETH_SPEED_100GBASE_SR4 |
- MLXSW_REG_PTYS_ETH_SPEED_100GBASE_KR4 |
- MLXSW_REG_PTYS_ETH_SPEED_100GBASE_LR4_ER4,
- .speed = 100000,
- },
-};
-
-#define MLXSW_SX_PORT_LINK_MODE_LEN ARRAY_SIZE(mlxsw_sx_port_link_mode)
-#define MLXSW_SX_PORT_BASE_SPEED 10000 /* Mb/s */
-
-static u32 mlxsw_sx_from_ptys_supported_port(u32 ptys_eth_proto)
-{
- if (ptys_eth_proto & (MLXSW_REG_PTYS_ETH_SPEED_10GBASE_CR |
- MLXSW_REG_PTYS_ETH_SPEED_10GBASE_SR |
- MLXSW_REG_PTYS_ETH_SPEED_40GBASE_CR4 |
- MLXSW_REG_PTYS_ETH_SPEED_40GBASE_SR4 |
- MLXSW_REG_PTYS_ETH_SPEED_100GBASE_SR4 |
- MLXSW_REG_PTYS_ETH_SPEED_SGMII))
- return SUPPORTED_FIBRE;
-
- if (ptys_eth_proto & (MLXSW_REG_PTYS_ETH_SPEED_10GBASE_KR |
- MLXSW_REG_PTYS_ETH_SPEED_10GBASE_KX4 |
- MLXSW_REG_PTYS_ETH_SPEED_40GBASE_KR4 |
- MLXSW_REG_PTYS_ETH_SPEED_100GBASE_KR4 |
- MLXSW_REG_PTYS_ETH_SPEED_1000BASE_KX))
- return SUPPORTED_Backplane;
- return 0;
-}
-
-static u32 mlxsw_sx_from_ptys_supported_link(u32 ptys_eth_proto)
-{
- u32 modes = 0;
- int i;
-
- for (i = 0; i < MLXSW_SX_PORT_LINK_MODE_LEN; i++) {
- if (ptys_eth_proto & mlxsw_sx_port_link_mode[i].mask)
- modes |= mlxsw_sx_port_link_mode[i].supported;
- }
- return modes;
-}
-
-static u32 mlxsw_sx_from_ptys_advert_link(u32 ptys_eth_proto)
-{
- u32 modes = 0;
- int i;
-
- for (i = 0; i < MLXSW_SX_PORT_LINK_MODE_LEN; i++) {
- if (ptys_eth_proto & mlxsw_sx_port_link_mode[i].mask)
- modes |= mlxsw_sx_port_link_mode[i].advertised;
- }
- return modes;
-}
-
-static void mlxsw_sx_from_ptys_speed_duplex(bool carrier_ok, u32 ptys_eth_proto,
- struct ethtool_link_ksettings *cmd)
-{
- u32 speed = SPEED_UNKNOWN;
- u8 duplex = DUPLEX_UNKNOWN;
- int i;
-
- if (!carrier_ok)
- goto out;
-
- for (i = 0; i < MLXSW_SX_PORT_LINK_MODE_LEN; i++) {
- if (ptys_eth_proto & mlxsw_sx_port_link_mode[i].mask) {
- speed = mlxsw_sx_port_link_mode[i].speed;
- duplex = DUPLEX_FULL;
- break;
- }
- }
-out:
- cmd->base.speed = speed;
- cmd->base.duplex = duplex;
-}
-
-static u8 mlxsw_sx_port_connector_port(u32 ptys_eth_proto)
-{
- if (ptys_eth_proto & (MLXSW_REG_PTYS_ETH_SPEED_10GBASE_SR |
- MLXSW_REG_PTYS_ETH_SPEED_40GBASE_SR4 |
- MLXSW_REG_PTYS_ETH_SPEED_100GBASE_SR4 |
- MLXSW_REG_PTYS_ETH_SPEED_SGMII))
- return PORT_FIBRE;
-
- if (ptys_eth_proto & (MLXSW_REG_PTYS_ETH_SPEED_10GBASE_CR |
- MLXSW_REG_PTYS_ETH_SPEED_40GBASE_CR4 |
- MLXSW_REG_PTYS_ETH_SPEED_100GBASE_CR4))
- return PORT_DA;
-
- if (ptys_eth_proto & (MLXSW_REG_PTYS_ETH_SPEED_10GBASE_KR |
- MLXSW_REG_PTYS_ETH_SPEED_10GBASE_KX4 |
- MLXSW_REG_PTYS_ETH_SPEED_40GBASE_KR4 |
- MLXSW_REG_PTYS_ETH_SPEED_100GBASE_KR4))
- return PORT_NONE;
-
- return PORT_OTHER;
-}
-
-static int
-mlxsw_sx_port_get_link_ksettings(struct net_device *dev,
- struct ethtool_link_ksettings *cmd)
-{
- struct mlxsw_sx_port *mlxsw_sx_port = netdev_priv(dev);
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- char ptys_pl[MLXSW_REG_PTYS_LEN];
- u32 eth_proto_cap;
- u32 eth_proto_admin;
- u32 eth_proto_oper;
- u32 supported, advertising, lp_advertising;
- int err;
-
- mlxsw_reg_ptys_eth_pack(ptys_pl, mlxsw_sx_port->local_port, 0, false);
- err = mlxsw_reg_query(mlxsw_sx->core, MLXSW_REG(ptys), ptys_pl);
- if (err) {
- netdev_err(dev, "Failed to get proto");
- return err;
- }
- mlxsw_reg_ptys_eth_unpack(ptys_pl, ð_proto_cap,
- ð_proto_admin, ð_proto_oper);
-
- supported = mlxsw_sx_from_ptys_supported_port(eth_proto_cap) |
- mlxsw_sx_from_ptys_supported_link(eth_proto_cap) |
- SUPPORTED_Pause | SUPPORTED_Asym_Pause;
- advertising = mlxsw_sx_from_ptys_advert_link(eth_proto_admin);
- mlxsw_sx_from_ptys_speed_duplex(netif_carrier_ok(dev),
- eth_proto_oper, cmd);
-
- eth_proto_oper = eth_proto_oper ? eth_proto_oper : eth_proto_cap;
- cmd->base.port = mlxsw_sx_port_connector_port(eth_proto_oper);
- lp_advertising = mlxsw_sx_from_ptys_advert_link(eth_proto_oper);
-
- ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
- supported);
- ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
- advertising);
- ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.lp_advertising,
- lp_advertising);
-
- return 0;
-}
-
-static u32 mlxsw_sx_to_ptys_advert_link(u32 advertising)
-{
- u32 ptys_proto = 0;
- int i;
-
- for (i = 0; i < MLXSW_SX_PORT_LINK_MODE_LEN; i++) {
- if (advertising & mlxsw_sx_port_link_mode[i].advertised)
- ptys_proto |= mlxsw_sx_port_link_mode[i].mask;
- }
- return ptys_proto;
-}
-
-static u32 mlxsw_sx_to_ptys_speed(u32 speed)
-{
- u32 ptys_proto = 0;
- int i;
-
- for (i = 0; i < MLXSW_SX_PORT_LINK_MODE_LEN; i++) {
- if (speed == mlxsw_sx_port_link_mode[i].speed)
- ptys_proto |= mlxsw_sx_port_link_mode[i].mask;
- }
- return ptys_proto;
-}
-
-static u32 mlxsw_sx_to_ptys_upper_speed(u32 upper_speed)
-{
- u32 ptys_proto = 0;
- int i;
-
- for (i = 0; i < MLXSW_SX_PORT_LINK_MODE_LEN; i++) {
- if (mlxsw_sx_port_link_mode[i].speed <= upper_speed)
- ptys_proto |= mlxsw_sx_port_link_mode[i].mask;
- }
- return ptys_proto;
-}
-
-static int
-mlxsw_sx_port_set_link_ksettings(struct net_device *dev,
- const struct ethtool_link_ksettings *cmd)
-{
- struct mlxsw_sx_port *mlxsw_sx_port = netdev_priv(dev);
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- char ptys_pl[MLXSW_REG_PTYS_LEN];
- u32 speed;
- u32 eth_proto_new;
- u32 eth_proto_cap;
- u32 eth_proto_admin;
- u32 advertising;
- bool is_up;
- int err;
-
- speed = cmd->base.speed;
-
- ethtool_convert_link_mode_to_legacy_u32(&advertising,
- cmd->link_modes.advertising);
-
- eth_proto_new = cmd->base.autoneg == AUTONEG_ENABLE ?
- mlxsw_sx_to_ptys_advert_link(advertising) :
- mlxsw_sx_to_ptys_speed(speed);
-
- mlxsw_reg_ptys_eth_pack(ptys_pl, mlxsw_sx_port->local_port, 0, false);
- err = mlxsw_reg_query(mlxsw_sx->core, MLXSW_REG(ptys), ptys_pl);
- if (err) {
- netdev_err(dev, "Failed to get proto");
- return err;
- }
- mlxsw_reg_ptys_eth_unpack(ptys_pl, ð_proto_cap, ð_proto_admin,
- NULL);
-
- eth_proto_new = eth_proto_new & eth_proto_cap;
- if (!eth_proto_new) {
- netdev_err(dev, "Not supported proto admin requested");
- return -EINVAL;
- }
- if (eth_proto_new == eth_proto_admin)
- return 0;
-
- mlxsw_reg_ptys_eth_pack(ptys_pl, mlxsw_sx_port->local_port,
- eth_proto_new, true);
- err = mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(ptys), ptys_pl);
- if (err) {
- netdev_err(dev, "Failed to set proto admin");
- return err;
- }
-
- err = mlxsw_sx_port_oper_status_get(mlxsw_sx_port, &is_up);
- if (err) {
- netdev_err(dev, "Failed to get oper status");
- return err;
- }
- if (!is_up)
- return 0;
-
- err = mlxsw_sx_port_admin_status_set(mlxsw_sx_port, false);
- if (err) {
- netdev_err(dev, "Failed to set admin status");
- return err;
- }
-
- err = mlxsw_sx_port_admin_status_set(mlxsw_sx_port, true);
- if (err) {
- netdev_err(dev, "Failed to set admin status");
- return err;
- }
-
- return 0;
-}
-
-static const struct ethtool_ops mlxsw_sx_port_ethtool_ops = {
- .get_drvinfo = mlxsw_sx_port_get_drvinfo,
- .get_link = ethtool_op_get_link,
- .get_strings = mlxsw_sx_port_get_strings,
- .get_ethtool_stats = mlxsw_sx_port_get_stats,
- .get_sset_count = mlxsw_sx_port_get_sset_count,
- .get_link_ksettings = mlxsw_sx_port_get_link_ksettings,
- .set_link_ksettings = mlxsw_sx_port_set_link_ksettings,
-};
-
-static int mlxsw_sx_hw_id_get(struct mlxsw_sx *mlxsw_sx)
-{
- char spad_pl[MLXSW_REG_SPAD_LEN] = {0};
- int err;
-
- err = mlxsw_reg_query(mlxsw_sx->core, MLXSW_REG(spad), spad_pl);
- if (err)
- return err;
- mlxsw_reg_spad_base_mac_memcpy_from(spad_pl, mlxsw_sx->hw_id);
- return 0;
-}
-
-static int mlxsw_sx_port_dev_addr_get(struct mlxsw_sx_port *mlxsw_sx_port)
-{
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- struct net_device *dev = mlxsw_sx_port->dev;
- char ppad_pl[MLXSW_REG_PPAD_LEN];
- int err;
-
- mlxsw_reg_ppad_pack(ppad_pl, false, 0);
- err = mlxsw_reg_query(mlxsw_sx->core, MLXSW_REG(ppad), ppad_pl);
- if (err)
- return err;
- mlxsw_reg_ppad_mac_memcpy_from(ppad_pl, dev->dev_addr);
- /* The last byte value in base mac address is guaranteed
- * to be such it does not overflow when adding local_port
- * value.
- */
- dev->dev_addr[ETH_ALEN - 1] += mlxsw_sx_port->local_port;
- return 0;
-}
-
-static int mlxsw_sx_port_stp_state_set(struct mlxsw_sx_port *mlxsw_sx_port,
- u16 vid, enum mlxsw_reg_spms_state state)
-{
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- char *spms_pl;
- int err;
-
- spms_pl = kmalloc(MLXSW_REG_SPMS_LEN, GFP_KERNEL);
- if (!spms_pl)
- return -ENOMEM;
- mlxsw_reg_spms_pack(spms_pl, mlxsw_sx_port->local_port);
- mlxsw_reg_spms_vid_pack(spms_pl, vid, state);
- err = mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(spms), spms_pl);
- kfree(spms_pl);
- return err;
-}
-
-static int mlxsw_sx_port_ib_speed_set(struct mlxsw_sx_port *mlxsw_sx_port,
- u16 speed, u16 width)
-{
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- char ptys_pl[MLXSW_REG_PTYS_LEN];
-
- mlxsw_reg_ptys_ib_pack(ptys_pl, mlxsw_sx_port->local_port, speed,
- width);
- return mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(ptys), ptys_pl);
-}
-
-static int
-mlxsw_sx_port_speed_by_width_set(struct mlxsw_sx_port *mlxsw_sx_port, u8 width)
-{
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- u32 upper_speed = MLXSW_SX_PORT_BASE_SPEED * width;
- char ptys_pl[MLXSW_REG_PTYS_LEN];
- u32 eth_proto_admin;
-
- eth_proto_admin = mlxsw_sx_to_ptys_upper_speed(upper_speed);
- mlxsw_reg_ptys_eth_pack(ptys_pl, mlxsw_sx_port->local_port,
- eth_proto_admin, true);
- return mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(ptys), ptys_pl);
-}
-
-static int
-mlxsw_sx_port_mac_learning_mode_set(struct mlxsw_sx_port *mlxsw_sx_port,
- enum mlxsw_reg_spmlr_learn_mode mode)
-{
- struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx;
- char spmlr_pl[MLXSW_REG_SPMLR_LEN];
-
- mlxsw_reg_spmlr_pack(spmlr_pl, mlxsw_sx_port->local_port, mode);
- return mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(spmlr), spmlr_pl);
-}
-
-static int __mlxsw_sx_port_eth_create(struct mlxsw_sx *mlxsw_sx, u8 local_port,
- u8 module, u8 width)
-{
- struct mlxsw_sx_port *mlxsw_sx_port;
- struct net_device *dev;
- int err;
-
- dev = alloc_etherdev(sizeof(struct mlxsw_sx_port));
- if (!dev)
- return -ENOMEM;
- SET_NETDEV_DEV(dev, mlxsw_sx->bus_info->dev);
- dev_net_set(dev, mlxsw_core_net(mlxsw_sx->core));
- mlxsw_sx_port = netdev_priv(dev);
- mlxsw_sx_port->dev = dev;
- mlxsw_sx_port->mlxsw_sx = mlxsw_sx;
- mlxsw_sx_port->local_port = local_port;
- mlxsw_sx_port->mapping.module = module;
-
- mlxsw_sx_port->pcpu_stats =
- netdev_alloc_pcpu_stats(struct mlxsw_sx_port_pcpu_stats);
- if (!mlxsw_sx_port->pcpu_stats) {
- err = -ENOMEM;
- goto err_alloc_stats;
- }
-
- dev->netdev_ops = &mlxsw_sx_port_netdev_ops;
- dev->ethtool_ops = &mlxsw_sx_port_ethtool_ops;
-
- err = mlxsw_sx_port_dev_addr_get(mlxsw_sx_port);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Unable to get port mac address\n",
- mlxsw_sx_port->local_port);
- goto err_dev_addr_get;
- }
-
- netif_carrier_off(dev);
-
- dev->features |= NETIF_F_NETNS_LOCAL | NETIF_F_LLTX | NETIF_F_SG |
- NETIF_F_VLAN_CHALLENGED;
-
- dev->min_mtu = 0;
- dev->max_mtu = ETH_MAX_MTU;
-
- /* Each packet needs to have a Tx header (metadata) on top all other
- * headers.
- */
- dev->needed_headroom = MLXSW_TXHDR_LEN;
-
- err = mlxsw_sx_port_system_port_mapping_set(mlxsw_sx_port);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to set system port mapping\n",
- mlxsw_sx_port->local_port);
- goto err_port_system_port_mapping_set;
- }
-
- err = mlxsw_sx_port_swid_set(mlxsw_sx_port, 0);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to set SWID\n",
- mlxsw_sx_port->local_port);
- goto err_port_swid_set;
- }
-
- err = mlxsw_sx_port_speed_by_width_set(mlxsw_sx_port, width);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to set speed\n",
- mlxsw_sx_port->local_port);
- goto err_port_speed_set;
- }
-
- err = mlxsw_sx_port_mtu_eth_set(mlxsw_sx_port, ETH_DATA_LEN);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to set MTU\n",
- mlxsw_sx_port->local_port);
- goto err_port_mtu_set;
- }
-
- err = mlxsw_sx_port_admin_status_set(mlxsw_sx_port, false);
- if (err)
- goto err_port_admin_status_set;
-
- err = mlxsw_sx_port_stp_state_set(mlxsw_sx_port,
- MLXSW_PORT_DEFAULT_VID,
- MLXSW_REG_SPMS_STATE_FORWARDING);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to set STP state\n",
- mlxsw_sx_port->local_port);
- goto err_port_stp_state_set;
- }
-
- err = mlxsw_sx_port_mac_learning_mode_set(mlxsw_sx_port,
- MLXSW_REG_SPMLR_LEARN_MODE_DISABLE);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to set MAC learning mode\n",
- mlxsw_sx_port->local_port);
- goto err_port_mac_learning_mode_set;
- }
-
- err = register_netdev(dev);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to register netdev\n",
- mlxsw_sx_port->local_port);
- goto err_register_netdev;
- }
-
- mlxsw_core_port_eth_set(mlxsw_sx->core, mlxsw_sx_port->local_port,
- mlxsw_sx_port, dev);
- mlxsw_sx->ports[local_port] = mlxsw_sx_port;
- return 0;
-
-err_register_netdev:
-err_port_mac_learning_mode_set:
-err_port_stp_state_set:
-err_port_admin_status_set:
-err_port_mtu_set:
-err_port_speed_set:
- mlxsw_sx_port_swid_set(mlxsw_sx_port, MLXSW_PORT_SWID_DISABLED_PORT);
-err_port_swid_set:
-err_port_system_port_mapping_set:
-err_dev_addr_get:
- free_percpu(mlxsw_sx_port->pcpu_stats);
-err_alloc_stats:
- free_netdev(dev);
- return err;
-}
-
-static int mlxsw_sx_port_eth_create(struct mlxsw_sx *mlxsw_sx, u8 local_port,
- u8 module, u8 width)
-{
- int err;
-
- err = mlxsw_core_port_init(mlxsw_sx->core, local_port,
- module + 1, false, 0, false, 0,
- mlxsw_sx->hw_id, sizeof(mlxsw_sx->hw_id));
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to init core port\n",
- local_port);
- return err;
- }
- err = __mlxsw_sx_port_eth_create(mlxsw_sx, local_port, module, width);
- if (err)
- goto err_port_create;
-
- return 0;
-
-err_port_create:
- mlxsw_core_port_fini(mlxsw_sx->core, local_port);
- return err;
-}
-
-static void __mlxsw_sx_port_eth_remove(struct mlxsw_sx *mlxsw_sx, u8 local_port)
-{
- struct mlxsw_sx_port *mlxsw_sx_port = mlxsw_sx->ports[local_port];
-
- mlxsw_core_port_clear(mlxsw_sx->core, local_port, mlxsw_sx);
- unregister_netdev(mlxsw_sx_port->dev); /* This calls ndo_stop */
- mlxsw_sx->ports[local_port] = NULL;
- mlxsw_sx_port_swid_set(mlxsw_sx_port, MLXSW_PORT_SWID_DISABLED_PORT);
- free_percpu(mlxsw_sx_port->pcpu_stats);
- free_netdev(mlxsw_sx_port->dev);
-}
-
-static bool mlxsw_sx_port_created(struct mlxsw_sx *mlxsw_sx, u8 local_port)
-{
- return mlxsw_sx->ports[local_port] != NULL;
-}
-
-static int __mlxsw_sx_port_ib_create(struct mlxsw_sx *mlxsw_sx, u8 local_port,
- u8 module, u8 width)
-{
- struct mlxsw_sx_port *mlxsw_sx_port;
- int err;
-
- mlxsw_sx_port = kzalloc(sizeof(*mlxsw_sx_port), GFP_KERNEL);
- if (!mlxsw_sx_port)
- return -ENOMEM;
- mlxsw_sx_port->mlxsw_sx = mlxsw_sx;
- mlxsw_sx_port->local_port = local_port;
- mlxsw_sx_port->mapping.module = module;
-
- err = mlxsw_sx_port_system_port_mapping_set(mlxsw_sx_port);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to set system port mapping\n",
- mlxsw_sx_port->local_port);
- goto err_port_system_port_mapping_set;
- }
-
- /* Adding port to Infiniband swid (1) */
- err = mlxsw_sx_port_swid_set(mlxsw_sx_port, 1);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to set SWID\n",
- mlxsw_sx_port->local_port);
- goto err_port_swid_set;
- }
-
- /* Expose the IB port number as it's front panel name */
- err = mlxsw_sx_port_ib_port_set(mlxsw_sx_port, module + 1);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to set IB port\n",
- mlxsw_sx_port->local_port);
- goto err_port_ib_set;
- }
-
- /* Supports all speeds from SDR to FDR (bitmask) and support bus width
- * of 1x, 2x and 4x (3 bits bitmask)
- */
- err = mlxsw_sx_port_ib_speed_set(mlxsw_sx_port,
- MLXSW_REG_PTYS_IB_SPEED_EDR - 1,
- BIT(3) - 1);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to set speed\n",
- mlxsw_sx_port->local_port);
- goto err_port_speed_set;
- }
-
- /* Change to the maximum MTU the device supports, the SMA will take
- * care of the active MTU
- */
- err = mlxsw_sx_port_mtu_ib_set(mlxsw_sx_port, MLXSW_IB_DEFAULT_MTU);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to set MTU\n",
- mlxsw_sx_port->local_port);
- goto err_port_mtu_set;
- }
-
- err = mlxsw_sx_port_admin_status_set(mlxsw_sx_port, true);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to change admin state to UP\n",
- mlxsw_sx_port->local_port);
- goto err_port_admin_set;
- }
-
- mlxsw_core_port_ib_set(mlxsw_sx->core, mlxsw_sx_port->local_port,
- mlxsw_sx_port);
- mlxsw_sx->ports[local_port] = mlxsw_sx_port;
- return 0;
-
-err_port_admin_set:
-err_port_mtu_set:
-err_port_speed_set:
-err_port_ib_set:
- mlxsw_sx_port_swid_set(mlxsw_sx_port, MLXSW_PORT_SWID_DISABLED_PORT);
-err_port_swid_set:
-err_port_system_port_mapping_set:
- kfree(mlxsw_sx_port);
- return err;
-}
-
-static void __mlxsw_sx_port_ib_remove(struct mlxsw_sx *mlxsw_sx, u8 local_port)
-{
- struct mlxsw_sx_port *mlxsw_sx_port = mlxsw_sx->ports[local_port];
-
- mlxsw_core_port_clear(mlxsw_sx->core, local_port, mlxsw_sx);
- mlxsw_sx->ports[local_port] = NULL;
- mlxsw_sx_port_admin_status_set(mlxsw_sx_port, false);
- mlxsw_sx_port_swid_set(mlxsw_sx_port, MLXSW_PORT_SWID_DISABLED_PORT);
- kfree(mlxsw_sx_port);
-}
-
-static void __mlxsw_sx_port_remove(struct mlxsw_sx *mlxsw_sx, u8 local_port)
-{
- enum devlink_port_type port_type =
- mlxsw_core_port_type_get(mlxsw_sx->core, local_port);
-
- if (port_type == DEVLINK_PORT_TYPE_ETH)
- __mlxsw_sx_port_eth_remove(mlxsw_sx, local_port);
- else if (port_type == DEVLINK_PORT_TYPE_IB)
- __mlxsw_sx_port_ib_remove(mlxsw_sx, local_port);
-}
-
-static void mlxsw_sx_port_remove(struct mlxsw_sx *mlxsw_sx, u8 local_port)
-{
- __mlxsw_sx_port_remove(mlxsw_sx, local_port);
- mlxsw_core_port_fini(mlxsw_sx->core, local_port);
-}
-
-static void mlxsw_sx_ports_remove(struct mlxsw_sx *mlxsw_sx)
-{
- int i;
-
- for (i = 1; i < mlxsw_core_max_ports(mlxsw_sx->core); i++)
- if (mlxsw_sx_port_created(mlxsw_sx, i))
- mlxsw_sx_port_remove(mlxsw_sx, i);
- kfree(mlxsw_sx->ports);
- mlxsw_sx->ports = NULL;
-}
-
-static int mlxsw_sx_ports_create(struct mlxsw_sx *mlxsw_sx)
-{
- unsigned int max_ports = mlxsw_core_max_ports(mlxsw_sx->core);
- size_t alloc_size;
- u8 module, width;
- int i;
- int err;
-
- alloc_size = sizeof(struct mlxsw_sx_port *) * max_ports;
- mlxsw_sx->ports = kzalloc(alloc_size, GFP_KERNEL);
- if (!mlxsw_sx->ports)
- return -ENOMEM;
-
- for (i = 1; i < max_ports; i++) {
- err = mlxsw_sx_port_module_info_get(mlxsw_sx, i, &module,
- &width);
- if (err)
- goto err_port_module_info_get;
- if (!width)
- continue;
- err = mlxsw_sx_port_eth_create(mlxsw_sx, i, module, width);
- if (err)
- goto err_port_create;
- }
- return 0;
-
-err_port_create:
-err_port_module_info_get:
- for (i--; i >= 1; i--)
- if (mlxsw_sx_port_created(mlxsw_sx, i))
- mlxsw_sx_port_remove(mlxsw_sx, i);
- kfree(mlxsw_sx->ports);
- mlxsw_sx->ports = NULL;
- return err;
-}
-
-static void mlxsw_sx_pude_eth_event_func(struct mlxsw_sx_port *mlxsw_sx_port,
- enum mlxsw_reg_pude_oper_status status)
-{
- if (status == MLXSW_PORT_OPER_STATUS_UP) {
- netdev_info(mlxsw_sx_port->dev, "link up\n");
- netif_carrier_on(mlxsw_sx_port->dev);
- } else {
- netdev_info(mlxsw_sx_port->dev, "link down\n");
- netif_carrier_off(mlxsw_sx_port->dev);
- }
-}
-
-static void mlxsw_sx_pude_ib_event_func(struct mlxsw_sx_port *mlxsw_sx_port,
- enum mlxsw_reg_pude_oper_status status)
-{
- if (status == MLXSW_PORT_OPER_STATUS_UP)
- pr_info("ib link for port %d - up\n",
- mlxsw_sx_port->mapping.module + 1);
- else
- pr_info("ib link for port %d - down\n",
- mlxsw_sx_port->mapping.module + 1);
-}
-
-static void mlxsw_sx_pude_event_func(const struct mlxsw_reg_info *reg,
- char *pude_pl, void *priv)
-{
- struct mlxsw_sx *mlxsw_sx = priv;
- struct mlxsw_sx_port *mlxsw_sx_port;
- enum mlxsw_reg_pude_oper_status status;
- enum devlink_port_type port_type;
- u8 local_port;
-
- local_port = mlxsw_reg_pude_local_port_get(pude_pl);
- mlxsw_sx_port = mlxsw_sx->ports[local_port];
- if (!mlxsw_sx_port) {
- dev_warn(mlxsw_sx->bus_info->dev, "Port %d: Link event received for non-existent port\n",
- local_port);
- return;
- }
-
- status = mlxsw_reg_pude_oper_status_get(pude_pl);
- port_type = mlxsw_core_port_type_get(mlxsw_sx->core, local_port);
- if (port_type == DEVLINK_PORT_TYPE_ETH)
- mlxsw_sx_pude_eth_event_func(mlxsw_sx_port, status);
- else if (port_type == DEVLINK_PORT_TYPE_IB)
- mlxsw_sx_pude_ib_event_func(mlxsw_sx_port, status);
-}
-
-static void mlxsw_sx_rx_listener_func(struct sk_buff *skb, u8 local_port,
- void *priv)
-{
- struct mlxsw_sx *mlxsw_sx = priv;
- struct mlxsw_sx_port *mlxsw_sx_port = mlxsw_sx->ports[local_port];
- struct mlxsw_sx_port_pcpu_stats *pcpu_stats;
-
- if (unlikely(!mlxsw_sx_port)) {
- dev_warn_ratelimited(mlxsw_sx->bus_info->dev, "Port %d: skb received for non-existent port\n",
- local_port);
- return;
- }
-
- skb->dev = mlxsw_sx_port->dev;
-
- pcpu_stats = this_cpu_ptr(mlxsw_sx_port->pcpu_stats);
- u64_stats_update_begin(&pcpu_stats->syncp);
- pcpu_stats->rx_packets++;
- pcpu_stats->rx_bytes += skb->len;
- u64_stats_update_end(&pcpu_stats->syncp);
-
- skb->protocol = eth_type_trans(skb, skb->dev);
- netif_receive_skb(skb);
-}
-
-static int mlxsw_sx_port_type_set(struct mlxsw_core *mlxsw_core, u8 local_port,
- enum devlink_port_type new_type)
-{
- struct mlxsw_sx *mlxsw_sx = mlxsw_core_driver_priv(mlxsw_core);
- u8 module, width;
- int err;
-
- if (!mlxsw_sx->ports || !mlxsw_sx->ports[local_port]) {
- dev_err(mlxsw_sx->bus_info->dev, "Port number \"%d\" does not exist\n",
- local_port);
- return -EINVAL;
- }
-
- if (new_type == DEVLINK_PORT_TYPE_AUTO)
- return -EOPNOTSUPP;
-
- __mlxsw_sx_port_remove(mlxsw_sx, local_port);
- err = mlxsw_sx_port_module_info_get(mlxsw_sx, local_port, &module,
- &width);
- if (err)
- goto err_port_module_info_get;
-
- if (new_type == DEVLINK_PORT_TYPE_ETH)
- err = __mlxsw_sx_port_eth_create(mlxsw_sx, local_port, module,
- width);
- else if (new_type == DEVLINK_PORT_TYPE_IB)
- err = __mlxsw_sx_port_ib_create(mlxsw_sx, local_port, module,
- width);
-
-err_port_module_info_get:
- return err;
-}
-
-enum {
- MLXSW_REG_HTGT_TRAP_GROUP_SX2_RX = 1,
- MLXSW_REG_HTGT_TRAP_GROUP_SX2_CTRL = 2,
-};
-
-#define MLXSW_SX_RXL(_trap_id) \
- MLXSW_RXL(mlxsw_sx_rx_listener_func, _trap_id, TRAP_TO_CPU, \
- false, SX2_RX, FORWARD)
-
-static const struct mlxsw_listener mlxsw_sx_listener[] = {
- MLXSW_EVENTL(mlxsw_sx_pude_event_func, PUDE, EMAD),
- MLXSW_SX_RXL(FDB_MC),
- MLXSW_SX_RXL(STP),
- MLXSW_SX_RXL(LACP),
- MLXSW_SX_RXL(EAPOL),
- MLXSW_SX_RXL(LLDP),
- MLXSW_SX_RXL(MMRP),
- MLXSW_SX_RXL(MVRP),
- MLXSW_SX_RXL(RPVST),
- MLXSW_SX_RXL(DHCP),
- MLXSW_SX_RXL(IGMP_QUERY),
- MLXSW_SX_RXL(IGMP_V1_REPORT),
- MLXSW_SX_RXL(IGMP_V2_REPORT),
- MLXSW_SX_RXL(IGMP_V2_LEAVE),
- MLXSW_SX_RXL(IGMP_V3_REPORT),
-};
-
-static int mlxsw_sx_traps_init(struct mlxsw_sx *mlxsw_sx)
-{
- char htgt_pl[MLXSW_REG_HTGT_LEN];
- int i;
- int err;
-
- mlxsw_reg_htgt_pack(htgt_pl, MLXSW_REG_HTGT_TRAP_GROUP_SX2_RX,
- MLXSW_REG_HTGT_INVALID_POLICER,
- MLXSW_REG_HTGT_DEFAULT_PRIORITY,
- MLXSW_REG_HTGT_DEFAULT_TC);
- mlxsw_reg_htgt_local_path_rdq_set(htgt_pl,
- MLXSW_REG_HTGT_LOCAL_PATH_RDQ_SX2_RX);
-
- err = mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(htgt), htgt_pl);
- if (err)
- return err;
-
- mlxsw_reg_htgt_pack(htgt_pl, MLXSW_REG_HTGT_TRAP_GROUP_SX2_CTRL,
- MLXSW_REG_HTGT_INVALID_POLICER,
- MLXSW_REG_HTGT_DEFAULT_PRIORITY,
- MLXSW_REG_HTGT_DEFAULT_TC);
- mlxsw_reg_htgt_local_path_rdq_set(htgt_pl,
- MLXSW_REG_HTGT_LOCAL_PATH_RDQ_SX2_CTRL);
-
- err = mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(htgt), htgt_pl);
- if (err)
- return err;
-
- for (i = 0; i < ARRAY_SIZE(mlxsw_sx_listener); i++) {
- err = mlxsw_core_trap_register(mlxsw_sx->core,
- &mlxsw_sx_listener[i],
- mlxsw_sx);
- if (err)
- goto err_listener_register;
-
- }
- return 0;
-
-err_listener_register:
- for (i--; i >= 0; i--) {
- mlxsw_core_trap_unregister(mlxsw_sx->core,
- &mlxsw_sx_listener[i],
- mlxsw_sx);
- }
- return err;
-}
-
-static void mlxsw_sx_traps_fini(struct mlxsw_sx *mlxsw_sx)
-{
- int i;
-
- for (i = 0; i < ARRAY_SIZE(mlxsw_sx_listener); i++) {
- mlxsw_core_trap_unregister(mlxsw_sx->core,
- &mlxsw_sx_listener[i],
- mlxsw_sx);
- }
-}
-
-static int mlxsw_sx_flood_init(struct mlxsw_sx *mlxsw_sx)
-{
- char sfgc_pl[MLXSW_REG_SFGC_LEN];
- char sgcr_pl[MLXSW_REG_SGCR_LEN];
- char *sftr_pl;
- int err;
-
- /* Configure a flooding table, which includes only CPU port. */
- sftr_pl = kmalloc(MLXSW_REG_SFTR_LEN, GFP_KERNEL);
- if (!sftr_pl)
- return -ENOMEM;
- mlxsw_reg_sftr_pack(sftr_pl, 0, 0, MLXSW_REG_SFGC_TABLE_TYPE_SINGLE, 0,
- MLXSW_PORT_CPU_PORT, true);
- err = mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(sftr), sftr_pl);
- kfree(sftr_pl);
- if (err)
- return err;
-
- /* Flood different packet types using the flooding table. */
- mlxsw_reg_sfgc_pack(sfgc_pl,
- MLXSW_REG_SFGC_TYPE_UNKNOWN_UNICAST,
- MLXSW_REG_SFGC_BRIDGE_TYPE_1Q_FID,
- MLXSW_REG_SFGC_TABLE_TYPE_SINGLE,
- 0);
- err = mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(sfgc), sfgc_pl);
- if (err)
- return err;
-
- mlxsw_reg_sfgc_pack(sfgc_pl,
- MLXSW_REG_SFGC_TYPE_BROADCAST,
- MLXSW_REG_SFGC_BRIDGE_TYPE_1Q_FID,
- MLXSW_REG_SFGC_TABLE_TYPE_SINGLE,
- 0);
- err = mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(sfgc), sfgc_pl);
- if (err)
- return err;
-
- mlxsw_reg_sfgc_pack(sfgc_pl,
- MLXSW_REG_SFGC_TYPE_UNREGISTERED_MULTICAST_NON_IP,
- MLXSW_REG_SFGC_BRIDGE_TYPE_1Q_FID,
- MLXSW_REG_SFGC_TABLE_TYPE_SINGLE,
- 0);
- err = mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(sfgc), sfgc_pl);
- if (err)
- return err;
-
- mlxsw_reg_sfgc_pack(sfgc_pl,
- MLXSW_REG_SFGC_TYPE_UNREGISTERED_MULTICAST_IPV6,
- MLXSW_REG_SFGC_BRIDGE_TYPE_1Q_FID,
- MLXSW_REG_SFGC_TABLE_TYPE_SINGLE,
- 0);
- err = mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(sfgc), sfgc_pl);
- if (err)
- return err;
-
- mlxsw_reg_sfgc_pack(sfgc_pl,
- MLXSW_REG_SFGC_TYPE_UNREGISTERED_MULTICAST_IPV4,
- MLXSW_REG_SFGC_BRIDGE_TYPE_1Q_FID,
- MLXSW_REG_SFGC_TABLE_TYPE_SINGLE,
- 0);
- err = mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(sfgc), sfgc_pl);
- if (err)
- return err;
-
- mlxsw_reg_sgcr_pack(sgcr_pl, true);
- return mlxsw_reg_write(mlxsw_sx->core, MLXSW_REG(sgcr), sgcr_pl);
-}
-
-static int mlxsw_sx_basic_trap_groups_set(struct mlxsw_core *mlxsw_core)
-{
- char htgt_pl[MLXSW_REG_HTGT_LEN];
-
- mlxsw_reg_htgt_pack(htgt_pl, MLXSW_REG_HTGT_TRAP_GROUP_EMAD,
- MLXSW_REG_HTGT_INVALID_POLICER,
- MLXSW_REG_HTGT_DEFAULT_PRIORITY,
- MLXSW_REG_HTGT_DEFAULT_TC);
- mlxsw_reg_htgt_swid_set(htgt_pl, MLXSW_PORT_SWID_ALL_SWIDS);
- mlxsw_reg_htgt_local_path_rdq_set(htgt_pl,
- MLXSW_REG_HTGT_LOCAL_PATH_RDQ_SX2_EMAD);
- return mlxsw_reg_write(mlxsw_core, MLXSW_REG(htgt), htgt_pl);
-}
-
-static int mlxsw_sx_init(struct mlxsw_core *mlxsw_core,
- const struct mlxsw_bus_info *mlxsw_bus_info,
- struct netlink_ext_ack *extack)
-{
- struct mlxsw_sx *mlxsw_sx = mlxsw_core_driver_priv(mlxsw_core);
- int err;
-
- mlxsw_sx->core = mlxsw_core;
- mlxsw_sx->bus_info = mlxsw_bus_info;
-
- err = mlxsw_sx_hw_id_get(mlxsw_sx);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Failed to get switch HW ID\n");
- return err;
- }
-
- err = mlxsw_sx_ports_create(mlxsw_sx);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Failed to create ports\n");
- return err;
- }
-
- err = mlxsw_sx_traps_init(mlxsw_sx);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Failed to set traps\n");
- goto err_listener_register;
- }
-
- err = mlxsw_sx_flood_init(mlxsw_sx);
- if (err) {
- dev_err(mlxsw_sx->bus_info->dev, "Failed to initialize flood tables\n");
- goto err_flood_init;
- }
-
- return 0;
-
-err_flood_init:
- mlxsw_sx_traps_fini(mlxsw_sx);
-err_listener_register:
- mlxsw_sx_ports_remove(mlxsw_sx);
- return err;
-}
-
-static void mlxsw_sx_fini(struct mlxsw_core *mlxsw_core)
-{
- struct mlxsw_sx *mlxsw_sx = mlxsw_core_driver_priv(mlxsw_core);
-
- mlxsw_sx_traps_fini(mlxsw_sx);
- mlxsw_sx_ports_remove(mlxsw_sx);
-}
-
-static const struct mlxsw_config_profile mlxsw_sx_config_profile = {
- .used_max_vepa_channels = 1,
- .max_vepa_channels = 0,
- .used_max_mid = 1,
- .max_mid = 7000,
- .used_max_pgt = 1,
- .max_pgt = 0,
- .used_max_system_port = 1,
- .max_system_port = 48000,
- .used_max_vlan_groups = 1,
- .max_vlan_groups = 127,
- .used_max_regions = 1,
- .max_regions = 400,
- .used_flood_tables = 1,
- .max_flood_tables = 2,
- .max_vid_flood_tables = 1,
- .used_flood_mode = 1,
- .flood_mode = 3,
- .used_max_ib_mc = 1,
- .max_ib_mc = 6,
- .used_max_pkey = 1,
- .max_pkey = 0,
- .swid_config = {
- {
- .used_type = 1,
- .type = MLXSW_PORT_SWID_TYPE_ETH,
- },
- {
- .used_type = 1,
- .type = MLXSW_PORT_SWID_TYPE_IB,
- }
- },
-};
-
-static struct mlxsw_driver mlxsw_sx_driver = {
- .kind = mlxsw_sx_driver_name,
- .priv_size = sizeof(struct mlxsw_sx),
- .init = mlxsw_sx_init,
- .fini = mlxsw_sx_fini,
- .basic_trap_groups_set = mlxsw_sx_basic_trap_groups_set,
- .txhdr_construct = mlxsw_sx_txhdr_construct,
- .txhdr_len = MLXSW_TXHDR_LEN,
- .profile = &mlxsw_sx_config_profile,
- .port_type_set = mlxsw_sx_port_type_set,
-};
-
-static const struct pci_device_id mlxsw_sx_pci_id_table[] = {
- {PCI_VDEVICE(MELLANOX, PCI_DEVICE_ID_MELLANOX_SWITCHX2), 0},
- {0, },
-};
-
-static struct pci_driver mlxsw_sx_pci_driver = {
- .name = mlxsw_sx_driver_name,
- .id_table = mlxsw_sx_pci_id_table,
-};
-
-static int __init mlxsw_sx_module_init(void)
-{
- int err;
-
- err = mlxsw_core_driver_register(&mlxsw_sx_driver);
- if (err)
- return err;
-
- err = mlxsw_pci_driver_register(&mlxsw_sx_pci_driver);
- if (err)
- goto err_pci_driver_register;
-
- return 0;
-
-err_pci_driver_register:
- mlxsw_core_driver_unregister(&mlxsw_sx_driver);
- return err;
-}
-
-static void __exit mlxsw_sx_module_exit(void)
-{
- mlxsw_pci_driver_unregister(&mlxsw_sx_pci_driver);
- mlxsw_core_driver_unregister(&mlxsw_sx_driver);
-}
-
-module_init(mlxsw_sx_module_init);
-module_exit(mlxsw_sx_module_exit);
-
-MODULE_LICENSE("Dual BSD/GPL");
-MODULE_AUTHOR("Jiri Pirko <jiri@mellanox.com>");
-MODULE_DESCRIPTION("Mellanox SwitchX-2 driver");
-MODULE_DEVICE_TABLE(pci, mlxsw_sx_pci_id_table);
unsigned i;
iomem = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!iomem) {
+ dev_err(&pdev->dev, "Invalid resource\n");
+ return -EINVAL;
+ }
if (!request_mem_region(iomem->start, resource_size(iomem), DRV_NAME))
goto err_mem_region;
*
* Read and check the TX/RX memory selftest information.
*/
-static int ks8851_read_selftest(struct ks8851_net *ks)
+static void ks8851_read_selftest(struct ks8851_net *ks)
{
unsigned both_done = MBIR_TXMBF | MBIR_RXMBF;
- int ret = 0;
unsigned rd;
rd = ks8851_rdreg16(ks, KS_MBIR);
if ((rd & both_done) != both_done) {
netdev_warn(ks->netdev, "Memory selftest not finished\n");
- return 0;
+ return;
}
- if (rd & MBIR_TXMBFA) {
+ if (rd & MBIR_TXMBFA)
netdev_err(ks->netdev, "TX memory selftest fail\n");
- ret |= 1;
- }
- if (rd & MBIR_RXMBFA) {
+ if (rd & MBIR_RXMBFA)
netdev_err(ks->netdev, "RX memory selftest fail\n");
- ret |= 2;
- }
-
- return 0;
}
/* driver bus management functions */
#include <linux/crc32.h>
#include <linux/sched.h>
#include <linux/slab.h>
+#include <linux/micrel_phy.h>
/* DMA Registers */
#define KS884X_PHY_CTRL_OFFSET 0x00
-/* Mode Control Register */
-#define PHY_REG_CTRL 0
-
-#define PHY_RESET 0x8000
-#define PHY_LOOPBACK 0x4000
-#define PHY_SPEED_100MBIT 0x2000
-#define PHY_AUTO_NEG_ENABLE 0x1000
-#define PHY_POWER_DOWN 0x0800
-#define PHY_MII_DISABLE 0x0400
-#define PHY_AUTO_NEG_RESTART 0x0200
-#define PHY_FULL_DUPLEX 0x0100
-#define PHY_COLLISION_TEST 0x0080
-#define PHY_HP_MDIX 0x0020
-#define PHY_FORCE_MDIX 0x0010
-#define PHY_AUTO_MDIX_DISABLE 0x0008
-#define PHY_REMOTE_FAULT_DISABLE 0x0004
-#define PHY_TRANSMIT_DISABLE 0x0002
-#define PHY_LED_DISABLE 0x0001
-
#define KS884X_PHY_STATUS_OFFSET 0x02
-/* Mode Status Register */
-#define PHY_REG_STATUS 1
-
-#define PHY_100BT4_CAPABLE 0x8000
-#define PHY_100BTX_FD_CAPABLE 0x4000
-#define PHY_100BTX_CAPABLE 0x2000
-#define PHY_10BT_FD_CAPABLE 0x1000
-#define PHY_10BT_CAPABLE 0x0800
-#define PHY_MII_SUPPRESS_CAPABLE 0x0040
-#define PHY_AUTO_NEG_ACKNOWLEDGE 0x0020
-#define PHY_REMOTE_FAULT 0x0010
-#define PHY_AUTO_NEG_CAPABLE 0x0008
-#define PHY_LINK_STATUS 0x0004
-#define PHY_JABBER_DETECT 0x0002
-#define PHY_EXTENDED_CAPABILITY 0x0001
-
#define KS884X_PHY_ID_1_OFFSET 0x04
#define KS884X_PHY_ID_2_OFFSET 0x06
-/* PHY Identifier Registers */
-#define PHY_REG_ID_1 2
-#define PHY_REG_ID_2 3
-
#define KS884X_PHY_AUTO_NEG_OFFSET 0x08
-/* Auto-Negotiation Advertisement Register */
-#define PHY_REG_AUTO_NEGOTIATION 4
-
-#define PHY_AUTO_NEG_NEXT_PAGE 0x8000
-#define PHY_AUTO_NEG_REMOTE_FAULT 0x2000
-/* Not supported. */
-#define PHY_AUTO_NEG_ASYM_PAUSE 0x0800
-#define PHY_AUTO_NEG_SYM_PAUSE 0x0400
-#define PHY_AUTO_NEG_100BT4 0x0200
-#define PHY_AUTO_NEG_100BTX_FD 0x0100
-#define PHY_AUTO_NEG_100BTX 0x0080
-#define PHY_AUTO_NEG_10BT_FD 0x0040
-#define PHY_AUTO_NEG_10BT 0x0020
-#define PHY_AUTO_NEG_SELECTOR 0x001F
-#define PHY_AUTO_NEG_802_3 0x0001
-
-#define PHY_AUTO_NEG_PAUSE (PHY_AUTO_NEG_SYM_PAUSE | PHY_AUTO_NEG_ASYM_PAUSE)
-
#define KS884X_PHY_REMOTE_CAP_OFFSET 0x0A
-/* Auto-Negotiation Link Partner Ability Register */
-#define PHY_REG_REMOTE_CAPABILITY 5
-
-#define PHY_REMOTE_NEXT_PAGE 0x8000
-#define PHY_REMOTE_ACKNOWLEDGE 0x4000
-#define PHY_REMOTE_REMOTE_FAULT 0x2000
-#define PHY_REMOTE_SYM_PAUSE 0x0400
-#define PHY_REMOTE_100BTX_FD 0x0100
-#define PHY_REMOTE_100BTX 0x0080
-#define PHY_REMOTE_10BT_FD 0x0040
-#define PHY_REMOTE_10BT 0x0020
-
/* P1VCT */
#define KS884X_P1VCT_P 0x04F0
#define KS884X_P1PHYCTRL_P 0x04F2
}
/**
- * sw_get_board_storm - get broadcast storm threshold
+ * sw_get_broad_storm - get broadcast storm threshold
* @hw: The hardware instance.
* @percent: Buffer to store the broadcast storm threshold percentage.
*
}
}
-#define PHY_LINK_SUPPORT \
- (PHY_AUTO_NEG_ASYM_PAUSE | \
- PHY_AUTO_NEG_SYM_PAUSE | \
- PHY_AUTO_NEG_100BT4 | \
- PHY_AUTO_NEG_100BTX_FD | \
- PHY_AUTO_NEG_100BTX | \
- PHY_AUTO_NEG_10BT_FD | \
- PHY_AUTO_NEG_10BT)
-
static inline void hw_r_phy_ctrl(struct ksz_hw *hw, int phy, u16 *data)
{
*data = readw(hw->io + phy + KS884X_PHY_CTRL_OFFSET);
}
/**
- * port_w_phy - write data to PHY register
+ * hw_w_phy - write data to PHY register
* @hw: The hardware instance.
* @port: Port to write.
* @reg: PHY register to write.
rx = tx = 0;
if (port->force_link)
rx = tx = 1;
- if (remote & PHY_AUTO_NEG_SYM_PAUSE) {
- if (local & PHY_AUTO_NEG_SYM_PAUSE) {
+ if (remote & LPA_PAUSE_CAP) {
+ if (local & ADVERTISE_PAUSE_CAP) {
rx = tx = 1;
- } else if ((remote & PHY_AUTO_NEG_ASYM_PAUSE) &&
- (local & PHY_AUTO_NEG_PAUSE) ==
- PHY_AUTO_NEG_ASYM_PAUSE) {
+ } else if ((remote & LPA_PAUSE_ASYM) &&
+ (local &
+ (ADVERTISE_PAUSE_CAP | ADVERTISE_PAUSE_ASYM)) ==
+ ADVERTISE_PAUSE_ASYM) {
tx = 1;
}
- } else if (remote & PHY_AUTO_NEG_ASYM_PAUSE) {
- if ((local & PHY_AUTO_NEG_PAUSE) == PHY_AUTO_NEG_PAUSE)
+ } else if (remote & LPA_PAUSE_ASYM) {
+ if ((local & (ADVERTISE_PAUSE_CAP | ADVERTISE_PAUSE_ASYM))
+ == (ADVERTISE_PAUSE_CAP | ADVERTISE_PAUSE_ASYM))
rx = 1;
}
if (!hw->ksz_switch)
phy = KS884X_PHY_1_CTRL_OFFSET + p * PHY_CTRL_INTERVAL;
hw_r_phy_ctrl(hw, phy, &data);
- data &= ~PHY_AUTO_NEG_ENABLE;
+ data &= ~BMCR_ANENABLE;
if (10 == port->speed)
- data &= ~PHY_SPEED_100MBIT;
+ data &= ~BMCR_SPEED100;
else if (100 == port->speed)
- data |= PHY_SPEED_100MBIT;
+ data |= BMCR_SPEED100;
if (1 == port->duplex)
- data &= ~PHY_FULL_DUPLEX;
+ data &= ~BMCR_FULLDPLX;
else if (2 == port->duplex)
- data |= PHY_FULL_DUPLEX;
+ data |= BMCR_FULLDPLX;
hw_w_phy_ctrl(hw, phy, data);
}
}
}
/**
- * transmit_done - transmit done processing
+ * tx_done - transmit done processing
* @hw_priv: Network device.
*
* This routine is called when the transmit interrupt is triggered, indicating
To compile this driver as a module, choose M here. The module will be
called lan743x.
+source "drivers/net/ethernet/microchip/sparx5/Kconfig"
+
endif # NET_VENDOR_MICROCHIP
obj-$(CONFIG_LAN743X) += lan743x.o
lan743x-objs := lan743x_main.o lan743x_ethtool.o lan743x_ptp.o
+
+obj-$(CONFIG_SPARX5_SWITCH) += sparx5/
// SPDX-License-Identifier: GPL-2.0-or-later
-/**
+/*
* Microchip ENCX24J600 ethernet driver
*
* Copyright (C) 2015 Gridpoint
/* SPDX-License-Identifier: GPL-2.0 */
-/**
+/*
* encx24j600_hw.h: Register definitions
*
*/
--- /dev/null
+config SPARX5_SWITCH
+ tristate "Sparx5 switch driver"
+ depends on NET_SWITCHDEV
+ depends on HAS_IOMEM
+ select PHYLINK
+ select PHY_SPARX5_SERDES
+ select RESET_CONTROLLER
+ help
+ This driver supports the Sparx5 network switch device.
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Makefile for the Microchip Sparx5 network device drivers.
+#
+
+obj-$(CONFIG_SPARX5_SWITCH) += sparx5-switch.o
+
+sparx5-switch-objs := sparx5_main.o sparx5_packet.o \
+ sparx5_netdev.o sparx5_phylink.o sparx5_port.o sparx5_mactable.o sparx5_vlan.o \
+ sparx5_switchdev.o sparx5_calendar.o sparx5_ethtool.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/* Microchip Sparx5 Switch driver
+ *
+ * Copyright (c) 2021 Microchip Technology Inc. and its subsidiaries.
+ */
+
+#include <linux/module.h>
+#include <linux/device.h>
+
+#include "sparx5_main_regs.h"
+#include "sparx5_main.h"
+
+/* QSYS calendar information */
+#define SPX5_PORTS_PER_CALREG 10 /* Ports mapped in a calendar register */
+#define SPX5_CALBITS_PER_PORT 3 /* Bit per port in calendar register */
+
+/* DSM calendar information */
+#define SPX5_DSM_CAL_LEN 64
+#define SPX5_DSM_CAL_EMPTY 0xFFFF
+#define SPX5_DSM_CAL_MAX_DEVS_PER_TAXI 13
+#define SPX5_DSM_CAL_TAXIS 8
+#define SPX5_DSM_CAL_BW_LOSS 553
+
+#define SPX5_TAXI_PORT_MAX 70
+
+#define SPEED_12500 12500
+
+/* Maps from taxis to port numbers */
+static u32 sparx5_taxi_ports[SPX5_DSM_CAL_TAXIS][SPX5_DSM_CAL_MAX_DEVS_PER_TAXI] = {
+ {57, 12, 0, 1, 2, 16, 17, 18, 19, 20, 21, 22, 23},
+ {58, 13, 3, 4, 5, 24, 25, 26, 27, 28, 29, 30, 31},
+ {59, 14, 6, 7, 8, 32, 33, 34, 35, 36, 37, 38, 39},
+ {60, 15, 9, 10, 11, 40, 41, 42, 43, 44, 45, 46, 47},
+ {61, 48, 49, 50, 99, 99, 99, 99, 99, 99, 99, 99, 99},
+ {62, 51, 52, 53, 99, 99, 99, 99, 99, 99, 99, 99, 99},
+ {56, 63, 54, 55, 99, 99, 99, 99, 99, 99, 99, 99, 99},
+ {64, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99},
+};
+
+struct sparx5_calendar_data {
+ u32 schedule[SPX5_DSM_CAL_LEN];
+ u32 avg_dist[SPX5_DSM_CAL_MAX_DEVS_PER_TAXI];
+ u32 taxi_ports[SPX5_DSM_CAL_MAX_DEVS_PER_TAXI];
+ u32 taxi_speeds[SPX5_DSM_CAL_MAX_DEVS_PER_TAXI];
+ u32 dev_slots[SPX5_DSM_CAL_MAX_DEVS_PER_TAXI];
+ u32 new_slots[SPX5_DSM_CAL_LEN];
+ u32 temp_sched[SPX5_DSM_CAL_LEN];
+ u32 indices[SPX5_DSM_CAL_LEN];
+ u32 short_list[SPX5_DSM_CAL_LEN];
+ u32 long_list[SPX5_DSM_CAL_LEN];
+};
+
+static u32 sparx5_target_bandwidth(struct sparx5 *sparx5)
+{
+ switch (sparx5->target_ct) {
+ case SPX5_TARGET_CT_7546:
+ case SPX5_TARGET_CT_7546TSN:
+ return 65000;
+ case SPX5_TARGET_CT_7549:
+ case SPX5_TARGET_CT_7549TSN:
+ return 91000;
+ case SPX5_TARGET_CT_7552:
+ case SPX5_TARGET_CT_7552TSN:
+ return 129000;
+ case SPX5_TARGET_CT_7556:
+ case SPX5_TARGET_CT_7556TSN:
+ return 161000;
+ case SPX5_TARGET_CT_7558:
+ case SPX5_TARGET_CT_7558TSN:
+ return 201000;
+ default:
+ return 0;
+ }
+}
+
+/* This is used in calendar configuration */
+enum sparx5_cal_bw {
+ SPX5_CAL_SPEED_NONE = 0,
+ SPX5_CAL_SPEED_1G = 1,
+ SPX5_CAL_SPEED_2G5 = 2,
+ SPX5_CAL_SPEED_5G = 3,
+ SPX5_CAL_SPEED_10G = 4,
+ SPX5_CAL_SPEED_25G = 5,
+ SPX5_CAL_SPEED_0G5 = 6,
+ SPX5_CAL_SPEED_12G5 = 7
+};
+
+static u32 sparx5_clk_to_bandwidth(enum sparx5_core_clockfreq cclock)
+{
+ switch (cclock) {
+ case SPX5_CORE_CLOCK_250MHZ: return 83000; /* 250000 / 3 */
+ case SPX5_CORE_CLOCK_500MHZ: return 166000; /* 500000 / 3 */
+ case SPX5_CORE_CLOCK_625MHZ: return 208000; /* 625000 / 3 */
+ default: return 0;
+ }
+ return 0;
+}
+
+static u32 sparx5_cal_speed_to_value(enum sparx5_cal_bw speed)
+{
+ switch (speed) {
+ case SPX5_CAL_SPEED_1G: return 1000;
+ case SPX5_CAL_SPEED_2G5: return 2500;
+ case SPX5_CAL_SPEED_5G: return 5000;
+ case SPX5_CAL_SPEED_10G: return 10000;
+ case SPX5_CAL_SPEED_25G: return 25000;
+ case SPX5_CAL_SPEED_0G5: return 500;
+ case SPX5_CAL_SPEED_12G5: return 12500;
+ default: return 0;
+ }
+}
+
+static u32 sparx5_bandwidth_to_calendar(u32 bw)
+{
+ switch (bw) {
+ case SPEED_10: return SPX5_CAL_SPEED_0G5;
+ case SPEED_100: return SPX5_CAL_SPEED_0G5;
+ case SPEED_1000: return SPX5_CAL_SPEED_1G;
+ case SPEED_2500: return SPX5_CAL_SPEED_2G5;
+ case SPEED_5000: return SPX5_CAL_SPEED_5G;
+ case SPEED_10000: return SPX5_CAL_SPEED_10G;
+ case SPEED_12500: return SPX5_CAL_SPEED_12G5;
+ case SPEED_25000: return SPX5_CAL_SPEED_25G;
+ case SPEED_UNKNOWN: return SPX5_CAL_SPEED_1G;
+ default: return SPX5_CAL_SPEED_NONE;
+ }
+}
+
+static enum sparx5_cal_bw sparx5_get_port_cal_speed(struct sparx5 *sparx5,
+ u32 portno)
+{
+ struct sparx5_port *port;
+
+ if (portno >= SPX5_PORTS) {
+ /* Internal ports */
+ if (portno == SPX5_PORT_CPU_0 || portno == SPX5_PORT_CPU_1) {
+ /* Equals 1.25G */
+ return SPX5_CAL_SPEED_2G5;
+ } else if (portno == SPX5_PORT_VD0) {
+ /* IPMC only idle BW */
+ return SPX5_CAL_SPEED_NONE;
+ } else if (portno == SPX5_PORT_VD1) {
+ /* OAM only idle BW */
+ return SPX5_CAL_SPEED_NONE;
+ } else if (portno == SPX5_PORT_VD2) {
+ /* IPinIP gets only idle BW */
+ return SPX5_CAL_SPEED_NONE;
+ }
+ /* not in port map */
+ return SPX5_CAL_SPEED_NONE;
+ }
+ /* Front ports - may be used */
+ port = sparx5->ports[portno];
+ if (!port)
+ return SPX5_CAL_SPEED_NONE;
+ return sparx5_bandwidth_to_calendar(port->conf.bandwidth);
+}
+
+/* Auto configure the QSYS calendar based on port configuration */
+int sparx5_config_auto_calendar(struct sparx5 *sparx5)
+{
+ u32 cal[7], value, idx, portno;
+ u32 max_core_bw;
+ u32 total_bw = 0, used_port_bw = 0;
+ int err = 0;
+ enum sparx5_cal_bw spd;
+
+ memset(cal, 0, sizeof(cal));
+
+ max_core_bw = sparx5_clk_to_bandwidth(sparx5->coreclock);
+ if (max_core_bw == 0) {
+ dev_err(sparx5->dev, "Core clock not supported");
+ return -EINVAL;
+ }
+
+ /* Setup the calendar with the bandwidth to each port */
+ for (portno = 0; portno < SPX5_PORTS_ALL; portno++) {
+ u64 reg, offset, this_bw;
+
+ spd = sparx5_get_port_cal_speed(sparx5, portno);
+ if (spd == SPX5_CAL_SPEED_NONE)
+ continue;
+
+ this_bw = sparx5_cal_speed_to_value(spd);
+ if (portno < SPX5_PORTS)
+ used_port_bw += this_bw;
+ else
+ /* Internal ports are granted half the value */
+ this_bw = this_bw / 2;
+ total_bw += this_bw;
+ reg = portno;
+ offset = do_div(reg, SPX5_PORTS_PER_CALREG);
+ cal[reg] |= spd << (offset * SPX5_CALBITS_PER_PORT);
+ }
+
+ if (used_port_bw > sparx5_target_bandwidth(sparx5)) {
+ dev_err(sparx5->dev,
+ "Port BW %u above target BW %u\n",
+ used_port_bw, sparx5_target_bandwidth(sparx5));
+ return -EINVAL;
+ }
+
+ if (total_bw > max_core_bw) {
+ dev_err(sparx5->dev,
+ "Total BW %u above switch core BW %u\n",
+ total_bw, max_core_bw);
+ return -EINVAL;
+ }
+
+ /* Halt the calendar while changing it */
+ spx5_rmw(QSYS_CAL_CTRL_CAL_MODE_SET(10),
+ QSYS_CAL_CTRL_CAL_MODE,
+ sparx5, QSYS_CAL_CTRL);
+
+ /* Assign port bandwidth to auto calendar */
+ for (idx = 0; idx < ARRAY_SIZE(cal); idx++)
+ spx5_wr(cal[idx], sparx5, QSYS_CAL_AUTO(idx));
+
+ /* Increase grant rate of all ports to account for
+ * core clock ppm deviations
+ */
+ spx5_rmw(QSYS_CAL_CTRL_CAL_AUTO_GRANT_RATE_SET(671), /* 672->671 */
+ QSYS_CAL_CTRL_CAL_AUTO_GRANT_RATE,
+ sparx5,
+ QSYS_CAL_CTRL);
+
+ /* Grant idle usage to VD 0-2 */
+ for (idx = 2; idx < 5; idx++)
+ spx5_wr(HSCH_OUTB_SHARE_ENA_OUTB_SHARE_ENA_SET(12),
+ sparx5,
+ HSCH_OUTB_SHARE_ENA(idx));
+
+ /* Enable Auto mode */
+ spx5_rmw(QSYS_CAL_CTRL_CAL_MODE_SET(8),
+ QSYS_CAL_CTRL_CAL_MODE,
+ sparx5, QSYS_CAL_CTRL);
+
+ /* Verify successful calendar config */
+ value = spx5_rd(sparx5, QSYS_CAL_CTRL);
+ if (QSYS_CAL_CTRL_CAL_AUTO_ERROR_GET(value)) {
+ dev_err(sparx5->dev, "QSYS calendar error\n");
+ err = -EINVAL;
+ }
+ return err;
+}
+
+static u32 sparx5_dsm_exb_gcd(u32 a, u32 b)
+{
+ if (b == 0)
+ return a;
+ return sparx5_dsm_exb_gcd(b, a % b);
+}
+
+static u32 sparx5_dsm_cal_len(u32 *cal)
+{
+ u32 idx = 0, len = 0;
+
+ while (idx < SPX5_DSM_CAL_LEN) {
+ if (cal[idx] != SPX5_DSM_CAL_EMPTY)
+ len++;
+ idx++;
+ }
+ return len;
+}
+
+static u32 sparx5_dsm_cp_cal(u32 *sched)
+{
+ u32 idx = 0, tmp;
+
+ while (idx < SPX5_DSM_CAL_LEN) {
+ if (sched[idx] != SPX5_DSM_CAL_EMPTY) {
+ tmp = sched[idx];
+ sched[idx] = SPX5_DSM_CAL_EMPTY;
+ return tmp;
+ }
+ idx++;
+ }
+ return SPX5_DSM_CAL_EMPTY;
+}
+
+static int sparx5_dsm_calendar_calc(struct sparx5 *sparx5, u32 taxi,
+ struct sparx5_calendar_data *data)
+{
+ bool slow_mode;
+ u32 gcd, idx, sum, min, factor;
+ u32 num_of_slots, slot_spd, empty_slots;
+ u32 taxi_bw, clk_period_ps;
+
+ clk_period_ps = sparx5_clk_period(sparx5->coreclock);
+ taxi_bw = 128 * 1000000 / clk_period_ps;
+ slow_mode = !!(clk_period_ps > 2000);
+ memcpy(data->taxi_ports, &sparx5_taxi_ports[taxi],
+ sizeof(data->taxi_ports));
+
+ for (idx = 0; idx < SPX5_DSM_CAL_LEN; idx++) {
+ data->new_slots[idx] = SPX5_DSM_CAL_EMPTY;
+ data->schedule[idx] = SPX5_DSM_CAL_EMPTY;
+ data->temp_sched[idx] = SPX5_DSM_CAL_EMPTY;
+ }
+ /* Default empty calendar */
+ data->schedule[0] = SPX5_DSM_CAL_MAX_DEVS_PER_TAXI;
+
+ /* Map ports to taxi positions */
+ for (idx = 0; idx < SPX5_DSM_CAL_MAX_DEVS_PER_TAXI; idx++) {
+ u32 portno = data->taxi_ports[idx];
+
+ if (portno < SPX5_TAXI_PORT_MAX) {
+ data->taxi_speeds[idx] = sparx5_cal_speed_to_value
+ (sparx5_get_port_cal_speed(sparx5, portno));
+ } else {
+ data->taxi_speeds[idx] = 0;
+ }
+ }
+
+ sum = 0;
+ min = 25000;
+ for (idx = 0; idx < ARRAY_SIZE(data->taxi_speeds); idx++) {
+ u32 jdx;
+
+ sum += data->taxi_speeds[idx];
+ if (data->taxi_speeds[idx] && data->taxi_speeds[idx] < min)
+ min = data->taxi_speeds[idx];
+ gcd = min;
+ for (jdx = 0; jdx < ARRAY_SIZE(data->taxi_speeds); jdx++)
+ gcd = sparx5_dsm_exb_gcd(gcd, data->taxi_speeds[jdx]);
+ }
+ if (sum == 0) /* Empty calendar */
+ return 0;
+ /* Make room for overhead traffic */
+ factor = 100 * 100 * 1000 / (100 * 100 - SPX5_DSM_CAL_BW_LOSS);
+
+ if (sum * factor > (taxi_bw * 1000)) {
+ dev_err(sparx5->dev,
+ "Taxi %u, Requested BW %u above available BW %u\n",
+ taxi, sum, taxi_bw);
+ return -EINVAL;
+ }
+ for (idx = 0; idx < 4; idx++) {
+ u32 raw_spd;
+
+ if (idx == 0)
+ raw_spd = gcd / 5;
+ else if (idx == 1)
+ raw_spd = gcd / 2;
+ else if (idx == 2)
+ raw_spd = gcd;
+ else
+ raw_spd = min;
+ slot_spd = raw_spd * factor / 1000;
+ num_of_slots = taxi_bw / slot_spd;
+ if (num_of_slots <= 64)
+ break;
+ }
+
+ num_of_slots = num_of_slots > 64 ? 64 : num_of_slots;
+ slot_spd = taxi_bw / num_of_slots;
+
+ sum = 0;
+ for (idx = 0; idx < ARRAY_SIZE(data->taxi_speeds); idx++) {
+ u32 spd = data->taxi_speeds[idx];
+ u32 adjusted_speed = data->taxi_speeds[idx] * factor / 1000;
+
+ if (adjusted_speed > 0) {
+ data->avg_dist[idx] = (128 * 1000000 * 10) /
+ (adjusted_speed * clk_period_ps);
+ } else {
+ data->avg_dist[idx] = -1;
+ }
+ data->dev_slots[idx] = ((spd * factor / slot_spd) + 999) / 1000;
+ if (spd != 25000 && (spd != 10000 || !slow_mode)) {
+ if (num_of_slots < (5 * data->dev_slots[idx])) {
+ dev_err(sparx5->dev,
+ "Taxi %u, speed %u, Low slot sep.\n",
+ taxi, spd);
+ return -EINVAL;
+ }
+ }
+ sum += data->dev_slots[idx];
+ if (sum > num_of_slots) {
+ dev_err(sparx5->dev,
+ "Taxi %u with overhead factor %u\n",
+ taxi, factor);
+ return -EINVAL;
+ }
+ }
+
+ empty_slots = num_of_slots - sum;
+
+ for (idx = 0; idx < empty_slots; idx++)
+ data->schedule[idx] = SPX5_DSM_CAL_MAX_DEVS_PER_TAXI;
+
+ for (idx = 1; idx < num_of_slots; idx++) {
+ u32 indices_len = 0;
+ u32 slot, jdx, kdx, ts;
+ s32 cnt;
+ u32 num_of_old_slots, num_of_new_slots, tgt_score;
+
+ for (slot = 0; slot < ARRAY_SIZE(data->dev_slots); slot++) {
+ if (data->dev_slots[slot] == idx) {
+ data->indices[indices_len] = slot;
+ indices_len++;
+ }
+ }
+ if (indices_len == 0)
+ continue;
+ kdx = 0;
+ for (slot = 0; slot < idx; slot++) {
+ for (jdx = 0; jdx < indices_len; jdx++, kdx++)
+ data->new_slots[kdx] = data->indices[jdx];
+ }
+
+ for (slot = 0; slot < SPX5_DSM_CAL_LEN; slot++) {
+ if (data->schedule[slot] == SPX5_DSM_CAL_EMPTY)
+ break;
+ }
+
+ num_of_old_slots = slot;
+ num_of_new_slots = kdx;
+ cnt = 0;
+ ts = 0;
+
+ if (num_of_new_slots > num_of_old_slots) {
+ memcpy(data->short_list, data->schedule,
+ sizeof(data->short_list));
+ memcpy(data->long_list, data->new_slots,
+ sizeof(data->long_list));
+ tgt_score = 100000 * num_of_old_slots /
+ num_of_new_slots;
+ } else {
+ memcpy(data->short_list, data->new_slots,
+ sizeof(data->short_list));
+ memcpy(data->long_list, data->schedule,
+ sizeof(data->long_list));
+ tgt_score = 100000 * num_of_new_slots /
+ num_of_old_slots;
+ }
+
+ while (sparx5_dsm_cal_len(data->short_list) > 0 ||
+ sparx5_dsm_cal_len(data->long_list) > 0) {
+ u32 act = 0;
+
+ if (sparx5_dsm_cal_len(data->short_list) > 0) {
+ data->temp_sched[ts] =
+ sparx5_dsm_cp_cal(data->short_list);
+ ts++;
+ cnt += 100000;
+ act = 1;
+ }
+ while (sparx5_dsm_cal_len(data->long_list) > 0 &&
+ cnt > 0) {
+ data->temp_sched[ts] =
+ sparx5_dsm_cp_cal(data->long_list);
+ ts++;
+ cnt -= tgt_score;
+ act = 1;
+ }
+ if (act == 0) {
+ dev_err(sparx5->dev,
+ "Error in DSM calendar calculation\n");
+ return -EINVAL;
+ }
+ }
+
+ for (slot = 0; slot < SPX5_DSM_CAL_LEN; slot++) {
+ if (data->temp_sched[slot] == SPX5_DSM_CAL_EMPTY)
+ break;
+ }
+ for (slot = 0; slot < SPX5_DSM_CAL_LEN; slot++) {
+ data->schedule[slot] = data->temp_sched[slot];
+ data->temp_sched[slot] = SPX5_DSM_CAL_EMPTY;
+ data->new_slots[slot] = SPX5_DSM_CAL_EMPTY;
+ }
+ }
+ return 0;
+}
+
+static int sparx5_dsm_calendar_check(struct sparx5 *sparx5,
+ struct sparx5_calendar_data *data)
+{
+ u32 num_of_slots, idx, port;
+ int cnt, max_dist;
+ u32 slot_indices[SPX5_DSM_CAL_LEN], distances[SPX5_DSM_CAL_LEN];
+ u32 cal_length = sparx5_dsm_cal_len(data->schedule);
+
+ for (port = 0; port < SPX5_DSM_CAL_MAX_DEVS_PER_TAXI; port++) {
+ num_of_slots = 0;
+ max_dist = data->avg_dist[port];
+ for (idx = 0; idx < SPX5_DSM_CAL_LEN; idx++) {
+ slot_indices[idx] = SPX5_DSM_CAL_EMPTY;
+ distances[idx] = SPX5_DSM_CAL_EMPTY;
+ }
+
+ for (idx = 0; idx < cal_length; idx++) {
+ if (data->schedule[idx] == port) {
+ slot_indices[num_of_slots] = idx;
+ num_of_slots++;
+ }
+ }
+
+ slot_indices[num_of_slots] = slot_indices[0] + cal_length;
+
+ for (idx = 0; idx < num_of_slots; idx++) {
+ distances[idx] = (slot_indices[idx + 1] -
+ slot_indices[idx]) * 10;
+ }
+
+ for (idx = 0; idx < num_of_slots; idx++) {
+ u32 jdx, kdx;
+
+ cnt = distances[idx] - max_dist;
+ if (cnt < 0)
+ cnt = -cnt;
+ kdx = 0;
+ for (jdx = (idx + 1) % num_of_slots;
+ jdx != idx;
+ jdx = (jdx + 1) % num_of_slots, kdx++) {
+ cnt = cnt + distances[jdx] - max_dist;
+ if (cnt < 0)
+ cnt = -cnt;
+ if (cnt > max_dist)
+ goto check_err;
+ }
+ }
+ }
+ return 0;
+check_err:
+ dev_err(sparx5->dev,
+ "Port %u: distance %u above limit %d\n",
+ port, cnt, max_dist);
+ return -EINVAL;
+}
+
+static int sparx5_dsm_calendar_update(struct sparx5 *sparx5, u32 taxi,
+ struct sparx5_calendar_data *data)
+{
+ u32 idx;
+ u32 cal_len = sparx5_dsm_cal_len(data->schedule), len;
+
+ spx5_wr(DSM_TAXI_CAL_CFG_CAL_PGM_ENA_SET(1),
+ sparx5,
+ DSM_TAXI_CAL_CFG(taxi));
+ for (idx = 0; idx < cal_len; idx++) {
+ spx5_rmw(DSM_TAXI_CAL_CFG_CAL_IDX_SET(idx),
+ DSM_TAXI_CAL_CFG_CAL_IDX,
+ sparx5,
+ DSM_TAXI_CAL_CFG(taxi));
+ spx5_rmw(DSM_TAXI_CAL_CFG_CAL_PGM_VAL_SET(data->schedule[idx]),
+ DSM_TAXI_CAL_CFG_CAL_PGM_VAL,
+ sparx5,
+ DSM_TAXI_CAL_CFG(taxi));
+ }
+ spx5_wr(DSM_TAXI_CAL_CFG_CAL_PGM_ENA_SET(0),
+ sparx5,
+ DSM_TAXI_CAL_CFG(taxi));
+ len = DSM_TAXI_CAL_CFG_CAL_CUR_LEN_GET(spx5_rd(sparx5,
+ DSM_TAXI_CAL_CFG(taxi)));
+ if (len != cal_len - 1)
+ goto update_err;
+ return 0;
+update_err:
+ dev_err(sparx5->dev, "Incorrect calendar length: %u\n", len);
+ return -EINVAL;
+}
+
+/* Configure the DSM calendar based on port configuration */
+int sparx5_config_dsm_calendar(struct sparx5 *sparx5)
+{
+ int taxi;
+ struct sparx5_calendar_data *data;
+ int err = 0;
+
+ data = kzalloc(sizeof(*data), GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+
+ for (taxi = 0; taxi < SPX5_DSM_CAL_TAXIS; ++taxi) {
+ err = sparx5_dsm_calendar_calc(sparx5, taxi, data);
+ if (err) {
+ dev_err(sparx5->dev, "DSM calendar calculation failed\n");
+ goto cal_out;
+ }
+ err = sparx5_dsm_calendar_check(sparx5, data);
+ if (err) {
+ dev_err(sparx5->dev, "DSM calendar check failed\n");
+ goto cal_out;
+ }
+ err = sparx5_dsm_calendar_update(sparx5, taxi, data);
+ if (err) {
+ dev_err(sparx5->dev, "DSM calendar update failed\n");
+ goto cal_out;
+ }
+ }
+cal_out:
+ kfree(data);
+ return err;
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/* Microchip Sparx5 Switch driver
+ *
+ * Copyright (c) 2021 Microchip Technology Inc. and its subsidiaries.
+ */
+
+#include <linux/ethtool.h>
+
+#include "sparx5_main_regs.h"
+#include "sparx5_main.h"
+#include "sparx5_port.h"
+
+/* Index of ANA_AC port counters */
+#define SPX5_PORT_POLICER_DROPS 0
+
+/* Add a potentially wrapping 32 bit value to a 64 bit counter */
+static void sparx5_update_counter(u64 *cnt, u32 val)
+{
+ if (val < (*cnt & U32_MAX))
+ *cnt += (u64)1 << 32; /* value has wrapped */
+ *cnt = (*cnt & ~(u64)U32_MAX) + val;
+}
+
+enum sparx5_stats_entry {
+ spx5_stats_rx_symbol_err_cnt = 0,
+ spx5_stats_pmac_rx_symbol_err_cnt = 1,
+ spx5_stats_tx_uc_cnt = 2,
+ spx5_stats_pmac_tx_uc_cnt = 3,
+ spx5_stats_tx_mc_cnt = 4,
+ spx5_stats_tx_bc_cnt = 5,
+ spx5_stats_tx_backoff1_cnt = 6,
+ spx5_stats_tx_multi_coll_cnt = 7,
+ spx5_stats_rx_uc_cnt = 8,
+ spx5_stats_pmac_rx_uc_cnt = 9,
+ spx5_stats_rx_mc_cnt = 10,
+ spx5_stats_rx_bc_cnt = 11,
+ spx5_stats_rx_crc_err_cnt = 12,
+ spx5_stats_pmac_rx_crc_err_cnt = 13,
+ spx5_stats_rx_alignment_lost_cnt = 14,
+ spx5_stats_pmac_rx_alignment_lost_cnt = 15,
+ spx5_stats_tx_ok_bytes_cnt = 16,
+ spx5_stats_pmac_tx_ok_bytes_cnt = 17,
+ spx5_stats_tx_defer_cnt = 18,
+ spx5_stats_tx_late_coll_cnt = 19,
+ spx5_stats_tx_xcoll_cnt = 20,
+ spx5_stats_tx_csense_cnt = 21,
+ spx5_stats_rx_ok_bytes_cnt = 22,
+ spx5_stats_pmac_rx_ok_bytes_cnt = 23,
+ spx5_stats_pmac_tx_mc_cnt = 24,
+ spx5_stats_pmac_tx_bc_cnt = 25,
+ spx5_stats_tx_xdefer_cnt = 26,
+ spx5_stats_pmac_rx_mc_cnt = 27,
+ spx5_stats_pmac_rx_bc_cnt = 28,
+ spx5_stats_rx_in_range_len_err_cnt = 29,
+ spx5_stats_pmac_rx_in_range_len_err_cnt = 30,
+ spx5_stats_rx_out_of_range_len_err_cnt = 31,
+ spx5_stats_pmac_rx_out_of_range_len_err_cnt = 32,
+ spx5_stats_rx_oversize_cnt = 33,
+ spx5_stats_pmac_rx_oversize_cnt = 34,
+ spx5_stats_tx_pause_cnt = 35,
+ spx5_stats_pmac_tx_pause_cnt = 36,
+ spx5_stats_rx_pause_cnt = 37,
+ spx5_stats_pmac_rx_pause_cnt = 38,
+ spx5_stats_rx_unsup_opcode_cnt = 39,
+ spx5_stats_pmac_rx_unsup_opcode_cnt = 40,
+ spx5_stats_rx_undersize_cnt = 41,
+ spx5_stats_pmac_rx_undersize_cnt = 42,
+ spx5_stats_rx_fragments_cnt = 43,
+ spx5_stats_pmac_rx_fragments_cnt = 44,
+ spx5_stats_rx_jabbers_cnt = 45,
+ spx5_stats_pmac_rx_jabbers_cnt = 46,
+ spx5_stats_rx_size64_cnt = 47,
+ spx5_stats_pmac_rx_size64_cnt = 48,
+ spx5_stats_rx_size65to127_cnt = 49,
+ spx5_stats_pmac_rx_size65to127_cnt = 50,
+ spx5_stats_rx_size128to255_cnt = 51,
+ spx5_stats_pmac_rx_size128to255_cnt = 52,
+ spx5_stats_rx_size256to511_cnt = 53,
+ spx5_stats_pmac_rx_size256to511_cnt = 54,
+ spx5_stats_rx_size512to1023_cnt = 55,
+ spx5_stats_pmac_rx_size512to1023_cnt = 56,
+ spx5_stats_rx_size1024to1518_cnt = 57,
+ spx5_stats_pmac_rx_size1024to1518_cnt = 58,
+ spx5_stats_rx_size1519tomax_cnt = 59,
+ spx5_stats_pmac_rx_size1519tomax_cnt = 60,
+ spx5_stats_tx_size64_cnt = 61,
+ spx5_stats_pmac_tx_size64_cnt = 62,
+ spx5_stats_tx_size65to127_cnt = 63,
+ spx5_stats_pmac_tx_size65to127_cnt = 64,
+ spx5_stats_tx_size128to255_cnt = 65,
+ spx5_stats_pmac_tx_size128to255_cnt = 66,
+ spx5_stats_tx_size256to511_cnt = 67,
+ spx5_stats_pmac_tx_size256to511_cnt = 68,
+ spx5_stats_tx_size512to1023_cnt = 69,
+ spx5_stats_pmac_tx_size512to1023_cnt = 70,
+ spx5_stats_tx_size1024to1518_cnt = 71,
+ spx5_stats_pmac_tx_size1024to1518_cnt = 72,
+ spx5_stats_tx_size1519tomax_cnt = 73,
+ spx5_stats_pmac_tx_size1519tomax_cnt = 74,
+ spx5_stats_mm_rx_assembly_err_cnt = 75,
+ spx5_stats_mm_rx_assembly_ok_cnt = 76,
+ spx5_stats_mm_rx_merge_frag_cnt = 77,
+ spx5_stats_mm_rx_smd_err_cnt = 78,
+ spx5_stats_mm_tx_pfragment_cnt = 79,
+ spx5_stats_rx_bad_bytes_cnt = 80,
+ spx5_stats_pmac_rx_bad_bytes_cnt = 81,
+ spx5_stats_rx_in_bytes_cnt = 82,
+ spx5_stats_rx_ipg_shrink_cnt = 83,
+ spx5_stats_rx_sync_lost_err_cnt = 84,
+ spx5_stats_rx_tagged_frms_cnt = 85,
+ spx5_stats_rx_untagged_frms_cnt = 86,
+ spx5_stats_tx_out_bytes_cnt = 87,
+ spx5_stats_tx_tagged_frms_cnt = 88,
+ spx5_stats_tx_untagged_frms_cnt = 89,
+ spx5_stats_rx_hih_cksm_err_cnt = 90,
+ spx5_stats_pmac_rx_hih_cksm_err_cnt = 91,
+ spx5_stats_rx_xgmii_prot_err_cnt = 92,
+ spx5_stats_pmac_rx_xgmii_prot_err_cnt = 93,
+ spx5_stats_ana_ac_port_stat_lsb_cnt = 94,
+ spx5_stats_green_p0_rx_fwd = 95,
+ spx5_stats_green_p0_rx_port_drop = 111,
+ spx5_stats_green_p0_tx_port = 127,
+ spx5_stats_rx_local_drop = 143,
+ spx5_stats_tx_local_drop = 144,
+ spx5_stats_count = 145,
+};
+
+static const char *const sparx5_stats_layout[] = {
+ "mm_rx_assembly_err_cnt",
+ "mm_rx_assembly_ok_cnt",
+ "mm_rx_merge_frag_cnt",
+ "mm_rx_smd_err_cnt",
+ "mm_tx_pfragment_cnt",
+ "rx_bad_bytes_cnt",
+ "pmac_rx_bad_bytes_cnt",
+ "rx_in_bytes_cnt",
+ "rx_ipg_shrink_cnt",
+ "rx_sync_lost_err_cnt",
+ "rx_tagged_frms_cnt",
+ "rx_untagged_frms_cnt",
+ "tx_out_bytes_cnt",
+ "tx_tagged_frms_cnt",
+ "tx_untagged_frms_cnt",
+ "rx_hih_cksm_err_cnt",
+ "pmac_rx_hih_cksm_err_cnt",
+ "rx_xgmii_prot_err_cnt",
+ "pmac_rx_xgmii_prot_err_cnt",
+ "rx_port_policer_drop",
+ "rx_fwd_green_p0",
+ "rx_fwd_green_p1",
+ "rx_fwd_green_p2",
+ "rx_fwd_green_p3",
+ "rx_fwd_green_p4",
+ "rx_fwd_green_p5",
+ "rx_fwd_green_p6",
+ "rx_fwd_green_p7",
+ "rx_fwd_yellow_p0",
+ "rx_fwd_yellow_p1",
+ "rx_fwd_yellow_p2",
+ "rx_fwd_yellow_p3",
+ "rx_fwd_yellow_p4",
+ "rx_fwd_yellow_p5",
+ "rx_fwd_yellow_p6",
+ "rx_fwd_yellow_p7",
+ "rx_port_drop_green_p0",
+ "rx_port_drop_green_p1",
+ "rx_port_drop_green_p2",
+ "rx_port_drop_green_p3",
+ "rx_port_drop_green_p4",
+ "rx_port_drop_green_p5",
+ "rx_port_drop_green_p6",
+ "rx_port_drop_green_p7",
+ "rx_port_drop_yellow_p0",
+ "rx_port_drop_yellow_p1",
+ "rx_port_drop_yellow_p2",
+ "rx_port_drop_yellow_p3",
+ "rx_port_drop_yellow_p4",
+ "rx_port_drop_yellow_p5",
+ "rx_port_drop_yellow_p6",
+ "rx_port_drop_yellow_p7",
+ "tx_port_green_p0",
+ "tx_port_green_p1",
+ "tx_port_green_p2",
+ "tx_port_green_p3",
+ "tx_port_green_p4",
+ "tx_port_green_p5",
+ "tx_port_green_p6",
+ "tx_port_green_p7",
+ "tx_port_yellow_p0",
+ "tx_port_yellow_p1",
+ "tx_port_yellow_p2",
+ "tx_port_yellow_p3",
+ "tx_port_yellow_p4",
+ "tx_port_yellow_p5",
+ "tx_port_yellow_p6",
+ "tx_port_yellow_p7",
+ "rx_local_drop",
+ "tx_local_drop",
+};
+
+static void sparx5_get_queue_sys_stats(struct sparx5 *sparx5, int portno)
+{
+ u64 *portstats;
+ u64 *stats;
+ u32 addr;
+ int idx;
+
+ portstats = &sparx5->stats[portno * sparx5->num_stats];
+ mutex_lock(&sparx5->queue_stats_lock);
+ spx5_wr(XQS_STAT_CFG_STAT_VIEW_SET(portno), sparx5, XQS_STAT_CFG);
+ addr = 0;
+ stats = &portstats[spx5_stats_green_p0_rx_fwd];
+ for (idx = 0; idx < 2 * SPX5_PRIOS; ++idx, ++addr, ++stats)
+ sparx5_update_counter(stats, spx5_rd(sparx5, XQS_CNT(addr)));
+ addr = 16;
+ stats = &portstats[spx5_stats_green_p0_rx_port_drop];
+ for (idx = 0; idx < 2 * SPX5_PRIOS; ++idx, ++addr, ++stats)
+ sparx5_update_counter(stats, spx5_rd(sparx5, XQS_CNT(addr)));
+ addr = 256;
+ stats = &portstats[spx5_stats_green_p0_tx_port];
+ for (idx = 0; idx < 2 * SPX5_PRIOS; ++idx, ++addr, ++stats)
+ sparx5_update_counter(stats, spx5_rd(sparx5, XQS_CNT(addr)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_local_drop],
+ spx5_rd(sparx5, XQS_CNT(32)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_local_drop],
+ spx5_rd(sparx5, XQS_CNT(272)));
+ mutex_unlock(&sparx5->queue_stats_lock);
+}
+
+static void sparx5_get_ana_ac_stats_stats(struct sparx5 *sparx5, int portno)
+{
+ u64 *portstats = &sparx5->stats[portno * sparx5->num_stats];
+
+ sparx5_update_counter(&portstats[spx5_stats_ana_ac_port_stat_lsb_cnt],
+ spx5_rd(sparx5, ANA_AC_PORT_STAT_LSB_CNT(portno,
+ SPX5_PORT_POLICER_DROPS)));
+}
+
+static void sparx5_get_dev_phy_stats(u64 *portstats, void __iomem *inst, u32
+ tinst)
+{
+ sparx5_update_counter(&portstats[spx5_stats_rx_symbol_err_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_SYMBOL_ERR_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_symbol_err_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_SYMBOL_ERR_CNT(tinst)));
+}
+
+static void sparx5_get_dev_mac_stats(u64 *portstats, void __iomem *inst, u32
+ tinst)
+{
+ sparx5_update_counter(&portstats[spx5_stats_tx_uc_cnt],
+ spx5_inst_rd(inst, DEV5G_TX_UC_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_uc_cnt],
+ spx5_inst_rd(inst, DEV5G_PMAC_TX_UC_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_mc_cnt],
+ spx5_inst_rd(inst, DEV5G_TX_MC_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_bc_cnt],
+ spx5_inst_rd(inst, DEV5G_TX_BC_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_uc_cnt],
+ spx5_inst_rd(inst, DEV5G_RX_UC_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_uc_cnt],
+ spx5_inst_rd(inst, DEV5G_PMAC_RX_UC_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_mc_cnt],
+ spx5_inst_rd(inst, DEV5G_RX_MC_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_bc_cnt],
+ spx5_inst_rd(inst, DEV5G_RX_BC_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_crc_err_cnt],
+ spx5_inst_rd(inst, DEV5G_RX_CRC_ERR_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_crc_err_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_CRC_ERR_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_alignment_lost_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_ALIGNMENT_LOST_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_alignment_lost_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_ALIGNMENT_LOST_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_ok_bytes_cnt],
+ spx5_inst_rd(inst, DEV5G_TX_OK_BYTES_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_ok_bytes_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_TX_OK_BYTES_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_ok_bytes_cnt],
+ spx5_inst_rd(inst, DEV5G_RX_OK_BYTES_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_ok_bytes_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_OK_BYTES_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_mc_cnt],
+ spx5_inst_rd(inst, DEV5G_PMAC_TX_MC_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_bc_cnt],
+ spx5_inst_rd(inst, DEV5G_PMAC_TX_BC_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_mc_cnt],
+ spx5_inst_rd(inst, DEV5G_PMAC_RX_MC_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_bc_cnt],
+ spx5_inst_rd(inst, DEV5G_PMAC_RX_BC_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_in_range_len_err_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_IN_RANGE_LEN_ERR_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_in_range_len_err_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_IN_RANGE_LEN_ERR_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_out_of_range_len_err_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_OUT_OF_RANGE_LEN_ERR_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_out_of_range_len_err_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_OUT_OF_RANGE_LEN_ERR_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_oversize_cnt],
+ spx5_inst_rd(inst, DEV5G_RX_OVERSIZE_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_oversize_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_OVERSIZE_CNT(tinst)));
+}
+
+static void sparx5_get_dev_mac_ctrl_stats(u64 *portstats, void __iomem *inst,
+ u32 tinst)
+{
+ sparx5_update_counter(&portstats[spx5_stats_tx_pause_cnt],
+ spx5_inst_rd(inst, DEV5G_TX_PAUSE_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_pause_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_TX_PAUSE_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_pause_cnt],
+ spx5_inst_rd(inst, DEV5G_RX_PAUSE_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_pause_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_PAUSE_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_unsup_opcode_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_UNSUP_OPCODE_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_unsup_opcode_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_UNSUP_OPCODE_CNT(tinst)));
+}
+
+static void sparx5_get_dev_rmon_stats(u64 *portstats, void __iomem *inst, u32
+ tinst)
+{
+ sparx5_update_counter(&portstats[spx5_stats_rx_undersize_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_UNDERSIZE_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_undersize_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_UNDERSIZE_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_oversize_cnt],
+ spx5_inst_rd(inst, DEV5G_RX_OVERSIZE_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_oversize_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_OVERSIZE_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_fragments_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_FRAGMENTS_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_fragments_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_FRAGMENTS_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_jabbers_cnt],
+ spx5_inst_rd(inst, DEV5G_RX_JABBERS_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_jabbers_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_JABBERS_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_size64_cnt],
+ spx5_inst_rd(inst, DEV5G_RX_SIZE64_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_size64_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_SIZE64_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_size65to127_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_SIZE65TO127_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_size65to127_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_SIZE65TO127_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_size128to255_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_SIZE128TO255_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_size128to255_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_SIZE128TO255_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_size256to511_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_SIZE256TO511_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_size256to511_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_SIZE256TO511_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_size512to1023_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_SIZE512TO1023_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_size512to1023_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_SIZE512TO1023_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_size1024to1518_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_SIZE1024TO1518_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_size1024to1518_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_SIZE1024TO1518_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_size1519tomax_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_SIZE1519TOMAX_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_size1519tomax_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_SIZE1519TOMAX_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_size64_cnt],
+ spx5_inst_rd(inst, DEV5G_TX_SIZE64_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_size64_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_TX_SIZE64_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_size65to127_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_TX_SIZE65TO127_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_size65to127_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_TX_SIZE65TO127_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_size128to255_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_TX_SIZE128TO255_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_size128to255_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_TX_SIZE128TO255_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_size256to511_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_TX_SIZE256TO511_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_size256to511_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_TX_SIZE256TO511_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_size512to1023_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_TX_SIZE512TO1023_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_size512to1023_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_TX_SIZE512TO1023_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_size1024to1518_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_TX_SIZE1024TO1518_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_size1024to1518_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_TX_SIZE1024TO1518_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_size1519tomax_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_TX_SIZE1519TOMAX_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_size1519tomax_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_TX_SIZE1519TOMAX_CNT(tinst)));
+}
+
+static void sparx5_get_dev_misc_stats(u64 *portstats, void __iomem *inst, u32
+ tinst)
+{
+ sparx5_update_counter(&portstats[spx5_stats_mm_rx_assembly_err_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_MM_RX_ASSEMBLY_ERR_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_mm_rx_assembly_ok_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_MM_RX_ASSEMBLY_OK_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_mm_rx_merge_frag_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_MM_RX_MERGE_FRAG_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_mm_rx_smd_err_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_MM_RX_SMD_ERR_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_mm_tx_pfragment_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_MM_TX_PFRAGMENT_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_bad_bytes_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_BAD_BYTES_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_bad_bytes_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_BAD_BYTES_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_in_bytes_cnt],
+ spx5_inst_rd(inst, DEV5G_RX_IN_BYTES_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_ipg_shrink_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_IPG_SHRINK_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_tagged_frms_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_TAGGED_FRMS_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_untagged_frms_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_UNTAGGED_FRMS_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_out_bytes_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_TX_OUT_BYTES_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_tagged_frms_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_TX_TAGGED_FRMS_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_untagged_frms_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_TX_UNTAGGED_FRMS_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_hih_cksm_err_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_HIH_CKSM_ERR_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_hih_cksm_err_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_HIH_CKSM_ERR_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_xgmii_prot_err_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_RX_XGMII_PROT_ERR_CNT(tinst)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_xgmii_prot_err_cnt],
+ spx5_inst_rd(inst,
+ DEV5G_PMAC_RX_XGMII_PROT_ERR_CNT(tinst)));
+}
+
+static void sparx5_get_device_stats(struct sparx5 *sparx5, int portno)
+{
+ u64 *portstats = &sparx5->stats[portno * sparx5->num_stats];
+ u32 tinst = sparx5_port_dev_index(portno);
+ u32 dev = sparx5_to_high_dev(portno);
+ void __iomem *inst;
+
+ inst = spx5_inst_get(sparx5, dev, tinst);
+ sparx5_get_dev_phy_stats(portstats, inst, tinst);
+ sparx5_get_dev_mac_stats(portstats, inst, tinst);
+ sparx5_get_dev_mac_ctrl_stats(portstats, inst, tinst);
+ sparx5_get_dev_rmon_stats(portstats, inst, tinst);
+ sparx5_get_dev_misc_stats(portstats, inst, tinst);
+}
+
+static void sparx5_get_asm_phy_stats(u64 *portstats, void __iomem *inst, int
+ portno)
+{
+ sparx5_update_counter(&portstats[spx5_stats_rx_symbol_err_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_SYMBOL_ERR_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_symbol_err_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_SYMBOL_ERR_CNT(portno)));
+}
+
+static void sparx5_get_asm_mac_stats(u64 *portstats, void __iomem *inst, int
+ portno)
+{
+ sparx5_update_counter(&portstats[spx5_stats_tx_uc_cnt],
+ spx5_inst_rd(inst, ASM_TX_UC_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_uc_cnt],
+ spx5_inst_rd(inst, ASM_PMAC_TX_UC_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_mc_cnt],
+ spx5_inst_rd(inst, ASM_TX_MC_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_bc_cnt],
+ spx5_inst_rd(inst, ASM_TX_BC_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_backoff1_cnt],
+ spx5_inst_rd(inst, ASM_TX_BACKOFF1_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_multi_coll_cnt],
+ spx5_inst_rd(inst,
+ ASM_TX_MULTI_COLL_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_uc_cnt],
+ spx5_inst_rd(inst, ASM_RX_UC_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_uc_cnt],
+ spx5_inst_rd(inst, ASM_PMAC_RX_UC_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_mc_cnt],
+ spx5_inst_rd(inst, ASM_RX_MC_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_bc_cnt],
+ spx5_inst_rd(inst, ASM_RX_BC_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_crc_err_cnt],
+ spx5_inst_rd(inst, ASM_RX_CRC_ERR_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_crc_err_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_CRC_ERR_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_alignment_lost_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_ALIGNMENT_LOST_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_alignment_lost_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_ALIGNMENT_LOST_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_ok_bytes_cnt],
+ spx5_inst_rd(inst, ASM_TX_OK_BYTES_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_ok_bytes_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_TX_OK_BYTES_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_defer_cnt],
+ spx5_inst_rd(inst, ASM_TX_DEFER_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_late_coll_cnt],
+ spx5_inst_rd(inst, ASM_TX_LATE_COLL_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_xcoll_cnt],
+ spx5_inst_rd(inst, ASM_TX_XCOLL_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_csense_cnt],
+ spx5_inst_rd(inst, ASM_TX_CSENSE_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_ok_bytes_cnt],
+ spx5_inst_rd(inst, ASM_RX_OK_BYTES_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_ok_bytes_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_OK_BYTES_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_mc_cnt],
+ spx5_inst_rd(inst, ASM_PMAC_TX_MC_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_bc_cnt],
+ spx5_inst_rd(inst, ASM_PMAC_TX_BC_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_xdefer_cnt],
+ spx5_inst_rd(inst, ASM_TX_XDEFER_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_mc_cnt],
+ spx5_inst_rd(inst, ASM_PMAC_RX_MC_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_bc_cnt],
+ spx5_inst_rd(inst, ASM_PMAC_RX_BC_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_in_range_len_err_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_IN_RANGE_LEN_ERR_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_in_range_len_err_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_IN_RANGE_LEN_ERR_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_out_of_range_len_err_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_OUT_OF_RANGE_LEN_ERR_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_out_of_range_len_err_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_OUT_OF_RANGE_LEN_ERR_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_oversize_cnt],
+ spx5_inst_rd(inst, ASM_RX_OVERSIZE_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_oversize_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_OVERSIZE_CNT(portno)));
+}
+
+static void sparx5_get_asm_mac_ctrl_stats(u64 *portstats, void __iomem *inst,
+ int portno)
+{
+ sparx5_update_counter(&portstats[spx5_stats_tx_pause_cnt],
+ spx5_inst_rd(inst, ASM_TX_PAUSE_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_pause_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_TX_PAUSE_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_pause_cnt],
+ spx5_inst_rd(inst, ASM_RX_PAUSE_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_pause_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_PAUSE_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_unsup_opcode_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_UNSUP_OPCODE_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_unsup_opcode_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_UNSUP_OPCODE_CNT(portno)));
+}
+
+static void sparx5_get_asm_rmon_stats(u64 *portstats, void __iomem *inst, int
+ portno)
+{
+ sparx5_update_counter(&portstats[spx5_stats_rx_undersize_cnt],
+ spx5_inst_rd(inst, ASM_RX_UNDERSIZE_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_undersize_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_UNDERSIZE_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_oversize_cnt],
+ spx5_inst_rd(inst, ASM_RX_OVERSIZE_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_oversize_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_OVERSIZE_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_fragments_cnt],
+ spx5_inst_rd(inst, ASM_RX_FRAGMENTS_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_fragments_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_FRAGMENTS_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_jabbers_cnt],
+ spx5_inst_rd(inst, ASM_RX_JABBERS_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_jabbers_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_JABBERS_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_size64_cnt],
+ spx5_inst_rd(inst, ASM_RX_SIZE64_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_size64_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_SIZE64_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_size65to127_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_SIZE65TO127_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_size65to127_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_SIZE65TO127_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_size128to255_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_SIZE128TO255_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_size128to255_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_SIZE128TO255_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_size256to511_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_SIZE256TO511_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_size256to511_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_SIZE256TO511_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_size512to1023_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_SIZE512TO1023_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_size512to1023_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_SIZE512TO1023_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_size1024to1518_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_SIZE1024TO1518_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_size1024to1518_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_SIZE1024TO1518_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_size1519tomax_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_SIZE1519TOMAX_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_size1519tomax_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_SIZE1519TOMAX_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_size64_cnt],
+ spx5_inst_rd(inst, ASM_TX_SIZE64_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_size64_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_TX_SIZE64_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_size65to127_cnt],
+ spx5_inst_rd(inst,
+ ASM_TX_SIZE65TO127_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_size65to127_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_TX_SIZE65TO127_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_size128to255_cnt],
+ spx5_inst_rd(inst,
+ ASM_TX_SIZE128TO255_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_size128to255_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_TX_SIZE128TO255_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_size256to511_cnt],
+ spx5_inst_rd(inst,
+ ASM_TX_SIZE256TO511_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_size256to511_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_TX_SIZE256TO511_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_size512to1023_cnt],
+ spx5_inst_rd(inst,
+ ASM_TX_SIZE512TO1023_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_size512to1023_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_TX_SIZE512TO1023_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_size1024to1518_cnt],
+ spx5_inst_rd(inst,
+ ASM_TX_SIZE1024TO1518_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_size1024to1518_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_TX_SIZE1024TO1518_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_size1519tomax_cnt],
+ spx5_inst_rd(inst,
+ ASM_TX_SIZE1519TOMAX_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_tx_size1519tomax_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_TX_SIZE1519TOMAX_CNT(portno)));
+}
+
+static void sparx5_get_asm_misc_stats(u64 *portstats, void __iomem *inst, int
+ portno)
+{
+ sparx5_update_counter(&portstats[spx5_stats_mm_rx_assembly_err_cnt],
+ spx5_inst_rd(inst,
+ ASM_MM_RX_ASSEMBLY_ERR_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_mm_rx_assembly_ok_cnt],
+ spx5_inst_rd(inst,
+ ASM_MM_RX_ASSEMBLY_OK_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_mm_rx_merge_frag_cnt],
+ spx5_inst_rd(inst,
+ ASM_MM_RX_MERGE_FRAG_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_mm_rx_smd_err_cnt],
+ spx5_inst_rd(inst,
+ ASM_MM_RX_SMD_ERR_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_mm_tx_pfragment_cnt],
+ spx5_inst_rd(inst,
+ ASM_MM_TX_PFRAGMENT_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_bad_bytes_cnt],
+ spx5_inst_rd(inst, ASM_RX_BAD_BYTES_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_pmac_rx_bad_bytes_cnt],
+ spx5_inst_rd(inst,
+ ASM_PMAC_RX_BAD_BYTES_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_in_bytes_cnt],
+ spx5_inst_rd(inst, ASM_RX_IN_BYTES_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_ipg_shrink_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_IPG_SHRINK_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_sync_lost_err_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_SYNC_LOST_ERR_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_tagged_frms_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_TAGGED_FRMS_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_rx_untagged_frms_cnt],
+ spx5_inst_rd(inst,
+ ASM_RX_UNTAGGED_FRMS_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_out_bytes_cnt],
+ spx5_inst_rd(inst, ASM_TX_OUT_BYTES_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_tagged_frms_cnt],
+ spx5_inst_rd(inst,
+ ASM_TX_TAGGED_FRMS_CNT(portno)));
+ sparx5_update_counter(&portstats[spx5_stats_tx_untagged_frms_cnt],
+ spx5_inst_rd(inst,
+ ASM_TX_UNTAGGED_FRMS_CNT(portno)));
+}
+
+static void sparx5_get_asm_stats(struct sparx5 *sparx5, int portno)
+{
+ u64 *portstats = &sparx5->stats[portno * sparx5->num_stats];
+ void __iomem *inst = spx5_inst_get(sparx5, TARGET_ASM, 0);
+
+ sparx5_get_asm_phy_stats(portstats, inst, portno);
+ sparx5_get_asm_mac_stats(portstats, inst, portno);
+ sparx5_get_asm_mac_ctrl_stats(portstats, inst, portno);
+ sparx5_get_asm_rmon_stats(portstats, inst, portno);
+ sparx5_get_asm_misc_stats(portstats, inst, portno);
+}
+
+static const struct ethtool_rmon_hist_range sparx5_rmon_ranges[] = {
+ { 0, 64 },
+ { 65, 127 },
+ { 128, 255 },
+ { 256, 511 },
+ { 512, 1023 },
+ { 1024, 1518 },
+ { 1519, 10239 },
+ {}
+};
+
+static void sparx5_get_eth_phy_stats(struct net_device *ndev,
+ struct ethtool_eth_phy_stats *phy_stats)
+{
+ struct sparx5_port *port = netdev_priv(ndev);
+ struct sparx5 *sparx5 = port->sparx5;
+ int portno = port->portno;
+ void __iomem *inst;
+ u64 *portstats;
+
+ portstats = &sparx5->stats[portno * sparx5->num_stats];
+ if (sparx5_is_baser(port->conf.portmode)) {
+ u32 tinst = sparx5_port_dev_index(portno);
+ u32 dev = sparx5_to_high_dev(portno);
+
+ inst = spx5_inst_get(sparx5, dev, tinst);
+ sparx5_get_dev_phy_stats(portstats, inst, tinst);
+ } else {
+ inst = spx5_inst_get(sparx5, TARGET_ASM, 0);
+ sparx5_get_asm_phy_stats(portstats, inst, portno);
+ }
+ phy_stats->SymbolErrorDuringCarrier =
+ portstats[spx5_stats_rx_symbol_err_cnt] +
+ portstats[spx5_stats_pmac_rx_symbol_err_cnt];
+}
+
+static void sparx5_get_eth_mac_stats(struct net_device *ndev,
+ struct ethtool_eth_mac_stats *mac_stats)
+{
+ struct sparx5_port *port = netdev_priv(ndev);
+ struct sparx5 *sparx5 = port->sparx5;
+ int portno = port->portno;
+ void __iomem *inst;
+ u64 *portstats;
+
+ portstats = &sparx5->stats[portno * sparx5->num_stats];
+ if (sparx5_is_baser(port->conf.portmode)) {
+ u32 tinst = sparx5_port_dev_index(portno);
+ u32 dev = sparx5_to_high_dev(portno);
+
+ inst = spx5_inst_get(sparx5, dev, tinst);
+ sparx5_get_dev_mac_stats(portstats, inst, tinst);
+ } else {
+ inst = spx5_inst_get(sparx5, TARGET_ASM, 0);
+ sparx5_get_asm_mac_stats(portstats, inst, portno);
+ }
+ mac_stats->FramesTransmittedOK = portstats[spx5_stats_tx_uc_cnt] +
+ portstats[spx5_stats_pmac_tx_uc_cnt] +
+ portstats[spx5_stats_tx_mc_cnt] +
+ portstats[spx5_stats_tx_bc_cnt];
+ mac_stats->SingleCollisionFrames =
+ portstats[spx5_stats_tx_backoff1_cnt];
+ mac_stats->MultipleCollisionFrames =
+ portstats[spx5_stats_tx_multi_coll_cnt];
+ mac_stats->FramesReceivedOK = portstats[spx5_stats_rx_uc_cnt] +
+ portstats[spx5_stats_pmac_rx_uc_cnt] +
+ portstats[spx5_stats_rx_mc_cnt] +
+ portstats[spx5_stats_rx_bc_cnt];
+ mac_stats->FrameCheckSequenceErrors =
+ portstats[spx5_stats_rx_crc_err_cnt] +
+ portstats[spx5_stats_pmac_rx_crc_err_cnt];
+ mac_stats->AlignmentErrors = portstats[spx5_stats_rx_alignment_lost_cnt]
+ + portstats[spx5_stats_pmac_rx_alignment_lost_cnt];
+ mac_stats->OctetsTransmittedOK = portstats[spx5_stats_tx_ok_bytes_cnt] +
+ portstats[spx5_stats_pmac_tx_ok_bytes_cnt];
+ mac_stats->FramesWithDeferredXmissions =
+ portstats[spx5_stats_tx_defer_cnt];
+ mac_stats->LateCollisions =
+ portstats[spx5_stats_tx_late_coll_cnt];
+ mac_stats->FramesAbortedDueToXSColls =
+ portstats[spx5_stats_tx_xcoll_cnt];
+ mac_stats->CarrierSenseErrors = portstats[spx5_stats_tx_csense_cnt];
+ mac_stats->OctetsReceivedOK = portstats[spx5_stats_rx_ok_bytes_cnt] +
+ portstats[spx5_stats_pmac_rx_ok_bytes_cnt];
+ mac_stats->MulticastFramesXmittedOK = portstats[spx5_stats_tx_mc_cnt] +
+ portstats[spx5_stats_pmac_tx_mc_cnt];
+ mac_stats->BroadcastFramesXmittedOK = portstats[spx5_stats_tx_bc_cnt] +
+ portstats[spx5_stats_pmac_tx_bc_cnt];
+ mac_stats->FramesWithExcessiveDeferral =
+ portstats[spx5_stats_tx_xdefer_cnt];
+ mac_stats->MulticastFramesReceivedOK = portstats[spx5_stats_rx_mc_cnt] +
+ portstats[spx5_stats_pmac_rx_mc_cnt];
+ mac_stats->BroadcastFramesReceivedOK = portstats[spx5_stats_rx_bc_cnt] +
+ portstats[spx5_stats_pmac_rx_bc_cnt];
+ mac_stats->InRangeLengthErrors =
+ portstats[spx5_stats_rx_in_range_len_err_cnt] +
+ portstats[spx5_stats_pmac_rx_in_range_len_err_cnt];
+ mac_stats->OutOfRangeLengthField =
+ portstats[spx5_stats_rx_out_of_range_len_err_cnt] +
+ portstats[spx5_stats_pmac_rx_out_of_range_len_err_cnt];
+ mac_stats->FrameTooLongErrors = portstats[spx5_stats_rx_oversize_cnt] +
+ portstats[spx5_stats_pmac_rx_oversize_cnt];
+}
+
+static void sparx5_get_eth_mac_ctrl_stats(struct net_device *ndev,
+ struct ethtool_eth_ctrl_stats *mac_ctrl_stats)
+{
+ struct sparx5_port *port = netdev_priv(ndev);
+ struct sparx5 *sparx5 = port->sparx5;
+ int portno = port->portno;
+ void __iomem *inst;
+ u64 *portstats;
+
+ portstats = &sparx5->stats[portno * sparx5->num_stats];
+ if (sparx5_is_baser(port->conf.portmode)) {
+ u32 tinst = sparx5_port_dev_index(portno);
+ u32 dev = sparx5_to_high_dev(portno);
+
+ inst = spx5_inst_get(sparx5, dev, tinst);
+ sparx5_get_dev_mac_ctrl_stats(portstats, inst, tinst);
+ } else {
+ inst = spx5_inst_get(sparx5, TARGET_ASM, 0);
+ sparx5_get_asm_mac_ctrl_stats(portstats, inst, portno);
+ }
+ mac_ctrl_stats->MACControlFramesTransmitted =
+ portstats[spx5_stats_tx_pause_cnt] +
+ portstats[spx5_stats_pmac_tx_pause_cnt];
+ mac_ctrl_stats->MACControlFramesReceived =
+ portstats[spx5_stats_rx_pause_cnt] +
+ portstats[spx5_stats_pmac_rx_pause_cnt];
+ mac_ctrl_stats->UnsupportedOpcodesReceived =
+ portstats[spx5_stats_rx_unsup_opcode_cnt] +
+ portstats[spx5_stats_pmac_rx_unsup_opcode_cnt];
+}
+
+static void sparx5_get_eth_rmon_stats(struct net_device *ndev,
+ struct ethtool_rmon_stats *rmon_stats,
+ const struct ethtool_rmon_hist_range **ranges)
+{
+ struct sparx5_port *port = netdev_priv(ndev);
+ struct sparx5 *sparx5 = port->sparx5;
+ int portno = port->portno;
+ void __iomem *inst;
+ u64 *portstats;
+
+ portstats = &sparx5->stats[portno * sparx5->num_stats];
+ if (sparx5_is_baser(port->conf.portmode)) {
+ u32 tinst = sparx5_port_dev_index(portno);
+ u32 dev = sparx5_to_high_dev(portno);
+
+ inst = spx5_inst_get(sparx5, dev, tinst);
+ sparx5_get_dev_rmon_stats(portstats, inst, tinst);
+ } else {
+ inst = spx5_inst_get(sparx5, TARGET_ASM, 0);
+ sparx5_get_asm_rmon_stats(portstats, inst, portno);
+ }
+ rmon_stats->undersize_pkts = portstats[spx5_stats_rx_undersize_cnt] +
+ portstats[spx5_stats_pmac_rx_undersize_cnt];
+ rmon_stats->oversize_pkts = portstats[spx5_stats_rx_oversize_cnt] +
+ portstats[spx5_stats_pmac_rx_oversize_cnt];
+ rmon_stats->fragments = portstats[spx5_stats_rx_fragments_cnt] +
+ portstats[spx5_stats_pmac_rx_fragments_cnt];
+ rmon_stats->jabbers = portstats[spx5_stats_rx_jabbers_cnt] +
+ portstats[spx5_stats_pmac_rx_jabbers_cnt];
+ rmon_stats->hist[0] = portstats[spx5_stats_rx_size64_cnt] +
+ portstats[spx5_stats_pmac_rx_size64_cnt];
+ rmon_stats->hist[1] = portstats[spx5_stats_rx_size65to127_cnt] +
+ portstats[spx5_stats_pmac_rx_size65to127_cnt];
+ rmon_stats->hist[2] = portstats[spx5_stats_rx_size128to255_cnt] +
+ portstats[spx5_stats_pmac_rx_size128to255_cnt];
+ rmon_stats->hist[3] = portstats[spx5_stats_rx_size256to511_cnt] +
+ portstats[spx5_stats_pmac_rx_size256to511_cnt];
+ rmon_stats->hist[4] = portstats[spx5_stats_rx_size512to1023_cnt] +
+ portstats[spx5_stats_pmac_rx_size512to1023_cnt];
+ rmon_stats->hist[5] = portstats[spx5_stats_rx_size1024to1518_cnt] +
+ portstats[spx5_stats_pmac_rx_size1024to1518_cnt];
+ rmon_stats->hist[6] = portstats[spx5_stats_rx_size1519tomax_cnt] +
+ portstats[spx5_stats_pmac_rx_size1519tomax_cnt];
+ rmon_stats->hist_tx[0] = portstats[spx5_stats_tx_size64_cnt] +
+ portstats[spx5_stats_pmac_tx_size64_cnt];
+ rmon_stats->hist_tx[1] = portstats[spx5_stats_tx_size65to127_cnt] +
+ portstats[spx5_stats_pmac_tx_size65to127_cnt];
+ rmon_stats->hist_tx[2] = portstats[spx5_stats_tx_size128to255_cnt] +
+ portstats[spx5_stats_pmac_tx_size128to255_cnt];
+ rmon_stats->hist_tx[3] = portstats[spx5_stats_tx_size256to511_cnt] +
+ portstats[spx5_stats_pmac_tx_size256to511_cnt];
+ rmon_stats->hist_tx[4] = portstats[spx5_stats_tx_size512to1023_cnt] +
+ portstats[spx5_stats_pmac_tx_size512to1023_cnt];
+ rmon_stats->hist_tx[5] = portstats[spx5_stats_tx_size1024to1518_cnt] +
+ portstats[spx5_stats_pmac_tx_size1024to1518_cnt];
+ rmon_stats->hist_tx[6] = portstats[spx5_stats_tx_size1519tomax_cnt] +
+ portstats[spx5_stats_pmac_tx_size1519tomax_cnt];
+ *ranges = sparx5_rmon_ranges;
+}
+
+static int sparx5_get_sset_count(struct net_device *ndev, int sset)
+{
+ struct sparx5_port *port = netdev_priv(ndev);
+ struct sparx5 *sparx5 = port->sparx5;
+
+ if (sset != ETH_SS_STATS)
+ return -EOPNOTSUPP;
+ return sparx5->num_ethtool_stats;
+}
+
+static void sparx5_get_sset_strings(struct net_device *ndev, u32 sset, u8 *data)
+{
+ struct sparx5_port *port = netdev_priv(ndev);
+ struct sparx5 *sparx5 = port->sparx5;
+ int idx;
+
+ if (sset != ETH_SS_STATS)
+ return;
+
+ for (idx = 0; idx < sparx5->num_ethtool_stats; idx++)
+ strncpy(data + idx * ETH_GSTRING_LEN,
+ sparx5->stats_layout[idx], ETH_GSTRING_LEN);
+}
+
+static void sparx5_get_sset_data(struct net_device *ndev,
+ struct ethtool_stats *stats, u64 *data)
+{
+ struct sparx5_port *port = netdev_priv(ndev);
+ struct sparx5 *sparx5 = port->sparx5;
+ int portno = port->portno;
+ void __iomem *inst;
+ u64 *portstats;
+ int idx;
+
+ portstats = &sparx5->stats[portno * sparx5->num_stats];
+ if (sparx5_is_baser(port->conf.portmode)) {
+ u32 tinst = sparx5_port_dev_index(portno);
+ u32 dev = sparx5_to_high_dev(portno);
+
+ inst = spx5_inst_get(sparx5, dev, tinst);
+ sparx5_get_dev_misc_stats(portstats, inst, tinst);
+ } else {
+ inst = spx5_inst_get(sparx5, TARGET_ASM, 0);
+ sparx5_get_asm_misc_stats(portstats, inst, portno);
+ }
+ sparx5_get_ana_ac_stats_stats(sparx5, portno);
+ sparx5_get_queue_sys_stats(sparx5, portno);
+ /* Copy port counters to the ethtool buffer */
+ for (idx = spx5_stats_mm_rx_assembly_err_cnt;
+ idx < spx5_stats_mm_rx_assembly_err_cnt +
+ sparx5->num_ethtool_stats; idx++)
+ *data++ = portstats[idx];
+}
+
+void sparx5_get_stats64(struct net_device *ndev,
+ struct rtnl_link_stats64 *stats)
+{
+ struct sparx5_port *port = netdev_priv(ndev);
+ struct sparx5 *sparx5 = port->sparx5;
+ u64 *portstats;
+ int idx;
+
+ if (!sparx5->stats)
+ return; /* Not initialized yet */
+
+ portstats = &sparx5->stats[port->portno * sparx5->num_stats];
+
+ stats->rx_packets = portstats[spx5_stats_rx_uc_cnt] +
+ portstats[spx5_stats_pmac_rx_uc_cnt] +
+ portstats[spx5_stats_rx_mc_cnt] +
+ portstats[spx5_stats_rx_bc_cnt];
+ stats->tx_packets = portstats[spx5_stats_tx_uc_cnt] +
+ portstats[spx5_stats_pmac_tx_uc_cnt] +
+ portstats[spx5_stats_tx_mc_cnt] +
+ portstats[spx5_stats_tx_bc_cnt];
+ stats->rx_bytes = portstats[spx5_stats_rx_ok_bytes_cnt] +
+ portstats[spx5_stats_pmac_rx_ok_bytes_cnt];
+ stats->tx_bytes = portstats[spx5_stats_tx_ok_bytes_cnt] +
+ portstats[spx5_stats_pmac_tx_ok_bytes_cnt];
+ stats->rx_errors = portstats[spx5_stats_rx_in_range_len_err_cnt] +
+ portstats[spx5_stats_pmac_rx_in_range_len_err_cnt] +
+ portstats[spx5_stats_rx_out_of_range_len_err_cnt] +
+ portstats[spx5_stats_pmac_rx_out_of_range_len_err_cnt] +
+ portstats[spx5_stats_rx_oversize_cnt] +
+ portstats[spx5_stats_pmac_rx_oversize_cnt] +
+ portstats[spx5_stats_rx_crc_err_cnt] +
+ portstats[spx5_stats_pmac_rx_crc_err_cnt] +
+ portstats[spx5_stats_rx_alignment_lost_cnt] +
+ portstats[spx5_stats_pmac_rx_alignment_lost_cnt];
+ stats->tx_errors = portstats[spx5_stats_tx_xcoll_cnt] +
+ portstats[spx5_stats_tx_csense_cnt] +
+ portstats[spx5_stats_tx_late_coll_cnt];
+ stats->multicast = portstats[spx5_stats_rx_mc_cnt] +
+ portstats[spx5_stats_pmac_rx_mc_cnt];
+ stats->collisions = portstats[spx5_stats_tx_late_coll_cnt] +
+ portstats[spx5_stats_tx_xcoll_cnt] +
+ portstats[spx5_stats_tx_backoff1_cnt];
+ stats->rx_length_errors = portstats[spx5_stats_rx_in_range_len_err_cnt] +
+ portstats[spx5_stats_pmac_rx_in_range_len_err_cnt] +
+ portstats[spx5_stats_rx_out_of_range_len_err_cnt] +
+ portstats[spx5_stats_pmac_rx_out_of_range_len_err_cnt] +
+ portstats[spx5_stats_rx_oversize_cnt] +
+ portstats[spx5_stats_pmac_rx_oversize_cnt];
+ stats->rx_crc_errors = portstats[spx5_stats_rx_crc_err_cnt] +
+ portstats[spx5_stats_pmac_rx_crc_err_cnt];
+ stats->rx_frame_errors = portstats[spx5_stats_rx_alignment_lost_cnt] +
+ portstats[spx5_stats_pmac_rx_alignment_lost_cnt];
+ stats->tx_aborted_errors = portstats[spx5_stats_tx_xcoll_cnt];
+ stats->tx_carrier_errors = portstats[spx5_stats_tx_csense_cnt];
+ stats->tx_window_errors = portstats[spx5_stats_tx_late_coll_cnt];
+ stats->rx_dropped = portstats[spx5_stats_ana_ac_port_stat_lsb_cnt];
+ for (idx = 0; idx < 2 * SPX5_PRIOS; ++idx, ++stats)
+ stats->rx_dropped += portstats[spx5_stats_green_p0_rx_port_drop
+ + idx];
+ stats->tx_dropped = portstats[spx5_stats_tx_local_drop];
+}
+
+static void sparx5_update_port_stats(struct sparx5 *sparx5, int portno)
+{
+ if (sparx5_is_baser(sparx5->ports[portno]->conf.portmode))
+ sparx5_get_device_stats(sparx5, portno);
+ else
+ sparx5_get_asm_stats(sparx5, portno);
+ sparx5_get_ana_ac_stats_stats(sparx5, portno);
+ sparx5_get_queue_sys_stats(sparx5, portno);
+}
+
+static void sparx5_update_stats(struct sparx5 *sparx5)
+{
+ int idx;
+
+ for (idx = 0; idx < SPX5_PORTS; idx++)
+ if (sparx5->ports[idx])
+ sparx5_update_port_stats(sparx5, idx);
+}
+
+static void sparx5_check_stats_work(struct work_struct *work)
+{
+ struct delayed_work *dwork = to_delayed_work(work);
+ struct sparx5 *sparx5 = container_of(dwork,
+ struct sparx5,
+ stats_work);
+
+ sparx5_update_stats(sparx5);
+
+ queue_delayed_work(sparx5->stats_queue, &sparx5->stats_work,
+ SPX5_STATS_CHECK_DELAY);
+}
+
+static int sparx5_get_link_settings(struct net_device *ndev,
+ struct ethtool_link_ksettings *cmd)
+{
+ struct sparx5_port *port = netdev_priv(ndev);
+
+ return phylink_ethtool_ksettings_get(port->phylink, cmd);
+}
+
+static int sparx5_set_link_settings(struct net_device *ndev,
+ const struct ethtool_link_ksettings *cmd)
+{
+ struct sparx5_port *port = netdev_priv(ndev);
+
+ return phylink_ethtool_ksettings_set(port->phylink, cmd);
+}
+
+static void sparx5_config_stats(struct sparx5 *sparx5)
+{
+ /* Enable global events for port policer drops */
+ spx5_rmw(ANA_AC_PORT_SGE_CFG_MASK_SET(0xf0f0),
+ ANA_AC_PORT_SGE_CFG_MASK,
+ sparx5,
+ ANA_AC_PORT_SGE_CFG(SPX5_PORT_POLICER_DROPS));
+}
+
+static void sparx5_config_port_stats(struct sparx5 *sparx5, int portno)
+{
+ /* Clear Queue System counters */
+ spx5_wr(XQS_STAT_CFG_STAT_VIEW_SET(portno) |
+ XQS_STAT_CFG_STAT_CLEAR_SHOT_SET(3), sparx5,
+ XQS_STAT_CFG);
+
+ /* Use counter for port policer drop count */
+ spx5_rmw(ANA_AC_PORT_STAT_CFG_CFG_CNT_FRM_TYPE_SET(1) |
+ ANA_AC_PORT_STAT_CFG_CFG_CNT_BYTE_SET(0) |
+ ANA_AC_PORT_STAT_CFG_CFG_PRIO_MASK_SET(0xff),
+ ANA_AC_PORT_STAT_CFG_CFG_CNT_FRM_TYPE |
+ ANA_AC_PORT_STAT_CFG_CFG_CNT_BYTE |
+ ANA_AC_PORT_STAT_CFG_CFG_PRIO_MASK,
+ sparx5, ANA_AC_PORT_STAT_CFG(portno, SPX5_PORT_POLICER_DROPS));
+}
+
+const struct ethtool_ops sparx5_ethtool_ops = {
+ .get_sset_count = sparx5_get_sset_count,
+ .get_strings = sparx5_get_sset_strings,
+ .get_ethtool_stats = sparx5_get_sset_data,
+ .get_link_ksettings = sparx5_get_link_settings,
+ .set_link_ksettings = sparx5_set_link_settings,
+ .get_link = ethtool_op_get_link,
+ .get_eth_phy_stats = sparx5_get_eth_phy_stats,
+ .get_eth_mac_stats = sparx5_get_eth_mac_stats,
+ .get_eth_ctrl_stats = sparx5_get_eth_mac_ctrl_stats,
+ .get_rmon_stats = sparx5_get_eth_rmon_stats,
+};
+
+int sparx_stats_init(struct sparx5 *sparx5)
+{
+ char queue_name[32];
+ int portno;
+
+ sparx5->stats_layout = sparx5_stats_layout;
+ sparx5->num_stats = spx5_stats_count;
+ sparx5->num_ethtool_stats = ARRAY_SIZE(sparx5_stats_layout);
+ sparx5->stats = devm_kcalloc(sparx5->dev,
+ SPX5_PORTS_ALL * sparx5->num_stats,
+ sizeof(u64), GFP_KERNEL);
+ if (!sparx5->stats)
+ return -ENOMEM;
+
+ mutex_init(&sparx5->queue_stats_lock);
+ sparx5_config_stats(sparx5);
+ for (portno = 0; portno < SPX5_PORTS; portno++)
+ if (sparx5->ports[portno])
+ sparx5_config_port_stats(sparx5, portno);
+
+ snprintf(queue_name, sizeof(queue_name), "%s-stats",
+ dev_name(sparx5->dev));
+ sparx5->stats_queue = create_singlethread_workqueue(queue_name);
+ INIT_DELAYED_WORK(&sparx5->stats_work, sparx5_check_stats_work);
+ queue_delayed_work(sparx5->stats_queue, &sparx5->stats_work,
+ SPX5_STATS_CHECK_DELAY);
+
+ return 0;
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/* Microchip Sparx5 Switch driver
+ *
+ * Copyright (c) 2021 Microchip Technology Inc. and its subsidiaries.
+ */
+
+#include <net/switchdev.h>
+#include <linux/if_bridge.h>
+#include <linux/iopoll.h>
+
+#include "sparx5_main_regs.h"
+#include "sparx5_main.h"
+
+/* Commands for Mac Table Command register */
+#define MAC_CMD_LEARN 0 /* Insert (Learn) 1 entry */
+#define MAC_CMD_UNLEARN 1 /* Unlearn (Forget) 1 entry */
+#define MAC_CMD_LOOKUP 2 /* Look up 1 entry */
+#define MAC_CMD_READ 3 /* Read entry at Mac Table Index */
+#define MAC_CMD_WRITE 4 /* Write entry at Mac Table Index */
+#define MAC_CMD_SCAN 5 /* Scan (Age or find next) */
+#define MAC_CMD_FIND_SMALLEST 6 /* Get next entry */
+#define MAC_CMD_CLEAR_ALL 7 /* Delete all entries in table */
+
+/* Commands for MAC_ENTRY_ADDR_TYPE */
+#define MAC_ENTRY_ADDR_TYPE_UPSID_PN 0
+#define MAC_ENTRY_ADDR_TYPE_UPSID_CPU_OR_INT 1
+#define MAC_ENTRY_ADDR_TYPE_GLAG 2
+#define MAC_ENTRY_ADDR_TYPE_MC_IDX 3
+
+#define TABLE_UPDATE_SLEEP_US 10
+#define TABLE_UPDATE_TIMEOUT_US 100000
+
+struct sparx5_mact_entry {
+ struct list_head list;
+ unsigned char mac[ETH_ALEN];
+ u32 flags;
+#define MAC_ENT_ALIVE BIT(0)
+#define MAC_ENT_MOVED BIT(1)
+#define MAC_ENT_LOCK BIT(2)
+ u16 vid;
+ u16 port;
+};
+
+static int sparx5_mact_get_status(struct sparx5 *sparx5)
+{
+ return spx5_rd(sparx5, LRN_COMMON_ACCESS_CTRL);
+}
+
+static int sparx5_mact_wait_for_completion(struct sparx5 *sparx5)
+{
+ u32 val;
+
+ return readx_poll_timeout(sparx5_mact_get_status,
+ sparx5, val,
+ LRN_COMMON_ACCESS_CTRL_MAC_TABLE_ACCESS_SHOT_GET(val) == 0,
+ TABLE_UPDATE_SLEEP_US, TABLE_UPDATE_TIMEOUT_US);
+}
+
+static void sparx5_mact_select(struct sparx5 *sparx5,
+ const unsigned char mac[ETH_ALEN],
+ u16 vid)
+{
+ u32 macl = 0, mach = 0;
+
+ /* Set the MAC address to handle and the vlan associated in a format
+ * understood by the hardware.
+ */
+ mach |= vid << 16;
+ mach |= mac[0] << 8;
+ mach |= mac[1] << 0;
+ macl |= mac[2] << 24;
+ macl |= mac[3] << 16;
+ macl |= mac[4] << 8;
+ macl |= mac[5] << 0;
+
+ spx5_wr(mach, sparx5, LRN_MAC_ACCESS_CFG_0);
+ spx5_wr(macl, sparx5, LRN_MAC_ACCESS_CFG_1);
+}
+
+int sparx5_mact_learn(struct sparx5 *sparx5, int pgid,
+ const unsigned char mac[ETH_ALEN], u16 vid)
+{
+ int addr, type, ret;
+
+ if (pgid < SPX5_PORTS) {
+ type = MAC_ENTRY_ADDR_TYPE_UPSID_PN;
+ addr = pgid % 32;
+ addr += (pgid / 32) << 5; /* Add upsid */
+ } else {
+ type = MAC_ENTRY_ADDR_TYPE_MC_IDX;
+ addr = pgid - SPX5_PORTS;
+ }
+
+ mutex_lock(&sparx5->lock);
+
+ sparx5_mact_select(sparx5, mac, vid);
+
+ /* MAC entry properties */
+ spx5_wr(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_ADDR_SET(addr) |
+ LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_ADDR_TYPE_SET(type) |
+ LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_VLD_SET(1) |
+ LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_LOCKED_SET(1),
+ sparx5, LRN_MAC_ACCESS_CFG_2);
+ spx5_wr(0, sparx5, LRN_MAC_ACCESS_CFG_3);
+
+ /* Insert/learn new entry */
+ spx5_wr(LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_CMD_SET(MAC_CMD_LEARN) |
+ LRN_COMMON_ACCESS_CTRL_MAC_TABLE_ACCESS_SHOT_SET(1),
+ sparx5, LRN_COMMON_ACCESS_CTRL);
+
+ ret = sparx5_mact_wait_for_completion(sparx5);
+
+ mutex_unlock(&sparx5->lock);
+
+ return ret;
+}
+
+int sparx5_mc_unsync(struct net_device *dev, const unsigned char *addr)
+{
+ struct sparx5_port *port = netdev_priv(dev);
+ struct sparx5 *sparx5 = port->sparx5;
+
+ return sparx5_mact_forget(sparx5, addr, port->pvid);
+}
+
+int sparx5_mc_sync(struct net_device *dev, const unsigned char *addr)
+{
+ struct sparx5_port *port = netdev_priv(dev);
+ struct sparx5 *sparx5 = port->sparx5;
+
+ return sparx5_mact_learn(sparx5, PGID_CPU, addr, port->pvid);
+}
+
+static int sparx5_mact_get(struct sparx5 *sparx5,
+ unsigned char mac[ETH_ALEN],
+ u16 *vid, u32 *pcfg2)
+{
+ u32 mach, macl, cfg2;
+ int ret = -ENOENT;
+
+ cfg2 = spx5_rd(sparx5, LRN_MAC_ACCESS_CFG_2);
+ if (LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_VLD_GET(cfg2)) {
+ mach = spx5_rd(sparx5, LRN_MAC_ACCESS_CFG_0);
+ macl = spx5_rd(sparx5, LRN_MAC_ACCESS_CFG_1);
+ mac[0] = ((mach >> 8) & 0xff);
+ mac[1] = ((mach >> 0) & 0xff);
+ mac[2] = ((macl >> 24) & 0xff);
+ mac[3] = ((macl >> 16) & 0xff);
+ mac[4] = ((macl >> 8) & 0xff);
+ mac[5] = ((macl >> 0) & 0xff);
+ *vid = mach >> 16;
+ *pcfg2 = cfg2;
+ ret = 0;
+ }
+
+ return ret;
+}
+
+bool sparx5_mact_getnext(struct sparx5 *sparx5,
+ unsigned char mac[ETH_ALEN], u16 *vid, u32 *pcfg2)
+{
+ u32 cfg2;
+ int ret;
+
+ mutex_lock(&sparx5->lock);
+
+ sparx5_mact_select(sparx5, mac, *vid);
+
+ spx5_wr(LRN_SCAN_NEXT_CFG_SCAN_NEXT_IGNORE_LOCKED_ENA_SET(1) |
+ LRN_SCAN_NEXT_CFG_SCAN_NEXT_UNTIL_FOUND_ENA_SET(1),
+ sparx5, LRN_SCAN_NEXT_CFG);
+ spx5_wr(LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_CMD_SET
+ (MAC_CMD_FIND_SMALLEST) |
+ LRN_COMMON_ACCESS_CTRL_MAC_TABLE_ACCESS_SHOT_SET(1),
+ sparx5, LRN_COMMON_ACCESS_CTRL);
+
+ ret = sparx5_mact_wait_for_completion(sparx5);
+ if (ret == 0) {
+ ret = sparx5_mact_get(sparx5, mac, vid, &cfg2);
+ if (ret == 0)
+ *pcfg2 = cfg2;
+ }
+
+ mutex_unlock(&sparx5->lock);
+
+ return ret == 0;
+}
+
+static int sparx5_mact_lookup(struct sparx5 *sparx5,
+ const unsigned char mac[ETH_ALEN],
+ u16 vid)
+{
+ int ret;
+
+ mutex_lock(&sparx5->lock);
+
+ sparx5_mact_select(sparx5, mac, vid);
+
+ /* Issue a lookup command */
+ spx5_wr(LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_CMD_SET(MAC_CMD_LOOKUP) |
+ LRN_COMMON_ACCESS_CTRL_MAC_TABLE_ACCESS_SHOT_SET(1),
+ sparx5, LRN_COMMON_ACCESS_CTRL);
+
+ ret = sparx5_mact_wait_for_completion(sparx5);
+ if (ret)
+ goto out;
+
+ ret = LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_VLD_GET
+ (spx5_rd(sparx5, LRN_MAC_ACCESS_CFG_2));
+
+out:
+ mutex_unlock(&sparx5->lock);
+
+ return ret;
+}
+
+int sparx5_mact_forget(struct sparx5 *sparx5,
+ const unsigned char mac[ETH_ALEN], u16 vid)
+{
+ int ret;
+
+ mutex_lock(&sparx5->lock);
+
+ sparx5_mact_select(sparx5, mac, vid);
+
+ /* Issue an unlearn command */
+ spx5_wr(LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_CMD_SET(MAC_CMD_UNLEARN) |
+ LRN_COMMON_ACCESS_CTRL_MAC_TABLE_ACCESS_SHOT_SET(1),
+ sparx5, LRN_COMMON_ACCESS_CTRL);
+
+ ret = sparx5_mact_wait_for_completion(sparx5);
+
+ mutex_unlock(&sparx5->lock);
+
+ return ret;
+}
+
+static struct sparx5_mact_entry *alloc_mact_entry(struct sparx5 *sparx5,
+ const unsigned char *mac,
+ u16 vid, u16 port_index)
+{
+ struct sparx5_mact_entry *mact_entry;
+
+ mact_entry = devm_kzalloc(sparx5->dev,
+ sizeof(*mact_entry), GFP_ATOMIC);
+ if (!mact_entry)
+ return NULL;
+
+ memcpy(mact_entry->mac, mac, ETH_ALEN);
+ mact_entry->vid = vid;
+ mact_entry->port = port_index;
+ return mact_entry;
+}
+
+static struct sparx5_mact_entry *find_mact_entry(struct sparx5 *sparx5,
+ const unsigned char *mac,
+ u16 vid, u16 port_index)
+{
+ struct sparx5_mact_entry *mact_entry;
+ struct sparx5_mact_entry *res = NULL;
+
+ mutex_lock(&sparx5->mact_lock);
+ list_for_each_entry(mact_entry, &sparx5->mact_entries, list) {
+ if (mact_entry->vid == vid &&
+ ether_addr_equal(mac, mact_entry->mac) &&
+ mact_entry->port == port_index) {
+ res = mact_entry;
+ break;
+ }
+ }
+ mutex_unlock(&sparx5->mact_lock);
+
+ return res;
+}
+
+static void sparx5_fdb_call_notifiers(enum switchdev_notifier_type type,
+ const char *mac, u16 vid,
+ struct net_device *dev, bool offloaded)
+{
+ struct switchdev_notifier_fdb_info info;
+
+ info.addr = mac;
+ info.vid = vid;
+ info.offloaded = offloaded;
+ call_switchdev_notifiers(type, dev, &info.info, NULL);
+}
+
+int sparx5_add_mact_entry(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ const unsigned char *addr, u16 vid)
+{
+ struct sparx5_mact_entry *mact_entry;
+ int ret;
+
+ ret = sparx5_mact_lookup(sparx5, addr, vid);
+ if (ret)
+ return 0;
+
+ /* In case the entry already exists, don't add it again to SW,
+ * just update HW, but we need to look in the actual HW because
+ * it is possible for an entry to be learn by HW and before the
+ * mact thread to start the frame will reach CPU and the CPU will
+ * add the entry but without the extern_learn flag.
+ */
+ mact_entry = find_mact_entry(sparx5, addr, vid, port->portno);
+ if (mact_entry)
+ goto update_hw;
+
+ /* Add the entry in SW MAC table not to get the notification when
+ * SW is pulling again
+ */
+ mact_entry = alloc_mact_entry(sparx5, addr, vid, port->portno);
+ if (!mact_entry)
+ return -ENOMEM;
+
+ mutex_lock(&sparx5->mact_lock);
+ list_add_tail(&mact_entry->list, &sparx5->mact_entries);
+ mutex_unlock(&sparx5->mact_lock);
+
+update_hw:
+ ret = sparx5_mact_learn(sparx5, port->portno, addr, vid);
+
+ /* New entry? */
+ if (mact_entry->flags == 0) {
+ mact_entry->flags |= MAC_ENT_LOCK; /* Don't age this */
+ sparx5_fdb_call_notifiers(SWITCHDEV_FDB_ADD_TO_BRIDGE, addr, vid,
+ port->ndev, true);
+ }
+
+ return ret;
+}
+
+int sparx5_del_mact_entry(struct sparx5 *sparx5,
+ const unsigned char *addr,
+ u16 vid)
+{
+ struct sparx5_mact_entry *mact_entry, *tmp;
+
+ /* Delete the entry in SW MAC table not to get the notification when
+ * SW is pulling again
+ */
+ mutex_lock(&sparx5->mact_lock);
+ list_for_each_entry_safe(mact_entry, tmp, &sparx5->mact_entries,
+ list) {
+ if ((vid == 0 || mact_entry->vid == vid) &&
+ ether_addr_equal(addr, mact_entry->mac)) {
+ list_del(&mact_entry->list);
+ devm_kfree(sparx5->dev, mact_entry);
+
+ sparx5_mact_forget(sparx5, addr, mact_entry->vid);
+ }
+ }
+ mutex_unlock(&sparx5->mact_lock);
+
+ return 0;
+}
+
+static void sparx5_mact_handle_entry(struct sparx5 *sparx5,
+ unsigned char mac[ETH_ALEN],
+ u16 vid, u32 cfg2)
+{
+ struct sparx5_mact_entry *mact_entry;
+ bool found = false;
+ u16 port;
+
+ if (LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_ADDR_TYPE_GET(cfg2) !=
+ MAC_ENTRY_ADDR_TYPE_UPSID_PN)
+ return;
+
+ port = LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_ADDR_GET(cfg2);
+ if (port >= SPX5_PORTS)
+ return;
+
+ if (!test_bit(port, sparx5->bridge_mask))
+ return;
+
+ mutex_lock(&sparx5->mact_lock);
+ list_for_each_entry(mact_entry, &sparx5->mact_entries, list) {
+ if (mact_entry->vid == vid &&
+ ether_addr_equal(mac, mact_entry->mac)) {
+ found = true;
+ mact_entry->flags |= MAC_ENT_ALIVE;
+ if (mact_entry->port != port) {
+ dev_warn(sparx5->dev, "Entry move: %d -> %d\n",
+ mact_entry->port, port);
+ mact_entry->port = port;
+ mact_entry->flags |= MAC_ENT_MOVED;
+ }
+ /* Entry handled */
+ break;
+ }
+ }
+ mutex_unlock(&sparx5->mact_lock);
+
+ if (found && !(mact_entry->flags & MAC_ENT_MOVED))
+ /* Present, not moved */
+ return;
+
+ if (!found) {
+ /* Entry not found - now add */
+ mact_entry = alloc_mact_entry(sparx5, mac, vid, port);
+ if (!mact_entry)
+ return;
+
+ mact_entry->flags |= MAC_ENT_ALIVE;
+ mutex_lock(&sparx5->mact_lock);
+ list_add_tail(&mact_entry->list, &sparx5->mact_entries);
+ mutex_unlock(&sparx5->mact_lock);
+ }
+
+ /* New or moved entry - notify bridge */
+ sparx5_fdb_call_notifiers(SWITCHDEV_FDB_ADD_TO_BRIDGE,
+ mac, vid, sparx5->ports[port]->ndev,
+ true);
+}
+
+void sparx5_mact_pull_work(struct work_struct *work)
+{
+ struct delayed_work *del_work = to_delayed_work(work);
+ struct sparx5 *sparx5 = container_of(del_work, struct sparx5,
+ mact_work);
+ struct sparx5_mact_entry *mact_entry, *tmp;
+ unsigned char mac[ETH_ALEN];
+ u32 cfg2;
+ u16 vid;
+ int ret;
+
+ /* Reset MAC entry flags */
+ mutex_lock(&sparx5->mact_lock);
+ list_for_each_entry(mact_entry, &sparx5->mact_entries, list)
+ mact_entry->flags &= MAC_ENT_LOCK;
+ mutex_unlock(&sparx5->mact_lock);
+
+ /* MAIN mac address processing loop */
+ vid = 0;
+ memset(mac, 0, sizeof(mac));
+ do {
+ mutex_lock(&sparx5->lock);
+ sparx5_mact_select(sparx5, mac, vid);
+ spx5_wr(LRN_SCAN_NEXT_CFG_SCAN_NEXT_UNTIL_FOUND_ENA_SET(1),
+ sparx5, LRN_SCAN_NEXT_CFG);
+ spx5_wr(LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_CMD_SET
+ (MAC_CMD_FIND_SMALLEST) |
+ LRN_COMMON_ACCESS_CTRL_MAC_TABLE_ACCESS_SHOT_SET(1),
+ sparx5, LRN_COMMON_ACCESS_CTRL);
+ ret = sparx5_mact_wait_for_completion(sparx5);
+ if (ret == 0)
+ ret = sparx5_mact_get(sparx5, mac, &vid, &cfg2);
+ mutex_unlock(&sparx5->lock);
+ if (ret == 0)
+ sparx5_mact_handle_entry(sparx5, mac, vid, cfg2);
+ } while (ret == 0);
+
+ mutex_lock(&sparx5->mact_lock);
+ list_for_each_entry_safe(mact_entry, tmp, &sparx5->mact_entries,
+ list) {
+ /* If the entry is in HW or permanent, then skip */
+ if (mact_entry->flags & (MAC_ENT_ALIVE | MAC_ENT_LOCK))
+ continue;
+
+ sparx5_fdb_call_notifiers(SWITCHDEV_FDB_DEL_TO_BRIDGE,
+ mact_entry->mac, mact_entry->vid,
+ sparx5->ports[mact_entry->port]->ndev,
+ true);
+
+ list_del(&mact_entry->list);
+ devm_kfree(sparx5->dev, mact_entry);
+ }
+ mutex_unlock(&sparx5->mact_lock);
+
+ queue_delayed_work(sparx5->mact_queue, &sparx5->mact_work,
+ SPX5_MACT_PULL_DELAY);
+}
+
+void sparx5_set_ageing(struct sparx5 *sparx5, int msecs)
+{
+ int value = max(1, msecs / 10); /* unit 10 ms */
+
+ spx5_rmw(LRN_AUTOAGE_CFG_UNIT_SIZE_SET(2) | /* 10 ms */
+ LRN_AUTOAGE_CFG_PERIOD_VAL_SET(value / 2), /* one bit ageing */
+ LRN_AUTOAGE_CFG_UNIT_SIZE |
+ LRN_AUTOAGE_CFG_PERIOD_VAL,
+ sparx5,
+ LRN_AUTOAGE_CFG(0));
+}
+
+void sparx5_mact_init(struct sparx5 *sparx5)
+{
+ mutex_init(&sparx5->lock);
+
+ /* Flush MAC table */
+ spx5_wr(LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_CMD_SET(MAC_CMD_CLEAR_ALL) |
+ LRN_COMMON_ACCESS_CTRL_MAC_TABLE_ACCESS_SHOT_SET(1),
+ sparx5, LRN_COMMON_ACCESS_CTRL);
+
+ if (sparx5_mact_wait_for_completion(sparx5) != 0)
+ dev_warn(sparx5->dev, "MAC flush error\n");
+
+ sparx5_set_ageing(sparx5, BR_DEFAULT_AGEING_TIME / HZ * 1000);
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/* Microchip Sparx5 Switch driver
+ *
+ * Copyright (c) 2021 Microchip Technology Inc. and its subsidiaries.
+ *
+ * The Sparx5 Chip Register Model can be browsed at this location:
+ * https://github.com/microchip-ung/sparx-5_reginfo
+ */
+#include <linux/module.h>
+#include <linux/device.h>
+#include <linux/netdevice.h>
+#include <linux/platform_device.h>
+#include <linux/interrupt.h>
+#include <linux/of.h>
+#include <linux/of_net.h>
+#include <linux/of_mdio.h>
+#include <net/switchdev.h>
+#include <linux/etherdevice.h>
+#include <linux/io.h>
+#include <linux/printk.h>
+#include <linux/iopoll.h>
+#include <linux/mfd/syscon.h>
+#include <linux/regmap.h>
+#include <linux/types.h>
+#include <linux/reset.h>
+
+#include "sparx5_main_regs.h"
+#include "sparx5_main.h"
+#include "sparx5_port.h"
+
+#define QLIM_WM(fraction) \
+ ((SPX5_BUFFER_MEMORY / SPX5_BUFFER_CELL_SZ - 100) * (fraction) / 100)
+#define IO_RANGES 3
+
+struct initial_port_config {
+ u32 portno;
+ struct device_node *node;
+ struct sparx5_port_config conf;
+ struct phy *serdes;
+};
+
+struct sparx5_ram_config {
+ void __iomem *init_reg;
+ u32 init_val;
+};
+
+struct sparx5_main_io_resource {
+ enum sparx5_target id;
+ phys_addr_t offset;
+ int range;
+};
+
+static const struct sparx5_main_io_resource sparx5_main_iomap[] = {
+ { TARGET_CPU, 0, 0 }, /* 0x600000000 */
+ { TARGET_FDMA, 0x80000, 0 }, /* 0x600080000 */
+ { TARGET_PCEP, 0x400000, 0 }, /* 0x600400000 */
+ { TARGET_DEV2G5, 0x10004000, 1 }, /* 0x610004000 */
+ { TARGET_DEV5G, 0x10008000, 1 }, /* 0x610008000 */
+ { TARGET_PCS5G_BR, 0x1000c000, 1 }, /* 0x61000c000 */
+ { TARGET_DEV2G5 + 1, 0x10010000, 1 }, /* 0x610010000 */
+ { TARGET_DEV5G + 1, 0x10014000, 1 }, /* 0x610014000 */
+ { TARGET_PCS5G_BR + 1, 0x10018000, 1 }, /* 0x610018000 */
+ { TARGET_DEV2G5 + 2, 0x1001c000, 1 }, /* 0x61001c000 */
+ { TARGET_DEV5G + 2, 0x10020000, 1 }, /* 0x610020000 */
+ { TARGET_PCS5G_BR + 2, 0x10024000, 1 }, /* 0x610024000 */
+ { TARGET_DEV2G5 + 6, 0x10028000, 1 }, /* 0x610028000 */
+ { TARGET_DEV5G + 6, 0x1002c000, 1 }, /* 0x61002c000 */
+ { TARGET_PCS5G_BR + 6, 0x10030000, 1 }, /* 0x610030000 */
+ { TARGET_DEV2G5 + 7, 0x10034000, 1 }, /* 0x610034000 */
+ { TARGET_DEV5G + 7, 0x10038000, 1 }, /* 0x610038000 */
+ { TARGET_PCS5G_BR + 7, 0x1003c000, 1 }, /* 0x61003c000 */
+ { TARGET_DEV2G5 + 8, 0x10040000, 1 }, /* 0x610040000 */
+ { TARGET_DEV5G + 8, 0x10044000, 1 }, /* 0x610044000 */
+ { TARGET_PCS5G_BR + 8, 0x10048000, 1 }, /* 0x610048000 */
+ { TARGET_DEV2G5 + 9, 0x1004c000, 1 }, /* 0x61004c000 */
+ { TARGET_DEV5G + 9, 0x10050000, 1 }, /* 0x610050000 */
+ { TARGET_PCS5G_BR + 9, 0x10054000, 1 }, /* 0x610054000 */
+ { TARGET_DEV2G5 + 10, 0x10058000, 1 }, /* 0x610058000 */
+ { TARGET_DEV5G + 10, 0x1005c000, 1 }, /* 0x61005c000 */
+ { TARGET_PCS5G_BR + 10, 0x10060000, 1 }, /* 0x610060000 */
+ { TARGET_DEV2G5 + 11, 0x10064000, 1 }, /* 0x610064000 */
+ { TARGET_DEV5G + 11, 0x10068000, 1 }, /* 0x610068000 */
+ { TARGET_PCS5G_BR + 11, 0x1006c000, 1 }, /* 0x61006c000 */
+ { TARGET_DEV2G5 + 12, 0x10070000, 1 }, /* 0x610070000 */
+ { TARGET_DEV10G, 0x10074000, 1 }, /* 0x610074000 */
+ { TARGET_PCS10G_BR, 0x10078000, 1 }, /* 0x610078000 */
+ { TARGET_DEV2G5 + 14, 0x1007c000, 1 }, /* 0x61007c000 */
+ { TARGET_DEV10G + 2, 0x10080000, 1 }, /* 0x610080000 */
+ { TARGET_PCS10G_BR + 2, 0x10084000, 1 }, /* 0x610084000 */
+ { TARGET_DEV2G5 + 15, 0x10088000, 1 }, /* 0x610088000 */
+ { TARGET_DEV10G + 3, 0x1008c000, 1 }, /* 0x61008c000 */
+ { TARGET_PCS10G_BR + 3, 0x10090000, 1 }, /* 0x610090000 */
+ { TARGET_DEV2G5 + 16, 0x10094000, 1 }, /* 0x610094000 */
+ { TARGET_DEV2G5 + 17, 0x10098000, 1 }, /* 0x610098000 */
+ { TARGET_DEV2G5 + 18, 0x1009c000, 1 }, /* 0x61009c000 */
+ { TARGET_DEV2G5 + 19, 0x100a0000, 1 }, /* 0x6100a0000 */
+ { TARGET_DEV2G5 + 20, 0x100a4000, 1 }, /* 0x6100a4000 */
+ { TARGET_DEV2G5 + 21, 0x100a8000, 1 }, /* 0x6100a8000 */
+ { TARGET_DEV2G5 + 22, 0x100ac000, 1 }, /* 0x6100ac000 */
+ { TARGET_DEV2G5 + 23, 0x100b0000, 1 }, /* 0x6100b0000 */
+ { TARGET_DEV2G5 + 32, 0x100b4000, 1 }, /* 0x6100b4000 */
+ { TARGET_DEV2G5 + 33, 0x100b8000, 1 }, /* 0x6100b8000 */
+ { TARGET_DEV2G5 + 34, 0x100bc000, 1 }, /* 0x6100bc000 */
+ { TARGET_DEV2G5 + 35, 0x100c0000, 1 }, /* 0x6100c0000 */
+ { TARGET_DEV2G5 + 36, 0x100c4000, 1 }, /* 0x6100c4000 */
+ { TARGET_DEV2G5 + 37, 0x100c8000, 1 }, /* 0x6100c8000 */
+ { TARGET_DEV2G5 + 38, 0x100cc000, 1 }, /* 0x6100cc000 */
+ { TARGET_DEV2G5 + 39, 0x100d0000, 1 }, /* 0x6100d0000 */
+ { TARGET_DEV2G5 + 40, 0x100d4000, 1 }, /* 0x6100d4000 */
+ { TARGET_DEV2G5 + 41, 0x100d8000, 1 }, /* 0x6100d8000 */
+ { TARGET_DEV2G5 + 42, 0x100dc000, 1 }, /* 0x6100dc000 */
+ { TARGET_DEV2G5 + 43, 0x100e0000, 1 }, /* 0x6100e0000 */
+ { TARGET_DEV2G5 + 44, 0x100e4000, 1 }, /* 0x6100e4000 */
+ { TARGET_DEV2G5 + 45, 0x100e8000, 1 }, /* 0x6100e8000 */
+ { TARGET_DEV2G5 + 46, 0x100ec000, 1 }, /* 0x6100ec000 */
+ { TARGET_DEV2G5 + 47, 0x100f0000, 1 }, /* 0x6100f0000 */
+ { TARGET_DEV2G5 + 57, 0x100f4000, 1 }, /* 0x6100f4000 */
+ { TARGET_DEV25G + 1, 0x100f8000, 1 }, /* 0x6100f8000 */
+ { TARGET_PCS25G_BR + 1, 0x100fc000, 1 }, /* 0x6100fc000 */
+ { TARGET_DEV2G5 + 59, 0x10104000, 1 }, /* 0x610104000 */
+ { TARGET_DEV25G + 3, 0x10108000, 1 }, /* 0x610108000 */
+ { TARGET_PCS25G_BR + 3, 0x1010c000, 1 }, /* 0x61010c000 */
+ { TARGET_DEV2G5 + 60, 0x10114000, 1 }, /* 0x610114000 */
+ { TARGET_DEV25G + 4, 0x10118000, 1 }, /* 0x610118000 */
+ { TARGET_PCS25G_BR + 4, 0x1011c000, 1 }, /* 0x61011c000 */
+ { TARGET_DEV2G5 + 64, 0x10124000, 1 }, /* 0x610124000 */
+ { TARGET_DEV5G + 12, 0x10128000, 1 }, /* 0x610128000 */
+ { TARGET_PCS5G_BR + 12, 0x1012c000, 1 }, /* 0x61012c000 */
+ { TARGET_PORT_CONF, 0x10130000, 1 }, /* 0x610130000 */
+ { TARGET_DEV2G5 + 3, 0x10404000, 1 }, /* 0x610404000 */
+ { TARGET_DEV5G + 3, 0x10408000, 1 }, /* 0x610408000 */
+ { TARGET_PCS5G_BR + 3, 0x1040c000, 1 }, /* 0x61040c000 */
+ { TARGET_DEV2G5 + 4, 0x10410000, 1 }, /* 0x610410000 */
+ { TARGET_DEV5G + 4, 0x10414000, 1 }, /* 0x610414000 */
+ { TARGET_PCS5G_BR + 4, 0x10418000, 1 }, /* 0x610418000 */
+ { TARGET_DEV2G5 + 5, 0x1041c000, 1 }, /* 0x61041c000 */
+ { TARGET_DEV5G + 5, 0x10420000, 1 }, /* 0x610420000 */
+ { TARGET_PCS5G_BR + 5, 0x10424000, 1 }, /* 0x610424000 */
+ { TARGET_DEV2G5 + 13, 0x10428000, 1 }, /* 0x610428000 */
+ { TARGET_DEV10G + 1, 0x1042c000, 1 }, /* 0x61042c000 */
+ { TARGET_PCS10G_BR + 1, 0x10430000, 1 }, /* 0x610430000 */
+ { TARGET_DEV2G5 + 24, 0x10434000, 1 }, /* 0x610434000 */
+ { TARGET_DEV2G5 + 25, 0x10438000, 1 }, /* 0x610438000 */
+ { TARGET_DEV2G5 + 26, 0x1043c000, 1 }, /* 0x61043c000 */
+ { TARGET_DEV2G5 + 27, 0x10440000, 1 }, /* 0x610440000 */
+ { TARGET_DEV2G5 + 28, 0x10444000, 1 }, /* 0x610444000 */
+ { TARGET_DEV2G5 + 29, 0x10448000, 1 }, /* 0x610448000 */
+ { TARGET_DEV2G5 + 30, 0x1044c000, 1 }, /* 0x61044c000 */
+ { TARGET_DEV2G5 + 31, 0x10450000, 1 }, /* 0x610450000 */
+ { TARGET_DEV2G5 + 48, 0x10454000, 1 }, /* 0x610454000 */
+ { TARGET_DEV10G + 4, 0x10458000, 1 }, /* 0x610458000 */
+ { TARGET_PCS10G_BR + 4, 0x1045c000, 1 }, /* 0x61045c000 */
+ { TARGET_DEV2G5 + 49, 0x10460000, 1 }, /* 0x610460000 */
+ { TARGET_DEV10G + 5, 0x10464000, 1 }, /* 0x610464000 */
+ { TARGET_PCS10G_BR + 5, 0x10468000, 1 }, /* 0x610468000 */
+ { TARGET_DEV2G5 + 50, 0x1046c000, 1 }, /* 0x61046c000 */
+ { TARGET_DEV10G + 6, 0x10470000, 1 }, /* 0x610470000 */
+ { TARGET_PCS10G_BR + 6, 0x10474000, 1 }, /* 0x610474000 */
+ { TARGET_DEV2G5 + 51, 0x10478000, 1 }, /* 0x610478000 */
+ { TARGET_DEV10G + 7, 0x1047c000, 1 }, /* 0x61047c000 */
+ { TARGET_PCS10G_BR + 7, 0x10480000, 1 }, /* 0x610480000 */
+ { TARGET_DEV2G5 + 52, 0x10484000, 1 }, /* 0x610484000 */
+ { TARGET_DEV10G + 8, 0x10488000, 1 }, /* 0x610488000 */
+ { TARGET_PCS10G_BR + 8, 0x1048c000, 1 }, /* 0x61048c000 */
+ { TARGET_DEV2G5 + 53, 0x10490000, 1 }, /* 0x610490000 */
+ { TARGET_DEV10G + 9, 0x10494000, 1 }, /* 0x610494000 */
+ { TARGET_PCS10G_BR + 9, 0x10498000, 1 }, /* 0x610498000 */
+ { TARGET_DEV2G5 + 54, 0x1049c000, 1 }, /* 0x61049c000 */
+ { TARGET_DEV10G + 10, 0x104a0000, 1 }, /* 0x6104a0000 */
+ { TARGET_PCS10G_BR + 10, 0x104a4000, 1 }, /* 0x6104a4000 */
+ { TARGET_DEV2G5 + 55, 0x104a8000, 1 }, /* 0x6104a8000 */
+ { TARGET_DEV10G + 11, 0x104ac000, 1 }, /* 0x6104ac000 */
+ { TARGET_PCS10G_BR + 11, 0x104b0000, 1 }, /* 0x6104b0000 */
+ { TARGET_DEV2G5 + 56, 0x104b4000, 1 }, /* 0x6104b4000 */
+ { TARGET_DEV25G, 0x104b8000, 1 }, /* 0x6104b8000 */
+ { TARGET_PCS25G_BR, 0x104bc000, 1 }, /* 0x6104bc000 */
+ { TARGET_DEV2G5 + 58, 0x104c4000, 1 }, /* 0x6104c4000 */
+ { TARGET_DEV25G + 2, 0x104c8000, 1 }, /* 0x6104c8000 */
+ { TARGET_PCS25G_BR + 2, 0x104cc000, 1 }, /* 0x6104cc000 */
+ { TARGET_DEV2G5 + 61, 0x104d4000, 1 }, /* 0x6104d4000 */
+ { TARGET_DEV25G + 5, 0x104d8000, 1 }, /* 0x6104d8000 */
+ { TARGET_PCS25G_BR + 5, 0x104dc000, 1 }, /* 0x6104dc000 */
+ { TARGET_DEV2G5 + 62, 0x104e4000, 1 }, /* 0x6104e4000 */
+ { TARGET_DEV25G + 6, 0x104e8000, 1 }, /* 0x6104e8000 */
+ { TARGET_PCS25G_BR + 6, 0x104ec000, 1 }, /* 0x6104ec000 */
+ { TARGET_DEV2G5 + 63, 0x104f4000, 1 }, /* 0x6104f4000 */
+ { TARGET_DEV25G + 7, 0x104f8000, 1 }, /* 0x6104f8000 */
+ { TARGET_PCS25G_BR + 7, 0x104fc000, 1 }, /* 0x6104fc000 */
+ { TARGET_DSM, 0x10504000, 1 }, /* 0x610504000 */
+ { TARGET_ASM, 0x10600000, 1 }, /* 0x610600000 */
+ { TARGET_GCB, 0x11010000, 2 }, /* 0x611010000 */
+ { TARGET_QS, 0x11030000, 2 }, /* 0x611030000 */
+ { TARGET_ANA_ACL, 0x11050000, 2 }, /* 0x611050000 */
+ { TARGET_LRN, 0x11060000, 2 }, /* 0x611060000 */
+ { TARGET_VCAP_SUPER, 0x11080000, 2 }, /* 0x611080000 */
+ { TARGET_QSYS, 0x110a0000, 2 }, /* 0x6110a0000 */
+ { TARGET_QFWD, 0x110b0000, 2 }, /* 0x6110b0000 */
+ { TARGET_XQS, 0x110c0000, 2 }, /* 0x6110c0000 */
+ { TARGET_CLKGEN, 0x11100000, 2 }, /* 0x611100000 */
+ { TARGET_ANA_AC_POL, 0x11200000, 2 }, /* 0x611200000 */
+ { TARGET_QRES, 0x11280000, 2 }, /* 0x611280000 */
+ { TARGET_EACL, 0x112c0000, 2 }, /* 0x6112c0000 */
+ { TARGET_ANA_CL, 0x11400000, 2 }, /* 0x611400000 */
+ { TARGET_ANA_L3, 0x11480000, 2 }, /* 0x611480000 */
+ { TARGET_HSCH, 0x11580000, 2 }, /* 0x611580000 */
+ { TARGET_REW, 0x11600000, 2 }, /* 0x611600000 */
+ { TARGET_ANA_L2, 0x11800000, 2 }, /* 0x611800000 */
+ { TARGET_ANA_AC, 0x11900000, 2 }, /* 0x611900000 */
+ { TARGET_VOP, 0x11a00000, 2 }, /* 0x611a00000 */
+};
+
+static int sparx5_create_targets(struct sparx5 *sparx5)
+{
+ struct resource *iores[IO_RANGES];
+ void __iomem *iomem[IO_RANGES];
+ void __iomem *begin[IO_RANGES];
+ int range_id[IO_RANGES];
+ int idx, jdx;
+
+ for (idx = 0, jdx = 0; jdx < ARRAY_SIZE(sparx5_main_iomap); jdx++) {
+ const struct sparx5_main_io_resource *iomap = &sparx5_main_iomap[jdx];
+
+ if (idx == iomap->range) {
+ range_id[idx] = jdx;
+ idx++;
+ }
+ }
+ for (idx = 0; idx < IO_RANGES; idx++) {
+ iores[idx] = platform_get_resource(sparx5->pdev, IORESOURCE_MEM,
+ idx);
+ iomem[idx] = devm_ioremap(sparx5->dev,
+ iores[idx]->start,
+ iores[idx]->end - iores[idx]->start
+ + 1);
+ if (IS_ERR(iomem[idx])) {
+ dev_err(sparx5->dev, "Unable to get switch registers: %s\n",
+ iores[idx]->name);
+ return PTR_ERR(iomem[idx]);
+ }
+ begin[idx] = iomem[idx] - sparx5_main_iomap[range_id[idx]].offset;
+ }
+ for (jdx = 0; jdx < ARRAY_SIZE(sparx5_main_iomap); jdx++) {
+ const struct sparx5_main_io_resource *iomap = &sparx5_main_iomap[jdx];
+
+ sparx5->regs[iomap->id] = begin[iomap->range] + iomap->offset;
+ }
+ return 0;
+}
+
+static int sparx5_create_port(struct sparx5 *sparx5,
+ struct initial_port_config *config)
+{
+ struct sparx5_port *spx5_port;
+ struct net_device *ndev;
+ struct phylink *phylink;
+ int err;
+
+ ndev = sparx5_create_netdev(sparx5, config->portno);
+ if (IS_ERR(ndev)) {
+ dev_err(sparx5->dev, "Could not create net device: %02u\n",
+ config->portno);
+ return PTR_ERR(ndev);
+ }
+ spx5_port = netdev_priv(ndev);
+ spx5_port->of_node = config->node;
+ spx5_port->serdes = config->serdes;
+ spx5_port->pvid = NULL_VID;
+ spx5_port->signd_internal = true;
+ spx5_port->signd_active_high = true;
+ spx5_port->signd_enable = true;
+ spx5_port->max_vlan_tags = SPX5_PORT_MAX_TAGS_NONE;
+ spx5_port->vlan_type = SPX5_VLAN_PORT_TYPE_UNAWARE;
+ spx5_port->custom_etype = 0x8880; /* Vitesse */
+ spx5_port->phylink_pcs.poll = true;
+ spx5_port->phylink_pcs.ops = &sparx5_phylink_pcs_ops;
+ sparx5->ports[config->portno] = spx5_port;
+
+ err = sparx5_port_init(sparx5, spx5_port, &config->conf);
+ if (err) {
+ dev_err(sparx5->dev, "port init failed\n");
+ return err;
+ }
+ spx5_port->conf = config->conf;
+
+ /* Setup VLAN */
+ sparx5_vlan_port_setup(sparx5, spx5_port->portno);
+
+ /* Create a phylink for PHY management. Also handles SFPs */
+ spx5_port->phylink_config.dev = &spx5_port->ndev->dev;
+ spx5_port->phylink_config.type = PHYLINK_NETDEV;
+ spx5_port->phylink_config.pcs_poll = true;
+
+ phylink = phylink_create(&spx5_port->phylink_config,
+ of_fwnode_handle(config->node),
+ config->conf.phy_mode,
+ &sparx5_phylink_mac_ops);
+ if (IS_ERR(phylink))
+ return PTR_ERR(phylink);
+
+ spx5_port->phylink = phylink;
+ phylink_set_pcs(phylink, &spx5_port->phylink_pcs);
+
+ return 0;
+}
+
+static int sparx5_init_ram(struct sparx5 *s5)
+{
+ const struct sparx5_ram_config spx5_ram_cfg[] = {
+ {spx5_reg_get(s5, ANA_AC_STAT_RESET), ANA_AC_STAT_RESET_RESET},
+ {spx5_reg_get(s5, ASM_STAT_CFG), ASM_STAT_CFG_STAT_CNT_CLR_SHOT},
+ {spx5_reg_get(s5, QSYS_RAM_INIT), QSYS_RAM_INIT_RAM_INIT},
+ {spx5_reg_get(s5, REW_RAM_INIT), QSYS_RAM_INIT_RAM_INIT},
+ {spx5_reg_get(s5, VOP_RAM_INIT), QSYS_RAM_INIT_RAM_INIT},
+ {spx5_reg_get(s5, ANA_AC_RAM_INIT), QSYS_RAM_INIT_RAM_INIT},
+ {spx5_reg_get(s5, ASM_RAM_INIT), QSYS_RAM_INIT_RAM_INIT},
+ {spx5_reg_get(s5, EACL_RAM_INIT), QSYS_RAM_INIT_RAM_INIT},
+ {spx5_reg_get(s5, VCAP_SUPER_RAM_INIT), QSYS_RAM_INIT_RAM_INIT},
+ {spx5_reg_get(s5, DSM_RAM_INIT), QSYS_RAM_INIT_RAM_INIT}
+ };
+ const struct sparx5_ram_config *cfg;
+ u32 value, pending, jdx, idx;
+
+ for (jdx = 0; jdx < 10; jdx++) {
+ pending = ARRAY_SIZE(spx5_ram_cfg);
+ for (idx = 0; idx < ARRAY_SIZE(spx5_ram_cfg); idx++) {
+ cfg = &spx5_ram_cfg[idx];
+ if (jdx == 0) {
+ writel(cfg->init_val, cfg->init_reg);
+ } else {
+ value = readl(cfg->init_reg);
+ if ((value & cfg->init_val) != cfg->init_val)
+ pending--;
+ }
+ }
+ if (!pending)
+ break;
+ usleep_range(USEC_PER_MSEC, 2 * USEC_PER_MSEC);
+ }
+
+ if (pending > 0) {
+ /* Still initializing, should be complete in
+ * less than 1ms
+ */
+ dev_err(s5->dev, "Memory initialization error\n");
+ return -EINVAL;
+ }
+ return 0;
+}
+
+static int sparx5_init_switchcore(struct sparx5 *sparx5)
+{
+ u32 value;
+ int err = 0;
+
+ spx5_rmw(EACL_POL_EACL_CFG_EACL_FORCE_INIT_SET(1),
+ EACL_POL_EACL_CFG_EACL_FORCE_INIT,
+ sparx5,
+ EACL_POL_EACL_CFG);
+
+ spx5_rmw(EACL_POL_EACL_CFG_EACL_FORCE_INIT_SET(0),
+ EACL_POL_EACL_CFG_EACL_FORCE_INIT,
+ sparx5,
+ EACL_POL_EACL_CFG);
+
+ /* Initialize memories, if not done already */
+ value = spx5_rd(sparx5, HSCH_RESET_CFG);
+ if (!(value & HSCH_RESET_CFG_CORE_ENA)) {
+ err = sparx5_init_ram(sparx5);
+ if (err)
+ return err;
+ }
+
+ /* Reset counters */
+ spx5_wr(ANA_AC_STAT_RESET_RESET_SET(1), sparx5, ANA_AC_STAT_RESET);
+ spx5_wr(ASM_STAT_CFG_STAT_CNT_CLR_SHOT_SET(1), sparx5, ASM_STAT_CFG);
+
+ /* Enable switch-core and queue system */
+ spx5_wr(HSCH_RESET_CFG_CORE_ENA_SET(1), sparx5, HSCH_RESET_CFG);
+
+ return 0;
+}
+
+static int sparx5_init_coreclock(struct sparx5 *sparx5)
+{
+ enum sparx5_core_clockfreq freq = sparx5->coreclock;
+ u32 clk_div, clk_period, pol_upd_int, idx;
+
+ /* Verify if core clock frequency is supported on target.
+ * If 'VTSS_CORE_CLOCK_DEFAULT' then the highest supported
+ * freq. is used
+ */
+ switch (sparx5->target_ct) {
+ case SPX5_TARGET_CT_7546:
+ if (sparx5->coreclock == SPX5_CORE_CLOCK_DEFAULT)
+ freq = SPX5_CORE_CLOCK_250MHZ;
+ else if (sparx5->coreclock != SPX5_CORE_CLOCK_250MHZ)
+ freq = 0; /* Not supported */
+ break;
+ case SPX5_TARGET_CT_7549:
+ case SPX5_TARGET_CT_7552:
+ case SPX5_TARGET_CT_7556:
+ if (sparx5->coreclock == SPX5_CORE_CLOCK_DEFAULT)
+ freq = SPX5_CORE_CLOCK_500MHZ;
+ else if (sparx5->coreclock != SPX5_CORE_CLOCK_500MHZ)
+ freq = 0; /* Not supported */
+ break;
+ case SPX5_TARGET_CT_7558:
+ case SPX5_TARGET_CT_7558TSN:
+ if (sparx5->coreclock == SPX5_CORE_CLOCK_DEFAULT)
+ freq = SPX5_CORE_CLOCK_625MHZ;
+ else if (sparx5->coreclock != SPX5_CORE_CLOCK_625MHZ)
+ freq = 0; /* Not supported */
+ break;
+ case SPX5_TARGET_CT_7546TSN:
+ if (sparx5->coreclock == SPX5_CORE_CLOCK_DEFAULT)
+ freq = SPX5_CORE_CLOCK_625MHZ;
+ break;
+ case SPX5_TARGET_CT_7549TSN:
+ case SPX5_TARGET_CT_7552TSN:
+ case SPX5_TARGET_CT_7556TSN:
+ if (sparx5->coreclock == SPX5_CORE_CLOCK_DEFAULT)
+ freq = SPX5_CORE_CLOCK_625MHZ;
+ else if (sparx5->coreclock == SPX5_CORE_CLOCK_250MHZ)
+ freq = 0; /* Not supported */
+ break;
+ default:
+ dev_err(sparx5->dev, "Target (%#04x) not supported\n",
+ sparx5->target_ct);
+ return -ENODEV;
+ }
+
+ switch (freq) {
+ case SPX5_CORE_CLOCK_250MHZ:
+ clk_div = 10;
+ pol_upd_int = 312;
+ break;
+ case SPX5_CORE_CLOCK_500MHZ:
+ clk_div = 5;
+ pol_upd_int = 624;
+ break;
+ case SPX5_CORE_CLOCK_625MHZ:
+ clk_div = 4;
+ pol_upd_int = 780;
+ break;
+ default:
+ dev_err(sparx5->dev, "%d coreclock not supported on (%#04x)\n",
+ sparx5->coreclock, sparx5->target_ct);
+ return -EINVAL;
+ }
+
+ /* Update state with chosen frequency */
+ sparx5->coreclock = freq;
+
+ /* Configure the LCPLL */
+ spx5_rmw(CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_CLK_DIV_SET(clk_div) |
+ CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_PRE_DIV_SET(0) |
+ CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_DIR_SET(0) |
+ CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_SEL_SET(0) |
+ CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_ENA_SET(0) |
+ CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_CLK_ENA_SET(1),
+ CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_CLK_DIV |
+ CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_PRE_DIV |
+ CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_DIR |
+ CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_SEL |
+ CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_ENA |
+ CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_CLK_ENA,
+ sparx5,
+ CLKGEN_LCPLL1_CORE_CLK_CFG);
+
+ clk_period = sparx5_clk_period(freq);
+
+ spx5_rmw(HSCH_SYS_CLK_PER_SYS_CLK_PER_100PS_SET(clk_period / 100),
+ HSCH_SYS_CLK_PER_SYS_CLK_PER_100PS,
+ sparx5,
+ HSCH_SYS_CLK_PER);
+
+ spx5_rmw(ANA_AC_POL_BDLB_DLB_CTRL_CLK_PERIOD_01NS_SET(clk_period / 100),
+ ANA_AC_POL_BDLB_DLB_CTRL_CLK_PERIOD_01NS,
+ sparx5,
+ ANA_AC_POL_BDLB_DLB_CTRL);
+
+ spx5_rmw(ANA_AC_POL_SLB_DLB_CTRL_CLK_PERIOD_01NS_SET(clk_period / 100),
+ ANA_AC_POL_SLB_DLB_CTRL_CLK_PERIOD_01NS,
+ sparx5,
+ ANA_AC_POL_SLB_DLB_CTRL);
+
+ spx5_rmw(LRN_AUTOAGE_CFG_1_CLK_PERIOD_01NS_SET(clk_period / 100),
+ LRN_AUTOAGE_CFG_1_CLK_PERIOD_01NS,
+ sparx5,
+ LRN_AUTOAGE_CFG_1);
+
+ for (idx = 0; idx < 3; idx++)
+ spx5_rmw(GCB_SIO_CLOCK_SYS_CLK_PERIOD_SET(clk_period / 100),
+ GCB_SIO_CLOCK_SYS_CLK_PERIOD,
+ sparx5,
+ GCB_SIO_CLOCK(idx));
+
+ spx5_rmw(HSCH_TAS_STATEMACHINE_CFG_REVISIT_DLY_SET
+ ((256 * 1000) / clk_period),
+ HSCH_TAS_STATEMACHINE_CFG_REVISIT_DLY,
+ sparx5,
+ HSCH_TAS_STATEMACHINE_CFG);
+
+ spx5_rmw(ANA_AC_POL_POL_UPD_INT_CFG_POL_UPD_INT_SET(pol_upd_int),
+ ANA_AC_POL_POL_UPD_INT_CFG_POL_UPD_INT,
+ sparx5,
+ ANA_AC_POL_POL_UPD_INT_CFG);
+
+ return 0;
+}
+
+static int sparx5_qlim_set(struct sparx5 *sparx5)
+{
+ u32 res, dp, prio;
+
+ for (res = 0; res < 2; res++) {
+ for (prio = 0; prio < 8; prio++)
+ spx5_wr(0xFFF, sparx5,
+ QRES_RES_CFG(prio + 630 + res * 1024));
+
+ for (dp = 0; dp < 4; dp++)
+ spx5_wr(0xFFF, sparx5,
+ QRES_RES_CFG(dp + 638 + res * 1024));
+ }
+
+ /* Set 80,90,95,100% of memory size for top watermarks */
+ spx5_wr(QLIM_WM(80), sparx5, XQS_QLIMIT_SHR_QLIM_CFG(0));
+ spx5_wr(QLIM_WM(90), sparx5, XQS_QLIMIT_SHR_CTOP_CFG(0));
+ spx5_wr(QLIM_WM(95), sparx5, XQS_QLIMIT_SHR_ATOP_CFG(0));
+ spx5_wr(QLIM_WM(100), sparx5, XQS_QLIMIT_SHR_TOP_CFG(0));
+
+ return 0;
+}
+
+/* Some boards needs to map the SGPIO for signal detect explicitly to the
+ * port module
+ */
+static void sparx5_board_init(struct sparx5 *sparx5)
+{
+ int idx;
+
+ if (!sparx5->sd_sgpio_remapping)
+ return;
+
+ /* Enable SGPIO Signal Detect remapping */
+ spx5_rmw(GCB_HW_SGPIO_SD_CFG_SD_MAP_SEL,
+ GCB_HW_SGPIO_SD_CFG_SD_MAP_SEL,
+ sparx5,
+ GCB_HW_SGPIO_SD_CFG);
+
+ /* Refer to LOS SGPIO */
+ for (idx = 0; idx < SPX5_PORTS; idx++)
+ if (sparx5->ports[idx])
+ if (sparx5->ports[idx]->conf.sd_sgpio != ~0)
+ spx5_wr(sparx5->ports[idx]->conf.sd_sgpio,
+ sparx5,
+ GCB_HW_SGPIO_TO_SD_MAP_CFG(idx));
+}
+
+static int sparx5_start(struct sparx5 *sparx5)
+{
+ u8 broadcast[ETH_ALEN] = {0xff, 0xff, 0xff, 0xff, 0xff, 0xff};
+ char queue_name[32];
+ u32 idx;
+ int err;
+
+ /* Setup own UPSIDs */
+ for (idx = 0; idx < 3; idx++) {
+ spx5_wr(idx, sparx5, ANA_AC_OWN_UPSID(idx));
+ spx5_wr(idx, sparx5, ANA_CL_OWN_UPSID(idx));
+ spx5_wr(idx, sparx5, ANA_L2_OWN_UPSID(idx));
+ spx5_wr(idx, sparx5, REW_OWN_UPSID(idx));
+ }
+
+ /* Enable CPU ports */
+ for (idx = SPX5_PORTS; idx < SPX5_PORTS_ALL; idx++)
+ spx5_rmw(QFWD_SWITCH_PORT_MODE_PORT_ENA_SET(1),
+ QFWD_SWITCH_PORT_MODE_PORT_ENA,
+ sparx5,
+ QFWD_SWITCH_PORT_MODE(idx));
+
+ /* Init masks */
+ sparx5_update_fwd(sparx5);
+
+ /* CPU copy CPU pgids */
+ spx5_wr(ANA_AC_PGID_MISC_CFG_PGID_CPU_COPY_ENA_SET(1),
+ sparx5, ANA_AC_PGID_MISC_CFG(PGID_CPU));
+ spx5_wr(ANA_AC_PGID_MISC_CFG_PGID_CPU_COPY_ENA_SET(1),
+ sparx5, ANA_AC_PGID_MISC_CFG(PGID_BCAST));
+
+ /* Recalc injected frame FCS */
+ for (idx = SPX5_PORT_CPU_0; idx <= SPX5_PORT_CPU_1; idx++)
+ spx5_rmw(ANA_CL_FILTER_CTRL_FORCE_FCS_UPDATE_ENA_SET(1),
+ ANA_CL_FILTER_CTRL_FORCE_FCS_UPDATE_ENA,
+ sparx5, ANA_CL_FILTER_CTRL(idx));
+
+ /* Init MAC table, ageing */
+ sparx5_mact_init(sparx5);
+
+ /* Setup VLANs */
+ sparx5_vlan_init(sparx5);
+
+ /* Add host mode BC address (points only to CPU) */
+ sparx5_mact_learn(sparx5, PGID_CPU, broadcast, NULL_VID);
+
+ /* Enable queue limitation watermarks */
+ sparx5_qlim_set(sparx5);
+
+ err = sparx5_config_auto_calendar(sparx5);
+ if (err)
+ return err;
+
+ err = sparx5_config_dsm_calendar(sparx5);
+ if (err)
+ return err;
+
+ /* Init stats */
+ err = sparx_stats_init(sparx5);
+ if (err)
+ return err;
+
+ /* Init mact_sw struct */
+ mutex_init(&sparx5->mact_lock);
+ INIT_LIST_HEAD(&sparx5->mact_entries);
+ snprintf(queue_name, sizeof(queue_name), "%s-mact",
+ dev_name(sparx5->dev));
+ sparx5->mact_queue = create_singlethread_workqueue(queue_name);
+ INIT_DELAYED_WORK(&sparx5->mact_work, sparx5_mact_pull_work);
+ queue_delayed_work(sparx5->mact_queue, &sparx5->mact_work,
+ SPX5_MACT_PULL_DELAY);
+
+ err = sparx5_register_netdevs(sparx5);
+ if (err)
+ return err;
+
+ sparx5_board_init(sparx5);
+ err = sparx5_register_notifier_blocks(sparx5);
+
+ /* Start register based INJ/XTR */
+ err = -ENXIO;
+ if (err && sparx5->xtr_irq >= 0) {
+ err = devm_request_irq(sparx5->dev, sparx5->xtr_irq,
+ sparx5_xtr_handler, IRQF_SHARED,
+ "sparx5-xtr", sparx5);
+ if (!err)
+ err = sparx5_manual_injection_mode(sparx5);
+ if (err)
+ sparx5->xtr_irq = -ENXIO;
+ } else {
+ sparx5->xtr_irq = -ENXIO;
+ }
+ return err;
+}
+
+static void sparx5_cleanup_ports(struct sparx5 *sparx5)
+{
+ sparx5_unregister_netdevs(sparx5);
+ sparx5_destroy_netdevs(sparx5);
+}
+
+static int mchp_sparx5_probe(struct platform_device *pdev)
+{
+ struct initial_port_config *configs, *config;
+ struct device_node *np = pdev->dev.of_node;
+ struct device_node *ports, *portnp;
+ struct reset_control *reset;
+ struct sparx5 *sparx5;
+ int idx = 0, err = 0;
+ u8 *mac_addr;
+
+ if (!np && !pdev->dev.platform_data)
+ return -ENODEV;
+
+ sparx5 = devm_kzalloc(&pdev->dev, sizeof(*sparx5), GFP_KERNEL);
+ if (!sparx5)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, sparx5);
+ sparx5->pdev = pdev;
+ sparx5->dev = &pdev->dev;
+
+ /* Do switch core reset if available */
+ reset = devm_reset_control_get_optional_shared(&pdev->dev, "switch");
+ if (IS_ERR(reset))
+ return dev_err_probe(&pdev->dev, PTR_ERR(reset),
+ "Failed to get switch reset controller.\n");
+ reset_control_reset(reset);
+
+ /* Default values, some from DT */
+ sparx5->coreclock = SPX5_CORE_CLOCK_DEFAULT;
+
+ ports = of_get_child_by_name(np, "ethernet-ports");
+ if (!ports) {
+ dev_err(sparx5->dev, "no ethernet-ports child node found\n");
+ return -ENODEV;
+ }
+ sparx5->port_count = of_get_child_count(ports);
+
+ configs = kcalloc(sparx5->port_count,
+ sizeof(struct initial_port_config), GFP_KERNEL);
+ if (!configs) {
+ err = -ENOMEM;
+ goto cleanup_pnode;
+ }
+
+ for_each_available_child_of_node(ports, portnp) {
+ struct sparx5_port_config *conf;
+ struct phy *serdes;
+ u32 portno;
+
+ err = of_property_read_u32(portnp, "reg", &portno);
+ if (err) {
+ dev_err(sparx5->dev, "port reg property error\n");
+ continue;
+ }
+ config = &configs[idx];
+ conf = &config->conf;
+ conf->speed = SPEED_UNKNOWN;
+ conf->bandwidth = SPEED_UNKNOWN;
+ err = of_get_phy_mode(portnp, &conf->phy_mode);
+ if (err) {
+ dev_err(sparx5->dev, "port %u: missing phy-mode\n",
+ portno);
+ continue;
+ }
+ err = of_property_read_u32(portnp, "microchip,bandwidth",
+ &conf->bandwidth);
+ if (err) {
+ dev_err(sparx5->dev, "port %u: missing bandwidth\n",
+ portno);
+ continue;
+ }
+ err = of_property_read_u32(portnp, "microchip,sd-sgpio", &conf->sd_sgpio);
+ if (err)
+ conf->sd_sgpio = ~0;
+ else
+ sparx5->sd_sgpio_remapping = true;
+ serdes = devm_of_phy_get(sparx5->dev, portnp, NULL);
+ if (IS_ERR(serdes)) {
+ err = dev_err_probe(sparx5->dev, PTR_ERR(serdes),
+ "port %u: missing serdes\n",
+ portno);
+ goto cleanup_config;
+ }
+ config->portno = portno;
+ config->node = portnp;
+ config->serdes = serdes;
+
+ conf->media = PHY_MEDIA_DAC;
+ conf->serdes_reset = true;
+ conf->portmode = conf->phy_mode;
+ conf->power_down = true;
+ idx++;
+ }
+
+ err = sparx5_create_targets(sparx5);
+ if (err)
+ goto cleanup_config;
+
+ if (of_get_mac_address(np, mac_addr)) {
+ dev_info(sparx5->dev, "MAC addr was not set, use random MAC\n");
+ eth_random_addr(sparx5->base_mac);
+ sparx5->base_mac[5] = 0;
+ } else {
+ ether_addr_copy(sparx5->base_mac, mac_addr);
+ }
+
+ sparx5->xtr_irq = platform_get_irq_byname(sparx5->pdev, "xtr");
+
+ /* Read chip ID to check CPU interface */
+ sparx5->chip_id = spx5_rd(sparx5, GCB_CHIP_ID);
+
+ sparx5->target_ct = (enum spx5_target_chiptype)
+ GCB_CHIP_ID_PART_ID_GET(sparx5->chip_id);
+
+ /* Initialize Switchcore and internal RAMs */
+ err = sparx5_init_switchcore(sparx5);
+ if (err) {
+ dev_err(sparx5->dev, "Switchcore initialization error\n");
+ goto cleanup_config;
+ }
+
+ /* Initialize the LC-PLL (core clock) and set affected registers */
+ err = sparx5_init_coreclock(sparx5);
+ if (err) {
+ dev_err(sparx5->dev, "LC-PLL initialization error\n");
+ goto cleanup_config;
+ }
+
+ for (idx = 0; idx < sparx5->port_count; ++idx) {
+ config = &configs[idx];
+ if (!config->node)
+ continue;
+
+ err = sparx5_create_port(sparx5, config);
+ if (err) {
+ dev_err(sparx5->dev, "port create error\n");
+ goto cleanup_ports;
+ }
+ }
+
+ err = sparx5_start(sparx5);
+ if (err) {
+ dev_err(sparx5->dev, "Start failed\n");
+ goto cleanup_ports;
+ }
+ goto cleanup_config;
+
+cleanup_ports:
+ sparx5_cleanup_ports(sparx5);
+cleanup_config:
+ kfree(configs);
+cleanup_pnode:
+ of_node_put(ports);
+ return err;
+}
+
+static int mchp_sparx5_remove(struct platform_device *pdev)
+{
+ struct sparx5 *sparx5 = platform_get_drvdata(pdev);
+
+ if (sparx5->xtr_irq) {
+ disable_irq(sparx5->xtr_irq);
+ sparx5->xtr_irq = -ENXIO;
+ }
+ sparx5_cleanup_ports(sparx5);
+ /* Unregister netdevs */
+ sparx5_unregister_notifier_blocks(sparx5);
+
+ return 0;
+}
+
+static const struct of_device_id mchp_sparx5_match[] = {
+ { .compatible = "microchip,sparx5-switch" },
+ { }
+};
+MODULE_DEVICE_TABLE(of, mchp_sparx5_match);
+
+static struct platform_driver mchp_sparx5_driver = {
+ .probe = mchp_sparx5_probe,
+ .remove = mchp_sparx5_remove,
+ .driver = {
+ .name = "sparx5-switch",
+ .of_match_table = mchp_sparx5_match,
+ },
+};
+
+module_platform_driver(mchp_sparx5_driver);
+
+MODULE_DESCRIPTION("Microchip Sparx5 switch driver");
+MODULE_AUTHOR("Steen Hegelund <steen.hegelund@microchip.com>");
+MODULE_LICENSE("Dual MIT/GPL");
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0+ */
+/* Microchip Sparx5 Switch driver
+ *
+ * Copyright (c) 2021 Microchip Technology Inc. and its subsidiaries.
+ */
+
+#ifndef __SPARX5_MAIN_H__
+#define __SPARX5_MAIN_H__
+
+#include <linux/types.h>
+#include <linux/phy/phy.h>
+#include <linux/netdevice.h>
+#include <linux/phy.h>
+#include <linux/if_vlan.h>
+#include <linux/bitmap.h>
+#include <linux/phylink.h>
+#include <linux/hrtimer.h>
+
+/* Target chip type */
+enum spx5_target_chiptype {
+ SPX5_TARGET_CT_7546 = 0x7546, /* SparX-5-64 Enterprise */
+ SPX5_TARGET_CT_7549 = 0x7549, /* SparX-5-90 Enterprise */
+ SPX5_TARGET_CT_7552 = 0x7552, /* SparX-5-128 Enterprise */
+ SPX5_TARGET_CT_7556 = 0x7556, /* SparX-5-160 Enterprise */
+ SPX5_TARGET_CT_7558 = 0x7558, /* SparX-5-200 Enterprise */
+ SPX5_TARGET_CT_7546TSN = 0x47546, /* SparX-5-64i Industrial */
+ SPX5_TARGET_CT_7549TSN = 0x47549, /* SparX-5-90i Industrial */
+ SPX5_TARGET_CT_7552TSN = 0x47552, /* SparX-5-128i Industrial */
+ SPX5_TARGET_CT_7556TSN = 0x47556, /* SparX-5-160i Industrial */
+ SPX5_TARGET_CT_7558TSN = 0x47558, /* SparX-5-200i Industrial */
+};
+
+enum sparx5_port_max_tags {
+ SPX5_PORT_MAX_TAGS_NONE, /* No extra tags allowed */
+ SPX5_PORT_MAX_TAGS_ONE, /* Single tag allowed */
+ SPX5_PORT_MAX_TAGS_TWO /* Single and double tag allowed */
+};
+
+enum sparx5_vlan_port_type {
+ SPX5_VLAN_PORT_TYPE_UNAWARE, /* VLAN unaware port */
+ SPX5_VLAN_PORT_TYPE_C, /* C-port */
+ SPX5_VLAN_PORT_TYPE_S, /* S-port */
+ SPX5_VLAN_PORT_TYPE_S_CUSTOM /* S-port using custom type */
+};
+
+#define SPX5_PORTS 65
+#define SPX5_PORT_CPU (SPX5_PORTS) /* Next port is CPU port */
+#define SPX5_PORT_CPU_0 (SPX5_PORT_CPU + 0) /* CPU Port 65 */
+#define SPX5_PORT_CPU_1 (SPX5_PORT_CPU + 1) /* CPU Port 66 */
+#define SPX5_PORT_VD0 (SPX5_PORT_CPU + 2) /* VD0/Port 67 used for IPMC */
+#define SPX5_PORT_VD1 (SPX5_PORT_CPU + 3) /* VD1/Port 68 used for AFI/OAM */
+#define SPX5_PORT_VD2 (SPX5_PORT_CPU + 4) /* VD2/Port 69 used for IPinIP*/
+#define SPX5_PORTS_ALL (SPX5_PORT_CPU + 5) /* Total number of ports */
+
+#define PGID_BASE SPX5_PORTS /* Starts after port PGIDs */
+#define PGID_UC_FLOOD (PGID_BASE + 0)
+#define PGID_MC_FLOOD (PGID_BASE + 1)
+#define PGID_IPV4_MC_DATA (PGID_BASE + 2)
+#define PGID_IPV4_MC_CTRL (PGID_BASE + 3)
+#define PGID_IPV6_MC_DATA (PGID_BASE + 4)
+#define PGID_IPV6_MC_CTRL (PGID_BASE + 5)
+#define PGID_BCAST (PGID_BASE + 6)
+#define PGID_CPU (PGID_BASE + 7)
+
+#define IFH_LEN 9 /* 36 bytes */
+#define NULL_VID 0
+#define SPX5_MACT_PULL_DELAY (2 * HZ)
+#define SPX5_STATS_CHECK_DELAY (1 * HZ)
+#define SPX5_PRIOS 8 /* Number of priority queues */
+#define SPX5_BUFFER_CELL_SZ 184 /* Cell size */
+#define SPX5_BUFFER_MEMORY 4194280 /* 22795 words * 184 bytes */
+
+#define XTR_QUEUE 0
+#define INJ_QUEUE 0
+
+struct sparx5;
+
+struct sparx5_port_config {
+ phy_interface_t portmode;
+ u32 bandwidth;
+ int speed;
+ int duplex;
+ enum phy_media media;
+ bool inband;
+ bool power_down;
+ bool autoneg;
+ bool serdes_reset;
+ u32 pause;
+ u32 pause_adv;
+ phy_interface_t phy_mode;
+ u32 sd_sgpio;
+};
+
+struct sparx5_port {
+ struct net_device *ndev;
+ struct sparx5 *sparx5;
+ struct device_node *of_node;
+ struct phy *serdes;
+ struct sparx5_port_config conf;
+ struct phylink_config phylink_config;
+ struct phylink *phylink;
+ struct phylink_pcs phylink_pcs;
+ u16 portno;
+ /* Ingress default VLAN (pvid) */
+ u16 pvid;
+ /* Egress default VLAN (vid) */
+ u16 vid;
+ bool signd_internal;
+ bool signd_active_high;
+ bool signd_enable;
+ bool flow_control;
+ enum sparx5_port_max_tags max_vlan_tags;
+ enum sparx5_vlan_port_type vlan_type;
+ u32 custom_etype;
+ u32 ifh[IFH_LEN];
+ bool vlan_aware;
+ struct hrtimer inj_timer;
+};
+
+enum sparx5_core_clockfreq {
+ SPX5_CORE_CLOCK_DEFAULT, /* Defaults to the highest supported frequency */
+ SPX5_CORE_CLOCK_250MHZ, /* 250MHZ core clock frequency */
+ SPX5_CORE_CLOCK_500MHZ, /* 500MHZ core clock frequency */
+ SPX5_CORE_CLOCK_625MHZ, /* 625MHZ core clock frequency */
+};
+
+struct sparx5 {
+ struct platform_device *pdev;
+ struct device *dev;
+ u32 chip_id;
+ enum spx5_target_chiptype target_ct;
+ void __iomem *regs[NUM_TARGETS];
+ int port_count;
+ struct mutex lock; /* MAC reg lock */
+ /* port structures are in net device */
+ struct sparx5_port *ports[SPX5_PORTS];
+ enum sparx5_core_clockfreq coreclock;
+ /* Statistics */
+ u32 num_stats;
+ u32 num_ethtool_stats;
+ const char * const *stats_layout;
+ u64 *stats;
+ /* Workqueue for reading stats */
+ struct mutex queue_stats_lock;
+ struct delayed_work stats_work;
+ struct workqueue_struct *stats_queue;
+ /* Notifiers */
+ struct notifier_block netdevice_nb;
+ struct notifier_block switchdev_nb;
+ struct notifier_block switchdev_blocking_nb;
+ /* Switch state */
+ u8 base_mac[ETH_ALEN];
+ /* Associated bridge device (when bridged) */
+ struct net_device *hw_bridge_dev;
+ /* Bridged interfaces */
+ DECLARE_BITMAP(bridge_mask, SPX5_PORTS);
+ DECLARE_BITMAP(bridge_fwd_mask, SPX5_PORTS);
+ DECLARE_BITMAP(bridge_lrn_mask, SPX5_PORTS);
+ DECLARE_BITMAP(vlan_mask[VLAN_N_VID], SPX5_PORTS);
+ /* SW MAC table */
+ struct list_head mact_entries;
+ /* mac table list (mact_entries) mutex */
+ struct mutex mact_lock;
+ struct delayed_work mact_work;
+ struct workqueue_struct *mact_queue;
+ /* Board specifics */
+ bool sd_sgpio_remapping;
+ /* Register based inj/xtr */
+ int xtr_irq;
+};
+
+/* sparx5_switchdev.c */
+int sparx5_register_notifier_blocks(struct sparx5 *sparx5);
+void sparx5_unregister_notifier_blocks(struct sparx5 *sparx5);
+
+/* sparx5_packet.c */
+irqreturn_t sparx5_xtr_handler(int irq, void *_priv);
+int sparx5_port_xmit_impl(struct sk_buff *skb, struct net_device *dev);
+int sparx5_manual_injection_mode(struct sparx5 *sparx5);
+void sparx5_port_inj_timer_setup(struct sparx5_port *port);
+
+/* sparx5_mactable.c */
+void sparx5_mact_pull_work(struct work_struct *work);
+int sparx5_mact_learn(struct sparx5 *sparx5, int port,
+ const unsigned char mac[ETH_ALEN], u16 vid);
+bool sparx5_mact_getnext(struct sparx5 *sparx5,
+ unsigned char mac[ETH_ALEN], u16 *vid, u32 *pcfg2);
+int sparx5_mact_forget(struct sparx5 *sparx5,
+ const unsigned char mac[ETH_ALEN], u16 vid);
+int sparx5_add_mact_entry(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ const unsigned char *addr, u16 vid);
+int sparx5_del_mact_entry(struct sparx5 *sparx5,
+ const unsigned char *addr,
+ u16 vid);
+int sparx5_mc_sync(struct net_device *dev, const unsigned char *addr);
+int sparx5_mc_unsync(struct net_device *dev, const unsigned char *addr);
+void sparx5_set_ageing(struct sparx5 *sparx5, int msecs);
+void sparx5_mact_init(struct sparx5 *sparx5);
+
+/* sparx5_vlan.c */
+void sparx5_pgid_update_mask(struct sparx5_port *port, int pgid, bool enable);
+void sparx5_update_fwd(struct sparx5 *sparx5);
+void sparx5_vlan_init(struct sparx5 *sparx5);
+void sparx5_vlan_port_setup(struct sparx5 *sparx5, int portno);
+int sparx5_vlan_vid_add(struct sparx5_port *port, u16 vid, bool pvid,
+ bool untagged);
+int sparx5_vlan_vid_del(struct sparx5_port *port, u16 vid);
+void sparx5_vlan_port_apply(struct sparx5 *sparx5, struct sparx5_port *port);
+
+/* sparx5_calendar.c */
+int sparx5_config_auto_calendar(struct sparx5 *sparx5);
+int sparx5_config_dsm_calendar(struct sparx5 *sparx5);
+
+/* sparx5_ethtool.c */
+void sparx5_get_stats64(struct net_device *ndev, struct rtnl_link_stats64 *stats);
+int sparx_stats_init(struct sparx5 *sparx5);
+
+/* sparx5_netdev.c */
+bool sparx5_netdevice_check(const struct net_device *dev);
+struct net_device *sparx5_create_netdev(struct sparx5 *sparx5, u32 portno);
+int sparx5_register_netdevs(struct sparx5 *sparx5);
+void sparx5_destroy_netdevs(struct sparx5 *sparx5);
+void sparx5_unregister_netdevs(struct sparx5 *sparx5);
+
+/* Clock period in picoseconds */
+static inline u32 sparx5_clk_period(enum sparx5_core_clockfreq cclock)
+{
+ switch (cclock) {
+ case SPX5_CORE_CLOCK_250MHZ:
+ return 4000;
+ case SPX5_CORE_CLOCK_500MHZ:
+ return 2000;
+ case SPX5_CORE_CLOCK_625MHZ:
+ default:
+ return 1600;
+ }
+}
+
+static inline bool sparx5_is_baser(phy_interface_t interface)
+{
+ return interface == PHY_INTERFACE_MODE_5GBASER ||
+ interface == PHY_INTERFACE_MODE_10GBASER ||
+ interface == PHY_INTERFACE_MODE_25GBASER;
+}
+
+extern const struct phylink_mac_ops sparx5_phylink_mac_ops;
+extern const struct phylink_pcs_ops sparx5_phylink_pcs_ops;
+extern const struct ethtool_ops sparx5_ethtool_ops;
+
+/* Calculate raw offset */
+static inline __pure int spx5_offset(int id, int tinst, int tcnt,
+ int gbase, int ginst,
+ int gcnt, int gwidth,
+ int raddr, int rinst,
+ int rcnt, int rwidth)
+{
+ WARN_ON((tinst) >= tcnt);
+ WARN_ON((ginst) >= gcnt);
+ WARN_ON((rinst) >= rcnt);
+ return gbase + ((ginst) * gwidth) +
+ raddr + ((rinst) * rwidth);
+}
+
+/* Read, Write and modify registers content.
+ * The register definition macros start at the id
+ */
+static inline void __iomem *spx5_addr(void __iomem *base[],
+ int id, int tinst, int tcnt,
+ int gbase, int ginst,
+ int gcnt, int gwidth,
+ int raddr, int rinst,
+ int rcnt, int rwidth)
+{
+ WARN_ON((tinst) >= tcnt);
+ WARN_ON((ginst) >= gcnt);
+ WARN_ON((rinst) >= rcnt);
+ return base[id + (tinst)] +
+ gbase + ((ginst) * gwidth) +
+ raddr + ((rinst) * rwidth);
+}
+
+static inline void __iomem *spx5_inst_addr(void __iomem *base,
+ int gbase, int ginst,
+ int gcnt, int gwidth,
+ int raddr, int rinst,
+ int rcnt, int rwidth)
+{
+ WARN_ON((ginst) >= gcnt);
+ WARN_ON((rinst) >= rcnt);
+ return base +
+ gbase + ((ginst) * gwidth) +
+ raddr + ((rinst) * rwidth);
+}
+
+static inline u32 spx5_rd(struct sparx5 *sparx5, int id, int tinst, int tcnt,
+ int gbase, int ginst, int gcnt, int gwidth,
+ int raddr, int rinst, int rcnt, int rwidth)
+{
+ return readl(spx5_addr(sparx5->regs, id, tinst, tcnt, gbase, ginst,
+ gcnt, gwidth, raddr, rinst, rcnt, rwidth));
+}
+
+static inline u32 spx5_inst_rd(void __iomem *iomem, int id, int tinst, int tcnt,
+ int gbase, int ginst, int gcnt, int gwidth,
+ int raddr, int rinst, int rcnt, int rwidth)
+{
+ return readl(spx5_inst_addr(iomem, gbase, ginst,
+ gcnt, gwidth, raddr, rinst, rcnt, rwidth));
+}
+
+static inline void spx5_wr(u32 val, struct sparx5 *sparx5,
+ int id, int tinst, int tcnt,
+ int gbase, int ginst, int gcnt, int gwidth,
+ int raddr, int rinst, int rcnt, int rwidth)
+{
+ writel(val, spx5_addr(sparx5->regs, id, tinst, tcnt,
+ gbase, ginst, gcnt, gwidth,
+ raddr, rinst, rcnt, rwidth));
+}
+
+static inline void spx5_inst_wr(u32 val, void __iomem *iomem,
+ int id, int tinst, int tcnt,
+ int gbase, int ginst, int gcnt, int gwidth,
+ int raddr, int rinst, int rcnt, int rwidth)
+{
+ writel(val, spx5_inst_addr(iomem,
+ gbase, ginst, gcnt, gwidth,
+ raddr, rinst, rcnt, rwidth));
+}
+
+static inline void spx5_rmw(u32 val, u32 mask, struct sparx5 *sparx5,
+ int id, int tinst, int tcnt,
+ int gbase, int ginst, int gcnt, int gwidth,
+ int raddr, int rinst, int rcnt, int rwidth)
+{
+ u32 nval;
+
+ nval = readl(spx5_addr(sparx5->regs, id, tinst, tcnt, gbase, ginst,
+ gcnt, gwidth, raddr, rinst, rcnt, rwidth));
+ nval = (nval & ~mask) | (val & mask);
+ writel(nval, spx5_addr(sparx5->regs, id, tinst, tcnt, gbase, ginst,
+ gcnt, gwidth, raddr, rinst, rcnt, rwidth));
+}
+
+static inline void spx5_inst_rmw(u32 val, u32 mask, void __iomem *iomem,
+ int id, int tinst, int tcnt,
+ int gbase, int ginst, int gcnt, int gwidth,
+ int raddr, int rinst, int rcnt, int rwidth)
+{
+ u32 nval;
+
+ nval = readl(spx5_inst_addr(iomem, gbase, ginst, gcnt, gwidth, raddr,
+ rinst, rcnt, rwidth));
+ nval = (nval & ~mask) | (val & mask);
+ writel(nval, spx5_inst_addr(iomem, gbase, ginst, gcnt, gwidth, raddr,
+ rinst, rcnt, rwidth));
+}
+
+static inline void __iomem *spx5_inst_get(struct sparx5 *sparx5, int id, int tinst)
+{
+ return sparx5->regs[id + tinst];
+}
+
+static inline void __iomem *spx5_reg_get(struct sparx5 *sparx5,
+ int id, int tinst, int tcnt,
+ int gbase, int ginst, int gcnt, int gwidth,
+ int raddr, int rinst, int rcnt, int rwidth)
+{
+ return spx5_addr(sparx5->regs, id, tinst, tcnt,
+ gbase, ginst, gcnt, gwidth,
+ raddr, rinst, rcnt, rwidth);
+}
+
+#endif /* __SPARX5_MAIN_H__ */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0+
+ * Microchip Sparx5 Switch driver
+ *
+ * Copyright (c) 2021 Microchip Technology Inc.
+ */
+
+/* This file is autogenerated by cml-utils 2021-05-06 13:06:37 +0200.
+ * Commit ID: 9ae4ec441e25e4b9003f4e514df5cb12a36b84d3
+ */
+
+#ifndef _SPARX5_MAIN_REGS_H_
+#define _SPARX5_MAIN_REGS_H_
+
+#include <linux/bitfield.h>
+#include <linux/types.h>
+#include <linux/bug.h>
+
+enum sparx5_target {
+ TARGET_ANA_AC = 1,
+ TARGET_ANA_ACL = 2,
+ TARGET_ANA_AC_POL = 4,
+ TARGET_ANA_CL = 6,
+ TARGET_ANA_L2 = 7,
+ TARGET_ANA_L3 = 8,
+ TARGET_ASM = 9,
+ TARGET_CLKGEN = 11,
+ TARGET_CPU = 12,
+ TARGET_DEV10G = 17,
+ TARGET_DEV25G = 29,
+ TARGET_DEV2G5 = 37,
+ TARGET_DEV5G = 102,
+ TARGET_DSM = 115,
+ TARGET_EACL = 116,
+ TARGET_FDMA = 117,
+ TARGET_GCB = 118,
+ TARGET_HSCH = 119,
+ TARGET_LRN = 122,
+ TARGET_PCEP = 129,
+ TARGET_PCS10G_BR = 132,
+ TARGET_PCS25G_BR = 144,
+ TARGET_PCS5G_BR = 160,
+ TARGET_PORT_CONF = 173,
+ TARGET_QFWD = 175,
+ TARGET_QRES = 176,
+ TARGET_QS = 177,
+ TARGET_QSYS = 178,
+ TARGET_REW = 179,
+ TARGET_VCAP_SUPER = 326,
+ TARGET_VOP = 327,
+ TARGET_XQS = 331,
+ NUM_TARGETS = 332
+};
+
+#define __REG(...) __VA_ARGS__
+
+/* ANA_AC:RAM_CTRL:RAM_INIT */
+#define ANA_AC_RAM_INIT __REG(TARGET_ANA_AC, 0, 1, 839108, 0, 1, 4, 0, 0, 1, 4)
+
+#define ANA_AC_RAM_INIT_RAM_INIT BIT(1)
+#define ANA_AC_RAM_INIT_RAM_INIT_SET(x)\
+ FIELD_PREP(ANA_AC_RAM_INIT_RAM_INIT, x)
+#define ANA_AC_RAM_INIT_RAM_INIT_GET(x)\
+ FIELD_GET(ANA_AC_RAM_INIT_RAM_INIT, x)
+
+#define ANA_AC_RAM_INIT_RAM_CFG_HOOK BIT(0)
+#define ANA_AC_RAM_INIT_RAM_CFG_HOOK_SET(x)\
+ FIELD_PREP(ANA_AC_RAM_INIT_RAM_CFG_HOOK, x)
+#define ANA_AC_RAM_INIT_RAM_CFG_HOOK_GET(x)\
+ FIELD_GET(ANA_AC_RAM_INIT_RAM_CFG_HOOK, x)
+
+/* ANA_AC:PS_COMMON:OWN_UPSID */
+#define ANA_AC_OWN_UPSID(r) __REG(TARGET_ANA_AC, 0, 1, 894472, 0, 1, 352, 52, r, 3, 4)
+
+#define ANA_AC_OWN_UPSID_OWN_UPSID GENMASK(4, 0)
+#define ANA_AC_OWN_UPSID_OWN_UPSID_SET(x)\
+ FIELD_PREP(ANA_AC_OWN_UPSID_OWN_UPSID, x)
+#define ANA_AC_OWN_UPSID_OWN_UPSID_GET(x)\
+ FIELD_GET(ANA_AC_OWN_UPSID_OWN_UPSID, x)
+
+/* ANA_AC:SRC:SRC_CFG */
+#define ANA_AC_SRC_CFG(g) __REG(TARGET_ANA_AC, 0, 1, 849920, g, 102, 16, 0, 0, 1, 4)
+
+/* ANA_AC:SRC:SRC_CFG1 */
+#define ANA_AC_SRC_CFG1(g) __REG(TARGET_ANA_AC, 0, 1, 849920, g, 102, 16, 4, 0, 1, 4)
+
+/* ANA_AC:SRC:SRC_CFG2 */
+#define ANA_AC_SRC_CFG2(g) __REG(TARGET_ANA_AC, 0, 1, 849920, g, 102, 16, 8, 0, 1, 4)
+
+#define ANA_AC_SRC_CFG2_PORT_MASK2 BIT(0)
+#define ANA_AC_SRC_CFG2_PORT_MASK2_SET(x)\
+ FIELD_PREP(ANA_AC_SRC_CFG2_PORT_MASK2, x)
+#define ANA_AC_SRC_CFG2_PORT_MASK2_GET(x)\
+ FIELD_GET(ANA_AC_SRC_CFG2_PORT_MASK2, x)
+
+/* ANA_AC:PGID:PGID_CFG */
+#define ANA_AC_PGID_CFG(g) __REG(TARGET_ANA_AC, 0, 1, 786432, g, 3290, 16, 0, 0, 1, 4)
+
+/* ANA_AC:PGID:PGID_CFG1 */
+#define ANA_AC_PGID_CFG1(g) __REG(TARGET_ANA_AC, 0, 1, 786432, g, 3290, 16, 4, 0, 1, 4)
+
+/* ANA_AC:PGID:PGID_CFG2 */
+#define ANA_AC_PGID_CFG2(g) __REG(TARGET_ANA_AC, 0, 1, 786432, g, 3290, 16, 8, 0, 1, 4)
+
+#define ANA_AC_PGID_CFG2_PORT_MASK2 BIT(0)
+#define ANA_AC_PGID_CFG2_PORT_MASK2_SET(x)\
+ FIELD_PREP(ANA_AC_PGID_CFG2_PORT_MASK2, x)
+#define ANA_AC_PGID_CFG2_PORT_MASK2_GET(x)\
+ FIELD_GET(ANA_AC_PGID_CFG2_PORT_MASK2, x)
+
+/* ANA_AC:PGID:PGID_MISC_CFG */
+#define ANA_AC_PGID_MISC_CFG(g) __REG(TARGET_ANA_AC, 0, 1, 786432, g, 3290, 16, 12, 0, 1, 4)
+
+#define ANA_AC_PGID_MISC_CFG_PGID_CPU_QU GENMASK(6, 4)
+#define ANA_AC_PGID_MISC_CFG_PGID_CPU_QU_SET(x)\
+ FIELD_PREP(ANA_AC_PGID_MISC_CFG_PGID_CPU_QU, x)
+#define ANA_AC_PGID_MISC_CFG_PGID_CPU_QU_GET(x)\
+ FIELD_GET(ANA_AC_PGID_MISC_CFG_PGID_CPU_QU, x)
+
+#define ANA_AC_PGID_MISC_CFG_STACK_TYPE_ENA BIT(1)
+#define ANA_AC_PGID_MISC_CFG_STACK_TYPE_ENA_SET(x)\
+ FIELD_PREP(ANA_AC_PGID_MISC_CFG_STACK_TYPE_ENA, x)
+#define ANA_AC_PGID_MISC_CFG_STACK_TYPE_ENA_GET(x)\
+ FIELD_GET(ANA_AC_PGID_MISC_CFG_STACK_TYPE_ENA, x)
+
+#define ANA_AC_PGID_MISC_CFG_PGID_CPU_COPY_ENA BIT(0)
+#define ANA_AC_PGID_MISC_CFG_PGID_CPU_COPY_ENA_SET(x)\
+ FIELD_PREP(ANA_AC_PGID_MISC_CFG_PGID_CPU_COPY_ENA, x)
+#define ANA_AC_PGID_MISC_CFG_PGID_CPU_COPY_ENA_GET(x)\
+ FIELD_GET(ANA_AC_PGID_MISC_CFG_PGID_CPU_COPY_ENA, x)
+
+/* ANA_AC:STAT_GLOBAL_CFG_PORT:STAT_GLOBAL_EVENT_MASK */
+#define ANA_AC_PORT_SGE_CFG(r) __REG(TARGET_ANA_AC, 0, 1, 851552, 0, 1, 20, 0, r, 4, 4)
+
+#define ANA_AC_PORT_SGE_CFG_MASK GENMASK(15, 0)
+#define ANA_AC_PORT_SGE_CFG_MASK_SET(x)\
+ FIELD_PREP(ANA_AC_PORT_SGE_CFG_MASK, x)
+#define ANA_AC_PORT_SGE_CFG_MASK_GET(x)\
+ FIELD_GET(ANA_AC_PORT_SGE_CFG_MASK, x)
+
+/* ANA_AC:STAT_GLOBAL_CFG_PORT:STAT_RESET */
+#define ANA_AC_STAT_RESET __REG(TARGET_ANA_AC, 0, 1, 851552, 0, 1, 20, 16, 0, 1, 4)
+
+#define ANA_AC_STAT_RESET_RESET BIT(0)
+#define ANA_AC_STAT_RESET_RESET_SET(x)\
+ FIELD_PREP(ANA_AC_STAT_RESET_RESET, x)
+#define ANA_AC_STAT_RESET_RESET_GET(x)\
+ FIELD_GET(ANA_AC_STAT_RESET_RESET, x)
+
+/* ANA_AC:STAT_CNT_CFG_PORT:STAT_CFG */
+#define ANA_AC_PORT_STAT_CFG(g, r) __REG(TARGET_ANA_AC, 0, 1, 843776, g, 70, 64, 4, r, 4, 4)
+
+#define ANA_AC_PORT_STAT_CFG_CFG_PRIO_MASK GENMASK(11, 4)
+#define ANA_AC_PORT_STAT_CFG_CFG_PRIO_MASK_SET(x)\
+ FIELD_PREP(ANA_AC_PORT_STAT_CFG_CFG_PRIO_MASK, x)
+#define ANA_AC_PORT_STAT_CFG_CFG_PRIO_MASK_GET(x)\
+ FIELD_GET(ANA_AC_PORT_STAT_CFG_CFG_PRIO_MASK, x)
+
+#define ANA_AC_PORT_STAT_CFG_CFG_CNT_FRM_TYPE GENMASK(3, 1)
+#define ANA_AC_PORT_STAT_CFG_CFG_CNT_FRM_TYPE_SET(x)\
+ FIELD_PREP(ANA_AC_PORT_STAT_CFG_CFG_CNT_FRM_TYPE, x)
+#define ANA_AC_PORT_STAT_CFG_CFG_CNT_FRM_TYPE_GET(x)\
+ FIELD_GET(ANA_AC_PORT_STAT_CFG_CFG_CNT_FRM_TYPE, x)
+
+#define ANA_AC_PORT_STAT_CFG_CFG_CNT_BYTE BIT(0)
+#define ANA_AC_PORT_STAT_CFG_CFG_CNT_BYTE_SET(x)\
+ FIELD_PREP(ANA_AC_PORT_STAT_CFG_CFG_CNT_BYTE, x)
+#define ANA_AC_PORT_STAT_CFG_CFG_CNT_BYTE_GET(x)\
+ FIELD_GET(ANA_AC_PORT_STAT_CFG_CFG_CNT_BYTE, x)
+
+/* ANA_AC:STAT_CNT_CFG_PORT:STAT_LSB_CNT */
+#define ANA_AC_PORT_STAT_LSB_CNT(g, r) __REG(TARGET_ANA_AC, 0, 1, 843776, g, 70, 64, 20, r, 4, 4)
+
+/* ANA_ACL:COMMON:OWN_UPSID */
+#define ANA_ACL_OWN_UPSID(r) __REG(TARGET_ANA_ACL, 0, 1, 32768, 0, 1, 592, 580, r, 3, 4)
+
+#define ANA_ACL_OWN_UPSID_OWN_UPSID GENMASK(4, 0)
+#define ANA_ACL_OWN_UPSID_OWN_UPSID_SET(x)\
+ FIELD_PREP(ANA_ACL_OWN_UPSID_OWN_UPSID, x)
+#define ANA_ACL_OWN_UPSID_OWN_UPSID_GET(x)\
+ FIELD_GET(ANA_ACL_OWN_UPSID_OWN_UPSID, x)
+
+/* ANA_AC_POL:POL_ALL_CFG:POL_UPD_INT_CFG */
+#define ANA_AC_POL_POL_UPD_INT_CFG __REG(TARGET_ANA_AC_POL, 0, 1, 75968, 0, 1, 1160, 1148, 0, 1, 4)
+
+#define ANA_AC_POL_POL_UPD_INT_CFG_POL_UPD_INT GENMASK(9, 0)
+#define ANA_AC_POL_POL_UPD_INT_CFG_POL_UPD_INT_SET(x)\
+ FIELD_PREP(ANA_AC_POL_POL_UPD_INT_CFG_POL_UPD_INT, x)
+#define ANA_AC_POL_POL_UPD_INT_CFG_POL_UPD_INT_GET(x)\
+ FIELD_GET(ANA_AC_POL_POL_UPD_INT_CFG_POL_UPD_INT, x)
+
+/* ANA_AC_POL:COMMON_BDLB:DLB_CTRL */
+#define ANA_AC_POL_BDLB_DLB_CTRL __REG(TARGET_ANA_AC_POL, 0, 1, 79048, 0, 1, 8, 0, 0, 1, 4)
+
+#define ANA_AC_POL_BDLB_DLB_CTRL_CLK_PERIOD_01NS GENMASK(26, 19)
+#define ANA_AC_POL_BDLB_DLB_CTRL_CLK_PERIOD_01NS_SET(x)\
+ FIELD_PREP(ANA_AC_POL_BDLB_DLB_CTRL_CLK_PERIOD_01NS, x)
+#define ANA_AC_POL_BDLB_DLB_CTRL_CLK_PERIOD_01NS_GET(x)\
+ FIELD_GET(ANA_AC_POL_BDLB_DLB_CTRL_CLK_PERIOD_01NS, x)
+
+#define ANA_AC_POL_BDLB_DLB_CTRL_BASE_TICK_CNT GENMASK(18, 4)
+#define ANA_AC_POL_BDLB_DLB_CTRL_BASE_TICK_CNT_SET(x)\
+ FIELD_PREP(ANA_AC_POL_BDLB_DLB_CTRL_BASE_TICK_CNT, x)
+#define ANA_AC_POL_BDLB_DLB_CTRL_BASE_TICK_CNT_GET(x)\
+ FIELD_GET(ANA_AC_POL_BDLB_DLB_CTRL_BASE_TICK_CNT, x)
+
+#define ANA_AC_POL_BDLB_DLB_CTRL_LEAK_ENA BIT(1)
+#define ANA_AC_POL_BDLB_DLB_CTRL_LEAK_ENA_SET(x)\
+ FIELD_PREP(ANA_AC_POL_BDLB_DLB_CTRL_LEAK_ENA, x)
+#define ANA_AC_POL_BDLB_DLB_CTRL_LEAK_ENA_GET(x)\
+ FIELD_GET(ANA_AC_POL_BDLB_DLB_CTRL_LEAK_ENA, x)
+
+#define ANA_AC_POL_BDLB_DLB_CTRL_DLB_ADD_ENA BIT(0)
+#define ANA_AC_POL_BDLB_DLB_CTRL_DLB_ADD_ENA_SET(x)\
+ FIELD_PREP(ANA_AC_POL_BDLB_DLB_CTRL_DLB_ADD_ENA, x)
+#define ANA_AC_POL_BDLB_DLB_CTRL_DLB_ADD_ENA_GET(x)\
+ FIELD_GET(ANA_AC_POL_BDLB_DLB_CTRL_DLB_ADD_ENA, x)
+
+/* ANA_AC_POL:COMMON_BUM_SLB:DLB_CTRL */
+#define ANA_AC_POL_SLB_DLB_CTRL __REG(TARGET_ANA_AC_POL, 0, 1, 79056, 0, 1, 20, 0, 0, 1, 4)
+
+#define ANA_AC_POL_SLB_DLB_CTRL_CLK_PERIOD_01NS GENMASK(26, 19)
+#define ANA_AC_POL_SLB_DLB_CTRL_CLK_PERIOD_01NS_SET(x)\
+ FIELD_PREP(ANA_AC_POL_SLB_DLB_CTRL_CLK_PERIOD_01NS, x)
+#define ANA_AC_POL_SLB_DLB_CTRL_CLK_PERIOD_01NS_GET(x)\
+ FIELD_GET(ANA_AC_POL_SLB_DLB_CTRL_CLK_PERIOD_01NS, x)
+
+#define ANA_AC_POL_SLB_DLB_CTRL_BASE_TICK_CNT GENMASK(18, 4)
+#define ANA_AC_POL_SLB_DLB_CTRL_BASE_TICK_CNT_SET(x)\
+ FIELD_PREP(ANA_AC_POL_SLB_DLB_CTRL_BASE_TICK_CNT, x)
+#define ANA_AC_POL_SLB_DLB_CTRL_BASE_TICK_CNT_GET(x)\
+ FIELD_GET(ANA_AC_POL_SLB_DLB_CTRL_BASE_TICK_CNT, x)
+
+#define ANA_AC_POL_SLB_DLB_CTRL_LEAK_ENA BIT(1)
+#define ANA_AC_POL_SLB_DLB_CTRL_LEAK_ENA_SET(x)\
+ FIELD_PREP(ANA_AC_POL_SLB_DLB_CTRL_LEAK_ENA, x)
+#define ANA_AC_POL_SLB_DLB_CTRL_LEAK_ENA_GET(x)\
+ FIELD_GET(ANA_AC_POL_SLB_DLB_CTRL_LEAK_ENA, x)
+
+#define ANA_AC_POL_SLB_DLB_CTRL_DLB_ADD_ENA BIT(0)
+#define ANA_AC_POL_SLB_DLB_CTRL_DLB_ADD_ENA_SET(x)\
+ FIELD_PREP(ANA_AC_POL_SLB_DLB_CTRL_DLB_ADD_ENA, x)
+#define ANA_AC_POL_SLB_DLB_CTRL_DLB_ADD_ENA_GET(x)\
+ FIELD_GET(ANA_AC_POL_SLB_DLB_CTRL_DLB_ADD_ENA, x)
+
+/* ANA_CL:PORT:FILTER_CTRL */
+#define ANA_CL_FILTER_CTRL(g) __REG(TARGET_ANA_CL, 0, 1, 131072, g, 70, 512, 4, 0, 1, 4)
+
+#define ANA_CL_FILTER_CTRL_FILTER_SMAC_MC_DIS BIT(2)
+#define ANA_CL_FILTER_CTRL_FILTER_SMAC_MC_DIS_SET(x)\
+ FIELD_PREP(ANA_CL_FILTER_CTRL_FILTER_SMAC_MC_DIS, x)
+#define ANA_CL_FILTER_CTRL_FILTER_SMAC_MC_DIS_GET(x)\
+ FIELD_GET(ANA_CL_FILTER_CTRL_FILTER_SMAC_MC_DIS, x)
+
+#define ANA_CL_FILTER_CTRL_FILTER_NULL_MAC_DIS BIT(1)
+#define ANA_CL_FILTER_CTRL_FILTER_NULL_MAC_DIS_SET(x)\
+ FIELD_PREP(ANA_CL_FILTER_CTRL_FILTER_NULL_MAC_DIS, x)
+#define ANA_CL_FILTER_CTRL_FILTER_NULL_MAC_DIS_GET(x)\
+ FIELD_GET(ANA_CL_FILTER_CTRL_FILTER_NULL_MAC_DIS, x)
+
+#define ANA_CL_FILTER_CTRL_FORCE_FCS_UPDATE_ENA BIT(0)
+#define ANA_CL_FILTER_CTRL_FORCE_FCS_UPDATE_ENA_SET(x)\
+ FIELD_PREP(ANA_CL_FILTER_CTRL_FORCE_FCS_UPDATE_ENA, x)
+#define ANA_CL_FILTER_CTRL_FORCE_FCS_UPDATE_ENA_GET(x)\
+ FIELD_GET(ANA_CL_FILTER_CTRL_FORCE_FCS_UPDATE_ENA, x)
+
+/* ANA_CL:PORT:VLAN_FILTER_CTRL */
+#define ANA_CL_VLAN_FILTER_CTRL(g, r) __REG(TARGET_ANA_CL, 0, 1, 131072, g, 70, 512, 8, r, 3, 4)
+
+#define ANA_CL_VLAN_FILTER_CTRL_TAG_REQUIRED_ENA BIT(10)
+#define ANA_CL_VLAN_FILTER_CTRL_TAG_REQUIRED_ENA_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_FILTER_CTRL_TAG_REQUIRED_ENA, x)
+#define ANA_CL_VLAN_FILTER_CTRL_TAG_REQUIRED_ENA_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_FILTER_CTRL_TAG_REQUIRED_ENA, x)
+
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_CTAG_DIS BIT(9)
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_CTAG_DIS_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_FILTER_CTRL_PRIO_CTAG_DIS, x)
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_CTAG_DIS_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_FILTER_CTRL_PRIO_CTAG_DIS, x)
+
+#define ANA_CL_VLAN_FILTER_CTRL_CTAG_DIS BIT(8)
+#define ANA_CL_VLAN_FILTER_CTRL_CTAG_DIS_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_FILTER_CTRL_CTAG_DIS, x)
+#define ANA_CL_VLAN_FILTER_CTRL_CTAG_DIS_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_FILTER_CTRL_CTAG_DIS, x)
+
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_STAG_DIS BIT(7)
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_STAG_DIS_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_FILTER_CTRL_PRIO_STAG_DIS, x)
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_STAG_DIS_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_FILTER_CTRL_PRIO_STAG_DIS, x)
+
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST1_STAG_DIS BIT(6)
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST1_STAG_DIS_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST1_STAG_DIS, x)
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST1_STAG_DIS_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST1_STAG_DIS, x)
+
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST2_STAG_DIS BIT(5)
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST2_STAG_DIS_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST2_STAG_DIS, x)
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST2_STAG_DIS_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST2_STAG_DIS, x)
+
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST3_STAG_DIS BIT(4)
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST3_STAG_DIS_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST3_STAG_DIS, x)
+#define ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST3_STAG_DIS_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_FILTER_CTRL_PRIO_CUST3_STAG_DIS, x)
+
+#define ANA_CL_VLAN_FILTER_CTRL_STAG_DIS BIT(3)
+#define ANA_CL_VLAN_FILTER_CTRL_STAG_DIS_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_FILTER_CTRL_STAG_DIS, x)
+#define ANA_CL_VLAN_FILTER_CTRL_STAG_DIS_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_FILTER_CTRL_STAG_DIS, x)
+
+#define ANA_CL_VLAN_FILTER_CTRL_CUST1_STAG_DIS BIT(2)
+#define ANA_CL_VLAN_FILTER_CTRL_CUST1_STAG_DIS_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_FILTER_CTRL_CUST1_STAG_DIS, x)
+#define ANA_CL_VLAN_FILTER_CTRL_CUST1_STAG_DIS_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_FILTER_CTRL_CUST1_STAG_DIS, x)
+
+#define ANA_CL_VLAN_FILTER_CTRL_CUST2_STAG_DIS BIT(1)
+#define ANA_CL_VLAN_FILTER_CTRL_CUST2_STAG_DIS_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_FILTER_CTRL_CUST2_STAG_DIS, x)
+#define ANA_CL_VLAN_FILTER_CTRL_CUST2_STAG_DIS_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_FILTER_CTRL_CUST2_STAG_DIS, x)
+
+#define ANA_CL_VLAN_FILTER_CTRL_CUST3_STAG_DIS BIT(0)
+#define ANA_CL_VLAN_FILTER_CTRL_CUST3_STAG_DIS_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_FILTER_CTRL_CUST3_STAG_DIS, x)
+#define ANA_CL_VLAN_FILTER_CTRL_CUST3_STAG_DIS_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_FILTER_CTRL_CUST3_STAG_DIS, x)
+
+/* ANA_CL:PORT:ETAG_FILTER_CTRL */
+#define ANA_CL_ETAG_FILTER_CTRL(g) __REG(TARGET_ANA_CL, 0, 1, 131072, g, 70, 512, 20, 0, 1, 4)
+
+#define ANA_CL_ETAG_FILTER_CTRL_ETAG_REQUIRED_ENA BIT(1)
+#define ANA_CL_ETAG_FILTER_CTRL_ETAG_REQUIRED_ENA_SET(x)\
+ FIELD_PREP(ANA_CL_ETAG_FILTER_CTRL_ETAG_REQUIRED_ENA, x)
+#define ANA_CL_ETAG_FILTER_CTRL_ETAG_REQUIRED_ENA_GET(x)\
+ FIELD_GET(ANA_CL_ETAG_FILTER_CTRL_ETAG_REQUIRED_ENA, x)
+
+#define ANA_CL_ETAG_FILTER_CTRL_ETAG_DIS BIT(0)
+#define ANA_CL_ETAG_FILTER_CTRL_ETAG_DIS_SET(x)\
+ FIELD_PREP(ANA_CL_ETAG_FILTER_CTRL_ETAG_DIS, x)
+#define ANA_CL_ETAG_FILTER_CTRL_ETAG_DIS_GET(x)\
+ FIELD_GET(ANA_CL_ETAG_FILTER_CTRL_ETAG_DIS, x)
+
+/* ANA_CL:PORT:VLAN_CTRL */
+#define ANA_CL_VLAN_CTRL(g) __REG(TARGET_ANA_CL, 0, 1, 131072, g, 70, 512, 32, 0, 1, 4)
+
+#define ANA_CL_VLAN_CTRL_PORT_VOE_TPID_AWARE_DIS GENMASK(30, 26)
+#define ANA_CL_VLAN_CTRL_PORT_VOE_TPID_AWARE_DIS_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_CTRL_PORT_VOE_TPID_AWARE_DIS, x)
+#define ANA_CL_VLAN_CTRL_PORT_VOE_TPID_AWARE_DIS_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_CTRL_PORT_VOE_TPID_AWARE_DIS, x)
+
+#define ANA_CL_VLAN_CTRL_PORT_VOE_DEFAULT_PCP GENMASK(25, 23)
+#define ANA_CL_VLAN_CTRL_PORT_VOE_DEFAULT_PCP_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_CTRL_PORT_VOE_DEFAULT_PCP, x)
+#define ANA_CL_VLAN_CTRL_PORT_VOE_DEFAULT_PCP_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_CTRL_PORT_VOE_DEFAULT_PCP, x)
+
+#define ANA_CL_VLAN_CTRL_PORT_VOE_DEFAULT_DEI BIT(22)
+#define ANA_CL_VLAN_CTRL_PORT_VOE_DEFAULT_DEI_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_CTRL_PORT_VOE_DEFAULT_DEI, x)
+#define ANA_CL_VLAN_CTRL_PORT_VOE_DEFAULT_DEI_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_CTRL_PORT_VOE_DEFAULT_DEI, x)
+
+#define ANA_CL_VLAN_CTRL_VLAN_PCP_DEI_TRANS_ENA BIT(21)
+#define ANA_CL_VLAN_CTRL_VLAN_PCP_DEI_TRANS_ENA_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_CTRL_VLAN_PCP_DEI_TRANS_ENA, x)
+#define ANA_CL_VLAN_CTRL_VLAN_PCP_DEI_TRANS_ENA_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_CTRL_VLAN_PCP_DEI_TRANS_ENA, x)
+
+#define ANA_CL_VLAN_CTRL_VLAN_TAG_SEL BIT(20)
+#define ANA_CL_VLAN_CTRL_VLAN_TAG_SEL_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_CTRL_VLAN_TAG_SEL, x)
+#define ANA_CL_VLAN_CTRL_VLAN_TAG_SEL_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_CTRL_VLAN_TAG_SEL, x)
+
+#define ANA_CL_VLAN_CTRL_VLAN_AWARE_ENA BIT(19)
+#define ANA_CL_VLAN_CTRL_VLAN_AWARE_ENA_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_CTRL_VLAN_AWARE_ENA, x)
+#define ANA_CL_VLAN_CTRL_VLAN_AWARE_ENA_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_CTRL_VLAN_AWARE_ENA, x)
+
+#define ANA_CL_VLAN_CTRL_VLAN_POP_CNT GENMASK(18, 17)
+#define ANA_CL_VLAN_CTRL_VLAN_POP_CNT_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_CTRL_VLAN_POP_CNT, x)
+#define ANA_CL_VLAN_CTRL_VLAN_POP_CNT_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_CTRL_VLAN_POP_CNT, x)
+
+#define ANA_CL_VLAN_CTRL_PORT_TAG_TYPE BIT(16)
+#define ANA_CL_VLAN_CTRL_PORT_TAG_TYPE_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_CTRL_PORT_TAG_TYPE, x)
+#define ANA_CL_VLAN_CTRL_PORT_TAG_TYPE_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_CTRL_PORT_TAG_TYPE, x)
+
+#define ANA_CL_VLAN_CTRL_PORT_PCP GENMASK(15, 13)
+#define ANA_CL_VLAN_CTRL_PORT_PCP_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_CTRL_PORT_PCP, x)
+#define ANA_CL_VLAN_CTRL_PORT_PCP_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_CTRL_PORT_PCP, x)
+
+#define ANA_CL_VLAN_CTRL_PORT_DEI BIT(12)
+#define ANA_CL_VLAN_CTRL_PORT_DEI_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_CTRL_PORT_DEI, x)
+#define ANA_CL_VLAN_CTRL_PORT_DEI_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_CTRL_PORT_DEI, x)
+
+#define ANA_CL_VLAN_CTRL_PORT_VID GENMASK(11, 0)
+#define ANA_CL_VLAN_CTRL_PORT_VID_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_CTRL_PORT_VID, x)
+#define ANA_CL_VLAN_CTRL_PORT_VID_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_CTRL_PORT_VID, x)
+
+/* ANA_CL:PORT:VLAN_CTRL_2 */
+#define ANA_CL_VLAN_CTRL_2(g) __REG(TARGET_ANA_CL, 0, 1, 131072, g, 70, 512, 36, 0, 1, 4)
+
+#define ANA_CL_VLAN_CTRL_2_VLAN_PUSH_CNT GENMASK(1, 0)
+#define ANA_CL_VLAN_CTRL_2_VLAN_PUSH_CNT_SET(x)\
+ FIELD_PREP(ANA_CL_VLAN_CTRL_2_VLAN_PUSH_CNT, x)
+#define ANA_CL_VLAN_CTRL_2_VLAN_PUSH_CNT_GET(x)\
+ FIELD_GET(ANA_CL_VLAN_CTRL_2_VLAN_PUSH_CNT, x)
+
+/* ANA_CL:PORT:CAPTURE_BPDU_CFG */
+#define ANA_CL_CAPTURE_BPDU_CFG(g) __REG(TARGET_ANA_CL, 0, 1, 131072, g, 70, 512, 196, 0, 1, 4)
+
+/* ANA_CL:COMMON:OWN_UPSID */
+#define ANA_CL_OWN_UPSID(r) __REG(TARGET_ANA_CL, 0, 1, 166912, 0, 1, 756, 0, r, 3, 4)
+
+#define ANA_CL_OWN_UPSID_OWN_UPSID GENMASK(4, 0)
+#define ANA_CL_OWN_UPSID_OWN_UPSID_SET(x)\
+ FIELD_PREP(ANA_CL_OWN_UPSID_OWN_UPSID, x)
+#define ANA_CL_OWN_UPSID_OWN_UPSID_GET(x)\
+ FIELD_GET(ANA_CL_OWN_UPSID_OWN_UPSID, x)
+
+/* ANA_L2:COMMON:AUTO_LRN_CFG */
+#define ANA_L2_AUTO_LRN_CFG __REG(TARGET_ANA_L2, 0, 1, 566024, 0, 1, 700, 24, 0, 1, 4)
+
+/* ANA_L2:COMMON:AUTO_LRN_CFG1 */
+#define ANA_L2_AUTO_LRN_CFG1 __REG(TARGET_ANA_L2, 0, 1, 566024, 0, 1, 700, 28, 0, 1, 4)
+
+/* ANA_L2:COMMON:AUTO_LRN_CFG2 */
+#define ANA_L2_AUTO_LRN_CFG2 __REG(TARGET_ANA_L2, 0, 1, 566024, 0, 1, 700, 32, 0, 1, 4)
+
+#define ANA_L2_AUTO_LRN_CFG2_AUTO_LRN_ENA2 BIT(0)
+#define ANA_L2_AUTO_LRN_CFG2_AUTO_LRN_ENA2_SET(x)\
+ FIELD_PREP(ANA_L2_AUTO_LRN_CFG2_AUTO_LRN_ENA2, x)
+#define ANA_L2_AUTO_LRN_CFG2_AUTO_LRN_ENA2_GET(x)\
+ FIELD_GET(ANA_L2_AUTO_LRN_CFG2_AUTO_LRN_ENA2, x)
+
+/* ANA_L2:COMMON:OWN_UPSID */
+#define ANA_L2_OWN_UPSID(r) __REG(TARGET_ANA_L2, 0, 1, 566024, 0, 1, 700, 672, r, 3, 4)
+
+#define ANA_L2_OWN_UPSID_OWN_UPSID GENMASK(4, 0)
+#define ANA_L2_OWN_UPSID_OWN_UPSID_SET(x)\
+ FIELD_PREP(ANA_L2_OWN_UPSID_OWN_UPSID, x)
+#define ANA_L2_OWN_UPSID_OWN_UPSID_GET(x)\
+ FIELD_GET(ANA_L2_OWN_UPSID_OWN_UPSID, x)
+
+/* ANA_L3:COMMON:VLAN_CTRL */
+#define ANA_L3_VLAN_CTRL __REG(TARGET_ANA_L3, 0, 1, 493632, 0, 1, 184, 4, 0, 1, 4)
+
+#define ANA_L3_VLAN_CTRL_VLAN_ENA BIT(0)
+#define ANA_L3_VLAN_CTRL_VLAN_ENA_SET(x)\
+ FIELD_PREP(ANA_L3_VLAN_CTRL_VLAN_ENA, x)
+#define ANA_L3_VLAN_CTRL_VLAN_ENA_GET(x)\
+ FIELD_GET(ANA_L3_VLAN_CTRL_VLAN_ENA, x)
+
+/* ANA_L3:VLAN:VLAN_CFG */
+#define ANA_L3_VLAN_CFG(g) __REG(TARGET_ANA_L3, 0, 1, 0, g, 5120, 64, 8, 0, 1, 4)
+
+#define ANA_L3_VLAN_CFG_VLAN_MSTP_PTR GENMASK(30, 24)
+#define ANA_L3_VLAN_CFG_VLAN_MSTP_PTR_SET(x)\
+ FIELD_PREP(ANA_L3_VLAN_CFG_VLAN_MSTP_PTR, x)
+#define ANA_L3_VLAN_CFG_VLAN_MSTP_PTR_GET(x)\
+ FIELD_GET(ANA_L3_VLAN_CFG_VLAN_MSTP_PTR, x)
+
+#define ANA_L3_VLAN_CFG_VLAN_FID GENMASK(20, 8)
+#define ANA_L3_VLAN_CFG_VLAN_FID_SET(x)\
+ FIELD_PREP(ANA_L3_VLAN_CFG_VLAN_FID, x)
+#define ANA_L3_VLAN_CFG_VLAN_FID_GET(x)\
+ FIELD_GET(ANA_L3_VLAN_CFG_VLAN_FID, x)
+
+#define ANA_L3_VLAN_CFG_VLAN_IGR_FILTER_ENA BIT(6)
+#define ANA_L3_VLAN_CFG_VLAN_IGR_FILTER_ENA_SET(x)\
+ FIELD_PREP(ANA_L3_VLAN_CFG_VLAN_IGR_FILTER_ENA, x)
+#define ANA_L3_VLAN_CFG_VLAN_IGR_FILTER_ENA_GET(x)\
+ FIELD_GET(ANA_L3_VLAN_CFG_VLAN_IGR_FILTER_ENA, x)
+
+#define ANA_L3_VLAN_CFG_VLAN_SEC_FWD_ENA BIT(5)
+#define ANA_L3_VLAN_CFG_VLAN_SEC_FWD_ENA_SET(x)\
+ FIELD_PREP(ANA_L3_VLAN_CFG_VLAN_SEC_FWD_ENA, x)
+#define ANA_L3_VLAN_CFG_VLAN_SEC_FWD_ENA_GET(x)\
+ FIELD_GET(ANA_L3_VLAN_CFG_VLAN_SEC_FWD_ENA, x)
+
+#define ANA_L3_VLAN_CFG_VLAN_FLOOD_DIS BIT(4)
+#define ANA_L3_VLAN_CFG_VLAN_FLOOD_DIS_SET(x)\
+ FIELD_PREP(ANA_L3_VLAN_CFG_VLAN_FLOOD_DIS, x)
+#define ANA_L3_VLAN_CFG_VLAN_FLOOD_DIS_GET(x)\
+ FIELD_GET(ANA_L3_VLAN_CFG_VLAN_FLOOD_DIS, x)
+
+#define ANA_L3_VLAN_CFG_VLAN_LRN_DIS BIT(3)
+#define ANA_L3_VLAN_CFG_VLAN_LRN_DIS_SET(x)\
+ FIELD_PREP(ANA_L3_VLAN_CFG_VLAN_LRN_DIS, x)
+#define ANA_L3_VLAN_CFG_VLAN_LRN_DIS_GET(x)\
+ FIELD_GET(ANA_L3_VLAN_CFG_VLAN_LRN_DIS, x)
+
+#define ANA_L3_VLAN_CFG_VLAN_RLEG_ENA BIT(2)
+#define ANA_L3_VLAN_CFG_VLAN_RLEG_ENA_SET(x)\
+ FIELD_PREP(ANA_L3_VLAN_CFG_VLAN_RLEG_ENA, x)
+#define ANA_L3_VLAN_CFG_VLAN_RLEG_ENA_GET(x)\
+ FIELD_GET(ANA_L3_VLAN_CFG_VLAN_RLEG_ENA, x)
+
+#define ANA_L3_VLAN_CFG_VLAN_PRIVATE_ENA BIT(1)
+#define ANA_L3_VLAN_CFG_VLAN_PRIVATE_ENA_SET(x)\
+ FIELD_PREP(ANA_L3_VLAN_CFG_VLAN_PRIVATE_ENA, x)
+#define ANA_L3_VLAN_CFG_VLAN_PRIVATE_ENA_GET(x)\
+ FIELD_GET(ANA_L3_VLAN_CFG_VLAN_PRIVATE_ENA, x)
+
+#define ANA_L3_VLAN_CFG_VLAN_MIRROR_ENA BIT(0)
+#define ANA_L3_VLAN_CFG_VLAN_MIRROR_ENA_SET(x)\
+ FIELD_PREP(ANA_L3_VLAN_CFG_VLAN_MIRROR_ENA, x)
+#define ANA_L3_VLAN_CFG_VLAN_MIRROR_ENA_GET(x)\
+ FIELD_GET(ANA_L3_VLAN_CFG_VLAN_MIRROR_ENA, x)
+
+/* ANA_L3:VLAN:VLAN_MASK_CFG */
+#define ANA_L3_VLAN_MASK_CFG(g) __REG(TARGET_ANA_L3, 0, 1, 0, g, 5120, 64, 16, 0, 1, 4)
+
+/* ANA_L3:VLAN:VLAN_MASK_CFG1 */
+#define ANA_L3_VLAN_MASK_CFG1(g) __REG(TARGET_ANA_L3, 0, 1, 0, g, 5120, 64, 20, 0, 1, 4)
+
+/* ANA_L3:VLAN:VLAN_MASK_CFG2 */
+#define ANA_L3_VLAN_MASK_CFG2(g) __REG(TARGET_ANA_L3, 0, 1, 0, g, 5120, 64, 24, 0, 1, 4)
+
+#define ANA_L3_VLAN_MASK_CFG2_VLAN_PORT_MASK2 BIT(0)
+#define ANA_L3_VLAN_MASK_CFG2_VLAN_PORT_MASK2_SET(x)\
+ FIELD_PREP(ANA_L3_VLAN_MASK_CFG2_VLAN_PORT_MASK2, x)
+#define ANA_L3_VLAN_MASK_CFG2_VLAN_PORT_MASK2_GET(x)\
+ FIELD_GET(ANA_L3_VLAN_MASK_CFG2_VLAN_PORT_MASK2, x)
+
+/* ASM:DEV_STATISTICS:RX_IN_BYTES_CNT */
+#define ASM_RX_IN_BYTES_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 0, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_SYMBOL_ERR_CNT */
+#define ASM_RX_SYMBOL_ERR_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 4, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_PAUSE_CNT */
+#define ASM_RX_PAUSE_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 8, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_UNSUP_OPCODE_CNT */
+#define ASM_RX_UNSUP_OPCODE_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 12, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_OK_BYTES_CNT */
+#define ASM_RX_OK_BYTES_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 16, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_BAD_BYTES_CNT */
+#define ASM_RX_BAD_BYTES_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 20, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_UC_CNT */
+#define ASM_RX_UC_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 24, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_MC_CNT */
+#define ASM_RX_MC_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 28, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_BC_CNT */
+#define ASM_RX_BC_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 32, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_CRC_ERR_CNT */
+#define ASM_RX_CRC_ERR_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 36, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_UNDERSIZE_CNT */
+#define ASM_RX_UNDERSIZE_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 40, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_FRAGMENTS_CNT */
+#define ASM_RX_FRAGMENTS_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 44, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_IN_RANGE_LEN_ERR_CNT */
+#define ASM_RX_IN_RANGE_LEN_ERR_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 48, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_OUT_OF_RANGE_LEN_ERR_CNT */
+#define ASM_RX_OUT_OF_RANGE_LEN_ERR_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 52, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_OVERSIZE_CNT */
+#define ASM_RX_OVERSIZE_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 56, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_JABBERS_CNT */
+#define ASM_RX_JABBERS_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 60, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_SIZE64_CNT */
+#define ASM_RX_SIZE64_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 64, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_SIZE65TO127_CNT */
+#define ASM_RX_SIZE65TO127_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 68, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_SIZE128TO255_CNT */
+#define ASM_RX_SIZE128TO255_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 72, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_SIZE256TO511_CNT */
+#define ASM_RX_SIZE256TO511_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 76, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_SIZE512TO1023_CNT */
+#define ASM_RX_SIZE512TO1023_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 80, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_SIZE1024TO1518_CNT */
+#define ASM_RX_SIZE1024TO1518_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 84, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_SIZE1519TOMAX_CNT */
+#define ASM_RX_SIZE1519TOMAX_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 88, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_IPG_SHRINK_CNT */
+#define ASM_RX_IPG_SHRINK_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 92, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_OUT_BYTES_CNT */
+#define ASM_TX_OUT_BYTES_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 96, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_PAUSE_CNT */
+#define ASM_TX_PAUSE_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 100, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_OK_BYTES_CNT */
+#define ASM_TX_OK_BYTES_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 104, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_UC_CNT */
+#define ASM_TX_UC_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 108, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_MC_CNT */
+#define ASM_TX_MC_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 112, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_BC_CNT */
+#define ASM_TX_BC_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 116, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_SIZE64_CNT */
+#define ASM_TX_SIZE64_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 120, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_SIZE65TO127_CNT */
+#define ASM_TX_SIZE65TO127_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 124, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_SIZE128TO255_CNT */
+#define ASM_TX_SIZE128TO255_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 128, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_SIZE256TO511_CNT */
+#define ASM_TX_SIZE256TO511_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 132, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_SIZE512TO1023_CNT */
+#define ASM_TX_SIZE512TO1023_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 136, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_SIZE1024TO1518_CNT */
+#define ASM_TX_SIZE1024TO1518_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 140, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_SIZE1519TOMAX_CNT */
+#define ASM_TX_SIZE1519TOMAX_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 144, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_ALIGNMENT_LOST_CNT */
+#define ASM_RX_ALIGNMENT_LOST_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 148, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_TAGGED_FRMS_CNT */
+#define ASM_RX_TAGGED_FRMS_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 152, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_UNTAGGED_FRMS_CNT */
+#define ASM_RX_UNTAGGED_FRMS_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 156, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_TAGGED_FRMS_CNT */
+#define ASM_TX_TAGGED_FRMS_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 160, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_UNTAGGED_FRMS_CNT */
+#define ASM_TX_UNTAGGED_FRMS_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 164, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_SYMBOL_ERR_CNT */
+#define ASM_PMAC_RX_SYMBOL_ERR_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 168, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_PAUSE_CNT */
+#define ASM_PMAC_RX_PAUSE_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 172, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_UNSUP_OPCODE_CNT */
+#define ASM_PMAC_RX_UNSUP_OPCODE_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 176, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_OK_BYTES_CNT */
+#define ASM_PMAC_RX_OK_BYTES_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 180, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_BAD_BYTES_CNT */
+#define ASM_PMAC_RX_BAD_BYTES_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 184, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_UC_CNT */
+#define ASM_PMAC_RX_UC_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 188, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_MC_CNT */
+#define ASM_PMAC_RX_MC_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 192, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_BC_CNT */
+#define ASM_PMAC_RX_BC_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 196, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_CRC_ERR_CNT */
+#define ASM_PMAC_RX_CRC_ERR_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 200, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_UNDERSIZE_CNT */
+#define ASM_PMAC_RX_UNDERSIZE_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 204, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_FRAGMENTS_CNT */
+#define ASM_PMAC_RX_FRAGMENTS_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 208, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_IN_RANGE_LEN_ERR_CNT */
+#define ASM_PMAC_RX_IN_RANGE_LEN_ERR_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 212, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_OUT_OF_RANGE_LEN_ERR_CNT */
+#define ASM_PMAC_RX_OUT_OF_RANGE_LEN_ERR_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 216, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_OVERSIZE_CNT */
+#define ASM_PMAC_RX_OVERSIZE_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 220, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_JABBERS_CNT */
+#define ASM_PMAC_RX_JABBERS_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 224, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_SIZE64_CNT */
+#define ASM_PMAC_RX_SIZE64_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 228, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_SIZE65TO127_CNT */
+#define ASM_PMAC_RX_SIZE65TO127_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 232, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_SIZE128TO255_CNT */
+#define ASM_PMAC_RX_SIZE128TO255_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 236, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_SIZE256TO511_CNT */
+#define ASM_PMAC_RX_SIZE256TO511_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 240, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_SIZE512TO1023_CNT */
+#define ASM_PMAC_RX_SIZE512TO1023_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 244, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_SIZE1024TO1518_CNT */
+#define ASM_PMAC_RX_SIZE1024TO1518_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 248, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_SIZE1519TOMAX_CNT */
+#define ASM_PMAC_RX_SIZE1519TOMAX_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 252, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_TX_PAUSE_CNT */
+#define ASM_PMAC_TX_PAUSE_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 256, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_TX_OK_BYTES_CNT */
+#define ASM_PMAC_TX_OK_BYTES_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 260, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_TX_UC_CNT */
+#define ASM_PMAC_TX_UC_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 264, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_TX_MC_CNT */
+#define ASM_PMAC_TX_MC_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 268, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_TX_BC_CNT */
+#define ASM_PMAC_TX_BC_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 272, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_TX_SIZE64_CNT */
+#define ASM_PMAC_TX_SIZE64_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 276, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_TX_SIZE65TO127_CNT */
+#define ASM_PMAC_TX_SIZE65TO127_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 280, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_TX_SIZE128TO255_CNT */
+#define ASM_PMAC_TX_SIZE128TO255_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 284, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_TX_SIZE256TO511_CNT */
+#define ASM_PMAC_TX_SIZE256TO511_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 288, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_TX_SIZE512TO1023_CNT */
+#define ASM_PMAC_TX_SIZE512TO1023_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 292, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_TX_SIZE1024TO1518_CNT */
+#define ASM_PMAC_TX_SIZE1024TO1518_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 296, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_TX_SIZE1519TOMAX_CNT */
+#define ASM_PMAC_TX_SIZE1519TOMAX_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 300, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_ALIGNMENT_LOST_CNT */
+#define ASM_PMAC_RX_ALIGNMENT_LOST_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 304, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:MM_RX_ASSEMBLY_ERR_CNT */
+#define ASM_MM_RX_ASSEMBLY_ERR_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 308, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:MM_RX_SMD_ERR_CNT */
+#define ASM_MM_RX_SMD_ERR_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 312, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:MM_RX_ASSEMBLY_OK_CNT */
+#define ASM_MM_RX_ASSEMBLY_OK_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 316, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:MM_RX_MERGE_FRAG_CNT */
+#define ASM_MM_RX_MERGE_FRAG_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 320, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:MM_TX_PFRAGMENT_CNT */
+#define ASM_MM_TX_PFRAGMENT_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 324, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_MULTI_COLL_CNT */
+#define ASM_TX_MULTI_COLL_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 328, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_LATE_COLL_CNT */
+#define ASM_TX_LATE_COLL_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 332, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_XCOLL_CNT */
+#define ASM_TX_XCOLL_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 336, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_DEFER_CNT */
+#define ASM_TX_DEFER_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 340, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_XDEFER_CNT */
+#define ASM_TX_XDEFER_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 344, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_BACKOFF1_CNT */
+#define ASM_TX_BACKOFF1_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 348, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:TX_CSENSE_CNT */
+#define ASM_TX_CSENSE_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 352, 0, 1, 4)
+
+/* ASM:DEV_STATISTICS:RX_IN_BYTES_MSB_CNT */
+#define ASM_RX_IN_BYTES_MSB_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 356, 0, 1, 4)
+
+#define ASM_RX_IN_BYTES_MSB_CNT_RX_IN_BYTES_MSB_CNT GENMASK(3, 0)
+#define ASM_RX_IN_BYTES_MSB_CNT_RX_IN_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(ASM_RX_IN_BYTES_MSB_CNT_RX_IN_BYTES_MSB_CNT, x)
+#define ASM_RX_IN_BYTES_MSB_CNT_RX_IN_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(ASM_RX_IN_BYTES_MSB_CNT_RX_IN_BYTES_MSB_CNT, x)
+
+/* ASM:DEV_STATISTICS:RX_OK_BYTES_MSB_CNT */
+#define ASM_RX_OK_BYTES_MSB_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 360, 0, 1, 4)
+
+#define ASM_RX_OK_BYTES_MSB_CNT_RX_OK_BYTES_MSB_CNT GENMASK(3, 0)
+#define ASM_RX_OK_BYTES_MSB_CNT_RX_OK_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(ASM_RX_OK_BYTES_MSB_CNT_RX_OK_BYTES_MSB_CNT, x)
+#define ASM_RX_OK_BYTES_MSB_CNT_RX_OK_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(ASM_RX_OK_BYTES_MSB_CNT_RX_OK_BYTES_MSB_CNT, x)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_OK_BYTES_MSB_CNT */
+#define ASM_PMAC_RX_OK_BYTES_MSB_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 364, 0, 1, 4)
+
+#define ASM_PMAC_RX_OK_BYTES_MSB_CNT_PMAC_RX_OK_BYTES_MSB_CNT GENMASK(3, 0)
+#define ASM_PMAC_RX_OK_BYTES_MSB_CNT_PMAC_RX_OK_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(ASM_PMAC_RX_OK_BYTES_MSB_CNT_PMAC_RX_OK_BYTES_MSB_CNT, x)
+#define ASM_PMAC_RX_OK_BYTES_MSB_CNT_PMAC_RX_OK_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(ASM_PMAC_RX_OK_BYTES_MSB_CNT_PMAC_RX_OK_BYTES_MSB_CNT, x)
+
+/* ASM:DEV_STATISTICS:RX_BAD_BYTES_MSB_CNT */
+#define ASM_RX_BAD_BYTES_MSB_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 368, 0, 1, 4)
+
+#define ASM_RX_BAD_BYTES_MSB_CNT_RX_BAD_BYTES_MSB_CNT GENMASK(3, 0)
+#define ASM_RX_BAD_BYTES_MSB_CNT_RX_BAD_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(ASM_RX_BAD_BYTES_MSB_CNT_RX_BAD_BYTES_MSB_CNT, x)
+#define ASM_RX_BAD_BYTES_MSB_CNT_RX_BAD_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(ASM_RX_BAD_BYTES_MSB_CNT_RX_BAD_BYTES_MSB_CNT, x)
+
+/* ASM:DEV_STATISTICS:PMAC_RX_BAD_BYTES_MSB_CNT */
+#define ASM_PMAC_RX_BAD_BYTES_MSB_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 372, 0, 1, 4)
+
+#define ASM_PMAC_RX_BAD_BYTES_MSB_CNT_PMAC_RX_BAD_BYTES_MSB_CNT GENMASK(3, 0)
+#define ASM_PMAC_RX_BAD_BYTES_MSB_CNT_PMAC_RX_BAD_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(ASM_PMAC_RX_BAD_BYTES_MSB_CNT_PMAC_RX_BAD_BYTES_MSB_CNT, x)
+#define ASM_PMAC_RX_BAD_BYTES_MSB_CNT_PMAC_RX_BAD_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(ASM_PMAC_RX_BAD_BYTES_MSB_CNT_PMAC_RX_BAD_BYTES_MSB_CNT, x)
+
+/* ASM:DEV_STATISTICS:TX_OUT_BYTES_MSB_CNT */
+#define ASM_TX_OUT_BYTES_MSB_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 376, 0, 1, 4)
+
+#define ASM_TX_OUT_BYTES_MSB_CNT_TX_OUT_BYTES_MSB_CNT GENMASK(3, 0)
+#define ASM_TX_OUT_BYTES_MSB_CNT_TX_OUT_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(ASM_TX_OUT_BYTES_MSB_CNT_TX_OUT_BYTES_MSB_CNT, x)
+#define ASM_TX_OUT_BYTES_MSB_CNT_TX_OUT_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(ASM_TX_OUT_BYTES_MSB_CNT_TX_OUT_BYTES_MSB_CNT, x)
+
+/* ASM:DEV_STATISTICS:TX_OK_BYTES_MSB_CNT */
+#define ASM_TX_OK_BYTES_MSB_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 380, 0, 1, 4)
+
+#define ASM_TX_OK_BYTES_MSB_CNT_TX_OK_BYTES_MSB_CNT GENMASK(3, 0)
+#define ASM_TX_OK_BYTES_MSB_CNT_TX_OK_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(ASM_TX_OK_BYTES_MSB_CNT_TX_OK_BYTES_MSB_CNT, x)
+#define ASM_TX_OK_BYTES_MSB_CNT_TX_OK_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(ASM_TX_OK_BYTES_MSB_CNT_TX_OK_BYTES_MSB_CNT, x)
+
+/* ASM:DEV_STATISTICS:PMAC_TX_OK_BYTES_MSB_CNT */
+#define ASM_PMAC_TX_OK_BYTES_MSB_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 384, 0, 1, 4)
+
+#define ASM_PMAC_TX_OK_BYTES_MSB_CNT_PMAC_TX_OK_BYTES_MSB_CNT GENMASK(3, 0)
+#define ASM_PMAC_TX_OK_BYTES_MSB_CNT_PMAC_TX_OK_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(ASM_PMAC_TX_OK_BYTES_MSB_CNT_PMAC_TX_OK_BYTES_MSB_CNT, x)
+#define ASM_PMAC_TX_OK_BYTES_MSB_CNT_PMAC_TX_OK_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(ASM_PMAC_TX_OK_BYTES_MSB_CNT_PMAC_TX_OK_BYTES_MSB_CNT, x)
+
+/* ASM:DEV_STATISTICS:RX_SYNC_LOST_ERR_CNT */
+#define ASM_RX_SYNC_LOST_ERR_CNT(g) __REG(TARGET_ASM, 0, 1, 0, g, 65, 512, 388, 0, 1, 4)
+
+/* ASM:CFG:STAT_CFG */
+#define ASM_STAT_CFG __REG(TARGET_ASM, 0, 1, 33280, 0, 1, 1088, 0, 0, 1, 4)
+
+#define ASM_STAT_CFG_STAT_CNT_CLR_SHOT BIT(0)
+#define ASM_STAT_CFG_STAT_CNT_CLR_SHOT_SET(x)\
+ FIELD_PREP(ASM_STAT_CFG_STAT_CNT_CLR_SHOT, x)
+#define ASM_STAT_CFG_STAT_CNT_CLR_SHOT_GET(x)\
+ FIELD_GET(ASM_STAT_CFG_STAT_CNT_CLR_SHOT, x)
+
+/* ASM:CFG:PORT_CFG */
+#define ASM_PORT_CFG(r) __REG(TARGET_ASM, 0, 1, 33280, 0, 1, 1088, 540, r, 67, 4)
+
+#define ASM_PORT_CFG_CSC_STAT_DIS BIT(12)
+#define ASM_PORT_CFG_CSC_STAT_DIS_SET(x)\
+ FIELD_PREP(ASM_PORT_CFG_CSC_STAT_DIS, x)
+#define ASM_PORT_CFG_CSC_STAT_DIS_GET(x)\
+ FIELD_GET(ASM_PORT_CFG_CSC_STAT_DIS, x)
+
+#define ASM_PORT_CFG_HIH_AFTER_PREAMBLE_ENA BIT(11)
+#define ASM_PORT_CFG_HIH_AFTER_PREAMBLE_ENA_SET(x)\
+ FIELD_PREP(ASM_PORT_CFG_HIH_AFTER_PREAMBLE_ENA, x)
+#define ASM_PORT_CFG_HIH_AFTER_PREAMBLE_ENA_GET(x)\
+ FIELD_GET(ASM_PORT_CFG_HIH_AFTER_PREAMBLE_ENA, x)
+
+#define ASM_PORT_CFG_IGN_TAXI_ABORT_ENA BIT(10)
+#define ASM_PORT_CFG_IGN_TAXI_ABORT_ENA_SET(x)\
+ FIELD_PREP(ASM_PORT_CFG_IGN_TAXI_ABORT_ENA, x)
+#define ASM_PORT_CFG_IGN_TAXI_ABORT_ENA_GET(x)\
+ FIELD_GET(ASM_PORT_CFG_IGN_TAXI_ABORT_ENA, x)
+
+#define ASM_PORT_CFG_NO_PREAMBLE_ENA BIT(9)
+#define ASM_PORT_CFG_NO_PREAMBLE_ENA_SET(x)\
+ FIELD_PREP(ASM_PORT_CFG_NO_PREAMBLE_ENA, x)
+#define ASM_PORT_CFG_NO_PREAMBLE_ENA_GET(x)\
+ FIELD_GET(ASM_PORT_CFG_NO_PREAMBLE_ENA, x)
+
+#define ASM_PORT_CFG_SKIP_PREAMBLE_ENA BIT(8)
+#define ASM_PORT_CFG_SKIP_PREAMBLE_ENA_SET(x)\
+ FIELD_PREP(ASM_PORT_CFG_SKIP_PREAMBLE_ENA, x)
+#define ASM_PORT_CFG_SKIP_PREAMBLE_ENA_GET(x)\
+ FIELD_GET(ASM_PORT_CFG_SKIP_PREAMBLE_ENA, x)
+
+#define ASM_PORT_CFG_FRM_AGING_DIS BIT(7)
+#define ASM_PORT_CFG_FRM_AGING_DIS_SET(x)\
+ FIELD_PREP(ASM_PORT_CFG_FRM_AGING_DIS, x)
+#define ASM_PORT_CFG_FRM_AGING_DIS_GET(x)\
+ FIELD_GET(ASM_PORT_CFG_FRM_AGING_DIS, x)
+
+#define ASM_PORT_CFG_PAD_ENA BIT(6)
+#define ASM_PORT_CFG_PAD_ENA_SET(x)\
+ FIELD_PREP(ASM_PORT_CFG_PAD_ENA, x)
+#define ASM_PORT_CFG_PAD_ENA_GET(x)\
+ FIELD_GET(ASM_PORT_CFG_PAD_ENA, x)
+
+#define ASM_PORT_CFG_INJ_DISCARD_CFG GENMASK(5, 4)
+#define ASM_PORT_CFG_INJ_DISCARD_CFG_SET(x)\
+ FIELD_PREP(ASM_PORT_CFG_INJ_DISCARD_CFG, x)
+#define ASM_PORT_CFG_INJ_DISCARD_CFG_GET(x)\
+ FIELD_GET(ASM_PORT_CFG_INJ_DISCARD_CFG, x)
+
+#define ASM_PORT_CFG_INJ_FORMAT_CFG GENMASK(3, 2)
+#define ASM_PORT_CFG_INJ_FORMAT_CFG_SET(x)\
+ FIELD_PREP(ASM_PORT_CFG_INJ_FORMAT_CFG, x)
+#define ASM_PORT_CFG_INJ_FORMAT_CFG_GET(x)\
+ FIELD_GET(ASM_PORT_CFG_INJ_FORMAT_CFG, x)
+
+#define ASM_PORT_CFG_VSTAX2_AWR_ENA BIT(1)
+#define ASM_PORT_CFG_VSTAX2_AWR_ENA_SET(x)\
+ FIELD_PREP(ASM_PORT_CFG_VSTAX2_AWR_ENA, x)
+#define ASM_PORT_CFG_VSTAX2_AWR_ENA_GET(x)\
+ FIELD_GET(ASM_PORT_CFG_VSTAX2_AWR_ENA, x)
+
+#define ASM_PORT_CFG_PFRM_FLUSH BIT(0)
+#define ASM_PORT_CFG_PFRM_FLUSH_SET(x)\
+ FIELD_PREP(ASM_PORT_CFG_PFRM_FLUSH, x)
+#define ASM_PORT_CFG_PFRM_FLUSH_GET(x)\
+ FIELD_GET(ASM_PORT_CFG_PFRM_FLUSH, x)
+
+/* ASM:RAM_CTRL:RAM_INIT */
+#define ASM_RAM_INIT __REG(TARGET_ASM, 0, 1, 34832, 0, 1, 4, 0, 0, 1, 4)
+
+#define ASM_RAM_INIT_RAM_INIT BIT(1)
+#define ASM_RAM_INIT_RAM_INIT_SET(x)\
+ FIELD_PREP(ASM_RAM_INIT_RAM_INIT, x)
+#define ASM_RAM_INIT_RAM_INIT_GET(x)\
+ FIELD_GET(ASM_RAM_INIT_RAM_INIT, x)
+
+#define ASM_RAM_INIT_RAM_CFG_HOOK BIT(0)
+#define ASM_RAM_INIT_RAM_CFG_HOOK_SET(x)\
+ FIELD_PREP(ASM_RAM_INIT_RAM_CFG_HOOK, x)
+#define ASM_RAM_INIT_RAM_CFG_HOOK_GET(x)\
+ FIELD_GET(ASM_RAM_INIT_RAM_CFG_HOOK, x)
+
+/* CLKGEN:LCPLL1:LCPLL1_CORE_CLK_CFG */
+#define CLKGEN_LCPLL1_CORE_CLK_CFG __REG(TARGET_CLKGEN, 0, 1, 12, 0, 1, 36, 0, 0, 1, 4)
+
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_CLK_DIV GENMASK(7, 0)
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_CLK_DIV_SET(x)\
+ FIELD_PREP(CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_CLK_DIV, x)
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_CLK_DIV_GET(x)\
+ FIELD_GET(CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_CLK_DIV, x)
+
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_PRE_DIV GENMASK(10, 8)
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_PRE_DIV_SET(x)\
+ FIELD_PREP(CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_PRE_DIV, x)
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_PRE_DIV_GET(x)\
+ FIELD_GET(CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_PRE_DIV, x)
+
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_DIR BIT(11)
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_DIR_SET(x)\
+ FIELD_PREP(CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_DIR, x)
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_DIR_GET(x)\
+ FIELD_GET(CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_DIR, x)
+
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_SEL GENMASK(13, 12)
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_SEL_SET(x)\
+ FIELD_PREP(CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_SEL, x)
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_SEL_GET(x)\
+ FIELD_GET(CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_SEL, x)
+
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_ENA BIT(14)
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_ENA_SET(x)\
+ FIELD_PREP(CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_ENA, x)
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_ENA_GET(x)\
+ FIELD_GET(CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_ROT_ENA, x)
+
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_CLK_ENA BIT(15)
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_CLK_ENA_SET(x)\
+ FIELD_PREP(CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_CLK_ENA, x)
+#define CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_CLK_ENA_GET(x)\
+ FIELD_GET(CLKGEN_LCPLL1_CORE_CLK_CFG_CORE_CLK_ENA, x)
+
+/* CPU:CPU_REGS:PROC_CTRL */
+#define CPU_PROC_CTRL __REG(TARGET_CPU, 0, 1, 0, 0, 1, 204, 176, 0, 1, 4)
+
+#define CPU_PROC_CTRL_AARCH64_MODE_ENA BIT(12)
+#define CPU_PROC_CTRL_AARCH64_MODE_ENA_SET(x)\
+ FIELD_PREP(CPU_PROC_CTRL_AARCH64_MODE_ENA, x)
+#define CPU_PROC_CTRL_AARCH64_MODE_ENA_GET(x)\
+ FIELD_GET(CPU_PROC_CTRL_AARCH64_MODE_ENA, x)
+
+#define CPU_PROC_CTRL_L2_RST_INVALIDATE_DIS BIT(11)
+#define CPU_PROC_CTRL_L2_RST_INVALIDATE_DIS_SET(x)\
+ FIELD_PREP(CPU_PROC_CTRL_L2_RST_INVALIDATE_DIS, x)
+#define CPU_PROC_CTRL_L2_RST_INVALIDATE_DIS_GET(x)\
+ FIELD_GET(CPU_PROC_CTRL_L2_RST_INVALIDATE_DIS, x)
+
+#define CPU_PROC_CTRL_L1_RST_INVALIDATE_DIS BIT(10)
+#define CPU_PROC_CTRL_L1_RST_INVALIDATE_DIS_SET(x)\
+ FIELD_PREP(CPU_PROC_CTRL_L1_RST_INVALIDATE_DIS, x)
+#define CPU_PROC_CTRL_L1_RST_INVALIDATE_DIS_GET(x)\
+ FIELD_GET(CPU_PROC_CTRL_L1_RST_INVALIDATE_DIS, x)
+
+#define CPU_PROC_CTRL_BE_EXCEP_MODE BIT(9)
+#define CPU_PROC_CTRL_BE_EXCEP_MODE_SET(x)\
+ FIELD_PREP(CPU_PROC_CTRL_BE_EXCEP_MODE, x)
+#define CPU_PROC_CTRL_BE_EXCEP_MODE_GET(x)\
+ FIELD_GET(CPU_PROC_CTRL_BE_EXCEP_MODE, x)
+
+#define CPU_PROC_CTRL_VINITHI BIT(8)
+#define CPU_PROC_CTRL_VINITHI_SET(x)\
+ FIELD_PREP(CPU_PROC_CTRL_VINITHI, x)
+#define CPU_PROC_CTRL_VINITHI_GET(x)\
+ FIELD_GET(CPU_PROC_CTRL_VINITHI, x)
+
+#define CPU_PROC_CTRL_CFGTE BIT(7)
+#define CPU_PROC_CTRL_CFGTE_SET(x)\
+ FIELD_PREP(CPU_PROC_CTRL_CFGTE, x)
+#define CPU_PROC_CTRL_CFGTE_GET(x)\
+ FIELD_GET(CPU_PROC_CTRL_CFGTE, x)
+
+#define CPU_PROC_CTRL_CP15S_DISABLE BIT(6)
+#define CPU_PROC_CTRL_CP15S_DISABLE_SET(x)\
+ FIELD_PREP(CPU_PROC_CTRL_CP15S_DISABLE, x)
+#define CPU_PROC_CTRL_CP15S_DISABLE_GET(x)\
+ FIELD_GET(CPU_PROC_CTRL_CP15S_DISABLE, x)
+
+#define CPU_PROC_CTRL_PROC_CRYPTO_DISABLE BIT(5)
+#define CPU_PROC_CTRL_PROC_CRYPTO_DISABLE_SET(x)\
+ FIELD_PREP(CPU_PROC_CTRL_PROC_CRYPTO_DISABLE, x)
+#define CPU_PROC_CTRL_PROC_CRYPTO_DISABLE_GET(x)\
+ FIELD_GET(CPU_PROC_CTRL_PROC_CRYPTO_DISABLE, x)
+
+#define CPU_PROC_CTRL_ACP_CACHE_FORCE_ENA BIT(4)
+#define CPU_PROC_CTRL_ACP_CACHE_FORCE_ENA_SET(x)\
+ FIELD_PREP(CPU_PROC_CTRL_ACP_CACHE_FORCE_ENA, x)
+#define CPU_PROC_CTRL_ACP_CACHE_FORCE_ENA_GET(x)\
+ FIELD_GET(CPU_PROC_CTRL_ACP_CACHE_FORCE_ENA, x)
+
+#define CPU_PROC_CTRL_ACP_AWCACHE BIT(3)
+#define CPU_PROC_CTRL_ACP_AWCACHE_SET(x)\
+ FIELD_PREP(CPU_PROC_CTRL_ACP_AWCACHE, x)
+#define CPU_PROC_CTRL_ACP_AWCACHE_GET(x)\
+ FIELD_GET(CPU_PROC_CTRL_ACP_AWCACHE, x)
+
+#define CPU_PROC_CTRL_ACP_ARCACHE BIT(2)
+#define CPU_PROC_CTRL_ACP_ARCACHE_SET(x)\
+ FIELD_PREP(CPU_PROC_CTRL_ACP_ARCACHE, x)
+#define CPU_PROC_CTRL_ACP_ARCACHE_GET(x)\
+ FIELD_GET(CPU_PROC_CTRL_ACP_ARCACHE, x)
+
+#define CPU_PROC_CTRL_L2_FLUSH_REQ BIT(1)
+#define CPU_PROC_CTRL_L2_FLUSH_REQ_SET(x)\
+ FIELD_PREP(CPU_PROC_CTRL_L2_FLUSH_REQ, x)
+#define CPU_PROC_CTRL_L2_FLUSH_REQ_GET(x)\
+ FIELD_GET(CPU_PROC_CTRL_L2_FLUSH_REQ, x)
+
+#define CPU_PROC_CTRL_ACP_DISABLE BIT(0)
+#define CPU_PROC_CTRL_ACP_DISABLE_SET(x)\
+ FIELD_PREP(CPU_PROC_CTRL_ACP_DISABLE, x)
+#define CPU_PROC_CTRL_ACP_DISABLE_GET(x)\
+ FIELD_GET(CPU_PROC_CTRL_ACP_DISABLE, x)
+
+/* DEV10G:MAC_CFG_STATUS:MAC_ENA_CFG */
+#define DEV10G_MAC_ENA_CFG(t) __REG(TARGET_DEV10G, t, 12, 0, 0, 1, 60, 0, 0, 1, 4)
+
+#define DEV10G_MAC_ENA_CFG_RX_ENA BIT(4)
+#define DEV10G_MAC_ENA_CFG_RX_ENA_SET(x)\
+ FIELD_PREP(DEV10G_MAC_ENA_CFG_RX_ENA, x)
+#define DEV10G_MAC_ENA_CFG_RX_ENA_GET(x)\
+ FIELD_GET(DEV10G_MAC_ENA_CFG_RX_ENA, x)
+
+#define DEV10G_MAC_ENA_CFG_TX_ENA BIT(0)
+#define DEV10G_MAC_ENA_CFG_TX_ENA_SET(x)\
+ FIELD_PREP(DEV10G_MAC_ENA_CFG_TX_ENA, x)
+#define DEV10G_MAC_ENA_CFG_TX_ENA_GET(x)\
+ FIELD_GET(DEV10G_MAC_ENA_CFG_TX_ENA, x)
+
+/* DEV10G:MAC_CFG_STATUS:MAC_MAXLEN_CFG */
+#define DEV10G_MAC_MAXLEN_CFG(t) __REG(TARGET_DEV10G, t, 12, 0, 0, 1, 60, 8, 0, 1, 4)
+
+#define DEV10G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK BIT(16)
+#define DEV10G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK_SET(x)\
+ FIELD_PREP(DEV10G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK, x)
+#define DEV10G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK_GET(x)\
+ FIELD_GET(DEV10G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK, x)
+
+#define DEV10G_MAC_MAXLEN_CFG_MAX_LEN GENMASK(15, 0)
+#define DEV10G_MAC_MAXLEN_CFG_MAX_LEN_SET(x)\
+ FIELD_PREP(DEV10G_MAC_MAXLEN_CFG_MAX_LEN, x)
+#define DEV10G_MAC_MAXLEN_CFG_MAX_LEN_GET(x)\
+ FIELD_GET(DEV10G_MAC_MAXLEN_CFG_MAX_LEN, x)
+
+/* DEV10G:MAC_CFG_STATUS:MAC_NUM_TAGS_CFG */
+#define DEV10G_MAC_NUM_TAGS_CFG(t) __REG(TARGET_DEV10G, t, 12, 0, 0, 1, 60, 12, 0, 1, 4)
+
+#define DEV10G_MAC_NUM_TAGS_CFG_NUM_TAGS GENMASK(1, 0)
+#define DEV10G_MAC_NUM_TAGS_CFG_NUM_TAGS_SET(x)\
+ FIELD_PREP(DEV10G_MAC_NUM_TAGS_CFG_NUM_TAGS, x)
+#define DEV10G_MAC_NUM_TAGS_CFG_NUM_TAGS_GET(x)\
+ FIELD_GET(DEV10G_MAC_NUM_TAGS_CFG_NUM_TAGS, x)
+
+/* DEV10G:MAC_CFG_STATUS:MAC_TAGS_CFG */
+#define DEV10G_MAC_TAGS_CFG(t, r) __REG(TARGET_DEV10G, t, 12, 0, 0, 1, 60, 16, r, 3, 4)
+
+#define DEV10G_MAC_TAGS_CFG_TAG_ID GENMASK(31, 16)
+#define DEV10G_MAC_TAGS_CFG_TAG_ID_SET(x)\
+ FIELD_PREP(DEV10G_MAC_TAGS_CFG_TAG_ID, x)
+#define DEV10G_MAC_TAGS_CFG_TAG_ID_GET(x)\
+ FIELD_GET(DEV10G_MAC_TAGS_CFG_TAG_ID, x)
+
+#define DEV10G_MAC_TAGS_CFG_TAG_ENA BIT(4)
+#define DEV10G_MAC_TAGS_CFG_TAG_ENA_SET(x)\
+ FIELD_PREP(DEV10G_MAC_TAGS_CFG_TAG_ENA, x)
+#define DEV10G_MAC_TAGS_CFG_TAG_ENA_GET(x)\
+ FIELD_GET(DEV10G_MAC_TAGS_CFG_TAG_ENA, x)
+
+/* DEV10G:MAC_CFG_STATUS:MAC_ADV_CHK_CFG */
+#define DEV10G_MAC_ADV_CHK_CFG(t) __REG(TARGET_DEV10G, t, 12, 0, 0, 1, 60, 28, 0, 1, 4)
+
+#define DEV10G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA BIT(24)
+#define DEV10G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA_SET(x)\
+ FIELD_PREP(DEV10G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA, x)
+#define DEV10G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA_GET(x)\
+ FIELD_GET(DEV10G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA, x)
+
+#define DEV10G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA BIT(20)
+#define DEV10G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA_SET(x)\
+ FIELD_PREP(DEV10G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA, x)
+#define DEV10G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA_GET(x)\
+ FIELD_GET(DEV10G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA, x)
+
+#define DEV10G_MAC_ADV_CHK_CFG_SFD_CHK_ENA BIT(16)
+#define DEV10G_MAC_ADV_CHK_CFG_SFD_CHK_ENA_SET(x)\
+ FIELD_PREP(DEV10G_MAC_ADV_CHK_CFG_SFD_CHK_ENA, x)
+#define DEV10G_MAC_ADV_CHK_CFG_SFD_CHK_ENA_GET(x)\
+ FIELD_GET(DEV10G_MAC_ADV_CHK_CFG_SFD_CHK_ENA, x)
+
+#define DEV10G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS BIT(12)
+#define DEV10G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS_SET(x)\
+ FIELD_PREP(DEV10G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS, x)
+#define DEV10G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS_GET(x)\
+ FIELD_GET(DEV10G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS, x)
+
+#define DEV10G_MAC_ADV_CHK_CFG_PRM_CHK_ENA BIT(8)
+#define DEV10G_MAC_ADV_CHK_CFG_PRM_CHK_ENA_SET(x)\
+ FIELD_PREP(DEV10G_MAC_ADV_CHK_CFG_PRM_CHK_ENA, x)
+#define DEV10G_MAC_ADV_CHK_CFG_PRM_CHK_ENA_GET(x)\
+ FIELD_GET(DEV10G_MAC_ADV_CHK_CFG_PRM_CHK_ENA, x)
+
+#define DEV10G_MAC_ADV_CHK_CFG_OOR_ERR_ENA BIT(4)
+#define DEV10G_MAC_ADV_CHK_CFG_OOR_ERR_ENA_SET(x)\
+ FIELD_PREP(DEV10G_MAC_ADV_CHK_CFG_OOR_ERR_ENA, x)
+#define DEV10G_MAC_ADV_CHK_CFG_OOR_ERR_ENA_GET(x)\
+ FIELD_GET(DEV10G_MAC_ADV_CHK_CFG_OOR_ERR_ENA, x)
+
+#define DEV10G_MAC_ADV_CHK_CFG_INR_ERR_ENA BIT(0)
+#define DEV10G_MAC_ADV_CHK_CFG_INR_ERR_ENA_SET(x)\
+ FIELD_PREP(DEV10G_MAC_ADV_CHK_CFG_INR_ERR_ENA, x)
+#define DEV10G_MAC_ADV_CHK_CFG_INR_ERR_ENA_GET(x)\
+ FIELD_GET(DEV10G_MAC_ADV_CHK_CFG_INR_ERR_ENA, x)
+
+/* DEV10G:MAC_CFG_STATUS:MAC_TX_MONITOR_STICKY */
+#define DEV10G_MAC_TX_MONITOR_STICKY(t) __REG(TARGET_DEV10G, t, 12, 0, 0, 1, 60, 48, 0, 1, 4)
+
+#define DEV10G_MAC_TX_MONITOR_STICKY_LOCAL_ERR_STATE_STICKY BIT(4)
+#define DEV10G_MAC_TX_MONITOR_STICKY_LOCAL_ERR_STATE_STICKY_SET(x)\
+ FIELD_PREP(DEV10G_MAC_TX_MONITOR_STICKY_LOCAL_ERR_STATE_STICKY, x)
+#define DEV10G_MAC_TX_MONITOR_STICKY_LOCAL_ERR_STATE_STICKY_GET(x)\
+ FIELD_GET(DEV10G_MAC_TX_MONITOR_STICKY_LOCAL_ERR_STATE_STICKY, x)
+
+#define DEV10G_MAC_TX_MONITOR_STICKY_REMOTE_ERR_STATE_STICKY BIT(3)
+#define DEV10G_MAC_TX_MONITOR_STICKY_REMOTE_ERR_STATE_STICKY_SET(x)\
+ FIELD_PREP(DEV10G_MAC_TX_MONITOR_STICKY_REMOTE_ERR_STATE_STICKY, x)
+#define DEV10G_MAC_TX_MONITOR_STICKY_REMOTE_ERR_STATE_STICKY_GET(x)\
+ FIELD_GET(DEV10G_MAC_TX_MONITOR_STICKY_REMOTE_ERR_STATE_STICKY, x)
+
+#define DEV10G_MAC_TX_MONITOR_STICKY_LINK_INTERRUPTION_STATE_STICKY BIT(2)
+#define DEV10G_MAC_TX_MONITOR_STICKY_LINK_INTERRUPTION_STATE_STICKY_SET(x)\
+ FIELD_PREP(DEV10G_MAC_TX_MONITOR_STICKY_LINK_INTERRUPTION_STATE_STICKY, x)
+#define DEV10G_MAC_TX_MONITOR_STICKY_LINK_INTERRUPTION_STATE_STICKY_GET(x)\
+ FIELD_GET(DEV10G_MAC_TX_MONITOR_STICKY_LINK_INTERRUPTION_STATE_STICKY, x)
+
+#define DEV10G_MAC_TX_MONITOR_STICKY_IDLE_STATE_STICKY BIT(1)
+#define DEV10G_MAC_TX_MONITOR_STICKY_IDLE_STATE_STICKY_SET(x)\
+ FIELD_PREP(DEV10G_MAC_TX_MONITOR_STICKY_IDLE_STATE_STICKY, x)
+#define DEV10G_MAC_TX_MONITOR_STICKY_IDLE_STATE_STICKY_GET(x)\
+ FIELD_GET(DEV10G_MAC_TX_MONITOR_STICKY_IDLE_STATE_STICKY, x)
+
+#define DEV10G_MAC_TX_MONITOR_STICKY_DIS_STATE_STICKY BIT(0)
+#define DEV10G_MAC_TX_MONITOR_STICKY_DIS_STATE_STICKY_SET(x)\
+ FIELD_PREP(DEV10G_MAC_TX_MONITOR_STICKY_DIS_STATE_STICKY, x)
+#define DEV10G_MAC_TX_MONITOR_STICKY_DIS_STATE_STICKY_GET(x)\
+ FIELD_GET(DEV10G_MAC_TX_MONITOR_STICKY_DIS_STATE_STICKY, x)
+
+/* DEV10G:DEV_CFG_STATUS:DEV_RST_CTRL */
+#define DEV10G_DEV_RST_CTRL(t) __REG(TARGET_DEV10G, t, 12, 436, 0, 1, 52, 0, 0, 1, 4)
+
+#define DEV10G_DEV_RST_CTRL_PARDET_MODE_ENA BIT(28)
+#define DEV10G_DEV_RST_CTRL_PARDET_MODE_ENA_SET(x)\
+ FIELD_PREP(DEV10G_DEV_RST_CTRL_PARDET_MODE_ENA, x)
+#define DEV10G_DEV_RST_CTRL_PARDET_MODE_ENA_GET(x)\
+ FIELD_GET(DEV10G_DEV_RST_CTRL_PARDET_MODE_ENA, x)
+
+#define DEV10G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS BIT(27)
+#define DEV10G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS_SET(x)\
+ FIELD_PREP(DEV10G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS, x)
+#define DEV10G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS_GET(x)\
+ FIELD_GET(DEV10G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS, x)
+
+#define DEV10G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS GENMASK(26, 25)
+#define DEV10G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS_SET(x)\
+ FIELD_PREP(DEV10G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS, x)
+#define DEV10G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS_GET(x)\
+ FIELD_GET(DEV10G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS, x)
+
+#define DEV10G_DEV_RST_CTRL_SERDES_SPEED_SEL GENMASK(24, 23)
+#define DEV10G_DEV_RST_CTRL_SERDES_SPEED_SEL_SET(x)\
+ FIELD_PREP(DEV10G_DEV_RST_CTRL_SERDES_SPEED_SEL, x)
+#define DEV10G_DEV_RST_CTRL_SERDES_SPEED_SEL_GET(x)\
+ FIELD_GET(DEV10G_DEV_RST_CTRL_SERDES_SPEED_SEL, x)
+
+#define DEV10G_DEV_RST_CTRL_SPEED_SEL GENMASK(22, 20)
+#define DEV10G_DEV_RST_CTRL_SPEED_SEL_SET(x)\
+ FIELD_PREP(DEV10G_DEV_RST_CTRL_SPEED_SEL, x)
+#define DEV10G_DEV_RST_CTRL_SPEED_SEL_GET(x)\
+ FIELD_GET(DEV10G_DEV_RST_CTRL_SPEED_SEL, x)
+
+#define DEV10G_DEV_RST_CTRL_PCS_TX_RST BIT(12)
+#define DEV10G_DEV_RST_CTRL_PCS_TX_RST_SET(x)\
+ FIELD_PREP(DEV10G_DEV_RST_CTRL_PCS_TX_RST, x)
+#define DEV10G_DEV_RST_CTRL_PCS_TX_RST_GET(x)\
+ FIELD_GET(DEV10G_DEV_RST_CTRL_PCS_TX_RST, x)
+
+#define DEV10G_DEV_RST_CTRL_PCS_RX_RST BIT(8)
+#define DEV10G_DEV_RST_CTRL_PCS_RX_RST_SET(x)\
+ FIELD_PREP(DEV10G_DEV_RST_CTRL_PCS_RX_RST, x)
+#define DEV10G_DEV_RST_CTRL_PCS_RX_RST_GET(x)\
+ FIELD_GET(DEV10G_DEV_RST_CTRL_PCS_RX_RST, x)
+
+#define DEV10G_DEV_RST_CTRL_MAC_TX_RST BIT(4)
+#define DEV10G_DEV_RST_CTRL_MAC_TX_RST_SET(x)\
+ FIELD_PREP(DEV10G_DEV_RST_CTRL_MAC_TX_RST, x)
+#define DEV10G_DEV_RST_CTRL_MAC_TX_RST_GET(x)\
+ FIELD_GET(DEV10G_DEV_RST_CTRL_MAC_TX_RST, x)
+
+#define DEV10G_DEV_RST_CTRL_MAC_RX_RST BIT(0)
+#define DEV10G_DEV_RST_CTRL_MAC_RX_RST_SET(x)\
+ FIELD_PREP(DEV10G_DEV_RST_CTRL_MAC_RX_RST, x)
+#define DEV10G_DEV_RST_CTRL_MAC_RX_RST_GET(x)\
+ FIELD_GET(DEV10G_DEV_RST_CTRL_MAC_RX_RST, x)
+
+/* DEV10G:PCS25G_CFG_STATUS:PCS25G_CFG */
+#define DEV10G_PCS25G_CFG(t) __REG(TARGET_DEV10G, t, 12, 488, 0, 1, 32, 0, 0, 1, 4)
+
+#define DEV10G_PCS25G_CFG_PCS25G_ENA BIT(0)
+#define DEV10G_PCS25G_CFG_PCS25G_ENA_SET(x)\
+ FIELD_PREP(DEV10G_PCS25G_CFG_PCS25G_ENA, x)
+#define DEV10G_PCS25G_CFG_PCS25G_ENA_GET(x)\
+ FIELD_GET(DEV10G_PCS25G_CFG_PCS25G_ENA, x)
+
+/* DEV10G:MAC_CFG_STATUS:MAC_ENA_CFG */
+#define DEV25G_MAC_ENA_CFG(t) __REG(TARGET_DEV25G, t, 8, 0, 0, 1, 60, 0, 0, 1, 4)
+
+#define DEV25G_MAC_ENA_CFG_RX_ENA BIT(4)
+#define DEV25G_MAC_ENA_CFG_RX_ENA_SET(x)\
+ FIELD_PREP(DEV25G_MAC_ENA_CFG_RX_ENA, x)
+#define DEV25G_MAC_ENA_CFG_RX_ENA_GET(x)\
+ FIELD_GET(DEV25G_MAC_ENA_CFG_RX_ENA, x)
+
+#define DEV25G_MAC_ENA_CFG_TX_ENA BIT(0)
+#define DEV25G_MAC_ENA_CFG_TX_ENA_SET(x)\
+ FIELD_PREP(DEV25G_MAC_ENA_CFG_TX_ENA, x)
+#define DEV25G_MAC_ENA_CFG_TX_ENA_GET(x)\
+ FIELD_GET(DEV25G_MAC_ENA_CFG_TX_ENA, x)
+
+/* DEV10G:MAC_CFG_STATUS:MAC_MAXLEN_CFG */
+#define DEV25G_MAC_MAXLEN_CFG(t) __REG(TARGET_DEV25G, t, 8, 0, 0, 1, 60, 8, 0, 1, 4)
+
+#define DEV25G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK BIT(16)
+#define DEV25G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK_SET(x)\
+ FIELD_PREP(DEV25G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK, x)
+#define DEV25G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK_GET(x)\
+ FIELD_GET(DEV25G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK, x)
+
+#define DEV25G_MAC_MAXLEN_CFG_MAX_LEN GENMASK(15, 0)
+#define DEV25G_MAC_MAXLEN_CFG_MAX_LEN_SET(x)\
+ FIELD_PREP(DEV25G_MAC_MAXLEN_CFG_MAX_LEN, x)
+#define DEV25G_MAC_MAXLEN_CFG_MAX_LEN_GET(x)\
+ FIELD_GET(DEV25G_MAC_MAXLEN_CFG_MAX_LEN, x)
+
+/* DEV10G:MAC_CFG_STATUS:MAC_ADV_CHK_CFG */
+#define DEV25G_MAC_ADV_CHK_CFG(t) __REG(TARGET_DEV25G, t, 8, 0, 0, 1, 60, 28, 0, 1, 4)
+
+#define DEV25G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA BIT(24)
+#define DEV25G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA_SET(x)\
+ FIELD_PREP(DEV25G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA, x)
+#define DEV25G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA_GET(x)\
+ FIELD_GET(DEV25G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA, x)
+
+#define DEV25G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA BIT(20)
+#define DEV25G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA_SET(x)\
+ FIELD_PREP(DEV25G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA, x)
+#define DEV25G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA_GET(x)\
+ FIELD_GET(DEV25G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA, x)
+
+#define DEV25G_MAC_ADV_CHK_CFG_SFD_CHK_ENA BIT(16)
+#define DEV25G_MAC_ADV_CHK_CFG_SFD_CHK_ENA_SET(x)\
+ FIELD_PREP(DEV25G_MAC_ADV_CHK_CFG_SFD_CHK_ENA, x)
+#define DEV25G_MAC_ADV_CHK_CFG_SFD_CHK_ENA_GET(x)\
+ FIELD_GET(DEV25G_MAC_ADV_CHK_CFG_SFD_CHK_ENA, x)
+
+#define DEV25G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS BIT(12)
+#define DEV25G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS_SET(x)\
+ FIELD_PREP(DEV25G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS, x)
+#define DEV25G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS_GET(x)\
+ FIELD_GET(DEV25G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS, x)
+
+#define DEV25G_MAC_ADV_CHK_CFG_PRM_CHK_ENA BIT(8)
+#define DEV25G_MAC_ADV_CHK_CFG_PRM_CHK_ENA_SET(x)\
+ FIELD_PREP(DEV25G_MAC_ADV_CHK_CFG_PRM_CHK_ENA, x)
+#define DEV25G_MAC_ADV_CHK_CFG_PRM_CHK_ENA_GET(x)\
+ FIELD_GET(DEV25G_MAC_ADV_CHK_CFG_PRM_CHK_ENA, x)
+
+#define DEV25G_MAC_ADV_CHK_CFG_OOR_ERR_ENA BIT(4)
+#define DEV25G_MAC_ADV_CHK_CFG_OOR_ERR_ENA_SET(x)\
+ FIELD_PREP(DEV25G_MAC_ADV_CHK_CFG_OOR_ERR_ENA, x)
+#define DEV25G_MAC_ADV_CHK_CFG_OOR_ERR_ENA_GET(x)\
+ FIELD_GET(DEV25G_MAC_ADV_CHK_CFG_OOR_ERR_ENA, x)
+
+#define DEV25G_MAC_ADV_CHK_CFG_INR_ERR_ENA BIT(0)
+#define DEV25G_MAC_ADV_CHK_CFG_INR_ERR_ENA_SET(x)\
+ FIELD_PREP(DEV25G_MAC_ADV_CHK_CFG_INR_ERR_ENA, x)
+#define DEV25G_MAC_ADV_CHK_CFG_INR_ERR_ENA_GET(x)\
+ FIELD_GET(DEV25G_MAC_ADV_CHK_CFG_INR_ERR_ENA, x)
+
+/* DEV10G:DEV_CFG_STATUS:DEV_RST_CTRL */
+#define DEV25G_DEV_RST_CTRL(t) __REG(TARGET_DEV25G, t, 8, 436, 0, 1, 52, 0, 0, 1, 4)
+
+#define DEV25G_DEV_RST_CTRL_PARDET_MODE_ENA BIT(28)
+#define DEV25G_DEV_RST_CTRL_PARDET_MODE_ENA_SET(x)\
+ FIELD_PREP(DEV25G_DEV_RST_CTRL_PARDET_MODE_ENA, x)
+#define DEV25G_DEV_RST_CTRL_PARDET_MODE_ENA_GET(x)\
+ FIELD_GET(DEV25G_DEV_RST_CTRL_PARDET_MODE_ENA, x)
+
+#define DEV25G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS BIT(27)
+#define DEV25G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS_SET(x)\
+ FIELD_PREP(DEV25G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS, x)
+#define DEV25G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS_GET(x)\
+ FIELD_GET(DEV25G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS, x)
+
+#define DEV25G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS GENMASK(26, 25)
+#define DEV25G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS_SET(x)\
+ FIELD_PREP(DEV25G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS, x)
+#define DEV25G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS_GET(x)\
+ FIELD_GET(DEV25G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS, x)
+
+#define DEV25G_DEV_RST_CTRL_SERDES_SPEED_SEL GENMASK(24, 23)
+#define DEV25G_DEV_RST_CTRL_SERDES_SPEED_SEL_SET(x)\
+ FIELD_PREP(DEV25G_DEV_RST_CTRL_SERDES_SPEED_SEL, x)
+#define DEV25G_DEV_RST_CTRL_SERDES_SPEED_SEL_GET(x)\
+ FIELD_GET(DEV25G_DEV_RST_CTRL_SERDES_SPEED_SEL, x)
+
+#define DEV25G_DEV_RST_CTRL_SPEED_SEL GENMASK(22, 20)
+#define DEV25G_DEV_RST_CTRL_SPEED_SEL_SET(x)\
+ FIELD_PREP(DEV25G_DEV_RST_CTRL_SPEED_SEL, x)
+#define DEV25G_DEV_RST_CTRL_SPEED_SEL_GET(x)\
+ FIELD_GET(DEV25G_DEV_RST_CTRL_SPEED_SEL, x)
+
+#define DEV25G_DEV_RST_CTRL_PCS_TX_RST BIT(12)
+#define DEV25G_DEV_RST_CTRL_PCS_TX_RST_SET(x)\
+ FIELD_PREP(DEV25G_DEV_RST_CTRL_PCS_TX_RST, x)
+#define DEV25G_DEV_RST_CTRL_PCS_TX_RST_GET(x)\
+ FIELD_GET(DEV25G_DEV_RST_CTRL_PCS_TX_RST, x)
+
+#define DEV25G_DEV_RST_CTRL_PCS_RX_RST BIT(8)
+#define DEV25G_DEV_RST_CTRL_PCS_RX_RST_SET(x)\
+ FIELD_PREP(DEV25G_DEV_RST_CTRL_PCS_RX_RST, x)
+#define DEV25G_DEV_RST_CTRL_PCS_RX_RST_GET(x)\
+ FIELD_GET(DEV25G_DEV_RST_CTRL_PCS_RX_RST, x)
+
+#define DEV25G_DEV_RST_CTRL_MAC_TX_RST BIT(4)
+#define DEV25G_DEV_RST_CTRL_MAC_TX_RST_SET(x)\
+ FIELD_PREP(DEV25G_DEV_RST_CTRL_MAC_TX_RST, x)
+#define DEV25G_DEV_RST_CTRL_MAC_TX_RST_GET(x)\
+ FIELD_GET(DEV25G_DEV_RST_CTRL_MAC_TX_RST, x)
+
+#define DEV25G_DEV_RST_CTRL_MAC_RX_RST BIT(0)
+#define DEV25G_DEV_RST_CTRL_MAC_RX_RST_SET(x)\
+ FIELD_PREP(DEV25G_DEV_RST_CTRL_MAC_RX_RST, x)
+#define DEV25G_DEV_RST_CTRL_MAC_RX_RST_GET(x)\
+ FIELD_GET(DEV25G_DEV_RST_CTRL_MAC_RX_RST, x)
+
+/* DEV10G:PCS25G_CFG_STATUS:PCS25G_CFG */
+#define DEV25G_PCS25G_CFG(t) __REG(TARGET_DEV25G, t, 8, 488, 0, 1, 32, 0, 0, 1, 4)
+
+#define DEV25G_PCS25G_CFG_PCS25G_ENA BIT(0)
+#define DEV25G_PCS25G_CFG_PCS25G_ENA_SET(x)\
+ FIELD_PREP(DEV25G_PCS25G_CFG_PCS25G_ENA, x)
+#define DEV25G_PCS25G_CFG_PCS25G_ENA_GET(x)\
+ FIELD_GET(DEV25G_PCS25G_CFG_PCS25G_ENA, x)
+
+/* DEV10G:PCS25G_CFG_STATUS:PCS25G_SD_CFG */
+#define DEV25G_PCS25G_SD_CFG(t) __REG(TARGET_DEV25G, t, 8, 488, 0, 1, 32, 4, 0, 1, 4)
+
+#define DEV25G_PCS25G_SD_CFG_SD_SEL BIT(8)
+#define DEV25G_PCS25G_SD_CFG_SD_SEL_SET(x)\
+ FIELD_PREP(DEV25G_PCS25G_SD_CFG_SD_SEL, x)
+#define DEV25G_PCS25G_SD_CFG_SD_SEL_GET(x)\
+ FIELD_GET(DEV25G_PCS25G_SD_CFG_SD_SEL, x)
+
+#define DEV25G_PCS25G_SD_CFG_SD_POL BIT(4)
+#define DEV25G_PCS25G_SD_CFG_SD_POL_SET(x)\
+ FIELD_PREP(DEV25G_PCS25G_SD_CFG_SD_POL, x)
+#define DEV25G_PCS25G_SD_CFG_SD_POL_GET(x)\
+ FIELD_GET(DEV25G_PCS25G_SD_CFG_SD_POL, x)
+
+#define DEV25G_PCS25G_SD_CFG_SD_ENA BIT(0)
+#define DEV25G_PCS25G_SD_CFG_SD_ENA_SET(x)\
+ FIELD_PREP(DEV25G_PCS25G_SD_CFG_SD_ENA, x)
+#define DEV25G_PCS25G_SD_CFG_SD_ENA_GET(x)\
+ FIELD_GET(DEV25G_PCS25G_SD_CFG_SD_ENA, x)
+
+/* DEV1G:DEV_CFG_STATUS:DEV_RST_CTRL */
+#define DEV2G5_DEV_RST_CTRL(t) __REG(TARGET_DEV2G5, t, 65, 0, 0, 1, 36, 0, 0, 1, 4)
+
+#define DEV2G5_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS BIT(23)
+#define DEV2G5_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS_SET(x)\
+ FIELD_PREP(DEV2G5_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS, x)
+#define DEV2G5_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS_GET(x)\
+ FIELD_GET(DEV2G5_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS, x)
+
+#define DEV2G5_DEV_RST_CTRL_SPEED_SEL GENMASK(22, 20)
+#define DEV2G5_DEV_RST_CTRL_SPEED_SEL_SET(x)\
+ FIELD_PREP(DEV2G5_DEV_RST_CTRL_SPEED_SEL, x)
+#define DEV2G5_DEV_RST_CTRL_SPEED_SEL_GET(x)\
+ FIELD_GET(DEV2G5_DEV_RST_CTRL_SPEED_SEL, x)
+
+#define DEV2G5_DEV_RST_CTRL_USX_PCS_TX_RST BIT(17)
+#define DEV2G5_DEV_RST_CTRL_USX_PCS_TX_RST_SET(x)\
+ FIELD_PREP(DEV2G5_DEV_RST_CTRL_USX_PCS_TX_RST, x)
+#define DEV2G5_DEV_RST_CTRL_USX_PCS_TX_RST_GET(x)\
+ FIELD_GET(DEV2G5_DEV_RST_CTRL_USX_PCS_TX_RST, x)
+
+#define DEV2G5_DEV_RST_CTRL_USX_PCS_RX_RST BIT(16)
+#define DEV2G5_DEV_RST_CTRL_USX_PCS_RX_RST_SET(x)\
+ FIELD_PREP(DEV2G5_DEV_RST_CTRL_USX_PCS_RX_RST, x)
+#define DEV2G5_DEV_RST_CTRL_USX_PCS_RX_RST_GET(x)\
+ FIELD_GET(DEV2G5_DEV_RST_CTRL_USX_PCS_RX_RST, x)
+
+#define DEV2G5_DEV_RST_CTRL_PCS_TX_RST BIT(12)
+#define DEV2G5_DEV_RST_CTRL_PCS_TX_RST_SET(x)\
+ FIELD_PREP(DEV2G5_DEV_RST_CTRL_PCS_TX_RST, x)
+#define DEV2G5_DEV_RST_CTRL_PCS_TX_RST_GET(x)\
+ FIELD_GET(DEV2G5_DEV_RST_CTRL_PCS_TX_RST, x)
+
+#define DEV2G5_DEV_RST_CTRL_PCS_RX_RST BIT(8)
+#define DEV2G5_DEV_RST_CTRL_PCS_RX_RST_SET(x)\
+ FIELD_PREP(DEV2G5_DEV_RST_CTRL_PCS_RX_RST, x)
+#define DEV2G5_DEV_RST_CTRL_PCS_RX_RST_GET(x)\
+ FIELD_GET(DEV2G5_DEV_RST_CTRL_PCS_RX_RST, x)
+
+#define DEV2G5_DEV_RST_CTRL_MAC_TX_RST BIT(4)
+#define DEV2G5_DEV_RST_CTRL_MAC_TX_RST_SET(x)\
+ FIELD_PREP(DEV2G5_DEV_RST_CTRL_MAC_TX_RST, x)
+#define DEV2G5_DEV_RST_CTRL_MAC_TX_RST_GET(x)\
+ FIELD_GET(DEV2G5_DEV_RST_CTRL_MAC_TX_RST, x)
+
+#define DEV2G5_DEV_RST_CTRL_MAC_RX_RST BIT(0)
+#define DEV2G5_DEV_RST_CTRL_MAC_RX_RST_SET(x)\
+ FIELD_PREP(DEV2G5_DEV_RST_CTRL_MAC_RX_RST, x)
+#define DEV2G5_DEV_RST_CTRL_MAC_RX_RST_GET(x)\
+ FIELD_GET(DEV2G5_DEV_RST_CTRL_MAC_RX_RST, x)
+
+/* DEV1G:MAC_CFG_STATUS:MAC_ENA_CFG */
+#define DEV2G5_MAC_ENA_CFG(t) __REG(TARGET_DEV2G5, t, 65, 52, 0, 1, 36, 0, 0, 1, 4)
+
+#define DEV2G5_MAC_ENA_CFG_RX_ENA BIT(4)
+#define DEV2G5_MAC_ENA_CFG_RX_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_ENA_CFG_RX_ENA, x)
+#define DEV2G5_MAC_ENA_CFG_RX_ENA_GET(x)\
+ FIELD_GET(DEV2G5_MAC_ENA_CFG_RX_ENA, x)
+
+#define DEV2G5_MAC_ENA_CFG_TX_ENA BIT(0)
+#define DEV2G5_MAC_ENA_CFG_TX_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_ENA_CFG_TX_ENA, x)
+#define DEV2G5_MAC_ENA_CFG_TX_ENA_GET(x)\
+ FIELD_GET(DEV2G5_MAC_ENA_CFG_TX_ENA, x)
+
+/* DEV1G:MAC_CFG_STATUS:MAC_MODE_CFG */
+#define DEV2G5_MAC_MODE_CFG(t) __REG(TARGET_DEV2G5, t, 65, 52, 0, 1, 36, 4, 0, 1, 4)
+
+#define DEV2G5_MAC_MODE_CFG_FC_WORD_SYNC_ENA BIT(8)
+#define DEV2G5_MAC_MODE_CFG_FC_WORD_SYNC_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_MODE_CFG_FC_WORD_SYNC_ENA, x)
+#define DEV2G5_MAC_MODE_CFG_FC_WORD_SYNC_ENA_GET(x)\
+ FIELD_GET(DEV2G5_MAC_MODE_CFG_FC_WORD_SYNC_ENA, x)
+
+#define DEV2G5_MAC_MODE_CFG_GIGA_MODE_ENA BIT(4)
+#define DEV2G5_MAC_MODE_CFG_GIGA_MODE_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_MODE_CFG_GIGA_MODE_ENA, x)
+#define DEV2G5_MAC_MODE_CFG_GIGA_MODE_ENA_GET(x)\
+ FIELD_GET(DEV2G5_MAC_MODE_CFG_GIGA_MODE_ENA, x)
+
+#define DEV2G5_MAC_MODE_CFG_FDX_ENA BIT(0)
+#define DEV2G5_MAC_MODE_CFG_FDX_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_MODE_CFG_FDX_ENA, x)
+#define DEV2G5_MAC_MODE_CFG_FDX_ENA_GET(x)\
+ FIELD_GET(DEV2G5_MAC_MODE_CFG_FDX_ENA, x)
+
+/* DEV1G:MAC_CFG_STATUS:MAC_MAXLEN_CFG */
+#define DEV2G5_MAC_MAXLEN_CFG(t) __REG(TARGET_DEV2G5, t, 65, 52, 0, 1, 36, 8, 0, 1, 4)
+
+#define DEV2G5_MAC_MAXLEN_CFG_MAX_LEN GENMASK(15, 0)
+#define DEV2G5_MAC_MAXLEN_CFG_MAX_LEN_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_MAXLEN_CFG_MAX_LEN, x)
+#define DEV2G5_MAC_MAXLEN_CFG_MAX_LEN_GET(x)\
+ FIELD_GET(DEV2G5_MAC_MAXLEN_CFG_MAX_LEN, x)
+
+/* DEV1G:MAC_CFG_STATUS:MAC_TAGS_CFG */
+#define DEV2G5_MAC_TAGS_CFG(t) __REG(TARGET_DEV2G5, t, 65, 52, 0, 1, 36, 12, 0, 1, 4)
+
+#define DEV2G5_MAC_TAGS_CFG_TAG_ID GENMASK(31, 16)
+#define DEV2G5_MAC_TAGS_CFG_TAG_ID_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_TAGS_CFG_TAG_ID, x)
+#define DEV2G5_MAC_TAGS_CFG_TAG_ID_GET(x)\
+ FIELD_GET(DEV2G5_MAC_TAGS_CFG_TAG_ID, x)
+
+#define DEV2G5_MAC_TAGS_CFG_VLAN_LEN_AWR_ENA BIT(3)
+#define DEV2G5_MAC_TAGS_CFG_VLAN_LEN_AWR_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_TAGS_CFG_VLAN_LEN_AWR_ENA, x)
+#define DEV2G5_MAC_TAGS_CFG_VLAN_LEN_AWR_ENA_GET(x)\
+ FIELD_GET(DEV2G5_MAC_TAGS_CFG_VLAN_LEN_AWR_ENA, x)
+
+#define DEV2G5_MAC_TAGS_CFG_PB_ENA GENMASK(2, 1)
+#define DEV2G5_MAC_TAGS_CFG_PB_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_TAGS_CFG_PB_ENA, x)
+#define DEV2G5_MAC_TAGS_CFG_PB_ENA_GET(x)\
+ FIELD_GET(DEV2G5_MAC_TAGS_CFG_PB_ENA, x)
+
+#define DEV2G5_MAC_TAGS_CFG_VLAN_AWR_ENA BIT(0)
+#define DEV2G5_MAC_TAGS_CFG_VLAN_AWR_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_TAGS_CFG_VLAN_AWR_ENA, x)
+#define DEV2G5_MAC_TAGS_CFG_VLAN_AWR_ENA_GET(x)\
+ FIELD_GET(DEV2G5_MAC_TAGS_CFG_VLAN_AWR_ENA, x)
+
+/* DEV1G:MAC_CFG_STATUS:MAC_TAGS_CFG2 */
+#define DEV2G5_MAC_TAGS_CFG2(t) __REG(TARGET_DEV2G5, t, 65, 52, 0, 1, 36, 16, 0, 1, 4)
+
+#define DEV2G5_MAC_TAGS_CFG2_TAG_ID3 GENMASK(31, 16)
+#define DEV2G5_MAC_TAGS_CFG2_TAG_ID3_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_TAGS_CFG2_TAG_ID3, x)
+#define DEV2G5_MAC_TAGS_CFG2_TAG_ID3_GET(x)\
+ FIELD_GET(DEV2G5_MAC_TAGS_CFG2_TAG_ID3, x)
+
+#define DEV2G5_MAC_TAGS_CFG2_TAG_ID2 GENMASK(15, 0)
+#define DEV2G5_MAC_TAGS_CFG2_TAG_ID2_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_TAGS_CFG2_TAG_ID2, x)
+#define DEV2G5_MAC_TAGS_CFG2_TAG_ID2_GET(x)\
+ FIELD_GET(DEV2G5_MAC_TAGS_CFG2_TAG_ID2, x)
+
+/* DEV1G:MAC_CFG_STATUS:MAC_ADV_CHK_CFG */
+#define DEV2G5_MAC_ADV_CHK_CFG(t) __REG(TARGET_DEV2G5, t, 65, 52, 0, 1, 36, 20, 0, 1, 4)
+
+#define DEV2G5_MAC_ADV_CHK_CFG_LEN_DROP_ENA BIT(0)
+#define DEV2G5_MAC_ADV_CHK_CFG_LEN_DROP_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_ADV_CHK_CFG_LEN_DROP_ENA, x)
+#define DEV2G5_MAC_ADV_CHK_CFG_LEN_DROP_ENA_GET(x)\
+ FIELD_GET(DEV2G5_MAC_ADV_CHK_CFG_LEN_DROP_ENA, x)
+
+/* DEV1G:MAC_CFG_STATUS:MAC_IFG_CFG */
+#define DEV2G5_MAC_IFG_CFG(t) __REG(TARGET_DEV2G5, t, 65, 52, 0, 1, 36, 24, 0, 1, 4)
+
+#define DEV2G5_MAC_IFG_CFG_RESTORE_OLD_IPG_CHECK BIT(17)
+#define DEV2G5_MAC_IFG_CFG_RESTORE_OLD_IPG_CHECK_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_IFG_CFG_RESTORE_OLD_IPG_CHECK, x)
+#define DEV2G5_MAC_IFG_CFG_RESTORE_OLD_IPG_CHECK_GET(x)\
+ FIELD_GET(DEV2G5_MAC_IFG_CFG_RESTORE_OLD_IPG_CHECK, x)
+
+#define DEV2G5_MAC_IFG_CFG_TX_IFG GENMASK(12, 8)
+#define DEV2G5_MAC_IFG_CFG_TX_IFG_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_IFG_CFG_TX_IFG, x)
+#define DEV2G5_MAC_IFG_CFG_TX_IFG_GET(x)\
+ FIELD_GET(DEV2G5_MAC_IFG_CFG_TX_IFG, x)
+
+#define DEV2G5_MAC_IFG_CFG_RX_IFG2 GENMASK(7, 4)
+#define DEV2G5_MAC_IFG_CFG_RX_IFG2_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_IFG_CFG_RX_IFG2, x)
+#define DEV2G5_MAC_IFG_CFG_RX_IFG2_GET(x)\
+ FIELD_GET(DEV2G5_MAC_IFG_CFG_RX_IFG2, x)
+
+#define DEV2G5_MAC_IFG_CFG_RX_IFG1 GENMASK(3, 0)
+#define DEV2G5_MAC_IFG_CFG_RX_IFG1_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_IFG_CFG_RX_IFG1, x)
+#define DEV2G5_MAC_IFG_CFG_RX_IFG1_GET(x)\
+ FIELD_GET(DEV2G5_MAC_IFG_CFG_RX_IFG1, x)
+
+/* DEV1G:MAC_CFG_STATUS:MAC_HDX_CFG */
+#define DEV2G5_MAC_HDX_CFG(t) __REG(TARGET_DEV2G5, t, 65, 52, 0, 1, 36, 28, 0, 1, 4)
+
+#define DEV2G5_MAC_HDX_CFG_BYPASS_COL_SYNC BIT(26)
+#define DEV2G5_MAC_HDX_CFG_BYPASS_COL_SYNC_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_HDX_CFG_BYPASS_COL_SYNC, x)
+#define DEV2G5_MAC_HDX_CFG_BYPASS_COL_SYNC_GET(x)\
+ FIELD_GET(DEV2G5_MAC_HDX_CFG_BYPASS_COL_SYNC, x)
+
+#define DEV2G5_MAC_HDX_CFG_SEED GENMASK(23, 16)
+#define DEV2G5_MAC_HDX_CFG_SEED_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_HDX_CFG_SEED, x)
+#define DEV2G5_MAC_HDX_CFG_SEED_GET(x)\
+ FIELD_GET(DEV2G5_MAC_HDX_CFG_SEED, x)
+
+#define DEV2G5_MAC_HDX_CFG_SEED_LOAD BIT(12)
+#define DEV2G5_MAC_HDX_CFG_SEED_LOAD_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_HDX_CFG_SEED_LOAD, x)
+#define DEV2G5_MAC_HDX_CFG_SEED_LOAD_GET(x)\
+ FIELD_GET(DEV2G5_MAC_HDX_CFG_SEED_LOAD, x)
+
+#define DEV2G5_MAC_HDX_CFG_RETRY_AFTER_EXC_COL_ENA BIT(8)
+#define DEV2G5_MAC_HDX_CFG_RETRY_AFTER_EXC_COL_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_HDX_CFG_RETRY_AFTER_EXC_COL_ENA, x)
+#define DEV2G5_MAC_HDX_CFG_RETRY_AFTER_EXC_COL_ENA_GET(x)\
+ FIELD_GET(DEV2G5_MAC_HDX_CFG_RETRY_AFTER_EXC_COL_ENA, x)
+
+#define DEV2G5_MAC_HDX_CFG_LATE_COL_POS GENMASK(6, 0)
+#define DEV2G5_MAC_HDX_CFG_LATE_COL_POS_SET(x)\
+ FIELD_PREP(DEV2G5_MAC_HDX_CFG_LATE_COL_POS, x)
+#define DEV2G5_MAC_HDX_CFG_LATE_COL_POS_GET(x)\
+ FIELD_GET(DEV2G5_MAC_HDX_CFG_LATE_COL_POS, x)
+
+/* DEV1G:PCS1G_CFG_STATUS:PCS1G_CFG */
+#define DEV2G5_PCS1G_CFG(t) __REG(TARGET_DEV2G5, t, 65, 88, 0, 1, 68, 0, 0, 1, 4)
+
+#define DEV2G5_PCS1G_CFG_LINK_STATUS_TYPE BIT(4)
+#define DEV2G5_PCS1G_CFG_LINK_STATUS_TYPE_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_CFG_LINK_STATUS_TYPE, x)
+#define DEV2G5_PCS1G_CFG_LINK_STATUS_TYPE_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_CFG_LINK_STATUS_TYPE, x)
+
+#define DEV2G5_PCS1G_CFG_AN_LINK_CTRL_ENA BIT(1)
+#define DEV2G5_PCS1G_CFG_AN_LINK_CTRL_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_CFG_AN_LINK_CTRL_ENA, x)
+#define DEV2G5_PCS1G_CFG_AN_LINK_CTRL_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_CFG_AN_LINK_CTRL_ENA, x)
+
+#define DEV2G5_PCS1G_CFG_PCS_ENA BIT(0)
+#define DEV2G5_PCS1G_CFG_PCS_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_CFG_PCS_ENA, x)
+#define DEV2G5_PCS1G_CFG_PCS_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_CFG_PCS_ENA, x)
+
+/* DEV1G:PCS1G_CFG_STATUS:PCS1G_MODE_CFG */
+#define DEV2G5_PCS1G_MODE_CFG(t) __REG(TARGET_DEV2G5, t, 65, 88, 0, 1, 68, 4, 0, 1, 4)
+
+#define DEV2G5_PCS1G_MODE_CFG_UNIDIR_MODE_ENA BIT(4)
+#define DEV2G5_PCS1G_MODE_CFG_UNIDIR_MODE_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_MODE_CFG_UNIDIR_MODE_ENA, x)
+#define DEV2G5_PCS1G_MODE_CFG_UNIDIR_MODE_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_MODE_CFG_UNIDIR_MODE_ENA, x)
+
+#define DEV2G5_PCS1G_MODE_CFG_SAVE_PREAMBLE_ENA BIT(1)
+#define DEV2G5_PCS1G_MODE_CFG_SAVE_PREAMBLE_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_MODE_CFG_SAVE_PREAMBLE_ENA, x)
+#define DEV2G5_PCS1G_MODE_CFG_SAVE_PREAMBLE_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_MODE_CFG_SAVE_PREAMBLE_ENA, x)
+
+#define DEV2G5_PCS1G_MODE_CFG_SGMII_MODE_ENA BIT(0)
+#define DEV2G5_PCS1G_MODE_CFG_SGMII_MODE_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_MODE_CFG_SGMII_MODE_ENA, x)
+#define DEV2G5_PCS1G_MODE_CFG_SGMII_MODE_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_MODE_CFG_SGMII_MODE_ENA, x)
+
+/* DEV1G:PCS1G_CFG_STATUS:PCS1G_SD_CFG */
+#define DEV2G5_PCS1G_SD_CFG(t) __REG(TARGET_DEV2G5, t, 65, 88, 0, 1, 68, 8, 0, 1, 4)
+
+#define DEV2G5_PCS1G_SD_CFG_SD_SEL BIT(8)
+#define DEV2G5_PCS1G_SD_CFG_SD_SEL_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_SD_CFG_SD_SEL, x)
+#define DEV2G5_PCS1G_SD_CFG_SD_SEL_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_SD_CFG_SD_SEL, x)
+
+#define DEV2G5_PCS1G_SD_CFG_SD_POL BIT(4)
+#define DEV2G5_PCS1G_SD_CFG_SD_POL_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_SD_CFG_SD_POL, x)
+#define DEV2G5_PCS1G_SD_CFG_SD_POL_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_SD_CFG_SD_POL, x)
+
+#define DEV2G5_PCS1G_SD_CFG_SD_ENA BIT(0)
+#define DEV2G5_PCS1G_SD_CFG_SD_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_SD_CFG_SD_ENA, x)
+#define DEV2G5_PCS1G_SD_CFG_SD_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_SD_CFG_SD_ENA, x)
+
+/* DEV1G:PCS1G_CFG_STATUS:PCS1G_ANEG_CFG */
+#define DEV2G5_PCS1G_ANEG_CFG(t) __REG(TARGET_DEV2G5, t, 65, 88, 0, 1, 68, 12, 0, 1, 4)
+
+#define DEV2G5_PCS1G_ANEG_CFG_ADV_ABILITY GENMASK(31, 16)
+#define DEV2G5_PCS1G_ANEG_CFG_ADV_ABILITY_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_ANEG_CFG_ADV_ABILITY, x)
+#define DEV2G5_PCS1G_ANEG_CFG_ADV_ABILITY_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_ANEG_CFG_ADV_ABILITY, x)
+
+#define DEV2G5_PCS1G_ANEG_CFG_SW_RESOLVE_ENA BIT(8)
+#define DEV2G5_PCS1G_ANEG_CFG_SW_RESOLVE_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_ANEG_CFG_SW_RESOLVE_ENA, x)
+#define DEV2G5_PCS1G_ANEG_CFG_SW_RESOLVE_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_ANEG_CFG_SW_RESOLVE_ENA, x)
+
+#define DEV2G5_PCS1G_ANEG_CFG_ANEG_RESTART_ONE_SHOT BIT(1)
+#define DEV2G5_PCS1G_ANEG_CFG_ANEG_RESTART_ONE_SHOT_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_ANEG_CFG_ANEG_RESTART_ONE_SHOT, x)
+#define DEV2G5_PCS1G_ANEG_CFG_ANEG_RESTART_ONE_SHOT_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_ANEG_CFG_ANEG_RESTART_ONE_SHOT, x)
+
+#define DEV2G5_PCS1G_ANEG_CFG_ANEG_ENA BIT(0)
+#define DEV2G5_PCS1G_ANEG_CFG_ANEG_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_ANEG_CFG_ANEG_ENA, x)
+#define DEV2G5_PCS1G_ANEG_CFG_ANEG_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_ANEG_CFG_ANEG_ENA, x)
+
+/* DEV1G:PCS1G_CFG_STATUS:PCS1G_LB_CFG */
+#define DEV2G5_PCS1G_LB_CFG(t) __REG(TARGET_DEV2G5, t, 65, 88, 0, 1, 68, 20, 0, 1, 4)
+
+#define DEV2G5_PCS1G_LB_CFG_RA_ENA BIT(4)
+#define DEV2G5_PCS1G_LB_CFG_RA_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_LB_CFG_RA_ENA, x)
+#define DEV2G5_PCS1G_LB_CFG_RA_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_LB_CFG_RA_ENA, x)
+
+#define DEV2G5_PCS1G_LB_CFG_GMII_PHY_LB_ENA BIT(1)
+#define DEV2G5_PCS1G_LB_CFG_GMII_PHY_LB_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_LB_CFG_GMII_PHY_LB_ENA, x)
+#define DEV2G5_PCS1G_LB_CFG_GMII_PHY_LB_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_LB_CFG_GMII_PHY_LB_ENA, x)
+
+#define DEV2G5_PCS1G_LB_CFG_TBI_HOST_LB_ENA BIT(0)
+#define DEV2G5_PCS1G_LB_CFG_TBI_HOST_LB_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_LB_CFG_TBI_HOST_LB_ENA, x)
+#define DEV2G5_PCS1G_LB_CFG_TBI_HOST_LB_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_LB_CFG_TBI_HOST_LB_ENA, x)
+
+/* DEV1G:PCS1G_CFG_STATUS:PCS1G_ANEG_STATUS */
+#define DEV2G5_PCS1G_ANEG_STATUS(t) __REG(TARGET_DEV2G5, t, 65, 88, 0, 1, 68, 32, 0, 1, 4)
+
+#define DEV2G5_PCS1G_ANEG_STATUS_LP_ADV_ABILITY GENMASK(31, 16)
+#define DEV2G5_PCS1G_ANEG_STATUS_LP_ADV_ABILITY_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_ANEG_STATUS_LP_ADV_ABILITY, x)
+#define DEV2G5_PCS1G_ANEG_STATUS_LP_ADV_ABILITY_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_ANEG_STATUS_LP_ADV_ABILITY, x)
+
+#define DEV2G5_PCS1G_ANEG_STATUS_PR BIT(4)
+#define DEV2G5_PCS1G_ANEG_STATUS_PR_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_ANEG_STATUS_PR, x)
+#define DEV2G5_PCS1G_ANEG_STATUS_PR_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_ANEG_STATUS_PR, x)
+
+#define DEV2G5_PCS1G_ANEG_STATUS_PAGE_RX_STICKY BIT(3)
+#define DEV2G5_PCS1G_ANEG_STATUS_PAGE_RX_STICKY_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_ANEG_STATUS_PAGE_RX_STICKY, x)
+#define DEV2G5_PCS1G_ANEG_STATUS_PAGE_RX_STICKY_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_ANEG_STATUS_PAGE_RX_STICKY, x)
+
+#define DEV2G5_PCS1G_ANEG_STATUS_ANEG_COMPLETE BIT(0)
+#define DEV2G5_PCS1G_ANEG_STATUS_ANEG_COMPLETE_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_ANEG_STATUS_ANEG_COMPLETE, x)
+#define DEV2G5_PCS1G_ANEG_STATUS_ANEG_COMPLETE_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_ANEG_STATUS_ANEG_COMPLETE, x)
+
+/* DEV1G:PCS1G_CFG_STATUS:PCS1G_LINK_STATUS */
+#define DEV2G5_PCS1G_LINK_STATUS(t) __REG(TARGET_DEV2G5, t, 65, 88, 0, 1, 68, 40, 0, 1, 4)
+
+#define DEV2G5_PCS1G_LINK_STATUS_DELAY_VAR GENMASK(15, 12)
+#define DEV2G5_PCS1G_LINK_STATUS_DELAY_VAR_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_LINK_STATUS_DELAY_VAR, x)
+#define DEV2G5_PCS1G_LINK_STATUS_DELAY_VAR_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_LINK_STATUS_DELAY_VAR, x)
+
+#define DEV2G5_PCS1G_LINK_STATUS_SIGNAL_DETECT BIT(8)
+#define DEV2G5_PCS1G_LINK_STATUS_SIGNAL_DETECT_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_LINK_STATUS_SIGNAL_DETECT, x)
+#define DEV2G5_PCS1G_LINK_STATUS_SIGNAL_DETECT_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_LINK_STATUS_SIGNAL_DETECT, x)
+
+#define DEV2G5_PCS1G_LINK_STATUS_LINK_STATUS BIT(4)
+#define DEV2G5_PCS1G_LINK_STATUS_LINK_STATUS_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_LINK_STATUS_LINK_STATUS, x)
+#define DEV2G5_PCS1G_LINK_STATUS_LINK_STATUS_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_LINK_STATUS_LINK_STATUS, x)
+
+#define DEV2G5_PCS1G_LINK_STATUS_SYNC_STATUS BIT(0)
+#define DEV2G5_PCS1G_LINK_STATUS_SYNC_STATUS_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_LINK_STATUS_SYNC_STATUS, x)
+#define DEV2G5_PCS1G_LINK_STATUS_SYNC_STATUS_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_LINK_STATUS_SYNC_STATUS, x)
+
+/* DEV1G:PCS1G_CFG_STATUS:PCS1G_STICKY */
+#define DEV2G5_PCS1G_STICKY(t) __REG(TARGET_DEV2G5, t, 65, 88, 0, 1, 68, 48, 0, 1, 4)
+
+#define DEV2G5_PCS1G_STICKY_LINK_DOWN_STICKY BIT(4)
+#define DEV2G5_PCS1G_STICKY_LINK_DOWN_STICKY_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_STICKY_LINK_DOWN_STICKY, x)
+#define DEV2G5_PCS1G_STICKY_LINK_DOWN_STICKY_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_STICKY_LINK_DOWN_STICKY, x)
+
+#define DEV2G5_PCS1G_STICKY_OUT_OF_SYNC_STICKY BIT(0)
+#define DEV2G5_PCS1G_STICKY_OUT_OF_SYNC_STICKY_SET(x)\
+ FIELD_PREP(DEV2G5_PCS1G_STICKY_OUT_OF_SYNC_STICKY, x)
+#define DEV2G5_PCS1G_STICKY_OUT_OF_SYNC_STICKY_GET(x)\
+ FIELD_GET(DEV2G5_PCS1G_STICKY_OUT_OF_SYNC_STICKY, x)
+
+/* DEV1G:PCS_FX100_CONFIGURATION:PCS_FX100_CFG */
+#define DEV2G5_PCS_FX100_CFG(t) __REG(TARGET_DEV2G5, t, 65, 164, 0, 1, 4, 0, 0, 1, 4)
+
+#define DEV2G5_PCS_FX100_CFG_SD_SEL BIT(26)
+#define DEV2G5_PCS_FX100_CFG_SD_SEL_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_CFG_SD_SEL, x)
+#define DEV2G5_PCS_FX100_CFG_SD_SEL_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_CFG_SD_SEL, x)
+
+#define DEV2G5_PCS_FX100_CFG_SD_POL BIT(25)
+#define DEV2G5_PCS_FX100_CFG_SD_POL_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_CFG_SD_POL, x)
+#define DEV2G5_PCS_FX100_CFG_SD_POL_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_CFG_SD_POL, x)
+
+#define DEV2G5_PCS_FX100_CFG_SD_ENA BIT(24)
+#define DEV2G5_PCS_FX100_CFG_SD_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_CFG_SD_ENA, x)
+#define DEV2G5_PCS_FX100_CFG_SD_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_CFG_SD_ENA, x)
+
+#define DEV2G5_PCS_FX100_CFG_LOOPBACK_ENA BIT(20)
+#define DEV2G5_PCS_FX100_CFG_LOOPBACK_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_CFG_LOOPBACK_ENA, x)
+#define DEV2G5_PCS_FX100_CFG_LOOPBACK_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_CFG_LOOPBACK_ENA, x)
+
+#define DEV2G5_PCS_FX100_CFG_SWAP_MII_ENA BIT(16)
+#define DEV2G5_PCS_FX100_CFG_SWAP_MII_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_CFG_SWAP_MII_ENA, x)
+#define DEV2G5_PCS_FX100_CFG_SWAP_MII_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_CFG_SWAP_MII_ENA, x)
+
+#define DEV2G5_PCS_FX100_CFG_RXBITSEL GENMASK(15, 12)
+#define DEV2G5_PCS_FX100_CFG_RXBITSEL_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_CFG_RXBITSEL, x)
+#define DEV2G5_PCS_FX100_CFG_RXBITSEL_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_CFG_RXBITSEL, x)
+
+#define DEV2G5_PCS_FX100_CFG_SIGDET_CFG GENMASK(10, 9)
+#define DEV2G5_PCS_FX100_CFG_SIGDET_CFG_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_CFG_SIGDET_CFG, x)
+#define DEV2G5_PCS_FX100_CFG_SIGDET_CFG_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_CFG_SIGDET_CFG, x)
+
+#define DEV2G5_PCS_FX100_CFG_LINKHYST_TM_ENA BIT(8)
+#define DEV2G5_PCS_FX100_CFG_LINKHYST_TM_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_CFG_LINKHYST_TM_ENA, x)
+#define DEV2G5_PCS_FX100_CFG_LINKHYST_TM_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_CFG_LINKHYST_TM_ENA, x)
+
+#define DEV2G5_PCS_FX100_CFG_LINKHYSTTIMER GENMASK(7, 4)
+#define DEV2G5_PCS_FX100_CFG_LINKHYSTTIMER_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_CFG_LINKHYSTTIMER, x)
+#define DEV2G5_PCS_FX100_CFG_LINKHYSTTIMER_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_CFG_LINKHYSTTIMER, x)
+
+#define DEV2G5_PCS_FX100_CFG_UNIDIR_MODE_ENA BIT(3)
+#define DEV2G5_PCS_FX100_CFG_UNIDIR_MODE_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_CFG_UNIDIR_MODE_ENA, x)
+#define DEV2G5_PCS_FX100_CFG_UNIDIR_MODE_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_CFG_UNIDIR_MODE_ENA, x)
+
+#define DEV2G5_PCS_FX100_CFG_FEFCHK_ENA BIT(2)
+#define DEV2G5_PCS_FX100_CFG_FEFCHK_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_CFG_FEFCHK_ENA, x)
+#define DEV2G5_PCS_FX100_CFG_FEFCHK_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_CFG_FEFCHK_ENA, x)
+
+#define DEV2G5_PCS_FX100_CFG_FEFGEN_ENA BIT(1)
+#define DEV2G5_PCS_FX100_CFG_FEFGEN_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_CFG_FEFGEN_ENA, x)
+#define DEV2G5_PCS_FX100_CFG_FEFGEN_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_CFG_FEFGEN_ENA, x)
+
+#define DEV2G5_PCS_FX100_CFG_PCS_ENA BIT(0)
+#define DEV2G5_PCS_FX100_CFG_PCS_ENA_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_CFG_PCS_ENA, x)
+#define DEV2G5_PCS_FX100_CFG_PCS_ENA_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_CFG_PCS_ENA, x)
+
+/* DEV1G:PCS_FX100_STATUS:PCS_FX100_STATUS */
+#define DEV2G5_PCS_FX100_STATUS(t) __REG(TARGET_DEV2G5, t, 65, 168, 0, 1, 4, 0, 0, 1, 4)
+
+#define DEV2G5_PCS_FX100_STATUS_EDGE_POS_PTP GENMASK(11, 8)
+#define DEV2G5_PCS_FX100_STATUS_EDGE_POS_PTP_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_STATUS_EDGE_POS_PTP, x)
+#define DEV2G5_PCS_FX100_STATUS_EDGE_POS_PTP_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_STATUS_EDGE_POS_PTP, x)
+
+#define DEV2G5_PCS_FX100_STATUS_PCS_ERROR_STICKY BIT(7)
+#define DEV2G5_PCS_FX100_STATUS_PCS_ERROR_STICKY_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_STATUS_PCS_ERROR_STICKY, x)
+#define DEV2G5_PCS_FX100_STATUS_PCS_ERROR_STICKY_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_STATUS_PCS_ERROR_STICKY, x)
+
+#define DEV2G5_PCS_FX100_STATUS_FEF_FOUND_STICKY BIT(6)
+#define DEV2G5_PCS_FX100_STATUS_FEF_FOUND_STICKY_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_STATUS_FEF_FOUND_STICKY, x)
+#define DEV2G5_PCS_FX100_STATUS_FEF_FOUND_STICKY_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_STATUS_FEF_FOUND_STICKY, x)
+
+#define DEV2G5_PCS_FX100_STATUS_SSD_ERROR_STICKY BIT(5)
+#define DEV2G5_PCS_FX100_STATUS_SSD_ERROR_STICKY_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_STATUS_SSD_ERROR_STICKY, x)
+#define DEV2G5_PCS_FX100_STATUS_SSD_ERROR_STICKY_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_STATUS_SSD_ERROR_STICKY, x)
+
+#define DEV2G5_PCS_FX100_STATUS_SYNC_LOST_STICKY BIT(4)
+#define DEV2G5_PCS_FX100_STATUS_SYNC_LOST_STICKY_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_STATUS_SYNC_LOST_STICKY, x)
+#define DEV2G5_PCS_FX100_STATUS_SYNC_LOST_STICKY_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_STATUS_SYNC_LOST_STICKY, x)
+
+#define DEV2G5_PCS_FX100_STATUS_FEF_STATUS BIT(2)
+#define DEV2G5_PCS_FX100_STATUS_FEF_STATUS_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_STATUS_FEF_STATUS, x)
+#define DEV2G5_PCS_FX100_STATUS_FEF_STATUS_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_STATUS_FEF_STATUS, x)
+
+#define DEV2G5_PCS_FX100_STATUS_SIGNAL_DETECT BIT(1)
+#define DEV2G5_PCS_FX100_STATUS_SIGNAL_DETECT_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_STATUS_SIGNAL_DETECT, x)
+#define DEV2G5_PCS_FX100_STATUS_SIGNAL_DETECT_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_STATUS_SIGNAL_DETECT, x)
+
+#define DEV2G5_PCS_FX100_STATUS_SYNC_STATUS BIT(0)
+#define DEV2G5_PCS_FX100_STATUS_SYNC_STATUS_SET(x)\
+ FIELD_PREP(DEV2G5_PCS_FX100_STATUS_SYNC_STATUS, x)
+#define DEV2G5_PCS_FX100_STATUS_SYNC_STATUS_GET(x)\
+ FIELD_GET(DEV2G5_PCS_FX100_STATUS_SYNC_STATUS, x)
+
+/* DEV10G:MAC_CFG_STATUS:MAC_ENA_CFG */
+#define DEV5G_MAC_ENA_CFG(t) __REG(TARGET_DEV5G, t, 13, 0, 0, 1, 60, 0, 0, 1, 4)
+
+#define DEV5G_MAC_ENA_CFG_RX_ENA BIT(4)
+#define DEV5G_MAC_ENA_CFG_RX_ENA_SET(x)\
+ FIELD_PREP(DEV5G_MAC_ENA_CFG_RX_ENA, x)
+#define DEV5G_MAC_ENA_CFG_RX_ENA_GET(x)\
+ FIELD_GET(DEV5G_MAC_ENA_CFG_RX_ENA, x)
+
+#define DEV5G_MAC_ENA_CFG_TX_ENA BIT(0)
+#define DEV5G_MAC_ENA_CFG_TX_ENA_SET(x)\
+ FIELD_PREP(DEV5G_MAC_ENA_CFG_TX_ENA, x)
+#define DEV5G_MAC_ENA_CFG_TX_ENA_GET(x)\
+ FIELD_GET(DEV5G_MAC_ENA_CFG_TX_ENA, x)
+
+/* DEV10G:MAC_CFG_STATUS:MAC_MAXLEN_CFG */
+#define DEV5G_MAC_MAXLEN_CFG(t) __REG(TARGET_DEV5G, t, 13, 0, 0, 1, 60, 8, 0, 1, 4)
+
+#define DEV5G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK BIT(16)
+#define DEV5G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK_SET(x)\
+ FIELD_PREP(DEV5G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK, x)
+#define DEV5G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK_GET(x)\
+ FIELD_GET(DEV5G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK, x)
+
+#define DEV5G_MAC_MAXLEN_CFG_MAX_LEN GENMASK(15, 0)
+#define DEV5G_MAC_MAXLEN_CFG_MAX_LEN_SET(x)\
+ FIELD_PREP(DEV5G_MAC_MAXLEN_CFG_MAX_LEN, x)
+#define DEV5G_MAC_MAXLEN_CFG_MAX_LEN_GET(x)\
+ FIELD_GET(DEV5G_MAC_MAXLEN_CFG_MAX_LEN, x)
+
+/* DEV10G:MAC_CFG_STATUS:MAC_ADV_CHK_CFG */
+#define DEV5G_MAC_ADV_CHK_CFG(t) __REG(TARGET_DEV5G, t, 13, 0, 0, 1, 60, 28, 0, 1, 4)
+
+#define DEV5G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA BIT(24)
+#define DEV5G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA_SET(x)\
+ FIELD_PREP(DEV5G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA, x)
+#define DEV5G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA_GET(x)\
+ FIELD_GET(DEV5G_MAC_ADV_CHK_CFG_EXT_EOP_CHK_ENA, x)
+
+#define DEV5G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA BIT(20)
+#define DEV5G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA_SET(x)\
+ FIELD_PREP(DEV5G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA, x)
+#define DEV5G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA_GET(x)\
+ FIELD_GET(DEV5G_MAC_ADV_CHK_CFG_EXT_SOP_CHK_ENA, x)
+
+#define DEV5G_MAC_ADV_CHK_CFG_SFD_CHK_ENA BIT(16)
+#define DEV5G_MAC_ADV_CHK_CFG_SFD_CHK_ENA_SET(x)\
+ FIELD_PREP(DEV5G_MAC_ADV_CHK_CFG_SFD_CHK_ENA, x)
+#define DEV5G_MAC_ADV_CHK_CFG_SFD_CHK_ENA_GET(x)\
+ FIELD_GET(DEV5G_MAC_ADV_CHK_CFG_SFD_CHK_ENA, x)
+
+#define DEV5G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS BIT(12)
+#define DEV5G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS_SET(x)\
+ FIELD_PREP(DEV5G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS, x)
+#define DEV5G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS_GET(x)\
+ FIELD_GET(DEV5G_MAC_ADV_CHK_CFG_PRM_SHK_CHK_DIS, x)
+
+#define DEV5G_MAC_ADV_CHK_CFG_PRM_CHK_ENA BIT(8)
+#define DEV5G_MAC_ADV_CHK_CFG_PRM_CHK_ENA_SET(x)\
+ FIELD_PREP(DEV5G_MAC_ADV_CHK_CFG_PRM_CHK_ENA, x)
+#define DEV5G_MAC_ADV_CHK_CFG_PRM_CHK_ENA_GET(x)\
+ FIELD_GET(DEV5G_MAC_ADV_CHK_CFG_PRM_CHK_ENA, x)
+
+#define DEV5G_MAC_ADV_CHK_CFG_OOR_ERR_ENA BIT(4)
+#define DEV5G_MAC_ADV_CHK_CFG_OOR_ERR_ENA_SET(x)\
+ FIELD_PREP(DEV5G_MAC_ADV_CHK_CFG_OOR_ERR_ENA, x)
+#define DEV5G_MAC_ADV_CHK_CFG_OOR_ERR_ENA_GET(x)\
+ FIELD_GET(DEV5G_MAC_ADV_CHK_CFG_OOR_ERR_ENA, x)
+
+#define DEV5G_MAC_ADV_CHK_CFG_INR_ERR_ENA BIT(0)
+#define DEV5G_MAC_ADV_CHK_CFG_INR_ERR_ENA_SET(x)\
+ FIELD_PREP(DEV5G_MAC_ADV_CHK_CFG_INR_ERR_ENA, x)
+#define DEV5G_MAC_ADV_CHK_CFG_INR_ERR_ENA_GET(x)\
+ FIELD_GET(DEV5G_MAC_ADV_CHK_CFG_INR_ERR_ENA, x)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_SYMBOL_ERR_CNT */
+#define DEV5G_RX_SYMBOL_ERR_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 0, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_PAUSE_CNT */
+#define DEV5G_RX_PAUSE_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 4, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_UNSUP_OPCODE_CNT */
+#define DEV5G_RX_UNSUP_OPCODE_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 8, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_UC_CNT */
+#define DEV5G_RX_UC_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 12, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_MC_CNT */
+#define DEV5G_RX_MC_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 16, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_BC_CNT */
+#define DEV5G_RX_BC_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 20, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_CRC_ERR_CNT */
+#define DEV5G_RX_CRC_ERR_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 24, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_UNDERSIZE_CNT */
+#define DEV5G_RX_UNDERSIZE_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 28, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_FRAGMENTS_CNT */
+#define DEV5G_RX_FRAGMENTS_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 32, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_IN_RANGE_LEN_ERR_CNT */
+#define DEV5G_RX_IN_RANGE_LEN_ERR_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 36, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_OUT_OF_RANGE_LEN_ERR_CNT */
+#define DEV5G_RX_OUT_OF_RANGE_LEN_ERR_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 40, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_OVERSIZE_CNT */
+#define DEV5G_RX_OVERSIZE_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 44, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_JABBERS_CNT */
+#define DEV5G_RX_JABBERS_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 48, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_SIZE64_CNT */
+#define DEV5G_RX_SIZE64_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 52, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_SIZE65TO127_CNT */
+#define DEV5G_RX_SIZE65TO127_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 56, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_SIZE128TO255_CNT */
+#define DEV5G_RX_SIZE128TO255_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 60, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_SIZE256TO511_CNT */
+#define DEV5G_RX_SIZE256TO511_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 64, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_SIZE512TO1023_CNT */
+#define DEV5G_RX_SIZE512TO1023_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 68, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_SIZE1024TO1518_CNT */
+#define DEV5G_RX_SIZE1024TO1518_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 72, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_SIZE1519TOMAX_CNT */
+#define DEV5G_RX_SIZE1519TOMAX_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 76, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_IPG_SHRINK_CNT */
+#define DEV5G_RX_IPG_SHRINK_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 80, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:TX_PAUSE_CNT */
+#define DEV5G_TX_PAUSE_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 84, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:TX_UC_CNT */
+#define DEV5G_TX_UC_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 88, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:TX_MC_CNT */
+#define DEV5G_TX_MC_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 92, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:TX_BC_CNT */
+#define DEV5G_TX_BC_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 96, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:TX_SIZE64_CNT */
+#define DEV5G_TX_SIZE64_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 100, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:TX_SIZE65TO127_CNT */
+#define DEV5G_TX_SIZE65TO127_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 104, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:TX_SIZE128TO255_CNT */
+#define DEV5G_TX_SIZE128TO255_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 108, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:TX_SIZE256TO511_CNT */
+#define DEV5G_TX_SIZE256TO511_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 112, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:TX_SIZE512TO1023_CNT */
+#define DEV5G_TX_SIZE512TO1023_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 116, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:TX_SIZE1024TO1518_CNT */
+#define DEV5G_TX_SIZE1024TO1518_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 120, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:TX_SIZE1519TOMAX_CNT */
+#define DEV5G_TX_SIZE1519TOMAX_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 124, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_ALIGNMENT_LOST_CNT */
+#define DEV5G_RX_ALIGNMENT_LOST_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 128, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_TAGGED_FRMS_CNT */
+#define DEV5G_RX_TAGGED_FRMS_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 132, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_UNTAGGED_FRMS_CNT */
+#define DEV5G_RX_UNTAGGED_FRMS_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 136, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:TX_TAGGED_FRMS_CNT */
+#define DEV5G_TX_TAGGED_FRMS_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 140, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:TX_UNTAGGED_FRMS_CNT */
+#define DEV5G_TX_UNTAGGED_FRMS_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 144, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_SYMBOL_ERR_CNT */
+#define DEV5G_PMAC_RX_SYMBOL_ERR_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 148, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_PAUSE_CNT */
+#define DEV5G_PMAC_RX_PAUSE_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 152, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_UNSUP_OPCODE_CNT */
+#define DEV5G_PMAC_RX_UNSUP_OPCODE_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 156, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_UC_CNT */
+#define DEV5G_PMAC_RX_UC_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 160, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_MC_CNT */
+#define DEV5G_PMAC_RX_MC_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 164, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_BC_CNT */
+#define DEV5G_PMAC_RX_BC_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 168, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_CRC_ERR_CNT */
+#define DEV5G_PMAC_RX_CRC_ERR_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 172, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_UNDERSIZE_CNT */
+#define DEV5G_PMAC_RX_UNDERSIZE_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 176, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_FRAGMENTS_CNT */
+#define DEV5G_PMAC_RX_FRAGMENTS_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 180, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_IN_RANGE_LEN_ERR_CNT */
+#define DEV5G_PMAC_RX_IN_RANGE_LEN_ERR_CNT(t) __REG(TARGET_DEV5G,\
+ t, 13, 60, 0, 1, 312, 184, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_OUT_OF_RANGE_LEN_ERR_CNT */
+#define DEV5G_PMAC_RX_OUT_OF_RANGE_LEN_ERR_CNT(t) __REG(TARGET_DEV5G,\
+ t, 13, 60, 0, 1, 312, 188, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_OVERSIZE_CNT */
+#define DEV5G_PMAC_RX_OVERSIZE_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 192, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_JABBERS_CNT */
+#define DEV5G_PMAC_RX_JABBERS_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 196, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_SIZE64_CNT */
+#define DEV5G_PMAC_RX_SIZE64_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 200, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_SIZE65TO127_CNT */
+#define DEV5G_PMAC_RX_SIZE65TO127_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 204, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_SIZE128TO255_CNT */
+#define DEV5G_PMAC_RX_SIZE128TO255_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 208, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_SIZE256TO511_CNT */
+#define DEV5G_PMAC_RX_SIZE256TO511_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 212, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_SIZE512TO1023_CNT */
+#define DEV5G_PMAC_RX_SIZE512TO1023_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 216, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_SIZE1024TO1518_CNT */
+#define DEV5G_PMAC_RX_SIZE1024TO1518_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 220, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_SIZE1519TOMAX_CNT */
+#define DEV5G_PMAC_RX_SIZE1519TOMAX_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 224, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_TX_PAUSE_CNT */
+#define DEV5G_PMAC_TX_PAUSE_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 228, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_TX_UC_CNT */
+#define DEV5G_PMAC_TX_UC_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 232, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_TX_MC_CNT */
+#define DEV5G_PMAC_TX_MC_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 236, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_TX_BC_CNT */
+#define DEV5G_PMAC_TX_BC_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 240, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_TX_SIZE64_CNT */
+#define DEV5G_PMAC_TX_SIZE64_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 244, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_TX_SIZE65TO127_CNT */
+#define DEV5G_PMAC_TX_SIZE65TO127_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 248, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_TX_SIZE128TO255_CNT */
+#define DEV5G_PMAC_TX_SIZE128TO255_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 252, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_TX_SIZE256TO511_CNT */
+#define DEV5G_PMAC_TX_SIZE256TO511_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 256, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_TX_SIZE512TO1023_CNT */
+#define DEV5G_PMAC_TX_SIZE512TO1023_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 260, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_TX_SIZE1024TO1518_CNT */
+#define DEV5G_PMAC_TX_SIZE1024TO1518_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 264, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_TX_SIZE1519TOMAX_CNT */
+#define DEV5G_PMAC_TX_SIZE1519TOMAX_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 268, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_ALIGNMENT_LOST_CNT */
+#define DEV5G_PMAC_RX_ALIGNMENT_LOST_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 272, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:MM_RX_ASSEMBLY_ERR_CNT */
+#define DEV5G_MM_RX_ASSEMBLY_ERR_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 276, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:MM_RX_SMD_ERR_CNT */
+#define DEV5G_MM_RX_SMD_ERR_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 280, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:MM_RX_ASSEMBLY_OK_CNT */
+#define DEV5G_MM_RX_ASSEMBLY_OK_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 284, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:MM_RX_MERGE_FRAG_CNT */
+#define DEV5G_MM_RX_MERGE_FRAG_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 288, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:MM_TX_PFRAGMENT_CNT */
+#define DEV5G_MM_TX_PFRAGMENT_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 292, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_HIH_CKSM_ERR_CNT */
+#define DEV5G_RX_HIH_CKSM_ERR_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 296, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:RX_XGMII_PROT_ERR_CNT */
+#define DEV5G_RX_XGMII_PROT_ERR_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 300, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_HIH_CKSM_ERR_CNT */
+#define DEV5G_PMAC_RX_HIH_CKSM_ERR_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 304, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_32BIT:PMAC_RX_XGMII_PROT_ERR_CNT */
+#define DEV5G_PMAC_RX_XGMII_PROT_ERR_CNT(t) __REG(TARGET_DEV5G, t, 13, 60, 0, 1, 312, 308, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_40BIT:RX_IN_BYTES_CNT */
+#define DEV5G_RX_IN_BYTES_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 0, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_40BIT:RX_IN_BYTES_MSB_CNT */
+#define DEV5G_RX_IN_BYTES_MSB_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 4, 0, 1, 4)
+
+#define DEV5G_RX_IN_BYTES_MSB_CNT_RX_IN_BYTES_MSB_CNT GENMASK(7, 0)
+#define DEV5G_RX_IN_BYTES_MSB_CNT_RX_IN_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(DEV5G_RX_IN_BYTES_MSB_CNT_RX_IN_BYTES_MSB_CNT, x)
+#define DEV5G_RX_IN_BYTES_MSB_CNT_RX_IN_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(DEV5G_RX_IN_BYTES_MSB_CNT_RX_IN_BYTES_MSB_CNT, x)
+
+/* DEV10G:DEV_STATISTICS_40BIT:RX_OK_BYTES_CNT */
+#define DEV5G_RX_OK_BYTES_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 8, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_40BIT:RX_OK_BYTES_MSB_CNT */
+#define DEV5G_RX_OK_BYTES_MSB_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 12, 0, 1, 4)
+
+#define DEV5G_RX_OK_BYTES_MSB_CNT_RX_OK_BYTES_MSB_CNT GENMASK(7, 0)
+#define DEV5G_RX_OK_BYTES_MSB_CNT_RX_OK_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(DEV5G_RX_OK_BYTES_MSB_CNT_RX_OK_BYTES_MSB_CNT, x)
+#define DEV5G_RX_OK_BYTES_MSB_CNT_RX_OK_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(DEV5G_RX_OK_BYTES_MSB_CNT_RX_OK_BYTES_MSB_CNT, x)
+
+/* DEV10G:DEV_STATISTICS_40BIT:RX_BAD_BYTES_CNT */
+#define DEV5G_RX_BAD_BYTES_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 16, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_40BIT:RX_BAD_BYTES_MSB_CNT */
+#define DEV5G_RX_BAD_BYTES_MSB_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 20, 0, 1, 4)
+
+#define DEV5G_RX_BAD_BYTES_MSB_CNT_RX_BAD_BYTES_MSB_CNT GENMASK(7, 0)
+#define DEV5G_RX_BAD_BYTES_MSB_CNT_RX_BAD_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(DEV5G_RX_BAD_BYTES_MSB_CNT_RX_BAD_BYTES_MSB_CNT, x)
+#define DEV5G_RX_BAD_BYTES_MSB_CNT_RX_BAD_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(DEV5G_RX_BAD_BYTES_MSB_CNT_RX_BAD_BYTES_MSB_CNT, x)
+
+/* DEV10G:DEV_STATISTICS_40BIT:TX_OUT_BYTES_CNT */
+#define DEV5G_TX_OUT_BYTES_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 24, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_40BIT:TX_OUT_BYTES_MSB_CNT */
+#define DEV5G_TX_OUT_BYTES_MSB_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 28, 0, 1, 4)
+
+#define DEV5G_TX_OUT_BYTES_MSB_CNT_TX_OUT_BYTES_MSB_CNT GENMASK(7, 0)
+#define DEV5G_TX_OUT_BYTES_MSB_CNT_TX_OUT_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(DEV5G_TX_OUT_BYTES_MSB_CNT_TX_OUT_BYTES_MSB_CNT, x)
+#define DEV5G_TX_OUT_BYTES_MSB_CNT_TX_OUT_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(DEV5G_TX_OUT_BYTES_MSB_CNT_TX_OUT_BYTES_MSB_CNT, x)
+
+/* DEV10G:DEV_STATISTICS_40BIT:TX_OK_BYTES_CNT */
+#define DEV5G_TX_OK_BYTES_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 32, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_40BIT:TX_OK_BYTES_MSB_CNT */
+#define DEV5G_TX_OK_BYTES_MSB_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 36, 0, 1, 4)
+
+#define DEV5G_TX_OK_BYTES_MSB_CNT_TX_OK_BYTES_MSB_CNT GENMASK(7, 0)
+#define DEV5G_TX_OK_BYTES_MSB_CNT_TX_OK_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(DEV5G_TX_OK_BYTES_MSB_CNT_TX_OK_BYTES_MSB_CNT, x)
+#define DEV5G_TX_OK_BYTES_MSB_CNT_TX_OK_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(DEV5G_TX_OK_BYTES_MSB_CNT_TX_OK_BYTES_MSB_CNT, x)
+
+/* DEV10G:DEV_STATISTICS_40BIT:PMAC_RX_OK_BYTES_CNT */
+#define DEV5G_PMAC_RX_OK_BYTES_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 40, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_40BIT:PMAC_RX_OK_BYTES_MSB_CNT */
+#define DEV5G_PMAC_RX_OK_BYTES_MSB_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 44, 0, 1, 4)
+
+#define DEV5G_PMAC_RX_OK_BYTES_MSB_CNT_PMAC_RX_OK_BYTES_MSB_CNT GENMASK(7, 0)
+#define DEV5G_PMAC_RX_OK_BYTES_MSB_CNT_PMAC_RX_OK_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(DEV5G_PMAC_RX_OK_BYTES_MSB_CNT_PMAC_RX_OK_BYTES_MSB_CNT, x)
+#define DEV5G_PMAC_RX_OK_BYTES_MSB_CNT_PMAC_RX_OK_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(DEV5G_PMAC_RX_OK_BYTES_MSB_CNT_PMAC_RX_OK_BYTES_MSB_CNT, x)
+
+/* DEV10G:DEV_STATISTICS_40BIT:PMAC_RX_BAD_BYTES_CNT */
+#define DEV5G_PMAC_RX_BAD_BYTES_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 48, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_40BIT:PMAC_RX_BAD_BYTES_MSB_CNT */
+#define DEV5G_PMAC_RX_BAD_BYTES_MSB_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 52, 0, 1, 4)
+
+#define DEV5G_PMAC_RX_BAD_BYTES_MSB_CNT_PMAC_RX_BAD_BYTES_MSB_CNT GENMASK(7, 0)
+#define DEV5G_PMAC_RX_BAD_BYTES_MSB_CNT_PMAC_RX_BAD_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(DEV5G_PMAC_RX_BAD_BYTES_MSB_CNT_PMAC_RX_BAD_BYTES_MSB_CNT, x)
+#define DEV5G_PMAC_RX_BAD_BYTES_MSB_CNT_PMAC_RX_BAD_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(DEV5G_PMAC_RX_BAD_BYTES_MSB_CNT_PMAC_RX_BAD_BYTES_MSB_CNT, x)
+
+/* DEV10G:DEV_STATISTICS_40BIT:PMAC_TX_OK_BYTES_CNT */
+#define DEV5G_PMAC_TX_OK_BYTES_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 56, 0, 1, 4)
+
+/* DEV10G:DEV_STATISTICS_40BIT:PMAC_TX_OK_BYTES_MSB_CNT */
+#define DEV5G_PMAC_TX_OK_BYTES_MSB_CNT(t) __REG(TARGET_DEV5G, t, 13, 372, 0, 1, 64, 60, 0, 1, 4)
+
+#define DEV5G_PMAC_TX_OK_BYTES_MSB_CNT_PMAC_TX_OK_BYTES_MSB_CNT GENMASK(7, 0)
+#define DEV5G_PMAC_TX_OK_BYTES_MSB_CNT_PMAC_TX_OK_BYTES_MSB_CNT_SET(x)\
+ FIELD_PREP(DEV5G_PMAC_TX_OK_BYTES_MSB_CNT_PMAC_TX_OK_BYTES_MSB_CNT, x)
+#define DEV5G_PMAC_TX_OK_BYTES_MSB_CNT_PMAC_TX_OK_BYTES_MSB_CNT_GET(x)\
+ FIELD_GET(DEV5G_PMAC_TX_OK_BYTES_MSB_CNT_PMAC_TX_OK_BYTES_MSB_CNT, x)
+
+/* DEV10G:DEV_CFG_STATUS:DEV_RST_CTRL */
+#define DEV5G_DEV_RST_CTRL(t) __REG(TARGET_DEV5G, t, 13, 436, 0, 1, 52, 0, 0, 1, 4)
+
+#define DEV5G_DEV_RST_CTRL_PARDET_MODE_ENA BIT(28)
+#define DEV5G_DEV_RST_CTRL_PARDET_MODE_ENA_SET(x)\
+ FIELD_PREP(DEV5G_DEV_RST_CTRL_PARDET_MODE_ENA, x)
+#define DEV5G_DEV_RST_CTRL_PARDET_MODE_ENA_GET(x)\
+ FIELD_GET(DEV5G_DEV_RST_CTRL_PARDET_MODE_ENA, x)
+
+#define DEV5G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS BIT(27)
+#define DEV5G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS_SET(x)\
+ FIELD_PREP(DEV5G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS, x)
+#define DEV5G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS_GET(x)\
+ FIELD_GET(DEV5G_DEV_RST_CTRL_USXGMII_OSET_FILTER_DIS, x)
+
+#define DEV5G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS GENMASK(26, 25)
+#define DEV5G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS_SET(x)\
+ FIELD_PREP(DEV5G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS, x)
+#define DEV5G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS_GET(x)\
+ FIELD_GET(DEV5G_DEV_RST_CTRL_MUXED_USXGMII_NETWORK_PORTS, x)
+
+#define DEV5G_DEV_RST_CTRL_SERDES_SPEED_SEL GENMASK(24, 23)
+#define DEV5G_DEV_RST_CTRL_SERDES_SPEED_SEL_SET(x)\
+ FIELD_PREP(DEV5G_DEV_RST_CTRL_SERDES_SPEED_SEL, x)
+#define DEV5G_DEV_RST_CTRL_SERDES_SPEED_SEL_GET(x)\
+ FIELD_GET(DEV5G_DEV_RST_CTRL_SERDES_SPEED_SEL, x)
+
+#define DEV5G_DEV_RST_CTRL_SPEED_SEL GENMASK(22, 20)
+#define DEV5G_DEV_RST_CTRL_SPEED_SEL_SET(x)\
+ FIELD_PREP(DEV5G_DEV_RST_CTRL_SPEED_SEL, x)
+#define DEV5G_DEV_RST_CTRL_SPEED_SEL_GET(x)\
+ FIELD_GET(DEV5G_DEV_RST_CTRL_SPEED_SEL, x)
+
+#define DEV5G_DEV_RST_CTRL_PCS_TX_RST BIT(12)
+#define DEV5G_DEV_RST_CTRL_PCS_TX_RST_SET(x)\
+ FIELD_PREP(DEV5G_DEV_RST_CTRL_PCS_TX_RST, x)
+#define DEV5G_DEV_RST_CTRL_PCS_TX_RST_GET(x)\
+ FIELD_GET(DEV5G_DEV_RST_CTRL_PCS_TX_RST, x)
+
+#define DEV5G_DEV_RST_CTRL_PCS_RX_RST BIT(8)
+#define DEV5G_DEV_RST_CTRL_PCS_RX_RST_SET(x)\
+ FIELD_PREP(DEV5G_DEV_RST_CTRL_PCS_RX_RST, x)
+#define DEV5G_DEV_RST_CTRL_PCS_RX_RST_GET(x)\
+ FIELD_GET(DEV5G_DEV_RST_CTRL_PCS_RX_RST, x)
+
+#define DEV5G_DEV_RST_CTRL_MAC_TX_RST BIT(4)
+#define DEV5G_DEV_RST_CTRL_MAC_TX_RST_SET(x)\
+ FIELD_PREP(DEV5G_DEV_RST_CTRL_MAC_TX_RST, x)
+#define DEV5G_DEV_RST_CTRL_MAC_TX_RST_GET(x)\
+ FIELD_GET(DEV5G_DEV_RST_CTRL_MAC_TX_RST, x)
+
+#define DEV5G_DEV_RST_CTRL_MAC_RX_RST BIT(0)
+#define DEV5G_DEV_RST_CTRL_MAC_RX_RST_SET(x)\
+ FIELD_PREP(DEV5G_DEV_RST_CTRL_MAC_RX_RST, x)
+#define DEV5G_DEV_RST_CTRL_MAC_RX_RST_GET(x)\
+ FIELD_GET(DEV5G_DEV_RST_CTRL_MAC_RX_RST, x)
+
+/* DSM:RAM_CTRL:RAM_INIT */
+#define DSM_RAM_INIT __REG(TARGET_DSM, 0, 1, 0, 0, 1, 4, 0, 0, 1, 4)
+
+#define DSM_RAM_INIT_RAM_INIT BIT(1)
+#define DSM_RAM_INIT_RAM_INIT_SET(x)\
+ FIELD_PREP(DSM_RAM_INIT_RAM_INIT, x)
+#define DSM_RAM_INIT_RAM_INIT_GET(x)\
+ FIELD_GET(DSM_RAM_INIT_RAM_INIT, x)
+
+#define DSM_RAM_INIT_RAM_CFG_HOOK BIT(0)
+#define DSM_RAM_INIT_RAM_CFG_HOOK_SET(x)\
+ FIELD_PREP(DSM_RAM_INIT_RAM_CFG_HOOK, x)
+#define DSM_RAM_INIT_RAM_CFG_HOOK_GET(x)\
+ FIELD_GET(DSM_RAM_INIT_RAM_CFG_HOOK, x)
+
+/* DSM:CFG:BUF_CFG */
+#define DSM_BUF_CFG(r) __REG(TARGET_DSM, 0, 1, 20, 0, 1, 3528, 0, r, 67, 4)
+
+#define DSM_BUF_CFG_CSC_STAT_DIS BIT(13)
+#define DSM_BUF_CFG_CSC_STAT_DIS_SET(x)\
+ FIELD_PREP(DSM_BUF_CFG_CSC_STAT_DIS, x)
+#define DSM_BUF_CFG_CSC_STAT_DIS_GET(x)\
+ FIELD_GET(DSM_BUF_CFG_CSC_STAT_DIS, x)
+
+#define DSM_BUF_CFG_AGING_ENA BIT(12)
+#define DSM_BUF_CFG_AGING_ENA_SET(x)\
+ FIELD_PREP(DSM_BUF_CFG_AGING_ENA, x)
+#define DSM_BUF_CFG_AGING_ENA_GET(x)\
+ FIELD_GET(DSM_BUF_CFG_AGING_ENA, x)
+
+#define DSM_BUF_CFG_UNDERFLOW_WATCHDOG_DIS BIT(11)
+#define DSM_BUF_CFG_UNDERFLOW_WATCHDOG_DIS_SET(x)\
+ FIELD_PREP(DSM_BUF_CFG_UNDERFLOW_WATCHDOG_DIS, x)
+#define DSM_BUF_CFG_UNDERFLOW_WATCHDOG_DIS_GET(x)\
+ FIELD_GET(DSM_BUF_CFG_UNDERFLOW_WATCHDOG_DIS, x)
+
+#define DSM_BUF_CFG_UNDERFLOW_WATCHDOG_TIMEOUT GENMASK(10, 0)
+#define DSM_BUF_CFG_UNDERFLOW_WATCHDOG_TIMEOUT_SET(x)\
+ FIELD_PREP(DSM_BUF_CFG_UNDERFLOW_WATCHDOG_TIMEOUT, x)
+#define DSM_BUF_CFG_UNDERFLOW_WATCHDOG_TIMEOUT_GET(x)\
+ FIELD_GET(DSM_BUF_CFG_UNDERFLOW_WATCHDOG_TIMEOUT, x)
+
+/* DSM:CFG:DEV_TX_STOP_WM_CFG */
+#define DSM_DEV_TX_STOP_WM_CFG(r) __REG(TARGET_DSM, 0, 1, 20, 0, 1, 3528, 1360, r, 67, 4)
+
+#define DSM_DEV_TX_STOP_WM_CFG_FAST_STARTUP_ENA BIT(9)
+#define DSM_DEV_TX_STOP_WM_CFG_FAST_STARTUP_ENA_SET(x)\
+ FIELD_PREP(DSM_DEV_TX_STOP_WM_CFG_FAST_STARTUP_ENA, x)
+#define DSM_DEV_TX_STOP_WM_CFG_FAST_STARTUP_ENA_GET(x)\
+ FIELD_GET(DSM_DEV_TX_STOP_WM_CFG_FAST_STARTUP_ENA, x)
+
+#define DSM_DEV_TX_STOP_WM_CFG_DEV10G_SHADOW_ENA BIT(8)
+#define DSM_DEV_TX_STOP_WM_CFG_DEV10G_SHADOW_ENA_SET(x)\
+ FIELD_PREP(DSM_DEV_TX_STOP_WM_CFG_DEV10G_SHADOW_ENA, x)
+#define DSM_DEV_TX_STOP_WM_CFG_DEV10G_SHADOW_ENA_GET(x)\
+ FIELD_GET(DSM_DEV_TX_STOP_WM_CFG_DEV10G_SHADOW_ENA, x)
+
+#define DSM_DEV_TX_STOP_WM_CFG_DEV_TX_STOP_WM GENMASK(7, 1)
+#define DSM_DEV_TX_STOP_WM_CFG_DEV_TX_STOP_WM_SET(x)\
+ FIELD_PREP(DSM_DEV_TX_STOP_WM_CFG_DEV_TX_STOP_WM, x)
+#define DSM_DEV_TX_STOP_WM_CFG_DEV_TX_STOP_WM_GET(x)\
+ FIELD_GET(DSM_DEV_TX_STOP_WM_CFG_DEV_TX_STOP_WM, x)
+
+#define DSM_DEV_TX_STOP_WM_CFG_DEV_TX_CNT_CLR BIT(0)
+#define DSM_DEV_TX_STOP_WM_CFG_DEV_TX_CNT_CLR_SET(x)\
+ FIELD_PREP(DSM_DEV_TX_STOP_WM_CFG_DEV_TX_CNT_CLR, x)
+#define DSM_DEV_TX_STOP_WM_CFG_DEV_TX_CNT_CLR_GET(x)\
+ FIELD_GET(DSM_DEV_TX_STOP_WM_CFG_DEV_TX_CNT_CLR, x)
+
+/* DSM:CFG:RX_PAUSE_CFG */
+#define DSM_RX_PAUSE_CFG(r) __REG(TARGET_DSM, 0, 1, 20, 0, 1, 3528, 1628, r, 67, 4)
+
+#define DSM_RX_PAUSE_CFG_RX_PAUSE_EN BIT(1)
+#define DSM_RX_PAUSE_CFG_RX_PAUSE_EN_SET(x)\
+ FIELD_PREP(DSM_RX_PAUSE_CFG_RX_PAUSE_EN, x)
+#define DSM_RX_PAUSE_CFG_RX_PAUSE_EN_GET(x)\
+ FIELD_GET(DSM_RX_PAUSE_CFG_RX_PAUSE_EN, x)
+
+#define DSM_RX_PAUSE_CFG_FC_OBEY_LOCAL BIT(0)
+#define DSM_RX_PAUSE_CFG_FC_OBEY_LOCAL_SET(x)\
+ FIELD_PREP(DSM_RX_PAUSE_CFG_FC_OBEY_LOCAL, x)
+#define DSM_RX_PAUSE_CFG_FC_OBEY_LOCAL_GET(x)\
+ FIELD_GET(DSM_RX_PAUSE_CFG_FC_OBEY_LOCAL, x)
+
+/* DSM:CFG:MAC_CFG */
+#define DSM_MAC_CFG(r) __REG(TARGET_DSM, 0, 1, 20, 0, 1, 3528, 2432, r, 67, 4)
+
+#define DSM_MAC_CFG_TX_PAUSE_VAL GENMASK(31, 16)
+#define DSM_MAC_CFG_TX_PAUSE_VAL_SET(x)\
+ FIELD_PREP(DSM_MAC_CFG_TX_PAUSE_VAL, x)
+#define DSM_MAC_CFG_TX_PAUSE_VAL_GET(x)\
+ FIELD_GET(DSM_MAC_CFG_TX_PAUSE_VAL, x)
+
+#define DSM_MAC_CFG_HDX_BACKPREASSURE BIT(2)
+#define DSM_MAC_CFG_HDX_BACKPREASSURE_SET(x)\
+ FIELD_PREP(DSM_MAC_CFG_HDX_BACKPREASSURE, x)
+#define DSM_MAC_CFG_HDX_BACKPREASSURE_GET(x)\
+ FIELD_GET(DSM_MAC_CFG_HDX_BACKPREASSURE, x)
+
+#define DSM_MAC_CFG_SEND_PAUSE_FRM_TWICE BIT(1)
+#define DSM_MAC_CFG_SEND_PAUSE_FRM_TWICE_SET(x)\
+ FIELD_PREP(DSM_MAC_CFG_SEND_PAUSE_FRM_TWICE, x)
+#define DSM_MAC_CFG_SEND_PAUSE_FRM_TWICE_GET(x)\
+ FIELD_GET(DSM_MAC_CFG_SEND_PAUSE_FRM_TWICE, x)
+
+#define DSM_MAC_CFG_TX_PAUSE_XON_XOFF BIT(0)
+#define DSM_MAC_CFG_TX_PAUSE_XON_XOFF_SET(x)\
+ FIELD_PREP(DSM_MAC_CFG_TX_PAUSE_XON_XOFF, x)
+#define DSM_MAC_CFG_TX_PAUSE_XON_XOFF_GET(x)\
+ FIELD_GET(DSM_MAC_CFG_TX_PAUSE_XON_XOFF, x)
+
+/* DSM:CFG:MAC_ADDR_BASE_HIGH_CFG */
+#define DSM_MAC_ADDR_BASE_HIGH_CFG(r) __REG(TARGET_DSM, 0, 1, 20, 0, 1, 3528, 2700, r, 65, 4)
+
+#define DSM_MAC_ADDR_BASE_HIGH_CFG_MAC_ADDR_HIGH GENMASK(23, 0)
+#define DSM_MAC_ADDR_BASE_HIGH_CFG_MAC_ADDR_HIGH_SET(x)\
+ FIELD_PREP(DSM_MAC_ADDR_BASE_HIGH_CFG_MAC_ADDR_HIGH, x)
+#define DSM_MAC_ADDR_BASE_HIGH_CFG_MAC_ADDR_HIGH_GET(x)\
+ FIELD_GET(DSM_MAC_ADDR_BASE_HIGH_CFG_MAC_ADDR_HIGH, x)
+
+/* DSM:CFG:MAC_ADDR_BASE_LOW_CFG */
+#define DSM_MAC_ADDR_BASE_LOW_CFG(r) __REG(TARGET_DSM, 0, 1, 20, 0, 1, 3528, 2960, r, 65, 4)
+
+#define DSM_MAC_ADDR_BASE_LOW_CFG_MAC_ADDR_LOW GENMASK(23, 0)
+#define DSM_MAC_ADDR_BASE_LOW_CFG_MAC_ADDR_LOW_SET(x)\
+ FIELD_PREP(DSM_MAC_ADDR_BASE_LOW_CFG_MAC_ADDR_LOW, x)
+#define DSM_MAC_ADDR_BASE_LOW_CFG_MAC_ADDR_LOW_GET(x)\
+ FIELD_GET(DSM_MAC_ADDR_BASE_LOW_CFG_MAC_ADDR_LOW, x)
+
+/* DSM:CFG:TAXI_CAL_CFG */
+#define DSM_TAXI_CAL_CFG(r) __REG(TARGET_DSM, 0, 1, 20, 0, 1, 3528, 3224, r, 9, 4)
+
+#define DSM_TAXI_CAL_CFG_CAL_IDX GENMASK(20, 15)
+#define DSM_TAXI_CAL_CFG_CAL_IDX_SET(x)\
+ FIELD_PREP(DSM_TAXI_CAL_CFG_CAL_IDX, x)
+#define DSM_TAXI_CAL_CFG_CAL_IDX_GET(x)\
+ FIELD_GET(DSM_TAXI_CAL_CFG_CAL_IDX, x)
+
+#define DSM_TAXI_CAL_CFG_CAL_CUR_LEN GENMASK(14, 9)
+#define DSM_TAXI_CAL_CFG_CAL_CUR_LEN_SET(x)\
+ FIELD_PREP(DSM_TAXI_CAL_CFG_CAL_CUR_LEN, x)
+#define DSM_TAXI_CAL_CFG_CAL_CUR_LEN_GET(x)\
+ FIELD_GET(DSM_TAXI_CAL_CFG_CAL_CUR_LEN, x)
+
+#define DSM_TAXI_CAL_CFG_CAL_CUR_VAL GENMASK(8, 5)
+#define DSM_TAXI_CAL_CFG_CAL_CUR_VAL_SET(x)\
+ FIELD_PREP(DSM_TAXI_CAL_CFG_CAL_CUR_VAL, x)
+#define DSM_TAXI_CAL_CFG_CAL_CUR_VAL_GET(x)\
+ FIELD_GET(DSM_TAXI_CAL_CFG_CAL_CUR_VAL, x)
+
+#define DSM_TAXI_CAL_CFG_CAL_PGM_VAL GENMASK(4, 1)
+#define DSM_TAXI_CAL_CFG_CAL_PGM_VAL_SET(x)\
+ FIELD_PREP(DSM_TAXI_CAL_CFG_CAL_PGM_VAL, x)
+#define DSM_TAXI_CAL_CFG_CAL_PGM_VAL_GET(x)\
+ FIELD_GET(DSM_TAXI_CAL_CFG_CAL_PGM_VAL, x)
+
+#define DSM_TAXI_CAL_CFG_CAL_PGM_ENA BIT(0)
+#define DSM_TAXI_CAL_CFG_CAL_PGM_ENA_SET(x)\
+ FIELD_PREP(DSM_TAXI_CAL_CFG_CAL_PGM_ENA, x)
+#define DSM_TAXI_CAL_CFG_CAL_PGM_ENA_GET(x)\
+ FIELD_GET(DSM_TAXI_CAL_CFG_CAL_PGM_ENA, x)
+
+/* EACL:POL_CFG:POL_EACL_CFG */
+#define EACL_POL_EACL_CFG __REG(TARGET_EACL, 0, 1, 150608, 0, 1, 780, 768, 0, 1, 4)
+
+#define EACL_POL_EACL_CFG_EACL_CNT_MARKED_AS_DROPPED BIT(5)
+#define EACL_POL_EACL_CFG_EACL_CNT_MARKED_AS_DROPPED_SET(x)\
+ FIELD_PREP(EACL_POL_EACL_CFG_EACL_CNT_MARKED_AS_DROPPED, x)
+#define EACL_POL_EACL_CFG_EACL_CNT_MARKED_AS_DROPPED_GET(x)\
+ FIELD_GET(EACL_POL_EACL_CFG_EACL_CNT_MARKED_AS_DROPPED, x)
+
+#define EACL_POL_EACL_CFG_EACL_ALLOW_FP_COPY BIT(4)
+#define EACL_POL_EACL_CFG_EACL_ALLOW_FP_COPY_SET(x)\
+ FIELD_PREP(EACL_POL_EACL_CFG_EACL_ALLOW_FP_COPY, x)
+#define EACL_POL_EACL_CFG_EACL_ALLOW_FP_COPY_GET(x)\
+ FIELD_GET(EACL_POL_EACL_CFG_EACL_ALLOW_FP_COPY, x)
+
+#define EACL_POL_EACL_CFG_EACL_ALLOW_CPU_COPY BIT(3)
+#define EACL_POL_EACL_CFG_EACL_ALLOW_CPU_COPY_SET(x)\
+ FIELD_PREP(EACL_POL_EACL_CFG_EACL_ALLOW_CPU_COPY, x)
+#define EACL_POL_EACL_CFG_EACL_ALLOW_CPU_COPY_GET(x)\
+ FIELD_GET(EACL_POL_EACL_CFG_EACL_ALLOW_CPU_COPY, x)
+
+#define EACL_POL_EACL_CFG_EACL_FORCE_CLOSE BIT(2)
+#define EACL_POL_EACL_CFG_EACL_FORCE_CLOSE_SET(x)\
+ FIELD_PREP(EACL_POL_EACL_CFG_EACL_FORCE_CLOSE, x)
+#define EACL_POL_EACL_CFG_EACL_FORCE_CLOSE_GET(x)\
+ FIELD_GET(EACL_POL_EACL_CFG_EACL_FORCE_CLOSE, x)
+
+#define EACL_POL_EACL_CFG_EACL_FORCE_OPEN BIT(1)
+#define EACL_POL_EACL_CFG_EACL_FORCE_OPEN_SET(x)\
+ FIELD_PREP(EACL_POL_EACL_CFG_EACL_FORCE_OPEN, x)
+#define EACL_POL_EACL_CFG_EACL_FORCE_OPEN_GET(x)\
+ FIELD_GET(EACL_POL_EACL_CFG_EACL_FORCE_OPEN, x)
+
+#define EACL_POL_EACL_CFG_EACL_FORCE_INIT BIT(0)
+#define EACL_POL_EACL_CFG_EACL_FORCE_INIT_SET(x)\
+ FIELD_PREP(EACL_POL_EACL_CFG_EACL_FORCE_INIT, x)
+#define EACL_POL_EACL_CFG_EACL_FORCE_INIT_GET(x)\
+ FIELD_GET(EACL_POL_EACL_CFG_EACL_FORCE_INIT, x)
+
+/* EACL:RAM_CTRL:RAM_INIT */
+#define EACL_RAM_INIT __REG(TARGET_EACL, 0, 1, 118736, 0, 1, 4, 0, 0, 1, 4)
+
+#define EACL_RAM_INIT_RAM_INIT BIT(1)
+#define EACL_RAM_INIT_RAM_INIT_SET(x)\
+ FIELD_PREP(EACL_RAM_INIT_RAM_INIT, x)
+#define EACL_RAM_INIT_RAM_INIT_GET(x)\
+ FIELD_GET(EACL_RAM_INIT_RAM_INIT, x)
+
+#define EACL_RAM_INIT_RAM_CFG_HOOK BIT(0)
+#define EACL_RAM_INIT_RAM_CFG_HOOK_SET(x)\
+ FIELD_PREP(EACL_RAM_INIT_RAM_CFG_HOOK, x)
+#define EACL_RAM_INIT_RAM_CFG_HOOK_GET(x)\
+ FIELD_GET(EACL_RAM_INIT_RAM_CFG_HOOK, x)
+
+/* FDMA:FDMA:FDMA_CH_ACTIVATE */
+#define FDMA_CH_ACTIVATE __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 0, 0, 1, 4)
+
+#define FDMA_CH_ACTIVATE_CH_ACTIVATE GENMASK(7, 0)
+#define FDMA_CH_ACTIVATE_CH_ACTIVATE_SET(x)\
+ FIELD_PREP(FDMA_CH_ACTIVATE_CH_ACTIVATE, x)
+#define FDMA_CH_ACTIVATE_CH_ACTIVATE_GET(x)\
+ FIELD_GET(FDMA_CH_ACTIVATE_CH_ACTIVATE, x)
+
+/* FDMA:FDMA:FDMA_CH_RELOAD */
+#define FDMA_CH_RELOAD __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 4, 0, 1, 4)
+
+#define FDMA_CH_RELOAD_CH_RELOAD GENMASK(7, 0)
+#define FDMA_CH_RELOAD_CH_RELOAD_SET(x)\
+ FIELD_PREP(FDMA_CH_RELOAD_CH_RELOAD, x)
+#define FDMA_CH_RELOAD_CH_RELOAD_GET(x)\
+ FIELD_GET(FDMA_CH_RELOAD_CH_RELOAD, x)
+
+/* FDMA:FDMA:FDMA_CH_DISABLE */
+#define FDMA_CH_DISABLE __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 8, 0, 1, 4)
+
+#define FDMA_CH_DISABLE_CH_DISABLE GENMASK(7, 0)
+#define FDMA_CH_DISABLE_CH_DISABLE_SET(x)\
+ FIELD_PREP(FDMA_CH_DISABLE_CH_DISABLE, x)
+#define FDMA_CH_DISABLE_CH_DISABLE_GET(x)\
+ FIELD_GET(FDMA_CH_DISABLE_CH_DISABLE, x)
+
+/* FDMA:FDMA:FDMA_DCB_LLP */
+#define FDMA_DCB_LLP(r) __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 52, r, 8, 4)
+
+/* FDMA:FDMA:FDMA_DCB_LLP1 */
+#define FDMA_DCB_LLP1(r) __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 84, r, 8, 4)
+
+/* FDMA:FDMA:FDMA_DCB_LLP_PREV */
+#define FDMA_DCB_LLP_PREV(r) __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 116, r, 8, 4)
+
+/* FDMA:FDMA:FDMA_DCB_LLP_PREV1 */
+#define FDMA_DCB_LLP_PREV1(r) __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 148, r, 8, 4)
+
+/* FDMA:FDMA:FDMA_CH_CFG */
+#define FDMA_CH_CFG(r) __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 224, r, 8, 4)
+
+#define FDMA_CH_CFG_CH_XTR_STATUS_MODE BIT(7)
+#define FDMA_CH_CFG_CH_XTR_STATUS_MODE_SET(x)\
+ FIELD_PREP(FDMA_CH_CFG_CH_XTR_STATUS_MODE, x)
+#define FDMA_CH_CFG_CH_XTR_STATUS_MODE_GET(x)\
+ FIELD_GET(FDMA_CH_CFG_CH_XTR_STATUS_MODE, x)
+
+#define FDMA_CH_CFG_CH_INTR_DB_EOF_ONLY BIT(6)
+#define FDMA_CH_CFG_CH_INTR_DB_EOF_ONLY_SET(x)\
+ FIELD_PREP(FDMA_CH_CFG_CH_INTR_DB_EOF_ONLY, x)
+#define FDMA_CH_CFG_CH_INTR_DB_EOF_ONLY_GET(x)\
+ FIELD_GET(FDMA_CH_CFG_CH_INTR_DB_EOF_ONLY, x)
+
+#define FDMA_CH_CFG_CH_INJ_PORT BIT(5)
+#define FDMA_CH_CFG_CH_INJ_PORT_SET(x)\
+ FIELD_PREP(FDMA_CH_CFG_CH_INJ_PORT, x)
+#define FDMA_CH_CFG_CH_INJ_PORT_GET(x)\
+ FIELD_GET(FDMA_CH_CFG_CH_INJ_PORT, x)
+
+#define FDMA_CH_CFG_CH_DCB_DB_CNT GENMASK(4, 1)
+#define FDMA_CH_CFG_CH_DCB_DB_CNT_SET(x)\
+ FIELD_PREP(FDMA_CH_CFG_CH_DCB_DB_CNT, x)
+#define FDMA_CH_CFG_CH_DCB_DB_CNT_GET(x)\
+ FIELD_GET(FDMA_CH_CFG_CH_DCB_DB_CNT, x)
+
+#define FDMA_CH_CFG_CH_MEM BIT(0)
+#define FDMA_CH_CFG_CH_MEM_SET(x)\
+ FIELD_PREP(FDMA_CH_CFG_CH_MEM, x)
+#define FDMA_CH_CFG_CH_MEM_GET(x)\
+ FIELD_GET(FDMA_CH_CFG_CH_MEM, x)
+
+/* FDMA:FDMA:FDMA_CH_TRANSLATE */
+#define FDMA_CH_TRANSLATE(r) __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 256, r, 8, 4)
+
+#define FDMA_CH_TRANSLATE_OFFSET GENMASK(15, 0)
+#define FDMA_CH_TRANSLATE_OFFSET_SET(x)\
+ FIELD_PREP(FDMA_CH_TRANSLATE_OFFSET, x)
+#define FDMA_CH_TRANSLATE_OFFSET_GET(x)\
+ FIELD_GET(FDMA_CH_TRANSLATE_OFFSET, x)
+
+/* FDMA:FDMA:FDMA_XTR_CFG */
+#define FDMA_XTR_CFG __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 364, 0, 1, 4)
+
+#define FDMA_XTR_CFG_XTR_FIFO_WM GENMASK(15, 11)
+#define FDMA_XTR_CFG_XTR_FIFO_WM_SET(x)\
+ FIELD_PREP(FDMA_XTR_CFG_XTR_FIFO_WM, x)
+#define FDMA_XTR_CFG_XTR_FIFO_WM_GET(x)\
+ FIELD_GET(FDMA_XTR_CFG_XTR_FIFO_WM, x)
+
+#define FDMA_XTR_CFG_XTR_ARB_SAT GENMASK(10, 0)
+#define FDMA_XTR_CFG_XTR_ARB_SAT_SET(x)\
+ FIELD_PREP(FDMA_XTR_CFG_XTR_ARB_SAT, x)
+#define FDMA_XTR_CFG_XTR_ARB_SAT_GET(x)\
+ FIELD_GET(FDMA_XTR_CFG_XTR_ARB_SAT, x)
+
+/* FDMA:FDMA:FDMA_PORT_CTRL */
+#define FDMA_PORT_CTRL(r) __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 376, r, 2, 4)
+
+#define FDMA_PORT_CTRL_INJ_STOP BIT(4)
+#define FDMA_PORT_CTRL_INJ_STOP_SET(x)\
+ FIELD_PREP(FDMA_PORT_CTRL_INJ_STOP, x)
+#define FDMA_PORT_CTRL_INJ_STOP_GET(x)\
+ FIELD_GET(FDMA_PORT_CTRL_INJ_STOP, x)
+
+#define FDMA_PORT_CTRL_INJ_STOP_FORCE BIT(3)
+#define FDMA_PORT_CTRL_INJ_STOP_FORCE_SET(x)\
+ FIELD_PREP(FDMA_PORT_CTRL_INJ_STOP_FORCE, x)
+#define FDMA_PORT_CTRL_INJ_STOP_FORCE_GET(x)\
+ FIELD_GET(FDMA_PORT_CTRL_INJ_STOP_FORCE, x)
+
+#define FDMA_PORT_CTRL_XTR_STOP BIT(2)
+#define FDMA_PORT_CTRL_XTR_STOP_SET(x)\
+ FIELD_PREP(FDMA_PORT_CTRL_XTR_STOP, x)
+#define FDMA_PORT_CTRL_XTR_STOP_GET(x)\
+ FIELD_GET(FDMA_PORT_CTRL_XTR_STOP, x)
+
+#define FDMA_PORT_CTRL_XTR_BUF_IS_EMPTY BIT(1)
+#define FDMA_PORT_CTRL_XTR_BUF_IS_EMPTY_SET(x)\
+ FIELD_PREP(FDMA_PORT_CTRL_XTR_BUF_IS_EMPTY, x)
+#define FDMA_PORT_CTRL_XTR_BUF_IS_EMPTY_GET(x)\
+ FIELD_GET(FDMA_PORT_CTRL_XTR_BUF_IS_EMPTY, x)
+
+#define FDMA_PORT_CTRL_XTR_BUF_RST BIT(0)
+#define FDMA_PORT_CTRL_XTR_BUF_RST_SET(x)\
+ FIELD_PREP(FDMA_PORT_CTRL_XTR_BUF_RST, x)
+#define FDMA_PORT_CTRL_XTR_BUF_RST_GET(x)\
+ FIELD_GET(FDMA_PORT_CTRL_XTR_BUF_RST, x)
+
+/* FDMA:FDMA:FDMA_INTR_DCB */
+#define FDMA_INTR_DCB __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 384, 0, 1, 4)
+
+#define FDMA_INTR_DCB_INTR_DCB GENMASK(7, 0)
+#define FDMA_INTR_DCB_INTR_DCB_SET(x)\
+ FIELD_PREP(FDMA_INTR_DCB_INTR_DCB, x)
+#define FDMA_INTR_DCB_INTR_DCB_GET(x)\
+ FIELD_GET(FDMA_INTR_DCB_INTR_DCB, x)
+
+/* FDMA:FDMA:FDMA_INTR_DCB_ENA */
+#define FDMA_INTR_DCB_ENA __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 388, 0, 1, 4)
+
+#define FDMA_INTR_DCB_ENA_INTR_DCB_ENA GENMASK(7, 0)
+#define FDMA_INTR_DCB_ENA_INTR_DCB_ENA_SET(x)\
+ FIELD_PREP(FDMA_INTR_DCB_ENA_INTR_DCB_ENA, x)
+#define FDMA_INTR_DCB_ENA_INTR_DCB_ENA_GET(x)\
+ FIELD_GET(FDMA_INTR_DCB_ENA_INTR_DCB_ENA, x)
+
+/* FDMA:FDMA:FDMA_INTR_DB */
+#define FDMA_INTR_DB __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 392, 0, 1, 4)
+
+#define FDMA_INTR_DB_INTR_DB GENMASK(7, 0)
+#define FDMA_INTR_DB_INTR_DB_SET(x)\
+ FIELD_PREP(FDMA_INTR_DB_INTR_DB, x)
+#define FDMA_INTR_DB_INTR_DB_GET(x)\
+ FIELD_GET(FDMA_INTR_DB_INTR_DB, x)
+
+/* FDMA:FDMA:FDMA_INTR_DB_ENA */
+#define FDMA_INTR_DB_ENA __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 396, 0, 1, 4)
+
+#define FDMA_INTR_DB_ENA_INTR_DB_ENA GENMASK(7, 0)
+#define FDMA_INTR_DB_ENA_INTR_DB_ENA_SET(x)\
+ FIELD_PREP(FDMA_INTR_DB_ENA_INTR_DB_ENA, x)
+#define FDMA_INTR_DB_ENA_INTR_DB_ENA_GET(x)\
+ FIELD_GET(FDMA_INTR_DB_ENA_INTR_DB_ENA, x)
+
+/* FDMA:FDMA:FDMA_INTR_ERR */
+#define FDMA_INTR_ERR __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 400, 0, 1, 4)
+
+#define FDMA_INTR_ERR_INTR_PORT_ERR GENMASK(9, 8)
+#define FDMA_INTR_ERR_INTR_PORT_ERR_SET(x)\
+ FIELD_PREP(FDMA_INTR_ERR_INTR_PORT_ERR, x)
+#define FDMA_INTR_ERR_INTR_PORT_ERR_GET(x)\
+ FIELD_GET(FDMA_INTR_ERR_INTR_PORT_ERR, x)
+
+#define FDMA_INTR_ERR_INTR_CH_ERR GENMASK(7, 0)
+#define FDMA_INTR_ERR_INTR_CH_ERR_SET(x)\
+ FIELD_PREP(FDMA_INTR_ERR_INTR_CH_ERR, x)
+#define FDMA_INTR_ERR_INTR_CH_ERR_GET(x)\
+ FIELD_GET(FDMA_INTR_ERR_INTR_CH_ERR, x)
+
+/* FDMA:FDMA:FDMA_ERRORS */
+#define FDMA_ERRORS __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 412, 0, 1, 4)
+
+#define FDMA_ERRORS_ERR_XTR_WR GENMASK(31, 30)
+#define FDMA_ERRORS_ERR_XTR_WR_SET(x)\
+ FIELD_PREP(FDMA_ERRORS_ERR_XTR_WR, x)
+#define FDMA_ERRORS_ERR_XTR_WR_GET(x)\
+ FIELD_GET(FDMA_ERRORS_ERR_XTR_WR, x)
+
+#define FDMA_ERRORS_ERR_XTR_OVF GENMASK(29, 28)
+#define FDMA_ERRORS_ERR_XTR_OVF_SET(x)\
+ FIELD_PREP(FDMA_ERRORS_ERR_XTR_OVF, x)
+#define FDMA_ERRORS_ERR_XTR_OVF_GET(x)\
+ FIELD_GET(FDMA_ERRORS_ERR_XTR_OVF, x)
+
+#define FDMA_ERRORS_ERR_XTR_TAXI32_OVF GENMASK(27, 26)
+#define FDMA_ERRORS_ERR_XTR_TAXI32_OVF_SET(x)\
+ FIELD_PREP(FDMA_ERRORS_ERR_XTR_TAXI32_OVF, x)
+#define FDMA_ERRORS_ERR_XTR_TAXI32_OVF_GET(x)\
+ FIELD_GET(FDMA_ERRORS_ERR_XTR_TAXI32_OVF, x)
+
+#define FDMA_ERRORS_ERR_DCB_XTR_DATAL GENMASK(25, 24)
+#define FDMA_ERRORS_ERR_DCB_XTR_DATAL_SET(x)\
+ FIELD_PREP(FDMA_ERRORS_ERR_DCB_XTR_DATAL, x)
+#define FDMA_ERRORS_ERR_DCB_XTR_DATAL_GET(x)\
+ FIELD_GET(FDMA_ERRORS_ERR_DCB_XTR_DATAL, x)
+
+#define FDMA_ERRORS_ERR_DCB_RD GENMASK(23, 16)
+#define FDMA_ERRORS_ERR_DCB_RD_SET(x)\
+ FIELD_PREP(FDMA_ERRORS_ERR_DCB_RD, x)
+#define FDMA_ERRORS_ERR_DCB_RD_GET(x)\
+ FIELD_GET(FDMA_ERRORS_ERR_DCB_RD, x)
+
+#define FDMA_ERRORS_ERR_INJ_RD GENMASK(15, 10)
+#define FDMA_ERRORS_ERR_INJ_RD_SET(x)\
+ FIELD_PREP(FDMA_ERRORS_ERR_INJ_RD, x)
+#define FDMA_ERRORS_ERR_INJ_RD_GET(x)\
+ FIELD_GET(FDMA_ERRORS_ERR_INJ_RD, x)
+
+#define FDMA_ERRORS_ERR_INJ_OUT_OF_SYNC GENMASK(9, 8)
+#define FDMA_ERRORS_ERR_INJ_OUT_OF_SYNC_SET(x)\
+ FIELD_PREP(FDMA_ERRORS_ERR_INJ_OUT_OF_SYNC, x)
+#define FDMA_ERRORS_ERR_INJ_OUT_OF_SYNC_GET(x)\
+ FIELD_GET(FDMA_ERRORS_ERR_INJ_OUT_OF_SYNC, x)
+
+#define FDMA_ERRORS_ERR_CH_WR GENMASK(7, 0)
+#define FDMA_ERRORS_ERR_CH_WR_SET(x)\
+ FIELD_PREP(FDMA_ERRORS_ERR_CH_WR, x)
+#define FDMA_ERRORS_ERR_CH_WR_GET(x)\
+ FIELD_GET(FDMA_ERRORS_ERR_CH_WR, x)
+
+/* FDMA:FDMA:FDMA_ERRORS_2 */
+#define FDMA_ERRORS_2 __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 416, 0, 1, 4)
+
+#define FDMA_ERRORS_2_ERR_XTR_FRAG GENMASK(1, 0)
+#define FDMA_ERRORS_2_ERR_XTR_FRAG_SET(x)\
+ FIELD_PREP(FDMA_ERRORS_2_ERR_XTR_FRAG, x)
+#define FDMA_ERRORS_2_ERR_XTR_FRAG_GET(x)\
+ FIELD_GET(FDMA_ERRORS_2_ERR_XTR_FRAG, x)
+
+/* FDMA:FDMA:FDMA_CTRL */
+#define FDMA_CTRL __REG(TARGET_FDMA, 0, 1, 8, 0, 1, 428, 424, 0, 1, 4)
+
+#define FDMA_CTRL_NRESET BIT(0)
+#define FDMA_CTRL_NRESET_SET(x)\
+ FIELD_PREP(FDMA_CTRL_NRESET, x)
+#define FDMA_CTRL_NRESET_GET(x)\
+ FIELD_GET(FDMA_CTRL_NRESET, x)
+
+/* DEVCPU_GCB:CHIP_REGS:CHIP_ID */
+#define GCB_CHIP_ID __REG(TARGET_GCB, 0, 1, 0, 0, 1, 424, 0, 0, 1, 4)
+
+#define GCB_CHIP_ID_REV_ID GENMASK(31, 28)
+#define GCB_CHIP_ID_REV_ID_SET(x)\
+ FIELD_PREP(GCB_CHIP_ID_REV_ID, x)
+#define GCB_CHIP_ID_REV_ID_GET(x)\
+ FIELD_GET(GCB_CHIP_ID_REV_ID, x)
+
+#define GCB_CHIP_ID_PART_ID GENMASK(27, 12)
+#define GCB_CHIP_ID_PART_ID_SET(x)\
+ FIELD_PREP(GCB_CHIP_ID_PART_ID, x)
+#define GCB_CHIP_ID_PART_ID_GET(x)\
+ FIELD_GET(GCB_CHIP_ID_PART_ID, x)
+
+#define GCB_CHIP_ID_MFG_ID GENMASK(11, 1)
+#define GCB_CHIP_ID_MFG_ID_SET(x)\
+ FIELD_PREP(GCB_CHIP_ID_MFG_ID, x)
+#define GCB_CHIP_ID_MFG_ID_GET(x)\
+ FIELD_GET(GCB_CHIP_ID_MFG_ID, x)
+
+#define GCB_CHIP_ID_ONE BIT(0)
+#define GCB_CHIP_ID_ONE_SET(x)\
+ FIELD_PREP(GCB_CHIP_ID_ONE, x)
+#define GCB_CHIP_ID_ONE_GET(x)\
+ FIELD_GET(GCB_CHIP_ID_ONE, x)
+
+/* DEVCPU_GCB:CHIP_REGS:SOFT_RST */
+#define GCB_SOFT_RST __REG(TARGET_GCB, 0, 1, 0, 0, 1, 424, 8, 0, 1, 4)
+
+#define GCB_SOFT_RST_SOFT_NON_CFG_RST BIT(2)
+#define GCB_SOFT_RST_SOFT_NON_CFG_RST_SET(x)\
+ FIELD_PREP(GCB_SOFT_RST_SOFT_NON_CFG_RST, x)
+#define GCB_SOFT_RST_SOFT_NON_CFG_RST_GET(x)\
+ FIELD_GET(GCB_SOFT_RST_SOFT_NON_CFG_RST, x)
+
+#define GCB_SOFT_RST_SOFT_SWC_RST BIT(1)
+#define GCB_SOFT_RST_SOFT_SWC_RST_SET(x)\
+ FIELD_PREP(GCB_SOFT_RST_SOFT_SWC_RST, x)
+#define GCB_SOFT_RST_SOFT_SWC_RST_GET(x)\
+ FIELD_GET(GCB_SOFT_RST_SOFT_SWC_RST, x)
+
+#define GCB_SOFT_RST_SOFT_CHIP_RST BIT(0)
+#define GCB_SOFT_RST_SOFT_CHIP_RST_SET(x)\
+ FIELD_PREP(GCB_SOFT_RST_SOFT_CHIP_RST, x)
+#define GCB_SOFT_RST_SOFT_CHIP_RST_GET(x)\
+ FIELD_GET(GCB_SOFT_RST_SOFT_CHIP_RST, x)
+
+/* DEVCPU_GCB:CHIP_REGS:HW_SGPIO_SD_CFG */
+#define GCB_HW_SGPIO_SD_CFG __REG(TARGET_GCB, 0, 1, 0, 0, 1, 424, 20, 0, 1, 4)
+
+#define GCB_HW_SGPIO_SD_CFG_SD_HIGH_ENA BIT(1)
+#define GCB_HW_SGPIO_SD_CFG_SD_HIGH_ENA_SET(x)\
+ FIELD_PREP(GCB_HW_SGPIO_SD_CFG_SD_HIGH_ENA, x)
+#define GCB_HW_SGPIO_SD_CFG_SD_HIGH_ENA_GET(x)\
+ FIELD_GET(GCB_HW_SGPIO_SD_CFG_SD_HIGH_ENA, x)
+
+#define GCB_HW_SGPIO_SD_CFG_SD_MAP_SEL BIT(0)
+#define GCB_HW_SGPIO_SD_CFG_SD_MAP_SEL_SET(x)\
+ FIELD_PREP(GCB_HW_SGPIO_SD_CFG_SD_MAP_SEL, x)
+#define GCB_HW_SGPIO_SD_CFG_SD_MAP_SEL_GET(x)\
+ FIELD_GET(GCB_HW_SGPIO_SD_CFG_SD_MAP_SEL, x)
+
+/* DEVCPU_GCB:CHIP_REGS:HW_SGPIO_TO_SD_MAP_CFG */
+#define GCB_HW_SGPIO_TO_SD_MAP_CFG(r) __REG(TARGET_GCB, 0, 1, 0, 0, 1, 424, 24, r, 65, 4)
+
+#define GCB_HW_SGPIO_TO_SD_MAP_CFG_SGPIO_TO_SD_SEL GENMASK(8, 0)
+#define GCB_HW_SGPIO_TO_SD_MAP_CFG_SGPIO_TO_SD_SEL_SET(x)\
+ FIELD_PREP(GCB_HW_SGPIO_TO_SD_MAP_CFG_SGPIO_TO_SD_SEL, x)
+#define GCB_HW_SGPIO_TO_SD_MAP_CFG_SGPIO_TO_SD_SEL_GET(x)\
+ FIELD_GET(GCB_HW_SGPIO_TO_SD_MAP_CFG_SGPIO_TO_SD_SEL, x)
+
+/* DEVCPU_GCB:SIO_CTRL:SIO_CLOCK */
+#define GCB_SIO_CLOCK(g) __REG(TARGET_GCB, 0, 1, 876, g, 3, 280, 20, 0, 1, 4)
+
+#define GCB_SIO_CLOCK_SIO_CLK_FREQ GENMASK(19, 8)
+#define GCB_SIO_CLOCK_SIO_CLK_FREQ_SET(x)\
+ FIELD_PREP(GCB_SIO_CLOCK_SIO_CLK_FREQ, x)
+#define GCB_SIO_CLOCK_SIO_CLK_FREQ_GET(x)\
+ FIELD_GET(GCB_SIO_CLOCK_SIO_CLK_FREQ, x)
+
+#define GCB_SIO_CLOCK_SYS_CLK_PERIOD GENMASK(7, 0)
+#define GCB_SIO_CLOCK_SYS_CLK_PERIOD_SET(x)\
+ FIELD_PREP(GCB_SIO_CLOCK_SYS_CLK_PERIOD, x)
+#define GCB_SIO_CLOCK_SYS_CLK_PERIOD_GET(x)\
+ FIELD_GET(GCB_SIO_CLOCK_SYS_CLK_PERIOD, x)
+
+/* HSCH:HSCH_MISC:SYS_CLK_PER */
+#define HSCH_SYS_CLK_PER __REG(TARGET_HSCH, 0, 1, 163104, 0, 1, 648, 640, 0, 1, 4)
+
+#define HSCH_SYS_CLK_PER_SYS_CLK_PER_100PS GENMASK(7, 0)
+#define HSCH_SYS_CLK_PER_SYS_CLK_PER_100PS_SET(x)\
+ FIELD_PREP(HSCH_SYS_CLK_PER_SYS_CLK_PER_100PS, x)
+#define HSCH_SYS_CLK_PER_SYS_CLK_PER_100PS_GET(x)\
+ FIELD_GET(HSCH_SYS_CLK_PER_SYS_CLK_PER_100PS, x)
+
+/* HSCH:SYSTEM:FLUSH_CTRL */
+#define HSCH_FLUSH_CTRL __REG(TARGET_HSCH, 0, 1, 184000, 0, 1, 312, 4, 0, 1, 4)
+
+#define HSCH_FLUSH_CTRL_FLUSH_ENA BIT(27)
+#define HSCH_FLUSH_CTRL_FLUSH_ENA_SET(x)\
+ FIELD_PREP(HSCH_FLUSH_CTRL_FLUSH_ENA, x)
+#define HSCH_FLUSH_CTRL_FLUSH_ENA_GET(x)\
+ FIELD_GET(HSCH_FLUSH_CTRL_FLUSH_ENA, x)
+
+#define HSCH_FLUSH_CTRL_FLUSH_SRC BIT(26)
+#define HSCH_FLUSH_CTRL_FLUSH_SRC_SET(x)\
+ FIELD_PREP(HSCH_FLUSH_CTRL_FLUSH_SRC, x)
+#define HSCH_FLUSH_CTRL_FLUSH_SRC_GET(x)\
+ FIELD_GET(HSCH_FLUSH_CTRL_FLUSH_SRC, x)
+
+#define HSCH_FLUSH_CTRL_FLUSH_DST BIT(25)
+#define HSCH_FLUSH_CTRL_FLUSH_DST_SET(x)\
+ FIELD_PREP(HSCH_FLUSH_CTRL_FLUSH_DST, x)
+#define HSCH_FLUSH_CTRL_FLUSH_DST_GET(x)\
+ FIELD_GET(HSCH_FLUSH_CTRL_FLUSH_DST, x)
+
+#define HSCH_FLUSH_CTRL_FLUSH_PORT GENMASK(24, 18)
+#define HSCH_FLUSH_CTRL_FLUSH_PORT_SET(x)\
+ FIELD_PREP(HSCH_FLUSH_CTRL_FLUSH_PORT, x)
+#define HSCH_FLUSH_CTRL_FLUSH_PORT_GET(x)\
+ FIELD_GET(HSCH_FLUSH_CTRL_FLUSH_PORT, x)
+
+#define HSCH_FLUSH_CTRL_FLUSH_QUEUE BIT(17)
+#define HSCH_FLUSH_CTRL_FLUSH_QUEUE_SET(x)\
+ FIELD_PREP(HSCH_FLUSH_CTRL_FLUSH_QUEUE, x)
+#define HSCH_FLUSH_CTRL_FLUSH_QUEUE_GET(x)\
+ FIELD_GET(HSCH_FLUSH_CTRL_FLUSH_QUEUE, x)
+
+#define HSCH_FLUSH_CTRL_FLUSH_SE BIT(16)
+#define HSCH_FLUSH_CTRL_FLUSH_SE_SET(x)\
+ FIELD_PREP(HSCH_FLUSH_CTRL_FLUSH_SE, x)
+#define HSCH_FLUSH_CTRL_FLUSH_SE_GET(x)\
+ FIELD_GET(HSCH_FLUSH_CTRL_FLUSH_SE, x)
+
+#define HSCH_FLUSH_CTRL_FLUSH_HIER GENMASK(15, 0)
+#define HSCH_FLUSH_CTRL_FLUSH_HIER_SET(x)\
+ FIELD_PREP(HSCH_FLUSH_CTRL_FLUSH_HIER, x)
+#define HSCH_FLUSH_CTRL_FLUSH_HIER_GET(x)\
+ FIELD_GET(HSCH_FLUSH_CTRL_FLUSH_HIER, x)
+
+/* HSCH:SYSTEM:PORT_MODE */
+#define HSCH_PORT_MODE(r) __REG(TARGET_HSCH, 0, 1, 184000, 0, 1, 312, 8, r, 70, 4)
+
+#define HSCH_PORT_MODE_DEQUEUE_DIS BIT(4)
+#define HSCH_PORT_MODE_DEQUEUE_DIS_SET(x)\
+ FIELD_PREP(HSCH_PORT_MODE_DEQUEUE_DIS, x)
+#define HSCH_PORT_MODE_DEQUEUE_DIS_GET(x)\
+ FIELD_GET(HSCH_PORT_MODE_DEQUEUE_DIS, x)
+
+#define HSCH_PORT_MODE_AGE_DIS BIT(3)
+#define HSCH_PORT_MODE_AGE_DIS_SET(x)\
+ FIELD_PREP(HSCH_PORT_MODE_AGE_DIS, x)
+#define HSCH_PORT_MODE_AGE_DIS_GET(x)\
+ FIELD_GET(HSCH_PORT_MODE_AGE_DIS, x)
+
+#define HSCH_PORT_MODE_TRUNC_ENA BIT(2)
+#define HSCH_PORT_MODE_TRUNC_ENA_SET(x)\
+ FIELD_PREP(HSCH_PORT_MODE_TRUNC_ENA, x)
+#define HSCH_PORT_MODE_TRUNC_ENA_GET(x)\
+ FIELD_GET(HSCH_PORT_MODE_TRUNC_ENA, x)
+
+#define HSCH_PORT_MODE_EIR_REMARK_ENA BIT(1)
+#define HSCH_PORT_MODE_EIR_REMARK_ENA_SET(x)\
+ FIELD_PREP(HSCH_PORT_MODE_EIR_REMARK_ENA, x)
+#define HSCH_PORT_MODE_EIR_REMARK_ENA_GET(x)\
+ FIELD_GET(HSCH_PORT_MODE_EIR_REMARK_ENA, x)
+
+#define HSCH_PORT_MODE_CPU_PRIO_MODE BIT(0)
+#define HSCH_PORT_MODE_CPU_PRIO_MODE_SET(x)\
+ FIELD_PREP(HSCH_PORT_MODE_CPU_PRIO_MODE, x)
+#define HSCH_PORT_MODE_CPU_PRIO_MODE_GET(x)\
+ FIELD_GET(HSCH_PORT_MODE_CPU_PRIO_MODE, x)
+
+/* HSCH:SYSTEM:OUTB_SHARE_ENA */
+#define HSCH_OUTB_SHARE_ENA(r) __REG(TARGET_HSCH, 0, 1, 184000, 0, 1, 312, 288, r, 5, 4)
+
+#define HSCH_OUTB_SHARE_ENA_OUTB_SHARE_ENA GENMASK(7, 0)
+#define HSCH_OUTB_SHARE_ENA_OUTB_SHARE_ENA_SET(x)\
+ FIELD_PREP(HSCH_OUTB_SHARE_ENA_OUTB_SHARE_ENA, x)
+#define HSCH_OUTB_SHARE_ENA_OUTB_SHARE_ENA_GET(x)\
+ FIELD_GET(HSCH_OUTB_SHARE_ENA_OUTB_SHARE_ENA, x)
+
+/* HSCH:MMGT:RESET_CFG */
+#define HSCH_RESET_CFG __REG(TARGET_HSCH, 0, 1, 162368, 0, 1, 16, 8, 0, 1, 4)
+
+#define HSCH_RESET_CFG_CORE_ENA BIT(0)
+#define HSCH_RESET_CFG_CORE_ENA_SET(x)\
+ FIELD_PREP(HSCH_RESET_CFG_CORE_ENA, x)
+#define HSCH_RESET_CFG_CORE_ENA_GET(x)\
+ FIELD_GET(HSCH_RESET_CFG_CORE_ENA, x)
+
+/* HSCH:TAS_CONFIG:TAS_STATEMACHINE_CFG */
+#define HSCH_TAS_STATEMACHINE_CFG __REG(TARGET_HSCH, 0, 1, 162384, 0, 1, 12, 8, 0, 1, 4)
+
+#define HSCH_TAS_STATEMACHINE_CFG_REVISIT_DLY GENMASK(7, 0)
+#define HSCH_TAS_STATEMACHINE_CFG_REVISIT_DLY_SET(x)\
+ FIELD_PREP(HSCH_TAS_STATEMACHINE_CFG_REVISIT_DLY, x)
+#define HSCH_TAS_STATEMACHINE_CFG_REVISIT_DLY_GET(x)\
+ FIELD_GET(HSCH_TAS_STATEMACHINE_CFG_REVISIT_DLY, x)
+
+/* LRN:COMMON:COMMON_ACCESS_CTRL */
+#define LRN_COMMON_ACCESS_CTRL __REG(TARGET_LRN, 0, 1, 0, 0, 1, 72, 0, 0, 1, 4)
+
+#define LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_COL GENMASK(21, 20)
+#define LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_COL_SET(x)\
+ FIELD_PREP(LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_COL, x)
+#define LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_COL_GET(x)\
+ FIELD_GET(LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_COL, x)
+
+#define LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_TYPE BIT(19)
+#define LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_TYPE_SET(x)\
+ FIELD_PREP(LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_TYPE, x)
+#define LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_TYPE_GET(x)\
+ FIELD_GET(LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_TYPE, x)
+
+#define LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_ROW GENMASK(18, 5)
+#define LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_ROW_SET(x)\
+ FIELD_PREP(LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_ROW, x)
+#define LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_ROW_GET(x)\
+ FIELD_GET(LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_DIRECT_ROW, x)
+
+#define LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_CMD GENMASK(4, 1)
+#define LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_CMD_SET(x)\
+ FIELD_PREP(LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_CMD, x)
+#define LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_CMD_GET(x)\
+ FIELD_GET(LRN_COMMON_ACCESS_CTRL_CPU_ACCESS_CMD, x)
+
+#define LRN_COMMON_ACCESS_CTRL_MAC_TABLE_ACCESS_SHOT BIT(0)
+#define LRN_COMMON_ACCESS_CTRL_MAC_TABLE_ACCESS_SHOT_SET(x)\
+ FIELD_PREP(LRN_COMMON_ACCESS_CTRL_MAC_TABLE_ACCESS_SHOT, x)
+#define LRN_COMMON_ACCESS_CTRL_MAC_TABLE_ACCESS_SHOT_GET(x)\
+ FIELD_GET(LRN_COMMON_ACCESS_CTRL_MAC_TABLE_ACCESS_SHOT, x)
+
+/* LRN:COMMON:MAC_ACCESS_CFG_0 */
+#define LRN_MAC_ACCESS_CFG_0 __REG(TARGET_LRN, 0, 1, 0, 0, 1, 72, 4, 0, 1, 4)
+
+#define LRN_MAC_ACCESS_CFG_0_MAC_ENTRY_FID GENMASK(28, 16)
+#define LRN_MAC_ACCESS_CFG_0_MAC_ENTRY_FID_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_0_MAC_ENTRY_FID, x)
+#define LRN_MAC_ACCESS_CFG_0_MAC_ENTRY_FID_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_0_MAC_ENTRY_FID, x)
+
+#define LRN_MAC_ACCESS_CFG_0_MAC_ENTRY_MAC_MSB GENMASK(15, 0)
+#define LRN_MAC_ACCESS_CFG_0_MAC_ENTRY_MAC_MSB_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_0_MAC_ENTRY_MAC_MSB, x)
+#define LRN_MAC_ACCESS_CFG_0_MAC_ENTRY_MAC_MSB_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_0_MAC_ENTRY_MAC_MSB, x)
+
+/* LRN:COMMON:MAC_ACCESS_CFG_1 */
+#define LRN_MAC_ACCESS_CFG_1 __REG(TARGET_LRN, 0, 1, 0, 0, 1, 72, 8, 0, 1, 4)
+
+/* LRN:COMMON:MAC_ACCESS_CFG_2 */
+#define LRN_MAC_ACCESS_CFG_2 __REG(TARGET_LRN, 0, 1, 0, 0, 1, 72, 12, 0, 1, 4)
+
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_SRC_KILL_FWD BIT(28)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_SRC_KILL_FWD_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_SRC_KILL_FWD, x)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_SRC_KILL_FWD_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_SRC_KILL_FWD, x)
+
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_NXT_LRN_ALL BIT(27)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_NXT_LRN_ALL_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_NXT_LRN_ALL, x)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_NXT_LRN_ALL_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_NXT_LRN_ALL, x)
+
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_CPU_QU GENMASK(26, 24)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_CPU_QU_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_CPU_QU, x)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_CPU_QU_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_CPU_QU, x)
+
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_CPU_COPY BIT(23)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_CPU_COPY_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_CPU_COPY, x)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_CPU_COPY_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_CPU_COPY, x)
+
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_VLAN_IGNORE BIT(22)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_VLAN_IGNORE_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_VLAN_IGNORE, x)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_VLAN_IGNORE_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_VLAN_IGNORE, x)
+
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_MIRROR BIT(21)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_MIRROR_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_MIRROR, x)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_MIRROR_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_MIRROR, x)
+
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_AGE_FLAG GENMASK(20, 19)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_AGE_FLAG_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_AGE_FLAG, x)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_AGE_FLAG_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_AGE_FLAG, x)
+
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_AGE_INTERVAL GENMASK(18, 17)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_AGE_INTERVAL_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_AGE_INTERVAL, x)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_AGE_INTERVAL_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_AGE_INTERVAL, x)
+
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_LOCKED BIT(16)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_LOCKED_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_LOCKED, x)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_LOCKED_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_LOCKED, x)
+
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_VLD BIT(15)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_VLD_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_VLD, x)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_VLD_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_VLD, x)
+
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_ADDR_TYPE GENMASK(14, 12)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_ADDR_TYPE_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_ADDR_TYPE, x)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_ADDR_TYPE_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_ADDR_TYPE, x)
+
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_ADDR GENMASK(11, 0)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_ADDR_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_ADDR, x)
+#define LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_ADDR_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_2_MAC_ENTRY_ADDR, x)
+
+/* LRN:COMMON:MAC_ACCESS_CFG_3 */
+#define LRN_MAC_ACCESS_CFG_3 __REG(TARGET_LRN, 0, 1, 0, 0, 1, 72, 16, 0, 1, 4)
+
+#define LRN_MAC_ACCESS_CFG_3_MAC_ENTRY_ISDX_LIMIT_IDX GENMASK(10, 0)
+#define LRN_MAC_ACCESS_CFG_3_MAC_ENTRY_ISDX_LIMIT_IDX_SET(x)\
+ FIELD_PREP(LRN_MAC_ACCESS_CFG_3_MAC_ENTRY_ISDX_LIMIT_IDX, x)
+#define LRN_MAC_ACCESS_CFG_3_MAC_ENTRY_ISDX_LIMIT_IDX_GET(x)\
+ FIELD_GET(LRN_MAC_ACCESS_CFG_3_MAC_ENTRY_ISDX_LIMIT_IDX, x)
+
+/* LRN:COMMON:SCAN_NEXT_CFG */
+#define LRN_SCAN_NEXT_CFG __REG(TARGET_LRN, 0, 1, 0, 0, 1, 72, 20, 0, 1, 4)
+
+#define LRN_SCAN_NEXT_CFG_SCAN_AGE_FLAG_UPDATE_SEL GENMASK(21, 19)
+#define LRN_SCAN_NEXT_CFG_SCAN_AGE_FLAG_UPDATE_SEL_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_SCAN_AGE_FLAG_UPDATE_SEL, x)
+#define LRN_SCAN_NEXT_CFG_SCAN_AGE_FLAG_UPDATE_SEL_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_SCAN_AGE_FLAG_UPDATE_SEL, x)
+
+#define LRN_SCAN_NEXT_CFG_SCAN_NXT_LRN_ALL_UPDATE_SEL GENMASK(18, 17)
+#define LRN_SCAN_NEXT_CFG_SCAN_NXT_LRN_ALL_UPDATE_SEL_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_SCAN_NXT_LRN_ALL_UPDATE_SEL, x)
+#define LRN_SCAN_NEXT_CFG_SCAN_NXT_LRN_ALL_UPDATE_SEL_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_SCAN_NXT_LRN_ALL_UPDATE_SEL, x)
+
+#define LRN_SCAN_NEXT_CFG_SCAN_AGE_FILTER_SEL GENMASK(16, 15)
+#define LRN_SCAN_NEXT_CFG_SCAN_AGE_FILTER_SEL_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_SCAN_AGE_FILTER_SEL, x)
+#define LRN_SCAN_NEXT_CFG_SCAN_AGE_FILTER_SEL_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_SCAN_AGE_FILTER_SEL, x)
+
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_MOVE_FOUND_ENA BIT(14)
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_MOVE_FOUND_ENA_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_SCAN_NEXT_MOVE_FOUND_ENA, x)
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_MOVE_FOUND_ENA_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_SCAN_NEXT_MOVE_FOUND_ENA, x)
+
+#define LRN_SCAN_NEXT_CFG_NXT_LRN_ALL_FILTER_ENA BIT(13)
+#define LRN_SCAN_NEXT_CFG_NXT_LRN_ALL_FILTER_ENA_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_NXT_LRN_ALL_FILTER_ENA, x)
+#define LRN_SCAN_NEXT_CFG_NXT_LRN_ALL_FILTER_ENA_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_NXT_LRN_ALL_FILTER_ENA, x)
+
+#define LRN_SCAN_NEXT_CFG_SCAN_USE_PORT_FILTER_ENA BIT(12)
+#define LRN_SCAN_NEXT_CFG_SCAN_USE_PORT_FILTER_ENA_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_SCAN_USE_PORT_FILTER_ENA, x)
+#define LRN_SCAN_NEXT_CFG_SCAN_USE_PORT_FILTER_ENA_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_SCAN_USE_PORT_FILTER_ENA, x)
+
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_REMOVE_FOUND_ENA BIT(11)
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_REMOVE_FOUND_ENA_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_SCAN_NEXT_REMOVE_FOUND_ENA, x)
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_REMOVE_FOUND_ENA_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_SCAN_NEXT_REMOVE_FOUND_ENA, x)
+
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_UNTIL_FOUND_ENA BIT(10)
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_UNTIL_FOUND_ENA_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_SCAN_NEXT_UNTIL_FOUND_ENA, x)
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_UNTIL_FOUND_ENA_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_SCAN_NEXT_UNTIL_FOUND_ENA, x)
+
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_INC_AGE_BITS_ENA BIT(9)
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_INC_AGE_BITS_ENA_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_SCAN_NEXT_INC_AGE_BITS_ENA, x)
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_INC_AGE_BITS_ENA_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_SCAN_NEXT_INC_AGE_BITS_ENA, x)
+
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_AGED_ONLY_ENA BIT(8)
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_AGED_ONLY_ENA_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_SCAN_NEXT_AGED_ONLY_ENA, x)
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_AGED_ONLY_ENA_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_SCAN_NEXT_AGED_ONLY_ENA, x)
+
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_IGNORE_LOCKED_ENA BIT(7)
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_IGNORE_LOCKED_ENA_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_SCAN_NEXT_IGNORE_LOCKED_ENA, x)
+#define LRN_SCAN_NEXT_CFG_SCAN_NEXT_IGNORE_LOCKED_ENA_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_SCAN_NEXT_IGNORE_LOCKED_ENA, x)
+
+#define LRN_SCAN_NEXT_CFG_SCAN_AGE_INTERVAL_MASK GENMASK(6, 3)
+#define LRN_SCAN_NEXT_CFG_SCAN_AGE_INTERVAL_MASK_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_SCAN_AGE_INTERVAL_MASK, x)
+#define LRN_SCAN_NEXT_CFG_SCAN_AGE_INTERVAL_MASK_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_SCAN_AGE_INTERVAL_MASK, x)
+
+#define LRN_SCAN_NEXT_CFG_ISDX_LIMIT_IDX_FILTER_ENA BIT(2)
+#define LRN_SCAN_NEXT_CFG_ISDX_LIMIT_IDX_FILTER_ENA_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_ISDX_LIMIT_IDX_FILTER_ENA, x)
+#define LRN_SCAN_NEXT_CFG_ISDX_LIMIT_IDX_FILTER_ENA_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_ISDX_LIMIT_IDX_FILTER_ENA, x)
+
+#define LRN_SCAN_NEXT_CFG_FID_FILTER_ENA BIT(1)
+#define LRN_SCAN_NEXT_CFG_FID_FILTER_ENA_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_FID_FILTER_ENA, x)
+#define LRN_SCAN_NEXT_CFG_FID_FILTER_ENA_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_FID_FILTER_ENA, x)
+
+#define LRN_SCAN_NEXT_CFG_ADDR_FILTER_ENA BIT(0)
+#define LRN_SCAN_NEXT_CFG_ADDR_FILTER_ENA_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_ADDR_FILTER_ENA, x)
+#define LRN_SCAN_NEXT_CFG_ADDR_FILTER_ENA_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_ADDR_FILTER_ENA, x)
+
+/* LRN:COMMON:SCAN_NEXT_CFG_1 */
+#define LRN_SCAN_NEXT_CFG_1 __REG(TARGET_LRN, 0, 1, 0, 0, 1, 72, 24, 0, 1, 4)
+
+#define LRN_SCAN_NEXT_CFG_1_PORT_MOVE_NEW_ADDR GENMASK(30, 16)
+#define LRN_SCAN_NEXT_CFG_1_PORT_MOVE_NEW_ADDR_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_1_PORT_MOVE_NEW_ADDR, x)
+#define LRN_SCAN_NEXT_CFG_1_PORT_MOVE_NEW_ADDR_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_1_PORT_MOVE_NEW_ADDR, x)
+
+#define LRN_SCAN_NEXT_CFG_1_SCAN_ENTRY_ADDR_MASK GENMASK(14, 0)
+#define LRN_SCAN_NEXT_CFG_1_SCAN_ENTRY_ADDR_MASK_SET(x)\
+ FIELD_PREP(LRN_SCAN_NEXT_CFG_1_SCAN_ENTRY_ADDR_MASK, x)
+#define LRN_SCAN_NEXT_CFG_1_SCAN_ENTRY_ADDR_MASK_GET(x)\
+ FIELD_GET(LRN_SCAN_NEXT_CFG_1_SCAN_ENTRY_ADDR_MASK, x)
+
+/* LRN:COMMON:AUTOAGE_CFG */
+#define LRN_AUTOAGE_CFG(r) __REG(TARGET_LRN, 0, 1, 0, 0, 1, 72, 36, r, 4, 4)
+
+#define LRN_AUTOAGE_CFG_UNIT_SIZE GENMASK(29, 28)
+#define LRN_AUTOAGE_CFG_UNIT_SIZE_SET(x)\
+ FIELD_PREP(LRN_AUTOAGE_CFG_UNIT_SIZE, x)
+#define LRN_AUTOAGE_CFG_UNIT_SIZE_GET(x)\
+ FIELD_GET(LRN_AUTOAGE_CFG_UNIT_SIZE, x)
+
+#define LRN_AUTOAGE_CFG_PERIOD_VAL GENMASK(27, 0)
+#define LRN_AUTOAGE_CFG_PERIOD_VAL_SET(x)\
+ FIELD_PREP(LRN_AUTOAGE_CFG_PERIOD_VAL, x)
+#define LRN_AUTOAGE_CFG_PERIOD_VAL_GET(x)\
+ FIELD_GET(LRN_AUTOAGE_CFG_PERIOD_VAL, x)
+
+/* LRN:COMMON:AUTOAGE_CFG_1 */
+#define LRN_AUTOAGE_CFG_1 __REG(TARGET_LRN, 0, 1, 0, 0, 1, 72, 52, 0, 1, 4)
+
+#define LRN_AUTOAGE_CFG_1_PAUSE_AUTO_AGE_ENA BIT(25)
+#define LRN_AUTOAGE_CFG_1_PAUSE_AUTO_AGE_ENA_SET(x)\
+ FIELD_PREP(LRN_AUTOAGE_CFG_1_PAUSE_AUTO_AGE_ENA, x)
+#define LRN_AUTOAGE_CFG_1_PAUSE_AUTO_AGE_ENA_GET(x)\
+ FIELD_GET(LRN_AUTOAGE_CFG_1_PAUSE_AUTO_AGE_ENA, x)
+
+#define LRN_AUTOAGE_CFG_1_CELLS_BETWEEN_ENTRY_SCAN GENMASK(24, 15)
+#define LRN_AUTOAGE_CFG_1_CELLS_BETWEEN_ENTRY_SCAN_SET(x)\
+ FIELD_PREP(LRN_AUTOAGE_CFG_1_CELLS_BETWEEN_ENTRY_SCAN, x)
+#define LRN_AUTOAGE_CFG_1_CELLS_BETWEEN_ENTRY_SCAN_GET(x)\
+ FIELD_GET(LRN_AUTOAGE_CFG_1_CELLS_BETWEEN_ENTRY_SCAN, x)
+
+#define LRN_AUTOAGE_CFG_1_CLK_PERIOD_01NS GENMASK(14, 7)
+#define LRN_AUTOAGE_CFG_1_CLK_PERIOD_01NS_SET(x)\
+ FIELD_PREP(LRN_AUTOAGE_CFG_1_CLK_PERIOD_01NS, x)
+#define LRN_AUTOAGE_CFG_1_CLK_PERIOD_01NS_GET(x)\
+ FIELD_GET(LRN_AUTOAGE_CFG_1_CLK_PERIOD_01NS, x)
+
+#define LRN_AUTOAGE_CFG_1_USE_PORT_FILTER_ENA BIT(6)
+#define LRN_AUTOAGE_CFG_1_USE_PORT_FILTER_ENA_SET(x)\
+ FIELD_PREP(LRN_AUTOAGE_CFG_1_USE_PORT_FILTER_ENA, x)
+#define LRN_AUTOAGE_CFG_1_USE_PORT_FILTER_ENA_GET(x)\
+ FIELD_GET(LRN_AUTOAGE_CFG_1_USE_PORT_FILTER_ENA, x)
+
+#define LRN_AUTOAGE_CFG_1_FORCE_HW_SCAN_SHOT GENMASK(5, 2)
+#define LRN_AUTOAGE_CFG_1_FORCE_HW_SCAN_SHOT_SET(x)\
+ FIELD_PREP(LRN_AUTOAGE_CFG_1_FORCE_HW_SCAN_SHOT, x)
+#define LRN_AUTOAGE_CFG_1_FORCE_HW_SCAN_SHOT_GET(x)\
+ FIELD_GET(LRN_AUTOAGE_CFG_1_FORCE_HW_SCAN_SHOT, x)
+
+#define LRN_AUTOAGE_CFG_1_FORCE_HW_SCAN_STOP_SHOT BIT(1)
+#define LRN_AUTOAGE_CFG_1_FORCE_HW_SCAN_STOP_SHOT_SET(x)\
+ FIELD_PREP(LRN_AUTOAGE_CFG_1_FORCE_HW_SCAN_STOP_SHOT, x)
+#define LRN_AUTOAGE_CFG_1_FORCE_HW_SCAN_STOP_SHOT_GET(x)\
+ FIELD_GET(LRN_AUTOAGE_CFG_1_FORCE_HW_SCAN_STOP_SHOT, x)
+
+#define LRN_AUTOAGE_CFG_1_FORCE_IDLE_ENA BIT(0)
+#define LRN_AUTOAGE_CFG_1_FORCE_IDLE_ENA_SET(x)\
+ FIELD_PREP(LRN_AUTOAGE_CFG_1_FORCE_IDLE_ENA, x)
+#define LRN_AUTOAGE_CFG_1_FORCE_IDLE_ENA_GET(x)\
+ FIELD_GET(LRN_AUTOAGE_CFG_1_FORCE_IDLE_ENA, x)
+
+/* LRN:COMMON:AUTOAGE_CFG_2 */
+#define LRN_AUTOAGE_CFG_2 __REG(TARGET_LRN, 0, 1, 0, 0, 1, 72, 56, 0, 1, 4)
+
+#define LRN_AUTOAGE_CFG_2_NEXT_ROW GENMASK(17, 4)
+#define LRN_AUTOAGE_CFG_2_NEXT_ROW_SET(x)\
+ FIELD_PREP(LRN_AUTOAGE_CFG_2_NEXT_ROW, x)
+#define LRN_AUTOAGE_CFG_2_NEXT_ROW_GET(x)\
+ FIELD_GET(LRN_AUTOAGE_CFG_2_NEXT_ROW, x)
+
+#define LRN_AUTOAGE_CFG_2_SCAN_ONGOING_STATUS GENMASK(3, 0)
+#define LRN_AUTOAGE_CFG_2_SCAN_ONGOING_STATUS_SET(x)\
+ FIELD_PREP(LRN_AUTOAGE_CFG_2_SCAN_ONGOING_STATUS, x)
+#define LRN_AUTOAGE_CFG_2_SCAN_ONGOING_STATUS_GET(x)\
+ FIELD_GET(LRN_AUTOAGE_CFG_2_SCAN_ONGOING_STATUS, x)
+
+/* PCIE_DM_EP:PF0_ATU_CAP:IATU_REGION_CTRL_2_OFF_OUTBOUND_0 */
+#define PCEP_RCTRL_2_OUT_0 __REG(TARGET_PCEP, 0, 1, 3145728, 0, 1, 130852, 4, 0, 1, 4)
+
+#define PCEP_RCTRL_2_OUT_0_MSG_CODE GENMASK(7, 0)
+#define PCEP_RCTRL_2_OUT_0_MSG_CODE_SET(x)\
+ FIELD_PREP(PCEP_RCTRL_2_OUT_0_MSG_CODE, x)
+#define PCEP_RCTRL_2_OUT_0_MSG_CODE_GET(x)\
+ FIELD_GET(PCEP_RCTRL_2_OUT_0_MSG_CODE, x)
+
+#define PCEP_RCTRL_2_OUT_0_TAG GENMASK(15, 8)
+#define PCEP_RCTRL_2_OUT_0_TAG_SET(x)\
+ FIELD_PREP(PCEP_RCTRL_2_OUT_0_TAG, x)
+#define PCEP_RCTRL_2_OUT_0_TAG_GET(x)\
+ FIELD_GET(PCEP_RCTRL_2_OUT_0_TAG, x)
+
+#define PCEP_RCTRL_2_OUT_0_TAG_SUBSTITUTE_EN BIT(16)
+#define PCEP_RCTRL_2_OUT_0_TAG_SUBSTITUTE_EN_SET(x)\
+ FIELD_PREP(PCEP_RCTRL_2_OUT_0_TAG_SUBSTITUTE_EN, x)
+#define PCEP_RCTRL_2_OUT_0_TAG_SUBSTITUTE_EN_GET(x)\
+ FIELD_GET(PCEP_RCTRL_2_OUT_0_TAG_SUBSTITUTE_EN, x)
+
+#define PCEP_RCTRL_2_OUT_0_FUNC_BYPASS BIT(19)
+#define PCEP_RCTRL_2_OUT_0_FUNC_BYPASS_SET(x)\
+ FIELD_PREP(PCEP_RCTRL_2_OUT_0_FUNC_BYPASS, x)
+#define PCEP_RCTRL_2_OUT_0_FUNC_BYPASS_GET(x)\
+ FIELD_GET(PCEP_RCTRL_2_OUT_0_FUNC_BYPASS, x)
+
+#define PCEP_RCTRL_2_OUT_0_SNP BIT(20)
+#define PCEP_RCTRL_2_OUT_0_SNP_SET(x)\
+ FIELD_PREP(PCEP_RCTRL_2_OUT_0_SNP, x)
+#define PCEP_RCTRL_2_OUT_0_SNP_GET(x)\
+ FIELD_GET(PCEP_RCTRL_2_OUT_0_SNP, x)
+
+#define PCEP_RCTRL_2_OUT_0_INHIBIT_PAYLOAD BIT(22)
+#define PCEP_RCTRL_2_OUT_0_INHIBIT_PAYLOAD_SET(x)\
+ FIELD_PREP(PCEP_RCTRL_2_OUT_0_INHIBIT_PAYLOAD, x)
+#define PCEP_RCTRL_2_OUT_0_INHIBIT_PAYLOAD_GET(x)\
+ FIELD_GET(PCEP_RCTRL_2_OUT_0_INHIBIT_PAYLOAD, x)
+
+#define PCEP_RCTRL_2_OUT_0_HEADER_SUBSTITUTE_EN BIT(23)
+#define PCEP_RCTRL_2_OUT_0_HEADER_SUBSTITUTE_EN_SET(x)\
+ FIELD_PREP(PCEP_RCTRL_2_OUT_0_HEADER_SUBSTITUTE_EN, x)
+#define PCEP_RCTRL_2_OUT_0_HEADER_SUBSTITUTE_EN_GET(x)\
+ FIELD_GET(PCEP_RCTRL_2_OUT_0_HEADER_SUBSTITUTE_EN, x)
+
+#define PCEP_RCTRL_2_OUT_0_CFG_SHIFT_MODE BIT(28)
+#define PCEP_RCTRL_2_OUT_0_CFG_SHIFT_MODE_SET(x)\
+ FIELD_PREP(PCEP_RCTRL_2_OUT_0_CFG_SHIFT_MODE, x)
+#define PCEP_RCTRL_2_OUT_0_CFG_SHIFT_MODE_GET(x)\
+ FIELD_GET(PCEP_RCTRL_2_OUT_0_CFG_SHIFT_MODE, x)
+
+#define PCEP_RCTRL_2_OUT_0_INVERT_MODE BIT(29)
+#define PCEP_RCTRL_2_OUT_0_INVERT_MODE_SET(x)\
+ FIELD_PREP(PCEP_RCTRL_2_OUT_0_INVERT_MODE, x)
+#define PCEP_RCTRL_2_OUT_0_INVERT_MODE_GET(x)\
+ FIELD_GET(PCEP_RCTRL_2_OUT_0_INVERT_MODE, x)
+
+#define PCEP_RCTRL_2_OUT_0_REGION_EN BIT(31)
+#define PCEP_RCTRL_2_OUT_0_REGION_EN_SET(x)\
+ FIELD_PREP(PCEP_RCTRL_2_OUT_0_REGION_EN, x)
+#define PCEP_RCTRL_2_OUT_0_REGION_EN_GET(x)\
+ FIELD_GET(PCEP_RCTRL_2_OUT_0_REGION_EN, x)
+
+/* PCIE_DM_EP:PF0_ATU_CAP:IATU_LWR_BASE_ADDR_OFF_OUTBOUND_0 */
+#define PCEP_ADDR_LWR_OUT_0 __REG(TARGET_PCEP, 0, 1, 3145728, 0, 1, 130852, 8, 0, 1, 4)
+
+#define PCEP_ADDR_LWR_OUT_0_LWR_BASE_HW GENMASK(15, 0)
+#define PCEP_ADDR_LWR_OUT_0_LWR_BASE_HW_SET(x)\
+ FIELD_PREP(PCEP_ADDR_LWR_OUT_0_LWR_BASE_HW, x)
+#define PCEP_ADDR_LWR_OUT_0_LWR_BASE_HW_GET(x)\
+ FIELD_GET(PCEP_ADDR_LWR_OUT_0_LWR_BASE_HW, x)
+
+#define PCEP_ADDR_LWR_OUT_0_LWR_BASE_RW GENMASK(31, 16)
+#define PCEP_ADDR_LWR_OUT_0_LWR_BASE_RW_SET(x)\
+ FIELD_PREP(PCEP_ADDR_LWR_OUT_0_LWR_BASE_RW, x)
+#define PCEP_ADDR_LWR_OUT_0_LWR_BASE_RW_GET(x)\
+ FIELD_GET(PCEP_ADDR_LWR_OUT_0_LWR_BASE_RW, x)
+
+/* PCIE_DM_EP:PF0_ATU_CAP:IATU_UPPER_BASE_ADDR_OFF_OUTBOUND_0 */
+#define PCEP_ADDR_UPR_OUT_0 __REG(TARGET_PCEP, 0, 1, 3145728, 0, 1, 130852, 12, 0, 1, 4)
+
+/* PCIE_DM_EP:PF0_ATU_CAP:IATU_LIMIT_ADDR_OFF_OUTBOUND_0 */
+#define PCEP_ADDR_LIM_OUT_0 __REG(TARGET_PCEP, 0, 1, 3145728, 0, 1, 130852, 16, 0, 1, 4)
+
+#define PCEP_ADDR_LIM_OUT_0_LIMIT_ADDR_HW GENMASK(15, 0)
+#define PCEP_ADDR_LIM_OUT_0_LIMIT_ADDR_HW_SET(x)\
+ FIELD_PREP(PCEP_ADDR_LIM_OUT_0_LIMIT_ADDR_HW, x)
+#define PCEP_ADDR_LIM_OUT_0_LIMIT_ADDR_HW_GET(x)\
+ FIELD_GET(PCEP_ADDR_LIM_OUT_0_LIMIT_ADDR_HW, x)
+
+#define PCEP_ADDR_LIM_OUT_0_LIMIT_ADDR_RW GENMASK(31, 16)
+#define PCEP_ADDR_LIM_OUT_0_LIMIT_ADDR_RW_SET(x)\
+ FIELD_PREP(PCEP_ADDR_LIM_OUT_0_LIMIT_ADDR_RW, x)
+#define PCEP_ADDR_LIM_OUT_0_LIMIT_ADDR_RW_GET(x)\
+ FIELD_GET(PCEP_ADDR_LIM_OUT_0_LIMIT_ADDR_RW, x)
+
+/* PCIE_DM_EP:PF0_ATU_CAP:IATU_LWR_TARGET_ADDR_OFF_OUTBOUND_0 */
+#define PCEP_ADDR_LWR_TGT_OUT_0 __REG(TARGET_PCEP, 0, 1, 3145728, 0, 1, 130852, 20, 0, 1, 4)
+
+/* PCIE_DM_EP:PF0_ATU_CAP:IATU_UPPER_TARGET_ADDR_OFF_OUTBOUND_0 */
+#define PCEP_ADDR_UPR_TGT_OUT_0 __REG(TARGET_PCEP, 0, 1, 3145728, 0, 1, 130852, 24, 0, 1, 4)
+
+/* PCIE_DM_EP:PF0_ATU_CAP:IATU_UPPR_LIMIT_ADDR_OFF_OUTBOUND_0 */
+#define PCEP_ADDR_UPR_LIM_OUT_0 __REG(TARGET_PCEP, 0, 1, 3145728, 0, 1, 130852, 32, 0, 1, 4)
+
+#define PCEP_ADDR_UPR_LIM_OUT_0_UPPR_LIMIT_ADDR_RW GENMASK(1, 0)
+#define PCEP_ADDR_UPR_LIM_OUT_0_UPPR_LIMIT_ADDR_RW_SET(x)\
+ FIELD_PREP(PCEP_ADDR_UPR_LIM_OUT_0_UPPR_LIMIT_ADDR_RW, x)
+#define PCEP_ADDR_UPR_LIM_OUT_0_UPPR_LIMIT_ADDR_RW_GET(x)\
+ FIELD_GET(PCEP_ADDR_UPR_LIM_OUT_0_UPPR_LIMIT_ADDR_RW, x)
+
+#define PCEP_ADDR_UPR_LIM_OUT_0_UPPR_LIMIT_ADDR_HW GENMASK(31, 2)
+#define PCEP_ADDR_UPR_LIM_OUT_0_UPPR_LIMIT_ADDR_HW_SET(x)\
+ FIELD_PREP(PCEP_ADDR_UPR_LIM_OUT_0_UPPR_LIMIT_ADDR_HW, x)
+#define PCEP_ADDR_UPR_LIM_OUT_0_UPPR_LIMIT_ADDR_HW_GET(x)\
+ FIELD_GET(PCEP_ADDR_UPR_LIM_OUT_0_UPPR_LIMIT_ADDR_HW, x)
+
+/* PCS_10GBASE_R:PCS_10GBR_CFG:PCS_CFG */
+#define PCS10G_BR_PCS_CFG(t) __REG(TARGET_PCS10G_BR, t, 12, 0, 0, 1, 56, 0, 0, 1, 4)
+
+#define PCS10G_BR_PCS_CFG_PCS_ENA BIT(31)
+#define PCS10G_BR_PCS_CFG_PCS_ENA_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_CFG_PCS_ENA, x)
+#define PCS10G_BR_PCS_CFG_PCS_ENA_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_CFG_PCS_ENA, x)
+
+#define PCS10G_BR_PCS_CFG_PMA_LOOPBACK_ENA BIT(30)
+#define PCS10G_BR_PCS_CFG_PMA_LOOPBACK_ENA_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_CFG_PMA_LOOPBACK_ENA, x)
+#define PCS10G_BR_PCS_CFG_PMA_LOOPBACK_ENA_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_CFG_PMA_LOOPBACK_ENA, x)
+
+#define PCS10G_BR_PCS_CFG_SH_CNT_MAX GENMASK(29, 24)
+#define PCS10G_BR_PCS_CFG_SH_CNT_MAX_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_CFG_SH_CNT_MAX, x)
+#define PCS10G_BR_PCS_CFG_SH_CNT_MAX_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_CFG_SH_CNT_MAX, x)
+
+#define PCS10G_BR_PCS_CFG_RX_DATA_FLIP BIT(18)
+#define PCS10G_BR_PCS_CFG_RX_DATA_FLIP_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_CFG_RX_DATA_FLIP, x)
+#define PCS10G_BR_PCS_CFG_RX_DATA_FLIP_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_CFG_RX_DATA_FLIP, x)
+
+#define PCS10G_BR_PCS_CFG_RESYNC_ENA BIT(15)
+#define PCS10G_BR_PCS_CFG_RESYNC_ENA_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_CFG_RESYNC_ENA, x)
+#define PCS10G_BR_PCS_CFG_RESYNC_ENA_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_CFG_RESYNC_ENA, x)
+
+#define PCS10G_BR_PCS_CFG_LF_GEN_DIS BIT(14)
+#define PCS10G_BR_PCS_CFG_LF_GEN_DIS_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_CFG_LF_GEN_DIS, x)
+#define PCS10G_BR_PCS_CFG_LF_GEN_DIS_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_CFG_LF_GEN_DIS, x)
+
+#define PCS10G_BR_PCS_CFG_RX_TEST_MODE BIT(13)
+#define PCS10G_BR_PCS_CFG_RX_TEST_MODE_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_CFG_RX_TEST_MODE, x)
+#define PCS10G_BR_PCS_CFG_RX_TEST_MODE_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_CFG_RX_TEST_MODE, x)
+
+#define PCS10G_BR_PCS_CFG_RX_SCR_DISABLE BIT(12)
+#define PCS10G_BR_PCS_CFG_RX_SCR_DISABLE_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_CFG_RX_SCR_DISABLE, x)
+#define PCS10G_BR_PCS_CFG_RX_SCR_DISABLE_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_CFG_RX_SCR_DISABLE, x)
+
+#define PCS10G_BR_PCS_CFG_TX_DATA_FLIP BIT(7)
+#define PCS10G_BR_PCS_CFG_TX_DATA_FLIP_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_CFG_TX_DATA_FLIP, x)
+#define PCS10G_BR_PCS_CFG_TX_DATA_FLIP_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_CFG_TX_DATA_FLIP, x)
+
+#define PCS10G_BR_PCS_CFG_AN_LINK_CTRL_ENA BIT(6)
+#define PCS10G_BR_PCS_CFG_AN_LINK_CTRL_ENA_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_CFG_AN_LINK_CTRL_ENA, x)
+#define PCS10G_BR_PCS_CFG_AN_LINK_CTRL_ENA_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_CFG_AN_LINK_CTRL_ENA, x)
+
+#define PCS10G_BR_PCS_CFG_TX_TEST_MODE BIT(4)
+#define PCS10G_BR_PCS_CFG_TX_TEST_MODE_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_CFG_TX_TEST_MODE, x)
+#define PCS10G_BR_PCS_CFG_TX_TEST_MODE_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_CFG_TX_TEST_MODE, x)
+
+#define PCS10G_BR_PCS_CFG_TX_SCR_DISABLE BIT(3)
+#define PCS10G_BR_PCS_CFG_TX_SCR_DISABLE_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_CFG_TX_SCR_DISABLE, x)
+#define PCS10G_BR_PCS_CFG_TX_SCR_DISABLE_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_CFG_TX_SCR_DISABLE, x)
+
+/* PCS_10GBASE_R:PCS_10GBR_CFG:PCS_SD_CFG */
+#define PCS10G_BR_PCS_SD_CFG(t) __REG(TARGET_PCS10G_BR, t, 12, 0, 0, 1, 56, 4, 0, 1, 4)
+
+#define PCS10G_BR_PCS_SD_CFG_SD_SEL BIT(8)
+#define PCS10G_BR_PCS_SD_CFG_SD_SEL_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_SD_CFG_SD_SEL, x)
+#define PCS10G_BR_PCS_SD_CFG_SD_SEL_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_SD_CFG_SD_SEL, x)
+
+#define PCS10G_BR_PCS_SD_CFG_SD_POL BIT(4)
+#define PCS10G_BR_PCS_SD_CFG_SD_POL_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_SD_CFG_SD_POL, x)
+#define PCS10G_BR_PCS_SD_CFG_SD_POL_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_SD_CFG_SD_POL, x)
+
+#define PCS10G_BR_PCS_SD_CFG_SD_ENA BIT(0)
+#define PCS10G_BR_PCS_SD_CFG_SD_ENA_SET(x)\
+ FIELD_PREP(PCS10G_BR_PCS_SD_CFG_SD_ENA, x)
+#define PCS10G_BR_PCS_SD_CFG_SD_ENA_GET(x)\
+ FIELD_GET(PCS10G_BR_PCS_SD_CFG_SD_ENA, x)
+
+/* PCS_10GBASE_R:PCS_10GBR_CFG:PCS_CFG */
+#define PCS25G_BR_PCS_CFG(t) __REG(TARGET_PCS25G_BR, t, 8, 0, 0, 1, 56, 0, 0, 1, 4)
+
+#define PCS25G_BR_PCS_CFG_PCS_ENA BIT(31)
+#define PCS25G_BR_PCS_CFG_PCS_ENA_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_CFG_PCS_ENA, x)
+#define PCS25G_BR_PCS_CFG_PCS_ENA_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_CFG_PCS_ENA, x)
+
+#define PCS25G_BR_PCS_CFG_PMA_LOOPBACK_ENA BIT(30)
+#define PCS25G_BR_PCS_CFG_PMA_LOOPBACK_ENA_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_CFG_PMA_LOOPBACK_ENA, x)
+#define PCS25G_BR_PCS_CFG_PMA_LOOPBACK_ENA_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_CFG_PMA_LOOPBACK_ENA, x)
+
+#define PCS25G_BR_PCS_CFG_SH_CNT_MAX GENMASK(29, 24)
+#define PCS25G_BR_PCS_CFG_SH_CNT_MAX_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_CFG_SH_CNT_MAX, x)
+#define PCS25G_BR_PCS_CFG_SH_CNT_MAX_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_CFG_SH_CNT_MAX, x)
+
+#define PCS25G_BR_PCS_CFG_RX_DATA_FLIP BIT(18)
+#define PCS25G_BR_PCS_CFG_RX_DATA_FLIP_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_CFG_RX_DATA_FLIP, x)
+#define PCS25G_BR_PCS_CFG_RX_DATA_FLIP_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_CFG_RX_DATA_FLIP, x)
+
+#define PCS25G_BR_PCS_CFG_RESYNC_ENA BIT(15)
+#define PCS25G_BR_PCS_CFG_RESYNC_ENA_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_CFG_RESYNC_ENA, x)
+#define PCS25G_BR_PCS_CFG_RESYNC_ENA_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_CFG_RESYNC_ENA, x)
+
+#define PCS25G_BR_PCS_CFG_LF_GEN_DIS BIT(14)
+#define PCS25G_BR_PCS_CFG_LF_GEN_DIS_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_CFG_LF_GEN_DIS, x)
+#define PCS25G_BR_PCS_CFG_LF_GEN_DIS_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_CFG_LF_GEN_DIS, x)
+
+#define PCS25G_BR_PCS_CFG_RX_TEST_MODE BIT(13)
+#define PCS25G_BR_PCS_CFG_RX_TEST_MODE_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_CFG_RX_TEST_MODE, x)
+#define PCS25G_BR_PCS_CFG_RX_TEST_MODE_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_CFG_RX_TEST_MODE, x)
+
+#define PCS25G_BR_PCS_CFG_RX_SCR_DISABLE BIT(12)
+#define PCS25G_BR_PCS_CFG_RX_SCR_DISABLE_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_CFG_RX_SCR_DISABLE, x)
+#define PCS25G_BR_PCS_CFG_RX_SCR_DISABLE_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_CFG_RX_SCR_DISABLE, x)
+
+#define PCS25G_BR_PCS_CFG_TX_DATA_FLIP BIT(7)
+#define PCS25G_BR_PCS_CFG_TX_DATA_FLIP_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_CFG_TX_DATA_FLIP, x)
+#define PCS25G_BR_PCS_CFG_TX_DATA_FLIP_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_CFG_TX_DATA_FLIP, x)
+
+#define PCS25G_BR_PCS_CFG_AN_LINK_CTRL_ENA BIT(6)
+#define PCS25G_BR_PCS_CFG_AN_LINK_CTRL_ENA_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_CFG_AN_LINK_CTRL_ENA, x)
+#define PCS25G_BR_PCS_CFG_AN_LINK_CTRL_ENA_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_CFG_AN_LINK_CTRL_ENA, x)
+
+#define PCS25G_BR_PCS_CFG_TX_TEST_MODE BIT(4)
+#define PCS25G_BR_PCS_CFG_TX_TEST_MODE_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_CFG_TX_TEST_MODE, x)
+#define PCS25G_BR_PCS_CFG_TX_TEST_MODE_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_CFG_TX_TEST_MODE, x)
+
+#define PCS25G_BR_PCS_CFG_TX_SCR_DISABLE BIT(3)
+#define PCS25G_BR_PCS_CFG_TX_SCR_DISABLE_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_CFG_TX_SCR_DISABLE, x)
+#define PCS25G_BR_PCS_CFG_TX_SCR_DISABLE_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_CFG_TX_SCR_DISABLE, x)
+
+/* PCS_10GBASE_R:PCS_10GBR_CFG:PCS_SD_CFG */
+#define PCS25G_BR_PCS_SD_CFG(t) __REG(TARGET_PCS25G_BR, t, 8, 0, 0, 1, 56, 4, 0, 1, 4)
+
+#define PCS25G_BR_PCS_SD_CFG_SD_SEL BIT(8)
+#define PCS25G_BR_PCS_SD_CFG_SD_SEL_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_SD_CFG_SD_SEL, x)
+#define PCS25G_BR_PCS_SD_CFG_SD_SEL_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_SD_CFG_SD_SEL, x)
+
+#define PCS25G_BR_PCS_SD_CFG_SD_POL BIT(4)
+#define PCS25G_BR_PCS_SD_CFG_SD_POL_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_SD_CFG_SD_POL, x)
+#define PCS25G_BR_PCS_SD_CFG_SD_POL_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_SD_CFG_SD_POL, x)
+
+#define PCS25G_BR_PCS_SD_CFG_SD_ENA BIT(0)
+#define PCS25G_BR_PCS_SD_CFG_SD_ENA_SET(x)\
+ FIELD_PREP(PCS25G_BR_PCS_SD_CFG_SD_ENA, x)
+#define PCS25G_BR_PCS_SD_CFG_SD_ENA_GET(x)\
+ FIELD_GET(PCS25G_BR_PCS_SD_CFG_SD_ENA, x)
+
+/* PCS_10GBASE_R:PCS_10GBR_CFG:PCS_CFG */
+#define PCS5G_BR_PCS_CFG(t) __REG(TARGET_PCS5G_BR, t, 13, 0, 0, 1, 56, 0, 0, 1, 4)
+
+#define PCS5G_BR_PCS_CFG_PCS_ENA BIT(31)
+#define PCS5G_BR_PCS_CFG_PCS_ENA_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_CFG_PCS_ENA, x)
+#define PCS5G_BR_PCS_CFG_PCS_ENA_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_CFG_PCS_ENA, x)
+
+#define PCS5G_BR_PCS_CFG_PMA_LOOPBACK_ENA BIT(30)
+#define PCS5G_BR_PCS_CFG_PMA_LOOPBACK_ENA_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_CFG_PMA_LOOPBACK_ENA, x)
+#define PCS5G_BR_PCS_CFG_PMA_LOOPBACK_ENA_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_CFG_PMA_LOOPBACK_ENA, x)
+
+#define PCS5G_BR_PCS_CFG_SH_CNT_MAX GENMASK(29, 24)
+#define PCS5G_BR_PCS_CFG_SH_CNT_MAX_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_CFG_SH_CNT_MAX, x)
+#define PCS5G_BR_PCS_CFG_SH_CNT_MAX_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_CFG_SH_CNT_MAX, x)
+
+#define PCS5G_BR_PCS_CFG_RX_DATA_FLIP BIT(18)
+#define PCS5G_BR_PCS_CFG_RX_DATA_FLIP_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_CFG_RX_DATA_FLIP, x)
+#define PCS5G_BR_PCS_CFG_RX_DATA_FLIP_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_CFG_RX_DATA_FLIP, x)
+
+#define PCS5G_BR_PCS_CFG_RESYNC_ENA BIT(15)
+#define PCS5G_BR_PCS_CFG_RESYNC_ENA_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_CFG_RESYNC_ENA, x)
+#define PCS5G_BR_PCS_CFG_RESYNC_ENA_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_CFG_RESYNC_ENA, x)
+
+#define PCS5G_BR_PCS_CFG_LF_GEN_DIS BIT(14)
+#define PCS5G_BR_PCS_CFG_LF_GEN_DIS_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_CFG_LF_GEN_DIS, x)
+#define PCS5G_BR_PCS_CFG_LF_GEN_DIS_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_CFG_LF_GEN_DIS, x)
+
+#define PCS5G_BR_PCS_CFG_RX_TEST_MODE BIT(13)
+#define PCS5G_BR_PCS_CFG_RX_TEST_MODE_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_CFG_RX_TEST_MODE, x)
+#define PCS5G_BR_PCS_CFG_RX_TEST_MODE_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_CFG_RX_TEST_MODE, x)
+
+#define PCS5G_BR_PCS_CFG_RX_SCR_DISABLE BIT(12)
+#define PCS5G_BR_PCS_CFG_RX_SCR_DISABLE_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_CFG_RX_SCR_DISABLE, x)
+#define PCS5G_BR_PCS_CFG_RX_SCR_DISABLE_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_CFG_RX_SCR_DISABLE, x)
+
+#define PCS5G_BR_PCS_CFG_TX_DATA_FLIP BIT(7)
+#define PCS5G_BR_PCS_CFG_TX_DATA_FLIP_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_CFG_TX_DATA_FLIP, x)
+#define PCS5G_BR_PCS_CFG_TX_DATA_FLIP_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_CFG_TX_DATA_FLIP, x)
+
+#define PCS5G_BR_PCS_CFG_AN_LINK_CTRL_ENA BIT(6)
+#define PCS5G_BR_PCS_CFG_AN_LINK_CTRL_ENA_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_CFG_AN_LINK_CTRL_ENA, x)
+#define PCS5G_BR_PCS_CFG_AN_LINK_CTRL_ENA_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_CFG_AN_LINK_CTRL_ENA, x)
+
+#define PCS5G_BR_PCS_CFG_TX_TEST_MODE BIT(4)
+#define PCS5G_BR_PCS_CFG_TX_TEST_MODE_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_CFG_TX_TEST_MODE, x)
+#define PCS5G_BR_PCS_CFG_TX_TEST_MODE_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_CFG_TX_TEST_MODE, x)
+
+#define PCS5G_BR_PCS_CFG_TX_SCR_DISABLE BIT(3)
+#define PCS5G_BR_PCS_CFG_TX_SCR_DISABLE_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_CFG_TX_SCR_DISABLE, x)
+#define PCS5G_BR_PCS_CFG_TX_SCR_DISABLE_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_CFG_TX_SCR_DISABLE, x)
+
+/* PCS_10GBASE_R:PCS_10GBR_CFG:PCS_SD_CFG */
+#define PCS5G_BR_PCS_SD_CFG(t) __REG(TARGET_PCS5G_BR, t, 13, 0, 0, 1, 56, 4, 0, 1, 4)
+
+#define PCS5G_BR_PCS_SD_CFG_SD_SEL BIT(8)
+#define PCS5G_BR_PCS_SD_CFG_SD_SEL_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_SD_CFG_SD_SEL, x)
+#define PCS5G_BR_PCS_SD_CFG_SD_SEL_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_SD_CFG_SD_SEL, x)
+
+#define PCS5G_BR_PCS_SD_CFG_SD_POL BIT(4)
+#define PCS5G_BR_PCS_SD_CFG_SD_POL_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_SD_CFG_SD_POL, x)
+#define PCS5G_BR_PCS_SD_CFG_SD_POL_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_SD_CFG_SD_POL, x)
+
+#define PCS5G_BR_PCS_SD_CFG_SD_ENA BIT(0)
+#define PCS5G_BR_PCS_SD_CFG_SD_ENA_SET(x)\
+ FIELD_PREP(PCS5G_BR_PCS_SD_CFG_SD_ENA, x)
+#define PCS5G_BR_PCS_SD_CFG_SD_ENA_GET(x)\
+ FIELD_GET(PCS5G_BR_PCS_SD_CFG_SD_ENA, x)
+
+/* PORT_CONF:HW_CFG:DEV5G_MODES */
+#define PORT_CONF_DEV5G_MODES __REG(TARGET_PORT_CONF, 0, 1, 0, 0, 1, 24, 0, 0, 1, 4)
+
+#define PORT_CONF_DEV5G_MODES_DEV5G_D0_MODE BIT(0)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D0_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV5G_MODES_DEV5G_D0_MODE, x)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D0_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV5G_MODES_DEV5G_D0_MODE, x)
+
+#define PORT_CONF_DEV5G_MODES_DEV5G_D1_MODE BIT(1)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D1_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV5G_MODES_DEV5G_D1_MODE, x)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D1_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV5G_MODES_DEV5G_D1_MODE, x)
+
+#define PORT_CONF_DEV5G_MODES_DEV5G_D2_MODE BIT(2)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D2_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV5G_MODES_DEV5G_D2_MODE, x)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D2_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV5G_MODES_DEV5G_D2_MODE, x)
+
+#define PORT_CONF_DEV5G_MODES_DEV5G_D3_MODE BIT(3)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D3_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV5G_MODES_DEV5G_D3_MODE, x)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D3_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV5G_MODES_DEV5G_D3_MODE, x)
+
+#define PORT_CONF_DEV5G_MODES_DEV5G_D4_MODE BIT(4)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D4_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV5G_MODES_DEV5G_D4_MODE, x)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D4_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV5G_MODES_DEV5G_D4_MODE, x)
+
+#define PORT_CONF_DEV5G_MODES_DEV5G_D5_MODE BIT(5)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D5_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV5G_MODES_DEV5G_D5_MODE, x)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D5_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV5G_MODES_DEV5G_D5_MODE, x)
+
+#define PORT_CONF_DEV5G_MODES_DEV5G_D6_MODE BIT(6)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D6_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV5G_MODES_DEV5G_D6_MODE, x)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D6_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV5G_MODES_DEV5G_D6_MODE, x)
+
+#define PORT_CONF_DEV5G_MODES_DEV5G_D7_MODE BIT(7)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D7_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV5G_MODES_DEV5G_D7_MODE, x)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D7_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV5G_MODES_DEV5G_D7_MODE, x)
+
+#define PORT_CONF_DEV5G_MODES_DEV5G_D8_MODE BIT(8)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D8_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV5G_MODES_DEV5G_D8_MODE, x)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D8_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV5G_MODES_DEV5G_D8_MODE, x)
+
+#define PORT_CONF_DEV5G_MODES_DEV5G_D9_MODE BIT(9)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D9_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV5G_MODES_DEV5G_D9_MODE, x)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D9_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV5G_MODES_DEV5G_D9_MODE, x)
+
+#define PORT_CONF_DEV5G_MODES_DEV5G_D10_MODE BIT(10)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D10_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV5G_MODES_DEV5G_D10_MODE, x)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D10_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV5G_MODES_DEV5G_D10_MODE, x)
+
+#define PORT_CONF_DEV5G_MODES_DEV5G_D11_MODE BIT(11)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D11_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV5G_MODES_DEV5G_D11_MODE, x)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D11_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV5G_MODES_DEV5G_D11_MODE, x)
+
+#define PORT_CONF_DEV5G_MODES_DEV5G_D64_MODE BIT(12)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D64_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV5G_MODES_DEV5G_D64_MODE, x)
+#define PORT_CONF_DEV5G_MODES_DEV5G_D64_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV5G_MODES_DEV5G_D64_MODE, x)
+
+/* PORT_CONF:HW_CFG:DEV10G_MODES */
+#define PORT_CONF_DEV10G_MODES __REG(TARGET_PORT_CONF, 0, 1, 0, 0, 1, 24, 4, 0, 1, 4)
+
+#define PORT_CONF_DEV10G_MODES_DEV10G_D12_MODE BIT(0)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D12_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV10G_MODES_DEV10G_D12_MODE, x)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D12_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV10G_MODES_DEV10G_D12_MODE, x)
+
+#define PORT_CONF_DEV10G_MODES_DEV10G_D13_MODE BIT(1)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D13_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV10G_MODES_DEV10G_D13_MODE, x)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D13_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV10G_MODES_DEV10G_D13_MODE, x)
+
+#define PORT_CONF_DEV10G_MODES_DEV10G_D14_MODE BIT(2)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D14_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV10G_MODES_DEV10G_D14_MODE, x)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D14_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV10G_MODES_DEV10G_D14_MODE, x)
+
+#define PORT_CONF_DEV10G_MODES_DEV10G_D15_MODE BIT(3)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D15_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV10G_MODES_DEV10G_D15_MODE, x)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D15_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV10G_MODES_DEV10G_D15_MODE, x)
+
+#define PORT_CONF_DEV10G_MODES_DEV10G_D48_MODE BIT(4)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D48_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV10G_MODES_DEV10G_D48_MODE, x)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D48_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV10G_MODES_DEV10G_D48_MODE, x)
+
+#define PORT_CONF_DEV10G_MODES_DEV10G_D49_MODE BIT(5)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D49_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV10G_MODES_DEV10G_D49_MODE, x)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D49_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV10G_MODES_DEV10G_D49_MODE, x)
+
+#define PORT_CONF_DEV10G_MODES_DEV10G_D50_MODE BIT(6)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D50_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV10G_MODES_DEV10G_D50_MODE, x)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D50_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV10G_MODES_DEV10G_D50_MODE, x)
+
+#define PORT_CONF_DEV10G_MODES_DEV10G_D51_MODE BIT(7)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D51_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV10G_MODES_DEV10G_D51_MODE, x)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D51_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV10G_MODES_DEV10G_D51_MODE, x)
+
+#define PORT_CONF_DEV10G_MODES_DEV10G_D52_MODE BIT(8)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D52_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV10G_MODES_DEV10G_D52_MODE, x)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D52_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV10G_MODES_DEV10G_D52_MODE, x)
+
+#define PORT_CONF_DEV10G_MODES_DEV10G_D53_MODE BIT(9)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D53_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV10G_MODES_DEV10G_D53_MODE, x)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D53_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV10G_MODES_DEV10G_D53_MODE, x)
+
+#define PORT_CONF_DEV10G_MODES_DEV10G_D54_MODE BIT(10)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D54_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV10G_MODES_DEV10G_D54_MODE, x)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D54_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV10G_MODES_DEV10G_D54_MODE, x)
+
+#define PORT_CONF_DEV10G_MODES_DEV10G_D55_MODE BIT(11)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D55_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV10G_MODES_DEV10G_D55_MODE, x)
+#define PORT_CONF_DEV10G_MODES_DEV10G_D55_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV10G_MODES_DEV10G_D55_MODE, x)
+
+/* PORT_CONF:HW_CFG:DEV25G_MODES */
+#define PORT_CONF_DEV25G_MODES __REG(TARGET_PORT_CONF, 0, 1, 0, 0, 1, 24, 8, 0, 1, 4)
+
+#define PORT_CONF_DEV25G_MODES_DEV25G_D56_MODE BIT(0)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D56_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV25G_MODES_DEV25G_D56_MODE, x)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D56_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV25G_MODES_DEV25G_D56_MODE, x)
+
+#define PORT_CONF_DEV25G_MODES_DEV25G_D57_MODE BIT(1)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D57_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV25G_MODES_DEV25G_D57_MODE, x)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D57_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV25G_MODES_DEV25G_D57_MODE, x)
+
+#define PORT_CONF_DEV25G_MODES_DEV25G_D58_MODE BIT(2)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D58_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV25G_MODES_DEV25G_D58_MODE, x)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D58_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV25G_MODES_DEV25G_D58_MODE, x)
+
+#define PORT_CONF_DEV25G_MODES_DEV25G_D59_MODE BIT(3)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D59_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV25G_MODES_DEV25G_D59_MODE, x)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D59_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV25G_MODES_DEV25G_D59_MODE, x)
+
+#define PORT_CONF_DEV25G_MODES_DEV25G_D60_MODE BIT(4)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D60_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV25G_MODES_DEV25G_D60_MODE, x)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D60_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV25G_MODES_DEV25G_D60_MODE, x)
+
+#define PORT_CONF_DEV25G_MODES_DEV25G_D61_MODE BIT(5)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D61_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV25G_MODES_DEV25G_D61_MODE, x)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D61_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV25G_MODES_DEV25G_D61_MODE, x)
+
+#define PORT_CONF_DEV25G_MODES_DEV25G_D62_MODE BIT(6)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D62_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV25G_MODES_DEV25G_D62_MODE, x)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D62_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV25G_MODES_DEV25G_D62_MODE, x)
+
+#define PORT_CONF_DEV25G_MODES_DEV25G_D63_MODE BIT(7)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D63_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_DEV25G_MODES_DEV25G_D63_MODE, x)
+#define PORT_CONF_DEV25G_MODES_DEV25G_D63_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_DEV25G_MODES_DEV25G_D63_MODE, x)
+
+/* PORT_CONF:HW_CFG:QSGMII_ENA */
+#define PORT_CONF_QSGMII_ENA __REG(TARGET_PORT_CONF, 0, 1, 0, 0, 1, 24, 12, 0, 1, 4)
+
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_0 BIT(0)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_0_SET(x)\
+ FIELD_PREP(PORT_CONF_QSGMII_ENA_QSGMII_ENA_0, x)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_0_GET(x)\
+ FIELD_GET(PORT_CONF_QSGMII_ENA_QSGMII_ENA_0, x)
+
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_1 BIT(1)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_1_SET(x)\
+ FIELD_PREP(PORT_CONF_QSGMII_ENA_QSGMII_ENA_1, x)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_1_GET(x)\
+ FIELD_GET(PORT_CONF_QSGMII_ENA_QSGMII_ENA_1, x)
+
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_2 BIT(2)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_2_SET(x)\
+ FIELD_PREP(PORT_CONF_QSGMII_ENA_QSGMII_ENA_2, x)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_2_GET(x)\
+ FIELD_GET(PORT_CONF_QSGMII_ENA_QSGMII_ENA_2, x)
+
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_3 BIT(3)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_3_SET(x)\
+ FIELD_PREP(PORT_CONF_QSGMII_ENA_QSGMII_ENA_3, x)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_3_GET(x)\
+ FIELD_GET(PORT_CONF_QSGMII_ENA_QSGMII_ENA_3, x)
+
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_4 BIT(4)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_4_SET(x)\
+ FIELD_PREP(PORT_CONF_QSGMII_ENA_QSGMII_ENA_4, x)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_4_GET(x)\
+ FIELD_GET(PORT_CONF_QSGMII_ENA_QSGMII_ENA_4, x)
+
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_5 BIT(5)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_5_SET(x)\
+ FIELD_PREP(PORT_CONF_QSGMII_ENA_QSGMII_ENA_5, x)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_5_GET(x)\
+ FIELD_GET(PORT_CONF_QSGMII_ENA_QSGMII_ENA_5, x)
+
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_6 BIT(6)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_6_SET(x)\
+ FIELD_PREP(PORT_CONF_QSGMII_ENA_QSGMII_ENA_6, x)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_6_GET(x)\
+ FIELD_GET(PORT_CONF_QSGMII_ENA_QSGMII_ENA_6, x)
+
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_7 BIT(7)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_7_SET(x)\
+ FIELD_PREP(PORT_CONF_QSGMII_ENA_QSGMII_ENA_7, x)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_7_GET(x)\
+ FIELD_GET(PORT_CONF_QSGMII_ENA_QSGMII_ENA_7, x)
+
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_8 BIT(8)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_8_SET(x)\
+ FIELD_PREP(PORT_CONF_QSGMII_ENA_QSGMII_ENA_8, x)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_8_GET(x)\
+ FIELD_GET(PORT_CONF_QSGMII_ENA_QSGMII_ENA_8, x)
+
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_9 BIT(9)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_9_SET(x)\
+ FIELD_PREP(PORT_CONF_QSGMII_ENA_QSGMII_ENA_9, x)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_9_GET(x)\
+ FIELD_GET(PORT_CONF_QSGMII_ENA_QSGMII_ENA_9, x)
+
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_10 BIT(10)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_10_SET(x)\
+ FIELD_PREP(PORT_CONF_QSGMII_ENA_QSGMII_ENA_10, x)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_10_GET(x)\
+ FIELD_GET(PORT_CONF_QSGMII_ENA_QSGMII_ENA_10, x)
+
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_11 BIT(11)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_11_SET(x)\
+ FIELD_PREP(PORT_CONF_QSGMII_ENA_QSGMII_ENA_11, x)
+#define PORT_CONF_QSGMII_ENA_QSGMII_ENA_11_GET(x)\
+ FIELD_GET(PORT_CONF_QSGMII_ENA_QSGMII_ENA_11, x)
+
+/* PORT_CONF:USGMII_CFG_STAT:USGMII_CFG */
+#define PORT_CONF_USGMII_CFG(g) __REG(TARGET_PORT_CONF, 0, 1, 72, g, 6, 8, 0, 0, 1, 4)
+
+#define PORT_CONF_USGMII_CFG_BYPASS_SCRAM BIT(9)
+#define PORT_CONF_USGMII_CFG_BYPASS_SCRAM_SET(x)\
+ FIELD_PREP(PORT_CONF_USGMII_CFG_BYPASS_SCRAM, x)
+#define PORT_CONF_USGMII_CFG_BYPASS_SCRAM_GET(x)\
+ FIELD_GET(PORT_CONF_USGMII_CFG_BYPASS_SCRAM, x)
+
+#define PORT_CONF_USGMII_CFG_BYPASS_DESCRAM BIT(8)
+#define PORT_CONF_USGMII_CFG_BYPASS_DESCRAM_SET(x)\
+ FIELD_PREP(PORT_CONF_USGMII_CFG_BYPASS_DESCRAM, x)
+#define PORT_CONF_USGMII_CFG_BYPASS_DESCRAM_GET(x)\
+ FIELD_GET(PORT_CONF_USGMII_CFG_BYPASS_DESCRAM, x)
+
+#define PORT_CONF_USGMII_CFG_FLIP_LANES BIT(7)
+#define PORT_CONF_USGMII_CFG_FLIP_LANES_SET(x)\
+ FIELD_PREP(PORT_CONF_USGMII_CFG_FLIP_LANES, x)
+#define PORT_CONF_USGMII_CFG_FLIP_LANES_GET(x)\
+ FIELD_GET(PORT_CONF_USGMII_CFG_FLIP_LANES, x)
+
+#define PORT_CONF_USGMII_CFG_SHYST_DIS BIT(6)
+#define PORT_CONF_USGMII_CFG_SHYST_DIS_SET(x)\
+ FIELD_PREP(PORT_CONF_USGMII_CFG_SHYST_DIS, x)
+#define PORT_CONF_USGMII_CFG_SHYST_DIS_GET(x)\
+ FIELD_GET(PORT_CONF_USGMII_CFG_SHYST_DIS, x)
+
+#define PORT_CONF_USGMII_CFG_E_DET_ENA BIT(5)
+#define PORT_CONF_USGMII_CFG_E_DET_ENA_SET(x)\
+ FIELD_PREP(PORT_CONF_USGMII_CFG_E_DET_ENA, x)
+#define PORT_CONF_USGMII_CFG_E_DET_ENA_GET(x)\
+ FIELD_GET(PORT_CONF_USGMII_CFG_E_DET_ENA, x)
+
+#define PORT_CONF_USGMII_CFG_USE_I1_ENA BIT(4)
+#define PORT_CONF_USGMII_CFG_USE_I1_ENA_SET(x)\
+ FIELD_PREP(PORT_CONF_USGMII_CFG_USE_I1_ENA, x)
+#define PORT_CONF_USGMII_CFG_USE_I1_ENA_GET(x)\
+ FIELD_GET(PORT_CONF_USGMII_CFG_USE_I1_ENA, x)
+
+#define PORT_CONF_USGMII_CFG_QUAD_MODE BIT(1)
+#define PORT_CONF_USGMII_CFG_QUAD_MODE_SET(x)\
+ FIELD_PREP(PORT_CONF_USGMII_CFG_QUAD_MODE, x)
+#define PORT_CONF_USGMII_CFG_QUAD_MODE_GET(x)\
+ FIELD_GET(PORT_CONF_USGMII_CFG_QUAD_MODE, x)
+
+/* QFWD:SYSTEM:SWITCH_PORT_MODE */
+#define QFWD_SWITCH_PORT_MODE(r) __REG(TARGET_QFWD, 0, 1, 0, 0, 1, 340, 0, r, 70, 4)
+
+#define QFWD_SWITCH_PORT_MODE_PORT_ENA BIT(19)
+#define QFWD_SWITCH_PORT_MODE_PORT_ENA_SET(x)\
+ FIELD_PREP(QFWD_SWITCH_PORT_MODE_PORT_ENA, x)
+#define QFWD_SWITCH_PORT_MODE_PORT_ENA_GET(x)\
+ FIELD_GET(QFWD_SWITCH_PORT_MODE_PORT_ENA, x)
+
+#define QFWD_SWITCH_PORT_MODE_FWD_URGENCY GENMASK(18, 10)
+#define QFWD_SWITCH_PORT_MODE_FWD_URGENCY_SET(x)\
+ FIELD_PREP(QFWD_SWITCH_PORT_MODE_FWD_URGENCY, x)
+#define QFWD_SWITCH_PORT_MODE_FWD_URGENCY_GET(x)\
+ FIELD_GET(QFWD_SWITCH_PORT_MODE_FWD_URGENCY, x)
+
+#define QFWD_SWITCH_PORT_MODE_YEL_RSRVD GENMASK(9, 6)
+#define QFWD_SWITCH_PORT_MODE_YEL_RSRVD_SET(x)\
+ FIELD_PREP(QFWD_SWITCH_PORT_MODE_YEL_RSRVD, x)
+#define QFWD_SWITCH_PORT_MODE_YEL_RSRVD_GET(x)\
+ FIELD_GET(QFWD_SWITCH_PORT_MODE_YEL_RSRVD, x)
+
+#define QFWD_SWITCH_PORT_MODE_INGRESS_DROP_MODE BIT(5)
+#define QFWD_SWITCH_PORT_MODE_INGRESS_DROP_MODE_SET(x)\
+ FIELD_PREP(QFWD_SWITCH_PORT_MODE_INGRESS_DROP_MODE, x)
+#define QFWD_SWITCH_PORT_MODE_INGRESS_DROP_MODE_GET(x)\
+ FIELD_GET(QFWD_SWITCH_PORT_MODE_INGRESS_DROP_MODE, x)
+
+#define QFWD_SWITCH_PORT_MODE_IGR_NO_SHARING BIT(4)
+#define QFWD_SWITCH_PORT_MODE_IGR_NO_SHARING_SET(x)\
+ FIELD_PREP(QFWD_SWITCH_PORT_MODE_IGR_NO_SHARING, x)
+#define QFWD_SWITCH_PORT_MODE_IGR_NO_SHARING_GET(x)\
+ FIELD_GET(QFWD_SWITCH_PORT_MODE_IGR_NO_SHARING, x)
+
+#define QFWD_SWITCH_PORT_MODE_EGR_NO_SHARING BIT(3)
+#define QFWD_SWITCH_PORT_MODE_EGR_NO_SHARING_SET(x)\
+ FIELD_PREP(QFWD_SWITCH_PORT_MODE_EGR_NO_SHARING, x)
+#define QFWD_SWITCH_PORT_MODE_EGR_NO_SHARING_GET(x)\
+ FIELD_GET(QFWD_SWITCH_PORT_MODE_EGR_NO_SHARING, x)
+
+#define QFWD_SWITCH_PORT_MODE_EGRESS_DROP_MODE BIT(2)
+#define QFWD_SWITCH_PORT_MODE_EGRESS_DROP_MODE_SET(x)\
+ FIELD_PREP(QFWD_SWITCH_PORT_MODE_EGRESS_DROP_MODE, x)
+#define QFWD_SWITCH_PORT_MODE_EGRESS_DROP_MODE_GET(x)\
+ FIELD_GET(QFWD_SWITCH_PORT_MODE_EGRESS_DROP_MODE, x)
+
+#define QFWD_SWITCH_PORT_MODE_EGRESS_RSRV_DIS BIT(1)
+#define QFWD_SWITCH_PORT_MODE_EGRESS_RSRV_DIS_SET(x)\
+ FIELD_PREP(QFWD_SWITCH_PORT_MODE_EGRESS_RSRV_DIS, x)
+#define QFWD_SWITCH_PORT_MODE_EGRESS_RSRV_DIS_GET(x)\
+ FIELD_GET(QFWD_SWITCH_PORT_MODE_EGRESS_RSRV_DIS, x)
+
+#define QFWD_SWITCH_PORT_MODE_LEARNALL_MORE BIT(0)
+#define QFWD_SWITCH_PORT_MODE_LEARNALL_MORE_SET(x)\
+ FIELD_PREP(QFWD_SWITCH_PORT_MODE_LEARNALL_MORE, x)
+#define QFWD_SWITCH_PORT_MODE_LEARNALL_MORE_GET(x)\
+ FIELD_GET(QFWD_SWITCH_PORT_MODE_LEARNALL_MORE, x)
+
+/* QRES:RES_CTRL:RES_CFG */
+#define QRES_RES_CFG(g) __REG(TARGET_QRES, 0, 1, 0, g, 5120, 16, 0, 0, 1, 4)
+
+#define QRES_RES_CFG_WM_HIGH GENMASK(11, 0)
+#define QRES_RES_CFG_WM_HIGH_SET(x)\
+ FIELD_PREP(QRES_RES_CFG_WM_HIGH, x)
+#define QRES_RES_CFG_WM_HIGH_GET(x)\
+ FIELD_GET(QRES_RES_CFG_WM_HIGH, x)
+
+/* QRES:RES_CTRL:RES_STAT */
+#define QRES_RES_STAT(g) __REG(TARGET_QRES, 0, 1, 0, g, 5120, 16, 4, 0, 1, 4)
+
+#define QRES_RES_STAT_MAXUSE GENMASK(20, 0)
+#define QRES_RES_STAT_MAXUSE_SET(x)\
+ FIELD_PREP(QRES_RES_STAT_MAXUSE, x)
+#define QRES_RES_STAT_MAXUSE_GET(x)\
+ FIELD_GET(QRES_RES_STAT_MAXUSE, x)
+
+/* QRES:RES_CTRL:RES_STAT_CUR */
+#define QRES_RES_STAT_CUR(g) __REG(TARGET_QRES, 0, 1, 0, g, 5120, 16, 8, 0, 1, 4)
+
+#define QRES_RES_STAT_CUR_INUSE GENMASK(20, 0)
+#define QRES_RES_STAT_CUR_INUSE_SET(x)\
+ FIELD_PREP(QRES_RES_STAT_CUR_INUSE, x)
+#define QRES_RES_STAT_CUR_INUSE_GET(x)\
+ FIELD_GET(QRES_RES_STAT_CUR_INUSE, x)
+
+/* DEVCPU_QS:XTR:XTR_GRP_CFG */
+#define QS_XTR_GRP_CFG(r) __REG(TARGET_QS, 0, 1, 0, 0, 1, 36, 0, r, 2, 4)
+
+#define QS_XTR_GRP_CFG_MODE GENMASK(3, 2)
+#define QS_XTR_GRP_CFG_MODE_SET(x)\
+ FIELD_PREP(QS_XTR_GRP_CFG_MODE, x)
+#define QS_XTR_GRP_CFG_MODE_GET(x)\
+ FIELD_GET(QS_XTR_GRP_CFG_MODE, x)
+
+#define QS_XTR_GRP_CFG_STATUS_WORD_POS BIT(1)
+#define QS_XTR_GRP_CFG_STATUS_WORD_POS_SET(x)\
+ FIELD_PREP(QS_XTR_GRP_CFG_STATUS_WORD_POS, x)
+#define QS_XTR_GRP_CFG_STATUS_WORD_POS_GET(x)\
+ FIELD_GET(QS_XTR_GRP_CFG_STATUS_WORD_POS, x)
+
+#define QS_XTR_GRP_CFG_BYTE_SWAP BIT(0)
+#define QS_XTR_GRP_CFG_BYTE_SWAP_SET(x)\
+ FIELD_PREP(QS_XTR_GRP_CFG_BYTE_SWAP, x)
+#define QS_XTR_GRP_CFG_BYTE_SWAP_GET(x)\
+ FIELD_GET(QS_XTR_GRP_CFG_BYTE_SWAP, x)
+
+/* DEVCPU_QS:XTR:XTR_RD */
+#define QS_XTR_RD(r) __REG(TARGET_QS, 0, 1, 0, 0, 1, 36, 8, r, 2, 4)
+
+/* DEVCPU_QS:XTR:XTR_FLUSH */
+#define QS_XTR_FLUSH __REG(TARGET_QS, 0, 1, 0, 0, 1, 36, 24, 0, 1, 4)
+
+#define QS_XTR_FLUSH_FLUSH GENMASK(1, 0)
+#define QS_XTR_FLUSH_FLUSH_SET(x)\
+ FIELD_PREP(QS_XTR_FLUSH_FLUSH, x)
+#define QS_XTR_FLUSH_FLUSH_GET(x)\
+ FIELD_GET(QS_XTR_FLUSH_FLUSH, x)
+
+/* DEVCPU_QS:XTR:XTR_DATA_PRESENT */
+#define QS_XTR_DATA_PRESENT __REG(TARGET_QS, 0, 1, 0, 0, 1, 36, 28, 0, 1, 4)
+
+#define QS_XTR_DATA_PRESENT_DATA_PRESENT GENMASK(1, 0)
+#define QS_XTR_DATA_PRESENT_DATA_PRESENT_SET(x)\
+ FIELD_PREP(QS_XTR_DATA_PRESENT_DATA_PRESENT, x)
+#define QS_XTR_DATA_PRESENT_DATA_PRESENT_GET(x)\
+ FIELD_GET(QS_XTR_DATA_PRESENT_DATA_PRESENT, x)
+
+/* DEVCPU_QS:INJ:INJ_GRP_CFG */
+#define QS_INJ_GRP_CFG(r) __REG(TARGET_QS, 0, 1, 36, 0, 1, 40, 0, r, 2, 4)
+
+#define QS_INJ_GRP_CFG_MODE GENMASK(3, 2)
+#define QS_INJ_GRP_CFG_MODE_SET(x)\
+ FIELD_PREP(QS_INJ_GRP_CFG_MODE, x)
+#define QS_INJ_GRP_CFG_MODE_GET(x)\
+ FIELD_GET(QS_INJ_GRP_CFG_MODE, x)
+
+#define QS_INJ_GRP_CFG_BYTE_SWAP BIT(0)
+#define QS_INJ_GRP_CFG_BYTE_SWAP_SET(x)\
+ FIELD_PREP(QS_INJ_GRP_CFG_BYTE_SWAP, x)
+#define QS_INJ_GRP_CFG_BYTE_SWAP_GET(x)\
+ FIELD_GET(QS_INJ_GRP_CFG_BYTE_SWAP, x)
+
+/* DEVCPU_QS:INJ:INJ_WR */
+#define QS_INJ_WR(r) __REG(TARGET_QS, 0, 1, 36, 0, 1, 40, 8, r, 2, 4)
+
+/* DEVCPU_QS:INJ:INJ_CTRL */
+#define QS_INJ_CTRL(r) __REG(TARGET_QS, 0, 1, 36, 0, 1, 40, 16, r, 2, 4)
+
+#define QS_INJ_CTRL_GAP_SIZE GENMASK(24, 21)
+#define QS_INJ_CTRL_GAP_SIZE_SET(x)\
+ FIELD_PREP(QS_INJ_CTRL_GAP_SIZE, x)
+#define QS_INJ_CTRL_GAP_SIZE_GET(x)\
+ FIELD_GET(QS_INJ_CTRL_GAP_SIZE, x)
+
+#define QS_INJ_CTRL_ABORT BIT(20)
+#define QS_INJ_CTRL_ABORT_SET(x)\
+ FIELD_PREP(QS_INJ_CTRL_ABORT, x)
+#define QS_INJ_CTRL_ABORT_GET(x)\
+ FIELD_GET(QS_INJ_CTRL_ABORT, x)
+
+#define QS_INJ_CTRL_EOF BIT(19)
+#define QS_INJ_CTRL_EOF_SET(x)\
+ FIELD_PREP(QS_INJ_CTRL_EOF, x)
+#define QS_INJ_CTRL_EOF_GET(x)\
+ FIELD_GET(QS_INJ_CTRL_EOF, x)
+
+#define QS_INJ_CTRL_SOF BIT(18)
+#define QS_INJ_CTRL_SOF_SET(x)\
+ FIELD_PREP(QS_INJ_CTRL_SOF, x)
+#define QS_INJ_CTRL_SOF_GET(x)\
+ FIELD_GET(QS_INJ_CTRL_SOF, x)
+
+#define QS_INJ_CTRL_VLD_BYTES GENMASK(17, 16)
+#define QS_INJ_CTRL_VLD_BYTES_SET(x)\
+ FIELD_PREP(QS_INJ_CTRL_VLD_BYTES, x)
+#define QS_INJ_CTRL_VLD_BYTES_GET(x)\
+ FIELD_GET(QS_INJ_CTRL_VLD_BYTES, x)
+
+/* DEVCPU_QS:INJ:INJ_STATUS */
+#define QS_INJ_STATUS __REG(TARGET_QS, 0, 1, 36, 0, 1, 40, 24, 0, 1, 4)
+
+#define QS_INJ_STATUS_WMARK_REACHED GENMASK(5, 4)
+#define QS_INJ_STATUS_WMARK_REACHED_SET(x)\
+ FIELD_PREP(QS_INJ_STATUS_WMARK_REACHED, x)
+#define QS_INJ_STATUS_WMARK_REACHED_GET(x)\
+ FIELD_GET(QS_INJ_STATUS_WMARK_REACHED, x)
+
+#define QS_INJ_STATUS_FIFO_RDY GENMASK(3, 2)
+#define QS_INJ_STATUS_FIFO_RDY_SET(x)\
+ FIELD_PREP(QS_INJ_STATUS_FIFO_RDY, x)
+#define QS_INJ_STATUS_FIFO_RDY_GET(x)\
+ FIELD_GET(QS_INJ_STATUS_FIFO_RDY, x)
+
+#define QS_INJ_STATUS_INJ_IN_PROGRESS GENMASK(1, 0)
+#define QS_INJ_STATUS_INJ_IN_PROGRESS_SET(x)\
+ FIELD_PREP(QS_INJ_STATUS_INJ_IN_PROGRESS, x)
+#define QS_INJ_STATUS_INJ_IN_PROGRESS_GET(x)\
+ FIELD_GET(QS_INJ_STATUS_INJ_IN_PROGRESS, x)
+
+/* QSYS:PAUSE_CFG:PAUSE_CFG */
+#define QSYS_PAUSE_CFG(r) __REG(TARGET_QSYS, 0, 1, 544, 0, 1, 1128, 0, r, 70, 4)
+
+#define QSYS_PAUSE_CFG_PAUSE_START GENMASK(25, 14)
+#define QSYS_PAUSE_CFG_PAUSE_START_SET(x)\
+ FIELD_PREP(QSYS_PAUSE_CFG_PAUSE_START, x)
+#define QSYS_PAUSE_CFG_PAUSE_START_GET(x)\
+ FIELD_GET(QSYS_PAUSE_CFG_PAUSE_START, x)
+
+#define QSYS_PAUSE_CFG_PAUSE_STOP GENMASK(13, 2)
+#define QSYS_PAUSE_CFG_PAUSE_STOP_SET(x)\
+ FIELD_PREP(QSYS_PAUSE_CFG_PAUSE_STOP, x)
+#define QSYS_PAUSE_CFG_PAUSE_STOP_GET(x)\
+ FIELD_GET(QSYS_PAUSE_CFG_PAUSE_STOP, x)
+
+#define QSYS_PAUSE_CFG_PAUSE_ENA BIT(1)
+#define QSYS_PAUSE_CFG_PAUSE_ENA_SET(x)\
+ FIELD_PREP(QSYS_PAUSE_CFG_PAUSE_ENA, x)
+#define QSYS_PAUSE_CFG_PAUSE_ENA_GET(x)\
+ FIELD_GET(QSYS_PAUSE_CFG_PAUSE_ENA, x)
+
+#define QSYS_PAUSE_CFG_AGGRESSIVE_TAILDROP_ENA BIT(0)
+#define QSYS_PAUSE_CFG_AGGRESSIVE_TAILDROP_ENA_SET(x)\
+ FIELD_PREP(QSYS_PAUSE_CFG_AGGRESSIVE_TAILDROP_ENA, x)
+#define QSYS_PAUSE_CFG_AGGRESSIVE_TAILDROP_ENA_GET(x)\
+ FIELD_GET(QSYS_PAUSE_CFG_AGGRESSIVE_TAILDROP_ENA, x)
+
+/* QSYS:PAUSE_CFG:ATOP */
+#define QSYS_ATOP(r) __REG(TARGET_QSYS, 0, 1, 544, 0, 1, 1128, 284, r, 70, 4)
+
+#define QSYS_ATOP_ATOP GENMASK(11, 0)
+#define QSYS_ATOP_ATOP_SET(x)\
+ FIELD_PREP(QSYS_ATOP_ATOP, x)
+#define QSYS_ATOP_ATOP_GET(x)\
+ FIELD_GET(QSYS_ATOP_ATOP, x)
+
+/* QSYS:PAUSE_CFG:FWD_PRESSURE */
+#define QSYS_FWD_PRESSURE(r) __REG(TARGET_QSYS, 0, 1, 544, 0, 1, 1128, 564, r, 70, 4)
+
+#define QSYS_FWD_PRESSURE_FWD_PRESSURE GENMASK(11, 1)
+#define QSYS_FWD_PRESSURE_FWD_PRESSURE_SET(x)\
+ FIELD_PREP(QSYS_FWD_PRESSURE_FWD_PRESSURE, x)
+#define QSYS_FWD_PRESSURE_FWD_PRESSURE_GET(x)\
+ FIELD_GET(QSYS_FWD_PRESSURE_FWD_PRESSURE, x)
+
+#define QSYS_FWD_PRESSURE_FWD_PRESSURE_DIS BIT(0)
+#define QSYS_FWD_PRESSURE_FWD_PRESSURE_DIS_SET(x)\
+ FIELD_PREP(QSYS_FWD_PRESSURE_FWD_PRESSURE_DIS, x)
+#define QSYS_FWD_PRESSURE_FWD_PRESSURE_DIS_GET(x)\
+ FIELD_GET(QSYS_FWD_PRESSURE_FWD_PRESSURE_DIS, x)
+
+/* QSYS:PAUSE_CFG:ATOP_TOT_CFG */
+#define QSYS_ATOP_TOT_CFG __REG(TARGET_QSYS, 0, 1, 544, 0, 1, 1128, 844, 0, 1, 4)
+
+#define QSYS_ATOP_TOT_CFG_ATOP_TOT GENMASK(11, 0)
+#define QSYS_ATOP_TOT_CFG_ATOP_TOT_SET(x)\
+ FIELD_PREP(QSYS_ATOP_TOT_CFG_ATOP_TOT, x)
+#define QSYS_ATOP_TOT_CFG_ATOP_TOT_GET(x)\
+ FIELD_GET(QSYS_ATOP_TOT_CFG_ATOP_TOT, x)
+
+/* QSYS:CALCFG:CAL_AUTO */
+#define QSYS_CAL_AUTO(r) __REG(TARGET_QSYS, 0, 1, 2304, 0, 1, 40, 0, r, 7, 4)
+
+#define QSYS_CAL_AUTO_CAL_AUTO GENMASK(29, 0)
+#define QSYS_CAL_AUTO_CAL_AUTO_SET(x)\
+ FIELD_PREP(QSYS_CAL_AUTO_CAL_AUTO, x)
+#define QSYS_CAL_AUTO_CAL_AUTO_GET(x)\
+ FIELD_GET(QSYS_CAL_AUTO_CAL_AUTO, x)
+
+/* QSYS:CALCFG:CAL_CTRL */
+#define QSYS_CAL_CTRL __REG(TARGET_QSYS, 0, 1, 2304, 0, 1, 40, 36, 0, 1, 4)
+
+#define QSYS_CAL_CTRL_CAL_MODE GENMASK(14, 11)
+#define QSYS_CAL_CTRL_CAL_MODE_SET(x)\
+ FIELD_PREP(QSYS_CAL_CTRL_CAL_MODE, x)
+#define QSYS_CAL_CTRL_CAL_MODE_GET(x)\
+ FIELD_GET(QSYS_CAL_CTRL_CAL_MODE, x)
+
+#define QSYS_CAL_CTRL_CAL_AUTO_GRANT_RATE GENMASK(10, 1)
+#define QSYS_CAL_CTRL_CAL_AUTO_GRANT_RATE_SET(x)\
+ FIELD_PREP(QSYS_CAL_CTRL_CAL_AUTO_GRANT_RATE, x)
+#define QSYS_CAL_CTRL_CAL_AUTO_GRANT_RATE_GET(x)\
+ FIELD_GET(QSYS_CAL_CTRL_CAL_AUTO_GRANT_RATE, x)
+
+#define QSYS_CAL_CTRL_CAL_AUTO_ERROR BIT(0)
+#define QSYS_CAL_CTRL_CAL_AUTO_ERROR_SET(x)\
+ FIELD_PREP(QSYS_CAL_CTRL_CAL_AUTO_ERROR, x)
+#define QSYS_CAL_CTRL_CAL_AUTO_ERROR_GET(x)\
+ FIELD_GET(QSYS_CAL_CTRL_CAL_AUTO_ERROR, x)
+
+/* QSYS:RAM_CTRL:RAM_INIT */
+#define QSYS_RAM_INIT __REG(TARGET_QSYS, 0, 1, 2344, 0, 1, 4, 0, 0, 1, 4)
+
+#define QSYS_RAM_INIT_RAM_INIT BIT(1)
+#define QSYS_RAM_INIT_RAM_INIT_SET(x)\
+ FIELD_PREP(QSYS_RAM_INIT_RAM_INIT, x)
+#define QSYS_RAM_INIT_RAM_INIT_GET(x)\
+ FIELD_GET(QSYS_RAM_INIT_RAM_INIT, x)
+
+#define QSYS_RAM_INIT_RAM_CFG_HOOK BIT(0)
+#define QSYS_RAM_INIT_RAM_CFG_HOOK_SET(x)\
+ FIELD_PREP(QSYS_RAM_INIT_RAM_CFG_HOOK, x)
+#define QSYS_RAM_INIT_RAM_CFG_HOOK_GET(x)\
+ FIELD_GET(QSYS_RAM_INIT_RAM_CFG_HOOK, x)
+
+/* REW:COMMON:OWN_UPSID */
+#define REW_OWN_UPSID(r) __REG(TARGET_REW, 0, 1, 387264, 0, 1, 1232, 0, r, 3, 4)
+
+#define REW_OWN_UPSID_OWN_UPSID GENMASK(4, 0)
+#define REW_OWN_UPSID_OWN_UPSID_SET(x)\
+ FIELD_PREP(REW_OWN_UPSID_OWN_UPSID, x)
+#define REW_OWN_UPSID_OWN_UPSID_GET(x)\
+ FIELD_GET(REW_OWN_UPSID_OWN_UPSID, x)
+
+/* REW:PORT:PORT_VLAN_CFG */
+#define REW_PORT_VLAN_CFG(g) __REG(TARGET_REW, 0, 1, 360448, g, 70, 256, 0, 0, 1, 4)
+
+#define REW_PORT_VLAN_CFG_PORT_PCP GENMASK(15, 13)
+#define REW_PORT_VLAN_CFG_PORT_PCP_SET(x)\
+ FIELD_PREP(REW_PORT_VLAN_CFG_PORT_PCP, x)
+#define REW_PORT_VLAN_CFG_PORT_PCP_GET(x)\
+ FIELD_GET(REW_PORT_VLAN_CFG_PORT_PCP, x)
+
+#define REW_PORT_VLAN_CFG_PORT_DEI BIT(12)
+#define REW_PORT_VLAN_CFG_PORT_DEI_SET(x)\
+ FIELD_PREP(REW_PORT_VLAN_CFG_PORT_DEI, x)
+#define REW_PORT_VLAN_CFG_PORT_DEI_GET(x)\
+ FIELD_GET(REW_PORT_VLAN_CFG_PORT_DEI, x)
+
+#define REW_PORT_VLAN_CFG_PORT_VID GENMASK(11, 0)
+#define REW_PORT_VLAN_CFG_PORT_VID_SET(x)\
+ FIELD_PREP(REW_PORT_VLAN_CFG_PORT_VID, x)
+#define REW_PORT_VLAN_CFG_PORT_VID_GET(x)\
+ FIELD_GET(REW_PORT_VLAN_CFG_PORT_VID, x)
+
+/* REW:PORT:TAG_CTRL */
+#define REW_TAG_CTRL(g) __REG(TARGET_REW, 0, 1, 360448, g, 70, 256, 132, 0, 1, 4)
+
+#define REW_TAG_CTRL_TAG_CFG_OBEY_WAS_TAGGED BIT(13)
+#define REW_TAG_CTRL_TAG_CFG_OBEY_WAS_TAGGED_SET(x)\
+ FIELD_PREP(REW_TAG_CTRL_TAG_CFG_OBEY_WAS_TAGGED, x)
+#define REW_TAG_CTRL_TAG_CFG_OBEY_WAS_TAGGED_GET(x)\
+ FIELD_GET(REW_TAG_CTRL_TAG_CFG_OBEY_WAS_TAGGED, x)
+
+#define REW_TAG_CTRL_TAG_CFG GENMASK(12, 11)
+#define REW_TAG_CTRL_TAG_CFG_SET(x)\
+ FIELD_PREP(REW_TAG_CTRL_TAG_CFG, x)
+#define REW_TAG_CTRL_TAG_CFG_GET(x)\
+ FIELD_GET(REW_TAG_CTRL_TAG_CFG, x)
+
+#define REW_TAG_CTRL_TAG_TPID_CFG GENMASK(10, 8)
+#define REW_TAG_CTRL_TAG_TPID_CFG_SET(x)\
+ FIELD_PREP(REW_TAG_CTRL_TAG_TPID_CFG, x)
+#define REW_TAG_CTRL_TAG_TPID_CFG_GET(x)\
+ FIELD_GET(REW_TAG_CTRL_TAG_TPID_CFG, x)
+
+#define REW_TAG_CTRL_TAG_VID_CFG GENMASK(7, 6)
+#define REW_TAG_CTRL_TAG_VID_CFG_SET(x)\
+ FIELD_PREP(REW_TAG_CTRL_TAG_VID_CFG, x)
+#define REW_TAG_CTRL_TAG_VID_CFG_GET(x)\
+ FIELD_GET(REW_TAG_CTRL_TAG_VID_CFG, x)
+
+#define REW_TAG_CTRL_TAG_PCP_CFG GENMASK(5, 3)
+#define REW_TAG_CTRL_TAG_PCP_CFG_SET(x)\
+ FIELD_PREP(REW_TAG_CTRL_TAG_PCP_CFG, x)
+#define REW_TAG_CTRL_TAG_PCP_CFG_GET(x)\
+ FIELD_GET(REW_TAG_CTRL_TAG_PCP_CFG, x)
+
+#define REW_TAG_CTRL_TAG_DEI_CFG GENMASK(2, 0)
+#define REW_TAG_CTRL_TAG_DEI_CFG_SET(x)\
+ FIELD_PREP(REW_TAG_CTRL_TAG_DEI_CFG, x)
+#define REW_TAG_CTRL_TAG_DEI_CFG_GET(x)\
+ FIELD_GET(REW_TAG_CTRL_TAG_DEI_CFG, x)
+
+/* REW:RAM_CTRL:RAM_INIT */
+#define REW_RAM_INIT __REG(TARGET_REW, 0, 1, 378696, 0, 1, 4, 0, 0, 1, 4)
+
+#define REW_RAM_INIT_RAM_INIT BIT(1)
+#define REW_RAM_INIT_RAM_INIT_SET(x)\
+ FIELD_PREP(REW_RAM_INIT_RAM_INIT, x)
+#define REW_RAM_INIT_RAM_INIT_GET(x)\
+ FIELD_GET(REW_RAM_INIT_RAM_INIT, x)
+
+#define REW_RAM_INIT_RAM_CFG_HOOK BIT(0)
+#define REW_RAM_INIT_RAM_CFG_HOOK_SET(x)\
+ FIELD_PREP(REW_RAM_INIT_RAM_CFG_HOOK, x)
+#define REW_RAM_INIT_RAM_CFG_HOOK_GET(x)\
+ FIELD_GET(REW_RAM_INIT_RAM_CFG_HOOK, x)
+
+/* VCAP_SUPER:RAM_CTRL:RAM_INIT */
+#define VCAP_SUPER_RAM_INIT __REG(TARGET_VCAP_SUPER, 0, 1, 1120, 0, 1, 4, 0, 0, 1, 4)
+
+#define VCAP_SUPER_RAM_INIT_RAM_INIT BIT(1)
+#define VCAP_SUPER_RAM_INIT_RAM_INIT_SET(x)\
+ FIELD_PREP(VCAP_SUPER_RAM_INIT_RAM_INIT, x)
+#define VCAP_SUPER_RAM_INIT_RAM_INIT_GET(x)\
+ FIELD_GET(VCAP_SUPER_RAM_INIT_RAM_INIT, x)
+
+#define VCAP_SUPER_RAM_INIT_RAM_CFG_HOOK BIT(0)
+#define VCAP_SUPER_RAM_INIT_RAM_CFG_HOOK_SET(x)\
+ FIELD_PREP(VCAP_SUPER_RAM_INIT_RAM_CFG_HOOK, x)
+#define VCAP_SUPER_RAM_INIT_RAM_CFG_HOOK_GET(x)\
+ FIELD_GET(VCAP_SUPER_RAM_INIT_RAM_CFG_HOOK, x)
+
+/* VOP:RAM_CTRL:RAM_INIT */
+#define VOP_RAM_INIT __REG(TARGET_VOP, 0, 1, 279176, 0, 1, 4, 0, 0, 1, 4)
+
+#define VOP_RAM_INIT_RAM_INIT BIT(1)
+#define VOP_RAM_INIT_RAM_INIT_SET(x)\
+ FIELD_PREP(VOP_RAM_INIT_RAM_INIT, x)
+#define VOP_RAM_INIT_RAM_INIT_GET(x)\
+ FIELD_GET(VOP_RAM_INIT_RAM_INIT, x)
+
+#define VOP_RAM_INIT_RAM_CFG_HOOK BIT(0)
+#define VOP_RAM_INIT_RAM_CFG_HOOK_SET(x)\
+ FIELD_PREP(VOP_RAM_INIT_RAM_CFG_HOOK, x)
+#define VOP_RAM_INIT_RAM_CFG_HOOK_GET(x)\
+ FIELD_GET(VOP_RAM_INIT_RAM_CFG_HOOK, x)
+
+/* XQS:SYSTEM:STAT_CFG */
+#define XQS_STAT_CFG __REG(TARGET_XQS, 0, 1, 6768, 0, 1, 872, 860, 0, 1, 4)
+
+#define XQS_STAT_CFG_STAT_CLEAR_SHOT GENMASK(21, 18)
+#define XQS_STAT_CFG_STAT_CLEAR_SHOT_SET(x)\
+ FIELD_PREP(XQS_STAT_CFG_STAT_CLEAR_SHOT, x)
+#define XQS_STAT_CFG_STAT_CLEAR_SHOT_GET(x)\
+ FIELD_GET(XQS_STAT_CFG_STAT_CLEAR_SHOT, x)
+
+#define XQS_STAT_CFG_STAT_VIEW GENMASK(17, 5)
+#define XQS_STAT_CFG_STAT_VIEW_SET(x)\
+ FIELD_PREP(XQS_STAT_CFG_STAT_VIEW, x)
+#define XQS_STAT_CFG_STAT_VIEW_GET(x)\
+ FIELD_GET(XQS_STAT_CFG_STAT_VIEW, x)
+
+#define XQS_STAT_CFG_STAT_SRV_PKT_ONLY BIT(4)
+#define XQS_STAT_CFG_STAT_SRV_PKT_ONLY_SET(x)\
+ FIELD_PREP(XQS_STAT_CFG_STAT_SRV_PKT_ONLY, x)
+#define XQS_STAT_CFG_STAT_SRV_PKT_ONLY_GET(x)\
+ FIELD_GET(XQS_STAT_CFG_STAT_SRV_PKT_ONLY, x)
+
+#define XQS_STAT_CFG_STAT_WRAP_DIS GENMASK(3, 0)
+#define XQS_STAT_CFG_STAT_WRAP_DIS_SET(x)\
+ FIELD_PREP(XQS_STAT_CFG_STAT_WRAP_DIS, x)
+#define XQS_STAT_CFG_STAT_WRAP_DIS_GET(x)\
+ FIELD_GET(XQS_STAT_CFG_STAT_WRAP_DIS, x)
+
+/* XQS:QLIMIT_SHR:QLIMIT_SHR_TOP_CFG */
+#define XQS_QLIMIT_SHR_TOP_CFG(g) __REG(TARGET_XQS, 0, 1, 7936, g, 4, 48, 0, 0, 1, 4)
+
+#define XQS_QLIMIT_SHR_TOP_CFG_QLIMIT_SHR_TOP GENMASK(14, 0)
+#define XQS_QLIMIT_SHR_TOP_CFG_QLIMIT_SHR_TOP_SET(x)\
+ FIELD_PREP(XQS_QLIMIT_SHR_TOP_CFG_QLIMIT_SHR_TOP, x)
+#define XQS_QLIMIT_SHR_TOP_CFG_QLIMIT_SHR_TOP_GET(x)\
+ FIELD_GET(XQS_QLIMIT_SHR_TOP_CFG_QLIMIT_SHR_TOP, x)
+
+/* XQS:QLIMIT_SHR:QLIMIT_SHR_ATOP_CFG */
+#define XQS_QLIMIT_SHR_ATOP_CFG(g) __REG(TARGET_XQS, 0, 1, 7936, g, 4, 48, 4, 0, 1, 4)
+
+#define XQS_QLIMIT_SHR_ATOP_CFG_QLIMIT_SHR_ATOP GENMASK(14, 0)
+#define XQS_QLIMIT_SHR_ATOP_CFG_QLIMIT_SHR_ATOP_SET(x)\
+ FIELD_PREP(XQS_QLIMIT_SHR_ATOP_CFG_QLIMIT_SHR_ATOP, x)
+#define XQS_QLIMIT_SHR_ATOP_CFG_QLIMIT_SHR_ATOP_GET(x)\
+ FIELD_GET(XQS_QLIMIT_SHR_ATOP_CFG_QLIMIT_SHR_ATOP, x)
+
+/* XQS:QLIMIT_SHR:QLIMIT_SHR_CTOP_CFG */
+#define XQS_QLIMIT_SHR_CTOP_CFG(g) __REG(TARGET_XQS, 0, 1, 7936, g, 4, 48, 8, 0, 1, 4)
+
+#define XQS_QLIMIT_SHR_CTOP_CFG_QLIMIT_SHR_CTOP GENMASK(14, 0)
+#define XQS_QLIMIT_SHR_CTOP_CFG_QLIMIT_SHR_CTOP_SET(x)\
+ FIELD_PREP(XQS_QLIMIT_SHR_CTOP_CFG_QLIMIT_SHR_CTOP, x)
+#define XQS_QLIMIT_SHR_CTOP_CFG_QLIMIT_SHR_CTOP_GET(x)\
+ FIELD_GET(XQS_QLIMIT_SHR_CTOP_CFG_QLIMIT_SHR_CTOP, x)
+
+/* XQS:QLIMIT_SHR:QLIMIT_SHR_QLIM_CFG */
+#define XQS_QLIMIT_SHR_QLIM_CFG(g) __REG(TARGET_XQS, 0, 1, 7936, g, 4, 48, 12, 0, 1, 4)
+
+#define XQS_QLIMIT_SHR_QLIM_CFG_QLIMIT_SHR_QLIM GENMASK(14, 0)
+#define XQS_QLIMIT_SHR_QLIM_CFG_QLIMIT_SHR_QLIM_SET(x)\
+ FIELD_PREP(XQS_QLIMIT_SHR_QLIM_CFG_QLIMIT_SHR_QLIM, x)
+#define XQS_QLIMIT_SHR_QLIM_CFG_QLIMIT_SHR_QLIM_GET(x)\
+ FIELD_GET(XQS_QLIMIT_SHR_QLIM_CFG_QLIMIT_SHR_QLIM, x)
+
+/* XQS:STAT:CNT */
+#define XQS_CNT(g) __REG(TARGET_XQS, 0, 1, 0, g, 1024, 4, 0, 0, 1, 4)
+
+#endif /* _SPARX5_MAIN_REGS_H_ */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/* Microchip Sparx5 Switch driver
+ *
+ * Copyright (c) 2021 Microchip Technology Inc. and its subsidiaries.
+ */
+
+#include "sparx5_main_regs.h"
+#include "sparx5_main.h"
+#include "sparx5_port.h"
+
+/* The IFH bit position of the first VSTAX bit. This is because the
+ * VSTAX bit positions in Data sheet is starting from zero.
+ */
+#define VSTAX 73
+
+static void ifh_encode_bitfield(void *ifh, u64 value, u32 pos, u32 width)
+{
+ u8 *ifh_hdr = ifh;
+ /* Calculate the Start IFH byte position of this IFH bit position */
+ u32 byte = (35 - (pos / 8));
+ /* Calculate the Start bit position in the Start IFH byte */
+ u32 bit = (pos % 8);
+ u64 encode = GENMASK(bit + width - 1, bit) & (value << bit);
+
+ /* Max width is 5 bytes - 40 bits. In worst case this will
+ * spread over 6 bytes - 48 bits
+ */
+ compiletime_assert(width <= 40, "Unsupported width, must be <= 40");
+
+ /* The b0-b7 goes into the start IFH byte */
+ if (encode & 0xFF)
+ ifh_hdr[byte] |= (u8)((encode & 0xFF));
+ /* The b8-b15 goes into the next IFH byte */
+ if (encode & 0xFF00)
+ ifh_hdr[byte - 1] |= (u8)((encode & 0xFF00) >> 8);
+ /* The b16-b23 goes into the next IFH byte */
+ if (encode & 0xFF0000)
+ ifh_hdr[byte - 2] |= (u8)((encode & 0xFF0000) >> 16);
+ /* The b24-b31 goes into the next IFH byte */
+ if (encode & 0xFF000000)
+ ifh_hdr[byte - 3] |= (u8)((encode & 0xFF000000) >> 24);
+ /* The b32-b39 goes into the next IFH byte */
+ if (encode & 0xFF00000000)
+ ifh_hdr[byte - 4] |= (u8)((encode & 0xFF00000000) >> 32);
+ /* The b40-b47 goes into the next IFH byte */
+ if (encode & 0xFF0000000000)
+ ifh_hdr[byte - 5] |= (u8)((encode & 0xFF0000000000) >> 40);
+}
+
+static void sparx5_set_port_ifh(void *ifh_hdr, u16 portno)
+{
+ /* VSTAX.RSV = 1. MSBit must be 1 */
+ ifh_encode_bitfield(ifh_hdr, 1, VSTAX + 79, 1);
+ /* VSTAX.INGR_DROP_MODE = Enable. Don't make head-of-line blocking */
+ ifh_encode_bitfield(ifh_hdr, 1, VSTAX + 55, 1);
+ /* MISC.CPU_MASK/DPORT = Destination port */
+ ifh_encode_bitfield(ifh_hdr, portno, 29, 8);
+ /* MISC.PIPELINE_PT */
+ ifh_encode_bitfield(ifh_hdr, 16, 37, 5);
+ /* MISC.PIPELINE_ACT */
+ ifh_encode_bitfield(ifh_hdr, 1, 42, 3);
+ /* FWD.SRC_PORT = CPU */
+ ifh_encode_bitfield(ifh_hdr, SPX5_PORT_CPU, 46, 7);
+ /* FWD.SFLOW_ID (disable SFlow sampling) */
+ ifh_encode_bitfield(ifh_hdr, 124, 57, 7);
+ /* FWD.UPDATE_FCS = Enable. Enforce update of FCS. */
+ ifh_encode_bitfield(ifh_hdr, 1, 67, 1);
+}
+
+static int sparx5_port_open(struct net_device *ndev)
+{
+ struct sparx5_port *port = netdev_priv(ndev);
+ int err = 0;
+
+ sparx5_port_enable(port, true);
+ err = phylink_of_phy_connect(port->phylink, port->of_node, 0);
+ if (err) {
+ netdev_err(ndev, "Could not attach to PHY\n");
+ return err;
+ }
+
+ phylink_start(port->phylink);
+
+ if (!ndev->phydev) {
+ /* power up serdes */
+ port->conf.power_down = false;
+ if (port->conf.serdes_reset)
+ err = sparx5_serdes_set(port->sparx5, port, &port->conf);
+ else
+ err = phy_power_on(port->serdes);
+ if (err)
+ netdev_err(ndev, "%s failed\n", __func__);
+ }
+
+ return err;
+}
+
+static int sparx5_port_stop(struct net_device *ndev)
+{
+ struct sparx5_port *port = netdev_priv(ndev);
+ int err = 0;
+
+ sparx5_port_enable(port, false);
+ phylink_stop(port->phylink);
+ phylink_disconnect_phy(port->phylink);
+
+ if (!ndev->phydev) {
+ /* power down serdes */
+ port->conf.power_down = true;
+ if (port->conf.serdes_reset)
+ err = sparx5_serdes_set(port->sparx5, port, &port->conf);
+ else
+ err = phy_power_off(port->serdes);
+ if (err)
+ netdev_err(ndev, "%s failed\n", __func__);
+ }
+ return 0;
+}
+
+static void sparx5_set_rx_mode(struct net_device *dev)
+{
+ struct sparx5_port *port = netdev_priv(dev);
+ struct sparx5 *sparx5 = port->sparx5;
+
+ if (!test_bit(port->portno, sparx5->bridge_mask))
+ __dev_mc_sync(dev, sparx5_mc_sync, sparx5_mc_unsync);
+}
+
+static int sparx5_port_get_phys_port_name(struct net_device *dev,
+ char *buf, size_t len)
+{
+ struct sparx5_port *port = netdev_priv(dev);
+ int ret;
+
+ ret = snprintf(buf, len, "p%d", port->portno);
+ if (ret >= len)
+ return -EINVAL;
+
+ return 0;
+}
+
+static int sparx5_set_mac_address(struct net_device *dev, void *p)
+{
+ struct sparx5_port *port = netdev_priv(dev);
+ struct sparx5 *sparx5 = port->sparx5;
+ const struct sockaddr *addr = p;
+
+ if (!is_valid_ether_addr(addr->sa_data))
+ return -EADDRNOTAVAIL;
+
+ /* Remove current */
+ sparx5_mact_forget(sparx5, dev->dev_addr, port->pvid);
+
+ /* Add new */
+ sparx5_mact_learn(sparx5, PGID_CPU, addr->sa_data, port->pvid);
+
+ /* Record the address */
+ ether_addr_copy(dev->dev_addr, addr->sa_data);
+
+ return 0;
+}
+
+static int sparx5_get_port_parent_id(struct net_device *dev,
+ struct netdev_phys_item_id *ppid)
+{
+ struct sparx5_port *sparx5_port = netdev_priv(dev);
+ struct sparx5 *sparx5 = sparx5_port->sparx5;
+
+ ppid->id_len = sizeof(sparx5->base_mac);
+ memcpy(&ppid->id, &sparx5->base_mac, ppid->id_len);
+
+ return 0;
+}
+
+static const struct net_device_ops sparx5_port_netdev_ops = {
+ .ndo_open = sparx5_port_open,
+ .ndo_stop = sparx5_port_stop,
+ .ndo_start_xmit = sparx5_port_xmit_impl,
+ .ndo_set_rx_mode = sparx5_set_rx_mode,
+ .ndo_get_phys_port_name = sparx5_port_get_phys_port_name,
+ .ndo_set_mac_address = sparx5_set_mac_address,
+ .ndo_validate_addr = eth_validate_addr,
+ .ndo_get_stats64 = sparx5_get_stats64,
+ .ndo_get_port_parent_id = sparx5_get_port_parent_id,
+};
+
+bool sparx5_netdevice_check(const struct net_device *dev)
+{
+ return dev && (dev->netdev_ops == &sparx5_port_netdev_ops);
+}
+
+struct net_device *sparx5_create_netdev(struct sparx5 *sparx5, u32 portno)
+{
+ struct sparx5_port *spx5_port;
+ struct net_device *ndev;
+ u64 val;
+
+ ndev = devm_alloc_etherdev(sparx5->dev, sizeof(struct sparx5_port));
+ if (!ndev)
+ return ERR_PTR(-ENOMEM);
+
+ SET_NETDEV_DEV(ndev, sparx5->dev);
+ spx5_port = netdev_priv(ndev);
+ spx5_port->ndev = ndev;
+ spx5_port->sparx5 = sparx5;
+ spx5_port->portno = portno;
+ sparx5_set_port_ifh(spx5_port->ifh, portno);
+
+ ndev->netdev_ops = &sparx5_port_netdev_ops;
+ ndev->ethtool_ops = &sparx5_ethtool_ops;
+
+ val = ether_addr_to_u64(sparx5->base_mac) + portno + 1;
+ u64_to_ether_addr(val, ndev->dev_addr);
+
+ return ndev;
+}
+
+int sparx5_register_netdevs(struct sparx5 *sparx5)
+{
+ int portno;
+ int err;
+
+ for (portno = 0; portno < SPX5_PORTS; portno++)
+ if (sparx5->ports[portno]) {
+ err = register_netdev(sparx5->ports[portno]->ndev);
+ if (err) {
+ dev_err(sparx5->dev,
+ "port: %02u: netdev registration failed\n",
+ portno);
+ return err;
+ }
+ sparx5_port_inj_timer_setup(sparx5->ports[portno]);
+ }
+ return 0;
+}
+
+void sparx5_destroy_netdevs(struct sparx5 *sparx5)
+{
+ struct sparx5_port *port;
+ int portno;
+
+ for (portno = 0; portno < SPX5_PORTS; portno++) {
+ port = sparx5->ports[portno];
+ if (port && port->phylink) {
+ /* Disconnect the phy */
+ rtnl_lock();
+ sparx5_port_stop(port->ndev);
+ phylink_disconnect_phy(port->phylink);
+ rtnl_unlock();
+ phylink_destroy(port->phylink);
+ port->phylink = NULL;
+ }
+ }
+}
+
+void sparx5_unregister_netdevs(struct sparx5 *sparx5)
+{
+ int portno;
+
+ for (portno = 0; portno < SPX5_PORTS; portno++)
+ if (sparx5->ports[portno])
+ unregister_netdev(sparx5->ports[portno]->ndev);
+}
+
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/* Microchip Sparx5 Switch driver
+ *
+ * Copyright (c) 2021 Microchip Technology Inc. and its subsidiaries.
+ */
+
+#include "sparx5_main_regs.h"
+#include "sparx5_main.h"
+
+#define XTR_EOF_0 ntohl((__force __be32)0x80000000u)
+#define XTR_EOF_1 ntohl((__force __be32)0x80000001u)
+#define XTR_EOF_2 ntohl((__force __be32)0x80000002u)
+#define XTR_EOF_3 ntohl((__force __be32)0x80000003u)
+#define XTR_PRUNED ntohl((__force __be32)0x80000004u)
+#define XTR_ABORT ntohl((__force __be32)0x80000005u)
+#define XTR_ESCAPE ntohl((__force __be32)0x80000006u)
+#define XTR_NOT_READY ntohl((__force __be32)0x80000007u)
+
+#define XTR_VALID_BYTES(x) (4 - ((x) & 3))
+
+#define INJ_TIMEOUT_NS 50000
+
+struct frame_info {
+ int src_port;
+};
+
+static void sparx5_xtr_flush(struct sparx5 *sparx5, u8 grp)
+{
+ /* Start flush */
+ spx5_wr(QS_XTR_FLUSH_FLUSH_SET(BIT(grp)), sparx5, QS_XTR_FLUSH);
+
+ /* Allow to drain */
+ mdelay(1);
+
+ /* All Queues normal */
+ spx5_wr(0, sparx5, QS_XTR_FLUSH);
+}
+
+static void sparx5_ifh_parse(u32 *ifh, struct frame_info *info)
+{
+ u8 *xtr_hdr = (u8 *)ifh;
+
+ /* FWD is bit 45-72 (28 bits), but we only read the 27 LSB for now */
+ u32 fwd =
+ ((u32)xtr_hdr[27] << 24) |
+ ((u32)xtr_hdr[28] << 16) |
+ ((u32)xtr_hdr[29] << 8) |
+ ((u32)xtr_hdr[30] << 0);
+ fwd = (fwd >> 5);
+ info->src_port = FIELD_GET(GENMASK(7, 1), fwd);
+}
+
+static void sparx5_xtr_grp(struct sparx5 *sparx5, u8 grp, bool byte_swap)
+{
+ bool eof_flag = false, pruned_flag = false, abort_flag = false;
+ struct net_device *netdev;
+ struct sparx5_port *port;
+ struct frame_info fi;
+ int i, byte_cnt = 0;
+ struct sk_buff *skb;
+ u32 ifh[IFH_LEN];
+ u32 *rxbuf;
+
+ /* Get IFH */
+ for (i = 0; i < IFH_LEN; i++)
+ ifh[i] = spx5_rd(sparx5, QS_XTR_RD(grp));
+
+ /* Decode IFH (whats needed) */
+ sparx5_ifh_parse(ifh, &fi);
+
+ /* Map to port netdev */
+ port = fi.src_port < SPX5_PORTS ?
+ sparx5->ports[fi.src_port] : NULL;
+ if (!port || !port->ndev) {
+ dev_err(sparx5->dev, "Data on inactive port %d\n", fi.src_port);
+ sparx5_xtr_flush(sparx5, grp);
+ return;
+ }
+
+ /* Have netdev, get skb */
+ netdev = port->ndev;
+ skb = netdev_alloc_skb(netdev, netdev->mtu + ETH_HLEN);
+ if (!skb) {
+ sparx5_xtr_flush(sparx5, grp);
+ dev_err(sparx5->dev, "No skb allocated\n");
+ netdev->stats.rx_dropped++;
+ return;
+ }
+ rxbuf = (u32 *)skb->data;
+
+ /* Now, pull frame data */
+ while (!eof_flag) {
+ u32 val = spx5_rd(sparx5, QS_XTR_RD(grp));
+ u32 cmp = val;
+
+ if (byte_swap)
+ cmp = ntohl((__force __be32)val);
+
+ switch (cmp) {
+ case XTR_NOT_READY:
+ break;
+ case XTR_ABORT:
+ /* No accompanying data */
+ abort_flag = true;
+ eof_flag = true;
+ break;
+ case XTR_EOF_0:
+ case XTR_EOF_1:
+ case XTR_EOF_2:
+ case XTR_EOF_3:
+ /* This assumes STATUS_WORD_POS == 1, Status
+ * just after last data
+ */
+ byte_cnt -= (4 - XTR_VALID_BYTES(val));
+ eof_flag = true;
+ break;
+ case XTR_PRUNED:
+ /* But get the last 4 bytes as well */
+ eof_flag = true;
+ pruned_flag = true;
+ fallthrough;
+ case XTR_ESCAPE:
+ *rxbuf = spx5_rd(sparx5, QS_XTR_RD(grp));
+ byte_cnt += 4;
+ rxbuf++;
+ break;
+ default:
+ *rxbuf = val;
+ byte_cnt += 4;
+ rxbuf++;
+ }
+ }
+
+ if (abort_flag || pruned_flag || !eof_flag) {
+ netdev_err(netdev, "Discarded frame: abort:%d pruned:%d eof:%d\n",
+ abort_flag, pruned_flag, eof_flag);
+ kfree_skb(skb);
+ netdev->stats.rx_dropped++;
+ return;
+ }
+
+ /* Everything we see on an interface that is in the HW bridge
+ * has already been forwarded
+ */
+ if (test_bit(port->portno, sparx5->bridge_mask))
+ skb->offload_fwd_mark = 1;
+
+ /* Finish up skb */
+ skb_put(skb, byte_cnt - ETH_FCS_LEN);
+ eth_skb_pad(skb);
+ skb->protocol = eth_type_trans(skb, netdev);
+ netif_rx(skb);
+ netdev->stats.rx_bytes += skb->len;
+ netdev->stats.rx_packets++;
+}
+
+static int sparx5_inject(struct sparx5 *sparx5,
+ u32 *ifh,
+ struct sk_buff *skb,
+ struct net_device *ndev)
+{
+ int grp = INJ_QUEUE;
+ u32 val, w, count;
+ u8 *buf;
+
+ val = spx5_rd(sparx5, QS_INJ_STATUS);
+ if (!(QS_INJ_STATUS_FIFO_RDY_GET(val) & BIT(grp))) {
+ pr_err_ratelimited("Injection: Queue not ready: 0x%lx\n",
+ QS_INJ_STATUS_FIFO_RDY_GET(val));
+ return -EBUSY;
+ }
+
+ /* Indicate SOF */
+ spx5_wr(QS_INJ_CTRL_SOF_SET(1) |
+ QS_INJ_CTRL_GAP_SIZE_SET(1),
+ sparx5, QS_INJ_CTRL(grp));
+
+ /* Write the IFH to the chip. */
+ for (w = 0; w < IFH_LEN; w++)
+ spx5_wr(ifh[w], sparx5, QS_INJ_WR(grp));
+
+ /* Write words, round up */
+ count = DIV_ROUND_UP(skb->len, 4);
+ buf = skb->data;
+ for (w = 0; w < count; w++, buf += 4) {
+ val = get_unaligned((const u32 *)buf);
+ spx5_wr(val, sparx5, QS_INJ_WR(grp));
+ }
+
+ /* Add padding */
+ while (w < (60 / 4)) {
+ spx5_wr(0, sparx5, QS_INJ_WR(grp));
+ w++;
+ }
+
+ /* Indicate EOF and valid bytes in last word */
+ spx5_wr(QS_INJ_CTRL_GAP_SIZE_SET(1) |
+ QS_INJ_CTRL_VLD_BYTES_SET(skb->len < 60 ? 0 : skb->len % 4) |
+ QS_INJ_CTRL_EOF_SET(1),
+ sparx5, QS_INJ_CTRL(grp));
+
+ /* Add dummy CRC */
+ spx5_wr(0, sparx5, QS_INJ_WR(grp));
+ w++;
+
+ val = spx5_rd(sparx5, QS_INJ_STATUS);
+ if (QS_INJ_STATUS_WMARK_REACHED_GET(val) & BIT(grp)) {
+ struct sparx5_port *port = netdev_priv(ndev);
+
+ pr_err_ratelimited("Injection: Watermark reached: 0x%lx\n",
+ QS_INJ_STATUS_WMARK_REACHED_GET(val));
+ netif_stop_queue(ndev);
+ hrtimer_start(&port->inj_timer, INJ_TIMEOUT_NS,
+ HRTIMER_MODE_REL);
+ }
+
+ return NETDEV_TX_OK;
+}
+
+int sparx5_port_xmit_impl(struct sk_buff *skb, struct net_device *dev)
+{
+ struct net_device_stats *stats = &dev->stats;
+ struct sparx5_port *port = netdev_priv(dev);
+ struct sparx5 *sparx5 = port->sparx5;
+ int ret;
+
+ ret = sparx5_inject(sparx5, port->ifh, skb, dev);
+
+ if (ret == NETDEV_TX_OK) {
+ stats->tx_bytes += skb->len;
+ stats->tx_packets++;
+ skb_tx_timestamp(skb);
+ dev_kfree_skb_any(skb);
+ } else {
+ stats->tx_dropped++;
+ }
+ return ret;
+}
+
+static enum hrtimer_restart sparx5_injection_timeout(struct hrtimer *tmr)
+{
+ struct sparx5_port *port = container_of(tmr, struct sparx5_port,
+ inj_timer);
+ int grp = INJ_QUEUE;
+ u32 val;
+
+ val = spx5_rd(port->sparx5, QS_INJ_STATUS);
+ if (QS_INJ_STATUS_WMARK_REACHED_GET(val) & BIT(grp)) {
+ pr_err_ratelimited("Injection: Reset watermark count\n");
+ /* Reset Watermark count to restart */
+ spx5_rmw(DSM_DEV_TX_STOP_WM_CFG_DEV_TX_CNT_CLR_SET(1),
+ DSM_DEV_TX_STOP_WM_CFG_DEV_TX_CNT_CLR,
+ port->sparx5,
+ DSM_DEV_TX_STOP_WM_CFG(port->portno));
+ }
+ netif_wake_queue(port->ndev);
+ return HRTIMER_NORESTART;
+}
+
+int sparx5_manual_injection_mode(struct sparx5 *sparx5)
+{
+ const int byte_swap = 1;
+ int portno;
+
+ /* Change mode to manual extraction and injection */
+ spx5_wr(QS_XTR_GRP_CFG_MODE_SET(1) |
+ QS_XTR_GRP_CFG_STATUS_WORD_POS_SET(1) |
+ QS_XTR_GRP_CFG_BYTE_SWAP_SET(byte_swap),
+ sparx5, QS_XTR_GRP_CFG(XTR_QUEUE));
+ spx5_wr(QS_INJ_GRP_CFG_MODE_SET(1) |
+ QS_INJ_GRP_CFG_BYTE_SWAP_SET(byte_swap),
+ sparx5, QS_INJ_GRP_CFG(INJ_QUEUE));
+
+ /* CPU ports capture setup */
+ for (portno = SPX5_PORT_CPU_0; portno <= SPX5_PORT_CPU_1; portno++) {
+ /* ASM CPU port: No preamble, IFH, enable padding */
+ spx5_wr(ASM_PORT_CFG_PAD_ENA_SET(1) |
+ ASM_PORT_CFG_NO_PREAMBLE_ENA_SET(1) |
+ ASM_PORT_CFG_INJ_FORMAT_CFG_SET(1), /* 1 = IFH */
+ sparx5, ASM_PORT_CFG(portno));
+
+ /* Reset WM cnt to unclog queued frames */
+ spx5_rmw(DSM_DEV_TX_STOP_WM_CFG_DEV_TX_CNT_CLR_SET(1),
+ DSM_DEV_TX_STOP_WM_CFG_DEV_TX_CNT_CLR,
+ sparx5,
+ DSM_DEV_TX_STOP_WM_CFG(portno));
+
+ /* Set Disassembler Stop Watermark level */
+ spx5_rmw(DSM_DEV_TX_STOP_WM_CFG_DEV_TX_STOP_WM_SET(0),
+ DSM_DEV_TX_STOP_WM_CFG_DEV_TX_STOP_WM,
+ sparx5,
+ DSM_DEV_TX_STOP_WM_CFG(portno));
+
+ /* Enable Disassembler buffer underrun watchdog
+ */
+ spx5_rmw(DSM_BUF_CFG_UNDERFLOW_WATCHDOG_DIS_SET(0),
+ DSM_BUF_CFG_UNDERFLOW_WATCHDOG_DIS,
+ sparx5,
+ DSM_BUF_CFG(portno));
+ }
+ return 0;
+}
+
+irqreturn_t sparx5_xtr_handler(int irq, void *_sparx5)
+{
+ struct sparx5 *s5 = _sparx5;
+ int poll = 64;
+
+ /* Check data in queue */
+ while (spx5_rd(s5, QS_XTR_DATA_PRESENT) & BIT(XTR_QUEUE) && poll-- > 0)
+ sparx5_xtr_grp(s5, XTR_QUEUE, false);
+
+ return IRQ_HANDLED;
+}
+
+void sparx5_port_inj_timer_setup(struct sparx5_port *port)
+{
+ hrtimer_init(&port->inj_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+ port->inj_timer.function = sparx5_injection_timeout;
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/* Microchip Sparx5 Switch driver
+ *
+ * Copyright (c) 2021 Microchip Technology Inc. and its subsidiaries.
+ */
+
+#include <linux/module.h>
+#include <linux/phylink.h>
+#include <linux/device.h>
+#include <linux/netdevice.h>
+#include <linux/sfp.h>
+
+#include "sparx5_main_regs.h"
+#include "sparx5_main.h"
+#include "sparx5_port.h"
+
+static bool port_conf_has_changed(struct sparx5_port_config *a, struct sparx5_port_config *b)
+{
+ if (a->speed != b->speed ||
+ a->portmode != b->portmode ||
+ a->autoneg != b->autoneg ||
+ a->pause_adv != b->pause_adv ||
+ a->power_down != b->power_down ||
+ a->media != b->media)
+ return true;
+ return false;
+}
+
+static void sparx5_phylink_validate(struct phylink_config *config,
+ unsigned long *supported,
+ struct phylink_link_state *state)
+{
+ struct sparx5_port *port = netdev_priv(to_net_dev(config->dev));
+ __ETHTOOL_DECLARE_LINK_MODE_MASK(mask) = { 0, };
+
+ phylink_set(mask, Autoneg);
+ phylink_set_port_modes(mask);
+ phylink_set(mask, Pause);
+ phylink_set(mask, Asym_Pause);
+
+ switch (state->interface) {
+ case PHY_INTERFACE_MODE_5GBASER:
+ case PHY_INTERFACE_MODE_10GBASER:
+ case PHY_INTERFACE_MODE_25GBASER:
+ case PHY_INTERFACE_MODE_NA:
+ if (port->conf.bandwidth == SPEED_5000)
+ phylink_set(mask, 5000baseT_Full);
+ if (port->conf.bandwidth == SPEED_10000) {
+ phylink_set(mask, 5000baseT_Full);
+ phylink_set(mask, 10000baseT_Full);
+ phylink_set(mask, 10000baseCR_Full);
+ phylink_set(mask, 10000baseSR_Full);
+ phylink_set(mask, 10000baseLR_Full);
+ phylink_set(mask, 10000baseLRM_Full);
+ phylink_set(mask, 10000baseER_Full);
+ }
+ if (port->conf.bandwidth == SPEED_25000) {
+ phylink_set(mask, 5000baseT_Full);
+ phylink_set(mask, 10000baseT_Full);
+ phylink_set(mask, 10000baseCR_Full);
+ phylink_set(mask, 10000baseSR_Full);
+ phylink_set(mask, 10000baseLR_Full);
+ phylink_set(mask, 10000baseLRM_Full);
+ phylink_set(mask, 10000baseER_Full);
+ phylink_set(mask, 25000baseCR_Full);
+ phylink_set(mask, 25000baseSR_Full);
+ }
+ if (state->interface != PHY_INTERFACE_MODE_NA)
+ break;
+ fallthrough;
+ case PHY_INTERFACE_MODE_SGMII:
+ case PHY_INTERFACE_MODE_QSGMII:
+ phylink_set(mask, 10baseT_Half);
+ phylink_set(mask, 10baseT_Full);
+ phylink_set(mask, 100baseT_Half);
+ phylink_set(mask, 100baseT_Full);
+ phylink_set(mask, 1000baseT_Full);
+ phylink_set(mask, 1000baseX_Full);
+ if (state->interface != PHY_INTERFACE_MODE_NA)
+ break;
+ fallthrough;
+ case PHY_INTERFACE_MODE_1000BASEX:
+ case PHY_INTERFACE_MODE_2500BASEX:
+ if (state->interface != PHY_INTERFACE_MODE_2500BASEX) {
+ phylink_set(mask, 1000baseT_Full);
+ phylink_set(mask, 1000baseX_Full);
+ }
+ if (state->interface == PHY_INTERFACE_MODE_2500BASEX ||
+ state->interface == PHY_INTERFACE_MODE_NA) {
+ phylink_set(mask, 2500baseT_Full);
+ phylink_set(mask, 2500baseX_Full);
+ }
+ break;
+ default:
+ bitmap_zero(supported, __ETHTOOL_LINK_MODE_MASK_NBITS);
+ return;
+ }
+ bitmap_and(supported, supported, mask, __ETHTOOL_LINK_MODE_MASK_NBITS);
+ bitmap_and(state->advertising, state->advertising, mask,
+ __ETHTOOL_LINK_MODE_MASK_NBITS);
+}
+
+static void sparx5_phylink_mac_config(struct phylink_config *config,
+ unsigned int mode,
+ const struct phylink_link_state *state)
+{
+ /* Currently not used */
+}
+
+static void sparx5_phylink_mac_link_up(struct phylink_config *config,
+ struct phy_device *phy,
+ unsigned int mode,
+ phy_interface_t interface,
+ int speed, int duplex,
+ bool tx_pause, bool rx_pause)
+{
+ struct sparx5_port *port = netdev_priv(to_net_dev(config->dev));
+ struct sparx5_port_config conf;
+ int err;
+
+ conf = port->conf;
+ conf.duplex = duplex;
+ conf.pause = 0;
+ conf.pause |= tx_pause ? MLO_PAUSE_TX : 0;
+ conf.pause |= rx_pause ? MLO_PAUSE_RX : 0;
+ conf.speed = speed;
+ /* Configure the port to speed/duplex/pause */
+ err = sparx5_port_config(port->sparx5, port, &conf);
+ if (err)
+ netdev_err(port->ndev, "port config failed: %d\n", err);
+}
+
+static void sparx5_phylink_mac_link_down(struct phylink_config *config,
+ unsigned int mode,
+ phy_interface_t interface)
+{
+ /* Currently not used */
+}
+
+static struct sparx5_port *sparx5_pcs_to_port(struct phylink_pcs *pcs)
+{
+ return container_of(pcs, struct sparx5_port, phylink_pcs);
+}
+
+static void sparx5_pcs_get_state(struct phylink_pcs *pcs,
+ struct phylink_link_state *state)
+{
+ struct sparx5_port *port = sparx5_pcs_to_port(pcs);
+ struct sparx5_port_status status;
+
+ sparx5_get_port_status(port->sparx5, port, &status);
+ state->link = status.link && !status.link_down;
+ state->an_complete = status.an_complete;
+ state->speed = status.speed;
+ state->duplex = status.duplex;
+ state->pause = status.pause;
+}
+
+static int sparx5_pcs_config(struct phylink_pcs *pcs,
+ unsigned int mode,
+ phy_interface_t interface,
+ const unsigned long *advertising,
+ bool permit_pause_to_mac)
+{
+ struct sparx5_port *port = sparx5_pcs_to_port(pcs);
+ struct sparx5_port_config conf;
+ int ret = 0;
+
+ conf = port->conf;
+ conf.power_down = false;
+ conf.portmode = interface;
+ conf.inband = phylink_autoneg_inband(mode);
+ conf.autoneg = phylink_test(advertising, Autoneg);
+ conf.pause_adv = 0;
+ if (phylink_test(advertising, Pause))
+ conf.pause_adv |= ADVERTISE_1000XPAUSE;
+ if (phylink_test(advertising, Asym_Pause))
+ conf.pause_adv |= ADVERTISE_1000XPSE_ASYM;
+ if (sparx5_is_baser(interface)) {
+ if (phylink_test(advertising, FIBRE))
+ conf.media = PHY_MEDIA_SR;
+ else
+ conf.media = PHY_MEDIA_DAC;
+ }
+ if (!port_conf_has_changed(&port->conf, &conf))
+ return ret;
+ /* Enable the PCS matching this interface type */
+ ret = sparx5_port_pcs_set(port->sparx5, port, &conf);
+ if (ret)
+ netdev_err(port->ndev, "port PCS config failed: %d\n", ret);
+ return ret;
+}
+
+static void sparx5_pcs_aneg_restart(struct phylink_pcs *pcs)
+{
+ /* Currently not used */
+}
+
+const struct phylink_pcs_ops sparx5_phylink_pcs_ops = {
+ .pcs_get_state = sparx5_pcs_get_state,
+ .pcs_config = sparx5_pcs_config,
+ .pcs_an_restart = sparx5_pcs_aneg_restart,
+};
+
+const struct phylink_mac_ops sparx5_phylink_mac_ops = {
+ .validate = sparx5_phylink_validate,
+ .mac_config = sparx5_phylink_mac_config,
+ .mac_link_down = sparx5_phylink_mac_link_down,
+ .mac_link_up = sparx5_phylink_mac_link_up,
+};
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/* Microchip Sparx5 Switch driver
+ *
+ * Copyright (c) 2021 Microchip Technology Inc. and its subsidiaries.
+ */
+
+#include <linux/module.h>
+#include <linux/phy/phy.h>
+
+#include "sparx5_main_regs.h"
+#include "sparx5_main.h"
+#include "sparx5_port.h"
+
+#define SPX5_ETYPE_TAG_C 0x8100
+#define SPX5_ETYPE_TAG_S 0x88a8
+
+#define SPX5_WAIT_US 1000
+#define SPX5_WAIT_MAX_US 2000
+
+enum port_error {
+ SPX5_PERR_SPEED,
+ SPX5_PERR_IFTYPE,
+};
+
+#define PAUSE_DISCARD 0xC
+#define ETH_MAXLEN (ETH_DATA_LEN + ETH_HLEN + ETH_FCS_LEN)
+
+static void decode_sgmii_word(u16 lp_abil, struct sparx5_port_status *status)
+{
+ status->an_complete = true;
+ if (!(lp_abil & LPA_SGMII_LINK)) {
+ status->link = false;
+ return;
+ }
+
+ switch (lp_abil & LPA_SGMII_SPD_MASK) {
+ case LPA_SGMII_10:
+ status->speed = SPEED_10;
+ break;
+ case LPA_SGMII_100:
+ status->speed = SPEED_100;
+ break;
+ case LPA_SGMII_1000:
+ status->speed = SPEED_1000;
+ break;
+ default:
+ status->link = false;
+ return;
+ }
+ if (lp_abil & LPA_SGMII_FULL_DUPLEX)
+ status->duplex = DUPLEX_FULL;
+ else
+ status->duplex = DUPLEX_HALF;
+}
+
+static void decode_cl37_word(u16 lp_abil, uint16_t ld_abil, struct sparx5_port_status *status)
+{
+ status->link = !(lp_abil & ADVERTISE_RFAULT) && status->link;
+ status->an_complete = true;
+ status->duplex = (ADVERTISE_1000XFULL & lp_abil) ?
+ DUPLEX_FULL : DUPLEX_UNKNOWN; // 1G HDX not supported
+
+ if ((ld_abil & ADVERTISE_1000XPAUSE) &&
+ (lp_abil & ADVERTISE_1000XPAUSE)) {
+ status->pause = MLO_PAUSE_RX | MLO_PAUSE_TX;
+ } else if ((ld_abil & ADVERTISE_1000XPSE_ASYM) &&
+ (lp_abil & ADVERTISE_1000XPSE_ASYM)) {
+ status->pause |= (lp_abil & ADVERTISE_1000XPAUSE) ?
+ MLO_PAUSE_TX : 0;
+ status->pause |= (ld_abil & ADVERTISE_1000XPAUSE) ?
+ MLO_PAUSE_RX : 0;
+ } else {
+ status->pause = MLO_PAUSE_NONE;
+ }
+}
+
+static int sparx5_get_dev2g5_status(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_status *status)
+{
+ u32 portno = port->portno;
+ u16 lp_adv, ld_adv;
+ u32 value;
+
+ /* Get PCS Link down sticky */
+ value = spx5_rd(sparx5, DEV2G5_PCS1G_STICKY(portno));
+ status->link_down = DEV2G5_PCS1G_STICKY_LINK_DOWN_STICKY_GET(value);
+ if (status->link_down) /* Clear the sticky */
+ spx5_wr(value, sparx5, DEV2G5_PCS1G_STICKY(portno));
+
+ /* Get both current Link and Sync status */
+ value = spx5_rd(sparx5, DEV2G5_PCS1G_LINK_STATUS(portno));
+ status->link = DEV2G5_PCS1G_LINK_STATUS_LINK_STATUS_GET(value) &&
+ DEV2G5_PCS1G_LINK_STATUS_SYNC_STATUS_GET(value);
+
+ if (port->conf.portmode == PHY_INTERFACE_MODE_1000BASEX)
+ status->speed = SPEED_1000;
+ else if (port->conf.portmode == PHY_INTERFACE_MODE_2500BASEX)
+ status->speed = SPEED_2500;
+
+ status->duplex = DUPLEX_FULL;
+
+ /* Get PCS ANEG status register */
+ value = spx5_rd(sparx5, DEV2G5_PCS1G_ANEG_STATUS(portno));
+
+ /* Aneg complete provides more information */
+ if (DEV2G5_PCS1G_ANEG_STATUS_ANEG_COMPLETE_GET(value)) {
+ lp_adv = DEV2G5_PCS1G_ANEG_STATUS_LP_ADV_ABILITY_GET(value);
+ if (port->conf.portmode == PHY_INTERFACE_MODE_SGMII) {
+ decode_sgmii_word(lp_adv, status);
+ } else {
+ value = spx5_rd(sparx5, DEV2G5_PCS1G_ANEG_CFG(portno));
+ ld_adv = DEV2G5_PCS1G_ANEG_CFG_ADV_ABILITY_GET(value);
+ decode_cl37_word(lp_adv, ld_adv, status);
+ }
+ }
+ return 0;
+}
+
+static int sparx5_get_sfi_status(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_status *status)
+{
+ bool high_speed_dev = sparx5_is_baser(port->conf.portmode);
+ u32 portno = port->portno;
+ u32 value, dev, tinst;
+ void __iomem *inst;
+
+ if (!high_speed_dev) {
+ netdev_err(port->ndev, "error: low speed and SFI mode\n");
+ return -EINVAL;
+ }
+
+ dev = sparx5_to_high_dev(portno);
+ tinst = sparx5_port_dev_index(portno);
+ inst = spx5_inst_get(sparx5, dev, tinst);
+
+ value = spx5_inst_rd(inst, DEV10G_MAC_TX_MONITOR_STICKY(0));
+ if (value != DEV10G_MAC_TX_MONITOR_STICKY_IDLE_STATE_STICKY) {
+ /* The link is or has been down. Clear the sticky bit */
+ status->link_down = 1;
+ spx5_inst_wr(0xffffffff, inst, DEV10G_MAC_TX_MONITOR_STICKY(0));
+ value = spx5_inst_rd(inst, DEV10G_MAC_TX_MONITOR_STICKY(0));
+ }
+ status->link = (value == DEV10G_MAC_TX_MONITOR_STICKY_IDLE_STATE_STICKY);
+ status->duplex = DUPLEX_FULL;
+ if (port->conf.portmode == PHY_INTERFACE_MODE_5GBASER)
+ status->speed = SPEED_5000;
+ else if (port->conf.portmode == PHY_INTERFACE_MODE_10GBASER)
+ status->speed = SPEED_10000;
+ else
+ status->speed = SPEED_25000;
+
+ return 0;
+}
+
+/* Get link status of 1000Base-X/in-band and SFI ports.
+ */
+int sparx5_get_port_status(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_status *status)
+{
+ memset(status, 0, sizeof(*status));
+ status->speed = port->conf.speed;
+ if (port->conf.power_down) {
+ status->link = false;
+ return 0;
+ }
+ switch (port->conf.portmode) {
+ case PHY_INTERFACE_MODE_SGMII:
+ case PHY_INTERFACE_MODE_QSGMII:
+ case PHY_INTERFACE_MODE_1000BASEX:
+ case PHY_INTERFACE_MODE_2500BASEX:
+ return sparx5_get_dev2g5_status(sparx5, port, status);
+ case PHY_INTERFACE_MODE_5GBASER:
+ case PHY_INTERFACE_MODE_10GBASER:
+ case PHY_INTERFACE_MODE_25GBASER:
+ return sparx5_get_sfi_status(sparx5, port, status);
+ case PHY_INTERFACE_MODE_NA:
+ return 0;
+ default:
+ netdev_err(port->ndev, "Status not supported");
+ return -ENODEV;
+ }
+ return 0;
+}
+
+static int sparx5_port_error(struct sparx5_port *port,
+ struct sparx5_port_config *conf,
+ enum port_error errtype)
+{
+ switch (errtype) {
+ case SPX5_PERR_SPEED:
+ netdev_err(port->ndev,
+ "Interface does not support speed: %u: for %s\n",
+ conf->speed, phy_modes(conf->portmode));
+ break;
+ case SPX5_PERR_IFTYPE:
+ netdev_err(port->ndev,
+ "Switch port does not support interface type: %s\n",
+ phy_modes(conf->portmode));
+ break;
+ default:
+ netdev_err(port->ndev,
+ "Interface configuration error\n");
+ }
+
+ return -EINVAL;
+}
+
+static int sparx5_port_verify_speed(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_config *conf)
+{
+ if ((sparx5_port_is_2g5(port->portno) &&
+ conf->speed > SPEED_2500) ||
+ (sparx5_port_is_5g(port->portno) &&
+ conf->speed > SPEED_5000) ||
+ (sparx5_port_is_10g(port->portno) &&
+ conf->speed > SPEED_10000))
+ return sparx5_port_error(port, conf, SPX5_PERR_SPEED);
+
+ switch (conf->portmode) {
+ case PHY_INTERFACE_MODE_NA:
+ return -EINVAL;
+ case PHY_INTERFACE_MODE_1000BASEX:
+ if (conf->speed != SPEED_1000 ||
+ sparx5_port_is_2g5(port->portno))
+ return sparx5_port_error(port, conf, SPX5_PERR_SPEED);
+ if (sparx5_port_is_2g5(port->portno))
+ return sparx5_port_error(port, conf, SPX5_PERR_IFTYPE);
+ break;
+ case PHY_INTERFACE_MODE_2500BASEX:
+ if (conf->speed != SPEED_2500 ||
+ sparx5_port_is_2g5(port->portno))
+ return sparx5_port_error(port, conf, SPX5_PERR_SPEED);
+ break;
+ case PHY_INTERFACE_MODE_QSGMII:
+ if (port->portno > 47)
+ return sparx5_port_error(port, conf, SPX5_PERR_IFTYPE);
+ fallthrough;
+ case PHY_INTERFACE_MODE_SGMII:
+ if (conf->speed != SPEED_1000 &&
+ conf->speed != SPEED_100 &&
+ conf->speed != SPEED_10 &&
+ conf->speed != SPEED_2500)
+ return sparx5_port_error(port, conf, SPX5_PERR_SPEED);
+ break;
+ case PHY_INTERFACE_MODE_5GBASER:
+ case PHY_INTERFACE_MODE_10GBASER:
+ case PHY_INTERFACE_MODE_25GBASER:
+ if ((conf->speed != SPEED_5000 &&
+ conf->speed != SPEED_10000 &&
+ conf->speed != SPEED_25000))
+ return sparx5_port_error(port, conf, SPX5_PERR_SPEED);
+ break;
+ default:
+ return sparx5_port_error(port, conf, SPX5_PERR_IFTYPE);
+ }
+ return 0;
+}
+
+static bool sparx5_dev_change(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_config *conf)
+{
+ return sparx5_is_baser(port->conf.portmode) ^
+ sparx5_is_baser(conf->portmode);
+}
+
+static int sparx5_port_flush_poll(struct sparx5 *sparx5, u32 portno)
+{
+ u32 value, resource, prio, delay_cnt = 0;
+ bool poll_src = true;
+ char *mem = "";
+
+ /* Resource == 0: Memory tracked per source (SRC-MEM)
+ * Resource == 1: Frame references tracked per source (SRC-REF)
+ * Resource == 2: Memory tracked per destination (DST-MEM)
+ * Resource == 3: Frame references tracked per destination. (DST-REF)
+ */
+ while (1) {
+ bool empty = true;
+
+ for (resource = 0; resource < (poll_src ? 2 : 1); resource++) {
+ u32 base;
+
+ base = (resource == 0 ? 2048 : 0) + SPX5_PRIOS * portno;
+ for (prio = 0; prio < SPX5_PRIOS; prio++) {
+ value = spx5_rd(sparx5,
+ QRES_RES_STAT(base + prio));
+ if (value) {
+ mem = resource == 0 ?
+ "DST-MEM" : "SRC-MEM";
+ empty = false;
+ }
+ }
+ }
+
+ if (empty)
+ break;
+
+ if (delay_cnt++ == 2000) {
+ dev_err(sparx5->dev,
+ "Flush timeout port %u. %s queue not empty\n",
+ portno, mem);
+ return -EINVAL;
+ }
+
+ usleep_range(SPX5_WAIT_US, SPX5_WAIT_MAX_US);
+ }
+ return 0;
+}
+
+static int sparx5_port_disable(struct sparx5 *sparx5, struct sparx5_port *port, bool high_spd_dev)
+{
+ u32 tinst = high_spd_dev ?
+ sparx5_port_dev_index(port->portno) : port->portno;
+ u32 dev = high_spd_dev ?
+ sparx5_to_high_dev(port->portno) : TARGET_DEV2G5;
+ void __iomem *devinst = spx5_inst_get(sparx5, dev, tinst);
+ u32 spd = port->conf.speed;
+ u32 spd_prm;
+ int err;
+
+ if (high_spd_dev) {
+ /* 1: Reset the PCS Rx clock domain */
+ spx5_inst_rmw(DEV10G_DEV_RST_CTRL_PCS_RX_RST,
+ DEV10G_DEV_RST_CTRL_PCS_RX_RST,
+ devinst,
+ DEV10G_DEV_RST_CTRL(0));
+
+ /* 2: Disable MAC frame reception */
+ spx5_inst_rmw(0,
+ DEV10G_MAC_ENA_CFG_RX_ENA,
+ devinst,
+ DEV10G_MAC_ENA_CFG(0));
+ } else {
+ /* 1: Reset the PCS Rx clock domain */
+ spx5_inst_rmw(DEV2G5_DEV_RST_CTRL_PCS_RX_RST,
+ DEV2G5_DEV_RST_CTRL_PCS_RX_RST,
+ devinst,
+ DEV2G5_DEV_RST_CTRL(0));
+ /* 2: Disable MAC frame reception */
+ spx5_inst_rmw(0,
+ DEV2G5_MAC_ENA_CFG_RX_ENA,
+ devinst,
+ DEV2G5_MAC_ENA_CFG(0));
+ }
+ /* 3: Disable traffic being sent to or from switch port->portno */
+ spx5_rmw(0,
+ QFWD_SWITCH_PORT_MODE_PORT_ENA,
+ sparx5,
+ QFWD_SWITCH_PORT_MODE(port->portno));
+
+ /* 4: Disable dequeuing from the egress queues */
+ spx5_rmw(HSCH_PORT_MODE_DEQUEUE_DIS,
+ HSCH_PORT_MODE_DEQUEUE_DIS,
+ sparx5,
+ HSCH_PORT_MODE(port->portno));
+
+ /* 5: Disable Flowcontrol */
+ spx5_rmw(QSYS_PAUSE_CFG_PAUSE_STOP_SET(0xFFF - 1),
+ QSYS_PAUSE_CFG_PAUSE_STOP,
+ sparx5,
+ QSYS_PAUSE_CFG(port->portno));
+
+ spd_prm = spd == SPEED_10 ? 1000 : spd == SPEED_100 ? 100 : 10;
+ /* 6: Wait while the last frame is exiting the queues */
+ usleep_range(8 * spd_prm, 10 * spd_prm);
+
+ /* 7: Flush the queues accociated with the port->portno */
+ spx5_rmw(HSCH_FLUSH_CTRL_FLUSH_PORT_SET(port->portno) |
+ HSCH_FLUSH_CTRL_FLUSH_DST_SET(1) |
+ HSCH_FLUSH_CTRL_FLUSH_SRC_SET(1) |
+ HSCH_FLUSH_CTRL_FLUSH_ENA_SET(1),
+ HSCH_FLUSH_CTRL_FLUSH_PORT |
+ HSCH_FLUSH_CTRL_FLUSH_DST |
+ HSCH_FLUSH_CTRL_FLUSH_SRC |
+ HSCH_FLUSH_CTRL_FLUSH_ENA,
+ sparx5,
+ HSCH_FLUSH_CTRL);
+
+ /* 8: Enable dequeuing from the egress queues */
+ spx5_rmw(0,
+ HSCH_PORT_MODE_DEQUEUE_DIS,
+ sparx5,
+ HSCH_PORT_MODE(port->portno));
+
+ /* 9: Wait until flushing is complete */
+ err = sparx5_port_flush_poll(sparx5, port->portno);
+ if (err)
+ return err;
+
+ /* 10: Reset the MAC clock domain */
+ if (high_spd_dev) {
+ spx5_inst_rmw(DEV10G_DEV_RST_CTRL_PCS_TX_RST_SET(1) |
+ DEV10G_DEV_RST_CTRL_MAC_RX_RST_SET(1) |
+ DEV10G_DEV_RST_CTRL_MAC_TX_RST_SET(1),
+ DEV10G_DEV_RST_CTRL_PCS_TX_RST |
+ DEV10G_DEV_RST_CTRL_MAC_RX_RST |
+ DEV10G_DEV_RST_CTRL_MAC_TX_RST,
+ devinst,
+ DEV10G_DEV_RST_CTRL(0));
+
+ } else {
+ spx5_inst_rmw(DEV2G5_DEV_RST_CTRL_SPEED_SEL_SET(3) |
+ DEV2G5_DEV_RST_CTRL_PCS_TX_RST_SET(1) |
+ DEV2G5_DEV_RST_CTRL_PCS_RX_RST_SET(1) |
+ DEV2G5_DEV_RST_CTRL_MAC_TX_RST_SET(1) |
+ DEV2G5_DEV_RST_CTRL_MAC_RX_RST_SET(1),
+ DEV2G5_DEV_RST_CTRL_SPEED_SEL |
+ DEV2G5_DEV_RST_CTRL_PCS_TX_RST |
+ DEV2G5_DEV_RST_CTRL_PCS_RX_RST |
+ DEV2G5_DEV_RST_CTRL_MAC_TX_RST |
+ DEV2G5_DEV_RST_CTRL_MAC_RX_RST,
+ devinst,
+ DEV2G5_DEV_RST_CTRL(0));
+ }
+ /* 11: Clear flushing */
+ spx5_rmw(HSCH_FLUSH_CTRL_FLUSH_PORT_SET(port->portno) |
+ HSCH_FLUSH_CTRL_FLUSH_ENA_SET(0),
+ HSCH_FLUSH_CTRL_FLUSH_PORT |
+ HSCH_FLUSH_CTRL_FLUSH_ENA,
+ sparx5,
+ HSCH_FLUSH_CTRL);
+
+ if (high_spd_dev) {
+ u32 pcs = sparx5_to_pcs_dev(port->portno);
+ void __iomem *pcsinst = spx5_inst_get(sparx5, pcs, tinst);
+
+ /* 12: Disable 5G/10G/25 BaseR PCS */
+ spx5_inst_rmw(PCS10G_BR_PCS_CFG_PCS_ENA_SET(0),
+ PCS10G_BR_PCS_CFG_PCS_ENA,
+ pcsinst,
+ PCS10G_BR_PCS_CFG(0));
+
+ if (sparx5_port_is_25g(port->portno))
+ /* Disable 25G PCS */
+ spx5_rmw(DEV25G_PCS25G_CFG_PCS25G_ENA_SET(0),
+ DEV25G_PCS25G_CFG_PCS25G_ENA,
+ sparx5,
+ DEV25G_PCS25G_CFG(tinst));
+ } else {
+ /* 12: Disable 1G PCS */
+ spx5_rmw(DEV2G5_PCS1G_CFG_PCS_ENA_SET(0),
+ DEV2G5_PCS1G_CFG_PCS_ENA,
+ sparx5,
+ DEV2G5_PCS1G_CFG(port->portno));
+ }
+
+ /* The port is now flushed and disabled */
+ return 0;
+}
+
+static int sparx5_port_fifo_sz(struct sparx5 *sparx5,
+ u32 portno, u32 speed)
+{
+ u32 sys_clk = sparx5_clk_period(sparx5->coreclock);
+ const u32 taxi_dist[SPX5_PORTS_ALL] = {
+ 6, 8, 10, 6, 8, 10, 6, 8, 10, 6, 8, 10,
+ 4, 4, 4, 4,
+ 11, 12, 13, 14, 15, 16, 17, 18,
+ 11, 12, 13, 14, 15, 16, 17, 18,
+ 11, 12, 13, 14, 15, 16, 17, 18,
+ 11, 12, 13, 14, 15, 16, 17, 18,
+ 4, 6, 8, 4, 6, 8, 6, 8,
+ 2, 2, 2, 2, 2, 2, 2, 4, 2
+ };
+ u32 mac_per = 6400, tmp1, tmp2, tmp3;
+ u32 fifo_width = 16;
+ u32 mac_width = 8;
+ u32 addition = 0;
+
+ switch (speed) {
+ case SPEED_25000:
+ return 0;
+ case SPEED_10000:
+ mac_per = 6400;
+ mac_width = 8;
+ addition = 1;
+ break;
+ case SPEED_5000:
+ mac_per = 12800;
+ mac_width = 8;
+ addition = 0;
+ break;
+ case SPEED_2500:
+ mac_per = 3200;
+ mac_width = 1;
+ addition = 0;
+ break;
+ case SPEED_1000:
+ mac_per = 8000;
+ mac_width = 1;
+ addition = 0;
+ break;
+ case SPEED_100:
+ case SPEED_10:
+ return 1;
+ default:
+ break;
+ }
+
+ tmp1 = 1000 * mac_width / fifo_width;
+ tmp2 = 3000 + ((12000 + 2 * taxi_dist[portno] * 1000)
+ * sys_clk / mac_per);
+ tmp3 = tmp1 * tmp2 / 1000;
+ return (tmp3 + 2000 + 999) / 1000 + addition;
+}
+
+/* Configure port muxing:
+ * QSGMII: 4x2G5 devices
+ */
+static int sparx5_port_mux_set(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_config *conf)
+{
+ u32 portno = port->portno;
+ u32 inst;
+
+ if (port->conf.portmode == conf->portmode)
+ return 0; /* Nothing to do */
+
+ switch (conf->portmode) {
+ case PHY_INTERFACE_MODE_QSGMII: /* QSGMII: 4x2G5 devices. Mode Q' */
+ inst = (portno - portno % 4) / 4;
+ spx5_rmw(BIT(inst),
+ BIT(inst),
+ sparx5,
+ PORT_CONF_QSGMII_ENA);
+
+ if ((portno / 4 % 2) == 0) {
+ /* Affects d0-d3,d8-d11..d40-d43 */
+ spx5_rmw(PORT_CONF_USGMII_CFG_BYPASS_SCRAM_SET(1) |
+ PORT_CONF_USGMII_CFG_BYPASS_DESCRAM_SET(1) |
+ PORT_CONF_USGMII_CFG_QUAD_MODE_SET(1),
+ PORT_CONF_USGMII_CFG_BYPASS_SCRAM |
+ PORT_CONF_USGMII_CFG_BYPASS_DESCRAM |
+ PORT_CONF_USGMII_CFG_QUAD_MODE,
+ sparx5,
+ PORT_CONF_USGMII_CFG((portno / 8)));
+ }
+ break;
+ default:
+ break;
+ }
+ return 0;
+}
+
+static int sparx5_port_max_tags_set(struct sparx5 *sparx5,
+ struct sparx5_port *port)
+{
+ enum sparx5_port_max_tags max_tags = port->max_vlan_tags;
+ int tag_ct = max_tags == SPX5_PORT_MAX_TAGS_ONE ? 1 :
+ max_tags == SPX5_PORT_MAX_TAGS_TWO ? 2 : 0;
+ bool dtag = max_tags == SPX5_PORT_MAX_TAGS_TWO;
+ enum sparx5_vlan_port_type vlan_type = port->vlan_type;
+ bool dotag = max_tags != SPX5_PORT_MAX_TAGS_NONE;
+ u32 dev = sparx5_to_high_dev(port->portno);
+ u32 tinst = sparx5_port_dev_index(port->portno);
+ void __iomem *inst = spx5_inst_get(sparx5, dev, tinst);
+ u32 etype;
+
+ etype = (vlan_type == SPX5_VLAN_PORT_TYPE_S_CUSTOM ?
+ port->custom_etype :
+ vlan_type == SPX5_VLAN_PORT_TYPE_C ?
+ SPX5_ETYPE_TAG_C : SPX5_ETYPE_TAG_S);
+
+ spx5_wr(DEV2G5_MAC_TAGS_CFG_TAG_ID_SET(etype) |
+ DEV2G5_MAC_TAGS_CFG_PB_ENA_SET(dtag) |
+ DEV2G5_MAC_TAGS_CFG_VLAN_AWR_ENA_SET(dotag) |
+ DEV2G5_MAC_TAGS_CFG_VLAN_LEN_AWR_ENA_SET(dotag),
+ sparx5,
+ DEV2G5_MAC_TAGS_CFG(port->portno));
+
+ if (sparx5_port_is_2g5(port->portno))
+ return 0;
+
+ spx5_inst_rmw(DEV10G_MAC_TAGS_CFG_TAG_ID_SET(etype) |
+ DEV10G_MAC_TAGS_CFG_TAG_ENA_SET(dotag),
+ DEV10G_MAC_TAGS_CFG_TAG_ID |
+ DEV10G_MAC_TAGS_CFG_TAG_ENA,
+ inst,
+ DEV10G_MAC_TAGS_CFG(0, 0));
+
+ spx5_inst_rmw(DEV10G_MAC_NUM_TAGS_CFG_NUM_TAGS_SET(tag_ct),
+ DEV10G_MAC_NUM_TAGS_CFG_NUM_TAGS,
+ inst,
+ DEV10G_MAC_NUM_TAGS_CFG(0));
+
+ spx5_inst_rmw(DEV10G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK_SET(dotag),
+ DEV10G_MAC_MAXLEN_CFG_MAX_LEN_TAG_CHK,
+ inst,
+ DEV10G_MAC_MAXLEN_CFG(0));
+ return 0;
+}
+
+static int sparx5_port_fwd_urg(struct sparx5 *sparx5, u32 speed)
+{
+ u32 clk_period_ps = 1600; /* 625Mhz for now */
+ u32 urg = 672000;
+
+ switch (speed) {
+ case SPEED_10:
+ case SPEED_100:
+ case SPEED_1000:
+ urg = 672000;
+ break;
+ case SPEED_2500:
+ urg = 270000;
+ break;
+ case SPEED_5000:
+ urg = 135000;
+ break;
+ case SPEED_10000:
+ urg = 67200;
+ break;
+ case SPEED_25000:
+ urg = 27000;
+ break;
+ }
+ return urg / clk_period_ps - 1;
+}
+
+static u16 sparx5_wm_enc(u16 value)
+{
+ if (value >= 2048)
+ return 2048 + value / 16;
+
+ return value;
+}
+
+static int sparx5_port_fc_setup(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_config *conf)
+{
+ bool fc_obey = conf->pause & MLO_PAUSE_RX ? 1 : 0;
+ u32 pause_stop = 0xFFF - 1; /* FC gen disabled */
+
+ if (conf->pause & MLO_PAUSE_TX)
+ pause_stop = sparx5_wm_enc(4 * (ETH_MAXLEN /
+ SPX5_BUFFER_CELL_SZ));
+
+ /* Set HDX flowcontrol */
+ spx5_rmw(DSM_MAC_CFG_HDX_BACKPREASSURE_SET(conf->duplex == DUPLEX_HALF),
+ DSM_MAC_CFG_HDX_BACKPREASSURE,
+ sparx5,
+ DSM_MAC_CFG(port->portno));
+
+ /* Obey flowcontrol */
+ spx5_rmw(DSM_RX_PAUSE_CFG_RX_PAUSE_EN_SET(fc_obey),
+ DSM_RX_PAUSE_CFG_RX_PAUSE_EN,
+ sparx5,
+ DSM_RX_PAUSE_CFG(port->portno));
+
+ /* Disable forward pressure */
+ spx5_rmw(QSYS_FWD_PRESSURE_FWD_PRESSURE_DIS_SET(fc_obey),
+ QSYS_FWD_PRESSURE_FWD_PRESSURE_DIS,
+ sparx5,
+ QSYS_FWD_PRESSURE(port->portno));
+
+ /* Generate pause frames */
+ spx5_rmw(QSYS_PAUSE_CFG_PAUSE_STOP_SET(pause_stop),
+ QSYS_PAUSE_CFG_PAUSE_STOP,
+ sparx5,
+ QSYS_PAUSE_CFG(port->portno));
+
+ return 0;
+}
+
+static u16 sparx5_get_aneg_word(struct sparx5_port_config *conf)
+{
+ if (conf->portmode == PHY_INTERFACE_MODE_1000BASEX) /* cl-37 aneg */
+ return (conf->pause_adv | ADVERTISE_LPACK | ADVERTISE_1000XFULL);
+ else
+ return 1; /* Enable SGMII Aneg */
+}
+
+int sparx5_serdes_set(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_config *conf)
+{
+ int portmode, err, speed = conf->speed;
+
+ if (conf->portmode == PHY_INTERFACE_MODE_QSGMII &&
+ ((port->portno % 4) != 0)) {
+ return 0;
+ }
+ if (sparx5_is_baser(conf->portmode)) {
+ if (conf->portmode == PHY_INTERFACE_MODE_25GBASER)
+ speed = SPEED_25000;
+ else if (conf->portmode == PHY_INTERFACE_MODE_10GBASER)
+ speed = SPEED_10000;
+ else
+ speed = SPEED_5000;
+ }
+
+ err = phy_set_media(port->serdes, conf->media);
+ if (err)
+ return err;
+ if (speed > 0) {
+ err = phy_set_speed(port->serdes, speed);
+ if (err)
+ return err;
+ }
+ if (conf->serdes_reset) {
+ err = phy_reset(port->serdes);
+ if (err)
+ return err;
+ }
+
+ /* Configure SerDes with port parameters
+ * For BaseR, the serdes driver supports 10GGBASE-R and speed 5G/10G/25G
+ */
+ portmode = conf->portmode;
+ if (sparx5_is_baser(conf->portmode))
+ portmode = PHY_INTERFACE_MODE_10GBASER;
+ err = phy_set_mode_ext(port->serdes, PHY_MODE_ETHERNET, portmode);
+ if (err)
+ return err;
+ conf->serdes_reset = false;
+ return err;
+}
+
+static int sparx5_port_pcs_low_set(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_config *conf)
+{
+ bool sgmii = false, inband_aneg = false;
+ int err;
+
+ if (port->conf.inband) {
+ if (conf->portmode == PHY_INTERFACE_MODE_SGMII ||
+ conf->portmode == PHY_INTERFACE_MODE_QSGMII)
+ inband_aneg = true; /* Cisco-SGMII in-band-aneg */
+ else if (conf->portmode == PHY_INTERFACE_MODE_1000BASEX &&
+ conf->autoneg)
+ inband_aneg = true; /* Clause-37 in-band-aneg */
+
+ err = sparx5_serdes_set(sparx5, port, conf);
+ if (err)
+ return -EINVAL;
+ } else {
+ sgmii = true; /* Phy is connnected to the MAC */
+ }
+
+ /* Choose SGMII or 1000BaseX/2500BaseX PCS mode */
+ spx5_rmw(DEV2G5_PCS1G_MODE_CFG_SGMII_MODE_ENA_SET(sgmii),
+ DEV2G5_PCS1G_MODE_CFG_SGMII_MODE_ENA,
+ sparx5,
+ DEV2G5_PCS1G_MODE_CFG(port->portno));
+
+ /* Enable PCS */
+ spx5_wr(DEV2G5_PCS1G_CFG_PCS_ENA_SET(1),
+ sparx5,
+ DEV2G5_PCS1G_CFG(port->portno));
+
+ if (inband_aneg) {
+ u16 abil = sparx5_get_aneg_word(conf);
+
+ /* Enable in-band aneg */
+ spx5_wr(DEV2G5_PCS1G_ANEG_CFG_ADV_ABILITY_SET(abil) |
+ DEV2G5_PCS1G_ANEG_CFG_SW_RESOLVE_ENA_SET(1) |
+ DEV2G5_PCS1G_ANEG_CFG_ANEG_ENA_SET(1) |
+ DEV2G5_PCS1G_ANEG_CFG_ANEG_RESTART_ONE_SHOT_SET(1),
+ sparx5,
+ DEV2G5_PCS1G_ANEG_CFG(port->portno));
+ } else {
+ spx5_wr(0, sparx5, DEV2G5_PCS1G_ANEG_CFG(port->portno));
+ }
+
+ /* Take PCS out of reset */
+ spx5_rmw(DEV2G5_DEV_RST_CTRL_SPEED_SEL_SET(2) |
+ DEV2G5_DEV_RST_CTRL_PCS_TX_RST_SET(0) |
+ DEV2G5_DEV_RST_CTRL_PCS_RX_RST_SET(0),
+ DEV2G5_DEV_RST_CTRL_SPEED_SEL |
+ DEV2G5_DEV_RST_CTRL_PCS_TX_RST |
+ DEV2G5_DEV_RST_CTRL_PCS_RX_RST,
+ sparx5,
+ DEV2G5_DEV_RST_CTRL(port->portno));
+
+ return 0;
+}
+
+static int sparx5_port_pcs_high_set(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_config *conf)
+{
+ u32 clk_spd = conf->portmode == PHY_INTERFACE_MODE_5GBASER ? 1 : 0;
+ u32 pix = sparx5_port_dev_index(port->portno);
+ u32 dev = sparx5_to_high_dev(port->portno);
+ u32 pcs = sparx5_to_pcs_dev(port->portno);
+ void __iomem *devinst;
+ void __iomem *pcsinst;
+ int err;
+
+ devinst = spx5_inst_get(sparx5, dev, pix);
+ pcsinst = spx5_inst_get(sparx5, pcs, pix);
+
+ /* SFI : No in-band-aneg. Speeds 5G/10G/25G */
+ err = sparx5_serdes_set(sparx5, port, conf);
+ if (err)
+ return -EINVAL;
+ if (conf->portmode == PHY_INTERFACE_MODE_25GBASER) {
+ /* Enable PCS for 25G device, speed 25G */
+ spx5_rmw(DEV25G_PCS25G_CFG_PCS25G_ENA_SET(1),
+ DEV25G_PCS25G_CFG_PCS25G_ENA,
+ sparx5,
+ DEV25G_PCS25G_CFG(pix));
+ } else {
+ /* Enable PCS for 5G/10G/25G devices, speed 5G/10G */
+ spx5_inst_rmw(PCS10G_BR_PCS_CFG_PCS_ENA_SET(1),
+ PCS10G_BR_PCS_CFG_PCS_ENA,
+ pcsinst,
+ PCS10G_BR_PCS_CFG(0));
+ }
+
+ /* Enable 5G/10G/25G MAC module */
+ spx5_inst_wr(DEV10G_MAC_ENA_CFG_RX_ENA_SET(1) |
+ DEV10G_MAC_ENA_CFG_TX_ENA_SET(1),
+ devinst,
+ DEV10G_MAC_ENA_CFG(0));
+
+ /* Take the device out of reset */
+ spx5_inst_rmw(DEV10G_DEV_RST_CTRL_PCS_RX_RST_SET(0) |
+ DEV10G_DEV_RST_CTRL_PCS_TX_RST_SET(0) |
+ DEV10G_DEV_RST_CTRL_MAC_RX_RST_SET(0) |
+ DEV10G_DEV_RST_CTRL_MAC_TX_RST_SET(0) |
+ DEV10G_DEV_RST_CTRL_SPEED_SEL_SET(clk_spd),
+ DEV10G_DEV_RST_CTRL_PCS_RX_RST |
+ DEV10G_DEV_RST_CTRL_PCS_TX_RST |
+ DEV10G_DEV_RST_CTRL_MAC_RX_RST |
+ DEV10G_DEV_RST_CTRL_MAC_TX_RST |
+ DEV10G_DEV_RST_CTRL_SPEED_SEL,
+ devinst,
+ DEV10G_DEV_RST_CTRL(0));
+
+ return 0;
+}
+
+/* Switch between 1G/2500 and 5G/10G/25G devices */
+static void sparx5_dev_switch(struct sparx5 *sparx5, int port, bool hsd)
+{
+ int bt_indx = BIT(sparx5_port_dev_index(port));
+
+ if (sparx5_port_is_5g(port)) {
+ spx5_rmw(hsd ? 0 : bt_indx,
+ bt_indx,
+ sparx5,
+ PORT_CONF_DEV5G_MODES);
+ } else if (sparx5_port_is_10g(port)) {
+ spx5_rmw(hsd ? 0 : bt_indx,
+ bt_indx,
+ sparx5,
+ PORT_CONF_DEV10G_MODES);
+ } else if (sparx5_port_is_25g(port)) {
+ spx5_rmw(hsd ? 0 : bt_indx,
+ bt_indx,
+ sparx5,
+ PORT_CONF_DEV25G_MODES);
+ }
+}
+
+/* Configure speed/duplex dependent registers */
+static int sparx5_port_config_low_set(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_config *conf)
+{
+ u32 clk_spd, gig_mode, tx_gap, hdx_gap_1, hdx_gap_2;
+ bool fdx = conf->duplex == DUPLEX_FULL;
+ int spd = conf->speed;
+
+ clk_spd = spd == SPEED_10 ? 0 : spd == SPEED_100 ? 1 : 2;
+ gig_mode = spd == SPEED_1000 || spd == SPEED_2500;
+ tx_gap = spd == SPEED_1000 ? 4 : fdx ? 6 : 5;
+ hdx_gap_1 = spd == SPEED_1000 ? 0 : spd == SPEED_100 ? 1 : 2;
+ hdx_gap_2 = spd == SPEED_1000 ? 0 : spd == SPEED_100 ? 4 : 1;
+
+ /* GIG/FDX mode */
+ spx5_rmw(DEV2G5_MAC_MODE_CFG_GIGA_MODE_ENA_SET(gig_mode) |
+ DEV2G5_MAC_MODE_CFG_FDX_ENA_SET(fdx),
+ DEV2G5_MAC_MODE_CFG_GIGA_MODE_ENA |
+ DEV2G5_MAC_MODE_CFG_FDX_ENA,
+ sparx5,
+ DEV2G5_MAC_MODE_CFG(port->portno));
+
+ /* Set MAC IFG Gaps */
+ spx5_wr(DEV2G5_MAC_IFG_CFG_TX_IFG_SET(tx_gap) |
+ DEV2G5_MAC_IFG_CFG_RX_IFG1_SET(hdx_gap_1) |
+ DEV2G5_MAC_IFG_CFG_RX_IFG2_SET(hdx_gap_2),
+ sparx5,
+ DEV2G5_MAC_IFG_CFG(port->portno));
+
+ /* Disabling frame aging when in HDX (due to HDX issue) */
+ spx5_rmw(HSCH_PORT_MODE_AGE_DIS_SET(fdx == 0),
+ HSCH_PORT_MODE_AGE_DIS,
+ sparx5,
+ HSCH_PORT_MODE(port->portno));
+
+ /* Enable MAC module */
+ spx5_wr(DEV2G5_MAC_ENA_CFG_RX_ENA |
+ DEV2G5_MAC_ENA_CFG_TX_ENA,
+ sparx5,
+ DEV2G5_MAC_ENA_CFG(port->portno));
+
+ /* Select speed and take MAC out of reset */
+ spx5_rmw(DEV2G5_DEV_RST_CTRL_SPEED_SEL_SET(clk_spd) |
+ DEV2G5_DEV_RST_CTRL_MAC_TX_RST_SET(0) |
+ DEV2G5_DEV_RST_CTRL_MAC_RX_RST_SET(0),
+ DEV2G5_DEV_RST_CTRL_SPEED_SEL |
+ DEV2G5_DEV_RST_CTRL_MAC_TX_RST |
+ DEV2G5_DEV_RST_CTRL_MAC_RX_RST,
+ sparx5,
+ DEV2G5_DEV_RST_CTRL(port->portno));
+
+ return 0;
+}
+
+int sparx5_port_pcs_set(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_config *conf)
+
+{
+ bool high_speed_dev = sparx5_is_baser(conf->portmode);
+ int err;
+
+ if (sparx5_dev_change(sparx5, port, conf)) {
+ /* switch device */
+ sparx5_dev_switch(sparx5, port->portno, high_speed_dev);
+
+ /* Disable the not-in-use device */
+ err = sparx5_port_disable(sparx5, port, !high_speed_dev);
+ if (err)
+ return err;
+ }
+ /* Disable the port before re-configuring */
+ err = sparx5_port_disable(sparx5, port, high_speed_dev);
+ if (err)
+ return -EINVAL;
+
+ if (high_speed_dev)
+ err = sparx5_port_pcs_high_set(sparx5, port, conf);
+ else
+ err = sparx5_port_pcs_low_set(sparx5, port, conf);
+
+ if (err)
+ return -EINVAL;
+
+ if (port->conf.inband) {
+ /* Enable/disable 1G counters in ASM */
+ spx5_rmw(ASM_PORT_CFG_CSC_STAT_DIS_SET(high_speed_dev),
+ ASM_PORT_CFG_CSC_STAT_DIS,
+ sparx5,
+ ASM_PORT_CFG(port->portno));
+
+ /* Enable/disable 1G counters in DSM */
+ spx5_rmw(DSM_BUF_CFG_CSC_STAT_DIS_SET(high_speed_dev),
+ DSM_BUF_CFG_CSC_STAT_DIS,
+ sparx5,
+ DSM_BUF_CFG(port->portno));
+ }
+
+ port->conf = *conf;
+
+ return 0;
+}
+
+int sparx5_port_config(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_config *conf)
+{
+ bool high_speed_dev = sparx5_is_baser(conf->portmode);
+ int err, urgency, stop_wm;
+
+ err = sparx5_port_verify_speed(sparx5, port, conf);
+ if (err)
+ return err;
+
+ /* high speed device is already configured */
+ if (!high_speed_dev)
+ sparx5_port_config_low_set(sparx5, port, conf);
+
+ /* Configure flow control */
+ err = sparx5_port_fc_setup(sparx5, port, conf);
+ if (err)
+ return err;
+
+ /* Set the DSM stop watermark */
+ stop_wm = sparx5_port_fifo_sz(sparx5, port->portno, conf->speed);
+ spx5_rmw(DSM_DEV_TX_STOP_WM_CFG_DEV_TX_STOP_WM_SET(stop_wm),
+ DSM_DEV_TX_STOP_WM_CFG_DEV_TX_STOP_WM,
+ sparx5,
+ DSM_DEV_TX_STOP_WM_CFG(port->portno));
+
+ /* Enable port in queue system */
+ urgency = sparx5_port_fwd_urg(sparx5, conf->speed);
+ spx5_rmw(QFWD_SWITCH_PORT_MODE_PORT_ENA_SET(1) |
+ QFWD_SWITCH_PORT_MODE_FWD_URGENCY_SET(urgency),
+ QFWD_SWITCH_PORT_MODE_PORT_ENA |
+ QFWD_SWITCH_PORT_MODE_FWD_URGENCY,
+ sparx5,
+ QFWD_SWITCH_PORT_MODE(port->portno));
+
+ /* Save the new values */
+ port->conf = *conf;
+
+ return 0;
+}
+
+/* Initialize port config to default */
+int sparx5_port_init(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_config *conf)
+{
+ u32 pause_start = sparx5_wm_enc(6 * (ETH_MAXLEN / SPX5_BUFFER_CELL_SZ));
+ u32 atop = sparx5_wm_enc(20 * (ETH_MAXLEN / SPX5_BUFFER_CELL_SZ));
+ u32 devhigh = sparx5_to_high_dev(port->portno);
+ u32 pix = sparx5_port_dev_index(port->portno);
+ u32 pcs = sparx5_to_pcs_dev(port->portno);
+ bool sd_pol = port->signd_active_high;
+ bool sd_sel = !port->signd_internal;
+ bool sd_ena = port->signd_enable;
+ u32 pause_stop = 0xFFF - 1; /* FC generate disabled */
+ void __iomem *devinst;
+ void __iomem *pcsinst;
+ int err;
+
+ devinst = spx5_inst_get(sparx5, devhigh, pix);
+ pcsinst = spx5_inst_get(sparx5, pcs, pix);
+
+ /* Set the mux port mode */
+ err = sparx5_port_mux_set(sparx5, port, conf);
+ if (err)
+ return err;
+
+ /* Configure MAC vlan awareness */
+ err = sparx5_port_max_tags_set(sparx5, port);
+ if (err)
+ return err;
+
+ /* Set Max Length */
+ spx5_rmw(DEV2G5_MAC_MAXLEN_CFG_MAX_LEN_SET(ETH_MAXLEN),
+ DEV2G5_MAC_MAXLEN_CFG_MAX_LEN,
+ sparx5,
+ DEV2G5_MAC_MAXLEN_CFG(port->portno));
+
+ /* 1G/2G5: Signal Detect configuration */
+ spx5_wr(DEV2G5_PCS1G_SD_CFG_SD_POL_SET(sd_pol) |
+ DEV2G5_PCS1G_SD_CFG_SD_SEL_SET(sd_sel) |
+ DEV2G5_PCS1G_SD_CFG_SD_ENA_SET(sd_ena),
+ sparx5,
+ DEV2G5_PCS1G_SD_CFG(port->portno));
+
+ /* Set Pause WM hysteresis */
+ spx5_rmw(QSYS_PAUSE_CFG_PAUSE_START_SET(pause_start) |
+ QSYS_PAUSE_CFG_PAUSE_STOP_SET(pause_stop) |
+ QSYS_PAUSE_CFG_PAUSE_ENA_SET(1),
+ QSYS_PAUSE_CFG_PAUSE_START |
+ QSYS_PAUSE_CFG_PAUSE_STOP |
+ QSYS_PAUSE_CFG_PAUSE_ENA,
+ sparx5,
+ QSYS_PAUSE_CFG(port->portno));
+
+ /* Port ATOP. Frames are tail dropped when this WM is hit */
+ spx5_wr(QSYS_ATOP_ATOP_SET(atop),
+ sparx5,
+ QSYS_ATOP(port->portno));
+
+ /* Discard pause frame 01-80-C2-00-00-01 */
+ spx5_wr(PAUSE_DISCARD, sparx5, ANA_CL_CAPTURE_BPDU_CFG(port->portno));
+
+ if (conf->portmode == PHY_INTERFACE_MODE_QSGMII ||
+ conf->portmode == PHY_INTERFACE_MODE_SGMII) {
+ err = sparx5_serdes_set(sparx5, port, conf);
+ if (err)
+ return err;
+
+ if (!sparx5_port_is_2g5(port->portno))
+ /* Enable shadow device */
+ spx5_rmw(DSM_DEV_TX_STOP_WM_CFG_DEV10G_SHADOW_ENA_SET(1),
+ DSM_DEV_TX_STOP_WM_CFG_DEV10G_SHADOW_ENA,
+ sparx5,
+ DSM_DEV_TX_STOP_WM_CFG(port->portno));
+
+ sparx5_dev_switch(sparx5, port->portno, false);
+ }
+ if (conf->portmode == PHY_INTERFACE_MODE_QSGMII) {
+ // All ports must be PCS enabled in QSGMII mode
+ spx5_rmw(DEV2G5_DEV_RST_CTRL_PCS_TX_RST_SET(0),
+ DEV2G5_DEV_RST_CTRL_PCS_TX_RST,
+ sparx5,
+ DEV2G5_DEV_RST_CTRL(port->portno));
+ }
+ /* Default IFGs for 1G */
+ spx5_wr(DEV2G5_MAC_IFG_CFG_TX_IFG_SET(6) |
+ DEV2G5_MAC_IFG_CFG_RX_IFG1_SET(0) |
+ DEV2G5_MAC_IFG_CFG_RX_IFG2_SET(0),
+ sparx5,
+ DEV2G5_MAC_IFG_CFG(port->portno));
+
+ if (sparx5_port_is_2g5(port->portno))
+ return 0; /* Low speed device only - return */
+
+ /* Now setup the high speed device */
+ if (conf->portmode == PHY_INTERFACE_MODE_NA)
+ conf->portmode = PHY_INTERFACE_MODE_10GBASER;
+
+ if (sparx5_is_baser(conf->portmode))
+ sparx5_dev_switch(sparx5, port->portno, true);
+
+ /* Set Max Length */
+ spx5_inst_rmw(DEV10G_MAC_MAXLEN_CFG_MAX_LEN_SET(ETH_MAXLEN),
+ DEV10G_MAC_MAXLEN_CFG_MAX_LEN,
+ devinst,
+ DEV10G_MAC_ENA_CFG(0));
+
+ /* Handle Signal Detect in 10G PCS */
+ spx5_inst_wr(PCS10G_BR_PCS_SD_CFG_SD_POL_SET(sd_pol) |
+ PCS10G_BR_PCS_SD_CFG_SD_SEL_SET(sd_sel) |
+ PCS10G_BR_PCS_SD_CFG_SD_ENA_SET(sd_ena),
+ pcsinst,
+ PCS10G_BR_PCS_SD_CFG(0));
+
+ if (sparx5_port_is_25g(port->portno)) {
+ /* Handle Signal Detect in 25G PCS */
+ spx5_wr(DEV25G_PCS25G_SD_CFG_SD_POL_SET(sd_pol) |
+ DEV25G_PCS25G_SD_CFG_SD_SEL_SET(sd_sel) |
+ DEV25G_PCS25G_SD_CFG_SD_ENA_SET(sd_ena),
+ sparx5,
+ DEV25G_PCS25G_SD_CFG(pix));
+ }
+
+ return 0;
+}
+
+void sparx5_port_enable(struct sparx5_port *port, bool enable)
+{
+ struct sparx5 *sparx5 = port->sparx5;
+
+ /* Enable port for frame transfer? */
+ spx5_rmw(QFWD_SWITCH_PORT_MODE_PORT_ENA_SET(enable),
+ QFWD_SWITCH_PORT_MODE_PORT_ENA,
+ sparx5,
+ QFWD_SWITCH_PORT_MODE(port->portno));
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0+ */
+/* Microchip Sparx5 Switch driver
+ *
+ * Copyright (c) 2021 Microchip Technology Inc. and its subsidiaries.
+ */
+
+#ifndef __SPARX5_PORT_H__
+#define __SPARX5_PORT_H__
+
+#include "sparx5_main.h"
+
+static inline bool sparx5_port_is_2g5(int portno)
+{
+ return portno >= 16 && portno <= 47;
+}
+
+static inline bool sparx5_port_is_5g(int portno)
+{
+ return portno <= 11 || portno == 64;
+}
+
+static inline bool sparx5_port_is_10g(int portno)
+{
+ return (portno >= 12 && portno <= 15) || (portno >= 48 && portno <= 55);
+}
+
+static inline bool sparx5_port_is_25g(int portno)
+{
+ return portno >= 56 && portno <= 63;
+}
+
+static inline u32 sparx5_to_high_dev(int port)
+{
+ if (sparx5_port_is_5g(port))
+ return TARGET_DEV5G;
+ if (sparx5_port_is_10g(port))
+ return TARGET_DEV10G;
+ return TARGET_DEV25G;
+}
+
+static inline u32 sparx5_to_pcs_dev(int port)
+{
+ if (sparx5_port_is_5g(port))
+ return TARGET_PCS5G_BR;
+ if (sparx5_port_is_10g(port))
+ return TARGET_PCS10G_BR;
+ return TARGET_PCS25G_BR;
+}
+
+static inline int sparx5_port_dev_index(int port)
+{
+ if (sparx5_port_is_2g5(port))
+ return port;
+ if (sparx5_port_is_5g(port))
+ return (port <= 11 ? port : 12);
+ if (sparx5_port_is_10g(port))
+ return (port >= 12 && port <= 15) ?
+ port - 12 : port - 44;
+ return (port - 56);
+}
+
+int sparx5_port_init(struct sparx5 *sparx5,
+ struct sparx5_port *spx5_port,
+ struct sparx5_port_config *conf);
+
+int sparx5_port_config(struct sparx5 *sparx5,
+ struct sparx5_port *spx5_port,
+ struct sparx5_port_config *conf);
+
+int sparx5_port_pcs_set(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_config *conf);
+
+int sparx5_serdes_set(struct sparx5 *sparx5,
+ struct sparx5_port *spx5_port,
+ struct sparx5_port_config *conf);
+
+struct sparx5_port_status {
+ bool link;
+ bool link_down;
+ int speed;
+ bool an_complete;
+ int duplex;
+ int pause;
+};
+
+int sparx5_get_port_status(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ struct sparx5_port_status *status);
+
+void sparx5_port_enable(struct sparx5_port *port, bool enable);
+
+#endif /* __SPARX5_PORT_H__ */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/* Microchip Sparx5 Switch driver
+ *
+ * Copyright (c) 2021 Microchip Technology Inc. and its subsidiaries.
+ */
+
+#include <linux/if_bridge.h>
+#include <net/switchdev.h>
+
+#include "sparx5_main_regs.h"
+#include "sparx5_main.h"
+
+static struct workqueue_struct *sparx5_owq;
+
+struct sparx5_switchdev_event_work {
+ struct work_struct work;
+ struct switchdev_notifier_fdb_info fdb_info;
+ struct net_device *dev;
+ unsigned long event;
+};
+
+static void sparx5_port_attr_bridge_flags(struct sparx5_port *port,
+ struct switchdev_brport_flags flags)
+{
+ if (flags.mask & BR_MCAST_FLOOD)
+ sparx5_pgid_update_mask(port, PGID_MC_FLOOD, true);
+}
+
+static void sparx5_attr_stp_state_set(struct sparx5_port *port,
+ u8 state)
+{
+ struct sparx5 *sparx5 = port->sparx5;
+
+ if (!test_bit(port->portno, sparx5->bridge_mask)) {
+ netdev_err(port->ndev,
+ "Controlling non-bridged port %d?\n", port->portno);
+ return;
+ }
+
+ switch (state) {
+ case BR_STATE_FORWARDING:
+ set_bit(port->portno, sparx5->bridge_fwd_mask);
+ fallthrough;
+ case BR_STATE_LEARNING:
+ set_bit(port->portno, sparx5->bridge_lrn_mask);
+ break;
+
+ default:
+ /* All other states treated as blocking */
+ clear_bit(port->portno, sparx5->bridge_fwd_mask);
+ clear_bit(port->portno, sparx5->bridge_lrn_mask);
+ break;
+ }
+
+ /* apply the bridge_fwd_mask to all the ports */
+ sparx5_update_fwd(sparx5);
+}
+
+static void sparx5_port_attr_ageing_set(struct sparx5_port *port,
+ unsigned long ageing_clock_t)
+{
+ unsigned long ageing_jiffies = clock_t_to_jiffies(ageing_clock_t);
+ u32 ageing_time = jiffies_to_msecs(ageing_jiffies);
+
+ sparx5_set_ageing(port->sparx5, ageing_time);
+}
+
+static int sparx5_port_attr_set(struct net_device *dev,
+ const struct switchdev_attr *attr,
+ struct netlink_ext_ack *extack)
+{
+ struct sparx5_port *port = netdev_priv(dev);
+
+ switch (attr->id) {
+ case SWITCHDEV_ATTR_ID_PORT_BRIDGE_FLAGS:
+ sparx5_port_attr_bridge_flags(port, attr->u.brport_flags);
+ break;
+ case SWITCHDEV_ATTR_ID_PORT_STP_STATE:
+ sparx5_attr_stp_state_set(port, attr->u.stp_state);
+ break;
+ case SWITCHDEV_ATTR_ID_BRIDGE_AGEING_TIME:
+ sparx5_port_attr_ageing_set(port, attr->u.ageing_time);
+ break;
+ case SWITCHDEV_ATTR_ID_BRIDGE_VLAN_FILTERING:
+ port->vlan_aware = attr->u.vlan_filtering;
+ sparx5_vlan_port_apply(port->sparx5, port);
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ return 0;
+}
+
+static int sparx5_port_bridge_join(struct sparx5_port *port,
+ struct net_device *bridge)
+{
+ struct sparx5 *sparx5 = port->sparx5;
+
+ if (bitmap_empty(sparx5->bridge_mask, SPX5_PORTS))
+ /* First bridged port */
+ sparx5->hw_bridge_dev = bridge;
+ else
+ if (sparx5->hw_bridge_dev != bridge)
+ /* This is adding the port to a second bridge, this is
+ * unsupported
+ */
+ return -ENODEV;
+
+ set_bit(port->portno, sparx5->bridge_mask);
+
+ /* Port enters in bridge mode therefor don't need to copy to CPU
+ * frames for multicast in case the bridge is not requesting them
+ */
+ __dev_mc_unsync(port->ndev, sparx5_mc_unsync);
+
+ return 0;
+}
+
+static void sparx5_port_bridge_leave(struct sparx5_port *port,
+ struct net_device *bridge)
+{
+ struct sparx5 *sparx5 = port->sparx5;
+
+ clear_bit(port->portno, sparx5->bridge_mask);
+ if (bitmap_empty(sparx5->bridge_mask, SPX5_PORTS))
+ sparx5->hw_bridge_dev = NULL;
+
+ /* Clear bridge vlan settings before updating the port settings */
+ port->vlan_aware = 0;
+ port->pvid = NULL_VID;
+ port->vid = NULL_VID;
+
+ /* Port enters in host more therefore restore mc list */
+ __dev_mc_sync(port->ndev, sparx5_mc_sync, sparx5_mc_unsync);
+}
+
+static int sparx5_port_changeupper(struct net_device *dev,
+ struct netdev_notifier_changeupper_info *info)
+{
+ struct sparx5_port *port = netdev_priv(dev);
+ int err = 0;
+
+ if (netif_is_bridge_master(info->upper_dev)) {
+ if (info->linking)
+ err = sparx5_port_bridge_join(port, info->upper_dev);
+ else
+ sparx5_port_bridge_leave(port, info->upper_dev);
+
+ sparx5_vlan_port_apply(port->sparx5, port);
+ }
+
+ return err;
+}
+
+static int sparx5_port_add_addr(struct net_device *dev, bool up)
+{
+ struct sparx5_port *port = netdev_priv(dev);
+ struct sparx5 *sparx5 = port->sparx5;
+ u16 vid = port->pvid;
+
+ if (up)
+ sparx5_mact_learn(sparx5, PGID_CPU, port->ndev->dev_addr, vid);
+ else
+ sparx5_mact_forget(sparx5, port->ndev->dev_addr, vid);
+
+ return 0;
+}
+
+static int sparx5_netdevice_port_event(struct net_device *dev,
+ struct notifier_block *nb,
+ unsigned long event, void *ptr)
+{
+ int err = 0;
+
+ if (!sparx5_netdevice_check(dev))
+ return 0;
+
+ switch (event) {
+ case NETDEV_CHANGEUPPER:
+ err = sparx5_port_changeupper(dev, ptr);
+ break;
+ case NETDEV_PRE_UP:
+ err = sparx5_port_add_addr(dev, true);
+ break;
+ case NETDEV_DOWN:
+ err = sparx5_port_add_addr(dev, false);
+ break;
+ }
+
+ return err;
+}
+
+static int sparx5_netdevice_event(struct notifier_block *nb,
+ unsigned long event, void *ptr)
+{
+ struct net_device *dev = netdev_notifier_info_to_dev(ptr);
+ int ret = 0;
+
+ ret = sparx5_netdevice_port_event(dev, nb, event, ptr);
+
+ return notifier_from_errno(ret);
+}
+
+static void sparx5_switchdev_bridge_fdb_event_work(struct work_struct *work)
+{
+ struct sparx5_switchdev_event_work *switchdev_work =
+ container_of(work, struct sparx5_switchdev_event_work, work);
+ struct net_device *dev = switchdev_work->dev;
+ struct switchdev_notifier_fdb_info *fdb_info;
+ struct sparx5_port *port;
+ struct sparx5 *sparx5;
+
+ rtnl_lock();
+ if (!sparx5_netdevice_check(dev))
+ goto out;
+
+ port = netdev_priv(dev);
+ sparx5 = port->sparx5;
+
+ fdb_info = &switchdev_work->fdb_info;
+
+ switch (switchdev_work->event) {
+ case SWITCHDEV_FDB_ADD_TO_DEVICE:
+ if (!fdb_info->added_by_user)
+ break;
+ sparx5_add_mact_entry(sparx5, port, fdb_info->addr,
+ fdb_info->vid);
+ break;
+ case SWITCHDEV_FDB_DEL_TO_DEVICE:
+ if (!fdb_info->added_by_user)
+ break;
+ sparx5_del_mact_entry(sparx5, fdb_info->addr, fdb_info->vid);
+ break;
+ }
+
+out:
+ rtnl_unlock();
+ kfree(switchdev_work->fdb_info.addr);
+ kfree(switchdev_work);
+ dev_put(dev);
+}
+
+static void sparx5_schedule_work(struct work_struct *work)
+{
+ queue_work(sparx5_owq, work);
+}
+
+static int sparx5_switchdev_event(struct notifier_block *unused,
+ unsigned long event, void *ptr)
+{
+ struct net_device *dev = switchdev_notifier_info_to_dev(ptr);
+ struct sparx5_switchdev_event_work *switchdev_work;
+ struct switchdev_notifier_fdb_info *fdb_info;
+ struct switchdev_notifier_info *info = ptr;
+ int err;
+
+ switch (event) {
+ case SWITCHDEV_PORT_ATTR_SET:
+ err = switchdev_handle_port_attr_set(dev, ptr,
+ sparx5_netdevice_check,
+ sparx5_port_attr_set);
+ return notifier_from_errno(err);
+ case SWITCHDEV_FDB_ADD_TO_DEVICE:
+ fallthrough;
+ case SWITCHDEV_FDB_DEL_TO_DEVICE:
+ switchdev_work = kzalloc(sizeof(*switchdev_work), GFP_ATOMIC);
+ if (!switchdev_work)
+ return NOTIFY_BAD;
+
+ switchdev_work->dev = dev;
+ switchdev_work->event = event;
+
+ fdb_info = container_of(info,
+ struct switchdev_notifier_fdb_info,
+ info);
+ INIT_WORK(&switchdev_work->work,
+ sparx5_switchdev_bridge_fdb_event_work);
+ memcpy(&switchdev_work->fdb_info, ptr,
+ sizeof(switchdev_work->fdb_info));
+ switchdev_work->fdb_info.addr = kzalloc(ETH_ALEN, GFP_ATOMIC);
+ if (!switchdev_work->fdb_info.addr)
+ goto err_addr_alloc;
+
+ ether_addr_copy((u8 *)switchdev_work->fdb_info.addr,
+ fdb_info->addr);
+ dev_hold(dev);
+
+ sparx5_schedule_work(&switchdev_work->work);
+ break;
+ }
+
+ return NOTIFY_DONE;
+err_addr_alloc:
+ kfree(switchdev_work);
+ return NOTIFY_BAD;
+}
+
+static void sparx5_sync_port_dev_addr(struct sparx5 *sparx5,
+ struct sparx5_port *port,
+ u16 vid, bool add)
+{
+ if (!port ||
+ !test_bit(port->portno, sparx5->bridge_mask))
+ return; /* Skip null/host interfaces */
+
+ /* Bridge connects to vid? */
+ if (add) {
+ /* Add port MAC address from the VLAN */
+ sparx5_mact_learn(sparx5, PGID_CPU,
+ port->ndev->dev_addr, vid);
+ } else {
+ /* Control port addr visibility depending on
+ * port VLAN connectivity.
+ */
+ if (test_bit(port->portno, sparx5->vlan_mask[vid]))
+ sparx5_mact_learn(sparx5, PGID_CPU,
+ port->ndev->dev_addr, vid);
+ else
+ sparx5_mact_forget(sparx5,
+ port->ndev->dev_addr, vid);
+ }
+}
+
+static void sparx5_sync_bridge_dev_addr(struct net_device *dev,
+ struct sparx5 *sparx5,
+ u16 vid, bool add)
+{
+ int i;
+
+ /* First, handle bridge address'es */
+ if (add) {
+ sparx5_mact_learn(sparx5, PGID_CPU, dev->dev_addr,
+ vid);
+ sparx5_mact_learn(sparx5, PGID_BCAST, dev->broadcast,
+ vid);
+ } else {
+ sparx5_mact_forget(sparx5, dev->dev_addr, vid);
+ sparx5_mact_forget(sparx5, dev->broadcast, vid);
+ }
+
+ /* Now look at bridged ports */
+ for (i = 0; i < SPX5_PORTS; i++)
+ sparx5_sync_port_dev_addr(sparx5, sparx5->ports[i], vid, add);
+}
+
+static int sparx5_handle_port_vlan_add(struct net_device *dev,
+ struct notifier_block *nb,
+ const struct switchdev_obj_port_vlan *v)
+{
+ struct sparx5_port *port = netdev_priv(dev);
+
+ if (netif_is_bridge_master(dev)) {
+ if (v->flags & BRIDGE_VLAN_INFO_BRENTRY) {
+ struct sparx5 *sparx5 =
+ container_of(nb, struct sparx5,
+ switchdev_blocking_nb);
+
+ sparx5_sync_bridge_dev_addr(dev, sparx5, v->vid, true);
+ }
+ return 0;
+ }
+
+ if (!sparx5_netdevice_check(dev))
+ return -EOPNOTSUPP;
+
+ return sparx5_vlan_vid_add(port, v->vid,
+ v->flags & BRIDGE_VLAN_INFO_PVID,
+ v->flags & BRIDGE_VLAN_INFO_UNTAGGED);
+}
+
+static int sparx5_handle_port_obj_add(struct net_device *dev,
+ struct notifier_block *nb,
+ struct switchdev_notifier_port_obj_info *info)
+{
+ const struct switchdev_obj *obj = info->obj;
+ int err;
+
+ switch (obj->id) {
+ case SWITCHDEV_OBJ_ID_PORT_VLAN:
+ err = sparx5_handle_port_vlan_add(dev, nb,
+ SWITCHDEV_OBJ_PORT_VLAN(obj));
+ break;
+ default:
+ err = -EOPNOTSUPP;
+ break;
+ }
+
+ info->handled = true;
+ return err;
+}
+
+static int sparx5_handle_port_vlan_del(struct net_device *dev,
+ struct notifier_block *nb,
+ u16 vid)
+{
+ struct sparx5_port *port = netdev_priv(dev);
+ int ret;
+
+ /* Master bridge? */
+ if (netif_is_bridge_master(dev)) {
+ struct sparx5 *sparx5 =
+ container_of(nb, struct sparx5,
+ switchdev_blocking_nb);
+
+ sparx5_sync_bridge_dev_addr(dev, sparx5, vid, false);
+ return 0;
+ }
+
+ if (!sparx5_netdevice_check(dev))
+ return -EOPNOTSUPP;
+
+ ret = sparx5_vlan_vid_del(port, vid);
+ if (ret)
+ return ret;
+
+ /* Delete the port MAC address with the matching VLAN information */
+ sparx5_mact_forget(port->sparx5, port->ndev->dev_addr, vid);
+
+ return 0;
+}
+
+static int sparx5_handle_port_obj_del(struct net_device *dev,
+ struct notifier_block *nb,
+ struct switchdev_notifier_port_obj_info *info)
+{
+ const struct switchdev_obj *obj = info->obj;
+ int err;
+
+ switch (obj->id) {
+ case SWITCHDEV_OBJ_ID_PORT_VLAN:
+ err = sparx5_handle_port_vlan_del(dev, nb,
+ SWITCHDEV_OBJ_PORT_VLAN(obj)->vid);
+ break;
+ default:
+ err = -EOPNOTSUPP;
+ break;
+ }
+
+ info->handled = true;
+ return err;
+}
+
+static int sparx5_switchdev_blocking_event(struct notifier_block *nb,
+ unsigned long event,
+ void *ptr)
+{
+ struct net_device *dev = switchdev_notifier_info_to_dev(ptr);
+ int err;
+
+ switch (event) {
+ case SWITCHDEV_PORT_OBJ_ADD:
+ err = sparx5_handle_port_obj_add(dev, nb, ptr);
+ return notifier_from_errno(err);
+ case SWITCHDEV_PORT_OBJ_DEL:
+ err = sparx5_handle_port_obj_del(dev, nb, ptr);
+ return notifier_from_errno(err);
+ case SWITCHDEV_PORT_ATTR_SET:
+ err = switchdev_handle_port_attr_set(dev, ptr,
+ sparx5_netdevice_check,
+ sparx5_port_attr_set);
+ return notifier_from_errno(err);
+ }
+
+ return NOTIFY_DONE;
+}
+
+int sparx5_register_notifier_blocks(struct sparx5 *s5)
+{
+ int err;
+
+ s5->netdevice_nb.notifier_call = sparx5_netdevice_event;
+ err = register_netdevice_notifier(&s5->netdevice_nb);
+ if (err)
+ return err;
+
+ s5->switchdev_nb.notifier_call = sparx5_switchdev_event;
+ err = register_switchdev_notifier(&s5->switchdev_nb);
+ if (err)
+ goto err_switchdev_nb;
+
+ s5->switchdev_blocking_nb.notifier_call = sparx5_switchdev_blocking_event;
+ err = register_switchdev_blocking_notifier(&s5->switchdev_blocking_nb);
+ if (err)
+ goto err_switchdev_blocking_nb;
+
+ sparx5_owq = alloc_ordered_workqueue("sparx5_order", 0);
+ if (!sparx5_owq)
+ goto err_switchdev_blocking_nb;
+
+ return 0;
+
+err_switchdev_blocking_nb:
+ unregister_switchdev_notifier(&s5->switchdev_nb);
+err_switchdev_nb:
+ unregister_netdevice_notifier(&s5->netdevice_nb);
+
+ return err;
+}
+
+void sparx5_unregister_notifier_blocks(struct sparx5 *s5)
+{
+ destroy_workqueue(sparx5_owq);
+
+ unregister_switchdev_blocking_notifier(&s5->switchdev_blocking_nb);
+ unregister_switchdev_notifier(&s5->switchdev_nb);
+ unregister_netdevice_notifier(&s5->netdevice_nb);
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/* Microchip Sparx5 Switch driver
+ *
+ * Copyright (c) 2021 Microchip Technology Inc. and its subsidiaries.
+ */
+
+#include "sparx5_main_regs.h"
+#include "sparx5_main.h"
+
+static int sparx5_vlant_set_mask(struct sparx5 *sparx5, u16 vid)
+{
+ u32 mask[3];
+
+ /* Divide up mask in 32 bit words */
+ bitmap_to_arr32(mask, sparx5->vlan_mask[vid], SPX5_PORTS);
+
+ /* Output mask to respective registers */
+ spx5_wr(mask[0], sparx5, ANA_L3_VLAN_MASK_CFG(vid));
+ spx5_wr(mask[1], sparx5, ANA_L3_VLAN_MASK_CFG1(vid));
+ spx5_wr(mask[2], sparx5, ANA_L3_VLAN_MASK_CFG2(vid));
+
+ return 0;
+}
+
+void sparx5_vlan_init(struct sparx5 *sparx5)
+{
+ u16 vid;
+
+ spx5_rmw(ANA_L3_VLAN_CTRL_VLAN_ENA_SET(1),
+ ANA_L3_VLAN_CTRL_VLAN_ENA,
+ sparx5,
+ ANA_L3_VLAN_CTRL);
+
+ /* Map VLAN = FID */
+ for (vid = NULL_VID; vid < VLAN_N_VID; vid++)
+ spx5_rmw(ANA_L3_VLAN_CFG_VLAN_FID_SET(vid),
+ ANA_L3_VLAN_CFG_VLAN_FID,
+ sparx5,
+ ANA_L3_VLAN_CFG(vid));
+}
+
+void sparx5_vlan_port_setup(struct sparx5 *sparx5, int portno)
+{
+ struct sparx5_port *port = sparx5->ports[portno];
+
+ /* Configure PVID */
+ spx5_rmw(ANA_CL_VLAN_CTRL_VLAN_AWARE_ENA_SET(0) |
+ ANA_CL_VLAN_CTRL_PORT_VID_SET(port->pvid),
+ ANA_CL_VLAN_CTRL_VLAN_AWARE_ENA |
+ ANA_CL_VLAN_CTRL_PORT_VID,
+ sparx5,
+ ANA_CL_VLAN_CTRL(port->portno));
+}
+
+int sparx5_vlan_vid_add(struct sparx5_port *port, u16 vid, bool pvid,
+ bool untagged)
+{
+ struct sparx5 *sparx5 = port->sparx5;
+ int ret;
+
+ /* Make the port a member of the VLAN */
+ set_bit(port->portno, sparx5->vlan_mask[vid]);
+ ret = sparx5_vlant_set_mask(sparx5, vid);
+ if (ret)
+ return ret;
+
+ /* Default ingress vlan classification */
+ if (pvid)
+ port->pvid = vid;
+
+ /* Untagged egress vlan classification */
+ if (untagged && port->vid != vid) {
+ if (port->vid) {
+ netdev_err(port->ndev,
+ "Port already has a native VLAN: %d\n",
+ port->vid);
+ return -EBUSY;
+ }
+ port->vid = vid;
+ }
+
+ sparx5_vlan_port_apply(sparx5, port);
+
+ return 0;
+}
+
+int sparx5_vlan_vid_del(struct sparx5_port *port, u16 vid)
+{
+ struct sparx5 *sparx5 = port->sparx5;
+ int ret;
+
+ /* 8021q removes VID 0 on module unload for all interfaces
+ * with VLAN filtering feature. We need to keep it to receive
+ * untagged traffic.
+ */
+ if (vid == 0)
+ return 0;
+
+ /* Stop the port from being a member of the vlan */
+ clear_bit(port->portno, sparx5->vlan_mask[vid]);
+ ret = sparx5_vlant_set_mask(sparx5, vid);
+ if (ret)
+ return ret;
+
+ /* Ingress */
+ if (port->pvid == vid)
+ port->pvid = 0;
+
+ /* Egress */
+ if (port->vid == vid)
+ port->vid = 0;
+
+ sparx5_vlan_port_apply(sparx5, port);
+
+ return 0;
+}
+
+void sparx5_pgid_update_mask(struct sparx5_port *port, int pgid, bool enable)
+{
+ struct sparx5 *sparx5 = port->sparx5;
+ u32 val, mask;
+
+ /* mask is spread across 3 registers x 32 bit */
+ if (port->portno < 32) {
+ mask = BIT(port->portno);
+ val = enable ? mask : 0;
+ spx5_rmw(val, mask, sparx5, ANA_AC_PGID_CFG(pgid));
+ } else if (port->portno < 64) {
+ mask = BIT(port->portno - 32);
+ val = enable ? mask : 0;
+ spx5_rmw(val, mask, sparx5, ANA_AC_PGID_CFG1(pgid));
+ } else if (port->portno < SPX5_PORTS) {
+ mask = BIT(port->portno - 64);
+ val = enable ? mask : 0;
+ spx5_rmw(val, mask, sparx5, ANA_AC_PGID_CFG2(pgid));
+ } else {
+ netdev_err(port->ndev, "Invalid port no: %d\n", port->portno);
+ }
+}
+
+void sparx5_update_fwd(struct sparx5 *sparx5)
+{
+ DECLARE_BITMAP(workmask, SPX5_PORTS);
+ u32 mask[3];
+ int port;
+
+ /* Divide up fwd mask in 32 bit words */
+ bitmap_to_arr32(mask, sparx5->bridge_fwd_mask, SPX5_PORTS);
+
+ /* Update flood masks */
+ for (port = PGID_UC_FLOOD; port <= PGID_BCAST; port++) {
+ spx5_wr(mask[0], sparx5, ANA_AC_PGID_CFG(port));
+ spx5_wr(mask[1], sparx5, ANA_AC_PGID_CFG1(port));
+ spx5_wr(mask[2], sparx5, ANA_AC_PGID_CFG2(port));
+ }
+
+ /* Update SRC masks */
+ for (port = 0; port < SPX5_PORTS; port++) {
+ if (test_bit(port, sparx5->bridge_fwd_mask)) {
+ /* Allow to send to all bridged but self */
+ bitmap_copy(workmask, sparx5->bridge_fwd_mask, SPX5_PORTS);
+ clear_bit(port, workmask);
+ bitmap_to_arr32(mask, workmask, SPX5_PORTS);
+ spx5_wr(mask[0], sparx5, ANA_AC_SRC_CFG(port));
+ spx5_wr(mask[1], sparx5, ANA_AC_SRC_CFG1(port));
+ spx5_wr(mask[2], sparx5, ANA_AC_SRC_CFG2(port));
+ } else {
+ spx5_wr(0, sparx5, ANA_AC_SRC_CFG(port));
+ spx5_wr(0, sparx5, ANA_AC_SRC_CFG1(port));
+ spx5_wr(0, sparx5, ANA_AC_SRC_CFG2(port));
+ }
+ }
+
+ /* Learning enabled only for bridged ports */
+ bitmap_and(workmask, sparx5->bridge_fwd_mask,
+ sparx5->bridge_lrn_mask, SPX5_PORTS);
+ bitmap_to_arr32(mask, workmask, SPX5_PORTS);
+
+ /* Apply learning mask */
+ spx5_wr(mask[0], sparx5, ANA_L2_AUTO_LRN_CFG);
+ spx5_wr(mask[1], sparx5, ANA_L2_AUTO_LRN_CFG1);
+ spx5_wr(mask[2], sparx5, ANA_L2_AUTO_LRN_CFG2);
+}
+
+void sparx5_vlan_port_apply(struct sparx5 *sparx5,
+ struct sparx5_port *port)
+
+{
+ u32 val;
+
+ /* Configure PVID, vlan aware */
+ val = ANA_CL_VLAN_CTRL_VLAN_AWARE_ENA_SET(port->vlan_aware) |
+ ANA_CL_VLAN_CTRL_VLAN_POP_CNT_SET(port->vlan_aware) |
+ ANA_CL_VLAN_CTRL_PORT_VID_SET(port->pvid);
+ spx5_wr(val, sparx5, ANA_CL_VLAN_CTRL(port->portno));
+
+ val = 0;
+ if (port->vlan_aware && !port->pvid)
+ /* If port is vlan-aware and tagged, drop untagged and
+ * priority tagged frames.
+ */
+ val = ANA_CL_VLAN_FILTER_CTRL_TAG_REQUIRED_ENA_SET(1) |
+ ANA_CL_VLAN_FILTER_CTRL_PRIO_CTAG_DIS_SET(1) |
+ ANA_CL_VLAN_FILTER_CTRL_PRIO_STAG_DIS_SET(1);
+ spx5_wr(val, sparx5,
+ ANA_CL_VLAN_FILTER_CTRL(port->portno, 0));
+
+ /* Egress configuration (REW_TAG_CFG): VLAN tag type to 8021Q */
+ val = REW_TAG_CTRL_TAG_TPID_CFG_SET(0);
+ if (port->vlan_aware) {
+ if (port->vid)
+ /* Tag all frames except when VID == DEFAULT_VLAN */
+ val |= REW_TAG_CTRL_TAG_CFG_SET(1);
+ else
+ val |= REW_TAG_CTRL_TAG_CFG_SET(3);
+ }
+ spx5_wr(val, sparx5, REW_TAG_CTRL(port->portno));
+
+ /* Egress VID */
+ spx5_rmw(REW_PORT_VLAN_CFG_PORT_VID_SET(port->vid),
+ REW_PORT_VLAN_CFG_PORT_VID,
+ sparx5,
+ REW_PORT_VLAN_CFG(port->portno));
+}
priv->ndev = ndev;
priv->pdev = pdev;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- ndev->base_addr = res->start;
- priv->base = devm_ioremap_resource(p_dev, res);
+ priv->base = devm_platform_get_and_ioremap_resource(pdev, 0, &res);
if (IS_ERR(priv->base)) {
ret = PTR_ERR(priv->base);
goto init_fail;
}
+ ndev->base_addr = res->start;
spin_lock_init(&priv->txlock);
int ocelot_port_flush(struct ocelot *ocelot, int port)
{
+ unsigned int pause_ena;
int err, val;
/* Disable dequeuing from the egress queues */
QSYS_PORT_MODE, port);
/* Disable flow control */
+ ocelot_fields_read(ocelot, port, SYS_PAUSE_CFG_PAUSE_ENA, &pause_ena);
ocelot_fields_write(ocelot, port, SYS_PAUSE_CFG_PAUSE_ENA, 0);
/* Disable priority flow control */
/* Clear flushing again. */
ocelot_rmw_gix(ocelot, 0, REW_PORT_CFG_FLUSH_ENA, REW_PORT_CFG, port);
+ /* Re-enable flow control */
+ ocelot_fields_write(ocelot, port, SYS_PAUSE_CFG_PAUSE_ENA, pause_ena);
+
return err;
}
EXPORT_SYMBOL(ocelot_port_flush);
dev_err(&pdev->dev,
"invalid sram_size %dB or board span %ldB\n",
mgp->sram_size, mgp->board_span);
+ status = -EINVAL;
goto abort_with_ioremap;
}
memcpy_fromio(mgp->eeprom_strings,
return 0;
err_create_file:
- unregister_netdev(dev);
+ unregister_netdev(dev);
err_register_netdev:
iounmap(ioaddr);
case SIOCSMIIREG: /* Write MII PHY register. */
if (dev->if_port == PORT_TP) {
if ((data->phy_id & 0x1f) == np->phy_addr_external) {
- if ((data->reg_num & 0x1f) == MII_ADVERTISE)
+ if ((data->reg_num & 0x1f) == MII_ADVERTISE)
np->advertising = data->val_in;
mdio_write(dev, data->reg_num & 0x1f,
data->val_in);
}
} else {
if ((data->phy_id & 0x1f) == np->phy_addr_external) {
- if ((data->reg_num & 0x1f) == MII_ADVERTISE)
+ if ((data->reg_num & 0x1f) == MII_ADVERTISE)
np->advertising = data->val_in;
}
move_int_phy(dev, data->phy_id & 0x1f);
* @nic: device private variable
* @link: link status (UP/DOWN) used to enable/disable continuous
* transmit interrupts
+ * @may_sleep: parameter indicates if sleeping when waiting for
+ * command complete
* Description: The function configures transmit traffic interrupts
* Return Value: SUCCESS on success and
* '-1' on failure
}
/**
- * s2io_poll - Rx interrupt handler for NAPI support
+ * s2io_poll_msix - Rx interrupt handler for NAPI support
* @napi : pointer to the napi structure.
* @budget : The number of packets that were budgeted to be processed
* during one pass through the 'Poll" function.
* @addr: address
* @busy_bit: bit to check for busy
* @bit_state: state to check
+ * @may_sleep: parameter indicates if sleeping when waiting for
+ * command complete
* Description: Function that waits for a command to Write into RMAC
* ADDR DATA registers to be completed and returns either success or
* error depending on whether the command was complete or not.
/**
* s2io_set_multicast - entry point for multicast address enable/disable.
* @dev : pointer to the device structure
+ * @may_sleep: parameter indicates if sleeping when waiting for command
+ * complete
* Description:
* This function is a driver entry point which gets called by the kernel
* whenever multicast addresses must be enabled/disabled. This also gets
}
/**
- * s2io_ethtol_get_link_ksettings - Return link specific information.
+ * s2io_ethtool_get_link_ksettings - Return link specific information.
* @dev: pointer to netdev
* @cmd : pointer to the structure with parameters given by ethtool
* to return link information.
}
/**
- * vxge_hw_vpath_rx_doorbell_post - Close the handle got from previous vpath
+ * vxge_hw_vpath_rx_doorbell_init - Close the handle got from previous vpath
* (vpath) open
* @vp: Handle got from previous vpath open
*
module_param_array(bw_percentage, uint, NULL, 0);
static struct vxge_drv_config *driver_config;
-static enum vxge_hw_status vxge_reset_all_vpaths(struct vxgedev *vdev);
+static void vxge_reset_all_vpaths(struct vxgedev *vdev);
static inline int is_vxge_card_up(struct vxgedev *vdev)
{
static int do_vxge_reset(struct vxgedev *vdev, int event)
{
- enum vxge_hw_status status;
int ret = 0, vp_id, i;
vxge_debug_entryexit(VXGE_TRACE, "%s:%d", __func__, __LINE__);
netif_tx_stop_all_queues(vdev->ndev);
if (event == VXGE_LL_FULL_RESET) {
- status = vxge_reset_all_vpaths(vdev);
- if (status != VXGE_HW_OK) {
- vxge_debug_init(VXGE_ERR,
- "fatal: %s: can not reset vpaths",
- vdev->ndev->name);
- ret = -EPERM;
- goto out;
- }
+ vxge_reset_all_vpaths(vdev);
}
if (event == VXGE_LL_COMPL_RESET) {
}
/**
- * vxge_poll - Receive handler when Receive Polling is used.
+ * vxge_poll_msix - Receive handler when Receive Polling is used.
* @napi: pointer to the napi structure.
* @budget: Number of packets budgeted to be processed in this iteration.
*
}
/* reset vpaths */
-static enum vxge_hw_status vxge_reset_all_vpaths(struct vxgedev *vdev)
+static void vxge_reset_all_vpaths(struct vxgedev *vdev)
{
- enum vxge_hw_status status = VXGE_HW_OK;
struct vxge_vpath *vpath;
int i;
"vxge_hw_vpath_recover_"
"from_reset failed for vpath: "
"%d", i);
- return status;
+ return;
}
} else {
vxge_debug_init(VXGE_ERR,
"vxge_hw_vpath_reset failed for "
"vpath:%d", i);
- return status;
+ return;
}
}
}
-
- return status;
}
/* close vpaths */
/* !netif_running() ensured by vxge_fix_features() */
vdev->devh->config.rth_en = !!(features & NETIF_F_RXHASH);
- if (vxge_reset_all_vpaths(vdev) != VXGE_HW_OK) {
- dev->features = features ^ NETIF_F_RXHASH;
- vdev->devh->config.rth_en = !!(dev->features & NETIF_F_RXHASH);
- return -EIO;
- }
+ vxge_reset_all_vpaths(vdev);
return 0;
}
driver_config->vpath_per_dev = 1;
for (i = 0; i < VXGE_HW_MAX_VIRTUAL_PATHS; i++)
- if (!vxge_bVALn(vpath_mask, i, 1))
- continue;
- else
+ if (vxge_bVALn(vpath_mask, i, 1))
default_no_vpath++;
+
if (default_no_vpath < driver_config->vpath_per_dev)
driver_config->vpath_per_dev = default_no_vpath;
}
/**
- * vxge_rem_nic - Free the PCI device
+ * vxge_remove - Free the PCI device
* @pdev: structure containing the PCI related information of the device.
* Description: This function is called by the Pci subsystem to release a
* PCI device and free up all resource held up by the device.
flower/metadata.o \
flower/offload.o \
flower/tunnel_conf.o \
- flower/qos_conf.o
+ flower/qos_conf.o \
+ flower/conntrack.o
endif
ifeq ($(CONFIG_BPF_SYSCALL),y)
};
/**
- * struct nfp_ccm_mbox_skb_cb - CCM mailbox specific info
+ * struct nfp_ccm_mbox_cmsg_cb - CCM mailbox specific info
* @state: processing state (/stage) of the message
* @err: error encountered during processing if any
* @max_len: max(request_len, reply_len)
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+/* Copyright (C) 2021 Corigine, Inc. */
+
+#include "conntrack.h"
+
+const struct rhashtable_params nfp_tc_ct_merge_params = {
+ .head_offset = offsetof(struct nfp_fl_ct_tc_merge,
+ hash_node),
+ .key_len = sizeof(unsigned long) * 2,
+ .key_offset = offsetof(struct nfp_fl_ct_tc_merge, cookie),
+ .automatic_shrinking = true,
+};
+
+const struct rhashtable_params nfp_nft_ct_merge_params = {
+ .head_offset = offsetof(struct nfp_fl_nft_tc_merge,
+ hash_node),
+ .key_len = sizeof(unsigned long) * 3,
+ .key_offset = offsetof(struct nfp_fl_nft_tc_merge, cookie),
+ .automatic_shrinking = true,
+};
+
+static struct flow_action_entry *get_flow_act(struct flow_rule *rule,
+ enum flow_action_id act_id);
+
+/**
+ * get_hashentry() - Wrapper around hashtable lookup.
+ * @ht: hashtable where entry could be found
+ * @key: key to lookup
+ * @params: hashtable params
+ * @size: size of entry to allocate if not in table
+ *
+ * Returns an entry from a hashtable. If entry does not exist
+ * yet allocate the memory for it and return the new entry.
+ */
+static void *get_hashentry(struct rhashtable *ht, void *key,
+ const struct rhashtable_params params, size_t size)
+{
+ void *result;
+
+ result = rhashtable_lookup_fast(ht, key, params);
+
+ if (result)
+ return result;
+
+ result = kzalloc(size, GFP_KERNEL);
+ if (!result)
+ return ERR_PTR(-ENOMEM);
+
+ return result;
+}
+
+bool is_pre_ct_flow(struct flow_cls_offload *flow)
+{
+ struct flow_action_entry *act;
+ int i;
+
+ flow_action_for_each(i, act, &flow->rule->action) {
+ if (act->id == FLOW_ACTION_CT && !act->ct.action)
+ return true;
+ }
+ return false;
+}
+
+bool is_post_ct_flow(struct flow_cls_offload *flow)
+{
+ struct flow_rule *rule = flow_cls_offload_flow_rule(flow);
+ struct flow_dissector *dissector = rule->match.dissector;
+ struct flow_match_ct ct;
+
+ if (dissector->used_keys & BIT(FLOW_DISSECTOR_KEY_CT)) {
+ flow_rule_match_ct(rule, &ct);
+ if (ct.key->ct_state & TCA_FLOWER_KEY_CT_FLAGS_ESTABLISHED)
+ return true;
+ }
+ return false;
+}
+
+static int nfp_ct_merge_check(struct nfp_fl_ct_flow_entry *entry1,
+ struct nfp_fl_ct_flow_entry *entry2)
+{
+ unsigned int ovlp_keys = entry1->rule->match.dissector->used_keys &
+ entry2->rule->match.dissector->used_keys;
+ bool out;
+
+ /* check the overlapped fields one by one, the unmasked part
+ * should not conflict with each other.
+ */
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_CONTROL)) {
+ struct flow_match_control match1, match2;
+
+ flow_rule_match_control(entry1->rule, &match1);
+ flow_rule_match_control(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_BASIC)) {
+ struct flow_match_basic match1, match2;
+
+ flow_rule_match_basic(entry1->rule, &match1);
+ flow_rule_match_basic(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_IPV4_ADDRS)) {
+ struct flow_match_ipv4_addrs match1, match2;
+
+ flow_rule_match_ipv4_addrs(entry1->rule, &match1);
+ flow_rule_match_ipv4_addrs(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_IPV6_ADDRS)) {
+ struct flow_match_ipv6_addrs match1, match2;
+
+ flow_rule_match_ipv6_addrs(entry1->rule, &match1);
+ flow_rule_match_ipv6_addrs(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_PORTS)) {
+ struct flow_match_ports match1, match2;
+
+ flow_rule_match_ports(entry1->rule, &match1);
+ flow_rule_match_ports(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_ETH_ADDRS)) {
+ struct flow_match_eth_addrs match1, match2;
+
+ flow_rule_match_eth_addrs(entry1->rule, &match1);
+ flow_rule_match_eth_addrs(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_VLAN)) {
+ struct flow_match_vlan match1, match2;
+
+ flow_rule_match_vlan(entry1->rule, &match1);
+ flow_rule_match_vlan(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_MPLS)) {
+ struct flow_match_mpls match1, match2;
+
+ flow_rule_match_mpls(entry1->rule, &match1);
+ flow_rule_match_mpls(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_TCP)) {
+ struct flow_match_tcp match1, match2;
+
+ flow_rule_match_tcp(entry1->rule, &match1);
+ flow_rule_match_tcp(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_IP)) {
+ struct flow_match_ip match1, match2;
+
+ flow_rule_match_ip(entry1->rule, &match1);
+ flow_rule_match_ip(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_ENC_KEYID)) {
+ struct flow_match_enc_keyid match1, match2;
+
+ flow_rule_match_enc_keyid(entry1->rule, &match1);
+ flow_rule_match_enc_keyid(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_ENC_IPV4_ADDRS)) {
+ struct flow_match_ipv4_addrs match1, match2;
+
+ flow_rule_match_enc_ipv4_addrs(entry1->rule, &match1);
+ flow_rule_match_enc_ipv4_addrs(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_ENC_IPV6_ADDRS)) {
+ struct flow_match_ipv6_addrs match1, match2;
+
+ flow_rule_match_enc_ipv6_addrs(entry1->rule, &match1);
+ flow_rule_match_enc_ipv6_addrs(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_ENC_CONTROL)) {
+ struct flow_match_control match1, match2;
+
+ flow_rule_match_enc_control(entry1->rule, &match1);
+ flow_rule_match_enc_control(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_ENC_IP)) {
+ struct flow_match_ip match1, match2;
+
+ flow_rule_match_enc_ip(entry1->rule, &match1);
+ flow_rule_match_enc_ip(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ if (ovlp_keys & BIT(FLOW_DISSECTOR_KEY_ENC_OPTS)) {
+ struct flow_match_enc_opts match1, match2;
+
+ flow_rule_match_enc_opts(entry1->rule, &match1);
+ flow_rule_match_enc_opts(entry2->rule, &match2);
+ COMPARE_UNMASKED_FIELDS(match1, match2, &out);
+ if (out)
+ goto check_failed;
+ }
+
+ return 0;
+
+check_failed:
+ return -EINVAL;
+}
+
+static int nfp_ct_check_mangle_merge(struct flow_action_entry *a_in,
+ struct flow_rule *rule)
+{
+ enum flow_action_mangle_base htype = a_in->mangle.htype;
+ u32 offset = a_in->mangle.offset;
+
+ switch (htype) {
+ case FLOW_ACT_MANGLE_HDR_TYPE_ETH:
+ if (flow_rule_match_key(rule, FLOW_DISSECTOR_KEY_ETH_ADDRS))
+ return -EOPNOTSUPP;
+ break;
+ case FLOW_ACT_MANGLE_HDR_TYPE_IP4:
+ if (flow_rule_match_key(rule, FLOW_DISSECTOR_KEY_IP)) {
+ struct flow_match_ip match;
+
+ flow_rule_match_ip(rule, &match);
+ if (offset == offsetof(struct iphdr, ttl) &&
+ match.mask->ttl)
+ return -EOPNOTSUPP;
+ if (offset == round_down(offsetof(struct iphdr, tos), 4) &&
+ match.mask->tos)
+ return -EOPNOTSUPP;
+ }
+ if (flow_rule_match_key(rule, FLOW_DISSECTOR_KEY_IPV4_ADDRS)) {
+ struct flow_match_ipv4_addrs match;
+
+ flow_rule_match_ipv4_addrs(rule, &match);
+ if (offset == offsetof(struct iphdr, saddr) &&
+ match.mask->src)
+ return -EOPNOTSUPP;
+ if (offset == offsetof(struct iphdr, daddr) &&
+ match.mask->dst)
+ return -EOPNOTSUPP;
+ }
+ break;
+ case FLOW_ACT_MANGLE_HDR_TYPE_IP6:
+ if (flow_rule_match_key(rule, FLOW_DISSECTOR_KEY_IP)) {
+ struct flow_match_ip match;
+
+ flow_rule_match_ip(rule, &match);
+ if (offset == round_down(offsetof(struct ipv6hdr, hop_limit), 4) &&
+ match.mask->ttl)
+ return -EOPNOTSUPP;
+ /* for ipv6, tos and flow_lbl are in the same word */
+ if (offset == round_down(offsetof(struct ipv6hdr, flow_lbl), 4) &&
+ match.mask->tos)
+ return -EOPNOTSUPP;
+ }
+ if (flow_rule_match_key(rule, FLOW_DISSECTOR_KEY_IPV6_ADDRS)) {
+ struct flow_match_ipv6_addrs match;
+
+ flow_rule_match_ipv6_addrs(rule, &match);
+ if (offset >= offsetof(struct ipv6hdr, saddr) &&
+ offset < offsetof(struct ipv6hdr, daddr) &&
+ memchr_inv(&match.mask->src, 0, sizeof(match.mask->src)))
+ return -EOPNOTSUPP;
+ if (offset >= offsetof(struct ipv6hdr, daddr) &&
+ offset < sizeof(struct ipv6hdr) &&
+ memchr_inv(&match.mask->dst, 0, sizeof(match.mask->dst)))
+ return -EOPNOTSUPP;
+ }
+ break;
+ case FLOW_ACT_MANGLE_HDR_TYPE_TCP:
+ case FLOW_ACT_MANGLE_HDR_TYPE_UDP:
+ /* currently only can modify ports */
+ if (flow_rule_match_key(rule, FLOW_DISSECTOR_KEY_PORTS))
+ return -EOPNOTSUPP;
+ break;
+ default:
+ break;
+ }
+ return 0;
+}
+
+static int nfp_ct_merge_act_check(struct nfp_fl_ct_flow_entry *pre_ct_entry,
+ struct nfp_fl_ct_flow_entry *post_ct_entry,
+ struct nfp_fl_ct_flow_entry *nft_entry)
+{
+ struct flow_action_entry *act;
+ int err, i;
+
+ /* Check for pre_ct->action conflicts */
+ flow_action_for_each(i, act, &pre_ct_entry->rule->action) {
+ switch (act->id) {
+ case FLOW_ACTION_MANGLE:
+ err = nfp_ct_check_mangle_merge(act, nft_entry->rule);
+ if (err)
+ return err;
+ err = nfp_ct_check_mangle_merge(act, post_ct_entry->rule);
+ if (err)
+ return err;
+ break;
+ case FLOW_ACTION_VLAN_PUSH:
+ case FLOW_ACTION_VLAN_POP:
+ case FLOW_ACTION_VLAN_MANGLE:
+ case FLOW_ACTION_MPLS_PUSH:
+ case FLOW_ACTION_MPLS_POP:
+ case FLOW_ACTION_MPLS_MANGLE:
+ return -EOPNOTSUPP;
+ default:
+ break;
+ }
+ }
+
+ /* Check for nft->action conflicts */
+ flow_action_for_each(i, act, &nft_entry->rule->action) {
+ switch (act->id) {
+ case FLOW_ACTION_MANGLE:
+ err = nfp_ct_check_mangle_merge(act, post_ct_entry->rule);
+ if (err)
+ return err;
+ break;
+ case FLOW_ACTION_VLAN_PUSH:
+ case FLOW_ACTION_VLAN_POP:
+ case FLOW_ACTION_VLAN_MANGLE:
+ case FLOW_ACTION_MPLS_PUSH:
+ case FLOW_ACTION_MPLS_POP:
+ case FLOW_ACTION_MPLS_MANGLE:
+ return -EOPNOTSUPP;
+ default:
+ break;
+ }
+ }
+ return 0;
+}
+
+static int nfp_ct_check_meta(struct nfp_fl_ct_flow_entry *post_ct_entry,
+ struct nfp_fl_ct_flow_entry *nft_entry)
+{
+ struct flow_dissector *dissector = post_ct_entry->rule->match.dissector;
+ struct flow_action_entry *ct_met;
+ struct flow_match_ct ct;
+ int i;
+
+ ct_met = get_flow_act(nft_entry->rule, FLOW_ACTION_CT_METADATA);
+ if (ct_met && (dissector->used_keys & BIT(FLOW_DISSECTOR_KEY_CT))) {
+ u32 *act_lbl;
+
+ act_lbl = ct_met->ct_metadata.labels;
+ flow_rule_match_ct(post_ct_entry->rule, &ct);
+ for (i = 0; i < 4; i++) {
+ if ((ct.key->ct_labels[i] & ct.mask->ct_labels[i]) ^
+ (act_lbl[i] & ct.mask->ct_labels[i]))
+ return -EINVAL;
+ }
+
+ if ((ct.key->ct_mark & ct.mask->ct_mark) ^
+ (ct_met->ct_metadata.mark & ct.mask->ct_mark))
+ return -EINVAL;
+
+ return 0;
+ }
+
+ return -EINVAL;
+}
+
+static int nfp_fl_ct_add_offload(struct nfp_fl_nft_tc_merge *m_entry)
+{
+ return 0;
+}
+
+static int nfp_fl_ct_del_offload(struct nfp_app *app, unsigned long cookie,
+ struct net_device *netdev)
+{
+ return 0;
+}
+
+static int nfp_ct_do_nft_merge(struct nfp_fl_ct_zone_entry *zt,
+ struct nfp_fl_ct_flow_entry *nft_entry,
+ struct nfp_fl_ct_tc_merge *tc_m_entry)
+{
+ struct nfp_fl_ct_flow_entry *post_ct_entry, *pre_ct_entry;
+ struct nfp_fl_nft_tc_merge *nft_m_entry;
+ unsigned long new_cookie[3];
+ int err;
+
+ pre_ct_entry = tc_m_entry->pre_ct_parent;
+ post_ct_entry = tc_m_entry->post_ct_parent;
+
+ err = nfp_ct_merge_act_check(pre_ct_entry, post_ct_entry, nft_entry);
+ if (err)
+ return err;
+
+ /* Check that the two tc flows are also compatible with
+ * the nft entry. No need to check the pre_ct and post_ct
+ * entries as that was already done during pre_merge.
+ * The nft entry does not have a netdev or chain populated, so
+ * skip this check.
+ */
+ err = nfp_ct_merge_check(pre_ct_entry, nft_entry);
+ if (err)
+ return err;
+ err = nfp_ct_merge_check(post_ct_entry, nft_entry);
+ if (err)
+ return err;
+ err = nfp_ct_check_meta(post_ct_entry, nft_entry);
+ if (err)
+ return err;
+
+ /* Combine tc_merge and nft cookies for this cookie. */
+ new_cookie[0] = tc_m_entry->cookie[0];
+ new_cookie[1] = tc_m_entry->cookie[1];
+ new_cookie[2] = nft_entry->cookie;
+ nft_m_entry = get_hashentry(&zt->nft_merge_tb,
+ &new_cookie,
+ nfp_nft_ct_merge_params,
+ sizeof(*nft_m_entry));
+
+ if (IS_ERR(nft_m_entry))
+ return PTR_ERR(nft_m_entry);
+
+ /* nft_m_entry already present, not merging again */
+ if (!memcmp(&new_cookie, nft_m_entry->cookie, sizeof(new_cookie)))
+ return 0;
+
+ memcpy(&nft_m_entry->cookie, &new_cookie, sizeof(new_cookie));
+ nft_m_entry->zt = zt;
+ nft_m_entry->tc_m_parent = tc_m_entry;
+ nft_m_entry->nft_parent = nft_entry;
+ nft_m_entry->tc_flower_cookie = 0;
+ /* Copy the netdev from one the pre_ct entry. When the tc_m_entry was created
+ * it only combined them if the netdevs were the same, so can use any of them.
+ */
+ nft_m_entry->netdev = pre_ct_entry->netdev;
+
+ /* Add this entry to the tc_m_list and nft_flow lists */
+ list_add(&nft_m_entry->tc_merge_list, &tc_m_entry->children);
+ list_add(&nft_m_entry->nft_flow_list, &nft_entry->children);
+
+ /* Generate offload structure and send to nfp */
+ err = nfp_fl_ct_add_offload(nft_m_entry);
+ if (err)
+ goto err_nft_ct_offload;
+
+ err = rhashtable_insert_fast(&zt->nft_merge_tb, &nft_m_entry->hash_node,
+ nfp_nft_ct_merge_params);
+ if (err)
+ goto err_nft_ct_merge_insert;
+
+ zt->nft_merge_count++;
+
+ return err;
+
+err_nft_ct_merge_insert:
+ nfp_fl_ct_del_offload(zt->priv->app, nft_m_entry->tc_flower_cookie,
+ nft_m_entry->netdev);
+err_nft_ct_offload:
+ list_del(&nft_m_entry->tc_merge_list);
+ list_del(&nft_m_entry->nft_flow_list);
+ kfree(nft_m_entry);
+ return err;
+}
+
+static int nfp_ct_do_tc_merge(struct nfp_fl_ct_zone_entry *zt,
+ struct nfp_fl_ct_flow_entry *ct_entry1,
+ struct nfp_fl_ct_flow_entry *ct_entry2)
+{
+ struct nfp_fl_ct_flow_entry *post_ct_entry, *pre_ct_entry;
+ struct nfp_fl_ct_flow_entry *nft_entry, *nft_tmp;
+ struct nfp_fl_ct_tc_merge *m_entry;
+ unsigned long new_cookie[2];
+ int err;
+
+ if (ct_entry1->type == CT_TYPE_PRE_CT) {
+ pre_ct_entry = ct_entry1;
+ post_ct_entry = ct_entry2;
+ } else {
+ post_ct_entry = ct_entry1;
+ pre_ct_entry = ct_entry2;
+ }
+
+ if (post_ct_entry->netdev != pre_ct_entry->netdev)
+ return -EINVAL;
+ /* Checks that the chain_index of the filter matches the
+ * chain_index of the GOTO action.
+ */
+ if (post_ct_entry->chain_index != pre_ct_entry->chain_index)
+ return -EINVAL;
+
+ err = nfp_ct_merge_check(post_ct_entry, pre_ct_entry);
+ if (err)
+ return err;
+
+ new_cookie[0] = pre_ct_entry->cookie;
+ new_cookie[1] = post_ct_entry->cookie;
+ m_entry = get_hashentry(&zt->tc_merge_tb, &new_cookie,
+ nfp_tc_ct_merge_params, sizeof(*m_entry));
+ if (IS_ERR(m_entry))
+ return PTR_ERR(m_entry);
+
+ /* m_entry already present, not merging again */
+ if (!memcmp(&new_cookie, m_entry->cookie, sizeof(new_cookie)))
+ return 0;
+
+ memcpy(&m_entry->cookie, &new_cookie, sizeof(new_cookie));
+ m_entry->zt = zt;
+ m_entry->post_ct_parent = post_ct_entry;
+ m_entry->pre_ct_parent = pre_ct_entry;
+
+ /* Add this entry to the pre_ct and post_ct lists */
+ list_add(&m_entry->post_ct_list, &post_ct_entry->children);
+ list_add(&m_entry->pre_ct_list, &pre_ct_entry->children);
+ INIT_LIST_HEAD(&m_entry->children);
+
+ err = rhashtable_insert_fast(&zt->tc_merge_tb, &m_entry->hash_node,
+ nfp_tc_ct_merge_params);
+ if (err)
+ goto err_ct_tc_merge_insert;
+ zt->tc_merge_count++;
+
+ /* Merge with existing nft flows */
+ list_for_each_entry_safe(nft_entry, nft_tmp, &zt->nft_flows_list,
+ list_node) {
+ nfp_ct_do_nft_merge(zt, nft_entry, m_entry);
+ }
+
+ return 0;
+
+err_ct_tc_merge_insert:
+ list_del(&m_entry->post_ct_list);
+ list_del(&m_entry->pre_ct_list);
+ kfree(m_entry);
+ return err;
+}
+
+static struct
+nfp_fl_ct_zone_entry *get_nfp_zone_entry(struct nfp_flower_priv *priv,
+ u16 zone, bool wildcarded)
+{
+ struct nfp_fl_ct_zone_entry *zt;
+ int err;
+
+ if (wildcarded && priv->ct_zone_wc)
+ return priv->ct_zone_wc;
+
+ if (!wildcarded) {
+ zt = get_hashentry(&priv->ct_zone_table, &zone,
+ nfp_zone_table_params, sizeof(*zt));
+
+ /* If priv is set this is an existing entry, just return it */
+ if (IS_ERR(zt) || zt->priv)
+ return zt;
+ } else {
+ zt = kzalloc(sizeof(*zt), GFP_KERNEL);
+ if (!zt)
+ return ERR_PTR(-ENOMEM);
+ }
+
+ zt->zone = zone;
+ zt->priv = priv;
+ zt->nft = NULL;
+
+ /* init the various hash tables and lists*/
+ INIT_LIST_HEAD(&zt->pre_ct_list);
+ INIT_LIST_HEAD(&zt->post_ct_list);
+ INIT_LIST_HEAD(&zt->nft_flows_list);
+
+ err = rhashtable_init(&zt->tc_merge_tb, &nfp_tc_ct_merge_params);
+ if (err)
+ goto err_tc_merge_tb_init;
+
+ err = rhashtable_init(&zt->nft_merge_tb, &nfp_nft_ct_merge_params);
+ if (err)
+ goto err_nft_merge_tb_init;
+
+ if (wildcarded) {
+ priv->ct_zone_wc = zt;
+ } else {
+ err = rhashtable_insert_fast(&priv->ct_zone_table,
+ &zt->hash_node,
+ nfp_zone_table_params);
+ if (err)
+ goto err_zone_insert;
+ }
+
+ return zt;
+
+err_zone_insert:
+ rhashtable_destroy(&zt->nft_merge_tb);
+err_nft_merge_tb_init:
+ rhashtable_destroy(&zt->tc_merge_tb);
+err_tc_merge_tb_init:
+ kfree(zt);
+ return ERR_PTR(err);
+}
+
+static struct
+nfp_fl_ct_flow_entry *nfp_fl_ct_add_flow(struct nfp_fl_ct_zone_entry *zt,
+ struct net_device *netdev,
+ struct flow_cls_offload *flow,
+ bool is_nft, struct netlink_ext_ack *extack)
+{
+ struct nf_flow_match *nft_match = NULL;
+ struct nfp_fl_ct_flow_entry *entry;
+ struct nfp_fl_ct_map_entry *map;
+ struct flow_action_entry *act;
+ int err, i;
+
+ entry = kzalloc(sizeof(*entry), GFP_KERNEL);
+ if (!entry)
+ return ERR_PTR(-ENOMEM);
+
+ entry->rule = flow_rule_alloc(flow->rule->action.num_entries);
+ if (!entry->rule) {
+ err = -ENOMEM;
+ goto err_pre_ct_rule;
+ }
+
+ /* nft flows gets destroyed after callback return, so need
+ * to do a full copy instead of just a reference.
+ */
+ if (is_nft) {
+ nft_match = kzalloc(sizeof(*nft_match), GFP_KERNEL);
+ if (!nft_match) {
+ err = -ENOMEM;
+ goto err_pre_ct_act;
+ }
+ memcpy(&nft_match->dissector, flow->rule->match.dissector,
+ sizeof(nft_match->dissector));
+ memcpy(&nft_match->mask, flow->rule->match.mask,
+ sizeof(nft_match->mask));
+ memcpy(&nft_match->key, flow->rule->match.key,
+ sizeof(nft_match->key));
+ entry->rule->match.dissector = &nft_match->dissector;
+ entry->rule->match.mask = &nft_match->mask;
+ entry->rule->match.key = &nft_match->key;
+ } else {
+ entry->rule->match.dissector = flow->rule->match.dissector;
+ entry->rule->match.mask = flow->rule->match.mask;
+ entry->rule->match.key = flow->rule->match.key;
+ }
+
+ entry->zt = zt;
+ entry->netdev = netdev;
+ entry->cookie = flow->cookie;
+ entry->chain_index = flow->common.chain_index;
+ entry->tun_offset = NFP_FL_CT_NO_TUN;
+
+ /* Copy over action data. Unfortunately we do not get a handle to the
+ * original tcf_action data, and the flow objects gets destroyed, so we
+ * cannot just save a pointer to this either, so need to copy over the
+ * data unfortunately.
+ */
+ entry->rule->action.num_entries = flow->rule->action.num_entries;
+ flow_action_for_each(i, act, &flow->rule->action) {
+ struct flow_action_entry *new_act;
+
+ new_act = &entry->rule->action.entries[i];
+ memcpy(new_act, act, sizeof(struct flow_action_entry));
+ /* Entunnel is a special case, need to allocate and copy
+ * tunnel info.
+ */
+ if (act->id == FLOW_ACTION_TUNNEL_ENCAP) {
+ struct ip_tunnel_info *tun = act->tunnel;
+ size_t tun_size = sizeof(*tun) + tun->options_len;
+
+ new_act->tunnel = kmemdup(tun, tun_size, GFP_ATOMIC);
+ if (!new_act->tunnel) {
+ err = -ENOMEM;
+ goto err_pre_ct_tun_cp;
+ }
+ entry->tun_offset = i;
+ }
+ }
+
+ INIT_LIST_HEAD(&entry->children);
+
+ /* Now add a ct map entry to flower-priv */
+ map = get_hashentry(&zt->priv->ct_map_table, &flow->cookie,
+ nfp_ct_map_params, sizeof(*map));
+ if (IS_ERR(map)) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "offload error: ct map entry creation failed");
+ err = -ENOMEM;
+ goto err_ct_flow_insert;
+ }
+ map->cookie = flow->cookie;
+ map->ct_entry = entry;
+ err = rhashtable_insert_fast(&zt->priv->ct_map_table,
+ &map->hash_node,
+ nfp_ct_map_params);
+ if (err) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "offload error: ct map entry table add failed");
+ goto err_map_insert;
+ }
+
+ return entry;
+
+err_map_insert:
+ kfree(map);
+err_ct_flow_insert:
+ if (entry->tun_offset != NFP_FL_CT_NO_TUN)
+ kfree(entry->rule->action.entries[entry->tun_offset].tunnel);
+err_pre_ct_tun_cp:
+ kfree(nft_match);
+err_pre_ct_act:
+ kfree(entry->rule);
+err_pre_ct_rule:
+ kfree(entry);
+ return ERR_PTR(err);
+}
+
+static void cleanup_nft_merge_entry(struct nfp_fl_nft_tc_merge *m_entry)
+{
+ struct nfp_fl_ct_zone_entry *zt;
+ int err;
+
+ zt = m_entry->zt;
+
+ /* Flow is in HW, need to delete */
+ if (m_entry->tc_flower_cookie) {
+ err = nfp_fl_ct_del_offload(zt->priv->app, m_entry->tc_flower_cookie,
+ m_entry->netdev);
+ if (err)
+ return;
+ }
+
+ WARN_ON_ONCE(rhashtable_remove_fast(&zt->nft_merge_tb,
+ &m_entry->hash_node,
+ nfp_nft_ct_merge_params));
+ zt->nft_merge_count--;
+ list_del(&m_entry->tc_merge_list);
+ list_del(&m_entry->nft_flow_list);
+
+ kfree(m_entry);
+}
+
+static void nfp_free_nft_merge_children(void *entry, bool is_nft_flow)
+{
+ struct nfp_fl_nft_tc_merge *m_entry, *tmp;
+
+ /* These post entries are parts of two lists, one is a list of nft_entries
+ * and the other is of from a list of tc_merge structures. Iterate
+ * through the relevant list and cleanup the entries.
+ */
+
+ if (is_nft_flow) {
+ /* Need to iterate through list of nft_flow entries*/
+ struct nfp_fl_ct_flow_entry *ct_entry = entry;
+
+ list_for_each_entry_safe(m_entry, tmp, &ct_entry->children,
+ nft_flow_list) {
+ cleanup_nft_merge_entry(m_entry);
+ }
+ } else {
+ /* Need to iterate through list of tc_merged_flow entries*/
+ struct nfp_fl_ct_tc_merge *ct_entry = entry;
+
+ list_for_each_entry_safe(m_entry, tmp, &ct_entry->children,
+ tc_merge_list) {
+ cleanup_nft_merge_entry(m_entry);
+ }
+ }
+}
+
+static void nfp_del_tc_merge_entry(struct nfp_fl_ct_tc_merge *m_ent)
+{
+ struct nfp_fl_ct_zone_entry *zt;
+ int err;
+
+ zt = m_ent->zt;
+ err = rhashtable_remove_fast(&zt->tc_merge_tb,
+ &m_ent->hash_node,
+ nfp_tc_ct_merge_params);
+ if (err)
+ pr_warn("WARNING: could not remove merge_entry from hashtable\n");
+ zt->tc_merge_count--;
+ list_del(&m_ent->post_ct_list);
+ list_del(&m_ent->pre_ct_list);
+
+ if (!list_empty(&m_ent->children))
+ nfp_free_nft_merge_children(m_ent, false);
+ kfree(m_ent);
+}
+
+static void nfp_free_tc_merge_children(struct nfp_fl_ct_flow_entry *entry)
+{
+ struct nfp_fl_ct_tc_merge *m_ent, *tmp;
+
+ switch (entry->type) {
+ case CT_TYPE_PRE_CT:
+ list_for_each_entry_safe(m_ent, tmp, &entry->children, pre_ct_list) {
+ nfp_del_tc_merge_entry(m_ent);
+ }
+ break;
+ case CT_TYPE_POST_CT:
+ list_for_each_entry_safe(m_ent, tmp, &entry->children, post_ct_list) {
+ nfp_del_tc_merge_entry(m_ent);
+ }
+ break;
+ default:
+ break;
+ }
+}
+
+void nfp_fl_ct_clean_flow_entry(struct nfp_fl_ct_flow_entry *entry)
+{
+ list_del(&entry->list_node);
+
+ if (!list_empty(&entry->children)) {
+ if (entry->type == CT_TYPE_NFT)
+ nfp_free_nft_merge_children(entry, true);
+ else
+ nfp_free_tc_merge_children(entry);
+ }
+
+ if (entry->tun_offset != NFP_FL_CT_NO_TUN)
+ kfree(entry->rule->action.entries[entry->tun_offset].tunnel);
+
+ if (entry->type == CT_TYPE_NFT) {
+ struct nf_flow_match *nft_match;
+
+ nft_match = container_of(entry->rule->match.dissector,
+ struct nf_flow_match, dissector);
+ kfree(nft_match);
+ }
+
+ kfree(entry->rule);
+ kfree(entry);
+}
+
+static struct flow_action_entry *get_flow_act(struct flow_rule *rule,
+ enum flow_action_id act_id)
+{
+ struct flow_action_entry *act = NULL;
+ int i;
+
+ flow_action_for_each(i, act, &rule->action) {
+ if (act->id == act_id)
+ return act;
+ }
+ return NULL;
+}
+
+static void
+nfp_ct_merge_tc_entries(struct nfp_fl_ct_flow_entry *ct_entry1,
+ struct nfp_fl_ct_zone_entry *zt_src,
+ struct nfp_fl_ct_zone_entry *zt_dst)
+{
+ struct nfp_fl_ct_flow_entry *ct_entry2, *ct_tmp;
+ struct list_head *ct_list;
+
+ if (ct_entry1->type == CT_TYPE_PRE_CT)
+ ct_list = &zt_src->post_ct_list;
+ else if (ct_entry1->type == CT_TYPE_POST_CT)
+ ct_list = &zt_src->pre_ct_list;
+ else
+ return;
+
+ list_for_each_entry_safe(ct_entry2, ct_tmp, ct_list,
+ list_node) {
+ nfp_ct_do_tc_merge(zt_dst, ct_entry2, ct_entry1);
+ }
+}
+
+static void
+nfp_ct_merge_nft_with_tc(struct nfp_fl_ct_flow_entry *nft_entry,
+ struct nfp_fl_ct_zone_entry *zt)
+{
+ struct nfp_fl_ct_tc_merge *tc_merge_entry;
+ struct rhashtable_iter iter;
+
+ rhashtable_walk_enter(&zt->tc_merge_tb, &iter);
+ rhashtable_walk_start(&iter);
+ while ((tc_merge_entry = rhashtable_walk_next(&iter)) != NULL) {
+ if (IS_ERR(tc_merge_entry))
+ continue;
+ rhashtable_walk_stop(&iter);
+ nfp_ct_do_nft_merge(zt, nft_entry, tc_merge_entry);
+ rhashtable_walk_start(&iter);
+ }
+ rhashtable_walk_stop(&iter);
+ rhashtable_walk_exit(&iter);
+}
+
+int nfp_fl_ct_handle_pre_ct(struct nfp_flower_priv *priv,
+ struct net_device *netdev,
+ struct flow_cls_offload *flow,
+ struct netlink_ext_ack *extack)
+{
+ struct flow_action_entry *ct_act, *ct_goto;
+ struct nfp_fl_ct_flow_entry *ct_entry;
+ struct nfp_fl_ct_zone_entry *zt;
+ int err;
+
+ ct_act = get_flow_act(flow->rule, FLOW_ACTION_CT);
+ if (!ct_act) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "unsupported offload: Conntrack action empty in conntrack offload");
+ return -EOPNOTSUPP;
+ }
+
+ ct_goto = get_flow_act(flow->rule, FLOW_ACTION_GOTO);
+ if (!ct_goto) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "unsupported offload: Conntrack requires ACTION_GOTO");
+ return -EOPNOTSUPP;
+ }
+
+ zt = get_nfp_zone_entry(priv, ct_act->ct.zone, false);
+ if (IS_ERR(zt)) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "offload error: Could not create zone table entry");
+ return PTR_ERR(zt);
+ }
+
+ if (!zt->nft) {
+ zt->nft = ct_act->ct.flow_table;
+ err = nf_flow_table_offload_add_cb(zt->nft, nfp_fl_ct_handle_nft_flow, zt);
+ if (err) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "offload error: Could not register nft_callback");
+ return err;
+ }
+ }
+
+ /* Add entry to pre_ct_list */
+ ct_entry = nfp_fl_ct_add_flow(zt, netdev, flow, false, extack);
+ if (IS_ERR(ct_entry))
+ return PTR_ERR(ct_entry);
+ ct_entry->type = CT_TYPE_PRE_CT;
+ ct_entry->chain_index = ct_goto->chain_index;
+ list_add(&ct_entry->list_node, &zt->pre_ct_list);
+ zt->pre_ct_count++;
+
+ nfp_ct_merge_tc_entries(ct_entry, zt, zt);
+
+ /* Need to check and merge with tables in the wc_zone as well */
+ if (priv->ct_zone_wc)
+ nfp_ct_merge_tc_entries(ct_entry, priv->ct_zone_wc, zt);
+
+ return 0;
+}
+
+int nfp_fl_ct_handle_post_ct(struct nfp_flower_priv *priv,
+ struct net_device *netdev,
+ struct flow_cls_offload *flow,
+ struct netlink_ext_ack *extack)
+{
+ struct flow_rule *rule = flow_cls_offload_flow_rule(flow);
+ struct nfp_fl_ct_flow_entry *ct_entry;
+ struct nfp_fl_ct_zone_entry *zt;
+ bool wildcarded = false;
+ struct flow_match_ct ct;
+
+ flow_rule_match_ct(rule, &ct);
+ if (!ct.mask->ct_zone) {
+ wildcarded = true;
+ } else if (ct.mask->ct_zone != U16_MAX) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "unsupported offload: partially wildcarded ct_zone is not supported");
+ return -EOPNOTSUPP;
+ }
+
+ zt = get_nfp_zone_entry(priv, ct.key->ct_zone, wildcarded);
+ if (IS_ERR(zt)) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "offload error: Could not create zone table entry");
+ return PTR_ERR(zt);
+ }
+
+ /* Add entry to post_ct_list */
+ ct_entry = nfp_fl_ct_add_flow(zt, netdev, flow, false, extack);
+ if (IS_ERR(ct_entry))
+ return PTR_ERR(ct_entry);
+
+ ct_entry->type = CT_TYPE_POST_CT;
+ ct_entry->chain_index = flow->common.chain_index;
+ list_add(&ct_entry->list_node, &zt->post_ct_list);
+ zt->post_ct_count++;
+
+ if (wildcarded) {
+ /* Iterate through all zone tables if not empty, look for merges with
+ * pre_ct entries and merge them.
+ */
+ struct rhashtable_iter iter;
+ struct nfp_fl_ct_zone_entry *zone_table;
+
+ rhashtable_walk_enter(&priv->ct_zone_table, &iter);
+ rhashtable_walk_start(&iter);
+ while ((zone_table = rhashtable_walk_next(&iter)) != NULL) {
+ if (IS_ERR(zone_table))
+ continue;
+ rhashtable_walk_stop(&iter);
+ nfp_ct_merge_tc_entries(ct_entry, zone_table, zone_table);
+ rhashtable_walk_start(&iter);
+ }
+ rhashtable_walk_stop(&iter);
+ rhashtable_walk_exit(&iter);
+ } else {
+ nfp_ct_merge_tc_entries(ct_entry, zt, zt);
+ }
+
+ return 0;
+}
+
+static int
+nfp_fl_ct_offload_nft_flow(struct nfp_fl_ct_zone_entry *zt, struct flow_cls_offload *flow)
+{
+ struct nfp_fl_ct_map_entry *ct_map_ent;
+ struct nfp_fl_ct_flow_entry *ct_entry;
+ struct netlink_ext_ack *extack = NULL;
+
+ ASSERT_RTNL();
+
+ extack = flow->common.extack;
+ switch (flow->command) {
+ case FLOW_CLS_REPLACE:
+ /* Netfilter can request offload multiple times for the same
+ * flow - protect against adding duplicates.
+ */
+ ct_map_ent = rhashtable_lookup_fast(&zt->priv->ct_map_table, &flow->cookie,
+ nfp_ct_map_params);
+ if (!ct_map_ent) {
+ ct_entry = nfp_fl_ct_add_flow(zt, NULL, flow, true, extack);
+ if (IS_ERR(ct_entry))
+ return PTR_ERR(ct_entry);
+ ct_entry->type = CT_TYPE_NFT;
+ list_add(&ct_entry->list_node, &zt->nft_flows_list);
+ zt->nft_flows_count++;
+ nfp_ct_merge_nft_with_tc(ct_entry, zt);
+ }
+ return 0;
+ case FLOW_CLS_DESTROY:
+ ct_map_ent = rhashtable_lookup_fast(&zt->priv->ct_map_table, &flow->cookie,
+ nfp_ct_map_params);
+ return nfp_fl_ct_del_flow(ct_map_ent);
+ case FLOW_CLS_STATS:
+ return 0;
+ default:
+ break;
+ }
+ return -EINVAL;
+}
+
+int nfp_fl_ct_handle_nft_flow(enum tc_setup_type type, void *type_data, void *cb_priv)
+{
+ struct flow_cls_offload *flow = type_data;
+ struct nfp_fl_ct_zone_entry *zt = cb_priv;
+ int err = -EOPNOTSUPP;
+
+ switch (type) {
+ case TC_SETUP_CLSFLOWER:
+ rtnl_lock();
+ err = nfp_fl_ct_offload_nft_flow(zt, flow);
+ rtnl_unlock();
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+ return err;
+}
+
+static void
+nfp_fl_ct_clean_nft_entries(struct nfp_fl_ct_zone_entry *zt)
+{
+ struct nfp_fl_ct_flow_entry *nft_entry, *ct_tmp;
+ struct nfp_fl_ct_map_entry *ct_map_ent;
+
+ list_for_each_entry_safe(nft_entry, ct_tmp, &zt->nft_flows_list,
+ list_node) {
+ ct_map_ent = rhashtable_lookup_fast(&zt->priv->ct_map_table,
+ &nft_entry->cookie,
+ nfp_ct_map_params);
+ nfp_fl_ct_del_flow(ct_map_ent);
+ }
+}
+
+int nfp_fl_ct_del_flow(struct nfp_fl_ct_map_entry *ct_map_ent)
+{
+ struct nfp_fl_ct_flow_entry *ct_entry;
+ struct nfp_fl_ct_zone_entry *zt;
+ struct rhashtable *m_table;
+
+ if (!ct_map_ent)
+ return -ENOENT;
+
+ zt = ct_map_ent->ct_entry->zt;
+ ct_entry = ct_map_ent->ct_entry;
+ m_table = &zt->priv->ct_map_table;
+
+ switch (ct_entry->type) {
+ case CT_TYPE_PRE_CT:
+ zt->pre_ct_count--;
+ rhashtable_remove_fast(m_table, &ct_map_ent->hash_node,
+ nfp_ct_map_params);
+ nfp_fl_ct_clean_flow_entry(ct_entry);
+ kfree(ct_map_ent);
+
+ /* If this is the last pre_ct_rule it means that it is
+ * very likely that the nft table will be cleaned up next,
+ * as this happens on the removal of the last act_ct flow.
+ * However we cannot deregister the callback on the removal
+ * of the last nft flow as this runs into a deadlock situation.
+ * So deregister the callback on removal of the last pre_ct flow
+ * and remove any remaining nft flow entries. We also cannot
+ * save this state and delete the callback later since the
+ * nft table would already have been freed at that time.
+ */
+ if (!zt->pre_ct_count) {
+ nf_flow_table_offload_del_cb(zt->nft,
+ nfp_fl_ct_handle_nft_flow,
+ zt);
+ zt->nft = NULL;
+ nfp_fl_ct_clean_nft_entries(zt);
+ }
+ break;
+ case CT_TYPE_POST_CT:
+ zt->post_ct_count--;
+ rhashtable_remove_fast(m_table, &ct_map_ent->hash_node,
+ nfp_ct_map_params);
+ nfp_fl_ct_clean_flow_entry(ct_entry);
+ kfree(ct_map_ent);
+ break;
+ case CT_TYPE_NFT:
+ zt->nft_flows_count--;
+ rhashtable_remove_fast(m_table, &ct_map_ent->hash_node,
+ nfp_ct_map_params);
+ nfp_fl_ct_clean_flow_entry(ct_map_ent->ct_entry);
+ kfree(ct_map_ent);
+ default:
+ break;
+ }
+
+ return 0;
+}
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */
+/* Copyright (C) 2021 Corigine, Inc. */
+
+#ifndef __NFP_FLOWER_CONNTRACK_H__
+#define __NFP_FLOWER_CONNTRACK_H__ 1
+
+#include <net/netfilter/nf_flow_table.h>
+#include "main.h"
+
+#define NFP_FL_CT_NO_TUN 0xff
+
+#define COMPARE_UNMASKED_FIELDS(__match1, __match2, __out) \
+ do { \
+ typeof(__match1) _match1 = (__match1); \
+ typeof(__match2) _match2 = (__match2); \
+ bool *_out = (__out); \
+ int i, size = sizeof(*(_match1).key); \
+ char *k1, *m1, *k2, *m2; \
+ *_out = false; \
+ k1 = (char *)_match1.key; \
+ m1 = (char *)_match1.mask; \
+ k2 = (char *)_match2.key; \
+ m2 = (char *)_match2.mask; \
+ for (i = 0; i < size; i++) \
+ if ((k1[i] & m1[i] & m2[i]) ^ \
+ (k2[i] & m1[i] & m2[i])) { \
+ *_out = true; \
+ break; \
+ } \
+ } while (0) \
+
+extern const struct rhashtable_params nfp_zone_table_params;
+extern const struct rhashtable_params nfp_ct_map_params;
+extern const struct rhashtable_params nfp_tc_ct_merge_params;
+extern const struct rhashtable_params nfp_nft_ct_merge_params;
+
+/**
+ * struct nfp_fl_ct_zone_entry - Zone entry containing conntrack flow information
+ * @zone: The zone number, used as lookup key in hashtable
+ * @hash_node: Used by the hashtable
+ * @priv: Pointer to nfp_flower_priv data
+ * @nft: Pointer to nf_flowtable for this zone
+ *
+ * @pre_ct_list: The pre_ct_list of nfp_fl_ct_flow_entry entries
+ * @pre_ct_count: Keep count of the number of pre_ct entries
+ *
+ * @post_ct_list: The post_ct_list of nfp_fl_ct_flow_entry entries
+ * @post_ct_count: Keep count of the number of post_ct entries
+ *
+ * @tc_merge_tb: The table of merged tc flows
+ * @tc_merge_count: Keep count of the number of merged tc entries
+ *
+ * @nft_flows_list: The list of nft relatednfp_fl_ct_flow_entry entries
+ * @nft_flows_count: Keep count of the number of nft_flow entries
+ *
+ * @nft_merge_tb: The table of merged tc+nft flows
+ * @nft_merge_count: Keep count of the number of merged tc+nft entries
+ */
+struct nfp_fl_ct_zone_entry {
+ u16 zone;
+ struct rhash_head hash_node;
+
+ struct nfp_flower_priv *priv;
+ struct nf_flowtable *nft;
+
+ struct list_head pre_ct_list;
+ unsigned int pre_ct_count;
+
+ struct list_head post_ct_list;
+ unsigned int post_ct_count;
+
+ struct rhashtable tc_merge_tb;
+ unsigned int tc_merge_count;
+
+ struct list_head nft_flows_list;
+ unsigned int nft_flows_count;
+
+ struct rhashtable nft_merge_tb;
+ unsigned int nft_merge_count;
+};
+
+enum ct_entry_type {
+ CT_TYPE_PRE_CT,
+ CT_TYPE_NFT,
+ CT_TYPE_POST_CT,
+};
+
+/**
+ * struct nfp_fl_ct_flow_entry - Flow entry containing conntrack flow information
+ * @cookie: Flow cookie, same as original TC flow, used as key
+ * @list_node: Used by the list
+ * @chain_index: Chain index of the original flow
+ * @netdev: netdev structure.
+ * @type: Type of pre-entry from enum ct_entry_type
+ * @zt: Reference to the zone table this belongs to
+ * @children: List of tc_merge flows this flow forms part of
+ * @rule: Reference to the original TC flow rule
+ * @stats: Used to cache stats for updating
+ * @tun_offset: Used to indicate tunnel action offset in action list
+ */
+struct nfp_fl_ct_flow_entry {
+ unsigned long cookie;
+ struct list_head list_node;
+ u32 chain_index;
+ enum ct_entry_type type;
+ struct net_device *netdev;
+ struct nfp_fl_ct_zone_entry *zt;
+ struct list_head children;
+ struct flow_rule *rule;
+ struct flow_stats stats;
+ u8 tun_offset; // Set to NFP_FL_CT_NO_TUN if no tun
+};
+
+/**
+ * struct nfp_fl_ct_tc_merge - Merge of two flows from tc
+ * @cookie: Flow cookie, combination of pre and post ct cookies
+ * @hash_node: Used by the hashtable
+ * @pre_ct_list: This entry is part of a pre_ct_list
+ * @post_ct_list: This entry is part of a post_ct_list
+ * @zt: Reference to the zone table this belongs to
+ * @pre_ct_parent: The pre_ct_parent
+ * @post_ct_parent: The post_ct_parent
+ * @children: List of nft merged entries
+ */
+struct nfp_fl_ct_tc_merge {
+ unsigned long cookie[2];
+ struct rhash_head hash_node;
+ struct list_head pre_ct_list;
+ struct list_head post_ct_list;
+ struct nfp_fl_ct_zone_entry *zt;
+ struct nfp_fl_ct_flow_entry *pre_ct_parent;
+ struct nfp_fl_ct_flow_entry *post_ct_parent;
+ struct list_head children;
+};
+
+/**
+ * struct nfp_fl_nft_tc_merge - Merge of tc_merge flows with nft flow
+ * @netdev: Ingress netdev name
+ * @cookie: Flow cookie, combination of tc_merge and nft cookies
+ * @hash_node: Used by the hashtable
+ * @zt: Reference to the zone table this belongs to
+ * @nft_flow_list: This entry is part of a nft_flows_list
+ * @tc_merge_list: This entry is part of a ct_merge_list
+ * @tc_m_parent: The tc_merge parent
+ * @nft_parent: The nft_entry parent
+ * @tc_flower_cookie: The cookie of the flow offloaded to the nfp
+ * @flow_pay: Reference to the offloaded flow struct
+ */
+struct nfp_fl_nft_tc_merge {
+ struct net_device *netdev;
+ unsigned long cookie[3];
+ struct rhash_head hash_node;
+ struct nfp_fl_ct_zone_entry *zt;
+ struct list_head nft_flow_list;
+ struct list_head tc_merge_list;
+ struct nfp_fl_ct_tc_merge *tc_m_parent;
+ struct nfp_fl_ct_flow_entry *nft_parent;
+ unsigned long tc_flower_cookie;
+ struct nfp_fl_payload *flow_pay;
+};
+
+/**
+ * struct nfp_fl_ct_map_entry - Map between flow cookie and specific ct_flow
+ * @cookie: Flow cookie, same as original TC flow, used as key
+ * @hash_node: Used by the hashtable
+ * @ct_entry: Pointer to corresponding ct_entry
+ */
+struct nfp_fl_ct_map_entry {
+ unsigned long cookie;
+ struct rhash_head hash_node;
+ struct nfp_fl_ct_flow_entry *ct_entry;
+};
+
+bool is_pre_ct_flow(struct flow_cls_offload *flow);
+bool is_post_ct_flow(struct flow_cls_offload *flow);
+
+/**
+ * nfp_fl_ct_handle_pre_ct() - Handles -trk conntrack rules
+ * @priv: Pointer to app priv
+ * @netdev: netdev structure.
+ * @flow: TC flower classifier offload structure.
+ * @extack: Extack pointer for errors
+ *
+ * Adds a new entry to the relevant zone table and tries to
+ * merge with other +trk+est entries and offload if possible.
+ *
+ * Return: negative value on error, 0 if configured successfully.
+ */
+int nfp_fl_ct_handle_pre_ct(struct nfp_flower_priv *priv,
+ struct net_device *netdev,
+ struct flow_cls_offload *flow,
+ struct netlink_ext_ack *extack);
+/**
+ * nfp_fl_ct_handle_post_ct() - Handles +trk+est conntrack rules
+ * @priv: Pointer to app priv
+ * @netdev: netdev structure.
+ * @flow: TC flower classifier offload structure.
+ * @extack: Extack pointer for errors
+ *
+ * Adds a new entry to the relevant zone table and tries to
+ * merge with other -trk entries and offload if possible.
+ *
+ * Return: negative value on error, 0 if configured successfully.
+ */
+int nfp_fl_ct_handle_post_ct(struct nfp_flower_priv *priv,
+ struct net_device *netdev,
+ struct flow_cls_offload *flow,
+ struct netlink_ext_ack *extack);
+
+/**
+ * nfp_fl_ct_clean_flow_entry() - Free a nfp_fl_ct_flow_entry
+ * @entry: Flow entry to cleanup
+ */
+void nfp_fl_ct_clean_flow_entry(struct nfp_fl_ct_flow_entry *entry);
+
+/**
+ * nfp_fl_ct_del_flow() - Handle flow_del callbacks for conntrack
+ * @ct_map_ent: ct map entry for the flow that needs deleting
+ */
+int nfp_fl_ct_del_flow(struct nfp_fl_ct_map_entry *ct_map_ent);
+
+/**
+ * nfp_fl_ct_handle_nft_flow() - Handle flower flow callbacks for nft table
+ * @type: Type provided by callback
+ * @type_data: Callback data
+ * @cb_priv: Pointer to data provided when registering the callback, in this
+ * case it's the zone table.
+ */
+int nfp_fl_ct_handle_nft_flow(enum tc_setup_type type, void *type_data,
+ void *cb_priv);
+#endif
* @qos_stats_lock: Lock on qos stats updates
* @pre_tun_rule_cnt: Number of pre-tunnel rules offloaded
* @merge_table: Hash table to store merged flows
+ * @ct_zone_table: Hash table used to store the different zones
+ * @ct_zone_wc: Special zone entry for wildcarded zone matches
+ * @ct_map_table: Hash table used to referennce ct flows
*/
struct nfp_flower_priv {
struct nfp_app *app;
spinlock_t qos_stats_lock; /* Protect the qos stats */
int pre_tun_rule_cnt;
struct rhashtable merge_table;
+ struct rhashtable ct_zone_table;
+ struct nfp_fl_ct_zone_entry *ct_zone_wc;
+ struct rhashtable ct_map_table;
};
/**
#include <net/pkt_cls.h>
#include "cmsg.h"
+#include "conntrack.h"
#include "main.h"
#include "../nfp_app.h"
.key_len = sizeof(u64),
};
+const struct rhashtable_params nfp_zone_table_params = {
+ .head_offset = offsetof(struct nfp_fl_ct_zone_entry, hash_node),
+ .key_len = sizeof(u16),
+ .key_offset = offsetof(struct nfp_fl_ct_zone_entry, zone),
+ .automatic_shrinking = false,
+};
+
+const struct rhashtable_params nfp_ct_map_params = {
+ .head_offset = offsetof(struct nfp_fl_ct_map_entry, hash_node),
+ .key_len = sizeof(unsigned long),
+ .key_offset = offsetof(struct nfp_fl_ct_map_entry, cookie),
+ .automatic_shrinking = true,
+};
+
int nfp_flower_metadata_init(struct nfp_app *app, u64 host_ctx_count,
unsigned int host_num_mems)
{
if (err)
goto err_free_stats_ctx_table;
+ err = rhashtable_init(&priv->ct_zone_table, &nfp_zone_table_params);
+ if (err)
+ goto err_free_merge_table;
+
+ err = rhashtable_init(&priv->ct_map_table, &nfp_ct_map_params);
+ if (err)
+ goto err_free_ct_zone_table;
+
get_random_bytes(&priv->mask_id_seed, sizeof(priv->mask_id_seed));
/* Init ring buffer and unallocated mask_ids. */
kmalloc_array(NFP_FLOWER_MASK_ENTRY_RS,
NFP_FLOWER_MASK_ELEMENT_RS, GFP_KERNEL);
if (!priv->mask_ids.mask_id_free_list.buf)
- goto err_free_merge_table;
+ goto err_free_ct_map_table;
priv->mask_ids.init_unallocated = NFP_FLOWER_MASK_ENTRY_RS - 1;
kfree(priv->mask_ids.last_used);
err_free_mask_id:
kfree(priv->mask_ids.mask_id_free_list.buf);
+err_free_ct_map_table:
+ rhashtable_destroy(&priv->ct_map_table);
+err_free_ct_zone_table:
+ rhashtable_destroy(&priv->ct_zone_table);
err_free_merge_table:
rhashtable_destroy(&priv->merge_table);
err_free_stats_ctx_table:
return -ENOMEM;
}
+static void nfp_zone_table_entry_destroy(struct nfp_fl_ct_zone_entry *zt)
+{
+ if (!zt)
+ return;
+
+ if (!list_empty(&zt->pre_ct_list)) {
+ struct rhashtable *m_table = &zt->priv->ct_map_table;
+ struct nfp_fl_ct_flow_entry *entry, *tmp;
+ struct nfp_fl_ct_map_entry *map;
+
+ WARN_ONCE(1, "pre_ct_list not empty as expected, cleaning up\n");
+ list_for_each_entry_safe(entry, tmp, &zt->pre_ct_list,
+ list_node) {
+ map = rhashtable_lookup_fast(m_table,
+ &entry->cookie,
+ nfp_ct_map_params);
+ WARN_ON_ONCE(rhashtable_remove_fast(m_table,
+ &map->hash_node,
+ nfp_ct_map_params));
+ nfp_fl_ct_clean_flow_entry(entry);
+ kfree(map);
+ }
+ }
+
+ if (!list_empty(&zt->post_ct_list)) {
+ struct rhashtable *m_table = &zt->priv->ct_map_table;
+ struct nfp_fl_ct_flow_entry *entry, *tmp;
+ struct nfp_fl_ct_map_entry *map;
+
+ WARN_ONCE(1, "post_ct_list not empty as expected, cleaning up\n");
+ list_for_each_entry_safe(entry, tmp, &zt->post_ct_list,
+ list_node) {
+ map = rhashtable_lookup_fast(m_table,
+ &entry->cookie,
+ nfp_ct_map_params);
+ WARN_ON_ONCE(rhashtable_remove_fast(m_table,
+ &map->hash_node,
+ nfp_ct_map_params));
+ nfp_fl_ct_clean_flow_entry(entry);
+ kfree(map);
+ }
+ }
+
+ if (zt->nft) {
+ nf_flow_table_offload_del_cb(zt->nft,
+ nfp_fl_ct_handle_nft_flow,
+ zt);
+ zt->nft = NULL;
+ }
+
+ if (!list_empty(&zt->nft_flows_list)) {
+ struct rhashtable *m_table = &zt->priv->ct_map_table;
+ struct nfp_fl_ct_flow_entry *entry, *tmp;
+ struct nfp_fl_ct_map_entry *map;
+
+ WARN_ONCE(1, "nft_flows_list not empty as expected, cleaning up\n");
+ list_for_each_entry_safe(entry, tmp, &zt->nft_flows_list,
+ list_node) {
+ map = rhashtable_lookup_fast(m_table,
+ &entry->cookie,
+ nfp_ct_map_params);
+ WARN_ON_ONCE(rhashtable_remove_fast(m_table,
+ &map->hash_node,
+ nfp_ct_map_params));
+ nfp_fl_ct_clean_flow_entry(entry);
+ kfree(map);
+ }
+ }
+
+ rhashtable_free_and_destroy(&zt->tc_merge_tb,
+ nfp_check_rhashtable_empty, NULL);
+ rhashtable_free_and_destroy(&zt->nft_merge_tb,
+ nfp_check_rhashtable_empty, NULL);
+
+ kfree(zt);
+}
+
+static void nfp_free_zone_table_entry(void *ptr, void *arg)
+{
+ struct nfp_fl_ct_zone_entry *zt = ptr;
+
+ nfp_zone_table_entry_destroy(zt);
+}
+
+static void nfp_free_map_table_entry(void *ptr, void *arg)
+{
+ struct nfp_fl_ct_map_entry *map = ptr;
+
+ if (!map)
+ return;
+
+ kfree(map);
+}
+
void nfp_flower_metadata_cleanup(struct nfp_app *app)
{
struct nfp_flower_priv *priv = app->priv;
nfp_check_rhashtable_empty, NULL);
rhashtable_free_and_destroy(&priv->merge_table,
nfp_check_rhashtable_empty, NULL);
+ rhashtable_free_and_destroy(&priv->ct_zone_table,
+ nfp_free_zone_table_entry, NULL);
+ nfp_zone_table_entry_destroy(priv->ct_zone_wc);
+
+ rhashtable_free_and_destroy(&priv->ct_map_table,
+ nfp_free_map_table_entry, NULL);
kvfree(priv->stats);
kfree(priv->mask_ids.mask_id_free_list.buf);
kfree(priv->mask_ids.last_used);
#include "cmsg.h"
#include "main.h"
+#include "conntrack.h"
#include "../nfpcore/nfp_cpp.h"
#include "../nfpcore/nfp_nsp.h"
#include "../nfp_app.h"
return 0;
}
+static bool offload_pre_check(struct flow_cls_offload *flow)
+{
+ struct flow_rule *rule = flow_cls_offload_flow_rule(flow);
+ struct flow_dissector *dissector = rule->match.dissector;
+
+ if (dissector->used_keys & BIT(FLOW_DISSECTOR_KEY_CT))
+ return false;
+
+ if (flow->common.chain_index)
+ return false;
+
+ return true;
+}
+
/**
* nfp_flower_add_offload() - Adds a new flow to hardware.
* @app: Pointer to the APP handle
if (nfp_netdev_is_nfp_repr(netdev))
port = nfp_port_from_netdev(netdev);
+ if (is_pre_ct_flow(flow))
+ return nfp_fl_ct_handle_pre_ct(priv, netdev, flow, extack);
+
+ if (is_post_ct_flow(flow))
+ return nfp_fl_ct_handle_post_ct(priv, netdev, flow, extack);
+
+ if (!offload_pre_check(flow))
+ return -EOPNOTSUPP;
+
key_layer = kmalloc(sizeof(*key_layer), GFP_KERNEL);
if (!key_layer)
return -ENOMEM;
struct flow_cls_offload *flow)
{
struct nfp_flower_priv *priv = app->priv;
+ struct nfp_fl_ct_map_entry *ct_map_ent;
struct netlink_ext_ack *extack = NULL;
struct nfp_fl_payload *nfp_flow;
struct nfp_port *port = NULL;
if (nfp_netdev_is_nfp_repr(netdev))
port = nfp_port_from_netdev(netdev);
+ /* Check ct_map_table */
+ ct_map_ent = rhashtable_lookup_fast(&priv->ct_map_table, &flow->cookie,
+ nfp_ct_map_params);
+ if (ct_map_ent) {
+ err = nfp_fl_ct_del_flow(ct_map_ent);
+ return err;
+ }
+
nfp_flow = nfp_flower_search_fl_table(app, flow->cookie, netdev);
if (!nfp_flow) {
NL_SET_ERR_MSG_MOD(extack, "invalid entry: cannot remove flow that does not exist");
static int nfp_flower_setup_tc_block_cb(enum tc_setup_type type,
void *type_data, void *cb_priv)
{
+ struct flow_cls_common_offload *common = type_data;
struct nfp_repr *repr = cb_priv;
- if (!tc_cls_can_offload_and_chain0(repr->netdev, type_data))
+ if (!tc_can_offload_extack(repr->netdev, common->extack))
return -EOPNOTSUPP;
switch (type) {
void *type_data, void *cb_priv)
{
struct nfp_flower_indr_block_cb_priv *priv = cb_priv;
- struct flow_cls_offload *flower = type_data;
-
- if (flower->common.chain_index)
- return -EOPNOTSUPP;
switch (type) {
case TC_SETUP_CLSFLOWER:
#define NFP_TUN_PRE_TUN_IPV6_BIT BIT(7)
/**
- * struct nfp_tun_pre_run_rule - rule matched before decap
+ * struct nfp_tun_pre_tun_rule - rule matched before decap
* @flags: options for the rule offset
* @port_idx: index of destination MAC address for the rule
* @vlan_tci: VLAN info associated with MAC
return;
/* Move to front of LRU */
- list_del(&cache->entry);
- list_add(&cache->entry, &cpp->area_cache_list);
+ list_move(&cache->entry, &cpp->area_cache_list);
mutex_unlock(&cpp->area_cache_mutex);
}
#define NFFW_FWID_ALL 255
-/**
+/*
* NFFW_INFO_VERSION history:
* 0: This was never actually used (before versioning), but it refers to
* the previous struct which had FWINFO_CNT = MEINFO_CNT = 120 that later
const struct of_device_id *of_id;
enum nixge_version version;
struct resource *ctrlres;
- struct resource *dmares;
struct net_device *ndev;
struct nixge_priv *priv;
version = (enum nixge_version)of_id->data;
if (version <= NIXGE_V2)
- dmares = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ priv->dma_regs = devm_platform_get_and_ioremap_resource(pdev, 0, NULL);
else
- dmares = platform_get_resource_byname(pdev, IORESOURCE_MEM,
- "dma");
-
- priv->dma_regs = devm_ioremap_resource(&pdev->dev, dmares);
+ priv->dma_regs = devm_platform_ioremap_resource_byname(pdev, "dma");
if (IS_ERR(priv->dma_regs)) {
netdev_err(ndev, "failed to map dma regs\n");
return PTR_ERR(priv->dma_regs);
config IONIC
tristate "Pensando Ethernet IONIC Support"
depends on 64BIT && PCI
+ depends on PTP_1588_CLOCK || !PTP_1588_CLOCK
select NET_DEVLINK
select DIMLIB
help
config QED_ISCSI
bool
+config QED_NVMETCP
+ bool
+
config QED_FCOE
bool
free_netdev(netdev);
err_out_free_res:
+ if (NX_IS_REVISION_P3(pdev->revision))
+ pci_disable_pcie_error_reporting(pdev);
pci_release_regions(pdev);
err_out_disable_pdev:
qed-$(CONFIG_QED_LL2) += qed_ll2.o
qed-$(CONFIG_QED_OOO) += qed_ooo.o
+qed-$(CONFIG_QED_NVMETCP) += \
+ qed_nvmetcp.o \
+ qed_nvmetcp_fw_funcs.o \
+ qed_nvmetcp_ip_services.o
+
qed-$(CONFIG_QED_RDMA) += \
qed_iwarp.o \
qed_rdma.o \
#define QED_MIN_WIDS (4)
#define QED_PF_DEMS_SIZE (4)
+#define QED_LLH_DONT_CARE 0
+
/* cau states */
enum qed_coalescing_mode {
QED_COAL_MODE_DISABLE,
QED_PCI_ETH,
QED_PCI_FCOE,
QED_PCI_ISCSI,
+ QED_PCI_NVMETCP,
QED_PCI_ETH_ROCE,
QED_PCI_ETH_IWARP,
QED_PCI_ETH_RDMA,
QED_PF_L2_QUE,
QED_VF,
QED_RDMA_CNQ,
+ QED_NVMETCP_CQ,
QED_ISCSI_CQ,
QED_FCOE_CQ,
QED_VF_L2_QUE,
((dev)->hw_info.personality == QED_PCI_FCOE)
#define QED_IS_ISCSI_PERSONALITY(dev) \
((dev)->hw_info.personality == QED_PCI_ISCSI)
+#define QED_IS_NVMETCP_PERSONALITY(dev) \
+ ((dev)->hw_info.personality == QED_PCI_NVMETCP)
/* Resource Allocation scheme results */
u32 resc_start[QED_MAX_RESC];
struct qed_ooo_info *p_ooo_info;
struct qed_rdma_info *p_rdma_info;
struct qed_iscsi_info *p_iscsi_info;
+ struct qed_nvmetcp_info *p_nvmetcp_info;
struct qed_fcoe_info *p_fcoe_info;
struct qed_pf_params pf_params;
struct qed_eth_cb_ops *eth;
struct qed_fcoe_cb_ops *fcoe;
struct qed_iscsi_cb_ops *iscsi;
+ struct qed_nvmetcp_cb_ops *nvmetcp;
} protocol_ops;
void *ops_cookie;
void qed_hw_info_set_offload_tc(struct qed_hw_info *p_info, u8 tc);
void qed_periodic_db_rec_start(struct qed_hwfn *p_hwfn);
+
+int qed_llh_add_src_tcp_port_filter(struct qed_dev *cdev, u16 src_port);
+int qed_llh_add_dst_tcp_port_filter(struct qed_dev *cdev, u16 dest_port);
+void qed_llh_remove_src_tcp_port_filter(struct qed_dev *cdev, u16 src_port);
+void qed_llh_remove_dst_tcp_port_filter(struct qed_dev *cdev, u16 src_port);
+void qed_llh_clear_all_filters(struct qed_dev *cdev);
#endif /* _QED_H */
static bool src_proto(enum protocol_type type)
{
- return type == PROTOCOLID_ISCSI ||
+ return type == PROTOCOLID_TCP_ULP ||
type == PROTOCOLID_FCOE ||
type == PROTOCOLID_IWARP;
}
static bool tm_cid_proto(enum protocol_type type)
{
- return type == PROTOCOLID_ISCSI ||
+ return type == PROTOCOLID_TCP_ULP ||
type == PROTOCOLID_FCOE ||
type == PROTOCOLID_ROCE ||
type == PROTOCOLID_IWARP;
PROTOCOLID_FCOE,
p_params->num_cons,
0);
-
qed_cxt_set_proto_tid_count(p_hwfn, PROTOCOLID_FCOE,
QED_CXT_FCOE_TID_SEG, 0,
p_params->num_tasks, true);
if (p_params->num_cons && p_params->num_tasks) {
qed_cxt_set_proto_cid_count(p_hwfn,
- PROTOCOLID_ISCSI,
+ PROTOCOLID_TCP_ULP,
p_params->num_cons,
0);
-
qed_cxt_set_proto_tid_count(p_hwfn,
- PROTOCOLID_ISCSI,
- QED_CXT_ISCSI_TID_SEG,
+ PROTOCOLID_TCP_ULP,
+ QED_CXT_TCP_ULP_TID_SEG,
0,
p_params->num_tasks,
true);
}
break;
}
+ case QED_PCI_NVMETCP:
+ {
+ struct qed_nvmetcp_pf_params *p_params;
+
+ p_params = &p_hwfn->pf_params.nvmetcp_pf_params;
+
+ if (p_params->num_cons && p_params->num_tasks) {
+ qed_cxt_set_proto_cid_count(p_hwfn,
+ PROTOCOLID_TCP_ULP,
+ p_params->num_cons,
+ 0);
+ qed_cxt_set_proto_tid_count(p_hwfn,
+ PROTOCOLID_TCP_ULP,
+ QED_CXT_TCP_ULP_TID_SEG,
+ 0,
+ p_params->num_tasks,
+ true);
+ } else {
+ DP_INFO(p_hwfn->cdev,
+ "NvmeTCP personality used without setting params!\n");
+ }
+ break;
+ }
default:
return -EINVAL;
}
seg = QED_CXT_FCOE_TID_SEG;
break;
case QED_PCI_ISCSI:
- proto = PROTOCOLID_ISCSI;
- seg = QED_CXT_ISCSI_TID_SEG;
+ case QED_PCI_NVMETCP:
+ proto = PROTOCOLID_TCP_ULP;
+ seg = QED_CXT_TCP_ULP_TID_SEG;
break;
default:
return -EINVAL;
seg = QED_CXT_FCOE_TID_SEG;
break;
case QED_PCI_ISCSI:
- proto = PROTOCOLID_ISCSI;
- seg = QED_CXT_ISCSI_TID_SEG;
+ case QED_PCI_NVMETCP:
+ proto = PROTOCOLID_TCP_ULP;
+ seg = QED_CXT_TCP_ULP_TID_SEG;
break;
default:
return -EINVAL;
int qed_cxt_get_tid_mem_info(struct qed_hwfn *p_hwfn,
struct qed_tid_mem *p_info);
-#define QED_CXT_ISCSI_TID_SEG PROTOCOLID_ISCSI
+#define QED_CXT_TCP_ULP_TID_SEG PROTOCOLID_TCP_ULP
#define QED_CXT_ROCE_TID_SEG PROTOCOLID_ROCE
#define QED_CXT_FCOE_TID_SEG PROTOCOLID_FCOE
enum qed_cxt_elem_type {
p_hwfn->p_dcbx_info->set.ver_num |= DCBX_CONFIG_VERSION_STATIC;
p_hwfn->p_dcbx_info->set.enabled = dcbx_info->operational.enabled;
+ BUILD_BUG_ON(sizeof(dcbx_info->operational.params) !=
+ sizeof(p_hwfn->p_dcbx_info->set.config.params));
memcpy(&p_hwfn->p_dcbx_info->set.config.params,
&dcbx_info->operational.params,
- sizeof(struct qed_dcbx_admin_params));
+ sizeof(p_hwfn->p_dcbx_info->set.config.params));
p_hwfn->p_dcbx_info->set.config.valid = true;
memcpy(params, &p_hwfn->p_dcbx_info->set, sizeof(struct qed_dcbx_set));
#include "qed_sriov.h"
#include "qed_vf.h"
#include "qed_rdma.h"
+#include "qed_nvmetcp.h"
static DEFINE_SPINLOCK(qm_lock);
}
/* Storage PF is bound to a single engine while L2 PF uses both */
- if (QED_IS_FCOE_PERSONALITY(p_hwfn) || QED_IS_ISCSI_PERSONALITY(p_hwfn))
+ if (QED_IS_FCOE_PERSONALITY(p_hwfn) || QED_IS_ISCSI_PERSONALITY(p_hwfn) ||
+ QED_IS_NVMETCP_PERSONALITY(p_hwfn))
eng = cdev->fir_affin ? QED_ENG1 : QED_ENG0;
else /* L2_PERSONALITY */
eng = QED_BOTH_ENG;
if (!test_bit(QED_MF_LLH_MAC_CLSS, &cdev->mf_bits))
goto out;
+ if (QED_IS_NVMETCP_PERSONALITY(p_hwfn))
+ return;
+
ether_addr_copy(filter.mac.addr, mac_addr);
rc = qed_llh_shadow_remove_filter(cdev, ppfid, &filter, &filter_idx,
&ref_cnt);
qed_ooo_free(p_hwfn);
}
+ if (p_hwfn->hw_info.personality == QED_PCI_NVMETCP) {
+ qed_nvmetcp_free(p_hwfn);
+ qed_ooo_free(p_hwfn);
+ }
+
if (QED_IS_RDMA_PERSONALITY(p_hwfn) && rdma_info) {
qed_spq_unregister_async_cb(p_hwfn, rdma_info->proto);
qed_rdma_info_free(p_hwfn);
flags |= PQ_FLAGS_OFLD;
break;
case QED_PCI_ISCSI:
+ case QED_PCI_NVMETCP:
flags |= PQ_FLAGS_ACK | PQ_FLAGS_OOO | PQ_FLAGS_OFLD;
break;
case QED_PCI_ETH_ROCE:
* at the same time
*/
n_eqes += num_cons + 2 * MAX_NUM_VFS_BB + n_srq;
- } else if (p_hwfn->hw_info.personality == QED_PCI_ISCSI) {
+ } else if (p_hwfn->hw_info.personality == QED_PCI_ISCSI ||
+ p_hwfn->hw_info.personality == QED_PCI_NVMETCP) {
num_cons =
qed_cxt_get_proto_cid_count(p_hwfn,
- PROTOCOLID_ISCSI,
+ PROTOCOLID_TCP_ULP,
NULL);
n_eqes += 2 * num_cons;
}
goto alloc_err;
}
+ if (p_hwfn->hw_info.personality == QED_PCI_NVMETCP) {
+ rc = qed_nvmetcp_alloc(p_hwfn);
+ if (rc)
+ goto alloc_err;
+ rc = qed_ooo_alloc(p_hwfn);
+ if (rc)
+ goto alloc_err;
+ }
+
if (QED_IS_RDMA_PERSONALITY(p_hwfn)) {
rc = qed_rdma_info_alloc(p_hwfn);
if (rc)
qed_iscsi_setup(p_hwfn);
qed_ooo_setup(p_hwfn);
}
+
+ if (p_hwfn->hw_info.personality == QED_PCI_NVMETCP) {
+ qed_nvmetcp_setup(p_hwfn);
+ qed_ooo_setup(p_hwfn);
+ }
}
}
/* Protocol Configuration */
STORE_RT_REG(p_hwfn, PRS_REG_SEARCH_TCP_RT_OFFSET,
- (p_hwfn->hw_info.personality == QED_PCI_ISCSI) ? 1 : 0);
+ ((p_hwfn->hw_info.personality == QED_PCI_ISCSI) ||
+ (p_hwfn->hw_info.personality == QED_PCI_NVMETCP)) ? 1 : 0);
STORE_RT_REG(p_hwfn, PRS_REG_SEARCH_FCOE_RT_OFFSET,
(p_hwfn->hw_info.personality == QED_PCI_FCOE) ? 1 : 0);
STORE_RT_REG(p_hwfn, PRS_REG_SEARCH_ROCE_RT_OFFSET, 0);
feat_num[QED_ISCSI_CQ] = min_t(u32, sb_cnt.cnt,
RESC_NUM(p_hwfn,
QED_CMDQS_CQS));
+
+ if (QED_IS_NVMETCP_PERSONALITY(p_hwfn))
+ feat_num[QED_NVMETCP_CQ] = min_t(u32, sb_cnt.cnt,
+ RESC_NUM(p_hwfn,
+ QED_CMDQS_CQS));
+
DP_VERBOSE(p_hwfn,
NETIF_MSG_PROBE,
- "#PF_L2_QUEUES=%d VF_L2_QUEUES=%d #ROCE_CNQ=%d FCOE_CQ=%d ISCSI_CQ=%d #SBS=%d\n",
+ "#PF_L2_QUEUES=%d VF_L2_QUEUES=%d #ROCE_CNQ=%d FCOE_CQ=%d ISCSI_CQ=%d NVMETCP_CQ=%d #SBS=%d\n",
(int)FEAT_NUM(p_hwfn, QED_PF_L2_QUE),
(int)FEAT_NUM(p_hwfn, QED_VF_L2_QUE),
(int)FEAT_NUM(p_hwfn, QED_RDMA_CNQ),
(int)FEAT_NUM(p_hwfn, QED_FCOE_CQ),
(int)FEAT_NUM(p_hwfn, QED_ISCSI_CQ),
+ (int)FEAT_NUM(p_hwfn, QED_NVMETCP_CQ),
(int)sb_cnt.cnt);
}
break;
case QED_BDQ:
if (p_hwfn->hw_info.personality != QED_PCI_ISCSI &&
- p_hwfn->hw_info.personality != QED_PCI_FCOE)
+ p_hwfn->hw_info.personality != QED_PCI_FCOE &&
+ p_hwfn->hw_info.personality != QED_PCI_NVMETCP)
*p_resc_num = 0;
else
*p_resc_num = 1;
*p_resc_start = 0;
else if (p_hwfn->cdev->num_ports_in_engine == 4)
*p_resc_start = p_hwfn->port_id;
- else if (p_hwfn->hw_info.personality == QED_PCI_ISCSI)
+ else if (p_hwfn->hw_info.personality == QED_PCI_ISCSI ||
+ p_hwfn->hw_info.personality == QED_PCI_NVMETCP)
*p_resc_start = p_hwfn->port_id;
else if (p_hwfn->hw_info.personality == QED_PCI_FCOE)
*p_resc_start = p_hwfn->port_id + 2;
((u8 *)fw_lsb)[0] = mac[5];
((u8 *)fw_lsb)[1] = mac[4];
}
+
+static int qed_llh_shadow_remove_all_filters(struct qed_dev *cdev, u8 ppfid)
+{
+ struct qed_llh_info *p_llh_info = cdev->p_llh_info;
+ struct qed_llh_filter_info *p_filters;
+ int rc;
+
+ rc = qed_llh_shadow_sanity(cdev, ppfid, 0, "remove_all");
+ if (rc)
+ return rc;
+
+ p_filters = p_llh_info->pp_filters[ppfid];
+ memset(p_filters, 0, NIG_REG_LLH_FUNC_FILTER_EN_SIZE *
+ sizeof(*p_filters));
+
+ return 0;
+}
+
+static void qed_llh_clear_ppfid_filters(struct qed_dev *cdev, u8 ppfid)
+{
+ struct qed_hwfn *p_hwfn = QED_LEADING_HWFN(cdev);
+ struct qed_ptt *p_ptt = qed_ptt_acquire(p_hwfn);
+ u8 filter_idx, abs_ppfid;
+ int rc = 0;
+
+ if (!p_ptt)
+ return;
+
+ if (!test_bit(QED_MF_LLH_PROTO_CLSS, &cdev->mf_bits) &&
+ !test_bit(QED_MF_LLH_MAC_CLSS, &cdev->mf_bits))
+ goto out;
+
+ rc = qed_llh_abs_ppfid(cdev, ppfid, &abs_ppfid);
+ if (rc)
+ goto out;
+
+ rc = qed_llh_shadow_remove_all_filters(cdev, ppfid);
+ if (rc)
+ goto out;
+
+ for (filter_idx = 0; filter_idx < NIG_REG_LLH_FUNC_FILTER_EN_SIZE;
+ filter_idx++) {
+ rc = qed_llh_remove_filter(p_hwfn, p_ptt,
+ abs_ppfid, filter_idx);
+ if (rc)
+ goto out;
+ }
+out:
+ qed_ptt_release(p_hwfn, p_ptt);
+}
+
+int qed_llh_add_src_tcp_port_filter(struct qed_dev *cdev, u16 src_port)
+{
+ return qed_llh_add_protocol_filter(cdev, 0,
+ QED_LLH_FILTER_TCP_SRC_PORT,
+ src_port, QED_LLH_DONT_CARE);
+}
+
+void qed_llh_remove_src_tcp_port_filter(struct qed_dev *cdev, u16 src_port)
+{
+ qed_llh_remove_protocol_filter(cdev, 0,
+ QED_LLH_FILTER_TCP_SRC_PORT,
+ src_port, QED_LLH_DONT_CARE);
+}
+
+int qed_llh_add_dst_tcp_port_filter(struct qed_dev *cdev, u16 dest_port)
+{
+ return qed_llh_add_protocol_filter(cdev, 0,
+ QED_LLH_FILTER_TCP_DEST_PORT,
+ QED_LLH_DONT_CARE, dest_port);
+}
+
+void qed_llh_remove_dst_tcp_port_filter(struct qed_dev *cdev, u16 dest_port)
+{
+ qed_llh_remove_protocol_filter(cdev, 0,
+ QED_LLH_FILTER_TCP_DEST_PORT,
+ QED_LLH_DONT_CARE, dest_port);
+}
+
+void qed_llh_clear_all_filters(struct qed_dev *cdev)
+{
+ u8 ppfid;
+
+ if (!test_bit(QED_MF_LLH_PROTO_CLSS, &cdev->mf_bits) &&
+ !test_bit(QED_MF_LLH_MAC_CLSS, &cdev->mf_bits))
+ return;
+
+ for (ppfid = 0; ppfid < cdev->p_llh_info->num_ppfid; ppfid++)
+ qed_llh_clear_ppfid_filters(cdev, ppfid);
+}
#include <linux/qed/fcoe_common.h>
#include <linux/qed/eth_common.h>
#include <linux/qed/iscsi_common.h>
+#include <linux/qed/nvmetcp_common.h>
#include <linux/qed/iwarp_common.h>
#include <linux/qed/rdma_common.h>
#include <linux/qed/roce_common.h>
/* personality per PF */
enum personality_type {
BAD_PERSONALITY_TYP,
- PERSONALITY_ISCSI,
+ PERSONALITY_TCP_ULP,
PERSONALITY_FCOE,
PERSONALITY_RDMA_AND_ETH,
PERSONALITY_RDMA,
#define FUNC_MF_CFG_PROTOCOL_ISCSI 0x00000010
#define FUNC_MF_CFG_PROTOCOL_FCOE 0x00000020
#define FUNC_MF_CFG_PROTOCOL_ROCE 0x00000030
-#define FUNC_MF_CFG_PROTOCOL_MAX 0x00000030
+#define FUNC_MF_CFG_PROTOCOL_NVMETCP 0x00000040
+#define FUNC_MF_CFG_PROTOCOL_MAX 0x00000040
#define FUNC_MF_CFG_MIN_BW_MASK 0x0000ff00
#define FUNC_MF_CFG_MIN_BW_SHIFT 8
rc = qed_sp_init_request(p_hwfn, &p_ent,
ISCSI_RAMROD_CMD_ID_INIT_FUNC,
- PROTOCOLID_ISCSI, &init_data);
+ PROTOCOLID_TCP_ULP, &init_data);
if (rc)
return rc;
p_hwfn->p_iscsi_info->event_context = event_context;
p_hwfn->p_iscsi_info->event_cb = async_event_cb;
- qed_spq_register_async_cb(p_hwfn, PROTOCOLID_ISCSI,
+ qed_spq_register_async_cb(p_hwfn, PROTOCOLID_TCP_ULP,
qed_iscsi_async_event);
return qed_spq_post(p_hwfn, p_ent, NULL);
rc = qed_sp_init_request(p_hwfn, &p_ent,
ISCSI_RAMROD_CMD_ID_OFFLOAD_CONN,
- PROTOCOLID_ISCSI, &init_data);
+ PROTOCOLID_TCP_ULP, &init_data);
if (rc)
return rc;
rc = qed_sp_init_request(p_hwfn, &p_ent,
ISCSI_RAMROD_CMD_ID_UPDATE_CONN,
- PROTOCOLID_ISCSI, &init_data);
+ PROTOCOLID_TCP_ULP, &init_data);
if (rc)
return rc;
rc = qed_sp_init_request(p_hwfn, &p_ent,
ISCSI_RAMROD_CMD_ID_MAC_UPDATE,
- PROTOCOLID_ISCSI, &init_data);
+ PROTOCOLID_TCP_ULP, &init_data);
if (rc)
return rc;
rc = qed_sp_init_request(p_hwfn, &p_ent,
ISCSI_RAMROD_CMD_ID_TERMINATION_CONN,
- PROTOCOLID_ISCSI, &init_data);
+ PROTOCOLID_TCP_ULP, &init_data);
if (rc)
return rc;
rc = qed_sp_init_request(p_hwfn, &p_ent,
ISCSI_RAMROD_CMD_ID_CLEAR_SQ,
- PROTOCOLID_ISCSI, &init_data);
+ PROTOCOLID_TCP_ULP, &init_data);
if (rc)
return rc;
rc = qed_sp_init_request(p_hwfn, &p_ent,
ISCSI_RAMROD_CMD_ID_DESTROY_FUNC,
- PROTOCOLID_ISCSI, &init_data);
+ PROTOCOLID_TCP_ULP, &init_data);
if (rc)
return rc;
rc = qed_spq_post(p_hwfn, p_ent, NULL);
- qed_spq_unregister_async_cb(p_hwfn, PROTOCOLID_ISCSI);
+ qed_spq_unregister_async_cb(p_hwfn, PROTOCOLID_TCP_ULP);
return rc;
}
u32 icid;
spin_lock_bh(&p_hwfn->p_iscsi_info->lock);
- rc = qed_cxt_acquire_cid(p_hwfn, PROTOCOLID_ISCSI, &icid);
+ rc = qed_cxt_acquire_cid(p_hwfn, PROTOCOLID_TCP_ULP, &icid);
spin_unlock_bh(&p_hwfn->p_iscsi_info->lock);
if (rc)
return rc;
if (test_bit(QED_MF_LL2_NON_UNICAST, &p_hwfn->cdev->mf_bits) &&
p_ramrod->main_func_queue && conn_type != QED_LL2_TYPE_ROCE &&
- conn_type != QED_LL2_TYPE_IWARP) {
+ conn_type != QED_LL2_TYPE_IWARP &&
+ (!QED_IS_NVMETCP_PERSONALITY(p_hwfn))) {
p_ramrod->mf_si_bcast_accept_all = 1;
p_ramrod->mf_si_mcast_accept_all = 1;
} else {
case QED_LL2_TYPE_FCOE:
p_ramrod->conn_type = PROTOCOLID_FCOE;
break;
- case QED_LL2_TYPE_ISCSI:
- p_ramrod->conn_type = PROTOCOLID_ISCSI;
+ case QED_LL2_TYPE_TCP_ULP:
+ p_ramrod->conn_type = PROTOCOLID_TCP_ULP;
break;
case QED_LL2_TYPE_ROCE:
p_ramrod->conn_type = PROTOCOLID_ROCE;
p_ramrod->conn_type = PROTOCOLID_IWARP;
break;
case QED_LL2_TYPE_OOO:
- if (p_hwfn->hw_info.personality == QED_PCI_ISCSI)
- p_ramrod->conn_type = PROTOCOLID_ISCSI;
+ if (p_hwfn->hw_info.personality == QED_PCI_ISCSI ||
+ p_hwfn->hw_info.personality == QED_PCI_NVMETCP)
+ p_ramrod->conn_type = PROTOCOLID_TCP_ULP;
else
p_ramrod->conn_type = PROTOCOLID_IWARP;
break;
if (rc)
goto out;
- if (!QED_IS_RDMA_PERSONALITY(p_hwfn))
+ if (!QED_IS_RDMA_PERSONALITY(p_hwfn) &&
+ !QED_IS_NVMETCP_PERSONALITY(p_hwfn))
qed_wr(p_hwfn, p_ptt, PRS_REG_USE_LIGHT_L2, 1);
qed_ll2_establish_connection_ooo(p_hwfn, p_ll2_conn);
static bool qed_ll2_is_storage_eng1(struct qed_dev *cdev)
{
return (QED_IS_FCOE_PERSONALITY(QED_LEADING_HWFN(cdev)) ||
- QED_IS_ISCSI_PERSONALITY(QED_LEADING_HWFN(cdev))) &&
+ QED_IS_ISCSI_PERSONALITY(QED_LEADING_HWFN(cdev)) ||
+ QED_IS_NVMETCP_PERSONALITY(QED_LEADING_HWFN(cdev))) &&
(QED_AFFIN_HWFN(cdev) != QED_LEADING_HWFN(cdev));
}
if (cdev->ll2->handle == QED_LL2_UNUSED_HANDLE)
return 0;
+ if (!QED_IS_NVMETCP_PERSONALITY(p_hwfn))
+ qed_llh_remove_mac_filter(cdev, 0, cdev->ll2_mac_address);
qed_llh_remove_mac_filter(cdev, 0, cdev->ll2_mac_address);
eth_zero_addr(cdev->ll2_mac_address);
- if (QED_IS_ISCSI_PERSONALITY(p_hwfn))
+ if (QED_IS_ISCSI_PERSONALITY(p_hwfn) || QED_IS_NVMETCP_PERSONALITY(p_hwfn))
qed_ll2_stop_ooo(p_hwfn);
/* In CMT mode, LL2 is always started on engine 0 for a storage PF */
conn_type = QED_LL2_TYPE_FCOE;
break;
case QED_PCI_ISCSI:
- conn_type = QED_LL2_TYPE_ISCSI;
+ case QED_PCI_NVMETCP:
+ conn_type = QED_LL2_TYPE_TCP_ULP;
break;
case QED_PCI_ETH_ROCE:
conn_type = QED_LL2_TYPE_ROCE;
}
}
- if (QED_IS_ISCSI_PERSONALITY(p_hwfn)) {
+ if (QED_IS_ISCSI_PERSONALITY(p_hwfn) || QED_IS_NVMETCP_PERSONALITY(p_hwfn)) {
DP_VERBOSE(cdev, QED_MSG_STORAGE, "Starting OOO LL2 queue\n");
rc = qed_ll2_start_ooo(p_hwfn, params);
if (rc) {
}
}
- rc = qed_llh_add_mac_filter(cdev, 0, params->ll2_mac_address);
- if (rc) {
- DP_NOTICE(cdev, "Failed to add an LLH filter\n");
- goto err3;
+ if (!QED_IS_NVMETCP_PERSONALITY(p_hwfn)) {
+ rc = qed_llh_add_mac_filter(cdev, 0, params->ll2_mac_address);
+ if (rc) {
+ DP_NOTICE(cdev, "Failed to add an LLH filter\n");
+ goto err3;
+ }
+
}
ether_addr_copy(cdev->ll2_mac_address, params->ll2_mac_address);
return 0;
err3:
- if (QED_IS_ISCSI_PERSONALITY(p_hwfn))
+ if (QED_IS_ISCSI_PERSONALITY(p_hwfn) || QED_IS_NVMETCP_PERSONALITY(p_hwfn))
qed_ll2_stop_ooo(p_hwfn);
err2:
if (b_is_storage_eng1)
case FUNC_MF_CFG_PROTOCOL_ISCSI:
*p_proto = QED_PCI_ISCSI;
break;
+ case FUNC_MF_CFG_PROTOCOL_NVMETCP:
+ *p_proto = QED_PCI_NVMETCP;
+ break;
case FUNC_MF_CFG_PROTOCOL_FCOE:
*p_proto = QED_PCI_FCOE;
break;
}
if ((tlv_group & QED_MFW_TLV_ISCSI) &&
- p_hwfn->hw_info.personality != QED_PCI_ISCSI) {
+ p_hwfn->hw_info.personality != QED_PCI_ISCSI &&
+ p_hwfn->hw_info.personality != QED_PCI_NVMETCP) {
DP_VERBOSE(p_hwfn, QED_MSG_SP,
"Skipping iSCSI TLVs for non-iSCSI function\n");
tlv_group &= ~QED_MFW_TLV_ISCSI;
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause)
+/* Copyright 2021 Marvell. All rights reserved. */
+
+#include <linux/types.h>
+#include <asm/byteorder.h>
+#include <asm/param.h>
+#include <linux/delay.h>
+#include <linux/dma-mapping.h>
+#include <linux/etherdevice.h>
+#include <linux/kernel.h>
+#include <linux/log2.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/stddef.h>
+#include <linux/string.h>
+#include <linux/errno.h>
+#include <linux/list.h>
+#include <linux/qed/qed_nvmetcp_if.h>
+#include "qed.h"
+#include "qed_cxt.h"
+#include "qed_dev_api.h"
+#include "qed_hsi.h"
+#include "qed_hw.h"
+#include "qed_int.h"
+#include "qed_nvmetcp.h"
+#include "qed_ll2.h"
+#include "qed_mcp.h"
+#include "qed_sp.h"
+#include "qed_reg_addr.h"
+#include "qed_nvmetcp_fw_funcs.h"
+
+static int qed_nvmetcp_async_event(struct qed_hwfn *p_hwfn, u8 fw_event_code,
+ u16 echo, union event_ring_data *data,
+ u8 fw_return_code)
+{
+ if (p_hwfn->p_nvmetcp_info->event_cb) {
+ struct qed_nvmetcp_info *p_nvmetcp = p_hwfn->p_nvmetcp_info;
+
+ return p_nvmetcp->event_cb(p_nvmetcp->event_context,
+ fw_event_code, data);
+ } else {
+ DP_NOTICE(p_hwfn, "nvmetcp async completion is not set\n");
+
+ return -EINVAL;
+ }
+}
+
+static int qed_sp_nvmetcp_func_start(struct qed_hwfn *p_hwfn,
+ enum spq_mode comp_mode,
+ struct qed_spq_comp_cb *p_comp_addr,
+ void *event_context,
+ nvmetcp_event_cb_t async_event_cb)
+{
+ struct nvmetcp_init_ramrod_params *p_ramrod = NULL;
+ struct qed_nvmetcp_pf_params *p_params = NULL;
+ struct scsi_init_func_queues *p_queue = NULL;
+ struct nvmetcp_spe_func_init *p_init = NULL;
+ struct qed_sp_init_data init_data = {};
+ struct qed_spq_entry *p_ent = NULL;
+ int rc = 0;
+ u16 val;
+ u8 i;
+
+ /* Get SPQ entry */
+ init_data.cid = qed_spq_get_cid(p_hwfn);
+ init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+ init_data.comp_mode = comp_mode;
+ init_data.p_comp_data = p_comp_addr;
+ rc = qed_sp_init_request(p_hwfn, &p_ent,
+ NVMETCP_RAMROD_CMD_ID_INIT_FUNC,
+ PROTOCOLID_TCP_ULP, &init_data);
+ if (rc)
+ return rc;
+
+ p_ramrod = &p_ent->ramrod.nvmetcp_init;
+ p_init = &p_ramrod->nvmetcp_init_spe;
+ p_params = &p_hwfn->pf_params.nvmetcp_pf_params;
+ p_queue = &p_init->q_params;
+ p_init->num_sq_pages_in_ring = p_params->num_sq_pages_in_ring;
+ p_init->num_r2tq_pages_in_ring = p_params->num_r2tq_pages_in_ring;
+ p_init->num_uhq_pages_in_ring = p_params->num_uhq_pages_in_ring;
+ p_init->ll2_rx_queue_id = RESC_START(p_hwfn, QED_LL2_RAM_QUEUE) +
+ p_params->ll2_ooo_queue_id;
+ SET_FIELD(p_init->flags, NVMETCP_SPE_FUNC_INIT_NVMETCP_MODE, 1);
+ p_init->func_params.log_page_size = ilog2(PAGE_SIZE);
+ p_init->func_params.num_tasks = cpu_to_le16(p_params->num_tasks);
+ p_init->debug_flags = p_params->debug_mode;
+ DMA_REGPAIR_LE(p_queue->glbl_q_params_addr,
+ p_params->glbl_q_params_addr);
+ p_queue->cq_num_entries = cpu_to_le16(QED_NVMETCP_FW_CQ_SIZE);
+ p_queue->num_queues = p_params->num_queues;
+ val = RESC_START(p_hwfn, QED_CMDQS_CQS);
+ p_queue->queue_relative_offset = cpu_to_le16((u16)val);
+ p_queue->cq_sb_pi = p_params->gl_rq_pi;
+
+ for (i = 0; i < p_params->num_queues; i++) {
+ val = qed_get_igu_sb_id(p_hwfn, i);
+ p_queue->cq_cmdq_sb_num_arr[i] = cpu_to_le16(val);
+ }
+
+ SET_FIELD(p_queue->q_validity,
+ SCSI_INIT_FUNC_QUEUES_CMD_VALID, 0);
+ p_queue->cmdq_num_entries = 0;
+ p_queue->bdq_resource_id = (u8)RESC_START(p_hwfn, QED_BDQ);
+ p_ramrod->tcp_init.two_msl_timer = cpu_to_le32(QED_TCP_TWO_MSL_TIMER);
+ p_ramrod->tcp_init.tx_sws_timer = cpu_to_le16(QED_TCP_SWS_TIMER);
+ p_init->half_way_close_timeout = cpu_to_le16(QED_TCP_HALF_WAY_CLOSE_TIMEOUT);
+ p_ramrod->tcp_init.max_fin_rt = QED_TCP_MAX_FIN_RT;
+ SET_FIELD(p_ramrod->nvmetcp_init_spe.params,
+ NVMETCP_SPE_FUNC_INIT_MAX_SYN_RT, QED_TCP_MAX_FIN_RT);
+ p_hwfn->p_nvmetcp_info->event_context = event_context;
+ p_hwfn->p_nvmetcp_info->event_cb = async_event_cb;
+ qed_spq_register_async_cb(p_hwfn, PROTOCOLID_TCP_ULP,
+ qed_nvmetcp_async_event);
+
+ return qed_spq_post(p_hwfn, p_ent, NULL);
+}
+
+static int qed_sp_nvmetcp_func_stop(struct qed_hwfn *p_hwfn,
+ enum spq_mode comp_mode,
+ struct qed_spq_comp_cb *p_comp_addr)
+{
+ struct qed_spq_entry *p_ent = NULL;
+ struct qed_sp_init_data init_data;
+ int rc;
+
+ /* Get SPQ entry */
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = qed_spq_get_cid(p_hwfn);
+ init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+ init_data.comp_mode = comp_mode;
+ init_data.p_comp_data = p_comp_addr;
+ rc = qed_sp_init_request(p_hwfn, &p_ent,
+ NVMETCP_RAMROD_CMD_ID_DESTROY_FUNC,
+ PROTOCOLID_TCP_ULP, &init_data);
+ if (rc)
+ return rc;
+
+ rc = qed_spq_post(p_hwfn, p_ent, NULL);
+ qed_spq_unregister_async_cb(p_hwfn, PROTOCOLID_TCP_ULP);
+
+ return rc;
+}
+
+static int qed_fill_nvmetcp_dev_info(struct qed_dev *cdev,
+ struct qed_dev_nvmetcp_info *info)
+{
+ struct qed_hwfn *hwfn = QED_AFFIN_HWFN(cdev);
+ int rc;
+
+ memset(info, 0, sizeof(*info));
+ rc = qed_fill_dev_info(cdev, &info->common);
+ info->port_id = MFW_PORT(hwfn);
+ info->num_cqs = FEAT_NUM(hwfn, QED_NVMETCP_CQ);
+
+ return rc;
+}
+
+static void qed_register_nvmetcp_ops(struct qed_dev *cdev,
+ struct qed_nvmetcp_cb_ops *ops,
+ void *cookie)
+{
+ cdev->protocol_ops.nvmetcp = ops;
+ cdev->ops_cookie = cookie;
+}
+
+static int qed_nvmetcp_stop(struct qed_dev *cdev)
+{
+ int rc;
+
+ if (!(cdev->flags & QED_FLAG_STORAGE_STARTED)) {
+ DP_NOTICE(cdev, "nvmetcp already stopped\n");
+
+ return 0;
+ }
+
+ if (!hash_empty(cdev->connections)) {
+ DP_NOTICE(cdev,
+ "Can't stop nvmetcp - not all connections were returned\n");
+
+ return -EINVAL;
+ }
+
+ /* Stop the nvmetcp */
+ rc = qed_sp_nvmetcp_func_stop(QED_AFFIN_HWFN(cdev), QED_SPQ_MODE_EBLOCK,
+ NULL);
+ cdev->flags &= ~QED_FLAG_STORAGE_STARTED;
+
+ return rc;
+}
+
+static int qed_nvmetcp_start(struct qed_dev *cdev,
+ struct qed_nvmetcp_tid *tasks,
+ void *event_context,
+ nvmetcp_event_cb_t async_event_cb)
+{
+ struct qed_tid_mem *tid_info;
+ int rc;
+
+ if (cdev->flags & QED_FLAG_STORAGE_STARTED) {
+ DP_NOTICE(cdev, "nvmetcp already started;\n");
+
+ return 0;
+ }
+
+ rc = qed_sp_nvmetcp_func_start(QED_AFFIN_HWFN(cdev),
+ QED_SPQ_MODE_EBLOCK, NULL,
+ event_context, async_event_cb);
+ if (rc) {
+ DP_NOTICE(cdev, "Failed to start nvmetcp\n");
+
+ return rc;
+ }
+
+ cdev->flags |= QED_FLAG_STORAGE_STARTED;
+ hash_init(cdev->connections);
+
+ if (!tasks)
+ return 0;
+
+ tid_info = kzalloc(sizeof(*tid_info), GFP_KERNEL);
+ if (!tid_info) {
+ qed_nvmetcp_stop(cdev);
+
+ return -ENOMEM;
+ }
+
+ rc = qed_cxt_get_tid_mem_info(QED_AFFIN_HWFN(cdev), tid_info);
+ if (rc) {
+ DP_NOTICE(cdev, "Failed to gather task information\n");
+ qed_nvmetcp_stop(cdev);
+ kfree(tid_info);
+
+ return rc;
+ }
+
+ /* Fill task information */
+ tasks->size = tid_info->tid_size;
+ tasks->num_tids_per_block = tid_info->num_tids_per_block;
+ memcpy(tasks->blocks, tid_info->blocks,
+ MAX_TID_BLOCKS_NVMETCP * sizeof(u8 *));
+ kfree(tid_info);
+
+ return 0;
+}
+
+static struct qed_hash_nvmetcp_con *qed_nvmetcp_get_hash(struct qed_dev *cdev,
+ u32 handle)
+{
+ struct qed_hash_nvmetcp_con *hash_con = NULL;
+
+ if (!(cdev->flags & QED_FLAG_STORAGE_STARTED))
+ return NULL;
+
+ hash_for_each_possible(cdev->connections, hash_con, node, handle) {
+ if (hash_con->con->icid == handle)
+ break;
+ }
+
+ if (!hash_con || hash_con->con->icid != handle)
+ return NULL;
+
+ return hash_con;
+}
+
+static int qed_sp_nvmetcp_conn_offload(struct qed_hwfn *p_hwfn,
+ struct qed_nvmetcp_conn *p_conn,
+ enum spq_mode comp_mode,
+ struct qed_spq_comp_cb *p_comp_addr)
+{
+ struct nvmetcp_spe_conn_offload *p_ramrod = NULL;
+ struct tcp_offload_params_opt2 *p_tcp = NULL;
+ struct qed_sp_init_data init_data = { 0 };
+ struct qed_spq_entry *p_ent = NULL;
+ dma_addr_t r2tq_pbl_addr;
+ dma_addr_t xhq_pbl_addr;
+ dma_addr_t uhq_pbl_addr;
+ u16 physical_q;
+ int rc = 0;
+ u8 i;
+
+ /* Get SPQ entry */
+ init_data.cid = p_conn->icid;
+ init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+ init_data.comp_mode = comp_mode;
+ init_data.p_comp_data = p_comp_addr;
+ rc = qed_sp_init_request(p_hwfn, &p_ent,
+ NVMETCP_RAMROD_CMD_ID_OFFLOAD_CONN,
+ PROTOCOLID_TCP_ULP, &init_data);
+ if (rc)
+ return rc;
+
+ p_ramrod = &p_ent->ramrod.nvmetcp_conn_offload;
+
+ /* Transmission PQ is the first of the PF */
+ physical_q = qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_OFLD);
+ p_conn->physical_q0 = cpu_to_le16(physical_q);
+ p_ramrod->nvmetcp.physical_q0 = cpu_to_le16(physical_q);
+
+ /* nvmetcp Pure-ACK PQ */
+ physical_q = qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_ACK);
+ p_conn->physical_q1 = cpu_to_le16(physical_q);
+ p_ramrod->nvmetcp.physical_q1 = cpu_to_le16(physical_q);
+ p_ramrod->conn_id = cpu_to_le16(p_conn->conn_id);
+ DMA_REGPAIR_LE(p_ramrod->nvmetcp.sq_pbl_addr, p_conn->sq_pbl_addr);
+ r2tq_pbl_addr = qed_chain_get_pbl_phys(&p_conn->r2tq);
+ DMA_REGPAIR_LE(p_ramrod->nvmetcp.r2tq_pbl_addr, r2tq_pbl_addr);
+ xhq_pbl_addr = qed_chain_get_pbl_phys(&p_conn->xhq);
+ DMA_REGPAIR_LE(p_ramrod->nvmetcp.xhq_pbl_addr, xhq_pbl_addr);
+ uhq_pbl_addr = qed_chain_get_pbl_phys(&p_conn->uhq);
+ DMA_REGPAIR_LE(p_ramrod->nvmetcp.uhq_pbl_addr, uhq_pbl_addr);
+ p_ramrod->nvmetcp.flags = p_conn->offl_flags;
+ p_ramrod->nvmetcp.default_cq = p_conn->default_cq;
+ p_ramrod->nvmetcp.initial_ack = 0;
+ DMA_REGPAIR_LE(p_ramrod->nvmetcp.nvmetcp.cccid_itid_table_addr,
+ p_conn->nvmetcp_cccid_itid_table_addr);
+ p_ramrod->nvmetcp.nvmetcp.cccid_max_range =
+ cpu_to_le16(p_conn->nvmetcp_cccid_max_range);
+ p_tcp = &p_ramrod->tcp;
+ qed_set_fw_mac_addr(&p_tcp->remote_mac_addr_hi,
+ &p_tcp->remote_mac_addr_mid,
+ &p_tcp->remote_mac_addr_lo, p_conn->remote_mac);
+ qed_set_fw_mac_addr(&p_tcp->local_mac_addr_hi,
+ &p_tcp->local_mac_addr_mid,
+ &p_tcp->local_mac_addr_lo, p_conn->local_mac);
+ p_tcp->vlan_id = cpu_to_le16(p_conn->vlan_id);
+ p_tcp->flags = cpu_to_le16(p_conn->tcp_flags);
+ p_tcp->ip_version = p_conn->ip_version;
+ if (p_tcp->ip_version == TCP_IPV6) {
+ for (i = 0; i < 4; i++) {
+ p_tcp->remote_ip[i] = cpu_to_le32(p_conn->remote_ip[i]);
+ p_tcp->local_ip[i] = cpu_to_le32(p_conn->local_ip[i]);
+ }
+ } else {
+ p_tcp->remote_ip[0] = cpu_to_le32(p_conn->remote_ip[0]);
+ p_tcp->local_ip[0] = cpu_to_le32(p_conn->local_ip[0]);
+ }
+
+ p_tcp->flow_label = cpu_to_le32(p_conn->flow_label);
+ p_tcp->ttl = p_conn->ttl;
+ p_tcp->tos_or_tc = p_conn->tos_or_tc;
+ p_tcp->remote_port = cpu_to_le16(p_conn->remote_port);
+ p_tcp->local_port = cpu_to_le16(p_conn->local_port);
+ p_tcp->mss = cpu_to_le16(p_conn->mss);
+ p_tcp->rcv_wnd_scale = p_conn->rcv_wnd_scale;
+ p_tcp->connect_mode = p_conn->connect_mode;
+ p_tcp->cwnd = cpu_to_le32(p_conn->cwnd);
+ p_tcp->ka_max_probe_cnt = p_conn->ka_max_probe_cnt;
+ p_tcp->ka_timeout = cpu_to_le32(p_conn->ka_timeout);
+ p_tcp->max_rt_time = cpu_to_le32(p_conn->max_rt_time);
+ p_tcp->ka_interval = cpu_to_le32(p_conn->ka_interval);
+
+ return qed_spq_post(p_hwfn, p_ent, NULL);
+}
+
+static int qed_sp_nvmetcp_conn_update(struct qed_hwfn *p_hwfn,
+ struct qed_nvmetcp_conn *p_conn,
+ enum spq_mode comp_mode,
+ struct qed_spq_comp_cb *p_comp_addr)
+{
+ struct nvmetcp_conn_update_ramrod_params *p_ramrod = NULL;
+ struct qed_spq_entry *p_ent = NULL;
+ struct qed_sp_init_data init_data;
+ int rc = -EINVAL;
+ u32 dval;
+
+ /* Get SPQ entry */
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = p_conn->icid;
+ init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+ init_data.comp_mode = comp_mode;
+ init_data.p_comp_data = p_comp_addr;
+
+ rc = qed_sp_init_request(p_hwfn, &p_ent,
+ NVMETCP_RAMROD_CMD_ID_UPDATE_CONN,
+ PROTOCOLID_TCP_ULP, &init_data);
+ if (rc)
+ return rc;
+
+ p_ramrod = &p_ent->ramrod.nvmetcp_conn_update;
+ p_ramrod->conn_id = cpu_to_le16(p_conn->conn_id);
+ p_ramrod->flags = p_conn->update_flag;
+ p_ramrod->max_seq_size = cpu_to_le32(p_conn->max_seq_size);
+ dval = p_conn->max_recv_pdu_length;
+ p_ramrod->max_recv_pdu_length = cpu_to_le32(dval);
+ dval = p_conn->max_send_pdu_length;
+ p_ramrod->max_send_pdu_length = cpu_to_le32(dval);
+ p_ramrod->first_seq_length = cpu_to_le32(p_conn->first_seq_length);
+
+ return qed_spq_post(p_hwfn, p_ent, NULL);
+}
+
+static int qed_sp_nvmetcp_conn_terminate(struct qed_hwfn *p_hwfn,
+ struct qed_nvmetcp_conn *p_conn,
+ enum spq_mode comp_mode,
+ struct qed_spq_comp_cb *p_comp_addr)
+{
+ struct nvmetcp_spe_conn_termination *p_ramrod = NULL;
+ struct qed_spq_entry *p_ent = NULL;
+ struct qed_sp_init_data init_data;
+ int rc = -EINVAL;
+
+ /* Get SPQ entry */
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = p_conn->icid;
+ init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+ init_data.comp_mode = comp_mode;
+ init_data.p_comp_data = p_comp_addr;
+ rc = qed_sp_init_request(p_hwfn, &p_ent,
+ NVMETCP_RAMROD_CMD_ID_TERMINATION_CONN,
+ PROTOCOLID_TCP_ULP, &init_data);
+ if (rc)
+ return rc;
+
+ p_ramrod = &p_ent->ramrod.nvmetcp_conn_terminate;
+ p_ramrod->conn_id = cpu_to_le16(p_conn->conn_id);
+ p_ramrod->abortive = p_conn->abortive_dsconnect;
+
+ return qed_spq_post(p_hwfn, p_ent, NULL);
+}
+
+static int qed_sp_nvmetcp_conn_clear_sq(struct qed_hwfn *p_hwfn,
+ struct qed_nvmetcp_conn *p_conn,
+ enum spq_mode comp_mode,
+ struct qed_spq_comp_cb *p_comp_addr)
+{
+ struct qed_spq_entry *p_ent = NULL;
+ struct qed_sp_init_data init_data;
+ int rc = -EINVAL;
+
+ /* Get SPQ entry */
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = p_conn->icid;
+ init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+ init_data.comp_mode = comp_mode;
+ init_data.p_comp_data = p_comp_addr;
+ rc = qed_sp_init_request(p_hwfn, &p_ent,
+ NVMETCP_RAMROD_CMD_ID_CLEAR_SQ,
+ PROTOCOLID_TCP_ULP, &init_data);
+ if (rc)
+ return rc;
+
+ return qed_spq_post(p_hwfn, p_ent, NULL);
+}
+
+static void __iomem *qed_nvmetcp_get_db_addr(struct qed_hwfn *p_hwfn, u32 cid)
+{
+ return (u8 __iomem *)p_hwfn->doorbells +
+ qed_db_addr(cid, DQ_DEMS_LEGACY);
+}
+
+static int qed_nvmetcp_allocate_connection(struct qed_hwfn *p_hwfn,
+ struct qed_nvmetcp_conn **p_out_conn)
+{
+ struct qed_chain_init_params params = {
+ .mode = QED_CHAIN_MODE_PBL,
+ .intended_use = QED_CHAIN_USE_TO_CONSUME_PRODUCE,
+ .cnt_type = QED_CHAIN_CNT_TYPE_U16,
+ };
+ struct qed_nvmetcp_pf_params *p_params = NULL;
+ struct qed_nvmetcp_conn *p_conn = NULL;
+ int rc = 0;
+
+ /* Try finding a free connection that can be used */
+ spin_lock_bh(&p_hwfn->p_nvmetcp_info->lock);
+ if (!list_empty(&p_hwfn->p_nvmetcp_info->free_list))
+ p_conn = list_first_entry(&p_hwfn->p_nvmetcp_info->free_list,
+ struct qed_nvmetcp_conn, list_entry);
+ if (p_conn) {
+ list_del(&p_conn->list_entry);
+ spin_unlock_bh(&p_hwfn->p_nvmetcp_info->lock);
+ *p_out_conn = p_conn;
+
+ return 0;
+ }
+ spin_unlock_bh(&p_hwfn->p_nvmetcp_info->lock);
+
+ /* Need to allocate a new connection */
+ p_params = &p_hwfn->pf_params.nvmetcp_pf_params;
+ p_conn = kzalloc(sizeof(*p_conn), GFP_KERNEL);
+ if (!p_conn)
+ return -ENOMEM;
+
+ params.num_elems = p_params->num_r2tq_pages_in_ring *
+ QED_CHAIN_PAGE_SIZE / sizeof(struct nvmetcp_wqe);
+ params.elem_size = sizeof(struct nvmetcp_wqe);
+ rc = qed_chain_alloc(p_hwfn->cdev, &p_conn->r2tq, ¶ms);
+ if (rc)
+ goto nomem_r2tq;
+
+ params.num_elems = p_params->num_uhq_pages_in_ring *
+ QED_CHAIN_PAGE_SIZE / sizeof(struct iscsi_uhqe);
+ params.elem_size = sizeof(struct iscsi_uhqe);
+ rc = qed_chain_alloc(p_hwfn->cdev, &p_conn->uhq, ¶ms);
+ if (rc)
+ goto nomem_uhq;
+
+ params.elem_size = sizeof(struct iscsi_xhqe);
+ rc = qed_chain_alloc(p_hwfn->cdev, &p_conn->xhq, ¶ms);
+ if (rc)
+ goto nomem;
+
+ p_conn->free_on_delete = true;
+ *p_out_conn = p_conn;
+
+ return 0;
+
+nomem:
+ qed_chain_free(p_hwfn->cdev, &p_conn->uhq);
+nomem_uhq:
+ qed_chain_free(p_hwfn->cdev, &p_conn->r2tq);
+nomem_r2tq:
+ kfree(p_conn);
+
+ return -ENOMEM;
+}
+
+static int qed_nvmetcp_acquire_connection(struct qed_hwfn *p_hwfn,
+ struct qed_nvmetcp_conn **p_out_conn)
+{
+ struct qed_nvmetcp_conn *p_conn = NULL;
+ int rc = 0;
+ u32 icid;
+
+ spin_lock_bh(&p_hwfn->p_nvmetcp_info->lock);
+ rc = qed_cxt_acquire_cid(p_hwfn, PROTOCOLID_TCP_ULP, &icid);
+ spin_unlock_bh(&p_hwfn->p_nvmetcp_info->lock);
+
+ if (rc)
+ return rc;
+
+ rc = qed_nvmetcp_allocate_connection(p_hwfn, &p_conn);
+ if (rc) {
+ spin_lock_bh(&p_hwfn->p_nvmetcp_info->lock);
+ qed_cxt_release_cid(p_hwfn, icid);
+ spin_unlock_bh(&p_hwfn->p_nvmetcp_info->lock);
+
+ return rc;
+ }
+
+ p_conn->icid = icid;
+ p_conn->conn_id = (u16)icid;
+ p_conn->fw_cid = (p_hwfn->hw_info.opaque_fid << 16) | icid;
+ *p_out_conn = p_conn;
+
+ return rc;
+}
+
+static void qed_nvmetcp_release_connection(struct qed_hwfn *p_hwfn,
+ struct qed_nvmetcp_conn *p_conn)
+{
+ spin_lock_bh(&p_hwfn->p_nvmetcp_info->lock);
+ list_add_tail(&p_conn->list_entry, &p_hwfn->p_nvmetcp_info->free_list);
+ qed_cxt_release_cid(p_hwfn, p_conn->icid);
+ spin_unlock_bh(&p_hwfn->p_nvmetcp_info->lock);
+}
+
+static void qed_nvmetcp_free_connection(struct qed_hwfn *p_hwfn,
+ struct qed_nvmetcp_conn *p_conn)
+{
+ qed_chain_free(p_hwfn->cdev, &p_conn->xhq);
+ qed_chain_free(p_hwfn->cdev, &p_conn->uhq);
+ qed_chain_free(p_hwfn->cdev, &p_conn->r2tq);
+ kfree(p_conn);
+}
+
+int qed_nvmetcp_alloc(struct qed_hwfn *p_hwfn)
+{
+ struct qed_nvmetcp_info *p_nvmetcp_info;
+
+ p_nvmetcp_info = kzalloc(sizeof(*p_nvmetcp_info), GFP_KERNEL);
+ if (!p_nvmetcp_info)
+ return -ENOMEM;
+
+ INIT_LIST_HEAD(&p_nvmetcp_info->free_list);
+ p_hwfn->p_nvmetcp_info = p_nvmetcp_info;
+
+ return 0;
+}
+
+void qed_nvmetcp_setup(struct qed_hwfn *p_hwfn)
+{
+ spin_lock_init(&p_hwfn->p_nvmetcp_info->lock);
+}
+
+void qed_nvmetcp_free(struct qed_hwfn *p_hwfn)
+{
+ struct qed_nvmetcp_conn *p_conn = NULL;
+
+ if (!p_hwfn->p_nvmetcp_info)
+ return;
+
+ while (!list_empty(&p_hwfn->p_nvmetcp_info->free_list)) {
+ p_conn = list_first_entry(&p_hwfn->p_nvmetcp_info->free_list,
+ struct qed_nvmetcp_conn, list_entry);
+ if (p_conn) {
+ list_del(&p_conn->list_entry);
+ qed_nvmetcp_free_connection(p_hwfn, p_conn);
+ }
+ }
+
+ kfree(p_hwfn->p_nvmetcp_info);
+ p_hwfn->p_nvmetcp_info = NULL;
+}
+
+static int qed_nvmetcp_acquire_conn(struct qed_dev *cdev,
+ u32 *handle,
+ u32 *fw_cid, void __iomem **p_doorbell)
+{
+ struct qed_hash_nvmetcp_con *hash_con;
+ int rc;
+
+ /* Allocate a hashed connection */
+ hash_con = kzalloc(sizeof(*hash_con), GFP_ATOMIC);
+ if (!hash_con)
+ return -ENOMEM;
+
+ /* Acquire the connection */
+ rc = qed_nvmetcp_acquire_connection(QED_AFFIN_HWFN(cdev),
+ &hash_con->con);
+ if (rc) {
+ DP_NOTICE(cdev, "Failed to acquire Connection\n");
+ kfree(hash_con);
+
+ return rc;
+ }
+
+ /* Added the connection to hash table */
+ *handle = hash_con->con->icid;
+ *fw_cid = hash_con->con->fw_cid;
+ hash_add(cdev->connections, &hash_con->node, *handle);
+ if (p_doorbell)
+ *p_doorbell = qed_nvmetcp_get_db_addr(QED_AFFIN_HWFN(cdev),
+ *handle);
+
+ return 0;
+}
+
+static int qed_nvmetcp_release_conn(struct qed_dev *cdev, u32 handle)
+{
+ struct qed_hash_nvmetcp_con *hash_con;
+
+ hash_con = qed_nvmetcp_get_hash(cdev, handle);
+ if (!hash_con) {
+ DP_NOTICE(cdev, "Failed to find connection for handle %d\n",
+ handle);
+
+ return -EINVAL;
+ }
+
+ hlist_del(&hash_con->node);
+ qed_nvmetcp_release_connection(QED_AFFIN_HWFN(cdev), hash_con->con);
+ kfree(hash_con);
+
+ return 0;
+}
+
+static int qed_nvmetcp_offload_conn(struct qed_dev *cdev, u32 handle,
+ struct qed_nvmetcp_params_offload *conn_info)
+{
+ struct qed_hash_nvmetcp_con *hash_con;
+ struct qed_nvmetcp_conn *con;
+
+ hash_con = qed_nvmetcp_get_hash(cdev, handle);
+ if (!hash_con) {
+ DP_NOTICE(cdev, "Failed to find connection for handle %d\n",
+ handle);
+
+ return -EINVAL;
+ }
+
+ /* Update the connection with information from the params */
+ con = hash_con->con;
+
+ /* FW initializations */
+ con->layer_code = NVMETCP_SLOW_PATH_LAYER_CODE;
+ con->sq_pbl_addr = conn_info->sq_pbl_addr;
+ con->nvmetcp_cccid_max_range = conn_info->nvmetcp_cccid_max_range;
+ con->nvmetcp_cccid_itid_table_addr = conn_info->nvmetcp_cccid_itid_table_addr;
+ con->default_cq = conn_info->default_cq;
+ SET_FIELD(con->offl_flags, NVMETCP_CONN_OFFLOAD_PARAMS_TARGET_MODE, 0);
+ SET_FIELD(con->offl_flags, NVMETCP_CONN_OFFLOAD_PARAMS_NVMETCP_MODE, 1);
+ SET_FIELD(con->offl_flags, NVMETCP_CONN_OFFLOAD_PARAMS_TCP_ON_CHIP_1B, 1);
+
+ /* Networking and TCP stack initializations */
+ ether_addr_copy(con->local_mac, conn_info->src.mac);
+ ether_addr_copy(con->remote_mac, conn_info->dst.mac);
+ memcpy(con->local_ip, conn_info->src.ip, sizeof(con->local_ip));
+ memcpy(con->remote_ip, conn_info->dst.ip, sizeof(con->remote_ip));
+ con->local_port = conn_info->src.port;
+ con->remote_port = conn_info->dst.port;
+ con->vlan_id = conn_info->vlan_id;
+
+ if (conn_info->timestamp_en)
+ SET_FIELD(con->tcp_flags, TCP_OFFLOAD_PARAMS_OPT2_TS_EN, 1);
+
+ if (conn_info->delayed_ack_en)
+ SET_FIELD(con->tcp_flags, TCP_OFFLOAD_PARAMS_OPT2_DA_EN, 1);
+
+ if (conn_info->tcp_keep_alive_en)
+ SET_FIELD(con->tcp_flags, TCP_OFFLOAD_PARAMS_OPT2_KA_EN, 1);
+
+ if (conn_info->ecn_en)
+ SET_FIELD(con->tcp_flags, TCP_OFFLOAD_PARAMS_OPT2_ECN_EN, 1);
+
+ con->ip_version = conn_info->ip_version;
+ con->flow_label = QED_TCP_FLOW_LABEL;
+ con->ka_max_probe_cnt = conn_info->ka_max_probe_cnt;
+ con->ka_timeout = conn_info->ka_timeout;
+ con->ka_interval = conn_info->ka_interval;
+ con->max_rt_time = conn_info->max_rt_time;
+ con->ttl = conn_info->ttl;
+ con->tos_or_tc = conn_info->tos_or_tc;
+ con->mss = conn_info->mss;
+ con->cwnd = conn_info->cwnd;
+ con->rcv_wnd_scale = conn_info->rcv_wnd_scale;
+ con->connect_mode = 0;
+
+ return qed_sp_nvmetcp_conn_offload(QED_AFFIN_HWFN(cdev), con,
+ QED_SPQ_MODE_EBLOCK, NULL);
+}
+
+static int qed_nvmetcp_update_conn(struct qed_dev *cdev,
+ u32 handle,
+ struct qed_nvmetcp_params_update *conn_info)
+{
+ struct qed_hash_nvmetcp_con *hash_con;
+ struct qed_nvmetcp_conn *con;
+
+ hash_con = qed_nvmetcp_get_hash(cdev, handle);
+ if (!hash_con) {
+ DP_NOTICE(cdev, "Failed to find connection for handle %d\n",
+ handle);
+
+ return -EINVAL;
+ }
+
+ /* Update the connection with information from the params */
+ con = hash_con->con;
+ SET_FIELD(con->update_flag,
+ ISCSI_CONN_UPDATE_RAMROD_PARAMS_INITIAL_R2T, 0);
+ SET_FIELD(con->update_flag,
+ ISCSI_CONN_UPDATE_RAMROD_PARAMS_IMMEDIATE_DATA, 1);
+ if (conn_info->hdr_digest_en)
+ SET_FIELD(con->update_flag, ISCSI_CONN_UPDATE_RAMROD_PARAMS_HD_EN, 1);
+
+ if (conn_info->data_digest_en)
+ SET_FIELD(con->update_flag, ISCSI_CONN_UPDATE_RAMROD_PARAMS_DD_EN, 1);
+
+ /* Placeholder - initialize pfv, cpda, hpda */
+
+ con->max_seq_size = conn_info->max_io_size;
+ con->max_recv_pdu_length = conn_info->max_recv_pdu_length;
+ con->max_send_pdu_length = conn_info->max_send_pdu_length;
+ con->first_seq_length = conn_info->max_io_size;
+
+ return qed_sp_nvmetcp_conn_update(QED_AFFIN_HWFN(cdev), con,
+ QED_SPQ_MODE_EBLOCK, NULL);
+}
+
+static int qed_nvmetcp_clear_conn_sq(struct qed_dev *cdev, u32 handle)
+{
+ struct qed_hash_nvmetcp_con *hash_con;
+
+ hash_con = qed_nvmetcp_get_hash(cdev, handle);
+ if (!hash_con) {
+ DP_NOTICE(cdev, "Failed to find connection for handle %d\n",
+ handle);
+
+ return -EINVAL;
+ }
+
+ return qed_sp_nvmetcp_conn_clear_sq(QED_AFFIN_HWFN(cdev), hash_con->con,
+ QED_SPQ_MODE_EBLOCK, NULL);
+}
+
+static int qed_nvmetcp_destroy_conn(struct qed_dev *cdev,
+ u32 handle, u8 abrt_conn)
+{
+ struct qed_hash_nvmetcp_con *hash_con;
+
+ hash_con = qed_nvmetcp_get_hash(cdev, handle);
+ if (!hash_con) {
+ DP_NOTICE(cdev, "Failed to find connection for handle %d\n",
+ handle);
+
+ return -EINVAL;
+ }
+
+ hash_con->con->abortive_dsconnect = abrt_conn;
+
+ return qed_sp_nvmetcp_conn_terminate(QED_AFFIN_HWFN(cdev), hash_con->con,
+ QED_SPQ_MODE_EBLOCK, NULL);
+}
+
+static const struct qed_nvmetcp_ops qed_nvmetcp_ops_pass = {
+ .common = &qed_common_ops_pass,
+ .ll2 = &qed_ll2_ops_pass,
+ .fill_dev_info = &qed_fill_nvmetcp_dev_info,
+ .register_ops = &qed_register_nvmetcp_ops,
+ .start = &qed_nvmetcp_start,
+ .stop = &qed_nvmetcp_stop,
+ .acquire_conn = &qed_nvmetcp_acquire_conn,
+ .release_conn = &qed_nvmetcp_release_conn,
+ .offload_conn = &qed_nvmetcp_offload_conn,
+ .update_conn = &qed_nvmetcp_update_conn,
+ .destroy_conn = &qed_nvmetcp_destroy_conn,
+ .clear_sq = &qed_nvmetcp_clear_conn_sq,
+ .add_src_tcp_port_filter = &qed_llh_add_src_tcp_port_filter,
+ .remove_src_tcp_port_filter = &qed_llh_remove_src_tcp_port_filter,
+ .add_dst_tcp_port_filter = &qed_llh_add_dst_tcp_port_filter,
+ .remove_dst_tcp_port_filter = &qed_llh_remove_dst_tcp_port_filter,
+ .clear_all_filters = &qed_llh_clear_all_filters,
+ .init_read_io = &init_nvmetcp_host_read_task,
+ .init_write_io = &init_nvmetcp_host_write_task,
+ .init_icreq_exchange = &init_nvmetcp_init_conn_req_task,
+ .init_task_cleanup = &init_cleanup_task_nvmetcp
+};
+
+const struct qed_nvmetcp_ops *qed_get_nvmetcp_ops(void)
+{
+ return &qed_nvmetcp_ops_pass;
+}
+EXPORT_SYMBOL(qed_get_nvmetcp_ops);
+
+void qed_put_nvmetcp_ops(void)
+{
+}
+EXPORT_SYMBOL(qed_put_nvmetcp_ops);
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) */
+/* Copyright 2021 Marvell. All rights reserved. */
+
+#ifndef _QED_NVMETCP_H
+#define _QED_NVMETCP_H
+
+#include <linux/types.h>
+#include <linux/list.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/qed/tcp_common.h>
+#include <linux/qed/qed_nvmetcp_if.h>
+#include <linux/qed/qed_chain.h>
+#include "qed.h"
+#include "qed_hsi.h"
+#include "qed_mcp.h"
+#include "qed_sp.h"
+
+#define QED_NVMETCP_FW_CQ_SIZE (4 * 1024)
+
+/* tcp parameters */
+#define QED_TCP_FLOW_LABEL 0
+#define QED_TCP_TWO_MSL_TIMER 4000
+#define QED_TCP_HALF_WAY_CLOSE_TIMEOUT 10
+#define QED_TCP_MAX_FIN_RT 2
+#define QED_TCP_SWS_TIMER 5000
+
+struct qed_nvmetcp_info {
+ spinlock_t lock; /* Connection resources. */
+ struct list_head free_list;
+ u16 max_num_outstanding_tasks;
+ void *event_context;
+ nvmetcp_event_cb_t event_cb;
+};
+
+struct qed_hash_nvmetcp_con {
+ struct hlist_node node;
+ struct qed_nvmetcp_conn *con;
+};
+
+struct qed_nvmetcp_conn {
+ struct list_head list_entry;
+ bool free_on_delete;
+ u16 conn_id;
+ u32 icid;
+ u32 fw_cid;
+ u8 layer_code;
+ u8 offl_flags;
+ u8 connect_mode;
+ dma_addr_t sq_pbl_addr;
+ struct qed_chain r2tq;
+ struct qed_chain xhq;
+ struct qed_chain uhq;
+ u8 local_mac[6];
+ u8 remote_mac[6];
+ u8 ip_version;
+ u8 ka_max_probe_cnt;
+ u16 vlan_id;
+ u16 tcp_flags;
+ u32 remote_ip[4];
+ u32 local_ip[4];
+ u32 flow_label;
+ u32 ka_timeout;
+ u32 ka_interval;
+ u32 max_rt_time;
+ u8 ttl;
+ u8 tos_or_tc;
+ u16 remote_port;
+ u16 local_port;
+ u16 mss;
+ u8 rcv_wnd_scale;
+ u32 rcv_wnd;
+ u32 cwnd;
+ u8 update_flag;
+ u8 default_cq;
+ u8 abortive_dsconnect;
+ u32 max_seq_size;
+ u32 max_recv_pdu_length;
+ u32 max_send_pdu_length;
+ u32 first_seq_length;
+ u16 physical_q0;
+ u16 physical_q1;
+ u16 nvmetcp_cccid_max_range;
+ dma_addr_t nvmetcp_cccid_itid_table_addr;
+};
+
+#if IS_ENABLED(CONFIG_QED_NVMETCP)
+int qed_nvmetcp_alloc(struct qed_hwfn *p_hwfn);
+void qed_nvmetcp_setup(struct qed_hwfn *p_hwfn);
+void qed_nvmetcp_free(struct qed_hwfn *p_hwfn);
+
+#else /* IS_ENABLED(CONFIG_QED_NVMETCP) */
+static inline int qed_nvmetcp_alloc(struct qed_hwfn *p_hwfn)
+{
+ return -EINVAL;
+}
+
+static inline void qed_nvmetcp_setup(struct qed_hwfn *p_hwfn) {}
+static inline void qed_nvmetcp_free(struct qed_hwfn *p_hwfn) {}
+
+#endif /* IS_ENABLED(CONFIG_QED_NVMETCP) */
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause)
+/* Copyright 2021 Marvell. All rights reserved. */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/mm.h>
+#include <linux/types.h>
+#include <asm/byteorder.h>
+#include <linux/qed/common_hsi.h>
+#include <linux/qed/storage_common.h>
+#include <linux/qed/nvmetcp_common.h>
+#include <linux/qed/qed_nvmetcp_if.h>
+#include "qed_nvmetcp_fw_funcs.h"
+
+#define NVMETCP_NUM_SGES_IN_CACHE 0x4
+
+bool nvmetcp_is_slow_sgl(u16 num_sges, bool small_mid_sge)
+{
+ return (num_sges > SCSI_NUM_SGES_SLOW_SGL_THR && small_mid_sge);
+}
+
+void init_scsi_sgl_context(struct scsi_sgl_params *ctx_sgl_params,
+ struct scsi_cached_sges *ctx_data_desc,
+ struct storage_sgl_task_params *sgl_params)
+{
+ u8 num_sges_to_init = (u8)(sgl_params->num_sges > NVMETCP_NUM_SGES_IN_CACHE ?
+ NVMETCP_NUM_SGES_IN_CACHE : sgl_params->num_sges);
+ u8 sge_index;
+
+ /* sgl params */
+ ctx_sgl_params->sgl_addr.lo = cpu_to_le32(sgl_params->sgl_phys_addr.lo);
+ ctx_sgl_params->sgl_addr.hi = cpu_to_le32(sgl_params->sgl_phys_addr.hi);
+ ctx_sgl_params->sgl_total_length = cpu_to_le32(sgl_params->total_buffer_size);
+ ctx_sgl_params->sgl_num_sges = cpu_to_le16(sgl_params->num_sges);
+
+ for (sge_index = 0; sge_index < num_sges_to_init; sge_index++) {
+ ctx_data_desc->sge[sge_index].sge_addr.lo =
+ cpu_to_le32(sgl_params->sgl[sge_index].sge_addr.lo);
+ ctx_data_desc->sge[sge_index].sge_addr.hi =
+ cpu_to_le32(sgl_params->sgl[sge_index].sge_addr.hi);
+ ctx_data_desc->sge[sge_index].sge_len =
+ cpu_to_le32(sgl_params->sgl[sge_index].sge_len);
+ }
+}
+
+static inline u32 calc_rw_task_size(struct nvmetcp_task_params *task_params,
+ enum nvmetcp_task_type task_type)
+{
+ u32 io_size;
+
+ if (task_type == NVMETCP_TASK_TYPE_HOST_WRITE)
+ io_size = task_params->tx_io_size;
+ else
+ io_size = task_params->rx_io_size;
+
+ if (unlikely(!io_size))
+ return 0;
+
+ return io_size;
+}
+
+static inline void init_sqe(struct nvmetcp_task_params *task_params,
+ struct storage_sgl_task_params *sgl_task_params,
+ enum nvmetcp_task_type task_type)
+{
+ if (!task_params->sqe)
+ return;
+
+ memset(task_params->sqe, 0, sizeof(*task_params->sqe));
+ task_params->sqe->task_id = cpu_to_le16(task_params->itid);
+
+ switch (task_type) {
+ case NVMETCP_TASK_TYPE_HOST_WRITE: {
+ u32 buf_size = 0;
+ u32 num_sges = 0;
+
+ SET_FIELD(task_params->sqe->contlen_cdbsize,
+ NVMETCP_WQE_CDB_SIZE_OR_NVMETCP_CMD, 1);
+ SET_FIELD(task_params->sqe->flags, NVMETCP_WQE_WQE_TYPE,
+ NVMETCP_WQE_TYPE_NORMAL);
+ if (task_params->tx_io_size) {
+ if (task_params->send_write_incapsule)
+ buf_size = calc_rw_task_size(task_params, task_type);
+
+ if (nvmetcp_is_slow_sgl(sgl_task_params->num_sges,
+ sgl_task_params->small_mid_sge))
+ num_sges = NVMETCP_WQE_NUM_SGES_SLOWIO;
+ else
+ num_sges = min((u16)sgl_task_params->num_sges,
+ (u16)SCSI_NUM_SGES_SLOW_SGL_THR);
+ }
+ SET_FIELD(task_params->sqe->flags, NVMETCP_WQE_NUM_SGES, num_sges);
+ SET_FIELD(task_params->sqe->contlen_cdbsize, NVMETCP_WQE_CONT_LEN, buf_size);
+ } break;
+
+ case NVMETCP_TASK_TYPE_HOST_READ: {
+ SET_FIELD(task_params->sqe->flags, NVMETCP_WQE_WQE_TYPE,
+ NVMETCP_WQE_TYPE_NORMAL);
+ SET_FIELD(task_params->sqe->contlen_cdbsize,
+ NVMETCP_WQE_CDB_SIZE_OR_NVMETCP_CMD, 1);
+ } break;
+
+ case NVMETCP_TASK_TYPE_INIT_CONN_REQUEST: {
+ SET_FIELD(task_params->sqe->flags, NVMETCP_WQE_WQE_TYPE,
+ NVMETCP_WQE_TYPE_MIDDLE_PATH);
+
+ if (task_params->tx_io_size) {
+ SET_FIELD(task_params->sqe->contlen_cdbsize, NVMETCP_WQE_CONT_LEN,
+ task_params->tx_io_size);
+ SET_FIELD(task_params->sqe->flags, NVMETCP_WQE_NUM_SGES,
+ min((u16)sgl_task_params->num_sges,
+ (u16)SCSI_NUM_SGES_SLOW_SGL_THR));
+ }
+ } break;
+
+ case NVMETCP_TASK_TYPE_CLEANUP:
+ SET_FIELD(task_params->sqe->flags, NVMETCP_WQE_WQE_TYPE,
+ NVMETCP_WQE_TYPE_TASK_CLEANUP);
+
+ default:
+ break;
+ }
+}
+
+/* The following function initializes of NVMeTCP task params */
+static inline void
+init_nvmetcp_task_params(struct e5_nvmetcp_task_context *context,
+ struct nvmetcp_task_params *task_params,
+ enum nvmetcp_task_type task_type)
+{
+ context->ystorm_st_context.state.cccid = task_params->host_cccid;
+ SET_FIELD(context->ustorm_st_context.error_flags, USTORM_NVMETCP_TASK_ST_CTX_NVME_TCP, 1);
+ context->ustorm_st_context.nvme_tcp_opaque_lo = cpu_to_le32(task_params->opq.lo);
+ context->ustorm_st_context.nvme_tcp_opaque_hi = cpu_to_le32(task_params->opq.hi);
+}
+
+/* The following function initializes default values to all tasks */
+static inline void
+init_default_nvmetcp_task(struct nvmetcp_task_params *task_params,
+ void *pdu_header, void *nvme_cmd,
+ enum nvmetcp_task_type task_type)
+{
+ struct e5_nvmetcp_task_context *context = task_params->context;
+ const u8 val_byte = context->mstorm_ag_context.cdu_validation;
+ u8 dw_index;
+
+ memset(context, 0, sizeof(*context));
+ init_nvmetcp_task_params(context, task_params,
+ (enum nvmetcp_task_type)task_type);
+
+ /* Swapping requirements used below, will be removed in future FW versions */
+ if (task_type == NVMETCP_TASK_TYPE_HOST_WRITE ||
+ task_type == NVMETCP_TASK_TYPE_HOST_READ) {
+ for (dw_index = 0;
+ dw_index < QED_NVMETCP_CMN_HDR_SIZE / sizeof(u32);
+ dw_index++)
+ context->ystorm_st_context.pdu_hdr.task_hdr.reg[dw_index] =
+ cpu_to_le32(__swab32(((u32 *)pdu_header)[dw_index]));
+
+ for (dw_index = QED_NVMETCP_CMN_HDR_SIZE / sizeof(u32);
+ dw_index < QED_NVMETCP_CMD_HDR_SIZE / sizeof(u32);
+ dw_index++)
+ context->ystorm_st_context.pdu_hdr.task_hdr.reg[dw_index] =
+ cpu_to_le32(__swab32(((u32 *)nvme_cmd)[dw_index - 2]));
+ } else {
+ for (dw_index = 0;
+ dw_index < QED_NVMETCP_NON_IO_HDR_SIZE / sizeof(u32);
+ dw_index++)
+ context->ystorm_st_context.pdu_hdr.task_hdr.reg[dw_index] =
+ cpu_to_le32(__swab32(((u32 *)pdu_header)[dw_index]));
+ }
+
+ /* M-Storm Context: */
+ context->mstorm_ag_context.cdu_validation = val_byte;
+ context->mstorm_st_context.task_type = (u8)(task_type);
+ context->mstorm_ag_context.task_cid = cpu_to_le16(task_params->conn_icid);
+
+ /* Ustorm Context: */
+ SET_FIELD(context->ustorm_ag_context.flags1, E5_USTORM_NVMETCP_TASK_AG_CTX_R2T2RECV, 1);
+ context->ustorm_st_context.task_type = (u8)(task_type);
+ context->ustorm_st_context.cq_rss_number = task_params->cq_rss_number;
+ context->ustorm_ag_context.icid = cpu_to_le16(task_params->conn_icid);
+}
+
+/* The following function initializes the U-Storm Task Contexts */
+static inline void
+init_ustorm_task_contexts(struct ustorm_nvmetcp_task_st_ctx *ustorm_st_context,
+ struct e5_ustorm_nvmetcp_task_ag_ctx *ustorm_ag_context,
+ u32 remaining_recv_len,
+ u32 expected_data_transfer_len, u8 num_sges,
+ bool tx_dif_conn_err_en)
+{
+ /* Remaining data to be received in bytes. Used in validations*/
+ ustorm_st_context->rem_rcv_len = cpu_to_le32(remaining_recv_len);
+ ustorm_ag_context->exp_data_acked = cpu_to_le32(expected_data_transfer_len);
+ ustorm_st_context->exp_data_transfer_len = cpu_to_le32(expected_data_transfer_len);
+ SET_FIELD(ustorm_st_context->reg1_map, REG1_NUM_SGES, num_sges);
+ SET_FIELD(ustorm_ag_context->flags2, E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_CF_EN,
+ tx_dif_conn_err_en ? 1 : 0);
+}
+
+/* The following function initializes Local Completion Contexts: */
+static inline void
+set_local_completion_context(struct e5_nvmetcp_task_context *context)
+{
+ SET_FIELD(context->ystorm_st_context.state.flags,
+ YSTORM_NVMETCP_TASK_STATE_LOCAL_COMP, 1);
+ SET_FIELD(context->ustorm_st_context.flags,
+ USTORM_NVMETCP_TASK_ST_CTX_LOCAL_COMP, 1);
+}
+
+/* Common Fastpath task init function: */
+static inline void
+init_rw_nvmetcp_task(struct nvmetcp_task_params *task_params,
+ enum nvmetcp_task_type task_type,
+ void *pdu_header, void *nvme_cmd,
+ struct storage_sgl_task_params *sgl_task_params)
+{
+ struct e5_nvmetcp_task_context *context = task_params->context;
+ u32 task_size = calc_rw_task_size(task_params, task_type);
+ bool slow_io = false;
+ u8 num_sges = 0;
+
+ init_default_nvmetcp_task(task_params, pdu_header, nvme_cmd, task_type);
+
+ /* Tx/Rx: */
+ if (task_params->tx_io_size) {
+ /* if data to transmit: */
+ init_scsi_sgl_context(&context->ystorm_st_context.state.sgl_params,
+ &context->ystorm_st_context.state.data_desc,
+ sgl_task_params);
+ slow_io = nvmetcp_is_slow_sgl(sgl_task_params->num_sges,
+ sgl_task_params->small_mid_sge);
+ num_sges =
+ (u8)(!slow_io ? min((u32)sgl_task_params->num_sges,
+ (u32)SCSI_NUM_SGES_SLOW_SGL_THR) :
+ NVMETCP_WQE_NUM_SGES_SLOWIO);
+ if (slow_io) {
+ SET_FIELD(context->ystorm_st_context.state.flags,
+ YSTORM_NVMETCP_TASK_STATE_SLOW_IO, 1);
+ }
+ } else if (task_params->rx_io_size) {
+ /* if data to receive: */
+ init_scsi_sgl_context(&context->mstorm_st_context.sgl_params,
+ &context->mstorm_st_context.data_desc,
+ sgl_task_params);
+ num_sges =
+ (u8)(!nvmetcp_is_slow_sgl(sgl_task_params->num_sges,
+ sgl_task_params->small_mid_sge) ?
+ min((u32)sgl_task_params->num_sges,
+ (u32)SCSI_NUM_SGES_SLOW_SGL_THR) :
+ NVMETCP_WQE_NUM_SGES_SLOWIO);
+ context->mstorm_st_context.rem_task_size = cpu_to_le32(task_size);
+ }
+
+ /* Ustorm context: */
+ init_ustorm_task_contexts(&context->ustorm_st_context,
+ &context->ustorm_ag_context,
+ /* Remaining Receive length is the Task Size */
+ task_size,
+ /* The size of the transmitted task */
+ task_size,
+ /* num_sges */
+ num_sges,
+ false);
+
+ /* Set exp_data_acked */
+ if (task_type == NVMETCP_TASK_TYPE_HOST_WRITE) {
+ if (task_params->send_write_incapsule)
+ context->ustorm_ag_context.exp_data_acked = task_size;
+ else
+ context->ustorm_ag_context.exp_data_acked = 0;
+ } else if (task_type == NVMETCP_TASK_TYPE_HOST_READ) {
+ context->ustorm_ag_context.exp_data_acked = 0;
+ }
+
+ context->ustorm_ag_context.exp_cont_len = 0;
+ init_sqe(task_params, sgl_task_params, task_type);
+}
+
+static void
+init_common_initiator_read_task(struct nvmetcp_task_params *task_params,
+ struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+ struct nvme_command *nvme_cmd,
+ struct storage_sgl_task_params *sgl_task_params)
+{
+ init_rw_nvmetcp_task(task_params, NVMETCP_TASK_TYPE_HOST_READ,
+ cmd_pdu_header, nvme_cmd, sgl_task_params);
+}
+
+void init_nvmetcp_host_read_task(struct nvmetcp_task_params *task_params,
+ struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+ struct nvme_command *nvme_cmd,
+ struct storage_sgl_task_params *sgl_task_params)
+{
+ init_common_initiator_read_task(task_params, (void *)cmd_pdu_header,
+ (void *)nvme_cmd, sgl_task_params);
+}
+
+static void
+init_common_initiator_write_task(struct nvmetcp_task_params *task_params,
+ struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+ struct nvme_command *nvme_cmd,
+ struct storage_sgl_task_params *sgl_task_params)
+{
+ init_rw_nvmetcp_task(task_params, NVMETCP_TASK_TYPE_HOST_WRITE,
+ cmd_pdu_header, nvme_cmd, sgl_task_params);
+}
+
+void init_nvmetcp_host_write_task(struct nvmetcp_task_params *task_params,
+ struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+ struct nvme_command *nvme_cmd,
+ struct storage_sgl_task_params *sgl_task_params)
+{
+ init_common_initiator_write_task(task_params, (void *)cmd_pdu_header,
+ (void *)nvme_cmd, sgl_task_params);
+}
+
+static void
+init_common_login_request_task(struct nvmetcp_task_params *task_params,
+ void *login_req_pdu_header,
+ struct storage_sgl_task_params *tx_sgl_task_params,
+ struct storage_sgl_task_params *rx_sgl_task_params)
+{
+ struct e5_nvmetcp_task_context *context = task_params->context;
+
+ init_default_nvmetcp_task(task_params, (void *)login_req_pdu_header, NULL,
+ NVMETCP_TASK_TYPE_INIT_CONN_REQUEST);
+
+ /* Ustorm Context: */
+ init_ustorm_task_contexts(&context->ustorm_st_context,
+ &context->ustorm_ag_context,
+
+ /* Remaining Receive length is the Task Size */
+ task_params->rx_io_size ?
+ rx_sgl_task_params->total_buffer_size : 0,
+
+ /* The size of the transmitted task */
+ task_params->tx_io_size ?
+ tx_sgl_task_params->total_buffer_size : 0,
+ 0, /* num_sges */
+ 0); /* tx_dif_conn_err_en */
+
+ /* SGL context: */
+ if (task_params->tx_io_size)
+ init_scsi_sgl_context(&context->ystorm_st_context.state.sgl_params,
+ &context->ystorm_st_context.state.data_desc,
+ tx_sgl_task_params);
+ if (task_params->rx_io_size)
+ init_scsi_sgl_context(&context->mstorm_st_context.sgl_params,
+ &context->mstorm_st_context.data_desc,
+ rx_sgl_task_params);
+
+ context->mstorm_st_context.rem_task_size =
+ cpu_to_le32(task_params->rx_io_size ?
+ rx_sgl_task_params->total_buffer_size : 0);
+ init_sqe(task_params, tx_sgl_task_params, NVMETCP_TASK_TYPE_INIT_CONN_REQUEST);
+}
+
+/* The following function initializes Login task in Host mode: */
+void init_nvmetcp_init_conn_req_task(struct nvmetcp_task_params *task_params,
+ struct nvme_tcp_icreq_pdu *init_conn_req_pdu_hdr,
+ struct storage_sgl_task_params *tx_sgl_task_params,
+ struct storage_sgl_task_params *rx_sgl_task_params)
+{
+ init_common_login_request_task(task_params, init_conn_req_pdu_hdr,
+ tx_sgl_task_params, rx_sgl_task_params);
+}
+
+void init_cleanup_task_nvmetcp(struct nvmetcp_task_params *task_params)
+{
+ init_sqe(task_params, NULL, NVMETCP_TASK_TYPE_CLEANUP);
+}
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) */
+/* Copyright 2021 Marvell. All rights reserved. */
+
+#ifndef _QED_NVMETCP_FW_FUNCS_H
+#define _QED_NVMETCP_FW_FUNCS_H
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/list.h>
+#include <linux/mm.h>
+#include <linux/types.h>
+#include <asm/byteorder.h>
+#include <linux/qed/common_hsi.h>
+#include <linux/qed/storage_common.h>
+#include <linux/qed/nvmetcp_common.h>
+#include <linux/qed/qed_nvmetcp_if.h>
+
+#if IS_ENABLED(CONFIG_QED_NVMETCP)
+
+void init_nvmetcp_host_read_task(struct nvmetcp_task_params *task_params,
+ struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+ struct nvme_command *nvme_cmd,
+ struct storage_sgl_task_params *sgl_task_params);
+void init_nvmetcp_host_write_task(struct nvmetcp_task_params *task_params,
+ struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+ struct nvme_command *nvme_cmd,
+ struct storage_sgl_task_params *sgl_task_params);
+void init_nvmetcp_init_conn_req_task(struct nvmetcp_task_params *task_params,
+ struct nvme_tcp_icreq_pdu *init_conn_req_pdu_hdr,
+ struct storage_sgl_task_params *tx_sgl_task_params,
+ struct storage_sgl_task_params *rx_sgl_task_params);
+void init_cleanup_task_nvmetcp(struct nvmetcp_task_params *task_params);
+
+#else /* IS_ENABLED(CONFIG_QED_NVMETCP) */
+
+#endif /* IS_ENABLED(CONFIG_QED_NVMETCP) */
+
+#endif /* _QED_NVMETCP_FW_FUNCS_H */
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause)
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+#include <linux/types.h>
+#include <asm/byteorder.h>
+#include <asm/param.h>
+#include <linux/delay.h>
+#include <linux/pci.h>
+#include <linux/dma-mapping.h>
+#include <linux/etherdevice.h>
+#include <linux/kernel.h>
+#include <linux/stddef.h>
+#include <linux/errno.h>
+
+#include <net/tcp.h>
+
+#include <linux/qed/qed_nvmetcp_ip_services_if.h>
+
+#define QED_IP_RESOL_TIMEOUT 4
+
+int qed_route_ipv4(struct sockaddr_storage *local_addr,
+ struct sockaddr_storage *remote_addr,
+ struct sockaddr *hardware_address,
+ struct net_device **ndev)
+{
+ struct neighbour *neigh = NULL;
+ __be32 *loc_ip, *rem_ip;
+ struct rtable *rt;
+ int rc = -ENXIO;
+ int retry;
+
+ loc_ip = &((struct sockaddr_in *)local_addr)->sin_addr.s_addr;
+ rem_ip = &((struct sockaddr_in *)remote_addr)->sin_addr.s_addr;
+ *ndev = NULL;
+ rt = ip_route_output(&init_net, *rem_ip, *loc_ip, 0/*tos*/, 0/*oif*/);
+ if (IS_ERR(rt)) {
+ pr_err("lookup route failed\n");
+ rc = PTR_ERR(rt);
+ goto return_err;
+ }
+
+ neigh = dst_neigh_lookup(&rt->dst, rem_ip);
+ if (!neigh) {
+ rc = -ENOMEM;
+ ip_rt_put(rt);
+ goto return_err;
+ }
+
+ *ndev = rt->dst.dev;
+ ip_rt_put(rt);
+
+ /* If not resolved, kick-off state machine towards resolution */
+ if (!(neigh->nud_state & NUD_VALID))
+ neigh_event_send(neigh, NULL);
+
+ /* query neighbor until resolved or timeout */
+ retry = QED_IP_RESOL_TIMEOUT;
+ while (!(neigh->nud_state & NUD_VALID) && retry > 0) {
+ msleep(1000);
+ retry--;
+ }
+
+ if (neigh->nud_state & NUD_VALID) {
+ /* copy resolved MAC address */
+ neigh_ha_snapshot(hardware_address->sa_data, neigh, *ndev);
+ hardware_address->sa_family = (*ndev)->type;
+ rc = 0;
+ }
+
+ neigh_release(neigh);
+ if (!(*loc_ip)) {
+ *loc_ip = inet_select_addr(*ndev, *rem_ip, RT_SCOPE_UNIVERSE);
+ local_addr->ss_family = AF_INET;
+ }
+
+return_err:
+
+ return rc;
+}
+EXPORT_SYMBOL(qed_route_ipv4);
+
+int qed_route_ipv6(struct sockaddr_storage *local_addr,
+ struct sockaddr_storage *remote_addr,
+ struct sockaddr *hardware_address,
+ struct net_device **ndev)
+{
+ struct neighbour *neigh = NULL;
+ struct dst_entry *dst;
+ struct flowi6 fl6;
+ int rc = -ENXIO;
+ int retry;
+
+ memset(&fl6, 0, sizeof(fl6));
+ fl6.saddr = ((struct sockaddr_in6 *)local_addr)->sin6_addr;
+ fl6.daddr = ((struct sockaddr_in6 *)remote_addr)->sin6_addr;
+ dst = ip6_route_output(&init_net, NULL, &fl6);
+ if (!dst || dst->error) {
+ if (dst) {
+ dst_release(dst);
+ pr_err("lookup route failed %d\n", dst->error);
+ }
+
+ goto out;
+ }
+
+ neigh = dst_neigh_lookup(dst, &fl6.daddr);
+ if (neigh) {
+ *ndev = ip6_dst_idev(dst)->dev;
+
+ /* If not resolved, kick-off state machine towards resolution */
+ if (!(neigh->nud_state & NUD_VALID))
+ neigh_event_send(neigh, NULL);
+
+ /* query neighbor until resolved or timeout */
+ retry = QED_IP_RESOL_TIMEOUT;
+ while (!(neigh->nud_state & NUD_VALID) && retry > 0) {
+ msleep(1000);
+ retry--;
+ }
+
+ if (neigh->nud_state & NUD_VALID) {
+ neigh_ha_snapshot((u8 *)hardware_address->sa_data,
+ neigh, *ndev);
+ hardware_address->sa_family = (*ndev)->type;
+ rc = 0;
+ }
+
+ neigh_release(neigh);
+
+ if (ipv6_addr_any(&fl6.saddr)) {
+ if (ipv6_dev_get_saddr(dev_net(*ndev), *ndev,
+ &fl6.daddr, 0, &fl6.saddr)) {
+ pr_err("Unable to find source IP address\n");
+ goto out;
+ }
+
+ local_addr->ss_family = AF_INET6;
+ ((struct sockaddr_in6 *)local_addr)->sin6_addr =
+ fl6.saddr;
+ }
+ }
+
+ dst_release(dst);
+
+out:
+
+ return rc;
+}
+EXPORT_SYMBOL(qed_route_ipv6);
+
+void qed_vlan_get_ndev(struct net_device **ndev, u16 *vlan_id)
+{
+ if (is_vlan_dev(*ndev)) {
+ *vlan_id = vlan_dev_vlan_id(*ndev);
+ *ndev = vlan_dev_real_dev(*ndev);
+ }
+}
+EXPORT_SYMBOL(qed_vlan_get_ndev);
+
+struct pci_dev *qed_validate_ndev(struct net_device *ndev)
+{
+ struct pci_dev *pdev = NULL;
+ struct net_device *upper;
+
+ for_each_pci_dev(pdev) {
+ if (pdev && pdev->driver &&
+ !strcmp(pdev->driver->name, "qede")) {
+ upper = pci_get_drvdata(pdev);
+ if (upper->ifindex == ndev->ifindex)
+ return pdev;
+ }
+ }
+
+ return NULL;
+}
+EXPORT_SYMBOL(qed_validate_ndev);
+
+__be16 qed_get_in_port(struct sockaddr_storage *sa)
+{
+ return sa->ss_family == AF_INET
+ ? ((struct sockaddr_in *)sa)->sin_port
+ : ((struct sockaddr_in6 *)sa)->sin6_port;
+}
+EXPORT_SYMBOL(qed_get_in_port);
+
+int qed_fetch_tcp_port(struct sockaddr_storage local_ip_addr,
+ struct socket **sock, u16 *port)
+{
+ struct sockaddr_storage sa;
+ int rc = 0;
+
+ rc = sock_create(local_ip_addr.ss_family, SOCK_STREAM, IPPROTO_TCP,
+ sock);
+ if (rc) {
+ pr_warn("failed to create socket: %d\n", rc);
+ goto err;
+ }
+
+ (*sock)->sk->sk_allocation = GFP_KERNEL;
+ sk_set_memalloc((*sock)->sk);
+
+ rc = kernel_bind(*sock, (struct sockaddr *)&local_ip_addr,
+ sizeof(local_ip_addr));
+
+ if (rc) {
+ pr_warn("failed to bind socket: %d\n", rc);
+ goto err_sock;
+ }
+
+ rc = kernel_getsockname(*sock, (struct sockaddr *)&sa);
+ if (rc < 0) {
+ pr_warn("getsockname() failed: %d\n", rc);
+ goto err_sock;
+ }
+
+ *port = ntohs(qed_get_in_port(&sa));
+
+ return 0;
+
+err_sock:
+ sock_release(*sock);
+ sock = NULL;
+err:
+
+ return rc;
+}
+EXPORT_SYMBOL(qed_fetch_tcp_port);
+
+void qed_return_tcp_port(struct socket *sock)
+{
+ if (sock && sock->sk) {
+ tcp_set_state(sock->sk, TCP_CLOSE);
+ sock_release(sock);
+ }
+}
+EXPORT_SYMBOL(qed_return_tcp_port);
#include "qed_ll2.h"
#include "qed_ooo.h"
#include "qed_cxt.h"
-
+#include "qed_nvmetcp.h"
static struct qed_ooo_archipelago
*qed_ooo_seek_archipelago(struct qed_hwfn *p_hwfn,
struct qed_ooo_info
switch (p_hwfn->hw_info.personality) {
case QED_PCI_ISCSI:
- proto = PROTOCOLID_ISCSI;
+ case QED_PCI_NVMETCP:
+ proto = PROTOCOLID_TCP_ULP;
break;
case QED_PCI_ETH_RDMA:
case QED_PCI_ETH_IWARP:
struct iscsi_spe_conn_mac_update iscsi_conn_mac_update;
struct iscsi_spe_conn_termination iscsi_conn_terminate;
+ struct nvmetcp_init_ramrod_params nvmetcp_init;
+ struct nvmetcp_spe_conn_offload nvmetcp_conn_offload;
+ struct nvmetcp_conn_update_ramrod_params nvmetcp_conn_update;
+ struct nvmetcp_spe_conn_termination nvmetcp_conn_terminate;
+
struct vf_start_ramrod_data vf_start;
struct vf_stop_ramrod_data vf_stop;
};
p_ramrod->personality = PERSONALITY_FCOE;
break;
case QED_PCI_ISCSI:
- p_ramrod->personality = PERSONALITY_ISCSI;
+ case QED_PCI_NVMETCP:
+ p_ramrod->personality = PERSONALITY_TCP_ULP;
break;
case QED_PCI_ETH_ROCE:
case QED_PCI_ETH_IWARP:
qede_rdma_get_free_event_node(struct qede_dev *edev)
{
struct qede_rdma_event_work *event_node = NULL;
- struct list_head *list_node = NULL;
bool found = false;
- list_for_each(list_node, &edev->rdma_info.rdma_event_list) {
- event_node = list_entry(list_node, struct qede_rdma_event_work,
- list);
+ list_for_each_entry(event_node, &edev->rdma_info.rdma_event_list,
+ list) {
if (!work_pending(&event_node->work)) {
found = true;
break;
value = readl(&port_regs->CommonRegs.semaphoreReg);
if ((value & (sem_mask >> 16)) == sem_bits)
return 0;
- ssleep(1);
+ mdelay(1000);
} while (--seconds);
return -1;
}
}
/**
- * qlcnic_83xx_idc_cold_state
+ * qlcnic_83xx_idc_cold_state_handler
*
* @adapter: adapter structure
*
}
/**
- * qlcnic_83xx_vnic_opmode
+ * qlcnic_83xx_config_vnic_opmode
*
* @adapter: adapter structure
* Identify virtual NIC operational modes.
for (i = 0; i < QLCNIC_NUM_ILB_PKT; i++) {
skb = netdev_alloc_skb(adapter->netdev, QLCNIC_ILB_PKT_SIZE);
if (!skb)
- break;
+ goto error;
qlcnic_create_loopback_buff(skb->data, adapter->mac_addr);
skb_put(skb, QLCNIC_ILB_PKT_SIZE);
adapter->ahw->diag_cnt = 0;
cnt++;
}
if (cnt != i) {
+error:
dev_err(&adapter->pdev->dev,
"LB Test: failed, TX[%d], RX[%d]\n", i, cnt);
if (mode != QLCNIC_ILB_MODE)
int qlcnic_nic_del_mac(struct qlcnic_adapter *adapter, const u8 *addr)
{
struct qlcnic_mac_vlan_list *cur;
- struct list_head *head;
int err = -EINVAL;
/* Delete MAC from the existing list */
- list_for_each(head, &adapter->mac_list) {
- cur = list_entry(head, struct qlcnic_mac_vlan_list, list);
+ list_for_each_entry(cur, &adapter->mac_list, list) {
if (ether_addr_equal(addr, cur->mac_addr)) {
err = qlcnic_sre_macaddr_change(adapter, cur->mac_addr,
0, QLCNIC_MAC_DEL);
enum qlcnic_mac_type mac_type)
{
struct qlcnic_mac_vlan_list *cur;
- struct list_head *head;
/* look up if already exists */
- list_for_each(head, &adapter->mac_list) {
- cur = list_entry(head, struct qlcnic_mac_vlan_list, list);
+ list_for_each_entry(cur, &adapter->mac_list, list) {
if (ether_addr_equal(addr, cur->mac_addr) &&
cur->vlan_id == vlan)
return 0;
int qlcnic_82xx_get_pci_info(struct qlcnic_adapter *, struct qlcnic_pci_info*);
int qlcnic_82xx_alloc_mbx_args(struct qlcnic_cmd_args *,
struct qlcnic_adapter *, u32);
-int qlcnic_82xx_hw_write_wx_2M(struct qlcnic_adapter *, ulong, u32);
int qlcnic_82xx_get_board_info(struct qlcnic_adapter *);
int qlcnic_82xx_config_led(struct qlcnic_adapter *, u32, u32);
void qlcnic_82xx_get_func_no(struct qlcnic_adapter *);
static void qlcnic_delete_adapter_mac(struct qlcnic_adapter *adapter)
{
struct qlcnic_mac_vlan_list *cur;
- struct list_head *head;
- list_for_each(head, &adapter->mac_list) {
- cur = list_entry(head, struct qlcnic_mac_vlan_list, list);
+ list_for_each_entry(cur, &adapter->mac_list, list) {
if (ether_addr_equal_unaligned(adapter->mac_addr, cur->mac_addr)) {
qlcnic_sre_macaddr_change(adapter, cur->mac_addr,
0, QLCNIC_MAC_DEL);
kfree(ahw);
err_out_free_res:
+ pci_disable_pcie_error_reporting(pdev);
pci_release_regions(pdev);
err_out_disable_pdev:
do {
msleep(1000);
prev_state = QLC_SHARED_REG_RD32(adapter, QLCNIC_CRB_DEV_STATE);
-
- if (prev_state == QLCNIC_DEV_QUISCENT)
- continue;
} while ((prev_state != QLCNIC_DEV_READY) && --dev_init_timeo);
if (!dev_init_timeo) {
struct ifla_rmnet_flags *flags;
flags = nla_data(data[IFLA_RMNET_FLAGS]);
- data_format = flags->flags & flags->mask;
+ data_format &= ~flags->mask;
+ data_format |= flags->flags & flags->mask;
}
netdev_dbg(dev, "data format [0x%08X]\n", data_format);
old_data_format = port->data_format;
flags = nla_data(data[IFLA_RMNET_FLAGS]);
- port->data_format = flags->flags & flags->mask;
+ port->data_format &= ~flags->mask;
+ port->data_format |= flags->flags & flags->mask;
if (rmnet_vnd_update_dev_mtu(port, real_dev)) {
port->data_format = old_data_format;
/* SPDX-License-Identifier: GPL-2.0-only */
-/* Copyright (c) 2013-2014, 2016-2018 The Linux Foundation. All rights reserved.
+/* Copyright (c) 2013-2014, 2016-2018, 2021 The Linux Foundation.
+ * All rights reserved.
*
* RMNET Data configuration engine
*/
struct rmnet_priv_stats {
u64 csum_ok;
+ u64 csum_ip4_header_bad;
u64 csum_valid_unset;
u64 csum_validation_failed;
u64 csum_err_bad_buffer;
u64 csum_fragmented_pkt;
u64 csum_skipped;
u64 csum_sw;
+ u64 csum_hw;
};
struct rmnet_priv {
// SPDX-License-Identifier: GPL-2.0-only
-/* Copyright (c) 2013-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2013-2018, 2021, The Linux Foundation. All rights reserved.
*
* RMNET Data ingress/egress handler
*/
skb->dev = ep->egress_dev;
- /* Subtract MAP header */
- skb_pull(skb, sizeof(struct rmnet_map_header));
- rmnet_set_skb_proto(skb);
-
- if (port->data_format & RMNET_FLAGS_INGRESS_MAP_CKSUMV4) {
- if (!rmnet_map_checksum_downlink_packet(skb, len + pad))
+ if ((port->data_format & RMNET_FLAGS_INGRESS_MAP_CKSUMV5) &&
+ (map_header->flags & MAP_NEXT_HEADER_FLAG)) {
+ if (rmnet_map_process_next_hdr_packet(skb, len))
+ goto free_skb;
+ skb_pull(skb, sizeof(*map_header));
+ rmnet_set_skb_proto(skb);
+ } else {
+ /* Subtract MAP header */
+ skb_pull(skb, sizeof(*map_header));
+ rmnet_set_skb_proto(skb);
+ if (port->data_format & RMNET_FLAGS_INGRESS_MAP_CKSUMV4 &&
+ !rmnet_map_checksum_downlink_packet(skb, len + pad))
skb->ip_summed = CHECKSUM_UNNECESSARY;
}
struct rmnet_port *port, u8 mux_id,
struct net_device *orig_dev)
{
- int required_headroom, additional_header_len;
+ int required_headroom, additional_header_len, csum_type = 0;
struct rmnet_map_header *map_header;
additional_header_len = 0;
if (port->data_format & RMNET_FLAGS_EGRESS_MAP_CKSUMV4) {
additional_header_len = sizeof(struct rmnet_map_ul_csum_header);
- required_headroom += additional_header_len;
+ csum_type = RMNET_FLAGS_EGRESS_MAP_CKSUMV4;
+ } else if (port->data_format & RMNET_FLAGS_EGRESS_MAP_CKSUMV5) {
+ additional_header_len = sizeof(struct rmnet_map_v5_csum_header);
+ csum_type = RMNET_FLAGS_EGRESS_MAP_CKSUMV5;
}
- if (skb_headroom(skb) < required_headroom) {
- if (pskb_expand_head(skb, required_headroom, 0, GFP_ATOMIC))
- return -ENOMEM;
- }
+ required_headroom += additional_header_len;
+
+ if (skb_cow_head(skb, required_headroom) < 0)
+ return -ENOMEM;
- if (port->data_format & RMNET_FLAGS_EGRESS_MAP_CKSUMV4)
- rmnet_map_checksum_uplink_packet(skb, orig_dev);
+ if (csum_type)
+ rmnet_map_checksum_uplink_packet(skb, port, orig_dev,
+ csum_type);
- map_header = rmnet_map_add_map_header(skb, additional_header_len, 0);
+ map_header = rmnet_map_add_map_header(skb, additional_header_len,
+ port, 0);
if (!map_header)
return -ENOMEM;
/* SPDX-License-Identifier: GPL-2.0-only */
-/* Copyright (c) 2013-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2013-2018, 2021, The Linux Foundation. All rights reserved.
*/
#ifndef _RMNET_MAP_H_
struct sk_buff *rmnet_map_deaggregate(struct sk_buff *skb,
struct rmnet_port *port);
struct rmnet_map_header *rmnet_map_add_map_header(struct sk_buff *skb,
- int hdrlen, int pad);
+ int hdrlen,
+ struct rmnet_port *port,
+ int pad);
void rmnet_map_command(struct sk_buff *skb, struct rmnet_port *port);
int rmnet_map_checksum_downlink_packet(struct sk_buff *skb, u16 len);
void rmnet_map_checksum_uplink_packet(struct sk_buff *skb,
- struct net_device *orig_dev);
+ struct rmnet_port *port,
+ struct net_device *orig_dev,
+ int csum_type);
+int rmnet_map_process_next_hdr_packet(struct sk_buff *skb, u16 len);
#endif /* _RMNET_MAP_H_ */
// SPDX-License-Identifier: GPL-2.0-only
-/* Copyright (c) 2013-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2013-2018, 2021, The Linux Foundation. All rights reserved.
*
* RMNET Data MAP protocol
*/
#include <linux/ip.h>
#include <linux/ipv6.h>
#include <net/ip6_checksum.h>
+#include <linux/bitfield.h>
#include "rmnet_config.h"
#include "rmnet_map.h"
#include "rmnet_private.h"
static __sum16 *rmnet_map_get_csum_field(unsigned char protocol,
const void *txporthdr)
{
- __sum16 *check = NULL;
+ if (protocol == IPPROTO_TCP)
+ return &((struct tcphdr *)txporthdr)->check;
- switch (protocol) {
- case IPPROTO_TCP:
- check = &(((struct tcphdr *)txporthdr)->check);
- break;
-
- case IPPROTO_UDP:
- check = &(((struct udphdr *)txporthdr)->check);
- break;
+ if (protocol == IPPROTO_UDP)
+ return &((struct udphdr *)txporthdr)->check;
- default:
- check = NULL;
- break;
- }
-
- return check;
+ return NULL;
}
static int
struct rmnet_map_dl_csum_trailer *csum_trailer,
struct rmnet_priv *priv)
{
- __sum16 *csum_field, csum_temp, pseudo_csum, hdr_csum, ip_payload_csum;
- u16 csum_value, csum_value_final;
- struct iphdr *ip4h;
- void *txporthdr;
- __be16 addend;
-
- ip4h = (struct iphdr *)(skb->data);
- if ((ntohs(ip4h->frag_off) & IP_MF) ||
- ((ntohs(ip4h->frag_off) & IP_OFFSET) > 0)) {
+ struct iphdr *ip4h = (struct iphdr *)skb->data;
+ void *txporthdr = skb->data + ip4h->ihl * 4;
+ __sum16 *csum_field, pseudo_csum;
+ __sum16 ip_payload_csum;
+
+ /* Computing the checksum over just the IPv4 header--including its
+ * checksum field--should yield 0. If it doesn't, the IP header
+ * is bad, so return an error and let the IP layer drop it.
+ */
+ if (ip_fast_csum(ip4h, ip4h->ihl)) {
+ priv->stats.csum_ip4_header_bad++;
+ return -EINVAL;
+ }
+
+ /* We don't support checksum offload on IPv4 fragments */
+ if (ip_is_fragment(ip4h)) {
priv->stats.csum_fragmented_pkt++;
return -EOPNOTSUPP;
}
- txporthdr = skb->data + ip4h->ihl * 4;
-
+ /* Checksum offload is only supported for UDP and TCP protocols */
csum_field = rmnet_map_get_csum_field(ip4h->protocol, txporthdr);
-
if (!csum_field) {
priv->stats.csum_err_invalid_transport++;
return -EPROTONOSUPPORT;
}
- /* RFC 768 - Skip IPv4 UDP packets where sender checksum field is 0 */
- if (*csum_field == 0 && ip4h->protocol == IPPROTO_UDP) {
+ /* RFC 768: UDP checksum is optional for IPv4, and is 0 if unused */
+ if (!*csum_field && ip4h->protocol == IPPROTO_UDP) {
priv->stats.csum_skipped++;
return 0;
}
- csum_value = ~ntohs(csum_trailer->csum_value);
- hdr_csum = ~ip_fast_csum(ip4h, (int)ip4h->ihl);
- ip_payload_csum = csum16_sub((__force __sum16)csum_value,
- (__force __be16)hdr_csum);
-
- pseudo_csum = ~csum_tcpudp_magic(ip4h->saddr, ip4h->daddr,
- ntohs(ip4h->tot_len) - ip4h->ihl * 4,
- ip4h->protocol, 0);
- addend = (__force __be16)ntohs((__force __be16)pseudo_csum);
- pseudo_csum = csum16_add(ip_payload_csum, addend);
-
- addend = (__force __be16)ntohs((__force __be16)*csum_field);
- csum_temp = ~csum16_sub(pseudo_csum, addend);
- csum_value_final = (__force u16)csum_temp;
-
- if (unlikely(csum_value_final == 0)) {
- switch (ip4h->protocol) {
- case IPPROTO_UDP:
- /* RFC 768 - DL4 1's complement rule for UDP csum 0 */
- csum_value_final = ~csum_value_final;
- break;
-
- case IPPROTO_TCP:
- /* DL4 Non-RFC compliant TCP checksum found */
- if (*csum_field == (__force __sum16)0xFFFF)
- csum_value_final = ~csum_value_final;
- break;
- }
- }
-
- if (csum_value_final == ntohs((__force __be16)*csum_field)) {
- priv->stats.csum_ok++;
- return 0;
- } else {
+ /* The checksum value in the trailer is computed over the entire
+ * IP packet, including the IP header and payload. To derive the
+ * transport checksum from this, we first subract the contribution
+ * of the IP header from the trailer checksum. We then add the
+ * checksum computed over the pseudo header.
+ *
+ * We verified above that the IP header contributes zero to the
+ * trailer checksum. Therefore the checksum in the trailer is
+ * just the checksum computed over the IP payload.
+
+ * If the IP payload arrives intact, adding the pseudo header
+ * checksum to the IP payload checksum will yield 0xffff (negative
+ * zero). This means the trailer checksum and the pseudo checksum
+ * are additive inverses of each other. Put another way, the
+ * message passes the checksum test if the trailer checksum value
+ * is the negated pseudo header checksum.
+ *
+ * Knowing this, we don't even need to examine the transport
+ * header checksum value; it is already accounted for in the
+ * checksum value found in the trailer.
+ */
+ ip_payload_csum = csum_trailer->csum_value;
+
+ pseudo_csum = csum_tcpudp_magic(ip4h->saddr, ip4h->daddr,
+ ntohs(ip4h->tot_len) - ip4h->ihl * 4,
+ ip4h->protocol, 0);
+
+ /* The cast is required to ensure only the low 16 bits are examined */
+ if (ip_payload_csum != (__sum16)~pseudo_csum) {
priv->stats.csum_validation_failed++;
return -EINVAL;
}
+
+ priv->stats.csum_ok++;
+ return 0;
}
#if IS_ENABLED(CONFIG_IPV6)
struct rmnet_map_dl_csum_trailer *csum_trailer,
struct rmnet_priv *priv)
{
- __sum16 *csum_field, ip6_payload_csum, pseudo_csum, csum_temp;
- u16 csum_value, csum_value_final;
- __be16 ip6_hdr_csum, addend;
- struct ipv6hdr *ip6h;
- void *txporthdr;
- u32 length;
-
- ip6h = (struct ipv6hdr *)(skb->data);
-
- txporthdr = skb->data + sizeof(struct ipv6hdr);
+ struct ipv6hdr *ip6h = (struct ipv6hdr *)skb->data;
+ void *txporthdr = skb->data + sizeof(*ip6h);
+ __sum16 *csum_field, pseudo_csum;
+ __sum16 ip6_payload_csum;
+ __be16 ip_header_csum;
+
+ /* Checksum offload is only supported for UDP and TCP protocols;
+ * the packet cannot include any IPv6 extension headers
+ */
csum_field = rmnet_map_get_csum_field(ip6h->nexthdr, txporthdr);
-
if (!csum_field) {
priv->stats.csum_err_invalid_transport++;
return -EPROTONOSUPPORT;
}
- csum_value = ~ntohs(csum_trailer->csum_value);
- ip6_hdr_csum = (__force __be16)
- ~ntohs((__force __be16)ip_compute_csum(ip6h,
- (int)(txporthdr - (void *)(skb->data))));
- ip6_payload_csum = csum16_sub((__force __sum16)csum_value,
- ip6_hdr_csum);
-
- length = (ip6h->nexthdr == IPPROTO_UDP) ?
- ntohs(((struct udphdr *)txporthdr)->len) :
- ntohs(ip6h->payload_len);
- pseudo_csum = ~(csum_ipv6_magic(&ip6h->saddr, &ip6h->daddr,
- length, ip6h->nexthdr, 0));
- addend = (__force __be16)ntohs((__force __be16)pseudo_csum);
- pseudo_csum = csum16_add(ip6_payload_csum, addend);
-
- addend = (__force __be16)ntohs((__force __be16)*csum_field);
- csum_temp = ~csum16_sub(pseudo_csum, addend);
- csum_value_final = (__force u16)csum_temp;
-
- if (unlikely(csum_value_final == 0)) {
- switch (ip6h->nexthdr) {
- case IPPROTO_UDP:
- /* RFC 2460 section 8.1
- * DL6 One's complement rule for UDP checksum 0
- */
- csum_value_final = ~csum_value_final;
- break;
-
- case IPPROTO_TCP:
- /* DL6 Non-RFC compliant TCP checksum found */
- if (*csum_field == (__force __sum16)0xFFFF)
- csum_value_final = ~csum_value_final;
- break;
- }
- }
-
- if (csum_value_final == ntohs((__force __be16)*csum_field)) {
- priv->stats.csum_ok++;
- return 0;
- } else {
+ /* The checksum value in the trailer is computed over the entire
+ * IP packet, including the IP header and payload. To derive the
+ * transport checksum from this, we first subract the contribution
+ * of the IP header from the trailer checksum. We then add the
+ * checksum computed over the pseudo header.
+ */
+ ip_header_csum = (__force __be16)ip_fast_csum(ip6h, sizeof(*ip6h) / 4);
+ ip6_payload_csum = csum16_sub(csum_trailer->csum_value, ip_header_csum);
+
+ pseudo_csum = csum_ipv6_magic(&ip6h->saddr, &ip6h->daddr,
+ ntohs(ip6h->payload_len),
+ ip6h->nexthdr, 0);
+
+ /* It's sufficient to compare the IP payload checksum with the
+ * negated pseudo checksum to determine whether the packet
+ * checksum was good. (See further explanation in comments
+ * in rmnet_map_ipv4_dl_csum_trailer()).
+ *
+ * The cast is required to ensure only the low 16 bits are
+ * examined.
+ */
+ if (ip6_payload_csum != (__sum16)~pseudo_csum) {
priv->stats.csum_validation_failed++;
return -EINVAL;
}
+
+ priv->stats.csum_ok++;
+ return 0;
+}
+#else
+static int
+rmnet_map_ipv6_dl_csum_trailer(struct sk_buff *skb,
+ struct rmnet_map_dl_csum_trailer *csum_trailer,
+ struct rmnet_priv *priv)
+{
+ return 0;
}
#endif
-static void rmnet_map_complement_ipv4_txporthdr_csum_field(void *iphdr)
+static void rmnet_map_complement_ipv4_txporthdr_csum_field(struct iphdr *ip4h)
{
- struct iphdr *ip4h = (struct iphdr *)iphdr;
void *txphdr;
u16 *csum;
- txphdr = iphdr + ip4h->ihl * 4;
+ txphdr = (void *)ip4h + ip4h->ihl * 4;
if (ip4h->protocol == IPPROTO_TCP || ip4h->protocol == IPPROTO_UDP) {
csum = (u16 *)rmnet_map_get_csum_field(ip4h->protocol, txphdr);
}
static void
-rmnet_map_ipv4_ul_csum_header(void *iphdr,
+rmnet_map_ipv4_ul_csum_header(struct iphdr *iphdr,
struct rmnet_map_ul_csum_header *ul_header,
struct sk_buff *skb)
{
- struct iphdr *ip4h = iphdr;
u16 val;
val = MAP_CSUM_UL_ENABLED_FLAG;
- if (ip4h->protocol == IPPROTO_UDP)
+ if (iphdr->protocol == IPPROTO_UDP)
val |= MAP_CSUM_UL_UDP_FLAG;
val |= skb->csum_offset & MAP_CSUM_UL_OFFSET_MASK;
}
#if IS_ENABLED(CONFIG_IPV6)
-static void rmnet_map_complement_ipv6_txporthdr_csum_field(void *ip6hdr)
+static void
+rmnet_map_complement_ipv6_txporthdr_csum_field(struct ipv6hdr *ip6h)
{
- struct ipv6hdr *ip6h = (struct ipv6hdr *)ip6hdr;
void *txphdr;
u16 *csum;
- txphdr = ip6hdr + sizeof(struct ipv6hdr);
+ txphdr = ip6h + 1;
if (ip6h->nexthdr == IPPROTO_TCP || ip6h->nexthdr == IPPROTO_UDP) {
csum = (u16 *)rmnet_map_get_csum_field(ip6h->nexthdr, txphdr);
}
static void
-rmnet_map_ipv6_ul_csum_header(void *ip6hdr,
+rmnet_map_ipv6_ul_csum_header(struct ipv6hdr *ipv6hdr,
struct rmnet_map_ul_csum_header *ul_header,
struct sk_buff *skb)
{
- struct ipv6hdr *ip6h = ip6hdr;
u16 val;
val = MAP_CSUM_UL_ENABLED_FLAG;
- if (ip6h->nexthdr == IPPROTO_UDP)
+ if (ipv6hdr->nexthdr == IPPROTO_UDP)
val |= MAP_CSUM_UL_UDP_FLAG;
val |= skb->csum_offset & MAP_CSUM_UL_OFFSET_MASK;
skb->ip_summed = CHECKSUM_NONE;
- rmnet_map_complement_ipv6_txporthdr_csum_field(ip6hdr);
+ rmnet_map_complement_ipv6_txporthdr_csum_field(ipv6hdr);
+}
+#else
+static void
+rmnet_map_ipv6_ul_csum_header(void *ip6hdr,
+ struct rmnet_map_ul_csum_header *ul_header,
+ struct sk_buff *skb)
+{
}
#endif
+static void rmnet_map_v5_checksum_uplink_packet(struct sk_buff *skb,
+ struct rmnet_port *port,
+ struct net_device *orig_dev)
+{
+ struct rmnet_priv *priv = netdev_priv(orig_dev);
+ struct rmnet_map_v5_csum_header *ul_header;
+
+ ul_header = skb_push(skb, sizeof(*ul_header));
+ memset(ul_header, 0, sizeof(*ul_header));
+ ul_header->header_info = u8_encode_bits(RMNET_MAP_HEADER_TYPE_CSUM_OFFLOAD,
+ MAPV5_HDRINFO_HDR_TYPE_FMASK);
+
+ if (skb->ip_summed == CHECKSUM_PARTIAL) {
+ void *iph = ip_hdr(skb);
+ __sum16 *check;
+ void *trans;
+ u8 proto;
+
+ if (skb->protocol == htons(ETH_P_IP)) {
+ u16 ip_len = ((struct iphdr *)iph)->ihl * 4;
+
+ proto = ((struct iphdr *)iph)->protocol;
+ trans = iph + ip_len;
+ } else if (IS_ENABLED(CONFIG_IPV6) &&
+ skb->protocol == htons(ETH_P_IPV6)) {
+ u16 ip_len = sizeof(struct ipv6hdr);
+
+ proto = ((struct ipv6hdr *)iph)->nexthdr;
+ trans = iph + ip_len;
+ } else {
+ priv->stats.csum_err_invalid_ip_version++;
+ goto sw_csum;
+ }
+
+ check = rmnet_map_get_csum_field(proto, trans);
+ if (check) {
+ skb->ip_summed = CHECKSUM_NONE;
+ /* Ask for checksum offloading */
+ ul_header->csum_info |= MAPV5_CSUMINFO_VALID_FLAG;
+ priv->stats.csum_hw++;
+ return;
+ }
+ }
+
+sw_csum:
+ priv->stats.csum_sw++;
+}
+
/* Adds MAP header to front of skb->data
* Padding is calculated and set appropriately in MAP header. Mux ID is
* initialized to 0.
*/
struct rmnet_map_header *rmnet_map_add_map_header(struct sk_buff *skb,
- int hdrlen, int pad)
+ int hdrlen,
+ struct rmnet_port *port,
+ int pad)
{
struct rmnet_map_header *map_header;
u32 padding, map_datalen;
skb_push(skb, sizeof(struct rmnet_map_header));
memset(map_header, 0, sizeof(struct rmnet_map_header));
+ /* Set next_hdr bit for csum offload packets */
+ if (port->data_format & RMNET_FLAGS_EGRESS_MAP_CKSUMV5)
+ map_header->flags |= MAP_NEXT_HEADER_FLAG;
+
if (pad == RMNET_MAP_NO_PAD_BYTES) {
map_header->pkt_len = htons(map_datalen);
return map_header;
struct sk_buff *rmnet_map_deaggregate(struct sk_buff *skb,
struct rmnet_port *port)
{
+ struct rmnet_map_v5_csum_header *next_hdr = NULL;
struct rmnet_map_header *maph;
+ void *data = skb->data;
struct sk_buff *skbn;
+ u8 nexthdr_type;
u32 packet_len;
if (skb->len == 0)
maph = (struct rmnet_map_header *)skb->data;
packet_len = ntohs(maph->pkt_len) + sizeof(*maph);
- if (port->data_format & RMNET_FLAGS_INGRESS_MAP_CKSUMV4)
+ if (port->data_format & RMNET_FLAGS_INGRESS_MAP_CKSUMV4) {
packet_len += sizeof(struct rmnet_map_dl_csum_trailer);
+ } else if (port->data_format & RMNET_FLAGS_INGRESS_MAP_CKSUMV5) {
+ if (!(maph->flags & MAP_CMD_FLAG)) {
+ packet_len += sizeof(*next_hdr);
+ if (maph->flags & MAP_NEXT_HEADER_FLAG)
+ next_hdr = data + sizeof(*maph);
+ else
+ /* Mapv5 data pkt without csum hdr is invalid */
+ return NULL;
+ }
+ }
if (((int)skb->len - (int)packet_len) < 0)
return NULL;
if (!maph->pkt_len)
return NULL;
+ if (next_hdr) {
+ nexthdr_type = u8_get_bits(next_hdr->header_info,
+ MAPV5_HDRINFO_HDR_TYPE_FMASK);
+ if (nexthdr_type != RMNET_MAP_HEADER_TYPE_CSUM_OFFLOAD)
+ return NULL;
+ }
+
skbn = alloc_skb(packet_len + RMNET_MAP_DEAGGR_SPACING, GFP_ATOMIC);
if (!skbn)
return NULL;
return -EINVAL;
}
- if (skb->protocol == htons(ETH_P_IP)) {
+ if (skb->protocol == htons(ETH_P_IP))
return rmnet_map_ipv4_dl_csum_trailer(skb, csum_trailer, priv);
- } else if (skb->protocol == htons(ETH_P_IPV6)) {
-#if IS_ENABLED(CONFIG_IPV6)
+
+ if (IS_ENABLED(CONFIG_IPV6) && skb->protocol == htons(ETH_P_IPV6))
return rmnet_map_ipv6_dl_csum_trailer(skb, csum_trailer, priv);
-#else
- priv->stats.csum_err_invalid_ip_version++;
- return -EPROTONOSUPPORT;
-#endif
- } else {
- priv->stats.csum_err_invalid_ip_version++;
- return -EPROTONOSUPPORT;
- }
- return 0;
+ priv->stats.csum_err_invalid_ip_version++;
+
+ return -EPROTONOSUPPORT;
}
-/* Generates UL checksum meta info header for IPv4 and IPv6 over TCP and UDP
- * packets that are supported for UL checksum offload.
- */
-void rmnet_map_checksum_uplink_packet(struct sk_buff *skb,
- struct net_device *orig_dev)
+static void rmnet_map_v4_checksum_uplink_packet(struct sk_buff *skb,
+ struct net_device *orig_dev)
{
struct rmnet_priv *priv = netdev_priv(orig_dev);
struct rmnet_map_ul_csum_header *ul_header;
(NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM))))
goto sw_csum;
- if (skb->ip_summed == CHECKSUM_PARTIAL) {
- iphdr = (char *)ul_header +
- sizeof(struct rmnet_map_ul_csum_header);
+ if (skb->ip_summed != CHECKSUM_PARTIAL)
+ goto sw_csum;
- if (skb->protocol == htons(ETH_P_IP)) {
- rmnet_map_ipv4_ul_csum_header(iphdr, ul_header, skb);
- return;
- } else if (skb->protocol == htons(ETH_P_IPV6)) {
-#if IS_ENABLED(CONFIG_IPV6)
- rmnet_map_ipv6_ul_csum_header(iphdr, ul_header, skb);
- return;
-#else
- priv->stats.csum_err_invalid_ip_version++;
- goto sw_csum;
-#endif
- } else {
- priv->stats.csum_err_invalid_ip_version++;
- }
+ iphdr = (char *)ul_header +
+ sizeof(struct rmnet_map_ul_csum_header);
+
+ if (skb->protocol == htons(ETH_P_IP)) {
+ rmnet_map_ipv4_ul_csum_header(iphdr, ul_header, skb);
+ priv->stats.csum_hw++;
+ return;
+ }
+
+ if (IS_ENABLED(CONFIG_IPV6) && skb->protocol == htons(ETH_P_IPV6)) {
+ rmnet_map_ipv6_ul_csum_header(iphdr, ul_header, skb);
+ priv->stats.csum_hw++;
+ return;
}
+ priv->stats.csum_err_invalid_ip_version++;
+
sw_csum:
memset(ul_header, 0, sizeof(*ul_header));
priv->stats.csum_sw++;
}
+
+/* Generates UL checksum meta info header for IPv4 and IPv6 over TCP and UDP
+ * packets that are supported for UL checksum offload.
+ */
+void rmnet_map_checksum_uplink_packet(struct sk_buff *skb,
+ struct rmnet_port *port,
+ struct net_device *orig_dev,
+ int csum_type)
+{
+ switch (csum_type) {
+ case RMNET_FLAGS_EGRESS_MAP_CKSUMV4:
+ rmnet_map_v4_checksum_uplink_packet(skb, orig_dev);
+ break;
+ case RMNET_FLAGS_EGRESS_MAP_CKSUMV5:
+ rmnet_map_v5_checksum_uplink_packet(skb, port, orig_dev);
+ break;
+ default:
+ break;
+ }
+}
+
+/* Process a MAPv5 packet header */
+int rmnet_map_process_next_hdr_packet(struct sk_buff *skb,
+ u16 len)
+{
+ struct rmnet_priv *priv = netdev_priv(skb->dev);
+ struct rmnet_map_v5_csum_header *next_hdr;
+ u8 nexthdr_type;
+
+ next_hdr = (struct rmnet_map_v5_csum_header *)(skb->data +
+ sizeof(struct rmnet_map_header));
+
+ nexthdr_type = u8_get_bits(next_hdr->header_info,
+ MAPV5_HDRINFO_HDR_TYPE_FMASK);
+
+ if (nexthdr_type != RMNET_MAP_HEADER_TYPE_CSUM_OFFLOAD)
+ return -EINVAL;
+
+ if (unlikely(!(skb->dev->features & NETIF_F_RXCSUM))) {
+ priv->stats.csum_sw++;
+ } else if (next_hdr->csum_info & MAPV5_CSUMINFO_VALID_FLAG) {
+ priv->stats.csum_ok++;
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
+ } else {
+ priv->stats.csum_valid_unset++;
+ }
+
+ /* Pull csum v5 header */
+ skb_pull(skb, sizeof(*next_hdr));
+
+ return 0;
+}
struct rtnl_link_stats64 *s)
{
struct rmnet_priv *priv = netdev_priv(dev);
- struct rmnet_vnd_stats total_stats;
+ struct rmnet_vnd_stats total_stats = { };
struct rmnet_pcpu_stats *pcpu_ptr;
+ struct rmnet_vnd_stats snapshot;
unsigned int cpu, start;
- memset(&total_stats, 0, sizeof(struct rmnet_vnd_stats));
-
for_each_possible_cpu(cpu) {
pcpu_ptr = per_cpu_ptr(priv->pcpu_stats, cpu);
do {
start = u64_stats_fetch_begin_irq(&pcpu_ptr->syncp);
- total_stats.rx_pkts += pcpu_ptr->stats.rx_pkts;
- total_stats.rx_bytes += pcpu_ptr->stats.rx_bytes;
- total_stats.tx_pkts += pcpu_ptr->stats.tx_pkts;
- total_stats.tx_bytes += pcpu_ptr->stats.tx_bytes;
+ snapshot = pcpu_ptr->stats; /* struct assignment */
} while (u64_stats_fetch_retry_irq(&pcpu_ptr->syncp, start));
- total_stats.tx_drops += pcpu_ptr->stats.tx_drops;
+ total_stats.rx_pkts += snapshot.rx_pkts;
+ total_stats.rx_bytes += snapshot.rx_bytes;
+ total_stats.tx_pkts += snapshot.tx_pkts;
+ total_stats.tx_bytes += snapshot.tx_bytes;
+ total_stats.tx_drops += snapshot.tx_drops;
}
s->rx_packets = total_stats.rx_pkts;
static const char rmnet_gstrings_stats[][ETH_GSTRING_LEN] = {
"Checksum ok",
+ "Bad IPv4 header checksum",
"Checksum valid bit not set",
"Checksum validation failed",
"Checksum error bad buffer",
"Checksum skipped on ip fragment",
"Checksum skipped",
"Checksum computed in software",
+ "Checksum computed in hardware",
};
static void rmnet_get_strings(struct net_device *dev, u32 stringset, u8 *buf)
}
return 0;
-}
\ No newline at end of file
+}
int limit = MAC_DEF_TIMEOUT;
u16 cmd;
- iowrite16(MDIO_READ + reg + (phy_addr << 8), ioaddr + MMDIO);
+ iowrite16(MDIO_READ | reg | (phy_addr << 8), ioaddr + MMDIO);
/* Wait for the read bit to be cleared */
while (limit--) {
cmd = ioread16(ioaddr + MMDIO);
iowrite16(val, ioaddr + MMWD);
/* Write the command to the MDIO bus */
- iowrite16(MDIO_WRITE + reg + (phy_addr << 8), ioaddr + MMDIO);
+ iowrite16(MDIO_WRITE | reg | (phy_addr << 8), ioaddr + MMDIO);
/* Wait for the write bit to be cleared */
while (limit--) {
cmd = ioread16(ioaddr + MMDIO);
skb_ptr->dev = priv->dev;
/* Do not count the CRC */
- skb_put(skb_ptr, descptr->len - 4);
+ skb_put(skb_ptr, descptr->len - ETH_FCS_LEN);
dma_unmap_single(&priv->pdev->dev, le32_to_cpu(descptr->buf),
MAX_BUF_SIZE, DMA_FROM_DEVICE);
skb_ptr->protocol = eth_type_trans(skb_ptr, priv->dev);
/* Send to upper layer */
netif_receive_skb(skb_ptr);
dev->stats.rx_packets++;
- dev->stats.rx_bytes += descptr->len - 4;
+ dev->stats.rx_bytes += descptr->len - ETH_FCS_LEN;
/* put new skb into descriptor */
descptr->skb_ptr = new_skb;
.get_ts_info = ethtool_op_get_ts_info,
.get_link_ksettings = phy_ethtool_get_link_ksettings,
.set_link_ksettings = phy_ethtool_set_link_ksettings,
+ .nway_reset = phy_ethtool_nway_reset,
};
static const struct net_device_ops r6040_netdev_ops = {
Copyright (C) 2000, 2001 David S. Miller (davem@redhat.com) [sungem.c]
Copyright 2001 Manfred Spraul [natsemi.c]
Copyright 1999-2001 by Donald Becker. [natsemi.c]
- Written 1997-2001 by Donald Becker. [8139too.c]
+ Written 1997-2001 by Donald Becker. [8139too.c]
Copyright 1998-2001 by Jes Sorensen, <jes@trained-monkey.org>. [acenic.c]
This software may be used and distributed according to the terms of
/* The chip only need report frame silently dropped. */
spin_lock_irqsave(&cp->lock, flags);
- if (netif_running(dev) && netif_device_present(dev))
- __cp_get_stats(cp);
+ if (netif_running(dev) && netif_device_present(dev))
+ __cp_get_stats(cp);
spin_unlock_irqrestore(&cp->lock, flags);
return &dev->stats;
-----<snip>-----
- Written 1997-2001 by Donald Becker.
+ Written 1997-2001 by Donald Becker.
This software may be used and distributed according to the
terms of the GNU General Public License (GPL), incorporated
herein by reference. Drivers based on or derived from this
{ "RTL-8100",
HW_REVID(1, 1, 1, 1, 0, 1, 0),
- HasLWake,
- },
+ HasLWake,
+ },
{ "RTL-8100B/8139D",
HW_REVID(1, 1, 1, 0, 1, 0, 1),
{
if (length & 1)
{
- length++;
- pad_len++;
+ length++;
+ pad_len++;
}
outb(EOC+MAR, ioaddr + PAR_DATA);
#include "r8169.h"
#include "r8169_firmware.h"
-#define MODULENAME "r8169"
-
#define FIRMWARE_8168D_1 "rtl_nic/rtl8168d-1.fw"
#define FIRMWARE_8168D_2 "rtl_nic/rtl8168d-2.fw"
#define FIRMWARE_8168E_1 "rtl_nic/rtl8168e-1.fw"
struct rtl8169_private *tp = netdev_priv(dev);
struct rtl_fw *rtl_fw = tp->rtl_fw;
- strlcpy(info->driver, MODULENAME, sizeof(info->driver));
+ strlcpy(info->driver, KBUILD_MODNAME, sizeof(info->driver));
strlcpy(info->bus_info, pci_name(tp->pci_dev), sizeof(info->bus_info));
BUILD_BUG_ON(sizeof(info->fw_version) < sizeof(rtl_fw->version));
if (rtl_fw)
{
switch(stringset) {
case ETH_SS_STATS:
- memcpy(data, *rtl8169_gstrings, sizeof(rtl8169_gstrings));
+ memcpy(data, rtl8169_gstrings, sizeof(rtl8169_gstrings));
break;
}
}
rtl_eri_write(tp, 0x1b0, ERIAR_MASK_0011, 0x0000);
rtl_pcie_state_l2l3_disable(tp);
- rtl_hw_aspm_clkreq_enable(tp, true);
}
DECLARE_RTL_COND(rtl_mac_ocp_e00e_cond)
case RTL_GIGA_MAC_VER_61:
case RTL_GIGA_MAC_VER_63:
padto = max_t(unsigned int, padto, ETH_ZLEN);
+ break;
default:
break;
}
return -ENODEV;
}
- rc = pcim_iomap_regions(pdev, BIT(region), MODULENAME);
+ rc = pcim_iomap_regions(pdev, BIT(region), KBUILD_MODNAME);
if (rc < 0) {
dev_err(&pdev->dev, "cannot remap MMIO, aborting\n");
return rc;
}
static struct pci_driver rtl8169_pci_driver = {
- .name = MODULENAME,
+ .name = KBUILD_MODNAME,
.id_table = rtl8169_pci_tbl,
.probe = rtl_init_one,
.remove = rtl_remove_one,
return -EINVAL;
}
- /* Get base address */
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!res) {
- dev_err(&pdev->dev, "invalid resource\n");
- return -EINVAL;
- }
-
ndev = alloc_etherdev_mqs(sizeof(struct ravb_private),
NUM_TX_QUEUE, NUM_RX_QUEUE);
if (!ndev)
pm_runtime_enable(&pdev->dev);
pm_runtime_get_sync(&pdev->dev);
- /* The Ether-specific entries in the device structure. */
- ndev->base_addr = res->start;
-
chip_id = (enum ravb_chip_id)of_device_get_match_data(&pdev->dev);
if (chip_id == RCAR_GEN3)
priv->num_rx_ring[RAVB_BE] = BE_RX_RING_SIZE;
priv->num_tx_ring[RAVB_NC] = NC_TX_RING_SIZE;
priv->num_rx_ring[RAVB_NC] = NC_RX_RING_SIZE;
- priv->addr = devm_ioremap_resource(&pdev->dev, res);
+ priv->addr = devm_platform_get_and_ioremap_resource(pdev, 0, &res);
if (IS_ERR(priv->addr)) {
error = PTR_ERR(priv->addr);
goto out_release;
}
+ /* The Ether-specific entries in the device structure. */
+ ndev->base_addr = res->start;
+
spin_lock_init(&priv->lock);
INIT_WORK(&priv->work, ravb_tx_timeout_work);
{
switch (stringset) {
case ETH_SS_STATS:
- memcpy(data, *sh_eth_gstrings_stats,
+ memcpy(data, sh_eth_gstrings_stats,
sizeof(sh_eth_gstrings_stats));
break;
}
struct net_device *ndev;
int ret;
- /* get base addr */
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
-
ndev = alloc_etherdev(sizeof(struct sh_eth_private));
if (!ndev)
return -ENOMEM;
mdp = netdev_priv(ndev);
mdp->num_tx_ring = TX_RING_SIZE;
mdp->num_rx_ring = RX_RING_SIZE;
- mdp->addr = devm_ioremap_resource(&pdev->dev, res);
+ mdp->addr = devm_platform_get_and_ioremap_resource(pdev, 0, &res);
if (IS_ERR(mdp->addr)) {
ret = PTR_ERR(mdp->addr);
goto out_release;
}
/**
- * sxgbe_tx_clean:
+ * sxgbe_tx_all_clean:
* @priv: driver private structure
* Description: it reclaims resources after transmission completes.
*/
}
/**
- * sxgbe_init_tx_coalesce: init tx mitigation options.
+ * sxgbe_tx_init_coalesce: init tx mitigation options.
* @priv: driver private structure
* Description:
* This inits the transmit coalesce parameters: i.e. timer rate,
break;
}
/*
- * ignore our own packets...
+ * ignore our own packets...
*/
if (!(*(unsigned long *)&dev->dev_addr[0] ^ *(unsigned long *)&addrs[2+6]) &&
!(*(unsigned short *)&dev->dev_addr[4] ^ *(unsigned short *)&addrs[2+10])) {
*/
if (!(ether3_inw(REG_STATUS) & STAT_RXON)) {
dev->stats.rx_dropped++;
- ether3_outw(next_ptr, REG_RECVPTR);
+ ether3_outw(next_ptr, REG_RECVPTR);
ether3_outw(priv(dev)->regs.command | CMD_RXON, REG_COMMAND);
}
do {
unsigned long status;
- /*
+ /*
* Read the packet header
- */
+ */
ether3_setbuffer(dev, buffer_read, tx_tail * 0x600);
- status = ether3_readlong(dev);
+ status = ether3_readlong(dev);
/*
* Check to see if this packet has been transmitted
return 0;
}
-static ssize_t efx_ef10_show_link_control_flag(struct device *dev,
- struct device_attribute *attr,
- char *buf)
+static ssize_t link_control_flag_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
{
struct efx_nic *efx = dev_get_drvdata(dev);
? 1 : 0);
}
-static ssize_t efx_ef10_show_primary_flag(struct device *dev,
- struct device_attribute *attr,
- char *buf)
+static ssize_t primary_flag_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
{
struct efx_nic *efx = dev_get_drvdata(dev);
mutex_unlock(&nic_data->vlan_lock);
}
-static DEVICE_ATTR(link_control_flag, 0444, efx_ef10_show_link_control_flag,
- NULL);
-static DEVICE_ATTR(primary_flag, 0444, efx_ef10_show_primary_flag, NULL);
+static DEVICE_ATTR_RO(link_control_flag);
+static DEVICE_ATTR_RO(primary_flag);
static int efx_ef10_probe(struct efx_nic *efx)
{
.notifier_call = efx_netdev_event,
};
-static ssize_t
-show_phy_type(struct device *dev, struct device_attribute *attr, char *buf)
+static ssize_t phy_type_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
{
struct efx_nic *efx = dev_get_drvdata(dev);
return sprintf(buf, "%d\n", efx->phy_type);
}
-static DEVICE_ATTR(phy_type, 0444, show_phy_type, NULL);
+static DEVICE_ATTR_RO(phy_type);
static int efx_register_netdev(struct efx_nic *efx)
{
efx->state = STATE_READY;
smp_mb(); /* ensure we change state before checking reset_pending */
if (efx->reset_pending) {
- netif_err(efx, probe, efx->net_dev,
- "aborting probe due to scheduled reset\n");
+ pci_err(efx->pci_dev, "aborting probe due to scheduled reset\n");
rc = -EIO;
goto fail_locked;
}
rc = efx->type->init(efx);
up_write(&efx->filter_sem);
if (rc) {
- netif_err(efx, probe, efx->net_dev,
- "failed to initialise NIC\n");
+ pci_err(efx->pci_dev, "failed to initialise NIC\n");
goto fail3;
}
if (efx->type->sriov_init) {
rc = efx->type->sriov_init(efx);
if (rc)
- netif_err(efx, probe, efx->net_dev,
- "SR-IOV can't be enabled rc %d\n", rc);
+ pci_err(efx->pci_dev, "SR-IOV can't be enabled rc %d\n",
+ rc);
}
/* Determine netdevice features */
if (rc)
goto fail1;
- netif_info(efx, probe, efx->net_dev,
- "Solarflare NIC detected\n");
+ pci_info(pci_dev, "Solarflare NIC detected\n");
if (!efx->type->is_vf)
efx_probe_vpd_strings(efx);
}
#ifdef CONFIG_SFC_MCDI_LOGGING
-static ssize_t show_mcdi_log(struct device *dev, struct device_attribute *attr,
- char *buf)
+static ssize_t mcdi_logging_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
{
struct efx_nic *efx = dev_get_drvdata(dev);
struct efx_mcdi_iface *mcdi = efx_mcdi(efx);
return scnprintf(buf, PAGE_SIZE, "%d\n", mcdi->logging_enabled);
}
-static ssize_t set_mcdi_log(struct device *dev, struct device_attribute *attr,
- const char *buf, size_t count)
+static ssize_t mcdi_logging_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct efx_nic *efx = dev_get_drvdata(dev);
struct efx_mcdi_iface *mcdi = efx_mcdi(efx);
return count;
}
-static DEVICE_ATTR(mcdi_logging, 0644, show_mcdi_log, set_mcdi_log);
+static DEVICE_ATTR_RW(mcdi_logging);
void efx_init_mcdi_logging(struct efx_nic *efx)
{
};
static ssize_t
-show_phy_type(struct device *dev, struct device_attribute *attr, char *buf)
+phy_type_show(struct device *dev, struct device_attribute *attr, char *buf)
{
struct ef4_nic *efx = dev_get_drvdata(dev);
return sprintf(buf, "%d\n", efx->phy_type);
}
-static DEVICE_ATTR(phy_type, 0444, show_phy_type, NULL);
+static DEVICE_ATTR_RO(phy_type);
static int ef4_register_netdev(struct ef4_nic *efx)
{
return rc;
}
-static ssize_t show_phy_flash_cfg(struct device *dev,
+static ssize_t phy_flash_cfg_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct ef4_nic *efx = dev_get_drvdata(dev);
return sprintf(buf, "%d\n", !!(efx->phy_mode & PHY_MODE_SPECIAL));
}
-static ssize_t set_phy_flash_cfg(struct device *dev,
- struct device_attribute *attr,
- const char *buf, size_t count)
+static ssize_t phy_flash_cfg_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct ef4_nic *efx = dev_get_drvdata(dev);
enum ef4_phy_mode old_mode, new_mode;
return err ? err : count;
}
-static DEVICE_ATTR(phy_flash_cfg, 0644, show_phy_flash_cfg, set_phy_flash_cfg);
+static DEVICE_ATTR_RW(phy_flash_cfg);
static void sfe4001_fini(struct ef4_nic *efx)
{
*/
void efx_farch_dimension_resources(struct efx_nic *efx, unsigned sram_lim_qw)
{
- unsigned vi_count, buftbl_min, total_tx_channels;
-
+ unsigned vi_count, total_tx_channels;
#ifdef CONFIG_SFC_SRIOV
- struct siena_nic_data *nic_data = efx->nic_data;
+ struct siena_nic_data *nic_data;
+ unsigned buftbl_min;
#endif
total_tx_channels = efx->n_tx_channels + efx->n_extra_tx_channels;
+ vi_count = max(efx->n_channels, total_tx_channels * EFX_MAX_TXQ_PER_CHANNEL);
+
+#ifdef CONFIG_SFC_SRIOV
+ nic_data = efx->nic_data;
/* Account for the buffer table entries backing the datapath channels
* and the descriptor caches for those channels.
*/
total_tx_channels * EFX_MAX_TXQ_PER_CHANNEL * EFX_MAX_DMAQ_SIZE +
efx->n_channels * EFX_MAX_EVQ_SIZE)
* sizeof(efx_qword_t) / EFX_BUF_SIZE);
- vi_count = max(efx->n_channels, total_tx_channels * EFX_MAX_TXQ_PER_CHANNEL);
-
-#ifdef CONFIG_SFC_SRIOV
if (efx->type->sriov_wanted) {
if (efx->type->sriov_wanted(efx)) {
unsigned vi_dc_entries, buftbl_free;
efx->pci_dev->irq);
goto fail1;
}
+ efx->irqs_hooked = true;
return 0;
}
int err;
regs = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!regs) {
+ dev_err(&pdev->dev, "Invalid resource\n");
+ return -EINVAL;
+ }
/* get mac addr from one wire prom */
if (ioc3eth_get_mac_addr(regs, mac_addr))
return -EPROBE_DEFER; /* not available yet */
/* Reset phy if default phy is internal sis900 */
if ((sis_priv->mii->phy_id0 == 0x001D) &&
((sis_priv->mii->phy_id1&0xFFF0) == 0x8000))
- status = sis900_reset_phy(net_dev, sis_priv->cur_phy);
+ status = sis900_reset_phy(net_dev, sis_priv->cur_phy);
/* workaround for ICS1893 PHY */
if ((sis_priv->mii->phy_id0 == 0x0015) &&
((sis_priv->mii->phy_id1&0xFFF0) == 0xF440))
- mdio_write(net_dev, sis_priv->cur_phy, 0x0018, 0xD200);
+ mdio_write(net_dev, sis_priv->cur_phy, 0x0018, 0xD200);
if(status & MII_STAT_LINK){
while (poll_bit) {
static u16 sis900_default_phy(struct net_device * net_dev)
{
struct sis900_private *sis_priv = netdev_priv(net_dev);
- struct mii_phy *phy = NULL, *phy_home = NULL,
+ struct mii_phy *phy = NULL, *phy_home = NULL,
*default_phy = NULL, *phy_lan = NULL;
u16 status;
} else {
/* Link ON -> OFF */
if (!(status & MII_STAT_LINK)){
- netif_carrier_off(net_dev);
+ netif_carrier_off(net_dev);
if(netif_msg_link(sis_priv))
- printk(KERN_INFO "%s: Media Link Off\n", net_dev->name);
+ printk(KERN_INFO "%s: Media Link Off\n", net_dev->name);
- /* Change mode issue */
- if ((mii_phy->phy_id0 == 0x001D) &&
- ((mii_phy->phy_id1 & 0xFFF0) == 0x8000))
- sis900_reset_phy(net_dev, sis_priv->cur_phy);
+ /* Change mode issue */
+ if ((mii_phy->phy_id0 == 0x001D) &&
+ ((mii_phy->phy_id1 & 0xFFF0) == 0x8000))
+ sis900_reset_phy(net_dev, sis_priv->cur_phy);
sis630_set_eq(net_dev, sis_priv->chipset_rev);
- goto LookForLink;
+ goto LookForLink;
}
}
case IF_PORT_10BASE2: /* 10Base2 */
case IF_PORT_AUI: /* AUI */
case IF_PORT_100BASEFX: /* 100BaseFx */
- /* These Modes are not supported (are they?)*/
+ /* These Modes are not supported (are they?)*/
return -EOPNOTSUPP;
default:
struct sk_buff * saved_skb;
/*
- . This keeps track of how many packets that I have
- . sent out. When an TX_EMPTY interrupt comes, I know
+ . This keeps track of how many packets that I have
+ . sent out. When an TX_EMPTY interrupt comes, I know
. that all of these have been sent.
*/
int packets_waiting;
/* Note: It doesn't seem that waiting for the MMU busy is needed here,
but this is a place where future chipsets _COULD_ break. Be wary
- of issuing another MMU command right after this */
+ of issuing another MMU command right after this */
outb( 0, ioaddr + INT_MASK );
}
SMC_SELECT_BANK( 2 );
outw( MC_ALLOC | numPages, ioaddr + MMU_CMD );
/*
- . Performance Hack
+ . Performance Hack
.
- . wait a short amount of time.. if I can send a packet now, I send
+ . wait a short amount of time.. if I can send a packet now, I send
. it now. Otherwise, I enable an interrupt and wait for one to be
. available.
.
if ( status & IM_ALLOC_INT ) {
/* acknowledge the interrupt */
outb( IM_ALLOC_INT, ioaddr + INTERRUPT );
- break;
+ break;
}
- } while ( -- time_out );
+ } while ( -- time_out );
- if ( !time_out ) {
+ if ( !time_out ) {
/* oh well, wait until the chip finds memory later */
SMC_ENABLE_INT( IM_ALLOC_INT );
PRINTK2((CARDNAME": memory allocation deferred.\n"));
/* it's deferred, but I'll handle it later */
return NETDEV_TX_OK;
- }
+ }
/* or YES! I can send the packet now.. */
smc_hardware_send_packet(dev);
netif_wake_queue(dev);
#endif
/* send the packet length ( +6 for status, length and ctl byte )
- and the status word ( set to zeros ) */
+ and the status word ( set to zeros ) */
#ifdef USE_32_BIT
outl( (length +6 ) << 16 , ioaddr + DATA_1 );
#else
/* send the actual data
. I _think_ it's faster to send the longs first, and then
. mop up by sending the last word. It depends heavily
- . on alignment, at least on the 486. Maybe it would be
- . a good idea to check which is optimal? But that could take
+ . on alignment, at least on the 486. Maybe it would be
+ . a good idea to check which is optimal? But that could take
. almost as much time as is saved?
*/
#ifdef USE_32_BIT
outb( IM_ALLOC_INT, ioaddr + INT_MASK );
/*
- . Allocate 512 bytes of memory. Note that the chip was just
+ . Allocate 512 bytes of memory. Note that the chip was just
. reset so all the memory is available
*/
outw( MC_ALLOC | 1, ioaddr + MMU_CMD );
goto err_out;
}
/* The above MIGHT indicate a device, but I need to write to further
- test this. */
+ test this. */
outw( 0x0, ioaddr + BANK_SELECT );
bank = inw( ioaddr + BANK_SELECT );
if ( (bank & 0xFF00 ) != 0x3300 ) {
goto err_out;
}
/* well, we've already written once, so hopefully another time won't
- hurt. This time, I need to switch the bank register to bank 1,
+ hurt. This time, I need to switch the bank register to bank 1,
so I can access the base address register */
SMC_SELECT_BANK(1);
base_address_register = inw( ioaddr + BASE );
dev->base_addr = ioaddr;
/*
- . Get the MAC address ( bank 1, regs 4 - 9 )
+ . Get the MAC address ( bank 1, regs 4 - 9 )
*/
SMC_SELECT_BANK( 1 );
for ( i = 0; i < 6; i += 2 ) {
/*
Now, I want to find out more about the chip. This is sort of
- redundant, but it's cleaner to have it in both, rather than having
- one VERY long probe procedure.
+ redundant, but it's cleaner to have it in both, rather than having
+ one VERY long probe procedure.
*/
SMC_SELECT_BANK(3);
revision_register = inw( ioaddr + REVISION );
/*
. If dev->irq is 0, then the device has to be banged on to see
. what the IRQ is.
- .
+ .
. This banging doesn't always detect the IRQ, for unknown reasons.
. a workaround is to reset the chip and try again.
.
.
. Specifying an IRQ is done with the assumption that the user knows
. what (s)he is doing. No checking is done!!!!
- .
+ .
*/
if ( dev->irq < 2 ) {
int trials;
}
/*
- According to Becker, I have to set the hardware address
+ According to Becker, I have to set the hardware address
at this point, because the (l)user can set it with an
ioctl. Easily done...
*/
status = SMC_GET_INT(lp);
if (status & IM_ALLOC_INT) {
SMC_ACK_INT(lp, IM_ALLOC_INT);
- break;
+ break;
}
- } while (--poll_count);
+ } while (--poll_count);
smc_special_unlock(&lp->lock, flags);
lp->pending_tx_skb = skb;
- if (!poll_count) {
+ if (!poll_count) {
/* oh well, wait until the chip finds memory later */
netif_stop_queue(dev);
DBG(2, dev, "TX memory allocation deferred.\n");
SMC_ENABLE_INT(lp, IM_ALLOC_INT);
- } else {
+ } else {
/*
* Allocation succeeded: push packet to the chip's own memory
* immediately.
SMC_SET_INT_MASK(lp, IM_ALLOC_INT);
/*
- * Allocate 512 bytes of memory. Note that the chip was just
+ * Allocate 512 bytes of memory. Note that the chip was just
* reset so all the memory is available
*/
SMC_SET_MMU_CMD(lp, MC_ALLOC | 1);
/* Grab the IRQ */
retval = request_irq(dev->irq, smc_interrupt, irq_flags, dev->name, dev);
- if (retval)
- goto err_out;
+ if (retval)
+ goto err_out;
#ifdef CONFIG_ARCH_PXA
# ifdef SMC_USE_PXA_DMA
// SPDX-License-Identifier: GPL-2.0
-/**
+/*
* sni_ave.c - Socionext UniPhier AVE ethernet driver
* Copyright 2014 Panasonic Corporation
* Copyright 2015-2017 Socionext Inc.
This selects the Anarion SoC glue layer support for the stmmac driver.
+config DWMAC_INGENIC
+ tristate "Ingenic MAC support"
+ default MACH_INGENIC
+ depends on OF && HAS_IOMEM && (MACH_INGENIC || COMPILE_TEST)
+ select MFD_SYSCON
+ help
+ Support for ethernet controller on Ingenic SoCs.
+
+ This selects Ingenic SoCs glue layer support for the stmmac
+ device driver. This driver is used on for the Ingenic SoCs
+ MAC ethernet controller.
+
config DWMAC_IPQ806X
tristate "QCA IPQ806x DWMAC support"
default ARCH_QCOM
This selects the Intel platform specific bus support for the
stmmac driver. This driver is used for Intel Quark/EHL/TGL.
+config DWMAC_LOONGSON
+ tristate "Loongson PCI DWMAC support"
+ default MACH_LOONGSON64
+ depends on STMMAC_ETH && PCI
+ depends on COMMON_CLK
+ help
+ This selects the LOONGSON PCI bus support for the stmmac driver,
+ Support for ethernet controller on Loongson-2K1000 SoC and LS7A1000 bridge.
+
config STMMAC_PCI
tristate "STMMAC PCI bus support"
depends on STMMAC_ETH && PCI
# Ordering matters. Generic driver must be last.
obj-$(CONFIG_STMMAC_PLATFORM) += stmmac-platform.o
obj-$(CONFIG_DWMAC_ANARION) += dwmac-anarion.o
+obj-$(CONFIG_DWMAC_INGENIC) += dwmac-ingenic.o
obj-$(CONFIG_DWMAC_IPQ806X) += dwmac-ipq806x.o
obj-$(CONFIG_DWMAC_LPC18XX) += dwmac-lpc18xx.o
obj-$(CONFIG_DWMAC_MEDIATEK) += dwmac-mediatek.o
obj-$(CONFIG_STMMAC_PCI) += stmmac-pci.o
obj-$(CONFIG_DWMAC_INTEL) += dwmac-intel.o
+obj-$(CONFIG_DWMAC_LOONGSON) += dwmac-loongson.o
stmmac-pci-objs:= stmmac_pci.o
const struct stmmac_hwtimestamp *ptp;
const struct stmmac_tc_ops *tc;
const struct stmmac_mmc_ops *mmc;
- const struct mdio_xpcs_ops *xpcs;
- struct mdio_xpcs_args xpcs_args;
+ struct dw_xpcs *xpcs;
struct mii_regs mii; /* MII register Addresses */
struct mac_link link;
void __iomem *pcsr; /* vpointer to device CSRs */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * dwmac-ingenic.c - Ingenic SoCs DWMAC specific glue layer
+ *
+ * Copyright (c) 2021 周琰杰 (Zhou Yanjie) <zhouyanjie@wanyeetech.com>
+ */
+
+#include <linux/bitfield.h>
+#include <linux/clk.h>
+#include <linux/kernel.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/of_net.h>
+#include <linux/phy.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include <linux/slab.h>
+#include <linux/stmmac.h>
+
+#include "stmmac_platform.h"
+
+#define MACPHYC_TXCLK_SEL_MASK GENMASK(31, 31)
+#define MACPHYC_TXCLK_SEL_OUTPUT 0x1
+#define MACPHYC_TXCLK_SEL_INPUT 0x0
+#define MACPHYC_MODE_SEL_MASK GENMASK(31, 31)
+#define MACPHYC_MODE_SEL_RMII 0x0
+#define MACPHYC_TX_SEL_MASK GENMASK(19, 19)
+#define MACPHYC_TX_SEL_ORIGIN 0x0
+#define MACPHYC_TX_SEL_DELAY 0x1
+#define MACPHYC_TX_DELAY_MASK GENMASK(18, 12)
+#define MACPHYC_RX_SEL_MASK GENMASK(11, 11)
+#define MACPHYC_RX_SEL_ORIGIN 0x0
+#define MACPHYC_RX_SEL_DELAY 0x1
+#define MACPHYC_RX_DELAY_MASK GENMASK(10, 4)
+#define MACPHYC_SOFT_RST_MASK GENMASK(3, 3)
+#define MACPHYC_PHY_INFT_MASK GENMASK(2, 0)
+#define MACPHYC_PHY_INFT_RMII 0x4
+#define MACPHYC_PHY_INFT_RGMII 0x1
+#define MACPHYC_PHY_INFT_GMII 0x0
+#define MACPHYC_PHY_INFT_MII 0x0
+
+#define MACPHYC_TX_DELAY_PS_MAX 2496
+#define MACPHYC_TX_DELAY_PS_MIN 20
+
+#define MACPHYC_RX_DELAY_PS_MAX 2496
+#define MACPHYC_RX_DELAY_PS_MIN 20
+
+enum ingenic_mac_version {
+ ID_JZ4775,
+ ID_X1000,
+ ID_X1600,
+ ID_X1830,
+ ID_X2000,
+};
+
+struct ingenic_mac {
+ const struct ingenic_soc_info *soc_info;
+ struct device *dev;
+ struct regmap *regmap;
+
+ int rx_delay;
+ int tx_delay;
+};
+
+struct ingenic_soc_info {
+ enum ingenic_mac_version version;
+ u32 mask;
+
+ int (*set_mode)(struct plat_stmmacenet_data *plat_dat);
+};
+
+static int ingenic_mac_init(struct plat_stmmacenet_data *plat_dat)
+{
+ struct ingenic_mac *mac = plat_dat->bsp_priv;
+ int ret;
+
+ if (mac->soc_info->set_mode) {
+ ret = mac->soc_info->set_mode(plat_dat);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int jz4775_mac_set_mode(struct plat_stmmacenet_data *plat_dat)
+{
+ struct ingenic_mac *mac = plat_dat->bsp_priv;
+ unsigned int val;
+
+ switch (plat_dat->interface) {
+ case PHY_INTERFACE_MODE_MII:
+ val = FIELD_PREP(MACPHYC_TXCLK_SEL_MASK, MACPHYC_TXCLK_SEL_INPUT) |
+ FIELD_PREP(MACPHYC_PHY_INFT_MASK, MACPHYC_PHY_INFT_MII);
+ dev_dbg(mac->dev, "MAC PHY Control Register: PHY_INTERFACE_MODE_MII\n");
+ break;
+
+ case PHY_INTERFACE_MODE_GMII:
+ val = FIELD_PREP(MACPHYC_TXCLK_SEL_MASK, MACPHYC_TXCLK_SEL_INPUT) |
+ FIELD_PREP(MACPHYC_PHY_INFT_MASK, MACPHYC_PHY_INFT_GMII);
+ dev_dbg(mac->dev, "MAC PHY Control Register: PHY_INTERFACE_MODE_GMII\n");
+ break;
+
+ case PHY_INTERFACE_MODE_RMII:
+ val = FIELD_PREP(MACPHYC_TXCLK_SEL_MASK, MACPHYC_TXCLK_SEL_INPUT) |
+ FIELD_PREP(MACPHYC_PHY_INFT_MASK, MACPHYC_PHY_INFT_RMII);
+ dev_dbg(mac->dev, "MAC PHY Control Register: PHY_INTERFACE_MODE_RMII\n");
+ break;
+
+ case PHY_INTERFACE_MODE_RGMII:
+ case PHY_INTERFACE_MODE_RGMII_ID:
+ case PHY_INTERFACE_MODE_RGMII_TXID:
+ case PHY_INTERFACE_MODE_RGMII_RXID:
+ val = FIELD_PREP(MACPHYC_TXCLK_SEL_MASK, MACPHYC_TXCLK_SEL_INPUT) |
+ FIELD_PREP(MACPHYC_PHY_INFT_MASK, MACPHYC_PHY_INFT_RGMII);
+ dev_dbg(mac->dev, "MAC PHY Control Register: PHY_INTERFACE_MODE_RGMII\n");
+ break;
+
+ default:
+ dev_err(mac->dev, "Unsupported interface %d", plat_dat->interface);
+ return -EINVAL;
+ }
+
+ /* Update MAC PHY control register */
+ return regmap_update_bits(mac->regmap, 0, mac->soc_info->mask, val);
+}
+
+static int x1000_mac_set_mode(struct plat_stmmacenet_data *plat_dat)
+{
+ struct ingenic_mac *mac = plat_dat->bsp_priv;
+
+ switch (plat_dat->interface) {
+ case PHY_INTERFACE_MODE_RMII:
+ dev_dbg(mac->dev, "MAC PHY Control Register: PHY_INTERFACE_MODE_RMII\n");
+ break;
+
+ default:
+ dev_err(mac->dev, "Unsupported interface %d", plat_dat->interface);
+ return -EINVAL;
+ }
+
+ /* Update MAC PHY control register */
+ return regmap_update_bits(mac->regmap, 0, mac->soc_info->mask, 0);
+}
+
+static int x1600_mac_set_mode(struct plat_stmmacenet_data *plat_dat)
+{
+ struct ingenic_mac *mac = plat_dat->bsp_priv;
+ unsigned int val;
+
+ switch (plat_dat->interface) {
+ case PHY_INTERFACE_MODE_RMII:
+ val = FIELD_PREP(MACPHYC_PHY_INFT_MASK, MACPHYC_PHY_INFT_RMII);
+ dev_dbg(mac->dev, "MAC PHY Control Register: PHY_INTERFACE_MODE_RMII\n");
+ break;
+
+ default:
+ dev_err(mac->dev, "Unsupported interface %d", plat_dat->interface);
+ return -EINVAL;
+ }
+
+ /* Update MAC PHY control register */
+ return regmap_update_bits(mac->regmap, 0, mac->soc_info->mask, val);
+}
+
+static int x1830_mac_set_mode(struct plat_stmmacenet_data *plat_dat)
+{
+ struct ingenic_mac *mac = plat_dat->bsp_priv;
+ unsigned int val;
+
+ switch (plat_dat->interface) {
+ case PHY_INTERFACE_MODE_RMII:
+ val = FIELD_PREP(MACPHYC_MODE_SEL_MASK, MACPHYC_MODE_SEL_RMII) |
+ FIELD_PREP(MACPHYC_PHY_INFT_MASK, MACPHYC_PHY_INFT_RMII);
+ dev_dbg(mac->dev, "MAC PHY Control Register: PHY_INTERFACE_MODE_RMII\n");
+ break;
+
+ default:
+ dev_err(mac->dev, "Unsupported interface %d", plat_dat->interface);
+ return -EINVAL;
+ }
+
+ /* Update MAC PHY control register */
+ return regmap_update_bits(mac->regmap, 0, mac->soc_info->mask, val);
+}
+
+static int x2000_mac_set_mode(struct plat_stmmacenet_data *plat_dat)
+{
+ struct ingenic_mac *mac = plat_dat->bsp_priv;
+ unsigned int val;
+
+ switch (plat_dat->interface) {
+ case PHY_INTERFACE_MODE_RMII:
+ val = FIELD_PREP(MACPHYC_TX_SEL_MASK, MACPHYC_TX_SEL_ORIGIN) |
+ FIELD_PREP(MACPHYC_RX_SEL_MASK, MACPHYC_RX_SEL_ORIGIN) |
+ FIELD_PREP(MACPHYC_PHY_INFT_MASK, MACPHYC_PHY_INFT_RMII);
+ dev_dbg(mac->dev, "MAC PHY Control Register: PHY_INTERFACE_MODE_RMII\n");
+ break;
+
+ case PHY_INTERFACE_MODE_RGMII:
+ case PHY_INTERFACE_MODE_RGMII_ID:
+ case PHY_INTERFACE_MODE_RGMII_TXID:
+ case PHY_INTERFACE_MODE_RGMII_RXID:
+ val = FIELD_PREP(MACPHYC_PHY_INFT_MASK, MACPHYC_PHY_INFT_RGMII);
+
+ if (mac->tx_delay == 0)
+ val |= FIELD_PREP(MACPHYC_TX_SEL_MASK, MACPHYC_TX_SEL_ORIGIN);
+ else
+ val |= FIELD_PREP(MACPHYC_TX_SEL_MASK, MACPHYC_TX_SEL_DELAY) |
+ FIELD_PREP(MACPHYC_TX_DELAY_MASK, (mac->tx_delay + 9750) / 19500 - 1);
+
+ if (mac->rx_delay == 0)
+ val |= FIELD_PREP(MACPHYC_RX_SEL_MASK, MACPHYC_RX_SEL_ORIGIN);
+ else
+ val |= FIELD_PREP(MACPHYC_RX_SEL_MASK, MACPHYC_RX_SEL_DELAY) |
+ FIELD_PREP(MACPHYC_RX_DELAY_MASK, (mac->rx_delay + 9750) / 19500 - 1);
+
+ dev_dbg(mac->dev, "MAC PHY Control Register: PHY_INTERFACE_MODE_RGMII\n");
+ break;
+
+ default:
+ dev_err(mac->dev, "Unsupported interface %d", plat_dat->interface);
+ return -EINVAL;
+ }
+
+ /* Update MAC PHY control register */
+ return regmap_update_bits(mac->regmap, 0, mac->soc_info->mask, val);
+}
+
+static int ingenic_mac_probe(struct platform_device *pdev)
+{
+ struct plat_stmmacenet_data *plat_dat;
+ struct stmmac_resources stmmac_res;
+ struct ingenic_mac *mac;
+ const struct ingenic_soc_info *data;
+ u32 tx_delay_ps, rx_delay_ps;
+ int ret;
+
+ ret = stmmac_get_platform_resources(pdev, &stmmac_res);
+ if (ret)
+ return ret;
+
+ plat_dat = stmmac_probe_config_dt(pdev, stmmac_res.mac);
+ if (IS_ERR(plat_dat))
+ return PTR_ERR(plat_dat);
+
+ mac = devm_kzalloc(&pdev->dev, sizeof(*mac), GFP_KERNEL);
+ if (!mac) {
+ ret = -ENOMEM;
+ goto err_remove_config_dt;
+ }
+
+ data = of_device_get_match_data(&pdev->dev);
+ if (!data) {
+ dev_err(&pdev->dev, "No of match data provided\n");
+ ret = -EINVAL;
+ goto err_remove_config_dt;
+ }
+
+ /* Get MAC PHY control register */
+ mac->regmap = syscon_regmap_lookup_by_phandle(pdev->dev.of_node, "mode-reg");
+ if (IS_ERR(mac->regmap)) {
+ dev_err(&pdev->dev, "%s: Failed to get syscon regmap\n", __func__);
+ ret = PTR_ERR(mac->regmap);
+ goto err_remove_config_dt;
+ }
+
+ if (!of_property_read_u32(pdev->dev.of_node, "tx-clk-delay-ps", &tx_delay_ps)) {
+ if (tx_delay_ps >= MACPHYC_TX_DELAY_PS_MIN &&
+ tx_delay_ps <= MACPHYC_TX_DELAY_PS_MAX) {
+ mac->tx_delay = tx_delay_ps * 1000;
+ } else {
+ dev_err(&pdev->dev, "Invalid TX clock delay: %dps\n", tx_delay_ps);
+ return -EINVAL;
+ }
+ }
+
+ if (!of_property_read_u32(pdev->dev.of_node, "rx-clk-delay-ps", &rx_delay_ps)) {
+ if (rx_delay_ps >= MACPHYC_RX_DELAY_PS_MIN &&
+ rx_delay_ps <= MACPHYC_RX_DELAY_PS_MAX) {
+ mac->rx_delay = rx_delay_ps * 1000;
+ } else {
+ dev_err(&pdev->dev, "Invalid RX clock delay: %dps\n", rx_delay_ps);
+ return -EINVAL;
+ }
+ }
+
+ mac->soc_info = data;
+ mac->dev = &pdev->dev;
+
+ plat_dat->bsp_priv = mac;
+
+ ret = ingenic_mac_init(plat_dat);
+ if (ret)
+ goto err_remove_config_dt;
+
+ ret = stmmac_dvr_probe(&pdev->dev, plat_dat, &stmmac_res);
+ if (ret)
+ goto err_remove_config_dt;
+
+ return 0;
+
+err_remove_config_dt:
+ stmmac_remove_config_dt(pdev, plat_dat);
+
+ return ret;
+}
+
+#ifdef CONFIG_PM_SLEEP
+static int ingenic_mac_suspend(struct device *dev)
+{
+ int ret;
+
+ ret = stmmac_suspend(dev);
+
+ return ret;
+}
+
+static int ingenic_mac_resume(struct device *dev)
+{
+ struct net_device *ndev = dev_get_drvdata(dev);
+ struct stmmac_priv *priv = netdev_priv(ndev);
+ int ret;
+
+ ret = ingenic_mac_init(priv->plat);
+ if (ret)
+ return ret;
+
+ ret = stmmac_resume(dev);
+
+ return ret;
+}
+#endif /* CONFIG_PM_SLEEP */
+
+static SIMPLE_DEV_PM_OPS(ingenic_mac_pm_ops, ingenic_mac_suspend, ingenic_mac_resume);
+
+static struct ingenic_soc_info jz4775_soc_info = {
+ .version = ID_JZ4775,
+ .mask = MACPHYC_TXCLK_SEL_MASK | MACPHYC_SOFT_RST_MASK | MACPHYC_PHY_INFT_MASK,
+
+ .set_mode = jz4775_mac_set_mode,
+};
+
+static struct ingenic_soc_info x1000_soc_info = {
+ .version = ID_X1000,
+ .mask = MACPHYC_SOFT_RST_MASK,
+
+ .set_mode = x1000_mac_set_mode,
+};
+
+static struct ingenic_soc_info x1600_soc_info = {
+ .version = ID_X1600,
+ .mask = MACPHYC_SOFT_RST_MASK | MACPHYC_PHY_INFT_MASK,
+
+ .set_mode = x1600_mac_set_mode,
+};
+
+static struct ingenic_soc_info x1830_soc_info = {
+ .version = ID_X1830,
+ .mask = MACPHYC_MODE_SEL_MASK | MACPHYC_SOFT_RST_MASK | MACPHYC_PHY_INFT_MASK,
+
+ .set_mode = x1830_mac_set_mode,
+};
+
+static struct ingenic_soc_info x2000_soc_info = {
+ .version = ID_X2000,
+ .mask = MACPHYC_TX_SEL_MASK | MACPHYC_TX_DELAY_MASK | MACPHYC_RX_SEL_MASK |
+ MACPHYC_RX_DELAY_MASK | MACPHYC_SOFT_RST_MASK | MACPHYC_PHY_INFT_MASK,
+
+ .set_mode = x2000_mac_set_mode,
+};
+
+static const struct of_device_id ingenic_mac_of_matches[] = {
+ { .compatible = "ingenic,jz4775-mac", .data = &jz4775_soc_info },
+ { .compatible = "ingenic,x1000-mac", .data = &x1000_soc_info },
+ { .compatible = "ingenic,x1600-mac", .data = &x1600_soc_info },
+ { .compatible = "ingenic,x1830-mac", .data = &x1830_soc_info },
+ { .compatible = "ingenic,x2000-mac", .data = &x2000_soc_info },
+ { }
+};
+MODULE_DEVICE_TABLE(of, ingenic_mac_of_matches);
+
+static struct platform_driver ingenic_mac_driver = {
+ .probe = ingenic_mac_probe,
+ .remove = stmmac_pltfr_remove,
+ .driver = {
+ .name = "ingenic-mac",
+ .pm = pm_ptr(&ingenic_mac_pm_ops),
+ .of_match_table = ingenic_mac_of_matches,
+ },
+};
+module_platform_driver(ingenic_mac_driver);
+
+MODULE_AUTHOR("周琰杰 (Zhou Yanjie) <zhouyanjie@wanyeetech.com>");
+MODULE_DESCRIPTION("Ingenic SoCs DWMAC specific glue layer");
+MODULE_LICENSE("GPL v2");
#include "stmmac.h"
#include "stmmac_ptp.h"
-#define INTEL_MGBE_ADHOC_ADDR 0x15
-#define INTEL_MGBE_XPCS_ADDR 0x16
-
-/* Selection for PTP Clock Freq belongs to PSE & PCH GbE */
-#define PSE_PTP_CLK_FREQ_MASK (GMAC_GPO0 | GMAC_GPO3)
-#define PSE_PTP_CLK_FREQ_19_2MHZ (GMAC_GPO0)
-#define PSE_PTP_CLK_FREQ_200MHZ (GMAC_GPO0 | GMAC_GPO3)
-#define PSE_PTP_CLK_FREQ_256MHZ (0)
-#define PCH_PTP_CLK_FREQ_MASK (GMAC_GPO0)
-#define PCH_PTP_CLK_FREQ_19_2MHZ (GMAC_GPO0)
-#define PCH_PTP_CLK_FREQ_200MHZ (0)
-
-/* Cross-timestamping defines */
-#define ART_CPUID_LEAF 0x15
-#define EHL_PSE_ART_MHZ 19200000
-
struct intel_priv_data {
int mdio_adhoc_addr; /* mdio address for serdes & etc */
unsigned long crossts_adj;
serdes_phy_addr = intel_priv->mdio_adhoc_addr;
+ /* Set the serdes rate and the PCLK rate */
+ data = mdiobus_read(priv->mii, serdes_phy_addr,
+ SERDES_GCR0);
+
+ data &= ~SERDES_RATE_MASK;
+ data &= ~SERDES_PCLK_MASK;
+
+ if (priv->plat->max_speed == 2500)
+ data |= SERDES_RATE_PCIE_GEN2 << SERDES_RATE_PCIE_SHIFT |
+ SERDES_PCLK_37p5MHZ << SERDES_PCLK_SHIFT;
+ else
+ data |= SERDES_RATE_PCIE_GEN1 << SERDES_RATE_PCIE_SHIFT |
+ SERDES_PCLK_70MHZ << SERDES_PCLK_SHIFT;
+
+ mdiobus_write(priv->mii, serdes_phy_addr, SERDES_GCR0, data);
+
/* assert clk_req */
data = mdiobus_read(priv->mii, serdes_phy_addr, SERDES_GCR0);
data |= SERDES_PLL_CLK;
}
}
+static void intel_speed_mode_2500(struct net_device *ndev, void *intel_data)
+{
+ struct intel_priv_data *intel_priv = intel_data;
+ struct stmmac_priv *priv = netdev_priv(ndev);
+ int serdes_phy_addr = 0;
+ u32 data = 0;
+
+ serdes_phy_addr = intel_priv->mdio_adhoc_addr;
+
+ /* Determine the link speed mode: 2.5Gbps/1Gbps */
+ data = mdiobus_read(priv->mii, serdes_phy_addr,
+ SERDES_GCR);
+
+ if (((data & SERDES_LINK_MODE_MASK) >> SERDES_LINK_MODE_SHIFT) ==
+ SERDES_LINK_MODE_2G5) {
+ dev_info(priv->device, "Link Speed Mode: 2.5Gbps\n");
+ priv->plat->max_speed = 2500;
+ priv->plat->phy_interface = PHY_INTERFACE_MODE_2500BASEX;
+ priv->plat->mdio_bus_data->xpcs_an_inband = false;
+ } else {
+ priv->plat->max_speed = 1000;
+ priv->plat->phy_interface = PHY_INTERFACE_MODE_SGMII;
+ priv->plat->mdio_bus_data->xpcs_an_inband = true;
+ }
+}
+
/* Program PTP Clock Frequency for different variant of
* Intel mGBE that has slightly different GPO mapping
*/
plat->force_sf_dma_mode = 0;
plat->tso_en = 1;
+ /* Multiplying factor to the clk_eee_i clock time
+ * period to make it closer to 100 ns. This value
+ * should be programmed such that the clk_eee_time_period *
+ * (MULT_FACT_100NS + 1) should be within 80 ns to 120 ns
+ * clk_eee frequency is 19.2Mhz
+ * clk_eee_time_period is 52ns
+ * 52ns * (1 + 1) = 104ns
+ * MULT_FACT_100NS = 1
+ */
+ plat->mult_fact_100ns = 1;
+
plat->rx_sched_algorithm = MTL_RX_ALGORITHM_SP;
for (i = 0; i < plat->rx_queues_to_use; i++) {
plat->tx_queues_to_use = 8;
plat->clk_ptp_rate = 200000000;
+ plat->safety_feat_cfg->tsoee = 1;
+ plat->safety_feat_cfg->mrxpee = 1;
+ plat->safety_feat_cfg->mestee = 1;
+ plat->safety_feat_cfg->mrxee = 1;
+ plat->safety_feat_cfg->mtxee = 1;
+ plat->safety_feat_cfg->epsi = 0;
+ plat->safety_feat_cfg->edpp = 0;
+ plat->safety_feat_cfg->prtyen = 0;
+ plat->safety_feat_cfg->tmouten = 0;
+
return intel_mgbe_common_data(pdev, plat);
}
{
plat->bus_id = 1;
plat->phy_interface = PHY_INTERFACE_MODE_SGMII;
-
+ plat->speed_mode_2500 = intel_speed_mode_2500;
plat->serdes_powerup = intel_serdes_powerup;
plat->serdes_powerdown = intel_serdes_powerdown;
struct plat_stmmacenet_data *plat)
{
plat->phy_interface = PHY_INTERFACE_MODE_SGMII;
+ plat->speed_mode_2500 = intel_speed_mode_2500;
plat->serdes_powerup = intel_serdes_powerup;
plat->serdes_powerdown = intel_serdes_powerdown;
return ehl_pse0_common_data(pdev, plat);
struct plat_stmmacenet_data *plat)
{
plat->phy_interface = PHY_INTERFACE_MODE_SGMII;
+ plat->speed_mode_2500 = intel_speed_mode_2500;
plat->serdes_powerup = intel_serdes_powerup;
plat->serdes_powerdown = intel_serdes_powerdown;
return ehl_pse1_common_data(pdev, plat);
plat->tx_queues_to_use = 4;
plat->clk_ptp_rate = 200000000;
+ plat->safety_feat_cfg->tsoee = 1;
+ plat->safety_feat_cfg->mrxpee = 0;
+ plat->safety_feat_cfg->mestee = 1;
+ plat->safety_feat_cfg->mrxee = 1;
+ plat->safety_feat_cfg->mtxee = 1;
+ plat->safety_feat_cfg->epsi = 0;
+ plat->safety_feat_cfg->edpp = 0;
+ plat->safety_feat_cfg->prtyen = 0;
+ plat->safety_feat_cfg->tmouten = 0;
+
return intel_mgbe_common_data(pdev, plat);
}
{
plat->bus_id = 1;
plat->phy_interface = PHY_INTERFACE_MODE_SGMII;
+ plat->speed_mode_2500 = intel_speed_mode_2500;
plat->serdes_powerup = intel_serdes_powerup;
plat->serdes_powerdown = intel_serdes_powerdown;
return tgl_common_data(pdev, plat);
{
plat->bus_id = 2;
plat->phy_interface = PHY_INTERFACE_MODE_SGMII;
+ plat->speed_mode_2500 = intel_speed_mode_2500;
plat->serdes_powerup = intel_serdes_powerup;
plat->serdes_powerdown = intel_serdes_powerdown;
return tgl_common_data(pdev, plat);
if (!plat->dma_cfg)
return -ENOMEM;
+ plat->safety_feat_cfg = devm_kzalloc(&pdev->dev,
+ sizeof(*plat->safety_feat_cfg),
+ GFP_KERNEL);
+ if (!plat->safety_feat_cfg)
+ return -ENOMEM;
+
/* Enable pci device */
ret = pcim_enable_device(pdev);
if (ret) {
/**
* intel_eth_pci_remove
*
- * @pdev: platform device pointer
+ * @pdev: pci device pointer
* Description: this function calls the main to free the net resources
* and releases the PCI resources.
*/
#define POLL_DELAY_US 8
/* SERDES Register */
+#define SERDES_GCR 0x0 /* Global Conguration */
#define SERDES_GSR0 0x5 /* Global Status Reg0 */
#define SERDES_GCR0 0xb /* Global Configuration Reg0 */
#define SERDES_PHY_RX_CLK BIT(1) /* PSE SGMII PHY rx clk */
#define SERDES_RST BIT(2) /* Serdes Reset */
#define SERDES_PWR_ST_MASK GENMASK(6, 4) /* Serdes Power state*/
+#define SERDES_RATE_MASK GENMASK(9, 8)
+#define SERDES_PCLK_MASK GENMASK(14, 12) /* PCLK rate to PHY */
+#define SERDES_LINK_MODE_MASK GENMASK(2, 1)
+#define SERDES_LINK_MODE_SHIFT 1
#define SERDES_PWR_ST_SHIFT 4
#define SERDES_PWR_ST_P0 0x0
#define SERDES_PWR_ST_P3 0x3
+#define SERDES_LINK_MODE_2G5 0x3
+#define SERSED_LINK_MODE_1G 0x2
+#define SERDES_PCLK_37p5MHZ 0x0
+#define SERDES_PCLK_70MHZ 0x1
+#define SERDES_RATE_PCIE_GEN1 0x0
+#define SERDES_RATE_PCIE_GEN2 0x1
+#define SERDES_RATE_PCIE_SHIFT 8
+#define SERDES_PCLK_SHIFT 12
+
+#define INTEL_MGBE_ADHOC_ADDR 0x15
+#define INTEL_MGBE_XPCS_ADDR 0x16
+
+/* Cross-timestamping defines */
+#define ART_CPUID_LEAF 0x15
+#define EHL_PSE_ART_MHZ 19200000
+
+/* Selection for PTP Clock Freq belongs to PSE & PCH GbE */
+#define PSE_PTP_CLK_FREQ_MASK (GMAC_GPO0 | GMAC_GPO3)
+#define PSE_PTP_CLK_FREQ_19_2MHZ (GMAC_GPO0)
+#define PSE_PTP_CLK_FREQ_200MHZ (GMAC_GPO0 | GMAC_GPO3)
+#define PSE_PTP_CLK_FREQ_256MHZ (0)
+#define PCH_PTP_CLK_FREQ_MASK (GMAC_GPO0)
+#define PCH_PTP_CLK_FREQ_19_2MHZ (GMAC_GPO0)
+#define PCH_PTP_CLK_FREQ_200MHZ (0)
#endif /* __DWMAC_INTEL_H__ */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2020, Loongson Corporation
+ */
+
+#include <linux/clk-provider.h>
+#include <linux/pci.h>
+#include <linux/dmi.h>
+#include <linux/device.h>
+#include <linux/of_irq.h>
+#include "stmmac.h"
+
+static int loongson_default_data(struct plat_stmmacenet_data *plat)
+{
+ plat->clk_csr = 2; /* clk_csr_i = 20-35MHz & MDC = clk_csr_i/16 */
+ plat->has_gmac = 1;
+ plat->force_sf_dma_mode = 1;
+
+ /* Set default value for multicast hash bins */
+ plat->multicast_filter_bins = HASH_TABLE_SIZE;
+
+ /* Set default value for unicast filter entries */
+ plat->unicast_filter_entries = 1;
+
+ /* Set the maxmtu to a default of JUMBO_LEN */
+ plat->maxmtu = JUMBO_LEN;
+
+ /* Set default number of RX and TX queues to use */
+ plat->tx_queues_to_use = 1;
+ plat->rx_queues_to_use = 1;
+
+ /* Disable Priority config by default */
+ plat->tx_queues_cfg[0].use_prio = false;
+ plat->rx_queues_cfg[0].use_prio = false;
+
+ /* Disable RX queues routing by default */
+ plat->rx_queues_cfg[0].pkt_route = 0x0;
+
+ /* Default to phy auto-detection */
+ plat->phy_addr = -1;
+
+ plat->dma_cfg->pbl = 32;
+ plat->dma_cfg->pblx8 = true;
+
+ plat->multicast_filter_bins = 256;
+ return 0;
+}
+
+static int loongson_dwmac_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+ struct plat_stmmacenet_data *plat;
+ struct stmmac_resources res;
+ bool mdio = false;
+ int ret, i;
+ struct device_node *np;
+
+ np = dev_of_node(&pdev->dev);
+
+ if (!np) {
+ pr_info("dwmac_loongson_pci: No OF node\n");
+ return -ENODEV;
+ }
+
+ if (!of_device_is_compatible(np, "loongson, pci-gmac")) {
+ pr_info("dwmac_loongson_pci: Incompatible OF node\n");
+ return -ENODEV;
+ }
+
+ plat = devm_kzalloc(&pdev->dev, sizeof(*plat), GFP_KERNEL);
+ if (!plat)
+ return -ENOMEM;
+
+ if (plat->mdio_node) {
+ dev_err(&pdev->dev, "Found MDIO subnode\n");
+ mdio = true;
+ }
+
+ if (mdio) {
+ plat->mdio_bus_data = devm_kzalloc(&pdev->dev,
+ sizeof(*plat->mdio_bus_data),
+ GFP_KERNEL);
+ if (!plat->mdio_bus_data)
+ return -ENOMEM;
+ plat->mdio_bus_data->needs_reset = true;
+ }
+
+ plat->dma_cfg = devm_kzalloc(&pdev->dev, sizeof(*plat->dma_cfg), GFP_KERNEL);
+ if (!plat->dma_cfg)
+ return -ENOMEM;
+
+ /* Enable pci device */
+ ret = pci_enable_device(pdev);
+ if (ret) {
+ dev_err(&pdev->dev, "%s: ERROR: failed to enable device\n", __func__);
+ return ret;
+ }
+
+ /* Get the base address of device */
+ for (i = 0; i < PCI_STD_NUM_BARS; i++) {
+ if (pci_resource_len(pdev, i) == 0)
+ continue;
+ ret = pcim_iomap_regions(pdev, BIT(0), pci_name(pdev));
+ if (ret)
+ return ret;
+ break;
+ }
+
+ plat->bus_id = of_alias_get_id(np, "ethernet");
+ if (plat->bus_id < 0)
+ plat->bus_id = pci_dev_id(pdev);
+
+ plat->phy_interface = device_get_phy_mode(&pdev->dev);
+ if (plat->phy_interface < 0)
+ dev_err(&pdev->dev, "phy_mode not found\n");
+
+ plat->interface = PHY_INTERFACE_MODE_GMII;
+
+ pci_set_master(pdev);
+
+ loongson_default_data(plat);
+ pci_enable_msi(pdev);
+ memset(&res, 0, sizeof(res));
+ res.addr = pcim_iomap_table(pdev)[0];
+
+ res.irq = of_irq_get_byname(np, "macirq");
+ if (res.irq < 0) {
+ dev_err(&pdev->dev, "IRQ macirq not found\n");
+ ret = -ENODEV;
+ }
+
+ res.wol_irq = of_irq_get_byname(np, "eth_wake_irq");
+ if (res.wol_irq < 0) {
+ dev_info(&pdev->dev, "IRQ eth_wake_irq not found, using macirq\n");
+ res.wol_irq = res.irq;
+ }
+
+ res.lpi_irq = of_irq_get_byname(np, "eth_lpi");
+ if (res.lpi_irq < 0) {
+ dev_err(&pdev->dev, "IRQ eth_lpi not found\n");
+ ret = -ENODEV;
+ }
+
+ return stmmac_dvr_probe(&pdev->dev, plat, &res);
+}
+
+static void loongson_dwmac_remove(struct pci_dev *pdev)
+{
+ int i;
+
+ stmmac_dvr_remove(&pdev->dev);
+
+ for (i = 0; i < PCI_STD_NUM_BARS; i++) {
+ if (pci_resource_len(pdev, i) == 0)
+ continue;
+ pcim_iounmap_regions(pdev, BIT(i));
+ break;
+ }
+
+ pci_disable_device(pdev);
+}
+
+static int __maybe_unused loongson_dwmac_suspend(struct device *dev)
+{
+ struct pci_dev *pdev = to_pci_dev(dev);
+ int ret;
+
+ ret = stmmac_suspend(dev);
+ if (ret)
+ return ret;
+
+ ret = pci_save_state(pdev);
+ if (ret)
+ return ret;
+
+ pci_disable_device(pdev);
+ pci_wake_from_d3(pdev, true);
+ return 0;
+}
+
+static int __maybe_unused loongson_dwmac_resume(struct device *dev)
+{
+ struct pci_dev *pdev = to_pci_dev(dev);
+ int ret;
+
+ pci_restore_state(pdev);
+ pci_set_power_state(pdev, PCI_D0);
+
+ ret = pci_enable_device(pdev);
+ if (ret)
+ return ret;
+
+ pci_set_master(pdev);
+
+ return stmmac_resume(dev);
+}
+
+static SIMPLE_DEV_PM_OPS(loongson_dwmac_pm_ops, loongson_dwmac_suspend,
+ loongson_dwmac_resume);
+
+static const struct pci_device_id loongson_dwmac_id_table[] = {
+ { PCI_VDEVICE(LOONGSON, 0x7a03) },
+ {}
+};
+MODULE_DEVICE_TABLE(pci, loongson_dwmac_id_table);
+
+struct pci_driver loongson_dwmac_driver = {
+ .name = "dwmac-loongson-pci",
+ .id_table = loongson_dwmac_id_table,
+ .probe = loongson_dwmac_probe,
+ .remove = loongson_dwmac_remove,
+ .driver = {
+ .pm = &loongson_dwmac_pm_ops,
+ },
+};
+
+module_pci_driver(loongson_dwmac_driver);
+
+MODULE_DESCRIPTION("Loongson DWMAC PCI driver");
+MODULE_AUTHOR("Qing Zhang <zhangqing@loongson.cn>");
+MODULE_LICENSE("GPL v2");
struct stmmac_resources stmmac_res;
const struct ethqos_emac_driver_data *data;
struct qcom_ethqos *ethqos;
- struct resource *res;
int ret;
ret = stmmac_get_platform_resources(pdev, &stmmac_res);
}
ethqos->pdev = pdev;
- res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "rgmii");
- ethqos->rgmii_base = devm_ioremap_resource(&pdev->dev, res);
+ ethqos->rgmii_base = devm_platform_ioremap_resource_byname(pdev, "rgmii");
if (IS_ERR(ethqos->rgmii_base)) {
ret = PTR_ERR(ethqos->rgmii_base);
goto err_mem;
void (*set_rgmii_speed)(struct rk_priv_data *bsp_priv, int speed);
void (*set_rmii_speed)(struct rk_priv_data *bsp_priv, int speed);
void (*integrated_phy_powerup)(struct rk_priv_data *bsp_priv);
+ u32 regs[];
};
struct rk_priv_data {
struct platform_device *pdev;
phy_interface_t phy_iface;
+ int id;
struct regulator *regulator;
bool suspended;
const struct rk_gmac_ops *ops;
.set_rmii_speed = rk3288_set_rmii_speed,
};
+#define RK3308_GRF_MAC_CON0 0x04a0
+
+/* RK3308_GRF_MAC_CON0 */
+#define RK3308_GMAC_PHY_INTF_SEL_RMII (GRF_CLR_BIT(2) | GRF_CLR_BIT(3) | \
+ GRF_BIT(4))
+#define RK3308_GMAC_FLOW_CTRL GRF_BIT(3)
+#define RK3308_GMAC_FLOW_CTRL_CLR GRF_CLR_BIT(3)
+#define RK3308_GMAC_SPEED_10M GRF_CLR_BIT(0)
+#define RK3308_GMAC_SPEED_100M GRF_BIT(0)
+
+static void rk3308_set_to_rmii(struct rk_priv_data *bsp_priv)
+{
+ struct device *dev = &bsp_priv->pdev->dev;
+
+ if (IS_ERR(bsp_priv->grf)) {
+ dev_err(dev, "Missing rockchip,grf property\n");
+ return;
+ }
+
+ regmap_write(bsp_priv->grf, RK3308_GRF_MAC_CON0,
+ RK3308_GMAC_PHY_INTF_SEL_RMII);
+}
+
+static void rk3308_set_rmii_speed(struct rk_priv_data *bsp_priv, int speed)
+{
+ struct device *dev = &bsp_priv->pdev->dev;
+
+ if (IS_ERR(bsp_priv->grf)) {
+ dev_err(dev, "Missing rockchip,grf property\n");
+ return;
+ }
+
+ if (speed == 10) {
+ regmap_write(bsp_priv->grf, RK3308_GRF_MAC_CON0,
+ RK3308_GMAC_SPEED_10M);
+ } else if (speed == 100) {
+ regmap_write(bsp_priv->grf, RK3308_GRF_MAC_CON0,
+ RK3308_GMAC_SPEED_100M);
+ } else {
+ dev_err(dev, "unknown speed value for RMII! speed=%d", speed);
+ }
+}
+
+static const struct rk_gmac_ops rk3308_ops = {
+ .set_to_rmii = rk3308_set_to_rmii,
+ .set_rmii_speed = rk3308_set_rmii_speed,
+};
+
#define RK3328_GRF_MAC_CON0 0x0900
#define RK3328_GRF_MAC_CON1 0x0904
#define RK3328_GRF_MAC_CON2 0x0908
.set_rmii_speed = rk3399_set_rmii_speed,
};
+#define RK3568_GRF_GMAC0_CON0 0x0380
+#define RK3568_GRF_GMAC0_CON1 0x0384
+#define RK3568_GRF_GMAC1_CON0 0x0388
+#define RK3568_GRF_GMAC1_CON1 0x038c
+
+/* RK3568_GRF_GMAC0_CON1 && RK3568_GRF_GMAC1_CON1 */
+#define RK3568_GMAC_PHY_INTF_SEL_RGMII \
+ (GRF_BIT(4) | GRF_CLR_BIT(5) | GRF_CLR_BIT(6))
+#define RK3568_GMAC_PHY_INTF_SEL_RMII \
+ (GRF_CLR_BIT(4) | GRF_CLR_BIT(5) | GRF_BIT(6))
+#define RK3568_GMAC_FLOW_CTRL GRF_BIT(3)
+#define RK3568_GMAC_FLOW_CTRL_CLR GRF_CLR_BIT(3)
+#define RK3568_GMAC_RXCLK_DLY_ENABLE GRF_BIT(1)
+#define RK3568_GMAC_RXCLK_DLY_DISABLE GRF_CLR_BIT(1)
+#define RK3568_GMAC_TXCLK_DLY_ENABLE GRF_BIT(0)
+#define RK3568_GMAC_TXCLK_DLY_DISABLE GRF_CLR_BIT(0)
+
+/* RK3568_GRF_GMAC0_CON0 && RK3568_GRF_GMAC1_CON0 */
+#define RK3568_GMAC_CLK_RX_DL_CFG(val) HIWORD_UPDATE(val, 0x7F, 8)
+#define RK3568_GMAC_CLK_TX_DL_CFG(val) HIWORD_UPDATE(val, 0x7F, 0)
+
+static void rk3568_set_to_rgmii(struct rk_priv_data *bsp_priv,
+ int tx_delay, int rx_delay)
+{
+ struct device *dev = &bsp_priv->pdev->dev;
+ u32 con0, con1;
+
+ if (IS_ERR(bsp_priv->grf)) {
+ dev_err(dev, "Missing rockchip,grf property\n");
+ return;
+ }
+
+ con0 = (bsp_priv->id == 1) ? RK3568_GRF_GMAC1_CON0 :
+ RK3568_GRF_GMAC0_CON0;
+ con1 = (bsp_priv->id == 1) ? RK3568_GRF_GMAC1_CON1 :
+ RK3568_GRF_GMAC0_CON1;
+
+ regmap_write(bsp_priv->grf, con0,
+ RK3568_GMAC_CLK_RX_DL_CFG(rx_delay) |
+ RK3568_GMAC_CLK_TX_DL_CFG(tx_delay));
+
+ regmap_write(bsp_priv->grf, con1,
+ RK3568_GMAC_PHY_INTF_SEL_RGMII |
+ RK3568_GMAC_RXCLK_DLY_ENABLE |
+ RK3568_GMAC_TXCLK_DLY_ENABLE);
+}
+
+static void rk3568_set_to_rmii(struct rk_priv_data *bsp_priv)
+{
+ struct device *dev = &bsp_priv->pdev->dev;
+ u32 con1;
+
+ if (IS_ERR(bsp_priv->grf)) {
+ dev_err(dev, "%s: Missing rockchip,grf property\n", __func__);
+ return;
+ }
+
+ con1 = (bsp_priv->id == 1) ? RK3568_GRF_GMAC1_CON1 :
+ RK3568_GRF_GMAC0_CON1;
+ regmap_write(bsp_priv->grf, con1, RK3568_GMAC_PHY_INTF_SEL_RMII);
+}
+
+static void rk3568_set_gmac_speed(struct rk_priv_data *bsp_priv, int speed)
+{
+ struct device *dev = &bsp_priv->pdev->dev;
+ unsigned long rate;
+ int ret;
+
+ switch (speed) {
+ case 10:
+ rate = 2500000;
+ break;
+ case 100:
+ rate = 25000000;
+ break;
+ case 1000:
+ rate = 125000000;
+ break;
+ default:
+ dev_err(dev, "unknown speed value for GMAC speed=%d", speed);
+ return;
+ }
+
+ ret = clk_set_rate(bsp_priv->clk_mac_speed, rate);
+ if (ret)
+ dev_err(dev, "%s: set clk_mac_speed rate %ld failed %d\n",
+ __func__, rate, ret);
+}
+
+static const struct rk_gmac_ops rk3568_ops = {
+ .set_to_rgmii = rk3568_set_to_rgmii,
+ .set_to_rmii = rk3568_set_to_rmii,
+ .set_rgmii_speed = rk3568_set_gmac_speed,
+ .set_rmii_speed = rk3568_set_gmac_speed,
+ .regs = {
+ 0xfe2a0000, /* gmac0 */
+ 0xfe010000, /* gmac1 */
+ 0x0, /* sentinel */
+ },
+};
+
#define RV1108_GRF_GMAC_CON0 0X0900
/* RV1108_GRF_GMAC_CON0 */
{
struct rk_priv_data *bsp_priv;
struct device *dev = &pdev->dev;
+ struct resource *res;
int ret;
const char *strings = NULL;
int value;
of_get_phy_mode(dev->of_node, &bsp_priv->phy_iface);
bsp_priv->ops = ops;
+ /* Some SoCs have multiple MAC controllers, which need
+ * to be distinguished.
+ */
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (res) {
+ int i = 0;
+
+ while (ops->regs[i]) {
+ if (ops->regs[i] == res->start) {
+ bsp_priv->id = i;
+ break;
+ }
+ i++;
+ }
+ }
+
bsp_priv->regulator = devm_regulator_get_optional(dev, "phy");
if (IS_ERR(bsp_priv->regulator)) {
if (PTR_ERR(bsp_priv->regulator) == -EPROBE_DEFER) {
return bsp_priv;
}
+static int rk_gmac_check_ops(struct rk_priv_data *bsp_priv)
+{
+ switch (bsp_priv->phy_iface) {
+ case PHY_INTERFACE_MODE_RGMII:
+ case PHY_INTERFACE_MODE_RGMII_ID:
+ case PHY_INTERFACE_MODE_RGMII_RXID:
+ case PHY_INTERFACE_MODE_RGMII_TXID:
+ if (!bsp_priv->ops->set_to_rgmii)
+ return -EINVAL;
+ break;
+ case PHY_INTERFACE_MODE_RMII:
+ if (!bsp_priv->ops->set_to_rmii)
+ return -EINVAL;
+ break;
+ default:
+ dev_err(&bsp_priv->pdev->dev,
+ "unsupported interface %d", bsp_priv->phy_iface);
+ }
+ return 0;
+}
+
static int rk_gmac_powerup(struct rk_priv_data *bsp_priv)
{
int ret;
struct device *dev = &bsp_priv->pdev->dev;
+ ret = rk_gmac_check_ops(bsp_priv);
+ if (ret)
+ return ret;
+
ret = gmac_clk_enable(bsp_priv, true);
if (ret)
return ret;
case PHY_INTERFACE_MODE_RGMII_ID:
case PHY_INTERFACE_MODE_RGMII_RXID:
case PHY_INTERFACE_MODE_RGMII_TXID:
- bsp_priv->ops->set_rgmii_speed(bsp_priv, speed);
+ if (bsp_priv->ops->set_rgmii_speed)
+ bsp_priv->ops->set_rgmii_speed(bsp_priv, speed);
break;
case PHY_INTERFACE_MODE_RMII:
- bsp_priv->ops->set_rmii_speed(bsp_priv, speed);
+ if (bsp_priv->ops->set_rmii_speed)
+ bsp_priv->ops->set_rmii_speed(bsp_priv, speed);
break;
default:
dev_err(dev, "unsupported interface %d", bsp_priv->phy_iface);
if (IS_ERR(plat_dat))
return PTR_ERR(plat_dat);
- plat_dat->has_gmac = true;
+ /* If the stmmac is not already selected as gmac4,
+ * then make sure we fallback to gmac.
+ */
+ if (!plat_dat->has_gmac4)
+ plat_dat->has_gmac = true;
plat_dat->fix_mac_speed = rk_fix_speed;
plat_dat->bsp_priv = rk_gmac_setup(pdev, plat_dat, data);
{ .compatible = "rockchip,rk3128-gmac", .data = &rk3128_ops },
{ .compatible = "rockchip,rk3228-gmac", .data = &rk3228_ops },
{ .compatible = "rockchip,rk3288-gmac", .data = &rk3288_ops },
+ { .compatible = "rockchip,rk3308-gmac", .data = &rk3308_ops },
{ .compatible = "rockchip,rk3328-gmac", .data = &rk3328_ops },
{ .compatible = "rockchip,rk3366-gmac", .data = &rk3366_ops },
{ .compatible = "rockchip,rk3368-gmac", .data = &rk3368_ops },
{ .compatible = "rockchip,rk3399-gmac", .data = &rk3399_ops },
+ { .compatible = "rockchip,rk3568-gmac", .data = &rk3568_ops },
{ .compatible = "rockchip,rv1108-gmac", .data = &rv1108_ops },
{ }
};
static int sun7i_gmac_init(struct platform_device *pdev, void *priv)
{
struct sunxi_priv_data *gmac = priv;
- int ret;
+ int ret = 0;
if (gmac->regulator) {
ret = regulator_enable(gmac->regulator);
} else {
clk_set_rate(gmac->tx_clk, SUN7I_GMAC_MII_RATE);
ret = clk_prepare(gmac->tx_clk);
- if (ret)
- return ret;
+ if (ret && gmac->regulator)
+ regulator_disable(gmac->regulator);
}
- return 0;
+ return ret;
}
static void sun7i_gmac_exit(struct platform_device *pdev, void *priv)
#define LPI_CTRL_STATUS_TLPIEN 0x00000001 /* Transmit LPI Entry */
/* GMAC HW ADDR regs */
-#define GMAC_ADDR_HIGH(reg) (((reg > 15) ? 0x00000800 : 0x00000040) + \
- (reg * 8))
-#define GMAC_ADDR_LOW(reg) (((reg > 15) ? 0x00000804 : 0x00000044) + \
- (reg * 8))
+#define GMAC_ADDR_HIGH(reg) ((reg > 15) ? 0x00000800 + (reg - 16) * 8 : \
+ 0x00000040 + (reg * 8))
+#define GMAC_ADDR_LOW(reg) ((reg > 15) ? 0x00000804 + (reg - 16) * 8 : \
+ 0x00000044 + (reg * 8))
#define GMAC_MAX_PERFECT_ADDRESSES 1
#define GMAC_PCS_BASE 0x000000c0 /* PCS register base */
mac->link.speed10 = GMAC_CONFIG_PS;
mac->link.speed100 = GMAC_CONFIG_FES | GMAC_CONFIG_PS;
mac->link.speed1000 = 0;
+ mac->link.speed2500 = GMAC_CONFIG_FES;
mac->link.speed_mask = GMAC_CONFIG_FES | GMAC_CONFIG_PS;
mac->mii.addr = GMAC_MDIO_ADDR;
mac->mii.data = GMAC_MDIO_DATA;
STAT_OFF(dma_errors), stats);
}
-int dwmac5_safety_feat_config(void __iomem *ioaddr, unsigned int asp)
+int dwmac5_safety_feat_config(void __iomem *ioaddr, unsigned int asp,
+ struct stmmac_safety_feature_cfg *safety_feat_cfg)
{
u32 value;
/* 1. Enable Safety Features */
value = readl(ioaddr + MTL_ECC_CONTROL);
value |= MEEAO; /* MTL ECC Error Addr Status Override */
- value |= TSOEE; /* TSO ECC */
- value |= MRXPEE; /* MTL RX Parser ECC */
- value |= MESTEE; /* MTL EST ECC */
- value |= MRXEE; /* MTL RX FIFO ECC */
- value |= MTXEE; /* MTL TX FIFO ECC */
+ if (safety_feat_cfg->tsoee)
+ value |= TSOEE; /* TSO ECC */
+ if (safety_feat_cfg->mrxpee)
+ value |= MRXPEE; /* MTL RX Parser ECC */
+ if (safety_feat_cfg->mestee)
+ value |= MESTEE; /* MTL EST ECC */
+ if (safety_feat_cfg->mrxee)
+ value |= MRXEE; /* MTL RX FIFO ECC */
+ if (safety_feat_cfg->mtxee)
+ value |= MTXEE; /* MTL TX FIFO ECC */
writel(value, ioaddr + MTL_ECC_CONTROL);
/* 2. Enable MTL Safety Interrupts */
/* 5. Enable Parity and Timeout for FSM */
value = readl(ioaddr + MAC_FSM_CONTROL);
- value |= PRTYEN; /* FSM Parity Feature */
- value |= TMOUTEN; /* FSM Timeout Feature */
+ if (safety_feat_cfg->prtyen)
+ value |= PRTYEN; /* FSM Parity Feature */
+ if (safety_feat_cfg->tmouten)
+ value |= TMOUTEN; /* FSM Timeout Feature */
writel(value, ioaddr + MAC_FSM_CONTROL);
/* 4. Enable Data Parity Protection */
value = readl(ioaddr + MTL_DPP_CONTROL);
- value |= EDPP;
+ if (safety_feat_cfg->edpp)
+ value |= EDPP;
writel(value, ioaddr + MTL_DPP_CONTROL);
/*
if (asp <= 0x2)
return 0;
- value |= EPSI;
+ if (safety_feat_cfg->epsi)
+ value |= EPSI;
writel(value, ioaddr + MTL_DPP_CONTROL);
return 0;
}
#define GMAC_INT_FPE_EN BIT(17)
-int dwmac5_safety_feat_config(void __iomem *ioaddr, unsigned int asp);
+int dwmac5_safety_feat_config(void __iomem *ioaddr, unsigned int asp,
+ struct stmmac_safety_feature_cfg *safety_cfg);
int dwmac5_safety_feat_irq_status(struct net_device *ndev,
void __iomem *ioaddr, unsigned int asp,
struct stmmac_safety_stats *stats);
dwxgmac3_dma_errors, STAT_OFF(dma_errors), stats);
}
-static int dwxgmac3_safety_feat_config(void __iomem *ioaddr, unsigned int asp)
+static int
+dwxgmac3_safety_feat_config(void __iomem *ioaddr, unsigned int asp,
+ struct stmmac_safety_feature_cfg *safety_cfg)
{
u32 value;
void (*pcs_rane)(void __iomem *ioaddr, bool restart);
void (*pcs_get_adv_lp)(void __iomem *ioaddr, struct rgmii_adv *adv);
/* Safety Features */
- int (*safety_feat_config)(void __iomem *ioaddr, unsigned int asp);
+ int (*safety_feat_config)(void __iomem *ioaddr, unsigned int asp,
+ struct stmmac_safety_feature_cfg *safety_cfg);
int (*safety_feat_irq_status)(struct net_device *ndev,
void __iomem *ioaddr, unsigned int asp,
struct stmmac_safety_stats *stats);
#define stmmac_mmc_read(__priv, __args...) \
stmmac_do_void_callback(__priv, mmc, read, __args)
-/* XPCS callbacks */
-#define stmmac_xpcs_validate(__priv, __args...) \
- stmmac_do_callback(__priv, xpcs, validate, __args)
-#define stmmac_xpcs_config(__priv, __args...) \
- stmmac_do_callback(__priv, xpcs, config, __args)
-#define stmmac_xpcs_get_state(__priv, __args...) \
- stmmac_do_callback(__priv, xpcs, get_state, __args)
-#define stmmac_xpcs_link_up(__priv, __args...) \
- stmmac_do_callback(__priv, xpcs, link_up, __args)
-#define stmmac_xpcs_probe(__priv, __args...) \
- stmmac_do_callback(__priv, xpcs, probe, __args)
-
struct stmmac_regs_off {
u32 ptp_off;
u32 mmc_off;
unsigned int cur_tx;
unsigned int dirty_tx;
dma_addr_t dma_tx_phy;
- u32 tx_tail_addr;
+ dma_addr_t tx_tail_addr;
u32 mss;
};
int stmmac_mdio_unregister(struct net_device *ndev);
int stmmac_mdio_register(struct net_device *ndev);
int stmmac_mdio_reset(struct mii_bus *mii);
+int stmmac_xpcs_setup(struct mii_bus *mii);
void stmmac_set_ethtool_ops(struct net_device *netdev);
void stmmac_ptp_register(struct stmmac_priv *priv);
static inline unsigned int stmmac_rx_offset(struct stmmac_priv *priv)
{
if (stmmac_xdp_is_enabled(priv))
- return XDP_PACKET_HEADROOM;
+ return XDP_PACKET_HEADROOM + NET_IP_ALIGN;
- return 0;
+ return NET_SKB_PAD + NET_IP_ALIGN;
}
void stmmac_disable_rx_queue(struct stmmac_priv *priv, u32 queue);
netdev_warn(priv->dev,
"Setting EEE tx-lpi is not supported\n");
+ if (priv->hw->xpcs) {
+ ret = xpcs_config_eee(priv->hw->xpcs,
+ priv->plat->mult_fact_100ns,
+ edata->eee_enabled);
+ if (ret)
+ return ret;
+ }
+
if (!edata->eee_enabled)
stmmac_disable_eee_mode(priv);
if ((max_speed > 0) && (max_speed < 1000)) {
phylink_set(mask, 1000baseT_Full);
phylink_set(mask, 1000baseX_Full);
+ } else if (priv->plat->has_gmac4) {
+ if (!max_speed || max_speed >= 2500) {
+ phylink_set(mac_supported, 2500baseT_Full);
+ phylink_set(mac_supported, 2500baseX_Full);
+ }
} else if (priv->plat->has_xgmac) {
if (!max_speed || (max_speed >= 2500)) {
phylink_set(mac_supported, 2500baseT_Full);
linkmode_andnot(state->advertising, state->advertising, mask);
/* If PCS is supported, check which modes it supports. */
- stmmac_xpcs_validate(priv, &priv->hw->xpcs_args, supported, state);
-}
-
-static void stmmac_mac_pcs_get_state(struct phylink_config *config,
- struct phylink_link_state *state)
-{
- struct stmmac_priv *priv = netdev_priv(to_net_dev(config->dev));
-
- state->link = 0;
- stmmac_xpcs_get_state(priv, &priv->hw->xpcs_args, state);
+ if (priv->hw->xpcs)
+ xpcs_validate(priv->hw->xpcs, supported, state);
}
static void stmmac_mac_config(struct phylink_config *config, unsigned int mode,
const struct phylink_link_state *state)
{
- struct stmmac_priv *priv = netdev_priv(to_net_dev(config->dev));
-
- stmmac_xpcs_config(priv, &priv->hw->xpcs_args, state);
-}
-
-static void stmmac_mac_an_restart(struct phylink_config *config)
-{
- /* Not Supported */
+ /* Nothing to do, xpcs_config() handles everything */
}
static void stmmac_fpe_link_state_handle(struct stmmac_priv *priv, bool is_up)
if (is_up && *hs_enable) {
stmmac_fpe_send_mpacket(priv, priv->ioaddr, MPACKET_VERIFY);
} else {
- *lo_state = FPE_EVENT_UNKNOWN;
- *lp_state = FPE_EVENT_UNKNOWN;
+ *lo_state = FPE_STATE_OFF;
+ *lp_state = FPE_STATE_OFF;
}
}
struct stmmac_priv *priv = netdev_priv(to_net_dev(config->dev));
u32 ctrl;
- stmmac_xpcs_link_up(priv, &priv->hw->xpcs_args, speed, interface);
-
ctrl = readl(priv->ioaddr + MAC_CTRL_REG);
ctrl &= ~priv->hw->link.speed_mask;
static const struct phylink_mac_ops stmmac_phylink_mac_ops = {
.validate = stmmac_validate,
- .mac_pcs_get_state = stmmac_mac_pcs_get_state,
.mac_config = stmmac_mac_config,
- .mac_an_restart = stmmac_mac_an_restart,
.mac_link_down = stmmac_mac_link_down,
.mac_link_up = stmmac_mac_link_up,
};
*/
static int stmmac_init_phy(struct net_device *dev)
{
- struct ethtool_wolinfo wol = { .cmd = ETHTOOL_GWOL };
struct stmmac_priv *priv = netdev_priv(dev);
struct device_node *node;
int ret;
ret = phylink_connect_phy(priv->phylink, phydev);
}
- phylink_ethtool_get_wol(priv->phylink, &wol);
- device_set_wakeup_capable(priv->device, !!wol.supported);
+ if (!priv->plat->pmt) {
+ struct ethtool_wolinfo wol = { .cmd = ETHTOOL_GWOL };
+
+ phylink_ethtool_get_wol(priv->phylink, &wol);
+ device_set_wakeup_capable(priv->device, !!wol.supported);
+ }
return ret;
}
static int stmmac_phy_setup(struct stmmac_priv *priv)
{
+ struct stmmac_mdio_bus_data *mdio_bus_data = priv->plat->mdio_bus_data;
struct fwnode_handle *fwnode = of_fwnode_handle(priv->plat->phylink_node);
int mode = priv->plat->phy_interface;
struct phylink *phylink;
priv->phylink_config.dev = &priv->dev->dev;
priv->phylink_config.type = PHYLINK_NETDEV;
priv->phylink_config.pcs_poll = true;
- priv->phylink_config.ovr_an_inband =
- priv->plat->mdio_bus_data->xpcs_an_inband;
+ if (priv->plat->mdio_bus_data)
+ priv->phylink_config.ovr_an_inband =
+ mdio_bus_data->xpcs_an_inband;
if (!fwnode)
fwnode = dev_fwnode(priv->device);
if (IS_ERR(phylink))
return PTR_ERR(phylink);
+ if (priv->hw->xpcs)
+ phylink_set_pcs(phylink, &priv->hw->xpcs->pcs);
+
priv->phylink = phylink;
return 0;
}
{
if (priv->dma_cap.asp) {
netdev_info(priv->dev, "Enabling Safety Features\n");
- stmmac_safety_feat_config(priv, priv->ioaddr, priv->dma_cap.asp);
+ stmmac_safety_feat_config(priv, priv->ioaddr, priv->dma_cap.asp,
+ priv->plat->safety_feat_cfg);
} else {
netdev_info(priv->dev, "No Safety Features support found\n");
}
static int stmmac_request_irq_multi_msi(struct net_device *dev)
{
- enum request_irq_err irq_err = REQ_IRQ_ERR_NO;
struct stmmac_priv *priv = netdev_priv(dev);
+ enum request_irq_err irq_err;
cpumask_t cpu_mask;
int irq_idx = 0;
char *int_name;
static int stmmac_request_irq_single(struct net_device *dev)
{
- enum request_irq_err irq_err = REQ_IRQ_ERR_NO;
struct stmmac_priv *priv = netdev_priv(dev);
+ enum request_irq_err irq_err;
int ret;
ret = request_irq(dev->irq, stmmac_interrupt,
"%s: ERROR: allocating the IRQ %d (error: %d)\n",
__func__, dev->irq, ret);
irq_err = REQ_IRQ_ERR_MAC;
- return ret;
+ goto irq_error;
}
/* Request the Wake IRQ in case of another line
"%s: ERROR: allocating the WoL IRQ %d (%d)\n",
__func__, priv->wol_irq, ret);
irq_err = REQ_IRQ_ERR_WOL;
- return ret;
+ goto irq_error;
}
}
int stmmac_open(struct net_device *dev)
{
struct stmmac_priv *priv = netdev_priv(dev);
+ int mode = priv->plat->phy_interface;
int bfsize = 0;
u32 chan;
int ret;
if (priv->hw->pcs != STMMAC_PCS_TBI &&
priv->hw->pcs != STMMAC_PCS_RTBI &&
- priv->hw->xpcs_args.an_mode != DW_AN_C73) {
+ (!priv->hw->xpcs ||
+ xpcs_get_an_mode(priv->hw->xpcs, mode) != DW_AN_C73)) {
ret = stmmac_init_phy(dev);
if (ret) {
netdev_err(priv->dev,
/* Buffer is good. Go on. */
- prefetch(page_address(buf->page));
+ prefetch(page_address(buf->page) + buf->page_offset);
if (buf->sec_page)
prefetch(page_address(buf->sec_page));
dma_sync_single_for_cpu(priv->device, buf->addr,
buf1_len, dma_dir);
- xdp.data = page_address(buf->page) + buf->page_offset;
- xdp.data_end = xdp.data + buf1_len;
- xdp.data_hard_start = page_address(buf->page);
- xdp_set_data_meta_invalid(&xdp);
- xdp.frame_sz = buf_sz;
- xdp.rxq = &rx_q->xdp_rxq;
+ xdp_init_buff(&xdp, buf_sz, &rx_q->xdp_rxq);
+ xdp_prepare_buff(&xdp, page_address(buf->page),
+ buf->page_offset, buf1_len, false);
pre_len = xdp.data_end - xdp.data_hard_start -
buf->page_offset;
struct stmmac_priv *priv = netdev_priv(ndev);
int ret = 0;
+ ret = pm_runtime_get_sync(priv->device);
+ if (ret < 0) {
+ pm_runtime_put_noidle(priv->device);
+ return ret;
+ }
+
ret = eth_mac_addr(ndev, addr);
if (ret)
- return ret;
+ goto set_mac_error;
stmmac_set_umac_addr(priv, priv->hw, ndev->dev_addr, 0);
+set_mac_error:
+ pm_runtime_put(priv->device);
+
return ret;
}
bool is_double = false;
int ret;
- ret = pm_runtime_get_sync(priv->device);
- if (ret < 0) {
- pm_runtime_put_noidle(priv->device);
- return ret;
- }
-
if (be16_to_cpu(proto) == ETH_P_8021AD)
is_double = true;
bool is_double = false;
int ret;
+ ret = pm_runtime_get_sync(priv->device);
+ if (ret < 0) {
+ pm_runtime_put_noidle(priv->device);
+ return ret;
+ }
+
if (be16_to_cpu(proto) == ETH_P_8021AD)
is_double = true;
reset_control_reset(priv->plat->stmmac_rst);
}
+ ret = reset_control_deassert(priv->plat->stmmac_ahb_rst);
+ if (ret == -ENOTSUPP)
+ dev_err(priv->device, "unable to bring out of ahb reset: %pe\n",
+ ERR_PTR(ret));
+
/* Init MAC and get the capabilities */
ret = stmmac_hw_init(priv);
if (ret)
}
}
+ if (priv->plat->speed_mode_2500)
+ priv->plat->speed_mode_2500(ndev, priv->plat->bsp_priv);
+
+ if (priv->plat->mdio_bus_data && priv->plat->mdio_bus_data->has_xpcs) {
+ ret = stmmac_xpcs_setup(priv->mii);
+ if (ret)
+ goto error_xpcs_setup;
+ }
+
ret = stmmac_phy_setup(priv);
if (ret) {
netdev_err(ndev, "failed to setup phy (%d)\n", ret);
unregister_netdev(ndev);
error_netdev_register:
phylink_destroy(priv->phylink);
+error_xpcs_setup:
error_phy_setup:
if (priv->hw->pcs != STMMAC_PCS_TBI &&
priv->hw->pcs != STMMAC_PCS_RTBI)
stmmac_napi_del(ndev);
error_hw_init:
destroy_workqueue(priv->wq);
- stmmac_bus_clks_config(priv, false);
bitmap_free(priv->af_xdp_zc_qps);
return ret;
phylink_destroy(priv->phylink);
if (priv->plat->stmmac_rst)
reset_control_assert(priv->plat->stmmac_rst);
+ reset_control_assert(priv->plat->stmmac_ahb_rst);
pm_runtime_put(dev);
pm_runtime_disable(dev);
if (priv->hw->pcs != STMMAC_PCS_TBI &&
return 0;
}
+int stmmac_xpcs_setup(struct mii_bus *bus)
+{
+ struct net_device *ndev = bus->priv;
+ struct mdio_device *mdiodev;
+ struct stmmac_priv *priv;
+ struct dw_xpcs *xpcs;
+ int mode, addr;
+
+ priv = netdev_priv(ndev);
+ mode = priv->plat->phy_interface;
+
+ /* Try to probe the XPCS by scanning all addresses. */
+ for (addr = 0; addr < PHY_MAX_ADDR; addr++) {
+ mdiodev = mdio_device_create(bus, addr);
+ if (IS_ERR(mdiodev))
+ continue;
+
+ xpcs = xpcs_create(mdiodev, mode);
+ if (IS_ERR_OR_NULL(xpcs)) {
+ mdio_device_free(mdiodev);
+ continue;
+ }
+
+ priv->hw->xpcs = xpcs;
+ break;
+ }
+
+ if (!priv->hw->xpcs) {
+ dev_warn(priv->device, "No xPCS found\n");
+ return -ENODEV;
+ }
+
+ return 0;
+}
+
/**
* stmmac_mdio_register
* @ndev: net device structure
max_addr = PHY_MAX_ADDR;
}
- if (mdio_bus_data->has_xpcs) {
- priv->hw->xpcs = mdio_xpcs_get_ops();
- if (!priv->hw->xpcs) {
- err = -ENODEV;
- goto bus_register_fail;
- }
- }
-
if (mdio_bus_data->needs_reset)
new_bus->reset = &stmmac_mdio_reset;
found = 1;
}
- /* Try to probe the XPCS by scanning all addresses. */
- if (priv->hw->xpcs) {
- struct mdio_xpcs_args *xpcs = &priv->hw->xpcs_args;
- int ret, mode = priv->plat->phy_interface;
- max_addr = PHY_MAX_ADDR;
-
- xpcs->bus = new_bus;
-
- for (addr = 0; addr < max_addr; addr++) {
- xpcs->addr = addr;
-
- ret = stmmac_xpcs_probe(priv, xpcs, mode);
- if (!ret) {
- found = 1;
- break;
- }
- }
- }
-
if (!found && !mdio_node) {
dev_warn(dev, "No PHY found\n");
- mdiobus_unregister(new_bus);
- mdiobus_free(new_bus);
- return -ENODEV;
+ err = -ENODEV;
+ goto no_phy_found;
}
bus_register_done:
return 0;
+no_phy_found:
+ mdiobus_unregister(new_bus);
bus_register_fail:
mdiobus_free(new_bus);
return err;
if (!priv->mii)
return 0;
+ if (priv->hw->xpcs) {
+ mdio_device_free(priv->hw->xpcs->mdiodev);
+ xpcs_destroy(priv->hw->xpcs);
+ }
+
mdiobus_unregister(priv->mii);
priv->mii->priv = NULL;
mdiobus_free(priv->mii);
if (!plat->dma_cfg)
return -ENOMEM;
+ plat->safety_feat_cfg = devm_kzalloc(&pdev->dev,
+ sizeof(*plat->safety_feat_cfg),
+ GFP_KERNEL);
+ if (!plat->safety_feat_cfg)
+ return -ENOMEM;
+
/* Enable pci device */
ret = pci_enable_device(pdev);
if (ret) {
res.wol_irq = pdev->irq;
res.irq = pdev->irq;
+ plat->safety_feat_cfg->tsoee = 1;
+ plat->safety_feat_cfg->mrxpee = 1;
+ plat->safety_feat_cfg->mestee = 1;
+ plat->safety_feat_cfg->mrxee = 1;
+ plat->safety_feat_cfg->mtxee = 1;
+ plat->safety_feat_cfg->epsi = 1;
+ plat->safety_feat_cfg->edpp = 1;
+ plat->safety_feat_cfg->prtyen = 1;
+ plat->safety_feat_cfg->tmouten = 1;
+
return stmmac_dvr_probe(&pdev->dev, plat, &res);
}
goto error_hw_init;
}
+ plat->stmmac_ahb_rst = devm_reset_control_get_optional_shared(
+ &pdev->dev, "ahb");
+ if (IS_ERR(plat->stmmac_ahb_rst)) {
+ ret = plat->stmmac_ahb_rst;
+ goto error_hw_init;
+ }
+
return plat;
error_hw_init:
void stmmac_remove_config_dt(struct platform_device *pdev,
struct plat_stmmacenet_data *plat)
{
+ clk_disable_unprepare(plat->stmmac_clk);
+ clk_disable_unprepare(plat->pclk);
of_node_put(plat->phy_node);
of_node_put(plat->mdio_node);
}
for (i = 0; i < ARRAY_SIZE(tc_flow_parsers); i++) {
ret = tc_flow_parsers[i].fn(priv, cls, entry);
- if (!ret) {
+ if (!ret)
entry->in_use = true;
- continue;
- }
}
if (!entry->in_use)
/* initialize spare pool of rx buffers, but allocate during the open */
static void cas_spare_init(struct cas *cp)
{
- spin_lock(&cp->rx_inuse_lock);
+ spin_lock(&cp->rx_inuse_lock);
INIT_LIST_HEAD(&cp->rx_inuse_list);
spin_unlock(&cp->rx_inuse_lock);
"VPD_SCAN: Reading in property [%s] len[%d]\n",
namebuf, prop_len);
for (i = 0; i < prop_len; i++) {
- err = niu_pci_eeprom_read(np, off + i);
- if (err >= 0)
- *prop_buf = err;
- ++prop_buf;
+ err = niu_pci_eeprom_read(np, off + i);
+ if (err < 0)
+ return err;
+ *prop_buf++ = err;
}
}
}
/* ESPC_PIO_EN_ENABLE must be set */
-static void niu_pci_vpd_fetch(struct niu *np, u32 start)
+static int niu_pci_vpd_fetch(struct niu *np, u32 start)
{
u32 offset;
int err;
err = niu_pci_eeprom_read16_swp(np, start + 1);
if (err < 0)
- return;
+ return err;
offset = err + 3;
u32 end;
err = niu_pci_eeprom_read(np, here);
+ if (err < 0)
+ return err;
if (err != 0x90)
- return;
+ return -EINVAL;
err = niu_pci_eeprom_read16_swp(np, here + 1);
if (err < 0)
- return;
+ return err;
here = start + offset + 3;
end = start + offset + err;
offset += err;
err = niu_pci_vpd_scan_props(np, here, end);
- if (err < 0 || err == 1)
- return;
+ if (err < 0)
+ return err;
+ if (err == 1)
+ return -EINVAL;
}
+ return 0;
}
/* ESPC_PIO_EN_ENABLE must be set */
offset = niu_pci_vpd_offset(np);
netif_printk(np, probe, KERN_DEBUG, np->dev,
"%s() VPD offset [%08x]\n", __func__, offset);
- if (offset)
- niu_pci_vpd_fetch(np, offset);
+ if (offset) {
+ err = niu_pci_vpd_fetch(np, offset);
+ if (err < 0)
+ return err;
+ }
nw64(ESPC_PIO_EN, 0);
if (np->flags & NIU_FLAGS_VPD_VALID) {
&advertising, ep->link_modes.advertising);
if (gp->phy_type != phy_mii_mdio0 &&
- gp->phy_type != phy_mii_mdio1)
- goto non_mii;
+ gp->phy_type != phy_mii_mdio1)
+ goto non_mii;
/* Setup advertise */
if (found_mii_phy(gp))
if (gp->phy_type == phy_serialink ||
gp->phy_type == phy_serdes) {
- u32 pcs_lpa = readl(gp->regs + PCS_MIILP);
+ u32 pcs_lpa = readl(gp->regs + PCS_MIILP);
if (pcs_lpa & (PCS_MIIADV_SP | PCS_MIIADV_AP))
pause = 1;
static void gem_init_pause_thresholds(struct gem *gp)
{
- u32 cfg;
+ u32 cfg;
/* Calculate pause thresholds. Setting the OFF threshold to the
* full RX fifo size effectively disables PAUSE generation which
/* Configure the chip "burst" DMA mode & enable some
* HW bug fixes on Apple version
*/
- cfg = 0;
- if (gp->pdev->vendor == PCI_VENDOR_ID_APPLE)
+ cfg = 0;
+ if (gp->pdev->vendor == PCI_VENDOR_ID_APPLE)
cfg |= GREG_CFG_RONPAULBIT | GREG_CFG_ENBUG2FIX;
#if !defined(CONFIG_SPARC64) && !defined(CONFIG_ALPHA)
- cfg |= GREG_CFG_IBURST;
+ cfg |= GREG_CFG_IBURST;
#endif
- cfg |= ((31 << 1) & GREG_CFG_TXDMALIM);
- cfg |= ((31 << 6) & GREG_CFG_RXDMALIM);
- writel(cfg, gp->regs + GREG_CFG);
+ cfg |= ((31 << 1) & GREG_CFG_TXDMALIM);
+ cfg |= ((31 << 6) & GREG_CFG_RXDMALIM);
+ writel(cfg, gp->regs + GREG_CFG);
/* If Infinite Burst didn't stick, then use different
* thresholds (and Apple bug fixes don't exist)
struct net_device *dev)
{
struct happy_meal *hp = netdev_priv(dev);
- int entry;
- u32 tx_flags;
+ int entry;
+ u32 tx_flags;
tx_flags = TXFLAG_OWN;
if (skb->ip_summed == CHECKSUM_PARTIAL) {
spin_lock_irq(&hp->happy_lock);
- if (TX_BUFFS_AVAIL(hp) <= (skb_shinfo(skb)->nr_frags + 1)) {
+ if (TX_BUFFS_AVAIL(hp) <= (skb_shinfo(skb)->nr_frags + 1)) {
netif_stop_queue(dev);
spin_unlock_irq(&hp->happy_lock);
printk(KERN_ERR "%s: BUG! Tx Ring full when queue awake!\n",
struct device_node *node = pdev->dev.of_node;
struct device *dev = &pdev->dev;
struct am65_cpts *cpts;
- struct resource *res;
void __iomem *base;
- res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "cpts");
- base = devm_ioremap_resource(dev, res);
+ base = devm_platform_ioremap_resource_byname(pdev, "cpts");
if (IS_ERR(base))
return PTR_ERR(base);
static int cpsw_phy_sel_probe(struct platform_device *pdev)
{
- struct resource *res;
const struct of_device_id *of_id;
struct cpsw_phy_sel_priv *priv;
priv->dev = &pdev->dev;
priv->cpsw_phy_sel = of_id->data;
- res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "gmii-sel");
- priv->gmii_sel = devm_ioremap_resource(&pdev->dev, res);
+ priv->gmii_sel = devm_platform_ioremap_resource_byname(pdev, "gmii-sel");
if (IS_ERR(priv->gmii_sel))
return PTR_ERR(priv->gmii_sel);
cpts_rx_timestamp(cpsw->cpts, skb);
skb->protocol = eth_type_trans(skb, ndev);
- /* unmap page as no netstack skb page recycling */
- page_pool_release_page(pool, page);
+ /* mark skb for recycling */
+ skb_mark_for_recycle(skb, page, pool);
netif_receive_skb(skb);
ndev->stats.rx_bytes += len;
}
cpsw->bus_freq_mhz = clk_get_rate(clk) / 1000000;
- ss_res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- ss_regs = devm_ioremap_resource(dev, ss_res);
+ ss_regs = devm_platform_get_and_ioremap_resource(pdev, 0, &ss_res);
if (IS_ERR(ss_regs))
return PTR_ERR(ss_regs);
cpsw->regs = ss_regs;
};
/**
- * struct ale_dev_id - The ALE version/SoC specific configuration
+ * struct cpsw_ale_dev_id - The ALE version/SoC specific configuration
* @dev_id: ALE version/SoC id
* @features: features supported by ALE
* @tbl_entries: number of ALE entries
cpts_rx_timestamp(cpsw->cpts, skb);
skb->protocol = eth_type_trans(skb, ndev);
- /* unmap page as no netstack skb page recycling */
- page_pool_release_page(pool, page);
+ /* mark skb for recycling */
+ skb_mark_for_recycle(skb, page, pool);
netif_receive_skb(skb);
ndev->stats.rx_bytes += len;
}
cpsw->bus_freq_mhz = clk_get_rate(clk) / 1000000;
- ss_res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- ss_regs = devm_ioremap_resource(dev, ss_res);
+ ss_regs = devm_platform_get_and_ioremap_resource(pdev, 0, &ss_res);
if (IS_ERR(ss_regs)) {
ret = PTR_ERR(ss_regs);
return ret;
priv->bus_freq_mhz = (u32)(emac_bus_frequency / 1000000);
/* Get EMAC platform data */
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- priv->emac_base_phys = res->start + pdata->ctrl_reg_offset;
- priv->remap_addr = devm_ioremap_resource(&pdev->dev, res);
+ priv->remap_addr = devm_platform_get_and_ioremap_resource(pdev, 0, &res);
if (IS_ERR(priv->remap_addr)) {
rc = PTR_ERR(priv->remap_addr);
goto no_pdata;
}
+ priv->emac_base_phys = res->start + pdata->ctrl_reg_offset;
res_ctrl = platform_get_resource(pdev, IORESOURCE_MEM, 1);
if (res_ctrl) {
tx_pipe->dma_queue = knav_queue_open(name, tx_pipe->dma_queue_id,
KNAV_QUEUE_SHARED);
if (IS_ERR(tx_pipe->dma_queue)) {
- dev_err(dev, "Could not open DMA queue for channel \"%s\": %d\n",
- name, ret);
+ dev_err(dev, "Could not open DMA queue for channel \"%s\": %pe\n",
+ name, tx_pipe->dma_queue);
ret = PTR_ERR(tx_pipe->dma_queue);
goto err;
}
}
/**
- * tx_srv - transmit interrupt service
+ * velocity_tx_srv - transmit interrupt service
* @vptr: Velocity
*
* Scan the queues looking for transmitted packets that
}
/**
- * velocity_get_status - statistics callback
+ * velocity_get_stats - statistics callback
* @dev: network device
*
* Callback from the network layer to allow driver statistics
}
/**
- * velocity_cleanup - module unload
+ * velocity_cleanup_module - module unload
*
* When the velocity hardware is unloaded this function is called.
* It will clean up the notifiers and the unregister the PCI
static int w5100_mmio_init(struct net_device *ndev)
{
struct platform_device *pdev = to_platform_device(ndev->dev.parent);
- struct w5100_priv *priv = netdev_priv(ndev);
struct w5100_mmio_priv *mmio_priv = w5100_mmio_priv(ndev);
- struct resource *mem;
spin_lock_init(&mmio_priv->reg_lock);
- mem = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- mmio_priv->base = devm_ioremap_resource(&pdev->dev, mem);
+ mmio_priv->base = devm_platform_get_and_ioremap_resource(pdev, 0, NULL);
if (IS_ERR(mmio_priv->base))
return PTR_ERR(mmio_priv->base);
- netdev_info(ndev, "at 0x%llx irq %d\n", (u64)mem->start, priv->irq);
-
return 0;
}
stat = be32_to_cpu(cur_p->app0);
while (stat & STS_CTRL_APP0_CMPLT) {
+ /* Make sure that the other fields are read after bd is
+ * released by dma
+ */
+ rmb();
dma_unmap_single(ndev->dev.parent, be32_to_cpu(cur_p->phys),
be32_to_cpu(cur_p->len), DMA_TO_DEVICE);
skb = (struct sk_buff *)ptr_from_txbd(cur_p);
if (skb)
dev_consume_skb_irq(skb);
- cur_p->app0 = 0;
cur_p->app1 = 0;
cur_p->app2 = 0;
cur_p->app3 = 0;
ndev->stats.tx_packets++;
ndev->stats.tx_bytes += be32_to_cpu(cur_p->len);
+ /* app0 must be visible last, as it is used to flag
+ * availability of the bd
+ */
+ smp_mb();
+ cur_p->app0 = 0;
+
lp->tx_bd_ci++;
if (lp->tx_bd_ci >= lp->tx_bd_num)
lp->tx_bd_ci = 0;
if (cur_p->app0)
return NETDEV_TX_BUSY;
+ /* Make sure to read next bd app0 after this one */
+ rmb();
+
tail++;
if (tail >= lp->tx_bd_num)
tail = 0;
smp_mb();
/* Space might have just been freed - check again */
- if (temac_check_tx_bd_space(lp, num_frag))
+ if (temac_check_tx_bd_space(lp, num_frag + 1))
return NETDEV_TX_BUSY;
netif_wake_queue(ndev);
return NETDEV_TX_OK;
}
cur_p->phys = cpu_to_be32(skb_dma_addr);
- ptr_to_txbd((void *)skb, cur_p);
for (ii = 0; ii < num_frag; ii++) {
if (++lp->tx_bd_tail >= lp->tx_bd_num)
}
cur_p->app0 |= cpu_to_be32(STS_CTRL_APP0_EOP);
+ /* Mark last fragment with skb address, so it can be consumed
+ * in temac_start_xmit_done()
+ */
+ ptr_to_txbd((void *)skb, cur_p);
+
tail_p = lp->tx_bd_p + sizeof(*lp->tx_bd_v) * lp->tx_bd_tail;
lp->tx_bd_tail++;
if (lp->tx_bd_tail >= lp->tx_bd_num)
wmb();
lp->dma_out(lp, TX_TAILDESC_PTR, tail_p); /* DMA start */
+ if (temac_check_tx_bd_space(lp, MAX_SKB_FRAGS + 1))
+ netif_stop_queue(ndev);
+
return NETDEV_TX_OK;
}
case PHY_INTERFACE_MODE_MII:
phylink_set(mask, 100baseT_Full);
phylink_set(mask, 10baseT_Full);
+ fallthrough;
default:
break;
}
goto cleanup_clk;
/* Map device registers */
- ethres = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- lp->regs = devm_ioremap_resource(&pdev->dev, ethres);
+ lp->regs = devm_platform_get_and_ioremap_resource(pdev, 0, ðres);
if (IS_ERR(lp->regs)) {
ret = PTR_ERR(lp->regs);
goto cleanup_clk;
lp->eth_irq = platform_get_irq_optional(pdev, 0);
} else {
/* Check for these resources directly on the Ethernet node. */
- struct resource *res = platform_get_resource(pdev,
- IORESOURCE_MEM, 1);
- lp->dma_regs = devm_ioremap_resource(&pdev->dev, res);
+ lp->dma_regs = devm_platform_get_and_ioremap_resource(pdev, 1, NULL);
lp->rx_irq = platform_get_irq(pdev, 1);
lp->tx_irq = platform_get_irq(pdev, 0);
lp->eth_irq = platform_get_irq_optional(pdev, 2);
}
dev_info(dev,
- "Xilinx EmacLite at 0x%08lX mapped to 0x%08lX, irq=%d\n",
- (unsigned long __force)ndev->mem_start,
- (unsigned long __force)lp->base_addr, ndev->irq);
+ "Xilinx EmacLite at 0x%08lX mapped to 0x%p, irq=%d\n",
+ (unsigned long __force)ndev->mem_start, lp->base_addr, ndev->irq);
return 0;
error:
if (pktlen < ETH_ZLEN)
{
if (skb_padto(skb, ETH_ZLEN))
- return NETDEV_TX_OK;
+ return NETDEV_TX_OK;
pktlen = ETH_ZLEN;
}
struct device_node *np = dev->of_node;
struct eth_plat_info *plat;
struct net_device *ndev;
- struct resource *res;
struct port *port;
int err;
port->id = plat->npe;
/* Get the port resource and remap */
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!res)
- return -ENODEV;
- port->regs = devm_ioremap_resource(dev, res);
+ port->regs = devm_platform_get_and_ioremap_resource(pdev, 0, NULL);
if (IS_ERR(port->regs))
return PTR_ERR(port->regs);
phydev = of_phy_get_and_connect(ndev, np, ixp4xx_adjust_link);
} else {
phydev = mdiobus_get_phy(mdio_bus, plat->phy);
- if (IS_ERR(phydev)) {
- err = PTR_ERR(phydev);
+ if (!phydev) {
+ err = -ENODEV;
dev_err(dev, "could not connect phydev (%d)\n", err);
goto err_free_mem;
}
* if the payload is greater than zero.
* For the SBAPayload and the SBAOverhead we have the following
* unite quations
- * _ _
+ * _ _
* | bytes |
* SBAPayload = | 8000 ------ |
* | s |
* - -
- * _ _
+ * _ _
* | bytes |
* SBAOverhead = | ------ |
* | T-NEG |
* - -
- *
+ *
* T-NEG is described by the equation:
*
* (-) fddiMACT-NEG
#define PLC_QELM_A_BIST 0x5b6b /* BIST signature of QELM Rev. A */
/*
- FDDI board recources
+ FDDI board recources
*/
/*
};
static struct resource fjes_resource[] = {
- {
- .flags = IORESOURCE_MEM,
- .start = 0,
- .end = 0,
- },
- {
- .flags = IORESOURCE_IRQ,
- .start = 0,
- .end = 0,
- },
+ DEFINE_RES_MEM(0, 1),
+ DEFINE_RES_IRQ(0)
};
static bool is_extended_socket_device(struct acpi_device *device)
adapter->interrupt_watch_enable = false;
res = platform_get_resource(plat_dev, IORESOURCE_MEM, 0);
+ if (!res) {
+ err = -EINVAL;
+ goto err_free_control_wq;
+ }
hw->hw_res.start = res->start;
hw->hw_res.size = resource_size(res);
hw->hw_res.irq = platform_get_irq(plat_dev, 0);
* calculate the transport header.
*/
skb_reset_network_header(skb);
+ skb_reset_mac_header(skb);
skb->dev = pctx->dev;
gtp1->length = htons(payload_len);
gtp1->tid = htonl(pctx->u.v1.o_tei);
- /* TODO: Suppport for extension header, sequence number and N-PDU.
+ /* TODO: Support for extension header, sequence number and N-PDU.
* Update the length field if any of them is available.
*/
}
err = 0;
break;
- case SIOCSIFHWADDR: {
- char addr[AX25_ADDR_LEN];
+ case SIOCSIFHWADDR: {
+ char addr[AX25_ADDR_LEN];
- if (copy_from_user(&addr,
- (void __user *) arg, AX25_ADDR_LEN)) {
+ if (copy_from_user(&addr,
+ (void __user *)arg, AX25_ADDR_LEN)) {
err = -EFAULT;
break;
}
netif_tx_lock_bh(dev);
memcpy(dev->dev_addr, &addr, AX25_ADDR_LEN);
netif_tx_unlock_bh(dev);
-
err = 0;
break;
}
-
default:
err = tty_mode_ioctl(tty, file, cmd, arg);
}
#if 0
static inline void append_crc_ccitt(unsigned char *buffer, int len)
{
- unsigned int crc = 0xffff;
+ unsigned int crc = 0xffff;
for (;len>0;len--)
crc = (crc >> 8) ^ crc_ccitt_table[(crc ^ *buffer++) & 0xff];
for (j = 0; j < 8; j++)
if (unlikely(!(notbitstream & (0x1f0 << j)))) {
bitstream &= ~(0x100 << j);
- bitbuf = (bitbuf & (((2 << j) << numbit) - 1)) |
+ bitbuf = (bitbuf & (((2 << j) << numbit) - 1)) |
((bitbuf & ~(((2 << j) << numbit) - 1)) << 1);
numbit++;
notbitstream = ~bitstream;
/* ------------------------------------------------------------------------ */
-
+#ifdef CONFIG_PROC_FS
/*
* Proc filesystem
*/
.stop = bpq_seq_stop,
.show = bpq_seq_show,
};
-
+#endif
/* ------------------------------------------------------------------------ */
static const struct net_device_ops bpq_netdev_ops = {
static inline void append_crc_ccitt(unsigned char *buffer, int len)
{
- unsigned int crc = crc_ccitt(0xffff, buffer, len) ^ 0xffff;
+ unsigned int crc = crc_ccitt(0xffff, buffer, len) ^ 0xffff;
buffer += len;
*buffer++ = crc;
*buffer++ = crc >> 8;
*/
*ax->rbuff &= ~0x20;
}
- }
+ }
count = ax->rcount;
default:
count = kiss_esc(p, ax->xbuff, len);
}
- }
+ }
spin_unlock_bh(&ax->buflock);
set_bit(TTY_DO_WRITE_WAKEUP, &ax->tty->flags);
ax->tty = NULL;
unregister_netdev(ax->dev);
+ free_netdev(ax->dev);
}
/* Perform I/O control on an active ax25 channel. */
dev = ax->dev;
switch (cmd) {
- case SIOCGIFNAME:
+ case SIOCGIFNAME:
err = copy_to_user((void __user *) arg, ax->dev->name,
strlen(ax->dev->name) + 1) ? -EFAULT : 0;
break;
unsigned long flags;
spin_lock_irqsave(&scc->lock, flags);
- del_timer(&scc->tx_wdog);
- scc_key_trx(scc, TX_OFF);
+ del_timer(&scc->tx_wdog);
+ scc_key_trx(scc, TX_OFF);
spin_unlock_irqrestore(&scc->lock, flags);
- if (scc->stat.tx_state == TXS_TIMEOUT) /* we had a timeout? */
- {
- scc->stat.tx_state = TXS_WAIT;
+ if (scc->stat.tx_state == TXS_TIMEOUT) /* we had a timeout? */
+ {
+ scc->stat.tx_state = TXS_WAIT;
scc_start_tx_timer(scc, t_dwait, scc->kiss.mintime*100);
- return;
- }
-
- scc->stat.tx_state = TXS_IDLE;
+ return;
+ }
+
+ scc->stat.tx_state = TXS_IDLE;
netif_wake_queue(scc->dev);
}
{
struct scc_channel *scc = (struct scc_channel *) dev->ml_priv;
- if (!scc->init)
+ if (!scc->init)
return -EINVAL;
scc->tx_buff = NULL;
}
yp->tx_len = skb->len - 1; /* strip KISS byte */
if (yp->tx_len >= YAM_MAX_FRAME || yp->tx_len < 2) {
- dev_kfree_skb_any(skb);
+ dev_kfree_skb_any(skb);
break;
}
skb_copy_from_linear_data_offset(skb, 1,
u32 info_buflen;
u32 info_buf_offset;
u32 dev_vc_handle;
+ u8 info_buf[];
};
/* Response to NdisSetRequest */
set = &request->request_msg.msg.set_req;
set->oid = RNDIS_OID_GEN_CURRENT_PACKET_FILTER;
set->info_buflen = sizeof(u32);
- set->info_buf_offset = sizeof(struct rndis_set_request);
-
- memcpy((void *)(unsigned long)set + sizeof(struct rndis_set_request),
- &new_filter, sizeof(u32));
+ set->info_buf_offset = offsetof(typeof(*set), info_buf);
+ memcpy(set->info_buf, &new_filter, sizeof(u32));
ret = rndis_filter_send_request(dev, request);
if (ret == 0) {
#include <linux/spi/spi.h>
#include <linux/interrupt.h>
+#include <linux/mod_devicetable.h>
#include <linux/module.h>
-#include <linux/of.h>
#include <linux/regmap.h>
#include <linux/ieee802154.h>
#include <linux/irq.h>
static struct spi_driver mrf24j40_driver = {
.driver = {
- .of_match_table = of_match_ptr(mrf24j40_of_match),
+ .of_match_table = mrf24j40_of_match,
.name = "mrf24j40",
},
.id_table = mrf24j40_ids,
by Patrick McHardy and then maintained by Andre Correa.
You need the tc action mirror or redirect to feed this device
- packets.
+ packets.
- Authors: Jamal Hadi Salim (2005)
+ Authors: Jamal Hadi Salim (2005)
*/
ipa_table.o ipa_interrupt.o gsi.o gsi_trans.o \
ipa_gsi.o ipa_smp2p.o ipa_uc.o \
ipa_endpoint.o ipa_cmd.o ipa_modem.o \
- ipa_resource.o ipa_qmi.o ipa_qmi_msg.o
+ ipa_resource.o ipa_qmi.o ipa_qmi_msg.o \
+ ipa_sysfs.o
-ipa-y += ipa_data-v3.5.1.o ipa_data-v4.2.o \
- ipa_data-v4.5.o ipa_data-v4.9.o \
- ipa_data-v4.11.o
+ipa-y += ipa_data-v3.1.o ipa_data-v3.5.1.o \
+ ipa_data-v4.2.o ipa_data-v4.5.o \
+ ipa_data-v4.9.o ipa_data-v4.11.o
iowrite32(0, gsi->virt + GSI_CNTXT_GLOB_IRQ_EN_OFFSET);
iowrite32(0, gsi->virt + GSI_CNTXT_SRC_IEOB_IRQ_MSK_OFFSET);
- /* The inter-EE registers are in the non-adjusted address range */
- iowrite32(0, gsi->virt_raw + GSI_INTER_EE_SRC_CH_IRQ_MSK_OFFSET);
- iowrite32(0, gsi->virt_raw + GSI_INTER_EE_SRC_EV_CH_IRQ_MSK_OFFSET);
+ /* The inter-EE interrupts are not supported for IPA v3.0-v3.1 */
+ if (gsi->version > IPA_VERSION_3_1) {
+ u32 offset;
+
+ /* These registers are in the non-adjusted address range */
+ offset = GSI_INTER_EE_SRC_CH_IRQ_MSK_OFFSET;
+ iowrite32(0, gsi->virt_raw + offset);
+ offset = GSI_INTER_EE_SRC_EV_CH_IRQ_MSK_OFFSET;
+ iowrite32(0, gsi->virt_raw + offset);
+ }
iowrite32(0, gsi->virt + GSI_CNTXT_GSI_IRQ_EN_OFFSET);
}
+/* Get # supported channel and event rings; there is no gsi_ring_teardown() */
+static int gsi_ring_setup(struct gsi *gsi)
+{
+ struct device *dev = gsi->dev;
+ u32 count;
+ u32 val;
+
+ if (gsi->version < IPA_VERSION_3_5_1) {
+ /* No HW_PARAM_2 register prior to IPA v3.5.1, assume the max */
+ gsi->channel_count = GSI_CHANNEL_COUNT_MAX;
+ gsi->evt_ring_count = GSI_EVT_RING_COUNT_MAX;
+
+ return 0;
+ }
+
+ val = ioread32(gsi->virt + GSI_GSI_HW_PARAM_2_OFFSET);
+
+ count = u32_get_bits(val, NUM_CH_PER_EE_FMASK);
+ if (!count) {
+ dev_err(dev, "GSI reports zero channels supported\n");
+ return -EINVAL;
+ }
+ if (count > GSI_CHANNEL_COUNT_MAX) {
+ dev_warn(dev, "limiting to %u channels; hardware supports %u\n",
+ GSI_CHANNEL_COUNT_MAX, count);
+ count = GSI_CHANNEL_COUNT_MAX;
+ }
+ gsi->channel_count = count;
+
+ count = u32_get_bits(val, NUM_EV_PER_EE_FMASK);
+ if (!count) {
+ dev_err(dev, "GSI reports zero event rings supported\n");
+ return -EINVAL;
+ }
+ if (count > GSI_EVT_RING_COUNT_MAX) {
+ dev_warn(dev,
+ "limiting to %u event rings; hardware supports %u\n",
+ GSI_EVT_RING_COUNT_MAX, count);
+ count = GSI_EVT_RING_COUNT_MAX;
+ }
+ gsi->evt_ring_count = count;
+
+ return 0;
+}
+
/* Event ring commands are performed one at a time. Their completion
* is signaled by the event ring control GSI interrupt type, which is
* only enabled when we issue an event ring command. Only the event
/* Setup function for GSI. GSI firmware must be loaded and initialized */
int gsi_setup(struct gsi *gsi)
{
- struct device *dev = gsi->dev;
u32 val;
+ int ret;
/* Here is where we first touch the GSI hardware */
val = ioread32(gsi->virt + GSI_GSI_STATUS_OFFSET);
if (!(val & ENABLED_FMASK)) {
- dev_err(dev, "GSI has not been enabled\n");
+ dev_err(gsi->dev, "GSI has not been enabled\n");
return -EIO;
}
gsi_irq_setup(gsi); /* No matching teardown required */
- val = ioread32(gsi->virt + GSI_GSI_HW_PARAM_2_OFFSET);
-
- gsi->channel_count = u32_get_bits(val, NUM_CH_PER_EE_FMASK);
- if (!gsi->channel_count) {
- dev_err(dev, "GSI reports zero channels supported\n");
- return -EINVAL;
- }
- if (gsi->channel_count > GSI_CHANNEL_COUNT_MAX) {
- dev_warn(dev,
- "limiting to %u channels; hardware supports %u\n",
- GSI_CHANNEL_COUNT_MAX, gsi->channel_count);
- gsi->channel_count = GSI_CHANNEL_COUNT_MAX;
- }
-
- gsi->evt_ring_count = u32_get_bits(val, NUM_EV_PER_EE_FMASK);
- if (!gsi->evt_ring_count) {
- dev_err(dev, "GSI reports zero event rings supported\n");
- return -EINVAL;
- }
- if (gsi->evt_ring_count > GSI_EVT_RING_COUNT_MAX) {
- dev_warn(dev,
- "limiting to %u event rings; hardware supports %u\n",
- GSI_EVT_RING_COUNT_MAX, gsi->evt_ring_count);
- gsi->evt_ring_count = GSI_EVT_RING_COUNT_MAX;
- }
+ ret = gsi_ring_setup(gsi); /* No matching teardown required */
+ if (ret)
+ return ret;
/* Initialize the error log */
iowrite32(0, gsi->virt + GSI_ERROR_LOG_OFFSET);
/* Maximum number of channels and event rings supported by the driver */
#define GSI_CHANNEL_COUNT_MAX 23
-#define GSI_EVT_RING_COUNT_MAX 20
+#define GSI_EVT_RING_COUNT_MAX 24
/* Maximum TLV FIFO size for a channel; 64 here is arbitrary (and high) */
#define GSI_TLV_MAX 64
*/
#define GSI_EE_REG_ADJUST 0x0000d000 /* IPA v4.5+ */
-/* The two inter-EE IRQ register offsets are relative to gsi->virt_raw */
+/* The inter-EE IRQ registers are relative to gsi->virt_raw (IPA v3.5+) */
+
#define GSI_INTER_EE_SRC_CH_IRQ_MSK_OFFSET \
GSI_INTER_EE_N_SRC_CH_IRQ_MSK_OFFSET(GSI_EE_AP)
#define GSI_INTER_EE_N_SRC_CH_IRQ_MSK_OFFSET(ee) \
* @mem_virt: Virtual address of IPA-local memory space
* @mem_offset: Offset from @mem_virt used for access to IPA memory
* @mem_size: Total size (bytes) of memory at @mem_virt
+ * @mem_count: Number of entries in the mem array
* @mem: Array of IPA-local memory region descriptors
* @imem_iova: I/O virtual address of IPA region in IMEM
* @imem_size: Size of IMEM region
void *mem_virt;
u32 mem_offset;
u32 mem_size;
+ u32 mem_count;
const struct ipa_mem *mem;
unsigned long imem_iova;
/* Validate the memory region that holds headers */
static bool ipa_cmd_header_valid(struct ipa *ipa)
{
- const struct ipa_mem *mem = &ipa->mem[IPA_MEM_MODEM_HEADER];
struct device *dev = &ipa->pdev->dev;
+ const struct ipa_mem *mem;
u32 offset_max;
u32 size_max;
+ u32 offset;
u32 size;
- /* In ipa_cmd_hdr_init_local_add() we record the offset and size
- * of the header table memory area. Make sure the offset and size
- * fit in the fields that need to hold them, and that the entire
- * range is within the overall IPA memory range.
+ /* In ipa_cmd_hdr_init_local_add() we record the offset and size of
+ * the header table memory area in an immediate command. Make sure
+ * the offset and size fit in the fields that need to hold them, and
+ * that the entire range is within the overall IPA memory range.
*/
offset_max = field_max(HDR_INIT_LOCAL_FLAGS_HDR_ADDR_FMASK);
- if (mem->offset > offset_max ||
- ipa->mem_offset > offset_max - mem->offset) {
+ size_max = field_max(HDR_INIT_LOCAL_FLAGS_TABLE_SIZE_FMASK);
+
+ /* The header memory area contains both the modem and AP header
+ * regions. The modem portion defines the address of the region.
+ */
+ mem = ipa_mem_find(ipa, IPA_MEM_MODEM_HEADER);
+ offset = mem->offset;
+ size = mem->size;
+
+ /* Make sure the offset fits in the IPA command */
+ if (offset > offset_max || ipa->mem_offset > offset_max - offset) {
dev_err(dev, "header table region offset too large\n");
dev_err(dev, " (0x%04x + 0x%04x > 0x%04x)\n",
- ipa->mem_offset, mem->offset, offset_max);
+ ipa->mem_offset, offset, offset_max);
return false;
}
- size_max = field_max(HDR_INIT_LOCAL_FLAGS_TABLE_SIZE_FMASK);
- size = ipa->mem[IPA_MEM_MODEM_HEADER].size;
- size += ipa->mem[IPA_MEM_AP_HEADER].size;
+ /* Add the size of the AP portion (if defined) to the combined size */
+ mem = ipa_mem_find(ipa, IPA_MEM_AP_HEADER);
+ if (mem)
+ size += mem->size;
+ /* Make sure the combined size fits in the IPA command */
if (size > size_max) {
dev_err(dev, "header table region size too large\n");
dev_err(dev, " (0x%04x > 0x%08x)\n", size, size_max);
return false;
}
- if (size > ipa->mem_size || mem->offset > ipa->mem_size - size) {
+
+ /* Make sure the entire combined area fits in IPA memory */
+ if (size > ipa->mem_size || offset > ipa->mem_size - size) {
dev_err(dev, "header table region out of range\n");
dev_err(dev, " (0x%04x + 0x%04x > 0x%04x)\n",
- mem->offset, size, ipa->mem_size);
+ offset, size, ipa->mem_size);
return false;
}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2019-2021 Linaro Ltd.
+ */
+
+#include <linux/log2.h>
+
+#include "gsi.h"
+#include "ipa_data.h"
+#include "ipa_endpoint.h"
+#include "ipa_mem.h"
+
+/** enum ipa_resource_type - IPA resource types for an SoC having IPA v3.1 */
+enum ipa_resource_type {
+ /* Source resource types; first must have value 0 */
+ IPA_RESOURCE_TYPE_SRC_PKT_CONTEXTS = 0,
+ IPA_RESOURCE_TYPE_SRC_HDR_SECTORS,
+ IPA_RESOURCE_TYPE_SRC_HDRI1_BUFFER,
+ IPA_RESOURCE_TYPE_SRC_DESCRIPTOR_LISTS,
+ IPA_RESOURCE_TYPE_SRC_DESCRIPTOR_BUFF,
+ IPA_RESOURCE_TYPE_SRC_HDRI2_BUFFERS,
+ IPA_RESOURCE_TYPE_SRC_HPS_DMARS,
+ IPA_RESOURCE_TYPE_SRC_ACK_ENTRIES,
+
+ /* Destination resource types; first must have value 0 */
+ IPA_RESOURCE_TYPE_DST_DATA_SECTORS = 0,
+ IPA_RESOURCE_TYPE_DST_DATA_SECTOR_LISTS,
+ IPA_RESOURCE_TYPE_DST_DPS_DMARS,
+};
+
+/* Resource groups used for an SoC having IPA v3.1 */
+enum ipa_rsrc_group_id {
+ /* Source resource group identifiers */
+ IPA_RSRC_GROUP_SRC_UL = 0,
+ IPA_RSRC_GROUP_SRC_DL,
+ IPA_RSRC_GROUP_SRC_DIAG,
+ IPA_RSRC_GROUP_SRC_DMA,
+ IPA_RSRC_GROUP_SRC_UNUSED,
+ IPA_RSRC_GROUP_SRC_UC_RX_Q,
+ IPA_RSRC_GROUP_SRC_COUNT, /* Last in set; not a source group */
+
+ /* Destination resource group identifiers */
+ IPA_RSRC_GROUP_DST_UL = 0,
+ IPA_RSRC_GROUP_DST_DL,
+ IPA_RSRC_GROUP_DST_DIAG_DPL,
+ IPA_RSRC_GROUP_DST_DMA,
+ IPA_RSRC_GROUP_DST_Q6ZIP_GENERAL,
+ IPA_RSRC_GROUP_DST_Q6ZIP_ENGINE,
+ IPA_RSRC_GROUP_DST_COUNT, /* Last; not a destination group */
+};
+
+/* QSB configuration data for an SoC having IPA v3.1 */
+static const struct ipa_qsb_data ipa_qsb_data[] = {
+ [IPA_QSB_MASTER_DDR] = {
+ .max_writes = 8,
+ .max_reads = 8,
+ },
+ [IPA_QSB_MASTER_PCIE] = {
+ .max_writes = 2,
+ .max_reads = 8,
+ },
+};
+
+/* Endpoint data for an SoC having IPA v3.1 */
+static const struct ipa_gsi_endpoint_data ipa_gsi_endpoint_data[] = {
+ [IPA_ENDPOINT_AP_COMMAND_TX] = {
+ .ee_id = GSI_EE_AP,
+ .channel_id = 6,
+ .endpoint_id = 22,
+ .toward_ipa = true,
+ .channel = {
+ .tre_count = 256,
+ .event_count = 256,
+ .tlv_count = 18,
+ },
+ .endpoint = {
+ .config = {
+ .resource_group = IPA_RSRC_GROUP_SRC_UL,
+ .dma_mode = true,
+ .dma_endpoint = IPA_ENDPOINT_AP_LAN_RX,
+ .tx = {
+ .seq_type = IPA_SEQ_DMA,
+ },
+ },
+ },
+ },
+ [IPA_ENDPOINT_AP_LAN_RX] = {
+ .ee_id = GSI_EE_AP,
+ .channel_id = 7,
+ .endpoint_id = 15,
+ .toward_ipa = false,
+ .channel = {
+ .tre_count = 256,
+ .event_count = 256,
+ .tlv_count = 8,
+ },
+ .endpoint = {
+ .config = {
+ .resource_group = IPA_RSRC_GROUP_SRC_UL,
+ .aggregation = true,
+ .status_enable = true,
+ .rx = {
+ .pad_align = ilog2(sizeof(u32)),
+ },
+ },
+ },
+ },
+ [IPA_ENDPOINT_AP_MODEM_TX] = {
+ .ee_id = GSI_EE_AP,
+ .channel_id = 5,
+ .endpoint_id = 3,
+ .toward_ipa = true,
+ .channel = {
+ .tre_count = 512,
+ .event_count = 512,
+ .tlv_count = 16,
+ },
+ .endpoint = {
+ .filter_support = true,
+ .config = {
+ .resource_group = IPA_RSRC_GROUP_SRC_UL,
+ .checksum = true,
+ .qmap = true,
+ .status_enable = true,
+ .tx = {
+ .seq_type = IPA_SEQ_2_PASS_SKIP_LAST_UC,
+ .status_endpoint =
+ IPA_ENDPOINT_MODEM_AP_RX,
+ },
+ },
+ },
+ },
+ [IPA_ENDPOINT_AP_MODEM_RX] = {
+ .ee_id = GSI_EE_AP,
+ .channel_id = 8,
+ .endpoint_id = 16,
+ .toward_ipa = false,
+ .channel = {
+ .tre_count = 256,
+ .event_count = 256,
+ .tlv_count = 8,
+ },
+ .endpoint = {
+ .config = {
+ .resource_group = IPA_RSRC_GROUP_DST_DL,
+ .checksum = true,
+ .qmap = true,
+ .aggregation = true,
+ .rx = {
+ .aggr_close_eof = true,
+ },
+ },
+ },
+ },
+ [IPA_ENDPOINT_MODEM_LAN_TX] = {
+ .ee_id = GSI_EE_MODEM,
+ .channel_id = 4,
+ .endpoint_id = 9,
+ .toward_ipa = true,
+ .endpoint = {
+ .filter_support = true,
+ },
+ },
+ [IPA_ENDPOINT_MODEM_AP_TX] = {
+ .ee_id = GSI_EE_MODEM,
+ .channel_id = 0,
+ .endpoint_id = 5,
+ .toward_ipa = true,
+ .endpoint = {
+ .filter_support = true,
+ },
+ },
+ [IPA_ENDPOINT_MODEM_AP_RX] = {
+ .ee_id = GSI_EE_MODEM,
+ .channel_id = 5,
+ .endpoint_id = 18,
+ .toward_ipa = false,
+ },
+};
+
+/* Source resource configuration data for an SoC having IPA v3.1 */
+static const struct ipa_resource ipa_resource_src[] = {
+ [IPA_RESOURCE_TYPE_SRC_PKT_CONTEXTS] = {
+ .limits[IPA_RSRC_GROUP_SRC_UL] = {
+ .min = 3, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DL] = {
+ .min = 3, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DIAG] = {
+ .min = 1, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DMA] = {
+ .min = 1, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_UC_RX_Q] = {
+ .min = 2, .max = 255,
+ },
+ },
+ [IPA_RESOURCE_TYPE_SRC_HDR_SECTORS] = {
+ .limits[IPA_RSRC_GROUP_SRC_UL] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DL] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DIAG] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DMA] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_UC_RX_Q] = {
+ .min = 0, .max = 255,
+ },
+ },
+ [IPA_RESOURCE_TYPE_SRC_HDRI1_BUFFER] = {
+ .limits[IPA_RSRC_GROUP_SRC_UL] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DL] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DIAG] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DMA] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_UC_RX_Q] = {
+ .min = 0, .max = 255,
+ },
+ },
+ [IPA_RESOURCE_TYPE_SRC_DESCRIPTOR_LISTS] = {
+ .limits[IPA_RSRC_GROUP_SRC_UL] = {
+ .min = 14, .max = 14,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DL] = {
+ .min = 16, .max = 16,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DIAG] = {
+ .min = 5, .max = 5,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DMA] = {
+ .min = 5, .max = 5,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_UC_RX_Q] = {
+ .min = 8, .max = 8,
+ },
+ },
+ [IPA_RESOURCE_TYPE_SRC_DESCRIPTOR_BUFF] = {
+ .limits[IPA_RSRC_GROUP_SRC_UL] = {
+ .min = 19, .max = 19,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DL] = {
+ .min = 26, .max = 26,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DIAG] = {
+ .min = 5, .max = 5, /* 3 downstream */
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DMA] = {
+ .min = 5, .max = 5, /* 7 downstream */
+ },
+ .limits[IPA_RSRC_GROUP_SRC_UC_RX_Q] = {
+ .min = 8, .max = 8,
+ },
+ },
+ [IPA_RESOURCE_TYPE_SRC_HDRI2_BUFFERS] = {
+ .limits[IPA_RSRC_GROUP_SRC_UL] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DL] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DIAG] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DMA] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_UC_RX_Q] = {
+ .min = 0, .max = 255,
+ },
+ },
+ [IPA_RESOURCE_TYPE_SRC_HPS_DMARS] = {
+ .limits[IPA_RSRC_GROUP_SRC_UL] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DL] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DIAG] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DMA] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_UC_RX_Q] = {
+ .min = 0, .max = 255,
+ },
+ },
+ [IPA_RESOURCE_TYPE_SRC_ACK_ENTRIES] = {
+ .limits[IPA_RSRC_GROUP_SRC_UL] = {
+ .min = 19, .max = 19,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DL] = {
+ .min = 26, .max = 26,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DIAG] = {
+ .min = 5, .max = 5,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_DMA] = {
+ .min = 5, .max = 5,
+ },
+ .limits[IPA_RSRC_GROUP_SRC_UC_RX_Q] = {
+ .min = 8, .max = 8,
+ },
+ },
+};
+
+/* Destination resource configuration data for an SoC having IPA v3.1 */
+static const struct ipa_resource ipa_resource_dst[] = {
+ [IPA_RESOURCE_TYPE_DST_DATA_SECTORS] = {
+ .limits[IPA_RSRC_GROUP_DST_UL] = {
+ .min = 3, .max = 3, /* 2 downstream */
+ },
+ .limits[IPA_RSRC_GROUP_DST_DL] = {
+ .min = 3, .max = 3,
+ },
+ .limits[IPA_RSRC_GROUP_DST_DIAG_DPL] = {
+ .min = 1, .max = 1, /* 0 downstream */
+ },
+ /* IPA_RSRC_GROUP_DST_DMA uses 2 downstream */
+ .limits[IPA_RSRC_GROUP_DST_Q6ZIP_GENERAL] = {
+ .min = 3, .max = 3,
+ },
+ .limits[IPA_RSRC_GROUP_DST_Q6ZIP_ENGINE] = {
+ .min = 3, .max = 3,
+ },
+ },
+ [IPA_RESOURCE_TYPE_DST_DATA_SECTOR_LISTS] = {
+ .limits[IPA_RSRC_GROUP_DST_UL] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_DST_DL] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_DST_DIAG_DPL] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_DST_DMA] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_DST_Q6ZIP_GENERAL] = {
+ .min = 0, .max = 255,
+ },
+ .limits[IPA_RSRC_GROUP_DST_Q6ZIP_ENGINE] = {
+ .min = 0, .max = 255,
+ },
+ },
+ [IPA_RESOURCE_TYPE_DST_DPS_DMARS] = {
+ .limits[IPA_RSRC_GROUP_DST_UL] = {
+ .min = 1, .max = 1,
+ },
+ .limits[IPA_RSRC_GROUP_DST_DL] = {
+ .min = 1, .max = 1,
+ },
+ .limits[IPA_RSRC_GROUP_DST_DIAG_DPL] = {
+ .min = 1, .max = 1,
+ },
+ .limits[IPA_RSRC_GROUP_DST_DMA] = {
+ .min = 1, .max = 1,
+ },
+ .limits[IPA_RSRC_GROUP_DST_Q6ZIP_GENERAL] = {
+ .min = 1, .max = 1,
+ },
+ },
+};
+
+/* Resource configuration data for an SoC having IPA v3.1 */
+static const struct ipa_resource_data ipa_resource_data = {
+ .rsrc_group_src_count = IPA_RSRC_GROUP_SRC_COUNT,
+ .rsrc_group_dst_count = IPA_RSRC_GROUP_DST_COUNT,
+ .resource_src_count = ARRAY_SIZE(ipa_resource_src),
+ .resource_src = ipa_resource_src,
+ .resource_dst_count = ARRAY_SIZE(ipa_resource_dst),
+ .resource_dst = ipa_resource_dst,
+};
+
+/* IPA-resident memory region data for an SoC having IPA v3.1 */
+static const struct ipa_mem ipa_mem_local_data[] = {
+ {
+ .id = IPA_MEM_UC_SHARED,
+ .offset = 0x0000,
+ .size = 0x0080,
+ .canary_count = 0,
+ },
+ {
+ .id = IPA_MEM_UC_INFO,
+ .offset = 0x0080,
+ .size = 0x0200,
+ .canary_count = 0,
+ },
+ {
+ .id = IPA_MEM_V4_FILTER_HASHED,
+ .offset = 0x0288,
+ .size = 0x0078,
+ .canary_count = 2,
+ },
+ {
+ .id = IPA_MEM_V4_FILTER,
+ .offset = 0x0308,
+ .size = 0x0078,
+ .canary_count = 2,
+ },
+ {
+ .id = IPA_MEM_V6_FILTER_HASHED,
+ .offset = 0x0388,
+ .size = 0x0078,
+ .canary_count = 2,
+ },
+ {
+ .id = IPA_MEM_V6_FILTER,
+ .offset = 0x0408,
+ .size = 0x0078,
+ .canary_count = 2,
+ },
+ {
+ .id = IPA_MEM_V4_ROUTE_HASHED,
+ .offset = 0x0488,
+ .size = 0x0078,
+ .canary_count = 2,
+ },
+ {
+ .id = IPA_MEM_V4_ROUTE,
+ .offset = 0x0508,
+ .size = 0x0078,
+ .canary_count = 2,
+ },
+ {
+ .id = IPA_MEM_V6_ROUTE_HASHED,
+ .offset = 0x0588,
+ .size = 0x0078,
+ .canary_count = 2,
+ },
+ {
+ .id = IPA_MEM_V6_ROUTE,
+ .offset = 0x0608,
+ .size = 0x0078,
+ .canary_count = 2,
+ },
+ {
+ .id = IPA_MEM_MODEM_HEADER,
+ .offset = 0x0688,
+ .size = 0x0140,
+ .canary_count = 2,
+ },
+ {
+ .id = IPA_MEM_MODEM_PROC_CTX,
+ .offset = 0x07d0,
+ .size = 0x0200,
+ .canary_count = 2,
+ },
+ {
+ .id = IPA_MEM_AP_PROC_CTX,
+ .offset = 0x09d0,
+ .size = 0x0200,
+ .canary_count = 0,
+ },
+ {
+ .id = IPA_MEM_MODEM,
+ .offset = 0x0bd8,
+ .size = 0x1424,
+ .canary_count = 0,
+ },
+ {
+ .id = IPA_MEM_END_MARKER,
+ .offset = 0x2000,
+ .size = 0,
+ .canary_count = 1,
+ },
+};
+
+/* Memory configuration data for an SoC having IPA v3.1 */
+static const struct ipa_mem_data ipa_mem_data = {
+ .local_count = ARRAY_SIZE(ipa_mem_local_data),
+ .local = ipa_mem_local_data,
+ .imem_addr = 0x146bd000,
+ .imem_size = 0x00002000,
+ .smem_id = 497,
+ .smem_size = 0x00002000,
+};
+
+/* Interconnect bandwidths are in 1000 byte/second units */
+static const struct ipa_interconnect_data ipa_interconnect_data[] = {
+ {
+ .name = "memory",
+ .peak_bandwidth = 640000, /* 640 MBps */
+ .average_bandwidth = 80000, /* 80 MBps */
+ },
+ {
+ .name = "imem",
+ .peak_bandwidth = 640000, /* 640 MBps */
+ .average_bandwidth = 80000, /* 80 MBps */
+ },
+ /* Average bandwidth is unused for the next interconnect */
+ {
+ .name = "config",
+ .peak_bandwidth = 80000, /* 80 MBps */
+ .average_bandwidth = 0, /* unused */
+ },
+};
+
+/* Clock and interconnect configuration data for an SoC having IPA v3.1 */
+static const struct ipa_clock_data ipa_clock_data = {
+ .core_clock_rate = 16 * 1000 * 1000, /* Hz */
+ .interconnect_count = ARRAY_SIZE(ipa_interconnect_data),
+ .interconnect_data = ipa_interconnect_data,
+};
+
+/* Configuration data for an SoC having IPA v3.1 */
+const struct ipa_data ipa_data_v3_1 = {
+ .version = IPA_VERSION_3_1,
+ .backward_compat = BCR_CMDQ_L_LACK_ONE_ENTRY_FMASK,
+ .qsb_count = ARRAY_SIZE(ipa_qsb_data),
+ .qsb_data = ipa_qsb_data,
+ .endpoint_count = ARRAY_SIZE(ipa_gsi_endpoint_data),
+ .endpoint_data = ipa_gsi_endpoint_data,
+ .resource_data = &ipa_resource_data,
+ .mem_data = &ipa_mem_data,
+ .clock_data = &ipa_clock_data,
+};
/* IPA-resident memory region data for an SoC having IPA v3.5.1 */
static const struct ipa_mem ipa_mem_local_data[] = {
- [IPA_MEM_UC_SHARED] = {
+ {
+ .id = IPA_MEM_UC_SHARED,
.offset = 0x0000,
.size = 0x0080,
.canary_count = 0,
},
- [IPA_MEM_UC_INFO] = {
+ {
+ .id = IPA_MEM_UC_INFO,
.offset = 0x0080,
.size = 0x0200,
.canary_count = 0,
},
- [IPA_MEM_V4_FILTER_HASHED] = {
+ {
+ .id = IPA_MEM_V4_FILTER_HASHED,
.offset = 0x0288,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V4_FILTER] = {
+ {
+ .id = IPA_MEM_V4_FILTER,
.offset = 0x0308,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_FILTER_HASHED] = {
+ {
+ .id = IPA_MEM_V6_FILTER_HASHED,
.offset = 0x0388,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_FILTER] = {
+ {
+ .id = IPA_MEM_V6_FILTER,
.offset = 0x0408,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V4_ROUTE_HASHED] = {
+ {
+ .id = IPA_MEM_V4_ROUTE_HASHED,
.offset = 0x0488,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V4_ROUTE] = {
+ {
+ .id = IPA_MEM_V4_ROUTE,
.offset = 0x0508,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_ROUTE_HASHED] = {
+ {
+ .id = IPA_MEM_V6_ROUTE_HASHED,
.offset = 0x0588,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_ROUTE] = {
+ {
+ .id = IPA_MEM_V6_ROUTE,
.offset = 0x0608,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_MODEM_HEADER] = {
+ {
+ .id = IPA_MEM_MODEM_HEADER,
.offset = 0x0688,
.size = 0x0140,
.canary_count = 2,
},
- [IPA_MEM_MODEM_PROC_CTX] = {
+ {
+ .id = IPA_MEM_MODEM_PROC_CTX,
.offset = 0x07d0,
.size = 0x0200,
.canary_count = 2,
},
- [IPA_MEM_AP_PROC_CTX] = {
+ {
+ .id = IPA_MEM_AP_PROC_CTX,
.offset = 0x09d0,
.size = 0x0200,
.canary_count = 0,
},
- [IPA_MEM_MODEM] = {
+ {
+ .id = IPA_MEM_MODEM,
.offset = 0x0bd8,
.size = 0x1024,
.canary_count = 0,
},
- [IPA_MEM_UC_EVENT_RING] = {
+ {
+ .id = IPA_MEM_UC_EVENT_RING,
.offset = 0x1c00,
.size = 0x0400,
.canary_count = 1,
/* IPA-resident memory region data for an SoC having IPA v4.11 */
static const struct ipa_mem ipa_mem_local_data[] = {
- [IPA_MEM_UC_SHARED] = {
+ {
+ .id = IPA_MEM_UC_SHARED,
.offset = 0x0000,
.size = 0x0080,
.canary_count = 0,
},
- [IPA_MEM_UC_INFO] = {
+ {
+ .id = IPA_MEM_UC_INFO,
.offset = 0x0080,
.size = 0x0200,
.canary_count = 0,
},
- [IPA_MEM_V4_FILTER_HASHED] = {
+ {
+ .id = IPA_MEM_V4_FILTER_HASHED,
.offset = 0x0288,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V4_FILTER] = {
+ {
+ .id = IPA_MEM_V4_FILTER,
.offset = 0x0308,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_FILTER_HASHED] = {
+ {
+ .id = IPA_MEM_V6_FILTER_HASHED,
.offset = 0x0388,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_FILTER] = {
+ {
+ .id = IPA_MEM_V6_FILTER,
.offset = 0x0408,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V4_ROUTE_HASHED] = {
+ {
+ .id = IPA_MEM_V4_ROUTE_HASHED,
.offset = 0x0488,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V4_ROUTE] = {
+ {
+ .id = IPA_MEM_V4_ROUTE,
.offset = 0x0508,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_ROUTE_HASHED] = {
+ {
+ .id = IPA_MEM_V6_ROUTE_HASHED,
.offset = 0x0588,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_ROUTE] = {
+ {
+ .id = IPA_MEM_V6_ROUTE,
.offset = 0x0608,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_MODEM_HEADER] = {
+ {
+ .id = IPA_MEM_MODEM_HEADER,
.offset = 0x0688,
.size = 0x0240,
.canary_count = 2,
},
- [IPA_MEM_AP_HEADER] = {
+ {
+ .id = IPA_MEM_AP_HEADER,
.offset = 0x08c8,
.size = 0x0200,
.canary_count = 0,
},
- [IPA_MEM_MODEM_PROC_CTX] = {
+ {
+ .id = IPA_MEM_MODEM_PROC_CTX,
.offset = 0x0ad0,
.size = 0x0200,
.canary_count = 2,
},
- [IPA_MEM_AP_PROC_CTX] = {
+ {
+ .id = IPA_MEM_AP_PROC_CTX,
.offset = 0x0cd0,
.size = 0x0200,
.canary_count = 0,
},
- [IPA_MEM_NAT_TABLE] = {
+ {
+ .id = IPA_MEM_NAT_TABLE,
.offset = 0x0ee0,
.size = 0x0d00,
.canary_count = 4,
},
- [IPA_MEM_PDN_CONFIG] = {
+ {
+ .id = IPA_MEM_PDN_CONFIG,
.offset = 0x1be8,
.size = 0x0050,
.canary_count = 0,
},
- [IPA_MEM_STATS_QUOTA_MODEM] = {
+ {
+ .id = IPA_MEM_STATS_QUOTA_MODEM,
.offset = 0x1c40,
.size = 0x0030,
.canary_count = 4,
},
- [IPA_MEM_STATS_QUOTA_AP] = {
+ {
+ .id = IPA_MEM_STATS_QUOTA_AP,
.offset = 0x1c70,
.size = 0x0048,
.canary_count = 0,
},
- [IPA_MEM_STATS_TETHERING] = {
+ {
+ .id = IPA_MEM_STATS_TETHERING,
.offset = 0x1cb8,
.size = 0x0238,
.canary_count = 0,
},
- [IPA_MEM_STATS_DROP] = {
+ {
+ .id = IPA_MEM_STATS_DROP,
.offset = 0x1ef0,
.size = 0x0020,
.canary_count = 0,
},
- [IPA_MEM_MODEM] = {
+ {
+ .id = IPA_MEM_MODEM,
.offset = 0x1f18,
.size = 0x100c,
.canary_count = 2,
},
- [IPA_MEM_UC_EVENT_RING] = {
+ {
+ .id = IPA_MEM_END_MARKER,
.offset = 0x3000,
.size = 0x0000,
.canary_count = 1,
/* IPA-resident memory region data for an SoC having IPA v4.2 */
static const struct ipa_mem ipa_mem_local_data[] = {
- [IPA_MEM_UC_SHARED] = {
+ {
+ .id = IPA_MEM_UC_SHARED,
.offset = 0x0000,
.size = 0x0080,
.canary_count = 0,
},
- [IPA_MEM_UC_INFO] = {
+ {
+ .id = IPA_MEM_UC_INFO,
.offset = 0x0080,
.size = 0x0200,
.canary_count = 0,
},
- [IPA_MEM_V4_FILTER_HASHED] = {
+ {
+ .id = IPA_MEM_V4_FILTER_HASHED,
.offset = 0x0288,
.size = 0,
.canary_count = 2,
},
- [IPA_MEM_V4_FILTER] = {
+ {
+ .id = IPA_MEM_V4_FILTER,
.offset = 0x0290,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_FILTER_HASHED] = {
+ {
+ .id = IPA_MEM_V6_FILTER_HASHED,
.offset = 0x0310,
.size = 0,
.canary_count = 2,
},
- [IPA_MEM_V6_FILTER] = {
+ {
+ .id = IPA_MEM_V6_FILTER,
.offset = 0x0318,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V4_ROUTE_HASHED] = {
+ {
+ .id = IPA_MEM_V4_ROUTE_HASHED,
.offset = 0x0398,
.size = 0,
.canary_count = 2,
},
- [IPA_MEM_V4_ROUTE] = {
+ {
+ .id = IPA_MEM_V4_ROUTE,
.offset = 0x03a0,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_ROUTE_HASHED] = {
+ {
+ .id = IPA_MEM_V6_ROUTE_HASHED,
.offset = 0x0420,
.size = 0,
.canary_count = 2,
},
- [IPA_MEM_V6_ROUTE] = {
+ {
+ .id = IPA_MEM_V6_ROUTE,
.offset = 0x0428,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_MODEM_HEADER] = {
+ {
+ .id = IPA_MEM_MODEM_HEADER,
.offset = 0x04a8,
.size = 0x0140,
.canary_count = 2,
},
- [IPA_MEM_MODEM_PROC_CTX] = {
+ {
+ .id = IPA_MEM_MODEM_PROC_CTX,
.offset = 0x05f0,
.size = 0x0200,
.canary_count = 2,
},
- [IPA_MEM_AP_PROC_CTX] = {
+ {
+ .id = IPA_MEM_AP_PROC_CTX,
.offset = 0x07f0,
.size = 0x0200,
.canary_count = 0,
},
- [IPA_MEM_PDN_CONFIG] = {
+ {
+ .id = IPA_MEM_PDN_CONFIG,
.offset = 0x09f8,
.size = 0x0050,
.canary_count = 2,
},
- [IPA_MEM_STATS_QUOTA_MODEM] = {
+ {
+ .id = IPA_MEM_STATS_QUOTA_MODEM,
.offset = 0x0a50,
.size = 0x0060,
.canary_count = 2,
},
- [IPA_MEM_STATS_TETHERING] = {
+ {
+ .id = IPA_MEM_STATS_TETHERING,
.offset = 0x0ab0,
.size = 0x0140,
.canary_count = 0,
},
- [IPA_MEM_MODEM] = {
+ {
+ .id = IPA_MEM_MODEM,
.offset = 0x0bf0,
.size = 0x140c,
.canary_count = 0,
},
- [IPA_MEM_UC_EVENT_RING] = {
+ {
+ .id = IPA_MEM_END_MARKER,
.offset = 0x2000,
.size = 0,
.canary_count = 1,
/* IPA-resident memory region data for an SoC having IPA v4.5 */
static const struct ipa_mem ipa_mem_local_data[] = {
- [IPA_MEM_UC_SHARED] = {
+ {
+ .id = IPA_MEM_UC_SHARED,
.offset = 0x0000,
.size = 0x0080,
.canary_count = 0,
},
- [IPA_MEM_UC_INFO] = {
+ {
+ .id = IPA_MEM_UC_INFO,
.offset = 0x0080,
.size = 0x0200,
.canary_count = 0,
},
- [IPA_MEM_V4_FILTER_HASHED] = {
+ {
+ .id = IPA_MEM_V4_FILTER_HASHED,
.offset = 0x0288,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V4_FILTER] = {
+ {
+ .id = IPA_MEM_V4_FILTER,
.offset = 0x0308,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_FILTER_HASHED] = {
+ {
+ .id = IPA_MEM_V6_FILTER_HASHED,
.offset = 0x0388,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_FILTER] = {
+ {
+ .id = IPA_MEM_V6_FILTER,
.offset = 0x0408,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V4_ROUTE_HASHED] = {
+ {
+ .id = IPA_MEM_V4_ROUTE_HASHED,
.offset = 0x0488,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V4_ROUTE] = {
+ {
+ .id = IPA_MEM_V4_ROUTE,
.offset = 0x0508,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_ROUTE_HASHED] = {
+ {
+ .id = IPA_MEM_V6_ROUTE_HASHED,
.offset = 0x0588,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_ROUTE] = {
+ {
+ .id = IPA_MEM_V6_ROUTE,
.offset = 0x0608,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_MODEM_HEADER] = {
+ {
+ .id = IPA_MEM_MODEM_HEADER,
.offset = 0x0688,
.size = 0x0240,
.canary_count = 2,
},
- [IPA_MEM_AP_HEADER] = {
+ {
+ .id = IPA_MEM_AP_HEADER,
.offset = 0x08c8,
.size = 0x0200,
.canary_count = 0,
},
- [IPA_MEM_MODEM_PROC_CTX] = {
+ {
+ .id = IPA_MEM_MODEM_PROC_CTX,
.offset = 0x0ad0,
.size = 0x0b20,
.canary_count = 2,
},
- [IPA_MEM_AP_PROC_CTX] = {
+ {
+ .id = IPA_MEM_AP_PROC_CTX,
.offset = 0x15f0,
.size = 0x0200,
.canary_count = 0,
},
- [IPA_MEM_NAT_TABLE] = {
+ {
+ .id = IPA_MEM_NAT_TABLE,
.offset = 0x1800,
.size = 0x0d00,
.canary_count = 4,
},
- [IPA_MEM_STATS_QUOTA_MODEM] = {
+ {
+ .id = IPA_MEM_STATS_QUOTA_MODEM,
.offset = 0x2510,
.size = 0x0030,
.canary_count = 4,
},
- [IPA_MEM_STATS_QUOTA_AP] = {
+ {
+ .id = IPA_MEM_STATS_QUOTA_AP,
.offset = 0x2540,
.size = 0x0048,
.canary_count = 0,
},
- [IPA_MEM_STATS_TETHERING] = {
+ {
+ .id = IPA_MEM_STATS_TETHERING,
.offset = 0x2588,
.size = 0x0238,
.canary_count = 0,
},
- [IPA_MEM_STATS_FILTER_ROUTE] = {
+ {
+ .id = IPA_MEM_STATS_FILTER_ROUTE,
.offset = 0x27c0,
.size = 0x0800,
.canary_count = 0,
},
- [IPA_MEM_STATS_DROP] = {
+ {
+ .id = IPA_MEM_STATS_DROP,
.offset = 0x2fc0,
.size = 0x0020,
.canary_count = 0,
},
- [IPA_MEM_MODEM] = {
+ {
+ .id = IPA_MEM_MODEM,
.offset = 0x2fe8,
.size = 0x0800,
.canary_count = 2,
},
- [IPA_MEM_UC_EVENT_RING] = {
+ {
+ .id = IPA_MEM_UC_EVENT_RING,
.offset = 0x3800,
.size = 0x1000,
.canary_count = 1,
},
- [IPA_MEM_PDN_CONFIG] = {
+ {
+ .id = IPA_MEM_PDN_CONFIG,
.offset = 0x4800,
.size = 0x0050,
.canary_count = 0,
/* IPA-resident memory region data for an SoC having IPA v4.9 */
static const struct ipa_mem ipa_mem_local_data[] = {
- [IPA_MEM_UC_SHARED] = {
+ {
+ .id = IPA_MEM_UC_SHARED,
.offset = 0x0000,
.size = 0x0080,
.canary_count = 0,
},
- [IPA_MEM_UC_INFO] = {
+ {
+ .id = IPA_MEM_UC_INFO,
.offset = 0x0080,
.size = 0x0200,
.canary_count = 0,
},
- [IPA_MEM_V4_FILTER_HASHED] = { .offset = 0x0288,
+ {
+ .id = IPA_MEM_V4_FILTER_HASHED,
+ .offset = 0x0288,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V4_FILTER] = {
+ {
+ .id = IPA_MEM_V4_FILTER,
.offset = 0x0308,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_FILTER_HASHED] = {
+ {
+ .id = IPA_MEM_V6_FILTER_HASHED,
.offset = 0x0388,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_FILTER] = {
+ {
+ .id = IPA_MEM_V6_FILTER,
.offset = 0x0408,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V4_ROUTE_HASHED] = {
+ {
+ .id = IPA_MEM_V4_ROUTE_HASHED,
.offset = 0x0488,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V4_ROUTE] = {
+ {
+ .id = IPA_MEM_V4_ROUTE,
.offset = 0x0508,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_ROUTE_HASHED] = {
+ {
+ .id = IPA_MEM_V6_ROUTE_HASHED,
.offset = 0x0588,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_V6_ROUTE] = {
+ {
+ .id = IPA_MEM_V6_ROUTE,
.offset = 0x0608,
.size = 0x0078,
.canary_count = 2,
},
- [IPA_MEM_MODEM_HEADER] = {
+ {
+ .id = IPA_MEM_MODEM_HEADER,
.offset = 0x0688,
.size = 0x0240,
.canary_count = 2,
},
- [IPA_MEM_AP_HEADER] = {
+ {
+ .id = IPA_MEM_AP_HEADER,
.offset = 0x08c8,
.size = 0x0200,
.canary_count = 0,
},
- [IPA_MEM_MODEM_PROC_CTX] = {
+ {
+ .id = IPA_MEM_MODEM_PROC_CTX,
.offset = 0x0ad0,
.size = 0x0b20,
.canary_count = 2,
},
- [IPA_MEM_AP_PROC_CTX] = {
+ {
+ .id = IPA_MEM_AP_PROC_CTX,
.offset = 0x15f0,
.size = 0x0200,
.canary_count = 0,
},
- [IPA_MEM_NAT_TABLE] = {
+ {
+ .id = IPA_MEM_NAT_TABLE,
.offset = 0x1800,
.size = 0x0d00,
.canary_count = 4,
},
- [IPA_MEM_STATS_QUOTA_MODEM] = {
+ {
+ .id = IPA_MEM_STATS_QUOTA_MODEM,
.offset = 0x2510,
.size = 0x0030,
.canary_count = 4,
},
- [IPA_MEM_STATS_QUOTA_AP] = {
+ {
+ .id = IPA_MEM_STATS_QUOTA_AP,
.offset = 0x2540,
.size = 0x0048,
.canary_count = 0,
},
- [IPA_MEM_STATS_TETHERING] = {
+ {
+ .id = IPA_MEM_STATS_TETHERING,
.offset = 0x2588,
.size = 0x0238,
.canary_count = 0,
},
- [IPA_MEM_STATS_FILTER_ROUTE] = {
+ {
+ .id = IPA_MEM_STATS_FILTER_ROUTE,
.offset = 0x27c0,
.size = 0x0800,
.canary_count = 0,
},
- [IPA_MEM_STATS_DROP] = {
+ {
+ .id = IPA_MEM_STATS_DROP,
.offset = 0x2fc0,
.size = 0x0020,
.canary_count = 0,
},
- [IPA_MEM_MODEM] = {
+ {
+ .id = IPA_MEM_MODEM,
.offset = 0x2fe8,
.size = 0x0800,
.canary_count = 2,
},
- [IPA_MEM_UC_EVENT_RING] = {
+ {
+ .id = IPA_MEM_UC_EVENT_RING,
.offset = 0x3800,
.size = 0x1000,
.canary_count = 1,
},
- [IPA_MEM_PDN_CONFIG] = {
+ {
+ .id = IPA_MEM_PDN_CONFIG,
.offset = 0x4800,
.size = 0x0050,
.canary_count = 0,
const struct ipa_clock_data *clock_data;
};
+extern const struct ipa_data ipa_data_v3_1;
extern const struct ipa_data ipa_data_v3_5_1;
extern const struct ipa_data ipa_data_v4_2;
extern const struct ipa_data ipa_data_v4_5;
#define IPA_STATUS_FLAGS1_RT_RULE_ID_FMASK GENMASK(31, 22)
#define IPA_STATUS_FLAGS2_TAG_FMASK GENMASK_ULL(63, 16)
-#ifdef IPA_VALIDATE
-
static bool ipa_endpoint_data_valid_one(struct ipa *ipa, u32 count,
const struct ipa_gsi_endpoint_data *all_data,
const struct ipa_gsi_endpoint_data *data)
if (ipa_gsi_endpoint_data_empty(data))
return true;
- /* IPA v4.5+ uses checksum offload, not yet supported by RMNet */
- if (ipa->version >= IPA_VERSION_4_5)
- if (data->endpoint.config.checksum)
- return false;
-
if (!data->toward_ipa) {
if (data->endpoint.filter_support) {
dev_err(dev, "filtering not supported for "
return true;
}
-#else /* !IPA_VALIDATE */
-
-static bool ipa_endpoint_data_valid(struct ipa *ipa, u32 count,
- const struct ipa_gsi_endpoint_data *data)
-{
- const struct ipa_gsi_endpoint_data *dp = data;
- enum ipa_endpoint_name name;
-
- if (ipa->version < IPA_VERSION_4_5)
- return true;
-
- /* IPA v4.5+ uses checksum offload, not yet supported by RMNet */
- for (name = 0; name < count; name++, dp++)
- if (data->endpoint.config.checksum)
- return false;
-
- return true;
-}
-
-#endif /* !IPA_VALIDATE */
-
/* Allocate a transaction to use on a non-command endpoint */
static struct gsi_trans *ipa_endpoint_trans_alloc(struct ipa_endpoint *endpoint,
u32 tre_count)
static void ipa_endpoint_init_cfg(struct ipa_endpoint *endpoint)
{
u32 offset = IPA_REG_ENDP_INIT_CFG_N_OFFSET(endpoint->endpoint_id);
+ enum ipa_cs_offload_en enabled;
u32 val = 0;
/* FRAG_OFFLOAD_EN is 0 */
if (endpoint->data->checksum) {
+ enum ipa_version version = endpoint->ipa->version;
+
if (endpoint->toward_ipa) {
u32 checksum_offset;
- val |= u32_encode_bits(IPA_CS_OFFLOAD_UL,
- CS_OFFLOAD_EN_FMASK);
/* Checksum header offset is in 4-byte units */
checksum_offset = sizeof(struct rmnet_map_header);
checksum_offset /= sizeof(u32);
val |= u32_encode_bits(checksum_offset,
CS_METADATA_HDR_OFFSET_FMASK);
+
+ enabled = version < IPA_VERSION_4_5
+ ? IPA_CS_OFFLOAD_UL
+ : IPA_CS_OFFLOAD_INLINE;
} else {
- val |= u32_encode_bits(IPA_CS_OFFLOAD_DL,
- CS_OFFLOAD_EN_FMASK);
+ enabled = version < IPA_VERSION_4_5
+ ? IPA_CS_OFFLOAD_DL
+ : IPA_CS_OFFLOAD_INLINE;
}
} else {
- val |= u32_encode_bits(IPA_CS_OFFLOAD_NONE,
- CS_OFFLOAD_EN_FMASK);
+ enabled = IPA_CS_OFFLOAD_NONE;
}
+ val |= u32_encode_bits(enabled, CS_OFFLOAD_EN_FMASK);
/* CS_GEN_QMB_MASTER_SEL is 0 */
iowrite32(val, endpoint->ipa->reg_virt + offset);
iowrite32(val, endpoint->ipa->reg_virt + offset);
}
+static u32
+ipa_qmap_header_size(enum ipa_version version, struct ipa_endpoint *endpoint)
+{
+ u32 header_size = sizeof(struct rmnet_map_header);
+
+ /* Without checksum offload, we just have the MAP header */
+ if (!endpoint->data->checksum)
+ return header_size;
+
+ if (version < IPA_VERSION_4_5) {
+ /* Checksum header inserted for AP TX endpoints only */
+ if (endpoint->toward_ipa)
+ header_size += sizeof(struct rmnet_map_ul_csum_header);
+ } else {
+ /* Checksum header is used in both directions */
+ header_size += sizeof(struct rmnet_map_v5_csum_header);
+ }
+
+ return header_size;
+}
+
/**
* ipa_endpoint_init_hdr() - Initialize HDR endpoint configuration register
* @endpoint: Endpoint pointer
u32 val = 0;
if (endpoint->data->qmap) {
- size_t header_size = sizeof(struct rmnet_map_header);
enum ipa_version version = ipa->version;
+ size_t header_size;
- /* We might supply a checksum header after the QMAP header */
- if (endpoint->toward_ipa && endpoint->data->checksum)
- header_size += sizeof(struct rmnet_map_ul_csum_header);
- val |= ipa_header_size_encoded(version, header_size);
+ header_size = ipa_qmap_header_size(version, endpoint);
+ val = ipa_header_size_encoded(version, header_size);
/* Define how to fill fields in a received QMAP header */
if (!endpoint->toward_ipa) {
u32 max;
u32 val;
+ /* Prior to IPAv3.5, the FLAVOR_0 register was not supported.
+ * Furthermore, the endpoints were not grouped such that TX
+ * endpoint numbers started with 0 and RX endpoints had numbers
+ * higher than all TX endpoints, so we can't do the simple
+ * direction check used for newer hardware below.
+ *
+ * For hardware that doesn't support the FLAVOR_0 register,
+ * just set the available mask to support any endpoint, and
+ * assume the configuration is valid.
+ */
+ if (ipa->version < IPA_VERSION_3_5) {
+ ipa->available = ~0;
+ return 0;
+ }
+
/* Find out about the endpoints supplied by the hardware, and ensure
* the highest one doesn't exceed the number we support.
*/
#include "ipa_uc.h"
#include "ipa_interrupt.h"
#include "gsi_trans.h"
+#include "ipa_sysfs.h"
/**
* DOC: The IP Accelerator
/* Implement some hardware workarounds */
if (version >= IPA_VERSION_4_0 && version < IPA_VERSION_4_5) {
- /* Enable open global clocks (not needed for IPA v4.5) */
- val = GLOBAL_FMASK;
- val |= GLOBAL_2X_CLK_FMASK;
- iowrite32(val, ipa->reg_virt + IPA_REG_CLKON_CFG_OFFSET);
-
/* Disable PA mask to allow HOLB drop */
val = ioread32(ipa->reg_virt + IPA_REG_TX_CFG_OFFSET);
val &= ~PA_MASK_EN_FMASK;
iowrite32(val, ipa->reg_virt + IPA_REG_TX_CFG_OFFSET);
+
+ /* Enable open global clocks in the CLKON configuration */
+ val = GLOBAL_FMASK | GLOBAL_2X_CLK_FMASK;
+ } else if (version == IPA_VERSION_3_1) {
+ val = MISC_FMASK; /* Disable MISC clock gating */
+ } else {
+ val = 0; /* No CLKON configuration needed */
}
+ if (val)
+ iowrite32(val, ipa->reg_virt + IPA_REG_CLKON_CFG_OFFSET);
ipa_hardware_config_comp(ipa);
}
ret = of_address_to_resource(node, 0, &res);
+ of_node_put(node);
if (ret) {
dev_err(dev, "error %d getting \"memory-region\" resource\n",
ret);
static const struct of_device_id ipa_match[] = {
{
+ .compatible = "qcom,msm8998-ipa",
+ .data = &ipa_data_v3_1,
+ },
+ {
.compatible = "qcom,sdm845-ipa",
.data = &ipa_data_v3_5_1,
},
#endif /* IPA_VALIDATE */
}
+static bool ipa_version_valid(enum ipa_version version)
+{
+ switch (version) {
+ case IPA_VERSION_3_0:
+ case IPA_VERSION_3_1:
+ case IPA_VERSION_3_5:
+ case IPA_VERSION_3_5_1:
+ case IPA_VERSION_4_0:
+ case IPA_VERSION_4_1:
+ case IPA_VERSION_4_2:
+ case IPA_VERSION_4_5:
+ case IPA_VERSION_4_7:
+ case IPA_VERSION_4_9:
+ case IPA_VERSION_4_11:
+ return true;
+
+ default:
+ return false;
+ }
+}
+
/**
* ipa_probe() - IPA platform driver probe function
* @pdev: Platform device pointer
/* Get configuration data early; needed for clock initialization */
data = of_device_get_match_data(dev);
if (!data) {
- /* This is really IPA_VALIDATE (should never happen) */
dev_err(dev, "matched hardware not supported\n");
return -ENODEV;
}
+ if (!ipa_version_valid(data->version)) {
+ dev_err(dev, "invalid IPA version\n");
+ return -EINVAL;
+ }
+
/* If we need Trust Zone, make sure it's available */
modem_init = of_property_read_bool(dev->of_node, "modem-init");
if (!modem_init)
.resume = ipa_resume,
};
+static const struct attribute_group *ipa_attribute_groups[] = {
+ &ipa_attribute_group,
+ &ipa_feature_attribute_group,
+ &ipa_modem_attribute_group,
+ NULL,
+};
+
static struct platform_driver ipa_driver = {
.probe = ipa_probe,
.remove = ipa_remove,
.name = "ipa",
.pm = &ipa_pm_ops,
.of_match_table = ipa_match,
+ .dev_groups = ipa_attribute_groups,
},
};
/* SMEM host id representing the modem. */
#define QCOM_SMEM_HOST_MODEM 1
+const struct ipa_mem *ipa_mem_find(struct ipa *ipa, enum ipa_mem_id mem_id)
+{
+ u32 i;
+
+ for (i = 0; i < ipa->mem_count; i++) {
+ const struct ipa_mem *mem = &ipa->mem[i];
+
+ if (mem->id == mem_id)
+ return mem;
+ }
+
+ return NULL;
+}
+
/* Add an immediate command to a transaction that zeroes a memory region */
static void
-ipa_mem_zero_region_add(struct gsi_trans *trans, const struct ipa_mem *mem)
+ipa_mem_zero_region_add(struct gsi_trans *trans, enum ipa_mem_id mem_id)
{
struct ipa *ipa = container_of(trans->gsi, struct ipa, gsi);
+ const struct ipa_mem *mem = ipa_mem_find(ipa, mem_id);
dma_addr_t addr = ipa->zero_addr;
if (!mem->size)
int ipa_mem_setup(struct ipa *ipa)
{
dma_addr_t addr = ipa->zero_addr;
+ const struct ipa_mem *mem;
struct gsi_trans *trans;
u32 offset;
u16 size;
return -EBUSY;
}
- /* Initialize IPA-local header memory. The modem and AP header
- * regions are contiguous, and initialized together.
+ /* Initialize IPA-local header memory. The AP header region, if
+ * present, is contiguous with and follows the modem header region,
+ * and they are initialized together.
*/
- offset = ipa->mem[IPA_MEM_MODEM_HEADER].offset;
- size = ipa->mem[IPA_MEM_MODEM_HEADER].size;
- size += ipa->mem[IPA_MEM_AP_HEADER].size;
+ mem = ipa_mem_find(ipa, IPA_MEM_MODEM_HEADER);
+ offset = mem->offset;
+ size = mem->size;
+ mem = ipa_mem_find(ipa, IPA_MEM_AP_HEADER);
+ if (mem)
+ size += mem->size;
ipa_cmd_hdr_init_local_add(trans, offset, size, addr);
- ipa_mem_zero_region_add(trans, &ipa->mem[IPA_MEM_MODEM_PROC_CTX]);
-
- ipa_mem_zero_region_add(trans, &ipa->mem[IPA_MEM_AP_PROC_CTX]);
-
- ipa_mem_zero_region_add(trans, &ipa->mem[IPA_MEM_MODEM]);
+ ipa_mem_zero_region_add(trans, IPA_MEM_MODEM_PROC_CTX);
+ ipa_mem_zero_region_add(trans, IPA_MEM_AP_PROC_CTX);
+ ipa_mem_zero_region_add(trans, IPA_MEM_MODEM);
gsi_trans_commit_wait(trans);
/* Tell the hardware where the processing context area is located */
- offset = ipa->mem_offset + ipa->mem[IPA_MEM_MODEM_PROC_CTX].offset;
+ mem = ipa_mem_find(ipa, IPA_MEM_MODEM_PROC_CTX);
+ offset = ipa->mem_offset + mem->offset;
val = proc_cntxt_base_addr_encoded(ipa->version, offset);
iowrite32(val, ipa->reg_virt + IPA_REG_LOCAL_PKT_PROC_CNTXT_OFFSET);
return 0;
}
-#ifdef IPA_VALIDATE
+/* Is the given memory region ID is valid for the current IPA version? */
+static bool ipa_mem_id_valid(struct ipa *ipa, enum ipa_mem_id mem_id)
+{
+ enum ipa_version version = ipa->version;
+
+ switch (mem_id) {
+ case IPA_MEM_UC_SHARED:
+ case IPA_MEM_UC_INFO:
+ case IPA_MEM_V4_FILTER_HASHED:
+ case IPA_MEM_V4_FILTER:
+ case IPA_MEM_V6_FILTER_HASHED:
+ case IPA_MEM_V6_FILTER:
+ case IPA_MEM_V4_ROUTE_HASHED:
+ case IPA_MEM_V4_ROUTE:
+ case IPA_MEM_V6_ROUTE_HASHED:
+ case IPA_MEM_V6_ROUTE:
+ case IPA_MEM_MODEM_HEADER:
+ case IPA_MEM_AP_HEADER:
+ case IPA_MEM_MODEM_PROC_CTX:
+ case IPA_MEM_AP_PROC_CTX:
+ case IPA_MEM_MODEM:
+ case IPA_MEM_UC_EVENT_RING:
+ case IPA_MEM_PDN_CONFIG:
+ case IPA_MEM_STATS_QUOTA_MODEM:
+ case IPA_MEM_STATS_QUOTA_AP:
+ case IPA_MEM_END_MARKER: /* pseudo region */
+ break;
+
+ case IPA_MEM_STATS_TETHERING:
+ case IPA_MEM_STATS_DROP:
+ if (version < IPA_VERSION_4_0)
+ return false;
+ break;
+
+ case IPA_MEM_STATS_V4_FILTER:
+ case IPA_MEM_STATS_V6_FILTER:
+ case IPA_MEM_STATS_V4_ROUTE:
+ case IPA_MEM_STATS_V6_ROUTE:
+ if (version < IPA_VERSION_4_0 || version > IPA_VERSION_4_2)
+ return false;
+ break;
+
+ case IPA_MEM_NAT_TABLE:
+ case IPA_MEM_STATS_FILTER_ROUTE:
+ if (version < IPA_VERSION_4_5)
+ return false;
+ break;
+
+ default:
+ return false;
+ }
+
+ return true;
+}
+
+/* Must the given memory region be present in the configuration? */
+static bool ipa_mem_id_required(struct ipa *ipa, enum ipa_mem_id mem_id)
+{
+ switch (mem_id) {
+ case IPA_MEM_UC_SHARED:
+ case IPA_MEM_UC_INFO:
+ case IPA_MEM_V4_FILTER_HASHED:
+ case IPA_MEM_V4_FILTER:
+ case IPA_MEM_V6_FILTER_HASHED:
+ case IPA_MEM_V6_FILTER:
+ case IPA_MEM_V4_ROUTE_HASHED:
+ case IPA_MEM_V4_ROUTE:
+ case IPA_MEM_V6_ROUTE_HASHED:
+ case IPA_MEM_V6_ROUTE:
+ case IPA_MEM_MODEM_HEADER:
+ case IPA_MEM_MODEM_PROC_CTX:
+ case IPA_MEM_AP_PROC_CTX:
+ case IPA_MEM_MODEM:
+ return true;
+
+ case IPA_MEM_PDN_CONFIG:
+ case IPA_MEM_STATS_QUOTA_MODEM:
+ case IPA_MEM_STATS_TETHERING:
+ return ipa->version >= IPA_VERSION_4_0;
-static bool ipa_mem_valid(struct ipa *ipa, enum ipa_mem_id mem_id)
+ default:
+ return false; /* Anything else is optional */
+ }
+}
+
+static bool ipa_mem_valid_one(struct ipa *ipa, const struct ipa_mem *mem)
{
- const struct ipa_mem *mem = &ipa->mem[mem_id];
struct device *dev = &ipa->pdev->dev;
+ enum ipa_mem_id mem_id = mem->id;
u16 size_multiple;
+ /* Make sure the memory region is valid for this version of IPA */
+ if (!ipa_mem_id_valid(ipa, mem_id)) {
+ dev_err(dev, "region id %u not valid\n", mem_id);
+ return false;
+ }
+
+ if (!mem->size && !mem->canary_count) {
+ dev_err(dev, "empty memory region %u\n", mem_id);
+ return false;
+ }
+
/* Other than modem memory, sizes must be a multiple of 8 */
size_multiple = mem_id == IPA_MEM_MODEM ? 4 : 8;
if (mem->size % size_multiple)
else if (mem->offset < mem->canary_count * sizeof(__le32))
dev_err(dev, "region %u offset too small for %hu canaries\n",
mem_id, mem->canary_count);
- else if (mem->offset + mem->size > ipa->mem_size)
- dev_err(dev, "region %u ends beyond memory limit (0x%08x)\n",
- mem_id, ipa->mem_size);
+ else if (mem_id == IPA_MEM_END_MARKER && mem->size)
+ dev_err(dev, "non-zero end marker region size\n");
else
return true;
return false;
}
-#else /* !IPA_VALIDATE */
-
-static bool ipa_mem_valid(struct ipa *ipa, enum ipa_mem_id mem_id)
+/* Verify each defined memory region is valid. */
+static bool ipa_mem_valid(struct ipa *ipa, const struct ipa_mem_data *mem_data)
{
+ DECLARE_BITMAP(regions, IPA_MEM_COUNT) = { };
+ struct device *dev = &ipa->pdev->dev;
+ enum ipa_mem_id mem_id;
+ u32 i;
+
+ if (mem_data->local_count > IPA_MEM_COUNT) {
+ dev_err(dev, "too many memory regions (%u > %u)\n",
+ mem_data->local_count, IPA_MEM_COUNT);
+ return false;
+ }
+
+ for (i = 0; i < mem_data->local_count; i++) {
+ const struct ipa_mem *mem = &mem_data->local[i];
+
+ if (__test_and_set_bit(mem->id, regions)) {
+ dev_err(dev, "duplicate memory region %u\n", mem->id);
+ return false;
+ }
+
+ /* Defined regions have non-zero size and/or canary count */
+ if (!ipa_mem_valid_one(ipa, mem))
+ return false;
+ }
+
+ /* Now see if any required regions are not defined */
+ for (mem_id = find_first_zero_bit(regions, IPA_MEM_COUNT);
+ mem_id < IPA_MEM_COUNT;
+ mem_id = find_next_zero_bit(regions, IPA_MEM_COUNT, mem_id + 1)) {
+ if (ipa_mem_id_required(ipa, mem_id))
+ dev_err(dev, "required memory region %u missing\n",
+ mem_id);
+ }
+
return true;
}
-#endif /*! IPA_VALIDATE */
+/* Do all memory regions fit within the IPA local memory? */
+static bool ipa_mem_size_valid(struct ipa *ipa)
+{
+ struct device *dev = &ipa->pdev->dev;
+ u32 limit = ipa->mem_size;
+ u32 i;
+
+ for (i = 0; i < ipa->mem_count; i++) {
+ const struct ipa_mem *mem = &ipa->mem[i];
+
+ if (mem->offset + mem->size <= limit)
+ continue;
+
+ dev_err(dev, "region %u ends beyond memory limit (0x%08x)\n",
+ mem->id, limit);
+
+ return false;
+ }
+
+ return true;
+}
/**
* ipa_mem_config() - Configure IPA shared memory
int ipa_mem_config(struct ipa *ipa)
{
struct device *dev = &ipa->pdev->dev;
- enum ipa_mem_id mem_id;
+ const struct ipa_mem *mem;
dma_addr_t addr;
u32 mem_size;
void *virt;
u32 val;
+ u32 i;
/* Check the advertised location and size of the shared memory area */
val = ioread32(ipa->reg_virt + IPA_REG_SHARED_MEM_SIZE_OFFSET);
mem_size);
}
+ /* We know our memory size; make sure regions are all in range */
+ if (!ipa_mem_size_valid(ipa))
+ return -EINVAL;
+
/* Prealloc DMA memory for zeroing regions */
virt = dma_alloc_coherent(dev, IPA_MEM_MAX, &addr, GFP_KERNEL);
if (!virt)
ipa->zero_virt = virt;
ipa->zero_size = IPA_MEM_MAX;
- /* Verify each defined memory region is valid, and if indicated
- * for the region, write "canary" values in the space prior to
- * the region's base address.
+ /* For each defined region, write "canary" values in the
+ * space prior to the region's base address if indicated.
*/
- for (mem_id = 0; mem_id < IPA_MEM_COUNT; mem_id++) {
- const struct ipa_mem *mem = &ipa->mem[mem_id];
- u16 canary_count;
+ for (i = 0; i < ipa->mem_count; i++) {
+ u16 canary_count = ipa->mem[i].canary_count;
__le32 *canary;
- /* Validate all regions (even undefined ones) */
- if (!ipa_mem_valid(ipa, mem_id))
- goto err_dma_free;
-
- /* Skip over undefined regions */
- if (!mem->offset && !mem->size)
- continue;
-
- canary_count = mem->canary_count;
if (!canary_count)
continue;
/* Write canary values in the space before the region */
- canary = ipa->mem_virt + ipa->mem_offset + mem->offset;
+ canary = ipa->mem_virt + ipa->mem_offset + ipa->mem[i].offset;
do
*--canary = IPA_MEM_CANARY_VAL;
while (--canary_count);
if (!ipa_cmd_data_valid(ipa))
goto err_dma_free;
- /* Verify the microcontroller ring alignment (0 is OK too) */
- if (ipa->mem[IPA_MEM_UC_EVENT_RING].offset % 1024) {
+ /* Verify the microcontroller ring alignment (if defined) */
+ mem = ipa_mem_find(ipa, IPA_MEM_UC_EVENT_RING);
+ if (mem && mem->offset % 1024) {
dev_err(dev, "microcontroller ring not 1024-byte aligned\n");
goto err_dma_free;
}
return -EBUSY;
}
- ipa_mem_zero_region_add(trans, &ipa->mem[IPA_MEM_MODEM_HEADER]);
-
- ipa_mem_zero_region_add(trans, &ipa->mem[IPA_MEM_MODEM_PROC_CTX]);
-
- ipa_mem_zero_region_add(trans, &ipa->mem[IPA_MEM_MODEM]);
+ ipa_mem_zero_region_add(trans, IPA_MEM_MODEM_HEADER);
+ ipa_mem_zero_region_add(trans, IPA_MEM_MODEM_PROC_CTX);
+ ipa_mem_zero_region_add(trans, IPA_MEM_MODEM);
gsi_trans_commit_wait(trans);
* (in this case, the modem). An allocation from SMEM is persistent
* until the AP reboots; there is no way to free an allocated SMEM
* region. Allocation only reserves the space; to use it you need
- * to "get" a pointer it (this implies no reference counting).
+ * to "get" a pointer it (this does not imply reference counting).
* The item might have already been allocated, in which case we
* use it unless the size isn't what we expect.
*/
struct resource *res;
int ret;
- if (mem_data->local_count > IPA_MEM_COUNT) {
- dev_err(dev, "to many memory regions (%u > %u)\n",
- mem_data->local_count, IPA_MEM_COUNT);
+ /* Make sure the set of defined memory regions is valid */
+ if (!ipa_mem_valid(ipa, mem_data))
return -EINVAL;
- }
+
+ ipa->mem_count = mem_data->local_count;
+ ipa->mem = mem_data->local;
ret = dma_set_mask_and_coherent(&ipa->pdev->dev, DMA_BIT_MASK(64));
if (ret) {
ipa->mem_addr = res->start;
ipa->mem_size = resource_size(res);
- /* The ipa->mem[] array is indexed by enum ipa_mem_id values */
- ipa->mem = mem_data->local;
-
ret = ipa_imem_init(ipa, mem_data->imem_addr, mem_data->imem_size);
if (ret)
goto err_unmap;
IPA_MEM_V6_ROUTE_HASHED, /* 2 canaries */
IPA_MEM_V6_ROUTE, /* 2 canaries */
IPA_MEM_MODEM_HEADER, /* 2 canaries */
- IPA_MEM_AP_HEADER, /* 0 canaries */
+ IPA_MEM_AP_HEADER, /* 0 canaries, optional */
IPA_MEM_MODEM_PROC_CTX, /* 2 canaries */
IPA_MEM_AP_PROC_CTX, /* 0 canaries */
- IPA_MEM_NAT_TABLE, /* 4 canaries (IPA v4.5 and above) */
- IPA_MEM_PDN_CONFIG, /* 0/2 canaries (IPA v4.0 and above) */
- IPA_MEM_STATS_QUOTA_MODEM, /* 2/4 canaries (IPA v4.0 and above) */
- IPA_MEM_STATS_QUOTA_AP, /* 0 canaries (IPA v4.0 and above) */
- IPA_MEM_STATS_TETHERING, /* 0 canaries (IPA v4.0 and above) */
+ IPA_MEM_MODEM, /* 0/2 canaries */
+ IPA_MEM_UC_EVENT_RING, /* 1 canary, optional */
+ IPA_MEM_PDN_CONFIG, /* 0/2 canaries (IPA v4.0+) */
+ IPA_MEM_STATS_QUOTA_MODEM, /* 2/4 canaries (IPA v4.0+) */
+ IPA_MEM_STATS_QUOTA_AP, /* 0 canaries, optional (IPA v4.0+) */
+ IPA_MEM_STATS_TETHERING, /* 0 canaries (IPA v4.0+) */
+ IPA_MEM_STATS_DROP, /* 0 canaries, optional (IPA v4.0+) */
+ /* The next 5 filter and route statistics regions are optional */
IPA_MEM_STATS_V4_FILTER, /* 0 canaries (IPA v4.0-v4.2) */
IPA_MEM_STATS_V6_FILTER, /* 0 canaries (IPA v4.0-v4.2) */
IPA_MEM_STATS_V4_ROUTE, /* 0 canaries (IPA v4.0-v4.2) */
IPA_MEM_STATS_V6_ROUTE, /* 0 canaries (IPA v4.0-v4.2) */
- IPA_MEM_STATS_FILTER_ROUTE, /* 0 canaries (IPA v4.5 and above) */
- IPA_MEM_STATS_DROP, /* 0 canaries (IPA v4.0 and above) */
- IPA_MEM_MODEM, /* 0/2 canaries */
- IPA_MEM_UC_EVENT_RING, /* 1 canary */
+ IPA_MEM_STATS_FILTER_ROUTE, /* 0 canaries (IPA v4.5+) */
+ IPA_MEM_NAT_TABLE, /* 4 canaries, optional (IPA v4.5+) */
+ IPA_MEM_END_MARKER, /* 1 canary (not a real region) */
IPA_MEM_COUNT, /* Number of regions (not an index) */
};
/**
* struct ipa_mem - IPA local memory region description
+ * @id: memory region identifier
* @offset: offset in IPA memory space to base of the region
* @size: size in bytes base of the region
* @canary_count: Number of 32-bit "canary" values that precede region
*/
struct ipa_mem {
+ enum ipa_mem_id id;
u32 offset;
u16 size;
u16 canary_count;
};
+const struct ipa_mem *ipa_mem_find(struct ipa *ipa, enum ipa_mem_id mem_id);
+
int ipa_mem_config(struct ipa *ipa);
void ipa_mem_deconfig(struct ipa *ipa);
req.platform_type_valid = 1;
req.platform_type = IPA_QMI_PLATFORM_TYPE_MSM_ANDROID;
- mem = &ipa->mem[IPA_MEM_MODEM_HEADER];
+ mem = ipa_mem_find(ipa, IPA_MEM_MODEM_HEADER);
if (mem->size) {
req.hdr_tbl_info_valid = 1;
req.hdr_tbl_info.start = ipa->mem_offset + mem->offset;
req.hdr_tbl_info.end = req.hdr_tbl_info.start + mem->size - 1;
}
- mem = &ipa->mem[IPA_MEM_V4_ROUTE];
+ mem = ipa_mem_find(ipa, IPA_MEM_V4_ROUTE);
req.v4_route_tbl_info_valid = 1;
req.v4_route_tbl_info.start = ipa->mem_offset + mem->offset;
req.v4_route_tbl_info.count = mem->size / sizeof(__le64);
- mem = &ipa->mem[IPA_MEM_V6_ROUTE];
+ mem = ipa_mem_find(ipa, IPA_MEM_V6_ROUTE);
req.v6_route_tbl_info_valid = 1;
req.v6_route_tbl_info.start = ipa->mem_offset + mem->offset;
req.v6_route_tbl_info.count = mem->size / sizeof(__le64);
- mem = &ipa->mem[IPA_MEM_V4_FILTER];
+ mem = ipa_mem_find(ipa, IPA_MEM_V4_FILTER);
req.v4_filter_tbl_start_valid = 1;
req.v4_filter_tbl_start = ipa->mem_offset + mem->offset;
- mem = &ipa->mem[IPA_MEM_V6_FILTER];
+ mem = ipa_mem_find(ipa, IPA_MEM_V6_FILTER);
req.v6_filter_tbl_start_valid = 1;
req.v6_filter_tbl_start = ipa->mem_offset + mem->offset;
- mem = &ipa->mem[IPA_MEM_MODEM];
+ mem = ipa_mem_find(ipa, IPA_MEM_MODEM);
if (mem->size) {
req.modem_mem_info_valid = 1;
req.modem_mem_info.start = ipa->mem_offset + mem->offset;
/* skip_uc_load_valid and skip_uc_load are set above */
- mem = &ipa->mem[IPA_MEM_MODEM_PROC_CTX];
+ mem = ipa_mem_find(ipa, IPA_MEM_MODEM_PROC_CTX);
if (mem->size) {
req.hdr_proc_ctx_tbl_info_valid = 1;
req.hdr_proc_ctx_tbl_info.start =
/* Nothing to report for the compression table (zip_tbl_info) */
- mem = &ipa->mem[IPA_MEM_V4_ROUTE_HASHED];
+ mem = ipa_mem_find(ipa, IPA_MEM_V4_ROUTE_HASHED);
if (mem->size) {
req.v4_hash_route_tbl_info_valid = 1;
req.v4_hash_route_tbl_info.start =
req.v4_hash_route_tbl_info.count = mem->size / sizeof(__le64);
}
- mem = &ipa->mem[IPA_MEM_V6_ROUTE_HASHED];
+ mem = ipa_mem_find(ipa, IPA_MEM_V6_ROUTE_HASHED);
if (mem->size) {
req.v6_hash_route_tbl_info_valid = 1;
req.v6_hash_route_tbl_info.start =
req.v6_hash_route_tbl_info.count = mem->size / sizeof(__le64);
}
- mem = &ipa->mem[IPA_MEM_V4_FILTER_HASHED];
+ mem = ipa_mem_find(ipa, IPA_MEM_V4_FILTER_HASHED);
if (mem->size) {
req.v4_hash_filter_tbl_start_valid = 1;
req.v4_hash_filter_tbl_start = ipa->mem_offset + mem->offset;
}
- mem = &ipa->mem[IPA_MEM_V6_FILTER_HASHED];
+ mem = ipa_mem_find(ipa, IPA_MEM_V6_FILTER_HASHED);
if (mem->size) {
req.v6_hash_filter_tbl_start_valid = 1;
req.v6_hash_filter_tbl_start = ipa->mem_offset + mem->offset;
}
- /* None of the stats fields are valid (IPA v4.0 and above) */
-
+ /* The stats fields are only valid for IPA v4.0+ */
if (ipa->version >= IPA_VERSION_4_0) {
- mem = &ipa->mem[IPA_MEM_STATS_QUOTA_MODEM];
+ mem = ipa_mem_find(ipa, IPA_MEM_STATS_QUOTA_MODEM);
if (mem->size) {
req.hw_stats_quota_base_addr_valid = 1;
req.hw_stats_quota_base_addr =
req.hw_stats_quota_size = ipa->mem_offset + mem->size;
}
- mem = &ipa->mem[IPA_MEM_STATS_DROP];
- if (mem->size) {
+ /* If the DROP stats region is defined, include it */
+ mem = ipa_mem_find(ipa, IPA_MEM_STATS_DROP);
+ if (mem && mem->size) {
req.hw_stats_drop_base_addr_valid = 1;
req.hw_stats_drop_base_addr =
ipa->mem_offset + mem->offset;
IPA_CS_OFFLOAD_NONE = 0x0,
IPA_CS_OFFLOAD_UL = 0x1, /* Before IPA v4.5 (TX) */
IPA_CS_OFFLOAD_DL = 0x2, /* Before IPA v4.5 (RX) */
+ IPA_CS_OFFLOAD_INLINE = 0x1, /* IPA v4.5 (TX and RX) */
};
/* Valid only for TX (IPA consumer) endpoints */
int ret;
ret = platform_get_irq_byname(smp2p->ipa->pdev, name);
- if (ret <= 0) {
- dev_err(dev, "DT error %d getting \"%s\" IRQ property\n",
- ret, name);
+ if (ret <= 0)
return ret ? : -EINVAL;
- }
irq = ret;
ret = request_threaded_irq(irq, NULL, handler, 0, name, smp2p);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (C) 2021 Linaro Ltd. */
+
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/sysfs.h>
+
+#include "ipa.h"
+#include "ipa_version.h"
+#include "ipa_sysfs.h"
+
+static const char *ipa_version_string(struct ipa *ipa)
+{
+ switch (ipa->version) {
+ case IPA_VERSION_3_0:
+ return "3.0";
+ case IPA_VERSION_3_1:
+ return "3.1";
+ case IPA_VERSION_3_5:
+ return "3.5";
+ case IPA_VERSION_3_5_1:
+ return "3.5.1";
+ case IPA_VERSION_4_0:
+ return "4.0";
+ case IPA_VERSION_4_1:
+ return "4.1";
+ case IPA_VERSION_4_2:
+ return "4.2";
+ case IPA_VERSION_4_5:
+ return "4.5";
+ case IPA_VERSION_4_7:
+ return "4.7";
+ case IPA_VERSION_4_9:
+ return "4.9";
+ case IPA_VERSION_4_11:
+ return "4.11";
+ default:
+ return "0.0"; /* Won't happen (checked at probe time) */
+ }
+}
+
+static ssize_t
+version_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct ipa *ipa = dev_get_drvdata(dev);
+
+ return scnprintf(buf, PAGE_SIZE, "%s\n", ipa_version_string(ipa));
+}
+
+static DEVICE_ATTR_RO(version);
+
+static struct attribute *ipa_attrs[] = {
+ &dev_attr_version.attr,
+ NULL
+};
+
+const struct attribute_group ipa_attribute_group = {
+ .attrs = ipa_attrs,
+};
+
+static const char *ipa_offload_string(struct ipa *ipa)
+{
+ return ipa->version < IPA_VERSION_4_5 ? "MAPv4" : "MAPv5";
+}
+
+static ssize_t rx_offload_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct ipa *ipa = dev_get_drvdata(dev);
+
+ return scnprintf(buf, PAGE_SIZE, "%s\n", ipa_offload_string(ipa));
+}
+
+static DEVICE_ATTR_RO(rx_offload);
+
+static ssize_t tx_offload_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct ipa *ipa = dev_get_drvdata(dev);
+
+ return scnprintf(buf, PAGE_SIZE, "%s\n", ipa_offload_string(ipa));
+}
+
+static DEVICE_ATTR_RO(tx_offload);
+
+static struct attribute *ipa_feature_attrs[] = {
+ &dev_attr_rx_offload.attr,
+ &dev_attr_tx_offload.attr,
+ NULL
+};
+
+const struct attribute_group ipa_feature_attribute_group = {
+ .name = "feature",
+ .attrs = ipa_feature_attrs,
+};
+
+static ssize_t
+ipa_endpoint_id_show(struct ipa *ipa, char *buf, enum ipa_endpoint_name name)
+{
+ u32 endpoint_id = ipa->name_map[name]->endpoint_id;
+
+ return scnprintf(buf, PAGE_SIZE, "%u\n", endpoint_id);
+}
+
+static ssize_t rx_endpoint_id_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct ipa *ipa = dev_get_drvdata(dev);
+
+ return ipa_endpoint_id_show(ipa, buf, IPA_ENDPOINT_AP_MODEM_RX);
+}
+
+static DEVICE_ATTR_RO(rx_endpoint_id);
+
+static ssize_t tx_endpoint_id_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct ipa *ipa = dev_get_drvdata(dev);
+
+ return ipa_endpoint_id_show(ipa, buf, IPA_ENDPOINT_AP_MODEM_TX);
+}
+
+static DEVICE_ATTR_RO(tx_endpoint_id);
+
+static struct attribute *ipa_modem_attrs[] = {
+ &dev_attr_rx_endpoint_id.attr,
+ &dev_attr_tx_endpoint_id.attr,
+ NULL
+};
+
+const struct attribute_group ipa_modem_attribute_group = {
+ .name = "modem",
+ .attrs = ipa_modem_attrs,
+};
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2019-2021 Linaro Ltd.
+ */
+#ifndef _IPA_SYSFS_H_
+#define _IPA_SYSFS_H_
+
+struct attribute_group;
+
+extern const struct attribute_group ipa_attribute_group;
+extern const struct attribute_group ipa_feature_attribute_group;
+extern const struct attribute_group ipa_modem_attribute_group;
+
+#endif /* _IPA_SYSFS_H_ */
}
static bool
-ipa_table_valid_one(struct ipa *ipa, bool route, bool ipv6, bool hashed)
+ipa_table_valid_one(struct ipa *ipa, enum ipa_mem_id mem_id, bool route)
{
+ const struct ipa_mem *mem = ipa_mem_find(ipa, mem_id);
struct device *dev = &ipa->pdev->dev;
- const struct ipa_mem *mem;
u32 size;
- if (route) {
- if (ipv6)
- mem = hashed ? &ipa->mem[IPA_MEM_V6_ROUTE_HASHED]
- : &ipa->mem[IPA_MEM_V6_ROUTE];
- else
- mem = hashed ? &ipa->mem[IPA_MEM_V4_ROUTE_HASHED]
- : &ipa->mem[IPA_MEM_V4_ROUTE];
+ if (route)
size = IPA_ROUTE_COUNT_MAX * sizeof(__le64);
- } else {
- if (ipv6)
- mem = hashed ? &ipa->mem[IPA_MEM_V6_FILTER_HASHED]
- : &ipa->mem[IPA_MEM_V6_FILTER];
- else
- mem = hashed ? &ipa->mem[IPA_MEM_V4_FILTER_HASHED]
- : &ipa->mem[IPA_MEM_V4_FILTER];
+ else
size = (1 + IPA_FILTER_COUNT_MAX) * sizeof(__le64);
- }
if (!ipa_cmd_table_valid(ipa, mem, route, ipv6, hashed))
return false;
if (hashed && !mem->size)
return true;
- dev_err(dev, "IPv%c %s%s table region size 0x%02x, expected 0x%02x\n",
- ipv6 ? '6' : '4', hashed ? "hashed " : "",
- route ? "route" : "filter", mem->size, size);
+ dev_err(dev, "%s table region %u size 0x%02x, expected 0x%02x\n",
+ route ? "route" : "filter", mem_id, mem->size, size);
return false;
}
/* Verify the filter and route table memory regions are the expected size */
bool ipa_table_valid(struct ipa *ipa)
{
- bool valid = true;
+ bool valid;
- valid = valid && ipa_table_valid_one(ipa, false, false, false);
- valid = valid && ipa_table_valid_one(ipa, false, false, true);
- valid = valid && ipa_table_valid_one(ipa, false, true, false);
- valid = valid && ipa_table_valid_one(ipa, false, true, true);
- valid = valid && ipa_table_valid_one(ipa, true, false, false);
- valid = valid && ipa_table_valid_one(ipa, true, false, true);
- valid = valid && ipa_table_valid_one(ipa, true, true, false);
- valid = valid && ipa_table_valid_one(ipa, true, true, true);
+ valid = ipa_table_valid_one(IPA_MEM_V4_FILTER, false);
+ valid = valid && ipa_table_valid_one(IPA_MEM_V4_FILTER_HASHED, false);
+ valid = valid && ipa_table_valid_one(IPA_MEM_V6_FILTER, false);
+ valid = valid && ipa_table_valid_one(IPA_MEM_V6_FILTER_HASHED, false);
+ valid = valid && ipa_table_valid_one(IPA_MEM_V4_ROUTE, true);
+ valid = valid && ipa_table_valid_one(IPA_MEM_V4_ROUTE_HASHED, true);
+ valid = valid && ipa_table_valid_one(IPA_MEM_V6_ROUTE, true);
+ valid = valid && ipa_table_valid_one(IPA_MEM_V6_ROUTE_HASHED, true);
return valid;
}
}
static void ipa_table_reset_add(struct gsi_trans *trans, bool filter,
- u16 first, u16 count, const struct ipa_mem *mem)
+ u16 first, u16 count, enum ipa_mem_id mem_id)
{
struct ipa *ipa = container_of(trans->gsi, struct ipa, gsi);
+ const struct ipa_mem *mem = ipa_mem_find(ipa, mem_id);
dma_addr_t addr;
u32 offset;
u16 size;
- /* Nothing to do if the table memory regions is empty */
+ /* Nothing to do if the table memory region is empty */
if (!mem->size)
return;
* for the IPv4 and IPv6 non-hashed and hashed filter tables.
*/
static int
-ipa_filter_reset_table(struct ipa *ipa, const struct ipa_mem *mem, bool modem)
+ipa_filter_reset_table(struct ipa *ipa, enum ipa_mem_id mem_id, bool modem)
{
u32 ep_mask = ipa->filter_map;
u32 count = hweight32(ep_mask);
struct gsi_trans *trans;
enum gsi_ee_id ee_id;
- if (!mem->size)
- return 0;
-
trans = ipa_cmd_trans_alloc(ipa, count);
if (!trans) {
dev_err(&ipa->pdev->dev,
if (endpoint->ee_id != ee_id)
continue;
- ipa_table_reset_add(trans, true, endpoint_id, 1, mem);
+ ipa_table_reset_add(trans, true, endpoint_id, 1, mem_id);
}
gsi_trans_commit_wait(trans);
{
int ret;
- ret = ipa_filter_reset_table(ipa, &ipa->mem[IPA_MEM_V4_FILTER], modem);
+ ret = ipa_filter_reset_table(ipa, IPA_MEM_V4_FILTER, modem);
if (ret)
return ret;
- ret = ipa_filter_reset_table(ipa, &ipa->mem[IPA_MEM_V4_FILTER_HASHED],
- modem);
+ ret = ipa_filter_reset_table(ipa, IPA_MEM_V4_FILTER_HASHED, modem);
if (ret)
return ret;
- ret = ipa_filter_reset_table(ipa, &ipa->mem[IPA_MEM_V6_FILTER], modem);
+ ret = ipa_filter_reset_table(ipa, IPA_MEM_V6_FILTER, modem);
if (ret)
return ret;
- ret = ipa_filter_reset_table(ipa, &ipa->mem[IPA_MEM_V6_FILTER_HASHED],
- modem);
+ ret = ipa_filter_reset_table(ipa, IPA_MEM_V6_FILTER_HASHED, modem);
return ret;
}
count = IPA_ROUTE_AP_COUNT;
}
+ ipa_table_reset_add(trans, false, first, count, IPA_MEM_V4_ROUTE);
ipa_table_reset_add(trans, false, first, count,
- &ipa->mem[IPA_MEM_V4_ROUTE]);
- ipa_table_reset_add(trans, false, first, count,
- &ipa->mem[IPA_MEM_V4_ROUTE_HASHED]);
+ IPA_MEM_V4_ROUTE_HASHED);
+ ipa_table_reset_add(trans, false, first, count, IPA_MEM_V6_ROUTE);
ipa_table_reset_add(trans, false, first, count,
- &ipa->mem[IPA_MEM_V6_ROUTE]);
- ipa_table_reset_add(trans, false, first, count,
- &ipa->mem[IPA_MEM_V6_ROUTE_HASHED]);
+ IPA_MEM_V6_ROUTE_HASHED);
gsi_trans_commit_wait(trans);
static void ipa_table_init_add(struct gsi_trans *trans, bool filter,
enum ipa_cmd_opcode opcode,
- const struct ipa_mem *mem,
- const struct ipa_mem *hash_mem)
+ enum ipa_mem_id mem_id,
+ enum ipa_mem_id hash_mem_id)
{
struct ipa *ipa = container_of(trans->gsi, struct ipa, gsi);
+ const struct ipa_mem *hash_mem = ipa_mem_find(ipa, hash_mem_id);
+ const struct ipa_mem *mem = ipa_mem_find(ipa, mem_id);
dma_addr_t hash_addr;
dma_addr_t addr;
u16 hash_count;
}
ipa_table_init_add(trans, false, IPA_CMD_IP_V4_ROUTING_INIT,
- &ipa->mem[IPA_MEM_V4_ROUTE],
- &ipa->mem[IPA_MEM_V4_ROUTE_HASHED]);
+ IPA_MEM_V4_ROUTE, IPA_MEM_V4_ROUTE_HASHED);
ipa_table_init_add(trans, false, IPA_CMD_IP_V6_ROUTING_INIT,
- &ipa->mem[IPA_MEM_V6_ROUTE],
- &ipa->mem[IPA_MEM_V6_ROUTE_HASHED]);
+ IPA_MEM_V6_ROUTE, IPA_MEM_V6_ROUTE_HASHED);
ipa_table_init_add(trans, true, IPA_CMD_IP_V4_FILTER_INIT,
- &ipa->mem[IPA_MEM_V4_FILTER],
- &ipa->mem[IPA_MEM_V4_FILTER_HASHED]);
+ IPA_MEM_V4_FILTER, IPA_MEM_V4_FILTER_HASHED);
ipa_table_init_add(trans, true, IPA_CMD_IP_V6_FILTER_INIT,
- &ipa->mem[IPA_MEM_V6_FILTER],
- &ipa->mem[IPA_MEM_V6_FILTER_HASHED]);
+ IPA_MEM_V6_FILTER, IPA_MEM_V6_FILTER_HASHED);
gsi_trans_commit_wait(trans);
static struct ipa_uc_mem_area *ipa_uc_shared(struct ipa *ipa)
{
- u32 offset = ipa->mem_offset + ipa->mem[IPA_MEM_UC_SHARED].offset;
+ const struct ipa_mem *mem = ipa_mem_find(ipa, IPA_MEM_UC_SHARED);
+ u32 offset = ipa->mem_offset + mem->offset;
return ipa->mem_virt + offset;
}
* @IPA_VERSION_4_11: IPA version 4.11/GSI version 2.11 (2.1.1)
*
* Defines the version of IPA (and GSI) hardware present on the platform.
+ * Please update ipa_version_valid() and ipa_version_string() whenever a
+ * new version is added.
*/
enum ipa_version {
IPA_VERSION_3_0,
unregister_netdevice_many(&list_kill);
break;
case NETDEV_PRE_TYPE_CHANGE:
- /* Forbid underlaying device to change its type. */
+ /* Forbid underlying device to change its type. */
return NOTIFY_BAD;
case NETDEV_NOTIFY_PEERS:
reflects whether the mdio_bus/mdio_device code is built as a
loadable module or built-in.
+config FWNODE_MDIO
+ def_tristate PHYLIB
+ depends on (ACPI || OF) || COMPILE_TEST
+ select FIXED_PHY
+ help
+ FWNODE MDIO bus (Ethernet PHY) accessors
+
config OF_MDIO
def_tristate PHYLIB
depends on OF
help
OpenFirmware MDIO bus (Ethernet PHY) accessors
+config ACPI_MDIO
+ def_tristate PHYLIB
+ depends on ACPI
+ depends on PHYLIB
+ help
+ ACPI MDIO bus (Ethernet PHY) accessors
+
if MDIO_BUS
config MDIO_DEVRES
# SPDX-License-Identifier: GPL-2.0
# Makefile for Linux MDIO bus drivers
-obj-$(CONFIG_OF_MDIO) += of_mdio.o
+obj-$(CONFIG_ACPI_MDIO) += acpi_mdio.o
+obj-$(CONFIG_FWNODE_MDIO) += fwnode_mdio.o
+obj-$(CONFIG_OF_MDIO) += of_mdio.o
obj-$(CONFIG_MDIO_ASPEED) += mdio-aspeed.o
obj-$(CONFIG_MDIO_BCM_IPROC) += mdio-bcm-iproc.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * ACPI helpers for the MDIO (Ethernet PHY) API
+ *
+ * This file provides helper functions for extracting PHY device information
+ * out of the ACPI ASL and using it to populate an mii_bus.
+ */
+
+#include <linux/acpi.h>
+#include <linux/acpi_mdio.h>
+#include <linux/bits.h>
+#include <linux/dev_printk.h>
+#include <linux/fwnode_mdio.h>
+#include <linux/module.h>
+#include <linux/types.h>
+
+MODULE_AUTHOR("Calvin Johnson <calvin.johnson@oss.nxp.com>");
+MODULE_LICENSE("GPL");
+
+/**
+ * acpi_mdiobus_register - Register mii_bus and create PHYs from the ACPI ASL.
+ * @mdio: pointer to mii_bus structure
+ * @fwnode: pointer to fwnode of MDIO bus. This fwnode is expected to represent
+ * an ACPI device object corresponding to the MDIO bus and its children are
+ * expected to correspond to the PHY devices on that bus.
+ *
+ * This function registers the mii_bus structure and registers a phy_device
+ * for each child node of @fwnode.
+ */
+int acpi_mdiobus_register(struct mii_bus *mdio, struct fwnode_handle *fwnode)
+{
+ struct fwnode_handle *child;
+ u32 addr;
+ int ret;
+
+ /* Mask out all PHYs from auto probing. */
+ mdio->phy_mask = GENMASK(31, 0);
+ ret = mdiobus_register(mdio);
+ if (ret)
+ return ret;
+
+ ACPI_COMPANION_SET(&mdio->dev, to_acpi_device_node(fwnode));
+
+ /* Loop over the child nodes and register a phy_device for each PHY */
+ fwnode_for_each_child_node(fwnode, child) {
+ ret = acpi_get_local_address(ACPI_HANDLE_FWNODE(child), &addr);
+ if (ret || addr >= PHY_MAX_ADDR)
+ continue;
+
+ ret = fwnode_mdiobus_register_phy(mdio, child, addr);
+ if (ret == -ENODEV)
+ dev_err(&mdio->dev,
+ "MDIO device at address %d is missing.\n",
+ addr);
+ }
+ return 0;
+}
+EXPORT_SYMBOL(acpi_mdiobus_register);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * fwnode helpers for the MDIO (Ethernet PHY) API
+ *
+ * This file provides helper functions for extracting PHY device information
+ * out of the fwnode and using it to populate an mii_bus.
+ */
+
+#include <linux/acpi.h>
+#include <linux/fwnode_mdio.h>
+#include <linux/of.h>
+#include <linux/phy.h>
+
+MODULE_AUTHOR("Calvin Johnson <calvin.johnson@oss.nxp.com>");
+MODULE_LICENSE("GPL");
+
+static struct mii_timestamper *
+fwnode_find_mii_timestamper(struct fwnode_handle *fwnode)
+{
+ struct of_phandle_args arg;
+ int err;
+
+ if (is_acpi_node(fwnode))
+ return NULL;
+
+ err = of_parse_phandle_with_fixed_args(to_of_node(fwnode),
+ "timestamper", 1, 0, &arg);
+ if (err == -ENOENT)
+ return NULL;
+ else if (err)
+ return ERR_PTR(err);
+
+ if (arg.args_count != 1)
+ return ERR_PTR(-EINVAL);
+
+ return register_mii_timestamper(arg.np, arg.args[0]);
+}
+
+int fwnode_mdiobus_phy_device_register(struct mii_bus *mdio,
+ struct phy_device *phy,
+ struct fwnode_handle *child, u32 addr)
+{
+ int rc;
+
+ rc = fwnode_irq_get(child, 0);
+ if (rc == -EPROBE_DEFER)
+ return rc;
+
+ if (rc > 0) {
+ phy->irq = rc;
+ mdio->irq[addr] = rc;
+ } else {
+ phy->irq = mdio->irq[addr];
+ }
+
+ if (fwnode_property_read_bool(child, "broken-turn-around"))
+ mdio->phy_ignore_ta_mask |= 1 << addr;
+
+ fwnode_property_read_u32(child, "reset-assert-us",
+ &phy->mdio.reset_assert_delay);
+ fwnode_property_read_u32(child, "reset-deassert-us",
+ &phy->mdio.reset_deassert_delay);
+
+ /* Associate the fwnode with the device structure so it
+ * can be looked up later
+ */
+ fwnode_handle_get(child);
+ device_set_node(&phy->mdio.dev, child);
+
+ /* All data is now stored in the phy struct;
+ * register it
+ */
+ rc = phy_device_register(phy);
+ if (rc) {
+ fwnode_handle_put(child);
+ return rc;
+ }
+
+ dev_dbg(&mdio->dev, "registered phy %p fwnode at address %i\n",
+ child, addr);
+ return 0;
+}
+EXPORT_SYMBOL(fwnode_mdiobus_phy_device_register);
+
+int fwnode_mdiobus_register_phy(struct mii_bus *bus,
+ struct fwnode_handle *child, u32 addr)
+{
+ struct mii_timestamper *mii_ts = NULL;
+ struct phy_device *phy;
+ bool is_c45 = false;
+ u32 phy_id;
+ int rc;
+
+ mii_ts = fwnode_find_mii_timestamper(child);
+ if (IS_ERR(mii_ts))
+ return PTR_ERR(mii_ts);
+
+ rc = fwnode_property_match_string(child, "compatible",
+ "ethernet-phy-ieee802.3-c45");
+ if (rc >= 0)
+ is_c45 = true;
+
+ if (is_c45 || fwnode_get_phy_id(child, &phy_id))
+ phy = get_phy_device(bus, addr, is_c45);
+ else
+ phy = phy_device_create(bus, addr, phy_id, 0, NULL);
+ if (IS_ERR(phy)) {
+ unregister_mii_timestamper(mii_ts);
+ return PTR_ERR(phy);
+ }
+
+ if (is_acpi_node(child)) {
+ phy->irq = bus->irq[addr];
+
+ /* Associate the fwnode with the device structure so it
+ * can be looked up later.
+ */
+ phy->mdio.dev.fwnode = child;
+
+ /* All data is now stored in the phy struct, so register it */
+ rc = phy_device_register(phy);
+ if (rc) {
+ phy_device_free(phy);
+ fwnode_handle_put(phy->mdio.dev.fwnode);
+ return rc;
+ }
+ } else if (is_of_node(child)) {
+ rc = fwnode_mdiobus_phy_device_register(bus, phy, child, addr);
+ if (rc) {
+ unregister_mii_timestamper(mii_ts);
+ phy_device_free(phy);
+ return rc;
+ }
+ }
+
+ /* phy->mii_ts may already be defined by the PHY driver. A
+ * mii_timestamper probed via the device tree will still have
+ * precedence.
+ */
+ if (mii_ts)
+ phy->mii_ts = mii_ts;
+ return 0;
+}
+EXPORT_SYMBOL(fwnode_mdiobus_register_phy);
return;
}
- /* The MDIO clock is the reference clock (typicaly 250Mhz) divided by
+ /* The MDIO clock is the reference clock (typically 250Mhz) divided by
* 2 x (MDIO_CLK_DIV + 1)
*/
reg = unimac_mdio_readl(priv, MDIO_CFG);
#include <linux/delay.h>
#include <linux/kernel.h>
-#include <linux/mfd/syscon.h>
#include <linux/module.h>
#include <linux/of_mdio.h>
-#include <linux/phy.h>
+#include <linux/of_address.h>
#include <linux/platform_device.h>
#include <linux/regmap.h>
/* MII address register definitions */
-#define MII_ADDR_REG_ADDR 0x10
-#define MII_BUSY BIT(0)
-#define MII_WRITE BIT(1)
-#define MII_CLKRANGE_60_100M (0 << 2)
-#define MII_CLKRANGE_100_150M (1 << 2)
-#define MII_CLKRANGE_20_35M (2 << 2)
-#define MII_CLKRANGE_35_60M (3 << 2)
-#define MII_CLKRANGE_150_250M (4 << 2)
-#define MII_CLKRANGE_250_300M (5 << 2)
+#define MII_ADDR_REG_ADDR 0x10
+#define MII_BUSY BIT(0)
+#define MII_WRITE BIT(1)
+#define MII_CLKRANGE(x) ((x) << 2)
+#define MII_CLKRANGE_60_100M MII_CLKRANGE(0)
+#define MII_CLKRANGE_100_150M MII_CLKRANGE(1)
+#define MII_CLKRANGE_20_35M MII_CLKRANGE(2)
+#define MII_CLKRANGE_35_60M MII_CLKRANGE(3)
+#define MII_CLKRANGE_150_250M MII_CLKRANGE(4)
+#define MII_CLKRANGE_250_300M MII_CLKRANGE(5)
#define MII_CLKRANGE_MASK GENMASK(4, 2)
#define MII_REG_SHIFT 6
#define MII_REG_MASK GENMASK(10, 6)
#define MII_ADDR_SHIFT 11
#define MII_ADDR_MASK GENMASK(15, 11)
-#define MII_DATA_REG_ADDR 0x14
+#define MII_DATA_REG_ADDR 0x14
-#define MII_MDIO_DELAY_USEC (1000)
-#define MII_MDIO_RETRY_MSEC (10)
+#define MII_MDIO_DELAY_USEC (1000)
+#define MII_MDIO_RETRY_MSEC (10)
struct ipq8064_mdio {
struct regmap *base; /* NSS_GMAC0_BASE */
((reg_offset << MII_REG_SHIFT) & MII_REG_MASK);
regmap_write(priv->base, MII_ADDR_REG_ADDR, miiaddr);
- usleep_range(8, 10);
+ usleep_range(10, 13);
err = ipq8064_mdio_wait_busy(priv);
if (err)
((reg_offset << MII_REG_SHIFT) & MII_REG_MASK);
regmap_write(priv->base, MII_ADDR_REG_ADDR, miiaddr);
- usleep_range(8, 10);
+
+ /* For the specific reg 31 extra time is needed or the next
+ * read will produce garbage data.
+ */
+ if (reg_offset == 31)
+ usleep_range(30, 43);
+ else
+ usleep_range(10, 13);
return ipq8064_mdio_wait_busy(priv);
}
+static const struct regmap_config ipq8064_mdio_regmap_config = {
+ .reg_bits = 32,
+ .reg_stride = 4,
+ .val_bits = 32,
+ .can_multi_write = false,
+ /* the mdio lock is used by any user of this mdio driver */
+ .disable_locking = true,
+
+ .cache_type = REGCACHE_NONE,
+};
+
static int
ipq8064_mdio_probe(struct platform_device *pdev)
{
struct device_node *np = pdev->dev.of_node;
struct ipq8064_mdio *priv;
+ struct resource res;
struct mii_bus *bus;
+ void __iomem *base;
int ret;
+ if (of_address_to_resource(np, 0, &res))
+ return -ENOMEM;
+
+ base = ioremap(res.start, resource_size(&res));
+ if (!base)
+ return -ENOMEM;
+
bus = devm_mdiobus_alloc_size(&pdev->dev, sizeof(*priv));
if (!bus)
return -ENOMEM;
bus->parent = &pdev->dev;
priv = bus->priv;
- priv->base = device_node_to_regmap(np);
- if (IS_ERR(priv->base)) {
- if (priv->base == ERR_PTR(-EPROBE_DEFER))
- return -EPROBE_DEFER;
-
- dev_err(&pdev->dev, "error getting device regmap, error=%pe\n",
- priv->base);
+ priv->base = devm_regmap_init_mmio(&pdev->dev, base,
+ &ipq8064_mdio_regmap_config);
+ if (IS_ERR(priv->base))
return PTR_ERR(priv->base);
- }
ret = of_mdiobus_register(bus, np);
if (ret)
struct mscc_miim_dev *dev;
int ret;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!res)
- return -ENODEV;
-
bus = devm_mdiobus_alloc_size(&pdev->dev, sizeof(*dev));
if (!bus)
return -ENOMEM;
bus->parent = &pdev->dev;
dev = bus->priv;
- dev->regs = devm_ioremap_resource(&pdev->dev, res);
+ dev->regs = devm_platform_get_and_ioremap_resource(pdev, 0, NULL);
if (IS_ERR(dev->regs)) {
dev_err(&pdev->dev, "Unable to map MIIM registers\n");
return PTR_ERR(dev->regs);
writel(val, md->base + MDIO_SCAN_CTRL_OFFSET);
if (md->core_clk) {
- /* use rate adjust regs to derrive the mdio's operating
+ /* use rate adjust regs to derive the mdio's operating
* frequency from the specified core clock
*/
divisor = clk_get_rate(md->core_clk) / MDIO_OPERATING_FREQUENCY;
return -ENOMEM;
md->dev = &pdev->dev;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ md->base = devm_platform_get_and_ioremap_resource(pdev, 0, &res);
+ if (IS_ERR(md->base))
+ return PTR_ERR(md->base);
if (res->start & 0xfff) {
/* For backward compatibility in case the
* base address is specified with an offset.
res->start &= ~0xfff;
res->end = res->start + MDIO_REG_ADDR_SPACE_SIZE - 1;
}
- md->base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(md->base))
- return PTR_ERR(md->base);
md->mii_bus = devm_mdiobus_alloc(&pdev->dev);
if (!md->mii_bus) {
/* Poll on the digital lock instead of the usual analog lock
* This is done because bit 31 is unreliable on some SoC. Bit
- * 31 may indicate that the PLL is not lock eventhough the clock
+ * 31 may indicate that the PLL is not lock even though the clock
* is actually running
*/
return readl_poll_timeout(pll->base + ETH_PLL_CTL0, val,
return 0;
fail_register:
- mdiobus_free(bus->mii_bus);
smi_en.u64 = 0;
oct_mdio_writeq(smi_en.u64, bus->register_base + SMI_EN);
return err;
bus = platform_get_drvdata(pdev);
mdiobus_unregister(bus->mii_bus);
- mdiobus_free(bus->mii_bus);
smi_en.u64 = 0;
oct_mdio_writeq(smi_en.u64, bus->register_base + SMI_EN);
return 0;
continue;
mdiobus_unregister(bus->mii_bus);
- mdiobus_free(bus->mii_bus);
oct_mdio_writeq(0, bus->register_base + SMI_EN);
}
pci_release_regions(pdev);
#include <linux/device.h>
#include <linux/err.h>
+#include <linux/fwnode_mdio.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/netdevice.h>
* ethernet-phy-idAAAA.BBBB */
static int of_get_phy_id(struct device_node *device, u32 *phy_id)
{
- struct property *prop;
- const char *cp;
- unsigned int upper, lower;
-
- of_property_for_each_string(device, "compatible", prop, cp) {
- if (sscanf(cp, "ethernet-phy-id%4x.%4x", &upper, &lower) == 2) {
- *phy_id = ((upper & 0xFFFF) << 16) | (lower & 0xFFFF);
- return 0;
- }
- }
- return -EINVAL;
-}
-
-static struct mii_timestamper *of_find_mii_timestamper(struct device_node *node)
-{
- struct of_phandle_args arg;
- int err;
-
- err = of_parse_phandle_with_fixed_args(node, "timestamper", 1, 0, &arg);
-
- if (err == -ENOENT)
- return NULL;
- else if (err)
- return ERR_PTR(err);
-
- if (arg.args_count != 1)
- return ERR_PTR(-EINVAL);
-
- return register_mii_timestamper(arg.np, arg.args[0]);
+ return fwnode_get_phy_id(of_fwnode_handle(device), phy_id);
}
int of_mdiobus_phy_device_register(struct mii_bus *mdio, struct phy_device *phy,
- struct device_node *child, u32 addr)
+ struct device_node *child, u32 addr)
{
- int rc;
-
- rc = of_irq_get(child, 0);
- if (rc == -EPROBE_DEFER)
- return rc;
-
- if (rc > 0) {
- phy->irq = rc;
- mdio->irq[addr] = rc;
- } else {
- phy->irq = mdio->irq[addr];
- }
-
- if (of_property_read_bool(child, "broken-turn-around"))
- mdio->phy_ignore_ta_mask |= 1 << addr;
-
- of_property_read_u32(child, "reset-assert-us",
- &phy->mdio.reset_assert_delay);
- of_property_read_u32(child, "reset-deassert-us",
- &phy->mdio.reset_deassert_delay);
-
- /* Associate the OF node with the device structure so it
- * can be looked up later */
- of_node_get(child);
- phy->mdio.dev.of_node = child;
- phy->mdio.dev.fwnode = of_fwnode_handle(child);
-
- /* All data is now stored in the phy struct;
- * register it */
- rc = phy_device_register(phy);
- if (rc) {
- of_node_put(child);
- return rc;
- }
-
- dev_dbg(&mdio->dev, "registered phy %pOFn at address %i\n",
- child, addr);
- return 0;
+ return fwnode_mdiobus_phy_device_register(mdio, phy,
+ of_fwnode_handle(child),
+ addr);
}
EXPORT_SYMBOL(of_mdiobus_phy_device_register);
static int of_mdiobus_register_phy(struct mii_bus *mdio,
struct device_node *child, u32 addr)
{
- struct mii_timestamper *mii_ts;
- struct phy_device *phy;
- bool is_c45;
- int rc;
- u32 phy_id;
-
- mii_ts = of_find_mii_timestamper(child);
- if (IS_ERR(mii_ts))
- return PTR_ERR(mii_ts);
-
- is_c45 = of_device_is_compatible(child,
- "ethernet-phy-ieee802.3-c45");
-
- if (!is_c45 && !of_get_phy_id(child, &phy_id))
- phy = phy_device_create(mdio, addr, phy_id, 0, NULL);
- else
- phy = get_phy_device(mdio, addr, is_c45);
- if (IS_ERR(phy)) {
- if (mii_ts)
- unregister_mii_timestamper(mii_ts);
- return PTR_ERR(phy);
- }
-
- rc = of_mdiobus_phy_device_register(mdio, phy, child, addr);
- if (rc) {
- if (mii_ts)
- unregister_mii_timestamper(mii_ts);
- phy_device_free(phy);
- return rc;
- }
-
- /* phy->mii_ts may already be defined by the PHY driver. A
- * mii_timestamper probed via the device tree will still have
- * precedence.
- */
- if (mii_ts)
- phy->mii_ts = mii_ts;
-
- return 0;
+ return fwnode_mdiobus_register_phy(mdio, of_fwnode_handle(child), addr);
}
static int of_mdiobus_register_device(struct mii_bus *mdio,
struct device_node *child, u32 addr)
{
+ struct fwnode_handle *fwnode = of_fwnode_handle(child);
struct mdio_device *mdiodev;
int rc;
/* Associate the OF node with the device structure so it
* can be looked up later.
*/
- of_node_get(child);
- mdiodev->dev.of_node = child;
- mdiodev->dev.fwnode = of_fwnode_handle(child);
+ fwnode_handle_get(fwnode);
+ device_set_node(&mdiodev->dev, fwnode);
/* All data is now stored in the mdiodev struct; register it. */
rc = mdio_device_register(mdiodev);
* the device tree are populated after the bus has been registered */
mdio->phy_mask = ~0;
- mdio->dev.of_node = np;
- mdio->dev.fwnode = of_fwnode_handle(np);
+ device_set_node(&mdio->dev, of_fwnode_handle(np));
/* Get bus level PHY reset GPIO details */
mdio->reset_delay_us = DEFAULT_GPIO_RESET_DELAY;
*/
struct mdio_device *of_mdio_find_device(struct device_node *np)
{
- struct device *d;
-
- if (!np)
- return NULL;
-
- d = bus_find_device_by_of_node(&mdio_bus_type, np);
- if (!d)
- return NULL;
-
- return to_mdio_device(d);
+ return fwnode_mdio_find_device(of_fwnode_handle(np));
}
EXPORT_SYMBOL(of_mdio_find_device);
*/
struct phy_device *of_phy_find_device(struct device_node *phy_np)
{
- struct mdio_device *mdiodev;
-
- mdiodev = of_mdio_find_device(phy_np);
- if (!mdiodev)
- return NULL;
-
- if (mdiodev->flags & MDIO_DEVICE_FLAG_PHY)
- return to_phy_device(&mdiodev->dev);
-
- put_device(&mdiodev->dev);
-
- return NULL;
+ return fwnode_phy_find_device(of_fwnode_handle(phy_np));
}
EXPORT_SYMBOL(of_phy_find_device);
* of_phy_is_fixed_link() and of_phy_register_fixed_link() must
* support two DT bindings:
* - the old DT binding, where 'fixed-link' was a property with 5
- * cells encoding various informations about the fixed PHY
+ * cells encoding various information about the fixed PHY
* - the new DT binding, where 'fixed-link' is a sub-node of the
* Ethernet device.
*/
#include <linux/netdevice.h>
#include <linux/skbuff.h>
#include <linux/u64_stats_sync.h>
+#include <linux/wwan.h>
#include "mhi.h"
#define MHI_NET_MAX_MTU 0xffff
#define MHI_NET_DEFAULT_MTU 0x4000
+/* When set to false, the default netdev (link 0) is not created, and it's up
+ * to user to create the link (via wwan rtnetlink).
+ */
+static bool create_default_iface = true;
+module_param(create_default_iface, bool, 0);
+
struct mhi_device_info {
const char *netname;
const struct mhi_net_proto *proto;
static int mhi_ndo_open(struct net_device *ndev)
{
- struct mhi_net_dev *mhi_netdev = netdev_priv(ndev);
+ struct mhi_net_dev *mhi_netdev = wwan_netdev_drvpriv(ndev);
/* Feed the rx buffer pool */
schedule_delayed_work(&mhi_netdev->rx_refill, 0);
static int mhi_ndo_stop(struct net_device *ndev)
{
- struct mhi_net_dev *mhi_netdev = netdev_priv(ndev);
+ struct mhi_net_dev *mhi_netdev = wwan_netdev_drvpriv(ndev);
netif_stop_queue(ndev);
netif_carrier_off(ndev);
return 0;
}
-static int mhi_ndo_xmit(struct sk_buff *skb, struct net_device *ndev)
+static netdev_tx_t mhi_ndo_xmit(struct sk_buff *skb, struct net_device *ndev)
{
- struct mhi_net_dev *mhi_netdev = netdev_priv(ndev);
+ struct mhi_net_dev *mhi_netdev = wwan_netdev_drvpriv(ndev);
const struct mhi_net_proto *proto = mhi_netdev->proto;
struct mhi_device *mdev = mhi_netdev->mdev;
int err;
static void mhi_ndo_get_stats64(struct net_device *ndev,
struct rtnl_link_stats64 *stats)
{
- struct mhi_net_dev *mhi_netdev = netdev_priv(ndev);
+ struct mhi_net_dev *mhi_netdev = wwan_netdev_drvpriv(ndev);
unsigned int start;
do {
schedule_delayed_work(&mhi_netdev->rx_refill, HZ / 2);
}
-static struct device_type wwan_type = {
- .name = "wwan",
-};
-
-static int mhi_net_probe(struct mhi_device *mhi_dev,
- const struct mhi_device_id *id)
+static int mhi_net_newlink(void *ctxt, struct net_device *ndev, u32 if_id,
+ struct netlink_ext_ack *extack)
{
- const struct mhi_device_info *info = (struct mhi_device_info *)id->driver_data;
- struct device *dev = &mhi_dev->dev;
+ const struct mhi_device_info *info;
+ struct mhi_device *mhi_dev = ctxt;
struct mhi_net_dev *mhi_netdev;
- struct net_device *ndev;
int err;
- ndev = alloc_netdev(sizeof(*mhi_netdev), info->netname,
- NET_NAME_PREDICTABLE, mhi_net_setup);
- if (!ndev)
- return -ENOMEM;
+ info = (struct mhi_device_info *)mhi_dev->id->driver_data;
+
+ /* For now we only support one link (link context 0), driver must be
+ * reworked to break 1:1 relationship for net MBIM and to forward setup
+ * call to rmnet(QMAP) otherwise.
+ */
+ if (if_id != 0)
+ return -EINVAL;
+
+ if (dev_get_drvdata(&mhi_dev->dev))
+ return -EBUSY;
+
+ mhi_netdev = wwan_netdev_drvpriv(ndev);
- mhi_netdev = netdev_priv(ndev);
- dev_set_drvdata(dev, mhi_netdev);
+ dev_set_drvdata(&mhi_dev->dev, mhi_netdev);
mhi_netdev->ndev = ndev;
mhi_netdev->mdev = mhi_dev;
mhi_netdev->skbagg_head = NULL;
mhi_netdev->proto = info->proto;
- SET_NETDEV_DEV(ndev, &mhi_dev->dev);
- SET_NETDEV_DEVTYPE(ndev, &wwan_type);
INIT_DELAYED_WORK(&mhi_netdev->rx_refill, mhi_net_rx_refill_work);
u64_stats_init(&mhi_netdev->stats.rx_syncp);
/* Number of transfer descriptors determines size of the queue */
mhi_netdev->rx_queue_sz = mhi_get_free_desc_count(mhi_dev, DMA_FROM_DEVICE);
- err = register_netdev(ndev);
+ if (extack)
+ err = register_netdevice(ndev);
+ else
+ err = register_netdev(ndev);
if (err)
goto out_err;
return 0;
out_err_proto:
- unregister_netdev(ndev);
+ unregister_netdevice(ndev);
out_err:
free_netdev(ndev);
return err;
}
-static void mhi_net_remove(struct mhi_device *mhi_dev)
+static void mhi_net_dellink(void *ctxt, struct net_device *ndev,
+ struct list_head *head)
{
- struct mhi_net_dev *mhi_netdev = dev_get_drvdata(&mhi_dev->dev);
+ struct mhi_net_dev *mhi_netdev = wwan_netdev_drvpriv(ndev);
+ struct mhi_device *mhi_dev = ctxt;
- unregister_netdev(mhi_netdev->ndev);
+ if (head)
+ unregister_netdevice_queue(ndev, head);
+ else
+ unregister_netdev(ndev);
- mhi_unprepare_from_transfer(mhi_netdev->mdev);
+ mhi_unprepare_from_transfer(mhi_dev);
kfree_skb(mhi_netdev->skbagg_head);
- free_netdev(mhi_netdev->ndev);
+ dev_set_drvdata(&mhi_dev->dev, NULL);
+}
+
+static const struct wwan_ops mhi_wwan_ops = {
+ .priv_size = sizeof(struct mhi_net_dev),
+ .setup = mhi_net_setup,
+ .newlink = mhi_net_newlink,
+ .dellink = mhi_net_dellink,
+};
+
+static int mhi_net_probe(struct mhi_device *mhi_dev,
+ const struct mhi_device_id *id)
+{
+ const struct mhi_device_info *info = (struct mhi_device_info *)id->driver_data;
+ struct mhi_controller *cntrl = mhi_dev->mhi_cntrl;
+ struct net_device *ndev;
+ int err;
+
+ err = wwan_register_ops(&cntrl->mhi_dev->dev, &mhi_wwan_ops, mhi_dev,
+ WWAN_NO_DEFAULT_LINK);
+ if (err)
+ return err;
+
+ if (!create_default_iface)
+ return 0;
+
+ /* Create a default interface which is used as either RMNET real-dev,
+ * MBIM link 0 or ip link 0)
+ */
+ ndev = alloc_netdev(sizeof(struct mhi_net_dev), info->netname,
+ NET_NAME_PREDICTABLE, mhi_net_setup);
+ if (!ndev) {
+ err = -ENOMEM;
+ goto err_unregister;
+ }
+
+ SET_NETDEV_DEV(ndev, &mhi_dev->dev);
+
+ err = mhi_net_newlink(mhi_dev, ndev, 0, NULL);
+ if (err)
+ goto err_release;
+
+ return 0;
+
+err_release:
+ free_netdev(ndev);
+err_unregister:
+ wwan_unregister_ops(&cntrl->mhi_dev->dev);
+
+ return err;
+}
+
+static void mhi_net_remove(struct mhi_device *mhi_dev)
+{
+ struct mhi_net_dev *mhi_netdev = dev_get_drvdata(&mhi_dev->dev);
+ struct mhi_controller *cntrl = mhi_dev->mhi_cntrl;
+
+ /* WWAN core takes care of removing remaining links */
+ wwan_unregister_ops(&cntrl->mhi_dev->dev);
+
+ if (create_default_iface)
+ mhi_net_dellink(mhi_dev, mhi_netdev->ndev, NULL);
}
static const struct mhi_device_info mhi_hwip0 = {
#include <linux/ip.h>
#include <linux/mii.h>
#include <linux/netdevice.h>
+#include <linux/wwan.h>
#include <linux/skbuff.h>
#include <linux/usb.h>
#include <linux/usb/cdc.h>
static int mbim_rx_verify_nth16(struct sk_buff *skb)
{
- struct mhi_net_dev *dev = netdev_priv(skb->dev);
+ struct mhi_net_dev *dev = wwan_netdev_drvpriv(skb->dev);
struct mbim_context *ctx = dev->proto_data;
struct usb_cdc_ncm_nth16 *nth16;
int len;
static int mbim_rx_verify_ndp16(struct sk_buff *skb, struct usb_cdc_ncm_ndp16 *ndp16)
{
- struct mhi_net_dev *dev = netdev_priv(skb->dev);
+ struct mhi_net_dev *dev = wwan_netdev_drvpriv(skb->dev);
int ret;
if (le16_to_cpu(ndp16->wLength) < USB_CDC_NCM_NDP16_LENGTH_MIN) {
bmcr = mii->mdio_read(dev, mii->phy_id, MII_BMCR);
bmsr = mii->mdio_read(dev, mii->phy_id, MII_BMSR);
if (mii->supports_gmii) {
- ctrl1000 = mii->mdio_read(dev, mii->phy_id, MII_CTRL1000);
+ ctrl1000 = mii->mdio_read(dev, mii->phy_id, MII_CTRL1000);
stat1000 = mii->mdio_read(dev, mii->phy_id, MII_STAT1000);
}
static int nsim_bus_dev_vfs_enable(struct nsim_bus_dev *nsim_bus_dev,
unsigned int num_vfs)
{
- nsim_bus_dev->vfconfigs = kcalloc(num_vfs,
- sizeof(struct nsim_vf_config),
- GFP_KERNEL | __GFP_NOWARN);
+ struct nsim_dev *nsim_dev;
+ int err = 0;
+
+ if (nsim_bus_dev->max_vfs < num_vfs)
+ return -ENOMEM;
+
if (!nsim_bus_dev->vfconfigs)
return -ENOMEM;
nsim_bus_dev->num_vfs = num_vfs;
- return 0;
+ nsim_dev = dev_get_drvdata(&nsim_bus_dev->dev);
+ if (nsim_esw_mode_is_switchdev(nsim_dev)) {
+ err = nsim_esw_switchdev_enable(nsim_dev, NULL);
+ if (err)
+ nsim_bus_dev->num_vfs = 0;
+ }
+
+ return err;
}
-static void nsim_bus_dev_vfs_disable(struct nsim_bus_dev *nsim_bus_dev)
+void nsim_bus_dev_vfs_disable(struct nsim_bus_dev *nsim_bus_dev)
{
- kfree(nsim_bus_dev->vfconfigs);
- nsim_bus_dev->vfconfigs = NULL;
+ struct nsim_dev *nsim_dev;
+
nsim_bus_dev->num_vfs = 0;
+ nsim_dev = dev_get_drvdata(&nsim_bus_dev->dev);
+ if (nsim_esw_mode_is_switchdev(nsim_dev))
+ nsim_esw_legacy_enable(nsim_dev, NULL);
}
static ssize_t
if (ret)
return ret;
- rtnl_lock();
+ mutex_lock(&nsim_bus_dev->vfs_lock);
if (nsim_bus_dev->num_vfs == num_vfs)
goto exit_good;
if (nsim_bus_dev->num_vfs && num_vfs) {
exit_good:
ret = count;
exit_unlock:
- rtnl_unlock();
+ mutex_unlock(&nsim_bus_dev->vfs_lock);
return ret;
}
__ATTR(sriov_numvfs, 0664, nsim_bus_dev_numvfs_show,
nsim_bus_dev_numvfs_store);
+ssize_t nsim_bus_dev_max_vfs_read(struct file *file,
+ char __user *data,
+ size_t count, loff_t *ppos)
+{
+ struct nsim_bus_dev *nsim_bus_dev = file->private_data;
+ char buf[11];
+ ssize_t len;
+
+ len = snprintf(buf, sizeof(buf), "%u\n", nsim_bus_dev->max_vfs);
+ if (len < 0)
+ return len;
+
+ return simple_read_from_buffer(data, count, ppos, buf, len);
+}
+
+ssize_t nsim_bus_dev_max_vfs_write(struct file *file,
+ const char __user *data,
+ size_t count, loff_t *ppos)
+{
+ struct nsim_bus_dev *nsim_bus_dev = file->private_data;
+ struct nsim_vf_config *vfconfigs;
+ ssize_t ret;
+ char buf[10];
+ u32 val;
+
+ if (*ppos != 0)
+ return 0;
+
+ if (count >= sizeof(buf))
+ return -ENOSPC;
+
+ mutex_lock(&nsim_bus_dev->vfs_lock);
+ /* Reject if VFs are configured */
+ if (nsim_bus_dev->num_vfs) {
+ ret = -EBUSY;
+ goto unlock;
+ }
+
+ ret = copy_from_user(buf, data, count);
+ if (ret) {
+ ret = -EFAULT;
+ goto unlock;
+ }
+
+ buf[count] = '\0';
+ ret = kstrtouint(buf, 10, &val);
+ if (ret) {
+ ret = -EIO;
+ goto unlock;
+ }
+
+ /* max_vfs limited by the maximum number of provided port indexes */
+ if (val > NSIM_DEV_VF_PORT_INDEX_MAX - NSIM_DEV_VF_PORT_INDEX_BASE) {
+ ret = -ERANGE;
+ goto unlock;
+ }
+
+ vfconfigs = kcalloc(val, sizeof(struct nsim_vf_config), GFP_KERNEL | __GFP_NOWARN);
+ if (!vfconfigs) {
+ ret = -ENOMEM;
+ goto unlock;
+ }
+
+ kfree(nsim_bus_dev->vfconfigs);
+ nsim_bus_dev->vfconfigs = vfconfigs;
+ nsim_bus_dev->max_vfs = val;
+ *ppos += count;
+ ret = count;
+unlock:
+ mutex_unlock(&nsim_bus_dev->vfs_lock);
+ return ret;
+}
+
static ssize_t
new_port_store(struct device *dev, struct device_attribute *attr,
const char *buf, size_t count)
mutex_lock(&nsim_bus_dev->nsim_bus_reload_lock);
devlink_reload_disable(devlink);
- ret = nsim_dev_port_add(nsim_bus_dev, port_index);
+ ret = nsim_dev_port_add(nsim_bus_dev, NSIM_DEV_PORT_TYPE_PF, port_index);
devlink_reload_enable(devlink);
mutex_unlock(&nsim_bus_dev->nsim_bus_reload_lock);
return ret ? ret : count;
mutex_lock(&nsim_bus_dev->nsim_bus_reload_lock);
devlink_reload_disable(devlink);
- ret = nsim_dev_port_del(nsim_bus_dev, port_index);
+ ret = nsim_dev_port_del(nsim_bus_dev, NSIM_DEV_PORT_TYPE_PF, port_index);
devlink_reload_enable(devlink);
mutex_unlock(&nsim_bus_dev->nsim_bus_reload_lock);
return ret ? ret : count;
static void nsim_bus_dev_release(struct device *dev)
{
- struct nsim_bus_dev *nsim_bus_dev = to_nsim_bus_dev(dev);
-
- nsim_bus_dev_vfs_disable(nsim_bus_dev);
}
static struct device_type nsim_bus_dev_type = {
.num_vf = nsim_num_vf,
};
+#define NSIM_BUS_DEV_MAX_VFS 4
+
static struct nsim_bus_dev *
nsim_bus_dev_new(unsigned int id, unsigned int port_count)
{
nsim_bus_dev->dev.type = &nsim_bus_dev_type;
nsim_bus_dev->port_count = port_count;
nsim_bus_dev->initial_net = current->nsproxy->net_ns;
+ nsim_bus_dev->max_vfs = NSIM_BUS_DEV_MAX_VFS;
mutex_init(&nsim_bus_dev->nsim_bus_reload_lock);
+ mutex_init(&nsim_bus_dev->vfs_lock);
/* Disallow using nsim_bus_dev */
smp_store_release(&nsim_bus_dev->init, false);
+ nsim_bus_dev->vfconfigs = kcalloc(nsim_bus_dev->max_vfs,
+ sizeof(struct nsim_vf_config),
+ GFP_KERNEL | __GFP_NOWARN);
+ if (!nsim_bus_dev->vfconfigs) {
+ err = -ENOMEM;
+ goto err_nsim_bus_dev_id_free;
+ }
+
err = device_register(&nsim_bus_dev->dev);
if (err)
- goto err_nsim_bus_dev_id_free;
+ goto err_nsim_vfs_free;
+
return nsim_bus_dev;
+err_nsim_vfs_free:
+ kfree(nsim_bus_dev->vfconfigs);
err_nsim_bus_dev_id_free:
ida_free(&nsim_bus_dev_ids, nsim_bus_dev->dev.id);
err_nsim_bus_dev_free:
smp_store_release(&nsim_bus_dev->init, false);
device_unregister(&nsim_bus_dev->dev);
ida_free(&nsim_bus_dev_ids, nsim_bus_dev->dev.id);
+ kfree(nsim_bus_dev->vfconfigs);
kfree(nsim_bus_dev);
}
#include "netdevsim.h"
+static unsigned int
+nsim_dev_port_index(enum nsim_dev_port_type type, unsigned int port_index)
+{
+ switch (type) {
+ case NSIM_DEV_PORT_TYPE_VF:
+ port_index = NSIM_DEV_VF_PORT_INDEX_BASE + port_index;
+ break;
+ case NSIM_DEV_PORT_TYPE_PF:
+ break;
+ }
+
+ return port_index;
+}
+
+static inline unsigned int nsim_dev_port_index_to_vf_index(unsigned int port_index)
+{
+ return port_index - NSIM_DEV_VF_PORT_INDEX_BASE;
+}
+
static struct dentry *nsim_dev_ddir;
#define NSIM_DEV_DUMMY_REGION_SIZE (1024 * 32)
.owner = THIS_MODULE,
};
+static const struct file_operations nsim_dev_max_vfs_fops = {
+ .open = simple_open,
+ .read = nsim_bus_dev_max_vfs_read,
+ .write = nsim_bus_dev_max_vfs_write,
+ .llseek = generic_file_llseek,
+ .owner = THIS_MODULE,
+};
+
static int nsim_dev_debugfs_init(struct nsim_dev *nsim_dev)
{
char dev_ddir_name[sizeof(DRV_NAME) + 10];
+ int err;
sprintf(dev_ddir_name, DRV_NAME "%u", nsim_dev->nsim_bus_dev->dev.id);
nsim_dev->ddir = debugfs_create_dir(dev_ddir_name, nsim_dev_ddir);
debugfs_create_bool("fail_trap_policer_counter_get", 0600,
nsim_dev->ddir,
&nsim_dev->fail_trap_policer_counter_get);
+ nsim_dev->max_vfs = debugfs_create_file("max_vfs",
+ 0600,
+ nsim_dev->ddir,
+ nsim_dev->nsim_bus_dev,
+ &nsim_dev_max_vfs_fops);
+ nsim_dev->nodes_ddir = debugfs_create_dir("rate_nodes", nsim_dev->ddir);
+ if (IS_ERR(nsim_dev->nodes_ddir)) {
+ err = PTR_ERR(nsim_dev->nodes_ddir);
+ goto err_out;
+ }
+ debugfs_create_bool("fail_trap_drop_counter_get", 0600,
+ nsim_dev->ddir,
+ &nsim_dev->fail_trap_drop_counter_get);
nsim_udp_tunnels_debugfs_create(nsim_dev);
return 0;
+
+err_out:
+ debugfs_remove_recursive(nsim_dev->ports_ddir);
+ debugfs_remove_recursive(nsim_dev->ddir);
+ return err;
}
static void nsim_dev_debugfs_exit(struct nsim_dev *nsim_dev)
{
+ debugfs_remove_recursive(nsim_dev->nodes_ddir);
debugfs_remove_recursive(nsim_dev->ports_ddir);
debugfs_remove_recursive(nsim_dev->ddir);
}
+static ssize_t nsim_dev_rate_parent_read(struct file *file,
+ char __user *data,
+ size_t count, loff_t *ppos)
+{
+ char **name_ptr = file->private_data;
+ size_t len;
+
+ if (!*name_ptr)
+ return 0;
+
+ len = strlen(*name_ptr);
+ return simple_read_from_buffer(data, count, ppos, *name_ptr, len);
+}
+
+static const struct file_operations nsim_dev_rate_parent_fops = {
+ .open = simple_open,
+ .read = nsim_dev_rate_parent_read,
+ .llseek = generic_file_llseek,
+ .owner = THIS_MODULE,
+};
+
static int nsim_dev_port_debugfs_init(struct nsim_dev *nsim_dev,
struct nsim_dev_port *nsim_dev_port)
{
+ struct nsim_bus_dev *nsim_bus_dev = nsim_dev->nsim_bus_dev;
+ unsigned int port_index = nsim_dev_port->port_index;
char port_ddir_name[16];
char dev_link_name[32];
- sprintf(port_ddir_name, "%u", nsim_dev_port->port_index);
+ sprintf(port_ddir_name, "%u", port_index);
nsim_dev_port->ddir = debugfs_create_dir(port_ddir_name,
nsim_dev->ports_ddir);
if (IS_ERR(nsim_dev_port->ddir))
return PTR_ERR(nsim_dev_port->ddir);
- sprintf(dev_link_name, "../../../" DRV_NAME "%u",
- nsim_dev->nsim_bus_dev->dev.id);
+ sprintf(dev_link_name, "../../../" DRV_NAME "%u", nsim_bus_dev->dev.id);
+ if (nsim_dev_port_is_vf(nsim_dev_port)) {
+ unsigned int vf_id = nsim_dev_port_index_to_vf_index(port_index);
+
+ debugfs_create_u16("tx_share", 0400, nsim_dev_port->ddir,
+ &nsim_bus_dev->vfconfigs[vf_id].min_tx_rate);
+ debugfs_create_u16("tx_max", 0400, nsim_dev_port->ddir,
+ &nsim_bus_dev->vfconfigs[vf_id].max_tx_rate);
+ nsim_dev_port->rate_parent = debugfs_create_file("rate_parent",
+ 0400,
+ nsim_dev_port->ddir,
+ &nsim_dev_port->parent_name,
+ &nsim_dev_rate_parent_fops);
+ }
debugfs_create_symlink("dev", nsim_dev_port->ddir, dev_link_name);
return 0;
devlink_region_destroy(nsim_dev->dummy_region);
}
+static void __nsim_dev_port_del(struct nsim_dev_port *nsim_dev_port);
+int nsim_esw_legacy_enable(struct nsim_dev *nsim_dev, struct netlink_ext_ack *extack)
+{
+ struct devlink *devlink = priv_to_devlink(nsim_dev);
+ struct nsim_dev_port *nsim_dev_port, *tmp;
+
+ devlink_rate_nodes_destroy(devlink);
+ mutex_lock(&nsim_dev->port_list_lock);
+ list_for_each_entry_safe(nsim_dev_port, tmp, &nsim_dev->port_list, list)
+ if (nsim_dev_port_is_vf(nsim_dev_port))
+ __nsim_dev_port_del(nsim_dev_port);
+ mutex_unlock(&nsim_dev->port_list_lock);
+ nsim_dev->esw_mode = DEVLINK_ESWITCH_MODE_LEGACY;
+ return 0;
+}
+
+int nsim_esw_switchdev_enable(struct nsim_dev *nsim_dev, struct netlink_ext_ack *extack)
+{
+ struct nsim_bus_dev *nsim_bus_dev = nsim_dev->nsim_bus_dev;
+ int i, err;
+
+ for (i = 0; i < nsim_bus_dev->num_vfs; i++) {
+ err = nsim_dev_port_add(nsim_bus_dev, NSIM_DEV_PORT_TYPE_VF, i);
+ if (err) {
+ NL_SET_ERR_MSG_MOD(extack, "Failed to initialize VFs' netdevsim ports");
+ pr_err("Failed to initialize VF id=%d. %d.\n", i, err);
+ goto err_port_add_vfs;
+ }
+ }
+ nsim_dev->esw_mode = DEVLINK_ESWITCH_MODE_SWITCHDEV;
+ return 0;
+
+err_port_add_vfs:
+ for (i--; i >= 0; i--)
+ nsim_dev_port_del(nsim_bus_dev, NSIM_DEV_PORT_TYPE_VF, i);
+ return err;
+}
+
+static int nsim_devlink_eswitch_mode_set(struct devlink *devlink, u16 mode,
+ struct netlink_ext_ack *extack)
+{
+ struct nsim_dev *nsim_dev = devlink_priv(devlink);
+ int err = 0;
+
+ mutex_lock(&nsim_dev->nsim_bus_dev->vfs_lock);
+ if (mode == nsim_dev->esw_mode)
+ goto unlock;
+
+ if (mode == DEVLINK_ESWITCH_MODE_LEGACY)
+ err = nsim_esw_legacy_enable(nsim_dev, extack);
+ else if (mode == DEVLINK_ESWITCH_MODE_SWITCHDEV)
+ err = nsim_esw_switchdev_enable(nsim_dev, extack);
+ else
+ err = -EINVAL;
+
+unlock:
+ mutex_unlock(&nsim_dev->nsim_bus_dev->vfs_lock);
+ return err;
+}
+
+static int nsim_devlink_eswitch_mode_get(struct devlink *devlink, u16 *mode)
+{
+ struct nsim_dev *nsim_dev = devlink_priv(devlink);
+
+ *mode = nsim_dev->esw_mode;
+ return 0;
+}
+
struct nsim_trap_item {
void *trap_ctx;
enum devlink_trap_action action;
struct delayed_work trap_report_dw;
struct nsim_trap_item *trap_items_arr;
u64 *trap_policers_cnt_arr;
+ u64 trap_pkt_cnt;
struct nsim_dev *nsim_dev;
spinlock_t trap_lock; /* Protects trap_items_arr */
};
return 0;
}
+#define NSIM_LINK_SPEED_MAX 5000 /* Mbps */
+#define NSIM_LINK_SPEED_UNIT 125000 /* 1 Mbps given in bytes/sec to avoid
+ * u64 overflow during conversion from
+ * bytes to bits.
+ */
+
+static int nsim_rate_bytes_to_units(char *name, u64 *rate, struct netlink_ext_ack *extack)
+{
+ u64 val;
+ u32 rem;
+
+ val = div_u64_rem(*rate, NSIM_LINK_SPEED_UNIT, &rem);
+ if (rem) {
+ pr_err("%s rate value %lluBps not in link speed units of 1Mbps.\n",
+ name, *rate);
+ NL_SET_ERR_MSG_MOD(extack, "TX rate value not in link speed units of 1Mbps.");
+ return -EINVAL;
+ }
+
+ if (val > NSIM_LINK_SPEED_MAX) {
+ pr_err("%s rate value %lluMbps exceed link maximum speed 5000Mbps.\n",
+ name, val);
+ NL_SET_ERR_MSG_MOD(extack, "TX rate value exceed link maximum speed 5000Mbps.");
+ return -EINVAL;
+ }
+ *rate = val;
+ return 0;
+}
+
+static int nsim_leaf_tx_share_set(struct devlink_rate *devlink_rate, void *priv,
+ u64 tx_share, struct netlink_ext_ack *extack)
+{
+ struct nsim_dev_port *nsim_dev_port = priv;
+ struct nsim_bus_dev *nsim_bus_dev = nsim_dev_port->ns->nsim_bus_dev;
+ int vf_id = nsim_dev_port_index_to_vf_index(nsim_dev_port->port_index);
+ int err;
+
+ err = nsim_rate_bytes_to_units("tx_share", &tx_share, extack);
+ if (err)
+ return err;
+
+ nsim_bus_dev->vfconfigs[vf_id].min_tx_rate = tx_share;
+ return 0;
+}
+
+static int nsim_leaf_tx_max_set(struct devlink_rate *devlink_rate, void *priv,
+ u64 tx_max, struct netlink_ext_ack *extack)
+{
+ struct nsim_dev_port *nsim_dev_port = priv;
+ struct nsim_bus_dev *nsim_bus_dev = nsim_dev_port->ns->nsim_bus_dev;
+ int vf_id = nsim_dev_port_index_to_vf_index(nsim_dev_port->port_index);
+ int err;
+
+ err = nsim_rate_bytes_to_units("tx_max", &tx_max, extack);
+ if (err)
+ return err;
+
+ nsim_bus_dev->vfconfigs[vf_id].max_tx_rate = tx_max;
+ return 0;
+}
+
+struct nsim_rate_node {
+ struct dentry *ddir;
+ struct dentry *rate_parent;
+ char *parent_name;
+ u16 tx_share;
+ u16 tx_max;
+};
+
+static int nsim_node_tx_share_set(struct devlink_rate *devlink_rate, void *priv,
+ u64 tx_share, struct netlink_ext_ack *extack)
+{
+ struct nsim_rate_node *nsim_node = priv;
+ int err;
+
+ err = nsim_rate_bytes_to_units("tx_share", &tx_share, extack);
+ if (err)
+ return err;
+
+ nsim_node->tx_share = tx_share;
+ return 0;
+}
+
+static int nsim_node_tx_max_set(struct devlink_rate *devlink_rate, void *priv,
+ u64 tx_max, struct netlink_ext_ack *extack)
+{
+ struct nsim_rate_node *nsim_node = priv;
+ int err;
+
+ err = nsim_rate_bytes_to_units("tx_max", &tx_max, extack);
+ if (err)
+ return err;
+
+ nsim_node->tx_max = tx_max;
+ return 0;
+}
+
+static int nsim_rate_node_new(struct devlink_rate *node, void **priv,
+ struct netlink_ext_ack *extack)
+{
+ struct nsim_dev *nsim_dev = devlink_priv(node->devlink);
+ struct nsim_rate_node *nsim_node;
+
+ if (!nsim_esw_mode_is_switchdev(nsim_dev)) {
+ NL_SET_ERR_MSG_MOD(extack, "Node creation allowed only in switchdev mode.");
+ return -EOPNOTSUPP;
+ }
+
+ nsim_node = kzalloc(sizeof(*nsim_node), GFP_KERNEL);
+ if (!nsim_node)
+ return -ENOMEM;
+
+ nsim_node->ddir = debugfs_create_dir(node->name, nsim_dev->nodes_ddir);
+
+ debugfs_create_u16("tx_share", 0400, nsim_node->ddir, &nsim_node->tx_share);
+ debugfs_create_u16("tx_max", 0400, nsim_node->ddir, &nsim_node->tx_max);
+ nsim_node->rate_parent = debugfs_create_file("rate_parent", 0400,
+ nsim_node->ddir,
+ &nsim_node->parent_name,
+ &nsim_dev_rate_parent_fops);
+
+ *priv = nsim_node;
+ return 0;
+}
+
+static int nsim_rate_node_del(struct devlink_rate *node, void *priv,
+ struct netlink_ext_ack *extack)
+{
+ struct nsim_rate_node *nsim_node = priv;
+
+ debugfs_remove(nsim_node->rate_parent);
+ debugfs_remove_recursive(nsim_node->ddir);
+ kfree(nsim_node);
+ return 0;
+}
+
+static int nsim_rate_leaf_parent_set(struct devlink_rate *child,
+ struct devlink_rate *parent,
+ void *priv_child, void *priv_parent,
+ struct netlink_ext_ack *extack)
+{
+ struct nsim_dev_port *nsim_dev_port = priv_child;
+
+ if (parent)
+ nsim_dev_port->parent_name = parent->name;
+ else
+ nsim_dev_port->parent_name = NULL;
+ return 0;
+}
+
+static int nsim_rate_node_parent_set(struct devlink_rate *child,
+ struct devlink_rate *parent,
+ void *priv_child, void *priv_parent,
+ struct netlink_ext_ack *extack)
+{
+ struct nsim_rate_node *nsim_node = priv_child;
+
+ if (parent)
+ nsim_node->parent_name = parent->name;
+ else
+ nsim_node->parent_name = NULL;
+ return 0;
+}
+
+static int
+nsim_dev_devlink_trap_drop_counter_get(struct devlink *devlink,
+ const struct devlink_trap *trap,
+ u64 *p_drops)
+{
+ struct nsim_dev *nsim_dev = devlink_priv(devlink);
+ u64 *cnt;
+
+ if (nsim_dev->fail_trap_drop_counter_get)
+ return -EINVAL;
+
+ cnt = &nsim_dev->trap_data->trap_pkt_cnt;
+ *p_drops = (*cnt)++;
+
+ return 0;
+}
+
static const struct devlink_ops nsim_dev_devlink_ops = {
+ .eswitch_mode_set = nsim_devlink_eswitch_mode_set,
+ .eswitch_mode_get = nsim_devlink_eswitch_mode_get,
.supported_flash_update_params = DEVLINK_SUPPORT_FLASH_UPDATE_COMPONENT |
DEVLINK_SUPPORT_FLASH_UPDATE_OVERWRITE_MASK,
.reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
.trap_group_set = nsim_dev_devlink_trap_group_set,
.trap_policer_set = nsim_dev_devlink_trap_policer_set,
.trap_policer_counter_get = nsim_dev_devlink_trap_policer_counter_get,
+ .rate_leaf_tx_share_set = nsim_leaf_tx_share_set,
+ .rate_leaf_tx_max_set = nsim_leaf_tx_max_set,
+ .rate_node_tx_share_set = nsim_node_tx_share_set,
+ .rate_node_tx_max_set = nsim_node_tx_max_set,
+ .rate_node_new = nsim_rate_node_new,
+ .rate_node_del = nsim_rate_node_del,
+ .rate_leaf_parent_set = nsim_rate_leaf_parent_set,
+ .rate_node_parent_set = nsim_rate_node_parent_set,
+ .trap_drop_counter_get = nsim_dev_devlink_trap_drop_counter_get,
};
#define NSIM_DEV_MAX_MACS_DEFAULT 32
#define NSIM_DEV_TEST1_DEFAULT true
-static int __nsim_dev_port_add(struct nsim_dev *nsim_dev,
+static int __nsim_dev_port_add(struct nsim_dev *nsim_dev, enum nsim_dev_port_type type,
unsigned int port_index)
{
+ struct nsim_bus_dev *nsim_bus_dev = nsim_dev->nsim_bus_dev;
struct devlink_port_attrs attrs = {};
struct nsim_dev_port *nsim_dev_port;
struct devlink_port *devlink_port;
int err;
+ if (type == NSIM_DEV_PORT_TYPE_VF && !nsim_bus_dev->num_vfs)
+ return -EINVAL;
+
nsim_dev_port = kzalloc(sizeof(*nsim_dev_port), GFP_KERNEL);
if (!nsim_dev_port)
return -ENOMEM;
- nsim_dev_port->port_index = port_index;
+ nsim_dev_port->port_index = nsim_dev_port_index(type, port_index);
+ nsim_dev_port->port_type = type;
devlink_port = &nsim_dev_port->devlink_port;
- attrs.flavour = DEVLINK_PORT_FLAVOUR_PHYSICAL;
- attrs.phys.port_number = port_index + 1;
+ if (nsim_dev_port_is_pf(nsim_dev_port)) {
+ attrs.flavour = DEVLINK_PORT_FLAVOUR_PHYSICAL;
+ attrs.phys.port_number = port_index + 1;
+ } else {
+ attrs.flavour = DEVLINK_PORT_FLAVOUR_PCI_VF;
+ attrs.pci_vf.pf = 0;
+ attrs.pci_vf.vf = port_index;
+ }
memcpy(attrs.switch_id.id, nsim_dev->switch_id.id, nsim_dev->switch_id.id_len);
attrs.switch_id.id_len = nsim_dev->switch_id.id_len;
devlink_port_attrs_set(devlink_port, &attrs);
err = devlink_port_register(priv_to_devlink(nsim_dev), devlink_port,
- port_index);
+ nsim_dev_port->port_index);
if (err)
goto err_port_free;
goto err_port_debugfs_exit;
}
+ if (nsim_dev_port_is_vf(nsim_dev_port)) {
+ err = devlink_rate_leaf_create(&nsim_dev_port->devlink_port,
+ nsim_dev_port);
+ if (err)
+ goto err_nsim_destroy;
+ }
+
devlink_port_type_eth_set(devlink_port, nsim_dev_port->ns->netdev);
list_add(&nsim_dev_port->list, &nsim_dev->port_list);
return 0;
+err_nsim_destroy:
+ nsim_destroy(nsim_dev_port->ns);
err_port_debugfs_exit:
nsim_dev_port_debugfs_exit(nsim_dev_port);
err_dl_port_unregister:
struct devlink_port *devlink_port = &nsim_dev_port->devlink_port;
list_del(&nsim_dev_port->list);
+ if (nsim_dev_port_is_vf(nsim_dev_port))
+ devlink_rate_leaf_destroy(&nsim_dev_port->devlink_port);
devlink_port_type_clear(devlink_port);
nsim_destroy(nsim_dev_port->ns);
nsim_dev_port_debugfs_exit(nsim_dev_port);
int i, err;
for (i = 0; i < port_count; i++) {
- err = __nsim_dev_port_add(nsim_dev, i);
+ err = __nsim_dev_port_add(nsim_dev, NSIM_DEV_PORT_TYPE_PF, i);
if (err)
goto err_port_del_all;
}
devlink_params_publish(devlink);
devlink_reload_enable(devlink);
+ nsim_dev->esw_mode = DEVLINK_ESWITCH_MODE_LEGACY;
return 0;
err_psample_exit:
if (devlink_is_reload_failed(devlink))
return;
debugfs_remove(nsim_dev->take_snapshot);
+
+ mutex_lock(&nsim_dev->nsim_bus_dev->vfs_lock);
+ if (nsim_dev->nsim_bus_dev->num_vfs)
+ nsim_bus_dev_vfs_disable(nsim_dev->nsim_bus_dev);
+ mutex_unlock(&nsim_dev->nsim_bus_dev->vfs_lock);
+
nsim_dev_port_del_all(nsim_dev);
nsim_dev_psample_exit(nsim_dev);
nsim_dev_health_exit(nsim_dev);
}
static struct nsim_dev_port *
-__nsim_dev_port_lookup(struct nsim_dev *nsim_dev, unsigned int port_index)
+__nsim_dev_port_lookup(struct nsim_dev *nsim_dev, enum nsim_dev_port_type type,
+ unsigned int port_index)
{
struct nsim_dev_port *nsim_dev_port;
+ port_index = nsim_dev_port_index(type, port_index);
list_for_each_entry(nsim_dev_port, &nsim_dev->port_list, list)
if (nsim_dev_port->port_index == port_index)
return nsim_dev_port;
return NULL;
}
-int nsim_dev_port_add(struct nsim_bus_dev *nsim_bus_dev,
+int nsim_dev_port_add(struct nsim_bus_dev *nsim_bus_dev, enum nsim_dev_port_type type,
unsigned int port_index)
{
struct nsim_dev *nsim_dev = dev_get_drvdata(&nsim_bus_dev->dev);
int err;
mutex_lock(&nsim_dev->port_list_lock);
- if (__nsim_dev_port_lookup(nsim_dev, port_index))
+ if (__nsim_dev_port_lookup(nsim_dev, type, port_index))
err = -EEXIST;
else
- err = __nsim_dev_port_add(nsim_dev, port_index);
+ err = __nsim_dev_port_add(nsim_dev, type, port_index);
mutex_unlock(&nsim_dev->port_list_lock);
return err;
}
-int nsim_dev_port_del(struct nsim_bus_dev *nsim_bus_dev,
+int nsim_dev_port_del(struct nsim_bus_dev *nsim_bus_dev, enum nsim_dev_port_type type,
unsigned int port_index)
{
struct nsim_dev *nsim_dev = dev_get_drvdata(&nsim_bus_dev->dev);
int err = 0;
mutex_lock(&nsim_dev->port_list_lock);
- nsim_dev_port = __nsim_dev_port_lookup(nsim_dev, port_index);
+ nsim_dev_port = __nsim_dev_port_lookup(nsim_dev, type, port_index);
if (!nsim_dev_port)
err = -ENOENT;
else
struct netdevsim *ns = netdev_priv(dev);
struct nsim_bus_dev *nsim_bus_dev = ns->nsim_bus_dev;
+ if (nsim_esw_mode_is_switchdev(ns->nsim_dev)) {
+ pr_err("Not supported in switchdev mode. Please use devlink API.\n");
+ return -EOPNOTSUPP;
+ }
+
if (vf >= nsim_bus_dev->num_vfs)
return -EINVAL;
.ndo_get_devlink_port = nsim_get_devlink_port,
};
+static const struct net_device_ops nsim_vf_netdev_ops = {
+ .ndo_start_xmit = nsim_start_xmit,
+ .ndo_set_rx_mode = nsim_set_rx_mode,
+ .ndo_set_mac_address = eth_mac_addr,
+ .ndo_validate_addr = eth_validate_addr,
+ .ndo_change_mtu = nsim_change_mtu,
+ .ndo_get_stats64 = nsim_get_stats64,
+ .ndo_setup_tc = nsim_setup_tc,
+ .ndo_set_features = nsim_set_features,
+ .ndo_get_devlink_port = nsim_get_devlink_port,
+};
+
static void nsim_setup(struct net_device *dev)
{
ether_setup(dev);
dev->max_mtu = ETH_MAX_MTU;
}
+static int nsim_init_netdevsim(struct netdevsim *ns)
+{
+ int err;
+
+ ns->netdev->netdev_ops = &nsim_netdev_ops;
+
+ err = nsim_udp_tunnels_info_create(ns->nsim_dev, ns->netdev);
+ if (err)
+ return err;
+
+ rtnl_lock();
+ err = nsim_bpf_init(ns);
+ if (err)
+ goto err_utn_destroy;
+
+ nsim_ipsec_init(ns);
+
+ err = register_netdevice(ns->netdev);
+ if (err)
+ goto err_ipsec_teardown;
+ rtnl_unlock();
+ return 0;
+
+err_ipsec_teardown:
+ nsim_ipsec_teardown(ns);
+ nsim_bpf_uninit(ns);
+err_utn_destroy:
+ rtnl_unlock();
+ nsim_udp_tunnels_info_destroy(ns->netdev);
+ return err;
+}
+
+static int nsim_init_netdevsim_vf(struct netdevsim *ns)
+{
+ int err;
+
+ ns->netdev->netdev_ops = &nsim_vf_netdev_ops;
+ rtnl_lock();
+ err = register_netdevice(ns->netdev);
+ rtnl_unlock();
+ return err;
+}
+
struct netdevsim *
nsim_create(struct nsim_dev *nsim_dev, struct nsim_dev_port *nsim_dev_port)
{
ns->nsim_dev_port = nsim_dev_port;
ns->nsim_bus_dev = nsim_dev->nsim_bus_dev;
SET_NETDEV_DEV(dev, &ns->nsim_bus_dev->dev);
- dev->netdev_ops = &nsim_netdev_ops;
nsim_ethtool_init(ns);
-
- err = nsim_udp_tunnels_info_create(nsim_dev, dev);
+ if (nsim_dev_port_is_pf(nsim_dev_port))
+ err = nsim_init_netdevsim(ns);
+ else
+ err = nsim_init_netdevsim_vf(ns);
if (err)
goto err_free_netdev;
-
- rtnl_lock();
- err = nsim_bpf_init(ns);
- if (err)
- goto err_utn_destroy;
-
- nsim_ipsec_init(ns);
-
- err = register_netdevice(dev);
- if (err)
- goto err_ipsec_teardown;
- rtnl_unlock();
-
return ns;
-err_ipsec_teardown:
- nsim_ipsec_teardown(ns);
- nsim_bpf_uninit(ns);
-err_utn_destroy:
- rtnl_unlock();
- nsim_udp_tunnels_info_destroy(dev);
err_free_netdev:
free_netdev(dev);
return ERR_PTR(err);
rtnl_lock();
unregister_netdevice(dev);
- nsim_ipsec_teardown(ns);
- nsim_bpf_uninit(ns);
+ if (nsim_dev_port_is_pf(ns->nsim_dev_port)) {
+ nsim_ipsec_teardown(ns);
+ nsim_bpf_uninit(ns);
+ }
rtnl_unlock();
- nsim_udp_tunnels_info_destroy(dev);
+ if (nsim_dev_port_is_pf(ns->nsim_dev_port))
+ nsim_udp_tunnels_info_destroy(dev);
free_netdev(dev);
}
}
#endif
+enum nsim_dev_port_type {
+ NSIM_DEV_PORT_TYPE_PF,
+ NSIM_DEV_PORT_TYPE_VF,
+};
+
+#define NSIM_DEV_VF_PORT_INDEX_BASE 128
+#define NSIM_DEV_VF_PORT_INDEX_MAX UINT_MAX
+
struct nsim_dev_port {
struct list_head list;
struct devlink_port devlink_port;
unsigned int port_index;
+ enum nsim_dev_port_type port_type;
struct dentry *ddir;
+ struct dentry *rate_parent;
+ char *parent_name;
struct netdevsim *ns;
};
struct dentry *ddir;
struct dentry *ports_ddir;
struct dentry *take_snapshot;
+ struct dentry *max_vfs;
+ struct dentry *nodes_ddir;
struct bpf_offload_dev *bpf_dev;
bool bpf_bind_accept;
bool bpf_bind_verifier_accept;
bool fail_trap_group_set;
bool fail_trap_policer_set;
bool fail_trap_policer_counter_get;
+ bool fail_trap_drop_counter_get;
struct {
struct udp_tunnel_nic_shared utn_shared;
u32 __ports[2][NSIM_UDP_TUNNEL_N_PORTS];
u32 sleep;
} udp_ports;
struct nsim_dev_psample *psample;
+ u16 esw_mode;
};
+int nsim_esw_legacy_enable(struct nsim_dev *nsim_dev, struct netlink_ext_ack *extack);
+int nsim_esw_switchdev_enable(struct nsim_dev *nsim_dev, struct netlink_ext_ack *extack);
+
+static inline bool nsim_esw_mode_is_legacy(struct nsim_dev *nsim_dev)
+{
+ return nsim_dev->esw_mode == DEVLINK_ESWITCH_MODE_LEGACY;
+}
+
+static inline bool nsim_esw_mode_is_switchdev(struct nsim_dev *nsim_dev)
+{
+ return nsim_dev->esw_mode == DEVLINK_ESWITCH_MODE_SWITCHDEV;
+}
+
static inline struct net *nsim_dev_net(struct nsim_dev *nsim_dev)
{
return devlink_net(priv_to_devlink(nsim_dev));
int nsim_dev_probe(struct nsim_bus_dev *nsim_bus_dev);
void nsim_dev_remove(struct nsim_bus_dev *nsim_bus_dev);
int nsim_dev_port_add(struct nsim_bus_dev *nsim_bus_dev,
+ enum nsim_dev_port_type type,
unsigned int port_index);
int nsim_dev_port_del(struct nsim_bus_dev *nsim_bus_dev,
+ enum nsim_dev_port_type type,
unsigned int port_index);
struct nsim_fib_data *nsim_fib_create(struct devlink *devlink,
u64 nsim_fib_get_val(struct nsim_fib_data *fib_data,
enum nsim_resource_id res_id, bool max);
+ssize_t nsim_bus_dev_max_vfs_read(struct file *file,
+ char __user *data,
+ size_t count, loff_t *ppos);
+ssize_t nsim_bus_dev_max_vfs_write(struct file *file,
+ const char __user *data,
+ size_t count, loff_t *ppos);
+void nsim_bus_dev_vfs_disable(struct nsim_bus_dev *nsim_bus_dev);
+
+static inline bool nsim_dev_port_is_pf(struct nsim_dev_port *nsim_dev_port)
+{
+ return nsim_dev_port->port_type == NSIM_DEV_PORT_TYPE_PF;
+}
+
+static inline bool nsim_dev_port_is_vf(struct nsim_dev_port *nsim_dev_port)
+{
+ return nsim_dev_port->port_type == NSIM_DEV_PORT_TYPE_VF;
+}
#if IS_ENABLED(CONFIG_XFRM_OFFLOAD)
void nsim_ipsec_init(struct netdevsim *ns);
void nsim_ipsec_teardown(struct netdevsim *ns);
struct net *initial_net; /* Purpose of this is to carry net pointer
* during the probe time only.
*/
+ unsigned int max_vfs;
unsigned int num_vfs;
+ struct mutex vfs_lock; /* Protects vfconfigs */
struct nsim_vf_config *vfconfigs;
/* Lock for devlink->reload_enabled in netdevsim module */
struct mutex nsim_bus_reload_lock;
# SPDX-License-Identifier: GPL-2.0
# Makefile for Linux PCS drivers
-obj-$(CONFIG_PCS_XPCS) += pcs-xpcs.o
+pcs_xpcs-$(CONFIG_PCS_XPCS) := pcs-xpcs.o pcs-xpcs-nxp.o
+
+obj-$(CONFIG_PCS_XPCS) += pcs_xpcs.o
obj-$(CONFIG_PCS_LYNX) += pcs-lynx.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright 2021 NXP Semiconductors
+ */
+#include <linux/pcs/pcs-xpcs.h>
+#include "pcs-xpcs.h"
+
+/* LANE_DRIVER1_0 register */
+#define SJA1110_LANE_DRIVER1_0 0x8038
+#define SJA1110_TXDRV(x) (((x) << 12) & GENMASK(14, 12))
+
+/* LANE_DRIVER2_0 register */
+#define SJA1110_LANE_DRIVER2_0 0x803a
+#define SJA1110_TXDRVTRIM_LSB(x) ((x) & GENMASK_ULL(15, 0))
+
+/* LANE_DRIVER2_1 register */
+#define SJA1110_LANE_DRIVER2_1 0x803b
+#define SJA1110_LANE_DRIVER2_1_RSV BIT(9)
+#define SJA1110_TXDRVTRIM_MSB(x) (((x) & GENMASK_ULL(23, 16)) >> 16)
+
+/* LANE_TRIM register */
+#define SJA1110_LANE_TRIM 0x8040
+#define SJA1110_TXTEN BIT(11)
+#define SJA1110_TXRTRIM(x) (((x) << 8) & GENMASK(10, 8))
+#define SJA1110_TXPLL_BWSEL BIT(7)
+#define SJA1110_RXTEN BIT(6)
+#define SJA1110_RXRTRIM(x) (((x) << 3) & GENMASK(5, 3))
+#define SJA1110_CDR_GAIN BIT(2)
+#define SJA1110_ACCOUPLE_RXVCM_EN BIT(0)
+
+/* LANE_DATAPATH_1 register */
+#define SJA1110_LANE_DATAPATH_1 0x8037
+
+/* POWERDOWN_ENABLE register */
+#define SJA1110_POWERDOWN_ENABLE 0x8041
+#define SJA1110_TXPLL_PD BIT(12)
+#define SJA1110_TXPD BIT(11)
+#define SJA1110_RXPKDETEN BIT(10)
+#define SJA1110_RXCH_PD BIT(9)
+#define SJA1110_RXBIAS_PD BIT(8)
+#define SJA1110_RESET_SER_EN BIT(7)
+#define SJA1110_RESET_SER BIT(6)
+#define SJA1110_RESET_DES BIT(5)
+#define SJA1110_RCVEN BIT(4)
+
+/* RXPLL_CTRL0 register */
+#define SJA1110_RXPLL_CTRL0 0x8065
+#define SJA1110_RXPLL_FBDIV(x) (((x) << 2) & GENMASK(9, 2))
+
+/* RXPLL_CTRL1 register */
+#define SJA1110_RXPLL_CTRL1 0x8066
+#define SJA1110_RXPLL_REFDIV(x) ((x) & GENMASK(4, 0))
+
+/* TXPLL_CTRL0 register */
+#define SJA1110_TXPLL_CTRL0 0x806d
+#define SJA1110_TXPLL_FBDIV(x) ((x) & GENMASK(11, 0))
+
+/* TXPLL_CTRL1 register */
+#define SJA1110_TXPLL_CTRL1 0x806e
+#define SJA1110_TXPLL_REFDIV(x) ((x) & GENMASK(5, 0))
+
+/* RX_DATA_DETECT register */
+#define SJA1110_RX_DATA_DETECT 0x8045
+
+/* RX_CDR_CTLE register */
+#define SJA1110_RX_CDR_CTLE 0x8042
+
+/* In NXP SJA1105, the PCS is integrated with a PMA that has the TX lane
+ * polarity inverted by default (PLUS is MINUS, MINUS is PLUS). To obtain
+ * normal non-inverted behavior, the TX lane polarity must be inverted in the
+ * PCS, via the DIGITAL_CONTROL_2 register.
+ */
+int nxp_sja1105_sgmii_pma_config(struct dw_xpcs *xpcs)
+{
+ return xpcs_write(xpcs, MDIO_MMD_VEND2, DW_VR_MII_DIG_CTRL2,
+ DW_VR_MII_DIG_CTRL2_TX_POL_INV);
+}
+
+static int nxp_sja1110_pma_config(struct dw_xpcs *xpcs,
+ u16 txpll_fbdiv, u16 txpll_refdiv,
+ u16 rxpll_fbdiv, u16 rxpll_refdiv,
+ u16 rx_cdr_ctle)
+{
+ u16 val;
+ int ret;
+
+ /* Program TX PLL feedback divider and reference divider settings for
+ * correct oscillation frequency.
+ */
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, SJA1110_TXPLL_CTRL0,
+ SJA1110_TXPLL_FBDIV(txpll_fbdiv));
+ if (ret < 0)
+ return ret;
+
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, SJA1110_TXPLL_CTRL1,
+ SJA1110_TXPLL_REFDIV(txpll_refdiv));
+ if (ret < 0)
+ return ret;
+
+ /* Program transmitter amplitude and disable amplitude trimming */
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, SJA1110_LANE_DRIVER1_0,
+ SJA1110_TXDRV(0x5));
+ if (ret < 0)
+ return ret;
+
+ val = SJA1110_TXDRVTRIM_LSB(0xffffffull);
+
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, SJA1110_LANE_DRIVER2_0, val);
+ if (ret < 0)
+ return ret;
+
+ val = SJA1110_TXDRVTRIM_MSB(0xffffffull) | SJA1110_LANE_DRIVER2_1_RSV;
+
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, SJA1110_LANE_DRIVER2_1, val);
+ if (ret < 0)
+ return ret;
+
+ /* Enable input and output resistor terminations for low BER. */
+ val = SJA1110_ACCOUPLE_RXVCM_EN | SJA1110_CDR_GAIN |
+ SJA1110_RXRTRIM(4) | SJA1110_RXTEN | SJA1110_TXPLL_BWSEL |
+ SJA1110_TXRTRIM(3) | SJA1110_TXTEN;
+
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, SJA1110_LANE_TRIM, val);
+ if (ret < 0)
+ return ret;
+
+ /* Select PCS as transmitter data source. */
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, SJA1110_LANE_DATAPATH_1, 0);
+ if (ret < 0)
+ return ret;
+
+ /* Program RX PLL feedback divider and reference divider for correct
+ * oscillation frequency.
+ */
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, SJA1110_RXPLL_CTRL0,
+ SJA1110_RXPLL_FBDIV(rxpll_fbdiv));
+ if (ret < 0)
+ return ret;
+
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, SJA1110_RXPLL_CTRL1,
+ SJA1110_RXPLL_REFDIV(rxpll_refdiv));
+ if (ret < 0)
+ return ret;
+
+ /* Program threshold for receiver signal detector.
+ * Enable control of RXPLL by receiver signal detector to disable RXPLL
+ * when an input signal is not present.
+ */
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, SJA1110_RX_DATA_DETECT, 0x0005);
+ if (ret < 0)
+ return ret;
+
+ /* Enable TX and RX PLLs and circuits.
+ * Release reset of PMA to enable data flow to/from PCS.
+ */
+ ret = xpcs_read(xpcs, MDIO_MMD_VEND2, SJA1110_POWERDOWN_ENABLE);
+ if (ret < 0)
+ return ret;
+
+ val = ret & ~(SJA1110_TXPLL_PD | SJA1110_TXPD | SJA1110_RXCH_PD |
+ SJA1110_RXBIAS_PD | SJA1110_RESET_SER_EN |
+ SJA1110_RESET_SER | SJA1110_RESET_DES);
+ val |= SJA1110_RXPKDETEN | SJA1110_RCVEN;
+
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, SJA1110_POWERDOWN_ENABLE, val);
+ if (ret < 0)
+ return ret;
+
+ /* Program continuous-time linear equalizer (CTLE) settings. */
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, SJA1110_RX_CDR_CTLE,
+ rx_cdr_ctle);
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+int nxp_sja1110_sgmii_pma_config(struct dw_xpcs *xpcs)
+{
+ return nxp_sja1110_pma_config(xpcs, 0x19, 0x1, 0x19, 0x1, 0x212a);
+}
+
+int nxp_sja1110_2500basex_pma_config(struct dw_xpcs *xpcs)
+{
+ return nxp_sja1110_pma_config(xpcs, 0x7d, 0x2, 0x7d, 0x2, 0x732a);
+}
#include <linux/mdio.h>
#include <linux/phylink.h>
#include <linux/workqueue.h>
+#include "pcs-xpcs.h"
-#define SYNOPSYS_XPCS_USXGMII_ID 0x7996ced0
-#define SYNOPSYS_XPCS_10GKR_ID 0x7996ced0
-#define SYNOPSYS_XPCS_XLGMII_ID 0x7996ced0
-#define SYNOPSYS_XPCS_SGMII_ID 0x7996ced0
-#define SYNOPSYS_XPCS_MASK 0xffffffff
-
-/* Vendor regs access */
-#define DW_VENDOR BIT(15)
-
-/* VR_XS_PCS */
-#define DW_USXGMII_RST BIT(10)
-#define DW_USXGMII_EN BIT(9)
-#define DW_VR_XS_PCS_DIG_STS 0x0010
-#define DW_RXFIFO_ERR GENMASK(6, 5)
-
-/* SR_MII */
-#define DW_USXGMII_FULL BIT(8)
-#define DW_USXGMII_SS_MASK (BIT(13) | BIT(6) | BIT(5))
-#define DW_USXGMII_10000 (BIT(13) | BIT(6))
-#define DW_USXGMII_5000 (BIT(13) | BIT(5))
-#define DW_USXGMII_2500 (BIT(5))
-#define DW_USXGMII_1000 (BIT(6))
-#define DW_USXGMII_100 (BIT(13))
-#define DW_USXGMII_10 (0)
-
-/* SR_AN */
-#define DW_SR_AN_ADV1 0x10
-#define DW_SR_AN_ADV2 0x11
-#define DW_SR_AN_ADV3 0x12
-#define DW_SR_AN_LP_ABL1 0x13
-#define DW_SR_AN_LP_ABL2 0x14
-#define DW_SR_AN_LP_ABL3 0x15
-
-/* Clause 73 Defines */
-/* AN_LP_ABL1 */
-#define DW_C73_PAUSE BIT(10)
-#define DW_C73_ASYM_PAUSE BIT(11)
-#define DW_C73_AN_ADV_SF 0x1
-/* AN_LP_ABL2 */
-#define DW_C73_1000KX BIT(5)
-#define DW_C73_10000KX4 BIT(6)
-#define DW_C73_10000KR BIT(7)
-/* AN_LP_ABL3 */
-#define DW_C73_2500KX BIT(0)
-#define DW_C73_5000KR BIT(1)
-
-/* Clause 37 Defines */
-/* VR MII MMD registers offsets */
-#define DW_VR_MII_DIG_CTRL1 0x8000
-#define DW_VR_MII_AN_CTRL 0x8001
-#define DW_VR_MII_AN_INTR_STS 0x8002
-
-/* VR_MII_DIG_CTRL1 */
-#define DW_VR_MII_DIG_CTRL1_MAC_AUTO_SW BIT(9)
-
-/* VR_MII_AN_CTRL */
-#define DW_VR_MII_AN_CTRL_TX_CONFIG_SHIFT 3
-#define DW_VR_MII_TX_CONFIG_MASK BIT(3)
-#define DW_VR_MII_TX_CONFIG_PHY_SIDE_SGMII 0x1
-#define DW_VR_MII_TX_CONFIG_MAC_SIDE_SGMII 0x0
-#define DW_VR_MII_AN_CTRL_PCS_MODE_SHIFT 1
-#define DW_VR_MII_PCS_MODE_MASK GENMASK(2, 1)
-#define DW_VR_MII_PCS_MODE_C37_1000BASEX 0x0
-#define DW_VR_MII_PCS_MODE_C37_SGMII 0x2
-
-/* VR_MII_AN_INTR_STS */
-#define DW_VR_MII_AN_STS_C37_ANSGM_FD BIT(1)
-#define DW_VR_MII_AN_STS_C37_ANSGM_SP_SHIFT 2
-#define DW_VR_MII_AN_STS_C37_ANSGM_SP GENMASK(3, 2)
-#define DW_VR_MII_C37_ANSGM_SP_10 0x0
-#define DW_VR_MII_C37_ANSGM_SP_100 0x1
-#define DW_VR_MII_C37_ANSGM_SP_1000 0x2
-#define DW_VR_MII_C37_ANSGM_SP_LNKSTS BIT(4)
+#define phylink_pcs_to_xpcs(pl_pcs) \
+ container_of((pl_pcs), struct dw_xpcs, pcs)
static const int xpcs_usxgmii_features[] = {
ETHTOOL_LINK_MODE_Pause_BIT,
__ETHTOOL_LINK_MODE_MASK_NBITS,
};
+static const int xpcs_2500basex_features[] = {
+ ETHTOOL_LINK_MODE_Asym_Pause_BIT,
+ ETHTOOL_LINK_MODE_Autoneg_BIT,
+ ETHTOOL_LINK_MODE_2500baseX_Full_BIT,
+ ETHTOOL_LINK_MODE_2500baseT_Full_BIT,
+ __ETHTOOL_LINK_MODE_MASK_NBITS,
+};
+
static const phy_interface_t xpcs_usxgmii_interfaces[] = {
PHY_INTERFACE_MODE_USXGMII,
- PHY_INTERFACE_MODE_MAX,
};
static const phy_interface_t xpcs_10gkr_interfaces[] = {
PHY_INTERFACE_MODE_10GKR,
- PHY_INTERFACE_MODE_MAX,
};
static const phy_interface_t xpcs_xlgmii_interfaces[] = {
PHY_INTERFACE_MODE_XLGMII,
- PHY_INTERFACE_MODE_MAX,
};
static const phy_interface_t xpcs_sgmii_interfaces[] = {
PHY_INTERFACE_MODE_SGMII,
+};
+
+static const phy_interface_t xpcs_2500basex_interfaces[] = {
+ PHY_INTERFACE_MODE_2500BASEX,
PHY_INTERFACE_MODE_MAX,
};
-static struct xpcs_id {
- u32 id;
- u32 mask;
+enum {
+ DW_XPCS_USXGMII,
+ DW_XPCS_10GKR,
+ DW_XPCS_XLGMII,
+ DW_XPCS_SGMII,
+ DW_XPCS_2500BASEX,
+ DW_XPCS_INTERFACE_MAX,
+};
+
+struct xpcs_compat {
const int *supported;
const phy_interface_t *interface;
+ int num_interfaces;
int an_mode;
-} xpcs_id_list[] = {
- {
- .id = SYNOPSYS_XPCS_USXGMII_ID,
- .mask = SYNOPSYS_XPCS_MASK,
- .supported = xpcs_usxgmii_features,
- .interface = xpcs_usxgmii_interfaces,
- .an_mode = DW_AN_C73,
- }, {
- .id = SYNOPSYS_XPCS_10GKR_ID,
- .mask = SYNOPSYS_XPCS_MASK,
- .supported = xpcs_10gkr_features,
- .interface = xpcs_10gkr_interfaces,
- .an_mode = DW_AN_C73,
- }, {
- .id = SYNOPSYS_XPCS_XLGMII_ID,
- .mask = SYNOPSYS_XPCS_MASK,
- .supported = xpcs_xlgmii_features,
- .interface = xpcs_xlgmii_interfaces,
- .an_mode = DW_AN_C73,
- }, {
- .id = SYNOPSYS_XPCS_SGMII_ID,
- .mask = SYNOPSYS_XPCS_MASK,
- .supported = xpcs_sgmii_features,
- .interface = xpcs_sgmii_interfaces,
- .an_mode = DW_AN_C37_SGMII,
- },
+ int (*pma_config)(struct dw_xpcs *xpcs);
};
-static int xpcs_read(struct mdio_xpcs_args *xpcs, int dev, u32 reg)
+struct xpcs_id {
+ u32 id;
+ u32 mask;
+ const struct xpcs_compat *compat;
+};
+
+static const struct xpcs_compat *xpcs_find_compat(const struct xpcs_id *id,
+ phy_interface_t interface)
+{
+ int i, j;
+
+ for (i = 0; i < DW_XPCS_INTERFACE_MAX; i++) {
+ const struct xpcs_compat *compat = &id->compat[i];
+
+ for (j = 0; j < compat->num_interfaces; j++)
+ if (compat->interface[j] == interface)
+ return compat;
+ }
+
+ return NULL;
+}
+
+int xpcs_get_an_mode(struct dw_xpcs *xpcs, phy_interface_t interface)
+{
+ const struct xpcs_compat *compat;
+
+ compat = xpcs_find_compat(xpcs->id, interface);
+ if (!compat)
+ return -ENODEV;
+
+ return compat->an_mode;
+}
+EXPORT_SYMBOL_GPL(xpcs_get_an_mode);
+
+static bool __xpcs_linkmode_supported(const struct xpcs_compat *compat,
+ enum ethtool_link_mode_bit_indices linkmode)
{
- u32 reg_addr = MII_ADDR_C45 | dev << 16 | reg;
+ int i;
- return mdiobus_read(xpcs->bus, xpcs->addr, reg_addr);
+ for (i = 0; compat->supported[i] != __ETHTOOL_LINK_MODE_MASK_NBITS; i++)
+ if (compat->supported[i] == linkmode)
+ return true;
+
+ return false;
}
-static int xpcs_write(struct mdio_xpcs_args *xpcs, int dev, u32 reg, u16 val)
+#define xpcs_linkmode_supported(compat, mode) \
+ __xpcs_linkmode_supported(compat, ETHTOOL_LINK_MODE_ ## mode ## _BIT)
+
+int xpcs_read(struct dw_xpcs *xpcs, int dev, u32 reg)
{
- u32 reg_addr = MII_ADDR_C45 | dev << 16 | reg;
+ u32 reg_addr = mdiobus_c45_addr(dev, reg);
+ struct mii_bus *bus = xpcs->mdiodev->bus;
+ int addr = xpcs->mdiodev->addr;
- return mdiobus_write(xpcs->bus, xpcs->addr, reg_addr, val);
+ return mdiobus_read(bus, addr, reg_addr);
}
-static int xpcs_read_vendor(struct mdio_xpcs_args *xpcs, int dev, u32 reg)
+int xpcs_write(struct dw_xpcs *xpcs, int dev, u32 reg, u16 val)
+{
+ u32 reg_addr = mdiobus_c45_addr(dev, reg);
+ struct mii_bus *bus = xpcs->mdiodev->bus;
+ int addr = xpcs->mdiodev->addr;
+
+ return mdiobus_write(bus, addr, reg_addr, val);
+}
+
+static int xpcs_read_vendor(struct dw_xpcs *xpcs, int dev, u32 reg)
{
return xpcs_read(xpcs, dev, DW_VENDOR | reg);
}
-static int xpcs_write_vendor(struct mdio_xpcs_args *xpcs, int dev, int reg,
+static int xpcs_write_vendor(struct dw_xpcs *xpcs, int dev, int reg,
u16 val)
{
return xpcs_write(xpcs, dev, DW_VENDOR | reg, val);
}
-static int xpcs_read_vpcs(struct mdio_xpcs_args *xpcs, int reg)
+static int xpcs_read_vpcs(struct dw_xpcs *xpcs, int reg)
{
return xpcs_read_vendor(xpcs, MDIO_MMD_PCS, reg);
}
-static int xpcs_write_vpcs(struct mdio_xpcs_args *xpcs, int reg, u16 val)
+static int xpcs_write_vpcs(struct dw_xpcs *xpcs, int reg, u16 val)
{
return xpcs_write_vendor(xpcs, MDIO_MMD_PCS, reg, val);
}
-static int xpcs_poll_reset(struct mdio_xpcs_args *xpcs, int dev)
+static int xpcs_poll_reset(struct dw_xpcs *xpcs, int dev)
{
/* Poll until the reset bit clears (50ms per retry == 0.6 sec) */
unsigned int retries = 12;
return (ret & MDIO_CTRL1_RESET) ? -ETIMEDOUT : 0;
}
-static int xpcs_soft_reset(struct mdio_xpcs_args *xpcs)
+static int xpcs_soft_reset(struct dw_xpcs *xpcs,
+ const struct xpcs_compat *compat)
{
int ret, dev;
- switch (xpcs->an_mode) {
+ switch (compat->an_mode) {
case DW_AN_C73:
dev = MDIO_MMD_PCS;
break;
case DW_AN_C37_SGMII:
+ case DW_2500BASEX:
dev = MDIO_MMD_VEND2;
break;
default:
#define xpcs_warn(__xpcs, __state, __args...) \
({ \
if ((__state)->link) \
- dev_warn(&(__xpcs)->bus->dev, ##__args); \
+ dev_warn(&(__xpcs)->mdiodev->dev, ##__args); \
})
-static int xpcs_read_fault_c73(struct mdio_xpcs_args *xpcs,
+static int xpcs_read_fault_c73(struct dw_xpcs *xpcs,
struct phylink_link_state *state)
{
int ret;
return 0;
}
-static int xpcs_read_link_c73(struct mdio_xpcs_args *xpcs, bool an)
+static int xpcs_read_link_c73(struct dw_xpcs *xpcs, bool an)
{
bool link = true;
int ret;
return max;
}
-static int xpcs_config_usxgmii(struct mdio_xpcs_args *xpcs, int speed)
+static void xpcs_config_usxgmii(struct dw_xpcs *xpcs, int speed)
{
int ret, speed_sel;
break;
default:
/* Nothing to do here */
- return -EINVAL;
+ return;
}
ret = xpcs_read_vpcs(xpcs, MDIO_CTRL1);
if (ret < 0)
- return ret;
+ goto out;
ret = xpcs_write_vpcs(xpcs, MDIO_CTRL1, ret | DW_USXGMII_EN);
if (ret < 0)
- return ret;
+ goto out;
ret = xpcs_read(xpcs, MDIO_MMD_VEND2, MDIO_CTRL1);
if (ret < 0)
- return ret;
+ goto out;
ret &= ~DW_USXGMII_SS_MASK;
ret |= speed_sel | DW_USXGMII_FULL;
ret = xpcs_write(xpcs, MDIO_MMD_VEND2, MDIO_CTRL1, ret);
if (ret < 0)
- return ret;
+ goto out;
ret = xpcs_read_vpcs(xpcs, MDIO_CTRL1);
if (ret < 0)
- return ret;
+ goto out;
- return xpcs_write_vpcs(xpcs, MDIO_CTRL1, ret | DW_USXGMII_RST);
+ ret = xpcs_write_vpcs(xpcs, MDIO_CTRL1, ret | DW_USXGMII_RST);
+ if (ret < 0)
+ goto out;
+
+ return;
+
+out:
+ pr_err("%s: XPCS access returned %pe\n", __func__, ERR_PTR(ret));
}
-static int _xpcs_config_aneg_c73(struct mdio_xpcs_args *xpcs)
+static int _xpcs_config_aneg_c73(struct dw_xpcs *xpcs,
+ const struct xpcs_compat *compat)
{
int ret, adv;
/* SR_AN_ADV3 */
adv = 0;
- if (phylink_test(xpcs->supported, 2500baseX_Full))
+ if (xpcs_linkmode_supported(compat, 2500baseX_Full))
adv |= DW_C73_2500KX;
/* TODO: 5000baseKR */
/* SR_AN_ADV2 */
adv = 0;
- if (phylink_test(xpcs->supported, 1000baseKX_Full))
+ if (xpcs_linkmode_supported(compat, 1000baseKX_Full))
adv |= DW_C73_1000KX;
- if (phylink_test(xpcs->supported, 10000baseKX4_Full))
+ if (xpcs_linkmode_supported(compat, 10000baseKX4_Full))
adv |= DW_C73_10000KX4;
- if (phylink_test(xpcs->supported, 10000baseKR_Full))
+ if (xpcs_linkmode_supported(compat, 10000baseKR_Full))
adv |= DW_C73_10000KR;
ret = xpcs_write(xpcs, MDIO_MMD_AN, DW_SR_AN_ADV2, adv);
/* SR_AN_ADV1 */
adv = DW_C73_AN_ADV_SF;
- if (phylink_test(xpcs->supported, Pause))
+ if (xpcs_linkmode_supported(compat, Pause))
adv |= DW_C73_PAUSE;
- if (phylink_test(xpcs->supported, Asym_Pause))
+ if (xpcs_linkmode_supported(compat, Asym_Pause))
adv |= DW_C73_ASYM_PAUSE;
return xpcs_write(xpcs, MDIO_MMD_AN, DW_SR_AN_ADV1, adv);
}
-static int xpcs_config_aneg_c73(struct mdio_xpcs_args *xpcs)
+static int xpcs_config_aneg_c73(struct dw_xpcs *xpcs,
+ const struct xpcs_compat *compat)
{
int ret;
- ret = _xpcs_config_aneg_c73(xpcs);
+ ret = _xpcs_config_aneg_c73(xpcs, compat);
if (ret < 0)
return ret;
return xpcs_write(xpcs, MDIO_MMD_AN, MDIO_CTRL1, ret);
}
-static int xpcs_aneg_done_c73(struct mdio_xpcs_args *xpcs,
- struct phylink_link_state *state)
+static int xpcs_aneg_done_c73(struct dw_xpcs *xpcs,
+ struct phylink_link_state *state,
+ const struct xpcs_compat *compat)
{
int ret;
/* Check if Aneg outcome is valid */
if (!(ret & DW_C73_AN_ADV_SF)) {
- xpcs_config_aneg_c73(xpcs);
+ xpcs_config_aneg_c73(xpcs, compat);
return 0;
}
return 0;
}
-static int xpcs_read_lpa_c73(struct mdio_xpcs_args *xpcs,
+static int xpcs_read_lpa_c73(struct dw_xpcs *xpcs,
struct phylink_link_state *state)
{
int ret;
return 0;
}
-static void xpcs_resolve_lpa_c73(struct mdio_xpcs_args *xpcs,
+static void xpcs_resolve_lpa_c73(struct dw_xpcs *xpcs,
struct phylink_link_state *state)
{
int max_speed = xpcs_get_max_usxgmii_speed(state->lp_advertising);
state->duplex = DUPLEX_FULL;
}
-static int xpcs_get_max_xlgmii_speed(struct mdio_xpcs_args *xpcs,
+static int xpcs_get_max_xlgmii_speed(struct dw_xpcs *xpcs,
struct phylink_link_state *state)
{
unsigned long *adv = state->advertising;
return speed;
}
-static void xpcs_resolve_pma(struct mdio_xpcs_args *xpcs,
+static void xpcs_resolve_pma(struct dw_xpcs *xpcs,
struct phylink_link_state *state)
{
state->pause = MLO_PAUSE_TX | MLO_PAUSE_RX;
}
}
-static int xpcs_validate(struct mdio_xpcs_args *xpcs,
- unsigned long *supported,
- struct phylink_link_state *state)
+void xpcs_validate(struct dw_xpcs *xpcs, unsigned long *supported,
+ struct phylink_link_state *state)
{
- linkmode_and(supported, supported, xpcs->supported);
- linkmode_and(state->advertising, state->advertising, xpcs->supported);
- return 0;
+ __ETHTOOL_DECLARE_LINK_MODE_MASK(xpcs_supported);
+ const struct xpcs_compat *compat;
+ int i;
+
+ /* phylink expects us to report all supported modes with
+ * PHY_INTERFACE_MODE_NA, just don't limit the supported and
+ * advertising masks and exit.
+ */
+ if (state->interface == PHY_INTERFACE_MODE_NA)
+ return;
+
+ bitmap_zero(xpcs_supported, __ETHTOOL_LINK_MODE_MASK_NBITS);
+
+ compat = xpcs_find_compat(xpcs->id, state->interface);
+
+ /* Populate the supported link modes for this
+ * PHY interface type
+ */
+ if (compat)
+ for (i = 0; compat->supported[i] != __ETHTOOL_LINK_MODE_MASK_NBITS; i++)
+ set_bit(compat->supported[i], xpcs_supported);
+
+ linkmode_and(supported, supported, xpcs_supported);
+ linkmode_and(state->advertising, state->advertising, xpcs_supported);
}
+EXPORT_SYMBOL_GPL(xpcs_validate);
-static int xpcs_config_aneg_c37_sgmii(struct mdio_xpcs_args *xpcs)
+int xpcs_config_eee(struct dw_xpcs *xpcs, int mult_fact_100ns, int enable)
+{
+ int ret;
+
+ if (enable) {
+ /* Enable EEE */
+ ret = DW_VR_MII_EEE_LTX_EN | DW_VR_MII_EEE_LRX_EN |
+ DW_VR_MII_EEE_TX_QUIET_EN | DW_VR_MII_EEE_RX_QUIET_EN |
+ DW_VR_MII_EEE_TX_EN_CTRL | DW_VR_MII_EEE_RX_EN_CTRL |
+ mult_fact_100ns << DW_VR_MII_EEE_MULT_FACT_100NS_SHIFT;
+ } else {
+ ret = xpcs_read(xpcs, MDIO_MMD_VEND2, DW_VR_MII_EEE_MCTRL0);
+ if (ret < 0)
+ return ret;
+ ret &= ~(DW_VR_MII_EEE_LTX_EN | DW_VR_MII_EEE_LRX_EN |
+ DW_VR_MII_EEE_TX_QUIET_EN | DW_VR_MII_EEE_RX_QUIET_EN |
+ DW_VR_MII_EEE_TX_EN_CTRL | DW_VR_MII_EEE_RX_EN_CTRL |
+ DW_VR_MII_EEE_MULT_FACT_100NS);
+ }
+
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, DW_VR_MII_EEE_MCTRL0, ret);
+ if (ret < 0)
+ return ret;
+
+ ret = xpcs_read(xpcs, MDIO_MMD_VEND2, DW_VR_MII_EEE_MCTRL1);
+ if (ret < 0)
+ return ret;
+
+ ret |= DW_VR_MII_EEE_TRN_LPI;
+ return xpcs_write(xpcs, MDIO_MMD_VEND2, DW_VR_MII_EEE_MCTRL1, ret);
+}
+EXPORT_SYMBOL_GPL(xpcs_config_eee);
+
+static int xpcs_config_aneg_c37_sgmii(struct dw_xpcs *xpcs, unsigned int mode)
{
int ret;
if (ret < 0)
return ret;
- ret |= DW_VR_MII_DIG_CTRL1_MAC_AUTO_SW;
+ if (phylink_autoneg_inband(mode))
+ ret |= DW_VR_MII_DIG_CTRL1_MAC_AUTO_SW;
+ else
+ ret &= ~DW_VR_MII_DIG_CTRL1_MAC_AUTO_SW;
return xpcs_write(xpcs, MDIO_MMD_VEND2, DW_VR_MII_DIG_CTRL1, ret);
}
-static int xpcs_config(struct mdio_xpcs_args *xpcs,
- const struct phylink_link_state *state)
+static int xpcs_config_2500basex(struct dw_xpcs *xpcs)
+{
+ int ret;
+
+ ret = xpcs_read(xpcs, MDIO_MMD_VEND2, DW_VR_MII_DIG_CTRL1);
+ if (ret < 0)
+ return ret;
+ ret |= DW_VR_MII_DIG_CTRL1_2G5_EN;
+ ret &= ~DW_VR_MII_DIG_CTRL1_MAC_AUTO_SW;
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, DW_VR_MII_DIG_CTRL1, ret);
+ if (ret < 0)
+ return ret;
+
+ ret = xpcs_read(xpcs, MDIO_MMD_VEND2, DW_VR_MII_MMD_CTRL);
+ if (ret < 0)
+ return ret;
+ ret &= ~AN_CL37_EN;
+ ret |= SGMII_SPEED_SS6;
+ ret &= ~SGMII_SPEED_SS13;
+ return xpcs_write(xpcs, MDIO_MMD_VEND2, DW_VR_MII_MMD_CTRL, ret);
+}
+
+int xpcs_do_config(struct dw_xpcs *xpcs, phy_interface_t interface,
+ unsigned int mode)
{
+ const struct xpcs_compat *compat;
int ret;
- switch (xpcs->an_mode) {
+ compat = xpcs_find_compat(xpcs->id, interface);
+ if (!compat)
+ return -ENODEV;
+
+ switch (compat->an_mode) {
case DW_AN_C73:
- if (state->an_enabled) {
- ret = xpcs_config_aneg_c73(xpcs);
+ if (phylink_autoneg_inband(mode)) {
+ ret = xpcs_config_aneg_c73(xpcs, compat);
if (ret)
return ret;
}
break;
case DW_AN_C37_SGMII:
- ret = xpcs_config_aneg_c37_sgmii(xpcs);
+ ret = xpcs_config_aneg_c37_sgmii(xpcs, mode);
+ if (ret)
+ return ret;
+ break;
+ case DW_2500BASEX:
+ ret = xpcs_config_2500basex(xpcs);
if (ret)
return ret;
break;
return -1;
}
+ if (compat->pma_config) {
+ ret = compat->pma_config(xpcs);
+ if (ret)
+ return ret;
+ }
+
return 0;
}
+EXPORT_SYMBOL_GPL(xpcs_do_config);
+
+static int xpcs_config(struct phylink_pcs *pcs, unsigned int mode,
+ phy_interface_t interface,
+ const unsigned long *advertising,
+ bool permit_pause_to_mac)
+{
+ struct dw_xpcs *xpcs = phylink_pcs_to_xpcs(pcs);
+
+ return xpcs_do_config(xpcs, interface, mode);
+}
-static int xpcs_get_state_c73(struct mdio_xpcs_args *xpcs,
- struct phylink_link_state *state)
+static int xpcs_get_state_c73(struct dw_xpcs *xpcs,
+ struct phylink_link_state *state,
+ const struct xpcs_compat *compat)
{
int ret;
/* ... and then we check the faults. */
ret = xpcs_read_fault_c73(xpcs, state);
if (ret) {
- ret = xpcs_soft_reset(xpcs);
+ ret = xpcs_soft_reset(xpcs, compat);
if (ret)
return ret;
state->link = 0;
- return xpcs_config(xpcs, state);
+ return xpcs_do_config(xpcs, state->interface, MLO_AN_INBAND);
}
- if (state->an_enabled && xpcs_aneg_done_c73(xpcs, state)) {
+ if (state->an_enabled && xpcs_aneg_done_c73(xpcs, state, compat)) {
state->an_complete = true;
xpcs_read_lpa_c73(xpcs, state);
xpcs_resolve_lpa_c73(xpcs, state);
return 0;
}
-static int xpcs_get_state_c37_sgmii(struct mdio_xpcs_args *xpcs,
+static int xpcs_get_state_c37_sgmii(struct dw_xpcs *xpcs,
struct phylink_link_state *state)
{
int ret;
return 0;
}
-static int xpcs_get_state(struct mdio_xpcs_args *xpcs,
- struct phylink_link_state *state)
+static void xpcs_get_state(struct phylink_pcs *pcs,
+ struct phylink_link_state *state)
{
+ struct dw_xpcs *xpcs = phylink_pcs_to_xpcs(pcs);
+ const struct xpcs_compat *compat;
int ret;
- switch (xpcs->an_mode) {
+ compat = xpcs_find_compat(xpcs->id, state->interface);
+ if (!compat)
+ return;
+
+ switch (compat->an_mode) {
case DW_AN_C73:
- ret = xpcs_get_state_c73(xpcs, state);
- if (ret)
- return ret;
+ ret = xpcs_get_state_c73(xpcs, state, compat);
+ if (ret) {
+ pr_err("xpcs_get_state_c73 returned %pe\n",
+ ERR_PTR(ret));
+ return;
+ }
break;
case DW_AN_C37_SGMII:
ret = xpcs_get_state_c37_sgmii(xpcs, state);
- if (ret)
- return ret;
+ if (ret) {
+ pr_err("xpcs_get_state_c37_sgmii returned %pe\n",
+ ERR_PTR(ret));
+ }
break;
default:
- return -1;
+ return;
}
+}
- return 0;
+static void xpcs_link_up_sgmii(struct dw_xpcs *xpcs, unsigned int mode,
+ int speed, int duplex)
+{
+ int val, ret;
+
+ if (phylink_autoneg_inband(mode))
+ return;
+
+ switch (speed) {
+ case SPEED_1000:
+ val = BMCR_SPEED1000;
+ break;
+ case SPEED_100:
+ val = BMCR_SPEED100;
+ break;
+ case SPEED_10:
+ val = BMCR_SPEED10;
+ break;
+ default:
+ return;
+ }
+
+ if (duplex == DUPLEX_FULL)
+ val |= BMCR_FULLDPLX;
+
+ ret = xpcs_write(xpcs, MDIO_MMD_VEND2, MDIO_CTRL1, val);
+ if (ret)
+ pr_err("%s: xpcs_write returned %pe\n", __func__, ERR_PTR(ret));
}
-static int xpcs_link_up(struct mdio_xpcs_args *xpcs, int speed,
- phy_interface_t interface)
+void xpcs_link_up(struct phylink_pcs *pcs, unsigned int mode,
+ phy_interface_t interface, int speed, int duplex)
{
+ struct dw_xpcs *xpcs = phylink_pcs_to_xpcs(pcs);
+
if (interface == PHY_INTERFACE_MODE_USXGMII)
return xpcs_config_usxgmii(xpcs, speed);
-
- return 0;
+ if (interface == PHY_INTERFACE_MODE_SGMII)
+ return xpcs_link_up_sgmii(xpcs, mode, speed, duplex);
}
+EXPORT_SYMBOL_GPL(xpcs_link_up);
-static u32 xpcs_get_id(struct mdio_xpcs_args *xpcs)
+static u32 xpcs_get_id(struct dw_xpcs *xpcs)
{
int ret;
u32 id;
if (ret < 0)
return 0xffffffff;
- /* If Device IDs are not all zeros, we found C73 AN-type device */
- if (id | ret)
+ /* If Device IDs are not all zeros or all ones,
+ * we found C73 AN-type device
+ */
+ if ((id | ret) && (id | ret) != 0xffffffff)
return id | ret;
/* Next, search C37 PCS using Vendor-Specific MII MMD */
return 0xffffffff;
}
-static bool xpcs_check_features(struct mdio_xpcs_args *xpcs,
- struct xpcs_id *match,
- phy_interface_t interface)
-{
- int i;
-
- for (i = 0; match->interface[i] != PHY_INTERFACE_MODE_MAX; i++) {
- if (match->interface[i] == interface)
- break;
- }
+static const struct xpcs_compat synopsys_xpcs_compat[DW_XPCS_INTERFACE_MAX] = {
+ [DW_XPCS_USXGMII] = {
+ .supported = xpcs_usxgmii_features,
+ .interface = xpcs_usxgmii_interfaces,
+ .num_interfaces = ARRAY_SIZE(xpcs_usxgmii_interfaces),
+ .an_mode = DW_AN_C73,
+ },
+ [DW_XPCS_10GKR] = {
+ .supported = xpcs_10gkr_features,
+ .interface = xpcs_10gkr_interfaces,
+ .num_interfaces = ARRAY_SIZE(xpcs_10gkr_interfaces),
+ .an_mode = DW_AN_C73,
+ },
+ [DW_XPCS_XLGMII] = {
+ .supported = xpcs_xlgmii_features,
+ .interface = xpcs_xlgmii_interfaces,
+ .num_interfaces = ARRAY_SIZE(xpcs_xlgmii_interfaces),
+ .an_mode = DW_AN_C73,
+ },
+ [DW_XPCS_SGMII] = {
+ .supported = xpcs_sgmii_features,
+ .interface = xpcs_sgmii_interfaces,
+ .num_interfaces = ARRAY_SIZE(xpcs_sgmii_interfaces),
+ .an_mode = DW_AN_C37_SGMII,
+ },
+ [DW_XPCS_2500BASEX] = {
+ .supported = xpcs_2500basex_features,
+ .interface = xpcs_2500basex_interfaces,
+ .num_interfaces = ARRAY_SIZE(xpcs_2500basex_features),
+ .an_mode = DW_2500BASEX,
+ },
+};
- if (match->interface[i] == PHY_INTERFACE_MODE_MAX)
- return false;
+static const struct xpcs_compat nxp_sja1105_xpcs_compat[DW_XPCS_INTERFACE_MAX] = {
+ [DW_XPCS_SGMII] = {
+ .supported = xpcs_sgmii_features,
+ .interface = xpcs_sgmii_interfaces,
+ .num_interfaces = ARRAY_SIZE(xpcs_sgmii_interfaces),
+ .an_mode = DW_AN_C37_SGMII,
+ .pma_config = nxp_sja1105_sgmii_pma_config,
+ },
+};
- for (i = 0; match->supported[i] != __ETHTOOL_LINK_MODE_MASK_NBITS; i++)
- set_bit(match->supported[i], xpcs->supported);
+static const struct xpcs_compat nxp_sja1110_xpcs_compat[DW_XPCS_INTERFACE_MAX] = {
+ [DW_XPCS_SGMII] = {
+ .supported = xpcs_sgmii_features,
+ .interface = xpcs_sgmii_interfaces,
+ .num_interfaces = ARRAY_SIZE(xpcs_sgmii_interfaces),
+ .an_mode = DW_AN_C37_SGMII,
+ .pma_config = nxp_sja1110_sgmii_pma_config,
+ },
+ [DW_XPCS_2500BASEX] = {
+ .supported = xpcs_2500basex_features,
+ .interface = xpcs_2500basex_interfaces,
+ .num_interfaces = ARRAY_SIZE(xpcs_2500basex_interfaces),
+ .an_mode = DW_2500BASEX,
+ .pma_config = nxp_sja1110_2500basex_pma_config,
+ },
+};
- xpcs->an_mode = match->an_mode;
+static const struct xpcs_id xpcs_id_list[] = {
+ {
+ .id = SYNOPSYS_XPCS_ID,
+ .mask = SYNOPSYS_XPCS_MASK,
+ .compat = synopsys_xpcs_compat,
+ }, {
+ .id = NXP_SJA1105_XPCS_ID,
+ .mask = SYNOPSYS_XPCS_MASK,
+ .compat = nxp_sja1105_xpcs_compat,
+ }, {
+ .id = NXP_SJA1110_XPCS_ID,
+ .mask = SYNOPSYS_XPCS_MASK,
+ .compat = nxp_sja1110_xpcs_compat,
+ },
+};
- return true;
-}
+static const struct phylink_pcs_ops xpcs_phylink_ops = {
+ .pcs_config = xpcs_config,
+ .pcs_get_state = xpcs_get_state,
+ .pcs_link_up = xpcs_link_up,
+};
-static int xpcs_probe(struct mdio_xpcs_args *xpcs, phy_interface_t interface)
+struct dw_xpcs *xpcs_create(struct mdio_device *mdiodev,
+ phy_interface_t interface)
{
- u32 xpcs_id = xpcs_get_id(xpcs);
- struct xpcs_id *match = NULL;
- int i;
+ struct dw_xpcs *xpcs;
+ u32 xpcs_id;
+ int i, ret;
+
+ xpcs = kzalloc(sizeof(*xpcs), GFP_KERNEL);
+ if (!xpcs)
+ return NULL;
+
+ xpcs->mdiodev = mdiodev;
+
+ xpcs_id = xpcs_get_id(xpcs);
for (i = 0; i < ARRAY_SIZE(xpcs_id_list); i++) {
- struct xpcs_id *entry = &xpcs_id_list[i];
+ const struct xpcs_id *entry = &xpcs_id_list[i];
+ const struct xpcs_compat *compat;
+
+ if ((xpcs_id & entry->mask) != entry->id)
+ continue;
- if ((xpcs_id & entry->mask) == entry->id) {
- match = entry;
+ xpcs->id = entry;
- if (xpcs_check_features(xpcs, match, interface))
- return xpcs_soft_reset(xpcs);
+ compat = xpcs_find_compat(entry, interface);
+ if (!compat) {
+ ret = -ENODEV;
+ goto out;
}
+
+ xpcs->pcs.ops = &xpcs_phylink_ops;
+ xpcs->pcs.poll = true;
+
+ ret = xpcs_soft_reset(xpcs, compat);
+ if (ret)
+ goto out;
+
+ return xpcs;
}
- return -ENODEV;
-}
+ ret = -ENODEV;
-static struct mdio_xpcs_ops xpcs_ops = {
- .validate = xpcs_validate,
- .config = xpcs_config,
- .get_state = xpcs_get_state,
- .link_up = xpcs_link_up,
- .probe = xpcs_probe,
-};
+out:
+ kfree(xpcs);
+
+ return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(xpcs_create);
-struct mdio_xpcs_ops *mdio_xpcs_get_ops(void)
+void xpcs_destroy(struct dw_xpcs *xpcs)
{
- return &xpcs_ops;
+ kfree(xpcs);
}
-EXPORT_SYMBOL_GPL(mdio_xpcs_get_ops);
+EXPORT_SYMBOL_GPL(xpcs_destroy);
MODULE_LICENSE("GPL v2");
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2020 Synopsys, Inc. and/or its affiliates.
+ * Synopsys DesignWare XPCS helpers
+ *
+ * Author: Jose Abreu <Jose.Abreu@synopsys.com>
+ */
+
+#define SYNOPSYS_XPCS_ID 0x7996ced0
+#define SYNOPSYS_XPCS_MASK 0xffffffff
+
+/* Vendor regs access */
+#define DW_VENDOR BIT(15)
+
+/* VR_XS_PCS */
+#define DW_USXGMII_RST BIT(10)
+#define DW_USXGMII_EN BIT(9)
+#define DW_VR_XS_PCS_DIG_STS 0x0010
+#define DW_RXFIFO_ERR GENMASK(6, 5)
+
+/* SR_MII */
+#define DW_USXGMII_FULL BIT(8)
+#define DW_USXGMII_SS_MASK (BIT(13) | BIT(6) | BIT(5))
+#define DW_USXGMII_10000 (BIT(13) | BIT(6))
+#define DW_USXGMII_5000 (BIT(13) | BIT(5))
+#define DW_USXGMII_2500 (BIT(5))
+#define DW_USXGMII_1000 (BIT(6))
+#define DW_USXGMII_100 (BIT(13))
+#define DW_USXGMII_10 (0)
+
+/* SR_AN */
+#define DW_SR_AN_ADV1 0x10
+#define DW_SR_AN_ADV2 0x11
+#define DW_SR_AN_ADV3 0x12
+#define DW_SR_AN_LP_ABL1 0x13
+#define DW_SR_AN_LP_ABL2 0x14
+#define DW_SR_AN_LP_ABL3 0x15
+
+/* Clause 73 Defines */
+/* AN_LP_ABL1 */
+#define DW_C73_PAUSE BIT(10)
+#define DW_C73_ASYM_PAUSE BIT(11)
+#define DW_C73_AN_ADV_SF 0x1
+/* AN_LP_ABL2 */
+#define DW_C73_1000KX BIT(5)
+#define DW_C73_10000KX4 BIT(6)
+#define DW_C73_10000KR BIT(7)
+/* AN_LP_ABL3 */
+#define DW_C73_2500KX BIT(0)
+#define DW_C73_5000KR BIT(1)
+
+/* Clause 37 Defines */
+/* VR MII MMD registers offsets */
+#define DW_VR_MII_MMD_CTRL 0x0000
+#define DW_VR_MII_DIG_CTRL1 0x8000
+#define DW_VR_MII_AN_CTRL 0x8001
+#define DW_VR_MII_AN_INTR_STS 0x8002
+/* Enable 2.5G Mode */
+#define DW_VR_MII_DIG_CTRL1_2G5_EN BIT(2)
+/* EEE Mode Control Register */
+#define DW_VR_MII_EEE_MCTRL0 0x8006
+#define DW_VR_MII_EEE_MCTRL1 0x800b
+#define DW_VR_MII_DIG_CTRL2 0x80e1
+
+/* VR_MII_DIG_CTRL1 */
+#define DW_VR_MII_DIG_CTRL1_MAC_AUTO_SW BIT(9)
+
+/* VR_MII_DIG_CTRL2 */
+#define DW_VR_MII_DIG_CTRL2_TX_POL_INV BIT(4)
+#define DW_VR_MII_DIG_CTRL2_RX_POL_INV BIT(0)
+
+/* VR_MII_AN_CTRL */
+#define DW_VR_MII_AN_CTRL_TX_CONFIG_SHIFT 3
+#define DW_VR_MII_TX_CONFIG_MASK BIT(3)
+#define DW_VR_MII_TX_CONFIG_PHY_SIDE_SGMII 0x1
+#define DW_VR_MII_TX_CONFIG_MAC_SIDE_SGMII 0x0
+#define DW_VR_MII_AN_CTRL_PCS_MODE_SHIFT 1
+#define DW_VR_MII_PCS_MODE_MASK GENMASK(2, 1)
+#define DW_VR_MII_PCS_MODE_C37_1000BASEX 0x0
+#define DW_VR_MII_PCS_MODE_C37_SGMII 0x2
+
+/* VR_MII_AN_INTR_STS */
+#define DW_VR_MII_AN_STS_C37_ANSGM_FD BIT(1)
+#define DW_VR_MII_AN_STS_C37_ANSGM_SP_SHIFT 2
+#define DW_VR_MII_AN_STS_C37_ANSGM_SP GENMASK(3, 2)
+#define DW_VR_MII_C37_ANSGM_SP_10 0x0
+#define DW_VR_MII_C37_ANSGM_SP_100 0x1
+#define DW_VR_MII_C37_ANSGM_SP_1000 0x2
+#define DW_VR_MII_C37_ANSGM_SP_LNKSTS BIT(4)
+
+/* SR MII MMD Control defines */
+#define AN_CL37_EN BIT(12) /* Enable Clause 37 auto-nego */
+#define SGMII_SPEED_SS13 BIT(13) /* SGMII speed along with SS6 */
+#define SGMII_SPEED_SS6 BIT(6) /* SGMII speed along with SS13 */
+
+/* VR MII EEE Control 0 defines */
+#define DW_VR_MII_EEE_LTX_EN BIT(0) /* LPI Tx Enable */
+#define DW_VR_MII_EEE_LRX_EN BIT(1) /* LPI Rx Enable */
+#define DW_VR_MII_EEE_TX_QUIET_EN BIT(2) /* Tx Quiet Enable */
+#define DW_VR_MII_EEE_RX_QUIET_EN BIT(3) /* Rx Quiet Enable */
+#define DW_VR_MII_EEE_TX_EN_CTRL BIT(4) /* Tx Control Enable */
+#define DW_VR_MII_EEE_RX_EN_CTRL BIT(7) /* Rx Control Enable */
+
+#define DW_VR_MII_EEE_MULT_FACT_100NS_SHIFT 8
+#define DW_VR_MII_EEE_MULT_FACT_100NS GENMASK(11, 8)
+
+/* VR MII EEE Control 1 defines */
+#define DW_VR_MII_EEE_TRN_LPI BIT(0) /* Transparent Mode Enable */
+
+int xpcs_read(struct dw_xpcs *xpcs, int dev, u32 reg);
+int xpcs_write(struct dw_xpcs *xpcs, int dev, u32 reg, u16 val);
+
+int nxp_sja1105_sgmii_pma_config(struct dw_xpcs *xpcs);
+int nxp_sja1110_sgmii_pma_config(struct dw_xpcs *xpcs);
+int nxp_sja1110_2500basex_pma_config(struct dw_xpcs *xpcs);
Support for the Marvell 88X2222 Dual-port Multi-speed Ethernet
Transceiver.
+config MEDIATEK_GE_PHY
+ tristate "MediaTek Gigabit Ethernet PHYs"
+ help
+ Supports the MediaTek Gigabit Ethernet PHYs.
+
config MICREL_PHY
tristate "Micrel PHYs"
help
help
Currently supports VSC8514, VSC8530, VSC8531, VSC8540 and VSC8541 PHYs
+config MOTORCOMM_PHY
+ tristate "Motorcomm PHYs"
+ help
+ Enables support for Motorcomm network PHYs.
+ Currently supports the YT8511 gigabit PHY.
+
config NATIONAL_PHY
tristate "National Semiconductor PHYs"
help
Currently supports the NXP TJA1100 and TJA1101 PHY.
config AT803X_PHY
- tristate "Qualcomm Atheros AR803X PHYs"
+ tristate "Qualcomm Atheros AR803X PHYs and QCA833x PHYs"
depends on REGULATOR
help
- Currently supports the AR8030, AR8031, AR8033 and AR8035 model
+ Currently supports the AR8030, AR8031, AR8033, AR8035 and internal
+ QCA8337(Internal qca8k PHY) model
config QSEMI_PHY
tristate "Quality Semiconductor PHYs"
obj-$(CONFIG_MARVELL_10G_PHY) += marvell10g.o
obj-$(CONFIG_MARVELL_PHY) += marvell.o
obj-$(CONFIG_MARVELL_88X2222_PHY) += marvell-88x2222.o
+obj-$(CONFIG_MEDIATEK_GE_PHY) += mediatek-ge.o
obj-$(CONFIG_MESON_GXL_PHY) += meson-gxl.o
obj-$(CONFIG_MICREL_KS8995MA) += spi_ks8995.o
obj-$(CONFIG_MICREL_PHY) += micrel.o
obj-$(CONFIG_MICROCHIP_PHY) += microchip.o
obj-$(CONFIG_MICROCHIP_T1_PHY) += microchip_t1.o
obj-$(CONFIG_MICROSEMI_PHY) += mscc/
+obj-$(CONFIG_MOTORCOMM_PHY) += motorcomm.o
obj-$(CONFIG_NATIONAL_PHY) += national.o
obj-$(CONFIG_NXP_C45_TJA11XX_PHY) += nxp-c45-tja11xx.o
obj-$(CONFIG_NXP_TJA11XX_PHY) += nxp-tja11xx.o
// SPDX-License-Identifier: GPL-2.0+
-/**
+/*
* Driver for Analog Devices Industrial Ethernet PHYs
*
* Copyright 2019 Analog Devices Inc.
#define AT803X_MODE_CFG_MASK 0x0F
#define AT803X_MODE_CFG_SGMII 0x01
-#define AT803X_PSSR 0x11 /*PHY-Specific Status Register*/
-#define AT803X_PSSR_MR_AN_COMPLETE 0x0200
+#define AT803X_PSSR 0x11 /*PHY-Specific Status Register*/
+#define AT803X_PSSR_MR_AN_COMPLETE 0x0200
#define AT803X_DEBUG_REG_0 0x00
#define AT803X_DEBUG_RX_CLK_DLY_EN BIT(15)
#define AT803X_DEBUG_REG_5 0x05
#define AT803X_DEBUG_TX_CLK_DLY_EN BIT(8)
+#define AT803X_DEBUG_REG_3C 0x3C
+
+#define AT803X_DEBUG_REG_3D 0x3D
+
#define AT803X_DEBUG_REG_1F 0x1F
#define AT803X_DEBUG_PLL_ON BIT(2)
#define AT803X_DEBUG_RGMII_1V8 BIT(3)
+#define MDIO_AZ_DEBUG 0x800D
+
/* AT803x supports either the XTAL input pad, an internal PLL or the
* DSP as clock reference for the clock output pad. The XTAL reference
* is only used for 25 MHz output, all other frequencies need the PLL.
#define AT803X_CLK_OUT_STRENGTH_HALF 1
#define AT803X_CLK_OUT_STRENGTH_QUARTER 2
-#define AT803X_DEFAULT_DOWNSHIFT 5
-#define AT803X_MIN_DOWNSHIFT 2
-#define AT803X_MAX_DOWNSHIFT 9
+#define AT803X_DEFAULT_DOWNSHIFT 5
+#define AT803X_MIN_DOWNSHIFT 2
+#define AT803X_MAX_DOWNSHIFT 9
#define AT803X_MMD3_SMARTEEE_CTL1 0x805b
#define AT803X_MMD3_SMARTEEE_CTL2 0x805c
#define AT803X_MMD3_SMARTEEE_CTL3 0x805d
#define AT803X_MMD3_SMARTEEE_CTL3_LPI_EN BIT(8)
-#define ATH9331_PHY_ID 0x004dd041
-#define ATH8030_PHY_ID 0x004dd076
-#define ATH8031_PHY_ID 0x004dd074
-#define ATH8032_PHY_ID 0x004dd023
-#define ATH8035_PHY_ID 0x004dd072
+#define ATH9331_PHY_ID 0x004dd041
+#define ATH8030_PHY_ID 0x004dd076
+#define ATH8031_PHY_ID 0x004dd074
+#define ATH8032_PHY_ID 0x004dd023
+#define ATH8035_PHY_ID 0x004dd072
#define AT8030_PHY_ID_MASK 0xffffffef
-#define AT803X_PAGE_FIBER 0
-#define AT803X_PAGE_COPPER 1
+#define QCA8327_PHY_ID 0x004dd034
+#define QCA8337_PHY_ID 0x004dd036
+#define QCA8K_PHY_ID_MASK 0xffffffff
+
+#define QCA8K_DEVFLAGS_REVISION_MASK GENMASK(2, 0)
+
+#define AT803X_PAGE_FIBER 0
+#define AT803X_PAGE_COPPER 1
+
+/* don't turn off internal PLL */
+#define AT803X_KEEP_PLL_ENABLED BIT(0)
+#define AT803X_DISABLE_SMARTEEE BIT(1)
MODULE_DESCRIPTION("Qualcomm Atheros AR803x PHY driver");
MODULE_AUTHOR("Matus Ujhelyi");
MODULE_LICENSE("GPL");
+enum stat_access_type {
+ PHY,
+ MMD
+};
+
+struct at803x_hw_stat {
+ const char *string;
+ u8 reg;
+ u32 mask;
+ enum stat_access_type access_type;
+};
+
+static struct at803x_hw_stat at803x_hw_stats[] = {
+ { "phy_idle_errors", 0xa, GENMASK(7, 0), PHY},
+ { "phy_receive_errors", 0x15, GENMASK(15, 0), PHY},
+ { "eee_wake_errors", 0x16, GENMASK(15, 0), MMD},
+};
+
struct at803x_priv {
int flags;
-#define AT803X_KEEP_PLL_ENABLED BIT(0) /* don't turn off internal PLL */
-#define AT803X_DISABLE_SMARTEEE BIT(1)
u16 clk_25m_reg;
u16 clk_25m_mask;
u8 smarteee_lpi_tw_1g;
struct regulator_dev *vddio_rdev;
struct regulator_dev *vddh_rdev;
struct regulator *vddio;
+ u64 stats[ARRAY_SIZE(at803x_hw_stats)];
};
struct at803x_context {
u16 led_control;
};
+static int at803x_debug_reg_write(struct phy_device *phydev, u16 reg, u16 data)
+{
+ int ret;
+
+ ret = phy_write(phydev, AT803X_DEBUG_ADDR, reg);
+ if (ret < 0)
+ return ret;
+
+ return phy_write(phydev, AT803X_DEBUG_DATA, data);
+}
+
static int at803x_debug_reg_read(struct phy_device *phydev, u16 reg)
{
int ret;
wol->wolopts |= WAKE_MAGIC;
}
+static int at803x_get_sset_count(struct phy_device *phydev)
+{
+ return ARRAY_SIZE(at803x_hw_stats);
+}
+
+static void at803x_get_strings(struct phy_device *phydev, u8 *data)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(at803x_hw_stats); i++) {
+ strscpy(data + i * ETH_GSTRING_LEN,
+ at803x_hw_stats[i].string, ETH_GSTRING_LEN);
+ }
+}
+
+static u64 at803x_get_stat(struct phy_device *phydev, int i)
+{
+ struct at803x_hw_stat stat = at803x_hw_stats[i];
+ struct at803x_priv *priv = phydev->priv;
+ int val;
+ u64 ret;
+
+ if (stat.access_type == MMD)
+ val = phy_read_mmd(phydev, MDIO_MMD_PCS, stat.reg);
+ else
+ val = phy_read(phydev, stat.reg);
+
+ if (val < 0) {
+ ret = U64_MAX;
+ } else {
+ val = val & stat.mask;
+ priv->stats[i] += val;
+ ret = priv->stats[i];
+ }
+
+ return ret;
+}
+
+static void at803x_get_stats(struct phy_device *phydev,
+ struct ethtool_stats *stats, u64 *data)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(at803x_hw_stats); i++)
+ data[i] = at803x_get_stat(phydev, i);
+}
+
static int at803x_suspend(struct phy_device *phydev)
{
int value;
return 0;
}
+static int qca83xx_config_init(struct phy_device *phydev)
+{
+ u8 switch_revision;
+
+ switch_revision = phydev->dev_flags & QCA8K_DEVFLAGS_REVISION_MASK;
+
+ switch (switch_revision) {
+ case 1:
+ /* For 100M waveform */
+ at803x_debug_reg_write(phydev, AT803X_DEBUG_REG_0, 0x02ea);
+ /* Turn on Gigabit clock */
+ at803x_debug_reg_write(phydev, AT803X_DEBUG_REG_3D, 0x68a0);
+ break;
+
+ case 2:
+ phy_write_mmd(phydev, MDIO_MMD_AN, MDIO_AN_EEE_ADV, 0x0);
+ fallthrough;
+ case 4:
+ phy_write_mmd(phydev, MDIO_MMD_PCS, MDIO_AZ_DEBUG, 0x803f);
+ at803x_debug_reg_write(phydev, AT803X_DEBUG_REG_3D, 0x6860);
+ at803x_debug_reg_write(phydev, AT803X_DEBUG_REG_5, 0x2c46);
+ at803x_debug_reg_write(phydev, AT803X_DEBUG_REG_3C, 0x6000);
+ break;
+ }
+
+ return 0;
+}
+
static struct phy_driver at803x_driver[] = {
{
/* Qualcomm Atheros AR8035 */
.read_status = at803x_read_status,
.soft_reset = genphy_soft_reset,
.config_aneg = at803x_config_aneg,
-} };
+}, {
+ /* QCA8337 */
+ .phy_id = QCA8337_PHY_ID,
+ .phy_id_mask = QCA8K_PHY_ID_MASK,
+ .name = "QCA PHY 8337",
+ /* PHY_GBIT_FEATURES */
+ .probe = at803x_probe,
+ .flags = PHY_IS_INTERNAL,
+ .config_init = qca83xx_config_init,
+ .soft_reset = genphy_soft_reset,
+ .get_sset_count = at803x_get_sset_count,
+ .get_strings = at803x_get_strings,
+ .get_stats = at803x_get_stats,
+}, };
module_phy_driver(at803x_driver);
#include <linux/mii.h>
#include <linux/phy.h>
+#define PHY_ID_ASIX_AX88772A 0x003b1861
+#define PHY_ID_ASIX_AX88772C 0x003b1881
#define PHY_ID_ASIX_AX88796B 0x003b1841
MODULE_DESCRIPTION("Asix PHY driver");
return genphy_soft_reset(phydev);
}
-static struct phy_driver asix_driver[] = { {
+/* AX88772A is not working properly with some old switches (NETGEAR EN 108TP):
+ * after autoneg is done and the link status is reported as active, the MII_LPA
+ * register is 0. This issue is not reproducible on AX88772C.
+ */
+static int asix_ax88772a_read_status(struct phy_device *phydev)
+{
+ int ret, val;
+
+ ret = genphy_update_link(phydev);
+ if (ret)
+ return ret;
+
+ if (!phydev->link)
+ return 0;
+
+ /* If MII_LPA is 0, phy_resolve_aneg_linkmode() will fail to resolve
+ * linkmode so use MII_BMCR as default values.
+ */
+ val = phy_read(phydev, MII_BMCR);
+ if (val < 0)
+ return val;
+
+ if (val & BMCR_SPEED100)
+ phydev->speed = SPEED_100;
+ else
+ phydev->speed = SPEED_10;
+
+ if (val & BMCR_FULLDPLX)
+ phydev->duplex = DUPLEX_FULL;
+ else
+ phydev->duplex = DUPLEX_HALF;
+
+ ret = genphy_read_lpa(phydev);
+ if (ret < 0)
+ return ret;
+
+ if (phydev->autoneg == AUTONEG_ENABLE && phydev->autoneg_complete)
+ phy_resolve_aneg_linkmode(phydev);
+
+ return 0;
+}
+
+static void asix_ax88772a_link_change_notify(struct phy_device *phydev)
+{
+ /* Reset PHY, otherwise MII_LPA will provide outdated information.
+ * This issue is reproducible only with some link partner PHYs
+ */
+ if (phydev->state == PHY_NOLINK && phydev->drv->soft_reset)
+ phydev->drv->soft_reset(phydev);
+}
+
+static struct phy_driver asix_driver[] = {
+{
+ PHY_ID_MATCH_EXACT(PHY_ID_ASIX_AX88772A),
+ .name = "Asix Electronics AX88772A",
+ .flags = PHY_IS_INTERNAL,
+ .read_status = asix_ax88772a_read_status,
+ .suspend = genphy_suspend,
+ .resume = genphy_resume,
+ .soft_reset = asix_soft_reset,
+ .link_change_notify = asix_ax88772a_link_change_notify,
+}, {
+ PHY_ID_MATCH_EXACT(PHY_ID_ASIX_AX88772C),
+ .name = "Asix Electronics AX88772C",
+ .flags = PHY_IS_INTERNAL,
+ .suspend = genphy_suspend,
+ .resume = genphy_resume,
+ .soft_reset = asix_soft_reset,
+}, {
.phy_id = PHY_ID_ASIX_AX88796B,
.name = "Asix Electronics AX88796B",
.phy_id_mask = 0xfffffff0,
module_phy_driver(asix_driver);
static struct mdio_device_id __maybe_unused asix_tbl[] = {
+ { PHY_ID_MATCH_EXACT(PHY_ID_ASIX_AX88772A) },
+ { PHY_ID_MATCH_EXACT(PHY_ID_ASIX_AX88772C) },
{ PHY_ID_ASIX_AX88796B, 0xfffffff0 },
{ }
};
u16 reg = be32_to_cpup(paddr++);
u16 mask = be32_to_cpup(paddr++);
u16 val_bits = be32_to_cpup(paddr++);
- int val;
u32 regnum = mdiobus_c45_addr(devid, reg);
- val = 0;
+ int val = 0;
+
if (mask) {
val = phy_read(phydev, regnum);
if (val < 0) {
#define MII_DM9161_INTR_DPLX_CHANGE 0x0010
#define MII_DM9161_INTR_SPD_CHANGE 0x0008
#define MII_DM9161_INTR_LINK_CHANGE 0x0004
-#define MII_DM9161_INTR_INIT 0x0000
+#define MII_DM9161_INTR_INIT 0x0000
#define MII_DM9161_INTR_STOP \
-(MII_DM9161_INTR_DPLX_MASK | MII_DM9161_INTR_SPD_MASK \
- | MII_DM9161_INTR_LINK_MASK | MII_DM9161_INTR_MASK)
+ (MII_DM9161_INTR_DPLX_MASK | MII_DM9161_INTR_SPD_MASK | \
+ MII_DM9161_INTR_LINK_MASK | MII_DM9161_INTR_MASK)
#define MII_DM9161_INTR_CHANGE \
(MII_DM9161_INTR_DPLX_CHANGE | \
MII_DM9161_INTR_SPD_CHANGE | \
module_param(chosen_phy, int, 0444);
module_param_array(gpio_tab, ushort, NULL, 0444);
-MODULE_PARM_DESC(chosen_phy, \
+MODULE_PARM_DESC(chosen_phy,
"The address of the PHY to use for the ancillary clock features");
-MODULE_PARM_DESC(gpio_tab, \
+MODULE_PARM_DESC(gpio_tab,
"Which GPIO line to use for which purpose: cal,perout,extts1,...,extts6");
static void dp83640_gpio_defaults(struct ptp_pin_desc *pd)
static void enable_broadcast(struct phy_device *phydev, int init_page, int on)
{
int val;
+
phy_write(phydev, PAGESEL, 0);
val = phy_read(phydev, PHYCR2);
if (on)
{
int err;
- err = phy_write(phydev, DP83867_CTRL, DP83867_SW_RESET);
+ err = phy_write(phydev, DP83867_CTRL, DP83867_SW_RESTART);
if (err < 0)
return err;
usleep_range(10, 20);
- /* After reset FORCE_LINK_GOOD bit is set. Although the
- * default value should be unset. Disable FORCE_LINK_GOOD
- * for the phy to work properly.
- */
return phy_modify(phydev, MII_DP83867_PHYCTRL,
DP83867_PHYCR_FORCE_LINK_GOOD, 0);
}
static int et1011c_config_aneg(struct phy_device *phydev)
{
- int ctl = 0;
- ctl = phy_read(phydev, MII_BMCR);
+ int ctl = phy_read(phydev, MII_BMCR);
+
if (ctl < 0)
return ctl;
ctl &= ~(BMCR_FULLDPLX | BMCR_SPEED100 | BMCR_SPEED1000 |
static int et1011c_read_status(struct phy_device *phydev)
{
+ static int speed;
int ret;
u32 val;
- static int speed;
+
ret = genphy_read_status(phydev);
if (speed != phydev->speed) {
ET1011C_GIGABIT_SPEED) {
val = phy_read(phydev, ET1011C_CONFIG_REG);
val &= ~ET1011C_TX_FIFO_MASK;
- phy_write(phydev, ET1011C_CONFIG_REG, val\
- | ET1011C_GMII_INTERFACE\
- | ET1011C_SYS_CLK_EN\
- | ET1011C_TX_FIFO_DEPTH_16);
+ phy_write(phydev, ET1011C_CONFIG_REG, val |
+ ET1011C_GMII_INTERFACE |
+ ET1011C_SYS_CLK_EN |
+ ET1011C_TX_FIFO_DEPTH_16);
}
}
}
int fixed_phy_add(unsigned int irq, int phy_addr,
- struct fixed_phy_status *status) {
-
+ struct fixed_phy_status *status)
+{
return fixed_phy_add_gpiod(irq, phy_addr, status, NULL);
}
EXPORT_SYMBOL_GPL(fixed_phy_add);
return lpa;
/* If both registers are equal, it is suspect but not
- * impossible, hence a new try
- */
+ * impossible, hence a new try
+ */
} while (lpa == adv && retry--);
mii_lpa_to_linkmode_lpa_t(phydev->lp_advertising, lpa);
static int marvell_set_polarity(struct phy_device *phydev, int polarity)
{
- int reg;
- int err;
- int val;
-
- /* get the current settings */
- reg = phy_read(phydev, MII_M1011_PHY_SCR);
- if (reg < 0)
- return reg;
+ u16 val;
- val = reg;
- val &= ~MII_M1011_PHY_SCR_AUTO_CROSS;
switch (polarity) {
case ETH_TP_MDI:
- val |= MII_M1011_PHY_SCR_MDI;
+ val = MII_M1011_PHY_SCR_MDI;
break;
case ETH_TP_MDI_X:
- val |= MII_M1011_PHY_SCR_MDI_X;
+ val = MII_M1011_PHY_SCR_MDI_X;
break;
case ETH_TP_MDI_AUTO:
case ETH_TP_MDI_INVALID:
default:
- val |= MII_M1011_PHY_SCR_AUTO_CROSS;
+ val = MII_M1011_PHY_SCR_AUTO_CROSS;
break;
}
- if (val != reg) {
- /* Set the new polarity value in the register */
- err = phy_write(phydev, MII_M1011_PHY_SCR, val);
- if (err)
- return err;
- }
-
- return val != reg;
+ return phy_modify_changed(phydev, MII_M1011_PHY_SCR,
+ MII_M1011_PHY_SCR_AUTO_CROSS, val);
}
static int marvell_config_aneg(struct phy_device *phydev)
{
int delay;
- if (phydev->interface == PHY_INTERFACE_MODE_RGMII_ID) {
+ switch (phydev->interface) {
+ case PHY_INTERFACE_MODE_RGMII_ID:
delay = MII_M1111_RGMII_RX_DELAY | MII_M1111_RGMII_TX_DELAY;
- } else if (phydev->interface == PHY_INTERFACE_MODE_RGMII_RXID) {
+ break;
+ case PHY_INTERFACE_MODE_RGMII_RXID:
delay = MII_M1111_RGMII_RX_DELAY;
- } else if (phydev->interface == PHY_INTERFACE_MODE_RGMII_TXID) {
+ break;
+ case PHY_INTERFACE_MODE_RGMII_TXID:
delay = MII_M1111_RGMII_TX_DELAY;
- } else {
+ break;
+ default:
delay = 0;
+ break;
}
return phy_modify(phydev, MII_M1111_PHY_EXT_CR,
static void mdiobus_release(struct device *d)
{
struct mii_bus *bus = to_mii_bus(d);
+
BUG_ON(bus->state != MDIOBUS_RELEASED &&
/* for compatibility with error handling in drivers */
bus->state != MDIOBUS_ALLOCATED);
continue;
if (addr == mdiodev->addr) {
- dev->of_node = child;
- dev->fwnode = of_fwnode_handle(child);
+ device_set_node(dev, of_fwnode_handle(child));
return;
}
}
struct mdio_device *mdiodev;
int i;
- BUG_ON(bus->state != MDIOBUS_REGISTERED);
+ if (WARN_ON_ONCE(bus->state != MDIOBUS_REGISTERED))
+ return;
bus->state = MDIOBUS_UNREGISTERED;
for (i = 0; i < PHY_MAX_ADDR; i++) {
{
int err;
- dev_dbg(&mdiodev->dev, "mdio_device_register\n");
+ dev_dbg(&mdiodev->dev, "%s\n", __func__);
err = mdiobus_register_device(mdiodev);
if (err)
struct mdio_driver_common *mdiodrv = &drv->mdiodrv;
int retval;
- pr_debug("mdio_driver_register: %s\n", mdiodrv->driver.name);
+ pr_debug("%s: %s\n", __func__, mdiodrv->driver.name);
mdiodrv->driver.bus = &mdio_bus_type;
mdiodrv->driver.probe = mdio_probe;
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+#include <linux/bitfield.h>
+#include <linux/module.h>
+#include <linux/phy.h>
+
+#define MTK_EXT_PAGE_ACCESS 0x1f
+#define MTK_PHY_PAGE_STANDARD 0x0000
+#define MTK_PHY_PAGE_EXTENDED 0x0001
+#define MTK_PHY_PAGE_EXTENDED_2 0x0002
+#define MTK_PHY_PAGE_EXTENDED_3 0x0003
+#define MTK_PHY_PAGE_EXTENDED_2A30 0x2a30
+#define MTK_PHY_PAGE_EXTENDED_52B5 0x52b5
+
+static int mtk_gephy_read_page(struct phy_device *phydev)
+{
+ return __phy_read(phydev, MTK_EXT_PAGE_ACCESS);
+}
+
+static int mtk_gephy_write_page(struct phy_device *phydev, int page)
+{
+ return __phy_write(phydev, MTK_EXT_PAGE_ACCESS, page);
+}
+
+static void mtk_gephy_config_init(struct phy_device *phydev)
+{
+ /* Disable EEE */
+ phy_write_mmd(phydev, MDIO_MMD_AN, MDIO_AN_EEE_ADV, 0);
+
+ /* Enable HW auto downshift */
+ phy_modify_paged(phydev, MTK_PHY_PAGE_EXTENDED, 0x14, 0, BIT(4));
+
+ /* Increase SlvDPSready time */
+ phy_select_page(phydev, MTK_PHY_PAGE_EXTENDED_52B5);
+ __phy_write(phydev, 0x10, 0xafae);
+ __phy_write(phydev, 0x12, 0x2f);
+ __phy_write(phydev, 0x10, 0x8fae);
+ phy_restore_page(phydev, MTK_PHY_PAGE_STANDARD, 0);
+
+ /* Adjust 100_mse_threshold */
+ phy_write_mmd(phydev, MDIO_MMD_VEND1, 0x123, 0xffff);
+
+ /* Disable mcc */
+ phy_write_mmd(phydev, MDIO_MMD_VEND1, 0xa6, 0x300);
+}
+
+static int mt7530_phy_config_init(struct phy_device *phydev)
+{
+ mtk_gephy_config_init(phydev);
+
+ /* Increase post_update_timer */
+ phy_write_paged(phydev, MTK_PHY_PAGE_EXTENDED_3, 0x11, 0x4b);
+
+ return 0;
+}
+
+static int mt7531_phy_config_init(struct phy_device *phydev)
+{
+ if (phydev->interface != PHY_INTERFACE_MODE_INTERNAL)
+ return -EINVAL;
+
+ mtk_gephy_config_init(phydev);
+
+ /* PHY link down power saving enable */
+ phy_set_bits(phydev, 0x17, BIT(4));
+ phy_clear_bits_mmd(phydev, MDIO_MMD_VEND1, 0xc6, 0x300);
+
+ /* Set TX Pair delay selection */
+ phy_write_mmd(phydev, MDIO_MMD_VEND1, 0x13, 0x404);
+ phy_write_mmd(phydev, MDIO_MMD_VEND1, 0x14, 0x404);
+
+ return 0;
+}
+
+static struct phy_driver mtk_gephy_driver[] = {
+ {
+ PHY_ID_MATCH_EXACT(0x03a29412),
+ .name = "MediaTek MT7530 PHY",
+ .config_init = mt7530_phy_config_init,
+ /* Interrupts are handled by the switch, not the PHY
+ * itself.
+ */
+ .config_intr = genphy_no_config_intr,
+ .handle_interrupt = genphy_handle_interrupt_no_ack,
+ .read_page = mtk_gephy_read_page,
+ .write_page = mtk_gephy_write_page,
+ },
+ {
+ PHY_ID_MATCH_EXACT(0x03a29441),
+ .name = "MediaTek MT7531 PHY",
+ .config_init = mt7531_phy_config_init,
+ /* Interrupts are handled by the switch, not the PHY
+ * itself.
+ */
+ .config_intr = genphy_no_config_intr,
+ .handle_interrupt = genphy_handle_interrupt_no_ack,
+ .read_page = mtk_gephy_read_page,
+ .write_page = mtk_gephy_write_page,
+ },
+};
+
+module_phy_driver(mtk_gephy_driver);
+
+static struct mdio_device_id __maybe_unused mtk_gephy_tbl[] = {
+ { PHY_ID_MATCH_VENDOR(0x03a29400) },
+ { }
+};
+
+MODULE_DESCRIPTION("MediaTek Gigabit Ethernet PHY driver");
+MODULE_AUTHOR("DENG, Qingfang <dqfext@gmail.com>");
+MODULE_LICENSE("GPL");
+
+MODULE_DEVICE_TABLE(mdio, mtk_gephy_tbl);
*/
#include <linux/bitfield.h>
+#include <linux/ethtool_netlink.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/phy.h>
/* general Interrupt control/status reg in vendor specific block. */
#define MII_KSZPHY_INTCS 0x1B
-#define KSZPHY_INTCS_JABBER BIT(15)
-#define KSZPHY_INTCS_RECEIVE_ERR BIT(14)
-#define KSZPHY_INTCS_PAGE_RECEIVE BIT(13)
-#define KSZPHY_INTCS_PARELLEL BIT(12)
-#define KSZPHY_INTCS_LINK_PARTNER_ACK BIT(11)
-#define KSZPHY_INTCS_LINK_DOWN BIT(10)
-#define KSZPHY_INTCS_REMOTE_FAULT BIT(9)
-#define KSZPHY_INTCS_LINK_UP BIT(8)
-#define KSZPHY_INTCS_ALL (KSZPHY_INTCS_LINK_UP |\
+#define KSZPHY_INTCS_JABBER BIT(15)
+#define KSZPHY_INTCS_RECEIVE_ERR BIT(14)
+#define KSZPHY_INTCS_PAGE_RECEIVE BIT(13)
+#define KSZPHY_INTCS_PARELLEL BIT(12)
+#define KSZPHY_INTCS_LINK_PARTNER_ACK BIT(11)
+#define KSZPHY_INTCS_LINK_DOWN BIT(10)
+#define KSZPHY_INTCS_REMOTE_FAULT BIT(9)
+#define KSZPHY_INTCS_LINK_UP BIT(8)
+#define KSZPHY_INTCS_ALL (KSZPHY_INTCS_LINK_UP |\
KSZPHY_INTCS_LINK_DOWN)
-#define KSZPHY_INTCS_LINK_DOWN_STATUS BIT(2)
-#define KSZPHY_INTCS_LINK_UP_STATUS BIT(0)
-#define KSZPHY_INTCS_STATUS (KSZPHY_INTCS_LINK_DOWN_STATUS |\
+#define KSZPHY_INTCS_LINK_DOWN_STATUS BIT(2)
+#define KSZPHY_INTCS_LINK_UP_STATUS BIT(0)
+#define KSZPHY_INTCS_STATUS (KSZPHY_INTCS_LINK_DOWN_STATUS |\
KSZPHY_INTCS_LINK_UP_STATUS)
+/* LinkMD Control/Status */
+#define KSZ8081_LMD 0x1d
+#define KSZ8081_LMD_ENABLE_TEST BIT(15)
+#define KSZ8081_LMD_STAT_NORMAL 0
+#define KSZ8081_LMD_STAT_OPEN 1
+#define KSZ8081_LMD_STAT_SHORT 2
+#define KSZ8081_LMD_STAT_FAIL 3
+#define KSZ8081_LMD_STAT_MASK GENMASK(14, 13)
+/* Short cable (<10 meter) has been detected by LinkMD */
+#define KSZ8081_LMD_SHORT_INDICATOR BIT(12)
+#define KSZ8081_LMD_DELTA_TIME_MASK GENMASK(8, 0)
+
/* PHY Control 1 */
-#define MII_KSZPHY_CTRL_1 0x1e
+#define MII_KSZPHY_CTRL_1 0x1e
+#define KSZ8081_CTRL1_MDIX_STAT BIT(4)
/* PHY Control 2 / PHY Control (if no PHY Control 1) */
-#define MII_KSZPHY_CTRL_2 0x1f
-#define MII_KSZPHY_CTRL MII_KSZPHY_CTRL_2
+#define MII_KSZPHY_CTRL_2 0x1f
+#define MII_KSZPHY_CTRL MII_KSZPHY_CTRL_2
/* bitmap of PHY register to set interrupt mode */
+#define KSZ8081_CTRL2_HP_MDIX BIT(15)
+#define KSZ8081_CTRL2_MDI_MDI_X_SELECT BIT(14)
+#define KSZ8081_CTRL2_DISABLE_AUTO_MDIX BIT(13)
+#define KSZ8081_CTRL2_FORCE_LINK BIT(11)
+#define KSZ8081_CTRL2_POWER_SAVING BIT(10)
#define KSZPHY_CTRL_INT_ACTIVE_HIGH BIT(9)
#define KSZPHY_RMII_REF_CLK_SEL BIT(7)
/* Write/read to/from extended registers */
-#define MII_KSZPHY_EXTREG 0x0b
-#define KSZPHY_EXTREG_WRITE 0x8000
+#define MII_KSZPHY_EXTREG 0x0b
+#define KSZPHY_EXTREG_WRITE 0x8000
-#define MII_KSZPHY_EXTREG_WRITE 0x0c
-#define MII_KSZPHY_EXTREG_READ 0x0d
+#define MII_KSZPHY_EXTREG_WRITE 0x0c
+#define MII_KSZPHY_EXTREG_READ 0x0d
/* Extended registers */
-#define MII_KSZPHY_CLK_CONTROL_PAD_SKEW 0x104
-#define MII_KSZPHY_RX_DATA_PAD_SKEW 0x105
-#define MII_KSZPHY_TX_DATA_PAD_SKEW 0x106
+#define MII_KSZPHY_CLK_CONTROL_PAD_SKEW 0x104
+#define MII_KSZPHY_RX_DATA_PAD_SKEW 0x105
+#define MII_KSZPHY_TX_DATA_PAD_SKEW 0x106
#define PS_TO_REG 200
return kszphy_config_init(phydev);
}
+static int ksz8081_config_mdix(struct phy_device *phydev, u8 ctrl)
+{
+ u16 val;
+
+ switch (ctrl) {
+ case ETH_TP_MDI:
+ val = KSZ8081_CTRL2_DISABLE_AUTO_MDIX;
+ break;
+ case ETH_TP_MDI_X:
+ val = KSZ8081_CTRL2_DISABLE_AUTO_MDIX |
+ KSZ8081_CTRL2_MDI_MDI_X_SELECT;
+ break;
+ case ETH_TP_MDI_AUTO:
+ val = 0;
+ break;
+ default:
+ return 0;
+ }
+
+ return phy_modify(phydev, MII_KSZPHY_CTRL_2,
+ KSZ8081_CTRL2_HP_MDIX |
+ KSZ8081_CTRL2_MDI_MDI_X_SELECT |
+ KSZ8081_CTRL2_DISABLE_AUTO_MDIX,
+ KSZ8081_CTRL2_HP_MDIX | val);
+}
+
+static int ksz8081_config_aneg(struct phy_device *phydev)
+{
+ int ret;
+
+ ret = genphy_config_aneg(phydev);
+ if (ret)
+ return ret;
+
+ /* The MDI-X configuration is automatically changed by the PHY after
+ * switching from autoneg off to on. So, take MDI-X configuration under
+ * own control and set it after autoneg configuration was done.
+ */
+ return ksz8081_config_mdix(phydev, phydev->mdix_ctrl);
+}
+
+static int ksz8081_mdix_update(struct phy_device *phydev)
+{
+ int ret;
+
+ ret = phy_read(phydev, MII_KSZPHY_CTRL_2);
+ if (ret < 0)
+ return ret;
+
+ if (ret & KSZ8081_CTRL2_DISABLE_AUTO_MDIX) {
+ if (ret & KSZ8081_CTRL2_MDI_MDI_X_SELECT)
+ phydev->mdix_ctrl = ETH_TP_MDI_X;
+ else
+ phydev->mdix_ctrl = ETH_TP_MDI;
+ } else {
+ phydev->mdix_ctrl = ETH_TP_MDI_AUTO;
+ }
+
+ ret = phy_read(phydev, MII_KSZPHY_CTRL_1);
+ if (ret < 0)
+ return ret;
+
+ if (ret & KSZ8081_CTRL1_MDIX_STAT)
+ phydev->mdix = ETH_TP_MDI;
+ else
+ phydev->mdix = ETH_TP_MDI_X;
+
+ return 0;
+}
+
+static int ksz8081_read_status(struct phy_device *phydev)
+{
+ int ret;
+
+ ret = ksz8081_mdix_update(phydev);
+ if (ret < 0)
+ return ret;
+
+ return genphy_read_status(phydev);
+}
+
static int ksz8061_config_init(struct phy_device *phydev)
{
int ret;
static int ksz9021_config_init(struct phy_device *phydev)
{
- const struct device *dev = &phydev->mdio.dev;
- const struct device_node *of_node = dev->of_node;
+ const struct device_node *of_node;
const struct device *dev_walker;
/* The Micrel driver has a deprecated option to place phy OF
static int ksz9031_config_init(struct phy_device *phydev)
{
- const struct device *dev = &phydev->mdio.dev;
- const struct device_node *of_node = dev->of_node;
+ const struct device_node *of_node;
static const char *clk_skews[2] = {"rxc-skew-ps", "txc-skew-ps"};
static const char *rx_data_skews[4] = {
"rxd0-skew-ps", "rxd1-skew-ps",
static int ksz9131_config_init(struct phy_device *phydev)
{
- const struct device *dev = &phydev->mdio.dev;
- struct device_node *of_node = dev->of_node;
+ struct device_node *of_node;
char *clk_skews[2] = {"rxc-skew-psec", "txc-skew-psec"};
char *rx_data_skews[4] = {
"rxd0-skew-psec", "rxd1-skew-psec",
return 0;
}
+static int ksz886x_config_mdix(struct phy_device *phydev, u8 ctrl)
+{
+ u16 val;
+
+ switch (ctrl) {
+ case ETH_TP_MDI:
+ val = KSZ886X_BMCR_DISABLE_AUTO_MDIX;
+ break;
+ case ETH_TP_MDI_X:
+ /* Note: The naming of the bit KSZ886X_BMCR_FORCE_MDI is bit
+ * counter intuitive, the "-X" in "1 = Force MDI" in the data
+ * sheet seems to be missing:
+ * 1 = Force MDI (sic!) (transmit on RX+/RX- pins)
+ * 0 = Normal operation (transmit on TX+/TX- pins)
+ */
+ val = KSZ886X_BMCR_DISABLE_AUTO_MDIX | KSZ886X_BMCR_FORCE_MDI;
+ break;
+ case ETH_TP_MDI_AUTO:
+ val = 0;
+ break;
+ default:
+ return 0;
+ }
+
+ return phy_modify(phydev, MII_BMCR,
+ KSZ886X_BMCR_HP_MDIX | KSZ886X_BMCR_FORCE_MDI |
+ KSZ886X_BMCR_DISABLE_AUTO_MDIX,
+ KSZ886X_BMCR_HP_MDIX | val);
+}
+
+static int ksz886x_config_aneg(struct phy_device *phydev)
+{
+ int ret;
+
+ ret = genphy_config_aneg(phydev);
+ if (ret)
+ return ret;
+
+ /* The MDI-X configuration is automatically changed by the PHY after
+ * switching from autoneg off to on. So, take MDI-X configuration under
+ * own control and set it after autoneg configuration was done.
+ */
+ return ksz886x_config_mdix(phydev, phydev->mdix_ctrl);
+}
+
+static int ksz886x_mdix_update(struct phy_device *phydev)
+{
+ int ret;
+
+ ret = phy_read(phydev, MII_BMCR);
+ if (ret < 0)
+ return ret;
+
+ if (ret & KSZ886X_BMCR_DISABLE_AUTO_MDIX) {
+ if (ret & KSZ886X_BMCR_FORCE_MDI)
+ phydev->mdix_ctrl = ETH_TP_MDI_X;
+ else
+ phydev->mdix_ctrl = ETH_TP_MDI;
+ } else {
+ phydev->mdix_ctrl = ETH_TP_MDI_AUTO;
+ }
+
+ ret = phy_read(phydev, MII_KSZPHY_CTRL);
+ if (ret < 0)
+ return ret;
+
+ /* Same reverse logic as KSZ886X_BMCR_FORCE_MDI */
+ if (ret & KSZ886X_CTRL_MDIX_STAT)
+ phydev->mdix = ETH_TP_MDI_X;
+ else
+ phydev->mdix = ETH_TP_MDI;
+
+ return 0;
+}
+
+static int ksz886x_read_status(struct phy_device *phydev)
+{
+ int ret;
+
+ ret = ksz886x_mdix_update(phydev);
+ if (ret < 0)
+ return ret;
+
+ return genphy_read_status(phydev);
+}
+
static int kszphy_get_sset_count(struct phy_device *phydev)
{
return ARRAY_SIZE(kszphy_hw_stats);
return 0;
}
+static int ksz886x_cable_test_start(struct phy_device *phydev)
+{
+ if (phydev->dev_flags & MICREL_KSZ8_P1_ERRATA)
+ return -EOPNOTSUPP;
+
+ /* If autoneg is enabled, we won't be able to test cross pair
+ * short. In this case, the PHY will "detect" a link and
+ * confuse the internal state machine - disable auto neg here.
+ * If autoneg is disabled, we should set the speed to 10mbit.
+ */
+ return phy_clear_bits(phydev, MII_BMCR, BMCR_ANENABLE | BMCR_SPEED100);
+}
+
+static int ksz886x_cable_test_result_trans(u16 status)
+{
+ switch (FIELD_GET(KSZ8081_LMD_STAT_MASK, status)) {
+ case KSZ8081_LMD_STAT_NORMAL:
+ return ETHTOOL_A_CABLE_RESULT_CODE_OK;
+ case KSZ8081_LMD_STAT_SHORT:
+ return ETHTOOL_A_CABLE_RESULT_CODE_SAME_SHORT;
+ case KSZ8081_LMD_STAT_OPEN:
+ return ETHTOOL_A_CABLE_RESULT_CODE_OPEN;
+ case KSZ8081_LMD_STAT_FAIL:
+ fallthrough;
+ default:
+ return ETHTOOL_A_CABLE_RESULT_CODE_UNSPEC;
+ }
+}
+
+static bool ksz886x_cable_test_failed(u16 status)
+{
+ return FIELD_GET(KSZ8081_LMD_STAT_MASK, status) ==
+ KSZ8081_LMD_STAT_FAIL;
+}
+
+static bool ksz886x_cable_test_fault_length_valid(u16 status)
+{
+ switch (FIELD_GET(KSZ8081_LMD_STAT_MASK, status)) {
+ case KSZ8081_LMD_STAT_OPEN:
+ fallthrough;
+ case KSZ8081_LMD_STAT_SHORT:
+ return true;
+ }
+ return false;
+}
+
+static int ksz886x_cable_test_fault_length(u16 status)
+{
+ int dt;
+
+ /* According to the data sheet the distance to the fault is
+ * DELTA_TIME * 0.4 meters.
+ */
+ dt = FIELD_GET(KSZ8081_LMD_DELTA_TIME_MASK, status);
+
+ return (dt * 400) / 10;
+}
+
+static int ksz886x_cable_test_wait_for_completion(struct phy_device *phydev)
+{
+ int val, ret;
+
+ ret = phy_read_poll_timeout(phydev, KSZ8081_LMD, val,
+ !(val & KSZ8081_LMD_ENABLE_TEST),
+ 30000, 100000, true);
+
+ return ret < 0 ? ret : 0;
+}
+
+static int ksz886x_cable_test_one_pair(struct phy_device *phydev, int pair)
+{
+ static const int ethtool_pair[] = {
+ ETHTOOL_A_CABLE_PAIR_A,
+ ETHTOOL_A_CABLE_PAIR_B,
+ };
+ int ret, val, mdix;
+
+ /* There is no way to choice the pair, like we do one ksz9031.
+ * We can workaround this limitation by using the MDI-X functionality.
+ */
+ if (pair == 0)
+ mdix = ETH_TP_MDI;
+ else
+ mdix = ETH_TP_MDI_X;
+
+ switch (phydev->phy_id & MICREL_PHY_ID_MASK) {
+ case PHY_ID_KSZ8081:
+ ret = ksz8081_config_mdix(phydev, mdix);
+ break;
+ case PHY_ID_KSZ886X:
+ ret = ksz886x_config_mdix(phydev, mdix);
+ break;
+ default:
+ ret = -ENODEV;
+ }
+
+ if (ret)
+ return ret;
+
+ /* Now we are ready to fire. This command will send a 100ns pulse
+ * to the pair.
+ */
+ ret = phy_write(phydev, KSZ8081_LMD, KSZ8081_LMD_ENABLE_TEST);
+ if (ret)
+ return ret;
+
+ ret = ksz886x_cable_test_wait_for_completion(phydev);
+ if (ret)
+ return ret;
+
+ val = phy_read(phydev, KSZ8081_LMD);
+ if (val < 0)
+ return val;
+
+ if (ksz886x_cable_test_failed(val))
+ return -EAGAIN;
+
+ ret = ethnl_cable_test_result(phydev, ethtool_pair[pair],
+ ksz886x_cable_test_result_trans(val));
+ if (ret)
+ return ret;
+
+ if (!ksz886x_cable_test_fault_length_valid(val))
+ return 0;
+
+ return ethnl_cable_test_fault_length(phydev, ethtool_pair[pair],
+ ksz886x_cable_test_fault_length(val));
+}
+
+static int ksz886x_cable_test_get_status(struct phy_device *phydev,
+ bool *finished)
+{
+ unsigned long pair_mask = 0x3;
+ int retries = 20;
+ int pair, ret;
+
+ *finished = false;
+
+ /* Try harder if link partner is active */
+ while (pair_mask && retries--) {
+ for_each_set_bit(pair, &pair_mask, 4) {
+ ret = ksz886x_cable_test_one_pair(phydev, pair);
+ if (ret == -EAGAIN)
+ continue;
+ if (ret < 0)
+ return ret;
+ clear_bit(pair, &pair_mask);
+ }
+ /* If link partner is in autonegotiation mode it will send 2ms
+ * of FLPs with at least 6ms of silence.
+ * Add 2ms sleep to have better chances to hit this silence.
+ */
+ if (pair_mask)
+ msleep(2);
+ }
+
+ *finished = true;
+
+ return ret;
+}
+
static struct phy_driver ksphy_driver[] = {
{
.phy_id = PHY_ID_KS8737,
.phy_id = PHY_ID_KSZ8081,
.name = "Micrel KSZ8081 or KSZ8091",
.phy_id_mask = MICREL_PHY_ID_MASK,
+ .flags = PHY_POLL_CABLE_TEST,
/* PHY_BASIC_FEATURES */
.driver_data = &ksz8081_type,
.probe = kszphy_probe,
.config_init = ksz8081_config_init,
.soft_reset = genphy_soft_reset,
+ .config_aneg = ksz8081_config_aneg,
+ .read_status = ksz8081_read_status,
.config_intr = kszphy_config_intr,
.handle_interrupt = kszphy_handle_interrupt,
.get_sset_count = kszphy_get_sset_count,
.get_stats = kszphy_get_stats,
.suspend = kszphy_suspend,
.resume = kszphy_resume,
+ .cable_test_start = ksz886x_cable_test_start,
+ .cable_test_get_status = ksz886x_cable_test_get_status,
}, {
.phy_id = PHY_ID_KSZ8061,
.name = "Micrel KSZ8061",
.phy_id_mask = MICREL_PHY_ID_MASK,
.name = "Micrel KSZ8851 Ethernet MAC or KSZ886X Switch",
/* PHY_BASIC_FEATURES */
+ .flags = PHY_POLL_CABLE_TEST,
.config_init = kszphy_config_init,
+ .config_aneg = ksz886x_config_aneg,
+ .read_status = ksz886x_read_status,
.suspend = genphy_suspend,
.resume = genphy_resume,
+ .cable_test_start = ksz886x_cable_test_start,
+ .cable_test_get_status = ksz886x_cable_test_get_status,
}, {
.name = "Micrel KSZ87XX Switch",
/* PHY_BASIC_FEATURES */
struct mii_timestamping_desc *desc;
struct list_head *this;
+ if (!mii_ts)
+ return;
+
/* mii_timestamper statically registered by the PHY driver won't use the
* register_mii_timestamper() and thus don't have ->device set. Don't
* try to unregister these.
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Driver for Motorcomm PHYs
+ *
+ * Author: Peter Geis <pgwipeout@gmail.com>
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/phy.h>
+
+#define PHY_ID_YT8511 0x0000010a
+
+#define YT8511_PAGE_SELECT 0x1e
+#define YT8511_PAGE 0x1f
+#define YT8511_EXT_CLK_GATE 0x0c
+#define YT8511_EXT_DELAY_DRIVE 0x0d
+#define YT8511_EXT_SLEEP_CTRL 0x27
+
+/* 2b00 25m from pll
+ * 2b01 25m from xtl *default*
+ * 2b10 62.m from pll
+ * 2b11 125m from pll
+ */
+#define YT8511_CLK_125M (BIT(2) | BIT(1))
+#define YT8511_PLLON_SLP BIT(14)
+
+/* RX Delay enabled = 1.8ns 1000T, 8ns 10/100T */
+#define YT8511_DELAY_RX BIT(0)
+
+/* TX Gig-E Delay is bits 7:4, default 0x5
+ * TX Fast-E Delay is bits 15:12, default 0xf
+ * Delay = 150ps * N - 250ps
+ * On = 2000ps, off = 50ps
+ */
+#define YT8511_DELAY_GE_TX_EN (0xf << 4)
+#define YT8511_DELAY_GE_TX_DIS (0x2 << 4)
+#define YT8511_DELAY_FE_TX_EN (0xf << 12)
+#define YT8511_DELAY_FE_TX_DIS (0x2 << 12)
+
+static int yt8511_read_page(struct phy_device *phydev)
+{
+ return __phy_read(phydev, YT8511_PAGE_SELECT);
+};
+
+static int yt8511_write_page(struct phy_device *phydev, int page)
+{
+ return __phy_write(phydev, YT8511_PAGE_SELECT, page);
+};
+
+static int yt8511_config_init(struct phy_device *phydev)
+{
+ int oldpage, ret = 0;
+ unsigned int ge, fe;
+
+ oldpage = phy_select_page(phydev, YT8511_EXT_CLK_GATE);
+ if (oldpage < 0)
+ goto err_restore_page;
+
+ /* set rgmii delay mode */
+ switch (phydev->interface) {
+ case PHY_INTERFACE_MODE_RGMII:
+ ge = YT8511_DELAY_GE_TX_DIS;
+ fe = YT8511_DELAY_FE_TX_DIS;
+ break;
+ case PHY_INTERFACE_MODE_RGMII_RXID:
+ ge = YT8511_DELAY_RX | YT8511_DELAY_GE_TX_DIS;
+ fe = YT8511_DELAY_FE_TX_DIS;
+ break;
+ case PHY_INTERFACE_MODE_RGMII_TXID:
+ ge = YT8511_DELAY_GE_TX_EN;
+ fe = YT8511_DELAY_FE_TX_EN;
+ break;
+ case PHY_INTERFACE_MODE_RGMII_ID:
+ ge = YT8511_DELAY_RX | YT8511_DELAY_GE_TX_EN;
+ fe = YT8511_DELAY_FE_TX_EN;
+ break;
+ default: /* do not support other modes */
+ ret = -EOPNOTSUPP;
+ goto err_restore_page;
+ }
+
+ ret = __phy_modify(phydev, YT8511_PAGE, (YT8511_DELAY_RX | YT8511_DELAY_GE_TX_EN), ge);
+ if (ret < 0)
+ goto err_restore_page;
+
+ /* set clock mode to 125mhz */
+ ret = __phy_modify(phydev, YT8511_PAGE, 0, YT8511_CLK_125M);
+ if (ret < 0)
+ goto err_restore_page;
+
+ /* fast ethernet delay is in a separate page */
+ ret = __phy_write(phydev, YT8511_PAGE_SELECT, YT8511_EXT_DELAY_DRIVE);
+ if (ret < 0)
+ goto err_restore_page;
+
+ ret = __phy_modify(phydev, YT8511_PAGE, YT8511_DELAY_FE_TX_EN, fe);
+ if (ret < 0)
+ goto err_restore_page;
+
+ /* leave pll enabled in sleep */
+ ret = __phy_write(phydev, YT8511_PAGE_SELECT, YT8511_EXT_SLEEP_CTRL);
+ if (ret < 0)
+ goto err_restore_page;
+
+ ret = __phy_modify(phydev, YT8511_PAGE, 0, YT8511_PLLON_SLP);
+ if (ret < 0)
+ goto err_restore_page;
+
+err_restore_page:
+ return phy_restore_page(phydev, oldpage, ret);
+}
+
+static struct phy_driver motorcomm_phy_drvs[] = {
+ {
+ PHY_ID_MATCH_EXACT(PHY_ID_YT8511),
+ .name = "YT8511 Gigabit Ethernet",
+ .config_init = yt8511_config_init,
+ .suspend = genphy_suspend,
+ .resume = genphy_resume,
+ .read_page = yt8511_read_page,
+ .write_page = yt8511_write_page,
+ },
+};
+
+module_phy_driver(motorcomm_phy_drvs);
+
+MODULE_DESCRIPTION("Motorcomm PHY driver");
+MODULE_AUTHOR("Peter Geis");
+MODULE_LICENSE("GPL");
+
+static const struct mdio_device_id __maybe_unused motorcomm_tbl[] = {
+ { PHY_ID_MATCH_EXACT(PHY_ID_YT8511) },
+ { /* sentinal */ }
+};
+
+MODULE_DEVICE_TABLE(mdio, motorcomm_tbl);
return ret;
/* Clear the interrupt status bit by writing a “1”
- * to the corresponding bit in INT_CLEAR (2:0 are reserved) */
+ * to the corresponding bit in INT_CLEAR (2:0 are reserved)
+ */
ret = phy_write(phydev, DP83865_INT_CLEAR, ret & ~0x7);
return ret;
{
ns_giga_speed_fallback(phydev, ALL_FALLBACK_ON);
/* In the latest MAC or switches design, the 10 Mbps loopback
- is desired to be turned off. */
+ * is desired to be turned off.
+ */
ns_10_base_t_hdx_loopack(phydev, hdx_loopback_off);
return ns_ack_interrupt(phydev);
}
{
ts->tv_nsec = hwts->nsec;
if ((ts->tv_sec & TS_SEC_MASK) < (hwts->sec & TS_SEC_MASK))
- ts->tv_sec -= BIT(2);
+ ts->tv_sec -= TS_SEC_MASK + 1;
ts->tv_sec &= ~TS_SEC_MASK;
ts->tv_sec |= hwts->sec & TS_SEC_MASK;
}
nxp_c45_process_txts(priv, &hwts);
}
- nxp_c45_ptp_gettimex64(&priv->caps, &ts, NULL);
while ((skb = skb_dequeue(&priv->rx_queue)) != NULL) {
+ nxp_c45_ptp_gettimex64(&priv->caps, &ts, NULL);
ts_raw = __be32_to_cpu(NXP_C45_SKB_CB(skb)->header->reserved2);
hwts.sec = ts_raw >> 30;
hwts.nsec = ts_raw & GENMASK(29, 0);
return ret;
}
+ /* Bug workaround for SJA1110 rev B: enable write access
+ * to MDIO_MMD_PMAPMD
+ */
+ phy_write_mmd(phydev, MDIO_MMD_VEND1, 0x01F8, 1);
+ phy_write_mmd(phydev, MDIO_MMD_VEND1, 0x01F9, 2);
+
phy_set_bits_mmd(phydev, MDIO_MMD_VEND1, VEND1_PHY_CONFIG,
PHY_CONFIG_AUTO);
VEND1_PORT_ABILITIES);
ptp_ability = !!(ptp_ability & PTP_ABILITY);
if (!ptp_ability) {
- phydev_info(phydev, "the phy does not support PTP");
+ phydev_dbg(phydev, "the phy does not support PTP");
goto no_ptp_support;
}
* @phydev: target phy_device struct
*
* Disable auto-negotiation in the Clause 45 PHY. The link parameters
- * parameters are controlled through the PMA/PMD MMD registers.
+ * are controlled through the PMA/PMD MMD registers.
*
* Returns zero on success, negative errno code on failure.
*/
/* A mapping of all SUPPORTED settings to speed/duplex. This table
* must be grouped by speed and sorted in descending match priority
- * - iow, descending speed. */
+ * - iow, descending speed.
+ */
#define PHY_SETTING(s, d, b) { .speed = SPEED_ ## s, .duplex = DUPLEX_ ## d, \
.bit = ETHTOOL_LINK_MODE_ ## b ## _BIT}
else if (val & BMCR_SPEED100)
phydev->speed = SPEED_100;
else phydev->speed = SPEED_10;
- }
- else {
+ } else {
if (phydev->autoneg == AUTONEG_DISABLE)
change_autoneg = true;
phydev->autoneg = AUTONEG_ENABLE;
else if (do_suspend)
phy_suspend(phydev);
+ if (err == -ENODEV)
+ return;
+
if (err < 0)
phy_error(phydev);
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/acpi.h>
#include <linux/bitmap.h>
#include <linux/delay.h>
#include <linux/errno.h>
return 0;
}
+/* Extract the phy ID from the compatible string of the form
+ * ethernet-phy-idAAAA.BBBB.
+ */
+int fwnode_get_phy_id(struct fwnode_handle *fwnode, u32 *phy_id)
+{
+ unsigned int upper, lower;
+ const char *cp;
+ int ret;
+
+ ret = fwnode_property_read_string(fwnode, "compatible", &cp);
+ if (ret)
+ return ret;
+
+ if (sscanf(cp, "ethernet-phy-id%4x.%4x", &upper, &lower) != 2)
+ return -EINVAL;
+
+ *phy_id = ((upper & GENMASK(15, 0)) << 16) | (lower & GENMASK(15, 0));
+ return 0;
+}
+EXPORT_SYMBOL(fwnode_get_phy_id);
+
/**
* get_phy_device - reads the specified PHY device and returns its @phy_device
* struct
if (r)
return ERR_PTR(r);
+ /* PHY device such as the Marvell Alaska 88E2110 will return a PHY ID
+ * of 0 when probed using get_phy_c22_id() with no error. Proceed to
+ * probe with C45 to see if we're able to get a valid PHY ID in the C45
+ * space, if successful, create the C45 PHY device.
+ */
+ if (!is_c45 && phy_id == 0 && bus->probe_capabilities >= MDIOBUS_C45) {
+ r = get_phy_c45_ids(bus, addr, &c45_ids);
+ if (!r)
+ return phy_device_create(bus, addr, phy_id,
+ true, &c45_ids);
+ }
+
return phy_device_create(bus, addr, phy_id, is_c45, &c45_ids);
}
EXPORT_SYMBOL(get_phy_device);
*/
void phy_device_remove(struct phy_device *phydev)
{
- if (phydev->mii_ts)
- unregister_mii_timestamper(phydev->mii_ts);
+ unregister_mii_timestamper(phydev->mii_ts);
device_del(&phydev->mdio.dev);
}
/**
+ * fwnode_mdio_find_device - Given a fwnode, find the mdio_device
+ * @fwnode: pointer to the mdio_device's fwnode
+ *
+ * If successful, returns a pointer to the mdio_device with the embedded
+ * struct device refcount incremented by one, or NULL on failure.
+ * The caller should call put_device() on the mdio_device after its use.
+ */
+struct mdio_device *fwnode_mdio_find_device(struct fwnode_handle *fwnode)
+{
+ struct device *d;
+
+ if (!fwnode)
+ return NULL;
+
+ d = bus_find_device_by_fwnode(&mdio_bus_type, fwnode);
+ if (!d)
+ return NULL;
+
+ return to_mdio_device(d);
+}
+EXPORT_SYMBOL(fwnode_mdio_find_device);
+
+/**
+ * fwnode_phy_find_device - For provided phy_fwnode, find phy_device.
+ *
+ * @phy_fwnode: Pointer to the phy's fwnode.
+ *
+ * If successful, returns a pointer to the phy_device with the embedded
+ * struct device refcount incremented by one, or NULL on failure.
+ */
+struct phy_device *fwnode_phy_find_device(struct fwnode_handle *phy_fwnode)
+{
+ struct mdio_device *mdiodev;
+
+ mdiodev = fwnode_mdio_find_device(phy_fwnode);
+ if (!mdiodev)
+ return NULL;
+
+ if (mdiodev->flags & MDIO_DEVICE_FLAG_PHY)
+ return to_phy_device(&mdiodev->dev);
+
+ put_device(&mdiodev->dev);
+
+ return NULL;
+}
+EXPORT_SYMBOL(fwnode_phy_find_device);
+
+/**
+ * device_phy_find_device - For the given device, get the phy_device
+ * @dev: Pointer to the given device
+ *
+ * Refer return conditions of fwnode_phy_find_device().
+ */
+struct phy_device *device_phy_find_device(struct device *dev)
+{
+ return fwnode_phy_find_device(dev_fwnode(dev));
+}
+EXPORT_SYMBOL_GPL(device_phy_find_device);
+
+/**
+ * fwnode_get_phy_node - Get the phy_node using the named reference.
+ * @fwnode: Pointer to fwnode from which phy_node has to be obtained.
+ *
+ * Refer return conditions of fwnode_find_reference().
+ * For ACPI, only "phy-handle" is supported. Legacy DT properties "phy"
+ * and "phy-device" are not supported in ACPI. DT supports all the three
+ * named references to the phy node.
+ */
+struct fwnode_handle *fwnode_get_phy_node(struct fwnode_handle *fwnode)
+{
+ struct fwnode_handle *phy_node;
+
+ /* Only phy-handle is used for ACPI */
+ phy_node = fwnode_find_reference(fwnode, "phy-handle", 0);
+ if (is_acpi_node(fwnode) || !IS_ERR(phy_node))
+ return phy_node;
+ phy_node = fwnode_find_reference(fwnode, "phy", 0);
+ if (IS_ERR(phy_node))
+ phy_node = fwnode_find_reference(fwnode, "phy-device", 0);
+ return phy_node;
+}
+EXPORT_SYMBOL_GPL(fwnode_get_phy_node);
+
+/**
* phy_probe - probe and init a PHY device
* @dev: device to probe and init
*
/* Disable the interrupt if the PHY doesn't support it
* but the interrupt is still a valid one
*/
- if (!phy_drv_supports_irq(phydrv) && phy_interrupt_is_valid(phydev))
+ if (!phy_drv_supports_irq(phydrv) && phy_interrupt_is_valid(phydev))
phydev->irq = PHY_POLL;
if (phydrv->flags & PHY_IS_INTERNAL)
* a controller will attach, and may modify one
* or both of these values
*/
- if (phydrv->features) {
+ if (phydrv->features)
linkmode_copy(phydev->supported, phydrv->features);
- } else if (phydrv->get_features) {
+ else if (phydrv->get_features)
err = phydrv->get_features(phydev);
- } else if (phydev->is_c45) {
+ else if (phydev->is_c45)
err = genphy_c45_pma_read_abilities(phydev);
- } else {
+ else
err = genphy_read_abilities(phydev);
- }
if (err)
goto out;
*
* Copyright (C) 2015 Russell King
*/
+#include <linux/acpi.h>
#include <linux/ethtool.h>
#include <linux/export.h>
#include <linux/gpio/consumer.h>
pl->link_config.duplex = DUPLEX_FULL;
/* We treat the "pause" and "asym-pause" terminology as
- * defining the link partner's ability. */
+ * defining the link partner's ability.
+ */
if (fwnode_property_read_bool(fixed_node, "pause"))
__set_bit(ETHTOOL_LINK_MODE_Pause_BIT,
pl->link_config.lp_advertising);
phylink_set(pl->supported, 5000baseT_Full);
break;
+ case PHY_INTERFACE_MODE_25GBASER:
+ phylink_set(pl->supported, 25000baseCR_Full);
+ phylink_set(pl->supported, 25000baseKR_Full);
+ phylink_set(pl->supported, 25000baseSR_Full);
+ fallthrough;
case PHY_INTERFACE_MODE_USXGMII:
case PHY_INTERFACE_MODE_10GKR:
case PHY_INTERFACE_MODE_10GBASER:
phylink_mac_pcs_get_state(pl, &link_state);
/* If we have a phy, the "up" state is the union of
- * both the PHY and the MAC */
+ * both the PHY and the MAC
+ */
if (pl->phydev)
link_state.link &= pl->phy_state.link;
link_state.interface = pl->phy_state.interface;
/* If we have a PHY, we need to update with
- * the PHY flow control bits. */
+ * the PHY flow control bits.
+ */
link_state.pause = pl->phy_state.pause;
mac_config = true;
}
int phylink_of_phy_connect(struct phylink *pl, struct device_node *dn,
u32 flags)
{
- struct device_node *phy_node;
+ return phylink_fwnode_phy_connect(pl, of_fwnode_handle(dn), flags);
+}
+EXPORT_SYMBOL_GPL(phylink_of_phy_connect);
+
+/**
+ * phylink_fwnode_phy_connect() - connect the PHY specified in the fwnode.
+ * @pl: a pointer to a &struct phylink returned from phylink_create()
+ * @fwnode: a pointer to a &struct fwnode_handle.
+ * @flags: PHY-specific flags to communicate to the PHY device driver
+ *
+ * Connect the phy specified @fwnode to the phylink instance specified
+ * by @pl.
+ *
+ * Returns 0 on success or a negative errno.
+ */
+int phylink_fwnode_phy_connect(struct phylink *pl,
+ struct fwnode_handle *fwnode,
+ u32 flags)
+{
+ struct fwnode_handle *phy_fwnode;
struct phy_device *phy_dev;
int ret;
phy_interface_mode_is_8023z(pl->link_interface)))
return 0;
- phy_node = of_parse_phandle(dn, "phy-handle", 0);
- if (!phy_node)
- phy_node = of_parse_phandle(dn, "phy", 0);
- if (!phy_node)
- phy_node = of_parse_phandle(dn, "phy-device", 0);
-
- if (!phy_node) {
+ phy_fwnode = fwnode_get_phy_node(fwnode);
+ if (IS_ERR(phy_fwnode)) {
if (pl->cfg_link_an_mode == MLO_AN_PHY)
return -ENODEV;
return 0;
}
- phy_dev = of_phy_find_device(phy_node);
+ phy_dev = fwnode_phy_find_device(phy_fwnode);
/* We're done with the phy_node handle */
- of_node_put(phy_node);
+ fwnode_handle_put(phy_fwnode);
if (!phy_dev)
return -ENODEV;
ret = phy_attach_direct(pl->netdev, phy_dev, flags,
pl->link_interface);
- if (ret)
+ if (ret) {
+ phy_device_free(phy_dev);
return ret;
+ }
ret = phylink_bringup_phy(pl, phy_dev, pl->link_config.interface);
if (ret)
return ret;
}
-EXPORT_SYMBOL_GPL(phylink_of_phy_connect);
+EXPORT_SYMBOL_GPL(phylink_fwnode_phy_connect);
/**
* phylink_disconnect_phy() - disconnect any PHY attached to the phylink
ASSERT_RTNL();
- if (pl->phydev) {
+ if (pl->phydev)
phy_ethtool_ksettings_get(pl->phydev, kset);
- } else {
+ else
kset->base.port = pl->link_port;
- }
linkmode_copy(kset->link_modes.supported, pl->supported);
static int qs6612_config_intr(struct phy_device *phydev)
{
int err;
+
if (phydev->interrupts == PHY_INTERRUPT_ENABLED) {
/* clear any interrupts before enabling them */
err = qs6612_ack_interrupt(phydev);
* Copyright (c) 2004 Freescale Semiconductor, Inc.
*/
#include <linux/bitops.h>
+#include <linux/of.h>
#include <linux/phy.h>
#include <linux/module.h>
#include <linux/delay.h>
#define RTL821x_PAGE_SELECT 0x1f
#define RTL8211F_PHYCR1 0x18
+#define RTL8211F_PHYCR2 0x19
#define RTL8211F_INSR 0x1d
#define RTL8211F_TX_DELAY BIT(8)
#define RTL8211E_TX_DELAY BIT(12)
#define RTL8211E_RX_DELAY BIT(11)
+#define RTL8211F_CLKOUT_EN BIT(0)
+
#define RTL8201F_ISR 0x1e
#define RTL8201F_ISR_ANERR BIT(15)
#define RTL8201F_ISR_DUPLEX BIT(13)
MODULE_AUTHOR("Johnson Leung");
MODULE_LICENSE("GPL");
+struct rtl821x_priv {
+ u16 phycr1;
+ u16 phycr2;
+};
+
static int rtl821x_read_page(struct phy_device *phydev)
{
return __phy_read(phydev, RTL821x_PAGE_SELECT);
return __phy_write(phydev, RTL821x_PAGE_SELECT, page);
}
+static int rtl821x_probe(struct phy_device *phydev)
+{
+ struct device *dev = &phydev->mdio.dev;
+ struct rtl821x_priv *priv;
+ int ret;
+
+ priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ ret = phy_read_paged(phydev, 0xa43, RTL8211F_PHYCR1);
+ if (ret < 0)
+ return ret;
+
+ priv->phycr1 = ret & (RTL8211F_ALDPS_PLL_OFF | RTL8211F_ALDPS_ENABLE | RTL8211F_ALDPS_XTAL_OFF);
+ if (of_property_read_bool(dev->of_node, "realtek,aldps-enable"))
+ priv->phycr1 |= RTL8211F_ALDPS_PLL_OFF | RTL8211F_ALDPS_ENABLE | RTL8211F_ALDPS_XTAL_OFF;
+
+ ret = phy_read_paged(phydev, 0xa43, RTL8211F_PHYCR2);
+ if (ret < 0)
+ return ret;
+
+ priv->phycr2 = ret & RTL8211F_CLKOUT_EN;
+ if (of_property_read_bool(dev->of_node, "realtek,clkout-disable"))
+ priv->phycr2 &= ~RTL8211F_CLKOUT_EN;
+
+ phydev->priv = priv;
+
+ return 0;
+}
+
static int rtl8201_ack_interrupt(struct phy_device *phydev)
{
int err;
static int rtl8211f_config_init(struct phy_device *phydev)
{
+ struct rtl821x_priv *priv = phydev->priv;
struct device *dev = &phydev->mdio.dev;
u16 val_txdly, val_rxdly;
- u16 val;
int ret;
- val = RTL8211F_ALDPS_ENABLE | RTL8211F_ALDPS_PLL_OFF | RTL8211F_ALDPS_XTAL_OFF;
- phy_modify_paged_changed(phydev, 0xa43, RTL8211F_PHYCR1, val, val);
+ ret = phy_modify_paged_changed(phydev, 0xa43, RTL8211F_PHYCR1,
+ RTL8211F_ALDPS_PLL_OFF | RTL8211F_ALDPS_ENABLE | RTL8211F_ALDPS_XTAL_OFF,
+ priv->phycr1);
+ if (ret < 0) {
+ dev_err(dev, "aldps mode configuration failed: %pe\n",
+ ERR_PTR(ret));
+ return ret;
+ }
switch (phydev->interface) {
case PHY_INTERFACE_MODE_RGMII:
val_rxdly ? "enabled" : "disabled");
}
+ ret = phy_modify_paged(phydev, 0xa43, RTL8211F_PHYCR2,
+ RTL8211F_CLKOUT_EN, priv->phycr2);
+ if (ret < 0) {
+ dev_err(dev, "clkout configuration failed: %pe\n",
+ ERR_PTR(ret));
+ return ret;
+ }
+
+ return genphy_soft_reset(phydev);
+}
+
+static int rtl821x_resume(struct phy_device *phydev)
+{
+ int ret;
+
+ ret = genphy_resume(phydev);
+ if (ret < 0)
+ return ret;
+
+ msleep(20);
+
return 0;
}
}, {
PHY_ID_MATCH_EXACT(0x001cc916),
.name = "RTL8211F Gigabit Ethernet",
+ .probe = rtl821x_probe,
.config_init = &rtl8211f_config_init,
.read_status = rtlgen_read_status,
.config_intr = &rtl8211f_config_intr,
.handle_interrupt = rtl8211f_handle_interrupt,
.suspend = genphy_suspend,
- .resume = genphy_resume,
+ .resume = rtl821x_resume,
.read_page = rtl821x_read_page,
.write_page = rtl821x_write_page,
}, {
// SPDX-License-Identifier: GPL-2.0+
-/**
+/*
* drivers/net/phy/rockchip.c
*
* Driver for ROCKCHIP Ethernet PHYs
phy_interface_t sfp_select_interface(struct sfp_bus *bus,
unsigned long *link_modes)
{
+ if (phylink_test(link_modes, 25000baseCR_Full) ||
+ phylink_test(link_modes, 25000baseKR_Full) ||
+ phylink_test(link_modes, 25000baseSR_Full))
+ return PHY_INTERFACE_MODE_25GBASER;
+
if (phylink_test(link_modes, 10000baseCR_Full) ||
phylink_test(link_modes, 10000baseSR_Full) ||
phylink_test(link_modes, 10000baseLR_Full) ||
* be put via sfp_bus_put() when done.
*
* Returns:
- * - on success, a pointer to the sfp_bus structure,
- * - %NULL if no SFP is specified,
- * - on failure, an error pointer value:
+ * - on success, a pointer to the sfp_bus structure,
+ * - %NULL if no SFP is specified,
+ * - on failure, an error pointer value:
*
- * - corresponding to the errors detailed for
- * fwnode_property_get_reference_args().
- * - %-ENOMEM if we failed to allocate the bus.
- * - an error from the upstream's connect_phy() method.
+ * - corresponding to the errors detailed for
+ * fwnode_property_get_reference_args().
+ * - %-ENOMEM if we failed to allocate the bus.
+ * - an error from the upstream's connect_phy() method.
*/
struct sfp_bus *sfp_bus_find_fwnode(struct fwnode_handle *fwnode)
{
* bus, so it is safe to put the bus after this call.
*
* Returns:
- * - on success, a pointer to the sfp_bus structure,
- * - %NULL if no SFP is specified,
- * - on failure, an error pointer value:
+ * - on success, a pointer to the sfp_bus structure,
+ * - %NULL if no SFP is specified,
+ * - on failure, an error pointer value:
*
- * - corresponding to the errors detailed for
- * fwnode_property_get_reference_args().
- * - %-ENOMEM if we failed to allocate the bus.
- * - an error from the upstream's connect_phy() method.
+ * - corresponding to the errors detailed for
+ * fwnode_property_get_reference_args().
+ * - %-ENOMEM if we failed to allocate the bus.
+ * - an error from the upstream's connect_phy() method.
*/
int sfp_bus_add_upstream(struct sfp_bus *bus, void *upstream,
const struct sfp_upstream_ops *ops)
case SFP_S_INIT:
if (event == SFP_E_TIMEOUT && sfp->state & SFP_F_TX_FAULT) {
- /* TX_FAULT is still asserted after t_init or
+ /* TX_FAULT is still asserted after t_init
* or t_start_up, so assume there is a fault.
*/
sfp_sm_fault(sfp, SFP_S_INIT_TX_FAULT,
MODULE_DEVICE_TABLE(spi, ks8995_id);
static const struct of_device_id ks8895_spi_of_match[] = {
- { .compatible = "micrel,ks8995" },
- { .compatible = "micrel,ksz8864" },
- { .compatible = "micrel,ksz8795" },
- { },
- };
+ { .compatible = "micrel,ks8995" },
+ { .compatible = "micrel,ksz8864" },
+ { .compatible = "micrel,ksz8795" },
+ { },
+};
MODULE_DEVICE_TABLE(of, ks8895_spi_of_match);
static inline u8 get_chip_id(u8 val)
#include <linux/mii.h>
#include <linux/phy.h>
-#define MII_XCIIS 0x11 /* Configuration Info IRQ & Status Reg */
-#define MII_XIE 0x12 /* Interrupt Enable Register */
+#define MII_XCIIS 0x11 /* Configuration Info IRQ & Status Reg */
+#define MII_XIE 0x12 /* Interrupt Enable Register */
#define MII_XIE_DEFAULT_MASK 0x0070 /* ANE complete, Remote Fault, Link Down */
#define STE101P_PHY_ID 0x00061c50
-#define STE100P_PHY_ID 0x1c040011
+#define STE100P_PHY_ID 0x1c040011
static int ste10Xp_config_init(struct phy_device *phydev)
{
/* This adds a skew for both TX and RX clocks, so the skew should only be
* applied to "rgmii-id" interfaces. It may not work as expected
- * on "rgmii-txid", "rgmii-rxid" or "rgmii" interfaces. */
+ * on "rgmii-txid", "rgmii-rxid" or "rgmii" interfaces.
+ */
static int vsc8601_add_skew(struct phy_device *phydev)
{
int ret;
* Initialize the data information for the compression code
*/
db->totlen = sizeof (struct bsd_db) +
- (sizeof (struct bsd_dict) * hsize);
+ (sizeof (struct bsd_dict) * hsize);
db->hsize = hsize;
db->hshift = hshift;
* Found it -- move to the front on the connection list.
*/
if(lcs == ocs) {
- /* found at most recently used */
+ /* found at most recently used */
} else if (cs == ocs) {
/* found at least recently used */
comp->xmit_oldest = lcs->cs_this;
return tun->flags & (TUN_FEATURES | IFF_PERSIST | IFF_TUN | IFF_TAP);
}
-static ssize_t tun_show_flags(struct device *dev, struct device_attribute *attr,
+static ssize_t tun_flags_show(struct device *dev, struct device_attribute *attr,
char *buf)
{
struct tun_struct *tun = netdev_priv(to_net_dev(dev));
return sprintf(buf, "0x%x\n", tun_flags(tun));
}
-static ssize_t tun_show_owner(struct device *dev, struct device_attribute *attr,
- char *buf)
+static ssize_t owner_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
{
struct tun_struct *tun = netdev_priv(to_net_dev(dev));
return uid_valid(tun->owner)?
sprintf(buf, "-1\n");
}
-static ssize_t tun_show_group(struct device *dev, struct device_attribute *attr,
- char *buf)
+static ssize_t group_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
{
struct tun_struct *tun = netdev_priv(to_net_dev(dev));
return gid_valid(tun->group) ?
sprintf(buf, "-1\n");
}
-static DEVICE_ATTR(tun_flags, 0444, tun_show_flags, NULL);
-static DEVICE_ATTR(owner, 0444, tun_show_owner, NULL);
-static DEVICE_ATTR(group, 0444, tun_show_group, NULL);
+static DEVICE_ATTR_RO(tun_flags);
+static DEVICE_ATTR_RO(owner);
+static DEVICE_ATTR_RO(group);
static struct attribute *tun_dev_attrs[] = {
&dev_attr_tun_flags.attr,
depends on USB_USBNET
select CRC32
select PHYLIB
+ select AX88796B_PHY
+ imply NET_SELFTESTS
default y
help
This option adds support for ASIX AX88xxx based USB 2.0
10/100 Ethernet adapters.
- This driver should work with at least the following devices:
+ This driver should work with at least the following devices:
* Aten UC210T
* ASIX AX88172
* Billionton Systems, USB2AR
CDC Ethernet is an implementation option for DOCSIS cable modems
that support USB connectivity, used for non-Microsoft USB hosts.
The Linux-USB CDC Ethernet Gadget driver is an open implementation.
- This driver should work with at least the following devices:
+ This driver should work with at least the following devices:
* Dell Wireless 5530 HSPA
- * Ericsson PipeRider (all variants)
+ * Ericsson PipeRider (all variants)
* Ericsson Mobile Broadband Module (all variants)
- * Motorola (DM100 and SB4100)
- * Broadcom Cable Modem (reference design)
+ * Motorola (DM100 and SB4100)
+ * Broadcom Cable Modem (reference design)
* Toshiba (PCX1100U and F3507g/F3607gw)
* ...
#include <linux/usb/usbnet.h>
#include <linux/slab.h>
#include <linux/if_vlan.h>
+#include <linux/phy.h>
+#include <net/selftests.h>
#define DRIVER_VERSION "22-Dec-2011"
#define DRIVER_NAME "asix"
u16 presvd_phy_advertise;
u16 presvd_phy_bmcr;
struct asix_rx_fixup_info rx_fixup_info;
+ struct mii_bus *mdio;
+ struct phy_device *phydev;
+ u16 phy_addr;
+ char phy_name[20];
};
extern const struct driver_info ax88172a_info;
int asix_set_sw_mii(struct usbnet *dev, int in_pm);
int asix_set_hw_mii(struct usbnet *dev, int in_pm);
-int asix_read_phy_addr(struct usbnet *dev, int internal);
-int asix_get_phy_addr(struct usbnet *dev);
+int asix_read_phy_addr(struct usbnet *dev, bool internal);
int asix_sw_reset(struct usbnet *dev, u8 flags, int in_pm);
u16 asix_read_medium_status(struct usbnet *dev, int in_pm);
int asix_write_medium_mode(struct usbnet *dev, u16 mode, int in_pm);
+void asix_adjust_link(struct net_device *netdev);
int asix_write_gpio(struct usbnet *dev, u16 value, int sleep, int in_pm);
int asix_mdio_read(struct net_device *netdev, int phy_id, int loc);
void asix_mdio_write(struct net_device *netdev, int phy_id, int loc, int val);
+int asix_mdio_bus_read(struct mii_bus *bus, int phy_id, int regnum);
+int asix_mdio_bus_write(struct mii_bus *bus, int phy_id, int regnum, u16 val);
+
int asix_mdio_read_nopm(struct net_device *netdev, int phy_id, int loc);
void asix_mdio_write_nopm(struct net_device *netdev, int phy_id, int loc,
int val);
return ret;
}
-int asix_read_phy_addr(struct usbnet *dev, int internal)
+int asix_read_phy_addr(struct usbnet *dev, bool internal)
{
- int offset = (internal ? 1 : 0);
+ int ret, offset;
u8 buf[2];
- int ret = asix_read_cmd(dev, AX_CMD_READ_PHY_ID, 0, 0, 2, buf, 0);
- netdev_dbg(dev->net, "asix_get_phy_addr()\n");
+ ret = asix_read_cmd(dev, AX_CMD_READ_PHY_ID, 0, 0, 2, buf, 0);
+ if (ret < 0)
+ goto error;
if (ret < 2) {
- netdev_err(dev->net, "Error reading PHYID register: %02x\n", ret);
- goto out;
+ ret = -EIO;
+ goto error;
}
- netdev_dbg(dev->net, "asix_get_phy_addr() returning 0x%04x\n",
- *((__le16 *)buf));
+
+ offset = (internal ? 1 : 0);
ret = buf[offset];
-out:
+ netdev_dbg(dev->net, "%s PHY address 0x%x\n",
+ internal ? "internal" : "external", ret);
+
return ret;
-}
-int asix_get_phy_addr(struct usbnet *dev)
-{
- /* return the address of the internal phy */
- return asix_read_phy_addr(dev, 1);
-}
+error:
+ netdev_err(dev->net, "Error reading PHY_ID register: %02x\n", ret);
+ return ret;
+}
int asix_sw_reset(struct usbnet *dev, u8 flags, int in_pm)
{
return ret;
}
+/* set MAC link settings according to information from phylib */
+void asix_adjust_link(struct net_device *netdev)
+{
+ struct phy_device *phydev = netdev->phydev;
+ struct usbnet *dev = netdev_priv(netdev);
+ u16 mode = 0;
+
+ if (phydev->link) {
+ mode = AX88772_MEDIUM_DEFAULT;
+
+ if (phydev->duplex == DUPLEX_HALF)
+ mode &= ~AX_MEDIUM_FD;
+
+ if (phydev->speed != SPEED_100)
+ mode &= ~AX_MEDIUM_PS;
+ }
+
+ asix_write_medium_mode(dev, mode, 0);
+ phy_print_status(phydev);
+}
+
int asix_write_gpio(struct usbnet *dev, u16 value, int sleep, int in_pm)
{
int ret;
return ret;
}
- asix_read_cmd(dev, AX_CMD_READ_MII_REG, phy_id,
- (__u16)loc, 2, &res, 0);
- asix_set_hw_mii(dev, 0);
+ ret = asix_read_cmd(dev, AX_CMD_READ_MII_REG, phy_id, (__u16)loc, 2,
+ &res, 0);
+ if (ret < 0)
+ goto out;
+
+ ret = asix_set_hw_mii(dev, 0);
+out:
mutex_unlock(&dev->phy_mutex);
netdev_dbg(dev->net, "asix_mdio_read() phy_id=0x%02x, loc=0x%02x, returns=0x%04x\n",
phy_id, loc, le16_to_cpu(res));
- return le16_to_cpu(res);
+ return ret < 0 ? ret : le16_to_cpu(res);
}
-void asix_mdio_write(struct net_device *netdev, int phy_id, int loc, int val)
+static int __asix_mdio_write(struct net_device *netdev, int phy_id, int loc,
+ int val)
{
struct usbnet *dev = netdev_priv(netdev);
__le16 res = cpu_to_le16(val);
ret = asix_read_cmd(dev, AX_CMD_STATMNGSTS_REG,
0, 0, 1, &smsr, 0);
} while (!(smsr & AX_HOST_EN) && (i++ < 30) && (ret != -ENODEV));
- if (ret == -ENODEV) {
- mutex_unlock(&dev->phy_mutex);
- return;
- }
- asix_write_cmd(dev, AX_CMD_WRITE_MII_REG, phy_id,
- (__u16)loc, 2, &res, 0);
- asix_set_hw_mii(dev, 0);
+ if (ret == -ENODEV)
+ goto out;
+
+ ret = asix_write_cmd(dev, AX_CMD_WRITE_MII_REG, phy_id, (__u16)loc, 2,
+ &res, 0);
+ if (ret < 0)
+ goto out;
+
+ ret = asix_set_hw_mii(dev, 0);
+out:
mutex_unlock(&dev->phy_mutex);
+
+ return ret < 0 ? ret : 0;
+}
+
+void asix_mdio_write(struct net_device *netdev, int phy_id, int loc, int val)
+{
+ __asix_mdio_write(netdev, phy_id, loc, val);
+}
+
+/* MDIO read and write wrappers for phylib */
+int asix_mdio_bus_read(struct mii_bus *bus, int phy_id, int regnum)
+{
+ struct usbnet *priv = bus->priv;
+
+ return asix_mdio_read(priv->net, phy_id, regnum);
+}
+
+int asix_mdio_bus_write(struct mii_bus *bus, int phy_id, int regnum, u16 val)
+{
+ struct usbnet *priv = bus->priv;
+
+ return __asix_mdio_write(priv->net, phy_id, regnum, val);
}
int asix_mdio_read_nopm(struct net_device *netdev, int phy_id, int loc)
dev->mii.mdio_write = asix_mdio_write;
dev->mii.phy_id_mask = 0x3f;
dev->mii.reg_num_mask = 0x1f;
- dev->mii.phy_id = asix_get_phy_addr(dev);
+
+ dev->mii.phy_id = asix_read_phy_addr(dev, true);
+ if (dev->mii.phy_id < 0)
+ return dev->mii.phy_id;
dev->net->netdev_ops = &ax88172_netdev_ops;
dev->net->ethtool_ops = &ax88172_ethtool_ops;
return ret;
}
+static void ax88772_ethtool_get_strings(struct net_device *netdev, u32 sset,
+ u8 *data)
+{
+ switch (sset) {
+ case ETH_SS_TEST:
+ net_selftest_get_strings(data);
+ break;
+ }
+}
+
+static int ax88772_ethtool_get_sset_count(struct net_device *ndev, int sset)
+{
+ switch (sset) {
+ case ETH_SS_TEST:
+ return net_selftest_get_count();
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+
static const struct ethtool_ops ax88772_ethtool_ops = {
.get_drvinfo = asix_get_drvinfo,
- .get_link = asix_get_link,
+ .get_link = usbnet_get_link,
.get_msglevel = usbnet_get_msglevel,
.set_msglevel = usbnet_set_msglevel,
.get_wol = asix_get_wol,
.get_eeprom_len = asix_get_eeprom_len,
.get_eeprom = asix_get_eeprom,
.set_eeprom = asix_set_eeprom,
- .nway_reset = usbnet_nway_reset,
- .get_link_ksettings = usbnet_get_link_ksettings_mii,
- .set_link_ksettings = usbnet_set_link_ksettings_mii,
+ .nway_reset = phy_ethtool_nway_reset,
+ .get_link_ksettings = phy_ethtool_get_link_ksettings,
+ .set_link_ksettings = phy_ethtool_set_link_ksettings,
+ .self_test = net_selftest,
+ .get_strings = ax88772_ethtool_get_strings,
+ .get_sset_count = ax88772_ethtool_get_sset_count,
};
-static int ax88772_link_reset(struct usbnet *dev)
-{
- u16 mode;
- struct ethtool_cmd ecmd = { .cmd = ETHTOOL_GSET };
-
- mii_check_media(&dev->mii, 1, 1);
- mii_ethtool_gset(&dev->mii, &ecmd);
- mode = AX88772_MEDIUM_DEFAULT;
-
- if (ethtool_cmd_speed(&ecmd) != SPEED_100)
- mode &= ~AX_MEDIUM_PS;
-
- if (ecmd.duplex != DUPLEX_FULL)
- mode &= ~AX_MEDIUM_FD;
-
- netdev_dbg(dev->net, "ax88772_link_reset() speed: %u duplex: %d setting mode to 0x%04x\n",
- ethtool_cmd_speed(&ecmd), ecmd.duplex, mode);
-
- asix_write_medium_mode(dev, mode, 0);
-
- return 0;
-}
-
static int ax88772_reset(struct usbnet *dev)
{
struct asix_data *data = (struct asix_data *)&dev->data;
+ struct asix_common_private *priv = dev->driver_priv;
int ret;
/* Rewrite MAC address */
if (ret < 0)
goto out;
+ phy_start(priv->phydev);
+
return 0;
out:
.ndo_get_stats64 = dev_get_tstats64,
.ndo_set_mac_address = asix_set_mac_address,
.ndo_validate_addr = eth_validate_addr,
- .ndo_do_ioctl = asix_ioctl,
+ .ndo_do_ioctl = phy_do_ioctl_running,
.ndo_set_rx_mode = asix_set_multicast,
};
struct asix_common_private *priv = dev->driver_priv;
u16 medium;
+ if (netif_running(dev->net))
+ phy_stop(priv->phydev);
+
/* Stop MAC operation */
medium = asix_read_medium_status(dev, 1);
medium &= ~AX_MEDIUM_RE;
netdev_dbg(dev->net, "ax88772_suspend: medium=0x%04x\n",
asix_read_medium_status(dev, 1));
-
- /* Preserve BMCR for restoring */
- priv->presvd_phy_bmcr =
- asix_mdio_read_nopm(dev->net, dev->mii.phy_id, MII_BMCR);
-
- /* Preserve ANAR for restoring */
- priv->presvd_phy_advertise =
- asix_mdio_read_nopm(dev->net, dev->mii.phy_id, MII_ADVERTISE);
}
static int asix_suspend(struct usb_interface *intf, pm_message_t message)
return usbnet_suspend(intf, message);
}
-static void ax88772_restore_phy(struct usbnet *dev)
-{
- struct asix_common_private *priv = dev->driver_priv;
-
- if (priv->presvd_phy_advertise) {
- /* Restore Advertisement control reg */
- asix_mdio_write_nopm(dev->net, dev->mii.phy_id, MII_ADVERTISE,
- priv->presvd_phy_advertise);
-
- /* Restore BMCR */
- if (priv->presvd_phy_bmcr & BMCR_ANENABLE)
- priv->presvd_phy_bmcr |= BMCR_ANRESTART;
-
- asix_mdio_write_nopm(dev->net, dev->mii.phy_id, MII_BMCR,
- priv->presvd_phy_bmcr);
-
- priv->presvd_phy_advertise = 0;
- priv->presvd_phy_bmcr = 0;
- }
-}
-
static void ax88772_resume(struct usbnet *dev)
{
+ struct asix_common_private *priv = dev->driver_priv;
int i;
for (i = 0; i < 3; i++)
if (!ax88772_hw_reset(dev, 1))
break;
- ax88772_restore_phy(dev);
+
+ if (netif_running(dev->net))
+ phy_start(priv->phydev);
}
static void ax88772a_resume(struct usbnet *dev)
{
+ struct asix_common_private *priv = dev->driver_priv;
int i;
for (i = 0; i < 3; i++) {
break;
}
- ax88772_restore_phy(dev);
+ if (netif_running(dev->net))
+ phy_start(priv->phydev);
}
static int asix_resume(struct usb_interface *intf)
return usbnet_resume(intf);
}
+static int ax88772_init_mdio(struct usbnet *dev)
+{
+ struct asix_common_private *priv = dev->driver_priv;
+
+ priv->mdio = devm_mdiobus_alloc(&dev->udev->dev);
+ if (!priv->mdio)
+ return -ENOMEM;
+
+ priv->mdio->priv = dev;
+ priv->mdio->read = &asix_mdio_bus_read;
+ priv->mdio->write = &asix_mdio_bus_write;
+ priv->mdio->name = "Asix MDIO Bus";
+ /* mii bus name is usb-<usb bus number>-<usb device number> */
+ snprintf(priv->mdio->id, MII_BUS_ID_SIZE, "usb-%03d:%03d",
+ dev->udev->bus->busnum, dev->udev->devnum);
+
+ return devm_mdiobus_register(&dev->udev->dev, priv->mdio);
+}
+
+static int ax88772_init_phy(struct usbnet *dev)
+{
+ struct asix_common_private *priv = dev->driver_priv;
+ int ret;
+
+ ret = asix_read_phy_addr(dev, true);
+ if (ret < 0)
+ return ret;
+
+ priv->phy_addr = ret;
+
+ snprintf(priv->phy_name, sizeof(priv->phy_name), PHY_ID_FMT,
+ priv->mdio->id, priv->phy_addr);
+
+ priv->phydev = phy_connect(dev->net, priv->phy_name, &asix_adjust_link,
+ PHY_INTERFACE_MODE_INTERNAL);
+ if (IS_ERR(priv->phydev)) {
+ netdev_err(dev->net, "Could not connect to PHY device %s\n",
+ priv->phy_name);
+ ret = PTR_ERR(priv->phydev);
+ return ret;
+ }
+
+ priv->phydev->mac_managed_pm = 1;
+
+ phy_attached_info(priv->phydev);
+
+ return 0;
+}
+
static int ax88772_bind(struct usbnet *dev, struct usb_interface *intf)
{
- int ret, i;
u8 buf[ETH_ALEN] = {0}, chipcode = 0;
- u32 phyid;
struct asix_common_private *priv;
+ int ret, i;
+ u32 phyid;
usbnet_get_endpoints(dev, intf);
asix_set_netdev_dev_addr(dev, buf);
- /* Initialize MII structure */
- dev->mii.dev = dev->net;
- dev->mii.mdio_read = asix_mdio_read;
- dev->mii.mdio_write = asix_mdio_write;
- dev->mii.phy_id_mask = 0x1f;
- dev->mii.reg_num_mask = 0x1f;
- dev->mii.phy_id = asix_get_phy_addr(dev);
-
dev->net->netdev_ops = &ax88772_netdev_ops;
dev->net->ethtool_ops = &ax88772_ethtool_ops;
dev->net->needed_headroom = 4; /* cf asix_tx_fixup() */
dev->rx_urb_size = 2048;
}
- dev->driver_priv = kzalloc(sizeof(struct asix_common_private), GFP_KERNEL);
- if (!dev->driver_priv)
+ priv = devm_kzalloc(&dev->udev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
return -ENOMEM;
- priv = dev->driver_priv;
+ dev->driver_priv = priv;
priv->presvd_phy_bmcr = 0;
priv->presvd_phy_advertise = 0;
priv->suspend = ax88772_suspend;
}
+ ret = ax88772_init_mdio(dev);
+ if (ret)
+ return ret;
+
+ return ax88772_init_phy(dev);
+}
+
+static int ax88772_stop(struct usbnet *dev)
+{
+ struct asix_common_private *priv = dev->driver_priv;
+
+ /* On unplugged USB, we will get MDIO communication errors and the
+ * PHY will be set in to PHY_HALTED state.
+ */
+ if (priv->phydev->state != PHY_HALTED)
+ phy_stop(priv->phydev);
+
return 0;
}
static void ax88772_unbind(struct usbnet *dev, struct usb_interface *intf)
{
+ struct asix_common_private *priv = dev->driver_priv;
+
+ phy_disconnect(priv->phydev);
asix_rx_fixup_common_free(dev->driver_priv);
- kfree(dev->driver_priv);
}
static const struct ethtool_ops ax88178_ethtool_ops = {
dev->mii.phy_id_mask = 0x1f;
dev->mii.reg_num_mask = 0xff;
dev->mii.supports_gmii = 1;
- dev->mii.phy_id = asix_get_phy_addr(dev);
+
+ dev->mii.phy_id = asix_read_phy_addr(dev, true);
+ if (dev->mii.phy_id < 0)
+ return dev->mii.phy_id;
dev->net->netdev_ops = &ax88178_netdev_ops;
dev->net->ethtool_ops = &ax88178_ethtool_ops;
.bind = ax88772_bind,
.unbind = ax88772_unbind,
.status = asix_status,
- .link_reset = ax88772_link_reset,
.reset = ax88772_reset,
+ .stop = ax88772_stop,
.flags = FLAG_ETHER | FLAG_FRAMING_AX | FLAG_LINK_INTR | FLAG_MULTI_PACKET,
.rx_fixup = asix_rx_fixup_common,
.tx_fixup = asix_tx_fixup,
.bind = ax88772_bind,
.unbind = ax88772_unbind,
.status = asix_status,
- .link_reset = ax88772_link_reset,
.reset = ax88772_reset,
.flags = FLAG_ETHER | FLAG_FRAMING_AX | FLAG_LINK_INTR |
FLAG_MULTI_PACKET,
.bind = ax88772_bind,
.unbind = ax88772_unbind,
.status = asix_status,
- .link_reset = ax88772_link_reset,
.reset = ax88772_reset,
.flags = FLAG_ETHER | FLAG_FRAMING_AX | FLAG_LINK_INTR |
FLAG_MULTI_PACKET,
struct asix_rx_fixup_info rx_fixup_info;
};
-/* MDIO read and write wrappers for phylib */
-static int asix_mdio_bus_read(struct mii_bus *bus, int phy_id, int regnum)
-{
- return asix_mdio_read(((struct usbnet *)bus->priv)->net, phy_id,
- regnum);
-}
-
-static int asix_mdio_bus_write(struct mii_bus *bus, int phy_id, int regnum,
- u16 val)
-{
- asix_mdio_write(((struct usbnet *)bus->priv)->net, phy_id, regnum, val);
- return 0;
-}
-
/* set MAC link settings according to information from phylib */
static void ax88172a_adjust_link(struct net_device *netdev)
{
goto free;
}
- priv->phy_addr = asix_read_phy_addr(dev, priv->use_embdphy);
+ ret = asix_read_phy_addr(dev, priv->use_embdphy);
+ if (ret < 0)
+ goto free;
+
+ priv->phy_addr = ret;
+
ax88172a_reset_phy(dev, priv->use_embdphy);
/* Asix framing packs multiple eth frames into a 2K usb bulk transfer */
* for transport over USB using a simpler USB device model than the
* previous CDC "Ethernet Control Model" (ECM, or "CDC Ethernet").
*
- * For details, see www.usb.org/developers/devclass_docs/CDC_EEM10.pdf
+ * For details, see https://usb.org/sites/default/files/CDC_EEM10.pdf
*
* This version has been tested with GIGAntIC WuaoW SIM Smart Card on 2.6.24,
* 2.6.27 and 2.6.30rc2 kernel.
}
skb2 = skb_copy_expand(skb, EEM_HEAD, ETH_FCS_LEN + padlen, flags);
+ dev_kfree_skb_any(skb);
if (!skb2)
return NULL;
- dev_kfree_skb_any(skb);
skb = skb2;
done:
goto bad_desc;
}
skip:
- /* Communcation class functions with bmCapabilities are not
+ /* Communication class functions with bmCapabilities are not
* RNDIS. But some Wireless class RNDIS functions use
* bmCapabilities for their own purpose. The failsafe is
* therefore applied only to Communication class RNDIS
return NULL;
}
-/* Some devices are known to send Neigbor Solicitation messages and
- * require Neigbor Advertisement replies. The IPv6 core will not
+/* Some devices are known to send Neighbor Solicitation messages and
+ * require Neighbor Advertisement replies. The IPv6 core will not
* respond since IFF_NOARP is set, so we must handle them ourselves.
*/
static void do_neigh_solicit(struct usbnet *dev, u8 *buf, u16 tci)
*
* Note: The current implementation of this feature restricts each NTB
* to a single NDP, implying that multiplexed sessions cannot share an
- * NTB. This might affect performace for multiplexed sessions.
+ * NTB. This might affect performance for multiplexed sessions.
*/
static const struct driver_info cdc_mbim_info_ndp_to_end = {
.description = "CDC MBIM",
return val;
}
-static ssize_t cdc_ncm_show_min_tx_pkt(struct device *d, struct device_attribute *attr, char *buf)
+static ssize_t min_tx_pkt_show(struct device *d,
+ struct device_attribute *attr, char *buf)
{
struct usbnet *dev = netdev_priv(to_net_dev(d));
struct cdc_ncm_ctx *ctx = (struct cdc_ncm_ctx *)dev->data[0];
return sprintf(buf, "%u\n", ctx->min_tx_pkt);
}
-static ssize_t cdc_ncm_show_rx_max(struct device *d, struct device_attribute *attr, char *buf)
+static ssize_t rx_max_show(struct device *d,
+ struct device_attribute *attr, char *buf)
{
struct usbnet *dev = netdev_priv(to_net_dev(d));
struct cdc_ncm_ctx *ctx = (struct cdc_ncm_ctx *)dev->data[0];
return sprintf(buf, "%u\n", ctx->rx_max);
}
-static ssize_t cdc_ncm_show_tx_max(struct device *d, struct device_attribute *attr, char *buf)
+static ssize_t tx_max_show(struct device *d,
+ struct device_attribute *attr, char *buf)
{
struct usbnet *dev = netdev_priv(to_net_dev(d));
struct cdc_ncm_ctx *ctx = (struct cdc_ncm_ctx *)dev->data[0];
return sprintf(buf, "%u\n", ctx->tx_max);
}
-static ssize_t cdc_ncm_show_tx_timer_usecs(struct device *d, struct device_attribute *attr, char *buf)
+static ssize_t tx_timer_usecs_show(struct device *d,
+ struct device_attribute *attr, char *buf)
{
struct usbnet *dev = netdev_priv(to_net_dev(d));
struct cdc_ncm_ctx *ctx = (struct cdc_ncm_ctx *)dev->data[0];
return sprintf(buf, "%u\n", ctx->timer_interval / (u32)NSEC_PER_USEC);
}
-static ssize_t cdc_ncm_store_min_tx_pkt(struct device *d, struct device_attribute *attr, const char *buf, size_t len)
+static ssize_t min_tx_pkt_store(struct device *d,
+ struct device_attribute *attr,
+ const char *buf, size_t len)
{
struct usbnet *dev = netdev_priv(to_net_dev(d));
struct cdc_ncm_ctx *ctx = (struct cdc_ncm_ctx *)dev->data[0];
return len;
}
-static ssize_t cdc_ncm_store_rx_max(struct device *d, struct device_attribute *attr, const char *buf, size_t len)
+static ssize_t rx_max_store(struct device *d,
+ struct device_attribute *attr,
+ const char *buf, size_t len)
{
struct usbnet *dev = netdev_priv(to_net_dev(d));
struct cdc_ncm_ctx *ctx = (struct cdc_ncm_ctx *)dev->data[0];
return len;
}
-static ssize_t cdc_ncm_store_tx_max(struct device *d, struct device_attribute *attr, const char *buf, size_t len)
+static ssize_t tx_max_store(struct device *d,
+ struct device_attribute *attr,
+ const char *buf, size_t len)
{
struct usbnet *dev = netdev_priv(to_net_dev(d));
struct cdc_ncm_ctx *ctx = (struct cdc_ncm_ctx *)dev->data[0];
return len;
}
-static ssize_t cdc_ncm_store_tx_timer_usecs(struct device *d, struct device_attribute *attr, const char *buf, size_t len)
+static ssize_t tx_timer_usecs_store(struct device *d,
+ struct device_attribute *attr,
+ const char *buf, size_t len)
{
struct usbnet *dev = netdev_priv(to_net_dev(d));
struct cdc_ncm_ctx *ctx = (struct cdc_ncm_ctx *)dev->data[0];
return len;
}
-static DEVICE_ATTR(min_tx_pkt, 0644, cdc_ncm_show_min_tx_pkt, cdc_ncm_store_min_tx_pkt);
-static DEVICE_ATTR(rx_max, 0644, cdc_ncm_show_rx_max, cdc_ncm_store_rx_max);
-static DEVICE_ATTR(tx_max, 0644, cdc_ncm_show_tx_max, cdc_ncm_store_tx_max);
-static DEVICE_ATTR(tx_timer_usecs, 0644, cdc_ncm_show_tx_timer_usecs, cdc_ncm_store_tx_timer_usecs);
+static DEVICE_ATTR_RW(min_tx_pkt);
+static DEVICE_ATTR_RW(rx_max);
+static DEVICE_ATTR_RW(tx_max);
+static DEVICE_ATTR_RW(tx_timer_usecs);
static ssize_t ndp_to_end_show(struct device *d, struct device_attribute *attr, char *buf)
{
/* set MTU to max supported by the device if necessary */
dev->net->mtu = min_t(int, dev->net->mtu, ctx->max_datagram_size - cdc_ncm_eth_hlen(dev));
- /* do not exceed operater preferred MTU */
+ /* do not exceed operator preferred MTU */
if (ctx->mbim_extended_desc) {
mbim_mtu = le16_to_cpu(ctx->mbim_extended_desc->wMTU);
if (mbim_mtu != 0 && mbim_mtu < dev->net->mtu)
struct cdc_ncm_ctx *ctx = (struct cdc_ncm_ctx *)dev->data[0];
u32 def_rx, def_tx;
- /* be conservative when selecting intial buffer size to
+ /* be conservative when selecting initial buffer size to
* increase the number of hosts this will work for
*/
def_rx = min_t(u32, CDC_NCM_NTB_DEF_SIZE_RX,
static const struct driver_info cdc_ncm_info = {
.description = "CDC NCM",
.flags = FLAG_POINTTOPOINT | FLAG_NO_SETINT | FLAG_MULTI_PACKET
- | FLAG_LINK_INTR,
+ | FLAG_LINK_INTR | FLAG_ETHER,
.bind = cdc_ncm_bind,
.unbind = cdc_ncm_unbind,
.manage_power = usbnet_manage_power,
MODULE_DEVICE_TABLE(usb, hso_ids);
/* Sysfs attribute */
-static ssize_t hso_sysfs_show_porttype(struct device *dev,
- struct device_attribute *attr,
- char *buf)
+static ssize_t hsotype_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
{
struct hso_device *hso_dev = dev_get_drvdata(dev);
char *port_name;
return sprintf(buf, "%s\n", port_name);
}
-static DEVICE_ATTR(hsotype, 0444, hso_sysfs_show_porttype, NULL);
+static DEVICE_ATTR_RO(hsotype);
static struct attribute *hso_serial_dev_attrs[] = {
&dev_attr_hsotype.attr,
spin_unlock_irqrestore(&serial->serial_lock, flags);
return usb_control_msg(serial->parent->usb,
- usb_rcvctrlpipe(serial->parent->usb, 0), 0x22,
+ usb_sndctrlpipe(serial->parent->usb, 0), 0x22,
0x21, val, if_num, NULL, 0,
USB_CTRL_SET_TIMEOUT);
}
if (hso_dev->usb_gone)
rv = 0;
else
- rv = usb_control_msg(hso_dev->usb, usb_rcvctrlpipe(hso_dev->usb, 0),
+ rv = usb_control_msg(hso_dev->usb, usb_sndctrlpipe(hso_dev->usb, 0),
enabled ? 0x82 : 0x81, 0x40, 0, 0, NULL, 0,
USB_CTRL_SET_TIMEOUT);
mutex_unlock(&hso_dev->mutex);
num_urbs = 2;
serial->tiocmget = kzalloc(sizeof(struct hso_tiocmget),
GFP_KERNEL);
+ if (!serial->tiocmget)
+ goto exit;
serial->tiocmget->serial_state_notification
= kzalloc(sizeof(struct hso_serial_state_notification),
GFP_KERNEL);
- /* it isn't going to break our heart if serial->tiocmget
- * allocation fails don't bother checking this.
- */
- if (serial->tiocmget && serial->tiocmget->serial_state_notification) {
- tiocmget = serial->tiocmget;
- tiocmget->endp = hso_get_ep(interface,
- USB_ENDPOINT_XFER_INT,
- USB_DIR_IN);
- if (!tiocmget->endp) {
- dev_err(&interface->dev, "Failed to find INT IN ep\n");
- goto exit;
- }
-
- tiocmget->urb = usb_alloc_urb(0, GFP_KERNEL);
- if (tiocmget->urb) {
- mutex_init(&tiocmget->mutex);
- init_waitqueue_head(&tiocmget->waitq);
- } else
- hso_free_tiomget(serial);
+ if (!serial->tiocmget->serial_state_notification)
+ goto exit;
+ tiocmget = serial->tiocmget;
+ tiocmget->endp = hso_get_ep(interface,
+ USB_ENDPOINT_XFER_INT,
+ USB_DIR_IN);
+ if (!tiocmget->endp) {
+ dev_err(&interface->dev, "Failed to find INT IN ep\n");
+ goto exit;
}
- }
- else
+
+ tiocmget->urb = usb_alloc_urb(0, GFP_KERNEL);
+ if (!tiocmget->urb)
+ goto exit;
+
+ mutex_init(&tiocmget->mutex);
+ init_waitqueue_head(&tiocmget->waitq);
+ } else {
num_urbs = 1;
+ }
if (hso_serial_common_create(serial, num_urbs, BULK_URB_RX_SIZE,
BULK_URB_TX_SIZE))
int need_tail = 0;
__le16 *len;
- /* if packet and our header is smaler than 64 pad to 64 (+ ZLP) */
+ /* if packet and our header is smaller than 64 pad to 64 (+ ZLP) */
if ((pack_with_header_len) < dev->maxpacket)
need_tail = dev->maxpacket - pack_with_header_len + 1;
/*
struct lan78xx_priv {
struct lan78xx_net *dev;
u32 rfe_ctl;
- u32 mchash_table[DP_SEL_VHF_HASH_LEN]; /* multicat hash table */
+ u32 mchash_table[DP_SEL_VHF_HASH_LEN]; /* multicast hash table */
u32 pfilter_table[NUM_OF_MAF][2]; /* perfect filter table */
u32 vlan_table[DP_SEL_VHF_VLAN_LEN];
struct mutex dataport_mutex; /* for dataport access */
.get_strings = lan78xx_get_strings,
.get_wol = lan78xx_get_wol,
.set_wol = lan78xx_set_wol,
+ .get_ts_info = ethtool_op_get_ts_info,
.get_eee = lan78xx_get_eee,
.set_eee = lan78xx_set_eee,
.get_pauseparam = lan78xx_get_pause,
* Windows/Mac drivers do send a couple of such frames to the device
* during initialisation, with protocol set to 0x0906 or 0x0b06 and (what
* seems to be) a flag in the .dummy_flags. This doesn't seem necessary
- * for modem operation but can possibly be used for GPS or other funcitons.
+ * for modem operation but can possibly be used for GPS or other functions.
*/
struct vl600_frame_hdr {
/* ARP packets don't go through, but they're also of no use. The
* subnet has only two hosts anyway: us and the gateway / DHCP
* server (probably simulated by modem firmware or network operator)
- * whose address changes everytime we connect to the intarwebz and
+ * whose address changes every time we connect to the intarwebz and
* who doesn't bother answering ARP requests either. So hardware
* addresses have no meaning, the destination and the source of every
* packet depend only on whether it is on the IN or OUT endpoint. */
static int mcs7830_reset_resume (struct usb_interface *intf)
{
- /* YES, this function is successful enough that ethtool -d
+ /* YES, this function is successful enough that ethtool -d
does show same output pre-/post-suspend */
struct usbnet *dev = usb_get_intfdata(intf);
if (info->flags & QMI_WWAN_FLAG_PASS_THROUGH) {
skb->protocol = htons(ETH_P_MAP);
- return (netif_rx(skb) == NET_RX_SUCCESS);
+ return 1;
}
switch (skb->data[0] & 0xf0) {
u32 rx_pending;
u32 fc_pause_on, fc_pause_off;
+ unsigned int pipe_in, pipe_out, pipe_intr, pipe_ctrl_in, pipe_ctrl_out;
+
u32 support_2500full:1;
u32 lenovo_macpassthru:1;
u32 dell_tb_rx_agg_bug:1;
if (!tmp)
return -ENOMEM;
- ret = usb_control_msg(tp->udev, usb_rcvctrlpipe(tp->udev, 0),
+ ret = usb_control_msg(tp->udev, tp->pipe_ctrl_in,
RTL8152_REQ_GET_REGS, RTL8152_REQT_READ,
value, index, tmp, size, 500);
if (ret < 0)
if (!tmp)
return -ENOMEM;
- ret = usb_control_msg(tp->udev, usb_sndctrlpipe(tp->udev, 0),
+ ret = usb_control_msg(tp->udev, tp->pipe_ctrl_out,
RTL8152_REQ_SET_REGS, RTL8152_REQT_WRITE,
value, index, tmp, size, 500);
goto err1;
tp->intr_interval = (int)ep_intr->desc.bInterval;
- usb_fill_int_urb(tp->intr_urb, tp->udev, usb_rcvintpipe(tp->udev, 3),
+ usb_fill_int_urb(tp->intr_urb, tp->udev, tp->pipe_intr,
tp->intr_buff, INTBUFSIZE, intr_callback,
tp, tp->intr_interval);
if (ret < 0)
goto out_tx_fill;
- usb_fill_bulk_urb(agg->urb, tp->udev, usb_sndbulkpipe(tp->udev, 2),
+ usb_fill_bulk_urb(agg->urb, tp->udev, tp->pipe_out,
agg->head, (int)(tx_data - (u8 *)agg->head),
(usb_complete_t)write_bulk_callback, agg);
unsigned int pkt_len, rx_frag_head_sz;
struct sk_buff *skb;
- /* limite the skb numbers for rx_queue */
+ /* limit the skb numbers for rx_queue */
if (unlikely(skb_queue_len(&tp->rx_queue) >= 1000))
break;
!test_bit(WORK_ENABLE, &tp->flags) || !netif_carrier_ok(tp->netdev))
return 0;
- usb_fill_bulk_urb(agg->urb, tp->udev, usb_rcvbulkpipe(tp->udev, 1),
+ usb_fill_bulk_urb(agg->urb, tp->udev, tp->pipe_in,
agg->buffer, tp->rx_buf_sz,
(usb_complete_t)read_bulk_callback, agg);
tp->coalesce = 15000; /* 15 us */
}
+static bool rtl_check_vendor_ok(struct usb_interface *intf)
+{
+ struct usb_host_interface *alt = intf->cur_altsetting;
+ struct usb_endpoint_descriptor *in, *out, *intr;
+
+ if (usb_find_common_endpoints(alt, &in, &out, &intr, NULL) < 0) {
+ dev_err(&intf->dev, "Expected endpoints are not found\n");
+ return false;
+ }
+
+ /* Check Rx endpoint address */
+ if (usb_endpoint_num(in) != 1) {
+ dev_err(&intf->dev, "Invalid Rx endpoint address\n");
+ return false;
+ }
+
+ /* Check Tx endpoint address */
+ if (usb_endpoint_num(out) != 2) {
+ dev_err(&intf->dev, "Invalid Tx endpoint address\n");
+ return false;
+ }
+
+ /* Check interrupt endpoint address */
+ if (usb_endpoint_num(intr) != 3) {
+ dev_err(&intf->dev, "Invalid interrupt endpoint address\n");
+ return false;
+ }
+
+ return true;
+}
+
static bool rtl_vendor_mode(struct usb_interface *intf)
{
struct usb_host_interface *alt = intf->cur_altsetting;
int i, num_configs;
if (alt->desc.bInterfaceClass == USB_CLASS_VENDOR_SPEC)
- return true;
+ return rtl_check_vendor_ok(intf);
/* The vendor mode is not always config #1, so to find it out. */
udev = interface_to_usbdev(intf);
c = udev->config;
num_configs = udev->descriptor.bNumConfigurations;
+ if (num_configs < 2)
+ return false;
+
for (i = 0; i < num_configs; (i++, c++)) {
struct usb_interface_descriptor *desc = NULL;
}
}
- WARN_ON_ONCE(i == num_configs);
+ if (i == num_configs)
+ dev_err(&intf->dev, "Unexpected Device\n");
return false;
}
if (!tp)
return 0;
- /* reset the MAC adddress in case of policy change */
+ /* reset the MAC address in case of policy change */
if (determine_ethernet_addr(tp, &sa) >= 0) {
rtnl_lock();
dev_set_mac_address (tp->netdev, &sa, NULL);
{
switch (stringset) {
case ETH_SS_STATS:
- memcpy(data, *rtl8152_gstrings, sizeof(rtl8152_gstrings));
+ memcpy(data, rtl8152_gstrings, sizeof(rtl8152_gstrings));
break;
}
}
return 0;
}
+static void rtl8152_get_pauseparam(struct net_device *netdev, struct ethtool_pauseparam *pause)
+{
+ struct r8152 *tp = netdev_priv(netdev);
+ u16 bmcr, lcladv, rmtadv;
+ u8 cap;
+
+ if (usb_autopm_get_interface(tp->intf) < 0)
+ return;
+
+ mutex_lock(&tp->control);
+
+ bmcr = r8152_mdio_read(tp, MII_BMCR);
+ lcladv = r8152_mdio_read(tp, MII_ADVERTISE);
+ rmtadv = r8152_mdio_read(tp, MII_LPA);
+
+ mutex_unlock(&tp->control);
+
+ usb_autopm_put_interface(tp->intf);
+
+ if (!(bmcr & BMCR_ANENABLE)) {
+ pause->autoneg = 0;
+ pause->rx_pause = 0;
+ pause->tx_pause = 0;
+ return;
+ }
+
+ pause->autoneg = 1;
+
+ cap = mii_resolve_flowctrl_fdx(lcladv, rmtadv);
+
+ if (cap & FLOW_CTRL_RX)
+ pause->rx_pause = 1;
+
+ if (cap & FLOW_CTRL_TX)
+ pause->tx_pause = 1;
+}
+
+static int rtl8152_set_pauseparam(struct net_device *netdev, struct ethtool_pauseparam *pause)
+{
+ struct r8152 *tp = netdev_priv(netdev);
+ u16 old, new1;
+ u8 cap = 0;
+ int ret;
+
+ ret = usb_autopm_get_interface(tp->intf);
+ if (ret < 0)
+ return ret;
+
+ mutex_lock(&tp->control);
+
+ if (pause->autoneg && !(r8152_mdio_read(tp, MII_BMCR) & BMCR_ANENABLE)) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ if (pause->rx_pause)
+ cap |= FLOW_CTRL_RX;
+
+ if (pause->tx_pause)
+ cap |= FLOW_CTRL_TX;
+
+ old = r8152_mdio_read(tp, MII_ADVERTISE);
+ new1 = (old & ~(ADVERTISE_PAUSE_CAP | ADVERTISE_PAUSE_ASYM)) | mii_advertise_flowctrl(cap);
+ if (old != new1)
+ r8152_mdio_write(tp, MII_ADVERTISE, new1);
+
+out:
+ mutex_unlock(&tp->control);
+ usb_autopm_put_interface(tp->intf);
+
+ return ret;
+}
+
static const struct ethtool_ops ops = {
.supported_coalesce_params = ETHTOOL_COALESCE_USECS,
.get_drvinfo = rtl8152_get_drvinfo,
.set_tunable = rtl8152_set_tunable,
.get_ringparam = rtl8152_get_ringparam,
.set_ringparam = rtl8152_set_ringparam,
+ .get_pauseparam = rtl8152_get_pauseparam,
+ .set_pauseparam = rtl8152_set_pauseparam,
};
static int rtl8152_ioctl(struct net_device *netdev, struct ifreq *rq, int cmd)
if (!rtl_vendor_mode(intf))
return -ENODEV;
- if (intf->cur_altsetting->desc.bNumEndpoints < 3)
- return -ENODEV;
-
usb_reset_device(udev);
netdev = alloc_etherdev(sizeof(struct r8152));
if (!netdev) {
tp->intf = intf;
tp->version = version;
+ tp->pipe_ctrl_in = usb_rcvctrlpipe(udev, 0);
+ tp->pipe_ctrl_out = usb_sndctrlpipe(udev, 0);
+ tp->pipe_in = usb_rcvbulkpipe(udev, 1);
+ tp->pipe_out = usb_sndbulkpipe(udev, 2);
+ tp->pipe_intr = usb_rcvintpipe(udev, 3);
+
switch (version) {
case RTL_VER_01:
case RTL_VER_02:
* For RX we handle drivers that zero-pad to end-of-packet.
* Don't let userspace change these settings.
*
- * NOTE: there still seems to be wierdness here, as if we need
+ * NOTE: there still seems to be weirdness here, as if we need
* to do some more things to make sure WinCE targets accept this.
* They default to jumbograms of 8KB or 16KB, which is absurd
* for such low data rates and which is also more than Linux
ret = smsc75xx_wait_ready(dev, 0);
if (ret < 0) {
netdev_warn(dev->net, "device not ready in smsc75xx_bind\n");
- return ret;
+ goto free_pdata;
}
smsc75xx_init_mac_address(dev);
ret = smsc75xx_reset(dev);
if (ret < 0) {
netdev_warn(dev->net, "smsc75xx_reset error %d\n", ret);
- return ret;
+ goto cancel_work;
}
dev->net->netdev_ops = &smsc75xx_netdev_ops;
dev->hard_mtu = dev->net->mtu + dev->net->hard_header_len;
dev->net->max_mtu = MAX_SINGLE_PACKET_SIZE;
return 0;
+
+cancel_work:
+ cancel_work_sync(&pdata->set_multicast);
+free_pdata:
+ kfree(pdata);
+ dev->data[0] = 0;
+ return ret;
}
static void smsc75xx_unbind(struct usbnet *dev, struct usb_interface *intf)
cancel_work_sync(&pdata->set_multicast);
netif_dbg(dev, ifdown, dev->net, "free pdata\n");
kfree(pdata);
- pdata = NULL;
dev->data[0] = 0;
}
}
/*-------------------------------------------------------------------------*/
+static const char * const usbnet_event_names[] = {
+ [EVENT_TX_HALT] = "EVENT_TX_HALT",
+ [EVENT_RX_HALT] = "EVENT_RX_HALT",
+ [EVENT_RX_MEMORY] = "EVENT_RX_MEMORY",
+ [EVENT_STS_SPLIT] = "EVENT_STS_SPLIT",
+ [EVENT_LINK_RESET] = "EVENT_LINK_RESET",
+ [EVENT_RX_PAUSED] = "EVENT_RX_PAUSED",
+ [EVENT_DEV_ASLEEP] = "EVENT_DEV_ASLEEP",
+ [EVENT_DEV_OPEN] = "EVENT_DEV_OPEN",
+ [EVENT_DEVICE_REPORT_IDLE] = "EVENT_DEVICE_REPORT_IDLE",
+ [EVENT_NO_RUNTIME_PM] = "EVENT_NO_RUNTIME_PM",
+ [EVENT_RX_KILL] = "EVENT_RX_KILL",
+ [EVENT_LINK_CHANGE] = "EVENT_LINK_CHANGE",
+ [EVENT_SET_RX_MODE] = "EVENT_SET_RX_MODE",
+ [EVENT_NO_IP_ALIGN] = "EVENT_NO_IP_ALIGN",
+};
+
/* handles CDC Ethernet and many other network "bulk data" interfaces */
int usbnet_get_endpoints(struct usbnet *dev, struct usb_interface *intf)
{
{
set_bit (work, &dev->flags);
if (!schedule_work (&dev->kevent))
- netdev_dbg(dev->net, "kevent %d may have been dropped\n", work);
+ netdev_dbg(dev->net, "kevent %s may have been dropped\n", usbnet_event_names[work]);
else
- netdev_dbg(dev->net, "kevent %d scheduled\n", work);
+ netdev_dbg(dev->net, "kevent %s scheduled\n", usbnet_event_names[work]);
}
EXPORT_SYMBOL_GPL(usbnet_defer_kevent);
xdev->bus->bus_name, xdev->devpath,
dev->driver_info->description);
+ if (dev->driver_info->unbind)
+ dev->driver_info->unbind(dev, intf);
+
net = dev->net;
unregister_netdev (net);
usb_scuttle_anchored_urbs(&dev->deferred);
- if (dev->driver_info->unbind)
- dev->driver_info->unbind (dev, intf);
-
usb_kill_urb(dev->interrupt);
usb_free_urb(dev->interrupt);
kfree(dev->padding_pkt);
/* If whole_page, there is an offset between the beginning of the
* data and the allocated space, otherwise the data and the allocated
* space are aligned.
+ *
+ * Buffers with headroom use PAGE_SIZE as alloc size, see
+ * add_recvbuf_mergeable() + get_mergeable_buf_len()
*/
if (whole_page) {
/* Buffers with whole_page use PAGE_SIZE as alloc size,
len -= vi->hdr_len;
stats->bytes += len;
+ if (unlikely(len > GOOD_PACKET_LEN)) {
+ pr_debug("%s: rx error: len %u exceeds max size %d\n",
+ dev->name, len, GOOD_PACKET_LEN);
+ dev->stats.rx_length_errors++;
+ goto err_len;
+ }
rcu_read_lock();
xdp_prog = rcu_dereference(rq->xdp_prog);
if (xdp_prog) {
err_xdp:
rcu_read_unlock();
stats->xdp_drops++;
+err_len:
stats->drops++;
put_page(page);
xdp_xmit:
head_skb = NULL;
stats->bytes += len - vi->hdr_len;
+ if (unlikely(len > truesize)) {
+ pr_debug("%s: rx error: len %u exceeds truesize %lu\n",
+ dev->name, len, (unsigned long)ctx);
+ dev->stats.rx_length_errors++;
+ goto err_skb;
+ }
rcu_read_lock();
xdp_prog = rcu_dereference(rq->xdp_prog);
if (xdp_prog) {
}
rcu_read_unlock();
- if (unlikely(len > truesize)) {
- pr_debug("%s: rx error: len %u exceeds truesize %lu\n",
- dev->name, len, (unsigned long)ctx);
- dev->stats.rx_length_errors++;
- goto err_skb;
- }
-
head_skb = page_to_skb(vi, rq, page, offset, len, truesize, !xdp_prog,
metasize, !!headroom);
curr_skb = head_skb;
if (virtio_net_hdr_from_skb(skb, &hdr->hdr,
virtio_is_little_endian(vi->vdev), false,
0))
- BUG();
+ return -EPROTO;
if (vi->mergeable_rx_bufs)
hdr->num_buffers = 0;
ctx[rxq2vq(i)] = true;
}
- ret = vi->vdev->config->find_vqs(vi->vdev, total_vqs, vqs, callbacks,
- names, ctx, NULL);
+ ret = virtio_find_vqs_ctx(vi->vdev, total_vqs, vqs, callbacks,
+ names, ctx, NULL);
if (ret)
goto err_find;
int res;
/* we pre-allocate elements used in the spin-locked section (so that we
- * keep the spinlock as short as possibile).
+ * keep the spinlock as short as possible).
*/
new_me = vrf_map_elem_alloc(GFP_KERNEL);
if (!new_me)
dev->flags = IFF_MASTER | IFF_NOARP;
- /* MTU is irrelevant for VRF device; set to 64k similar to lo */
- dev->mtu = 64 * 1024;
-
/* similarly, oper state is irrelevant; set to up to avoid confusion */
dev->operstate = IF_OPER_UP;
netdev_lockdep_set_classes(dev);
* which breaks networking.
*/
dev->min_mtu = IPV6_MIN_MTU;
- dev->max_mtu = ETH_MAX_MTU;
+ dev->max_mtu = IP6_MAX_MTU;
+ dev->mtu = dev->max_mtu;
}
static int vrf_validate(struct nlattr *tb[], struct nlattr *data[],
network device.
You will need user-space utilities COSA or SRP boards for downloading
- the firmware to the cards and to set them up. Look at the
+ the firmware to the cards and to set them up. Look at the
<http://www.fi.muni.cz/~kas/cosa/> for more information. You can also
read the comment at the top of the <file:drivers/net/wan/cosa.c> for
details about the cards and the driver itself.
Generic HDLC driver currently supports raw HDLC, Cisco HDLC, Frame
Relay, synchronous Point-to-Point Protocol (PPP) and X.25.
- To compile this driver as a module, choose M here: the
+ To compile this driver as a module, choose M here: the
module will be called hdlc.
If unsure, say N.
#include "hd64570.h"
-
-static const char* version = "Moxa C101 driver version: 1.15";
-static const char* devname = "C101";
+static const char *version = "Moxa C101 driver version: 1.15";
+static const char *devname = "C101";
#undef DEBUG_PKT
#define DEBUG_RINGS
static char *hw; /* pointer to hw=xxx command line string */
-
typedef struct card_s {
struct net_device *dev;
spinlock_t lock; /* TX lock */
u8 page;
struct card_s *next_card;
-}card_t;
+} card_t;
typedef card_t port_t;
static card_t *first_card;
static card_t **new_card = &first_card;
-
#define sca_in(reg, card) readb((card)->win0base + C101_SCA + (reg))
#define sca_out(value, reg, card) writeb(value, (card)->win0base + C101_SCA + (reg))
#define sca_inw(reg, card) readw((card)->win0base + C101_SCA + (reg))
/* EDA address register must be set in EDAL, EDAH order - 8 bit ISA bus */
#define sca_outw(value, reg, card) do { \
writeb(value & 0xFF, (card)->win0base + C101_SCA + (reg)); \
- writeb((value >> 8 ) & 0xFF, (card)->win0base + C101_SCA + (reg + 1));\
-} while(0)
+ writeb((value >> 8) & 0xFF, (card)->win0base + C101_SCA + (reg + 1));\
+} while (0)
#define port_to_card(port) (port)
#define log_node(port) (0)
#define phy_node(port) (0)
#define winsize(card) (C101_WINDOW_SIZE)
#define win0base(card) ((card)->win0base)
-#define winbase(card) ((card)->win0base + 0x2000)
+#define winbase(card) ((card)->win0base + 0x2000)
#define get_port(card, port) (card)
static void sca_msci_intr(port_t *port);
-
static inline u8 sca_get_page(card_t *card)
{
return card->page;
writeb(page, card->win0base + C101_PAGE);
}
-
#include "hd64570.c"
-
static inline void set_carrier(port_t *port)
{
if (!(sca_in(MSCI1_OFFSET + ST3, port) & ST3_DCD))
netif_carrier_off(port_to_dev(port));
}
-
static void sca_msci_intr(port_t *port)
{
u8 stat = sca_in(MSCI0_OFFSET + ST1, port); /* read MSCI ST1 status */
set_carrier(port);
}
-
static void c101_set_iface(port_t *port)
{
u8 rxs = port->rxs & CLK_BRG_MASK;
u8 txs = port->txs & CLK_BRG_MASK;
- switch(port->settings.clock_type) {
+ switch (port->settings.clock_type) {
case CLOCK_INT:
rxs |= CLK_BRG_RX; /* TX clock */
txs |= CLK_RXCLK_TX; /* BRG output */
sca_set_port(port);
}
-
static int c101_open(struct net_device *dev)
{
port_t *port = dev_to_port(dev);
return 0;
}
-
static int c101_close(struct net_device *dev)
{
port_t *port = dev_to_port(dev);
return 0;
}
-
static int c101_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd)
{
const size_t size = sizeof(sync_serial_settings);
if (cmd != SIOCWANDEV)
return hdlc_ioctl(dev, ifr, cmd);
- switch(ifr->ifr_settings.type) {
+ switch (ifr->ifr_settings.type) {
case IF_GET_IFACE:
ifr->ifr_settings.type = IF_IFACE_SYNC_SERIAL;
if (ifr->ifr_settings.size < size) {
return 0;
case IF_IFACE_SYNC_SERIAL:
- if(!capable(CAP_NET_ADMIN))
+ if (!capable(CAP_NET_ADMIN))
return -EPERM;
if (copy_from_user(&new_line, line, size))
}
}
-
-
static void c101_destroy_card(card_t *card)
{
readb(card->win0base + C101_PAGE); /* Resets SCA? */
card_t *card;
int result;
- if (irq<3 || irq>15 || irq == 6) /* FIXME */ {
+ if (irq < 3 || irq > 15 || irq == 6) /* FIXME */ {
pr_err("invalid IRQ value\n");
return -ENODEV;
}
- if (winbase < 0xC0000 || winbase > 0xDFFFF || (winbase & 0x3FFF) !=0) {
+ if (winbase < 0xC0000 || winbase > 0xDFFFF || (winbase & 0x3FFF) != 0) {
pr_err("invalid RAM value\n");
return -ENODEV;
}
card = kzalloc(sizeof(card_t), GFP_KERNEL);
- if (card == NULL)
+ if (!card)
return -ENOBUFS;
card->dev = alloc_hdlcdev(card);
return 0;
}
-
-
static int __init c101_init(void)
{
- if (hw == NULL) {
+ if (!hw) {
#ifdef MODULE
pr_info("no card initialized\n");
#endif
if (*hw == '\x0')
return first_card ? 0 : -EINVAL;
- }while(*hw++ == ':');
+ } while (*hw++ == ':');
pr_err("invalid hardware parameters\n");
return first_card ? 0 : -EINVAL;
}
-
static void __exit c101_cleanup(void)
{
card_t *card = first_card;
while (card) {
card_t *ptr = card;
+
card = card->next_card;
unregister_hdlc_device(port_to_dev(ptr));
c101_destroy_card(ptr);
}
}
-
module_init(c101_init);
module_exit(c101_cleanup);
// SPDX-License-Identifier: GPL-2.0-or-later
/* $Id: cosa.c,v 1.31 2000/03/08 17:47:16 kas Exp $ */
-/*
- * Copyright (C) 1995-1997 Jan "Yenya" Kasprzak <kas@fi.muni.cz>
+/* Copyright (C) 1995-1997 Jan "Yenya" Kasprzak <kas@fi.muni.cz>
* Generic HDLC port Copyright (C) 2008 Krzysztof Halasa <khc@pm.waw.pl>
*/
-/*
- * The driver for the SRP and COSA synchronous serial cards.
+/* The driver for the SRP and COSA synchronous serial cards.
*
* HARDWARE INFO
*
#define COSA_MAX_ID_STRING 128
/* Maximum length of the channel name */
-#define COSA_MAX_NAME (sizeof("cosaXXXcXXX")+1)
+#define COSA_MAX_NAME (sizeof("cosaXXXcXXX") + 1)
/* Per-channel data structure */
};
/* cosa->firmware_status bits */
-#define COSA_FW_RESET (1<<0) /* Is the ROM monitor active? */
-#define COSA_FW_DOWNLOAD (1<<1) /* Is the microcode downloaded? */
-#define COSA_FW_START (1<<2) /* Is the microcode running? */
+#define COSA_FW_RESET BIT(0) /* Is the ROM monitor active? */
+#define COSA_FW_DOWNLOAD BIT(1) /* Is the microcode downloaded? */
+#define COSA_FW_START BIT(2) /* Is the microcode running? */
struct cosa_data {
int num; /* Card number */
char *type; /* card type */
};
-/*
- * Define this if you want all the possible ports to be autoprobed.
+/* Define this if you want all the possible ports to be autoprobed.
* It is here but it probably is not a good idea to use this.
*/
-/* #define COSA_ISA_AUTOPROBE 1 */
+/* #define COSA_ISA_AUTOPROBE 1*/
-/*
- * Character device major number. 117 was allocated for us.
+/* Character device major number. 117 was allocated for us.
* The value of 0 means to allocate a first free one.
*/
static DEFINE_MUTEX(cosa_chardev_mutex);
static int cosa_major = 117;
-/*
- * Encoding of the minor numbers:
+/* Encoding of the minor numbers:
* The lowest CARD_MINOR_BITS bits means the channel on the single card,
* the highest bits means the card number.
*/
#define CARD_MINOR_BITS 4 /* How many bits in minor number are reserved
- * for the single card */
-/*
- * The following depends on CARD_MINOR_BITS. Unfortunately, the "MODULE_STRING"
+ * for the single card
+ */
+/* The following depends on CARD_MINOR_BITS. Unfortunately, the "MODULE_STRING"
* macro doesn't like anything other than the raw number as an argument :-(
*/
#define MAX_CARDS 16
#define DRIVER_TXMAP_SHIFT 2
#define DRIVER_TXMAP_MASK 0x0c /* FIXME: 0xfc for 8-channel version */
-/*
- * for cosa->rxtx - indicates whether either transmit or receive is
+/* for cosa->rxtx - indicates whether either transmit or receive is
* in progress. These values are mean number of the bit.
*/
#define TXBIT 0
#undef DEBUG_IRQS //1 /* Print the message when the IRQ is received */
#undef DEBUG_IO //1 /* Dump the I/O traffic */
-#define TX_TIMEOUT (5*HZ)
+#define TX_TIMEOUT (5 * HZ)
/* Maybe the following should be allocated dynamically */
static struct cosa_data cosa_cards[MAX_CARDS];
static int nr_cards;
#ifdef COSA_ISA_AUTOPROBE
-static int io[MAX_CARDS+1] = { 0x220, 0x228, 0x210, 0x218, 0, };
+static int io[MAX_CARDS + 1] = {0x220, 0x228, 0x210, 0x218, 0,};
/* NOTE: DMA is not autoprobed!!! */
-static int dma[MAX_CARDS+1] = { 1, 7, 1, 7, 1, 7, 1, 7, 0, };
+static int dma[MAX_CARDS + 1] = {1, 7, 1, 7, 1, 7, 1, 7, 0,};
#else
-static int io[MAX_CARDS+1];
-static int dma[MAX_CARDS+1];
+static int io[MAX_CARDS + 1];
+static int dma[MAX_CARDS + 1];
#endif
/* IRQ can be safely autoprobed */
-static int irq[MAX_CARDS+1] = { -1, -1, -1, -1, -1, -1, 0, };
+static int irq[MAX_CARDS + 1] = {-1, -1, -1, -1, -1, -1, 0,};
/* for class stuff*/
static struct class *cosa_class;
#define cosa_inw inw
#endif
-#define is_8bit(cosa) (!(cosa->datareg & 0x08))
+#define is_8bit(cosa) (!((cosa)->datareg & 0x08))
-#define cosa_getstatus(cosa) (cosa_inb(cosa->statusreg))
-#define cosa_putstatus(cosa, stat) (cosa_outb(stat, cosa->statusreg))
-#define cosa_getdata16(cosa) (cosa_inw(cosa->datareg))
-#define cosa_getdata8(cosa) (cosa_inb(cosa->datareg))
-#define cosa_putdata16(cosa, dt) (cosa_outw(dt, cosa->datareg))
-#define cosa_putdata8(cosa, dt) (cosa_outb(dt, cosa->datareg))
+#define cosa_getstatus(cosa) (cosa_inb((cosa)->statusreg))
+#define cosa_putstatus(cosa, stat) (cosa_outb(stat, (cosa)->statusreg))
+#define cosa_getdata16(cosa) (cosa_inw((cosa)->datareg))
+#define cosa_getdata8(cosa) (cosa_inb((cosa)->datareg))
+#define cosa_putdata16(cosa, dt) (cosa_outw(dt, (cosa)->datareg))
+#define cosa_putdata8(cosa, dt) (cosa_outb(dt, (cosa)->datareg))
/* Initialization stuff */
static int cosa_probe(int ioaddr, int irq, int dma);
static int chrdev_rx_done(struct channel_data *channel);
static int chrdev_tx_done(struct channel_data *channel, int size);
static ssize_t cosa_read(struct file *file,
- char __user *buf, size_t count, loff_t *ppos);
+ char __user *buf, size_t count, loff_t *ppos);
static ssize_t cosa_write(struct file *file,
- const char __user *buf, size_t count, loff_t *ppos);
+ const char __user *buf, size_t count, loff_t *ppos);
static unsigned int cosa_poll(struct file *file, poll_table *poll);
static int cosa_open(struct inode *inode, struct file *file);
static int cosa_release(struct inode *inode, struct file *file);
static long cosa_chardev_ioctl(struct file *file, unsigned int cmd,
- unsigned long arg);
+ unsigned long arg);
#ifdef COSA_FASYNC_WORKING
static int cosa_fasync(struct inode *inode, struct file *file, int on);
#endif
static void debug_status_out(struct cosa_data *cosa, int status);
#endif
-static inline struct channel_data* dev_to_chan(struct net_device *dev)
+static inline struct channel_data *dev_to_chan(struct net_device *dev)
{
return (struct channel_data *)dev_to_hdlc(dev)->priv;
}
goto out;
}
} else {
- if (!(cosa_major=register_chrdev(0, "cosa", &cosa_fops))) {
+ cosa_major = register_chrdev(0, "cosa", &cosa_fops);
+ if (!cosa_major) {
pr_warn("unable to register chardev\n");
err = -EIO;
goto out;
}
}
- for (i=0; i<MAX_CARDS; i++)
+ for (i = 0; i < MAX_CARDS; i++)
cosa_cards[i].num = -1;
- for (i=0; io[i] != 0 && i < MAX_CARDS; i++)
+ for (i = 0; io[i] != 0 && i < MAX_CARDS; i++)
cosa_probe(io[i], irq[i], dma[i]);
if (!nr_cards) {
pr_warn("no devices found\n");
static int cosa_probe(int base, int irq, int dma)
{
- struct cosa_data *cosa = cosa_cards+nr_cards;
+ struct cosa_data *cosa = cosa_cards + nr_cards;
int i, err = 0;
memset(cosa, 0, sizeof(struct cosa_data));
return -1;
}
/* I/O address should be between 0x100 and 0x3ff and should be
- * multiple of 8. */
+ * multiple of 8.
+ */
if (base < 0x100 || base > 0x3ff || base & 0x7) {
pr_info("invalid I/O address 0x%x\n", base);
return -1;
pr_info("invalid DMA %d\n", dma);
return -1;
}
- /* and finally, on 16-bit COSA DMA should be 4-7 and
- * I/O base should not be multiple of 0x10 */
+ /* and finally, on 16-bit COSA DMA should be 4-7 and
+ * I/O base should not be multiple of 0x10
+ */
if (((base & 0x8) && dma < 4) || (!(base & 0x8) && dma > 3)) {
pr_info("8/16 bit base and DMA mismatch (base=0x%x, dma=%d)\n",
base, dma);
cosa->dma = dma;
cosa->datareg = base;
- cosa->statusreg = is_8bit(cosa)?base+1:base+2;
+ cosa->statusreg = is_8bit(cosa) ? base + 1 : base + 2;
spin_lock_init(&cosa->lock);
- if (!request_region(base, is_8bit(cosa)?2:4,"cosa"))
+ if (!request_region(base, is_8bit(cosa) ? 2 : 4, "cosa"))
return -1;
-
+
if (cosa_reset_and_read_id(cosa, cosa->id_string) < 0) {
printk(KERN_DEBUG "probe at 0x%x failed.\n", base);
err = -1;
}
/* Test the validity of identification string */
- if (!strncmp(cosa->id_string, "SRP", 3))
+ if (!strncmp(cosa->id_string, "SRP", 3)) {
cosa->type = "srp";
- else if (!strncmp(cosa->id_string, "COSA", 4))
- cosa->type = is_8bit(cosa)? "cosa8": "cosa16";
- else {
+ } else if (!strncmp(cosa->id_string, "COSA", 4)) {
+ cosa->type = is_8bit(cosa) ? "cosa8" : "cosa16";
+ } else {
/* Print a warning only if we are not autoprobing */
#ifndef COSA_ISA_AUTOPROBE
pr_info("valid signature not found at 0x%x\n", base);
err = -1;
goto err_out;
}
- /* Update the name of the region now we know the type of card */
- release_region(base, is_8bit(cosa)?2:4);
- if (!request_region(base, is_8bit(cosa)?2:4, cosa->type)) {
+ /* Update the name of the region now we know the type of card */
+ release_region(base, is_8bit(cosa) ? 2 : 4);
+ if (!request_region(base, is_8bit(cosa) ? 2 : 4, cosa->type)) {
printk(KERN_DEBUG "changing name at 0x%x failed.\n", base);
return -1;
}
unsigned long irqs;
/* pr_info("IRQ autoprobe\n"); */
irqs = probe_irq_on();
- /*
- * Enable interrupt on tx buffer empty (it sure is)
+ /* Enable interrupt on tx buffer empty (it sure is)
* really sure ?
* FIXME: When this code is not used as module, we should
* probably call udelay() instead of the interruptible sleep.
err = -1;
goto err_out1;
}
-
- cosa->bouncebuf = kmalloc(COSA_MTU, GFP_KERNEL|GFP_DMA);
+
+ cosa->bouncebuf = kmalloc(COSA_MTU, GFP_KERNEL | GFP_DMA);
if (!cosa->bouncebuf) {
err = -ENOMEM;
goto err_out2;
sema_init(&chan->wsem, 1);
/* Register the network interface */
- if (!(chan->netdev = alloc_hdlcdev(chan))) {
+ chan->netdev = alloc_hdlcdev(chan);
+ if (!chan->netdev) {
pr_warn("%s: alloc_hdlcdev failed\n", chan->name);
err = -ENOMEM;
goto err_hdlcdev;
err_out1:
free_irq(cosa->irq, cosa);
err_out:
- release_region(cosa->datareg,is_8bit(cosa)?2:4);
+ release_region(cosa->datareg, is_8bit(cosa) ? 2 : 4);
pr_notice("cosa%d: allocating resources failed\n", cosa->num);
return err;
}
-\f
/*---------- network device ---------- */
static int cosa_net_attach(struct net_device *dev, unsigned short encoding,
}
static netdev_tx_t cosa_net_tx(struct sk_buff *skb,
- struct net_device *dev)
+ struct net_device *dev)
{
struct channel_data *chan = dev_to_chan(dev);
static char *cosa_net_setup_rx(struct channel_data *chan, int size)
{
- /*
- * We can safely fall back to non-dma-able memory, because we have
+ /* We can safely fall back to non-dma-able memory, because we have
* the cosa->bouncebuf pre-allocated.
*/
kfree_skb(chan->rx_skb);
chan->rx_skb = dev_alloc_skb(size);
- if (chan->rx_skb == NULL) {
+ if (!chan->rx_skb) {
pr_notice("%s: Memory squeeze, dropping packet\n", chan->name);
chan->netdev->stats.rx_dropped++;
return NULL;
/*---------- Character device ---------- */
static ssize_t cosa_read(struct file *file,
- char __user *buf, size_t count, loff_t *ppos)
+ char __user *buf, size_t count, loff_t *ppos)
{
DECLARE_WAITQUEUE(wait, current);
unsigned long flags;
}
if (mutex_lock_interruptible(&chan->rlock))
return -ERESTARTSYS;
-
- chan->rxdata = kmalloc(COSA_MTU, GFP_DMA|GFP_KERNEL);
- if (chan->rxdata == NULL) {
+
+ chan->rxdata = kmalloc(COSA_MTU, GFP_DMA | GFP_KERNEL);
+ if (!chan->rxdata) {
mutex_unlock(&chan->rlock);
return -ENOMEM;
}
return 1;
}
-
static ssize_t cosa_write(struct file *file,
- const char __user *buf, size_t count, loff_t *ppos)
+ const char __user *buf, size_t count, loff_t *ppos)
{
DECLARE_WAITQUEUE(wait, current);
struct channel_data *chan = file->private_data;
if (count > COSA_MTU)
count = COSA_MTU;
-
+
/* Allocate the buffer */
- kbuf = kmalloc(count, GFP_KERNEL|GFP_DMA);
- if (kbuf == NULL) {
+ kbuf = kmalloc(count, GFP_KERNEL | GFP_DMA);
+ if (!kbuf) {
up(&chan->wsem);
return -ENOMEM;
}
kfree(kbuf);
return -EFAULT;
}
- chan->tx_status=0;
+ chan->tx_status = 0;
cosa_start_tx(chan, kbuf, count);
spin_lock_irqsave(&cosa->lock, flags);
int ret = 0;
mutex_lock(&cosa_chardev_mutex);
- if ((n=iminor(file_inode(file))>>CARD_MINOR_BITS)
- >= nr_cards) {
+ n = iminor(file_inode(file)) >> CARD_MINOR_BITS;
+ if (n >= nr_cards) {
ret = -ENODEV;
goto out;
}
- cosa = cosa_cards+n;
+ cosa = cosa_cards + n;
- if ((n=iminor(file_inode(file))
- & ((1<<CARD_MINOR_BITS)-1)) >= cosa->nchannels) {
+ n = iminor(file_inode(file)) & ((1 << CARD_MINOR_BITS) - 1);
+ if (n >= cosa->nchannels) {
ret = -ENODEV;
goto out;
}
chan = cosa->chan + n;
-
+
file->private_data = chan;
spin_lock_irqsave(&cosa->lock, flags);
/* To be done ... */
static int cosa_fasync(struct inode *inode, struct file *file, int on)
{
- int port = iminor(inode);
+ int port = iminor(inode);
return fasync_helper(inode, file, on, &fasync[port]);
}
#endif
-\f
/* ---------- Ioctls ---------- */
-/*
- * Ioctl subroutines can safely be made inline, because they are called
+/* Ioctl subroutines can safely be made inline, because they are called
* only from cosa_ioctl().
*/
static inline int cosa_reset(struct cosa_data *cosa)
{
char idstring[COSA_MAX_ID_STRING];
+
if (cosa->usage > 1)
pr_info("cosa%d: WARNING: reset requested with cosa->usage > 1 (%d). Odd things may happen.\n",
cosa->num, cosa->usage);
- cosa->firmware_status &= ~(COSA_FW_RESET|COSA_FW_START);
+ cosa->firmware_status &= ~(COSA_FW_RESET | COSA_FW_START);
if (cosa_reset_and_read_id(cosa, idstring) < 0) {
pr_notice("cosa%d: reset failed\n", cosa->num);
return -EIO;
cosa->name, cosa->firmware_status);
return -EPERM;
}
-
+
if (copy_from_user(&d, arg, sizeof(d)))
return -EFAULT;
if (d.len < 0 || d.len > COSA_MAX_FIRMWARE_SIZE)
return -EINVAL;
-
/* If something fails, force the user to reset the card */
- cosa->firmware_status &= ~(COSA_FW_RESET|COSA_FW_DOWNLOAD);
+ cosa->firmware_status &= ~(COSA_FW_RESET | COSA_FW_DOWNLOAD);
i = download(cosa, d.code, d.len, d.addr);
if (i < 0) {
}
pr_info("cosa%d: downloading microcode - 0x%04x bytes at 0x%04x\n",
cosa->num, d.len, d.addr);
- cosa->firmware_status |= COSA_FW_RESET|COSA_FW_DOWNLOAD;
+ cosa->firmware_status |= COSA_FW_RESET | COSA_FW_DOWNLOAD;
return 0;
}
pr_info("cosa%d: WARNING: start microcode requested with cosa->usage > 1 (%d). Odd things may happen.\n",
cosa->num, cosa->usage);
- if ((cosa->firmware_status & (COSA_FW_RESET|COSA_FW_DOWNLOAD))
- != (COSA_FW_RESET|COSA_FW_DOWNLOAD)) {
+ if ((cosa->firmware_status & (COSA_FW_RESET | COSA_FW_DOWNLOAD))
+ != (COSA_FW_RESET | COSA_FW_DOWNLOAD)) {
pr_notice("%s: download the microcode and/or reset the card first (status %d)\n",
cosa->name, cosa->firmware_status);
return -EPERM;
}
cosa->firmware_status &= ~COSA_FW_RESET;
- if ((i=startmicrocode(cosa, address)) < 0) {
+ i = startmicrocode(cosa, address);
+ if (i < 0) {
pr_notice("cosa%d: start microcode at 0x%04x failed: %d\n",
cosa->num, address, i);
return -EIO;
cosa->firmware_status |= COSA_FW_START;
return 0;
}
-
+
/* Buffer of size at least COSA_MAX_ID_STRING is expected */
static inline int cosa_getidstr(struct cosa_data *cosa, char __user *string)
{
- int l = strlen(cosa->id_string)+1;
+ int l = strlen(cosa->id_string) + 1;
+
if (copy_to_user(string, cosa->id_string, l))
return -EFAULT;
return l;
/* Buffer of size at least COSA_MAX_ID_STRING is expected */
static inline int cosa_gettype(struct cosa_data *cosa, char __user *string)
{
- int l = strlen(cosa->type)+1;
+ int l = strlen(cosa->type) + 1;
+
if (copy_to_user(string, cosa->type, l))
return -EFAULT;
return l;
}
static int cosa_ioctl_common(struct cosa_data *cosa,
- struct channel_data *channel, unsigned int cmd, unsigned long arg)
+ struct channel_data *channel, unsigned int cmd,
+ unsigned long arg)
{
void __user *argp = (void __user *)arg;
+
switch (cmd) {
case COSAIORSET: /* Reset the device */
if (!capable(CAP_NET_ADMIN))
case COSAIODOWNLD: /* Download the firmware */
if (!capable(CAP_SYS_RAWIO))
return -EACCES;
-
+
return cosa_download(cosa, argp);
case COSAIORMEM:
if (!capable(CAP_SYS_RAWIO))
{
int rv;
struct channel_data *chan = dev_to_chan(dev);
+
rv = cosa_ioctl_common(chan->cosa, chan, cmd,
(unsigned long)ifr->ifr_data);
if (rv != -ENOIOCTLCMD)
}
static long cosa_chardev_ioctl(struct file *file, unsigned int cmd,
- unsigned long arg)
+ unsigned long arg)
{
struct channel_data *channel = file->private_data;
struct cosa_data *cosa;
return ret;
}
-\f
/*---------- HW layer interface ---------- */
-/*
- * The higher layer can bind itself to the HW layer by setting the callbacks
+/* The higher layer can bind itself to the HW layer by setting the callbacks
* in the channel_data structure and by using these routines.
*/
static void cosa_enable_rx(struct channel_data *chan)
put_driver_status(cosa);
}
-/*
- * FIXME: This routine probably should check for cosa_start_tx() called when
+/* FIXME: This routine probably should check for cosa_start_tx() called when
* the previous transmit is still unfinished. In this case the non-zero
* return value should indicate to the caller that the queuing(sp?) up
* the transmit has failed.
pr_info("cosa%dc%d: starting tx(0x%x)",
chan->cosa->num, chan->num, len);
- for (i=0; i<len; i++)
+ for (i = 0; i < len; i++)
pr_cont(" %02x", buf[i]&0xff);
pr_cont("\n");
#endif
status = (cosa->rxbitmap ? DRIVER_RX_READY : 0)
| (cosa->txbitmap ? DRIVER_TX_READY : 0)
- | (cosa->txbitmap? ~(cosa->txbitmap<<DRIVER_TXMAP_SHIFT)
- &DRIVER_TXMAP_MASK : 0);
+ | (cosa->txbitmap ? ~(cosa->txbitmap << DRIVER_TXMAP_SHIFT)
+ & DRIVER_TXMAP_MASK : 0);
if (!cosa->rxtx) {
- if (cosa->rxbitmap|cosa->txbitmap) {
+ if (cosa->rxbitmap | cosa->txbitmap) {
if (!cosa->enabled) {
cosa_putstatus(cosa, SR_RX_INT_ENA);
#ifdef DEBUG_IO
status = (cosa->rxbitmap ? DRIVER_RX_READY : 0)
| (cosa->txbitmap ? DRIVER_TX_READY : 0)
- | (cosa->txbitmap? ~(cosa->txbitmap<<DRIVER_TXMAP_SHIFT)
- &DRIVER_TXMAP_MASK : 0);
+ | (cosa->txbitmap ? ~(cosa->txbitmap << DRIVER_TXMAP_SHIFT)
+ & DRIVER_TXMAP_MASK : 0);
- if (cosa->rxbitmap|cosa->txbitmap) {
+ if (cosa->rxbitmap | cosa->txbitmap) {
cosa_putstatus(cosa, SR_RX_INT_ENA);
#ifdef DEBUG_IO
debug_status_out(cosa, SR_RX_INT_ENA);
#endif
}
-/*
- * The "kickme" function: When the DMA times out, this is called to
+/* The "kickme" function: When the DMA times out, this is called to
* clean up the driver status.
* FIXME: Preliminary support, the interface is probably wrong.
*/
udelay(100);
cosa_putstatus(cosa, 0);
udelay(100);
- (void) cosa_getdata8(cosa);
+ (void)cosa_getdata8(cosa);
udelay(100);
cosa_putdata8(cosa, 0);
udelay(100);
spin_unlock_irqrestore(&cosa->lock, flags);
}
-/*
- * Check if the whole buffer is DMA-able. It means it is below the 16M of
+/* Check if the whole buffer is DMA-able. It means it is below the 16M of
* physical memory and doesn't span the 64k boundary. For now it seems
* SKB's never do this, but we'll check this anyway.
*/
{
static int count;
unsigned long b = (unsigned long)buf;
- if (b+len >= MAX_DMA_ADDRESS)
+
+ if (b + len >= MAX_DMA_ADDRESS)
return 0;
- if ((b^ (b+len)) & 0x10000) {
+ if ((b ^ (b + len)) & 0x10000) {
if (count++ < 5)
pr_info("%s: packet spanning a 64k boundary\n",
chan->name);
return 1;
}
-\f
/* ---------- The SRP/COSA ROM monitor functions ---------- */
-/*
- * Downloading SRP microcode: say "w" to SRP monitor, it answers by "w=",
+/* Downloading SRP microcode: say "w" to SRP monitor, it answers by "w=",
* drivers need to say 4-digit hex number meaning start address of the microcode
* separated by a single space. Monitor replies by saying " =". Now driver
* has to write 4-digit hex number meaning the last byte address ended
{
int i;
- if (put_wait_data(cosa, 'w') == -1) return -1;
+ if (put_wait_data(cosa, 'w') == -1)
+ return -1;
if ((i=get_wait_data(cosa)) != 'w') { printk("dnld: 0x%04x\n",i); return -2;}
- if (get_wait_data(cosa) != '=') return -3;
-
- if (puthexnumber(cosa, address) < 0) return -4;
- if (put_wait_data(cosa, ' ') == -1) return -10;
- if (get_wait_data(cosa) != ' ') return -11;
- if (get_wait_data(cosa) != '=') return -12;
-
- if (puthexnumber(cosa, address+length-1) < 0) return -13;
- if (put_wait_data(cosa, ' ') == -1) return -18;
- if (get_wait_data(cosa) != ' ') return -19;
+ if (get_wait_data(cosa) != '=')
+ return -3;
+
+ if (puthexnumber(cosa, address) < 0)
+ return -4;
+ if (put_wait_data(cosa, ' ') == -1)
+ return -10;
+ if (get_wait_data(cosa) != ' ')
+ return -11;
+ if (get_wait_data(cosa) != '=')
+ return -12;
+
+ if (puthexnumber(cosa, address + length - 1) < 0)
+ return -13;
+ if (put_wait_data(cosa, ' ') == -1)
+ return -18;
+ if (get_wait_data(cosa) != ' ')
+ return -19;
while (length--) {
char c;
microcode++;
}
- if (get_wait_data(cosa) != '\r') return -21;
- if (get_wait_data(cosa) != '\n') return -22;
- if (get_wait_data(cosa) != '.') return -23;
+ if (get_wait_data(cosa) != '\r')
+ return -21;
+ if (get_wait_data(cosa) != '\n')
+ return -22;
+ if (get_wait_data(cosa) != '.')
+ return -23;
#if 0
printk(KERN_DEBUG "cosa%d: download completed.\n", cosa->num);
#endif
return 0;
}
-
-/*
- * Starting microcode is done via the "g" command of the SRP monitor.
+/* Starting microcode is done via the "g" command of the SRP monitor.
* The chat should be the following: "g" "g=" "<addr><CR>"
* "<CR><CR><LF><CR><LF>".
*/
static int startmicrocode(struct cosa_data *cosa, int address)
{
- if (put_wait_data(cosa, 'g') == -1) return -1;
- if (get_wait_data(cosa) != 'g') return -2;
- if (get_wait_data(cosa) != '=') return -3;
-
- if (puthexnumber(cosa, address) < 0) return -4;
- if (put_wait_data(cosa, '\r') == -1) return -5;
-
- if (get_wait_data(cosa) != '\r') return -6;
- if (get_wait_data(cosa) != '\r') return -7;
- if (get_wait_data(cosa) != '\n') return -8;
- if (get_wait_data(cosa) != '\r') return -9;
- if (get_wait_data(cosa) != '\n') return -10;
+ if (put_wait_data(cosa, 'g') == -1)
+ return -1;
+ if (get_wait_data(cosa) != 'g')
+ return -2;
+ if (get_wait_data(cosa) != '=')
+ return -3;
+
+ if (puthexnumber(cosa, address) < 0)
+ return -4;
+ if (put_wait_data(cosa, '\r') == -1)
+ return -5;
+
+ if (get_wait_data(cosa) != '\r')
+ return -6;
+ if (get_wait_data(cosa) != '\r')
+ return -7;
+ if (get_wait_data(cosa) != '\n')
+ return -8;
+ if (get_wait_data(cosa) != '\r')
+ return -9;
+ if (get_wait_data(cosa) != '\n')
+ return -10;
#if 0
printk(KERN_DEBUG "cosa%d: microcode started\n", cosa->num);
#endif
return 0;
}
-/*
- * Reading memory is done via the "r" command of the SRP monitor.
+/* Reading memory is done via the "r" command of the SRP monitor.
* The chat is the following "r" "r=" "<addr> " " =" "<last_byte> " " "
* Then driver can read the data and the conversation is finished
* by SRP monitor sending "<CR><LF>." (dot at the end).
*/
static int readmem(struct cosa_data *cosa, char __user *microcode, int length, int address)
{
- if (put_wait_data(cosa, 'r') == -1) return -1;
- if ((get_wait_data(cosa)) != 'r') return -2;
- if ((get_wait_data(cosa)) != '=') return -3;
-
- if (puthexnumber(cosa, address) < 0) return -4;
- if (put_wait_data(cosa, ' ') == -1) return -5;
- if (get_wait_data(cosa) != ' ') return -6;
- if (get_wait_data(cosa) != '=') return -7;
-
- if (puthexnumber(cosa, address+length-1) < 0) return -8;
- if (put_wait_data(cosa, ' ') == -1) return -9;
- if (get_wait_data(cosa) != ' ') return -10;
+ if (put_wait_data(cosa, 'r') == -1)
+ return -1;
+ if ((get_wait_data(cosa)) != 'r')
+ return -2;
+ if ((get_wait_data(cosa)) != '=')
+ return -3;
+
+ if (puthexnumber(cosa, address) < 0)
+ return -4;
+ if (put_wait_data(cosa, ' ') == -1)
+ return -5;
+ if (get_wait_data(cosa) != ' ')
+ return -6;
+ if (get_wait_data(cosa) != '=')
+ return -7;
+
+ if (puthexnumber(cosa, address + length - 1) < 0)
+ return -8;
+ if (put_wait_data(cosa, ' ') == -1)
+ return -9;
+ if (get_wait_data(cosa) != ' ')
+ return -10;
while (length--) {
char c;
int i;
- if ((i=get_wait_data(cosa)) == -1) {
+
+ i = get_wait_data(cosa);
+ if (i == -1) {
pr_info("0x%04x bytes remaining\n", length);
return -11;
}
- c=i;
+ c = i;
#if 1
if (put_user(c, microcode))
return -23; /* ??? */
microcode++;
}
- if (get_wait_data(cosa) != '\r') return -21;
- if (get_wait_data(cosa) != '\n') return -22;
- if (get_wait_data(cosa) != '.') return -23;
+ if (get_wait_data(cosa) != '\r')
+ return -21;
+ if (get_wait_data(cosa) != '\n')
+ return -22;
+ if (get_wait_data(cosa) != '.')
+ return -23;
#if 0
printk(KERN_DEBUG "cosa%d: readmem completed.\n", cosa->num);
#endif
return 0;
}
-/*
- * This function resets the device and reads the initial prompt
+/* This function resets the device and reads the initial prompt
* of the device's ROM monitor.
*/
static int cosa_reset_and_read_id(struct cosa_data *cosa, char *idstring)
{
- int i=0, id=0, prev=0, curr=0;
+ int i = 0, id = 0, prev = 0, curr = 0;
/* Reset the card ... */
cosa_putstatus(cosa, 0);
/* Disable all IRQs from the card */
cosa_putstatus(cosa, 0);
- /*
- * Try to read the ID string. The card then prints out the
+ /* Try to read the ID string. The card then prints out the
* identification string ended by the "\n\x2e".
*
* The following loop is indexed through i (instead of id)
* to avoid looping forever when for any reason
* the port returns '\r', '\n' or '\x2e' permanently.
*/
- for (i=0; i<COSA_MAX_ID_STRING-1; i++, prev=curr) {
- if ((curr = get_wait_data(cosa)) == -1) {
+ for (i = 0; i < COSA_MAX_ID_STRING - 1; i++, prev = curr) {
+ curr = get_wait_data(cosa);
+ if (curr == -1)
return -1;
- }
+
curr &= 0xff;
if (curr != '\r' && curr != '\n' && curr != 0x2e)
idstring[id++] = curr;
return id;
}
-\f
/* ---------- Auxiliary routines for COSA/SRP monitor ---------- */
-/*
- * This routine gets the data byte from the card waiting for the SR_RX_RDY
+/* This routine gets the data byte from the card waiting for the SR_RX_RDY
* bit to be set in a loop. It should be used in the exceptional cases
* only (for example when resetting the card or downloading the firmware.
*/
/* read data and return them */
if (cosa_getstatus(cosa) & SR_RX_RDY) {
short r;
+
r = cosa_getdata8(cosa);
#if 0
pr_info("get_wait_data returning after %d retries\n",
- 999-retries);
+ 999 - retries);
#endif
return r;
}
return -1;
}
-/*
- * This routine puts the data byte to the card waiting for the SR_TX_RDY
+/* This routine puts the data byte to the card waiting for the SR_TX_RDY
* bit to be set in a loop. It should be used in the exceptional cases
* only (for example when resetting the card or downloading the firmware).
*/
static int put_wait_data(struct cosa_data *cosa, int data)
{
int retries = 1000;
+
while (--retries) {
/* read data and return them */
if (cosa_getstatus(cosa) & SR_TX_RDY) {
cosa_putdata8(cosa, data);
#if 0
- pr_info("Putdata: %d retries\n", 999-retries);
+ pr_info("Putdata: %d retries\n", 999 - retries);
#endif
return 0;
}
cosa->num, cosa_getstatus(cosa));
return -1;
}
-
-/*
- * The following routine puts the hexadecimal number into the SRP monitor
+
+/* The following routine puts the hexadecimal number into the SRP monitor
* and verifies the proper echo of the sent bytes. Returns 0 on success,
* negative number on failure (-1,-3,-5,-7) means that put_wait_data() failed,
* (-2,-4,-6,-8) means that reading echo failed.
/* Well, I should probably replace this by something faster. */
sprintf(temp, "%04X", number);
- for (i=0; i<4; i++) {
+ for (i = 0; i < 4; i++) {
if (put_wait_data(cosa, temp[i]) == -1) {
pr_notice("cosa%d: puthexnumber failed to write byte %d\n",
cosa->num, i);
- return -1-2*i;
+ return -1 - 2 * i;
}
if (get_wait_data(cosa) != temp[i]) {
pr_notice("cosa%d: puthexhumber failed to read echo of byte %d\n",
cosa->num, i);
- return -2-2*i;
+ return -2 - 2 * i;
}
}
return 0;
}
-\f
/* ---------- Interrupt routines ---------- */
-/*
- * There are three types of interrupt:
+/* There are three types of interrupt:
* At the beginning of transmit - this handled is in tx_interrupt(),
* at the beginning of receive - it is in rx_interrupt() and
* at the end of transmit/receive - it is the eot_interrupt() function.
* COSA status byte. I have moved the rx/tx/eot interrupt handling into
* separate functions to make it more readable. These functions are inline,
* so there should be no overhead of function call.
- *
+ *
* In the COSA bus-master mode, we need to tell the card the address of a
* buffer. Unfortunately, COSA may be too slow for us, so we must busy-wait.
* It's time to use the bottom half :-(
*/
-/*
- * Transmit interrupt routine - called when COSA is willing to obtain
+/* Transmit interrupt routine - called when COSA is willing to obtain
* data from the OS. The most tricky part of the routine is selection
* of channel we (OS) want to send packet for. For SRP we should probably
* use the round-robin approach. The newer COSA firmwares have a simple
set_bit(TXBIT, &cosa->rxtx);
if (!test_bit(IRQBIT, &cosa->rxtx)) {
/* flow control, see the comment above */
- int i=0;
+ int i = 0;
+
if (!cosa->txbitmap) {
pr_warn("%s: No channel wants data in TX IRQ. Expect DMA timeout.\n",
cosa->name);
i++;
if (cosa->txchan >= cosa->nchannels)
cosa->txchan = 0;
- if (!(cosa->txbitmap & (1<<cosa->txchan)))
+ if (!(cosa->txbitmap & (1 << cosa->txchan)))
continue;
- if (~status & (1 << (cosa->txchan+DRIVER_TXMAP_SHIFT)))
+ if (~status &
+ (1 << (cosa->txchan + DRIVER_TXMAP_SHIFT)))
break;
/* in second pass, accept first ready-to-TX channel */
if (i > cosa->nchannels) {
}
cosa->txsize = cosa->chan[cosa->txchan].txsize;
- if (cosa_dma_able(cosa->chan+cosa->txchan,
- cosa->chan[cosa->txchan].txbuf, cosa->txsize)) {
+ if (cosa_dma_able(cosa->chan + cosa->txchan,
+ cosa->chan[cosa->txchan].txbuf,
+ cosa->txsize)) {
cosa->txbuf = cosa->chan[cosa->txchan].txbuf;
} else {
memcpy(cosa->bouncebuf, cosa->chan[cosa->txchan].txbuf,
- cosa->txsize);
+ cosa->txsize);
cosa->txbuf = cosa->bouncebuf;
}
}
if (is_8bit(cosa)) {
if (!test_bit(IRQBIT, &cosa->rxtx)) {
cosa_putstatus(cosa, SR_TX_INT_ENA);
- cosa_putdata8(cosa, ((cosa->txchan << 5) & 0xe0)|
+ cosa_putdata8(cosa, ((cosa->txchan << 5) & 0xe0) |
((cosa->txsize >> 8) & 0x1f));
#ifdef DEBUG_IO
debug_status_out(cosa, SR_TX_INT_ENA);
- debug_data_out(cosa, ((cosa->txchan << 5) & 0xe0)|
- ((cosa->txsize >> 8) & 0x1f));
+ debug_data_out(cosa, ((cosa->txchan << 5) & 0xe0) |
+ ((cosa->txsize >> 8) & 0x1f));
debug_data_in(cosa, cosa_getdata8(cosa));
#else
cosa_getdata8(cosa);
} else {
clear_bit(IRQBIT, &cosa->rxtx);
cosa_putstatus(cosa, 0);
- cosa_putdata8(cosa, cosa->txsize&0xff);
+ cosa_putdata8(cosa, cosa->txsize & 0xff);
#ifdef DEBUG_IO
debug_status_out(cosa, 0);
- debug_data_out(cosa, cosa->txsize&0xff);
+ debug_data_out(cosa, cosa->txsize & 0xff);
#endif
}
} else {
cosa_putstatus(cosa, SR_TX_INT_ENA);
- cosa_putdata16(cosa, ((cosa->txchan<<13) & 0xe000)
+ cosa_putdata16(cosa, ((cosa->txchan << 13) & 0xe000)
| (cosa->txsize & 0x1fff));
#ifdef DEBUG_IO
debug_status_out(cosa, SR_TX_INT_ENA);
- debug_data_out(cosa, ((cosa->txchan<<13) & 0xe000)
- | (cosa->txsize & 0x1fff));
+ debug_data_out(cosa, ((cosa->txchan << 13) & 0xe000) |
+ (cosa->txsize & 0x1fff));
debug_data_in(cosa, cosa_getdata8(cosa));
debug_status_out(cosa, 0);
#else
if (cosa->busmaster) {
unsigned long addr = virt_to_bus(cosa->txbuf);
- int count=0;
+ int count = 0;
+
pr_info("busmaster IRQ\n");
- while (!(cosa_getstatus(cosa)&SR_TX_RDY)) {
+ while (!(cosa_getstatus(cosa) & SR_TX_RDY)) {
count++;
udelay(10);
- if (count > 1000) break;
+ if (count > 1000)
+ break;
}
pr_info("status %x\n", cosa_getstatus(cosa));
pr_info("ready after %d loops\n", count);
- cosa_putdata16(cosa, (addr >> 16)&0xffff);
+ cosa_putdata16(cosa, (addr >> 16) & 0xffff);
count = 0;
- while (!(cosa_getstatus(cosa)&SR_TX_RDY)) {
+ while (!(cosa_getstatus(cosa) & SR_TX_RDY)) {
count++;
- if (count > 1000) break;
+ if (count > 1000)
+ break;
udelay(10);
}
pr_info("ready after %d loops\n", count);
- cosa_putdata16(cosa, addr &0xffff);
+ cosa_putdata16(cosa, addr & 0xffff);
flags1 = claim_dma_lock();
set_dma_mode(cosa->dma, DMA_MODE_CASCADE);
enable_dma(cosa->dma);
enable_dma(cosa->dma);
release_dma_lock(flags1);
}
- cosa_putstatus(cosa, SR_TX_DMA_ENA|SR_USR_INT_ENA);
+ cosa_putstatus(cosa, SR_TX_DMA_ENA | SR_USR_INT_ENA);
#ifdef DEBUG_IO
- debug_status_out(cosa, SR_TX_DMA_ENA|SR_USR_INT_ENA);
+ debug_status_out(cosa, SR_TX_DMA_ENA | SR_USR_INT_ENA);
#endif
spin_unlock_irqrestore(&cosa->lock, flags);
}
if (!test_bit(IRQBIT, &cosa->rxtx)) {
set_bit(IRQBIT, &cosa->rxtx);
put_driver_status_nolock(cosa);
- cosa->rxsize = cosa_getdata8(cosa) <<8;
+ cosa->rxsize = cosa_getdata8(cosa) << 8;
#ifdef DEBUG_IO
debug_data_in(cosa, cosa->rxsize >> 8);
#endif
disable_dma(cosa->dma);
clear_dma_ff(cosa->dma);
set_dma_mode(cosa->dma, DMA_MODE_READ);
- if (cosa_dma_able(cosa->rxchan, cosa->rxbuf, cosa->rxsize & 0x1fff)) {
+ if (cosa_dma_able(cosa->rxchan, cosa->rxbuf, cosa->rxsize & 0x1fff))
set_dma_addr(cosa->dma, virt_to_bus(cosa->rxbuf));
- } else {
+ else
set_dma_addr(cosa->dma, virt_to_bus(cosa->bouncebuf));
- }
- set_dma_count(cosa->dma, (cosa->rxsize&0x1fff));
+
+ set_dma_count(cosa->dma, (cosa->rxsize & 0x1fff));
enable_dma(cosa->dma);
release_dma_lock(flags);
spin_lock_irqsave(&cosa->lock, flags);
- cosa_putstatus(cosa, SR_RX_DMA_ENA|SR_USR_INT_ENA);
+ cosa_putstatus(cosa, SR_RX_DMA_ENA | SR_USR_INT_ENA);
if (!is_8bit(cosa) && (status & SR_TX_RDY))
cosa_putdata8(cosa, DRIVER_RX_READY);
#ifdef DEBUG_IO
- debug_status_out(cosa, SR_RX_DMA_ENA|SR_USR_INT_ENA);
+ debug_status_out(cosa, SR_RX_DMA_ENA | SR_USR_INT_ENA);
if (!is_8bit(cosa) && (status & SR_TX_RDY))
debug_data_cmd(cosa, DRIVER_RX_READY);
#endif
static inline void eot_interrupt(struct cosa_data *cosa, int status)
{
unsigned long flags, flags1;
+
spin_lock_irqsave(&cosa->lock, flags);
flags1 = claim_dma_lock();
disable_dma(cosa->dma);
clear_dma_ff(cosa->dma);
release_dma_lock(flags1);
if (test_bit(TXBIT, &cosa->rxtx)) {
- struct channel_data *chan = cosa->chan+cosa->txchan;
+ struct channel_data *chan = cosa->chan + cosa->txchan;
+
if (chan->tx_done)
if (chan->tx_done(chan, cosa->txsize))
clear_bit(chan->num, &cosa->txbitmap);
#ifdef DEBUG_DATA
{
int i;
+
pr_info("cosa%dc%d: done rx(0x%x)",
cosa->num, cosa->rxchan->num, cosa->rxsize);
- for (i=0; i<cosa->rxsize; i++)
+ for (i = 0; i < cosa->rxsize; i++)
pr_cont(" %02x", cosa->rxbuf[i]&0xff);
pr_cont("\n");
}
} else {
pr_notice("cosa%d: unexpected EOT interrupt\n", cosa->num);
}
- /*
- * Clear the RXBIT, TXBIT and IRQBIT (the latest should be
+ /* Clear the RXBIT, TXBIT and IRQBIT (the latest should be
* cleared anyway). We should do it as soon as possible
* so that we can tell the COSA we are done and to give it a time
* for recovery.
return IRQ_HANDLED;
}
-\f
/* ---------- I/O debugging routines ---------- */
-/*
- * These routines can be used to monitor COSA/SRP I/O and to printk()
+/* These routines can be used to monitor COSA/SRP I/O and to printk()
* the data being transferred on the data and status I/O port in a
* readable way.
*/
static void debug_status_in(struct cosa_data *cosa, int status)
{
char *s;
+
switch (status & SR_CMD_FROM_SRP_MASK) {
case SR_UP_REQUEST:
s = "RX_REQ";
// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * FarSync WAN driver for Linux (2.6.x kernel version)
+/* FarSync WAN driver for Linux (2.6.x kernel version)
*
* Actually sync driver for X.21, V.35 and V.24 on FarSync T-series cards
*
#include "farsync.h"
-/*
- * Module info
+/* Module info
*/
MODULE_AUTHOR("R.J.Dunlop <bob.dunlop@farsite.co.uk>");
MODULE_DESCRIPTION("FarSync T-Series WAN driver. FarSite Communications Ltd.");
/* Default parameters for the link
*/
#define FST_TX_QUEUE_LEN 100 /* At 8Mbps a longer queue length is
- * useful */
+ * useful
+ */
#define FST_TXQ_DEPTH 16 /* This one is for the buffering
* of frames on the way down to the card
* so that we can keep the card busy
* and maximise throughput
*/
#define FST_HIGH_WATER_MARK 12 /* Point at which we flow control
- * network layer */
+ * network layer
+ */
#define FST_LOW_WATER_MARK 8 /* Point at which we remove flow
- * control from network layer */
+ * control from network layer
+ */
#define FST_MAX_MTU 8000 /* Huge but possible */
#define FST_DEF_MTU 1500 /* Common sane value */
-#define FST_TX_TIMEOUT (2*HZ)
+#define FST_TX_TIMEOUT (2 * HZ)
#ifdef ARPHRD_RAWHDLC
#define ARPHRD_MYTYPE ARPHRD_RAWHDLC /* Raw frames */
#define ARPHRD_MYTYPE ARPHRD_HDLC /* Cisco-HDLC (keepalives etc) */
#endif
-/*
- * Modules parameters and associated variables
+/* Modules parameters and associated variables
*/
static int fst_txq_low = FST_LOW_WATER_MARK;
static int fst_txq_high = FST_HIGH_WATER_MARK;
static int fst_max_reads = 7;
-static int fst_excluded_cards = 0;
+static int fst_excluded_cards;
static int fst_excluded_list[FST_MAX_CARDS];
module_param(fst_txq_low, int, 0);
#define FST_MEMSIZE 0x100000 /* Size of card memory (1Mb) */
#define SMC_BASE 0x00002000L /* Base offset of the shared memory window main
- * configuration structure */
+ * configuration structure
+ */
#define BFM_BASE 0x00010000L /* Base offset of the shared memory window DMA
- * buffers */
+ * buffers
+ */
#define LEN_TX_BUFFER 8192 /* Size of packet buffers */
#define LEN_RX_BUFFER 8192
#define INTCSR_9054 0x68 /* Interrupt control/status register */
/* 9054 DMA Registers */
-/*
- * Note that we will be using DMA Channel 0 for copying rx data
+/* Note that we will be using DMA Channel 0 for copying rx data
* and Channel 1 for copying tx data
*/
#define DMAMODE0 0x80
/* Per port (line or channel) information
*/
struct fst_port_info {
- struct net_device *dev; /* Device struct - must be first */
+ struct net_device *dev; /* Device struct - must be first */
struct fst_card_info *card; /* Card we're associated with */
int index; /* Port index on the card */
int hwif; /* Line hardware (lineInterface copy) */
int txpos; /* Next Tx buffer to use */
int txipos; /* Next Tx buffer to check for free */
int start; /* Indication of start/stop to network */
- /*
- * A sixteen entry transmit queue
+ /* A sixteen entry transmit queue
*/
int txqs; /* index to get next buffer to tx */
int txqe; /* index to queue next packet */
#define dev_to_port(D) (dev_to_hdlc(D)->priv)
#define port_to_dev(P) ((P)->dev)
-
-/*
- * Shared memory window access macros
+/* Shared memory window access macros
*
* We have a nice memory based structure above, which could be directly
* mapped on i386 but might not work on other architectures unless we use
*/
#define WIN_OFFSET(X) ((long)&(((struct fst_shared *)SMC_BASE)->X))
-#define FST_RDB(C,E) readb ((C)->mem + WIN_OFFSET(E))
-#define FST_RDW(C,E) readw ((C)->mem + WIN_OFFSET(E))
-#define FST_RDL(C,E) readl ((C)->mem + WIN_OFFSET(E))
+#define FST_RDB(C, E) (readb((C)->mem + WIN_OFFSET(E)))
+#define FST_RDW(C, E) (readw((C)->mem + WIN_OFFSET(E)))
+#define FST_RDL(C, E) (readl((C)->mem + WIN_OFFSET(E)))
-#define FST_WRB(C,E,B) writeb ((B), (C)->mem + WIN_OFFSET(E))
-#define FST_WRW(C,E,W) writew ((W), (C)->mem + WIN_OFFSET(E))
-#define FST_WRL(C,E,L) writel ((L), (C)->mem + WIN_OFFSET(E))
+#define FST_WRB(C, E, B) (writeb((B), (C)->mem + WIN_OFFSET(E)))
+#define FST_WRW(C, E, W) (writew((W), (C)->mem + WIN_OFFSET(E)))
+#define FST_WRL(C, E, L) (writel((L), (C)->mem + WIN_OFFSET(E)))
-/*
- * Debug support
+/* Debug support
*/
#if FST_DEBUG
} while (0)
#endif
-/*
- * PCI ID lookup table
+/* PCI ID lookup table
*/
static const struct pci_device_id fst_pci_dev_id[] = {
- {PCI_VENDOR_ID_FARSITE, PCI_DEVICE_ID_FARSITE_T2P, PCI_ANY_ID,
+ {PCI_VENDOR_ID_FARSITE, PCI_DEVICE_ID_FARSITE_T2P, PCI_ANY_ID,
PCI_ANY_ID, 0, 0, FST_TYPE_T2P},
- {PCI_VENDOR_ID_FARSITE, PCI_DEVICE_ID_FARSITE_T4P, PCI_ANY_ID,
+ {PCI_VENDOR_ID_FARSITE, PCI_DEVICE_ID_FARSITE_T4P, PCI_ANY_ID,
PCI_ANY_ID, 0, 0, FST_TYPE_T4P},
- {PCI_VENDOR_ID_FARSITE, PCI_DEVICE_ID_FARSITE_T1U, PCI_ANY_ID,
+ {PCI_VENDOR_ID_FARSITE, PCI_DEVICE_ID_FARSITE_T1U, PCI_ANY_ID,
PCI_ANY_ID, 0, 0, FST_TYPE_T1U},
- {PCI_VENDOR_ID_FARSITE, PCI_DEVICE_ID_FARSITE_T2U, PCI_ANY_ID,
+ {PCI_VENDOR_ID_FARSITE, PCI_DEVICE_ID_FARSITE_T2U, PCI_ANY_ID,
PCI_ANY_ID, 0, 0, FST_TYPE_T2U},
- {PCI_VENDOR_ID_FARSITE, PCI_DEVICE_ID_FARSITE_T4U, PCI_ANY_ID,
+ {PCI_VENDOR_ID_FARSITE, PCI_DEVICE_ID_FARSITE_T4U, PCI_ANY_ID,
PCI_ANY_ID, 0, 0, FST_TYPE_T4U},
- {PCI_VENDOR_ID_FARSITE, PCI_DEVICE_ID_FARSITE_TE1, PCI_ANY_ID,
+ {PCI_VENDOR_ID_FARSITE, PCI_DEVICE_ID_FARSITE_TE1, PCI_ANY_ID,
PCI_ANY_ID, 0, 0, FST_TYPE_TE1},
- {PCI_VENDOR_ID_FARSITE, PCI_DEVICE_ID_FARSITE_TE1C, PCI_ANY_ID,
+ {PCI_VENDOR_ID_FARSITE, PCI_DEVICE_ID_FARSITE_TE1C, PCI_ANY_ID,
PCI_ANY_ID, 0, 0, FST_TYPE_TE1},
{0,} /* End */
};
MODULE_DEVICE_TABLE(pci, fst_pci_dev_id);
-/*
- * Device Driver Work Queues
+/* Device Driver Work Queues
*
- * So that we don't spend too much time processing events in the
- * Interrupt Service routine, we will declare a work queue per Card
+ * So that we don't spend too much time processing events in the
+ * Interrupt Service routine, we will declare a work queue per Card
* and make the ISR schedule a task in the queue for later execution.
* In the 2.4 Kernel we used to use the immediate queue for BH's
- * Now that they are gone, tasklets seem to be much better than work
+ * Now that they are gone, tasklets seem to be much better than work
* queues.
*/
static u64 fst_work_intq;
static void
-fst_q_work_item(u64 * queue, int card_index)
+fst_q_work_item(u64 *queue, int card_index)
{
unsigned long flags;
u64 mask;
- /*
- * Grab the queue exclusively
+ /* Grab the queue exclusively
*/
spin_lock_irqsave(&fst_work_q_lock, flags);
- /*
- * Making an entry in the queue is simply a matter of setting
+ /* Making an entry in the queue is simply a matter of setting
* a bit for the card indicating that there is work to do in the
* bottom half for the card. Note the limitation of 64 cards.
* That ought to be enough
u64 work_txq;
int i;
- /*
- * Grab the queue exclusively
+ /* Grab the queue exclusively
*/
dbg(DBG_TX, "fst_process_tx_work_q\n");
spin_lock_irqsave(&fst_work_q_lock, flags);
fst_work_txq = 0;
spin_unlock_irqrestore(&fst_work_q_lock, flags);
- /*
- * Call the bottom half for each card with work waiting
+ /* Call the bottom half for each card with work waiting
*/
for (i = 0; i < FST_MAX_CARDS; i++) {
if (work_txq & 0x01) {
- if (fst_card_array[i] != NULL) {
+ if (fst_card_array[i]) {
dbg(DBG_TX, "Calling tx bh for card %d\n", i);
do_bottom_half_tx(fst_card_array[i]);
}
u64 work_intq;
int i;
- /*
- * Grab the queue exclusively
+ /* Grab the queue exclusively
*/
dbg(DBG_INTR, "fst_process_int_work_q\n");
spin_lock_irqsave(&fst_work_q_lock, flags);
fst_work_intq = 0;
spin_unlock_irqrestore(&fst_work_q_lock, flags);
- /*
- * Call the bottom half for each card with work waiting
+ /* Call the bottom half for each card with work waiting
*/
for (i = 0; i < FST_MAX_CARDS; i++) {
if (work_intq & 0x01) {
- if (fst_card_array[i] != NULL) {
+ if (fst_card_array[i]) {
dbg(DBG_INTR,
"Calling rx & tx bh for card %d\n", i);
do_bottom_half_rx(fst_card_array[i]);
dbg(DBG_ASS,
"Error in reading interrupt line register\n");
}
- /*
- * Assert PLX software reset and Am186 hardware reset
+ /* Assert PLX software reset and Am186 hardware reset
* and then deassert the PLX software reset but 186 still in reset
*/
outw(0x440f, card->pci_conf + CNTRL_9054 + 2);
outw(0x040f, card->pci_conf + CNTRL_9054 + 2);
- /*
- * We are delaying here to allow the 9054 to reset itself
+ /* We are delaying here to allow the 9054 to reset itself
*/
usleep_range(10, 20);
outw(0x240f, card->pci_conf + CNTRL_9054 + 2);
- /*
- * We are delaying here to allow the 9054 to reload its eeprom
+ /* We are delaying here to allow the 9054 to reload its eeprom
*/
usleep_range(10, 20);
outw(0x040f, card->pci_conf + CNTRL_9054 + 2);
fst_cpurelease(struct fst_card_info *card)
{
if (card->family == FST_FAMILY_TXU) {
- /*
- * Force posted writes to complete
+ /* Force posted writes to complete
*/
- (void) readb(card->mem);
+ (void)readb(card->mem);
- /*
- * Release LRESET DO = 1
+ /* Release LRESET DO = 1
* Then release Local Hold, DO = 1
*/
outw(0x040e, card->pci_conf + CNTRL_9054 + 2);
outw(0x040f, card->pci_conf + CNTRL_9054 + 2);
} else {
- (void) readb(card->ctlmem);
+ (void)readb(card->ctlmem);
}
}
fst_clear_intr(struct fst_card_info *card)
{
if (card->family == FST_FAMILY_TXU) {
- (void) readb(card->ctlmem);
+ (void)readb(card->ctlmem);
} else {
/* Poke the appropriate PLX chip register (same as enabling interrupts)
*/
static inline void
fst_enable_intr(struct fst_card_info *card)
{
- if (card->family == FST_FAMILY_TXU) {
+ if (card->family == FST_FAMILY_TXU)
outl(0x0f0c0900, card->pci_conf + INTCSR_9054);
- } else {
+ else
outw(0x0543, card->pci_conf + INTCSR_9052);
- }
}
/* Disable card interrupts
static inline void
fst_disable_intr(struct fst_card_info *card)
{
- if (card->family == FST_FAMILY_TXU) {
+ if (card->family == FST_FAMILY_TXU)
outl(0x00000000, card->pci_conf + INTCSR_9054);
- } else {
+ else
outw(0x0000, card->pci_conf + INTCSR_9052);
- }
}
/* Process the result of trying to pass a received frame up the stack
switch (rx_status) {
case NET_RX_SUCCESS:
{
- /*
- * Nothing to do here
+ /* Nothing to do here
*/
break;
}
static inline void
fst_init_dma(struct fst_card_info *card)
{
- /*
- * This is only required for the PLX 9054
+ /* This is only required for the PLX 9054
*/
if (card->family == FST_FAMILY_TXU) {
- pci_set_master(card->device);
+ pci_set_master(card->device);
outl(0x00020441, card->pci_conf + DMAMODE0);
outl(0x00020441, card->pci_conf + DMAMODE1);
outl(0x0, card->pci_conf + DMATHR);
{
struct net_device *dev = port_to_dev(port);
- /*
- * Everything is now set, just tell the card to go
+ /* Everything is now set, just tell the card to go
*/
dbg(DBG_TX, "fst_tx_dma_complete\n");
FST_WRB(card, txDescrRing[port->index][txpos].bits,
netif_trans_update(dev);
}
-/*
- * Mark it for our own raw sockets interface
+/* Mark it for our own raw sockets interface
*/
static __be16 farsync_type_trans(struct sk_buff *skb, struct net_device *dev)
{
dev->stats.rx_dropped++;
}
-/*
- * Receive a frame through the DMA
+/* Receive a frame through the DMA
*/
static inline void
fst_rx_dma(struct fst_card_info *card, dma_addr_t dma, u32 mem, int len)
{
- /*
- * This routine will setup the DMA and start it
+ /* This routine will setup the DMA and start it
*/
dbg(DBG_RX, "In fst_rx_dma %x %x %d\n", (u32)dma, mem, len);
- if (card->dmarx_in_progress) {
+ if (card->dmarx_in_progress)
dbg(DBG_ASS, "In fst_rx_dma while dma in progress\n");
- }
outl(dma, card->pci_conf + DMAPADR0); /* Copy to here */
outl(mem, card->pci_conf + DMALADR0); /* from here */
outl(len, card->pci_conf + DMASIZ0); /* for this length */
outl(0x00000000c, card->pci_conf + DMADPR0); /* In this direction */
- /*
- * We use the dmarx_in_progress flag to flag the channel as busy
+ /* We use the dmarx_in_progress flag to flag the channel as busy
*/
card->dmarx_in_progress = 1;
outb(0x03, card->pci_conf + DMACSR0); /* Start the transfer */
}
-/*
- * Send a frame through the DMA
+/* Send a frame through the DMA
*/
static inline void
fst_tx_dma(struct fst_card_info *card, dma_addr_t dma, u32 mem, int len)
{
- /*
- * This routine will setup the DMA and start it.
+ /* This routine will setup the DMA and start it.
*/
dbg(DBG_TX, "In fst_tx_dma %x %x %d\n", (u32)dma, mem, len);
- if (card->dmatx_in_progress) {
+ if (card->dmatx_in_progress)
dbg(DBG_ASS, "In fst_tx_dma while dma in progress\n");
- }
outl(dma, card->pci_conf + DMAPADR1); /* Copy from here */
outl(mem, card->pci_conf + DMALADR1); /* to here */
outl(len, card->pci_conf + DMASIZ1); /* for this length */
outl(0x000000004, card->pci_conf + DMADPR1); /* In this direction */
- /*
- * We use the dmatx_in_progress to flag the channel as busy
+ /* We use the dmatx_in_progress to flag the channel as busy
*/
card->dmatx_in_progress = 1;
outb(0x03, card->pci_conf + DMACSR1); /* Start the transfer */
mbval = FST_RDW(card, portMailbox[port->index][0]);
}
- if (safety > 0) {
+ if (safety > 0)
dbg(DBG_CMD, "Mailbox clear after %d jiffies\n", safety);
- }
- if (mbval == NAK) {
+
+ if (mbval == NAK)
dbg(DBG_CMD, "issue_cmd: previous command was NAK'd\n");
- }
FST_WRW(card, portMailbox[port->index][0], cmd);
fst_issue_cmd(port, SETV24O);
}
-/*
- * Setup port Rx buffers
+/* Setup port Rx buffers
*/
static void
fst_rx_config(struct fst_port_info *port)
for (i = 0; i < NUM_RX_BUFFER; i++) {
offset = BUF_OFFSET(rxBuffer[pi][i][0]);
- FST_WRW(card, rxDescrRing[pi][i].ladr, (u16) offset);
- FST_WRB(card, rxDescrRing[pi][i].hadr, (u8) (offset >> 16));
+ FST_WRW(card, rxDescrRing[pi][i].ladr, (u16)offset);
+ FST_WRB(card, rxDescrRing[pi][i].hadr, (u8)(offset >> 16));
FST_WRW(card, rxDescrRing[pi][i].bcnt, cnv_bcnt(LEN_RX_BUFFER));
FST_WRW(card, rxDescrRing[pi][i].mcnt, LEN_RX_BUFFER);
FST_WRB(card, rxDescrRing[pi][i].bits, DMA_OWN);
spin_unlock_irqrestore(&card->card_lock, flags);
}
-/*
- * Setup port Tx buffers
+/* Setup port Tx buffers
*/
static void
fst_tx_config(struct fst_port_info *port)
for (i = 0; i < NUM_TX_BUFFER; i++) {
offset = BUF_OFFSET(txBuffer[pi][i][0]);
- FST_WRW(card, txDescrRing[pi][i].ladr, (u16) offset);
- FST_WRB(card, txDescrRing[pi][i].hadr, (u8) (offset >> 16));
+ FST_WRW(card, txDescrRing[pi][i].ladr, (u16)offset);
+ FST_WRB(card, txDescrRing[pi][i].hadr, (u8)(offset >> 16));
FST_WRW(card, txDescrRing[pi][i].bcnt, 0);
FST_WRB(card, txDescrRing[pi][i].bits, 0);
}
ais = FST_RDB(card, suStatus.alarmIndicationSignal);
if (los) {
- /*
- * Lost the link
+ /* Lost the link
*/
if (netif_carrier_ok(port_to_dev(port))) {
dbg(DBG_INTR, "Net carrier off\n");
netif_carrier_off(port_to_dev(port));
}
} else {
- /*
- * Link available
+ /* Link available
*/
if (!netif_carrier_ok(port_to_dev(port))) {
dbg(DBG_INTR, "Net carrier on\n");
signals = FST_RDL(card, v24DebouncedSts[port->index]);
- if (signals & (((port->hwif == X21) || (port->hwif == X21D))
+ if (signals & ((port->hwif == X21 || port->hwif == X21D)
? IPSTS_INDICATE : IPSTS_DCD)) {
if (!netif_carrier_ok(port_to_dev(port))) {
dbg(DBG_INTR, "DCD active\n");
{
struct net_device *dev = port_to_dev(port);
- /*
- * Increment the appropriate error counter
+ /* Increment the appropriate error counter
*/
dev->stats.rx_errors++;
if (dmabits & RX_OFLO) {
int pi;
pi = port->index;
- /*
- * Discard buffer descriptors until we see the start of the
+ /* Discard buffer descriptors until we see the start of the
* next frame. Note that for long frames this could be in
- * a subsequent interrupt.
+ * a subsequent interrupt.
*/
i = 0;
while ((dmabits & (DMA_OWN | RX_STP)) == 0) {
FST_WRB(card, rxDescrRing[pi][rxp].bits, DMA_OWN);
- rxp = (rxp+1) % NUM_RX_BUFFER;
+ rxp = (rxp + 1) % NUM_RX_BUFFER;
if (++i > NUM_RX_BUFFER) {
dbg(DBG_ASS, "intr_rx: Discarding more bufs"
" than we have\n");
/* Discard the terminal buffer */
if (!(dmabits & DMA_OWN)) {
FST_WRB(card, rxDescrRing[pi][rxp].bits, DMA_OWN);
- rxp = (rxp+1) % NUM_RX_BUFFER;
+ rxp = (rxp + 1) % NUM_RX_BUFFER;
}
port->rxpos = rxp;
- return;
-
}
/* Rx complete interrupt
pi, rxp);
return;
}
- if (card->dmarx_in_progress) {
+ if (card->dmarx_in_progress)
return;
- }
/* Get buffer length */
len = FST_RDW(card, rxDescrRing[pi][rxp].mcnt);
/* Discard the CRC */
len -= 2;
if (len == 0) {
- /*
- * This seems to happen on the TE1 interface sometimes
+ /* This seems to happen on the TE1 interface sometimes
* so throw the frame away and log the event.
*/
pr_err("Frame received with 0 length. Card %d Port %d\n",
/* Return descriptor to card */
FST_WRB(card, rxDescrRing[pi][rxp].bits, DMA_OWN);
- rxp = (rxp+1) % NUM_RX_BUFFER;
+ rxp = (rxp + 1) % NUM_RX_BUFFER;
port->rxpos = rxp;
return;
}
}
/* Allocate SKB */
- if ((skb = dev_alloc_skb(len)) == NULL) {
+ skb = dev_alloc_skb(len);
+ if (!skb) {
dbg(DBG_RX, "intr_rx: can't allocate buffer\n");
dev->stats.rx_dropped++;
/* Return descriptor to card */
FST_WRB(card, rxDescrRing[pi][rxp].bits, DMA_OWN);
- rxp = (rxp+1) % NUM_RX_BUFFER;
+ rxp = (rxp + 1) % NUM_RX_BUFFER;
port->rxpos = rxp;
return;
}
- /*
- * We know the length we need to receive, len.
+ /* We know the length we need to receive, len.
* It's not worth using the DMA for reads of less than
* FST_MIN_DMA_LEN
*/
- if ((len < FST_MIN_DMA_LEN) || (card->family == FST_FAMILY_TXP)) {
+ if (len < FST_MIN_DMA_LEN || card->family == FST_FAMILY_TXP) {
memcpy_fromio(skb_put(skb, len),
card->mem + BUF_OFFSET(rxBuffer[pi][rxp][0]),
len);
dbg(DBG_ASS, "About to increment rxpos by more than 1\n");
dbg(DBG_ASS, "rxp = %d rxpos = %d\n", rxp, port->rxpos);
}
- rxp = (rxp+1) % NUM_RX_BUFFER;
+ rxp = (rxp + 1) % NUM_RX_BUFFER;
port->rxpos = rxp;
}
-/*
- * The bottom halfs to the ISR
+/* The bottom half to the ISR
*
*/
unsigned long flags;
struct net_device *dev;
- /*
- * Find a free buffer for the transmit
+ /* Find a free buffer for the transmit
* Step through each port on this card
*/
while (!(FST_RDB(card, txDescrRing[pi][port->txpos].bits) &
DMA_OWN) &&
!(card->dmatx_in_progress)) {
- /*
- * There doesn't seem to be a txdone event per-se
+ /* There doesn't seem to be a txdone event per-se
* We seem to have to deduce it, by checking the DMA_OWN
* bit on the next buffer we think we can use
*/
spin_lock_irqsave(&card->card_lock, flags);
- if ((txq_length = port->txqe - port->txqs) < 0) {
- /*
- * This is the case where one has wrapped and the
+ txq_length = port->txqe - port->txqs;
+ if (txq_length < 0) {
+ /* This is the case where one has wrapped and the
* maths gives us a negative number
*/
txq_length = txq_length + FST_TXQ_DEPTH;
}
spin_unlock_irqrestore(&card->card_lock, flags);
if (txq_length > 0) {
- /*
- * There is something to send
+ /* There is something to send
*/
spin_lock_irqsave(&card->card_lock, flags);
skb = port->txq[port->txqs];
port->txqs++;
- if (port->txqs == FST_TXQ_DEPTH) {
+ if (port->txqs == FST_TXQ_DEPTH)
port->txqs = 0;
- }
+
spin_unlock_irqrestore(&card->card_lock, flags);
- /*
- * copy the data and set the required indicators on the
+ /* copy the data and set the required indicators on the
* card.
*/
FST_WRW(card, txDescrRing[pi][port->txpos].bcnt,
cnv_bcnt(skb->len));
- if ((skb->len < FST_MIN_DMA_LEN) ||
- (card->family == FST_FAMILY_TXP)) {
+ if (skb->len < FST_MIN_DMA_LEN ||
+ card->family == FST_FAMILY_TXP) {
/* Enqueue the packet with normal io */
memcpy_toio(card->mem +
BUF_OFFSET(txBuffer[pi]
}
if (++port->txpos >= NUM_TX_BUFFER)
port->txpos = 0;
- /*
- * If we have flow control on, can we now release it?
+ /* If we have flow control on, can we now release it?
*/
if (port->start) {
if (txq_length < fst_txq_low) {
}
dev_kfree_skb(skb);
} else {
- /*
- * Nothing to send so break out of the while loop
+ /* Nothing to send so break out of the while loop
*/
break;
}
while (!(FST_RDB(card, rxDescrRing[pi][port->rxpos].bits)
& DMA_OWN) && !(card->dmarx_in_progress)) {
if (rx_count > fst_max_reads) {
- /*
- * Don't spend forever in receive processing
+ /* Don't spend forever in receive processing
* Schedule another event
*/
fst_q_work_item(&fst_work_intq, card->card_no);
}
}
-/*
- * The interrupt service routine
+/* The interrupt service routine
* Dev_id is our fst_card_info pointer
*/
static irqreturn_t
unsigned int do_card_interrupt;
unsigned int int_retry_count;
- /*
- * Check to see if the interrupt was for this card
+ /* Check to see if the interrupt was for this card
* return if not
* Note that the call to clear the interrupt is important
*/
pr_err("Interrupt received for card %d in a non running state (%d)\n",
card->card_no, card->state);
- /*
- * It is possible to really be running, i.e. we have re-loaded
+ /* It is possible to really be running, i.e. we have re-loaded
* a running card
- * Clear and reprime the interrupt source
+ * Clear and reprime the interrupt source
*/
fst_clear_intr(card);
return IRQ_HANDLED;
/* Clear and reprime the interrupt source */
fst_clear_intr(card);
- /*
- * Is the interrupt for this card (handshake == 1)
+ /* Is the interrupt for this card (handshake == 1)
*/
do_card_interrupt = 0;
if (FST_RDB(card, interruptHandshake) == 1) {
FST_WRB(card, interruptHandshake, 0xEE);
}
if (card->family == FST_FAMILY_TXU) {
- /*
- * Is it a DMA Interrupt
+ /* Is it a DMA Interrupt
*/
dma_intcsr = inl(card->pci_conf + INTCSR_9054);
if (dma_intcsr & 0x00200000) {
- /*
- * DMA Channel 0 (Rx transfer complete)
+ /* DMA Channel 0 (Rx transfer complete)
*/
dbg(DBG_RX, "DMA Rx xfer complete\n");
outb(0x8, card->pci_conf + DMACSR0);
do_card_interrupt += FST_RX_DMA_INT;
}
if (dma_intcsr & 0x00400000) {
- /*
- * DMA Channel 1 (Tx transfer complete)
+ /* DMA Channel 1 (Tx transfer complete)
*/
dbg(DBG_TX, "DMA Tx xfer complete\n");
outb(0x8, card->pci_conf + DMACSR1);
}
}
- /*
- * Have we been missing Interrupts
+ /* Have we been missing Interrupts
*/
int_retry_count = FST_RDL(card, interruptRetryCount);
if (int_retry_count) {
FST_WRL(card, interruptRetryCount, 0);
}
- if (!do_card_interrupt) {
+ if (!do_card_interrupt)
return IRQ_HANDLED;
- }
/* Scehdule the bottom half of the ISR */
fst_q_work_item(&fst_work_intq, card->card_no);
rdidx = 0;
}
FST_WRB(card, interruptEvent.rdindex, rdidx);
- return IRQ_HANDLED;
+ return IRQ_HANDLED;
}
/* Check that the shared memory configuration is one that we can handle
return;
}
/* Firmware status flag, 0x00 = initialising, 0x01 = OK, 0xFF = fail */
- if ((i = FST_RDB(card, taskStatus)) == 0x01) {
+ i = FST_RDB(card, taskStatus);
+ if (i == 0x01) {
card->state = FST_RUNNING;
} else if (i == 0xFF) {
pr_err("Firmware initialisation failed. Card halted\n");
int err;
unsigned char my_framing;
- /* Set things according to the user set valid flags
- * Several of the old options have been invalidated/replaced by the
+ /* Set things according to the user set valid flags
+ * Several of the old options have been invalidated/replaced by the
* generic hdlc package.
*/
err = 0;
#endif
}
#if FST_DEBUG
- if (info->valid & FSTVAL_DEBUG) {
+ if (info->valid & FSTVAL_DEBUG)
fst_debug_mask = info->debug;
- }
#endif
return err;
{
int i;
- memset(info, 0, sizeof (struct fstioc_info));
+ memset(info, 0, sizeof(struct fstioc_info));
i = port->index;
info->kernelVersion = LINUX_VERSION_CODE;
info->cardMode = FST_RDW(card, cardMode);
info->smcFirmwareVersion = FST_RDL(card, smcFirmwareVersion);
- /*
- * The T2U can report cable presence for both A or B
- * in bits 0 and 1 of cableStatus. See which port we are and
+ /* The T2U can report cable presence for both A or B
+ * in bits 0 and 1 of cableStatus. See which port we are and
* do the mapping.
*/
if (card->family == FST_FAMILY_TXU) {
if (port->index == 0) {
- /*
- * Port A
+ /* Port A
*/
info->cableStatus = info->cableStatus & 1;
} else {
- /*
- * Port B
+ /* Port B
*/
info->cableStatus = info->cableStatus >> 1;
info->cableStatus = info->cableStatus & 1;
}
}
- /*
- * Some additional bits if we are TE1
+ /* Some additional bits if we are TE1
*/
if (card->type == FST_TYPE_TE1) {
info->lineSpeed = FST_RDL(card, suConfig.dataRate);
sync_serial_settings sync;
int i;
- if (ifr->ifr_settings.size != sizeof (sync)) {
+ if (ifr->ifr_settings.size != sizeof(sync))
return -ENOMEM;
- }
if (copy_from_user
- (&sync, ifr->ifr_settings.ifs_ifsu.sync, sizeof (sync))) {
+ (&sync, ifr->ifr_settings.ifs_ifsu.sync, sizeof(sync)))
return -EFAULT;
- }
if (sync.loopback)
return -EINVAL;
ifr->ifr_settings.type = IF_IFACE_X21;
break;
}
- if (ifr->ifr_settings.size == 0) {
+ if (ifr->ifr_settings.size == 0)
return 0; /* only type requested */
- }
- if (ifr->ifr_settings.size < sizeof (sync)) {
+
+ if (ifr->ifr_settings.size < sizeof(sync))
return -ENOMEM;
- }
i = port->index;
memset(&sync, 0, sizeof(sync));
INTCLK ? CLOCK_INT : CLOCK_EXT;
sync.loopback = 0;
- if (copy_to_user(ifr->ifr_settings.ifs_ifsu.sync, &sync, sizeof (sync))) {
+ if (copy_to_user(ifr->ifr_settings.ifs_ifsu.sync, &sync, sizeof(sync)))
return -EFAULT;
- }
- ifr->ifr_settings.size = sizeof (sync);
+ ifr->ifr_settings.size = sizeof(sync);
return 0;
}
/* First copy in the header with the length and offset of data
* to write
*/
- if (ifr->ifr_data == NULL) {
+ if (!ifr->ifr_data)
return -EINVAL;
- }
+
if (copy_from_user(&wrthdr, ifr->ifr_data,
- sizeof (struct fstioc_write))) {
+ sizeof(struct fstioc_write)))
return -EFAULT;
- }
/* Sanity check the parameters. We don't support partial writes
* when going over the top
*/
if (wrthdr.size > FST_MEMSIZE || wrthdr.offset > FST_MEMSIZE ||
- wrthdr.size + wrthdr.offset > FST_MEMSIZE) {
+ wrthdr.size + wrthdr.offset > FST_MEMSIZE)
return -ENXIO;
- }
/* Now copy the data to the card. */
/* Writes to the memory of a card in the reset state constitute
* a download
*/
- if (card->state == FST_RESET) {
+ if (card->state == FST_RESET)
card->state = FST_DOWNLOAD;
- }
+
return 0;
case FSTGETCONF:
}
}
- if (ifr->ifr_data == NULL) {
+ if (!ifr->ifr_data)
return -EINVAL;
- }
gather_conf_info(card, port, &info);
- if (copy_to_user(ifr->ifr_data, &info, sizeof (info))) {
+ if (copy_to_user(ifr->ifr_data, &info, sizeof(info)))
return -EFAULT;
- }
+
return 0;
case FSTSETCONF:
-
- /*
- * Most of the settings have been moved to the generic ioctls
+ /* Most of the settings have been moved to the generic ioctls
* this just covers debug and board ident now
*/
card->card_no, card->state);
return -EIO;
}
- if (copy_from_user(&info, ifr->ifr_data, sizeof (info))) {
+ if (copy_from_user(&info, ifr->ifr_data, sizeof(info)))
return -EFAULT;
- }
return set_conf_from_info(card, port, &info);
port->run = 1;
signals = FST_RDL(port->card, v24DebouncedSts[port->index]);
- if (signals & (((port->hwif == X21) || (port->hwif == X21D))
+ if (signals & ((port->hwif == X21 || port->hwif == X21D)
? IPSTS_INDICATE : IPSTS_DCD))
netif_carrier_on(port_to_dev(port));
else
port->txqe = 0;
port->txqs = 0;
}
-
}
static void
port = dev_to_port(dev);
if (!try_module_get(THIS_MODULE))
- return -EBUSY;
+ return -EBUSY;
if (port->mode != FST_RAW) {
err = hdlc_open(dev);
netif_stop_queue(dev);
fst_closeport(dev_to_port(dev));
- if (port->mode != FST_RAW) {
+ if (port->mode != FST_RAW)
hdlc_close(dev);
- }
+
module_put(THIS_MODULE);
return 0;
}
static int
fst_attach(struct net_device *dev, unsigned short encoding, unsigned short parity)
{
- /*
- * Setting currently fixed in FarSync card so we check and forget
+ /* Setting currently fixed in FarSync card so we check and forget
*/
if (encoding != ENCODING_NRZ || parity != PARITY_CRC16_PR1_CCITT)
return -EINVAL;
return NETDEV_TX_OK;
}
- /*
- * We are always going to queue the packet
+ /* We are always going to queue the packet
* so that the bottom half is the only place we tx from
* Check there is room in the port txq
*/
spin_lock_irqsave(&card->card_lock, flags);
- if ((txq_length = port->txqe - port->txqs) < 0) {
- /*
- * This is the case where the next free has wrapped but the
+ txq_length = port->txqe - port->txqs;
+ if (txq_length < 0) {
+ /* This is the case where the next free has wrapped but the
* last used hasn't
*/
txq_length = txq_length + FST_TXQ_DEPTH;
}
spin_unlock_irqrestore(&card->card_lock, flags);
if (txq_length > fst_txq_high) {
- /*
- * We have got enough buffers in the pipeline. Ask the network
+ /* We have got enough buffers in the pipeline. Ask the network
* layer to stop sending frames down
*/
netif_stop_queue(dev);
}
if (txq_length == FST_TXQ_DEPTH - 1) {
- /*
- * This shouldn't have happened but such is life
+ /* This shouldn't have happened but such is life
*/
dev_kfree_skb(skb);
dev->stats.tx_errors++;
return NETDEV_TX_OK;
}
- /*
- * queue the buffer
+ /* queue the buffer
*/
spin_lock_irqsave(&card->card_lock, flags);
port->txq[port->txqe] = skb;
return NETDEV_TX_OK;
}
-/*
- * Card setup having checked hardware resources.
+/* Card setup having checked hardware resources.
* Should be pretty bizarre if we get an error here (kernel memory
* exhaustion is one possibility). If we do see a problem we report it
* via a printk and leave the corresponding interface and all that follow
err = register_hdlc_device(card->ports[i].dev);
if (err < 0) {
pr_err("Cannot register HDLC device for port %d (errno %d)\n",
- i, -err);
+ i, -err);
while (i--)
unregister_hdlc_device(card->ports[i].dev);
return err;
.ndo_tx_timeout = fst_tx_timeout,
};
-/*
- * Initialise card when detected.
+/* Initialise card when detected.
* Returns 0 to indicate success, or errno otherwise.
*/
static int
fst_add_one(struct pci_dev *pdev, const struct pci_device_id *ent)
{
- static int no_of_cards_added = 0;
+ static int no_of_cards_added;
struct fst_card_info *card;
int err = 0;
int i;
#if FST_DEBUG
dbg(DBG_ASS, "The value of debug mask is %x\n", fst_debug_mask);
#endif
- /*
- * We are going to be clever and allow certain cards not to be
+ /* We are going to be clever and allow certain cards not to be
* configured. An exclude list can be provided in /etc/modules.conf
*/
if (fst_excluded_cards != 0) {
- /*
- * There are cards to exclude
+ /* There are cards to exclude
*
*/
for (i = 0; i < fst_excluded_cards; i++) {
- if ((pdev->devfn) >> 3 == fst_excluded_list[i]) {
+ if (pdev->devfn >> 3 == fst_excluded_list[i]) {
pr_info("FarSync PCI device %d not assigned\n",
(pdev->devfn) >> 3);
return -EBUSY;
/* Allocate driver private data */
card = kzalloc(sizeof(struct fst_card_info), GFP_KERNEL);
- if (card == NULL)
+ if (!card)
return -ENOMEM;
/* Try to enable the device */
- if ((err = pci_enable_device(pdev)) != 0) {
+ err = pci_enable_device(pdev);
+ if (err) {
pr_err("Failed to enable card. Err %d\n", -err);
goto enable_fail;
}
- if ((err = pci_request_regions(pdev, "FarSync")) !=0) {
+ err = pci_request_regions(pdev, "FarSync");
+ if (err) {
pr_err("Failed to allocate regions. Err %d\n", -err);
goto regions_fail;
}
card->pci_conf = pci_resource_start(pdev, 1);
card->phys_mem = pci_resource_start(pdev, 2);
card->phys_ctlmem = pci_resource_start(pdev, 3);
- if ((card->mem = ioremap(card->phys_mem, FST_MEMSIZE)) == NULL) {
+ card->mem = ioremap(card->phys_mem, FST_MEMSIZE);
+ if (!card->mem) {
pr_err("Physical memory remap failed\n");
err = -ENODEV;
goto ioremap_physmem_fail;
}
- if ((card->ctlmem = ioremap(card->phys_ctlmem, 0x10)) == NULL) {
+ card->ctlmem = ioremap(card->phys_ctlmem, 0x10);
+ if (!card->ctlmem) {
pr_err("Control memory remap failed\n");
err = -ENODEV;
goto ioremap_ctlmem_fail;
card->family = ((ent->driver_data == FST_TYPE_T2P) ||
(ent->driver_data == FST_TYPE_T4P))
? FST_FAMILY_TXP : FST_FAMILY_TXU;
- if ((ent->driver_data == FST_TYPE_T1U) ||
- (ent->driver_data == FST_TYPE_TE1))
+ if (ent->driver_data == FST_TYPE_T1U ||
+ ent->driver_data == FST_TYPE_TE1)
card->nports = 1;
else
card->nports = ((ent->driver_data == FST_TYPE_T2P) ||
(ent->driver_data == FST_TYPE_T2U)) ? 2 : 4;
card->state = FST_UNINIT;
- spin_lock_init ( &card->card_lock );
+ spin_lock_init(&card->card_lock);
- for ( i = 0 ; i < card->nports ; i++ ) {
+ for (i = 0; i < card->nports; i++) {
struct net_device *dev = alloc_hdlcdev(&card->ports[i]);
hdlc_device *hdlc;
+
if (!dev) {
while (i--)
free_netdev(card->ports[i].dev);
goto hdlcdev_fail;
}
card->ports[i].dev = dev;
- card->ports[i].card = card;
- card->ports[i].index = i;
- card->ports[i].run = 0;
+ card->ports[i].card = card;
+ card->ports[i].index = i;
+ card->ports[i].run = 0;
hdlc = dev_to_hdlc(dev);
- /* Fill in the net device info */
+ /* Fill in the net device info */
/* Since this is a PCI setup this is purely
* informational. Give them the buffer addresses
* and basic card I/O.
*/
- dev->mem_start = card->phys_mem
- + BUF_OFFSET ( txBuffer[i][0][0]);
- dev->mem_end = card->phys_mem
- + BUF_OFFSET ( txBuffer[i][NUM_TX_BUFFER - 1][LEN_RX_BUFFER - 1]);
- dev->base_addr = card->pci_conf;
- dev->irq = card->irq;
+ dev->mem_start = card->phys_mem
+ + BUF_OFFSET(txBuffer[i][0][0]);
+ dev->mem_end = card->phys_mem
+ + BUF_OFFSET(txBuffer[i][NUM_TX_BUFFER - 1][LEN_RX_BUFFER - 1]);
+ dev->base_addr = card->pci_conf;
+ dev->irq = card->irq;
dev->netdev_ops = &fst_ops;
dev->tx_queue_len = FST_TX_QUEUE_LEN;
dev->watchdog_timeo = FST_TX_TIMEOUT;
- hdlc->attach = fst_attach;
- hdlc->xmit = fst_start_xmit;
+ hdlc->attach = fst_attach;
+ hdlc->xmit = fst_start_xmit;
}
card->device = pdev;
if (err)
goto init_card_fail;
if (card->family == FST_FAMILY_TXU) {
- /*
- * Allocate a dma buffer for transmit and receives
+ /* Allocate a dma buffer for transmit and receives
*/
card->rx_dma_handle_host =
dma_alloc_coherent(&card->device->dev, FST_MAX_MTU,
&card->rx_dma_handle_card, GFP_KERNEL);
- if (card->rx_dma_handle_host == NULL) {
+ if (!card->rx_dma_handle_host) {
pr_err("Could not allocate rx dma buffer\n");
err = -ENOMEM;
goto rx_dma_fail;
card->tx_dma_handle_host =
dma_alloc_coherent(&card->device->dev, FST_MAX_MTU,
&card->tx_dma_handle_card, GFP_KERNEL);
- if (card->tx_dma_handle_host == NULL) {
+ if (!card->tx_dma_handle_host) {
pr_err("Could not allocate tx dma buffer\n");
err = -ENOMEM;
goto tx_dma_fail;
return err;
}
-/*
- * Cleanup and close down a card
+/* Cleanup and close down a card
*/
static void
fst_remove_one(struct pci_dev *pdev)
for (i = 0; i < card->nports; i++) {
struct net_device *dev = port_to_dev(&card->ports[i]);
+
unregister_hdlc_device(dev);
}
iounmap(card->mem);
pci_release_regions(pdev);
if (card->family == FST_FAMILY_TXU) {
- /*
- * Free dma buffers
+ /* Free dma buffers
*/
dma_free_coherent(&card->device->dev, FST_MAX_MTU,
card->rx_dma_handle_host,
ut_info->uf_info.irq = irq_of_parse_and_map(np, 0);
uhdlc_priv = kzalloc(sizeof(*uhdlc_priv), GFP_KERNEL);
- if (!uhdlc_priv) {
+ if (!uhdlc_priv)
return -ENOMEM;
- }
dev_set_drvdata(&pdev->dev, uhdlc_priv);
uhdlc_priv->dev = &pdev->dev;
#define SCA_INTR_DMAC_RX(node) (node ? 0x20 : 0x02)
#define SCA_INTR_DMAC_TX(node) (node ? 0x40 : 0x04)
-
static inline struct net_device *port_to_dev(port_t *port)
{
return port->dev;
u8 isr0 = sca_in(ISR0, card);
u8 isr1 = sca_in(ISR1, card);
- if (isr1 & 0x03) result |= SCA_INTR_DMAC_RX(0);
- if (isr1 & 0x0C) result |= SCA_INTR_DMAC_TX(0);
- if (isr1 & 0x30) result |= SCA_INTR_DMAC_RX(1);
- if (isr1 & 0xC0) result |= SCA_INTR_DMAC_TX(1);
- if (isr0 & 0x0F) result |= SCA_INTR_MSCI(0);
- if (isr0 & 0xF0) result |= SCA_INTR_MSCI(1);
+ if (isr1 & 0x03)
+ result |= SCA_INTR_DMAC_RX(0);
+ if (isr1 & 0x0C)
+ result |= SCA_INTR_DMAC_TX(0);
+ if (isr1 & 0x30)
+ result |= SCA_INTR_DMAC_RX(1);
+ if (isr1 & 0xC0)
+ result |= SCA_INTR_DMAC_TX(1);
+ if (isr0 & 0x0F)
+ result |= SCA_INTR_MSCI(0);
+ if (isr0 & 0xF0)
+ result |= SCA_INTR_MSCI(1);
if (!(result & SCA_INTR_DMAC_TX(0)))
if (sca_in(DSR_TX(0), card) & DSR_EOM)
return result;
}
-static inline port_t* dev_to_port(struct net_device *dev)
+static inline port_t *dev_to_port(struct net_device *dev)
{
return dev_to_hdlc(dev)->priv;
}
: port_to_card(port)->rx_ring_buffers);
}
-
static inline u16 desc_abs_number(port_t *port, u16 desc, int transmit)
{
u16 rx_buffs = port_to_card(port)->rx_ring_buffers;
transmit * rx_buffs + desc;
}
-
static inline u16 desc_offset(port_t *port, u16 desc, int transmit)
{
/* Descriptor offset always fits in 16 bits */
return desc_abs_number(port, desc, transmit) * sizeof(pkt_desc);
}
-
static inline pkt_desc __iomem *desc_address(port_t *port, u16 desc,
int transmit)
{
#endif
}
-
static inline u32 buffer_offset(port_t *port, u16 desc, int transmit)
{
return port_to_card(port)->buff_offset +
desc_abs_number(port, desc, transmit) * (u32)HDLC_MAX_MRU;
}
-
static inline void sca_set_carrier(port_t *port)
{
if (!(sca_in(get_msci(port) + ST3, port_to_card(port)) & ST3_DCD)) {
}
}
-
static void sca_init_port(port_t *port)
{
card_t *card = port_to_card(port);
sca_set_carrier(port);
}
-
#ifdef NEED_SCA_MSCI_INTR
/* MSCI interrupt service */
static inline void sca_msci_intr(port_t *port)
{
u16 msci = get_msci(port);
- card_t* card = port_to_card(port);
+ card_t *card = port_to_card(port);
u8 stat = sca_in(msci + ST1, card); /* read MSCI ST1 status */
/* Reset MSCI TX underrun and CDCD status bit */
}
#endif
-
static inline void sca_rx(card_t *card, port_t *port, pkt_desc __iomem *desc,
u16 rxin)
{
memcpy_fromio(skb->data, winbase(card) + buff, maxlen);
openwin(card, page + 1);
memcpy_fromio(skb->data + maxlen, winbase(card), len - maxlen);
- } else
+ } else {
memcpy_fromio(skb->data, winbase(card) + buff, len);
+ }
#ifndef PAGE0_ALWAYS_MAPPED
openwin(card, 0); /* select pkt_desc table page back */
netif_rx(skb);
}
-
/* Receive DMA interrupt service */
static inline void sca_rx_intr(port_t *port)
{
pkt_desc __iomem *desc;
u32 cda = sca_inw(dmac + CDAL, card);
- if ((cda >= desc_off) && (cda < desc_off + sizeof(pkt_desc)))
+ if (cda >= desc_off && (cda < desc_off + sizeof(pkt_desc)))
break; /* No frame received */
desc = desc_address(port, port->rxin, 0);
dev->stats.rx_crc_errors++;
if (stat & ST_RX_EOM)
port->rxpart = 0; /* received last fragment */
- } else
+ } else {
sca_rx(card, port, desc, port->rxin);
+ }
/* Set new error descriptor address */
sca_outw(desc_off, dmac + EDAL, card);
sca_out(DSR_DE, DSR_RX(phy_node(port)), card);
}
-
/* Transmit DMA interrupt service */
static inline void sca_tx_intr(port_t *port)
{
struct net_device *dev = port_to_dev(port);
u16 dmac = get_dmac_tx(port);
- card_t* card = port_to_card(port);
+ card_t *card = port_to_card(port);
u8 stat;
spin_lock(&port->lock);
u32 desc_off = desc_offset(port, port->txlast, 1);
u32 cda = sca_inw(dmac + CDAL, card);
- if ((cda >= desc_off) && (cda < desc_off + sizeof(pkt_desc)))
+
+ if (cda >= desc_off && (cda < desc_off + sizeof(pkt_desc)))
break; /* Transmitter is/will_be sending this frame */
desc = desc_address(port, port->txlast, 1);
spin_unlock(&port->lock);
}
-
-static irqreturn_t sca_intr(int irq, void* dev_id)
+static irqreturn_t sca_intr(int irq, void *dev_id)
{
card_t *card = dev_id;
int i;
int handled = 0;
u8 page = sca_get_page(card);
- while((stat = sca_intr_status(card)) != 0) {
+ while ((stat = sca_intr_status(card)) != 0) {
handled = 1;
for (i = 0; i < 2; i++) {
port_t *port = get_port(card, i);
+
if (port) {
if (stat & SCA_INTR_MSCI(i))
sca_msci_intr(port);
return IRQ_RETVAL(handled);
}
-
static void sca_set_port(port_t *port)
{
- card_t* card = port_to_card(port);
+ card_t *card = port_to_card(port);
u16 msci = get_msci(port);
u8 md2 = sca_in(msci + MD2, card);
unsigned int tmc, br = 10, brv = 1024;
-
if (port->settings.clock_rate > 0) {
/* Try lower br for better accuracy*/
do {
/* Baud Rate = CLOCK_BASE / TMC / 2^BR */
tmc = CLOCK_BASE / brv / port->settings.clock_rate;
- }while (br > 1 && tmc <= 128);
+ } while (br > 1 && tmc <= 128);
if (tmc < 1) {
tmc = 1;
br = 0; /* For baud=CLOCK_BASE we use tmc=1 br=0 */
brv = 1;
- } else if (tmc > 255)
+ } else if (tmc > 255) {
tmc = 256; /* tmc=0 means 256 - low baud rates */
+ }
port->settings.clock_rate = CLOCK_BASE / brv / tmc;
} else {
md2 &= ~MD2_LOOPBACK;
sca_out(md2, msci + MD2, card);
-
}
-
static void sca_open(struct net_device *dev)
{
port_t *port = dev_to_port(dev);
- card_t* card = port_to_card(port);
+ card_t *card = port_to_card(port);
u16 msci = get_msci(port);
u8 md0, md2;
- switch(port->encoding) {
- case ENCODING_NRZ: md2 = MD2_NRZ; break;
- case ENCODING_NRZI: md2 = MD2_NRZI; break;
- case ENCODING_FM_MARK: md2 = MD2_FM_MARK; break;
- case ENCODING_FM_SPACE: md2 = MD2_FM_SPACE; break;
- default: md2 = MD2_MANCHESTER;
+ switch (port->encoding) {
+ case ENCODING_NRZ:
+ md2 = MD2_NRZ;
+ break;
+ case ENCODING_NRZI:
+ md2 = MD2_NRZI;
+ break;
+ case ENCODING_FM_MARK:
+ md2 = MD2_FM_MARK;
+ break;
+ case ENCODING_FM_SPACE:
+ md2 = MD2_FM_SPACE;
+ break;
+ default:
+ md2 = MD2_MANCHESTER;
}
if (port->settings.loopback)
md2 |= MD2_LOOPBACK;
- switch(port->parity) {
- case PARITY_CRC16_PR0: md0 = MD0_HDLC | MD0_CRC_16_0; break;
- case PARITY_CRC16_PR1: md0 = MD0_HDLC | MD0_CRC_16; break;
- case PARITY_CRC16_PR0_CCITT: md0 = MD0_HDLC | MD0_CRC_ITU_0; break;
- case PARITY_CRC16_PR1_CCITT: md0 = MD0_HDLC | MD0_CRC_ITU; break;
- default: md0 = MD0_HDLC | MD0_CRC_NONE;
+ switch (port->parity) {
+ case PARITY_CRC16_PR0:
+ md0 = MD0_HDLC | MD0_CRC_16_0;
+ break;
+ case PARITY_CRC16_PR1:
+ md0 = MD0_HDLC | MD0_CRC_16;
+ break;
+ case PARITY_CRC16_PR0_CCITT:
+ md0 = MD0_HDLC | MD0_CRC_ITU_0;
+ break;
+ case PARITY_CRC16_PR1_CCITT:
+ md0 = MD0_HDLC | MD0_CRC_ITU;
+ break;
+ default:
+ md0 = MD0_HDLC | MD0_CRC_NONE;
}
sca_out(CMD_RESET, msci + CMD, card);
sca_out(0x14, msci + TRC1, card); /* +1=TXRDY/DMA deactiv condition */
/* We're using the following interrupts:
- - TXINT (DMAC completed all transmisions, underrun or DCD change)
- - all DMA interrupts
-*/
+ * - TXINT (DMAC completed all transmisions, underrun or DCD change)
+ * - all DMA interrupts
+ */
sca_set_carrier(port);
/* MSCI TX INT and RX INT A IRQ enable */
netif_start_queue(dev);
}
-
static void sca_close(struct net_device *dev)
{
port_t *port = dev_to_port(dev);
- card_t* card = port_to_card(port);
+ card_t *card = port_to_card(port);
/* reset channel */
sca_out(CMD_RESET, get_msci(port) + CMD, port_to_card(port));
netif_stop_queue(dev);
}
-
static int sca_attach(struct net_device *dev, unsigned short encoding,
unsigned short parity)
{
return 0;
}
-
#ifdef DEBUG_RINGS
static void sca_dump_rings(struct net_device *dev)
{
}
#endif /* DEBUG_RINGS */
-
static netdev_tx_t sca_xmit(struct sk_buff *skb, struct net_device *dev)
{
port_t *port = dev_to_port(dev);
memcpy_toio(winbase(card) + buff, skb->data, maxlen);
openwin(card, page + 1);
memcpy_toio(winbase(card), skb->data + maxlen, len - maxlen);
- } else
+ } else {
memcpy_toio(winbase(card) + buff, skb->data, len);
+ }
#ifndef PAGE0_ALWAYS_MAPPED
openwin(card, 0); /* select pkt_desc table page back */
return NETDEV_TX_OK;
}
-
#ifdef NEED_DETECT_RAM
static u32 sca_detect_ram(card_t *card, u8 __iomem *rambase, u32 ramsize)
{
}
#endif /* NEED_DETECT_RAM */
-
static void sca_init(card_t *card, int wait_states)
{
sca_out(wait_states, WCRL, card); /* Wait Control */
#define NAPI_WEIGHT 16
-#define get_msci(port) (port->chan ? MSCI1_OFFSET : MSCI0_OFFSET)
-#define get_dmac_rx(port) (port->chan ? DMAC1RX_OFFSET : DMAC0RX_OFFSET)
-#define get_dmac_tx(port) (port->chan ? DMAC1TX_OFFSET : DMAC0TX_OFFSET)
+#define get_msci(port) ((port)->chan ? MSCI1_OFFSET : MSCI0_OFFSET)
+#define get_dmac_rx(port) ((port)->chan ? DMAC1RX_OFFSET : DMAC0RX_OFFSET)
+#define get_dmac_tx(port) ((port)->chan ? DMAC1TX_OFFSET : DMAC0TX_OFFSET)
-#define sca_in(reg, card) readb(card->scabase + (reg))
-#define sca_out(value, reg, card) writeb(value, card->scabase + (reg))
-#define sca_inw(reg, card) readw(card->scabase + (reg))
-#define sca_outw(value, reg, card) writew(value, card->scabase + (reg))
-#define sca_inl(reg, card) readl(card->scabase + (reg))
-#define sca_outl(value, reg, card) writel(value, card->scabase + (reg))
+#define sca_in(reg, card) readb((card)->scabase + (reg))
+#define sca_out(value, reg, card) writeb(value, (card)->scabase + (reg))
+#define sca_inw(reg, card) readw((card)->scabase + (reg))
+#define sca_outw(value, reg, card) writew(value, (card)->scabase + (reg))
+#define sca_inl(reg, card) readl((card)->scabase + (reg))
+#define sca_outl(value, reg, card) writel(value, (card)->scabase + (reg))
static int sca_poll(struct napi_struct *napi, int budget);
-static inline port_t* dev_to_port(struct net_device *dev)
+static inline port_t *dev_to_port(struct net_device *dev)
{
return dev_to_hdlc(dev)->priv;
}
return port->chan * (rx_buffs + tx_buffs) + transmit * rx_buffs + desc;
}
-
static inline u16 desc_offset(port_t *port, u16 desc, int transmit)
{
/* Descriptor offset always fits in 16 bits */
return desc_abs_number(port, desc, transmit) * sizeof(pkt_desc);
}
-
static inline pkt_desc __iomem *desc_address(port_t *port, u16 desc,
int transmit)
{
desc_offset(port, desc, transmit));
}
-
static inline u32 buffer_offset(port_t *port, u16 desc, int transmit)
{
return port->card->buff_offset +
desc_abs_number(port, desc, transmit) * (u32)HDLC_MAX_MRU;
}
-
static inline void sca_set_carrier(port_t *port)
{
if (!(sca_in(get_msci(port) + ST3, port->card) & ST3_DCD)) {
}
}
-
static void sca_init_port(port_t *port)
{
card_t *card = port->card;
netif_napi_add(port->netdev, &port->napi, sca_poll, NAPI_WEIGHT);
}
-
/* MSCI interrupt service */
static inline void sca_msci_intr(port_t *port)
{
u16 msci = get_msci(port);
- card_t* card = port->card;
+ card_t *card = port->card;
if (sca_in(msci + ST1, card) & ST1_CDCD) {
/* Reset MSCI CDCD status bit */
}
}
-
static inline void sca_rx(card_t *card, port_t *port, pkt_desc __iomem *desc,
u16 rxin)
{
netif_receive_skb(skb);
}
-
/* Receive DMA service */
static inline int sca_rx_done(port_t *port, int budget)
{
return received;
}
-
/* Transmit DMA service */
static inline void sca_tx_done(port_t *port)
{
struct net_device *dev = port->netdev;
- card_t* card = port->card;
+ card_t *card = port->card;
u8 stat;
unsigned count = 0;
spin_unlock(&port->lock);
}
-
static int sca_poll(struct napi_struct *napi, int budget)
{
port_t *port = container_of(napi, port_t, napi);
return IRQ_RETVAL(handled);
}
-
static void sca_set_port(port_t *port)
{
- card_t* card = port->card;
+ card_t *card = port->card;
u16 msci = get_msci(port);
u8 md2 = sca_in(msci + MD2, card);
unsigned int tmc, br = 10, brv = 1024;
-
if (port->settings.clock_rate > 0) {
/* Try lower br for better accuracy*/
do {
/* Baud Rate = CLOCK_BASE / TMC / 2^BR */
tmc = CLOCK_BASE / brv / port->settings.clock_rate;
- }while (br > 1 && tmc <= 128);
+ } while (br > 1 && tmc <= 128);
if (tmc < 1) {
tmc = 1;
br = 0; /* For baud=CLOCK_BASE we use tmc=1 br=0 */
brv = 1;
- } else if (tmc > 255)
+ } else if (tmc > 255) {
tmc = 256; /* tmc=0 means 256 - low baud rates */
+ }
port->settings.clock_rate = CLOCK_BASE / brv / tmc;
} else {
md2 &= ~MD2_LOOPBACK;
sca_out(md2, msci + MD2, card);
-
}
-
static void sca_open(struct net_device *dev)
{
port_t *port = dev_to_port(dev);
- card_t* card = port->card;
+ card_t *card = port->card;
u16 msci = get_msci(port);
u8 md0, md2;
- switch(port->encoding) {
- case ENCODING_NRZ: md2 = MD2_NRZ; break;
- case ENCODING_NRZI: md2 = MD2_NRZI; break;
- case ENCODING_FM_MARK: md2 = MD2_FM_MARK; break;
- case ENCODING_FM_SPACE: md2 = MD2_FM_SPACE; break;
- default: md2 = MD2_MANCHESTER;
+ switch (port->encoding) {
+ case ENCODING_NRZ:
+ md2 = MD2_NRZ;
+ break;
+ case ENCODING_NRZI:
+ md2 = MD2_NRZI;
+ break;
+ case ENCODING_FM_MARK:
+ md2 = MD2_FM_MARK;
+ break;
+ case ENCODING_FM_SPACE:
+ md2 = MD2_FM_SPACE;
+ break;
+ default:
+ md2 = MD2_MANCHESTER;
}
if (port->settings.loopback)
md2 |= MD2_LOOPBACK;
- switch(port->parity) {
- case PARITY_CRC16_PR0: md0 = MD0_HDLC | MD0_CRC_16_0; break;
- case PARITY_CRC16_PR1: md0 = MD0_HDLC | MD0_CRC_16; break;
- case PARITY_CRC32_PR1_CCITT: md0 = MD0_HDLC | MD0_CRC_ITU32; break;
- case PARITY_CRC16_PR1_CCITT: md0 = MD0_HDLC | MD0_CRC_ITU; break;
- default: md0 = MD0_HDLC | MD0_CRC_NONE;
+ switch (port->parity) {
+ case PARITY_CRC16_PR0:
+ md0 = MD0_HDLC | MD0_CRC_16_0;
+ break;
+ case PARITY_CRC16_PR1:
+ md0 = MD0_HDLC | MD0_CRC_16;
+ break;
+ case PARITY_CRC32_PR1_CCITT:
+ md0 = MD0_HDLC | MD0_CRC_ITU32;
+ break;
+ case PARITY_CRC16_PR1_CCITT:
+ md0 = MD0_HDLC | MD0_CRC_ITU;
+ break;
+ default:
+ md0 = MD0_HDLC | MD0_CRC_NONE;
}
sca_out(CMD_RESET, msci + CMD, card);
netif_start_queue(dev);
}
-
static void sca_close(struct net_device *dev)
{
port_t *port = dev_to_port(dev);
netif_stop_queue(dev);
}
-
static int sca_attach(struct net_device *dev, unsigned short encoding,
unsigned short parity)
{
return 0;
}
-
#ifdef DEBUG_RINGS
static void sca_dump_rings(struct net_device *dev)
{
}
#endif /* DEBUG_RINGS */
-
static netdev_tx_t sca_xmit(struct sk_buff *skb, struct net_device *dev)
{
port_t *port = dev_to_port(dev);
return NETDEV_TX_OK;
}
-
static u32 sca_detect_ram(card_t *card, u8 __iomem *rambase, u32 ramsize)
{
/* Round RAM size to 32 bits, fill from end to start */
return i;
}
-
static void sca_init(card_t *card, int wait_states)
{
sca_out(wait_states, WCRL, card); /* Wait Control */
#include <linux/slab.h>
#include <net/net_namespace.h>
-
-static const char* version = "HDLC support module revision 1.22";
+static const char *version = "HDLC support module revision 1.22";
#undef DEBUG_LINK
return hdlc->xmit(skb, dev); /* call hardware driver directly */
}
+EXPORT_SYMBOL(hdlc_start_xmit);
static inline void hdlc_proto_start(struct net_device *dev)
{
hdlc_device *hdlc = dev_to_hdlc(dev);
+
if (hdlc->proto->start)
hdlc->proto->start(dev);
}
-
-
static inline void hdlc_proto_stop(struct net_device *dev)
{
hdlc_device *hdlc = dev_to_hdlc(dev);
+
if (hdlc->proto->stop)
hdlc->proto->stop(dev);
}
-
-
static int hdlc_device_event(struct notifier_block *this, unsigned long event,
void *ptr)
{
return NOTIFY_DONE;
}
-
-
/* Must be called by hardware driver when HDLC device is being opened */
int hdlc_open(struct net_device *dev)
{
hdlc->carrier, hdlc->open);
#endif
- if (hdlc->proto == NULL)
+ if (!hdlc->proto)
return -ENOSYS; /* no protocol attached */
if (hdlc->proto->open) {
int result = hdlc->proto->open(dev);
+
if (result)
return result;
}
if (hdlc->carrier) {
netdev_info(dev, "Carrier detected\n");
hdlc_proto_start(dev);
- } else
+ } else {
netdev_info(dev, "No carrier\n");
+ }
hdlc->open = 1;
spin_unlock_irq(&hdlc->state_lock);
return 0;
}
-
-
+EXPORT_SYMBOL(hdlc_open);
/* Must be called by hardware driver when HDLC device is being closed */
void hdlc_close(struct net_device *dev)
if (hdlc->proto->close)
hdlc->proto->close(dev);
}
-
-
+EXPORT_SYMBOL(hdlc_close);
int hdlc_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd)
{
/* Not handled by currently attached protocol (if any) */
while (proto) {
- if ((result = proto->ioctl(dev, ifr)) != -EINVAL)
+ result = proto->ioctl(dev, ifr);
+ if (result != -EINVAL)
return result;
proto = proto->next;
}
return -EINVAL;
}
+EXPORT_SYMBOL(hdlc_ioctl);
static const struct header_ops hdlc_null_ops;
struct net_device *alloc_hdlcdev(void *priv)
{
struct net_device *dev;
+
dev = alloc_netdev(sizeof(struct hdlc_device), "hdlc%d",
NET_NAME_UNKNOWN, hdlc_setup);
if (dev)
dev_to_hdlc(dev)->priv = priv;
return dev;
}
+EXPORT_SYMBOL(alloc_hdlcdev);
void unregister_hdlc_device(struct net_device *dev)
{
unregister_netdevice(dev);
rtnl_unlock();
}
-
-
+EXPORT_SYMBOL(unregister_hdlc_device);
int attach_hdlc_protocol(struct net_device *dev, struct hdlc_proto *proto,
size_t size)
if (size) {
dev_to_hdlc(dev)->state = kmalloc(size, GFP_KERNEL);
- if (dev_to_hdlc(dev)->state == NULL) {
+ if (!dev_to_hdlc(dev)->state) {
module_put(proto->module);
return -ENOBUFS;
}
return 0;
}
-
+EXPORT_SYMBOL(attach_hdlc_protocol);
int detach_hdlc_protocol(struct net_device *dev)
{
return 0;
}
-
+EXPORT_SYMBOL(detach_hdlc_protocol);
void register_hdlc_protocol(struct hdlc_proto *proto)
{
first_proto = proto;
rtnl_unlock();
}
-
+EXPORT_SYMBOL(register_hdlc_protocol);
void unregister_hdlc_protocol(struct hdlc_proto *proto)
{
*p = proto->next;
rtnl_unlock();
}
-
-
+EXPORT_SYMBOL(unregister_hdlc_protocol);
MODULE_AUTHOR("Krzysztof Halasa <khc@pm.waw.pl>");
MODULE_DESCRIPTION("HDLC support module");
MODULE_LICENSE("GPL v2");
-EXPORT_SYMBOL(hdlc_start_xmit);
-EXPORT_SYMBOL(hdlc_open);
-EXPORT_SYMBOL(hdlc_close);
-EXPORT_SYMBOL(hdlc_ioctl);
-EXPORT_SYMBOL(alloc_hdlcdev);
-EXPORT_SYMBOL(unregister_hdlc_device);
-EXPORT_SYMBOL(register_hdlc_protocol);
-EXPORT_SYMBOL(unregister_hdlc_protocol);
-EXPORT_SYMBOL(attach_hdlc_protocol);
-EXPORT_SYMBOL(detach_hdlc_protocol);
-
static struct packet_type hdlc_packet_type __read_mostly = {
.type = cpu_to_be16(ETH_P_HDLC),
.func = hdlc_rcv,
};
-
static struct notifier_block hdlc_notifier = {
.notifier_call = hdlc_device_event,
};
-
static int __init hdlc_module_init(void)
{
int result;
pr_info("%s\n", version);
- if ((result = register_netdevice_notifier(&hdlc_notifier)) != 0)
+ result = register_netdevice_notifier(&hdlc_notifier);
+ if (result)
return result;
dev_add_pack(&hdlc_packet_type);
return 0;
}
-
-
static void __exit hdlc_module_exit(void)
{
dev_remove_pack(&hdlc_packet_type);
unregister_netdevice_notifier(&hdlc_notifier);
}
-
module_init(hdlc_module_init);
module_exit(hdlc_module_exit);
#define CISCO_ADDR_REPLY 1 /* Cisco address reply */
#define CISCO_KEEPALIVE_REQ 2 /* Cisco keepalive request */
-
struct hdlc_header {
u8 address;
u8 control;
__be16 protocol;
-}__packed;
-
+} __packed;
struct cisco_packet {
__be32 type; /* code */
__be32 par2;
__be16 rel; /* reliability */
__be32 time;
-}__packed;
+} __packed;
#define CISCO_PACKET_LEN 18
#define CISCO_BIG_PACKET_LEN 20
-
struct cisco_state {
cisco_proto settings;
u32 rxseq; /* RX sequence number */
};
-
static int cisco_ioctl(struct net_device *dev, struct ifreq *ifr);
-
-static inline struct cisco_state* state(hdlc_device *hdlc)
+static inline struct cisco_state *state(hdlc_device *hdlc)
{
return (struct cisco_state *)hdlc->state;
}
-
static int cisco_hard_header(struct sk_buff *skb, struct net_device *dev,
u16 type, const void *daddr, const void *saddr,
unsigned int len)
#endif
skb_push(skb, sizeof(struct hdlc_header));
- data = (struct hdlc_header*)skb->data;
+ data = (struct hdlc_header *)skb->data;
if (type == CISCO_KEEPALIVE)
data->address = CISCO_MULTICAST;
else
return sizeof(struct hdlc_header);
}
-
-
static void cisco_keepalive_send(struct net_device *dev, u32 type,
__be32 par1, __be32 par2)
{
skb = dev_alloc_skb(sizeof(struct hdlc_header) +
sizeof(struct cisco_packet));
- if (!skb) {
- netdev_warn(dev, "Memory squeeze on %s()\n", __func__);
+ if (!skb)
return;
- }
+
skb_reserve(skb, 4);
cisco_hard_header(skb, dev, CISCO_KEEPALIVE, NULL, NULL, 0);
- data = (struct cisco_packet*)(skb->data + 4);
+ data = (struct cisco_packet *)(skb->data + 4);
data->type = htonl(type);
data->par1 = par1;
dev_queue_xmit(skb);
}
-
-
static __be16 cisco_type_trans(struct sk_buff *skb, struct net_device *dev)
{
- struct hdlc_header *data = (struct hdlc_header*)skb->data;
+ struct hdlc_header *data = (struct hdlc_header *)skb->data;
if (skb->len < sizeof(struct hdlc_header))
return cpu_to_be16(ETH_P_HDLC);
}
}
-
static int cisco_rx(struct sk_buff *skb)
{
struct net_device *dev = skb->dev;
hdlc_device *hdlc = dev_to_hdlc(dev);
struct cisco_state *st = state(hdlc);
- struct hdlc_header *data = (struct hdlc_header*)skb->data;
+ struct hdlc_header *data = (struct hdlc_header *)skb->data;
struct cisco_packet *cisco_data;
struct in_device *in_dev;
__be32 addr, mask;
goto rx_error;
}
- cisco_data = (struct cisco_packet*)(skb->data + sizeof
+ cisco_data = (struct cisco_packet *)(skb->data + sizeof
(struct hdlc_header));
- switch (ntohl (cisco_data->type)) {
+ switch (ntohl(cisco_data->type)) {
case CISCO_ADDR_REQ: /* Stolen from syncppp.c :-) */
rcu_read_lock();
in_dev = __in_dev_get_rcu(dev);
st->last_poll = jiffies;
if (!st->up) {
u32 sec, min, hrs, days;
+
sec = ntohl(cisco_data->time) / 1000;
min = sec / 60; sec -= min * 60;
hrs = min / 60; min -= hrs * 60;
return NET_RX_DROP;
}
-
-
static void cisco_timer(struct timer_list *t)
{
struct cisco_state *st = from_timer(st, t, timer);
add_timer(&st->timer);
}
-
-
static void cisco_start(struct net_device *dev)
{
hdlc_device *hdlc = dev_to_hdlc(dev);
add_timer(&st->timer);
}
-
-
static void cisco_stop(struct net_device *dev)
{
hdlc_device *hdlc = dev_to_hdlc(dev);
spin_unlock_irqrestore(&st->lock, flags);
}
-
static struct hdlc_proto proto = {
.start = cisco_start,
.stop = cisco_stop,
new_settings.timeout < 2)
return -EINVAL;
- result = hdlc->attach(dev, ENCODING_NRZ,PARITY_CRC16_PR1_CCITT);
+ result = hdlc->attach(dev, ENCODING_NRZ,
+ PARITY_CRC16_PR1_CCITT);
if (result)
return result;
return -EINVAL;
}
-
static int __init mod_init(void)
{
register_hdlc_protocol(&proto);
return 0;
}
-
-
static void __exit mod_exit(void)
{
unregister_hdlc_protocol(&proto);
}
-
module_init(mod_init);
module_exit(mod_exit);
* Copyright (C) 1999 - 2006 Krzysztof Halasa <khc@pm.waw.pl>
*
- Theory of PVC state
+ Theory of PVC state
DCE mode:
(exist,new) -> 0,0 when "PVC create" or if "link unreliable"
- 0,x -> 1,1 if "link reliable" when sending FULL STATUS
- 1,1 -> 1,0 if received FULL STATUS ACK
+ 0,x -> 1,1 if "link reliable" when sending FULL STATUS
+ 1,1 -> 1,0 if received FULL STATUS ACK
(active) -> 0 when "ifconfig PVC down" or "link unreliable" or "PVC create"
- -> 1 when "PVC up" and (exist,new) = 1,0
+ -> 1 when "PVC up" and (exist,new) = 1,0
DTE mode:
(exist,new,active) = FULL STATUS if "link reliable"
#define NLPID_CCITT_ANSI_LMI 0x08
#define NLPID_CISCO_LMI 0x09
-
#define LMI_CCITT_ANSI_DLCI 0 /* LMI DLCI */
#define LMI_CISCO_DLCI 1023
#define LMI_CCITT_CISCO_LENGTH 13 /* LMI frame lengths */
#define LMI_ANSI_LENGTH 14
-
struct fr_hdr {
#if defined(__LITTLE_ENDIAN_BITFIELD)
unsigned ea1: 1;
#endif
} __packed;
-
struct pvc_device {
struct net_device *frad;
struct net_device *main;
unsigned int fecn: 1;
unsigned int becn: 1;
unsigned int bandwidth; /* Cisco LMI reporting only */
- }state;
+ } state;
};
struct frad_state {
u8 rxseq; /* RX sequence number */
};
-
static int fr_ioctl(struct net_device *dev, struct ifreq *ifr);
-
static inline u16 q922_to_dlci(u8 *hdr)
{
return ((hdr[0] & 0xFC) << 2) | ((hdr[1] & 0xF0) >> 4);
}
-
static inline void dlci_to_q922(u8 *hdr, u16 dlci)
{
hdr[0] = (dlci >> 2) & 0xFC;
hdr[1] = ((dlci << 4) & 0xF0) | 0x01;
}
-
-static inline struct frad_state* state(hdlc_device *hdlc)
+static inline struct frad_state *state(hdlc_device *hdlc)
{
- return(struct frad_state *)(hdlc->state);
+ return (struct frad_state *)(hdlc->state);
}
-
static inline struct pvc_device *find_pvc(hdlc_device *hdlc, u16 dlci)
{
struct pvc_device *pvc = state(hdlc)->first_pvc;
return NULL;
}
-
static struct pvc_device *add_pvc(struct net_device *dev, u16 dlci)
{
hdlc_device *hdlc = dev_to_hdlc(dev);
return pvc;
}
-
static inline int pvc_is_used(struct pvc_device *pvc)
{
return pvc->main || pvc->ether;
}
-
static inline void pvc_carrier(int on, struct pvc_device *pvc)
{
if (on) {
}
}
-
static inline void delete_unused_pvcs(hdlc_device *hdlc)
{
struct pvc_device **pvc_p = &state(hdlc)->first_pvc;
}
}
-
static inline struct net_device **get_dev_p(struct pvc_device *pvc,
int type)
{
return &pvc->main;
}
-
static int fr_hard_header(struct sk_buff *skb, u16 dlci)
{
if (!skb->dev) { /* Control packets */
return 0;
}
-
-
static int pvc_open(struct net_device *dev)
{
struct pvc_device *pvc = dev->ml_priv;
if (pvc->open_count++ == 0) {
hdlc_device *hdlc = dev_to_hdlc(pvc->frad);
+
if (state(hdlc)->settings.lmi == LMI_NONE)
pvc->state.active = netif_carrier_ok(pvc->frad);
return 0;
}
-
-
static int pvc_close(struct net_device *dev)
{
struct pvc_device *pvc = dev->ml_priv;
if (--pvc->open_count == 0) {
hdlc_device *hdlc = dev_to_hdlc(pvc->frad);
+
if (state(hdlc)->settings.lmi == LMI_NONE)
pvc->state.active = 0;
return 0;
}
-
-
static int pvc_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd)
{
struct pvc_device *pvc = dev->ml_priv;
pvc->state.active ? "active" : "inactive");
}
-
-
static inline u8 fr_lmi_nextseq(u8 x)
{
x++;
return x ? x : 1;
}
-
static void fr_lmi_send(struct net_device *dev, int fullrep)
{
hdlc_device *hdlc = dev_to_hdlc(dev);
}
skb = dev_alloc_skb(len);
- if (!skb) {
- netdev_warn(dev, "Memory squeeze on fr_lmi_send()\n");
+ if (!skb)
return;
- }
+
memset(skb->data, 0, len);
skb_reserve(skb, 4);
- if (lmi == LMI_CISCO) {
+ if (lmi == LMI_CISCO)
fr_hard_header(skb, LMI_CISCO_DLCI);
- } else {
+ else
fr_hard_header(skb, LMI_CCITT_ANSI_DLCI);
- }
+
data = skb_tail_pointer(skb);
data[i++] = LMI_CALLREF;
data[i++] = dce ? LMI_STATUS : LMI_STATUS_ENQUIRY;
dev_queue_xmit(skb);
}
-
-
static void fr_set_link_state(int reliable, struct net_device *dev)
{
hdlc_device *hdlc = dev_to_hdlc(dev);
}
}
-
static void fr_timer(struct timer_list *t)
{
struct frad_state *st = from_timer(st, t, timer);
fr_set_link_state(reliable, dev);
}
- if (state(hdlc)->settings.dce)
+ if (state(hdlc)->settings.dce) {
state(hdlc)->timer.expires = jiffies +
state(hdlc)->settings.t392 * HZ;
- else {
+ } else {
if (state(hdlc)->n391cnt)
state(hdlc)->n391cnt--;
add_timer(&state(hdlc)->timer);
}
-
static int fr_lmi_recv(struct net_device *dev, struct sk_buff *skb)
{
hdlc_device *hdlc = dev_to_hdlc(dev);
return 1;
}
i = 7;
- } else
+ } else {
i = 6;
+ }
if (skb->data[i] != (lmi == LMI_CCITT ? LMI_CCITT_REPTYPE :
LMI_ANSI_CISCO_REPTYPE)) {
}
i++;
- new = !! (skb->data[i + 2] & 0x08);
- active = !! (skb->data[i + 2] & 0x02);
+ new = !!(skb->data[i + 2] & 0x08);
+ active = !!(skb->data[i + 2] & 0x02);
if (lmi == LMI_CISCO) {
dlci = (skb->data[i] << 8) | skb->data[i + 1];
bw = (skb->data[i + 3] << 16) |
pvc->state.becn ^= 1;
}
-
- if ((skb = skb_share_check(skb, GFP_ATOMIC)) == NULL) {
+ skb = skb_share_check(skb, GFP_ATOMIC);
+ if (!skb) {
frad->stats.rx_dropped++;
return NET_RX_DROP;
}
return NET_RX_DROP;
}
-
-
static void fr_start(struct net_device *dev)
{
hdlc_device *hdlc = dev_to_hdlc(dev);
/* First poll after 1 s */
state(hdlc)->timer.expires = jiffies + HZ;
add_timer(&state(hdlc)->timer);
- } else
+ } else {
fr_set_link_state(1, dev);
+ }
}
-
static void fr_stop(struct net_device *dev)
{
hdlc_device *hdlc = dev_to_hdlc(dev);
fr_set_link_state(0, dev);
}
-
static void fr_close(struct net_device *dev)
{
hdlc_device *hdlc = dev_to_hdlc(dev);
}
}
-
static void pvc_setup(struct net_device *dev)
{
dev->type = ARPHRD_DLCI;
struct net_device *dev;
int used;
- if ((pvc = add_pvc(frad, dlci)) == NULL) {
+ pvc = add_pvc(frad, dlci);
+ if (!pvc) {
netdev_warn(frad, "Memory squeeze on fr_add_pvc()\n");
return -ENOBUFS;
}
dev->priv_flags &= ~IFF_TX_SKB_SHARING;
eth_hw_addr_random(dev);
} else {
- *(__be16*)dev->dev_addr = htons(dlci);
+ *(__be16 *)dev->dev_addr = htons(dlci);
dlci_to_q922(dev->broadcast, dlci);
}
dev->netdev_ops = &pvc_ops;
return 0;
}
-
-
static int fr_del_pvc(hdlc_device *hdlc, unsigned int dlci, int type)
{
struct pvc_device *pvc;
struct net_device *dev;
- if ((pvc = find_pvc(hdlc, dlci)) == NULL)
+ pvc = find_pvc(hdlc, dlci);
+ if (!pvc)
return -ENOENT;
- if ((dev = *get_dev_p(pvc, type)) == NULL)
+ dev = *get_dev_p(pvc, type);
+ if (!dev)
return -ENOENT;
if (dev->flags & IFF_UP)
return 0;
}
-
-
static void fr_destroy(struct net_device *frad)
{
hdlc_device *hdlc = dev_to_hdlc(frad);
struct pvc_device *pvc = state(hdlc)->first_pvc;
+
state(hdlc)->first_pvc = NULL; /* All PVCs destroyed */
state(hdlc)->dce_pvc_count = 0;
state(hdlc)->dce_changed = 1;
}
}
-
static struct hdlc_proto proto = {
.close = fr_close,
.start = fr_start,
.module = THIS_MODULE,
};
-
static int fr_ioctl(struct net_device *dev, struct ifreq *ifr)
{
fr_proto __user *fr_s = ifr->ifr_settings.ifs_ifsu.fr;
new_settings.dce != 1))
return -EINVAL;
- result=hdlc->attach(dev, ENCODING_NRZ,PARITY_CRC16_PR1_CCITT);
+ result = hdlc->attach(dev, ENCODING_NRZ,
+ PARITY_CRC16_PR1_CCITT);
if (result)
return result;
return -EINVAL;
}
-
static int __init mod_init(void)
{
register_hdlc_protocol(&proto);
return 0;
}
-
static void __exit mod_exit(void)
{
unregister_hdlc_protocol(&proto);
}
-
module_init(mod_init);
module_exit(mod_exit);
"0", "ConfReq", "ConfAck", "ConfNak", "ConfRej", "TermReq",
"TermAck", "CodeRej", "ProtoRej", "EchoReq", "EchoReply", "Discard"
};
+
static char debug_buffer[64 + 3 * DEBUG_CP];
#endif
__be16 len;
};
-
struct proto {
struct net_device *dev;
struct timer_list timer;
"Closed", "Stopped", "Stopping", "ReqSent", "AckRecv", "AckSent",
"Opened"
};
+
static const char *const event_names[EVENTS] = {
"Start", "Stop", "TO+", "TO-", "RCR+", "RCR-", "RCA", "RCN",
"RTR", "RTA", "RUC", "RXJ+", "RXJ-"
static int ppp_ioctl(struct net_device *dev, struct ifreq *ifr);
-static inline struct ppp* get_ppp(struct net_device *dev)
+static inline struct ppp *get_ppp(struct net_device *dev)
{
return (struct ppp *)dev_to_hdlc(dev)->state;
}
-static inline struct proto* get_proto(struct net_device *dev, u16 pid)
+static inline struct proto *get_proto(struct net_device *dev, u16 pid)
{
struct ppp *ppp = get_ppp(dev);
}
}
-static inline const char* proto_name(u16 pid)
+static inline const char *proto_name(u16 pid)
{
switch (pid) {
case PID_LCP:
static __be16 ppp_type_trans(struct sk_buff *skb, struct net_device *dev)
{
- struct hdlc_header *data = (struct hdlc_header*)skb->data;
+ struct hdlc_header *data = (struct hdlc_header *)skb->data;
if (skb->len < sizeof(struct hdlc_header))
return htons(ETH_P_HDLC);
}
}
-
static int ppp_hard_header(struct sk_buff *skb, struct net_device *dev,
u16 type, const void *daddr, const void *saddr,
unsigned int len)
#endif
skb_push(skb, sizeof(struct hdlc_header));
- data = (struct hdlc_header*)skb->data;
+ data = (struct hdlc_header *)skb->data;
data->address = HDLC_ADDR_ALLSTATIONS;
data->control = HDLC_CTRL_UI;
return sizeof(struct hdlc_header);
}
-
static void ppp_tx_flush(void)
{
struct sk_buff *skb;
+
while ((skb = skb_dequeue(&tx_queue)) != NULL)
dev_queue_xmit(skb);
}
skb = dev_alloc_skb(sizeof(struct hdlc_header) +
sizeof(struct cp_header) + magic_len + len);
- if (!skb) {
- netdev_warn(dev, "out of memory in ppp_tx_cp()\n");
+ if (!skb)
return;
- }
+
skb_reserve(skb, sizeof(struct hdlc_header));
cp = skb_put(skb, sizeof(struct cp_header));
skb_queue_tail(&tx_queue, skb);
}
-
/* State transition table (compare STD-51)
Events Actions
TO+ = Timeout with counter > 0 irc = Initialize-Restart-Count
{ 0 , 1 , 1 , 1 , 1 , 1 ,IRC|STR|2}, /* RXJ- */
};
-
/* SCA: RCR+ must supply id, len and data
SCN: RCR- must supply code, id, len and data
STA: RTR must supply id
#endif
}
-
static void ppp_cp_parse_cr(struct net_device *dev, u16 pid, u8 id,
unsigned int req_len, const u8 *data)
{
u8 *out;
unsigned int len = req_len, nak_len = 0, rej_len = 0;
- if (!(out = kmalloc(len, GFP_ATOMIC))) {
+ out = kmalloc(len, GFP_ATOMIC);
+ if (!out) {
dev->stats.rx_dropped++;
return; /* out of memory, ignore CR packet */
}
static int ppp_rx(struct sk_buff *skb)
{
- struct hdlc_header *hdr = (struct hdlc_header*)skb->data;
+ struct hdlc_header *hdr = (struct hdlc_header *)skb->data;
struct net_device *dev = skb->dev;
struct ppp *ppp = get_ppp(dev);
struct proto *proto;
if (pid == PID_LCP)
switch (cp->code) {
case LCP_PROTO_REJ:
- pid = ntohs(*(__be16*)skb->data);
+ pid = ntohs(*(__be16 *)skb->data);
if (pid == PID_LCP || pid == PID_IPCP ||
pid == PID_IPV6CP)
ppp_cp_event(dev, pid, RXJ_BAD, 0, 0,
ppp_tx_flush();
}
-
static void ppp_start(struct net_device *dev)
{
struct ppp *ppp = get_ppp(dev);
for (i = 0; i < IDX_COUNT; i++) {
struct proto *proto = &ppp->protos[i];
+
proto->dev = dev;
timer_setup(&proto->timer, ppp_timer, 0);
proto->state = CLOSED;
/* no settable parameters */
- result = hdlc->attach(dev, ENCODING_NRZ,PARITY_CRC16_PR1_CCITT);
+ result = hdlc->attach(dev, ENCODING_NRZ,
+ PARITY_CRC16_PR1_CCITT);
if (result)
return result;
return -EINVAL;
}
-
static int __init mod_init(void)
{
skb_queue_head_init(&tx_queue);
unregister_hdlc_protocol(&proto);
}
-
module_init(mod_init);
module_exit(mod_exit);
unsigned char *ptr;
skb = __dev_alloc_skb(1, GFP_ATOMIC | __GFP_NOMEMALLOC);
- if (!skb) {
- netdev_err(dev, "out of memory\n");
+ if (!skb)
return;
- }
ptr = skb_put(skb, 1);
*ptr = code;
tasklet_schedule(&x25st->rx_tasklet);
}
-
-
static void x25_connected(struct net_device *dev, int reason)
{
x25_connect_disconnect(dev, reason, X25_IFACE_CONNECT);
}
-
-
static void x25_disconnected(struct net_device *dev, int reason)
{
x25_connect_disconnect(dev, reason, X25_IFACE_DISCONNECT);
}
-
-
static int x25_data_indication(struct net_device *dev, struct sk_buff *skb)
{
struct x25_state *x25st = state(dev_to_hdlc(dev));
return NET_RX_SUCCESS;
}
-
-
static void x25_data_transmit(struct net_device *dev, struct sk_buff *skb)
{
hdlc_device *hdlc = dev_to_hdlc(dev);
hdlc->xmit(skb, dev); /* Ignore return value :-( */
}
-
-
static netdev_tx_t x25_xmit(struct sk_buff *skb, struct net_device *dev)
{
hdlc_device *hdlc = dev_to_hdlc(dev);
switch (skb->data[0]) {
case X25_IFACE_DATA: /* Data to be transmitted */
skb_pull(skb, 1);
- if ((result = lapb_data_request(dev, skb)) != LAPB_OK)
+ result = lapb_data_request(dev, skb);
+ if (result != LAPB_OK)
dev_kfree_skb(skb);
spin_unlock_bh(&x25st->up_lock);
return NETDEV_TX_OK;
case X25_IFACE_CONNECT:
- if ((result = lapb_connect_request(dev))!= LAPB_OK) {
+ result = lapb_connect_request(dev);
+ if (result != LAPB_OK) {
if (result == LAPB_CONNECTED)
/* Send connect confirm. msg to level 3 */
x25_connected(dev, 0);
break;
case X25_IFACE_DISCONNECT:
- if ((result = lapb_disconnect_request(dev)) != LAPB_OK) {
+ result = lapb_disconnect_request(dev);
+ if (result != LAPB_OK) {
if (result == LAPB_NOTCONNECTED)
/* Send disconnect confirm. msg to level 3 */
x25_disconnected(dev, 0);
return NETDEV_TX_OK;
}
-
-
static int x25_open(struct net_device *dev)
{
static const struct lapb_register_struct cb = {
return 0;
}
-
-
static void x25_close(struct net_device *dev)
{
hdlc_device *hdlc = dev_to_hdlc(dev);
tasklet_kill(&x25st->rx_tasklet);
}
-
-
static int x25_rx(struct sk_buff *skb)
{
struct net_device *dev = skb->dev;
hdlc_device *hdlc = dev_to_hdlc(dev);
struct x25_state *x25st = state(hdlc);
- if ((skb = skb_share_check(skb, GFP_ATOMIC)) == NULL) {
+ skb = skb_share_check(skb, GFP_ATOMIC);
+ if (!skb) {
dev->stats.rx_dropped++;
return NET_RX_DROP;
}
return NET_RX_DROP;
}
-
static struct hdlc_proto proto = {
.open = x25_open,
.close = x25_close,
.module = THIS_MODULE,
};
-
static int x25_ioctl(struct net_device *dev, struct ifreq *ifr)
{
x25_hdlc_proto __user *x25_s = ifr->ifr_settings.ifs_ifsu.x25;
new_settings.t1 = 3;
new_settings.t2 = 1;
new_settings.n2 = 10;
- }
- else {
+ } else {
if (copy_from_user(&new_settings, x25_s, size))
return -EFAULT;
if ((new_settings.dce != 0 &&
- new_settings.dce != 1) ||
- (new_settings.modulo != 8 &&
- new_settings.modulo != 128) ||
- new_settings.window < 1 ||
- (new_settings.modulo == 8 &&
- new_settings.window > 7) ||
- (new_settings.modulo == 128 &&
- new_settings.window > 127) ||
- new_settings.t1 < 1 ||
- new_settings.t1 > 255 ||
- new_settings.t2 < 1 ||
- new_settings.t2 > 255 ||
- new_settings.n2 < 1 ||
- new_settings.n2 > 255)
+ new_settings.dce != 1) ||
+ (new_settings.modulo != 8 &&
+ new_settings.modulo != 128) ||
+ new_settings.window < 1 ||
+ (new_settings.modulo == 8 &&
+ new_settings.window > 7) ||
+ (new_settings.modulo == 128 &&
+ new_settings.window > 127) ||
+ new_settings.t1 < 1 ||
+ new_settings.t1 > 255 ||
+ new_settings.t2 < 1 ||
+ new_settings.t2 > 255 ||
+ new_settings.n2 < 1 ||
+ new_settings.n2 > 255)
return -EINVAL;
}
- result=hdlc->attach(dev, ENCODING_NRZ,PARITY_CRC16_PR1_CCITT);
+ result = hdlc->attach(dev, ENCODING_NRZ,
+ PARITY_CRC16_PR1_CCITT);
if (result)
return result;
- if ((result = attach_hdlc_protocol(dev, &proto,
- sizeof(struct x25_state))))
+ result = attach_hdlc_protocol(dev, &proto,
+ sizeof(struct x25_state));
+ if (result)
return result;
memcpy(&state(hdlc)->settings, &new_settings, size);
return -EINVAL;
}
-
static int __init mod_init(void)
{
register_hdlc_protocol(&proto);
return 0;
}
-
-
static void __exit mod_exit(void)
{
unregister_hdlc_protocol(&proto);
}
-
module_init(mod_init);
module_exit(mod_exit);
// SPDX-License-Identifier: GPL-2.0-only
-/*
- * Comtrol SV11 card driver
+/* Comtrol SV11 card driver
*
* This is a slightly odd Z85230 synchronous driver. All you need to
* know basically is
*
* It supports DMA using two DMA channels in SYNC mode. The driver doesn't
* use these facilities
- *
+ *
* The control port is at io+1, the data at io+3 and turning off the DMA
* is done by writing 0 to io+4
*
static int dma;
-/*
- * Network driver support routines
+/* Network driver support routines
*/
-static inline struct z8530_dev* dev_to_sv(struct net_device *dev)
+static inline struct z8530_dev *dev_to_sv(struct net_device *dev)
{
return (struct z8530_dev *)dev_to_hdlc(dev)->priv;
}
-/*
- * Frame receive. Simple for our card as we do HDLC and there
+/* Frame receive. Simple for our card as we do HDLC and there
* is no funny garbage involved
*/
skb->protocol = hdlc_type_trans(skb, c->netdevice);
skb_reset_mac_header(skb);
skb->dev = c->netdevice;
- /*
- * Send it to the PPP layer. We don't have time to process
+ /* Send it to the PPP layer. We don't have time to process
* it right now.
*/
netif_rx(skb);
}
-/*
- * We've been placed in the UP state
+/* We've been placed in the UP state
*/
static int hostess_open(struct net_device *d)
struct z8530_dev *sv11 = dev_to_sv(d);
int err = -1;
- /*
- * Link layer up
+ /* Link layer up
*/
switch (dma) {
- case 0:
- err = z8530_sync_open(d, &sv11->chanA);
- break;
- case 1:
- err = z8530_sync_dma_open(d, &sv11->chanA);
- break;
- case 2:
- err = z8530_sync_txdma_open(d, &sv11->chanA);
- break;
+ case 0:
+ err = z8530_sync_open(d, &sv11->chanA);
+ break;
+ case 1:
+ err = z8530_sync_dma_open(d, &sv11->chanA);
+ break;
+ case 2:
+ err = z8530_sync_txdma_open(d, &sv11->chanA);
+ break;
}
if (err)
err = hdlc_open(d);
if (err) {
switch (dma) {
- case 0:
- z8530_sync_close(d, &sv11->chanA);
- break;
- case 1:
- z8530_sync_dma_close(d, &sv11->chanA);
- break;
- case 2:
- z8530_sync_txdma_close(d, &sv11->chanA);
- break;
+ case 0:
+ z8530_sync_close(d, &sv11->chanA);
+ break;
+ case 1:
+ z8530_sync_dma_close(d, &sv11->chanA);
+ break;
+ case 2:
+ z8530_sync_txdma_close(d, &sv11->chanA);
+ break;
}
return err;
}
static int hostess_close(struct net_device *d)
{
struct z8530_dev *sv11 = dev_to_sv(d);
- /*
- * Discard new frames
+ /* Discard new frames
*/
sv11->chanA.rx_function = z8530_null_rx;
netif_stop_queue(d);
switch (dma) {
- case 0:
- z8530_sync_close(d, &sv11->chanA);
- break;
- case 1:
- z8530_sync_dma_close(d, &sv11->chanA);
- break;
- case 2:
- z8530_sync_txdma_close(d, &sv11->chanA);
- break;
+ case 0:
+ z8530_sync_close(d, &sv11->chanA);
+ break;
+ case 1:
+ z8530_sync_dma_close(d, &sv11->chanA);
+ break;
+ case 2:
+ z8530_sync_txdma_close(d, &sv11->chanA);
+ break;
}
return 0;
}
static int hostess_ioctl(struct net_device *d, struct ifreq *ifr, int cmd)
{
- /* struct z8530_dev *sv11=dev_to_sv(d);
- z8530_ioctl(d,&sv11->chanA,ifr,cmd) */
return hdlc_ioctl(d, ifr, cmd);
}
-/*
- * Passed network frames, fire them downwind.
+/* Passed network frames, fire them downwind.
*/
static netdev_tx_t hostess_queue_xmit(struct sk_buff *skb,
- struct net_device *d)
+ struct net_device *d)
{
return z8530_queue_xmit(&dev_to_sv(d)->chanA, skb);
}
return -EINVAL;
}
-/*
- * Description block for a Comtrol Hostess SV11 card
+/* Description block for a Comtrol Hostess SV11 card
*/
static const struct net_device_ops hostess_ops = {
{
struct z8530_dev *sv;
struct net_device *netdev;
- /*
- * Get the needed I/O space
+ /* Get the needed I/O space
*/
if (!request_region(iobase, 8, "Comtrol SV11")) {
if (!sv)
goto err_kzalloc;
- /*
- * Stuff in the I/O addressing
+ /* Stuff in the I/O addressing
*/
sv->active = 0;
outb(0, iobase + 4); /* DMA off */
/* We want a fast IRQ for this device. Actually we'd like an even faster
- IRQ ;) - This is one driver RtLinux is made for */
+ * IRQ ;) - This is one driver RtLinux is made for
+ */
if (request_irq(irq, z8530_interrupt, 0,
"Hostess SV11", sv) < 0) {
sv->chanB.dev = sv;
if (dma) {
- /*
- * You can have DMA off or 1 and 3 thats the lot
+ /* You can have DMA off or 1 and 3 thats the lot
* on the Comtrol.
*/
sv->chanA.txdma = 3;
}
/* Kill our private IRQ line the hostess can end up chattering
- until the configuration is set */
+ * until the configuration is set
+ */
disable_irq(irq);
- /*
- * Begin normal initialise
+ /* Begin normal initialise
*/
if (z8530_init(sv)) {
enable_irq(irq);
- /*
- * Now we can take the IRQ
+ /* Now we can take the IRQ
*/
sv->chanA.netdevice = netdev = alloc_hdlcdev(sv);
int init_module(void)
{
- if ((sv11_unit = sv11_init(io, irq)) == NULL)
+ sv11_unit = sv11_init(io, irq);
+ if (!sv11_unit)
return -ENODEV;
return 0;
}
#define PKT_HDLC_CRC_32 0x2 /* default = CRC-16 */
#define PKT_HDLC_MSB_ENDIAN 0x4 /* default = LE */
-
/* hss_config, PCRs */
/* Frame sync sampling, default = active low */
#define PCR_FRM_SYNC_ACTIVE_HIGH 0x40000000
/* HSS number, default = 0 (first) */
#define CCR_SECOND_HSS 0x01000000
-
/* hss_config, clkCR: main:10, num:10, denom:12 */
-#define CLK42X_SPEED_EXP ((0x3FF << 22) | ( 2 << 12) | 15) /*65 KHz*/
-
-#define CLK42X_SPEED_512KHZ (( 130 << 22) | ( 2 << 12) | 15)
-#define CLK42X_SPEED_1536KHZ (( 43 << 22) | ( 18 << 12) | 47)
-#define CLK42X_SPEED_1544KHZ (( 43 << 22) | ( 33 << 12) | 192)
-#define CLK42X_SPEED_2048KHZ (( 32 << 22) | ( 34 << 12) | 63)
-#define CLK42X_SPEED_4096KHZ (( 16 << 22) | ( 34 << 12) | 127)
-#define CLK42X_SPEED_8192KHZ (( 8 << 22) | ( 34 << 12) | 255)
-
-#define CLK46X_SPEED_512KHZ (( 130 << 22) | ( 24 << 12) | 127)
-#define CLK46X_SPEED_1536KHZ (( 43 << 22) | (152 << 12) | 383)
-#define CLK46X_SPEED_1544KHZ (( 43 << 22) | ( 66 << 12) | 385)
-#define CLK46X_SPEED_2048KHZ (( 32 << 22) | (280 << 12) | 511)
-#define CLK46X_SPEED_4096KHZ (( 16 << 22) | (280 << 12) | 1023)
-#define CLK46X_SPEED_8192KHZ (( 8 << 22) | (280 << 12) | 2047)
-
-/*
- * HSS_CONFIG_CLOCK_CR register consists of 3 parts:
+#define CLK42X_SPEED_EXP ((0x3FF << 22) | (2 << 12) | 15) /*65 KHz*/
+
+#define CLK42X_SPEED_512KHZ ((130 << 22) | (2 << 12) | 15)
+#define CLK42X_SPEED_1536KHZ ((43 << 22) | (18 << 12) | 47)
+#define CLK42X_SPEED_1544KHZ ((43 << 22) | (33 << 12) | 192)
+#define CLK42X_SPEED_2048KHZ ((32 << 22) | (34 << 12) | 63)
+#define CLK42X_SPEED_4096KHZ ((16 << 22) | (34 << 12) | 127)
+#define CLK42X_SPEED_8192KHZ ((8 << 22) | (34 << 12) | 255)
+
+#define CLK46X_SPEED_512KHZ ((130 << 22) | (24 << 12) | 127)
+#define CLK46X_SPEED_1536KHZ ((43 << 22) | (152 << 12) | 383)
+#define CLK46X_SPEED_1544KHZ ((43 << 22) | (66 << 12) | 385)
+#define CLK46X_SPEED_2048KHZ ((32 << 22) | (280 << 12) | 511)
+#define CLK46X_SPEED_4096KHZ ((16 << 22) | (280 << 12) | 1023)
+#define CLK46X_SPEED_8192KHZ ((8 << 22) | (280 << 12) | 2047)
+
+/* HSS_CONFIG_CLOCK_CR register consists of 3 parts:
* A (10 bits), B (10 bits) and C (12 bits).
* IXP42x HSS clock generator operation (verified with an oscilloscope):
* Each clock bit takes 7.5 ns (1 / 133.xx MHz).
#define HSS_CONFIG_TX_LUT 0x18 /* channel look-up tables */
#define HSS_CONFIG_RX_LUT 0x38
-
/* NPE command codes */
/* writes the ConfigWord value to the location specified by offset */
#define PORT_CONFIG_WRITE 0x40
#define PORT_ERROR_READ 0x42
/* triggers the NPE to reset internal status and enable the HssPacketized
- operation for the flow specified by pPipe */
+ * operation for the flow specified by pPipe
+ */
#define PKT_PIPE_FLOW_ENABLE 0x50
#define PKT_PIPE_FLOW_DISABLE 0x51
#define PKT_NUM_PIPES_WRITE 0x52
#define ERR_HDLC_ALIGN 2 /* HDLC alignment error */
#define ERR_HDLC_FCS 3 /* HDLC Frame Check Sum error */
#define ERR_RXFREE_Q_EMPTY 4 /* RX-free queue became empty while receiving
- this packet (if buf_len < pkt_len) */
+ * this packet (if buf_len < pkt_len)
+ */
#define ERR_HDLC_TOO_LONG 5 /* HDLC frame size too long */
#define ERR_HDLC_ABORT 6 /* abort sequence received */
#define ERR_DISCONNECTING 7 /* disconnect is in progress */
-
#ifdef __ARMEB__
typedef struct sk_buff buffer_t;
#define free_buffer dev_kfree_skb
u32 __reserved1[4];
};
-
#define rx_desc_phys(port, n) ((port)->desc_tab_phys + \
(n) * sizeof(struct desc))
#define rx_desc_ptr(port, n) (&(port)->desc_tab[n])
static const struct {
int tx, txdone, rx, rxfree;
-}queue_ids[2] = {{HSS0_PKT_TX0_QUEUE, HSS0_PKT_TXDONE_QUEUE, HSS0_PKT_RX_QUEUE,
+} queue_ids[2] = {{HSS0_PKT_TX0_QUEUE, HSS0_PKT_TXDONE_QUEUE, HSS0_PKT_RX_QUEUE,
HSS0_PKT_RXFREE0_QUEUE},
{HSS1_PKT_TX0_QUEUE, HSS1_PKT_TXDONE_QUEUE, HSS1_PKT_RX_QUEUE,
HSS1_PKT_RXFREE0_QUEUE},
* utility functions
****************************************************************************/
-static inline struct port* dev_to_port(struct net_device *dev)
+static inline struct port *dev_to_port(struct net_device *dev)
{
return dev_to_hdlc(dev)->priv;
}
static inline void memcpy_swab32(u32 *dest, u32 *src, int cnt)
{
int i;
+
for (i = 0; i < cnt; i++)
dest[i] = swab32(src[i]);
}
* HSS access
****************************************************************************/
-static void hss_npe_send(struct port *port, struct msg *msg, const char* what)
+static void hss_npe_send(struct port *port, struct msg *msg, const char *what)
{
- u32 *val = (u32*)msg;
+ u32 *val = (u32 *)msg;
+
if (npe_send_message(port->npe, msg, what)) {
pr_crit("HSS-%i: unable to send command [%08X:%08X] to %s\n",
port->id, val[0], val[1], npe_name(port->npe));
if (port->initialized)
return 0;
- if (!npe_running(port->npe) &&
- (err = npe_load_firmware(port->npe, npe_name(port->npe),
- port->dev)))
- return err;
+ if (!npe_running(port->npe)) {
+ err = npe_load_firmware(port->npe, npe_name(port->npe),
+ port->dev);
+ if (err)
+ return err;
+ }
/* HDLC mode configuration */
memset(&msg, 0, sizeof(msg));
#endif
}
-
static inline void debug_desc(u32 phys, struct desc *desc)
{
#if DEBUG_DESC
u32 phys, tab_phys, n_desc;
struct desc *tab;
- if (!(phys = qmgr_get_entry(queue)))
+ phys = qmgr_get_entry(queue);
+ if (!phys)
return -1;
BUG_ON(phys & 0x1F);
BUG_ON(phys & 0x1F);
qmgr_put_entry(queue, phys);
/* Don't check for queue overflow here, we've allocated sufficient
- length and queues >= 32 don't support this check anyway. */
+ * length and queues >= 32 don't support this check anyway.
+ */
}
-
static inline void dma_unmap_tx(struct port *port, struct desc *desc)
{
#ifdef __ARMEB__
#endif
}
-
static void hss_hdlc_set_carrier(void *pdev, int carrier)
{
struct net_device *netdev = pdev;
u32 phys;
#endif
- if ((n = queue_get_desc(rxq, port, 0)) < 0) {
+ n = queue_get_desc(rxq, port, 0);
+ if (n < 0) {
#if DEBUG_RX
printk(KERN_DEBUG "%s: hss_hdlc_poll"
" napi_complete\n", dev->name);
switch (desc->status) {
case 0:
#ifdef __ARMEB__
- if ((skb = netdev_alloc_skb(dev, RX_SIZE)) != NULL) {
+ skb = netdev_alloc_skb(dev, RX_SIZE);
+ if (skb) {
phys = dma_map_single(&dev->dev, skb->data,
RX_SIZE,
DMA_FROM_DEVICE);
return received; /* not all work done */
}
-
static void hss_hdlc_txdone_irq(void *pdev)
{
struct net_device *dev = pdev;
#else
offset = (int)skb->data & 3; /* keep 32-bit alignment */
bytes = ALIGN(offset + len, 4);
- if (!(mem = kmalloc(bytes, GFP_ATOMIC))) {
+ mem = kmalloc(bytes, GFP_ATOMIC);
+ if (!mem) {
dev_kfree_skb(skb);
dev->stats.tx_dropped++;
return NETDEV_TX_OK;
return NETDEV_TX_OK;
}
-
static int request_hdlc_queues(struct port *port)
{
int err;
return -ENOMEM;
}
- if (!(port->desc_tab = dma_pool_alloc(dma_pool, GFP_KERNEL,
- &port->desc_tab_phys)))
+ port->desc_tab = dma_pool_alloc(dma_pool, GFP_KERNEL,
+ &port->desc_tab_phys);
+ if (!port->desc_tab)
return -ENOMEM;
memset(port->desc_tab, 0, POOL_ALLOC_SIZE);
memset(port->rx_buff_tab, 0, sizeof(port->rx_buff_tab)); /* tables */
buffer_t *buff;
void *data;
#ifdef __ARMEB__
- if (!(buff = netdev_alloc_skb(port->netdev, RX_SIZE)))
+ buff = netdev_alloc_skb(port->netdev, RX_SIZE);
+ if (!buff)
return -ENOMEM;
data = buff->data;
#else
- if (!(buff = kmalloc(RX_SIZE, GFP_KERNEL)))
+ buff = kmalloc(RX_SIZE, GFP_KERNEL);
+ if (!buff)
return -ENOMEM;
data = buff;
#endif
for (i = 0; i < RX_DESCS; i++) {
struct desc *desc = rx_desc_ptr(port, i);
buffer_t *buff = port->rx_buff_tab[i];
+
if (buff) {
dma_unmap_single(&port->netdev->dev,
desc->data, RX_SIZE,
for (i = 0; i < TX_DESCS; i++) {
struct desc *desc = tx_desc_ptr(port, i);
buffer_t *buff = port->tx_buff_tab[i];
+
if (buff) {
dma_unmap_tx(port, desc);
free_buffer(buff);
unsigned long flags;
int i, err = 0;
- if ((err = hdlc_open(dev)))
+ err = hdlc_open(dev);
+ if (err)
return err;
- if ((err = hss_load_firmware(port)))
+ err = hss_load_firmware(port);
+ if (err)
goto err_hdlc_close;
- if ((err = request_hdlc_queues(port)))
+ err = request_hdlc_queues(port);
+ if (err)
goto err_hdlc_close;
- if ((err = init_hdlc_queues(port)))
+ err = init_hdlc_queues(port);
+ if (err)
goto err_destroy_queues;
spin_lock_irqsave(&npe_lock, flags);
- if (port->plat->open)
- if ((err = port->plat->open(port->id, dev,
- hss_hdlc_set_carrier)))
+ if (port->plat->open) {
+ err = port->plat->open(port->id, dev, hss_hdlc_set_carrier);
+ if (err)
goto err_unlock;
+ }
+
spin_unlock_irqrestore(&npe_lock, flags);
/* Populate queues with buffers, no failure after this point */
return 0;
}
-
static int hss_hdlc_attach(struct net_device *dev, unsigned short encoding,
unsigned short parity)
{
if (encoding != ENCODING_NRZ)
return -EINVAL;
- switch(parity) {
+ switch (parity) {
case PARITY_CRC16_PR1_CCITT:
port->hdlc_cfg = 0;
return 0;
for (b = 0; b < 0x400; b++) {
u64 c = (b + 1) * (u64)rate;
+
do_div(c, timer_freq - rate * a);
c--;
if (c >= 0xFFF) { /* 12-bit - no need to check more 'b's */
if (cmd != SIOCWANDEV)
return hdlc_ioctl(dev, ifr, cmd);
- switch(ifr->ifr_settings.type) {
+ switch (ifr->ifr_settings.type) {
case IF_GET_IFACE:
ifr->ifr_settings.type = IF_IFACE_V35;
if (ifr->ifr_settings.size < size) {
case IF_IFACE_SYNC_SERIAL:
case IF_IFACE_V35:
- if(!capable(CAP_NET_ADMIN))
+ if (!capable(CAP_NET_ADMIN))
return -EPERM;
if (copy_from_user(&new_line, line, size))
return -EFAULT;
return -EINVAL;
port->clock_type = clk; /* Update settings */
- if (clk == CLOCK_INT)
+ if (clk == CLOCK_INT) {
find_best_clock(port->plat->timer_freq,
new_line.clock_rate,
&port->clock_rate, &port->clock_reg);
- else {
+ } else {
port->clock_rate = 0;
port->clock_reg = CLK42X_SPEED_2048KHZ;
}
hdlc_device *hdlc;
int err;
- if ((port = kzalloc(sizeof(*port), GFP_KERNEL)) == NULL)
+ port = kzalloc(sizeof(*port), GFP_KERNEL);
+ if (!port)
return -ENOMEM;
- if ((port->npe = npe_request(0)) == NULL) {
+ port->npe = npe_request(0);
+ if (!port->npe) {
err = -ENODEV;
goto err_free;
}
- if ((port->netdev = dev = alloc_hdlcdev(port)) == NULL) {
+ dev = alloc_hdlcdev(port);
+ port->netdev = alloc_hdlcdev(port);
+ if (!port->netdev) {
err = -ENOMEM;
goto err_plat;
}
port->plat = pdev->dev.platform_data;
netif_napi_add(dev, &port->napi, hss_hdlc_poll, NAPI_WEIGHT);
- if ((err = register_hdlc_device(dev)))
+ err = register_hdlc_device(dev);
+ if (err)
goto err_free_netdev;
platform_set_drvdata(pdev, port);
*
* This is a "pseudo" network driver to allow LAPB over Ethernet.
*
- * This driver can use any ethernet destination address, and can be
+ * This driver can use any ethernet destination address, and can be
* limited to accept frames from one dedicated ethernet card only.
*
* History
static const u8 bcast_addr[6] = { 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF };
/* If this number is made larger, check that the temporary string buffer
- * in lapbeth_new_device is large enough to store the probe device name.*/
+ * in lapbeth_new_device is large enough to store the probe device name.
+ */
#define MAXLAPBDEV 100
struct lapbethdev {
/* ------------------------------------------------------------------------ */
-/*
- * Get the LAPB device for the ethernet device
+/* Get the LAPB device for the ethernet device
*/
static struct lapbethdev *lapbeth_get_x25_dev(struct net_device *dev)
{
struct lapbethdev *lapbeth;
list_for_each_entry_rcu(lapbeth, &lapbeth_devices, node, lockdep_rtnl_is_held()) {
- if (lapbeth->ethdev == dev)
+ if (lapbeth->ethdev == dev)
return lapbeth;
}
return NULL;
return processed;
}
-/*
- * Receive a LAPB frame via an ethernet interface.
+/* Receive a LAPB frame via an ethernet interface.
*/
-static int lapbeth_rcv(struct sk_buff *skb, struct net_device *dev, struct packet_type *ptype, struct net_device *orig_dev)
+static int lapbeth_rcv(struct sk_buff *skb, struct net_device *dev,
+ struct packet_type *ptype, struct net_device *orig_dev)
{
int len, err;
struct lapbethdev *lapbeth;
if (dev_net(dev) != &init_net)
goto drop;
- if ((skb = skb_share_check(skb, GFP_ATOMIC)) == NULL)
+ skb = skb_share_check(skb, GFP_ATOMIC);
+ if (!skb)
return NET_RX_DROP;
if (!pskb_may_pull(skb, 2))
skb_pull(skb, 2); /* Remove the length bytes */
skb_trim(skb, len); /* Set the length of the data */
- if ((err = lapb_data_received(lapbeth->axdev, skb)) != LAPB_OK) {
+ err = lapb_data_received(lapbeth->axdev, skb);
+ if (err != LAPB_OK) {
printk(KERN_DEBUG "lapbether: lapb_data_received err - %d\n", err);
goto drop_unlock;
}
return NET_RX_SUCCESS;
}
-/*
- * Send a LAPB frame via an ethernet interface
+/* Send a LAPB frame via an ethernet interface
*/
static netdev_tx_t lapbeth_xmit(struct sk_buff *skb,
- struct net_device *dev)
+ struct net_device *dev)
{
struct lapbethdev *lapbeth = netdev_priv(dev);
int err;
skb_pull(skb, 1);
- if ((err = lapb_data_request(dev, skb)) != LAPB_OK) {
+ err = lapb_data_request(dev, skb);
+ if (err != LAPB_OK) {
pr_err("lapb_data_request error - %d\n", err);
goto drop;
}
unsigned char *ptr;
struct sk_buff *skb = __dev_alloc_skb(1, GFP_ATOMIC | __GFP_NOMEMALLOC);
- if (!skb) {
- pr_err("out of memory\n");
+ if (!skb)
return;
- }
ptr = skb_put(skb, 1);
*ptr = X25_IFACE_CONNECT;
unsigned char *ptr;
struct sk_buff *skb = __dev_alloc_skb(1, GFP_ATOMIC | __GFP_NOMEMALLOC);
- if (!skb) {
- pr_err("out of memory\n");
+ if (!skb)
return;
- }
ptr = skb_put(skb, 1);
*ptr = X25_IFACE_DISCONNECT;
napi_schedule(&lapbeth->napi);
}
-/*
- * Set AX.25 callsign
+/* Set AX.25 callsign
*/
static int lapbeth_set_mac_address(struct net_device *dev, void *addr)
{
struct sockaddr *sa = addr;
+
memcpy(dev->dev_addr, sa->sa_data, dev->addr_len);
return 0;
}
-
static const struct lapb_register_struct lapbeth_callbacks = {
.connect_confirmation = lapbeth_connected,
.connect_indication = lapbeth_connected,
.data_transmit = lapbeth_data_transmit,
};
-/*
- * open/close a device
+/* open/close a device
*/
static int lapbeth_open(struct net_device *dev)
{
napi_enable(&lapbeth->napi);
- if ((err = lapb_register(dev, &lapbeth_callbacks)) != LAPB_OK) {
+ err = lapb_register(dev, &lapbeth_callbacks);
+ if (err != LAPB_OK) {
pr_err("lapb_register error: %d\n", err);
return -ENODEV;
}
lapbeth->up = false;
spin_unlock_bh(&lapbeth->up_lock);
- if ((err = lapb_unregister(dev)) != LAPB_OK)
+ err = lapb_unregister(dev);
+ if (err != LAPB_OK)
pr_err("lapb_unregister error: %d\n", err);
napi_disable(&lapbeth->napi);
dev->addr_len = 0;
}
-/*
- * Setup a new device.
+/* Setup a new device.
*/
static int lapbeth_new_device(struct net_device *dev)
{
goto out;
}
-/*
- * Free a lapb network device.
+/* Free a lapb network device.
*/
static void lapbeth_free_device(struct lapbethdev *lapbeth)
{
unregister_netdevice(lapbeth->axdev);
}
-/*
- * Handle device status changes.
+/* Handle device status changes.
*
* Called from notifier with RTNL held.
*/
switch (event) {
case NETDEV_UP:
/* New ethernet device -> new LAPB interface */
- if (lapbeth_get_x25_dev(dev) == NULL)
+ if (!lapbeth_get_x25_dev(dev))
lapbeth_new_device(dev);
break;
case NETDEV_GOING_DOWN:
/* ethernet device closes -> close LAPB interface */
lapbeth = lapbeth_get_x25_dev(dev);
- if (lapbeth)
+ if (lapbeth)
dev_close(lapbeth->axdev);
break;
case NETDEV_UNREGISTER:
*/
int lmc_probe(struct net_device * dev);
unsigned lmc_mii_readreg(lmc_softc_t * const sc, unsigned
- devaddr, unsigned regno);
+ devaddr, unsigned regno);
void lmc_mii_writereg(lmc_softc_t * const sc, unsigned devaddr,
unsigned regno, unsigned data);
void lmc_led_on(lmc_softc_t * const, u32);
#include <asm/io.h>
#include "hd64570.h"
-
-static const char* version = "SDL RISCom/N2 driver version: 1.15";
-static const char* devname = "RISCom/N2";
+static const char *version = "SDL RISCom/N2 driver version: 1.15";
+static const char *devname = "RISCom/N2";
#undef DEBUG_PKT
#define DEBUG_RINGS
#define PCR_ENWIN 4 /* Open window */
#define PCR_BUS16 8 /* 16-bit bus */
-
/* Memory Base Address Register */
#define N2_BAR 2
-
/* Page Scan Register */
#define N2_PSR 4
#define WIN16K 0x00
#define PSR_DMAEN 0x80
#define PSR_PAGEBITS 0x0F
-
/* Modem Control Reg */
#define N2_MCR 6
#define CLOCK_OUT_PORT1 0x80
#define DTR_PORT1 0x02
#define DTR_PORT0 0x01
-
typedef struct port_s {
struct net_device *dev;
struct card_s *card;
u8 rxs, txs, tmc; /* SCA registers */
u8 phy_node; /* physical port # - 0 or 1 */
u8 log_node; /* logical port # */
-}port_t;
-
-
+} port_t;
typedef struct card_s {
u8 __iomem *winbase; /* ISA window base address */
port_t ports[2];
struct card_s *next_card;
-}card_t;
-
+} card_t;
static card_t *first_card;
static card_t **new_card = &first_card;
-
#define sca_reg(reg, card) (0x8000 | (card)->io | \
((reg) & 0x0F) | (((reg) & 0xF0) << 6))
#define sca_in(reg, card) inb(sca_reg(reg, card))
#define get_port(card, port) ((card)->ports[port].valid ? \
&(card)->ports[port] : NULL)
-
static __inline__ u8 sca_get_page(card_t *card)
{
return inb(card->io + N2_PSR) & PSR_PAGEBITS;
}
-
static __inline__ void openwin(card_t *card, u8 page)
{
u8 psr = inb(card->io + N2_PSR);
+
outb((psr & ~PSR_PAGEBITS) | page, card->io + N2_PSR);
}
-
#include "hd64570.c"
-
static void n2_set_iface(port_t *port)
{
card_t *card = port->card;
u8 rxs = port->rxs & CLK_BRG_MASK;
u8 txs = port->txs & CLK_BRG_MASK;
- switch(port->settings.clock_type) {
+ switch (port->settings.clock_type) {
case CLOCK_INT:
mcr |= port->phy_node ? CLOCK_OUT_PORT1 : CLOCK_OUT_PORT0;
rxs |= CLK_BRG_RX; /* BRG output */
sca_set_port(port);
}
-
-
static int n2_open(struct net_device *dev)
{
port_t *port = dev_to_port(dev);
int io = port->card->io;
- u8 mcr = inb(io + N2_MCR) | (port->phy_node ? TX422_PORT1:TX422_PORT0);
+ u8 mcr = inb(io + N2_MCR) |
+ (port->phy_node ? TX422_PORT1 : TX422_PORT0);
int result;
result = hdlc_open(dev);
return 0;
}
-
-
static int n2_close(struct net_device *dev)
{
port_t *port = dev_to_port(dev);
int io = port->card->io;
- u8 mcr = inb(io+N2_MCR) | (port->phy_node ? TX422_PORT1 : TX422_PORT0);
+ u8 mcr = inb(io + N2_MCR) |
+ (port->phy_node ? TX422_PORT1 : TX422_PORT0);
sca_close(dev);
mcr |= port->phy_node ? DTR_PORT1 : DTR_PORT0; /* set DTR OFF */
return 0;
}
-
-
static int n2_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd)
{
const size_t size = sizeof(sync_serial_settings);
if (cmd != SIOCWANDEV)
return hdlc_ioctl(dev, ifr, cmd);
- switch(ifr->ifr_settings.type) {
+ switch (ifr->ifr_settings.type) {
case IF_GET_IFACE:
ifr->ifr_settings.type = IF_IFACE_SYNC_SERIAL;
if (ifr->ifr_settings.size < size) {
return 0;
case IF_IFACE_SYNC_SERIAL:
- if(!capable(CAP_NET_ADMIN))
+ if (!capable(CAP_NET_ADMIN))
return -EPERM;
if (copy_from_user(&new_line, line, size))
}
}
-
-
static void n2_destroy_card(card_t *card)
{
int cnt;
for (cnt = 0; cnt < 2; cnt++)
if (card->ports[cnt].card) {
struct net_device *dev = port_to_dev(&card->ports[cnt]);
+
unregister_hdlc_device(dev);
}
}
card = kzalloc(sizeof(card_t), GFP_KERNEL);
- if (card == NULL)
+ if (!card)
return -ENOBUFS;
card->ports[0].dev = alloc_hdlcdev(&card->ports[0]);
return 0;
}
-
-
static int __init n2_init(void)
{
- if (hw==NULL) {
+ if (!hw) {
#ifdef MODULE
pr_info("no card initialized\n");
#endif
if (*hw++ != ',')
break;
- while(1) {
+ while (1) {
if (*hw == '0' && !valid[0])
valid[0] = 1; /* Port 0 enabled */
else if (*hw == '1' && !valid[1])
if (*hw == '\x0')
return first_card ? 0 : -EINVAL;
- }while(*hw++ == ':');
+ } while (*hw++ == ':');
pr_err("invalid hardware parameters\n");
return first_card ? 0 : -EINVAL;
}
-
static void __exit n2_cleanup(void)
{
card_t *card = first_card;
while (card) {
card_t *ptr = card;
+
card = card->next_card;
n2_destroy_card(ptr);
}
}
-
module_init(n2_init);
module_exit(n2_cleanup);
#define MAX_TX_BUFFERS 10
static int pci_clock_freq = 33000000;
-static int use_crystal_clock = 0;
+static int use_crystal_clock;
static unsigned int CLOCK_BASE;
/* Masks to access the init_ctrl PLX register */
#define PC300_CHMEDIA_MASK(port) (0x00000020UL << ((port) * 3))
#define PC300_CTYPE_MASK (0x00000800UL)
-
enum { PC300_RSV = 1, PC300_X21, PC300_TE }; /* card types */
-/*
- * PLX PCI9050-1 local configuration and shared runtime registers.
+/* PLX PCI9050-1 local configuration and shared runtime registers.
* This structure can be used to access 9050 registers (memory mapped).
*/
typedef struct {
u32 cs_base[4]; /* 3C-48h : Chip Select Base Addrs */
u32 intr_ctrl_stat; /* 4Ch : Interrupt Control/Status */
u32 init_ctrl; /* 50h : EEPROM ctrl, Init Ctrl, etc */
-}plx9050;
-
-
+} plx9050;
typedef struct port_s {
struct napi_struct napi;
u16 txlast;
u8 rxs, txs, tmc; /* SCA registers */
u8 chan; /* physical port # - 0 or 1 */
-}port_t;
-
-
+} port_t;
typedef struct card_s {
int type; /* RSV, X21, etc. */
u8 irq; /* interrupt request level */
port_t ports[2];
-}card_t;
-
+} card_t;
#define get_port(card, port) ((port) < (card)->n_ports ? \
(&(card)->ports[port]) : (NULL))
#include "hd64572.c"
-
static void pc300_set_iface(port_t *port)
{
card_t *card = port->card;
- u32 __iomem * init_ctrl = &card->plxbase->init_ctrl;
+ u32 __iomem *init_ctrl = &card->plxbase->init_ctrl;
u16 msci = get_msci(port);
u8 rxs = port->rxs & CLK_BRG_MASK;
u8 txs = port->txs & CLK_BRG_MASK;
sca_out(EXS_TES1, (port->chan ? MSCI1_OFFSET : MSCI0_OFFSET) + EXS,
port->card);
- switch(port->settings.clock_type) {
+ switch (port->settings.clock_type) {
case CLOCK_INT:
rxs |= CLK_BRG; /* BRG output */
txs |= CLK_PIN_OUT | CLK_TX_RXCLK; /* RX clock */
}
}
-
-
static int pc300_open(struct net_device *dev)
{
port_t *port = dev_to_port(dev);
-
int result = hdlc_open(dev);
+
if (result)
return result;
return 0;
}
-
-
static int pc300_close(struct net_device *dev)
{
sca_close(dev);
return 0;
}
-
-
static int pc300_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd)
{
const size_t size = sizeof(sync_serial_settings);
if (copy_to_user(line, &port->settings, size))
return -EFAULT;
return 0;
-
}
if (port->card->type == PC300_X21 &&
return 0;
}
-
-
static void pc300_pci_remove_one(struct pci_dev *pdev)
{
int i;
}
card = kzalloc(sizeof(card_t), GFP_KERNEL);
- if (card == NULL) {
+ if (!card) {
pci_release_regions(pdev);
pci_disable_device(pdev);
return -ENOBUFS;
ramphys = pci_resource_start(pdev, 3) & PCI_BASE_ADDRESS_MEM_MASK;
card->rambase = pci_ioremap_bar(pdev, 3);
- if (card->plxbase == NULL ||
- card->scabase == NULL ||
- card->rambase == NULL) {
+ if (!card->plxbase || !card->scabase || !card->rambase) {
pr_err("ioremap() failed\n");
pc300_pci_remove_one(pdev);
return -ENOMEM;
else
card->n_ports = 2;
- for (i = 0; i < card->n_ports; i++)
- if (!(card->ports[i].netdev = alloc_hdlcdev(&card->ports[i]))) {
+ for (i = 0; i < card->n_ports; i++) {
+ card->ports[i].netdev = alloc_hdlcdev(&card->ports[i]);
+ if (!card->ports[i].netdev) {
pr_err("unable to allocate memory\n");
pc300_pci_remove_one(pdev);
return -ENOMEM;
}
+ }
/* Reset PLX */
p = &card->plxbase->init_ctrl;
port_t *port = &card->ports[i];
struct net_device *dev = port->netdev;
hdlc_device *hdlc = dev_to_hdlc(dev);
+
port->chan = i;
spin_lock_init(&port->lock);
return 0;
}
-
-
static const struct pci_device_id pc300_pci_tbl[] = {
{ PCI_VENDOR_ID_CYCLADES, PCI_DEVICE_ID_PC300_RX_1, PCI_ANY_ID,
PCI_ANY_ID, 0, 0, 0 },
{ 0, }
};
-
static struct pci_driver pc300_pci_driver = {
.name = "PC300",
.id_table = pc300_pci_tbl,
.remove = pc300_pci_remove_one,
};
-
static int __init pc300_init_module(void)
{
if (pci_clock_freq < 1000000 || pci_clock_freq > 80000000) {
return pci_register_driver(&pc300_pci_driver);
}
-
-
static void __exit pc300_cleanup_module(void)
{
pci_unregister_driver(&pc300_pci_driver);
static int pci_clock_freq = 33000000;
#define CLOCK_BASE pci_clock_freq
-/*
- * PLX PCI9052 local configuration and shared runtime registers.
+/* PLX PCI9052 local configuration and shared runtime registers.
* This structure can be used to access 9052 registers (memory mapped).
*/
typedef struct {
u32 cs_base[4]; /* 3C-48h : Chip Select Base Addrs */
u32 intr_ctrl_stat; /* 4Ch : Interrupt Control/Status */
u32 init_ctrl; /* 50h : EEPROM ctrl, Init Ctrl, etc */
-}plx9052;
-
-
+} plx9052;
typedef struct port_s {
struct napi_struct napi;
u16 txlast;
u8 rxs, txs, tmc; /* SCA registers */
u8 chan; /* physical port # - 0 or 1 */
-}port_t;
-
-
+} port_t;
typedef struct card_s {
u8 __iomem *rambase; /* buffer memory base (virtual) */
u8 irq; /* interrupt request level */
port_t ports[2];
-}card_t;
-
+} card_t;
-#define get_port(card, port) (&card->ports[port])
+#define get_port(card, port) (&(card)->ports[port])
#define sca_flush(card) (sca_in(IER0, card))
static inline void new_memcpy_toio(char __iomem *dest, char *src, int length)
{
int len;
+
do {
len = length > 256 ? 256 : length;
memcpy_toio(dest, src, len);
#include "hd64572.c"
-
static void pci200_set_iface(port_t *port)
{
card_t *card = port->card;
sca_out(EXS_TES1, (port->chan ? MSCI1_OFFSET : MSCI0_OFFSET) + EXS,
port->card);
- switch(port->settings.clock_type) {
+ switch (port->settings.clock_type) {
case CLOCK_INT:
rxs |= CLK_BRG; /* BRG output */
txs |= CLK_PIN_OUT | CLK_TX_RXCLK; /* RX clock */
sca_set_port(port);
}
-
-
static int pci200_open(struct net_device *dev)
{
port_t *port = dev_to_port(dev);
-
int result = hdlc_open(dev);
+
if (result)
return result;
return 0;
}
-
-
static int pci200_close(struct net_device *dev)
{
sca_close(dev);
return 0;
}
-
-
static int pci200_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd)
{
const size_t size = sizeof(sync_serial_settings);
if (cmd != SIOCWANDEV)
return hdlc_ioctl(dev, ifr, cmd);
- switch(ifr->ifr_settings.type) {
+ switch (ifr->ifr_settings.type) {
case IF_GET_IFACE:
ifr->ifr_settings.type = IF_IFACE_V35;
if (ifr->ifr_settings.size < size) {
}
}
-
-
static void pci200_pci_remove_one(struct pci_dev *pdev)
{
int i;
}
card = kzalloc(sizeof(card_t), GFP_KERNEL);
- if (card == NULL) {
+ if (!card) {
pci_release_regions(pdev);
pci_disable_device(pdev);
return -ENOBUFS;
return -EFAULT;
}
- plxphys = pci_resource_start(pdev,0) & PCI_BASE_ADDRESS_MEM_MASK;
+ plxphys = pci_resource_start(pdev, 0) & PCI_BASE_ADDRESS_MEM_MASK;
card->plxbase = ioremap(plxphys, PCI200SYN_PLX_SIZE);
- scaphys = pci_resource_start(pdev,2) & PCI_BASE_ADDRESS_MEM_MASK;
+ scaphys = pci_resource_start(pdev, 2) & PCI_BASE_ADDRESS_MEM_MASK;
card->scabase = ioremap(scaphys, PCI200SYN_SCA_SIZE);
- ramphys = pci_resource_start(pdev,3) & PCI_BASE_ADDRESS_MEM_MASK;
+ ramphys = pci_resource_start(pdev, 3) & PCI_BASE_ADDRESS_MEM_MASK;
card->rambase = pci_ioremap_bar(pdev, 3);
- if (card->plxbase == NULL ||
- card->scabase == NULL ||
- card->rambase == NULL) {
+ if (!card->plxbase || !card->scabase || !card->rambase) {
pr_err("ioremap() failed\n");
pci200_pci_remove_one(pdev);
return -EFAULT;
port_t *port = &card->ports[i];
struct net_device *dev = port->netdev;
hdlc_device *hdlc = dev_to_hdlc(dev);
+
port->chan = i;
spin_lock_init(&port->lock);
return 0;
}
-
-
static const struct pci_device_id pci200_pci_tbl[] = {
{ PCI_VENDOR_ID_PLX, PCI_DEVICE_ID_PLX_9050, PCI_VENDOR_ID_PLX,
PCI_DEVICE_ID_PLX_PCI200SYN, 0, 0, 0 },
{ 0, }
};
-
static struct pci_driver pci200_pci_driver = {
.name = "PCI200SYN",
.id_table = pci200_pci_tbl,
.remove = pci200_pci_remove_one,
};
-
static int __init pci200_init_module(void)
{
if (pci_clock_freq < 1000000 || pci_clock_freq > 80000000) {
return pci_register_driver(&pci200_pci_driver);
}
-
-
static void __exit pci200_cleanup_module(void)
{
pci_unregister_driver(&pci200_pci_driver);
// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * Sealevel Systems 4021 driver.
+/* Sealevel Systems 4021 driver.
*
* (c) Copyright 1999, 2001 Alan Cox
* (c) Copyright 2001 Red Hat Inc.
#include <asm/byteorder.h>
#include "z85230.h"
-
-struct slvl_device
-{
+struct slvl_device {
struct z8530_channel *chan;
int channel;
};
-
-struct slvl_board
-{
+struct slvl_board {
struct slvl_device dev[2];
struct z8530_dev board;
int iobase;
};
-/*
- * Network driver support routines
- */
+ /* Network driver support routines */
-static inline struct slvl_device* dev_to_chan(struct net_device *dev)
+static inline struct slvl_device *dev_to_chan(struct net_device *dev)
{
return (struct slvl_device *)dev_to_hdlc(dev)->priv;
}
-/*
- * Frame receive. Simple for our card as we do HDLC and there
+/* Frame receive. Simple for our card as we do HDLC and there
* is no funny garbage involved
*/
netif_rx(skb);
}
-/*
- * We've been placed in the UP state
- */
+ /* We've been placed in the UP state */
static int sealevel_open(struct net_device *d)
{
int err = -1;
int unit = slvl->channel;
- /*
- * Link layer up.
- */
+ /* Link layer up. */
switch (unit) {
- case 0:
- err = z8530_sync_dma_open(d, slvl->chan);
- break;
- case 1:
- err = z8530_sync_open(d, slvl->chan);
- break;
+ case 0:
+ err = z8530_sync_dma_open(d, slvl->chan);
+ break;
+ case 1:
+ err = z8530_sync_open(d, slvl->chan);
+ break;
}
if (err)
err = hdlc_open(d);
if (err) {
switch (unit) {
- case 0:
- z8530_sync_dma_close(d, slvl->chan);
- break;
- case 1:
- z8530_sync_close(d, slvl->chan);
- break;
+ case 0:
+ z8530_sync_dma_close(d, slvl->chan);
+ break;
+ case 1:
+ z8530_sync_close(d, slvl->chan);
+ break;
}
return err;
}
slvl->chan->rx_function = sealevel_input;
- /*
- * Go go go
- */
netif_start_queue(d);
return 0;
}
struct slvl_device *slvl = dev_to_chan(d);
int unit = slvl->channel;
- /*
- * Discard new frames
- */
+ /* Discard new frames */
slvl->chan->rx_function = z8530_null_rx;
netif_stop_queue(d);
switch (unit) {
- case 0:
- z8530_sync_dma_close(d, slvl->chan);
- break;
- case 1:
- z8530_sync_close(d, slvl->chan);
- break;
+ case 0:
+ z8530_sync_dma_close(d, slvl->chan);
+ break;
+ case 1:
+ z8530_sync_close(d, slvl->chan);
+ break;
}
return 0;
}
static int sealevel_ioctl(struct net_device *d, struct ifreq *ifr, int cmd)
{
/* struct slvl_device *slvl=dev_to_chan(d);
- z8530_ioctl(d,&slvl->sync.chanA,ifr,cmd) */
+ * z8530_ioctl(d,&slvl->sync.chanA,ifr,cmd)
+ */
return hdlc_ioctl(d, ifr, cmd);
}
-/*
- * Passed network frames, fire them downwind.
- */
+/* Passed network frames, fire them downwind. */
static netdev_tx_t sealevel_queue_xmit(struct sk_buff *skb,
- struct net_device *d)
+ struct net_device *d)
{
return z8530_queue_xmit(dev_to_chan(d)->chan, skb);
}
static int slvl_setup(struct slvl_device *sv, int iobase, int irq)
{
struct net_device *dev = alloc_hdlcdev(sv);
+
if (!dev)
return -1;
return 0;
}
-
-/*
- * Allocate and setup Sealevel board.
- */
+/* Allocate and setup Sealevel board. */
static __init struct slvl_board *slvl_init(int iobase, int irq,
int txdma, int rxdma, int slow)
struct z8530_dev *dev;
struct slvl_board *b;
- /*
- * Get the needed I/O space
- */
+ /* Get the needed I/O space */
if (!request_region(iobase, 8, "Sealevel 4021")) {
pr_warn("I/O 0x%X already in use\n", iobase);
dev = &b->board;
- /*
- * Stuff in the I/O addressing
- */
+ /* Stuff in the I/O addressing */
dev->active = 0;
b->iobase = iobase;
- /*
- * Select 8530 delays for the old board
- */
+ /* Select 8530 delays for the old board */
if (slow)
iobase |= Z8530_PORT_SLEEP;
dev->chanA.irqs = &z8530_nop;
dev->chanB.irqs = &z8530_nop;
- /*
- * Assert DTR enable DMA
- */
+ /* Assert DTR enable DMA */
outb(3 | (1 << 7), b->iobase + 4);
-
/* We want a fast IRQ for this device. Actually we'd like an even faster
- IRQ ;) - This is one driver RtLinux is made for */
+ * IRQ ;) - This is one driver RtLinux is made for
+ */
if (request_irq(irq, z8530_interrupt, 0,
"SeaLevel", dev) < 0) {
disable_irq(irq);
- /*
- * Begin normal initialise
- */
+ /* Begin normal initialise */
if (z8530_init(dev) != 0) {
pr_err("Z8530 series device not found\n");
z8530_channel_load(&dev->chanB, z8530_hdlc_kilostream_85230);
}
- /*
- * Now we can take the IRQ
- */
+ /* Now we can take the IRQ */
enable_irq(irq);
for (u = 0; u < 2; u++) {
struct net_device *d = b->dev[u].chan->netdevice;
+
unregister_hdlc_device(d);
free_netdev(d);
}
kfree(b);
}
-
-static int io=0x238;
-static int txdma=1;
-static int rxdma=3;
-static int irq=5;
-static bool slow=false;
+static int io = 0x238;
+static int txdma = 1;
+static int rxdma = 3;
+static int irq = 5;
+static bool slow;
module_param_hw(io, int, ioport, 0);
MODULE_PARM_DESC(io, "The I/O base of the Sealevel card");
#include "wanxl.h"
-static const char* version = "wanXL serial card driver version: 0.48";
+static const char *version = "wanXL serial card driver version: 0.48";
#define PLX_CTL_RESET 0x40000000 /* adapter reset */
/* MAILBOX #2 - DRAM SIZE */
#define MBX2_MEMSZ_MASK 0xFFFF0000 /* PUTS Memory Size Register mask */
-
struct port {
struct net_device *dev;
struct card *card;
spinlock_t lock; /* for wanxl_xmit */
- int node; /* physical port #0 - 3 */
+ int node; /* physical port #0 - 3 */
unsigned int clock_type;
int tx_in, tx_out;
struct sk_buff *tx_skbs[TX_BUFFERS];
};
-
struct card_status {
desc_t rx_descs[RX_QUEUE_LENGTH];
port_status_t port_status[4];
};
-
struct card {
int n_ports; /* 1, 2 or 4 ports */
u8 irq;
struct port ports[]; /* 1 - 4 port structures follow */
};
-
-
static inline struct port *dev_to_port(struct net_device *dev)
{
return (struct port *)dev_to_hdlc(dev)->priv;
}
-
static inline port_status_t *get_status(struct port *port)
{
return &port->card->status->port_status[port->node];
}
-
#ifdef DEBUG_PCI
static inline dma_addr_t pci_map_single_debug(struct pci_dev *pdev, void *ptr,
size_t size, int direction)
{
dma_addr_t addr = dma_map_single(&pdev->dev, ptr, size, direction);
+
if (addr + size > 0x100000000LL)
pr_crit("%s: pci_map_single() returned memory at 0x%llx!\n",
pci_name(pdev), (unsigned long long)addr);
#define pci_map_single pci_map_single_debug
#endif
-
/* Cable and/or personality module change interrupt service */
static inline void wanxl_cable_intr(struct port *port)
{
int valid = 1;
const char *cable, *pm, *dte = "", *dsr = "", *dcd = "";
- switch(value & 0x7) {
- case STATUS_CABLE_V35: cable = "V.35"; break;
- case STATUS_CABLE_X21: cable = "X.21"; break;
- case STATUS_CABLE_V24: cable = "V.24"; break;
- case STATUS_CABLE_EIA530: cable = "EIA530"; break;
- case STATUS_CABLE_NONE: cable = "no"; break;
- default: cable = "invalid";
+ switch (value & 0x7) {
+ case STATUS_CABLE_V35:
+ cable = "V.35";
+ break;
+ case STATUS_CABLE_X21:
+ cable = "X.21";
+ break;
+ case STATUS_CABLE_V24:
+ cable = "V.24";
+ break;
+ case STATUS_CABLE_EIA530:
+ cable = "EIA530";
+ break;
+ case STATUS_CABLE_NONE:
+ cable = "no";
+ break;
+ default:
+ cable = "invalid";
}
- switch((value >> STATUS_CABLE_PM_SHIFT) & 0x7) {
- case STATUS_CABLE_V35: pm = "V.35"; break;
- case STATUS_CABLE_X21: pm = "X.21"; break;
- case STATUS_CABLE_V24: pm = "V.24"; break;
- case STATUS_CABLE_EIA530: pm = "EIA530"; break;
- case STATUS_CABLE_NONE: pm = "no personality"; valid = 0; break;
- default: pm = "invalid personality"; valid = 0;
+ switch ((value >> STATUS_CABLE_PM_SHIFT) & 0x7) {
+ case STATUS_CABLE_V35:
+ pm = "V.35";
+ break;
+ case STATUS_CABLE_X21:
+ pm = "X.21";
+ break;
+ case STATUS_CABLE_V24:
+ pm = "V.24";
+ break;
+ case STATUS_CABLE_EIA530:
+ pm = "EIA530";
+ break;
+ case STATUS_CABLE_NONE:
+ pm = "no personality";
+ valid = 0;
+ break;
+ default:
+ pm = "invalid personality";
+ valid = 0;
}
if (valid) {
netif_carrier_off(port->dev);
}
-
-
/* Transmit complete interrupt service */
static inline void wanxl_tx_intr(struct port *port)
{
struct net_device *dev = port->dev;
+
while (1) {
- desc_t *desc = &get_status(port)->tx_descs[port->tx_in];
+ desc_t *desc = &get_status(port)->tx_descs[port->tx_in];
struct sk_buff *skb = port->tx_skbs[port->tx_in];
switch (desc->stat) {
dev->stats.tx_packets++;
dev->stats.tx_bytes += skb->len;
}
- desc->stat = PACKET_EMPTY; /* Free descriptor */
+ desc->stat = PACKET_EMPTY; /* Free descriptor */
dma_unmap_single(&port->card->pdev->dev, desc->address,
skb->len, DMA_TO_DEVICE);
dev_consume_skb_irq(skb);
- port->tx_in = (port->tx_in + 1) % TX_BUFFERS;
- }
+ port->tx_in = (port->tx_in + 1) % TX_BUFFERS;
+ }
}
-
-
/* Receive complete interrupt service */
static inline void wanxl_rx_intr(struct card *card)
{
desc_t *desc;
+
while (desc = &card->status->rx_descs[card->rx_in],
desc->stat != PACKET_EMPTY) {
- if ((desc->stat & PACKET_PORT_MASK) > card->n_ports)
+ if ((desc->stat & PACKET_PORT_MASK) > card->n_ports) {
pr_crit("%s: received packet for nonexistent port\n",
pci_name(card->pdev));
- else {
+ } else {
struct sk_buff *skb = card->rx_skbs[card->rx_in];
struct port *port = &card->ports[desc->stat &
PACKET_PORT_MASK];
struct net_device *dev = port->dev;
- if (!skb)
+ if (!skb) {
dev->stats.rx_dropped++;
- else {
+ } else {
dma_unmap_single(&card->pdev->dev,
desc->address, BUFFER_LENGTH,
DMA_FROM_DEVICE);
}
}
-
-
-static irqreturn_t wanxl_intr(int irq, void* dev_id)
+static irqreturn_t wanxl_intr(int irq, void *dev_id)
{
struct card *card = dev_id;
- int i;
- u32 stat;
- int handled = 0;
-
+ int i;
+ u32 stat;
+ int handled = 0;
- while((stat = readl(card->plx + PLX_DOORBELL_FROM_CARD)) != 0) {
- handled = 1;
+ while ((stat = readl(card->plx + PLX_DOORBELL_FROM_CARD)) != 0) {
+ handled = 1;
writel(stat, card->plx + PLX_DOORBELL_FROM_CARD);
- for (i = 0; i < card->n_ports; i++) {
+ for (i = 0; i < card->n_ports; i++) {
if (stat & (1 << (DOORBELL_FROM_CARD_TX_0 + i)))
wanxl_tx_intr(&card->ports[i]);
if (stat & (1 << (DOORBELL_FROM_CARD_CABLE_0 + i)))
}
if (stat & (1 << DOORBELL_FROM_CARD_RX))
wanxl_rx_intr(card);
- }
+ }
- return IRQ_RETVAL(handled);
+ return IRQ_RETVAL(handled);
}
-
-
static netdev_tx_t wanxl_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct port *port = dev_to_port(dev);
desc_t *desc;
- spin_lock(&port->lock);
+ spin_lock(&port->lock);
desc = &get_status(port)->tx_descs[port->tx_out];
- if (desc->stat != PACKET_EMPTY) {
- /* should never happen - previous xmit should stop queue */
+ if (desc->stat != PACKET_EMPTY) {
+ /* should never happen - previous xmit should stop queue */
#ifdef DEBUG_PKT
printk(KERN_DEBUG "%s: transmitter buffer full\n", dev->name);
#endif
return NETDEV_TX_OK;
}
-
-
static int wanxl_attach(struct net_device *dev, unsigned short encoding,
unsigned short parity)
{
return 0;
}
-
-
static int wanxl_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd)
{
const size_t size = sizeof(sync_serial_settings);
default:
return hdlc_ioctl(dev, ifr, cmd);
- }
+ }
}
-
-
static int wanxl_open(struct net_device *dev)
{
struct port *port = dev_to_port(dev);
netdev_err(dev, "port already open\n");
return -EIO;
}
- if ((i = hdlc_open(dev)) != 0)
+
+ i = hdlc_open(dev);
+ if (i)
return i;
port->tx_in = port->tx_out = 0;
return -EFAULT;
}
-
-
static int wanxl_close(struct net_device *dev)
{
struct port *port = dev_to_port(dev);
return 0;
}
-
-
static struct net_device_stats *wanxl_get_stats(struct net_device *dev)
{
struct port *port = dev_to_port(dev);
return &dev->stats;
}
-
-
static int wanxl_puts_command(struct card *card, u32 cmd)
{
unsigned long timeout = jiffies + 5 * HZ;
return 0;
schedule();
- }while (time_after(timeout, jiffies));
+ } while (time_after(timeout, jiffies));
return -1;
}
-
-
static void wanxl_reset(struct card *card)
{
u32 old_value = readl(card->plx + PLX_CONTROL) & ~PLX_CTL_RESET;
readl(card->plx + PLX_CONTROL); /* wait for posted write */
}
-
-
static void wanxl_pci_remove_one(struct pci_dev *pdev)
{
struct card *card = pci_get_drvdata(pdev);
kfree(card);
}
-
#include "wanxlfw.inc"
static const struct net_device_ops wanxl_ops = {
return i;
/* QUICC can only access first 256 MB of host RAM directly,
- but PLX9060 DMA does 32-bits for actual packet data transfers */
+ * but PLX9060 DMA does 32-bits for actual packet data transfers
+ */
/* FIXME when PCI/DMA subsystems are fixed.
- We set both dma_mask and consistent_dma_mask to 28 bits
- and pray pci_alloc_consistent() will use this info. It should
- work on most platforms */
+ * We set both dma_mask and consistent_dma_mask to 28 bits
+ * and pray pci_alloc_consistent() will use this info. It should
+ * work on most platforms
+ */
if (dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(28)) ||
dma_set_mask(&pdev->dev, DMA_BIT_MASK(28))) {
pr_err("No usable DMA configuration\n");
}
switch (pdev->device) {
- case PCI_DEVICE_ID_SBE_WANXL100: ports = 1; break;
- case PCI_DEVICE_ID_SBE_WANXL200: ports = 2; break;
- default: ports = 4;
+ case PCI_DEVICE_ID_SBE_WANXL100:
+ ports = 1;
+ break;
+ case PCI_DEVICE_ID_SBE_WANXL200:
+ ports = 2;
+ break;
+ default:
+ ports = 4;
}
card = kzalloc(struct_size(card, ports, ports), GFP_KERNEL);
- if (card == NULL) {
+ if (!card) {
pci_release_regions(pdev);
pci_disable_device(pdev);
return -ENOBUFS;
card->status = dma_alloc_coherent(&pdev->dev,
sizeof(struct card_status),
&card->status_address, GFP_KERNEL);
- if (card->status == NULL) {
+ if (!card->status) {
wanxl_pci_remove_one(pdev);
return -ENOBUFS;
}
#endif
/* FIXME when PCI/DMA subsystems are fixed.
- We set both dma_mask and consistent_dma_mask back to 32 bits
- to indicate the card can do 32-bit DMA addressing */
+ * We set both dma_mask and consistent_dma_mask back to 32 bits
+ * to indicate the card can do 32-bit DMA addressing
+ */
if (dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(32)) ||
dma_set_mask(&pdev->dev, DMA_BIT_MASK(32))) {
pr_err("No usable DMA configuration\n");
card->plx = ioremap(plx_phy, 0x70);
if (!card->plx) {
pr_err("ioremap() failed\n");
- wanxl_pci_remove_one(pdev);
+ wanxl_pci_remove_one(pdev);
return -EFAULT;
}
return -ENODEV;
}
- switch(stat & 0xC0) {
+ switch (stat & 0xC0) {
case 0x00: /* hmm - PUTS completed with non-zero code? */
case 0x80: /* PUTS still testing the hardware */
break;
/* set up on-board RAM mapping */
mem_phy = pci_resource_start(pdev, 2);
-
/* sanity check the board's reported memory size */
if (ramsize < BUFFERS_ADDR +
(TX_BUFFERS + RX_BUFFERS) * BUFFER_LENGTH * ports) {
for (i = 0; i < RX_QUEUE_LENGTH; i++) {
struct sk_buff *skb = dev_alloc_skb(BUFFER_LENGTH);
+
card->rx_skbs[i] = skb;
if (skb)
card->status->rx_descs[i].address =
mem = ioremap(mem_phy, PDM_OFFSET + sizeof(firmware));
if (!mem) {
pr_err("ioremap() failed\n");
- wanxl_pci_remove_one(pdev);
+ wanxl_pci_remove_one(pdev);
return -EFAULT;
}
for (i = 0; i < sizeof(firmware); i += 4)
- writel(ntohl(*(__be32*)(firmware + i)), mem + PDM_OFFSET + i);
+ writel(ntohl(*(__be32 *)(firmware + i)), mem + PDM_OFFSET + i);
for (i = 0; i < ports; i++)
writel(card->status_address +
timeout = jiffies + 5 * HZ;
do {
- if ((stat = readl(card->plx + PLX_MAILBOX_5)) != 0)
+ stat = readl(card->plx + PLX_MAILBOX_5);
+ if (stat)
break;
schedule();
- }while (time_after(timeout, jiffies));
+ } while (time_after(timeout, jiffies));
if (!stat) {
pr_warn("%s: timeout while initializing card firmware\n",
hdlc_device *hdlc;
struct port *port = &card->ports[i];
struct net_device *dev = alloc_hdlcdev(port);
+
if (!dev) {
pr_err("%s: unable to allocate memory\n",
pci_name(pdev));
{ 0, }
};
-
static struct pci_driver wanxl_pci_driver = {
.name = "wanXL",
.id_table = wanxl_pci_tbl,
.remove = wanxl_pci_remove_one,
};
-
static int __init wanxl_init_module(void)
{
#ifdef MODULE
pci_unregister_driver(&wanxl_pci_driver);
}
-
MODULE_AUTHOR("Krzysztof Halasa <khc@pm.waw.pl>");
MODULE_DESCRIPTION("SBE Inc. wanXL serial port driver");
MODULE_LICENSE("GPL v2");
// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- *
- * (c) Copyright 1998 Alan Cox <alan@lxorguk.ukuu.org.uk>
+/* (c) Copyright 1998 Alan Cox <alan@lxorguk.ukuu.org.uk>
* (c) Copyright 2000, 2001 Red Hat Inc
*
* Development of this driver was funded by Equiinet Ltd
* Asynchronous mode dropped for 2.2. For 2.5 we will attempt the
* unification of all the Z85x30 asynchronous drivers for real.
*
- * DMA now uses get_free_page as kmalloc buffers may span a 64K
+ * DMA now uses get_free_page as kmalloc buffers may span a 64K
* boundary.
*
* Modified for SMP safety and SMP locking by Alan Cox
#include "z85230.h"
-
/**
* z8530_read_port - Architecture specific interface function
* @p: port to read
*
* Provided port access methods. The Comtrol SV11 requires no delays
* between accesses and uses PC I/O. Some drivers may need a 5uS delay
- *
+ *
* In the longer term this should become an architecture specific
* section so that this can become a generic driver interface for all
* platforms. For now we only handle PC I/O ports with or without the
static inline int z8530_read_port(unsigned long p)
{
- u8 r=inb(Z8530_PORT_OF(p));
- if(p&Z8530_PORT_SLEEP) /* gcc should figure this out efficiently ! */
+ u8 r = inb(Z8530_PORT_OF(p));
+
+ if (p & Z8530_PORT_SLEEP) /* gcc should figure this out efficiently ! */
udelay(5);
return r;
}
* dread 5uS sanity delay.
*/
-
static inline void z8530_write_port(unsigned long p, u8 d)
{
- outb(d,Z8530_PORT_OF(p));
- if(p&Z8530_PORT_SLEEP)
+ outb(d, Z8530_PORT_OF(p));
+ if (p & Z8530_PORT_SLEEP)
udelay(5);
}
-
-
static void z8530_rx_done(struct z8530_channel *c);
static void z8530_tx_done(struct z8530_channel *c);
-
/**
- * read_zsreg - Read a register from a Z85230
+ * read_zsreg - Read a register from a Z85230
* @c: Z8530 channel to read from (2 per chip)
* @reg: Register to read
* FIXME: Use a spinlock.
- *
+ *
* Most of the Z8530 registers are indexed off the control registers.
* A read is done by writing to the control register and reading the
* register back. The caller must hold the lock
*/
-
+
static inline u8 read_zsreg(struct z8530_channel *c, u8 reg)
{
- if(reg)
+ if (reg)
z8530_write_port(c->ctrlio, reg);
return z8530_read_port(c->ctrlio);
}
static inline u8 read_zsdata(struct z8530_channel *c)
{
u8 r;
- r=z8530_read_port(c->dataio);
+
+ r = z8530_read_port(c->dataio);
return r;
}
*/
static inline void write_zsreg(struct z8530_channel *c, u8 reg, u8 val)
{
- if(reg)
+ if (reg)
z8530_write_port(c->ctrlio, reg);
z8530_write_port(c->ctrlio, val);
-
}
/**
*
* Write directly to the data register on the Z8530
*/
-
-
static inline void write_zsdata(struct z8530_channel *c, u8 val)
{
z8530_write_port(c->dataio, val);
}
-/*
- * Register loading parameters for a dead port
+/* Register loading parameters for a dead port
*/
-
-u8 z8530_dead_port[]=
-{
+
+u8 z8530_dead_port[] = {
255
};
-
EXPORT_SYMBOL(z8530_dead_port);
-/*
- * Register loading parameters for currently supported circuit types
+/* Register loading parameters for currently supported circuit types
*/
-
-/*
- * Data clocked by telco end. This is the correct data for the UK
+/* Data clocked by telco end. This is the correct data for the UK
* "kilostream" service, and most other similar services.
*/
-
-u8 z8530_hdlc_kilostream[]=
-{
- 4, SYNC_ENAB|SDLC|X1CLK,
+
+u8 z8530_hdlc_kilostream[] = {
+ 4, SYNC_ENAB | SDLC | X1CLK,
2, 0, /* No vector */
1, 0,
- 3, ENT_HM|RxCRC_ENAB|Rx8,
- 5, TxCRC_ENAB|RTS|TxENAB|Tx8|DTR,
+ 3, ENT_HM | RxCRC_ENAB | Rx8,
+ 5, TxCRC_ENAB | RTS | TxENAB | Tx8 | DTR,
9, 0, /* Disable interrupts */
6, 0xFF,
7, FLAG,
- 10, ABUNDER|NRZ|CRCPS,/*MARKIDLE ??*/
+ 10, ABUNDER | NRZ | CRCPS,/*MARKIDLE ??*/
11, TCTRxCP,
14, DISDPLL,
- 15, DCDIE|SYNCIE|CTSIE|TxUIE|BRKIE,
- 1, EXT_INT_ENAB|TxINT_ENAB|INT_ALL_Rx,
- 9, NV|MIE|NORESET,
+ 15, DCDIE | SYNCIE | CTSIE | TxUIE | BRKIE,
+ 1, EXT_INT_ENAB | TxINT_ENAB | INT_ALL_Rx,
+ 9, NV | MIE | NORESET,
255
};
-
EXPORT_SYMBOL(z8530_hdlc_kilostream);
-/*
- * As above but for enhanced chips.
+/* As above but for enhanced chips.
*/
-
-u8 z8530_hdlc_kilostream_85230[]=
-{
- 4, SYNC_ENAB|SDLC|X1CLK,
+
+u8 z8530_hdlc_kilostream_85230[] = {
+ 4, SYNC_ENAB | SDLC | X1CLK,
2, 0, /* No vector */
1, 0,
- 3, ENT_HM|RxCRC_ENAB|Rx8,
- 5, TxCRC_ENAB|RTS|TxENAB|Tx8|DTR,
+ 3, ENT_HM | RxCRC_ENAB | Rx8,
+ 5, TxCRC_ENAB | RTS | TxENAB | Tx8 | DTR,
9, 0, /* Disable interrupts */
6, 0xFF,
7, FLAG,
- 10, ABUNDER|NRZ|CRCPS, /* MARKIDLE?? */
+ 10, ABUNDER | NRZ | CRCPS, /* MARKIDLE?? */
11, TCTRxCP,
14, DISDPLL,
- 15, DCDIE|SYNCIE|CTSIE|TxUIE|BRKIE,
- 1, EXT_INT_ENAB|TxINT_ENAB|INT_ALL_Rx,
- 9, NV|MIE|NORESET,
+ 15, DCDIE | SYNCIE | CTSIE | TxUIE | BRKIE,
+ 1, EXT_INT_ENAB | TxINT_ENAB | INT_ALL_Rx,
+ 9, NV | MIE | NORESET,
23, 3, /* Extended mode AUTO TX and EOM*/
-
+
255
};
-
EXPORT_SYMBOL(z8530_hdlc_kilostream_85230);
/**
* z8530_flush_fifo - Flush on chip RX FIFO
* @c: Channel to flush
*
- * Flush the receive FIFO. There is no specific option for this, we
+ * Flush the receive FIFO. There is no specific option for this, we
* blindly read bytes and discard them. Reading when there is no data
* is harmless. The 8530 has a 4 byte FIFO, the 85230 has 8 bytes.
- *
+ *
* All locking is handled for the caller. On return data may still be
* present if it arrived during the flush.
*/
-
+
static void z8530_flush_fifo(struct z8530_channel *c)
{
read_zsreg(c, R1);
read_zsreg(c, R1);
read_zsreg(c, R1);
read_zsreg(c, R1);
- if(c->dev->type==Z85230)
- {
+ if (c->dev->type == Z85230) {
read_zsreg(c, R1);
read_zsreg(c, R1);
read_zsreg(c, R1);
read_zsreg(c, R1);
}
-}
+}
/**
* z8530_rtsdtr - Control the outgoing DTS/RTS line
* z8530_rx - Handle a PIO receive event
* @c: Z8530 channel to process
*
- * Receive handler for receiving in PIO mode. This is much like the
+ * Receive handler for receiving in PIO mode. This is much like the
* async one but not quite the same or as complex
*
* Note: Its intended that this handler can easily be separated from
* other code - this is true in the RT case too.
*
* We only cover the sync cases for this. If you want 2Mbit async
- * do it yourself but consider medical assistance first. This non DMA
- * synchronous mode is portable code. The DMA mode assumes PCI like
+ * do it yourself but consider medical assistance first. This non DMA
+ * synchronous mode is portable code. The DMA mode assumes PCI like
* ISA DMA
*
* Called with the device lock held
*/
-
+
static void z8530_rx(struct z8530_channel *c)
{
- u8 ch,stat;
+ u8 ch, stat;
- while(1)
- {
+ while (1) {
/* FIFO empty ? */
- if(!(read_zsreg(c, R0)&1))
+ if (!(read_zsreg(c, R0) & 1))
break;
- ch=read_zsdata(c);
- stat=read_zsreg(c, R1);
-
- /*
- * Overrun ?
+ ch = read_zsdata(c);
+ stat = read_zsreg(c, R1);
+
+ /* Overrun ?
*/
- if(c->count < c->max)
- {
- *c->dptr++=ch;
+ if (c->count < c->max) {
+ *c->dptr++ = ch;
c->count++;
}
- if(stat&END_FR)
- {
-
- /*
- * Error ?
+ if (stat & END_FR) {
+ /* Error ?
*/
- if(stat&(Rx_OVR|CRC_ERR))
- {
+ if (stat & (Rx_OVR | CRC_ERR)) {
/* Rewind the buffer and return */
- if(c->skb)
- c->dptr=c->skb->data;
- c->count=0;
- if(stat&Rx_OVR)
- {
+ if (c->skb)
+ c->dptr = c->skb->data;
+ c->count = 0;
+ if (stat & Rx_OVR) {
pr_warn("%s: overrun\n", c->dev->name);
c->rx_overrun++;
}
- if(stat&CRC_ERR)
- {
+ if (stat & CRC_ERR) {
c->rx_crc_err++;
/* printk("crc error\n"); */
}
/* Shove the frame upstream */
- }
- else
- {
- /*
- * Drop the lock for RX processing, or
- * there are deadlocks
- */
+ } else {
+ /* Drop the lock for RX processing, or
+ * there are deadlocks
+ */
z8530_rx_done(c);
write_zsctrl(c, RES_Rx_CRC);
}
}
}
- /*
- * Clear irq
+ /* Clear irq
*/
write_zsctrl(c, ERR_RES);
write_zsctrl(c, RES_H_IUS);
}
-
/**
* z8530_tx - Handle a PIO transmit event
* @c: Z8530 channel to process
* in as possible, its quite possible that we won't keep up with the
* data rate otherwise.
*/
-
+
static void z8530_tx(struct z8530_channel *c)
{
- while(c->txcount) {
+ while (c->txcount) {
/* FIFO full ? */
- if(!(read_zsreg(c, R0)&4))
+ if (!(read_zsreg(c, R0) & 4))
return;
c->txcount--;
- /*
- * Shovel out the byte
+ /* Shovel out the byte
*/
write_zsreg(c, R8, *c->tx_ptr++);
write_zsctrl(c, RES_H_IUS);
/* We are about to underflow */
- if(c->txcount==0)
- {
+ if (c->txcount == 0) {
write_zsctrl(c, RES_EOM_L);
- write_zsreg(c, R10, c->regs[10]&~ABUNDER);
+ write_zsreg(c, R10, c->regs[10] & ~ABUNDER);
}
}
-
- /*
- * End of frame TX - fire another one
+ /* End of frame TX - fire another one
*/
-
+
write_zsctrl(c, RES_Tx_P);
- z8530_tx_done(c);
+ z8530_tx_done(c);
write_zsctrl(c, RES_H_IUS);
}
z8530_tx_done(chan);
}
- if (altered & chan->dcdcheck)
- {
+ if (altered & chan->dcdcheck) {
if (status & chan->dcdcheck) {
pr_info("%s: DCD raised\n", chan->dev->name);
write_zsreg(chan, R3, chan->regs[3] | RxENABLE);
if (chan->netdevice)
netif_carrier_off(chan->netdevice);
}
-
}
write_zsctrl(chan, RES_EXT_INT);
write_zsctrl(chan, RES_H_IUS);
.tx = z8530_tx,
.status = z8530_status,
};
-
EXPORT_SYMBOL(z8530_sync);
/**
* events are handled by the DMA hardware. We get a kick here only if
* a frame ended.
*/
-
+
static void z8530_dma_rx(struct z8530_channel *chan)
{
- if(chan->rxdma_on)
- {
+ if (chan->rxdma_on) {
/* Special condition check only */
u8 status;
-
+
read_zsreg(chan, R7);
read_zsreg(chan, R6);
-
- status=read_zsreg(chan, R1);
-
- if(status&END_FR)
- {
+
+ status = read_zsreg(chan, R1);
+
+ if (status & END_FR)
z8530_rx_done(chan); /* Fire up the next one */
- }
+
write_zsctrl(chan, ERR_RES);
write_zsctrl(chan, RES_H_IUS);
- }
- else
- {
+ } else {
/* DMA is off right now, drain the slow way */
z8530_rx(chan);
- }
+ }
}
/**
* We have received an interrupt while doing DMA transmissions. It
* shouldn't happen. Scream loudly if it does.
*/
-
static void z8530_dma_tx(struct z8530_channel *chan)
{
- if(!chan->dma_tx)
- {
+ if (!chan->dma_tx) {
pr_warn("Hey who turned the DMA off?\n");
z8530_tx(chan);
return;
/**
* z8530_dma_status - Handle a DMA status exception
* @chan: Z8530 channel to process
- *
+ *
* A status event occurred on the Z8530. We receive these for two reasons
* when in DMA mode. Firstly if we finished a packet transfer we get one
* and kick the next packet out. Secondly we may see a DCD change.
*
*/
-
static void z8530_dma_status(struct z8530_channel *chan)
{
u8 status, altered;
- status=read_zsreg(chan, R0);
- altered=chan->status^status;
-
- chan->status=status;
+ status = read_zsreg(chan, R0);
+ altered = chan->status ^ status;
+ chan->status = status;
- if(chan->dma_tx)
- {
- if(status&TxEOM)
- {
+ if (chan->dma_tx) {
+ if (status & TxEOM) {
unsigned long flags;
-
- flags=claim_dma_lock();
+
+ flags = claim_dma_lock();
disable_dma(chan->txdma);
- clear_dma_ff(chan->txdma);
- chan->txdma_on=0;
+ clear_dma_ff(chan->txdma);
+ chan->txdma_on = 0;
release_dma_lock(flags);
z8530_tx_done(chan);
}
}
- if (altered & chan->dcdcheck)
- {
+ if (altered & chan->dcdcheck) {
if (status & chan->dcdcheck) {
pr_info("%s: DCD raised\n", chan->dev->name);
write_zsreg(chan, R3, chan->regs[3] | RxENABLE);
* (eg the MacII) we must clear the interrupt cause or die.
*/
-
static void z8530_rx_clear(struct z8530_channel *c)
{
- /*
- * Data and status bytes
+ /* Data and status bytes
*/
u8 stat;
read_zsdata(c);
- stat=read_zsreg(c, R1);
-
- if(stat&END_FR)
+ stat = read_zsreg(c, R1);
+
+ if (stat & END_FR)
write_zsctrl(c, RES_Rx_CRC);
- /*
- * Clear irq
+ /* Clear irq
*/
write_zsctrl(c, ERR_RES);
write_zsctrl(c, RES_H_IUS);
static void z8530_status_clear(struct z8530_channel *chan)
{
- u8 status=read_zsreg(chan, R0);
- if(status&TxEOM)
+ u8 status = read_zsreg(chan, R0);
+
+ if (status & TxEOM)
write_zsctrl(chan, ERR_RES);
write_zsctrl(chan, RES_EXT_INT);
write_zsctrl(chan, RES_H_IUS);
.tx = z8530_tx_clear,
.status = z8530_status_clear,
};
-
-
EXPORT_SYMBOL(z8530_nop);
/**
* z8530_interrupt - Handle an interrupt from a Z8530
- * @irq: Interrupt number
+ * @irq: Interrupt number
* @dev_id: The Z8530 device that is interrupting.
*
* A Z85[2]30 device has stuck its hand in the air for attention.
irqreturn_t z8530_interrupt(int irq, void *dev_id)
{
- struct z8530_dev *dev=dev_id;
+ struct z8530_dev *dev = dev_id;
u8 intr;
static volatile int locker=0;
- int work=0;
+ int work = 0;
struct z8530_irqhandler *irqs;
-
- if(locker)
- {
+
+ if (locker) {
pr_err("IRQ re-enter\n");
return IRQ_NONE;
}
- locker=1;
+ locker = 1;
spin_lock(&dev->lock);
- while(++work<5000)
- {
-
+ while (++work < 5000) {
intr = read_zsreg(&dev->chanA, R3);
- if(!(intr & (CHARxIP|CHATxIP|CHAEXT|CHBRxIP|CHBTxIP|CHBEXT)))
+ if (!(intr &
+ (CHARxIP | CHATxIP | CHAEXT | CHBRxIP | CHBTxIP | CHBEXT)))
break;
-
- /* This holds the IRQ status. On the 8530 you must read it from chan
- A even though it applies to the whole chip */
-
+
+ /* This holds the IRQ status. On the 8530 you must read it
+ * from chan A even though it applies to the whole chip
+ */
+
/* Now walk the chip and see what it is wanting - it may be
- an IRQ for someone else remember */
-
- irqs=dev->chanA.irqs;
+ * an IRQ for someone else remember
+ */
+
+ irqs = dev->chanA.irqs;
- if(intr & (CHARxIP|CHATxIP|CHAEXT))
- {
- if(intr&CHARxIP)
+ if (intr & (CHARxIP | CHATxIP | CHAEXT)) {
+ if (intr & CHARxIP)
irqs->rx(&dev->chanA);
- if(intr&CHATxIP)
+ if (intr & CHATxIP)
irqs->tx(&dev->chanA);
- if(intr&CHAEXT)
+ if (intr & CHAEXT)
irqs->status(&dev->chanA);
}
- irqs=dev->chanB.irqs;
+ irqs = dev->chanB.irqs;
- if(intr & (CHBRxIP|CHBTxIP|CHBEXT))
- {
- if(intr&CHBRxIP)
+ if (intr & (CHBRxIP | CHBTxIP | CHBEXT)) {
+ if (intr & CHBRxIP)
irqs->rx(&dev->chanB);
- if(intr&CHBTxIP)
+ if (intr & CHBTxIP)
irqs->tx(&dev->chanB);
- if(intr&CHBEXT)
+ if (intr & CHBEXT)
irqs->status(&dev->chanB);
}
}
spin_unlock(&dev->lock);
- if(work==5000)
+ if (work == 5000)
pr_err("%s: interrupt jammed - abort(0x%X)!\n",
dev->name, intr);
/* Ok all done */
- locker=0;
+ locker = 0;
return IRQ_HANDLED;
}
-
EXPORT_SYMBOL(z8530_interrupt);
-static const u8 reg_init[16]=
-{
- 0,0,0,0,
- 0,0,0,0,
- 0,0,0,0,
- 0x55,0,0,0
+static const u8 reg_init[16] = {
+ 0, 0, 0, 0,
+ 0, 0, 0, 0,
+ 0, 0, 0, 0,
+ 0x55, 0, 0, 0
};
-
/**
* z8530_sync_open - Open a Z8530 channel for PIO
* @dev: The network interface we are using
* Switch a Z8530 into synchronous mode without DMA assist. We
* raise the RTS/DTR and commence network operation.
*/
-
int z8530_sync_open(struct net_device *dev, struct z8530_channel *c)
{
unsigned long flags;
spin_lock_irqsave(c->lock, flags);
c->sync = 1;
- c->mtu = dev->mtu+64;
+ c->mtu = dev->mtu + 64;
c->count = 0;
c->skb = NULL;
c->skb2 = NULL;
/* This loads the double buffer up */
z8530_rx_done(c); /* Load the frame ring */
z8530_rx_done(c); /* Load the backup frame */
- z8530_rtsdtr(c,1);
+ z8530_rtsdtr(c, 1);
c->dma_tx = 0;
- c->regs[R1]|=TxINT_ENAB;
+ c->regs[R1] |= TxINT_ENAB;
write_zsreg(c, R1, c->regs[R1]);
- write_zsreg(c, R3, c->regs[R3]|RxENABLE);
+ write_zsreg(c, R3, c->regs[R3] | RxENABLE);
spin_unlock_irqrestore(c->lock, flags);
return 0;
}
-
-
EXPORT_SYMBOL(z8530_sync_open);
/**
* Close down a Z8530 interface and switch its interrupt handlers
* to discard future events.
*/
-
int z8530_sync_close(struct net_device *dev, struct z8530_channel *c)
{
u8 chk;
unsigned long flags;
-
+
spin_lock_irqsave(c->lock, flags);
c->irqs = &z8530_nop;
c->max = 0;
c->sync = 0;
-
- chk=read_zsreg(c,R0);
+
+ chk = read_zsreg(c, R0);
write_zsreg(c, R3, c->regs[R3]);
- z8530_rtsdtr(c,0);
+ z8530_rtsdtr(c, 0);
spin_unlock_irqrestore(c->lock, flags);
return 0;
}
-
EXPORT_SYMBOL(z8530_sync_close);
/**
* ISA DMA channels must be available for this to work. We assume ISA
* DMA driven I/O and PC limits on access.
*/
-
int z8530_sync_dma_open(struct net_device *dev, struct z8530_channel *c)
{
unsigned long cflags, dflags;
-
+
c->sync = 1;
- c->mtu = dev->mtu+64;
+ c->mtu = dev->mtu + 64;
c->count = 0;
c->skb = NULL;
c->skb2 = NULL;
- /*
- * Load the DMA interfaces up
+
+ /* Load the DMA interfaces up
*/
c->rxdma_on = 0;
c->txdma_on = 0;
-
- /*
- * Allocate the DMA flip buffers. Limit by page size.
+
+ /* Allocate the DMA flip buffers. Limit by page size.
* Everyone runs 1500 mtu or less on wan links so this
* should be fine.
*/
-
- if(c->mtu > PAGE_SIZE/2)
+
+ if (c->mtu > PAGE_SIZE / 2)
return -EMSGSIZE;
-
- c->rx_buf[0]=(void *)get_zeroed_page(GFP_KERNEL|GFP_DMA);
- if(c->rx_buf[0]==NULL)
+
+ c->rx_buf[0] = (void *)get_zeroed_page(GFP_KERNEL | GFP_DMA);
+ if (!c->rx_buf[0])
return -ENOBUFS;
- c->rx_buf[1]=c->rx_buf[0]+PAGE_SIZE/2;
-
- c->tx_dma_buf[0]=(void *)get_zeroed_page(GFP_KERNEL|GFP_DMA);
- if(c->tx_dma_buf[0]==NULL)
- {
+ c->rx_buf[1] = c->rx_buf[0] + PAGE_SIZE / 2;
+
+ c->tx_dma_buf[0] = (void *)get_zeroed_page(GFP_KERNEL | GFP_DMA);
+ if (!c->tx_dma_buf[0]) {
free_page((unsigned long)c->rx_buf[0]);
- c->rx_buf[0]=NULL;
+ c->rx_buf[0] = NULL;
return -ENOBUFS;
}
- c->tx_dma_buf[1]=c->tx_dma_buf[0]+PAGE_SIZE/2;
+ c->tx_dma_buf[1] = c->tx_dma_buf[0] + PAGE_SIZE / 2;
- c->tx_dma_used=0;
+ c->tx_dma_used = 0;
c->dma_tx = 1;
- c->dma_num=0;
- c->dma_ready=1;
-
- /*
- * Enable DMA control mode
+ c->dma_num = 0;
+ c->dma_ready = 1;
+
+ /* Enable DMA control mode
*/
spin_lock_irqsave(c->lock, cflags);
-
- /*
- * TX DMA via DIR/REQ
+
+ /* TX DMA via DIR/REQ
+ */
+
+ c->regs[R14] |= DTRREQ;
+ write_zsreg(c, R14, c->regs[R14]);
+
+ c->regs[R1] &= ~TxINT_ENAB;
+ write_zsreg(c, R1, c->regs[R1]);
+
+ /* RX DMA via W/Req
*/
-
- c->regs[R14]|= DTRREQ;
- write_zsreg(c, R14, c->regs[R14]);
- c->regs[R1]&= ~TxINT_ENAB;
+ c->regs[R1] |= WT_FN_RDYFN;
+ c->regs[R1] |= WT_RDY_RT;
+ c->regs[R1] |= INT_ERR_Rx;
+ c->regs[R1] &= ~TxINT_ENAB;
write_zsreg(c, R1, c->regs[R1]);
-
- /*
- * RX DMA via W/Req
- */
-
- c->regs[R1]|= WT_FN_RDYFN;
- c->regs[R1]|= WT_RDY_RT;
- c->regs[R1]|= INT_ERR_Rx;
- c->regs[R1]&= ~TxINT_ENAB;
+ c->regs[R1] |= WT_RDY_ENAB;
write_zsreg(c, R1, c->regs[R1]);
- c->regs[R1]|= WT_RDY_ENAB;
- write_zsreg(c, R1, c->regs[R1]);
-
- /*
- * DMA interrupts
+
+ /* DMA interrupts
+ */
+
+ /* Set up the DMA configuration
*/
-
- /*
- * Set up the DMA configuration
- */
-
- dflags=claim_dma_lock();
-
+
+ dflags = claim_dma_lock();
+
disable_dma(c->rxdma);
clear_dma_ff(c->rxdma);
- set_dma_mode(c->rxdma, DMA_MODE_READ|0x10);
+ set_dma_mode(c->rxdma, DMA_MODE_READ | 0x10);
set_dma_addr(c->rxdma, virt_to_bus(c->rx_buf[0]));
set_dma_count(c->rxdma, c->mtu);
enable_dma(c->rxdma);
clear_dma_ff(c->txdma);
set_dma_mode(c->txdma, DMA_MODE_WRITE);
disable_dma(c->txdma);
-
+
release_dma_lock(dflags);
-
- /*
- * Select the DMA interrupt handlers
+
+ /* Select the DMA interrupt handlers
*/
c->rxdma_on = 1;
c->txdma_on = 1;
c->tx_dma_used = 1;
-
+
c->irqs = &z8530_dma_sync;
- z8530_rtsdtr(c,1);
- write_zsreg(c, R3, c->regs[R3]|RxENABLE);
+ z8530_rtsdtr(c, 1);
+ write_zsreg(c, R3, c->regs[R3] | RxENABLE);
spin_unlock_irqrestore(c->lock, cflags);
-
+
return 0;
}
-
EXPORT_SYMBOL(z8530_sync_dma_open);
/**
* Shut down a DMA mode synchronous interface. Halt the DMA, and
* free the buffers.
*/
-
int z8530_sync_dma_close(struct net_device *dev, struct z8530_channel *c)
{
u8 chk;
unsigned long flags;
-
+
c->irqs = &z8530_nop;
c->max = 0;
c->sync = 0;
-
- /*
- * Disable the PC DMA channels
+
+ /* Disable the PC DMA channels
*/
-
- flags=claim_dma_lock();
+
+ flags = claim_dma_lock();
disable_dma(c->rxdma);
clear_dma_ff(c->rxdma);
-
+
c->rxdma_on = 0;
-
+
disable_dma(c->txdma);
clear_dma_ff(c->txdma);
release_dma_lock(flags);
-
+
c->txdma_on = 0;
c->tx_dma_used = 0;
spin_lock_irqsave(c->lock, flags);
- /*
- * Disable DMA control mode
+ /* Disable DMA control mode
*/
-
- c->regs[R1]&= ~WT_RDY_ENAB;
- write_zsreg(c, R1, c->regs[R1]);
- c->regs[R1]&= ~(WT_RDY_RT|WT_FN_RDYFN|INT_ERR_Rx);
- c->regs[R1]|= INT_ALL_Rx;
+
+ c->regs[R1] &= ~WT_RDY_ENAB;
write_zsreg(c, R1, c->regs[R1]);
- c->regs[R14]&= ~DTRREQ;
- write_zsreg(c, R14, c->regs[R14]);
-
- if(c->rx_buf[0])
- {
+ c->regs[R1] &= ~(WT_RDY_RT | WT_FN_RDYFN | INT_ERR_Rx);
+ c->regs[R1] |= INT_ALL_Rx;
+ write_zsreg(c, R1, c->regs[R1]);
+ c->regs[R14] &= ~DTRREQ;
+ write_zsreg(c, R14, c->regs[R14]);
+
+ if (c->rx_buf[0]) {
free_page((unsigned long)c->rx_buf[0]);
- c->rx_buf[0]=NULL;
+ c->rx_buf[0] = NULL;
}
- if(c->tx_dma_buf[0])
- {
+ if (c->tx_dma_buf[0]) {
free_page((unsigned long)c->tx_dma_buf[0]);
- c->tx_dma_buf[0]=NULL;
+ c->tx_dma_buf[0] = NULL;
}
- chk=read_zsreg(c,R0);
+ chk = read_zsreg(c, R0);
write_zsreg(c, R3, c->regs[R3]);
- z8530_rtsdtr(c,0);
+ z8530_rtsdtr(c, 0);
spin_unlock_irqrestore(c->lock, flags);
return 0;
}
-
EXPORT_SYMBOL(z8530_sync_dma_close);
/**
printk("Opening sync interface for TX-DMA\n");
c->sync = 1;
- c->mtu = dev->mtu+64;
+ c->mtu = dev->mtu + 64;
c->count = 0;
c->skb = NULL;
c->skb2 = NULL;
-
- /*
- * Allocate the DMA flip buffers. Limit by page size.
+
+ /* Allocate the DMA flip buffers. Limit by page size.
* Everyone runs 1500 mtu or less on wan links so this
* should be fine.
*/
-
- if(c->mtu > PAGE_SIZE/2)
+
+ if (c->mtu > PAGE_SIZE / 2)
return -EMSGSIZE;
-
- c->tx_dma_buf[0]=(void *)get_zeroed_page(GFP_KERNEL|GFP_DMA);
- if(c->tx_dma_buf[0]==NULL)
- return -ENOBUFS;
- c->tx_dma_buf[1] = c->tx_dma_buf[0] + PAGE_SIZE/2;
+ c->tx_dma_buf[0] = (void *)get_zeroed_page(GFP_KERNEL | GFP_DMA);
+ if (!c->tx_dma_buf[0])
+ return -ENOBUFS;
+ c->tx_dma_buf[1] = c->tx_dma_buf[0] + PAGE_SIZE / 2;
spin_lock_irqsave(c->lock, cflags);
- /*
- * Load the PIO receive ring
+ /* Load the PIO receive ring
*/
z8530_rx_done(c);
z8530_rx_done(c);
- /*
- * Load the DMA interfaces up
+ /* Load the DMA interfaces up
*/
c->rxdma_on = 0;
c->txdma_on = 0;
-
- c->tx_dma_used=0;
- c->dma_num=0;
- c->dma_ready=1;
+
+ c->tx_dma_used = 0;
+ c->dma_num = 0;
+ c->dma_ready = 1;
c->dma_tx = 1;
- /*
- * Enable DMA control mode
+ /* Enable DMA control mode
*/
- /*
- * TX DMA via DIR/REQ
- */
- c->regs[R14]|= DTRREQ;
- write_zsreg(c, R14, c->regs[R14]);
-
- c->regs[R1]&= ~TxINT_ENAB;
+ /* TX DMA via DIR/REQ
+ */
+ c->regs[R14] |= DTRREQ;
+ write_zsreg(c, R14, c->regs[R14]);
+
+ c->regs[R1] &= ~TxINT_ENAB;
write_zsreg(c, R1, c->regs[R1]);
-
- /*
- * Set up the DMA configuration
- */
-
+
+ /* Set up the DMA configuration
+ */
+
dflags = claim_dma_lock();
disable_dma(c->txdma);
disable_dma(c->txdma);
release_dma_lock(dflags);
-
- /*
- * Select the DMA interrupt handlers
+
+ /* Select the DMA interrupt handlers
*/
c->rxdma_on = 0;
c->txdma_on = 1;
c->tx_dma_used = 1;
-
+
c->irqs = &z8530_txdma_sync;
- z8530_rtsdtr(c,1);
- write_zsreg(c, R3, c->regs[R3]|RxENABLE);
+ z8530_rtsdtr(c, 1);
+ write_zsreg(c, R3, c->regs[R3] | RxENABLE);
spin_unlock_irqrestore(c->lock, cflags);
-
+
return 0;
}
-
EXPORT_SYMBOL(z8530_sync_txdma_open);
/**
* @dev: Network device to detach
* @c: Z8530 channel to move into discard mode
*
- * Shut down a DMA/PIO split mode synchronous interface. Halt the DMA,
+ * Shut down a DMA/PIO split mode synchronous interface. Halt the DMA,
* and free the buffers.
*/
unsigned long dflags, cflags;
u8 chk;
-
spin_lock_irqsave(c->lock, cflags);
-
+
c->irqs = &z8530_nop;
c->max = 0;
c->sync = 0;
-
- /*
- * Disable the PC DMA channels
+
+ /* Disable the PC DMA channels
*/
-
+
dflags = claim_dma_lock();
disable_dma(c->txdma);
release_dma_lock(dflags);
- /*
- * Disable DMA control mode
+ /* Disable DMA control mode
*/
-
- c->regs[R1]&= ~WT_RDY_ENAB;
- write_zsreg(c, R1, c->regs[R1]);
- c->regs[R1]&= ~(WT_RDY_RT|WT_FN_RDYFN|INT_ERR_Rx);
- c->regs[R1]|= INT_ALL_Rx;
+
+ c->regs[R1] &= ~WT_RDY_ENAB;
+ write_zsreg(c, R1, c->regs[R1]);
+ c->regs[R1] &= ~(WT_RDY_RT | WT_FN_RDYFN | INT_ERR_Rx);
+ c->regs[R1] |= INT_ALL_Rx;
write_zsreg(c, R1, c->regs[R1]);
- c->regs[R14]&= ~DTRREQ;
- write_zsreg(c, R14, c->regs[R14]);
-
- if(c->tx_dma_buf[0])
- {
+ c->regs[R14] &= ~DTRREQ;
+ write_zsreg(c, R14, c->regs[R14]);
+
+ if (c->tx_dma_buf[0]) {
free_page((unsigned long)c->tx_dma_buf[0]);
- c->tx_dma_buf[0]=NULL;
+ c->tx_dma_buf[0] = NULL;
}
- chk=read_zsreg(c,R0);
+ chk = read_zsreg(c, R0);
write_zsreg(c, R3, c->regs[R3]);
- z8530_rtsdtr(c,0);
+ z8530_rtsdtr(c, 0);
spin_unlock_irqrestore(c->lock, cflags);
return 0;
}
-
-
EXPORT_SYMBOL(z8530_sync_txdma_close);
-
-/*
- * Name strings for Z8530 chips. SGI claim to have a 130, Zilog deny
+/* Name strings for Z8530 chips. SGI claim to have a 130, Zilog deny
* it exists...
*/
-
-static const char *z8530_type_name[]={
+static const char * const z8530_type_name[] = {
"Z8530",
"Z85C30",
"Z85230"
void z8530_describe(struct z8530_dev *dev, char *mapping, unsigned long io)
{
pr_info("%s: %s found at %s 0x%lX, IRQ %d\n",
- dev->name,
+ dev->name,
z8530_type_name[dev->type],
mapping,
Z8530_PORT_OF(io),
dev->irq);
}
-
EXPORT_SYMBOL(z8530_describe);
-/*
- * Locked operation part of the z8530 init code
+/* Locked operation part of the z8530 init code
*/
-
static inline int do_z8530_init(struct z8530_dev *dev)
{
/* NOP the interrupt handlers first - we might get a
- floating IRQ transition when we reset the chip */
- dev->chanA.irqs=&z8530_nop;
- dev->chanB.irqs=&z8530_nop;
- dev->chanA.dcdcheck=DCD;
- dev->chanB.dcdcheck=DCD;
+ * floating IRQ transition when we reset the chip
+ */
+ dev->chanA.irqs = &z8530_nop;
+ dev->chanB.irqs = &z8530_nop;
+ dev->chanA.dcdcheck = DCD;
+ dev->chanB.dcdcheck = DCD;
/* Reset the chip */
write_zsreg(&dev->chanA, R9, 0xC0);
udelay(200);
/* Now check its valid */
write_zsreg(&dev->chanA, R12, 0xAA);
- if(read_zsreg(&dev->chanA, R12)!=0xAA)
+ if (read_zsreg(&dev->chanA, R12) != 0xAA)
return -ENODEV;
write_zsreg(&dev->chanA, R12, 0x55);
- if(read_zsreg(&dev->chanA, R12)!=0x55)
+ if (read_zsreg(&dev->chanA, R12) != 0x55)
return -ENODEV;
-
- dev->type=Z8530;
-
- /*
- * See the application note.
+
+ dev->type = Z8530;
+
+ /* See the application note.
*/
-
+
write_zsreg(&dev->chanA, R15, 0x01);
-
- /*
- * If we can set the low bit of R15 then
+
+ /* If we can set the low bit of R15 then
* the chip is enhanced.
*/
-
- if(read_zsreg(&dev->chanA, R15)==0x01)
- {
+
+ if (read_zsreg(&dev->chanA, R15) == 0x01) {
/* This C30 versus 230 detect is from Klaus Kudielka's dmascc */
/* Put a char in the fifo */
write_zsreg(&dev->chanA, R8, 0);
- if(read_zsreg(&dev->chanA, R0)&Tx_BUF_EMP)
+ if (read_zsreg(&dev->chanA, R0) & Tx_BUF_EMP)
dev->type = Z85230; /* Has a FIFO */
else
dev->type = Z85C30; /* Z85C30, 1 byte FIFO */
}
-
- /*
- * The code assumes R7' and friends are
+
+ /* The code assumes R7' and friends are
* off. Use write_zsext() for these and keep
* this bit clear.
*/
-
+
write_zsreg(&dev->chanA, R15, 0);
-
- /*
- * At this point it looks like the chip is behaving
+
+ /* At this point it looks like the chip is behaving
*/
-
+
memcpy(dev->chanA.regs, reg_init, 16);
- memcpy(dev->chanB.regs, reg_init ,16);
-
+ memcpy(dev->chanB.regs, reg_init, 16);
+
return 0;
}
return ret;
}
-
-
EXPORT_SYMBOL(z8530_init);
/**
* z8530_shutdown - Shutdown a Z8530 device
* @dev: The Z8530 chip to shutdown
*
- * We set the interrupt handlers to silence any interrupts. We then
+ * We set the interrupt handlers to silence any interrupts. We then
* reset the chip and wait 100uS to be sure the reset completed. Just
* in case the caller then tries to do stuff.
*
* This is called without the lock held
*/
-
int z8530_shutdown(struct z8530_dev *dev)
{
unsigned long flags;
/* Reset the chip */
spin_lock_irqsave(&dev->lock, flags);
- dev->chanA.irqs=&z8530_nop;
- dev->chanB.irqs=&z8530_nop;
+ dev->chanA.irqs = &z8530_nop;
+ dev->chanB.irqs = &z8530_nop;
write_zsreg(&dev->chanA, R9, 0xC0);
/* We must lock the udelay, the chip is offlimits here */
udelay(100);
spin_unlock_irqrestore(&dev->lock, flags);
return 0;
}
-
EXPORT_SYMBOL(z8530_shutdown);
/**
* @rtable: table of register, value pairs
* FIXME: ioctl to allow user uploaded tables
*
- * Load a Z8530 channel up from the system data. We use +16 to
+ * Load a Z8530 channel up from the system data. We use +16 to
* indicate the "prime" registers. The value 255 terminates the
* table.
*/
spin_lock_irqsave(c->lock, flags);
- while(*rtable!=255)
- {
- int reg=*rtable++;
- if(reg>0x0F)
- write_zsreg(c, R15, c->regs[15]|1);
- write_zsreg(c, reg&0x0F, *rtable);
- if(reg>0x0F)
- write_zsreg(c, R15, c->regs[15]&~1);
- c->regs[reg]=*rtable++;
+ while (*rtable != 255) {
+ int reg = *rtable++;
+
+ if (reg > 0x0F)
+ write_zsreg(c, R15, c->regs[15] | 1);
+ write_zsreg(c, reg & 0x0F, *rtable);
+ if (reg > 0x0F)
+ write_zsreg(c, R15, c->regs[15] & ~1);
+ c->regs[reg] = *rtable++;
}
- c->rx_function=z8530_null_rx;
- c->skb=NULL;
- c->tx_skb=NULL;
- c->tx_next_skb=NULL;
- c->mtu=1500;
- c->max=0;
- c->count=0;
- c->status=read_zsreg(c, R0);
- c->sync=1;
- write_zsreg(c, R3, c->regs[R3]|RxENABLE);
+ c->rx_function = z8530_null_rx;
+ c->skb = NULL;
+ c->tx_skb = NULL;
+ c->tx_next_skb = NULL;
+ c->mtu = 1500;
+ c->max = 0;
+ c->count = 0;
+ c->status = read_zsreg(c, R0);
+ c->sync = 1;
+ write_zsreg(c, R3, c->regs[R3] | RxENABLE);
spin_unlock_irqrestore(c->lock, flags);
return 0;
}
-
EXPORT_SYMBOL(z8530_channel_load);
-
/**
* z8530_tx_begin - Begin packet transmission
* @c: The Z8530 channel to kick
*
* This is the speed sensitive side of transmission. If we are called
* and no buffer is being transmitted we commence the next buffer. If
- * nothing is queued we idle the sync.
+ * nothing is queued we idle the sync.
*
* Note: We are handling this code path in the interrupt path, keep it
* fast or bad things will happen.
static void z8530_tx_begin(struct z8530_channel *c)
{
unsigned long flags;
- if(c->tx_skb)
+
+ if (c->tx_skb)
return;
-
- c->tx_skb=c->tx_next_skb;
- c->tx_next_skb=NULL;
- c->tx_ptr=c->tx_next_ptr;
-
- if(c->tx_skb==NULL)
- {
+
+ c->tx_skb = c->tx_next_skb;
+ c->tx_next_skb = NULL;
+ c->tx_ptr = c->tx_next_ptr;
+
+ if (!c->tx_skb) {
/* Idle on */
- if(c->dma_tx)
- {
- flags=claim_dma_lock();
+ if (c->dma_tx) {
+ flags = claim_dma_lock();
disable_dma(c->txdma);
- /*
- * Check if we crapped out.
+ /* Check if we crapped out.
*/
- if (get_dma_residue(c->txdma))
- {
+ if (get_dma_residue(c->txdma)) {
c->netdevice->stats.tx_dropped++;
c->netdevice->stats.tx_fifo_errors++;
}
release_dma_lock(flags);
}
- c->txcount=0;
- }
- else
- {
- c->txcount=c->tx_skb->len;
-
-
- if(c->dma_tx)
- {
- /*
- * FIXME. DMA is broken for the original 8530,
+ c->txcount = 0;
+ } else {
+ c->txcount = c->tx_skb->len;
+
+ if (c->dma_tx) {
+ /* FIXME. DMA is broken for the original 8530,
* on the older parts we need to set a flag and
* wait for a further TX interrupt to fire this
- * stage off
+ * stage off
*/
-
- flags=claim_dma_lock();
+
+ flags = claim_dma_lock();
disable_dma(c->txdma);
- /*
- * These two are needed by the 8530/85C30
+ /* These two are needed by the 8530/85C30
* and must be issued when idling.
*/
-
- if(c->dev->type!=Z85230)
- {
+ if (c->dev->type != Z85230) {
write_zsctrl(c, RES_Tx_CRC);
write_zsctrl(c, RES_EOM_L);
- }
- write_zsreg(c, R10, c->regs[10]&~ABUNDER);
+ }
+ write_zsreg(c, R10, c->regs[10] & ~ABUNDER);
clear_dma_ff(c->txdma);
set_dma_addr(c->txdma, virt_to_bus(c->tx_ptr));
set_dma_count(c->txdma, c->txcount);
enable_dma(c->txdma);
release_dma_lock(flags);
write_zsctrl(c, RES_EOM_L);
- write_zsreg(c, R5, c->regs[R5]|TxENAB);
- }
- else
- {
-
+ write_zsreg(c, R5, c->regs[R5] | TxENAB);
+ } else {
/* ABUNDER off */
write_zsreg(c, R10, c->regs[10]);
write_zsctrl(c, RES_Tx_CRC);
-
- while(c->txcount && (read_zsreg(c,R0)&Tx_BUF_EMP))
- {
+
+ while (c->txcount && (read_zsreg(c, R0) & Tx_BUF_EMP)) {
write_zsreg(c, R8, *c->tx_ptr++);
c->txcount--;
}
-
}
}
- /*
- * Since we emptied tx_skb we can ask for more
+ /* Since we emptied tx_skb we can ask for more
*/
netif_wake_queue(c->netdevice);
}
struct sk_buff *skb;
/* Actually this can happen.*/
- if (c->tx_skb == NULL)
+ if (!c->tx_skb)
return;
skb = c->tx_skb;
* We point the receive handler at this function when idle. Instead
* of processing the frames we get to throw them away.
*/
-
void z8530_null_rx(struct z8530_channel *c, struct sk_buff *skb)
{
dev_kfree_skb_any(skb);
}
-
EXPORT_SYMBOL(z8530_null_rx);
/**
*
* Called with the lock held
*/
-
static void z8530_rx_done(struct z8530_channel *c)
{
struct sk_buff *skb;
int ct;
-
- /*
- * Is our receive engine in DMA mode
+
+ /* Is our receive engine in DMA mode
*/
-
- if(c->rxdma_on)
- {
- /*
- * Save the ready state and the buffer currently
+ if (c->rxdma_on) {
+ /* Save the ready state and the buffer currently
* being used as the DMA target
*/
-
- int ready=c->dma_ready;
- unsigned char *rxb=c->rx_buf[c->dma_num];
+ int ready = c->dma_ready;
+ unsigned char *rxb = c->rx_buf[c->dma_num];
unsigned long flags;
-
- /*
- * Complete this DMA. Necessary to find the length
- */
-
- flags=claim_dma_lock();
-
+
+ /* Complete this DMA. Necessary to find the length
+ */
+ flags = claim_dma_lock();
+
disable_dma(c->rxdma);
clear_dma_ff(c->rxdma);
- c->rxdma_on=0;
- ct=c->mtu-get_dma_residue(c->rxdma);
- if(ct<0)
- ct=2; /* Shit happens.. */
- c->dma_ready=0;
-
- /*
- * Normal case: the other slot is free, start the next DMA
+ c->rxdma_on = 0;
+ ct = c->mtu - get_dma_residue(c->rxdma);
+ if (ct < 0)
+ ct = 2; /* Shit happens.. */
+ c->dma_ready = 0;
+
+ /* Normal case: the other slot is free, start the next DMA
* into it immediately.
*/
-
- if(ready)
- {
- c->dma_num^=1;
- set_dma_mode(c->rxdma, DMA_MODE_READ|0x10);
+
+ if (ready) {
+ c->dma_num ^= 1;
+ set_dma_mode(c->rxdma, DMA_MODE_READ | 0x10);
set_dma_addr(c->rxdma, virt_to_bus(c->rx_buf[c->dma_num]));
set_dma_count(c->rxdma, c->mtu);
c->rxdma_on = 1;
enable_dma(c->rxdma);
- /* Stop any frames that we missed the head of
- from passing */
+ /* Stop any frames that we missed the head of
+ * from passing
+ */
write_zsreg(c, R0, RES_Rx_CRC);
- }
- else
+ } else {
/* Can't occur as we dont reenable the DMA irq until
- after the flip is done */
+ * after the flip is done
+ */
netdev_warn(c->netdevice, "DMA flip overrun!\n");
+ }
release_dma_lock(flags);
- /*
- * Shove the old buffer into an sk_buff. We can't DMA
+ /* Shove the old buffer into an sk_buff. We can't DMA
* directly into one on a PC - it might be above the 16Mb
* boundary. Optimisation - we could check to see if we
* can avoid the copy. Optimisation 2 - make the memcpy
*/
skb = dev_alloc_skb(ct);
- if (skb == NULL) {
+ if (!skb) {
c->netdevice->stats.rx_dropped++;
netdev_warn(c->netdevice, "Memory squeeze\n");
} else {
RT_LOCK;
skb = c->skb;
- /*
- * The game we play for non DMA is similar. We want to
+ /* The game we play for non DMA is similar. We want to
* get the controller set up for the next packet as fast
* as possible. We potentially only have one byte + the
* fifo length for this. Thus we want to flip to the new
* sync IRQ for the RT_LOCK area.
*
*/
- ct=c->count;
+ ct = c->count;
c->skb = c->skb2;
c->count = 0;
RT_UNLOCK;
c->skb2 = dev_alloc_skb(c->mtu);
- if (c->skb2 == NULL)
- netdev_warn(c->netdevice, "memory squeeze\n");
- else
+ if (c->skb2)
skb_put(c->skb2, c->mtu);
+
c->netdevice->stats.rx_packets++;
c->netdevice->stats.rx_bytes += ct;
}
- /*
- * If we received a frame we must now process it.
+ /* If we received a frame we must now process it.
*/
if (skb) {
skb_trim(skb, ct);
static inline int spans_boundary(struct sk_buff *skb)
{
- unsigned long a=(unsigned long)skb->data;
- a^=(a+skb->len);
- if(a&0x00010000) /* If the 64K bit is different.. */
+ unsigned long a = (unsigned long)skb->data;
+
+ a ^= (a + skb->len);
+ if (a & 0x00010000) /* If the 64K bit is different.. */
return 1;
return 0;
}
* @skb: The packet to kick down the channel
*
* Queue a packet for transmission. Because we have rather
- * hard to hit interrupt latencies for the Z85230 per packet
+ * hard to hit interrupt latencies for the Z85230 per packet
* even in DMA mode we do the flip to DMA buffer if needed here
* not in the IRQ.
*
- * Called from the network code. The lock is not held at this
+ * Called from the network code. The lock is not held at this
* point.
*/
-
netdev_tx_t z8530_queue_xmit(struct z8530_channel *c, struct sk_buff *skb)
{
unsigned long flags;
-
+
netif_stop_queue(c->netdevice);
- if(c->tx_next_skb)
+ if (c->tx_next_skb)
return NETDEV_TX_BUSY;
-
/* PC SPECIFIC - DMA limits */
-
- /*
- * If we will DMA the transmit and its gone over the ISA bus
+ /* If we will DMA the transmit and its gone over the ISA bus
* limit, then copy to the flip buffer
*/
-
- if(c->dma_tx && ((unsigned long)(virt_to_bus(skb->data+skb->len))>=16*1024*1024 || spans_boundary(skb)))
- {
- /*
- * Send the flip buffer, and flip the flippy bit.
+
+ if (c->dma_tx &&
+ ((unsigned long)(virt_to_bus(skb->data + skb->len)) >=
+ 16 * 1024 * 1024 || spans_boundary(skb))) {
+ /* Send the flip buffer, and flip the flippy bit.
* We don't care which is used when just so long as
* we never use the same buffer twice in a row. Since
* only one buffer can be going out at a time the other
* has to be safe.
*/
- c->tx_next_ptr=c->tx_dma_buf[c->tx_dma_used];
- c->tx_dma_used^=1; /* Flip temp buffer */
+ c->tx_next_ptr = c->tx_dma_buf[c->tx_dma_used];
+ c->tx_dma_used ^= 1; /* Flip temp buffer */
skb_copy_from_linear_data(skb, c->tx_next_ptr, skb->len);
+ } else {
+ c->tx_next_ptr = skb->data;
}
- else
- c->tx_next_ptr=skb->data;
RT_LOCK;
- c->tx_next_skb=skb;
+ c->tx_next_skb = skb;
RT_UNLOCK;
-
+
spin_lock_irqsave(c->lock, flags);
z8530_tx_begin(c);
spin_unlock_irqrestore(c->lock, flags);
-
+
return NETDEV_TX_OK;
}
-
EXPORT_SYMBOL(z8530_queue_xmit);
-/*
- * Module support
+/* Module support
*/
static const char banner[] __initconst =
KERN_INFO "Generic Z85C30/Z85230 interface driver v0.02\n";
-ccflags-y := -O3
-ccflags-y += -D'pr_fmt(fmt)=KBUILD_MODNAME ": " fmt'
+ccflags-y := -D'pr_fmt(fmt)=KBUILD_MODNAME ": " fmt'
ccflags-$(CONFIG_WIREGUARD_DEBUG) += -DDEBUG
wireguard-y := main.o
wireguard-y += noise.o
#include "allowedips.h"
#include "peer.h"
+static struct kmem_cache *node_cache;
+
static void swap_endian(u8 *dst, const u8 *src, u8 bits)
{
if (bits == 32) {
node->bitlen = bits;
memcpy(node->bits, src, bits / 8U);
}
-#define CHOOSE_NODE(parent, key) \
- parent->bit[(key[parent->bit_at_a] >> parent->bit_at_b) & 1]
+
+static inline u8 choose(struct allowedips_node *node, const u8 *key)
+{
+ return (key[node->bit_at_a] >> node->bit_at_b) & 1;
+}
static void push_rcu(struct allowedips_node **stack,
struct allowedips_node __rcu *p, unsigned int *len)
}
}
+static void node_free_rcu(struct rcu_head *rcu)
+{
+ kmem_cache_free(node_cache, container_of(rcu, struct allowedips_node, rcu));
+}
+
static void root_free_rcu(struct rcu_head *rcu)
{
struct allowedips_node *node, *stack[128] = {
while (len > 0 && (node = stack[--len])) {
push_rcu(stack, node->bit[0], &len);
push_rcu(stack, node->bit[1], &len);
- kfree(node);
+ kmem_cache_free(node_cache, node);
}
}
}
}
-static void walk_remove_by_peer(struct allowedips_node __rcu **top,
- struct wg_peer *peer, struct mutex *lock)
-{
-#define REF(p) rcu_access_pointer(p)
-#define DEREF(p) rcu_dereference_protected(*(p), lockdep_is_held(lock))
-#define PUSH(p) ({ \
- WARN_ON(IS_ENABLED(DEBUG) && len >= 128); \
- stack[len++] = p; \
- })
-
- struct allowedips_node __rcu **stack[128], **nptr;
- struct allowedips_node *node, *prev;
- unsigned int len;
-
- if (unlikely(!peer || !REF(*top)))
- return;
-
- for (prev = NULL, len = 0, PUSH(top); len > 0; prev = node) {
- nptr = stack[len - 1];
- node = DEREF(nptr);
- if (!node) {
- --len;
- continue;
- }
- if (!prev || REF(prev->bit[0]) == node ||
- REF(prev->bit[1]) == node) {
- if (REF(node->bit[0]))
- PUSH(&node->bit[0]);
- else if (REF(node->bit[1]))
- PUSH(&node->bit[1]);
- } else if (REF(node->bit[0]) == prev) {
- if (REF(node->bit[1]))
- PUSH(&node->bit[1]);
- } else {
- if (rcu_dereference_protected(node->peer,
- lockdep_is_held(lock)) == peer) {
- RCU_INIT_POINTER(node->peer, NULL);
- list_del_init(&node->peer_list);
- if (!node->bit[0] || !node->bit[1]) {
- rcu_assign_pointer(*nptr, DEREF(
- &node->bit[!REF(node->bit[0])]));
- kfree_rcu(node, rcu);
- node = DEREF(nptr);
- }
- }
- --len;
- }
- }
-
-#undef REF
-#undef DEREF
-#undef PUSH
-}
-
static unsigned int fls128(u64 a, u64 b)
{
return a ? fls64(a) + 64U : fls64(b);
found = node;
if (node->cidr == bits)
break;
- node = rcu_dereference_bh(CHOOSE_NODE(node, key));
+ node = rcu_dereference_bh(node->bit[choose(node, key)]);
}
return found;
}
u8 cidr, u8 bits, struct allowedips_node **rnode,
struct mutex *lock)
{
- struct allowedips_node *node = rcu_dereference_protected(trie,
- lockdep_is_held(lock));
+ struct allowedips_node *node = rcu_dereference_protected(trie, lockdep_is_held(lock));
struct allowedips_node *parent = NULL;
bool exact = false;
exact = true;
break;
}
- node = rcu_dereference_protected(CHOOSE_NODE(parent, key),
- lockdep_is_held(lock));
+ node = rcu_dereference_protected(parent->bit[choose(parent, key)], lockdep_is_held(lock));
}
*rnode = parent;
return exact;
}
+static inline void connect_node(struct allowedips_node **parent, u8 bit, struct allowedips_node *node)
+{
+ node->parent_bit_packed = (unsigned long)parent | bit;
+ rcu_assign_pointer(*parent, node);
+}
+
+static inline void choose_and_connect_node(struct allowedips_node *parent, struct allowedips_node *node)
+{
+ u8 bit = choose(parent, node->bits);
+ connect_node(&parent->bit[bit], bit, node);
+}
+
static int add(struct allowedips_node __rcu **trie, u8 bits, const u8 *key,
u8 cidr, struct wg_peer *peer, struct mutex *lock)
{
return -EINVAL;
if (!rcu_access_pointer(*trie)) {
- node = kzalloc(sizeof(*node), GFP_KERNEL);
+ node = kmem_cache_zalloc(node_cache, GFP_KERNEL);
if (unlikely(!node))
return -ENOMEM;
RCU_INIT_POINTER(node->peer, peer);
list_add_tail(&node->peer_list, &peer->allowedips_list);
copy_and_assign_cidr(node, key, cidr, bits);
- rcu_assign_pointer(*trie, node);
+ connect_node(trie, 2, node);
return 0;
}
if (node_placement(*trie, key, cidr, bits, &node, lock)) {
return 0;
}
- newnode = kzalloc(sizeof(*newnode), GFP_KERNEL);
+ newnode = kmem_cache_zalloc(node_cache, GFP_KERNEL);
if (unlikely(!newnode))
return -ENOMEM;
RCU_INIT_POINTER(newnode->peer, peer);
if (!node) {
down = rcu_dereference_protected(*trie, lockdep_is_held(lock));
} else {
- down = rcu_dereference_protected(CHOOSE_NODE(node, key),
- lockdep_is_held(lock));
+ const u8 bit = choose(node, key);
+ down = rcu_dereference_protected(node->bit[bit], lockdep_is_held(lock));
if (!down) {
- rcu_assign_pointer(CHOOSE_NODE(node, key), newnode);
+ connect_node(&node->bit[bit], bit, newnode);
return 0;
}
}
parent = node;
if (newnode->cidr == cidr) {
- rcu_assign_pointer(CHOOSE_NODE(newnode, down->bits), down);
+ choose_and_connect_node(newnode, down);
if (!parent)
- rcu_assign_pointer(*trie, newnode);
+ connect_node(trie, 2, newnode);
else
- rcu_assign_pointer(CHOOSE_NODE(parent, newnode->bits),
- newnode);
- } else {
- node = kzalloc(sizeof(*node), GFP_KERNEL);
- if (unlikely(!node)) {
- list_del(&newnode->peer_list);
- kfree(newnode);
- return -ENOMEM;
- }
- INIT_LIST_HEAD(&node->peer_list);
- copy_and_assign_cidr(node, newnode->bits, cidr, bits);
+ choose_and_connect_node(parent, newnode);
+ return 0;
+ }
- rcu_assign_pointer(CHOOSE_NODE(node, down->bits), down);
- rcu_assign_pointer(CHOOSE_NODE(node, newnode->bits), newnode);
- if (!parent)
- rcu_assign_pointer(*trie, node);
- else
- rcu_assign_pointer(CHOOSE_NODE(parent, node->bits),
- node);
+ node = kmem_cache_zalloc(node_cache, GFP_KERNEL);
+ if (unlikely(!node)) {
+ list_del(&newnode->peer_list);
+ kmem_cache_free(node_cache, newnode);
+ return -ENOMEM;
}
+ INIT_LIST_HEAD(&node->peer_list);
+ copy_and_assign_cidr(node, newnode->bits, cidr, bits);
+
+ choose_and_connect_node(node, down);
+ choose_and_connect_node(node, newnode);
+ if (!parent)
+ connect_node(trie, 2, node);
+ else
+ choose_and_connect_node(parent, node);
return 0;
}
void wg_allowedips_remove_by_peer(struct allowedips *table,
struct wg_peer *peer, struct mutex *lock)
{
+ struct allowedips_node *node, *child, **parent_bit, *parent, *tmp;
+ bool free_parent;
+
+ if (list_empty(&peer->allowedips_list))
+ return;
++table->seq;
- walk_remove_by_peer(&table->root4, peer, lock);
- walk_remove_by_peer(&table->root6, peer, lock);
+ list_for_each_entry_safe(node, tmp, &peer->allowedips_list, peer_list) {
+ list_del_init(&node->peer_list);
+ RCU_INIT_POINTER(node->peer, NULL);
+ if (node->bit[0] && node->bit[1])
+ continue;
+ child = rcu_dereference_protected(node->bit[!rcu_access_pointer(node->bit[0])],
+ lockdep_is_held(lock));
+ if (child)
+ child->parent_bit_packed = node->parent_bit_packed;
+ parent_bit = (struct allowedips_node **)(node->parent_bit_packed & ~3UL);
+ *parent_bit = child;
+ parent = (void *)parent_bit -
+ offsetof(struct allowedips_node, bit[node->parent_bit_packed & 1]);
+ free_parent = !rcu_access_pointer(node->bit[0]) &&
+ !rcu_access_pointer(node->bit[1]) &&
+ (node->parent_bit_packed & 3) <= 1 &&
+ !rcu_access_pointer(parent->peer);
+ if (free_parent)
+ child = rcu_dereference_protected(
+ parent->bit[!(node->parent_bit_packed & 1)],
+ lockdep_is_held(lock));
+ call_rcu(&node->rcu, node_free_rcu);
+ if (!free_parent)
+ continue;
+ if (child)
+ child->parent_bit_packed = parent->parent_bit_packed;
+ *(struct allowedips_node **)(parent->parent_bit_packed & ~3UL) = child;
+ call_rcu(&parent->rcu, node_free_rcu);
+ }
}
int wg_allowedips_read_node(struct allowedips_node *node, u8 ip[16], u8 *cidr)
return NULL;
}
+int __init wg_allowedips_slab_init(void)
+{
+ node_cache = KMEM_CACHE(allowedips_node, 0);
+ return node_cache ? 0 : -ENOMEM;
+}
+
+void wg_allowedips_slab_uninit(void)
+{
+ rcu_barrier();
+ kmem_cache_destroy(node_cache);
+}
+
#include "selftest/allowedips.c"
struct allowedips_node {
struct wg_peer __rcu *peer;
struct allowedips_node __rcu *bit[2];
- /* While it may seem scandalous that we waste space for v4,
- * we're alloc'ing to the nearest power of 2 anyway, so this
- * doesn't actually make a difference.
- */
- u8 bits[16] __aligned(__alignof(u64));
u8 cidr, bit_at_a, bit_at_b, bitlen;
+ u8 bits[16] __aligned(__alignof(u64));
- /* Keep rarely used list at bottom to be beyond cache line. */
+ /* Keep rarely used members at bottom to be beyond cache line. */
+ unsigned long parent_bit_packed;
union {
struct list_head peer_list;
struct rcu_head rcu;
struct allowedips_node __rcu *root4;
struct allowedips_node __rcu *root6;
u64 seq;
-};
+} __aligned(4); /* We pack the lower 2 bits of &root, but m68k only gives 16-bit alignment. */
void wg_allowedips_init(struct allowedips *table);
void wg_allowedips_free(struct allowedips *table, struct mutex *mutex);
bool wg_allowedips_selftest(void);
#endif
+int wg_allowedips_slab_init(void);
+void wg_allowedips_slab_uninit(void);
+
#endif /* _WG_ALLOWEDIPS_H */
{
int ret;
+ ret = wg_allowedips_slab_init();
+ if (ret < 0)
+ goto err_allowedips;
+
#ifdef DEBUG
+ ret = -ENOTRECOVERABLE;
if (!wg_allowedips_selftest() || !wg_packet_counter_selftest() ||
!wg_ratelimiter_selftest())
- return -ENOTRECOVERABLE;
+ goto err_peer;
#endif
wg_noise_init();
+ ret = wg_peer_init();
+ if (ret < 0)
+ goto err_peer;
+
ret = wg_device_init();
if (ret < 0)
goto err_device;
err_netlink:
wg_device_uninit();
err_device:
+ wg_peer_uninit();
+err_peer:
+ wg_allowedips_slab_uninit();
+err_allowedips:
return ret;
}
{
wg_genetlink_uninit();
wg_device_uninit();
+ wg_peer_uninit();
+ wg_allowedips_slab_uninit();
}
module_init(mod_init);
#include <linux/rcupdate.h>
#include <linux/list.h>
+static struct kmem_cache *peer_cache;
static atomic64_t peer_counter = ATOMIC64_INIT(0);
struct wg_peer *wg_peer_create(struct wg_device *wg,
if (wg->num_peers >= MAX_PEERS_PER_DEVICE)
return ERR_PTR(ret);
- peer = kzalloc(sizeof(*peer), GFP_KERNEL);
+ peer = kmem_cache_zalloc(peer_cache, GFP_KERNEL);
if (unlikely(!peer))
return ERR_PTR(ret);
- if (dst_cache_init(&peer->endpoint_cache, GFP_KERNEL))
+ if (unlikely(dst_cache_init(&peer->endpoint_cache, GFP_KERNEL)))
goto err;
peer->device = wg;
return peer;
err:
- kfree(peer);
+ kmem_cache_free(peer_cache, peer);
return ERR_PTR(ret);
}
/* Mark as dead, so that we don't allow jumping contexts after. */
WRITE_ONCE(peer->is_dead, true);
- /* The caller must now synchronize_rcu() for this to take effect. */
+ /* The caller must now synchronize_net() for this to take effect. */
}
static void peer_remove_after_dead(struct wg_peer *peer)
lockdep_assert_held(&peer->device->device_update_lock);
peer_make_dead(peer);
- synchronize_rcu();
+ synchronize_net();
peer_remove_after_dead(peer);
}
peer_make_dead(peer);
list_add_tail(&peer->peer_list, &dead_peers);
}
- synchronize_rcu();
+ synchronize_net();
list_for_each_entry_safe(peer, temp, &dead_peers, peer_list)
peer_remove_after_dead(peer);
}
/* The final zeroing takes care of clearing any remaining handshake key
* material and other potentially sensitive information.
*/
- kfree_sensitive(peer);
+ memzero_explicit(peer, sizeof(*peer));
+ kmem_cache_free(peer_cache, peer);
}
static void kref_release(struct kref *refcount)
return;
kref_put(&peer->refcount, kref_release);
}
+
+int __init wg_peer_init(void)
+{
+ peer_cache = KMEM_CACHE(wg_peer, 0);
+ return peer_cache ? 0 : -ENOMEM;
+}
+
+void wg_peer_uninit(void)
+{
+ kmem_cache_destroy(peer_cache);
+}
void wg_peer_remove(struct wg_peer *peer);
void wg_peer_remove_all(struct wg_device *wg);
+int wg_peer_init(void);
+void wg_peer_uninit(void);
+
#endif /* _WG_PEER_H */
#include <linux/siphash.h>
-static __init void swap_endian_and_apply_cidr(u8 *dst, const u8 *src, u8 bits,
- u8 cidr)
-{
- swap_endian(dst, src, bits);
- memset(dst + (cidr + 7) / 8, 0, bits / 8 - (cidr + 7) / 8);
- if (cidr)
- dst[(cidr + 7) / 8 - 1] &= ~0U << ((8 - (cidr % 8)) % 8);
-}
-
static __init void print_node(struct allowedips_node *node, u8 bits)
{
char *fmt_connection = KERN_DEBUG "\t\"%p/%d\" -> \"%p/%d\";\n";
- char *fmt_declaration = KERN_DEBUG
- "\t\"%p/%d\"[style=%s, color=\"#%06x\"];\n";
+ char *fmt_declaration = KERN_DEBUG "\t\"%p/%d\"[style=%s, color=\"#%06x\"];\n";
+ u8 ip1[16], ip2[16], cidr1, cidr2;
char *style = "dotted";
- u8 ip1[16], ip2[16];
u32 color = 0;
+ if (node == NULL)
+ return;
if (bits == 32) {
fmt_connection = KERN_DEBUG "\t\"%pI4/%d\" -> \"%pI4/%d\";\n";
- fmt_declaration = KERN_DEBUG
- "\t\"%pI4/%d\"[style=%s, color=\"#%06x\"];\n";
+ fmt_declaration = KERN_DEBUG "\t\"%pI4/%d\"[style=%s, color=\"#%06x\"];\n";
} else if (bits == 128) {
fmt_connection = KERN_DEBUG "\t\"%pI6/%d\" -> \"%pI6/%d\";\n";
- fmt_declaration = KERN_DEBUG
- "\t\"%pI6/%d\"[style=%s, color=\"#%06x\"];\n";
+ fmt_declaration = KERN_DEBUG "\t\"%pI6/%d\"[style=%s, color=\"#%06x\"];\n";
}
if (node->peer) {
hsiphash_key_t key = { { 0 } };
hsiphash_1u32(0xabad1dea, &key) % 200;
style = "bold";
}
- swap_endian_and_apply_cidr(ip1, node->bits, bits, node->cidr);
- printk(fmt_declaration, ip1, node->cidr, style, color);
+ wg_allowedips_read_node(node, ip1, &cidr1);
+ printk(fmt_declaration, ip1, cidr1, style, color);
if (node->bit[0]) {
- swap_endian_and_apply_cidr(ip2,
- rcu_dereference_raw(node->bit[0])->bits, bits,
- node->cidr);
- printk(fmt_connection, ip1, node->cidr, ip2,
- rcu_dereference_raw(node->bit[0])->cidr);
- print_node(rcu_dereference_raw(node->bit[0]), bits);
+ wg_allowedips_read_node(rcu_dereference_raw(node->bit[0]), ip2, &cidr2);
+ printk(fmt_connection, ip1, cidr1, ip2, cidr2);
}
if (node->bit[1]) {
- swap_endian_and_apply_cidr(ip2,
- rcu_dereference_raw(node->bit[1])->bits,
- bits, node->cidr);
- printk(fmt_connection, ip1, node->cidr, ip2,
- rcu_dereference_raw(node->bit[1])->cidr);
- print_node(rcu_dereference_raw(node->bit[1]), bits);
+ wg_allowedips_read_node(rcu_dereference_raw(node->bit[1]), ip2, &cidr2);
+ printk(fmt_connection, ip1, cidr1, ip2, cidr2);
}
+ if (node->bit[0])
+ print_node(rcu_dereference_raw(node->bit[0]), bits);
+ if (node->bit[1])
+ print_node(rcu_dereference_raw(node->bit[1]), bits);
}
static __init void print_tree(struct allowedips_node __rcu *top, u8 bits)
{
union nf_inet_addr mask;
- memset(&mask, 0x00, 128 / 8);
- memset(&mask, 0xff, cidr / 8);
+ memset(&mask, 0, sizeof(mask));
+ memset(&mask.all, 0xff, cidr / 8);
if (cidr % 32)
mask.all[cidr / 32] = (__force u32)htonl(
(0xFFFFFFFFUL << (32 - (cidr % 32))) & 0xFFFFFFFFUL);
}
static __init inline bool
-horrible_match_v4(const struct horrible_allowedips_node *node,
- struct in_addr *ip)
+horrible_match_v4(const struct horrible_allowedips_node *node, struct in_addr *ip)
{
return (ip->s_addr & node->mask.ip) == node->ip.ip;
}
static __init inline bool
-horrible_match_v6(const struct horrible_allowedips_node *node,
- struct in6_addr *ip)
+horrible_match_v6(const struct horrible_allowedips_node *node, struct in6_addr *ip)
{
- return (ip->in6_u.u6_addr32[0] & node->mask.ip6[0]) ==
- node->ip.ip6[0] &&
- (ip->in6_u.u6_addr32[1] & node->mask.ip6[1]) ==
- node->ip.ip6[1] &&
- (ip->in6_u.u6_addr32[2] & node->mask.ip6[2]) ==
- node->ip.ip6[2] &&
+ return (ip->in6_u.u6_addr32[0] & node->mask.ip6[0]) == node->ip.ip6[0] &&
+ (ip->in6_u.u6_addr32[1] & node->mask.ip6[1]) == node->ip.ip6[1] &&
+ (ip->in6_u.u6_addr32[2] & node->mask.ip6[2]) == node->ip.ip6[2] &&
(ip->in6_u.u6_addr32[3] & node->mask.ip6[3]) == node->ip.ip6[3];
}
static __init void
-horrible_insert_ordered(struct horrible_allowedips *table,
- struct horrible_allowedips_node *node)
+horrible_insert_ordered(struct horrible_allowedips *table, struct horrible_allowedips_node *node)
{
struct horrible_allowedips_node *other = NULL, *where = NULL;
u8 my_cidr = horrible_mask_to_cidr(node->mask);
hlist_for_each_entry(other, &table->head, table) {
- if (!memcmp(&other->mask, &node->mask,
- sizeof(union nf_inet_addr)) &&
- !memcmp(&other->ip, &node->ip,
- sizeof(union nf_inet_addr)) &&
- other->ip_version == node->ip_version) {
+ if (other->ip_version == node->ip_version &&
+ !memcmp(&other->mask, &node->mask, sizeof(union nf_inet_addr)) &&
+ !memcmp(&other->ip, &node->ip, sizeof(union nf_inet_addr))) {
other->value = node->value;
kfree(node);
return;
}
+ }
+ hlist_for_each_entry(other, &table->head, table) {
where = other;
if (horrible_mask_to_cidr(other->mask) <= my_cidr)
break;
horrible_allowedips_insert_v4(struct horrible_allowedips *table,
struct in_addr *ip, u8 cidr, void *value)
{
- struct horrible_allowedips_node *node = kzalloc(sizeof(*node),
- GFP_KERNEL);
+ struct horrible_allowedips_node *node = kzalloc(sizeof(*node), GFP_KERNEL);
if (unlikely(!node))
return -ENOMEM;
horrible_allowedips_insert_v6(struct horrible_allowedips *table,
struct in6_addr *ip, u8 cidr, void *value)
{
- struct horrible_allowedips_node *node = kzalloc(sizeof(*node),
- GFP_KERNEL);
+ struct horrible_allowedips_node *node = kzalloc(sizeof(*node), GFP_KERNEL);
if (unlikely(!node))
return -ENOMEM;
}
static __init void *
-horrible_allowedips_lookup_v4(struct horrible_allowedips *table,
- struct in_addr *ip)
+horrible_allowedips_lookup_v4(struct horrible_allowedips *table, struct in_addr *ip)
{
struct horrible_allowedips_node *node;
- void *ret = NULL;
hlist_for_each_entry(node, &table->head, table) {
- if (node->ip_version != 4)
- continue;
- if (horrible_match_v4(node, ip)) {
- ret = node->value;
- break;
- }
+ if (node->ip_version == 4 && horrible_match_v4(node, ip))
+ return node->value;
}
- return ret;
+ return NULL;
}
static __init void *
-horrible_allowedips_lookup_v6(struct horrible_allowedips *table,
- struct in6_addr *ip)
+horrible_allowedips_lookup_v6(struct horrible_allowedips *table, struct in6_addr *ip)
{
struct horrible_allowedips_node *node;
- void *ret = NULL;
hlist_for_each_entry(node, &table->head, table) {
- if (node->ip_version != 6)
+ if (node->ip_version == 6 && horrible_match_v6(node, ip))
+ return node->value;
+ }
+ return NULL;
+}
+
+
+static __init void
+horrible_allowedips_remove_by_value(struct horrible_allowedips *table, void *value)
+{
+ struct horrible_allowedips_node *node;
+ struct hlist_node *h;
+
+ hlist_for_each_entry_safe(node, h, &table->head, table) {
+ if (node->value != value)
continue;
- if (horrible_match_v6(node, ip)) {
- ret = node->value;
- break;
- }
+ hlist_del(&node->table);
+ kfree(node);
}
- return ret;
+
}
static __init bool randomized_test(void)
goto free;
}
kref_init(&peers[i]->refcount);
+ INIT_LIST_HEAD(&peers[i]->allowedips_list);
}
mutex_lock(&mutex);
if (wg_allowedips_insert_v4(&t,
(struct in_addr *)mutated,
cidr, peer, &mutex) < 0) {
- pr_err("allowedips random malloc: FAIL\n");
+ pr_err("allowedips random self-test malloc: FAIL\n");
goto free_locked;
}
if (horrible_allowedips_insert_v4(&h,
print_tree(t.root6, 128);
}
- for (i = 0; i < NUM_QUERIES; ++i) {
- prandom_bytes(ip, 4);
- if (lookup(t.root4, 32, ip) !=
- horrible_allowedips_lookup_v4(&h, (struct in_addr *)ip)) {
- pr_err("allowedips random self-test: FAIL\n");
- goto free;
+ for (j = 0;; ++j) {
+ for (i = 0; i < NUM_QUERIES; ++i) {
+ prandom_bytes(ip, 4);
+ if (lookup(t.root4, 32, ip) != horrible_allowedips_lookup_v4(&h, (struct in_addr *)ip)) {
+ horrible_allowedips_lookup_v4(&h, (struct in_addr *)ip);
+ pr_err("allowedips random v4 self-test: FAIL\n");
+ goto free;
+ }
+ prandom_bytes(ip, 16);
+ if (lookup(t.root6, 128, ip) != horrible_allowedips_lookup_v6(&h, (struct in6_addr *)ip)) {
+ pr_err("allowedips random v6 self-test: FAIL\n");
+ goto free;
+ }
}
+ if (j >= NUM_PEERS)
+ break;
+ mutex_lock(&mutex);
+ wg_allowedips_remove_by_peer(&t, peers[j], &mutex);
+ mutex_unlock(&mutex);
+ horrible_allowedips_remove_by_value(&h, peers[j]);
}
- for (i = 0; i < NUM_QUERIES; ++i) {
- prandom_bytes(ip, 16);
- if (lookup(t.root6, 128, ip) !=
- horrible_allowedips_lookup_v6(&h, (struct in6_addr *)ip)) {
- pr_err("allowedips random self-test: FAIL\n");
- goto free;
- }
+ if (t.root4 || t.root6) {
+ pr_err("allowedips random self-test removal: FAIL\n");
+ goto free;
}
+
ret = true;
free:
if (new4)
wg->incoming_port = ntohs(inet_sk(new4)->inet_sport);
mutex_unlock(&wg->socket_update_lock);
- synchronize_rcu();
+ synchronize_net();
sock_free(old4);
sock_free(old6);
}
pdev = ar_ahb->pdev;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!res) {
- ath10k_err(ar, "failed to get memory resource\n");
- ret = -ENXIO;
- goto out;
- }
-
- ar_ahb->mem = devm_ioremap_resource(&pdev->dev, res);
+ ar_ahb->mem = devm_platform_get_and_ioremap_resource(pdev, 0, &res);
if (IS_ERR(ar_ahb->mem)) {
ath10k_err(ar, "mem ioremap error\n");
ret = PTR_ERR(ar_ahb->mem);
s32 underrun;
u32 hw_paused;
s32 tx_abort;
- s32 mpdus_requed;
+ s32 mpdus_requeued;
u32 tx_ko;
u32 data_rc;
u32 self_triggers;
"d_tx_ppdu_reaped",
"d_tx_fifo_underrun",
"d_tx_ppdu_abort",
- "d_tx_mpdu_requed",
+ "d_tx_mpdu_requeued",
"d_tx_excessive_retries",
"d_tx_hw_rate",
"d_tx_dropped_sw_retries",
data[i++] = pdev_stats->hw_reaped;
data[i++] = pdev_stats->underrun;
data[i++] = pdev_stats->tx_abort;
- data[i++] = pdev_stats->mpdus_requed;
+ data[i++] = pdev_stats->mpdus_requeued;
data[i++] = pdev_stats->tx_ko;
data[i++] = pdev_stats->data_rc;
data[i++] = pdev_stats->sw_retry_failure;
#define ATH10K_HTT_TXRX_PEER_SECURITY_MAX 2
#define ATH10K_TXRX_NUM_EXT_TIDS 19
+#define ATH10K_TXRX_NON_QOS_TID 16
enum htt_security_flags {
#define HTT_SECURITY_TYPE_MASK 0x7F
/* Num PPDUs cleaned up in TX abort */
__le32 tx_abort;
- /* Num MPDUs requed by SW */
- __le32 mpdus_requed;
+ /* Num MPDUs requeued by SW */
+ __le32 mpdus_requeued;
/* excessive retries */
__le32 tx_ko;
msdu->ip_summed = ath10k_htt_rx_get_csum_state(msdu);
}
+static u64 ath10k_htt_rx_h_get_pn(struct ath10k *ar, struct sk_buff *skb,
+ u16 offset,
+ enum htt_rx_mpdu_encrypt_type enctype)
+{
+ struct ieee80211_hdr *hdr;
+ u64 pn = 0;
+ u8 *ehdr;
+
+ hdr = (struct ieee80211_hdr *)(skb->data + offset);
+ ehdr = skb->data + offset + ieee80211_hdrlen(hdr->frame_control);
+
+ if (enctype == HTT_RX_MPDU_ENCRYPT_AES_CCM_WPA2) {
+ pn = ehdr[0];
+ pn |= (u64)ehdr[1] << 8;
+ pn |= (u64)ehdr[4] << 16;
+ pn |= (u64)ehdr[5] << 24;
+ pn |= (u64)ehdr[6] << 32;
+ pn |= (u64)ehdr[7] << 40;
+ }
+ return pn;
+}
+
+static bool ath10k_htt_rx_h_frag_multicast_check(struct ath10k *ar,
+ struct sk_buff *skb,
+ u16 offset)
+{
+ struct ieee80211_hdr *hdr;
+
+ hdr = (struct ieee80211_hdr *)(skb->data + offset);
+ return !is_multicast_ether_addr(hdr->addr1);
+}
+
+static bool ath10k_htt_rx_h_frag_pn_check(struct ath10k *ar,
+ struct sk_buff *skb,
+ u16 peer_id,
+ u16 offset,
+ enum htt_rx_mpdu_encrypt_type enctype)
+{
+ struct ath10k_peer *peer;
+ union htt_rx_pn_t *last_pn, new_pn = {0};
+ struct ieee80211_hdr *hdr;
+ u8 tid, frag_number;
+ u32 seq;
+
+ peer = ath10k_peer_find_by_id(ar, peer_id);
+ if (!peer) {
+ ath10k_dbg(ar, ATH10K_DBG_HTT, "invalid peer for frag pn check\n");
+ return false;
+ }
+
+ hdr = (struct ieee80211_hdr *)(skb->data + offset);
+ if (ieee80211_is_data_qos(hdr->frame_control))
+ tid = ieee80211_get_tid(hdr);
+ else
+ tid = ATH10K_TXRX_NON_QOS_TID;
+
+ last_pn = &peer->frag_tids_last_pn[tid];
+ new_pn.pn48 = ath10k_htt_rx_h_get_pn(ar, skb, offset, enctype);
+ frag_number = le16_to_cpu(hdr->seq_ctrl) & IEEE80211_SCTL_FRAG;
+ seq = (__le16_to_cpu(hdr->seq_ctrl) & IEEE80211_SCTL_SEQ) >> 4;
+
+ if (frag_number == 0) {
+ last_pn->pn48 = new_pn.pn48;
+ peer->frag_tids_seq[tid] = seq;
+ } else {
+ if (seq != peer->frag_tids_seq[tid])
+ return false;
+
+ if (new_pn.pn48 != last_pn->pn48 + 1)
+ return false;
+
+ last_pn->pn48 = new_pn.pn48;
+ }
+
+ return true;
+}
+
static void ath10k_htt_rx_h_mpdu(struct ath10k *ar,
struct sk_buff_head *amsdu,
struct ieee80211_rx_status *status,
bool fill_crypt_header,
u8 *rx_hdr,
- enum ath10k_pkt_rx_err *err)
+ enum ath10k_pkt_rx_err *err,
+ u16 peer_id,
+ bool frag)
{
struct sk_buff *first;
struct sk_buff *last;
- struct sk_buff *msdu;
+ struct sk_buff *msdu, *temp;
struct htt_rx_desc *rxd;
struct ieee80211_hdr *hdr;
enum htt_rx_mpdu_encrypt_type enctype;
bool is_decrypted;
bool is_mgmt;
u32 attention;
+ bool frag_pn_check = true, multicast_check = true;
if (skb_queue_empty(amsdu))
return;
}
skb_queue_walk(amsdu, msdu) {
+ if (frag && !fill_crypt_header && is_decrypted &&
+ enctype == HTT_RX_MPDU_ENCRYPT_AES_CCM_WPA2)
+ frag_pn_check = ath10k_htt_rx_h_frag_pn_check(ar,
+ msdu,
+ peer_id,
+ 0,
+ enctype);
+
+ if (frag)
+ multicast_check = ath10k_htt_rx_h_frag_multicast_check(ar,
+ msdu,
+ 0);
+
+ if (!frag_pn_check || !multicast_check) {
+ /* Discard the fragment with invalid PN or multicast DA
+ */
+ temp = msdu->prev;
+ __skb_unlink(msdu, amsdu);
+ dev_kfree_skb_any(msdu);
+ msdu = temp;
+ frag_pn_check = true;
+ multicast_check = true;
+ continue;
+ }
+
ath10k_htt_rx_h_csum_offload(msdu);
+
+ if (frag && !fill_crypt_header &&
+ enctype == HTT_RX_MPDU_ENCRYPT_TKIP_WPA)
+ status->flag &= ~RX_FLAG_MMIC_STRIPPED;
+
ath10k_htt_rx_h_undecap(ar, msdu, status, first_hdr, enctype,
is_decrypted);
hdr = (void *)msdu->data;
hdr->frame_control &= ~__cpu_to_le16(IEEE80211_FCTL_PROTECTED);
+
+ if (frag && !fill_crypt_header &&
+ enctype == HTT_RX_MPDU_ENCRYPT_TKIP_WPA)
+ status->flag &= ~RX_FLAG_IV_STRIPPED &
+ ~RX_FLAG_MMIC_STRIPPED;
}
}
ath10k_unchain_msdu(amsdu, unchain_cnt);
}
+static bool ath10k_htt_rx_validate_amsdu(struct ath10k *ar,
+ struct sk_buff_head *amsdu)
+{
+ u8 *subframe_hdr;
+ struct sk_buff *first;
+ bool is_first, is_last;
+ struct htt_rx_desc *rxd;
+ struct ieee80211_hdr *hdr;
+ size_t hdr_len, crypto_len;
+ enum htt_rx_mpdu_encrypt_type enctype;
+ int bytes_aligned = ar->hw_params.decap_align_bytes;
+
+ first = skb_peek(amsdu);
+
+ rxd = (void *)first->data - sizeof(*rxd);
+ hdr = (void *)rxd->rx_hdr_status;
+
+ is_first = !!(rxd->msdu_end.common.info0 &
+ __cpu_to_le32(RX_MSDU_END_INFO0_FIRST_MSDU));
+ is_last = !!(rxd->msdu_end.common.info0 &
+ __cpu_to_le32(RX_MSDU_END_INFO0_LAST_MSDU));
+
+ /* Return in case of non-aggregated msdu */
+ if (is_first && is_last)
+ return true;
+
+ /* First msdu flag is not set for the first msdu of the list */
+ if (!is_first)
+ return false;
+
+ enctype = MS(__le32_to_cpu(rxd->mpdu_start.info0),
+ RX_MPDU_START_INFO0_ENCRYPT_TYPE);
+
+ hdr_len = ieee80211_hdrlen(hdr->frame_control);
+ crypto_len = ath10k_htt_rx_crypto_param_len(ar, enctype);
+
+ subframe_hdr = (u8 *)hdr + round_up(hdr_len, bytes_aligned) +
+ crypto_len;
+
+ /* Validate if the amsdu has a proper first subframe.
+ * There are chances a single msdu can be received as amsdu when
+ * the unauthenticated amsdu flag of a QoS header
+ * gets flipped in non-SPP AMSDU's, in such cases the first
+ * subframe has llc/snap header in place of a valid da.
+ * return false if the da matches rfc1042 pattern
+ */
+ if (ether_addr_equal(subframe_hdr, rfc1042_header))
+ return false;
+
+ return true;
+}
+
static bool ath10k_htt_rx_amsdu_allowed(struct ath10k *ar,
struct sk_buff_head *amsdu,
struct ieee80211_rx_status *rx_status)
{
- /* FIXME: It might be a good idea to do some fuzzy-testing to drop
- * invalid/dangerous frames.
- */
-
if (!rx_status->freq) {
ath10k_dbg(ar, ATH10K_DBG_HTT, "no channel configured; ignoring frame(s)!\n");
return false;
return false;
}
+ if (!ath10k_htt_rx_validate_amsdu(ar, amsdu)) {
+ ath10k_dbg(ar, ATH10K_DBG_HTT, "invalid amsdu received\n");
+ return false;
+ }
+
return true;
}
ath10k_htt_rx_h_unchain(ar, &amsdu, &drop_cnt, &unchain_cnt);
ath10k_htt_rx_h_filter(ar, &amsdu, rx_status, &drop_cnt_filter);
- ath10k_htt_rx_h_mpdu(ar, &amsdu, rx_status, true, first_hdr, &err);
+ ath10k_htt_rx_h_mpdu(ar, &amsdu, rx_status, true, first_hdr, &err, 0,
+ false);
msdus_to_queue = skb_queue_len(&amsdu);
ath10k_htt_rx_h_enqueue(ar, &amsdu, rx_status);
fw_desc = &rx->fw_desc;
rx_desc_len = fw_desc->len;
+ if (fw_desc->u.bits.discard) {
+ ath10k_dbg(ar, ATH10K_DBG_HTT, "htt discard mpdu\n");
+ goto err;
+ }
+
/* I have not yet seen any case where num_mpdu_ranges > 1.
* qcacld does not seem handle that case either, so we introduce the
* same limitiation here as well.
rx_desc = (struct htt_hl_rx_desc *)(skb->data + tot_hdr_len);
rx_desc_info = __le32_to_cpu(rx_desc->info);
+ hdr = (struct ieee80211_hdr *)((u8 *)rx_desc + rx_hl->fw_desc.len);
+
+ if (is_multicast_ether_addr(hdr->addr1)) {
+ /* Discard the fragment with multicast DA */
+ goto err;
+ }
+
if (!MS(rx_desc_info, HTT_RX_DESC_HL_INFO_ENCRYPTED)) {
spin_unlock_bh(&ar->data_lock);
return ath10k_htt_rx_proc_rx_ind_hl(htt, &resp->rx_ind_hl, skb,
HTT_RX_NON_TKIP_MIC);
}
- hdr = (struct ieee80211_hdr *)((u8 *)rx_desc + rx_hl->fw_desc.len);
-
if (ieee80211_has_retry(hdr->frame_control))
goto err;
ath10k_htt_rx_h_ppdu(ar, &amsdu, status, vdev_id);
ath10k_htt_rx_h_filter(ar, &amsdu, status, NULL);
ath10k_htt_rx_h_mpdu(ar, &amsdu, status, false, NULL,
- NULL);
+ NULL, peer_id, frag);
ath10k_htt_rx_h_enqueue(ar, &amsdu, status);
break;
case -EAGAIN:
if (arvif->nohwcrypt &&
!test_bit(ATH10K_FLAG_RAW_MODE, &ar->dev_flags)) {
+ ret = -EINVAL;
ath10k_warn(ar, "cryptmode module param needed for sw crypto\n");
goto err;
}
ath10k_pci_soc_read32(ar, SOC_CHIP_ID_ADDRESS);
if (bus_params.chip_id != 0xffffffff) {
if (!ath10k_pci_chip_is_supported(pdev->device,
- bus_params.chip_id))
+ bus_params.chip_id)) {
+ ret = -ENODEV;
goto err_unsupported;
+ }
}
}
}
bus_params.chip_id = ath10k_pci_soc_read32(ar, SOC_CHIP_ID_ADDRESS);
- if (bus_params.chip_id == 0xffffffff)
+ if (bus_params.chip_id == 0xffffffff) {
+ ret = -ENODEV;
goto err_unsupported;
+ }
- if (!ath10k_pci_chip_is_supported(pdev->device, bus_params.chip_id))
- goto err_free_irq;
+ if (!ath10k_pci_chip_is_supported(pdev->device, bus_params.chip_id)) {
+ ret = -ENODEV;
+ goto err_unsupported;
+ }
ret = ath10k_core_register(ar, &bus_params);
if (ret) {
void ath10k_pci_hif_power_down(struct ath10k *ar);
int ath10k_pci_alloc_pipes(struct ath10k *ar);
void ath10k_pci_free_pipes(struct ath10k *ar);
-void ath10k_pci_free_pipes(struct ath10k *ar);
void ath10k_pci_rx_replenish_retry(struct timer_list *t);
void ath10k_pci_ce_deinit(struct ath10k *ar);
void ath10k_pci_init_napi(struct ath10k *ar);
#define FW_RX_DESC_UDP (1 << 6)
struct fw_rx_desc_hl {
- u8 info0;
+ union {
+ struct {
+ u8 discard:1,
+ forward:1,
+ any_err:1,
+ dup_err:1,
+ reserved:1,
+ inspect:1,
+ extension:2;
+ } bits;
+ u8 info0;
+ } u;
+
u8 version;
u8 len;
u8 flags;
switch (ar->scan.state) {
case ATH10K_SCAN_IDLE:
case ATH10K_SCAN_STARTING:
- ath10k_warn(ar, "received chan info event without a scan request, ignoring\n");
+ ath10k_dbg(ar, ATH10K_DBG_WMI, "received chan info event without a scan request, ignoring\n");
goto exit;
case ATH10K_SCAN_RUNNING:
case ATH10K_SCAN_ABORTING:
dst->hw_reaped = __le32_to_cpu(src->hw_reaped);
dst->underrun = __le32_to_cpu(src->underrun);
dst->tx_abort = __le32_to_cpu(src->tx_abort);
- dst->mpdus_requed = __le32_to_cpu(src->mpdus_requed);
+ dst->mpdus_requeued = __le32_to_cpu(src->mpdus_requeued);
dst->tx_ko = __le32_to_cpu(src->tx_ko);
dst->data_rc = __le32_to_cpu(src->data_rc);
dst->self_triggers = __le32_to_cpu(src->self_triggers);
dst->hw_reaped = __le32_to_cpu(src->hw_reaped);
dst->underrun = __le32_to_cpu(src->underrun);
dst->tx_abort = __le32_to_cpu(src->tx_abort);
- dst->mpdus_requed = __le32_to_cpu(src->mpdus_requed);
+ dst->mpdus_requeued = __le32_to_cpu(src->mpdus_requeued);
dst->tx_ko = __le32_to_cpu(src->tx_ko);
dst->data_rc = __le32_to_cpu(src->data_rc);
dst->self_triggers = __le32_to_cpu(src->self_triggers);
len += scnprintf(buf + len, buf_len - len, "%30s %10d\n",
"PPDUs cleaned", pdev->tx_abort);
len += scnprintf(buf + len, buf_len - len, "%30s %10d\n",
- "MPDUs requed", pdev->mpdus_requed);
+ "MPDUs requeued", pdev->mpdus_requeued);
len += scnprintf(buf + len, buf_len - len, "%30s %10d\n",
"Excessive retries", pdev->tx_ko);
len += scnprintf(buf + len, buf_len - len, "%30s %10d\n",
/* Num PPDUs cleaned up in TX abort */
__le32 tx_abort;
- /* Num MPDUs requed by SW */
- __le32 mpdus_requed;
+ /* Num MPDUs requeued by SW */
+ __le32 mpdus_requeued;
/* excessive retries */
__le32 tx_ko;
/* Num PPDUs cleaned up in TX abort */
__le32 tx_abort;
- /* Num MPDUs requed by SW */
- __le32 mpdus_requed;
+ /* Num MPDUs requeued by SW */
+ __le32 mpdus_requeued;
/* excessive retries */
__le32 tx_ko;
struct sk_buff *ath10k_wmi_alloc_skb(struct ath10k *ar, u32 len);
int ath10k_wmi_connect(struct ath10k *ar);
-struct sk_buff *ath10k_wmi_alloc_skb(struct ath10k *ar, u32 len);
int ath10k_wmi_cmd_send(struct ath10k *ar, struct sk_buff *skb, u32 cmd_id);
int ath10k_wmi_cmd_send_nowait(struct ath10k *ar, struct sk_buff *skb,
u32 cmd_id);
.cold_boot_calib = true,
.supports_suspend = false,
.hal_desc_sz = sizeof(struct hal_rx_desc_ipq8074),
+ .fix_l1ss = true,
},
{
.hw_rev = ATH11K_HW_IPQ6018_HW10,
.cold_boot_calib = true,
.supports_suspend = false,
.hal_desc_sz = sizeof(struct hal_rx_desc_ipq8074),
+ .fix_l1ss = true,
},
{
.name = "qca6390 hw2.0",
.cold_boot_calib = false,
.supports_suspend = true,
.hal_desc_sz = sizeof(struct hal_rx_desc_ipq8074),
+ .fix_l1ss = true,
},
{
.name = "qcn9074 hw1.0",
.cold_boot_calib = false,
.supports_suspend = false,
.hal_desc_sz = sizeof(struct hal_rx_desc_qcn9074),
+ .fix_l1ss = true,
+ },
+ {
+ .name = "wcn6855 hw2.0",
+ .hw_rev = ATH11K_HW_WCN6855_HW20,
+ .fw = {
+ .dir = "WCN6855/hw2.0",
+ .board_size = 256 * 1024,
+ .cal_size = 256 * 1024,
+ },
+ .max_radios = 3,
+ .bdf_addr = 0x4B0C0000,
+ .hw_ops = &wcn6855_ops,
+ .ring_mask = &ath11k_hw_ring_mask_qca6390,
+ .internal_sleep_clock = true,
+ .regs = &wcn6855_regs,
+ .qmi_service_ins_id = ATH11K_QMI_WLFW_SERVICE_INS_ID_V01_QCA6390,
+ .host_ce_config = ath11k_host_ce_config_qca6390,
+ .ce_count = 9,
+ .target_ce_config = ath11k_target_ce_config_wlan_qca6390,
+ .target_ce_count = 9,
+ .svc_to_ce_map = ath11k_target_service_to_ce_map_wlan_qca6390,
+ .svc_to_ce_map_len = 14,
+ .single_pdev_only = true,
+ .rxdma1_enable = false,
+ .num_rxmda_per_pdev = 2,
+ .rx_mac_buf_ring = true,
+ .vdev_start_delay = true,
+ .htt_peer_map_v2 = false,
+ .tcl_0_only = true,
+ .spectral_fft_sz = 0,
+
+ .interface_modes = BIT(NL80211_IFTYPE_STATION) |
+ BIT(NL80211_IFTYPE_AP),
+ .supports_monitor = false,
+ .supports_shadow_regs = true,
+ .idle_ps = true,
+ .cold_boot_calib = false,
+ .supports_suspend = true,
+ .hal_desc_sz = sizeof(struct hal_rx_desc_wcn6855),
+ .fix_l1ss = false,
},
};
if (len < ALIGN(ie_len, 4)) {
ath11k_err(ab, "invalid length for board ie_id %d ie_len %zu len %zu\n",
ie_id, ie_len, len);
- return -EINVAL;
+ ret = -EINVAL;
+ goto err;
}
switch (ie_id) {
ATH11K_HW_QCA6390_HW20,
ATH11K_HW_IPQ6018_HW10,
ATH11K_HW_QCN9074_HW10,
+ ATH11K_HW_WCN6855_HW20,
};
enum ath11k_firmware_mode {
s32 underrun;
/* Num PPDUs cleaned up in TX abort */
s32 tx_abort;
- /* Num MPDUs requed by SW */
- s32 mpdus_requed;
+ /* Num MPDUs requeued by SW */
+ s32 mpdus_requeued;
/* excessive retries */
u32 tx_ko;
/* data hw rate code */
len += HTT_DBG_OUT(buf + len, buf_len - len, "tx_abort = %u",
htt_stats_buf->tx_abort);
len += HTT_DBG_OUT(buf + len, buf_len - len, "mpdu_requeued = %u",
- htt_stats_buf->mpdu_requed);
+ htt_stats_buf->mpdu_requeued);
len += HTT_DBG_OUT(buf + len, buf_len - len, "tx_xretry = %u",
htt_stats_buf->tx_xretry);
len += HTT_DBG_OUT(buf + len, buf_len - len, "data_rc = %u",
u32 hw_flush;
u32 hw_filt;
u32 tx_abort;
- u32 mpdu_requed;
+ u32 mpdu_requeued;
u32 tx_xretry;
u32 data_rc;
u32 mpdu_dropped_xretry;
struct ath11k_dp *dp = &ab->dp;
struct hal_srng *srng;
int i, ret;
- u32 ring_hash_map;
ret = ath11k_dp_srng_setup(ab, &dp->wbm_desc_rel_ring,
HAL_SW2WBM_RELEASE, 0, 0,
}
/* When hash based routing of rx packet is enabled, 32 entries to map
- * the hash values to the ring will be configured. Each hash entry uses
- * three bits to map to a particular ring. The ring mapping will be
- * 0:TCL, 1:SW1, 2:SW2, 3:SW3, 4:SW4, 5:Release, 6:FW and 7:Not used.
+ * the hash values to the ring will be configured.
*/
- ring_hash_map = HAL_HASH_ROUTING_RING_SW1 << 0 |
- HAL_HASH_ROUTING_RING_SW2 << 3 |
- HAL_HASH_ROUTING_RING_SW3 << 6 |
- HAL_HASH_ROUTING_RING_SW4 << 9 |
- HAL_HASH_ROUTING_RING_SW1 << 12 |
- HAL_HASH_ROUTING_RING_SW2 << 15 |
- HAL_HASH_ROUTING_RING_SW3 << 18 |
- HAL_HASH_ROUTING_RING_SW4 << 21;
-
- ath11k_hal_reo_hw_setup(ab, ring_hash_map);
+ ab->hw_params.hw_ops->reo_setup(ab);
return 0;
ab->hw_params.hw_ops->rx_desc_set_msdu_len(desc, len);
}
+static bool ath11k_dp_rx_h_attn_is_mcbc(struct ath11k_base *ab,
+ struct hal_rx_desc *desc)
+{
+ struct rx_attention *attn = ath11k_dp_rx_get_attention(ab, desc);
+
+ return ath11k_dp_rx_h_msdu_end_first_msdu(ab, desc) &&
+ (!!FIELD_GET(RX_ATTENTION_INFO1_MCAST_BCAST,
+ __le32_to_cpu(attn->info1)));
+}
+
static void ath11k_dp_service_mon_ring(struct timer_list *t)
{
struct ath11k_base *ab = from_timer(ab, t, mon_reap_timer);
__skb_queue_purge(&rx_tid->rx_frags);
}
+void ath11k_peer_frags_flush(struct ath11k *ar, struct ath11k_peer *peer)
+{
+ struct dp_rx_tid *rx_tid;
+ int i;
+
+ lockdep_assert_held(&ar->ab->base_lock);
+
+ for (i = 0; i <= IEEE80211_NUM_TIDS; i++) {
+ rx_tid = &peer->rx_tid[i];
+
+ spin_unlock_bh(&ar->ab->base_lock);
+ del_timer_sync(&rx_tid->frag_timer);
+ spin_lock_bh(&ar->ab->base_lock);
+
+ ath11k_dp_rx_frags_cleanup(rx_tid, true);
+ }
+}
+
void ath11k_peer_rx_tid_cleanup(struct ath11k *ar, struct ath11k_peer *peer)
{
struct dp_rx_tid *rx_tid;
u8 tid;
int ret = 0;
bool more_frags;
+ bool is_mcbc;
rx_desc = (struct hal_rx_desc *)msdu->data;
peer_id = ath11k_dp_rx_h_mpdu_start_peer_id(ar->ab, rx_desc);
seqno = ath11k_dp_rx_h_mpdu_start_seq_no(ar->ab, rx_desc);
frag_no = ath11k_dp_rx_h_mpdu_start_frag_no(ar->ab, msdu);
more_frags = ath11k_dp_rx_h_mpdu_start_more_frags(ar->ab, msdu);
+ is_mcbc = ath11k_dp_rx_h_attn_is_mcbc(ar->ab, rx_desc);
+
+ /* Multicast/Broadcast fragments are not expected */
+ if (is_mcbc)
+ return -EINVAL;
if (!ath11k_dp_rx_h_mpdu_start_seq_ctrl_valid(ar->ab, rx_desc) ||
!ath11k_dp_rx_h_mpdu_start_fc_valid(ar->ab, rx_desc) ||
const u8 *peer_addr,
enum set_key_cmd key_cmd,
struct ieee80211_key_conf *key);
+void ath11k_peer_frags_flush(struct ath11k *ar, struct ath11k_peer *peer);
void ath11k_peer_rx_tid_cleanup(struct ath11k *ar, struct ath11k_peer *peer);
void ath11k_peer_rx_tid_delete(struct ath11k *ar,
struct ath11k_peer *peer, u8 tid);
val = FIELD_PREP(HAL_REO1_RING_ID_ENTRY_SIZE, srng->entry_size);
ath11k_hif_write32(ab, reg_base + HAL_TCL1_RING_ID_OFFSET(ab), val);
+ if (srng->ring_id == HAL_SRNG_RING_ID_WBM_IDLE_LINK) {
+ ath11k_hif_write32(ab, reg_base, (u32)srng->ring_base_paddr);
+ val = FIELD_PREP(HAL_TCL1_RING_BASE_MSB_RING_BASE_ADDR_MSB,
+ ((u64)srng->ring_base_paddr >>
+ HAL_ADDR_MSB_REG_SHIFT)) |
+ FIELD_PREP(HAL_TCL1_RING_BASE_MSB_RING_SIZE,
+ (srng->entry_size * srng->num_entries));
+ ath11k_hif_write32(ab, reg_base + HAL_TCL1_RING_BASE_MSB_OFFSET(ab), val);
+ }
+
/* interrupt setup */
/* NOTE: IPQ8074 v2 requires the interrupt timer threshold in the
* unit of 8 usecs instead of 1 usec (as required by v1).
#define HAL_REO1_DEST_RING_CTRL_IX_1 0x00000008
#define HAL_REO1_DEST_RING_CTRL_IX_2 0x0000000c
#define HAL_REO1_DEST_RING_CTRL_IX_3 0x00000010
+#define HAL_REO1_MISC_CTL 0x00000630
#define HAL_REO1_RING_BASE_LSB(ab) ab->hw_params.regs->hal_reo1_ring_base_lsb
#define HAL_REO1_RING_BASE_MSB(ab) ab->hw_params.regs->hal_reo1_ring_base_msb
#define HAL_REO1_RING_ID(ab) ab->hw_params.regs->hal_reo1_ring_id
#define HAL_REO1_GEN_ENABLE_FRAG_DST_RING GENMASK(25, 23)
#define HAL_REO1_GEN_ENABLE_AGING_LIST_ENABLE BIT(2)
#define HAL_REO1_GEN_ENABLE_AGING_FLUSH_ENABLE BIT(3)
+#define HAL_REO1_MISC_CTL_FRAGMENT_DST_RING GENMASK(20, 17)
/* CE ring bit field mask and shift */
#define HAL_CE_DST_R0_DEST_CTRL_MAX_LEN GENMASK(15, 0)
u32 start_seq, enum hal_pn_type type);
void ath11k_hal_reo_init_cmd_ring(struct ath11k_base *ab,
struct hal_srng *srng);
-void ath11k_hal_reo_hw_setup(struct ath11k_base *ab, u32 ring_hash_map);
void ath11k_hal_setup_link_idle_list(struct ath11k_base *ab,
struct hal_wbm_idle_scatter_list *sbuf,
u32 nsbufs, u32 tot_link_desc,
}
}
-void ath11k_hal_reo_hw_setup(struct ath11k_base *ab, u32 ring_hash_map)
-{
- u32 reo_base = HAL_SEQ_WCSS_UMAC_REO_REG;
- u32 val;
-
- val = ath11k_hif_read32(ab, reo_base + HAL_REO1_GEN_ENABLE);
-
- val &= ~HAL_REO1_GEN_ENABLE_FRAG_DST_RING;
- val |= FIELD_PREP(HAL_REO1_GEN_ENABLE_FRAG_DST_RING,
- HAL_SRNG_RING_ID_REO2SW1) |
- FIELD_PREP(HAL_REO1_GEN_ENABLE_AGING_LIST_ENABLE, 1) |
- FIELD_PREP(HAL_REO1_GEN_ENABLE_AGING_FLUSH_ENABLE, 1);
- ath11k_hif_write32(ab, reo_base + HAL_REO1_GEN_ENABLE, val);
-
- ath11k_hif_write32(ab, reo_base + HAL_REO1_AGING_THRESH_IX_0(ab),
- HAL_DEFAULT_REO_TIMEOUT_USEC);
- ath11k_hif_write32(ab, reo_base + HAL_REO1_AGING_THRESH_IX_1(ab),
- HAL_DEFAULT_REO_TIMEOUT_USEC);
- ath11k_hif_write32(ab, reo_base + HAL_REO1_AGING_THRESH_IX_2(ab),
- HAL_DEFAULT_REO_TIMEOUT_USEC);
- ath11k_hif_write32(ab, reo_base + HAL_REO1_AGING_THRESH_IX_3(ab),
- HAL_DEFAULT_REO_TIMEOUT_USEC);
-
- ath11k_hif_write32(ab, reo_base + HAL_REO1_DEST_RING_CTRL_IX_0,
- FIELD_PREP(HAL_REO_DEST_RING_CTRL_HASH_RING_MAP,
- ring_hash_map));
- ath11k_hif_write32(ab, reo_base + HAL_REO1_DEST_RING_CTRL_IX_1,
- FIELD_PREP(HAL_REO_DEST_RING_CTRL_HASH_RING_MAP,
- ring_hash_map));
- ath11k_hif_write32(ab, reo_base + HAL_REO1_DEST_RING_CTRL_IX_2,
- FIELD_PREP(HAL_REO_DEST_RING_CTRL_HASH_RING_MAP,
- ring_hash_map));
- ath11k_hif_write32(ab, reo_base + HAL_REO1_DEST_RING_CTRL_IX_3,
- FIELD_PREP(HAL_REO_DEST_RING_CTRL_HASH_RING_MAP,
- ring_hash_map));
-}
-
static enum hal_rx_mon_status
ath11k_hal_rx_parse_mon_status_tlv(struct ath11k_base *ab,
struct hal_rx_mon_ppdu_info *ppdu_info,
break;
}
case HAL_RX_MPDU_START: {
- struct hal_rx_mpdu_info *mpdu_info =
- (struct hal_rx_mpdu_info *)tlv_data;
u16 peer_id;
- peer_id = FIELD_GET(HAL_RX_MPDU_INFO_INFO0_PEERID,
- __le32_to_cpu(mpdu_info->info0));
+ peer_id = ab->hw_params.hw_ops->mpdu_info_get_peerid(tlv_data);
if (peer_id)
ppdu_info->peer_id = peer_id;
break;
} __packed;
#define HAL_RX_MPDU_INFO_INFO0_PEERID GENMASK(31, 16)
+#define HAL_RX_MPDU_INFO_INFO0_PEERID_WCN6855 GENMASK(15, 0)
+
struct hal_rx_mpdu_info {
__le32 rsvd0;
__le32 info0;
__le32 rsvd1[21];
} __packed;
+struct hal_rx_mpdu_info_wcn6855 {
+ __le32 rsvd0[8];
+ __le32 info0;
+ __le32 rsvd1[14];
+} __packed;
+
#define HAL_RX_PPDU_END_DURATION GENMASK(23, 0)
struct hal_rx_ppdu_end_duration {
__le32 rsvd0[9];
#include "hw.h"
#include "core.h"
#include "ce.h"
+#include "hif.h"
/* Map from pdev index to hw mac index */
static u8 ath11k_hw_ipq8074_mac_from_pdev_id(int pdev_idx)
true);
}
+static void ath11k_hw_wcn6855_tx_mesh_enable(struct ath11k_base *ab,
+ struct hal_tcl_data_cmd *tcl_cmd)
+{
+ tcl_cmd->info3 |= FIELD_PREP(HAL_QCN9074_TCL_DATA_CMD_INFO3_MESH_ENABLE,
+ true);
+}
+
static void ath11k_init_wmi_config_qca6390(struct ath11k_base *ab,
struct target_resource_config *config)
{
config->num_keep_alive_pattern = 0;
}
+static void ath11k_hw_ipq8074_reo_setup(struct ath11k_base *ab)
+{
+ u32 reo_base = HAL_SEQ_WCSS_UMAC_REO_REG;
+ u32 val;
+ /* Each hash entry uses three bits to map to a particular ring. */
+ u32 ring_hash_map = HAL_HASH_ROUTING_RING_SW1 << 0 |
+ HAL_HASH_ROUTING_RING_SW2 << 3 |
+ HAL_HASH_ROUTING_RING_SW3 << 6 |
+ HAL_HASH_ROUTING_RING_SW4 << 9 |
+ HAL_HASH_ROUTING_RING_SW1 << 12 |
+ HAL_HASH_ROUTING_RING_SW2 << 15 |
+ HAL_HASH_ROUTING_RING_SW3 << 18 |
+ HAL_HASH_ROUTING_RING_SW4 << 21;
+
+ val = ath11k_hif_read32(ab, reo_base + HAL_REO1_GEN_ENABLE);
+
+ val &= ~HAL_REO1_GEN_ENABLE_FRAG_DST_RING;
+ val |= FIELD_PREP(HAL_REO1_GEN_ENABLE_FRAG_DST_RING,
+ HAL_SRNG_RING_ID_REO2SW1) |
+ FIELD_PREP(HAL_REO1_GEN_ENABLE_AGING_LIST_ENABLE, 1) |
+ FIELD_PREP(HAL_REO1_GEN_ENABLE_AGING_FLUSH_ENABLE, 1);
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_GEN_ENABLE, val);
+
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_AGING_THRESH_IX_0(ab),
+ HAL_DEFAULT_REO_TIMEOUT_USEC);
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_AGING_THRESH_IX_1(ab),
+ HAL_DEFAULT_REO_TIMEOUT_USEC);
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_AGING_THRESH_IX_2(ab),
+ HAL_DEFAULT_REO_TIMEOUT_USEC);
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_AGING_THRESH_IX_3(ab),
+ HAL_DEFAULT_REO_TIMEOUT_USEC);
+
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_DEST_RING_CTRL_IX_0,
+ FIELD_PREP(HAL_REO_DEST_RING_CTRL_HASH_RING_MAP,
+ ring_hash_map));
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_DEST_RING_CTRL_IX_1,
+ FIELD_PREP(HAL_REO_DEST_RING_CTRL_HASH_RING_MAP,
+ ring_hash_map));
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_DEST_RING_CTRL_IX_2,
+ FIELD_PREP(HAL_REO_DEST_RING_CTRL_HASH_RING_MAP,
+ ring_hash_map));
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_DEST_RING_CTRL_IX_3,
+ FIELD_PREP(HAL_REO_DEST_RING_CTRL_HASH_RING_MAP,
+ ring_hash_map));
+}
+
static void ath11k_init_wmi_config_ipq8074(struct ath11k_base *ab,
struct target_resource_config *config)
{
return &desc->u.qcn9074.msdu_payload[0];
}
+static bool ath11k_hw_wcn6855_rx_desc_get_first_msdu(struct hal_rx_desc *desc)
+{
+ return !!FIELD_GET(RX_MSDU_END_INFO2_FIRST_MSDU_WCN6855,
+ __le32_to_cpu(desc->u.wcn6855.msdu_end.info2));
+}
+
+static bool ath11k_hw_wcn6855_rx_desc_get_last_msdu(struct hal_rx_desc *desc)
+{
+ return !!FIELD_GET(RX_MSDU_END_INFO2_LAST_MSDU_WCN6855,
+ __le32_to_cpu(desc->u.wcn6855.msdu_end.info2));
+}
+
+static u8 ath11k_hw_wcn6855_rx_desc_get_l3_pad_bytes(struct hal_rx_desc *desc)
+{
+ return FIELD_GET(RX_MSDU_END_INFO2_L3_HDR_PADDING,
+ __le32_to_cpu(desc->u.wcn6855.msdu_end.info2));
+}
+
+static u8 *ath11k_hw_wcn6855_rx_desc_get_hdr_status(struct hal_rx_desc *desc)
+{
+ return desc->u.wcn6855.hdr_status;
+}
+
+static bool ath11k_hw_wcn6855_rx_desc_encrypt_valid(struct hal_rx_desc *desc)
+{
+ return __le32_to_cpu(desc->u.wcn6855.mpdu_start.info1) &
+ RX_MPDU_START_INFO1_ENCRYPT_INFO_VALID;
+}
+
+static u32 ath11k_hw_wcn6855_rx_desc_get_encrypt_type(struct hal_rx_desc *desc)
+{
+ return FIELD_GET(RX_MPDU_START_INFO2_ENC_TYPE,
+ __le32_to_cpu(desc->u.wcn6855.mpdu_start.info2));
+}
+
+static u8 ath11k_hw_wcn6855_rx_desc_get_decap_type(struct hal_rx_desc *desc)
+{
+ return FIELD_GET(RX_MSDU_START_INFO2_DECAP_FORMAT,
+ __le32_to_cpu(desc->u.wcn6855.msdu_start.info2));
+}
+
+static u8 ath11k_hw_wcn6855_rx_desc_get_mesh_ctl(struct hal_rx_desc *desc)
+{
+ return FIELD_GET(RX_MSDU_START_INFO2_MESH_CTRL_PRESENT,
+ __le32_to_cpu(desc->u.wcn6855.msdu_start.info2));
+}
+
+static bool ath11k_hw_wcn6855_rx_desc_get_mpdu_seq_ctl_vld(struct hal_rx_desc *desc)
+{
+ return !!FIELD_GET(RX_MPDU_START_INFO1_MPDU_SEQ_CTRL_VALID,
+ __le32_to_cpu(desc->u.wcn6855.mpdu_start.info1));
+}
+
+static bool ath11k_hw_wcn6855_rx_desc_get_mpdu_fc_valid(struct hal_rx_desc *desc)
+{
+ return !!FIELD_GET(RX_MPDU_START_INFO1_MPDU_FCTRL_VALID,
+ __le32_to_cpu(desc->u.wcn6855.mpdu_start.info1));
+}
+
+static u16 ath11k_hw_wcn6855_rx_desc_get_mpdu_start_seq_no(struct hal_rx_desc *desc)
+{
+ return FIELD_GET(RX_MPDU_START_INFO1_MPDU_SEQ_NUM,
+ __le32_to_cpu(desc->u.wcn6855.mpdu_start.info1));
+}
+
+static u16 ath11k_hw_wcn6855_rx_desc_get_msdu_len(struct hal_rx_desc *desc)
+{
+ return FIELD_GET(RX_MSDU_START_INFO1_MSDU_LENGTH,
+ __le32_to_cpu(desc->u.wcn6855.msdu_start.info1));
+}
+
+static u8 ath11k_hw_wcn6855_rx_desc_get_msdu_sgi(struct hal_rx_desc *desc)
+{
+ return FIELD_GET(RX_MSDU_START_INFO3_SGI,
+ __le32_to_cpu(desc->u.wcn6855.msdu_start.info3));
+}
+
+static u8 ath11k_hw_wcn6855_rx_desc_get_msdu_rate_mcs(struct hal_rx_desc *desc)
+{
+ return FIELD_GET(RX_MSDU_START_INFO3_RATE_MCS,
+ __le32_to_cpu(desc->u.wcn6855.msdu_start.info3));
+}
+
+static u8 ath11k_hw_wcn6855_rx_desc_get_msdu_rx_bw(struct hal_rx_desc *desc)
+{
+ return FIELD_GET(RX_MSDU_START_INFO3_RECV_BW,
+ __le32_to_cpu(desc->u.wcn6855.msdu_start.info3));
+}
+
+static u32 ath11k_hw_wcn6855_rx_desc_get_msdu_freq(struct hal_rx_desc *desc)
+{
+ return __le32_to_cpu(desc->u.wcn6855.msdu_start.phy_meta_data);
+}
+
+static u8 ath11k_hw_wcn6855_rx_desc_get_msdu_pkt_type(struct hal_rx_desc *desc)
+{
+ return FIELD_GET(RX_MSDU_START_INFO3_PKT_TYPE,
+ __le32_to_cpu(desc->u.wcn6855.msdu_start.info3));
+}
+
+static u8 ath11k_hw_wcn6855_rx_desc_get_msdu_nss(struct hal_rx_desc *desc)
+{
+ return FIELD_GET(RX_MSDU_START_INFO3_MIMO_SS_BITMAP,
+ __le32_to_cpu(desc->u.wcn6855.msdu_start.info3));
+}
+
+static u8 ath11k_hw_wcn6855_rx_desc_get_mpdu_tid(struct hal_rx_desc *desc)
+{
+ return FIELD_GET(RX_MPDU_START_INFO2_TID_WCN6855,
+ __le32_to_cpu(desc->u.wcn6855.mpdu_start.info2));
+}
+
+static u16 ath11k_hw_wcn6855_rx_desc_get_mpdu_peer_id(struct hal_rx_desc *desc)
+{
+ return __le16_to_cpu(desc->u.wcn6855.mpdu_start.sw_peer_id);
+}
+
+static void ath11k_hw_wcn6855_rx_desc_copy_attn_end(struct hal_rx_desc *fdesc,
+ struct hal_rx_desc *ldesc)
+{
+ memcpy((u8 *)&fdesc->u.wcn6855.msdu_end, (u8 *)&ldesc->u.wcn6855.msdu_end,
+ sizeof(struct rx_msdu_end_wcn6855));
+ memcpy((u8 *)&fdesc->u.wcn6855.attention, (u8 *)&ldesc->u.wcn6855.attention,
+ sizeof(struct rx_attention));
+ memcpy((u8 *)&fdesc->u.wcn6855.mpdu_end, (u8 *)&ldesc->u.wcn6855.mpdu_end,
+ sizeof(struct rx_mpdu_end));
+}
+
+static u32 ath11k_hw_wcn6855_rx_desc_get_mpdu_start_tag(struct hal_rx_desc *desc)
+{
+ return FIELD_GET(HAL_TLV_HDR_TAG,
+ __le32_to_cpu(desc->u.wcn6855.mpdu_start_tag));
+}
+
+static u32 ath11k_hw_wcn6855_rx_desc_get_mpdu_ppdu_id(struct hal_rx_desc *desc)
+{
+ return __le16_to_cpu(desc->u.wcn6855.mpdu_start.phy_ppdu_id);
+}
+
+static void ath11k_hw_wcn6855_rx_desc_set_msdu_len(struct hal_rx_desc *desc, u16 len)
+{
+ u32 info = __le32_to_cpu(desc->u.wcn6855.msdu_start.info1);
+
+ info &= ~RX_MSDU_START_INFO1_MSDU_LENGTH;
+ info |= FIELD_PREP(RX_MSDU_START_INFO1_MSDU_LENGTH, len);
+
+ desc->u.wcn6855.msdu_start.info1 = __cpu_to_le32(info);
+}
+
+static
+struct rx_attention *ath11k_hw_wcn6855_rx_desc_get_attention(struct hal_rx_desc *desc)
+{
+ return &desc->u.wcn6855.attention;
+}
+
+static u8 *ath11k_hw_wcn6855_rx_desc_get_msdu_payload(struct hal_rx_desc *desc)
+{
+ return &desc->u.wcn6855.msdu_payload[0];
+}
+
+static void ath11k_hw_wcn6855_reo_setup(struct ath11k_base *ab)
+{
+ u32 reo_base = HAL_SEQ_WCSS_UMAC_REO_REG;
+ u32 val;
+ /* Each hash entry uses four bits to map to a particular ring. */
+ u32 ring_hash_map = HAL_HASH_ROUTING_RING_SW1 << 0 |
+ HAL_HASH_ROUTING_RING_SW2 << 4 |
+ HAL_HASH_ROUTING_RING_SW3 << 8 |
+ HAL_HASH_ROUTING_RING_SW4 << 12 |
+ HAL_HASH_ROUTING_RING_SW1 << 16 |
+ HAL_HASH_ROUTING_RING_SW2 << 20 |
+ HAL_HASH_ROUTING_RING_SW3 << 24 |
+ HAL_HASH_ROUTING_RING_SW4 << 28;
+
+ val = ath11k_hif_read32(ab, reo_base + HAL_REO1_GEN_ENABLE);
+ val |= FIELD_PREP(HAL_REO1_GEN_ENABLE_AGING_LIST_ENABLE, 1) |
+ FIELD_PREP(HAL_REO1_GEN_ENABLE_AGING_FLUSH_ENABLE, 1);
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_GEN_ENABLE, val);
+
+ val = ath11k_hif_read32(ab, reo_base + HAL_REO1_MISC_CTL);
+ val &= ~HAL_REO1_MISC_CTL_FRAGMENT_DST_RING;
+ val |= FIELD_PREP(HAL_REO1_MISC_CTL_FRAGMENT_DST_RING, HAL_SRNG_RING_ID_REO2SW1);
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_MISC_CTL, val);
+
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_AGING_THRESH_IX_0(ab),
+ HAL_DEFAULT_REO_TIMEOUT_USEC);
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_AGING_THRESH_IX_1(ab),
+ HAL_DEFAULT_REO_TIMEOUT_USEC);
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_AGING_THRESH_IX_2(ab),
+ HAL_DEFAULT_REO_TIMEOUT_USEC);
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_AGING_THRESH_IX_3(ab),
+ HAL_DEFAULT_REO_TIMEOUT_USEC);
+
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_DEST_RING_CTRL_IX_2,
+ ring_hash_map);
+ ath11k_hif_write32(ab, reo_base + HAL_REO1_DEST_RING_CTRL_IX_3,
+ ring_hash_map);
+}
+
+static u16 ath11k_hw_ipq8074_mpdu_info_get_peerid(u8 *tlv_data)
+{
+ u16 peer_id = 0;
+ struct hal_rx_mpdu_info *mpdu_info =
+ (struct hal_rx_mpdu_info *)tlv_data;
+
+ peer_id = FIELD_GET(HAL_RX_MPDU_INFO_INFO0_PEERID,
+ __le32_to_cpu(mpdu_info->info0));
+
+ return peer_id;
+}
+
+static u16 ath11k_hw_wcn6855_mpdu_info_get_peerid(u8 *tlv_data)
+{
+ u16 peer_id = 0;
+ struct hal_rx_mpdu_info_wcn6855 *mpdu_info =
+ (struct hal_rx_mpdu_info_wcn6855 *)tlv_data;
+
+ peer_id = FIELD_GET(HAL_RX_MPDU_INFO_INFO0_PEERID_WCN6855,
+ __le32_to_cpu(mpdu_info->info0));
+ return peer_id;
+}
+
const struct ath11k_hw_ops ipq8074_ops = {
.get_hw_mac_from_pdev_id = ath11k_hw_ipq8074_mac_from_pdev_id,
.wmi_init_config = ath11k_init_wmi_config_ipq8074,
.rx_desc_set_msdu_len = ath11k_hw_ipq8074_rx_desc_set_msdu_len,
.rx_desc_get_attention = ath11k_hw_ipq8074_rx_desc_get_attention,
.rx_desc_get_msdu_payload = ath11k_hw_ipq8074_rx_desc_get_msdu_payload,
+ .reo_setup = ath11k_hw_ipq8074_reo_setup,
+ .mpdu_info_get_peerid = ath11k_hw_ipq8074_mpdu_info_get_peerid,
};
const struct ath11k_hw_ops ipq6018_ops = {
.rx_desc_set_msdu_len = ath11k_hw_ipq8074_rx_desc_set_msdu_len,
.rx_desc_get_attention = ath11k_hw_ipq8074_rx_desc_get_attention,
.rx_desc_get_msdu_payload = ath11k_hw_ipq8074_rx_desc_get_msdu_payload,
+ .reo_setup = ath11k_hw_ipq8074_reo_setup,
+ .mpdu_info_get_peerid = ath11k_hw_ipq8074_mpdu_info_get_peerid,
};
const struct ath11k_hw_ops qca6390_ops = {
.rx_desc_set_msdu_len = ath11k_hw_ipq8074_rx_desc_set_msdu_len,
.rx_desc_get_attention = ath11k_hw_ipq8074_rx_desc_get_attention,
.rx_desc_get_msdu_payload = ath11k_hw_ipq8074_rx_desc_get_msdu_payload,
+ .reo_setup = ath11k_hw_ipq8074_reo_setup,
+ .mpdu_info_get_peerid = ath11k_hw_ipq8074_mpdu_info_get_peerid,
};
const struct ath11k_hw_ops qcn9074_ops = {
.rx_desc_set_msdu_len = ath11k_hw_qcn9074_rx_desc_set_msdu_len,
.rx_desc_get_attention = ath11k_hw_qcn9074_rx_desc_get_attention,
.rx_desc_get_msdu_payload = ath11k_hw_qcn9074_rx_desc_get_msdu_payload,
+ .reo_setup = ath11k_hw_ipq8074_reo_setup,
+ .mpdu_info_get_peerid = ath11k_hw_ipq8074_mpdu_info_get_peerid,
+};
+
+const struct ath11k_hw_ops wcn6855_ops = {
+ .get_hw_mac_from_pdev_id = ath11k_hw_ipq8074_mac_from_pdev_id,
+ .wmi_init_config = ath11k_init_wmi_config_qca6390,
+ .mac_id_to_pdev_id = ath11k_hw_mac_id_to_pdev_id_qca6390,
+ .mac_id_to_srng_id = ath11k_hw_mac_id_to_srng_id_qca6390,
+ .tx_mesh_enable = ath11k_hw_wcn6855_tx_mesh_enable,
+ .rx_desc_get_first_msdu = ath11k_hw_wcn6855_rx_desc_get_first_msdu,
+ .rx_desc_get_last_msdu = ath11k_hw_wcn6855_rx_desc_get_last_msdu,
+ .rx_desc_get_l3_pad_bytes = ath11k_hw_wcn6855_rx_desc_get_l3_pad_bytes,
+ .rx_desc_get_hdr_status = ath11k_hw_wcn6855_rx_desc_get_hdr_status,
+ .rx_desc_encrypt_valid = ath11k_hw_wcn6855_rx_desc_encrypt_valid,
+ .rx_desc_get_encrypt_type = ath11k_hw_wcn6855_rx_desc_get_encrypt_type,
+ .rx_desc_get_decap_type = ath11k_hw_wcn6855_rx_desc_get_decap_type,
+ .rx_desc_get_mesh_ctl = ath11k_hw_wcn6855_rx_desc_get_mesh_ctl,
+ .rx_desc_get_mpdu_seq_ctl_vld = ath11k_hw_wcn6855_rx_desc_get_mpdu_seq_ctl_vld,
+ .rx_desc_get_mpdu_fc_valid = ath11k_hw_wcn6855_rx_desc_get_mpdu_fc_valid,
+ .rx_desc_get_mpdu_start_seq_no = ath11k_hw_wcn6855_rx_desc_get_mpdu_start_seq_no,
+ .rx_desc_get_msdu_len = ath11k_hw_wcn6855_rx_desc_get_msdu_len,
+ .rx_desc_get_msdu_sgi = ath11k_hw_wcn6855_rx_desc_get_msdu_sgi,
+ .rx_desc_get_msdu_rate_mcs = ath11k_hw_wcn6855_rx_desc_get_msdu_rate_mcs,
+ .rx_desc_get_msdu_rx_bw = ath11k_hw_wcn6855_rx_desc_get_msdu_rx_bw,
+ .rx_desc_get_msdu_freq = ath11k_hw_wcn6855_rx_desc_get_msdu_freq,
+ .rx_desc_get_msdu_pkt_type = ath11k_hw_wcn6855_rx_desc_get_msdu_pkt_type,
+ .rx_desc_get_msdu_nss = ath11k_hw_wcn6855_rx_desc_get_msdu_nss,
+ .rx_desc_get_mpdu_tid = ath11k_hw_wcn6855_rx_desc_get_mpdu_tid,
+ .rx_desc_get_mpdu_peer_id = ath11k_hw_wcn6855_rx_desc_get_mpdu_peer_id,
+ .rx_desc_copy_attn_end_tlv = ath11k_hw_wcn6855_rx_desc_copy_attn_end,
+ .rx_desc_get_mpdu_start_tag = ath11k_hw_wcn6855_rx_desc_get_mpdu_start_tag,
+ .rx_desc_get_mpdu_ppdu_id = ath11k_hw_wcn6855_rx_desc_get_mpdu_ppdu_id,
+ .rx_desc_set_msdu_len = ath11k_hw_wcn6855_rx_desc_set_msdu_len,
+ .rx_desc_get_attention = ath11k_hw_wcn6855_rx_desc_get_attention,
+ .rx_desc_get_msdu_payload = ath11k_hw_wcn6855_rx_desc_get_msdu_payload,
+ .reo_setup = ath11k_hw_wcn6855_reo_setup,
+ .mpdu_info_get_peerid = ath11k_hw_wcn6855_mpdu_info_get_peerid,
};
#define ATH11K_TX_RING_MASK_0 0x1
.pcie_qserdes_sysclk_en_sel = 0x01e0e0a8,
.pcie_pcs_osc_dtct_config_base = 0x01e0f45c,
};
+
+const struct ath11k_hw_regs wcn6855_regs = {
+ /* SW2TCL(x) R0 ring configuration address */
+ .hal_tcl1_ring_base_lsb = 0x00000690,
+ .hal_tcl1_ring_base_msb = 0x00000694,
+ .hal_tcl1_ring_id = 0x00000698,
+ .hal_tcl1_ring_misc = 0x000006a0,
+ .hal_tcl1_ring_tp_addr_lsb = 0x000006ac,
+ .hal_tcl1_ring_tp_addr_msb = 0x000006b0,
+ .hal_tcl1_ring_consumer_int_setup_ix0 = 0x000006c0,
+ .hal_tcl1_ring_consumer_int_setup_ix1 = 0x000006c4,
+ .hal_tcl1_ring_msi1_base_lsb = 0x000006d8,
+ .hal_tcl1_ring_msi1_base_msb = 0x000006dc,
+ .hal_tcl1_ring_msi1_data = 0x000006e0,
+ .hal_tcl2_ring_base_lsb = 0x000006e8,
+ .hal_tcl_ring_base_lsb = 0x00000798,
+
+ /* TCL STATUS ring address */
+ .hal_tcl_status_ring_base_lsb = 0x000008a0,
+
+ /* REO2SW(x) R0 ring configuration address */
+ .hal_reo1_ring_base_lsb = 0x00000244,
+ .hal_reo1_ring_base_msb = 0x00000248,
+ .hal_reo1_ring_id = 0x0000024c,
+ .hal_reo1_ring_misc = 0x00000254,
+ .hal_reo1_ring_hp_addr_lsb = 0x00000258,
+ .hal_reo1_ring_hp_addr_msb = 0x0000025c,
+ .hal_reo1_ring_producer_int_setup = 0x00000268,
+ .hal_reo1_ring_msi1_base_lsb = 0x0000028c,
+ .hal_reo1_ring_msi1_base_msb = 0x00000290,
+ .hal_reo1_ring_msi1_data = 0x00000294,
+ .hal_reo2_ring_base_lsb = 0x0000029c,
+ .hal_reo1_aging_thresh_ix_0 = 0x000005bc,
+ .hal_reo1_aging_thresh_ix_1 = 0x000005c0,
+ .hal_reo1_aging_thresh_ix_2 = 0x000005c4,
+ .hal_reo1_aging_thresh_ix_3 = 0x000005c8,
+
+ /* REO2SW(x) R2 ring pointers (head/tail) address */
+ .hal_reo1_ring_hp = 0x00003030,
+ .hal_reo1_ring_tp = 0x00003034,
+ .hal_reo2_ring_hp = 0x00003038,
+
+ /* REO2TCL R0 ring configuration address */
+ .hal_reo_tcl_ring_base_lsb = 0x00000454,
+ .hal_reo_tcl_ring_hp = 0x00003060,
+
+ /* REO status address */
+ .hal_reo_status_ring_base_lsb = 0x0000055c,
+ .hal_reo_status_hp = 0x00003078,
+
+ /* WCSS relative address */
+ .hal_seq_wcss_umac_ce0_src_reg = 0x1b80000,
+ .hal_seq_wcss_umac_ce0_dst_reg = 0x1b81000,
+ .hal_seq_wcss_umac_ce1_src_reg = 0x1b82000,
+ .hal_seq_wcss_umac_ce1_dst_reg = 0x1b83000,
+
+ /* WBM Idle address */
+ .hal_wbm_idle_link_ring_base_lsb = 0x00000870,
+ .hal_wbm_idle_link_ring_misc = 0x00000880,
+
+ /* SW2WBM release address */
+ .hal_wbm_release_ring_base_lsb = 0x000001e8,
+
+ /* WBM2SW release address */
+ .hal_wbm0_release_ring_base_lsb = 0x00000920,
+ .hal_wbm1_release_ring_base_lsb = 0x00000978,
+
+ /* PCIe base address */
+ .pcie_qserdes_sysclk_en_sel = 0x01e0c0ac,
+ .pcie_pcs_osc_dtct_config_base = 0x01e0c628,
+};
bool cold_boot_calib;
bool supports_suspend;
u32 hal_desc_sz;
+ bool fix_l1ss;
};
struct ath11k_hw_ops {
void (*rx_desc_set_msdu_len)(struct hal_rx_desc *desc, u16 len);
struct rx_attention *(*rx_desc_get_attention)(struct hal_rx_desc *desc);
u8 *(*rx_desc_get_msdu_payload)(struct hal_rx_desc *desc);
+ void (*reo_setup)(struct ath11k_base *ab);
+ u16 (*mpdu_info_get_peerid)(u8 *tlv_data);
};
extern const struct ath11k_hw_ops ipq8074_ops;
extern const struct ath11k_hw_ops ipq6018_ops;
extern const struct ath11k_hw_ops qca6390_ops;
extern const struct ath11k_hw_ops qcn9074_ops;
+extern const struct ath11k_hw_ops wcn6855_ops;
extern const struct ath11k_hw_ring_mask ath11k_hw_ring_mask_ipq8074;
extern const struct ath11k_hw_ring_mask ath11k_hw_ring_mask_qca6390;
extern const struct ath11k_hw_regs ipq8074_regs;
extern const struct ath11k_hw_regs qca6390_regs;
extern const struct ath11k_hw_regs qcn9074_regs;
+extern const struct ath11k_hw_regs wcn6855_regs;
#endif
arg->he_flag = true;
- memcpy(&arg->peer_he_cap_macinfo, he_cap->he_cap_elem.mac_cap_info,
- sizeof(arg->peer_he_cap_macinfo));
- memcpy(&arg->peer_he_cap_phyinfo, he_cap->he_cap_elem.phy_cap_info,
- sizeof(arg->peer_he_cap_phyinfo));
+ memcpy_and_pad(&arg->peer_he_cap_macinfo,
+ sizeof(arg->peer_he_cap_macinfo),
+ he_cap->he_cap_elem.mac_cap_info,
+ sizeof(he_cap->he_cap_elem.mac_cap_info),
+ 0);
+ memcpy_and_pad(&arg->peer_he_cap_phyinfo,
+ sizeof(arg->peer_he_cap_phyinfo),
+ he_cap->he_cap_elem.phy_cap_info,
+ sizeof(he_cap->he_cap_elem.phy_cap_info),
+ 0);
arg->peer_he_ops = vif->bss_conf.he_oper.params;
/* the top most byte is used to indicate BSS color info */
*/
spin_lock_bh(&ab->base_lock);
peer = ath11k_peer_find(ab, arvif->vdev_id, peer_addr);
+
+ /* flush the fragments cache during key (re)install to
+ * ensure all frags in the new frag list belong to the same key.
+ */
+ if (peer && cmd == SET_KEY)
+ ath11k_peer_frags_flush(ar, peer);
spin_unlock_bh(&ab->base_lock);
if (!peer) {
if (WARN_ON(!arvif->is_up))
continue;
- ret = ath11k_mac_setup_bcn_tmpl(arvif);
- if (ret)
- ath11k_warn(ab, "failed to update bcn tmpl during csa: %d\n",
- ret);
-
ret = ath11k_mac_vdev_restart(arvif, &vifs[i].new_ctx->def);
if (ret) {
ath11k_warn(ab, "failed to restart vdev %d: %d\n",
continue;
}
+ ret = ath11k_mac_setup_bcn_tmpl(arvif);
+ if (ret)
+ ath11k_warn(ab, "failed to update bcn tmpl during csa: %d\n",
+ ret);
+
ret = ath11k_wmi_vdev_up(arvif->ar, arvif->vdev_id, arvif->aid,
arvif->bssid);
if (ret) {
ath11k_mhi_config = &ath11k_mhi_config_qcn9074;
break;
case ATH11K_HW_QCA6390_HW20:
+ case ATH11K_HW_WCN6855_HW20:
ath11k_mhi_config = &ath11k_mhi_config_qca6390;
break;
default:
#define QCA6390_DEVICE_ID 0x1101
#define QCN9074_DEVICE_ID 0x1104
+#define WCN6855_DEVICE_ID 0x1103
static const struct pci_device_id ath11k_pci_id_table[] = {
{ PCI_VDEVICE(QCOM, QCA6390_DEVICE_ID) },
- /* TODO: add QCN9074_DEVICE_ID) once firmware issues are resolved */
+ { PCI_VDEVICE(QCOM, WCN6855_DEVICE_ID) },
+ { PCI_VDEVICE(QCOM, QCN9074_DEVICE_ID) },
{0}
};
ath11k_pci_enable_ltssm(ab);
ath11k_pci_clear_all_intrs(ab);
ath11k_pci_set_wlaon_pwr_ctrl(ab);
- ath11k_pci_fix_l1ss(ab);
+ if (ab->hw_params.fix_l1ss)
+ ath11k_pci_fix_l1ss(ab);
}
ath11k_mhi_clear_vector(ab);
.get_ce_msi_idx = ath11k_pci_get_ce_msi_idx,
};
+static void ath11k_pci_read_hw_version(struct ath11k_base *ab, u32 *major, u32 *minor)
+{
+ u32 soc_hw_version;
+
+ soc_hw_version = ath11k_pci_read32(ab, TCSR_SOC_HW_VERSION);
+ *major = FIELD_GET(TCSR_SOC_HW_VERSION_MAJOR_MASK,
+ soc_hw_version);
+ *minor = FIELD_GET(TCSR_SOC_HW_VERSION_MINOR_MASK,
+ soc_hw_version);
+
+ ath11k_dbg(ab, ATH11K_DBG_PCI, "pci tcsr_soc_hw_version major %d minor %d\n",
+ *major, *minor);
+}
+
static int ath11k_pci_probe(struct pci_dev *pdev,
const struct pci_device_id *pci_dev)
{
struct ath11k_base *ab;
struct ath11k_pci *ab_pci;
- u32 soc_hw_version, soc_hw_version_major, soc_hw_version_minor;
+ u32 soc_hw_version_major, soc_hw_version_minor;
int ret;
ab = ath11k_core_alloc(&pdev->dev, sizeof(*ab_pci), ATH11K_BUS_PCI,
switch (pci_dev->device) {
case QCA6390_DEVICE_ID:
- soc_hw_version = ath11k_pci_read32(ab, TCSR_SOC_HW_VERSION);
- soc_hw_version_major = FIELD_GET(TCSR_SOC_HW_VERSION_MAJOR_MASK,
- soc_hw_version);
- soc_hw_version_minor = FIELD_GET(TCSR_SOC_HW_VERSION_MINOR_MASK,
- soc_hw_version);
-
- ath11k_dbg(ab, ATH11K_DBG_PCI, "pci tcsr_soc_hw_version major %d minor %d\n",
- soc_hw_version_major, soc_hw_version_minor);
-
+ ath11k_pci_read_hw_version(ab, &soc_hw_version_major,
+ &soc_hw_version_minor);
switch (soc_hw_version_major) {
case 2:
ab->hw_rev = ATH11K_HW_QCA6390_HW20;
ab->bus_params.static_window_map = true;
ab->hw_rev = ATH11K_HW_QCN9074_HW10;
break;
+ case WCN6855_DEVICE_ID:
+ ath11k_pci_read_hw_version(ab, &soc_hw_version_major,
+ &soc_hw_version_minor);
+ switch (soc_hw_version_major) {
+ case 2:
+ ab->hw_rev = ATH11K_HW_WCN6855_HW20;
+ break;
+ default:
+ dev_err(&pdev->dev, "Unsupported WCN6855 SOC hardware version: %d %d\n",
+ soc_hw_version_major, soc_hw_version_minor);
+ ret = -EOPNOTSUPP;
+ goto err_pci_free_region;
+ }
+ ab_pci->msi_config = &ath11k_msi_config[0];
+ break;
default:
dev_err(&pdev->dev, "Unknown PCI device found: 0x%x\n",
pci_dev->device);
#define RX_MPDU_START_INFO2_BSSID_HIT BIT(9)
#define RX_MPDU_START_INFO2_BSSID_NUM GENMASK(13, 10)
#define RX_MPDU_START_INFO2_TID GENMASK(17, 14)
+#define RX_MPDU_START_INFO2_TID_WCN6855 GENMASK(18, 15)
#define RX_MPDU_START_INFO3_REO_DEST_IND GENMASK(4, 0)
#define RX_MPDU_START_INFO3_FLOW_ID_TOEPLITZ BIT(7)
__le32 ht_ctrl;
} __packed;
+struct rx_mpdu_start_wcn6855 {
+ __le32 info3;
+ __le32 reo_queue_desc_lo;
+ __le32 info4;
+ __le32 pn[4];
+ __le32 info2;
+ __le32 peer_meta_data;
+ __le16 info0;
+ __le16 phy_ppdu_id;
+ __le16 ast_index;
+ __le16 sw_peer_id;
+ __le32 info1;
+ __le32 info5;
+ __le32 info6;
+ __le16 frame_ctrl;
+ __le16 duration;
+ u8 addr1[ETH_ALEN];
+ u8 addr2[ETH_ALEN];
+ u8 addr3[ETH_ALEN];
+ __le16 seq_ctrl;
+ u8 addr4[ETH_ALEN];
+ __le16 qos_ctrl;
+ __le32 ht_ctrl;
+} __packed;
+
/* rx_mpdu_start
*
* rxpcu_mpdu_filter_in_category
__le16 vlan_stag_c1;
} __packed;
+struct rx_msdu_start_wcn6855 {
+ __le16 info0;
+ __le16 phy_ppdu_id;
+ __le32 info1;
+ __le32 info2;
+ __le32 toeplitz_hash;
+ __le32 flow_id_toeplitz;
+ __le32 info3;
+ __le32 ppdu_start_timestamp;
+ __le32 phy_meta_data;
+ __le16 vlan_ctag_ci;
+ __le16 vlan_stag_ci;
+} __packed;
+
/* rx_msdu_start
*
* rxpcu_mpdu_filter_in_category
#define RX_MSDU_END_INFO2_REPORTED_MPDU_LEN GENMASK(13, 0)
#define RX_MSDU_END_INFO2_FIRST_MSDU BIT(14)
+#define RX_MSDU_END_INFO2_FIRST_MSDU_WCN6855 BIT(28)
#define RX_MSDU_END_INFO2_LAST_MSDU BIT(15)
+#define RX_MSDU_END_INFO2_LAST_MSDU_WCN6855 BIT(29)
#define RX_MSDU_END_INFO2_SA_IDX_TIMEOUT BIT(16)
#define RX_MSDU_END_INFO2_DA_IDX_TIMEOUT BIT(17)
#define RX_MSDU_END_INFO2_MSDU_LIMIT_ERR BIT(18)
__le16 sa_sw_peer_id;
} __packed;
+struct rx_msdu_end_wcn6855 {
+ __le16 info0;
+ __le16 phy_ppdu_id;
+ __le16 ip_hdr_cksum;
+ __le16 reported_mpdu_len;
+ __le32 info1;
+ __le32 ext_wapi_pn[2];
+ __le32 info4;
+ __le32 ipv6_options_crc;
+ __le32 tcp_seq_num;
+ __le32 tcp_ack_num;
+ __le16 info3;
+ __le16 window_size;
+ __le32 info2;
+ __le16 sa_idx;
+ __le16 da_idx;
+ __le32 info5;
+ __le32 fse_metadata;
+ __le16 cce_metadata;
+ __le16 sa_sw_peer_id;
+ __le32 rule_indication[2];
+ __le32 info6;
+ __le32 info7;
+} __packed;
+
#define RX_MSDU_END_MPDU_LENGTH_INFO GENMASK(13, 0)
#define RX_MSDU_END_INFO2_DA_OFFSET GENMASK(5, 0)
u8 msdu_payload[0];
} __packed;
+struct hal_rx_desc_wcn6855 {
+ __le32 msdu_end_tag;
+ struct rx_msdu_end_wcn6855 msdu_end;
+ __le32 rx_attn_tag;
+ struct rx_attention attention;
+ __le32 msdu_start_tag;
+ struct rx_msdu_start_wcn6855 msdu_start;
+ u8 rx_padding0[HAL_RX_DESC_PADDING0_BYTES];
+ __le32 mpdu_start_tag;
+ struct rx_mpdu_start_wcn6855 mpdu_start;
+ __le32 mpdu_end_tag;
+ struct rx_mpdu_end mpdu_end;
+ u8 rx_padding1[HAL_RX_DESC_PADDING1_BYTES];
+ __le32 hdr_status_tag;
+ __le32 phy_ppdu_id;
+ u8 hdr_status[HAL_RX_DESC_HDR_STATUS_LEN];
+ u8 msdu_payload[0];
+} __packed;
+
struct hal_rx_desc {
union {
struct hal_rx_desc_ipq8074 ipq8074;
struct hal_rx_desc_qcn9074 qcn9074;
+ struct hal_rx_desc_wcn6855 wcn6855;
} u;
} __packed;
dst->hw_reaped = src->hw_reaped;
dst->underrun = src->underrun;
dst->tx_abort = src->tx_abort;
- dst->mpdus_requed = src->mpdus_requed;
+ dst->mpdus_requeued = src->mpdus_requeued;
dst->tx_ko = src->tx_ko;
dst->data_rc = src->data_rc;
dst->self_triggers = src->self_triggers;
len += scnprintf(buf + len, buf_len - len, "%30s %10d\n",
"PPDUs cleaned", pdev->tx_abort);
len += scnprintf(buf + len, buf_len - len, "%30s %10d\n",
- "MPDUs requed", pdev->mpdus_requed);
+ "MPDUs requeued", pdev->mpdus_requeued);
len += scnprintf(buf + len, buf_len - len, "%30s %10u\n",
"Excessive retries", pdev->tx_ko);
len += scnprintf(buf + len, buf_len - len, "%30s %10u\n",
/* Num PPDUs cleaned up in TX abort */
s32 tx_abort;
- /* Num MPDUs requed by SW */
- s32 mpdus_requed;
+ /* Num MPDUs requeued by SW */
+ s32 mpdus_requeued;
/* excessive retries */
u32 tx_ko;
}
/**
- * at5k_hw_stop_rx_pcu() - Stop RX engine
+ * ath5k_hw_stop_rx_pcu() - Stop RX engine
* @ah: The &struct ath5k_hw
*
* Stops RX engine on PCU
if (ret < 0)
return ret;
} else {
- ret = ath6kl_wmi_bssfilter_cmd(ar->wmi, vif->fw_vif_idx,
- MATCHED_SSID_FILTER, 0);
+ ret = ath6kl_wmi_bssfilter_cmd(ar->wmi, vif->fw_vif_idx,
+ MATCHED_SSID_FILTER, 0);
if (ret < 0)
return ret;
}
{
struct ath6kl *ar = file->private_data;
unsigned long lrssi_roam_threshold;
+ int ret;
if (kstrtoul_from_user(user_buf, count, 0, &lrssi_roam_threshold))
return -EINVAL;
ar->lrssi_roam_threshold = lrssi_roam_threshold;
- ath6kl_wmi_set_roam_lrssi_cmd(ar->wmi, ar->lrssi_roam_threshold);
+ ret = ath6kl_wmi_set_roam_lrssi_cmd(ar->wmi, ar->lrssi_roam_threshold);
+ if (ret)
+ return ret;
return count;
}
rxs->rs_moreaggr = (rxsp->status11 & AR_RxMoreAggr) ? 1 : 0;
rxs->rs_antenna = (MS(rxsp->status4, AR_RxAntenna) & 0x7);
rxs->enc_flags |= (rxsp->status4 & AR_GI) ? RX_ENC_FLAG_SHORT_GI : 0;
+ rxs->enc_flags |=
+ (rxsp->status4 & AR_STBC) ? (1 << RX_ENC_FLAG_STBC_SHIFT) : 0;
rxs->bw = (rxsp->status4 & AR_2040) ? RATE_INFO_BW_40 : RATE_INFO_BW_20;
rxs->evm0 = rxsp->status6;
hchan = ah->curchan;
}
+ if (!hchan) {
+ fastcc = false;
+ hchan = ath9k_cmn_get_channel(sc->hw, ah, &sc->cur_chan->chandef);
+ }
+
if (!ath_prepare_reset(sc))
fastcc = false;
static void ath9k_mgd_prepare_tx(struct ieee80211_hw *hw,
struct ieee80211_vif *vif,
- u16 duration)
+ struct ieee80211_prep_tx_info *info)
{
struct ath_softc *sc = hw->priv;
struct ath_common *common = ath9k_hw_common(sc->sc_ah);
config CARL9170_LEDS
bool "SoftLED Support"
- depends on CARL9170
- select MAC80211_LEDS
- select LEDS_CLASS
- select NEW_LEDS
default y
+ depends on CARL9170
+ depends on MAC80211_LEDS
help
- This option is necessary, if you want your device' LEDs to blink
+ This option is necessary, if you want your device's LEDs to blink.
Say Y, unless you need the LEDs for firmware debugging.
#define REG_WRITE(_ah, _reg, _val) (common->ops->write)(_ah, _val, _reg)
/**
- * ath_hw_set_bssid_mask - filter out bssids we listen
+ * ath_hw_setbssidmask - filter out bssids we listen
*
* @common: the ath_common struct for the device.
*
(char *)ctl_skb->skb->data, ctl_skb->skb->len);
/* Move the head of the ring to the next empty descriptor */
- ch->head_blk_ctl = ctl_skb->next;
+ ch->head_blk_ctl = ctl_skb->next;
/* Commit all previous writes and set descriptors to VALID */
wmb();
#define WCN36XX_HAL_OFFLOAD_DISABLE 0
#define WCN36XX_HAL_OFFLOAD_ENABLE 1
#define WCN36XX_HAL_OFFLOAD_BCAST_FILTER_ENABLE 0x2
+#define WCN36XX_HAL_OFFLOAD_MCAST_FILTER_ENABLE 0x4
+#define WCN36XX_HAL_OFFLOAD_NS_AND_MCAST_FILTER_ENABLE \
+ (WCN36XX_HAL_OFFLOAD_ENABLE | WCN36XX_HAL_OFFLOAD_MCAST_FILTER_ENABLE)
#define WCN36XX_HAL_OFFLOAD_ARP_AND_BCAST_FILTER_ENABLE \
- (HAL_OFFLOAD_ENABLE|HAL_OFFLOAD_BCAST_FILTER_ENABLE)
+ (WCN36XX_HAL_OFFLOAD_ENABLE | WCN36XX_HAL_OFFLOAD_BCAST_FILTER_ENABLE)
+#define WCN36XX_HAL_IPV6_OFFLOAD_ADDR_MAX 0x02
struct wcn36xx_hal_ns_offload_params {
u8 src_ipv6_addr[WCN36XX_HAL_IPV6_ADDR_LEN];
/* slot index for this offload */
u32 slot_index;
u8 bss_index;
-};
+} __packed;
struct wcn36xx_hal_host_offload_req {
- u8 offload_Type;
+ u8 offload_type;
/* enable or disable */
u8 enable;
u8 host_ipv4_addr[4];
u8 host_ipv6_addr[WCN36XX_HAL_IPV6_ADDR_LEN];
} u;
-};
+} __packed;
struct wcn36xx_hal_host_offload_req_msg {
struct wcn36xx_hal_msg_header header;
struct wcn36xx_hal_host_offload_req host_offload_params;
struct wcn36xx_hal_ns_offload_params ns_offload_params;
-};
+} __packed;
/* Packet Types. */
#define WCN36XX_HAL_KEEP_ALIVE_NULL_PKT 1
u64 key_replay_counter;
u8 bss_index;
-};
+} __packed;
struct wcn36xx_hal_gtk_offload_rsp_msg {
struct wcn36xx_hal_msg_header header;
struct wcn36xx_hal_gtk_offload_get_info_req_msg {
struct wcn36xx_hal_msg_header header;
u8 bss_index;
-};
+} __packed;
struct wcn36xx_hal_gtk_offload_get_info_rsp_msg {
struct wcn36xx_hal_msg_header header;
u32 igtk_rekey_count;
u8 bss_index;
-};
+} __packed;
struct dhcp_info {
/* Indicates the device mode which indicates about the DHCP activity */
#include <linux/rpmsg.h>
#include <linux/soc/qcom/smem_state.h>
#include <linux/soc/qcom/wcnss_ctrl.h>
+#include <net/ipv6.h>
#include "wcn36xx.h"
#include "testmode.h"
#ifdef CONFIG_PM
static const struct wiphy_wowlan_support wowlan_support = {
- .flags = WIPHY_WOWLAN_ANY
+ .flags = WIPHY_WOWLAN_ANY |
+ WIPHY_WOWLAN_MAGIC_PKT |
+ WIPHY_WOWLAN_SUPPORTS_GTK_REKEY
};
#endif
goto out_free_dxe_pool;
}
- wcn->hal_buf = kmalloc(WCN36XX_HAL_BUF_SIZE, GFP_KERNEL);
- if (!wcn->hal_buf) {
- wcn36xx_err("Failed to allocate smd buf\n");
- ret = -ENOMEM;
- goto out_free_dxe_ctl;
- }
-
ret = wcn36xx_smd_load_nv(wcn);
if (ret) {
wcn36xx_err("Failed to push NV to chip\n");
- goto out_free_smd_buf;
+ goto out_free_dxe_ctl;
}
ret = wcn36xx_smd_start(wcn);
if (ret) {
wcn36xx_err("Failed to start chip\n");
- goto out_free_smd_buf;
+ goto out_free_dxe_ctl;
}
if (!wcn36xx_is_fw_version(wcn, 1, 2, 2, 24)) {
out_smd_stop:
wcn36xx_smd_stop(wcn);
-out_free_smd_buf:
- kfree(wcn->hal_buf);
out_free_dxe_ctl:
wcn36xx_dxe_free_ctl_blks(wcn);
out_free_dxe_pool:
wcn36xx_dxe_free_mem_pools(wcn);
wcn36xx_dxe_free_ctl_blks(wcn);
-
- kfree(wcn->hal_buf);
}
static void wcn36xx_change_ps(struct wcn36xx *wcn, bool enable)
#ifdef CONFIG_PM
+static struct ieee80211_vif *wcn36xx_get_first_assoc_vif(struct wcn36xx *wcn)
+{
+ struct wcn36xx_vif *vif_priv = NULL;
+ struct ieee80211_vif *vif = NULL;
+
+ list_for_each_entry(vif_priv, &wcn->vif_list, list) {
+ if (vif_priv->sta_assoc) {
+ vif = wcn36xx_priv_to_vif(vif_priv);
+ break;
+ }
+ }
+ return vif;
+}
+
static int wcn36xx_suspend(struct ieee80211_hw *hw, struct cfg80211_wowlan *wow)
{
struct wcn36xx *wcn = hw->priv;
+ struct ieee80211_vif *vif = NULL;
+ int ret = 0;
wcn36xx_dbg(WCN36XX_DBG_MAC, "mac suspend\n");
- flush_workqueue(wcn->hal_ind_wq);
- wcn36xx_smd_set_power_params(wcn, true);
- return 0;
+ mutex_lock(&wcn->conf_mutex);
+
+ vif = wcn36xx_get_first_assoc_vif(wcn);
+ if (vif) {
+ ret = wcn36xx_smd_arp_offload(wcn, vif, true);
+ if (ret)
+ goto out;
+ ret = wcn36xx_smd_ipv6_ns_offload(wcn, vif, true);
+ if (ret)
+ goto out;
+ ret = wcn36xx_smd_gtk_offload(wcn, vif, true);
+ if (ret)
+ goto out;
+ ret = wcn36xx_smd_set_power_params(wcn, true);
+ if (ret)
+ goto out;
+ ret = wcn36xx_smd_wlan_host_suspend_ind(wcn);
+ }
+out:
+ mutex_unlock(&wcn->conf_mutex);
+ return ret;
}
static int wcn36xx_resume(struct ieee80211_hw *hw)
{
struct wcn36xx *wcn = hw->priv;
+ struct ieee80211_vif *vif = NULL;
wcn36xx_dbg(WCN36XX_DBG_MAC, "mac resume\n");
- flush_workqueue(wcn->hal_ind_wq);
- wcn36xx_smd_set_power_params(wcn, false);
+ mutex_lock(&wcn->conf_mutex);
+ vif = wcn36xx_get_first_assoc_vif(wcn);
+ if (vif) {
+ wcn36xx_smd_host_resume(wcn);
+ wcn36xx_smd_set_power_params(wcn, false);
+ wcn36xx_smd_gtk_offload_get_info(wcn, vif);
+ wcn36xx_smd_gtk_offload(wcn, vif, false);
+ wcn36xx_smd_ipv6_ns_offload(wcn, vif, false);
+ wcn36xx_smd_arp_offload(wcn, vif, false);
+ }
+ mutex_unlock(&wcn->conf_mutex);
+
return 0;
}
+static void wcn36xx_set_rekey_data(struct ieee80211_hw *hw,
+ struct ieee80211_vif *vif,
+ struct cfg80211_gtk_rekey_data *data)
+{
+ struct wcn36xx *wcn = hw->priv;
+ struct wcn36xx_vif *vif_priv = wcn36xx_vif_to_priv(vif);
+
+ mutex_lock(&wcn->conf_mutex);
+
+ memcpy(vif_priv->rekey_data.kek, data->kek, NL80211_KEK_LEN);
+ memcpy(vif_priv->rekey_data.kck, data->kck, NL80211_KCK_LEN);
+ vif_priv->rekey_data.replay_ctr =
+ cpu_to_le64(be64_to_cpup((__be64 *)data->replay_ctr));
+ vif_priv->rekey_data.valid = true;
+
+ mutex_unlock(&wcn->conf_mutex);
+}
+
#endif
static int wcn36xx_ampdu_action(struct ieee80211_hw *hw,
return ret;
}
+#if IS_ENABLED(CONFIG_IPV6)
+static void wcn36xx_ipv6_addr_change(struct ieee80211_hw *hw,
+ struct ieee80211_vif *vif,
+ struct inet6_dev *idev)
+{
+ struct wcn36xx_vif *vif_priv = wcn36xx_vif_to_priv(vif);
+ struct inet6_ifaddr *ifa;
+ int idx = 0;
+
+ memset(vif_priv->tentative_addrs, 0, sizeof(vif_priv->tentative_addrs));
+
+ read_lock_bh(&idev->lock);
+ list_for_each_entry(ifa, &idev->addr_list, if_list) {
+ vif_priv->target_ipv6_addrs[idx] = ifa->addr;
+ if (ifa->flags & IFA_F_TENTATIVE)
+ __set_bit(idx, vif_priv->tentative_addrs);
+ idx++;
+ if (idx >= WCN36XX_HAL_IPV6_OFFLOAD_ADDR_MAX)
+ break;
+ wcn36xx_dbg(WCN36XX_DBG_MAC, "%pI6 %s\n", &ifa->addr,
+ (ifa->flags & IFA_F_TENTATIVE) ? "tentative" : NULL);
+ }
+ read_unlock_bh(&idev->lock);
+
+ vif_priv->num_target_ipv6_addrs = idx;
+}
+#endif
+
static const struct ieee80211_ops wcn36xx_ops = {
.start = wcn36xx_start,
.stop = wcn36xx_stop,
#ifdef CONFIG_PM
.suspend = wcn36xx_suspend,
.resume = wcn36xx_resume,
+ .set_rekey_data = wcn36xx_set_rekey_data,
#endif
.config = wcn36xx_config,
.prepare_multicast = wcn36xx_prepare_multicast,
.sta_add = wcn36xx_sta_add,
.sta_remove = wcn36xx_sta_remove,
.ampdu_action = wcn36xx_ampdu_action,
+#if IS_ENABLED(CONFIG_IPV6)
+ .ipv6_addr_change = wcn36xx_ipv6_addr_change,
+#endif
CFG80211_TESTMODE_CMD(wcn36xx_tm_cmd)
};
mutex_init(&wcn->hal_mutex);
mutex_init(&wcn->scan_lock);
+ wcn->hal_buf = devm_kmalloc(wcn->dev, WCN36XX_HAL_BUF_SIZE, GFP_KERNEL);
+ if (!wcn->hal_buf) {
+ ret = -ENOMEM;
+ goto out_wq;
+ }
+
ret = dma_set_mask_and_coherent(wcn->dev, DMA_BIT_MASK(32));
if (ret < 0) {
wcn36xx_err("failed to set DMA mask: %d\n", ret);
return ret;
}
-static void init_hal_msg(struct wcn36xx_hal_msg_header *hdr,
- enum wcn36xx_hal_host_msg_type msg_type,
- size_t msg_size)
-{
- memset(hdr, 0, msg_size + sizeof(*hdr));
- hdr->msg_type = msg_type;
- hdr->msg_version = WCN36XX_HAL_MSG_VERSION0;
- hdr->len = msg_size + sizeof(*hdr);
-}
-
#define __INIT_HAL_MSG(msg_body, type, version) \
do { \
- memset(&msg_body, 0, sizeof(msg_body)); \
- msg_body.header.msg_type = type; \
- msg_body.header.msg_version = version; \
- msg_body.header.len = sizeof(msg_body); \
+ memset(&(msg_body), 0, sizeof(msg_body)); \
+ (msg_body).header.msg_type = type; \
+ (msg_body).header.msg_version = version; \
+ (msg_body).header.len = sizeof(msg_body); \
} while (0) \
#define INIT_HAL_MSG(msg_body, type) \
msg_body = (struct wcn36xx_hal_rcv_flt_pkt_set_mc_list_req_msg *)
wcn->hal_buf;
- init_hal_msg(&msg_body->header, WCN36XX_HAL_8023_MULTICAST_LIST_REQ,
- sizeof(msg_body->mc_addr_list));
+ INIT_HAL_MSG(*msg_body, WCN36XX_HAL_8023_MULTICAST_LIST_REQ);
/* An empty list means all mc traffic will be received */
if (fp)
return ret;
}
+int wcn36xx_smd_arp_offload(struct wcn36xx *wcn, struct ieee80211_vif *vif,
+ bool enable)
+{
+ struct wcn36xx_vif *vif_priv = wcn36xx_vif_to_priv(vif);
+ struct wcn36xx_hal_host_offload_req_msg msg_body;
+ int ret;
+
+ mutex_lock(&wcn->hal_mutex);
+
+ INIT_HAL_MSG(msg_body, WCN36XX_HAL_HOST_OFFLOAD_REQ);
+ msg_body.host_offload_params.offload_type =
+ WCN36XX_HAL_IPV4_ARP_REPLY_OFFLOAD;
+ if (enable) {
+ msg_body.host_offload_params.enable =
+ WCN36XX_HAL_OFFLOAD_ARP_AND_BCAST_FILTER_ENABLE;
+ memcpy(&msg_body.host_offload_params.u,
+ &vif->bss_conf.arp_addr_list[0], sizeof(__be32));
+ }
+ msg_body.ns_offload_params.bss_index = vif_priv->bss_index;
+
+ PREPARE_HAL_BUF(wcn->hal_buf, msg_body);
+
+ ret = wcn36xx_smd_send_and_wait(wcn, msg_body.header.len);
+ if (ret) {
+ wcn36xx_err("Sending host_offload_arp failed\n");
+ goto out;
+ }
+ ret = wcn36xx_smd_rsp_status_check(wcn->hal_buf, wcn->hal_rsp_len);
+ if (ret) {
+ wcn36xx_err("host_offload_arp failed err=%d\n", ret);
+ goto out;
+ }
+out:
+ mutex_unlock(&wcn->hal_mutex);
+ return ret;
+}
+
+#if IS_ENABLED(CONFIG_IPV6)
+int wcn36xx_smd_ipv6_ns_offload(struct wcn36xx *wcn, struct ieee80211_vif *vif,
+ bool enable)
+{
+ struct wcn36xx_vif *vif_priv = wcn36xx_vif_to_priv(vif);
+ struct wcn36xx_hal_host_offload_req_msg msg_body;
+ struct wcn36xx_hal_ns_offload_params *ns_params;
+ struct wcn36xx_hal_host_offload_req *ho_params;
+ int ret;
+
+ mutex_lock(&wcn->hal_mutex);
+
+ INIT_HAL_MSG(msg_body, WCN36XX_HAL_HOST_OFFLOAD_REQ);
+ ho_params = &msg_body.host_offload_params;
+ ns_params = &msg_body.ns_offload_params;
+
+ ho_params->offload_type = WCN36XX_HAL_IPV6_NS_OFFLOAD;
+ if (enable) {
+ ho_params->enable =
+ WCN36XX_HAL_OFFLOAD_NS_AND_MCAST_FILTER_ENABLE;
+ if (vif_priv->num_target_ipv6_addrs) {
+ memcpy(&ho_params->u,
+ &vif_priv->target_ipv6_addrs[0].in6_u,
+ sizeof(struct in6_addr));
+ memcpy(&ns_params->target_ipv6_addr1,
+ &vif_priv->target_ipv6_addrs[0].in6_u,
+ sizeof(struct in6_addr));
+ ns_params->target_ipv6_addr1_valid = 1;
+ }
+ if (vif_priv->num_target_ipv6_addrs > 1) {
+ memcpy(&ns_params->target_ipv6_addr2,
+ &vif_priv->target_ipv6_addrs[1].in6_u,
+ sizeof(struct in6_addr));
+ ns_params->target_ipv6_addr2_valid = 1;
+ }
+ }
+ memcpy(&ns_params->self_addr, vif->addr, ETH_ALEN);
+ ns_params->bss_index = vif_priv->bss_index;
+
+ PREPARE_HAL_BUF(wcn->hal_buf, msg_body);
+
+ ret = wcn36xx_smd_send_and_wait(wcn, msg_body.header.len);
+ if (ret) {
+ wcn36xx_err("Sending host_offload_arp failed\n");
+ goto out;
+ }
+ ret = wcn36xx_smd_rsp_status_check(wcn->hal_buf, wcn->hal_rsp_len);
+ if (ret) {
+ wcn36xx_err("host_offload_arp failed err=%d\n", ret);
+ goto out;
+ }
+out:
+ mutex_unlock(&wcn->hal_mutex);
+ return ret;
+}
+#else
+int wcn36xx_smd_ipv6_ns_offload(struct wcn36xx *wcn, struct ieee80211_vif *vif,
+ bool enable)
+{
+ return 0;
+}
+#endif
+
+int wcn36xx_smd_gtk_offload(struct wcn36xx *wcn, struct ieee80211_vif *vif,
+ bool enable)
+{
+ struct wcn36xx_vif *vif_priv = wcn36xx_vif_to_priv(vif);
+ struct wcn36xx_hal_gtk_offload_req_msg msg_body;
+ int ret;
+
+ mutex_lock(&wcn->hal_mutex);
+
+ INIT_HAL_MSG(msg_body, WCN36XX_HAL_GTK_OFFLOAD_REQ);
+
+ if (enable) {
+ memcpy(&msg_body.kek, vif_priv->rekey_data.kek, NL80211_KEK_LEN);
+ memcpy(&msg_body.kck, vif_priv->rekey_data.kck, NL80211_KCK_LEN);
+ msg_body.key_replay_counter =
+ le64_to_cpu(vif_priv->rekey_data.replay_ctr);
+ msg_body.bss_index = vif_priv->bss_index;
+ } else {
+ msg_body.flags = WCN36XX_HAL_GTK_OFFLOAD_FLAGS_DISABLE;
+ }
+
+ PREPARE_HAL_BUF(wcn->hal_buf, msg_body);
+
+ ret = wcn36xx_smd_send_and_wait(wcn, msg_body.header.len);
+ if (ret) {
+ wcn36xx_err("Sending host_offload_arp failed\n");
+ goto out;
+ }
+ ret = wcn36xx_smd_rsp_status_check(wcn->hal_buf, wcn->hal_rsp_len);
+ if (ret) {
+ wcn36xx_err("host_offload_arp failed err=%d\n", ret);
+ goto out;
+ }
+out:
+ mutex_unlock(&wcn->hal_mutex);
+ return ret;
+}
+
+static int wcn36xx_smd_gtk_offload_get_info_rsp(struct wcn36xx *wcn,
+ struct ieee80211_vif *vif)
+{
+ struct wcn36xx_vif *vif_priv = wcn36xx_vif_to_priv(vif);
+ struct wcn36xx_hal_gtk_offload_get_info_rsp_msg *rsp;
+ __be64 replay_ctr;
+
+ if (wcn36xx_smd_rsp_status_check(wcn->hal_buf, wcn->hal_rsp_len))
+ return -EIO;
+
+ rsp = (struct wcn36xx_hal_gtk_offload_get_info_rsp_msg *)wcn->hal_buf;
+
+ if (rsp->bss_index != vif_priv->bss_index) {
+ wcn36xx_err("gtk_offload_info invalid response bss index %d\n",
+ rsp->bss_index);
+ return -ENOENT;
+ }
+
+ if (vif_priv->rekey_data.replay_ctr != cpu_to_le64(rsp->key_replay_counter)) {
+ replay_ctr = cpu_to_be64(rsp->key_replay_counter);
+ vif_priv->rekey_data.replay_ctr =
+ cpu_to_le64(rsp->key_replay_counter);
+ ieee80211_gtk_rekey_notify(vif, vif->bss_conf.bssid,
+ (void *)&replay_ctr, GFP_KERNEL);
+ wcn36xx_dbg(WCN36XX_DBG_HAL,
+ "GTK replay counter increment %llu\n",
+ rsp->key_replay_counter);
+ }
+
+ wcn36xx_dbg(WCN36XX_DBG_HAL,
+ "gtk offload info status %d last_rekey_status %d "
+ "replay_counter %llu total_rekey_count %d gtk_rekey_count %d "
+ "igtk_rekey_count %d bss_index %d\n",
+ rsp->status, rsp->last_rekey_status,
+ rsp->key_replay_counter, rsp->total_rekey_count,
+ rsp->gtk_rekey_count, rsp->igtk_rekey_count,
+ rsp->bss_index);
+
+ return 0;
+}
+
+int wcn36xx_smd_gtk_offload_get_info(struct wcn36xx *wcn,
+ struct ieee80211_vif *vif)
+{
+ struct wcn36xx_vif *vif_priv = wcn36xx_vif_to_priv(vif);
+ struct wcn36xx_hal_gtk_offload_get_info_req_msg msg_body;
+ int ret;
+
+ mutex_lock(&wcn->hal_mutex);
+
+ INIT_HAL_MSG(msg_body, WCN36XX_HAL_GTK_OFFLOAD_GETINFO_REQ);
+
+ msg_body.bss_index = vif_priv->bss_index;
+
+ PREPARE_HAL_BUF(wcn->hal_buf, msg_body);
+
+ ret = wcn36xx_smd_send_and_wait(wcn, msg_body.header.len);
+ if (ret) {
+ wcn36xx_err("Sending gtk_offload_get_info failed\n");
+ goto out;
+ }
+ ret = wcn36xx_smd_rsp_status_check(wcn->hal_buf, wcn->hal_rsp_len);
+ if (ret) {
+ wcn36xx_err("gtk_offload_get_info failed err=%d\n", ret);
+ goto out;
+ }
+ ret = wcn36xx_smd_gtk_offload_get_info_rsp(wcn, vif);
+out:
+ mutex_unlock(&wcn->hal_mutex);
+ return ret;
+}
+
+int wcn36xx_smd_wlan_host_suspend_ind(struct wcn36xx *wcn)
+{
+ struct wcn36xx_hal_wlan_host_suspend_ind_msg msg_body;
+ int ret;
+
+ mutex_lock(&wcn->hal_mutex);
+
+ INIT_HAL_MSG(msg_body, WCN36XX_HAL_HOST_SUSPEND_IND);
+ msg_body.configured_mcst_bcst_filter_setting = 0;
+ msg_body.active_session_count = 1;
+ PREPARE_HAL_BUF(wcn->hal_buf, msg_body);
+
+ ret = rpmsg_send(wcn->smd_channel, wcn->hal_buf, msg_body.header.len);
+
+ mutex_unlock(&wcn->hal_mutex);
+
+ return ret;
+}
+
+int wcn36xx_smd_host_resume(struct wcn36xx *wcn)
+{
+ struct wcn36xx_hal_wlan_host_resume_req_msg msg_body;
+ struct wcn36xx_hal_host_resume_rsp_msg *rsp;
+ int ret;
+
+ mutex_lock(&wcn->hal_mutex);
+
+ INIT_HAL_MSG(msg_body, WCN36XX_HAL_HOST_RESUME_REQ);
+ msg_body.configured_mcst_bcst_filter_setting = 0;
+
+ PREPARE_HAL_BUF(wcn->hal_buf, msg_body);
+
+ ret = wcn36xx_smd_send_and_wait(wcn, msg_body.header.len);
+ if (ret) {
+ wcn36xx_err("Sending wlan_host_resume failed\n");
+ goto out;
+ }
+ ret = wcn36xx_smd_rsp_status_check(wcn->hal_buf, wcn->hal_rsp_len);
+ if (ret) {
+ wcn36xx_err("wlan_host_resume err=%d\n", ret);
+ goto out;
+ }
+
+ rsp = (struct wcn36xx_hal_host_resume_rsp_msg *)wcn->hal_buf;
+ if (rsp->status)
+ wcn36xx_warn("wlan_host_resume status=%d\n", rsp->status);
+
+out:
+ mutex_unlock(&wcn->hal_mutex);
+
+ return ret;
+}
+
int wcn36xx_smd_rsp_process(struct rpmsg_device *rpdev,
void *buf, int len, void *priv, u32 addr)
{
case WCN36XX_HAL_8023_MULTICAST_LIST_RSP:
case WCN36XX_HAL_START_SCAN_OFFLOAD_RSP:
case WCN36XX_HAL_STOP_SCAN_OFFLOAD_RSP:
+ case WCN36XX_HAL_HOST_OFFLOAD_RSP:
+ case WCN36XX_HAL_GTK_OFFLOAD_RSP:
+ case WCN36XX_HAL_GTK_OFFLOAD_GETINFO_RSP:
+ case WCN36XX_HAL_HOST_RESUME_RSP:
memcpy(wcn->hal_buf, buf, len);
wcn->hal_rsp_len = len;
complete(&wcn->hal_rsp_compl);
int wcn36xx_smd_set_mc_list(struct wcn36xx *wcn,
struct ieee80211_vif *vif,
struct wcn36xx_hal_rcv_flt_mc_addr_list_type *fp);
+
+int wcn36xx_smd_arp_offload(struct wcn36xx *wcn, struct ieee80211_vif *vif,
+ bool enable);
+
+int wcn36xx_smd_ipv6_ns_offload(struct wcn36xx *wcn, struct ieee80211_vif *vif,
+ bool enable);
+
+int wcn36xx_smd_gtk_offload(struct wcn36xx *wcn, struct ieee80211_vif *vif,
+ bool enable);
+
+int wcn36xx_smd_gtk_offload_get_info(struct wcn36xx *wcn,
+ struct ieee80211_vif *vif);
+
+int wcn36xx_smd_wlan_host_suspend_ind(struct wcn36xx *wcn);
+
+int wcn36xx_smd_host_resume(struct wcn36xx *wcn);
+
#endif /* _SMD_H_ */
#define _WCN36XX_H_
#include <linux/completion.h>
+#include <linux/in6.h>
#include <linux/printk.h>
#include <linux/spinlock.h>
#include <net/mac80211.h>
u8 self_dpu_desc_index;
u8 self_ucast_dpu_sign;
+#if IS_ENABLED(CONFIG_IPV6)
+ /* IPv6 addresses for WoWLAN */
+ struct in6_addr target_ipv6_addrs[WCN36XX_HAL_IPV6_OFFLOAD_ADDR_MAX];
+ unsigned long tentative_addrs[BITS_TO_LONGS(WCN36XX_HAL_IPV6_OFFLOAD_ADDR_MAX)];
+ int num_target_ipv6_addrs;
+#endif
+ /* WoWLAN GTK rekey data */
+ struct {
+ u8 kck[NL80211_KCK_LEN], kek[NL80211_KEK_LEN];
+ __le64 replay_ctr;
+ bool valid;
+ } rekey_data;
+
struct list_head sta_list;
};
wil->radio_wdev = wil->main_ndev->ieee80211_ptr;
mutex_unlock(&wil->vif_mutex);
if (p2p_wdev) {
- wiphy_lock(wil->wiphy);
cfg80211_unregister_wdev(p2p_wdev);
- wiphy_unlock(wil->wiphy);
kfree(p2p_wdev);
}
}
#include "wil6210.h"
#include "trace.h"
-/**
+/*
* Theory of operation:
*
* There is ISR pseudo-cause register,
u8 led_polarity = LED_POLARITY_LOW_ACTIVE;
/**
- * return AHB address for given firmware internal (linker) address
+ * wmi_addr_remap - return AHB address for given firmware internal (linker) address
* @x: internal address
* If address have no valid AHB mapping, return 0
*/
}
/**
- * find fw_mapping entry by section name
+ * wil_find_fw_mapping - find fw_mapping entry by section name
* @section: section name
*
* Return pointer to section or NULL if not found
}
/**
- * Check address validity for WMI buffer; remap if needed
+ * wmi_buffer_block - Check address validity for WMI buffer; remap if needed
* @wil: driver data
* @ptr_: internal (linker) fw/ucode address
* @size: if non zero, validate the block does not
{
struct b43_phy_n *nphy = dev->phy.n;
- u8 channel = dev->phy.channel;
- int tone[2] = { 57, 58 };
- u32 noise[2] = { 0x3FF, 0x3FF };
-
B43_WARN_ON(dev->phy.rev < 3);
if (nphy->hang_avoid)
b43_nphy_stay_in_carrier_search(dev, 1);
- if (nphy->gband_spurwar_en) {
- /* TODO: N PHY Adjust Analog Pfbw (7) */
- if (channel == 11 && b43_is_40mhz(dev)) {
- ; /* TODO: N PHY Adjust Min Noise Var(2, tone, noise)*/
- } else {
- ; /* TODO: N PHY Adjust Min Noise Var(0, NULL, NULL)*/
- }
- /* TODO: N PHY Adjust CRS Min Power (0x1E) */
- }
-
- if (nphy->aband_spurwar_en) {
- if (channel == 54) {
- tone[0] = 0x20;
- noise[0] = 0x25F;
- } else if (channel == 38 || channel == 102 || channel == 118) {
- if (0 /* FIXME */) {
- tone[0] = 0x20;
- noise[0] = 0x21F;
- } else {
- tone[0] = 0;
- noise[0] = 0;
- }
- } else if (channel == 134) {
- tone[0] = 0x20;
- noise[0] = 0x21F;
- } else if (channel == 151) {
- tone[0] = 0x10;
- noise[0] = 0x23F;
- } else if (channel == 153 || channel == 161) {
- tone[0] = 0x30;
- noise[0] = 0x23F;
- } else {
- tone[0] = 0;
- noise[0] = 0;
- }
-
- if (!tone[0] && !noise[0]) {
- ; /* TODO: N PHY Adjust Min Noise Var(1, tone, noise)*/
- } else {
- ; /* TODO: N PHY Adjust Min Noise Var(0, NULL, NULL)*/
- }
- }
-
if (nphy->hang_avoid)
b43_nphy_stay_in_carrier_search(dev, 0);
}
return ring;
}
-/* Bcm4301-ring to mac80211-queue mapping */
-static inline int txring_to_priority(struct b43legacy_dmaring *ring)
-{
- static const u8 idx_to_prio[] =
- { 3, 2, 1, 0, 4, 5, };
-
-/*FIXME: have only one queue, for now */
-return 0;
-
- return idx_to_prio[ring->index];
-}
-
-
static u16 b43legacy_dmacontroller_base(enum b43legacy_dmatype type,
int controller_idx)
{
* registers, we should take care of register overflows.
* In theory, the whole tsf read process should be atomic.
* We try to be atomic here, by restaring the read process,
- * if any of the high registers changed (overflew).
+ * if any of the high registers changed (overflowed).
*/
if (dev->dev->id.revision >= 3) {
u32 low;
},
};
-void brcmf_sdio_register(void)
+int brcmf_sdio_register(void)
{
- int ret;
-
- ret = sdio_register_driver(&brcmf_sdmmc_driver);
- if (ret)
- brcmf_err("sdio_register_driver failed: %d\n", ret);
+ return sdio_register_driver(&brcmf_sdmmc_driver);
}
void brcmf_sdio_exit(void)
#ifdef CONFIG_BRCMFMAC_SDIO
void brcmf_sdio_exit(void);
-void brcmf_sdio_register(void);
+int brcmf_sdio_register(void);
+#else
+static inline void brcmf_sdio_exit(void) { }
+static inline int brcmf_sdio_register(void) { return 0; }
#endif
+
#ifdef CONFIG_BRCMFMAC_USB
void brcmf_usb_exit(void);
-void brcmf_usb_register(void);
+int brcmf_usb_register(void);
+#else
+static inline void brcmf_usb_exit(void) { }
+static inline int brcmf_usb_register(void) { return 0; }
+#endif
+
+#ifdef CONFIG_BRCMFMAC_PCIE
+void brcmf_pcie_exit(void);
+int brcmf_pcie_register(void);
+#else
+static inline void brcmf_pcie_exit(void) { }
+static inline int brcmf_pcie_register(void) { return 0; }
#endif
#endif /* BRCMFMAC_BUS_H */
struct brcmf_sta_info_le sta_info_le;
u32 sta_flags;
u32 is_tdls_peer;
- s32 total_rssi;
- s32 count_rssi;
+ s32 total_rssi_avg = 0;
+ s32 total_rssi = 0;
+ s32 count_rssi = 0;
int rssi;
u32 i;
sinfo->filled |= BIT_ULL(NL80211_STA_INFO_RX_BYTES);
sinfo->rx_bytes = le64_to_cpu(sta_info_le.rx_tot_bytes);
}
- total_rssi = 0;
- count_rssi = 0;
for (i = 0; i < BRCMF_ANT_MAX; i++) {
- if (sta_info_le.rssi[i]) {
- sinfo->chain_signal_avg[count_rssi] =
- sta_info_le.rssi[i];
- sinfo->chain_signal[count_rssi] =
- sta_info_le.rssi[i];
- total_rssi += sta_info_le.rssi[i];
- count_rssi++;
- }
+ if (sta_info_le.rssi[i] == 0 ||
+ sta_info_le.rx_lastpkt_rssi[i] == 0)
+ continue;
+ sinfo->chains |= BIT(count_rssi);
+ sinfo->chain_signal[count_rssi] =
+ sta_info_le.rx_lastpkt_rssi[i];
+ sinfo->chain_signal_avg[count_rssi] =
+ sta_info_le.rssi[i];
+ total_rssi += sta_info_le.rx_lastpkt_rssi[i];
+ total_rssi_avg += sta_info_le.rssi[i];
+ count_rssi++;
}
if (count_rssi) {
- sinfo->filled |= BIT_ULL(NL80211_STA_INFO_CHAIN_SIGNAL);
- sinfo->chains = count_rssi;
-
sinfo->filled |= BIT_ULL(NL80211_STA_INFO_SIGNAL);
- total_rssi /= count_rssi;
- sinfo->signal = total_rssi;
+ sinfo->filled |= BIT_ULL(NL80211_STA_INFO_SIGNAL_AVG);
+ sinfo->filled |= BIT_ULL(NL80211_STA_INFO_CHAIN_SIGNAL);
+ sinfo->filled |=
+ BIT_ULL(NL80211_STA_INFO_CHAIN_SIGNAL_AVG);
+ sinfo->signal = total_rssi / count_rssi;
+ sinfo->signal_avg = total_rssi_avg / count_rssi;
} else if (test_bit(BRCMF_VIF_STATUS_CONNECTED,
&ifp->vif->sme_state)) {
memset(&scb_val, 0, sizeof(scb_val));
&cfg->assoclist,
sizeof(cfg->assoclist));
if (err) {
- bphy_err(drvr, "BRCMF_C_GET_ASSOCLIST unsupported, err=%d\n",
- err);
+ /* GET_ASSOCLIST unsupported by firmware of older chips */
+ if (err == -EBADE)
+ bphy_info_once(drvr, "BRCMF_C_GET_ASSOCLIST unsupported\n");
+ else
+ bphy_err(drvr, "BRCMF_C_GET_ASSOCLIST failed, err=%d\n",
+ err);
+
cfg->assoclist.count = 0;
return -EOPNOTSUPP;
}
err = brcmf_fil_iovar_int_get(ifp, "rxchain", &rxchain);
if (err) {
- bphy_err(drvr, "rxchain error (%d)\n", err);
+ /* rxchain unsupported by firmware of older chips */
+ if (err == -EBADE)
+ bphy_info_once(drvr, "rxchain unsupported\n");
+ else
+ bphy_err(drvr, "rxchain error (%d)\n", err);
+
nchain = 1;
} else {
for (nchain = 0; rxchain; nchain++)
s32 found_index;
int i;
- country_codes = drvr->settings->country_codes;
- if (!country_codes) {
- brcmf_dbg(TRACE, "No country codes configured for device\n");
- return -EINVAL;
- }
-
if ((alpha2[0] == ccreq->country_abbrev[0]) &&
(alpha2[1] == ccreq->country_abbrev[1])) {
brcmf_dbg(TRACE, "Country code already set\n");
return -EAGAIN;
}
+ country_codes = drvr->settings->country_codes;
+ if (!country_codes) {
+ brcmf_dbg(TRACE, "No country codes configured for device, using ISO3166 code and 0 rev\n");
+ memset(ccreq, 0, sizeof(*ccreq));
+ ccreq->country_abbrev[0] = alpha2[0];
+ ccreq->country_abbrev[1] = alpha2[1];
+ ccreq->ccode[0] = alpha2[0];
+ ccreq->ccode[1] = alpha2[1];
+ return 0;
+ }
+
found_index = -1;
for (i = 0; i < country_codes->table_size; i++) {
cc = &country_codes->table[i];
/*Finally, pick up the PROMISC flag */
cmd_value = (ndev->flags & IFF_PROMISC) ? true : false;
err = brcmf_fil_cmd_int_set(ifp, BRCMF_C_SET_PROMISC, cmd_value);
- if (err < 0)
- bphy_err(drvr, "Setting BRCMF_C_SET_PROMISC failed, %d\n",
- err);
+ if (err < 0) {
+ /* PROMISC unsupported by firmware of older chips */
+ if (err == -EBADE)
+ bphy_info_once(drvr, "BRCMF_C_SET_PROMISC unsupported\n");
+ else
+ bphy_err(drvr, "Setting BRCMF_C_SET_PROMISC failed, err=%d\n",
+ err);
+ }
brcmf_configure_arp_nd_offload(ifp, !cmd_value);
}
}
}
-static void brcmf_driver_register(struct work_struct *work)
-{
-#ifdef CONFIG_BRCMFMAC_SDIO
- brcmf_sdio_register();
-#endif
-#ifdef CONFIG_BRCMFMAC_USB
- brcmf_usb_register();
-#endif
-#ifdef CONFIG_BRCMFMAC_PCIE
- brcmf_pcie_register();
-#endif
-}
-static DECLARE_WORK(brcmf_driver_work, brcmf_driver_register);
-
int __init brcmf_core_init(void)
{
- if (!schedule_work(&brcmf_driver_work))
- return -EBUSY;
+ int err;
+ err = brcmf_sdio_register();
+ if (err)
+ return err;
+
+ err = brcmf_usb_register();
+ if (err)
+ goto error_usb_register;
+
+ err = brcmf_pcie_register();
+ if (err)
+ goto error_pcie_register;
return 0;
+
+error_pcie_register:
+ brcmf_usb_exit();
+error_usb_register:
+ brcmf_sdio_exit();
+ return err;
}
void __exit brcmf_core_exit(void)
{
- cancel_work_sync(&brcmf_driver_work);
-
-#ifdef CONFIG_BRCMFMAC_SDIO
brcmf_sdio_exit();
-#endif
-#ifdef CONFIG_BRCMFMAC_USB
brcmf_usb_exit();
-#endif
-#ifdef CONFIG_BRCMFMAC_PCIE
brcmf_pcie_exit();
-#endif
}
##__VA_ARGS__); \
} while (0)
+#define bphy_info_once(drvr, fmt, ...) \
+ wiphy_info_once((drvr)->wiphy, "%s: " fmt, __func__, \
+ ##__VA_ARGS__)
+
#if defined(DEBUG) || defined(CONFIG_BRCM_TRACING)
/* For debug/tracing purposes treat info messages as errors */
BRCMF_FW_DEFAULT_PATH fw_base; \
MODULE_FIRMWARE(BRCMF_FW_DEFAULT_PATH fw_base ".bin")
+/* Firmware and Country Local Matrix files */
+#define BRCMF_FW_CLM_DEF(fw_name, fw_base) \
+static const char BRCM_ ## fw_name ## _FIRMWARE_BASENAME[] = \
+ BRCMF_FW_DEFAULT_PATH fw_base; \
+MODULE_FIRMWARE(BRCMF_FW_DEFAULT_PATH fw_base ".bin"); \
+MODULE_FIRMWARE(BRCMF_FW_DEFAULT_PATH fw_base ".clm_blob")
+
#define BRCMF_FW_ENTRY(chipid, mask, name) \
{ chipid, mask, BRCM_ ## name ## _FIRMWARE_BASENAME }
#include "common.h"
#include "of.h"
+static int brcmf_of_get_country_codes(struct device *dev,
+ struct brcmf_mp_device *settings)
+{
+ struct device_node *np = dev->of_node;
+ struct brcmfmac_pd_cc_entry *cce;
+ struct brcmfmac_pd_cc *cc;
+ int count;
+ int i;
+
+ count = of_property_count_strings(np, "brcm,ccode-map");
+ if (count < 0) {
+ /* The property is optional, so return success if it doesn't
+ * exist. Otherwise propagate the error code.
+ */
+ return (count == -EINVAL) ? 0 : count;
+ }
+
+ cc = devm_kzalloc(dev, sizeof(*cc) + count * sizeof(*cce), GFP_KERNEL);
+ if (!cc)
+ return -ENOMEM;
+
+ cc->table_size = count;
+
+ for (i = 0; i < count; i++) {
+ const char *map;
+
+ cce = &cc->table[i];
+
+ if (of_property_read_string_index(np, "brcm,ccode-map",
+ i, &map))
+ continue;
+
+ /* String format e.g. US-Q2-86 */
+ if (sscanf(map, "%2c-%2c-%d", cce->iso3166, cce->cc,
+ &cce->rev) != 3)
+ brcmf_err("failed to read country map %s\n", map);
+ else
+ brcmf_dbg(INFO, "%s-%s-%d\n", cce->iso3166, cce->cc,
+ cce->rev);
+ }
+
+ settings->country_codes = cc;
+
+ return 0;
+}
+
void brcmf_of_probe(struct device *dev, enum brcmf_bus_type bus_type,
struct brcmf_mp_device *settings)
{
struct brcmfmac_sdio_pd *sdio = &settings->bus.sdio;
struct device_node *root, *np = dev->of_node;
int irq;
+ int err;
u32 irqf;
u32 val;
of_node_put(root);
}
- if (!np || bus_type != BRCMF_BUSTYPE_SDIO ||
- !of_device_is_compatible(np, "brcm,bcm4329-fmac"))
+ if (!np || !of_device_is_compatible(np, "brcm,bcm4329-fmac"))
+ return;
+
+ err = brcmf_of_get_country_codes(dev, settings);
+ if (err)
+ brcmf_err("failed to get OF country code map (err=%d)\n", err);
+
+ if (bus_type != BRCMF_BUSTYPE_SDIO)
return;
if (of_property_read_u32(np, "brcm,drive-strength", &val) == 0)
}
/**
- * Change a P2P Role.
+ * brcmf_p2p_ifchange - Change a P2P Role.
* @cfg: driver private data for cfg80211 interface.
* @if_type: interface type.
* Returns 0 if success.
BRCMF_FW_DEF(43602, "brcmfmac43602-pcie");
BRCMF_FW_DEF(4350, "brcmfmac4350-pcie");
BRCMF_FW_DEF(4350C, "brcmfmac4350c2-pcie");
-BRCMF_FW_DEF(4356, "brcmfmac4356-pcie");
-BRCMF_FW_DEF(43570, "brcmfmac43570-pcie");
+BRCMF_FW_CLM_DEF(4356, "brcmfmac4356-pcie");
+BRCMF_FW_CLM_DEF(43570, "brcmfmac43570-pcie");
BRCMF_FW_DEF(4358, "brcmfmac4358-pcie");
BRCMF_FW_DEF(4359, "brcmfmac4359-pcie");
BRCMF_FW_DEF(4364, "brcmfmac4364-pcie");
};
-void brcmf_pcie_register(void)
+int brcmf_pcie_register(void)
{
- int err;
-
brcmf_dbg(PCIE, "Enter\n");
- err = pci_register_driver(&brcmf_pciedrvr);
- if (err)
- brcmf_err(NULL, "PCIE driver registration failed, err=%d\n",
- err);
+ return pci_register_driver(&brcmf_pciedrvr);
}
struct brcmf_pciedev_info *devinfo;
};
-
-void brcmf_pcie_exit(void);
-void brcmf_pcie_register(void);
-
-
#endif /* BRCMFMAC_PCIE_H */
BRCMF_FW_DEF(4339, "brcmfmac4339-sdio");
BRCMF_FW_DEF(43430A0, "brcmfmac43430a0-sdio");
/* Note the names are not postfixed with a1 for backward compatibility */
-BRCMF_FW_DEF(43430A1, "brcmfmac43430-sdio");
-BRCMF_FW_DEF(43455, "brcmfmac43455-sdio");
+BRCMF_FW_CLM_DEF(43430A1, "brcmfmac43430-sdio");
+BRCMF_FW_CLM_DEF(43455, "brcmfmac43455-sdio");
BRCMF_FW_DEF(43456, "brcmfmac43456-sdio");
-BRCMF_FW_DEF(4354, "brcmfmac4354-sdio");
-BRCMF_FW_DEF(4356, "brcmfmac4356-sdio");
+BRCMF_FW_CLM_DEF(4354, "brcmfmac4354-sdio");
+BRCMF_FW_CLM_DEF(4356, "brcmfmac4356-sdio");
BRCMF_FW_DEF(4359, "brcmfmac4359-sdio");
-BRCMF_FW_DEF(4373, "brcmfmac4373-sdio");
-BRCMF_FW_DEF(43012, "brcmfmac43012-sdio");
+BRCMF_FW_CLM_DEF(4373, "brcmfmac4373-sdio");
+BRCMF_FW_CLM_DEF(43012, "brcmfmac43012-sdio");
/* firmware config files */
-MODULE_FIRMWARE(BRCMF_FW_DEFAULT_PATH "brcm/brcmfmac*-sdio.*.txt");
-MODULE_FIRMWARE(BRCMF_FW_DEFAULT_PATH "brcm/brcmfmac*-pcie.*.txt");
+MODULE_FIRMWARE(BRCMF_FW_DEFAULT_PATH "brcmfmac*-sdio.*.txt");
+MODULE_FIRMWARE(BRCMF_FW_DEFAULT_PATH "brcmfmac*-pcie.*.txt");
static const struct brcmf_firmware_mapping brcmf_sdio_fwnames[] = {
BRCMF_FW_ENTRY(BRCM_CC_43143_CHIP_ID, 0xFFFFFFFF, 43143),
}
}
-/**
+/*
* brcmfmac sdio bus specific header
* This is the lowest layer header wrapped on the packets transmitted between
* host and WiFi dongle which contains information needed for SDIO core and
if (ret) {
brcmf_err("Failed to probe after sdio device reset: ret %d\n",
ret);
- brcmf_sdiod_remove(sdiodev);
}
return ret;
usb_deregister(&brcmf_usbdrvr);
}
-void brcmf_usb_register(void)
+int brcmf_usb_register(void)
{
- int ret;
-
brcmf_dbg(USB, "Enter\n");
- ret = usb_register(&brcmf_usbdrvr);
- if (ret)
- brcmf_err("usb_register failed %d\n", ret);
+ return usb_register(&brcmf_usbdrvr);
}
sii = container_of(sih, struct si_info, pub);
- if (sii == NULL)
- return;
-
kfree(sii);
}
{
struct brcms_info *wl;
struct ieee80211_hw *hw;
+ int ret;
dev_info(&pdev->dev, "mfg %x core %x rev %d class %d irq %d\n",
pdev->id.manuf, pdev->id.id, pdev->id.rev, pdev->id.class,
wl = brcms_attach(pdev);
if (!wl) {
pr_err("%s: brcms_attach failed!\n", __func__);
- return -ENODEV;
+ ret = -ENODEV;
+ goto err_free_ieee80211;
}
brcms_led_register(wl);
return 0;
+
+err_free_ieee80211:
+ ieee80211_free_hw(hw);
+ return ret;
}
static int brcms_suspend(struct bcma_device *pdev)
rts->frame_control = cpu_to_le16(IEEE80211_FTYPE_CTL |
IEEE80211_STYPE_RTS);
- memcpy(&rts->ra, &h->addr1, 2 * ETH_ALEN);
+ memcpy(&rts->ra, &h->addr1, ETH_ALEN);
+ memcpy(&rts->ta, &h->addr2, ETH_ALEN);
}
/* mainrate
void brcms_c_stf_phy_txant_upd(struct brcms_c_info *wlc);
int brcms_c_stf_txchain_set(struct brcms_c_info *wlc, s32 int_val, bool force);
bool brcms_c_stf_stbc_rx_set(struct brcms_c_info *wlc, s32 int_val);
-void brcms_c_stf_phy_txant_upd(struct brcms_c_info *wlc);
void brcms_c_stf_phy_chain_calc(struct brcms_c_info *wlc);
u16 brcms_c_stf_phytxchain_sel(struct brcms_c_info *wlc, u32 rspec);
u16 brcms_c_stf_d11hdrs_phyctl_txant(struct brcms_c_info *wlc, u32 rspec);
#define WEP_STR_128(x) x[0],x[1],x[2],x[3],x[4],x[5],x[6],x[7],x[8],x[9],x[10]
/**
- * Set a the wep key
+ * ipw2100_set_key() - Set a the wep key
*
* @priv: struct to work on
* @idx: index of the key we want to set
iwlwifi-objs += queue/tx.o
iwlwifi-objs += fw/img.o fw/notif-wait.o
-iwlwifi-objs += fw/dbg.o fw/pnvm.o
+iwlwifi-objs += fw/dbg.o fw/pnvm.o fw/dump.o
iwlwifi-$(CONFIG_IWLMVM) += fw/paging.o fw/smem.o fw/init.o
iwlwifi-$(CONFIG_ACPI) += fw/acpi.o
+iwlwifi-$(CONFIG_EFI) += fw/uefi.o
iwlwifi-$(CONFIG_IWLWIFI_DEBUGFS) += fw/debugfs.o
iwlwifi-objs += $(iwlwifi-m)
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/*
* Copyright (C) 2015-2017 Intel Deutschland GmbH
- * Copyright (C) 2018-2020 Intel Corporation
+ * Copyright (C) 2018-2021 Intel Corporation
*/
#include <linux/module.h>
#include <linux/stringify.h>
#include "iwl-prph.h"
/* Highest firmware API version supported */
-#define IWL_22000_UCODE_API_MAX 63
+#define IWL_22000_UCODE_API_MAX 64
/* Lowest firmware API version supported */
#define IWL_22000_UCODE_API_MIN 39
#define IWL_MA_A_GF_A_FW_PRE "iwlwifi-ma-a0-gf-a0-"
#define IWL_MA_A_GF4_A_FW_PRE "iwlwifi-ma-a0-gf4-a0-"
#define IWL_MA_A_MR_A_FW_PRE "iwlwifi-ma-a0-mr-a0-"
+#define IWL_MA_A_FM_A_FW_PRE "iwlwifi-ma-a0-fm-a0-"
#define IWL_SNJ_A_MR_A_FW_PRE "iwlwifi-SoSnj-a0-mr-a0-"
#define IWL_BZ_A_HR_B_FW_PRE "iwlwifi-bz-a0-hr-b0-"
#define IWL_BZ_A_GF_A_FW_PRE "iwlwifi-bz-a0-gf-a0-"
IWL_MA_A_GF4_A_FW_PRE __stringify(api) ".ucode"
#define IWL_MA_A_MR_A_FW_MODULE_FIRMWARE(api) \
IWL_MA_A_MR_A_FW_PRE __stringify(api) ".ucode"
+#define IWL_MA_A_FM_A_FW_MODULE_FIRMWARE(api) \
+ IWL_MA_A_FM_A_FW_PRE __stringify(api) ".ucode"
#define IWL_SNJ_A_MR_A_MODULE_FIRMWARE(api) \
IWL_SNJ_A_MR_A_FW_PRE __stringify(api) ".ucode"
#define IWL_BZ_A_HR_B_MODULE_FIRMWARE(api) \
const char iwl_ax203_name[] = "Intel(R) Wi-Fi 6 AX203";
const char iwl_ax211_name[] = "Intel(R) Wi-Fi 6E AX211 160MHz";
const char iwl_ax221_name[] = "Intel(R) Wi-Fi 6E AX221 160MHz";
+const char iwl_ax231_name[] = "Intel(R) Wi-Fi 6E AX231 160MHz";
const char iwl_ax411_name[] = "Intel(R) Wi-Fi 6E AX411 160MHz";
const char iwl_ax200_killer_1650w_name[] =
.num_rbds = IWL_NUM_RBDS_AX210_HE,
};
+const struct iwl_cfg iwl_cfg_ma_a0_fm_a0 = {
+ .fw_name_pre = IWL_MA_A_FM_A_FW_PRE,
+ .uhb_supported = true,
+ IWL_DEVICE_AX210,
+ .num_rbds = IWL_NUM_RBDS_AX210_HE,
+};
+
const struct iwl_cfg iwl_cfg_snj_a0_mr_a0 = {
.fw_name_pre = IWL_SNJ_A_MR_A_FW_PRE,
.uhb_supported = true,
MODULE_FIRMWARE(IWL_MA_A_GF_A_FW_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX));
MODULE_FIRMWARE(IWL_MA_A_GF4_A_FW_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX));
MODULE_FIRMWARE(IWL_MA_A_MR_A_FW_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX));
+MODULE_FIRMWARE(IWL_MA_A_FM_A_FW_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX));
MODULE_FIRMWARE(IWL_SNJ_A_MR_A_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX));
MODULE_FIRMWARE(IWL_BZ_A_HR_B_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX));
MODULE_FIRMWARE(IWL_BZ_A_GF_A_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX));
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/*
* Copyright (C) 2015-2017 Intel Deutschland GmbH
- * Copyright (C) 2018-2020 Intel Corporation
+ * Copyright (C) 2018-2021 Intel Corporation
*/
#include <linux/module.h>
#include <linux/stringify.h>
"Killer (R) Wireless-AC 1550 Wireless Network Adapter (9260NGW) 160MHz";
const char iwl9560_killer_1550i_name[] =
"Killer (R) Wireless-AC 1550i Wireless Network Adapter (9560NGW)";
+const char iwl9560_killer_1550i_160_name[] =
+ "Killer(R) Wireless-AC 1550i Wireless Network Adapter (9560NGW) 160MHz";
const char iwl9560_killer_1550s_name[] =
"Killer (R) Wireless-AC 1550s Wireless Network Adapter (9560NGW)";
+const char iwl9560_killer_1550s_160_name[] =
+ "Killer(R) Wireless-AC 1550s Wireless Network Adapter (9560D2W) 160MHz";
const struct iwl_cfg iwl9260_2ac_cfg = {
.fw_name_pre = IWL9260_FW_PRE,
}
IWL_EXPORT_SYMBOL(iwl_acpi_get_dsm_u8);
+/*
+ * Evaluate a DSM with no arguments and a u32 return value,
+ */
+int iwl_acpi_get_dsm_u32(struct device *dev, int rev, int func,
+ const guid_t *guid, u32 *value)
+{
+ int ret;
+ u64 val;
+
+ ret = iwl_acpi_get_dsm_integer(dev, rev, func,
+ guid, &val, sizeof(u32));
+
+ if (ret < 0)
+ return ret;
+
+ /* cast val (u64) to be u32 */
+ *value = (u32)val;
+ return 0;
+}
+IWL_EXPORT_SYMBOL(iwl_acpi_get_dsm_u32);
+
union acpi_object *iwl_acpi_get_wifi_pkg(struct device *dev,
union acpi_object *data,
int data_size, int *tbl_rev)
}
IWL_EXPORT_SYMBOL(iwl_sar_geo_init);
-static u32 iwl_acpi_eval_dsm_func(struct device *dev, enum iwl_dsm_funcs_rev_0 eval_func)
-{
- union acpi_object *obj;
- u32 ret;
-
- obj = iwl_acpi_get_dsm_object(dev, 0,
- eval_func, NULL,
- &iwl_guid);
-
- if (IS_ERR(obj)) {
- IWL_DEBUG_DEV_RADIO(dev,
- "ACPI: DSM func '%d': Got Error in obj = %ld\n",
- eval_func,
- PTR_ERR(obj));
- return 0;
- }
-
- if (obj->type != ACPI_TYPE_INTEGER) {
- IWL_DEBUG_DEV_RADIO(dev,
- "ACPI: DSM func '%d' did not return a valid object, type=%d\n",
- eval_func,
- obj->type);
- ret = 0;
- goto out;
- }
-
- ret = obj->integer.value;
- IWL_DEBUG_DEV_RADIO(dev,
- "ACPI: DSM method evaluated: func='%d', ret=%d\n",
- eval_func,
- ret);
-out:
- ACPI_FREE(obj);
- return ret;
-}
-
__le32 iwl_acpi_get_lari_config_bitmap(struct iwl_fw_runtime *fwrt)
{
- u32 ret;
+ int ret;
+ u8 value;
__le32 config_bitmap = 0;
/*
** Evaluate func 'DSM_FUNC_ENABLE_INDONESIA_5G2'
*/
- ret = iwl_acpi_eval_dsm_func(fwrt->dev, DSM_FUNC_ENABLE_INDONESIA_5G2);
+ ret = iwl_acpi_get_dsm_u8(fwrt->dev, 0,
+ DSM_FUNC_ENABLE_INDONESIA_5G2,
+ &iwl_guid, &value);
- if (ret == DSM_VALUE_INDONESIA_ENABLE)
+ if (!ret && value == DSM_VALUE_INDONESIA_ENABLE)
config_bitmap |=
cpu_to_le32(LARI_CONFIG_ENABLE_5G2_IN_INDONESIA_MSK);
/*
** Evaluate func 'DSM_FUNC_DISABLE_SRD'
*/
- ret = iwl_acpi_eval_dsm_func(fwrt->dev, DSM_FUNC_DISABLE_SRD);
-
- if (ret == DSM_VALUE_SRD_PASSIVE)
- config_bitmap |=
- cpu_to_le32(LARI_CONFIG_CHANGE_ETSI_TO_PASSIVE_MSK);
-
- else if (ret == DSM_VALUE_SRD_DISABLE)
- config_bitmap |=
- cpu_to_le32(LARI_CONFIG_CHANGE_ETSI_TO_DISABLED_MSK);
+ ret = iwl_acpi_get_dsm_u8(fwrt->dev, 0,
+ DSM_FUNC_DISABLE_SRD,
+ &iwl_guid, &value);
+ if (!ret) {
+ if (value == DSM_VALUE_SRD_PASSIVE)
+ config_bitmap |=
+ cpu_to_le32(LARI_CONFIG_CHANGE_ETSI_TO_PASSIVE_MSK);
+ else if (value == DSM_VALUE_SRD_DISABLE)
+ config_bitmap |=
+ cpu_to_le32(LARI_CONFIG_CHANGE_ETSI_TO_DISABLED_MSK);
+ }
return config_bitmap;
}
DSM_FUNC_DISABLE_SRD = 1,
DSM_FUNC_ENABLE_INDONESIA_5G2 = 2,
DSM_FUNC_11AX_ENABLEMENT = 6,
+ DSM_FUNC_ENABLE_UNII4_CHAN = 7
};
enum iwl_dsm_values_srd {
int iwl_acpi_get_dsm_u8(struct device *dev, int rev, int func,
const guid_t *guid, u8 *value);
+int iwl_acpi_get_dsm_u32(struct device *dev, int rev, int func,
+ const guid_t *guid, u32 *value);
+
union acpi_object *iwl_acpi_get_wifi_pkg(struct device *dev,
union acpi_object *data,
int data_size, int *tbl_rev);
return -ENOENT;
}
+static inline int iwl_acpi_get_dsm_u32(struct device *dev, int rev, int func,
+ const guid_t *guid, u32 *value)
+{
+ return -ENOENT;
+}
+
static inline union acpi_object *iwl_acpi_get_wifi_pkg(struct device *dev,
union acpi_object *data,
int data_size,
OFFLOADS_QUERY_CMD = 0xd5,
/**
- * @REMOTE_WAKE_CONFIG_CMD: &struct iwl_wowlan_remote_wake_config
- */
- REMOTE_WAKE_CONFIG_CMD = 0xd6,
-
- /**
* @D0I3_END_CMD: End D0i3/D3 state, no command data
*/
D0I3_END_CMD = 0xed,
/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
/*
- * Copyright (C) 2012-2014, 2018-2020 Intel Corporation
+ * Copyright (C) 2012-2014, 2018-2021 Intel Corporation
* Copyright (C) 2013-2014 Intel Mobile Communications GmbH
* Copyright (C) 2015-2017 Intel Deutschland GmbH
*/
struct iwl_ns_config ns_config[IWL_PROTO_OFFLOAD_NUM_NS_CONFIG_V3L];
} __packed; /* PROT_OFFLOAD_CONFIG_CMD_DB_S_VER_3 */
+/**
+ * struct iwl_proto_offload_cmd_v4 - ARP/NS offload configuration
+ * @sta_id: station id
+ * @common: common/IPv4 configuration
+ * @num_valid_ipv6_addrs: number of valid IPv6 addresses
+ * @targ_addrs: target IPv6 addresses
+ * @ns_config: NS offload configurations
+ */
+struct iwl_proto_offload_cmd_v4 {
+ __le32 sta_id;
+ struct iwl_proto_offload_cmd_common common;
+ __le32 num_valid_ipv6_addrs;
+ struct iwl_targ_addr targ_addrs[IWL_PROTO_OFFLOAD_NUM_IPV6_ADDRS_V3L];
+ struct iwl_ns_config ns_config[IWL_PROTO_OFFLOAD_NUM_NS_CONFIG_V3L];
+} __packed; /* PROT_OFFLOAD_CONFIG_CMD_DB_S_VER_4 */
+
/*
* WOWLAN_PATTERNS
*/
/**
* @n_patterns: number of patterns
*/
- __le32 n_patterns;
+ u8 n_patterns;
+
+ /**
+ * @n_patterns: sta_id
+ */
+ u8 sta_id;
+
+ /**
+ * @reserved: reserved for alignment
+ */
+ __le16 reserved;
/**
* @patterns: the patterns, array length in @n_patterns
*/
struct iwl_wowlan_pattern_v2 patterns[];
-} __packed; /* WOWLAN_PATTERN_ARRAY_API_S_VER_2 */
+} __packed; /* WOWLAN_PATTERN_ARRAY_API_S_VER_3 */
enum iwl_wowlan_wakeup_filters {
IWL_WOWLAN_WAKEUP_MAGIC_PACKET = BIT(0),
};
/**
- * struct iwl_wowlan_config_cmd - WoWLAN configuration
+ * struct iwl_wowlan_config_cmd - WoWLAN configuration (versions 5 and 6)
* @wakeup_filter: filter from &enum iwl_wowlan_wakeup_filters
- * @non_qos_seq: non-QoS sequence counter to use next
+ * @non_qos_seq: non-QoS sequence counter to use next.
+ * Reserved if the struct has version >= 6.
* @qos_seq: QoS sequence counters to use next
* @wowlan_ba_teardown_tids: bitmap of BA sessions to tear down
* @is_11n_connection: indicates HT connection
__le32 bigtk_cipher;
} __packed; /* KEK_KCK_MATERIAL_API_S_VER_3 */
+struct iwl_wowlan_kek_kck_material_cmd_v4 {
+ __le32 sta_id;
+ u8 kck[IWL_KCK_MAX_SIZE];
+ u8 kek[IWL_KEK_MAX_SIZE];
+ __le16 kck_len;
+ __le16 kek_len;
+ __le64 replay_ctr;
+ __le32 akm;
+ __le32 gtk_cipher;
+ __le32 igtk_cipher;
+ __le32 bigtk_cipher;
+} __packed; /* KEK_KCK_MATERIAL_API_S_VER_4 */
+
+struct iwl_wowlan_get_status_cmd {
+ __le32 sta_id;
+} __packed; /* WOWLAN_GET_STATUSES_CMD_API_S_VER_1 */
+
#define RF_KILL_INDICATOR_FOR_WOWLAN 0x87
enum iwl_wowlan_rekey_status {
} __packed; /* WOWLAN_STATUSES_API_S_VER_7 */
/**
- * struct iwl_wowlan_status_v9 - WoWLAN status (version 9)
+ * struct iwl_wowlan_status_v9 - WoWLAN status (versions 9 and 10)
* @gtk: GTK data
* @igtk: IGTK data
* @replay_ctr: GTK rekey replay counter
* @pattern_number: number of the matched pattern
- * @non_qos_seq_ctr: non-QoS sequence counter to use next
+ * @non_qos_seq_ctr: non-QoS sequence counter to use next.
+ * Reserved if the struct has version >= 10.
* @qos_seq_ctr: QoS sequence counters to use next
* @wakeup_reasons: wakeup reasons, see &enum iwl_wowlan_wakeup_reason
* @num_of_gtk_rekeys: number of GTK rekeys
u8 tid_tear_down;
u8 reserved[3];
u8 wake_packet[]; /* can be truncated from _length to _bufsize */
-} __packed; /* WOWLAN_STATUSES_API_S_VER_9 */
+} __packed; /* WOWLAN_STATUSES_RSP_API_S_VER_9 */
/**
* struct iwl_wowlan_status - WoWLAN status
return gtk->key_flags & IWL_WOWLAN_GTK_IDX_MASK;
}
-#define IWL_WOWLAN_TCP_MAX_PACKET_LEN 64
-#define IWL_WOWLAN_REMOTE_WAKE_MAX_PACKET_LEN 128
-#define IWL_WOWLAN_REMOTE_WAKE_MAX_TOKENS 2048
-
-struct iwl_tcp_packet_info {
- __le16 tcp_pseudo_header_checksum;
- __le16 tcp_payload_length;
-} __packed; /* TCP_PACKET_INFO_API_S_VER_2 */
-
-struct iwl_tcp_packet {
- struct iwl_tcp_packet_info info;
- u8 rx_mask[IWL_WOWLAN_MAX_PATTERN_LEN / 8];
- u8 data[IWL_WOWLAN_TCP_MAX_PACKET_LEN];
-} __packed; /* TCP_PROTOCOL_PACKET_API_S_VER_1 */
-
-struct iwl_remote_wake_packet {
- struct iwl_tcp_packet_info info;
- u8 rx_mask[IWL_WOWLAN_MAX_PATTERN_LEN / 8];
- u8 data[IWL_WOWLAN_REMOTE_WAKE_MAX_PACKET_LEN];
-} __packed; /* TCP_PROTOCOL_PACKET_API_S_VER_1 */
-
-struct iwl_wowlan_remote_wake_config {
- __le32 connection_max_time; /* unused */
- /* TCP_PROTOCOL_CONFIG_API_S_VER_1 */
- u8 max_syn_retries;
- u8 max_data_retries;
- u8 tcp_syn_ack_timeout;
- u8 tcp_ack_timeout;
-
- struct iwl_tcp_packet syn_tx;
- struct iwl_tcp_packet synack_rx;
- struct iwl_tcp_packet keepalive_ack_rx;
- struct iwl_tcp_packet fin_tx;
-
- struct iwl_remote_wake_packet keepalive_tx;
- struct iwl_remote_wake_packet wake_rx;
-
- /* REMOTE_WAKE_OFFSET_INFO_API_S_VER_1 */
- u8 sequence_number_offset;
- u8 sequence_number_length;
- u8 token_offset;
- u8 token_length;
- /* REMOTE_WAKE_PROTOCOL_PARAMS_API_S_VER_1 */
- __le32 initial_sequence_number;
- __le16 keepalive_interval;
- __le16 num_tokens;
- u8 tokens[IWL_WOWLAN_REMOTE_WAKE_MAX_TOKENS];
-} __packed; /* REMOTE_WAKE_CONFIG_API_S_VER_2 */
-
/* TODO: NetDetect API */
#endif /* __iwl_fw_api_d3_h__ */
RX_NO_DATA_NOTIF = 0xF5,
/**
+ * @THERMAL_DUAL_CHAIN_DISABLE_REQ: firmware request for SMPS mode,
+ * &struct iwl_thermal_dual_chain_request
+ */
+ THERMAL_DUAL_CHAIN_REQUEST = 0xF6,
+
+ /**
* @TLC_MNG_UPDATE_NOTIF: &struct iwl_tlc_update_notif
*/
TLC_MNG_UPDATE_NOTIF = 0xF7,
u8 reserved[3];
} __packed; /* MONITOR_NTF_API_S_VER_1 */
+/**
+ * enum iwl_thermal_dual_chain_req_events - firmware SMPS request event
+ * @THERMAL_DUAL_CHAIN_REQ_ENABLE: (re-)enable dual-chain operation
+ * (subject to other constraints)
+ * @THERMAL_DUAL_CHAIN_REQ_DISABLE: disable dual-chain operation
+ * (static SMPS)
+ */
+enum iwl_thermal_dual_chain_req_events {
+ THERMAL_DUAL_CHAIN_REQ_ENABLE,
+ THERMAL_DUAL_CHAIN_REQ_DISABLE,
+}; /* THERMAL_DUAL_CHAIN_DISABLE_STATE_API_E_VER_1 */
+
+/**
+ * struct iwl_thermal_dual_chain_request - SMPS request
+ * @event: the type of request, see &enum iwl_thermal_dual_chain_req_events
+ */
+struct iwl_thermal_dual_chain_request {
+ __le32 event;
+} __packed; /* THERMAL_DUAL_CHAIN_DISABLE_REQ_NTFY_API_S_VER_1 */
+
#endif /* __iwl_fw_api_datapath_h__ */
/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
/*
- * Copyright (C) 2018-2020 Intel Corporation
+ * Copyright (C) 2018-2021 Intel Corporation
*/
#ifndef __iwl_fw_dbg_tlv_h__
#define __iwl_fw_dbg_tlv_h__
#define IWL_FW_INI_MAX_NAME 32
#define IWL_FW_INI_MAX_CFG_NAME 64
#define IWL_FW_INI_DOMAIN_ALWAYS_ON 0
+#define IWL_FW_INI_REGION_V2_MASK 0x0000FFFF
/**
* struct iwl_fw_ini_hcmd
} __packed; /* LARI_CHANGE_CONF_CMD_S_VER_3 */
/**
+ * struct iwl_lari_config_change_cmd_v4 - change LARI configuration
+ * @config_bitmap: Bitmap of the config commands. Each bit will trigger a
+ * different predefined FW config operation.
+ * @oem_uhb_allow_bitmap: Bitmap of UHB enabled MCC sets.
+ * @oem_11ax_allow_bitmap: Bitmap of 11ax allowed MCCs. There are two bits
+ * per country, one to indicate whether to override and the other to
+ * indicate the value to use.
+ * @oem_unii4_allow_bitmap: Bitmap of unii4 allowed MCCs.There are two bits
+ * per country, one to indicate whether to override and the other to
+ * indicate allow/disallow unii4 channels.
+ */
+struct iwl_lari_config_change_cmd_v4 {
+ __le32 config_bitmap;
+ __le32 oem_uhb_allow_bitmap;
+ __le32 oem_11ax_allow_bitmap;
+ __le32 oem_unii4_allow_bitmap;
+} __packed; /* LARI_CHANGE_CONF_CMD_S_VER_4 */
+
+/**
* struct iwl_pnvm_init_complete_ntfy - PNVM initialization complete
* @status: PNVM image loading status
*/
u32 num_of_ranges, i, size;
void *range;
+ /*
+ * The higher part of the ID in version 2 is irrelevant for
+ * us, so mask it out.
+ */
+ if (le32_to_cpu(reg->hdr.version) == 2)
+ id &= IWL_FW_INI_REGION_V2_MASK;
+
if (!ops->get_num_of_ranges || !ops->get_size || !ops->fill_mem_hdr ||
!ops->fill_range)
return 0;
num_of_ranges = ops->get_num_of_ranges(fwrt, reg_data);
header = (void *)tlv->data;
- header->region_id = reg->id;
+ header->region_id = cpu_to_le32(id);
header->num_of_ranges = cpu_to_le32(num_of_ranges);
header->name_len = cpu_to_le32(IWL_FW_INI_MAX_NAME);
memcpy(header->name, reg->name, IWL_FW_INI_MAX_NAME);
}
IWL_EXPORT_SYMBOL(iwl_fw_dbg_stop_sync);
-#define FSEQ_REG(x) { .addr = (x), .str = #x, }
-
-void iwl_fw_error_print_fseq_regs(struct iwl_fw_runtime *fwrt)
-{
- struct iwl_trans *trans = fwrt->trans;
- int i;
- struct {
- u32 addr;
- const char *str;
- } fseq_regs[] = {
- FSEQ_REG(FSEQ_ERROR_CODE),
- FSEQ_REG(FSEQ_TOP_INIT_VERSION),
- FSEQ_REG(FSEQ_CNVIO_INIT_VERSION),
- FSEQ_REG(FSEQ_OTP_VERSION),
- FSEQ_REG(FSEQ_TOP_CONTENT_VERSION),
- FSEQ_REG(FSEQ_ALIVE_TOKEN),
- FSEQ_REG(FSEQ_CNVI_ID),
- FSEQ_REG(FSEQ_CNVR_ID),
- FSEQ_REG(CNVI_AUX_MISC_CHIP),
- FSEQ_REG(CNVR_AUX_MISC_CHIP),
- FSEQ_REG(CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM),
- FSEQ_REG(CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR),
- };
-
- if (!iwl_trans_grab_nic_access(trans))
- return;
-
- IWL_ERR(fwrt, "Fseq Registers:\n");
-
- for (i = 0; i < ARRAY_SIZE(fseq_regs); i++)
- IWL_ERR(fwrt, "0x%08X | %s\n",
- iwl_read_prph_no_grab(trans, fseq_regs[i].addr),
- fseq_regs[i].str);
-
- iwl_trans_release_nic_access(trans);
-}
-IWL_EXPORT_SYMBOL(iwl_fw_error_print_fseq_regs);
-
static int iwl_fw_dbg_suspend_resume_hcmd(struct iwl_trans *trans, bool suspend)
{
struct iwl_dbg_suspend_resume_cmd cmd = {
/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
/*
- * Copyright (C) 2005-2014, 2018-2019 Intel Corporation
+ * Copyright (C) 2005-2014, 2018-2019, 2021 Intel Corporation
* Copyright (C) 2013-2015 Intel Mobile Communications GmbH
* Copyright (C) 2015-2017 Intel Deutschland GmbH
*/
fwrt->dump.fw_ver.umac_minor = le32_to_cpu(umac->umac_minor);
}
}
+
+void iwl_fwrt_dump_error_logs(struct iwl_fw_runtime *fwrt);
#endif /* __iwl_fw_dbg_h__ */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * Copyright (C) 2012-2014, 2018-2021 Intel Corporation
+ * Copyright (C) 2013-2014 Intel Mobile Communications GmbH
+ * Copyright (C) 2015-2017 Intel Deutschland GmbH
+ */
+#include <linux/devcoredump.h>
+#include "iwl-drv.h"
+#include "runtime.h"
+#include "dbg.h"
+#include "debugfs.h"
+#include "iwl-io.h"
+#include "iwl-prph.h"
+#include "iwl-csr.h"
+
+/*
+ * Note: This structure is read from the device with IO accesses,
+ * and the reading already does the endian conversion. As it is
+ * read with u32-sized accesses, any members with a different size
+ * need to be ordered correctly though!
+ */
+struct iwl_error_event_table_v1 {
+ u32 valid; /* (nonzero) valid, (0) log is empty */
+ u32 error_id; /* type of error */
+ u32 pc; /* program counter */
+ u32 blink1; /* branch link */
+ u32 blink2; /* branch link */
+ u32 ilink1; /* interrupt link */
+ u32 ilink2; /* interrupt link */
+ u32 data1; /* error-specific data */
+ u32 data2; /* error-specific data */
+ u32 data3; /* error-specific data */
+ u32 bcon_time; /* beacon timer */
+ u32 tsf_low; /* network timestamp function timer */
+ u32 tsf_hi; /* network timestamp function timer */
+ u32 gp1; /* GP1 timer register */
+ u32 gp2; /* GP2 timer register */
+ u32 gp3; /* GP3 timer register */
+ u32 ucode_ver; /* uCode version */
+ u32 hw_ver; /* HW Silicon version */
+ u32 brd_ver; /* HW board version */
+ u32 log_pc; /* log program counter */
+ u32 frame_ptr; /* frame pointer */
+ u32 stack_ptr; /* stack pointer */
+ u32 hcmd; /* last host command header */
+ u32 isr0; /* isr status register LMPM_NIC_ISR0:
+ * rxtx_flag */
+ u32 isr1; /* isr status register LMPM_NIC_ISR1:
+ * host_flag */
+ u32 isr2; /* isr status register LMPM_NIC_ISR2:
+ * enc_flag */
+ u32 isr3; /* isr status register LMPM_NIC_ISR3:
+ * time_flag */
+ u32 isr4; /* isr status register LMPM_NIC_ISR4:
+ * wico interrupt */
+ u32 isr_pref; /* isr status register LMPM_NIC_PREF_STAT */
+ u32 wait_event; /* wait event() caller address */
+ u32 l2p_control; /* L2pControlField */
+ u32 l2p_duration; /* L2pDurationField */
+ u32 l2p_mhvalid; /* L2pMhValidBits */
+ u32 l2p_addr_match; /* L2pAddrMatchStat */
+ u32 lmpm_pmg_sel; /* indicate which clocks are turned on
+ * (LMPM_PMG_SEL) */
+ u32 u_timestamp; /* indicate when the date and time of the
+ * compilation */
+ u32 flow_handler; /* FH read/write pointers, RX credit */
+} __packed /* LOG_ERROR_TABLE_API_S_VER_1 */;
+
+struct iwl_error_event_table {
+ u32 valid; /* (nonzero) valid, (0) log is empty */
+ u32 error_id; /* type of error */
+ u32 trm_hw_status0; /* TRM HW status */
+ u32 trm_hw_status1; /* TRM HW status */
+ u32 blink2; /* branch link */
+ u32 ilink1; /* interrupt link */
+ u32 ilink2; /* interrupt link */
+ u32 data1; /* error-specific data */
+ u32 data2; /* error-specific data */
+ u32 data3; /* error-specific data */
+ u32 bcon_time; /* beacon timer */
+ u32 tsf_low; /* network timestamp function timer */
+ u32 tsf_hi; /* network timestamp function timer */
+ u32 gp1; /* GP1 timer register */
+ u32 gp2; /* GP2 timer register */
+ u32 fw_rev_type; /* firmware revision type */
+ u32 major; /* uCode version major */
+ u32 minor; /* uCode version minor */
+ u32 hw_ver; /* HW Silicon version */
+ u32 brd_ver; /* HW board version */
+ u32 log_pc; /* log program counter */
+ u32 frame_ptr; /* frame pointer */
+ u32 stack_ptr; /* stack pointer */
+ u32 hcmd; /* last host command header */
+ u32 isr0; /* isr status register LMPM_NIC_ISR0:
+ * rxtx_flag */
+ u32 isr1; /* isr status register LMPM_NIC_ISR1:
+ * host_flag */
+ u32 isr2; /* isr status register LMPM_NIC_ISR2:
+ * enc_flag */
+ u32 isr3; /* isr status register LMPM_NIC_ISR3:
+ * time_flag */
+ u32 isr4; /* isr status register LMPM_NIC_ISR4:
+ * wico interrupt */
+ u32 last_cmd_id; /* last HCMD id handled by the firmware */
+ u32 wait_event; /* wait event() caller address */
+ u32 l2p_control; /* L2pControlField */
+ u32 l2p_duration; /* L2pDurationField */
+ u32 l2p_mhvalid; /* L2pMhValidBits */
+ u32 l2p_addr_match; /* L2pAddrMatchStat */
+ u32 lmpm_pmg_sel; /* indicate which clocks are turned on
+ * (LMPM_PMG_SEL) */
+ u32 u_timestamp; /* indicate when the date and time of the
+ * compilation */
+ u32 flow_handler; /* FH read/write pointers, RX credit */
+} __packed /* LOG_ERROR_TABLE_API_S_VER_3 */;
+
+/*
+ * UMAC error struct - relevant starting from family 8000 chip.
+ * Note: This structure is read from the device with IO accesses,
+ * and the reading already does the endian conversion. As it is
+ * read with u32-sized accesses, any members with a different size
+ * need to be ordered correctly though!
+ */
+struct iwl_umac_error_event_table {
+ u32 valid; /* (nonzero) valid, (0) log is empty */
+ u32 error_id; /* type of error */
+ u32 blink1; /* branch link */
+ u32 blink2; /* branch link */
+ u32 ilink1; /* interrupt link */
+ u32 ilink2; /* interrupt link */
+ u32 data1; /* error-specific data */
+ u32 data2; /* error-specific data */
+ u32 data3; /* error-specific data */
+ u32 umac_major;
+ u32 umac_minor;
+ u32 frame_pointer; /* core register 27*/
+ u32 stack_pointer; /* core register 28 */
+ u32 cmd_header; /* latest host cmd sent to UMAC */
+ u32 nic_isr_pref; /* ISR status register */
+} __packed;
+
+#define ERROR_START_OFFSET (1 * sizeof(u32))
+#define ERROR_ELEM_SIZE (7 * sizeof(u32))
+
+static void iwl_fwrt_dump_umac_error_log(struct iwl_fw_runtime *fwrt)
+{
+ struct iwl_trans *trans = fwrt->trans;
+ struct iwl_umac_error_event_table table = {};
+ u32 base = fwrt->trans->dbg.umac_error_event_table;
+
+ if (!base &&
+ !(fwrt->trans->dbg.error_event_table_tlv_status &
+ IWL_ERROR_EVENT_TABLE_UMAC))
+ return;
+
+ iwl_trans_read_mem_bytes(trans, base, &table, sizeof(table));
+
+ if (table.valid)
+ fwrt->dump.umac_err_id = table.error_id;
+
+ if (ERROR_START_OFFSET <= table.valid * ERROR_ELEM_SIZE) {
+ IWL_ERR(trans, "Start IWL Error Log Dump:\n");
+ IWL_ERR(trans, "Transport status: 0x%08lX, valid: %d\n",
+ fwrt->trans->status, table.valid);
+ }
+
+ IWL_ERR(fwrt, "0x%08X | %s\n", table.error_id,
+ iwl_fw_lookup_assert_desc(table.error_id));
+ IWL_ERR(fwrt, "0x%08X | umac branchlink1\n", table.blink1);
+ IWL_ERR(fwrt, "0x%08X | umac branchlink2\n", table.blink2);
+ IWL_ERR(fwrt, "0x%08X | umac interruptlink1\n", table.ilink1);
+ IWL_ERR(fwrt, "0x%08X | umac interruptlink2\n", table.ilink2);
+ IWL_ERR(fwrt, "0x%08X | umac data1\n", table.data1);
+ IWL_ERR(fwrt, "0x%08X | umac data2\n", table.data2);
+ IWL_ERR(fwrt, "0x%08X | umac data3\n", table.data3);
+ IWL_ERR(fwrt, "0x%08X | umac major\n", table.umac_major);
+ IWL_ERR(fwrt, "0x%08X | umac minor\n", table.umac_minor);
+ IWL_ERR(fwrt, "0x%08X | frame pointer\n", table.frame_pointer);
+ IWL_ERR(fwrt, "0x%08X | stack pointer\n", table.stack_pointer);
+ IWL_ERR(fwrt, "0x%08X | last host cmd\n", table.cmd_header);
+ IWL_ERR(fwrt, "0x%08X | isr status reg\n", table.nic_isr_pref);
+}
+
+static void iwl_fwrt_dump_lmac_error_log(struct iwl_fw_runtime *fwrt, u8 lmac_num)
+{
+ struct iwl_trans *trans = fwrt->trans;
+ struct iwl_error_event_table table = {};
+ u32 val, base = fwrt->trans->dbg.lmac_error_event_table[lmac_num];
+
+ if (fwrt->cur_fw_img == IWL_UCODE_INIT) {
+ if (!base)
+ base = fwrt->fw->init_errlog_ptr;
+ } else {
+ if (!base)
+ base = fwrt->fw->inst_errlog_ptr;
+ }
+
+ if (base < 0x400000) {
+ IWL_ERR(fwrt,
+ "Not valid error log pointer 0x%08X for %s uCode\n",
+ base,
+ (fwrt->cur_fw_img == IWL_UCODE_INIT)
+ ? "Init" : "RT");
+ return;
+ }
+
+ /* check if there is a HW error */
+ val = iwl_trans_read_mem32(trans, base);
+ if (((val & ~0xf) == 0xa5a5a5a0) || ((val & ~0xf) == 0x5a5a5a50)) {
+ int err;
+
+ IWL_ERR(trans, "HW error, resetting before reading\n");
+
+ /* reset the device */
+ iwl_trans_sw_reset(trans);
+
+ err = iwl_finish_nic_init(trans, trans->trans_cfg);
+ if (err)
+ return;
+ }
+
+ iwl_trans_read_mem_bytes(trans, base, &table, sizeof(table));
+
+ if (table.valid)
+ fwrt->dump.lmac_err_id[lmac_num] = table.error_id;
+
+ if (ERROR_START_OFFSET <= table.valid * ERROR_ELEM_SIZE) {
+ IWL_ERR(trans, "Start IWL Error Log Dump:\n");
+ IWL_ERR(trans, "Transport status: 0x%08lX, valid: %d\n",
+ fwrt->trans->status, table.valid);
+ }
+
+ /* Do not change this output - scripts rely on it */
+
+ IWL_ERR(fwrt, "Loaded firmware version: %s\n", fwrt->fw->fw_version);
+
+ IWL_ERR(fwrt, "0x%08X | %-28s\n", table.error_id,
+ iwl_fw_lookup_assert_desc(table.error_id));
+ IWL_ERR(fwrt, "0x%08X | trm_hw_status0\n", table.trm_hw_status0);
+ IWL_ERR(fwrt, "0x%08X | trm_hw_status1\n", table.trm_hw_status1);
+ IWL_ERR(fwrt, "0x%08X | branchlink2\n", table.blink2);
+ IWL_ERR(fwrt, "0x%08X | interruptlink1\n", table.ilink1);
+ IWL_ERR(fwrt, "0x%08X | interruptlink2\n", table.ilink2);
+ IWL_ERR(fwrt, "0x%08X | data1\n", table.data1);
+ IWL_ERR(fwrt, "0x%08X | data2\n", table.data2);
+ IWL_ERR(fwrt, "0x%08X | data3\n", table.data3);
+ IWL_ERR(fwrt, "0x%08X | beacon time\n", table.bcon_time);
+ IWL_ERR(fwrt, "0x%08X | tsf low\n", table.tsf_low);
+ IWL_ERR(fwrt, "0x%08X | tsf hi\n", table.tsf_hi);
+ IWL_ERR(fwrt, "0x%08X | time gp1\n", table.gp1);
+ IWL_ERR(fwrt, "0x%08X | time gp2\n", table.gp2);
+ IWL_ERR(fwrt, "0x%08X | uCode revision type\n", table.fw_rev_type);
+ IWL_ERR(fwrt, "0x%08X | uCode version major\n", table.major);
+ IWL_ERR(fwrt, "0x%08X | uCode version minor\n", table.minor);
+ IWL_ERR(fwrt, "0x%08X | hw version\n", table.hw_ver);
+ IWL_ERR(fwrt, "0x%08X | board version\n", table.brd_ver);
+ IWL_ERR(fwrt, "0x%08X | hcmd\n", table.hcmd);
+ IWL_ERR(fwrt, "0x%08X | isr0\n", table.isr0);
+ IWL_ERR(fwrt, "0x%08X | isr1\n", table.isr1);
+ IWL_ERR(fwrt, "0x%08X | isr2\n", table.isr2);
+ IWL_ERR(fwrt, "0x%08X | isr3\n", table.isr3);
+ IWL_ERR(fwrt, "0x%08X | isr4\n", table.isr4);
+ IWL_ERR(fwrt, "0x%08X | last cmd Id\n", table.last_cmd_id);
+ IWL_ERR(fwrt, "0x%08X | wait_event\n", table.wait_event);
+ IWL_ERR(fwrt, "0x%08X | l2p_control\n", table.l2p_control);
+ IWL_ERR(fwrt, "0x%08X | l2p_duration\n", table.l2p_duration);
+ IWL_ERR(fwrt, "0x%08X | l2p_mhvalid\n", table.l2p_mhvalid);
+ IWL_ERR(fwrt, "0x%08X | l2p_addr_match\n", table.l2p_addr_match);
+ IWL_ERR(fwrt, "0x%08X | lmpm_pmg_sel\n", table.lmpm_pmg_sel);
+ IWL_ERR(fwrt, "0x%08X | timestamp\n", table.u_timestamp);
+ IWL_ERR(fwrt, "0x%08X | flow_handler\n", table.flow_handler);
+}
+
+/*
+ * TCM error struct.
+ * Note: This structure is read from the device with IO accesses,
+ * and the reading already does the endian conversion. As it is
+ * read with u32-sized accesses, any members with a different size
+ * need to be ordered correctly though!
+ */
+struct iwl_tcm_error_event_table {
+ u32 valid;
+ u32 error_id;
+ u32 blink2;
+ u32 ilink1;
+ u32 ilink2;
+ u32 data1, data2, data3;
+ u32 logpc;
+ u32 frame_pointer;
+ u32 stack_pointer;
+ u32 msgid;
+ u32 isr;
+ u32 hw_status[5];
+ u32 sw_status[1];
+ u32 reserved[4];
+} __packed; /* TCM_LOG_ERROR_TABLE_API_S_VER_1 */
+
+static void iwl_fwrt_dump_tcm_error_log(struct iwl_fw_runtime *fwrt)
+{
+ struct iwl_trans *trans = fwrt->trans;
+ struct iwl_tcm_error_event_table table = {};
+ u32 base = fwrt->trans->dbg.tcm_error_event_table;
+ int i;
+
+ if (!base ||
+ !(fwrt->trans->dbg.error_event_table_tlv_status &
+ IWL_ERROR_EVENT_TABLE_TCM))
+ return;
+
+ iwl_trans_read_mem_bytes(trans, base, &table, sizeof(table));
+
+ IWL_ERR(fwrt, "TCM status:\n");
+ IWL_ERR(fwrt, "0x%08X | error ID\n", table.error_id);
+ IWL_ERR(fwrt, "0x%08X | tcm branchlink2\n", table.blink2);
+ IWL_ERR(fwrt, "0x%08X | tcm interruptlink1\n", table.ilink1);
+ IWL_ERR(fwrt, "0x%08X | tcm interruptlink2\n", table.ilink2);
+ IWL_ERR(fwrt, "0x%08X | tcm data1\n", table.data1);
+ IWL_ERR(fwrt, "0x%08X | tcm data2\n", table.data2);
+ IWL_ERR(fwrt, "0x%08X | tcm data3\n", table.data3);
+ IWL_ERR(fwrt, "0x%08X | tcm log PC\n", table.logpc);
+ IWL_ERR(fwrt, "0x%08X | tcm frame pointer\n", table.frame_pointer);
+ IWL_ERR(fwrt, "0x%08X | tcm stack pointer\n", table.stack_pointer);
+ IWL_ERR(fwrt, "0x%08X | tcm msg ID\n", table.msgid);
+ IWL_ERR(fwrt, "0x%08X | tcm ISR status\n", table.isr);
+ for (i = 0; i < ARRAY_SIZE(table.hw_status); i++)
+ IWL_ERR(fwrt, "0x%08X | tcm HW status[%d]\n",
+ table.hw_status[i], i);
+ for (i = 0; i < ARRAY_SIZE(table.sw_status); i++)
+ IWL_ERR(fwrt, "0x%08X | tcm SW status[%d]\n",
+ table.sw_status[i], i);
+}
+
+static void iwl_fwrt_dump_iml_error_log(struct iwl_fw_runtime *fwrt)
+{
+ struct iwl_trans *trans = fwrt->trans;
+ u32 error, data1;
+
+ if (fwrt->trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_22000) {
+ error = UMAG_SB_CPU_2_STATUS;
+ data1 = UMAG_SB_CPU_1_STATUS;
+ } else if (fwrt->trans->trans_cfg->device_family >=
+ IWL_DEVICE_FAMILY_8000) {
+ error = SB_CPU_2_STATUS;
+ data1 = SB_CPU_1_STATUS;
+ } else {
+ return;
+ }
+
+ error = iwl_read_umac_prph(trans, UMAG_SB_CPU_2_STATUS);
+
+ IWL_ERR(trans, "IML/ROM dump:\n");
+
+ if (error & 0xFFFF0000)
+ IWL_ERR(trans, "0x%04X | IML/ROM SYSASSERT\n", error >> 16);
+
+ IWL_ERR(fwrt, "0x%08X | IML/ROM error/state\n", error);
+ IWL_ERR(fwrt, "0x%08X | IML/ROM data1\n",
+ iwl_read_umac_prph(trans, data1));
+
+ if (fwrt->trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_22000)
+ IWL_ERR(fwrt, "0x%08X | IML/ROM WFPM_AUTH_KEY_0\n",
+ iwl_read_umac_prph(trans, SB_MODIFY_CFG_FLAG));
+}
+
+#define FSEQ_REG(x) { .addr = (x), .str = #x, }
+
+static void iwl_fwrt_dump_fseq_regs(struct iwl_fw_runtime *fwrt)
+{
+ struct iwl_trans *trans = fwrt->trans;
+ int i;
+ struct {
+ u32 addr;
+ const char *str;
+ } fseq_regs[] = {
+ FSEQ_REG(FSEQ_ERROR_CODE),
+ FSEQ_REG(FSEQ_TOP_INIT_VERSION),
+ FSEQ_REG(FSEQ_CNVIO_INIT_VERSION),
+ FSEQ_REG(FSEQ_OTP_VERSION),
+ FSEQ_REG(FSEQ_TOP_CONTENT_VERSION),
+ FSEQ_REG(FSEQ_ALIVE_TOKEN),
+ FSEQ_REG(FSEQ_CNVI_ID),
+ FSEQ_REG(FSEQ_CNVR_ID),
+ FSEQ_REG(CNVI_AUX_MISC_CHIP),
+ FSEQ_REG(CNVR_AUX_MISC_CHIP),
+ FSEQ_REG(CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM),
+ FSEQ_REG(CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR),
+ };
+
+ if (!iwl_trans_grab_nic_access(trans))
+ return;
+
+ IWL_ERR(fwrt, "Fseq Registers:\n");
+
+ for (i = 0; i < ARRAY_SIZE(fseq_regs); i++)
+ IWL_ERR(fwrt, "0x%08X | %s\n",
+ iwl_read_prph_no_grab(trans, fseq_regs[i].addr),
+ fseq_regs[i].str);
+
+ iwl_trans_release_nic_access(trans);
+}
+
+void iwl_fwrt_dump_error_logs(struct iwl_fw_runtime *fwrt)
+{
+ if (!test_bit(STATUS_DEVICE_ENABLED, &fwrt->trans->status)) {
+ IWL_ERR(fwrt,
+ "DEVICE_ENABLED bit is not set. Aborting dump.\n");
+ return;
+ }
+
+ iwl_fwrt_dump_lmac_error_log(fwrt, 0);
+ if (fwrt->trans->dbg.lmac_error_event_table[1])
+ iwl_fwrt_dump_lmac_error_log(fwrt, 1);
+ iwl_fwrt_dump_umac_error_log(fwrt);
+ iwl_fwrt_dump_tcm_error_log(fwrt);
+ iwl_fwrt_dump_iml_error_log(fwrt);
+ iwl_fwrt_dump_fseq_regs(fwrt);
+}
+IWL_EXPORT_SYMBOL(iwl_fwrt_dump_error_logs);
/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
/*
- * Copyright (C) 2008-2014, 2018-2020 Intel Corporation
+ * Copyright (C) 2008-2014, 2018-2021 Intel Corporation
* Copyright (C) 2013-2015 Intel Mobile Communications GmbH
* Copyright (C) 2016-2017 Intel Deutschland GmbH
*/
IWL_UCODE_TLV_INIT_DATA = 4,
IWL_UCODE_TLV_BOOT = 5,
IWL_UCODE_TLV_PROBE_MAX_LEN = 6, /* a u32 value */
- IWL_UCODE_TLV_PAN = 7,
+ IWL_UCODE_TLV_PAN = 7, /* deprecated -- only used in DVM */
+ IWL_UCODE_TLV_MEM_DESC = 7, /* replaces PAN in non-DVM */
IWL_UCODE_TLV_RUNT_EVTLOG_PTR = 8,
IWL_UCODE_TLV_RUNT_EVTLOG_SIZE = 9,
IWL_UCODE_TLV_RUNT_ERRLOG_PTR = 10,
IWL_UCODE_TLV_PNVM_VERSION = 62,
IWL_UCODE_TLV_PNVM_SKU = 64,
+ IWL_UCODE_TLV_TCM_DEBUG_ADDRS = 65,
IWL_UCODE_TLV_FW_NUM_STATIONS = IWL_UCODE_TLV_CONST_BASE + 0,
IWL_UCODE_TLV_API_BAND_IN_RX_DATA = (__force iwl_ucode_tlv_api_t)59,
- NUM_IWL_UCODE_TLV_API
#ifdef __CHECKER__
- /* sparse says it cannot increment the previous enum member */
- = 128
+ /* sparse says it cannot increment the previous enum member */
+#define NUM_IWL_UCODE_TLV_API 128
+#else
+ NUM_IWL_UCODE_TLV_API
#endif
};
IWL_UCODE_TLV_CAPA_PROTECTED_TWT = (__force iwl_ucode_tlv_capa_t)56,
IWL_UCODE_TLV_CAPA_FW_RESET_HANDSHAKE = (__force iwl_ucode_tlv_capa_t)57,
IWL_UCODE_TLV_CAPA_PASSIVE_6GHZ_SCAN = (__force iwl_ucode_tlv_capa_t)58,
+ IWL_UCODE_TLV_CAPA_BROADCAST_TWT = (__force iwl_ucode_tlv_capa_t)60,
/* set 2 */
IWL_UCODE_TLV_CAPA_EXTENDED_DTS_MEASURE = (__force iwl_ucode_tlv_capa_t)64,
IWL_UCODE_TLV_CAPA_BIGTK_SUPPORT = (__force iwl_ucode_tlv_capa_t)100,
IWL_UCODE_TLV_CAPA_RFIM_SUPPORT = (__force iwl_ucode_tlv_capa_t)102,
- NUM_IWL_UCODE_TLV_CAPA
#ifdef __CHECKER__
- /* sparse says it cannot increment the previous enum member */
- = 128
+ /* sparse says it cannot increment the previous enum member */
+#define NUM_IWL_UCODE_TLV_CAPA 128
+#else
+ NUM_IWL_UCODE_TLV_CAPA
#endif
};
u8 notif_ver;
} __packed;
+struct iwl_fw_tcm_error_addr {
+ __le32 addr;
+}; /* FW_TLV_TCM_ERROR_INFO_ADDRS_S */
+
static inline size_t _iwl_tlv_array_len(const struct iwl_ucode_tlv *tlv,
size_t fixed_size, size_t var_size)
{
#include "fw/api/commands.h"
#include "fw/api/nvm-reg.h"
#include "fw/api/alive.h"
-#include <linux/efi.h>
+#include "fw/uefi.h"
struct iwl_pnvm_section {
__le32 offset;
return -ENOENT;
}
-#if defined(CONFIG_EFI)
-
-#define IWL_EFI_VAR_GUID EFI_GUID(0x92daaf2f, 0xc02b, 0x455b, \
- 0xb2, 0xec, 0xf5, 0xa3, \
- 0x59, 0x4f, 0x4a, 0xea)
-
-#define IWL_UEFI_OEM_PNVM_NAME L"UefiCnvWlanOemSignedPnvm"
-
-#define IWL_HARDCODED_PNVM_SIZE 4096
-
-struct pnvm_sku_package {
- u8 rev;
- u8 reserved1[3];
- u32 total_size;
- u8 n_skus;
- u8 reserved2[11];
- u8 data[];
-};
-
-static int iwl_pnvm_get_from_efi(struct iwl_trans *trans,
- u8 **data, size_t *len)
-{
- struct efivar_entry *pnvm_efivar;
- struct pnvm_sku_package *package;
- unsigned long package_size;
- int err;
-
- pnvm_efivar = kzalloc(sizeof(*pnvm_efivar), GFP_KERNEL);
- if (!pnvm_efivar)
- return -ENOMEM;
-
- memcpy(&pnvm_efivar->var.VariableName, IWL_UEFI_OEM_PNVM_NAME,
- sizeof(IWL_UEFI_OEM_PNVM_NAME));
- pnvm_efivar->var.VendorGuid = IWL_EFI_VAR_GUID;
-
- /*
- * TODO: we hardcode a maximum length here, because reading
- * from the UEFI is not working. To implement this properly,
- * we have to call efivar_entry_size().
- */
- package_size = IWL_HARDCODED_PNVM_SIZE;
-
- package = kmalloc(package_size, GFP_KERNEL);
- if (!package) {
- err = -ENOMEM;
- goto out;
- }
-
- err = efivar_entry_get(pnvm_efivar, NULL, &package_size, package);
- if (err) {
- IWL_DEBUG_FW(trans,
- "PNVM UEFI variable not found %d (len %lu)\n",
- err, package_size);
- goto out;
- }
-
- IWL_DEBUG_FW(trans, "Read PNVM fro UEFI with size %lu\n", package_size);
-
- *data = kmemdup(package->data, *len, GFP_KERNEL);
- if (!*data)
- err = -ENOMEM;
- *len = package_size - sizeof(*package);
-
-out:
- kfree(package);
- kfree(pnvm_efivar);
-
- return err;
-}
-#else /* CONFIG_EFI */
-static inline int iwl_pnvm_get_from_efi(struct iwl_trans *trans,
- u8 **data, size_t *len)
-{
- return -EOPNOTSUPP;
-}
-#endif /* CONFIG_EFI */
-
static int iwl_pnvm_get_from_fs(struct iwl_trans *trans, u8 **data, size_t *len)
{
const struct firmware *pnvm;
{
u8 *data;
size_t len;
+ struct pnvm_sku_package *package;
struct iwl_notification_wait pnvm_wait;
static const u16 ntf_cmds[] = { WIDE_ID(REGULATORY_AND_NVM_GROUP,
PNVM_INIT_COMPLETE_NTFY) };
}
/* First attempt to get the PNVM from BIOS */
- ret = iwl_pnvm_get_from_efi(trans, &data, &len);
- if (!ret)
- goto parse;
+ package = iwl_uefi_get_pnvm(trans, &len);
+ if (!IS_ERR_OR_NULL(package)) {
+ data = kmemdup(package->data, len, GFP_KERNEL);
+
+ /* free package regardless of whether kmemdup succeeded */
+ kfree(package);
+
+ if (data) {
+ /* we need only the data size */
+ len -= sizeof(*package);
+ goto parse;
+ }
+ }
/* If it's not available, try from the filesystem */
ret = iwl_pnvm_get_from_fs(trans, &data, &len);
kfree(data);
skip_parse:
+ data = NULL;
+ /* now try to get the reduce power table, if not loaded yet */
+ if (!trans->reduce_power_loaded) {
+ data = iwl_uefi_get_reduced_power(trans, &len);
+ if (IS_ERR_OR_NULL(data)) {
+ /*
+ * Pretend we've loaded it - at least we've tried and
+ * couldn't load it at all, so there's no point in
+ * trying again over and over.
+ */
+ trans->reduce_power_loaded = true;
+
+ goto skip_reduce_power;
+ }
+ }
+
+ ret = iwl_trans_set_reduce_power(trans, data, len);
+ if (ret)
+ IWL_DEBUG_FW(trans,
+ "Failed to set reduce power table %d\n",
+ ret);
+ kfree(data);
+
+skip_reduce_power:
iwl_init_notification_wait(notif_wait, &pnvm_wait,
ntf_cmds, ARRAY_SIZE(ntf_cmds),
iwl_pnvm_complete_fn, trans);
/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
/******************************************************************************
*
- * Copyright(c) 2020 Intel Corporation
+ * Copyright(c) 2020-2021 Intel Corporation
*
*****************************************************************************/
#include "fw/notif-wait.h"
-#define MVM_UCODE_PNVM_TIMEOUT (HZ / 10)
+#define MVM_UCODE_PNVM_TIMEOUT (HZ / 4)
int iwl_pnvm_load(struct iwl_trans *trans,
struct iwl_notif_wait_data *notif_wait);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * Copyright(c) 2021 Intel Corporation
+ */
+
+#include "iwl-drv.h"
+#include "pnvm.h"
+#include "iwl-prph.h"
+#include "iwl-io.h"
+
+#include "fw/uefi.h"
+#include "fw/api/alive.h"
+#include <linux/efi.h>
+
+#define IWL_EFI_VAR_GUID EFI_GUID(0x92daaf2f, 0xc02b, 0x455b, \
+ 0xb2, 0xec, 0xf5, 0xa3, \
+ 0x59, 0x4f, 0x4a, 0xea)
+
+void *iwl_uefi_get_pnvm(struct iwl_trans *trans, size_t *len)
+{
+ struct efivar_entry *pnvm_efivar;
+ void *data;
+ unsigned long package_size;
+ int err;
+
+ *len = 0;
+
+ pnvm_efivar = kzalloc(sizeof(*pnvm_efivar), GFP_KERNEL);
+ if (!pnvm_efivar)
+ return ERR_PTR(-ENOMEM);
+
+ memcpy(&pnvm_efivar->var.VariableName, IWL_UEFI_OEM_PNVM_NAME,
+ sizeof(IWL_UEFI_OEM_PNVM_NAME));
+ pnvm_efivar->var.VendorGuid = IWL_EFI_VAR_GUID;
+
+ /*
+ * TODO: we hardcode a maximum length here, because reading
+ * from the UEFI is not working. To implement this properly,
+ * we have to call efivar_entry_size().
+ */
+ package_size = IWL_HARDCODED_PNVM_SIZE;
+
+ data = kmalloc(package_size, GFP_KERNEL);
+ if (!data) {
+ data = ERR_PTR(-ENOMEM);
+ goto out;
+ }
+
+ err = efivar_entry_get(pnvm_efivar, NULL, &package_size, data);
+ if (err) {
+ IWL_DEBUG_FW(trans,
+ "PNVM UEFI variable not found %d (len %zd)\n",
+ err, package_size);
+ kfree(data);
+ data = ERR_PTR(err);
+ goto out;
+ }
+
+ IWL_DEBUG_FW(trans, "Read PNVM from UEFI with size %zd\n", package_size);
+ *len = package_size;
+
+out:
+ kfree(pnvm_efivar);
+
+ return data;
+}
+
+static void *iwl_uefi_reduce_power_section(struct iwl_trans *trans,
+ const u8 *data, size_t len)
+{
+ struct iwl_ucode_tlv *tlv;
+ u8 *reduce_power_data = NULL, *tmp;
+ u32 size = 0;
+
+ IWL_DEBUG_FW(trans, "Handling REDUCE_POWER section\n");
+
+ while (len >= sizeof(*tlv)) {
+ u32 tlv_len, tlv_type;
+
+ len -= sizeof(*tlv);
+ tlv = (void *)data;
+
+ tlv_len = le32_to_cpu(tlv->length);
+ tlv_type = le32_to_cpu(tlv->type);
+
+ if (len < tlv_len) {
+ IWL_ERR(trans, "invalid TLV len: %zd/%u\n",
+ len, tlv_len);
+ reduce_power_data = ERR_PTR(-EINVAL);
+ goto out;
+ }
+
+ data += sizeof(*tlv);
+
+ switch (tlv_type) {
+ case IWL_UCODE_TLV_MEM_DESC: {
+ IWL_DEBUG_FW(trans,
+ "Got IWL_UCODE_TLV_MEM_DESC len %d\n",
+ tlv_len);
+
+ IWL_DEBUG_FW(trans, "Adding data (size %d)\n", tlv_len);
+
+ tmp = krealloc(reduce_power_data, size + tlv_len, GFP_KERNEL);
+ if (!tmp) {
+ IWL_DEBUG_FW(trans,
+ "Couldn't allocate (more) reduce_power_data\n");
+
+ reduce_power_data = ERR_PTR(-ENOMEM);
+ goto out;
+ }
+
+ reduce_power_data = tmp;
+
+ memcpy(reduce_power_data + size, data, tlv_len);
+
+ size += tlv_len;
+
+ break;
+ }
+ case IWL_UCODE_TLV_PNVM_SKU:
+ IWL_DEBUG_FW(trans,
+ "New REDUCE_POWER section started, stop parsing.\n");
+ goto done;
+ default:
+ IWL_DEBUG_FW(trans, "Found TLV 0x%0x, len %d\n",
+ tlv_type, tlv_len);
+ break;
+ }
+
+ len -= ALIGN(tlv_len, 4);
+ data += ALIGN(tlv_len, 4);
+ }
+
+done:
+ if (!size) {
+ IWL_DEBUG_FW(trans, "Empty REDUCE_POWER, skipping.\n");
+ reduce_power_data = ERR_PTR(-ENOENT);
+ goto out;
+ }
+
+ IWL_INFO(trans, "loaded REDUCE_POWER\n");
+
+out:
+ return reduce_power_data;
+}
+
+static void *iwl_uefi_reduce_power_parse(struct iwl_trans *trans,
+ const u8 *data, size_t len)
+{
+ struct iwl_ucode_tlv *tlv;
+ void *sec_data;
+
+ IWL_DEBUG_FW(trans, "Parsing REDUCE_POWER data\n");
+
+ while (len >= sizeof(*tlv)) {
+ u32 tlv_len, tlv_type;
+
+ len -= sizeof(*tlv);
+ tlv = (void *)data;
+
+ tlv_len = le32_to_cpu(tlv->length);
+ tlv_type = le32_to_cpu(tlv->type);
+
+ if (len < tlv_len) {
+ IWL_ERR(trans, "invalid TLV len: %zd/%u\n",
+ len, tlv_len);
+ return ERR_PTR(-EINVAL);
+ }
+
+ if (tlv_type == IWL_UCODE_TLV_PNVM_SKU) {
+ struct iwl_sku_id *sku_id =
+ (void *)(data + sizeof(*tlv));
+
+ IWL_DEBUG_FW(trans,
+ "Got IWL_UCODE_TLV_PNVM_SKU len %d\n",
+ tlv_len);
+ IWL_DEBUG_FW(trans, "sku_id 0x%0x 0x%0x 0x%0x\n",
+ le32_to_cpu(sku_id->data[0]),
+ le32_to_cpu(sku_id->data[1]),
+ le32_to_cpu(sku_id->data[2]));
+
+ data += sizeof(*tlv) + ALIGN(tlv_len, 4);
+ len -= ALIGN(tlv_len, 4);
+
+ if (trans->sku_id[0] == le32_to_cpu(sku_id->data[0]) &&
+ trans->sku_id[1] == le32_to_cpu(sku_id->data[1]) &&
+ trans->sku_id[2] == le32_to_cpu(sku_id->data[2])) {
+ sec_data = iwl_uefi_reduce_power_section(trans,
+ data,
+ len);
+ if (!IS_ERR(sec_data))
+ return sec_data;
+ } else {
+ IWL_DEBUG_FW(trans, "SKU ID didn't match!\n");
+ }
+ } else {
+ data += sizeof(*tlv) + ALIGN(tlv_len, 4);
+ len -= ALIGN(tlv_len, 4);
+ }
+ }
+
+ return ERR_PTR(-ENOENT);
+}
+
+void *iwl_uefi_get_reduced_power(struct iwl_trans *trans, size_t *len)
+{
+ struct efivar_entry *reduce_power_efivar;
+ struct pnvm_sku_package *package;
+ void *data = NULL;
+ unsigned long package_size;
+ int err;
+
+ *len = 0;
+
+ reduce_power_efivar = kzalloc(sizeof(*reduce_power_efivar), GFP_KERNEL);
+ if (!reduce_power_efivar)
+ return ERR_PTR(-ENOMEM);
+
+ memcpy(&reduce_power_efivar->var.VariableName, IWL_UEFI_REDUCED_POWER_NAME,
+ sizeof(IWL_UEFI_REDUCED_POWER_NAME));
+ reduce_power_efivar->var.VendorGuid = IWL_EFI_VAR_GUID;
+
+ /*
+ * TODO: we hardcode a maximum length here, because reading
+ * from the UEFI is not working. To implement this properly,
+ * we have to call efivar_entry_size().
+ */
+ package_size = IWL_HARDCODED_REDUCE_POWER_SIZE;
+
+ package = kmalloc(package_size, GFP_KERNEL);
+ if (!package) {
+ package = ERR_PTR(-ENOMEM);
+ goto out;
+ }
+
+ err = efivar_entry_get(reduce_power_efivar, NULL, &package_size, package);
+ if (err) {
+ IWL_DEBUG_FW(trans,
+ "Reduced Power UEFI variable not found %d (len %lu)\n",
+ err, package_size);
+ kfree(package);
+ data = ERR_PTR(err);
+ goto out;
+ }
+
+ IWL_DEBUG_FW(trans, "Read reduced power from UEFI with size %lu\n",
+ package_size);
+ *len = package_size;
+
+ IWL_DEBUG_FW(trans, "rev %d, total_size %d, n_skus %d\n",
+ package->rev, package->total_size, package->n_skus);
+
+ data = iwl_uefi_reduce_power_parse(trans, package->data,
+ *len - sizeof(*package));
+
+ kfree(package);
+
+out:
+ kfree(reduce_power_efivar);
+
+ return data;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright(c) 2021 Intel Corporation
+ */
+
+
+#define IWL_UEFI_OEM_PNVM_NAME L"UefiCnvWlanOemSignedPnvm"
+#define IWL_UEFI_REDUCED_POWER_NAME L"UefiCnvWlanReducedPower"
+
+/*
+ * TODO: we have these hardcoded values that the caller must pass,
+ * because reading from the UEFI is not working. To implement this
+ * properly, we have to change iwl_pnvm_get_from_uefi() to call
+ * efivar_entry_size() and return the value to the caller instead.
+ */
+#define IWL_HARDCODED_PNVM_SIZE 4096
+#define IWL_HARDCODED_REDUCE_POWER_SIZE 32768
+
+struct pnvm_sku_package {
+ u8 rev;
+ u32 total_size;
+ u8 n_skus;
+ u32 reserved[2];
+ u8 data[];
+} __packed;
+
+#ifdef CONFIG_EFI
+void *iwl_uefi_get_pnvm(struct iwl_trans *trans, size_t *len);
+void *iwl_uefi_get_reduced_power(struct iwl_trans *trans, size_t *len);
+#else /* CONFIG_EFI */
+static inline
+void *iwl_uefi_get_pnvm(struct iwl_trans *trans, size_t *len)
+{
+ return ERR_PTR(-EOPNOTSUPP);
+}
+
+static inline
+void *iwl_uefi_get_reduced_power(struct iwl_trans *trans, size_t *len)
+{
+ return ERR_PTR(-EOPNOTSUPP);
+}
+#endif /* CONFIG_EFI */
#define IWL_CFG_RF_TYPE_HR1 0x10C
#define IWL_CFG_RF_TYPE_GF 0x10D
#define IWL_CFG_RF_TYPE_MR 0x110
+#define IWL_CFG_RF_TYPE_FM 0x112
#define IWL_CFG_RF_ID_TH 0x1
#define IWL_CFG_RF_ID_TH1 0x1
extern const char iwl_ax201_killer_1650i_name[];
extern const char iwl_ax210_killer_1675w_name[];
extern const char iwl_ax210_killer_1675x_name[];
+extern const char iwl9560_killer_1550i_160_name[];
+extern const char iwl9560_killer_1550s_160_name[];
extern const char iwl_ax211_name[];
extern const char iwl_ax221_name[];
+extern const char iwl_ax231_name[];
extern const char iwl_ax411_name[];
#if IS_ENABLED(CONFIG_IWLDVM)
extern const struct iwl_cfg iwl5300_agn_cfg;
extern const struct iwl_cfg iwl_qu_c0_hr_b0;
extern const struct iwl_cfg iwl_ax200_cfg_cc;
extern const struct iwl_cfg iwl_ax201_cfg_qu_hr;
-extern const struct iwl_cfg iwl_ax201_cfg_qu_hr;
extern const struct iwl_cfg iwl_ax201_cfg_qu_c0_hr_b0;
extern const struct iwl_cfg iwl_ax201_cfg_quz_hr;
extern const struct iwl_cfg iwl_ax1650i_cfg_quz_hr;
extern const struct iwl_cfg iwl_cfg_ma_a0_gf_a0;
extern const struct iwl_cfg iwl_cfg_ma_a0_gf4_a0;
extern const struct iwl_cfg iwl_cfg_ma_a0_mr_a0;
+extern const struct iwl_cfg iwl_cfg_ma_a0_fm_a0;
extern const struct iwl_cfg iwl_cfg_snj_a0_mr_a0;
extern const struct iwl_cfg iwl_cfg_so_a0_hr_a0;
extern const struct iwl_cfg iwl_cfg_quz_a0_hr_b0;
/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
/*
- * Copyright (C) 2018, 2020 Intel Corporation
+ * Copyright (C) 2018, 2020-2021 Intel Corporation
*/
#ifndef __iwl_context_info_file_gen3_h__
#define __iwl_context_info_file_gen3_h__
} __packed; /* PERIPH_SCRATCH_RBD_CFG_S */
/*
+ * struct iwl_prph_scratch_uefi_cfg - prph scratch reduce power table
+ * @base_addr: reduce power table address
+ * @size: table size in dwords
+ */
+struct iwl_prph_scratch_uefi_cfg {
+ __le64 base_addr;
+ __le32 size;
+ __le32 reserved;
+} __packed; /* PERIPH_SCRATCH_UEFI_CFG_S */
+
+/*
* struct iwl_prph_scratch_ctrl_cfg - prph scratch ctrl and config
* @version: version information of context info and HW
* @control: control flags of FH configurations
struct iwl_prph_scratch_pnvm_cfg pnvm_cfg;
struct iwl_prph_scratch_hwm_cfg hwm_cfg;
struct iwl_prph_scratch_rbd_cfg rbd_cfg;
+ struct iwl_prph_scratch_uefi_cfg reduce_power_cfg;
} __packed; /* PERIPH_SCRATCH_CTRL_CFG_S */
/*
*/
struct iwl_prph_scratch {
struct iwl_prph_scratch_ctrl_cfg ctrl_cfg;
- __le32 reserved[16];
+ __le32 reserved[12];
struct iwl_context_info_dram dram;
} __packed; /* PERIPH_SCRATCH_S */
int iwl_pcie_ctxt_info_gen3_init(struct iwl_trans *trans,
const struct fw_img *fw);
-void iwl_pcie_ctxt_info_gen3_free(struct iwl_trans *trans);
+void iwl_pcie_ctxt_info_gen3_free(struct iwl_trans *trans, bool alive);
int iwl_trans_pcie_ctx_info_gen3_set_pnvm(struct iwl_trans *trans,
const void *data, u32 len);
+int iwl_trans_pcie_ctx_info_gen3_set_reduce_power(struct iwl_trans *trans,
+ const void *data, u32 len);
#endif /* __iwl_context_info_file_gen3_h__ */
/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
/*
- * Copyright (C) 2005-2014, 2018-2020 Intel Corporation
+ * Copyright (C) 2005-2014, 2018-2021 Intel Corporation
* Copyright (C) 2013-2014 Intel Mobile Communications GmbH
* Copyright (C) 2016 Intel Deutschland GmbH
*/
#define CSR_HW_RF_ID_TYPE_GF (0x0010D000)
#define CSR_HW_RF_ID_TYPE_GF4 (0x0010E000)
-/* HW_RF CHIP ID */
-#define CSR_HW_RF_ID_TYPE_CHIP_ID(_val) (((_val) >> 12) & 0xFFF)
-
/* HW_RF CHIP STEP */
#define CSR_HW_RF_STEP(_val) (((_val) >> 8) & 0xF)
[IWL_DBG_TLV_TYPE_DEBUG_INFO] = {.min_ver = 1, .max_ver = 1,},
[IWL_DBG_TLV_TYPE_BUF_ALLOC] = {.min_ver = 1, .max_ver = 1,},
[IWL_DBG_TLV_TYPE_HCMD] = {.min_ver = 1, .max_ver = 1,},
- [IWL_DBG_TLV_TYPE_REGION] = {.min_ver = 1, .max_ver = 1,},
+ [IWL_DBG_TLV_TYPE_REGION] = {.min_ver = 1, .max_ver = 2,},
[IWL_DBG_TLV_TYPE_TRIGGER] = {.min_ver = 1, .max_ver = 1,},
};
u32 type = le32_to_cpu(reg->type);
u32 tlv_len = sizeof(*tlv) + le32_to_cpu(tlv->length);
+ /*
+ * The higher part of the ID in version 2 is irrelevant for
+ * us, so mask it out.
+ */
+ if (le32_to_cpu(reg->hdr.version) == 2)
+ id &= IWL_FW_INI_REGION_V2_MASK;
+
if (le32_to_cpu(tlv->length) < sizeof(*reg))
return -EINVAL;
+ /* for safe use of a string from FW, limit it to IWL_FW_INI_MAX_NAME */
+ IWL_DEBUG_FW(trans, "WRT: parsing region: %.*s\n",
+ IWL_FW_INI_MAX_NAME, reg->name);
+
if (id >= IWL_FW_INI_MAX_REGION_ID) {
IWL_ERR(trans, "WRT: Invalid region id %u\n", id);
return -EINVAL;
IWL_ERROR_EVENT_TABLE_LMAC1;
break;
}
+ case IWL_UCODE_TLV_TCM_DEBUG_ADDRS: {
+ struct iwl_fw_tcm_error_addr *ptr = (void *)tlv_data;
+
+ if (tlv_len != sizeof(*ptr))
+ goto invalid_tlv_len;
+ drv->trans->dbg.tcm_error_event_table =
+ le32_to_cpu(ptr->addr) & ~FW_ADDR_CACHE_CONTROL;
+ drv->trans->dbg.error_event_table_tlv_status |=
+ IWL_ERROR_EVENT_TABLE_TCM;
+ break;
+ }
case IWL_UCODE_TLV_TYPE_DEBUG_INFO:
case IWL_UCODE_TLV_TYPE_BUFFER_ALLOCATION:
case IWL_UCODE_TLV_TYPE_HCMD:
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/*
- * Copyright (C) 2005-2014, 2018-2020 Intel Corporation
+ * Copyright (C) 2005-2014, 2018-2021 Intel Corporation
* Copyright (C) 2013-2015 Intel Mobile Communications GmbH
* Copyright (C) 2016-2017 Intel Deutschland GmbH
*/
.mac_cap_info[2] =
IEEE80211_HE_MAC_CAP2_32BIT_BA_BITMAP,
.mac_cap_info[3] =
- IEEE80211_HE_MAC_CAP3_OMI_CONTROL |
- IEEE80211_HE_MAC_CAP3_MAX_AMPDU_LEN_EXP_EXT_2,
+ IEEE80211_HE_MAC_CAP3_OMI_CONTROL,
.mac_cap_info[4] =
IEEE80211_HE_MAC_CAP4_AMSDU_IN_AMPDU |
IEEE80211_HE_MAC_CAP4_MULTI_TID_AGG_TX_QOS_B39,
IEEE80211_HE_PHY_CAP4_SU_BEAMFORMEE |
IEEE80211_HE_PHY_CAP4_BEAMFORMEE_MAX_STS_ABOVE_80MHZ_8 |
IEEE80211_HE_PHY_CAP4_BEAMFORMEE_MAX_STS_UNDER_80MHZ_8,
- .phy_cap_info[5] =
- IEEE80211_HE_PHY_CAP5_BEAMFORMEE_NUM_SND_DIM_UNDER_80MHZ_2 |
- IEEE80211_HE_PHY_CAP5_BEAMFORMEE_NUM_SND_DIM_ABOVE_80MHZ_2,
.phy_cap_info[6] =
IEEE80211_HE_PHY_CAP6_TRIG_SU_BEAMFORMING_FB |
IEEE80211_HE_PHY_CAP6_TRIG_MU_BEAMFORMING_PARTIAL_BW_FB |
IEEE80211_HE_PHY_CAP6_PPE_THRESHOLD_PRESENT,
.phy_cap_info[7] =
IEEE80211_HE_PHY_CAP7_POWER_BOOST_FACTOR_SUPP |
- IEEE80211_HE_PHY_CAP7_HE_SU_MU_PPDU_4XLTF_AND_08_US_GI |
- IEEE80211_HE_PHY_CAP7_MAX_NC_1,
+ IEEE80211_HE_PHY_CAP7_HE_SU_MU_PPDU_4XLTF_AND_08_US_GI,
.phy_cap_info[8] =
IEEE80211_HE_PHY_CAP8_HE_ER_SU_PPDU_4XLTF_AND_08_US_GI |
IEEE80211_HE_PHY_CAP8_20MHZ_IN_40MHZ_HE_PPDU_IN_2G |
IEEE80211_HE_PHY_CAP8_20MHZ_IN_160MHZ_HE_PPDU |
IEEE80211_HE_PHY_CAP8_80MHZ_IN_160MHZ_HE_PPDU |
- IEEE80211_HE_PHY_CAP8_DCM_MAX_RU_2x996,
+ IEEE80211_HE_PHY_CAP8_DCM_MAX_RU_242,
.phy_cap_info[9] =
- IEEE80211_HE_PHY_CAP9_NON_TRIGGERED_CQI_FEEDBACK |
IEEE80211_HE_PHY_CAP9_RX_FULL_BW_SU_USING_MU_WITH_COMP_SIGB |
IEEE80211_HE_PHY_CAP9_RX_FULL_BW_SU_USING_MU_WITH_NON_COMP_SIGB |
IEEE80211_HE_PHY_CAP9_NOMIMAL_PKT_PADDING_RESERVED,
.mac_cap_info[1] =
IEEE80211_HE_MAC_CAP1_TF_MAC_PAD_DUR_16US |
IEEE80211_HE_MAC_CAP1_MULTI_TID_AGG_RX_QOS_8,
- .mac_cap_info[2] =
- IEEE80211_HE_MAC_CAP2_BSR,
.mac_cap_info[3] =
- IEEE80211_HE_MAC_CAP3_OMI_CONTROL |
- IEEE80211_HE_MAC_CAP3_MAX_AMPDU_LEN_EXP_EXT_2,
- .mac_cap_info[4] =
- IEEE80211_HE_MAC_CAP4_AMSDU_IN_AMPDU,
- .mac_cap_info[5] =
- IEEE80211_HE_MAC_CAP5_UL_2x996_TONE_RU,
+ IEEE80211_HE_MAC_CAP3_OMI_CONTROL,
.phy_cap_info[0] =
IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_40MHZ_IN_2G |
- IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_40MHZ_80MHZ_IN_5G |
- IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_160MHZ_IN_5G,
+ IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_40MHZ_80MHZ_IN_5G,
.phy_cap_info[1] =
IEEE80211_HE_PHY_CAP1_LDPC_CODING_IN_PAYLOAD,
.phy_cap_info[2] =
IEEE80211_HE_PHY_CAP3_DCM_MAX_TX_NSS_1 |
IEEE80211_HE_PHY_CAP3_DCM_MAX_CONST_RX_NO_DCM |
IEEE80211_HE_PHY_CAP3_DCM_MAX_RX_NSS_1,
- .phy_cap_info[4] =
- IEEE80211_HE_PHY_CAP4_SU_BEAMFORMEE |
- IEEE80211_HE_PHY_CAP4_BEAMFORMEE_MAX_STS_ABOVE_80MHZ_8 |
- IEEE80211_HE_PHY_CAP4_BEAMFORMEE_MAX_STS_UNDER_80MHZ_8,
- .phy_cap_info[5] =
- IEEE80211_HE_PHY_CAP5_BEAMFORMEE_NUM_SND_DIM_UNDER_80MHZ_2 |
- IEEE80211_HE_PHY_CAP5_BEAMFORMEE_NUM_SND_DIM_ABOVE_80MHZ_2,
.phy_cap_info[6] =
IEEE80211_HE_PHY_CAP6_PPE_THRESHOLD_PRESENT,
.phy_cap_info[7] =
- IEEE80211_HE_PHY_CAP7_HE_SU_MU_PPDU_4XLTF_AND_08_US_GI |
- IEEE80211_HE_PHY_CAP7_MAX_NC_1,
+ IEEE80211_HE_PHY_CAP7_HE_SU_MU_PPDU_4XLTF_AND_08_US_GI,
.phy_cap_info[8] =
IEEE80211_HE_PHY_CAP8_HE_ER_SU_PPDU_4XLTF_AND_08_US_GI |
- IEEE80211_HE_PHY_CAP8_20MHZ_IN_40MHZ_HE_PPDU_IN_2G |
- IEEE80211_HE_PHY_CAP8_20MHZ_IN_160MHZ_HE_PPDU |
- IEEE80211_HE_PHY_CAP8_80MHZ_IN_160MHZ_HE_PPDU |
- IEEE80211_HE_PHY_CAP8_DCM_MAX_RU_2x996,
+ IEEE80211_HE_PHY_CAP8_DCM_MAX_RU_242,
.phy_cap_info[9] =
- IEEE80211_HE_PHY_CAP9_RX_FULL_BW_SU_USING_MU_WITH_COMP_SIGB |
- IEEE80211_HE_PHY_CAP9_RX_FULL_BW_SU_USING_MU_WITH_NON_COMP_SIGB |
IEEE80211_HE_PHY_CAP9_NOMIMAL_PKT_PADDING_RESERVED,
},
/*
iftype_data[i].he_6ghz_capa.capa = cpu_to_le16(he_6ghz_capa);
}
+static void
+iwl_nvm_fixup_sband_iftd(struct iwl_trans *trans,
+ struct ieee80211_supported_band *sband,
+ struct ieee80211_sband_iftype_data *iftype_data,
+ u8 tx_chains, u8 rx_chains,
+ const struct iwl_fw *fw)
+{
+ bool is_ap = iftype_data->types_mask & BIT(NL80211_IFTYPE_AP);
+
+ /* Advertise an A-MPDU exponent extension based on
+ * operating band
+ */
+ if (sband->band != NL80211_BAND_2GHZ)
+ iftype_data->he_cap.he_cap_elem.mac_cap_info[3] |=
+ IEEE80211_HE_MAC_CAP3_MAX_AMPDU_LEN_EXP_EXT_1;
+ else
+ iftype_data->he_cap.he_cap_elem.mac_cap_info[3] |=
+ IEEE80211_HE_MAC_CAP3_MAX_AMPDU_LEN_EXP_EXT_3;
+
+ if (is_ap && iwlwifi_mod_params.nvm_file)
+ iftype_data->he_cap.he_cap_elem.phy_cap_info[0] |=
+ IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_160MHZ_IN_5G;
+
+ if ((tx_chains & rx_chains) == ANT_AB) {
+ iftype_data->he_cap.he_cap_elem.phy_cap_info[5] |=
+ IEEE80211_HE_PHY_CAP5_BEAMFORMEE_NUM_SND_DIM_UNDER_80MHZ_2 |
+ IEEE80211_HE_PHY_CAP5_BEAMFORMEE_NUM_SND_DIM_ABOVE_80MHZ_2;
+ if (!is_ap)
+ iftype_data->he_cap.he_cap_elem.phy_cap_info[7] |=
+ IEEE80211_HE_PHY_CAP7_MAX_NC_2;
+ } else if (!is_ap) {
+ /* If not 2x2, we need to indicate 1x1 in the
+ * Midamble RX Max NSTS - but not for AP mode
+ */
+ iftype_data->he_cap.he_cap_elem.phy_cap_info[1] &=
+ ~IEEE80211_HE_PHY_CAP1_MIDAMBLE_RX_TX_MAX_NSTS;
+ iftype_data->he_cap.he_cap_elem.phy_cap_info[2] &=
+ ~IEEE80211_HE_PHY_CAP2_MIDAMBLE_RX_TX_MAX_NSTS;
+ iftype_data->he_cap.he_cap_elem.phy_cap_info[7] |=
+ IEEE80211_HE_PHY_CAP7_MAX_NC_1;
+ }
+
+ switch (CSR_HW_RFID_TYPE(trans->hw_rf_id)) {
+ case IWL_CFG_RF_TYPE_GF:
+ case IWL_CFG_RF_TYPE_MR:
+ iftype_data->he_cap.he_cap_elem.phy_cap_info[9] |=
+ IEEE80211_HE_PHY_CAP9_TX_1024_QAM_LESS_THAN_242_TONE_RU;
+ if (!is_ap)
+ iftype_data->he_cap.he_cap_elem.phy_cap_info[9] |=
+ IEEE80211_HE_PHY_CAP9_RX_1024_QAM_LESS_THAN_242_TONE_RU;
+ break;
+ }
+
+ if (fw_has_capa(&fw->ucode_capa, IWL_UCODE_TLV_CAPA_BROADCAST_TWT))
+ iftype_data->he_cap.he_cap_elem.mac_cap_info[2] |=
+ IEEE80211_HE_MAC_CAP2_BCAST_TWT;
+}
+
static void iwl_init_he_hw_capab(struct iwl_trans *trans,
struct iwl_nvm_data *data,
struct ieee80211_supported_band *sband,
- u8 tx_chains, u8 rx_chains)
+ u8 tx_chains, u8 rx_chains,
+ const struct iwl_fw *fw)
{
struct ieee80211_sband_iftype_data *iftype_data;
+ int i;
/* should only initialize once */
if (WARN_ON(sband->iftype_data))
sband->iftype_data = iftype_data;
sband->n_iftype_data = ARRAY_SIZE(iwl_he_capa);
- /* If not 2x2, we need to indicate 1x1 in the Midamble RX Max NSTS */
- if ((tx_chains & rx_chains) != ANT_AB) {
- int i;
-
- for (i = 0; i < sband->n_iftype_data; i++) {
- iftype_data[i].he_cap.he_cap_elem.phy_cap_info[1] &=
- ~IEEE80211_HE_PHY_CAP1_MIDAMBLE_RX_TX_MAX_NSTS;
- iftype_data[i].he_cap.he_cap_elem.phy_cap_info[2] &=
- ~IEEE80211_HE_PHY_CAP2_MIDAMBLE_RX_TX_MAX_NSTS;
- iftype_data[i].he_cap.he_cap_elem.phy_cap_info[7] &=
- ~IEEE80211_HE_PHY_CAP7_MAX_NC_MASK;
- }
- }
+ for (i = 0; i < sband->n_iftype_data; i++)
+ iwl_nvm_fixup_sband_iftd(trans, sband, &iftype_data[i],
+ tx_chains, rx_chains, fw);
+
iwl_init_he_6ghz_capa(trans, data, sband, tx_chains, rx_chains);
}
static void iwl_init_sbands(struct iwl_trans *trans,
struct iwl_nvm_data *data,
const void *nvm_ch_flags, u8 tx_chains,
- u8 rx_chains, u32 sbands_flags, bool v4)
+ u8 rx_chains, u32 sbands_flags, bool v4,
+ const struct iwl_fw *fw)
{
struct device *dev = trans->dev;
const struct iwl_cfg *cfg = trans->cfg;
tx_chains, rx_chains);
if (data->sku_cap_11ax_enable && !iwlwifi_mod_params.disable_11ax)
- iwl_init_he_hw_capab(trans, data, sband, tx_chains, rx_chains);
+ iwl_init_he_hw_capab(trans, data, sband, tx_chains, rx_chains,
+ fw);
sband = &data->bands[NL80211_BAND_5GHZ];
sband->band = NL80211_BAND_5GHZ;
tx_chains, rx_chains);
if (data->sku_cap_11ax_enable && !iwlwifi_mod_params.disable_11ax)
- iwl_init_he_hw_capab(trans, data, sband, tx_chains, rx_chains);
+ iwl_init_he_hw_capab(trans, data, sband, tx_chains, rx_chains,
+ fw);
/* 6GHz band. */
sband = &data->bands[NL80211_BAND_6GHZ];
NL80211_BAND_6GHZ);
if (data->sku_cap_11ax_enable && !iwlwifi_mod_params.disable_11ax)
- iwl_init_he_hw_capab(trans, data, sband, tx_chains, rx_chains);
+ iwl_init_he_hw_capab(trans, data, sband, tx_chains, rx_chains,
+ fw);
else
sband->n_channels = 0;
if (n_channels != n_used)
sbands_flags |= IWL_NVM_SBANDS_FLAGS_NO_WIDE_IN_5GHZ;
iwl_init_sbands(trans, data, ch_section, tx_chains, rx_chains,
- sbands_flags, false);
+ sbands_flags, false, fw);
data->calib_version = 255;
return data;
channel_profile,
nvm->valid_tx_ant & fw->valid_tx_ant,
nvm->valid_rx_ant & fw->valid_rx_ant,
- sbands_flags, v4);
+ sbands_flags, v4, fw);
iwl_free_resp(&hcmd);
return nvm;
/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
/*
- * Copyright (C) 2005-2014, 2018-2020 Intel Corporation
+ * Copyright (C) 2005-2014, 2018-2021 Intel Corporation
* Copyright (C) 2013-2015 Intel Mobile Communications GmbH
* Copyright (C) 2016 Intel Deutschland GmbH
*/
#define UREG_DOORBELL_TO_ISR6_RESUME BIT(19)
#define UREG_DOORBELL_TO_ISR6_PNVM BIT(20)
+#define CNVI_MBOX_C 0xA3400C
+
#define FSEQ_ERROR_CODE 0xA340C8
#define FSEQ_TOP_INIT_VERSION 0xA34038
#define FSEQ_CNVIO_INIT_VERSION 0xA3403C
IWL_ERROR_EVENT_TABLE_LMAC1 = BIT(0),
IWL_ERROR_EVENT_TABLE_LMAC2 = BIT(1),
IWL_ERROR_EVENT_TABLE_UMAC = BIT(2),
+ IWL_ERROR_EVENT_TABLE_TCM = BIT(3),
};
/**
void (*debugfs_cleanup)(struct iwl_trans *trans);
void (*sync_nmi)(struct iwl_trans *trans);
int (*set_pnvm)(struct iwl_trans *trans, const void *data, u32 len);
+ int (*set_reduce_power)(struct iwl_trans *trans,
+ const void *data, u32 len);
void (*interrupts)(struct iwl_trans *trans, bool enable);
};
* @trigger_tlv: array of pointers to triggers TLVs for debug
* @lmac_error_event_table: addrs of lmacs error tables
* @umac_error_event_table: addr of umac error table
+ * @tcm_error_event_table: address of TCM error table
* @error_event_table_tlv_status: bitmap that indicates what error table
* pointers was recevied via TLV. uses enum &iwl_error_event_table_status
* @internal_ini_cfg: internal debug cfg state. Uses &enum iwl_ini_cfg_state
u32 lmac_error_event_table[2];
u32 umac_error_event_table;
+ u32 tcm_error_event_table;
unsigned int error_event_table_tlv_status;
enum iwl_ini_cfg_state internal_ini_cfg;
bool pm_support;
bool ltr_enabled;
u8 pnvm_loaded:1;
+ u8 reduce_power_loaded:1;
const struct iwl_hcmd_arr *command_groups;
int command_groups_size;
return 0;
}
+static inline int iwl_trans_set_reduce_power(struct iwl_trans *trans,
+ const void *data, u32 len)
+{
+ if (trans->ops->set_reduce_power) {
+ int ret = trans->ops->set_reduce_power(trans, data, len);
+
+ if (ret)
+ return ret;
+ }
+
+ trans->reduce_power_loaded = true;
+ return 0;
+}
+
static inline bool iwl_trans_dbg_ini_valid(struct iwl_trans *trans)
{
return trans->dbg.internal_ini_cfg != IWL_INI_CFG_STATE_NOT_LOADED ||
struct wowlan_key_data {
struct iwl_wowlan_rsc_tsc_params_cmd *rsc_tsc;
struct iwl_wowlan_tkip_params_cmd *tkip;
- struct iwl_wowlan_kek_kck_material_cmd_v3 *kek_kck_cmd;
+ struct iwl_wowlan_kek_kck_material_cmd_v4 *kek_kck_cmd;
bool error, use_rsc_tsc, use_tkip, configure_keys;
int wep_key_idx;
};
}
static int iwl_mvm_send_patterns(struct iwl_mvm *mvm,
+ struct ieee80211_vif *vif,
struct cfg80211_wowlan *wowlan)
{
+ struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
struct iwl_wowlan_patterns_cmd *pattern_cmd;
struct iwl_host_cmd cmd = {
.id = WOWLAN_PATTERNS,
.dataflags[0] = IWL_HCMD_DFL_NOCOPY,
};
int i, err;
+ int ver = iwl_fw_lookup_cmd_ver(mvm->fw, LONG_GROUP,
+ WOWLAN_PATTERNS,
+ IWL_FW_CMD_VER_UNKNOWN);
if (!wowlan->n_patterns)
return 0;
cmd.len[0] = sizeof(*pattern_cmd) +
wowlan->n_patterns * sizeof(struct iwl_wowlan_pattern_v2);
- pattern_cmd = kmalloc(cmd.len[0], GFP_KERNEL);
+ pattern_cmd = kzalloc(cmd.len[0], GFP_KERNEL);
if (!pattern_cmd)
return -ENOMEM;
- pattern_cmd->n_patterns = cpu_to_le32(wowlan->n_patterns);
+ pattern_cmd->n_patterns = wowlan->n_patterns;
+ if (ver >= 3)
+ pattern_cmd->sta_id = mvmvif->ap_sta_id;
for (i = 0; i < wowlan->n_patterns; i++) {
int mask_len = DIV_ROUND_UP(wowlan->patterns[i].pattern_len, 8);
struct ieee80211_vif *vif, struct iwl_mvm_vif *mvmvif,
struct ieee80211_sta *ap_sta)
{
- int ret;
struct iwl_mvm_sta *mvm_ap_sta = iwl_mvm_sta_from_mac80211(ap_sta);
/* TODO: wowlan_config_cmd->wowlan_ba_teardown_tids */
wowlan_config_cmd->flags = ENABLE_L3_FILTERING |
ENABLE_NBNS_FILTERING | ENABLE_DHCP_FILTERING;
- /* Query the last used seqno and set it */
- ret = iwl_mvm_get_last_nonqos_seq(mvm, vif);
- if (ret < 0)
- return ret;
+ if (iwl_fw_lookup_cmd_ver(mvm->fw, LONG_GROUP,
+ WOWLAN_CONFIGURATION, 0) < 6) {
+ /* Query the last used seqno and set it */
+ int ret = iwl_mvm_get_last_nonqos_seq(mvm, vif);
- wowlan_config_cmd->non_qos_seq = cpu_to_le16(ret);
+ if (ret < 0)
+ return ret;
+
+ wowlan_config_cmd->non_qos_seq = cpu_to_le16(ret);
+ }
iwl_mvm_set_wowlan_qos_seq(mvm_ap_sta, wowlan_config_cmd);
struct ieee80211_vif *vif,
u32 cmd_flags)
{
- struct iwl_wowlan_kek_kck_material_cmd_v3 kek_kck_cmd = {};
+ struct iwl_wowlan_kek_kck_material_cmd_v4 kek_kck_cmd = {};
+ struct iwl_wowlan_kek_kck_material_cmd_v4 *_kek_kck_cmd = &kek_kck_cmd;
struct iwl_wowlan_tkip_params_cmd tkip_cmd = {};
bool unified = fw_has_capa(&mvm->fw->ucode_capa,
IWL_UCODE_TLV_CAPA_CNSLDTD_D3_D0_IMG);
.use_rsc_tsc = false,
.tkip = &tkip_cmd,
.use_tkip = false,
- .kek_kck_cmd = &kek_kck_cmd,
+ .kek_kck_cmd = _kek_kck_cmd,
};
struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
int ret;
IWL_ALWAYS_LONG_GROUP,
WOWLAN_KEK_KCK_MATERIAL,
IWL_FW_CMD_VER_UNKNOWN);
- if (WARN_ON(cmd_ver != 2 && cmd_ver != 3 &&
+ if (WARN_ON(cmd_ver != 2 && cmd_ver != 3 && cmd_ver != 4 &&
cmd_ver != IWL_FW_CMD_VER_UNKNOWN))
return -EINVAL;
- if (cmd_ver == 3)
- cmd_size = sizeof(struct iwl_wowlan_kek_kck_material_cmd_v3);
- else
- cmd_size = sizeof(struct iwl_wowlan_kek_kck_material_cmd_v2);
memcpy(kek_kck_cmd.kck, mvmvif->rekey_data.kck,
mvmvif->rekey_data.kck_len);
kek_kck_cmd.kek_len = cpu_to_le16(mvmvif->rekey_data.kek_len);
kek_kck_cmd.replay_ctr = mvmvif->rekey_data.replay_ctr;
kek_kck_cmd.akm = cpu_to_le32(mvmvif->rekey_data.akm);
+ kek_kck_cmd.sta_id = cpu_to_le32(mvmvif->ap_sta_id);
+
+ if (cmd_ver == 4) {
+ cmd_size = sizeof(struct iwl_wowlan_kek_kck_material_cmd_v4);
+ } else {
+ if (cmd_ver == 3)
+ cmd_size =
+ sizeof(struct iwl_wowlan_kek_kck_material_cmd_v3);
+ else
+ cmd_size =
+ sizeof(struct iwl_wowlan_kek_kck_material_cmd_v2);
+ /* skip the sta_id at the beginning */
+ _kek_kck_cmd = (void *)
+ ((u8 *)_kek_kck_cmd) + sizeof(kek_kck_cmd.sta_id);
+ }
IWL_DEBUG_WOWLAN(mvm, "setting akm %d\n",
mvmvif->rekey_data.akm);
ret = iwl_mvm_send_cmd_pdu(mvm,
WOWLAN_KEK_KCK_MATERIAL, cmd_flags,
cmd_size,
- &kek_kck_cmd);
+ _kek_kck_cmd);
if (ret)
goto out;
}
if (fw_has_api(&mvm->fw->ucode_capa,
IWL_UCODE_TLV_API_WOWLAN_TCP_SYN_WAKE))
- ret = iwl_mvm_send_patterns(mvm, wowlan);
+ ret = iwl_mvm_send_patterns(mvm, vif, wowlan);
else
ret = iwl_mvm_send_patterns_v1(mvm, wowlan);
if (ret)
}
out:
- mvmvif->seqno_valid = true;
- /* +0x10 because the set API expects next-to-use, not last-used */
- mvmvif->seqno = le16_to_cpu(status->non_qos_seq_ctr) + 0x10;
+ if (iwl_fw_lookup_notif_ver(mvm->fw, LONG_GROUP,
+ WOWLAN_GET_STATUSES, 0) < 10) {
+ mvmvif->seqno_valid = true;
+ /* +0x10 because the set API expects next-to-use, not last-used */
+ mvmvif->seqno = le16_to_cpu(status->non_qos_seq_ctr) + 0x10;
+ }
return true;
}
iwl_mvm_parse_wowlan_status_common(v7)
iwl_mvm_parse_wowlan_status_common(v9)
-struct iwl_wowlan_status *iwl_mvm_send_wowlan_get_status(struct iwl_mvm *mvm)
+static struct iwl_wowlan_status *
+iwl_mvm_send_wowlan_get_status(struct iwl_mvm *mvm, u8 sta_id)
{
struct iwl_wowlan_status *status;
+ struct iwl_wowlan_get_status_cmd get_status_cmd = {
+ .sta_id = cpu_to_le32(sta_id),
+ };
struct iwl_host_cmd cmd = {
.id = WOWLAN_GET_STATUSES,
.flags = CMD_WANT_SKB,
+ .data = { &get_status_cmd, },
+ .len = { sizeof(get_status_cmd), },
};
int ret, len;
u8 notif_ver;
+ u8 cmd_ver = iwl_fw_lookup_cmd_ver(mvm->fw, LONG_GROUP,
+ WOWLAN_GET_STATUSES,
+ IWL_FW_CMD_VER_UNKNOWN);
+
+ if (cmd_ver == IWL_FW_CMD_VER_UNKNOWN)
+ cmd.len[0] = 0;
lockdep_assert_held(&mvm->mutex);
len = iwl_rx_packet_payload_len(cmd.resp_pkt);
/* default to 7 (when we have IWL_UCODE_TLV_API_WOWLAN_KEY_MATERIAL) */
- notif_ver = iwl_fw_lookup_notif_ver(mvm->fw, LEGACY_GROUP,
- WOWLAN_GET_STATUSES, 7);
+ notif_ver = iwl_fw_lookup_notif_ver(mvm->fw, LONG_GROUP,
+ WOWLAN_GET_STATUSES, 0);
+ if (!notif_ver)
+ notif_ver = iwl_fw_lookup_notif_ver(mvm->fw, LEGACY_GROUP,
+ WOWLAN_GET_STATUSES, 7);
if (!fw_has_api(&mvm->fw->ucode_capa,
IWL_UCODE_TLV_API_WOWLAN_KEY_MATERIAL)) {
status->gtk[0] = v7->gtk[0];
status->igtk[0] = v7->igtk[0];
- } else if (notif_ver == 9) {
+ } else if (notif_ver == 9 || notif_ver == 10) {
struct iwl_wowlan_status_v9 *v9 = (void *)cmd.resp_pkt->data;
status = iwl_mvm_parse_wowlan_status_common_v9(mvm,
}
static struct iwl_wowlan_status *
-iwl_mvm_get_wakeup_status(struct iwl_mvm *mvm)
+iwl_mvm_get_wakeup_status(struct iwl_mvm *mvm, u8 sta_id)
{
- int ret;
-
- /* only for tracing for now */
- ret = iwl_mvm_send_cmd_pdu(mvm, OFFLOADS_QUERY_CMD, 0, 0, NULL);
- if (ret)
- IWL_ERR(mvm, "failed to query offload statistics (%d)\n", ret);
+ u8 cmd_ver = iwl_fw_lookup_cmd_ver(mvm->fw, LONG_GROUP,
+ OFFLOADS_QUERY_CMD,
+ IWL_FW_CMD_VER_UNKNOWN);
+ __le32 station_id = cpu_to_le32(sta_id);
+ u32 cmd_size = cmd_ver != IWL_FW_CMD_VER_UNKNOWN ? sizeof(station_id) : 0;
+
+ if (!mvm->net_detect) {
+ /* only for tracing for now */
+ int ret = iwl_mvm_send_cmd_pdu(mvm, OFFLOADS_QUERY_CMD, 0,
+ cmd_size, &station_id);
+ if (ret)
+ IWL_ERR(mvm, "failed to query offload statistics (%d)\n", ret);
+ }
- return iwl_mvm_send_wowlan_get_status(mvm);
+ return iwl_mvm_send_wowlan_get_status(mvm, sta_id);
}
/* releases the MVM mutex */
static bool iwl_mvm_query_wakeup_reasons(struct iwl_mvm *mvm,
struct ieee80211_vif *vif)
{
+ struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
struct iwl_wowlan_status_data status;
struct iwl_wowlan_status *fw_status;
int i;
bool keep;
struct iwl_mvm_sta *mvm_ap_sta;
- fw_status = iwl_mvm_get_wakeup_status(mvm);
+ fw_status = iwl_mvm_get_wakeup_status(mvm, mvmvif->ap_sta_id);
if (IS_ERR_OR_NULL(fw_status))
goto out_unlock;
u32 reasons = 0;
int i, n_matches, ret;
- fw_status = iwl_mvm_get_wakeup_status(mvm);
+ fw_status = iwl_mvm_get_wakeup_status(mvm, IWL_MVM_INVALID_STA);
if (!IS_ERR_OR_NULL(fw_status)) {
reasons = le32_to_cpu(fw_status->wakeup_reasons);
kfree(fw_status);
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/*
- * Copyright (C) 2012-2014, 2018-2020 Intel Corporation
+ * Copyright (C) 2012-2014, 2018-2021 Intel Corporation
* Copyright (C) 2013-2015 Intel Mobile Communications GmbH
* Copyright (C) 2016-2017 Intel Deutschland GmbH
*/
int pos = 0;
mutex_lock(&mvm->mutex);
- iwl_mvm_get_sync_time(mvm, &curr_gp2, &curr_os);
+ iwl_mvm_get_sync_time(mvm, CLOCK_BOOTTIME, &curr_gp2, &curr_os, NULL);
mutex_unlock(&mvm->mutex);
do_div(curr_os, NSEC_PER_USEC);
mvm->fw_restart++;
/* take the return value to make compiler happy - it will fail anyway */
- ret = iwl_mvm_send_cmd_pdu(mvm, REPLY_ERROR, 0, 0, NULL);
+ ret = iwl_mvm_send_cmd_pdu(mvm,
+ WIDE_ID(LONG_GROUP, REPLY_ERROR),
+ 0, 0, NULL);
mutex_unlock(&mvm->mutex);
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/*
* Copyright (C) 2015-2017 Intel Deutschland GmbH
- * Copyright (C) 2018-2020 Intel Corporation
+ * Copyright (C) 2018-2021 Intel Corporation
*/
#include <linux/etherdevice.h>
#include <linux/math64.h>
FTM_PUT_FLAG(TB);
else if (peer->ftm.non_trigger_based)
FTM_PUT_FLAG(NON_TB);
+
+ if ((peer->ftm.trigger_based || peer->ftm.non_trigger_based) &&
+ peer->ftm.lmr_feedback)
+ FTM_PUT_FLAG(LMR_FEEDBACK);
}
static int
u32 curr_gp2, diff;
u64 now_from_boot_ns;
- iwl_mvm_get_sync_time(mvm, &curr_gp2, &now_from_boot_ns);
+ iwl_mvm_get_sync_time(mvm, CLOCK_BOOTTIME, &curr_gp2,
+ &now_from_boot_ns, NULL);
if (curr_gp2 >= gp2_ts)
diff = curr_gp2 - gp2_ts;
static void iwl_mvm_lari_cfg(struct iwl_mvm *mvm)
{
- int cmd_ret;
- struct iwl_lari_config_change_cmd_v3 cmd = {};
+ int ret;
+ u32 value;
+ struct iwl_lari_config_change_cmd_v4 cmd = {};
cmd.config_bitmap = iwl_acpi_get_lari_config_bitmap(&mvm->fwrt);
+ ret = iwl_acpi_get_dsm_u32((&mvm->fwrt)->dev, 0, DSM_FUNC_11AX_ENABLEMENT,
+ &iwl_guid, &value);
+ if (!ret)
+ cmd.oem_11ax_allow_bitmap = cpu_to_le32(value);
/* apply more config masks here */
- if (cmd.config_bitmap) {
+ ret = iwl_acpi_get_dsm_u32((&mvm->fwrt)->dev, 0,
+ DSM_FUNC_ENABLE_UNII4_CHAN,
+ &iwl_guid, &value);
+ if (!ret)
+ cmd.oem_unii4_allow_bitmap = cpu_to_le32(value);
+
+ if (cmd.config_bitmap ||
+ cmd.oem_11ax_allow_bitmap ||
+ cmd.oem_unii4_allow_bitmap) {
size_t cmd_size;
u8 cmd_ver = iwl_fw_lookup_cmd_ver(mvm->fw,
REGULATORY_AND_NVM_GROUP,
LARI_CONFIG_CHANGE, 1);
- if (cmd_ver == 3)
+ if (cmd_ver == 4)
+ cmd_size = sizeof(struct iwl_lari_config_change_cmd_v4);
+ else if (cmd_ver == 3)
cmd_size = sizeof(struct iwl_lari_config_change_cmd_v3);
else if (cmd_ver == 2)
cmd_size = sizeof(struct iwl_lari_config_change_cmd_v2);
cmd_size = sizeof(struct iwl_lari_config_change_cmd_v1);
IWL_DEBUG_RADIO(mvm,
- "sending LARI_CONFIG_CHANGE, config_bitmap=0x%x\n",
- le32_to_cpu(cmd.config_bitmap));
- cmd_ret = iwl_mvm_send_cmd_pdu(mvm,
- WIDE_ID(REGULATORY_AND_NVM_GROUP,
- LARI_CONFIG_CHANGE),
- 0, cmd_size, &cmd);
- if (cmd_ret < 0)
+ "sending LARI_CONFIG_CHANGE, config_bitmap=0x%x, oem_11ax_allow_bitmap=0x%x\n",
+ le32_to_cpu(cmd.config_bitmap),
+ le32_to_cpu(cmd.oem_11ax_allow_bitmap));
+ IWL_DEBUG_RADIO(mvm,
+ "sending LARI_CONFIG_CHANGE, oem_unii4_allow_bitmap=0x%x, cmd_ver=%d\n",
+ le32_to_cpu(cmd.oem_unii4_allow_bitmap),
+ cmd_ver);
+ ret = iwl_mvm_send_cmd_pdu(mvm,
+ WIDE_ID(REGULATORY_AND_NVM_GROUP,
+ LARI_CONFIG_CHANGE),
+ 0, cmd_size, &cmd);
+ if (ret < 0)
IWL_DEBUG_RADIO(mvm,
"Failed to send LARI_CONFIG_CHANGE (%d)\n",
- cmd_ret);
+ ret);
}
}
#else /* CONFIG_ACPI */
static void iwl_mvm_mac_mgd_prepare_tx(struct ieee80211_hw *hw,
struct ieee80211_vif *vif,
- u16 req_duration)
+ struct ieee80211_prep_tx_info *info)
{
struct iwl_mvm *mvm = IWL_MAC80211_GET_MVM(hw);
u32 duration = IWL_MVM_TE_SESSION_PROTECTION_MAX_TIME_MS;
u32 min_duration = IWL_MVM_TE_SESSION_PROTECTION_MIN_TIME_MS;
- if (req_duration > duration)
- duration = req_duration;
+ if (info->duration > duration)
+ duration = info->duration;
mutex_lock(&mvm->mutex);
/* Try really hard to protect the session and hear a beacon
struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
struct cfg80211_chan_def chandef;
struct iwl_mvm_phy_ctxt *phy_ctxt;
+ bool band_change_removal;
int ret, i;
IWL_DEBUG_MAC80211(mvm, "enter (%d, %d, %d)\n", channel->hw_value,
cfg80211_chandef_create(&chandef, channel, NL80211_CHAN_NO_HT);
/*
- * Change the PHY context configuration as it is currently referenced
- * only by the P2P Device MAC
+ * Check if the remain-on-channel is on a different band and that
+ * requires context removal, see iwl_mvm_phy_ctxt_changed(). If
+ * so, we'll need to release and then re-configure here, since we
+ * must not remove a PHY context that's part of a binding.
*/
- if (mvmvif->phy_ctxt->ref == 1) {
+ band_change_removal =
+ fw_has_capa(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_CAPA_BINDING_CDB_SUPPORT) &&
+ mvmvif->phy_ctxt->channel->band != chandef.chan->band;
+
+ if (mvmvif->phy_ctxt->ref == 1 && !band_change_removal) {
+ /*
+ * Change the PHY context configuration as it is currently
+ * referenced only by the P2P Device MAC (and we can modify it)
+ */
ret = iwl_mvm_phy_ctxt_changed(mvm, mvmvif->phy_ctxt,
&chandef, 1, 1);
if (ret)
goto out_unlock;
} else {
/*
- * The PHY context is shared with other MACs. Need to remove the
- * P2P Device from the binding, allocate an new PHY context and
- * create a new binding
+ * The PHY context is shared with other MACs (or we're trying to
+ * switch bands), so remove the P2P Device from the binding,
+ * allocate an new PHY context and create a new binding.
*/
phy_ctxt = iwl_mvm_get_free_phy_ctxt(mvm);
if (!phy_ctxt) {
struct ieee80211_vif *disabled_vif = NULL;
lockdep_assert_held(&mvm->mutex);
-
iwl_mvm_remove_time_event(mvm, mvmvif, &mvmvif->time_event_data);
switch (vif->type) {
#include <linux/thermal.h>
#endif
+#include <linux/ktime.h>
+
#include "iwl-op-mode.h"
#include "iwl-trans.h"
#include "fw/notif-wait.h"
IWL_MVM_SMPS_REQ_BT_COEX,
IWL_MVM_SMPS_REQ_TT,
IWL_MVM_SMPS_REQ_PROT,
+ IWL_MVM_SMPS_REQ_FW,
NUM_IWL_MVM_SMPS_REQ,
};
*/
bool temperature_test; /* Debug test temperature is enabled */
+ bool fw_static_smps_request;
+
unsigned long bt_coex_last_tcm_ts;
struct iwl_mvm_tcm tcm;
struct ieee80211_tx_rate *r);
u8 iwl_mvm_mac80211_idx_to_hwrate(int rate_idx);
u8 iwl_mvm_mac80211_ac_to_ucode_ac(enum ieee80211_ac_numbers ac);
-void iwl_mvm_dump_nic_error_log(struct iwl_mvm *mvm);
+
+static inline void iwl_mvm_dump_nic_error_log(struct iwl_mvm *mvm)
+{
+ iwl_fwrt_dump_error_logs(&mvm->fwrt);
+}
+
u8 first_antenna(u8 mask);
u8 iwl_mvm_next_antenna(struct iwl_mvm *mvm, u8 valid, u8 last_idx);
-void iwl_mvm_get_sync_time(struct iwl_mvm *mvm, u32 *gp2, u64 *boottime);
+void iwl_mvm_get_sync_time(struct iwl_mvm *mvm, int clock_type, u32 *gp2,
+ u64 *boottime, ktime_t *realtime);
u32 iwl_mvm_get_systime(struct iwl_mvm *mvm);
/* Tx / Host Commands */
void iwl_mvm_set_default_unicast_key(struct ieee80211_hw *hw,
struct ieee80211_vif *vif, int idx);
extern const struct file_operations iwl_dbgfs_d3_test_ops;
-struct iwl_wowlan_status *iwl_mvm_send_wowlan_get_status(struct iwl_mvm *mvm);
#ifdef CONFIG_PM
void iwl_mvm_set_last_nonqos_seq(struct iwl_mvm *mvm,
struct ieee80211_vif *vif);
void iwl_mvm_update_smps(struct iwl_mvm *mvm, struct ieee80211_vif *vif,
enum iwl_mvm_smps_type_request req_type,
enum ieee80211_smps_mode smps_request);
-bool iwl_mvm_rx_diversity_allowed(struct iwl_mvm *mvm);
+bool iwl_mvm_rx_diversity_allowed(struct iwl_mvm *mvm,
+ struct iwl_mvm_phy_ctxt *ctxt);
+void iwl_mvm_apply_fw_smps_request(struct ieee80211_vif *vif);
/* Low latency */
int iwl_mvm_update_low_latency(struct iwl_mvm *mvm, struct ieee80211_vif *vif,
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/*
- * Copyright (C) 2012-2014 Intel Corporation
+ * Copyright (C) 2012-2014, 2021 Intel Corporation
* Copyright (C) 2013-2014 Intel Mobile Communications GmbH
* Copyright (C) 2015 Intel Deutschland GmbH
*/
struct iwl_proto_offload_cmd_v1 v1;
struct iwl_proto_offload_cmd_v2 v2;
struct iwl_proto_offload_cmd_v3_small v3s;
- struct iwl_proto_offload_cmd_v3_large v3l;
+ struct iwl_proto_offload_cmd_v4 v4;
} cmd = {};
struct iwl_host_cmd hcmd = {
.id = PROT_OFFLOAD_CONFIG_CMD,
struct iwl_proto_offload_cmd_common *common;
u32 enabled = 0, size;
u32 capa_flags = mvm->fw->ucode_capa.flags;
+ int ver = iwl_fw_lookup_cmd_ver(mvm->fw, LONG_GROUP,
+ PROT_OFFLOAD_CONFIG_CMD, 0);
+
#if IS_ENABLED(CONFIG_IPV6)
struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
int i;
addrs = cmd.v3s.targ_addrs;
n_addrs = IWL_PROTO_OFFLOAD_NUM_IPV6_ADDRS_V3S;
} else {
- nsc = cmd.v3l.ns_config;
+ nsc = cmd.v4.ns_config;
n_nsc = IWL_PROTO_OFFLOAD_NUM_NS_CONFIG_V3L;
- addrs = cmd.v3l.targ_addrs;
+ addrs = cmd.v4.targ_addrs;
n_addrs = IWL_PROTO_OFFLOAD_NUM_IPV6_ADDRS_V3L;
}
cmd.v3s.num_valid_ipv6_addrs =
cpu_to_le32(i - num_skipped);
else
- cmd.v3l.num_valid_ipv6_addrs =
+ cmd.v4.num_valid_ipv6_addrs =
cpu_to_le32(i - num_skipped);
} else if (capa_flags & IWL_UCODE_TLV_FLAGS_D3_6_IPV6_ADDRS) {
bool found = false;
common = &cmd.v3s.common;
size = sizeof(cmd.v3s);
} else if (capa_flags & IWL_UCODE_TLV_FLAGS_NEW_NSOFFL_LARGE) {
- common = &cmd.v3l.common;
- size = sizeof(cmd.v3l);
+ common = &cmd.v4.common;
+ size = sizeof(cmd.v4);
+ if (ver < 4) {
+ /*
+ * This basically uses iwl_proto_offload_cmd_v3_large
+ * which doesn't have the sta_id parameter before the
+ * common part.
+ */
+ size -= sizeof(cmd.v4.sta_id);
+ hcmd.data[0] = common;
+ }
} else if (capa_flags & IWL_UCODE_TLV_FLAGS_D3_6_IPV6_ADDRS) {
common = &cmd.v2.common;
size = sizeof(cmd.v2);
ieee80211_disconnect(vif, true);
}
+void iwl_mvm_apply_fw_smps_request(struct ieee80211_vif *vif)
+{
+ struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
+ struct iwl_mvm *mvm = mvmvif->mvm;
+
+ iwl_mvm_update_smps(mvm, vif, IWL_MVM_SMPS_REQ_FW,
+ mvm->fw_static_smps_request ?
+ IEEE80211_SMPS_STATIC :
+ IEEE80211_SMPS_AUTOMATIC);
+}
+
+static void iwl_mvm_intf_dual_chain_req(void *data, u8 *mac,
+ struct ieee80211_vif *vif)
+{
+ iwl_mvm_apply_fw_smps_request(vif);
+}
+
+static void iwl_mvm_rx_thermal_dual_chain_req(struct iwl_mvm *mvm,
+ struct iwl_rx_cmd_buffer *rxb)
+{
+ struct iwl_rx_packet *pkt = rxb_addr(rxb);
+ struct iwl_thermal_dual_chain_request *req = (void *)pkt->data;
+
+ /*
+ * We could pass it to the iterator data, but also need to remember
+ * it for new interfaces that are added while in this state.
+ */
+ mvm->fw_static_smps_request =
+ req->event == cpu_to_le32(THERMAL_DUAL_CHAIN_REQ_DISABLE);
+ ieee80211_iterate_interfaces(mvm->hw, IEEE80211_IFACE_ITER_NORMAL,
+ iwl_mvm_intf_dual_chain_req, NULL);
+}
+
/**
* enum iwl_rx_handler_context context for Rx handler
* @RX_HANDLER_SYNC : this means that it will be called in the Rx path
RX_HANDLER_GRP(DATA_PATH_GROUP, MONITOR_NOTIF,
iwl_mvm_rx_monitor_notif, RX_HANDLER_ASYNC_LOCKED,
struct iwl_datapath_monitor_notif),
+
+ RX_HANDLER_GRP(DATA_PATH_GROUP, THERMAL_DUAL_CHAIN_REQUEST,
+ iwl_mvm_rx_thermal_dual_chain_req,
+ RX_HANDLER_ASYNC_LOCKED,
+ struct iwl_thermal_dual_chain_request),
};
#undef RX_HANDLER
#undef RX_HANDLER_GRP
HCMD_NAME(D3_CONFIG_CMD),
HCMD_NAME(PROT_OFFLOAD_CONFIG_CMD),
HCMD_NAME(OFFLOADS_QUERY_CMD),
- HCMD_NAME(REMOTE_WAKE_CONFIG_CMD),
HCMD_NAME(MATCH_FOUND_NOTIFICATION),
HCMD_NAME(DTS_MEASUREMENT_NOTIFICATION),
HCMD_NAME(WOWLAN_PATTERNS),
HCMD_NAME(TLC_MNG_CONFIG_CMD),
HCMD_NAME(CHEST_COLLECTOR_FILTER_CONFIG_CMD),
HCMD_NAME(MONITOR_NOTIF),
+ HCMD_NAME(THERMAL_DUAL_CHAIN_REQUEST),
HCMD_NAME(STA_PM_NOTIF),
HCMD_NAME(MU_GROUP_MGMT_NOTIF),
HCMD_NAME(RX_QUEUES_NOTIFICATION),
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/*
- * Copyright (C) 2012-2014, 2018-2020 Intel Corporation
+ * Copyright (C) 2012-2014, 2018-2021 Intel Corporation
* Copyright (C) 2013-2014 Intel Mobile Communications GmbH
* Copyright (C) 2017 Intel Deutschland GmbH
*/
}
static void iwl_mvm_phy_ctxt_set_rxchain(struct iwl_mvm *mvm,
+ struct iwl_mvm_phy_ctxt *ctxt,
__le32 *rxchain_info,
u8 chains_static,
u8 chains_dynamic)
* between the two antennas is sufficiently different to impact
* performance.
*/
- if (active_cnt == 1 && iwl_mvm_rx_diversity_allowed(mvm)) {
+ if (active_cnt == 1 && iwl_mvm_rx_diversity_allowed(mvm, ctxt)) {
idle_cnt = 2;
active_cnt = 2;
}
+ /*
+ * If the firmware requested it, then we know that it supports
+ * getting zero for the values to indicate "use one, but pick
+ * which one yourself", which means it can dynamically pick one
+ * that e.g. has better RSSI.
+ */
+ if (mvm->fw_static_smps_request && active_cnt == 1 && idle_cnt == 1) {
+ idle_cnt = 0;
+ active_cnt = 0;
+ }
+
*rxchain_info = cpu_to_le32(iwl_mvm_get_valid_rx_ant(mvm) <<
PHY_RX_CHAIN_VALID_POS);
*rxchain_info |= cpu_to_le32(idle_cnt << PHY_RX_CHAIN_CNT_POS);
* Add the phy configuration to the PHY context command
*/
static void iwl_mvm_phy_ctxt_cmd_data_v1(struct iwl_mvm *mvm,
+ struct iwl_mvm_phy_ctxt *ctxt,
struct iwl_phy_context_cmd_v1 *cmd,
struct cfg80211_chan_def *chandef,
u8 chains_static, u8 chains_dynamic)
/* Set the channel info data */
iwl_mvm_set_chan_info_chandef(mvm, &cmd->ci, chandef);
- iwl_mvm_phy_ctxt_set_rxchain(mvm, &tail->rxchain_info,
+ iwl_mvm_phy_ctxt_set_rxchain(mvm, ctxt, &tail->rxchain_info,
chains_static, chains_dynamic);
tail->txchain_info = cpu_to_le32(iwl_mvm_get_valid_tx_ant(mvm));
* Add the phy configuration to the PHY context command
*/
static void iwl_mvm_phy_ctxt_cmd_data(struct iwl_mvm *mvm,
+ struct iwl_mvm_phy_ctxt *ctxt,
struct iwl_phy_context_cmd *cmd,
struct cfg80211_chan_def *chandef,
u8 chains_static, u8 chains_dynamic)
/* Set the channel info data */
iwl_mvm_set_chan_info_chandef(mvm, &cmd->ci, chandef);
- iwl_mvm_phy_ctxt_set_rxchain(mvm, &cmd->rxchain_info,
+ iwl_mvm_phy_ctxt_set_rxchain(mvm, ctxt, &cmd->rxchain_info,
chains_static, chains_dynamic);
}
iwl_mvm_phy_ctxt_cmd_hdr(ctxt, &cmd, action);
/* Set the command data */
- iwl_mvm_phy_ctxt_cmd_data(mvm, &cmd, chandef,
+ iwl_mvm_phy_ctxt_cmd_data(mvm, ctxt, &cmd, chandef,
chains_static,
chains_dynamic);
action);
/* Set the command data */
- iwl_mvm_phy_ctxt_cmd_data_v1(mvm, &cmd, chandef,
+ iwl_mvm_phy_ctxt_cmd_data_v1(mvm, ctxt, &cmd, chandef,
chains_static,
chains_dynamic);
ret = iwl_mvm_send_cmd_pdu(mvm, PHY_CONTEXT_CMD,
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/*
- * Copyright (C) 2012-2014, 2018-2020 Intel Corporation
+ * Copyright (C) 2012-2014, 2018-2021 Intel Corporation
* Copyright (C) 2013-2015 Intel Mobile Communications GmbH
* Copyright (C) 2015-2017 Intel Deutschland GmbH
*/
struct sk_buff *skb;
u8 channel, energy_a, energy_b;
struct iwl_mvm_rx_phy_data phy_data = {
+ .info_type = le32_get_bits(desc->phy_info[1],
+ IWL_RX_PHY_DATA1_INFO_TYPE_MASK),
.d0 = desc->phy_info[0],
- .info_type = IWL_RX_PHY_INFO_TYPE_NONE,
+ .d1 = desc->phy_info[1],
};
if (unlikely(iwl_rx_packet_payload_len(pkt) < sizeof(*desc)))
energy_b = (rssi & RX_NO_DATA_CHAIN_B_MSK) >> RX_NO_DATA_CHAIN_B_POS;
channel = (rssi & RX_NO_DATA_CHANNEL_MSK) >> RX_NO_DATA_CHANNEL_POS;
- phy_data.info_type =
- le32_get_bits(desc->phy_info[1],
- IWL_RX_PHY_DATA1_INFO_TYPE_MASK);
-
/* Dont use dev_alloc_skb(), we'll have enough headroom once
* ieee80211_hdr pulled.
*/
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/*
- * Copyright (C) 2012-2014, 2018-2020 Intel Corporation
+ * Copyright (C) 2012-2014, 2018-2021 Intel Corporation
* Copyright (C) 2013-2015 Intel Mobile Communications GmbH
* Copyright (C) 2016-2017 Intel Deutschland GmbH
*/
&scan_p->general_params,
gen_flags);
- ret = iwl_mvm_fill_scan_sched_params(params,
- scan_p->periodic_params.schedule,
- &scan_p->periodic_params.delay);
+ ret = iwl_mvm_fill_scan_sched_params(params,
+ scan_p->periodic_params.schedule,
+ &scan_p->periodic_params.delay);
if (ret)
return ret;
&scan_p->general_params,
gen_flags);
- ret = iwl_mvm_fill_scan_sched_params(params,
- scan_p->periodic_params.schedule,
- &scan_p->periodic_params.delay);
+ ret = iwl_mvm_fill_scan_sched_params(params,
+ scan_p->periodic_params.schedule,
+ &scan_p->periodic_params.delay);
if (ret)
return ret;
mvm_sta->disable_tx = disable;
- /* Tell mac80211 to start/stop queuing tx for this station */
- ieee80211_sta_block_awake(mvm->hw, sta, disable);
+ /*
+ * If sta PS state is handled by mac80211, tell it to start/stop
+ * queuing tx for this station.
+ */
+ if (!ieee80211_hw_check(mvm->hw, AP_LINK_PS))
+ ieee80211_sta_block_awake(mvm->hw, sta, disable);
iwl_mvm_sta_modify_disable_tx(mvm, mvm_sta, disable);
return;
list_del(&te_data->list);
+
+ /*
+ * the list is only used for AUX ROC events so make sure it is always
+ * initialized
+ */
+ INIT_LIST_HEAD(&te_data->list);
+
te_data->running = false;
te_data->uid = 0;
te_data->id = TE_MAX;
* and know the dtim period.
*/
iwl_mvm_te_check_disconnect(mvm, te_data->vif,
+ !te_data->vif->bss_conf.assoc ?
+ "Not associated and the time event is over already..." :
"No beacon heard and the time event is over already...");
break;
default:
}
static void iwl_mvm_cancel_session_protection(struct iwl_mvm *mvm,
- struct iwl_mvm_vif *mvmvif)
+ struct iwl_mvm_vif *mvmvif,
+ u32 id)
{
struct iwl_mvm_session_prot_cmd cmd = {
.id_and_color =
cpu_to_le32(FW_CMD_ID_AND_COLOR(mvmvif->id,
mvmvif->color)),
.action = cpu_to_le32(FW_CTXT_ACTION_REMOVE),
- .conf_id = cpu_to_le32(mvmvif->time_event_data.id),
+ .conf_id = cpu_to_le32(id),
};
int ret;
{
u32 id;
struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(te_data->vif);
+ enum nl80211_iftype iftype;
+
+ if (!te_data->vif)
+ return false;
+
+ iftype = te_data->vif->type;
/*
* It is possible that by the time we got to this point the time
IWL_UCODE_TLV_CAPA_SESSION_PROT_CMD)) {
if (mvmvif && id < SESSION_PROTECT_CONF_MAX_ID) {
/* Session protection is still ongoing. Cancel it */
- iwl_mvm_cancel_session_protection(mvm, mvmvif);
- if (te_data->vif->type == NL80211_IFTYPE_P2P_DEVICE) {
+ iwl_mvm_cancel_session_protection(mvm, mvmvif, id);
+ if (iftype == NL80211_IFTYPE_P2P_DEVICE) {
set_bit(IWL_MVM_STATUS_NEED_FLUSH_P2P, &mvm->status);
iwl_mvm_roc_finished(mvm);
}
IWL_ERR(mvm, "Couldn't remove the time event\n");
}
-/*
- * When the firmware supports the session protection API,
- * this is not needed since it'll automatically remove the
- * session protection after association + beacon reception.
- */
void iwl_mvm_stop_session_protection(struct iwl_mvm *mvm,
struct ieee80211_vif *vif)
{
id = te_data->id;
spin_unlock_bh(&mvm->time_event_lock);
- if (id != TE_BSS_STA_AGGRESSIVE_ASSOC) {
+ if (fw_has_capa(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_CAPA_SESSION_PROT_CMD)) {
+ if (id != SESSION_PROTECT_CONF_ASSOC) {
+ IWL_DEBUG_TE(mvm,
+ "don't remove session protection id=%u\n",
+ id);
+ return;
+ }
+ } else if (id != TE_BSS_STA_AGGRESSIVE_ASSOC) {
IWL_DEBUG_TE(mvm,
"don't remove TE with id=%u (not session protection)\n",
id);
* and know the dtim period.
*/
iwl_mvm_te_check_disconnect(mvm, vif,
+ !vif->bss_conf.assoc ?
+ "Not associated and the session protection is over already..." :
"No beacon heard and the session protection is over already...");
spin_lock_bh(&mvm->time_event_lock);
iwl_mvm_te_clear_data(mvm, te_data);
mvmvif = iwl_mvm_vif_from_mac80211(vif);
if (vif->type == NL80211_IFTYPE_P2P_DEVICE) {
- iwl_mvm_cancel_session_protection(mvm, mvmvif);
+ iwl_mvm_cancel_session_protection(mvm, mvmvif,
+ mvmvif->time_event_data.id);
set_bit(IWL_MVM_STATUS_NEED_FLUSH_P2P, &mvm->status);
} else {
iwl_mvm_remove_aux_roc_te(mvm, mvmvif,
iwl_mvm_te_clear_data(mvm, te_data);
te_data->duration = le32_to_cpu(cmd.duration_tu);
+ te_data->vif = vif;
spin_unlock_bh(&mvm->time_event_lock);
IWL_DEBUG_TE(mvm, "Add new session protection, duration %d TU\n",
if (WARN_ON_ONCE(mvmsta->sta_id == IWL_MVM_INVALID_STA))
return -1;
+ if (unlikely(ieee80211_is_any_nullfunc(fc)) && sta->he_cap.has_he)
+ return -1;
+
if (unlikely(ieee80211_is_probe_resp(fc)))
iwl_mvm_probe_resp_set_noa(mvm, skb);
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/*
- * Copyright (C) 2012-2014, 2018-2020 Intel Corporation
+ * Copyright (C) 2012-2014, 2018-2021 Intel Corporation
* Copyright (C) 2013-2014 Intel Mobile Communications GmbH
* Copyright (C) 2015-2017 Intel Deutschland GmbH
*/
return last_idx;
}
-/*
- * Note: This structure is read from the device with IO accesses,
- * and the reading already does the endian conversion. As it is
- * read with u32-sized accesses, any members with a different size
- * need to be ordered correctly though!
- */
-struct iwl_error_event_table_v1 {
- u32 valid; /* (nonzero) valid, (0) log is empty */
- u32 error_id; /* type of error */
- u32 pc; /* program counter */
- u32 blink1; /* branch link */
- u32 blink2; /* branch link */
- u32 ilink1; /* interrupt link */
- u32 ilink2; /* interrupt link */
- u32 data1; /* error-specific data */
- u32 data2; /* error-specific data */
- u32 data3; /* error-specific data */
- u32 bcon_time; /* beacon timer */
- u32 tsf_low; /* network timestamp function timer */
- u32 tsf_hi; /* network timestamp function timer */
- u32 gp1; /* GP1 timer register */
- u32 gp2; /* GP2 timer register */
- u32 gp3; /* GP3 timer register */
- u32 ucode_ver; /* uCode version */
- u32 hw_ver; /* HW Silicon version */
- u32 brd_ver; /* HW board version */
- u32 log_pc; /* log program counter */
- u32 frame_ptr; /* frame pointer */
- u32 stack_ptr; /* stack pointer */
- u32 hcmd; /* last host command header */
- u32 isr0; /* isr status register LMPM_NIC_ISR0:
- * rxtx_flag */
- u32 isr1; /* isr status register LMPM_NIC_ISR1:
- * host_flag */
- u32 isr2; /* isr status register LMPM_NIC_ISR2:
- * enc_flag */
- u32 isr3; /* isr status register LMPM_NIC_ISR3:
- * time_flag */
- u32 isr4; /* isr status register LMPM_NIC_ISR4:
- * wico interrupt */
- u32 isr_pref; /* isr status register LMPM_NIC_PREF_STAT */
- u32 wait_event; /* wait event() caller address */
- u32 l2p_control; /* L2pControlField */
- u32 l2p_duration; /* L2pDurationField */
- u32 l2p_mhvalid; /* L2pMhValidBits */
- u32 l2p_addr_match; /* L2pAddrMatchStat */
- u32 lmpm_pmg_sel; /* indicate which clocks are turned on
- * (LMPM_PMG_SEL) */
- u32 u_timestamp; /* indicate when the date and time of the
- * compilation */
- u32 flow_handler; /* FH read/write pointers, RX credit */
-} __packed /* LOG_ERROR_TABLE_API_S_VER_1 */;
-
-struct iwl_error_event_table {
- u32 valid; /* (nonzero) valid, (0) log is empty */
- u32 error_id; /* type of error */
- u32 trm_hw_status0; /* TRM HW status */
- u32 trm_hw_status1; /* TRM HW status */
- u32 blink2; /* branch link */
- u32 ilink1; /* interrupt link */
- u32 ilink2; /* interrupt link */
- u32 data1; /* error-specific data */
- u32 data2; /* error-specific data */
- u32 data3; /* error-specific data */
- u32 bcon_time; /* beacon timer */
- u32 tsf_low; /* network timestamp function timer */
- u32 tsf_hi; /* network timestamp function timer */
- u32 gp1; /* GP1 timer register */
- u32 gp2; /* GP2 timer register */
- u32 fw_rev_type; /* firmware revision type */
- u32 major; /* uCode version major */
- u32 minor; /* uCode version minor */
- u32 hw_ver; /* HW Silicon version */
- u32 brd_ver; /* HW board version */
- u32 log_pc; /* log program counter */
- u32 frame_ptr; /* frame pointer */
- u32 stack_ptr; /* stack pointer */
- u32 hcmd; /* last host command header */
- u32 isr0; /* isr status register LMPM_NIC_ISR0:
- * rxtx_flag */
- u32 isr1; /* isr status register LMPM_NIC_ISR1:
- * host_flag */
- u32 isr2; /* isr status register LMPM_NIC_ISR2:
- * enc_flag */
- u32 isr3; /* isr status register LMPM_NIC_ISR3:
- * time_flag */
- u32 isr4; /* isr status register LMPM_NIC_ISR4:
- * wico interrupt */
- u32 last_cmd_id; /* last HCMD id handled by the firmware */
- u32 wait_event; /* wait event() caller address */
- u32 l2p_control; /* L2pControlField */
- u32 l2p_duration; /* L2pDurationField */
- u32 l2p_mhvalid; /* L2pMhValidBits */
- u32 l2p_addr_match; /* L2pAddrMatchStat */
- u32 lmpm_pmg_sel; /* indicate which clocks are turned on
- * (LMPM_PMG_SEL) */
- u32 u_timestamp; /* indicate when the date and time of the
- * compilation */
- u32 flow_handler; /* FH read/write pointers, RX credit */
-} __packed /* LOG_ERROR_TABLE_API_S_VER_3 */;
-
-/*
- * UMAC error struct - relevant starting from family 8000 chip.
- * Note: This structure is read from the device with IO accesses,
- * and the reading already does the endian conversion. As it is
- * read with u32-sized accesses, any members with a different size
- * need to be ordered correctly though!
- */
-struct iwl_umac_error_event_table {
- u32 valid; /* (nonzero) valid, (0) log is empty */
- u32 error_id; /* type of error */
- u32 blink1; /* branch link */
- u32 blink2; /* branch link */
- u32 ilink1; /* interrupt link */
- u32 ilink2; /* interrupt link */
- u32 data1; /* error-specific data */
- u32 data2; /* error-specific data */
- u32 data3; /* error-specific data */
- u32 umac_major;
- u32 umac_minor;
- u32 frame_pointer; /* core register 27*/
- u32 stack_pointer; /* core register 28 */
- u32 cmd_header; /* latest host cmd sent to UMAC */
- u32 nic_isr_pref; /* ISR status register */
-} __packed;
-
-#define ERROR_START_OFFSET (1 * sizeof(u32))
-#define ERROR_ELEM_SIZE (7 * sizeof(u32))
-
-static void iwl_mvm_dump_umac_error_log(struct iwl_mvm *mvm)
-{
- struct iwl_trans *trans = mvm->trans;
- struct iwl_umac_error_event_table table = {};
- u32 base = mvm->trans->dbg.umac_error_event_table;
-
- if (!base &&
- !(mvm->trans->dbg.error_event_table_tlv_status &
- IWL_ERROR_EVENT_TABLE_UMAC))
- return;
-
- iwl_trans_read_mem_bytes(trans, base, &table, sizeof(table));
-
- if (table.valid)
- mvm->fwrt.dump.umac_err_id = table.error_id;
-
- if (ERROR_START_OFFSET <= table.valid * ERROR_ELEM_SIZE) {
- IWL_ERR(trans, "Start IWL Error Log Dump:\n");
- IWL_ERR(trans, "Status: 0x%08lX, count: %d\n",
- mvm->status, table.valid);
- }
-
- IWL_ERR(mvm, "0x%08X | %s\n", table.error_id,
- iwl_fw_lookup_assert_desc(table.error_id));
- IWL_ERR(mvm, "0x%08X | umac branchlink1\n", table.blink1);
- IWL_ERR(mvm, "0x%08X | umac branchlink2\n", table.blink2);
- IWL_ERR(mvm, "0x%08X | umac interruptlink1\n", table.ilink1);
- IWL_ERR(mvm, "0x%08X | umac interruptlink2\n", table.ilink2);
- IWL_ERR(mvm, "0x%08X | umac data1\n", table.data1);
- IWL_ERR(mvm, "0x%08X | umac data2\n", table.data2);
- IWL_ERR(mvm, "0x%08X | umac data3\n", table.data3);
- IWL_ERR(mvm, "0x%08X | umac major\n", table.umac_major);
- IWL_ERR(mvm, "0x%08X | umac minor\n", table.umac_minor);
- IWL_ERR(mvm, "0x%08X | frame pointer\n", table.frame_pointer);
- IWL_ERR(mvm, "0x%08X | stack pointer\n", table.stack_pointer);
- IWL_ERR(mvm, "0x%08X | last host cmd\n", table.cmd_header);
- IWL_ERR(mvm, "0x%08X | isr status reg\n", table.nic_isr_pref);
-}
-
-static void iwl_mvm_dump_lmac_error_log(struct iwl_mvm *mvm, u8 lmac_num)
-{
- struct iwl_trans *trans = mvm->trans;
- struct iwl_error_event_table table = {};
- u32 val, base = mvm->trans->dbg.lmac_error_event_table[lmac_num];
-
- if (mvm->fwrt.cur_fw_img == IWL_UCODE_INIT) {
- if (!base)
- base = mvm->fw->init_errlog_ptr;
- } else {
- if (!base)
- base = mvm->fw->inst_errlog_ptr;
- }
-
- if (base < 0x400000) {
- IWL_ERR(mvm,
- "Not valid error log pointer 0x%08X for %s uCode\n",
- base,
- (mvm->fwrt.cur_fw_img == IWL_UCODE_INIT)
- ? "Init" : "RT");
- return;
- }
-
- /* check if there is a HW error */
- val = iwl_trans_read_mem32(trans, base);
- if (((val & ~0xf) == 0xa5a5a5a0) || ((val & ~0xf) == 0x5a5a5a50)) {
- int err;
-
- IWL_ERR(trans, "HW error, resetting before reading\n");
-
- /* reset the device */
- iwl_trans_sw_reset(trans);
-
- err = iwl_finish_nic_init(trans, trans->trans_cfg);
- if (err)
- return;
- }
-
- iwl_trans_read_mem_bytes(trans, base, &table, sizeof(table));
-
- if (table.valid)
- mvm->fwrt.dump.lmac_err_id[lmac_num] = table.error_id;
-
- if (ERROR_START_OFFSET <= table.valid * ERROR_ELEM_SIZE) {
- IWL_ERR(trans, "Start IWL Error Log Dump:\n");
- IWL_ERR(trans, "Status: 0x%08lX, count: %d\n",
- mvm->status, table.valid);
- }
-
- /* Do not change this output - scripts rely on it */
-
- IWL_ERR(mvm, "Loaded firmware version: %s\n", mvm->fw->fw_version);
-
- IWL_ERR(mvm, "0x%08X | %-28s\n", table.error_id,
- iwl_fw_lookup_assert_desc(table.error_id));
- IWL_ERR(mvm, "0x%08X | trm_hw_status0\n", table.trm_hw_status0);
- IWL_ERR(mvm, "0x%08X | trm_hw_status1\n", table.trm_hw_status1);
- IWL_ERR(mvm, "0x%08X | branchlink2\n", table.blink2);
- IWL_ERR(mvm, "0x%08X | interruptlink1\n", table.ilink1);
- IWL_ERR(mvm, "0x%08X | interruptlink2\n", table.ilink2);
- IWL_ERR(mvm, "0x%08X | data1\n", table.data1);
- IWL_ERR(mvm, "0x%08X | data2\n", table.data2);
- IWL_ERR(mvm, "0x%08X | data3\n", table.data3);
- IWL_ERR(mvm, "0x%08X | beacon time\n", table.bcon_time);
- IWL_ERR(mvm, "0x%08X | tsf low\n", table.tsf_low);
- IWL_ERR(mvm, "0x%08X | tsf hi\n", table.tsf_hi);
- IWL_ERR(mvm, "0x%08X | time gp1\n", table.gp1);
- IWL_ERR(mvm, "0x%08X | time gp2\n", table.gp2);
- IWL_ERR(mvm, "0x%08X | uCode revision type\n", table.fw_rev_type);
- IWL_ERR(mvm, "0x%08X | uCode version major\n", table.major);
- IWL_ERR(mvm, "0x%08X | uCode version minor\n", table.minor);
- IWL_ERR(mvm, "0x%08X | hw version\n", table.hw_ver);
- IWL_ERR(mvm, "0x%08X | board version\n", table.brd_ver);
- IWL_ERR(mvm, "0x%08X | hcmd\n", table.hcmd);
- IWL_ERR(mvm, "0x%08X | isr0\n", table.isr0);
- IWL_ERR(mvm, "0x%08X | isr1\n", table.isr1);
- IWL_ERR(mvm, "0x%08X | isr2\n", table.isr2);
- IWL_ERR(mvm, "0x%08X | isr3\n", table.isr3);
- IWL_ERR(mvm, "0x%08X | isr4\n", table.isr4);
- IWL_ERR(mvm, "0x%08X | last cmd Id\n", table.last_cmd_id);
- IWL_ERR(mvm, "0x%08X | wait_event\n", table.wait_event);
- IWL_ERR(mvm, "0x%08X | l2p_control\n", table.l2p_control);
- IWL_ERR(mvm, "0x%08X | l2p_duration\n", table.l2p_duration);
- IWL_ERR(mvm, "0x%08X | l2p_mhvalid\n", table.l2p_mhvalid);
- IWL_ERR(mvm, "0x%08X | l2p_addr_match\n", table.l2p_addr_match);
- IWL_ERR(mvm, "0x%08X | lmpm_pmg_sel\n", table.lmpm_pmg_sel);
- IWL_ERR(mvm, "0x%08X | timestamp\n", table.u_timestamp);
- IWL_ERR(mvm, "0x%08X | flow_handler\n", table.flow_handler);
-}
-
-static void iwl_mvm_dump_iml_error_log(struct iwl_mvm *mvm)
-{
- struct iwl_trans *trans = mvm->trans;
- u32 error, data1;
-
- if (mvm->trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_22000) {
- error = UMAG_SB_CPU_2_STATUS;
- data1 = UMAG_SB_CPU_1_STATUS;
- } else if (mvm->trans->trans_cfg->device_family >=
- IWL_DEVICE_FAMILY_8000) {
- error = SB_CPU_2_STATUS;
- data1 = SB_CPU_1_STATUS;
- } else {
- return;
- }
-
- error = iwl_read_umac_prph(trans, UMAG_SB_CPU_2_STATUS);
-
- IWL_ERR(trans, "IML/ROM dump:\n");
-
- if (error & 0xFFFF0000)
- IWL_ERR(trans, "0x%04X | IML/ROM SYSASSERT\n", error >> 16);
-
- IWL_ERR(mvm, "0x%08X | IML/ROM error/state\n", error);
- IWL_ERR(mvm, "0x%08X | IML/ROM data1\n",
- iwl_read_umac_prph(trans, data1));
-
- if (mvm->trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_22000)
- IWL_ERR(mvm, "0x%08X | IML/ROM WFPM_AUTH_KEY_0\n",
- iwl_read_umac_prph(trans, SB_MODIFY_CFG_FLAG));
-}
-
-void iwl_mvm_dump_nic_error_log(struct iwl_mvm *mvm)
-{
- if (!test_bit(STATUS_DEVICE_ENABLED, &mvm->trans->status)) {
- IWL_ERR(mvm,
- "DEVICE_ENABLED bit is not set. Aborting dump.\n");
- return;
- }
-
- iwl_mvm_dump_lmac_error_log(mvm, 0);
-
- if (mvm->trans->dbg.lmac_error_event_table[1])
- iwl_mvm_dump_lmac_error_log(mvm, 1);
-
- iwl_mvm_dump_umac_error_log(mvm);
-
- iwl_mvm_dump_iml_error_log(mvm);
-
- iwl_fw_error_print_fseq_regs(&mvm->fwrt);
-}
-
int iwl_mvm_reconfig_scd(struct iwl_mvm *mvm, int queue, int fifo, int sta_id,
int tid, int frame_limit, u16 ssn)
{
enum ieee80211_smps_mode smps_request)
{
struct iwl_mvm_vif *mvmvif;
- enum ieee80211_smps_mode smps_mode;
+ enum ieee80211_smps_mode smps_mode = IEEE80211_SMPS_AUTOMATIC;
int i;
lockdep_assert_held(&mvm->mutex);
if (num_of_ant(iwl_mvm_get_valid_rx_ant(mvm)) == 1)
return;
- if (vif->type == NL80211_IFTYPE_AP)
- smps_mode = IEEE80211_SMPS_OFF;
- else
- smps_mode = IEEE80211_SMPS_AUTOMATIC;
+ if (vif->type != NL80211_IFTYPE_STATION)
+ return;
mvmvif = iwl_mvm_vif_from_mac80211(vif);
mvmvif->smps_requests[req_type] = smps_request;
mvm->accu_radio_stats.on_time_scan += mvm->radio_stats.on_time_scan;
}
+struct iwl_mvm_diversity_iter_data {
+ struct iwl_mvm_phy_ctxt *ctxt;
+ bool result;
+};
+
static void iwl_mvm_diversity_iter(void *_data, u8 *mac,
struct ieee80211_vif *vif)
{
struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
- bool *result = _data;
+ struct iwl_mvm_diversity_iter_data *data = _data;
int i;
+ if (mvmvif->phy_ctxt != data->ctxt)
+ return;
+
for (i = 0; i < NUM_IWL_MVM_SMPS_REQ; i++) {
if (mvmvif->smps_requests[i] == IEEE80211_SMPS_STATIC ||
- mvmvif->smps_requests[i] == IEEE80211_SMPS_DYNAMIC)
- *result = false;
+ mvmvif->smps_requests[i] == IEEE80211_SMPS_DYNAMIC) {
+ data->result = false;
+ break;
+ }
}
}
-bool iwl_mvm_rx_diversity_allowed(struct iwl_mvm *mvm)
+bool iwl_mvm_rx_diversity_allowed(struct iwl_mvm *mvm,
+ struct iwl_mvm_phy_ctxt *ctxt)
{
- bool result = true;
+ struct iwl_mvm_diversity_iter_data data = {
+ .ctxt = ctxt,
+ .result = true,
+ };
lockdep_assert_held(&mvm->mutex);
ieee80211_iterate_active_interfaces_atomic(
mvm->hw, IEEE80211_IFACE_ITER_NORMAL,
- iwl_mvm_diversity_iter, &result);
+ iwl_mvm_diversity_iter, &data);
- return result;
+ return data.result;
}
void iwl_mvm_send_low_latency_cmd(struct iwl_mvm *mvm,
return iwl_read_prph(mvm->trans, reg_addr);
}
-void iwl_mvm_get_sync_time(struct iwl_mvm *mvm, u32 *gp2, u64 *boottime)
+void iwl_mvm_get_sync_time(struct iwl_mvm *mvm, int clock_type,
+ u32 *gp2, u64 *boottime, ktime_t *realtime)
{
bool ps_disabled;
}
*gp2 = iwl_mvm_get_systime(mvm);
- *boottime = ktime_get_boottime_ns();
+
+ if (clock_type == CLOCK_BOOTTIME && boottime)
+ *boottime = ktime_get_boottime_ns();
+ else if (clock_type == CLOCK_REALTIME && realtime)
+ *realtime = ktime_get_real();
if (!ps_disabled) {
mvm->ps_disabled = ps_disabled;
struct iwl_prph_scratch *prph_scratch;
struct iwl_prph_scratch_ctrl_cfg *prph_sc_ctrl;
struct iwl_prph_info *prph_info;
- void *iml_img;
u32 control_flags = 0;
int ret;
int cmdq_size = max_t(u32, IWL_CMD_QUEUE_SIZE,
/* Allocate prph information
* currently we don't assign to the prph info anything, but it would get
- * assigned later */
- prph_info = dma_alloc_coherent(trans->dev, sizeof(*prph_info),
+ * assigned later
+ *
+ * We also use the second half of this page to give the device some
+ * dummy TR/CR tail pointers - which shouldn't be necessary as we don't
+ * use this, but the hardware still reads/writes there and we can't let
+ * it go do that with a NULL pointer.
+ */
+ BUILD_BUG_ON(sizeof(*prph_info) > PAGE_SIZE / 2);
+ prph_info = dma_alloc_coherent(trans->dev, PAGE_SIZE,
&trans_pcie->prph_info_dma_addr,
GFP_KERNEL);
if (!prph_info) {
ctxt_info_gen3->cr_head_idx_arr_base_addr =
cpu_to_le64(trans_pcie->rxq->rb_stts_dma);
ctxt_info_gen3->tr_tail_idx_arr_base_addr =
- cpu_to_le64(trans_pcie->rxq->tr_tail_dma);
+ cpu_to_le64(trans_pcie->prph_info_dma_addr + PAGE_SIZE / 2);
ctxt_info_gen3->cr_tail_idx_arr_base_addr =
- cpu_to_le64(trans_pcie->rxq->cr_tail_dma);
- ctxt_info_gen3->cr_idx_arr_size =
- cpu_to_le16(IWL_NUM_OF_COMPLETION_RINGS);
- ctxt_info_gen3->tr_idx_arr_size =
- cpu_to_le16(IWL_NUM_OF_TRANSFER_RINGS);
+ cpu_to_le64(trans_pcie->prph_info_dma_addr + 3 * PAGE_SIZE / 4);
ctxt_info_gen3->mtr_base_addr =
cpu_to_le64(trans->txqs.txq[trans->txqs.cmd.q_id]->dma_addr);
ctxt_info_gen3->mcr_base_addr =
trans_pcie->prph_scratch = prph_scratch;
/* Allocate IML */
- iml_img = dma_alloc_coherent(trans->dev, trans->iml_len,
- &trans_pcie->iml_dma_addr, GFP_KERNEL);
- if (!iml_img) {
+ trans_pcie->iml = dma_alloc_coherent(trans->dev, trans->iml_len,
+ &trans_pcie->iml_dma_addr,
+ GFP_KERNEL);
+ if (!trans_pcie->iml) {
ret = -ENOMEM;
goto err_free_ctxt_info;
}
- memcpy(iml_img, trans->iml, trans->iml_len);
+ memcpy(trans_pcie->iml, trans->iml, trans->iml_len);
iwl_enable_fw_load_int_ctx_info(trans);
trans_pcie->ctxt_info_dma_addr);
trans_pcie->ctxt_info_gen3 = NULL;
err_free_prph_info:
- dma_free_coherent(trans->dev,
- sizeof(*prph_info),
- prph_info,
- trans_pcie->prph_info_dma_addr);
+ dma_free_coherent(trans->dev, PAGE_SIZE, prph_info,
+ trans_pcie->prph_info_dma_addr);
err_free_prph_scratch:
dma_free_coherent(trans->dev,
}
-void iwl_pcie_ctxt_info_gen3_free(struct iwl_trans *trans)
+void iwl_pcie_ctxt_info_gen3_free(struct iwl_trans *trans, bool alive)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ if (trans_pcie->iml) {
+ dma_free_coherent(trans->dev, trans->iml_len, trans_pcie->iml,
+ trans_pcie->iml_dma_addr);
+ trans_pcie->iml_dma_addr = 0;
+ trans_pcie->iml = NULL;
+ }
+
+ iwl_pcie_ctxt_info_free_fw_img(trans);
+
+ if (alive)
+ return;
+
if (!trans_pcie->ctxt_info_gen3)
return;
+ /* ctxt_info_gen3 and prph_scratch are still needed for PNVM load */
dma_free_coherent(trans->dev, sizeof(*trans_pcie->ctxt_info_gen3),
trans_pcie->ctxt_info_gen3,
trans_pcie->ctxt_info_dma_addr);
trans_pcie->ctxt_info_dma_addr = 0;
trans_pcie->ctxt_info_gen3 = NULL;
- iwl_pcie_ctxt_info_free_fw_img(trans);
-
dma_free_coherent(trans->dev, sizeof(*trans_pcie->prph_scratch),
trans_pcie->prph_scratch,
trans_pcie->prph_scratch_dma_addr);
trans_pcie->prph_scratch_dma_addr = 0;
trans_pcie->prph_scratch = NULL;
- dma_free_coherent(trans->dev, sizeof(*trans_pcie->prph_info),
- trans_pcie->prph_info,
+ /* this is needed for the entire lifetime */
+ dma_free_coherent(trans->dev, PAGE_SIZE, trans_pcie->prph_info,
trans_pcie->prph_info_dma_addr);
trans_pcie->prph_info_dma_addr = 0;
trans_pcie->prph_info = NULL;
return 0;
}
+
+int iwl_trans_pcie_ctx_info_gen3_set_reduce_power(struct iwl_trans *trans,
+ const void *data, u32 len)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ struct iwl_prph_scratch_ctrl_cfg *prph_sc_ctrl =
+ &trans_pcie->prph_scratch->ctrl_cfg;
+ int ret;
+
+ if (trans->trans_cfg->device_family < IWL_DEVICE_FAMILY_AX210)
+ return 0;
+
+ /* only allocate the DRAM if not allocated yet */
+ if (!trans->reduce_power_loaded) {
+ if (WARN_ON(prph_sc_ctrl->reduce_power_cfg.size))
+ return -EBUSY;
+
+ ret = iwl_pcie_ctxt_info_alloc_dma(trans, data, len,
+ &trans_pcie->reduce_power_dram);
+ if (ret < 0) {
+ IWL_DEBUG_FW(trans,
+ "Failed to allocate reduce power DMA %d.\n",
+ ret);
+ return ret;
+ }
+ }
+
+ prph_sc_ctrl->reduce_power_cfg.base_addr =
+ cpu_to_le64(trans_pcie->reduce_power_dram.physical);
+ prph_sc_ctrl->reduce_power_cfg.size =
+ cpu_to_le32(trans_pcie->reduce_power_dram.size);
+
+ return 0;
+}
IWL_DEV_INFO(0x31DC, 0x1552, iwl9560_2ac_cfg_soc, iwl9560_killer_1550i_name),
IWL_DEV_INFO(0xA370, 0x1551, iwl9560_2ac_cfg_soc, iwl9560_killer_1550s_name),
IWL_DEV_INFO(0xA370, 0x1552, iwl9560_2ac_cfg_soc, iwl9560_killer_1550i_name),
+ IWL_DEV_INFO(0x51F0, 0x1552, iwl9560_2ac_cfg_soc, iwl9560_killer_1550s_160_name),
+ IWL_DEV_INFO(0x51F0, 0x1551, iwl9560_2ac_cfg_soc, iwl9560_killer_1550i_160_name),
IWL_DEV_INFO(0x271C, 0x0214, iwl9260_2ac_cfg, iwl9260_1_name),
IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB,
iwl_cfg_ma_a0_mr_a0, iwl_ax221_name),
_IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY,
+ IWL_CFG_MAC_TYPE_MA, IWL_CFG_ANY,
+ IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY,
+ IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB,
+ iwl_cfg_ma_a0_fm_a0, iwl_ax231_name),
+ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY,
IWL_CFG_MAC_TYPE_SNJ, IWL_CFG_ANY,
IWL_CFG_RF_TYPE_MR, IWL_CFG_ANY,
IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB,
if (cfg == &iwlax210_2ax_cfg_so_hr_a0) {
if (iwl_trans->hw_rev == CSR_HW_REV_TYPE_TY) {
iwl_trans->cfg = &iwlax210_2ax_cfg_ty_gf_a0;
- } else if (CSR_HW_RF_ID_TYPE_CHIP_ID(iwl_trans->hw_rf_id) ==
- CSR_HW_RF_ID_TYPE_CHIP_ID(CSR_HW_RF_ID_TYPE_JF)) {
+ } else if (CSR_HW_RFID_TYPE(iwl_trans->hw_rf_id) ==
+ CSR_HW_RFID_TYPE(CSR_HW_RF_ID_TYPE_JF)) {
iwl_trans->cfg = &iwlax210_2ax_cfg_so_jf_b0;
- } else if (CSR_HW_RF_ID_TYPE_CHIP_ID(iwl_trans->hw_rf_id) ==
- CSR_HW_RF_ID_TYPE_CHIP_ID(CSR_HW_RF_ID_TYPE_GF)) {
+ } else if (CSR_HW_RFID_TYPE(iwl_trans->hw_rf_id) ==
+ CSR_HW_RFID_TYPE(CSR_HW_RF_ID_TYPE_GF)) {
iwl_trans->cfg = &iwlax211_2ax_cfg_so_gf_a0;
- } else if (CSR_HW_RF_ID_TYPE_CHIP_ID(iwl_trans->hw_rf_id) ==
- CSR_HW_RF_ID_TYPE_CHIP_ID(CSR_HW_RF_ID_TYPE_GF4)) {
+ } else if (CSR_HW_RFID_TYPE(iwl_trans->hw_rf_id) ==
+ CSR_HW_RFID_TYPE(CSR_HW_RF_ID_TYPE_GF4)) {
iwl_trans->cfg = &iwlax411_2ax_cfg_so_gf4_a0;
}
}
/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
/*
- * Copyright (C) 2003-2015, 2018-2020 Intel Corporation
+ * Copyright (C) 2003-2015, 2018-2021 Intel Corporation
* Copyright (C) 2013-2015 Intel Mobile Communications GmbH
* Copyright (C) 2016-2017 Intel Deutschland GmbH
*/
* Address size is 32 bit in pre-9000 devices and 64 bit in 9000 devices.
* In AX210 devices it is a pointer to a list of iwl_rx_transfer_desc's
* @bd_dma: bus address of buffer of receive buffer descriptors (rbd)
- * @ubd: driver's pointer to buffer of used receive buffer descriptors (rbd)
- * @ubd_dma: physical address of buffer of used receive buffer descriptors (rbd)
- * @tr_tail: driver's pointer to the transmission ring tail buffer
- * @tr_tail_dma: physical address of the buffer for the transmission ring tail
- * @cr_tail: driver's pointer to the completion ring tail buffer
- * @cr_tail_dma: physical address of the buffer for the completion ring tail
+ * @used_bd: driver's pointer to buffer of used receive buffer descriptors (rbd)
+ * @used_bd_dma: physical address of buffer of used receive buffer descriptors (rbd)
* @read: Shared index to newest available Rx buffer
* @write: Shared index to oldest written Rx packet
* @free_count: Number of pre-allocated buffers in rx_free
struct iwl_rx_completion_desc *cd;
};
dma_addr_t used_bd_dma;
- __le16 *tr_tail;
- dma_addr_t tr_tail_dma;
- __le16 *cr_tail;
- dma_addr_t cr_tail_dma;
u32 read;
u32 write;
u32 free_count;
* Context information addresses will be taken from here.
* This is driver's local copy for keeping track of size and
* count for allocating and freeing the memory.
+ * @iml: image loader image virtual address
+ * @iml_dma_addr: image loader image DMA address
* @trans: pointer to the generic transport area
* @scd_base_addr: scheduler sram base address in SRAM
* @kw: keep warm address
* @alloc_page_lock: spinlock for the page allocator
* @alloc_page: allocated page to still use parts of
* @alloc_page_used: how much of the allocated page was already used (bytes)
+ * @rf_name: name/version of the CRF, if any
*/
struct iwl_trans_pcie {
struct iwl_rxq *rxq;
};
struct iwl_prph_info *prph_info;
struct iwl_prph_scratch *prph_scratch;
+ void *iml;
dma_addr_t ctxt_info_dma_addr;
dma_addr_t prph_info_dma_addr;
dma_addr_t prph_scratch_dma_addr;
struct iwl_dma_ptr kw;
struct iwl_dram_data pnvm_dram;
+ struct iwl_dram_data reduce_power_dram;
struct iwl_txq *txq_memory;
bool fw_reset_handshake;
bool fw_reset_done;
wait_queue_head_t fw_reset_waitq;
+
+ char rf_name[32];
};
static inline struct iwl_trans_pcie *
IWL_DEBUG_ISR(trans, "Disabled interrupts\n");
}
-#define IWL_NUM_OF_COMPLETION_RINGS 31
-#define IWL_NUM_OF_TRANSFER_RINGS 527
-
static inline int iwl_pcie_get_num_sections(const struct fw_img *fw,
int start)
{
static void iwl_pcie_free_rxq_dma(struct iwl_trans *trans,
struct iwl_rxq *rxq)
{
- struct device *dev = trans->dev;
bool use_rx_td = (trans->trans_cfg->device_family >=
IWL_DEVICE_FAMILY_AX210);
int free_size = iwl_pcie_free_bd_size(trans, use_rx_td);
rxq->used_bd, rxq->used_bd_dma);
rxq->used_bd_dma = 0;
rxq->used_bd = NULL;
-
- if (trans->trans_cfg->device_family < IWL_DEVICE_FAMILY_AX210)
- return;
-
- if (rxq->tr_tail)
- dma_free_coherent(dev, sizeof(__le16),
- rxq->tr_tail, rxq->tr_tail_dma);
- rxq->tr_tail_dma = 0;
- rxq->tr_tail = NULL;
-
- if (rxq->cr_tail)
- dma_free_coherent(dev, sizeof(__le16),
- rxq->cr_tail, rxq->cr_tail_dma);
- rxq->cr_tail_dma = 0;
- rxq->cr_tail = NULL;
}
static int iwl_pcie_alloc_rxq_dma(struct iwl_trans *trans,
rxq->rb_stts_dma =
trans_pcie->base_rb_stts_dma + rxq->id * rb_stts_size;
- if (!use_rx_td)
- return 0;
-
- /* Allocate the driver's pointer to TR tail */
- rxq->tr_tail = dma_alloc_coherent(dev, sizeof(__le16),
- &rxq->tr_tail_dma, GFP_KERNEL);
- if (!rxq->tr_tail)
- goto err;
-
- /* Allocate the driver's pointer to CR tail */
- rxq->cr_tail = dma_alloc_coherent(dev, sizeof(__le16),
- &rxq->cr_tail_dma, GFP_KERNEL);
- if (!rxq->cr_tail)
- goto err;
-
return 0;
err:
out:
/* Backtrack one entry */
rxq->read = i;
- /* update cr tail with the rxq read pointer */
- if (trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_AX210)
- *rxq->cr_tail = cpu_to_le16(r);
spin_unlock(&rxq->lock);
/*
iwl_pcie_ctxt_info_free_paging(trans);
if (trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_AX210)
- iwl_pcie_ctxt_info_gen3_free(trans);
+ iwl_pcie_ctxt_info_gen3_free(trans, false);
else
iwl_pcie_ctxt_info_free(trans);
return 0;
}
+static void iwl_pcie_get_rf_name(struct iwl_trans *trans)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ char *buf = trans_pcie->rf_name;
+ size_t buflen = sizeof(trans_pcie->rf_name);
+ size_t pos;
+ u32 version;
+
+ if (buf[0])
+ return;
+
+ switch (CSR_HW_RFID_TYPE(trans->hw_rf_id)) {
+ case CSR_HW_RFID_TYPE(CSR_HW_RF_ID_TYPE_JF):
+ pos = scnprintf(buf, buflen, "JF");
+ break;
+ case CSR_HW_RFID_TYPE(CSR_HW_RF_ID_TYPE_GF):
+ pos = scnprintf(buf, buflen, "GF");
+ break;
+ case CSR_HW_RFID_TYPE(CSR_HW_RF_ID_TYPE_GF4):
+ pos = scnprintf(buf, buflen, "GF4");
+ break;
+ case CSR_HW_RFID_TYPE(CSR_HW_RF_ID_TYPE_HR):
+ pos = scnprintf(buf, buflen, "HR");
+ break;
+ case CSR_HW_RFID_TYPE(CSR_HW_RF_ID_TYPE_HR1):
+ pos = scnprintf(buf, buflen, "HR1");
+ break;
+ case CSR_HW_RFID_TYPE(CSR_HW_RF_ID_TYPE_HRCDB):
+ pos = scnprintf(buf, buflen, "HRCDB");
+ break;
+ default:
+ return;
+ }
+
+ switch (CSR_HW_RFID_TYPE(trans->hw_rf_id)) {
+ case CSR_HW_RFID_TYPE(CSR_HW_RF_ID_TYPE_HR):
+ case CSR_HW_RFID_TYPE(CSR_HW_RF_ID_TYPE_HR1):
+ case CSR_HW_RFID_TYPE(CSR_HW_RF_ID_TYPE_HRCDB):
+ version = iwl_read_prph(trans, CNVI_MBOX_C);
+ switch (version) {
+ case 0x20000:
+ pos += scnprintf(buf + pos, buflen - pos, " B3");
+ break;
+ case 0x120000:
+ pos += scnprintf(buf + pos, buflen - pos, " B5");
+ break;
+ default:
+ pos += scnprintf(buf + pos, buflen - pos,
+ " (0x%x)", version);
+ break;
+ }
+ break;
+ default:
+ break;
+ }
+
+ pos += scnprintf(buf + pos, buflen - pos, ", rfid=0x%x",
+ trans->hw_rf_id);
+
+ IWL_INFO(trans, "Detected RF %s\n", buf);
+
+ /*
+ * also add a \n for debugfs - need to do it after printing
+ * since our IWL_INFO machinery wants to see a static \n at
+ * the end of the string
+ */
+ pos += scnprintf(buf + pos, buflen - pos, "\n");
+}
+
void iwl_trans_pcie_gen2_fw_alive(struct iwl_trans *trans, u32 scd_addr)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
/* now that we got alive we can free the fw image & the context info.
* paging memory cannot be freed included since FW will still use it
*/
- iwl_pcie_ctxt_info_free(trans);
+ if (trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_AX210)
+ iwl_pcie_ctxt_info_gen3_free(trans, true);
+ else
+ iwl_pcie_ctxt_info_free(trans);
/*
* Re-enable all the interrupts, including the RF-Kill one, now that
iwl_enable_interrupts(trans);
mutex_lock(&trans_pcie->mutex);
iwl_pcie_check_hw_rf_kill(trans);
+
+ iwl_pcie_get_rf_name(trans);
mutex_unlock(&trans_pcie->mutex);
}
if (ret)
IWL_ERR(trans_pcie->trans,
"Failed to set affinity mask for IRQ %d\n",
- i);
+ trans_pcie->msix_entries[i].vector);
}
}
trans_pcie->pnvm_dram.block,
trans_pcie->pnvm_dram.physical);
+ if (trans_pcie->reduce_power_dram.size)
+ dma_free_coherent(trans->dev,
+ trans_pcie->reduce_power_dram.size,
+ trans_pcie->reduce_power_dram.block,
+ trans_pcie->reduce_power_dram.physical);
+
mutex_destroy(&trans_pcie->mutex);
iwl_trans_free(trans);
}
return bytes_copied;
}
+static ssize_t iwl_dbgfs_rf_read(struct file *file,
+ char __user *user_buf,
+ size_t count, loff_t *ppos)
+{
+ struct iwl_trans *trans = file->private_data;
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+
+ if (!trans_pcie->rf_name[0])
+ return -ENODEV;
+
+ return simple_read_from_buffer(user_buf, count, ppos,
+ trans_pcie->rf_name,
+ strlen(trans_pcie->rf_name));
+}
+
DEBUGFS_READ_WRITE_FILE_OPS(interrupt);
DEBUGFS_READ_FILE_OPS(fh_reg);
DEBUGFS_READ_FILE_OPS(rx_queue);
DEBUGFS_WRITE_FILE_OPS(csr);
DEBUGFS_READ_WRITE_FILE_OPS(rfkill);
+DEBUGFS_READ_FILE_OPS(rf);
+
static const struct file_operations iwl_dbgfs_tx_queue_ops = {
.owner = THIS_MODULE,
.open = iwl_dbgfs_tx_queue_open,
DEBUGFS_ADD_FILE(fh_reg, dir, 0400);
DEBUGFS_ADD_FILE(rfkill, dir, 0600);
DEBUGFS_ADD_FILE(monitor_data, dir, 0400);
+ DEBUGFS_ADD_FILE(rf, dir, 0400);
}
static void iwl_trans_pcie_debugfs_cleanup(struct iwl_trans *trans)
.wait_txq_empty = iwl_trans_pcie_wait_txq_empty,
.rxq_dma_data = iwl_trans_pcie_rxq_dma_data,
.set_pnvm = iwl_trans_pcie_ctx_info_gen3_set_pnvm,
+ .set_reduce_power = iwl_trans_pcie_ctx_info_gen3_set_reduce_power,
#ifdef CONFIG_IWLWIFI_DEBUGFS
.debugfs_cleanup = iwl_trans_pcie_debugfs_cleanup,
#endif
struct iwl_trans *trans;
int ret, addr_size;
const struct iwl_trans_ops *ops = &trans_ops_pcie_gen2;
+ void __iomem * const *table;
if (!cfg_trans->gen2)
ops = &trans_ops_pcie;
goto out_no_pci;
}
- trans_pcie->hw_base = pcim_iomap_table(pdev)[0];
- if (!trans_pcie->hw_base) {
+ table = pcim_iomap_table(pdev);
+ if (!table) {
dev_err(&pdev->dev, "pcim_iomap_table failed\n");
+ ret = -ENOMEM;
+ goto out_no_pci;
+ }
+
+ trans_pcie->hw_base = table[0];
+ if (!trans_pcie->hw_base) {
+ dev_err(&pdev->dev, "couldn't find IO mem in first BAR\n");
ret = -ENODEV;
goto out_no_pci;
}
* tsc must be NULL or up to 8 bytes
*/
int __orinoco_hw_set_tkip_key(struct orinoco_private *priv, int key_idx,
- int set_tx, const u8 *key, const u8 *rsc,
- size_t rsc_len, const u8 *tsc, size_t tsc_len)
+ int set_tx, const u8 *key, size_t key_len,
+ const u8 *rsc, size_t rsc_len,
+ const u8 *tsc, size_t tsc_len)
{
struct {
__le16 idx;
u8 rsc[ORINOCO_SEQ_LEN];
- u8 key[TKIP_KEYLEN];
- u8 tx_mic[MIC_KEYLEN];
- u8 rx_mic[MIC_KEYLEN];
+ struct {
+ u8 key[TKIP_KEYLEN];
+ u8 tx_mic[MIC_KEYLEN];
+ u8 rx_mic[MIC_KEYLEN];
+ } tkip;
u8 tsc[ORINOCO_SEQ_LEN];
} __packed buf;
struct hermes *hw = &priv->hw;
key_idx |= 0x8000;
buf.idx = cpu_to_le16(key_idx);
- memcpy(buf.key, key,
- sizeof(buf.key) + sizeof(buf.tx_mic) + sizeof(buf.rx_mic));
+ if (key_len != sizeof(buf.tkip))
+ return -EINVAL;
+ memcpy(&buf.tkip, key, sizeof(buf.tkip));
if (rsc_len > sizeof(buf.rsc))
rsc_len = sizeof(buf.rsc);
int __orinoco_hw_setup_wepkeys(struct orinoco_private *priv);
int __orinoco_hw_setup_enc(struct orinoco_private *priv);
int __orinoco_hw_set_tkip_key(struct orinoco_private *priv, int key_idx,
- int set_tx, const u8 *key, const u8 *rsc,
- size_t rsc_len, const u8 *tsc, size_t tsc_len);
+ int set_tx, const u8 *key, size_t key_len,
+ const u8 *rsc, size_t rsc_len,
+ const u8 *tsc, size_t tsc_len);
int orinoco_clear_tkip_key(struct orinoco_private *priv, int key_idx);
int __orinoco_hw_set_multicast_list(struct orinoco_private *priv,
struct net_device *dev,
err = __orinoco_hw_set_tkip_key(priv, idx,
ext->ext_flags & IW_ENCODE_EXT_SET_TX_KEY,
- priv->keys[idx].key,
+ priv->keys[idx].key, priv->keys[idx].key_len,
tkip_iv, ORINOCO_SEQ_LEN, NULL, 0);
if (err)
printk(KERN_ERR "%s: Error %d setting TKIP key"
u32 ciphers[ARRAY_SIZE(hwsim_ciphers)];
struct mac_address addresses[2];
+ struct ieee80211_chanctx_conf *chanctx;
int channels, idx;
bool use_chanctx;
bool destroy_on_close;
static void mac80211_hwsim_tx_frame_nl(struct ieee80211_hw *hw,
struct sk_buff *my_skb,
- int dst_portid)
+ int dst_portid,
+ struct ieee80211_channel *channel)
{
struct sk_buff *skb;
struct mac80211_hwsim_data *data = hw->priv;
if (nla_put_u32(skb, HWSIM_ATTR_FLAGS, hwsim_flags))
goto nla_put_failure;
- if (nla_put_u32(skb, HWSIM_ATTR_FREQ, data->channel->center_freq))
+ if (nla_put_u32(skb, HWSIM_ATTR_FREQ, channel->center_freq))
goto nla_put_failure;
/* We get the tx control (rate and retries) info*/
_portid = READ_ONCE(data->wmediumd);
if (_portid || hwsim_virtio_enabled)
- return mac80211_hwsim_tx_frame_nl(hw, skb, _portid);
+ return mac80211_hwsim_tx_frame_nl(hw, skb, _portid, channel);
/* NO wmediumd detected, perfect medium simulation */
data->tx_pkts++;
static void mac80211_hwsim_stop(struct ieee80211_hw *hw)
{
struct mac80211_hwsim_data *data = hw->priv;
+
data->started = false;
hrtimer_cancel(&data->beacon_timer);
+
+ while (!skb_queue_empty(&data->pending))
+ ieee80211_free_txskb(hw, skb_dequeue(&data->pending));
+
wiphy_dbg(hw->wiphy, "%s\n", __func__);
}
mac80211_hwsim_monitor_rx(hw, skb, chan);
if (_pid || hwsim_virtio_enabled)
- return mac80211_hwsim_tx_frame_nl(hw, skb, _pid);
+ return mac80211_hwsim_tx_frame_nl(hw, skb, _pid, chan);
+ data->tx_pkts++;
+ data->tx_bytes += skb->len;
mac80211_hwsim_tx_frame_no_nl(hw, skb, chan);
dev_kfree_skb(skb);
}
static int mac80211_hwsim_add_chanctx(struct ieee80211_hw *hw,
struct ieee80211_chanctx_conf *ctx)
{
+ struct mac80211_hwsim_data *hwsim = hw->priv;
+
+ mutex_lock(&hwsim->mutex);
+ hwsim->chanctx = ctx;
+ mutex_unlock(&hwsim->mutex);
hwsim_set_chanctx_magic(ctx);
wiphy_dbg(hw->wiphy,
"add channel context control: %d MHz/width: %d/cfreqs:%d/%d MHz\n",
static void mac80211_hwsim_remove_chanctx(struct ieee80211_hw *hw,
struct ieee80211_chanctx_conf *ctx)
{
+ struct mac80211_hwsim_data *hwsim = hw->priv;
+
+ mutex_lock(&hwsim->mutex);
+ hwsim->chanctx = NULL;
+ mutex_unlock(&hwsim->mutex);
wiphy_dbg(hw->wiphy,
"remove channel context control: %d MHz/width: %d/cfreqs:%d/%d MHz\n",
ctx->def.chan->center_freq, ctx->def.width,
struct ieee80211_chanctx_conf *ctx,
u32 changed)
{
+ struct mac80211_hwsim_data *hwsim = hw->priv;
+
+ mutex_lock(&hwsim->mutex);
+ hwsim->chanctx = ctx;
+ mutex_unlock(&hwsim->mutex);
hwsim_check_chanctx_magic(ctx);
wiphy_dbg(hw->wiphy,
"change channel context control: %d MHz/width: %d/cfreqs:%d/%d MHz\n",
hw->wiphy->max_remain_on_channel_duration = 1000;
data->if_combination.radar_detect_widths = 0;
data->if_combination.num_different_channels = data->channels;
+ data->chanctx = NULL;
} else {
data->if_combination.num_different_channels = 1;
data->if_combination.radar_detect_widths =
int frame_data_len;
void *frame_data;
struct sk_buff *skb = NULL;
+ struct ieee80211_channel *channel = NULL;
if (!info->attrs[HWSIM_ATTR_ADDR_RECEIVER] ||
!info->attrs[HWSIM_ATTR_FRAME] ||
if (!data2)
goto out;
+ if (data2->use_chanctx) {
+ if (data2->tmp_chan)
+ channel = data2->tmp_chan;
+ else if (data2->chanctx)
+ channel = data2->chanctx->def.chan;
+ } else {
+ channel = data2->channel;
+ }
+ if (!channel)
+ goto out;
+
if (!hwsim_virtio_enabled) {
if (hwsim_net_get_netgroup(genl_info_net(info)) !=
data2->netgroup)
/* check if radio is configured properly */
- if (data2->idle || !data2->started)
+ if ((data2->idle && !data2->tmp_chan) || !data2->started)
goto out;
/* A frame is received from user space */
mutex_lock(&data2->mutex);
rx_status.freq = nla_get_u32(info->attrs[HWSIM_ATTR_FREQ]);
- if (rx_status.freq != data2->channel->center_freq &&
- (!data2->tmp_chan ||
- rx_status.freq != data2->tmp_chan->center_freq)) {
+ if (rx_status.freq != channel->center_freq) {
mutex_unlock(&data2->mutex);
goto out;
}
mutex_unlock(&data2->mutex);
} else {
- rx_status.freq = data2->channel->center_freq;
+ rx_status.freq = channel->center_freq;
}
- rx_status.band = data2->channel->band;
+ rx_status.band = channel->band;
rx_status.rate_idx = nla_get_u32(info->attrs[HWSIM_ATTR_RX_RATE]);
rx_status.signal = nla_get_u32(info->attrs[HWSIM_ATTR_SIGNAL]);
return -EINVAL;
}
- if (param.channels > CFG80211_MAX_NUM_DIFFERENT_CHANNELS) {
- GENL_SET_ERR_MSG(info, "too many channels specified");
- return -EINVAL;
- }
-
if (info->attrs[HWSIM_ATTR_NO_VIF])
param.no_vif = true;
wdev->netdev = dev;
priv->dev = dev;
- dev->netdev_ops = &lbs_netdev_ops;
+ dev->netdev_ops = &lbs_netdev_ops;
dev->watchdog_timeo = 5 * HZ;
dev->ethtool_ops = &lbs_ethtool_ops;
dev->flags |= IFF_BROADCAST | IFF_MULTICAST;
*/
/**
- * lbs_anycast_get - Get function for sysfs attribute anycast_mask
+ * anycast_mask_show - Get function for sysfs attribute anycast_mask
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer where data will be returned
*/
-static ssize_t lbs_anycast_get(struct device *dev,
- struct device_attribute *attr, char * buf)
+static ssize_t anycast_mask_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
{
struct lbs_private *priv = to_net_dev(dev)->ml_priv;
struct cmd_ds_mesh_access mesh_access;
}
/**
- * lbs_anycast_set - Set function for sysfs attribute anycast_mask
+ * anycast_mask_store - Set function for sysfs attribute anycast_mask
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer that contains new attribute value
* @count: size of buffer
*/
-static ssize_t lbs_anycast_set(struct device *dev,
- struct device_attribute *attr, const char * buf, size_t count)
+static ssize_t anycast_mask_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct lbs_private *priv = to_net_dev(dev)->ml_priv;
struct cmd_ds_mesh_access mesh_access;
}
/**
- * lbs_prb_rsp_limit_get - Get function for sysfs attribute prb_rsp_limit
+ * prb_rsp_limit_show - Get function for sysfs attribute prb_rsp_limit
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer where data will be returned
*/
-static ssize_t lbs_prb_rsp_limit_get(struct device *dev,
- struct device_attribute *attr, char *buf)
+static ssize_t prb_rsp_limit_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
{
struct lbs_private *priv = to_net_dev(dev)->ml_priv;
struct cmd_ds_mesh_access mesh_access;
}
/**
- * lbs_prb_rsp_limit_set - Set function for sysfs attribute prb_rsp_limit
+ * prb_rsp_limit_store - Set function for sysfs attribute prb_rsp_limit
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer that contains new attribute value
* @count: size of buffer
*/
-static ssize_t lbs_prb_rsp_limit_set(struct device *dev,
- struct device_attribute *attr, const char *buf, size_t count)
+static ssize_t prb_rsp_limit_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct lbs_private *priv = to_net_dev(dev)->ml_priv;
struct cmd_ds_mesh_access mesh_access;
}
/**
- * lbs_mesh_get - Get function for sysfs attribute mesh
+ * lbs_mesh_show - Get function for sysfs attribute mesh
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer where data will be returned
*/
-static ssize_t lbs_mesh_get(struct device *dev,
- struct device_attribute *attr, char * buf)
+static ssize_t lbs_mesh_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
{
struct lbs_private *priv = to_net_dev(dev)->ml_priv;
return snprintf(buf, 5, "0x%X\n", !!priv->mesh_dev);
}
/**
- * lbs_mesh_set - Set function for sysfs attribute mesh
+ * lbs_mesh_store - Set function for sysfs attribute mesh
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer that contains new attribute value
* @count: size of buffer
*/
-static ssize_t lbs_mesh_set(struct device *dev,
- struct device_attribute *attr, const char * buf, size_t count)
+static ssize_t lbs_mesh_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct lbs_private *priv = to_net_dev(dev)->ml_priv;
int enable;
* lbs_mesh attribute to be exported per ethX interface
* through sysfs (/sys/class/net/ethX/lbs_mesh)
*/
-static DEVICE_ATTR(lbs_mesh, 0644, lbs_mesh_get, lbs_mesh_set);
+static DEVICE_ATTR_RW(lbs_mesh);
/*
* anycast_mask attribute to be exported per mshX interface
* through sysfs (/sys/class/net/mshX/anycast_mask)
*/
-static DEVICE_ATTR(anycast_mask, 0644, lbs_anycast_get, lbs_anycast_set);
+static DEVICE_ATTR_RW(anycast_mask);
/*
* prb_rsp_limit attribute to be exported per mshX interface
* through sysfs (/sys/class/net/mshX/prb_rsp_limit)
*/
-static DEVICE_ATTR(prb_rsp_limit, 0644, lbs_prb_rsp_limit_get,
- lbs_prb_rsp_limit_set);
+static DEVICE_ATTR_RW(prb_rsp_limit);
static struct attribute *lbs_mesh_sysfs_entries[] = {
&dev_attr_anycast_mask.attr,
}
/**
- * bootflag_get - Get function for sysfs attribute bootflag
+ * bootflag_show - Get function for sysfs attribute bootflag
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer where data will be returned
*/
-static ssize_t bootflag_get(struct device *dev,
- struct device_attribute *attr, char *buf)
+static ssize_t bootflag_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
{
struct mrvl_mesh_defaults defs;
int ret;
}
/**
- * bootflag_set - Set function for sysfs attribute bootflag
+ * bootflag_store - Set function for sysfs attribute bootflag
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer that contains new attribute value
* @count: size of buffer
*/
-static ssize_t bootflag_set(struct device *dev, struct device_attribute *attr,
- const char *buf, size_t count)
+static ssize_t bootflag_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct lbs_private *priv = to_net_dev(dev)->ml_priv;
struct cmd_ds_mesh_config cmd;
}
/**
- * boottime_get - Get function for sysfs attribute boottime
+ * boottime_show - Get function for sysfs attribute boottime
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer where data will be returned
*/
-static ssize_t boottime_get(struct device *dev,
- struct device_attribute *attr, char *buf)
+static ssize_t boottime_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
{
struct mrvl_mesh_defaults defs;
int ret;
}
/**
- * boottime_set - Set function for sysfs attribute boottime
+ * boottime_store - Set function for sysfs attribute boottime
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer that contains new attribute value
* @count: size of buffer
*/
-static ssize_t boottime_set(struct device *dev,
- struct device_attribute *attr, const char *buf, size_t count)
+static ssize_t boottime_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct lbs_private *priv = to_net_dev(dev)->ml_priv;
struct cmd_ds_mesh_config cmd;
}
/**
- * channel_get - Get function for sysfs attribute channel
+ * channel_show - Get function for sysfs attribute channel
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer where data will be returned
*/
-static ssize_t channel_get(struct device *dev,
- struct device_attribute *attr, char *buf)
+static ssize_t channel_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
{
struct mrvl_mesh_defaults defs;
int ret;
}
/**
- * channel_set - Set function for sysfs attribute channel
+ * channel_store - Set function for sysfs attribute channel
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer that contains new attribute value
* @count: size of buffer
*/
-static ssize_t channel_set(struct device *dev, struct device_attribute *attr,
- const char *buf, size_t count)
+static ssize_t channel_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct lbs_private *priv = to_net_dev(dev)->ml_priv;
struct cmd_ds_mesh_config cmd;
}
/**
- * mesh_id_get - Get function for sysfs attribute mesh_id
+ * mesh_id_show - Get function for sysfs attribute mesh_id
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer where data will be returned
*/
-static ssize_t mesh_id_get(struct device *dev, struct device_attribute *attr,
- char *buf)
+static ssize_t mesh_id_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
{
struct mrvl_mesh_defaults defs;
int ret;
}
/**
- * mesh_id_set - Set function for sysfs attribute mesh_id
+ * mesh_id_store - Set function for sysfs attribute mesh_id
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer that contains new attribute value
* @count: size of buffer
*/
-static ssize_t mesh_id_set(struct device *dev, struct device_attribute *attr,
- const char *buf, size_t count)
+static ssize_t mesh_id_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct cmd_ds_mesh_config cmd;
struct mrvl_mesh_defaults defs;
}
/**
- * protocol_id_get - Get function for sysfs attribute protocol_id
+ * protocol_id_show - Get function for sysfs attribute protocol_id
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer where data will be returned
*/
-static ssize_t protocol_id_get(struct device *dev,
- struct device_attribute *attr, char *buf)
+static ssize_t protocol_id_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
{
struct mrvl_mesh_defaults defs;
int ret;
}
/**
- * protocol_id_set - Set function for sysfs attribute protocol_id
+ * protocol_id_store - Set function for sysfs attribute protocol_id
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer that contains new attribute value
* @count: size of buffer
*/
-static ssize_t protocol_id_set(struct device *dev,
- struct device_attribute *attr, const char *buf, size_t count)
+static ssize_t protocol_id_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct cmd_ds_mesh_config cmd;
struct mrvl_mesh_defaults defs;
}
/**
- * metric_id_get - Get function for sysfs attribute metric_id
+ * metric_id_show - Get function for sysfs attribute metric_id
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer where data will be returned
*/
-static ssize_t metric_id_get(struct device *dev,
- struct device_attribute *attr, char *buf)
+static ssize_t metric_id_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
{
struct mrvl_mesh_defaults defs;
int ret;
}
/**
- * metric_id_set - Set function for sysfs attribute metric_id
+ * metric_id_store - Set function for sysfs attribute metric_id
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer that contains new attribute value
* @count: size of buffer
*/
-static ssize_t metric_id_set(struct device *dev, struct device_attribute *attr,
- const char *buf, size_t count)
+static ssize_t metric_id_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct cmd_ds_mesh_config cmd;
struct mrvl_mesh_defaults defs;
}
/**
- * capability_get - Get function for sysfs attribute capability
+ * capability_show - Get function for sysfs attribute capability
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer where data will be returned
*/
-static ssize_t capability_get(struct device *dev,
- struct device_attribute *attr, char *buf)
+static ssize_t capability_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
{
struct mrvl_mesh_defaults defs;
int ret;
}
/**
- * capability_set - Set function for sysfs attribute capability
+ * capability_store - Set function for sysfs attribute capability
* @dev: the &struct device
* @attr: device attributes
* @buf: buffer that contains new attribute value
* @count: size of buffer
*/
-static ssize_t capability_set(struct device *dev, struct device_attribute *attr,
- const char *buf, size_t count)
+static ssize_t capability_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct cmd_ds_mesh_config cmd;
struct mrvl_mesh_defaults defs;
}
-static DEVICE_ATTR(bootflag, 0644, bootflag_get, bootflag_set);
-static DEVICE_ATTR(boottime, 0644, boottime_get, boottime_set);
-static DEVICE_ATTR(channel, 0644, channel_get, channel_set);
-static DEVICE_ATTR(mesh_id, 0644, mesh_id_get, mesh_id_set);
-static DEVICE_ATTR(protocol_id, 0644, protocol_id_get, protocol_id_set);
-static DEVICE_ATTR(metric_id, 0644, metric_id_get, metric_id_set);
-static DEVICE_ATTR(capability, 0644, capability_get, capability_set);
+static DEVICE_ATTR_RW(bootflag);
+static DEVICE_ATTR_RW(boottime);
+static DEVICE_ATTR_RW(channel);
+static DEVICE_ATTR_RW(mesh_id);
+static DEVICE_ATTR_RW(protocol_id);
+static DEVICE_ATTR_RW(metric_id);
+static DEVICE_ATTR_RW(capability);
static struct attribute *boot_opts_attrs[] = {
&dev_attr_bootflag.attr,
.attrs = mesh_ie_attrs,
};
-static void lbs_persist_config_init(struct net_device *dev)
-{
- int ret;
- ret = sysfs_create_group(&(dev->dev.kobj), &boot_opts_group);
- if (ret)
- pr_err("failed to create boot_opts_group.\n");
-
- ret = sysfs_create_group(&(dev->dev.kobj), &mesh_ie_group);
- if (ret)
- pr_err("failed to create mesh_ie_group.\n");
-}
-
-static void lbs_persist_config_remove(struct net_device *dev)
-{
- sysfs_remove_group(&(dev->dev.kobj), &boot_opts_group);
- sysfs_remove_group(&(dev->dev.kobj), &mesh_ie_group);
-}
-
/***************************************************************************
* Initializing and starting, stopping mesh
SET_NETDEV_DEV(priv->mesh_dev, priv->dev->dev.parent);
mesh_dev->flags |= IFF_BROADCAST | IFF_MULTICAST;
+ mesh_dev->sysfs_groups[0] = &lbs_mesh_attr_group;
+ mesh_dev->sysfs_groups[1] = &boot_opts_group;
+ mesh_dev->sysfs_groups[2] = &mesh_ie_group;
+
/* Register virtual mesh interface */
ret = register_netdev(mesh_dev);
if (ret) {
goto err_free_netdev;
}
- ret = sysfs_create_group(&(mesh_dev->dev.kobj), &lbs_mesh_attr_group);
- if (ret)
- goto err_unregister;
-
- lbs_persist_config_init(mesh_dev);
-
/* Everything successful */
ret = 0;
goto done;
-err_unregister:
- unregister_netdev(mesh_dev);
-
err_free_netdev:
free_netdev(mesh_dev);
netif_stop_queue(mesh_dev);
netif_carrier_off(mesh_dev);
- sysfs_remove_group(&(mesh_dev->dev.kobj), &lbs_mesh_attr_group);
- lbs_persist_config_remove(mesh_dev);
unregister_netdev(mesh_dev);
priv->mesh_dev = NULL;
kfree(mesh_dev->ieee80211_ptr);
static int if_usb_reset_device(struct lbtf_private *priv);
/**
- * if_usb_wrike_bulk_callback - call back to handle URB status
+ * if_usb_write_bulk_callback - call back to handle URB status
*
* @urb: pointer to urb structure
*/
struct mwifiex_ie_type_key_param_set key_param_set;
} __packed;
+struct host_cmd_ds_802_11_key_material_wep {
+ __le16 action;
+ struct mwifiex_ie_type_key_param_set key_param_set[NUM_WEP_KEYS];
+} __packed;
+
struct host_cmd_ds_gen {
__le16 command;
__le16 size;
struct host_cmd_ds_wmm_get_status get_wmm_status;
struct host_cmd_ds_802_11_key_material key_material;
struct host_cmd_ds_802_11_key_material_v2 key_material_v2;
+ struct host_cmd_ds_802_11_key_material_wep key_material_wep;
struct host_cmd_ds_version_ext verext;
struct host_cmd_ds_mgmt_frame_reg reg_mask;
struct host_cmd_ds_remain_on_chan roc_cfg;
}
if (!enc_key) {
- memset(&key_material->key_param_set, 0,
- (NUM_WEP_KEYS *
- sizeof(struct mwifiex_ie_type_key_param_set)));
+ struct host_cmd_ds_802_11_key_material_wep *key_material_wep =
+ (struct host_cmd_ds_802_11_key_material_wep *)key_material;
+ memset(key_material_wep->key_param_set, 0,
+ sizeof(key_material_wep->key_param_set));
ret = mwifiex_set_keyparamset_wep(priv,
- &key_material->key_param_set,
+ &key_material_wep->key_param_set[0],
&key_param_len);
cmd->size = cpu_to_le16(key_param_len +
- sizeof(key_material->action) + S_DS_GEN);
+ sizeof(key_material_wep->action) + S_DS_GEN);
return ret;
} else
memset(&key_material->key_param_set, 0,
else
rates = sta->supp_rates[NL80211_BAND_5GHZ] << 5;
legacy_rate_mask_to_array(p->legacy_rates, rates);
- memcpy(p->ht_rates, sta->ht_cap.mcs.rx_mask, 16);
+ memcpy(p->ht_rates, &sta->ht_cap.mcs, 16);
p->interop = 1;
p->amsdu_enabled = 0;
ap_legacy_rates =
ap->supp_rates[NL80211_BAND_5GHZ] << 5;
}
- memcpy(ap_mcs_rates, ap->ht_cap.mcs.rx_mask, 16);
+ memcpy(ap_mcs_rates, &ap->ht_cap.mcs, 16);
rcu_read_unlock();
q->entry[idx].txwi = txwi;
q->entry[idx].skb = skb;
+ q->entry[idx].wcid = 0xffff;
return idx;
}
struct sk_buff *skb, struct mt76_wcid *wcid,
struct ieee80211_sta *sta)
{
+ struct ieee80211_tx_status status = {
+ .sta = sta,
+ };
struct mt76_tx_info tx_info = {
.skb = skb,
};
u8 *txwi;
t = mt76_get_txwi(dev);
- if (!t) {
- hw = mt76_tx_status_get_hw(dev, skb);
- ieee80211_free_txskb(hw, skb);
- return -ENOMEM;
- }
+ if (!t)
+ goto free_skb;
+
txwi = mt76_get_txwi_ptr(dev, t);
skb->prev = skb->next = NULL;
}
#endif
- dev_kfree_skb(tx_info.skb);
mt76_put_txwi(dev, t);
+
+free_skb:
+ status.skb = tx_info.skb;
+ hw = mt76_tx_status_get_hw(dev, tx_info.skb);
+ ieee80211_tx_status_ext(hw, &status);
+
return ret;
}
{ .throughput = 300 * 1024, .blink_time = 50 },
};
+struct ieee80211_rate mt76_rates[] = {
+ CCK_RATE(0, 10),
+ CCK_RATE(1, 20),
+ CCK_RATE(2, 55),
+ CCK_RATE(3, 110),
+ OFDM_RATE(11, 60),
+ OFDM_RATE(15, 90),
+ OFDM_RATE(10, 120),
+ OFDM_RATE(14, 180),
+ OFDM_RATE(9, 240),
+ OFDM_RATE(13, 360),
+ OFDM_RATE(8, 480),
+ OFDM_RATE(12, 540),
+};
+EXPORT_SYMBOL_GPL(mt76_rates);
+
static int mt76_led_init(struct mt76_dev *dev)
{
struct device_node *np = dev->dev->of_node;
ieee80211_hw_set(hw, MFP_CAPABLE);
ieee80211_hw_set(hw, AP_LINK_PS);
ieee80211_hw_set(hw, REPORTS_TX_ACK_STATUS);
-
- wiphy->flags |= WIPHY_FLAG_IBSS_RSN;
- wiphy->interface_modes =
- BIT(NL80211_IFTYPE_STATION) |
- BIT(NL80211_IFTYPE_AP) |
-#ifdef CONFIG_MAC80211_MESH
- BIT(NL80211_IFTYPE_MESH_POINT) |
-#endif
- BIT(NL80211_IFTYPE_P2P_CLIENT) |
- BIT(NL80211_IFTYPE_P2P_GO) |
- BIT(NL80211_IFTYPE_ADHOC);
}
struct mt76_phy *
phy->hw = hw;
phy->priv = hw->priv + phy_size;
+ hw->wiphy->flags |= WIPHY_FLAG_IBSS_RSN;
+ hw->wiphy->interface_modes =
+ BIT(NL80211_IFTYPE_STATION) |
+ BIT(NL80211_IFTYPE_AP) |
+#ifdef CONFIG_MAC80211_MESH
+ BIT(NL80211_IFTYPE_MESH_POINT) |
+#endif
+ BIT(NL80211_IFTYPE_P2P_CLIENT) |
+ BIT(NL80211_IFTYPE_P2P_GO) |
+ BIT(NL80211_IFTYPE_ADHOC);
+
return phy;
}
EXPORT_SYMBOL_GPL(mt76_alloc_phy);
mutex_init(&dev->mcu.mutex);
dev->tx_worker.fn = mt76_tx_worker;
+ hw->wiphy->flags |= WIPHY_FLAG_IBSS_RSN;
+ hw->wiphy->interface_modes =
+ BIT(NL80211_IFTYPE_STATION) |
+ BIT(NL80211_IFTYPE_AP) |
+#ifdef CONFIG_MAC80211_MESH
+ BIT(NL80211_IFTYPE_MESH_POINT) |
+#endif
+ BIT(NL80211_IFTYPE_P2P_CLIENT) |
+ BIT(NL80211_IFTYPE_P2P_GO) |
+ BIT(NL80211_IFTYPE_ADHOC);
+
spin_lock_init(&dev->token_lock);
idr_init(&dev->token);
static void mt76_rx_release_amsdu(struct mt76_phy *phy, enum mt76_rxq_id q)
{
struct sk_buff *skb = phy->rx_amsdu[q].head;
+ struct mt76_rx_status *status = (struct mt76_rx_status *)skb->cb;
struct mt76_dev *dev = phy->dev;
phy->rx_amsdu[q].head = NULL;
phy->rx_amsdu[q].tail = NULL;
+
+ /*
+ * Validate if the amsdu has a proper first subframe.
+ * A single MSDU can be parsed as A-MSDU when the unauthenticated A-MSDU
+ * flag of the QoS header gets flipped. In such cases, the first
+ * subframe has a LLC/SNAP header in the location of the destination
+ * address.
+ */
+ if (skb_shinfo(skb)->frag_list) {
+ int offset = 0;
+
+ if (!(status->flag & RX_FLAG_8023)) {
+ offset = ieee80211_get_hdrlen_from_skb(skb);
+
+ if ((status->flag &
+ (RX_FLAG_DECRYPTED | RX_FLAG_IV_STRIPPED)) ==
+ RX_FLAG_DECRYPTED)
+ offset += 8;
+ }
+
+ if (ether_addr_equal(skb->data + offset, rfc1042_header)) {
+ dev_kfree_skb(skb);
+ return;
+ }
+ }
__skb_queue_tail(&dev->rx_skb[q], skb);
}
}
EXPORT_SYMBOL_GPL(mt76_update_survey_active_time);
-void mt76_update_survey(struct mt76_dev *dev)
+void mt76_update_survey(struct mt76_phy *phy)
{
+ struct mt76_dev *dev = phy->dev;
ktime_t cur_time;
if (dev->drv->update_survey)
- dev->drv->update_survey(dev);
+ dev->drv->update_survey(phy);
cur_time = ktime_get_boottime();
- mt76_update_survey_active_time(&dev->phy, cur_time);
- if (dev->phy2)
- mt76_update_survey_active_time(dev->phy2, cur_time);
+ mt76_update_survey_active_time(phy, cur_time);
if (dev->drv->drv_flags & MT_DRV_SW_RX_AIRTIME) {
- struct mt76_channel_state *state = dev->phy.chan_state;
+ struct mt76_channel_state *state = phy->chan_state;
spin_lock_bh(&dev->cc_lock);
state->cc_bss_rx += dev->cur_cc_bss_rx;
int timeout = HZ / 5;
wait_event_timeout(dev->tx_wait, !mt76_has_tx_pending(phy), timeout);
- mt76_update_survey(dev);
+ mt76_update_survey(phy);
phy->chandef = *chandef;
phy->chan_state = mt76_channel_state(phy, chandef->chan);
mutex_lock(&dev->mutex);
if (idx == 0 && dev->drv->update_survey)
- mt76_update_survey(dev);
+ mt76_update_survey(phy);
sband = &phy->sband_2g;
if (idx >= sband->sband.n_channels) {
__MT_RXQ_MAX
};
+enum mt76_cipher_type {
+ MT_CIPHER_NONE,
+ MT_CIPHER_WEP40,
+ MT_CIPHER_TKIP,
+ MT_CIPHER_TKIP_NO_MIC,
+ MT_CIPHER_AES_CCMP,
+ MT_CIPHER_WEP104,
+ MT_CIPHER_BIP_CMAC_128,
+ MT_CIPHER_WEP128,
+ MT_CIPHER_WAPI,
+ MT_CIPHER_CCMP_CCX,
+ MT_CIPHER_CCMP_256,
+ MT_CIPHER_GCMP,
+ MT_CIPHER_GCMP_256,
+};
+
struct mt76_queue_buf {
dma_addr_t addr;
u16 len;
struct mt76_hw_cap {
bool has_2ghz;
bool has_5ghz;
+ bool has_6ghz;
};
#define MT_DRV_TXWI_NO_FREE BIT(0)
u16 token_size;
u8 mcs_rates;
- void (*update_survey)(struct mt76_dev *dev);
+ void (*update_survey)(struct mt76_phy *phy);
int (*tx_prepare_skb)(struct mt76_dev *dev, void *txwi_ptr,
enum mt76_txq_id qid, struct mt76_wcid *wcid,
MT_PHY_TYPE_HE_MU,
};
+#define CCK_RATE(_idx, _rate) { \
+ .bitrate = _rate, \
+ .flags = IEEE80211_RATE_SHORT_PREAMBLE, \
+ .hw_value = (MT_PHY_TYPE_CCK << 8) | (_idx), \
+ .hw_value_short = (MT_PHY_TYPE_CCK << 8) | (4 + _idx), \
+}
+
+#define OFDM_RATE(_idx, _rate) { \
+ .bitrate = _rate, \
+ .hw_value = (MT_PHY_TYPE_OFDM << 8) | (_idx), \
+ .hw_value_short = (MT_PHY_TYPE_OFDM << 8) | (_idx), \
+}
+
+extern struct ieee80211_rate mt76_rates[12];
+
#define __mt76_rr(dev, ...) (dev)->bus->rr((dev), __VA_ARGS__)
#define __mt76_wr(dev, ...) (dev)->bus->wr((dev), __VA_ARGS__)
#define __mt76_rmw(dev, ...) (dev)->bus->rmw((dev), __VA_ARGS__)
bool more_data);
bool mt76_has_tx_pending(struct mt76_phy *phy);
void mt76_set_channel(struct mt76_phy *phy);
-void mt76_update_survey(struct mt76_dev *dev);
+void mt76_update_survey(struct mt76_phy *phy);
void mt76_update_survey_active_time(struct mt76_phy *phy, ktime_t time);
int mt76_get_survey(struct ieee80211_hw *hw, int idx,
struct survey_info *survey);
struct sk_buff_head *list);
void mt76_tx_status_skb_done(struct mt76_dev *dev, struct sk_buff *skb,
struct sk_buff_head *list);
-void mt76_tx_complete_skb(struct mt76_dev *dev, u16 wcid, struct sk_buff *skb);
+void __mt76_tx_complete_skb(struct mt76_dev *dev, u16 wcid, struct sk_buff *skb,
+ struct list_head *free_list);
+static inline void
+mt76_tx_complete_skb(struct mt76_dev *dev, u16 wcid, struct sk_buff *skb)
+{
+ __mt76_tx_complete_skb(dev, wcid, skb, NULL);
+}
+
void mt76_tx_status_check(struct mt76_dev *dev, struct mt76_wcid *wcid,
bool flush);
int mt76_sta_state(struct ieee80211_hw *hw, struct ieee80211_vif *vif,
return txwi;
}
+
+static inline int
+mt76_get_next_pkt_id(struct mt76_wcid *wcid)
+{
+ wcid->packet_id = (wcid->packet_id + 1) & MT_PACKET_ID_MASK;
+ if (wcid->packet_id == MT_PACKET_ID_NO_ACK ||
+ wcid->packet_id == MT_PACKET_ID_NO_SKB)
+ wcid->packet_id = MT_PACKET_ID_FIRST;
+
+ return wcid->packet_id;
+}
#endif
return 0;
}
-#define CCK_RATE(_idx, _rate) { \
- .bitrate = _rate, \
- .flags = IEEE80211_RATE_SHORT_PREAMBLE, \
- .hw_value = (MT_PHY_TYPE_CCK << 8) | (_idx), \
- .hw_value_short = (MT_PHY_TYPE_CCK << 8) | (4 + _idx), \
-}
-
-#define OFDM_RATE(_idx, _rate) { \
- .bitrate = _rate, \
- .hw_value = (MT_PHY_TYPE_OFDM << 8) | (_idx), \
- .hw_value_short = (MT_PHY_TYPE_OFDM << 8) | (_idx), \
-}
-
-static struct ieee80211_rate mt7603_rates[] = {
- CCK_RATE(0, 10),
- CCK_RATE(1, 20),
- CCK_RATE(2, 55),
- CCK_RATE(3, 110),
- OFDM_RATE(11, 60),
- OFDM_RATE(15, 90),
- OFDM_RATE(10, 120),
- OFDM_RATE(14, 180),
- OFDM_RATE(9, 240),
- OFDM_RATE(13, 360),
- OFDM_RATE(8, 480),
- OFDM_RATE(12, 540),
-};
-
static const struct ieee80211_iface_limit if_limits[] = {
{
.max = 1,
wiphy->reg_notifier = mt7603_regd_notifier;
- ret = mt76_register_device(&dev->mt76, true, mt7603_rates,
- ARRAY_SIZE(mt7603_rates));
+ ret = mt76_register_device(&dev->mt76, true, mt76_rates,
+ ARRAY_SIZE(mt76_rates));
if (ret)
return ret;
u8 *data = (u8 *)rxd;
if (status->flag & RX_FLAG_DECRYPTED) {
- status->iv[0] = data[5];
- status->iv[1] = data[4];
- status->iv[2] = data[3];
- status->iv[3] = data[2];
- status->iv[4] = data[1];
- status->iv[5] = data[0];
-
- insert_ccmp_hdr = FIELD_GET(MT_RXD2_NORMAL_FRAG, rxd2);
+ switch (FIELD_GET(MT_RXD2_NORMAL_SEC_MODE, rxd2)) {
+ case MT_CIPHER_AES_CCMP:
+ case MT_CIPHER_CCMP_CCX:
+ case MT_CIPHER_CCMP_256:
+ insert_ccmp_hdr =
+ FIELD_GET(MT_RXD2_NORMAL_FRAG, rxd2);
+ fallthrough;
+ case MT_CIPHER_TKIP:
+ case MT_CIPHER_TKIP_NO_MIC:
+ case MT_CIPHER_GCMP:
+ case MT_CIPHER_GCMP_256:
+ status->iv[0] = data[5];
+ status->iv[1] = data[4];
+ status->iv[2] = data[3];
+ status->iv[3] = data[2];
+ status->iv[4] = data[1];
+ status->iv[5] = data[0];
+ break;
+ default:
+ break;
+ }
}
rxd += 4;
sta->wcid.tx_info |= MT_WCID_TX_INFO_SET;
}
-static enum mt7603_cipher_type
+static enum mt76_cipher_type
mt7603_mac_get_key_info(struct ieee80211_key_conf *key, u8 *key_data)
{
memset(key_data, 0, 32);
int mt7603_wtbl_set_key(struct mt7603_dev *dev, int wcid,
struct ieee80211_key_conf *key)
{
- enum mt7603_cipher_type cipher;
+ enum mt76_cipher_type cipher;
u32 addr = mt7603_wtbl3_addr(wcid);
u8 key_data[32];
int key_len = sizeof(key_data);
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
if (!mt7603_fill_txs(dev, sta, info, txs_data)) {
- ieee80211_tx_info_clear_status(info);
+ info->status.rates[0].count = 0;
info->status.rates[0].idx = -1;
}
return true;
}
-void mt7603_update_channel(struct mt76_dev *mdev)
+void mt7603_update_channel(struct mt76_phy *mphy)
{
- struct mt7603_dev *dev = container_of(mdev, struct mt7603_dev, mt76);
+ struct mt7603_dev *dev = container_of(mphy->dev, struct mt7603_dev, mt76);
struct mt76_channel_state *state;
- state = mdev->phy.chan_state;
+ state = mphy->chan_state;
state->cc_busy += mt76_rr(dev, MT_MIB_STAT_CCA);
}
mutex_lock(&dev->mt76.mutex);
dev->mphy.mac_work_count++;
- mt76_update_survey(&dev->mt76);
+ mt76_update_survey(&dev->mphy);
mt7603_edcca_check(dev);
for (i = 0, idx = 0; i < 2; i++) {
void mt7603_pre_tbtt_tasklet(struct tasklet_struct *t);
-void mt7603_update_channel(struct mt76_dev *mdev);
+void mt7603_update_channel(struct mt76_phy *mphy);
void mt7603_edcca_set_strict(struct mt7603_dev *dev, bool val);
void mt7603_cca_stats_reset(struct mt7603_dev *dev);
#define MT_WTBL1_OR (MT_WTBL1_BASE + 0x2300)
#define MT_WTBL1_OR_PSM_WRITE BIT(31)
-enum mt7603_cipher_type {
- MT_CIPHER_NONE,
- MT_CIPHER_WEP40,
- MT_CIPHER_TKIP,
- MT_CIPHER_TKIP_NO_MIC,
- MT_CIPHER_AES_CCMP,
- MT_CIPHER_WEP104,
- MT_CIPHER_BIP_CMAC_128,
- MT_CIPHER_WEP128,
- MT_CIPHER_WAPI,
-};
-
#endif
-#SPDX-License-Identifier: ISC
+# SPDX-License-Identifier: ISC
obj-$(CONFIG_MT7615_COMMON) += mt7615-common.o
obj-$(CONFIG_MT7615E) += mt7615e.o
if (!mt7615_wait_for_mcu_init(dev))
return 0;
- if (!mt7615_firmware_offload(dev) || !mt76_is_mmio(&dev->mt76))
+ if (!mt7615_firmware_offload(dev) || mt76_is_usb(&dev->mt76))
return -EOPNOTSUPP;
if (val == pm->enable)
return 0;
}
-static int mt7615_read_temperature(struct seq_file *s, void *data)
-{
- struct mt7615_dev *dev = dev_get_drvdata(s->private);
- int temp;
-
- if (!mt7615_wait_for_mcu_init(dev))
- return 0;
-
- /* cpu */
- mt7615_mutex_acquire(dev);
- temp = mt7615_mcu_get_temperature(dev, 0);
- mt7615_mutex_release(dev);
-
- seq_printf(s, "Temperature: %d\n", temp);
-
- return 0;
-}
-
static int
mt7615_queues_acq(struct seq_file *s, void *data)
{
debugfs_create_file("reset_test", 0200, dir, dev,
&fops_reset_test);
- debugfs_create_devm_seqfile(dev->mt76.dev, "temperature", dir,
- mt7615_read_temperature);
debugfs_create_file("ext_mac_addr", 0600, dir, dev, &fops_ext_mac_addr);
debugfs_create_u32("rf_wfidx", 0600, dir, &dev->debugfs_rf_wf);
if (napi_complete(napi))
mt7615_irq_enable(dev, mt7615_tx_mcu_int_mask(dev));
- mt76_connac_pm_unref(&dev->pm);
+ mt76_connac_pm_unref(&dev->mphy, &dev->pm);
return 0;
}
return 0;
}
done = mt76_dma_rx_poll(napi, budget);
- mt76_connac_pm_unref(&dev->pm);
+ mt76_connac_pm_unref(&dev->mphy, &dev->pm);
return done;
}
int mt7615_dma_init(struct mt7615_dev *dev)
{
int rx_ring_size = MT7615_RX_RING_SIZE;
- int rx_buf_size = MT_RX_BUF_SIZE;
u32 mask;
int ret;
- /* Increase buffer size to receive large VHT MPDUs */
- if (dev->mphy.cap.has_5ghz)
- rx_buf_size *= 2;
-
mt76_dma_attach(&dev->mt76);
mt76_wr(dev, MT_WPDMA_GLO_CFG,
/* init rx queues */
ret = mt76_queue_alloc(dev, &dev->mt76.q_rx[MT_RXQ_MCU], 1,
- MT7615_RX_MCU_RING_SIZE, rx_buf_size,
+ MT7615_RX_MCU_RING_SIZE, MT_RX_BUF_SIZE,
MT_RX_RING_BASE);
if (ret)
return ret;
rx_ring_size /= 2;
ret = mt76_queue_alloc(dev, &dev->mt76.q_rx[MT_RXQ_MAIN], 0,
- rx_ring_size, rx_buf_size, MT_RX_RING_BASE);
+ rx_ring_size, MT_RX_BUF_SIZE, MT_RX_RING_BASE);
if (ret)
return ret;
*/
#include <linux/etherdevice.h>
+#include <linux/hwmon.h>
+#include <linux/hwmon-sysfs.h>
#include "mt7615.h"
#include "mac.h"
#include "mcu.h"
#include "eeprom.h"
+static ssize_t mt7615_thermal_show_temp(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct mt7615_dev *mdev = dev_get_drvdata(dev);
+ int temperature;
+
+ if (!mt7615_wait_for_mcu_init(mdev))
+ return 0;
+
+ mt7615_mutex_acquire(mdev);
+ temperature = mt7615_mcu_get_temperature(mdev);
+ mt7615_mutex_release(mdev);
+
+ if (temperature < 0)
+ return temperature;
+
+ /* display in millidegree celcius */
+ return sprintf(buf, "%u\n", temperature * 1000);
+}
+
+static SENSOR_DEVICE_ATTR(temp1_input, 0444, mt7615_thermal_show_temp,
+ NULL, 0);
+
+static struct attribute *mt7615_hwmon_attrs[] = {
+ &sensor_dev_attr_temp1_input.dev_attr.attr,
+ NULL,
+};
+ATTRIBUTE_GROUPS(mt7615_hwmon);
+
+int mt7615_thermal_init(struct mt7615_dev *dev)
+{
+ struct wiphy *wiphy = mt76_hw(dev)->wiphy;
+ struct device *hwmon;
+
+ if (!IS_REACHABLE(CONFIG_HWMON))
+ return 0;
+
+ hwmon = devm_hwmon_device_register_with_groups(&wiphy->dev,
+ wiphy_name(wiphy), dev,
+ mt7615_hwmon_groups);
+ if (IS_ERR(hwmon))
+ return PTR_ERR(hwmon);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mt7615_thermal_init);
+
static void
mt7615_phy_init(struct mt7615_dev *dev)
{
}
EXPORT_SYMBOL_GPL(mt7615_wait_for_mcu_init);
-#define CCK_RATE(_idx, _rate) { \
- .bitrate = _rate, \
- .flags = IEEE80211_RATE_SHORT_PREAMBLE, \
- .hw_value = (MT_PHY_TYPE_CCK << 8) | (_idx), \
- .hw_value_short = (MT_PHY_TYPE_CCK << 8) | (4 + (_idx)), \
-}
-
-#define OFDM_RATE(_idx, _rate) { \
- .bitrate = _rate, \
- .hw_value = (MT_PHY_TYPE_OFDM << 8) | (_idx), \
- .hw_value_short = (MT_PHY_TYPE_OFDM << 8) | (_idx), \
-}
-
-struct ieee80211_rate mt7615_rates[] = {
- CCK_RATE(0, 10),
- CCK_RATE(1, 20),
- CCK_RATE(2, 55),
- CCK_RATE(3, 110),
- OFDM_RATE(11, 60),
- OFDM_RATE(15, 90),
- OFDM_RATE(10, 120),
- OFDM_RATE(14, 180),
- OFDM_RATE(9, 240),
- OFDM_RATE(13, 360),
- OFDM_RATE(8, 480),
- OFDM_RATE(12, 540),
-};
-EXPORT_SYMBOL_GPL(mt7615_rates);
-
static const struct ieee80211_iface_limit if_limits[] = {
{
.max = 1,
wiphy->reg_notifier = mt7615_regd_notifier;
wiphy->max_sched_scan_plan_interval =
- MT76_CONNAC_MAX_SCHED_SCAN_INTERVAL;
+ MT76_CONNAC_MAX_TIME_SCHED_SCAN_INTERVAL;
wiphy->max_sched_scan_ie_len = IEEE80211_MAX_DATA_LEN;
wiphy->max_scan_ie_len = MT76_CONNAC_SCAN_IE_LEN;
wiphy->max_sched_scan_ssids = MT76_CONNAC_MAX_SCHED_SCAN_SSID;
for (i = 0; i <= MT_TXQ_PSD ; i++)
mphy->q_tx[i] = dev->mphy.q_tx[i];
- ret = mt76_register_phy(mphy, true, mt7615_rates,
- ARRAY_SIZE(mt7615_rates));
+ ret = mt76_register_phy(mphy, true, mt76_rates,
+ ARRAY_SIZE(mt76_rates));
if (ret)
ieee80211_free_hw(mphy->hw);
mutex_init(&dev->pm.mutex);
init_waitqueue_head(&dev->pm.wait);
spin_lock_init(&dev->pm.txq_lock);
- set_bit(MT76_STATE_PM, &dev->mphy.state);
INIT_DELAYED_WORK(&dev->mphy.mac_work, mt7615_mac_work);
INIT_DELAYED_WORK(&dev->phy.scan_work, mt7615_scan_work);
INIT_DELAYED_WORK(&dev->coredump.work, mt7615_coredump_work);
#define to_rssi(field, rxv) ((FIELD_GET(field, rxv) - 220) / 2)
static const struct mt7615_dfs_radar_spec etsi_radar_specs = {
- .pulse_th = { 40, -10, -80, 800, 3360, 128, 5200 },
+ .pulse_th = { 110, -10, -80, 40, 5200, 128, 5200 },
.radar_pattern = {
[5] = { 1, 0, 6, 32, 28, 0, 17, 990, 5010, 1, 1 },
[6] = { 1, 0, 9, 32, 28, 0, 27, 615, 5010, 1, 1 },
};
static const struct mt7615_dfs_radar_spec fcc_radar_specs = {
- .pulse_th = { 40, -10, -80, 800, 3360, 128, 5200 },
+ .pulse_th = { 110, -10, -80, 40, 5200, 128, 5200 },
.radar_pattern = {
[0] = { 1, 0, 9, 32, 28, 0, 13, 508, 3076, 1, 1 },
[1] = { 1, 0, 12, 32, 28, 0, 17, 140, 240, 1, 1 },
};
static const struct mt7615_dfs_radar_spec jp_radar_specs = {
- .pulse_th = { 40, -10, -80, 800, 3360, 128, 5200 },
+ .pulse_th = { 110, -10, -80, 40, 5200, 128, 5200 },
.radar_pattern = {
[0] = { 1, 0, 8, 32, 28, 0, 13, 508, 3076, 1, 1 },
[1] = { 1, 0, 12, 32, 28, 0, 17, 140, 240, 1, 1 },
},
};
+static enum mt76_cipher_type
+mt7615_mac_get_cipher(int cipher)
+{
+ switch (cipher) {
+ case WLAN_CIPHER_SUITE_WEP40:
+ return MT_CIPHER_WEP40;
+ case WLAN_CIPHER_SUITE_WEP104:
+ return MT_CIPHER_WEP104;
+ case WLAN_CIPHER_SUITE_TKIP:
+ return MT_CIPHER_TKIP;
+ case WLAN_CIPHER_SUITE_AES_CMAC:
+ return MT_CIPHER_BIP_CMAC_128;
+ case WLAN_CIPHER_SUITE_CCMP:
+ return MT_CIPHER_AES_CCMP;
+ case WLAN_CIPHER_SUITE_CCMP_256:
+ return MT_CIPHER_CCMP_256;
+ case WLAN_CIPHER_SUITE_GCMP:
+ return MT_CIPHER_GCMP;
+ case WLAN_CIPHER_SUITE_GCMP_256:
+ return MT_CIPHER_GCMP_256;
+ case WLAN_CIPHER_SUITE_SMS4:
+ return MT_CIPHER_WAPI;
+ default:
+ return MT_CIPHER_NONE;
+ }
+}
+
static struct mt76_wcid *mt7615_rx_get_wcid(struct mt7615_dev *dev,
u8 idx, bool unicast)
{
u8 *data = (u8 *)rxd;
if (status->flag & RX_FLAG_DECRYPTED) {
- status->iv[0] = data[5];
- status->iv[1] = data[4];
- status->iv[2] = data[3];
- status->iv[3] = data[2];
- status->iv[4] = data[1];
- status->iv[5] = data[0];
-
- insert_ccmp_hdr = FIELD_GET(MT_RXD2_NORMAL_FRAG, rxd2);
+ switch (FIELD_GET(MT_RXD2_NORMAL_SEC_MODE, rxd2)) {
+ case MT_CIPHER_AES_CCMP:
+ case MT_CIPHER_CCMP_CCX:
+ case MT_CIPHER_CCMP_256:
+ insert_ccmp_hdr =
+ FIELD_GET(MT_RXD2_NORMAL_FRAG, rxd2);
+ fallthrough;
+ case MT_CIPHER_TKIP:
+ case MT_CIPHER_TKIP_NO_MIC:
+ case MT_CIPHER_GCMP:
+ case MT_CIPHER_GCMP_256:
+ status->iv[0] = data[5];
+ status->iv[1] = data[4];
+ status->iv[2] = data[3];
+ status->iv[3] = data[2];
+ status->iv[4] = data[1];
+ status->iv[5] = data[0];
+ break;
+ default:
+ break;
+ }
}
rxd += 4;
if ((u8 *)rxd - skb->data >= skb->len)
idx = idx > HW_BSSID_MAX ? HW_BSSID_0 : idx;
addr = idx > 1 ? MT_LPON_TCR2(idx): MT_LPON_TCR0(idx);
- mt76_set(dev, addr, MT_LPON_TCR_MODE); /* TSF read */
+ mt76_rmw(dev, addr, MT_LPON_TCR_MODE, MT_LPON_TCR_READ); /* TSF read */
sta->rate_set_tsf = mt76_rr(dev, MT_LPON_UTTR0) & ~BIT(0);
sta->rate_set_tsf |= rd.rateset;
static int
mt7615_mac_wtbl_update_key(struct mt7615_dev *dev, struct mt76_wcid *wcid,
struct ieee80211_key_conf *key,
- enum mt7615_cipher_type cipher, u16 cipher_mask,
+ enum mt76_cipher_type cipher, u16 cipher_mask,
enum set_key_cmd cmd)
{
u32 addr = mt7615_mac_wtbl_addr(dev, wcid->idx) + 30 * 4;
static int
mt7615_mac_wtbl_update_pk(struct mt7615_dev *dev, struct mt76_wcid *wcid,
- enum mt7615_cipher_type cipher, u16 cipher_mask,
+ enum mt76_cipher_type cipher, u16 cipher_mask,
int keyidx, enum set_key_cmd cmd)
{
u32 addr = mt7615_mac_wtbl_addr(dev, wcid->idx), w0, w1;
static void
mt7615_mac_wtbl_update_cipher(struct mt7615_dev *dev, struct mt76_wcid *wcid,
- enum mt7615_cipher_type cipher, u16 cipher_mask,
+ enum mt76_cipher_type cipher, u16 cipher_mask,
enum set_key_cmd cmd)
{
u32 addr = mt7615_mac_wtbl_addr(dev, wcid->idx);
struct ieee80211_key_conf *key,
enum set_key_cmd cmd)
{
- enum mt7615_cipher_type cipher;
+ enum mt76_cipher_type cipher;
u16 cipher_mask = wcid->cipher;
int err;
int first_idx = 0, last_idx;
int i, idx, count;
bool fixed_rate, ack_timeout;
- bool probe, ampdu, cck = false;
+ bool ampdu, cck = false;
bool rs_idx;
u32 rate_set_tsf;
u32 final_rate, final_rate_flags, final_nss, txs;
- fixed_rate = info->status.rates[0].count;
- probe = !!(info->flags & IEEE80211_TX_CTL_RATE_CTRL_PROBE);
-
txs = le32_to_cpu(txs_data[1]);
- ampdu = !fixed_rate && (txs & MT_TXS1_AMPDU);
+ ampdu = txs & MT_TXS1_AMPDU;
txs = le32_to_cpu(txs_data[3]);
count = FIELD_GET(MT_TXS3_TX_COUNT, txs);
last_idx = FIELD_GET(MT_TXS3_LAST_TX_RATE, txs);
txs = le32_to_cpu(txs_data[0]);
+ fixed_rate = txs & MT_TXS0_FIXED_RATE;
final_rate = FIELD_GET(MT_TXS0_TX_RATE, txs);
ack_timeout = txs & MT_TXS0_ACK_TIMEOUT;
first_idx = max_t(int, 0, last_idx - (count - 1) / MT7615_RATE_RETRY);
- if (fixed_rate && !probe) {
+ if (fixed_rate) {
info->status.rates[0].count = count;
i = 0;
goto out;
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
if (!mt7615_fill_txs(dev, sta, info, txs_data)) {
- ieee80211_tx_info_clear_status(info);
+ info->status.rates[0].count = 0;
info->status.rates[0].idx = -1;
}
state->noise = -(phy->noise >> 4);
}
-static void __mt7615_update_channel(struct mt7615_dev *dev)
+static void mt7615_update_survey(struct mt7615_dev *dev)
{
struct mt76_dev *mdev = &dev->mt76;
+ ktime_t cur_time;
+
+ /* MT7615 can only update both phys simultaneously
+ * since some reisters are shared across bands.
+ */
mt7615_phy_update_channel(&mdev->phy, 0);
if (mdev->phy2)
mt7615_phy_update_channel(mdev->phy2, 1);
+ cur_time = ktime_get_boottime();
+
+ mt76_update_survey_active_time(&mdev->phy, cur_time);
+ if (mdev->phy2)
+ mt76_update_survey_active_time(mdev->phy2, cur_time);
+
/* reset obss airtime */
mt76_set(dev, MT_WF_RMAC_MIB_TIME0, MT_WF_RMAC_MIB_RXTIME_CLR);
}
-void mt7615_update_channel(struct mt76_dev *mdev)
+void mt7615_update_channel(struct mt76_phy *mphy)
{
- struct mt7615_dev *dev = container_of(mdev, struct mt7615_dev, mt76);
+ struct mt7615_dev *dev = container_of(mphy->dev, struct mt7615_dev, mt76);
if (mt76_connac_pm_wake(&dev->mphy, &dev->pm))
return;
- __mt7615_update_channel(dev);
+ mt7615_update_survey(dev);
mt76_connac_power_save_sched(&dev->mphy, &dev->pm);
}
EXPORT_SYMBOL_GPL(mt7615_update_channel);
-static void mt7615_update_survey(struct mt7615_dev *dev)
-{
- struct mt76_dev *mdev = &dev->mt76;
- ktime_t cur_time;
-
- __mt7615_update_channel(dev);
- cur_time = ktime_get_boottime();
-
- mt76_update_survey_active_time(&mdev->phy, cur_time);
- if (mdev->phy2)
- mt76_update_survey_active_time(mdev->phy2, cur_time);
-}
-
static void
mt7615_mac_update_mib_stats(struct mt7615_phy *phy)
{
mphy = dev->phy.mt76;
if (!mt7615_mcu_set_drv_ctrl(dev)) {
+ struct mt76_dev *mdev = &dev->mt76;
int i;
- mt76_for_each_q_rx(&dev->mt76, i)
- napi_schedule(&dev->mt76.napi[i]);
- mt76_connac_pm_dequeue_skbs(mphy, &dev->pm);
- mt76_queue_tx_cleanup(dev, dev->mt76.q_mcu[MT_MCUQ_WM], false);
- ieee80211_queue_delayed_work(mphy->hw, &mphy->mac_work,
- MT7615_WATCHDOG_TIME);
+ if (mt76_is_sdio(mdev)) {
+ mt76_worker_schedule(&mdev->sdio.txrx_worker);
+ } else {
+ mt76_for_each_q_rx(mdev, i)
+ napi_schedule(&mdev->napi[i]);
+ mt76_connac_pm_dequeue_skbs(mphy, &dev->pm);
+ mt76_queue_tx_cleanup(dev, mdev->q_mcu[MT_MCUQ_WM],
+ false);
+ }
+
+ if (test_bit(MT76_STATE_RUNNING, &mphy->state)) {
+ unsigned long timeout;
+
+ timeout = mt7615_get_macwork_timeout(dev);
+ ieee80211_queue_delayed_work(mphy->hw, &mphy->mac_work,
+ timeout);
+ }
}
ieee80211_wake_queues(mphy->hw);
{
struct mt7615_phy *phy;
struct mt76_phy *mphy;
+ unsigned long timeout;
mphy = (struct mt76_phy *)container_of(work, struct mt76_phy,
mac_work.work);
mt7615_mutex_release(phy->dev);
mt76_tx_status_check(mphy->dev, NULL, false);
- ieee80211_queue_delayed_work(mphy->hw, &mphy->mac_work,
- MT7615_WATCHDOG_TIME);
+
+ timeout = mt7615_get_macwork_timeout(phy->dev);
+ ieee80211_queue_delayed_work(mphy->hw, &mphy->mac_work, timeout);
}
void mt7615_tx_token_put(struct mt7615_dev *dev)
{
const struct mt7615_dfs_radar_spec *radar_specs;
struct mt7615_dev *dev = phy->dev;
- int err, i;
+ int err, i, lpn = 500;
switch (dev->mt76.region) {
case NL80211_DFS_FCC:
radar_specs = &fcc_radar_specs;
- err = mt7615_mcu_set_fcc5_lpn(dev, 8);
- if (err < 0)
- return err;
+ lpn = 8;
break;
case NL80211_DFS_ETSI:
radar_specs = &etsi_radar_specs;
return -EINVAL;
}
+ /* avoid FCC radar detection in non-FCC region */
+ err = mt7615_mcu_set_fcc5_lpn(dev, lpn);
+ if (err < 0)
+ return err;
+
for (i = 0; i < ARRAY_SIZE(radar_specs->radar_pattern); i++) {
err = mt7615_mcu_set_radar_th(dev, i,
&radar_specs->radar_pattern[i]);
struct mt7615_dfs_pattern radar_pattern[16];
};
-enum mt7615_cipher_type {
- MT_CIPHER_NONE,
- MT_CIPHER_WEP40,
- MT_CIPHER_TKIP,
- MT_CIPHER_TKIP_NO_MIC,
- MT_CIPHER_AES_CCMP,
- MT_CIPHER_WEP104,
- MT_CIPHER_BIP_CMAC_128,
- MT_CIPHER_WEP128,
- MT_CIPHER_WAPI,
- MT_CIPHER_CCMP_256 = 10,
- MT_CIPHER_GCMP,
- MT_CIPHER_GCMP_256,
-};
-
-static inline enum mt7615_cipher_type
-mt7615_mac_get_cipher(int cipher)
-{
- switch (cipher) {
- case WLAN_CIPHER_SUITE_WEP40:
- return MT_CIPHER_WEP40;
- case WLAN_CIPHER_SUITE_WEP104:
- return MT_CIPHER_WEP104;
- case WLAN_CIPHER_SUITE_TKIP:
- return MT_CIPHER_TKIP;
- case WLAN_CIPHER_SUITE_AES_CMAC:
- return MT_CIPHER_BIP_CMAC_128;
- case WLAN_CIPHER_SUITE_CCMP:
- return MT_CIPHER_AES_CCMP;
- case WLAN_CIPHER_SUITE_CCMP_256:
- return MT_CIPHER_CCMP_256;
- case WLAN_CIPHER_SUITE_GCMP:
- return MT_CIPHER_GCMP;
- case WLAN_CIPHER_SUITE_GCMP_256:
- return MT_CIPHER_GCMP_256;
- case WLAN_CIPHER_SUITE_SMS4:
- return MT_CIPHER_WAPI;
- default:
- return MT_CIPHER_NONE;
- }
-}
-
static inline struct mt7615_txp_common *
mt7615_txwi_to_txp(struct mt76_dev *dev, struct mt76_txwi_cache *t)
{
{
struct mt7615_dev *dev = mt7615_hw_dev(hw);
struct mt7615_phy *phy = mt7615_hw_phy(hw);
+ unsigned long timeout;
bool running;
int ret;
set_bit(MT76_STATE_RUNNING, &phy->mt76->state);
- ieee80211_queue_delayed_work(hw, &phy->mt76->mac_work,
- MT7615_WATCHDOG_TIME);
+ timeout = mt7615_get_macwork_timeout(dev);
+ ieee80211_queue_delayed_work(hw, &phy->mt76->mac_work, timeout);
if (!running)
mt7615_mac_reset_counters(dev);
}
ret = mt7615_mcu_add_dev_info(phy, vif, true);
- if (ret)
- goto out;
out:
mt7615_mutex_release(dev);
mt7615_mutex_release(dev);
mt76_worker_schedule(&dev->mt76.tx_worker);
- if (!mt76_testmode_enabled(phy->mt76))
+ if (!mt76_testmode_enabled(phy->mt76)) {
+ unsigned long timeout = mt7615_get_macwork_timeout(dev);
+
ieee80211_queue_delayed_work(phy->mt76->hw,
- &phy->mt76->mac_work,
- MT7615_WATCHDOG_TIME);
+ &phy->mt76->mac_work, timeout);
+ }
return ret;
}
msta->n_rates = i;
if (mt76_connac_pm_ref(phy->mt76, &dev->pm)) {
mt7615_mac_set_rates(phy, msta, NULL, msta->rates);
- mt76_connac_pm_unref(&dev->pm);
+ mt76_connac_pm_unref(phy->mt76, &dev->pm);
}
spin_unlock_bh(&dev->mt76.lock);
}
}
mt76_tx_worker_run(&dev->mt76);
- mt76_connac_pm_unref(&dev->pm);
+ mt76_connac_pm_unref(&dev->mphy, &dev->pm);
}
static void mt7615_tx(struct ieee80211_hw *hw,
if (mt76_connac_pm_ref(mphy, &dev->pm)) {
mt76_tx(mphy, control->sta, wcid, skb);
- mt76_connac_pm_unref(&dev->pm);
+ mt76_connac_pm_unref(mphy, &dev->pm);
return;
}
mt7615_mutex_acquire(dev);
- mt76_set(dev, reg, MT_LPON_TCR_MODE); /* TSF read */
+ /* TSF read */
+ mt76_rmw(dev, reg, MT_LPON_TCR_MODE, MT_LPON_TCR_READ);
tsf.t32[0] = mt76_rr(dev, MT_LPON_UTTR0);
tsf.t32[1] = mt76_rr(dev, MT_LPON_UTTR1);
mt76_wr(dev, MT_LPON_UTTR0, tsf.t32[0]);
mt76_wr(dev, MT_LPON_UTTR1, tsf.t32[1]);
/* TSF software overwrite */
- mt76_set(dev, reg, MT_LPON_TCR_WRITE);
+ mt76_rmw(dev, reg, MT_LPON_TCR_MODE, MT_LPON_TCR_WRITE);
+
+ mt7615_mutex_release(dev);
+}
+
+static void
+mt7615_offset_tsf(struct ieee80211_hw *hw, struct ieee80211_vif *vif,
+ s64 timestamp)
+{
+ struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv;
+ struct mt7615_dev *dev = mt7615_hw_dev(hw);
+ union {
+ u64 t64;
+ u32 t32[2];
+ } tsf = { .t64 = timestamp, };
+ u16 idx = mvif->mt76.omac_idx;
+ u32 reg;
+
+ idx = idx > HW_BSSID_MAX ? HW_BSSID_0 : idx;
+ reg = idx > 1 ? MT_LPON_TCR2(idx): MT_LPON_TCR0(idx);
+
+ mt7615_mutex_acquire(dev);
+
+ mt76_wr(dev, MT_LPON_UTTR0, tsf.t32[0]);
+ mt76_wr(dev, MT_LPON_UTTR1, tsf.t32[1]);
+ /* TSF software adjust*/
+ mt76_rmw(dev, reg, MT_LPON_TCR_MODE, MT_LPON_TCR_ADJUST);
mt7615_mutex_release(dev);
}
else
clear_bit(MT_WCID_FLAG_HDR_TRANS, &msta->wcid.flags);
- mt7615_mcu_sta_update_hdr_trans(dev, vif, sta);
+ mt7615_mcu_set_sta_decap_offload(dev, vif, sta);
}
#ifdef CONFIG_PM
{
struct mt7615_phy *phy = mt7615_hw_phy(hw);
struct mt7615_dev *dev = mt7615_hw_dev(hw);
+ unsigned long timeout;
bool running;
mt7615_mutex_acquire(dev);
mt76_connac_mcu_set_suspend_iter,
phy->mt76);
- ieee80211_queue_delayed_work(hw, &phy->mt76->mac_work,
- MT7615_WATCHDOG_TIME);
+ timeout = mt7615_get_macwork_timeout(dev);
+ ieee80211_queue_delayed_work(hw, &phy->mt76->mac_work, timeout);
mt7615_mutex_release(dev);
.get_stats = mt7615_get_stats,
.get_tsf = mt7615_get_tsf,
.set_tsf = mt7615_set_tsf,
+ .offset_tsf = mt7615_offset_tsf,
.get_survey = mt76_get_survey,
.get_antenna = mt76_get_antenna,
.set_antenna = mt7615_set_antenna,
c = (struct mt7615_mcu_csa_notify *)skb->data;
+ if (c->omac_idx > EXT_BSSID_MAX)
+ return;
+
if (ext_phy && ext_phy->omac_mask & BIT_ULL(c->omac_idx))
mphy = dev->mt76.phy2;
r = (struct mt7615_mcu_rdd_report *)skb->data;
+ if (!dev->radar_pattern.n_pulses && !r->long_detected &&
+ !r->constant_prf_detected && !r->staggered_prf_detected)
+ return;
+
if (r->band_idx && dev->mt76.phy2)
mphy = dev->mt76.phy2;
if (IS_ERR(sskb))
return PTR_ERR(sskb);
- mt76_connac_mcu_sta_basic_tlv(sskb, vif, sta, enable);
+ mt76_connac_mcu_sta_basic_tlv(sskb, vif, sta, enable, true);
if (enable && sta)
- mt76_connac_mcu_sta_tlv(phy->mt76, sskb, sta, vif, 0);
+ mt76_connac_mcu_sta_tlv(phy->mt76, sskb, sta, vif, 0,
+ MT76_STA_INFO_STATE_ASSOC);
wtbl_hdr = mt76_connac_mcu_alloc_wtbl_req(&dev->mt76, &msta->wcid,
WTBL_RESET_AND_SET, NULL,
if (sta)
mt76_connac_mcu_wtbl_ht_tlv(&dev->mt76, wskb, sta,
NULL, wtbl_hdr);
- mt76_connac_mcu_wtbl_hdr_trans_tlv(wskb, &msta->wcid, NULL,
- wtbl_hdr);
+ mt76_connac_mcu_wtbl_hdr_trans_tlv(wskb, vif, &msta->wcid,
+ NULL, wtbl_hdr);
}
cmd = enable ? MCU_EXT_CMD_WTBL_UPDATE : MCU_EXT_CMD_STA_REC_UPDATE;
return mt76_mcu_skb_send_msg(&dev->mt76, skb, cmd, true);
}
+static int
+mt7615_mcu_wtbl_update_hdr_trans(struct mt7615_dev *dev,
+ struct ieee80211_vif *vif,
+ struct ieee80211_sta *sta)
+{
+ struct mt7615_sta *msta = (struct mt7615_sta *)sta->drv_priv;
+ struct wtbl_req_hdr *wtbl_hdr;
+ struct sk_buff *skb = NULL;
+
+ wtbl_hdr = mt76_connac_mcu_alloc_wtbl_req(&dev->mt76, &msta->wcid,
+ WTBL_SET, NULL, &skb);
+ if (IS_ERR(wtbl_hdr))
+ return PTR_ERR(wtbl_hdr);
+
+ mt76_connac_mcu_wtbl_hdr_trans_tlv(skb, vif, &msta->wcid, NULL,
+ wtbl_hdr);
+ return mt76_mcu_skb_send_msg(&dev->mt76, skb, MCU_EXT_CMD_WTBL_UPDATE,
+ true);
+}
+
static const struct mt7615_mcu_ops wtbl_update_ops = {
.add_beacon_offload = mt7615_mcu_add_beacon_offload,
.set_pm_state = mt7615_mcu_ctrl_pm_state,
.sta_add = mt7615_mcu_wtbl_sta_add,
.set_drv_ctrl = mt7615_mcu_drv_pmctrl,
.set_fw_ctrl = mt7615_mcu_fw_pmctrl,
+ .set_sta_decap_offload = mt7615_mcu_wtbl_update_hdr_trans,
};
static int
static int
__mt7615_mcu_add_sta(struct mt76_phy *phy, struct ieee80211_vif *vif,
- struct ieee80211_sta *sta, bool enable, int cmd)
+ struct ieee80211_sta *sta, bool enable, int cmd,
+ bool offload_fw)
{
struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv;
struct mt76_sta_cmd_info info = {
.sta = sta,
.vif = vif,
+ .offload_fw = offload_fw,
.enable = enable,
+ .newly = true,
.cmd = cmd,
};
info.wcid = sta ? (struct mt76_wcid *)sta->drv_priv : &mvif->sta.wcid;
- return mt76_connac_mcu_add_sta_cmd(phy, &info);
+ return mt76_connac_mcu_sta_cmd(phy, &info);
}
static int
struct ieee80211_sta *sta, bool enable)
{
return __mt7615_mcu_add_sta(phy->mt76, vif, sta, enable,
- MCU_EXT_CMD_STA_REC_UPDATE);
+ MCU_EXT_CMD_STA_REC_UPDATE, false);
+}
+
+static int
+mt7615_mcu_sta_update_hdr_trans(struct mt7615_dev *dev,
+ struct ieee80211_vif *vif,
+ struct ieee80211_sta *sta)
+{
+ struct mt7615_sta *msta = (struct mt7615_sta *)sta->drv_priv;
+
+ return mt76_connac_mcu_sta_update_hdr_trans(&dev->mt76,
+ vif, &msta->wcid,
+ MCU_EXT_CMD_STA_REC_UPDATE);
}
static const struct mt7615_mcu_ops sta_update_ops = {
.sta_add = mt7615_mcu_add_sta,
.set_drv_ctrl = mt7615_mcu_drv_pmctrl,
.set_fw_ctrl = mt7615_mcu_fw_pmctrl,
+ .set_sta_decap_offload = mt7615_mcu_sta_update_hdr_trans,
};
-int mt7615_mcu_sta_update_hdr_trans(struct mt7615_dev *dev,
- struct ieee80211_vif *vif,
- struct ieee80211_sta *sta)
-{
- struct mt7615_sta *msta = (struct mt7615_sta *)sta->drv_priv;
- struct wtbl_req_hdr *wtbl_hdr;
- struct sk_buff *skb = NULL;
-
- wtbl_hdr = mt76_connac_mcu_alloc_wtbl_req(&dev->mt76, &msta->wcid,
- WTBL_SET, NULL, &skb);
- if (IS_ERR(wtbl_hdr))
- return PTR_ERR(wtbl_hdr);
-
- mt76_connac_mcu_wtbl_hdr_trans_tlv(skb, &msta->wcid, NULL, wtbl_hdr);
-
- return mt76_mcu_skb_send_msg(&dev->mt76, skb, MCU_EXT_CMD_WTBL_UPDATE,
- true);
-}
-
static int
mt7615_mcu_uni_ctrl_pm_state(struct mt7615_dev *dev, int band, int state)
{
struct ieee80211_sta *sta, bool enable)
{
return __mt7615_mcu_add_sta(phy->mt76, vif, sta, enable,
- MCU_UNI_CMD_STA_REC_UPDATE);
+ MCU_UNI_CMD_STA_REC_UPDATE, true);
}
static int
MCU_UNI_CMD_STA_REC_UPDATE, true);
}
+static int
+mt7615_mcu_sta_uni_update_hdr_trans(struct mt7615_dev *dev,
+ struct ieee80211_vif *vif,
+ struct ieee80211_sta *sta)
+{
+ struct mt7615_sta *msta = (struct mt7615_sta *)sta->drv_priv;
+
+ return mt76_connac_mcu_sta_update_hdr_trans(&dev->mt76,
+ vif, &msta->wcid,
+ MCU_UNI_CMD_STA_REC_UPDATE);
+}
+
static const struct mt7615_mcu_ops uni_update_ops = {
.add_beacon_offload = mt7615_mcu_uni_add_beacon_offload,
.set_pm_state = mt7615_mcu_uni_ctrl_pm_state,
.sta_add = mt7615_mcu_uni_add_sta,
.set_drv_ctrl = mt7615_mcu_lp_drv_pmctrl,
.set_fw_ctrl = mt7615_mcu_fw_pmctrl,
+ .set_sta_decap_offload = mt7615_mcu_sta_uni_update_hdr_trans,
};
int mt7615_mcu_restart(struct mt76_dev *dev)
return mt76_mcu_send_msg(&dev->mt76, cmd, &req, sizeof(req), true);
}
-int mt7615_mcu_get_temperature(struct mt7615_dev *dev, int index)
+int mt7615_mcu_get_temperature(struct mt7615_dev *dev)
{
struct {
u8 action;
u8 rsv[3];
- } req = {
- .action = index,
- };
+ } req = {};
return mt76_mcu_send_msg(&dev->mt76, MCU_EXT_CMD_GET_TEMP, &req,
sizeof(req), true);
GFP_KERNEL);
if (!bus_ops) {
ret = -ENOMEM;
- goto error;
+ goto err_free_dev;
}
bus_ops->rr = mt7615_rr;
ret = devm_request_irq(mdev->dev, irq, mt7615_irq_handler,
IRQF_SHARED, KBUILD_MODNAME, dev);
if (ret)
- goto error;
+ goto err_free_dev;
if (is_mt7663(mdev))
mt76_wr(dev, MT_PCIE_IRQ_ENABLE, 1);
ret = mt7615_register_device(dev);
if (ret)
- goto error;
+ goto err_free_irq;
return 0;
-error:
+
+err_free_irq:
+ devm_free_irq(pdev, irq, dev);
+err_free_dev:
mt76_free_device(&dev->mt76);
return ret;
MT7615_MAX_INTERFACES)
#define MT7615_PM_TIMEOUT (HZ / 12)
-#define MT7615_WATCHDOG_TIME (HZ / 10)
#define MT7615_HW_SCAN_TIMEOUT (HZ / 10)
#define MT7615_RESET_TIMEOUT (30 * HZ)
#define MT7615_RATE_RETRY 2
#define mt7615_mcu_set_pm(dev, ...) (dev)->mcu_ops->set_pm_state((dev), __VA_ARGS__)
#define mt7615_mcu_set_drv_ctrl(dev) (dev)->mcu_ops->set_drv_ctrl((dev))
#define mt7615_mcu_set_fw_ctrl(dev) (dev)->mcu_ops->set_fw_ctrl((dev))
+#define mt7615_mcu_set_sta_decap_offload(dev, ...) (dev)->mcu_ops->set_sta_decap_offload((dev), __VA_ARGS__)
struct mt7615_mcu_ops {
int (*add_tx_ba)(struct mt7615_dev *dev,
struct ieee80211_ampdu_params *params,
int (*set_pm_state)(struct mt7615_dev *dev, int band, int state);
int (*set_drv_ctrl)(struct mt7615_dev *dev);
int (*set_fw_ctrl)(struct mt7615_dev *dev);
+ int (*set_sta_decap_offload)(struct mt7615_dev *dev,
+ struct ieee80211_vif *vif,
+ struct ieee80211_sta *sta);
};
struct mt7615_dev {
}
#endif
+int mt7615_thermal_init(struct mt7615_dev *dev);
int mt7615_mmio_probe(struct device *pdev, void __iomem *mem_base,
int irq, const u32 *map);
u32 mt7615_reg_map(struct mt7615_dev *dev, u32 addr);
return MT_INT_TX_DONE(dev->mt76.q_mcu[MT_MCUQ_WM]->hw_idx);
}
+static inline unsigned long
+mt7615_get_macwork_timeout(struct mt7615_dev *dev)
+{
+ return dev->pm.enable ? HZ / 3 : HZ / 10;
+}
+
void mt7615_dma_reset(struct mt7615_dev *dev);
void mt7615_scan_work(struct work_struct *work);
void mt7615_roc_work(struct work_struct *work);
void mt7615_init_work(struct mt7615_dev *dev);
int mt7615_mcu_restart(struct mt76_dev *dev);
-void mt7615_update_channel(struct mt76_dev *mdev);
+void mt7615_update_channel(struct mt76_phy *mphy);
bool mt7615_mac_wtbl_update(struct mt7615_dev *dev, int idx, u32 mask);
void mt7615_mac_reset_counters(struct mt7615_dev *dev);
void mt7615_mac_cca_stats_reset(struct mt7615_phy *phy);
int mt7615_rf_wr(struct mt7615_dev *dev, u32 wf, u32 reg, u32 val);
int mt7615_mcu_set_dbdc(struct mt7615_dev *dev);
int mt7615_mcu_set_eeprom(struct mt7615_dev *dev);
-int mt7615_mcu_get_temperature(struct mt7615_dev *dev, int index);
+int mt7615_mcu_get_temperature(struct mt7615_dev *dev);
int mt7615_mcu_set_tx_power(struct mt7615_phy *phy);
void mt7615_mcu_exit(struct mt7615_dev *dev);
void mt7615_mcu_fill_msg(struct mt7615_dev *dev, struct sk_buff *skb,
void mt7615_mac_work(struct work_struct *work);
void mt7615_txp_skb_unmap(struct mt76_dev *dev,
struct mt76_txwi_cache *txwi);
-int mt7615_mcu_sta_update_hdr_trans(struct mt7615_dev *dev,
- struct ieee80211_vif *vif,
- struct ieee80211_sta *sta);
int mt7615_mcu_set_rx_hdr_trans_blacklist(struct mt7615_dev *dev);
int mt7615_mcu_set_fcc5_lpn(struct mt7615_dev *dev, int val);
int mt7615_mcu_set_pulse_th(struct mt7615_dev *dev,
addr = mt7615_reg_map(dev, MT_LED_CTRL);
mt76_wr(dev, addr, val);
- mt76_connac_pm_unref(&dev->pm);
+ mt76_connac_pm_unref(&dev->mphy, &dev->pm);
}
static int
if (ret)
return ret;
- ret = mt76_register_device(&dev->mt76, true, mt7615_rates,
- ARRAY_SIZE(mt7615_rates));
+ ret = mt76_register_device(&dev->mt76, true, mt76_rates,
+ ARRAY_SIZE(mt76_rates));
+ if (ret)
+ return ret;
+
+ ret = mt7615_thermal_init(dev);
if (ret)
return ret;
struct mt76_tx_info *tx_info)
{
struct mt7615_dev *dev = container_of(mdev, struct mt7615_dev, mt76);
- struct mt7615_sta *msta = container_of(wcid, struct mt7615_sta, wcid);
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(tx_info->skb);
struct ieee80211_key_conf *key = info->control.hw_key;
int pid, id;
u8 *txwi = (u8 *)txwi_ptr;
struct mt76_txwi_cache *t;
+ struct mt7615_sta *msta;
void *txp;
+ msta = wcid ? container_of(wcid, struct mt7615_sta, wcid) : NULL;
if (!wcid)
wcid = &dev->mt76.global_wcid;
pid = mt76_tx_status_skb_add(mdev, wcid, tx_info->skb);
- if (info->flags & IEEE80211_TX_CTL_RATE_CTRL_PROBE) {
+ if ((info->flags & IEEE80211_TX_CTL_RATE_CTRL_PROBE) && msta) {
struct mt7615_phy *phy = &dev->phy;
if ((info->hw_queue & MT_TX_HW_QUEUE_EXT_PHY) && mdev->phy2)
struct mt7615_phy *phy2;
struct mt76_phy *ext_phy;
struct mt7615_dev *dev;
+ unsigned long timeout;
dev = container_of(work, struct mt7615_dev, reset_work);
ext_phy = dev->mt76.phy2;
mt7615_mutex_release(dev);
+ timeout = mt7615_get_macwork_timeout(dev);
ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mphy.mac_work,
- MT7615_WATCHDOG_TIME);
+ timeout);
if (phy2)
ieee80211_queue_delayed_work(ext_phy->hw,
- &phy2->mt76->mac_work,
- MT7615_WATCHDOG_TIME);
+ &phy2->mt76->mac_work, timeout);
}
#define MT_LPON_TCR0(_n) MT_LPON(0x010 + ((_n) * 4))
#define MT_LPON_TCR2(_n) MT_LPON(0x0f8 + ((_n) - 2) * 4)
#define MT_LPON_TCR_MODE GENMASK(1, 0)
+#define MT_LPON_TCR_READ GENMASK(1, 0)
#define MT_LPON_TCR_WRITE BIT(0)
+#define MT_LPON_TCR_ADJUST BIT(1)
#define MT_LPON_UTTR0 MT_LPON(0x018)
#define MT_LPON_UTTR1 MT_LPON(0x01c)
-// SPDX-License-Identifier: ISC
+/* SPDX-License-Identifier: ISC */
/* Copyright (C) 2020 MediaTek Inc.
*
* Author: Sean Wang <sean.wang@mediatek.com>
return ret;
}
-static int mt7663s_mcu_drv_pmctrl(struct mt7615_dev *dev)
+static int __mt7663s_mcu_drv_pmctrl(struct mt7615_dev *dev)
{
struct sdio_func *func = dev->mt76.sdio.func;
struct mt76_phy *mphy = &dev->mt76.phy;
+ struct mt76_connac_pm *pm = &dev->pm;
u32 status;
int ret;
- if (!test_and_clear_bit(MT76_STATE_PM, &mphy->state))
- goto out;
-
sdio_claim_host(func);
sdio_writel(func, WHLPCR_FW_OWN_REQ_CLR, MCR_WHLPCR, NULL);
status & WHLPCR_IS_DRIVER_OWN, 2000, 1000000);
if (ret < 0) {
dev_err(dev->mt76.dev, "Cannot get ownership from device");
- set_bit(MT76_STATE_PM, &mphy->state);
- sdio_release_host(func);
+ } else {
+ clear_bit(MT76_STATE_PM, &mphy->state);
- return ret;
+ pm->stats.last_wake_event = jiffies;
+ pm->stats.doze_time += pm->stats.last_wake_event -
+ pm->stats.last_doze_event;
}
-
sdio_release_host(func);
-out:
- dev->pm.last_activity = jiffies;
+ return ret;
+}
- return 0;
+static int mt7663s_mcu_drv_pmctrl(struct mt7615_dev *dev)
+{
+ struct mt76_phy *mphy = &dev->mt76.phy;
+ int ret = 0;
+
+ mutex_lock(&dev->pm.mutex);
+
+ if (test_bit(MT76_STATE_PM, &mphy->state))
+ ret = __mt7663s_mcu_drv_pmctrl(dev);
+
+ mutex_unlock(&dev->pm.mutex);
+
+ return ret;
}
static int mt7663s_mcu_fw_pmctrl(struct mt7615_dev *dev)
{
struct sdio_func *func = dev->mt76.sdio.func;
struct mt76_phy *mphy = &dev->mt76.phy;
+ struct mt76_connac_pm *pm = &dev->pm;
+ int ret = 0;
u32 status;
- int ret;
- if (test_and_set_bit(MT76_STATE_PM, &mphy->state))
- return 0;
+ mutex_lock(&pm->mutex);
+
+ if (mt76_connac_skip_fw_pmctrl(mphy, pm))
+ goto out;
sdio_claim_host(func);
if (ret < 0) {
dev_err(dev->mt76.dev, "Cannot set ownership to device");
clear_bit(MT76_STATE_PM, &mphy->state);
+ } else {
+ pm->stats.last_doze_event = jiffies;
+ pm->stats.awake_time += pm->stats.last_doze_event -
+ pm->stats.last_wake_event;
}
sdio_release_host(func);
+out:
+ mutex_unlock(&pm->mutex);
return ret;
}
struct mt7615_mcu_ops *mcu_ops;
int ret;
- ret = mt7663s_mcu_drv_pmctrl(dev);
+ ret = __mt7663s_mcu_drv_pmctrl(dev);
if (ret)
return ret;
{
struct mt76_sdio *sdio = container_of(w, struct mt76_sdio,
txrx_worker);
- struct mt76_dev *dev = container_of(sdio, struct mt76_dev, sdio);
+ struct mt76_dev *mdev = container_of(sdio, struct mt76_dev, sdio);
+ struct mt7615_dev *dev = container_of(mdev, struct mt7615_dev, mt76);
int i, nframes, ret;
+ if (!mt76_connac_pm_ref(&dev->mphy, &dev->pm)) {
+ queue_work(mdev->wq, &dev->pm.wake_work);
+ return;
+ }
+
/* disable interrupt */
sdio_claim_host(sdio->func);
sdio_writel(sdio->func, WHLPCR_INT_EN_CLR, MCR_WHLPCR, NULL);
/* tx */
for (i = 0; i <= MT_TXQ_PSD; i++) {
- ret = mt7663s_tx_run_queue(dev, dev->phy.q_tx[i]);
+ ret = mt7663s_tx_run_queue(mdev, mdev->phy.q_tx[i]);
if (ret > 0)
nframes += ret;
}
- ret = mt7663s_tx_run_queue(dev, dev->q_mcu[MT_MCUQ_WM]);
+ ret = mt7663s_tx_run_queue(mdev, mdev->q_mcu[MT_MCUQ_WM]);
if (ret > 0)
nframes += ret;
/* rx */
- ret = mt7663s_rx_handler(dev);
+ ret = mt7663s_rx_handler(mdev);
if (ret > 0)
nframes += ret;
} while (nframes > 0);
/* enable interrupt */
sdio_writel(sdio->func, WHLPCR_INT_EN_SET, MCR_WHLPCR, NULL);
sdio_release_host(sdio->func);
+
+ mt76_connac_pm_unref(&dev->mphy, &dev->pm);
}
void mt7663s_sdio_irq(struct sdio_func *func)
static int mt7622_wmac_probe(struct platform_device *pdev)
{
- struct resource *res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
void __iomem *mem_base;
int irq;
if (irq < 0)
return irq;
- mem_base = devm_ioremap_resource(&pdev->dev, res);
+ mem_base = devm_platform_get_and_ioremap_resource(pdev, 0, NULL);
if (IS_ERR(mem_base))
return PTR_ERR(mem_base);
dev->mt76.mcu_ops = &mt7663u_mcu_ops,
- /* usb does not support runtime-pm */
- clear_bit(MT76_STATE_PM, &dev->mphy.state);
mt76_set(dev, MT_UDMA_TX_QSEL, MT_FW_DL_EN);
-
if (test_and_clear_bit(MT76_STATE_POWER_OFF, &dev->mphy.state)) {
mt7615_mcu_restart(&dev->mt76);
if (!mt76_poll_msec(dev, MT_CONN_ON_MISC,
idx = idx > HW_BSSID_MAX ? HW_BSSID_0 : idx;
addr = idx > 1 ? MT_LPON_TCR2(idx): MT_LPON_TCR0(idx);
- mt76_set(dev, addr, MT_LPON_TCR_MODE); /* TSF read */
+ mt76_rmw(dev, addr, MT_LPON_TCR_MODE, MT_LPON_TCR_READ); /* TSF read */
val = mt76_rr(dev, MT_LPON_UTTR0);
sta->rate_set_tsf = (val & ~BIT(0)) | rate->rateset;
struct ieee80211_sta *sta,
struct mt76_tx_info *tx_info)
{
- struct mt7615_sta *msta = container_of(wcid, struct mt7615_sta, wcid);
struct mt7615_dev *dev = container_of(mdev, struct mt7615_dev, mt76);
struct sk_buff *skb = tx_info->skb;
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
+ struct mt7615_sta *msta;
int pad;
+ msta = wcid ? container_of(wcid, struct mt7615_sta, wcid) : NULL;
if ((info->flags & IEEE80211_TX_CTL_RATE_CTRL_PROBE) &&
- !msta->rate_probe) {
+ msta && !msta->rate_probe) {
/* request to configure sampling rate */
spin_lock_bh(&dev->mt76.lock);
mt7615_mac_set_rates(&dev->phy, msta, &info->control.rates[0],
hw->max_tx_fragments = 1;
}
- err = mt76_register_device(&dev->mt76, true, mt7615_rates,
- ARRAY_SIZE(mt7615_rates));
+ err = mt76_register_device(&dev->mt76, true, mt76_rates,
+ ARRAY_SIZE(mt76_rates));
if (err < 0)
return err;
#include "mt76.h"
#define MT76_CONNAC_SCAN_IE_LEN 600
-#define MT76_CONNAC_MAX_SCHED_SCAN_INTERVAL 10
+#define MT76_CONNAC_MAX_NUM_SCHED_SCAN_INTERVAL 10
+#define MT76_CONNAC_MAX_TIME_SCHED_SCAN_INTERVAL U16_MAX
#define MT76_CONNAC_MAX_SCHED_SCAN_SSID 10
#define MT76_CONNAC_MAX_SCAN_MATCH 16
#define MT76_CONNAC_COREDUMP_TIMEOUT (HZ / 20)
-#define MT76_CONNAC_COREDUMP_SZ (128 * 1024)
+#define MT76_CONNAC_COREDUMP_SZ (1300 * 1024)
enum {
CMD_CBW_20MHZ = IEEE80211_STA_RX_BW_20,
struct mt76_connac_pm {
bool enable;
+ bool ds_enable;
+ bool suspended;
spinlock_t txq_lock;
struct {
}
static inline void
-mt76_connac_pm_unref(struct mt76_connac_pm *pm)
+mt76_connac_pm_unref(struct mt76_phy *phy, struct mt76_connac_pm *pm)
{
spin_lock_bh(&pm->wake.lock);
- pm->wake.count--;
+
pm->last_activity = jiffies;
+ if (--pm->wake.count == 0 &&
+ test_bit(MT76_STATE_MCU_RUNNING, &phy->state))
+ mt76_connac_power_save_sched(phy, pm);
+
spin_unlock_bh(&pm->wake.lock);
}
static inline bool
mt76_connac_skip_fw_pmctrl(struct mt76_phy *phy, struct mt76_connac_pm *pm)
{
+ struct mt76_dev *dev = phy->dev;
bool ret;
+ if (dev->token_count)
+ return true;
+
spin_lock_bh(&pm->wake.lock);
ret = pm->wake.count || test_and_set_bit(MT76_STATE_PM, &phy->state);
spin_unlock_bh(&pm->wake.lock);
if (!pm->enable)
return 0;
- if (!mt76_is_mmio(dev))
+ if (mt76_is_usb(dev))
return 0;
cancel_delayed_work_sync(&pm->ps_work);
if (!test_bit(MT76_STATE_PM, &phy->state))
return 0;
+ if (pm->suspended)
+ return 0;
+
queue_work(dev->wq, &pm->wake_work);
if (!wait_event_timeout(pm->wait,
!test_bit(MT76_STATE_PM, &phy->state),
{
struct mt76_dev *dev = phy->dev;
- if (!mt76_is_mmio(dev))
+ if (mt76_is_usb(dev))
return;
if (!pm->enable)
return;
+ if (pm->suspended)
+ return;
+
pm->last_activity = jiffies;
if (!test_bit(MT76_STATE_PM, &phy->state)) {
void mt76_connac_mcu_sta_basic_tlv(struct sk_buff *skb,
struct ieee80211_vif *vif,
struct ieee80211_sta *sta,
- bool enable)
+ bool enable, bool newly)
{
struct sta_rec_basic *basic;
struct tlv *tlv;
basic->extra_info = cpu_to_le16(EXTRA_INFO_VER);
if (enable) {
- basic->extra_info |= cpu_to_le16(EXTRA_INFO_NEW);
+ if (newly)
+ basic->extra_info |= cpu_to_le16(EXTRA_INFO_NEW);
basic->conn_state = CONN_STATE_PORT_SECURE;
} else {
basic->conn_state = CONN_STATE_DISCONNECT;
}
void mt76_connac_mcu_wtbl_hdr_trans_tlv(struct sk_buff *skb,
+ struct ieee80211_vif *vif,
struct mt76_wcid *wcid,
void *sta_wtbl, void *wtbl_tlv)
{
wtbl_tlv, sta_wtbl);
htr = (struct wtbl_hdr_trans *)tlv;
htr->no_rx_trans = !test_bit(MT_WCID_FLAG_HDR_TRANS, &wcid->flags);
+
+ if (vif->type == NL80211_IFTYPE_STATION)
+ htr->to_ds = true;
+ else
+ htr->from_ds = true;
+
+ if (test_bit(MT_WCID_FLAG_4ADDR, &wcid->flags)) {
+ htr->to_ds = true;
+ htr->from_ds = true;
+ }
}
EXPORT_SYMBOL_GPL(mt76_connac_mcu_wtbl_hdr_trans_tlv);
+int mt76_connac_mcu_sta_update_hdr_trans(struct mt76_dev *dev,
+ struct ieee80211_vif *vif,
+ struct mt76_wcid *wcid, int cmd)
+{
+ struct mt76_vif *mvif = (struct mt76_vif *)vif->drv_priv;
+ struct wtbl_req_hdr *wtbl_hdr;
+ struct tlv *sta_wtbl;
+ struct sk_buff *skb;
+
+ skb = mt76_connac_mcu_alloc_sta_req(dev, mvif, wcid);
+ if (IS_ERR(skb))
+ return PTR_ERR(skb);
+
+ sta_wtbl = mt76_connac_mcu_add_tlv(skb, STA_REC_WTBL,
+ sizeof(struct tlv));
+
+ wtbl_hdr = mt76_connac_mcu_alloc_wtbl_req(dev, wcid, WTBL_SET,
+ sta_wtbl, &skb);
+ if (IS_ERR(wtbl_hdr))
+ return PTR_ERR(wtbl_hdr);
+
+ mt76_connac_mcu_wtbl_hdr_trans_tlv(skb, vif, wcid, sta_wtbl, wtbl_hdr);
+
+ return mt76_mcu_skb_send_msg(dev, skb, cmd, true);
+}
+EXPORT_SYMBOL_GPL(mt76_connac_mcu_sta_update_hdr_trans);
+
void mt76_connac_mcu_wtbl_generic_tlv(struct mt76_dev *dev,
struct sk_buff *skb,
struct ieee80211_vif *vif,
void mt76_connac_mcu_sta_tlv(struct mt76_phy *mphy, struct sk_buff *skb,
struct ieee80211_sta *sta,
struct ieee80211_vif *vif,
- u8 rcpi)
+ u8 rcpi, u8 sta_state)
{
struct cfg80211_chan_def *chandef = &mphy->chandef;
enum nl80211_band band = chandef->chan->band;
phy->phy_type = mt76_connac_get_phy_mode_v2(mphy, vif, band, sta);
phy->basic_rate = cpu_to_le16((u16)vif->bss_conf.basic_rates);
phy->rcpi = rcpi;
+ phy->ampdu = FIELD_PREP(IEEE80211_HT_AMPDU_PARM_FACTOR,
+ sta->ht_cap.ampdu_factor) |
+ FIELD_PREP(IEEE80211_HT_AMPDU_PARM_DENSITY,
+ sta->ht_cap.ampdu_density);
tlv = mt76_connac_mcu_add_tlv(skb, STA_REC_RA, sizeof(*ra_info));
ra_info = (struct sta_rec_ra_info *)tlv;
tlv = mt76_connac_mcu_add_tlv(skb, STA_REC_STATE, sizeof(*state));
state = (struct sta_rec_state *)tlv;
- state->state = 2;
+ state->state = sta_state;
if (sta->vht_cap.vht_supported) {
state->vht_opmode = sta->bandwidth;
}
EXPORT_SYMBOL_GPL(mt76_connac_mcu_wtbl_ht_tlv);
-int mt76_connac_mcu_add_sta_cmd(struct mt76_phy *phy,
- struct mt76_sta_cmd_info *info)
+int mt76_connac_mcu_sta_cmd(struct mt76_phy *phy,
+ struct mt76_sta_cmd_info *info)
{
struct mt76_vif *mvif = (struct mt76_vif *)info->vif->drv_priv;
struct mt76_dev *dev = phy->dev;
if (IS_ERR(skb))
return PTR_ERR(skb);
- mt76_connac_mcu_sta_basic_tlv(skb, info->vif, info->sta, info->enable);
- if (info->enable && info->sta)
- mt76_connac_mcu_sta_tlv(phy, skb, info->sta, info->vif,
- info->rcpi);
+ if (info->sta || !info->offload_fw)
+ mt76_connac_mcu_sta_basic_tlv(skb, info->vif, info->sta,
+ info->enable, info->newly);
+ if (info->sta && info->enable)
+ mt76_connac_mcu_sta_tlv(phy, skb, info->sta,
+ info->vif, info->rcpi,
+ info->state);
sta_wtbl = mt76_connac_mcu_add_tlv(skb, STA_REC_WTBL,
sizeof(struct tlv));
mt76_connac_mcu_wtbl_generic_tlv(dev, skb, info->vif,
info->sta, sta_wtbl,
wtbl_hdr);
+ mt76_connac_mcu_wtbl_hdr_trans_tlv(skb, info->vif, info->wcid,
+ sta_wtbl, wtbl_hdr);
if (info->sta)
mt76_connac_mcu_wtbl_ht_tlv(dev, skb, info->sta,
sta_wtbl, wtbl_hdr);
return mt76_mcu_skb_send_msg(dev, skb, info->cmd, true);
}
-EXPORT_SYMBOL_GPL(mt76_connac_mcu_add_sta_cmd);
+EXPORT_SYMBOL_GPL(mt76_connac_mcu_sta_cmd);
void mt76_connac_mcu_wtbl_ba_tlv(struct mt76_dev *dev, struct sk_buff *skb,
struct ieee80211_ampdu_params *params,
ba->rst_ba_sb = 1;
}
- if (is_mt7921(dev))
+ if (is_mt7921(dev)) {
+ ba->ba_winsize = enable ? cpu_to_le16(params->buf_size) : 0;
return;
+ }
if (enable && tx) {
u8 ba_range[] = { 4, 8, 12, 24, 36, 48, 54, 64 };
u8 pad[3];
} __packed hdr;
struct bss_info_uni_he he;
+ struct bss_info_uni_bss_color bss_color;
} he_req = {
.hdr = {
.bss_idx = mvif->idx,
.tag = cpu_to_le16(UNI_BSS_INFO_HE_BASIC),
.len = cpu_to_le16(sizeof(struct bss_info_uni_he)),
},
+ .bss_color = {
+ .tag = cpu_to_le16(UNI_BSS_INFO_BSS_COLOR),
+ .len = cpu_to_le16(sizeof(struct bss_info_uni_bss_color)),
+ .enable = 0,
+ .bss_color = 0,
+ },
};
+ if (enable) {
+ he_req.bss_color.enable =
+ vif->bss_conf.he_bss_color.enabled;
+ he_req.bss_color.bss_color =
+ vif->bss_conf.he_bss_color.color;
+ }
+
mt76_connac_mcu_uni_bss_he_tlv(phy, vif,
(struct tlv *)&he_req.he);
err = mt76_mcu_send_msg(mdev, MCU_UNI_CMD_BSS_INFO_UPDATE,
req->version = 1;
req->seq_num = mvif->scan_seq_num | ext_phy << 7;
- if (is_mt7663(phy->dev) &&
- (sreq->flags & NL80211_SCAN_FLAG_RANDOM_ADDR)) {
- get_random_mask_addr(req->mt7663.random_mac, sreq->mac_addr,
- sreq->mac_addr_mask);
+ if (sreq->flags & NL80211_SCAN_FLAG_RANDOM_ADDR) {
+ u8 *addr = is_mt7663(phy->dev) ? req->mt7663.random_mac
+ : req->mt7921.random_mac;
+
req->scan_func = 1;
- } else if (is_mt7921(phy->dev)) {
- req->mt7921.bss_idx = mvif->idx;
+ get_random_mask_addr(addr, sreq->mac_addr,
+ sreq->mac_addr_mask);
}
+ if (is_mt7921(phy->dev))
+ req->mt7921.bss_idx = mvif->idx;
req->ssids_num = sreq->n_ssids;
for (i = 0; i < req->ssids_num; i++) {
}
EXPORT_SYMBOL_GPL(mt76_connac_mcu_set_deep_sleep);
+int mt76_connac_sta_state_dp(struct mt76_dev *dev,
+ enum ieee80211_sta_state old_state,
+ enum ieee80211_sta_state new_state)
+{
+ if ((old_state == IEEE80211_STA_ASSOC &&
+ new_state == IEEE80211_STA_AUTHORIZED) ||
+ (old_state == IEEE80211_STA_NONE &&
+ new_state == IEEE80211_STA_NOTEXIST))
+ mt76_connac_mcu_set_deep_sleep(dev, true);
+
+ if ((old_state == IEEE80211_STA_NOTEXIST &&
+ new_state == IEEE80211_STA_NONE) ||
+ (old_state == IEEE80211_STA_AUTHORIZED &&
+ new_state == IEEE80211_STA_ASSOC))
+ mt76_connac_mcu_set_deep_sleep(dev, false);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mt76_connac_sta_state_dp);
+
void mt76_connac_mcu_coredump_event(struct mt76_dev *dev, struct sk_buff *skb,
struct mt76_connac_coredump *coredump)
{
}
EXPORT_SYMBOL_GPL(mt76_connac_mcu_coredump_event);
+int mt76_connac_mcu_get_nic_capability(struct mt76_phy *phy)
+{
+ struct mt76_connac_cap_hdr {
+ __le16 n_element;
+ u8 rsv[2];
+ } __packed * hdr;
+ struct sk_buff *skb;
+ int ret, i;
+
+ ret = mt76_mcu_send_and_get_msg(phy->dev, MCU_CMD_GET_NIC_CAPAB, NULL,
+ 0, true, &skb);
+ if (ret)
+ return ret;
+
+ hdr = (struct mt76_connac_cap_hdr *)skb->data;
+ if (skb->len < sizeof(*hdr)) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ skb_pull(skb, sizeof(*hdr));
+
+ for (i = 0; i < le16_to_cpu(hdr->n_element); i++) {
+ struct tlv_hdr {
+ __le32 type;
+ __le32 len;
+ } __packed * tlv = (struct tlv_hdr *)skb->data;
+ int len;
+
+ if (skb->len < sizeof(*tlv))
+ break;
+
+ skb_pull(skb, sizeof(*tlv));
+
+ len = le32_to_cpu(tlv->len);
+ if (skb->len < len)
+ break;
+
+ switch (le32_to_cpu(tlv->type)) {
+ case MT_NIC_CAP_6G:
+ phy->cap.has_6ghz = skb->data[0];
+ break;
+ default:
+ break;
+ }
+ skb_pull(skb, len);
+ }
+out:
+ dev_kfree_skb(skb);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(mt76_connac_mcu_get_nic_capability);
+
static void
mt76_connac_mcu_build_sku(struct mt76_dev *dev, s8 *sku,
struct mt76_power_limits *limits,
142, 144, 149, 151, 153, 155, 157,
159, 161, 165
};
+ int i, n_chan, batch_size, idx = 0, tx_power, last_ch;
struct mt76_connac_sku_tlv sku_tlbv;
- int i, n_chan, batch_size, idx = 0;
struct mt76_power_limits limits;
const u8 *ch_list;
sku_len = is_mt7921(dev) ? sizeof(sku_tlbv) : sizeof(sku_tlbv) - 92;
+ tx_power = 2 * phy->hw->conf.power_level;
+ if (!tx_power)
+ tx_power = 127;
if (band == NL80211_BAND_2GHZ) {
n_chan = ARRAY_SIZE(chan_list_2ghz);
}
batch_size = DIV_ROUND_UP(n_chan, batch_len);
+ if (!phy->cap.has_5ghz)
+ last_ch = chan_list_2ghz[n_chan - 1];
+ else
+ last_ch = chan_list_5ghz[n_chan - 1];
+
for (i = 0; i < batch_size; i++) {
- bool last_msg = i == batch_size - 1;
- int num_ch = last_msg ? n_chan % batch_len : batch_len;
struct mt76_connac_tx_power_limit_tlv tx_power_tlv = {
.band = band == NL80211_BAND_2GHZ ? 1 : 2,
- .n_chan = num_ch,
- .last_msg = last_msg,
};
+ int j, err, msg_len, num_ch;
struct sk_buff *skb;
- int j, err, msg_len;
+ num_ch = i == batch_size - 1 ? n_chan % batch_len : batch_len;
msg_len = sizeof(tx_power_tlv) + num_ch * sizeof(sku_tlbv);
skb = mt76_mcu_msg_alloc(dev, NULL, msg_len);
if (!skb)
return -ENOMEM;
+ skb_reserve(skb, sizeof(tx_power_tlv));
+
BUILD_BUG_ON(sizeof(dev->alpha2) > sizeof(tx_power_tlv.alpha2));
memcpy(tx_power_tlv.alpha2, dev->alpha2, sizeof(dev->alpha2));
+ tx_power_tlv.n_chan = num_ch;
- skb_put_data(skb, &tx_power_tlv, sizeof(tx_power_tlv));
for (j = 0; j < num_ch; j++, idx++) {
struct ieee80211_channel chan = {
.hw_value = ch_list[idx],
.band = band,
};
- mt76_get_rate_power_limits(phy, &chan, &limits, 127);
+ mt76_get_rate_power_limits(phy, &chan, &limits,
+ tx_power);
+ tx_power_tlv.last_msg = ch_list[idx] == last_ch;
sku_tlbv.channel = ch_list[idx];
+
mt76_connac_mcu_build_sku(dev, sku_tlbv.pwr_limit,
&limits, band);
skb_put_data(skb, &sku_tlbv, sku_len);
}
+ __skb_push(skb, sizeof(tx_power_tlv));
+ memcpy(skb->data, &tx_power_tlv, sizeof(tx_power_tlv));
err = mt76_mcu_skb_send_msg(dev, skb,
MCU_CMD_SET_RATE_TX_POWER, false);
{
int err;
- err = mt76_connac_mcu_rate_txpower_band(phy, NL80211_BAND_2GHZ);
- if (err < 0)
- return err;
+ if (phy->cap.has_2ghz) {
+ err = mt76_connac_mcu_rate_txpower_band(phy,
+ NL80211_BAND_2GHZ);
+ if (err < 0)
+ return err;
+ }
+ if (phy->cap.has_5ghz) {
+ err = mt76_connac_mcu_rate_txpower_band(phy,
+ NL80211_BAND_5GHZ);
+ if (err < 0)
+ return err;
+ }
- return mt76_connac_mcu_rate_txpower_band(phy, NL80211_BAND_5GHZ);
+ return 0;
}
EXPORT_SYMBOL_GPL(mt76_connac_mcu_set_rate_txpower);
ptlv->index = index;
memcpy(ptlv->pattern, pattern->pattern, pattern->pattern_len);
- memcpy(ptlv->mask, pattern->mask, pattern->pattern_len / 8);
+ memcpy(ptlv->mask, pattern->mask, DIV_ROUND_UP(pattern->pattern_len, 8));
return mt76_mcu_skb_send_msg(dev, skb, MCU_UNI_CMD_SUSPEND, true);
}
};
if (wowlan->magic_pkt)
- req.wow_ctrl_tlv.trigger |= BIT(0);
+ req.wow_ctrl_tlv.trigger |= UNI_WOW_DETECT_TYPE_MAGIC;
if (wowlan->disconnect)
- req.wow_ctrl_tlv.trigger |= BIT(2);
+ req.wow_ctrl_tlv.trigger |= (UNI_WOW_DETECT_TYPE_DISCONNECT |
+ UNI_WOW_DETECT_TYPE_BCN_LOST);
if (wowlan->nd_config) {
mt76_connac_mcu_sched_scan_req(phy, vif, wowlan->nd_config);
- req.wow_ctrl_tlv.trigger |= BIT(5);
+ req.wow_ctrl_tlv.trigger |= UNI_WOW_DETECT_TYPE_SCH_SCAN_HIT;
mt76_connac_mcu_sched_scan_enable(phy, vif, suspend);
}
+ if (wowlan->n_patterns)
+ req.wow_ctrl_tlv.trigger |= UNI_WOW_DETECT_TYPE_BITMAP;
if (mt76_is_mmio(dev))
req.wow_ctrl_tlv.wakeup_hif = WOW_PCIE;
MCU_CMD_SET_RATE_TX_POWER = MCU_CE_PREFIX | 0x5d,
MCU_CMD_SCHED_SCAN_ENABLE = MCU_CE_PREFIX | 0x61,
MCU_CMD_SCHED_SCAN_REQ = MCU_CE_PREFIX | 0x62,
+ MCU_CMD_GET_NIC_CAPAB = MCU_CE_PREFIX | 0x8a,
MCU_CMD_REG_WRITE = MCU_CE_PREFIX | 0xc0,
MCU_CMD_REG_READ = MCU_CE_PREFIX | MCU_QUERY_MASK | 0xc0,
MCU_CMD_CHIP_CONFIG = MCU_CE_PREFIX | 0xca,
enum {
UNI_BSS_INFO_BASIC = 0,
UNI_BSS_INFO_RLM = 2,
+ UNI_BSS_INFO_BSS_COLOR = 4,
UNI_BSS_INFO_HE_BASIC = 5,
UNI_BSS_INFO_BCN_CONTENT = 7,
UNI_BSS_INFO_QBSS = 15,
};
enum {
+ MT_NIC_CAP_TX_RESOURCE,
+ MT_NIC_CAP_TX_EFUSE_ADDR,
+ MT_NIC_CAP_COEX,
+ MT_NIC_CAP_SINGLE_SKU,
+ MT_NIC_CAP_CSUM_OFFLOAD,
+ MT_NIC_CAP_HW_VER,
+ MT_NIC_CAP_SW_VER,
+ MT_NIC_CAP_MAC_ADDR,
+ MT_NIC_CAP_PHY,
+ MT_NIC_CAP_MAC,
+ MT_NIC_CAP_FRAME_BUF,
+ MT_NIC_CAP_BEAM_FORM,
+ MT_NIC_CAP_LOCATION,
+ MT_NIC_CAP_MUMIMO,
+ MT_NIC_CAP_BUFFER_MODE_INFO,
+ MT_NIC_CAP_HW_ADIE_VERSION = 0x14,
+ MT_NIC_CAP_ANTSWP = 0x16,
+ MT_NIC_CAP_WFDMA_REALLOC,
+ MT_NIC_CAP_6G,
+};
+
+#define UNI_WOW_DETECT_TYPE_MAGIC BIT(0)
+#define UNI_WOW_DETECT_TYPE_ANY BIT(1)
+#define UNI_WOW_DETECT_TYPE_DISCONNECT BIT(2)
+#define UNI_WOW_DETECT_TYPE_GTK_REKEY_FAIL BIT(3)
+#define UNI_WOW_DETECT_TYPE_BCN_LOST BIT(4)
+#define UNI_WOW_DETECT_TYPE_SCH_SCAN_HIT BIT(5)
+#define UNI_WOW_DETECT_TYPE_BITMAP BIT(6)
+
+enum {
UNI_SUSPEND_MODE_SETTING,
UNI_SUSPEND_WOW_CTRL,
UNI_SUSPEND_WOW_GPIO_PARAM,
u8 intervals_num;
u8 scan_func; /* MT7663: BIT(0) eable random mac address */
struct mt76_connac_mcu_scan_channel channels[64];
- __le16 intervals[MT76_CONNAC_MAX_SCHED_SCAN_INTERVAL];
+ __le16 intervals[MT76_CONNAC_MAX_NUM_SCHED_SCAN_INTERVAL];
union {
struct {
u8 random_mac[ETH_ALEN];
} mt7663;
struct {
u8 bss_idx;
- u8 pad2[63];
+ u8 pad2[19];
+ u8 random_mac[ETH_ALEN];
+ u8 pad3[38];
} mt7921;
};
} __packed;
__le16 pad;
} __packed;
+struct bss_info_uni_bss_color {
+ __le16 tag;
+ __le16 len;
+ u8 enable;
+ u8 bss_color;
+ u8 rsv[2];
+} __packed;
+
struct bss_info_uni_he {
__le16 tag;
__le16 len;
u8 pad[5];
} __packed;
+enum mt76_sta_info_state {
+ MT76_STA_INFO_STATE_NONE,
+ MT76_STA_INFO_STATE_AUTH,
+ MT76_STA_INFO_STATE_ASSOC
+};
+
struct mt76_sta_cmd_info {
struct ieee80211_sta *sta;
struct mt76_wcid *wcid;
struct ieee80211_vif *vif;
+ bool offload_fw;
bool enable;
+ bool newly;
int cmd;
u8 rcpi;
+ u8 state;
};
#define MT_SKU_POWER_LIMIT 161
int mt76_connac_mcu_set_vif_ps(struct mt76_dev *dev, struct ieee80211_vif *vif);
void mt76_connac_mcu_sta_basic_tlv(struct sk_buff *skb,
struct ieee80211_vif *vif,
- struct ieee80211_sta *sta, bool enable);
+ struct ieee80211_sta *sta, bool enable,
+ bool newly);
void mt76_connac_mcu_wtbl_generic_tlv(struct mt76_dev *dev, struct sk_buff *skb,
struct ieee80211_vif *vif,
struct ieee80211_sta *sta, void *sta_wtbl,
void *wtbl_tlv);
void mt76_connac_mcu_wtbl_hdr_trans_tlv(struct sk_buff *skb,
+ struct ieee80211_vif *vif,
struct mt76_wcid *wcid,
void *sta_wtbl, void *wtbl_tlv);
+int mt76_connac_mcu_sta_update_hdr_trans(struct mt76_dev *dev,
+ struct ieee80211_vif *vif,
+ struct mt76_wcid *wcid, int cmd);
void mt76_connac_mcu_sta_tlv(struct mt76_phy *mphy, struct sk_buff *skb,
struct ieee80211_sta *sta,
struct ieee80211_vif *vif,
- u8 rcpi);
+ u8 rcpi, u8 state);
void mt76_connac_mcu_wtbl_ht_tlv(struct mt76_dev *dev, struct sk_buff *skb,
struct ieee80211_sta *sta, void *sta_wtbl,
void *wtbl_tlv);
struct ieee80211_vif *vif,
struct mt76_wcid *wcid,
bool enable);
-int mt76_connac_mcu_add_sta_cmd(struct mt76_phy *phy,
- struct mt76_sta_cmd_info *info);
+int mt76_connac_mcu_sta_cmd(struct mt76_phy *phy,
+ struct mt76_sta_cmd_info *info);
void mt76_connac_mcu_beacon_loss_iter(void *priv, u8 *mac,
struct ieee80211_vif *vif);
int mt76_connac_mcu_set_rts_thresh(struct mt76_dev *dev, u32 val, u8 band);
int mt76_connac_mcu_start_patch(struct mt76_dev *dev);
int mt76_connac_mcu_patch_sem_ctrl(struct mt76_dev *dev, bool get);
int mt76_connac_mcu_start_firmware(struct mt76_dev *dev, u32 addr, u32 option);
+int mt76_connac_mcu_get_nic_capability(struct mt76_phy *phy);
int mt76_connac_mcu_hw_scan(struct mt76_phy *phy, struct ieee80211_vif *vif,
struct ieee80211_scan_request *scan_req);
int mt76_connac_mcu_set_hif_suspend(struct mt76_dev *dev, bool suspend);
void mt76_connac_mcu_set_suspend_iter(void *priv, u8 *mac,
struct ieee80211_vif *vif);
+int mt76_connac_sta_state_dp(struct mt76_dev *dev,
+ enum ieee80211_sta_state old_state,
+ enum ieee80211_sta_state new_state);
int mt76_connac_mcu_chip_config(struct mt76_dev *dev);
int mt76_connac_mcu_set_deep_sleep(struct mt76_dev *dev, bool enable);
void mt76_connac_mcu_coredump_event(struct mt76_dev *dev, struct sk_buff *skb,
nic_conf1 &= 0xff00;
if (nic_conf1 & MT_EE_NIC_CONF_1_HW_RF_CTRL)
- dev_err(dev->mt76.dev,
+ dev_dbg(dev->mt76.dev,
"driver does not support HW RF ctrl\n");
if (!mt76x02_field_valid(nic_conf0 >> 8))
.reconfig_complete = mt76x02_reconfig_complete,
};
-static int mt76x0e_register_device(struct mt76x02_dev *dev)
+static int mt76x0e_init_hardware(struct mt76x02_dev *dev, bool resume)
{
int err;
if (err < 0)
return err;
- err = mt76x02_dma_init(dev);
- if (err < 0)
- return err;
+ if (!resume) {
+ err = mt76x02_dma_init(dev);
+ if (err < 0)
+ return err;
+ }
err = mt76x0_init_hardware(dev);
if (err < 0)
mt76_clear(dev, 0x110, BIT(9));
mt76_set(dev, MT_MAX_LEN_CFG, BIT(13));
+ return 0;
+}
+
+static int mt76x0e_register_device(struct mt76x02_dev *dev)
+{
+ int err;
+
+ err = mt76x0e_init_hardware(dev, false);
+ if (err < 0)
+ return err;
+
err = mt76x0_register_device(dev);
if (err < 0)
return err;
if (ret)
return ret;
+ mt76_pci_disable_aspm(pdev);
+
mdev = mt76_alloc_device(&pdev->dev, sizeof(*dev), &mt76x0e_ops,
&drv_ops);
if (!mdev)
mt76_free_device(mdev);
}
+#ifdef CONFIG_PM
+static int mt76x0e_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+ struct mt76_dev *mdev = pci_get_drvdata(pdev);
+ struct mt76x02_dev *dev = container_of(mdev, struct mt76x02_dev, mt76);
+ int i;
+
+ mt76_worker_disable(&mdev->tx_worker);
+ for (i = 0; i < ARRAY_SIZE(mdev->phy.q_tx); i++)
+ mt76_queue_tx_cleanup(dev, mdev->phy.q_tx[i], true);
+ for (i = 0; i < ARRAY_SIZE(mdev->q_mcu); i++)
+ mt76_queue_tx_cleanup(dev, mdev->q_mcu[i], true);
+ napi_disable(&mdev->tx_napi);
+
+ mt76_for_each_q_rx(mdev, i)
+ napi_disable(&mdev->napi[i]);
+
+ mt76x02_dma_disable(dev);
+ mt76x02_mcu_cleanup(dev);
+ mt76x0_chip_onoff(dev, false, false);
+
+ pci_enable_wake(pdev, pci_choose_state(pdev, state), true);
+ pci_save_state(pdev);
+
+ return pci_set_power_state(pdev, pci_choose_state(pdev, state));
+}
+
+static int mt76x0e_resume(struct pci_dev *pdev)
+{
+ struct mt76_dev *mdev = pci_get_drvdata(pdev);
+ struct mt76x02_dev *dev = container_of(mdev, struct mt76x02_dev, mt76);
+ int err, i;
+
+ err = pci_set_power_state(pdev, PCI_D0);
+ if (err)
+ return err;
+
+ pci_restore_state(pdev);
+
+ mt76_worker_enable(&mdev->tx_worker);
+
+ mt76_for_each_q_rx(mdev, i) {
+ mt76_queue_rx_reset(dev, i);
+ napi_enable(&mdev->napi[i]);
+ napi_schedule(&mdev->napi[i]);
+ }
+
+ napi_enable(&mdev->tx_napi);
+ napi_schedule(&mdev->tx_napi);
+
+ return mt76x0e_init_hardware(dev, true);
+}
+#endif /* CONFIG_PM */
+
static const struct pci_device_id mt76x0e_device_table[] = {
{ PCI_DEVICE(PCI_VENDOR_ID_MEDIATEK, 0x7610) },
{ PCI_DEVICE(PCI_VENDOR_ID_MEDIATEK, 0x7630) },
.id_table = mt76x0e_device_table,
.probe = mt76x0e_probe,
.remove = mt76x0e_remove,
+#ifdef CONFIG_PM
+ .suspend = mt76x0e_suspend,
+ .resume = mt76x0e_resume,
+#endif /* CONFIG_PM */
};
module_pci_driver(mt76x0e_driver);
{
memset(key_data, 0, 32);
if (!key)
- return MT_CIPHER_NONE;
+ return MT76X02_CIPHER_NONE;
if (key->keylen > 32)
- return MT_CIPHER_NONE;
+ return MT76X02_CIPHER_NONE;
memcpy(key_data, key->key, key->keylen);
switch (key->cipher) {
case WLAN_CIPHER_SUITE_WEP40:
- return MT_CIPHER_WEP40;
+ return MT76X02_CIPHER_WEP40;
case WLAN_CIPHER_SUITE_WEP104:
- return MT_CIPHER_WEP104;
+ return MT76X02_CIPHER_WEP104;
case WLAN_CIPHER_SUITE_TKIP:
- return MT_CIPHER_TKIP;
+ return MT76X02_CIPHER_TKIP;
case WLAN_CIPHER_SUITE_CCMP:
- return MT_CIPHER_AES_CCMP;
+ return MT76X02_CIPHER_AES_CCMP;
default:
- return MT_CIPHER_NONE;
+ return MT76X02_CIPHER_NONE;
}
}
u32 val;
cipher = mt76x02_mac_get_key_info(key, key_data);
- if (cipher == MT_CIPHER_NONE && key)
+ if (cipher == MT76X02_CIPHER_NONE && key)
return -EOPNOTSUPP;
val = mt76_rr(dev, MT_SKEY_MODE(vif_idx));
eiv = mt76_rr(dev, MT_WCID_IV(idx) + 4);
pn = (u64)eiv << 16;
- if (cipher == MT_CIPHER_TKIP) {
+ if (cipher == MT76X02_CIPHER_TKIP) {
pn |= (iv >> 16) & 0xff;
pn |= (iv & 0xff) << 8;
- } else if (cipher >= MT_CIPHER_AES_CCMP) {
+ } else if (cipher >= MT76X02_CIPHER_AES_CCMP) {
pn |= iv & 0xffff;
} else {
return;
u64 pn;
cipher = mt76x02_mac_get_key_info(key, key_data);
- if (cipher == MT_CIPHER_NONE && key)
+ if (cipher == MT76X02_CIPHER_NONE && key)
return -EOPNOTSUPP;
mt76_wr_copy(dev, MT_WCID_KEY(idx), key_data, sizeof(key_data));
pn = atomic64_read(&key->tx_pn);
iv_data[3] = key->keyidx << 6;
- if (cipher >= MT_CIPHER_TKIP) {
+ if (cipher >= MT76X02_CIPHER_TKIP) {
iv_data[3] |= 0x20;
put_unaligned_le32(pn >> 16, &iv_data[4]);
}
- if (cipher == MT_CIPHER_TKIP) {
+ if (cipher == MT76X02_CIPHER_TKIP) {
iv_data[0] = (pn >> 8) & 0xff;
iv_data[1] = (iv_data[0] | 0x20) & 0x7f;
iv_data[2] = pn & 0xff;
- } else if (cipher >= MT_CIPHER_AES_CCMP) {
+ } else if (cipher >= MT76X02_CIPHER_AES_CCMP) {
put_unaligned_le16((pn & 0xffff), &iv_data[0]);
}
}
mt76_wr(dev, MT_TX_PROT_CFG6 + i * 4, vht_prot[i]);
}
-void mt76x02_update_channel(struct mt76_dev *mdev)
+void mt76x02_update_channel(struct mt76_phy *mphy)
{
- struct mt76x02_dev *dev = container_of(mdev, struct mt76x02_dev, mt76);
+ struct mt76x02_dev *dev = container_of(mphy->dev, struct mt76x02_dev, mt76);
struct mt76_channel_state *state;
- state = mdev->phy.chan_state;
+ state = mphy->chan_state;
state->cc_busy += mt76_rr(dev, MT_CH_BUSY);
spin_lock_bh(&dev->mt76.cc_lock);
mutex_lock(&dev->mt76.mutex);
- mt76_update_survey(&dev->mt76);
+ mt76_update_survey(&dev->mphy);
for (i = 0, idx = 0; i < 16; i++) {
u32 val = mt76_rr(dev, MT_TX_AGG_CNT(i));
struct ieee80211_sta *sta, int len);
void mt76x02_mac_poll_tx_status(struct mt76x02_dev *dev, bool irq);
void mt76x02_tx_complete_skb(struct mt76_dev *mdev, struct mt76_queue_entry *e);
-void mt76x02_update_channel(struct mt76_dev *mdev);
+void mt76x02_update_channel(struct mt76_phy *mphy);
void mt76x02_mac_work(struct work_struct *work);
void mt76x02_mac_cc_reset(struct mt76x02_dev *dev);
} __packed __aligned(4);
enum mt76x02_cipher_type {
- MT_CIPHER_NONE,
- MT_CIPHER_WEP40,
- MT_CIPHER_WEP104,
- MT_CIPHER_TKIP,
- MT_CIPHER_AES_CCMP,
- MT_CIPHER_CKIP40,
- MT_CIPHER_CKIP104,
- MT_CIPHER_CKIP128,
- MT_CIPHER_WAPI,
+ MT76X02_CIPHER_NONE,
+ MT76X02_CIPHER_WEP40,
+ MT76X02_CIPHER_WEP104,
+ MT76X02_CIPHER_TKIP,
+ MT76X02_CIPHER_AES_CCMP,
+ MT76X02_CIPHER_CKIP40,
+ MT76X02_CIPHER_CKIP104,
+ MT76X02_CIPHER_CKIP128,
+ MT76X02_CIPHER_WAPI,
};
#endif
#include <linux/module.h>
#include "mt76x02.h"
-#define CCK_RATE(_idx, _rate) { \
+#define MT76x02_CCK_RATE(_idx, _rate) { \
.bitrate = _rate, \
.flags = IEEE80211_RATE_SHORT_PREAMBLE, \
.hw_value = (MT_PHY_TYPE_CCK << 8) | (_idx), \
.hw_value_short = (MT_PHY_TYPE_CCK << 8) | (8 + (_idx)), \
}
-#define OFDM_RATE(_idx, _rate) { \
- .bitrate = _rate, \
- .hw_value = (MT_PHY_TYPE_OFDM << 8) | (_idx), \
- .hw_value_short = (MT_PHY_TYPE_OFDM << 8) | (_idx), \
-}
-
struct ieee80211_rate mt76x02_rates[] = {
- CCK_RATE(0, 10),
- CCK_RATE(1, 20),
- CCK_RATE(2, 55),
- CCK_RATE(3, 110),
+ MT76x02_CCK_RATE(0, 10),
+ MT76x02_CCK_RATE(1, 20),
+ MT76x02_CCK_RATE(2, 55),
+ MT76x02_CCK_RATE(3, 110),
OFDM_RATE(0, 60),
OFDM_RATE(1, 90),
OFDM_RATE(2, 120),
-#SPDX-License-Identifier: ISC
+# SPDX-License-Identifier: ISC
obj-$(CONFIG_MT7915E) += mt7915e.o
#include "mt7915.h"
#include "eeprom.h"
+#include "mcu.h"
/** global debugfs **/
dev->ibf = !!val;
- return mt7915_mcu_set_txbf_type(dev);
+ return mt7915_mcu_set_txbf(dev, MT_BF_TYPE_UPDATE);
}
static int
{
struct mt7915_dev *dev = s->private;
bool ext_phy = phy != &dev->phy;
+ static const char * const bw[] = {
+ "BW20", "BW40", "BW80", "BW160"
+ };
int cnt;
if (!phy)
seq_puts(s, "Tx Beamformer Rx feedback statistics: ");
cnt = mt76_rr(dev, MT_ETBF_RX_FB_CNT(ext_phy));
- seq_printf(s, "All: %ld, HE: %ld, VHT: %ld, HT: %ld\n",
+ seq_printf(s, "All: %ld, HE: %ld, VHT: %ld, HT: %ld, ",
FIELD_GET(MT_ETBF_RX_FB_ALL, cnt),
FIELD_GET(MT_ETBF_RX_FB_HE, cnt),
FIELD_GET(MT_ETBF_RX_FB_VHT, cnt),
FIELD_GET(MT_ETBF_RX_FB_HT, cnt));
+ cnt = mt76_rr(dev, MT_ETBF_RX_FB_CONT(ext_phy));
+ seq_printf(s, "%s, NC: %ld, NR: %ld\n",
+ bw[FIELD_GET(MT_ETBF_RX_FB_BW, cnt)],
+ FIELD_GET(MT_ETBF_RX_FB_NC, cnt),
+ FIELD_GET(MT_ETBF_RX_FB_NR, cnt));
/* Tx Beamformee Rx NDPA & Tx feedback report */
cnt = mt76_rr(dev, MT_ETBF_TX_NDP_BFRP(ext_phy));
mt7915_txbf_stat_read_phy(mt7915_ext_phy(dev), file);
/* Tx amsdu info */
- seq_puts(file, "Tx MSDU stat:\n");
+ seq_puts(file, "Tx MSDU statistics:\n");
for (i = 0, n = 0; i < ARRAY_SIZE(stat); i++) {
stat[i] = mt76_rr(dev, MT_PLE_AMSDU_PACK_MSDU_CNT(i));
n += stat[i];
DEFINE_SHOW_ATTRIBUTE(mt7915_tx_stats);
-static int mt7915_read_temperature(struct seq_file *s, void *data)
-{
- struct mt7915_dev *dev = dev_get_drvdata(s->private);
- int temp;
-
- /* cpu */
- temp = mt7915_mcu_get_temperature(dev, 0);
- seq_printf(s, "Temperature: %d\n", temp);
-
- return 0;
-}
-
static int
mt7915_queues_acq(struct seq_file *s, void *data)
{
"RU26", "RU52", "RU106", "RU242/SU20",
"RU484/SU40", "RU996/SU80", "RU2x996/SU160"
};
- struct mt7915_dev *dev = dev_get_drvdata(s->private);
- bool ext_phy = phy != &dev->phy;
- u32 reg_base;
- int i, idx = 0;
+ s8 txpower[MT7915_SKU_RATE_NUM], *buf;
+ int i;
if (!phy)
return;
- reg_base = MT_TMAC_FP0R0(ext_phy);
- seq_printf(s, "\nBand %d\n", ext_phy);
+ seq_printf(s, "\nBand %d\n", phy != &phy->dev->phy);
- for (i = 0; i < ARRAY_SIZE(mt7915_sku_group_len); i++) {
- u8 cnt, mcs_num = mt7915_sku_group_len[i];
- s8 txpower[12];
- int j;
+ mt7915_mcu_get_txpower_sku(phy, txpower, sizeof(txpower));
+ for (i = 0, buf = txpower; i < ARRAY_SIZE(mt7915_sku_group_len); i++) {
+ u8 mcs_num = mt7915_sku_group_len[i];
- if (i == SKU_HT_BW20 || i == SKU_HT_BW40) {
- mcs_num = 8;
- } else if (i >= SKU_VHT_BW20 && i <= SKU_VHT_BW160) {
+ if (i >= SKU_VHT_BW20 && i <= SKU_VHT_BW160)
mcs_num = 10;
- } else if (i == SKU_HE_RU26) {
- reg_base = MT_TMAC_FP0R18(ext_phy);
- idx = 0;
- }
-
- for (j = 0, cnt = 0; j < DIV_ROUND_UP(mcs_num, 4); j++) {
- u32 val;
-
- if (i == SKU_VHT_BW160 && idx == 60) {
- reg_base = MT_TMAC_FP0R15(ext_phy);
- idx = 0;
- }
-
- val = mt76_rr(dev, reg_base + (idx / 4) * 4);
-
- if (idx && idx % 4)
- val >>= (idx % 4) * 8;
-
- while (val > 0 && cnt < mcs_num) {
- s8 pwr = FIELD_GET(MT_TMAC_FP_MASK, val);
-
- txpower[cnt++] = pwr;
- val >>= 8;
- idx++;
- }
- }
- mt76_seq_puts_array(s, sku_group_name[i], txpower, mcs_num);
+ mt76_seq_puts_array(s, sku_group_name[i], buf, mcs_num);
+ buf += mt7915_sku_group_len[i];
}
}
debugfs_create_file("radar_trigger", 0200, dir, dev,
&fops_radar_trigger);
debugfs_create_file("ser_trigger", 0200, dir, dev, &fops_ser_trigger);
- debugfs_create_devm_seqfile(dev->mt76.dev, "temperature", dir,
- mt7915_read_temperature);
debugfs_create_devm_seqfile(dev->mt76.dev, "txpower_sku", dir,
mt7915_read_rate_txpower);
return 0;
}
-void mt7915_queue_rx_skb(struct mt76_dev *mdev, enum mt76_rxq_id q,
- struct sk_buff *skb)
-{
- struct mt7915_dev *dev = container_of(mdev, struct mt7915_dev, mt76);
- __le32 *rxd = (__le32 *)skb->data;
- enum rx_pkt_type type;
-
- type = FIELD_GET(MT_RXD0_PKT_TYPE, le32_to_cpu(rxd[0]));
-
- switch (type) {
- case PKT_TYPE_TXRX_NOTIFY:
- mt7915_mac_tx_free(dev, skb);
- break;
- case PKT_TYPE_RX_EVENT:
- mt7915_mcu_rx_event(dev, skb);
- break;
-#ifdef CONFIG_NL80211_TESTMODE
- case PKT_TYPE_TXRXV:
- mt7915_mac_fill_rx_vector(dev, skb);
- break;
-#endif
- case PKT_TYPE_NORMAL:
- if (!mt7915_mac_fill_rx(dev, skb)) {
- mt76_rx(&dev->mt76, q, skb);
- return;
- }
- fallthrough;
- default:
- dev_kfree_skb(skb);
- break;
- }
-}
-
static void
mt7915_tx_cleanup(struct mt7915_dev *dev)
{
int mt7915_dma_init(struct mt7915_dev *dev)
{
- /* Increase buffer size to receive large VHT/HE MPDUs */
- int rx_buf_size = MT_RX_BUF_SIZE * 2;
u32 hif1_ofs = 0;
int ret;
/* event from WM */
ret = mt76_queue_alloc(dev, &dev->mt76.q_rx[MT_RXQ_MCU],
MT7915_RXQ_MCU_WM, MT7915_RX_MCU_RING_SIZE,
- rx_buf_size, MT_RX_EVENT_RING_BASE);
+ MT_RX_BUF_SIZE, MT_RX_EVENT_RING_BASE);
if (ret)
return ret;
/* event from WA */
ret = mt76_queue_alloc(dev, &dev->mt76.q_rx[MT_RXQ_MCU_WA],
MT7915_RXQ_MCU_WA, MT7915_RX_MCU_RING_SIZE,
- rx_buf_size, MT_RX_EVENT_RING_BASE);
+ MT_RX_BUF_SIZE, MT_RX_EVENT_RING_BASE);
if (ret)
return ret;
/* rx data queue */
ret = mt76_queue_alloc(dev, &dev->mt76.q_rx[MT_RXQ_MAIN],
MT7915_RXQ_BAND0, MT7915_RX_RING_SIZE,
- rx_buf_size, MT_RX_DATA_RING_BASE);
+ MT_RX_BUF_SIZE, MT_RX_DATA_RING_BASE);
if (ret)
return ret;
if (dev->dbdc_support) {
ret = mt76_queue_alloc(dev, &dev->mt76.q_rx[MT_RXQ_EXT],
MT7915_RXQ_BAND1, MT7915_RX_RING_SIZE,
- rx_buf_size,
+ MT_RX_BUF_SIZE,
MT_RX_DATA_RING_BASE + hif1_ofs);
if (ret)
return ret;
ret = mt76_queue_alloc(dev, &dev->mt76.q_rx[MT_RXQ_EXT_WA],
MT7915_RXQ_MCU_WA_EXT,
MT7915_RX_MCU_RING_SIZE,
- rx_buf_size,
+ MT_RX_BUF_SIZE,
MT_RX_EVENT_RING_BASE + hif1_ofs);
if (ret)
return ret;
#include "mt7915.h"
#include "eeprom.h"
-static u32 mt7915_eeprom_read(struct mt7915_dev *dev, u32 offset)
-{
- u8 *data = dev->mt76.eeprom.data;
-
- if (data[offset] == 0xff && !dev->flash_mode)
- mt7915_mcu_get_eeprom(dev, offset);
-
- return data[offset];
-}
-
static int mt7915_eeprom_load_precal(struct mt7915_dev *dev)
{
struct mt76_dev *mdev = &dev->mt76;
- u32 val;
+ u8 *eeprom = mdev->eeprom.data;
+ u32 val = eeprom[MT_EE_DO_PRE_CAL];
- val = mt7915_eeprom_read(dev, MT_EE_DO_PRE_CAL);
if (val != (MT_EE_WIFI_CAL_DPD | MT_EE_WIFI_CAL_GROUP))
return 0;
dev->flash_mode = true;
ret = mt7915_eeprom_load_precal(dev);
} else {
- memset(dev->mt76.eeprom.data, -1, MT7915_EEPROM_SIZE);
+ u32 block_num, i;
+
+ block_num = DIV_ROUND_UP(MT7915_EEPROM_SIZE,
+ MT7915_EEPROM_BLOCK_SIZE);
+ for (i = 0; i < block_num; i++)
+ mt7915_mcu_get_eeprom(dev,
+ i * MT7915_EEPROM_BLOCK_SIZE);
}
return ret;
static int mt7915_check_eeprom(struct mt7915_dev *dev)
{
u8 *eeprom = dev->mt76.eeprom.data;
- u16 val;
-
- mt7915_eeprom_read(dev, MT_EE_CHIP_ID);
- val = get_unaligned_le16(eeprom);
+ u16 val = get_unaligned_le16(eeprom);
switch (val) {
case 0x7915:
{
struct mt7915_dev *dev = phy->dev;
bool ext_phy = phy != &dev->phy;
+ u8 *eeprom = dev->mt76.eeprom.data;
u32 val;
- val = mt7915_eeprom_read(dev, MT_EE_WIFI_CONF + ext_phy);
+ val = eeprom[MT_EE_WIFI_CONF + ext_phy];
val = FIELD_GET(MT_EE_WIFI_CONF0_BAND_SEL, val);
if (val == MT_EE_BAND_SEL_DEFAULT && dev->dbdc_support)
val = ext_phy ? MT_EE_BAND_SEL_5GHZ : MT_EE_BAND_SEL_2GHZ;
struct ieee80211_channel *chan,
u8 chain_idx)
{
+ u8 *eeprom = dev->mt76.eeprom.data;
int index, target_power;
bool tssi_on;
if (chan->band == NL80211_BAND_2GHZ) {
index = MT_EE_TX0_POWER_2G + chain_idx * 3;
- target_power = mt7915_eeprom_read(dev, index);
+ target_power = eeprom[index];
if (!tssi_on)
- target_power += mt7915_eeprom_read(dev, index + 1);
+ target_power += eeprom[index + 1];
} else {
int group = mt7915_get_channel_group(chan->hw_value);
index = MT_EE_TX0_POWER_5G + chain_idx * 12;
- target_power = mt7915_eeprom_read(dev, index + group);
+ target_power = eeprom[index + group];
if (!tssi_on)
- target_power += mt7915_eeprom_read(dev, index + 8);
+ target_power += eeprom[index + 8];
}
return target_power;
s8 mt7915_eeprom_get_power_delta(struct mt7915_dev *dev, int band)
{
+ u8 *eeprom = dev->mt76.eeprom.data;
u32 val;
s8 delta;
if (band == NL80211_BAND_2GHZ)
- val = mt7915_eeprom_read(dev, MT_EE_RATE_DELTA_2G);
+ val = eeprom[MT_EE_RATE_DELTA_2G];
else
- val = mt7915_eeprom_read(dev, MT_EE_RATE_DELTA_5G);
+ val = eeprom[MT_EE_RATE_DELTA_5G];
if (!(val & MT_EE_RATE_DELTA_EN))
return 0;
#define MT_EE_WIFI_CAL_GROUP BIT(0)
#define MT_EE_WIFI_CAL_DPD GENMASK(2, 1)
#define MT_EE_CAL_UNIT 1024
-#define MT_EE_CAL_GROUP_SIZE (44 * MT_EE_CAL_UNIT)
+#define MT_EE_CAL_GROUP_SIZE (49 * MT_EE_CAL_UNIT + 16)
#define MT_EE_CAL_DPD_SIZE (54 * MT_EE_CAL_UNIT)
#define MT_EE_WIFI_CONF0_TX_PATH GENMASK(2, 0)
mt7915_tssi_enabled(struct mt7915_dev *dev, enum nl80211_band band)
{
u8 *eep = dev->mt76.eeprom.data;
+ u8 val = eep[MT_EE_WIFI_CONF + 7];
- /* TODO: DBDC */
- if (band == NL80211_BAND_5GHZ)
- return eep[MT_EE_WIFI_CONF + 7] & MT_EE_WIFI_CONF7_TSSI0_5G;
+ if (band == NL80211_BAND_2GHZ)
+ return val & MT_EE_WIFI_CONF7_TSSI0_2G;
+
+ if (dev->dbdc_support)
+ return val & MT_EE_WIFI_CONF7_TSSI1_5G;
else
- return eep[MT_EE_WIFI_CONF + 7] & MT_EE_WIFI_CONF7_TSSI0_2G;
+ return val & MT_EE_WIFI_CONF7_TSSI0_5G;
}
extern const u8 mt7915_sku_group_len[MAX_SKU_RATE_GROUP_NUM];
/* Copyright (C) 2020 MediaTek Inc. */
#include <linux/etherdevice.h>
+#include <linux/hwmon.h>
+#include <linux/hwmon-sysfs.h>
+#include <linux/thermal.h>
#include "mt7915.h"
#include "mac.h"
#include "mcu.h"
#include "eeprom.h"
-#define CCK_RATE(_idx, _rate) { \
- .bitrate = _rate, \
- .flags = IEEE80211_RATE_SHORT_PREAMBLE, \
- .hw_value = (MT_PHY_TYPE_CCK << 8) | (_idx), \
- .hw_value_short = (MT_PHY_TYPE_CCK << 8) | (4 + (_idx)), \
-}
-
-#define OFDM_RATE(_idx, _rate) { \
- .bitrate = _rate, \
- .hw_value = (MT_PHY_TYPE_OFDM << 8) | (_idx), \
- .hw_value_short = (MT_PHY_TYPE_OFDM << 8) | (_idx), \
-}
-
-static struct ieee80211_rate mt7915_rates[] = {
- CCK_RATE(0, 10),
- CCK_RATE(1, 20),
- CCK_RATE(2, 55),
- CCK_RATE(3, 110),
- OFDM_RATE(11, 60),
- OFDM_RATE(15, 90),
- OFDM_RATE(10, 120),
- OFDM_RATE(14, 180),
- OFDM_RATE(9, 240),
- OFDM_RATE(13, 360),
- OFDM_RATE(8, 480),
- OFDM_RATE(12, 540),
-};
-
static const struct ieee80211_iface_limit if_limits[] = {
{
.max = 1,
}
};
+static ssize_t mt7915_thermal_show_temp(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct mt7915_phy *phy = dev_get_drvdata(dev);
+ int temperature;
+
+ temperature = mt7915_mcu_get_temperature(phy);
+ if (temperature < 0)
+ return temperature;
+
+ /* display in millidegree celcius */
+ return sprintf(buf, "%u\n", temperature * 1000);
+}
+
+static SENSOR_DEVICE_ATTR(temp1_input, 0444, mt7915_thermal_show_temp,
+ NULL, 0);
+
+static struct attribute *mt7915_hwmon_attrs[] = {
+ &sensor_dev_attr_temp1_input.dev_attr.attr,
+ NULL,
+};
+ATTRIBUTE_GROUPS(mt7915_hwmon);
+
+static int
+mt7915_thermal_get_max_throttle_state(struct thermal_cooling_device *cdev,
+ unsigned long *state)
+{
+ *state = MT7915_THERMAL_THROTTLE_MAX;
+
+ return 0;
+}
+
+static int
+mt7915_thermal_get_cur_throttle_state(struct thermal_cooling_device *cdev,
+ unsigned long *state)
+{
+ struct mt7915_phy *phy = cdev->devdata;
+
+ *state = phy->throttle_state;
+
+ return 0;
+}
+
+static int
+mt7915_thermal_set_cur_throttle_state(struct thermal_cooling_device *cdev,
+ unsigned long state)
+{
+ struct mt7915_phy *phy = cdev->devdata;
+ int ret;
+
+ if (state > MT7915_THERMAL_THROTTLE_MAX)
+ return -EINVAL;
+
+ if (state == phy->throttle_state)
+ return 0;
+
+ ret = mt7915_mcu_set_thermal_throttling(phy, state);
+ if (ret)
+ return ret;
+
+ phy->throttle_state = state;
+
+ return 0;
+}
+
+static const struct thermal_cooling_device_ops mt7915_thermal_ops = {
+ .get_max_state = mt7915_thermal_get_max_throttle_state,
+ .get_cur_state = mt7915_thermal_get_cur_throttle_state,
+ .set_cur_state = mt7915_thermal_set_cur_throttle_state,
+};
+
+static void mt7915_unregister_thermal(struct mt7915_phy *phy)
+{
+ struct wiphy *wiphy = phy->mt76->hw->wiphy;
+
+ if (!phy->cdev)
+ return;
+
+ sysfs_remove_link(&wiphy->dev.kobj, "cooling_device");
+ thermal_cooling_device_unregister(phy->cdev);
+}
+
+static int mt7915_thermal_init(struct mt7915_phy *phy)
+{
+ struct wiphy *wiphy = phy->mt76->hw->wiphy;
+ struct thermal_cooling_device *cdev;
+ struct device *hwmon;
+
+ cdev = thermal_cooling_device_register(wiphy_name(wiphy), phy,
+ &mt7915_thermal_ops);
+ if (!IS_ERR(cdev)) {
+ if (sysfs_create_link(&wiphy->dev.kobj, &cdev->device.kobj,
+ "cooling_device") < 0)
+ thermal_cooling_device_unregister(cdev);
+ else
+ phy->cdev = cdev;
+ }
+
+ if (!IS_REACHABLE(CONFIG_HWMON))
+ return 0;
+
+ hwmon = devm_hwmon_device_register_with_groups(&wiphy->dev,
+ wiphy_name(wiphy), phy,
+ mt7915_hwmon_groups);
+ if (IS_ERR(hwmon))
+ return PTR_ERR(hwmon);
+
+ return 0;
+}
+
static void
mt7915_init_txpower(struct mt7915_dev *dev,
struct ieee80211_supported_band *sband)
FIELD_PREP(MT_MDP_RCFR1_RX_DROPPED_MCAST, MT_MDP_TO_HIF);
mt76_rmw(dev, MT_MDP_BNRCFR1(band), mask, set);
- mt76_set(dev, MT_WF_RMAC_MIB_TIME0(band), MT_WF_RMAC_MIB_RXTIME_EN);
mt76_set(dev, MT_WF_RMAC_MIB_AIRTIME0(band), MT_WF_RMAC_MIB_RXTIME_EN);
mt76_rmw_field(dev, MT_DMA_DCR0(band), MT_DMA_DCR0_MAX_RX_LEN, 1536);
{
int ret;
-
if (dev->dbdc_support) {
- ret = mt7915_mcu_set_txbf_module(dev);
+ ret = mt7915_mcu_set_txbf(dev, MT_BF_MODULE_UPDATE);
if (ret)
return ret;
}
/* trigger sounding packets */
- ret = mt7915_mcu_set_txbf_sounding(dev);
+ ret = mt7915_mcu_set_txbf(dev, MT_BF_SOUNDING_ON);
if (ret)
return ret;
/* enable eBF */
- return mt7915_mcu_set_txbf_type(dev);
+ return mt7915_mcu_set_txbf(dev, MT_BF_TYPE_UPDATE);
}
static int mt7915_register_ext_phy(struct mt7915_dev *dev)
if (ret)
goto error;
- ret = mt76_register_phy(mphy, true, mt7915_rates,
- ARRAY_SIZE(mt7915_rates));
+ ret = mt76_register_phy(mphy, true, mt76_rates,
+ ARRAY_SIZE(mt76_rates));
+ if (ret)
+ goto error;
+
+ ret = mt7915_thermal_init(phy);
if (ret)
goto error;
if (nss < 2)
return;
+ /* the maximum cap is 4 x 3, (Nr, Nc) = (3, 2) */
+ elem->phy_cap_info[7] |= min_t(int, nss - 1, 2) << 3;
+
if (vif != NL80211_IFTYPE_AP)
return;
c = IEEE80211_HE_PHY_CAP6_TRIG_SU_BEAMFORMING_FB |
IEEE80211_HE_PHY_CAP6_TRIG_MU_BEAMFORMING_PARTIAL_BW_FB;
elem->phy_cap_info[6] |= c;
-
- /* the maximum cap is 4 x 3, (Nr, Nc) = (3, 2) */
- elem->phy_cap_info[7] |= min_t(int, nss - 1, 2) << 3;
}
static void
switch (i) {
case NL80211_IFTYPE_AP:
- he_cap_elem->mac_cap_info[0] |=
- IEEE80211_HE_MAC_CAP0_TWT_RES;
he_cap_elem->mac_cap_info[2] |=
IEEE80211_HE_MAC_CAP2_BSR;
he_cap_elem->mac_cap_info[4] |=
IEEE80211_HE_PHY_CAP6_PPE_THRESHOLD_PRESENT;
break;
case NL80211_IFTYPE_STATION:
- he_cap_elem->mac_cap_info[0] |=
- IEEE80211_HE_MAC_CAP0_TWT_REQ;
he_cap_elem->mac_cap_info[1] |=
IEEE80211_HE_MAC_CAP1_TF_MAC_PAD_DUR_16US;
if (!phy)
return;
+ mt7915_unregister_thermal(phy);
mt76_unregister_phy(mphy);
ieee80211_free_hw(mphy->hw);
}
dev->mt76.test_ops = &mt7915_testmode_ops;
#endif
- ret = mt76_register_device(&dev->mt76, true, mt7915_rates,
- ARRAY_SIZE(mt7915_rates));
+ ret = mt76_register_device(&dev->mt76, true, mt76_rates,
+ ARRAY_SIZE(mt76_rates));
+ if (ret)
+ return ret;
+
+ ret = mt7915_thermal_init(&dev->phy);
if (ret)
return ret;
void mt7915_unregister_device(struct mt7915_dev *dev)
{
mt7915_unregister_ext_phy(dev);
+ mt7915_unregister_thermal(&dev->phy);
mt76_unregister_device(&dev->mt76);
mt7915_mcu_exit(dev);
mt7915_tx_token_put(dev);
mt7915_dma_cleanup(dev);
+ tasklet_disable(&dev->irq_tasklet);
mt76_free_device(&dev->mt76);
}
}
}
-int mt7915_mac_fill_rx(struct mt7915_dev *dev, struct sk_buff *skb)
+static int
+mt7915_mac_fill_rx(struct mt7915_dev *dev, struct sk_buff *skb)
{
struct mt76_rx_status *status = (struct mt76_rx_status *)skb->cb;
struct mt76_phy *mphy = &dev->mt76.phy;
u8 *data = (u8 *)rxd;
if (status->flag & RX_FLAG_DECRYPTED) {
- status->iv[0] = data[5];
- status->iv[1] = data[4];
- status->iv[2] = data[3];
- status->iv[3] = data[2];
- status->iv[4] = data[1];
- status->iv[5] = data[0];
-
- insert_ccmp_hdr = FIELD_GET(MT_RXD2_NORMAL_FRAG, rxd2);
+ switch (FIELD_GET(MT_RXD1_NORMAL_SEC_MODE, rxd1)) {
+ case MT_CIPHER_AES_CCMP:
+ case MT_CIPHER_CCMP_CCX:
+ case MT_CIPHER_CCMP_256:
+ insert_ccmp_hdr =
+ FIELD_GET(MT_RXD2_NORMAL_FRAG, rxd2);
+ fallthrough;
+ case MT_CIPHER_TKIP:
+ case MT_CIPHER_TKIP_NO_MIC:
+ case MT_CIPHER_GCMP:
+ case MT_CIPHER_GCMP_256:
+ status->iv[0] = data[5];
+ status->iv[1] = data[4];
+ status->iv[2] = data[3];
+ status->iv[3] = data[2];
+ status->iv[4] = data[1];
+ status->iv[5] = data[0];
+ break;
+ default:
+ break;
+ }
}
rxd += 4;
if ((u8 *)rxd - skb->data >= skb->len)
return 0;
}
-#ifdef CONFIG_NL80211_TESTMODE
-void mt7915_mac_fill_rx_vector(struct mt7915_dev *dev, struct sk_buff *skb)
+static void
+mt7915_mac_fill_rx_vector(struct mt7915_dev *dev, struct sk_buff *skb)
{
+#ifdef CONFIG_NL80211_TESTMODE
struct mt7915_phy *phy = &dev->phy;
__le32 *rxd = (__le32 *)skb->data;
__le32 *rxv_hdr = rxd + 2;
phy->test.last_freq_offset = foe;
phy->test.last_snr = snr;
+#endif
dev_kfree_skb(skb);
}
-#endif
static void
mt7915_mac_write_txwi_tm(struct mt7915_phy *phy, __le32 *txwi,
}
void mt7915_mac_write_txwi(struct mt7915_dev *dev, __le32 *txwi,
- struct sk_buff *skb, struct mt76_wcid *wcid,
+ struct sk_buff *skb, struct mt76_wcid *wcid, int pid,
struct ieee80211_key_conf *key, bool beacon)
{
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
txwi[3] = cpu_to_le32(val);
txwi[4] = 0;
- txwi[5] = 0;
+
+ val = FIELD_PREP(MT_TXD5_PID, pid);
+ if (pid >= MT_PACKET_ID_FIRST)
+ val |= MT_TXD5_TX_STATUS_HOST;
+ txwi[5] = cpu_to_le32(val);
+
txwi[6] = 0;
txwi[7] = wcid->amsdu ? cpu_to_le32(MT_TXD7_HW_AMSDU) : 0;
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(tx_info->skb);
struct ieee80211_key_conf *key = info->control.hw_key;
struct ieee80211_vif *vif = info->control.vif;
- struct mt76_tx_cb *cb = mt76_tx_skb_cb(tx_info->skb);
struct mt76_txwi_cache *t;
struct mt7915_txp *txp;
int id, i, nbuf = tx_info->nbuf - 1;
u8 *txwi = (u8 *)txwi_ptr;
+ int pid;
if (unlikely(tx_info->skb->len <= ETH_HLEN))
return -EINVAL;
if (!wcid)
wcid = &dev->mt76.global_wcid;
- mt7915_mac_write_txwi(dev, txwi_ptr, tx_info->skb, wcid, key,
- false);
+ pid = mt76_tx_status_skb_add(mdev, wcid, tx_info->skb);
- cb->wcid = wcid->idx;
+ mt7915_mac_write_txwi(dev, txwi_ptr, tx_info->skb, wcid, pid, key,
+ false);
txp = (struct mt7915_txp *)(txwi + MT_TXD_SIZE);
for (i = 0; i < nbuf; i++) {
}
static void
-mt7915_tx_complete_status(struct mt76_dev *mdev, struct sk_buff *skb,
- struct ieee80211_sta *sta, u8 stat,
- struct list_head *free_list)
-{
- struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
- struct ieee80211_tx_status status = {
- .sta = sta,
- .info = info,
- .skb = skb,
- .free_list = free_list,
- };
- struct ieee80211_hw *hw;
-
- if (sta) {
- struct mt7915_sta *msta;
-
- msta = (struct mt7915_sta *)sta->drv_priv;
- status.rate = &msta->stats.tx_rate;
- }
-
-#ifdef CONFIG_NL80211_TESTMODE
- if (mt76_is_testmode_skb(mdev, skb, &hw)) {
- struct mt7915_phy *phy = mt7915_hw_phy(hw);
- struct ieee80211_vif *vif = phy->monitor_vif;
- struct mt7915_vif *mvif = (struct mt7915_vif *)vif->drv_priv;
-
- mt76_tx_complete_skb(mdev, mvif->sta.wcid.idx, skb);
- return;
- }
-#endif
-
- hw = mt76_tx_status_get_hw(mdev, skb);
-
- if (info->flags & IEEE80211_TX_CTL_AMPDU)
- info->flags |= IEEE80211_TX_STAT_AMPDU;
-
- if (stat)
- ieee80211_tx_info_clear_status(info);
-
- if (!(info->flags & IEEE80211_TX_CTL_NO_ACK))
- info->flags |= IEEE80211_TX_STAT_ACK;
-
- info->status.tx_time = 0;
- ieee80211_tx_status_ext(hw, &status);
-}
-
-void mt7915_txp_skb_unmap(struct mt76_dev *dev,
- struct mt76_txwi_cache *t)
+mt7915_txp_skb_unmap(struct mt76_dev *dev, struct mt76_txwi_cache *t)
{
struct mt7915_txp *txp;
int i;
le16_to_cpu(txp->len[i]), DMA_TO_DEVICE);
}
-void mt7915_mac_tx_free(struct mt7915_dev *dev, struct sk_buff *skb)
+static void
+mt7915_txwi_free(struct mt7915_dev *dev, struct mt76_txwi_cache *t,
+ struct ieee80211_sta *sta, struct list_head *free_list)
+{
+ struct mt76_dev *mdev = &dev->mt76;
+ struct mt76_wcid *wcid;
+ __le32 *txwi;
+ u16 wcid_idx;
+
+ mt7915_txp_skb_unmap(mdev, t);
+ if (!t->skb)
+ goto out;
+
+ txwi = (__le32 *)mt76_get_txwi_ptr(mdev, t);
+ if (sta) {
+ wcid = (struct mt76_wcid *)sta->drv_priv;
+ wcid_idx = wcid->idx;
+
+ if (likely(t->skb->protocol != cpu_to_be16(ETH_P_PAE)))
+ mt7915_tx_check_aggr(sta, txwi);
+ } else {
+ wcid_idx = FIELD_GET(MT_TXD1_WLAN_IDX, le32_to_cpu(txwi[1]));
+ }
+
+ __mt76_tx_complete_skb(mdev, wcid_idx, t->skb, free_list);
+
+out:
+ t->skb = NULL;
+ mt76_put_txwi(mdev, t);
+}
+
+static void
+mt7915_mac_tx_free(struct mt7915_dev *dev, struct sk_buff *skb)
{
struct mt7915_tx_free *free = (struct mt7915_tx_free *)skb->data;
struct mt76_dev *mdev = &dev->mt76;
if (!txwi)
continue;
- mt7915_txp_skb_unmap(mdev, txwi);
- if (txwi->skb) {
- struct ieee80211_tx_info *info = IEEE80211_SKB_CB(txwi->skb);
- void *txwi_ptr = mt76_get_txwi_ptr(mdev, txwi);
-
- if (likely(txwi->skb->protocol != cpu_to_be16(ETH_P_PAE)))
- mt7915_tx_check_aggr(sta, txwi_ptr);
-
- if (sta && !info->tx_time_est) {
- struct mt76_wcid *wcid = (struct mt76_wcid *)sta->drv_priv;
- int pending;
-
- pending = atomic_dec_return(&wcid->non_aql_packets);
- if (pending < 0)
- atomic_cmpxchg(&wcid->non_aql_packets, pending, 0);
- }
-
- mt7915_tx_complete_status(mdev, txwi->skb, sta, stat, &free_list);
- txwi->skb = NULL;
- }
-
- mt76_put_txwi(mdev, txwi);
+ mt7915_txwi_free(dev, txwi, sta, &free_list);
}
mt7915_mac_sta_poll(dev);
}
}
+static bool
+mt7915_mac_add_txs_skb(struct mt7915_dev *dev, struct mt76_wcid *wcid, int pid,
+ __le32 *txs_data)
+{
+ struct mt76_dev *mdev = &dev->mt76;
+ struct ieee80211_tx_info *info;
+ struct sk_buff_head list;
+ struct sk_buff *skb;
+
+ mt76_tx_status_lock(mdev, &list);
+ skb = mt76_tx_status_skb_get(mdev, wcid, pid, &list);
+ if (!skb)
+ goto out;
+
+ info = IEEE80211_SKB_CB(skb);
+ if (!(txs_data[0] & le32_to_cpu(MT_TXS0_ACK_ERROR_MASK)))
+ info->flags |= IEEE80211_TX_STAT_ACK;
+
+ info->status.ampdu_len = 1;
+ info->status.ampdu_ack_len = !!(info->flags &
+ IEEE80211_TX_STAT_ACK);
+
+ info->status.rates[0].idx = -1;
+ mt76_tx_status_skb_done(mdev, skb, &list);
+
+out:
+ mt76_tx_status_unlock(mdev, &list);
+
+ return !!skb;
+}
+
+static void mt7915_mac_add_txs(struct mt7915_dev *dev, void *data)
+{
+ struct mt7915_sta *msta = NULL;
+ struct mt76_wcid *wcid;
+ __le32 *txs_data = data;
+ u16 wcidx;
+ u32 txs;
+ u8 pid;
+
+ txs = le32_to_cpu(txs_data[0]);
+ if (FIELD_GET(MT_TXS0_TXS_FORMAT, txs) > 1)
+ return;
+
+ txs = le32_to_cpu(txs_data[2]);
+ wcidx = FIELD_GET(MT_TXS2_WCID, txs);
+
+ txs = le32_to_cpu(txs_data[3]);
+ pid = FIELD_GET(MT_TXS3_PID, txs);
+
+ if (pid < MT_PACKET_ID_FIRST)
+ return;
+
+ if (wcidx >= MT7915_WTBL_SIZE)
+ return;
+
+ rcu_read_lock();
+
+ wcid = rcu_dereference(dev->mt76.wcid[wcidx]);
+ if (!wcid)
+ goto out;
+
+ mt7915_mac_add_txs_skb(dev, wcid, pid, txs_data);
+
+ if (!wcid->sta)
+ goto out;
+
+ msta = container_of(wcid, struct mt7915_sta, wcid);
+ spin_lock_bh(&dev->sta_poll_lock);
+ if (list_empty(&msta->poll_list))
+ list_add_tail(&msta->poll_list, &dev->sta_poll_list);
+ spin_unlock_bh(&dev->sta_poll_lock);
+
+out:
+ rcu_read_unlock();
+}
+
+void mt7915_queue_rx_skb(struct mt76_dev *mdev, enum mt76_rxq_id q,
+ struct sk_buff *skb)
+{
+ struct mt7915_dev *dev = container_of(mdev, struct mt7915_dev, mt76);
+ __le32 *rxd = (__le32 *)skb->data;
+ __le32 *end = (__le32 *)&skb->data[skb->len];
+ enum rx_pkt_type type;
+
+ type = FIELD_GET(MT_RXD0_PKT_TYPE, le32_to_cpu(rxd[0]));
+
+ switch (type) {
+ case PKT_TYPE_TXRX_NOTIFY:
+ mt7915_mac_tx_free(dev, skb);
+ break;
+ case PKT_TYPE_RX_EVENT:
+ mt7915_mcu_rx_event(dev, skb);
+ break;
+ case PKT_TYPE_TXRXV:
+ mt7915_mac_fill_rx_vector(dev, skb);
+ break;
+ case PKT_TYPE_TXS:
+ for (rxd += 2; rxd + 8 <= end; rxd += 8)
+ mt7915_mac_add_txs(dev, rxd);
+ dev_kfree_skb(skb);
+ break;
+ case PKT_TYPE_NORMAL:
+ if (!mt7915_mac_fill_rx(dev, skb)) {
+ mt76_rx(&dev->mt76, q, skb);
+ return;
+ }
+ fallthrough;
+ default:
+ dev_kfree_skb(skb);
+ break;
+ }
+}
+
void mt7915_tx_complete_skb(struct mt76_dev *mdev, struct mt76_queue_entry *e)
{
struct mt7915_dev *dev;
e->skb = t ? t->skb : NULL;
}
- if (e->skb) {
- struct mt76_tx_cb *cb = mt76_tx_skb_cb(e->skb);
- struct mt76_wcid *wcid;
-
- wcid = rcu_dereference(dev->mt76.wcid[cb->wcid]);
-
- mt7915_tx_complete_status(mdev, e->skb, wcid_to_sta(wcid), 0,
- NULL);
- }
+ if (e->skb)
+ mt76_tx_complete_skb(mdev, e->wcid, e->skb);
}
void mt7915_mac_cca_stats_reset(struct mt7915_phy *phy)
memset(&dev->mt76.aggr_stats[i], 0, sizeof(dev->mt76.aggr_stats) / 2);
/* reset airtime counters */
- mt76_rr(dev, MT_MIB_SDR9(ext_phy));
- mt76_rr(dev, MT_MIB_SDR36(ext_phy));
- mt76_rr(dev, MT_MIB_SDR37(ext_phy));
-
- mt76_set(dev, MT_WF_RMAC_MIB_TIME0(ext_phy),
- MT_WF_RMAC_MIB_RXTIME_CLR);
mt76_set(dev, MT_WF_RMAC_MIB_AIRTIME0(ext_phy),
MT_WF_RMAC_MIB_RXTIME_CLR);
+
+ mt7915_mcu_get_chan_mib_info(phy, true);
}
void mt7915_mac_set_timing(struct mt7915_phy *phy)
return sum / n;
}
-static void
-mt7915_phy_update_channel(struct mt76_phy *mphy, int idx)
+void mt7915_update_channel(struct mt76_phy *mphy)
{
- struct mt7915_dev *dev = container_of(mphy->dev, struct mt7915_dev, mt76);
struct mt7915_phy *phy = (struct mt7915_phy *)mphy->priv;
- struct mt76_channel_state *state;
- u64 busy_time, tx_time, rx_time, obss_time;
+ struct mt76_channel_state *state = mphy->chan_state;
+ bool ext_phy = phy != &phy->dev->phy;
int nf;
- busy_time = mt76_get_field(dev, MT_MIB_SDR9(idx),
- MT_MIB_SDR9_BUSY_MASK);
- tx_time = mt76_get_field(dev, MT_MIB_SDR36(idx),
- MT_MIB_SDR36_TXTIME_MASK);
- rx_time = mt76_get_field(dev, MT_MIB_SDR37(idx),
- MT_MIB_SDR37_RXTIME_MASK);
- obss_time = mt76_get_field(dev, MT_WF_RMAC_MIB_AIRTIME14(idx),
- MT_MIB_OBSSTIME_MASK);
+ mt7915_mcu_get_chan_mib_info(phy, false);
- nf = mt7915_phy_get_nf(phy, idx);
+ nf = mt7915_phy_get_nf(phy, ext_phy);
if (!phy->noise)
phy->noise = nf << 4;
else if (nf)
phy->noise += nf - (phy->noise >> 4);
- state = mphy->chan_state;
- state->cc_busy += busy_time;
- state->cc_tx += tx_time;
- state->cc_rx += rx_time + obss_time;
- state->cc_bss_rx += rx_time;
state->noise = -(phy->noise >> 4);
}
-void mt7915_update_channel(struct mt76_dev *mdev)
-{
- struct mt7915_dev *dev = container_of(mdev, struct mt7915_dev, mt76);
-
- mt7915_phy_update_channel(&mdev->phy, 0);
- if (mdev->phy2)
- mt7915_phy_update_channel(mdev->phy2, 1);
-
- /* reset obss airtime */
- mt76_set(dev, MT_WF_RMAC_MIB_TIME0(0), MT_WF_RMAC_MIB_RXTIME_CLR);
- if (mdev->phy2)
- mt76_set(dev, MT_WF_RMAC_MIB_TIME0(1),
- MT_WF_RMAC_MIB_RXTIME_CLR);
-}
-
static bool
mt7915_wait_reset_state(struct mt7915_dev *dev, u32 state)
{
mt76_set(dev, MT_WFDMA0_GLO_CFG,
MT_WFDMA0_GLO_CFG_TX_DMA_EN | MT_WFDMA0_GLO_CFG_RX_DMA_EN);
mt76_set(dev, MT_WFDMA1_GLO_CFG,
- MT_WFDMA1_GLO_CFG_TX_DMA_EN | MT_WFDMA1_GLO_CFG_RX_DMA_EN);
+ MT_WFDMA1_GLO_CFG_TX_DMA_EN | MT_WFDMA1_GLO_CFG_RX_DMA_EN |
+ MT_WFDMA1_GLO_CFG_OMIT_TX_INFO |
+ MT_WFDMA1_GLO_CFG_OMIT_RX_INFO);
if (dev->hif2) {
mt76_set(dev, MT_WFDMA0_GLO_CFG + hif1_ofs,
(MT_WFDMA0_GLO_CFG_TX_DMA_EN |
MT_WFDMA0_GLO_CFG_RX_DMA_EN));
mt76_set(dev, MT_WFDMA1_GLO_CFG + hif1_ofs,
(MT_WFDMA1_GLO_CFG_TX_DMA_EN |
- MT_WFDMA1_GLO_CFG_RX_DMA_EN));
+ MT_WFDMA1_GLO_CFG_RX_DMA_EN |
+ MT_WFDMA1_GLO_CFG_OMIT_TX_INFO |
+ MT_WFDMA1_GLO_CFG_OMIT_RX_INFO));
}
}
spin_lock_bh(&dev->mt76.token_lock);
idr_for_each_entry(&dev->mt76.token, txwi, id) {
- mt7915_txp_skb_unmap(&dev->mt76, txwi);
- if (txwi->skb) {
- struct ieee80211_hw *hw;
-
- hw = mt76_tx_status_get_hw(&dev->mt76, txwi->skb);
- ieee80211_free_txskb(hw, txwi->skb);
- }
- mt76_put_txwi(&dev->mt76, txwi);
+ mt7915_txwi_free(dev, txwi, NULL, NULL);
dev->mt76.token_count--;
}
spin_unlock_bh(&dev->mt76.token_lock);
set_bit(MT76_RESET, &phy2->mt76->state);
cancel_delayed_work_sync(&phy2->mt76->mac_work);
}
- /* lock/unlock all queues to ensure that no tx is pending */
- mt76_txq_schedule_all(&dev->mphy);
- if (ext_phy)
- mt76_txq_schedule_all(ext_phy);
-
mt76_worker_disable(&dev->mt76.tx_worker);
napi_disable(&dev->mt76.napi[0]);
napi_disable(&dev->mt76.napi[1]);
if (phy2)
clear_bit(MT76_RESET, &phy2->mt76->state);
- mt76_worker_enable(&dev->mt76.tx_worker);
- napi_enable(&dev->mt76.tx_napi);
- napi_schedule(&dev->mt76.tx_napi);
-
napi_enable(&dev->mt76.napi[0]);
napi_schedule(&dev->mt76.napi[0]);
napi_enable(&dev->mt76.napi[2]);
napi_schedule(&dev->mt76.napi[2]);
+ tasklet_schedule(&dev->irq_tasklet);
+
+ mt76_wr(dev, MT_MCU_INT_EVENT, MT_MCU_INT_EVENT_RESET_DONE);
+ mt7915_wait_reset_state(dev, MT_MCU_CMD_NORMAL_STATE);
+
+ mt76_worker_enable(&dev->mt76.tx_worker);
+
+ napi_enable(&dev->mt76.tx_napi);
+ napi_schedule(&dev->mt76.tx_napi);
ieee80211_wake_queues(mt76_hw(dev));
if (ext_phy)
ieee80211_wake_queues(ext_phy->hw);
- mt76_wr(dev, MT_MCU_INT_EVENT, MT_MCU_INT_EVENT_RESET_DONE);
- mt7915_wait_reset_state(dev, MT_MCU_CMD_NORMAL_STATE);
-
mutex_unlock(&dev->mt76.mutex);
mt7915_update_beacons(dev);
}
static void
-mt7915_mac_update_mib_stats(struct mt7915_phy *phy)
+mt7915_mac_update_stats(struct mt7915_phy *phy)
{
struct mt7915_dev *dev = phy->dev;
struct mib_stats *mib = &phy->mib;
if (changed & (IEEE80211_RC_SUPP_RATES_CHANGED |
IEEE80211_RC_NSS_CHANGED |
- IEEE80211_RC_BW_CHANGED))
+ IEEE80211_RC_BW_CHANGED)) {
+ mt7915_mcu_add_he(dev, vif, sta);
mt7915_mcu_add_rate_ctrl(dev, vif, sta);
+ }
if (changed & IEEE80211_RC_SMPS_CHANGED)
mt7915_mcu_add_smps(dev, vif, sta);
mutex_lock(&mphy->dev->mutex);
- mt76_update_survey(mphy->dev);
+ mt76_update_survey(mphy);
if (++mphy->mac_work_count == 5) {
mphy->mac_work_count = 0;
- mt7915_mac_update_mib_stats(phy);
+ mt7915_mac_update_stats(phy);
}
if (++phy->sta_work_count == 10) {
mutex_unlock(&mphy->dev->mutex);
+ mt76_tx_status_check(mphy->dev, NULL, false);
+
ieee80211_queue_delayed_work(mphy->hw, &mphy->mac_work,
MT7915_WATCHDOG_TIME);
}
/* will support this field in further revision */
#define MT_TX_FREE_RATE GENMASK(13, 0)
+#define MT_TXS0_FIXED_RATE BIT(31)
+#define MT_TXS0_BW GENMASK(30, 29)
+#define MT_TXS0_TID GENMASK(28, 26)
+#define MT_TXS0_AMPDU BIT(25)
+#define MT_TXS0_TXS_FORMAT GENMASK(24, 23)
+#define MT_TXS0_BA_ERROR BIT(22)
+#define MT_TXS0_PS_FLAG BIT(21)
+#define MT_TXS0_TXOP_TIMEOUT BIT(20)
+#define MT_TXS0_BIP_ERROR BIT(19)
+
+#define MT_TXS0_QUEUE_TIMEOUT BIT(18)
+#define MT_TXS0_RTS_TIMEOUT BIT(17)
+#define MT_TXS0_ACK_TIMEOUT BIT(16)
+#define MT_TXS0_ACK_ERROR_MASK GENMASK(18, 16)
+
+#define MT_TXS0_TX_STATUS_HOST BIT(15)
+#define MT_TXS0_TX_STATUS_MCU BIT(14)
+#define MT_TXS0_TX_RATE GENMASK(13, 0)
+
+#define MT_TXS1_SEQNO GENMASK(31, 20)
+#define MT_TXS1_RESP_RATE GENMASK(19, 16)
+#define MT_TXS1_RXV_SEQNO GENMASK(15, 8)
+#define MT_TXS1_TX_POWER_DBM GENMASK(7, 0)
+
+#define MT_TXS2_BF_STATUS GENMASK(31, 30)
+#define MT_TXS2_LAST_TX_RATE GENMASK(29, 27)
+#define MT_TXS2_SHARED_ANTENNA BIT(26)
+#define MT_TXS2_WCID GENMASK(25, 16)
+#define MT_TXS2_TX_DELAY GENMASK(15, 0)
+
+#define MT_TXS3_PID GENMASK(31, 24)
+#define MT_TXS3_ANT_ID GENMASK(23, 0)
+
+#define MT_TXS4_TIMESTAMP GENMASK(31, 0)
+
+#define MT_TXS5_F0_FINAL_MPDU BIT(31)
+#define MT_TXS5_F0_QOS BIT(30)
+#define MT_TXS5_F0_TX_COUNT GENMASK(29, 25)
+#define MT_TXS5_F0_FRONT_TIME GENMASK(24, 0)
+#define MT_TXS5_F1_MPDU_TX_COUNT GENMASK(31, 24)
+#define MT_TXS5_F1_MPDU_TX_BYTES GENMASK(23, 0)
+
+#define MT_TXS6_F0_NOISE_3 GENMASK(31, 24)
+#define MT_TXS6_F0_NOISE_2 GENMASK(23, 16)
+#define MT_TXS6_F0_NOISE_1 GENMASK(15, 8)
+#define MT_TXS6_F0_NOISE_0 GENMASK(7, 0)
+#define MT_TXS6_F1_MPDU_FAIL_COUNT GENMASK(31, 24)
+#define MT_TXS6_F1_MPDU_FAIL_BYTES GENMASK(23, 0)
+
+#define MT_TXS7_F0_RCPI_3 GENMASK(31, 24)
+#define MT_TXS7_F0_RCPI_2 GENMASK(23, 16)
+#define MT_TXS7_F0_RCPI_1 GENMASK(15, 8)
+#define MT_TXS7_F0_RCPI_0 GENMASK(7, 0)
+#define MT_TXS7_F1_MPDU_RETRY_COUNT GENMASK(31, 24)
+#define MT_TXS7_F1_MPDU_RETRY_BYTES GENMASK(23, 0)
+
struct mt7915_dfs_pulse {
u32 max_width; /* us */
int max_pwr; /* dbm */
if (type != NL80211_IFTYPE_STATION)
break;
- /* next, try to find a free repeater entry for the sta */
- i = get_free_idx(mask >> REPEATER_BSSID_START, 0,
- REPEATER_BSSID_MAX - REPEATER_BSSID_START);
- if (i)
- return i + 32 - 1;
-
i = get_free_idx(mask, EXT_BSSID_1, EXT_BSSID_MAX);
if (i)
return i - 1;
return -1;
}
+static void mt7915_init_bitrate_mask(struct ieee80211_vif *vif)
+{
+ struct mt7915_vif *mvif = (struct mt7915_vif *)vif->drv_priv;
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(mvif->bitrate_mask.control); i++) {
+ mvif->bitrate_mask.control[i].legacy = GENMASK(31, 0);
+ memset(mvif->bitrate_mask.control[i].ht_mcs, GENMASK(7, 0),
+ sizeof(mvif->bitrate_mask.control[i].ht_mcs));
+ memset(mvif->bitrate_mask.control[i].vht_mcs, GENMASK(15, 0),
+ sizeof(mvif->bitrate_mask.control[i].vht_mcs));
+ memset(mvif->bitrate_mask.control[i].he_mcs, GENMASK(15, 0),
+ sizeof(mvif->bitrate_mask.control[i].he_mcs));
+ }
+}
+
static int mt7915_add_interface(struct ieee80211_hw *hw,
struct ieee80211_vif *vif)
{
vif->offload_flags = 0;
vif->offload_flags |= IEEE80211_OFFLOAD_ENCAP_4ADDR;
+ mt7915_init_bitrate_mask(vif);
+
out:
mutex_unlock(&dev->mt76.mutex);
n = mvif->omac_idx > HW_BSSID_MAX ? HW_BSSID_0 : mvif->omac_idx;
/* TSF software read */
- mt76_set(dev, MT_LPON_TCR(band, n), MT_LPON_TCR_SW_MODE);
+ mt76_rmw(dev, MT_LPON_TCR(band, n), MT_LPON_TCR_SW_MODE,
+ MT_LPON_TCR_SW_READ);
tsf.t32[0] = mt76_rr(dev, MT_LPON_UTTR0(band));
tsf.t32[1] = mt76_rr(dev, MT_LPON_UTTR1(band));
mt76_wr(dev, MT_LPON_UTTR0(band), tsf.t32[0]);
mt76_wr(dev, MT_LPON_UTTR1(band), tsf.t32[1]);
/* TSF software overwrite */
- mt76_set(dev, MT_LPON_TCR(band, n), MT_LPON_TCR_SW_WRITE);
+ mt76_rmw(dev, MT_LPON_TCR(band, n), MT_LPON_TCR_SW_MODE,
+ MT_LPON_TCR_SW_WRITE);
+
+ mutex_unlock(&dev->mt76.mutex);
+}
+
+static void
+mt7915_offset_tsf(struct ieee80211_hw *hw, struct ieee80211_vif *vif,
+ s64 timestamp)
+{
+ struct mt7915_vif *mvif = (struct mt7915_vif *)vif->drv_priv;
+ struct mt7915_dev *dev = mt7915_hw_dev(hw);
+ struct mt7915_phy *phy = mt7915_hw_phy(hw);
+ bool band = phy != &dev->phy;
+ union {
+ u64 t64;
+ u32 t32[2];
+ } tsf = { .t64 = timestamp, };
+ u16 n;
+
+ mutex_lock(&dev->mt76.mutex);
+
+ n = mvif->omac_idx > HW_BSSID_MAX ? HW_BSSID_0 : mvif->omac_idx;
+ mt76_wr(dev, MT_LPON_UTTR0(band), tsf.t32[0]);
+ mt76_wr(dev, MT_LPON_UTTR1(band), tsf.t32[1]);
+ /* TSF software adjust*/
+ mt76_rmw(dev, MT_LPON_TCR(band, n), MT_LPON_TCR_SW_MODE,
+ MT_LPON_TCR_SW_ADJUST);
mutex_unlock(&dev->mt76.mutex);
}
sinfo->filled |= BIT_ULL(NL80211_STA_INFO_TX_BITRATE);
}
-static void
-mt7915_sta_rc_update(struct ieee80211_hw *hw,
- struct ieee80211_vif *vif,
- struct ieee80211_sta *sta,
- u32 changed)
+static void mt7915_sta_rc_work(void *data, struct ieee80211_sta *sta)
{
- struct mt7915_dev *dev = mt7915_hw_dev(hw);
struct mt7915_sta *msta = (struct mt7915_sta *)sta->drv_priv;
+ struct mt7915_dev *dev = msta->vif->phy->dev;
+ struct ieee80211_hw *hw = msta->vif->phy->mt76->hw;
+ u32 *changed = data;
spin_lock_bh(&dev->sta_poll_lock);
- msta->stats.changed |= changed;
+ msta->stats.changed |= *changed;
if (list_empty(&msta->rc_list))
list_add_tail(&msta->rc_list, &dev->sta_rc_list);
spin_unlock_bh(&dev->sta_poll_lock);
ieee80211_queue_work(hw, &dev->rc_work);
}
+static void mt7915_sta_rc_update(struct ieee80211_hw *hw,
+ struct ieee80211_vif *vif,
+ struct ieee80211_sta *sta,
+ u32 changed)
+{
+ mt7915_sta_rc_work(&changed, sta);
+}
+
+static int
+mt7915_set_bitrate_mask(struct ieee80211_hw *hw, struct ieee80211_vif *vif,
+ const struct cfg80211_bitrate_mask *mask)
+{
+ struct mt7915_vif *mvif = (struct mt7915_vif *)vif->drv_priv;
+ enum nl80211_band band = mvif->phy->mt76->chandef.chan->band;
+ u32 changed;
+
+ if (mask->control[band].gi == NL80211_TXRATE_FORCE_LGI)
+ return -EINVAL;
+
+ changed = IEEE80211_RC_SUPP_RATES_CHANGED;
+ mvif->bitrate_mask = *mask;
+
+ /* Update firmware rate control to add a boundary on top of table
+ * to limit the rate selection for each peer, so when set bitrates
+ * vht-mcs-5 1:9, which actually means nss = 1 mcs = 0~9. This only
+ * applies to data frames as for the other mgmt, mcast, bcast still
+ * use legacy rates as it is.
+ */
+ ieee80211_iterate_stations_atomic(hw, mt7915_sta_rc_work, &changed);
+
+ return 0;
+}
+
static void mt7915_sta_set_4addr(struct ieee80211_hw *hw,
struct ieee80211_vif *vif,
struct ieee80211_sta *sta,
.get_stats = mt7915_get_stats,
.get_tsf = mt7915_get_tsf,
.set_tsf = mt7915_set_tsf,
+ .offset_tsf = mt7915_offset_tsf,
.get_survey = mt76_get_survey,
.get_antenna = mt76_get_antenna,
.set_antenna = mt7915_set_antenna,
+ .set_bitrate_mask = mt7915_set_bitrate_mask,
.set_coverage_class = mt7915_set_coverage_class,
.sta_statistics = mt7915_sta_statistics,
.sta_set_4addr = mt7915_sta_set_4addr,
#define HE_PHY(p, c) u8_get_bits(c, IEEE80211_HE_PHY_##p)
#define HE_MAC(m, c) u8_get_bits(c, IEEE80211_HE_MAC_##m)
-static enum mt7915_cipher_type
+static enum mcu_cipher_type
mt7915_mcu_get_cipher(int cipher)
{
switch (cipher) {
case WLAN_CIPHER_SUITE_WEP40:
- return MT_CIPHER_WEP40;
+ return MCU_CIPHER_WEP40;
case WLAN_CIPHER_SUITE_WEP104:
- return MT_CIPHER_WEP104;
+ return MCU_CIPHER_WEP104;
case WLAN_CIPHER_SUITE_TKIP:
- return MT_CIPHER_TKIP;
+ return MCU_CIPHER_TKIP;
case WLAN_CIPHER_SUITE_AES_CMAC:
- return MT_CIPHER_BIP_CMAC_128;
+ return MCU_CIPHER_BIP_CMAC_128;
case WLAN_CIPHER_SUITE_CCMP:
- return MT_CIPHER_AES_CCMP;
+ return MCU_CIPHER_AES_CCMP;
case WLAN_CIPHER_SUITE_CCMP_256:
- return MT_CIPHER_CCMP_256;
+ return MCU_CIPHER_CCMP_256;
case WLAN_CIPHER_SUITE_GCMP:
- return MT_CIPHER_GCMP;
+ return MCU_CIPHER_GCMP;
case WLAN_CIPHER_SUITE_GCMP_256:
- return MT_CIPHER_GCMP_256;
+ return MCU_CIPHER_GCMP_256;
case WLAN_CIPHER_SUITE_SMS4:
- return MT_CIPHER_WAPI;
+ return MCU_CIPHER_WAPI;
default:
return MT_CIPHER_NONE;
}
}
static u8
-mt7915_get_phy_mode(struct mt76_phy *mphy, struct ieee80211_vif *vif,
- struct ieee80211_sta *sta)
+mt7915_get_phy_mode(struct ieee80211_vif *vif, struct ieee80211_sta *sta)
{
- enum nl80211_band band = mphy->chandef.chan->band;
+ struct mt7915_vif *mvif = (struct mt7915_vif *)vif->drv_priv;
+ enum nl80211_band band = mvif->phy->mt76->chandef.chan->band;
struct ieee80211_sta_ht_cap *ht_cap;
struct ieee80211_sta_vht_cap *vht_cap;
const struct ieee80211_sta_he_cap *he_cap;
} else {
struct ieee80211_supported_band *sband;
- sband = mphy->hw->wiphy->bands[band];
+ sband = mvif->phy->mt76->hw->wiphy->bands[band];
ht_cap = &sband->ht_cap;
vht_cap = &sband->vht_cap;
return nss - 1;
}
+static void
+mt7915_mcu_set_sta_he_mcs(struct ieee80211_sta *sta, __le16 *he_mcs,
+ const u16 *mask)
+{
+ struct mt7915_sta *msta = (struct mt7915_sta *)sta->drv_priv;
+ struct cfg80211_chan_def *chandef = &msta->vif->phy->mt76->chandef;
+ int nss, max_nss = sta->rx_nss > 3 ? 4 : sta->rx_nss;
+ u16 mcs_map;
+
+ switch (chandef->width) {
+ case NL80211_CHAN_WIDTH_80P80:
+ mcs_map = le16_to_cpu(sta->he_cap.he_mcs_nss_supp.rx_mcs_80p80);
+ break;
+ case NL80211_CHAN_WIDTH_160:
+ mcs_map = le16_to_cpu(sta->he_cap.he_mcs_nss_supp.rx_mcs_160);
+ break;
+ default:
+ mcs_map = le16_to_cpu(sta->he_cap.he_mcs_nss_supp.rx_mcs_80);
+ break;
+ }
+
+ for (nss = 0; nss < max_nss; nss++) {
+ int mcs;
+
+ switch ((mcs_map >> (2 * nss)) & 0x3) {
+ case IEEE80211_HE_MCS_SUPPORT_0_11:
+ mcs = GENMASK(11, 0);
+ break;
+ case IEEE80211_HE_MCS_SUPPORT_0_9:
+ mcs = GENMASK(9, 0);
+ break;
+ case IEEE80211_HE_MCS_SUPPORT_0_7:
+ mcs = GENMASK(7, 0);
+ break;
+ default:
+ mcs = 0;
+ }
+
+ mcs = mcs ? fls(mcs & mask[nss]) - 1 : -1;
+
+ switch (mcs) {
+ case 0 ... 7:
+ mcs = IEEE80211_HE_MCS_SUPPORT_0_7;
+ break;
+ case 8 ... 9:
+ mcs = IEEE80211_HE_MCS_SUPPORT_0_9;
+ break;
+ case 10 ... 11:
+ mcs = IEEE80211_HE_MCS_SUPPORT_0_11;
+ break;
+ default:
+ mcs = IEEE80211_HE_MCS_NOT_SUPPORTED;
+ break;
+ }
+ mcs_map &= ~(0x3 << (nss * 2));
+ mcs_map |= mcs << (nss * 2);
+
+ /* only support 2ss on 160MHz */
+ if (nss > 1 && (sta->bandwidth == IEEE80211_STA_RX_BW_160))
+ break;
+ }
+
+ *he_mcs = cpu_to_le16(mcs_map);
+}
+
+static void
+mt7915_mcu_set_sta_vht_mcs(struct ieee80211_sta *sta, __le16 *vht_mcs,
+ const u16 *mask)
+{
+ u16 mcs_map = le16_to_cpu(sta->vht_cap.vht_mcs.rx_mcs_map);
+ int nss, max_nss = sta->rx_nss > 3 ? 4 : sta->rx_nss;
+ u16 mcs;
+
+ for (nss = 0; nss < max_nss; nss++, mcs_map >>= 2) {
+ switch (mcs_map & 0x3) {
+ case IEEE80211_VHT_MCS_SUPPORT_0_9:
+ mcs = GENMASK(9, 0);
+ break;
+ case IEEE80211_VHT_MCS_SUPPORT_0_8:
+ mcs = GENMASK(8, 0);
+ break;
+ case IEEE80211_VHT_MCS_SUPPORT_0_7:
+ mcs = GENMASK(7, 0);
+ break;
+ default:
+ mcs = 0;
+ }
+
+ vht_mcs[nss] = cpu_to_le16(mcs & mask[nss]);
+
+ /* only support 2ss on 160MHz */
+ if (nss > 1 && (sta->bandwidth == IEEE80211_STA_RX_BW_160))
+ break;
+ }
+}
+
+static void
+mt7915_mcu_set_sta_ht_mcs(struct ieee80211_sta *sta, u8 *ht_mcs,
+ const u8 *mask)
+{
+ int nss, max_nss = sta->rx_nss > 3 ? 4 : sta->rx_nss;
+
+ for (nss = 0; nss < max_nss; nss++)
+ ht_mcs[nss] = sta->ht_cap.mcs.rx_mask[nss] & mask[nss];
+}
+
static int
mt7915_mcu_parse_response(struct mt76_dev *mdev, int cmd,
struct sk_buff *skb, int seq)
}
static void
+mt7915_mcu_rx_thermal_notify(struct mt7915_dev *dev, struct sk_buff *skb)
+{
+ struct mt76_phy *mphy = &dev->mt76.phy;
+ struct mt7915_mcu_thermal_notify *t;
+ struct mt7915_phy *phy;
+
+ t = (struct mt7915_mcu_thermal_notify *)skb->data;
+ if (t->ctrl.ctrl_id != THERMAL_PROTECT_ENABLE)
+ return;
+
+ if (t->ctrl.band_idx && dev->mt76.phy2)
+ mphy = dev->mt76.phy2;
+
+ phy = (struct mt7915_phy *)mphy->priv;
+ phy->throttle_state = t->ctrl.duty.duty_cycle;
+}
+
+static void
mt7915_mcu_rx_radar_detected(struct mt7915_dev *dev, struct sk_buff *skb)
{
struct mt76_phy *mphy = &dev->mt76.phy;
u16 attempts = le16_to_cpu(ra->attempts);
u16 curr = le16_to_cpu(ra->curr_rate);
u16 wcidx = le16_to_cpu(ra->wlan_idx);
+ struct ieee80211_tx_status status = {};
struct mt76_phy *mphy = &dev->mphy;
struct mt7915_sta_stats *stats;
struct mt7915_sta *msta;
stats->per = 1000 * (attempts - success) / attempts;
}
+
+ status.sta = wcid_to_sta(wcid);
+ if (!status.sta)
+ return;
+
+ status.rate = &stats->tx_rate;
+ ieee80211_tx_status_ext(mphy->hw, &status);
}
static void
struct mt7915_mcu_rxd *rxd = (struct mt7915_mcu_rxd *)skb->data;
switch (rxd->ext_eid) {
+ case MCU_EXT_EVENT_THERMAL_PROTECT:
+ mt7915_mcu_rx_thermal_notify(dev, skb);
+ break;
case MCU_EXT_EVENT_RDD_REPORT:
mt7915_mcu_rx_radar_detected(dev, skb);
break;
memcpy(bss->bssid, vif->bss_conf.bssid, ETH_ALEN);
bss->bcn_interval = cpu_to_le16(vif->bss_conf.beacon_int);
bss->dtim_period = vif->bss_conf.dtim_period;
- bss->phy_mode = mt7915_get_phy_mode(phy->mt76, vif, NULL);
+ bss->phy_mode = mt7915_get_phy_mode(vif, NULL);
} else {
memcpy(bss->bssid, phy->mt76->macaddr, ETH_ALEN);
}
sec_key = &sec->key[0];
sec_key->cipher_len = sizeof(*sec_key);
- if (cipher == MT_CIPHER_BIP_CMAC_128) {
- sec_key->cipher_id = MT_CIPHER_AES_CCMP;
+ if (cipher == MCU_CIPHER_BIP_CMAC_128) {
+ sec_key->cipher_id = MCU_CIPHER_AES_CCMP;
sec_key->key_id = bip->keyidx;
sec_key->key_len = 16;
memcpy(sec_key->key, bip->key, 16);
sec_key = &sec->key[1];
- sec_key->cipher_id = MT_CIPHER_BIP_CMAC_128;
+ sec_key->cipher_id = MCU_CIPHER_BIP_CMAC_128;
sec_key->cipher_len = sizeof(*sec_key);
sec_key->key_len = 16;
memcpy(sec_key->key, key->key, 16);
sec_key->key_len = key->keylen;
memcpy(sec_key->key, key->key, key->keylen);
- if (cipher == MT_CIPHER_TKIP) {
+ if (cipher == MCU_CIPHER_TKIP) {
/* Rx/Tx MIC keys are swapped */
memcpy(sec_key->key + 16, key->key + 24, 8);
memcpy(sec_key->key + 24, key->key + 16, 8);
}
/* store key_conf for BIP batch update */
- if (cipher == MT_CIPHER_AES_CCMP) {
+ if (cipher == MCU_CIPHER_AES_CCMP) {
memcpy(bip->key, key->key, key->keylen);
bip->keyidx = key->keyidx;
}
static void
mt7915_mcu_sta_he_tlv(struct sk_buff *skb, struct ieee80211_sta *sta)
{
+ struct mt7915_sta *msta = (struct mt7915_sta *)sta->drv_priv;
struct ieee80211_sta_he_cap *he_cap = &sta->he_cap;
struct ieee80211_he_cap_elem *elem = &he_cap->he_cap_elem;
+ enum nl80211_band band = msta->vif->phy->mt76->chandef.chan->band;
+ const u16 *mcs_mask = msta->vif->bitrate_mask.control[band].he_mcs;
struct sta_rec_he *he;
struct tlv *tlv;
u32 cap = 0;
case IEEE80211_STA_RX_BW_160:
if (elem->phy_cap_info[0] &
IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_80PLUS80_MHZ_IN_5G)
- he->max_nss_mcs[CMD_HE_MCS_BW8080] =
- he_cap->he_mcs_nss_supp.rx_mcs_80p80;
+ mt7915_mcu_set_sta_he_mcs(sta,
+ &he->max_nss_mcs[CMD_HE_MCS_BW8080],
+ mcs_mask);
- he->max_nss_mcs[CMD_HE_MCS_BW160] =
- he_cap->he_mcs_nss_supp.rx_mcs_160;
+ mt7915_mcu_set_sta_he_mcs(sta,
+ &he->max_nss_mcs[CMD_HE_MCS_BW160],
+ mcs_mask);
fallthrough;
default:
- he->max_nss_mcs[CMD_HE_MCS_BW80] =
- he_cap->he_mcs_nss_supp.rx_mcs_80;
+ mt7915_mcu_set_sta_he_mcs(sta,
+ &he->max_nss_mcs[CMD_HE_MCS_BW80],
+ mcs_mask);
break;
}
HE_PHY(CAP2_UL_MU_PARTIAL_MU_MIMO, elem->phy_cap_info[2]);
}
-static int
-mt7915_mcu_add_mu(struct mt7915_dev *dev, struct ieee80211_vif *vif,
- struct ieee80211_sta *sta)
+static void
+mt7915_mcu_sta_vht_tlv(struct sk_buff *skb, struct ieee80211_sta *sta)
{
- struct mt7915_vif *mvif = (struct mt7915_vif *)vif->drv_priv;
- struct mt7915_sta *msta = (struct mt7915_sta *)sta->drv_priv;
- struct sk_buff *skb;
- int len = sizeof(struct sta_req_hdr) + sizeof(struct sta_rec_muru);
-
- if (!sta->vht_cap.vht_supported && !sta->he_cap.has_he)
- return 0;
-
- skb = mt7915_mcu_alloc_sta_req(dev, mvif, msta, len);
- if (IS_ERR(skb))
- return PTR_ERR(skb);
+ struct sta_rec_vht *vht;
+ struct tlv *tlv;
- /* starec muru */
- mt7915_mcu_sta_muru_tlv(skb, sta);
+ tlv = mt7915_mcu_add_tlv(skb, STA_REC_VHT, sizeof(*vht));
- return mt76_mcu_skb_send_msg(&dev->mt76, skb,
- MCU_EXT_CMD(STA_REC_UPDATE), true);
+ vht = (struct sta_rec_vht *)tlv;
+ vht->vht_cap = cpu_to_le32(sta->vht_cap.cap);
+ vht->vht_rx_mcs_map = sta->vht_cap.vht_mcs.rx_mcs_map;
+ vht->vht_tx_mcs_map = sta->vht_cap.vht_mcs.tx_mcs_map;
}
static void
mt7915_mcu_sta_amsdu_tlv(skb, sta);
}
- /* starec vht */
- if (sta->vht_cap.vht_supported) {
- struct sta_rec_vht *vht;
-
- tlv = mt7915_mcu_add_tlv(skb, STA_REC_VHT, sizeof(*vht));
- vht = (struct sta_rec_vht *)tlv;
- vht->vht_cap = cpu_to_le32(sta->vht_cap.cap);
- vht->vht_rx_mcs_map = sta->vht_cap.vht_mcs.rx_mcs_map;
- vht->vht_tx_mcs_map = sta->vht_cap.vht_mcs.tx_mcs_map;
- }
-
/* starec he */
if (sta->he_cap.has_he)
mt7915_mcu_sta_he_tlv(skb, sta);
vc = mt7915_get_he_phy_cap(phy, vif);
ve = &vc->he_cap_elem;
- ebfee = !!((HE_PHY(CAP3_SU_BEAMFORMER, pe->phy_cap_info[3]) ||
- HE_PHY(CAP4_MU_BEAMFORMER, pe->phy_cap_info[4])) &&
+ ebfee = !!(HE_PHY(CAP3_SU_BEAMFORMER, pe->phy_cap_info[3]) &&
HE_PHY(CAP4_SU_BEAMFORMEE, ve->phy_cap_info[4]));
- ebf = !!((HE_PHY(CAP3_SU_BEAMFORMER, ve->phy_cap_info[3]) ||
- HE_PHY(CAP4_MU_BEAMFORMER, ve->phy_cap_info[4])) &&
+ ebf = !!(HE_PHY(CAP3_SU_BEAMFORMER, ve->phy_cap_info[3]) &&
HE_PHY(CAP4_SU_BEAMFORMEE, pe->phy_cap_info[4]));
} else if (sta->vht_cap.vht_supported) {
struct ieee80211_sta_vht_cap *pc;
struct ieee80211_sta_vht_cap *vc;
- u32 cr, ce;
pc = &sta->vht_cap;
vc = &phy->mt76->sband_5g.sband.vht_cap;
- cr = IEEE80211_VHT_CAP_SU_BEAMFORMER_CAPABLE |
- IEEE80211_VHT_CAP_MU_BEAMFORMER_CAPABLE;
- ce = IEEE80211_VHT_CAP_SU_BEAMFORMEE_CAPABLE |
- IEEE80211_VHT_CAP_MU_BEAMFORMEE_CAPABLE;
- ebfee = !!((pc->cap & cr) && (vc->cap & ce));
- ebf = !!((vc->cap & cr) && (pc->cap & ce));
+ ebfee = !!((pc->cap & IEEE80211_VHT_CAP_SU_BEAMFORMER_CAPABLE) &&
+ (vc->cap & IEEE80211_VHT_CAP_SU_BEAMFORMEE_CAPABLE));
+ ebf = !!((vc->cap & IEEE80211_VHT_CAP_SU_BEAMFORMER_CAPABLE) &&
+ (pc->cap & IEEE80211_VHT_CAP_SU_BEAMFORMEE_CAPABLE));
}
/* must keep each tag independent */
mt7915_mcu_sta_rate_ctrl_tlv(struct sk_buff *skb, struct mt7915_dev *dev,
struct ieee80211_vif *vif, struct ieee80211_sta *sta)
{
- struct mt7915_sta *msta = (struct mt7915_sta *)sta->drv_priv;
- struct mt76_phy *mphy = &dev->mphy;
- enum nl80211_band band;
+ struct mt7915_vif *mvif = (struct mt7915_vif *)vif->drv_priv;
+ struct cfg80211_chan_def *chandef = &mvif->phy->mt76->chandef;
+ struct cfg80211_bitrate_mask *mask = &mvif->bitrate_mask;
+ enum nl80211_band band = chandef->chan->band;
struct sta_rec_ra *ra;
struct tlv *tlv;
- u32 supp_rate, n_rates, cap = sta->wme ? STA_CAP_WMM : 0;
- u8 i, nss = sta->rx_nss, mcs = 0;
+ u32 supp_rate = sta->supp_rates[band];
+ u32 cap = sta->wme ? STA_CAP_WMM : 0;
tlv = mt7915_mcu_add_tlv(skb, STA_REC_RA, sizeof(*ra));
ra = (struct sta_rec_ra *)tlv;
- if (msta->wcid.ext_phy && dev->mt76.phy2)
- mphy = dev->mt76.phy2;
-
- band = mphy->chandef.chan->band;
- supp_rate = sta->supp_rates[band];
- n_rates = hweight32(supp_rate);
-
ra->valid = true;
ra->auto_rate = true;
- ra->phy_mode = mt7915_get_phy_mode(mphy, vif, sta);
- ra->channel = mphy->chandef.chan->hw_value;
+ ra->phy_mode = mt7915_get_phy_mode(vif, sta);
+ ra->channel = chandef->chan->hw_value;
ra->bw = sta->bandwidth;
- ra->rate_len = n_rates;
ra->phy.bw = sta->bandwidth;
- if (n_rates) {
+ if (supp_rate) {
+ supp_rate &= mask->control[band].legacy;
+ ra->rate_len = hweight32(supp_rate);
+
if (band == NL80211_BAND_2GHZ) {
ra->supp_mode = MODE_CCK;
ra->supp_cck_rate = supp_rate & GENMASK(3, 0);
- ra->phy.type = MT_PHY_TYPE_CCK;
- if (n_rates > 4) {
+ if (ra->rate_len > 4) {
ra->supp_mode |= MODE_OFDM;
ra->supp_ofdm_rate = supp_rate >> 4;
- ra->phy.type = MT_PHY_TYPE_OFDM;
}
} else {
ra->supp_mode = MODE_OFDM;
ra->supp_ofdm_rate = supp_rate;
- ra->phy.type = MT_PHY_TYPE_OFDM;
}
}
if (sta->ht_cap.ht_supported) {
- for (i = 0; i < nss; i++)
- ra->ht_mcs[i] = sta->ht_cap.mcs.rx_mask[i];
+ const u8 *mcs_mask = mask->control[band].ht_mcs;
- ra->supp_ht_mcs = *(__le32 *)ra->ht_mcs;
ra->supp_mode |= MODE_HT;
- mcs = hweight32(le32_to_cpu(ra->supp_ht_mcs)) - 1;
ra->af = sta->ht_cap.ampdu_factor;
ra->ht_gf = !!(sta->ht_cap.cap & IEEE80211_HT_CAP_GRN_FLD);
cap |= STA_CAP_RX_STBC;
if (sta->ht_cap.cap & IEEE80211_HT_CAP_LDPC_CODING)
cap |= STA_CAP_LDPC;
+
+ mt7915_mcu_set_sta_ht_mcs(sta, ra->ht_mcs, mcs_mask);
+ ra->supp_ht_mcs = *(__le32 *)ra->ht_mcs;
}
if (sta->vht_cap.vht_supported) {
- u16 mcs_map = le16_to_cpu(sta->vht_cap.vht_mcs.rx_mcs_map);
- u16 vht_mcs;
- u8 af, mcs_prev;
+ const u16 *mcs_mask = mask->control[band].vht_mcs;
+ u8 af;
+ ra->supp_mode |= MODE_VHT;
af = FIELD_GET(IEEE80211_VHT_CAP_MAX_A_MPDU_LENGTH_EXPONENT_MASK,
sta->vht_cap.cap);
ra->af = max_t(u8, ra->af, af);
if (sta->vht_cap.cap & IEEE80211_VHT_CAP_RXLDPC)
cap |= STA_CAP_VHT_LDPC;
- ra->supp_mode |= MODE_VHT;
- for (mcs = 0, i = 0; i < nss; i++, mcs_map >>= 2) {
- switch (mcs_map & 0x3) {
- case IEEE80211_VHT_MCS_SUPPORT_0_9:
- vht_mcs = GENMASK(9, 0);
- break;
- case IEEE80211_VHT_MCS_SUPPORT_0_8:
- vht_mcs = GENMASK(8, 0);
- break;
- case IEEE80211_VHT_MCS_SUPPORT_0_7:
- vht_mcs = GENMASK(7, 0);
- break;
- default:
- vht_mcs = 0;
- }
-
- ra->supp_vht_mcs[i] = cpu_to_le16(vht_mcs);
-
- mcs_prev = hweight16(vht_mcs) - 1;
- if (mcs_prev > mcs)
- mcs = mcs_prev;
-
- /* only support 2ss on 160MHz */
- if (i > 1 && (ra->bw == CMD_CBW_160MHZ ||
- ra->bw == CMD_CBW_8080MHZ))
- break;
- }
+ mt7915_mcu_set_sta_vht_mcs(sta, ra->supp_vht_mcs, mcs_mask);
}
if (sta->he_cap.has_he) {
cap |= STA_CAP_HE;
}
- ra->sta_status = cpu_to_le32(cap);
-
- switch (BIT(fls(ra->supp_mode) - 1)) {
- case MODE_VHT:
- ra->phy.type = MT_PHY_TYPE_VHT;
- ra->phy.mcs = mcs;
- ra->phy.nss = nss;
- ra->phy.stbc = !!(sta->vht_cap.cap & IEEE80211_VHT_CAP_TXSTBC);
- ra->phy.ldpc = !!(sta->vht_cap.cap & IEEE80211_VHT_CAP_RXLDPC);
- ra->phy.sgi =
- !!(sta->vht_cap.cap & IEEE80211_VHT_CAP_SHORT_GI_80);
- break;
- case MODE_HT:
- ra->phy.type = MT_PHY_TYPE_HT;
- ra->phy.mcs = mcs;
- ra->phy.ldpc = sta->ht_cap.cap & IEEE80211_HT_CAP_LDPC_CODING;
- ra->phy.stbc = !!(sta->ht_cap.cap & IEEE80211_HT_CAP_TX_STBC);
- ra->phy.sgi = !!(sta->ht_cap.cap & IEEE80211_HT_CAP_SGI_20);
- break;
- default:
- break;
- }
+ ra->sta_cap = cpu_to_le32(cap);
}
int mt7915_mcu_add_rate_ctrl(struct mt7915_dev *dev, struct ieee80211_vif *vif,
MCU_EXT_CMD(STA_REC_UPDATE), true);
}
+int mt7915_mcu_add_he(struct mt7915_dev *dev, struct ieee80211_vif *vif,
+ struct ieee80211_sta *sta)
+{
+ struct mt7915_vif *mvif = (struct mt7915_vif *)vif->drv_priv;
+ struct mt7915_sta *msta = (struct mt7915_sta *)sta->drv_priv;
+ struct sk_buff *skb;
+ int len;
+
+ if (!sta->he_cap.has_he)
+ return 0;
+
+ len = sizeof(struct sta_req_hdr) + sizeof(struct sta_rec_he);
+
+ skb = mt7915_mcu_alloc_sta_req(dev, mvif, msta, len);
+ if (IS_ERR(skb))
+ return PTR_ERR(skb);
+
+ mt7915_mcu_sta_he_tlv(skb, sta);
+
+ return mt76_mcu_skb_send_msg(&dev->mt76, skb,
+ MCU_EXT_CMD(STA_REC_UPDATE), true);
+}
+
+static int
+mt7915_mcu_add_group(struct mt7915_dev *dev, struct ieee80211_vif *vif,
+ struct ieee80211_sta *sta)
+{
+#define MT_STA_BSS_GROUP 1
+ struct mt7915_vif *mvif = (struct mt7915_vif *)vif->drv_priv;
+ struct mt7915_sta *msta = (struct mt7915_sta *)sta->drv_priv;
+ struct {
+ __le32 action;
+ u8 wlan_idx_lo;
+ u8 status;
+ u8 wlan_idx_hi;
+ u8 rsv0[5];
+ __le32 val;
+ u8 rsv1[8];
+ } __packed req = {
+ .action = cpu_to_le32(MT_STA_BSS_GROUP),
+ .wlan_idx_lo = to_wcid_lo(msta->wcid.idx),
+ .wlan_idx_hi = to_wcid_hi(msta->wcid.idx),
+ .val = cpu_to_le32(mvif->idx % 16),
+ };
+
+ return mt76_mcu_send_msg(&dev->mt76, MCU_EXT_CMD(SET_DRR_CTRL), &req,
+ sizeof(req), true);
+}
+
+static int
+mt7915_mcu_add_mu(struct mt7915_dev *dev, struct ieee80211_vif *vif,
+ struct ieee80211_sta *sta)
+{
+ struct mt7915_vif *mvif = (struct mt7915_vif *)vif->drv_priv;
+ struct mt7915_sta *msta = (struct mt7915_sta *)sta->drv_priv;
+ struct sk_buff *skb;
+ int ret;
+
+ if (!sta->vht_cap.vht_supported && !sta->he_cap.has_he)
+ return 0;
+
+ ret = mt7915_mcu_add_group(dev, vif, sta);
+ if (ret)
+ return ret;
+
+ skb = mt7915_mcu_alloc_sta_req(dev, mvif, msta,
+ MT7915_STA_UPDATE_MAX_SIZE);
+ if (IS_ERR(skb))
+ return PTR_ERR(skb);
+
+ /* wait until TxBF and MU ready to update stare vht */
+
+ /* starec muru */
+ mt7915_mcu_sta_muru_tlv(skb, sta);
+ /* starec vht */
+ mt7915_mcu_sta_vht_tlv(skb, sta);
+
+ return mt76_mcu_skb_send_msg(&dev->mt76, skb,
+ MCU_EXT_CMD(STA_REC_UPDATE), true);
+}
+
int mt7915_mcu_add_sta_adv(struct mt7915_dev *dev, struct ieee80211_vif *vif,
struct ieee80211_sta *sta, bool enable)
{
/* must keep the order */
ret = mt7915_mcu_add_txbf(dev, vif, sta, enable);
- if (ret)
+ if (ret || !enable)
return ret;
ret = mt7915_mcu_add_mu(dev, vif, sta);
if (ret)
return ret;
- if (enable)
- return mt7915_mcu_add_rate_ctrl(dev, vif, sta);
-
- return 0;
+ return mt7915_mcu_add_rate_ctrl(dev, vif, sta);
}
int mt7915_mcu_add_sta(struct mt7915_dev *dev, struct ieee80211_vif *vif,
cont->csa_ofs = cpu_to_le16(offs->cntdwn_counter_offs[0] - 4);
buf = (u8 *)tlv + sizeof(*cont);
- mt7915_mac_write_txwi(dev, (__le32 *)buf, skb, wcid, NULL,
+ mt7915_mac_write_txwi(dev, (__le32 *)buf, skb, wcid, 0, NULL,
true);
memcpy(buf + MT_TXD_SIZE, skb->data, skb->len);
}
int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset)
{
struct mt7915_mcu_eeprom_info req = {
- .addr = cpu_to_le32(round_down(offset, 16)),
+ .addr = cpu_to_le32(round_down(offset,
+ MT7915_EEPROM_BLOCK_SIZE)),
};
struct mt7915_mcu_eeprom_info *res;
struct sk_buff *skb;
res = (struct mt7915_mcu_eeprom_info *)skb->data;
buf = dev->mt76.eeprom.data + le32_to_cpu(res->addr);
- memcpy(buf, res->data, 16);
+ memcpy(buf, res->data, MT7915_EEPROM_BLOCK_SIZE);
dev_kfree_skb(skb);
return 0;
{
struct mt7915_dev *dev = phy->dev;
struct cfg80211_chan_def *chandef = &phy->mt76->chandef;
- u16 total = 2, idx, center_freq = chandef->center_freq1;
+ u16 total = 2, center_freq = chandef->center_freq1;
u8 *cal = dev->cal, *eep = dev->mt76.eeprom.data;
+ int idx;
if (!(eep[MT_EE_DO_PRE_CAL] & MT_EE_WIFI_CAL_DPD))
return 0;
return 0;
}
-int mt7915_mcu_get_temperature(struct mt7915_dev *dev, int index)
+int mt7915_mcu_get_chan_mib_info(struct mt7915_phy *phy, bool chan_switch)
+{
+ /* strict order */
+ static const enum mt7915_chan_mib_offs offs[] = {
+ MIB_BUSY_TIME, MIB_TX_TIME, MIB_RX_TIME, MIB_OBSS_AIRTIME
+ };
+ struct mt76_channel_state *state = phy->mt76->chan_state;
+ struct mt76_channel_state *state_ts = &phy->state_ts;
+ struct mt7915_dev *dev = phy->dev;
+ struct mt7915_mcu_mib *res, req[4];
+ struct sk_buff *skb;
+ int i, ret;
+
+ for (i = 0; i < 4; i++) {
+ req[i].band = cpu_to_le32(phy != &dev->phy);
+ req[i].offs = cpu_to_le32(offs[i]);
+ }
+
+ ret = mt76_mcu_send_and_get_msg(&dev->mt76, MCU_EXT_CMD(GET_MIB_INFO),
+ req, sizeof(req), true, &skb);
+ if (ret)
+ return ret;
+
+ res = (struct mt7915_mcu_mib *)(skb->data + 20);
+
+ if (chan_switch)
+ goto out;
+
+#define __res_u64(s) le64_to_cpu(res[s].data)
+ state->cc_busy += __res_u64(0) - state_ts->cc_busy;
+ state->cc_tx += __res_u64(1) - state_ts->cc_tx;
+ state->cc_bss_rx += __res_u64(2) - state_ts->cc_bss_rx;
+ state->cc_rx += __res_u64(2) + __res_u64(3) - state_ts->cc_rx;
+
+out:
+ state_ts->cc_busy = __res_u64(0);
+ state_ts->cc_tx = __res_u64(1);
+ state_ts->cc_bss_rx = __res_u64(2);
+ state_ts->cc_rx = __res_u64(2) + __res_u64(3);
+#undef __res_u64
+
+ dev_kfree_skb(skb);
+
+ return 0;
+}
+
+int mt7915_mcu_get_temperature(struct mt7915_phy *phy)
{
+ struct mt7915_dev *dev = phy->dev;
struct {
u8 ctrl_id;
u8 action;
- u8 band;
+ u8 dbdc_idx;
u8 rsv[5];
} req = {
.ctrl_id = THERMAL_SENSOR_TEMP_QUERY,
- .action = index,
+ .dbdc_idx = phy != &dev->phy,
};
return mt76_mcu_send_msg(&dev->mt76, MCU_EXT_CMD(THERMAL_CTRL), &req,
sizeof(req), true);
}
+int mt7915_mcu_set_thermal_throttling(struct mt7915_phy *phy, u8 state)
+{
+ struct mt7915_dev *dev = phy->dev;
+ struct {
+ struct mt7915_mcu_thermal_ctrl ctrl;
+
+ __le32 trigger_temp;
+ __le32 restore_temp;
+ __le16 sustain_time;
+ u8 rsv[2];
+ } __packed req = {
+ .ctrl = {
+ .band_idx = phy != &dev->phy,
+ },
+ };
+ int level;
+
+#define TRIGGER_TEMPERATURE 122
+#define RESTORE_TEMPERATURE 116
+#define SUSTAIN_PERIOD 10
+
+ if (!state) {
+ req.ctrl.ctrl_id = THERMAL_PROTECT_DISABLE;
+ goto out;
+ }
+
+ /* set duty cycle and level */
+ for (level = 0; level < 4; level++) {
+ int ret;
+
+ req.ctrl.ctrl_id = THERMAL_PROTECT_DUTY_CONFIG;
+ req.ctrl.duty.duty_level = level;
+ req.ctrl.duty.duty_cycle = state;
+ state = state * 4 / 5;
+
+ ret = mt76_mcu_send_msg(&dev->mt76, MCU_EXT_CMD(THERMAL_PROT),
+ &req, sizeof(req.ctrl), false);
+ if (ret)
+ return ret;
+ }
+
+ /* currently use fixed values for throttling, and would be better
+ * to implement thermal zone for dynamic trip in the long run.
+ */
+
+ /* set high-temperature trigger threshold */
+ req.ctrl.ctrl_id = THERMAL_PROTECT_ENABLE;
+ req.trigger_temp = cpu_to_le32(TRIGGER_TEMPERATURE);
+ req.restore_temp = cpu_to_le32(RESTORE_TEMPERATURE);
+ req.sustain_time = cpu_to_le16(SUSTAIN_PERIOD);
+
+out:
+ req.ctrl.type.protect_type = 1;
+ req.ctrl.type.trigger_type = 1;
+
+ return mt76_mcu_send_msg(&dev->mt76, MCU_EXT_CMD(THERMAL_PROT),
+ &req, sizeof(req), false);
+}
+
int mt7915_mcu_get_tx_rate(struct mt7915_dev *dev, u32 cmd, u16 wlan_idx)
{
struct {
int mt7915_mcu_set_txpower_sku(struct mt7915_phy *phy)
{
-#define MT7915_SKU_RATE_NUM 161
struct mt7915_dev *dev = phy->dev;
struct mt76_phy *mphy = phy->mt76;
struct ieee80211_hw *hw = mphy->hw;
sizeof(req), true);
}
+int mt7915_mcu_get_txpower_sku(struct mt7915_phy *phy, s8 *txpower, int len)
+{
+#define RATE_POWER_INFO 2
+ struct mt7915_dev *dev = phy->dev;
+ struct {
+ u8 format_id;
+ u8 category;
+ u8 band;
+ u8 _rsv;
+ } __packed req = {
+ .format_id = 7,
+ .category = RATE_POWER_INFO,
+ .band = phy != &dev->phy,
+ };
+ s8 res[MT7915_SKU_RATE_NUM][2];
+ struct sk_buff *skb;
+ int ret, i;
+
+ ret = mt76_mcu_send_and_get_msg(&dev->mt76,
+ MCU_EXT_CMD(TX_POWER_FEATURE_CTRL),
+ &req, sizeof(req), true, &skb);
+ if (ret)
+ return ret;
+
+ memcpy(res, skb->data + 4, sizeof(res));
+ for (i = 0; i < len; i++)
+ txpower[i] = res[i][req.band];
+
+ dev_kfree_skb(skb);
+
+ return 0;
+}
+
int mt7915_mcu_set_test_param(struct mt7915_dev *dev, u8 param, bool test_mode,
u8 en)
{
&req, sizeof(req), false);
}
-int mt7915_mcu_set_txbf_module(struct mt7915_dev *dev)
-{
-#define MT_BF_MODULE_UPDATE 25
- struct {
- u8 action;
- u8 bf_num;
- u8 bf_bitmap;
- u8 bf_sel[8];
- u8 rsv[8];
- } __packed req = {
- .action = MT_BF_MODULE_UPDATE,
- .bf_num = 2,
- .bf_bitmap = GENMASK(1, 0),
- };
-
- return mt76_mcu_send_msg(&dev->mt76, MCU_EXT_CMD(TXBF_ACTION), &req,
- sizeof(req), true);
-}
-
-int mt7915_mcu_set_txbf_type(struct mt7915_dev *dev)
+int mt7915_mcu_set_txbf(struct mt7915_dev *dev, u8 action)
{
-#define MT_BF_TYPE_UPDATE 20
struct {
u8 action;
- bool ebf;
- bool ibf;
- u8 rsv;
+ union {
+ struct {
+ u8 snd_mode;
+ u8 sta_num;
+ u8 rsv;
+ u8 wlan_idx[4];
+ __le32 snd_period; /* ms */
+ } __packed snd;
+ struct {
+ bool ebf;
+ bool ibf;
+ u8 rsv;
+ } __packed type;
+ struct {
+ u8 bf_num;
+ u8 bf_bitmap;
+ u8 bf_sel[8];
+ u8 rsv[5];
+ } __packed mod;
+ };
} __packed req = {
- .action = MT_BF_TYPE_UPDATE,
- .ebf = true,
- .ibf = dev->ibf,
+ .action = action,
};
- return mt76_mcu_send_msg(&dev->mt76, MCU_EXT_CMD(TXBF_ACTION), &req,
- sizeof(req), true);
-}
-
-int mt7915_mcu_set_txbf_sounding(struct mt7915_dev *dev)
-{
-#define MT_BF_PROCESSING 4
- struct {
- u8 action;
- u8 snd_mode;
- u8 sta_num;
- u8 rsv;
- u8 wlan_idx[4];
- __le32 snd_period; /* ms */
- } __packed req = {
- .action = true,
- .snd_mode = MT_BF_PROCESSING,
- };
+#define MT_BF_PROCESSING 4
+ switch (action) {
+ case MT_BF_SOUNDING_ON:
+ req.snd.snd_mode = MT_BF_PROCESSING;
+ break;
+ case MT_BF_TYPE_UPDATE:
+ req.type.ebf = true;
+ req.type.ibf = dev->ibf;
+ break;
+ case MT_BF_MODULE_UPDATE:
+ req.mod.bf_num = 2;
+ req.mod.bf_bitmap = GENMASK(1, 0);
+ break;
+ default:
+ return -EINVAL;
+ }
return mt76_mcu_send_msg(&dev->mt76, MCU_EXT_CMD(TXBF_ACTION), &req,
sizeof(req), true);
u8 s2d_index;
};
+struct mt7915_mcu_thermal_ctrl {
+ u8 ctrl_id;
+ u8 band_idx;
+ union {
+ struct {
+ u8 protect_type; /* 1: duty admit, 2: radio off */
+ u8 trigger_type; /* 0: low, 1: high */
+ } __packed type;
+ struct {
+ u8 duty_level; /* level 0~3 */
+ u8 duty_cycle;
+ } __packed duty;
+ };
+} __packed;
+
+struct mt7915_mcu_thermal_notify {
+ struct mt7915_mcu_rxd rxd;
+
+ struct mt7915_mcu_thermal_ctrl ctrl;
+ __le32 temperature;
+ u8 rsv[8];
+} __packed;
+
struct mt7915_mcu_csa_notify {
struct mt7915_mcu_rxd rxd;
#define MT_RA_RATE_DCM_EN BIT(4)
#define MT_RA_RATE_BW GENMASK(14, 13)
+struct mt7915_mcu_mib {
+ __le32 band;
+ __le32 offs;
+ __le64 data;
+} __packed;
+
+enum mt7915_chan_mib_offs {
+ MIB_BUSY_TIME = 14,
+ MIB_TX_TIME = 81,
+ MIB_RX_TIME,
+ MIB_OBSS_AIRTIME = 86
+};
+
struct edca {
u8 queue;
u8 set;
MCU_EXT_CMD_FW_LOG_2_HOST = 0x13,
MCU_EXT_CMD_TXBF_ACTION = 0x1e,
MCU_EXT_CMD_EFUSE_BUFFER_MODE = 0x21,
+ MCU_EXT_CMD_THERMAL_PROT = 0x23,
MCU_EXT_CMD_STA_REC_UPDATE = 0x25,
MCU_EXT_CMD_BSS_INFO_UPDATE = 0x26,
MCU_EXT_CMD_EDCA_UPDATE = 0x27,
MCU_EXT_CMD_MUAR_UPDATE = 0x48,
MCU_EXT_CMD_SET_RX_PATH = 0x4e,
MCU_EXT_CMD_TX_POWER_FEATURE_CTRL = 0x58,
+ MCU_EXT_CMD_GET_MIB_INFO = 0x5a,
MCU_EXT_CMD_MWDS_SUPPORT = 0x80,
MCU_EXT_CMD_SET_SER_TRIGGER = 0x81,
MCU_EXT_CMD_SCS_CTRL = 0x82,
u8 op_vht_rx_nss;
u8 op_vht_rx_nss_type;
- __le32 sta_status;
+ __le32 sta_cap;
struct ra_phy phy;
} __packed;
STA_REC_MAX_NUM
};
-enum mt7915_cipher_type {
- MT_CIPHER_NONE,
- MT_CIPHER_WEP40,
- MT_CIPHER_WEP104,
- MT_CIPHER_WEP128,
- MT_CIPHER_TKIP,
- MT_CIPHER_AES_CCMP,
- MT_CIPHER_CCMP_256,
- MT_CIPHER_GCMP,
- MT_CIPHER_GCMP_256,
- MT_CIPHER_WAPI,
- MT_CIPHER_BIP_CMAC_128,
+enum mcu_cipher_type {
+ MCU_CIPHER_WEP40 = 1,
+ MCU_CIPHER_WEP104,
+ MCU_CIPHER_WEP128,
+ MCU_CIPHER_TKIP,
+ MCU_CIPHER_AES_CCMP,
+ MCU_CIPHER_CCMP_256,
+ MCU_CIPHER_GCMP,
+ MCU_CIPHER_GCMP_256,
+ MCU_CIPHER_WAPI,
+ MCU_CIPHER_BIP_CMAC_128,
};
enum {
};
enum {
+ THERMAL_PROTECT_PARAMETER_CTRL,
+ THERMAL_PROTECT_BASIC_INFO,
+ THERMAL_PROTECT_ENABLE,
+ THERMAL_PROTECT_DISABLE,
+ THERMAL_PROTECT_DUTY_CONFIG,
+ THERMAL_PROTECT_MECH_INFO,
+ THERMAL_PROTECT_DUTY_INFO,
+ THERMAL_PROTECT_STATE_ACT,
+};
+
+enum {
MT_EBF = BIT(0), /* explicit beamforming */
MT_IBF = BIT(1) /* implicit beamforming */
};
+enum {
+ MT_BF_SOUNDING_ON = 1,
+ MT_BF_TYPE_UPDATE = 20,
+ MT_BF_MODULE_UPDATE = 25
+};
+
#define MT7915_WTBL_UPDATE_MAX_SIZE (sizeof(struct wtbl_req_hdr) + \
sizeof(struct wtbl_generic) + \
sizeof(struct wtbl_rx) + \
#include "../mt76.h"
#include "regs.h"
-#define MT7915_MAX_INTERFACES 32
+#define MT7915_MAX_INTERFACES 19
#define MT7915_MAX_WMM_SETS 4
#define MT7915_WTBL_SIZE 288
#define MT7915_WTBL_RESERVED (MT7915_WTBL_SIZE - 1)
#define MT7915_ROM_PATCH "mediatek/mt7915_rom_patch.bin"
#define MT7915_EEPROM_SIZE 3584
+#define MT7915_EEPROM_BLOCK_SIZE 16
#define MT7915_TOKEN_SIZE 8192
#define MT7915_CFEND_RATE_DEFAULT 0x49 /* OFDM 24M */
#define MT7915_5G_RATE_DEFAULT 0x4b /* OFDM 6M */
#define MT7915_2G_RATE_DEFAULT 0x0 /* CCK 1M */
+#define MT7915_THERMAL_THROTTLE_MAX 100
+
+#define MT7915_SKU_RATE_NUM 161
+
struct mt7915_vif;
struct mt7915_sta;
struct mt7915_dfs_pulse;
struct mt7915_phy *phy;
struct ieee80211_tx_queue_params queue_params[IEEE80211_NUM_ACS];
+ struct cfg80211_bitrate_mask bitrate_mask;
};
struct mib_stats {
struct ieee80211_vif *monitor_vif;
+ struct thermal_cooling_device *cdev;
+ u8 throttle_state;
+
u32 rxfilter;
u64 omac_mask;
u32 ampdu_ref;
struct mib_stats mib;
+ struct mt76_channel_state state_ts;
struct list_head stats_list;
u8 sta_work_count;
struct mt7915_hif *hif2;
const struct mt76_bus_ops *bus_ops;
+ struct tasklet_struct irq_tasklet;
struct mt7915_phy phy;
u16 chainmask;
bool enable);
int mt7915_mcu_add_rate_ctrl(struct mt7915_dev *dev, struct ieee80211_vif *vif,
struct ieee80211_sta *sta);
+int mt7915_mcu_add_he(struct mt7915_dev *dev, struct ieee80211_vif *vif,
+ struct ieee80211_sta *sta);
int mt7915_mcu_add_smps(struct mt7915_dev *dev, struct ieee80211_vif *vif,
struct ieee80211_sta *sta);
int mt7915_set_channel(struct mt7915_phy *phy);
int mt7915_mcu_set_pm(struct mt7915_dev *dev, int band, int enter);
int mt7915_mcu_set_sku_en(struct mt7915_phy *phy, bool enable);
int mt7915_mcu_set_txpower_sku(struct mt7915_phy *phy);
-int mt7915_mcu_set_txbf_type(struct mt7915_dev *dev);
-int mt7915_mcu_set_txbf_module(struct mt7915_dev *dev);
-int mt7915_mcu_set_txbf_sounding(struct mt7915_dev *dev);
+int mt7915_mcu_get_txpower_sku(struct mt7915_phy *phy, s8 *txpower, int len);
+int mt7915_mcu_set_txbf(struct mt7915_dev *dev, u8 action);
int mt7915_mcu_set_fcc5_lpn(struct mt7915_dev *dev, int val);
int mt7915_mcu_set_pulse_th(struct mt7915_dev *dev,
const struct mt7915_dfs_pulse *pulse);
const struct mt7915_dfs_pattern *pattern);
int mt7915_mcu_apply_group_cal(struct mt7915_dev *dev);
int mt7915_mcu_apply_tx_dpd(struct mt7915_phy *phy);
-int mt7915_mcu_get_temperature(struct mt7915_dev *dev, int index);
+int mt7915_mcu_get_chan_mib_info(struct mt7915_phy *phy, bool chan_switch);
+int mt7915_mcu_get_temperature(struct mt7915_phy *phy);
+int mt7915_mcu_set_thermal_throttling(struct mt7915_phy *phy, u8 state);
int mt7915_mcu_get_tx_rate(struct mt7915_dev *dev, u32 cmd, u16 wlan_idx);
int mt7915_mcu_get_rx_rate(struct mt7915_phy *phy, struct ieee80211_vif *vif,
struct ieee80211_sta *sta, struct rate_info *rate);
static inline void mt7915_irq_enable(struct mt7915_dev *dev, u32 mask)
{
if (dev->hif2)
- mt7915_dual_hif_set_irq_mask(dev, true, 0, mask);
+ mt7915_dual_hif_set_irq_mask(dev, false, 0, mask);
else
- mt76_set_irq_mask(&dev->mt76, MT_INT_MASK_CSR, 0, mask);
+ mt76_set_irq_mask(&dev->mt76, 0, 0, mask);
+
+ tasklet_schedule(&dev->irq_tasklet);
}
static inline void mt7915_irq_disable(struct mt7915_dev *dev, u32 mask)
void mt7915_mac_cca_stats_reset(struct mt7915_phy *phy);
void mt7915_mac_enable_nf(struct mt7915_dev *dev, bool ext_phy);
void mt7915_mac_write_txwi(struct mt7915_dev *dev, __le32 *txwi,
- struct sk_buff *skb, struct mt76_wcid *wcid,
+ struct sk_buff *skb, struct mt76_wcid *wcid, int pid,
struct ieee80211_key_conf *key, bool beacon);
void mt7915_mac_set_timing(struct mt7915_phy *phy);
-int mt7915_mac_fill_rx(struct mt7915_dev *dev, struct sk_buff *skb);
-void mt7915_mac_fill_rx_vector(struct mt7915_dev *dev, struct sk_buff *skb);
-void mt7915_mac_tx_free(struct mt7915_dev *dev, struct sk_buff *skb);
int mt7915_mac_sta_add(struct mt76_dev *mdev, struct ieee80211_vif *vif,
struct ieee80211_sta *sta);
void mt7915_mac_sta_remove(struct mt76_dev *mdev, struct ieee80211_vif *vif,
struct sk_buff *skb);
void mt7915_sta_ps(struct mt76_dev *mdev, struct ieee80211_sta *sta, bool ps);
void mt7915_stats_work(struct work_struct *work);
-void mt7915_txp_skb_unmap(struct mt76_dev *dev,
- struct mt76_txwi_cache *txwi);
int mt76_dfs_start_rdd(struct mt7915_dev *dev, bool force);
int mt7915_dfs_init_radar_detector(struct mt7915_phy *phy);
void mt7915_set_stream_he_caps(struct mt7915_phy *phy);
void mt7915_set_stream_vht_txbf_caps(struct mt7915_phy *phy);
-void mt7915_update_channel(struct mt76_dev *mdev);
+void mt7915_update_channel(struct mt76_phy *mphy);
int mt7915_init_debugfs(struct mt7915_dev *dev);
#ifdef CONFIG_MAC80211_DEBUGFS
void mt7915_sta_add_debugfs(struct ieee80211_hw *hw, struct ieee80211_vif *vif,
}
/* TODO: support 2/4/6/8 MSI-X vectors */
-static irqreturn_t mt7915_irq_handler(int irq, void *dev_instance)
+static void mt7915_irq_tasklet(struct tasklet_struct *t)
{
- struct mt7915_dev *dev = dev_instance;
+ struct mt7915_dev *dev = from_tasklet(dev, t, irq_tasklet);
u32 intr, intr1, mask;
+ mt76_wr(dev, MT_INT_MASK_CSR, 0);
+ if (dev->hif2)
+ mt76_wr(dev, MT_INT1_MASK_CSR, 0);
+
intr = mt76_rr(dev, MT_INT_SOURCE_CSR);
intr &= dev->mt76.mmio.irqmask;
mt76_wr(dev, MT_INT_SOURCE_CSR, intr);
intr |= intr1;
}
- if (!test_bit(MT76_STATE_INITIALIZED, &dev->mphy.state))
- return IRQ_NONE;
-
trace_dev_irq(&dev->mt76, intr, dev->mt76.mmio.irqmask);
mask = intr & MT_INT_RX_DONE_ALL;
wake_up(&dev->reset_wait);
}
}
+}
+
+static irqreturn_t mt7915_irq_handler(int irq, void *dev_instance)
+{
+ struct mt7915_dev *dev = dev_instance;
+
+ mt76_wr(dev, MT_INT_MASK_CSR, 0);
+ if (dev->hif2)
+ mt76_wr(dev, MT_INT1_MASK_CSR, 0);
+
+ if (!test_bit(MT76_STATE_INITIALIZED, &dev->mphy.state))
+ return IRQ_NONE;
+
+ tasklet_schedule(&dev->irq_tasklet);
return IRQ_HANDLED;
}
if (ret)
return ret;
+ mt76_pci_disable_aspm(pdev);
+
if (id->device == 0x7916)
return mt7915_pci_hif2_probe(pdev);
dev = container_of(mdev, struct mt7915_dev, mt76);
+ ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_ALL_TYPES);
+ if (ret < 0)
+ goto free;
+
ret = mt7915_mmio_init(mdev, pcim_iomap_table(pdev)[0], pdev->irq);
if (ret)
goto error;
+ tasklet_setup(&dev->irq_tasklet, mt7915_irq_tasklet);
+
+ mt76_wr(dev, MT_INT_MASK_CSR, 0);
+
/* master switch of PCIe tnterrupt enable */
mt76_wr(dev, MT_PCIE_MAC_INT_ENABLE, 0xff);
ret = mt7915_register_device(dev);
if (ret)
- goto error;
+ goto free_irq;
return 0;
+free_irq:
+ devm_free_irq(mdev->dev, pdev->irq, dev);
error:
+ pci_free_irq_vectors(pdev);
+free:
mt76_free_device(&dev->mt76);
return ret;
#define MT_TMAC_CTCR0_INS_DDLMT_EN BIT(17)
#define MT_TMAC_CTCR0_INS_DDLMT_VHT_SMPDU_EN BIT(18)
-#define MT_TMAC_FP0R0(_band) MT_WF_TMAC(_band, 0x020)
-#define MT_TMAC_FP0R15(_band) MT_WF_TMAC(_band, 0x080)
-#define MT_TMAC_FP0R18(_band) MT_WF_TMAC(_band, 0x270)
-#define MT_TMAC_FP_MASK GENMASK(7, 0)
-
#define MT_TMAC_TFCR0(_band) MT_WF_TMAC(_band, 0x1e0)
#define MT_WF_DMA_BASE(_band) ((_band) ? 0xa1e00 : 0x21e00)
#define MT_ETBF_TX_FB_CPL GENMASK(31, 16)
#define MT_ETBF_TX_FB_TRI GENMASK(15, 0)
+#define MT_ETBF_RX_FB_CONT(_band) MT_WF_ETBF(_band, 0x068)
+#define MT_ETBF_RX_FB_BW GENMASK(7, 6)
+#define MT_ETBF_RX_FB_NC GENMASK(5, 3)
+#define MT_ETBF_RX_FB_NR GENMASK(2, 0)
+
#define MT_ETBF_TX_APP_CNT(_band) MT_WF_ETBF(_band, 0x0f0)
#define MT_ETBF_TX_IBF_CNT GENMASK(31, 16)
#define MT_ETBF_TX_EBF_CNT GENMASK(15, 0)
#define MT_LPON_TCR(_band, n) MT_WF_LPON(_band, 0x0a8 + (n) * 4)
#define MT_LPON_TCR_SW_MODE GENMASK(1, 0)
#define MT_LPON_TCR_SW_WRITE BIT(0)
+#define MT_LPON_TCR_SW_ADJUST BIT(1)
+#define MT_LPON_TCR_SW_READ GENMASK(1, 0)
/* MIB: band 0(0x24800), band 1(0xa4800) */
#define MT_WF_MIB_BASE(_band) ((_band) ? 0xa4800 : 0x24800)
#define MT_MIB_SDR3(_band) MT_WF_MIB(_band, 0x014)
#define MT_MIB_SDR3_FCS_ERR_MASK GENMASK(15, 0)
-#define MT_MIB_SDR9(_band) MT_WF_MIB(_band, 0x02c)
-#define MT_MIB_SDR9_BUSY_MASK GENMASK(23, 0)
-
-#define MT_MIB_SDR16(_band) MT_WF_MIB(_band, 0x048)
-#define MT_MIB_SDR16_BUSY_MASK GENMASK(23, 0)
-
#define MT_MIB_SDR34(_band) MT_WF_MIB(_band, 0x090)
#define MT_MIB_MU_BF_TX_CNT GENMASK(15, 0)
-#define MT_MIB_SDR36(_band) MT_WF_MIB(_band, 0x098)
-#define MT_MIB_SDR36_TXTIME_MASK GENMASK(23, 0)
-#define MT_MIB_SDR37(_band) MT_WF_MIB(_band, 0x09c)
-#define MT_MIB_SDR37_RXTIME_MASK GENMASK(23, 0)
-
#define MT_MIB_DR8(_band) MT_WF_MIB(_band, 0x0c0)
#define MT_MIB_DR9(_band) MT_WF_MIB(_band, 0x0c4)
#define MT_MIB_DR11(_band) MT_WF_MIB(_band, 0x0cc)
#define MT_MIB_BA_MISS_COUNT_MASK GENMASK(15, 0)
#define MT_MIB_ACK_FAIL_COUNT_MASK GENMASK(31, 16)
-#define MT_MIB_MB_SDR2(_band, n) MT_WF_MIB(_band, 0x108 + ((n) << 4))
-#define MT_MIB_FRAME_RETRIES_COUNT_MASK GENMASK(15, 0)
-
#define MT_TX_AGG_CNT(_band, n) MT_WF_MIB(_band, 0x0a8 + ((n) << 2))
#define MT_TX_AGG_CNT2(_band, n) MT_WF_MIB(_band, 0x164 + ((n) << 2))
#define MT_MIB_ARNG(_band, n) MT_WF_MIB(_band, 0x4b8 + ((n) << 2))
#define MT_WF_RFCR1_DROP_CFEND BIT(7)
#define MT_WF_RFCR1_DROP_CFACK BIT(8)
-#define MT_WF_RMAC_MIB_TIME0(_band) MT_WF_RMAC(_band, 0x03c4)
+#define MT_WF_RMAC_MIB_AIRTIME0(_band) MT_WF_RMAC(_band, 0x0380)
#define MT_WF_RMAC_MIB_RXTIME_CLR BIT(31)
#define MT_WF_RMAC_MIB_RXTIME_EN BIT(30)
-#define MT_WF_RMAC_MIB_AIRTIME14(_band) MT_WF_RMAC(_band, 0x03b8)
-#define MT_MIB_OBSSTIME_MASK GENMASK(23, 0)
-#define MT_WF_RMAC_MIB_AIRTIME0(_band) MT_WF_RMAC(_band, 0x0380)
-
/* WFDMA0 */
#define MT_WFDMA0_BASE 0xd4000
#define MT_WFDMA0(ofs) (MT_WFDMA0_BASE + (ofs))
static void
mt7915_tm_set_rx_frames(struct mt7915_phy *phy, bool en)
{
- if (en)
+ mt7915_tm_set_trx(phy, TM_MAC_RX_RXV, false);
+
+ if (en) {
+ struct mt7915_dev *dev = phy->dev;
+
mt7915_tm_update_channel(phy);
- mt7915_tm_set_trx(phy, TM_MAC_RX_RXV, en);
+ /* read-clear */
+ mt76_rr(dev, MT_MIB_SDR3(phy != &dev->phy));
+ mt7915_tm_set_trx(phy, TM_MAC_RX_RXV, en);
+ }
}
static int
mt7915_tm_dump_stats(struct mt76_phy *mphy, struct sk_buff *msg)
{
struct mt7915_phy *phy = mphy->priv;
+ struct mt7915_dev *dev = phy->dev;
+ bool ext_phy = phy != &dev->phy;
+ enum mt76_rxq_id q;
void *rx, *rssi;
+ u16 fcs_err;
int i;
rx = nla_nest_start(msg, MT76_TM_STATS_ATTR_LAST_RX);
nla_nest_end(msg, rx);
+ fcs_err = mt76_get_field(dev, MT_MIB_SDR3(ext_phy),
+ MT_MIB_SDR3_FCS_ERR_MASK);
+ q = ext_phy ? MT_RXQ_EXT : MT_RXQ_MAIN;
+ mphy->test.rx_stats.packets[q] += fcs_err;
+ mphy->test.rx_stats.fcs_error[q] += fcs_err;
+
return 0;
}
-// SPDX-License-Identifier: ISC
+/* SPDX-License-Identifier: ISC */
/* Copyright (C) 2020 MediaTek Inc. */
#ifndef __MT7915_TESTMODE_H
-#SPDX-License-Identifier: ISC
+# SPDX-License-Identifier: ISC
obj-$(CONFIG_MT7921E) += mt7921e.o
struct mt7921_txpwr txpwr;
int ret;
+ mt7921_mutex_acquire(dev);
ret = mt7921_get_txpwr_info(dev, &txpwr);
+ mt7921_mutex_release(dev);
+
if (ret)
return ret;
ieee80211_iterate_active_interfaces(mphy->hw,
IEEE80211_IFACE_ITER_RESUME_ALL,
mt7921_pm_interface_iter, mphy->priv);
+
+ mt76_connac_mcu_set_deep_sleep(&dev->mt76, pm->ds_enable);
+
mt7921_mutex_release(dev);
return 0;
DEFINE_DEBUGFS_ATTRIBUTE(fops_pm, mt7921_pm_get, mt7921_pm_set, "%lld\n");
static int
+mt7921_deep_sleep_set(void *data, u64 val)
+{
+ struct mt7921_dev *dev = data;
+ struct mt76_connac_pm *pm = &dev->pm;
+ bool enable = !!val;
+
+ mt7921_mutex_acquire(dev);
+ if (pm->ds_enable != enable) {
+ mt76_connac_mcu_set_deep_sleep(&dev->mt76, enable);
+ pm->ds_enable = enable;
+ }
+ mt7921_mutex_release(dev);
+
+ return 0;
+}
+
+static int
+mt7921_deep_sleep_get(void *data, u64 *val)
+{
+ struct mt7921_dev *dev = data;
+
+ *val = dev->pm.ds_enable;
+
+ return 0;
+}
+
+DEFINE_DEBUGFS_ATTRIBUTE(fops_ds, mt7921_deep_sleep_get,
+ mt7921_deep_sleep_set, "%lld\n");
+
+static int
mt7921_pm_stats(struct seq_file *s, void *data)
{
struct mt7921_dev *dev = dev_get_drvdata(s->private);
debugfs_create_file("chip_reset", 0600, dir, dev, &fops_reset);
debugfs_create_devm_seqfile(dev->mt76.dev, "runtime_pm_stats", dir,
mt7921_pm_stats);
+ debugfs_create_file("deep-sleep", 0600, dir, dev, &fops_ds);
return 0;
}
mt7921_tx_cleanup(dev);
if (napi_complete(napi))
mt7921_irq_enable(dev, MT_INT_TX_DONE_ALL);
- mt76_connac_pm_unref(&dev->pm);
+ mt76_connac_pm_unref(&dev->mphy, &dev->pm);
return 0;
}
return 0;
}
done = mt76_dma_rx_poll(napi, budget);
- mt76_connac_pm_unref(&dev->pm);
+ mt76_connac_pm_unref(&dev->mphy, &dev->pm);
return done;
}
int mt7921_wfsys_reset(struct mt7921_dev *dev)
{
- mt76_set(dev, 0x70002600, BIT(0));
- msleep(200);
- mt76_clear(dev, 0x70002600, BIT(0));
+ mt76_clear(dev, MT_WFSYS_SW_RST_B, WFSYS_SW_RST_B);
+ msleep(50);
+ mt76_set(dev, MT_WFSYS_SW_RST_B, WFSYS_SW_RST_B);
if (!__mt76_poll_msec(&dev->mt76, MT_WFSYS_SW_RST_B,
WFSYS_SW_INIT_DONE, WFSYS_SW_INIT_DONE, 500))
int mt7921_dma_init(struct mt7921_dev *dev)
{
- /* Increase buffer size to receive large VHT/HE MPDUs */
struct mt76_bus_ops *bus_ops;
- int rx_buf_size = MT_RX_BUF_SIZE * 2;
int ret;
dev->bus_ops = dev->mt76.bus;
if (ret)
return ret;
+ ret = mt7921_wfsys_reset(dev);
+ if (ret)
+ return ret;
+
/* init tx queue */
ret = mt7921_init_tx_queues(&dev->phy, MT7921_TXQ_BAND0,
MT7921_TX_RING_SIZE);
ret = mt76_queue_alloc(dev, &dev->mt76.q_rx[MT_RXQ_MCU],
MT7921_RXQ_MCU_WM,
MT7921_RX_MCU_RING_SIZE,
- rx_buf_size, MT_RX_EVENT_RING_BASE);
+ MT_RX_BUF_SIZE, MT_RX_EVENT_RING_BASE);
if (ret)
return ret;
ret = mt76_queue_alloc(dev, &dev->mt76.q_rx[MT_RXQ_MCU_WA],
MT7921_RXQ_MCU_WM,
MT7921_RX_MCU_RING_SIZE,
- rx_buf_size, MT_WFDMA0(0x540));
+ MT_RX_BUF_SIZE, MT_WFDMA0(0x540));
if (ret)
return ret;
/* rx data */
ret = mt76_queue_alloc(dev, &dev->mt76.q_rx[MT_RXQ_MAIN],
MT7921_RXQ_BAND0, MT7921_RX_RING_SIZE,
- rx_buf_size, MT_RX_DATA_RING_BASE);
+ MT_RX_BUF_SIZE, MT_RX_DATA_RING_BASE);
if (ret)
return ret;
#include "mcu.h"
#include "eeprom.h"
-#define CCK_RATE(_idx, _rate) { \
- .bitrate = _rate, \
- .flags = IEEE80211_RATE_SHORT_PREAMBLE, \
- .hw_value = (MT_PHY_TYPE_CCK << 8) | (_idx), \
- .hw_value_short = (MT_PHY_TYPE_CCK << 8) | (4 + (_idx)), \
-}
-
-#define OFDM_RATE(_idx, _rate) { \
- .bitrate = _rate, \
- .hw_value = (MT_PHY_TYPE_OFDM << 8) | (_idx), \
- .hw_value_short = (MT_PHY_TYPE_OFDM << 8) | (_idx), \
-}
-
-static struct ieee80211_rate mt7921_rates[] = {
- CCK_RATE(0, 10),
- CCK_RATE(1, 20),
- CCK_RATE(2, 55),
- CCK_RATE(3, 110),
- OFDM_RATE(11, 60),
- OFDM_RATE(15, 90),
- OFDM_RATE(10, 120),
- OFDM_RATE(14, 180),
- OFDM_RATE(9, 240),
- OFDM_RATE(13, 360),
- OFDM_RATE(8, 480),
- OFDM_RATE(12, 540),
-};
-
static const struct ieee80211_iface_limit if_limits[] = {
{
.max = MT7921_MAX_INTERFACES,
mt7921_init_wiphy(struct ieee80211_hw *hw)
{
struct mt7921_phy *phy = mt7921_hw_phy(hw);
+ struct mt7921_dev *dev = phy->dev;
struct wiphy *wiphy = hw->wiphy;
hw->queues = 4;
- hw->max_rx_aggregation_subframes = IEEE80211_MAX_AMPDU_BUF;
- hw->max_tx_aggregation_subframes = IEEE80211_MAX_AMPDU_BUF;
+ hw->max_rx_aggregation_subframes = 64;
+ hw->max_tx_aggregation_subframes = 128;
+ hw->netdev_features = NETIF_F_RXCSUM;
hw->radiotap_timestamp.units_pos =
IEEE80211_RADIOTAP_TIMESTAMP_UNIT_US;
hw->vif_data_size = sizeof(struct mt7921_vif);
wiphy->iface_combinations = if_comb;
+ wiphy->flags &= ~WIPHY_FLAG_IBSS_RSN;
+ wiphy->interface_modes = BIT(NL80211_IFTYPE_STATION);
wiphy->n_iface_combinations = ARRAY_SIZE(if_comb);
wiphy->max_scan_ie_len = MT76_CONNAC_SCAN_IE_LEN;
wiphy->max_scan_ssids = 4;
wiphy->max_sched_scan_plan_interval =
- MT76_CONNAC_MAX_SCHED_SCAN_INTERVAL;
+ MT76_CONNAC_MAX_TIME_SCHED_SCAN_INTERVAL;
wiphy->max_sched_scan_ie_len = IEEE80211_MAX_DATA_LEN;
wiphy->max_sched_scan_ssids = MT76_CONNAC_MAX_SCHED_SCAN_SSID;
wiphy->max_match_sets = MT76_CONNAC_MAX_SCAN_MATCH;
wiphy->flags |= WIPHY_FLAG_HAS_CHANNEL_SWITCH;
wiphy->reg_notifier = mt7921_regd_notifier;
- wiphy->features |= NL80211_FEATURE_SCAN_RANDOM_MAC_ADDR;
+ wiphy->features |= NL80211_FEATURE_SCHED_SCAN_RANDOM_MAC_ADDR |
+ NL80211_FEATURE_SCAN_RANDOM_MAC_ADDR;
wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_SET_SCAN_DWELL);
ieee80211_hw_set(hw, SINGLE_SCAN_ON_ALL_BANDS);
ieee80211_hw_set(hw, HAS_RATE_CONTROL);
ieee80211_hw_set(hw, SUPPORTS_TX_ENCAP_OFFLOAD);
+ ieee80211_hw_set(hw, SUPPORTS_RX_DECAP_OFFLOAD);
ieee80211_hw_set(hw, WANT_MONITOR_VIF);
ieee80211_hw_set(hw, SUPPORTS_PS);
ieee80211_hw_set(hw, SUPPORTS_DYNAMIC_PS);
+ if (dev->pm.enable)
+ ieee80211_hw_set(hw, CONNECTION_MONITOR);
+
hw->max_tx_fragments = 4;
}
static void
mt7921_mac_init_band(struct mt7921_dev *dev, u8 band)
{
- u32 mask, set;
-
mt76_rmw_field(dev, MT_TMAC_CTCR0(band),
MT_TMAC_CTCR0_INS_DDLMT_REFTIME, 0x3f);
mt76_set(dev, MT_TMAC_CTCR0(band),
MT_TMAC_CTCR0_INS_DDLMT_VHT_SMPDU_EN |
MT_TMAC_CTCR0_INS_DDLMT_EN);
- mask = MT_MDP_RCFR0_MCU_RX_MGMT |
- MT_MDP_RCFR0_MCU_RX_CTL_NON_BAR |
- MT_MDP_RCFR0_MCU_RX_CTL_BAR;
- set = FIELD_PREP(MT_MDP_RCFR0_MCU_RX_MGMT, MT_MDP_TO_HIF) |
- FIELD_PREP(MT_MDP_RCFR0_MCU_RX_CTL_NON_BAR, MT_MDP_TO_HIF) |
- FIELD_PREP(MT_MDP_RCFR0_MCU_RX_CTL_BAR, MT_MDP_TO_HIF);
- mt76_rmw(dev, MT_MDP_BNRCFR0(band), mask, set);
-
- mask = MT_MDP_RCFR1_MCU_RX_BYPASS |
- MT_MDP_RCFR1_RX_DROPPED_UCAST |
- MT_MDP_RCFR1_RX_DROPPED_MCAST;
- set = FIELD_PREP(MT_MDP_RCFR1_MCU_RX_BYPASS, MT_MDP_TO_HIF) |
- FIELD_PREP(MT_MDP_RCFR1_RX_DROPPED_UCAST, MT_MDP_TO_HIF) |
- FIELD_PREP(MT_MDP_RCFR1_RX_DROPPED_MCAST, MT_MDP_TO_HIF);
- mt76_rmw(dev, MT_MDP_BNRCFR1(band), mask, set);
-
mt76_set(dev, MT_WF_RMAC_MIB_TIME0(band), MT_WF_RMAC_MIB_RXTIME_EN);
mt76_set(dev, MT_WF_RMAC_MIB_AIRTIME0(band), MT_WF_RMAC_MIB_RXTIME_EN);
mt76_clear(dev, MT_DMA_DCR0(band), MT_DMA_DCR0_RXD_G5_EN);
}
-void mt7921_mac_init(struct mt7921_dev *dev)
+int mt7921_mac_init(struct mt7921_dev *dev)
{
int i;
mt76_rmw_field(dev, MT_MDP_DCR1, MT_MDP_DCR1_MAX_RX_LEN, 1536);
- /* disable hardware de-agg */
- mt76_clear(dev, MT_MDP_DCR0, MT_MDP_DCR0_DAMSDU_EN);
- mt76_clear(dev, MT_MDP_DCR0, MT_MDP_DCR0_RX_HDR_TRANS_EN);
+ /* enable hardware de-agg */
+ mt76_set(dev, MT_MDP_DCR0, MT_MDP_DCR0_DAMSDU_EN);
+ /* enable hardware rx header translation */
+ mt76_set(dev, MT_MDP_DCR0, MT_MDP_DCR0_RX_HDR_TRANS_EN);
for (i = 0; i < MT7921_WTBL_SIZE; i++)
mt7921_mac_wtbl_update(dev, i,
for (i = 0; i < 2; i++)
mt7921_mac_init_band(dev, i);
- mt76_connac_mcu_set_rts_thresh(&dev->mt76, 0x92b, 0);
+ return mt76_connac_mcu_set_rts_thresh(&dev->mt76, 0x92b, 0);
}
static int mt7921_init_hardware(struct mt7921_dev *dev)
dev->mt76.global_wcid.tx_info |= MT_WCID_TX_INFO_SET;
rcu_assign_pointer(dev->mt76.wcid[idx], &dev->mt76.global_wcid);
- mt7921_mac_init(dev);
-
- return 0;
+ return mt7921_mac_init(dev);
}
int mt7921_register_device(struct mt7921_dev *dev)
mutex_init(&dev->pm.mutex);
init_waitqueue_head(&dev->pm.wait);
spin_lock_init(&dev->pm.txq_lock);
- set_bit(MT76_STATE_PM, &dev->mphy.state);
INIT_LIST_HEAD(&dev->phy.stats_list);
INIT_DELAYED_WORK(&dev->mphy.mac_work, mt7921_mac_work);
INIT_DELAYED_WORK(&dev->phy.scan_work, mt7921_scan_work);
dev->pm.idle_timeout = MT7921_PM_TIMEOUT;
dev->pm.stats.last_wake_event = jiffies;
dev->pm.stats.last_doze_event = jiffies;
+ dev->pm.enable = true;
+ dev->pm.ds_enable = true;
ret = mt7921_init_hardware(dev);
if (ret)
IEEE80211_HT_CAP_MAX_AMSDU;
dev->mphy.sband_5g.sband.vht_cap.cap |=
IEEE80211_VHT_CAP_MAX_MPDU_LENGTH_7991 |
- IEEE80211_VHT_CAP_MAX_A_MPDU_LENGTH_EXPONENT_MASK;
+ IEEE80211_VHT_CAP_MAX_A_MPDU_LENGTH_EXPONENT_MASK |
+ IEEE80211_VHT_CAP_SU_BEAMFORMEE_CAPABLE |
+ IEEE80211_VHT_CAP_MU_BEAMFORMEE_CAPABLE |
+ (3 << IEEE80211_VHT_CAP_BEAMFORMEE_STS_SHIFT);
+
dev->mphy.hw->wiphy->available_antennas_rx = dev->mphy.chainmask;
dev->mphy.hw->wiphy->available_antennas_tx = dev->mphy.chainmask;
mt76_set_stream_caps(&dev->mphy, true);
mt7921_set_stream_he_caps(&dev->phy);
- ret = mt76_register_device(&dev->mt76, true, mt7921_rates,
- ARRAY_SIZE(mt7921_rates));
+ ret = mt76_register_device(&dev->mt76, true, mt76_rates,
+ ARRAY_SIZE(mt76_rates));
+ if (ret)
+ return ret;
+
+ ret = mt7921_init_debugfs(dev);
if (ret)
return ret;
- return mt7921_init_debugfs(dev);
+ ret = mt76_connac_mcu_set_deep_sleep(&dev->mt76, dev->pm.ds_enable);
+ if (ret)
+ return ret;
+
+ dev->hw_init_done = true;
+
+ return 0;
}
void mt7921_unregister_device(struct mt7921_dev *dev)
int mt7921_mac_fill_rx(struct mt7921_dev *dev, struct sk_buff *skb)
{
+ u32 csum_mask = MT_RXD0_NORMAL_IP_SUM | MT_RXD0_NORMAL_UDP_TCP_SUM;
struct mt76_rx_status *status = (struct mt76_rx_status *)skb->cb;
+ bool hdr_trans, unicast, insert_ccmp_hdr = false;
+ u8 chfreq, qos_ctl = 0, remove_pad, amsdu_info;
+ __le32 *rxv = NULL, *rxd = (__le32 *)skb->data;
struct mt76_phy *mphy = &dev->mt76.phy;
struct mt7921_phy *phy = &dev->phy;
struct ieee80211_supported_band *sband;
struct ieee80211_hdr *hdr;
- __le32 *rxd = (__le32 *)skb->data;
- __le32 *rxv = NULL;
- u32 mode = 0;
+ u32 rxd0 = le32_to_cpu(rxd[0]);
u32 rxd1 = le32_to_cpu(rxd[1]);
u32 rxd2 = le32_to_cpu(rxd[2]);
u32 rxd3 = le32_to_cpu(rxd[3]);
- bool unicast, insert_ccmp_hdr = false;
- u8 remove_pad;
+ u32 rxd4 = le32_to_cpu(rxd[4]);
+ u16 seq_ctrl = 0;
+ __le16 fc = 0;
+ u32 mode = 0;
int i, idx;
- u8 chfreq;
memset(status, 0, sizeof(*status));
if (!test_bit(MT76_STATE_RUNNING, &mphy->state))
return -EINVAL;
+ if (rxd2 & MT_RXD2_NORMAL_AMSDU_ERR)
+ return -EINVAL;
+
chfreq = FIELD_GET(MT_RXD3_NORMAL_CH_FREQ, rxd3);
unicast = FIELD_GET(MT_RXD3_NORMAL_ADDR_TYPE, rxd3) == MT_RXD3_NORMAL_U2M;
idx = FIELD_GET(MT_RXD1_NORMAL_WLAN_IDX, rxd1);
+ hdr_trans = rxd2 & MT_RXD2_NORMAL_HDR_TRANS;
status->wcid = mt7921_rx_get_wcid(dev, idx, unicast);
if (status->wcid) {
if (!sband->channels)
return -EINVAL;
+ if ((rxd0 & csum_mask) == csum_mask)
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
+
if (rxd1 & MT_RXD1_NORMAL_FCS_ERR)
status->flag |= RX_FLAG_FAILED_FCS_CRC;
rxd += 6;
if (rxd1 & MT_RXD1_NORMAL_GROUP_4) {
+ u32 v0 = le32_to_cpu(rxd[0]);
+ u32 v2 = le32_to_cpu(rxd[2]);
+
+ fc = cpu_to_le16(FIELD_GET(MT_RXD6_FRAME_CONTROL, v0));
+ seq_ctrl = FIELD_GET(MT_RXD8_SEQ_CTRL, v2);
+ qos_ctl = FIELD_GET(MT_RXD8_QOS_CTL, v2);
+
rxd += 4;
if ((u8 *)rxd - skb->data >= skb->len)
return -EINVAL;
u8 *data = (u8 *)rxd;
if (status->flag & RX_FLAG_DECRYPTED) {
- status->iv[0] = data[5];
- status->iv[1] = data[4];
- status->iv[2] = data[3];
- status->iv[3] = data[2];
- status->iv[4] = data[1];
- status->iv[5] = data[0];
-
- insert_ccmp_hdr = FIELD_GET(MT_RXD2_NORMAL_FRAG, rxd2);
+ switch (FIELD_GET(MT_RXD1_NORMAL_SEC_MODE, rxd1)) {
+ case MT_CIPHER_AES_CCMP:
+ case MT_CIPHER_CCMP_CCX:
+ case MT_CIPHER_CCMP_256:
+ insert_ccmp_hdr =
+ FIELD_GET(MT_RXD2_NORMAL_FRAG, rxd2);
+ fallthrough;
+ case MT_CIPHER_TKIP:
+ case MT_CIPHER_TKIP_NO_MIC:
+ case MT_CIPHER_GCMP:
+ case MT_CIPHER_GCMP_256:
+ status->iv[0] = data[5];
+ status->iv[1] = data[4];
+ status->iv[2] = data[3];
+ status->iv[3] = data[2];
+ status->iv[4] = data[1];
+ status->iv[5] = data[0];
+ break;
+ default:
+ break;
+ }
}
rxd += 4;
if ((u8 *)rxd - skb->data >= skb->len)
status->chain_signal[1] = to_rssi(MT_PRXV_RCPI1, v1);
status->chain_signal[2] = to_rssi(MT_PRXV_RCPI2, v1);
status->chain_signal[3] = to_rssi(MT_PRXV_RCPI3, v1);
- status->signal = status->chain_signal[0];
-
- for (i = 1; i < hweight8(mphy->antenna_mask); i++) {
- if (!(status->chains & BIT(i)))
+ status->signal = -128;
+ for (i = 0; i < hweight8(mphy->antenna_mask); i++) {
+ if (!(status->chains & BIT(i)) ||
+ status->chain_signal[i] >= 0)
continue;
status->signal = max(status->signal,
status->chain_signal[i]);
}
+ if (status->signal == -128)
+ status->flag |= RX_FLAG_NO_SIGNAL_VAL;
+
stbc = FIELD_GET(MT_PRXV_STBC, v0);
gi = FIELD_GET(MT_PRXV_SGI, v0);
cck = false;
skb_pull(skb, (u8 *)rxd - skb->data + 2 * remove_pad);
- if (insert_ccmp_hdr) {
- u8 key_id = FIELD_GET(MT_RXD1_NORMAL_KEY_ID, rxd1);
+ amsdu_info = FIELD_GET(MT_RXD4_NORMAL_PAYLOAD_FORMAT, rxd4);
+ status->amsdu = !!amsdu_info;
+ if (status->amsdu) {
+ status->first_amsdu = amsdu_info == MT_RXD4_FIRST_AMSDU_FRAME;
+ status->last_amsdu = amsdu_info == MT_RXD4_LAST_AMSDU_FRAME;
+ if (!hdr_trans) {
+ memmove(skb->data + 2, skb->data,
+ ieee80211_get_hdrlen_from_skb(skb));
+ skb_pull(skb, 2);
+ }
+ }
+
+ if (!hdr_trans) {
+ if (insert_ccmp_hdr) {
+ u8 key_id = FIELD_GET(MT_RXD1_NORMAL_KEY_ID, rxd1);
+
+ mt76_insert_ccmp_hdr(skb, key_id);
+ }
- mt76_insert_ccmp_hdr(skb, key_id);
+ hdr = mt76_skb_get_hdr(skb);
+ fc = hdr->frame_control;
+ if (ieee80211_is_data_qos(fc)) {
+ seq_ctrl = le16_to_cpu(hdr->seq_ctrl);
+ qos_ctl = *ieee80211_get_qos_ctl(hdr);
+ }
+ } else {
+ status->flag &= ~(RX_FLAG_RADIOTAP_HE |
+ RX_FLAG_RADIOTAP_HE_MU);
+ status->flag |= RX_FLAG_8023;
}
mt7921_mac_assoc_rssi(dev, skb);
if (rxv && status->flag & RX_FLAG_RADIOTAP_HE)
mt7921_mac_decode_he_radiotap(skb, status, rxv, mode);
- hdr = mt76_skb_get_hdr(skb);
- if (!status->wcid || !ieee80211_is_data_qos(hdr->frame_control))
+ if (!status->wcid || !ieee80211_is_data_qos(fc))
return 0;
- status->aggr = unicast &&
- !ieee80211_is_qos_nullfunc(hdr->frame_control);
- status->qos_ctl = *ieee80211_get_qos_ctl(hdr);
- status->seqno = IEEE80211_SEQ_TO_SN(le16_to_cpu(hdr->seq_ctrl));
+ status->aggr = unicast && !ieee80211_is_qos_nullfunc(fc);
+ status->seqno = IEEE80211_SEQ_TO_SN(seq_ctrl);
+ status->qos_ctl = qos_ctl;
return 0;
}
txwi[7] |= cpu_to_le32(val);
}
+static void mt7921_update_txs(struct mt76_wcid *wcid, __le32 *txwi)
+{
+ struct mt7921_sta *msta = container_of(wcid, struct mt7921_sta, wcid);
+ u32 pid, frame_type = FIELD_GET(MT_TXD2_FRAME_TYPE, txwi[2]);
+
+ if (!(frame_type & (IEEE80211_FTYPE_DATA >> 2)))
+ return;
+
+ if (time_is_after_eq_jiffies(msta->next_txs_ts))
+ return;
+
+ msta->next_txs_ts = jiffies + msecs_to_jiffies(250);
+ pid = mt76_get_next_pkt_id(wcid);
+ txwi[5] |= cpu_to_le32(MT_TXD5_TX_STATUS_MCU |
+ FIELD_PREP(MT_TXD5_PID, pid));
+}
+
void mt7921_mac_write_txwi(struct mt7921_dev *dev, __le32 *txwi,
struct sk_buff *skb, struct mt76_wcid *wcid,
struct ieee80211_key_conf *key, bool beacon)
txwi[6] |= cpu_to_le32(val);
txwi[3] |= cpu_to_le32(MT_TXD3_BA_DISABLE);
}
+
+ mt7921_update_txs(wcid, txwi);
}
static void
state->noise = -(phy->noise >> 4);
}
-void mt7921_update_channel(struct mt76_dev *mdev)
+void mt7921_update_channel(struct mt76_phy *mphy)
{
- struct mt7921_dev *dev = container_of(mdev, struct mt7921_dev, mt76);
+ struct mt7921_dev *dev = container_of(mphy->dev, struct mt7921_dev, mt76);
- if (mt76_connac_pm_wake(&dev->mphy, &dev->pm))
+ if (mt76_connac_pm_wake(mphy, &dev->pm))
return;
- mt7921_phy_update_channel(&mdev->phy, 0);
+ mt7921_phy_update_channel(mphy, 0);
/* reset obss airtime */
mt76_set(dev, MT_WF_RMAC_MIB_TIME0(0), MT_WF_RMAC_MIB_RXTIME_CLR);
- mt76_connac_power_save_sched(&dev->mphy, &dev->pm);
+ mt76_connac_power_save_sched(mphy, &dev->pm);
}
void mt7921_tx_token_put(struct mt7921_dev *dev)
struct mt7921_vif *mvif = (struct mt7921_vif *)vif->drv_priv;
struct mt7921_dev *dev = mvif->phy->dev;
- ieee80211_disconnect(vif, true);
+ if (vif->type == NL80211_IFTYPE_STATION)
+ ieee80211_disconnect(vif, true);
mt76_connac_mcu_uni_add_dev(&dev->mphy, vif, &mvif->sta.wcid, true);
mt7921_mcu_set_tx(dev, vif);
mt76_wr(dev, MT_WFDMA0_HOST_INT_ENA, 0);
mt76_wr(dev, MT_PCIE_MAC_INT_ENABLE, 0x0);
+ set_bit(MT76_RESET, &dev->mphy.state);
set_bit(MT76_MCU_RESET, &dev->mphy.state);
wake_up(&dev->mt76.mcu.wait);
skb_queue_purge(&dev->mt76.mcu.res_q);
mt7921_tx_token_put(dev);
idr_init(&dev->mt76.token);
- err = mt7921_wpdma_reset(dev, true);
- if (err)
- return err;
+ mt7921_wpdma_reset(dev, true);
mt76_for_each_q_rx(&dev->mt76, i) {
napi_enable(&dev->mt76.napi[i]);
napi_schedule(&dev->mt76.napi[i]);
}
- napi_enable(&dev->mt76.tx_napi);
- napi_schedule(&dev->mt76.tx_napi);
- mt76_worker_enable(&dev->mt76.tx_worker);
-
clear_bit(MT76_MCU_RESET, &dev->mphy.state);
- clear_bit(MT76_STATE_PM, &dev->mphy.state);
- mt76_wr(dev, MT_WFDMA0_HOST_INT_ENA, 0);
+ mt76_wr(dev, MT_WFDMA0_HOST_INT_ENA,
+ MT_INT_RX_DONE_ALL | MT_INT_TX_DONE_ALL |
+ MT_INT_MCU_CMD);
mt76_wr(dev, MT_PCIE_MAC_INT_ENABLE, 0xff);
err = mt7921_run_firmware(dev);
if (err)
- return err;
+ goto out;
err = mt7921_mcu_set_eeprom(dev);
if (err)
- return err;
+ goto out;
- mt7921_mac_init(dev);
- return __mt7921_start(&dev->phy);
+ err = mt7921_mac_init(dev);
+ if (err)
+ goto out;
+
+ err = __mt7921_start(&dev->phy);
+out:
+ clear_bit(MT76_RESET, &dev->mphy.state);
+
+ napi_enable(&dev->mt76.tx_napi);
+ napi_schedule(&dev->mt76.tx_napi);
+ mt76_worker_enable(&dev->mt76.tx_worker);
+
+ return err;
}
/* system error recovery */
void mt7921_mac_reset_work(struct work_struct *work)
{
- struct ieee80211_hw *hw;
- struct mt7921_dev *dev;
+ struct mt7921_dev *dev = container_of(work, struct mt7921_dev,
+ reset_work);
+ struct ieee80211_hw *hw = mt76_hw(dev);
+ struct mt76_connac_pm *pm = &dev->pm;
int i;
- dev = container_of(work, struct mt7921_dev, reset_work);
- hw = mt76_hw(dev);
-
dev_err(dev->mt76.dev, "chip reset\n");
+ dev->hw_full_reset = true;
ieee80211_stop_queues(hw);
cancel_delayed_work_sync(&dev->mphy.mac_work);
- cancel_delayed_work_sync(&dev->pm.ps_work);
- cancel_work_sync(&dev->pm.wake_work);
+ cancel_delayed_work_sync(&pm->ps_work);
+ cancel_work_sync(&pm->wake_work);
mutex_lock(&dev->mt76.mutex);
for (i = 0; i < 10; i++) {
+ __mt7921_mcu_drv_pmctrl(dev);
+
if (!mt7921_mac_reset(dev))
break;
}
ieee80211_scan_completed(dev->mphy.hw, &info);
}
+ dev->hw_full_reset = false;
ieee80211_wake_queues(hw);
ieee80211_iterate_active_interfaces(hw,
IEEE80211_IFACE_ITER_RESUME_ALL,
mt7921_vif_connect_iter, NULL);
+ mt76_connac_power_save_sched(&dev->mt76.phy, pm);
}
void mt7921_reset(struct mt76_dev *mdev)
{
struct mt7921_dev *dev = container_of(mdev, struct mt7921_dev, mt76);
+ if (!dev->hw_init_done)
+ return;
+
+ if (dev->hw_full_reset)
+ return;
+
queue_work(dev->mt76.wq, &dev->reset_work);
}
}
}
-static void
-mt7921_mac_sta_stats_work(struct mt7921_phy *phy)
-{
- struct mt7921_dev *dev = phy->dev;
- struct mt7921_sta *msta;
- LIST_HEAD(list);
-
- spin_lock_bh(&dev->sta_poll_lock);
- list_splice_init(&phy->stats_list, &list);
-
- while (!list_empty(&list)) {
- msta = list_first_entry(&list, struct mt7921_sta, stats_list);
- list_del_init(&msta->stats_list);
- spin_unlock_bh(&dev->sta_poll_lock);
-
- /* query wtbl info to report tx rate for further devices */
- mt7921_get_wtbl_info(dev, msta->wcid.idx);
-
- spin_lock_bh(&dev->sta_poll_lock);
- }
-
- spin_unlock_bh(&dev->sta_poll_lock);
-}
-
void mt7921_mac_work(struct work_struct *work)
{
struct mt7921_phy *phy;
mt7921_mutex_acquire(phy->dev);
- mt76_update_survey(mphy->dev);
+ mt76_update_survey(mphy);
if (++mphy->mac_work_count == 2) {
mphy->mac_work_count = 0;
mt7921_mac_update_mib_stats(phy);
}
- if (++phy->sta_work_count == 4) {
- phy->sta_work_count = 0;
- mt7921_mac_sta_stats_work(phy);
- }
mt7921_mutex_release(phy->dev);
ieee80211_queue_delayed_work(phy->mt76->hw, &mphy->mac_work,
napi_schedule(&dev->mt76.napi[i]);
mt76_connac_pm_dequeue_skbs(mphy, &dev->pm);
mt7921_tx_cleanup(dev);
- ieee80211_queue_delayed_work(mphy->hw, &mphy->mac_work,
- MT7921_WATCHDOG_TIME);
+ if (test_bit(MT76_STATE_RUNNING, &mphy->state))
+ ieee80211_queue_delayed_work(mphy->hw, &mphy->mac_work,
+ MT7921_WATCHDOG_TIME);
}
ieee80211_wake_queues(mphy->hw);
{
struct mt7921_dev *dev;
unsigned long delta;
+ struct mt76_phy *mphy;
dev = (struct mt7921_dev *)container_of(work, struct mt7921_dev,
pm.ps_work.work);
+ mphy = dev->phy.mt76;
delta = dev->pm.idle_timeout;
- if (test_bit(MT76_HW_SCANNING, &dev->mphy.state) ||
- test_bit(MT76_HW_SCHED_SCANNING, &dev->mphy.state))
+ if (test_bit(MT76_HW_SCANNING, &mphy->state) ||
+ test_bit(MT76_HW_SCHED_SCANNING, &mphy->state))
goto out;
if (time_is_after_jiffies(dev->pm.last_activity + delta)) {
goto out;
}
- if (!mt7921_mcu_fw_pmctrl(dev))
+ if (!mt7921_mcu_fw_pmctrl(dev)) {
+ cancel_delayed_work_sync(&mphy->mac_work);
return;
+ }
out:
queue_delayed_work(dev->mt76.wq, &dev->pm.ps_work, delta);
}
break;
skb_pull(skb, sizeof(struct mt7921_mcu_rxd));
- if (data + skb->len - dump > MT76_CONNAC_COREDUMP_SZ) {
+ if (!dump || data + skb->len - dump > MT76_CONNAC_COREDUMP_SZ) {
dev_kfree_skb(skb);
continue;
}
dev_kfree_skb(skb);
}
- dev_coredumpv(dev->mt76.dev, dump, MT76_CONNAC_COREDUMP_SZ,
- GFP_KERNEL);
+
+ if (dump)
+ dev_coredumpv(dev->mt76.dev, dump, MT76_CONNAC_COREDUMP_SZ,
+ GFP_KERNEL);
+
mt7921_reset(&dev->mt76);
}
/* RXD DW4 */
#define MT_RXD4_NORMAL_PAYLOAD_FORMAT GENMASK(1, 0)
+#define MT_RXD4_FIRST_AMSDU_FRAME GENMASK(1, 0)
+#define MT_RXD4_MID_AMSDU_FRAME BIT(1)
+#define MT_RXD4_LAST_AMSDU_FRAME BIT(0)
#define MT_RXD4_NORMAL_PATTERN_DROP BIT(9)
#define MT_RXD4_NORMAL_CLS BIT(10)
#define MT_RXD4_NORMAL_OFLD GENMASK(12, 11)
#define MT_RXD3_NORMAL_PF_MODE BIT(29)
#define MT_RXD3_NORMAL_PF_STS GENMASK(31, 30)
+/* RXD GROUP4 */
+#define MT_RXD6_FRAME_CONTROL GENMASK(15, 0)
+#define MT_RXD6_TA_LO GENMASK(31, 16)
+
+#define MT_RXD7_TA_HI GENMASK(31, 0)
+
+#define MT_RXD8_SEQ_CTRL GENMASK(15, 0)
+#define MT_RXD8_QOS_CTL GENMASK(31, 16)
+
+#define MT_RXD9_HT_CONTROL GENMASK(31, 0)
+
/* P-RXV DW0 */
#define MT_PRXV_TX_RATE GENMASK(6, 0)
#define MT_PRXV_TX_DCM BIT(4)
IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_40MHZ_IN_2G;
else if (band == NL80211_BAND_5GHZ)
he_cap_elem->phy_cap_info[0] =
- IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_40MHZ_80MHZ_IN_5G |
- IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_80PLUS80_MHZ_IN_5G;
+ IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_40MHZ_80MHZ_IN_5G;
he_cap_elem->phy_cap_info[1] =
IEEE80211_HE_PHY_CAP1_LDPC_CODING_IN_PAYLOAD;
he_cap_elem->phy_cap_info[2] =
+ IEEE80211_HE_PHY_CAP2_NDP_4x_LTF_AND_3_2US |
IEEE80211_HE_PHY_CAP2_STBC_TX_UNDER_80MHZ |
- IEEE80211_HE_PHY_CAP2_STBC_RX_UNDER_80MHZ;
+ IEEE80211_HE_PHY_CAP2_STBC_RX_UNDER_80MHZ |
+ IEEE80211_HE_PHY_CAP2_UL_MU_FULL_MU_MIMO |
+ IEEE80211_HE_PHY_CAP2_UL_MU_PARTIAL_MU_MIMO;
switch (i) {
case NL80211_IFTYPE_STATION:
- he_cap_elem->mac_cap_info[0] |=
- IEEE80211_HE_MAC_CAP0_TWT_REQ;
he_cap_elem->mac_cap_info[1] |=
IEEE80211_HE_MAC_CAP1_TF_MAC_PAD_DUR_16US;
he_cap_elem->phy_cap_info[3] |=
IEEE80211_HE_PHY_CAP3_DCM_MAX_CONST_TX_QPSK |
IEEE80211_HE_PHY_CAP3_DCM_MAX_CONST_RX_QPSK;
+ he_cap_elem->phy_cap_info[4] |=
+ IEEE80211_HE_PHY_CAP4_SU_BEAMFORMEE |
+ IEEE80211_HE_PHY_CAP4_BEAMFORMEE_MAX_STS_UNDER_80MHZ_4;
+ he_cap_elem->phy_cap_info[5] |=
+ IEEE80211_HE_PHY_CAP5_NG16_SU_FEEDBACK |
+ IEEE80211_HE_PHY_CAP5_NG16_MU_FEEDBACK;
he_cap_elem->phy_cap_info[6] |=
+ IEEE80211_HE_PHY_CAP6_CODEBOOK_SIZE_42_SU |
+ IEEE80211_HE_PHY_CAP6_CODEBOOK_SIZE_75_MU |
IEEE80211_HE_PHY_CAP6_TRIG_CQI_FB |
IEEE80211_HE_PHY_CAP6_PARTIAL_BW_EXT_RANGE |
IEEE80211_HE_PHY_CAP6_PPE_THRESHOLD_PRESENT;
mt7921_mutex_release(dev);
}
-static inline int get_free_idx(u32 mask, u8 start, u8 end)
-{
- return ffs(~mask & GENMASK(end, start));
-}
-
-static int get_omac_idx(enum nl80211_iftype type, u64 mask)
-{
- int i;
-
- switch (type) {
- case NL80211_IFTYPE_STATION:
- /* prefer hw bssid slot 1-3 */
- i = get_free_idx(mask, HW_BSSID_1, HW_BSSID_3);
- if (i)
- return i - 1;
-
- /* next, try to find a free repeater entry for the sta */
- i = get_free_idx(mask >> REPEATER_BSSID_START, 0,
- REPEATER_BSSID_MAX - REPEATER_BSSID_START);
- if (i)
- return i + 32 - 1;
-
- i = get_free_idx(mask, EXT_BSSID_1, EXT_BSSID_MAX);
- if (i)
- return i - 1;
-
- if (~mask & BIT(HW_BSSID_0))
- return HW_BSSID_0;
-
- break;
- case NL80211_IFTYPE_MONITOR:
- /* ap uses hw bssid 0 and ext bssid */
- if (~mask & BIT(HW_BSSID_0))
- return HW_BSSID_0;
-
- i = get_free_idx(mask, EXT_BSSID_1, EXT_BSSID_MAX);
- if (i)
- return i - 1;
-
- break;
- default:
- WARN_ON(1);
- break;
- }
-
- return -1;
-}
-
static int mt7921_add_interface(struct ieee80211_hw *hw,
struct ieee80211_vif *vif)
{
goto out;
}
- idx = get_omac_idx(vif->type, phy->omac_mask);
- if (idx < 0) {
- ret = -ENOSPC;
- goto out;
- }
- mvif->mt76.omac_idx = idx;
+ mvif->mt76.omac_idx = mvif->mt76.idx;
mvif->phy = phy;
mvif->mt76.band_idx = 0;
mvif->mt76.wmm_idx = mvif->mt76.idx % MT7921_MAX_WMM_SETS;
spin_unlock_bh(&dev->sta_poll_lock);
}
-int mt7921_set_channel(struct mt7921_phy *phy)
+static int mt7921_set_channel(struct mt7921_phy *phy)
{
struct mt7921_dev *dev = phy->dev;
int ret;
key->flags |= IEEE80211_KEY_FLAG_GENERATE_MMIE;
wcid_keyidx = &wcid->hw_key_idx2;
break;
+ case WLAN_CIPHER_SUITE_WEP40:
+ case WLAN_CIPHER_SUITE_WEP104:
+ if (!mvif->wep_sta)
+ return -EOPNOTSUPP;
case WLAN_CIPHER_SUITE_TKIP:
case WLAN_CIPHER_SUITE_CCMP:
case WLAN_CIPHER_SUITE_CCMP_256:
case WLAN_CIPHER_SUITE_GCMP_256:
case WLAN_CIPHER_SUITE_SMS4:
break;
- case WLAN_CIPHER_SUITE_WEP40:
- case WLAN_CIPHER_SUITE_WEP104:
default:
return -EOPNOTSUPP;
}
cmd == SET_KEY ? key : NULL);
err = mt7921_mcu_add_key(dev, vif, msta, key, cmd);
+ if (err)
+ goto out;
+
+ if (key->cipher == WLAN_CIPHER_SUITE_WEP104 ||
+ key->cipher == WLAN_CIPHER_SUITE_WEP40)
+ err = mt7921_mcu_add_key(dev, vif, mvif->wep_sta, key, cmd);
out:
mt7921_mutex_release(dev);
mt7921_mutex_acquire(dev);
+ if (changed & IEEE80211_CONF_CHANGE_POWER)
+ mt76_connac_mcu_set_rate_txpower(phy->mt76);
+
if (changed & IEEE80211_CONF_CHANGE_MONITOR) {
bool enabled = !!(hw->conf.flags & IEEE80211_CONF_MONITOR);
mt7921_mcu_uni_bss_ps(dev, vif);
if (changed & BSS_CHANGED_ASSOC) {
- mt7921_mcu_sta_add(dev, NULL, vif, true);
+ mt7921_mcu_sta_update(dev, NULL, vif, true,
+ MT76_STA_INFO_STATE_ASSOC);
mt7921_bss_bcnft_apply(dev, vif, info->assoc);
}
if (ret)
return ret;
- if (vif->type == NL80211_IFTYPE_STATION && !sta->tdls)
- mt76_connac_mcu_uni_add_bss(&dev->mphy, vif, &mvif->sta.wcid,
- true);
+ if (vif->type == NL80211_IFTYPE_STATION)
+ mvif->wep_sta = msta;
mt7921_mac_wtbl_update(dev, idx,
MT_WTBL_UPDATE_ADM_COUNT_CLEAR);
- ret = mt7921_mcu_sta_add(dev, sta, vif, true);
+ ret = mt7921_mcu_sta_update(dev, sta, vif, true,
+ MT76_STA_INFO_STATE_NONE);
if (ret)
return ret;
return 0;
}
+void mt7921_mac_sta_assoc(struct mt76_dev *mdev, struct ieee80211_vif *vif,
+ struct ieee80211_sta *sta)
+{
+ struct mt7921_dev *dev = container_of(mdev, struct mt7921_dev, mt76);
+ struct mt7921_sta *msta = (struct mt7921_sta *)sta->drv_priv;
+ struct mt7921_vif *mvif = (struct mt7921_vif *)vif->drv_priv;
+
+ mt7921_mutex_acquire(dev);
+
+ if (vif->type == NL80211_IFTYPE_STATION && !sta->tdls)
+ mt76_connac_mcu_uni_add_bss(&dev->mphy, vif, &mvif->sta.wcid,
+ true);
+
+ mt7921_mac_wtbl_update(dev, msta->wcid.idx,
+ MT_WTBL_UPDATE_ADM_COUNT_CLEAR);
+
+ mt7921_mcu_sta_update(dev, sta, vif, true, MT76_STA_INFO_STATE_ASSOC);
+
+ mt7921_mutex_release(dev);
+}
+
void mt7921_mac_sta_remove(struct mt76_dev *mdev, struct ieee80211_vif *vif,
struct ieee80211_sta *sta)
{
mt76_connac_free_pending_tx_skbs(&dev->pm, &msta->wcid);
mt76_connac_pm_wake(&dev->mphy, &dev->pm);
- mt7921_mcu_sta_add(dev, sta, vif, false);
+ mt7921_mcu_sta_update(dev, sta, vif, false, MT76_STA_INFO_STATE_NONE);
mt7921_mac_wtbl_update(dev, msta->wcid.idx,
MT_WTBL_UPDATE_ADM_COUNT_CLEAR);
if (vif->type == NL80211_IFTYPE_STATION) {
struct mt7921_vif *mvif = (struct mt7921_vif *)vif->drv_priv;
+ mvif->wep_sta = NULL;
ewma_rssi_init(&mvif->rssi);
if (!sta->tdls)
mt76_connac_mcu_uni_add_bss(&dev->mphy, vif,
}
mt76_txq_schedule_all(&dev->mphy);
- mt76_connac_pm_unref(&dev->pm);
+ mt76_connac_pm_unref(&dev->mphy, &dev->pm);
}
static void mt7921_tx(struct ieee80211_hw *hw,
if (mt76_connac_pm_ref(mphy, &dev->pm)) {
mt76_tx(mphy, control->sta, wcid, skb);
- mt76_connac_pm_unref(&dev->pm);
+ mt76_connac_pm_unref(mphy, &dev->pm);
return;
}
return ret;
}
-static int
-mt7921_sta_add(struct ieee80211_hw *hw, struct ieee80211_vif *vif,
- struct ieee80211_sta *sta)
+static int mt7921_sta_state(struct ieee80211_hw *hw,
+ struct ieee80211_vif *vif,
+ struct ieee80211_sta *sta,
+ enum ieee80211_sta_state old_state,
+ enum ieee80211_sta_state new_state)
{
- return mt76_sta_state(hw, vif, sta, IEEE80211_STA_NOTEXIST,
- IEEE80211_STA_NONE);
-}
+ struct mt7921_dev *dev = mt7921_hw_dev(hw);
-static int
-mt7921_sta_remove(struct ieee80211_hw *hw, struct ieee80211_vif *vif,
- struct ieee80211_sta *sta)
-{
- return mt76_sta_state(hw, vif, sta, IEEE80211_STA_NONE,
- IEEE80211_STA_NOTEXIST);
+ if (dev->pm.ds_enable) {
+ mt7921_mutex_acquire(dev);
+ mt76_connac_sta_state_dp(&dev->mt76, old_state, new_state);
+ mt7921_mutex_release(dev);
+ }
+
+ return mt76_sta_state(hw, vif, sta, old_state, new_state);
}
static int
HZ / 2);
}
+static void mt7921_sta_set_decap_offload(struct ieee80211_hw *hw,
+ struct ieee80211_vif *vif,
+ struct ieee80211_sta *sta,
+ bool enabled)
+{
+ struct mt7921_sta *msta = (struct mt7921_sta *)sta->drv_priv;
+ struct mt7921_dev *dev = mt7921_hw_dev(hw);
+
+ if (enabled)
+ set_bit(MT_WCID_FLAG_HDR_TRANS, &msta->wcid.flags);
+ else
+ clear_bit(MT_WCID_FLAG_HDR_TRANS, &msta->wcid.flags);
+
+ mt76_connac_mcu_sta_update_hdr_trans(&dev->mt76, vif, &msta->wcid,
+ MCU_UNI_CMD_STA_REC_UPDATE);
+}
+
const struct ieee80211_ops mt7921_ops = {
.tx = mt7921_tx,
.start = mt7921_start,
.conf_tx = mt7921_conf_tx,
.configure_filter = mt7921_configure_filter,
.bss_info_changed = mt7921_bss_info_changed,
- .sta_add = mt7921_sta_add,
- .sta_remove = mt7921_sta_remove,
+ .sta_state = mt7921_sta_state,
.sta_pre_rcu_remove = mt76_sta_pre_rcu_remove,
.set_key = mt7921_set_key,
+ .sta_set_decap_offload = mt7921_sta_set_decap_offload,
.ampdu_action = mt7921_ampdu_action,
.set_rts_threshold = mt7921_set_rts_threshold,
.wake_tx_queue = mt76_wake_tx_queue,
#define to_wcid_lo(id) FIELD_GET(GENMASK(7, 0), (u16)id)
#define to_wcid_hi(id) FIELD_GET(GENMASK(9, 8), (u16)id)
-static enum mt7921_cipher_type
+static enum mcu_cipher_type
mt7921_mcu_get_cipher(int cipher)
{
switch (cipher) {
case WLAN_CIPHER_SUITE_WEP40:
- return MT_CIPHER_WEP40;
+ return MCU_CIPHER_WEP40;
case WLAN_CIPHER_SUITE_WEP104:
- return MT_CIPHER_WEP104;
+ return MCU_CIPHER_WEP104;
case WLAN_CIPHER_SUITE_TKIP:
- return MT_CIPHER_TKIP;
+ return MCU_CIPHER_TKIP;
case WLAN_CIPHER_SUITE_AES_CMAC:
- return MT_CIPHER_BIP_CMAC_128;
+ return MCU_CIPHER_BIP_CMAC_128;
case WLAN_CIPHER_SUITE_CCMP:
- return MT_CIPHER_AES_CCMP;
+ return MCU_CIPHER_AES_CCMP;
case WLAN_CIPHER_SUITE_CCMP_256:
- return MT_CIPHER_CCMP_256;
+ return MCU_CIPHER_CCMP_256;
case WLAN_CIPHER_SUITE_GCMP:
- return MT_CIPHER_GCMP;
+ return MCU_CIPHER_GCMP;
case WLAN_CIPHER_SUITE_GCMP_256:
- return MT_CIPHER_GCMP_256;
+ return MCU_CIPHER_GCMP_256;
case WLAN_CIPHER_SUITE_SMS4:
- return MT_CIPHER_WAPI;
+ return MCU_CIPHER_WAPI;
default:
return MT_CIPHER_NONE;
}
}
static void
-mt7921_mcu_tx_rate_report(struct mt7921_dev *dev, struct sk_buff *skb,
- u16 wlan_idx)
-{
- struct mt7921_mcu_wlan_info_event *wtbl_info =
- (struct mt7921_mcu_wlan_info_event *)(skb->data);
- struct rate_info rate = {};
- u8 curr_idx = wtbl_info->rate_info.rate_idx;
- u16 curr = le16_to_cpu(wtbl_info->rate_info.rate[curr_idx]);
- struct mt7921_mcu_peer_cap peer = wtbl_info->peer_cap;
- struct mt76_phy *mphy = &dev->mphy;
- struct mt7921_sta_stats *stats;
- struct mt7921_sta *msta;
- struct mt76_wcid *wcid;
-
- if (wlan_idx >= MT76_N_WCIDS)
- return;
-
- rcu_read_lock();
-
- wcid = rcu_dereference(dev->mt76.wcid[wlan_idx]);
- if (!wcid)
- goto out;
-
- msta = container_of(wcid, struct mt7921_sta, wcid);
- stats = &msta->stats;
-
- /* current rate */
- mt7921_mcu_tx_rate_parse(mphy, &peer, &rate, curr);
- stats->tx_rate = rate;
-out:
- rcu_read_unlock();
-}
-
-static void
mt7921_mcu_scan_event(struct mt7921_dev *dev, struct sk_buff *skb)
{
struct mt76_phy *mphy = &dev->mt76.phy;
}
static void
-mt7921_mcu_beacon_loss_event(struct mt7921_dev *dev, struct sk_buff *skb)
+mt7921_mcu_connection_loss_iter(void *priv, u8 *mac,
+ struct ieee80211_vif *vif)
+{
+ struct mt76_vif *mvif = (struct mt76_vif *)vif->drv_priv;
+ struct mt76_connac_beacon_loss_event *event = priv;
+
+ if (mvif->idx != event->bss_idx)
+ return;
+
+ if (!(vif->driver_flags & IEEE80211_VIF_BEACON_FILTER))
+ return;
+
+ ieee80211_connection_loss(vif);
+}
+
+static void
+mt7921_mcu_connection_loss_event(struct mt7921_dev *dev, struct sk_buff *skb)
{
struct mt76_connac_beacon_loss_event *event;
- struct mt76_phy *mphy;
- u8 band_idx = 0; /* DBDC support */
+ struct mt76_phy *mphy = &dev->mt76.phy;
skb_pull(skb, sizeof(struct mt7921_mcu_rxd));
event = (struct mt76_connac_beacon_loss_event *)skb->data;
- if (band_idx && dev->mt76.phy2)
- mphy = dev->mt76.phy2;
- else
- mphy = &dev->mt76.phy;
ieee80211_iterate_active_interfaces_atomic(mphy->hw,
IEEE80211_IFACE_ITER_RESUME_ALL,
- mt76_connac_mcu_beacon_loss_iter, event);
+ mt7921_mcu_connection_loss_iter, event);
}
static void
}
static void
+mt7921_mcu_tx_done_event(struct mt7921_dev *dev, struct sk_buff *skb)
+{
+ struct mt7921_mcu_tx_done_event *event;
+ struct mt7921_sta *msta;
+ struct mt7921_phy *mphy = &dev->phy;
+ struct mt7921_mcu_peer_cap peer;
+ struct ieee80211_sta *sta;
+ LIST_HEAD(list);
+
+ skb_pull(skb, sizeof(struct mt7921_mcu_rxd));
+ event = (struct mt7921_mcu_tx_done_event *)skb->data;
+
+ spin_lock_bh(&dev->sta_poll_lock);
+ list_splice_init(&mphy->stats_list, &list);
+
+ while (!list_empty(&list)) {
+ msta = list_first_entry(&list, struct mt7921_sta, stats_list);
+ list_del_init(&msta->stats_list);
+
+ if (msta->wcid.idx != event->wlan_idx)
+ continue;
+
+ spin_unlock_bh(&dev->sta_poll_lock);
+
+ sta = wcid_to_sta(&msta->wcid);
+
+ /* peer config based on IEEE SPEC */
+ memset(&peer, 0x0, sizeof(peer));
+ peer.bw = event->bw;
+ peer.g2 = !!(sta->ht_cap.cap & IEEE80211_HT_CAP_SGI_20);
+ peer.g4 = !!(sta->ht_cap.cap & IEEE80211_HT_CAP_SGI_40);
+ peer.g8 = !!(sta->vht_cap.cap & IEEE80211_VHT_CAP_SHORT_GI_80);
+ peer.g16 = !!(sta->vht_cap.cap & IEEE80211_VHT_CAP_SHORT_GI_160);
+ mt7921_mcu_tx_rate_parse(mphy->mt76, &peer,
+ &msta->stats.tx_rate, event->tx_rate);
+
+ spin_lock_bh(&dev->sta_poll_lock);
+ break;
+ }
+ spin_unlock_bh(&dev->sta_poll_lock);
+}
+
+static void
mt7921_mcu_rx_unsolicited_event(struct mt7921_dev *dev, struct sk_buff *skb)
{
struct mt7921_mcu_rxd *rxd = (struct mt7921_mcu_rxd *)skb->data;
switch (rxd->eid) {
case MCU_EVENT_BSS_BEACON_LOSS:
- mt7921_mcu_beacon_loss_event(dev, skb);
+ mt7921_mcu_connection_loss_event(dev, skb);
break;
case MCU_EVENT_SCHED_SCAN_DONE:
case MCU_EVENT_SCAN_DONE:
case MCU_EVENT_LP_INFO:
mt7921_mcu_low_power_event(dev, skb);
break;
+ case MCU_EVENT_TX_DONE:
+ mt7921_mcu_tx_done_event(dev, skb);
+ break;
default:
break;
}
rxd->eid == MCU_EVENT_SCHED_SCAN_DONE ||
rxd->eid == MCU_EVENT_BSS_ABSENCE ||
rxd->eid == MCU_EVENT_SCAN_DONE ||
+ rxd->eid == MCU_EVENT_TX_DONE ||
rxd->eid == MCU_EVENT_DBG_MSG ||
rxd->eid == MCU_EVENT_COREDUMP ||
rxd->eid == MCU_EVENT_LP_INFO ||
sec_key = &sec->key[0];
sec_key->cipher_len = sizeof(*sec_key);
- if (cipher == MT_CIPHER_BIP_CMAC_128) {
- sec_key->cipher_id = MT_CIPHER_AES_CCMP;
+ if (cipher == MCU_CIPHER_BIP_CMAC_128) {
+ sec_key->cipher_id = MCU_CIPHER_AES_CCMP;
sec_key->key_id = bip->keyidx;
sec_key->key_len = 16;
memcpy(sec_key->key, bip->key, 16);
sec_key = &sec->key[1];
- sec_key->cipher_id = MT_CIPHER_BIP_CMAC_128;
+ sec_key->cipher_id = MCU_CIPHER_BIP_CMAC_128;
sec_key->cipher_len = sizeof(*sec_key);
sec_key->key_len = 16;
memcpy(sec_key->key, key->key, 16);
sec_key->key_len = key->keylen;
memcpy(sec_key->key, key->key, key->keylen);
- if (cipher == MT_CIPHER_TKIP) {
+ if (cipher == MCU_CIPHER_TKIP) {
/* Rx/Tx MIC keys are swapped */
memcpy(sec_key->key + 16, key->key + 24, 8);
memcpy(sec_key->key + 24, key->key + 16, 8);
}
/* store key_conf for BIP batch update */
- if (cipher == MT_CIPHER_AES_CCMP) {
+ if (cipher == MCU_CIPHER_AES_CCMP) {
memcpy(bip->key, key->key, key->keylen);
bip->keyidx = key->keyidx;
}
dev->mt76.hw->wiphy->wowlan = &mt76_connac_wowlan_support;
#endif /* CONFIG_PM */
- clear_bit(MT76_STATE_PM, &dev->mphy.state);
-
dev_err(dev->mt76.dev, "Firmware init done\n");
return 0;
set_bit(MT76_STATE_MCU_RUNNING, &dev->mphy.state);
mt7921_mcu_fw_log_2_host(dev, 1);
- return 0;
+ return mt76_connac_mcu_get_nic_capability(&dev->mphy);
}
int mt7921_mcu_init(struct mt7921_dev *dev)
return 0;
}
-u32 mt7921_get_wtbl_info(struct mt7921_dev *dev, u32 wlan_idx)
-{
- struct mt7921_mcu_wlan_info wtbl_info = {
- .wlan_idx = cpu_to_le32(wlan_idx),
- };
- struct sk_buff *skb;
- int ret;
-
- ret = mt76_mcu_send_and_get_msg(&dev->mt76, MCU_CMD_GET_WTBL,
- &wtbl_info, sizeof(wtbl_info), true,
- &skb);
- if (ret)
- return ret;
-
- mt7921_mcu_tx_rate_report(dev, skb, wlan_idx);
- dev_kfree_skb(skb);
-
- return 0;
-}
-
int mt7921_mcu_uni_bss_ps(struct mt7921_dev *dev, struct ieee80211_vif *vif)
{
struct mt7921_vif *mvif = (struct mt7921_vif *)vif->drv_priv;
sizeof(req), false);
}
-int mt7921_mcu_sta_add(struct mt7921_dev *dev, struct ieee80211_sta *sta,
- struct ieee80211_vif *vif, bool enable)
+int mt7921_mcu_sta_update(struct mt7921_dev *dev, struct ieee80211_sta *sta,
+ struct ieee80211_vif *vif, bool enable,
+ enum mt76_sta_info_state state)
{
struct mt7921_vif *mvif = (struct mt7921_vif *)vif->drv_priv;
int rssi = -ewma_rssi_read(&mvif->rssi);
.vif = vif,
.enable = enable,
.cmd = MCU_UNI_CMD_STA_REC_UPDATE,
+ .state = state,
+ .offload_fw = true,
.rcpi = to_rcpi(rssi),
};
struct mt7921_sta *msta;
msta = sta ? (struct mt7921_sta *)sta->drv_priv : NULL;
info.wcid = msta ? &msta->wcid : &mvif->sta.wcid;
+ info.newly = msta ? state != MT76_STA_INFO_STATE_ASSOC : true;
- return mt76_connac_mcu_add_sta_cmd(&dev->mphy, &info);
+ return mt76_connac_mcu_sta_cmd(&dev->mphy, &info);
}
-int mt7921_mcu_drv_pmctrl(struct mt7921_dev *dev)
+int __mt7921_mcu_drv_pmctrl(struct mt7921_dev *dev)
{
struct mt76_phy *mphy = &dev->mt76.phy;
struct mt76_connac_pm *pm = &dev->pm;
int i, err = 0;
- mutex_lock(&pm->mutex);
-
- if (!test_bit(MT76_STATE_PM, &mphy->state))
- goto out;
-
for (i = 0; i < MT7921_DRV_OWN_RETRY_COUNT; i++) {
mt76_wr(dev, MT_CONN_ON_LPCTL, PCIE_LPCR_HOST_CLR_OWN);
if (mt76_poll_msec(dev, MT_CONN_ON_LPCTL,
pm->stats.doze_time += pm->stats.last_wake_event -
pm->stats.last_doze_event;
out:
+ return err;
+}
+
+int mt7921_mcu_drv_pmctrl(struct mt7921_dev *dev)
+{
+ struct mt76_phy *mphy = &dev->mt76.phy;
+ struct mt76_connac_pm *pm = &dev->pm;
+ int err = 0;
+
+ mutex_lock(&pm->mutex);
+
+ if (!test_bit(MT76_STATE_PM, &mphy->state))
+ goto out;
+
+ err = __mt7921_mcu_drv_pmctrl(dev);
+out:
mutex_unlock(&pm->mutex);
if (err)
{
struct mt7921_phy *phy = priv;
struct mt7921_dev *dev = phy->dev;
+ struct ieee80211_hw *hw = mt76_hw(dev);
int ret;
if (dev->pm.enable)
if (dev->pm.enable) {
vif->driver_flags |= IEEE80211_VIF_BEACON_FILTER;
+ ieee80211_hw_set(hw, CONNECTION_MONITOR);
mt76_set(dev, MT_WF_RFCR(0), MT_WF_RFCR_DROP_OTHER_BEACON);
} else {
vif->driver_flags &= ~IEEE80211_VIF_BEACON_FILTER;
+ __clear_bit(IEEE80211_HW_CONNECTION_MONITOR, hw->flags);
mt76_clear(dev, MT_WF_RFCR(0), MT_WF_RFCR_DROP_OTHER_BEACON);
}
}
MCU_EVENT_REG_ACCESS = 0x05,
MCU_EVENT_LP_INFO = 0x07,
MCU_EVENT_SCAN_DONE = 0x0d,
+ MCU_EVENT_TX_DONE = 0x0f,
MCU_EVENT_BSS_ABSENCE = 0x11,
MCU_EVENT_BSS_BEACON_LOSS = 0x13,
MCU_EVENT_CH_PRIVILEGE = 0x18,
struct sec_key key[2];
} __packed;
-enum mt7921_cipher_type {
- MT_CIPHER_NONE,
- MT_CIPHER_WEP40,
- MT_CIPHER_WEP104,
- MT_CIPHER_WEP128,
- MT_CIPHER_TKIP,
- MT_CIPHER_AES_CCMP,
- MT_CIPHER_CCMP_256,
- MT_CIPHER_GCMP,
- MT_CIPHER_GCMP_256,
- MT_CIPHER_WAPI,
- MT_CIPHER_BIP_CMAC_128,
+enum mcu_cipher_type {
+ MCU_CIPHER_WEP40 = 1,
+ MCU_CIPHER_WEP104,
+ MCU_CIPHER_WEP128,
+ MCU_CIPHER_TKIP,
+ MCU_CIPHER_AES_CCMP,
+ MCU_CIPHER_CCMP_256,
+ MCU_CIPHER_GCMP,
+ MCU_CIPHER_GCMP_256,
+ MCU_CIPHER_WAPI,
+ MCU_CIPHER_BIP_CMAC_128,
};
enum {
__le32 val;
} __packed;
-struct mt7921_mcu_tx_config {
- u8 peer_addr[ETH_ALEN];
- u8 sw;
- u8 dis_rx_hdr_tran;
-
- u8 aad_om;
- u8 pfmu_idx;
- __le16 partial_aid;
-
- u8 ibf;
- u8 ebf;
- u8 is_ht;
- u8 is_vht;
-
- u8 mesh;
- u8 baf_en;
- u8 cf_ack;
- u8 rdg_ba;
-
- u8 rdg;
- u8 pm;
- u8 rts;
- u8 smps;
-
- u8 txop_ps;
- u8 not_update_ipsm;
- u8 skip_tx;
- u8 ldpc;
-
- u8 qos;
- u8 from_ds;
- u8 to_ds;
- u8 dyn_bw;
-
- u8 amdsu_cross_lg;
- u8 check_per;
- u8 gid_63;
- u8 he;
-
- u8 vht_ibf;
- u8 vht_ebf;
- u8 vht_ldpc;
- u8 he_ldpc;
-} __packed;
-
-struct mt7921_mcu_sec_config {
- u8 wpi_flag;
- u8 rv;
- u8 ikv;
- u8 rkv;
-
- u8 rcid;
- u8 rca1;
- u8 rca2;
- u8 even_pn;
-
- u8 key_id;
- u8 muar_idx;
- u8 cipher_suit;
- u8 rsv[1];
-} __packed;
-
-struct mt7921_mcu_key_config {
- u8 key[32];
-} __packed;
-
-struct mt7921_mcu_rate_info {
- u8 mpdu_fail;
- u8 mpdu_tx;
- u8 rate_idx;
- u8 rsv[1];
- __le16 rate[8];
-} __packed;
-
-struct mt7921_mcu_ba_config {
- u8 ba_en;
- u8 rsv[3];
- __le32 ba_winsize;
-} __packed;
-
struct mt7921_mcu_ant_id_config {
u8 ant_id[4];
} __packed;
u8 rsv[1];
} __packed;
-struct mt7921_mcu_rx_cnt {
- u8 rx_rcpi[4];
- u8 rx_cc[4];
- u8 rx_cc_sel;
- u8 ce_rmsd;
- u8 rsv[2];
-} __packed;
-
-struct mt7921_mcu_tx_cnt {
- __le16 rate1_cnt;
- __le16 rate1_fail_cnt;
- __le16 rate2_cnt;
- __le16 rate3_cnt;
- __le16 cur_bw_tx_cnt;
- __le16 cur_bw_tx_fail_cnt;
- __le16 other_bw_tx_cnt;
- __le16 other_bw_tx_fail_cnt;
-} __packed;
-
-struct mt7921_mcu_wlan_info_event {
- struct mt7921_mcu_tx_config tx_config;
- struct mt7921_mcu_sec_config sec_config;
- struct mt7921_mcu_key_config key_config;
- struct mt7921_mcu_rate_info rate_info;
- struct mt7921_mcu_ba_config ba_config;
- struct mt7921_mcu_peer_cap peer_cap;
- struct mt7921_mcu_rx_cnt rx_cnt;
- struct mt7921_mcu_tx_cnt tx_cnt;
-} __packed;
-
-struct mt7921_mcu_wlan_info {
- __le32 wlan_idx;
- struct mt7921_mcu_wlan_info_event event;
-} __packed;
-
struct mt7921_txpwr_req {
u8 ver;
u8 action;
struct mt7921_txpwr txpwr;
} __packed;
+struct mt7921_mcu_tx_done_event {
+ u8 pid;
+ u8 status;
+ u16 seq;
+
+ u8 wlan_idx;
+ u8 tx_cnt;
+ u16 tx_rate;
+
+ u8 flag;
+ u8 tid;
+ u8 rsp_rate;
+ u8 mcs;
+
+ u8 bw;
+ u8 tx_pwr;
+ u8 reason;
+ u8 rsv0[1];
+
+ u32 delay;
+ u32 timestamp;
+ u32 applied_flag;
+
+ u8 txs[28];
+
+ u8 rsv1[32];
+} __packed;
#endif
unsigned long ampdu_state;
struct mt7921_sta_key_conf bip;
+
+ unsigned long next_txs_ts;
};
DECLARE_EWMA(rssi, 10, 8);
struct mt76_vif mt76; /* must be first */
struct mt7921_sta sta;
+ struct mt7921_sta *wep_sta;
+
struct mt7921_phy *phy;
struct ewma_rssi rssi;
u16 chainmask;
struct work_struct reset_work;
+ bool hw_full_reset:1;
+ bool hw_init_done:1;
struct list_head sta_poll_list;
spinlock_t sta_poll_lock;
int mt7921_mcu_add_key(struct mt7921_dev *dev, struct ieee80211_vif *vif,
struct mt7921_sta *msta, struct ieee80211_key_conf *key,
enum set_key_cmd cmd);
-int mt7921_set_channel(struct mt7921_phy *phy);
-int mt7921_mcu_sta_add(struct mt7921_dev *dev, struct ieee80211_sta *sta,
- struct ieee80211_vif *vif, bool enable);
+int mt7921_mcu_sta_update(struct mt7921_dev *dev, struct ieee80211_sta *sta,
+ struct ieee80211_vif *vif, bool enable,
+ enum mt76_sta_info_state state);
int mt7921_mcu_set_chan_info(struct mt7921_phy *phy, int cmd);
int mt7921_mcu_set_tx(struct mt7921_dev *dev, struct ieee80211_vif *vif);
int mt7921_mcu_set_eeprom(struct mt7921_dev *dev);
return !mt76_get_field(dev, MT_WFDMA_DUMMY_CR, MT_WFDMA_NEED_REINIT);
}
-void mt7921_mac_init(struct mt7921_dev *dev);
+int mt7921_mac_init(struct mt7921_dev *dev);
bool mt7921_mac_wtbl_update(struct mt7921_dev *dev, int idx, u32 mask);
void mt7921_mac_reset_counters(struct mt7921_phy *phy);
void mt7921_mac_write_txwi(struct mt7921_dev *dev, __le32 *txwi,
void mt7921_mac_tx_free(struct mt7921_dev *dev, struct sk_buff *skb);
int mt7921_mac_sta_add(struct mt76_dev *mdev, struct ieee80211_vif *vif,
struct ieee80211_sta *sta);
+void mt7921_mac_sta_assoc(struct mt76_dev *mdev, struct ieee80211_vif *vif,
+ struct ieee80211_sta *sta);
void mt7921_mac_sta_remove(struct mt76_dev *mdev, struct ieee80211_vif *vif,
struct ieee80211_sta *sta);
void mt7921_mac_work(struct work_struct *work);
void mt7921_txp_skb_unmap(struct mt76_dev *dev,
struct mt76_txwi_cache *txwi);
void mt7921_set_stream_he_caps(struct mt7921_phy *phy);
-void mt7921_update_channel(struct mt76_dev *mdev);
+void mt7921_update_channel(struct mt76_phy *mphy);
int mt7921_init_debugfs(struct mt7921_dev *dev);
int mt7921_mcu_uni_tx_ba(struct mt7921_dev *dev,
struct ieee80211_ampdu_params *params,
bool enable);
void mt7921_scan_work(struct work_struct *work);
-u32 mt7921_get_wtbl_info(struct mt7921_dev *dev, u32 wlan_idx);
int mt7921_mcu_uni_bss_ps(struct mt7921_dev *dev, struct ieee80211_vif *vif);
int mt7921_mcu_uni_bss_bcnft(struct mt7921_dev *dev, struct ieee80211_vif *vif,
bool enable);
int mt7921_mcu_set_bss_pm(struct mt7921_dev *dev, struct ieee80211_vif *vif,
bool enable);
+int __mt7921_mcu_drv_pmctrl(struct mt7921_dev *dev);
int mt7921_mcu_drv_pmctrl(struct mt7921_dev *dev);
int mt7921_mcu_fw_pmctrl(struct mt7921_dev *dev);
void mt7921_pm_wake_work(struct work_struct *work);
.rx_poll_complete = mt7921_rx_poll_complete,
.sta_ps = mt7921_sta_ps,
.sta_add = mt7921_mac_sta_add,
+ .sta_assoc = mt7921_mac_sta_assoc,
.sta_remove = mt7921_mac_sta_remove,
.update_survey = mt7921_update_channel,
};
{
struct mt76_dev *mdev = pci_get_drvdata(pdev);
struct mt7921_dev *dev = container_of(mdev, struct mt7921_dev, mt76);
+ struct mt76_connac_pm *pm = &dev->pm;
bool hif_suspend;
int i, err;
- err = mt76_connac_pm_wake(&dev->mphy, &dev->pm);
+ pm->suspended = true;
+ cancel_delayed_work_sync(&pm->ps_work);
+ cancel_work_sync(&pm->wake_work);
+
+ err = mt7921_mcu_drv_pmctrl(dev);
if (err < 0)
- return err;
+ goto restore_suspend;
hif_suspend = !test_bit(MT76_STATE_SUSPEND, &dev->mphy.state);
if (hif_suspend) {
err = mt76_connac_mcu_set_hif_suspend(mdev, true);
if (err)
- return err;
+ goto restore_suspend;
}
- if (!dev->pm.enable)
- mt76_connac_mcu_set_deep_sleep(&dev->mt76, true);
+ /* always enable deep sleep during suspend to reduce
+ * power consumption
+ */
+ mt76_connac_mcu_set_deep_sleep(&dev->mt76, true);
napi_disable(&mdev->tx_napi);
mt76_worker_disable(&mdev->tx_worker);
err = mt7921_mcu_fw_pmctrl(dev);
if (err)
- goto restore;
+ goto restore_napi;
pci_save_state(pdev);
err = pci_set_power_state(pdev, pci_choose_state(pdev, state));
if (err)
- goto restore;
+ goto restore_napi;
return 0;
-restore:
+restore_napi:
mt76_for_each_q_rx(mdev, i) {
napi_enable(&mdev->napi[i]);
}
napi_enable(&mdev->tx_napi);
- if (!dev->pm.enable)
+ if (!pm->ds_enable)
mt76_connac_mcu_set_deep_sleep(&dev->mt76, false);
if (hif_suspend)
mt76_connac_mcu_set_hif_suspend(mdev, false);
+restore_suspend:
+ pm->suspended = false;
+
return err;
}
{
struct mt76_dev *mdev = pci_get_drvdata(pdev);
struct mt7921_dev *dev = container_of(mdev, struct mt7921_dev, mt76);
+ struct mt76_connac_pm *pm = &dev->pm;
int i, err;
+ pm->suspended = false;
err = pci_set_power_state(pdev, PCI_D0);
if (err)
return err;
napi_enable(&mdev->tx_napi);
napi_schedule(&mdev->tx_napi);
- if (!dev->pm.enable)
+ /* restore previous ds setting */
+ if (!pm->ds_enable)
mt76_connac_mcu_set_deep_sleep(&dev->mt76, false);
if (!test_bit(MT76_STATE_SUSPEND, &dev->mphy.state))
if (!q->queued)
wake_up(&dev->tx_wait);
- if (!mcu)
- mt76_txq_schedule(&dev->phy, q->qid);
-
return nframes;
}
struct mt76_sdio *sdio = container_of(w, struct mt76_sdio,
status_worker);
struct mt76_dev *dev = container_of(sdio, struct mt76_dev, sdio);
+ bool resched = false;
int i, nframes;
do {
+ int ndata_frames = 0;
+
nframes = mt76s_process_tx_queue(dev, dev->q_mcu[MT_MCUQ_WM]);
for (i = 0; i <= MT_TXQ_PSD; i++)
- nframes += mt76s_process_tx_queue(dev,
- dev->phy.q_tx[i]);
+ ndata_frames += mt76s_process_tx_queue(dev,
+ dev->phy.q_tx[i]);
+ nframes += ndata_frames;
+ if (ndata_frames > 0)
+ resched = true;
if (dev->drv->tx_status_data &&
!test_and_set_bit(MT76_READING_STATS, &dev->phy.state))
queue_work(dev->wq, &dev->sdio.stat_work);
} while (nframes > 0);
+
+ if (resched)
+ mt76_worker_schedule(&dev->sdio.txrx_worker);
}
static void mt76s_tx_status_data(struct work_struct *work)
q->entry[q->head].skb = tx_info.skb;
q->entry[q->head].buf_sz = len;
+ q->entry[q->head].wcid = 0xffff;
smp_wmb();
mt76_testmode_free_skb(struct mt76_phy *phy)
{
struct mt76_testmode_data *td = &phy->test;
- struct sk_buff *skb = td->tx_skb;
-
- if (!skb)
- return;
- if (skb_has_frag_list(skb)) {
- kfree_skb_list(skb_shinfo(skb)->frag_list);
- skb_shinfo(skb)->frag_list = NULL;
- }
-
- dev_kfree_skb(skb);
+ dev_kfree_skb(td->tx_skb);
td->tx_skb = NULL;
}
frag_len = MT_TXP_MAX_LEN;
frag = alloc_skb(frag_len, GFP_KERNEL);
- if (!frag)
+ if (!frag) {
+ mt76_testmode_free_skb(phy);
+ dev_kfree_skb(head);
return -ENOMEM;
+ }
__skb_put_zero(frag, frag_len);
head->len += frag->len;
head->data_len += frag->len;
- if (*frag_tail) {
- (*frag_tail)->next = frag;
- frag_tail = &frag;
- } else {
- *frag_tail = frag;
- }
+ *frag_tail = frag;
+ frag_tail = &(*frag_tail)->next;
}
mt76_testmode_free_skb(phy);
u64 rx_fcs_error = 0;
int i;
+ if (dev->test_ops->dump_stats) {
+ int ret;
+
+ ret = dev->test_ops->dump_stats(phy, msg);
+ if (ret)
+ return ret;
+ }
+
for (i = 0; i < ARRAY_SIZE(td->rx_stats.packets); i++) {
rx_packets += td->rx_stats.packets[i];
rx_fcs_error += td->rx_stats.fcs_error[i];
MT76_TM_STATS_ATTR_PAD))
return -EMSGSIZE;
- if (dev->test_ops->dump_stats)
- return dev->test_ops->dump_stats(phy, msg);
-
return 0;
}
spin_unlock_bh(&dev->status_list.lock);
+ rcu_read_lock();
while ((skb = __skb_dequeue(list)) != NULL) {
+ struct ieee80211_tx_status status = {
+ .skb = skb,
+ .info = IEEE80211_SKB_CB(skb),
+ };
+ struct mt76_tx_cb *cb = mt76_tx_skb_cb(skb);
+ struct mt76_wcid *wcid;
+
+ wcid = rcu_dereference(dev->wcid[cb->wcid]);
+ if (wcid)
+ status.sta = wcid_to_sta(wcid);
+
hw = mt76_tx_status_get_hw(dev, skb);
- ieee80211_tx_status(hw, skb);
+ ieee80211_tx_status_ext(hw, &status);
}
-
+ rcu_read_unlock();
}
EXPORT_SYMBOL_GPL(mt76_tx_status_unlock);
/* Tx status can be unreliable. if it fails, mark the frame as ACKed */
if (flags & MT_TX_CB_TXS_FAILED) {
- ieee80211_tx_info_clear_status(info);
+ info->status.rates[0].count = 0;
info->status.rates[0].idx = -1;
info->flags |= IEEE80211_TX_STAT_ACK;
}
spin_lock_bh(&dev->status_list.lock);
memset(cb, 0, sizeof(*cb));
- wcid->packet_id = (wcid->packet_id + 1) & MT_PACKET_ID_MASK;
- if (wcid->packet_id == MT_PACKET_ID_NO_ACK ||
- wcid->packet_id == MT_PACKET_ID_NO_SKB)
- wcid->packet_id = MT_PACKET_ID_FIRST;
-
- pid = wcid->packet_id;
+ pid = mt76_get_next_pkt_id(wcid);
cb->wcid = wcid->idx;
cb->pktid = pid;
cb->jiffies = jiffies;
EXPORT_SYMBOL_GPL(mt76_tx_status_check);
static void
-mt76_tx_check_non_aql(struct mt76_dev *dev, u16 wcid_idx, struct sk_buff *skb)
+mt76_tx_check_non_aql(struct mt76_dev *dev, struct mt76_wcid *wcid,
+ struct sk_buff *skb)
{
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
- struct mt76_wcid *wcid;
int pending;
- if (info->tx_time_est)
- return;
-
- if (wcid_idx >= ARRAY_SIZE(dev->wcid))
+ if (!wcid || info->tx_time_est)
return;
- rcu_read_lock();
-
- wcid = rcu_dereference(dev->wcid[wcid_idx]);
- if (wcid) {
- pending = atomic_dec_return(&wcid->non_aql_packets);
- if (pending < 0)
- atomic_cmpxchg(&wcid->non_aql_packets, pending, 0);
- }
-
- rcu_read_unlock();
+ pending = atomic_dec_return(&wcid->non_aql_packets);
+ if (pending < 0)
+ atomic_cmpxchg(&wcid->non_aql_packets, pending, 0);
}
-void mt76_tx_complete_skb(struct mt76_dev *dev, u16 wcid_idx, struct sk_buff *skb)
+void __mt76_tx_complete_skb(struct mt76_dev *dev, u16 wcid_idx, struct sk_buff *skb,
+ struct list_head *free_list)
{
+ struct ieee80211_tx_status status = {
+ .skb = skb,
+ .free_list = free_list,
+ };
+ struct mt76_wcid *wcid = NULL;
struct ieee80211_hw *hw;
struct sk_buff_head list;
- mt76_tx_check_non_aql(dev, wcid_idx, skb);
+ rcu_read_lock();
+
+ if (wcid_idx < ARRAY_SIZE(dev->wcid))
+ wcid = rcu_dereference(dev->wcid[wcid_idx]);
+
+ mt76_tx_check_non_aql(dev, wcid, skb);
#ifdef CONFIG_NL80211_TESTMODE
if (mt76_is_testmode_skb(dev, skb, &hw)) {
wake_up(&dev->tx_wait);
dev_kfree_skb_any(skb);
- return;
+ goto out;
}
#endif
if (!skb->prev) {
hw = mt76_tx_status_get_hw(dev, skb);
- ieee80211_free_txskb(hw, skb);
- return;
+ status.sta = wcid_to_sta(wcid);
+ ieee80211_tx_status_ext(hw, &status);
+ goto out;
}
mt76_tx_status_lock(dev, &list);
__mt76_tx_status_skb_done(dev, skb, MT_TX_CB_DMA_DONE, &list);
mt76_tx_status_unlock(dev, &list);
+
+out:
+ rcu_read_unlock();
}
-EXPORT_SYMBOL_GPL(mt76_tx_complete_skb);
+EXPORT_SYMBOL_GPL(__mt76_tx_complete_skb);
static int
__mt76_tx_queue_skb(struct mt76_phy *phy, int qid, struct sk_buff *skb,
non_aql = !info->tx_time_est;
idx = dev->queue_ops->tx_queue_skb(dev, q, skb, wcid, sta);
- if (idx < 0 || !sta || !non_aql)
+ if (idx < 0 || !sta)
return idx;
wcid = (struct mt76_wcid *)sta->drv_priv;
q->entry[idx].wcid = wcid->idx;
+
+ if (!non_aql)
+ return idx;
+
pending = atomic_inc_return(&wcid->non_aql_packets);
if (stop && pending >= MT_MAX_NON_AQL_PKT)
*stop = true;
skb_set_queue_mapping(skb, qid);
}
- if (!(wcid->tx_info & MT_WCID_TX_INFO_SET))
+ if (wcid && !(wcid->tx_info & MT_WCID_TX_INFO_SET))
ieee80211_get_tx_rates(info->control.vif, sta, skb,
info->control.rates, 1);
q->head = (q->head + 1) % q->ndesc;
q->entry[idx].skb = tx_info.skb;
+ q->entry[idx].wcid = 0xffff;
q->queued++;
return idx;
{ USB_DEVICE(0x2717, 0x4106) },
{ USB_DEVICE(0x2955, 0x0001) },
{ USB_DEVICE(0x2955, 0x1001) },
+ { USB_DEVICE(0x2955, 0x1003) },
{ USB_DEVICE(0x2a5f, 0x1000) },
{ USB_DEVICE(0x7392, 0x7710) },
{ 0, }
wilc->bus_data = spi_priv;
wilc->dev_irq_num = spi->irq;
- wilc->rtc_clk = devm_clk_get(&spi->dev, "rtc_clk");
+ wilc->rtc_clk = devm_clk_get(&spi->dev, "rtc");
if (PTR_ERR_OR_ZERO(wilc->rtc_clk) == -EPROBE_DEFER) {
kfree(spi_priv);
return -EPROBE_DEFER;
* FIXME: if we do not find matching entry, we tell that frame was
* posted without any retries. We need to find a way to fix that
* and provide retry count.
- */
+ */
if (unlikely((aggr == 1 && ampdu == 0 && real_mcs != mcs)) || !match) {
rt2800_rate_from_status(skbdesc, status, rt2x00dev->curr_band);
mcs = real_mcs;
* Beacons and probe responses require the tsf timestamp
* to be inserted into the frame.
*/
- if (ieee80211_is_beacon(hdr->frame_control) ||
- ieee80211_is_probe_resp(hdr->frame_control))
+ if ((ieee80211_is_beacon(hdr->frame_control) ||
+ ieee80211_is_probe_resp(hdr->frame_control)) &&
+ !(tx_info->flags & IEEE80211_TX_CTL_INJECTED))
__set_bit(ENTRY_TXD_REQ_TIMESTAMP, &txdesc->flags);
if ((tx_info->flags & IEEE80211_TX_CTL_FIRST_FRAGMENT) &&
u8 usb_optional_function;
u8 res9[2];
u8 mac_addr[ETH_ALEN]; /* 0xd7 */
- u8 res10[2];
- u8 vendor_name[7];
- u8 res11[2];
- u8 device_name[0x0b]; /* 0xe8 */
- u8 res12[2];
- u8 serial[0x0b]; /* 0xf5 */
- u8 res13[0x30];
+ u8 device_info[80];
+ u8 res11[3];
u8 unknown[0x0d]; /* 0x130 */
- u8 res14[0xc3];
+ u8 res12[0xc3];
};
struct rtl8xxxu_reg8val {
}
}
+static void rtl8192eu_log_next_device_info(struct rtl8xxxu_priv *priv,
+ char *record_name,
+ char *device_info,
+ unsigned int *record_offset)
+{
+ char *record = device_info + *record_offset;
+
+ /* A record is [ total length | 0x03 | value ] */
+ unsigned char l = record[0];
+
+ /*
+ * The whole device info section seems to be 80 characters, make sure
+ * we don't read further.
+ */
+ if (*record_offset + l > 80) {
+ dev_warn(&priv->udev->dev,
+ "invalid record length %d while parsing \"%s\" at offset %u.\n",
+ l, record_name, *record_offset);
+ return;
+ }
+
+ if (l >= 2) {
+ char value[80];
+
+ memcpy(value, &record[2], l - 2);
+ value[l - 2] = '\0';
+ dev_info(&priv->udev->dev, "%s: %s\n", record_name, value);
+ *record_offset = *record_offset + l;
+ } else {
+ dev_info(&priv->udev->dev, "%s not available.\n", record_name);
+ }
+}
+
static int rtl8192eu_parse_efuse(struct rtl8xxxu_priv *priv)
{
struct rtl8192eu_efuse *efuse = &priv->efuse_wifi.efuse8192eu;
+ unsigned int record_offset;
int i;
if (efuse->rtl_id != cpu_to_le16(0x8129))
priv->has_xtalk = 1;
priv->xtalk = priv->efuse_wifi.efuse8192eu.xtal_k & 0x3f;
- dev_info(&priv->udev->dev, "Vendor: %.7s\n", efuse->vendor_name);
- dev_info(&priv->udev->dev, "Product: %.11s\n", efuse->device_name);
- if (memchr_inv(efuse->serial, 0xff, 11))
- dev_info(&priv->udev->dev, "Serial: %.11s\n", efuse->serial);
- else
- dev_info(&priv->udev->dev, "Serial not available.\n");
+ /*
+ * device_info section seems to be laid out as records
+ * [ total length | 0x03 | value ] so:
+ * - vendor length + 2
+ * - 0x03
+ * - vendor string (not null terminated)
+ * - product length + 2
+ * - 0x03
+ * - product string (not null terminated)
+ * Then there is one or 2 0x00 on all the 4 devices I own or found
+ * dumped online.
+ * As previous version of the code handled an optional serial
+ * string, I now assume there may be a third record if the
+ * length is not 0.
+ */
+ record_offset = 0;
+ rtl8192eu_log_next_device_info(priv, "Vendor", efuse->device_info, &record_offset);
+ rtl8192eu_log_next_device_info(priv, "Product", efuse->device_info, &record_offset);
+ rtl8192eu_log_next_device_info(priv, "Serial", efuse->device_info, &record_offset);
if (rtl8xxxu_debug & RTL8XXXU_DEBUG_EFUSE) {
unsigned char *raw = priv->efuse_wifi.raw;
urb_len = skb->len;
pkt_cnt = 0;
+ if (urb_len < sizeof(struct rtl8xxxu_rxdesc16)) {
+ kfree_skb(skb);
+ return RX_TYPE_ERROR;
+ }
+
do {
rx_desc = (struct rtl8xxxu_rxdesc16 *)skb->data;
_rx_desc_le = (__le32 *)skb->data;
* at least cover the rx descriptor
*/
if (pkt_cnt > 1 &&
- urb_len > (pkt_offset + sizeof(struct rtl8xxxu_rxdesc16)))
+ urb_len >= (pkt_offset + sizeof(struct rtl8xxxu_rxdesc16)))
next_skb = skb_clone(skb, GFP_ATOMIC);
rx_status = IEEE80211_SKB_RXCB(skb);
pkt_cnt--;
urb_len -= pkt_offset;
- } while (skb && urb_len > 0 && pkt_cnt > 0);
+ next_skb = NULL;
+ } while (skb && pkt_cnt > 0 &&
+ urb_len >= sizeof(struct rtl8xxxu_rxdesc16));
return RX_TYPE_DATA_PKT;
}
static void rtl_fwevt_wq_callback(struct work_struct *work);
static void rtl_c2hcmd_wq_callback(struct work_struct *work);
-static void _rtl_init_deferred_work(struct ieee80211_hw *hw)
+static int _rtl_init_deferred_work(struct ieee80211_hw *hw)
{
struct rtl_priv *rtlpriv = rtl_priv(hw);
+ struct workqueue_struct *wq;
+
+ wq = alloc_workqueue("%s", 0, 0, rtlpriv->cfg->name);
+ if (!wq)
+ return -ENOMEM;
/* <1> timer */
timer_setup(&rtlpriv->works.watchdog_timer,
rtl_easy_concurrent_retrytimer_callback, 0);
/* <2> work queue */
rtlpriv->works.hw = hw;
- rtlpriv->works.rtl_wq = alloc_workqueue("%s", 0, 0, rtlpriv->cfg->name);
- if (unlikely(!rtlpriv->works.rtl_wq)) {
- pr_err("Failed to allocate work queue\n");
- return;
- }
+ rtlpriv->works.rtl_wq = wq;
INIT_DELAYED_WORK(&rtlpriv->works.watchdog_wq,
rtl_watchdog_wq_callback);
rtl_swlps_rfon_wq_callback);
INIT_DELAYED_WORK(&rtlpriv->works.fwevt_wq, rtl_fwevt_wq_callback);
INIT_DELAYED_WORK(&rtlpriv->works.c2hcmd_wq, rtl_c2hcmd_wq_callback);
+ return 0;
}
void rtl_deinit_deferred_work(struct ieee80211_hw *hw, bool ips_wq)
rtlmac->link_state = MAC80211_NOLINK;
/* <6> init deferred work */
- _rtl_init_deferred_work(hw);
-
- return 0;
+ return _rtl_init_deferred_work(hw);
}
EXPORT_SYMBOL_GPL(rtl_init_core);
btc8821a2ant_ps_tdma(btcoexist,
NORMAL_EXEC, true, 14);
coex_dm->ps_tdma_du_adj_type = 14;
- } else if (max_interval == 3) {
- btc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC, true, 15);
- coex_dm->ps_tdma_du_adj_type = 15;
} else {
btc8821a2ant_ps_tdma(btcoexist,
NORMAL_EXEC, true, 15);
btc8821a2ant_ps_tdma(btcoexist,
NORMAL_EXEC, true, 10);
coex_dm->ps_tdma_du_adj_type = 10;
- } else if (max_interval == 3) {
- btc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC, true, 11);
- coex_dm->ps_tdma_du_adj_type = 11;
} else {
btc8821a2ant_ps_tdma(btcoexist,
NORMAL_EXEC, true, 11);
btc8821a2ant_ps_tdma(btcoexist,
NORMAL_EXEC, true, 6);
coex_dm->ps_tdma_du_adj_type = 6;
- } else if (max_interval == 3) {
- btc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC, true, 7);
- coex_dm->ps_tdma_du_adj_type = 7;
} else {
btc8821a2ant_ps_tdma(btcoexist,
NORMAL_EXEC, true, 7);
btc8821a2ant_ps_tdma(btcoexist,
NORMAL_EXEC, true, 2);
coex_dm->ps_tdma_du_adj_type = 2;
- } else if (max_interval == 3) {
- btc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC, true, 3);
- coex_dm->ps_tdma_du_adj_type = 3;
} else {
btc8821a2ant_ps_tdma(btcoexist,
NORMAL_EXEC, true, 3);
0x4);
}
+ /* preserve identical branches for further fine-tuning */
if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
(bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 23);
0x4);
}
+ /* preserve identical branches for further fine-tuning */
if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
(bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH))
btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 26);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
+ /* preserve identical branches for further fine-tuning */
if (wifi_bw == BTC_WIFI_BW_LEGACY) {
/* for HID at 11b/g mode */
btc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
0x4);
}
+ /* preserve identical branches for further fine-tuning */
if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
(bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 23);
u32 ul_command;
u32 ul_content;
- u32 ul_enc_algo = rtlpriv->cfg->maps[SEC_CAM_AES];
+ u32 ul_enc_algo;
switch (rtlpriv->sec.pairwise_enc_algorithm) {
case WEP40_ENCRYPTION:
}
/**
- * writeLLT - LLT table write access
+ * rtl92c_llt_write - LLT table write access
* @hw: Pointer to the ieee80211_hw structure.
* @address: LLT logical address.
* @data: LLT data content
}
/**
- * rtl92c_init_LLT_table - Init LLT table
+ * rtl92c_init_llt_table - Init LLT table
* @hw: Pointer to the ieee80211_hw structure.
* @boundary: Page boundary.
*
RTPRINT(rtlpriv, FINIT, INIT_IQK,
"path-B / 2.4G LCK\n");
}
- memset(&curvecount_val[0], 0, CV_CURVE_CNT * 2);
+ memset(curvecount_val, 0, sizeof(curvecount_val));
/* Set LC calibration off */
rtl_set_rfreg(hw, (enum radio_path)index, RF_CHNLBW,
0x08000, 0x0);
/* This bit indicate this packet is used for FW download. */
if (tcb_desc->cmd_or_init == DESC_PACKET_TYPE_INIT) {
- /* For firmware downlaod we only need to set LINIP */
+ /* For firmware download we only need to set LINIP */
set_tx_desc_linip(pdesc, tcb_desc->last_inipkt);
/* 92SE must set as 1 for firmware download HW DMA error */
struct rtl_phy *rtlphy = &(rtlpriv->phy);
struct rtl_ps_ctl *ppsc = rtl_psc(rtl_priv(hw));
struct rtl_pci *rtlpci = rtl_pcidev(rtl_pcipriv(hw));
- bool rtstatus = true;
+ bool rtstatus;
int err;
u8 tmp_u1b;
unsigned long flags;
struct rtw_coex *coex = &rtwdev->coex;
u8 *payload = get_payload_from_coex_resp(skb);
- if (payload[0] != COEX_RESP_ACK_BY_WL_FW)
+ if (payload[0] != COEX_RESP_ACK_BY_WL_FW) {
+ dev_kfree_skb_any(skb);
return;
+ }
skb_queue_tail(&coex->queue, skb);
wake_up(&coex->wait);
struct rtw_coex_info_req req = {0};
struct sk_buff *skb;
u8 *payload;
- bool ret = false;
req.op_code = BT_MP_INFO_OP_SCAN_TYPE;
skb = rtw_coex_info_request(rtwdev, &req);
if (!skb)
- goto out;
+ return false;
payload = get_payload_from_coex_resp(skb);
*scan_type = GET_COEX_RESP_BT_SCAN_TYPE(payload);
dev_kfree_skb_any(skb);
- ret = true;
-
-out:
- return ret;
+ return true;
}
static bool rtw_coex_set_lna_constrain_level(struct rtw_dev *rtwdev,
{
struct rtw_coex_info_req req = {0};
struct sk_buff *skb;
- bool ret = false;
req.op_code = BT_MP_INFO_OP_LNA_CONSTRAINT;
req.para1 = lna_constrain_level;
skb = rtw_coex_info_request(rtwdev, &req);
if (!skb)
- goto out;
+ return false;
dev_kfree_skb_any(skb);
- ret = true;
-
-out:
- return ret;
+ return true;
}
#define case_BTSTATUS(src) \
payload = get_payload_from_coex_resp(skb);
*val = GET_COEX_RESP_BT_REG_VAL(payload);
+ dev_kfree_skb_any(skb);
return true;
}
struct rtw_coex_info_req req = {0};
struct sk_buff *skb;
u8 *payload;
- bool ret = false;
req.op_code = BT_MP_INFO_OP_PATCH_VER;
skb = rtw_coex_info_request(rtwdev, &req);
if (!skb)
- goto out;
+ return false;
payload = get_payload_from_coex_resp(skb);
*patch_version = GET_COEX_RESP_BT_PATCH_VER(payload);
- ret = true;
+ dev_kfree_skb_any(skb);
-out:
- return ret;
+ return true;
}
static bool rtw_coex_get_bt_supported_version(struct rtw_dev *rtwdev,
struct rtw_coex_info_req req = {0};
struct sk_buff *skb;
u8 *payload;
- bool ret = false;
req.op_code = BT_MP_INFO_OP_SUPP_VER;
skb = rtw_coex_info_request(rtwdev, &req);
if (!skb)
- goto out;
+ return false;
payload = get_payload_from_coex_resp(skb);
*supported_version = GET_COEX_RESP_BT_SUPP_VER(payload);
- ret = true;
+ dev_kfree_skb_any(skb);
-out:
- return ret;
+ return true;
}
static bool rtw_coex_get_bt_supported_feature(struct rtw_dev *rtwdev,
struct rtw_coex_info_req req = {0};
struct sk_buff *skb;
u8 *payload;
- bool ret = false;
req.op_code = BT_MP_INFO_OP_SUPP_FEAT;
skb = rtw_coex_info_request(rtwdev, &req);
if (!skb)
- goto out;
+ return false;
payload = get_payload_from_coex_resp(skb);
*supported_feature = GET_COEX_RESP_BT_SUPP_FEAT(payload);
- ret = true;
+ dev_kfree_skb_any(skb);
-out:
- return ret;
+ return true;
}
struct rtw_coex_sta_stat_iter_data {
#include "debug.h"
#include "phy.h"
#include "reg.h"
+#include "ps.h"
#ifdef CONFIG_RTW88_DEBUGFS
if (!input)
return -EINVAL;
+ if (test_bit(RTW_FLAG_RESTARTING, rtwdev->flags))
+ return -EINPROGRESS;
+
+ mutex_lock(&rtwdev->mutex);
+ rtw_leave_lps_deep(rtwdev);
rtw_write8(rtwdev, REG_HRCV_MSG, 1);
+ mutex_unlock(&rtwdev->mutex);
return count;
}
RTW_DBG_BF = 0x00000800,
RTW_DBG_WOW = 0x00001000,
RTW_DBG_CFO = 0x00002000,
+ RTW_DBG_PATH_DIV = 0x00004000,
RTW_DBG_ALL = 0xffffffff
};
rtw_iterate_stas_atomic(rtwdev, rtw_fw_ra_report_iter, &ra_data);
}
+struct rtw_beacon_filter_iter_data {
+ struct rtw_dev *rtwdev;
+ u8 *payload;
+};
+
+static void rtw_fw_bcn_filter_notify_vif_iter(void *data, u8 *mac,
+ struct ieee80211_vif *vif)
+{
+ struct rtw_beacon_filter_iter_data *iter_data = data;
+ struct rtw_dev *rtwdev = iter_data->rtwdev;
+ u8 *payload = iter_data->payload;
+ u8 type = GET_BCN_FILTER_NOTIFY_TYPE(payload);
+ u8 event = GET_BCN_FILTER_NOTIFY_EVENT(payload);
+ s8 sig = (s8)GET_BCN_FILTER_NOTIFY_RSSI(payload);
+
+ switch (type) {
+ case BCN_FILTER_NOTIFY_SIGNAL_CHANGE:
+ event = event ? NL80211_CQM_RSSI_THRESHOLD_EVENT_HIGH :
+ NL80211_CQM_RSSI_THRESHOLD_EVENT_LOW;
+ ieee80211_cqm_rssi_notify(vif, event, sig, GFP_KERNEL);
+ break;
+ case BCN_FILTER_CONNECTION_LOSS:
+ ieee80211_connection_loss(vif);
+ break;
+ case BCN_FILTER_CONNECTED:
+ rtwdev->beacon_loss = false;
+ break;
+ case BCN_FILTER_NOTIFY_BEACON_LOSS:
+ rtwdev->beacon_loss = true;
+ rtw_leave_lps(rtwdev);
+ break;
+ }
+}
+
+static void rtw_fw_bcn_filter_notify(struct rtw_dev *rtwdev, u8 *payload,
+ u8 length)
+{
+ struct rtw_beacon_filter_iter_data dev_iter_data;
+
+ dev_iter_data.rtwdev = rtwdev;
+ dev_iter_data.payload = payload;
+ rtw_iterate_vifs(rtwdev, rtw_fw_bcn_filter_notify_vif_iter,
+ &dev_iter_data);
+}
+
+static void rtw_fw_scan_result(struct rtw_dev *rtwdev, u8 *payload,
+ u8 length)
+{
+ struct rtw_dm_info *dm_info = &rtwdev->dm_info;
+
+ dm_info->scan_density = payload[0];
+
+ rtw_dbg(rtwdev, RTW_DBG_FW, "scan.density = %x\n",
+ dm_info->scan_density);
+}
+
void rtw_fw_c2h_cmd_handle(struct rtw_dev *rtwdev, struct sk_buff *skb)
{
struct rtw_c2h_cmd *c2h;
case C2H_WLAN_INFO:
rtw_coex_wl_fwdbginfo_notify(rtwdev, c2h->payload, len);
break;
+ case C2H_BCN_FILTER_NOTIFY:
+ rtw_fw_bcn_filter_notify(rtwdev, c2h->payload, len);
+ break;
case C2H_HALMAC:
rtw_fw_c2h_cmd_handle_ext(rtwdev, skb);
break;
break;
case C2H_WLAN_RFON:
complete(&rtwdev->lps_leave_check);
+ dev_kfree_skb_any(skb);
+ break;
+ case C2H_SCAN_RESULT:
+ complete(&rtwdev->fw_scan_density);
+ rtw_fw_scan_result(rtwdev, c2h->payload, len);
+ dev_kfree_skb_any(skb);
break;
default:
/* pass offset for further operation */
rtw_fw_send_h2c_command(rtwdev, h2c_pkt);
}
+void rtw_fw_beacon_filter_config(struct rtw_dev *rtwdev, bool connect,
+ struct ieee80211_vif *vif)
+{
+ struct ieee80211_bss_conf *bss_conf = &vif->bss_conf;
+ struct ieee80211_sta *sta = ieee80211_find_sta(vif, bss_conf->bssid);
+ static const u8 rssi_min = 0, rssi_max = 100, rssi_offset = 100;
+ struct rtw_sta_info *si =
+ sta ? (struct rtw_sta_info *)sta->drv_priv : NULL;
+ s32 threshold = bss_conf->cqm_rssi_thold + rssi_offset;
+ u8 h2c_pkt[H2C_PKT_SIZE] = {0};
+
+ if (!rtw_fw_feature_check(&rtwdev->fw, FW_FEATURE_BCN_FILTER) || !si)
+ return;
+
+ if (!connect) {
+ SET_H2C_CMD_ID_CLASS(h2c_pkt, H2C_CMD_BCN_FILTER_OFFLOAD_P1);
+ SET_BCN_FILTER_OFFLOAD_P1_ENABLE(h2c_pkt, connect);
+ rtw_fw_send_h2c_command(rtwdev, h2c_pkt);
+
+ return;
+ }
+ SET_H2C_CMD_ID_CLASS(h2c_pkt, H2C_CMD_BCN_FILTER_OFFLOAD_P0);
+ ether_addr_copy(&h2c_pkt[1], bss_conf->bssid);
+ rtw_fw_send_h2c_command(rtwdev, h2c_pkt);
+
+ memset(h2c_pkt, 0, sizeof(h2c_pkt));
+ threshold = clamp_t(s32, threshold, rssi_min, rssi_max);
+ SET_H2C_CMD_ID_CLASS(h2c_pkt, H2C_CMD_BCN_FILTER_OFFLOAD_P1);
+ SET_BCN_FILTER_OFFLOAD_P1_ENABLE(h2c_pkt, connect);
+ SET_BCN_FILTER_OFFLOAD_P1_OFFLOAD_MODE(h2c_pkt,
+ BCN_FILTER_OFFLOAD_MODE_DEFAULT);
+ SET_BCN_FILTER_OFFLOAD_P1_THRESHOLD(h2c_pkt, (u8)threshold);
+ SET_BCN_FILTER_OFFLOAD_P1_BCN_LOSS_CNT(h2c_pkt, BCN_LOSS_CNT);
+ SET_BCN_FILTER_OFFLOAD_P1_MACID(h2c_pkt, si->mac_id);
+ SET_BCN_FILTER_OFFLOAD_P1_HYST(h2c_pkt, bss_conf->cqm_rssi_hyst);
+ SET_BCN_FILTER_OFFLOAD_P1_BCN_INTERVAL(h2c_pkt, bss_conf->beacon_int);
+ rtw_fw_send_h2c_command(rtwdev, h2c_pkt);
+}
+
void rtw_fw_set_pwr_mode(struct rtw_dev *rtwdev)
{
struct rtw_lps_conf *conf = &rtwdev->lps_conf;
rtw_fw_send_h2c_packet(rtwdev, h2c_pkt);
}
+
+void rtw_fw_scan_notify(struct rtw_dev *rtwdev, bool start)
+{
+ u8 h2c_pkt[H2C_PKT_SIZE] = {0};
+
+ SET_H2C_CMD_ID_CLASS(h2c_pkt, H2C_CMD_SCAN);
+ SET_SCAN_START(h2c_pkt, start);
+
+ rtw_fw_send_h2c_command(rtwdev, h2c_pkt);
+}
#define DLFW_BLK_SIZE_LEGACY 4
#define FW_START_ADDR_LEGACY 0x1000
+#define BCN_LOSS_CNT 10
+#define BCN_FILTER_NOTIFY_SIGNAL_CHANGE 0
+#define BCN_FILTER_CONNECTION_LOSS 1
+#define BCN_FILTER_CONNECTED 2
+#define BCN_FILTER_NOTIFY_BEACON_LOSS 3
+
+#define SCAN_NOTIFY_TIMEOUT msecs_to_jiffies(10)
+
enum rtw_c2h_cmd_id {
C2H_CCX_TX_RPT = 0x03,
C2H_BT_INFO = 0x09,
C2H_HW_FEATURE_REPORT = 0x19,
C2H_WLAN_INFO = 0x27,
C2H_WLAN_RFON = 0x32,
+ C2H_BCN_FILTER_NOTIFY = 0x36,
+ C2H_SCAN_RESULT = 0x38,
C2H_HW_FEATURE_DUMP = 0xfd,
C2H_HALMAC = 0xff,
};
FW_FEATURE_LPS_C2H = BIT(1),
FW_FEATURE_LCLK = BIT(2),
FW_FEATURE_PG = BIT(3),
+ FW_FEATURE_BCN_FILTER = BIT(5),
+ FW_FEATURE_NOTIFY_SCAN = BIT(6),
FW_FEATURE_MAX = BIT(31),
};
+enum rtw_beacon_filter_offload_mode {
+ BCN_FILTER_OFFLOAD_MODE_0 = 0,
+ BCN_FILTER_OFFLOAD_MODE_1,
+ BCN_FILTER_OFFLOAD_MODE_2,
+ BCN_FILTER_OFFLOAD_MODE_3,
+
+ BCN_FILTER_OFFLOAD_MODE_DEFAULT = BCN_FILTER_OFFLOAD_MODE_1,
+};
+
struct rtw_coex_info_req {
u8 seq;
u8 op_code;
#define GET_RA_REPORT_BW(c2h_payload) (c2h_payload[6])
#define GET_RA_REPORT_MACID(c2h_payload) (c2h_payload[1])
+#define GET_BCN_FILTER_NOTIFY_TYPE(c2h_payload) (c2h_payload[1] & 0xf)
+#define GET_BCN_FILTER_NOTIFY_EVENT(c2h_payload) (c2h_payload[1] & 0x10)
+#define GET_BCN_FILTER_NOTIFY_RSSI(c2h_payload) (c2h_payload[2] - 100)
+
/* PKT H2C */
#define H2C_PKT_CMD_ID 0xFF
#define H2C_PKT_CATEGORY 0x01
#define H2C_CMD_LPS_PG_INFO 0x2b
#define H2C_CMD_RA_INFO 0x40
#define H2C_CMD_RSSI_MONITOR 0x42
+#define H2C_CMD_BCN_FILTER_OFFLOAD_P0 0x56
+#define H2C_CMD_BCN_FILTER_OFFLOAD_P1 0x57
#define H2C_CMD_WL_PHY_INFO 0x58
+#define H2C_CMD_SCAN 0x59
#define H2C_CMD_COEX_TDMA_TYPE 0x60
#define H2C_CMD_QUERY_BT_INFO 0x61
le32p_replace_bits((__le32 *)(h2c_pkt) + 0x01, value, GENMASK(15, 8))
#define SET_WL_PHY_INFO_RX_EVM(h2c_pkt, value) \
le32p_replace_bits((__le32 *)(h2c_pkt) + 0x01, value, GENMASK(23, 16))
+#define SET_BCN_FILTER_OFFLOAD_P1_MACID(h2c_pkt, value) \
+ le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(15, 8))
+#define SET_BCN_FILTER_OFFLOAD_P1_ENABLE(h2c_pkt, value) \
+ le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, BIT(16))
+#define SET_BCN_FILTER_OFFLOAD_P1_HYST(h2c_pkt, value) \
+ le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(20, 17))
+#define SET_BCN_FILTER_OFFLOAD_P1_OFFLOAD_MODE(h2c_pkt, value) \
+ le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(23, 21))
+#define SET_BCN_FILTER_OFFLOAD_P1_THRESHOLD(h2c_pkt, value) \
+ le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(31, 24))
+#define SET_BCN_FILTER_OFFLOAD_P1_BCN_LOSS_CNT(h2c_pkt, value) \
+ le32p_replace_bits((__le32 *)(h2c_pkt) + 0x01, value, GENMASK(3, 0))
+#define SET_BCN_FILTER_OFFLOAD_P1_BCN_INTERVAL(h2c_pkt, value) \
+ le32p_replace_bits((__le32 *)(h2c_pkt) + 0x01, value, GENMASK(13, 4))
+
+#define SET_SCAN_START(h2c_pkt, value) \
+ le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, BIT(8))
#define SET_PWR_MODE_SET_MODE(h2c_pkt, value) \
le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(14, 8))
return (struct rtw_c2h_cmd *)(skb->data + pkt_offset);
}
+static inline bool rtw_fw_feature_check(struct rtw_fw_state *fw,
+ enum rtw_fw_feature feature)
+{
+ return !!(fw->feature & feature);
+}
+
void rtw_fw_c2h_cmd_rx_irqsafe(struct rtw_dev *rtwdev, u32 pkt_offset,
struct sk_buff *skb);
void rtw_fw_c2h_cmd_handle(struct rtw_dev *rtwdev, struct sk_buff *skb);
void rtw_fw_send_ra_info(struct rtw_dev *rtwdev, struct rtw_sta_info *si);
void rtw_fw_media_status_report(struct rtw_dev *rtwdev, u8 mac_id, bool conn);
void rtw_fw_update_wl_phy_info(struct rtw_dev *rtwdev);
+void rtw_fw_beacon_filter_config(struct rtw_dev *rtwdev, bool connect,
+ struct ieee80211_vif *vif);
int rtw_fw_write_data_rsvd_page(struct rtw_dev *rtwdev, u16 pg_addr,
u8 *buf, u32 size);
void rtw_remove_rsvd_page(struct rtw_dev *rtwdev,
void rtw_fw_c2h_cmd_isr(struct rtw_dev *rtwdev);
int rtw_fw_dump_fifo(struct rtw_dev *rtwdev, u8 fifo_sel, u32 addr, u32 size,
u32 *buffer);
-
+void rtw_fw_scan_notify(struct rtw_dev *rtwdev, bool start);
#endif
u8 port = 0;
u8 bcn_ctrl = 0;
+ if (rtw_fw_feature_check(&rtwdev->fw, FW_FEATURE_BCN_FILTER))
+ vif->driver_flags |= IEEE80211_VIF_BEACON_FILTER |
+ IEEE80211_VIF_SUPPORTS_CQM_RSSI;
rtwvif->port = port;
rtwvif->stats.tx_unicast = 0;
rtwvif->stats.rx_unicast = 0;
rtw_write32_clr(rtwdev, REG_FWHW_TXQ_CTRL,
BIT_EN_BCNQ_DL);
}
+ if (changed & BSS_CHANGED_CQM)
+ rtw_fw_beacon_filter_config(rtwdev, true, vif);
if (changed & BSS_CHANGED_MU_GROUPS)
rtw_chip_set_gid_table(rtwdev, vif, conf);
{
struct rtw_dev *rtwdev = hw->priv;
+ rtw_fw_beacon_filter_config(rtwdev, false, vif);
mutex_lock(&rtwdev->mutex);
rtw_sta_remove(rtwdev, sta, true);
mutex_unlock(&rtwdev->mutex);
rtw_vif_port_config(rtwdev, rtwvif, config);
rtw_coex_scan_notify(rtwdev, COEX_SCAN_START);
+ rtw_core_fw_scan_notify(rtwdev, true);
set_bit(RTW_FLAG_DIG_DISABLE, rtwdev->flags);
set_bit(RTW_FLAG_SCANNING, rtwdev->flags);
clear_bit(RTW_FLAG_SCANNING, rtwdev->flags);
clear_bit(RTW_FLAG_DIG_DISABLE, rtwdev->flags);
+ rtw_core_fw_scan_notify(rtwdev, false);
+
ether_addr_copy(rtwvif->mac_addr, vif->addr);
config |= PORT_SET_MAC_ADDR;
rtw_vif_port_config(rtwdev, rtwvif, config);
static void rtw_ops_mgd_prepare_tx(struct ieee80211_hw *hw,
struct ieee80211_vif *vif,
- u16 duration)
+ struct ieee80211_prep_tx_info *info)
{
struct rtw_dev *rtwdev = hw->priv;
/* Copyright(c) 2018-2019 Realtek Corporation
*/
+#include <linux/devcoredump.h>
+
#include "main.h"
#include "regd.h"
#include "fw.h"
* get that vif and check if device is having traffic more than the
* threshold.
*/
- if (rtwdev->ps_enabled && data.rtwvif && !ps_active)
+ if (rtwdev->ps_enabled && data.rtwvif && !ps_active &&
+ !rtwdev->beacon_loss)
rtw_enter_lps(rtwdev, data.rtwvif->port);
rtwdev->watch_dog_cnt++;
rtw_fw_media_status_report(rtwdev, si->mac_id, true);
rtwdev->sta_cnt++;
+ rtwdev->beacon_loss = false;
rtw_info(rtwdev, "sta %pM joined with macid %d\n",
sta->addr, si->mac_id);
sta->addr, si->mac_id);
}
-static bool rtw_fw_dump_crash_log(struct rtw_dev *rtwdev)
+struct rtw_fwcd_hdr {
+ u32 item;
+ u32 size;
+ u32 padding1;
+ u32 padding2;
+} __packed;
+
+static int rtw_fwcd_prep(struct rtw_dev *rtwdev)
+{
+ struct rtw_chip_info *chip = rtwdev->chip;
+ struct rtw_fwcd_desc *desc = &rtwdev->fw.fwcd_desc;
+ const struct rtw_fwcd_segs *segs = chip->fwcd_segs;
+ u32 prep_size = chip->fw_rxff_size + sizeof(struct rtw_fwcd_hdr);
+ u8 i;
+
+ if (segs) {
+ prep_size += segs->num * sizeof(struct rtw_fwcd_hdr);
+
+ for (i = 0; i < segs->num; i++)
+ prep_size += segs->segs[i];
+ }
+
+ desc->data = vmalloc(prep_size);
+ if (!desc->data)
+ return -ENOMEM;
+
+ desc->size = prep_size;
+ desc->next = desc->data;
+
+ return 0;
+}
+
+static u8 *rtw_fwcd_next(struct rtw_dev *rtwdev, u32 item, u32 size)
+{
+ struct rtw_fwcd_desc *desc = &rtwdev->fw.fwcd_desc;
+ struct rtw_fwcd_hdr *hdr;
+ u8 *next;
+
+ if (!desc->data) {
+ rtw_dbg(rtwdev, RTW_DBG_FW, "fwcd isn't prepared successfully\n");
+ return NULL;
+ }
+
+ next = desc->next + sizeof(struct rtw_fwcd_hdr);
+ if (next - desc->data + size > desc->size) {
+ rtw_dbg(rtwdev, RTW_DBG_FW, "fwcd isn't prepared enough\n");
+ return NULL;
+ }
+
+ hdr = (struct rtw_fwcd_hdr *)(desc->next);
+ hdr->item = item;
+ hdr->size = size;
+ hdr->padding1 = 0x01234567;
+ hdr->padding2 = 0x89abcdef;
+ desc->next = next + size;
+
+ return next;
+}
+
+static void rtw_fwcd_dump(struct rtw_dev *rtwdev)
+{
+ struct rtw_fwcd_desc *desc = &rtwdev->fw.fwcd_desc;
+
+ rtw_dbg(rtwdev, RTW_DBG_FW, "dump fwcd\n");
+
+ /* Data will be freed after lifetime of device coredump. After calling
+ * dev_coredump, data is supposed to be handled by the device coredump
+ * framework. Note that a new dump will be discarded if a previous one
+ * hasn't been released yet.
+ */
+ dev_coredumpv(rtwdev->dev, desc->data, desc->size, GFP_KERNEL);
+}
+
+static void rtw_fwcd_free(struct rtw_dev *rtwdev, bool free_self)
+{
+ struct rtw_fwcd_desc *desc = &rtwdev->fw.fwcd_desc;
+
+ if (free_self) {
+ rtw_dbg(rtwdev, RTW_DBG_FW, "free fwcd by self\n");
+ vfree(desc->data);
+ }
+
+ desc->data = NULL;
+ desc->next = NULL;
+}
+
+static int rtw_fw_dump_crash_log(struct rtw_dev *rtwdev)
{
u32 size = rtwdev->chip->fw_rxff_size;
u32 *buf;
u8 seq;
- bool ret = true;
- buf = vmalloc(size);
+ buf = (u32 *)rtw_fwcd_next(rtwdev, RTW_FWCD_TLV, size);
if (!buf)
- goto exit;
+ return -ENOMEM;
if (rtw_fw_dump_fifo(rtwdev, RTW_FW_FIFO_SEL_RXBUF_FW, 0, size, buf)) {
rtw_dbg(rtwdev, RTW_DBG_FW, "dump fw fifo fail\n");
- goto free_buf;
+ return -EINVAL;
}
if (GET_FW_DUMP_LEN(buf) == 0) {
rtw_dbg(rtwdev, RTW_DBG_FW, "fw crash dump's length is 0\n");
- goto free_buf;
+ return -EINVAL;
}
seq = GET_FW_DUMP_SEQ(buf);
- if (seq > 0 && seq != (rtwdev->fw.prev_dump_seq + 1)) {
+ if (seq > 0) {
rtw_dbg(rtwdev, RTW_DBG_FW,
"fw crash dump's seq is wrong: %d\n", seq);
- goto free_buf;
- }
-
- print_hex_dump(KERN_ERR, "rtw88 fw dump: ", DUMP_PREFIX_OFFSET, 16, 1,
- buf, size, true);
-
- if (GET_FW_DUMP_MORE(buf) == 1) {
- rtwdev->fw.prev_dump_seq = seq;
- ret = false;
+ return -EINVAL;
}
-free_buf:
- vfree(buf);
-exit:
- rtw_write8(rtwdev, REG_MCU_TST_CFG, 0);
-
- return ret;
+ return 0;
}
int rtw_dump_fw(struct rtw_dev *rtwdev, const u32 ocp_src, u32 size,
- const char *prefix_str)
+ u32 fwcd_item)
{
u32 rxff = rtwdev->chip->fw_rxff_size;
u32 dump_size, done_size = 0;
u8 *buf;
int ret;
- buf = vzalloc(size);
+ buf = rtw_fwcd_next(rtwdev, fwcd_item, size);
if (!buf)
return -ENOMEM;
rtw_err(rtwdev,
"ddma fw 0x%x [+0x%x] to fw fifo fail\n",
ocp_src, done_size);
- goto exit;
+ return ret;
}
ret = rtw_fw_dump_fifo(rtwdev, RTW_FW_FIFO_SEL_RXBUF_FW, 0,
rtw_err(rtwdev,
"dump fw 0x%x [+0x%x] from fw fifo fail\n",
ocp_src, done_size);
- goto exit;
+ return ret;
}
size -= dump_size;
done_size += dump_size;
}
- print_hex_dump(KERN_ERR, prefix_str, DUMP_PREFIX_OFFSET, 16, 1,
- buf, done_size, true);
-
-exit:
- vfree(buf);
- return ret;
+ return 0;
}
EXPORT_SYMBOL(rtw_dump_fw);
-int rtw_dump_reg(struct rtw_dev *rtwdev, const u32 addr, const u32 size,
- const char *prefix_str)
+int rtw_dump_reg(struct rtw_dev *rtwdev, const u32 addr, const u32 size)
{
u8 *buf;
u32 i;
return -EINVAL;
}
- buf = vzalloc(size);
+ buf = rtw_fwcd_next(rtwdev, RTW_FWCD_REG, size);
if (!buf)
return -ENOMEM;
for (i = 0; i < size; i += 4)
*(u32 *)(buf + i) = rtw_read32(rtwdev, addr + i);
- print_hex_dump(KERN_ERR, prefix_str, DUMP_PREFIX_OFFSET, 16, 4, buf,
- size, true);
-
- vfree(buf);
return 0;
}
EXPORT_SYMBOL(rtw_dump_reg);
static void __fw_recovery_work(struct rtw_dev *rtwdev)
{
-
- /* rtw_fw_dump_crash_log() returns false indicates that there are
- * still more log to dump. Driver set 0x1cf[7:0] = 0x1 to tell firmware
- * to dump the remaining part of the log, and firmware will trigger an
- * IMR_C2HCMD interrupt to inform driver the log is ready.
- */
- if (!rtw_fw_dump_crash_log(rtwdev)) {
- rtw_write8(rtwdev, REG_HRCV_MSG, 1);
- return;
- }
- rtwdev->fw.prev_dump_seq = 0;
+ int ret = 0;
set_bit(RTW_FLAG_RESTARTING, rtwdev->flags);
- rtw_chip_dump_fw_crash(rtwdev);
+
+ ret = rtw_fwcd_prep(rtwdev);
+ if (ret)
+ goto free;
+ ret = rtw_fw_dump_crash_log(rtwdev);
+ if (ret)
+ goto free;
+ ret = rtw_chip_dump_fw_crash(rtwdev);
+ if (ret)
+ goto free;
+
+ rtw_fwcd_dump(rtwdev);
+free:
+ rtw_fwcd_free(rtwdev, !!ret);
+ rtw_write8(rtwdev, REG_MCU_TST_CFG, 0);
WARN(1, "firmware crash, start reset and recover\n");
return LPS_DEEP_MODE_NONE;
if ((chip->lps_deep_mode_supported & BIT(LPS_DEEP_MODE_PG)) &&
- (fw->feature & FW_FEATURE_PG))
+ rtw_fw_feature_check(fw, FW_FEATURE_PG))
return LPS_DEEP_MODE_PG;
if ((chip->lps_deep_mode_supported & BIT(LPS_DEEP_MODE_LCLK)) &&
- (fw->feature & FW_FEATURE_LCLK))
+ rtw_fw_feature_check(fw, FW_FEATURE_LCLK))
return LPS_DEEP_MODE_LCLK;
return LPS_DEEP_MODE_NONE;
return ret;
}
+void rtw_core_fw_scan_notify(struct rtw_dev *rtwdev, bool start)
+{
+ if (!rtw_fw_feature_check(&rtwdev->fw, FW_FEATURE_NOTIFY_SCAN))
+ return;
+
+ if (start) {
+ rtw_fw_scan_notify(rtwdev, true);
+ } else {
+ reinit_completion(&rtwdev->fw_scan_density);
+ rtw_fw_scan_notify(rtwdev, false);
+ if (!wait_for_completion_timeout(&rtwdev->fw_scan_density,
+ SCAN_NOTIFY_TIMEOUT))
+ rtw_warn(rtwdev, "firmware failed to report density after scan\n");
+ }
+}
+
int rtw_core_start(struct rtw_dev *rtwdev)
{
int ret;
init_waitqueue_head(&rtwdev->coex.wait);
init_completion(&rtwdev->lps_leave_check);
+ init_completion(&rtwdev->fw_scan_density);
rtwdev->sec.total_cam_num = 32;
rtwdev->hal.current_channel = 1;
destroy_workqueue(rtwdev->tx_wq);
spin_lock_irqsave(&rtwdev->tx_report.q_lock, flags);
skb_queue_purge(&rtwdev->tx_report.queue);
+ skb_queue_purge(&rtwdev->coex.queue);
spin_unlock_irqrestore(&rtwdev->tx_report.q_lock, flags);
list_for_each_entry_safe(rsvd_pkt, tmp, &rtwdev->rsvd_page_list,
struct rtw_chip_ops {
int (*mac_init)(struct rtw_dev *rtwdev);
- void (*dump_fw_crash)(struct rtw_dev *rtwdev);
+ int (*dump_fw_crash)(struct rtw_dev *rtwdev);
void (*shutdown)(struct rtw_dev *rtwdev);
int (*read_efuse)(struct rtw_dev *rtwdev, u8 *map);
void (*phy_set_param)(struct rtw_dev *rtwdev);
u8 fixrate_en, u8 *new_rate);
void (*cfo_init)(struct rtw_dev *rtwdev);
void (*cfo_track)(struct rtw_dev *rtwdev);
+ void (*config_tx_path)(struct rtw_dev *rtwdev, u8 tx_path,
+ enum rtw_bb_path tx_path_1ss,
+ enum rtw_bb_path tx_path_cck,
+ bool is_tx2_path);
/* for coex */
void (*coex_set_init)(struct rtw_dev *rtwdev);
RTW_FW_FIFO_MAX,
};
+enum rtw_fwcd_item {
+ RTW_FWCD_TLV,
+ RTW_FWCD_REG,
+ RTW_FWCD_ROM,
+ RTW_FWCD_IMEM,
+ RTW_FWCD_DMEM,
+ RTW_FWCD_EMEM,
+};
+
/* hardware configuration for each IC */
struct rtw_chip_info {
struct rtw_chip_ops *ops;
u8 max_power_index;
u16 fw_fifo_addr[RTW_FW_FIFO_MAX];
+ const struct rtw_fwcd_segs *fwcd_segs;
+
+ u8 default_1ss_tx_path;
+ bool path_div_supported;
bool ht_supported;
bool vht_supported;
u8 lps_deep_mode_supported;
struct rtw_iqk_info iqk;
struct rtw_gapk_info gapk;
bool is_bt_iqk_timeout;
+
+ u8 scan_density;
};
struct rtw_efuse {
const struct rtw_rqpn *rqpn;
};
+struct rtw_fwcd_desc {
+ u32 size;
+ u8 *next;
+ u8 *data;
+};
+
+struct rtw_fwcd_segs {
+ const u32 *segs;
+ u8 num;
+};
+
#define FW_CD_TYPE 0xffff
#define FW_CD_LEN 4
#define FW_CD_VAL 0xaabbccdd
const struct firmware *firmware;
struct rtw_dev *rtwdev;
struct completion completion;
+ struct rtw_fwcd_desc fwcd_desc;
u16 version;
u8 sub_version;
u8 sub_index;
u16 h2c_version;
- u8 prev_dump_seq;
u32 feature;
};
[DESC_RATE_MAX];
};
+struct rtw_path_div {
+ enum rtw_bb_path current_tx_path;
+ u32 path_a_sum;
+ u32 path_b_sum;
+ u16 path_a_cnt;
+ u16 path_b_cnt;
+};
+
struct rtw_dev {
struct ieee80211_hw *hw;
struct device *dev;
/* lps power state & handler work */
struct rtw_lps_conf lps_conf;
bool ps_enabled;
+ bool beacon_loss;
struct completion lps_leave_check;
struct dentry *debugfs;
DECLARE_BITMAP(flags, NUM_OF_RTW_FLAGS);
u8 mp_mode;
+ struct rtw_path_div dm_path_div;
struct rtw_fw_state wow_fw;
struct rtw_wow_param wow;
bool need_rfk;
+ struct completion fw_scan_density;
/* hci related data, must be last */
u8 priv[] __aligned(sizeof(void *));
clear_bit(mac_id, rtwdev->mac_id_map);
}
-static inline void rtw_chip_dump_fw_crash(struct rtw_dev *rtwdev)
+static inline int rtw_chip_dump_fw_crash(struct rtw_dev *rtwdev)
{
if (rtwdev->chip->ops->dump_fw_crash)
- rtwdev->chip->ops->dump_fw_crash(rtwdev);
+ return rtwdev->chip->ops->dump_fw_crash(rtwdev);
+
+ return 0;
}
void rtw_get_channel_params(struct cfg80211_chan_def *chandef,
void rtw_sta_remove(struct rtw_dev *rtwdev, struct ieee80211_sta *sta,
bool fw_exist);
void rtw_fw_recovery(struct rtw_dev *rtwdev);
+void rtw_core_fw_scan_notify(struct rtw_dev *rtwdev, bool start);
int rtw_dump_fw(struct rtw_dev *rtwdev, const u32 ocp_src, u32 size,
- const char *prefix_str);
-int rtw_dump_reg(struct rtw_dev *rtwdev, const u32 addr, const u32 size,
- const char *prefix_str);
+ u32 fwcd_item);
+int rtw_dump_reg(struct rtw_dev *rtwdev, const u32 addr, const u32 size);
#endif
/* Copyright(c) 2018-2019 Realtek Corporation
*/
+#include <linux/dmi.h>
#include <linux/module.h>
#include <linux/pci.h>
#include "main.h"
netif_napi_del(&rtwpci->napi);
}
+enum rtw88_quirk_dis_pci_caps {
+ QUIRK_DIS_PCI_CAP_MSI,
+ QUIRK_DIS_PCI_CAP_ASPM,
+};
+
+static int disable_pci_caps(const struct dmi_system_id *dmi)
+{
+ uintptr_t dis_caps = (uintptr_t)dmi->driver_data;
+
+ if (dis_caps & BIT(QUIRK_DIS_PCI_CAP_MSI))
+ rtw_disable_msi = true;
+ if (dis_caps & BIT(QUIRK_DIS_PCI_CAP_ASPM))
+ rtw_pci_disable_aspm = true;
+
+ return 1;
+}
+
+static const struct dmi_system_id rtw88_pci_quirks[] = {
+ {
+ .callback = disable_pci_caps,
+ .ident = "Protempo Ltd L116HTN6SPW",
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Protempo Ltd"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "L116HTN6SPW"),
+ },
+ .driver_data = (void *)BIT(QUIRK_DIS_PCI_CAP_ASPM),
+ },
+ {}
+};
+
int rtw_pci_probe(struct pci_dev *pdev,
const struct pci_device_id *id)
{
goto err_destroy_pci;
}
+ dmi_check_system(rtw88_pci_quirks);
rtw_pci_phy_cfg(rtwdev);
ret = rtw_register_hw(rtwdev, hw);
chip->ops->cfo_init(rtwdev);
}
+static void rtw_phy_tx_path_div_init(struct rtw_dev *rtwdev)
+{
+ struct rtw_path_div *path_div = &rtwdev->dm_path_div;
+
+ path_div->current_tx_path = rtwdev->chip->default_1ss_tx_path;
+ path_div->path_a_cnt = 0;
+ path_div->path_a_sum = 0;
+ path_div->path_b_cnt = 0;
+ path_div->path_b_sum = 0;
+}
+
void rtw_phy_init(struct rtw_dev *rtwdev)
{
struct rtw_chip_info *chip = rtwdev->chip;
dm_info->iqk.done = false;
rtw_phy_cfo_init(rtwdev);
+ rtw_phy_tx_path_div_init(rtwdev);
}
EXPORT_SYMBOL(rtw_phy_init);
rtw_phy_dig(rtwdev);
rtw_phy_cck_pd(rtwdev);
rtw_phy_ra_track(rtwdev);
+ rtw_phy_tx_path_diversity(rtwdev);
rtw_phy_cfo_track(rtwdev);
rtw_phy_dpk_track(rtwdev);
rtw_phy_pwr_track(rtwdev);
return false;
}
EXPORT_SYMBOL(rtw_phy_pwrtrack_need_iqk);
+
+static void rtw_phy_set_tx_path_by_reg(struct rtw_dev *rtwdev,
+ enum rtw_bb_path tx_path_sel_1ss)
+{
+ struct rtw_path_div *path_div = &rtwdev->dm_path_div;
+ enum rtw_bb_path tx_path_sel_cck = tx_path_sel_1ss;
+ struct rtw_chip_info *chip = rtwdev->chip;
+
+ if (tx_path_sel_1ss == path_div->current_tx_path)
+ return;
+
+ path_div->current_tx_path = tx_path_sel_1ss;
+ rtw_dbg(rtwdev, RTW_DBG_PATH_DIV, "Switch TX path=%s\n",
+ tx_path_sel_1ss == BB_PATH_A ? "A" : "B");
+ chip->ops->config_tx_path(rtwdev, rtwdev->hal.antenna_tx,
+ tx_path_sel_1ss, tx_path_sel_cck, false);
+}
+
+static void rtw_phy_tx_path_div_select(struct rtw_dev *rtwdev)
+{
+ struct rtw_path_div *path_div = &rtwdev->dm_path_div;
+ enum rtw_bb_path path = path_div->current_tx_path;
+ s32 rssi_a = 0, rssi_b = 0;
+
+ if (path_div->path_a_cnt)
+ rssi_a = path_div->path_a_sum / path_div->path_a_cnt;
+ else
+ rssi_a = 0;
+ if (path_div->path_b_cnt)
+ rssi_b = path_div->path_b_sum / path_div->path_b_cnt;
+ else
+ rssi_b = 0;
+
+ if (rssi_a != rssi_b)
+ path = (rssi_a > rssi_b) ? BB_PATH_A : BB_PATH_B;
+
+ path_div->path_a_cnt = 0;
+ path_div->path_a_sum = 0;
+ path_div->path_b_cnt = 0;
+ path_div->path_b_sum = 0;
+ rtw_phy_set_tx_path_by_reg(rtwdev, path);
+}
+
+static void rtw_phy_tx_path_diversity_2ss(struct rtw_dev *rtwdev)
+{
+ if (rtwdev->hal.antenna_rx != BB_PATH_AB) {
+ rtw_dbg(rtwdev, RTW_DBG_PATH_DIV,
+ "[Return] tx_Path_en=%d, rx_Path_en=%d\n",
+ rtwdev->hal.antenna_tx, rtwdev->hal.antenna_rx);
+ return;
+ }
+ if (rtwdev->sta_cnt == 0) {
+ rtw_dbg(rtwdev, RTW_DBG_PATH_DIV, "No Link\n");
+ return;
+ }
+
+ rtw_phy_tx_path_div_select(rtwdev);
+}
+
+void rtw_phy_tx_path_diversity(struct rtw_dev *rtwdev)
+{
+ struct rtw_chip_info *chip = rtwdev->chip;
+
+ if (!chip->path_div_supported)
+ return;
+
+ rtw_phy_tx_path_diversity_2ss(rtwdev);
+}
struct rtw_swing_table *swing_table);
void rtw_phy_parsing_cfo(struct rtw_dev *rtwdev,
struct rtw_rx_pkt_stat *pkt_stat);
+void rtw_phy_tx_path_diversity(struct rtw_dev *rtwdev);
struct rtw_txpwr_lmt_cfg_pair {
u8 regd;
else
fw = &rtwdev->fw;
- if (fw->feature & FW_FEATURE_LPS_C2H)
+ if (rtw_fw_feature_check(fw, FW_FEATURE_LPS_C2H))
ret = __rtw_fw_leave_lps_check_c2h(rtwdev);
else
ret = __rtw_fw_leave_lps_check_reg(rtwdev);
else
fw = &rtwdev->fw;
- if (fw->feature & FW_FEATURE_LPS_C2H)
+ if (rtw_fw_feature_check(fw, FW_FEATURE_LPS_C2H))
reinit_completion(&rtwdev->lps_leave_check);
}
#include "util.h"
#include "bf.h"
#include "efuse.h"
-#include "coex.h"
#define IQK_DONE_8822C 0xaa
rtw_write32_set(rtwdev, REG_ENCCK, BIT_CCK_OFDM_BLK_EN);
}
+static void rtw8822c_bb_reset(struct rtw_dev *rtwdev)
+{
+ rtw_write16_set(rtwdev, REG_SYS_FUNC_EN, BIT_FEN_BB_RSTB);
+ rtw_write16_clr(rtwdev, REG_SYS_FUNC_EN, BIT_FEN_BB_RSTB);
+ rtw_write16_set(rtwdev, REG_SYS_FUNC_EN, BIT_FEN_BB_RSTB);
+}
+
static void rtw8822c_dac_backup_reg(struct rtw_dev *rtwdev,
struct rtw_backup_info *backup,
struct rtw_backup_info *backup_rf)
return 0;
}
-static void rtw8822c_dump_fw_crash(struct rtw_dev *rtwdev)
+#define FWCD_SIZE_REG_8822C 0x2000
+#define FWCD_SIZE_DMEM_8822C 0x10000
+#define FWCD_SIZE_IMEM_8822C 0x10000
+#define FWCD_SIZE_EMEM_8822C 0x20000
+#define FWCD_SIZE_ROM_8822C 0x10000
+
+static const u32 __fwcd_segs_8822c[] = {
+ FWCD_SIZE_REG_8822C,
+ FWCD_SIZE_DMEM_8822C,
+ FWCD_SIZE_IMEM_8822C,
+ FWCD_SIZE_EMEM_8822C,
+ FWCD_SIZE_ROM_8822C,
+};
+
+static const struct rtw_fwcd_segs rtw8822c_fwcd_segs = {
+ .segs = __fwcd_segs_8822c,
+ .num = ARRAY_SIZE(__fwcd_segs_8822c),
+};
+
+static int rtw8822c_dump_fw_crash(struct rtw_dev *rtwdev)
{
- rtw_dump_reg(rtwdev, 0x0, 0x2000, "rtw8822c reg_");
- rtw_dump_fw(rtwdev, OCPBASE_DMEM_88XX, 0x10000, "rtw8822c DMEM_");
- rtw_dump_fw(rtwdev, OCPBASE_IMEM_88XX, 0x10000, "rtw8822c IMEM_");
- rtw_dump_fw(rtwdev, OCPBASE_EMEM_88XX, 0x20000, "rtw8822c EMEM_");
- rtw_dump_fw(rtwdev, OCPBASE_ROM_88XX, 0x10000, "rtw8822c ROM_");
+#define __dump_fw_8822c(_dev, _mem) \
+ rtw_dump_fw(_dev, OCPBASE_ ## _mem ## _88XX, \
+ FWCD_SIZE_ ## _mem ## _8822C, RTW_FWCD_ ## _mem)
+ int ret;
+
+ ret = rtw_dump_reg(rtwdev, 0x0, FWCD_SIZE_REG_8822C);
+ if (ret)
+ return ret;
+ ret = __dump_fw_8822c(rtwdev, DMEM);
+ if (ret)
+ return ret;
+ ret = __dump_fw_8822c(rtwdev, IMEM);
+ if (ret)
+ return ret;
+ ret = __dump_fw_8822c(rtwdev, EMEM);
+ if (ret)
+ return ret;
+ ret = __dump_fw_8822c(rtwdev, ROM);
+ if (ret)
+ return ret;
+
+ return 0;
+
+#undef __dump_fw_8822c
}
static void rtw8822c_rstb_3wire(struct rtw_dev *rtwdev, bool enable)
else
rtw_write32_mask(rtwdev, REG_RXCCKSEL, 0xf0000000, 0x8);
}
+ rtw8822c_bb_reset(rtwdev);
}
static void rtw8822c_config_ofdm_tx_path(struct rtw_dev *rtwdev, u8 tx_path,
- bool is_tx2_path)
+ enum rtw_bb_path tx_path_sel_1ss)
{
if (tx_path == BB_PATH_A) {
rtw_write32_mask(rtwdev, REG_ANTMAP0, 0xff, 0x11);
rtw_write32_mask(rtwdev, REG_ANTMAP0, 0xff, 0x12);
rtw_write32_mask(rtwdev, REG_TXLGMAP, 0xff, 0x0);
} else {
- if (is_tx2_path) {
+ if (tx_path_sel_1ss == BB_PATH_AB) {
rtw_write32_mask(rtwdev, REG_ANTMAP0, 0xff, 0x33);
rtw_write32_mask(rtwdev, REG_TXLGMAP, 0xffff, 0x0404);
- } else {
+ } else if (tx_path_sel_1ss == BB_PATH_B) {
+ rtw_write32_mask(rtwdev, REG_ANTMAP0, 0xff, 0x32);
+ rtw_write32_mask(rtwdev, REG_TXLGMAP, 0xffff, 0x0400);
+ } else if (tx_path_sel_1ss == BB_PATH_A) {
rtw_write32_mask(rtwdev, REG_ANTMAP0, 0xff, 0x31);
rtw_write32_mask(rtwdev, REG_TXLGMAP, 0xffff, 0x0400);
}
}
+ rtw8822c_bb_reset(rtwdev);
}
static void rtw8822c_config_tx_path(struct rtw_dev *rtwdev, u8 tx_path,
+ enum rtw_bb_path tx_path_sel_1ss,
+ enum rtw_bb_path tx_path_cck,
bool is_tx2_path)
{
- rtw8822c_config_cck_tx_path(rtwdev, tx_path, is_tx2_path);
- rtw8822c_config_ofdm_tx_path(rtwdev, tx_path, is_tx2_path);
+ rtw8822c_config_cck_tx_path(rtwdev, tx_path_cck, is_tx2_path);
+ rtw8822c_config_ofdm_tx_path(rtwdev, tx_path, tx_path_sel_1ss);
+ rtw8822c_bb_reset(rtwdev);
}
static void rtw8822c_config_trx_mode(struct rtw_dev *rtwdev, u8 tx_path,
rtw_write32_mask(rtwdev, REG_ORITXCODE2, MASK20BITS, 0x11111);
rtw8822c_config_rx_path(rtwdev, rx_path);
- rtw8822c_config_tx_path(rtwdev, tx_path, is_tx2_path);
+ rtw8822c_config_tx_path(rtwdev, tx_path, BB_PATH_A, BB_PATH_A,
+ is_tx2_path);
rtw8822c_toggle_igi(rtwdev);
}
static void query_phy_status_page1(struct rtw_dev *rtwdev, u8 *phy_status,
struct rtw_rx_pkt_stat *pkt_stat)
{
+ struct rtw_path_div *p_div = &rtwdev->dm_path_div;
struct rtw_dm_info *dm_info = &rtwdev->dm_info;
u8 rxsc, bw;
s8 min_rx_power = -120;
for (path = 0; path <= rtwdev->hal.rf_path_num; path++) {
rssi = rtw_phy_rf_power_2_rssi(&pkt_stat->rx_power[path], 1);
dm_info->rssi[path] = rssi;
+ if (path == RF_PATH_A) {
+ p_div->path_a_sum += rssi;
+ p_div->path_a_cnt++;
+ } else if (path == RF_PATH_B) {
+ p_div->path_b_sum += rssi;
+ p_div->path_b_cnt++;
+ }
dm_info->rx_snr[path] = pkt_stat->rx_snr[path] >> 1;
dm_info->cfo_tail[path] = (pkt_stat->cfo_tail[path] * 5) >> 1;
}
}
-static void rtw8822c_pwr_track_path(struct rtw_dev *rtwdev,
- struct rtw_swing_table *swing_table,
- u8 path)
+static void rtw8822c_pwr_track_stats(struct rtw_dev *rtwdev, u8 path)
{
- struct rtw_dm_info *dm_info = &rtwdev->dm_info;
- u8 thermal_value, delta;
+ u8 thermal_value;
if (rtwdev->efuse.thermal_meter[path] == 0xff)
return;
thermal_value = rtw_read_rf(rtwdev, path, RF_T_METER, 0x7e);
-
rtw_phy_pwrtrack_avg(rtwdev, thermal_value, path);
+}
- delta = rtw_phy_pwrtrack_get_delta(rtwdev, path);
+static void rtw8822c_pwr_track_path(struct rtw_dev *rtwdev,
+ struct rtw_swing_table *swing_table,
+ u8 path)
+{
+ struct rtw_dm_info *dm_info = &rtwdev->dm_info;
+ u8 delta;
+ delta = rtw_phy_pwrtrack_get_delta(rtwdev, path);
dm_info->delta_power_index[path] =
rtw_phy_pwrtrack_get_pwridx(rtwdev, swing_table, path, path,
delta);
-
rtw8822c_pwrtrack_set(rtwdev, path);
}
rtw_phy_config_swing_table(rtwdev, &swing_table);
+ for (i = 0; i < rtwdev->hal.rf_path_num; i++)
+ rtw8822c_pwr_track_stats(rtwdev, i);
if (rtw_phy_pwrtrack_need_lck(rtwdev))
rtw8822c_do_lck(rtwdev);
-
for (i = 0; i < rtwdev->hal.rf_path_num; i++)
rtw8822c_pwr_track_path(rtwdev, &swing_table, i);
-
}
static void rtw8822c_pwr_track(struct rtw_dev *rtwdev)
.cfg_csi_rate = rtw_bf_cfg_csi_rate,
.cfo_init = rtw8822c_cfo_init,
.cfo_track = rtw8822c_cfo_track,
+ .config_tx_path = rtw8822c_config_tx_path,
.coex_set_init = rtw8822c_coex_cfg_init,
.coex_set_ant_switch = NULL,
.band = RTW_BAND_2G | RTW_BAND_5G,
.page_size = 128,
.dig_min = 0x20,
+ .default_1ss_tx_path = BB_PATH_A,
+ .path_div_supported = true,
.ht_supported = true,
.vht_supported = true,
.lps_deep_mode_supported = BIT(LPS_DEEP_MODE_LCLK) | BIT(LPS_DEEP_MODE_PG),
.coex_info_hw_regs = coex_info_hw_regs_8822c,
.fw_fifo_addr = {0x780, 0x700, 0x780, 0x660, 0x650, 0x680},
+ .fwcd_segs = &rtw8822c_fwcd_segs,
};
EXPORT_SYMBOL(rtw8822c_hw_spec);
0x92000002, 0x00000000, 0x40000000, 0x00000000,
0x03F, 0x00010E46,
0x93000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x93000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x93000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x93000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x93000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x93000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x93000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x93000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x94000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x94000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x94000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x94000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x94000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x94000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x94000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x94000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x95000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x95000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x95000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x95000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x95000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x95000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x95000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0x95000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00030246,
+ 0x03F, 0x0003D646,
0xA0000000, 0x00000000,
0x03F, 0x00002A46,
0xB0000000, 0x00000000,
0x92000002, 0x00000000, 0x40000000, 0x00000000,
0x03F, 0x0000EA46,
0x93000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0xA0000000, 0x00000000,
0x03F, 0x00002A46,
0xB0000000, 0x00000000,
0x92000002, 0x00000000, 0x40000000, 0x00000000,
0x03F, 0x0000EA46,
0x93000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0xA0000000, 0x00000000,
0x03F, 0x00002A46,
0xB0000000, 0x00000000,
0x92000002, 0x00000000, 0x40000000, 0x00000000,
0x03F, 0x0000EA46,
0x93000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0xA0000000, 0x00000000,
0x03F, 0x00002A46,
0xB0000000, 0x00000000,
0x92000002, 0x00000000, 0x40000000, 0x00000000,
0x03F, 0x0000EA46,
0x93000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x93000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x94000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000001, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000002, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000003, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000004, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000005, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000006, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000015, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0x95000016, 0x00000000, 0x40000000, 0x00000000,
- 0x03F, 0x00031E46,
+ 0x03F, 0x0003D646,
0xA0000000, 0x00000000,
0x03F, 0x00002A46,
0xB0000000, 0x00000000,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x93000002, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x93000003, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x93000004, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x93000005, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x93000006, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x93000015, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x93000016, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000001, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000002, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000003, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000004, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000005, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000006, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000015, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000016, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000001, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000002, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000003, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000004, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000005, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000006, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000015, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000016, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x000008C8,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x000008CB,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x000008CE,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x000008D1,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x000008D4,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000DD1,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0xA0000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000487,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x93000002, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x93000003, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x93000004, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x93000005, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x93000006, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x93000015, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x93000016, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000001, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000002, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000003, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000004, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000005, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000006, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000015, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x94000016, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000001, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000002, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000003, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000004, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000005, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000006, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000015, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0x95000016, 0x00000000, 0x40000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000467,
0x033, 0x00000062,
0x03F, 0x00000908,
0x033, 0x00000063,
- 0x03F, 0x00000D09,
+ 0x03F, 0x00000CC6,
0x033, 0x00000064,
- 0x03F, 0x00000D49,
+ 0x03F, 0x00000CC9,
0x033, 0x00000065,
- 0x03F, 0x00000D8A,
+ 0x03F, 0x00000CCC,
0x033, 0x00000066,
- 0x03F, 0x00000DEB,
+ 0x03F, 0x00000CCF,
0x033, 0x00000067,
- 0x03F, 0x00000DEE,
+ 0x03F, 0x00000CD2,
0x033, 0x00000068,
- 0x03F, 0x00000DF1,
+ 0x03F, 0x00000CD5,
0x033, 0x00000069,
- 0x03F, 0x00000DF4,
+ 0x03F, 0x00000DD4,
0x033, 0x0000006A,
- 0x03F, 0x00000DF7,
+ 0x03F, 0x00000DD7,
0xA0000000, 0x00000000,
0x033, 0x00000060,
0x03F, 0x00000487,
{
struct rndis_wlan_private *priv = get_rndis_wlan_priv(usbdev);
u8 bssid[ETH_ALEN];
- int ret;
if (!priv->radio_on)
return false;
- ret = get_bssid(usbdev, bssid);
-
- return (ret == 0 && !is_zero_ether_addr(bssid));
+ return (get_bssid(usbdev, bssid) == 0 && !is_zero_ether_addr(bssid));
}
static int disassociate(struct usbnet *usbdev, bool reset_ssid)
wh->frame_control |= cpu_to_le16(RSI_SET_PS_ENABLE);
if ((!(info->flags & IEEE80211_TX_INTFL_DONT_ENCRYPT)) &&
- (common->secinfo.security_enable)) {
+ info->control.hw_key) {
if (rsi_is_cipher_wep(common))
ieee80211_size += 4;
else
}
if (common->band == NL80211_BAND_2GHZ)
- bcn_frm->bbp_info |= cpu_to_le16(RSI_RATE_1);
+ bcn_frm->rate_info |= cpu_to_le16(RSI_RATE_1);
else
- bcn_frm->bbp_info |= cpu_to_le16(RSI_RATE_6);
+ bcn_frm->rate_info |= cpu_to_le16(RSI_RATE_6);
if (mac_bcn->data[tim_offset + 2] == 0)
bcn_frm->frame_info |= cpu_to_le16(RSI_DATA_DESC_DTIM_BEACON);
common->cqm_info.rssi_hyst);
}
+ if (changed & BSS_CHANGED_BEACON_INT) {
+ rsi_dbg(INFO_ZONE, "%s: Changed Beacon interval: %d\n",
+ __func__, bss_conf->beacon_int);
+ if (common->beacon_interval != bss->beacon_int) {
+ common->beacon_interval = bss->beacon_int;
+ if (vif->type == NL80211_IFTYPE_AP) {
+ struct vif_priv *vif_info = (struct vif_priv *)vif->drv_priv;
+
+ rsi_set_vap_capabilities(common, RSI_OPMODE_AP,
+ vif->addr, vif_info->vap_id,
+ VAP_UPDATE);
+ }
+ }
+ adapter->ps_info.listen_interval =
+ bss->beacon_int * adapter->ps_info.num_bcns_per_lis_int;
+ }
+
if ((changed & BSS_CHANGED_BEACON_ENABLED) &&
((vif->type == NL80211_IFTYPE_AP) ||
(vif->type == NL80211_IFTYPE_P2P_GO))) {
mutex_lock(&common->mutex);
switch (cmd) {
case SET_KEY:
- secinfo->security_enable = true;
status = rsi_hal_key_config(hw, vif, key, sta);
if (status) {
mutex_unlock(&common->mutex);
break;
case DISABLE_KEY:
- if (vif->type == NL80211_IFTYPE_STATION)
- secinfo->security_enable = false;
rsi_dbg(ERR_ZONE, "%s: RSI del key\n", __func__);
memset(key, 0, sizeof(struct ieee80211_key_conf));
status = rsi_hal_key_config(hw, vif, key, sta);
}
/**
- * This function sends a frame to block/unblock
- * data queues in the firmware
+ * rsi_send_block_unblock_frame() - This function sends a frame to block/unblock
+ * data queues in the firmware
*
* @common: Pointer to the driver private structure.
* @block_event: Event block if true, unblock if false
RSI_WIFI_MGMT_Q);
cmd_frame->desc.desc_dword0.frame_type = WOWLAN_CONFIG_PARAMS;
cmd_frame->host_sleep_status = sleep_status;
- if (common->secinfo.security_enable &&
- common->secinfo.gtk_cipher)
+ if (common->secinfo.gtk_cipher)
flags |= RSI_WOW_GTK_REKEY;
if (sleep_status)
cmd_frame->wow_flags = flags;
};
struct security_info {
- bool security_enable;
u32 ptk_cipher;
u32 gtk_cipher;
};
{ SDIO_DEVICE(SDIO_VENDOR_ID_STE, SDIO_DEVICE_ID_STE_CW1200) },
{ /* end: all zeroes */ },
};
+MODULE_DEVICE_TABLE(sdio, cw1200_sdio_ids);
/* hwbus_ops implemetation */
if (req->n_ssids > WSM_SCAN_MAX_NUM_OF_SSIDS)
return -EINVAL;
- /* will be unlocked in cw1200_scan_work() */
- down(&priv->scan.lock);
- mutex_lock(&priv->conf_mutex);
-
frame.skb = ieee80211_probereq_get(hw, priv->vif->addr, NULL, 0,
req->ie_len);
- if (!frame.skb) {
- mutex_unlock(&priv->conf_mutex);
- up(&priv->scan.lock);
+ if (!frame.skb)
return -ENOMEM;
- }
if (req->ie_len)
skb_put_data(frame.skb, req->ie, req->ie_len);
+ /* will be unlocked in cw1200_scan_work() */
+ down(&priv->scan.lock);
+ mutex_lock(&priv->conf_mutex);
+
ret = wsm_set_template_frame(priv, &frame);
if (!ret) {
/* Host want to be the probe responder. */
ret = wsm_set_probe_responder(priv, true);
}
if (ret) {
- dev_kfree_skb(frame.skb);
mutex_unlock(&priv->conf_mutex);
up(&priv->scan.lock);
+ dev_kfree_skb(frame.skb);
return ret;
}
++priv->scan.n_ssids;
}
- dev_kfree_skb(frame.skb);
mutex_unlock(&priv->conf_mutex);
+ dev_kfree_skb(frame.skb);
queue_work(priv->workqueue, &priv->scan.work);
return 0;
}
#include "acx.h"
/**
- * send command to firmware
+ * wl1251_cmd_send - Send command to firmware
*
* @wl: wl struct
* @id: command id
}
/**
- * send test command to firmware
+ * wl1251_cmd_test - Send test command to firmware
*
* @wl: wl struct
* @buf: buffer containing the command, with all headers, must work with dma
}
/**
- * read acx from firmware
+ * wl1251_cmd_interrogate - Read acx from firmware
*
* @wl: wl struct
* @id: acx id
}
/**
- * write acx value to firmware
+ * wl1251_cmd_configure - Write acx value to firmware
*
* @wl: wl struct
* @id: acx id
cmd->channels[i].channel = channels[i]->hw_value;
}
- cmd->params.ssid_len = ssid_len;
- if (ssid)
- memcpy(cmd->params.ssid, ssid, ssid_len);
+ if (ssid) {
+ int len = clamp_val(ssid_len, 0, IEEE80211_MAX_SSID_LEN);
+
+ cmd->params.ssid_len = len;
+ memcpy(cmd->params.ssid, ssid, len);
+ }
ret = wl1251_cmd_send(wl, CMD_SCAN, cmd, sizeof(*cmd));
if (ret < 0) {
u32 mac1, mac2;
int ret;
+ /* Device may be in ELP from the bootloader or kexec */
+ ret = wlcore_write32(wl, WL12XX_WELP_ARM_COMMAND, WELP_ARM_COMMAND_VAL);
+ if (ret < 0)
+ goto out;
+
+ usleep_range(500000, 700000);
+
ret = wlcore_set_partition(wl, &wl->ptable[PART_DRPW]);
if (ret < 0)
goto out;
/**
- * send test command to firmware
+ * wl1271_cmd_test - send test command to firmware
*
* @wl: wl struct
* @buf: buffer containing the command, with all headers, must work with dma
EXPORT_SYMBOL_GPL(wl1271_cmd_test);
/**
- * read acx from firmware
+ * wl1271_cmd_interrogate - read acx from firmware
*
* @wl: wl struct
* @id: acx id
}
/**
- * write acx value to firmware
+ * wlcore_cmd_configure_failsafe - write acx value to firmware
*
* @wl: wl struct
* @id: acx id
u8 *buffer;
u32 internal_fw_addrbase = WL18XX_DATA_RAM_BASE_ADDRESS;
u32 addr = WL18XX_LOGGER_SDIO_BUFF_ADDR;
- u32 end_buff_addr = WL18XX_LOGGER_SDIO_BUFF_ADDR +
- WL18XX_LOGGER_BUFF_OFFSET;
+ u32 addr_ptr;
+ u32 buff_start_ptr;
+ u32 buff_read_ptr;
+ u32 buff_end_ptr;
u32 available_len;
u32 actual_len;
- u32 clear_addr;
+ u32 clear_ptr;
size_t len;
u32 start_loc;
buffer = kzalloc(WL18XX_LOGGER_SDIO_BUFF_MAX, GFP_KERNEL);
if (!buffer) {
wl1271_error("Fail to allocate fw logger memory");
- fw_log.actual_buff_size = cpu_to_le32(0);
+ actual_len = 0;
goto out;
}
if (ret < 0) {
wl1271_error("Fail to read logger buffer, error_id = %d",
ret);
- fw_log.actual_buff_size = cpu_to_le32(0);
+ actual_len = 0;
goto free_out;
}
memcpy(&fw_log, buffer, sizeof(fw_log));
- if (le32_to_cpu(fw_log.actual_buff_size) == 0)
+ actual_len = le32_to_cpu(fw_log.actual_buff_size);
+ if (actual_len == 0)
goto free_out;
- actual_len = le32_to_cpu(fw_log.actual_buff_size);
- start_loc = (le32_to_cpu(fw_log.buff_read_ptr) -
- internal_fw_addrbase) - addr;
- end_buff_addr += le32_to_cpu(fw_log.max_buff_size);
- available_len = end_buff_addr -
- (le32_to_cpu(fw_log.buff_read_ptr) -
- internal_fw_addrbase);
- actual_len = min(actual_len, available_len);
- len = actual_len;
+ /* Calculate the internal pointer to the fwlog structure */
+ addr_ptr = internal_fw_addrbase + addr;
+
+ /* Calculate the internal pointers to the start and end of log buffer */
+ buff_start_ptr = addr_ptr + WL18XX_LOGGER_BUFF_OFFSET;
+ buff_end_ptr = buff_start_ptr + le32_to_cpu(fw_log.max_buff_size);
+ /* Read the read pointer and validate it */
+ buff_read_ptr = le32_to_cpu(fw_log.buff_read_ptr);
+ if (buff_read_ptr < buff_start_ptr ||
+ buff_read_ptr >= buff_end_ptr) {
+ wl1271_error("buffer read pointer out of bounds: %x not in (%x-%x)\n",
+ buff_read_ptr, buff_start_ptr, buff_end_ptr);
+ goto free_out;
+ }
+
+ start_loc = buff_read_ptr - addr_ptr;
+ available_len = buff_end_ptr - buff_read_ptr;
+
+ /* Copy initial part up to the end of ring buffer */
+ len = min(actual_len, available_len);
wl12xx_copy_fwlog(wl, &buffer[start_loc], len);
- clear_addr = addr + start_loc + le32_to_cpu(fw_log.actual_buff_size) +
- internal_fw_addrbase;
+ clear_ptr = addr_ptr + start_loc + actual_len;
+ if (clear_ptr == buff_end_ptr)
+ clear_ptr = buff_start_ptr;
- len = le32_to_cpu(fw_log.actual_buff_size) - len;
+ /* Copy any remaining part from beginning of ring buffer */
+ len = actual_len - len;
if (len) {
wl12xx_copy_fwlog(wl,
&buffer[WL18XX_LOGGER_BUFF_OFFSET],
len);
- clear_addr = addr + WL18XX_LOGGER_BUFF_OFFSET + len +
- internal_fw_addrbase;
- }
-
- /* double check that clear address and write pointer are the same */
- if (clear_addr != le32_to_cpu(fw_log.buff_write_ptr)) {
- wl1271_error("Calculate of clear addr Clear = %x, write = %x",
- clear_addr, le32_to_cpu(fw_log.buff_write_ptr));
+ clear_ptr = addr_ptr + WL18XX_LOGGER_BUFF_OFFSET + len;
}
- /* indicate FW about Clear buffer */
+ /* Update the read pointer */
ret = wlcore_write32(wl, addr + WL18XX_LOGGER_READ_POINT_OFFSET,
- fw_log.buff_write_ptr);
+ clear_ptr);
free_out:
kfree(buffer);
out:
- return le32_to_cpu(fw_log.actual_buff_size);
+ return actual_len;
}
EXPORT_SYMBOL_GPL(wlcore_event_fw_logger);
* the firmware filters so that all multicast packets are passed
* This is mandatory for MDNS based discovery protocols
*/
- if (wlvif->bss_type == BSS_TYPE_AP_BSS) {
- if (*total & FIF_ALLMULTI) {
+ if (wlvif->bss_type == BSS_TYPE_AP_BSS) {
+ if (*total & FIF_ALLMULTI) {
ret = wl1271_acx_group_address_tbl(wl, wlvif,
false,
NULL, 0);
#include "debug.h"
#include "sysfs.h"
-static ssize_t wl1271_sysfs_show_bt_coex_state(struct device *dev,
- struct device_attribute *attr,
- char *buf)
+static ssize_t bt_coex_state_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
{
struct wl1271 *wl = dev_get_drvdata(dev);
ssize_t len;
}
-static ssize_t wl1271_sysfs_store_bt_coex_state(struct device *dev,
- struct device_attribute *attr,
- const char *buf, size_t count)
+static ssize_t bt_coex_state_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct wl1271 *wl = dev_get_drvdata(dev);
unsigned long res;
return count;
}
-static DEVICE_ATTR(bt_coex_state, 0644,
- wl1271_sysfs_show_bt_coex_state,
- wl1271_sysfs_store_bt_coex_state);
+static DEVICE_ATTR_RW(bt_coex_state);
-static ssize_t wl1271_sysfs_show_hw_pg_ver(struct device *dev,
- struct device_attribute *attr,
- char *buf)
+static ssize_t hw_pg_ver_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
{
struct wl1271 *wl = dev_get_drvdata(dev);
ssize_t len;
return len;
}
-static DEVICE_ATTR(hw_pg_ver, 0444, wl1271_sysfs_show_hw_pg_ver, NULL);
+static DEVICE_ATTR_RO(hw_pg_ver);
static ssize_t wl1271_sysfs_read_fwlog(struct file *filp, struct kobject *kobj,
struct bin_attribute *bin_attr,
zd_workqueue = create_singlethread_workqueue(driver.name);
if (zd_workqueue == NULL) {
- printk(KERN_ERR "%s couldn't create workqueue\n", driver.name);
+ pr_err("%s couldn't create workqueue\n", driver.name);
return -ENOMEM;
}
r = usb_register(&driver);
if (r) {
destroy_workqueue(zd_workqueue);
- printk(KERN_ERR "%s usb_register() failed. Error number %d\n",
+ pr_err("%s usb_register() failed. Error number %d\n",
driver.name, r);
return r;
}
# Wireless WAN device configuration
#
-menuconfig WWAN
- bool "Wireless WAN"
- help
- This section contains Wireless WAN configuration for WWAN framework
- and drivers.
-
-if WWAN
+menu "Wireless WAN"
-config WWAN_CORE
+config WWAN
tristate "WWAN Driver Core"
help
Say Y here if you want to use the WWAN driver core. This driver
To compile this driver as a module, choose M here: the module will be
called wwan.
+if WWAN
+
+config WWAN_HWSIM
+ tristate "Simulated WWAN device"
+ help
+ This driver is a developer testing tool that can be used to test WWAN
+ framework.
+
+ To compile this driver as a module, choose M here: the module will be
+ called wwan_hwsim. If unsure, say N.
+
config MHI_WWAN_CTRL
tristate "MHI WWAN control driver for QCOM-based PCIe modems"
- select WWAN_CORE
depends on MHI_BUS
help
MHI WWAN CTRL allows QCOM-based PCIe modems to expose different modem
To compile this driver as a module, choose M here: the module will be
called mhi_wwan_ctrl.
+config RPMSG_WWAN_CTRL
+ tristate "RPMSG WWAN control driver"
+ depends on RPMSG
+ help
+ RPMSG WWAN CTRL allows modems available via RPMSG channels to expose
+ different modem protocols/ports to userspace, including AT and QMI.
+ These protocols can be accessed directly from userspace
+ (e.g. AT commands) or via libraries/tools (e.g. libqmi, libqcdm...).
+
+ This is mainly used for modems integrated into many Qualcomm SoCs,
+ e.g. for AT and QMI on Qualcomm MSM8916 or MSM8974. Note that many
+ newer Qualcomm SoCs (e.g. SDM845) still provide an AT port through
+ this driver but the QMI messages can only be sent through
+ QRTR network sockets (CONFIG_QRTR).
+
+ To compile this driver as a module, choose M here: the module will be
+ called rpmsg_wwan_ctrl.
+
+config IOSM
+ tristate "IOSM Driver for Intel M.2 WWAN Device"
+ depends on INTEL_IOMMU
+ help
+ This driver enables Intel M.2 WWAN Device communication.
+
+ If you have one of those Intel M.2 WWAN Modules and wish to use it in
+ Linux say Y/M here.
+
+ If unsure, say N.
+
endif # WWAN
+
+endmenu
# Makefile for the Linux WWAN device drivers.
#
-obj-$(CONFIG_WWAN_CORE) += wwan.o
+obj-$(CONFIG_WWAN) += wwan.o
wwan-objs += wwan_core.o
+obj-$(CONFIG_WWAN_HWSIM) += wwan_hwsim.o
+
obj-$(CONFIG_MHI_WWAN_CTRL) += mhi_wwan_ctrl.o
+obj-$(CONFIG_RPMSG_WWAN_CTRL) += rpmsg_wwan_ctrl.o
+obj-$(CONFIG_IOSM) += iosm/
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only)
+#
+# Copyright (C) 2020-21 Intel Corporation.
+#
+
+iosm-y = \
+ iosm_ipc_task_queue.o \
+ iosm_ipc_imem.o \
+ iosm_ipc_imem_ops.o \
+ iosm_ipc_mmio.o \
+ iosm_ipc_port.o \
+ iosm_ipc_wwan.o \
+ iosm_ipc_uevent.o \
+ iosm_ipc_pm.o \
+ iosm_ipc_pcie.o \
+ iosm_ipc_irq.o \
+ iosm_ipc_chnl_cfg.o \
+ iosm_ipc_protocol.o \
+ iosm_ipc_protocol_ops.o \
+ iosm_ipc_mux.o \
+ iosm_ipc_mux_codec.o
+
+obj-$(CONFIG_IOSM) := iosm.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include <linux/wwan.h>
+
+#include "iosm_ipc_chnl_cfg.h"
+
+/* Max. sizes of a downlink buffers */
+#define IPC_MEM_MAX_DL_FLASH_BUF_SIZE (16 * 1024)
+#define IPC_MEM_MAX_DL_LOOPBACK_SIZE (1 * 1024 * 1024)
+#define IPC_MEM_MAX_DL_AT_BUF_SIZE 2048
+#define IPC_MEM_MAX_DL_RPC_BUF_SIZE (32 * 1024)
+#define IPC_MEM_MAX_DL_MBIM_BUF_SIZE IPC_MEM_MAX_DL_RPC_BUF_SIZE
+
+/* Max. transfer descriptors for a pipe. */
+#define IPC_MEM_MAX_TDS_FLASH_DL 3
+#define IPC_MEM_MAX_TDS_FLASH_UL 6
+#define IPC_MEM_MAX_TDS_AT 4
+#define IPC_MEM_MAX_TDS_RPC 4
+#define IPC_MEM_MAX_TDS_MBIM IPC_MEM_MAX_TDS_RPC
+#define IPC_MEM_MAX_TDS_LOOPBACK 11
+
+/* Accumulation backoff usec */
+#define IRQ_ACC_BACKOFF_OFF 0
+
+/* MUX acc backoff 1ms */
+#define IRQ_ACC_BACKOFF_MUX 1000
+
+/* Modem channel configuration table
+ * Always reserve element zero for flash channel.
+ */
+static struct ipc_chnl_cfg modem_cfg[] = {
+ /* IP Mux */
+ { IPC_MEM_IP_CHL_ID_0, IPC_MEM_PIPE_0, IPC_MEM_PIPE_1,
+ IPC_MEM_MAX_TDS_MUX_LITE_UL, IPC_MEM_MAX_TDS_MUX_LITE_DL,
+ IPC_MEM_MAX_DL_MUX_LITE_BUF_SIZE, WWAN_PORT_UNKNOWN },
+ /* RPC - 0 */
+ { IPC_MEM_CTRL_CHL_ID_1, IPC_MEM_PIPE_2, IPC_MEM_PIPE_3,
+ IPC_MEM_MAX_TDS_RPC, IPC_MEM_MAX_TDS_RPC,
+ IPC_MEM_MAX_DL_RPC_BUF_SIZE, WWAN_PORT_UNKNOWN },
+ /* IAT0 */
+ { IPC_MEM_CTRL_CHL_ID_2, IPC_MEM_PIPE_4, IPC_MEM_PIPE_5,
+ IPC_MEM_MAX_TDS_AT, IPC_MEM_MAX_TDS_AT, IPC_MEM_MAX_DL_AT_BUF_SIZE,
+ WWAN_PORT_AT },
+ /* Trace */
+ { IPC_MEM_CTRL_CHL_ID_3, IPC_MEM_PIPE_6, IPC_MEM_PIPE_7,
+ IPC_MEM_TDS_TRC, IPC_MEM_TDS_TRC, IPC_MEM_MAX_DL_TRC_BUF_SIZE,
+ WWAN_PORT_UNKNOWN },
+ /* IAT1 */
+ { IPC_MEM_CTRL_CHL_ID_4, IPC_MEM_PIPE_8, IPC_MEM_PIPE_9,
+ IPC_MEM_MAX_TDS_AT, IPC_MEM_MAX_TDS_AT, IPC_MEM_MAX_DL_AT_BUF_SIZE,
+ WWAN_PORT_AT },
+ /* Loopback */
+ { IPC_MEM_CTRL_CHL_ID_5, IPC_MEM_PIPE_10, IPC_MEM_PIPE_11,
+ IPC_MEM_MAX_TDS_LOOPBACK, IPC_MEM_MAX_TDS_LOOPBACK,
+ IPC_MEM_MAX_DL_LOOPBACK_SIZE, WWAN_PORT_UNKNOWN },
+ /* MBIM Channel */
+ { IPC_MEM_CTRL_CHL_ID_6, IPC_MEM_PIPE_12, IPC_MEM_PIPE_13,
+ IPC_MEM_MAX_TDS_MBIM, IPC_MEM_MAX_TDS_MBIM,
+ IPC_MEM_MAX_DL_MBIM_BUF_SIZE, WWAN_PORT_MBIM },
+};
+
+int ipc_chnl_cfg_get(struct ipc_chnl_cfg *chnl_cfg, int index)
+{
+ int array_size = ARRAY_SIZE(modem_cfg);
+
+ if (index >= array_size) {
+ pr_err("index: %d and array_size %d", index, array_size);
+ return -ECHRNG;
+ }
+
+ if (index == IPC_MEM_MUX_IP_CH_IF_ID)
+ chnl_cfg->accumulation_backoff = IRQ_ACC_BACKOFF_MUX;
+ else
+ chnl_cfg->accumulation_backoff = IRQ_ACC_BACKOFF_OFF;
+
+ chnl_cfg->ul_nr_of_entries = modem_cfg[index].ul_nr_of_entries;
+ chnl_cfg->dl_nr_of_entries = modem_cfg[index].dl_nr_of_entries;
+ chnl_cfg->dl_buf_size = modem_cfg[index].dl_buf_size;
+ chnl_cfg->id = modem_cfg[index].id;
+ chnl_cfg->ul_pipe = modem_cfg[index].ul_pipe;
+ chnl_cfg->dl_pipe = modem_cfg[index].dl_pipe;
+ chnl_cfg->wwan_port_type = modem_cfg[index].wwan_port_type;
+
+ return 0;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation
+ */
+
+#ifndef IOSM_IPC_CHNL_CFG_H
+#define IOSM_IPC_CHNL_CFG_H
+
+#include "iosm_ipc_mux.h"
+
+/* Number of TDs on the trace channel */
+#define IPC_MEM_TDS_TRC 32
+
+/* Trace channel TD buffer size. */
+#define IPC_MEM_MAX_DL_TRC_BUF_SIZE 8192
+
+/* Channel ID */
+enum ipc_channel_id {
+ IPC_MEM_IP_CHL_ID_0 = 0,
+ IPC_MEM_CTRL_CHL_ID_1,
+ IPC_MEM_CTRL_CHL_ID_2,
+ IPC_MEM_CTRL_CHL_ID_3,
+ IPC_MEM_CTRL_CHL_ID_4,
+ IPC_MEM_CTRL_CHL_ID_5,
+ IPC_MEM_CTRL_CHL_ID_6,
+};
+
+/**
+ * struct ipc_chnl_cfg - IPC channel configuration structure
+ * @id: Interface ID
+ * @ul_pipe: Uplink datastream
+ * @dl_pipe: Downlink datastream
+ * @ul_nr_of_entries: Number of Transfer descriptor uplink pipe
+ * @dl_nr_of_entries: Number of Transfer descriptor downlink pipe
+ * @dl_buf_size: Downlink buffer size
+ * @wwan_port_type: Wwan subsystem port type
+ * @accumulation_backoff: Time in usec for data accumalation
+ */
+struct ipc_chnl_cfg {
+ u32 id;
+ u32 ul_pipe;
+ u32 dl_pipe;
+ u32 ul_nr_of_entries;
+ u32 dl_nr_of_entries;
+ u32 dl_buf_size;
+ u32 wwan_port_type;
+ u32 accumulation_backoff;
+};
+
+/**
+ * ipc_chnl_cfg_get - Get pipe configuration.
+ * @chnl_cfg: Array of ipc_chnl_cfg struct
+ * @index: Channel index (upto MAX_CHANNELS)
+ *
+ * Return: 0 on success and failure value on error
+ */
+int ipc_chnl_cfg_get(struct ipc_chnl_cfg *chnl_cfg, int index);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include <linux/delay.h>
+
+#include "iosm_ipc_chnl_cfg.h"
+#include "iosm_ipc_imem.h"
+#include "iosm_ipc_port.h"
+
+/* Check the wwan ips if it is valid with Channel as input. */
+static int ipc_imem_check_wwan_ips(struct ipc_mem_channel *chnl)
+{
+ if (chnl)
+ return chnl->ctype == IPC_CTYPE_WWAN &&
+ chnl->if_id == IPC_MEM_MUX_IP_CH_IF_ID;
+ return false;
+}
+
+static int ipc_imem_msg_send_device_sleep(struct iosm_imem *ipc_imem, u32 state)
+{
+ union ipc_msg_prep_args prep_args = {
+ .sleep.target = 1,
+ .sleep.state = state,
+ };
+
+ ipc_imem->device_sleep = state;
+
+ return ipc_protocol_tq_msg_send(ipc_imem->ipc_protocol,
+ IPC_MSG_PREP_SLEEP, &prep_args, NULL);
+}
+
+static bool ipc_imem_dl_skb_alloc(struct iosm_imem *ipc_imem,
+ struct ipc_pipe *pipe)
+{
+ /* limit max. nr of entries */
+ if (pipe->nr_of_queued_entries >= pipe->max_nr_of_queued_entries)
+ return false;
+
+ return ipc_protocol_dl_td_prepare(ipc_imem->ipc_protocol, pipe);
+}
+
+/* This timer handler will retry DL buff allocation if a pipe has no free buf
+ * and gives doorbell if TD is available
+ */
+static int ipc_imem_tq_td_alloc_timer(struct iosm_imem *ipc_imem, int arg,
+ void *msg, size_t size)
+{
+ bool new_buffers_available = false;
+ bool retry_allocation = false;
+ int i;
+
+ for (i = 0; i < IPC_MEM_MAX_CHANNELS; i++) {
+ struct ipc_pipe *pipe = &ipc_imem->channels[i].dl_pipe;
+
+ if (!pipe->is_open || pipe->nr_of_queued_entries > 0)
+ continue;
+
+ while (ipc_imem_dl_skb_alloc(ipc_imem, pipe))
+ new_buffers_available = true;
+
+ if (pipe->nr_of_queued_entries == 0)
+ retry_allocation = true;
+ }
+
+ if (new_buffers_available)
+ ipc_protocol_doorbell_trigger(ipc_imem->ipc_protocol,
+ IPC_HP_DL_PROCESS);
+
+ if (retry_allocation) {
+ ipc_imem->hrtimer_period =
+ ktime_set(0, IPC_TD_ALLOC_TIMER_PERIOD_MS * 1000 * 1000ULL);
+ if (!hrtimer_active(&ipc_imem->td_alloc_timer))
+ hrtimer_start(&ipc_imem->td_alloc_timer,
+ ipc_imem->hrtimer_period,
+ HRTIMER_MODE_REL);
+ }
+ return 0;
+}
+
+static enum hrtimer_restart ipc_imem_td_alloc_timer_cb(struct hrtimer *hr_timer)
+{
+ struct iosm_imem *ipc_imem =
+ container_of(hr_timer, struct iosm_imem, td_alloc_timer);
+ /* Post an async tasklet event to trigger HP update Doorbell */
+ ipc_task_queue_send_task(ipc_imem, ipc_imem_tq_td_alloc_timer, 0, NULL,
+ 0, false);
+ return HRTIMER_NORESTART;
+}
+
+/* Fast update timer tasklet handler to trigger HP update */
+static int ipc_imem_tq_fast_update_timer_cb(struct iosm_imem *ipc_imem, int arg,
+ void *msg, size_t size)
+{
+ ipc_protocol_doorbell_trigger(ipc_imem->ipc_protocol,
+ IPC_HP_FAST_TD_UPD_TMR);
+
+ return 0;
+}
+
+static enum hrtimer_restart
+ipc_imem_fast_update_timer_cb(struct hrtimer *hr_timer)
+{
+ struct iosm_imem *ipc_imem =
+ container_of(hr_timer, struct iosm_imem, fast_update_timer);
+ /* Post an async tasklet event to trigger HP update Doorbell */
+ ipc_task_queue_send_task(ipc_imem, ipc_imem_tq_fast_update_timer_cb, 0,
+ NULL, 0, false);
+ return HRTIMER_NORESTART;
+}
+
+static int ipc_imem_setup_cp_mux_cap_init(struct iosm_imem *ipc_imem,
+ struct ipc_mux_config *cfg)
+{
+ ipc_mmio_update_cp_capability(ipc_imem->mmio);
+
+ if (!ipc_imem->mmio->has_mux_lite) {
+ dev_err(ipc_imem->dev, "Failed to get Mux capability.");
+ return -EINVAL;
+ }
+
+ cfg->protocol = MUX_LITE;
+
+ cfg->ul_flow = (ipc_imem->mmio->has_ul_flow_credit == 1) ?
+ MUX_UL_ON_CREDITS :
+ MUX_UL;
+
+ /* The instance ID is same as channel ID because this is been reused
+ * for channel alloc function.
+ */
+ cfg->instance_id = IPC_MEM_MUX_IP_CH_IF_ID;
+ cfg->nr_sessions = IPC_MEM_MUX_IP_SESSION_ENTRIES;
+
+ return 0;
+}
+
+void ipc_imem_msg_send_feature_set(struct iosm_imem *ipc_imem,
+ unsigned int reset_enable, bool atomic_ctx)
+{
+ union ipc_msg_prep_args prep_args = { .feature_set.reset_enable =
+ reset_enable };
+
+ if (atomic_ctx)
+ ipc_protocol_tq_msg_send(ipc_imem->ipc_protocol,
+ IPC_MSG_PREP_FEATURE_SET, &prep_args,
+ NULL);
+ else
+ ipc_protocol_msg_send(ipc_imem->ipc_protocol,
+ IPC_MSG_PREP_FEATURE_SET, &prep_args);
+}
+
+void ipc_imem_td_update_timer_start(struct iosm_imem *ipc_imem)
+{
+ /* Use the TD update timer only in the runtime phase */
+ if (!ipc_imem->enter_runtime || ipc_imem->td_update_timer_suspended) {
+ /* trigger the doorbell irq on CP directly. */
+ ipc_protocol_doorbell_trigger(ipc_imem->ipc_protocol,
+ IPC_HP_TD_UPD_TMR_START);
+ return;
+ }
+
+ if (!hrtimer_active(&ipc_imem->tdupdate_timer)) {
+ ipc_imem->hrtimer_period =
+ ktime_set(0, TD_UPDATE_DEFAULT_TIMEOUT_USEC * 1000ULL);
+ if (!hrtimer_active(&ipc_imem->tdupdate_timer))
+ hrtimer_start(&ipc_imem->tdupdate_timer,
+ ipc_imem->hrtimer_period,
+ HRTIMER_MODE_REL);
+ }
+}
+
+void ipc_imem_hrtimer_stop(struct hrtimer *hr_timer)
+{
+ if (hrtimer_active(hr_timer))
+ hrtimer_cancel(hr_timer);
+}
+
+bool ipc_imem_ul_write_td(struct iosm_imem *ipc_imem)
+{
+ struct ipc_mem_channel *channel;
+ struct sk_buff_head *ul_list;
+ bool hpda_pending = false;
+ bool forced_hpdu = false;
+ struct ipc_pipe *pipe;
+ int i;
+
+ /* Analyze the uplink pipe of all active channels. */
+ for (i = 0; i < ipc_imem->nr_of_channels; i++) {
+ channel = &ipc_imem->channels[i];
+
+ if (channel->state != IMEM_CHANNEL_ACTIVE)
+ continue;
+
+ pipe = &channel->ul_pipe;
+
+ /* Get the reference to the skbuf accumulator list. */
+ ul_list = &channel->ul_list;
+
+ /* Fill the transfer descriptor with the uplink buffer info. */
+ hpda_pending |= ipc_protocol_ul_td_send(ipc_imem->ipc_protocol,
+ pipe, ul_list);
+
+ /* forced HP update needed for non data channels */
+ if (hpda_pending && !ipc_imem_check_wwan_ips(channel))
+ forced_hpdu = true;
+ }
+
+ if (forced_hpdu) {
+ hpda_pending = false;
+ ipc_protocol_doorbell_trigger(ipc_imem->ipc_protocol,
+ IPC_HP_UL_WRITE_TD);
+ }
+
+ return hpda_pending;
+}
+
+void ipc_imem_ipc_init_check(struct iosm_imem *ipc_imem)
+{
+ int timeout = IPC_MODEM_BOOT_TIMEOUT;
+
+ ipc_imem->ipc_requested_state = IPC_MEM_DEVICE_IPC_INIT;
+
+ /* Trigger the CP interrupt to enter the init state. */
+ ipc_doorbell_fire(ipc_imem->pcie, IPC_DOORBELL_IRQ_IPC,
+ IPC_MEM_DEVICE_IPC_INIT);
+ /* Wait for the CP update. */
+ do {
+ if (ipc_mmio_get_ipc_state(ipc_imem->mmio) ==
+ ipc_imem->ipc_requested_state) {
+ /* Prepare the MMIO space */
+ ipc_mmio_config(ipc_imem->mmio);
+
+ /* Trigger the CP irq to enter the running state. */
+ ipc_imem->ipc_requested_state =
+ IPC_MEM_DEVICE_IPC_RUNNING;
+ ipc_doorbell_fire(ipc_imem->pcie, IPC_DOORBELL_IRQ_IPC,
+ IPC_MEM_DEVICE_IPC_RUNNING);
+
+ return;
+ }
+ msleep(20);
+ } while (--timeout);
+
+ /* timeout */
+ dev_err(ipc_imem->dev, "%s: ipc_status(%d) ne. IPC_MEM_DEVICE_IPC_INIT",
+ ipc_imem_phase_get_string(ipc_imem->phase),
+ ipc_mmio_get_ipc_state(ipc_imem->mmio));
+
+ ipc_uevent_send(ipc_imem->dev, UEVENT_MDM_TIMEOUT);
+}
+
+/* Analyze the packet type and distribute it. */
+static void ipc_imem_dl_skb_process(struct iosm_imem *ipc_imem,
+ struct ipc_pipe *pipe, struct sk_buff *skb)
+{
+ u16 port_id;
+
+ if (!skb)
+ return;
+
+ /* An AT/control or IP packet is expected. */
+ switch (pipe->channel->ctype) {
+ case IPC_CTYPE_CTRL:
+ port_id = pipe->channel->channel_id;
+
+ /* Pass the packet to the wwan layer. */
+ wwan_port_rx(ipc_imem->ipc_port[port_id]->iosm_port, skb);
+ break;
+
+ case IPC_CTYPE_WWAN:
+ if (pipe->channel->if_id == IPC_MEM_MUX_IP_CH_IF_ID)
+ ipc_mux_dl_decode(ipc_imem->mux, skb);
+ break;
+ default:
+ dev_err(ipc_imem->dev, "Invalid channel type");
+ break;
+ }
+}
+
+/* Process the downlink data and pass them to the char or net layer. */
+static void ipc_imem_dl_pipe_process(struct iosm_imem *ipc_imem,
+ struct ipc_pipe *pipe)
+{
+ s32 cnt = 0, processed_td_cnt = 0;
+ struct ipc_mem_channel *channel;
+ u32 head = 0, tail = 0;
+ bool processed = false;
+ struct sk_buff *skb;
+
+ channel = pipe->channel;
+
+ ipc_protocol_get_head_tail_index(ipc_imem->ipc_protocol, pipe, &head,
+ &tail);
+ if (pipe->old_tail != tail) {
+ if (pipe->old_tail < tail)
+ cnt = tail - pipe->old_tail;
+ else
+ cnt = pipe->nr_of_entries - pipe->old_tail + tail;
+ }
+
+ processed_td_cnt = cnt;
+
+ /* Seek for pipes with pending DL data. */
+ while (cnt--) {
+ skb = ipc_protocol_dl_td_process(ipc_imem->ipc_protocol, pipe);
+
+ /* Analyze the packet type and distribute it. */
+ ipc_imem_dl_skb_process(ipc_imem, pipe, skb);
+ }
+
+ /* try to allocate new empty DL SKbs from head..tail - 1*/
+ while (ipc_imem_dl_skb_alloc(ipc_imem, pipe))
+ processed = true;
+
+ if (processed && !ipc_imem_check_wwan_ips(channel)) {
+ /* Force HP update for non IP channels */
+ ipc_protocol_doorbell_trigger(ipc_imem->ipc_protocol,
+ IPC_HP_DL_PROCESS);
+ processed = false;
+
+ /* If Fast Update timer is already running then stop */
+ ipc_imem_hrtimer_stop(&ipc_imem->fast_update_timer);
+ }
+
+ /* Any control channel process will get immediate HP update.
+ * Start Fast update timer only for IP channel if all the TDs were
+ * used in last process.
+ */
+ if (processed && (processed_td_cnt == pipe->nr_of_entries - 1)) {
+ ipc_imem->hrtimer_period =
+ ktime_set(0, FORCE_UPDATE_DEFAULT_TIMEOUT_USEC * 1000ULL);
+ hrtimer_start(&ipc_imem->fast_update_timer,
+ ipc_imem->hrtimer_period, HRTIMER_MODE_REL);
+ }
+
+ if (ipc_imem->app_notify_dl_pend)
+ complete(&ipc_imem->dl_pend_sem);
+}
+
+/* process open uplink pipe */
+static void ipc_imem_ul_pipe_process(struct iosm_imem *ipc_imem,
+ struct ipc_pipe *pipe)
+{
+ struct ipc_mem_channel *channel;
+ u32 tail = 0, head = 0;
+ struct sk_buff *skb;
+ s32 cnt = 0;
+
+ channel = pipe->channel;
+
+ /* Get the internal phase. */
+ ipc_protocol_get_head_tail_index(ipc_imem->ipc_protocol, pipe, &head,
+ &tail);
+
+ if (pipe->old_tail != tail) {
+ if (pipe->old_tail < tail)
+ cnt = tail - pipe->old_tail;
+ else
+ cnt = pipe->nr_of_entries - pipe->old_tail + tail;
+ }
+
+ /* Free UL buffers. */
+ while (cnt--) {
+ skb = ipc_protocol_ul_td_process(ipc_imem->ipc_protocol, pipe);
+
+ if (!skb)
+ continue;
+
+ /* If the user app was suspended in uplink direction - blocking
+ * write, resume it.
+ */
+ if (IPC_CB(skb)->op_type == UL_USR_OP_BLOCKED)
+ complete(&channel->ul_sem);
+
+ /* Free the skbuf element. */
+ if (IPC_CB(skb)->op_type == UL_MUX_OP_ADB) {
+ if (channel->if_id == IPC_MEM_MUX_IP_CH_IF_ID)
+ ipc_mux_ul_encoded_process(ipc_imem->mux, skb);
+ else
+ dev_err(ipc_imem->dev,
+ "OP Type is UL_MUX, unknown if_id %d",
+ channel->if_id);
+ } else {
+ ipc_pcie_kfree_skb(ipc_imem->pcie, skb);
+ }
+ }
+
+ /* Trace channel stats for IP UL pipe. */
+ if (ipc_imem_check_wwan_ips(pipe->channel))
+ ipc_mux_check_n_restart_tx(ipc_imem->mux);
+
+ if (ipc_imem->app_notify_ul_pend)
+ complete(&ipc_imem->ul_pend_sem);
+}
+
+/* Executes the irq. */
+static void ipc_imem_rom_irq_exec(struct iosm_imem *ipc_imem)
+{
+ struct ipc_mem_channel *channel;
+
+ if (ipc_imem->flash_channel_id < 0) {
+ ipc_imem->rom_exit_code = IMEM_ROM_EXIT_FAIL;
+ dev_err(ipc_imem->dev, "Missing flash app:%d",
+ ipc_imem->flash_channel_id);
+ return;
+ }
+
+ ipc_imem->rom_exit_code = ipc_mmio_get_rom_exit_code(ipc_imem->mmio);
+
+ /* Wake up the flash app to continue or to terminate depending
+ * on the CP ROM exit code.
+ */
+ channel = &ipc_imem->channels[ipc_imem->flash_channel_id];
+ complete(&channel->ul_sem);
+}
+
+/* Execute the UL bundle timer actions, generating the doorbell irq. */
+static int ipc_imem_tq_td_update_timer_cb(struct iosm_imem *ipc_imem, int arg,
+ void *msg, size_t size)
+{
+ ipc_protocol_doorbell_trigger(ipc_imem->ipc_protocol,
+ IPC_HP_TD_UPD_TMR);
+ return 0;
+}
+
+/* Consider link power management in the runtime phase. */
+static void ipc_imem_slp_control_exec(struct iosm_imem *ipc_imem)
+{
+ /* link will go down, Test pending UL packets.*/
+ if (ipc_protocol_pm_dev_sleep_handle(ipc_imem->ipc_protocol) &&
+ hrtimer_active(&ipc_imem->tdupdate_timer)) {
+ /* Generate the doorbell irq. */
+ ipc_imem_tq_td_update_timer_cb(ipc_imem, 0, NULL, 0);
+ /* Stop the TD update timer. */
+ ipc_imem_hrtimer_stop(&ipc_imem->tdupdate_timer);
+ /* Stop the fast update timer. */
+ ipc_imem_hrtimer_stop(&ipc_imem->fast_update_timer);
+ }
+}
+
+/* Execute startup timer and wait for delayed start (e.g. NAND) */
+static int ipc_imem_tq_startup_timer_cb(struct iosm_imem *ipc_imem, int arg,
+ void *msg, size_t size)
+{
+ /* Update & check the current operation phase. */
+ if (ipc_imem_phase_update(ipc_imem) != IPC_P_RUN)
+ return -EIO;
+
+ if (ipc_mmio_get_ipc_state(ipc_imem->mmio) ==
+ IPC_MEM_DEVICE_IPC_UNINIT) {
+ ipc_imem->ipc_requested_state = IPC_MEM_DEVICE_IPC_INIT;
+
+ ipc_doorbell_fire(ipc_imem->pcie, IPC_DOORBELL_IRQ_IPC,
+ IPC_MEM_DEVICE_IPC_INIT);
+
+ ipc_imem->hrtimer_period = ktime_set(0, 100 * 1000UL * 1000ULL);
+ /* reduce period to 100 ms to check for mmio init state */
+ if (!hrtimer_active(&ipc_imem->startup_timer))
+ hrtimer_start(&ipc_imem->startup_timer,
+ ipc_imem->hrtimer_period,
+ HRTIMER_MODE_REL);
+ } else if (ipc_mmio_get_ipc_state(ipc_imem->mmio) ==
+ IPC_MEM_DEVICE_IPC_INIT) {
+ /* Startup complete - disable timer */
+ ipc_imem_hrtimer_stop(&ipc_imem->startup_timer);
+
+ /* Prepare the MMIO space */
+ ipc_mmio_config(ipc_imem->mmio);
+ ipc_imem->ipc_requested_state = IPC_MEM_DEVICE_IPC_RUNNING;
+ ipc_doorbell_fire(ipc_imem->pcie, IPC_DOORBELL_IRQ_IPC,
+ IPC_MEM_DEVICE_IPC_RUNNING);
+ }
+
+ return 0;
+}
+
+static enum hrtimer_restart ipc_imem_startup_timer_cb(struct hrtimer *hr_timer)
+{
+ enum hrtimer_restart result = HRTIMER_NORESTART;
+ struct iosm_imem *ipc_imem =
+ container_of(hr_timer, struct iosm_imem, startup_timer);
+
+ if (ktime_to_ns(ipc_imem->hrtimer_period)) {
+ hrtimer_forward(&ipc_imem->startup_timer, ktime_get(),
+ ipc_imem->hrtimer_period);
+ result = HRTIMER_RESTART;
+ }
+
+ ipc_task_queue_send_task(ipc_imem, ipc_imem_tq_startup_timer_cb, 0,
+ NULL, 0, false);
+ return result;
+}
+
+/* Get the CP execution stage */
+static enum ipc_mem_exec_stage
+ipc_imem_get_exec_stage_buffered(struct iosm_imem *ipc_imem)
+{
+ return (ipc_imem->phase == IPC_P_RUN &&
+ ipc_imem->ipc_status == IPC_MEM_DEVICE_IPC_RUNNING) ?
+ ipc_protocol_get_ap_exec_stage(ipc_imem->ipc_protocol) :
+ ipc_mmio_get_exec_stage(ipc_imem->mmio);
+}
+
+/* Callback to send the modem ready uevent */
+static int ipc_imem_send_mdm_rdy_cb(struct iosm_imem *ipc_imem, int arg,
+ void *msg, size_t size)
+{
+ enum ipc_mem_exec_stage exec_stage =
+ ipc_imem_get_exec_stage_buffered(ipc_imem);
+
+ if (exec_stage == IPC_MEM_EXEC_STAGE_RUN)
+ ipc_uevent_send(ipc_imem->dev, UEVENT_MDM_READY);
+
+ return 0;
+}
+
+/* This function is executed in a task context via an ipc_worker object,
+ * as the creation or removal of device can't be done from tasklet.
+ */
+static void ipc_imem_run_state_worker(struct work_struct *instance)
+{
+ struct ipc_chnl_cfg chnl_cfg_port = { 0 };
+ struct ipc_mux_config mux_cfg;
+ struct iosm_imem *ipc_imem;
+ u8 ctrl_chl_idx = 0;
+
+ ipc_imem = container_of(instance, struct iosm_imem, run_state_worker);
+
+ if (ipc_imem->phase != IPC_P_RUN) {
+ dev_err(ipc_imem->dev,
+ "Modem link down. Exit run state worker.");
+ return;
+ }
+
+ if (!ipc_imem_setup_cp_mux_cap_init(ipc_imem, &mux_cfg))
+ ipc_imem->mux = ipc_mux_init(&mux_cfg, ipc_imem);
+
+ ipc_imem_wwan_channel_init(ipc_imem, mux_cfg.protocol);
+ if (ipc_imem->mux)
+ ipc_imem->mux->wwan = ipc_imem->wwan;
+
+ while (ctrl_chl_idx < IPC_MEM_MAX_CHANNELS) {
+ if (!ipc_chnl_cfg_get(&chnl_cfg_port, ctrl_chl_idx)) {
+ ipc_imem->ipc_port[ctrl_chl_idx] = NULL;
+ if (chnl_cfg_port.wwan_port_type != WWAN_PORT_UNKNOWN) {
+ ipc_imem_channel_init(ipc_imem, IPC_CTYPE_CTRL,
+ chnl_cfg_port,
+ IRQ_MOD_OFF);
+ ipc_imem->ipc_port[ctrl_chl_idx] =
+ ipc_port_init(ipc_imem, chnl_cfg_port);
+ }
+ }
+ ctrl_chl_idx++;
+ }
+
+ ipc_task_queue_send_task(ipc_imem, ipc_imem_send_mdm_rdy_cb, 0, NULL, 0,
+ false);
+
+ /* Complete all memory stores before setting bit */
+ smp_mb__before_atomic();
+
+ set_bit(FULLY_FUNCTIONAL, &ipc_imem->flag);
+
+ /* Complete all memory stores after setting bit */
+ smp_mb__after_atomic();
+}
+
+static void ipc_imem_handle_irq(struct iosm_imem *ipc_imem, int irq)
+{
+ enum ipc_mem_device_ipc_state curr_ipc_status;
+ enum ipc_phase old_phase, phase;
+ bool retry_allocation = false;
+ bool ul_pending = false;
+ int ch_id, i;
+
+ if (irq != IMEM_IRQ_DONT_CARE)
+ ipc_imem->ev_irq_pending[irq] = false;
+
+ /* Get the internal phase. */
+ old_phase = ipc_imem->phase;
+
+ if (old_phase == IPC_P_OFF_REQ) {
+ dev_dbg(ipc_imem->dev,
+ "[%s]: Ignoring MSI. Deinit sequence in progress!",
+ ipc_imem_phase_get_string(old_phase));
+ return;
+ }
+
+ /* Update the phase controlled by CP. */
+ phase = ipc_imem_phase_update(ipc_imem);
+
+ switch (phase) {
+ case IPC_P_RUN:
+ if (!ipc_imem->enter_runtime) {
+ /* Excute the transition from flash/boot to runtime. */
+ ipc_imem->enter_runtime = 1;
+
+ /* allow device to sleep, default value is
+ * IPC_HOST_SLEEP_ENTER_SLEEP
+ */
+ ipc_imem_msg_send_device_sleep(ipc_imem,
+ ipc_imem->device_sleep);
+
+ ipc_imem_msg_send_feature_set(ipc_imem,
+ IPC_MEM_INBAND_CRASH_SIG,
+ true);
+ }
+
+ curr_ipc_status =
+ ipc_protocol_get_ipc_status(ipc_imem->ipc_protocol);
+
+ /* check ipc_status change */
+ if (ipc_imem->ipc_status != curr_ipc_status) {
+ ipc_imem->ipc_status = curr_ipc_status;
+
+ if (ipc_imem->ipc_status ==
+ IPC_MEM_DEVICE_IPC_RUNNING) {
+ schedule_work(&ipc_imem->run_state_worker);
+ }
+ }
+
+ /* Consider power management in the runtime phase. */
+ ipc_imem_slp_control_exec(ipc_imem);
+ break; /* Continue with skbuf processing. */
+
+ /* Unexpected phases. */
+ case IPC_P_OFF:
+ case IPC_P_OFF_REQ:
+ dev_err(ipc_imem->dev, "confused phase %s",
+ ipc_imem_phase_get_string(phase));
+ return;
+
+ case IPC_P_PSI:
+ if (old_phase != IPC_P_ROM)
+ break;
+
+ fallthrough;
+ /* On CP the PSI phase is already active. */
+
+ case IPC_P_ROM:
+ /* Before CP ROM driver starts the PSI image, it sets
+ * the exit_code field on the doorbell scratchpad and
+ * triggers the irq.
+ */
+ ipc_imem_rom_irq_exec(ipc_imem);
+ return;
+
+ default:
+ break;
+ }
+
+ /* process message ring */
+ ipc_protocol_msg_process(ipc_imem, irq);
+
+ /* process all open pipes */
+ for (i = 0; i < IPC_MEM_MAX_CHANNELS; i++) {
+ struct ipc_pipe *ul_pipe = &ipc_imem->channels[i].ul_pipe;
+ struct ipc_pipe *dl_pipe = &ipc_imem->channels[i].dl_pipe;
+
+ if (dl_pipe->is_open &&
+ (irq == IMEM_IRQ_DONT_CARE || irq == dl_pipe->irq)) {
+ ipc_imem_dl_pipe_process(ipc_imem, dl_pipe);
+
+ if (dl_pipe->nr_of_queued_entries == 0)
+ retry_allocation = true;
+ }
+
+ if (ul_pipe->is_open)
+ ipc_imem_ul_pipe_process(ipc_imem, ul_pipe);
+ }
+
+ /* Try to generate new ADB or ADGH. */
+ if (ipc_mux_ul_data_encode(ipc_imem->mux))
+ ipc_imem_td_update_timer_start(ipc_imem);
+
+ /* Continue the send procedure with accumulated SIO or NETIF packets.
+ * Reset the debounce flags.
+ */
+ ul_pending |= ipc_imem_ul_write_td(ipc_imem);
+
+ /* if UL data is pending restart TD update timer */
+ if (ul_pending) {
+ ipc_imem->hrtimer_period =
+ ktime_set(0, TD_UPDATE_DEFAULT_TIMEOUT_USEC * 1000ULL);
+ if (!hrtimer_active(&ipc_imem->tdupdate_timer))
+ hrtimer_start(&ipc_imem->tdupdate_timer,
+ ipc_imem->hrtimer_period,
+ HRTIMER_MODE_REL);
+ }
+
+ /* If CP has executed the transition
+ * from IPC_INIT to IPC_RUNNING in the PSI
+ * phase, wake up the flash app to open the pipes.
+ */
+ if ((phase == IPC_P_PSI || phase == IPC_P_EBL) &&
+ ipc_imem->ipc_requested_state == IPC_MEM_DEVICE_IPC_RUNNING &&
+ ipc_mmio_get_ipc_state(ipc_imem->mmio) ==
+ IPC_MEM_DEVICE_IPC_RUNNING &&
+ ipc_imem->flash_channel_id >= 0) {
+ /* Wake up the flash app to open the pipes. */
+ ch_id = ipc_imem->flash_channel_id;
+ complete(&ipc_imem->channels[ch_id].ul_sem);
+ }
+
+ /* Reset the expected CP state. */
+ ipc_imem->ipc_requested_state = IPC_MEM_DEVICE_IPC_DONT_CARE;
+
+ if (retry_allocation) {
+ ipc_imem->hrtimer_period =
+ ktime_set(0, IPC_TD_ALLOC_TIMER_PERIOD_MS * 1000 * 1000ULL);
+ if (!hrtimer_active(&ipc_imem->td_alloc_timer))
+ hrtimer_start(&ipc_imem->td_alloc_timer,
+ ipc_imem->hrtimer_period,
+ HRTIMER_MODE_REL);
+ }
+}
+
+/* Callback by tasklet for handling interrupt events. */
+static int ipc_imem_tq_irq_cb(struct iosm_imem *ipc_imem, int arg, void *msg,
+ size_t size)
+{
+ ipc_imem_handle_irq(ipc_imem, arg);
+
+ return 0;
+}
+
+void ipc_imem_ul_send(struct iosm_imem *ipc_imem)
+{
+ /* start doorbell irq delay timer if UL is pending */
+ if (ipc_imem_ul_write_td(ipc_imem))
+ ipc_imem_td_update_timer_start(ipc_imem);
+}
+
+/* Check the execution stage and update the AP phase */
+static enum ipc_phase ipc_imem_phase_update_check(struct iosm_imem *ipc_imem,
+ enum ipc_mem_exec_stage stage)
+{
+ switch (stage) {
+ case IPC_MEM_EXEC_STAGE_BOOT:
+ if (ipc_imem->phase != IPC_P_ROM) {
+ /* Send this event only once */
+ ipc_uevent_send(ipc_imem->dev, UEVENT_ROM_READY);
+ }
+
+ ipc_imem->phase = IPC_P_ROM;
+ break;
+
+ case IPC_MEM_EXEC_STAGE_PSI:
+ ipc_imem->phase = IPC_P_PSI;
+ break;
+
+ case IPC_MEM_EXEC_STAGE_EBL:
+ ipc_imem->phase = IPC_P_EBL;
+ break;
+
+ case IPC_MEM_EXEC_STAGE_RUN:
+ if (ipc_imem->phase != IPC_P_RUN &&
+ ipc_imem->ipc_status == IPC_MEM_DEVICE_IPC_RUNNING) {
+ ipc_uevent_send(ipc_imem->dev, UEVENT_MDM_READY);
+ }
+ ipc_imem->phase = IPC_P_RUN;
+ break;
+
+ case IPC_MEM_EXEC_STAGE_CRASH:
+ if (ipc_imem->phase != IPC_P_CRASH)
+ ipc_uevent_send(ipc_imem->dev, UEVENT_CRASH);
+
+ ipc_imem->phase = IPC_P_CRASH;
+ break;
+
+ case IPC_MEM_EXEC_STAGE_CD_READY:
+ if (ipc_imem->phase != IPC_P_CD_READY)
+ ipc_uevent_send(ipc_imem->dev, UEVENT_CD_READY);
+ ipc_imem->phase = IPC_P_CD_READY;
+ break;
+
+ default:
+ /* unknown exec stage:
+ * assume that link is down and send info to listeners
+ */
+ ipc_uevent_send(ipc_imem->dev, UEVENT_CD_READY_LINK_DOWN);
+ break;
+ }
+
+ return ipc_imem->phase;
+}
+
+/* Send msg to device to open pipe */
+static bool ipc_imem_pipe_open(struct iosm_imem *ipc_imem,
+ struct ipc_pipe *pipe)
+{
+ union ipc_msg_prep_args prep_args = {
+ .pipe_open.pipe = pipe,
+ };
+
+ if (ipc_protocol_msg_send(ipc_imem->ipc_protocol,
+ IPC_MSG_PREP_PIPE_OPEN, &prep_args) == 0)
+ pipe->is_open = true;
+
+ return pipe->is_open;
+}
+
+/* Allocates the TDs for the given pipe along with firing HP update DB. */
+static int ipc_imem_tq_pipe_td_alloc(struct iosm_imem *ipc_imem, int arg,
+ void *msg, size_t size)
+{
+ struct ipc_pipe *dl_pipe = msg;
+ bool processed = false;
+ int i;
+
+ for (i = 0; i < dl_pipe->nr_of_entries - 1; i++)
+ processed |= ipc_imem_dl_skb_alloc(ipc_imem, dl_pipe);
+
+ /* Trigger the doorbell irq to inform CP that new downlink buffers are
+ * available.
+ */
+ if (processed)
+ ipc_protocol_doorbell_trigger(ipc_imem->ipc_protocol, arg);
+
+ return 0;
+}
+
+static enum hrtimer_restart
+ipc_imem_td_update_timer_cb(struct hrtimer *hr_timer)
+{
+ struct iosm_imem *ipc_imem =
+ container_of(hr_timer, struct iosm_imem, tdupdate_timer);
+
+ ipc_task_queue_send_task(ipc_imem, ipc_imem_tq_td_update_timer_cb, 0,
+ NULL, 0, false);
+ return HRTIMER_NORESTART;
+}
+
+/* Get the CP execution state and map it to the AP phase. */
+enum ipc_phase ipc_imem_phase_update(struct iosm_imem *ipc_imem)
+{
+ enum ipc_mem_exec_stage exec_stage =
+ ipc_imem_get_exec_stage_buffered(ipc_imem);
+ /* If the CP stage is undef, return the internal precalculated phase. */
+ return ipc_imem->phase == IPC_P_OFF_REQ ?
+ ipc_imem->phase :
+ ipc_imem_phase_update_check(ipc_imem, exec_stage);
+}
+
+const char *ipc_imem_phase_get_string(enum ipc_phase phase)
+{
+ switch (phase) {
+ case IPC_P_RUN:
+ return "A-RUN";
+
+ case IPC_P_OFF:
+ return "A-OFF";
+
+ case IPC_P_ROM:
+ return "A-ROM";
+
+ case IPC_P_PSI:
+ return "A-PSI";
+
+ case IPC_P_EBL:
+ return "A-EBL";
+
+ case IPC_P_CRASH:
+ return "A-CRASH";
+
+ case IPC_P_CD_READY:
+ return "A-CD_READY";
+
+ case IPC_P_OFF_REQ:
+ return "A-OFF_REQ";
+
+ default:
+ return "A-???";
+ }
+}
+
+void ipc_imem_pipe_close(struct iosm_imem *ipc_imem, struct ipc_pipe *pipe)
+{
+ union ipc_msg_prep_args prep_args = { .pipe_close.pipe = pipe };
+
+ pipe->is_open = false;
+ ipc_protocol_msg_send(ipc_imem->ipc_protocol, IPC_MSG_PREP_PIPE_CLOSE,
+ &prep_args);
+
+ ipc_imem_pipe_cleanup(ipc_imem, pipe);
+}
+
+void ipc_imem_channel_close(struct iosm_imem *ipc_imem, int channel_id)
+{
+ struct ipc_mem_channel *channel;
+
+ if (channel_id < 0 || channel_id >= ipc_imem->nr_of_channels) {
+ dev_err(ipc_imem->dev, "invalid channel id %d", channel_id);
+ return;
+ }
+
+ channel = &ipc_imem->channels[channel_id];
+
+ if (channel->state == IMEM_CHANNEL_FREE) {
+ dev_err(ipc_imem->dev, "ch[%d]: invalid channel state %d",
+ channel_id, channel->state);
+ return;
+ }
+
+ /* Free only the channel id in the CP power off mode. */
+ if (channel->state == IMEM_CHANNEL_RESERVED)
+ /* Release only the channel id. */
+ goto channel_free;
+
+ if (ipc_imem->phase == IPC_P_RUN) {
+ ipc_imem_pipe_close(ipc_imem, &channel->ul_pipe);
+ ipc_imem_pipe_close(ipc_imem, &channel->dl_pipe);
+ }
+
+ ipc_imem_pipe_cleanup(ipc_imem, &channel->ul_pipe);
+ ipc_imem_pipe_cleanup(ipc_imem, &channel->dl_pipe);
+
+channel_free:
+ ipc_imem_channel_free(channel);
+}
+
+struct ipc_mem_channel *ipc_imem_channel_open(struct iosm_imem *ipc_imem,
+ int channel_id, u32 db_id)
+{
+ struct ipc_mem_channel *channel;
+
+ if (channel_id < 0 || channel_id >= IPC_MEM_MAX_CHANNELS) {
+ dev_err(ipc_imem->dev, "invalid channel ID: %d", channel_id);
+ return NULL;
+ }
+
+ channel = &ipc_imem->channels[channel_id];
+
+ channel->state = IMEM_CHANNEL_ACTIVE;
+
+ if (!ipc_imem_pipe_open(ipc_imem, &channel->ul_pipe))
+ goto ul_pipe_err;
+
+ if (!ipc_imem_pipe_open(ipc_imem, &channel->dl_pipe))
+ goto dl_pipe_err;
+
+ /* Allocate the downlink buffers in tasklet context. */
+ if (ipc_task_queue_send_task(ipc_imem, ipc_imem_tq_pipe_td_alloc, db_id,
+ &channel->dl_pipe, 0, false)) {
+ dev_err(ipc_imem->dev, "td allocation failed : %d", channel_id);
+ goto task_failed;
+ }
+
+ /* Active channel. */
+ return channel;
+task_failed:
+ ipc_imem_pipe_close(ipc_imem, &channel->dl_pipe);
+dl_pipe_err:
+ ipc_imem_pipe_close(ipc_imem, &channel->ul_pipe);
+ul_pipe_err:
+ ipc_imem_channel_free(channel);
+ return NULL;
+}
+
+void ipc_imem_pm_suspend(struct iosm_imem *ipc_imem)
+{
+ ipc_protocol_suspend(ipc_imem->ipc_protocol);
+}
+
+void ipc_imem_pm_s2idle_sleep(struct iosm_imem *ipc_imem, bool sleep)
+{
+ ipc_protocol_s2idle_sleep(ipc_imem->ipc_protocol, sleep);
+}
+
+void ipc_imem_pm_resume(struct iosm_imem *ipc_imem)
+{
+ enum ipc_mem_exec_stage stage;
+
+ if (ipc_protocol_resume(ipc_imem->ipc_protocol)) {
+ stage = ipc_mmio_get_exec_stage(ipc_imem->mmio);
+ ipc_imem_phase_update_check(ipc_imem, stage);
+ }
+}
+
+void ipc_imem_channel_free(struct ipc_mem_channel *channel)
+{
+ /* Reset dynamic channel elements. */
+ channel->state = IMEM_CHANNEL_FREE;
+}
+
+int ipc_imem_channel_alloc(struct iosm_imem *ipc_imem, int index,
+ enum ipc_ctype ctype)
+{
+ struct ipc_mem_channel *channel;
+ int i;
+
+ /* Find channel of given type/index */
+ for (i = 0; i < ipc_imem->nr_of_channels; i++) {
+ channel = &ipc_imem->channels[i];
+ if (channel->ctype == ctype && channel->index == index)
+ break;
+ }
+
+ if (i >= ipc_imem->nr_of_channels) {
+ dev_dbg(ipc_imem->dev,
+ "no channel definition for index=%d ctype=%d", index,
+ ctype);
+ return -ECHRNG;
+ }
+
+ if (ipc_imem->channels[i].state != IMEM_CHANNEL_FREE) {
+ dev_dbg(ipc_imem->dev, "channel is in use");
+ return -EBUSY;
+ }
+
+ if (channel->ctype == IPC_CTYPE_WWAN &&
+ index == IPC_MEM_MUX_IP_CH_IF_ID)
+ channel->if_id = index;
+
+ channel->channel_id = index;
+ channel->state = IMEM_CHANNEL_RESERVED;
+
+ return i;
+}
+
+void ipc_imem_channel_init(struct iosm_imem *ipc_imem, enum ipc_ctype ctype,
+ struct ipc_chnl_cfg chnl_cfg, u32 irq_moderation)
+{
+ struct ipc_mem_channel *channel;
+
+ if (chnl_cfg.ul_pipe >= IPC_MEM_MAX_PIPES ||
+ chnl_cfg.dl_pipe >= IPC_MEM_MAX_PIPES) {
+ dev_err(ipc_imem->dev, "invalid pipe: ul_pipe=%d, dl_pipe=%d",
+ chnl_cfg.ul_pipe, chnl_cfg.dl_pipe);
+ return;
+ }
+
+ if (ipc_imem->nr_of_channels >= IPC_MEM_MAX_CHANNELS) {
+ dev_err(ipc_imem->dev, "too many channels");
+ return;
+ }
+
+ channel = &ipc_imem->channels[ipc_imem->nr_of_channels];
+ channel->channel_id = ipc_imem->nr_of_channels;
+ channel->ctype = ctype;
+ channel->index = chnl_cfg.id;
+ channel->net_err_count = 0;
+ channel->state = IMEM_CHANNEL_FREE;
+ ipc_imem->nr_of_channels++;
+
+ ipc_imem_channel_update(ipc_imem, channel->channel_id, chnl_cfg,
+ IRQ_MOD_OFF);
+
+ skb_queue_head_init(&channel->ul_list);
+
+ init_completion(&channel->ul_sem);
+}
+
+void ipc_imem_channel_update(struct iosm_imem *ipc_imem, int id,
+ struct ipc_chnl_cfg chnl_cfg, u32 irq_moderation)
+{
+ struct ipc_mem_channel *channel;
+
+ if (id < 0 || id >= ipc_imem->nr_of_channels) {
+ dev_err(ipc_imem->dev, "invalid channel id %d", id);
+ return;
+ }
+
+ channel = &ipc_imem->channels[id];
+
+ if (channel->state != IMEM_CHANNEL_FREE &&
+ channel->state != IMEM_CHANNEL_RESERVED) {
+ dev_err(ipc_imem->dev, "invalid channel state %d",
+ channel->state);
+ return;
+ }
+
+ channel->ul_pipe.nr_of_entries = chnl_cfg.ul_nr_of_entries;
+ channel->ul_pipe.pipe_nr = chnl_cfg.ul_pipe;
+ channel->ul_pipe.is_open = false;
+ channel->ul_pipe.irq = IPC_UL_PIPE_IRQ_VECTOR;
+ channel->ul_pipe.channel = channel;
+ channel->ul_pipe.dir = IPC_MEM_DIR_UL;
+ channel->ul_pipe.accumulation_backoff = chnl_cfg.accumulation_backoff;
+ channel->ul_pipe.irq_moderation = irq_moderation;
+ channel->ul_pipe.buf_size = 0;
+
+ channel->dl_pipe.nr_of_entries = chnl_cfg.dl_nr_of_entries;
+ channel->dl_pipe.pipe_nr = chnl_cfg.dl_pipe;
+ channel->dl_pipe.is_open = false;
+ channel->dl_pipe.irq = IPC_DL_PIPE_IRQ_VECTOR;
+ channel->dl_pipe.channel = channel;
+ channel->dl_pipe.dir = IPC_MEM_DIR_DL;
+ channel->dl_pipe.accumulation_backoff = chnl_cfg.accumulation_backoff;
+ channel->dl_pipe.irq_moderation = irq_moderation;
+ channel->dl_pipe.buf_size = chnl_cfg.dl_buf_size;
+}
+
+static void ipc_imem_channel_reset(struct iosm_imem *ipc_imem)
+{
+ int i;
+
+ for (i = 0; i < ipc_imem->nr_of_channels; i++) {
+ struct ipc_mem_channel *channel;
+
+ channel = &ipc_imem->channels[i];
+
+ ipc_imem_pipe_cleanup(ipc_imem, &channel->dl_pipe);
+ ipc_imem_pipe_cleanup(ipc_imem, &channel->ul_pipe);
+
+ ipc_imem_channel_free(channel);
+ }
+}
+
+void ipc_imem_pipe_cleanup(struct iosm_imem *ipc_imem, struct ipc_pipe *pipe)
+{
+ struct sk_buff *skb;
+
+ /* Force pipe to closed state also when not explicitly closed through
+ * ipc_imem_pipe_close()
+ */
+ pipe->is_open = false;
+
+ /* Empty the uplink skb accumulator. */
+ while ((skb = skb_dequeue(&pipe->channel->ul_list)))
+ ipc_pcie_kfree_skb(ipc_imem->pcie, skb);
+
+ ipc_protocol_pipe_cleanup(ipc_imem->ipc_protocol, pipe);
+}
+
+/* Send IPC protocol uninit to the modem when Link is active. */
+static void ipc_imem_device_ipc_uninit(struct iosm_imem *ipc_imem)
+{
+ int timeout = IPC_MODEM_UNINIT_TIMEOUT_MS;
+ enum ipc_mem_device_ipc_state ipc_state;
+
+ /* When PCIe link is up set IPC_UNINIT
+ * of the modem otherwise ignore it when PCIe link down happens.
+ */
+ if (ipc_pcie_check_data_link_active(ipc_imem->pcie)) {
+ /* set modem to UNINIT
+ * (in case we want to reload the AP driver without resetting
+ * the modem)
+ */
+ ipc_doorbell_fire(ipc_imem->pcie, IPC_DOORBELL_IRQ_IPC,
+ IPC_MEM_DEVICE_IPC_UNINIT);
+ ipc_state = ipc_mmio_get_ipc_state(ipc_imem->mmio);
+
+ /* Wait for maximum 30ms to allow the Modem to uninitialize the
+ * protocol.
+ */
+ while ((ipc_state <= IPC_MEM_DEVICE_IPC_DONT_CARE) &&
+ (ipc_state != IPC_MEM_DEVICE_IPC_UNINIT) &&
+ (timeout > 0)) {
+ usleep_range(1000, 1250);
+ timeout--;
+ ipc_state = ipc_mmio_get_ipc_state(ipc_imem->mmio);
+ }
+ }
+}
+
+void ipc_imem_cleanup(struct iosm_imem *ipc_imem)
+{
+ ipc_imem->phase = IPC_P_OFF_REQ;
+
+ /* forward MDM_NOT_READY to listeners */
+ ipc_uevent_send(ipc_imem->dev, UEVENT_MDM_NOT_READY);
+
+ hrtimer_cancel(&ipc_imem->td_alloc_timer);
+ hrtimer_cancel(&ipc_imem->tdupdate_timer);
+ hrtimer_cancel(&ipc_imem->fast_update_timer);
+ hrtimer_cancel(&ipc_imem->startup_timer);
+
+ /* cancel the workqueue */
+ cancel_work_sync(&ipc_imem->run_state_worker);
+
+ if (test_and_clear_bit(FULLY_FUNCTIONAL, &ipc_imem->flag)) {
+ ipc_mux_deinit(ipc_imem->mux);
+ ipc_wwan_deinit(ipc_imem->wwan);
+ ipc_port_deinit(ipc_imem->ipc_port);
+ }
+
+ ipc_imem_device_ipc_uninit(ipc_imem);
+ ipc_imem_channel_reset(ipc_imem);
+
+ ipc_protocol_deinit(ipc_imem->ipc_protocol);
+ ipc_task_deinit(ipc_imem->ipc_task);
+
+ kfree(ipc_imem->ipc_task);
+ kfree(ipc_imem->mmio);
+
+ ipc_imem->phase = IPC_P_OFF;
+}
+
+/* After CP has unblocked the PCIe link, save the start address of the doorbell
+ * scratchpad and prepare the shared memory region. If the flashing to RAM
+ * procedure shall be executed, copy the chip information from the doorbell
+ * scratchtpad to the application buffer and wake up the flash app.
+ */
+static int ipc_imem_config(struct iosm_imem *ipc_imem)
+{
+ enum ipc_phase phase;
+
+ /* Initialize the semaphore for the blocking read UL/DL transfer. */
+ init_completion(&ipc_imem->ul_pend_sem);
+
+ init_completion(&ipc_imem->dl_pend_sem);
+
+ /* clear internal flags */
+ ipc_imem->ipc_status = IPC_MEM_DEVICE_IPC_UNINIT;
+ ipc_imem->enter_runtime = 0;
+
+ phase = ipc_imem_phase_update(ipc_imem);
+
+ /* Either CP shall be in the power off or power on phase. */
+ switch (phase) {
+ case IPC_P_ROM:
+ ipc_imem->hrtimer_period = ktime_set(0, 1000 * 1000 * 1000ULL);
+ /* poll execution stage (for delayed start, e.g. NAND) */
+ if (!hrtimer_active(&ipc_imem->startup_timer))
+ hrtimer_start(&ipc_imem->startup_timer,
+ ipc_imem->hrtimer_period,
+ HRTIMER_MODE_REL);
+ return 0;
+
+ case IPC_P_PSI:
+ case IPC_P_EBL:
+ case IPC_P_RUN:
+ /* The initial IPC state is IPC_MEM_DEVICE_IPC_UNINIT. */
+ ipc_imem->ipc_requested_state = IPC_MEM_DEVICE_IPC_UNINIT;
+
+ /* Verify the exepected initial state. */
+ if (ipc_imem->ipc_requested_state ==
+ ipc_mmio_get_ipc_state(ipc_imem->mmio)) {
+ ipc_imem_ipc_init_check(ipc_imem);
+
+ return 0;
+ }
+ dev_err(ipc_imem->dev,
+ "ipc_status(%d) != IPC_MEM_DEVICE_IPC_UNINIT",
+ ipc_mmio_get_ipc_state(ipc_imem->mmio));
+ break;
+ case IPC_P_CRASH:
+ case IPC_P_CD_READY:
+ dev_dbg(ipc_imem->dev,
+ "Modem is in phase %d, reset Modem to collect CD",
+ phase);
+ return 0;
+ default:
+ dev_err(ipc_imem->dev, "unexpected operation phase %d", phase);
+ break;
+ }
+
+ complete(&ipc_imem->dl_pend_sem);
+ complete(&ipc_imem->ul_pend_sem);
+ ipc_imem->phase = IPC_P_OFF;
+ return -EIO;
+}
+
+/* Pass the dev ptr to the shared memory driver and request the entry points */
+struct iosm_imem *ipc_imem_init(struct iosm_pcie *pcie, unsigned int device_id,
+ void __iomem *mmio, struct device *dev)
+{
+ struct iosm_imem *ipc_imem = kzalloc(sizeof(*pcie->imem), GFP_KERNEL);
+
+ if (!ipc_imem)
+ return NULL;
+
+ /* Save the device address. */
+ ipc_imem->pcie = pcie;
+ ipc_imem->dev = dev;
+
+ ipc_imem->pci_device_id = device_id;
+
+ ipc_imem->ev_cdev_write_pending = false;
+ ipc_imem->cp_version = 0;
+ ipc_imem->device_sleep = IPC_HOST_SLEEP_ENTER_SLEEP;
+
+ /* Reset the flash channel id. */
+ ipc_imem->flash_channel_id = -1;
+
+ /* Reset the max number of configured channels */
+ ipc_imem->nr_of_channels = 0;
+
+ /* allocate IPC MMIO */
+ ipc_imem->mmio = ipc_mmio_init(mmio, ipc_imem->dev);
+ if (!ipc_imem->mmio) {
+ dev_err(ipc_imem->dev, "failed to initialize mmio region");
+ goto mmio_init_fail;
+ }
+
+ ipc_imem->ipc_task = kzalloc(sizeof(*ipc_imem->ipc_task),
+ GFP_KERNEL);
+
+ /* Create tasklet for event handling*/
+ if (!ipc_imem->ipc_task)
+ goto ipc_task_fail;
+
+ if (ipc_task_init(ipc_imem->ipc_task))
+ goto ipc_task_init_fail;
+
+ ipc_imem->ipc_task->dev = ipc_imem->dev;
+
+ INIT_WORK(&ipc_imem->run_state_worker, ipc_imem_run_state_worker);
+
+ ipc_imem->ipc_protocol = ipc_protocol_init(ipc_imem);
+
+ if (!ipc_imem->ipc_protocol)
+ goto protocol_init_fail;
+
+ /* The phase is set to power off. */
+ ipc_imem->phase = IPC_P_OFF;
+
+ hrtimer_init(&ipc_imem->startup_timer, CLOCK_MONOTONIC,
+ HRTIMER_MODE_REL);
+ ipc_imem->startup_timer.function = ipc_imem_startup_timer_cb;
+
+ hrtimer_init(&ipc_imem->tdupdate_timer, CLOCK_MONOTONIC,
+ HRTIMER_MODE_REL);
+ ipc_imem->tdupdate_timer.function = ipc_imem_td_update_timer_cb;
+
+ hrtimer_init(&ipc_imem->fast_update_timer, CLOCK_MONOTONIC,
+ HRTIMER_MODE_REL);
+ ipc_imem->fast_update_timer.function = ipc_imem_fast_update_timer_cb;
+
+ hrtimer_init(&ipc_imem->td_alloc_timer, CLOCK_MONOTONIC,
+ HRTIMER_MODE_REL);
+ ipc_imem->td_alloc_timer.function = ipc_imem_td_alloc_timer_cb;
+
+ if (ipc_imem_config(ipc_imem)) {
+ dev_err(ipc_imem->dev, "failed to initialize the imem");
+ goto imem_config_fail;
+ }
+
+ return ipc_imem;
+
+imem_config_fail:
+ hrtimer_cancel(&ipc_imem->td_alloc_timer);
+ hrtimer_cancel(&ipc_imem->fast_update_timer);
+ hrtimer_cancel(&ipc_imem->tdupdate_timer);
+ hrtimer_cancel(&ipc_imem->startup_timer);
+protocol_init_fail:
+ cancel_work_sync(&ipc_imem->run_state_worker);
+ ipc_task_deinit(ipc_imem->ipc_task);
+ipc_task_init_fail:
+ kfree(ipc_imem->ipc_task);
+ipc_task_fail:
+ kfree(ipc_imem->mmio);
+mmio_init_fail:
+ kfree(ipc_imem);
+ return NULL;
+}
+
+void ipc_imem_irq_process(struct iosm_imem *ipc_imem, int irq)
+{
+ /* Debounce IPC_EV_IRQ. */
+ if (ipc_imem && !ipc_imem->ev_irq_pending[irq]) {
+ ipc_imem->ev_irq_pending[irq] = true;
+ ipc_task_queue_send_task(ipc_imem, ipc_imem_tq_irq_cb, irq,
+ NULL, 0, false);
+ }
+}
+
+void ipc_imem_td_update_timer_suspend(struct iosm_imem *ipc_imem, bool suspend)
+{
+ ipc_imem->td_update_timer_suspended = suspend;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#ifndef IOSM_IPC_IMEM_H
+#define IOSM_IPC_IMEM_H
+
+#include <linux/skbuff.h>
+#include <stdbool.h>
+
+#include "iosm_ipc_mmio.h"
+#include "iosm_ipc_pcie.h"
+#include "iosm_ipc_uevent.h"
+#include "iosm_ipc_wwan.h"
+#include "iosm_ipc_task_queue.h"
+
+struct ipc_chnl_cfg;
+
+/* IRQ moderation in usec */
+#define IRQ_MOD_OFF 0
+#define IRQ_MOD_NET 1000
+#define IRQ_MOD_TRC 4000
+
+/* Either the PSI image is accepted by CP or the suspended flash tool is waken,
+ * informed that the CP ROM driver is not ready to process the PSI image.
+ * unit : milliseconds
+ */
+#define IPC_PSI_TRANSFER_TIMEOUT 3000
+
+/* Timeout in 20 msec to wait for the modem to boot up to
+ * IPC_MEM_DEVICE_IPC_INIT state.
+ * unit : milliseconds (500 * ipc_util_msleep(20))
+ */
+#define IPC_MODEM_BOOT_TIMEOUT 500
+
+/* Wait timeout for ipc status reflects IPC_MEM_DEVICE_IPC_UNINIT
+ * unit : milliseconds
+ */
+#define IPC_MODEM_UNINIT_TIMEOUT_MS 30
+
+/* Pending time for processing data.
+ * unit : milliseconds
+ */
+#define IPC_PEND_DATA_TIMEOUT 500
+
+/* The timeout in milliseconds for application to wait for remote time. */
+#define IPC_REMOTE_TS_TIMEOUT_MS 10
+
+/* Timeout for TD allocation retry.
+ * unit : milliseconds
+ */
+#define IPC_TD_ALLOC_TIMER_PERIOD_MS 100
+
+/* Host sleep target is host */
+#define IPC_HOST_SLEEP_HOST 0
+
+/* Host sleep target is device */
+#define IPC_HOST_SLEEP_DEVICE 1
+
+/* Sleep message, target host: AP enters sleep / target device: CP is
+ * allowed to enter sleep and shall use the host sleep protocol
+ */
+#define IPC_HOST_SLEEP_ENTER_SLEEP 0
+
+/* Sleep_message, target host: AP exits sleep / target device: CP is
+ * NOT allowed to enter sleep
+ */
+#define IPC_HOST_SLEEP_EXIT_SLEEP 1
+
+#define IMEM_IRQ_DONT_CARE (-1)
+
+#define IPC_MEM_MAX_CHANNELS 7
+
+#define IPC_MEM_MUX_IP_SESSION_ENTRIES 8
+
+#define IPC_MEM_MUX_IP_CH_IF_ID 0
+
+#define TD_UPDATE_DEFAULT_TIMEOUT_USEC 1900
+
+#define FORCE_UPDATE_DEFAULT_TIMEOUT_USEC 500
+
+/* Sleep_message, target host: not applicable / target device: CP is
+ * allowed to enter sleep and shall NOT use the device sleep protocol
+ */
+#define IPC_HOST_SLEEP_ENTER_SLEEP_NO_PROTOCOL 2
+
+/* in_band_crash_signal IPC_MEM_INBAND_CRASH_SIG
+ * Modem crash notification configuration. If this value is non-zero then
+ * FEATURE_SET message will be sent to the Modem as a result the Modem will
+ * signal Crash via Execution Stage register. If this value is zero then Modem
+ * will use out-of-band method to notify about it's Crash.
+ */
+#define IPC_MEM_INBAND_CRASH_SIG 1
+
+/* Extra headroom to be allocated for DL SKBs to allow addition of Ethernet
+ * header
+ */
+#define IPC_MEM_DL_ETH_OFFSET 16
+
+#define IPC_CB(skb) ((struct ipc_skb_cb *)((skb)->cb))
+
+#define FULLY_FUNCTIONAL 0
+
+/* List of the supported UL/DL pipes. */
+enum ipc_mem_pipes {
+ IPC_MEM_PIPE_0 = 0,
+ IPC_MEM_PIPE_1,
+ IPC_MEM_PIPE_2,
+ IPC_MEM_PIPE_3,
+ IPC_MEM_PIPE_4,
+ IPC_MEM_PIPE_5,
+ IPC_MEM_PIPE_6,
+ IPC_MEM_PIPE_7,
+ IPC_MEM_PIPE_8,
+ IPC_MEM_PIPE_9,
+ IPC_MEM_PIPE_10,
+ IPC_MEM_PIPE_11,
+ IPC_MEM_PIPE_12,
+ IPC_MEM_PIPE_13,
+ IPC_MEM_PIPE_14,
+ IPC_MEM_PIPE_15,
+ IPC_MEM_PIPE_16,
+ IPC_MEM_PIPE_17,
+ IPC_MEM_PIPE_18,
+ IPC_MEM_PIPE_19,
+ IPC_MEM_PIPE_20,
+ IPC_MEM_PIPE_21,
+ IPC_MEM_PIPE_22,
+ IPC_MEM_PIPE_23,
+ IPC_MEM_MAX_PIPES
+};
+
+/* Enum defining channel states. */
+enum ipc_channel_state {
+ IMEM_CHANNEL_FREE,
+ IMEM_CHANNEL_RESERVED,
+ IMEM_CHANNEL_ACTIVE,
+ IMEM_CHANNEL_CLOSING,
+};
+
+/* Time Unit */
+enum ipc_time_unit {
+ IPC_SEC = 0,
+ IPC_MILLI_SEC = 1,
+ IPC_MICRO_SEC = 2,
+ IPC_NANO_SEC = 3,
+ IPC_PICO_SEC = 4,
+ IPC_FEMTO_SEC = 5,
+ IPC_ATTO_SEC = 6,
+};
+
+/**
+ * enum ipc_ctype - Enum defining supported channel type needed for control
+ * /IP traffic.
+ * @IPC_CTYPE_WWAN: Used for IP traffic
+ * @IPC_CTYPE_CTRL: Used for Control Communication
+ */
+enum ipc_ctype {
+ IPC_CTYPE_WWAN,
+ IPC_CTYPE_CTRL,
+};
+
+/* Pipe direction. */
+enum ipc_mem_pipe_dir {
+ IPC_MEM_DIR_UL,
+ IPC_MEM_DIR_DL,
+};
+
+/* HP update identifier. To be used as data for ipc_cp_irq_hpda_update() */
+enum ipc_hp_identifier {
+ IPC_HP_MR = 0,
+ IPC_HP_PM_TRIGGER,
+ IPC_HP_WAKEUP_SPEC_TMR,
+ IPC_HP_TD_UPD_TMR_START,
+ IPC_HP_TD_UPD_TMR,
+ IPC_HP_FAST_TD_UPD_TMR,
+ IPC_HP_UL_WRITE_TD,
+ IPC_HP_DL_PROCESS,
+ IPC_HP_NET_CHANNEL_INIT,
+ IPC_HP_CDEV_OPEN,
+};
+
+/**
+ * struct ipc_pipe - Structure for Pipe.
+ * @tdr_start: Ipc private protocol Transfer Descriptor Ring
+ * @channel: Id of the sio device, set by imem_sio_open,
+ * needed to pass DL char to the user terminal
+ * @skbr_start: Circular buffer for skbuf and the buffer
+ * reference in a tdr_start entry.
+ * @phy_tdr_start: Transfer descriptor start address
+ * @old_head: last head pointer reported to CP.
+ * @old_tail: AP read position before CP moves the read
+ * position to write/head. If CP has consumed the
+ * buffers, AP has to freed the skbuf starting at
+ * tdr_start[old_tail].
+ * @nr_of_entries: Number of elements of skb_start and tdr_start.
+ * @max_nr_of_queued_entries: Maximum number of queued entries in TDR
+ * @accumulation_backoff: Accumulation in usec for accumulation
+ * backoff (0 = no acc backoff)
+ * @irq_moderation: timer in usec for irq_moderation
+ * (0=no irq moderation)
+ * @pipe_nr: Pipe identification number
+ * @irq: Interrupt vector
+ * @dir: Direction of data stream in pipe
+ * @td_tag: Unique tag of the buffer queued
+ * @buf_size: Buffer size (in bytes) for preallocated
+ * buffers (for DL pipes)
+ * @nr_of_queued_entries: Aueued number of entries
+ * @is_open: Check for open pipe status
+ */
+struct ipc_pipe {
+ struct ipc_protocol_td *tdr_start;
+ struct ipc_mem_channel *channel;
+ struct sk_buff **skbr_start;
+ dma_addr_t phy_tdr_start;
+ u32 old_head;
+ u32 old_tail;
+ u32 nr_of_entries;
+ u32 max_nr_of_queued_entries;
+ u32 accumulation_backoff;
+ u32 irq_moderation;
+ u32 pipe_nr;
+ u32 irq;
+ enum ipc_mem_pipe_dir dir;
+ u32 td_tag;
+ u32 buf_size;
+ u16 nr_of_queued_entries;
+ u8 is_open:1;
+};
+
+/**
+ * struct ipc_mem_channel - Structure for Channel.
+ * @channel_id: Instance of the channel list and is return to the user
+ * at the end of the open operation.
+ * @ctype: Control or netif channel.
+ * @index: unique index per ctype
+ * @ul_pipe: pipe objects
+ * @dl_pipe: pipe objects
+ * @if_id: Interface ID
+ * @net_err_count: Number of downlink errors returned by ipc_wwan_receive
+ * interface at the entry point of the IP stack.
+ * @state: Free, reserved or busy (in use).
+ * @ul_sem: Needed for the blocking write or uplink transfer.
+ * @ul_list: Uplink accumulator which is filled by the uplink
+ * char app or IP stack. The socket buffer pointer are
+ * added to the descriptor list in the kthread context.
+ */
+struct ipc_mem_channel {
+ int channel_id;
+ enum ipc_ctype ctype;
+ int index;
+ struct ipc_pipe ul_pipe;
+ struct ipc_pipe dl_pipe;
+ int if_id;
+ u32 net_err_count;
+ enum ipc_channel_state state;
+ struct completion ul_sem;
+ struct sk_buff_head ul_list;
+};
+
+/**
+ * enum ipc_phase - Different AP and CP phases.
+ * The enums defined after "IPC_P_ROM" and before
+ * "IPC_P_RUN" indicates the operating state where CP can
+ * respond to any requests. So while introducing new phase
+ * this shall be taken into consideration.
+ * @IPC_P_OFF: On host PC, the PCIe device link settings are known
+ * about the combined power on. PC is running, the driver
+ * is loaded and CP is in power off mode. The PCIe bus
+ * driver call the device power mode D3hot. In this phase
+ * the driver the polls the device, until the device is in
+ * the power on state and signals the power mode D0.
+ * @IPC_P_OFF_REQ: The intermediate phase between cleanup activity starts
+ * and ends.
+ * @IPC_P_CRASH: The phase indicating CP crash
+ * @IPC_P_CD_READY: The phase indicating CP core dump is ready
+ * @IPC_P_ROM: After power on, CP starts in ROM mode and the IPC ROM
+ * driver is waiting 150 ms for the AP active notification
+ * saved in the PCI link status register.
+ * @IPC_P_PSI: Primary signed image download phase
+ * @IPC_P_EBL: Extended bootloader pahse
+ * @IPC_P_RUN: The phase after flashing to RAM is the RUNTIME phase.
+ */
+enum ipc_phase {
+ IPC_P_OFF,
+ IPC_P_OFF_REQ,
+ IPC_P_CRASH,
+ IPC_P_CD_READY,
+ IPC_P_ROM,
+ IPC_P_PSI,
+ IPC_P_EBL,
+ IPC_P_RUN,
+};
+
+/**
+ * struct iosm_imem - Current state of the IPC shared memory.
+ * @mmio: mmio instance to access CP MMIO area /
+ * doorbell scratchpad.
+ * @ipc_protocol: IPC Protocol instance
+ * @ipc_task: Task for entry into ipc task queue
+ * @wwan: WWAN device pointer
+ * @mux: IP Data multiplexing state.
+ * @sio: IPC SIO data structure pointer
+ * @ipc_port: IPC PORT data structure pointer
+ * @pcie: IPC PCIe
+ * @dev: Pointer to device structure
+ * @flash_channel_id: Reserved channel id for flashing to RAM.
+ * @ipc_requested_state: Expected IPC state on CP.
+ * @channels: Channel list with UL/DL pipe pairs.
+ * @ipc_status: local ipc_status
+ * @nr_of_channels: number of configured channels
+ * @startup_timer: startup timer for NAND support.
+ * @hrtimer_period: Hr timer period
+ * @tdupdate_timer: Delay the TD update doorbell.
+ * @fast_update_timer: forced head pointer update delay timer.
+ * @td_alloc_timer: Timer for DL pipe TD allocation retry
+ * @rom_exit_code: Mapped boot rom exit code.
+ * @enter_runtime: 1 means the transition to runtime phase was
+ * executed.
+ * @ul_pend_sem: Semaphore to wait/complete of UL TDs
+ * before closing pipe.
+ * @app_notify_ul_pend: Signal app if UL TD is pending
+ * @dl_pend_sem: Semaphore to wait/complete of DL TDs
+ * before closing pipe.
+ * @app_notify_dl_pend: Signal app if DL TD is pending
+ * @phase: Operating phase like runtime.
+ * @pci_device_id: Device ID
+ * @cp_version: CP version
+ * @device_sleep: Device sleep state
+ * @run_state_worker: Pointer to worker component for device
+ * setup operations to be called when modem
+ * reaches RUN state
+ * @ev_irq_pending: 0 means inform the IPC tasklet to
+ * process the irq actions.
+ * @flag: Flag to monitor the state of driver
+ * @td_update_timer_suspended: if true then td update timer suspend
+ * @ev_cdev_write_pending: 0 means inform the IPC tasklet to pass
+ * the accumulated uplink buffers to CP.
+ * @ev_mux_net_transmit_pending:0 means inform the IPC tasklet to pass
+ * @reset_det_n: Reset detect flag
+ * @pcie_wake_n: Pcie wake flag
+ */
+struct iosm_imem {
+ struct iosm_mmio *mmio;
+ struct iosm_protocol *ipc_protocol;
+ struct ipc_task *ipc_task;
+ struct iosm_wwan *wwan;
+ struct iosm_mux *mux;
+ struct iosm_cdev *ipc_port[IPC_MEM_MAX_CHANNELS];
+ struct iosm_pcie *pcie;
+ struct device *dev;
+ int flash_channel_id;
+ enum ipc_mem_device_ipc_state ipc_requested_state;
+ struct ipc_mem_channel channels[IPC_MEM_MAX_CHANNELS];
+ u32 ipc_status;
+ u32 nr_of_channels;
+ struct hrtimer startup_timer;
+ ktime_t hrtimer_period;
+ struct hrtimer tdupdate_timer;
+ struct hrtimer fast_update_timer;
+ struct hrtimer td_alloc_timer;
+ enum rom_exit_code rom_exit_code;
+ u32 enter_runtime;
+ struct completion ul_pend_sem;
+ u32 app_notify_ul_pend;
+ struct completion dl_pend_sem;
+ u32 app_notify_dl_pend;
+ enum ipc_phase phase;
+ u16 pci_device_id;
+ int cp_version;
+ int device_sleep;
+ struct work_struct run_state_worker;
+ u8 ev_irq_pending[IPC_IRQ_VECTORS];
+ unsigned long flag;
+ u8 td_update_timer_suspended:1,
+ ev_cdev_write_pending:1,
+ ev_mux_net_transmit_pending:1,
+ reset_det_n:1,
+ pcie_wake_n:1;
+};
+
+/**
+ * ipc_imem_init - Initialize the shared memory region
+ * @pcie: Pointer to core driver data-struct
+ * @device_id: PCI device ID
+ * @mmio: Pointer to the mmio area
+ * @dev: Pointer to device structure
+ *
+ * Returns: Initialized imem pointer on success else NULL
+ */
+struct iosm_imem *ipc_imem_init(struct iosm_pcie *pcie, unsigned int device_id,
+ void __iomem *mmio, struct device *dev);
+
+/**
+ * ipc_imem_pm_s2idle_sleep - Set PM variables to sleep/active for
+ * s2idle sleep/active
+ * @ipc_imem: Pointer to imem data-struct
+ * @sleep: Set PM Variable to sleep/active
+ */
+void ipc_imem_pm_s2idle_sleep(struct iosm_imem *ipc_imem, bool sleep);
+
+/**
+ * ipc_imem_pm_suspend - The HAL shall ask the shared memory layer
+ * whether D3 is allowed.
+ * @ipc_imem: Pointer to imem data-struct
+ */
+void ipc_imem_pm_suspend(struct iosm_imem *ipc_imem);
+
+/**
+ * ipc_imem_pm_resume - The HAL shall inform the shared memory layer
+ * that the device is active.
+ * @ipc_imem: Pointer to imem data-struct
+ */
+void ipc_imem_pm_resume(struct iosm_imem *ipc_imem);
+
+/**
+ * ipc_imem_cleanup - Inform CP and free the shared memory resources.
+ * @ipc_imem: Pointer to imem data-struct
+ */
+void ipc_imem_cleanup(struct iosm_imem *ipc_imem);
+
+/**
+ * ipc_imem_irq_process - Shift the IRQ actions to the IPC thread.
+ * @ipc_imem: Pointer to imem data-struct
+ * @irq: Irq number
+ */
+void ipc_imem_irq_process(struct iosm_imem *ipc_imem, int irq);
+
+/**
+ * imem_get_device_sleep_state - Get the device sleep state value.
+ * @ipc_imem: Pointer to imem instance
+ *
+ * Returns: device sleep state
+ */
+int imem_get_device_sleep_state(struct iosm_imem *ipc_imem);
+
+/**
+ * ipc_imem_td_update_timer_suspend - Updates the TD Update Timer suspend flag.
+ * @ipc_imem: Pointer to imem data-struct
+ * @suspend: Flag to update. If TRUE then HP update doorbell is triggered to
+ * device without any wait. If FALSE then HP update doorbell is
+ * delayed until timeout.
+ */
+void ipc_imem_td_update_timer_suspend(struct iosm_imem *ipc_imem, bool suspend);
+
+/**
+ * ipc_imem_channel_close - Release the channel resources.
+ * @ipc_imem: Pointer to imem data-struct
+ * @channel_id: Channel ID to be cleaned up.
+ */
+void ipc_imem_channel_close(struct iosm_imem *ipc_imem, int channel_id);
+
+/**
+ * ipc_imem_channel_alloc - Reserves a channel
+ * @ipc_imem: Pointer to imem data-struct
+ * @index: ID to lookup from the preallocated list.
+ * @ctype: Channel type.
+ *
+ * Returns: Index on success and failure value on error
+ */
+int ipc_imem_channel_alloc(struct iosm_imem *ipc_imem, int index,
+ enum ipc_ctype ctype);
+
+/**
+ * ipc_imem_channel_open - Establish the pipes.
+ * @ipc_imem: Pointer to imem data-struct
+ * @channel_id: Channel ID returned during alloc.
+ * @db_id: Doorbell ID for trigger identifier.
+ *
+ * Returns: Pointer of ipc_mem_channel on success and NULL on failure.
+ */
+struct ipc_mem_channel *ipc_imem_channel_open(struct iosm_imem *ipc_imem,
+ int channel_id, u32 db_id);
+
+/**
+ * ipc_imem_td_update_timer_start - Starts the TD Update Timer if not running.
+ * @ipc_imem: Pointer to imem data-struct
+ */
+void ipc_imem_td_update_timer_start(struct iosm_imem *ipc_imem);
+
+/**
+ * ipc_imem_ul_write_td - Pass the channel UL list to protocol layer for TD
+ * preparation and sending them to the device.
+ * @ipc_imem: Pointer to imem data-struct
+ *
+ * Returns: TRUE of HP Doorbell trigger is pending. FALSE otherwise.
+ */
+bool ipc_imem_ul_write_td(struct iosm_imem *ipc_imem);
+
+/**
+ * ipc_imem_ul_send - Dequeue SKB from channel list and start with
+ * the uplink transfer.If HP Doorbell is pending to be
+ * triggered then starts the TD Update Timer.
+ * @ipc_imem: Pointer to imem data-struct
+ */
+void ipc_imem_ul_send(struct iosm_imem *ipc_imem);
+
+/**
+ * ipc_imem_channel_update - Set or modify pipe config of an existing channel
+ * @ipc_imem: Pointer to imem data-struct
+ * @id: Channel config index
+ * @chnl_cfg: Channel config struct
+ * @irq_moderation: Timer in usec for irq_moderation
+ */
+void ipc_imem_channel_update(struct iosm_imem *ipc_imem, int id,
+ struct ipc_chnl_cfg chnl_cfg, u32 irq_moderation);
+
+/**
+ * ipc_imem_channel_free -Free an IPC channel.
+ * @channel: Channel to be freed
+ */
+void ipc_imem_channel_free(struct ipc_mem_channel *channel);
+
+/**
+ * ipc_imem_hrtimer_stop - Stop the hrtimer
+ * @hr_timer: Pointer to hrtimer instance
+ */
+void ipc_imem_hrtimer_stop(struct hrtimer *hr_timer);
+
+/**
+ * ipc_imem_pipe_cleanup - Reset volatile pipe content for all channels
+ * @ipc_imem: Pointer to imem data-struct
+ * @pipe: Pipe to cleaned up
+ */
+void ipc_imem_pipe_cleanup(struct iosm_imem *ipc_imem, struct ipc_pipe *pipe);
+
+/**
+ * ipc_imem_pipe_close - Send msg to device to close pipe
+ * @ipc_imem: Pointer to imem data-struct
+ * @pipe: Pipe to be closed
+ */
+void ipc_imem_pipe_close(struct iosm_imem *ipc_imem, struct ipc_pipe *pipe);
+
+/**
+ * ipc_imem_phase_update - Get the CP execution state
+ * and map it to the AP phase.
+ * @ipc_imem: Pointer to imem data-struct
+ *
+ * Returns: Current ap updated phase
+ */
+enum ipc_phase ipc_imem_phase_update(struct iosm_imem *ipc_imem);
+
+/**
+ * ipc_imem_phase_get_string - Return the current operation
+ * phase as string.
+ * @phase: AP phase
+ *
+ * Returns: AP phase string
+ */
+const char *ipc_imem_phase_get_string(enum ipc_phase phase);
+
+/**
+ * ipc_imem_msg_send_feature_set - Send feature set message to modem
+ * @ipc_imem: Pointer to imem data-struct
+ * @reset_enable: 0 = out-of-band, 1 = in-band-crash notification
+ * @atomic_ctx: if disabled call in tasklet context
+ *
+ */
+void ipc_imem_msg_send_feature_set(struct iosm_imem *ipc_imem,
+ unsigned int reset_enable, bool atomic_ctx);
+
+/**
+ * ipc_imem_ipc_init_check - Send the init event to CP, wait a certain time and
+ * set CP to runtime with the context information
+ * @ipc_imem: Pointer to imem data-struct
+ */
+void ipc_imem_ipc_init_check(struct iosm_imem *ipc_imem);
+
+/**
+ * ipc_imem_channel_init - Initialize the channel list with UL/DL pipe pairs.
+ * @ipc_imem: Pointer to imem data-struct
+ * @ctype: Channel type
+ * @chnl_cfg: Channel configuration struct
+ * @irq_moderation: Timer in usec for irq_moderation
+ */
+void ipc_imem_channel_init(struct iosm_imem *ipc_imem, enum ipc_ctype ctype,
+ struct ipc_chnl_cfg chnl_cfg, u32 irq_moderation);
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include <linux/delay.h>
+
+#include "iosm_ipc_chnl_cfg.h"
+#include "iosm_ipc_imem.h"
+#include "iosm_ipc_imem_ops.h"
+#include "iosm_ipc_port.h"
+#include "iosm_ipc_task_queue.h"
+
+/* Open a packet data online channel between the network layer and CP. */
+int ipc_imem_sys_wwan_open(struct iosm_imem *ipc_imem, int if_id)
+{
+ dev_dbg(ipc_imem->dev, "%s if id: %d",
+ ipc_imem_phase_get_string(ipc_imem->phase), if_id);
+
+ /* The network interface is only supported in the runtime phase. */
+ if (ipc_imem_phase_update(ipc_imem) != IPC_P_RUN) {
+ dev_err(ipc_imem->dev, "net:%d : refused phase %s", if_id,
+ ipc_imem_phase_get_string(ipc_imem->phase));
+ return -EIO;
+ }
+
+ /* check for the interafce id
+ * if if_id 1 to 8 then create IP MUX channel sessions.
+ * To start MUX session from 0 as network interface id would start
+ * from 1 so map it to if_id = if_id - 1
+ */
+ if (if_id >= IP_MUX_SESSION_START && if_id <= IP_MUX_SESSION_END)
+ return ipc_mux_open_session(ipc_imem->mux, if_id - 1);
+
+ return -EINVAL;
+}
+
+/* Release a net link to CP. */
+void ipc_imem_sys_wwan_close(struct iosm_imem *ipc_imem, int if_id,
+ int channel_id)
+{
+ if (ipc_imem->mux && if_id >= IP_MUX_SESSION_START &&
+ if_id <= IP_MUX_SESSION_END)
+ ipc_mux_close_session(ipc_imem->mux, if_id - 1);
+}
+
+/* Tasklet call to do uplink transfer. */
+static int ipc_imem_tq_cdev_write(struct iosm_imem *ipc_imem, int arg,
+ void *msg, size_t size)
+{
+ ipc_imem->ev_cdev_write_pending = false;
+ ipc_imem_ul_send(ipc_imem);
+
+ return 0;
+}
+
+/* Through tasklet to do sio write. */
+static int ipc_imem_call_cdev_write(struct iosm_imem *ipc_imem)
+{
+ if (ipc_imem->ev_cdev_write_pending)
+ return -1;
+
+ ipc_imem->ev_cdev_write_pending = true;
+
+ return ipc_task_queue_send_task(ipc_imem, ipc_imem_tq_cdev_write, 0,
+ NULL, 0, false);
+}
+
+/* Function for transfer UL data */
+int ipc_imem_sys_wwan_transmit(struct iosm_imem *ipc_imem,
+ int if_id, int channel_id, struct sk_buff *skb)
+{
+ int ret = -EINVAL;
+
+ if (!ipc_imem || channel_id < 0)
+ goto out;
+
+ /* Is CP Running? */
+ if (ipc_imem->phase != IPC_P_RUN) {
+ dev_dbg(ipc_imem->dev, "phase %s transmit",
+ ipc_imem_phase_get_string(ipc_imem->phase));
+ ret = -EIO;
+ goto out;
+ }
+
+ if (if_id >= IP_MUX_SESSION_START && if_id <= IP_MUX_SESSION_END)
+ /* Route the UL packet through IP MUX Layer */
+ ret = ipc_mux_ul_trigger_encode(ipc_imem->mux,
+ if_id - 1, skb);
+ else
+ dev_err(ipc_imem->dev,
+ "invalid if_id %d: ", if_id);
+out:
+ return ret;
+}
+
+/* Initialize wwan channel */
+void ipc_imem_wwan_channel_init(struct iosm_imem *ipc_imem,
+ enum ipc_mux_protocol mux_type)
+{
+ struct ipc_chnl_cfg chnl_cfg = { 0 };
+
+ ipc_imem->cp_version = ipc_mmio_get_cp_version(ipc_imem->mmio);
+
+ /* If modem version is invalid (0xffffffff), do not initialize WWAN. */
+ if (ipc_imem->cp_version == -1) {
+ dev_err(ipc_imem->dev, "invalid CP version");
+ return;
+ }
+
+ ipc_chnl_cfg_get(&chnl_cfg, ipc_imem->nr_of_channels);
+ ipc_imem_channel_init(ipc_imem, IPC_CTYPE_WWAN, chnl_cfg,
+ IRQ_MOD_OFF);
+
+ /* WWAN registration. */
+ ipc_imem->wwan = ipc_wwan_init(ipc_imem, ipc_imem->dev);
+ if (!ipc_imem->wwan)
+ dev_err(ipc_imem->dev,
+ "failed to register the ipc_wwan interfaces");
+}
+
+/* Map SKB to DMA for transfer */
+static int ipc_imem_map_skb_to_dma(struct iosm_imem *ipc_imem,
+ struct sk_buff *skb)
+{
+ struct iosm_pcie *ipc_pcie = ipc_imem->pcie;
+ char *buf = skb->data;
+ int len = skb->len;
+ dma_addr_t mapping;
+ int ret;
+
+ ret = ipc_pcie_addr_map(ipc_pcie, buf, len, &mapping, DMA_TO_DEVICE);
+
+ if (ret)
+ goto err;
+
+ BUILD_BUG_ON(sizeof(*IPC_CB(skb)) > sizeof(skb->cb));
+
+ IPC_CB(skb)->mapping = mapping;
+ IPC_CB(skb)->direction = DMA_TO_DEVICE;
+ IPC_CB(skb)->len = len;
+ IPC_CB(skb)->op_type = (u8)UL_DEFAULT;
+
+err:
+ return ret;
+}
+
+/* return true if channel is ready for use */
+static bool ipc_imem_is_channel_active(struct iosm_imem *ipc_imem,
+ struct ipc_mem_channel *channel)
+{
+ enum ipc_phase phase;
+
+ /* Update the current operation phase. */
+ phase = ipc_imem->phase;
+
+ /* Select the operation depending on the execution stage. */
+ switch (phase) {
+ case IPC_P_RUN:
+ case IPC_P_PSI:
+ case IPC_P_EBL:
+ break;
+
+ case IPC_P_ROM:
+ /* Prepare the PSI image for the CP ROM driver and
+ * suspend the flash app.
+ */
+ if (channel->state != IMEM_CHANNEL_RESERVED) {
+ dev_err(ipc_imem->dev,
+ "ch[%d]:invalid channel state %d,expected %d",
+ channel->channel_id, channel->state,
+ IMEM_CHANNEL_RESERVED);
+ goto channel_unavailable;
+ }
+ goto channel_available;
+
+ default:
+ /* Ignore uplink actions in all other phases. */
+ dev_err(ipc_imem->dev, "ch[%d]: confused phase %d",
+ channel->channel_id, phase);
+ goto channel_unavailable;
+ }
+ /* Check the full availability of the channel. */
+ if (channel->state != IMEM_CHANNEL_ACTIVE) {
+ dev_err(ipc_imem->dev, "ch[%d]: confused channel state %d",
+ channel->channel_id, channel->state);
+ goto channel_unavailable;
+ }
+
+channel_available:
+ return true;
+
+channel_unavailable:
+ return false;
+}
+
+/* Release a sio link to CP. */
+void ipc_imem_sys_cdev_close(struct iosm_cdev *ipc_cdev)
+{
+ struct iosm_imem *ipc_imem = ipc_cdev->ipc_imem;
+ struct ipc_mem_channel *channel = ipc_cdev->channel;
+ enum ipc_phase curr_phase;
+ int status = 0;
+ u32 tail = 0;
+
+ curr_phase = ipc_imem->phase;
+
+ /* If current phase is IPC_P_OFF or SIO ID is -ve then
+ * channel is already freed. Nothing to do.
+ */
+ if (curr_phase == IPC_P_OFF) {
+ dev_err(ipc_imem->dev,
+ "nothing to do. Current Phase: %s",
+ ipc_imem_phase_get_string(curr_phase));
+ return;
+ }
+
+ if (channel->state == IMEM_CHANNEL_FREE) {
+ dev_err(ipc_imem->dev, "ch[%d]: invalid channel state %d",
+ channel->channel_id, channel->state);
+ return;
+ }
+
+ /* If there are any pending TDs then wait for Timeout/Completion before
+ * closing pipe.
+ */
+ if (channel->ul_pipe.old_tail != channel->ul_pipe.old_head) {
+ ipc_imem->app_notify_ul_pend = 1;
+
+ /* Suspend the user app and wait a certain time for processing
+ * UL Data.
+ */
+ status = wait_for_completion_interruptible_timeout
+ (&ipc_imem->ul_pend_sem,
+ msecs_to_jiffies(IPC_PEND_DATA_TIMEOUT));
+ if (status == 0) {
+ dev_dbg(ipc_imem->dev,
+ "Pend data Timeout UL-Pipe:%d Head:%d Tail:%d",
+ channel->ul_pipe.pipe_nr,
+ channel->ul_pipe.old_head,
+ channel->ul_pipe.old_tail);
+ }
+
+ ipc_imem->app_notify_ul_pend = 0;
+ }
+
+ /* If there are any pending TDs then wait for Timeout/Completion before
+ * closing pipe.
+ */
+ ipc_protocol_get_head_tail_index(ipc_imem->ipc_protocol,
+ &channel->dl_pipe, NULL, &tail);
+
+ if (tail != channel->dl_pipe.old_tail) {
+ ipc_imem->app_notify_dl_pend = 1;
+
+ /* Suspend the user app and wait a certain time for processing
+ * DL Data.
+ */
+ status = wait_for_completion_interruptible_timeout
+ (&ipc_imem->dl_pend_sem,
+ msecs_to_jiffies(IPC_PEND_DATA_TIMEOUT));
+ if (status == 0) {
+ dev_dbg(ipc_imem->dev,
+ "Pend data Timeout DL-Pipe:%d Head:%d Tail:%d",
+ channel->dl_pipe.pipe_nr,
+ channel->dl_pipe.old_head,
+ channel->dl_pipe.old_tail);
+ }
+
+ ipc_imem->app_notify_dl_pend = 0;
+ }
+
+ /* Due to wait for completion in messages, there is a small window
+ * between closing the pipe and updating the channel is closed. In this
+ * small window there could be HP update from Host Driver. Hence update
+ * the channel state as CLOSING to aviod unnecessary interrupt
+ * towards CP.
+ */
+ channel->state = IMEM_CHANNEL_CLOSING;
+
+ ipc_imem_pipe_close(ipc_imem, &channel->ul_pipe);
+ ipc_imem_pipe_close(ipc_imem, &channel->dl_pipe);
+
+ ipc_imem_channel_free(channel);
+}
+
+/* Open a PORT link to CP and return the channel */
+struct ipc_mem_channel *ipc_imem_sys_port_open(struct iosm_imem *ipc_imem,
+ int chl_id, int hp_id)
+{
+ struct ipc_mem_channel *channel;
+ int ch_id;
+
+ /* The PORT interface is only supported in the runtime phase. */
+ if (ipc_imem_phase_update(ipc_imem) != IPC_P_RUN) {
+ dev_err(ipc_imem->dev, "PORT open refused, phase %s",
+ ipc_imem_phase_get_string(ipc_imem->phase));
+ return NULL;
+ }
+
+ ch_id = ipc_imem_channel_alloc(ipc_imem, chl_id, IPC_CTYPE_CTRL);
+
+ if (ch_id < 0) {
+ dev_err(ipc_imem->dev, "reservation of an PORT chnl id failed");
+ return NULL;
+ }
+
+ channel = ipc_imem_channel_open(ipc_imem, ch_id, hp_id);
+
+ if (!channel) {
+ dev_err(ipc_imem->dev, "PORT channel id open failed");
+ return NULL;
+ }
+
+ return channel;
+}
+
+/* transfer skb to modem */
+int ipc_imem_sys_cdev_write(struct iosm_cdev *ipc_cdev, struct sk_buff *skb)
+{
+ struct ipc_mem_channel *channel = ipc_cdev->channel;
+ struct iosm_imem *ipc_imem = ipc_cdev->ipc_imem;
+ int ret = -EIO;
+
+ if (!ipc_imem_is_channel_active(ipc_imem, channel) ||
+ ipc_imem->phase == IPC_P_OFF_REQ)
+ goto out;
+
+ ret = ipc_imem_map_skb_to_dma(ipc_imem, skb);
+
+ if (ret)
+ goto out;
+
+ /* Add skb to the uplink skbuf accumulator. */
+ skb_queue_tail(&channel->ul_list, skb);
+
+ ret = ipc_imem_call_cdev_write(ipc_imem);
+
+ if (ret) {
+ skb_dequeue_tail(&channel->ul_list);
+ dev_err(ipc_cdev->dev, "channel id[%d] write failed\n",
+ ipc_cdev->channel->channel_id);
+ }
+out:
+ return ret;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#ifndef IOSM_IPC_IMEM_OPS_H
+#define IOSM_IPC_IMEM_OPS_H
+
+#include "iosm_ipc_mux_codec.h"
+
+/* Maximum wait time for blocking read */
+#define IPC_READ_TIMEOUT 500
+
+/* The delay in ms for defering the unregister */
+#define SIO_UNREGISTER_DEFER_DELAY_MS 1
+
+/* Default delay till CP PSI image is running and modem updates the
+ * execution stage.
+ * unit : milliseconds
+ */
+#define PSI_START_DEFAULT_TIMEOUT 3000
+
+/* Default time out when closing SIO, till the modem is in
+ * running state.
+ * unit : milliseconds
+ */
+#define BOOT_CHECK_DEFAULT_TIMEOUT 400
+
+/* IP MUX channel range */
+#define IP_MUX_SESSION_START 1
+#define IP_MUX_SESSION_END 8
+
+/* Default IP MUX channel */
+#define IP_MUX_SESSION_DEFAULT 1
+
+/**
+ * ipc_imem_sys_port_open - Open a port link to CP.
+ * @ipc_imem: Imem instance.
+ * @chl_id: Channel Indentifier.
+ * @hp_id: HP Indentifier.
+ *
+ * Return: channel instance on success, NULL for failure
+ */
+struct ipc_mem_channel *ipc_imem_sys_port_open(struct iosm_imem *ipc_imem,
+ int chl_id, int hp_id);
+
+/**
+ * ipc_imem_sys_cdev_close - Release a sio link to CP.
+ * @ipc_cdev: iosm sio instance.
+ */
+void ipc_imem_sys_cdev_close(struct iosm_cdev *ipc_cdev);
+
+/**
+ * ipc_imem_sys_cdev_write - Route the uplink buffer to CP.
+ * @ipc_cdev: iosm_cdev instance.
+ * @skb: Pointer to skb.
+ *
+ * Return: 0 on success and failure value on error
+ */
+int ipc_imem_sys_cdev_write(struct iosm_cdev *ipc_cdev, struct sk_buff *skb);
+
+/**
+ * ipc_imem_sys_wwan_open - Open packet data online channel between network
+ * layer and CP.
+ * @ipc_imem: Imem instance.
+ * @if_id: ip link tag of the net device.
+ *
+ * Return: Channel ID on success and failure value on error
+ */
+int ipc_imem_sys_wwan_open(struct iosm_imem *ipc_imem, int if_id);
+
+/**
+ * ipc_imem_sys_wwan_close - Close packet data online channel between network
+ * layer and CP.
+ * @ipc_imem: Imem instance.
+ * @if_id: IP link id net device.
+ * @channel_id: Channel ID to be closed.
+ */
+void ipc_imem_sys_wwan_close(struct iosm_imem *ipc_imem, int if_id,
+ int channel_id);
+
+/**
+ * ipc_imem_sys_wwan_transmit - Function for transfer UL data
+ * @ipc_imem: Imem instance.
+ * @if_id: link ID of the device.
+ * @channel_id: Channel ID used
+ * @skb: Pointer to sk buffer
+ *
+ * Return: 0 on success and failure value on error
+ */
+int ipc_imem_sys_wwan_transmit(struct iosm_imem *ipc_imem, int if_id,
+ int channel_id, struct sk_buff *skb);
+/**
+ * ipc_imem_wwan_channel_init - Initializes WWAN channels and the channel for
+ * MUX.
+ * @ipc_imem: Pointer to iosm_imem struct.
+ * @mux_type: Type of mux protocol.
+ */
+void ipc_imem_wwan_channel_init(struct iosm_imem *ipc_imem,
+ enum ipc_mux_protocol mux_type);
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include "iosm_ipc_pcie.h"
+#include "iosm_ipc_protocol.h"
+
+static void ipc_write_dbell_reg(struct iosm_pcie *ipc_pcie, int irq_n, u32 data)
+{
+ void __iomem *write_reg;
+
+ /* Select the first doorbell register, which is only currently needed
+ * by CP.
+ */
+ write_reg = (void __iomem *)((u8 __iomem *)ipc_pcie->ipc_regs +
+ ipc_pcie->doorbell_write +
+ (irq_n * ipc_pcie->doorbell_reg_offset));
+
+ /* Fire the doorbell irq by writing data on the doorbell write pointer
+ * register.
+ */
+ iowrite32(data, write_reg);
+}
+
+void ipc_doorbell_fire(struct iosm_pcie *ipc_pcie, int irq_n, u32 data)
+{
+ ipc_write_dbell_reg(ipc_pcie, irq_n, data);
+}
+
+/* Threaded Interrupt handler for MSI interrupts */
+static irqreturn_t ipc_msi_interrupt(int irq, void *dev_id)
+{
+ struct iosm_pcie *ipc_pcie = dev_id;
+ int instance = irq - ipc_pcie->pci->irq;
+
+ /* Shift the MSI irq actions to the IPC tasklet. IRQ_NONE means the
+ * irq was not from the IPC device or could not be served.
+ */
+ if (instance >= ipc_pcie->nvec)
+ return IRQ_NONE;
+
+ if (!test_bit(0, &ipc_pcie->suspend))
+ ipc_imem_irq_process(ipc_pcie->imem, instance);
+
+ return IRQ_HANDLED;
+}
+
+void ipc_release_irq(struct iosm_pcie *ipc_pcie)
+{
+ struct pci_dev *pdev = ipc_pcie->pci;
+
+ if (pdev->msi_enabled) {
+ while (--ipc_pcie->nvec >= 0)
+ free_irq(pdev->irq + ipc_pcie->nvec, ipc_pcie);
+ }
+ pci_free_irq_vectors(pdev);
+}
+
+int ipc_acquire_irq(struct iosm_pcie *ipc_pcie)
+{
+ struct pci_dev *pdev = ipc_pcie->pci;
+ int i, rc = -EINVAL;
+
+ ipc_pcie->nvec = pci_alloc_irq_vectors(pdev, IPC_MSI_VECTORS,
+ IPC_MSI_VECTORS, PCI_IRQ_MSI);
+
+ if (ipc_pcie->nvec < 0) {
+ rc = ipc_pcie->nvec;
+ goto error;
+ }
+
+ if (!pdev->msi_enabled)
+ goto error;
+
+ for (i = 0; i < ipc_pcie->nvec; ++i) {
+ rc = request_threaded_irq(pdev->irq + i, NULL,
+ ipc_msi_interrupt, IRQF_ONESHOT,
+ KBUILD_MODNAME, ipc_pcie);
+ if (rc) {
+ dev_err(ipc_pcie->dev, "unable to grab IRQ, rc=%d", rc);
+ ipc_pcie->nvec = i;
+ ipc_release_irq(ipc_pcie);
+ goto error;
+ }
+ }
+
+error:
+ return rc;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#ifndef IOSM_IPC_IRQ_H
+#define IOSM_IPC_IRQ_H
+
+struct iosm_pcie;
+
+/**
+ * ipc_doorbell_fire - fire doorbell to CP
+ * @ipc_pcie: Pointer to iosm_pcie
+ * @irq_n: Doorbell type
+ * @data: ipc state
+ */
+void ipc_doorbell_fire(struct iosm_pcie *ipc_pcie, int irq_n, u32 data);
+
+/**
+ * ipc_release_irq - Release the IRQ handler.
+ * @ipc_pcie: Pointer to iosm_pcie struct
+ */
+void ipc_release_irq(struct iosm_pcie *ipc_pcie);
+
+/**
+ * ipc_acquire_irq - acquire IRQ & register IRQ handler.
+ * @ipc_pcie: Pointer to iosm_pcie struct
+ *
+ * Return: 0 on success and failure value on error
+ */
+int ipc_acquire_irq(struct iosm_pcie *ipc_pcie);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/io.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/slab.h>
+
+#include "iosm_ipc_mmio.h"
+
+/* Definition of MMIO offsets
+ * note that MMIO_CI offsets are relative to end of chip info structure
+ */
+
+/* MMIO chip info size in bytes */
+#define MMIO_CHIP_INFO_SIZE 60
+
+/* CP execution stage */
+#define MMIO_OFFSET_EXECUTION_STAGE 0x00
+
+/* Boot ROM Chip Info struct */
+#define MMIO_OFFSET_CHIP_INFO 0x04
+
+#define MMIO_OFFSET_ROM_EXIT_CODE 0x40
+
+#define MMIO_OFFSET_PSI_ADDRESS 0x54
+
+#define MMIO_OFFSET_PSI_SIZE 0x5C
+
+#define MMIO_OFFSET_IPC_STATUS 0x60
+
+#define MMIO_OFFSET_CONTEXT_INFO 0x64
+
+#define MMIO_OFFSET_BASE_ADDR 0x6C
+
+#define MMIO_OFFSET_END_ADDR 0x74
+
+#define MMIO_OFFSET_CP_VERSION 0xF0
+
+#define MMIO_OFFSET_CP_CAPABILITIES 0xF4
+
+/* Timeout in 50 msec to wait for the modem boot code to write a valid
+ * execution stage into mmio area
+ */
+#define IPC_MMIO_EXEC_STAGE_TIMEOUT 50
+
+/* check if exec stage has one of the valid values */
+static bool ipc_mmio_is_valid_exec_stage(enum ipc_mem_exec_stage stage)
+{
+ switch (stage) {
+ case IPC_MEM_EXEC_STAGE_BOOT:
+ case IPC_MEM_EXEC_STAGE_PSI:
+ case IPC_MEM_EXEC_STAGE_EBL:
+ case IPC_MEM_EXEC_STAGE_RUN:
+ case IPC_MEM_EXEC_STAGE_CRASH:
+ case IPC_MEM_EXEC_STAGE_CD_READY:
+ return true;
+ default:
+ return false;
+ }
+}
+
+void ipc_mmio_update_cp_capability(struct iosm_mmio *ipc_mmio)
+{
+ u32 cp_cap;
+ unsigned int ver;
+
+ ver = ipc_mmio_get_cp_version(ipc_mmio);
+ cp_cap = readl(ipc_mmio->base + ipc_mmio->offset.cp_capability);
+
+ ipc_mmio->has_mux_lite = (ver >= IOSM_CP_VERSION) &&
+ !(cp_cap & DL_AGGR) && !(cp_cap & UL_AGGR);
+
+ ipc_mmio->has_ul_flow_credit =
+ (ver >= IOSM_CP_VERSION) && (cp_cap & UL_FLOW_CREDIT);
+}
+
+struct iosm_mmio *ipc_mmio_init(void __iomem *mmio, struct device *dev)
+{
+ struct iosm_mmio *ipc_mmio = kzalloc(sizeof(*ipc_mmio), GFP_KERNEL);
+ int retries = IPC_MMIO_EXEC_STAGE_TIMEOUT;
+ enum ipc_mem_exec_stage stage;
+
+ if (!ipc_mmio)
+ return NULL;
+
+ ipc_mmio->dev = dev;
+
+ ipc_mmio->base = mmio;
+
+ ipc_mmio->offset.exec_stage = MMIO_OFFSET_EXECUTION_STAGE;
+
+ /* Check for a valid execution stage to make sure that the boot code
+ * has correctly initialized the MMIO area.
+ */
+ do {
+ stage = ipc_mmio_get_exec_stage(ipc_mmio);
+ if (ipc_mmio_is_valid_exec_stage(stage))
+ break;
+
+ msleep(20);
+ } while (retries-- > 0);
+
+ if (!retries) {
+ dev_err(ipc_mmio->dev, "invalid exec stage %X", stage);
+ goto init_fail;
+ }
+
+ ipc_mmio->offset.chip_info = MMIO_OFFSET_CHIP_INFO;
+
+ /* read chip info size and version from chip info structure */
+ ipc_mmio->chip_info_version =
+ ioread8(ipc_mmio->base + ipc_mmio->offset.chip_info);
+
+ /* Increment of 2 is needed as the size value in the chip info
+ * excludes the version and size field, which are always present
+ */
+ ipc_mmio->chip_info_size =
+ ioread8(ipc_mmio->base + ipc_mmio->offset.chip_info + 1) + 2;
+
+ if (ipc_mmio->chip_info_size != MMIO_CHIP_INFO_SIZE) {
+ dev_err(ipc_mmio->dev, "Unexpected Chip Info");
+ goto init_fail;
+ }
+
+ ipc_mmio->offset.rom_exit_code = MMIO_OFFSET_ROM_EXIT_CODE;
+
+ ipc_mmio->offset.psi_address = MMIO_OFFSET_PSI_ADDRESS;
+ ipc_mmio->offset.psi_size = MMIO_OFFSET_PSI_SIZE;
+ ipc_mmio->offset.ipc_status = MMIO_OFFSET_IPC_STATUS;
+ ipc_mmio->offset.context_info = MMIO_OFFSET_CONTEXT_INFO;
+ ipc_mmio->offset.ap_win_base = MMIO_OFFSET_BASE_ADDR;
+ ipc_mmio->offset.ap_win_end = MMIO_OFFSET_END_ADDR;
+
+ ipc_mmio->offset.cp_version = MMIO_OFFSET_CP_VERSION;
+ ipc_mmio->offset.cp_capability = MMIO_OFFSET_CP_CAPABILITIES;
+
+ return ipc_mmio;
+
+init_fail:
+ kfree(ipc_mmio);
+ return NULL;
+}
+
+enum ipc_mem_exec_stage ipc_mmio_get_exec_stage(struct iosm_mmio *ipc_mmio)
+{
+ if (!ipc_mmio)
+ return IPC_MEM_EXEC_STAGE_INVALID;
+
+ return (enum ipc_mem_exec_stage)readl(ipc_mmio->base +
+ ipc_mmio->offset.exec_stage);
+}
+
+void ipc_mmio_copy_chip_info(struct iosm_mmio *ipc_mmio, void *dest,
+ size_t size)
+{
+ if (ipc_mmio && dest)
+ memcpy_fromio(dest, ipc_mmio->base + ipc_mmio->offset.chip_info,
+ size);
+}
+
+enum ipc_mem_device_ipc_state ipc_mmio_get_ipc_state(struct iosm_mmio *ipc_mmio)
+{
+ if (!ipc_mmio)
+ return IPC_MEM_DEVICE_IPC_INVALID;
+
+ return (enum ipc_mem_device_ipc_state)
+ readl(ipc_mmio->base + ipc_mmio->offset.ipc_status);
+}
+
+enum rom_exit_code ipc_mmio_get_rom_exit_code(struct iosm_mmio *ipc_mmio)
+{
+ if (!ipc_mmio)
+ return IMEM_ROM_EXIT_FAIL;
+
+ return (enum rom_exit_code)readl(ipc_mmio->base +
+ ipc_mmio->offset.rom_exit_code);
+}
+
+void ipc_mmio_config(struct iosm_mmio *ipc_mmio)
+{
+ if (!ipc_mmio)
+ return;
+
+ /* AP memory window (full window is open and active so that modem checks
+ * each AP address) 0 means don't check on modem side.
+ */
+ iowrite64_lo_hi(0, ipc_mmio->base + ipc_mmio->offset.ap_win_base);
+ iowrite64_lo_hi(0, ipc_mmio->base + ipc_mmio->offset.ap_win_end);
+
+ iowrite64_lo_hi(ipc_mmio->context_info_addr,
+ ipc_mmio->base + ipc_mmio->offset.context_info);
+}
+
+void ipc_mmio_set_psi_addr_and_size(struct iosm_mmio *ipc_mmio, dma_addr_t addr,
+ u32 size)
+{
+ if (!ipc_mmio)
+ return;
+
+ iowrite64_lo_hi(addr, ipc_mmio->base + ipc_mmio->offset.psi_address);
+ writel(size, ipc_mmio->base + ipc_mmio->offset.psi_size);
+}
+
+void ipc_mmio_set_contex_info_addr(struct iosm_mmio *ipc_mmio, phys_addr_t addr)
+{
+ if (!ipc_mmio)
+ return;
+
+ /* store context_info address. This will be stored in the mmio area
+ * during IPC_MEM_DEVICE_IPC_INIT state via ipc_mmio_config()
+ */
+ ipc_mmio->context_info_addr = addr;
+}
+
+int ipc_mmio_get_cp_version(struct iosm_mmio *ipc_mmio)
+{
+ return ipc_mmio ? readl(ipc_mmio->base + ipc_mmio->offset.cp_version) :
+ -EFAULT;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#ifndef IOSM_IPC_MMIO_H
+#define IOSM_IPC_MMIO_H
+
+/* Minimal IOSM CP VERSION which has valid CP_CAPABILITIES field */
+#define IOSM_CP_VERSION 0x0100UL
+
+/* DL dir Aggregation support mask */
+#define DL_AGGR BIT(23)
+
+/* UL dir Aggregation support mask */
+#define UL_AGGR BIT(22)
+
+/* UL flow credit support mask */
+#define UL_FLOW_CREDIT BIT(21)
+
+/* Possible states of the IPC finite state machine. */
+enum ipc_mem_device_ipc_state {
+ IPC_MEM_DEVICE_IPC_UNINIT,
+ IPC_MEM_DEVICE_IPC_INIT,
+ IPC_MEM_DEVICE_IPC_RUNNING,
+ IPC_MEM_DEVICE_IPC_RECOVERY,
+ IPC_MEM_DEVICE_IPC_ERROR,
+ IPC_MEM_DEVICE_IPC_DONT_CARE,
+ IPC_MEM_DEVICE_IPC_INVALID = -1
+};
+
+/* Boot ROM exit status. */
+enum rom_exit_code {
+ IMEM_ROM_EXIT_OPEN_EXT = 0x01,
+ IMEM_ROM_EXIT_OPEN_MEM = 0x02,
+ IMEM_ROM_EXIT_CERT_EXT = 0x10,
+ IMEM_ROM_EXIT_CERT_MEM = 0x20,
+ IMEM_ROM_EXIT_FAIL = 0xFF
+};
+
+/* Boot stages */
+enum ipc_mem_exec_stage {
+ IPC_MEM_EXEC_STAGE_RUN = 0x600DF00D,
+ IPC_MEM_EXEC_STAGE_CRASH = 0x8BADF00D,
+ IPC_MEM_EXEC_STAGE_CD_READY = 0xBADC0DED,
+ IPC_MEM_EXEC_STAGE_BOOT = 0xFEEDB007,
+ IPC_MEM_EXEC_STAGE_PSI = 0xFEEDBEEF,
+ IPC_MEM_EXEC_STAGE_EBL = 0xFEEDCAFE,
+ IPC_MEM_EXEC_STAGE_INVALID = 0xFFFFFFFF
+};
+
+/* mmio scratchpad info */
+struct mmio_offset {
+ int exec_stage;
+ int chip_info;
+ int rom_exit_code;
+ int psi_address;
+ int psi_size;
+ int ipc_status;
+ int context_info;
+ int ap_win_base;
+ int ap_win_end;
+ int cp_version;
+ int cp_capability;
+};
+
+/**
+ * struct iosm_mmio - MMIO region mapped to the doorbell scratchpad.
+ * @base: Base address of MMIO region
+ * @dev: Pointer to device structure
+ * @offset: Start offset
+ * @context_info_addr: Physical base address of context info structure
+ * @chip_info_version: Version of chip info structure
+ * @chip_info_size: Size of chip info structure
+ * @has_mux_lite: It doesn't support mux aggergation
+ * @has_ul_flow_credit: Ul flow credit support
+ * @has_slp_no_prot: Device sleep no protocol support
+ * @has_mcr_support: Usage of mcr support
+ */
+struct iosm_mmio {
+ unsigned char __iomem *base;
+ struct device *dev;
+ struct mmio_offset offset;
+ phys_addr_t context_info_addr;
+ unsigned int chip_info_version;
+ unsigned int chip_info_size;
+ u8 has_mux_lite:1,
+ has_ul_flow_credit:1,
+ has_slp_no_prot:1,
+ has_mcr_support:1;
+};
+
+/**
+ * ipc_mmio_init - Allocate mmio instance data
+ * @mmio_addr: Mapped AP base address of the MMIO area.
+ * @dev: Pointer to device structure
+ *
+ * Returns: address of mmio instance data or NULL if fails.
+ */
+struct iosm_mmio *ipc_mmio_init(void __iomem *mmio_addr, struct device *dev);
+
+/**
+ * ipc_mmio_set_psi_addr_and_size - Set start address and size of the
+ * primary system image (PSI) for the
+ * FW dowload.
+ * @ipc_mmio: Pointer to mmio instance
+ * @addr: PSI address
+ * @size: PSI immage size
+ */
+void ipc_mmio_set_psi_addr_and_size(struct iosm_mmio *ipc_mmio, dma_addr_t addr,
+ u32 size);
+
+/**
+ * ipc_mmio_set_contex_info_addr - Stores the Context Info Address in
+ * MMIO instance to share it with CP during
+ * mmio_init.
+ * @ipc_mmio: Pointer to mmio instance
+ * @addr: 64-bit address of AP context information.
+ */
+void ipc_mmio_set_contex_info_addr(struct iosm_mmio *ipc_mmio,
+ phys_addr_t addr);
+
+/**
+ * ipc_mmio_get_cp_version - Get the CP IPC version
+ * @ipc_mmio: Pointer to mmio instance
+ *
+ * Returns: version number on success and failure value on error.
+ */
+int ipc_mmio_get_cp_version(struct iosm_mmio *ipc_mmio);
+
+/**
+ * ipc_mmio_get_rom_exit_code - Get exit code from CP boot rom download app
+ * @ipc_mmio: Pointer to mmio instance
+ *
+ * Returns: exit code from CP boot rom download APP
+ */
+enum rom_exit_code ipc_mmio_get_rom_exit_code(struct iosm_mmio *ipc_mmio);
+
+/**
+ * ipc_mmio_get_exec_stage - Query CP execution stage
+ * @ipc_mmio: Pointer to mmio instance
+ *
+ * Returns: CP execution stage
+ */
+enum ipc_mem_exec_stage ipc_mmio_get_exec_stage(struct iosm_mmio *ipc_mmio);
+
+/**
+ * ipc_mmio_get_ipc_state - Query CP IPC state
+ * @ipc_mmio: Pointer to mmio instance
+ *
+ * Returns: CP IPC state
+ */
+enum ipc_mem_device_ipc_state
+ipc_mmio_get_ipc_state(struct iosm_mmio *ipc_mmio);
+
+/**
+ * ipc_mmio_copy_chip_info - Copy size bytes of CP chip info structure
+ * into caller provided buffer
+ * @ipc_mmio: Pointer to mmio instance
+ * @dest: Pointer to caller provided buff
+ * @size: Number of bytes to copy
+ */
+void ipc_mmio_copy_chip_info(struct iosm_mmio *ipc_mmio, void *dest,
+ size_t size);
+
+/**
+ * ipc_mmio_config - Write context info and AP memory range addresses.
+ * This needs to be called when CP is in
+ * IPC_MEM_DEVICE_IPC_INIT state
+ *
+ * @ipc_mmio: Pointer to mmio instance
+ */
+void ipc_mmio_config(struct iosm_mmio *ipc_mmio);
+
+/**
+ * ipc_mmio_update_cp_capability - Read and update modem capability, from mmio
+ * capability offset
+ *
+ * @ipc_mmio: Pointer to mmio instance
+ */
+void ipc_mmio_update_cp_capability(struct iosm_mmio *ipc_mmio);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include "iosm_ipc_mux_codec.h"
+
+/* At the begin of the runtime phase the IP MUX channel shall created. */
+static int ipc_mux_channel_create(struct iosm_mux *ipc_mux)
+{
+ int channel_id;
+
+ channel_id = ipc_imem_channel_alloc(ipc_mux->imem, ipc_mux->instance_id,
+ IPC_CTYPE_WWAN);
+
+ if (channel_id < 0) {
+ dev_err(ipc_mux->dev,
+ "allocation of the MUX channel id failed");
+ ipc_mux->state = MUX_S_ERROR;
+ ipc_mux->event = MUX_E_NOT_APPLICABLE;
+ goto no_channel;
+ }
+
+ /* Establish the MUX channel in blocking mode. */
+ ipc_mux->channel = ipc_imem_channel_open(ipc_mux->imem, channel_id,
+ IPC_HP_NET_CHANNEL_INIT);
+
+ if (!ipc_mux->channel) {
+ dev_err(ipc_mux->dev, "ipc_imem_channel_open failed");
+ ipc_mux->state = MUX_S_ERROR;
+ ipc_mux->event = MUX_E_NOT_APPLICABLE;
+ return -ENODEV; /* MUX channel is not available. */
+ }
+
+ /* Define the MUX active state properties. */
+ ipc_mux->state = MUX_S_ACTIVE;
+ ipc_mux->event = MUX_E_NO_ORDERS;
+
+no_channel:
+ return channel_id;
+}
+
+/* Reset the session/if id state. */
+static void ipc_mux_session_free(struct iosm_mux *ipc_mux, int if_id)
+{
+ struct mux_session *if_entry;
+
+ if_entry = &ipc_mux->session[if_id];
+ /* Reset the session state. */
+ if_entry->wwan = NULL;
+}
+
+/* Create and send the session open command. */
+static struct mux_cmd_open_session_resp *
+ipc_mux_session_open_send(struct iosm_mux *ipc_mux, int if_id)
+{
+ struct mux_cmd_open_session_resp *open_session_resp;
+ struct mux_acb *acb = &ipc_mux->acb;
+ union mux_cmd_param param;
+
+ /* open_session commands to one ACB and start transmission. */
+ param.open_session.flow_ctrl = 0;
+ param.open_session.ipv4v6_hints = 0;
+ param.open_session.reserved2 = 0;
+ param.open_session.dl_head_pad_len = cpu_to_le32(IPC_MEM_DL_ETH_OFFSET);
+
+ /* Finish and transfer ACB. The user thread is suspended.
+ * It is a blocking function call, until CP responds or timeout.
+ */
+ acb->wanted_response = MUX_CMD_OPEN_SESSION_RESP;
+ if (ipc_mux_dl_acb_send_cmds(ipc_mux, MUX_CMD_OPEN_SESSION, if_id, 0,
+ ¶m, sizeof(param.open_session), true,
+ false) ||
+ acb->got_response != MUX_CMD_OPEN_SESSION_RESP) {
+ dev_err(ipc_mux->dev, "if_id %d: OPEN_SESSION send failed",
+ if_id);
+ return NULL;
+ }
+
+ open_session_resp = &ipc_mux->acb.got_param.open_session_resp;
+ if (open_session_resp->response != cpu_to_le32(MUX_CMD_RESP_SUCCESS)) {
+ dev_err(ipc_mux->dev,
+ "if_id %d,session open failed,response=%d", if_id,
+ open_session_resp->response);
+ return NULL;
+ }
+
+ return open_session_resp;
+}
+
+/* Open the first IP session. */
+static bool ipc_mux_session_open(struct iosm_mux *ipc_mux,
+ struct mux_session_open *session_open)
+{
+ struct mux_cmd_open_session_resp *open_session_resp;
+ int if_id;
+
+ /* Search for a free session interface id. */
+ if_id = le32_to_cpu(session_open->if_id);
+ if (if_id < 0 || if_id >= ipc_mux->nr_sessions) {
+ dev_err(ipc_mux->dev, "invalid interface id=%d", if_id);
+ return false;
+ }
+
+ /* Create and send the session open command.
+ * It is a blocking function call, until CP responds or timeout.
+ */
+ open_session_resp = ipc_mux_session_open_send(ipc_mux, if_id);
+ if (!open_session_resp) {
+ ipc_mux_session_free(ipc_mux, if_id);
+ session_open->if_id = cpu_to_le32(-1);
+ return false;
+ }
+
+ /* Initialize the uplink skb accumulator. */
+ skb_queue_head_init(&ipc_mux->session[if_id].ul_list);
+
+ ipc_mux->session[if_id].dl_head_pad_len = IPC_MEM_DL_ETH_OFFSET;
+ ipc_mux->session[if_id].ul_head_pad_len =
+ le32_to_cpu(open_session_resp->ul_head_pad_len);
+ ipc_mux->session[if_id].wwan = ipc_mux->wwan;
+
+ /* Reset the flow ctrl stats of the session */
+ ipc_mux->session[if_id].flow_ctl_en_cnt = 0;
+ ipc_mux->session[if_id].flow_ctl_dis_cnt = 0;
+ ipc_mux->session[if_id].ul_flow_credits = 0;
+ ipc_mux->session[if_id].net_tx_stop = false;
+ ipc_mux->session[if_id].flow_ctl_mask = 0;
+
+ /* Save and return the assigned if id. */
+ session_open->if_id = cpu_to_le32(if_id);
+
+ return true;
+}
+
+/* Free pending session UL packet. */
+static void ipc_mux_session_reset(struct iosm_mux *ipc_mux, int if_id)
+{
+ /* Reset the session/if id state. */
+ ipc_mux_session_free(ipc_mux, if_id);
+
+ /* Empty the uplink skb accumulator. */
+ skb_queue_purge(&ipc_mux->session[if_id].ul_list);
+}
+
+static void ipc_mux_session_close(struct iosm_mux *ipc_mux,
+ struct mux_session_close *msg)
+{
+ int if_id;
+
+ /* Copy the session interface id. */
+ if_id = le32_to_cpu(msg->if_id);
+
+ if (if_id < 0 || if_id >= ipc_mux->nr_sessions) {
+ dev_err(ipc_mux->dev, "invalid session id %d", if_id);
+ return;
+ }
+
+ /* Create and send the session close command.
+ * It is a blocking function call, until CP responds or timeout.
+ */
+ if (ipc_mux_dl_acb_send_cmds(ipc_mux, MUX_CMD_CLOSE_SESSION, if_id, 0,
+ NULL, 0, true, false))
+ dev_err(ipc_mux->dev, "if_id %d: CLOSE_SESSION send failed",
+ if_id);
+
+ /* Reset the flow ctrl stats of the session */
+ ipc_mux->session[if_id].flow_ctl_en_cnt = 0;
+ ipc_mux->session[if_id].flow_ctl_dis_cnt = 0;
+ ipc_mux->session[if_id].flow_ctl_mask = 0;
+
+ ipc_mux_session_reset(ipc_mux, if_id);
+}
+
+static void ipc_mux_channel_close(struct iosm_mux *ipc_mux,
+ struct mux_channel_close *channel_close_p)
+{
+ int i;
+
+ /* Free pending session UL packet. */
+ for (i = 0; i < ipc_mux->nr_sessions; i++)
+ if (ipc_mux->session[i].wwan)
+ ipc_mux_session_reset(ipc_mux, i);
+
+ ipc_imem_channel_close(ipc_mux->imem, ipc_mux->channel_id);
+
+ /* Reset the MUX object. */
+ ipc_mux->state = MUX_S_INACTIVE;
+ ipc_mux->event = MUX_E_INACTIVE;
+}
+
+/* CP has interrupted AP. If AP is in IP MUX mode, execute the pending ops. */
+static int ipc_mux_schedule(struct iosm_mux *ipc_mux, union mux_msg *msg)
+{
+ enum mux_event order;
+ bool success;
+ int ret = -EIO;
+
+ if (!ipc_mux->initialized) {
+ ret = -EAGAIN;
+ goto out;
+ }
+
+ order = msg->common.event;
+
+ switch (ipc_mux->state) {
+ case MUX_S_INACTIVE:
+ if (order != MUX_E_MUX_SESSION_OPEN)
+ goto out; /* Wait for the request to open a session */
+
+ if (ipc_mux->event == MUX_E_INACTIVE)
+ /* Establish the MUX channel and the new state. */
+ ipc_mux->channel_id = ipc_mux_channel_create(ipc_mux);
+
+ if (ipc_mux->state != MUX_S_ACTIVE) {
+ ret = ipc_mux->channel_id; /* Missing the MUX channel */
+ goto out;
+ }
+
+ /* Disable the TD update timer and open the first IP session. */
+ ipc_imem_td_update_timer_suspend(ipc_mux->imem, true);
+ ipc_mux->event = MUX_E_MUX_SESSION_OPEN;
+ success = ipc_mux_session_open(ipc_mux, &msg->session_open);
+
+ ipc_imem_td_update_timer_suspend(ipc_mux->imem, false);
+ if (success)
+ ret = ipc_mux->channel_id;
+ goto out;
+
+ case MUX_S_ACTIVE:
+ switch (order) {
+ case MUX_E_MUX_SESSION_OPEN:
+ /* Disable the TD update timer and open a session */
+ ipc_imem_td_update_timer_suspend(ipc_mux->imem, true);
+ ipc_mux->event = MUX_E_MUX_SESSION_OPEN;
+ success = ipc_mux_session_open(ipc_mux,
+ &msg->session_open);
+ ipc_imem_td_update_timer_suspend(ipc_mux->imem, false);
+ if (success)
+ ret = ipc_mux->channel_id;
+ goto out;
+
+ case MUX_E_MUX_SESSION_CLOSE:
+ /* Release an IP session. */
+ ipc_mux->event = MUX_E_MUX_SESSION_CLOSE;
+ ipc_mux_session_close(ipc_mux, &msg->session_close);
+ ret = ipc_mux->channel_id;
+ goto out;
+
+ case MUX_E_MUX_CHANNEL_CLOSE:
+ /* Close the MUX channel pipes. */
+ ipc_mux->event = MUX_E_MUX_CHANNEL_CLOSE;
+ ipc_mux_channel_close(ipc_mux, &msg->channel_close);
+ ret = ipc_mux->channel_id;
+ goto out;
+
+ default:
+ /* Invalid order. */
+ goto out;
+ }
+
+ default:
+ dev_err(ipc_mux->dev,
+ "unexpected MUX transition: state=%d, event=%d",
+ ipc_mux->state, ipc_mux->event);
+ }
+out:
+ return ret;
+}
+
+struct iosm_mux *ipc_mux_init(struct ipc_mux_config *mux_cfg,
+ struct iosm_imem *imem)
+{
+ struct iosm_mux *ipc_mux = kzalloc(sizeof(*ipc_mux), GFP_KERNEL);
+ int i, ul_tds, ul_td_size;
+ struct sk_buff_head *free_list;
+ struct sk_buff *skb;
+
+ if (!ipc_mux)
+ return NULL;
+
+ ipc_mux->protocol = mux_cfg->protocol;
+ ipc_mux->ul_flow = mux_cfg->ul_flow;
+ ipc_mux->nr_sessions = mux_cfg->nr_sessions;
+ ipc_mux->instance_id = mux_cfg->instance_id;
+ ipc_mux->wwan_q_offset = 0;
+
+ ipc_mux->pcie = imem->pcie;
+ ipc_mux->imem = imem;
+ ipc_mux->ipc_protocol = imem->ipc_protocol;
+ ipc_mux->dev = imem->dev;
+ ipc_mux->wwan = imem->wwan;
+
+ /* Get the reference to the UL ADB list. */
+ free_list = &ipc_mux->ul_adb.free_list;
+
+ /* Initialize the list with free ADB. */
+ skb_queue_head_init(free_list);
+
+ ul_td_size = IPC_MEM_MAX_DL_MUX_LITE_BUF_SIZE;
+
+ ul_tds = IPC_MEM_MAX_TDS_MUX_LITE_UL;
+
+ ipc_mux->ul_adb.dest_skb = NULL;
+
+ ipc_mux->initialized = true;
+ ipc_mux->adb_prep_ongoing = false;
+ ipc_mux->size_needed = 0;
+ ipc_mux->ul_data_pend_bytes = 0;
+ ipc_mux->state = MUX_S_INACTIVE;
+ ipc_mux->ev_mux_net_transmit_pending = false;
+ ipc_mux->tx_transaction_id = 0;
+ ipc_mux->rr_next_session = 0;
+ ipc_mux->event = MUX_E_INACTIVE;
+ ipc_mux->channel_id = -1;
+ ipc_mux->channel = NULL;
+
+ /* Allocate the list of UL ADB. */
+ for (i = 0; i < ul_tds; i++) {
+ dma_addr_t mapping;
+
+ skb = ipc_pcie_alloc_skb(ipc_mux->pcie, ul_td_size, GFP_ATOMIC,
+ &mapping, DMA_TO_DEVICE, 0);
+ if (!skb) {
+ ipc_mux_deinit(ipc_mux);
+ return NULL;
+ }
+ /* Extend the UL ADB list. */
+ skb_queue_tail(free_list, skb);
+ }
+
+ return ipc_mux;
+}
+
+/* Informs the network stack to restart transmission for all opened session if
+ * Flow Control is not ON for that session.
+ */
+static void ipc_mux_restart_tx_for_all_sessions(struct iosm_mux *ipc_mux)
+{
+ struct mux_session *session;
+ int idx;
+
+ for (idx = 0; idx < ipc_mux->nr_sessions; idx++) {
+ session = &ipc_mux->session[idx];
+
+ if (!session->wwan)
+ continue;
+
+ /* If flow control of the session is OFF and if there was tx
+ * stop then restart. Inform the network interface to restart
+ * sending data.
+ */
+ if (session->flow_ctl_mask == 0) {
+ session->net_tx_stop = false;
+ ipc_mux_netif_tx_flowctrl(session, idx, false);
+ }
+ }
+}
+
+/* Informs the network stack to stop sending further pkt for all opened
+ * sessions
+ */
+static void ipc_mux_stop_netif_for_all_sessions(struct iosm_mux *ipc_mux)
+{
+ struct mux_session *session;
+ int idx;
+
+ for (idx = 0; idx < ipc_mux->nr_sessions; idx++) {
+ session = &ipc_mux->session[idx];
+
+ if (!session->wwan)
+ continue;
+
+ ipc_mux_netif_tx_flowctrl(session, session->if_id, true);
+ }
+}
+
+void ipc_mux_check_n_restart_tx(struct iosm_mux *ipc_mux)
+{
+ if (ipc_mux->ul_flow == MUX_UL) {
+ int low_thresh = IPC_MEM_MUX_UL_FLOWCTRL_LOW_B;
+
+ if (ipc_mux->ul_data_pend_bytes < low_thresh)
+ ipc_mux_restart_tx_for_all_sessions(ipc_mux);
+ }
+}
+
+int ipc_mux_get_max_sessions(struct iosm_mux *ipc_mux)
+{
+ return ipc_mux ? ipc_mux->nr_sessions : -EFAULT;
+}
+
+enum ipc_mux_protocol ipc_mux_get_active_protocol(struct iosm_mux *ipc_mux)
+{
+ return ipc_mux ? ipc_mux->protocol : MUX_UNKNOWN;
+}
+
+int ipc_mux_open_session(struct iosm_mux *ipc_mux, int session_nr)
+{
+ struct mux_session_open *session_open;
+ union mux_msg mux_msg;
+
+ session_open = &mux_msg.session_open;
+ session_open->event = MUX_E_MUX_SESSION_OPEN;
+
+ session_open->if_id = cpu_to_le32(session_nr);
+ ipc_mux->session[session_nr].flags |= IPC_MEM_WWAN_MUX;
+ return ipc_mux_schedule(ipc_mux, &mux_msg);
+}
+
+int ipc_mux_close_session(struct iosm_mux *ipc_mux, int session_nr)
+{
+ struct mux_session_close *session_close;
+ union mux_msg mux_msg;
+ int ret_val;
+
+ session_close = &mux_msg.session_close;
+ session_close->event = MUX_E_MUX_SESSION_CLOSE;
+
+ session_close->if_id = cpu_to_le32(session_nr);
+ ret_val = ipc_mux_schedule(ipc_mux, &mux_msg);
+ ipc_mux->session[session_nr].flags &= ~IPC_MEM_WWAN_MUX;
+
+ return ret_val;
+}
+
+void ipc_mux_deinit(struct iosm_mux *ipc_mux)
+{
+ struct mux_channel_close *channel_close;
+ struct sk_buff_head *free_list;
+ union mux_msg mux_msg;
+ struct sk_buff *skb;
+
+ if (!ipc_mux->initialized)
+ return;
+ ipc_mux_stop_netif_for_all_sessions(ipc_mux);
+
+ channel_close = &mux_msg.channel_close;
+ channel_close->event = MUX_E_MUX_CHANNEL_CLOSE;
+ ipc_mux_schedule(ipc_mux, &mux_msg);
+
+ /* Empty the ADB free list. */
+ free_list = &ipc_mux->ul_adb.free_list;
+
+ /* Remove from the head of the downlink queue. */
+ while ((skb = skb_dequeue(free_list)))
+ ipc_pcie_kfree_skb(ipc_mux->pcie, skb);
+
+ if (ipc_mux->channel) {
+ ipc_mux->channel->ul_pipe.is_open = false;
+ ipc_mux->channel->dl_pipe.is_open = false;
+ }
+
+ kfree(ipc_mux);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#ifndef IOSM_IPC_MUX_H
+#define IOSM_IPC_MUX_H
+
+#include "iosm_ipc_protocol.h"
+
+/* Size of the buffer for the IP MUX data buffer. */
+#define IPC_MEM_MAX_DL_MUX_BUF_SIZE (16 * 1024)
+#define IPC_MEM_MAX_UL_ADB_BUF_SIZE IPC_MEM_MAX_DL_MUX_BUF_SIZE
+
+/* Size of the buffer for the IP MUX Lite data buffer. */
+#define IPC_MEM_MAX_DL_MUX_LITE_BUF_SIZE (2 * 1024)
+
+/* TD counts for IP MUX Lite */
+#define IPC_MEM_MAX_TDS_MUX_LITE_UL 800
+#define IPC_MEM_MAX_TDS_MUX_LITE_DL 1200
+
+/* open session request (AP->CP) */
+#define MUX_CMD_OPEN_SESSION 1
+
+/* response to open session request (CP->AP) */
+#define MUX_CMD_OPEN_SESSION_RESP 2
+
+/* close session request (AP->CP) */
+#define MUX_CMD_CLOSE_SESSION 3
+
+/* response to close session request (CP->AP) */
+#define MUX_CMD_CLOSE_SESSION_RESP 4
+
+/* Flow control command with mask of the flow per queue/flow. */
+#define MUX_LITE_CMD_FLOW_CTL 5
+
+/* ACK the flow control command. Shall have the same Transaction ID as the
+ * matching FLOW_CTL command.
+ */
+#define MUX_LITE_CMD_FLOW_CTL_ACK 6
+
+/* Command for report packet indicating link quality metrics. */
+#define MUX_LITE_CMD_LINK_STATUS_REPORT 7
+
+/* Response to a report packet */
+#define MUX_LITE_CMD_LINK_STATUS_REPORT_RESP 8
+
+/* Used to reset a command/response state. */
+#define MUX_CMD_INVALID 255
+
+/* command response : command processed successfully */
+#define MUX_CMD_RESP_SUCCESS 0
+
+/* MUX for route link devices */
+#define IPC_MEM_WWAN_MUX BIT(0)
+
+/* Initiated actions to change the state of the MUX object. */
+enum mux_event {
+ MUX_E_INACTIVE, /* No initiated actions. */
+ MUX_E_MUX_SESSION_OPEN, /* Create the MUX channel and a session. */
+ MUX_E_MUX_SESSION_CLOSE, /* Release a session. */
+ MUX_E_MUX_CHANNEL_CLOSE, /* Release the MUX channel. */
+ MUX_E_NO_ORDERS, /* No MUX order. */
+ MUX_E_NOT_APPLICABLE, /* Defect IP MUX. */
+};
+
+/* MUX session open command. */
+struct mux_session_open {
+ enum mux_event event;
+ __le32 if_id;
+};
+
+/* MUX session close command. */
+struct mux_session_close {
+ enum mux_event event;
+ __le32 if_id;
+};
+
+/* MUX channel close command. */
+struct mux_channel_close {
+ enum mux_event event;
+};
+
+/* Default message type to find out the right message type. */
+struct mux_common {
+ enum mux_event event;
+};
+
+/* List of ops in MUX mode. */
+union mux_msg {
+ struct mux_session_open session_open;
+ struct mux_session_close session_close;
+ struct mux_channel_close channel_close;
+ struct mux_common common;
+};
+
+/* Parameter definition of the open session command. */
+struct mux_cmd_open_session {
+ u8 flow_ctrl; /* 0: Flow control disabled (flow allowed). */
+ /* 1: Flow control enabled (flow not allowed)*/
+ u8 ipv4v6_hints; /* 0: IPv4/IPv6 hints not supported.*/
+ /* 1: IPv4/IPv6 hints supported*/
+ __le16 reserved2; /* Reserved. Set to zero. */
+ __le32 dl_head_pad_len; /* Maximum length supported */
+ /* for DL head padding on a datagram. */
+};
+
+/* Parameter definition of the open session response. */
+struct mux_cmd_open_session_resp {
+ __le32 response; /* Response code */
+ u8 flow_ctrl; /* 0: Flow control disabled (flow allowed). */
+ /* 1: Flow control enabled (flow not allowed) */
+ u8 ipv4v6_hints; /* 0: IPv4/IPv6 hints not supported */
+ /* 1: IPv4/IPv6 hints supported */
+ __le16 reserved2; /* Reserved. Set to zero. */
+ __le32 ul_head_pad_len; /* Actual length supported for */
+ /* UL head padding on adatagram.*/
+};
+
+/* Parameter definition of the close session response code */
+struct mux_cmd_close_session_resp {
+ __le32 response;
+};
+
+/* Parameter definition of the flow control command. */
+struct mux_cmd_flow_ctl {
+ __le32 mask; /* indicating the desired flow control */
+ /* state for various flows/queues */
+};
+
+/* Parameter definition of the link status report code*/
+struct mux_cmd_link_status_report {
+ u8 payload;
+};
+
+/* Parameter definition of the link status report response code. */
+struct mux_cmd_link_status_report_resp {
+ __le32 response;
+};
+
+/**
+ * union mux_cmd_param - Union-definition of the command parameters.
+ * @open_session: Inband command for open session
+ * @open_session_resp: Inband command for open session response
+ * @close_session_resp: Inband command for close session response
+ * @flow_ctl: In-band flow control on the opened interfaces
+ * @link_status: In-band Link Status Report
+ * @link_status_resp: In-band command for link status report response
+ */
+union mux_cmd_param {
+ struct mux_cmd_open_session open_session;
+ struct mux_cmd_open_session_resp open_session_resp;
+ struct mux_cmd_close_session_resp close_session_resp;
+ struct mux_cmd_flow_ctl flow_ctl;
+ struct mux_cmd_link_status_report link_status;
+ struct mux_cmd_link_status_report_resp link_status_resp;
+};
+
+/* States of the MUX object.. */
+enum mux_state {
+ MUX_S_INACTIVE, /* IP MUX is unused. */
+ MUX_S_ACTIVE, /* IP MUX channel is available. */
+ MUX_S_ERROR, /* Defect IP MUX. */
+};
+
+/* Supported MUX protocols. */
+enum ipc_mux_protocol {
+ MUX_UNKNOWN,
+ MUX_LITE,
+};
+
+/* Supported UL data transfer methods. */
+enum ipc_mux_ul_flow {
+ MUX_UL_UNKNOWN,
+ MUX_UL, /* Normal UL data transfer */
+ MUX_UL_ON_CREDITS, /* UL data transfer will be based on credits */
+};
+
+/* List of the MUX session. */
+struct mux_session {
+ struct iosm_wwan *wwan; /*Network i/f used for communication*/
+ int if_id; /* i/f id for session open message.*/
+ u32 flags;
+ u32 ul_head_pad_len; /* Nr of bytes for UL head padding. */
+ u32 dl_head_pad_len; /* Nr of bytes for DL head padding. */
+ struct sk_buff_head ul_list; /* skb entries for an ADT. */
+ u32 flow_ctl_mask; /* UL flow control */
+ u32 flow_ctl_en_cnt; /* Flow control Enable cmd count */
+ u32 flow_ctl_dis_cnt; /* Flow Control Disable cmd count */
+ int ul_flow_credits; /* UL flow credits */
+ u8 net_tx_stop:1,
+ flush:1; /* flush net interface ? */
+};
+
+/* State of a single UL data block. */
+struct mux_adb {
+ struct sk_buff *dest_skb; /* Current UL skb for the data block. */
+ u8 *buf; /* ADB memory. */
+ struct mux_adgh *adgh; /* ADGH pointer */
+ struct sk_buff *qlth_skb; /* QLTH pointer */
+ u32 *next_table_index; /* Pointer to next table index. */
+ struct sk_buff_head free_list; /* List of alloc. ADB for the UL sess.*/
+ int size; /* Size of the ADB memory. */
+ u32 if_cnt; /* Statistic counter */
+ u32 dg_cnt_total;
+ u32 payload_size;
+};
+
+/* Temporary ACB state. */
+struct mux_acb {
+ struct sk_buff *skb; /* Used UL skb. */
+ int if_id; /* Session id. */
+ u32 wanted_response;
+ u32 got_response;
+ u32 cmd;
+ union mux_cmd_param got_param; /* Received command/response parameter */
+};
+
+/**
+ * struct iosm_mux - Structure of the data multiplexing over an IP channel.
+ * @dev: Pointer to device structure
+ * @session: Array of the MUX sessions.
+ * @channel: Reference to the IP MUX channel
+ * @pcie: Pointer to iosm_pcie struct
+ * @imem: Pointer to iosm_imem
+ * @wwan: Poinetr to iosm_wwan
+ * @ipc_protocol: Pointer to iosm_protocol
+ * @channel_id: Channel ID for MUX
+ * @protocol: Type of the MUX protocol
+ * @ul_flow: UL Flow type
+ * @nr_sessions: Number of sessions
+ * @instance_id: Instance ID
+ * @state: States of the MUX object
+ * @event: Initiated actions to change the state of the MUX object
+ * @tx_transaction_id: Transaction id for the ACB command.
+ * @rr_next_session: Next session number for round robin.
+ * @ul_adb: State of the UL ADB/ADGH.
+ * @size_needed: Variable to store the size needed during ADB preparation
+ * @ul_data_pend_bytes: Pending UL data to be processed in bytes
+ * @acb: Temporary ACB state
+ * @wwan_q_offset: This will hold the offset of the given instance
+ * Useful while passing or receiving packets from
+ * wwan/imem layer.
+ * @initialized: MUX object is initialized
+ * @ev_mux_net_transmit_pending:
+ * 0 means inform the IPC tasklet to pass the
+ * accumulated uplink ADB to CP.
+ * @adb_prep_ongoing: Flag for ADB preparation status
+ */
+struct iosm_mux {
+ struct device *dev;
+ struct mux_session session[IPC_MEM_MUX_IP_SESSION_ENTRIES];
+ struct ipc_mem_channel *channel;
+ struct iosm_pcie *pcie;
+ struct iosm_imem *imem;
+ struct iosm_wwan *wwan;
+ struct iosm_protocol *ipc_protocol;
+ int channel_id;
+ enum ipc_mux_protocol protocol;
+ enum ipc_mux_ul_flow ul_flow;
+ int nr_sessions;
+ int instance_id;
+ enum mux_state state;
+ enum mux_event event;
+ u32 tx_transaction_id;
+ int rr_next_session;
+ struct mux_adb ul_adb;
+ int size_needed;
+ long long ul_data_pend_bytes;
+ struct mux_acb acb;
+ int wwan_q_offset;
+ u8 initialized:1,
+ ev_mux_net_transmit_pending:1,
+ adb_prep_ongoing:1;
+};
+
+/* MUX configuration structure */
+struct ipc_mux_config {
+ enum ipc_mux_protocol protocol;
+ enum ipc_mux_ul_flow ul_flow;
+ int nr_sessions;
+ int instance_id;
+};
+
+/**
+ * ipc_mux_init - Allocates and Init MUX instance
+ * @mux_cfg: Pointer to MUX configuration structure
+ * @ipc_imem: Pointer to imem data-struct
+ *
+ * Returns: Initialized mux pointer on success else NULL
+ */
+struct iosm_mux *ipc_mux_init(struct ipc_mux_config *mux_cfg,
+ struct iosm_imem *ipc_imem);
+
+/**
+ * ipc_mux_deinit - Deallocates MUX instance
+ * @ipc_mux: Pointer to the MUX instance.
+ */
+void ipc_mux_deinit(struct iosm_mux *ipc_mux);
+
+/**
+ * ipc_mux_check_n_restart_tx - Checks for pending UL date bytes and then
+ * it restarts the net interface tx queue if
+ * device has set flow control as off.
+ * @ipc_mux: Pointer to MUX data-struct
+ */
+void ipc_mux_check_n_restart_tx(struct iosm_mux *ipc_mux);
+
+/**
+ * ipc_mux_get_active_protocol - Returns the active MUX protocol type.
+ * @ipc_mux: Pointer to MUX data-struct
+ *
+ * Returns: enum of type ipc_mux_protocol
+ */
+enum ipc_mux_protocol ipc_mux_get_active_protocol(struct iosm_mux *ipc_mux);
+
+/**
+ * ipc_mux_open_session - Opens a MUX session for IP traffic.
+ * @ipc_mux: Pointer to MUX data-struct
+ * @session_nr: Interface ID or session number
+ *
+ * Returns: channel id on success, failure value on error
+ */
+int ipc_mux_open_session(struct iosm_mux *ipc_mux, int session_nr);
+
+/**
+ * ipc_mux_close_session - Closes a MUX session.
+ * @ipc_mux: Pointer to MUX data-struct
+ * @session_nr: Interface ID or session number
+ *
+ * Returns: channel id on success, failure value on error
+ */
+int ipc_mux_close_session(struct iosm_mux *ipc_mux, int session_nr);
+
+/**
+ * ipc_mux_get_max_sessions - Retuns the maximum sessions supported on the
+ * provided MUX instance..
+ * @ipc_mux: Pointer to MUX data-struct
+ *
+ * Returns: Number of sessions supported on Success and failure value on error
+ */
+int ipc_mux_get_max_sessions(struct iosm_mux *ipc_mux);
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include <linux/nospec.h>
+
+#include "iosm_ipc_imem_ops.h"
+#include "iosm_ipc_mux_codec.h"
+#include "iosm_ipc_task_queue.h"
+
+/* Test the link power state and send a MUX command in blocking mode. */
+static int ipc_mux_tq_cmd_send(struct iosm_imem *ipc_imem, int arg, void *msg,
+ size_t size)
+{
+ struct iosm_mux *ipc_mux = ipc_imem->mux;
+ const struct mux_acb *acb = msg;
+
+ skb_queue_tail(&ipc_mux->channel->ul_list, acb->skb);
+ ipc_imem_ul_send(ipc_mux->imem);
+
+ return 0;
+}
+
+static int ipc_mux_acb_send(struct iosm_mux *ipc_mux, bool blocking)
+{
+ struct completion *completion = &ipc_mux->channel->ul_sem;
+ int ret = ipc_task_queue_send_task(ipc_mux->imem, ipc_mux_tq_cmd_send,
+ 0, &ipc_mux->acb,
+ sizeof(ipc_mux->acb), false);
+ if (ret) {
+ dev_err(ipc_mux->dev, "unable to send mux command");
+ return ret;
+ }
+
+ /* if blocking, suspend the app and wait for irq in the flash or
+ * crash phase. return false on timeout to indicate failure.
+ */
+ if (blocking) {
+ u32 wait_time_milliseconds = IPC_MUX_CMD_RUN_DEFAULT_TIMEOUT;
+
+ reinit_completion(completion);
+
+ if (wait_for_completion_interruptible_timeout
+ (completion, msecs_to_jiffies(wait_time_milliseconds)) ==
+ 0) {
+ dev_err(ipc_mux->dev, "ch[%d] timeout",
+ ipc_mux->channel_id);
+ ipc_uevent_send(ipc_mux->imem->dev, UEVENT_MDM_TIMEOUT);
+ return -ETIMEDOUT;
+ }
+ }
+
+ return 0;
+}
+
+/* Prepare mux Command */
+static struct mux_lite_cmdh *ipc_mux_lite_add_cmd(struct iosm_mux *ipc_mux,
+ u32 cmd, struct mux_acb *acb,
+ void *param, u32 param_size)
+{
+ struct mux_lite_cmdh *cmdh = (struct mux_lite_cmdh *)acb->skb->data;
+
+ cmdh->signature = cpu_to_le32(MUX_SIG_CMDH);
+ cmdh->command_type = cpu_to_le32(cmd);
+ cmdh->if_id = acb->if_id;
+
+ acb->cmd = cmd;
+
+ cmdh->cmd_len = cpu_to_le16(offsetof(struct mux_lite_cmdh, param) +
+ param_size);
+ cmdh->transaction_id = cpu_to_le32(ipc_mux->tx_transaction_id++);
+
+ if (param)
+ memcpy(&cmdh->param, param, param_size);
+
+ skb_put(acb->skb, le16_to_cpu(cmdh->cmd_len));
+
+ return cmdh;
+}
+
+static int ipc_mux_acb_alloc(struct iosm_mux *ipc_mux)
+{
+ struct mux_acb *acb = &ipc_mux->acb;
+ struct sk_buff *skb;
+ dma_addr_t mapping;
+
+ /* Allocate skb memory for the uplink buffer. */
+ skb = ipc_pcie_alloc_skb(ipc_mux->pcie, MUX_MAX_UL_ACB_BUF_SIZE,
+ GFP_ATOMIC, &mapping, DMA_TO_DEVICE, 0);
+ if (!skb)
+ return -ENOMEM;
+
+ /* Save the skb address. */
+ acb->skb = skb;
+
+ memset(skb->data, 0, MUX_MAX_UL_ACB_BUF_SIZE);
+
+ return 0;
+}
+
+int ipc_mux_dl_acb_send_cmds(struct iosm_mux *ipc_mux, u32 cmd_type, u8 if_id,
+ u32 transaction_id, union mux_cmd_param *param,
+ size_t res_size, bool blocking, bool respond)
+{
+ struct mux_acb *acb = &ipc_mux->acb;
+ struct mux_lite_cmdh *ack_lite;
+ int ret = 0;
+
+ acb->if_id = if_id;
+ ret = ipc_mux_acb_alloc(ipc_mux);
+ if (ret)
+ return ret;
+
+ ack_lite = ipc_mux_lite_add_cmd(ipc_mux, cmd_type, acb, param,
+ res_size);
+ if (respond)
+ ack_lite->transaction_id = cpu_to_le32(transaction_id);
+
+ ret = ipc_mux_acb_send(ipc_mux, blocking);
+
+ return ret;
+}
+
+void ipc_mux_netif_tx_flowctrl(struct mux_session *session, int idx, bool on)
+{
+ /* Inform the network interface to start/stop flow ctrl */
+ ipc_wwan_tx_flowctrl(session->wwan, idx, on);
+}
+
+static int ipc_mux_dl_cmdresps_decode_process(struct iosm_mux *ipc_mux,
+ struct mux_lite_cmdh *cmdh)
+{
+ struct mux_acb *acb = &ipc_mux->acb;
+
+ switch (le32_to_cpu(cmdh->command_type)) {
+ case MUX_CMD_OPEN_SESSION_RESP:
+ case MUX_CMD_CLOSE_SESSION_RESP:
+ /* Resume the control application. */
+ acb->got_param = cmdh->param;
+ break;
+
+ case MUX_LITE_CMD_FLOW_CTL_ACK:
+ /* This command type is not expected as response for
+ * Aggregation version of the protocol. So return non-zero.
+ */
+ if (ipc_mux->protocol != MUX_LITE)
+ return -EINVAL;
+
+ dev_dbg(ipc_mux->dev, "if %u FLOW_CTL_ACK %u received",
+ cmdh->if_id, le32_to_cpu(cmdh->transaction_id));
+ break;
+
+ default:
+ return -EINVAL;
+ }
+
+ acb->wanted_response = MUX_CMD_INVALID;
+ acb->got_response = le32_to_cpu(cmdh->command_type);
+ complete(&ipc_mux->channel->ul_sem);
+
+ return 0;
+}
+
+static int ipc_mux_dl_dlcmds_decode_process(struct iosm_mux *ipc_mux,
+ struct mux_lite_cmdh *cmdh)
+{
+ union mux_cmd_param *param = &cmdh->param;
+ struct mux_session *session;
+ int new_size;
+
+ dev_dbg(ipc_mux->dev, "if_id[%d]: dlcmds decode process %d",
+ cmdh->if_id, le32_to_cpu(cmdh->command_type));
+
+ switch (le32_to_cpu(cmdh->command_type)) {
+ case MUX_LITE_CMD_FLOW_CTL:
+
+ if (cmdh->if_id >= ipc_mux->nr_sessions) {
+ dev_err(ipc_mux->dev, "if_id [%d] not valid",
+ cmdh->if_id);
+ return -EINVAL; /* No session interface id. */
+ }
+
+ session = &ipc_mux->session[cmdh->if_id];
+
+ new_size = offsetof(struct mux_lite_cmdh, param) +
+ sizeof(param->flow_ctl);
+ if (param->flow_ctl.mask == cpu_to_le32(0xFFFFFFFF)) {
+ /* Backward Compatibility */
+ if (cmdh->cmd_len == cpu_to_le16(new_size))
+ session->flow_ctl_mask =
+ le32_to_cpu(param->flow_ctl.mask);
+ else
+ session->flow_ctl_mask = ~0;
+ /* if CP asks for FLOW CTRL Enable
+ * then set our internal flow control Tx flag
+ * to limit uplink session queueing
+ */
+ session->net_tx_stop = true;
+ /* Update the stats */
+ session->flow_ctl_en_cnt++;
+ } else if (param->flow_ctl.mask == 0) {
+ /* Just reset the Flow control mask and let
+ * mux_flow_ctrl_low_thre_b take control on
+ * our internal Tx flag and enabling kernel
+ * flow control
+ */
+ /* Backward Compatibility */
+ if (cmdh->cmd_len == cpu_to_le16(new_size))
+ session->flow_ctl_mask =
+ le32_to_cpu(param->flow_ctl.mask);
+ else
+ session->flow_ctl_mask = 0;
+ /* Update the stats */
+ session->flow_ctl_dis_cnt++;
+ } else {
+ break;
+ }
+
+ dev_dbg(ipc_mux->dev, "if[%u] FLOW CTRL 0x%08X", cmdh->if_id,
+ le32_to_cpu(param->flow_ctl.mask));
+ break;
+
+ case MUX_LITE_CMD_LINK_STATUS_REPORT:
+ break;
+
+ default:
+ return -EINVAL;
+ }
+ return 0;
+}
+
+/* Decode and Send appropriate response to a command block. */
+static void ipc_mux_dl_cmd_decode(struct iosm_mux *ipc_mux, struct sk_buff *skb)
+{
+ struct mux_lite_cmdh *cmdh = (struct mux_lite_cmdh *)skb->data;
+ __le32 trans_id = cmdh->transaction_id;
+
+ if (ipc_mux_dl_cmdresps_decode_process(ipc_mux, cmdh)) {
+ /* Unable to decode command response indicates the cmd_type
+ * may be a command instead of response. So try to decoding it.
+ */
+ if (!ipc_mux_dl_dlcmds_decode_process(ipc_mux, cmdh)) {
+ /* Decoded command may need a response. Give the
+ * response according to the command type.
+ */
+ union mux_cmd_param *mux_cmd = NULL;
+ size_t size = 0;
+ u32 cmd = MUX_LITE_CMD_LINK_STATUS_REPORT_RESP;
+
+ if (cmdh->command_type ==
+ cpu_to_le32(MUX_LITE_CMD_LINK_STATUS_REPORT)) {
+ mux_cmd = &cmdh->param;
+ mux_cmd->link_status_resp.response =
+ cpu_to_le32(MUX_CMD_RESP_SUCCESS);
+ /* response field is u32 */
+ size = sizeof(u32);
+ } else if (cmdh->command_type ==
+ cpu_to_le32(MUX_LITE_CMD_FLOW_CTL)) {
+ cmd = MUX_LITE_CMD_FLOW_CTL_ACK;
+ } else {
+ return;
+ }
+
+ if (ipc_mux_dl_acb_send_cmds(ipc_mux, cmd, cmdh->if_id,
+ le32_to_cpu(trans_id),
+ mux_cmd, size, false,
+ true))
+ dev_err(ipc_mux->dev,
+ "if_id %d: cmd send failed",
+ cmdh->if_id);
+ }
+ }
+}
+
+/* Pass the DL packet to the netif layer. */
+static int ipc_mux_net_receive(struct iosm_mux *ipc_mux, int if_id,
+ struct iosm_wwan *wwan, u32 offset,
+ u8 service_class, struct sk_buff *skb)
+{
+ struct sk_buff *dest_skb = skb_clone(skb, GFP_ATOMIC);
+
+ if (!dest_skb)
+ return -ENOMEM;
+
+ skb_pull(dest_skb, offset);
+ skb_set_tail_pointer(dest_skb, dest_skb->len);
+ /* Pass the packet to the netif layer. */
+ dest_skb->priority = service_class;
+
+ return ipc_wwan_receive(wwan, dest_skb, false, if_id + 1);
+}
+
+/* Decode Flow Credit Table in the block */
+static void ipc_mux_dl_fcth_decode(struct iosm_mux *ipc_mux,
+ unsigned char *block)
+{
+ struct ipc_mem_lite_gen_tbl *fct = (struct ipc_mem_lite_gen_tbl *)block;
+ struct iosm_wwan *wwan;
+ int ul_credits;
+ int if_id;
+
+ if (fct->vfl_length != sizeof(fct->vfl.nr_of_bytes)) {
+ dev_err(ipc_mux->dev, "unexpected FCT length: %d",
+ fct->vfl_length);
+ return;
+ }
+
+ if_id = fct->if_id;
+ if (if_id >= ipc_mux->nr_sessions) {
+ dev_err(ipc_mux->dev, "not supported if_id: %d", if_id);
+ return;
+ }
+
+ /* Is the session active ? */
+ if_id = array_index_nospec(if_id, ipc_mux->nr_sessions);
+ wwan = ipc_mux->session[if_id].wwan;
+ if (!wwan) {
+ dev_err(ipc_mux->dev, "session Net ID is NULL");
+ return;
+ }
+
+ ul_credits = fct->vfl.nr_of_bytes;
+
+ dev_dbg(ipc_mux->dev, "Flow_Credit:: if_id[%d] Old: %d Grants: %d",
+ if_id, ipc_mux->session[if_id].ul_flow_credits, ul_credits);
+
+ /* Update the Flow Credit information from ADB */
+ ipc_mux->session[if_id].ul_flow_credits += ul_credits;
+
+ /* Check whether the TX can be started */
+ if (ipc_mux->session[if_id].ul_flow_credits > 0) {
+ ipc_mux->session[if_id].net_tx_stop = false;
+ ipc_mux_netif_tx_flowctrl(&ipc_mux->session[if_id],
+ ipc_mux->session[if_id].if_id, false);
+ }
+}
+
+/* Decode non-aggregated datagram */
+static void ipc_mux_dl_adgh_decode(struct iosm_mux *ipc_mux,
+ struct sk_buff *skb)
+{
+ u32 pad_len, packet_offset;
+ struct iosm_wwan *wwan;
+ struct mux_adgh *adgh;
+ u8 *block = skb->data;
+ int rc = 0;
+ u8 if_id;
+
+ adgh = (struct mux_adgh *)block;
+
+ if (adgh->signature != cpu_to_le32(MUX_SIG_ADGH)) {
+ dev_err(ipc_mux->dev, "invalid ADGH signature received");
+ return;
+ }
+
+ if_id = adgh->if_id;
+ if (if_id >= ipc_mux->nr_sessions) {
+ dev_err(ipc_mux->dev, "invalid if_id while decoding %d", if_id);
+ return;
+ }
+
+ /* Is the session active ? */
+ if_id = array_index_nospec(if_id, ipc_mux->nr_sessions);
+ wwan = ipc_mux->session[if_id].wwan;
+ if (!wwan) {
+ dev_err(ipc_mux->dev, "session Net ID is NULL");
+ return;
+ }
+
+ /* Store the pad len for the corresponding session
+ * Pad bytes as negotiated in the open session less the header size
+ * (see session management chapter for details).
+ * If resulting padding is zero or less, the additional head padding is
+ * omitted. For e.g., if HEAD_PAD_LEN = 16 or less, this field is
+ * omitted if HEAD_PAD_LEN = 20, then this field will have 4 bytes
+ * set to zero
+ */
+ pad_len =
+ ipc_mux->session[if_id].dl_head_pad_len - IPC_MEM_DL_ETH_OFFSET;
+ packet_offset = sizeof(*adgh) + pad_len;
+
+ if_id += ipc_mux->wwan_q_offset;
+
+ /* Pass the packet to the netif layer */
+ rc = ipc_mux_net_receive(ipc_mux, if_id, wwan, packet_offset,
+ adgh->service_class, skb);
+ if (rc) {
+ dev_err(ipc_mux->dev, "mux adgh decoding error");
+ return;
+ }
+ ipc_mux->session[if_id].flush = 1;
+}
+
+void ipc_mux_dl_decode(struct iosm_mux *ipc_mux, struct sk_buff *skb)
+{
+ u32 signature;
+
+ if (!skb->data)
+ return;
+
+ /* Decode the MUX header type. */
+ signature = le32_to_cpup((__le32 *)skb->data);
+
+ switch (signature) {
+ case MUX_SIG_ADGH:
+ ipc_mux_dl_adgh_decode(ipc_mux, skb);
+ break;
+
+ case MUX_SIG_FCTH:
+ ipc_mux_dl_fcth_decode(ipc_mux, skb->data);
+ break;
+
+ case MUX_SIG_CMDH:
+ ipc_mux_dl_cmd_decode(ipc_mux, skb);
+ break;
+
+ default:
+ dev_err(ipc_mux->dev, "invalid ABH signature");
+ }
+
+ ipc_pcie_kfree_skb(ipc_mux->pcie, skb);
+}
+
+static int ipc_mux_ul_skb_alloc(struct iosm_mux *ipc_mux,
+ struct mux_adb *ul_adb, u32 type)
+{
+ /* Take the first element of the free list. */
+ struct sk_buff *skb = skb_dequeue(&ul_adb->free_list);
+ int qlt_size;
+
+ if (!skb)
+ return -EBUSY; /* Wait for a free ADB skb. */
+
+ /* Mark it as UL ADB to select the right free operation. */
+ IPC_CB(skb)->op_type = (u8)UL_MUX_OP_ADB;
+
+ switch (type) {
+ case MUX_SIG_ADGH:
+ /* Save the ADB memory settings. */
+ ul_adb->dest_skb = skb;
+ ul_adb->buf = skb->data;
+ ul_adb->size = IPC_MEM_MAX_DL_MUX_LITE_BUF_SIZE;
+ /* reset statistic counter */
+ ul_adb->if_cnt = 0;
+ ul_adb->payload_size = 0;
+ ul_adb->dg_cnt_total = 0;
+
+ ul_adb->adgh = (struct mux_adgh *)skb->data;
+ memset(ul_adb->adgh, 0, sizeof(struct mux_adgh));
+ break;
+
+ case MUX_SIG_QLTH:
+ qlt_size = offsetof(struct ipc_mem_lite_gen_tbl, vfl) +
+ (MUX_QUEUE_LEVEL * sizeof(struct mux_lite_vfl));
+
+ if (qlt_size > IPC_MEM_MAX_DL_MUX_LITE_BUF_SIZE) {
+ dev_err(ipc_mux->dev,
+ "can't support. QLT size:%d SKB size: %d",
+ qlt_size, IPC_MEM_MAX_DL_MUX_LITE_BUF_SIZE);
+ return -ERANGE;
+ }
+
+ ul_adb->qlth_skb = skb;
+ memset((ul_adb->qlth_skb)->data, 0, qlt_size);
+ skb_put(skb, qlt_size);
+ break;
+ }
+
+ return 0;
+}
+
+static void ipc_mux_ul_adgh_finish(struct iosm_mux *ipc_mux)
+{
+ struct mux_adb *ul_adb = &ipc_mux->ul_adb;
+ u16 adgh_len;
+ long long bytes;
+ char *str;
+
+ if (!ul_adb->dest_skb) {
+ dev_err(ipc_mux->dev, "no dest skb");
+ return;
+ }
+
+ adgh_len = le16_to_cpu(ul_adb->adgh->length);
+ skb_put(ul_adb->dest_skb, adgh_len);
+ skb_queue_tail(&ipc_mux->channel->ul_list, ul_adb->dest_skb);
+ ul_adb->dest_skb = NULL;
+
+ if (ipc_mux->ul_flow == MUX_UL_ON_CREDITS) {
+ struct mux_session *session;
+
+ session = &ipc_mux->session[ul_adb->adgh->if_id];
+ str = "available_credits";
+ bytes = (long long)session->ul_flow_credits;
+
+ } else {
+ str = "pend_bytes";
+ bytes = ipc_mux->ul_data_pend_bytes;
+ ipc_mux->ul_data_pend_bytes = ipc_mux->ul_data_pend_bytes +
+ adgh_len;
+ }
+
+ dev_dbg(ipc_mux->dev, "UL ADGH: size=%u, if_id=%d, payload=%d, %s=%lld",
+ adgh_len, ul_adb->adgh->if_id, ul_adb->payload_size,
+ str, bytes);
+}
+
+/* Allocates an ADB from the free list and initializes it with ADBH */
+static bool ipc_mux_ul_adb_allocate(struct iosm_mux *ipc_mux,
+ struct mux_adb *adb, int *size_needed,
+ u32 type)
+{
+ bool ret_val = false;
+ int status;
+
+ if (!adb->dest_skb) {
+ /* Allocate memory for the ADB including of the
+ * datagram table header.
+ */
+ status = ipc_mux_ul_skb_alloc(ipc_mux, adb, type);
+ if (status)
+ /* Is a pending ADB available ? */
+ ret_val = true; /* None. */
+
+ /* Update size need to zero only for new ADB memory */
+ *size_needed = 0;
+ }
+
+ return ret_val;
+}
+
+/* Informs the network stack to stop sending further packets for all opened
+ * sessions
+ */
+static void ipc_mux_stop_tx_for_all_sessions(struct iosm_mux *ipc_mux)
+{
+ struct mux_session *session;
+ int idx;
+
+ for (idx = 0; idx < ipc_mux->nr_sessions; idx++) {
+ session = &ipc_mux->session[idx];
+
+ if (!session->wwan)
+ continue;
+
+ session->net_tx_stop = true;
+ }
+}
+
+/* Sends Queue Level Table of all opened sessions */
+static bool ipc_mux_lite_send_qlt(struct iosm_mux *ipc_mux)
+{
+ struct ipc_mem_lite_gen_tbl *qlt;
+ struct mux_session *session;
+ bool qlt_updated = false;
+ int i;
+ int qlt_size;
+
+ if (!ipc_mux->initialized || ipc_mux->state != MUX_S_ACTIVE)
+ return qlt_updated;
+
+ qlt_size = offsetof(struct ipc_mem_lite_gen_tbl, vfl) +
+ MUX_QUEUE_LEVEL * sizeof(struct mux_lite_vfl);
+
+ for (i = 0; i < ipc_mux->nr_sessions; i++) {
+ session = &ipc_mux->session[i];
+
+ if (!session->wwan || session->flow_ctl_mask)
+ continue;
+
+ if (ipc_mux_ul_skb_alloc(ipc_mux, &ipc_mux->ul_adb,
+ MUX_SIG_QLTH)) {
+ dev_err(ipc_mux->dev,
+ "no reserved mem to send QLT of if_id: %d", i);
+ break;
+ }
+
+ /* Prepare QLT */
+ qlt = (struct ipc_mem_lite_gen_tbl *)(ipc_mux->ul_adb.qlth_skb)
+ ->data;
+ qlt->signature = cpu_to_le32(MUX_SIG_QLTH);
+ qlt->length = cpu_to_le16(qlt_size);
+ qlt->if_id = i;
+ qlt->vfl_length = MUX_QUEUE_LEVEL * sizeof(struct mux_lite_vfl);
+ qlt->reserved[0] = 0;
+ qlt->reserved[1] = 0;
+
+ qlt->vfl.nr_of_bytes = session->ul_list.qlen;
+
+ /* Add QLT to the transfer list. */
+ skb_queue_tail(&ipc_mux->channel->ul_list,
+ ipc_mux->ul_adb.qlth_skb);
+
+ qlt_updated = true;
+ ipc_mux->ul_adb.qlth_skb = NULL;
+ }
+
+ if (qlt_updated)
+ /* Updates the TDs with ul_list */
+ (void)ipc_imem_ul_write_td(ipc_mux->imem);
+
+ return qlt_updated;
+}
+
+/* Checks the available credits for the specified session and returns
+ * number of packets for which credits are available.
+ */
+static int ipc_mux_ul_bytes_credits_check(struct iosm_mux *ipc_mux,
+ struct mux_session *session,
+ struct sk_buff_head *ul_list,
+ int max_nr_of_pkts)
+{
+ int pkts_to_send = 0;
+ struct sk_buff *skb;
+ int credits = 0;
+
+ if (ipc_mux->ul_flow == MUX_UL_ON_CREDITS) {
+ credits = session->ul_flow_credits;
+ if (credits <= 0) {
+ dev_dbg(ipc_mux->dev,
+ "FC::if_id[%d] Insuff.Credits/Qlen:%d/%u",
+ session->if_id, session->ul_flow_credits,
+ session->ul_list.qlen); /* nr_of_bytes */
+ return 0;
+ }
+ } else {
+ credits = IPC_MEM_MUX_UL_FLOWCTRL_HIGH_B -
+ ipc_mux->ul_data_pend_bytes;
+ if (credits <= 0) {
+ ipc_mux_stop_tx_for_all_sessions(ipc_mux);
+
+ dev_dbg(ipc_mux->dev,
+ "if_id[%d] encod. fail Bytes: %llu, thresh: %d",
+ session->if_id, ipc_mux->ul_data_pend_bytes,
+ IPC_MEM_MUX_UL_FLOWCTRL_HIGH_B);
+ return 0;
+ }
+ }
+
+ /* Check if there are enough credits/bytes available to send the
+ * requested max_nr_of_pkts. Otherwise restrict the nr_of_pkts
+ * depending on available credits.
+ */
+ skb_queue_walk(ul_list, skb)
+ {
+ if (!(credits >= skb->len && pkts_to_send < max_nr_of_pkts))
+ break;
+ credits -= skb->len;
+ pkts_to_send++;
+ }
+
+ return pkts_to_send;
+}
+
+/* Encode the UL IP packet according to Lite spec. */
+static int ipc_mux_ul_adgh_encode(struct iosm_mux *ipc_mux, int session_id,
+ struct mux_session *session,
+ struct sk_buff_head *ul_list,
+ struct mux_adb *adb, int nr_of_pkts)
+{
+ int offset = sizeof(struct mux_adgh);
+ int adb_updated = -EINVAL;
+ struct sk_buff *src_skb;
+ int aligned_size = 0;
+ int nr_of_skb = 0;
+ u32 pad_len = 0;
+
+ /* Re-calculate the number of packets depending on number of bytes to be
+ * processed/available credits.
+ */
+ nr_of_pkts = ipc_mux_ul_bytes_credits_check(ipc_mux, session, ul_list,
+ nr_of_pkts);
+
+ /* If calculated nr_of_pkts from available credits is <= 0
+ * then nothing to do.
+ */
+ if (nr_of_pkts <= 0)
+ return 0;
+
+ /* Read configured UL head_pad_length for session.*/
+ if (session->ul_head_pad_len > IPC_MEM_DL_ETH_OFFSET)
+ pad_len = session->ul_head_pad_len - IPC_MEM_DL_ETH_OFFSET;
+
+ /* Process all pending UL packets for this session
+ * depending on the allocated datagram table size.
+ */
+ while (nr_of_pkts > 0) {
+ /* get destination skb allocated */
+ if (ipc_mux_ul_adb_allocate(ipc_mux, adb, &ipc_mux->size_needed,
+ MUX_SIG_ADGH)) {
+ dev_err(ipc_mux->dev, "no reserved memory for ADGH");
+ return -ENOMEM;
+ }
+
+ /* Peek at the head of the list. */
+ src_skb = skb_peek(ul_list);
+ if (!src_skb) {
+ dev_err(ipc_mux->dev,
+ "skb peek return NULL with count : %d",
+ nr_of_pkts);
+ break;
+ }
+
+ /* Calculate the memory value. */
+ aligned_size = ALIGN((pad_len + src_skb->len), 4);
+
+ ipc_mux->size_needed = sizeof(struct mux_adgh) + aligned_size;
+
+ if (ipc_mux->size_needed > adb->size) {
+ dev_dbg(ipc_mux->dev, "size needed %d, adgh size %d",
+ ipc_mux->size_needed, adb->size);
+ /* Return 1 if any IP packet is added to the transfer
+ * list.
+ */
+ return nr_of_skb ? 1 : 0;
+ }
+
+ /* Add buffer (without head padding to next pending transfer) */
+ memcpy(adb->buf + offset + pad_len, src_skb->data,
+ src_skb->len);
+
+ adb->adgh->signature = cpu_to_le32(MUX_SIG_ADGH);
+ adb->adgh->if_id = session_id;
+ adb->adgh->length =
+ cpu_to_le16(sizeof(struct mux_adgh) + pad_len +
+ src_skb->len);
+ adb->adgh->service_class = src_skb->priority;
+ adb->adgh->next_count = --nr_of_pkts;
+ adb->dg_cnt_total++;
+ adb->payload_size += src_skb->len;
+
+ if (ipc_mux->ul_flow == MUX_UL_ON_CREDITS)
+ /* Decrement the credit value as we are processing the
+ * datagram from the UL list.
+ */
+ session->ul_flow_credits -= src_skb->len;
+
+ /* Remove the processed elements and free it. */
+ src_skb = skb_dequeue(ul_list);
+ dev_kfree_skb(src_skb);
+ nr_of_skb++;
+
+ ipc_mux_ul_adgh_finish(ipc_mux);
+ }
+
+ if (nr_of_skb) {
+ /* Send QLT info to modem if pending bytes > high watermark
+ * in case of mux lite
+ */
+ if (ipc_mux->ul_flow == MUX_UL_ON_CREDITS ||
+ ipc_mux->ul_data_pend_bytes >=
+ IPC_MEM_MUX_UL_FLOWCTRL_LOW_B)
+ adb_updated = ipc_mux_lite_send_qlt(ipc_mux);
+ else
+ adb_updated = 1;
+
+ /* Updates the TDs with ul_list */
+ (void)ipc_imem_ul_write_td(ipc_mux->imem);
+ }
+
+ return adb_updated;
+}
+
+bool ipc_mux_ul_data_encode(struct iosm_mux *ipc_mux)
+{
+ struct sk_buff_head *ul_list;
+ struct mux_session *session;
+ int updated = 0;
+ int session_id;
+ int dg_n;
+ int i;
+
+ if (!ipc_mux || ipc_mux->state != MUX_S_ACTIVE ||
+ ipc_mux->adb_prep_ongoing)
+ return false;
+
+ ipc_mux->adb_prep_ongoing = true;
+
+ for (i = 0; i < ipc_mux->nr_sessions; i++) {
+ session_id = ipc_mux->rr_next_session;
+ session = &ipc_mux->session[session_id];
+
+ /* Go to next handle rr_next_session overflow */
+ ipc_mux->rr_next_session++;
+ if (ipc_mux->rr_next_session >= ipc_mux->nr_sessions)
+ ipc_mux->rr_next_session = 0;
+
+ if (!session->wwan || session->flow_ctl_mask ||
+ session->net_tx_stop)
+ continue;
+
+ ul_list = &session->ul_list;
+
+ /* Is something pending in UL and flow ctrl off */
+ dg_n = skb_queue_len(ul_list);
+ if (dg_n > MUX_MAX_UL_DG_ENTRIES)
+ dg_n = MUX_MAX_UL_DG_ENTRIES;
+
+ if (dg_n == 0)
+ /* Nothing to do for ipc_mux session
+ * -> try next session id.
+ */
+ continue;
+
+ updated = ipc_mux_ul_adgh_encode(ipc_mux, session_id, session,
+ ul_list, &ipc_mux->ul_adb,
+ dg_n);
+ }
+
+ ipc_mux->adb_prep_ongoing = false;
+ return updated == 1;
+}
+
+void ipc_mux_ul_encoded_process(struct iosm_mux *ipc_mux, struct sk_buff *skb)
+{
+ struct mux_adgh *adgh;
+ u16 adgh_len;
+
+ adgh = (struct mux_adgh *)skb->data;
+ adgh_len = le16_to_cpu(adgh->length);
+
+ if (adgh->signature == cpu_to_le32(MUX_SIG_ADGH) &&
+ ipc_mux->ul_flow == MUX_UL)
+ ipc_mux->ul_data_pend_bytes = ipc_mux->ul_data_pend_bytes -
+ adgh_len;
+
+ if (ipc_mux->ul_flow == MUX_UL)
+ dev_dbg(ipc_mux->dev, "ul_data_pend_bytes: %lld",
+ ipc_mux->ul_data_pend_bytes);
+
+ /* Reset the skb settings. */
+ skb->tail = 0;
+ skb->len = 0;
+
+ /* Add the consumed ADB to the free list. */
+ skb_queue_tail((&ipc_mux->ul_adb.free_list), skb);
+}
+
+/* Start the NETIF uplink send transfer in MUX mode. */
+static int ipc_mux_tq_ul_trigger_encode(struct iosm_imem *ipc_imem, int arg,
+ void *msg, size_t size)
+{
+ struct iosm_mux *ipc_mux = ipc_imem->mux;
+ bool ul_data_pend = false;
+
+ /* Add session UL data to a ADB and ADGH */
+ ul_data_pend = ipc_mux_ul_data_encode(ipc_mux);
+ if (ul_data_pend)
+ /* Delay the doorbell irq */
+ ipc_imem_td_update_timer_start(ipc_mux->imem);
+
+ /* reset the debounce flag */
+ ipc_mux->ev_mux_net_transmit_pending = false;
+
+ return 0;
+}
+
+int ipc_mux_ul_trigger_encode(struct iosm_mux *ipc_mux, int if_id,
+ struct sk_buff *skb)
+{
+ struct mux_session *session = &ipc_mux->session[if_id];
+ int ret = -EINVAL;
+
+ if (ipc_mux->channel &&
+ ipc_mux->channel->state != IMEM_CHANNEL_ACTIVE) {
+ dev_err(ipc_mux->dev,
+ "channel state is not IMEM_CHANNEL_ACTIVE");
+ goto out;
+ }
+
+ if (!session->wwan) {
+ dev_err(ipc_mux->dev, "session net ID is NULL");
+ ret = -EFAULT;
+ goto out;
+ }
+
+ /* Session is under flow control.
+ * Check if packet can be queued in session list, if not
+ * suspend net tx
+ */
+ if (skb_queue_len(&session->ul_list) >=
+ (session->net_tx_stop ?
+ IPC_MEM_MUX_UL_SESS_FCON_THRESHOLD :
+ (IPC_MEM_MUX_UL_SESS_FCON_THRESHOLD *
+ IPC_MEM_MUX_UL_SESS_FCOFF_THRESHOLD_FACTOR))) {
+ ipc_mux_netif_tx_flowctrl(session, session->if_id, true);
+ ret = -EBUSY;
+ goto out;
+ }
+
+ /* Add skb to the uplink skb accumulator. */
+ skb_queue_tail(&session->ul_list, skb);
+
+ /* Inform the IPC kthread to pass uplink IP packets to CP. */
+ if (!ipc_mux->ev_mux_net_transmit_pending) {
+ ipc_mux->ev_mux_net_transmit_pending = true;
+ ret = ipc_task_queue_send_task(ipc_mux->imem,
+ ipc_mux_tq_ul_trigger_encode, 0,
+ NULL, 0, false);
+ if (ret)
+ goto out;
+ }
+ dev_dbg(ipc_mux->dev, "mux ul if[%d] qlen=%d/%u, len=%d/%d, prio=%d",
+ if_id, skb_queue_len(&session->ul_list), session->ul_list.qlen,
+ skb->len, skb->truesize, skb->priority);
+ ret = 0;
+out:
+ return ret;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#ifndef IOSM_IPC_MUX_CODEC_H
+#define IOSM_IPC_MUX_CODEC_H
+
+#include "iosm_ipc_mux.h"
+
+/* Queue level size and reporting
+ * >1 is enable, 0 is disable
+ */
+#define MUX_QUEUE_LEVEL 1
+
+/* Size of the buffer for the IP MUX commands. */
+#define MUX_MAX_UL_ACB_BUF_SIZE 256
+
+/* Maximum number of packets in a go per session */
+#define MUX_MAX_UL_DG_ENTRIES 100
+
+/* ADGH: Signature of the Datagram Header. */
+#define MUX_SIG_ADGH 0x48474441
+
+/* CMDH: Signature of the Command Header. */
+#define MUX_SIG_CMDH 0x48444D43
+
+/* QLTH: Signature of the Queue Level Table */
+#define MUX_SIG_QLTH 0x48544C51
+
+/* FCTH: Signature of the Flow Credit Table */
+#define MUX_SIG_FCTH 0x48544346
+
+/* MUX UL session threshold factor */
+#define IPC_MEM_MUX_UL_SESS_FCOFF_THRESHOLD_FACTOR (4)
+
+/* Size of the buffer for the IP MUX Lite data buffer. */
+#define IPC_MEM_MAX_DL_MUX_LITE_BUF_SIZE (2 * 1024)
+
+/* MUX UL session threshold in number of packets */
+#define IPC_MEM_MUX_UL_SESS_FCON_THRESHOLD (64)
+
+/* Default time out for sending IPC session commands like
+ * open session, close session etc
+ * unit : milliseconds
+ */
+#define IPC_MUX_CMD_RUN_DEFAULT_TIMEOUT 1000 /* 1 second */
+
+/* MUX UL flow control lower threshold in bytes */
+#define IPC_MEM_MUX_UL_FLOWCTRL_LOW_B 10240 /* 10KB */
+
+/* MUX UL flow control higher threshold in bytes (5ms worth of data)*/
+#define IPC_MEM_MUX_UL_FLOWCTRL_HIGH_B (110 * 1024)
+
+/**
+ * struct mux_adgh - Aggregated Datagram Header.
+ * @signature: Signature of the Aggregated Datagram Header(0x48474441)
+ * @length: Length (in bytes) of the datagram header. This length
+ * shall include the header size. Min value: 0x10
+ * @if_id: ID of the interface the datagrams belong to
+ * @opt_ipv4v6: Indicates IPv4(=0)/IPv6(=1), It is optional if not
+ * used set it to zero.
+ * @reserved: Reserved bits. Set to zero.
+ * @service_class: Service class identifier for the datagram.
+ * @next_count: Count of the datagrams that shall be following this
+ * datagrams for this interface. A count of zero means
+ * the next datagram may not belong to this interface.
+ * @reserved1: Reserved bytes, Set to zero
+ */
+struct mux_adgh {
+ __le32 signature;
+ __le16 length;
+ u8 if_id;
+ u8 opt_ipv4v6;
+ u8 service_class;
+ u8 next_count;
+ u8 reserved1[6];
+};
+
+/**
+ * struct mux_lite_cmdh - MUX Lite Command Header
+ * @signature: Signature of the Command Header(0x48444D43)
+ * @cmd_len: Length (in bytes) of the command. This length shall
+ * include the header size. Minimum value: 0x10
+ * @if_id: ID of the interface the commands in the table belong to.
+ * @reserved: Reserved Set to zero.
+ * @command_type: Command Enum.
+ * @transaction_id: 4 byte value shall be generated and sent along with a
+ * command Responses and ACKs shall have the same
+ * Transaction ID as their commands. It shall be unique to
+ * the command transaction on the given interface.
+ * @param: Optional parameters used with the command.
+ */
+struct mux_lite_cmdh {
+ __le32 signature;
+ __le16 cmd_len;
+ u8 if_id;
+ u8 reserved;
+ __le32 command_type;
+ __le32 transaction_id;
+ union mux_cmd_param param;
+};
+
+/**
+ * struct mux_lite_vfl - value field in generic table
+ * @nr_of_bytes: Number of bytes available to transmit in the queue.
+ */
+struct mux_lite_vfl {
+ u32 nr_of_bytes;
+};
+
+/**
+ * struct ipc_mem_lite_gen_tbl - Generic table format for Queue Level
+ * and Flow Credit
+ * @signature: Signature of the table
+ * @length: Length of the table
+ * @if_id: ID of the interface the table belongs to
+ * @vfl_length: Value field length
+ * @reserved: Reserved
+ * @vfl: Value field of variable length
+ */
+struct ipc_mem_lite_gen_tbl {
+ __le32 signature;
+ __le16 length;
+ u8 if_id;
+ u8 vfl_length;
+ u32 reserved[2];
+ struct mux_lite_vfl vfl;
+};
+
+/**
+ * ipc_mux_dl_decode -Route the DL packet through the IP MUX layer
+ * depending on Header.
+ * @ipc_mux: Pointer to MUX data-struct
+ * @skb: Pointer to ipc_skb.
+ */
+void ipc_mux_dl_decode(struct iosm_mux *ipc_mux, struct sk_buff *skb);
+
+/**
+ * ipc_mux_dl_acb_send_cmds - Respond to the Command blocks.
+ * @ipc_mux: Pointer to MUX data-struct
+ * @cmd_type: Command
+ * @if_id: Session interface id.
+ * @transaction_id: Command transaction id.
+ * @param: Pointer to command params.
+ * @res_size: Response size
+ * @blocking: True for blocking send
+ * @respond: If true return transaction ID
+ *
+ * Returns: 0 in success and failure value on error
+ */
+int ipc_mux_dl_acb_send_cmds(struct iosm_mux *ipc_mux, u32 cmd_type, u8 if_id,
+ u32 transaction_id, union mux_cmd_param *param,
+ size_t res_size, bool blocking, bool respond);
+
+/**
+ * ipc_mux_netif_tx_flowctrl - Enable/Disable TX flow control on MUX sessions.
+ * @session: Pointer to mux_session struct
+ * @idx: Session ID
+ * @on: true for Enable and false for disable flow control
+ */
+void ipc_mux_netif_tx_flowctrl(struct mux_session *session, int idx, bool on);
+
+/**
+ * ipc_mux_ul_trigger_encode - Route the UL packet through the IP MUX layer
+ * for encoding.
+ * @ipc_mux: Pointer to MUX data-struct
+ * @if_id: Session ID.
+ * @skb: Pointer to ipc_skb.
+ *
+ * Returns: 0 if successfully encoded
+ * failure value on error
+ * -EBUSY if packet has to be retransmitted.
+ */
+int ipc_mux_ul_trigger_encode(struct iosm_mux *ipc_mux, int if_id,
+ struct sk_buff *skb);
+/**
+ * ipc_mux_ul_data_encode - UL encode function for calling from Tasklet context.
+ * @ipc_mux: Pointer to MUX data-struct
+ *
+ * Returns: TRUE if any packet of any session is encoded FALSE otherwise.
+ */
+bool ipc_mux_ul_data_encode(struct iosm_mux *ipc_mux);
+
+/**
+ * ipc_mux_ul_encoded_process - Handles the Modem processed UL data by adding
+ * the SKB to the UL free list.
+ * @ipc_mux: Pointer to MUX data-struct
+ * @skb: Pointer to ipc_skb.
+ */
+void ipc_mux_ul_encoded_process(struct iosm_mux *ipc_mux, struct sk_buff *skb);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include <linux/acpi.h>
+#include <linux/bitfield.h>
+#include <linux/module.h>
+#include <net/rtnetlink.h>
+
+#include "iosm_ipc_imem.h"
+#include "iosm_ipc_pcie.h"
+#include "iosm_ipc_protocol.h"
+
+MODULE_DESCRIPTION("IOSM Driver");
+MODULE_LICENSE("GPL v2");
+
+/* WWAN GUID */
+static guid_t wwan_acpi_guid = GUID_INIT(0xbad01b75, 0x22a8, 0x4f48, 0x87, 0x92,
+ 0xbd, 0xde, 0x94, 0x67, 0x74, 0x7d);
+
+static void ipc_pcie_resources_release(struct iosm_pcie *ipc_pcie)
+{
+ /* Free the MSI resources. */
+ ipc_release_irq(ipc_pcie);
+
+ /* Free mapped doorbell scratchpad bus memory into CPU space. */
+ iounmap(ipc_pcie->scratchpad);
+
+ /* Free mapped IPC_REGS bus memory into CPU space. */
+ iounmap(ipc_pcie->ipc_regs);
+
+ /* Releases all PCI I/O and memory resources previously reserved by a
+ * successful call to pci_request_regions. Call this function only
+ * after all use of the PCI regions has ceased.
+ */
+ pci_release_regions(ipc_pcie->pci);
+}
+
+static void ipc_pcie_cleanup(struct iosm_pcie *ipc_pcie)
+{
+ /* Free the shared memory resources. */
+ ipc_imem_cleanup(ipc_pcie->imem);
+
+ ipc_pcie_resources_release(ipc_pcie);
+
+ /* Signal to the system that the PCI device is not in use. */
+ pci_disable_device(ipc_pcie->pci);
+}
+
+static void ipc_pcie_deinit(struct iosm_pcie *ipc_pcie)
+{
+ kfree(ipc_pcie->imem);
+ kfree(ipc_pcie);
+}
+
+static void ipc_pcie_remove(struct pci_dev *pci)
+{
+ struct iosm_pcie *ipc_pcie = pci_get_drvdata(pci);
+
+ ipc_pcie_cleanup(ipc_pcie);
+
+ ipc_pcie_deinit(ipc_pcie);
+}
+
+static int ipc_pcie_resources_request(struct iosm_pcie *ipc_pcie)
+{
+ struct pci_dev *pci = ipc_pcie->pci;
+ u32 cap = 0;
+ u32 ret;
+
+ /* Reserved PCI I/O and memory resources.
+ * Mark all PCI regions associated with PCI device pci as
+ * being reserved by owner IOSM_IPC.
+ */
+ ret = pci_request_regions(pci, "IOSM_IPC");
+ if (ret) {
+ dev_err(ipc_pcie->dev, "failed pci request regions");
+ goto pci_request_region_fail;
+ }
+
+ /* Reserve the doorbell IPC REGS memory resources.
+ * Remap the memory into CPU space. Arrange for the physical address
+ * (BAR) to be visible from this driver.
+ * pci_ioremap_bar() ensures that the memory is marked uncachable.
+ */
+ ipc_pcie->ipc_regs = pci_ioremap_bar(pci, ipc_pcie->ipc_regs_bar_nr);
+
+ if (!ipc_pcie->ipc_regs) {
+ dev_err(ipc_pcie->dev, "IPC REGS ioremap error");
+ ret = -EBUSY;
+ goto ipc_regs_remap_fail;
+ }
+
+ /* Reserve the MMIO scratchpad memory resources.
+ * Remap the memory into CPU space. Arrange for the physical address
+ * (BAR) to be visible from this driver.
+ * pci_ioremap_bar() ensures that the memory is marked uncachable.
+ */
+ ipc_pcie->scratchpad =
+ pci_ioremap_bar(pci, ipc_pcie->scratchpad_bar_nr);
+
+ if (!ipc_pcie->scratchpad) {
+ dev_err(ipc_pcie->dev, "doorbell scratchpad ioremap error");
+ ret = -EBUSY;
+ goto scratch_remap_fail;
+ }
+
+ /* Install the irq handler triggered by CP. */
+ ret = ipc_acquire_irq(ipc_pcie);
+ if (ret) {
+ dev_err(ipc_pcie->dev, "acquiring MSI irq failed!");
+ goto irq_acquire_fail;
+ }
+
+ /* Enable bus-mastering for the IOSM IPC device. */
+ pci_set_master(pci);
+
+ /* Enable LTR if possible
+ * This is needed for L1.2!
+ */
+ pcie_capability_read_dword(ipc_pcie->pci, PCI_EXP_DEVCAP2, &cap);
+ if (cap & PCI_EXP_DEVCAP2_LTR)
+ pcie_capability_set_word(ipc_pcie->pci, PCI_EXP_DEVCTL2,
+ PCI_EXP_DEVCTL2_LTR_EN);
+
+ dev_dbg(ipc_pcie->dev, "link between AP and CP is fully on");
+
+ return ret;
+
+irq_acquire_fail:
+ iounmap(ipc_pcie->scratchpad);
+scratch_remap_fail:
+ iounmap(ipc_pcie->ipc_regs);
+ipc_regs_remap_fail:
+ pci_release_regions(pci);
+pci_request_region_fail:
+ return ret;
+}
+
+bool ipc_pcie_check_aspm_enabled(struct iosm_pcie *ipc_pcie,
+ bool parent)
+{
+ struct pci_dev *pdev;
+ u16 value = 0;
+ u32 enabled;
+
+ if (parent)
+ pdev = ipc_pcie->pci->bus->self;
+ else
+ pdev = ipc_pcie->pci;
+
+ pcie_capability_read_word(pdev, PCI_EXP_LNKCTL, &value);
+ enabled = value & PCI_EXP_LNKCTL_ASPMC;
+ dev_dbg(ipc_pcie->dev, "ASPM L1: 0x%04X 0x%03X", pdev->device, value);
+
+ return (enabled == PCI_EXP_LNKCTL_ASPM_L1 ||
+ enabled == PCI_EXP_LNKCTL_ASPMC);
+}
+
+bool ipc_pcie_check_data_link_active(struct iosm_pcie *ipc_pcie)
+{
+ struct pci_dev *parent;
+ u16 link_status = 0;
+
+ if (!ipc_pcie->pci->bus || !ipc_pcie->pci->bus->self) {
+ dev_err(ipc_pcie->dev, "root port not found");
+ return false;
+ }
+
+ parent = ipc_pcie->pci->bus->self;
+
+ pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &link_status);
+ dev_dbg(ipc_pcie->dev, "Link status: 0x%04X", link_status);
+
+ return link_status & PCI_EXP_LNKSTA_DLLLA;
+}
+
+static bool ipc_pcie_check_aspm_supported(struct iosm_pcie *ipc_pcie,
+ bool parent)
+{
+ struct pci_dev *pdev;
+ u32 support;
+ u32 cap = 0;
+
+ if (parent)
+ pdev = ipc_pcie->pci->bus->self;
+ else
+ pdev = ipc_pcie->pci;
+ pcie_capability_read_dword(pdev, PCI_EXP_LNKCAP, &cap);
+ support = u32_get_bits(cap, PCI_EXP_LNKCAP_ASPMS);
+ if (support < PCI_EXP_LNKCTL_ASPM_L1) {
+ dev_dbg(ipc_pcie->dev, "ASPM L1 not supported: 0x%04X",
+ pdev->device);
+ return false;
+ }
+ return true;
+}
+
+void ipc_pcie_config_aspm(struct iosm_pcie *ipc_pcie)
+{
+ bool parent_aspm_enabled, dev_aspm_enabled;
+
+ /* check if both root port and child supports ASPM L1 */
+ if (!ipc_pcie_check_aspm_supported(ipc_pcie, true) ||
+ !ipc_pcie_check_aspm_supported(ipc_pcie, false))
+ return;
+
+ parent_aspm_enabled = ipc_pcie_check_aspm_enabled(ipc_pcie, true);
+ dev_aspm_enabled = ipc_pcie_check_aspm_enabled(ipc_pcie, false);
+
+ dev_dbg(ipc_pcie->dev, "ASPM parent: %s device: %s",
+ parent_aspm_enabled ? "Enabled" : "Disabled",
+ dev_aspm_enabled ? "Enabled" : "Disabled");
+}
+
+/* Initializes PCIe endpoint configuration */
+static void ipc_pcie_config_init(struct iosm_pcie *ipc_pcie)
+{
+ /* BAR0 is used for doorbell */
+ ipc_pcie->ipc_regs_bar_nr = IPC_DOORBELL_BAR0;
+
+ /* update HW configuration */
+ ipc_pcie->scratchpad_bar_nr = IPC_SCRATCHPAD_BAR2;
+ ipc_pcie->doorbell_reg_offset = IPC_DOORBELL_CH_OFFSET;
+ ipc_pcie->doorbell_write = IPC_WRITE_PTR_REG_0;
+ ipc_pcie->doorbell_capture = IPC_CAPTURE_PTR_REG_0;
+}
+
+/* This will read the BIOS WWAN RTD3 settings:
+ * D0L1.2/D3L2/Disabled
+ */
+static enum ipc_pcie_sleep_state ipc_pcie_read_bios_cfg(struct device *dev)
+{
+ union acpi_object *object;
+ acpi_handle handle_acpi;
+
+ handle_acpi = ACPI_HANDLE(dev);
+ if (!handle_acpi) {
+ pr_debug("pci device is NOT ACPI supporting device\n");
+ goto default_ret;
+ }
+
+ object = acpi_evaluate_dsm(handle_acpi, &wwan_acpi_guid, 0, 3, NULL);
+
+ if (object && object->integer.value == 3)
+ return IPC_PCIE_D3L2;
+
+default_ret:
+ return IPC_PCIE_D0L12;
+}
+
+static int ipc_pcie_probe(struct pci_dev *pci,
+ const struct pci_device_id *pci_id)
+{
+ struct iosm_pcie *ipc_pcie = kzalloc(sizeof(*ipc_pcie), GFP_KERNEL);
+
+ pr_debug("Probing device 0x%X from the vendor 0x%X", pci_id->device,
+ pci_id->vendor);
+
+ if (!ipc_pcie)
+ goto ret_fail;
+
+ /* Initialize ipc dbg component for the PCIe device */
+ ipc_pcie->dev = &pci->dev;
+
+ /* Set the driver specific data. */
+ pci_set_drvdata(pci, ipc_pcie);
+
+ /* Save the address of the PCI device configuration. */
+ ipc_pcie->pci = pci;
+
+ /* Update platform configuration */
+ ipc_pcie_config_init(ipc_pcie);
+
+ /* Initialize the device before it is used. Ask low-level code
+ * to enable I/O and memory. Wake up the device if it was suspended.
+ */
+ if (pci_enable_device(pci)) {
+ dev_err(ipc_pcie->dev, "failed to enable the AP PCIe device");
+ /* If enable of PCIe device has failed then calling
+ * ipc_pcie_cleanup will panic the system. More over
+ * ipc_pcie_cleanup() is required to be called after
+ * ipc_imem_mount()
+ */
+ goto pci_enable_fail;
+ }
+
+ ipc_pcie_config_aspm(ipc_pcie);
+ dev_dbg(ipc_pcie->dev, "PCIe device enabled.");
+
+ /* Read WWAN RTD3 BIOS Setting
+ */
+ ipc_pcie->d3l2_support = ipc_pcie_read_bios_cfg(&pci->dev);
+
+ ipc_pcie->suspend = 0;
+
+ if (ipc_pcie_resources_request(ipc_pcie))
+ goto resources_req_fail;
+
+ /* Establish the link to the imem layer. */
+ ipc_pcie->imem = ipc_imem_init(ipc_pcie, pci->device,
+ ipc_pcie->scratchpad, ipc_pcie->dev);
+ if (!ipc_pcie->imem) {
+ dev_err(ipc_pcie->dev, "failed to init imem");
+ goto imem_init_fail;
+ }
+
+ return 0;
+
+imem_init_fail:
+ ipc_pcie_resources_release(ipc_pcie);
+resources_req_fail:
+ pci_disable_device(pci);
+pci_enable_fail:
+ kfree(ipc_pcie);
+ret_fail:
+ return -EIO;
+}
+
+static const struct pci_device_id iosm_ipc_ids[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_INTEL, INTEL_CP_DEVICE_7560_ID) },
+ {}
+};
+MODULE_DEVICE_TABLE(pci, iosm_ipc_ids);
+
+/* Enter sleep in s2idle case
+ */
+static int __maybe_unused ipc_pcie_suspend_s2idle(struct iosm_pcie *ipc_pcie)
+{
+ ipc_cp_irq_sleep_control(ipc_pcie, IPC_MEM_DEV_PM_FORCE_SLEEP);
+
+ /* Complete all memory stores before setting bit */
+ smp_mb__before_atomic();
+
+ set_bit(0, &ipc_pcie->suspend);
+
+ /* Complete all memory stores after setting bit */
+ smp_mb__after_atomic();
+
+ ipc_imem_pm_s2idle_sleep(ipc_pcie->imem, true);
+
+ return 0;
+}
+
+/* Resume from sleep in s2idle case
+ */
+static int __maybe_unused ipc_pcie_resume_s2idle(struct iosm_pcie *ipc_pcie)
+{
+ ipc_cp_irq_sleep_control(ipc_pcie, IPC_MEM_DEV_PM_FORCE_ACTIVE);
+
+ ipc_imem_pm_s2idle_sleep(ipc_pcie->imem, false);
+
+ /* Complete all memory stores before clearing bit. */
+ smp_mb__before_atomic();
+
+ clear_bit(0, &ipc_pcie->suspend);
+
+ /* Complete all memory stores after clearing bit. */
+ smp_mb__after_atomic();
+ return 0;
+}
+
+int __maybe_unused ipc_pcie_suspend(struct iosm_pcie *ipc_pcie)
+{
+ struct pci_dev *pdev;
+ int ret;
+
+ pdev = ipc_pcie->pci;
+
+ /* Execute D3 one time. */
+ if (pdev->current_state != PCI_D0) {
+ dev_dbg(ipc_pcie->dev, "done for PM=%d", pdev->current_state);
+ return 0;
+ }
+
+ /* The HAL shall ask the shared memory layer whether D3 is allowed. */
+ ipc_imem_pm_suspend(ipc_pcie->imem);
+
+ /* Save the PCI configuration space of a device before suspending. */
+ ret = pci_save_state(pdev);
+
+ if (ret) {
+ dev_err(ipc_pcie->dev, "pci_save_state error=%d", ret);
+ return ret;
+ }
+
+ /* Set the power state of a PCI device.
+ * Transition a device to a new power state, using the device's PCI PM
+ * registers.
+ */
+ ret = pci_set_power_state(pdev, PCI_D3cold);
+
+ if (ret) {
+ dev_err(ipc_pcie->dev, "pci_set_power_state error=%d", ret);
+ return ret;
+ }
+
+ dev_dbg(ipc_pcie->dev, "SUSPEND done");
+ return ret;
+}
+
+int __maybe_unused ipc_pcie_resume(struct iosm_pcie *ipc_pcie)
+{
+ int ret;
+
+ /* Set the power state of a PCI device.
+ * Transition a device to a new power state, using the device's PCI PM
+ * registers.
+ */
+ ret = pci_set_power_state(ipc_pcie->pci, PCI_D0);
+
+ if (ret) {
+ dev_err(ipc_pcie->dev, "pci_set_power_state error=%d", ret);
+ return ret;
+ }
+
+ pci_restore_state(ipc_pcie->pci);
+
+ /* The HAL shall inform the shared memory layer that the device is
+ * active.
+ */
+ ipc_imem_pm_resume(ipc_pcie->imem);
+
+ dev_dbg(ipc_pcie->dev, "RESUME done");
+ return ret;
+}
+
+static int __maybe_unused ipc_pcie_suspend_cb(struct device *dev)
+{
+ struct iosm_pcie *ipc_pcie;
+ struct pci_dev *pdev;
+
+ pdev = to_pci_dev(dev);
+
+ ipc_pcie = pci_get_drvdata(pdev);
+
+ switch (ipc_pcie->d3l2_support) {
+ case IPC_PCIE_D0L12:
+ ipc_pcie_suspend_s2idle(ipc_pcie);
+ break;
+ case IPC_PCIE_D3L2:
+ ipc_pcie_suspend(ipc_pcie);
+ break;
+ }
+
+ return 0;
+}
+
+static int __maybe_unused ipc_pcie_resume_cb(struct device *dev)
+{
+ struct iosm_pcie *ipc_pcie;
+ struct pci_dev *pdev;
+
+ pdev = to_pci_dev(dev);
+
+ ipc_pcie = pci_get_drvdata(pdev);
+
+ switch (ipc_pcie->d3l2_support) {
+ case IPC_PCIE_D0L12:
+ ipc_pcie_resume_s2idle(ipc_pcie);
+ break;
+ case IPC_PCIE_D3L2:
+ ipc_pcie_resume(ipc_pcie);
+ break;
+ }
+
+ return 0;
+}
+
+static SIMPLE_DEV_PM_OPS(iosm_ipc_pm, ipc_pcie_suspend_cb, ipc_pcie_resume_cb);
+
+static struct pci_driver iosm_ipc_driver = {
+ .name = KBUILD_MODNAME,
+ .probe = ipc_pcie_probe,
+ .remove = ipc_pcie_remove,
+ .driver = {
+ .pm = &iosm_ipc_pm,
+ },
+ .id_table = iosm_ipc_ids,
+};
+
+int ipc_pcie_addr_map(struct iosm_pcie *ipc_pcie, unsigned char *data,
+ size_t size, dma_addr_t *mapping, int direction)
+{
+ if (ipc_pcie->pci) {
+ *mapping = dma_map_single(&ipc_pcie->pci->dev, data, size,
+ direction);
+ if (dma_mapping_error(&ipc_pcie->pci->dev, *mapping)) {
+ dev_err(ipc_pcie->dev, "dma mapping failed");
+ return -EINVAL;
+ }
+ }
+ return 0;
+}
+
+void ipc_pcie_addr_unmap(struct iosm_pcie *ipc_pcie, size_t size,
+ dma_addr_t mapping, int direction)
+{
+ if (!mapping)
+ return;
+ if (ipc_pcie->pci)
+ dma_unmap_single(&ipc_pcie->pci->dev, mapping, size, direction);
+}
+
+struct sk_buff *ipc_pcie_alloc_local_skb(struct iosm_pcie *ipc_pcie,
+ gfp_t flags, size_t size)
+{
+ struct sk_buff *skb;
+
+ if (!ipc_pcie || !size) {
+ pr_err("invalid pcie object or size");
+ return NULL;
+ }
+
+ skb = __netdev_alloc_skb(NULL, size, flags);
+ if (!skb)
+ return NULL;
+
+ IPC_CB(skb)->op_type = (u8)UL_DEFAULT;
+ IPC_CB(skb)->mapping = 0;
+
+ return skb;
+}
+
+struct sk_buff *ipc_pcie_alloc_skb(struct iosm_pcie *ipc_pcie, size_t size,
+ gfp_t flags, dma_addr_t *mapping,
+ int direction, size_t headroom)
+{
+ struct sk_buff *skb = ipc_pcie_alloc_local_skb(ipc_pcie, flags,
+ size + headroom);
+ if (!skb)
+ return NULL;
+
+ if (headroom)
+ skb_reserve(skb, headroom);
+
+ if (ipc_pcie_addr_map(ipc_pcie, skb->data, size, mapping, direction)) {
+ dev_kfree_skb(skb);
+ return NULL;
+ }
+
+ BUILD_BUG_ON(sizeof(*IPC_CB(skb)) > sizeof(skb->cb));
+
+ /* Store the mapping address in skb scratch pad for later usage */
+ IPC_CB(skb)->mapping = *mapping;
+ IPC_CB(skb)->direction = direction;
+ IPC_CB(skb)->len = size;
+
+ return skb;
+}
+
+void ipc_pcie_kfree_skb(struct iosm_pcie *ipc_pcie, struct sk_buff *skb)
+{
+ if (!skb)
+ return;
+
+ ipc_pcie_addr_unmap(ipc_pcie, IPC_CB(skb)->len, IPC_CB(skb)->mapping,
+ IPC_CB(skb)->direction);
+ IPC_CB(skb)->mapping = 0;
+ dev_kfree_skb(skb);
+}
+
+static int __init iosm_ipc_driver_init(void)
+{
+ if (pci_register_driver(&iosm_ipc_driver)) {
+ pr_err("registering of IOSM PCIe driver failed");
+ return -1;
+ }
+
+ return 0;
+}
+
+static void __exit iosm_ipc_driver_exit(void)
+{
+ pci_unregister_driver(&iosm_ipc_driver);
+}
+
+module_init(iosm_ipc_driver_init);
+module_exit(iosm_ipc_driver_exit);
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#ifndef IOSM_IPC_PCIE_H
+#define IOSM_IPC_PCIE_H
+
+#include <linux/device.h>
+#include <linux/pci.h>
+#include <linux/skbuff.h>
+
+#include "iosm_ipc_irq.h"
+
+/* Device ID */
+#define INTEL_CP_DEVICE_7560_ID 0x7560
+
+/* Define for BAR area usage */
+#define IPC_DOORBELL_BAR0 0
+#define IPC_SCRATCHPAD_BAR2 2
+
+/* Defines for DOORBELL registers information */
+#define IPC_DOORBELL_CH_OFFSET BIT(5)
+#define IPC_WRITE_PTR_REG_0 BIT(4)
+#define IPC_CAPTURE_PTR_REG_0 BIT(3)
+
+/* Number of MSI used for IPC */
+#define IPC_MSI_VECTORS 1
+
+/* Total number of Maximum IPC IRQ vectors used for IPC */
+#define IPC_IRQ_VECTORS IPC_MSI_VECTORS
+
+/**
+ * enum ipc_pcie_sleep_state - Enum type to different sleep state transitions
+ * @IPC_PCIE_D0L12: Put the sleep state in D0L12
+ * @IPC_PCIE_D3L2: Put the sleep state in D3L2
+ */
+enum ipc_pcie_sleep_state {
+ IPC_PCIE_D0L12,
+ IPC_PCIE_D3L2,
+};
+
+/**
+ * struct iosm_pcie - IPC_PCIE struct.
+ * @pci: Address of the device description
+ * @dev: Pointer to generic device structure
+ * @ipc_regs: Remapped CP doorbell address of the irq register
+ * set, to fire the doorbell irq.
+ * @scratchpad: Remapped CP scratchpad address, to send the
+ * configuration. tuple and the IPC descriptors
+ * to CP in the ROM phase. The config tuple
+ * information are saved on the MSI scratchpad.
+ * @imem: Pointer to imem data struct
+ * @ipc_regs_bar_nr: BAR number to be used for IPC doorbell
+ * @scratchpad_bar_nr: BAR number to be used for Scratchpad
+ * @nvec: number of requested irq vectors
+ * @doorbell_reg_offset: doorbell_reg_offset
+ * @doorbell_write: doorbell write register
+ * @doorbell_capture: doorbell capture resgister
+ * @suspend: S2IDLE sleep/active
+ * @d3l2_support: Read WWAN RTD3 BIOS setting for D3L2 support
+ */
+struct iosm_pcie {
+ struct pci_dev *pci;
+ struct device *dev;
+ void __iomem *ipc_regs;
+ void __iomem *scratchpad;
+ struct iosm_imem *imem;
+ int ipc_regs_bar_nr;
+ int scratchpad_bar_nr;
+ int nvec;
+ u32 doorbell_reg_offset;
+ u32 doorbell_write;
+ u32 doorbell_capture;
+ unsigned long suspend;
+ enum ipc_pcie_sleep_state d3l2_support;
+};
+
+/**
+ * struct ipc_skb_cb - Struct definition of the socket buffer which is mapped to
+ * the cb field of sbk
+ * @mapping: Store physical or IOVA mapped address of skb virtual add.
+ * @direction: DMA direction
+ * @len: Length of the DMA mapped region
+ * @op_type: Expected values are defined about enum ipc_ul_usr_op.
+ */
+struct ipc_skb_cb {
+ dma_addr_t mapping;
+ int direction;
+ int len;
+ u8 op_type;
+};
+
+/**
+ * enum ipc_ul_usr_op - Control operation to execute the right action on
+ * the user interface.
+ * @UL_USR_OP_BLOCKED: The uplink app was blocked until CP confirms that the
+ * uplink buffer was consumed triggered by the IRQ.
+ * @UL_MUX_OP_ADB: In MUX mode the UL ADB shall be addedd to the free list.
+ * @UL_DEFAULT: SKB in non muxing mode
+ */
+enum ipc_ul_usr_op {
+ UL_USR_OP_BLOCKED,
+ UL_MUX_OP_ADB,
+ UL_DEFAULT,
+};
+
+/**
+ * ipc_pcie_addr_map - Maps the kernel's virtual address to either IOVA
+ * address space or Physical address space, the mapping is
+ * stored in the skb's cb.
+ * @ipc_pcie: Pointer to struct iosm_pcie
+ * @data: Skb mem containing data
+ * @size: Data size
+ * @mapping: Dma mapping address
+ * @direction: Data direction
+ *
+ * Returns: 0 on success and failure value on error
+ */
+int ipc_pcie_addr_map(struct iosm_pcie *ipc_pcie, unsigned char *data,
+ size_t size, dma_addr_t *mapping, int direction);
+
+/**
+ * ipc_pcie_addr_unmap - Unmaps the skb memory region from IOVA address space
+ * @ipc_pcie: Pointer to struct iosm_pcie
+ * @size: Data size
+ * @mapping: Dma mapping address
+ * @direction: Data direction
+ */
+void ipc_pcie_addr_unmap(struct iosm_pcie *ipc_pcie, size_t size,
+ dma_addr_t mapping, int direction);
+
+/**
+ * ipc_pcie_alloc_skb - Allocate an uplink SKB for the given size.
+ * @ipc_pcie: Pointer to struct iosm_pcie
+ * @size: Size of the SKB required.
+ * @flags: Allocation flags
+ * @mapping: Copies either mapped IOVA add. or converted Phy address
+ * @direction: DMA data direction
+ * @headroom: Header data offset
+ *
+ * Returns: Pointer to ipc_skb on Success, NULL on failure.
+ */
+struct sk_buff *ipc_pcie_alloc_skb(struct iosm_pcie *ipc_pcie, size_t size,
+ gfp_t flags, dma_addr_t *mapping,
+ int direction, size_t headroom);
+
+/**
+ * ipc_pcie_alloc_local_skb - Allocate a local SKB for the given size.
+ * @ipc_pcie: Pointer to struct iosm_pcie
+ * @flags: Allocation flags
+ * @size: Size of the SKB required.
+ *
+ * Returns: Pointer to ipc_skb on Success, NULL on failure.
+ */
+struct sk_buff *ipc_pcie_alloc_local_skb(struct iosm_pcie *ipc_pcie,
+ gfp_t flags, size_t size);
+
+/**
+ * ipc_pcie_kfree_skb - Free skb allocated by ipc_pcie_alloc_*_skb().
+ * @ipc_pcie: Pointer to struct iosm_pcie
+ * @skb: Pointer to the skb
+ */
+void ipc_pcie_kfree_skb(struct iosm_pcie *ipc_pcie, struct sk_buff *skb);
+
+/**
+ * ipc_pcie_check_data_link_active - Check Data Link Layer Active
+ * @ipc_pcie: Pointer to struct iosm_pcie
+ *
+ * Returns: true if active, otherwise false
+ */
+bool ipc_pcie_check_data_link_active(struct iosm_pcie *ipc_pcie);
+
+/**
+ * ipc_pcie_suspend - Callback invoked by pm_runtime_suspend. It decrements
+ * the device's usage count then, carry out a suspend,
+ * either synchronous or asynchronous.
+ * @ipc_pcie: Pointer to struct iosm_pcie
+ *
+ * Returns: 0 on success and failure value on error
+ */
+int ipc_pcie_suspend(struct iosm_pcie *ipc_pcie);
+
+/**
+ * ipc_pcie_resume - Callback invoked by pm_runtime_resume. It increments
+ * the device's usage count then, carry out a resume,
+ * either synchronous or asynchronous.
+ * @ipc_pcie: Pointer to struct iosm_pcie
+ *
+ * Returns: 0 on success and failure value on error
+ */
+int ipc_pcie_resume(struct iosm_pcie *ipc_pcie);
+
+/**
+ * ipc_pcie_check_aspm_enabled - Check if ASPM L1 is already enabled
+ * @ipc_pcie: Pointer to struct iosm_pcie
+ * @parent: True if checking ASPM L1 for parent else false
+ *
+ * Returns: true if ASPM is already enabled else false
+ */
+bool ipc_pcie_check_aspm_enabled(struct iosm_pcie *ipc_pcie,
+ bool parent);
+/**
+ * ipc_pcie_config_aspm - Configure ASPM L1
+ * @ipc_pcie: Pointer to struct iosm_pcie
+ */
+void ipc_pcie_config_aspm(struct iosm_pcie *ipc_pcie);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include "iosm_ipc_protocol.h"
+
+/* Timeout value in MS for the PM to wait for device to reach active state */
+#define IPC_PM_ACTIVE_TIMEOUT_MS (500)
+
+/* Note that here "active" has the value 1, as compared to the enums
+ * ipc_mem_host_pm_state or ipc_mem_dev_pm_state, where "active" is 0
+ */
+#define IPC_PM_SLEEP (0)
+#define CONSUME_STATE (0)
+#define IPC_PM_ACTIVE (1)
+
+void ipc_pm_signal_hpda_doorbell(struct iosm_pm *ipc_pm, u32 identifier,
+ bool host_slp_check)
+{
+ if (host_slp_check && ipc_pm->host_pm_state != IPC_MEM_HOST_PM_ACTIVE &&
+ ipc_pm->host_pm_state != IPC_MEM_HOST_PM_ACTIVE_WAIT) {
+ ipc_pm->pending_hpda_update = true;
+ dev_dbg(ipc_pm->dev,
+ "Pend HPDA update set. Host PM_State: %d identifier:%d",
+ ipc_pm->host_pm_state, identifier);
+ return;
+ }
+
+ if (!ipc_pm_trigger(ipc_pm, IPC_PM_UNIT_IRQ, true)) {
+ ipc_pm->pending_hpda_update = true;
+ dev_dbg(ipc_pm->dev, "Pending HPDA update set. identifier:%d",
+ identifier);
+ return;
+ }
+ ipc_pm->pending_hpda_update = false;
+
+ /* Trigger the irq towards CP */
+ ipc_cp_irq_hpda_update(ipc_pm->pcie, identifier);
+
+ ipc_pm_trigger(ipc_pm, IPC_PM_UNIT_IRQ, false);
+}
+
+/* Wake up the device if it is in low power mode. */
+static bool ipc_pm_link_activate(struct iosm_pm *ipc_pm)
+{
+ if (ipc_pm->cp_state == IPC_MEM_DEV_PM_ACTIVE)
+ return true;
+
+ if (ipc_pm->cp_state == IPC_MEM_DEV_PM_SLEEP) {
+ if (ipc_pm->ap_state == IPC_MEM_DEV_PM_SLEEP) {
+ /* Wake up the device. */
+ ipc_cp_irq_sleep_control(ipc_pm->pcie,
+ IPC_MEM_DEV_PM_WAKEUP);
+ ipc_pm->ap_state = IPC_MEM_DEV_PM_ACTIVE_WAIT;
+
+ goto not_active;
+ }
+
+ if (ipc_pm->ap_state == IPC_MEM_DEV_PM_ACTIVE_WAIT)
+ goto not_active;
+
+ return true;
+ }
+
+not_active:
+ /* link is not ready */
+ return false;
+}
+
+bool ipc_pm_wait_for_device_active(struct iosm_pm *ipc_pm)
+{
+ bool ret_val = false;
+
+ if (ipc_pm->ap_state != IPC_MEM_DEV_PM_ACTIVE) {
+ /* Complete all memory stores before setting bit */
+ smp_mb__before_atomic();
+
+ /* Wait for IPC_PM_ACTIVE_TIMEOUT_MS for Device sleep state
+ * machine to enter ACTIVE state.
+ */
+ set_bit(0, &ipc_pm->host_sleep_pend);
+
+ /* Complete all memory stores after setting bit */
+ smp_mb__after_atomic();
+
+ if (!wait_for_completion_interruptible_timeout
+ (&ipc_pm->host_sleep_complete,
+ msecs_to_jiffies(IPC_PM_ACTIVE_TIMEOUT_MS))) {
+ dev_err(ipc_pm->dev,
+ "PM timeout. Expected State:%d. Actual: %d",
+ IPC_MEM_DEV_PM_ACTIVE, ipc_pm->ap_state);
+ goto active_timeout;
+ }
+ }
+
+ ret_val = true;
+active_timeout:
+ /* Complete all memory stores before clearing bit */
+ smp_mb__before_atomic();
+
+ /* Reset the atomic variable in any case as device sleep
+ * state machine change is no longer of interest.
+ */
+ clear_bit(0, &ipc_pm->host_sleep_pend);
+
+ /* Complete all memory stores after clearing bit */
+ smp_mb__after_atomic();
+
+ return ret_val;
+}
+
+static void ipc_pm_on_link_sleep(struct iosm_pm *ipc_pm)
+{
+ /* pending sleep ack and all conditions are cleared
+ * -> signal SLEEP__ACK to CP
+ */
+ ipc_pm->cp_state = IPC_MEM_DEV_PM_SLEEP;
+ ipc_pm->ap_state = IPC_MEM_DEV_PM_SLEEP;
+
+ ipc_cp_irq_sleep_control(ipc_pm->pcie, IPC_MEM_DEV_PM_SLEEP);
+}
+
+static void ipc_pm_on_link_wake(struct iosm_pm *ipc_pm, bool ack)
+{
+ ipc_pm->ap_state = IPC_MEM_DEV_PM_ACTIVE;
+
+ if (ack) {
+ ipc_pm->cp_state = IPC_MEM_DEV_PM_ACTIVE;
+
+ ipc_cp_irq_sleep_control(ipc_pm->pcie, IPC_MEM_DEV_PM_ACTIVE);
+
+ /* check the consume state !!! */
+ if (test_bit(CONSUME_STATE, &ipc_pm->host_sleep_pend))
+ complete(&ipc_pm->host_sleep_complete);
+ }
+
+ /* Check for pending HPDA update.
+ * Pending HP update could be because of sending message was
+ * put on hold due to Device sleep state or due to TD update
+ * which could be because of Device Sleep and Host Sleep
+ * states.
+ */
+ if (ipc_pm->pending_hpda_update &&
+ ipc_pm->host_pm_state == IPC_MEM_HOST_PM_ACTIVE)
+ ipc_pm_signal_hpda_doorbell(ipc_pm, IPC_HP_PM_TRIGGER, true);
+}
+
+bool ipc_pm_trigger(struct iosm_pm *ipc_pm, enum ipc_pm_unit unit, bool active)
+{
+ union ipc_pm_cond old_cond;
+ union ipc_pm_cond new_cond;
+ bool link_active;
+
+ /* Save the current D3 state. */
+ new_cond = ipc_pm->pm_cond;
+ old_cond = ipc_pm->pm_cond;
+
+ /* Calculate the power state only in the runtime phase. */
+ switch (unit) {
+ case IPC_PM_UNIT_IRQ: /* CP irq */
+ new_cond.irq = active;
+ break;
+
+ case IPC_PM_UNIT_LINK: /* Device link state. */
+ new_cond.link = active;
+ break;
+
+ case IPC_PM_UNIT_HS: /* Host sleep trigger requires Link. */
+ new_cond.hs = active;
+ break;
+
+ default:
+ break;
+ }
+
+ /* Something changed ? */
+ if (old_cond.raw == new_cond.raw) {
+ /* Stay in the current PM state. */
+ link_active = old_cond.link == IPC_PM_ACTIVE;
+ goto ret;
+ }
+
+ ipc_pm->pm_cond = new_cond;
+
+ if (new_cond.link)
+ ipc_pm_on_link_wake(ipc_pm, unit == IPC_PM_UNIT_LINK);
+ else if (unit == IPC_PM_UNIT_LINK)
+ ipc_pm_on_link_sleep(ipc_pm);
+
+ if (old_cond.link == IPC_PM_SLEEP && new_cond.raw) {
+ link_active = ipc_pm_link_activate(ipc_pm);
+ goto ret;
+ }
+
+ link_active = old_cond.link == IPC_PM_ACTIVE;
+
+ret:
+ return link_active;
+}
+
+bool ipc_pm_prepare_host_sleep(struct iosm_pm *ipc_pm)
+{
+ /* suspend not allowed if host_pm_state is not IPC_MEM_HOST_PM_ACTIVE */
+ if (ipc_pm->host_pm_state != IPC_MEM_HOST_PM_ACTIVE) {
+ dev_err(ipc_pm->dev, "host_pm_state=%d\tExpected to be: %d",
+ ipc_pm->host_pm_state, IPC_MEM_HOST_PM_ACTIVE);
+ return false;
+ }
+
+ ipc_pm->host_pm_state = IPC_MEM_HOST_PM_SLEEP_WAIT_D3;
+
+ return true;
+}
+
+bool ipc_pm_prepare_host_active(struct iosm_pm *ipc_pm)
+{
+ if (ipc_pm->host_pm_state != IPC_MEM_HOST_PM_SLEEP) {
+ dev_err(ipc_pm->dev, "host_pm_state=%d\tExpected to be: %d",
+ ipc_pm->host_pm_state, IPC_MEM_HOST_PM_SLEEP);
+ return false;
+ }
+
+ /* Sending Sleep Exit message to CP. Update the state */
+ ipc_pm->host_pm_state = IPC_MEM_HOST_PM_ACTIVE_WAIT;
+
+ return true;
+}
+
+void ipc_pm_set_s2idle_sleep(struct iosm_pm *ipc_pm, bool sleep)
+{
+ if (sleep) {
+ ipc_pm->ap_state = IPC_MEM_DEV_PM_SLEEP;
+ ipc_pm->cp_state = IPC_MEM_DEV_PM_SLEEP;
+ ipc_pm->device_sleep_notification = IPC_MEM_DEV_PM_SLEEP;
+ } else {
+ ipc_pm->ap_state = IPC_MEM_DEV_PM_ACTIVE;
+ ipc_pm->cp_state = IPC_MEM_DEV_PM_ACTIVE;
+ ipc_pm->device_sleep_notification = IPC_MEM_DEV_PM_ACTIVE;
+ ipc_pm->pm_cond.link = IPC_PM_ACTIVE;
+ }
+}
+
+bool ipc_pm_dev_slp_notification(struct iosm_pm *ipc_pm, u32 cp_pm_req)
+{
+ if (cp_pm_req == ipc_pm->device_sleep_notification)
+ return false;
+
+ ipc_pm->device_sleep_notification = cp_pm_req;
+
+ /* Evaluate the PM request. */
+ switch (ipc_pm->cp_state) {
+ case IPC_MEM_DEV_PM_ACTIVE:
+ switch (cp_pm_req) {
+ case IPC_MEM_DEV_PM_ACTIVE:
+ break;
+
+ case IPC_MEM_DEV_PM_SLEEP:
+ /* Inform the PM that the device link can go down. */
+ ipc_pm_trigger(ipc_pm, IPC_PM_UNIT_LINK, false);
+ return true;
+
+ default:
+ dev_err(ipc_pm->dev,
+ "loc-pm=%d active: confused req-pm=%d",
+ ipc_pm->cp_state, cp_pm_req);
+ break;
+ }
+ break;
+
+ case IPC_MEM_DEV_PM_SLEEP:
+ switch (cp_pm_req) {
+ case IPC_MEM_DEV_PM_ACTIVE:
+ /* Inform the PM that the device link is active. */
+ ipc_pm_trigger(ipc_pm, IPC_PM_UNIT_LINK, true);
+ break;
+
+ case IPC_MEM_DEV_PM_SLEEP:
+ break;
+
+ default:
+ dev_err(ipc_pm->dev,
+ "loc-pm=%d sleep: confused req-pm=%d",
+ ipc_pm->cp_state, cp_pm_req);
+ break;
+ }
+ break;
+
+ default:
+ dev_err(ipc_pm->dev, "confused loc-pm=%d, req-pm=%d",
+ ipc_pm->cp_state, cp_pm_req);
+ break;
+ }
+
+ return false;
+}
+
+void ipc_pm_init(struct iosm_protocol *ipc_protocol)
+{
+ struct iosm_imem *ipc_imem = ipc_protocol->imem;
+ struct iosm_pm *ipc_pm = &ipc_protocol->pm;
+
+ ipc_pm->pcie = ipc_imem->pcie;
+ ipc_pm->dev = ipc_imem->dev;
+
+ ipc_pm->pm_cond.irq = IPC_PM_SLEEP;
+ ipc_pm->pm_cond.hs = IPC_PM_SLEEP;
+ ipc_pm->pm_cond.link = IPC_PM_ACTIVE;
+
+ ipc_pm->cp_state = IPC_MEM_DEV_PM_ACTIVE;
+ ipc_pm->ap_state = IPC_MEM_DEV_PM_ACTIVE;
+ ipc_pm->host_pm_state = IPC_MEM_HOST_PM_ACTIVE;
+
+ /* Create generic wait-for-completion handler for Host Sleep
+ * and device sleep coordination.
+ */
+ init_completion(&ipc_pm->host_sleep_complete);
+
+ /* Complete all memory stores before clearing bit */
+ smp_mb__before_atomic();
+
+ clear_bit(0, &ipc_pm->host_sleep_pend);
+
+ /* Complete all memory stores after clearing bit */
+ smp_mb__after_atomic();
+}
+
+void ipc_pm_deinit(struct iosm_protocol *proto)
+{
+ struct iosm_pm *ipc_pm = &proto->pm;
+
+ complete(&ipc_pm->host_sleep_complete);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#ifndef IOSM_IPC_PM_H
+#define IOSM_IPC_PM_H
+
+/* Trigger the doorbell interrupt on cp to change the PM sleep/active status */
+#define ipc_cp_irq_sleep_control(ipc_pcie, data) \
+ ipc_doorbell_fire(ipc_pcie, IPC_DOORBELL_IRQ_SLEEP, data)
+
+/* Trigger the doorbell interrupt on CP to do hpda update */
+#define ipc_cp_irq_hpda_update(ipc_pcie, data) \
+ ipc_doorbell_fire(ipc_pcie, IPC_DOORBELL_IRQ_HPDA, 0xFF & (data))
+
+/**
+ * union ipc_pm_cond - Conditions for D3 and the sleep message to CP.
+ * @raw: raw/combined value for faster check
+ * @irq: IRQ towards CP
+ * @hs: Host Sleep
+ * @link: Device link state.
+ */
+union ipc_pm_cond {
+ unsigned int raw;
+
+ struct {
+ unsigned int irq:1,
+ hs:1,
+ link:1;
+ };
+};
+
+/**
+ * enum ipc_mem_host_pm_state - Possible states of the HOST SLEEP finite state
+ * machine.
+ * @IPC_MEM_HOST_PM_ACTIVE: Host is active
+ * @IPC_MEM_HOST_PM_ACTIVE_WAIT: Intermediate state before going to
+ * active
+ * @IPC_MEM_HOST_PM_SLEEP_WAIT_IDLE: Intermediate state to wait for idle
+ * before going into sleep
+ * @IPC_MEM_HOST_PM_SLEEP_WAIT_D3: Intermediate state to wait for D3
+ * before going to sleep
+ * @IPC_MEM_HOST_PM_SLEEP: after this state the interface is not
+ * accessible host is in suspend to RAM
+ * @IPC_MEM_HOST_PM_SLEEP_WAIT_EXIT_SLEEP: Intermediate state before exiting
+ * sleep
+ */
+enum ipc_mem_host_pm_state {
+ IPC_MEM_HOST_PM_ACTIVE,
+ IPC_MEM_HOST_PM_ACTIVE_WAIT,
+ IPC_MEM_HOST_PM_SLEEP_WAIT_IDLE,
+ IPC_MEM_HOST_PM_SLEEP_WAIT_D3,
+ IPC_MEM_HOST_PM_SLEEP,
+ IPC_MEM_HOST_PM_SLEEP_WAIT_EXIT_SLEEP,
+};
+
+/**
+ * enum ipc_mem_dev_pm_state - Possible states of the DEVICE SLEEP finite state
+ * machine.
+ * @IPC_MEM_DEV_PM_ACTIVE: IPC_MEM_DEV_PM_ACTIVE is the initial
+ * power management state.
+ * IRQ(struct ipc_mem_device_info:
+ * device_sleep_notification)
+ * and DOORBELL-IRQ-HPDA(data) values.
+ * @IPC_MEM_DEV_PM_SLEEP: IPC_MEM_DEV_PM_SLEEP is PM state for
+ * sleep.
+ * @IPC_MEM_DEV_PM_WAKEUP: DOORBELL-IRQ-DEVICE_WAKE(data).
+ * @IPC_MEM_DEV_PM_HOST_SLEEP: DOORBELL-IRQ-HOST_SLEEP(data).
+ * @IPC_MEM_DEV_PM_ACTIVE_WAIT: Local intermediate states.
+ * @IPC_MEM_DEV_PM_FORCE_SLEEP: DOORBELL-IRQ-FORCE_SLEEP.
+ * @IPC_MEM_DEV_PM_FORCE_ACTIVE: DOORBELL-IRQ-FORCE_ACTIVE.
+ */
+enum ipc_mem_dev_pm_state {
+ IPC_MEM_DEV_PM_ACTIVE,
+ IPC_MEM_DEV_PM_SLEEP,
+ IPC_MEM_DEV_PM_WAKEUP,
+ IPC_MEM_DEV_PM_HOST_SLEEP,
+ IPC_MEM_DEV_PM_ACTIVE_WAIT,
+ IPC_MEM_DEV_PM_FORCE_SLEEP = 7,
+ IPC_MEM_DEV_PM_FORCE_ACTIVE,
+};
+
+/**
+ * struct iosm_pm - Power management instance
+ * @pcie: Pointer to iosm_pcie structure
+ * @dev: Pointer to device structure
+ * @host_pm_state: PM states for host
+ * @host_sleep_pend: Variable to indicate Host Sleep Pending
+ * @host_sleep_complete: Generic wait-for-completion used in
+ * case of Host Sleep
+ * @pm_cond: Conditions for power management
+ * @ap_state: Current power management state, the
+ * initial state is IPC_MEM_DEV_PM_ACTIVE eq. 0.
+ * @cp_state: PM State of CP
+ * @device_sleep_notification: last handled device_sleep_notfication
+ * @pending_hpda_update: is a HPDA update pending?
+ */
+struct iosm_pm {
+ struct iosm_pcie *pcie;
+ struct device *dev;
+ enum ipc_mem_host_pm_state host_pm_state;
+ unsigned long host_sleep_pend;
+ struct completion host_sleep_complete;
+ union ipc_pm_cond pm_cond;
+ enum ipc_mem_dev_pm_state ap_state;
+ enum ipc_mem_dev_pm_state cp_state;
+ u32 device_sleep_notification;
+ u8 pending_hpda_update:1;
+};
+
+/**
+ * enum ipc_pm_unit - Power management units.
+ * @IPC_PM_UNIT_IRQ: IRQ towards CP
+ * @IPC_PM_UNIT_HS: Host Sleep for converged protocol
+ * @IPC_PM_UNIT_LINK: Link state controlled by CP.
+ */
+enum ipc_pm_unit {
+ IPC_PM_UNIT_IRQ,
+ IPC_PM_UNIT_HS,
+ IPC_PM_UNIT_LINK,
+};
+
+/**
+ * ipc_pm_init - Allocate power management component
+ * @ipc_protocol: Pointer to iosm_protocol structure
+ */
+void ipc_pm_init(struct iosm_protocol *ipc_protocol);
+
+/**
+ * ipc_pm_deinit - Free power management component, invalidating its pointer.
+ * @ipc_protocol: Pointer to iosm_protocol structure
+ */
+void ipc_pm_deinit(struct iosm_protocol *ipc_protocol);
+
+/**
+ * ipc_pm_dev_slp_notification - Handle a sleep notification message from the
+ * device. This can be called from interrupt state
+ * This function handles Host Sleep requests too
+ * if the Host Sleep protocol is register based.
+ * @ipc_pm: Pointer to power management component
+ * @sleep_notification: Actual notification from device
+ *
+ * Returns: true if dev sleep state has to be checked, false otherwise.
+ */
+bool ipc_pm_dev_slp_notification(struct iosm_pm *ipc_pm,
+ u32 sleep_notification);
+
+/**
+ * ipc_pm_set_s2idle_sleep - Set PM variables to sleep/active
+ * @ipc_pm: Pointer to power management component
+ * @sleep: true to enter sleep/false to exit sleep
+ */
+void ipc_pm_set_s2idle_sleep(struct iosm_pm *ipc_pm, bool sleep);
+
+/**
+ * ipc_pm_prepare_host_sleep - Prepare the PM for sleep by entering
+ * IPC_MEM_HOST_PM_SLEEP_WAIT_D3 state.
+ * @ipc_pm: Pointer to power management component
+ *
+ * Returns: true on success, false if the host was not active.
+ */
+bool ipc_pm_prepare_host_sleep(struct iosm_pm *ipc_pm);
+
+/**
+ * ipc_pm_prepare_host_active - Prepare the PM for wakeup by entering
+ * IPC_MEM_HOST_PM_ACTIVE_WAIT state.
+ * @ipc_pm: Pointer to power management component
+ *
+ * Returns: true on success, false if the host was not sleeping.
+ */
+bool ipc_pm_prepare_host_active(struct iosm_pm *ipc_pm);
+
+/**
+ * ipc_pm_wait_for_device_active - Wait upto IPC_PM_ACTIVE_TIMEOUT_MS ms
+ * for the device to reach active state
+ * @ipc_pm: Pointer to power management component
+ *
+ * Returns: true if device is active, false on timeout
+ */
+bool ipc_pm_wait_for_device_active(struct iosm_pm *ipc_pm);
+
+/**
+ * ipc_pm_signal_hpda_doorbell - Wake up the device if it is in low power mode
+ * and trigger a head pointer update interrupt.
+ * @ipc_pm: Pointer to power management component
+ * @identifier: specifies what component triggered hpda update irq
+ * @host_slp_check: if set to true then Host Sleep state machine check will
+ * be performed. If Host Sleep state machine allows HP
+ * update then only doorbell is triggered otherwise pending
+ * flag will be set. If set to false then Host Sleep check
+ * will not be performed. This is helpful for Host Sleep
+ * negotiation through message ring.
+ */
+void ipc_pm_signal_hpda_doorbell(struct iosm_pm *ipc_pm, u32 identifier,
+ bool host_slp_check);
+/**
+ * ipc_pm_trigger - Update power manager and wake up the link if needed
+ * @ipc_pm: Pointer to power management component
+ * @unit: Power management units
+ * @active: Device link state
+ *
+ * Returns: true if link is unchanged or active, false otherwise
+ */
+bool ipc_pm_trigger(struct iosm_pm *ipc_pm, enum ipc_pm_unit unit, bool active);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include "iosm_ipc_chnl_cfg.h"
+#include "iosm_ipc_imem_ops.h"
+#include "iosm_ipc_port.h"
+
+/* open logical channel for control communication */
+static int ipc_port_ctrl_start(struct wwan_port *port)
+{
+ struct iosm_cdev *ipc_port = wwan_port_get_drvdata(port);
+ int ret = 0;
+
+ ipc_port->channel = ipc_imem_sys_port_open(ipc_port->ipc_imem,
+ ipc_port->chl_id,
+ IPC_HP_CDEV_OPEN);
+ if (!ipc_port->channel)
+ ret = -EIO;
+
+ return ret;
+}
+
+/* close logical channel */
+static void ipc_port_ctrl_stop(struct wwan_port *port)
+{
+ struct iosm_cdev *ipc_port = wwan_port_get_drvdata(port);
+
+ ipc_imem_sys_cdev_close(ipc_port);
+}
+
+/* transfer control data to modem */
+static int ipc_port_ctrl_tx(struct wwan_port *port, struct sk_buff *skb)
+{
+ struct iosm_cdev *ipc_port = wwan_port_get_drvdata(port);
+
+ return ipc_imem_sys_cdev_write(ipc_port, skb);
+}
+
+static const struct wwan_port_ops ipc_wwan_ctrl_ops = {
+ .start = ipc_port_ctrl_start,
+ .stop = ipc_port_ctrl_stop,
+ .tx = ipc_port_ctrl_tx,
+};
+
+/* Port init func */
+struct iosm_cdev *ipc_port_init(struct iosm_imem *ipc_imem,
+ struct ipc_chnl_cfg ipc_port_cfg)
+{
+ struct iosm_cdev *ipc_port = kzalloc(sizeof(*ipc_port), GFP_KERNEL);
+ enum wwan_port_type port_type = ipc_port_cfg.wwan_port_type;
+ enum ipc_channel_id chl_id = ipc_port_cfg.id;
+
+ if (!ipc_port)
+ return NULL;
+
+ ipc_port->dev = ipc_imem->dev;
+ ipc_port->pcie = ipc_imem->pcie;
+
+ ipc_port->port_type = port_type;
+ ipc_port->chl_id = chl_id;
+ ipc_port->ipc_imem = ipc_imem;
+
+ ipc_port->iosm_port = wwan_create_port(ipc_port->dev, port_type,
+ &ipc_wwan_ctrl_ops, ipc_port);
+
+ return ipc_port;
+}
+
+/* Port deinit func */
+void ipc_port_deinit(struct iosm_cdev *port[])
+{
+ struct iosm_cdev *ipc_port;
+ u8 ctrl_chl_nr;
+
+ for (ctrl_chl_nr = 0; ctrl_chl_nr < IPC_MEM_MAX_CHANNELS;
+ ctrl_chl_nr++) {
+ if (port[ctrl_chl_nr]) {
+ ipc_port = port[ctrl_chl_nr];
+ wwan_remove_port(ipc_port->iosm_port);
+ kfree(ipc_port);
+ }
+ }
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#ifndef IOSM_IPC_PORT_H
+#define IOSM_IPC_PORT_H
+
+#include <linux/wwan.h>
+
+#include "iosm_ipc_imem_ops.h"
+
+/**
+ * struct iosm_cdev - State of the char driver layer.
+ * @iosm_port: Pointer of type wwan_port
+ * @ipc_imem: imem instance
+ * @dev: Pointer to device struct
+ * @pcie: PCIe component
+ * @port_type: WWAN port type
+ * @channel: Channel instance
+ * @chl_id: Channel Indentifier
+ */
+struct iosm_cdev {
+ struct wwan_port *iosm_port;
+ struct iosm_imem *ipc_imem;
+ struct device *dev;
+ struct iosm_pcie *pcie;
+ enum wwan_port_type port_type;
+ struct ipc_mem_channel *channel;
+ enum ipc_channel_id chl_id;
+};
+
+/**
+ * ipc_port_init - Allocate IPC port & register to wwan subsystem for AT/MBIM
+ * communication.
+ * @ipc_imem: Pointer to iosm_imem structure
+ * @ipc_port_cfg: IPC Port Config
+ *
+ * Returns: 0 on success & NULL on failure
+ */
+struct iosm_cdev *ipc_port_init(struct iosm_imem *ipc_imem,
+ struct ipc_chnl_cfg ipc_port_cfg);
+
+/**
+ * ipc_port_deinit - Free IPC port & unregister port with wwan subsystem.
+ * @ipc_port: Array of pointer to the ipc port data-struct
+ */
+void ipc_port_deinit(struct iosm_cdev *ipc_port[]);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include "iosm_ipc_imem.h"
+#include "iosm_ipc_protocol.h"
+#include "iosm_ipc_protocol_ops.h"
+#include "iosm_ipc_pm.h"
+#include "iosm_ipc_task_queue.h"
+
+int ipc_protocol_tq_msg_send(struct iosm_protocol *ipc_protocol,
+ enum ipc_msg_prep_type msg_type,
+ union ipc_msg_prep_args *prep_args,
+ struct ipc_rsp *response)
+{
+ int index = ipc_protocol_msg_prep(ipc_protocol->imem, msg_type,
+ prep_args);
+
+ /* Store reference towards caller specified response in response ring
+ * and signal CP
+ */
+ if (index >= 0 && index < IPC_MEM_MSG_ENTRIES) {
+ ipc_protocol->rsp_ring[index] = response;
+ ipc_protocol_msg_hp_update(ipc_protocol->imem);
+ }
+
+ return index;
+}
+
+/* Callback for message send */
+static int ipc_protocol_tq_msg_send_cb(struct iosm_imem *ipc_imem, int arg,
+ void *msg, size_t size)
+{
+ struct ipc_call_msg_send_args *send_args = msg;
+ struct iosm_protocol *ipc_protocol = ipc_imem->ipc_protocol;
+
+ return ipc_protocol_tq_msg_send(ipc_protocol, send_args->msg_type,
+ send_args->prep_args,
+ send_args->response);
+}
+
+/* Remove reference to a response. This is typically used when a requestor timed
+ * out and is no longer interested in the response.
+ */
+static int ipc_protocol_tq_msg_remove(struct iosm_imem *ipc_imem, int arg,
+ void *msg, size_t size)
+{
+ struct iosm_protocol *ipc_protocol = ipc_imem->ipc_protocol;
+
+ ipc_protocol->rsp_ring[arg] = NULL;
+ return 0;
+}
+
+int ipc_protocol_msg_send(struct iosm_protocol *ipc_protocol,
+ enum ipc_msg_prep_type prep,
+ union ipc_msg_prep_args *prep_args)
+{
+ struct ipc_call_msg_send_args send_args;
+ unsigned int exec_timeout;
+ struct ipc_rsp response;
+ int index;
+
+ exec_timeout = (ipc_protocol_get_ap_exec_stage(ipc_protocol) ==
+ IPC_MEM_EXEC_STAGE_RUN ?
+ IPC_MSG_COMPLETE_RUN_DEFAULT_TIMEOUT :
+ IPC_MSG_COMPLETE_BOOT_DEFAULT_TIMEOUT);
+
+ /* Trap if called from non-preemptible context */
+ might_sleep();
+
+ response.status = IPC_MEM_MSG_CS_INVALID;
+ init_completion(&response.completion);
+
+ send_args.msg_type = prep;
+ send_args.prep_args = prep_args;
+ send_args.response = &response;
+
+ /* Allocate and prepare message to be sent in tasklet context.
+ * A positive index returned form tasklet_call references the message
+ * in case it needs to be cancelled when there is a timeout.
+ */
+ index = ipc_task_queue_send_task(ipc_protocol->imem,
+ ipc_protocol_tq_msg_send_cb, 0,
+ &send_args, 0, true);
+
+ if (index < 0) {
+ dev_err(ipc_protocol->dev, "msg %d failed", prep);
+ return index;
+ }
+
+ /* Wait for the device to respond to the message */
+ switch (wait_for_completion_timeout(&response.completion,
+ msecs_to_jiffies(exec_timeout))) {
+ case 0:
+ /* Timeout, there was no response from the device.
+ * Remove the reference to the local response completion
+ * object as we are no longer interested in the response.
+ */
+ ipc_task_queue_send_task(ipc_protocol->imem,
+ ipc_protocol_tq_msg_remove, index,
+ NULL, 0, true);
+ dev_err(ipc_protocol->dev, "msg timeout");
+ ipc_uevent_send(ipc_protocol->pcie->dev, UEVENT_MDM_TIMEOUT);
+ break;
+ default:
+ /* We got a response in time; check completion status: */
+ if (response.status != IPC_MEM_MSG_CS_SUCCESS) {
+ dev_err(ipc_protocol->dev,
+ "msg completion status error %d",
+ response.status);
+ return -EIO;
+ }
+ }
+
+ return 0;
+}
+
+static int ipc_protocol_msg_send_host_sleep(struct iosm_protocol *ipc_protocol,
+ u32 state)
+{
+ union ipc_msg_prep_args prep_args = {
+ .sleep.target = 0,
+ .sleep.state = state,
+ };
+
+ return ipc_protocol_msg_send(ipc_protocol, IPC_MSG_PREP_SLEEP,
+ &prep_args);
+}
+
+void ipc_protocol_doorbell_trigger(struct iosm_protocol *ipc_protocol,
+ u32 identifier)
+{
+ ipc_pm_signal_hpda_doorbell(&ipc_protocol->pm, identifier, true);
+}
+
+bool ipc_protocol_pm_dev_sleep_handle(struct iosm_protocol *ipc_protocol)
+{
+ u32 ipc_status = ipc_protocol_get_ipc_status(ipc_protocol);
+ u32 requested;
+
+ if (ipc_status != IPC_MEM_DEVICE_IPC_RUNNING) {
+ dev_err(ipc_protocol->dev,
+ "irq ignored, CP IPC state is %d, should be RUNNING",
+ ipc_status);
+
+ /* Stop further processing. */
+ return false;
+ }
+
+ /* Get a copy of the requested PM state by the device and the local
+ * device PM state.
+ */
+ requested = ipc_protocol_pm_dev_get_sleep_notification(ipc_protocol);
+
+ return ipc_pm_dev_slp_notification(&ipc_protocol->pm, requested);
+}
+
+static int ipc_protocol_tq_wakeup_dev_slp(struct iosm_imem *ipc_imem, int arg,
+ void *msg, size_t size)
+{
+ struct iosm_pm *ipc_pm = &ipc_imem->ipc_protocol->pm;
+
+ /* Wakeup from device sleep if it is not ACTIVE */
+ ipc_pm_trigger(ipc_pm, IPC_PM_UNIT_HS, true);
+
+ ipc_pm_trigger(ipc_pm, IPC_PM_UNIT_HS, false);
+
+ return 0;
+}
+
+void ipc_protocol_s2idle_sleep(struct iosm_protocol *ipc_protocol, bool sleep)
+{
+ ipc_pm_set_s2idle_sleep(&ipc_protocol->pm, sleep);
+}
+
+bool ipc_protocol_suspend(struct iosm_protocol *ipc_protocol)
+{
+ if (!ipc_pm_prepare_host_sleep(&ipc_protocol->pm))
+ goto err;
+
+ ipc_task_queue_send_task(ipc_protocol->imem,
+ ipc_protocol_tq_wakeup_dev_slp, 0, NULL, 0,
+ true);
+
+ if (!ipc_pm_wait_for_device_active(&ipc_protocol->pm)) {
+ ipc_uevent_send(ipc_protocol->pcie->dev, UEVENT_MDM_TIMEOUT);
+ goto err;
+ }
+
+ /* Send the sleep message for sync sys calls. */
+ dev_dbg(ipc_protocol->dev, "send TARGET_HOST, ENTER_SLEEP");
+ if (ipc_protocol_msg_send_host_sleep(ipc_protocol,
+ IPC_HOST_SLEEP_ENTER_SLEEP)) {
+ /* Sending ENTER_SLEEP message failed, we are still active */
+ ipc_protocol->pm.host_pm_state = IPC_MEM_HOST_PM_ACTIVE;
+ goto err;
+ }
+
+ ipc_protocol->pm.host_pm_state = IPC_MEM_HOST_PM_SLEEP;
+ return true;
+err:
+ return false;
+}
+
+bool ipc_protocol_resume(struct iosm_protocol *ipc_protocol)
+{
+ if (!ipc_pm_prepare_host_active(&ipc_protocol->pm))
+ return false;
+
+ dev_dbg(ipc_protocol->dev, "send TARGET_HOST, EXIT_SLEEP");
+ if (ipc_protocol_msg_send_host_sleep(ipc_protocol,
+ IPC_HOST_SLEEP_EXIT_SLEEP)) {
+ ipc_protocol->pm.host_pm_state = IPC_MEM_HOST_PM_SLEEP;
+ return false;
+ }
+
+ ipc_protocol->pm.host_pm_state = IPC_MEM_HOST_PM_ACTIVE;
+
+ return true;
+}
+
+struct iosm_protocol *ipc_protocol_init(struct iosm_imem *ipc_imem)
+{
+ struct iosm_protocol *ipc_protocol =
+ kzalloc(sizeof(*ipc_protocol), GFP_KERNEL);
+ struct ipc_protocol_context_info *p_ci;
+ u64 addr;
+
+ if (!ipc_protocol)
+ return NULL;
+
+ ipc_protocol->dev = ipc_imem->dev;
+ ipc_protocol->pcie = ipc_imem->pcie;
+ ipc_protocol->imem = ipc_imem;
+ ipc_protocol->p_ap_shm = NULL;
+ ipc_protocol->phy_ap_shm = 0;
+
+ ipc_protocol->old_msg_tail = 0;
+
+ ipc_protocol->p_ap_shm =
+ pci_alloc_consistent(ipc_protocol->pcie->pci,
+ sizeof(*ipc_protocol->p_ap_shm),
+ &ipc_protocol->phy_ap_shm);
+
+ if (!ipc_protocol->p_ap_shm) {
+ dev_err(ipc_protocol->dev, "pci shm alloc error");
+ kfree(ipc_protocol);
+ return NULL;
+ }
+
+ /* Prepare the context info for CP. */
+ addr = ipc_protocol->phy_ap_shm;
+ p_ci = &ipc_protocol->p_ap_shm->ci;
+ p_ci->device_info_addr =
+ addr + offsetof(struct ipc_protocol_ap_shm, device_info);
+ p_ci->head_array =
+ addr + offsetof(struct ipc_protocol_ap_shm, head_array);
+ p_ci->tail_array =
+ addr + offsetof(struct ipc_protocol_ap_shm, tail_array);
+ p_ci->msg_head = addr + offsetof(struct ipc_protocol_ap_shm, msg_head);
+ p_ci->msg_tail = addr + offsetof(struct ipc_protocol_ap_shm, msg_tail);
+ p_ci->msg_ring_addr =
+ addr + offsetof(struct ipc_protocol_ap_shm, msg_ring);
+ p_ci->msg_ring_entries = cpu_to_le16(IPC_MEM_MSG_ENTRIES);
+ p_ci->msg_irq_vector = IPC_MSG_IRQ_VECTOR;
+ p_ci->device_info_irq_vector = IPC_DEVICE_IRQ_VECTOR;
+
+ ipc_mmio_set_contex_info_addr(ipc_imem->mmio, addr);
+
+ ipc_pm_init(ipc_protocol);
+
+ return ipc_protocol;
+}
+
+void ipc_protocol_deinit(struct iosm_protocol *proto)
+{
+ pci_free_consistent(proto->pcie->pci, sizeof(*proto->p_ap_shm),
+ proto->p_ap_shm, proto->phy_ap_shm);
+
+ ipc_pm_deinit(proto);
+ kfree(proto);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#ifndef IOSM_IPC_PROTOCOL_H
+#define IOSM_IPC_PROTOCOL_H
+
+#include "iosm_ipc_imem.h"
+#include "iosm_ipc_pm.h"
+#include "iosm_ipc_protocol_ops.h"
+
+/* Trigger the doorbell interrupt on CP. */
+#define IPC_DOORBELL_IRQ_HPDA 0
+#define IPC_DOORBELL_IRQ_IPC 1
+#define IPC_DOORBELL_IRQ_SLEEP 2
+
+/* IRQ vector number. */
+#define IPC_DEVICE_IRQ_VECTOR 0
+#define IPC_MSG_IRQ_VECTOR 0
+#define IPC_UL_PIPE_IRQ_VECTOR 0
+#define IPC_DL_PIPE_IRQ_VECTOR 0
+
+#define IPC_MEM_MSG_ENTRIES 128
+
+/* Default time out for sending IPC messages like open pipe, close pipe etc.
+ * during run mode.
+ *
+ * If the message interface lock to CP times out, the link to CP is broken.
+ * mode : run mode (IPC_MEM_EXEC_STAGE_RUN)
+ * unit : milliseconds
+ */
+#define IPC_MSG_COMPLETE_RUN_DEFAULT_TIMEOUT 500 /* 0.5 seconds */
+
+/* Default time out for sending IPC messages like open pipe, close pipe etc.
+ * during boot mode.
+ *
+ * If the message interface lock to CP times out, the link to CP is broken.
+ * mode : boot mode
+ * (IPC_MEM_EXEC_STAGE_BOOT | IPC_MEM_EXEC_STAGE_PSI | IPC_MEM_EXEC_STAGE_EBL)
+ * unit : milliseconds
+ */
+#define IPC_MSG_COMPLETE_BOOT_DEFAULT_TIMEOUT 500 /* 0.5 seconds */
+
+/**
+ * struct ipc_protocol_context_info - Structure of the context info
+ * @device_info_addr: 64 bit address to device info
+ * @head_array: 64 bit address to head pointer arr for the pipes
+ * @tail_array: 64 bit address to tail pointer arr for the pipes
+ * @msg_head: 64 bit address to message head pointer
+ * @msg_tail: 64 bit address to message tail pointer
+ * @msg_ring_addr: 64 bit pointer to the message ring buffer
+ * @msg_ring_entries: This field provides the number of entries which
+ * the MR can hold
+ * @msg_irq_vector: This field provides the IRQ which shall be
+ * generated by the EP device when generating
+ * completion for Messages.
+ * @device_info_irq_vector: This field provides the IRQ which shall be
+ * generated by the EP dev after updating Dev. Info
+ */
+struct ipc_protocol_context_info {
+ phys_addr_t device_info_addr;
+ phys_addr_t head_array;
+ phys_addr_t tail_array;
+ phys_addr_t msg_head;
+ phys_addr_t msg_tail;
+ phys_addr_t msg_ring_addr;
+ __le16 msg_ring_entries;
+ u8 msg_irq_vector;
+ u8 device_info_irq_vector;
+};
+
+/**
+ * struct ipc_protocol_device_info - Structure for the device information
+ * @execution_stage: CP execution stage
+ * @ipc_status: IPC states
+ * @device_sleep_notification: Requested device pm states
+ */
+struct ipc_protocol_device_info {
+ __le32 execution_stage;
+ __le32 ipc_status;
+ __le32 device_sleep_notification;
+};
+
+/**
+ * struct ipc_protocol_ap_shm - Protocol Shared Memory Structure
+ * @ci: Context information struct
+ * @device_info: Device information struct
+ * @msg_head: Point to msg head
+ * @head_array: Array of head pointer
+ * @msg_tail: Point to msg tail
+ * @tail_array: Array of tail pointer
+ * @msg_ring: Circular buffers for the read/tail and write/head
+ * indeces.
+ */
+struct ipc_protocol_ap_shm {
+ struct ipc_protocol_context_info ci;
+ struct ipc_protocol_device_info device_info;
+ __le32 msg_head;
+ __le32 head_array[IPC_MEM_MAX_PIPES];
+ __le32 msg_tail;
+ __le32 tail_array[IPC_MEM_MAX_PIPES];
+ union ipc_mem_msg_entry msg_ring[IPC_MEM_MSG_ENTRIES];
+};
+
+/**
+ * struct iosm_protocol - Structure for IPC protocol.
+ * @p_ap_shm: Pointer to Protocol Shared Memory Structure
+ * @pm: Instance to struct iosm_pm
+ * @pcie: Pointer to struct iosm_pcie
+ * @imem: Pointer to struct iosm_imem
+ * @rsp_ring: Array of OS completion objects to be triggered once CP
+ * acknowledges a request in the message ring
+ * @dev: Pointer to device structure
+ * @phy_ap_shm: Physical/Mapped representation of the shared memory info
+ * @old_msg_tail: Old msg tail ptr, until AP has handled ACK's from CP
+ */
+struct iosm_protocol {
+ struct ipc_protocol_ap_shm *p_ap_shm;
+ struct iosm_pm pm;
+ struct iosm_pcie *pcie;
+ struct iosm_imem *imem;
+ struct ipc_rsp *rsp_ring[IPC_MEM_MSG_ENTRIES];
+ struct device *dev;
+ phys_addr_t phy_ap_shm;
+ u32 old_msg_tail;
+};
+
+/**
+ * struct ipc_call_msg_send_args - Structure for message argument for
+ * tasklet function.
+ * @prep_args: Arguments for message preparation function
+ * @response: Can be NULL if result can be ignored
+ * @msg_type: Message Type
+ */
+struct ipc_call_msg_send_args {
+ union ipc_msg_prep_args *prep_args;
+ struct ipc_rsp *response;
+ enum ipc_msg_prep_type msg_type;
+};
+
+/**
+ * ipc_protocol_tq_msg_send - prepare the msg and send to CP
+ * @ipc_protocol: Pointer to ipc_protocol instance
+ * @msg_type: Message type
+ * @prep_args: Message arguments
+ * @response: Pointer to a response object which has a
+ * completion object and return code.
+ *
+ * Returns: 0 on success and failure value on error
+ */
+int ipc_protocol_tq_msg_send(struct iosm_protocol *ipc_protocol,
+ enum ipc_msg_prep_type msg_type,
+ union ipc_msg_prep_args *prep_args,
+ struct ipc_rsp *response);
+
+/**
+ * ipc_protocol_msg_send - Send ipc control message to CP and wait for response
+ * @ipc_protocol: Pointer to ipc_protocol instance
+ * @prep: Message type
+ * @prep_args: Message arguments
+ *
+ * Returns: 0 on success and failure value on error
+ */
+int ipc_protocol_msg_send(struct iosm_protocol *ipc_protocol,
+ enum ipc_msg_prep_type prep,
+ union ipc_msg_prep_args *prep_args);
+
+/**
+ * ipc_protocol_suspend - Signal to CP that host wants to go to sleep (suspend).
+ * @ipc_protocol: Pointer to ipc_protocol instance
+ *
+ * Returns: true if host can suspend, false if suspend must be aborted.
+ */
+bool ipc_protocol_suspend(struct iosm_protocol *ipc_protocol);
+
+/**
+ * ipc_protocol_s2idle_sleep - Call PM function to set PM variables in s2idle
+ * sleep/active case
+ * @ipc_protocol: Pointer to ipc_protocol instance
+ * @sleep: True for sleep/False for active
+ */
+void ipc_protocol_s2idle_sleep(struct iosm_protocol *ipc_protocol, bool sleep);
+
+/**
+ * ipc_protocol_resume - Signal to CP that host wants to resume operation.
+ * @ipc_protocol: Pointer to ipc_protocol instance
+ *
+ * Returns: true if host can resume, false if there is a problem.
+ */
+bool ipc_protocol_resume(struct iosm_protocol *ipc_protocol);
+
+/**
+ * ipc_protocol_pm_dev_sleep_handle - Handles the Device Sleep state change
+ * notification.
+ * @ipc_protocol: Pointer to ipc_protocol instance.
+ *
+ * Returns: true if sleep notification handled, false otherwise.
+ */
+bool ipc_protocol_pm_dev_sleep_handle(struct iosm_protocol *ipc_protocol);
+
+/**
+ * ipc_protocol_doorbell_trigger - Wrapper for PM function which wake up the
+ * device if it is in low power mode
+ * and trigger a head pointer update interrupt.
+ * @ipc_protocol: Pointer to ipc_protocol instance.
+ * @identifier: Specifies what component triggered hpda
+ * update irq
+ */
+void ipc_protocol_doorbell_trigger(struct iosm_protocol *ipc_protocol,
+ u32 identifier);
+
+/**
+ * ipc_protocol_sleep_notification_string - Returns last Sleep Notification as
+ * string.
+ * @ipc_protocol: Instance pointer of Protocol module.
+ *
+ * Returns: Pointer to string.
+ */
+const char *
+ipc_protocol_sleep_notification_string(struct iosm_protocol *ipc_protocol);
+
+/**
+ * ipc_protocol_init - Allocates IPC protocol instance
+ * @ipc_imem: Pointer to iosm_imem structure
+ *
+ * Returns: Address of IPC protocol instance on success & NULL on failure.
+ */
+struct iosm_protocol *ipc_protocol_init(struct iosm_imem *ipc_imem);
+
+/**
+ * ipc_protocol_deinit - Deallocates IPC protocol instance
+ * @ipc_protocol: pointer to the IPC protocol instance
+ */
+void ipc_protocol_deinit(struct iosm_protocol *ipc_protocol);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include "iosm_ipc_protocol.h"
+#include "iosm_ipc_protocol_ops.h"
+
+/* Get the next free message element.*/
+static union ipc_mem_msg_entry *
+ipc_protocol_free_msg_get(struct iosm_protocol *ipc_protocol, int *index)
+{
+ u32 head = le32_to_cpu(ipc_protocol->p_ap_shm->msg_head);
+ u32 new_head = (head + 1) % IPC_MEM_MSG_ENTRIES;
+ union ipc_mem_msg_entry *msg;
+
+ if (new_head == le32_to_cpu(ipc_protocol->p_ap_shm->msg_tail)) {
+ dev_err(ipc_protocol->dev, "message ring is full");
+ return NULL;
+ }
+
+ /* Get the pointer to the next free message element,
+ * reset the fields and mark is as invalid.
+ */
+ msg = &ipc_protocol->p_ap_shm->msg_ring[head];
+ memset(msg, 0, sizeof(*msg));
+
+ /* return index in message ring */
+ *index = head;
+
+ return msg;
+}
+
+/* Updates the message ring Head pointer */
+void ipc_protocol_msg_hp_update(struct iosm_imem *ipc_imem)
+{
+ struct iosm_protocol *ipc_protocol = ipc_imem->ipc_protocol;
+ u32 head = le32_to_cpu(ipc_protocol->p_ap_shm->msg_head);
+ u32 new_head = (head + 1) % IPC_MEM_MSG_ENTRIES;
+
+ /* Update head pointer and fire doorbell. */
+ ipc_protocol->p_ap_shm->msg_head = cpu_to_le32(new_head);
+ ipc_protocol->old_msg_tail =
+ le32_to_cpu(ipc_protocol->p_ap_shm->msg_tail);
+
+ ipc_pm_signal_hpda_doorbell(&ipc_protocol->pm, IPC_HP_MR, false);
+}
+
+/* Allocate and prepare a OPEN_PIPE message.
+ * This also allocates the memory for the new TDR structure and
+ * updates the pipe structure referenced in the preparation arguments.
+ */
+static int ipc_protocol_msg_prepipe_open(struct iosm_protocol *ipc_protocol,
+ union ipc_msg_prep_args *args)
+{
+ int index;
+ union ipc_mem_msg_entry *msg =
+ ipc_protocol_free_msg_get(ipc_protocol, &index);
+ struct ipc_pipe *pipe = args->pipe_open.pipe;
+ struct ipc_protocol_td *tdr;
+ struct sk_buff **skbr;
+
+ if (!msg) {
+ dev_err(ipc_protocol->dev, "failed to get free message");
+ return -EIO;
+ }
+
+ /* Allocate the skbuf elements for the skbuf which are on the way.
+ * SKB ring is internal memory allocation for driver. No need to
+ * re-calculate the start and end addresses.
+ */
+ skbr = kcalloc(pipe->nr_of_entries, sizeof(*skbr), GFP_ATOMIC);
+ if (!skbr)
+ return -ENOMEM;
+
+ /* Allocate the transfer descriptors for the pipe. */
+ tdr = pci_alloc_consistent(ipc_protocol->pcie->pci,
+ pipe->nr_of_entries * sizeof(*tdr),
+ &pipe->phy_tdr_start);
+ if (!tdr) {
+ kfree(skbr);
+ dev_err(ipc_protocol->dev, "tdr alloc error");
+ return -ENOMEM;
+ }
+
+ pipe->max_nr_of_queued_entries = pipe->nr_of_entries - 1;
+ pipe->nr_of_queued_entries = 0;
+ pipe->tdr_start = tdr;
+ pipe->skbr_start = skbr;
+ pipe->old_tail = 0;
+
+ ipc_protocol->p_ap_shm->head_array[pipe->pipe_nr] = 0;
+
+ msg->open_pipe.type_of_message = IPC_MEM_MSG_OPEN_PIPE;
+ msg->open_pipe.pipe_nr = pipe->pipe_nr;
+ msg->open_pipe.tdr_addr = cpu_to_le64(pipe->phy_tdr_start);
+ msg->open_pipe.tdr_entries = cpu_to_le16(pipe->nr_of_entries);
+ msg->open_pipe.accumulation_backoff =
+ cpu_to_le32(pipe->accumulation_backoff);
+ msg->open_pipe.irq_vector = cpu_to_le32(pipe->irq);
+
+ return index;
+}
+
+static int ipc_protocol_msg_prepipe_close(struct iosm_protocol *ipc_protocol,
+ union ipc_msg_prep_args *args)
+{
+ int index = -1;
+ union ipc_mem_msg_entry *msg =
+ ipc_protocol_free_msg_get(ipc_protocol, &index);
+ struct ipc_pipe *pipe = args->pipe_close.pipe;
+
+ if (!msg)
+ return -EIO;
+
+ msg->close_pipe.type_of_message = IPC_MEM_MSG_CLOSE_PIPE;
+ msg->close_pipe.pipe_nr = pipe->pipe_nr;
+
+ dev_dbg(ipc_protocol->dev, "IPC_MEM_MSG_CLOSE_PIPE(pipe_nr=%d)",
+ msg->close_pipe.pipe_nr);
+
+ return index;
+}
+
+static int ipc_protocol_msg_prep_sleep(struct iosm_protocol *ipc_protocol,
+ union ipc_msg_prep_args *args)
+{
+ int index = -1;
+ union ipc_mem_msg_entry *msg =
+ ipc_protocol_free_msg_get(ipc_protocol, &index);
+
+ if (!msg) {
+ dev_err(ipc_protocol->dev, "failed to get free message");
+ return -EIO;
+ }
+
+ /* Prepare and send the host sleep message to CP to enter or exit D3. */
+ msg->host_sleep.type_of_message = IPC_MEM_MSG_SLEEP;
+ msg->host_sleep.target = args->sleep.target; /* 0=host, 1=device */
+
+ /* state; 0=enter, 1=exit 2=enter w/o protocol */
+ msg->host_sleep.state = args->sleep.state;
+
+ dev_dbg(ipc_protocol->dev, "IPC_MEM_MSG_SLEEP(target=%d; state=%d)",
+ msg->host_sleep.target, msg->host_sleep.state);
+
+ return index;
+}
+
+static int ipc_protocol_msg_prep_feature_set(struct iosm_protocol *ipc_protocol,
+ union ipc_msg_prep_args *args)
+{
+ int index = -1;
+ union ipc_mem_msg_entry *msg =
+ ipc_protocol_free_msg_get(ipc_protocol, &index);
+
+ if (!msg) {
+ dev_err(ipc_protocol->dev, "failed to get free message");
+ return -EIO;
+ }
+
+ msg->feature_set.type_of_message = IPC_MEM_MSG_FEATURE_SET;
+ msg->feature_set.reset_enable = args->feature_set.reset_enable <<
+ RESET_BIT;
+
+ dev_dbg(ipc_protocol->dev, "IPC_MEM_MSG_FEATURE_SET(reset_enable=%d)",
+ msg->feature_set.reset_enable >> RESET_BIT);
+
+ return index;
+}
+
+/* Processes the message consumed by CP. */
+bool ipc_protocol_msg_process(struct iosm_imem *ipc_imem, int irq)
+{
+ struct iosm_protocol *ipc_protocol = ipc_imem->ipc_protocol;
+ struct ipc_rsp **rsp_ring = ipc_protocol->rsp_ring;
+ bool msg_processed = false;
+ u32 i;
+
+ if (le32_to_cpu(ipc_protocol->p_ap_shm->msg_tail) >=
+ IPC_MEM_MSG_ENTRIES) {
+ dev_err(ipc_protocol->dev, "msg_tail out of range: %d",
+ le32_to_cpu(ipc_protocol->p_ap_shm->msg_tail));
+ return msg_processed;
+ }
+
+ if (irq != IMEM_IRQ_DONT_CARE &&
+ irq != ipc_protocol->p_ap_shm->ci.msg_irq_vector)
+ return msg_processed;
+
+ for (i = ipc_protocol->old_msg_tail;
+ i != le32_to_cpu(ipc_protocol->p_ap_shm->msg_tail);
+ i = (i + 1) % IPC_MEM_MSG_ENTRIES) {
+ union ipc_mem_msg_entry *msg =
+ &ipc_protocol->p_ap_shm->msg_ring[i];
+
+ dev_dbg(ipc_protocol->dev, "msg[%d]: type=%u status=%d", i,
+ msg->common.type_of_message,
+ msg->common.completion_status);
+
+ /* Update response with status and wake up waiting requestor */
+ if (rsp_ring[i]) {
+ rsp_ring[i]->status =
+ le32_to_cpu(msg->common.completion_status);
+ complete(&rsp_ring[i]->completion);
+ rsp_ring[i] = NULL;
+ }
+ msg_processed = true;
+ }
+
+ ipc_protocol->old_msg_tail = i;
+ return msg_processed;
+}
+
+/* Sends data from UL list to CP for the provided pipe by updating the Head
+ * pointer of given pipe.
+ */
+bool ipc_protocol_ul_td_send(struct iosm_protocol *ipc_protocol,
+ struct ipc_pipe *pipe,
+ struct sk_buff_head *p_ul_list)
+{
+ struct ipc_protocol_td *td;
+ bool hpda_pending = false;
+ struct sk_buff *skb;
+ s32 free_elements;
+ u32 head;
+ u32 tail;
+
+ if (!ipc_protocol->p_ap_shm) {
+ dev_err(ipc_protocol->dev, "driver is not initialized");
+ return false;
+ }
+
+ /* Get head and tail of the td list and calculate
+ * the number of free elements.
+ */
+ head = le32_to_cpu(ipc_protocol->p_ap_shm->head_array[pipe->pipe_nr]);
+ tail = pipe->old_tail;
+
+ while (!skb_queue_empty(p_ul_list)) {
+ if (head < tail)
+ free_elements = tail - head - 1;
+ else
+ free_elements =
+ pipe->nr_of_entries - head + ((s32)tail - 1);
+
+ if (free_elements <= 0) {
+ dev_dbg(ipc_protocol->dev,
+ "no free td elements for UL pipe %d",
+ pipe->pipe_nr);
+ break;
+ }
+
+ /* Get the td address. */
+ td = &pipe->tdr_start[head];
+
+ /* Take the first element of the uplink list and add it
+ * to the td list.
+ */
+ skb = skb_dequeue(p_ul_list);
+ if (WARN_ON(!skb))
+ break;
+
+ /* Save the reference to the uplink skbuf. */
+ pipe->skbr_start[head] = skb;
+
+ td->buffer.address = IPC_CB(skb)->mapping;
+ td->scs = cpu_to_le32(skb->len) & cpu_to_le32(SIZE_MASK);
+ td->next = 0;
+
+ pipe->nr_of_queued_entries++;
+
+ /* Calculate the new head and save it. */
+ head++;
+ if (head >= pipe->nr_of_entries)
+ head = 0;
+
+ ipc_protocol->p_ap_shm->head_array[pipe->pipe_nr] =
+ cpu_to_le32(head);
+ }
+
+ if (pipe->old_head != head) {
+ dev_dbg(ipc_protocol->dev, "New UL TDs Pipe:%d", pipe->pipe_nr);
+
+ pipe->old_head = head;
+ /* Trigger doorbell because of pending UL packets. */
+ hpda_pending = true;
+ }
+
+ return hpda_pending;
+}
+
+/* Checks for Tail pointer update from CP and returns the data as SKB. */
+struct sk_buff *ipc_protocol_ul_td_process(struct iosm_protocol *ipc_protocol,
+ struct ipc_pipe *pipe)
+{
+ struct ipc_protocol_td *p_td = &pipe->tdr_start[pipe->old_tail];
+ struct sk_buff *skb = pipe->skbr_start[pipe->old_tail];
+
+ pipe->nr_of_queued_entries--;
+ pipe->old_tail++;
+ if (pipe->old_tail >= pipe->nr_of_entries)
+ pipe->old_tail = 0;
+
+ if (!p_td->buffer.address) {
+ dev_err(ipc_protocol->dev, "Td buffer address is NULL");
+ return NULL;
+ }
+
+ if (p_td->buffer.address != IPC_CB(skb)->mapping) {
+ dev_err(ipc_protocol->dev,
+ "pipe %d: invalid buf_addr or skb_data",
+ pipe->pipe_nr);
+ return NULL;
+ }
+
+ return skb;
+}
+
+/* Allocates an SKB for CP to send data and updates the Head Pointer
+ * of the given Pipe#.
+ */
+bool ipc_protocol_dl_td_prepare(struct iosm_protocol *ipc_protocol,
+ struct ipc_pipe *pipe)
+{
+ struct ipc_protocol_td *td;
+ dma_addr_t mapping = 0;
+ u32 head, new_head;
+ struct sk_buff *skb;
+ u32 tail;
+
+ /* Get head and tail of the td list and calculate
+ * the number of free elements.
+ */
+ head = le32_to_cpu(ipc_protocol->p_ap_shm->head_array[pipe->pipe_nr]);
+ tail = le32_to_cpu(ipc_protocol->p_ap_shm->tail_array[pipe->pipe_nr]);
+
+ new_head = head + 1;
+ if (new_head >= pipe->nr_of_entries)
+ new_head = 0;
+
+ if (new_head == tail)
+ return false;
+
+ /* Get the td address. */
+ td = &pipe->tdr_start[head];
+
+ /* Allocate the skbuf for the descriptor. */
+ skb = ipc_pcie_alloc_skb(ipc_protocol->pcie, pipe->buf_size, GFP_ATOMIC,
+ &mapping, DMA_FROM_DEVICE,
+ IPC_MEM_DL_ETH_OFFSET);
+ if (!skb)
+ return false;
+
+ td->buffer.address = mapping;
+ td->scs = cpu_to_le32(pipe->buf_size) & cpu_to_le32(SIZE_MASK);
+ td->next = 0;
+
+ /* store the new head value. */
+ ipc_protocol->p_ap_shm->head_array[pipe->pipe_nr] =
+ cpu_to_le32(new_head);
+
+ /* Save the reference to the skbuf. */
+ pipe->skbr_start[head] = skb;
+
+ pipe->nr_of_queued_entries++;
+
+ return true;
+}
+
+/* Processes DL TD's */
+struct sk_buff *ipc_protocol_dl_td_process(struct iosm_protocol *ipc_protocol,
+ struct ipc_pipe *pipe)
+{
+ u32 tail =
+ le32_to_cpu(ipc_protocol->p_ap_shm->tail_array[pipe->pipe_nr]);
+ struct ipc_protocol_td *p_td;
+ struct sk_buff *skb;
+
+ if (!pipe->tdr_start)
+ return NULL;
+
+ /* Copy the reference to the downlink buffer. */
+ p_td = &pipe->tdr_start[pipe->old_tail];
+ skb = pipe->skbr_start[pipe->old_tail];
+
+ /* Reset the ring elements. */
+ pipe->skbr_start[pipe->old_tail] = NULL;
+
+ pipe->nr_of_queued_entries--;
+
+ pipe->old_tail++;
+ if (pipe->old_tail >= pipe->nr_of_entries)
+ pipe->old_tail = 0;
+
+ if (!skb) {
+ dev_err(ipc_protocol->dev, "skb is null");
+ goto ret;
+ } else if (!p_td->buffer.address) {
+ dev_err(ipc_protocol->dev, "td/buffer address is null");
+ ipc_pcie_kfree_skb(ipc_protocol->pcie, skb);
+ skb = NULL;
+ goto ret;
+ }
+
+ if (!IPC_CB(skb)) {
+ dev_err(ipc_protocol->dev, "pipe# %d, tail: %d skb_cb is NULL",
+ pipe->pipe_nr, tail);
+ ipc_pcie_kfree_skb(ipc_protocol->pcie, skb);
+ skb = NULL;
+ goto ret;
+ }
+
+ if (p_td->buffer.address != IPC_CB(skb)->mapping) {
+ dev_err(ipc_protocol->dev, "invalid buf=%p or skb=%p",
+ (void *)p_td->buffer.address, skb->data);
+ ipc_pcie_kfree_skb(ipc_protocol->pcie, skb);
+ skb = NULL;
+ goto ret;
+ } else if ((le32_to_cpu(p_td->scs) & SIZE_MASK) > pipe->buf_size) {
+ dev_err(ipc_protocol->dev, "invalid buffer size %d > %d",
+ le32_to_cpu(p_td->scs) & SIZE_MASK,
+ pipe->buf_size);
+ ipc_pcie_kfree_skb(ipc_protocol->pcie, skb);
+ skb = NULL;
+ goto ret;
+ } else if (le32_to_cpu(p_td->scs) >> COMPLETION_STATUS ==
+ IPC_MEM_TD_CS_ABORT) {
+ /* Discard aborted buffers. */
+ dev_dbg(ipc_protocol->dev, "discard 'aborted' buffers");
+ ipc_pcie_kfree_skb(ipc_protocol->pcie, skb);
+ skb = NULL;
+ goto ret;
+ }
+
+ /* Set the length field in skbuf. */
+ skb_put(skb, le32_to_cpu(p_td->scs) & SIZE_MASK);
+
+ret:
+ return skb;
+}
+
+void ipc_protocol_get_head_tail_index(struct iosm_protocol *ipc_protocol,
+ struct ipc_pipe *pipe, u32 *head,
+ u32 *tail)
+{
+ struct ipc_protocol_ap_shm *ipc_ap_shm = ipc_protocol->p_ap_shm;
+
+ if (head)
+ *head = le32_to_cpu(ipc_ap_shm->head_array[pipe->pipe_nr]);
+
+ if (tail)
+ *tail = le32_to_cpu(ipc_ap_shm->tail_array[pipe->pipe_nr]);
+}
+
+/* Frees the TDs given to CP. */
+void ipc_protocol_pipe_cleanup(struct iosm_protocol *ipc_protocol,
+ struct ipc_pipe *pipe)
+{
+ struct sk_buff *skb;
+ u32 head;
+ u32 tail;
+
+ /* Get the start and the end of the buffer list. */
+ head = le32_to_cpu(ipc_protocol->p_ap_shm->head_array[pipe->pipe_nr]);
+ tail = pipe->old_tail;
+
+ /* Reset tail and head to 0. */
+ ipc_protocol->p_ap_shm->tail_array[pipe->pipe_nr] = 0;
+ ipc_protocol->p_ap_shm->head_array[pipe->pipe_nr] = 0;
+
+ /* Free pending uplink and downlink buffers. */
+ if (pipe->skbr_start) {
+ while (head != tail) {
+ /* Get the reference to the skbuf,
+ * which is on the way and free it.
+ */
+ skb = pipe->skbr_start[tail];
+ if (skb)
+ ipc_pcie_kfree_skb(ipc_protocol->pcie, skb);
+
+ tail++;
+ if (tail >= pipe->nr_of_entries)
+ tail = 0;
+ }
+
+ kfree(pipe->skbr_start);
+ pipe->skbr_start = NULL;
+ }
+
+ pipe->old_tail = 0;
+
+ /* Free and reset the td and skbuf circular buffers. kfree is save! */
+ if (pipe->tdr_start) {
+ pci_free_consistent(ipc_protocol->pcie->pci,
+ sizeof(*pipe->tdr_start) *
+ pipe->nr_of_entries,
+ pipe->tdr_start, pipe->phy_tdr_start);
+
+ pipe->tdr_start = NULL;
+ }
+}
+
+enum ipc_mem_device_ipc_state ipc_protocol_get_ipc_status(struct iosm_protocol
+ *ipc_protocol)
+{
+ return (enum ipc_mem_device_ipc_state)
+ le32_to_cpu(ipc_protocol->p_ap_shm->device_info.ipc_status);
+}
+
+enum ipc_mem_exec_stage
+ipc_protocol_get_ap_exec_stage(struct iosm_protocol *ipc_protocol)
+{
+ return le32_to_cpu(ipc_protocol->p_ap_shm->device_info.execution_stage);
+}
+
+int ipc_protocol_msg_prep(struct iosm_imem *ipc_imem,
+ enum ipc_msg_prep_type msg_type,
+ union ipc_msg_prep_args *args)
+{
+ struct iosm_protocol *ipc_protocol = ipc_imem->ipc_protocol;
+
+ switch (msg_type) {
+ case IPC_MSG_PREP_SLEEP:
+ return ipc_protocol_msg_prep_sleep(ipc_protocol, args);
+
+ case IPC_MSG_PREP_PIPE_OPEN:
+ return ipc_protocol_msg_prepipe_open(ipc_protocol, args);
+
+ case IPC_MSG_PREP_PIPE_CLOSE:
+ return ipc_protocol_msg_prepipe_close(ipc_protocol, args);
+
+ case IPC_MSG_PREP_FEATURE_SET:
+ return ipc_protocol_msg_prep_feature_set(ipc_protocol, args);
+
+ /* Unsupported messages in protocol */
+ case IPC_MSG_PREP_MAP:
+ case IPC_MSG_PREP_UNMAP:
+ default:
+ dev_err(ipc_protocol->dev,
+ "unsupported message type: %d in protocol", msg_type);
+ return -EINVAL;
+ }
+}
+
+u32
+ipc_protocol_pm_dev_get_sleep_notification(struct iosm_protocol *ipc_protocol)
+{
+ struct ipc_protocol_ap_shm *ipc_ap_shm = ipc_protocol->p_ap_shm;
+
+ return le32_to_cpu(ipc_ap_shm->device_info.device_sleep_notification);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#ifndef IOSM_IPC_PROTOCOL_OPS_H
+#define IOSM_IPC_PROTOCOL_OPS_H
+
+#define SIZE_MASK 0x00FFFFFF
+#define COMPLETION_STATUS 24
+#define RESET_BIT 7
+
+/**
+ * enum ipc_mem_td_cs - Completion status of a TD
+ * @IPC_MEM_TD_CS_INVALID: Initial status - td not yet used.
+ * @IPC_MEM_TD_CS_PARTIAL_TRANSFER: More data pending -> next TD used for this
+ * @IPC_MEM_TD_CS_END_TRANSFER: IO transfer is complete.
+ * @IPC_MEM_TD_CS_OVERFLOW: IO transfer to small for the buff to write
+ * @IPC_MEM_TD_CS_ABORT: TD marked as abort and shall be discarded
+ * by AP.
+ * @IPC_MEM_TD_CS_ERROR: General error.
+ */
+enum ipc_mem_td_cs {
+ IPC_MEM_TD_CS_INVALID,
+ IPC_MEM_TD_CS_PARTIAL_TRANSFER,
+ IPC_MEM_TD_CS_END_TRANSFER,
+ IPC_MEM_TD_CS_OVERFLOW,
+ IPC_MEM_TD_CS_ABORT,
+ IPC_MEM_TD_CS_ERROR,
+};
+
+/**
+ * enum ipc_mem_msg_cs - Completion status of IPC Message
+ * @IPC_MEM_MSG_CS_INVALID: Initial status.
+ * @IPC_MEM_MSG_CS_SUCCESS: IPC Message completion success.
+ * @IPC_MEM_MSG_CS_ERROR: Message send error.
+ */
+enum ipc_mem_msg_cs {
+ IPC_MEM_MSG_CS_INVALID,
+ IPC_MEM_MSG_CS_SUCCESS,
+ IPC_MEM_MSG_CS_ERROR,
+};
+
+/**
+ * struct ipc_msg_prep_args_pipe - struct for pipe args for message preparation
+ * @pipe: Pipe to open/close
+ */
+struct ipc_msg_prep_args_pipe {
+ struct ipc_pipe *pipe;
+};
+
+/**
+ * struct ipc_msg_prep_args_sleep - struct for sleep args for message
+ * preparation
+ * @target: 0=host, 1=device
+ * @state: 0=enter sleep, 1=exit sleep
+ */
+struct ipc_msg_prep_args_sleep {
+ unsigned int target;
+ unsigned int state;
+};
+
+/**
+ * struct ipc_msg_prep_feature_set - struct for feature set argument for
+ * message preparation
+ * @reset_enable: 0=out-of-band, 1=in-band-crash notification
+ */
+struct ipc_msg_prep_feature_set {
+ u8 reset_enable;
+};
+
+/**
+ * struct ipc_msg_prep_map - struct for map argument for message preparation
+ * @region_id: Region to map
+ * @addr: Pcie addr of region to map
+ * @size: Size of the region to map
+ */
+struct ipc_msg_prep_map {
+ unsigned int region_id;
+ unsigned long addr;
+ size_t size;
+};
+
+/**
+ * struct ipc_msg_prep_unmap - struct for unmap argument for message preparation
+ * @region_id: Region to unmap
+ */
+struct ipc_msg_prep_unmap {
+ unsigned int region_id;
+};
+
+/**
+ * struct ipc_msg_prep_args - Union to handle different message types
+ * @pipe_open: Pipe open message preparation struct
+ * @pipe_close: Pipe close message preparation struct
+ * @sleep: Sleep message preparation struct
+ * @feature_set: Feature set message preparation struct
+ * @map: Memory map message preparation struct
+ * @unmap: Memory unmap message preparation struct
+ */
+union ipc_msg_prep_args {
+ struct ipc_msg_prep_args_pipe pipe_open;
+ struct ipc_msg_prep_args_pipe pipe_close;
+ struct ipc_msg_prep_args_sleep sleep;
+ struct ipc_msg_prep_feature_set feature_set;
+ struct ipc_msg_prep_map map;
+ struct ipc_msg_prep_unmap unmap;
+};
+
+/**
+ * enum ipc_msg_prep_type - Enum for message prepare actions
+ * @IPC_MSG_PREP_SLEEP: Sleep message preparation type
+ * @IPC_MSG_PREP_PIPE_OPEN: Pipe open message preparation type
+ * @IPC_MSG_PREP_PIPE_CLOSE: Pipe close message preparation type
+ * @IPC_MSG_PREP_FEATURE_SET: Feature set message preparation type
+ * @IPC_MSG_PREP_MAP: Memory map message preparation type
+ * @IPC_MSG_PREP_UNMAP: Memory unmap message preparation type
+ */
+enum ipc_msg_prep_type {
+ IPC_MSG_PREP_SLEEP,
+ IPC_MSG_PREP_PIPE_OPEN,
+ IPC_MSG_PREP_PIPE_CLOSE,
+ IPC_MSG_PREP_FEATURE_SET,
+ IPC_MSG_PREP_MAP,
+ IPC_MSG_PREP_UNMAP,
+};
+
+/**
+ * struct ipc_rsp - Response to sent message
+ * @completion: For waking up requestor
+ * @status: Completion status
+ */
+struct ipc_rsp {
+ struct completion completion;
+ enum ipc_mem_msg_cs status;
+};
+
+/**
+ * enum ipc_mem_msg - Type-definition of the messages.
+ * @IPC_MEM_MSG_OPEN_PIPE: AP ->CP: Open a pipe
+ * @IPC_MEM_MSG_CLOSE_PIPE: AP ->CP: Close a pipe
+ * @IPC_MEM_MSG_ABORT_PIPE: AP ->CP: wait for completion of the
+ * running transfer and abort all pending
+ * IO-transfers for the pipe
+ * @IPC_MEM_MSG_SLEEP: AP ->CP: host enter or exit sleep
+ * @IPC_MEM_MSG_FEATURE_SET: AP ->CP: Intel feature configuration
+ */
+enum ipc_mem_msg {
+ IPC_MEM_MSG_OPEN_PIPE = 0x01,
+ IPC_MEM_MSG_CLOSE_PIPE = 0x02,
+ IPC_MEM_MSG_ABORT_PIPE = 0x03,
+ IPC_MEM_MSG_SLEEP = 0x04,
+ IPC_MEM_MSG_FEATURE_SET = 0xF0,
+};
+
+/**
+ * struct ipc_mem_msg_open_pipe - Message structure for open pipe
+ * @tdr_addr: Tdr address
+ * @tdr_entries: Tdr entries
+ * @pipe_nr: Pipe number
+ * @type_of_message: Message type
+ * @irq_vector: MSI vector number
+ * @accumulation_backoff: Time in usec for data accumalation
+ * @completion_status: Message Completion Status
+ */
+struct ipc_mem_msg_open_pipe {
+ __le64 tdr_addr;
+ __le16 tdr_entries;
+ u8 pipe_nr;
+ u8 type_of_message;
+ __le32 irq_vector;
+ __le32 accumulation_backoff;
+ __le32 completion_status;
+};
+
+/**
+ * struct ipc_mem_msg_close_pipe - Message structure for close pipe
+ * @reserved1: Reserved
+ * @reserved2: Reserved
+ * @pipe_nr: Pipe number
+ * @type_of_message: Message type
+ * @reserved3: Reserved
+ * @reserved4: Reserved
+ * @completion_status: Message Completion Status
+ */
+struct ipc_mem_msg_close_pipe {
+ __le32 reserved1[2];
+ __le16 reserved2;
+ u8 pipe_nr;
+ u8 type_of_message;
+ __le32 reserved3;
+ __le32 reserved4;
+ __le32 completion_status;
+};
+
+/**
+ * struct ipc_mem_msg_abort_pipe - Message structure for abort pipe
+ * @reserved1: Reserved
+ * @reserved2: Reserved
+ * @pipe_nr: Pipe number
+ * @type_of_message: Message type
+ * @reserved3: Reserved
+ * @reserved4: Reserved
+ * @completion_status: Message Completion Status
+ */
+struct ipc_mem_msg_abort_pipe {
+ __le32 reserved1[2];
+ __le16 reserved2;
+ u8 pipe_nr;
+ u8 type_of_message;
+ __le32 reserved3;
+ __le32 reserved4;
+ __le32 completion_status;
+};
+
+/**
+ * struct ipc_mem_msg_host_sleep - Message structure for sleep message.
+ * @reserved1: Reserved
+ * @target: 0=host, 1=device, host or EP devie
+ * is the message target
+ * @state: 0=enter sleep, 1=exit sleep,
+ * 2=enter sleep no protocol
+ * @reserved2: Reserved
+ * @type_of_message: Message type
+ * @reserved3: Reserved
+ * @reserved4: Reserved
+ * @completion_status: Message Completion Status
+ */
+struct ipc_mem_msg_host_sleep {
+ __le32 reserved1[2];
+ u8 target;
+ u8 state;
+ u8 reserved2;
+ u8 type_of_message;
+ __le32 reserved3;
+ __le32 reserved4;
+ __le32 completion_status;
+};
+
+/**
+ * struct ipc_mem_msg_feature_set - Message structure for feature_set message
+ * @reserved1: Reserved
+ * @reserved2: Reserved
+ * @reset_enable: 0=out-of-band, 1=in-band-crash notification
+ * @type_of_message: Message type
+ * @reserved3: Reserved
+ * @reserved4: Reserved
+ * @completion_status: Message Completion Status
+ */
+struct ipc_mem_msg_feature_set {
+ __le32 reserved1[2];
+ __le16 reserved2;
+ u8 reset_enable;
+ u8 type_of_message;
+ __le32 reserved3;
+ __le32 reserved4;
+ __le32 completion_status;
+};
+
+/**
+ * struct ipc_mem_msg_common - Message structure for completion status update.
+ * @reserved1: Reserved
+ * @reserved2: Reserved
+ * @type_of_message: Message type
+ * @reserved3: Reserved
+ * @reserved4: Reserved
+ * @completion_status: Message Completion Status
+ */
+struct ipc_mem_msg_common {
+ __le32 reserved1[2];
+ u8 reserved2[3];
+ u8 type_of_message;
+ __le32 reserved3;
+ __le32 reserved4;
+ __le32 completion_status;
+};
+
+/**
+ * union ipc_mem_msg_entry - Union with all possible messages.
+ * @open_pipe: Open pipe message struct
+ * @close_pipe: Close pipe message struct
+ * @abort_pipe: Abort pipe message struct
+ * @host_sleep: Host sleep message struct
+ * @feature_set: Featuer set message struct
+ * @common: Used to access msg_type and to set the completion status
+ */
+union ipc_mem_msg_entry {
+ struct ipc_mem_msg_open_pipe open_pipe;
+ struct ipc_mem_msg_close_pipe close_pipe;
+ struct ipc_mem_msg_abort_pipe abort_pipe;
+ struct ipc_mem_msg_host_sleep host_sleep;
+ struct ipc_mem_msg_feature_set feature_set;
+ struct ipc_mem_msg_common common;
+};
+
+/* Transfer descriptor definition. */
+struct ipc_protocol_td {
+ union {
+ /* 0 : 63 - 64-bit address of a buffer in host memory. */
+ dma_addr_t address;
+ struct {
+ /* 0 : 31 - 32 bit address */
+ __le32 address;
+ /* 32 : 63 - corresponding descriptor */
+ __le32 desc;
+ } __packed shm;
+ } buffer;
+
+ /* 0 - 2nd byte - Size of the buffer.
+ * The host provides the size of the buffer queued.
+ * The EP device reads this value and shall update
+ * it for downlink transfers to indicate the
+ * amount of data written in buffer.
+ * 3rd byte - This field provides the completion status
+ * of the TD. When queuing the TD, the host sets
+ * the status to 0. The EP device updates this
+ * field when completing the TD.
+ */
+ __le32 scs;
+
+ /* 0th - nr of following descriptors
+ * 1 - 3rd byte - reserved
+ */
+ __le32 next;
+} __packed;
+
+/**
+ * ipc_protocol_msg_prep - Prepare message based upon message type
+ * @ipc_imem: iosm_protocol instance
+ * @msg_type: message prepare type
+ * @args: message arguments
+ *
+ * Return: 0 on success and failure value on error
+ */
+int ipc_protocol_msg_prep(struct iosm_imem *ipc_imem,
+ enum ipc_msg_prep_type msg_type,
+ union ipc_msg_prep_args *args);
+
+/**
+ * ipc_protocol_msg_hp_update - Function for head pointer update
+ * of message ring
+ * @ipc_imem: iosm_protocol instance
+ */
+void ipc_protocol_msg_hp_update(struct iosm_imem *ipc_imem);
+
+/**
+ * ipc_protocol_msg_process - Function for processing responses
+ * to IPC messages
+ * @ipc_imem: iosm_protocol instance
+ * @irq: IRQ vector
+ *
+ * Return: True on success, false if error
+ */
+bool ipc_protocol_msg_process(struct iosm_imem *ipc_imem, int irq);
+
+/**
+ * ipc_protocol_ul_td_send - Function for sending the data to CP
+ * @ipc_protocol: iosm_protocol instance
+ * @pipe: Pipe instance
+ * @p_ul_list: uplink sk_buff list
+ *
+ * Return: true in success, false in case of error
+ */
+bool ipc_protocol_ul_td_send(struct iosm_protocol *ipc_protocol,
+ struct ipc_pipe *pipe,
+ struct sk_buff_head *p_ul_list);
+
+/**
+ * ipc_protocol_ul_td_process - Function for processing the sent data
+ * @ipc_protocol: iosm_protocol instance
+ * @pipe: Pipe instance
+ *
+ * Return: sk_buff instance
+ */
+struct sk_buff *ipc_protocol_ul_td_process(struct iosm_protocol *ipc_protocol,
+ struct ipc_pipe *pipe);
+
+/**
+ * ipc_protocol_dl_td_prepare - Function for providing DL TDs to CP
+ * @ipc_protocol: iosm_protocol instance
+ * @pipe: Pipe instance
+ *
+ * Return: true in success, false in case of error
+ */
+bool ipc_protocol_dl_td_prepare(struct iosm_protocol *ipc_protocol,
+ struct ipc_pipe *pipe);
+
+/**
+ * ipc_protocol_dl_td_process - Function for processing the DL data
+ * @ipc_protocol: iosm_protocol instance
+ * @pipe: Pipe instance
+ *
+ * Return: sk_buff instance
+ */
+struct sk_buff *ipc_protocol_dl_td_process(struct iosm_protocol *ipc_protocol,
+ struct ipc_pipe *pipe);
+
+/**
+ * ipc_protocol_get_head_tail_index - Function for getting Head and Tail
+ * pointer index of given pipe
+ * @ipc_protocol: iosm_protocol instance
+ * @pipe: Pipe Instance
+ * @head: head pointer index of the given pipe
+ * @tail: tail pointer index of the given pipe
+ */
+void ipc_protocol_get_head_tail_index(struct iosm_protocol *ipc_protocol,
+ struct ipc_pipe *pipe, u32 *head,
+ u32 *tail);
+/**
+ * ipc_protocol_get_ipc_status - Function for getting the IPC Status
+ * @ipc_protocol: iosm_protocol instance
+ *
+ * Return: Returns IPC State
+ */
+enum ipc_mem_device_ipc_state ipc_protocol_get_ipc_status(struct iosm_protocol
+ *ipc_protocol);
+
+/**
+ * ipc_protocol_pipe_cleanup - Function to cleanup pipe resources
+ * @ipc_protocol: iosm_protocol instance
+ * @pipe: Pipe instance
+ */
+void ipc_protocol_pipe_cleanup(struct iosm_protocol *ipc_protocol,
+ struct ipc_pipe *pipe);
+
+/**
+ * ipc_protocol_get_ap_exec_stage - Function for getting AP Exec Stage
+ * @ipc_protocol: pointer to struct iosm protocol
+ *
+ * Return: returns BOOT Stages
+ */
+enum ipc_mem_exec_stage
+ipc_protocol_get_ap_exec_stage(struct iosm_protocol *ipc_protocol);
+
+/**
+ * ipc_protocol_pm_dev_get_sleep_notification - Function for getting Dev Sleep
+ * notification
+ * @ipc_protocol: iosm_protocol instance
+ *
+ * Return: Returns dev PM State
+ */
+u32 ipc_protocol_pm_dev_get_sleep_notification(struct iosm_protocol
+ *ipc_protocol);
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include "iosm_ipc_imem.h"
+#include "iosm_ipc_task_queue.h"
+
+/* Actual tasklet function, will be called whenever tasklet is scheduled.
+ * Calls event handler involves callback for each element in the message queue
+ */
+static void ipc_task_queue_handler(unsigned long data)
+{
+ struct ipc_task_queue *ipc_task = (struct ipc_task_queue *)data;
+ unsigned int q_rpos = ipc_task->q_rpos;
+
+ /* Loop over the input queue contents. */
+ while (q_rpos != ipc_task->q_wpos) {
+ /* Get the current first queue element. */
+ struct ipc_task_queue_args *args = &ipc_task->args[q_rpos];
+
+ /* Process the input message. */
+ if (args->func)
+ args->response = args->func(args->ipc_imem, args->arg,
+ args->msg, args->size);
+
+ /* Signal completion for synchronous calls */
+ if (args->completion)
+ complete(args->completion);
+
+ /* Free message if copy was allocated. */
+ if (args->is_copy)
+ kfree(args->msg);
+
+ /* Set invalid queue element. Technically
+ * spin_lock_irqsave is not required here as
+ * the array element has been processed already
+ * so we can assume that immediately after processing
+ * ipc_task element, queue will not rotate again to
+ * ipc_task same element within such short time.
+ */
+ args->completion = NULL;
+ args->func = NULL;
+ args->msg = NULL;
+ args->size = 0;
+ args->is_copy = false;
+
+ /* calculate the new read ptr and update the volatile read
+ * ptr
+ */
+ q_rpos = (q_rpos + 1) % IPC_THREAD_QUEUE_SIZE;
+ ipc_task->q_rpos = q_rpos;
+ }
+}
+
+/* Free memory alloc and trigger completions left in the queue during dealloc */
+static void ipc_task_queue_cleanup(struct ipc_task_queue *ipc_task)
+{
+ unsigned int q_rpos = ipc_task->q_rpos;
+
+ while (q_rpos != ipc_task->q_wpos) {
+ struct ipc_task_queue_args *args = &ipc_task->args[q_rpos];
+
+ if (args->completion)
+ complete(args->completion);
+
+ if (args->is_copy)
+ kfree(args->msg);
+
+ q_rpos = (q_rpos + 1) % IPC_THREAD_QUEUE_SIZE;
+ ipc_task->q_rpos = q_rpos;
+ }
+}
+
+/* Add a message to the queue and trigger the ipc_task. */
+static int
+ipc_task_queue_add_task(struct iosm_imem *ipc_imem,
+ int arg, void *msg,
+ int (*func)(struct iosm_imem *ipc_imem, int arg,
+ void *msg, size_t size),
+ size_t size, bool is_copy, bool wait)
+{
+ struct tasklet_struct *ipc_tasklet = ipc_imem->ipc_task->ipc_tasklet;
+ struct ipc_task_queue *ipc_task = &ipc_imem->ipc_task->ipc_queue;
+ struct completion completion;
+ unsigned int pos, nextpos;
+ unsigned long flags;
+ int result = -EIO;
+
+ init_completion(&completion);
+
+ /* tasklet send may be called from both interrupt or thread
+ * context, therefore protect queue operation by spinlock
+ */
+ spin_lock_irqsave(&ipc_task->q_lock, flags);
+
+ pos = ipc_task->q_wpos;
+ nextpos = (pos + 1) % IPC_THREAD_QUEUE_SIZE;
+
+ /* Get next queue position. */
+ if (nextpos != ipc_task->q_rpos) {
+ /* Get the reference to the queue element and save the passed
+ * values.
+ */
+ ipc_task->args[pos].arg = arg;
+ ipc_task->args[pos].msg = msg;
+ ipc_task->args[pos].func = func;
+ ipc_task->args[pos].ipc_imem = ipc_imem;
+ ipc_task->args[pos].size = size;
+ ipc_task->args[pos].is_copy = is_copy;
+ ipc_task->args[pos].completion = wait ? &completion : NULL;
+ ipc_task->args[pos].response = -1;
+
+ /* apply write barrier so that ipc_task->q_rpos elements
+ * are updated before ipc_task->q_wpos is being updated.
+ */
+ smp_wmb();
+
+ /* Update the status of the free queue space. */
+ ipc_task->q_wpos = nextpos;
+ result = 0;
+ }
+
+ spin_unlock_irqrestore(&ipc_task->q_lock, flags);
+
+ if (result == 0) {
+ tasklet_schedule(ipc_tasklet);
+
+ if (wait) {
+ wait_for_completion(&completion);
+ result = ipc_task->args[pos].response;
+ }
+ } else {
+ dev_err(ipc_imem->ipc_task->dev, "queue is full");
+ }
+
+ return result;
+}
+
+int ipc_task_queue_send_task(struct iosm_imem *imem,
+ int (*func)(struct iosm_imem *ipc_imem, int arg,
+ void *msg, size_t size),
+ int arg, void *msg, size_t size, bool wait)
+{
+ bool is_copy = false;
+ void *copy = msg;
+ int ret = -ENOMEM;
+
+ if (size > 0) {
+ copy = kmemdup(msg, size, GFP_ATOMIC);
+ if (!copy)
+ goto out;
+
+ is_copy = true;
+ }
+
+ ret = ipc_task_queue_add_task(imem, arg, copy, func,
+ size, is_copy, wait);
+ if (ret < 0) {
+ dev_err(imem->ipc_task->dev,
+ "add task failed for %ps %d, %p, %zu, %d", func, arg,
+ copy, size, is_copy);
+ if (is_copy)
+ kfree(copy);
+ goto out;
+ }
+
+ ret = 0;
+out:
+ return ret;
+}
+
+int ipc_task_init(struct ipc_task *ipc_task)
+{
+ struct ipc_task_queue *ipc_queue = &ipc_task->ipc_queue;
+
+ ipc_task->ipc_tasklet = kzalloc(sizeof(*ipc_task->ipc_tasklet),
+ GFP_KERNEL);
+
+ if (!ipc_task->ipc_tasklet)
+ return -ENOMEM;
+
+ /* Initialize the spinlock needed to protect the message queue of the
+ * ipc_task
+ */
+ spin_lock_init(&ipc_queue->q_lock);
+
+ tasklet_init(ipc_task->ipc_tasklet, ipc_task_queue_handler,
+ (unsigned long)ipc_queue);
+ return 0;
+}
+
+void ipc_task_deinit(struct ipc_task *ipc_task)
+{
+ tasklet_kill(ipc_task->ipc_tasklet);
+
+ kfree(ipc_task->ipc_tasklet);
+ /* This will free/complete any outstanding messages,
+ * without calling the actual handler
+ */
+ ipc_task_queue_cleanup(&ipc_task->ipc_queue);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#ifndef IOSM_IPC_TASK_QUEUE_H
+#define IOSM_IPC_TASK_QUEUE_H
+
+/* Number of available element for the input message queue of the IPC
+ * ipc_task
+ */
+#define IPC_THREAD_QUEUE_SIZE 256
+
+/**
+ * struct ipc_task_queue_args - Struct for Task queue elements
+ * @ipc_imem: Pointer to struct iosm_imem
+ * @msg: Message argument for tasklet function. (optional, can be NULL)
+ * @completion: OS object used to wait for the tasklet function to finish for
+ * synchronous calls
+ * @func: Function to be called in tasklet (tl) context
+ * @arg: Generic integer argument for tasklet function (optional)
+ * @size: Message size argument for tasklet function (optional)
+ * @response: Return code of tasklet function for synchronous calls
+ * @is_copy: Is true if msg contains a pointer to a copy of the original msg
+ * for async. calls that needs to be freed once the tasklet returns
+ */
+struct ipc_task_queue_args {
+ struct iosm_imem *ipc_imem;
+ void *msg;
+ struct completion *completion;
+ int (*func)(struct iosm_imem *ipc_imem, int arg, void *msg,
+ size_t size);
+ int arg;
+ size_t size;
+ int response;
+ u8 is_copy:1;
+};
+
+/**
+ * struct ipc_task_queue - Struct for Task queue
+ * @q_lock: Protect the message queue of the ipc ipc_task
+ * @args: Message queue of the IPC ipc_task
+ * @q_rpos: First queue element to process.
+ * @q_wpos: First free element of the input queue.
+ */
+struct ipc_task_queue {
+ spinlock_t q_lock; /* for atomic operation on queue */
+ struct ipc_task_queue_args args[IPC_THREAD_QUEUE_SIZE];
+ unsigned int q_rpos;
+ unsigned int q_wpos;
+};
+
+/**
+ * struct ipc_task - Struct for Task
+ * @dev: Pointer to device structure
+ * @ipc_tasklet: Tasklet for serialized work offload
+ * from interrupts and OS callbacks
+ * @ipc_queue: Task for entry into ipc task queue
+ */
+struct ipc_task {
+ struct device *dev;
+ struct tasklet_struct *ipc_tasklet;
+ struct ipc_task_queue ipc_queue;
+};
+
+/**
+ * ipc_task_init - Allocate a tasklet
+ * @ipc_task: Pointer to ipc_task structure
+ * Returns: 0 on success and failure value on error.
+ */
+int ipc_task_init(struct ipc_task *ipc_task);
+
+/**
+ * ipc_task_deinit - Free a tasklet, invalidating its pointer.
+ * @ipc_task: Pointer to ipc_task structure
+ */
+void ipc_task_deinit(struct ipc_task *ipc_task);
+
+/**
+ * ipc_task_queue_send_task - Synchronously/Asynchronously call a function in
+ * tasklet context.
+ * @imem: Pointer to iosm_imem struct
+ * @func: Function to be called in tasklet context
+ * @arg: Integer argument for func
+ * @msg: Message pointer argument for func
+ * @size: Size argument for func
+ * @wait: if true wait for result
+ *
+ * Returns: Result value returned by func or failure value if func could not
+ * be called.
+ */
+int ipc_task_queue_send_task(struct iosm_imem *imem,
+ int (*func)(struct iosm_imem *ipc_imem, int arg,
+ void *msg, size_t size),
+ int arg, void *msg, size_t size, bool wait);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include <linux/device.h>
+#include <linux/kobject.h>
+#include <linux/slab.h>
+
+#include "iosm_ipc_uevent.h"
+
+/* Update the uevent in work queue context */
+static void ipc_uevent_work(struct work_struct *data)
+{
+ struct ipc_uevent_info *info;
+ char *envp[2] = { NULL, NULL };
+
+ info = container_of(data, struct ipc_uevent_info, work);
+
+ envp[0] = info->uevent;
+
+ if (kobject_uevent_env(&info->dev->kobj, KOBJ_CHANGE, envp))
+ pr_err("uevent %s failed to sent", info->uevent);
+
+ kfree(info);
+}
+
+void ipc_uevent_send(struct device *dev, char *uevent)
+{
+ struct ipc_uevent_info *info = kzalloc(sizeof(*info), GFP_ATOMIC);
+
+ if (!info)
+ return;
+
+ /* Initialize the kernel work queue */
+ INIT_WORK(&info->work, ipc_uevent_work);
+
+ /* Store the device and event information */
+ info->dev = dev;
+ snprintf(info->uevent, MAX_UEVENT_LEN, "%s: %s", dev_name(dev), uevent);
+
+ /* Schedule uevent in process context using work queue */
+ schedule_work(&info->work);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#ifndef IOSM_IPC_UEVENT_H
+#define IOSM_IPC_UEVENT_H
+
+/* Baseband event strings */
+#define UEVENT_MDM_NOT_READY "MDM_NOT_READY"
+#define UEVENT_ROM_READY "ROM_READY"
+#define UEVENT_MDM_READY "MDM_READY"
+#define UEVENT_CRASH "CRASH"
+#define UEVENT_CD_READY "CD_READY"
+#define UEVENT_CD_READY_LINK_DOWN "CD_READY_LINK_DOWN"
+#define UEVENT_MDM_TIMEOUT "MDM_TIMEOUT"
+
+/* Maximum length of user events */
+#define MAX_UEVENT_LEN 64
+
+/**
+ * struct ipc_uevent_info - Uevent information structure.
+ * @dev: Pointer to device structure
+ * @uevent: Uevent information
+ * @work: Uevent work struct
+ */
+struct ipc_uevent_info {
+ struct device *dev;
+ char uevent[MAX_UEVENT_LEN];
+ struct work_struct work;
+};
+
+/**
+ * ipc_uevent_send - Send modem event to user space.
+ * @dev: Generic device pointer
+ * @uevent: Uevent information
+ *
+ */
+void ipc_uevent_send(struct device *dev, char *uevent);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#include <linux/etherdevice.h>
+#include <linux/if_arp.h>
+#include <linux/if_link.h>
+#include <linux/rtnetlink.h>
+#include <linux/wwan.h>
+
+#include "iosm_ipc_chnl_cfg.h"
+#include "iosm_ipc_imem_ops.h"
+#include "iosm_ipc_wwan.h"
+
+#define IOSM_IP_TYPE_MASK 0xF0
+#define IOSM_IP_TYPE_IPV4 0x40
+#define IOSM_IP_TYPE_IPV6 0x60
+
+#define IOSM_IF_ID_PAYLOAD 2
+
+/**
+ * struct iosm_netdev_priv - netdev WWAN driver specific private data
+ * @ipc_wwan: Pointer to iosm_wwan struct
+ * @netdev: Pointer to network interface device structure
+ * @if_id: Interface id for device.
+ * @ch_id: IPC channel number for which interface device is created.
+ */
+struct iosm_netdev_priv {
+ struct iosm_wwan *ipc_wwan;
+ struct net_device *netdev;
+ int if_id;
+ int ch_id;
+};
+
+/**
+ * struct iosm_wwan - This structure contains information about WWAN root device
+ * and interface to the IPC layer.
+ * @ipc_imem: Pointer to imem data-struct
+ * @sub_netlist: List of active netdevs
+ * @dev: Pointer device structure
+ * @if_mutex: Mutex used for add and remove interface id
+ */
+struct iosm_wwan {
+ struct iosm_imem *ipc_imem;
+ struct iosm_netdev_priv __rcu *sub_netlist[IP_MUX_SESSION_END + 1];
+ struct device *dev;
+ struct mutex if_mutex; /* Mutex used for add and remove interface id */
+};
+
+/* Bring-up the wwan net link */
+static int ipc_wwan_link_open(struct net_device *netdev)
+{
+ struct iosm_netdev_priv *priv = wwan_netdev_drvpriv(netdev);
+ struct iosm_wwan *ipc_wwan = priv->ipc_wwan;
+ int if_id = priv->if_id;
+ int ret;
+
+ if (if_id < IP_MUX_SESSION_START ||
+ if_id >= ARRAY_SIZE(ipc_wwan->sub_netlist))
+ return -EINVAL;
+
+ mutex_lock(&ipc_wwan->if_mutex);
+
+ /* get channel id */
+ priv->ch_id = ipc_imem_sys_wwan_open(ipc_wwan->ipc_imem, if_id);
+
+ if (priv->ch_id < 0) {
+ dev_err(ipc_wwan->dev,
+ "cannot connect wwan0 & id %d to the IPC mem layer",
+ if_id);
+ ret = -ENODEV;
+ goto out;
+ }
+
+ /* enable tx path, DL data may follow */
+ netif_start_queue(netdev);
+
+ dev_dbg(ipc_wwan->dev, "Channel id %d allocated to if_id %d",
+ priv->ch_id, priv->if_id);
+
+ ret = 0;
+out:
+ mutex_unlock(&ipc_wwan->if_mutex);
+ return ret;
+}
+
+/* Bring-down the wwan net link */
+static int ipc_wwan_link_stop(struct net_device *netdev)
+{
+ struct iosm_netdev_priv *priv = wwan_netdev_drvpriv(netdev);
+
+ netif_stop_queue(netdev);
+
+ mutex_lock(&priv->ipc_wwan->if_mutex);
+ ipc_imem_sys_wwan_close(priv->ipc_wwan->ipc_imem, priv->if_id,
+ priv->ch_id);
+ priv->ch_id = -1;
+ mutex_unlock(&priv->ipc_wwan->if_mutex);
+
+ return 0;
+}
+
+/* Transmit a packet */
+static int ipc_wwan_link_transmit(struct sk_buff *skb,
+ struct net_device *netdev)
+{
+ struct iosm_netdev_priv *priv = wwan_netdev_drvpriv(netdev);
+ struct iosm_wwan *ipc_wwan = priv->ipc_wwan;
+ int if_id = priv->if_id;
+ int ret;
+
+ /* Interface IDs from 1 to 8 are for IP data
+ * & from 257 to 261 are for non-IP data
+ */
+ if (if_id < IP_MUX_SESSION_START ||
+ if_id >= ARRAY_SIZE(ipc_wwan->sub_netlist))
+ return -EINVAL;
+
+ /* Send the SKB to device for transmission */
+ ret = ipc_imem_sys_wwan_transmit(ipc_wwan->ipc_imem,
+ if_id, priv->ch_id, skb);
+
+ /* Return code of zero is success */
+ if (ret == 0) {
+ ret = NETDEV_TX_OK;
+ } else if (ret == -EBUSY) {
+ ret = NETDEV_TX_BUSY;
+ dev_err(ipc_wwan->dev, "unable to push packets");
+ } else {
+ goto exit;
+ }
+
+ return ret;
+
+exit:
+ /* Log any skb drop */
+ if (if_id)
+ dev_dbg(ipc_wwan->dev, "skb dropped. IF_ID: %d, ret: %d", if_id,
+ ret);
+
+ dev_kfree_skb_any(skb);
+ return ret;
+}
+
+/* Ops structure for wwan net link */
+static const struct net_device_ops ipc_inm_ops = {
+ .ndo_open = ipc_wwan_link_open,
+ .ndo_stop = ipc_wwan_link_stop,
+ .ndo_start_xmit = ipc_wwan_link_transmit,
+};
+
+/* Setup function for creating new net link */
+static void ipc_wwan_setup(struct net_device *iosm_dev)
+{
+ iosm_dev->header_ops = NULL;
+ iosm_dev->hard_header_len = 0;
+ iosm_dev->priv_flags |= IFF_NO_QUEUE;
+
+ iosm_dev->type = ARPHRD_NONE;
+ iosm_dev->min_mtu = ETH_MIN_MTU;
+ iosm_dev->max_mtu = ETH_MAX_MTU;
+
+ iosm_dev->flags = IFF_POINTOPOINT | IFF_NOARP;
+
+ iosm_dev->netdev_ops = &ipc_inm_ops;
+}
+
+/* Create new wwan net link */
+static int ipc_wwan_newlink(void *ctxt, struct net_device *dev,
+ u32 if_id, struct netlink_ext_ack *extack)
+{
+ struct iosm_wwan *ipc_wwan = ctxt;
+ struct iosm_netdev_priv *priv;
+ int err;
+
+ if (if_id < IP_MUX_SESSION_START ||
+ if_id >= ARRAY_SIZE(ipc_wwan->sub_netlist))
+ return -EINVAL;
+
+ priv = wwan_netdev_drvpriv(dev);
+ priv->if_id = if_id;
+ priv->netdev = dev;
+ priv->ipc_wwan = ipc_wwan;
+
+ mutex_lock(&ipc_wwan->if_mutex);
+ if (rcu_access_pointer(ipc_wwan->sub_netlist[if_id])) {
+ err = -EBUSY;
+ goto out_unlock;
+ }
+
+ err = register_netdevice(dev);
+ if (err)
+ goto out_unlock;
+
+ rcu_assign_pointer(ipc_wwan->sub_netlist[if_id], priv);
+ mutex_unlock(&ipc_wwan->if_mutex);
+
+ netif_device_attach(dev);
+
+ return 0;
+
+out_unlock:
+ mutex_unlock(&ipc_wwan->if_mutex);
+ return err;
+}
+
+static void ipc_wwan_dellink(void *ctxt, struct net_device *dev,
+ struct list_head *head)
+{
+ struct iosm_netdev_priv *priv = wwan_netdev_drvpriv(dev);
+ struct iosm_wwan *ipc_wwan = ctxt;
+ int if_id = priv->if_id;
+
+ if (WARN_ON(if_id < IP_MUX_SESSION_START ||
+ if_id >= ARRAY_SIZE(ipc_wwan->sub_netlist)))
+ return;
+
+ mutex_lock(&ipc_wwan->if_mutex);
+
+ if (WARN_ON(rcu_access_pointer(ipc_wwan->sub_netlist[if_id]) != priv))
+ goto unlock;
+
+ RCU_INIT_POINTER(ipc_wwan->sub_netlist[if_id], NULL);
+ /* unregistering includes synchronize_net() */
+ unregister_netdevice(dev);
+
+unlock:
+ mutex_unlock(&ipc_wwan->if_mutex);
+}
+
+static const struct wwan_ops iosm_wwan_ops = {
+ .priv_size = sizeof(struct iosm_netdev_priv),
+ .setup = ipc_wwan_setup,
+ .newlink = ipc_wwan_newlink,
+ .dellink = ipc_wwan_dellink,
+};
+
+int ipc_wwan_receive(struct iosm_wwan *ipc_wwan, struct sk_buff *skb_arg,
+ bool dss, int if_id)
+{
+ struct sk_buff *skb = skb_arg;
+ struct net_device_stats *stats;
+ struct iosm_netdev_priv *priv;
+ int ret;
+
+ if ((skb->data[0] & IOSM_IP_TYPE_MASK) == IOSM_IP_TYPE_IPV4)
+ skb->protocol = htons(ETH_P_IP);
+ else if ((skb->data[0] & IOSM_IP_TYPE_MASK) ==
+ IOSM_IP_TYPE_IPV6)
+ skb->protocol = htons(ETH_P_IPV6);
+
+ skb->pkt_type = PACKET_HOST;
+
+ if (if_id < (IP_MUX_SESSION_START - 1) ||
+ if_id > (IP_MUX_SESSION_END - 1)) {
+ ret = -EINVAL;
+ goto free;
+ }
+
+ rcu_read_lock();
+ priv = rcu_dereference(ipc_wwan->sub_netlist[if_id]);
+ if (!priv) {
+ ret = -EINVAL;
+ goto unlock;
+ }
+ skb->dev = priv->netdev;
+ stats = &priv->netdev->stats;
+ stats->rx_packets++;
+ stats->rx_bytes += skb->len;
+
+ ret = netif_rx(skb);
+ skb = NULL;
+unlock:
+ rcu_read_unlock();
+free:
+ dev_kfree_skb(skb);
+ return ret;
+}
+
+void ipc_wwan_tx_flowctrl(struct iosm_wwan *ipc_wwan, int if_id, bool on)
+{
+ struct net_device *netdev;
+ struct iosm_netdev_priv *priv;
+ bool is_tx_blk;
+
+ rcu_read_lock();
+ priv = rcu_dereference(ipc_wwan->sub_netlist[if_id]);
+ if (!priv) {
+ rcu_read_unlock();
+ return;
+ }
+
+ netdev = priv->netdev;
+
+ is_tx_blk = netif_queue_stopped(netdev);
+
+ if (on)
+ dev_dbg(ipc_wwan->dev, "session id[%d]: flowctrl enable",
+ if_id);
+
+ if (on && !is_tx_blk)
+ netif_stop_queue(netdev);
+ else if (!on && is_tx_blk)
+ netif_wake_queue(netdev);
+ rcu_read_unlock();
+}
+
+struct iosm_wwan *ipc_wwan_init(struct iosm_imem *ipc_imem, struct device *dev)
+{
+ struct iosm_wwan *ipc_wwan;
+
+ ipc_wwan = kzalloc(sizeof(*ipc_wwan), GFP_KERNEL);
+ if (!ipc_wwan)
+ return NULL;
+
+ ipc_wwan->dev = dev;
+ ipc_wwan->ipc_imem = ipc_imem;
+
+ /* WWAN core will create a netdev for the default IP MUX channel */
+ if (wwan_register_ops(ipc_wwan->dev, &iosm_wwan_ops, ipc_wwan,
+ IP_MUX_SESSION_DEFAULT)) {
+ kfree(ipc_wwan);
+ return NULL;
+ }
+
+ mutex_init(&ipc_wwan->if_mutex);
+
+ return ipc_wwan;
+}
+
+void ipc_wwan_deinit(struct iosm_wwan *ipc_wwan)
+{
+ /* This call will remove all child netdev(s) */
+ wwan_unregister_ops(ipc_wwan->dev);
+
+ mutex_destroy(&ipc_wwan->if_mutex);
+
+ kfree(ipc_wwan);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (C) 2020-21 Intel Corporation.
+ */
+
+#ifndef IOSM_IPC_WWAN_H
+#define IOSM_IPC_WWAN_H
+
+/**
+ * ipc_wwan_init - Allocate, Init and register WWAN device
+ * @ipc_imem: Pointer to imem data-struct
+ * @dev: Pointer to device structure
+ *
+ * Returns: Pointer to instance on success else NULL
+ */
+struct iosm_wwan *ipc_wwan_init(struct iosm_imem *ipc_imem, struct device *dev);
+
+/**
+ * ipc_wwan_deinit - Unregister and free WWAN device, clear pointer
+ * @ipc_wwan: Pointer to wwan instance data
+ */
+void ipc_wwan_deinit(struct iosm_wwan *ipc_wwan);
+
+/**
+ * ipc_wwan_receive - Receive a downlink packet from CP.
+ * @ipc_wwan: Pointer to wwan instance
+ * @skb_arg: Pointer to struct sk_buff
+ * @dss: Set to true if interafce id is from 257 to 261,
+ * else false
+ * @if_id: Interface ID
+ *
+ * Return: 0 on success and failure value on error
+ */
+int ipc_wwan_receive(struct iosm_wwan *ipc_wwan, struct sk_buff *skb_arg,
+ bool dss, int if_id);
+
+/**
+ * ipc_wwan_tx_flowctrl - Enable/Disable TX flow control
+ * @ipc_wwan: Pointer to wwan instance
+ * @id: Ipc mux channel session id
+ * @on: if true then flow ctrl would be enabled else disable
+ *
+ */
+void ipc_wwan_tx_flowctrl(struct iosm_wwan *ipc_wwan, int id, bool on);
+
+/**
+ * ipc_wwan_is_tx_stopped - Checks if Tx stopped for a Interface id.
+ * @ipc_wwan: Pointer to wwan instance
+ * @id: Ipc mux channel session id
+ *
+ * Return: true if stopped, false otherwise
+ */
+bool ipc_wwan_is_tx_stopped(struct iosm_wwan *ipc_wwan, int id);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2021, Stephan Gerhold <stephan@gerhold.net> */
+#include <linux/kernel.h>
+#include <linux/mod_devicetable.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/rpmsg.h>
+#include <linux/wwan.h>
+
+struct rpmsg_wwan_dev {
+ /* Lower level is a rpmsg dev, upper level is a wwan port */
+ struct rpmsg_device *rpdev;
+ struct wwan_port *wwan_port;
+ struct rpmsg_endpoint *ept;
+};
+
+static int rpmsg_wwan_ctrl_callback(struct rpmsg_device *rpdev,
+ void *buf, int len, void *priv, u32 src)
+{
+ struct rpmsg_wwan_dev *rpwwan = priv;
+ struct sk_buff *skb;
+
+ skb = alloc_skb(len, GFP_ATOMIC);
+ if (!skb)
+ return -ENOMEM;
+
+ skb_put_data(skb, buf, len);
+ wwan_port_rx(rpwwan->wwan_port, skb);
+ return 0;
+}
+
+static int rpmsg_wwan_ctrl_start(struct wwan_port *port)
+{
+ struct rpmsg_wwan_dev *rpwwan = wwan_port_get_drvdata(port);
+ struct rpmsg_channel_info chinfo = {
+ .src = rpwwan->rpdev->src,
+ .dst = RPMSG_ADDR_ANY,
+ };
+
+ strncpy(chinfo.name, rpwwan->rpdev->id.name, RPMSG_NAME_SIZE);
+ rpwwan->ept = rpmsg_create_ept(rpwwan->rpdev, rpmsg_wwan_ctrl_callback,
+ rpwwan, chinfo);
+ if (!rpwwan->ept)
+ return -EREMOTEIO;
+
+ return 0;
+}
+
+static void rpmsg_wwan_ctrl_stop(struct wwan_port *port)
+{
+ struct rpmsg_wwan_dev *rpwwan = wwan_port_get_drvdata(port);
+
+ rpmsg_destroy_ept(rpwwan->ept);
+ rpwwan->ept = NULL;
+}
+
+static int rpmsg_wwan_ctrl_tx(struct wwan_port *port, struct sk_buff *skb)
+{
+ struct rpmsg_wwan_dev *rpwwan = wwan_port_get_drvdata(port);
+ int ret;
+
+ ret = rpmsg_trysend(rpwwan->ept, skb->data, skb->len);
+ if (ret)
+ return ret;
+
+ consume_skb(skb);
+ return 0;
+}
+
+static int rpmsg_wwan_ctrl_tx_blocking(struct wwan_port *port, struct sk_buff *skb)
+{
+ struct rpmsg_wwan_dev *rpwwan = wwan_port_get_drvdata(port);
+ int ret;
+
+ ret = rpmsg_send(rpwwan->ept, skb->data, skb->len);
+ if (ret)
+ return ret;
+
+ consume_skb(skb);
+ return 0;
+}
+
+static __poll_t rpmsg_wwan_ctrl_tx_poll(struct wwan_port *port,
+ struct file *filp, poll_table *wait)
+{
+ struct rpmsg_wwan_dev *rpwwan = wwan_port_get_drvdata(port);
+
+ return rpmsg_poll(rpwwan->ept, filp, wait);
+}
+
+static const struct wwan_port_ops rpmsg_wwan_pops = {
+ .start = rpmsg_wwan_ctrl_start,
+ .stop = rpmsg_wwan_ctrl_stop,
+ .tx = rpmsg_wwan_ctrl_tx,
+ .tx_blocking = rpmsg_wwan_ctrl_tx_blocking,
+ .tx_poll = rpmsg_wwan_ctrl_tx_poll,
+};
+
+static struct device *rpmsg_wwan_find_parent(struct device *dev)
+{
+ /* Select first platform device as parent for the WWAN ports.
+ * On Qualcomm platforms this is usually the platform device that
+ * represents the modem remote processor. This might need to be
+ * adjusted when adding device IDs for other platforms.
+ */
+ for (dev = dev->parent; dev; dev = dev->parent) {
+ if (dev_is_platform(dev))
+ return dev;
+ }
+ return NULL;
+}
+
+static int rpmsg_wwan_ctrl_probe(struct rpmsg_device *rpdev)
+{
+ struct rpmsg_wwan_dev *rpwwan;
+ struct wwan_port *port;
+ struct device *parent;
+
+ parent = rpmsg_wwan_find_parent(&rpdev->dev);
+ if (!parent)
+ return -ENODEV;
+
+ rpwwan = devm_kzalloc(&rpdev->dev, sizeof(*rpwwan), GFP_KERNEL);
+ if (!rpwwan)
+ return -ENOMEM;
+
+ rpwwan->rpdev = rpdev;
+ dev_set_drvdata(&rpdev->dev, rpwwan);
+
+ /* Register as a wwan port, id.driver_data contains wwan port type */
+ port = wwan_create_port(parent, rpdev->id.driver_data,
+ &rpmsg_wwan_pops, rpwwan);
+ if (IS_ERR(port))
+ return PTR_ERR(port);
+
+ rpwwan->wwan_port = port;
+
+ return 0;
+};
+
+static void rpmsg_wwan_ctrl_remove(struct rpmsg_device *rpdev)
+{
+ struct rpmsg_wwan_dev *rpwwan = dev_get_drvdata(&rpdev->dev);
+
+ wwan_remove_port(rpwwan->wwan_port);
+}
+
+static const struct rpmsg_device_id rpmsg_wwan_ctrl_id_table[] = {
+ /* RPMSG channels for Qualcomm SoCs with integrated modem */
+ { .name = "DATA5_CNTL", .driver_data = WWAN_PORT_QMI },
+ { .name = "DATA4", .driver_data = WWAN_PORT_AT },
+ {},
+};
+MODULE_DEVICE_TABLE(rpmsg, rpmsg_wwan_ctrl_id_table);
+
+static struct rpmsg_driver rpmsg_wwan_ctrl_driver = {
+ .drv.name = "rpmsg_wwan_ctrl",
+ .id_table = rpmsg_wwan_ctrl_id_table,
+ .probe = rpmsg_wwan_ctrl_probe,
+ .remove = rpmsg_wwan_ctrl_remove,
+};
+module_rpmsg_driver(rpmsg_wwan_ctrl_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("RPMSG WWAN CTRL Driver");
+MODULE_AUTHOR("Stephan Gerhold <stephan@gerhold.net>");
#include <linux/skbuff.h>
#include <linux/slab.h>
#include <linux/types.h>
+#include <linux/termios.h>
#include <linux/wwan.h>
+#include <net/rtnetlink.h>
+#include <uapi/linux/wwan.h>
-#define WWAN_MAX_MINORS 256 /* 256 minors allowed with register_chrdev() */
+/* Maximum number of minors in use */
+#define WWAN_MAX_MINORS (1 << MINORBITS)
static DEFINE_MUTEX(wwan_register_lock); /* WWAN device create|remove lock */
static DEFINE_IDA(minors); /* minors for WWAN port chardevs */
* @id: WWAN device unique ID.
* @dev: Underlying device.
* @port_id: Current available port ID to pick.
+ * @ops: wwan device ops
+ * @ops_ctxt: context to pass to ops
*/
struct wwan_device {
unsigned int id;
struct device dev;
atomic_t port_id;
+ const struct wwan_ops *ops;
+ void *ops_ctxt;
};
/**
* @dev: Underlying device
* @rxq: Buffer inbound queue
* @waitqueue: The waitqueue for port fops (read/write/poll)
+ * @data_lock: Port specific data access serialization
+ * @at_data: AT port specific data
*/
struct wwan_port {
enum wwan_port_type type;
struct device dev;
struct sk_buff_head rxq;
wait_queue_head_t waitqueue;
+ struct mutex data_lock; /* Port specific data access serialization */
+ union {
+ struct {
+ struct ktermios termios;
+ int mdmbits;
+ } at_data;
+ };
};
+static ssize_t index_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct wwan_device *wwan = to_wwan_dev(dev);
+
+ return sprintf(buf, "%d\n", wwan->id);
+}
+static DEVICE_ATTR_RO(index);
+
+static struct attribute *wwan_dev_attrs[] = {
+ &dev_attr_index.attr,
+ NULL,
+};
+ATTRIBUTE_GROUPS(wwan_dev);
+
static void wwan_dev_destroy(struct device *dev)
{
struct wwan_device *wwandev = to_wwan_dev(dev);
static const struct device_type wwan_dev_type = {
.name = "wwan_dev",
.release = wwan_dev_destroy,
+ .groups = wwan_dev_groups,
};
static int wwan_dev_parent_match(struct device *dev, const void *parent)
{
- return (dev->type == &wwan_dev_type && dev->parent == parent);
+ return (dev->type == &wwan_dev_type &&
+ (dev->parent == parent || dev == parent));
}
static struct wwan_device *wwan_dev_get_by_parent(struct device *parent)
return to_wwan_dev(dev);
}
+static int wwan_dev_name_match(struct device *dev, const void *name)
+{
+ return dev->type == &wwan_dev_type &&
+ strcmp(dev_name(dev), name) == 0;
+}
+
+static struct wwan_device *wwan_dev_get_by_name(const char *name)
+{
+ struct device *dev;
+
+ dev = class_find_device(wwan_class, NULL, name, wwan_dev_name_match);
+ if (!dev)
+ return ERR_PTR(-ENODEV);
+
+ return to_wwan_dev(dev);
+}
+
/* This function allocates and registers a new WWAN device OR if a WWAN device
* already exist for the given parent, it gets a reference and return it.
* This function is not exported (for now), it is called indirectly via
/* WWAN device is created and registered (get+add) along with its first
* child port, and subsequent port registrations only grab a reference
* (get). The WWAN device must then be unregistered (del+put) along with
- * its latest port, and reference simply dropped (put) otherwise.
+ * its last port, and reference simply dropped (put) otherwise. In the
+ * same fashion, we must not unregister it when the ops are still there.
*/
- ret = device_for_each_child(&wwandev->dev, NULL, is_wwan_child);
+ if (wwandev->ops)
+ ret = 1;
+ else
+ ret = device_for_each_child(&wwandev->dev, NULL, is_wwan_child);
+
if (!ret)
device_unregister(&wwandev->dev);
else
/* ------- WWAN port management ------- */
+static const struct {
+ const char * const name; /* Port type name */
+ const char * const devsuf; /* Port devce name suffix */
+} wwan_port_types[WWAN_PORT_MAX + 1] = {
+ [WWAN_PORT_AT] = {
+ .name = "AT",
+ .devsuf = "at",
+ },
+ [WWAN_PORT_MBIM] = {
+ .name = "MBIM",
+ .devsuf = "mbim",
+ },
+ [WWAN_PORT_QMI] = {
+ .name = "QMI",
+ .devsuf = "qmi",
+ },
+ [WWAN_PORT_QCDM] = {
+ .name = "QCDM",
+ .devsuf = "qcdm",
+ },
+ [WWAN_PORT_FIREHOSE] = {
+ .name = "FIREHOSE",
+ .devsuf = "firehose",
+ },
+};
+
+static ssize_t type_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct wwan_port *port = to_wwan_port(dev);
+
+ return sprintf(buf, "%s\n", wwan_port_types[port->type].name);
+}
+static DEVICE_ATTR_RO(type);
+
+static struct attribute *wwan_port_attrs[] = {
+ &dev_attr_type.attr,
+ NULL,
+};
+ATTRIBUTE_GROUPS(wwan_port);
+
static void wwan_port_destroy(struct device *dev)
{
struct wwan_port *port = to_wwan_port(dev);
ida_free(&minors, MINOR(port->dev.devt));
- skb_queue_purge(&port->rxq);
+ mutex_destroy(&port->data_lock);
mutex_destroy(&port->ops_lock);
kfree(port);
}
static const struct device_type wwan_port_dev_type = {
.name = "wwan_port",
.release = wwan_port_destroy,
+ .groups = wwan_port_groups,
};
static int wwan_port_minor_match(struct device *dev, const void *minor)
return to_wwan_port(dev);
}
-/* Keep aligned with wwan_port_type enum */
-static const char * const wwan_port_type_str[] = {
- "AT",
- "MBIM",
- "QMI",
- "QCDM",
- "FIREHOSE"
-};
+/* Allocate and set unique name based on passed format
+ *
+ * Name allocation approach is highly inspired by the __dev_alloc_name()
+ * function.
+ *
+ * To avoid names collision, the caller must prevent the new port device
+ * registration as well as concurrent invocation of this function.
+ */
+static int __wwan_port_dev_assign_name(struct wwan_port *port, const char *fmt)
+{
+ struct wwan_device *wwandev = to_wwan_dev(port->dev.parent);
+ const unsigned int max_ports = PAGE_SIZE * 8;
+ struct class_dev_iter iter;
+ unsigned long *idmap;
+ struct device *dev;
+ char buf[0x20];
+ int id;
+
+ idmap = (unsigned long *)get_zeroed_page(GFP_KERNEL);
+ if (!idmap)
+ return -ENOMEM;
+
+ /* Collect ids of same name format ports */
+ class_dev_iter_init(&iter, wwan_class, NULL, &wwan_port_dev_type);
+ while ((dev = class_dev_iter_next(&iter))) {
+ if (dev->parent != &wwandev->dev)
+ continue;
+ if (sscanf(dev_name(dev), fmt, &id) != 1)
+ continue;
+ if (id < 0 || id >= max_ports)
+ continue;
+ set_bit(id, idmap);
+ }
+ class_dev_iter_exit(&iter);
+
+ /* Allocate unique id */
+ id = find_first_zero_bit(idmap, max_ports);
+ free_page((unsigned long)idmap);
+
+ snprintf(buf, sizeof(buf), fmt, id); /* Name generation */
+
+ dev = device_find_child_by_name(&wwandev->dev, buf);
+ if (dev) {
+ put_device(dev);
+ return -ENFILE;
+ }
+
+ return dev_set_name(&port->dev, buf);
+}
struct wwan_port *wwan_create_port(struct device *parent,
enum wwan_port_type type,
struct wwan_device *wwandev;
struct wwan_port *port;
int minor, err = -ENOMEM;
+ char namefmt[0x20];
- if (type >= WWAN_PORT_MAX || !ops)
+ if (type > WWAN_PORT_MAX || !ops)
return ERR_PTR(-EINVAL);
/* A port is always a child of a WWAN device, retrieve (allocate or
mutex_init(&port->ops_lock);
skb_queue_head_init(&port->rxq);
init_waitqueue_head(&port->waitqueue);
+ mutex_init(&port->data_lock);
port->dev.parent = &wwandev->dev;
port->dev.class = wwan_class;
port->dev.devt = MKDEV(wwan_major, minor);
dev_set_drvdata(&port->dev, drvdata);
- /* create unique name based on wwan device id, port index and type */
- dev_set_name(&port->dev, "wwan%up%u%s", wwandev->id,
- atomic_inc_return(&wwandev->port_id),
- wwan_port_type_str[port->type]);
+ /* allocate unique name based on wwan device id, port type and number */
+ snprintf(namefmt, sizeof(namefmt), "wwan%u%s%%d", wwandev->id,
+ wwan_port_types[port->type].devsuf);
+ /* Serialize ports registration */
+ mutex_lock(&wwan_register_lock);
+
+ __wwan_port_dev_assign_name(port, namefmt);
err = device_register(&port->dev);
+
+ mutex_unlock(&wwan_register_lock);
+
if (err)
goto error_put_device;
{
mutex_lock(&port->ops_lock);
port->start_count--;
- if (port->ops && !port->start_count)
- port->ops->stop(port);
+ if (!port->start_count) {
+ if (port->ops)
+ port->ops->stop(port);
+ skb_queue_purge(&port->rxq);
+ }
mutex_unlock(&port->ops_lock);
}
-static int wwan_port_op_tx(struct wwan_port *port, struct sk_buff *skb)
+static int wwan_port_op_tx(struct wwan_port *port, struct sk_buff *skb,
+ bool nonblock)
{
int ret;
goto out_unlock;
}
- ret = port->ops->tx(port, skb);
+ if (nonblock || !port->ops->tx_blocking)
+ ret = port->ops->tx(port, skb);
+ else
+ ret = port->ops->tx_blocking(port, skb);
out_unlock:
mutex_unlock(&port->ops_lock);
return -EFAULT;
}
- ret = wwan_port_op_tx(port, skb);
+ ret = wwan_port_op_tx(port, skb, !!(filp->f_flags & O_NONBLOCK));
if (ret) {
kfree_skb(skb);
return ret;
poll_wait(filp, &port->waitqueue, wait);
- if (!is_write_blocked(port))
+ mutex_lock(&port->ops_lock);
+ if (port->ops && port->ops->tx_poll)
+ mask |= port->ops->tx_poll(port, filp, wait);
+ else if (!is_write_blocked(port))
mask |= EPOLLOUT | EPOLLWRNORM;
if (!is_read_blocked(port))
mask |= EPOLLIN | EPOLLRDNORM;
if (!port->ops)
mask |= EPOLLHUP | EPOLLERR;
+ mutex_unlock(&port->ops_lock);
return mask;
}
+/* Implements minimalistic stub terminal IOCTLs support */
+static long wwan_port_fops_at_ioctl(struct wwan_port *port, unsigned int cmd,
+ unsigned long arg)
+{
+ int ret = 0;
+
+ mutex_lock(&port->data_lock);
+
+ switch (cmd) {
+ case TCFLSH:
+ break;
+
+ case TCGETS:
+ if (copy_to_user((void __user *)arg, &port->at_data.termios,
+ sizeof(struct termios)))
+ ret = -EFAULT;
+ break;
+
+ case TCSETS:
+ case TCSETSW:
+ case TCSETSF:
+ if (copy_from_user(&port->at_data.termios, (void __user *)arg,
+ sizeof(struct termios)))
+ ret = -EFAULT;
+ break;
+
+#ifdef TCGETS2
+ case TCGETS2:
+ if (copy_to_user((void __user *)arg, &port->at_data.termios,
+ sizeof(struct termios2)))
+ ret = -EFAULT;
+ break;
+
+ case TCSETS2:
+ case TCSETSW2:
+ case TCSETSF2:
+ if (copy_from_user(&port->at_data.termios, (void __user *)arg,
+ sizeof(struct termios2)))
+ ret = -EFAULT;
+ break;
+#endif
+
+ case TIOCMGET:
+ ret = put_user(port->at_data.mdmbits, (int __user *)arg);
+ break;
+
+ case TIOCMSET:
+ case TIOCMBIC:
+ case TIOCMBIS: {
+ int mdmbits;
+
+ if (copy_from_user(&mdmbits, (int __user *)arg, sizeof(int))) {
+ ret = -EFAULT;
+ break;
+ }
+ if (cmd == TIOCMBIC)
+ port->at_data.mdmbits &= ~mdmbits;
+ else if (cmd == TIOCMBIS)
+ port->at_data.mdmbits |= mdmbits;
+ else
+ port->at_data.mdmbits = mdmbits;
+ break;
+ }
+
+ default:
+ ret = -ENOIOCTLCMD;
+ }
+
+ mutex_unlock(&port->data_lock);
+
+ return ret;
+}
+
+static long wwan_port_fops_ioctl(struct file *filp, unsigned int cmd,
+ unsigned long arg)
+{
+ struct wwan_port *port = filp->private_data;
+ int res;
+
+ if (port->type == WWAN_PORT_AT) { /* AT port specific IOCTLs */
+ res = wwan_port_fops_at_ioctl(port, cmd, arg);
+ if (res != -ENOIOCTLCMD)
+ return res;
+ }
+
+ switch (cmd) {
+ case TIOCINQ: { /* aka SIOCINQ aka FIONREAD */
+ unsigned long flags;
+ struct sk_buff *skb;
+ int amount = 0;
+
+ spin_lock_irqsave(&port->rxq.lock, flags);
+ skb_queue_walk(&port->rxq, skb)
+ amount += skb->len;
+ spin_unlock_irqrestore(&port->rxq.lock, flags);
+
+ return put_user(amount, (int __user *)arg);
+ }
+
+ default:
+ return -ENOIOCTLCMD;
+ }
+}
+
static const struct file_operations wwan_port_fops = {
.owner = THIS_MODULE,
.open = wwan_port_fops_open,
.read = wwan_port_fops_read,
.write = wwan_port_fops_write,
.poll = wwan_port_fops_poll,
+ .unlocked_ioctl = wwan_port_fops_ioctl,
+#ifdef CONFIG_COMPAT
+ .compat_ioctl = compat_ptr_ioctl,
+#endif
.llseek = noop_llseek,
};
+static int wwan_rtnl_validate(struct nlattr *tb[], struct nlattr *data[],
+ struct netlink_ext_ack *extack)
+{
+ if (!data)
+ return -EINVAL;
+
+ if (!tb[IFLA_PARENT_DEV_NAME])
+ return -EINVAL;
+
+ if (!data[IFLA_WWAN_LINK_ID])
+ return -EINVAL;
+
+ return 0;
+}
+
+static struct device_type wwan_type = { .name = "wwan" };
+
+static struct net_device *wwan_rtnl_alloc(struct nlattr *tb[],
+ const char *ifname,
+ unsigned char name_assign_type,
+ unsigned int num_tx_queues,
+ unsigned int num_rx_queues)
+{
+ const char *devname = nla_data(tb[IFLA_PARENT_DEV_NAME]);
+ struct wwan_device *wwandev = wwan_dev_get_by_name(devname);
+ struct net_device *dev;
+ unsigned int priv_size;
+
+ if (IS_ERR(wwandev))
+ return ERR_CAST(wwandev);
+
+ /* only supported if ops were registered (not just ports) */
+ if (!wwandev->ops) {
+ dev = ERR_PTR(-EOPNOTSUPP);
+ goto out;
+ }
+
+ priv_size = sizeof(struct wwan_netdev_priv) + wwandev->ops->priv_size;
+ dev = alloc_netdev_mqs(priv_size, ifname, name_assign_type,
+ wwandev->ops->setup, num_tx_queues, num_rx_queues);
+
+ if (dev) {
+ SET_NETDEV_DEV(dev, &wwandev->dev);
+ SET_NETDEV_DEVTYPE(dev, &wwan_type);
+ }
+
+out:
+ /* release the reference */
+ put_device(&wwandev->dev);
+ return dev;
+}
+
+static int wwan_rtnl_newlink(struct net *src_net, struct net_device *dev,
+ struct nlattr *tb[], struct nlattr *data[],
+ struct netlink_ext_ack *extack)
+{
+ struct wwan_device *wwandev = wwan_dev_get_by_parent(dev->dev.parent);
+ u32 link_id = nla_get_u32(data[IFLA_WWAN_LINK_ID]);
+ struct wwan_netdev_priv *priv = netdev_priv(dev);
+ int ret;
+
+ if (IS_ERR(wwandev))
+ return PTR_ERR(wwandev);
+
+ /* shouldn't have a netdev (left) with us as parent so WARN */
+ if (WARN_ON(!wwandev->ops)) {
+ ret = -EOPNOTSUPP;
+ goto out;
+ }
+
+ priv->link_id = link_id;
+ if (wwandev->ops->newlink)
+ ret = wwandev->ops->newlink(wwandev->ops_ctxt, dev,
+ link_id, extack);
+ else
+ ret = register_netdevice(dev);
+
+out:
+ /* release the reference */
+ put_device(&wwandev->dev);
+ return ret;
+}
+
+static void wwan_rtnl_dellink(struct net_device *dev, struct list_head *head)
+{
+ struct wwan_device *wwandev = wwan_dev_get_by_parent(dev->dev.parent);
+
+ if (IS_ERR(wwandev))
+ return;
+
+ /* shouldn't have a netdev (left) with us as parent so WARN */
+ if (WARN_ON(!wwandev->ops))
+ goto out;
+
+ if (wwandev->ops->dellink)
+ wwandev->ops->dellink(wwandev->ops_ctxt, dev, head);
+ else
+ unregister_netdevice_queue(dev, head);
+
+out:
+ /* release the reference */
+ put_device(&wwandev->dev);
+}
+
+static size_t wwan_rtnl_get_size(const struct net_device *dev)
+{
+ return
+ nla_total_size(4) + /* IFLA_WWAN_LINK_ID */
+ 0;
+}
+
+static int wwan_rtnl_fill_info(struct sk_buff *skb,
+ const struct net_device *dev)
+{
+ struct wwan_netdev_priv *priv = netdev_priv(dev);
+
+ if (nla_put_u32(skb, IFLA_WWAN_LINK_ID, priv->link_id))
+ goto nla_put_failure;
+
+ return 0;
+
+nla_put_failure:
+ return -EMSGSIZE;
+}
+
+static const struct nla_policy wwan_rtnl_policy[IFLA_WWAN_MAX + 1] = {
+ [IFLA_WWAN_LINK_ID] = { .type = NLA_U32 },
+};
+
+static struct rtnl_link_ops wwan_rtnl_link_ops __read_mostly = {
+ .kind = "wwan",
+ .maxtype = __IFLA_WWAN_MAX,
+ .alloc = wwan_rtnl_alloc,
+ .validate = wwan_rtnl_validate,
+ .newlink = wwan_rtnl_newlink,
+ .dellink = wwan_rtnl_dellink,
+ .get_size = wwan_rtnl_get_size,
+ .fill_info = wwan_rtnl_fill_info,
+ .policy = wwan_rtnl_policy,
+};
+
+static void wwan_create_default_link(struct wwan_device *wwandev,
+ u32 def_link_id)
+{
+ struct nlattr *tb[IFLA_MAX + 1], *linkinfo[IFLA_INFO_MAX + 1];
+ struct nlattr *data[IFLA_WWAN_MAX + 1];
+ struct net_device *dev;
+ struct nlmsghdr *nlh;
+ struct sk_buff *msg;
+
+ /* Forge attributes required to create a WWAN netdev. We first
+ * build a netlink message and then parse it. This looks
+ * odd, but such approach is less error prone.
+ */
+ msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
+ if (WARN_ON(!msg))
+ return;
+ nlh = nlmsg_put(msg, 0, 0, RTM_NEWLINK, 0, 0);
+ if (WARN_ON(!nlh))
+ goto free_attrs;
+
+ if (nla_put_string(msg, IFLA_PARENT_DEV_NAME, dev_name(&wwandev->dev)))
+ goto free_attrs;
+ tb[IFLA_LINKINFO] = nla_nest_start(msg, IFLA_LINKINFO);
+ if (!tb[IFLA_LINKINFO])
+ goto free_attrs;
+ linkinfo[IFLA_INFO_DATA] = nla_nest_start(msg, IFLA_INFO_DATA);
+ if (!linkinfo[IFLA_INFO_DATA])
+ goto free_attrs;
+ if (nla_put_u32(msg, IFLA_WWAN_LINK_ID, def_link_id))
+ goto free_attrs;
+ nla_nest_end(msg, linkinfo[IFLA_INFO_DATA]);
+ nla_nest_end(msg, tb[IFLA_LINKINFO]);
+
+ nlmsg_end(msg, nlh);
+
+ /* The next three parsing calls can not fail */
+ nlmsg_parse_deprecated(nlh, 0, tb, IFLA_MAX, NULL, NULL);
+ nla_parse_nested_deprecated(linkinfo, IFLA_INFO_MAX, tb[IFLA_LINKINFO],
+ NULL, NULL);
+ nla_parse_nested_deprecated(data, IFLA_WWAN_MAX,
+ linkinfo[IFLA_INFO_DATA], NULL, NULL);
+
+ rtnl_lock();
+
+ dev = rtnl_create_link(&init_net, "wwan%d", NET_NAME_ENUM,
+ &wwan_rtnl_link_ops, tb, NULL);
+ if (WARN_ON(IS_ERR(dev)))
+ goto unlock;
+
+ if (WARN_ON(wwan_rtnl_newlink(&init_net, dev, tb, data, NULL))) {
+ free_netdev(dev);
+ goto unlock;
+ }
+
+unlock:
+ rtnl_unlock();
+
+free_attrs:
+ nlmsg_free(msg);
+}
+
+/**
+ * wwan_register_ops - register WWAN device ops
+ * @parent: Device to use as parent and shared by all WWAN ports and
+ * created netdevs
+ * @ops: operations to register
+ * @ctxt: context to pass to operations
+ * @def_link_id: id of the default link that will be automatically created by
+ * the WWAN core for the WWAN device. The default link will not be created
+ * if the passed value is WWAN_NO_DEFAULT_LINK.
+ *
+ * Returns: 0 on success, a negative error code on failure
+ */
+int wwan_register_ops(struct device *parent, const struct wwan_ops *ops,
+ void *ctxt, u32 def_link_id)
+{
+ struct wwan_device *wwandev;
+
+ if (WARN_ON(!parent || !ops || !ops->setup))
+ return -EINVAL;
+
+ wwandev = wwan_create_dev(parent);
+ if (!wwandev)
+ return -ENOMEM;
+
+ if (WARN_ON(wwandev->ops)) {
+ wwan_remove_dev(wwandev);
+ return -EBUSY;
+ }
+
+ wwandev->ops = ops;
+ wwandev->ops_ctxt = ctxt;
+
+ /* NB: we do not abort ops registration in case of default link
+ * creation failure. Link ops is the management interface, while the
+ * default link creation is a service option. And we should not prevent
+ * a user from manually creating a link latter if service option failed
+ * now.
+ */
+ if (def_link_id != WWAN_NO_DEFAULT_LINK)
+ wwan_create_default_link(wwandev, def_link_id);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(wwan_register_ops);
+
+/* Enqueue child netdev deletion */
+static int wwan_child_dellink(struct device *dev, void *data)
+{
+ struct list_head *kill_list = data;
+
+ if (dev->type == &wwan_type)
+ wwan_rtnl_dellink(to_net_dev(dev), kill_list);
+
+ return 0;
+}
+
+/**
+ * wwan_unregister_ops - remove WWAN device ops
+ * @parent: Device to use as parent and shared by all WWAN ports and
+ * created netdevs
+ */
+void wwan_unregister_ops(struct device *parent)
+{
+ struct wwan_device *wwandev = wwan_dev_get_by_parent(parent);
+ LIST_HEAD(kill_list);
+
+ if (WARN_ON(IS_ERR(wwandev)))
+ return;
+ if (WARN_ON(!wwandev->ops)) {
+ put_device(&wwandev->dev);
+ return;
+ }
+
+ /* put the reference obtained by wwan_dev_get_by_parent(),
+ * we should still have one (that the owner is giving back
+ * now) due to the ops being assigned.
+ */
+ put_device(&wwandev->dev);
+
+ rtnl_lock(); /* Prevent concurent netdev(s) creation/destroying */
+
+ /* Remove all child netdev(s), using batch removing */
+ device_for_each_child(&wwandev->dev, &kill_list,
+ wwan_child_dellink);
+ unregister_netdevice_many(&kill_list);
+
+ wwandev->ops = NULL; /* Finally remove ops */
+
+ rtnl_unlock();
+
+ wwandev->ops_ctxt = NULL;
+ wwan_remove_dev(wwandev);
+}
+EXPORT_SYMBOL_GPL(wwan_unregister_ops);
+
static int __init wwan_init(void)
{
+ int err;
+
+ err = rtnl_link_register(&wwan_rtnl_link_ops);
+ if (err)
+ return err;
+
wwan_class = class_create(THIS_MODULE, "wwan");
- if (IS_ERR(wwan_class))
- return PTR_ERR(wwan_class);
+ if (IS_ERR(wwan_class)) {
+ err = PTR_ERR(wwan_class);
+ goto unregister;
+ }
/* chrdev used for wwan ports */
- wwan_major = register_chrdev(0, "wwan_port", &wwan_port_fops);
+ wwan_major = __register_chrdev(0, 0, WWAN_MAX_MINORS, "wwan_port",
+ &wwan_port_fops);
if (wwan_major < 0) {
- class_destroy(wwan_class);
- return wwan_major;
+ err = wwan_major;
+ goto destroy;
}
return 0;
+
+destroy:
+ class_destroy(wwan_class);
+unregister:
+ rtnl_link_unregister(&wwan_rtnl_link_ops);
+ return err;
}
static void __exit wwan_exit(void)
{
- unregister_chrdev(wwan_major, "wwan_port");
+ __unregister_chrdev(wwan_major, 0, WWAN_MAX_MINORS, "wwan_port");
+ rtnl_link_unregister(&wwan_rtnl_link_ops);
class_destroy(wwan_class);
}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * WWAN device simulator for WWAN framework testing.
+ *
+ * Copyright (c) 2021, Sergey Ryazanov <ryazanov.s.a@gmail.com>
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/device.h>
+#include <linux/spinlock.h>
+#include <linux/list.h>
+#include <linux/skbuff.h>
+#include <linux/netdevice.h>
+#include <linux/wwan.h>
+#include <linux/debugfs.h>
+#include <linux/workqueue.h>
+
+#include <net/arp.h>
+
+static int wwan_hwsim_devsnum = 2;
+module_param_named(devices, wwan_hwsim_devsnum, int, 0444);
+MODULE_PARM_DESC(devices, "Number of simulated devices");
+
+static struct class *wwan_hwsim_class;
+
+static struct dentry *wwan_hwsim_debugfs_topdir;
+static struct dentry *wwan_hwsim_debugfs_devcreate;
+
+static DEFINE_SPINLOCK(wwan_hwsim_devs_lock);
+static LIST_HEAD(wwan_hwsim_devs);
+static unsigned int wwan_hwsim_dev_idx;
+
+struct wwan_hwsim_dev {
+ struct list_head list;
+ unsigned int id;
+ struct device dev;
+ struct work_struct del_work;
+ struct dentry *debugfs_topdir;
+ struct dentry *debugfs_portcreate;
+ spinlock_t ports_lock; /* Serialize ports creation/deletion */
+ unsigned int port_idx;
+ struct list_head ports;
+};
+
+struct wwan_hwsim_port {
+ struct list_head list;
+ unsigned int id;
+ struct wwan_hwsim_dev *dev;
+ struct wwan_port *wwan;
+ struct work_struct del_work;
+ struct dentry *debugfs_topdir;
+ enum { /* AT command parser state */
+ AT_PARSER_WAIT_A,
+ AT_PARSER_WAIT_T,
+ AT_PARSER_WAIT_TERM,
+ AT_PARSER_SKIP_LINE,
+ } pstate;
+};
+
+static const struct file_operations wwan_hwsim_debugfs_portdestroy_fops;
+static const struct file_operations wwan_hwsim_debugfs_portcreate_fops;
+static const struct file_operations wwan_hwsim_debugfs_devdestroy_fops;
+static void wwan_hwsim_port_del_work(struct work_struct *work);
+static void wwan_hwsim_dev_del_work(struct work_struct *work);
+
+static netdev_tx_t wwan_hwsim_netdev_xmit(struct sk_buff *skb,
+ struct net_device *ndev)
+{
+ ndev->stats.tx_packets++;
+ ndev->stats.tx_bytes += skb->len;
+ consume_skb(skb);
+ return NETDEV_TX_OK;
+}
+
+static const struct net_device_ops wwan_hwsim_netdev_ops = {
+ .ndo_start_xmit = wwan_hwsim_netdev_xmit,
+};
+
+static void wwan_hwsim_netdev_setup(struct net_device *ndev)
+{
+ ndev->netdev_ops = &wwan_hwsim_netdev_ops;
+ ndev->needs_free_netdev = true;
+
+ ndev->mtu = ETH_DATA_LEN;
+ ndev->min_mtu = ETH_MIN_MTU;
+ ndev->max_mtu = ETH_MAX_MTU;
+
+ ndev->type = ARPHRD_NONE;
+ ndev->flags = IFF_POINTOPOINT | IFF_NOARP;
+}
+
+static const struct wwan_ops wwan_hwsim_wwan_rtnl_ops = {
+ .priv_size = 0, /* No private data */
+ .setup = wwan_hwsim_netdev_setup,
+};
+
+static int wwan_hwsim_port_start(struct wwan_port *wport)
+{
+ struct wwan_hwsim_port *port = wwan_port_get_drvdata(wport);
+
+ port->pstate = AT_PARSER_WAIT_A;
+
+ return 0;
+}
+
+static void wwan_hwsim_port_stop(struct wwan_port *wport)
+{
+}
+
+/* Implements a minimalistic AT commands parser that echo input back and
+ * reply with 'OK' to each input command. See AT command protocol details in the
+ * ITU-T V.250 recomendations document.
+ *
+ * Be aware that this processor is not fully V.250 compliant.
+ */
+static int wwan_hwsim_port_tx(struct wwan_port *wport, struct sk_buff *in)
+{
+ struct wwan_hwsim_port *port = wwan_port_get_drvdata(wport);
+ struct sk_buff *out;
+ int i, n, s;
+
+ /* Estimate a max possible number of commands by counting the number of
+ * termination chars (S3 param, CR by default). And then allocate the
+ * output buffer that will be enough to fit the echo and result codes of
+ * all commands.
+ */
+ for (i = 0, n = 0; i < in->len; ++i)
+ if (in->data[i] == '\r')
+ n++;
+ n = in->len + n * (2 + 2 + 2); /* Output buffer size */
+ out = alloc_skb(n, GFP_KERNEL);
+ if (!out)
+ return -ENOMEM;
+
+ for (i = 0, s = 0; i < in->len; ++i) {
+ char c = in->data[i];
+
+ if (port->pstate == AT_PARSER_WAIT_A) {
+ if (c == 'A' || c == 'a')
+ port->pstate = AT_PARSER_WAIT_T;
+ else if (c != '\n') /* Ignore formating char */
+ port->pstate = AT_PARSER_SKIP_LINE;
+ } else if (port->pstate == AT_PARSER_WAIT_T) {
+ if (c == 'T' || c == 't')
+ port->pstate = AT_PARSER_WAIT_TERM;
+ else
+ port->pstate = AT_PARSER_SKIP_LINE;
+ } else if (port->pstate == AT_PARSER_WAIT_TERM) {
+ if (c != '\r')
+ continue;
+ /* Consume the trailing formatting char as well */
+ if ((i + 1) < in->len && in->data[i + 1] == '\n')
+ i++;
+ n = i - s + 1;
+ memcpy(skb_put(out, n), &in->data[s], n);/* Echo */
+ memcpy(skb_put(out, 6), "\r\nOK\r\n", 6);
+ s = i + 1;
+ port->pstate = AT_PARSER_WAIT_A;
+ } else if (port->pstate == AT_PARSER_SKIP_LINE) {
+ if (c != '\r')
+ continue;
+ port->pstate = AT_PARSER_WAIT_A;
+ }
+ }
+
+ if (i > s) {
+ /* Echo the processed portion of a not yet completed command */
+ n = i - s;
+ memcpy(skb_put(out, n), &in->data[s], n);
+ }
+
+ consume_skb(in);
+
+ wwan_port_rx(wport, out);
+
+ return 0;
+}
+
+static const struct wwan_port_ops wwan_hwsim_port_ops = {
+ .start = wwan_hwsim_port_start,
+ .stop = wwan_hwsim_port_stop,
+ .tx = wwan_hwsim_port_tx,
+};
+
+static struct wwan_hwsim_port *wwan_hwsim_port_new(struct wwan_hwsim_dev *dev)
+{
+ struct wwan_hwsim_port *port;
+ char name[0x10];
+ int err;
+
+ port = kzalloc(sizeof(*port), GFP_KERNEL);
+ if (!port)
+ return ERR_PTR(-ENOMEM);
+
+ port->dev = dev;
+
+ spin_lock(&dev->ports_lock);
+ port->id = dev->port_idx++;
+ spin_unlock(&dev->ports_lock);
+
+ port->wwan = wwan_create_port(&dev->dev, WWAN_PORT_AT,
+ &wwan_hwsim_port_ops,
+ port);
+ if (IS_ERR(port->wwan)) {
+ err = PTR_ERR(port->wwan);
+ goto err_free_port;
+ }
+
+ INIT_WORK(&port->del_work, wwan_hwsim_port_del_work);
+
+ snprintf(name, sizeof(name), "port%u", port->id);
+ port->debugfs_topdir = debugfs_create_dir(name, dev->debugfs_topdir);
+ debugfs_create_file("destroy", 0200, port->debugfs_topdir, port,
+ &wwan_hwsim_debugfs_portdestroy_fops);
+
+ return port;
+
+err_free_port:
+ kfree(port);
+
+ return ERR_PTR(err);
+}
+
+static void wwan_hwsim_port_del(struct wwan_hwsim_port *port)
+{
+ debugfs_remove(port->debugfs_topdir);
+
+ /* Make sure that there is no pending deletion work */
+ if (current_work() != &port->del_work)
+ cancel_work_sync(&port->del_work);
+
+ wwan_remove_port(port->wwan);
+ kfree(port);
+}
+
+static void wwan_hwsim_port_del_work(struct work_struct *work)
+{
+ struct wwan_hwsim_port *port =
+ container_of(work, typeof(*port), del_work);
+ struct wwan_hwsim_dev *dev = port->dev;
+
+ spin_lock(&dev->ports_lock);
+ if (list_empty(&port->list)) {
+ /* Someone else deleting port at the moment */
+ spin_unlock(&dev->ports_lock);
+ return;
+ }
+ list_del_init(&port->list);
+ spin_unlock(&dev->ports_lock);
+
+ wwan_hwsim_port_del(port);
+}
+
+static void wwan_hwsim_dev_release(struct device *sysdev)
+{
+ struct wwan_hwsim_dev *dev = container_of(sysdev, typeof(*dev), dev);
+
+ kfree(dev);
+}
+
+static struct wwan_hwsim_dev *wwan_hwsim_dev_new(void)
+{
+ struct wwan_hwsim_dev *dev;
+ int err;
+
+ dev = kzalloc(sizeof(*dev), GFP_KERNEL);
+ if (!dev)
+ return ERR_PTR(-ENOMEM);
+
+ spin_lock(&wwan_hwsim_devs_lock);
+ dev->id = wwan_hwsim_dev_idx++;
+ spin_unlock(&wwan_hwsim_devs_lock);
+
+ dev->dev.release = wwan_hwsim_dev_release;
+ dev->dev.class = wwan_hwsim_class;
+ dev_set_name(&dev->dev, "hwsim%u", dev->id);
+
+ spin_lock_init(&dev->ports_lock);
+ INIT_LIST_HEAD(&dev->ports);
+
+ err = device_register(&dev->dev);
+ if (err)
+ goto err_free_dev;
+
+ INIT_WORK(&dev->del_work, wwan_hwsim_dev_del_work);
+
+ err = wwan_register_ops(&dev->dev, &wwan_hwsim_wwan_rtnl_ops, dev, 1);
+ if (err)
+ goto err_unreg_dev;
+
+ dev->debugfs_topdir = debugfs_create_dir(dev_name(&dev->dev),
+ wwan_hwsim_debugfs_topdir);
+ debugfs_create_file("destroy", 0200, dev->debugfs_topdir, dev,
+ &wwan_hwsim_debugfs_devdestroy_fops);
+ dev->debugfs_portcreate =
+ debugfs_create_file("portcreate", 0200,
+ dev->debugfs_topdir, dev,
+ &wwan_hwsim_debugfs_portcreate_fops);
+
+ return dev;
+
+err_unreg_dev:
+ device_unregister(&dev->dev);
+ /* Memory will be freed in the device release callback */
+
+ return ERR_PTR(err);
+
+err_free_dev:
+ kfree(dev);
+
+ return ERR_PTR(err);
+}
+
+static void wwan_hwsim_dev_del(struct wwan_hwsim_dev *dev)
+{
+ debugfs_remove(dev->debugfs_portcreate); /* Avoid new ports */
+
+ spin_lock(&dev->ports_lock);
+ while (!list_empty(&dev->ports)) {
+ struct wwan_hwsim_port *port;
+
+ port = list_first_entry(&dev->ports, struct wwan_hwsim_port,
+ list);
+ list_del_init(&port->list);
+ spin_unlock(&dev->ports_lock);
+ wwan_hwsim_port_del(port);
+ spin_lock(&dev->ports_lock);
+ }
+ spin_unlock(&dev->ports_lock);
+
+ debugfs_remove(dev->debugfs_topdir);
+
+ /* This will remove all child netdev(s) */
+ wwan_unregister_ops(&dev->dev);
+
+ /* Make sure that there is no pending deletion work */
+ if (current_work() != &dev->del_work)
+ cancel_work_sync(&dev->del_work);
+
+ device_unregister(&dev->dev);
+ /* Memory will be freed in the device release callback */
+}
+
+static void wwan_hwsim_dev_del_work(struct work_struct *work)
+{
+ struct wwan_hwsim_dev *dev = container_of(work, typeof(*dev), del_work);
+
+ spin_lock(&wwan_hwsim_devs_lock);
+ if (list_empty(&dev->list)) {
+ /* Someone else deleting device at the moment */
+ spin_unlock(&wwan_hwsim_devs_lock);
+ return;
+ }
+ list_del_init(&dev->list);
+ spin_unlock(&wwan_hwsim_devs_lock);
+
+ wwan_hwsim_dev_del(dev);
+}
+
+static ssize_t wwan_hwsim_debugfs_portdestroy_write(struct file *file,
+ const char __user *usrbuf,
+ size_t count, loff_t *ppos)
+{
+ struct wwan_hwsim_port *port = file->private_data;
+
+ /* We can not delete port here since it will cause a deadlock due to
+ * waiting this callback to finish in the debugfs_remove() call. So,
+ * use workqueue.
+ */
+ schedule_work(&port->del_work);
+
+ return count;
+}
+
+static const struct file_operations wwan_hwsim_debugfs_portdestroy_fops = {
+ .write = wwan_hwsim_debugfs_portdestroy_write,
+ .open = simple_open,
+ .llseek = noop_llseek,
+};
+
+static ssize_t wwan_hwsim_debugfs_portcreate_write(struct file *file,
+ const char __user *usrbuf,
+ size_t count, loff_t *ppos)
+{
+ struct wwan_hwsim_dev *dev = file->private_data;
+ struct wwan_hwsim_port *port;
+
+ port = wwan_hwsim_port_new(dev);
+ if (IS_ERR(port))
+ return PTR_ERR(port);
+
+ spin_lock(&dev->ports_lock);
+ list_add_tail(&port->list, &dev->ports);
+ spin_unlock(&dev->ports_lock);
+
+ return count;
+}
+
+static const struct file_operations wwan_hwsim_debugfs_portcreate_fops = {
+ .write = wwan_hwsim_debugfs_portcreate_write,
+ .open = simple_open,
+ .llseek = noop_llseek,
+};
+
+static ssize_t wwan_hwsim_debugfs_devdestroy_write(struct file *file,
+ const char __user *usrbuf,
+ size_t count, loff_t *ppos)
+{
+ struct wwan_hwsim_dev *dev = file->private_data;
+
+ /* We can not delete device here since it will cause a deadlock due to
+ * waiting this callback to finish in the debugfs_remove() call. So,
+ * use workqueue.
+ */
+ schedule_work(&dev->del_work);
+
+ return count;
+}
+
+static const struct file_operations wwan_hwsim_debugfs_devdestroy_fops = {
+ .write = wwan_hwsim_debugfs_devdestroy_write,
+ .open = simple_open,
+ .llseek = noop_llseek,
+};
+
+static ssize_t wwan_hwsim_debugfs_devcreate_write(struct file *file,
+ const char __user *usrbuf,
+ size_t count, loff_t *ppos)
+{
+ struct wwan_hwsim_dev *dev;
+
+ dev = wwan_hwsim_dev_new();
+ if (IS_ERR(dev))
+ return PTR_ERR(dev);
+
+ spin_lock(&wwan_hwsim_devs_lock);
+ list_add_tail(&dev->list, &wwan_hwsim_devs);
+ spin_unlock(&wwan_hwsim_devs_lock);
+
+ return count;
+}
+
+static const struct file_operations wwan_hwsim_debugfs_devcreate_fops = {
+ .write = wwan_hwsim_debugfs_devcreate_write,
+ .open = simple_open,
+ .llseek = noop_llseek,
+};
+
+static int __init wwan_hwsim_init_devs(void)
+{
+ struct wwan_hwsim_dev *dev;
+ int i, j;
+
+ for (i = 0; i < wwan_hwsim_devsnum; ++i) {
+ dev = wwan_hwsim_dev_new();
+ if (IS_ERR(dev))
+ return PTR_ERR(dev);
+
+ spin_lock(&wwan_hwsim_devs_lock);
+ list_add_tail(&dev->list, &wwan_hwsim_devs);
+ spin_unlock(&wwan_hwsim_devs_lock);
+
+ /* Create a couple of ports per each device to accelerate
+ * the simulator readiness time.
+ */
+ for (j = 0; j < 2; ++j) {
+ struct wwan_hwsim_port *port;
+
+ port = wwan_hwsim_port_new(dev);
+ if (IS_ERR(port))
+ return PTR_ERR(port);
+
+ spin_lock(&dev->ports_lock);
+ list_add_tail(&port->list, &dev->ports);
+ spin_unlock(&dev->ports_lock);
+ }
+ }
+
+ return 0;
+}
+
+static void wwan_hwsim_free_devs(void)
+{
+ struct wwan_hwsim_dev *dev;
+
+ spin_lock(&wwan_hwsim_devs_lock);
+ while (!list_empty(&wwan_hwsim_devs)) {
+ dev = list_first_entry(&wwan_hwsim_devs, struct wwan_hwsim_dev,
+ list);
+ list_del_init(&dev->list);
+ spin_unlock(&wwan_hwsim_devs_lock);
+ wwan_hwsim_dev_del(dev);
+ spin_lock(&wwan_hwsim_devs_lock);
+ }
+ spin_unlock(&wwan_hwsim_devs_lock);
+}
+
+static int __init wwan_hwsim_init(void)
+{
+ int err;
+
+ if (wwan_hwsim_devsnum < 0 || wwan_hwsim_devsnum > 128)
+ return -EINVAL;
+
+ wwan_hwsim_class = class_create(THIS_MODULE, "wwan_hwsim");
+ if (IS_ERR(wwan_hwsim_class))
+ return PTR_ERR(wwan_hwsim_class);
+
+ wwan_hwsim_debugfs_topdir = debugfs_create_dir("wwan_hwsim", NULL);
+ wwan_hwsim_debugfs_devcreate =
+ debugfs_create_file("devcreate", 0200,
+ wwan_hwsim_debugfs_topdir, NULL,
+ &wwan_hwsim_debugfs_devcreate_fops);
+
+ err = wwan_hwsim_init_devs();
+ if (err)
+ goto err_clean_devs;
+
+ return 0;
+
+err_clean_devs:
+ wwan_hwsim_free_devs();
+ debugfs_remove(wwan_hwsim_debugfs_topdir);
+ class_destroy(wwan_hwsim_class);
+
+ return err;
+}
+
+static void __exit wwan_hwsim_exit(void)
+{
+ debugfs_remove(wwan_hwsim_debugfs_devcreate); /* Avoid new devs */
+ wwan_hwsim_free_devs();
+ flush_scheduled_work(); /* Wait deletion works completion */
+ debugfs_remove(wwan_hwsim_debugfs_topdir);
+ class_destroy(wwan_hwsim_class);
+}
+
+module_init(wwan_hwsim_init);
+module_exit(wwan_hwsim_exit);
+
+MODULE_AUTHOR("Sergey Ryazanov");
+MODULE_DESCRIPTION("Device simulator for WWAN framework");
+MODULE_LICENSE("GPL");
{
if (queue->task) {
kthread_stop(queue->task);
+ put_task_struct(queue->task);
queue->task = NULL;
}
if (IS_ERR(task))
goto kthread_err;
queue->task = task;
+ /*
+ * Take a reference to the task in order to prevent it from being freed
+ * if the thread function returns before kthread_stop is called.
+ */
+ get_task_struct(task);
task = kthread_run(xenvif_dealloc_kthread, queue,
"%s-dealloc", queue->name);
wake_up(&info->setup_wq);
}
-/**
+/*
* Register a packet sent counter and a callback
*
* We have no other way of knowing when all firmware packets were sent out
info->data_pkt_counter_cb = cb;
}
-/**
+/*
* The device is expecting a stream of packets. All packets need to
* have the PBF flag set to 0x0 (last packet) even if the firmware
* file is segmented and there are multiple packets. If we give the
static int fdp_nci_open(struct nci_dev *ndev)
{
struct fdp_nci_info *info = nci_get_drvdata(ndev);
- struct device *dev = &info->phy->i2c_dev->dev;
-
- dev_dbg(dev, "%s\n", __func__);
return info->phy_ops->enable(info->phy);
}
static int fdp_nci_close(struct nci_dev *ndev)
{
- struct fdp_nci_info *info = nci_get_drvdata(ndev);
- struct device *dev = &info->phy->i2c_dev->dev;
-
- dev_dbg(dev, "%s\n", __func__);
return 0;
}
static int fdp_nci_send(struct nci_dev *ndev, struct sk_buff *skb)
{
struct fdp_nci_info *info = nci_get_drvdata(ndev);
- struct device *dev = &info->phy->i2c_dev->dev;
-
- dev_dbg(dev, "%s\n", __func__);
if (atomic_dec_and_test(&info->data_pkt_counter))
info->data_pkt_counter_cb(ndev);
return info->phy_ops->write(info->phy, skb);
}
-int fdp_nci_recv_frame(struct nci_dev *ndev, struct sk_buff *skb)
-{
- struct fdp_nci_info *info = nci_get_drvdata(ndev);
- struct device *dev = &info->phy->i2c_dev->dev;
-
- dev_dbg(dev, "%s\n", __func__);
- return nci_recv_frame(ndev, skb);
-}
-EXPORT_SYMBOL(fdp_nci_recv_frame);
-
static int fdp_nci_request_firmware(struct nci_dev *ndev)
{
struct fdp_nci_info *info = nci_get_drvdata(ndev);
r = request_firmware(&info->ram_patch, FDP_RAM_PATCH_NAME, dev);
if (r < 0) {
nfc_err(dev, "RAM patch request error\n");
- goto error;
+ return r;
}
data = (u8 *) info->ram_patch->data;
r = request_firmware(&info->otp_patch, FDP_OTP_PATCH_NAME, dev);
if (r < 0) {
nfc_err(dev, "OTP patch request error\n");
- goto out;
+ return 0;
}
data = (u8 *) info->otp_patch->data;
dev_dbg(dev, "OTP patch version: %d, size: %d\n",
info->otp_patch_version, (int) info->otp_patch->size);
-out:
return 0;
-error:
- return r;
}
static void fdp_nci_release_firmware(struct nci_dev *ndev)
int r;
u8 patched = 0;
- dev_dbg(dev, "%s\n", __func__);
-
r = nci_core_init(ndev);
if (r)
goto error;
struct sk_buff *skb)
{
struct fdp_nci_info *info = nci_get_drvdata(ndev);
- struct device *dev = &info->phy->i2c_dev->dev;
- dev_dbg(dev, "%s\n", __func__);
info->setup_reset_ntf = 1;
wake_up(&info->setup_wq);
struct sk_buff *skb)
{
struct fdp_nci_info *info = nci_get_drvdata(ndev);
- struct device *dev = &info->phy->i2c_dev->dev;
- dev_dbg(dev, "%s\n", __func__);
info->setup_patch_ntf = 1;
info->setup_patch_status = skb->data[0];
wake_up(&info->setup_wq);
void fdp_nci_remove(struct nci_dev *ndev)
{
- struct fdp_nci_info *info = nci_get_drvdata(ndev);
- struct device *dev = &info->phy->i2c_dev->dev;
-
- dev_dbg(dev, "%s\n", __func__);
-
nci_unregister_device(ndev);
nci_free_device(ndev);
}
struct nci_dev **ndev, int tx_headroom, int tx_tailroom,
u8 clock_type, u32 clock_freq, u8 *fw_vsc_cfg);
void fdp_nci_remove(struct nci_dev *ndev);
-int fdp_nci_recv_frame(struct nci_dev *ndev, struct sk_buff *skb);
#endif /* __LOCAL_FDP_H_ */
{
struct fdp_i2c_phy *phy = phy_id;
- dev_dbg(&phy->i2c_dev->dev, "%s\n", __func__);
fdp_nci_i2c_reset(phy);
return 0;
{
struct fdp_i2c_phy *phy = phy_id;
- dev_dbg(&phy->i2c_dev->dev, "%s\n", __func__);
fdp_nci_i2c_reset(phy);
}
static irqreturn_t fdp_nci_i2c_irq_thread_fn(int irq, void *phy_id)
{
struct fdp_i2c_phy *phy = phy_id;
- struct i2c_client *client;
struct sk_buff *skb;
int r;
return IRQ_NONE;
}
- client = phy->i2c_dev;
- dev_dbg(&client->dev, "%s\n", __func__);
-
r = fdp_nci_i2c_read(phy, &skb);
if (r == -EREMOTEIO)
return IRQ_HANDLED;
if (skb != NULL)
- fdp_nci_recv_frame(phy->ndev, skb);
+ nci_recv_frame(phy->ndev, skb);
return IRQ_HANDLED;
}
u32 clock_freq;
int r = 0;
- dev_dbg(dev, "%s\n", __func__);
-
if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) {
nfc_err(dev, "No I2C_FUNC_I2C support\n");
return -ENODEV;
{
struct fdp_i2c_phy *phy = i2c_get_clientdata(client);
- dev_dbg(&client->dev, "%s\n", __func__);
-
fdp_nci_remove(phy->ndev);
fdp_nci_i2c_disable(phy);
static struct i2c_driver fdp_nci_i2c_driver = {
.driver = {
.name = FDP_I2C_DRIVER_NAME,
- .acpi_match_table = ACPI_PTR(fdp_nci_i2c_acpi_match),
+ .acpi_match_table = fdp_nci_i2c_acpi_match,
},
.probe_new = fdp_nci_i2c_probe,
.remove = fdp_nci_i2c_remove,
size_t if_version_length;
int bytes_recv, r;
- pr_info("%s\n", __func__);
-
memset(&cmd, 0, sizeof(struct mei_nfc_cmd));
cmd.hdr.cmd = MEI_NFC_CMD_MAINTENANCE;
cmd.hdr.data_size = 1;
size_t connect_length, connect_resp_length;
int bytes_recv, r;
- pr_info("%s\n", __func__);
-
connect_length = sizeof(struct mei_nfc_cmd) +
sizeof(struct mei_nfc_connect);
int r;
struct nfc_mei_phy *phy = phy_id;
- pr_info("%s\n", __func__);
-
if (phy->powered == 1)
return 0;
{
struct nfc_mei_phy *phy = phy_id;
- pr_info("%s\n", __func__);
-
mei_cldev_disable(phy->cldev);
phy->powered = 0;
case MICROREAD_CB_TYPE_READER_ALL:
if (err == 0) {
if (skb->len == 0) {
- err = -EPROTO;
kfree_skb(skb);
info->async_cb(info->async_cb_context, NULL,
-EPROTO);
+// SPDX-License-Identifier: GPL-2.0-only
/*
* Marvell NFC driver: Firmware downloader
*
* Copyright (C) 2015, Marvell International Ltd.
- *
- * This software file (the "File") is distributed by Marvell International
- * Ltd. under the terms of the GNU General Public License Version 2, June 1991
- * (the "License"). You may use, redistribute and/or modify this File in
- * accordance with the terms and conditions of the License, a copy of which
- * is available on the worldwide web at
- * http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt.
- *
- * THE FILE IS DISTRIBUTED AS-IS, WITHOUT WARRANTY OF ANY KIND, AND THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
- * ARE EXPRESSLY DISCLAIMED. The License provides additional details about
- * this warranty disclaimer.
*/
#include <linux/module.h>
};
/*
-** Patterns for responses
-*/
+ * Patterns for responses
+ */
static const uint8_t nci_pattern_core_reset_ntf[] = {
0x60, 0x00, 0x02, 0xA0, 0x01
}
}
-int nfcmrvl_fw_dnld_init(struct nfcmrvl_private *priv)
+int nfcmrvl_fw_dnld_init(struct nfcmrvl_private *priv)
{
char name[32];
return 0;
}
-void nfcmrvl_fw_dnld_deinit(struct nfcmrvl_private *priv)
+void nfcmrvl_fw_dnld_deinit(struct nfcmrvl_private *priv)
{
destroy_workqueue(priv->fw_dnld.rx_wq);
}
-void nfcmrvl_fw_dnld_recv_frame(struct nfcmrvl_private *priv,
- struct sk_buff *skb)
+void nfcmrvl_fw_dnld_recv_frame(struct nfcmrvl_private *priv,
+ struct sk_buff *skb)
{
/* Discard command timer */
if (timer_pending(&priv->ndev->cmd_timer))
-/**
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
* Marvell NFC driver: Firmware downloader
*
* Copyright (C) 2015, Marvell International Ltd.
- *
- * This software file (the "File") is distributed by Marvell International
- * Ltd. under the terms of the GNU General Public License Version 2, June 1991
- * (the "License"). You may use, redistribute and/or modify this File in
- * accordance with the terms and conditions of the License, a copy of which
- * is available on the worldwide web at
- * http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt.
- *
- * THE FILE IS DISTRIBUTED AS-IS, WITHOUT WARRANTY OF ANY KIND, AND THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
- * ARE EXPRESSLY DISCLAIMED. The License provides additional details about
- * this warranty disclaimer.
- **/
+ */
#ifndef __NFCMRVL_FW_DNLD_H__
#define __NFCMRVL_FW_DNLD_H__
-/**
+// SPDX-License-Identifier: GPL-2.0-only
+/*
* Marvell NFC-over-I2C driver: I2C interface related functions
*
* Copyright (C) 2015, Marvell International Ltd.
- *
- * This software file (the "File") is distributed by Marvell International
- * Ltd. under the terms of the GNU General Public License Version 2, June 1991
- * (the "License"). You may use, redistribute and/or modify this File in
- * accordance with the terms and conditions of the License, a copy of which
- * is available on the worldwide web at
- * http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt.
- *
- * THE FILE IS DISTRIBUTED AS-IS, WITHOUT WARRANTY OF ANY KIND, AND THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
- * ARE EXPRESSLY DISCLAIMED. The License provides additional details about
- * this warranty disclaimer.
- **/
+ */
#include <linux/module.h>
#include <linux/interrupt.h>
return -EBADMSG;
}
- if (nci_hdr.plen > NCI_MAX_PAYLOAD_SIZE) {
- nfc_err(&drv_data->i2c->dev, "invalid packet payload size\n");
- return -EBADMSG;
- }
-
*skb = nci_skb_alloc(drv_data->priv->ndev,
nci_hdr.plen + NCI_CTRL_HDR_SIZE, GFP_KERNEL);
if (!*skb)
}
-static const struct of_device_id of_nfcmrvl_i2c_match[] = {
+static const struct of_device_id of_nfcmrvl_i2c_match[] __maybe_unused = {
{ .compatible = "marvell,nfc-i2c", },
{},
};
+// SPDX-License-Identifier: GPL-2.0-only
/*
* Marvell NFC driver: major functions
*
* Copyright (C) 2014-2015 Marvell International Ltd.
- *
- * This software file (the "File") is distributed by Marvell International
- * Ltd. under the terms of the GNU General Public License Version 2, June 1991
- * (the "License"). You may use, redistribute and/or modify this File in
- * accordance with the terms and conditions of the License, a copy of which
- * is available on the worldwide web at
- * http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt.
- *
- * THE FILE IS DISTRIBUTED AS-IS, WITHOUT WARRANTY OF ANY KIND, AND THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
- * ARE EXPRESSLY DISCLAIMED. The License provides additional details about
- * this warranty disclaimer.
*/
#include <linux/module.h>
-/**
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
* Marvell NFC driver
*
* Copyright (C) 2014-2015, Marvell International Ltd.
- *
- * This software file (the "File") is distributed by Marvell International
- * Ltd. under the terms of the GNU General Public License Version 2, June 1991
- * (the "License"). You may use, redistribute and/or modify this File in
- * accordance with the terms and conditions of the License, a copy of which
- * is available on the worldwide web at
- * http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt.
- *
- * THE FILE IS DISTRIBUTED AS-IS, WITHOUT WARRANTY OF ANY KIND, AND THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
- * ARE EXPRESSLY DISCLAIMED. The License provides additional details about
- * this warranty disclaimer.
- **/
+ */
#ifndef _NFCMRVL_H_
#define _NFCMRVL_H_
#define NFCMRVL_NCI_MAX_EVENT_SIZE 260
/*
-** NCI FW Parmaters
-*/
+ * NCI FW Parameters
+ */
#define NFCMRVL_PB_BAIL_OUT 0x11
#define NFCMRVL_PROP_REF_CLOCK 0xF0
#define NFCMRVL_PROP_SET_HI_CONFIG 0xF1
/*
-** HCI defines
-*/
+ * HCI defines
+ */
#define NFCMRVL_HCI_EVENT_HEADER_SIZE 0x04
#define NFCMRVL_HCI_EVENT_CODE 0x04
bool support_fw_dnld;
/*
- ** PHY related information
- */
+ * PHY related information
+ */
/* PHY driver context */
void *drv_data;
-/**
+// SPDX-License-Identifier: GPL-2.0-only
+/*
* Marvell NFC-over-SPI driver: SPI interface related functions
*
* Copyright (C) 2015, Marvell International Ltd.
- *
- * This software file (the "File") is distributed by Marvell International
- * Ltd. under the terms of the GNU General Public License Version 2, June 1991
- * (the "License"). You may use, redistribute and/or modify this File in
- * accordance with the terms and conditions of the License, a copy of which
- * is available on the worldwide web at
- * http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt.
- *
- * THE FILE IS DISTRIBUTED AS-IS, WITHOUT WARRANTY OF ANY KIND, AND THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
- * ARE EXPRESSLY DISCLAIMED. The License provides additional details about
- * this warranty disclaimer.
- **/
+ */
#include <linux/module.h>
#include <linux/interrupt.h>
return 0;
}
-static const struct of_device_id of_nfcmrvl_spi_match[] = {
+static const struct of_device_id of_nfcmrvl_spi_match[] __maybe_unused = {
{ .compatible = "marvell,nfc-spi", },
{},
};
-/**
+// SPDX-License-Identifier: GPL-2.0-only
+/*
* Marvell NFC-over-UART driver
*
* Copyright (C) 2015, Marvell International Ltd.
- *
- * This software file (the "File") is distributed by Marvell International
- * Ltd. under the terms of the GNU General Public License Version 2, June 1991
- * (the "License"). You may use, redistribute and/or modify this File in
- * accordance with the terms and conditions of the License, a copy of which
- * is available on the worldwide web at
- * http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt.
- *
- * THE FILE IS DISTRIBUTED AS-IS, WITHOUT WARRANTY OF ANY KIND, AND THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
- * ARE EXPRESSLY DISCLAIMED. The License provides additional details about
- * this warranty disclaimer.
*/
#include <linux/module.h>
static int reset_n_io = -EINVAL;
/*
-** NFCMRVL NCI OPS
-*/
+ * NFCMRVL NCI OPS
+ */
static int nfcmrvl_uart_nci_open(struct nfcmrvl_private *priv)
{
}
/*
-** NCI UART OPS
-*/
+ * NCI UART OPS
+ */
static int nfcmrvl_nci_uart_open(struct nci_uart *nu)
{
return;
/*
- ** To ensure that if the NFCC goes in DEEP SLEEP sate we can wake him
- ** up. we set BREAK. Once we will be ready to send again we will remove
- ** it.
- */
+ * To ensure that if the NFCC goes in DEEP SLEEP sate we can wake him
+ * up. we set BREAK. Once we will be ready to send again we will remove
+ * it.
+ */
if (priv->config.break_control && nu->tty->ops->break_ctl) {
nu->tty->ops->break_ctl(nu->tty, -1);
usleep_range(1000, 3000);
.tx_done = nfcmrvl_nci_uart_tx_done,
}
};
-
-/*
-** Module init
-*/
-
-static int nfcmrvl_uart_init_module(void)
-{
- return nci_uart_register(&nfcmrvl_nci_uart);
-}
-
-static void nfcmrvl_uart_exit_module(void)
-{
- nci_uart_unregister(&nfcmrvl_nci_uart);
-}
-
-module_init(nfcmrvl_uart_init_module);
-module_exit(nfcmrvl_uart_exit_module);
+module_driver(nfcmrvl_nci_uart, nci_uart_register, nci_uart_unregister);
MODULE_AUTHOR("Marvell International Ltd.");
MODULE_DESCRIPTION("Marvell NFC-over-UART");
-/**
+// SPDX-License-Identifier: GPL-2.0-only
+/*
* Marvell NFC-over-USB driver: USB interface related functions
*
* Copyright (C) 2014, Marvell International Ltd.
- *
- * This software file (the "File") is distributed by Marvell International
- * Ltd. under the terms of the GNU General Public License Version 2, June 1991
- * (the "License"). You may use, redistribute and/or modify this File in
- * accordance with the terms and conditions of the License, a copy of which
- * is available on the worldwide web at
- * http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt.
- *
- * THE FILE IS DISTRIBUTED AS-IS, WITHOUT WARRANTY OF ANY KIND, AND THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
- * ARE EXPRESSLY DISCLAIMED. The License provides additional details about
- * this warranty disclaimer.
- **/
+ */
#include <linux/module.h>
#include <linux/usb.h>
static void nfcmrvl_bulk_complete(struct urb *urb)
{
struct nfcmrvl_usb_drv_data *drv_data = urb->context;
- struct sk_buff *skb;
int err;
dev_dbg(&drv_data->udev->dev, "urb %p status %d count %d\n",
return;
if (!urb->status) {
+ struct sk_buff *skb;
+
skb = nci_skb_alloc(drv_data->priv->ndev, urb->actual_length,
GFP_ATOMIC);
if (!skb) {
static int nfcmrvl_probe(struct usb_interface *intf,
const struct usb_device_id *id)
{
- struct usb_endpoint_descriptor *ep_desc;
struct nfcmrvl_usb_drv_data *drv_data;
struct nfcmrvl_private *priv;
int i;
return -ENOMEM;
for (i = 0; i < intf->cur_altsetting->desc.bNumEndpoints; i++) {
+ struct usb_endpoint_descriptor *ep_desc;
+
ep_desc = &intf->cur_altsetting->endpoint[i].desc;
if (!drv_data->bulk_tx_ep &&
usb_endpoint_is_bulk_out(ep_desc)) {
drv_data->bulk_tx_ep = ep_desc;
- continue;
- }
-
- if (!drv_data->bulk_rx_ep &&
- usb_endpoint_is_bulk_in(ep_desc)) {
+ } else if (!drv_data->bulk_rx_ep &&
+ usb_endpoint_is_bulk_in(ep_desc)) {
drv_data->bulk_rx_ep = ep_desc;
- continue;
}
}
struct nxp_nci_info *info = nci_get_drvdata(ndev);
int r;
- if (!info->phy_ops->write) {
- r = -ENOTSUPP;
- goto send_exit;
- }
+ if (!info->phy_ops->write)
+ return -EOPNOTSUPP;
- if (info->mode != NXP_NCI_MODE_NCI) {
- r = -EINVAL;
- goto send_exit;
- }
+ if (info->mode != NXP_NCI_MODE_NCI)
+ return -EINVAL;
r = info->phy_ops->write(info->phy_id, skb);
if (r < 0)
kfree_skb(skb);
-send_exit:
return r;
}
int r;
info = devm_kzalloc(pdev, sizeof(struct nxp_nci_info), GFP_KERNEL);
- if (!info) {
- r = -ENOMEM;
- goto probe_exit;
- }
+ if (!info)
+ return -ENOMEM;
info->phy_id = phy_id;
info->pdev = pdev;
if (info->phy_ops->set_mode) {
r = info->phy_ops->set_mode(info->phy_id, NXP_NCI_MODE_COLD);
if (r < 0)
- goto probe_exit;
+ return r;
}
info->mode = NXP_NCI_MODE_COLD;
info->ndev = nci_allocate_device(&nxp_nci_ops, NXP_NCI_NFC_PROTOCOLS,
NXP_NCI_HDR_LEN, 0);
- if (!info->ndev) {
- r = -ENOMEM;
- goto probe_exit;
- }
+ if (!info->ndev)
+ return -ENOMEM;
nci_set_parent_dev(info->ndev, pdev);
nci_set_drvdata(info->ndev, info);
r = nci_register_device(info->ndev);
- if (r < 0)
- goto probe_exit_free_nci;
+ if (r < 0) {
+ nci_free_device(info->ndev);
+ return r;
+ }
*ndev = info->ndev;
-
- goto probe_exit;
-
-probe_exit_free_nci:
- nci_free_device(info->ndev);
-probe_exit:
return r;
}
EXPORT_SYMBOL(nxp_nci_probe);
int r;
skb = nci_skb_alloc(info->ndev, info->max_payload, GFP_KERNEL);
- if (!skb) {
- r = -ENOMEM;
- goto chunk_exit;
- }
+ if (!skb)
+ return -ENOMEM;
chunk_len = info->max_payload - NXP_NCI_FW_HDR_LEN - NXP_NCI_FW_CRC_LEN;
remaining_len = fw_info->frame_size - fw_info->written;
kfree_skb(skb);
-chunk_exit:
return r;
}
struct pn533 *priv;
int r = 0;
- dev_dbg(&client->dev, "%s\n", __func__);
- dev_dbg(&client->dev, "IRQ: %d\n", client->irq);
-
if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) {
nfc_err(&client->dev, "Need I2C_FUNC_I2C\n");
return -ENODEV;
phy, &i2c_phy_ops, NULL,
&phy->i2c_dev->dev);
- if (IS_ERR(priv)) {
+ if (IS_ERR(priv))
return PTR_ERR(priv);
- }
phy->priv = priv;
r = pn532_i2c_nfc_alloc(priv, PN533_NO_TYPE_B_PROTOCOLS, &client->dev);
{
struct pn533_i2c_phy *phy = i2c_get_clientdata(client);
- dev_dbg(&client->dev, "%s\n", __func__);
-
free_irq(client->irq, phy);
pn53x_unregister_nfc(phy->priv);
return 0;
}
-static const struct of_device_id of_pn533_i2c_match[] = {
+static const struct of_device_id of_pn533_i2c_match[] __maybe_unused = {
{ .compatible = "nxp,pn532", },
/*
* NOTE: The use of the compatibles with the trailing "...-i2c" is
u8 status, ret, mi;
int rc;
- dev_dbg(dev->dev, "%s\n", __func__);
-
if (IS_ERR(resp)) {
skb_queue_purge(&dev->resp_q);
return PTR_ERR(resp);
struct sk_buff *skb;
int rc;
- dev_dbg(dev->dev, "%s\n", __func__);
-
skb = pn533_alloc_skb(dev, 0);
if (!skb)
return;
struct sk_buff *skb;
int rc;
- dev_dbg(dev->dev, "%s\n", __func__);
-
/* Grab the first skb in the queue */
skb = skb_dequeue(&dev->fragment_skb);
if (skb == NULL) { /* No more data */
struct sk_buff *skb;
int rc;
- dev_dbg(dev->dev, "%s\n", __func__);
-
skb = pn533_alloc_skb(dev, 0);
if (!skb)
return;
size_t gb_len;
int rc;
- dev_dbg(dev->dev, "%s\n", __func__);
-
if (resp->len < ATR_REQ_GB_OFFSET + 1)
return -EINVAL;
{
int rc = 0;
- dev_dbg(dev->dev, "%s\n", __func__);
-
if (IS_ERR(resp)) {
rc = PTR_ERR(resp);
struct sk_buff *skb;
int rc;
- dev_dbg(dev->dev, "%s\n", __func__);
-
skb = pn533_alloc_skb(dev, 2);
if (!skb)
return;
u8 *next, nfcid3[NFC_NFCID3_MAXSIZE];
u8 passive_data[PASSIVE_DATA_LEN] = {0x00, 0xff, 0xff, 0x00, 0x3};
- dev_dbg(dev->dev, "%s", __func__);
-
if (!dev->gb) {
dev->gb = nfc_get_local_general_bytes(nfc_dev, &dev->gb_len);
struct pn533_poll_modulations *cur_mod;
int rc;
- dev_dbg(dev->dev, "%s\n", __func__);
-
if (IS_ERR(resp)) {
rc = PTR_ERR(resp);
struct sk_buff *skb;
struct sk_buff *resp;
- dev_dbg(dev->dev, "%s\n", __func__);
-
skb = pn533_alloc_skb(dev, sizeof(u8) * 2); /*TG + Next*/
if (!skb)
return -ENOMEM;
{
int rc = 0;
- dev_dbg(dev->dev, "%s\n", __func__);
-
if (IS_ERR(resp)) {
rc = PTR_ERR(resp);
struct sk_buff *skb;
int rc;
- dev_dbg(dev->dev, "%s\n", __func__);
-
if (!dev->tgt_active_prot) {
nfc_err(dev->dev, "There is no active target\n");
return;
u8 *next, *arg, nfcid3[NFC_NFCID3_MAXSIZE];
u8 passive_data[PASSIVE_DATA_LEN] = {0x00, 0xff, 0xff, 0x00, 0x3};
- dev_dbg(dev->dev, "%s\n", __func__);
-
if (dev->poll_mod_count) {
nfc_err(dev->dev,
"Cannot bring the DEP link up while polling\n");
{
struct pn533 *dev = nfc_get_drvdata(nfc_dev);
- dev_dbg(dev->dev, "%s\n", __func__);
-
pn533_poll_reset_mod_list(dev);
if (dev->tgt_mode || dev->tgt_active_prot)
struct sk_buff *skb, *tmp, *t;
unsigned int skb_len = 0, tmp_len = 0;
- dev_dbg(dev->dev, "%s\n", __func__);
-
if (skb_queue_empty(&dev->resp_q))
return NULL;
int rc = 0;
u8 status, ret, mi;
- dev_dbg(dev->dev, "%s\n", __func__);
-
if (IS_ERR(resp)) {
rc = PTR_ERR(resp);
goto _error;
struct pn533_data_exchange_arg *arg = NULL;
int rc;
- dev_dbg(dev->dev, "%s\n", __func__);
-
if (!dev->tgt_active_prot) {
nfc_err(dev->dev,
"Can't exchange data if there is no active target\n");
{
u8 status;
- dev_dbg(dev->dev, "%s\n", __func__);
-
if (IS_ERR(resp))
return PTR_ERR(resp);
struct pn533 *dev = nfc_get_drvdata(nfc_dev);
int rc;
- dev_dbg(dev->dev, "%s\n", __func__);
-
/* let's split in multiple chunks if size's too big */
if (skb->len > PN533_CMD_DATAEXCH_DATA_MAXLEN) {
rc = pn533_fill_fragment_skbs(dev, skb);
struct sk_buff *skb;
int rc;
- dev_dbg(dev->dev, "%s\n", __func__);
-
skb = pn533_alloc_skb(dev, PN533_CMD_DATAEXCH_HEAD_LEN);
if (!skb)
goto error;
struct sk_buff *skb;
int rc;
- dev_dbg(dev->dev, "%s\n", __func__);
-
/* Grab the first skb in the queue */
skb = skb_dequeue(&dev->fragment_skb);
struct sk_buff *resp;
int skb_len;
- dev_dbg(dev->dev, "%s\n", __func__);
-
skb_len = sizeof(cfgitem) + cfgdata_len; /* cfgitem + cfgdata */
skb = pn533_alloc_skb(dev, skb_len);
struct sk_buff *skb;
struct sk_buff *resp;
- dev_dbg(dev->dev, "%s\n", __func__);
-
skb = pn533_alloc_skb(dev, sizeof(u8));
if (!skb)
return -ENOMEM;
.remove = pn532_uart_remove,
.driver = {
.name = "pn532_uart",
- .of_match_table = of_match_ptr(pn532_uart_of_match),
+ .of_match_table = pn532_uart_of_match,
},
};
{
struct pn533_acr122_poweron_rdr_arg *arg = urb->context;
- dev_dbg(&urb->dev->dev, "%s\n", __func__);
-
print_hex_dump_debug("ACR122 RX: ", DUMP_PREFIX_NONE, 16, 1,
urb->transfer_buffer, urb->transfer_buffer_length,
false);
void *cntx;
struct pn533_acr122_poweron_rdr_arg arg;
- dev_dbg(&phy->udev->dev, "%s\n", __func__);
-
buffer = kmemdup(cmd, sizeof(cmd), GFP_KERNEL);
if (!buffer)
return -ENOMEM;
MODULE_DEVICE_TABLE(i2c, pn544_hci_i2c_id_table);
-static const struct acpi_device_id pn544_hci_i2c_acpi_match[] = {
+static const struct acpi_device_id pn544_hci_i2c_acpi_match[] __maybe_unused = {
{"NXP5440", 0},
{}
};
{
struct pn544_i2c_phy *phy = phy_id;
- pr_info("%s\n", __func__);
-
pn544_hci_i2c_enable_mode(phy, PN544_HCI_MODE);
phy->powered = 1;
struct pn544_i2c_phy *phy;
int r = 0;
- dev_dbg(&client->dev, "%s\n", __func__);
- dev_dbg(&client->dev, "IRQ: %d\n", client->irq);
-
if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) {
nfc_err(&client->dev, "Need I2C_FUNC_I2C\n");
return -ENODEV;
{
struct pn544_i2c_phy *phy = i2c_get_clientdata(client);
- dev_dbg(&client->dev, "%s\n", __func__);
-
cancel_work_sync(&phy->fw_work);
if (phy->fw_work_state != FW_WORK_STATE_IDLE)
pn544_hci_i2c_fw_work_complete(phy, -ENODEV);
return 0;
}
-static const struct of_device_id of_pn544_i2c_match[] = {
+static const struct of_device_id of_pn544_i2c_match[] __maybe_unused = {
{ .compatible = "nxp,pn544-i2c", },
{},
};
typedef void (*port100_send_async_complete_t)(struct port100 *dev, void *arg,
struct sk_buff *resp);
-/**
+/*
* Setting sets structure for in_set_rf command
*
* @in_*_set_number: Represent the entry indexes in the port-100 RF Base Table.
};
/**
- * Setting sets structure for tg_set_rf command
+ * struct port100_tg_rf_setting - Setting sets structure for tg_set_rf command
*
* @tg_set_number: Represents the entry index in the port-100 RF Base Table.
* This table contains multiple RF setting sets required for RF
* Robert Baldyga <r.baldyga@samsung.com>
*/
+#include <linux/clk.h>
#include <linux/i2c.h>
#include <linux/gpio.h>
#include <linux/delay.h>
struct s3fwrn5_i2c_phy {
struct phy_common common;
struct i2c_client *i2c_dev;
+ struct clk *clk;
unsigned int irq_skip:1;
};
if (ret < 0)
return ret;
+ phy->clk = devm_clk_get_optional(&client->dev, NULL);
+ if (IS_ERR(phy->clk))
+ return dev_err_probe(&client->dev, PTR_ERR(phy->clk),
+ "failed to get clock\n");
+
+ /*
+ * S3FWRN5 depends on a clock input ("XI" pin) to function properly.
+ * Depending on the hardware configuration this could be an always-on
+ * oscillator or some external clock that must be explicitly enabled.
+ * Make sure the clock is running before starting S3FWRN5.
+ */
+ ret = clk_prepare_enable(phy->clk);
+ if (ret < 0) {
+ dev_err(&client->dev, "failed to enable clock: %d\n", ret);
+ return ret;
+ }
+
ret = s3fwrn5_probe(&phy->common.ndev, phy, &phy->i2c_dev->dev,
&i2c_phy_ops);
if (ret < 0)
- return ret;
+ goto disable_clk;
ret = devm_request_threaded_irq(&client->dev, phy->i2c_dev->irq, NULL,
s3fwrn5_i2c_irq_thread_fn, IRQF_ONESHOT,
S3FWRN5_I2C_DRIVER_NAME, phy);
if (ret)
- s3fwrn5_remove(phy->common.ndev);
+ goto s3fwrn5_remove;
+ return 0;
+
+s3fwrn5_remove:
+ s3fwrn5_remove(phy->common.ndev);
+disable_clk:
+ clk_disable_unprepare(phy->clk);
return ret;
}
struct s3fwrn5_i2c_phy *phy = i2c_get_clientdata(client);
s3fwrn5_remove(phy->common.ndev);
+ clk_disable_unprepare(phy->clk);
return 0;
}
};
MODULE_DEVICE_TABLE(i2c, s3fwrn5_i2c_id_table);
-static const struct of_device_id of_s3fwrn5_i2c_match[] = {
+static const struct of_device_id of_s3fwrn5_i2c_match[] __maybe_unused = {
{ .compatible = "samsung,s3fwrn5-i2c", },
{}
};
struct st_nci_i2c_phy *phy;
int r;
- dev_dbg(&client->dev, "%s\n", __func__);
- dev_dbg(&client->dev, "IRQ: %d\n", client->irq);
-
if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) {
nfc_err(&client->dev, "Need I2C_FUNC_I2C\n");
return -ENODEV;
{
struct st_nci_i2c_phy *phy = i2c_get_clientdata(client);
- dev_dbg(&client->dev, "%s\n", __func__);
-
ndlc_remove(phy->ndlc);
return 0;
};
MODULE_DEVICE_TABLE(i2c, st_nci_i2c_id_table);
-static const struct acpi_device_id st_nci_i2c_acpi_match[] = {
+static const struct acpi_device_id st_nci_i2c_acpi_match[] __maybe_unused = {
{"SMO2101"},
{"SMO2102"},
{}
};
MODULE_DEVICE_TABLE(acpi, st_nci_i2c_acpi_match);
-static const struct of_device_id of_st_nci_i2c_match[] = {
+static const struct of_device_id of_st_nci_i2c_match[] __maybe_unused = {
{ .compatible = "st,st21nfcb-i2c", },
{ .compatible = "st,st21nfcb_i2c", },
{ .compatible = "st,st21nfcc-i2c", },
{
int r;
- pr_debug("st_nci_disable_se\n");
-
/*
* According to upper layer, se_idx == NFC_SE_UICC when
* info->se_info.se_status->is_uicc_enable is true should never happen
{
int r;
- pr_debug("st_nci_enable_se\n");
-
/*
* According to upper layer, se_idx == NFC_SE_UICC when
* info->se_info.se_status->is_uicc_enable is true should never happen.
dest_params =
kzalloc(sizeof(struct core_conn_create_dest_spec_params) +
sizeof(struct dest_spec_params), GFP_KERNEL);
- if (dest_params == NULL) {
- r = -ENOMEM;
- goto exit;
- }
+ if (dest_params == NULL)
+ return -ENOMEM;
dest_params->type = NCI_DESTINATION_SPECIFIC_PARAM_NFCEE_TYPE;
dest_params->length = sizeof(struct dest_spec_params);
free_dest_params:
kfree(dest_params);
-
-exit:
return r;
}
int se_count = 0;
struct st_nci_info *info = nci_get_drvdata(ndev);
- pr_debug("st_nci_discover_se\n");
-
r = st_nci_hci_network_init(ndev);
if (r != 0)
return r;
struct st_nci_spi_phy *phy;
int r;
- dev_dbg(&dev->dev, "%s\n", __func__);
- dev_dbg(&dev->dev, "IRQ: %d\n", dev->irq);
-
/* Check SPI platform functionnalities */
if (!dev) {
pr_debug("%s: dev is NULL. Device is not accessible.\n",
{
struct st_nci_spi_phy *phy = spi_get_drvdata(dev);
- dev_dbg(&dev->dev, "%s\n", __func__);
-
ndlc_remove(phy->ndlc);
return 0;
};
MODULE_DEVICE_TABLE(spi, st_nci_spi_id_table);
-static const struct acpi_device_id st_nci_spi_acpi_match[] = {
+static const struct acpi_device_id st_nci_spi_acpi_match[] __maybe_unused = {
{"SMO2101", 0},
{}
};
MODULE_DEVICE_TABLE(acpi, st_nci_spi_acpi_match);
-static const struct of_device_id of_st_nci_spi_match[] = {
+static const struct of_device_id of_st_nci_spi_match[] __maybe_unused = {
{ .compatible = "st,st21nfcb-spi", },
{}
};
r = nci_hci_send_cmd(ndev, ST_NCI_DEVICE_MGNT_GATE, ST_NCI_HCI_DM_GETINFO,
data, data_len, &skb);
if (r)
- goto exit;
+ return r;
msg = nfc_vendor_cmd_alloc_reply_skb(dev, ST_NCI_VENDOR_OUI,
HCI_DM_GET_INFO, skb->len);
free_skb:
kfree_skb(skb);
-exit:
return r;
}
r = nci_hci_send_cmd(ndev, ST_NCI_DEVICE_MGNT_GATE, ST_NCI_HCI_DM_GETDATA,
data, data_len, &skb);
if (r)
- goto exit;
+ return r;
msg = nfc_vendor_cmd_alloc_reply_skb(dev, ST_NCI_VENDOR_OUI,
HCI_DM_GET_DATA, skb->len);
free_skb:
kfree_skb(skb);
-exit:
return r;
}
r = nci_hci_get_param(ndev, param->gate, param->data, &skb);
if (r)
- goto exit;
+ return r;
msg = nfc_vendor_cmd_alloc_reply_skb(dev, ST_NCI_VENDOR_OUI,
HCI_GET_PARAM, skb->len);
free_skb:
kfree_skb(skb);
-exit:
return r;
}
ST_NCI_HCI_DM_VDC_MEASUREMENT_VALUE,
data, data_len, &skb);
if (r)
- goto exit;
+ return r;
msg = nfc_vendor_cmd_alloc_reply_skb(dev, ST_NCI_VENDOR_OUI,
HCI_DM_VDC_MEASUREMENT_VALUE, skb->len);
free_skb:
kfree_skb(skb);
-exit:
return r;
}
ST_NCI_HCI_DM_VDC_VALUE_COMPARISON,
data, data_len, &skb);
if (r)
- goto exit;
+ return r;
msg = nfc_vendor_cmd_alloc_reply_skb(dev, ST_NCI_VENDOR_OUI,
HCI_DM_VDC_VALUE_COMPARISON, skb->len);
free_skb:
kfree_skb(skb);
-exit:
return r;
}
skb_trim(skb, skb->len - 1);
- if (!skb->len) {
- r = -EIO;
- goto exit;
- }
+ if (!skb->len)
+ return -EIO;
- if (skb->len < ST21NFCA_ATR_REQ_MIN_SIZE) {
- r = -EPROTO;
- goto exit;
- }
+ if (skb->len < ST21NFCA_ATR_REQ_MIN_SIZE)
+ return -EPROTO;
atr_req = (struct st21nfca_atr_req *)skb->data;
- if (atr_req->length < sizeof(struct st21nfca_atr_req)) {
- r = -EPROTO;
- goto exit;
- }
+ if (atr_req->length < sizeof(struct st21nfca_atr_req))
+ return -EPROTO;
r = st21nfca_tm_send_atr_res(hdev, atr_req);
if (r)
- goto exit;
+ return r;
gb_len = skb->len - sizeof(struct st21nfca_atr_req);
r = nfc_tm_activated(hdev->ndev, NFC_PROTO_NFC_DEP_MASK,
NFC_COMM_PASSIVE, atr_req->gbi, gb_len);
if (r)
- goto exit;
-
- r = 0;
+ return r;
-exit:
- return r;
+ return 0;
}
static int st21nfca_tm_send_psl_res(struct nfc_hci_dev *hdev,
struct sk_buff *skb)
{
struct st21nfca_psl_req *psl_req;
- int r;
skb_trim(skb, skb->len - 1);
- if (!skb->len) {
- r = -EIO;
- goto exit;
- }
+ if (!skb->len)
+ return -EIO;
psl_req = (struct st21nfca_psl_req *)skb->data;
- if (skb->len < sizeof(struct st21nfca_psl_req)) {
- r = -EIO;
- goto exit;
- }
+ if (skb->len < sizeof(struct st21nfca_psl_req))
+ return -EIO;
- r = st21nfca_tm_send_psl_res(hdev, psl_req);
-exit:
- return r;
+ return st21nfca_tm_send_psl_res(hdev, psl_req);
}
int st21nfca_tm_send_dep_res(struct nfc_hci_dev *hdev, struct sk_buff *skb)
{
struct st21nfca_dep_req_res *dep_req;
u8 size;
- int r;
struct st21nfca_hci_info *info = nfc_hci_get_clientdata(hdev);
skb_trim(skb, skb->len - 1);
size = 4;
dep_req = (struct st21nfca_dep_req_res *)skb->data;
- if (skb->len < size) {
- r = -EIO;
- goto exit;
- }
+ if (skb->len < size)
+ return -EIO;
if (ST21NFCA_NFC_DEP_DID_BIT_SET(dep_req->pfb))
size++;
if (ST21NFCA_NFC_DEP_NAD_BIT_SET(dep_req->pfb))
size++;
- if (skb->len < size) {
- r = -EIO;
- goto exit;
- }
+ if (skb->len < size)
+ return -EIO;
/* Receiving DEP_REQ - Decoding */
switch (ST21NFCA_NFC_DEP_PFB_TYPE(dep_req->pfb)) {
skb_pull(skb, size);
return nfc_tm_data_received(hdev->ndev, skb);
-exit:
- return r;
}
static int st21nfca_tm_event_send_data(struct nfc_hci_dev *hdev,
struct st21nfca_i2c_phy *phy;
int r;
- dev_dbg(&client->dev, "%s\n", __func__);
- dev_dbg(&client->dev, "IRQ: %d\n", client->irq);
-
if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) {
nfc_err(&client->dev, "Need I2C_FUNC_I2C\n");
return -ENODEV;
{
struct st21nfca_i2c_phy *phy = i2c_get_clientdata(client);
- dev_dbg(&client->dev, "%s\n", __func__);
-
st21nfca_hci_remove(phy->hdev);
if (phy->powered)
};
MODULE_DEVICE_TABLE(i2c, st21nfca_hci_i2c_id_table);
-static const struct acpi_device_id st21nfca_hci_i2c_acpi_match[] = {
+static const struct acpi_device_id st21nfca_hci_i2c_acpi_match[] __maybe_unused = {
{"SMO2100", 0},
{}
};
MODULE_DEVICE_TABLE(acpi, st21nfca_hci_i2c_acpi_match);
-static const struct of_device_id of_st21nfca_i2c_match[] = {
+static const struct of_device_id of_st21nfca_i2c_match[] __maybe_unused = {
{ .compatible = "st,st21nfca-i2c", },
{ .compatible = "st,st21nfca_i2c", },
{}
int len_data_to_tag = 0;
skb_resp = nfc_alloc_recv_skb(MAX_RESPONSE_BUFFER_SIZE, GFP_KERNEL);
- if (!skb_resp) {
- rc = -ENOMEM;
- goto error;
- }
+ if (!skb_resp)
+ return -ENOMEM;
switch (stcontext->current_rf_tech) {
case NFC_DIGITAL_RF_TECH_106A:
free_skb_resp:
kfree_skb(skb_resp);
-error:
return rc;
}
};
MODULE_DEVICE_TABLE(spi, st95hf_id);
-static const struct of_device_id st95hf_spi_of_match[] = {
- { .compatible = "st,st95hf" },
- { },
+static const struct of_device_id st95hf_spi_of_match[] __maybe_unused = {
+ { .compatible = "st,st95hf" },
+ {},
};
MODULE_DEVICE_TABLE(of, st95hf_spi_of_match);
config NVME_TCP
tristate "NVM Express over Fabrics TCP host driver"
depends on INET
- depends on BLK_DEV_NVME
+ depends on BLOCK
+ select NVME_CORE
select NVME_FABRICS
select CRYPTO
select CRYPTO_CRC32C
ctrl->hmmaxd = le16_to_cpu(id->hmmaxd);
}
- ret = nvme_mpath_init(ctrl, id);
+ ret = nvme_mpath_init_identify(ctrl, id);
if (ret < 0)
goto out_free;
cdev_init(cdev, fops);
cdev->owner = owner;
ret = cdev_device_add(cdev, cdev_device);
- if (ret)
+ if (ret) {
+ put_device(cdev_device);
ida_simple_remove(&nvme_ns_chr_minor_ida, minor);
+ }
return ret;
}
min(default_ps_max_latency_us, (unsigned long)S32_MAX));
nvme_fault_inject_init(&ctrl->fault_inject, dev_name(ctrl->device));
+ nvme_mpath_init_ctrl(ctrl);
return 0;
out_free_name:
cmd->connect.recfmt);
break;
+ case NVME_SC_HOST_PATH_ERROR:
+ dev_err(ctrl->device,
+ "Connect command failed: host path error\n");
+ break;
+
default:
dev_err(ctrl->device,
"Connect command failed, error wo/DNR bit: %d\n",
static void
__nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
{
+ int q;
+
+ /*
+ * if aborting io, the queues are no longer good, mark them
+ * all as not live.
+ */
+ if (ctrl->ctrl.queue_count > 1) {
+ for (q = 1; q < ctrl->ctrl.queue_count; q++)
+ clear_bit(NVME_FC_Q_LIVE, &ctrl->queues[q].flags);
+ }
+ clear_bit(NVME_FC_Q_LIVE, &ctrl->queues[0].flags);
+
/*
* If io queues are present, stop them and terminate all outstanding
* ios on them. As FC allocates FC exchange for each io, the
if (ctrl->ctrl.icdoff) {
dev_err(ctrl->ctrl.device, "icdoff %d is not supported!\n",
ctrl->ctrl.icdoff);
+ ret = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
goto out_disconnect_admin_queue;
}
if (!(ctrl->ctrl.sgls & ((1 << 0) | (1 << 1)))) {
dev_err(ctrl->ctrl.device,
"Mandatory sgls are not supported!\n");
+ ret = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
goto out_disconnect_admin_queue;
}
if (ctrl->ctrl.state != NVME_CTRL_CONNECTING)
return;
- if (portptr->port_state == FC_OBJSTATE_ONLINE)
+ if (portptr->port_state == FC_OBJSTATE_ONLINE) {
dev_info(ctrl->ctrl.device,
"NVME-FC{%d}: reset: Reconnect attempt failed (%d)\n",
ctrl->cnum, status);
- else if (time_after_eq(jiffies, rport->dev_loss_end))
+ if (status > 0 && (status & NVME_SC_DNR))
+ recon = false;
+ } else if (time_after_eq(jiffies, rport->dev_loss_end))
recon = false;
if (recon && nvmf_should_reconnect(&ctrl->ctrl)) {
queue_delayed_work(nvme_wq, &ctrl->connect_work, recon_delay);
} else {
- if (portptr->port_state == FC_OBJSTATE_ONLINE)
- dev_warn(ctrl->ctrl.device,
- "NVME-FC{%d}: Max reconnect attempts (%d) "
- "reached.\n",
- ctrl->cnum, ctrl->ctrl.nr_reconnects);
- else
+ if (portptr->port_state == FC_OBJSTATE_ONLINE) {
+ if (status > 0 && (status & NVME_SC_DNR))
+ dev_warn(ctrl->ctrl.device,
+ "NVME-FC{%d}: reconnect failure\n",
+ ctrl->cnum);
+ else
+ dev_warn(ctrl->ctrl.device,
+ "NVME-FC{%d}: Max reconnect attempts "
+ "(%d) reached.\n",
+ ctrl->cnum, ctrl->ctrl.nr_reconnects);
+ } else
dev_warn(ctrl->ctrl.device,
"NVME-FC{%d}: dev_loss_tmo (%d) expired "
"while waiting for remoteport connectivity.\n",
put_disk(head->disk);
}
-int nvme_mpath_init(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
+void nvme_mpath_init_ctrl(struct nvme_ctrl *ctrl)
{
- int error;
+ mutex_init(&ctrl->ana_lock);
+ timer_setup(&ctrl->anatt_timer, nvme_anatt_timeout, 0);
+ INIT_WORK(&ctrl->ana_work, nvme_ana_work);
+}
+
+int nvme_mpath_init_identify(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
+{
+ size_t max_transfer_size = ctrl->max_hw_sectors << SECTOR_SHIFT;
+ size_t ana_log_size;
+ int error = 0;
/* check if multipath is enabled and we have the capability */
if (!multipath || !ctrl->subsys ||
ctrl->nanagrpid = le32_to_cpu(id->nanagrpid);
ctrl->anagrpmax = le32_to_cpu(id->anagrpmax);
- mutex_init(&ctrl->ana_lock);
- timer_setup(&ctrl->anatt_timer, nvme_anatt_timeout, 0);
- ctrl->ana_log_size = sizeof(struct nvme_ana_rsp_hdr) +
- ctrl->nanagrpid * sizeof(struct nvme_ana_group_desc);
- ctrl->ana_log_size += ctrl->max_namespaces * sizeof(__le32);
-
- if (ctrl->ana_log_size > ctrl->max_hw_sectors << SECTOR_SHIFT) {
+ ana_log_size = sizeof(struct nvme_ana_rsp_hdr) +
+ ctrl->nanagrpid * sizeof(struct nvme_ana_group_desc) +
+ ctrl->max_namespaces * sizeof(__le32);
+ if (ana_log_size > max_transfer_size) {
dev_err(ctrl->device,
- "ANA log page size (%zd) larger than MDTS (%d).\n",
- ctrl->ana_log_size,
- ctrl->max_hw_sectors << SECTOR_SHIFT);
+ "ANA log page size (%zd) larger than MDTS (%zd).\n",
+ ana_log_size, max_transfer_size);
dev_err(ctrl->device, "disabling ANA support.\n");
- return 0;
+ goto out_uninit;
}
-
- INIT_WORK(&ctrl->ana_work, nvme_ana_work);
- kfree(ctrl->ana_log_buf);
- ctrl->ana_log_buf = kmalloc(ctrl->ana_log_size, GFP_KERNEL);
- if (!ctrl->ana_log_buf) {
- error = -ENOMEM;
- goto out;
+ if (ana_log_size > ctrl->ana_log_size) {
+ nvme_mpath_stop(ctrl);
+ kfree(ctrl->ana_log_buf);
+ ctrl->ana_log_buf = kmalloc(ana_log_size, GFP_KERNEL);
+ if (!ctrl->ana_log_buf)
+ return -ENOMEM;
}
-
+ ctrl->ana_log_size = ana_log_size;
error = nvme_read_ana_log(ctrl);
if (error)
- goto out_free_ana_log_buf;
+ goto out_uninit;
return 0;
-out_free_ana_log_buf:
- kfree(ctrl->ana_log_buf);
- ctrl->ana_log_buf = NULL;
-out:
+
+out_uninit:
+ nvme_mpath_uninit(ctrl);
return error;
}
int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl,struct nvme_ns_head *head);
void nvme_mpath_add_disk(struct nvme_ns *ns, struct nvme_id_ns *id);
void nvme_mpath_remove_disk(struct nvme_ns_head *head);
-int nvme_mpath_init(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id);
+int nvme_mpath_init_identify(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id);
+void nvme_mpath_init_ctrl(struct nvme_ctrl *ctrl);
void nvme_mpath_uninit(struct nvme_ctrl *ctrl);
void nvme_mpath_stop(struct nvme_ctrl *ctrl);
bool nvme_mpath_clear_current_path(struct nvme_ns *ns);
static inline void nvme_trace_bio_complete(struct request *req)
{
}
-static inline int nvme_mpath_init(struct nvme_ctrl *ctrl,
+static inline void nvme_mpath_init_ctrl(struct nvme_ctrl *ctrl)
+{
+}
+static inline int nvme_mpath_init_identify(struct nvme_ctrl *ctrl,
struct nvme_id_ctrl *id)
{
if (ctrl->subsys->cmic & NVME_CTRL_CMIC_ANA)
int count)
{
struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
- struct scatterlist *sgl = req->data_sgl.sg_table.sgl;
struct ib_sge *sge = &req->sge[1];
+ struct scatterlist *sgl;
u32 len = 0;
int i;
- for (i = 0; i < count; i++, sgl++, sge++) {
+ for_each_sg(req->data_sgl.sg_table.sgl, sgl, count, i) {
sge->addr = sg_dma_address(sgl);
sge->length = sg_dma_len(sgl);
sge->lkey = queue->device->pd->local_dma_lkey;
len += sge->length;
+ sge++;
}
sg->addr = cpu_to_le64(queue->ctrl->ctrl.icdoff);
if (ret <= 0)
return ret;
- nvme_tcp_advance_req(req, ret);
if (queue->data_digest)
nvme_tcp_ddgst_update(queue->snd_hash, page,
offset, ret);
}
return 1;
}
+ nvme_tcp_advance_req(req, ret);
}
return -EAGAIN;
}
pending = true;
else if (unlikely(result < 0))
break;
- }
+ } else
+ pending = !llist_empty(&queue->req_list);
result = nvme_tcp_try_recv(queue);
if (result > 0)
case nvme_admin_keep_alive:
req->execute = nvmet_execute_keep_alive;
return 0;
+ default:
+ return nvmet_report_invalid_opcode(req);
}
-
- pr_debug("unhandled cmd %d on qid %d\n", cmd->common.opcode,
- req->sq->qid);
- req->error_loc = offsetof(struct nvme_common_command, opcode);
- return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
}
{
struct nvmet_ctrl *ctrl = container_of(to_delayed_work(work),
struct nvmet_ctrl, ka_work);
- bool cmd_seen = ctrl->cmd_seen;
+ bool reset_tbkas = ctrl->reset_tbkas;
- ctrl->cmd_seen = false;
- if (cmd_seen) {
+ ctrl->reset_tbkas = false;
+ if (reset_tbkas) {
pr_debug("ctrl %d reschedule traffic based keep-alive timer\n",
ctrl->cntlid);
schedule_delayed_work(&ctrl->ka_work, ctrl->kato * HZ);
percpu_ref_exit(&sq->ref);
if (ctrl) {
+ /*
+ * The teardown flow may take some time, and the host may not
+ * send us keep-alive during this period, hence reset the
+ * traffic based keep-alive timer so we don't trigger a
+ * controller teardown as a result of a keep-alive expiration.
+ */
+ ctrl->reset_tbkas = true;
nvmet_ctrl_put(ctrl);
sq->ctrl = NULL; /* allows reusing the queue later */
}
}
if (sq->ctrl)
- sq->ctrl->cmd_seen = true;
+ sq->ctrl->reset_tbkas = true;
return true;
return req->transfer_len - req->metadata_len;
}
-static int nvmet_req_alloc_p2pmem_sgls(struct nvmet_req *req)
+static int nvmet_req_alloc_p2pmem_sgls(struct pci_dev *p2p_dev,
+ struct nvmet_req *req)
{
- req->sg = pci_p2pmem_alloc_sgl(req->p2p_dev, &req->sg_cnt,
+ req->sg = pci_p2pmem_alloc_sgl(p2p_dev, &req->sg_cnt,
nvmet_data_transfer_len(req));
if (!req->sg)
goto out_err;
if (req->metadata_len) {
- req->metadata_sg = pci_p2pmem_alloc_sgl(req->p2p_dev,
+ req->metadata_sg = pci_p2pmem_alloc_sgl(p2p_dev,
&req->metadata_sg_cnt, req->metadata_len);
if (!req->metadata_sg)
goto out_free_sg;
}
+
+ req->p2p_dev = p2p_dev;
+
return 0;
out_free_sg:
pci_p2pmem_free_sgl(req->p2p_dev, req->sg);
return -ENOMEM;
}
-static bool nvmet_req_find_p2p_dev(struct nvmet_req *req)
+static struct pci_dev *nvmet_req_find_p2p_dev(struct nvmet_req *req)
{
- if (!IS_ENABLED(CONFIG_PCI_P2PDMA))
- return false;
-
- if (req->sq->ctrl && req->sq->qid && req->ns) {
- req->p2p_dev = radix_tree_lookup(&req->sq->ctrl->p2p_ns_map,
- req->ns->nsid);
- if (req->p2p_dev)
- return true;
- }
-
- req->p2p_dev = NULL;
- return false;
+ if (!IS_ENABLED(CONFIG_PCI_P2PDMA) ||
+ !req->sq->ctrl || !req->sq->qid || !req->ns)
+ return NULL;
+ return radix_tree_lookup(&req->sq->ctrl->p2p_ns_map, req->ns->nsid);
}
int nvmet_req_alloc_sgls(struct nvmet_req *req)
{
- if (nvmet_req_find_p2p_dev(req) && !nvmet_req_alloc_p2pmem_sgls(req))
+ struct pci_dev *p2p_dev = nvmet_req_find_p2p_dev(req);
+
+ if (p2p_dev && !nvmet_req_alloc_p2pmem_sgls(p2p_dev, req))
return 0;
req->sg = sgl_alloc(nvmet_data_transfer_len(req), GFP_KERNEL,
pci_p2pmem_free_sgl(req->p2p_dev, req->sg);
if (req->metadata_sg)
pci_p2pmem_free_sgl(req->p2p_dev, req->metadata_sg);
+ req->p2p_dev = NULL;
} else {
sgl_free(req->sg);
if (req->metadata_sg)
goto out_free_changed_ns_list;
if (subsys->cntlid_min > subsys->cntlid_max)
- goto out_free_changed_ns_list;
+ goto out_free_sqs;
ret = ida_simple_get(&cntlid_ida,
subsys->cntlid_min, subsys->cntlid_max,
req->execute = nvmet_execute_disc_identify;
return 0;
default:
- pr_err("unhandled cmd %d\n", cmd->common.opcode);
+ pr_debug("unhandled cmd %d\n", cmd->common.opcode);
req->error_loc = offsetof(struct nvme_common_command, opcode);
return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
}
req->execute = nvmet_execute_prop_get;
break;
default:
- pr_err("received unknown capsule type 0x%x\n",
+ pr_debug("received unknown capsule type 0x%x\n",
cmd->fabrics.fctype);
req->error_loc = offsetof(struct nvmf_common_command, fctype);
return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
struct nvme_command *cmd = req->cmd;
if (!nvme_is_fabrics(cmd)) {
- pr_err("invalid command 0x%x on unconnected queue.\n",
+ pr_debug("invalid command 0x%x on unconnected queue.\n",
cmd->fabrics.opcode);
req->error_loc = offsetof(struct nvme_common_command, opcode);
return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
}
if (cmd->fabrics.fctype != nvme_fabrics_type_connect) {
- pr_err("invalid capsule type 0x%x on unconnected queue.\n",
+ pr_debug("invalid capsule type 0x%x on unconnected queue.\n",
cmd->fabrics.fctype);
req->error_loc = offsetof(struct nvmf_common_command, fctype);
return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
sector = nvmet_lba_to_sect(req->ns, req->cmd->rw.slba);
- if (req->transfer_len <= NVMET_MAX_INLINE_DATA_LEN) {
+ if (nvmet_use_inline_bvec(req)) {
bio = &req->b.inline_bio;
bio_init(bio, req->inline_bvec, ARRAY_SIZE(req->inline_bvec));
} else {
ns->file = filp_open(ns->device_path, flags, 0);
if (IS_ERR(ns->file)) {
- pr_err("failed to open file %s: (%ld)\n",
- ns->device_path, PTR_ERR(ns->file));
- return PTR_ERR(ns->file);
+ ret = PTR_ERR(ns->file);
+ pr_err("failed to open file %s: (%d)\n",
+ ns->device_path, ret);
+ ns->file = NULL;
+ return ret;
}
ret = nvmet_file_ns_revalidate(ns);
static void nvme_loop_destroy_admin_queue(struct nvme_loop_ctrl *ctrl)
{
- clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
+ if (!test_and_clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags))
+ return;
nvmet_sq_destroy(&ctrl->queues[0].nvme_sq);
blk_cleanup_queue(ctrl->ctrl.admin_q);
blk_cleanup_queue(ctrl->ctrl.fabrics_q);
clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[i].flags);
nvmet_sq_destroy(&ctrl->queues[i].nvme_sq);
}
+ ctrl->ctrl.queue_count = 1;
}
static int nvme_loop_init_io_queues(struct nvme_loop_ctrl *ctrl)
return 0;
out_cleanup_queue:
+ clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
blk_cleanup_queue(ctrl->ctrl.admin_q);
out_cleanup_fabrics_q:
blk_cleanup_queue(ctrl->ctrl.fabrics_q);
nvme_loop_shutdown_ctrl(ctrl);
if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) {
- /* state change failure should never happen */
- WARN_ON_ONCE(1);
+ if (ctrl->ctrl.state != NVME_CTRL_DELETING &&
+ ctrl->ctrl.state != NVME_CTRL_DELETING_NOIO)
+ /* state change failure for non-deleted ctrl? */
+ WARN_ON_ONCE(1);
return;
}
ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_loop_ctrl_ops,
0 /* no quirks, we're perfect! */);
- if (ret)
+ if (ret) {
+ kfree(ctrl);
goto out;
+ }
if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING))
WARN_ON_ONCE(1);
struct nvmet_subsys *subsys;
struct nvmet_sq **sqs;
- bool cmd_seen;
+ bool reset_tbkas;
struct mutex lock;
u64 cap;
return le64_to_cpu(lba) << (ns->blksize_shift - SECTOR_SHIFT);
}
+static inline bool nvmet_use_inline_bvec(struct nvmet_req *req)
+{
+ return req->transfer_len <= NVMET_MAX_INLINE_DATA_LEN &&
+ req->sg_cnt <= NVMET_MAX_INLINE_BIOVEC;
+}
+
#endif /* _NVMET_H */
if (req->sg_cnt > BIO_MAX_VECS)
return -EINVAL;
- if (req->transfer_len <= NVMET_MAX_INLINE_DATA_LEN) {
+ if (nvmet_use_inline_bvec(req)) {
bio = &req->p.inline_bio;
bio_init(bio, req->inline_bvec, ARRAY_SIZE(req->inline_bvec));
} else {
{
struct nvmet_rdma_rsp *rsp =
container_of(wc->wr_cqe, struct nvmet_rdma_rsp, send_cqe);
- struct nvmet_rdma_queue *queue = cq->cq_context;
+ struct nvmet_rdma_queue *queue = wc->qp->qp_context;
nvmet_rdma_release_rsp(rsp);
{
struct nvmet_rdma_rsp *rsp =
container_of(wc->wr_cqe, struct nvmet_rdma_rsp, write_cqe);
- struct nvmet_rdma_queue *queue = cq->cq_context;
+ struct nvmet_rdma_queue *queue = wc->qp->qp_context;
struct rdma_cm_id *cm_id = rsp->queue->cm_id;
u16 status;
* nvmet_req_init is completed.
*/
if (queue->rcv_state == NVMET_TCP_RECV_PDU &&
- len && len < cmd->req.port->inline_data_size &&
+ len && len <= cmd->req.port->inline_data_size &&
nvme_is_write(cmd->req.cmd))
return;
}
obj-$(CONFIG_PCIE_KIRIN) += pcie-kirin.o
obj-$(CONFIG_PCIE_HISI_STB) += pcie-histb.o
obj-$(CONFIG_PCI_MESON) += pci-meson.o
+obj-$(CONFIG_PCIE_TEGRA194) += pcie-tegra194.o
obj-$(CONFIG_PCIE_UNIPHIER) += pcie-uniphier.o
obj-$(CONFIG_PCIE_UNIPHIER_EP) += pcie-uniphier-ep.o
ifdef CONFIG_PCI_QUIRKS
obj-$(CONFIG_ARM64) += pcie-al.o
obj-$(CONFIG_ARM64) += pcie-hisi.o
-obj-$(CONFIG_ARM64) += pcie-tegra194.o
+obj-$(CONFIG_ARM64) += pcie-tegra194-acpi.o
endif
endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * ACPI quirks for Tegra194 PCIe host controller
+ *
+ * Copyright (C) 2021 NVIDIA Corporation.
+ *
+ * Author: Vidya Sagar <vidyas@nvidia.com>
+ */
+
+#include <linux/pci.h>
+#include <linux/pci-acpi.h>
+#include <linux/pci-ecam.h>
+
+#include "pcie-designware.h"
+
+struct tegra194_pcie_ecam {
+ void __iomem *config_base;
+ void __iomem *iatu_base;
+ void __iomem *dbi_base;
+};
+
+static int tegra194_acpi_init(struct pci_config_window *cfg)
+{
+ struct device *dev = cfg->parent;
+ struct tegra194_pcie_ecam *pcie_ecam;
+
+ pcie_ecam = devm_kzalloc(dev, sizeof(*pcie_ecam), GFP_KERNEL);
+ if (!pcie_ecam)
+ return -ENOMEM;
+
+ pcie_ecam->config_base = cfg->win;
+ pcie_ecam->iatu_base = cfg->win + SZ_256K;
+ pcie_ecam->dbi_base = cfg->win + SZ_512K;
+ cfg->priv = pcie_ecam;
+
+ return 0;
+}
+
+static void atu_reg_write(struct tegra194_pcie_ecam *pcie_ecam, int index,
+ u32 val, u32 reg)
+{
+ u32 offset = PCIE_GET_ATU_OUTB_UNR_REG_OFFSET(index);
+
+ writel(val, pcie_ecam->iatu_base + offset + reg);
+}
+
+static void program_outbound_atu(struct tegra194_pcie_ecam *pcie_ecam,
+ int index, int type, u64 cpu_addr,
+ u64 pci_addr, u64 size)
+{
+ atu_reg_write(pcie_ecam, index, lower_32_bits(cpu_addr),
+ PCIE_ATU_LOWER_BASE);
+ atu_reg_write(pcie_ecam, index, upper_32_bits(cpu_addr),
+ PCIE_ATU_UPPER_BASE);
+ atu_reg_write(pcie_ecam, index, lower_32_bits(pci_addr),
+ PCIE_ATU_LOWER_TARGET);
+ atu_reg_write(pcie_ecam, index, lower_32_bits(cpu_addr + size - 1),
+ PCIE_ATU_LIMIT);
+ atu_reg_write(pcie_ecam, index, upper_32_bits(pci_addr),
+ PCIE_ATU_UPPER_TARGET);
+ atu_reg_write(pcie_ecam, index, type, PCIE_ATU_CR1);
+ atu_reg_write(pcie_ecam, index, PCIE_ATU_ENABLE, PCIE_ATU_CR2);
+}
+
+static void __iomem *tegra194_map_bus(struct pci_bus *bus,
+ unsigned int devfn, int where)
+{
+ struct pci_config_window *cfg = bus->sysdata;
+ struct tegra194_pcie_ecam *pcie_ecam = cfg->priv;
+ u32 busdev;
+ int type;
+
+ if (bus->number < cfg->busr.start || bus->number > cfg->busr.end)
+ return NULL;
+
+ if (bus->number == cfg->busr.start) {
+ if (PCI_SLOT(devfn) == 0)
+ return pcie_ecam->dbi_base + where;
+ else
+ return NULL;
+ }
+
+ busdev = PCIE_ATU_BUS(bus->number) | PCIE_ATU_DEV(PCI_SLOT(devfn)) |
+ PCIE_ATU_FUNC(PCI_FUNC(devfn));
+
+ if (bus->parent->number == cfg->busr.start) {
+ if (PCI_SLOT(devfn) == 0)
+ type = PCIE_ATU_TYPE_CFG0;
+ else
+ return NULL;
+ } else {
+ type = PCIE_ATU_TYPE_CFG1;
+ }
+
+ program_outbound_atu(pcie_ecam, 0, type, cfg->res.start, busdev,
+ SZ_256K);
+
+ return pcie_ecam->config_base + where;
+}
+
+const struct pci_ecam_ops tegra194_pcie_ops = {
+ .init = tegra194_acpi_init,
+ .pci_ops = {
+ .map_bus = tegra194_map_bus,
+ .read = pci_generic_config_read,
+ .write = pci_generic_config_write,
+ }
+};
#include <linux/of_irq.h>
#include <linux/of_pci.h>
#include <linux/pci.h>
-#include <linux/pci-acpi.h>
-#include <linux/pci-ecam.h>
#include <linux/phy/phy.h>
#include <linux/pinctrl/consumer.h>
#include <linux/platform_device.h>
GEN4_CORE_CLK_FREQ
};
-static const u32 event_cntr_ctrl_offset[] = {
- 0x1d8,
- 0x1a8,
- 0x1a8,
- 0x1a8,
- 0x1c4,
- 0x1d8
-};
-
-static const u32 event_cntr_data_offset[] = {
- 0x1dc,
- 0x1ac,
- 0x1ac,
- 0x1ac,
- 0x1c8,
- 0x1dc
-};
-
struct tegra_pcie_dw {
struct device *dev;
struct resource *appl_res;
enum dw_pcie_device_mode mode;
};
-#if defined(CONFIG_ACPI) && defined(CONFIG_PCI_QUIRKS)
-struct tegra194_pcie_ecam {
- void __iomem *config_base;
- void __iomem *iatu_base;
- void __iomem *dbi_base;
-};
-
-static int tegra194_acpi_init(struct pci_config_window *cfg)
-{
- struct device *dev = cfg->parent;
- struct tegra194_pcie_ecam *pcie_ecam;
-
- pcie_ecam = devm_kzalloc(dev, sizeof(*pcie_ecam), GFP_KERNEL);
- if (!pcie_ecam)
- return -ENOMEM;
-
- pcie_ecam->config_base = cfg->win;
- pcie_ecam->iatu_base = cfg->win + SZ_256K;
- pcie_ecam->dbi_base = cfg->win + SZ_512K;
- cfg->priv = pcie_ecam;
-
- return 0;
-}
-
-static void atu_reg_write(struct tegra194_pcie_ecam *pcie_ecam, int index,
- u32 val, u32 reg)
-{
- u32 offset = PCIE_GET_ATU_OUTB_UNR_REG_OFFSET(index);
-
- writel(val, pcie_ecam->iatu_base + offset + reg);
-}
-
-static void program_outbound_atu(struct tegra194_pcie_ecam *pcie_ecam,
- int index, int type, u64 cpu_addr,
- u64 pci_addr, u64 size)
-{
- atu_reg_write(pcie_ecam, index, lower_32_bits(cpu_addr),
- PCIE_ATU_LOWER_BASE);
- atu_reg_write(pcie_ecam, index, upper_32_bits(cpu_addr),
- PCIE_ATU_UPPER_BASE);
- atu_reg_write(pcie_ecam, index, lower_32_bits(pci_addr),
- PCIE_ATU_LOWER_TARGET);
- atu_reg_write(pcie_ecam, index, lower_32_bits(cpu_addr + size - 1),
- PCIE_ATU_LIMIT);
- atu_reg_write(pcie_ecam, index, upper_32_bits(pci_addr),
- PCIE_ATU_UPPER_TARGET);
- atu_reg_write(pcie_ecam, index, type, PCIE_ATU_CR1);
- atu_reg_write(pcie_ecam, index, PCIE_ATU_ENABLE, PCIE_ATU_CR2);
-}
-
-static void __iomem *tegra194_map_bus(struct pci_bus *bus,
- unsigned int devfn, int where)
-{
- struct pci_config_window *cfg = bus->sysdata;
- struct tegra194_pcie_ecam *pcie_ecam = cfg->priv;
- u32 busdev;
- int type;
-
- if (bus->number < cfg->busr.start || bus->number > cfg->busr.end)
- return NULL;
-
- if (bus->number == cfg->busr.start) {
- if (PCI_SLOT(devfn) == 0)
- return pcie_ecam->dbi_base + where;
- else
- return NULL;
- }
-
- busdev = PCIE_ATU_BUS(bus->number) | PCIE_ATU_DEV(PCI_SLOT(devfn)) |
- PCIE_ATU_FUNC(PCI_FUNC(devfn));
-
- if (bus->parent->number == cfg->busr.start) {
- if (PCI_SLOT(devfn) == 0)
- type = PCIE_ATU_TYPE_CFG0;
- else
- return NULL;
- } else {
- type = PCIE_ATU_TYPE_CFG1;
- }
-
- program_outbound_atu(pcie_ecam, 0, type, cfg->res.start, busdev,
- SZ_256K);
-
- return pcie_ecam->config_base + where;
-}
-
-const struct pci_ecam_ops tegra194_pcie_ops = {
- .init = tegra194_acpi_init,
- .pci_ops = {
- .map_bus = tegra194_map_bus,
- .read = pci_generic_config_read,
- .write = pci_generic_config_write,
- }
-};
-#endif /* defined(CONFIG_ACPI) && defined(CONFIG_PCI_QUIRKS) */
-
-#ifdef CONFIG_PCIE_TEGRA194
-
static inline struct tegra_pcie_dw *to_tegra_pcie(struct dw_pcie *pci)
{
return container_of(pci, struct tegra_pcie_dw, pci);
};
#if defined(CONFIG_PCIEASPM)
+static const u32 event_cntr_ctrl_offset[] = {
+ 0x1d8,
+ 0x1a8,
+ 0x1a8,
+ 0x1a8,
+ 0x1c4,
+ 0x1d8
+};
+
+static const u32 event_cntr_data_offset[] = {
+ 0x1dc,
+ 0x1ac,
+ 0x1ac,
+ 0x1ac,
+ 0x1c8,
+ 0x1dc
+};
+
static void disable_aspm_l11(struct tegra_pcie_dw *pcie)
{
u32 val;
MODULE_AUTHOR("Vidya Sagar <vidyas@nvidia.com>");
MODULE_DESCRIPTION("NVIDIA PCIe host controller driver");
MODULE_LICENSE("GPL v2");
-
-#endif /* CONFIG_PCIE_TEGRA194 */
udelay(PIO_RETRY_DELAY);
}
- dev_err(dev, "config read/write timed out\n");
+ dev_err(dev, "PIO read/write transfer time out\n");
return -ETIMEDOUT;
}
return true;
}
+static bool advk_pcie_pio_is_running(struct advk_pcie *pcie)
+{
+ struct device *dev = &pcie->pdev->dev;
+
+ /*
+ * Trying to start a new PIO transfer when previous has not completed
+ * cause External Abort on CPU which results in kernel panic:
+ *
+ * SError Interrupt on CPU0, code 0xbf000002 -- SError
+ * Kernel panic - not syncing: Asynchronous SError Interrupt
+ *
+ * Functions advk_pcie_rd_conf() and advk_pcie_wr_conf() are protected
+ * by raw_spin_lock_irqsave() at pci_lock_config() level to prevent
+ * concurrent calls at the same time. But because PIO transfer may take
+ * about 1.5s when link is down or card is disconnected, it means that
+ * advk_pcie_wait_pio() does not always have to wait for completion.
+ *
+ * Some versions of ARM Trusted Firmware handles this External Abort at
+ * EL3 level and mask it to prevent kernel panic. Relevant TF-A commit:
+ * https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git/commit/?id=3c7dcdac5c50
+ */
+ if (advk_readl(pcie, PIO_START)) {
+ dev_err(dev, "Previous PIO read/write transfer is still running\n");
+ return true;
+ }
+
+ return false;
+}
+
static int advk_pcie_rd_conf(struct pci_bus *bus, u32 devfn,
int where, int size, u32 *val)
{
return pci_bridge_emul_conf_read(&pcie->bridge, where,
size, val);
- /* Start PIO */
- advk_writel(pcie, 0, PIO_START);
- advk_writel(pcie, 1, PIO_ISR);
+ if (advk_pcie_pio_is_running(pcie)) {
+ *val = 0xffffffff;
+ return PCIBIOS_SET_FAILED;
+ }
/* Program the control register */
reg = advk_readl(pcie, PIO_CTRL);
/* Program the data strobe */
advk_writel(pcie, 0xf, PIO_WR_DATA_STRB);
- /* Start the transfer */
+ /* Clear PIO DONE ISR and start the transfer */
+ advk_writel(pcie, 1, PIO_ISR);
advk_writel(pcie, 1, PIO_START);
ret = advk_pcie_wait_pio(pcie);
if (where % size)
return PCIBIOS_SET_FAILED;
- /* Start PIO */
- advk_writel(pcie, 0, PIO_START);
- advk_writel(pcie, 1, PIO_ISR);
+ if (advk_pcie_pio_is_running(pcie))
+ return PCIBIOS_SET_FAILED;
/* Program the control register */
reg = advk_readl(pcie, PIO_CTRL);
/* Program the data strobe */
advk_writel(pcie, data_strobe, PIO_WR_DATA_STRB);
- /* Start the transfer */
+ /* Clear PIO DONE ISR and start the transfer */
+ advk_writel(pcie, 1, PIO_ISR);
advk_writel(pcie, 1, PIO_START);
ret = advk_pcie_wait_pio(pcie);
#endif
}
+bool pci_host_of_has_msi_map(struct device *dev)
+{
+ if (dev && dev->of_node)
+ return of_get_property(dev->of_node, "msi-map", NULL);
+ return false;
+}
+
static inline int __of_pci_pci_compare(struct device_node *node,
unsigned int data)
{
dev_warn(dev, "More than one I/O resource converted for %pOF. CPU base address for old range lost!\n",
dev_node);
*io_base = range.cpu_addr;
+ } else if (resource_type(res) == IORESOURCE_MEM) {
+ res->flags &= ~IORESOURCE_MEM_64;
}
pci_add_resource_offset(resources, res, res->start - range.pci_addr);
device_enable_async_suspend(bus->bridge);
pci_set_bus_of_node(bus);
pci_set_bus_msi_domain(bus);
- if (bridge->msi_domain && !dev_get_msi_domain(&bus->dev))
+ if (bridge->msi_domain && !dev_get_msi_domain(&bus->dev) &&
+ !pci_host_of_has_msi_map(parent))
bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI;
if (!parent)
}
/*
+ * Some NVIDIA GPU devices do not work with bus reset, SBR needs to be
+ * prevented for those affected devices.
+ */
+static void quirk_nvidia_no_bus_reset(struct pci_dev *dev)
+{
+ if ((dev->device & 0xffc0) == 0x2340)
+ quirk_no_bus_reset(dev);
+}
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
+ quirk_nvidia_no_bus_reset);
+
+/*
* Some Atheros AR9xxx and QCA988x chips do not behave after a bus reset.
* The device will throw a Link Down error on AER-capable systems and
* regardless of AER, config space of the device is never accessible again
*/
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_CAVIUM, 0xa100, quirk_no_bus_reset);
+/*
+ * Some TI KeyStone C667X devices do not support bus/hot reset. The PCIESS
+ * automatically disables LTSSM when Secondary Bus Reset is received and
+ * the device stops working. Prevent bus reset for these devices. With
+ * this change, the device can be assigned to VMs with VFIO, but it will
+ * leak state between VMs. Reference
+ * https://e2e.ti.com/support/processors/f/791/t/954382
+ */
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_TI, 0xb005, quirk_no_bus_reset);
+
static void quirk_no_pm_reset(struct pci_dev *dev)
{
/*
return 0;
}
+#define PCI_DEVICE_ID_HINIC_VF 0x375E
+#define HINIC_VF_FLR_TYPE 0x1000
+#define HINIC_VF_FLR_CAP_BIT (1UL << 30)
+#define HINIC_VF_OP 0xE80
+#define HINIC_VF_FLR_PROC_BIT (1UL << 18)
+#define HINIC_OPERATION_TIMEOUT 15000 /* 15 seconds */
+
+/* Device-specific reset method for Huawei Intelligent NIC virtual functions */
+static int reset_hinic_vf_dev(struct pci_dev *pdev, int probe)
+{
+ unsigned long timeout;
+ void __iomem *bar;
+ u32 val;
+
+ if (probe)
+ return 0;
+
+ bar = pci_iomap(pdev, 0, 0);
+ if (!bar)
+ return -ENOTTY;
+
+ /* Get and check firmware capabilities */
+ val = ioread32be(bar + HINIC_VF_FLR_TYPE);
+ if (!(val & HINIC_VF_FLR_CAP_BIT)) {
+ pci_iounmap(pdev, bar);
+ return -ENOTTY;
+ }
+
+ /* Set HINIC_VF_FLR_PROC_BIT for the start of FLR */
+ val = ioread32be(bar + HINIC_VF_OP);
+ val = val | HINIC_VF_FLR_PROC_BIT;
+ iowrite32be(val, bar + HINIC_VF_OP);
+
+ pcie_flr(pdev);
+
+ /*
+ * The device must recapture its Bus and Device Numbers after FLR
+ * in order generate Completions. Issue a config write to let the
+ * device capture this information.
+ */
+ pci_write_config_word(pdev, PCI_VENDOR_ID, 0);
+
+ /* Firmware clears HINIC_VF_FLR_PROC_BIT when reset is complete */
+ timeout = jiffies + msecs_to_jiffies(HINIC_OPERATION_TIMEOUT);
+ do {
+ val = ioread32be(bar + HINIC_VF_OP);
+ if (!(val & HINIC_VF_FLR_PROC_BIT))
+ goto reset_complete;
+ msleep(20);
+ } while (time_before(jiffies, timeout));
+
+ val = ioread32be(bar + HINIC_VF_OP);
+ if (!(val & HINIC_VF_FLR_PROC_BIT))
+ goto reset_complete;
+
+ pci_warn(pdev, "Reset dev timeout, FLR ack reg: %#010x\n", val);
+
+reset_complete:
+ pci_iounmap(pdev, bar);
+
+ return 0;
+}
+
static const struct pci_dev_reset_methods pci_dev_reset_methods[] = {
{ PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_82599_SFP_VF,
reset_intel_82599_sfp_virtfn },
{ PCI_VENDOR_ID_INTEL, 0x0a54, delay_250ms_after_flr },
{ PCI_VENDOR_ID_CHELSIO, PCI_ANY_ID,
reset_chelsio_generic_dev },
+ { PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HINIC_VF,
+ reset_hinic_vf_dev },
{ 0 }
};
{ PCI_VENDOR_ID_AMPERE, 0xE00A, pci_quirk_xgene_acs },
{ PCI_VENDOR_ID_AMPERE, 0xE00B, pci_quirk_xgene_acs },
{ PCI_VENDOR_ID_AMPERE, 0xE00C, pci_quirk_xgene_acs },
+ /* Broadcom multi-function device */
+ { PCI_VENDOR_ID_BROADCOM, 0x16D7, pci_quirk_mf_endpoint_acs },
{ PCI_VENDOR_ID_BROADCOM, 0xD714, pci_quirk_brcm_acs },
/* Amazon Annapurna Labs */
{ PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031, pci_quirk_al_acs },
static void quirk_amd_harvest_no_ats(struct pci_dev *pdev)
{
if ((pdev->device == 0x7312 && pdev->revision != 0x00) ||
- (pdev->device == 0x7340 && pdev->revision != 0xc5))
+ (pdev->device == 0x7340 && pdev->revision != 0xc5) ||
+ (pdev->device == 0x7341 && pdev->revision != 0x00))
return;
if (pdev->device == 0x15d8) {
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x7312, quirk_amd_harvest_no_ats);
/* AMD Navi14 dGPU */
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x7340, quirk_amd_harvest_no_ats);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x7341, quirk_amd_harvest_no_ats);
/* AMD Raven platform iGPU */
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x15d8, quirk_amd_harvest_no_ats);
#endif /* CONFIG_PCI_ATS */
* Other architectures (e.g., ARM) either do not support big endian, or
* else leave I/O in little endian mode.
*/
- if (IS_ENABLED(CONFIG_MIPS) && IS_ENABLED(__BIG_ENDIAN))
+ if (IS_ENABLED(CONFIG_MIPS) && IS_ENABLED(CONFIG_CPU_BIG_ENDIAN))
return __raw_readl(addr);
else
return readl_relaxed(addr);
static inline void brcm_usb_writel(u32 val, void __iomem *addr)
{
/* See brcmnand_readl() comments */
- if (IS_ENABLED(CONFIG_MIPS) && IS_ENABLED(__BIG_ENDIAN))
+ if (IS_ENABLED(CONFIG_MIPS) && IS_ENABLED(CONFIG_CPU_BIG_ENDIAN))
__raw_writel(val, addr);
else
writel_relaxed(val, addr);
sp->nsubnodes = node;
if (sp->num_lanes > SIERRA_MAX_LANES) {
+ ret = -EINVAL;
dev_err(dev, "Invalid lane configuration\n");
goto put_child2;
}
break;
default:
dev_err(tphy->dev, "incompatible PHY type\n");
+ clk_disable_unprepare(instance->ref_clk);
+ clk_disable_unprepare(instance->da_ref_clk);
return -EINVAL;
}
priv->coreclock = clock;
iores = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!iores) {
+ dev_err(priv->dev, "Invalid resource\n");
+ return -EINVAL;
+ }
iomem = devm_ioremap(priv->dev, iores->start, resource_size(iores));
if (IS_ERR(iomem)) {
dev_err(priv->dev, "Unable to get serdes registers: %s\n",
.probe = mt7621_pci_phy_probe,
.driver = {
.name = "mt7621-pci-phy",
- .of_match_table = of_match_ptr(mt7621_pci_phy_ids),
+ .of_match_table = mt7621_pci_phy_ids,
},
};
if (wiz->typec_dir_delay < WIZ_TYPEC_DIR_DEBOUNCE_MIN ||
wiz->typec_dir_delay > WIZ_TYPEC_DIR_DEBOUNCE_MAX) {
+ ret = -EINVAL;
dev_err(dev, "Invalid typec-dir-debounce property\n");
goto err_addr_to_resource;
}
}
/**
- * Configure a pin's signal by applying an expression's descriptor state for
- * all descriptors in the expression.
+ * aspeed_g5_sig_expr_set() - Configure a pin's signal by applying an
+ * expression's descriptor state for all descriptors in the expression.
*
* @ctx: The pinmux context
* @expr: The expression associated with the function whose signal is to be
};
/**
- * Configure a pin's signal by applying an expression's descriptor state for
- * all descriptors in the expression.
+ * aspeed_g6_sig_expr_set() - Configure a pin's signal by applying an
+ * expression's descriptor state for all descriptors in the expression.
*
* @ctx: The pinmux context
* @expr: The expression associated with the function whose signal is to be
}
/**
- * Disable a signal on a pin by disabling all provided signal expressions.
+ * aspeed_disable_sig() - Disable a signal on a pin by disabling all provided
+ * signal expressions.
*
* @ctx: The pinmux context
* @exprs: The list of signal expressions (from a priority level on a pin)
}
/**
- * Query the enabled or disabled state of a signal descriptor
+ * aspeed_sig_desc_eval() - Query the enabled or disabled state of a signal
+ * descriptor.
*
* @desc: The signal descriptor of interest
* @enabled: True to query the enabled state, false to query disabled state
config PINCTRL_SC8180X
tristate "Qualcomm Technologies Inc SC8180x pin controller driver"
depends on GPIOLIB && (OF || ACPI)
- select PINCTRL_MSM
+ depends on PINCTRL_MSM
help
This is the pinctrl, pinmux, pinconf and gpiolib driver for the
Qualcomm Technologies Inc TLMM block found on the Qualcomm
"gpio29", "gpio30", "gpio31", "gpio32", "gpio33", "gpio34", "gpio35",
"gpio36", "gpio37", "gpio38", "gpio39", "gpio40", "gpio41", "gpio42",
"gpio43", "gpio44", "gpio45", "gpio46", "gpio47", "gpio48", "gpio49",
- "gpio50", "gpio51", "gpio52", "gpio52", "gpio53", "gpio53", "gpio54",
- "gpio55", "gpio56", "gpio57", "gpio58", "gpio59", "gpio60", "gpio61",
- "gpio62", "gpio63", "gpio64", "gpio65", "gpio66", "gpio67", "gpio68",
- "gpio69", "gpio70", "gpio71", "gpio72", "gpio73", "gpio74", "gpio75",
- "gpio76", "gpio77", "gpio78", "gpio79", "gpio80", "gpio81", "gpio82",
- "gpio83", "gpio84", "gpio85", "gpio86", "gpio87", "gpio88", "gpio89",
- "gpio90", "gpio91", "gpio92", "gpio93", "gpio94", "gpio95", "gpio96",
- "gpio97", "gpio98", "gpio99", "gpio100", "gpio101", "gpio102",
- "gpio103", "gpio104", "gpio105", "gpio106", "gpio107",
+ "gpio50", "gpio51", "gpio52", "gpio53", "gpio54", "gpio55", "gpio56",
+ "gpio57", "gpio58", "gpio59", "gpio60", "gpio61", "gpio62", "gpio63",
+ "gpio64", "gpio65", "gpio66", "gpio67", "gpio68", "gpio69", "gpio70",
+ "gpio71", "gpio72", "gpio73", "gpio74", "gpio75", "gpio76", "gpio77",
+ "gpio78", "gpio79", "gpio80", "gpio81", "gpio82", "gpio83", "gpio84",
+ "gpio85", "gpio86", "gpio87", "gpio88", "gpio89", "gpio90", "gpio91",
+ "gpio92", "gpio93", "gpio94", "gpio95", "gpio96", "gpio97", "gpio98",
+ "gpio99", "gpio100", "gpio101", "gpio102", "gpio103", "gpio104",
+ "gpio105", "gpio106", "gpio107",
};
static const char * const qdss_stm_groups[] = {
if (p->groups[group].enabled) {
dev_err(p->dev, "%s is already enabled\n",
p->groups[group].name);
- return -EBUSY;
+ return 0;
}
p->groups[group].enabled = 1;
if (vring->next_avail == virtio16_to_cpu(vdev, vr->avail->idx))
return NULL;
+ /* Make sure 'avail->idx' is visible already. */
+ virtio_rmb(false);
+
idx = vring->next_avail % vr->num;
head = virtio16_to_cpu(vdev, vr->avail->ring[idx]);
if (WARN_ON(head >= vr->num))
* done or not. Add a memory barrier here to make sure the update above
* completes before updating the idx.
*/
- mb();
+ virtio_mb(false);
vr->used->idx = cpu_to_virtio16(vdev, vr_idx + 1);
}
desc = NULL;
fifo->vring[is_rx] = NULL;
+ /*
+ * Make sure the load/store are in order before
+ * returning back to virtio.
+ */
+ virtio_mb(false);
+
/* Notify upper layer that packet is done. */
spin_lock_irqsave(&fifo->spin_lock[is_rx], flags);
vring_interrupt(0, vring->vq);
err = devm_request_irq(&pdev->dev, priv->irq,
mlxreg_hotplug_irq_handler, IRQF_TRIGGER_FALLING
- | IRQF_SHARED | IRQF_NO_AUTOEN,
- "mlxreg-hotplug", priv);
+ | IRQF_SHARED, "mlxreg-hotplug", priv);
if (err) {
dev_err(&pdev->dev, "Failed to request irq: %d\n", err);
return err;
}
+ disable_irq(priv->irq);
spin_lock_init(&priv->lock);
INIT_DELAYED_WORK(&priv->dwork_irq, mlxreg_hotplug_work_handler);
dev_set_drvdata(&pdev->dev, priv);
{
int status;
- status = __ssam_ssh_event_request(ctrl, reg, reg.cid_enable, id, flags);
+ status = __ssam_ssh_event_request(ctrl, reg, reg.cid_disable, id, flags);
if (status < 0 && status != -EINVAL) {
ssam_err(ctrl,
* interrupt, and let the SAM resume callback during the controller
* resume process clear it.
*/
- const int irqf = IRQF_SHARED | IRQF_ONESHOT |
- IRQF_TRIGGER_RISING | IRQF_NO_AUTOEN;
+ const int irqf = IRQF_ONESHOT | IRQF_TRIGGER_RISING | IRQF_NO_AUTOEN;
gpiod = gpiod_get(dev, "ssam_wakeup-int", GPIOD_ASIS);
if (IS_ERR(gpiod))
NULL,
};
-/* Devices for Surface Laptop 3. */
+/* Devices for Surface Laptop 3 and 4. */
static const struct software_node *ssam_node_group_sl3[] = {
&ssam_node_root,
&ssam_node_bat_ac,
/* Surface Laptop 3 (13", Intel) */
{ "MSHW0114", (unsigned long)ssam_node_group_sl3 },
- /* Surface Laptop 3 (15", AMD) */
+ /* Surface Laptop 3 (15", AMD) and 4 (15", AMD) */
{ "MSHW0110", (unsigned long)ssam_node_group_sl3 },
+ /* Surface Laptop 4 (13", Intel) */
+ { "MSHW0250", (unsigned long)ssam_node_group_sl3 },
+
/* Surface Laptop Go 1 */
{ "MSHW0118", (unsigned long)ssam_node_group_slg1 },
*/
if (test_bit(SDTX_DEVICE_SHUTDOWN_BIT, &ddev->flags)) {
up_write(&ddev->client_lock);
+ mutex_destroy(&client->read_lock);
sdtx_device_put(client->ddev);
kfree(client);
return -ENODEV;
struct sdtx_client *client = file->private_data;
__poll_t events = 0;
- if (down_read_killable(&client->ddev->lock))
- return -ERESTARTSYS;
-
- if (test_bit(SDTX_DEVICE_SHUTDOWN_BIT, &client->ddev->flags)) {
- up_read(&client->ddev->lock);
+ if (test_bit(SDTX_DEVICE_SHUTDOWN_BIT, &client->ddev->flags))
return EPOLLHUP | EPOLLERR;
- }
poll_wait(file, &client->ddev->waitq, pt);
if (!kfifo_is_empty(&client->buffer))
events |= EPOLLIN | EPOLLRDNORM;
- up_read(&client->ddev->lock);
return events;
}
config INTEL_INT0002_VGPIO
tristate "Intel ACPI INT0002 Virtual GPIO driver"
- depends on GPIOLIB && ACPI
+ depends on GPIOLIB && ACPI && PM_SLEEP
select GPIOLIB_IRQCHIP
help
Some peripherals on Bay Trail and Cherry Trail platforms signal a
void exit_dell_smbios_wmi(void)
{
- wmi_driver_unregister(&dell_smbios_wmi_driver);
+ if (wmi_supported)
+ wmi_driver_unregister(&dell_smbios_wmi_driver);
}
MODULE_DEVICE_TABLE(wmi, dell_smbios_wmi_id_table);
return r;
}
+#define DMI_EXACT_MATCH_GIGABYTE_BOARD_NAME(name) \
+ { .matches = { \
+ DMI_EXACT_MATCH(DMI_BOARD_VENDOR, "Gigabyte Technology Co., Ltd."), \
+ DMI_EXACT_MATCH(DMI_BOARD_NAME, name), \
+ }}
+
static const struct dmi_system_id gigabyte_wmi_known_working_platforms[] = {
- { .matches = {
- DMI_EXACT_MATCH(DMI_BOARD_VENDOR, "Gigabyte Technology Co., Ltd."),
- DMI_EXACT_MATCH(DMI_BOARD_NAME, "B550 GAMING X V2"),
- }},
- { .matches = {
- DMI_EXACT_MATCH(DMI_BOARD_VENDOR, "Gigabyte Technology Co., Ltd."),
- DMI_EXACT_MATCH(DMI_BOARD_NAME, "B550M AORUS PRO-P"),
- }},
- { .matches = {
- DMI_EXACT_MATCH(DMI_BOARD_VENDOR, "Gigabyte Technology Co., Ltd."),
- DMI_EXACT_MATCH(DMI_BOARD_NAME, "B550M DS3H"),
- }},
- { .matches = {
- DMI_EXACT_MATCH(DMI_BOARD_VENDOR, "Gigabyte Technology Co., Ltd."),
- DMI_EXACT_MATCH(DMI_BOARD_NAME, "Z390 I AORUS PRO WIFI-CF"),
- }},
- { .matches = {
- DMI_EXACT_MATCH(DMI_BOARD_VENDOR, "Gigabyte Technology Co., Ltd."),
- DMI_EXACT_MATCH(DMI_BOARD_NAME, "X570 AORUS ELITE"),
- }},
- { .matches = {
- DMI_EXACT_MATCH(DMI_BOARD_VENDOR, "Gigabyte Technology Co., Ltd."),
- DMI_EXACT_MATCH(DMI_BOARD_NAME, "X570 I AORUS PRO WIFI"),
- }},
+ DMI_EXACT_MATCH_GIGABYTE_BOARD_NAME("B550 AORUS ELITE"),
+ DMI_EXACT_MATCH_GIGABYTE_BOARD_NAME("B550 GAMING X V2"),
+ DMI_EXACT_MATCH_GIGABYTE_BOARD_NAME("B550M AORUS PRO-P"),
+ DMI_EXACT_MATCH_GIGABYTE_BOARD_NAME("B550M DS3H"),
+ DMI_EXACT_MATCH_GIGABYTE_BOARD_NAME("Z390 I AORUS PRO WIFI-CF"),
+ DMI_EXACT_MATCH_GIGABYTE_BOARD_NAME("X570 AORUS ELITE"),
+ DMI_EXACT_MATCH_GIGABYTE_BOARD_NAME("X570 I AORUS PRO WIFI"),
+ DMI_EXACT_MATCH_GIGABYTE_BOARD_NAME("X570 UD"),
{ }
};
MODULE_AUTHOR("Alex Hung");
MODULE_ALIAS("acpi*:HPQ6001:*");
MODULE_ALIAS("acpi*:WSTADEF:*");
+MODULE_ALIAS("acpi*:AMDI0051:*");
static struct input_dev *hpwl_input_dev;
static const struct acpi_device_id hpwl_ids[] = {
{"HPQ6001", 0},
{"WSTADEF", 0},
+ {"AMDI0051", 0},
{"", 0},
};
static int lis3lv02d_acpi_init(struct lis3lv02d *lis3)
{
struct acpi_device *dev = lis3->bus_priv;
+ if (!lis3->init_required)
+ return 0;
+
if (acpi_evaluate_object(dev->handle, METHOD_NAME__INI,
NULL, NULL) != AE_OK)
return -EINVAL;
}
/* call the core layer do its init */
+ lis3_dev.init_required = true;
ret = lis3lv02d_init_device(&lis3_dev);
if (ret)
return ret;
static int lis3lv02d_resume(struct device *dev)
{
+ lis3_dev.init_required = false;
+ lis3lv02d_poweron(&lis3_dev);
+ return 0;
+}
+
+static int lis3lv02d_restore(struct device *dev)
+{
+ lis3_dev.init_required = true;
lis3lv02d_poweron(&lis3_dev);
return 0;
}
-static SIMPLE_DEV_PM_OPS(hp_accel_pm, lis3lv02d_suspend, lis3lv02d_resume);
+static const struct dev_pm_ops hp_accel_pm = {
+ .suspend = lis3lv02d_suspend,
+ .resume = lis3lv02d_resume,
+ .freeze = lis3lv02d_suspend,
+ .thaw = lis3lv02d_resume,
+ .poweroff = lis3lv02d_suspend,
+ .restore = lis3lv02d_restore,
+};
+
#define HP_ACCEL_PM (&hp_accel_pm)
#else
#define HP_ACCEL_PM NULL
};
enum {
- SMBC_CONSERVATION_ON = 3,
- SMBC_CONSERVATION_OFF = 5,
+ SBMC_CONSERVATION_ON = 3,
+ SBMC_CONSERVATION_OFF = 5,
};
enum {
return eval_int(handle, "GBMD", res);
}
-static int exec_smbc(acpi_handle handle, unsigned long arg)
+static int exec_sbmc(acpi_handle handle, unsigned long arg)
{
- return exec_simple_method(handle, "SMBC", arg);
+ return exec_simple_method(handle, "SBMC", arg);
}
static int eval_hals(acpi_handle handle, unsigned long *res)
if (err)
return err;
- err = exec_smbc(priv->adev->handle, state ? SMBC_CONSERVATION_ON : SMBC_CONSERVATION_OFF);
+ err = exec_sbmc(priv->adev->handle, state ? SBMC_CONSERVATION_ON : SBMC_CONSERVATION_OFF);
if (err)
return err;
{
struct ideapad_dytc_priv *dytc = container_of(pprof, struct ideapad_dytc_priv, pprof);
struct ideapad_private *priv = dytc->priv;
+ unsigned long output;
int err;
err = mutex_lock_interruptible(&dytc->mutex);
/* Determine if we are in CQL mode. This alters the commands we do */
err = dytc_cql_command(priv, DYTC_SET_COMMAND(DYTC_FUNCTION_MMC, perfmode, 1),
- NULL);
+ &output);
if (err)
goto unlock;
}
#define GPE0A_STS_PORT 0x420
#define GPE0A_EN_PORT 0x428
+struct int0002_data {
+ struct gpio_chip chip;
+ int parent_irq;
+ int wake_enable_count;
+};
+
/*
* As this is not a real GPIO at all, but just a hack to model an event in
* ACPI the get / set functions are dummy functions.
static int int0002_irq_set_wake(struct irq_data *data, unsigned int on)
{
struct gpio_chip *chip = irq_data_get_irq_chip_data(data);
- struct platform_device *pdev = to_platform_device(chip->parent);
- int irq = platform_get_irq(pdev, 0);
+ struct int0002_data *int0002 = container_of(chip, struct int0002_data, chip);
- /* Propagate to parent irq */
+ /*
+ * Applying of the wakeup flag to our parent IRQ is delayed till system
+ * suspend, because we only want to do this when using s2idle.
+ */
if (on)
- enable_irq_wake(irq);
+ int0002->wake_enable_count++;
else
- disable_irq_wake(irq);
+ int0002->wake_enable_count--;
return 0;
}
return (gpe_sts_reg & GPE0A_PME_B0_STS_BIT);
}
-static struct irq_chip int0002_byt_irqchip = {
+static struct irq_chip int0002_irqchip = {
.name = DRV_NAME,
.irq_ack = int0002_irq_ack,
.irq_mask = int0002_irq_mask,
.irq_set_wake = int0002_irq_set_wake,
};
-static struct irq_chip int0002_cht_irqchip = {
- .name = DRV_NAME,
- .irq_ack = int0002_irq_ack,
- .irq_mask = int0002_irq_mask,
- .irq_unmask = int0002_irq_unmask,
- /*
- * No set_wake, on CHT the IRQ is typically shared with the ACPI SCI
- * and we don't want to mess with the ACPI SCI irq settings.
- */
- .flags = IRQCHIP_SKIP_SET_WAKE,
-};
-
static const struct x86_cpu_id int0002_cpu_ids[] = {
- X86_MATCH_INTEL_FAM6_MODEL(ATOM_SILVERMONT, &int0002_byt_irqchip),
- X86_MATCH_INTEL_FAM6_MODEL(ATOM_AIRMONT, &int0002_cht_irqchip),
+ X86_MATCH_INTEL_FAM6_MODEL(ATOM_SILVERMONT, NULL),
+ X86_MATCH_INTEL_FAM6_MODEL(ATOM_AIRMONT, NULL),
{}
};
{
struct device *dev = &pdev->dev;
const struct x86_cpu_id *cpu_id;
- struct gpio_chip *chip;
+ struct int0002_data *int0002;
struct gpio_irq_chip *girq;
+ struct gpio_chip *chip;
int irq, ret;
/* Menlow has a different INT0002 device? <sigh> */
if (irq < 0)
return irq;
- chip = devm_kzalloc(dev, sizeof(*chip), GFP_KERNEL);
- if (!chip)
+ int0002 = devm_kzalloc(dev, sizeof(*int0002), GFP_KERNEL);
+ if (!int0002)
return -ENOMEM;
+ int0002->parent_irq = irq;
+
+ chip = &int0002->chip;
chip->label = DRV_NAME;
chip->parent = dev;
chip->owner = THIS_MODULE;
}
girq = &chip->irq;
- girq->chip = (struct irq_chip *)cpu_id->driver_data;
+ girq->chip = &int0002_irqchip;
/* This let us handle the parent IRQ in the driver */
girq->parent_handler = NULL;
girq->num_parents = 0;
acpi_register_wakeup_handler(irq, int0002_check_wake, NULL);
device_init_wakeup(dev, true);
+ dev_set_drvdata(dev, int0002);
return 0;
}
return 0;
}
+static int int0002_suspend(struct device *dev)
+{
+ struct int0002_data *int0002 = dev_get_drvdata(dev);
+
+ /*
+ * The INT0002 parent IRQ is often shared with the ACPI GPE IRQ, don't
+ * muck with it when firmware based suspend is used, otherwise we may
+ * cause spurious wakeups from firmware managed suspend.
+ */
+ if (!pm_suspend_via_firmware() && int0002->wake_enable_count)
+ enable_irq_wake(int0002->parent_irq);
+
+ return 0;
+}
+
+static int int0002_resume(struct device *dev)
+{
+ struct int0002_data *int0002 = dev_get_drvdata(dev);
+
+ if (!pm_suspend_via_firmware() && int0002->wake_enable_count)
+ disable_irq_wake(int0002->parent_irq);
+
+ return 0;
+}
+
+static const struct dev_pm_ops int0002_pm_ops = {
+ .suspend = int0002_suspend,
+ .resume = int0002_resume,
+};
+
static const struct acpi_device_id int0002_acpi_ids[] = {
{ "INT0002", 0 },
{ },
.driver = {
.name = DRV_NAME,
.acpi_match_table = int0002_acpi_ids,
+ .pm = &int0002_pm_ops,
},
.probe = int0002_probe,
.remove = int0002_remove,
{ "INT34D4", 0 },
{ }
};
+MODULE_DEVICE_TABLE(acpi, punit_ipc_acpi_ids);
static struct platform_driver intel_punit_ipc_driver = {
.probe = intel_punit_ipc_probe,
TPACPI_Q_LNV3('N', '2', 'O', TPACPI_FAN_2CTL), /* P1 / X1 Extreme (2nd gen) */
TPACPI_Q_LNV3('N', '2', 'V', TPACPI_FAN_2CTL), /* P1 / X1 Extreme (3nd gen) */
TPACPI_Q_LNV3('N', '3', '0', TPACPI_FAN_2CTL), /* P15 (1st gen) / P15v (1st gen) */
+ TPACPI_Q_LNV3('N', '3', '2', TPACPI_FAN_2CTL), /* X1 Carbon (9th gen) */
};
static int __init fan_init(struct ibm_init_struct *iibm)
.properties = chuwi_hi10_plus_props,
};
+static const struct property_entry chuwi_hi10_pro_props[] = {
+ PROPERTY_ENTRY_U32("touchscreen-min-x", 8),
+ PROPERTY_ENTRY_U32("touchscreen-min-y", 8),
+ PROPERTY_ENTRY_U32("touchscreen-size-x", 1912),
+ PROPERTY_ENTRY_U32("touchscreen-size-y", 1272),
+ PROPERTY_ENTRY_BOOL("touchscreen-swapped-x-y"),
+ PROPERTY_ENTRY_STRING("firmware-name", "gsl1680-chuwi-hi10-pro.fw"),
+ PROPERTY_ENTRY_U32("silead,max-fingers", 10),
+ PROPERTY_ENTRY_BOOL("silead,home-button"),
+ { }
+};
+
+static const struct ts_dmi_data chuwi_hi10_pro_data = {
+ .embedded_fw = {
+ .name = "silead/gsl1680-chuwi-hi10-pro.fw",
+ .prefix = { 0xf0, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00 },
+ .length = 42504,
+ .sha256 = { 0xdb, 0x92, 0x68, 0xa8, 0xdb, 0x81, 0x31, 0x00,
+ 0x1f, 0x58, 0x89, 0xdb, 0x19, 0x1b, 0x15, 0x8c,
+ 0x05, 0x14, 0xf4, 0x95, 0xba, 0x15, 0x45, 0x98,
+ 0x42, 0xa3, 0xbb, 0x65, 0xe3, 0x30, 0xa5, 0x93 },
+ },
+ .acpi_name = "MSSL1680:00",
+ .properties = chuwi_hi10_pro_props,
+};
+
static const struct property_entry chuwi_vi8_props[] = {
PROPERTY_ENTRY_U32("touchscreen-min-x", 4),
PROPERTY_ENTRY_U32("touchscreen-min-y", 6),
},
},
{
+ /* Chuwi Hi10 Prus (CWI597) */
+ .driver_data = (void *)&chuwi_hi10_pro_data,
+ .matches = {
+ DMI_MATCH(DMI_BOARD_VENDOR, "Hampoo"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "Hi10 pro tablet"),
+ DMI_MATCH(DMI_BOARD_NAME, "Cherry Trail CR"),
+ },
+ },
+ {
/* Chuwi Vi8 (CWI506) */
.driver_data = (void *)&chuwi_vi8_data,
.matches = {
},
},
{
+ /* Mediacom WinPad 7.0 W700 (same hw as Wintron surftab 7") */
+ .driver_data = (void *)&trekstor_surftab_wintron70_data,
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "MEDIACOM"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "WinPad 7 W10 - WPW700"),
+ },
+ },
+ {
/* Mediacom Flexbook Edge 11 (same hw as TS Primebook C11) */
.driver_data = (void *)&trekstor_primebook_c11_data,
.matches = {
delta = ktime_to_ns(kt);
err = ops->adjtime(ops, delta);
} else if (tx->modes & ADJ_FREQUENCY) {
- s32 ppb = scaled_ppm_to_ppb(tx->freq);
+ long ppb = scaled_ppm_to_ppb(tx->freq);
if (ppb > ops->max_adj || ppb < -ops->max_adj)
return -ERANGE;
if (ops->adjfine)
if (!bp->base) {
dev_err(&pdev->dev, "io_remap bar0\n");
err = -ENOMEM;
- goto out;
+ goto out_release_regions;
}
bp->reg = bp->base + OCP_REGISTER_OFFSET;
bp->tod = bp->base + TOD_REGISTER_OFFSET;
return 0;
out:
+ pci_iounmap(pdev, bp->base);
+out_release_regions:
pci_release_regions(pdev);
out_disable:
pci_disable_device(pdev);
return -ENODEV;
}
+ cm->rx_wq = create_workqueue(DRV_NAME "/rxq");
+ if (!cm->rx_wq) {
+ rio_release_inb_mbox(mport, cmbox);
+ rio_release_outb_mbox(mport, cmbox);
+ kfree(cm);
+ return -ENOMEM;
+ }
+
/*
* Allocate and register inbound messaging buffers to be ready
* to receive channel and system management requests
cm->rx_slots = RIOCM_RX_RING_SIZE;
mutex_init(&cm->rx_lock);
riocm_rx_fill(cm, RIOCM_RX_RING_SIZE);
- cm->rx_wq = create_workqueue(DRV_NAME "/rxq");
- if (!cm->rx_wq) {
- riocm_error("failed to allocate IBMBOX_%d on %s",
- cmbox, mport->name);
- rio_release_outb_mbox(mport, cmbox);
- kfree(cm);
- return -ENOMEM;
- }
-
INIT_WORK(&cm->rx_work, rio_ibmsg_handler);
cm->tx_slot = 0;
current source, LDO and Buck.
config REGULATOR_RTMV20
- tristate "RTMV20 Laser Diode Regulator"
+ tristate "Richtek RTMV20 Laser Diode Regulator"
depends on I2C
select REGMAP_I2C
help
static const struct linear_range atc2609a_ldo_voltage_ranges0[] = {
REGULATOR_LINEAR_RANGE(700000, 0, 15, 100000),
- REGULATOR_LINEAR_RANGE(2100000, 16, 28, 100000),
+ REGULATOR_LINEAR_RANGE(2100000, 0, 12, 100000),
};
static const struct linear_range atc2609a_ldo_voltage_ranges1[] = {
REGULATOR_LINEAR_RANGE(850000, 0, 15, 100000),
- REGULATOR_LINEAR_RANGE(2100000, 16, 27, 100000),
+ REGULATOR_LINEAR_RANGE(2100000, 0, 11, 100000),
};
static const unsigned int atc260x_ldo_voltage_range_sel[] = {
- 0x0, 0x1,
+ 0x0, 0x20,
};
static int atc260x_dcdc_set_voltage_time_sel(struct regulator_dev *rdev,
.owner = THIS_MODULE, \
}
-#define atc2609a_reg_desc_ldo_range_pick(num, n_range) { \
+#define atc2609a_reg_desc_ldo_range_pick(num, n_range, n_volt) { \
.name = "LDO"#num, \
.supply_name = "ldo"#num, \
.of_match = of_match_ptr("ldo"#num), \
.type = REGULATOR_VOLTAGE, \
.linear_ranges = atc2609a_ldo_voltage_ranges##n_range, \
.n_linear_ranges = ARRAY_SIZE(atc2609a_ldo_voltage_ranges##n_range), \
+ .n_voltages = n_volt, \
.vsel_reg = ATC2609A_PMU_LDO##num##_CTL0, \
.vsel_mask = GENMASK(4, 1), \
.vsel_range_reg = ATC2609A_PMU_LDO##num##_CTL0, \
atc2609a_reg_desc_ldo_bypass(0),
atc2609a_reg_desc_ldo_bypass(1),
atc2609a_reg_desc_ldo_bypass(2),
- atc2609a_reg_desc_ldo_range_pick(3, 0),
- atc2609a_reg_desc_ldo_range_pick(4, 0),
+ atc2609a_reg_desc_ldo_range_pick(3, 0, 29),
+ atc2609a_reg_desc_ldo_range_pick(4, 0, 29),
atc2609a_reg_desc_ldo(5),
- atc2609a_reg_desc_ldo_range_pick(6, 1),
- atc2609a_reg_desc_ldo_range_pick(7, 0),
- atc2609a_reg_desc_ldo_range_pick(8, 0),
+ atc2609a_reg_desc_ldo_range_pick(6, 1, 28),
+ atc2609a_reg_desc_ldo_range_pick(7, 0, 29),
+ atc2609a_reg_desc_ldo_range_pick(8, 0, 29),
atc2609a_reg_desc_ldo_fixed(9),
};
NULL);
BD718XX_OPS(bd71837_buck_regulator_nolinear_ops, regulator_list_voltage_table,
- regulator_map_voltage_ascend, bd718xx_set_voltage_sel_restricted,
+ regulator_map_voltage_ascend, bd71837_set_voltage_sel_restricted,
regulator_get_voltage_sel_regmap, regulator_set_voltage_time_sel,
NULL);
/*
* and we have control then make sure it is enabled.
*/
if (rdev->constraints->always_on || rdev->constraints->boot_on) {
+ /* If we want to enable this regulator, make sure that we know
+ * the supplying regulator.
+ */
+ if (rdev->supply_name && !rdev->supply)
+ return -EPROBE_DEFER;
+
if (rdev->supply) {
ret = regulator_enable(rdev->supply);
if (ret < 0) {
drvdata->dev = devm_regulator_register(dev, &drvdata->desc, &cfg);
if (IS_ERR(drvdata->dev)) {
+ ret = PTR_ERR(drvdata->dev);
dev_err(&pdev->dev, "Failed to register regulator: %d\n", ret);
- return PTR_ERR(drvdata->dev);
+ return ret;
}
platform_set_drvdata(pdev, drvdata);
case DA9121_BUCK_MODE_FORCE_PFM:
return REGULATOR_MODE_STANDBY;
default:
- return -EINVAL;
+ return REGULATOR_MODE_INVALID;
}
}
{
struct da9121 *chip = rdev_get_drvdata(rdev);
int id = rdev_get_id(rdev);
- unsigned int val;
+ unsigned int val, mode;
int ret = 0;
ret = regmap_read(chip->regmap, da9121_mode_field[id].reg, &val);
return -EINVAL;
}
- return da9121_map_mode(val & da9121_mode_field[id].msk);
+ mode = da9121_map_mode(val & da9121_mode_field[id].msk);
+ if (mode == REGULATOR_MODE_INVALID)
+ return -EINVAL;
+
+ return mode;
}
static const struct regulator_ops da9121_buck_ops = {
#define FAN53555_NVOLTAGES 64 /* Numbers of voltages */
#define FAN53526_NVOLTAGES 128
-#define TCS4525_NVOLTAGES 127 /* Numbers of voltages */
#define TCS_VSEL_NSEL_MASK 0x7f
#define TCS_VSEL0_MODE (1 << 7)
/* Init voltage range and step */
di->vsel_min = 600000;
di->vsel_step = 6250;
- di->vsel_count = TCS4525_NVOLTAGES;
+ di->vsel_count = FAN53526_NVOLTAGES;
return 0;
}
REGULATOR_LINEAR_RANGE(800000, 0xf, 0x73, 25000), \
}, \
.n_linear_ranges = 2, \
+ .n_voltages = 0x74, \
.vsel_reg = FAN53880_LDO ## _num ## VOUT, \
.vsel_mask = 0x7f, \
.enable_reg = FAN53880_ENABLE, \
REGULATOR_LINEAR_RANGE(600000, 0x1f, 0xf7, 12500),
},
.n_linear_ranges = 2,
+ .n_voltages = 0xf8,
.vsel_reg = FAN53880_BUCKVOUT,
.vsel_mask = 0x7f,
.enable_reg = FAN53880_ENABLE,
REGULATOR_LINEAR_RANGE(3000000, 0x4, 0x70, 25000),
},
.n_linear_ranges = 2,
+ .n_voltages = 0x71,
.vsel_reg = FAN53880_BOOSTVOUT,
.vsel_mask = 0x7f,
.enable_reg = FAN53880_ENABLE_BOOST,
{
struct fixed_voltage_data *priv = rdev_get_drvdata(rdev);
struct device *dev = rdev->dev.parent;
+ int ret;
+
+ ret = dev_pm_genpd_set_performance_state(dev, 0);
+ if (ret)
+ return ret;
priv->enable_counter--;
- return dev_pm_genpd_set_performance_state(dev, 0);
+ return 0;
}
static int reg_is_enabled(struct regulator_dev *rdev)
int ret;
unsigned int sel;
- if (!rdev->desc->n_ramp_values)
+ if (WARN_ON(!rdev->desc->n_ramp_values || !rdev->desc->ramp_delay_table))
return -EINVAL;
ret = find_closest_bigger(ramp_delay, rdev->desc->ramp_delay_table,
// Device driver for regulators in Hisi IC
//
// Copyright (c) 2013 Linaro Ltd.
-// Copyright (c) 2011 Hisilicon.
+// Copyright (c) 2011 HiSilicon Ltd.
// Copyright (c) 2020-2021 Huawei Technologies Co., Ltd
//
// Guodong Xu <guodong.xu@linaro.org>
.owner = THIS_MODULE, \
.volt_table = vtable, \
.n_voltages = ARRAY_SIZE(vtable), \
- .vsel_mask = (1 << (ARRAY_SIZE(vtable) - 1)) - 1, \
+ .vsel_mask = ARRAY_SIZE(vtable) - 1, \
.vsel_reg = vreg, \
.enable_reg = ereg, \
.enable_mask = emask, \
//
// Device driver for regulators in Hi655x IC
//
-// Copyright (c) 2016 Hisilicon.
+// Copyright (c) 2016 HiSilicon Ltd.
//
// Authors:
// Chen Feng <puck.chen@hisilicon.com>
config.dev = dev;
config.driver_data = pmic;
+ /*
+ * Set of_node_reuse flag to prevent driver core from attempting to
+ * claim any pinmux resources already claimed by the parent device.
+ * Otherwise PMIC driver will fail to re-probe.
+ */
+ device_set_of_node_from_dev(&pdev->dev, pdev->dev.parent);
+
for (id = 0; id < MAX77620_NUM_REGS; id++) {
struct regulator_dev *rdev;
struct regulator_desc *rdesc;
return ret;
rdev = devm_regulator_register(dev, rdesc, &config);
- if (IS_ERR(rdev)) {
- ret = PTR_ERR(rdev);
- dev_err(dev, "Regulator registration %s failed: %d\n",
- rdesc->name, ret);
- return ret;
- }
+ if (IS_ERR(rdev))
+ return dev_err_probe(dev, PTR_ERR(rdev),
+ "Regulator registration %s failed\n",
+ rdesc->name);
}
return 0;
REGULATOR_LINEAR_RANGE(0, 0, 0xbf, 6250),
};
-static unsigned int mt6315_map_mode(u32 mode)
+static unsigned int mt6315_map_mode(unsigned int mode)
{
switch (mode) {
case MT6315_BUCK_MODE_AUTO:
struct gpio_descs *gpios = priv->enable_gpios;
int id = rdev_get_id(rdev), ret;
- if (gpios->ndescs <= id) {
+ if (!gpios || gpios->ndescs <= id) {
dev_warn(&rdev->dev, "no dedicated gpio can control\n");
goto bypass_gpio;
}
struct gpio_descs *gpios = priv->enable_gpios;
int id = rdev_get_id(rdev);
- if (gpios->ndescs <= id) {
+ if (!gpios || gpios->ndescs <= id) {
dev_warn(&rdev->dev, "no dedicated gpio can control\n");
goto bypass_gpio;
}
#define RTMV20_REG_LDIRQ 0x30
#define RTMV20_REG_LDSTAT 0x40
#define RTMV20_REG_LDMASK 0x50
+#define RTMV20_MAX_REGS (RTMV20_REG_LDMASK + 1)
#define RTMV20_VID_MASK GENMASK(7, 4)
#define RICHTEK_VID 0x80
return 0;
}
+static int rtmv20_lsw_set_current_limit(struct regulator_dev *rdev, int min_uA,
+ int max_uA)
+{
+ int sel;
+
+ if (min_uA > RTMV20_LSW_MAXUA || max_uA < RTMV20_LSW_MINUA)
+ return -EINVAL;
+
+ if (max_uA > RTMV20_LSW_MAXUA)
+ max_uA = RTMV20_LSW_MAXUA;
+
+ sel = (max_uA - RTMV20_LSW_MINUA) / RTMV20_LSW_STEPUA;
+
+ /* Ensure the selected setting is still in range */
+ if ((sel * RTMV20_LSW_STEPUA + RTMV20_LSW_MINUA) < min_uA)
+ return -EINVAL;
+
+ sel <<= ffs(rdev->desc->csel_mask) - 1;
+
+ return regmap_update_bits(rdev->regmap, rdev->desc->csel_reg,
+ rdev->desc->csel_mask, sel);
+}
+
+static int rtmv20_lsw_get_current_limit(struct regulator_dev *rdev)
+{
+ unsigned int val;
+ int ret;
+
+ ret = regmap_read(rdev->regmap, rdev->desc->csel_reg, &val);
+ if (ret)
+ return ret;
+
+ val &= rdev->desc->csel_mask;
+ val >>= ffs(rdev->desc->csel_mask) - 1;
+
+ return val * RTMV20_LSW_STEPUA + RTMV20_LSW_MINUA;
+}
+
static const struct regulator_ops rtmv20_regulator_ops = {
- .set_current_limit = regulator_set_current_limit_regmap,
- .get_current_limit = regulator_get_current_limit_regmap,
+ .set_current_limit = rtmv20_lsw_set_current_limit,
+ .get_current_limit = rtmv20_lsw_get_current_limit,
.enable = rtmv20_lsw_enable,
.disable = rtmv20_lsw_disable,
.is_enabled = regulator_is_enabled_regmap,
.val_bits = 8,
.cache_type = REGCACHE_RBTREE,
.max_register = RTMV20_REG_LDMASK,
+ .num_reg_defaults_raw = RTMV20_MAX_REGS,
.writeable_reg = rtmv20_is_accessible_reg,
.readable_reg = rtmv20_is_accessible_reg,
sreg->desc.uV_step =
vinfo->levels_uv[SCMI_VOLTAGE_SEGMENT_STEP];
sreg->desc.linear_min_sel = 0;
- sreg->desc.n_voltages = delta_uV / sreg->desc.uV_step;
+ sreg->desc.n_voltages = (delta_uV / sreg->desc.uV_step) + 1;
sreg->desc.ops = &scmi_reg_linear_ops;
}
if (ids)
for (i = 0; ids[i].name[0]; i++)
- if (rpmsg_id_match(rpdev, &ids[i]))
+ if (rpmsg_id_match(rpdev, &ids[i])) {
+ rpdev->id.driver_data = ids[i].driver_data;
return 1;
+ }
return of_driver_match_device(dev, drv);
}
blk_queue_segment_boundary(q, PAGE_SIZE - 1);
}
+static int dasd_diag_pe_handler(struct dasd_device *device,
+ __u8 tbvpm, __u8 fcsecpm)
+{
+ return dasd_generic_verify_path(device, tbvpm);
+}
+
static struct dasd_discipline dasd_diag_discipline = {
.owner = THIS_MODULE,
.name = "DIAG",
.ebcname = "DIAG",
.check_device = dasd_diag_check_device,
- .verify_path = dasd_generic_verify_path,
+ .pe_handler = dasd_diag_pe_handler,
.fill_geometry = dasd_diag_fill_geometry,
.setup_blk_queue = dasd_diag_setup_blk_queue,
.start_IO = dasd_start_diag,
blk_queue_flag_set(QUEUE_FLAG_DISCARD, q);
}
+static int dasd_fba_pe_handler(struct dasd_device *device,
+ __u8 tbvpm, __u8 fcsecpm)
+{
+ return dasd_generic_verify_path(device, tbvpm);
+}
+
static struct dasd_discipline dasd_fba_discipline = {
.owner = THIS_MODULE,
.name = "FBA ",
.ebcname = "FBA ",
.check_device = dasd_fba_check_characteristics,
.do_analysis = dasd_fba_do_analysis,
- .verify_path = dasd_generic_verify_path,
+ .pe_handler = dasd_fba_pe_handler,
.setup_blk_queue = dasd_fba_setup_blk_queue,
.fill_geometry = dasd_fba_fill_geometry,
.start_IO = dasd_start_IO,
* e.g. verify that new path is compatible with the current
* configuration.
*/
- int (*verify_path)(struct dasd_device *, __u8);
int (*pe_handler)(struct dasd_device *, __u8, __u8);
/*
static DEFINE_RATELIMIT_STATE(ratelimit_state, 5 * HZ, 1);
int ret;
+ /* this is an error in the caller */
+ if (cp->initialized)
+ return -EBUSY;
+
/*
* We only support prefetching the channel program. We assume all channel
* programs executed by supported guests likewise support prefetching.
struct vfio_ccw_private *private;
struct irb *irb;
bool is_final;
+ bool cp_is_finished = false;
private = container_of(work, struct vfio_ccw_private, io_work);
irb = &private->irb;
(SCSW_ACTL_DEVACT | SCSW_ACTL_SCHACT));
if (scsw_is_solicited(&irb->scsw)) {
cp_update_scsw(&private->cp, &irb->scsw);
- if (is_final && private->state == VFIO_CCW_STATE_CP_PENDING)
+ if (is_final && private->state == VFIO_CCW_STATE_CP_PENDING) {
cp_free(&private->cp);
+ cp_is_finished = true;
+ }
}
mutex_lock(&private->io_mutex);
memcpy(private->io_region->irb_area, irb, sizeof(*irb));
mutex_unlock(&private->io_mutex);
- if (private->mdev && is_final)
+ /*
+ * Reset to IDLE only if processing of a channel program
+ * has finished. Do not overwrite a possible processing
+ * state if the final interrupt was for HSCH or CSCH.
+ */
+ if (private->mdev && cp_is_finished)
private->state = VFIO_CCW_STATE_IDLE;
if (private->io_trigger)
}
err_out:
+ private->state = VFIO_CCW_STATE_IDLE;
trace_vfio_ccw_fsm_io_request(scsw->cmd.fctl, schid,
io_region->ret_code, errstr);
}
}
vfio_ccw_fsm_event(private, VFIO_CCW_EVENT_IO_REQ);
- if (region->ret_code != 0)
- private->state = VFIO_CCW_STATE_IDLE;
ret = (region->ret_code != 0) ? region->ret_code : count;
out_unlock:
.bus = &iucv_bus,
};
-static int netiucv_callback_connreq(struct iucv_path *, u8 *, u8 *);
-static void netiucv_callback_connack(struct iucv_path *, u8 *);
-static void netiucv_callback_connrej(struct iucv_path *, u8 *);
-static void netiucv_callback_connsusp(struct iucv_path *, u8 *);
-static void netiucv_callback_connres(struct iucv_path *, u8 *);
-static void netiucv_callback_rx(struct iucv_path *, struct iucv_message *);
-static void netiucv_callback_txdone(struct iucv_path *, struct iucv_message *);
-
-static struct iucv_handler netiucv_handler = {
- .path_pending = netiucv_callback_connreq,
- .path_complete = netiucv_callback_connack,
- .path_severed = netiucv_callback_connrej,
- .path_quiesced = netiucv_callback_connsusp,
- .path_resumed = netiucv_callback_connres,
- .message_pending = netiucv_callback_rx,
- .message_complete = netiucv_callback_txdone
-};
-
/**
* Per connection profiling data
*/
}
}
+static struct iucv_handler netiucv_handler = {
+ .path_pending = netiucv_callback_connreq,
+ .path_complete = netiucv_callback_connack,
+ .path_severed = netiucv_callback_connrej,
+ .path_quiesced = netiucv_callback_connsusp,
+ .path_resumed = netiucv_callback_connres,
+ .message_pending = netiucv_callback_rx,
+ .message_complete = netiucv_callback_txdone,
+};
+
static void conn_action_connaccept(fsm_instance *fi, int event, void *arg)
{
struct iucv_event *ev = arg;
QETH_QDIO_BUF_EMPTY,
/* Filled by driver; owned by hardware in order to be sent. */
QETH_QDIO_BUF_PRIMED,
- /* Discovered by the TX completion code: */
- QETH_QDIO_BUF_PENDING,
- /* Finished by the TX completion code: */
- QETH_QDIO_BUF_NEED_QAOB,
- /* Received QAOB notification on CQ: */
- QETH_QDIO_BUF_QAOB_OK,
- QETH_QDIO_BUF_QAOB_ERROR,
+};
+
+enum qeth_qaob_state {
+ QETH_QAOB_ISSUED,
+ QETH_QAOB_PENDING,
+ QETH_QAOB_DONE,
+};
+
+struct qeth_qaob_priv1 {
+ unsigned int state;
+ u8 queue_no;
};
struct qeth_qdio_out_buffer {
unsigned int frames;
unsigned int bytes;
struct sk_buff_head skb_list;
- int is_header[QDIO_MAX_ELEMENTS_PER_BUFFER];
+ DECLARE_BITMAP(from_kmem_cache, QDIO_MAX_ELEMENTS_PER_BUFFER);
- struct qeth_qdio_out_q *q;
struct list_head list_entry;
struct qaob *aob;
};
u64 stopped;
u64 doorbell;
u64 coal_frames;
+ u64 completion_irq;
u64 completion_yield;
u64 completion_timer;
unsigned int coalesce_usecs;
unsigned int max_coalesced_frames;
+ unsigned int rescan_usecs;
};
#define qeth_for_each_output_queue(card, q, i) \
struct ccw_device *ccwdev;
struct qeth_cmd_buffer *active_cmd;
enum qeth_channel_states state;
- atomic_t irq_pending;
};
struct qeth_reply {
return (struct ccw1 *)(iob->data + ALIGN(iob->length, 8));
}
-static inline bool qeth_trylock_channel(struct qeth_channel *channel)
-{
- return atomic_cmpxchg(&channel->irq_pending, 0, 1) == 0;
-}
-
/**
* OSA card related definitions
*/
return card->state == CARD_STATE_SOFTSETUP;
}
+static inline bool qeth_use_tx_irqs(struct qeth_card *card)
+{
+ return !IS_IQD(card);
+}
+
static inline void qeth_unlock_channel(struct qeth_card *card,
struct qeth_channel *channel)
{
- atomic_set(&channel->irq_pending, 0);
+ xchg(&channel->active_cmd, NULL);
wake_up(&card->wait_q);
}
+static inline bool qeth_trylock_channel(struct qeth_channel *channel,
+ struct qeth_cmd_buffer *cmd)
+{
+ return cmpxchg(&channel->active_cmd, NULL, cmd) == NULL;
+}
+
struct qeth_trap_id {
__u16 lparnr;
char vmname[8];
unsigned int data_length);
static int qeth_qdio_establish(struct qeth_card *);
static void qeth_free_qdio_queues(struct qeth_card *card);
-static void qeth_notify_skbs(struct qeth_qdio_out_q *queue,
- struct qeth_qdio_out_buffer *buf,
- enum iucv_tx_notify notification);
static void qeth_close_dev_handler(struct work_struct *work)
{
return n;
}
-static void qeth_qdio_handle_aob(struct qeth_card *card,
- unsigned long phys_aob_addr)
-{
- enum qeth_qdio_out_buffer_state new_state = QETH_QDIO_BUF_QAOB_OK;
- struct qaob *aob;
- struct qeth_qdio_out_buffer *buffer;
- enum iucv_tx_notify notification;
- struct qeth_qdio_out_q *queue;
- unsigned int i;
-
- aob = (struct qaob *) phys_to_virt(phys_aob_addr);
- QETH_CARD_TEXT(card, 5, "haob");
- QETH_CARD_TEXT_(card, 5, "%lx", phys_aob_addr);
- buffer = (struct qeth_qdio_out_buffer *) aob->user1;
- QETH_CARD_TEXT_(card, 5, "%lx", aob->user1);
-
- if (aob->aorc) {
- QETH_CARD_TEXT_(card, 2, "aorc%02X", aob->aorc);
- new_state = QETH_QDIO_BUF_QAOB_ERROR;
- }
-
- switch (atomic_xchg(&buffer->state, new_state)) {
- case QETH_QDIO_BUF_PRIMED:
- /* Faster than TX completion code, let it handle the async
- * completion for us. It will also recycle the QAOB.
- */
- break;
- case QETH_QDIO_BUF_PENDING:
- /* TX completion code is active and will handle the async
- * completion for us. It will also recycle the QAOB.
- */
- break;
- case QETH_QDIO_BUF_NEED_QAOB:
- /* TX completion code is already finished. */
- notification = qeth_compute_cq_notification(aob->aorc, 1);
- qeth_notify_skbs(buffer->q, buffer, notification);
-
- /* Free dangling allocations. The attached skbs are handled by
- * qeth_tx_complete_pending_bufs(), and so is the QAOB.
- */
- for (i = 0;
- i < aob->sb_count && i < QETH_MAX_BUFFER_ELEMENTS(card);
- i++) {
- void *data = phys_to_virt(aob->sba[i]);
-
- if (data && buffer->is_header[i])
- kmem_cache_free(qeth_core_header_cache, data);
- buffer->is_header[i] = 0;
- }
-
- queue = buffer->q;
- atomic_set(&buffer->state, QETH_QDIO_BUF_EMPTY);
- napi_schedule(&queue->napi);
- break;
- default:
- WARN_ON_ONCE(1);
- }
-}
-
static void qeth_setup_ccw(struct ccw1 *ccw, u8 cmd_code, u8 flags, u32 len,
void *data)
{
iob = (struct qeth_cmd_buffer *) (addr_t)intparm;
}
- channel->active_cmd = NULL;
qeth_unlock_channel(card, channel);
rc = qeth_check_irb_error(card, cdev, irb);
}
}
-static void qeth_tx_complete_buf(struct qeth_qdio_out_buffer *buf, bool error,
+static void qeth_tx_complete_buf(struct qeth_qdio_out_q *queue,
+ struct qeth_qdio_out_buffer *buf, bool error,
int budget)
{
- struct qeth_qdio_out_q *queue = buf->q;
struct sk_buff *skb;
/* Empty buffer? */
int i;
/* is PCI flag set on buffer? */
- if (buf->buffer->element[0].sflags & SBAL_SFLAGS0_PCI_REQ)
+ if (buf->buffer->element[0].sflags & SBAL_SFLAGS0_PCI_REQ) {
atomic_dec(&queue->set_pci_flags_count);
+ QETH_TXQ_STAT_INC(queue, completion_irq);
+ }
- qeth_tx_complete_buf(buf, error, budget);
+ qeth_tx_complete_buf(queue, buf, error, budget);
for (i = 0; i < queue->max_elements; ++i) {
void *data = phys_to_virt(buf->buffer->element[i].addr);
- if (data && buf->is_header[i])
+ if (__test_and_clear_bit(i, buf->from_kmem_cache) && data)
kmem_cache_free(qeth_core_header_cache, data);
- buf->is_header[i] = 0;
}
qeth_scrub_qdio_buffer(buf->buffer, queue->max_elements);
struct qeth_qdio_out_buffer *buf, *tmp;
list_for_each_entry_safe(buf, tmp, &queue->pending_bufs, list_entry) {
- if (drain || atomic_read(&buf->state) == QETH_QDIO_BUF_EMPTY) {
+ struct qeth_qaob_priv1 *priv;
+ struct qaob *aob = buf->aob;
+ enum iucv_tx_notify notify;
+ unsigned int i;
+
+ priv = (struct qeth_qaob_priv1 *)&aob->user1;
+ if (drain || READ_ONCE(priv->state) == QETH_QAOB_DONE) {
QETH_CARD_TEXT(card, 5, "fp");
QETH_CARD_TEXT_(card, 5, "%lx", (long) buf);
- if (drain)
- qeth_notify_skbs(queue, buf,
- TX_NOTIFY_GENERALERROR);
- qeth_tx_complete_buf(buf, drain, budget);
+ notify = drain ? TX_NOTIFY_GENERALERROR :
+ qeth_compute_cq_notification(aob->aorc, 1);
+ qeth_notify_skbs(queue, buf, notify);
+ qeth_tx_complete_buf(queue, buf, drain, budget);
+
+ for (i = 0;
+ i < aob->sb_count && i < queue->max_elements;
+ i++) {
+ void *data = phys_to_virt(aob->sba[i]);
+
+ if (test_bit(i, buf->from_kmem_cache) && data)
+ kmem_cache_free(qeth_core_header_cache,
+ data);
+ }
list_del(&buf->list_entry);
qeth_free_out_buf(buf);
rc = ccw_device_set_offline(cdev);
spin_lock_irq(get_ccwdev_lock(cdev));
- if (channel->active_cmd) {
+ if (channel->active_cmd)
dev_err(&cdev->dev, "Stopped channel while cmd %px was still active\n",
channel->active_cmd);
- channel->active_cmd = NULL;
- }
+
cdev->handler = NULL;
spin_unlock_irq(get_ccwdev_lock(cdev));
int rc;
channel->state = CH_STATE_DOWN;
- atomic_set(&channel->irq_pending, 0);
+ xchg(&channel->active_cmd, NULL);
spin_lock_irq(get_ccwdev_lock(cdev));
cdev->handler = qeth_irq;
reply->param = reply_param;
timeout = wait_event_interruptible_timeout(card->wait_q,
- qeth_trylock_channel(channel),
+ qeth_trylock_channel(channel, iob),
timeout);
if (timeout <= 0) {
qeth_put_cmd(iob);
spin_lock_irq(get_ccwdev_lock(channel->ccwdev));
rc = ccw_device_start_timeout(channel->ccwdev, __ccw_from_cmd(iob),
(addr_t) iob, 0, 0, timeout);
- if (!rc)
- channel->active_cmd = iob;
spin_unlock_irq(get_ccwdev_lock(channel->ccwdev));
if (rc) {
QETH_DBF_MESSAGE(2, "qeth_send_control_data on device %x: ccw_device_start rc = %i\n",
newbuf->buffer = q->qdio_bufs[bidx];
skb_queue_head_init(&newbuf->skb_list);
lockdep_set_class(&newbuf->skb_list.lock, &qdio_out_skb_queue_key);
- newbuf->q = q;
atomic_set(&newbuf->state, QETH_QDIO_BUF_EMPTY);
q->bufs[bidx] = newbuf;
return 0;
INIT_LIST_HEAD(&queue->pending_bufs);
spin_lock_init(&queue->lock);
timer_setup(&queue->timer, qeth_tx_completion_timer, 0);
- queue->coalesce_usecs = QETH_TX_COALESCE_USECS;
- queue->max_coalesced_frames = QETH_TX_MAX_COALESCED_FRAMES;
+ if (IS_IQD(card)) {
+ queue->coalesce_usecs = QETH_TX_COALESCE_USECS;
+ queue->max_coalesced_frames = QETH_TX_MAX_COALESCED_FRAMES;
+ queue->rescan_usecs = QETH_TX_TIMER_USECS;
+ } else {
+ queue->coalesce_usecs = USEC_PER_SEC;
+ queue->max_coalesced_frames = 0;
+ queue->rescan_usecs = 10 * USEC_PER_SEC;
+ }
queue->priority = QETH_QIB_PQUE_PRIO_DEFAULT;
}
int count)
{
struct qeth_qdio_out_buffer *buf = queue->bufs[index];
- unsigned int qdio_flags = QDIO_FLAG_SYNC_OUTPUT;
struct qeth_card *card = queue->card;
+ unsigned int frames, usecs;
struct qaob *aob = NULL;
int rc;
int i;
if (!buf->aob)
buf->aob = qdio_allocate_aob();
if (buf->aob) {
+ struct qeth_qaob_priv1 *priv;
+
aob = buf->aob;
- aob->user1 = (u64) buf;
+ priv = (struct qeth_qaob_priv1 *)&aob->user1;
+ priv->state = QETH_QAOB_ISSUED;
+ priv->queue_no = queue->queue_no;
}
}
} else {
buf->buffer->element[0].sflags |= SBAL_SFLAGS0_PCI_REQ;
}
}
-
- if (atomic_read(&queue->set_pci_flags_count))
- qdio_flags |= QDIO_FLAG_PCI_OUT;
}
QETH_TXQ_STAT_INC(queue, doorbell);
- rc = do_QDIO(CARD_DDEV(card), qdio_flags, queue->queue_no, index, count,
- aob);
+ rc = do_QDIO(CARD_DDEV(card), QDIO_FLAG_SYNC_OUTPUT, queue->queue_no,
+ index, count, aob);
switch (rc) {
case 0:
/* ignore temporary SIGA errors without busy condition */
/* Fake the TX completion interrupt: */
- if (IS_IQD(card)) {
- unsigned int frames = READ_ONCE(queue->max_coalesced_frames);
- unsigned int usecs = READ_ONCE(queue->coalesce_usecs);
+ frames = READ_ONCE(queue->max_coalesced_frames);
+ usecs = READ_ONCE(queue->coalesce_usecs);
- if (frames && queue->coalesced_frames >= frames) {
- napi_schedule(&queue->napi);
- queue->coalesced_frames = 0;
- QETH_TXQ_STAT_INC(queue, coal_frames);
- } else if (usecs) {
- qeth_tx_arm_timer(queue, usecs);
- }
+ if (frames && queue->coalesced_frames >= frames) {
+ napi_schedule(&queue->napi);
+ queue->coalesced_frames = 0;
+ QETH_TXQ_STAT_INC(queue, coal_frames);
+ } else if (qeth_use_tx_irqs(card) &&
+ atomic_read(&queue->used_buffers) >= 32) {
+ /* Old behaviour carried over from the qdio layer: */
+ napi_schedule(&queue->napi);
+ QETH_TXQ_STAT_INC(queue, coal_frames);
+ } else if (usecs) {
+ qeth_tx_arm_timer(queue, usecs);
}
break;
}
EXPORT_SYMBOL_GPL(qeth_configure_cq);
+static void qeth_qdio_handle_aob(struct qeth_card *card, struct qaob *aob)
+{
+ struct qeth_qaob_priv1 *priv = (struct qeth_qaob_priv1 *)&aob->user1;
+ unsigned int queue_no = priv->queue_no;
+
+ BUILD_BUG_ON(sizeof(*priv) > ARRAY_SIZE(aob->user1));
+
+ if (xchg(&priv->state, QETH_QAOB_DONE) == QETH_QAOB_PENDING &&
+ queue_no < card->qdio.no_out_queues)
+ napi_schedule(&card->qdio.out_qs[queue_no]->napi);
+}
+
static void qeth_qdio_cq_handler(struct qeth_card *card, unsigned int qdio_err,
unsigned int queue, int first_element,
int count)
buffer->element[e].addr) {
unsigned long phys_aob_addr = buffer->element[e].addr;
- qeth_qdio_handle_aob(card, phys_aob_addr);
+ qeth_qdio_handle_aob(card, phys_to_virt(phys_aob_addr));
++e;
}
qeth_scrub_qdio_buffer(buffer, QDIO_MAX_ELEMENTS_PER_BUFFER);
unsigned long card_ptr)
{
struct qeth_card *card = (struct qeth_card *) card_ptr;
- struct qeth_qdio_out_q *queue = card->qdio.out_qs[__queue];
struct net_device *dev = card->dev;
- struct netdev_queue *txq;
- int i;
QETH_CARD_TEXT(card, 6, "qdouhdl");
if (qdio_error & QDIO_ERROR_FATAL) {
QETH_CARD_TEXT(card, 2, "achkcond");
netif_tx_stop_all_queues(dev);
qeth_schedule_recovery(card);
- return;
}
-
- for (i = first_element; i < (first_element + count); ++i) {
- struct qeth_qdio_out_buffer *buf = queue->bufs[QDIO_BUFNR(i)];
-
- qeth_handle_send_error(card, buf, qdio_error);
- qeth_clear_output_buffer(queue, buf, qdio_error, 0);
- }
-
- atomic_sub(count, &queue->used_buffers);
- qeth_check_outbound_queue(queue);
-
- txq = netdev_get_tx_queue(dev, __queue);
- /* xmit may have observed the full-condition, but not yet stopped the
- * txq. In which case the code below won't trigger. So before returning,
- * xmit will re-check the txq's fill level and wake it up if needed.
- */
- if (netif_tx_queue_stopped(txq) && !qeth_out_queue_is_full(queue))
- netif_tx_wake_queue(txq);
}
/**
/* HW header is allocated from cache: */
if ((void *)hdr != skb->data)
- buf->is_header[element] = 1;
+ __set_bit(element, buf->from_kmem_cache);
/* HW header was pushed and is contiguous with linear part: */
else if (length > 0 && !PAGE_ALIGNED(data) &&
(data == (char *)hdr + hd_len))
init_data.int_parm = (unsigned long) card;
init_data.input_sbal_addr_array = in_sbal_ptrs;
init_data.output_sbal_addr_array = out_sbal_ptrs;
- init_data.scan_threshold = IS_IQD(card) ? 0 : 32;
if (atomic_cmpxchg(&card->qdio.state, QETH_QDIO_ALLOCATED,
QETH_QDIO_ESTABLISHED) == QETH_QDIO_ALLOCATED) {
/* Fetch completed RX buffers: */
if (!card->rx.b_count) {
card->rx.qdio_err = 0;
- card->rx.b_count = qdio_get_next_buffers(
- card->data.ccwdev, 0, &card->rx.b_index,
- &card->rx.qdio_err);
+ card->rx.b_count = qdio_inspect_queue(CARD_DDEV(card),
+ 0, true,
+ &card->rx.b_index,
+ &card->rx.qdio_err);
if (card->rx.b_count <= 0) {
card->rx.b_count = 0;
break;
work_done = qeth_rx_poll(card, budget);
+ if (qeth_use_tx_irqs(card)) {
+ struct qeth_qdio_out_q *queue;
+ unsigned int i;
+
+ qeth_for_each_output_queue(card, queue, i) {
+ if (!qeth_out_queue_is_empty(queue))
+ napi_schedule(&queue->napi);
+ }
+ }
+
if (card->options.cq == QETH_CQ_ENABLED)
qeth_cq_poll(card);
if (qdio_error == QDIO_ERROR_SLSB_PENDING) {
struct qaob *aob = buffer->aob;
+ struct qeth_qaob_priv1 *priv;
+ enum iucv_tx_notify notify;
if (!aob) {
netdev_WARN_ONCE(card->dev,
QETH_CARD_TEXT_(card, 5, "pel%u", bidx);
- switch (atomic_cmpxchg(&buffer->state,
- QETH_QDIO_BUF_PRIMED,
- QETH_QDIO_BUF_PENDING)) {
- case QETH_QDIO_BUF_PRIMED:
- /* We have initial ownership, no QAOB (yet): */
+ priv = (struct qeth_qaob_priv1 *)&aob->user1;
+ /* QAOB hasn't completed yet: */
+ if (xchg(&priv->state, QETH_QAOB_PENDING) != QETH_QAOB_DONE) {
qeth_notify_skbs(queue, buffer, TX_NOTIFY_PENDING);
- /* Handle race with qeth_qdio_handle_aob(): */
- switch (atomic_xchg(&buffer->state,
- QETH_QDIO_BUF_NEED_QAOB)) {
- case QETH_QDIO_BUF_PENDING:
- /* No concurrent QAOB notification. */
-
- /* Prepare the queue slot for immediate re-use: */
- qeth_scrub_qdio_buffer(buffer->buffer, queue->max_elements);
- if (qeth_alloc_out_buf(queue, bidx,
- GFP_ATOMIC)) {
- QETH_CARD_TEXT(card, 2, "outofbuf");
- qeth_schedule_recovery(card);
- }
-
- list_add(&buffer->list_entry,
- &queue->pending_bufs);
- /* Skip clearing the buffer: */
- return;
- case QETH_QDIO_BUF_QAOB_OK:
- qeth_notify_skbs(queue, buffer,
- TX_NOTIFY_DELAYED_OK);
- error = false;
- break;
- case QETH_QDIO_BUF_QAOB_ERROR:
- qeth_notify_skbs(queue, buffer,
- TX_NOTIFY_DELAYED_GENERALERROR);
- error = true;
- break;
- default:
- WARN_ON_ONCE(1);
+ /* Prepare the queue slot for immediate re-use: */
+ qeth_scrub_qdio_buffer(buffer->buffer, queue->max_elements);
+ if (qeth_alloc_out_buf(queue, bidx, GFP_ATOMIC)) {
+ QETH_CARD_TEXT(card, 2, "outofbuf");
+ qeth_schedule_recovery(card);
}
- break;
- case QETH_QDIO_BUF_QAOB_OK:
- /* qeth_qdio_handle_aob() already received a QAOB: */
- qeth_notify_skbs(queue, buffer, TX_NOTIFY_OK);
- error = false;
- break;
- case QETH_QDIO_BUF_QAOB_ERROR:
- /* qeth_qdio_handle_aob() already received a QAOB: */
- qeth_notify_skbs(queue, buffer, TX_NOTIFY_GENERALERROR);
- error = true;
- break;
- default:
- WARN_ON_ONCE(1);
+ list_add(&buffer->list_entry, &queue->pending_bufs);
+ /* Skip clearing the buffer: */
+ return;
}
+ /* QAOB already completed: */
+ notify = qeth_compute_cq_notification(aob->aorc, 0);
+ qeth_notify_skbs(queue, buffer, notify);
+ error = !!aob->aorc;
memset(aob, 0, sizeof(*aob));
} else if (card->options.cq == QETH_CQ_ENABLED) {
qeth_notify_skbs(queue, buffer,
unsigned int work_done = 0;
struct netdev_queue *txq;
- txq = netdev_get_tx_queue(dev, qeth_iqd_translate_txq(dev, queue_no));
+ if (IS_IQD(card))
+ txq = netdev_get_tx_queue(dev, qeth_iqd_translate_txq(dev, queue_no));
+ else
+ txq = netdev_get_tx_queue(dev, queue_no);
while (1) {
unsigned int start, error, i;
&start, &error);
if (completed <= 0) {
/* Ensure we see TX completion for pending work: */
- if (napi_complete_done(napi, 0))
- qeth_tx_arm_timer(queue, QETH_TX_TIMER_USECS);
+ if (napi_complete_done(napi, 0) &&
+ !atomic_read(&queue->set_pci_flags_count))
+ qeth_tx_arm_timer(queue, queue->rescan_usecs);
return 0;
}
bytes += buffer->bytes;
qeth_handle_send_error(card, buffer, error);
- qeth_iqd_tx_complete(queue, bidx, error, budget);
+ if (IS_IQD(card))
+ qeth_iqd_tx_complete(queue, bidx, error, budget);
+ else
+ qeth_clear_output_buffer(queue, buffer, error,
+ budget);
}
- netdev_tx_completed_queue(txq, packets, bytes);
atomic_sub(completed, &queue->used_buffers);
work_done += completed;
+ if (IS_IQD(card))
+ netdev_tx_completed_queue(txq, packets, bytes);
+ else
+ qeth_check_outbound_queue(queue);
/* xmit may have observed the full-condition, but not yet
* stopped the txq. In which case the code below won't trigger.
int qeth_open(struct net_device *dev)
{
struct qeth_card *card = dev->ml_priv;
+ struct qeth_qdio_out_q *queue;
+ unsigned int i;
QETH_CARD_TEXT(card, 4, "qethopen");
netif_tx_start_all_queues(dev);
local_bh_disable();
- if (IS_IQD(card)) {
- struct qeth_qdio_out_q *queue;
- unsigned int i;
-
- qeth_for_each_output_queue(card, queue, i) {
- netif_tx_napi_add(dev, &queue->napi, qeth_tx_poll,
- QETH_NAPI_WEIGHT);
- napi_enable(&queue->napi);
- napi_schedule(&queue->napi);
- }
+ qeth_for_each_output_queue(card, queue, i) {
+ netif_tx_napi_add(dev, &queue->napi, qeth_tx_poll,
+ QETH_NAPI_WEIGHT);
+ napi_enable(&queue->napi);
+ napi_schedule(&queue->napi);
}
napi_enable(&card->napi);
int qeth_stop(struct net_device *dev)
{
struct qeth_card *card = dev->ml_priv;
+ struct qeth_qdio_out_q *queue;
+ unsigned int i;
QETH_CARD_TEXT(card, 4, "qethstop");
cancel_delayed_work_sync(&card->buffer_reclaim_work);
qdio_stop_irq(CARD_DDEV(card));
- if (IS_IQD(card)) {
- struct qeth_qdio_out_q *queue;
- unsigned int i;
-
- /* Quiesce the NAPI instances: */
- qeth_for_each_output_queue(card, queue, i)
- napi_disable(&queue->napi);
+ /* Quiesce the NAPI instances: */
+ qeth_for_each_output_queue(card, queue, i)
+ napi_disable(&queue->napi);
- /* Stop .ndo_start_xmit, might still access queue->napi. */
- netif_tx_disable(dev);
+ /* Stop .ndo_start_xmit, might still access queue->napi. */
+ netif_tx_disable(dev);
- qeth_for_each_output_queue(card, queue, i) {
- del_timer_sync(&queue->timer);
- /* Queues may get re-allocated, so remove the NAPIs. */
- netif_napi_del(&queue->napi);
- }
- } else {
- netif_tx_disable(dev);
+ qeth_for_each_output_queue(card, queue, i) {
+ del_timer_sync(&queue->timer);
+ /* Queues may get re-allocated, so remove the NAPIs. */
+ netif_napi_del(&queue->napi);
}
return 0;
QETH_TXQ_STAT("Queue stopped", stopped),
QETH_TXQ_STAT("Doorbell", doorbell),
QETH_TXQ_STAT("IRQ for frames", coal_frames),
+ QETH_TXQ_STAT("Completion IRQ", completion_irq),
QETH_TXQ_STAT("Completion yield", completion_yield),
QETH_TXQ_STAT("Completion timer", completion_timer),
};
{
unsigned int i;
- for (i = 0; i < size; i++) {
- snprintf(*data, ETH_GSTRING_LEN, "%s%s", prefix, stats[i].name);
- *data += ETH_GSTRING_LEN;
- }
+ for (i = 0; i < size; i++)
+ ethtool_sprintf(data, "%s%s", prefix, stats[i].name);
}
static int qeth_get_sset_count(struct net_device *dev, int stringset)
if (!netif_device_present(dev))
return -ENODEV;
- if (!(priv->brport_hw_features))
- return -EOPNOTSUPP;
nlmsg_for_each_attr(attr, nlh, sizeof(struct ifinfomsg), rem1) {
if (nla_type(attr) == IFLA_PROTINFO) {
return 0;
if (!bp_tb[IFLA_BRPORT_LEARNING_SYNC])
return -EINVAL;
+ if (!(priv->brport_hw_features & BR_LEARNING_SYNC)) {
+ NL_SET_ERR_MSG_ATTR(extack, bp_tb[IFLA_BRPORT_LEARNING_SYNC],
+ "Operation not supported by HW");
+ return -EOPNOTSUPP;
+ }
+ if (!IS_ENABLED(CONFIG_NET_SWITCHDEV)) {
+ NL_SET_ERR_MSG_ATTR(extack, bp_tb[IFLA_BRPORT_LEARNING_SYNC],
+ "Requires NET_SWITCHDEV");
+ return -EOPNOTSUPP;
+ }
enable = !!nla_get_u8(bp_tb[IFLA_BRPORT_LEARNING_SYNC]);
if (enable == !!(priv->brport_features & BR_LEARNING_SYNC))
ccb->opcode = BLOGIC_INITIATOR_CCB_SG;
ccb->datalen = count * sizeof(struct blogic_sg_seg);
if (blogic_multimaster_type(adapter))
- ccb->data = (void *)((unsigned int) ccb->dma_handle +
+ ccb->data = (unsigned int) ccb->dma_handle +
((unsigned long) &ccb->sglist -
- (unsigned long) ccb));
+ (unsigned long) ccb);
else
- ccb->data = ccb->sglist;
+ ccb->data = virt_to_32bit_virt(ccb->sglist);
scsi_for_each_sg(command, sg, count, i) {
ccb->sglist[i].segbytes = sg_dma_len(sg);
unsigned char cdblen; /* Byte 2 */
unsigned char sense_datalen; /* Byte 3 */
u32 datalen; /* Bytes 4-7 */
- void *data; /* Bytes 8-11 */
+ u32 data; /* Bytes 8-11 */
unsigned char:8; /* Byte 12 */
unsigned char:8; /* Byte 13 */
enum blogic_adapter_status adapter_status; /* Byte 14 */
#include "aicasm_symbol.h"
#include "aicasm_insformat.h"
-int yylineno;
char *yyfilename;
char stock_prefix[] = "aic_";
char *prefix = stock_prefix;
regex_t arg_regex;
char *replacement_text;
};
-STAILQ_HEAD(macro_arg_list, macro_arg) args;
+STAILQ_HEAD(macro_arg_list, macro_arg);
struct macro_info {
struct macro_arg_list args;
* $FreeBSD: src/sys/cam/scsi/scsi_message.h,v 1.2 2000/05/01 20:21:29 peter Exp $
*/
+/* Messages (1 byte) */ /* I/T (M)andatory or (O)ptional */
+#define MSG_SAVEDATAPOINTER 0x02 /* O/O */
+#define MSG_RESTOREPOINTERS 0x03 /* O/O */
+#define MSG_DISCONNECT 0x04 /* O/O */
+#define MSG_MESSAGE_REJECT 0x07 /* M/M */
+#define MSG_NOOP 0x08 /* M/M */
+
+/* Messages (2 byte) */
+#define MSG_SIMPLE_Q_TAG 0x20 /* O/O */
+#define MSG_IGN_WIDE_RESIDUE 0x23 /* O/O */
+
/* Identify message */ /* M/M */
#define MSG_IDENTIFYFLAG 0x80
#define MSG_IDENTIFY_DISCFLAG 0x40
was a result from the ABTS request rather than the CLEANUP
request */
set_bit(BNX2FC_FLAG_IO_CLEANUP, &io_req->req_flags);
+ rc = FAILED;
goto done;
}
{
int i;
- free_irq(pci_irq_vector(pdev, 1), hisi_hba);
- free_irq(pci_irq_vector(pdev, 2), hisi_hba);
- free_irq(pci_irq_vector(pdev, 11), hisi_hba);
+ devm_free_irq(&pdev->dev, pci_irq_vector(pdev, 1), hisi_hba);
+ devm_free_irq(&pdev->dev, pci_irq_vector(pdev, 2), hisi_hba);
+ devm_free_irq(&pdev->dev, pci_irq_vector(pdev, 11), hisi_hba);
for (i = 0; i < hisi_hba->cq_nvecs; i++) {
struct hisi_sas_cq *cq = &hisi_hba->cq[i];
int nr = hisi_sas_intr_conv ? 16 : 16 + i;
- free_irq(pci_irq_vector(pdev, nr), cq);
+ devm_free_irq(&pdev->dev, pci_irq_vector(pdev, nr), cq);
}
pci_free_irq_vectors(pdev);
}
device_enable_async_suspend(&shost->shost_dev);
+ get_device(&shost->shost_gendev);
error = device_add(&shost->shost_dev);
if (error)
goto out_del_gendev;
- get_device(&shost->shost_gendev);
-
if (shost->transportt->host_size) {
shost->shost_data = kzalloc(shost->transportt->host_size,
GFP_KERNEL);
if (!shost->work_q) {
error = -EINVAL;
- goto out_free_shost_data;
+ goto out_del_dev;
}
}
error = scsi_sysfs_add_host(shost);
if (error)
- goto out_destroy_host;
+ goto out_del_dev;
scsi_proc_host_add(shost);
scsi_autopm_put_host(shost);
return error;
- out_destroy_host:
- if (shost->work_q)
- destroy_workqueue(shost->work_q);
- out_free_shost_data:
- kfree(shost->shost_data);
+ /*
+ * Any host allocation in this function will be freed in
+ * scsi_host_dev_release().
+ */
out_del_dev:
device_del(&shost->shost_dev);
out_del_gendev:
+ /*
+ * Host state is SHOST_RUNNING so we have to explicitly release
+ * ->shost_dev.
+ */
+ put_device(&shost->shost_dev);
device_del(&shost->shost_gendev);
out_disable_runtime_pm:
device_disable_async_suspend(&shost->shost_gendev);
pm_runtime_disable(&shost->shost_gendev);
pm_runtime_set_suspended(&shost->shost_gendev);
pm_runtime_put_noidle(&shost->shost_gendev);
- scsi_mq_destroy_tags(shost);
fail:
return error;
}
ida_simple_remove(&host_index_ida, shost->host_no);
- if (parent)
+ if (shost->shost_state != SHOST_CREATED)
put_device(parent);
kfree(shost);
}
mutex_init(&shost->scan_mutex);
index = ida_simple_get(&host_index_ida, 0, 0, GFP_KERNEL);
- if (index < 0)
- goto fail_kfree;
+ if (index < 0) {
+ kfree(shost);
+ return NULL;
+ }
shost->host_no = index;
shost->dma_channel = 0xff;
shost_printk(KERN_WARNING, shost,
"error handler thread failed to spawn, error = %ld\n",
PTR_ERR(shost->ehandler));
- goto fail_index_remove;
+ goto fail;
}
shost->tmf_work_q = alloc_workqueue("scsi_tmf_%d",
if (!shost->tmf_work_q) {
shost_printk(KERN_WARNING, shost,
"failed to create tmf workq\n");
- goto fail_kthread;
+ goto fail;
}
scsi_proc_hostdir_add(shost->hostt);
return shost;
+ fail:
+ /*
+ * Host state is still SHOST_CREATED and that is enough to release
+ * ->shost_gendev. scsi_host_dev_release() will free
+ * dev_name(&shost->shost_dev).
+ */
+ put_device(&shost->shost_gendev);
- fail_kthread:
- kthread_stop(shost->ehandler);
- fail_index_remove:
- ida_simple_remove(&host_index_ida, shost->host_no);
- fail_kfree:
- kfree(shost);
return NULL;
}
EXPORT_SYMBOL(scsi_host_alloc);
static void sas_resume_port(struct asd_sas_phy *phy)
{
- struct domain_device *dev;
+ struct domain_device *dev, *n;
struct asd_sas_port *port = phy->port;
struct sas_ha_struct *sas_ha = phy->ha;
struct sas_internal *si = to_sas_internal(sas_ha->core.shost->transportt);
* 1/ presume every device came back
* 2/ force the next revalidation to check all expander phys
*/
- list_for_each_entry(dev, &port->dev_list, dev_list_node) {
+ list_for_each_entry_safe(dev, n, &port->dev_list, dev_list_node) {
int i, rc;
rc = sas_notify_lldd_dev_found(dev);
abtswqe = &abtsiocb->wqe;
memset(abtswqe, 0, sizeof(*abtswqe));
- if (lpfc_is_link_up(phba))
+ if (!lpfc_is_link_up(phba))
bf_set(abort_cmd_ia, &abtswqe->abort_cmd, 1);
- else
- bf_set(abort_cmd_ia, &abtswqe->abort_cmd, 0);
bf_set(abort_cmd_criteria, &abtswqe->abort_cmd, T_XRI_TAG);
abtswqe->abort_cmd.rsrvd5 = 0;
abtswqe->abort_cmd.wqe_com.abort_tag = xritag;
case HW_EVENT_PHY_START_STATUS:
pm8001_dbg(pm8001_ha, MSG, "HW_EVENT_PHY_START_STATUS status = %x\n",
status);
- if (status == 0) {
+ if (status == 0)
phy->phy_state = 1;
- if (pm8001_ha->flags == PM8001F_RUN_TIME &&
- phy->enable_completion != NULL)
- complete(phy->enable_completion);
+
+ if (pm8001_ha->flags == PM8001F_RUN_TIME &&
+ phy->enable_completion != NULL) {
+ complete(phy->enable_completion);
+ phy->enable_completion = NULL;
}
break;
case HW_EVENT_SAS_PHY_UP:
goto err_out_shost;
}
list_add_tail(&pm8001_ha->list, &hba_list);
- scsi_scan_host(pm8001_ha->shost);
pm8001_ha->flags = PM8001F_RUN_TIME;
+ scsi_scan_host(pm8001_ha->shost);
return 0;
err_out_shost:
int i;
struct pm8001_hba_info *pm8001_ha;
struct sas_ha_struct *sha = SHOST_TO_SAS_HA(shost);
+ DECLARE_COMPLETION_ONSTACK(completion);
pm8001_ha = sha->lldd_ha;
/* SAS_RE_INITIALIZATION not available in SPCv/ve */
if (pm8001_ha->chip_id == chip_8001)
PM8001_CHIP_DISP->sas_re_init_req(pm8001_ha);
- for (i = 0; i < pm8001_ha->chip->n_phy; ++i)
+ for (i = 0; i < pm8001_ha->chip->n_phy; ++i) {
+ pm8001_ha->phy[i].enable_completion = &completion;
PM8001_CHIP_DISP->phy_start_req(pm8001_ha, i);
+ wait_for_completion(&completion);
+ msleep(300);
+ }
}
int pm8001_scan_finished(struct Scsi_Host *shost, unsigned long time)
pm8001_dbg(pm8001_ha, INIT,
"phy start resp status:0x%x, phyid:0x%x\n",
status, phy_id);
- if (status == 0) {
+ if (status == 0)
phy->phy_state = PHY_LINK_DOWN;
- if (pm8001_ha->flags == PM8001F_RUN_TIME &&
- phy->enable_completion != NULL) {
- complete(phy->enable_completion);
- phy->enable_completion = NULL;
- }
+
+ if (pm8001_ha->flags == PM8001F_RUN_TIME &&
+ phy->enable_completion != NULL) {
+ complete(phy->enable_completion);
+ phy->enable_completion = NULL;
}
return 0;
if (linkmode_intersects(link->supported_caps, sup_caps))
lport->link_supported_speeds |= FC_PORTSPEED_20GBIT;
- fc_host_supported_speeds(lport->host) = lport->link_supported_speeds;
+ if (lport->host && lport->host->shost_data)
+ fc_host_supported_speeds(lport->host) =
+ lport->link_supported_speeds;
}
static void qedf_bw_update(void *dev)
fcoe_wwn_to_str(vport->port_name, buf, sizeof(buf));
QEDF_WARN(&(base_qedf->dbg_ctx), "Failed to create vport, "
"WWPN (0x%s) already exists.\n", buf);
- goto err1;
+ return rc;
}
if (atomic_read(&base_qedf->link_state) != QEDF_LINK_UP) {
QEDF_WARN(&(base_qedf->dbg_ctx), "Cannot create vport "
"because link is not up.\n");
- rc = -EIO;
- goto err1;
+ return -EIO;
}
vn_port = libfc_vport_create(vport, sizeof(struct qedf_ctx));
if (!vn_port) {
QEDF_WARN(&(base_qedf->dbg_ctx), "Could not create lport "
"for vport.\n");
- rc = -ENOMEM;
- goto err1;
+ return -ENOMEM;
}
fcoe_wwn_to_str(vport->port_name, buf, sizeof(buf));
if (rc) {
QEDF_ERR(&(base_qedf->dbg_ctx), "Could not allocate memory "
"for lport stats.\n");
- goto err2;
+ goto err;
}
fc_set_wwnn(vn_port, vport->node_name);
if (rc) {
QEDF_WARN(&base_qedf->dbg_ctx,
"Error adding Scsi_Host rc=0x%x.\n", rc);
- goto err2;
+ goto err;
}
/* Set default dev_loss_tmo based on module parameter */
vport_qedf->dbg_ctx.host_no = vn_port->host->host_no;
vport_qedf->dbg_ctx.pdev = base_qedf->pdev;
-err2:
+ return 0;
+
+err:
scsi_host_put(vn_port->host);
-err1:
return rc;
}
fc_lport_free_stats(vn_port);
/* Release Scsi_Host */
- if (vn_port->host)
- scsi_host_put(vn_port->host);
+ scsi_host_put(vn_port->host);
out:
return 0;
return ret;
}
- if (qla82xx_flash_set_write_enable(ha))
+ ret = qla82xx_flash_set_write_enable(ha);
+ if (ret < 0)
goto done_write;
qla82xx_wr_32(ha, QLA82XX_ROMUSB_ROM_WDATA, data);
return;
}
+ mutex_lock(&tgt->ha->optrom_mutex);
mutex_lock(&vha->vha_tgt.tgt_mutex);
tgt->tgt_stop = 0;
tgt->tgt_stopped = 1;
mutex_unlock(&vha->vha_tgt.tgt_mutex);
+ mutex_unlock(&tgt->ha->optrom_mutex);
ql_dbg(ql_dbg_tgt_mgt, vha, 0xf00c, "Stop of tgt %p finished\n",
tgt);
{"HP", "C3323-300", "4269", BLIST_NOTQ},
{"HP", "C5713A", NULL, BLIST_NOREPORTLUN},
{"HP", "DISK-SUBSYSTEM", "*", BLIST_REPORTLUN2},
+ {"HPE", "OPEN-", "*", BLIST_REPORTLUN2 | BLIST_TRY_VPD_PAGES},
{"IBM", "AuSaV1S2", NULL, BLIST_FORCELUN},
{"IBM", "ProFibre 4000R", "*", BLIST_SPARSELUN | BLIST_LARGELUN},
{"IBM", "2105", NULL, BLIST_RETRY_HWERROR},
host->hba = hba;
ufshcd_set_variant(hba, host);
- host->rst = devm_reset_control_get(dev, "rst");
+ host->rst = devm_reset_control_get(dev, "rst");
if (IS_ERR(host->rst)) {
dev_err(dev, "%s: failed to get reset control\n", __func__);
- return PTR_ERR(host->rst);
+ err = PTR_ERR(host->rst);
+ goto error;
}
ufs_hisi_set_pm_lvl(hba);
err = ufs_hisi_get_resource(host);
- if (err) {
- ufshcd_set_variant(hba, NULL);
- return err;
- }
+ if (err)
+ goto error;
return 0;
+
+error:
+ ufshcd_set_variant(hba, NULL);
+ return err;
}
static int ufs_hi3660_init(struct ufs_hba *hba)
ret = ufshcd_dme_get(hba, UIC_ARG_MIB(PA_LOCALVERINFO), &ver);
if (!ret) {
- if (ver >= UFS_UNIPRO_VER_1_8)
+ if (ver >= UFS_UNIPRO_VER_1_8) {
host->hw_ver.major = 3;
+ /*
+ * Fix HCI version for some platforms with
+ * incorrect version
+ */
+ if (hba->ufs_version < ufshci_version(3, 0))
+ hba->ufs_version = ufshci_version(3, 0);
+ }
}
}
+static u32 ufs_mtk_get_ufs_hci_version(struct ufs_hba *hba)
+{
+ return hba->ufs_version;
+}
+
/**
* ufs_mtk_init - find other essential mmio bases
* @hba: host controller instance
static int ufs_mtk_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)
{
int err;
+ struct arm_smccc_res res;
if (ufshcd_is_link_hibern8(hba)) {
err = ufs_mtk_link_set_lpm(hba);
goto fail;
}
+ if (ufshcd_is_link_off(hba))
+ ufs_mtk_device_reset_ctrl(0, res);
+
return 0;
fail:
/*
static const struct ufs_hba_variant_ops ufs_hba_mtk_vops = {
.name = "mediatek.ufshci",
.init = ufs_mtk_init,
+ .get_ufs_hci_version = ufs_mtk_get_ufs_hci_version,
.setup_clocks = ufs_mtk_setup_clocks,
.hce_enable_notify = ufs_mtk_hce_enable_notify,
.link_startup_notify = ufs_mtk_link_startup_notify,
* ufshcd_exec_dev_cmd - API for sending device management requests
* @hba: UFS hba
* @cmd_type: specifies the type (NOP, Query...)
- * @timeout: time in seconds
+ * @timeout: timeout in milliseconds
*
* NOTE: Since there is only one available tag for device management commands,
* it is expected you hold the hba->dev_cmd.lock mutex.
}
tag = req->tag;
WARN_ON_ONCE(!ufshcd_valid_tag(hba, tag));
+ /* Set the timeout such that the SCSI error handler is not activated. */
+ req->timeout = msecs_to_jiffies(2 * timeout);
+ blk_mq_start_request(req);
init_completion(&wait);
lrbp = &hba->lrb[tag];
case BTSTAT_SUCCESS:
case BTSTAT_LINKED_COMMAND_COMPLETED:
case BTSTAT_LINKED_COMMAND_COMPLETED_WITH_FLAG:
- /* If everything went fine, let's move on.. */
+ /*
+ * Commands like INQUIRY may transfer less data than
+ * requested by the initiator via bufflen. Set residual
+ * count to make upper layer aware of the actual amount
+ * of data returned.
+ */
+ scsi_set_resid(cmd, scsi_bufflen(cmd) - e->dataLen);
cmd->result = (DID_OK << 16);
break;
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(base)) {
- dev_err(&pdev->dev, "io resource mapping failed\n");
+ if (IS_ERR(base))
return PTR_ERR(base);
- }
priv->regmap = devm_regmap_init_mmio(&pdev->dev, base,
&meson_clk_msr_regmap_config);
ret = of_property_read_u8_array(np, "qcom,ports-block-pack-mode",
bp_mode, nports);
- if (ret)
- return ret;
+ if (ret) {
+ u32 version;
+
+ ctrl->reg_read(ctrl, SWRM_COMP_HW_VERSION, &version);
+
+ if (version <= 0x01030000)
+ memset(bp_mode, SWR_INVALID_PARAM, QCOM_SDW_MAX_PORTS);
+ else
+ return ret;
+ }
memset(hstart, SWR_INVALID_PARAM, QCOM_SDW_MAX_PORTS);
of_property_read_u8_array(np, "qcom,ports-hstart", hstart, nports);
This is the driver for the Altera SPI Controller.
config SPI_ALTERA_CORE
- tristate "Altera SPI Controller core code"
+ tristate "Altera SPI Controller core code" if COMPILE_TEST
select REGMAP
help
"The core code for the Altera SPI Controller"
#define BCM2835_SPI_FIFO_SIZE 64
#define BCM2835_SPI_FIFO_SIZE_3_4 48
#define BCM2835_SPI_DMA_MIN_LENGTH 96
-#define BCM2835_SPI_NUM_CS 4 /* raise as necessary */
+#define BCM2835_SPI_NUM_CS 24 /* raise as necessary */
#define BCM2835_SPI_MODE_BITS (SPI_CPOL | SPI_CPHA | SPI_CS_HIGH \
| SPI_NO_CS | SPI_3WIRE)
struct gpio_chip *chip;
u32 cs;
+ if (spi->chip_select >= BCM2835_SPI_NUM_CS) {
+ dev_err(&spi->dev, "only %d chip-selects supported\n",
+ BCM2835_SPI_NUM_CS - 1);
+ return -EINVAL;
+ }
+
/*
* Precalculate SPI slave's CS register value for ->prepare_message():
* The driver always uses software-controlled GPIO chip select, hence
ctlr->use_gpio_descriptors = true;
ctlr->mode_bits = BCM2835_SPI_MODE_BITS;
ctlr->bits_per_word_mask = SPI_BPW_MASK(8);
- ctlr->num_chipselect = BCM2835_SPI_NUM_CS;
+ ctlr->num_chipselect = 3;
ctlr->setup = bcm2835_spi_setup;
ctlr->transfer_one = bcm2835_spi_transfer_one;
ctlr->handle_err = bcm2835_spi_handle_err;
{
struct spi_bitbang_cs *cs = spi->controller_state;
struct spi_bitbang *bitbang;
+ bool initial_setup = false;
+ int retval;
bitbang = spi_master_get_devdata(spi->master);
if (!cs)
return -ENOMEM;
spi->controller_state = cs;
+ initial_setup = true;
}
/* per-word shift register access, in hardware or bitbanging */
cs->txrx_word = bitbang->txrx_word[spi->mode & (SPI_CPOL|SPI_CPHA)];
- if (!cs->txrx_word)
- return -EINVAL;
+ if (!cs->txrx_word) {
+ retval = -EINVAL;
+ goto err_free;
+ }
if (bitbang->setup_transfer) {
- int retval = bitbang->setup_transfer(spi, NULL);
+ retval = bitbang->setup_transfer(spi, NULL);
if (retval < 0)
- return retval;
+ goto err_free;
}
dev_dbg(&spi->dev, "%s, %u nsec/bit\n", __func__, 2 * cs->nsecs);
return 0;
+
+err_free:
+ if (initial_setup)
+ kfree(cs);
+ return retval;
}
EXPORT_SYMBOL_GPL(spi_bitbang_setup);
ret = spi_register_controller(ctlr);
if (ret != 0) {
dev_err(&pdev->dev, "Problem registering DSPI ctlr\n");
- goto out_free_irq;
+ goto out_release_dma;
}
return ret;
+out_release_dma:
+ dspi_release_dma(dspi);
out_free_irq:
if (dspi->irq)
free_irq(dspi->irq, dspi);
{
struct mpc8xxx_spi *mpc8xxx_spi;
struct fsl_spi_reg __iomem *reg_base;
+ bool initial_setup = false;
int retval;
u32 hw_mode;
struct spi_mpc8xxx_cs *cs = spi_get_ctldata(spi);
if (!cs)
return -ENOMEM;
spi_set_ctldata(spi, cs);
+ initial_setup = true;
}
mpc8xxx_spi = spi_master_get_devdata(spi->master);
retval = fsl_spi_setup_transfer(spi, NULL);
if (retval < 0) {
cs->hw_mode = hw_mode; /* Restore settings */
+ if (initial_setup)
+ kfree(cs);
return retval;
}
static int uwire_setup(struct spi_device *spi)
{
struct uwire_state *ust = spi->controller_state;
+ bool initial_setup = false;
+ int status;
if (ust == NULL) {
ust = kzalloc(sizeof(*ust), GFP_KERNEL);
if (ust == NULL)
return -ENOMEM;
spi->controller_state = ust;
+ initial_setup = true;
}
- return uwire_setup_transfer(spi, NULL);
+ status = uwire_setup_transfer(spi, NULL);
+ if (status && initial_setup)
+ kfree(ust);
+
+ return status;
}
static void uwire_cleanup(struct spi_device *spi)
}
}
+static void omap2_mcspi_cleanup(struct spi_device *spi)
+{
+ struct omap2_mcspi_cs *cs;
+
+ if (spi->controller_state) {
+ /* Unlink controller state from context save list */
+ cs = spi->controller_state;
+ list_del(&cs->node);
+
+ kfree(cs);
+ }
+}
+
static int omap2_mcspi_setup(struct spi_device *spi)
{
+ bool initial_setup = false;
int ret;
struct omap2_mcspi *mcspi = spi_master_get_devdata(spi->master);
struct omap2_mcspi_regs *ctx = &mcspi->ctx;
spi->controller_state = cs;
/* Link this to context save list */
list_add_tail(&cs->node, &ctx->cs);
+ initial_setup = true;
}
ret = pm_runtime_get_sync(mcspi->dev);
if (ret < 0) {
pm_runtime_put_noidle(mcspi->dev);
+ if (initial_setup)
+ omap2_mcspi_cleanup(spi);
return ret;
}
ret = omap2_mcspi_setup_transfer(spi, NULL);
+ if (ret && initial_setup)
+ omap2_mcspi_cleanup(spi);
+
pm_runtime_mark_last_busy(mcspi->dev);
pm_runtime_put_autosuspend(mcspi->dev);
return ret;
}
-static void omap2_mcspi_cleanup(struct spi_device *spi)
-{
- struct omap2_mcspi_cs *cs;
-
- if (spi->controller_state) {
- /* Unlink controller state from context save list */
- cs = spi->controller_state;
- list_del(&cs->node);
-
- kfree(cs);
- }
-}
-
static irqreturn_t omap2_mcspi_irq_handler(int irq, void *data)
{
struct omap2_mcspi *mcspi = data;
chip->gpio_cs_inverted = spi->mode & SPI_CS_HIGH;
err = gpiod_direction_output(gpiod, !chip->gpio_cs_inverted);
+ if (err)
+ gpiod_put(chip->gpiod_cs);
}
return err;
struct driver_data *drv_data =
spi_controller_get_devdata(spi->controller);
uint tx_thres, tx_hi_thres, rx_thres;
+ int err;
switch (drv_data->ssp_type) {
case QUARK_X1000_SSP:
if (drv_data->ssp_type == CE4100_SSP)
return 0;
- return setup_cs(spi, chip, chip_info);
+ err = setup_cs(spi, chip, chip_info);
+ if (err)
+ kfree(chip);
+
+ return err;
}
static void cleanup(struct spi_device *spi)
static int sc18is602_check_transfer(struct spi_device *spi,
struct spi_transfer *t, int tlen)
{
- if (t && t->len + tlen > SC18IS602_BUFSIZ)
+ if (t && t->len + tlen > SC18IS602_BUFSIZ + 1)
return -EINVAL;
return 0;
return status;
}
+static size_t sc18is602_max_transfer_size(struct spi_device *spi)
+{
+ return SC18IS602_BUFSIZ;
+}
+
static int sc18is602_setup(struct spi_device *spi)
{
struct sc18is602 *hw = spi_master_get_devdata(spi->master);
master->bits_per_word_mask = SPI_BPW_MASK(8);
master->setup = sc18is602_setup;
master->transfer_one_message = sc18is602_transfer_one;
+ master->max_transfer_size = sc18is602_max_transfer_size;
+ master->max_message_size = sc18is602_max_transfer_size;
master->dev.of_node = np;
master->min_speed_hz = hw->freq / 128;
master->max_speed_hz = hw->freq / 4;
{ .compatible = "sprd,sc9860-spi", },
{ /* sentinel */ }
};
+MODULE_DEVICE_TABLE(of, sprd_spi_of_match);
static struct platform_driver sprd_spi_driver = {
.driver = {
int err = 0;
if (!op->data.nbytes)
- return stm32_qspi_wait_nobusy(qspi);
+ goto wait_nobusy;
if (readl_relaxed(qspi->io_base + QSPI_SR) & SR_TCF)
goto out;
out:
/* clear flags */
writel_relaxed(FCR_CTCF | FCR_CTEF, qspi->io_base + QSPI_FCR);
+wait_nobusy:
+ if (!err)
+ err = stm32_qspi_wait_nobusy(qspi);
return err;
}
}
/**
- * zynq_qspi_setup - Configure the QSPI controller
+ * zynq_qspi_setup_op - Configure the QSPI controller
* @spi: Pointer to the spi_device structure
*
* Sets the operational mode of QSPI controller for the next QSPI transfer, baud
struct zynq_qspi *xqspi = spi_controller_get_devdata(mem->spi->master);
int err = 0, i;
u8 *tmpbuf;
- u8 opcode = op->cmd.opcode;
dev_dbg(xqspi->dev, "cmd:%#x mode:%d.%d.%d.%d\n",
- opcode, op->cmd.buswidth, op->addr.buswidth,
+ op->cmd.opcode, op->cmd.buswidth, op->addr.buswidth,
op->dummy.buswidth, op->data.buswidth);
zynq_qspi_chipselect(mem->spi, true);
zynq_qspi_config_op(xqspi, mem->spi);
- if (op->cmd.nbytes) {
+ if (op->cmd.opcode) {
reinit_completion(&xqspi->data_completion);
- xqspi->txbuf = &opcode;
+ xqspi->txbuf = (u8 *)&op->cmd.opcode;
xqspi->rxbuf = NULL;
xqspi->tx_bytes = op->cmd.nbytes;
xqspi->rx_bytes = op->cmd.nbytes;
xqspi->irq = platform_get_irq(pdev, 0);
if (xqspi->irq <= 0) {
ret = -ENXIO;
- goto remove_master;
+ goto clk_dis_all;
}
ret = devm_request_irq(&pdev->dev, xqspi->irq, zynq_qspi_irq,
0, pdev->name, xqspi);
if (ret != 0) {
ret = -ENXIO;
dev_err(&pdev->dev, "request_irq failed\n");
- goto remove_master;
+ goto clk_dis_all;
}
ret = of_property_read_u32(np, "num-cs",
if (ret < 0) {
ctlr->num_chipselect = 1;
} else if (num_cs > ZYNQ_QSPI_MAX_NUM_CS) {
+ ret = -EINVAL;
dev_err(&pdev->dev, "only 2 chip selects are available\n");
- goto remove_master;
+ goto clk_dis_all;
} else {
ctlr->num_chipselect = num_cs;
}
{
struct spi_device *spi = to_spi_device(dev);
- /* spi controllers may cleanup for released devices */
- if (spi->controller->cleanup)
- spi->controller->cleanup(spi);
-
spi_controller_put(spi->controller);
kfree(spi->driver_override);
kfree(spi);
return 0;
}
+static void spi_cleanup(struct spi_device *spi)
+{
+ if (spi->controller->cleanup)
+ spi->controller->cleanup(spi);
+}
+
/**
* spi_add_device - Add spi_device allocated with spi_alloc_device
* @spi: spi_device to register
/* Device may be bound to an active driver when this returns */
status = device_add(&spi->dev);
- if (status < 0)
+ if (status < 0) {
dev_err(dev, "can't add %s, status %d\n",
dev_name(&spi->dev), status);
- else
+ spi_cleanup(spi);
+ } else {
dev_dbg(dev, "registered child %s\n", dev_name(&spi->dev));
+ }
done:
mutex_unlock(&spi_add_lock);
if (ACPI_COMPANION(&spi->dev))
acpi_device_clear_enumerated(ACPI_COMPANION(&spi->dev));
device_remove_software_node(&spi->dev);
- device_unregister(&spi->dev);
+ device_del(&spi->dev);
+ spi_cleanup(spi);
+ put_device(&spi->dev);
}
EXPORT_SYMBOL_GPL(spi_unregister_device);
if (spi->cs_gpiod || gpio_is_valid(spi->cs_gpio)) {
if (!(spi->mode & SPI_NO_CS)) {
- if (spi->cs_gpiod)
- /* polarity handled by gpiolib */
- gpiod_set_value_cansleep(spi->cs_gpiod, activate);
- else
+ if (spi->cs_gpiod) {
+ /*
+ * Historically ACPI has no means of the GPIO polarity and
+ * thus the SPISerialBus() resource defines it on the per-chip
+ * basis. In order to avoid a chain of negations, the GPIO
+ * polarity is considered being Active High. Even for the cases
+ * when _DSD() is involved (in the updated versions of ACPI)
+ * the GPIO CS polarity must be defined Active High to avoid
+ * ambiguity. That's why we use enable, that takes SPI_CS_HIGH
+ * into account.
+ */
+ if (has_acpi_companion(&spi->dev))
+ gpiod_set_value_cansleep(spi->cs_gpiod, !enable);
+ else
+ /* Polarity handled by GPIO library */
+ gpiod_set_value_cansleep(spi->cs_gpiod, activate);
+ } else {
/*
* invert the enable line, as active low is
* default for SPI.
*/
gpio_set_value_cansleep(spi->cs_gpio, !enable);
+ }
}
/* Some SPI masters need both GPIO CS & slave_select */
if ((spi->controller->flags & SPI_MASTER_GPIO_SS) &&
if (spi->controller->set_cs_timing &&
!(spi->cs_gpiod || gpio_is_valid(spi->cs_gpio))) {
+ mutex_lock(&spi->controller->io_mutex);
+
if (spi->controller->auto_runtime_pm) {
status = pm_runtime_get_sync(parent);
if (status < 0) {
+ mutex_unlock(&spi->controller->io_mutex);
pm_runtime_put_noidle(parent);
dev_err(&spi->controller->dev, "Failed to power device: %d\n",
status);
hold, inactive);
pm_runtime_mark_last_busy(parent);
pm_runtime_put_autosuspend(parent);
- return status;
} else {
- return spi->controller->set_cs_timing(spi, setup, hold,
+ status = spi->controller->set_cs_timing(spi, setup, hold,
inactive);
}
+
+ mutex_unlock(&spi->controller->io_mutex);
+ return status;
}
if ((setup && setup->unit == SPI_DELAY_UNIT_SCK) ||
chip->ngpio = 16;
/* There is just one SoC in one device and its GPIO addresses should be
* deterministic to address them more easily. The other buses could get
- * a random base number. */
+ * a random base number.
+ */
if (bus->bustype == SSB_BUSTYPE_SSB)
chip->base = 0;
else
chip->ngpio = 5;
/* There is just one SoC in one device and its GPIO addresses should be
* deterministic to address them more easily. The other buses could get
- * a random base number. */
+ * a random base number.
+ */
if (bus->bustype == SSB_BUSTYPE_SSB)
chip->base = 0;
else
#include <asm/paccess.h>
/* Probe a 32bit value on the bus and catch bus exceptions.
* Returns nonzero on a bus exception.
- * This is MIPS specific */
+ * This is MIPS specific
+ */
#define mips_busprobe32(val, addr) get_dbe((val), ((u32 *)(addr)))
/* Assume one-hot slot wiring */
};
/* This function is called when doing a pci_enable_device().
- * We must first check if the device is a device on the PCI-core bridge. */
+ * We must first check if the device is a device on the PCI-core bridge.
+ */
int ssb_pcicore_plat_dev_init(struct pci_dev *d)
{
if (d->bus->ops != &ssb_pcicore_pciops) {
/* Ok, ready to run, register it to the system.
* The following needs change, if we want to port hostmode
- * to non-MIPS platform. */
+ * to non-MIPS platform.
+ */
ssb_pcicore_controller.io_map_base = (unsigned long)ioremap(SSB_PCI_MEM, 0x04000000);
set_io_port_base(ssb_pcicore_controller.io_map_base);
/* Give some time to the PCI controller to configure itself with the new
- * values. Not waiting at this point causes crashes of the machine. */
+ * values. Not waiting at this point causes crashes of the machine.
+ */
mdelay(10);
register_pci_controller(&ssb_pcicore_controller);
}
return 0;
/* The 200-pin BCM4712 package does not bond out PCI. Even when
- * PCI is bonded out, some boards may leave the pins floating. */
+ * PCI is bonded out, some boards may leave the pins floating.
+ */
if (bus->chip_id == 0x4712) {
if (bus->chip_package == SSB_CHIPPACK_BCM4712S)
return 0;
if (dev->bus->bustype != SSB_BUSTYPE_PCI) {
/* This SSB device is not on a PCI host-bus. So the IRQs are
* not routed through the PCI core.
- * So we must not enable routing through the PCI core. */
+ * So we must not enable routing through the PCI core.
+ */
goto out;
}
/* Software ID counter */
static unsigned int next_busnumber;
/* buses_mutes locks the two buslists and the next_busnumber.
- * Don't lock this directly, but use ssb_buses_[un]lock() below. */
+ * Don't lock this directly, but use ssb_buses_[un]lock() below.
+ */
static DEFINE_MUTEX(buses_mutex);
/* There are differences in the codeflow, if the bus is
* are not available early. This is a mechanism to delay
* these initializations to after early boot has finished.
* It's also used to avoid mutex locking, as that's not
- * available and needed early. */
+ * available and needed early.
+ */
static bool ssb_is_early_boot = 1;
static void ssb_buses_lock(void);
int err;
/* Reset HW state information in memory, so that HW is
- * completely reinitialized. */
+ * completely reinitialized.
+ */
bus->mapped_device = NULL;
#ifdef CONFIG_SSB_DRIVER_PCICORE
bus->pcicore.setup_done = 0;
int err;
err = ssb_gpio_unregister(bus);
- if (err == -EBUSY)
- pr_debug("Some GPIOs are still in use\n");
- else if (err)
+ if (err)
pr_debug("Can not unregister GPIO driver: %i\n", err);
ssb_buses_lock();
sdev = &(bus->devices[i]);
/* We don't register SSB-system devices to the kernel,
- * as the drivers for them are built into SSB. */
+ * as the drivers for them are built into SSB.
+ */
switch (sdev->id.coreid) {
case SSB_DEV_CHIPCOMMON:
case SSB_DEV_PCI:
if (err) {
pr_err("Could not register %s\n", dev_name(dev));
/* Set dev to NULL to not unregister
- * dev on error unwinding. */
+ * dev on error unwinding.
+ */
sdev->dev = NULL;
put_device(dev);
goto error;
ssb_bus_may_powerdown(bus);
/* Queue it for attach.
- * See the comment at the ssb_is_early_boot definition. */
+ * See the comment at the ssb_is_early_boot definition.
+ */
list_add_tail(&bus->list, &attach_queue);
if (!ssb_is_early_boot) {
/* This is not early boot, so we must attach the bus now */
* a machine check exception otherwise.
* Do this by reading the register back to commit the
* PCI write and delay an additional usec for the device
- * to react to the change. */
+ * to react to the change.
+ */
ssb_read32(dev, SSB_TMSLOW);
udelay(1);
}
EXPORT_SYMBOL(ssb_device_enable);
/* Wait for bitmask in a register to get set or cleared.
- * timeout is in units of ten-microseconds */
+ * timeout is in units of ten-microseconds
+ */
static int ssb_wait_bits(struct ssb_device *dev, u16 reg, u32 bitmask,
int timeout, int set)
{
/* On buses where more than one core may be working
* at a time, we must not powerdown stuff if there are
- * still cores that may want to run. */
+ * still cores that may want to run.
+ */
if (bus->bustype == SSB_BUSTYPE_SSB)
goto out;
if (err) {
pr_err("Broadcom 43xx PCI-SSB-bridge initialization failed\n");
/* don't fail SSB init because of this */
- err = 0;
}
err = ssb_host_pcmcia_init();
if (err) {
pr_err("PCMCIA host initialization failed\n");
/* don't fail SSB init because of this */
- err = 0;
}
err = ssb_gige_init();
if (err) {
}
/* ssb must be initialized after PCI but before the ssb drivers.
* That means we must use some initcall between subsys_initcall
- * and device_initcall. */
+ * and device_initcall.
+ */
fs_initcall(ssb_modinit);
static void __exit ssb_modexit(void)
#endif
};
-static ssize_t ssb_pci_attr_sprom_show(struct device *pcidev,
- struct device_attribute *attr,
- char *buf)
+static ssize_t ssb_sprom_show(struct device *pcidev,
+ struct device_attribute *attr,
+ char *buf)
{
struct pci_dev *pdev = container_of(pcidev, struct pci_dev, dev);
struct ssb_bus *bus;
return ssb_attr_sprom_show(bus, buf, sprom_do_read);
}
-static ssize_t ssb_pci_attr_sprom_store(struct device *pcidev,
- struct device_attribute *attr,
- const char *buf, size_t count)
+static ssize_t ssb_sprom_store(struct device *pcidev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct pci_dev *pdev = container_of(pcidev, struct pci_dev, dev);
struct ssb_bus *bus;
sprom_check_crc, sprom_do_write);
}
-static DEVICE_ATTR(ssb_sprom, 0600,
- ssb_pci_attr_sprom_show,
- ssb_pci_attr_sprom_store);
+static DEVICE_ATTR_ADMIN_RW(ssb_sprom);
void ssb_pci_exit(struct ssb_bus *bus)
{
return -ENODEV;
}
-static ssize_t ssb_pcmcia_attr_sprom_show(struct device *pcmciadev,
- struct device_attribute *attr,
- char *buf)
+static ssize_t ssb_sprom_show(struct device *pcmciadev,
+ struct device_attribute *attr,
+ char *buf)
{
struct pcmcia_device *pdev =
container_of(pcmciadev, struct pcmcia_device, dev);
ssb_pcmcia_sprom_read_all);
}
-static ssize_t ssb_pcmcia_attr_sprom_store(struct device *pcmciadev,
- struct device_attribute *attr,
- const char *buf, size_t count)
+static ssize_t ssb_sprom_store(struct device *pcmciadev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
{
struct pcmcia_device *pdev =
container_of(pcmciadev, struct pcmcia_device, dev);
ssb_pcmcia_sprom_write_all);
}
-static DEVICE_ATTR(ssb_sprom, 0600,
- ssb_pcmcia_attr_sprom_show,
- ssb_pcmcia_attr_sprom_store);
+static DEVICE_ATTR_ADMIN_RW(ssb_sprom);
static int ssb_pcmcia_cor_setup(struct ssb_bus *bus, u8 cor)
{
if (bus->nr_devices > ARRAY_SIZE(bus->devices)) {
pr_err("More than %d ssb cores found (%d)\n",
SSB_MAX_NR_CORES, bus->nr_devices);
+ err = -EINVAL;
goto err_unmap;
}
if (bus->bustype == SSB_BUSTYPE_SSB) {
sdio_claim_host(bus->host_sdio);
if (unlikely(ssb_sdio_switch_core(bus, dev))) {
error = -EIO;
- memset((void *)buffer, 0xff, count);
goto err_out;
}
offset |= bus->sdio_sbaddr & 0xffff;
struct nbu2ss_ep *ep,
int status)
{
- struct nbu2ss_req *req;
+ struct nbu2ss_req *req, *n;
/* Endpoint Disable */
_nbu2ss_epn_exit(udc, ep);
return 0;
/* called with irqs blocked */
- list_for_each_entry(req, &ep->queue, queue) {
+ list_for_each_entry_safe(req, n, &ep->queue, queue) {
_nbu2ss_ep_done(ep, req, status);
}
indio_dev->num_channels = ARRAY_SIZE(ad7746_channels);
else
indio_dev->num_channels = ARRAY_SIZE(ad7746_channels) - 2;
- indio_dev->num_channels = ARRAY_SIZE(ad7746_channels);
indio_dev->modes = INDIO_DIRECT_MODE;
if (pdata) {
mediatek,mcm;
resets = <&rstctrl 2>;
reset-names = "mcm";
+ interrupt-controller;
+ #interrupt-cells = <1>;
+ interrupt-parent = <&gic>;
+ interrupts = <GIC_SHARED 23 IRQ_TYPE_LEVEL_HIGH>;
ports {
#address-cells = <1>;
// SPDX-License-Identifier: GPL-2.0+
/*
- * Copyright (C) 2013, Lars-Peter Clausen <lars@metafoo.de>
* GDMA4740 DMAC support
*/
};
module_platform_driver(gdma_dma_driver);
-MODULE_AUTHOR("Lars-Peter Clausen <lars@metafoo.de>");
MODULE_DESCRIPTION("Ralink/MTK DMA driver");
MODULE_LICENSE("GPL v2");
struct mlme_priv *pmlmepriv = &padapter->mlmepriv;
struct security_priv *psecuritypriv = &(padapter->securitypriv);
struct sta_priv *pstapriv = &padapter->stapriv;
+ char *grpkey = padapter->securitypriv.dot118021XGrpKey[param->u.crypt.idx].skey;
+ char *txkey = padapter->securitypriv.dot118021XGrptxmickey[param->u.crypt.idx].skey;
+ char *rxkey = padapter->securitypriv.dot118021XGrprxmickey[param->u.crypt.idx].skey;
param->u.crypt.err = 0;
param->u.crypt.alg[IEEE_CRYPT_ALG_NAME_LEN - 1] = '\0';
{
if (strcmp(param->u.crypt.alg, "WEP") == 0)
{
- memcpy(psecuritypriv->dot118021XGrpKey[param->u.crypt.idx].skey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
+ memcpy(grpkey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
psecuritypriv->dot118021XGrpPrivacy = _WEP40_;
if (param->u.crypt.key_len == 13)
{
psecuritypriv->dot118021XGrpPrivacy = _TKIP_;
- memcpy(psecuritypriv->dot118021XGrpKey[param->u.crypt.idx].skey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
+ memcpy(grpkey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
/* DEBUG_ERR("set key length :param->u.crypt.key_len =%d\n", param->u.crypt.key_len); */
/* set mic key */
- memcpy(psecuritypriv->dot118021XGrptxmickey[param->u.crypt.idx].skey, &(param->u.crypt.key[16]), 8);
- memcpy(psecuritypriv->dot118021XGrprxmickey[param->u.crypt.idx].skey, &(param->u.crypt.key[24]), 8);
+ memcpy(txkey, &(param->u.crypt.key[16]), 8);
+ memcpy(rxkey, &(param->u.crypt.key[24]), 8);
psecuritypriv->busetkipkey = true;
{
psecuritypriv->dot118021XGrpPrivacy = _AES_;
- memcpy(psecuritypriv->dot118021XGrpKey[param->u.crypt.idx].skey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
+ memcpy(grpkey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
}
else
{
{
if (strcmp(param->u.crypt.alg, "WEP") == 0)
{
- memcpy(psecuritypriv->dot118021XGrpKey[param->u.crypt.idx].skey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
+ memcpy(grpkey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
psecuritypriv->dot118021XGrpPrivacy = _WEP40_;
if (param->u.crypt.key_len == 13)
{
psecuritypriv->dot118021XGrpPrivacy = _TKIP_;
- memcpy(psecuritypriv->dot118021XGrpKey[param->u.crypt.idx].skey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
+ memcpy(grpkey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
/* DEBUG_ERR("set key length :param->u.crypt.key_len =%d\n", param->u.crypt.key_len); */
/* set mic key */
- memcpy(psecuritypriv->dot118021XGrptxmickey[param->u.crypt.idx].skey, &(param->u.crypt.key[16]), 8);
- memcpy(psecuritypriv->dot118021XGrprxmickey[param->u.crypt.idx].skey, &(param->u.crypt.key[24]), 8);
+ memcpy(txkey, &(param->u.crypt.key[16]), 8);
+ memcpy(rxkey, &(param->u.crypt.key[24]), 8);
psecuritypriv->busetkipkey = true;
{
psecuritypriv->dot118021XGrpPrivacy = _AES_;
- memcpy(psecuritypriv->dot118021XGrpKey[param->u.crypt.idx].skey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
+ memcpy(grpkey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
}
else
{
struct net_device *ndev = padapter->pnetdev;
{
- struct station_info sinfo;
+ struct station_info sinfo = {};
u8 ie_offset;
if (GetFrameSubType(pmgmt_frame) == WIFI_ASSOCREQ)
ie_offset = _ASOCREQ_IE_OFFSET_;
mon_wdev->iftype = NL80211_IFTYPE_MONITOR;
mon_ndev->ieee80211_ptr = mon_wdev;
- ret = register_netdevice(mon_ndev);
+ ret = cfg80211_register_netdevice(mon_ndev);
if (ret) {
goto out;
}
adapter = rtw_netdev_priv(ndev);
pwdev_priv = adapter_wdev_data(adapter);
- unregister_netdevice(ndev);
+ cfg80211_unregister_netdevice(ndev);
if (ndev == pwdev_priv->pmon_ndev) {
pwdev_priv->pmon_ndev = NULL;
struct mlme_priv *pmlmepriv = &padapter->mlmepriv;
struct security_priv *psecuritypriv = &(padapter->securitypriv);
struct sta_priv *pstapriv = &padapter->stapriv;
+ char *txkey = padapter->securitypriv.dot118021XGrptxmickey[param->u.crypt.idx].skey;
+ char *rxkey = padapter->securitypriv.dot118021XGrprxmickey[param->u.crypt.idx].skey;
+ char *grpkey = psecuritypriv->dot118021XGrpKey[param->u.crypt.idx].skey;
param->u.crypt.err = 0;
param->u.crypt.alg[IEEE_CRYPT_ALG_NAME_LEN - 1] = '\0';
if (!psta && check_fwstate(pmlmepriv, WIFI_AP_STATE)) { /* group key */
if (param->u.crypt.set_tx == 1) {
if (strcmp(param->u.crypt.alg, "WEP") == 0) {
- memcpy(psecuritypriv->dot118021XGrpKey[param->u.crypt.idx].skey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
+ memcpy(grpkey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
psecuritypriv->dot118021XGrpPrivacy = _WEP40_;
if (param->u.crypt.key_len == 13)
} else if (strcmp(param->u.crypt.alg, "TKIP") == 0) {
psecuritypriv->dot118021XGrpPrivacy = _TKIP_;
- memcpy(psecuritypriv->dot118021XGrpKey[param->u.crypt.idx].skey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
+ memcpy(grpkey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
/* DEBUG_ERR("set key length :param->u.crypt.key_len =%d\n", param->u.crypt.key_len); */
/* set mic key */
- memcpy(psecuritypriv->dot118021XGrptxmickey[param->u.crypt.idx].skey, &(param->u.crypt.key[16]), 8);
+ memcpy(txkey, &(param->u.crypt.key[16]), 8);
memcpy(psecuritypriv->dot118021XGrprxmickey[param->u.crypt.idx].skey, &(param->u.crypt.key[24]), 8);
psecuritypriv->busetkipkey = true;
else if (strcmp(param->u.crypt.alg, "CCMP") == 0) {
psecuritypriv->dot118021XGrpPrivacy = _AES_;
- memcpy(psecuritypriv->dot118021XGrpKey[param->u.crypt.idx].skey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
+ memcpy(grpkey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
} else {
psecuritypriv->dot118021XGrpPrivacy = _NO_PRIVACY_;
}
} else { /* group key??? */
if (strcmp(param->u.crypt.alg, "WEP") == 0) {
- memcpy(psecuritypriv->dot118021XGrpKey[param->u.crypt.idx].skey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
+ memcpy(grpkey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
psecuritypriv->dot118021XGrpPrivacy = _WEP40_;
if (param->u.crypt.key_len == 13)
} else if (strcmp(param->u.crypt.alg, "TKIP") == 0) {
psecuritypriv->dot118021XGrpPrivacy = _TKIP_;
- memcpy(psecuritypriv->dot118021XGrpKey[param->u.crypt.idx].skey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
+ memcpy(grpkey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
/* DEBUG_ERR("set key length :param->u.crypt.key_len =%d\n", param->u.crypt.key_len); */
/* set mic key */
- memcpy(psecuritypriv->dot118021XGrptxmickey[param->u.crypt.idx].skey, &(param->u.crypt.key[16]), 8);
- memcpy(psecuritypriv->dot118021XGrprxmickey[param->u.crypt.idx].skey, &(param->u.crypt.key[24]), 8);
+ memcpy(txkey, &(param->u.crypt.key[16]), 8);
+ memcpy(rxkey, &(param->u.crypt.key[24]), 8);
psecuritypriv->busetkipkey = true;
} else if (strcmp(param->u.crypt.alg, "CCMP") == 0) {
psecuritypriv->dot118021XGrpPrivacy = _AES_;
- memcpy(psecuritypriv->dot118021XGrpKey[param->u.crypt.idx].skey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
+ memcpy(grpkey, param->u.crypt.key, (param->u.crypt.key_len > 16 ? 16 : param->u.crypt.key_len));
} else {
psecuritypriv->dot118021XGrpPrivacy = _NO_PRIVACY_;
}
struct iblock_dev_plug *ib_dev_plug;
/*
- * Each se_device has a per cpu work this can be run from. Wwe
+ * Each se_device has a per cpu work this can be run from. We
* shouldn't have multiple threads on the same cpu calling this
* at the same time.
*/
- ib_dev_plug = &ib_dev->ibd_plug[smp_processor_id()];
+ ib_dev_plug = &ib_dev->ibd_plug[raw_smp_processor_id()];
if (test_and_set_bit(IBD_PLUGF_PLUGGED, &ib_dev_plug->flags))
return NULL;
cmd->orig_fe_lun = unpacked_lun;
if (!(cmd->se_cmd_flags & SCF_USE_CPUID))
- cmd->cpuid = smp_processor_id();
+ cmd->cpuid = raw_smp_processor_id();
cmd->state_active = false;
}
__releases(&cmd->t_state_lock)
__acquires(&cmd->t_state_lock)
{
-
- assert_spin_locked(&cmd->t_state_lock);
- WARN_ON_ONCE(!irqs_disabled());
+ lockdep_assert_held(&cmd->t_state_lock);
if (fabric_stop)
cmd->transport_state |= CMD_T_FABRIC_STOP;
dpi = dbi * udev->data_pages_per_blk;
/* Count the number of already allocated pages */
xas_set(&xas, dpi);
+ rcu_read_lock();
for (cnt = 0; xas_next(&xas) && cnt < page_cnt;)
cnt++;
+ rcu_read_unlock();
for (i = cnt; i < page_cnt; i++) {
/* try to get new page from the mm */
struct scatterlist *sg, unsigned int sg_nents,
struct iovec **iov, size_t data_len)
{
- XA_STATE(xas, &udev->data_pages, 0);
/* start value of dbi + 1 must not be a valid dbi */
int dbi = -2;
size_t page_remaining, cp_len;
- int page_cnt, page_inx;
+ int page_cnt, page_inx, dpi;
struct sg_mapping_iter sg_iter;
unsigned int sg_flags;
struct page *page;
if (page_cnt > udev->data_pages_per_blk)
page_cnt = udev->data_pages_per_blk;
- xas_set(&xas, dbi * udev->data_pages_per_blk);
- for (page_inx = 0; page_inx < page_cnt && data_len; page_inx++) {
- page = xas_next(&xas);
+ dpi = dbi * udev->data_pages_per_blk;
+ for (page_inx = 0; page_inx < page_cnt && data_len;
+ page_inx++, dpi++) {
+ page = xa_load(&udev->data_pages, dpi);
if (direction == TCMU_DATA_AREA_TO_SG)
flush_dcache_page(page);
#define TEEC_SUCCESS 0x00000000
#define TEEC_ERROR_GENERIC 0xFFFF0000
#define TEEC_ERROR_BAD_PARAMETERS 0xFFFF0006
+#define TEEC_ERROR_OUT_OF_MEMORY 0xFFFF000C
#define TEEC_ERROR_COMMUNICATION 0xFFFF000E
#define TEEC_ORIGIN_COMMS 0x00000002
u32 buf_id;
};
+/**
+ * struct amdtee_ta_data - Keeps track of all TAs loaded in AMD Secure
+ * Processor
+ * @ta_handle: Handle to TA loaded in TEE
+ * @refcount: Reference count for the loaded TA
+ */
+struct amdtee_ta_data {
+ struct list_head list_node;
+ u32 ta_handle;
+ u32 refcount;
+};
+
#define LOWER_TWO_BYTE_MASK 0x0000FFFF
/**
return ret;
}
+static DEFINE_MUTEX(ta_refcount_mutex);
+static struct list_head ta_list = LIST_HEAD_INIT(ta_list);
+
+static u32 get_ta_refcount(u32 ta_handle)
+{
+ struct amdtee_ta_data *ta_data;
+ u32 count = 0;
+
+ /* Caller must hold a mutex */
+ list_for_each_entry(ta_data, &ta_list, list_node)
+ if (ta_data->ta_handle == ta_handle)
+ return ++ta_data->refcount;
+
+ ta_data = kzalloc(sizeof(*ta_data), GFP_KERNEL);
+ if (ta_data) {
+ ta_data->ta_handle = ta_handle;
+ ta_data->refcount = 1;
+ count = ta_data->refcount;
+ list_add(&ta_data->list_node, &ta_list);
+ }
+
+ return count;
+}
+
+static u32 put_ta_refcount(u32 ta_handle)
+{
+ struct amdtee_ta_data *ta_data;
+ u32 count = 0;
+
+ /* Caller must hold a mutex */
+ list_for_each_entry(ta_data, &ta_list, list_node)
+ if (ta_data->ta_handle == ta_handle) {
+ count = --ta_data->refcount;
+ if (count == 0) {
+ list_del(&ta_data->list_node);
+ kfree(ta_data);
+ break;
+ }
+ }
+
+ return count;
+}
+
int handle_unload_ta(u32 ta_handle)
{
struct tee_cmd_unload_ta cmd = {0};
- u32 status;
+ u32 status, count;
int ret;
if (!ta_handle)
return -EINVAL;
+ mutex_lock(&ta_refcount_mutex);
+
+ count = put_ta_refcount(ta_handle);
+
+ if (count) {
+ pr_debug("unload ta: not unloading %u count %u\n",
+ ta_handle, count);
+ ret = -EBUSY;
+ goto unlock;
+ }
+
cmd.ta_handle = ta_handle;
ret = psp_tee_process_cmd(TEE_CMD_ID_UNLOAD_TA, (void *)&cmd,
if (!ret && status != 0) {
pr_err("unload ta: status = 0x%x\n", status);
ret = -EBUSY;
+ } else {
+ pr_debug("unloaded ta handle %u\n", ta_handle);
}
+unlock:
+ mutex_unlock(&ta_refcount_mutex);
return ret;
}
int handle_load_ta(void *data, u32 size, struct tee_ioctl_open_session_arg *arg)
{
- struct tee_cmd_load_ta cmd = {0};
+ struct tee_cmd_unload_ta unload_cmd = {};
+ struct tee_cmd_load_ta load_cmd = {};
phys_addr_t blob;
int ret;
return -EINVAL;
}
- cmd.hi_addr = upper_32_bits(blob);
- cmd.low_addr = lower_32_bits(blob);
- cmd.size = size;
+ load_cmd.hi_addr = upper_32_bits(blob);
+ load_cmd.low_addr = lower_32_bits(blob);
+ load_cmd.size = size;
- ret = psp_tee_process_cmd(TEE_CMD_ID_LOAD_TA, (void *)&cmd,
- sizeof(cmd), &arg->ret);
+ mutex_lock(&ta_refcount_mutex);
+
+ ret = psp_tee_process_cmd(TEE_CMD_ID_LOAD_TA, (void *)&load_cmd,
+ sizeof(load_cmd), &arg->ret);
if (ret) {
arg->ret_origin = TEEC_ORIGIN_COMMS;
arg->ret = TEEC_ERROR_COMMUNICATION;
- } else {
- set_session_id(cmd.ta_handle, 0, &arg->session);
+ } else if (arg->ret == TEEC_SUCCESS) {
+ ret = get_ta_refcount(load_cmd.ta_handle);
+ if (!ret) {
+ arg->ret_origin = TEEC_ORIGIN_COMMS;
+ arg->ret = TEEC_ERROR_OUT_OF_MEMORY;
+
+ /* Unload the TA on error */
+ unload_cmd.ta_handle = load_cmd.ta_handle;
+ psp_tee_process_cmd(TEE_CMD_ID_UNLOAD_TA,
+ (void *)&unload_cmd,
+ sizeof(unload_cmd), &ret);
+ } else {
+ set_session_id(load_cmd.ta_handle, 0, &arg->session);
+ }
}
+ mutex_unlock(&ta_refcount_mutex);
pr_debug("load TA: TA handle = 0x%x, RO = 0x%x, ret = 0x%x\n",
- cmd.ta_handle, arg->ret_origin, arg->ret);
+ load_cmd.ta_handle, arg->ret_origin, arg->ret);
return 0;
}
continue;
handle_close_session(sess->ta_handle, sess->session_info[i]);
+ handle_unload_ta(sess->ta_handle);
}
- /* Unload Trusted Application once all sessions are closed */
- handle_unload_ta(sess->ta_handle);
kfree(sess);
}
struct amdtee_session *sess = container_of(ref, struct amdtee_session,
refcount);
- /* Unload the TA from TEE */
- handle_unload_ta(sess->ta_handle);
mutex_lock(&session_list_mutex);
list_del(&sess->list_node);
mutex_unlock(&session_list_mutex);
{
struct amdtee_context_data *ctxdata = ctx->data;
struct amdtee_session *sess = NULL;
- u32 session_info;
+ u32 session_info, ta_handle;
size_t ta_size;
int rc, i;
void *ta;
if (arg->ret != TEEC_SUCCESS)
goto out;
+ ta_handle = get_ta_handle(arg->session);
+
mutex_lock(&session_list_mutex);
sess = alloc_session(ctxdata, arg->session);
mutex_unlock(&session_list_mutex);
if (!sess) {
+ handle_unload_ta(ta_handle);
rc = -ENOMEM;
goto out;
}
if (i >= TEE_NUM_SESSIONS) {
pr_err("reached maximum session count %d\n", TEE_NUM_SESSIONS);
+ handle_unload_ta(ta_handle);
kref_put(&sess->refcount, destroy_session);
rc = -ENOMEM;
goto out;
spin_lock(&sess->lock);
clear_bit(i, sess->sess_mask);
spin_unlock(&sess->lock);
+ handle_unload_ta(ta_handle);
kref_put(&sess->refcount, destroy_session);
goto out;
}
sess->session_info[i] = session_info;
- set_session_id(sess->ta_handle, i, &arg->session);
+ set_session_id(ta_handle, i, &arg->session);
out:
free_pages((u64)ta, get_order(ta_size));
return rc;
/* Close the session */
handle_close_session(ta_handle, session_info);
+ handle_unload_ta(ta_handle);
kref_put(&sess->refcount, destroy_session);
struct optee_msg_arg *msg_arg;
phys_addr_t msg_parg;
struct optee_session *sess = NULL;
+ uuid_t client_uuid;
/* +2 for the meta parameters added below */
shm = get_msg_arg(ctx, arg->num_params + 2, &msg_arg, &msg_parg);
memcpy(&msg_arg->params[0].u.value, arg->uuid, sizeof(arg->uuid));
msg_arg->params[1].u.value.c = arg->clnt_login;
- rc = tee_session_calc_client_uuid((uuid_t *)&msg_arg->params[1].u.value,
- arg->clnt_login, arg->clnt_uuid);
+ rc = tee_session_calc_client_uuid(&client_uuid, arg->clnt_login,
+ arg->clnt_uuid);
if (rc)
goto out;
+ export_uuid(msg_arg->params[1].u.octets, &client_uuid);
rc = optee_to_msg_param(msg_arg->params + 2, arg->num_params, param);
if (rc)
#include <linux/types.h>
/*
- * This file defines the OP-TEE message protocol used to communicate
+ * This file defines the OP-TEE message protocol (ABI) used to communicate
* with an instance of OP-TEE running in secure world.
*
* This file is divided into two sections.
* @tmem: parameter by temporary memory reference
* @rmem: parameter by registered memory reference
* @value: parameter by opaque value
+ * @octets: parameter by octet string
*
* @attr & OPTEE_MSG_ATTR_TYPE_MASK indicates if tmem, rmem or value is used in
- * the union. OPTEE_MSG_ATTR_TYPE_VALUE_* indicates value,
+ * the union. OPTEE_MSG_ATTR_TYPE_VALUE_* indicates value or octets,
* OPTEE_MSG_ATTR_TYPE_TMEM_* indicates @tmem and
* OPTEE_MSG_ATTR_TYPE_RMEM_* indicates @rmem,
* OPTEE_MSG_ATTR_TYPE_NONE indicates that none of the members are used.
struct optee_msg_param_tmem tmem;
struct optee_msg_param_rmem rmem;
struct optee_msg_param_value value;
+ u8 octets[24];
} u;
};
if (ACPI_FAILURE(status))
trip_cnt = 0;
else {
+ int i;
+
int34x_thermal_zone->aux_trips =
kcalloc(trip_cnt,
sizeof(*int34x_thermal_zone->aux_trips),
}
trip_mask = BIT(trip_cnt) - 1;
int34x_thermal_zone->aux_trip_nr = trip_cnt;
+ for (i = 0; i < trip_cnt; ++i)
+ int34x_thermal_zone->aux_trips[i] = THERMAL_TEMP_INVALID;
}
trip_cnt = int340x_thermal_read_trips(int34x_thermal_zone);
return atomic_read(&therm_throt_en);
}
+void __init therm_lvt_init(void)
+{
+ /*
+ * This function is only called on boot CPU. Save the init thermal
+ * LVT value on BSP and use that value to restore APs' thermal LVT
+ * entry BIOS programmed later
+ */
+ if (intel_thermal_supported(&boot_cpu_data))
+ lvtthmr_init = apic_read(APIC_LVTTHMR);
+}
+
void intel_init_thermal(struct cpuinfo_x86 *c)
{
unsigned int cpu = smp_processor_id();
if (!intel_thermal_supported(c))
return;
- /* On the BSP? */
- if (c == &boot_cpu_data)
- lvtthmr_init = apic_read(APIC_LVTTHMR);
-
/*
* First check if its enabled already, in which case there might
* be some SMM goo which handles it, so we can't even put a handler
if (thres_reg_value)
*temp = zonedev->tj_max - thres_reg_value * 1000;
else
- *temp = 0;
+ *temp = THERMAL_TEMP_INVALID;
pr_debug("sys_get_trip_temp %d\n", *temp);
return 0;
if (args.args_count != 1 || args.args[0] >= ADC5_MAX_CHANNEL) {
dev_err(dev, "%s: invalid ADC channel number %d\n", name, chan);
- return ret;
+ return -EINVAL;
}
channel->adc_channel = args.args[0];
}
/**
- * ti_bandgap_alert_init() - setup and initialize talert handling
+ * ti_bandgap_talert_init() - setup and initialize talert handling
* @bgp: pointer to struct ti_bandgap
* @pdev: pointer to device struct platform_device
*
void *buf, size_t size)
{
unsigned int retries = DMA_PORT_RETRIES;
- unsigned int offset;
-
- offset = address & 3;
- address = address & ~3;
do {
- u32 nbytes = min_t(u32, size, MAIL_DATA_DWORDS * 4);
+ unsigned int offset;
+ size_t nbytes;
int ret;
+ offset = address & 3;
+ nbytes = min_t(size_t, size + offset, MAIL_DATA_DWORDS * 4);
+
ret = dma_port_flash_read_block(dma, address, dma->buf,
ALIGN(nbytes, 4));
if (ret) {
return ret;
}
+ nbytes -= offset;
memcpy(buf, dma->buf + offset, nbytes);
size -= nbytes;
unsigned int retries = USB4_DATA_RETRIES;
unsigned int offset;
- offset = address & 3;
- address = address & ~3;
-
do {
- size_t nbytes = min_t(size_t, size, USB4_DATA_DWORDS * 4);
unsigned int dwaddress, dwords;
u8 data[USB4_DATA_DWORDS * 4];
+ size_t nbytes;
int ret;
+ offset = address & 3;
+ nbytes = min_t(size_t, size + offset, USB4_DATA_DWORDS * 4);
+
dwaddress = address / 4;
dwords = ALIGN(nbytes, 4) / 4;
return ret;
}
+ nbytes -= offset;
memcpy(buf, data + offset, nbytes);
size -= nbytes;
* Copyright (C) 2001 Russell King.
*/
+#include <linux/bits.h>
#include <linux/serial_8250.h>
#include <linux/serial_reg.h>
#include <linux/dmaengine.h>
unsigned int flags;
};
-#define UART_CAP_FIFO (1 << 8) /* UART has FIFO */
-#define UART_CAP_EFR (1 << 9) /* UART has EFR */
-#define UART_CAP_SLEEP (1 << 10) /* UART has IER sleep */
-#define UART_CAP_AFE (1 << 11) /* MCR-based hw flow control */
-#define UART_CAP_UUE (1 << 12) /* UART needs IER bit 6 set (Xscale) */
-#define UART_CAP_RTOIE (1 << 13) /* UART needs IER bit 4 set (Xscale, Tegra) */
-#define UART_CAP_HFIFO (1 << 14) /* UART has a "hidden" FIFO */
-#define UART_CAP_RPM (1 << 15) /* Runtime PM is active while idle */
-#define UART_CAP_IRDA (1 << 16) /* UART supports IrDA line discipline */
-#define UART_CAP_MINI (1 << 17) /* Mini UART on BCM283X family lacks:
+#define UART_CAP_FIFO BIT(8) /* UART has FIFO */
+#define UART_CAP_EFR BIT(9) /* UART has EFR */
+#define UART_CAP_SLEEP BIT(10) /* UART has IER sleep */
+#define UART_CAP_AFE BIT(11) /* MCR-based hw flow control */
+#define UART_CAP_UUE BIT(12) /* UART needs IER bit 6 set (Xscale) */
+#define UART_CAP_RTOIE BIT(13) /* UART needs IER bit 4 set (Xscale, Tegra) */
+#define UART_CAP_HFIFO BIT(14) /* UART has a "hidden" FIFO */
+#define UART_CAP_RPM BIT(15) /* Runtime PM is active while idle */
+#define UART_CAP_IRDA BIT(16) /* UART supports IrDA line discipline */
+#define UART_CAP_MINI BIT(17) /* Mini UART on BCM283X family lacks:
* STOP PARITY EPAR SPAR WLEN5 WLEN6
*/
-#define UART_BUG_QUOT (1 << 0) /* UART has buggy quot LSB */
-#define UART_BUG_TXEN (1 << 1) /* UART has buggy TX IIR status */
-#define UART_BUG_NOMSR (1 << 2) /* UART has buggy MSR status bits (Au1x00) */
-#define UART_BUG_THRE (1 << 3) /* UART has buggy THRE reassertion */
-#define UART_BUG_PARITY (1 << 4) /* UART mishandles parity if FIFO enabled */
+#define UART_BUG_QUOT BIT(0) /* UART has buggy quot LSB */
+#define UART_BUG_TXEN BIT(1) /* UART has buggy TX IIR status */
+#define UART_BUG_NOMSR BIT(2) /* UART has buggy MSR status bits (Au1x00) */
+#define UART_BUG_THRE BIT(3) /* UART has buggy THRE reassertion */
+#define UART_BUG_PARITY BIT(4) /* UART mishandles parity if FIFO enabled */
+#define UART_BUG_TXRACE BIT(5) /* UART Tx fails to set remote DR */
#ifdef CONFIG_SERIAL_8250_SHARE_IRQ
port.port.status = UPSTAT_SYNC_FIFO;
port.port.dev = &pdev->dev;
port.port.has_sysrq = IS_ENABLED(CONFIG_SERIAL_8250_CONSOLE);
+ port.bugs |= UART_BUG_TXRACE;
rc = sysfs_create_group(&vuart->dev->kobj, &aspeed_vuart_attr_group);
if (rc < 0)
{ "APMC0D08", 0},
{ "AMD0020", 0 },
{ "AMDI0020", 0 },
+ { "AMDI0022", 0 },
{ "BRCM2032", 0 },
{ "HISI0031", 0 },
{ },
{
struct exar8250 *priv = pci_get_drvdata(pcidev);
struct uart_8250_port *port = serial8250_get_port(priv->line[0]);
- struct platform_device *pdev = port->port.private_data;
+ struct platform_device *pdev;
+
+ pdev = port->port.private_data;
+ if (!pdev)
+ return;
device_remove_software_node(&pdev->dev);
platform_device_unregister(pdev);
int line[];
};
+#define PCI_DEVICE_ID_HPE_PCI_SERIAL 0x37e
+
static const struct pci_device_id pci_use_msi[] = {
{ PCI_DEVICE_SUB(PCI_VENDOR_ID_NETMOS, PCI_DEVICE_ID_NETMOS_9900,
0xA000, 0x1000) },
0xA000, 0x1000) },
{ PCI_DEVICE_SUB(PCI_VENDOR_ID_NETMOS, PCI_DEVICE_ID_NETMOS_9922,
0xA000, 0x1000) },
+ { PCI_DEVICE_SUB(PCI_VENDOR_ID_HP_3PAR, PCI_DEVICE_ID_HPE_PCI_SERIAL,
+ PCI_ANY_ID, PCI_ANY_ID) },
{ }
};
.setup = pci_hp_diva_setup,
},
/*
+ * HPE PCI serial device
+ */
+ {
+ .vendor = PCI_VENDOR_ID_HP_3PAR,
+ .device = PCI_DEVICE_ID_HPE_PCI_SERIAL,
+ .subvendor = PCI_ANY_ID,
+ .subdevice = PCI_ANY_ID,
+ .setup = pci_hp_diva_setup,
+ },
+ /*
* Intel
*/
{
uart.port.flags = UPF_SKIP_TEST | UPF_BOOT_AUTOCONF | UPF_SHARE_IRQ;
uart.port.uartclk = board->base_baud * 16;
- if (pci_match_id(pci_use_msi, dev)) {
- dev_dbg(&dev->dev, "Using MSI(-X) interrupts\n");
- pci_set_master(dev);
- rc = pci_alloc_irq_vectors(dev, 1, 1, PCI_IRQ_ALL_TYPES);
+ if (board->flags & FL_NOIRQ) {
+ uart.port.irq = 0;
} else {
- dev_dbg(&dev->dev, "Using legacy interrupts\n");
- rc = pci_alloc_irq_vectors(dev, 1, 1, PCI_IRQ_LEGACY);
- }
- if (rc < 0) {
- kfree(priv);
- priv = ERR_PTR(rc);
- goto err_deinit;
+ if (pci_match_id(pci_use_msi, dev)) {
+ dev_dbg(&dev->dev, "Using MSI(-X) interrupts\n");
+ pci_set_master(dev);
+ rc = pci_alloc_irq_vectors(dev, 1, 1, PCI_IRQ_ALL_TYPES);
+ } else {
+ dev_dbg(&dev->dev, "Using legacy interrupts\n");
+ rc = pci_alloc_irq_vectors(dev, 1, 1, PCI_IRQ_LEGACY);
+ }
+ if (rc < 0) {
+ kfree(priv);
+ priv = ERR_PTR(rc);
+ goto err_deinit;
+ }
+
+ uart.port.irq = pci_irq_vector(dev, 0);
}
- uart.port.irq = pci_irq_vector(dev, 0);
uart.port.dev = &dev->dev;
for (i = 0; i < nr_ports; i++) {
{ PCI_VENDOR_ID_HP, PCI_DEVICE_ID_HP_DIVA_AUX,
PCI_ANY_ID, PCI_ANY_ID, 0, 0,
pbn_b2_1_115200 },
+ /* HPE PCI serial device */
+ { PCI_VENDOR_ID_HP_3PAR, PCI_DEVICE_ID_HPE_PCI_SERIAL,
+ PCI_ANY_ID, PCI_ANY_ID, 0, 0,
+ pbn_b1_1_115200 },
{ PCI_VENDOR_ID_DCI, PCI_DEVICE_ID_DCI_PCCOM2,
PCI_ANY_ID, PCI_ANY_ID, 0, 0,
count = up->tx_loadsz;
do {
serial_out(up, UART_TX, xmit->buf[xmit->tail]);
+ if (up->bugs & UART_BUG_TXRACE) {
+ /*
+ * The Aspeed BMC virtual UARTs have a bug where data
+ * may get stuck in the BMC's Tx FIFO from bursts of
+ * writes on the APB interface.
+ *
+ * Delay back-to-back writes by a read cycle to avoid
+ * stalling the VUART. Read a register that won't have
+ * side-effects and discard the result.
+ */
+ serial_in(up, UART_SCR);
+ }
xmit->tail = (xmit->tail + 1) & (UART_XMIT_SIZE - 1);
port->icount.tx++;
if (uart_circ_empty(xmit))
#ifdef CONFIG_SPI_MASTER
ret = spi_register_driver(&max310x_spi_driver);
+ if (ret)
+ uart_unregister_driver(&max310x_uart);
#endif
return ret;
return -EINVAL;
}
- if (!match)
- return -ENODEV;
-
/* Assume that all UART ports have a DT alias or none has */
id = of_alias_get_id(pdev->dev.of_node, "serial");
if (!pdev->dev.of_node || id < 0)
void __iomem *bar0;
void __iomem *bar1;
spinlock_t card_lock;
- struct completion fw_loaded;
};
#define RP_ID(prod) PCI_VDEVICE(RP, (prod))
card->initialized_ports = 0;
}
-static void rp2_fw_cb(const struct firmware *fw, void *context)
+static int rp2_load_firmware(struct rp2_card *card, const struct firmware *fw)
{
- struct rp2_card *card = context;
resource_size_t phys_base;
- int i, rc = -ENOENT;
-
- if (!fw) {
- dev_err(&card->pdev->dev, "cannot find '%s' firmware image\n",
- RP2_FW_NAME);
- goto no_fw;
- }
+ int i, rc = 0;
phys_base = pci_resource_start(card->pdev, 1);
card->initialized_ports++;
}
- release_firmware(fw);
-no_fw:
- /*
- * rp2_fw_cb() is called from a workqueue long after rp2_probe()
- * has already returned success. So if something failed here,
- * we'll just leave the now-dormant device in place until somebody
- * unbinds it.
- */
- if (rc)
- dev_warn(&card->pdev->dev, "driver initialization failed\n");
-
- complete(&card->fw_loaded);
+ return rc;
}
static int rp2_probe(struct pci_dev *pdev,
const struct pci_device_id *id)
{
+ const struct firmware *fw;
struct rp2_card *card;
struct rp2_uart_port *ports;
void __iomem * const *bars;
return -ENOMEM;
pci_set_drvdata(pdev, card);
spin_lock_init(&card->card_lock);
- init_completion(&card->fw_loaded);
rc = pcim_enable_device(pdev);
if (rc)
return -ENOMEM;
card->ports = ports;
- rc = devm_request_irq(&pdev->dev, pdev->irq, rp2_uart_interrupt,
- IRQF_SHARED, DRV_NAME, card);
- if (rc)
+ rc = request_firmware(&fw, RP2_FW_NAME, &pdev->dev);
+ if (rc < 0) {
+ dev_err(&pdev->dev, "cannot find '%s' firmware image\n",
+ RP2_FW_NAME);
return rc;
+ }
- /*
- * Only catastrophic errors (e.g. ENOMEM) are reported here.
- * If the FW image is missing, we'll find out in rp2_fw_cb()
- * and print an error message.
- */
- rc = request_firmware_nowait(THIS_MODULE, 1, RP2_FW_NAME, &pdev->dev,
- GFP_KERNEL, card, rp2_fw_cb);
+ rc = rp2_load_firmware(card, fw);
+
+ release_firmware(fw);
+ if (rc < 0)
+ return rc;
+
+ rc = devm_request_irq(&pdev->dev, pdev->irq, rp2_uart_interrupt,
+ IRQF_SHARED, DRV_NAME, card);
if (rc)
return rc;
- dev_dbg(&pdev->dev, "waiting for firmware blob...\n");
return 0;
}
{
struct rp2_card *card = pci_get_drvdata(pdev);
- wait_for_completion(&card->fw_loaded);
rp2_remove_ports(card);
}
do {
lsr = tegra_uart_read(tup, UART_LSR);
- if ((lsr | UART_LSR_TEMT) && !(lsr & UART_LSR_DR))
+ if ((lsr & UART_LSR_TEMT) && !(lsr & UART_LSR_DR))
break;
udelay(1);
} while (--tmout);
goto check_and_exit;
}
- retval = security_locked_down(LOCKDOWN_TIOCSSERIAL);
- if (retval && (change_irq || change_port))
- goto exit;
+ if (change_irq || change_port) {
+ retval = security_locked_down(LOCKDOWN_TIOCSSERIAL);
+ if (retval)
+ goto exit;
+ }
/*
* Ask the low level driver to verify the settings.
{
unsigned int bits;
+ if (rx_trig >= port->fifosize)
+ rx_trig = port->fifosize - 1;
if (rx_trig < 1)
rx_trig = 1;
- if (rx_trig >= port->fifosize)
- rx_trig = port->fifosize;
/* HSCIF can be set to an arbitrary level. */
if (sci_getreg(port, HSRTRGR)->size) {
/* Resizes the resolution of the display adapater */
int err = 0;
- if (vc->vc_mode != KD_GRAPHICS && vc->vc_sw->con_resize)
+ if (vc->vc_sw->con_resize)
err = vc->vc_sw->con_resize(vc, width, height, user);
return err;
if (copy_from_user(&v, cs, sizeof(struct vt_consize)))
return -EFAULT;
- if (v.v_vlin)
- pr_info_once("\"struct vt_consize\"->v_vlin is ignored. Please report if you need this.\n");
- if (v.v_clin)
- pr_info_once("\"struct vt_consize\"->v_clin is ignored. Please report if you need this.\n");
+ /* FIXME: Should check the copies properly */
+ if (!v.v_vlin)
+ v.v_vlin = vc->vc_scan_lines;
+
+ if (v.v_clin) {
+ int rows = v.v_vlin / v.v_clin;
+ if (v.v_rows != rows) {
+ if (v.v_rows) /* Parameters don't add up */
+ return -EINVAL;
+ v.v_rows = rows;
+ }
+ }
+
+ if (v.v_vcol && v.v_ccol) {
+ int cols = v.v_vcol / v.v_ccol;
+ if (v.v_cols != cols) {
+ if (v.v_cols)
+ return -EINVAL;
+ v.v_cols = cols;
+ }
+ }
+
+ if (v.v_clin > 32)
+ return -EINVAL;
- console_lock();
for (i = 0; i < MAX_NR_CONSOLES; i++) {
- vc = vc_cons[i].d;
+ struct vc_data *vcp;
- if (vc) {
- vc->vc_resize_user = 1;
- vc_resize(vc, v.v_cols, v.v_rows);
+ if (!vc_cons[i].d)
+ continue;
+ console_lock();
+ vcp = vc_cons[i].d;
+ if (vcp) {
+ int ret;
+ int save_scan_lines = vcp->vc_scan_lines;
+ int save_cell_height = vcp->vc_cell_height;
+
+ if (v.v_vlin)
+ vcp->vc_scan_lines = v.v_vlin;
+ if (v.v_clin)
+ vcp->vc_cell_height = v.v_clin;
+ vcp->vc_resize_user = 1;
+ ret = vc_resize(vcp, v.v_cols, v.v_rows);
+ if (ret) {
+ vcp->vc_scan_lines = save_scan_lines;
+ vcp->vc_cell_height = save_cell_height;
+ console_unlock();
+ return ret;
+ }
}
+ console_unlock();
}
- console_unlock();
return 0;
}
pdata->recv_buf = vzalloc(RECV_BUFFER_SIZE);
if (pdata->recv_buf == NULL) {
ret = -ENOMEM;
- goto fail_close;
+ goto fail_free_ring;
}
ret = vmbus_establish_gpadl(channel, pdata->recv_buf,
RECV_BUFFER_SIZE, &pdata->recv_gpadl);
- if (ret)
+ if (ret) {
+ vfree(pdata->recv_buf);
goto fail_close;
+ }
/* put Global Physical Address Label in name */
snprintf(pdata->recv_name, sizeof(pdata->recv_name),
ret = vmbus_establish_gpadl(channel, pdata->send_buf,
SEND_BUFFER_SIZE, &pdata->send_gpadl);
- if (ret)
+ if (ret) {
+ vfree(pdata->send_buf);
goto fail_close;
+ }
snprintf(pdata->send_name, sizeof(pdata->send_name),
"send:%u", pdata->send_gpadl);
fail_close:
hv_uio_cleanup(dev, pdata);
+fail_free_ring:
+ vmbus_free_ring(dev->channel);
return ret;
}
}
if (pdev->irq && !pci_intx_mask_supported(pdev))
- return -ENOMEM;
+ return -ENODEV;
gdev = devm_kzalloc(&pdev->dev, sizeof(struct uio_pci_generic_dev), GFP_KERNEL);
if (!gdev)
else
mask = BIT(priv_ep->num);
- if (priv_ep->type != USB_ENDPOINT_XFER_ISOC) {
+ if (priv_ep->type != USB_ENDPOINT_XFER_ISOC && !priv_ep->dir) {
cdns3_set_register_bit(®s->tdl_from_trb, mask);
cdns3_set_register_bit(®s->tdl_beh, mask);
cdns3_set_register_bit(®s->tdl_beh2, mask);
case USB_ENDPOINT_XFER_INT:
ep_cfg = EP_CFG_EPTYPE(USB_ENDPOINT_XFER_INT);
- if ((priv_dev->dev_ver == DEV_VER_V2 && !priv_ep->dir) ||
- priv_dev->dev_ver > DEV_VER_V2)
+ if (priv_dev->dev_ver >= DEV_VER_V2 && !priv_ep->dir)
ep_cfg |= EP_CFG_TDL_CHK;
break;
case USB_ENDPOINT_XFER_BULK:
ep_cfg = EP_CFG_EPTYPE(USB_ENDPOINT_XFER_BULK);
- if ((priv_dev->dev_ver == DEV_VER_V2 && !priv_ep->dir) ||
- priv_dev->dev_ver > DEV_VER_V2)
+ if (priv_dev->dev_ver >= DEV_VER_V2 && !priv_ep->dir)
ep_cfg |= EP_CFG_TDL_CHK;
break;
default:
pm_runtime_get_sync(cdns->dev);
ret = cdns3_gadget_start(cdns);
- if (ret)
+ if (ret) {
+ pm_runtime_put_sync(cdns->dev);
return ret;
+ }
/*
* Because interrupt line can be shared with other components in
int cdnsp_ep_dequeue(struct cdnsp_ep *pep, struct cdnsp_request *preq)
{
struct cdnsp_device *pdev = pep->pdev;
- int ret;
+ int ret_stop = 0;
+ int ret_rem;
trace_cdnsp_request_dequeue(preq);
- if (GET_EP_CTX_STATE(pep->out_ctx) == EP_STATE_RUNNING) {
- ret = cdnsp_cmd_stop_ep(pdev, pep);
- if (ret)
- return ret;
- }
+ if (GET_EP_CTX_STATE(pep->out_ctx) == EP_STATE_RUNNING)
+ ret_stop = cdnsp_cmd_stop_ep(pdev, pep);
+
+ ret_rem = cdnsp_remove_request(pdev, preq, pep);
- return cdnsp_remove_request(pdev, preq, pep);
+ return ret_rem ? ret_rem : ret_stop;
}
static void cdnsp_zero_in_ctx(struct cdnsp_device *pdev)
{
struct cdnsp_device *pdev = (struct cdnsp_device *)data;
union cdnsp_trb *event_ring_deq;
+ unsigned long flags;
int counter = 0;
- spin_lock(&pdev->lock);
+ spin_lock_irqsave(&pdev->lock, flags);
if (pdev->cdnsp_state & (CDNSP_STATE_HALTED | CDNSP_STATE_DYING)) {
cdnsp_died(pdev);
- spin_unlock(&pdev->lock);
+ spin_unlock_irqrestore(&pdev->lock, flags);
return IRQ_HANDLED;
}
cdnsp_update_erst_dequeue(pdev, event_ring_deq, 1);
- spin_unlock(&pdev->lock);
+ spin_unlock_irqrestore(&pdev->lock, flags);
return IRQ_HANDLED;
}
ci->gadget.name = ci->platdata->name;
ci->gadget.otg_caps = otg_caps;
ci->gadget.sg_supported = 1;
+ ci->gadget.irq = ci->irq;
if (ci->platdata->flags & CI_HDRC_REQUIRES_ALIGNED_DMA)
ci->gadget.quirk_avoids_skb_reserve = 1;
int val;
unsigned long flags;
+ /* Clear VDATSRCENB0 to disable VDP_SRC and IDM_SNK required by BC 1.2 spec */
+ spin_lock_irqsave(&usbmisc->lock, flags);
+ val = readl(usbmisc->base + MX7D_USB_OTG_PHY_CFG2);
+ val &= ~MX7D_USB_OTG_PHY_CFG2_CHRG_VDATSRCENB0;
+ writel(val, usbmisc->base + MX7D_USB_OTG_PHY_CFG2);
+ spin_unlock_irqrestore(&usbmisc->lock, flags);
+
+ /* TVDMSRC_DIS */
+ msleep(20);
+
/* VDM_SRC is connected to D- and IDP_SINK is connected to D+ */
spin_lock_irqsave(&usbmisc->lock, flags);
val = readl(usbmisc->base + MX7D_USB_OTG_PHY_CFG2);
usbmisc->base + MX7D_USB_OTG_PHY_CFG2);
spin_unlock_irqrestore(&usbmisc->lock, flags);
- usleep_range(1000, 2000);
+ /* TVDMSRC_ON */
+ msleep(40);
/*
* Per BC 1.2, check voltage of D+:
usbmisc->base + MX7D_USB_OTG_PHY_CFG2);
spin_unlock_irqrestore(&usbmisc->lock, flags);
- usleep_range(1000, 2000);
+ /* TVDPSRC_ON */
+ msleep(40);
/* Check if D- is less than VDAT_REF to determine an SDP per BC 1.2 */
val = readl(usbmisc->base + MX7D_USB_OTG_PHY_STATUS);
}
-static void kill_urbs(struct wdm_device *desc)
+static void poison_urbs(struct wdm_device *desc)
{
/* the order here is essential */
- usb_kill_urb(desc->command);
- usb_kill_urb(desc->validity);
- usb_kill_urb(desc->response);
+ usb_poison_urb(desc->command);
+ usb_poison_urb(desc->validity);
+ usb_poison_urb(desc->response);
+}
+
+static void unpoison_urbs(struct wdm_device *desc)
+{
+ /*
+ * the order here is not essential
+ * it is symmetrical just to be nice
+ */
+ usb_unpoison_urb(desc->response);
+ usb_unpoison_urb(desc->validity);
+ usb_unpoison_urb(desc->command);
}
static void free_urbs(struct wdm_device *desc)
if (!desc->count) {
if (!test_bit(WDM_DISCONNECTING, &desc->flags)) {
dev_dbg(&desc->intf->dev, "wdm_release: cleanup\n");
- kill_urbs(desc);
+ poison_urbs(desc);
spin_lock_irq(&desc->iuspin);
desc->resp_count = 0;
spin_unlock_irq(&desc->iuspin);
desc->manage_power(desc->intf, 0);
+ unpoison_urbs(desc);
} else {
/* must avoid dev_printk here as desc->intf is invalid */
pr_debug(KBUILD_MODNAME " %s: device gone - cleaning up\n", __func__);
struct wdm_device *desc = wwan_port_get_drvdata(port);
/* Stop all transfers and disable WWAN mode */
- kill_urbs(desc);
+ poison_urbs(desc);
desc->manage_power(desc->intf, 0);
clear_bit(WDM_READ, &desc->flags);
clear_bit(WDM_WWAN_IN_USE, &desc->flags);
+ unpoison_urbs(desc);
}
static void wdm_wwan_port_tx_complete(struct urb *urb)
wake_up_all(&desc->wait);
mutex_lock(&desc->rlock);
mutex_lock(&desc->wlock);
+ poison_urbs(desc);
cancel_work_sync(&desc->rxwork);
cancel_work_sync(&desc->service_outs_intr);
- kill_urbs(desc);
mutex_unlock(&desc->wlock);
mutex_unlock(&desc->rlock);
set_bit(WDM_SUSPENDING, &desc->flags);
spin_unlock_irq(&desc->iuspin);
/* callback submits work - order is essential */
- kill_urbs(desc);
+ poison_urbs(desc);
cancel_work_sync(&desc->rxwork);
cancel_work_sync(&desc->service_outs_intr);
+ unpoison_urbs(desc);
}
if (!PMSG_IS_AUTO(message)) {
mutex_unlock(&desc->wlock);
wake_up_all(&desc->wait);
mutex_lock(&desc->rlock);
mutex_lock(&desc->wlock);
- kill_urbs(desc);
+ poison_urbs(desc);
cancel_work_sync(&desc->rxwork);
cancel_work_sync(&desc->service_outs_intr);
return 0;
struct wdm_device *desc = wdm_find_device(intf);
int rv;
+ unpoison_urbs(desc);
clear_bit(WDM_OVERFLOW, &desc->flags);
clear_bit(WDM_RESETTING, &desc->flags);
rv = recover_from_urb_loss(desc);
ret = usbfs_increase_memory_usage(len1 + sizeof(struct urb));
if (ret)
return ret;
- tbuf = kmalloc(len1, GFP_KERNEL);
+
+ /*
+ * len1 can be almost arbitrarily large. Don't WARN if it's
+ * too big, just fail the request.
+ */
+ tbuf = kmalloc(len1, GFP_KERNEL | __GFP_NOWARN);
if (!tbuf) {
ret = -ENOMEM;
goto done;
if (num_sgs) {
as->urb->sg = kmalloc_array(num_sgs,
sizeof(struct scatterlist),
- GFP_KERNEL);
+ GFP_KERNEL | __GFP_NOWARN);
if (!as->urb->sg) {
ret = -ENOMEM;
goto error;
(uurb_start - as->usbm->vm_start);
} else {
as->urb->transfer_buffer = kmalloc(uurb->buffer_length,
- GFP_KERNEL);
+ GFP_KERNEL | __GFP_NOWARN);
if (!as->urb->transfer_buffer) {
ret = -ENOMEM;
goto error;
#define USB_VENDOR_GENESYS_LOGIC 0x05e3
#define USB_VENDOR_SMSC 0x0424
#define USB_PRODUCT_USB5534B 0x5534
+#define USB_VENDOR_CYPRESS 0x04b4
+#define USB_PRODUCT_CY7C65632 0x6570
#define HUB_QUIRK_CHECK_PORT_AUTOSUSPEND 0x01
#define HUB_QUIRK_DISABLE_AUTOSUSPEND 0x02
* sequence.
*/
status = hub_port_status(hub, port1, &portstatus, &portchange);
-
- /* TRSMRCY = 10 msec */
- msleep(10);
}
SuspendCleared:
usb_clear_port_feature(hub->hdev, port1,
USB_PORT_FEAT_C_SUSPEND);
}
+
+ /* TRSMRCY = 10 msec */
+ msleep(10);
}
if (udev->persist_enabled)
.bInterfaceClass = USB_CLASS_HUB,
.driver_info = HUB_QUIRK_DISABLE_AUTOSUSPEND},
{ .match_flags = USB_DEVICE_ID_MATCH_VENDOR
+ | USB_DEVICE_ID_MATCH_PRODUCT,
+ .idVendor = USB_VENDOR_CYPRESS,
+ .idProduct = USB_PRODUCT_CY7C65632,
+ .driver_info = HUB_QUIRK_DISABLE_AUTOSUSPEND},
+ { .match_flags = USB_DEVICE_ID_MATCH_VENDOR
| USB_DEVICE_ID_MATCH_INT_CLASS,
.idVendor = USB_VENDOR_GENESYS_LOGIC,
.bInterfaceClass = USB_CLASS_HUB,
* @debugfs: File entry for debugfs file for this endpoint.
* @dir_in: Set to true if this endpoint is of the IN direction, which
* means that it is sending data to the Host.
+ * @map_dir: Set to the value of dir_in when the DMA buffer is mapped.
* @index: The index for the endpoint registers.
* @mc: Multi Count - number of transactions per microframe
* @interval: Interval for periodic endpoints, in frames or microframes.
unsigned short fifo_index;
unsigned char dir_in;
+ unsigned char map_dir;
unsigned char index;
unsigned char mc;
u16 interval;
{
struct usb_request *req = &hs_req->req;
- usb_gadget_unmap_request(&hsotg->gadget, req, hs_ep->dir_in);
+ usb_gadget_unmap_request(&hsotg->gadget, req, hs_ep->map_dir);
}
/*
{
int ret;
+ hs_ep->map_dir = hs_ep->dir_in;
ret = usb_gadget_map_request(&hsotg->gadget, req, hs_ep->dir_in);
if (ret)
goto dma_error;
};
module_platform_driver(dwc2_platform_driver);
-
-MODULE_DESCRIPTION("DESIGNWARE HS OTG Platform Glue");
-MODULE_AUTHOR("Matthijs Kooijman <matthijs@stdin.nl>");
-MODULE_LICENSE("Dual BSD/GPL");
pm_runtime_get_sync(&pdev->dev);
- dwc3_debugfs_exit(dwc);
dwc3_core_exit_mode(dwc);
+ dwc3_debugfs_exit(dwc);
dwc3_core_exit(dwc);
dwc3_ulpi_exit(dwc);
return 0;
}
-static void dwc3_shutdown(struct platform_device *pdev)
-{
- dwc3_remove(pdev);
-}
-
#ifdef CONFIG_PM
static int dwc3_core_init_for_resume(struct dwc3 *dwc)
{
static struct platform_driver dwc3_driver = {
.probe = dwc3_probe,
.remove = dwc3_remove,
- .shutdown = dwc3_shutdown,
.driver = {
.name = "dwc3",
.of_match_table = of_match_ptr(of_dwc3_match),
#define DWC3_DEVICE_EVENT_LINK_STATUS_CHANGE 3
#define DWC3_DEVICE_EVENT_WAKEUP 4
#define DWC3_DEVICE_EVENT_HIBER_REQ 5
-#define DWC3_DEVICE_EVENT_EOPF 6
+#define DWC3_DEVICE_EVENT_SUSPEND 6
#define DWC3_DEVICE_EVENT_SOF 7
#define DWC3_DEVICE_EVENT_ERRATIC_ERROR 9
#define DWC3_DEVICE_EVENT_CMD_CMPL 10
#define DWC3_DEVTEN_CMDCMPLTEN BIT(10)
#define DWC3_DEVTEN_ERRTICERREN BIT(9)
#define DWC3_DEVTEN_SOFEN BIT(7)
-#define DWC3_DEVTEN_EOPFEN BIT(6)
+#define DWC3_DEVTEN_U3L2L1SUSPEN BIT(6)
#define DWC3_DEVTEN_HIBERNATIONREQEVTEN BIT(5)
#define DWC3_DEVTEN_WKUPEVTEN BIT(4)
#define DWC3_DEVTEN_ULSTCNGEN BIT(3)
* @hwparams6: GHWPARAMS6
* @hwparams7: GHWPARAMS7
* @hwparams8: GHWPARAMS8
+ * @hwparams9: GHWPARAMS9
*/
struct dwc3_hwparams {
u32 hwparams0;
* 3 - ULStChng
* 4 - WkUpEvt
* 5 - Reserved
- * 6 - EOPF
+ * 6 - Suspend (EOPF on revisions 2.10a and prior)
* 7 - SOF
* 8 - Reserved
* 9 - ErrticErr
snprintf(str, size, "WakeUp [%s]",
dwc3_gadget_link_string(state));
break;
- case DWC3_DEVICE_EVENT_EOPF:
- snprintf(str, size, "End-Of-Frame [%s]",
+ case DWC3_DEVICE_EVENT_SUSPEND:
+ snprintf(str, size, "Suspend [%s]",
dwc3_gadget_link_string(state));
break;
case DWC3_DEVICE_EVENT_SOF:
return "Wake-Up";
case DWC3_DEVICE_EVENT_HIBER_REQ:
return "Hibernation";
- case DWC3_DEVICE_EVENT_EOPF:
- return "End of Periodic Frame";
+ case DWC3_DEVICE_EVENT_SUSPEND:
+ return "Suspend";
case DWC3_DEVICE_EVENT_SOF:
return "Start of Frame";
case DWC3_DEVICE_EVENT_ERRATIC_ERROR:
#ifdef CONFIG_DEBUG_FS
+extern void dwc3_debugfs_create_endpoint_dir(struct dwc3_ep *dep);
extern void dwc3_debugfs_init(struct dwc3 *d);
extern void dwc3_debugfs_exit(struct dwc3 *d);
#else
+static inline void dwc3_debugfs_create_endpoint_dir(struct dwc3_ep *dep)
+{ }
static inline void dwc3_debugfs_init(struct dwc3 *d)
{ }
static inline void dwc3_debugfs_exit(struct dwc3 *d)
}
}
-static void dwc3_debugfs_create_endpoint_dir(struct dwc3_ep *dep,
- struct dentry *parent)
+void dwc3_debugfs_create_endpoint_dir(struct dwc3_ep *dep)
{
struct dentry *dir;
- dir = debugfs_create_dir(dep->name, parent);
+ dir = debugfs_create_dir(dep->name, dep->dwc->root);
dwc3_debugfs_create_endpoint_files(dep, dir);
}
-static void dwc3_debugfs_create_endpoint_dirs(struct dwc3 *dwc,
- struct dentry *parent)
-{
- int i;
-
- for (i = 0; i < dwc->num_eps; i++) {
- struct dwc3_ep *dep = dwc->eps[i];
-
- if (!dep)
- continue;
-
- dwc3_debugfs_create_endpoint_dir(dep, parent);
- }
-}
-
void dwc3_debugfs_init(struct dwc3 *dwc)
{
struct dentry *root;
&dwc3_testmode_fops);
debugfs_create_file("link_state", 0644, root, dwc,
&dwc3_link_state_fops);
- dwc3_debugfs_create_endpoint_dirs(dwc, root);
}
}
if (err < 0)
goto disable_rpm;
- dwc3_np = of_get_child_by_name(node, "dwc3");
+ dwc3_np = of_get_compatible_child(node, "snps,dwc3");
if (!dwc3_np) {
+ err = -ENODEV;
dev_err(dev, "failed to find dwc3 core child\n");
goto disable_rpm;
}
return PTR_ERR(priv->usb_glue_regmap);
/* Create a regmap for each USB2 PHY control register set */
- for (i = 0; i < priv->usb2_ports; i++) {
+ for (i = 0; i < priv->drvdata->num_phys; i++) {
struct regmap_config u2p_regmap_config = {
.reg_bits = 8,
.val_bits = 32,
.max_register = U2P_R1,
};
+ if (!strstr(priv->drvdata->phy_names[i], "usb2"))
+ continue;
+
u2p_regmap_config.name = devm_kasprintf(priv->dev, GFP_KERNEL,
"u2p-%d", i);
if (!u2p_regmap_config.name)
ret = priv->drvdata->usb_init(priv);
if (ret)
- goto err_disable_clks;
+ goto err_disable_regulator;
/* Init PHYs */
for (i = 0 ; i < PHY_COUNT ; ++i) {
ret = phy_init(priv->phys[i]);
if (ret)
- goto err_disable_clks;
+ goto err_disable_regulator;
}
/* Set PHY Power */
for (i = 0 ; i < PHY_COUNT ; ++i)
phy_exit(priv->phys[i]);
+err_disable_regulator:
+ if (priv->vbus)
+ regulator_disable(priv->vbus);
+
err_disable_clks:
clk_bulk_disable_unprepare(priv->drvdata->num_clks,
priv->drvdata->clks);
if (extcon_get_state(edev, EXTCON_USB) == true)
dwc3_omap_set_mailbox(omap, OMAP_DWC3_VBUS_VALID);
+ else
+ dwc3_omap_set_mailbox(omap, OMAP_DWC3_VBUS_OFF);
+
if (extcon_get_state(edev, EXTCON_USB_HOST) == true)
dwc3_omap_set_mailbox(omap, OMAP_DWC3_ID_GROUND);
+ else
+ dwc3_omap_set_mailbox(omap, OMAP_DWC3_ID_FLOAT);
omap->edev = edev;
}
PROPERTY_ENTRY_STRING("linux,extcon-name", "mrfld_bcove_pwrsrc"),
PROPERTY_ENTRY_BOOL("snps,dis_u3_susphy_quirk"),
PROPERTY_ENTRY_BOOL("snps,dis_u2_susphy_quirk"),
+ PROPERTY_ENTRY_BOOL("snps,usb2-gadget-lpm-disable"),
PROPERTY_ENTRY_BOOL("linux,sysdev_is_parent"),
{}
};
epnum |= 1;
dep = dwc->eps[epnum];
+ if (dep == NULL)
+ return NULL;
+
if (dep->flags & DWC3_EP_ENABLED)
return dep;
req->start_sg = sg_next(s);
req->num_queued_sgs++;
+ req->num_pending_sgs--;
/*
* The number of pending SG entries may not correspond to the
* don't include unused SG entries.
*/
if (length == 0) {
- req->num_pending_sgs -= req->request.num_mapped_sgs - req->num_queued_sgs;
+ req->num_pending_sgs = 0;
break;
}
}
}
- return __dwc3_gadget_kick_transfer(dep);
+ __dwc3_gadget_kick_transfer(dep);
+
+ return 0;
}
static int dwc3_gadget_ep_queue(struct usb_ep *ep, struct usb_request *request,
}
/*
- * Synchronize any pending event handling before executing the controller
- * halt routine.
+ * Synchronize and disable any further event handling while controller
+ * is being enabled/disabled.
*/
- if (!is_on) {
- dwc3_gadget_disable_irq(dwc);
- synchronize_irq(dwc->irq_gadget);
- }
+ disable_irq(dwc->irq_gadget);
spin_lock_irqsave(&dwc->lock, flags);
ret = dwc3_gadget_run_stop(dwc, is_on, false);
spin_unlock_irqrestore(&dwc->lock, flags);
+ enable_irq(dwc->irq_gadget);
+
pm_runtime_put(dwc->dev);
return ret;
if (DWC3_VER_IS_PRIOR(DWC3, 250A))
reg |= DWC3_DEVTEN_ULSTCNGEN;
+ /* On 2.30a and above this bit enables U3/L2-L1 Suspend Events */
+ if (!DWC3_VER_IS_PRIOR(DWC3, 230A))
+ reg |= DWC3_DEVTEN_U3L2L1SUSPEN;
+
dwc3_writel(dwc->regs, DWC3_DEVTEN, reg);
}
INIT_LIST_HEAD(&dep->started_list);
INIT_LIST_HEAD(&dep->cancelled_list);
+ dwc3_debugfs_create_endpoint_dir(dep);
+
return 0;
}
list_del(&dep->endpoint.ep_list);
}
+ debugfs_remove_recursive(debugfs_lookup(dep->name, dwc->root));
kfree(dep);
}
}
struct dwc3_trb *trb = &dep->trb_pool[dep->trb_dequeue];
struct scatterlist *sg = req->sg;
struct scatterlist *s;
- unsigned int pending = req->num_pending_sgs;
+ unsigned int num_queued = req->num_queued_sgs;
unsigned int i;
int ret = 0;
- for_each_sg(sg, s, pending, i) {
+ for_each_sg(sg, s, num_queued, i) {
trb = &dep->trb_pool[dep->trb_dequeue];
req->sg = sg_next(s);
- req->num_pending_sgs--;
+ req->num_queued_sgs--;
ret = dwc3_gadget_ep_reclaim_completed_trb(dep, req,
trb, event, status, true);
static bool dwc3_gadget_ep_request_completed(struct dwc3_request *req)
{
- return req->num_pending_sgs == 0;
+ return req->num_pending_sgs == 0 && req->num_queued_sgs == 0;
}
static int dwc3_gadget_ep_cleanup_completed_request(struct dwc3_ep *dep,
{
int ret;
- if (req->num_pending_sgs)
+ if (req->request.num_mapped_sgs)
ret = dwc3_gadget_ep_reclaim_trb_sg(dep, req, event,
status);
else
case DWC3_DEVICE_EVENT_LINK_STATUS_CHANGE:
dwc3_gadget_linksts_change_interrupt(dwc, event->event_info);
break;
- case DWC3_DEVICE_EVENT_EOPF:
+ case DWC3_DEVICE_EVENT_SUSPEND:
/* It changed to be suspend event for version 2.30a and above */
if (!DWC3_VER_IS_PRIOR(DWC3, 230A)) {
/*
dwc3_gadget_free_endpoints(dwc);
err4:
usb_put_gadget(dwc->gadget);
+ dwc->gadget = NULL;
err3:
dma_free_coherent(dwc->sysdev, DWC3_BOUNCE_SIZE, dwc->bounce,
dwc->bounce_addr);
void dwc3_gadget_exit(struct dwc3 *dwc)
{
- usb_del_gadget_udc(dwc->gadget);
+ if (!dwc->gadget)
+ return;
+
+ usb_del_gadget(dwc->gadget);
dwc3_gadget_free_endpoints(dwc);
+ usb_put_gadget(dwc->gadget);
dma_free_coherent(dwc->sysdev, DWC3_BOUNCE_SIZE, dwc->bounce,
dwc->bounce_addr);
kfree(dwc->setup_buf);
{
struct usb_gadget *g = f->config->cdev->gadget;
+ /* super-speed-plus descriptor falls back to super-speed one,
+ * if such a descriptor was provided, thus avoiding a NULL
+ * pointer dereference if a 5gbps capable gadget is used with
+ * a 10gbps capable config (device port + cable + host port)
+ */
+ if (!ssp)
+ ssp = ss;
+
if (fs) {
f->fs_descriptors = usb_copy_descriptors(fs);
if (!f->fs_descriptors)
fs_ecm_notify_desc.bEndpointAddress;
status = usb_assign_descriptors(f, ecm_fs_function, ecm_hs_function,
- ecm_ss_function, NULL);
+ ecm_ss_function, ecm_ss_function);
if (status)
goto fail;
eem_ss_out_desc.bEndpointAddress = eem_fs_out_desc.bEndpointAddress;
status = usb_assign_descriptors(f, eem_fs_function, eem_hs_function,
- eem_ss_function, NULL);
+ eem_ss_function, eem_ss_function);
if (status)
goto fail;
skb2 = skb_clone(skb, GFP_ATOMIC);
if (unlikely(!skb2)) {
DBG(cdev, "unable to unframe EEM packet\n");
- continue;
+ goto next;
}
skb_trim(skb2, len - ETH_FCS_LEN);
GFP_ATOMIC);
if (unlikely(!skb3)) {
dev_kfree_skb_any(skb2);
- continue;
+ goto next;
}
dev_kfree_skb_any(skb2);
skb_queue_tail(list, skb3);
ffs->func = NULL;
}
+ /* Drain any pending AIO completions */
+ drain_workqueue(ffs->io_completion_wq);
+
if (!--opts->refcnt)
functionfs_unbind(ffs);
hidg_fs_out_ep_desc.bEndpointAddress;
status = usb_assign_descriptors(f, hidg_fs_descriptors,
- hidg_hs_descriptors, hidg_ss_descriptors, NULL);
+ hidg_hs_descriptors, hidg_ss_descriptors,
+ hidg_ss_descriptors);
if (status)
goto fail;
ss_loop_sink_desc.bEndpointAddress = fs_loop_sink_desc.bEndpointAddress;
ret = usb_assign_descriptors(f, fs_loopback_descs, hs_loopback_descs,
- ss_loopback_descs, NULL);
+ ss_loopback_descs, ss_loopback_descs);
if (ret)
return ret;
data[0] = cpu_to_le32(ncm_bitrate(cdev->gadget));
data[1] = data[0];
- DBG(cdev, "notify speed %d\n", ncm_bitrate(cdev->gadget));
+ DBG(cdev, "notify speed %u\n", ncm_bitrate(cdev->gadget));
ncm->notify_state = NCM_NOTIFY_CONNECT;
break;
}
ncm->ndp_dgram_count = 1;
/* Note: we skip opts->next_ndp_index */
- }
- /* Delay the timer. */
- hrtimer_start(&ncm->task_timer, TX_TIMEOUT_NSECS,
- HRTIMER_MODE_REL_SOFT);
+ /* Start the timer. */
+ hrtimer_start(&ncm->task_timer, TX_TIMEOUT_NSECS,
+ HRTIMER_MODE_REL_SOFT);
+ }
/* Add the datagram position entries */
ntb_ndp = skb_put_zero(ncm->skb_tx_ndp, dgram_idx_len);
ss_ep_out_desc.bEndpointAddress = fs_ep_out_desc.bEndpointAddress;
ret = usb_assign_descriptors(f, fs_printer_function,
- hs_printer_function, ss_printer_function, NULL);
+ hs_printer_function, ss_printer_function,
+ ss_printer_function);
if (ret)
return ret;
ss_notify_desc.bEndpointAddress = fs_notify_desc.bEndpointAddress;
status = usb_assign_descriptors(f, eth_fs_function, eth_hs_function,
- eth_ss_function, NULL);
+ eth_ss_function, eth_ss_function);
if (status)
goto fail;
gser_ss_out_desc.bEndpointAddress = gser_fs_out_desc.bEndpointAddress;
status = usb_assign_descriptors(f, gser_fs_function, gser_hs_function,
- gser_ss_function, NULL);
+ gser_ss_function, gser_ss_function);
if (status)
goto fail;
dev_dbg(&cdev->gadget->dev, "generic ttyGS%d: %s speed IN/%s OUT/%s\n",
ss_iso_sink_desc.bEndpointAddress = fs_iso_sink_desc.bEndpointAddress;
ret = usb_assign_descriptors(f, fs_source_sink_descs,
- hs_source_sink_descs, ss_source_sink_descs, NULL);
+ hs_source_sink_descs, ss_source_sink_descs,
+ ss_source_sink_descs);
if (ret)
return ret;
fs_subset_out_desc.bEndpointAddress;
status = usb_assign_descriptors(f, fs_eth_function, hs_eth_function,
- ss_eth_function, NULL);
+ ss_eth_function, ss_eth_function);
if (status)
goto fail;
uasp_fs_cmd_desc.bEndpointAddress = uasp_ss_cmd_desc.bEndpointAddress;
ret = usb_assign_descriptors(f, uasp_fs_function_desc,
- uasp_hs_function_desc, uasp_ss_function_desc, NULL);
+ uasp_hs_function_desc, uasp_ss_function_desc,
+ uasp_ss_function_desc);
if (ret)
goto ep_fail;
struct renesas_usb3_request *usb3_req)
{
struct renesas_usb3 *usb3 = usb3_ep_to_usb3(usb3_ep);
- struct renesas_usb3_request *usb3_req_first = usb3_get_request(usb3_ep);
+ struct renesas_usb3_request *usb3_req_first;
unsigned long flags;
int ret = -EAGAIN;
u32 enable_bits = 0;
spin_lock_irqsave(&usb3->lock, flags);
if (usb3_ep->halt || usb3_ep->started)
goto out;
- if (usb3_req != usb3_req_first)
+ usb3_req_first = __usb3_get_request(usb3_ep);
+ if (!usb3_req_first || usb3_req != usb3_req_first)
goto out;
if (usb3_pn_change(usb3, usb3_ep->num) < 0)
struct usb_hcd *hcd;
struct resource *res;
int irq;
- int retval = -ENODEV;
+ int retval;
struct fotg210_hcd *fotg210;
if (usb_disabled())
hcd = usb_create_hcd(&fotg210_fotg210_hc_driver, dev,
dev_name(dev));
if (!hcd) {
- dev_err(dev, "failed to create hcd with err %d\n", retval);
+ dev_err(dev, "failed to create hcd\n");
retval = -ENOMEM;
goto fail_create_hcd;
}
* Author: Sarah Sharp
* Some code borrowed from the Linux EHCI driver.
*/
-/* Up to 16 ms to halt an HC */
-#define XHCI_MAX_HALT_USEC (16*1000)
+
+/* HC should halt within 16 ms, but use 32 ms as some hosts take longer */
+#define XHCI_MAX_HALT_USEC (32 * 1000)
/* HC not running - set to 1 when run/stop bit is cleared. */
#define XHCI_STS_HALT (1<<0)
#define PCI_DEVICE_ID_INTEL_CML_XHCI 0xa3af
#define PCI_DEVICE_ID_INTEL_TIGER_LAKE_XHCI 0x9a13
#define PCI_DEVICE_ID_INTEL_MAPLE_RIDGE_XHCI 0x1138
+#define PCI_DEVICE_ID_INTEL_ALDER_LAKE_XHCI 0x461e
+#define PCI_DEVICE_ID_AMD_RENOIR_XHCI 0x1639
#define PCI_DEVICE_ID_AMD_PROMONTORYA_4 0x43b9
#define PCI_DEVICE_ID_AMD_PROMONTORYA_3 0x43ba
#define PCI_DEVICE_ID_AMD_PROMONTORYA_2 0x43bb
(pdev->device == 0x15e0 || pdev->device == 0x15e1))
xhci->quirks |= XHCI_SNPS_BROKEN_SUSPEND;
- if (pdev->vendor == PCI_VENDOR_ID_AMD && pdev->device == 0x15e5)
+ if (pdev->vendor == PCI_VENDOR_ID_AMD && pdev->device == 0x15e5) {
xhci->quirks |= XHCI_DISABLE_SPARSE;
+ xhci->quirks |= XHCI_RESET_ON_RESUME;
+ }
if (pdev->vendor == PCI_VENDOR_ID_AMD)
xhci->quirks |= XHCI_TRUST_TX_LENGTH;
(pdev->device == PCI_DEVICE_ID_AMD_PROMONTORYA_1)))
xhci->quirks |= XHCI_U2_DISABLE_WAKE;
+ if (pdev->vendor == PCI_VENDOR_ID_AMD &&
+ pdev->device == PCI_DEVICE_ID_AMD_RENOIR_XHCI)
+ xhci->quirks |= XHCI_BROKEN_D3COLD;
+
if (pdev->vendor == PCI_VENDOR_ID_INTEL) {
xhci->quirks |= XHCI_LPM_SUPPORT;
xhci->quirks |= XHCI_INTEL_HOST;
pdev->device == PCI_DEVICE_ID_INTEL_TITAN_RIDGE_DD_XHCI ||
pdev->device == PCI_DEVICE_ID_INTEL_ICE_LAKE_XHCI ||
pdev->device == PCI_DEVICE_ID_INTEL_TIGER_LAKE_XHCI ||
- pdev->device == PCI_DEVICE_ID_INTEL_MAPLE_RIDGE_XHCI))
+ pdev->device == PCI_DEVICE_ID_INTEL_MAPLE_RIDGE_XHCI ||
+ pdev->device == PCI_DEVICE_ID_INTEL_ALDER_LAKE_XHCI))
xhci->quirks |= XHCI_DEFAULT_PM_RUNTIME_ALLOW;
if (pdev->vendor == PCI_VENDOR_ID_ETRON &&
* Systems with the TI redriver that loses port status change events
* need to have the registers polled during D3, so avoid D3cold.
*/
- if (xhci->quirks & XHCI_COMP_MODE_QUIRK)
+ if (xhci->quirks & (XHCI_COMP_MODE_QUIRK | XHCI_BROKEN_D3COLD))
pci_d3cold_disable(pdev);
if (xhci->quirks & XHCI_PME_STUCK_QUIRK)
list_for_each_entry_safe(td, tmp_td, &ep->cancelled_td_list,
cancelled_td_list) {
- /*
- * Doesn't matter what we pass for status, since the core will
- * just overwrite it (because the URB has been unlinked).
- */
ring = xhci_urb_to_transfer_ring(ep->xhci, td->urb);
if (td->cancel_status == TD_CLEARED)
- xhci_td_cleanup(ep->xhci, td, ring, 0);
+ xhci_td_cleanup(ep->xhci, td, ring, td->status);
if (ep->xhci->xhc_state & XHCI_STATE_DYING)
return;
return ret;
}
-static void xhci_handle_halted_endpoint(struct xhci_hcd *xhci,
+static int xhci_handle_halted_endpoint(struct xhci_hcd *xhci,
struct xhci_virt_ep *ep, unsigned int stream_id,
struct xhci_td *td,
enum xhci_ep_reset_type reset_type)
* Device will be reset soon to recover the link so don't do anything
*/
if (ep->vdev->flags & VDEV_PORT_ERROR)
- return;
+ return -ENODEV;
/* add td to cancelled list and let reset ep handler take care of it */
if (reset_type == EP_HARD_RESET) {
if (ep->ep_state & EP_HALTED) {
xhci_dbg(xhci, "Reset ep command already pending\n");
- return;
+ return 0;
}
err = xhci_reset_halted_ep(xhci, slot_id, ep->ep_index, reset_type);
if (err)
- return;
+ return err;
ep->ep_state |= EP_HALTED;
xhci_ring_cmd_db(xhci);
+
+ return 0;
}
/*
continue;
}
/*
- * If ring stopped on the TD we need to cancel, then we have to
+ * If a ring stopped on the TD we need to cancel then we have to
* move the xHC endpoint ring dequeue pointer past this TD.
+ * Rings halted due to STALL may show hw_deq is past the stalled
+ * TD, but still require a set TR Deq command to flush xHC cache.
*/
hw_deq = xhci_get_hw_deq(xhci, ep->vdev, ep->ep_index,
td->urb->stream_id);
hw_deq &= ~0xf;
- if (trb_in_td(xhci, td->start_seg, td->first_trb,
+ if (td->cancel_status == TD_HALTED) {
+ cached_td = td;
+ } else if (trb_in_td(xhci, td->start_seg, td->first_trb,
td->last_trb, hw_deq, false)) {
switch (td->cancel_status) {
case TD_CLEARED: /* TD is already no-op */
struct xhci_td *td = NULL;
enum xhci_ep_reset_type reset_type;
struct xhci_command *command;
+ int err;
if (unlikely(TRB_TO_SUSPEND_PORT(le32_to_cpu(trb->generic.field[3])))) {
if (!xhci->devs[slot_id])
td->status = -EPROTO;
}
/* reset ep, reset handler cleans up cancelled tds */
- xhci_handle_halted_endpoint(xhci, ep, 0, td, reset_type);
+ err = xhci_handle_halted_endpoint(xhci, ep, 0, td,
+ reset_type);
+ if (err)
+ break;
xhci_stop_watchdog_timer_in_irq(xhci, ep);
return;
case EP_STATE_RUNNING:
* we need to issue an evaluate context command and wait on it.
*/
static int xhci_check_maxpacket(struct xhci_hcd *xhci, unsigned int slot_id,
- unsigned int ep_index, struct urb *urb)
+ unsigned int ep_index, struct urb *urb, gfp_t mem_flags)
{
struct xhci_container_ctx *out_ctx;
struct xhci_input_control_ctx *ctrl_ctx;
* changes max packet sizes.
*/
- command = xhci_alloc_command(xhci, true, GFP_KERNEL);
+ command = xhci_alloc_command(xhci, true, mem_flags);
if (!command)
return -ENOMEM;
*/
if (urb->dev->speed == USB_SPEED_FULL) {
ret = xhci_check_maxpacket(xhci, slot_id,
- ep_index, urb);
+ ep_index, urb, mem_flags);
if (ret < 0) {
xhci_urb_free_priv(urb_priv);
urb->hcpriv = NULL;
#define XHCI_DISABLE_SPARSE BIT_ULL(38)
#define XHCI_SG_TRB_CACHE_SIZE_QUIRK BIT_ULL(39)
#define XHCI_NO_SOFT_RETRY BIT_ULL(40)
+#define XHCI_BROKEN_D3COLD BIT_ULL(41)
unsigned int num_active_eps;
unsigned int limit_active_eps;
return -EINVAL;
r = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!r)
+ return -EINVAL;
pdata = devm_kzalloc(&pdev->dev,
sizeof(*pdata) +
/* Set speed */
retval = usb_control_msg(tv->udev, usb_sndctrlpipe(tv->udev, 0),
0x01, /* vendor request: set speed */
- USB_DIR_IN | USB_TYPE_VENDOR | USB_RECIP_OTHER,
+ USB_DIR_OUT | USB_TYPE_VENDOR | USB_RECIP_OTHER,
tv->speed, /* speed value */
- 0, NULL, 0, USB_CTRL_GET_TIMEOUT);
+ 0, NULL, 0, USB_CTRL_SET_TIMEOUT);
if (retval) {
tv->speed = old;
dev_dbg(&tv->udev->dev, "retval = %d\n", retval);
parport_announce_port(pp);
usb_set_intfdata(intf, pp);
+ usb_put_dev(usbdev);
return 0;
probe_abort:
glue->xceiv = devm_usb_get_phy(dev, USB_PHY_TYPE_USB2);
if (IS_ERR(glue->xceiv)) {
- dev_err(dev, "fail to getting usb-phy %d\n", ret);
ret = PTR_ERR(glue->xceiv);
+ dev_err(dev, "fail to getting usb-phy %d\n", ret);
goto err_unregister_usb_phy;
}
schedule_delayed_work(&musb->irq_work,
msecs_to_jiffies(1000));
musb->quirk_retries--;
- break;
}
- fallthrough;
+ break;
case MUSB_QUIRK_B_INVALID_VBUS_91:
if (musb->quirk_retries && !musb->flush_irq_work) {
musb_dbg(musb,
u8 gpio_input;
#endif
u8 partnum;
+ u32 fw_version;
speed_t min_speed;
speed_t max_speed;
bool use_actual_rate;
+ bool no_flow_control;
};
enum cp210x_event_state {
/* CP210X_VENDOR_SPECIFIC values */
#define CP210X_READ_2NCONFIG 0x000E
+#define CP210X_GET_FW_VER_2N 0x0010
#define CP210X_READ_LATCH 0x00C2
#define CP210X_GET_PARTNUM 0x370B
#define CP210X_GET_PORTCONFIG 0x370C
#define CP210X_2NCONFIG_GPIO_RSTLATCH_IDX 587
#define CP210X_2NCONFIG_GPIO_CONTROL_IDX 600
+/* CP2102N QFN20 port configuration values */
+#define CP2102N_QFN20_GPIO2_TXLED_MODE BIT(2)
+#define CP2102N_QFN20_GPIO3_RXLED_MODE BIT(3)
+#define CP2102N_QFN20_GPIO1_RS485_MODE BIT(4)
+#define CP2102N_QFN20_GPIO0_CLK_MODE BIT(6)
+
/* CP210X_VENDOR_SPECIFIC, CP210X_WRITE_LATCH call writes these 0x2 bytes. */
struct cp210x_gpio_write {
u8 mask;
static void cp210x_set_flow_control(struct tty_struct *tty,
struct usb_serial_port *port, struct ktermios *old_termios)
{
+ struct cp210x_serial_private *priv = usb_get_serial_data(port->serial);
struct cp210x_port_private *port_priv = usb_get_serial_port_data(port);
struct cp210x_special_chars chars;
struct cp210x_flow_ctl flow_ctl;
u32 ctl_hs;
int ret;
+ /*
+ * Some CP2102N interpret ulXonLimit as ulFlowReplace (erratum
+ * CP2102N_E104). Report back that flow control is not supported.
+ */
+ if (priv->no_flow_control) {
+ tty->termios.c_cflag &= ~CRTSCTS;
+ tty->termios.c_iflag &= ~(IXON | IXOFF);
+ }
+
if (old_termios &&
C_CRTSCTS(tty) == (old_termios->c_cflag & CRTSCTS) &&
I_IXON(tty) == (old_termios->c_iflag & IXON) &&
port_priv->crtscts = false;
}
- if (I_IXOFF(tty))
+ if (I_IXOFF(tty)) {
flow_repl |= CP210X_SERIAL_AUTO_RECEIVE;
- else
+
+ flow_ctl.ulXonLimit = cpu_to_le32(128);
+ flow_ctl.ulXoffLimit = cpu_to_le32(128);
+ } else {
flow_repl &= ~CP210X_SERIAL_AUTO_RECEIVE;
+ }
if (I_IXON(tty))
flow_repl |= CP210X_SERIAL_AUTO_TRANSMIT;
else
flow_repl &= ~CP210X_SERIAL_AUTO_TRANSMIT;
- flow_ctl.ulXonLimit = cpu_to_le32(128);
- flow_ctl.ulXoffLimit = cpu_to_le32(128);
-
dev_dbg(&port->dev, "%s - ctrl = 0x%02x, flow = 0x%02x\n", __func__,
ctl_hs, flow_repl);
priv->gpio_pushpull = (gpio_pushpull >> 3) & 0x0f;
/* 0 indicates GPIO mode, 1 is alternate function */
- priv->gpio_altfunc = (gpio_ctrl >> 2) & 0x0f;
+ if (priv->partnum == CP210X_PARTNUM_CP2102N_QFN20) {
+ /* QFN20 is special... */
+ if (gpio_ctrl & CP2102N_QFN20_GPIO0_CLK_MODE) /* GPIO 0 */
+ priv->gpio_altfunc |= BIT(0);
+ if (gpio_ctrl & CP2102N_QFN20_GPIO1_RS485_MODE) /* GPIO 1 */
+ priv->gpio_altfunc |= BIT(1);
+ if (gpio_ctrl & CP2102N_QFN20_GPIO2_TXLED_MODE) /* GPIO 2 */
+ priv->gpio_altfunc |= BIT(2);
+ if (gpio_ctrl & CP2102N_QFN20_GPIO3_RXLED_MODE) /* GPIO 3 */
+ priv->gpio_altfunc |= BIT(3);
+ } else {
+ priv->gpio_altfunc = (gpio_ctrl >> 2) & 0x0f;
+ }
if (priv->partnum == CP210X_PARTNUM_CP2102N_QFN28) {
/*
priv->use_actual_rate = use_actual_rate;
}
+static int cp210x_get_fw_version(struct usb_serial *serial, u16 value)
+{
+ struct cp210x_serial_private *priv = usb_get_serial_data(serial);
+ u8 ver[3];
+ int ret;
+
+ ret = cp210x_read_vendor_block(serial, REQTYPE_DEVICE_TO_HOST, value,
+ ver, sizeof(ver));
+ if (ret)
+ return ret;
+
+ dev_dbg(&serial->interface->dev, "%s - %d.%d.%d\n", __func__,
+ ver[0], ver[1], ver[2]);
+
+ priv->fw_version = ver[0] << 16 | ver[1] << 8 | ver[2];
+
+ return 0;
+}
+
+static void cp210x_determine_quirks(struct usb_serial *serial)
+{
+ struct cp210x_serial_private *priv = usb_get_serial_data(serial);
+ int ret;
+
+ switch (priv->partnum) {
+ case CP210X_PARTNUM_CP2102N_QFN28:
+ case CP210X_PARTNUM_CP2102N_QFN24:
+ case CP210X_PARTNUM_CP2102N_QFN20:
+ ret = cp210x_get_fw_version(serial, CP210X_GET_FW_VER_2N);
+ if (ret)
+ break;
+ if (priv->fw_version <= 0x10004)
+ priv->no_flow_control = true;
+ break;
+ default:
+ break;
+ }
+}
+
static int cp210x_attach(struct usb_serial *serial)
{
int result;
usb_set_serial_data(serial, priv);
+ cp210x_determine_quirks(serial);
cp210x_init_max_speed(serial);
result = cp210x_gpio_init(serial);
.driver_info = (kernel_ulong_t)&ftdi_jtag_quirk },
{ USB_DEVICE(FTDI_VID, FTDI_NT_ORIONLX_PLUS_PID) },
{ USB_DEVICE(FTDI_VID, FTDI_NT_ORION_IO_PID) },
+ { USB_DEVICE(FTDI_VID, FTDI_NT_ORIONMX_PID) },
{ USB_DEVICE(FTDI_VID, FTDI_SYNAPSE_SS200_PID) },
{ USB_DEVICE(FTDI_VID, FTDI_CUSTOMWARE_MINIPLEX_PID) },
{ USB_DEVICE(FTDI_VID, FTDI_CUSTOMWARE_MINIPLEX2_PID) },
/* Sienna devices */
{ USB_DEVICE(FTDI_VID, FTDI_SIENNA_PID) },
{ USB_DEVICE(ECHELON_VID, ECHELON_U20_PID) },
+ /* IDS GmbH devices */
+ { USB_DEVICE(IDS_VID, IDS_SI31A_PID) },
+ { USB_DEVICE(IDS_VID, IDS_CM31A_PID) },
/* U-Blox devices */
{ USB_DEVICE(UBLOX_VID, UBLOX_C099F9P_ZED_PID) },
{ USB_DEVICE(UBLOX_VID, UBLOX_C099F9P_ODIN_PID) },
#define FTDI_NT_ORIONLXM_PID 0x7c90 /* OrionLXm Substation Automation Platform */
#define FTDI_NT_ORIONLX_PLUS_PID 0x7c91 /* OrionLX+ Substation Automation Platform */
#define FTDI_NT_ORION_IO_PID 0x7c92 /* Orion I/O */
+#define FTDI_NT_ORIONMX_PID 0x7c93 /* OrionMX */
/*
* Synapse Wireless product ids (FTDI_VID)
#define UNJO_ISODEBUG_V1_PID 0x150D
/*
+ * IDS GmbH
+ */
+#define IDS_VID 0x2CAF
+#define IDS_SI31A_PID 0x13A2
+#define IDS_CM31A_PID 0x13A3
+
+/*
* U-Blox products (http://www.u-blox.com).
*/
#define UBLOX_VID 0x1546
// SPDX-License-Identifier: GPL-2.0
/*
- * USB ZyXEL omni.net LCD PLUS driver
+ * USB ZyXEL omni.net driver
*
* Copyright (C) 2013,2017 Johan Hovold <johan@kernel.org>
*
#include <linux/usb/serial.h>
#define DRIVER_AUTHOR "Alessandro Zummo"
-#define DRIVER_DESC "USB ZyXEL omni.net LCD PLUS Driver"
+#define DRIVER_DESC "USB ZyXEL omni.net Driver"
#define ZYXEL_VENDOR_ID 0x0586
#define ZYXEL_OMNINET_ID 0x1000
+#define ZYXEL_OMNI_56K_PLUS_ID 0x1500
/* This one seems to be a re-branded ZyXEL device */
#define BT_IGNITIONPRO_ID 0x2000
static const struct usb_device_id id_table[] = {
{ USB_DEVICE(ZYXEL_VENDOR_ID, ZYXEL_OMNINET_ID) },
+ { USB_DEVICE(ZYXEL_VENDOR_ID, ZYXEL_OMNI_56K_PLUS_ID) },
{ USB_DEVICE(ZYXEL_VENDOR_ID, BT_IGNITIONPRO_ID) },
{ } /* Terminating entry */
};
.owner = THIS_MODULE,
.name = "omninet",
},
- .description = "ZyXEL - omni.net lcd plus usb",
+ .description = "ZyXEL - omni.net usb",
.id_table = id_table,
.num_bulk_out = 2,
.calc_num_ports = omninet_calc_num_ports,
.driver_info = NCTRL(0) | RSVD(1) },
{ USB_DEVICE_INTERFACE_CLASS(TELIT_VENDOR_ID, 0x1901, 0xff), /* Telit LN940 (MBIM) */
.driver_info = NCTRL(0) },
+ { USB_DEVICE_INTERFACE_CLASS(TELIT_VENDOR_ID, 0x7010, 0xff), /* Telit LE910-S1 (RNDIS) */
+ .driver_info = NCTRL(2) },
+ { USB_DEVICE_INTERFACE_CLASS(TELIT_VENDOR_ID, 0x7011, 0xff), /* Telit LE910-S1 (ECM) */
+ .driver_info = NCTRL(2) },
{ USB_DEVICE(TELIT_VENDOR_ID, 0x9010), /* Telit SBL FN980 flashing device */
.driver_info = NCTRL(0) | ZLP },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, ZTE_PRODUCT_MF622, 0xff, 0xff, 0xff) }, /* ZTE WCDMA products */
{ USB_DEVICE(SONY_VENDOR_ID, SONY_QN3USB_PRODUCT_ID) },
{ USB_DEVICE(SANWA_VENDOR_ID, SANWA_PRODUCT_ID) },
{ USB_DEVICE(ADLINK_VENDOR_ID, ADLINK_ND6530_PRODUCT_ID) },
+ { USB_DEVICE(ADLINK_VENDOR_ID, ADLINK_ND6530GC_PRODUCT_ID) },
{ USB_DEVICE(SMART_VENDOR_ID, SMART_PRODUCT_ID) },
{ USB_DEVICE(AT_VENDOR_ID, AT_VTKIT3_PRODUCT_ID) },
{ } /* Terminating entry */
/* ADLINK ND-6530 RS232,RS485 and RS422 adapter */
#define ADLINK_VENDOR_ID 0x0b63
#define ADLINK_ND6530_PRODUCT_ID 0x6530
+#define ADLINK_ND6530GC_PRODUCT_ID 0x653a
/* SMART USB Serial Adapter */
#define SMART_VENDOR_ID 0x0b8c
/* flush the port transmit buffer */
i = usb_control_msg(serial->dev,
- usb_rcvctrlpipe(serial->dev, 0),
+ usb_sndctrlpipe(serial->dev, 0),
QT2_FLUSH_DEVICE, 0x40, 1,
port_priv->device_port, NULL, 0, QT2_USB_TIMEOUT);
/* flush the port receive buffer */
i = usb_control_msg(serial->dev,
- usb_rcvctrlpipe(serial->dev, 0),
+ usb_sndctrlpipe(serial->dev, 0),
QT2_FLUSH_DEVICE, 0x40, 0,
port_priv->device_port, NULL, 0, QT2_USB_TIMEOUT);
int status;
/* power on unit */
- status = usb_control_msg(serial->dev, usb_rcvctrlpipe(serial->dev, 0),
+ status = usb_control_msg(serial->dev, usb_sndctrlpipe(serial->dev, 0),
0xc2, 0x40, 0x8000, 0, NULL, 0,
QT2_USB_TIMEOUT);
if (status < 0) {
/* Vendor and product ids */
#define TI_VENDOR_ID 0x0451
#define IBM_VENDOR_ID 0x04b3
+#define STARTECH_VENDOR_ID 0x14b0
#define TI_3410_PRODUCT_ID 0x3410
#define IBM_4543_PRODUCT_ID 0x4543
#define IBM_454B_PRODUCT_ID 0x454b
{ USB_DEVICE(MXU1_VENDOR_ID, MXU1_1131_PRODUCT_ID) },
{ USB_DEVICE(MXU1_VENDOR_ID, MXU1_1150_PRODUCT_ID) },
{ USB_DEVICE(MXU1_VENDOR_ID, MXU1_1151_PRODUCT_ID) },
+ { USB_DEVICE(STARTECH_VENDOR_ID, TI_3410_PRODUCT_ID) },
{ } /* terminator */
};
{ USB_DEVICE(MXU1_VENDOR_ID, MXU1_1131_PRODUCT_ID) },
{ USB_DEVICE(MXU1_VENDOR_ID, MXU1_1150_PRODUCT_ID) },
{ USB_DEVICE(MXU1_VENDOR_ID, MXU1_1151_PRODUCT_ID) },
+ { USB_DEVICE(STARTECH_VENDOR_ID, TI_3410_PRODUCT_ID) },
{ } /* terminator */
};
bool match;
int nval;
u16 *val;
+ int ret;
int i;
/*
if (!val)
return ERR_PTR(-ENOMEM);
- nval = fwnode_property_read_u16_array(fwnode, "svid", val, nval);
- if (nval < 0) {
+ ret = fwnode_property_read_u16_array(fwnode, "svid", val, nval);
+ if (ret < 0) {
kfree(val);
- return ERR_PTR(nval);
+ return ERR_PTR(ret);
}
for (i = 0; i < nval; i++) {
dev = class_find_device(&typec_mux_class, NULL, fwnode,
mux_fwnode_match);
- return dev ? to_typec_switch(dev) : ERR_PTR(-EPROBE_DEFER);
+ return dev ? to_typec_mux(dev) : ERR_PTR(-EPROBE_DEFER);
}
/**
acpi_dev_free_resource_list(&resource_list);
if (!pmc->iom_base) {
- put_device(&adev->dev);
+ acpi_dev_put(adev);
return -ENOMEM;
}
+ if (IS_ERR(pmc->iom_base)) {
+ acpi_dev_put(adev);
+ return PTR_ERR(pmc->iom_base);
+ }
+
pmc->iom_adev = adev;
return 0;
break;
ret = pmc_usb_register_port(pmc, i, fwnode);
- if (ret)
+ if (ret) {
+ fwnode_handle_put(fwnode);
goto err_remove_ports;
+ }
}
platform_set_drvdata(pdev, pmc);
usb_role_switch_unregister(pmc->port[i].usb_sw);
}
- put_device(&pmc->iom_adev->dev);
+ acpi_dev_put(pmc->iom_adev);
return ret;
}
usb_role_switch_unregister(pmc->port[i].usb_sw);
}
- put_device(&pmc->iom_adev->dev);
+ acpi_dev_put(pmc->iom_adev);
return 0;
}
#define ALTMODE_DISCOVERY_MAX (SVID_DISCOVERY_MAX * MODE_DISCOVERY_MAX)
#define GET_SINK_CAP_RETRY_MS 100
+#define SEND_DISCOVER_RETRY_MS 100
struct pd_mode_data {
int svid_index; /* current SVID index */
struct kthread_work vdm_state_machine;
struct hrtimer enable_frs_timer;
struct kthread_work enable_frs;
+ struct hrtimer send_discover_timer;
+ struct kthread_work send_discover_work;
bool state_machine_running;
bool vdm_sm_running;
unsigned int nr_src_pdo;
u32 snk_pdo[PDO_MAX_OBJECTS];
unsigned int nr_snk_pdo;
+ u32 snk_vdo_v1[VDO_MAX_OBJECTS];
+ unsigned int nr_snk_vdo_v1;
u32 snk_vdo[VDO_MAX_OBJECTS];
unsigned int nr_snk_vdo;
}
}
+static void mod_send_discover_delayed_work(struct tcpm_port *port, unsigned int delay_ms)
+{
+ if (delay_ms) {
+ hrtimer_start(&port->send_discover_timer, ms_to_ktime(delay_ms), HRTIMER_MODE_REL);
+ } else {
+ hrtimer_cancel(&port->send_discover_timer);
+ kthread_queue_work(port->wq, &port->send_discover_work);
+ }
+}
+
static void tcpm_set_state(struct tcpm_port *port, enum tcpm_state state,
unsigned int delay_ms)
{
if (PD_VDO_VID(p[0]) != USB_SID_PD)
break;
- if (PD_VDO_SVDM_VER(p[0]) < svdm_version)
+ if (PD_VDO_SVDM_VER(p[0]) < svdm_version) {
typec_partner_set_svdm_version(port->partner,
PD_VDO_SVDM_VER(p[0]));
- /* 6.4.4.3.1: Only respond as UFP (device) */
- if (port->data_role == TYPEC_DEVICE &&
+ svdm_version = PD_VDO_SVDM_VER(p[0]);
+ }
+
+ port->ams = DISCOVER_IDENTITY;
+ /*
+ * PD2.0 Spec 6.10.3: respond with NAK as DFP (data host)
+ * PD3.1 Spec 6.4.4.2.5.1: respond with NAK if "invalid field" or
+ * "wrong configuation" or "Unrecognized"
+ */
+ if ((port->data_role == TYPEC_DEVICE || svdm_version >= SVDM_VER_2_0) &&
port->nr_snk_vdo) {
- /*
- * Product Type DFP and Connector Type are not defined in SVDM
- * version 1.0 and shall be set to zero.
- */
- if (typec_get_negotiated_svdm_version(typec) < SVDM_VER_2_0)
- response[1] = port->snk_vdo[0] & ~IDH_DFP_MASK
- & ~IDH_CONN_MASK;
- else
- response[1] = port->snk_vdo[0];
- for (i = 1; i < port->nr_snk_vdo; i++)
- response[i + 1] = port->snk_vdo[i];
- rlen = port->nr_snk_vdo + 1;
+ if (svdm_version < SVDM_VER_2_0) {
+ for (i = 0; i < port->nr_snk_vdo_v1; i++)
+ response[i + 1] = port->snk_vdo_v1[i];
+ rlen = port->nr_snk_vdo_v1 + 1;
+
+ } else {
+ for (i = 0; i < port->nr_snk_vdo; i++)
+ response[i + 1] = port->snk_vdo[i];
+ rlen = port->nr_snk_vdo + 1;
+ }
}
break;
case CMD_DISCOVER_SVID:
+ port->ams = DISCOVER_SVIDS;
break;
case CMD_DISCOVER_MODES:
+ port->ams = DISCOVER_MODES;
break;
case CMD_ENTER_MODE:
+ port->ams = DFP_TO_UFP_ENTER_MODE;
break;
case CMD_EXIT_MODE:
+ port->ams = DFP_TO_UFP_EXIT_MODE;
break;
case CMD_ATTENTION:
/* Attention command does not have response */
res = tcpm_ams_start(port, DISCOVER_IDENTITY);
if (res == 0)
port->send_discover = false;
+ else if (res == -EAGAIN)
+ mod_send_discover_delayed_work(port,
+ SEND_DISCOVER_RETRY_MS);
break;
case CMD_DISCOVER_SVID:
res = tcpm_ams_start(port, DISCOVER_SVIDS);
}
if (res < 0) {
- port->vdm_sm_running = false;
+ port->vdm_state = VDM_STATE_ERR_BUSY;
return;
}
}
port->vdo_data[0] = port->vdo_retry;
port->vdo_count = 1;
port->vdm_state = VDM_STATE_READY;
+ tcpm_ams_finish(port);
break;
case VDM_STATE_BUSY:
port->vdm_state = VDM_STATE_ERR_TMOUT;
tcpm_log(port, "VDM Tx error, retry");
port->vdm_retries++;
port->vdm_state = VDM_STATE_READY;
+ if (PD_VDO_SVDM(vdo_hdr) && PD_VDO_CMDT(vdo_hdr) == CMDT_INIT)
+ tcpm_ams_finish(port);
+ } else {
tcpm_ams_finish(port);
}
break;
port->vdm_state != VDM_STATE_BUSY &&
port->vdm_state != VDM_STATE_SEND_MESSAGE);
- if (port->vdm_state == VDM_STATE_ERR_TMOUT)
+ if (port->vdm_state < VDM_STATE_READY)
port->vdm_sm_running = false;
mutex_unlock(&port->lock);
if (!type) {
tcpm_log(port, "Alert message received with no type");
+ tcpm_queue_message(port, PD_MSG_CTRL_NOT_SUPP);
return;
}
/* Just handling non-battery alerts for now */
if (!(type & USB_PD_ADO_TYPE_BATT_STATUS_CHANGE)) {
- switch (port->state) {
- case SRC_READY:
- case SNK_READY:
+ if (port->pwr_role == TYPEC_SOURCE) {
+ port->upcoming_state = GET_STATUS_SEND;
+ tcpm_ams_start(port, GETTING_SOURCE_SINK_STATUS);
+ } else {
+ /*
+ * Do not check SinkTxOk here in case the Source doesn't set its Rp to
+ * SinkTxOk in time.
+ */
+ port->ams = GETTING_SOURCE_SINK_STATUS;
tcpm_set_state(port, GET_STATUS_SEND, 0);
- break;
- default:
- tcpm_queue_message(port, PD_MSG_CTRL_WAIT);
- break;
}
+ } else {
+ tcpm_queue_message(port, PD_MSG_CTRL_NOT_SUPP);
}
}
bool frs_enable;
int ret;
+ if (tcpm_vdm_ams(port) && type != PD_DATA_VENDOR_DEF) {
+ port->vdm_state = VDM_STATE_ERR_BUSY;
+ tcpm_ams_finish(port);
+ mod_vdm_delayed_work(port, 0);
+ }
+
switch (type) {
case PD_DATA_SOURCE_CAP:
for (i = 0; i < cnt; i++)
port->nr_sink_caps = cnt;
port->sink_cap_done = true;
if (port->ams == GET_SINK_CAPABILITIES)
- tcpm_pd_handle_state(port, ready_state(port), NONE_AMS, 0);
+ tcpm_set_state(port, ready_state(port), 0);
/* Unexpected Sink Capabilities */
else
tcpm_pd_handle_msg(port,
NONE_AMS);
break;
case PD_DATA_VENDOR_DEF:
- tcpm_handle_vdm_request(port, msg->payload, cnt);
+ if (tcpm_vdm_ams(port) || port->nr_snk_vdo)
+ tcpm_handle_vdm_request(port, msg->payload, cnt);
+ else if (port->negotiated_rev > PD_REV20)
+ tcpm_pd_handle_msg(port, PD_MSG_CTRL_NOT_SUPP, NONE_AMS);
break;
case PD_DATA_BIST:
port->bist_request = le32_to_cpu(msg->payload[0]);
tcpm_pd_handle_state(port, BIST_RX, BIST, 0);
break;
case PD_DATA_ALERT:
- tcpm_handle_alert(port, msg->payload, cnt);
+ if (port->state != SRC_READY && port->state != SNK_READY)
+ tcpm_pd_handle_state(port, port->pwr_role == TYPEC_SOURCE ?
+ SRC_SOFT_RESET_WAIT_SNK_TX : SNK_SOFT_RESET,
+ NONE_AMS, 0);
+ else
+ tcpm_handle_alert(port, msg->payload, cnt);
break;
case PD_DATA_BATT_STATUS:
case PD_DATA_GET_COUNTRY_INFO:
enum pd_ctrl_msg_type type = pd_header_type_le(msg->header);
enum tcpm_state next_state;
+ /*
+ * Stop VDM state machine if interrupted by other Messages while NOT_SUPP is allowed in
+ * VDM AMS if waiting for VDM responses and will be handled later.
+ */
+ if (tcpm_vdm_ams(port) && type != PD_CTRL_NOT_SUPP && type != PD_CTRL_GOOD_CRC) {
+ port->vdm_state = VDM_STATE_ERR_BUSY;
+ tcpm_ams_finish(port);
+ mod_vdm_delayed_work(port, 0);
+ }
+
switch (type) {
case PD_CTRL_GOOD_CRC:
case PD_CTRL_PING:
port->sink_cap_done = true;
tcpm_set_state(port, ready_state(port), 0);
break;
+ case SRC_READY:
+ case SNK_READY:
+ if (port->vdm_state > VDM_STATE_READY) {
+ port->vdm_state = VDM_STATE_DONE;
+ if (tcpm_vdm_ams(port))
+ tcpm_ams_finish(port);
+ mod_vdm_delayed_work(port, 0);
+ break;
+ }
+ fallthrough;
default:
tcpm_pd_handle_state(port,
port->pwr_role == TYPEC_SOURCE ?
enum pd_ext_msg_type type = pd_header_type_le(msg->header);
unsigned int data_size = pd_ext_header_data_size_le(msg->ext_msg.header);
- if (!(msg->ext_msg.header & PD_EXT_HDR_CHUNKED)) {
+ /* stopping VDM state machine if interrupted by other Messages */
+ if (tcpm_vdm_ams(port)) {
+ port->vdm_state = VDM_STATE_ERR_BUSY;
+ tcpm_ams_finish(port);
+ mod_vdm_delayed_work(port, 0);
+ }
+
+ if (!(le16_to_cpu(msg->ext_msg.header) & PD_EXT_HDR_CHUNKED)) {
tcpm_pd_handle_msg(port, PD_MSG_CTRL_NOT_SUPP, NONE_AMS);
tcpm_log(port, "Unchunked extended messages unsupported");
return;
switch (type) {
case PD_EXT_STATUS:
- /*
- * If PPS related events raised then get PPS status to clear
- * (see USB PD 3.0 Spec, 6.5.2.4)
- */
- if (msg->ext_msg.data[USB_PD_EXT_SDB_EVENT_FLAGS] &
- USB_PD_EXT_SDB_PPS_EVENTS)
- tcpm_pd_handle_state(port, GET_PPS_STATUS_SEND,
- GETTING_SOURCE_SINK_STATUS, 0);
-
- else
- tcpm_pd_handle_state(port, ready_state(port), NONE_AMS, 0);
- break;
case PD_EXT_PPS_STATUS:
- /*
- * For now the PPS status message is used to clear events
- * and nothing more.
- */
- tcpm_pd_handle_state(port, ready_state(port), NONE_AMS, 0);
+ if (port->ams == GETTING_SOURCE_SINK_STATUS) {
+ tcpm_ams_finish(port);
+ tcpm_set_state(port, ready_state(port), 0);
+ } else {
+ /* unexpected Status or PPS_Status Message */
+ tcpm_pd_handle_state(port, port->pwr_role == TYPEC_SOURCE ?
+ SRC_SOFT_RESET_WAIT_SNK_TX : SNK_SOFT_RESET,
+ NONE_AMS, 0);
+ }
break;
case PD_EXT_SOURCE_CAP_EXT:
case PD_EXT_GET_BATT_CAP:
"Data role mismatch, initiating error recovery");
tcpm_set_state(port, ERROR_RECOVERY, 0);
} else {
- if (msg->header & PD_HEADER_EXT_HDR)
+ if (le16_to_cpu(msg->header) & PD_HEADER_EXT_HDR)
tcpm_pd_ext_msg_request(port, msg);
else if (cnt)
tcpm_pd_data_request(port, msg);
return SNK_UNATTACHED;
}
-static void tcpm_check_send_discover(struct tcpm_port *port)
-{
- if ((port->data_role == TYPEC_HOST || port->negotiated_rev > PD_REV20) &&
- port->send_discover && port->pd_capable)
- tcpm_send_vdm(port, USB_SID_PD, CMD_DISCOVER_IDENT, NULL, 0);
- port->send_discover = false;
-}
-
static void tcpm_swap_complete(struct tcpm_port *port, int result)
{
if (port->swap_pending) {
break;
}
- tcpm_check_send_discover(port);
+ /*
+ * 6.4.4.3.1 Discover Identity
+ * "The Discover Identity Command Shall only be sent to SOP when there is an
+ * Explicit Contract."
+ * For now, this driver only supports SOP for DISCOVER_IDENTITY, thus using
+ * port->explicit_contract to decide whether to send the command.
+ */
+ if (port->explicit_contract)
+ mod_send_discover_delayed_work(port, 0);
+ else
+ port->send_discover = false;
+
/*
* 6.3.5
* Sending ping messages is not necessary if
if (port->vbus_present) {
u32 current_lim = tcpm_get_current_limit(port);
- if (port->slow_charger_loop || (current_lim > PD_P_SNK_STDBY_MW / 5))
+ if (port->slow_charger_loop && (current_lim > PD_P_SNK_STDBY_MW / 5))
current_lim = PD_P_SNK_STDBY_MW / 5;
tcpm_set_current_limit(port, current_lim, 5000);
tcpm_set_charge(port, true);
break;
}
- tcpm_check_send_discover(port);
+ /*
+ * 6.4.4.3.1 Discover Identity
+ * "The Discover Identity Command Shall only be sent to SOP when there is an
+ * Explicit Contract."
+ * For now, this driver only supports SOP for DISCOVER_IDENTITY, thus using
+ * port->explicit_contract.
+ */
+ if (port->explicit_contract)
+ mod_send_discover_delayed_work(port, 0);
+ else
+ port->send_discover = false;
+
power_supply_changed(port->psy);
break;
mutex_unlock(&port->lock);
}
+static void tcpm_send_discover_work(struct kthread_work *work)
+{
+ struct tcpm_port *port = container_of(work, struct tcpm_port, send_discover_work);
+
+ mutex_lock(&port->lock);
+ /* No need to send DISCOVER_IDENTITY anymore */
+ if (!port->send_discover)
+ goto unlock;
+
+ /* Retry if the port is not idle */
+ if ((port->state != SRC_READY && port->state != SNK_READY) || port->vdm_sm_running) {
+ mod_send_discover_delayed_work(port, SEND_DISCOVER_RETRY_MS);
+ goto unlock;
+ }
+
+ /* Only send the Message if the port is host for PD rev2.0 */
+ if (port->data_role == TYPEC_HOST || port->negotiated_rev > PD_REV20)
+ tcpm_send_vdm(port, USB_SID_PD, CMD_DISCOVER_IDENT, NULL, 0);
+
+unlock:
+ mutex_unlock(&port->lock);
+}
+
static int tcpm_dr_set(struct typec_port *p, enum typec_data_role data)
{
struct tcpm_port *port = typec_get_drvdata(p);
if (!fwnode)
return -EINVAL;
+ /*
+ * This fwnode has a "compatible" property, but is never populated as a
+ * struct device. Instead we simply parse it to read the properties.
+ * This it breaks fw_devlink=on. To maintain backward compatibility
+ * with existing DT files, we work around this by deleting any
+ * fwnode_links to/from this fwnode.
+ */
+ fw_devlink_purge_absent_suppliers(fwnode);
+
/* USB data support is optional */
ret = fwnode_property_read_string(fwnode, "data-role", &cap_str);
if (ret == 0) {
return ret;
}
+ /* If sink-vdos is found, sink-vdos-v1 is expected for backward compatibility. */
+ if (port->nr_snk_vdo) {
+ ret = fwnode_property_count_u32(fwnode, "sink-vdos-v1");
+ if (ret < 0)
+ return ret;
+ else if (ret == 0)
+ return -ENODATA;
+
+ port->nr_snk_vdo_v1 = min(ret, VDO_MAX_OBJECTS);
+ ret = fwnode_property_read_u32_array(fwnode, "sink-vdos-v1",
+ port->snk_vdo_v1,
+ port->nr_snk_vdo_v1);
+ if (ret < 0)
+ return ret;
+ }
+
return 0;
}
return HRTIMER_NORESTART;
}
+static enum hrtimer_restart send_discover_timer_handler(struct hrtimer *timer)
+{
+ struct tcpm_port *port = container_of(timer, struct tcpm_port, send_discover_timer);
+
+ kthread_queue_work(port->wq, &port->send_discover_work);
+ return HRTIMER_NORESTART;
+}
+
struct tcpm_port *tcpm_register_port(struct device *dev, struct tcpc_dev *tcpc)
{
struct tcpm_port *port;
kthread_init_work(&port->vdm_state_machine, vdm_state_machine_work);
kthread_init_work(&port->event_work, tcpm_pd_event_handler);
kthread_init_work(&port->enable_frs, tcpm_enable_frs_work);
+ kthread_init_work(&port->send_discover_work, tcpm_send_discover_work);
hrtimer_init(&port->state_machine_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
port->state_machine_timer.function = state_machine_timer_handler;
hrtimer_init(&port->vdm_state_machine_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
port->vdm_state_machine_timer.function = vdm_state_machine_timer_handler;
hrtimer_init(&port->enable_frs_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
port->enable_frs_timer.function = enable_frs_timer_handler;
+ hrtimer_init(&port->send_discover_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+ port->send_discover_timer.function = send_discover_timer_handler;
spin_lock_init(&port->pd_event_lock);
{
int i;
+ hrtimer_cancel(&port->send_discover_timer);
+ hrtimer_cancel(&port->enable_frs_timer);
+ hrtimer_cancel(&port->vdm_state_machine_timer);
+ hrtimer_cancel(&port->state_machine_timer);
+
tcpm_reset_port(port);
for (i = 0; i < ARRAY_SIZE(port->port_altmode); i++)
typec_unregister_altmode(port->port_altmode[i]);
const u8 *data = (void *)msg;
int i;
- for (i = 0; i < pd_header_cnt(msg->header) * 4 + 2; i++) {
+ for (i = 0; i < pd_header_cnt_le(msg->header) * 4 + 2; i++) {
ret = regmap_write(wcove->regmap, USBC_TX_DATA + i,
data[i]);
if (ret)
}
}
-static void ucsi_get_pdos(struct ucsi_connector *con, int is_partner)
+static int ucsi_get_pdos(struct ucsi_connector *con, int is_partner,
+ u32 *pdos, int offset, int num_pdos)
{
struct ucsi *ucsi = con->ucsi;
u64 command;
command = UCSI_COMMAND(UCSI_GET_PDOS) | UCSI_CONNECTOR_NUMBER(con->num);
command |= UCSI_GET_PDOS_PARTNER_PDO(is_partner);
- command |= UCSI_GET_PDOS_NUM_PDOS(UCSI_MAX_PDOS - 1);
+ command |= UCSI_GET_PDOS_PDO_OFFSET(offset);
+ command |= UCSI_GET_PDOS_NUM_PDOS(num_pdos - 1);
command |= UCSI_GET_PDOS_SRC_PDOS;
- ret = ucsi_send_command(ucsi, command, con->src_pdos,
- sizeof(con->src_pdos));
- if (ret < 0) {
+ ret = ucsi_send_command(ucsi, command, pdos + offset,
+ num_pdos * sizeof(u32));
+ if (ret < 0)
dev_err(ucsi->dev, "UCSI_GET_PDOS failed (%d)\n", ret);
+ if (ret == 0 && offset == 0)
+ dev_warn(ucsi->dev, "UCSI_GET_PDOS returned 0 bytes\n");
+
+ return ret;
+}
+
+static void ucsi_get_src_pdos(struct ucsi_connector *con, int is_partner)
+{
+ int ret;
+
+ /* UCSI max payload means only getting at most 4 PDOs at a time */
+ ret = ucsi_get_pdos(con, 1, con->src_pdos, 0, UCSI_MAX_PDOS);
+ if (ret < 0)
return;
- }
+
con->num_pdos = ret / sizeof(u32); /* number of bytes to 32-bit PDOs */
- if (ret == 0)
- dev_warn(ucsi->dev, "UCSI_GET_PDOS returned 0 bytes\n");
+ if (con->num_pdos < UCSI_MAX_PDOS)
+ return;
+
+ /* get the remaining PDOs, if any */
+ ret = ucsi_get_pdos(con, 1, con->src_pdos, UCSI_MAX_PDOS,
+ PDO_MAX_OBJECTS - UCSI_MAX_PDOS);
+ if (ret < 0)
+ return;
+
+ con->num_pdos += ret / sizeof(u32);
}
static void ucsi_pwr_opmode_change(struct ucsi_connector *con)
case UCSI_CONSTAT_PWR_OPMODE_PD:
con->rdo = con->status.request_data_obj;
typec_set_pwr_opmode(con->port, TYPEC_PWR_MODE_PD);
- ucsi_get_pdos(con, 1);
+ ucsi_get_src_pdos(con, 1);
break;
case UCSI_CONSTAT_PWR_OPMODE_TYPEC1_5:
con->rdo = 0;
ucsi_send_command(con->ucsi, command, NULL, 0);
/* 3. ACK connector change */
- clear_bit(EVENT_PENDING, &ucsi->flags);
ret = ucsi_acknowledge_connector_change(ucsi);
+ clear_bit(EVENT_PENDING, &ucsi->flags);
if (ret) {
dev_err(ucsi->dev, "%s: ACK failed (%d)", __func__, ret);
goto out_unlock;
.pr_set = ucsi_pr_swap
};
+/* Caller must call fwnode_handle_put() after use */
static struct fwnode_handle *ucsi_find_fwnode(struct ucsi_connector *con)
{
struct fwnode_handle *fwnode;
command |= UCSI_CONNECTOR_NUMBER(con->num);
ret = ucsi_send_command(ucsi, command, &con->cap, sizeof(con->cap));
if (ret < 0)
- goto out;
+ goto out_unlock;
if (con->cap.op_mode & UCSI_CONCAP_OPMODE_DRP)
cap->data = TYPEC_PORT_DRD;
trace_ucsi_register_port(con->num, &con->status);
out:
+ fwnode_handle_put(cap->fwnode);
+out_unlock:
mutex_unlock(&con->lock);
return ret;
}
}
err_reset:
+ memset(&ucsi->cap, 0, sizeof(ucsi->cap));
ucsi_reset_ppm(ucsi);
err:
return ret;
#include <linux/power_supply.h>
#include <linux/types.h>
#include <linux/usb/typec.h>
+#include <linux/usb/pd.h>
#include <linux/usb/role.h>
/* -------------------------------------------------------------------------- */
/* GET_PDOS command bits */
#define UCSI_GET_PDOS_PARTNER_PDO(_r_) ((u64)(_r_) << 23)
+#define UCSI_GET_PDOS_PDO_OFFSET(_r_) ((u64)(_r_) << 24)
#define UCSI_GET_PDOS_NUM_PDOS(_r_) ((u64)(_r_) << 32)
+#define UCSI_MAX_PDOS (4)
#define UCSI_GET_PDOS_SRC_PDOS ((u64)1 << 34)
/* -------------------------------------------------------------------------- */
#define UCSI_MAX_SVID 5
#define UCSI_MAX_ALTMODES (UCSI_MAX_SVID * 6)
-#define UCSI_MAX_PDOS (4)
#define UCSI_TYPEC_VSAFE5V 5000
#define UCSI_TYPEC_1_5_CURRENT 1500
struct power_supply *psy;
struct power_supply_desc psy_desc;
u32 rdo;
- u32 src_pdos[UCSI_MAX_PDOS];
+ u32 src_pdos[PDO_MAX_OBJECTS];
int num_pdos;
struct usb_role_switch *usb_role_sw;
#include <linux/mlx5/vport.h>
#include <linux/mlx5/fs.h>
#include <linux/mlx5/mlx5_ifc_vdpa.h>
+#include <linux/mlx5/mpfs.h>
#include "mlx5_vdpa.h"
MODULE_AUTHOR("Eli Cohen <eli@mellanox.com>");
static void mlx5_vdpa_free(struct vdpa_device *vdev)
{
struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev);
+ struct mlx5_core_dev *pfmdev;
struct mlx5_vdpa_net *ndev;
ndev = to_mlx5_vdpa_ndev(mvdev);
free_resources(ndev);
+ if (!is_zero_ether_addr(ndev->config.mac)) {
+ pfmdev = pci_get_drvdata(pci_physfn(mvdev->mdev->pdev));
+ mlx5_mpfs_del_mac(pfmdev, ndev->config.mac);
+ }
mlx5_vdpa_free_resources(&ndev->mvdev);
mutex_destroy(&ndev->reslock);
}
{
struct mlx5_vdpa_mgmtdev *mgtdev = container_of(v_mdev, struct mlx5_vdpa_mgmtdev, mgtdev);
struct virtio_net_config *config;
+ struct mlx5_core_dev *pfmdev;
struct mlx5_vdpa_dev *mvdev;
struct mlx5_vdpa_net *ndev;
struct mlx5_core_dev *mdev;
if (err)
goto err_mtu;
+ if (!is_zero_ether_addr(config->mac)) {
+ pfmdev = pci_get_drvdata(pci_physfn(mdev->pdev));
+ err = mlx5_mpfs_add_mac(pfmdev, config->mac);
+ if (err)
+ goto err_mtu;
+ }
+
mvdev->vdev.dma_dev = mdev->device;
err = mlx5_vdpa_alloc_resources(&ndev->mvdev);
if (err)
- goto err_mtu;
+ goto err_mpfs;
err = alloc_resources(ndev);
if (err)
free_resources(ndev);
err_res:
mlx5_vdpa_free_resources(&ndev->mvdev);
+err_mpfs:
+ if (!is_zero_ether_addr(config->mac))
+ mlx5_mpfs_del_mac(pfmdev, config->mac);
err_mtu:
mutex_destroy(&ndev->reslock);
put_device(&mvdev->vdev.dev);
config VFIO_PCI
tristate "VFIO support for PCI devices"
depends on VFIO && PCI && EVENTFD
+ depends on MMU
select VFIO_VIRQFD
select IRQ_BYPASS_MANAGER
help
if (len == 0xFF) {
len = vfio_ext_cap_len(vdev, ecap, epos);
if (len < 0)
- return ret;
+ return len;
}
}
vfio_platform_regions_cleanup(vdev);
err_reg:
mutex_unlock(&driver_lock);
- module_put(THIS_MODULE);
+ module_put(vdev->parent_module);
return ret;
}
return 0;
}
- size = sizeof(*cap_iovas) + (iovas * sizeof(*cap_iovas->iova_ranges));
+ size = struct_size(cap_iovas, iova_ranges, iovas);
cap_iovas = kzalloc(size, GFP_KERNEL);
if (!cap_iovas)
if (copied != len)
return -EFAULT;
- xdp->data_hard_start = buf;
- xdp->data = buf + pad;
- xdp->data_end = xdp->data + len;
+ xdp_init_buff(xdp, buflen, NULL);
+ xdp_prepare_buff(xdp, buf, pad, len, true);
hdr->buflen = buflen;
- xdp->frame_sz = buflen;
--net->refcnt_bias;
alloc_frag->offset += buflen;
enum {
VHOST_VSOCK_FEATURES = VHOST_FEATURES |
- (1ULL << VIRTIO_F_ACCESS_PLATFORM)
+ (1ULL << VIRTIO_F_ACCESS_PLATFORM) |
+ (1ULL << VIRTIO_VSOCK_F_SEQPACKET)
};
enum {
atomic_t queued_replies;
u32 guest_cid;
+ bool seqpacket_allow;
};
static u32 vhost_transport_get_local_cid(void)
size_t nbytes;
size_t iov_len, payload_len;
int head;
+ bool restore_flag = false;
spin_lock_bh(&vsock->send_pkt_list_lock);
if (list_empty(&vsock->send_pkt_list)) {
/* If the packet is greater than the space available in the
* buffer, we split it using multiple buffers.
*/
- if (payload_len > iov_len - sizeof(pkt->hdr))
+ if (payload_len > iov_len - sizeof(pkt->hdr)) {
payload_len = iov_len - sizeof(pkt->hdr);
+ /* As we are copying pieces of large packet's buffer to
+ * small rx buffers, headers of packets in rx queue are
+ * created dynamically and are initialized with header
+ * of current packet(except length). But in case of
+ * SOCK_SEQPACKET, we also must clear record delimeter
+ * bit(VIRTIO_VSOCK_SEQ_EOR). Otherwise, instead of one
+ * packet with delimeter(which marks end of record),
+ * there will be sequence of packets with delimeter
+ * bit set. After initialized header will be copied to
+ * rx buffer, this bit will be restored.
+ */
+ if (le32_to_cpu(pkt->hdr.flags) & VIRTIO_VSOCK_SEQ_EOR) {
+ pkt->hdr.flags &= ~cpu_to_le32(VIRTIO_VSOCK_SEQ_EOR);
+ restore_flag = true;
+ }
+ }
+
/* Set the correct length in the header */
pkt->hdr.len = cpu_to_le32(payload_len);
* to send it with the next available buffer.
*/
if (pkt->off < pkt->len) {
+ if (restore_flag)
+ pkt->hdr.flags |= cpu_to_le32(VIRTIO_VSOCK_SEQ_EOR);
+
/* We are queueing the same virtio_vsock_pkt to handle
* the remaining bytes, and we want to deliver it
* to monitoring devices in the next iteration.
return NULL;
}
- if (le16_to_cpu(pkt->hdr.type) == VIRTIO_VSOCK_TYPE_STREAM)
- pkt->len = le32_to_cpu(pkt->hdr.len);
+ pkt->len = le32_to_cpu(pkt->hdr.len);
/* No payload */
if (!pkt->len)
return val < vq->num;
}
+static bool vhost_transport_seqpacket_allow(u32 remote_cid);
+
static struct virtio_transport vhost_transport = {
.transport = {
.module = THIS_MODULE,
.stream_is_active = virtio_transport_stream_is_active,
.stream_allow = virtio_transport_stream_allow,
+ .seqpacket_dequeue = virtio_transport_seqpacket_dequeue,
+ .seqpacket_enqueue = virtio_transport_seqpacket_enqueue,
+ .seqpacket_allow = vhost_transport_seqpacket_allow,
+ .seqpacket_has_data = virtio_transport_seqpacket_has_data,
+
.notify_poll_in = virtio_transport_notify_poll_in,
.notify_poll_out = virtio_transport_notify_poll_out,
.notify_recv_init = virtio_transport_notify_recv_init,
.send_pkt = vhost_transport_send_pkt,
};
+static bool vhost_transport_seqpacket_allow(u32 remote_cid)
+{
+ struct vhost_vsock *vsock;
+ bool seqpacket_allow = false;
+
+ rcu_read_lock();
+ vsock = vhost_vsock_get(remote_cid);
+
+ if (vsock)
+ seqpacket_allow = vsock->seqpacket_allow;
+
+ rcu_read_unlock();
+
+ return seqpacket_allow;
+}
+
static void vhost_vsock_handle_tx_kick(struct vhost_work *work)
{
struct vhost_virtqueue *vq = container_of(work, struct vhost_virtqueue,
goto err;
}
+ if (features & (1ULL << VIRTIO_VSOCK_F_SEQPACKET))
+ vsock->seqpacket_allow = true;
+
for (i = 0; i < ARRAY_SIZE(vsock->vqs); i++) {
vq = &vsock->vqs[i];
mutex_lock(&vq->mutex);
vc_resize(c, vga_video_num_columns, vga_video_num_lines);
c->vc_scan_lines = vga_scan_lines;
- c->vc_font.height = vga_video_font_height;
+ c->vc_font.height = c->vc_cell_height = vga_video_font_height;
c->vc_complement_mask = 0x7700;
if (vga_512_chars)
c->vc_hi_font_mask = 0x0800;
switch (CUR_SIZE(c->vc_cursor_type)) {
case CUR_UNDERLINE:
vgacon_set_cursor_size(c->state.x,
- c->vc_font.height -
- (c->vc_font.height <
+ c->vc_cell_height -
+ (c->vc_cell_height <
10 ? 2 : 3),
- c->vc_font.height -
- (c->vc_font.height <
+ c->vc_cell_height -
+ (c->vc_cell_height <
10 ? 1 : 2));
break;
case CUR_TWO_THIRDS:
vgacon_set_cursor_size(c->state.x,
- c->vc_font.height / 3,
- c->vc_font.height -
- (c->vc_font.height <
+ c->vc_cell_height / 3,
+ c->vc_cell_height -
+ (c->vc_cell_height <
10 ? 1 : 2));
break;
case CUR_LOWER_THIRD:
vgacon_set_cursor_size(c->state.x,
- (c->vc_font.height * 2) / 3,
- c->vc_font.height -
- (c->vc_font.height <
+ (c->vc_cell_height * 2) / 3,
+ c->vc_cell_height -
+ (c->vc_cell_height <
10 ? 1 : 2));
break;
case CUR_LOWER_HALF:
vgacon_set_cursor_size(c->state.x,
- c->vc_font.height / 2,
- c->vc_font.height -
- (c->vc_font.height <
+ c->vc_cell_height / 2,
+ c->vc_cell_height -
+ (c->vc_cell_height <
10 ? 1 : 2));
break;
case CUR_NONE:
break;
default:
vgacon_set_cursor_size(c->state.x, 1,
- c->vc_font.height);
+ c->vc_cell_height);
break;
}
break;
unsigned int width, unsigned int height)
{
unsigned long flags;
- unsigned int scanlines = height * c->vc_font.height;
+ unsigned int scanlines = height * c->vc_cell_height;
u8 scanlines_lo = 0, r7 = 0, vsync_end = 0, mode, max_scan;
raw_spin_lock_irqsave(&vga_lock, flags);
vgacon_xres = width * VGA_FONTWIDTH;
- vgacon_yres = height * c->vc_font.height;
+ vgacon_yres = height * c->vc_cell_height;
if (vga_video_type >= VIDEO_TYPE_VGAC) {
outb_p(VGA_CRTC_MAX_SCAN, vga_video_port_reg);
max_scan = inb_p(vga_video_port_val);
static int vgacon_switch(struct vc_data *c)
{
int x = c->vc_cols * VGA_FONTWIDTH;
- int y = c->vc_rows * c->vc_font.height;
+ int y = c->vc_rows * c->vc_cell_height;
int rows = screen_info.orig_video_lines * vga_default_font_height/
- c->vc_font.height;
+ c->vc_cell_height;
/*
* We need to save screen size here as it's the only way
* we can spot the screen has been resized and we need to
cursor_size_lastto = 0;
c->vc_sw->con_cursor(c, CM_DRAW);
}
- c->vc_font.height = fontheight;
+ c->vc_font.height = c->vc_cell_height = fontheight;
vc_resize(c, 0, rows); /* Adjust console size */
}
}
if ((width << 1) * height > vga_vram_size)
return -EINVAL;
+ if (user) {
+ /*
+ * Ho ho! Someone (svgatextmode, eh?) may have reprogrammed
+ * the video mode! Set the new defaults then and go away.
+ */
+ screen_info.orig_video_cols = width;
+ screen_info.orig_video_lines = height;
+ vga_default_font_height = c->vc_cell_height;
+ return 0;
+ }
if (width % 2 || width > screen_info.orig_video_cols ||
height > (screen_info.orig_video_lines * vga_default_font_height)/
- c->vc_font.height)
- /* let svgatextmode tinker with video timings and
- return success */
- return (user) ? 0 : -EINVAL;
+ c->vc_cell_height)
+ return -EINVAL;
if (con_is_visible(c) && !vga_is_gfx) /* who knows */
vgacon_doresize(c, width, height);
return VM_FAULT_SIGBUS;
get_page(page);
+
+ if (vmf->vma->vm_file)
+ page->mapping = vmf->vma->vm_file->f_mapping;
+ else
+ printk(KERN_ERR "no mapping available\n");
+
+ BUG_ON(!page->mapping);
page->index = vmf->pgoff;
vmf->page = page;
.page_mkwrite = fb_deferred_io_mkwrite,
};
+static int fb_deferred_io_set_page_dirty(struct page *page)
+{
+ if (!PageDirty(page))
+ SetPageDirty(page);
+ return 0;
+}
+
+static const struct address_space_operations fb_deferred_io_aops = {
+ .set_page_dirty = fb_deferred_io_set_page_dirty,
+};
+
int fb_deferred_io_mmap(struct fb_info *info, struct vm_area_struct *vma)
{
vma->vm_ops = &fb_deferred_io_vm_ops;
}
EXPORT_SYMBOL_GPL(fb_deferred_io_init);
+void fb_deferred_io_open(struct fb_info *info,
+ struct inode *inode,
+ struct file *file)
+{
+ file->f_mapping->a_ops = &fb_deferred_io_aops;
+}
+EXPORT_SYMBOL_GPL(fb_deferred_io_open);
+
void fb_deferred_io_cleanup(struct fb_info *info)
{
struct fb_deferred_io *fbdefio = info->fbdefio;
+ struct page *page;
+ int i;
BUG_ON(!fbdefio);
cancel_delayed_work_sync(&info->deferred_work);
+
+ /* clear out the mapping that we setup */
+ for (i = 0 ; i < info->fix.smem_len; i += PAGE_SIZE) {
+ page = fb_deferred_io_page(info, i);
+ page->mapping = NULL;
+ }
+
mutex_destroy(&fbdefio->lock);
}
EXPORT_SYMBOL_GPL(fb_deferred_io_cleanup);
return -EINVAL;
pr_debug("resize now %ix%i\n", var.xres, var.yres);
- if (con_is_visible(vc)) {
+ if (con_is_visible(vc) && vc->vc_mode == KD_TEXT) {
var.activate = FB_ACTIVATE_NOW |
FB_ACTIVATE_FORCE;
fb_set_var(info, &var);
return 0;
}
-static const struct seq_operations proc_fb_seq_ops = {
+static const struct seq_operations __maybe_unused proc_fb_seq_ops = {
.start = fb_seq_start,
.next = fb_seq_next,
.stop = fb_seq_stop,
if (res)
module_put(info->fbops->owner);
}
+#ifdef CONFIG_FB_DEFERRED_IO
+ if (info->fbdefio)
+ fb_deferred_io_open(info, inode, file);
+#endif
out:
unlock_fb_info(info);
if (res)
hga_vram = ioremap(0xb0000, hga_vram_len);
if (!hga_vram)
- goto error;
+ return -ENOMEM;
if (request_region(0x3b0, 12, "hgafb"))
release_io_ports = 1;
hga_type_name = "Hercules";
break;
}
- return 1;
+ return 0;
error:
if (release_io_ports)
release_region(0x3b0, 12);
if (release_io_port)
release_region(0x3bf, 1);
- return 0;
+
+ iounmap(hga_vram);
+
+ pr_err("hgafb: HGA card not detected.\n");
+
+ return -EINVAL;
}
/**
static int hgafb_probe(struct platform_device *pdev)
{
struct fb_info *info;
+ int ret;
- if (! hga_card_detect()) {
- printk(KERN_INFO "hgafb: HGA card not detected.\n");
- if (hga_vram)
- iounmap(hga_vram);
- return -EINVAL;
- }
+ ret = hga_card_detect();
+ if (ret)
+ return ret;
printk(KERN_INFO "hgafb: %s with %ldK of memory detected.\n",
hga_type_name, hga_vram_len/1024);
struct imstt_par *par;
struct fb_info *info;
struct device_node *dp;
+ int ret = -ENOMEM;
dp = pci_device_to_OF_node(pdev);
if(dp)
default:
printk(KERN_INFO "imsttfb: Device 0x%x unknown, "
"contact maintainer.\n", pdev->device);
- release_mem_region(addr, size);
- framebuffer_release(info);
- return -ENODEV;
+ ret = -ENODEV;
+ goto error;
}
info->fix.smem_start = addr;
info->screen_base = (__u8 *)ioremap(addr, par->ramdac == IBM ?
0x400000 : 0x800000);
- if (!info->screen_base) {
- release_mem_region(addr, size);
- framebuffer_release(info);
- return -ENOMEM;
- }
+ if (!info->screen_base)
+ goto error;
info->fix.mmio_start = addr + 0x800000;
par->dc_regs = ioremap(addr + 0x800000, 0x1000);
+ if (!par->dc_regs)
+ goto error;
par->cmap_regs_phys = addr + 0x840000;
par->cmap_regs = (__u8 *)ioremap(addr + 0x840000, 0x1000);
+ if (!par->cmap_regs)
+ goto error;
info->pseudo_palette = par->palette;
init_imstt(info);
pci_set_drvdata(pdev, info);
return 0;
+
+error:
+ if (par->dc_regs)
+ iounmap(par->dc_regs);
+ if (info->screen_base)
+ iounmap(info->screen_base);
+ release_mem_region(addr, size);
+ framebuffer_release(info);
+ return ret;
}
static void imsttfb_remove(struct pci_dev *pdev)
err = mmu_interval_notifier_insert_locked(
&map->notifier, vma->vm_mm, vma->vm_start,
vma->vm_end - vma->vm_start, &gntdev_mmu_ops);
- if (err)
+ if (err) {
+ map->vma = NULL;
goto out_unlock_put;
+ }
}
mutex_unlock(&priv->lock);
int rc = -ENOMEM;
char *start;
+ if (io_tlb_default_mem != NULL) {
+ pr_warn("swiotlb buffer already initialized\n");
+ return -EEXIST;
+ }
+
retry:
m_ret = XEN_SWIOTLB_ENOMEM;
order = get_order(bytes);
}
pgmap = kzalloc(sizeof(*pgmap), GFP_KERNEL);
- if (!pgmap)
+ if (!pgmap) {
+ ret = -ENOMEM;
goto err_pgmap;
+ }
pgmap->type = MEMORY_DEVICE_GENERIC;
pgmap->range = (struct range) {
struct pci_dev *dev, int devid,
publish_pci_dev_cb publish_cb)
{
- int err = 0, slot, func = -1;
+ int err = 0, slot, func = PCI_FUNC(dev->devfn);
struct pci_dev_entry *t, *dev_entry;
struct vpci_dev_data *vpci_dev = pdev->pci_dev_data;
/*
* Keep multi-function devices together on the virtual PCI bus, except
- * virtual functions.
+ * that we want to keep virtual functions at func 0 on their own. They
+ * aren't multi-function devices and hence their presence at func 0
+ * may cause guests to not scan the other functions.
*/
- if (!dev->is_virtfn) {
+ if (!dev->is_virtfn || func) {
for (slot = 0; slot < PCI_SLOT_MAX; slot++) {
if (list_empty(&vpci_dev->dev_list[slot]))
continue;
t = list_entry(list_first(&vpci_dev->dev_list[slot]),
struct pci_dev_entry, list);
+ if (t->dev->is_virtfn && !PCI_FUNC(t->dev->devfn))
+ continue;
if (match_slot(dev, t->dev)) {
dev_info(&dev->dev, "vpci: assign to virtual slot %d func %d\n",
- slot, PCI_FUNC(dev->devfn));
+ slot, func);
list_add_tail(&dev_entry->list,
&vpci_dev->dev_list[slot]);
- func = PCI_FUNC(dev->devfn);
goto unlock;
}
}
slot);
list_add_tail(&dev_entry->list,
&vpci_dev->dev_list[slot]);
- func = dev->is_virtfn ? 0 : PCI_FUNC(dev->devfn);
goto unlock;
}
}
return err;
}
-static int xen_pcibk_reconfigure(struct xen_pcibk_device *pdev)
+static int xen_pcibk_reconfigure(struct xen_pcibk_device *pdev,
+ enum xenbus_state state)
{
int err = 0;
int num_devs;
dev_dbg(&pdev->xdev->dev, "Reconfiguring device ...\n");
mutex_lock(&pdev->dev_lock);
- /* Make sure we only reconfigure once */
- if (xenbus_read_driver_state(pdev->xdev->nodename) !=
- XenbusStateReconfiguring)
+ if (xenbus_read_driver_state(pdev->xdev->nodename) != state)
goto out;
err = xenbus_scanf(XBT_NIL, pdev->xdev->nodename, "num_devs", "%d",
}
}
+ if (state != XenbusStateReconfiguring)
+ /* Make sure we only reconfigure once. */
+ goto out;
+
err = xenbus_switch_state(pdev->xdev, XenbusStateReconfigured);
if (err) {
xenbus_dev_fatal(pdev->xdev, err,
break;
case XenbusStateReconfiguring:
- xen_pcibk_reconfigure(pdev);
+ xen_pcibk_reconfigure(pdev, XenbusStateReconfiguring);
break;
case XenbusStateConnected:
xen_pcibk_setup_backend(pdev);
break;
+ case XenbusStateInitialised:
+ /*
+ * We typically move to Initialised when the first device was
+ * added. Hence subsequent devices getting added may need
+ * reconfiguring.
+ */
+ xen_pcibk_reconfigure(pdev, XenbusStateInitialised);
+ break;
+
default:
break;
}
return ret;
call->unmarshall++;
+ fallthrough;
+
case 5:
break;
}
r->node[loop] = ntohl(b[loop + 5]);
call->unmarshall++;
+ fallthrough;
case 2:
break;
r->node[loop] = ntohl(b[loop + 5]);
call->unmarshall++;
+ fallthrough;
case 2:
break;
afs_extract_to_tmp(call);
call->unmarshall++;
+ fallthrough;
case 3:
break;
new_inode = d_inode(new_dentry);
if (new_inode) {
spin_lock(&new_inode->i_lock);
- if (new_inode->i_nlink > 0)
+ if (S_ISDIR(new_inode->i_mode))
+ clear_nlink(new_inode);
+ else if (new_inode->i_nlink > 0)
drop_nlink(new_inode);
spin_unlock(&new_inode->i_lock);
}
req->file_size = vp->scb.status.size;
call->unmarshall++;
+ fallthrough;
case 5:
break;
_debug("motd '%s'", p);
call->unmarshall++;
+ fallthrough;
case 8:
break;
xdr_decode_AFSVolSync(&bp, &op->volsync);
call->unmarshall++;
+ fallthrough;
case 6:
break;
xdr_decode_AFSVolSync(&bp, &op->volsync);
call->unmarshall++;
+ fallthrough;
case 4:
break;
goto error_fs;
afs_proc_symlink = proc_symlink("fs/afs", NULL, "../self/net/afs");
- if (IS_ERR(afs_proc_symlink)) {
- ret = PTR_ERR(afs_proc_symlink);
+ if (!afs_proc_symlink) {
+ ret = -ENOMEM;
goto error_proc;
}
if (ret < 0)
return ret;
call->unmarshall = 6;
+ fallthrough;
case 6:
break;
return ret;
}
- start += ret * PAGE_SIZE;
+ start += ret;
cond_resched();
} while (wbc->nr_to_write > 0);
struct inode *inode = file_inode(file);
struct afs_vnode *vnode = AFS_FS_I(inode);
unsigned long priv;
+ vm_fault_t ret = VM_FAULT_RETRY;
_enter("{{%llx:%llu}},{%lx}", vnode->fid.vid, vnode->fid.vnode, page->index);
#ifdef CONFIG_AFS_FSCACHE
if (PageFsCache(page) &&
wait_on_page_fscache_killable(page) < 0)
- return VM_FAULT_RETRY;
+ goto out;
#endif
if (wait_on_page_writeback_killable(page))
- return VM_FAULT_RETRY;
+ goto out;
if (lock_page_killable(page) < 0)
- return VM_FAULT_RETRY;
+ goto out;
/* We mustn't change page->private until writeback is complete as that
* details the portion of the page we need to write back and we might
*/
if (wait_on_page_writeback_killable(page) < 0) {
unlock_page(page);
- return VM_FAULT_RETRY;
+ goto out;
}
priv = afs_page_dirty(page, 0, thp_size(page));
}
file_update_time(file);
+ ret = VM_FAULT_LOCKED;
+out:
sb_end_pagefault(inode->i_sb);
- return VM_FAULT_LOCKED;
+ return ret;
}
/*
lockdep_assert_held(&bdev->bd_mutex);
+ if (!(disk->flags & GENHD_FL_UP))
+ return -ENXIO;
+
rescan:
if (bdev->bd_part_count)
return -EBUSY;
struct gendisk *disk = bdev->bd_disk;
int ret = 0;
+ if (!(disk->flags & GENHD_FL_UP))
+ return -ENXIO;
+
if (!bdev->bd_openers) {
if (!bdev_is_partition(bdev)) {
ret = 0;
whole->bd_part_count++;
mutex_unlock(&whole->bd_mutex);
- if (!(disk->flags & GENHD_FL_UP) ||
- !bdev_nr_sectors(bdev)) {
+ if (!bdev_nr_sectors(bdev)) {
__blkdev_put(whole, mode, 1);
bdput(whole);
return -ENXIO;
struct block_device *bdev;
struct gendisk *disk;
- down_read(&bdev_lookup_sem);
bdev = bdget(dev);
if (!bdev) {
- up_read(&bdev_lookup_sem);
blk_request_module(dev);
- down_read(&bdev_lookup_sem);
-
bdev = bdget(dev);
if (!bdev)
- goto unlock;
+ return NULL;
}
disk = bdev->bd_disk;
goto put_disk;
if (!try_module_get(bdev->bd_disk->fops->owner))
goto put_disk;
- up_read(&bdev_lookup_sem);
return bdev;
put_disk:
put_disk(disk);
bdput:
bdput(bdev);
-unlock:
- up_read(&bdev_lookup_sem);
return NULL;
}
spin_lock(&sinfo->lock);
spin_lock(&cache->lock);
if (!--cache->ro) {
- num_bytes = cache->length - cache->reserved -
- cache->pinned - cache->bytes_super -
- cache->zone_unusable - cache->used;
- sinfo->bytes_readonly -= num_bytes;
if (btrfs_is_zoned(cache->fs_info)) {
/* Migrate zone_unusable bytes back */
cache->zone_unusable = cache->alloc_offset - cache->used;
sinfo->bytes_zone_unusable += cache->zone_unusable;
sinfo->bytes_readonly -= cache->zone_unusable;
}
+ num_bytes = cache->length - cache->reserved -
+ cache->pinned - cache->bytes_super -
+ cache->zone_unusable - cache->used;
+ sinfo->bytes_readonly -= num_bytes;
list_del_init(&cache->ro_list);
}
spin_unlock(&cache->lock);
#include "compression.h"
#include "extent_io.h"
#include "extent_map.h"
+#include "zoned.h"
static const char* const btrfs_compress_types[] = { "", "zlib", "lzo", "zstd" };
*/
inode = cb->inode;
cb->compressed_pages[0]->mapping = cb->inode->i_mapping;
+ btrfs_record_physical_zoned(inode, cb->start, bio);
btrfs_writepage_endio_finish_ordered(cb->compressed_pages[0],
cb->start, cb->start + cb->len - 1,
bio->bi_status == BLK_STS_OK);
u64 first_byte = disk_start;
blk_status_t ret;
int skip_sum = inode->flags & BTRFS_INODE_NODATASUM;
+ const bool use_append = btrfs_use_zone_append(inode, disk_start);
+ const unsigned int bio_op = use_append ? REQ_OP_ZONE_APPEND : REQ_OP_WRITE;
WARN_ON(!PAGE_ALIGNED(start));
cb = kmalloc(compressed_bio_size(fs_info, compressed_len), GFP_NOFS);
cb->nr_pages = nr_pages;
bio = btrfs_bio_alloc(first_byte);
- bio->bi_opf = REQ_OP_WRITE | write_flags;
+ bio->bi_opf = bio_op | write_flags;
bio->bi_private = cb;
bio->bi_end_io = end_compressed_bio_write;
+ if (use_append) {
+ struct extent_map *em;
+ struct map_lookup *map;
+ struct block_device *bdev;
+
+ em = btrfs_get_chunk_map(fs_info, disk_start, PAGE_SIZE);
+ if (IS_ERR(em)) {
+ kfree(cb);
+ bio_put(bio);
+ return BLK_STS_NOTSUPP;
+ }
+
+ map = em->map_lookup;
+ /* We only support single profile for now */
+ ASSERT(map->num_stripes == 1);
+ bdev = map->stripes[0].dev->bdev;
+
+ bio_set_dev(bio, bdev);
+ free_extent_map(em);
+ }
+
if (blkcg_css) {
bio->bi_opf |= REQ_CGROUP_PUNT;
kthread_associate_blkcg(blkcg_css);
bytes_left = compressed_len;
for (pg_index = 0; pg_index < cb->nr_pages; pg_index++) {
int submit = 0;
+ int len = 0;
page = compressed_pages[pg_index];
page->mapping = inode->vfs_inode.i_mapping;
submit = btrfs_bio_fits_in_stripe(page, PAGE_SIZE, bio,
0);
+ /*
+ * Page can only be added to bio if the current bio fits in
+ * stripe.
+ */
+ if (!submit) {
+ if (pg_index == 0 && use_append)
+ len = bio_add_zone_append_page(bio, page,
+ PAGE_SIZE, 0);
+ else
+ len = bio_add_page(bio, page, PAGE_SIZE, 0);
+ }
+
page->mapping = NULL;
- if (submit || bio_add_page(bio, page, PAGE_SIZE, 0) <
- PAGE_SIZE) {
+ if (submit || len < PAGE_SIZE) {
/*
* inc the count before we submit the bio so
* we know the end IO handler won't happen before
}
bio = btrfs_bio_alloc(first_byte);
- bio->bi_opf = REQ_OP_WRITE | write_flags;
+ bio->bi_opf = bio_op | write_flags;
bio->bi_private = cb;
bio->bi_end_io = end_compressed_bio_write;
if (blkcg_css)
bio->bi_opf |= REQ_CGROUP_PUNT;
+ /*
+ * Use bio_add_page() to ensure the bio has at least one
+ * page.
+ */
bio_add_page(bio, page, PAGE_SIZE, 0);
}
if (bytes_left < PAGE_SIZE) {
struct btrfs_inode *inode, u64 new_size,
u32 min_type);
-int btrfs_start_delalloc_snapshot(struct btrfs_root *root);
+int btrfs_start_delalloc_snapshot(struct btrfs_root *root, bool in_reclaim_context);
int btrfs_start_delalloc_roots(struct btrfs_fs_info *fs_info, long nr,
bool in_reclaim_context);
int btrfs_set_extent_delalloc(struct btrfs_inode *inode, u64 start, u64 end,
ret = -EINVAL;
}
+ if (memcmp(fs_info->fs_devices->fsid, fs_info->super_copy->fsid,
+ BTRFS_FSID_SIZE)) {
+ btrfs_err(fs_info,
+ "superblock fsid doesn't match fsid of fs_devices: %pU != %pU",
+ fs_info->super_copy->fsid, fs_info->fs_devices->fsid);
+ ret = -EINVAL;
+ }
+
+ if (btrfs_fs_incompat(fs_info, METADATA_UUID) &&
+ memcmp(fs_info->fs_devices->metadata_uuid,
+ fs_info->super_copy->metadata_uuid, BTRFS_FSID_SIZE)) {
+ btrfs_err(fs_info,
+"superblock metadata_uuid doesn't match metadata uuid of fs_devices: %pU != %pU",
+ fs_info->super_copy->metadata_uuid,
+ fs_info->fs_devices->metadata_uuid);
+ ret = -EINVAL;
+ }
+
if (memcmp(fs_info->fs_devices->metadata_uuid, sb->dev_item.fsid,
BTRFS_FSID_SIZE) != 0) {
btrfs_err(fs_info,
disk_super = fs_info->super_copy;
- ASSERT(!memcmp(fs_info->fs_devices->fsid, fs_info->super_copy->fsid,
- BTRFS_FSID_SIZE));
-
- if (btrfs_fs_incompat(fs_info, METADATA_UUID)) {
- ASSERT(!memcmp(fs_info->fs_devices->metadata_uuid,
- fs_info->super_copy->metadata_uuid,
- BTRFS_FSID_SIZE));
- }
features = btrfs_super_flags(disk_super);
if (features & BTRFS_SUPER_FLAG_CHANGING_FSID_V2) {
stripe = bbio->stripes;
for (i = 0; i < bbio->num_stripes; i++, stripe++) {
u64 bytes;
+ struct btrfs_device *device = stripe->dev;
- if (!stripe->dev->bdev) {
+ if (!device->bdev) {
ASSERT(btrfs_test_opt(fs_info, DEGRADED));
continue;
}
+ if (!test_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state))
+ continue;
+
ret = do_discard_extent(stripe, &bytes);
if (!ret) {
discarded_bytes += bytes;
trace_run_delayed_ref_head(fs_info, head, 0);
btrfs_delayed_ref_unlock(head);
btrfs_put_delayed_ref_head(head);
- return 0;
+ return ret;
}
static struct btrfs_delayed_ref_head *btrfs_obtain_ref_head(
/* Note that em_end from extent_map_end() is exclusive */
iosize = min(em_end, end + 1) - cur;
- if (btrfs_use_zone_append(inode, em))
+ if (btrfs_use_zone_append(inode, em->block_start))
opf = REQ_OP_ZONE_APPEND;
free_extent_map(em);
u64 start, u64 len)
{
int ret = 0;
- u64 off = start;
+ u64 off;
u64 max = start + len;
u32 flags = 0;
u32 found_type;
goto out_free_ulist;
}
+ /*
+ * We can't initialize that to 'start' as this could miss extents due
+ * to extent item merging
+ */
+ off = 0;
start = round_down(start, btrfs_inode_sectorsize(inode));
len = round_up(max, btrfs_inode_sectorsize(inode)) - start;
u64 end_byte = bytenr + len;
u64 csum_end;
struct extent_buffer *leaf;
- int ret;
+ int ret = 0;
const u32 csum_size = fs_info->csum_size;
u32 blocksize_bits = fs_info->sectorsize_bits;
ret = btrfs_search_slot(trans, root, &key, path, -1, 1);
if (ret > 0) {
+ ret = 0;
if (path->slots[0] == 0)
break;
path->slots[0]--;
ret = btrfs_del_items(trans, root, path,
path->slots[0], del_nr);
if (ret)
- goto out;
+ break;
if (key.offset == bytenr)
break;
} else if (key.offset < bytenr && csum_end > end_byte) {
ret = btrfs_split_item(trans, root, path, &key, offset);
if (ret && ret != -EAGAIN) {
btrfs_abort_transaction(trans, ret);
- goto out;
+ break;
}
+ ret = 0;
key.offset = end_byte - 1;
} else {
}
btrfs_release_path(path);
}
- ret = 0;
-out:
btrfs_free_path(path);
return ret;
}
+static int find_next_csum_offset(struct btrfs_root *root,
+ struct btrfs_path *path,
+ u64 *next_offset)
+{
+ const u32 nritems = btrfs_header_nritems(path->nodes[0]);
+ struct btrfs_key found_key;
+ int slot = path->slots[0] + 1;
+ int ret;
+
+ if (nritems == 0 || slot >= nritems) {
+ ret = btrfs_next_leaf(root, path);
+ if (ret < 0) {
+ return ret;
+ } else if (ret > 0) {
+ *next_offset = (u64)-1;
+ return 0;
+ }
+ slot = path->slots[0];
+ }
+
+ btrfs_item_key_to_cpu(path->nodes[0], &found_key, slot);
+
+ if (found_key.objectid != BTRFS_EXTENT_CSUM_OBJECTID ||
+ found_key.type != BTRFS_EXTENT_CSUM_KEY)
+ *next_offset = (u64)-1;
+ else
+ *next_offset = found_key.offset;
+
+ return 0;
+}
+
int btrfs_csum_file_blocks(struct btrfs_trans_handle *trans,
struct btrfs_root *root,
struct btrfs_ordered_sum *sums)
u64 total_bytes = 0;
u64 csum_offset;
u64 bytenr;
- u32 nritems;
u32 ins_size;
int index = 0;
int found_next;
goto insert;
}
} else {
- int slot = path->slots[0] + 1;
- /* we didn't find a csum item, insert one */
- nritems = btrfs_header_nritems(path->nodes[0]);
- if (!nritems || (path->slots[0] >= nritems - 1)) {
- ret = btrfs_next_leaf(root, path);
- if (ret < 0) {
- goto out;
- } else if (ret > 0) {
- found_next = 1;
- goto insert;
- }
- slot = path->slots[0];
- }
- btrfs_item_key_to_cpu(path->nodes[0], &found_key, slot);
- if (found_key.objectid != BTRFS_EXTENT_CSUM_OBJECTID ||
- found_key.type != BTRFS_EXTENT_CSUM_KEY) {
- found_next = 1;
- goto insert;
- }
- next_offset = found_key.offset;
+ /* We didn't find a csum item, insert one. */
+ ret = find_next_csum_offset(root, path, &next_offset);
+ if (ret < 0)
+ goto out;
found_next = 1;
goto insert;
}
tmp = sums->len - total_bytes;
tmp >>= fs_info->sectorsize_bits;
WARN_ON(tmp < 1);
+ extend_nr = max_t(int, 1, tmp);
+
+ /*
+ * A log tree can already have checksum items with a subset of
+ * the checksums we are trying to log. This can happen after
+ * doing a sequence of partial writes into prealloc extents and
+ * fsyncs in between, with a full fsync logging a larger subrange
+ * of an extent for which a previous fast fsync logged a smaller
+ * subrange. And this happens in particular due to merging file
+ * extent items when we complete an ordered extent for a range
+ * covered by a prealloc extent - this is done at
+ * btrfs_mark_extent_written().
+ *
+ * So if we try to extend the previous checksum item, which has
+ * a range that ends at the start of the range we want to insert,
+ * make sure we don't extend beyond the start offset of the next
+ * checksum item. If we are at the last item in the leaf, then
+ * forget the optimization of extending and add a new checksum
+ * item - it is not worth the complexity of releasing the path,
+ * getting the first key for the next leaf, repeat the btree
+ * search, etc, because log trees are temporary anyway and it
+ * would only save a few bytes of leaf space.
+ */
+ if (root->root_key.objectid == BTRFS_TREE_LOG_OBJECTID) {
+ if (path->slots[0] + 1 >=
+ btrfs_header_nritems(path->nodes[0])) {
+ ret = find_next_csum_offset(root, path, &next_offset);
+ if (ret < 0)
+ goto out;
+ found_next = 1;
+ goto insert;
+ }
+
+ ret = find_next_csum_offset(root, path, &next_offset);
+ if (ret < 0)
+ goto out;
+
+ tmp = (next_offset - bytenr) >> fs_info->sectorsize_bits;
+ if (tmp <= INT_MAX)
+ extend_nr = min_t(int, extend_nr, tmp);
+ }
- extend_nr = max_t(int, 1, (int)tmp);
diff = (csum_offset + extend_nr) * csum_size;
diff = min(diff,
MAX_CSUM_ITEMS(fs_info, csum_size) * csum_size);
int del_nr = 0;
int del_slot = 0;
int recow;
- int ret;
+ int ret = 0;
u64 ino = btrfs_ino(inode);
path = btrfs_alloc_path();
}
out:
btrfs_free_path(path);
- return 0;
+ return ret;
}
/*
return ret;
}
+static inline bool skip_inode_logging(const struct btrfs_log_ctx *ctx)
+{
+ struct btrfs_inode *inode = BTRFS_I(ctx->inode);
+ struct btrfs_fs_info *fs_info = inode->root->fs_info;
+
+ if (btrfs_inode_in_log(inode, fs_info->generation) &&
+ list_empty(&ctx->ordered_extents))
+ return true;
+
+ /*
+ * If we are doing a fast fsync we can not bail out if the inode's
+ * last_trans is <= then the last committed transaction, because we only
+ * update the last_trans of the inode during ordered extent completion,
+ * and for a fast fsync we don't wait for that, we only wait for the
+ * writeback to complete.
+ */
+ if (inode->last_trans <= fs_info->last_trans_committed &&
+ (test_bit(BTRFS_INODE_NEEDS_FULL_SYNC, &inode->runtime_flags) ||
+ list_empty(&ctx->ordered_extents)))
+ return true;
+
+ return false;
+}
+
/*
* fsync call for both files and directories. This logs the inode into
* the tree log instead of forcing full commits whenever possible.
atomic_inc(&root->log_batch);
- /*
- * If we are doing a fast fsync we can not bail out if the inode's
- * last_trans is <= then the last committed transaction, because we only
- * update the last_trans of the inode during ordered extent completion,
- * and for a fast fsync we don't wait for that, we only wait for the
- * writeback to complete.
- */
smp_mb();
- if (btrfs_inode_in_log(BTRFS_I(inode), fs_info->generation) ||
- (BTRFS_I(inode)->last_trans <= fs_info->last_trans_committed &&
- (full_sync || list_empty(&ctx.ordered_extents)))) {
+ if (skip_inode_logging(&ctx)) {
/*
* We've had everything committed since the last time we were
* modified so clear this flag in case it was set for whatever
{
struct btrfs_block_group *block_group;
struct rb_node *node;
- int ret;
+ int ret = 0;
btrfs_info(fs_info, "cleaning free space cache v1");
if (ret || truncated) {
u64 unwritten_start = start;
+ /*
+ * If we failed to finish this ordered extent for any reason we
+ * need to make sure BTRFS_ORDERED_IOERR is set on the ordered
+ * extent, and mark the inode with the error if it wasn't
+ * already set. Any error during writeback would have already
+ * set the mapping error, so we need to set it if we're the ones
+ * marking this ordered extent as failed.
+ */
+ if (ret && !test_and_set_bit(BTRFS_ORDERED_IOERR,
+ &ordered_extent->flags))
+ mapping_set_error(ordered_extent->inode->i_mapping, -EIO);
+
if (truncated)
unwritten_start += logical_len;
clear_extent_uptodate(io_tree, unwritten_start, end, NULL);
inode = list_first_entry(&fs_info->delayed_iputs,
struct btrfs_inode, delayed_iput);
run_delayed_iput_locked(fs_info, inode);
+ cond_resched_lock(&fs_info->delayed_iput_lock);
}
spin_unlock(&fs_info->delayed_iput_lock);
}
iomap->bdev = fs_info->fs_devices->latest_bdev;
iomap->length = len;
- if (write && btrfs_use_zone_append(BTRFS_I(inode), em))
+ if (write && btrfs_use_zone_append(BTRFS_I(inode), em->block_start))
iomap->flags |= IOMAP_F_ZONE_APPEND;
free_extent_map(em);
int ret2;
bool root_log_pinned = false;
bool dest_log_pinned = false;
+ bool need_abort = false;
/* we only allow rename subvolume link between subvolumes */
if (old_ino != BTRFS_FIRST_FREE_OBJECTID && root != dest)
old_idx);
if (ret)
goto out_fail;
+ need_abort = true;
}
/* And now for the dest. */
new_ino,
btrfs_ino(BTRFS_I(old_dir)),
new_idx);
- if (ret)
+ if (ret) {
+ if (need_abort)
+ btrfs_abort_transaction(trans, ret);
goto out_fail;
+ }
}
/* Update inode version and ctime/mtime. */
return ret;
}
-int btrfs_start_delalloc_snapshot(struct btrfs_root *root)
+int btrfs_start_delalloc_snapshot(struct btrfs_root *root, bool in_reclaim_context)
{
struct writeback_control wbc = {
.nr_to_write = LONG_MAX,
if (test_bit(BTRFS_FS_STATE_ERROR, &fs_info->fs_state))
return -EROFS;
- return start_delalloc_inodes(root, &wbc, true, false);
+ return start_delalloc_inodes(root, &wbc, true, in_reclaim_context);
}
int btrfs_start_delalloc_roots(struct btrfs_fs_info *fs_info, long nr,
if (!fa->flags_valid) {
/* 1 item for the inode */
trans = btrfs_start_transaction(root, 1);
+ if (IS_ERR(trans))
+ return PTR_ERR(trans);
goto update_flags;
}
*/
btrfs_drew_read_lock(&root->snapshot_lock);
- ret = btrfs_start_delalloc_snapshot(root);
+ ret = btrfs_start_delalloc_snapshot(root, false);
if (ret)
goto out;
if (pre)
ret = clone_ordered_extent(ordered, 0, pre);
- if (post)
+ if (ret == 0 && post)
ret = clone_ordered_extent(ordered, pre + ordered->disk_num_bytes,
post);
struct btrfs_trans_handle *trans;
int ret;
- /* Can't hold an open transaction or we run the risk of deadlocking */
- ASSERT(current->journal_info == NULL ||
- current->journal_info == BTRFS_SEND_TRANS_STUB);
- if (WARN_ON(current->journal_info &&
- current->journal_info != BTRFS_SEND_TRANS_STUB))
+ /*
+ * Can't hold an open transaction or we run the risk of deadlocking,
+ * and can't either be under the context of a send operation (where
+ * current->journal_info is set to BTRFS_SEND_TRANS_STUB), as that
+ * would result in a crash when starting a transaction and does not
+ * make sense either (send is a read-only operation).
+ */
+ ASSERT(current->journal_info == NULL);
+ if (WARN_ON(current->journal_info))
return 0;
/*
return 0;
}
- ret = btrfs_start_delalloc_snapshot(root);
+ ret = btrfs_start_delalloc_snapshot(root, true);
if (ret < 0)
goto out;
btrfs_wait_ordered_extents(root, U64_MAX, 0, (u64)-1);
* inline extent's data to the page.
*/
ASSERT(key.offset > 0);
- ret = copy_inline_to_page(BTRFS_I(dst), new_key->offset,
- inline_data, size, datal,
- comp_type);
- goto out;
+ goto copy_to_page;
}
} else if (i_size_read(dst) <= datal) {
struct btrfs_file_extent_item *ei;
BTRFS_FILE_EXTENT_INLINE)
goto copy_inline_extent;
- ret = copy_inline_to_page(BTRFS_I(dst), new_key->offset,
- inline_data, size, datal, comp_type);
- goto out;
+ goto copy_to_page;
}
copy_inline_extent:
- ret = 0;
/*
* We have no extent items, or we have an extent at offset 0 which may
* or may not be inlined. All these cases are dealt the same way.
* clone. Deal with all these cases by copying the inline extent
* data into the respective page at the destination inode.
*/
- ret = copy_inline_to_page(BTRFS_I(dst), new_key->offset,
- inline_data, size, datal, comp_type);
- goto out;
+ goto copy_to_page;
}
+ /*
+ * Release path before starting a new transaction so we don't hold locks
+ * that would confuse lockdep.
+ */
btrfs_release_path(path);
/*
* If we end up here it means were copy the inline extent into a leaf
*trans_out = trans;
return ret;
+
+copy_to_page:
+ /*
+ * Release our path because we don't need it anymore and also because
+ * copy_inline_to_page() needs to reserve data and metadata, which may
+ * need to flush delalloc when we are low on available space and
+ * therefore cause a deadlock if writeback of an inline extent needs to
+ * write to the same leaf or an ordered extent completion needs to write
+ * to the same leaf.
+ */
+ btrfs_release_path(path);
+
+ ret = copy_inline_to_page(BTRFS_I(dst), new_key->offset,
+ inline_data, size, datal, comp_type);
+ goto out;
}
/**
int i;
if (root) {
- ret = btrfs_start_delalloc_snapshot(root);
+ ret = btrfs_start_delalloc_snapshot(root, false);
if (ret)
return ret;
btrfs_wait_ordered_extents(root, U64_MAX, 0, U64_MAX);
for (i = 0; i < sctx->clone_roots_cnt; i++) {
root = sctx->clone_roots[i].root;
- ret = btrfs_start_delalloc_snapshot(root);
+ ret = btrfs_start_delalloc_snapshot(root, false);
if (ret)
return ret;
btrfs_wait_ordered_extents(root, U64_MAX, 0, U64_MAX);
if (ret)
goto out;
- btrfs_update_inode(trans, root, BTRFS_I(inode));
+ ret = btrfs_update_inode(trans, root, BTRFS_I(inode));
+ if (ret)
+ goto out;
}
ref_ptr = (unsigned long)(ref_ptr + ref_struct_size) + namelen;
if (nlink != inode->i_nlink) {
set_nlink(inode, nlink);
- btrfs_update_inode(trans, root, BTRFS_I(inode));
+ ret = btrfs_update_inode(trans, root, BTRFS_I(inode));
+ if (ret)
+ goto out;
}
BTRFS_I(inode)->index_cnt = (u64)-1;
break;
if (ret == 1) {
+ ret = 0;
if (path->slots[0] == 0)
break;
path->slots[0]--;
ret = btrfs_del_item(trans, root, path);
if (ret)
- goto out;
+ break;
btrfs_release_path(path);
inode = read_one_inode(root, key.offset);
- if (!inode)
- return -EIO;
+ if (!inode) {
+ ret = -EIO;
+ break;
+ }
ret = fixup_inode_link_count(trans, root, inode);
iput(inode);
if (ret)
- goto out;
+ break;
/*
* fixup on a directory may create new entries,
*/
key.offset = (u64)-1;
}
- ret = 0;
-out:
btrfs_release_path(path);
return ret;
}
ret = btrfs_update_inode(trans, root, BTRFS_I(inode));
} else if (ret == -EEXIST) {
ret = 0;
- } else {
- BUG(); /* Logic Error */
}
iput(inode);
* begins and releases it only after writing its superblock.
*/
mutex_lock(&fs_info->tree_log_mutex);
+
+ /*
+ * The previous transaction writeout phase could have failed, and thus
+ * marked the fs in an error state. We must not commit here, as we
+ * could have updated our generation in the super_for_commit and
+ * writing the super here would result in transid mismatches. If there
+ * is an error here just bail.
+ */
+ if (test_bit(BTRFS_FS_STATE_ERROR, &fs_info->fs_state)) {
+ ret = -EIO;
+ btrfs_set_log_full_commit(trans);
+ btrfs_abort_transaction(trans, ret);
+ mutex_unlock(&fs_info->tree_log_mutex);
+ goto out_wake_log_root;
+ }
+
btrfs_set_super_log_root(fs_info->super_for_commit, log_root_start);
btrfs_set_super_log_root_level(fs_info->super_for_commit, log_root_level);
ret = write_all_supers(fs_info, 1);
* (since logging them is pointless, a link count of 0 means they
* will never be accessible).
*/
- if (btrfs_inode_in_log(inode, trans->transid) ||
+ if ((btrfs_inode_in_log(inode, trans->transid) &&
+ list_empty(&ctx->ordered_extents)) ||
inode->vfs_inode.i_nlink == 0) {
ret = BTRFS_NO_LOG_SYNC;
goto end_no_trans;
(!old_dir || old_dir->logged_trans < trans->transid))
return;
+ /*
+ * If we are doing a rename (old_dir is not NULL) from a directory that
+ * was previously logged, make sure the next log attempt on the directory
+ * is not skipped and logs the inode again. This is because the log may
+ * not currently be authoritative for a range including the old
+ * BTRFS_DIR_ITEM_KEY and BTRFS_DIR_INDEX_KEY keys, so we want to make
+ * sure after a log replay we do not end up with both the new and old
+ * dentries around (in case the inode is a directory we would have a
+ * directory with two hard links and 2 inode references for different
+ * parents). The next log attempt of old_dir will happen at
+ * btrfs_log_all_parents(), called through btrfs_log_inode_parent()
+ * below, because we have previously set inode->last_unlink_trans to the
+ * current transaction ID, either here or at btrfs_record_unlink_dir() in
+ * case inode is a directory.
+ */
+ if (old_dir)
+ old_dir->logged_trans = 0;
+
btrfs_init_log_ctx(&ctx, &inode->vfs_inode);
ctx.logging_new_name = true;
/*
/* Given hole range was invalid (outside of device) */
if (ret == -ERANGE) {
*hole_start += *hole_size;
- *hole_size = false;
+ *hole_size = 0;
return true;
}
return (u32)zone;
}
+static inline sector_t zone_start_sector(u32 zone_number,
+ struct block_device *bdev)
+{
+ return (sector_t)zone_number << ilog2(bdev_zone_sectors(bdev));
+}
+
+static inline u64 zone_start_physical(u32 zone_number,
+ struct btrfs_zoned_device_info *zone_info)
+{
+ return (u64)zone_number << zone_info->zone_size_shift;
+}
+
/*
* Emulate blkdev_report_zones() for a non-zoned device. It slices up the block
* device into static sized chunks and fake a conventional zone on each of
if (sb_zone + 1 >= zone_info->nr_zones)
continue;
- sector = sb_zone << (zone_info->zone_size_shift - SECTOR_SHIFT);
- ret = btrfs_get_dev_zones(device, sector << SECTOR_SHIFT,
+ ret = btrfs_get_dev_zones(device,
+ zone_start_physical(sb_zone, zone_info),
&zone_info->sb_zones[sb_pos],
&nr_zones);
if (ret)
if (sb_zone + 1 >= nr_zones)
return -ENOENT;
- ret = blkdev_report_zones(bdev, sb_zone << zone_sectors_shift,
+ ret = blkdev_report_zones(bdev, zone_start_sector(sb_zone, bdev),
BTRFS_NR_SB_LOG_ZONES, copy_zone_info_cb,
zones);
if (ret < 0)
return -ENOENT;
return blkdev_zone_mgmt(bdev, REQ_OP_ZONE_RESET,
- sb_zone << zone_sectors_shift,
+ zone_start_sector(sb_zone, bdev),
zone_sectors * BTRFS_NR_SB_LOG_ZONES, GFP_NOFS);
}
if (!(end <= sb_zone ||
sb_zone + BTRFS_NR_SB_LOG_ZONES <= begin)) {
have_sb = true;
- pos = ((u64)sb_zone + BTRFS_NR_SB_LOG_ZONES) << shift;
+ pos = zone_start_physical(
+ sb_zone + BTRFS_NR_SB_LOG_ZONES, zinfo);
break;
}
goto out;
}
+ if (zone.type == BLK_ZONE_TYPE_CONVENTIONAL) {
+ ret = -EIO;
+ goto out;
+ }
+
switch (zone.cond) {
case BLK_ZONE_COND_OFFLINE:
case BLK_ZONE_COND_READONLY:
spin_unlock(&trans->releasing_ebs_lock);
}
-bool btrfs_use_zone_append(struct btrfs_inode *inode, struct extent_map *em)
+bool btrfs_use_zone_append(struct btrfs_inode *inode, u64 start)
{
struct btrfs_fs_info *fs_info = inode->root->fs_info;
struct btrfs_block_group *cache;
if (!is_data_inode(&inode->vfs_inode))
return false;
- cache = btrfs_lookup_block_group(fs_info, em->block_start);
+ cache = btrfs_lookup_block_group(fs_info, start);
ASSERT(cache);
if (!cache)
return false;
void btrfs_redirty_list_add(struct btrfs_transaction *trans,
struct extent_buffer *eb);
void btrfs_free_redirty_list(struct btrfs_transaction *trans);
-bool btrfs_use_zone_append(struct btrfs_inode *inode, struct extent_map *em);
+bool btrfs_use_zone_append(struct btrfs_inode *inode, u64 start);
void btrfs_record_physical_zoned(struct inode *inode, u64 file_offset,
struct bio *bio);
void btrfs_rewrite_logical_zoned(struct btrfs_ordered_extent *ordered);
struct extent_buffer *eb) { }
static inline void btrfs_free_redirty_list(struct btrfs_transaction *trans) { }
-static inline bool btrfs_use_zone_append(struct btrfs_inode *inode,
- struct extent_map *em)
+static inline bool btrfs_use_zone_append(struct btrfs_inode *inode, u64 start)
{
return false;
}
} __packed;
/*
- * Dump full key (32 byte encrypt/decrypt keys instead of 16 bytes)
- * is needed if GCM256 (stronger encryption) negotiated
+ * Dump variable-sized keys
*/
struct smb3_full_key_debug_info {
- __u64 Suid;
+ /* INPUT: size of userspace buffer */
+ __u32 in_size;
+
+ /*
+ * INPUT: 0 for current user, otherwise session to dump
+ * OUTPUT: session id that was dumped
+ */
+ __u64 session_id;
__u16 cipher_type;
- __u8 auth_key[16]; /* SMB2_NTLMV2_SESSKEY_SIZE */
- __u8 smb3encryptionkey[32]; /* SMB3_ENC_DEC_KEY_SIZE */
- __u8 smb3decryptionkey[32]; /* SMB3_ENC_DEC_KEY_SIZE */
+ __u8 session_key_length;
+ __u8 server_in_key_length;
+ __u8 server_out_key_length;
+ __u8 data[];
+ /*
+ * return this struct with the keys appended at the end:
+ * __u8 session_key[session_key_length];
+ * __u8 server_in_key[server_in_key_length];
+ * __u8 server_out_key[server_out_key_length];
+ */
} __packed;
struct smb3_notify {
struct workqueue_struct *decrypt_wq;
struct workqueue_struct *fileinfo_put_wq;
struct workqueue_struct *cifsoplockd_wq;
-struct workqueue_struct *deferredclose_wq;
+struct workqueue_struct *deferredclose_wq;
__u32 cifs_lock_secret;
/*
struct work_struct oplock_break; /* work for oplock breaks */
struct work_struct put; /* work for the final part of _put */
struct delayed_work deferred;
- bool oplock_break_received; /* Flag to indicate oplock break */
- bool deferred_scheduled;
+ bool deferred_close_scheduled; /* Flag to indicate close is scheduled */
};
struct cifs_io_parms {
struct inode vfs_inode;
struct list_head deferred_closes; /* list of deferred closes */
spinlock_t deferred_lock; /* protection on deferred list */
+ bool lease_granted; /* Flag to indicate whether lease or oplock is granted. */
};
static inline struct cifsInodeInfo *
#define SMB3_SIGN_KEY_SIZE (16)
/*
- * Size of the smb3 encryption/decryption keys
+ * Size of the smb3 encryption/decryption key storage.
+ * This size is big enough to store any cipher key types.
*/
#define SMB3_ENC_DEC_KEY_SIZE (32)
cfile->dentry = dget(dentry);
cfile->f_flags = file->f_flags;
cfile->invalidHandle = false;
- cfile->oplock_break_received = false;
- cfile->deferred_scheduled = false;
+ cfile->deferred_close_scheduled = false;
cfile->tlink = cifs_get_tlink(tlink);
INIT_WORK(&cfile->oplock_break, cifs_oplock_break);
INIT_WORK(&cfile->put, cifsFileInfo_put_work);
file->f_op = &cifs_file_direct_ops;
}
- spin_lock(&CIFS_I(inode)->deferred_lock);
/* Get the cached handle as SMB2 close is deferred */
rc = cifs_get_readable_path(tcon, full_path, &cfile);
if (rc == 0) {
if (file->f_flags == cfile->f_flags) {
file->private_data = cfile;
+ spin_lock(&CIFS_I(inode)->deferred_lock);
cifs_del_deferred_close(cfile);
spin_unlock(&CIFS_I(inode)->deferred_lock);
goto out;
} else {
- spin_unlock(&CIFS_I(inode)->deferred_lock);
_cifsFileInfo_put(cfile, true, false);
}
- } else {
- spin_unlock(&CIFS_I(inode)->deferred_lock);
}
if (server->oplocks)
struct cifsFileInfo, deferred.work);
spin_lock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
- if (!cfile->deferred_scheduled) {
- spin_unlock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
- return;
- }
cifs_del_deferred_close(cfile);
- cfile->deferred_scheduled = false;
+ cfile->deferred_close_scheduled = false;
spin_unlock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
_cifsFileInfo_put(cfile, true, false);
}
file->private_data = NULL;
dclose = kmalloc(sizeof(struct cifs_deferred_close), GFP_KERNEL);
if ((cinode->oplock == CIFS_CACHE_RHW_FLG) &&
+ cinode->lease_granted &&
dclose) {
if (test_bit(CIFS_INO_MODIFIED_ATTR, &cinode->flags))
inode->i_ctime = inode->i_mtime = current_time(inode);
spin_lock(&cinode->deferred_lock);
cifs_add_deferred_close(cfile, dclose);
- if (cfile->deferred_scheduled) {
- mod_delayed_work(deferredclose_wq,
- &cfile->deferred, cifs_sb->ctx->acregmax);
+ if (cfile->deferred_close_scheduled &&
+ delayed_work_pending(&cfile->deferred)) {
+ /*
+ * If there is no pending work, mod_delayed_work queues new work.
+ * So, Increase the ref count to avoid use-after-free.
+ */
+ if (!mod_delayed_work(deferredclose_wq,
+ &cfile->deferred, cifs_sb->ctx->acregmax))
+ cifsFileInfo_get(cfile);
} else {
/* Deferred close for files */
queue_delayed_work(deferredclose_wq,
&cfile->deferred, cifs_sb->ctx->acregmax);
- cfile->deferred_scheduled = true;
+ cfile->deferred_close_scheduled = true;
spin_unlock(&cinode->deferred_lock);
return 0;
}
if (fsuid_only && !uid_eq(open_file->uid, current_fsuid()))
continue;
if (OPEN_FMODE(open_file->f_flags) & FMODE_READ) {
- if ((!open_file->invalidHandle) &&
- (!open_file->oplock_break_received)) {
+ if ((!open_file->invalidHandle)) {
/* found a good file */
/* lock it so it will not be closed on us */
cifsFileInfo_get(open_file);
}
/*
* When oplock break is received and there are no active
- * file handles but cached, then set the flag oplock_break_received.
+ * file handles but cached, then schedule deferred close immediately.
* So, new open will not use cached handle.
*/
spin_lock(&CIFS_I(inode)->deferred_lock);
is_deferred = cifs_is_deferred_close(cfile, &dclose);
- if (is_deferred && cfile->deferred_scheduled) {
- cfile->oplock_break_received = true;
- mod_delayed_work(deferredclose_wq, &cfile->deferred, 0);
+ if (is_deferred &&
+ cfile->deferred_close_scheduled &&
+ delayed_work_pending(&cfile->deferred)) {
+ /*
+ * If there is no pending work, mod_delayed_work queues new work.
+ * So, Increase the ref count to avoid use-after-free.
+ */
+ if (!mod_delayed_work(deferredclose_wq, &cfile->deferred, 0))
+ cifsFileInfo_get(cfile);
}
spin_unlock(&CIFS_I(inode)->deferred_lock);
_cifsFileInfo_put(cfile, false /* do not wait for ourself */, false);
goto cifs_parse_mount_err;
}
ctx->max_channels = result.uint_32;
+ /* If more than one channel requested ... they want multichan */
+ if (result.uint_32 > 1)
+ ctx->multichannel = true;
break;
case Opt_handletimeout:
ctx->handle_timeout = result.uint_32;
/* if iocharset not set then load_nls_default
* is used by caller
*/
- cifs_dbg(FYI, "iocharset set to %s\n", ctx->iocharset);
+ cifs_dbg(FYI, "iocharset set to %s\n", ctx->iocharset);
break;
case Opt_netbiosname:
memset(ctx->source_rfc1001_name, 0x20,
#include "cifsfs.h"
#include "cifs_ioctl.h"
#include "smb2proto.h"
+#include "smb2glob.h"
#include <linux/btrfs.h>
static long cifs_ioctl_query_info(unsigned int xid, struct file *filep,
return 0;
}
-static int cifs_dump_full_key(struct cifs_tcon *tcon, unsigned long arg)
+static int cifs_dump_full_key(struct cifs_tcon *tcon, struct smb3_full_key_debug_info __user *in)
{
- struct smb3_full_key_debug_info pfull_key_inf;
- __u64 suid;
- struct list_head *tmp;
+ struct smb3_full_key_debug_info out;
struct cifs_ses *ses;
+ int rc = 0;
bool found = false;
+ u8 __user *end;
- if (!smb3_encryption_required(tcon))
- return -EOPNOTSUPP;
+ if (!smb3_encryption_required(tcon)) {
+ rc = -EOPNOTSUPP;
+ goto out;
+ }
+
+ /* copy user input into our output buffer */
+ if (copy_from_user(&out, in, sizeof(out))) {
+ rc = -EINVAL;
+ goto out;
+ }
+
+ if (!out.session_id) {
+ /* if ses id is 0, use current user session */
+ ses = tcon->ses;
+ } else {
+ /* otherwise if a session id is given, look for it in all our sessions */
+ struct cifs_ses *ses_it = NULL;
+ struct TCP_Server_Info *server_it = NULL;
- ses = tcon->ses; /* default to user id for current user */
- if (get_user(suid, (__u64 __user *)arg))
- suid = 0;
- if (suid) {
- /* search to see if there is a session with a matching SMB UID */
spin_lock(&cifs_tcp_ses_lock);
- list_for_each(tmp, &tcon->ses->server->smb_ses_list) {
- ses = list_entry(tmp, struct cifs_ses, smb_ses_list);
- if (ses->Suid == suid) {
- found = true;
- break;
+ list_for_each_entry(server_it, &cifs_tcp_ses_list, tcp_ses_list) {
+ list_for_each_entry(ses_it, &server_it->smb_ses_list, smb_ses_list) {
+ if (ses_it->Suid == out.session_id) {
+ ses = ses_it;
+ /*
+ * since we are using the session outside the crit
+ * section, we need to make sure it won't be released
+ * so increment its refcount
+ */
+ ses->ses_count++;
+ found = true;
+ goto search_end;
+ }
}
}
+search_end:
spin_unlock(&cifs_tcp_ses_lock);
- if (found == false)
- return -EINVAL;
- } /* else uses default user's SMB UID (ie current user) */
-
- pfull_key_inf.cipher_type = le16_to_cpu(ses->server->cipher_type);
- pfull_key_inf.Suid = ses->Suid;
- memcpy(pfull_key_inf.auth_key, ses->auth_key.response,
- 16 /* SMB2_NTLMV2_SESSKEY_SIZE */);
- memcpy(pfull_key_inf.smb3decryptionkey, ses->smb3decryptionkey,
- 32 /* SMB3_ENC_DEC_KEY_SIZE */);
- memcpy(pfull_key_inf.smb3encryptionkey,
- ses->smb3encryptionkey, 32 /* SMB3_ENC_DEC_KEY_SIZE */);
- if (copy_to_user((void __user *)arg, &pfull_key_inf,
- sizeof(struct smb3_full_key_debug_info)))
- return -EFAULT;
+ if (!found) {
+ rc = -ENOENT;
+ goto out;
+ }
+ }
- return 0;
+ switch (ses->server->cipher_type) {
+ case SMB2_ENCRYPTION_AES128_CCM:
+ case SMB2_ENCRYPTION_AES128_GCM:
+ out.session_key_length = CIFS_SESS_KEY_SIZE;
+ out.server_in_key_length = out.server_out_key_length = SMB3_GCM128_CRYPTKEY_SIZE;
+ break;
+ case SMB2_ENCRYPTION_AES256_CCM:
+ case SMB2_ENCRYPTION_AES256_GCM:
+ out.session_key_length = CIFS_SESS_KEY_SIZE;
+ out.server_in_key_length = out.server_out_key_length = SMB3_GCM256_CRYPTKEY_SIZE;
+ break;
+ default:
+ rc = -EOPNOTSUPP;
+ goto out;
+ }
+
+ /* check if user buffer is big enough to store all the keys */
+ if (out.in_size < sizeof(out) + out.session_key_length + out.server_in_key_length
+ + out.server_out_key_length) {
+ rc = -ENOBUFS;
+ goto out;
+ }
+
+ out.session_id = ses->Suid;
+ out.cipher_type = le16_to_cpu(ses->server->cipher_type);
+
+ /* overwrite user input with our output */
+ if (copy_to_user(in, &out, sizeof(out))) {
+ rc = -EINVAL;
+ goto out;
+ }
+
+ /* append all the keys at the end of the user buffer */
+ end = in->data;
+ if (copy_to_user(end, ses->auth_key.response, out.session_key_length)) {
+ rc = -EINVAL;
+ goto out;
+ }
+ end += out.session_key_length;
+
+ if (copy_to_user(end, ses->smb3encryptionkey, out.server_in_key_length)) {
+ rc = -EINVAL;
+ goto out;
+ }
+ end += out.server_in_key_length;
+
+ if (copy_to_user(end, ses->smb3decryptionkey, out.server_out_key_length)) {
+ rc = -EINVAL;
+ goto out;
+ }
+
+out:
+ if (found)
+ cifs_put_smb_ses(ses);
+ return rc;
}
long cifs_ioctl(struct file *filep, unsigned int command, unsigned long arg)
rc = -EOPNOTSUPP;
break;
case CIFS_DUMP_KEY:
+ /*
+ * Dump encryption keys. This is an old ioctl that only
+ * handles AES-128-{CCM,GCM}.
+ */
if (pSMBFile == NULL)
break;
if (!capable(CAP_SYS_ADMIN)) {
else
rc = 0;
break;
- /*
- * Dump full key (32 bytes instead of 16 bytes) is
- * needed if GCM256 (stronger encryption) negotiated
- */
case CIFS_DUMP_FULL_KEY:
+ /*
+ * Dump encryption keys (handles any key sizes)
+ */
if (pSMBFile == NULL)
break;
if (!capable(CAP_SYS_ADMIN)) {
break;
}
tcon = tlink_tcon(pSMBFile->tlink);
- rc = cifs_dump_full_key(tcon, arg);
-
+ rc = cifs_dump_full_key(tcon, (void __user *)arg);
break;
case CIFS_IOC_NOTIFY:
if (!S_ISDIR(inode->i_mode)) {
spin_unlock(&tlink_tcon(open->tlink)->open_file_lock);
}
+/*
+ * Critical section which runs after acquiring deferred_lock.
+ * As there is no reference count on cifs_deferred_close, pdclose
+ * should not be used outside deferred_lock.
+ */
bool
cifs_is_deferred_close(struct cifsFileInfo *cfile, struct cifs_deferred_close **pdclose)
{
return false;
}
+/*
+ * Critical section which runs after acquiring deferred_lock.
+ */
void
cifs_add_deferred_close(struct cifsFileInfo *cfile, struct cifs_deferred_close *dclose)
{
list_add_tail(&dclose->dlist, &CIFS_I(d_inode(cfile->dentry))->deferred_closes);
}
+/*
+ * Critical section which runs after acquiring deferred_lock.
+ */
void
cifs_del_deferred_close(struct cifsFileInfo *cfile)
{
cifs_close_all_deferred_files(struct cifs_tcon *tcon)
{
struct cifsFileInfo *cfile;
- struct cifsInodeInfo *cinode;
struct list_head *tmp;
spin_lock(&tcon->open_file_lock);
list_for_each(tmp, &tcon->openFileList) {
cfile = list_entry(tmp, struct cifsFileInfo, tlist);
- cinode = CIFS_I(d_inode(cfile->dentry));
- if (delayed_work_pending(&cfile->deferred))
- mod_delayed_work(deferredclose_wq, &cfile->deferred, 0);
+ if (delayed_work_pending(&cfile->deferred)) {
+ /*
+ * If there is no pending work, mod_delayed_work queues new work.
+ * So, Increase the ref count to avoid use-after-free.
+ */
+ if (!mod_delayed_work(deferredclose_wq, &cfile->deferred, 0))
+ cifsFileInfo_get(cfile);
+ }
}
spin_unlock(&tcon->open_file_lock);
}
return 0;
}
+ if (!(ses->server->capabilities & SMB2_GLOBAL_CAP_MULTI_CHANNEL)) {
+ cifs_dbg(VFS, "server %s does not support multichannel\n", ses->server->hostname);
+ ses->chan_max = 1;
+ return 0;
+ }
+
/*
* Make a copy of the iface list at the time and use that
* instead so as to not hold the iface spinlock for opening
cpu_to_le32(min_t(u32, len, tcon->max_bytes_chunk));
/* Request server copy to target from src identified by key */
+ kfree(retbuf);
+ retbuf = NULL;
rc = SMB2_ioctl(xid, tcon, trgtfile->fid.persistent_fid,
trgtfile->fid.volatile_fid, FSCTL_SRV_COPYCHUNK_WRITE,
true /* is_fsctl */, (char *)pcchunk,
unsigned int epoch, bool *purge_cache)
{
oplock &= 0xFF;
+ cinode->lease_granted = false;
if (oplock == SMB2_OPLOCK_LEVEL_NOCHANGE)
return;
if (oplock == SMB2_OPLOCK_LEVEL_BATCH) {
unsigned int new_oplock = 0;
oplock &= 0xFF;
+ cinode->lease_granted = true;
if (oplock == SMB2_OPLOCK_LEVEL_NOCHANGE)
return;
req->SecurityMode = 0;
req->Capabilities = cpu_to_le32(server->vals->req_capabilities);
+ if (ses->chan_max > 1)
+ req->Capabilities |= cpu_to_le32(SMB2_GLOBAL_CAP_MULTI_CHANNEL);
/* ClientGUID must be zero for SMB2.02 dialect */
if (server->vals->protocol_id == SMB20_PROT_ID)
/* Internal types */
server->capabilities |= SMB2_NT_FIND | SMB2_LARGE_FILES;
+ /*
+ * SMB3.0 supports only 1 cipher and doesn't have a encryption neg context
+ * Set the cipher type manually.
+ */
+ if (server->dialect == SMB30_PROT_ID && (server->capabilities & SMB2_GLOBAL_CAP_ENCRYPTION))
+ server->cipher_type = SMB2_ENCRYPTION_AES128_CCM;
+
security_blob = smb2_get_data_area_len(&blob_offset, &blob_length,
(struct smb2_sync_hdr *)rsp);
/*
pneg_inbuf->Capabilities =
cpu_to_le32(server->vals->req_capabilities);
+ if (tcon->ses->chan_max > 1)
+ pneg_inbuf->Capabilities |= cpu_to_le32(SMB2_GLOBAL_CAP_MULTI_CHANNEL);
+
memcpy(pneg_inbuf->Guid, server->client_guid,
SMB2_CLIENT_GUID_SIZE);
* Related requests use info from previous read request
* in chain.
*/
- shdr->SessionId = 0xFFFFFFFF;
+ shdr->SessionId = 0xFFFFFFFFFFFFFFFF;
shdr->TreeId = 0xFFFFFFFF;
- req->PersistentFileId = 0xFFFFFFFF;
- req->VolatileFileId = 0xFFFFFFFF;
+ req->PersistentFileId = 0xFFFFFFFFFFFFFFFF;
+ req->VolatileFileId = 0xFFFFFFFFFFFFFFFF;
}
}
if (remaining_bytes > io_parms->length)
#include <linux/tracepoint.h>
+/*
+ * Please use this 3-part article as a reference for writing new tracepoints:
+ * https://lwn.net/Articles/379903/
+ */
+
/* For logging errors in read or write */
DECLARE_EVENT_CLASS(smb3_rw_err_class,
TP_PROTO(unsigned int xid,
TP_ARGS(xid, func_name, rc),
TP_STRUCT__entry(
__field(unsigned int, xid)
- __field(const char *, func_name)
+ __string(func_name, func_name)
__field(int, rc)
),
TP_fast_assign(
__entry->xid = xid;
- __entry->func_name = func_name;
+ __assign_str(func_name, func_name);
__entry->rc = rc;
),
TP_printk("\t%s: xid=%u rc=%d",
- __entry->func_name, __entry->xid, __entry->rc)
+ __get_str(func_name), __entry->xid, __entry->rc)
)
#define DEFINE_SMB3_EXIT_ERR_EVENT(name) \
TP_ARGS(xid, func_name),
TP_STRUCT__entry(
__field(unsigned int, xid)
- __field(const char *, func_name)
+ __string(func_name, func_name)
),
TP_fast_assign(
__entry->xid = xid;
- __entry->func_name = func_name;
+ __assign_str(func_name, func_name);
),
TP_printk("\t%s: xid=%u",
- __entry->func_name, __entry->xid)
+ __get_str(func_name), __entry->xid)
)
#define DEFINE_SMB3_ENTER_EXIT_EVENT(name) \
TP_STRUCT__entry(
__field(__u64, currmid)
__field(__u64, conn_id)
- __field(char *, hostname)
+ __string(hostname, hostname)
),
TP_fast_assign(
__entry->currmid = currmid;
__entry->conn_id = conn_id;
- __entry->hostname = hostname;
+ __assign_str(hostname, hostname);
),
TP_printk("conn_id=0x%llx server=%s current_mid=%llu",
__entry->conn_id,
- __entry->hostname,
+ __get_str(hostname),
__entry->currmid)
)
TP_STRUCT__entry(
__field(__u64, currmid)
__field(__u64, conn_id)
- __field(char *, hostname)
+ __string(hostname, hostname)
__field(int, credits)
__field(int, credits_to_add)
__field(int, in_flight)
TP_fast_assign(
__entry->currmid = currmid;
__entry->conn_id = conn_id;
- __entry->hostname = hostname;
+ __assign_str(hostname, hostname);
__entry->credits = credits;
__entry->credits_to_add = credits_to_add;
__entry->in_flight = in_flight;
TP_printk("conn_id=0x%llx server=%s current_mid=%llu "
"credits=%d credit_change=%d in_flight=%d",
__entry->conn_id,
- __entry->hostname,
+ __get_str(hostname),
__entry->currmid,
__entry->credits,
__entry->credits_to_add,
* but then we need to teach dump_write() to restart and clear
* TIF_SIGPENDING.
*/
- return signal_pending(current);
+ return fatal_signal_pending(current) || freezing(current);
}
static void wait_for_dump_helpers(struct file *file)
struct exceptional_entry_key key;
};
+/**
+ * enum dax_wake_mode: waitqueue wakeup behaviour
+ * @WAKE_ALL: wake all waiters in the waitqueue
+ * @WAKE_NEXT: wake only the first waiter in the waitqueue
+ */
+enum dax_wake_mode {
+ WAKE_ALL,
+ WAKE_NEXT,
+};
+
static wait_queue_head_t *dax_entry_waitqueue(struct xa_state *xas,
void *entry, struct exceptional_entry_key *key)
{
* The important information it's conveying is whether the entry at
* this index used to be a PMD entry.
*/
-static void dax_wake_entry(struct xa_state *xas, void *entry, bool wake_all)
+static void dax_wake_entry(struct xa_state *xas, void *entry,
+ enum dax_wake_mode mode)
{
struct exceptional_entry_key key;
wait_queue_head_t *wq;
* must be in the waitqueue and the following check will see them.
*/
if (waitqueue_active(wq))
- __wake_up(wq, TASK_NORMAL, wake_all ? 0 : 1, &key);
+ __wake_up(wq, TASK_NORMAL, mode == WAKE_ALL ? 0 : 1, &key);
}
/*
finish_wait(wq, &ewait.wait);
}
-static void put_unlocked_entry(struct xa_state *xas, void *entry)
+static void put_unlocked_entry(struct xa_state *xas, void *entry,
+ enum dax_wake_mode mode)
{
- /* If we were the only waiter woken, wake the next one */
if (entry && !dax_is_conflict(entry))
- dax_wake_entry(xas, entry, false);
+ dax_wake_entry(xas, entry, mode);
}
/*
old = xas_store(xas, entry);
xas_unlock_irq(xas);
BUG_ON(!dax_is_locked(old));
- dax_wake_entry(xas, entry, false);
+ dax_wake_entry(xas, entry, WAKE_NEXT);
}
/*
dax_disassociate_entry(entry, mapping, false);
xas_store(xas, NULL); /* undo the PMD join */
- dax_wake_entry(xas, entry, true);
+ dax_wake_entry(xas, entry, WAKE_ALL);
mapping->nrpages -= PG_PMD_NR;
entry = NULL;
xas_set(xas, index);
entry = get_unlocked_entry(&xas, 0);
if (entry)
page = dax_busy_page(entry);
- put_unlocked_entry(&xas, entry);
+ put_unlocked_entry(&xas, entry, WAKE_NEXT);
if (page)
break;
if (++scanned % XA_CHECK_SCHED)
mapping->nrpages -= 1UL << dax_entry_order(entry);
ret = 1;
out:
- put_unlocked_entry(&xas, entry);
+ put_unlocked_entry(&xas, entry, WAKE_ALL);
xas_unlock_irq(&xas);
return ret;
}
xas_lock_irq(xas);
xas_store(xas, entry);
xas_clear_mark(xas, PAGECACHE_TAG_DIRTY);
- dax_wake_entry(xas, entry, false);
+ dax_wake_entry(xas, entry, WAKE_NEXT);
trace_dax_writeback_one(mapping->host, index, count);
return ret;
put_unlocked:
- put_unlocked_entry(xas, entry);
+ put_unlocked_entry(xas, entry, WAKE_NEXT);
return ret;
}
/* Did we race with someone splitting entry or so? */
if (!entry || dax_is_conflict(entry) ||
(order == 0 && !dax_is_pte_entry(entry))) {
- put_unlocked_entry(&xas, entry);
+ put_unlocked_entry(&xas, entry, WAKE_NEXT);
xas_unlock_irq(&xas);
trace_dax_insert_pfn_mkwrite_no_entry(mapping->host, vmf,
VM_FAULT_NOPAGE);
copy[copy_len] = '\n';
- ret = simple_read_from_buffer(user_buf, count, ppos, copy, copy_len);
+ ret = simple_read_from_buffer(user_buf, count, ppos, copy, len);
kfree(copy);
return ret;
static int debugfs_setattr(struct user_namespace *mnt_userns,
struct dentry *dentry, struct iattr *ia)
{
- int ret = security_locked_down(LOCKDOWN_DEBUGFS);
+ int ret;
- if (ret && (ia->ia_valid & (ATTR_MODE | ATTR_UID | ATTR_GID)))
- return ret;
+ if (ia->ia_valid & (ATTR_MODE | ATTR_UID | ATTR_GID)) {
+ ret = security_locked_down(LOCKDOWN_DEBUGFS);
+ if (ret)
+ return ret;
+ }
return simple_setattr(&init_user_ns, dentry, ia);
}
struct extent_crypt_result ecr;
int rc = 0;
- if (!crypt_stat || !crypt_stat->tfm
- || !(crypt_stat->flags & ECRYPTFS_STRUCT_INITIALIZED))
- return -EINVAL;
-
if (unlikely(ecryptfs_verbosity > 0)) {
ecryptfs_printk(KERN_DEBUG, "Key size [%zd]; key:\n",
crypt_stat->key_size);
lcn = m->lcn + 1;
if (m->compressedlcs)
goto out;
- if (lcn == initial_lcn)
- goto err_bonus_cblkcnt;
err = z_erofs_load_cluster_from_disk(m, lcn);
if (err)
return err;
+ /*
+ * If the 1st NONHEAD lcluster has already been handled initially w/o
+ * valid compressedlcs, which means at least it mustn't be CBLKCNT, or
+ * an internal implemenatation error is detected.
+ *
+ * The following code can also handle it properly anyway, but let's
+ * BUG_ON in the debugging mode only for developers to notice that.
+ */
+ DBG_BUGON(lcn == initial_lcn &&
+ m->type == Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD);
+
switch (m->type) {
+ case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:
+ case Z_EROFS_VLE_CLUSTER_TYPE_HEAD:
+ /*
+ * if the 1st NONHEAD lcluster is actually PLAIN or HEAD type
+ * rather than CBLKCNT, it's a 1 lcluster-sized pcluster.
+ */
+ m->compressedlcs = 1;
+ break;
case Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD:
if (m->delta[0] != 1)
goto err_bonus_cblkcnt;
ext4_ext_mark_unwritten(ex2);
err = ext4_ext_insert_extent(handle, inode, ppath, &newex, flags);
- if (err == -ENOSPC && (EXT4_EXT_MAY_ZEROOUT & split_flag)) {
+ if (err != -ENOSPC && err != -EDQUOT)
+ goto out;
+
+ if (EXT4_EXT_MAY_ZEROOUT & split_flag) {
if (split_flag & (EXT4_EXT_DATA_VALID1|EXT4_EXT_DATA_VALID2)) {
if (split_flag & EXT4_EXT_DATA_VALID1) {
err = ext4_ext_zeroout(inode, ex2);
ext4_ext_pblock(&orig_ex));
}
- if (err)
- goto fix_extent_len;
- /* update the extent length and mark as initialized */
- ex->ee_len = cpu_to_le16(ee_len);
- ext4_ext_try_to_merge(handle, inode, path, ex);
- err = ext4_ext_dirty(handle, inode, path + path->p_depth);
- if (err)
- goto fix_extent_len;
-
- /* update extent status tree */
- err = ext4_zeroout_es(inode, &zero_ex);
-
- goto out;
- } else if (err)
- goto fix_extent_len;
-
-out:
- ext4_ext_show_leaf(inode, path);
- return err;
+ if (!err) {
+ /* update the extent length and mark as initialized */
+ ex->ee_len = cpu_to_le16(ee_len);
+ ext4_ext_try_to_merge(handle, inode, path, ex);
+ err = ext4_ext_dirty(handle, inode, path + path->p_depth);
+ if (!err)
+ /* update extent status tree */
+ err = ext4_zeroout_es(inode, &zero_ex);
+ /* If we failed at this point, we don't know in which
+ * state the extent tree exactly is so don't try to fix
+ * length of the original extent as it may do even more
+ * damage.
+ */
+ goto out;
+ }
+ }
fix_extent_len:
ex->ee_len = orig_ex.ee_len;
*/
ext4_ext_dirty(handle, inode, path + path->p_depth);
return err;
+out:
+ ext4_ext_show_leaf(inode, path);
+ return err;
}
/*
};
static inline void tl_to_darg(struct dentry_info_args *darg,
- struct ext4_fc_tl *tl)
+ struct ext4_fc_tl *tl, u8 *val)
{
- struct ext4_fc_dentry_info *fcd;
+ struct ext4_fc_dentry_info fcd;
- fcd = (struct ext4_fc_dentry_info *)ext4_fc_tag_val(tl);
+ memcpy(&fcd, val, sizeof(fcd));
- darg->parent_ino = le32_to_cpu(fcd->fc_parent_ino);
- darg->ino = le32_to_cpu(fcd->fc_ino);
- darg->dname = fcd->fc_dname;
- darg->dname_len = ext4_fc_tag_len(tl) -
- sizeof(struct ext4_fc_dentry_info);
+ darg->parent_ino = le32_to_cpu(fcd.fc_parent_ino);
+ darg->ino = le32_to_cpu(fcd.fc_ino);
+ darg->dname = val + offsetof(struct ext4_fc_dentry_info, fc_dname);
+ darg->dname_len = le16_to_cpu(tl->fc_len) -
+ sizeof(struct ext4_fc_dentry_info);
}
/* Unlink replay function */
-static int ext4_fc_replay_unlink(struct super_block *sb, struct ext4_fc_tl *tl)
+static int ext4_fc_replay_unlink(struct super_block *sb, struct ext4_fc_tl *tl,
+ u8 *val)
{
struct inode *inode, *old_parent;
struct qstr entry;
struct dentry_info_args darg;
int ret = 0;
- tl_to_darg(&darg, tl);
+ tl_to_darg(&darg, tl, val);
trace_ext4_fc_replay(sb, EXT4_FC_TAG_UNLINK, darg.ino,
darg.parent_ino, darg.dname_len);
}
/* Link replay function */
-static int ext4_fc_replay_link(struct super_block *sb, struct ext4_fc_tl *tl)
+static int ext4_fc_replay_link(struct super_block *sb, struct ext4_fc_tl *tl,
+ u8 *val)
{
struct inode *inode;
struct dentry_info_args darg;
int ret = 0;
- tl_to_darg(&darg, tl);
+ tl_to_darg(&darg, tl, val);
trace_ext4_fc_replay(sb, EXT4_FC_TAG_LINK, darg.ino,
darg.parent_ino, darg.dname_len);
/*
* Inode replay function
*/
-static int ext4_fc_replay_inode(struct super_block *sb, struct ext4_fc_tl *tl)
+static int ext4_fc_replay_inode(struct super_block *sb, struct ext4_fc_tl *tl,
+ u8 *val)
{
- struct ext4_fc_inode *fc_inode;
+ struct ext4_fc_inode fc_inode;
struct ext4_inode *raw_inode;
struct ext4_inode *raw_fc_inode;
struct inode *inode = NULL;
int inode_len, ino, ret, tag = le16_to_cpu(tl->fc_tag);
struct ext4_extent_header *eh;
- fc_inode = (struct ext4_fc_inode *)ext4_fc_tag_val(tl);
+ memcpy(&fc_inode, val, sizeof(fc_inode));
- ino = le32_to_cpu(fc_inode->fc_ino);
+ ino = le32_to_cpu(fc_inode.fc_ino);
trace_ext4_fc_replay(sb, tag, ino, 0, 0);
inode = ext4_iget(sb, ino, EXT4_IGET_NORMAL);
ext4_fc_record_modified_inode(sb, ino);
- raw_fc_inode = (struct ext4_inode *)fc_inode->fc_raw_inode;
+ raw_fc_inode = (struct ext4_inode *)
+ (val + offsetof(struct ext4_fc_inode, fc_raw_inode));
ret = ext4_get_fc_inode_loc(sb, ino, &iloc);
if (ret)
goto out;
- inode_len = ext4_fc_tag_len(tl) - sizeof(struct ext4_fc_inode);
+ inode_len = le16_to_cpu(tl->fc_len) - sizeof(struct ext4_fc_inode);
raw_inode = ext4_raw_inode(&iloc);
memcpy(raw_inode, raw_fc_inode, offsetof(struct ext4_inode, i_block));
* inode for which we are trying to create a dentry here, should already have
* been replayed before we start here.
*/
-static int ext4_fc_replay_create(struct super_block *sb, struct ext4_fc_tl *tl)
+static int ext4_fc_replay_create(struct super_block *sb, struct ext4_fc_tl *tl,
+ u8 *val)
{
int ret = 0;
struct inode *inode = NULL;
struct inode *dir = NULL;
struct dentry_info_args darg;
- tl_to_darg(&darg, tl);
+ tl_to_darg(&darg, tl, val);
trace_ext4_fc_replay(sb, EXT4_FC_TAG_CREAT, darg.ino,
darg.parent_ino, darg.dname_len);
/* Replay add range tag */
static int ext4_fc_replay_add_range(struct super_block *sb,
- struct ext4_fc_tl *tl)
+ struct ext4_fc_tl *tl, u8 *val)
{
- struct ext4_fc_add_range *fc_add_ex;
+ struct ext4_fc_add_range fc_add_ex;
struct ext4_extent newex, *ex;
struct inode *inode;
ext4_lblk_t start, cur;
struct ext4_ext_path *path = NULL;
int ret;
- fc_add_ex = (struct ext4_fc_add_range *)ext4_fc_tag_val(tl);
- ex = (struct ext4_extent *)&fc_add_ex->fc_ex;
+ memcpy(&fc_add_ex, val, sizeof(fc_add_ex));
+ ex = (struct ext4_extent *)&fc_add_ex.fc_ex;
trace_ext4_fc_replay(sb, EXT4_FC_TAG_ADD_RANGE,
- le32_to_cpu(fc_add_ex->fc_ino), le32_to_cpu(ex->ee_block),
+ le32_to_cpu(fc_add_ex.fc_ino), le32_to_cpu(ex->ee_block),
ext4_ext_get_actual_len(ex));
- inode = ext4_iget(sb, le32_to_cpu(fc_add_ex->fc_ino),
- EXT4_IGET_NORMAL);
+ inode = ext4_iget(sb, le32_to_cpu(fc_add_ex.fc_ino), EXT4_IGET_NORMAL);
if (IS_ERR(inode)) {
jbd_debug(1, "Inode not found.");
return 0;
/* Replay DEL_RANGE tag */
static int
-ext4_fc_replay_del_range(struct super_block *sb, struct ext4_fc_tl *tl)
+ext4_fc_replay_del_range(struct super_block *sb, struct ext4_fc_tl *tl,
+ u8 *val)
{
struct inode *inode;
- struct ext4_fc_del_range *lrange;
+ struct ext4_fc_del_range lrange;
struct ext4_map_blocks map;
ext4_lblk_t cur, remaining;
int ret;
- lrange = (struct ext4_fc_del_range *)ext4_fc_tag_val(tl);
- cur = le32_to_cpu(lrange->fc_lblk);
- remaining = le32_to_cpu(lrange->fc_len);
+ memcpy(&lrange, val, sizeof(lrange));
+ cur = le32_to_cpu(lrange.fc_lblk);
+ remaining = le32_to_cpu(lrange.fc_len);
trace_ext4_fc_replay(sb, EXT4_FC_TAG_DEL_RANGE,
- le32_to_cpu(lrange->fc_ino), cur, remaining);
+ le32_to_cpu(lrange.fc_ino), cur, remaining);
- inode = ext4_iget(sb, le32_to_cpu(lrange->fc_ino), EXT4_IGET_NORMAL);
+ inode = ext4_iget(sb, le32_to_cpu(lrange.fc_ino), EXT4_IGET_NORMAL);
if (IS_ERR(inode)) {
- jbd_debug(1, "Inode %d not found", le32_to_cpu(lrange->fc_ino));
+ jbd_debug(1, "Inode %d not found", le32_to_cpu(lrange.fc_ino));
return 0;
}
ret = ext4_fc_record_modified_inode(sb, inode->i_ino);
jbd_debug(1, "DEL_RANGE, inode %ld, lblk %d, len %d\n",
- inode->i_ino, le32_to_cpu(lrange->fc_lblk),
- le32_to_cpu(lrange->fc_len));
+ inode->i_ino, le32_to_cpu(lrange.fc_lblk),
+ le32_to_cpu(lrange.fc_len));
while (remaining > 0) {
map.m_lblk = cur;
map.m_len = remaining;
}
ret = ext4_punch_hole(inode,
- le32_to_cpu(lrange->fc_lblk) << sb->s_blocksize_bits,
- le32_to_cpu(lrange->fc_len) << sb->s_blocksize_bits);
+ le32_to_cpu(lrange.fc_lblk) << sb->s_blocksize_bits,
+ le32_to_cpu(lrange.fc_len) << sb->s_blocksize_bits);
if (ret)
jbd_debug(1, "ext4_punch_hole returned %d", ret);
ext4_ext_replay_shrink_inode(inode,
struct ext4_sb_info *sbi = EXT4_SB(sb);
struct ext4_fc_replay_state *state;
int ret = JBD2_FC_REPLAY_CONTINUE;
- struct ext4_fc_add_range *ext;
- struct ext4_fc_tl *tl;
- struct ext4_fc_tail *tail;
- __u8 *start, *end;
- struct ext4_fc_head *head;
+ struct ext4_fc_add_range ext;
+ struct ext4_fc_tl tl;
+ struct ext4_fc_tail tail;
+ __u8 *start, *end, *cur, *val;
+ struct ext4_fc_head head;
struct ext4_extent *ex;
state = &sbi->s_fc_replay_state;
}
state->fc_replay_expected_off++;
- fc_for_each_tl(start, end, tl) {
+ for (cur = start; cur < end; cur = cur + sizeof(tl) + le16_to_cpu(tl.fc_len)) {
+ memcpy(&tl, cur, sizeof(tl));
+ val = cur + sizeof(tl);
jbd_debug(3, "Scan phase, tag:%s, blk %lld\n",
- tag2str(le16_to_cpu(tl->fc_tag)), bh->b_blocknr);
- switch (le16_to_cpu(tl->fc_tag)) {
+ tag2str(le16_to_cpu(tl.fc_tag)), bh->b_blocknr);
+ switch (le16_to_cpu(tl.fc_tag)) {
case EXT4_FC_TAG_ADD_RANGE:
- ext = (struct ext4_fc_add_range *)ext4_fc_tag_val(tl);
- ex = (struct ext4_extent *)&ext->fc_ex;
+ memcpy(&ext, val, sizeof(ext));
+ ex = (struct ext4_extent *)&ext.fc_ex;
ret = ext4_fc_record_regions(sb,
- le32_to_cpu(ext->fc_ino),
+ le32_to_cpu(ext.fc_ino),
le32_to_cpu(ex->ee_block), ext4_ext_pblock(ex),
ext4_ext_get_actual_len(ex));
if (ret < 0)
case EXT4_FC_TAG_INODE:
case EXT4_FC_TAG_PAD:
state->fc_cur_tag++;
- state->fc_crc = ext4_chksum(sbi, state->fc_crc, tl,
- sizeof(*tl) + ext4_fc_tag_len(tl));
+ state->fc_crc = ext4_chksum(sbi, state->fc_crc, cur,
+ sizeof(tl) + le16_to_cpu(tl.fc_len));
break;
case EXT4_FC_TAG_TAIL:
state->fc_cur_tag++;
- tail = (struct ext4_fc_tail *)ext4_fc_tag_val(tl);
- state->fc_crc = ext4_chksum(sbi, state->fc_crc, tl,
- sizeof(*tl) +
+ memcpy(&tail, val, sizeof(tail));
+ state->fc_crc = ext4_chksum(sbi, state->fc_crc, cur,
+ sizeof(tl) +
offsetof(struct ext4_fc_tail,
fc_crc));
- if (le32_to_cpu(tail->fc_tid) == expected_tid &&
- le32_to_cpu(tail->fc_crc) == state->fc_crc) {
+ if (le32_to_cpu(tail.fc_tid) == expected_tid &&
+ le32_to_cpu(tail.fc_crc) == state->fc_crc) {
state->fc_replay_num_tags = state->fc_cur_tag;
state->fc_regions_valid =
state->fc_regions_used;
state->fc_crc = 0;
break;
case EXT4_FC_TAG_HEAD:
- head = (struct ext4_fc_head *)ext4_fc_tag_val(tl);
- if (le32_to_cpu(head->fc_features) &
+ memcpy(&head, val, sizeof(head));
+ if (le32_to_cpu(head.fc_features) &
~EXT4_FC_SUPPORTED_FEATURES) {
ret = -EOPNOTSUPP;
break;
}
- if (le32_to_cpu(head->fc_tid) != expected_tid) {
+ if (le32_to_cpu(head.fc_tid) != expected_tid) {
ret = JBD2_FC_REPLAY_STOP;
break;
}
state->fc_cur_tag++;
- state->fc_crc = ext4_chksum(sbi, state->fc_crc, tl,
- sizeof(*tl) + ext4_fc_tag_len(tl));
+ state->fc_crc = ext4_chksum(sbi, state->fc_crc, cur,
+ sizeof(tl) + le16_to_cpu(tl.fc_len));
break;
default:
ret = state->fc_replay_num_tags ?
{
struct super_block *sb = journal->j_private;
struct ext4_sb_info *sbi = EXT4_SB(sb);
- struct ext4_fc_tl *tl;
- __u8 *start, *end;
+ struct ext4_fc_tl tl;
+ __u8 *start, *end, *cur, *val;
int ret = JBD2_FC_REPLAY_CONTINUE;
struct ext4_fc_replay_state *state = &sbi->s_fc_replay_state;
- struct ext4_fc_tail *tail;
+ struct ext4_fc_tail tail;
if (pass == PASS_SCAN) {
state->fc_current_pass = PASS_SCAN;
start = (u8 *)bh->b_data;
end = (__u8 *)bh->b_data + journal->j_blocksize - 1;
- fc_for_each_tl(start, end, tl) {
+ for (cur = start; cur < end; cur = cur + sizeof(tl) + le16_to_cpu(tl.fc_len)) {
+ memcpy(&tl, cur, sizeof(tl));
+ val = cur + sizeof(tl);
+
if (state->fc_replay_num_tags == 0) {
ret = JBD2_FC_REPLAY_STOP;
ext4_fc_set_bitmaps_and_counters(sb);
break;
}
jbd_debug(3, "Replay phase, tag:%s\n",
- tag2str(le16_to_cpu(tl->fc_tag)));
+ tag2str(le16_to_cpu(tl.fc_tag)));
state->fc_replay_num_tags--;
- switch (le16_to_cpu(tl->fc_tag)) {
+ switch (le16_to_cpu(tl.fc_tag)) {
case EXT4_FC_TAG_LINK:
- ret = ext4_fc_replay_link(sb, tl);
+ ret = ext4_fc_replay_link(sb, &tl, val);
break;
case EXT4_FC_TAG_UNLINK:
- ret = ext4_fc_replay_unlink(sb, tl);
+ ret = ext4_fc_replay_unlink(sb, &tl, val);
break;
case EXT4_FC_TAG_ADD_RANGE:
- ret = ext4_fc_replay_add_range(sb, tl);
+ ret = ext4_fc_replay_add_range(sb, &tl, val);
break;
case EXT4_FC_TAG_CREAT:
- ret = ext4_fc_replay_create(sb, tl);
+ ret = ext4_fc_replay_create(sb, &tl, val);
break;
case EXT4_FC_TAG_DEL_RANGE:
- ret = ext4_fc_replay_del_range(sb, tl);
+ ret = ext4_fc_replay_del_range(sb, &tl, val);
break;
case EXT4_FC_TAG_INODE:
- ret = ext4_fc_replay_inode(sb, tl);
+ ret = ext4_fc_replay_inode(sb, &tl, val);
break;
case EXT4_FC_TAG_PAD:
trace_ext4_fc_replay(sb, EXT4_FC_TAG_PAD, 0,
- ext4_fc_tag_len(tl), 0);
+ le16_to_cpu(tl.fc_len), 0);
break;
case EXT4_FC_TAG_TAIL:
trace_ext4_fc_replay(sb, EXT4_FC_TAG_TAIL, 0,
- ext4_fc_tag_len(tl), 0);
- tail = (struct ext4_fc_tail *)ext4_fc_tag_val(tl);
- WARN_ON(le32_to_cpu(tail->fc_tid) != expected_tid);
+ le16_to_cpu(tl.fc_len), 0);
+ memcpy(&tail, val, sizeof(tail));
+ WARN_ON(le32_to_cpu(tail.fc_tid) != expected_tid);
break;
case EXT4_FC_TAG_HEAD:
break;
default:
- trace_ext4_fc_replay(sb, le16_to_cpu(tl->fc_tag), 0,
- ext4_fc_tag_len(tl), 0);
+ trace_ext4_fc_replay(sb, le16_to_cpu(tl.fc_tag), 0,
+ le16_to_cpu(tl.fc_len), 0);
ret = -ECANCELED;
break;
}
#define region_last(__region) (((__region)->lblk) + ((__region)->len) - 1)
#endif
-#define fc_for_each_tl(__start, __end, __tl) \
- for (tl = (struct ext4_fc_tl *)(__start); \
- (__u8 *)tl < (__u8 *)(__end); \
- tl = (struct ext4_fc_tl *)((__u8 *)tl + \
- sizeof(struct ext4_fc_tl) + \
- + le16_to_cpu(tl->fc_len)))
-
static inline const char *tag2str(__u16 tag)
{
switch (tag) {
}
}
-/* Get length of a particular tlv */
-static inline int ext4_fc_tag_len(struct ext4_fc_tl *tl)
-{
- return le16_to_cpu(tl->fc_len);
-}
-
-/* Get a pointer to "value" of a tlv */
-static inline __u8 *ext4_fc_tag_val(struct ext4_fc_tl *tl)
-{
- return (__u8 *)tl + sizeof(*tl);
-}
-
#endif /* __FAST_COMMIT_H__ */
if (is_directory) {
count = ext4_used_dirs_count(sb, gdp) - 1;
ext4_used_dirs_set(sb, gdp, count);
- percpu_counter_dec(&sbi->s_dirs_counter);
+ if (percpu_counter_initialized(&sbi->s_dirs_counter))
+ percpu_counter_dec(&sbi->s_dirs_counter);
}
ext4_inode_bitmap_csum_set(sb, block_group, gdp, bitmap_bh,
EXT4_INODES_PER_GROUP(sb) / 8);
ext4_group_desc_csum_set(sb, block_group, gdp);
ext4_unlock_group(sb, block_group);
- percpu_counter_inc(&sbi->s_freeinodes_counter);
+ if (percpu_counter_initialized(&sbi->s_freeinodes_counter))
+ percpu_counter_inc(&sbi->s_freeinodes_counter);
if (sbi->s_log_groups_per_flex) {
struct flex_groups *fg;
*/
if (sbi->s_es->s_log_groups_per_flex >= 32) {
ext4_msg(sb, KERN_ERR, "too many log groups per flexible block group");
- goto err_freesgi;
+ goto err_freebuddy;
}
sbi->s_mb_prefetch = min_t(uint, 1 << sbi->s_es->s_log_groups_per_flex,
BLK_MAX_SEGMENT_SIZE >> (sb->s_blocksize_bits - 9));
struct dx_hash_info *hinfo = &name->hinfo;
int len;
- if (!IS_CASEFOLDED(dir) || !dir->i_sb->s_encoding) {
+ if (!IS_CASEFOLDED(dir) || !dir->i_sb->s_encoding ||
+ (IS_ENCRYPTED(dir) && !fscrypt_has_encryption_key(dir))) {
cf_name->name = NULL;
return 0;
}
#endif
#ifdef CONFIG_UNICODE
- if (parent->i_sb->s_encoding && IS_CASEFOLDED(parent)) {
+ if (parent->i_sb->s_encoding && IS_CASEFOLDED(parent) &&
+ (!IS_ENCRYPTED(parent) || fscrypt_has_encryption_key(parent))) {
if (fname->cf_name.name) {
struct qstr cf = {.name = fname->cf_name.name,
.len = fname->cf_name.len};
}
if (sb->s_blocksize != blocksize) {
+ /*
+ * bh must be released before kill_bdev(), otherwise
+ * it won't be freed and its page also. kill_bdev()
+ * is called by sb_set_blocksize().
+ */
+ brelse(bh);
/* Validate the filesystem blocksize */
if (!sb_set_blocksize(sb, blocksize)) {
ext4_msg(sb, KERN_ERR, "bad block size %d",
blocksize);
+ bh = NULL;
goto failed_mount;
}
- brelse(bh);
logical_sb_block = sb_block * EXT4_MIN_BLOCK_SIZE;
offset = do_div(logical_sb_block, blocksize);
bh = ext4_sb_bread_unmovable(sb, logical_sb_block);
kfree(get_qf_name(sb, sbi, i));
#endif
fscrypt_free_dummy_policy(&sbi->s_dummy_enc_policy);
- ext4_blkdev_remove(sbi);
+ /* ext4_blkdev_remove() calls kill_bdev(), release bh before it. */
brelse(bh);
+ ext4_blkdev_remove(sbi);
out_fail:
sb->s_fs_info = NULL;
kfree(sbi->s_blockgroup_lock);
#endif
EXT4_ATTR_FEATURE(metadata_csum_seed);
EXT4_ATTR_FEATURE(fast_commit);
+#if defined(CONFIG_UNICODE) && defined(CONFIG_FS_ENCRYPTION)
EXT4_ATTR_FEATURE(encrypted_casefold);
+#endif
static struct attribute *ext4_feat_attrs[] = {
ATTR_LIST(lazy_itable_init),
#endif
ATTR_LIST(metadata_csum_seed),
ATTR_LIST(fast_commit),
+#if defined(CONFIG_UNICODE) && defined(CONFIG_FS_ENCRYPTION)
ATTR_LIST(encrypted_casefold),
+#endif
NULL,
};
ATTRIBUTE_GROUPS(ext4_feat);
f2fs_drop_rpages(cc, len, true);
}
-static void f2fs_put_rpages_mapping(struct address_space *mapping,
- pgoff_t start, int len)
-{
- int i;
-
- for (i = 0; i < len; i++) {
- struct page *page = find_get_page(mapping, start + i);
-
- put_page(page);
- put_page(page);
- }
-}
-
static void f2fs_put_rpages_wbc(struct compress_ctx *cc,
struct writeback_control *wbc, bool redirty, int unlock)
{
return cc->rpages ? 0 : -ENOMEM;
}
-void f2fs_destroy_compress_ctx(struct compress_ctx *cc)
+void f2fs_destroy_compress_ctx(struct compress_ctx *cc, bool reuse)
{
page_array_free(cc->inode, cc->rpages, cc->cluster_size);
cc->rpages = NULL;
cc->nr_rpages = 0;
cc->nr_cpages = 0;
- cc->cluster_idx = NULL_CLUSTER;
+ if (!reuse)
+ cc->cluster_idx = NULL_CLUSTER;
}
void f2fs_compress_ctx_add_page(struct compress_ctx *cc, struct page *page)
}
if (PageUptodate(page))
- unlock_page(page);
+ f2fs_put_page(page, 1);
else
f2fs_compress_ctx_add_page(cc, page);
}
ret = f2fs_read_multi_pages(cc, &bio, cc->cluster_size,
&last_block_in_bio, false, true);
- f2fs_destroy_compress_ctx(cc);
+ f2fs_put_rpages(cc);
+ f2fs_destroy_compress_ctx(cc, true);
if (ret)
- goto release_pages;
+ goto out;
if (bio)
f2fs_submit_bio(sbi, bio, DATA);
ret = f2fs_init_compress_ctx(cc);
if (ret)
- goto release_pages;
+ goto out;
}
for (i = 0; i < cc->cluster_size; i++) {
f2fs_bug_on(sbi, cc->rpages[i]);
page = find_lock_page(mapping, start_idx + i);
- f2fs_bug_on(sbi, !page);
+ if (!page) {
+ /* page can be truncated */
+ goto release_and_retry;
+ }
f2fs_wait_on_page_writeback(page, DATA, true, true);
-
f2fs_compress_ctx_add_page(cc, page);
- f2fs_put_page(page, 0);
if (!PageUptodate(page)) {
+release_and_retry:
+ f2fs_put_rpages(cc);
f2fs_unlock_rpages(cc, i + 1);
- f2fs_put_rpages_mapping(mapping, start_idx,
- cc->cluster_size);
- f2fs_destroy_compress_ctx(cc);
+ f2fs_destroy_compress_ctx(cc, true);
goto retry;
}
}
}
unlock_pages:
+ f2fs_put_rpages(cc);
f2fs_unlock_rpages(cc, i);
-release_pages:
- f2fs_put_rpages_mapping(mapping, start_idx, i);
- f2fs_destroy_compress_ctx(cc);
+ f2fs_destroy_compress_ctx(cc, true);
+out:
return ret;
}
set_cluster_dirty(&cc);
f2fs_put_rpages_wbc(&cc, NULL, false, 1);
- f2fs_destroy_compress_ctx(&cc);
+ f2fs_destroy_compress_ctx(&cc, false);
return first_index;
}
f2fs_put_rpages(cc);
page_array_free(cc->inode, cc->cpages, cc->nr_cpages);
cc->cpages = NULL;
- f2fs_destroy_compress_ctx(cc);
+ f2fs_destroy_compress_ctx(cc, false);
return 0;
out_destroy_crypt:
for (i = 0; i < cc->nr_cpages; i++) {
if (!cc->cpages[i])
continue;
- f2fs_put_page(cc->cpages[i], 1);
+ f2fs_compress_free_page(cc->cpages[i]);
+ cc->cpages[i] = NULL;
}
out_put_cic:
kmem_cache_free(cic_entry_slab, cic);
err = f2fs_write_raw_pages(cc, submitted, wbc, io_type);
f2fs_put_rpages_wbc(cc, wbc, false, 0);
destroy_out:
- f2fs_destroy_compress_ctx(cc);
+ f2fs_destroy_compress_ctx(cc, false);
return err;
}
max_nr_pages,
&last_block_in_bio,
rac != NULL, false);
- f2fs_destroy_compress_ctx(&cc);
+ f2fs_destroy_compress_ctx(&cc, false);
if (ret)
goto set_error_page;
}
max_nr_pages,
&last_block_in_bio,
rac != NULL, false);
- f2fs_destroy_compress_ctx(&cc);
+ f2fs_destroy_compress_ctx(&cc, false);
}
}
#endif
}
}
if (f2fs_compressed_file(inode))
- f2fs_destroy_compress_ctx(&cc);
+ f2fs_destroy_compress_ctx(&cc, false);
#endif
if (retry) {
index = 0;
block_t pblock;
unsigned long nr_pblocks;
unsigned int blocks_per_sec = BLKS_PER_SEC(sbi);
+ unsigned int not_aligned = 0;
int ret = 0;
cur_lblock = 0;
if ((pblock - main_blkaddr) & (blocks_per_sec - 1) ||
nr_pblocks & (blocks_per_sec - 1)) {
- f2fs_err(sbi, "Swapfile does not align to section");
- ret = -EINVAL;
- goto out;
+ if (f2fs_is_pinned_file(inode)) {
+ f2fs_err(sbi, "Swapfile does not align to section");
+ ret = -EINVAL;
+ goto out;
+ }
+ not_aligned++;
}
cur_lblock += nr_pblocks;
}
+ if (not_aligned)
+ f2fs_warn(sbi, "Swapfile (%u) is not align to section: \n"
+ "\t1) creat(), 2) ioctl(F2FS_IOC_SET_PIN_FILE), 3) fallocate()",
+ not_aligned);
out:
return ret;
}
int nr_extents = 0;
unsigned long nr_pblocks;
unsigned int blocks_per_sec = BLKS_PER_SEC(sbi);
+ unsigned int not_aligned = 0;
int ret = 0;
/*
/* hole */
if (!(map.m_flags & F2FS_MAP_FLAGS)) {
f2fs_err(sbi, "Swapfile has holes\n");
- ret = -ENOENT;
+ ret = -EINVAL;
goto out;
}
if ((pblock - SM_I(sbi)->main_blkaddr) & (blocks_per_sec - 1) ||
nr_pblocks & (blocks_per_sec - 1)) {
- f2fs_err(sbi, "Swapfile does not align to section");
- ret = -EINVAL;
- goto out;
+ if (f2fs_is_pinned_file(inode)) {
+ f2fs_err(sbi, "Swapfile does not align to section");
+ ret = -EINVAL;
+ goto out;
+ }
+ not_aligned++;
}
if (cur_lblock + nr_pblocks >= sis->max)
sis->max = cur_lblock;
sis->pages = cur_lblock - 1;
sis->highest_bit = cur_lblock - 1;
+
+ if (not_aligned)
+ f2fs_warn(sbi, "Swapfile (%u) is not align to section: \n"
+ "\t1) creat(), 2) ioctl(F2FS_IOC_SET_PIN_FILE), 3) fallocate()",
+ not_aligned);
out:
return ret;
}
return ret;
bad_bmap:
f2fs_err(sbi, "Swapfile has holes\n");
- return -ENOENT;
+ return -EINVAL;
}
static int f2fs_swap_activate(struct swap_info_struct *sis, struct file *file,
void f2fs_decompress_end_io(struct decompress_io_ctx *dic, bool failed);
void f2fs_put_page_dic(struct page *page);
int f2fs_init_compress_ctx(struct compress_ctx *cc);
-void f2fs_destroy_compress_ctx(struct compress_ctx *cc);
+void f2fs_destroy_compress_ctx(struct compress_ctx *cc, bool reuse);
void f2fs_init_compress_info(struct f2fs_sb_info *sbi);
int f2fs_init_page_array_cache(struct f2fs_sb_info *sbi);
void f2fs_destroy_page_array_cache(struct f2fs_sb_info *sbi);
struct f2fs_inode_info *fi = F2FS_I(inode);
u32 masked_flags = fi->i_flags & mask;
- f2fs_bug_on(F2FS_I_SB(inode), (iflags & ~mask));
+ /* mask can be shrunk by flags_valid selector */
+ iflags &= mask;
/* Is it quota file? Do not allow user to mess with it */
if (IS_NOQUOTA(inode))
return err;
drop_bio:
- if (fio->bio) {
+ if (fio->bio && *(fio->bio)) {
struct bio *bio = *(fio->bio);
bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
- fio->bio = NULL;
+ *(fio->bio) = NULL;
}
return err;
}
current->backing_dev_info = inode_to_bdi(inode);
buffered = iomap_file_buffered_write(iocb, from, &gfs2_iomap_ops);
current->backing_dev_info = NULL;
- if (unlikely(buffered <= 0))
+ if (unlikely(buffered <= 0)) {
+ if (!ret)
+ ret = buffered;
goto out_unlock;
+ }
/*
* We need to ensure that the page cache pages are written to
spin_unlock(&gl->gl_lockref.lock);
}
+static bool is_system_glock(struct gfs2_glock *gl)
+{
+ struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;
+ struct gfs2_inode *m_ip = GFS2_I(sdp->sd_statfs_inode);
+
+ if (gl == m_ip->i_gl)
+ return true;
+ return false;
+}
+
/**
* do_xmote - Calls the DLM to change the state of a lock
* @gl: The lock state
* to see sd_log_error and withdraw, and in the meantime, requeue the
* work for later.
*
+ * We make a special exception for some system glocks, such as the
+ * system statfs inode glock, which needs to be granted before the
+ * gfs2_quotad daemon can exit, and that exit needs to finish before
+ * we can unmount the withdrawn file system.
+ *
* However, if we're just unlocking the lock (say, for unmount, when
* gfs2_gl_hash_clear calls clear_glock) and recovery is complete
* then it's okay to tell dlm to unlock it.
*/
if (unlikely(sdp->sd_log_error && !gfs2_withdrawn(sdp)))
gfs2_withdraw_delayed(sdp);
- if (glock_blocked_by_withdraw(gl)) {
- if (target != LM_ST_UNLOCKED ||
- test_bit(SDF_WITHDRAW_RECOVERY, &sdp->sd_flags)) {
+ if (glock_blocked_by_withdraw(gl) &&
+ (target != LM_ST_UNLOCKED ||
+ test_bit(SDF_WITHDRAW_RECOVERY, &sdp->sd_flags))) {
+ if (!is_system_glock(gl)) {
gfs2_glock_queue_work(gl, GL_GLOCK_DFT_HOLD);
goto out;
+ } else {
+ clear_bit(GLF_INVALIDATE_IN_PROGRESS, &gl->gl_flags);
}
}
glock_blocked_by_withdraw(gl) &&
gh->gh_gl != sdp->sd_jinode_gl) {
sdp->sd_glock_dqs_held++;
+ spin_unlock(&gl->gl_lockref.lock);
might_sleep();
wait_on_bit(&sdp->sd_flags, SDF_WITHDRAW_RECOVERY,
TASK_UNINTERRUPTIBLE);
+ spin_lock(&gl->gl_lockref.lock);
}
if (gh->gh_flags & GL_NOCACHE)
handle_callback(gl, LM_ST_UNLOCKED, 0, false);
while(!list_empty(list)) {
gl = list_first_entry(list, struct gfs2_glock, gl_lru);
list_del_init(&gl->gl_lru);
+ clear_bit(GLF_LRU, &gl->gl_flags);
if (!spin_trylock(&gl->gl_lockref.lock)) {
add_back_to_lru:
list_add(&gl->gl_lru, &lru_list);
if (!test_bit(GLF_LOCK, &gl->gl_flags)) {
list_move(&gl->gl_lru, &dispose);
atomic_dec(&lru_count);
- clear_bit(GLF_LRU, &gl->gl_flags);
freed++;
continue;
}
struct timespec64 atime;
u16 height, depth;
umode_t mode = be32_to_cpu(str->di_mode);
- bool is_new = ip->i_inode.i_flags & I_NEW;
+ bool is_new = ip->i_inode.i_state & I_NEW;
if (unlikely(ip->i_no_addr != be64_to_cpu(str->di_num.no_addr)))
goto corrupt;
}
/**
- * ail_drain - drain the ail lists after a withdraw
+ * gfs2_ail_drain - drain the ail lists after a withdraw
* @sdp: Pointer to GFS2 superblock
*/
-static void ail_drain(struct gfs2_sbd *sdp)
+void gfs2_ail_drain(struct gfs2_sbd *sdp)
{
struct gfs2_trans *tr;
list_del(&tr->tr_list);
gfs2_trans_free(sdp, tr);
}
+ gfs2_drain_revokes(sdp);
spin_unlock(&sdp->sd_ail_lock);
}
if (tr && list_empty(&tr->tr_list))
list_add(&tr->tr_list, &sdp->sd_ail1_list);
spin_unlock(&sdp->sd_ail_lock);
- ail_drain(sdp); /* frees all transactions */
tr = NULL;
goto out_end;
}
extern void gfs2_add_revoke(struct gfs2_sbd *sdp, struct gfs2_bufdata *bd);
extern void gfs2_glock_remove_revoke(struct gfs2_glock *gl);
extern void gfs2_flush_revokes(struct gfs2_sbd *sdp);
+extern void gfs2_ail_drain(struct gfs2_sbd *sdp);
#endif /* __LOG_DOT_H__ */
gfs2_log_write_page(sdp, page);
}
-static void revoke_lo_after_commit(struct gfs2_sbd *sdp, struct gfs2_trans *tr)
+void gfs2_drain_revokes(struct gfs2_sbd *sdp)
{
struct list_head *head = &sdp->sd_log_revokes;
struct gfs2_bufdata *bd;
}
}
+static void revoke_lo_after_commit(struct gfs2_sbd *sdp, struct gfs2_trans *tr)
+{
+ gfs2_drain_revokes(sdp);
+}
+
static void revoke_lo_before_scan(struct gfs2_jdesc *jd,
struct gfs2_log_header_host *head, int pass)
{
extern void gfs2_pin(struct gfs2_sbd *sdp, struct buffer_head *bh);
extern int gfs2_find_jhead(struct gfs2_jdesc *jd,
struct gfs2_log_header_host *head, bool keep_cache);
+extern void gfs2_drain_revokes(struct gfs2_sbd *sdp);
static inline unsigned int buf_limit(struct gfs2_sbd *sdp)
{
return sdp->sd_ldptrs;
if (test_bit(SDF_NORECOVERY, &sdp->sd_flags) || !sdp->sd_jdesc)
return;
+ gfs2_ail_drain(sdp); /* frees all transactions */
inode = sdp->sd_jdesc->jd_inode;
ip = GFS2_I(inode);
i_gl = ip->i_gl;
res = __hfsplus_ext_cache_extent(&fd, inode, alloc_cnt);
if (res)
break;
- hfs_brec_remove(&fd);
- mutex_unlock(&fd.tree->tree_lock);
start = hip->cached_start;
+ if (blk_cnt <= start)
+ hfs_brec_remove(&fd);
+ mutex_unlock(&fd.tree->tree_lock);
hfsplus_free_extents(sb, hip->cached_extents,
alloc_cnt - start, alloc_cnt - blk_cnt);
hfsplus_dump_extent(hip->cached_extents);
+ mutex_lock(&fd.tree->tree_lock);
if (blk_cnt > start) {
hip->extent_state |= HFSPLUS_EXT_DIRTY;
break;
alloc_cnt = start;
hip->cached_start = hip->cached_blocks = 0;
hip->extent_state &= ~(HFSPLUS_EXT_DIRTY | HFSPLUS_EXT_NEW);
- mutex_lock(&fd.tree->tree_lock);
}
hfs_find_exit(&fd);
static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma)
{
struct inode *inode = file_inode(file);
+ struct hugetlbfs_inode_info *info = HUGETLBFS_I(inode);
loff_t len, vma_len;
int ret;
struct hstate *h = hstate_file(file);
vma->vm_flags |= VM_HUGETLB | VM_DONTEXPAND;
vma->vm_ops = &hugetlb_vm_ops;
+ ret = seal_check_future_write(info->seals, vma);
+ if (ret)
+ return ret;
+
/*
* page based offset in vm_pgoff could be sufficiently large to
* overflow a loff_t when converted to byte offset. This can
* the subpool and global reserve usage count can need
* to be adjusted.
*/
- VM_BUG_ON(PagePrivate(page));
+ VM_BUG_ON(HPageRestoreReserve(page));
remove_huge_page(page);
freed++;
if (!truncate_op) {
__SetPageUptodate(page);
error = huge_add_to_page_cache(page, mapping, index);
if (unlikely(error)) {
+ restore_reserve_on_error(h, &pseudo_vma, addr, page);
put_page(page);
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
goto out;
return cwd->wqe->wq == data;
}
+void io_wq_exit_start(struct io_wq *wq)
+{
+ set_bit(IO_WQ_BIT_EXIT, &wq->state);
+}
+
static void io_wq_exit_workers(struct io_wq *wq)
{
struct callback_head *cb;
int node;
- set_bit(IO_WQ_BIT_EXIT, &wq->state);
-
if (!wq->task)
return;
struct io_wqe *wqe = wq->wqes[node];
io_wq_for_each_worker(wqe, io_wq_worker_wake, NULL);
- spin_lock_irq(&wq->hash->wait.lock);
- list_del_init(&wq->wqes[node]->wait.entry);
- spin_unlock_irq(&wq->hash->wait.lock);
}
rcu_read_unlock();
io_worker_ref_put(wq);
wait_for_completion(&wq->worker_done);
+
+ for_each_node(node) {
+ spin_lock_irq(&wq->hash->wait.lock);
+ list_del_init(&wq->wqes[node]->wait.entry);
+ spin_unlock_irq(&wq->hash->wait.lock);
+ }
put_task_struct(wq->task);
wq->task = NULL;
}
cpuhp_state_remove_instance_nocalls(io_wq_online, &wq->cpuhp_node);
- io_wq_exit_workers(wq);
-
for_each_node(node) {
struct io_wqe *wqe = wq->wqes[node];
struct io_cb_cancel_data match = {
kfree(wq);
}
-void io_wq_put(struct io_wq *wq)
-{
- if (refcount_dec_and_test(&wq->refs))
- io_wq_destroy(wq);
-}
-
void io_wq_put_and_exit(struct io_wq *wq)
{
+ WARN_ON_ONCE(!test_bit(IO_WQ_BIT_EXIT, &wq->state));
+
io_wq_exit_workers(wq);
- io_wq_put(wq);
+ if (refcount_dec_and_test(&wq->refs))
+ io_wq_destroy(wq);
}
static bool io_wq_worker_affinity(struct io_worker *worker, void *data)
};
struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data);
-void io_wq_put(struct io_wq *wq);
+void io_wq_exit_start(struct io_wq *wq);
void io_wq_put_and_exit(struct io_wq *wq);
void io_wq_enqueue(struct io_wq *wq, struct io_wq_work *work);
#define IORING_MAX_RESTRICTIONS (IORING_RESTRICTION_LAST + \
IORING_REGISTER_LAST + IORING_OP_LAST)
+#define IORING_MAX_REG_BUFFERS (1U << 14)
+
#define SQE_VALID_FLAGS (IOSQE_FIXED_FILE|IOSQE_IO_DRAIN|IOSQE_IO_LINK| \
IOSQE_IO_HARDLINK | IOSQE_ASYNC | \
IOSQE_BUFFER_SELECT)
task_work_func_t func;
};
+enum {
+ IORING_RSRC_FILE = 0,
+ IORING_RSRC_BUFFER = 1,
+};
+
/*
* NOTE! Each of the iocb union members has the file pointer
* as the first entry in their struct definition. So you can
#if defined(CONFIG_EPOLL)
if (sqe->ioprio || sqe->buf_index)
return -EINVAL;
- if (unlikely(req->ctx->flags & (IORING_SETUP_IOPOLL | IORING_SETUP_SQPOLL)))
+ if (unlikely(req->ctx->flags & IORING_SETUP_IOPOLL))
return -EINVAL;
req->epoll.epfd = READ_ONCE(sqe->fd);
static int io_statx_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
{
- if (unlikely(req->ctx->flags & (IORING_SETUP_IOPOLL | IORING_SETUP_SQPOLL)))
+ if (unlikely(req->ctx->flags & IORING_SETUP_IOPOLL))
return -EINVAL;
if (sqe->ioprio || sqe->buf_index)
return -EINVAL;
* Can't handle multishot for double wait for now, turn it
* into one-shot mode.
*/
- if (!(req->poll.events & EPOLLONESHOT))
- req->poll.events |= EPOLLONESHOT;
+ if (!(poll_one->events & EPOLLONESHOT))
+ poll_one->events |= EPOLLONESHOT;
/* double add on the same waitqueue head, ignore */
- if (poll->head == head)
+ if (poll_one->head == head)
return;
poll = kmalloc(sizeof(*poll), GFP_ATOMIC);
if (!poll) {
static int io_rsrc_update_prep(struct io_kiocb *req,
const struct io_uring_sqe *sqe)
{
- if (unlikely(req->ctx->flags & IORING_SETUP_SQPOLL))
- return -EINVAL;
if (unlikely(req->flags & (REQ_F_FIXED_FILE | REQ_F_BUFFER_SELECT)))
return -EINVAL;
if (sqe->ioprio || sqe->rw_flags)
* We don't expect the list to be empty, that will only happen if we
* race with the completion of the linked work.
*/
- if (prev && req_ref_inc_not_zero(prev))
+ if (prev) {
io_remove_next_linked(prev);
- else
- prev = NULL;
+ if (!req_ref_inc_not_zero(prev))
+ prev = NULL;
+ }
spin_unlock_irqrestore(&ctx->completion_lock, flags);
if (prev) {
io_async_find_and_cancel(ctx, req, prev->user_data, -ETIME);
io_put_req_deferred(prev, 1);
+ io_put_req_deferred(req, 1);
} else {
io_req_complete_post(req, -ETIME, 0);
}
- io_put_req_deferred(req, 1);
return HRTIMER_NORESTART;
}
{
int i, ret;
+ imu->acct_pages = 0;
for (i = 0; i < nr_pages; i++) {
if (!PageCompound(pages[i])) {
imu->acct_pages++;
if (ctx->user_bufs)
return -EBUSY;
- if (!nr_args || nr_args > UIO_MAXIOV)
+ if (!nr_args || nr_args > IORING_MAX_REG_BUFFERS)
return -EINVAL;
ret = io_rsrc_node_switch_start(ctx);
if (ret)
static void io_uring_clean_tctx(struct io_uring_task *tctx)
{
+ struct io_wq *wq = tctx->io_wq;
struct io_tctx_node *node;
unsigned long index;
xa_for_each(&tctx->xa, index, node)
io_uring_del_task_file(index);
- if (tctx->io_wq) {
- io_wq_put_and_exit(tctx->io_wq);
+ if (wq) {
+ /*
+ * Must be after io_uring_del_task_file() (removes nodes under
+ * uring_lock) to avoid race with io_uring_try_cancel_iowq().
+ */
tctx->io_wq = NULL;
+ io_wq_put_and_exit(wq);
}
}
if (!current->io_uring)
return;
+ if (tctx->io_wq)
+ io_wq_exit_start(tctx->io_wq);
+
WARN_ON_ONCE(!sqd || sqd->thread != current);
atomic_inc(&tctx->in_idle);
DEFINE_WAIT(wait);
s64 inflight;
+ if (tctx->io_wq)
+ io_wq_exit_start(tctx->io_wq);
+
/* make sure overflow events are dropped */
atomic_inc(&tctx->in_idle);
do {
IORING_FEAT_SUBMIT_STABLE | IORING_FEAT_RW_CUR_POS |
IORING_FEAT_CUR_PERSONALITY | IORING_FEAT_FAST_POLL |
IORING_FEAT_POLL_32BITS | IORING_FEAT_SQPOLL_NONFIXED |
- IORING_FEAT_EXT_ARG | IORING_FEAT_NATIVE_WORKERS;
+ IORING_FEAT_EXT_ARG | IORING_FEAT_NATIVE_WORKERS |
+ IORING_FEAT_RSRC_TAGS;
if (copy_to_user(params, p, sizeof(*p))) {
ret = -EFAULT;
}
static int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg,
- unsigned size)
+ unsigned size, unsigned type)
{
struct io_uring_rsrc_update2 up;
return -EINVAL;
if (copy_from_user(&up, arg, sizeof(up)))
return -EFAULT;
- if (!up.nr)
+ if (!up.nr || up.resv)
return -EINVAL;
- return __io_register_rsrc_update(ctx, up.type, &up, up.nr);
+ return __io_register_rsrc_update(ctx, type, &up, up.nr);
}
static int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg,
- unsigned int size)
+ unsigned int size, unsigned int type)
{
struct io_uring_rsrc_register rr;
memset(&rr, 0, sizeof(rr));
if (copy_from_user(&rr, arg, size))
return -EFAULT;
- if (!rr.nr)
+ if (!rr.nr || rr.resv || rr.resv2)
return -EINVAL;
- switch (rr.type) {
+ switch (type) {
case IORING_RSRC_FILE:
return io_sqe_files_register(ctx, u64_to_user_ptr(rr.data),
rr.nr, u64_to_user_ptr(rr.tags));
case IORING_REGISTER_PROBE:
case IORING_REGISTER_PERSONALITY:
case IORING_UNREGISTER_PERSONALITY:
- case IORING_REGISTER_RSRC:
- case IORING_REGISTER_RSRC_UPDATE:
+ case IORING_REGISTER_FILES2:
+ case IORING_REGISTER_FILES_UPDATE2:
+ case IORING_REGISTER_BUFFERS2:
+ case IORING_REGISTER_BUFFERS_UPDATE:
return false;
default:
return true;
case IORING_REGISTER_RESTRICTIONS:
ret = io_register_restrictions(ctx, arg, nr_args);
break;
- case IORING_REGISTER_RSRC:
- ret = io_register_rsrc(ctx, arg, nr_args);
+ case IORING_REGISTER_FILES2:
+ ret = io_register_rsrc(ctx, arg, nr_args, IORING_RSRC_FILE);
+ break;
+ case IORING_REGISTER_FILES_UPDATE2:
+ ret = io_register_rsrc_update(ctx, arg, nr_args,
+ IORING_RSRC_FILE);
+ break;
+ case IORING_REGISTER_BUFFERS2:
+ ret = io_register_rsrc(ctx, arg, nr_args, IORING_RSRC_BUFFER);
break;
- case IORING_REGISTER_RSRC_UPDATE:
- ret = io_register_rsrc_update(ctx, arg, nr_args);
+ case IORING_REGISTER_BUFFERS_UPDATE:
+ ret = io_register_rsrc_update(ctx, arg, nr_args,
+ IORING_RSRC_BUFFER);
break;
default:
ret = -EINVAL;
{
struct inode *inode = rac->mapping->host;
loff_t pos = readahead_pos(rac);
- loff_t length = readahead_length(rac);
+ size_t length = readahead_length(rac);
struct iomap_readpage_ctx ctx = {
.rac = rac,
};
trace_iomap_readahead(inode, readahead_count(rac));
while (length > 0) {
- loff_t ret = iomap_apply(inode, pos, length, 0, ops,
+ ssize_t ret = iomap_apply(inode, pos, length, 0, ops,
&ctx, iomap_readahead_actor);
if (ret <= 0) {
WARN_ON_ONCE(ret == 0);
if (!(m->mnt_sb->s_type->fs_flags & FS_ALLOW_IDMAP))
return -EINVAL;
+ /* Don't yet support filesystem mountable in user namespaces. */
+ if (m->mnt_sb->s_user_ns != &init_user_ns)
+ return -EINVAL;
+
/* We're not controlling the superblock. */
- if (!ns_capable(m->mnt_sb->s_user_ns, CAP_SYS_ADMIN))
+ if (!capable(CAP_SYS_ADMIN))
return -EPERM;
/* Mount has already been visible in the filesystem hierarchy. */
# SPDX-License-Identifier: GPL-2.0-only
config NETFS_SUPPORT
- tristate "Support for network filesystem high-level I/O"
+ tristate
help
This option enables support for network filesystems, including
helpers for high-level buffered I/O, abstracting out read
DEFINE_READAHEAD(ractl, file, NULL, mapping, index);
retry:
- page = grab_cache_page_write_begin(mapping, index, 0);
+ page = grab_cache_page_write_begin(mapping, index, flags);
if (!page)
return -ENOMEM;
if (cl_init->hostname == NULL) {
WARN_ON(1);
- return NULL;
+ return ERR_PTR(-EINVAL);
}
/* see if the client already exists */
if (unlikely(!p))
goto out_err;
fl->fh_array[i]->size = be32_to_cpup(p++);
- if (sizeof(struct nfs_fh) < fl->fh_array[i]->size) {
+ if (fl->fh_array[i]->size > NFS_MAXFHSIZE) {
printk(KERN_ERR "NFS: Too big fh %d received %d\n",
i, fl->fh_array[i]->size);
goto out_err;
.set = param_set_nfs_timeout,
.get = param_get_nfs_timeout,
};
-#define param_check_nfs_timeout(name, p) __param_check(name, p, int);
+#define param_check_nfs_timeout(name, p) __param_check(name, p, int)
module_param(nfs_mountpoint_expiry_timeout, nfs_timeout, 0644);
MODULE_PARM_DESC(nfs_mountpoint_expiry_timeout,
struct inode *inode;
nfs4_stateid *stateid;
long timeout;
+ unsigned char task_is_privileged : 1;
unsigned char delay : 1,
recovering : 1,
retry : 1;
*/
nfs_mark_client_ready(clp, -EPERM);
}
- nfs_put_client(clp);
clear_bit(NFS_CS_TSM_POSSIBLE, &clp->cl_flags);
+ nfs_put_client(clp);
return old;
error:
case SEEK_HOLE:
case SEEK_DATA:
ret = nfs42_proc_llseek(filep, offset, whence);
- if (ret != -ENOTSUPP)
+ if (ret != -EOPNOTSUPP)
return ret;
fallthrough;
default:
goto out_retry;
}
if (exception->recovering) {
+ if (exception->task_is_privileged)
+ return -EDEADLOCK;
ret = nfs4_wait_clnt_recover(clp);
if (test_bit(NFS_MIG_FAILED, &server->mig_status))
return -EIO;
goto out_retry;
}
if (exception->recovering) {
+ if (exception->task_is_privileged)
+ return -EDEADLOCK;
rpc_sleep_on(&clp->cl_rpcwaitq, task, NULL);
if (test_bit(NFS4CLNT_MANAGER_RUNNING, &clp->cl_state) == 0)
rpc_wake_up_queued_task(&clp->cl_rpcwaitq, task);
rcu_read_unlock();
trace_nfs4_open_stateid_update_wait(state->inode, stateid, 0);
- if (!signal_pending(current)) {
+ if (!fatal_signal_pending(current)) {
if (schedule_timeout(5*HZ) == 0)
status = -EAGAIN;
else
write_sequnlock(&state->seqlock);
trace_nfs4_close_stateid_update_wait(state->inode, dst, 0);
- if (signal_pending(current))
+ if (fatal_signal_pending(current))
status = -EINTR;
else
if (schedule_timeout(5*HZ) != 0)
server->caps |= NFS_CAP_HARDLINKS;
if (res.has_symlinks != 0)
server->caps |= NFS_CAP_SYMLINKS;
+#ifdef CONFIG_NFS_V4_SECURITY_LABEL
+ if (res.attr_bitmask[2] & FATTR4_WORD2_SECURITY_LABEL)
+ server->caps |= NFS_CAP_SECURITY_LABEL;
+#endif
if (!(res.attr_bitmask[0] & FATTR4_WORD0_FILEID))
server->fattr_valid &= ~NFS_ATTR_FATTR_FILEID;
if (!(res.attr_bitmask[1] & FATTR4_WORD1_MODE))
server->fattr_valid &= ~NFS_ATTR_FATTR_CTIME;
if (!(res.attr_bitmask[1] & FATTR4_WORD1_TIME_MODIFY))
server->fattr_valid &= ~NFS_ATTR_FATTR_MTIME;
-#ifdef CONFIG_NFS_V4_SECURITY_LABEL
- if (!(res.attr_bitmask[2] & FATTR4_WORD2_SECURITY_LABEL))
- server->fattr_valid &= ~NFS_ATTR_FATTR_V4_SECURITY_LABEL;
-#endif
memcpy(server->attr_bitmask_nl, res.attr_bitmask,
sizeof(server->attr_bitmask));
server->attr_bitmask_nl[2] &= ~FATTR4_WORD2_SECURITY_LABEL;
do {
err = __nfs4_proc_set_acl(inode, buf, buflen);
trace_nfs4_set_acl(inode, err);
+ if (err == -NFS4ERR_BADOWNER || err == -NFS4ERR_BADNAME) {
+ /*
+ * no need to retry since the kernel
+ * isn't involved in encoding the ACEs.
+ */
+ err = -EINVAL;
+ break;
+ }
err = nfs4_handle_exception(NFS_SERVER(inode), err,
&exception);
} while (exception.retry);
struct nfs4_exception exception = {
.inode = data->inode,
.stateid = &data->stateid,
+ .task_is_privileged = data->args.seq_args.sa_privileged,
};
if (!nfs4_sequence_done(task, &data->res.seq_res))
data = kzalloc(sizeof(*data), GFP_NOFS);
if (data == NULL)
return -ENOMEM;
- nfs4_init_sequence(&data->args.seq_args, &data->res.seq_res, 1, 0);
nfs4_state_protect(server->nfs_client,
NFS_SP4_MACH_CRED_CLEANUP,
}
}
+ if (!data->inode)
+ nfs4_init_sequence(&data->args.seq_args, &data->res.seq_res, 1,
+ 1);
+ else
+ nfs4_init_sequence(&data->args.seq_args, &data->res.seq_res, 1,
+ 0);
task_setup_data.callback_data = data;
msg.rpc_argp = &data->args;
msg.rpc_resp = &data->res;
&task_setup_data.rpc_client, &msg);
dprintk("--> %s\n", __func__);
+ lrp->inode = nfs_igrab_and_active(lrp->args.inode);
if (!sync) {
- lrp->inode = nfs_igrab_and_active(lrp->args.inode);
if (!lrp->inode) {
nfs4_layoutreturn_release(lrp);
return -EAGAIN;
}
task_setup_data.flags |= RPC_TASK_ASYNC;
}
- nfs4_init_sequence(&lrp->args.seq_args, &lrp->res.seq_res, 1, 0);
+ if (!lrp->inode)
+ nfs4_init_sequence(&lrp->args.seq_args, &lrp->res.seq_res, 1,
+ 1);
+ else
+ nfs4_init_sequence(&lrp->args.seq_args, &lrp->res.seq_res, 1,
+ 0);
task = rpc_run_task(&task_setup_data);
if (IS_ERR(task))
return PTR_ERR(task);
{ O_NOATIME, "O_NOATIME" }, \
{ O_CLOEXEC, "O_CLOEXEC" })
-TRACE_DEFINE_ENUM(FMODE_READ);
-TRACE_DEFINE_ENUM(FMODE_WRITE);
-TRACE_DEFINE_ENUM(FMODE_EXEC);
-
#define show_fmode_flags(mode) \
__print_flags(mode, "|", \
{ ((__force unsigned long)FMODE_READ), "READ" }, \
struct nfs_page *prev = NULL;
unsigned int size;
- if (mirror->pg_count != 0) {
- prev = nfs_list_entry(mirror->pg_list.prev);
- } else {
+ if (list_empty(&mirror->pg_list)) {
if (desc->pg_ops->pg_init)
desc->pg_ops->pg_init(desc, req);
if (desc->pg_error < 0)
return 0;
mirror->pg_base = req->wb_pgbase;
- }
+ mirror->pg_count = 0;
+ mirror->pg_recoalesce = 0;
+ } else
+ prev = nfs_list_entry(mirror->pg_list.prev);
if (desc->pg_maxretrans && req->wb_nio > desc->pg_maxretrans) {
if (NFS_SERVER(desc->pg_inode)->flags & NFS_MOUNT_SOFTERR)
{
struct nfs_pgio_mirror *mirror = nfs_pgio_current_mirror(desc);
-
if (!list_empty(&mirror->pg_list)) {
int error = desc->pg_ops->pg_doio(desc);
if (error < 0)
desc->pg_error = error;
- else
+ if (list_empty(&mirror->pg_list))
mirror->pg_bytes_written += mirror->pg_count;
}
- if (list_empty(&mirror->pg_list)) {
- mirror->pg_count = 0;
- mirror->pg_base = 0;
- }
}
static void
do {
list_splice_init(&mirror->pg_list, &head);
- mirror->pg_bytes_written -= mirror->pg_count;
- mirror->pg_count = 0;
- mirror->pg_base = 0;
- mirror->pg_recoalesce = 0;
while (!list_empty(&head)) {
struct nfs_page *req;
{
struct pnfs_layout_hdr *lo = NULL;
struct nfs_inode *nfsi = NFS_I(ino);
+ struct pnfs_layout_range range = {
+ .iomode = IOMODE_ANY,
+ .offset = 0,
+ .length = NFS4_MAX_UINT64,
+ };
LIST_HEAD(tmp_list);
const struct cred *cred;
nfs4_stateid stateid;
}
valid_layout = pnfs_layout_is_valid(lo);
pnfs_clear_layoutcommit(ino, &tmp_list);
- pnfs_mark_matching_lsegs_return(lo, &tmp_list, NULL, 0);
+ pnfs_mark_matching_lsegs_return(lo, &tmp_list, &range, 0);
- if (NFS_SERVER(ino)->pnfs_curr_ld->return_range) {
- struct pnfs_layout_range range = {
- .iomode = IOMODE_ANY,
- .offset = 0,
- .length = NFS4_MAX_UINT64,
- };
+ if (NFS_SERVER(ino)->pnfs_curr_ld->return_range)
NFS_SERVER(ino)->pnfs_curr_ld->return_range(lo, &range);
- }
/* Don't send a LAYOUTRETURN if list was initially empty */
if (!test_bit(NFS_LAYOUT_RETURN_REQUESTED, &lo->plh_flags) ||
void
pnfs_generic_pg_init_read(struct nfs_pageio_descriptor *pgio, struct nfs_page *req)
{
- u64 rd_size = req->wb_bytes;
+ u64 rd_size;
pnfs_generic_pg_check_layout(pgio);
pnfs_generic_pg_check_range(pgio, req);
.set = param_set_portnr,
.get = param_get_uint,
};
-#define param_check_portnr(name, p) __param_check(name, p, unsigned int);
+#define param_check_portnr(name, p) __param_check(name, p, unsigned int)
module_param_named(callback_tcpport, nfs_callback_set_tcpport, portnr, 0644);
module_param_named(callback_nr_threads, nfs_callback_nr_threads, ushort, 0644);
* events generated by the listener process itself, without disclosing
* the pids of other processes.
*/
- if (!capable(CAP_SYS_ADMIN) &&
+ if (FAN_GROUP_FLAG(group, FANOTIFY_UNPRIV) &&
task_tgid(current) != event->pid)
metadata.pid = 0;
- if (path && path->mnt && path->dentry) {
+ /*
+ * For now, fid mode is required for an unprivileged listener and
+ * fid mode does not report fd in events. Keep this check anyway
+ * for safety in case fid mode requirement is relaxed in the future
+ * to allow unprivileged listener to get events with no fd and no fid.
+ */
+ if (!FAN_GROUP_FLAG(group, FANOTIFY_UNPRIV) &&
+ path && path->mnt && path->dentry) {
fd = create_fd(group, path, &f);
if (fd < 0)
return fd;
info_type, fanotify_info_name(info),
info->name_len, buf, count);
if (ret < 0)
- return ret;
+ goto out_close_fd;
buf += ret;
count -= ret;
fanotify_event_object_fh(event),
info_type, dot, dot_len, buf, count);
if (ret < 0)
- return ret;
+ goto out_close_fd;
buf += ret;
count -= ret;
int f_flags, fd;
unsigned int fid_mode = flags & FANOTIFY_FID_BITS;
unsigned int class = flags & FANOTIFY_CLASS_BITS;
+ unsigned int internal_flags = 0;
pr_debug("%s: flags=%x event_f_flags=%x\n",
__func__, flags, event_f_flags);
*/
if ((flags & FANOTIFY_ADMIN_INIT_FLAGS) || !fid_mode)
return -EPERM;
+
+ /*
+ * Setting the internal flag FANOTIFY_UNPRIV on the group
+ * prevents setting mount/filesystem marks on this group and
+ * prevents reporting pid and open fd in events.
+ */
+ internal_flags |= FANOTIFY_UNPRIV;
}
#ifdef CONFIG_AUDITSYSCALL
goto out_destroy_group;
}
- group->fanotify_data.flags = flags;
+ group->fanotify_data.flags = flags | internal_flags;
group->memcg = get_mem_cgroup_from_mm(current->mm);
group->fanotify_data.merge_hash = fanotify_alloc_merge_hash();
group = f.file->private_data;
/*
- * An unprivileged user is not allowed to watch a mount point nor
- * a filesystem.
+ * An unprivileged user is not allowed to setup mount nor filesystem
+ * marks. This also includes setting up such marks by a group that
+ * was initialized by an unprivileged user.
*/
ret = -EPERM;
- if (!capable(CAP_SYS_ADMIN) &&
+ if ((!capable(CAP_SYS_ADMIN) ||
+ FAN_GROUP_FLAG(group, FANOTIFY_UNPRIV)) &&
mark_type != FAN_MARK_INODE)
goto fput_and_out;
max_marks = clamp(max_marks, FANOTIFY_OLD_DEFAULT_MAX_MARKS,
FANOTIFY_DEFAULT_MAX_USER_MARKS);
+ BUILD_BUG_ON(FANOTIFY_INIT_FLAGS & FANOTIFY_INTERNAL_GROUP_FLAGS);
BUILD_BUG_ON(HWEIGHT32(FANOTIFY_INIT_FLAGS) != 10);
BUILD_BUG_ON(HWEIGHT32(FANOTIFY_MARK_FLAGS) != 9);
struct fsnotify_group *group = f->private_data;
seq_printf(m, "fanotify flags:%x event-flags:%x\n",
- group->fanotify_data.flags,
+ group->fanotify_data.flags & FANOTIFY_INIT_FLAGS,
group->fanotify_data.f_flags);
show_fdinfo(m, f, fanotify_fdinfo);
}
/*
+ * zero out partial blocks of one cluster.
+ *
+ * start: file offset where zero starts, will be made upper block aligned.
+ * len: it will be trimmed to the end of current cluster if "start + len"
+ * is bigger than it.
+ */
+static int ocfs2_zeroout_partial_cluster(struct inode *inode,
+ u64 start, u64 len)
+{
+ int ret;
+ u64 start_block, end_block, nr_blocks;
+ u64 p_block, offset;
+ u32 cluster, p_cluster, nr_clusters;
+ struct super_block *sb = inode->i_sb;
+ u64 end = ocfs2_align_bytes_to_clusters(sb, start);
+
+ if (start + len < end)
+ end = start + len;
+
+ start_block = ocfs2_blocks_for_bytes(sb, start);
+ end_block = ocfs2_blocks_for_bytes(sb, end);
+ nr_blocks = end_block - start_block;
+ if (!nr_blocks)
+ return 0;
+
+ cluster = ocfs2_bytes_to_clusters(sb, start);
+ ret = ocfs2_get_clusters(inode, cluster, &p_cluster,
+ &nr_clusters, NULL);
+ if (ret)
+ return ret;
+ if (!p_cluster)
+ return 0;
+
+ offset = start_block - ocfs2_clusters_to_blocks(sb, cluster);
+ p_block = ocfs2_clusters_to_blocks(sb, p_cluster) + offset;
+ return sb_issue_zeroout(sb, p_block, nr_blocks, GFP_NOFS);
+}
+
+/*
* Parts of this function taken from xfs_change_file_space()
*/
static int __ocfs2_change_file_space(struct file *file, struct inode *inode,
{
int ret;
s64 llen;
- loff_t size;
+ loff_t size, orig_isize;
struct ocfs2_super *osb = OCFS2_SB(inode->i_sb);
struct buffer_head *di_bh = NULL;
handle_t *handle;
goto out_inode_unlock;
}
+ orig_isize = i_size_read(inode);
switch (sr->l_whence) {
case 0: /*SEEK_SET*/
break;
sr->l_start += f_pos;
break;
case 2: /*SEEK_END*/
- sr->l_start += i_size_read(inode);
+ sr->l_start += orig_isize;
break;
default:
ret = -EINVAL;
default:
ret = -EINVAL;
}
+
+ /* zeroout eof blocks in the cluster. */
+ if (!ret && change_size && orig_isize < size) {
+ ret = ocfs2_zeroout_partial_cluster(inode, orig_isize,
+ size - orig_isize);
+ if (!ret)
+ i_size_write(inode, size);
+ }
up_write(&OCFS2_I(inode)->ip_alloc_sem);
if (ret) {
mlog_errno(ret);
goto out_inode_unlock;
}
- if (change_size && i_size_read(inode) < size)
- i_size_write(inode, size);
-
inode->i_ctime = inode->i_mtime = current_time(inode);
ret = ocfs2_mark_inode_dirty(handle, inode, di_bh);
if (ret < 0)
}
#ifdef CONFIG_SECURITY
+static int proc_pid_attr_open(struct inode *inode, struct file *file)
+{
+ file->private_data = NULL;
+ __mem_open(inode, file, PTRACE_MODE_READ_FSCREDS);
+ return 0;
+}
+
static ssize_t proc_pid_attr_read(struct file * file, char __user * buf,
size_t count, loff_t *ppos)
{
void *page;
int rv;
+ /* A task may only write when it was the opener. */
+ if (file->private_data != current->mm)
+ return -EPERM;
+
rcu_read_lock();
task = pid_task(proc_pid(inode), PIDTYPE_PID);
if (!task) {
}
static const struct file_operations proc_pid_attr_operations = {
+ .open = proc_pid_attr_open,
.read = proc_pid_attr_read,
.write = proc_pid_attr_write,
.llseek = generic_file_llseek,
+ .release = mem_release,
};
#define LSM_DIR_OPS(LSM) \
static struct dquot *find_dquot(unsigned int hashent, struct super_block *sb,
struct kqid qid)
{
- struct hlist_node *node;
struct dquot *dquot;
- hlist_for_each (node, dquot_hash+hashent) {
- dquot = hlist_entry(node, struct dquot, dq_hash);
+ hlist_for_each_entry(dquot, dquot_hash+hashent, dq_hash)
if (dquot->dq_sb == sb && qid_eq(dquot->dq_id, qid))
return dquot;
- }
+
return NULL;
}
break;
case SIL_FAULT_BNDERR:
case SIL_FAULT_PKUERR:
+ case SIL_PERF_EVENT:
/*
- * Fall through to the SIL_FAULT case. Both SIL_FAULT_BNDERR
- * and SIL_FAULT_PKUERR are only generated by faults that
- * deliver them synchronously to userspace. In case someone
- * injects one of these signals and signalfd catches it treat
- * it as SIL_FAULT.
+ * Fall through to the SIL_FAULT case. SIL_FAULT_BNDERR,
+ * SIL_FAULT_PKUERR, and SIL_PERF_EVENT are only
+ * generated by faults that deliver them synchronously to
+ * userspace. In case someone injects one of these signals
+ * and signalfd catches it treat it as SIL_FAULT.
*/
case SIL_FAULT:
new.ssi_addr = (long) kinfo->si_addr;
-#ifdef __ARCH_SI_TRAPNO
- new.ssi_trapno = kinfo->si_trapno;
-#endif
break;
- case SIL_FAULT_MCEERR:
+ case SIL_FAULT_TRAPNO:
new.ssi_addr = (long) kinfo->si_addr;
-#ifdef __ARCH_SI_TRAPNO
new.ssi_trapno = kinfo->si_trapno;
-#endif
- new.ssi_addr_lsb = (short) kinfo->si_addr_lsb;
break;
- case SIL_PERF_EVENT:
+ case SIL_FAULT_MCEERR:
new.ssi_addr = (long) kinfo->si_addr;
- new.ssi_perf = kinfo->si_perf;
+ new.ssi_addr_lsb = (short) kinfo->si_addr_lsb;
break;
case SIL_CHLD:
new.ssi_pid = kinfo->si_pid;
* If the skip factor is limited in this way then the file will use multiple
* slots.
*/
-static inline int calculate_skip(int blocks)
+static inline int calculate_skip(u64 blocks)
{
- int skip = blocks / ((SQUASHFS_META_ENTRIES + 1)
+ u64 skip = blocks / ((SQUASHFS_META_ENTRIES + 1)
* SQUASHFS_META_INDEXES);
- return min(SQUASHFS_CACHED_BLKS - 1, skip + 1);
+ return min((u64) SQUASHFS_CACHED_BLKS - 1, skip + 1);
}
error2 = xfs_alloc_pagf_init(mp, tp, pag->pag_agno, 0);
if (error2)
return error2;
- ASSERT(xfs_perag_resv(pag, XFS_AG_RESV_METADATA)->ar_reserved +
- xfs_perag_resv(pag, XFS_AG_RESV_RMAPBT)->ar_reserved <=
- pag->pagf_freeblks + pag->pagf_flcount);
+
+ /*
+ * If there isn't enough space in the AG to satisfy the
+ * reservation, let the caller know that there wasn't enough
+ * space. Callers are responsible for deciding what to do
+ * next, since (in theory) we can stumble along with
+ * insufficient reservation if data blocks are being freed to
+ * replenish the AG's free space.
+ */
+ if (!error &&
+ xfs_perag_resv(pag, XFS_AG_RESV_METADATA)->ar_reserved +
+ xfs_perag_resv(pag, XFS_AG_RESV_RMAPBT)->ar_reserved >
+ pag->pagf_freeblks + pag->pagf_flcount)
+ error = -ENOSPC;
}
+
return error;
}
ASSERT(cur);
ASSERT(whichfork != XFS_COW_FORK);
- ASSERT(!xfs_need_iread_extents(ifp));
ASSERT(ifp->if_format == XFS_DINODE_FMT_BTREE);
ASSERT(be16_to_cpu(rblock->bb_level) == 1);
ASSERT(be16_to_cpu(rblock->bb_numrecs) == 1);
xfs_fsblock_t sum;
xfs_filblks_t len = *rlen; /* length to unmap in file */
xfs_fileoff_t max_len;
- xfs_agnumber_t prev_agno = NULLAGNUMBER, agno;
xfs_fileoff_t end;
struct xfs_iext_cursor icur;
bool done = false;
del = got;
wasdel = isnullstartblock(del.br_startblock);
- /*
- * Make sure we don't touch multiple AGF headers out of order
- * in a single transaction, as that could cause AB-BA deadlocks.
- */
- if (!wasdel && !isrt) {
- agno = XFS_FSB_TO_AGNO(mp, del.br_startblock);
- if (prev_agno != NULLAGNUMBER && prev_agno > agno)
- break;
- prev_agno = agno;
- }
if (got.br_startoff < start) {
del.br_startoff = start;
del.br_blockcount -= start - got.br_startoff;
/*
* ioctl commands that are used by Linux filesystems
*/
+#define XFS_IOC_GETXFLAGS FS_IOC_GETFLAGS
+#define XFS_IOC_SETXFLAGS FS_IOC_SETFLAGS
#define XFS_IOC_GETVERSION FS_IOC_GETVERSION
/*
#define XFS_IOC_ALLOCSP _IOW ('X', 10, struct xfs_flock64)
#define XFS_IOC_FREESP _IOW ('X', 11, struct xfs_flock64)
#define XFS_IOC_DIOINFO _IOR ('X', 30, struct dioattr)
+#define XFS_IOC_FSGETXATTR FS_IOC_FSGETXATTR
+#define XFS_IOC_FSSETXATTR FS_IOC_FSSETXATTR
#define XFS_IOC_ALLOCSP64 _IOW ('X', 36, struct xfs_flock64)
#define XFS_IOC_FREESP64 _IOW ('X', 37, struct xfs_flock64)
#define XFS_IOC_GETBMAP _IOWR('X', 38, struct getbmap)
/*
* Validate di_extsize hint.
*
- * The rules are documented at xfs_ioctl_setattr_check_extsize().
- * These functions must be kept in sync with each other.
+ * 1. Extent size hint is only valid for directories and regular files.
+ * 2. FS_XFLAG_EXTSIZE is only valid for regular files.
+ * 3. FS_XFLAG_EXTSZINHERIT is only valid for directories.
+ * 4. Hint cannot be larger than MAXTEXTLEN.
+ * 5. Can be changed on directories at any time.
+ * 6. Hint value of 0 turns off hints, clears inode flags.
+ * 7. Extent size must be a multiple of the appropriate block size.
+ * For realtime files, this is the rt extent size.
+ * 8. For non-realtime files, the extent size hint must be limited
+ * to half the AG size to avoid alignment extending the extent beyond the
+ * limits of the AG.
*/
xfs_failaddr_t
xfs_inode_validate_extsize(
inherit_flag = (flags & XFS_DIFLAG_EXTSZINHERIT);
extsize_bytes = XFS_FSB_TO_B(mp, extsize);
+ /*
+ * This comment describes a historic gap in this verifier function.
+ *
+ * On older kernels, the extent size hint verifier doesn't check that
+ * the extent size hint is an integer multiple of the realtime extent
+ * size on a directory with both RTINHERIT and EXTSZINHERIT flags set.
+ * The verifier has always enforced the alignment rule for regular
+ * files with the REALTIME flag set.
+ *
+ * If a directory with a misaligned extent size hint is allowed to
+ * propagate that hint into a new regular realtime file, the result
+ * is that the inode cluster buffer verifier will trigger a corruption
+ * shutdown the next time it is run.
+ *
+ * Unfortunately, there could be filesystems with these misconfigured
+ * directories in the wild, so we cannot add a check to this verifier
+ * at this time because that will result a new source of directory
+ * corruption errors when reading an existing filesystem. Instead, we
+ * permit the misconfiguration to pass through the verifiers so that
+ * callers of this function can correct and mitigate externally.
+ */
+
if (rt_flag)
blocksize_bytes = mp->m_sb.sb_rextsize << mp->m_sb.sb_blocklog;
else
/*
* Validate di_cowextsize hint.
*
- * The rules are documented at xfs_ioctl_setattr_check_cowextsize().
- * These functions must be kept in sync with each other.
+ * 1. CoW extent size hint can only be set if reflink is enabled on the fs.
+ * The inode does not have to have any shared blocks, but it must be a v3.
+ * 2. FS_XFLAG_COWEXTSIZE is only valid for directories and regular files;
+ * for a directory, the hint is propagated to new files.
+ * 3. Can be changed on files & directories at any time.
+ * 4. Hint value of 0 turns off hints, clears inode flags.
+ * 5. Extent size must be a multiple of the appropriate block size.
+ * 6. The extent size hint must be limited to half the AG size to avoid
+ * alignment extending the extent beyond the limits of the AG.
*/
xfs_failaddr_t
xfs_inode_validate_cowextsize(
}
/*
+ * Inode verifiers on older kernels don't check that the extent size
+ * hint is an integer multiple of the rt extent size on a directory
+ * with both rtinherit and extszinherit flags set. If we're logging a
+ * directory that is misconfigured in this way, clear the hint.
+ */
+ if ((ip->i_diflags & XFS_DIFLAG_RTINHERIT) &&
+ (ip->i_diflags & XFS_DIFLAG_EXTSZINHERIT) &&
+ (ip->i_extsize % ip->i_mount->m_sb.sb_rextsize) > 0) {
+ xfs_info_once(ip->i_mount,
+ "Correcting misaligned extent size hint in inode 0x%llx.", ip->i_ino);
+ ip->i_diflags &= ~(XFS_DIFLAG_EXTSIZE |
+ XFS_DIFLAG_EXTSZINHERIT);
+ ip->i_extsize = 0;
+ flags |= XFS_ILOG_CORE;
+ }
+
+ /*
* Record the specific change for fdatasync optimisation. This allows
* fdatasync to skip log forces for inodes that are only timestamp
* dirty.
return true;
case -EDEADLOCK:
/* Used to restart an op with deadlock avoidance. */
- trace_xchk_deadlock_retry(sc->ip, sc->sm, *error);
+ trace_xchk_deadlock_retry(
+ sc->ip ? sc->ip : XFS_I(file_inode(sc->file)),
+ sc->sm, *error);
break;
case -EFSBADCRC:
case -EFSCORRUPTED:
#ifdef CONFIG_XFS_RT
int
xfs_bmap_rtalloc(
- struct xfs_bmalloca *ap) /* bmap alloc argument struct */
+ struct xfs_bmalloca *ap)
{
- int error; /* error return value */
- xfs_mount_t *mp; /* mount point structure */
- xfs_extlen_t prod = 0; /* product factor for allocators */
- xfs_extlen_t mod = 0; /* product factor for allocators */
- xfs_extlen_t ralen = 0; /* realtime allocation length */
- xfs_extlen_t align; /* minimum allocation alignment */
- xfs_rtblock_t rtb;
-
- mp = ap->ip->i_mount;
+ struct xfs_mount *mp = ap->ip->i_mount;
+ xfs_fileoff_t orig_offset = ap->offset;
+ xfs_rtblock_t rtb;
+ xfs_extlen_t prod = 0; /* product factor for allocators */
+ xfs_extlen_t mod = 0; /* product factor for allocators */
+ xfs_extlen_t ralen = 0; /* realtime allocation length */
+ xfs_extlen_t align; /* minimum allocation alignment */
+ xfs_extlen_t orig_length = ap->length;
+ xfs_extlen_t minlen = mp->m_sb.sb_rextsize;
+ xfs_extlen_t raminlen;
+ bool rtlocked = false;
+ bool ignore_locality = false;
+ int error;
+
align = xfs_get_extsz_hint(ap->ip);
+retry:
prod = align / mp->m_sb.sb_rextsize;
error = xfs_bmap_extsize_align(mp, &ap->got, &ap->prev,
align, 1, ap->eof, 0,
ASSERT(ap->length % mp->m_sb.sb_rextsize == 0);
/*
+ * If we shifted the file offset downward to satisfy an extent size
+ * hint, increase minlen by that amount so that the allocator won't
+ * give us an allocation that's too short to cover at least one of the
+ * blocks that the caller asked for.
+ */
+ if (ap->offset != orig_offset)
+ minlen += orig_offset - ap->offset;
+
+ /*
* If the offset & length are not perfectly aligned
* then kill prod, it will just get us in trouble.
*/
/*
* Lock out modifications to both the RT bitmap and summary inodes
*/
- xfs_ilock(mp->m_rbmip, XFS_ILOCK_EXCL|XFS_ILOCK_RTBITMAP);
- xfs_trans_ijoin(ap->tp, mp->m_rbmip, XFS_ILOCK_EXCL);
- xfs_ilock(mp->m_rsumip, XFS_ILOCK_EXCL|XFS_ILOCK_RTSUM);
- xfs_trans_ijoin(ap->tp, mp->m_rsumip, XFS_ILOCK_EXCL);
+ if (!rtlocked) {
+ xfs_ilock(mp->m_rbmip, XFS_ILOCK_EXCL|XFS_ILOCK_RTBITMAP);
+ xfs_trans_ijoin(ap->tp, mp->m_rbmip, XFS_ILOCK_EXCL);
+ xfs_ilock(mp->m_rsumip, XFS_ILOCK_EXCL|XFS_ILOCK_RTSUM);
+ xfs_trans_ijoin(ap->tp, mp->m_rsumip, XFS_ILOCK_EXCL);
+ rtlocked = true;
+ }
/*
* If it's an allocation to an empty file at offset 0,
/*
* Realtime allocation, done through xfs_rtallocate_extent.
*/
- do_div(ap->blkno, mp->m_sb.sb_rextsize);
+ if (ignore_locality)
+ ap->blkno = 0;
+ else
+ do_div(ap->blkno, mp->m_sb.sb_rextsize);
rtb = ap->blkno;
ap->length = ralen;
- error = xfs_rtallocate_extent(ap->tp, ap->blkno, 1, ap->length,
- &ralen, ap->wasdel, prod, &rtb);
+ raminlen = max_t(xfs_extlen_t, 1, minlen / mp->m_sb.sb_rextsize);
+ error = xfs_rtallocate_extent(ap->tp, ap->blkno, raminlen, ap->length,
+ &ralen, ap->wasdel, prod, &rtb);
if (error)
return error;
- ap->blkno = rtb;
- if (ap->blkno != NULLFSBLOCK) {
- ap->blkno *= mp->m_sb.sb_rextsize;
- ralen *= mp->m_sb.sb_rextsize;
- ap->length = ralen;
- ap->ip->i_nblocks += ralen;
+ if (rtb != NULLRTBLOCK) {
+ ap->blkno = rtb * mp->m_sb.sb_rextsize;
+ ap->length = ralen * mp->m_sb.sb_rextsize;
+ ap->ip->i_nblocks += ap->length;
xfs_trans_log_inode(ap->tp, ap->ip, XFS_ILOG_CORE);
if (ap->wasdel)
- ap->ip->i_delayed_blks -= ralen;
+ ap->ip->i_delayed_blks -= ap->length;
/*
* Adjust the disk quota also. This was reserved
* earlier.
*/
xfs_trans_mod_dquot_byino(ap->tp, ap->ip,
ap->wasdel ? XFS_TRANS_DQ_DELRTBCOUNT :
- XFS_TRANS_DQ_RTBCOUNT, (long) ralen);
- } else {
- ap->length = 0;
+ XFS_TRANS_DQ_RTBCOUNT, ap->length);
+ return 0;
}
+
+ if (align > mp->m_sb.sb_rextsize) {
+ /*
+ * We previously enlarged the request length to try to satisfy
+ * an extent size hint. The allocator didn't return anything,
+ * so reset the parameters to the original values and try again
+ * without alignment criteria.
+ */
+ ap->offset = orig_offset;
+ ap->length = orig_length;
+ minlen = align = mp->m_sb.sb_rextsize;
+ goto retry;
+ }
+
+ if (!ignore_locality && ap->blkno != 0) {
+ /*
+ * If we can't allocate near a specific rt extent, try again
+ * without locality criteria.
+ */
+ ignore_locality = true;
+ goto retry;
+ }
+
+ ap->blkno = NULLFSBLOCK;
+ ap->length = 0;
return 0;
}
#endif /* CONFIG_XFS_RT */
const struct xfs_inode *pip)
{
unsigned int di_flags = 0;
+ xfs_failaddr_t failaddr;
umode_t mode = VFS_I(ip)->i_mode;
if (S_ISDIR(mode)) {
di_flags |= XFS_DIFLAG_FILESTREAM;
ip->i_diflags |= di_flags;
+
+ /*
+ * Inode verifiers on older kernels only check that the extent size
+ * hint is an integer multiple of the rt extent size on realtime files.
+ * They did not check the hint alignment on a directory with both
+ * rtinherit and extszinherit flags set. If the misaligned hint is
+ * propagated from a directory into a new realtime file, new file
+ * allocations will fail due to math errors in the rt allocator and/or
+ * trip the verifiers. Validate the hint settings in the new file so
+ * that we don't let broken hints propagate.
+ */
+ failaddr = xfs_inode_validate_extsize(ip->i_mount, ip->i_extsize,
+ VFS_I(ip)->i_mode, ip->i_diflags);
+ if (failaddr) {
+ ip->i_diflags &= ~(XFS_DIFLAG_EXTSIZE |
+ XFS_DIFLAG_EXTSZINHERIT);
+ ip->i_extsize = 0;
+ }
}
/* Propagate di_flags2 from a parent inode to a child inode. */
struct xfs_inode *ip,
const struct xfs_inode *pip)
{
+ xfs_failaddr_t failaddr;
+
if (pip->i_diflags2 & XFS_DIFLAG2_COWEXTSIZE) {
ip->i_diflags2 |= XFS_DIFLAG2_COWEXTSIZE;
ip->i_cowextsize = pip->i_cowextsize;
}
if (pip->i_diflags2 & XFS_DIFLAG2_DAX)
ip->i_diflags2 |= XFS_DIFLAG2_DAX;
+
+ /* Don't let invalid cowextsize hints propagate. */
+ failaddr = xfs_inode_validate_cowextsize(ip->i_mount, ip->i_cowextsize,
+ VFS_I(ip)->i_mode, ip->i_diflags, ip->i_diflags2);
+ if (failaddr) {
+ ip->i_diflags2 &= ~XFS_DIFLAG2_COWEXTSIZE;
+ ip->i_cowextsize = 0;
+ }
}
/*
}
/*
- * extent size hint validation is somewhat cumbersome. Rules are:
- *
- * 1. extent size hint is only valid for directories and regular files
- * 2. FS_XFLAG_EXTSIZE is only valid for regular files
- * 3. FS_XFLAG_EXTSZINHERIT is only valid for directories.
- * 4. can only be changed on regular files if no extents are allocated
- * 5. can be changed on directories at any time
- * 6. extsize hint of 0 turns off hints, clears inode flags.
- * 7. Extent size must be a multiple of the appropriate block size.
- * 8. for non-realtime files, the extent size hint must be limited
- * to half the AG size to avoid alignment extending the extent beyond the
- * limits of the AG.
- *
- * Please keep this function in sync with xfs_scrub_inode_extsize.
+ * Validate a proposed extent size hint. For regular files, the hint can only
+ * be changed if no extents are allocated.
*/
static int
xfs_ioctl_setattr_check_extsize(
struct fileattr *fa)
{
struct xfs_mount *mp = ip->i_mount;
- xfs_extlen_t size;
- xfs_fsblock_t extsize_fsb;
+ xfs_failaddr_t failaddr;
+ uint16_t new_diflags;
if (!fa->fsx_valid)
return 0;
if (S_ISREG(VFS_I(ip)->i_mode) && ip->i_df.if_nextents &&
- ((ip->i_extsize << mp->m_sb.sb_blocklog) != fa->fsx_extsize))
+ XFS_FSB_TO_B(mp, ip->i_extsize) != fa->fsx_extsize)
return -EINVAL;
- if (fa->fsx_extsize == 0)
- return 0;
-
- extsize_fsb = XFS_B_TO_FSB(mp, fa->fsx_extsize);
- if (extsize_fsb > MAXEXTLEN)
+ if (fa->fsx_extsize & mp->m_blockmask)
return -EINVAL;
- if (XFS_IS_REALTIME_INODE(ip) ||
- (fa->fsx_xflags & FS_XFLAG_REALTIME)) {
- size = mp->m_sb.sb_rextsize << mp->m_sb.sb_blocklog;
- } else {
- size = mp->m_sb.sb_blocksize;
- if (extsize_fsb > mp->m_sb.sb_agblocks / 2)
+ new_diflags = xfs_flags2diflags(ip, fa->fsx_xflags);
+
+ /*
+ * Inode verifiers on older kernels don't check that the extent size
+ * hint is an integer multiple of the rt extent size on a directory
+ * with both rtinherit and extszinherit flags set. Don't let sysadmins
+ * misconfigure directories.
+ */
+ if ((new_diflags & XFS_DIFLAG_RTINHERIT) &&
+ (new_diflags & XFS_DIFLAG_EXTSZINHERIT)) {
+ unsigned int rtextsize_bytes;
+
+ rtextsize_bytes = XFS_FSB_TO_B(mp, mp->m_sb.sb_rextsize);
+ if (fa->fsx_extsize % rtextsize_bytes)
return -EINVAL;
}
- if (fa->fsx_extsize % size)
- return -EINVAL;
-
- return 0;
+ failaddr = xfs_inode_validate_extsize(ip->i_mount,
+ XFS_B_TO_FSB(mp, fa->fsx_extsize),
+ VFS_I(ip)->i_mode, new_diflags);
+ return failaddr != NULL ? -EINVAL : 0;
}
-/*
- * CoW extent size hint validation rules are:
- *
- * 1. CoW extent size hint can only be set if reflink is enabled on the fs.
- * The inode does not have to have any shared blocks, but it must be a v3.
- * 2. FS_XFLAG_COWEXTSIZE is only valid for directories and regular files;
- * for a directory, the hint is propagated to new files.
- * 3. Can be changed on files & directories at any time.
- * 4. CoW extsize hint of 0 turns off hints, clears inode flags.
- * 5. Extent size must be a multiple of the appropriate block size.
- * 6. The extent size hint must be limited to half the AG size to avoid
- * alignment extending the extent beyond the limits of the AG.
- *
- * Please keep this function in sync with xfs_scrub_inode_cowextsize.
- */
static int
xfs_ioctl_setattr_check_cowextsize(
struct xfs_inode *ip,
struct fileattr *fa)
{
struct xfs_mount *mp = ip->i_mount;
- xfs_extlen_t size;
- xfs_fsblock_t cowextsize_fsb;
+ xfs_failaddr_t failaddr;
+ uint64_t new_diflags2;
+ uint16_t new_diflags;
if (!fa->fsx_valid)
return 0;
- if (!(fa->fsx_xflags & FS_XFLAG_COWEXTSIZE))
- return 0;
-
- if (!xfs_sb_version_hasreflink(&ip->i_mount->m_sb))
+ if (fa->fsx_cowextsize & mp->m_blockmask)
return -EINVAL;
- if (fa->fsx_cowextsize == 0)
- return 0;
+ new_diflags = xfs_flags2diflags(ip, fa->fsx_xflags);
+ new_diflags2 = xfs_flags2diflags2(ip, fa->fsx_xflags);
- cowextsize_fsb = XFS_B_TO_FSB(mp, fa->fsx_cowextsize);
- if (cowextsize_fsb > MAXEXTLEN)
- return -EINVAL;
-
- size = mp->m_sb.sb_blocksize;
- if (cowextsize_fsb > mp->m_sb.sb_agblocks / 2)
- return -EINVAL;
-
- if (fa->fsx_cowextsize % size)
- return -EINVAL;
-
- return 0;
+ failaddr = xfs_inode_validate_cowextsize(ip->i_mount,
+ XFS_B_TO_FSB(mp, fa->fsx_cowextsize),
+ VFS_I(ip)->i_mode, new_diflags, new_diflags2);
+ return failaddr != NULL ? -EINVAL : 0;
}
static int
xfs_printk_once(xfs_warn, dev, fmt, ##__VA_ARGS__)
#define xfs_notice_once(dev, fmt, ...) \
xfs_printk_once(xfs_notice, dev, fmt, ##__VA_ARGS__)
+#define xfs_info_once(dev, fmt, ...) \
+ xfs_printk_once(xfs_info, dev, fmt, ##__VA_ARGS__)
void assfail(struct xfs_mount *mp, char *expr, char *f, int l);
void asswarn(struct xfs_mount *mp, char *expr, char *f, int l);
#ifdef CONFIG_AMD_MEM_ENCRYPT
#define PERCPU_DECRYPTED_SECTION \
. = ALIGN(PAGE_SIZE); \
+ *(.data..decrypted) \
*(.data..percpu..decrypted) \
. = ALIGN(PAGE_SIZE);
#else
* <20:16> :: Reserved, Shall be set to zero
* <15:0> :: USB-IF assigned VID for this cable vendor
*/
+
+/* PD Rev2.0 definition */
+#define IDH_PTYPE_UNDEF 0
+
/* SOP Product Type (UFP) */
#define IDH_PTYPE_NOT_UFP 0
#define IDH_PTYPE_HUB 1
#define UFP_VDO_VER1_2 2
/* Device Capability */
-#define DEV_USB2_CAPABLE BIT(0)
-#define DEV_USB2_BILLBOARD BIT(1)
-#define DEV_USB3_CAPABLE BIT(2)
-#define DEV_USB4_CAPABLE BIT(3)
+#define DEV_USB2_CAPABLE (1 << 0)
+#define DEV_USB2_BILLBOARD (1 << 1)
+#define DEV_USB3_CAPABLE (1 << 2)
+#define DEV_USB4_CAPABLE (1 << 3)
/* Connector Type */
#define UFP_RECEPTACLE 2
/* Alternate Modes */
#define UFP_ALTMODE_NOT_SUPP 0
-#define UFP_ALTMODE_TBT3 BIT(0)
-#define UFP_ALTMODE_RECFG BIT(1)
-#define UFP_ALTMODE_NO_RECFG BIT(2)
+#define UFP_ALTMODE_TBT3 (1 << 0)
+#define UFP_ALTMODE_RECFG (1 << 1)
+#define UFP_ALTMODE_NO_RECFG (1 << 2)
/* USB Highest Speed */
#define UFP_USB2_ONLY 0
* <4:0> :: Port number
*/
#define DFP_VDO_VER1_1 1
-#define HOST_USB2_CAPABLE BIT(0)
-#define HOST_USB3_CAPABLE BIT(1)
-#define HOST_USB4_CAPABLE BIT(2)
+#define HOST_USB2_CAPABLE (1 << 0)
+#define HOST_USB3_CAPABLE (1 << 1)
+#define HOST_USB4_CAPABLE (1 << 2)
#define DFP_RECEPTACLE 2
#define DFP_CAPTIVE 3
| ((pnum) & 0x1f))
/*
- * Passive Cable VDO
+ * Cable VDO (for both Passive and Active Cable VDO in PD Rev2.0)
+ * ---------
+ * <31:28> :: Cable HW version
+ * <27:24> :: Cable FW version
+ * <23:20> :: Reserved, Shall be set to zero
+ * <19:18> :: type-C to Type-A/B/C/Captive (00b == A, 01 == B, 10 == C, 11 == Captive)
+ * <17> :: Reserved, Shall be set to zero
+ * <16:13> :: cable latency (0001 == <10ns(~1m length))
+ * <12:11> :: cable termination type (11b == both ends active VCONN req)
+ * <10> :: SSTX1 Directionality support (0b == fixed, 1b == cfgable)
+ * <9> :: SSTX2 Directionality support
+ * <8> :: SSRX1 Directionality support
+ * <7> :: SSRX2 Directionality support
+ * <6:5> :: Vbus current handling capability (01b == 3A, 10b == 5A)
+ * <4> :: Vbus through cable (0b == no, 1b == yes)
+ * <3> :: SOP" controller present? (0b == no, 1b == yes)
+ * <2:0> :: USB SS Signaling support
+ *
+ * Passive Cable VDO (PD Rev3.0+)
* ---------
* <31:28> :: Cable HW version
* <27:24> :: Cable FW version
* <4:3> :: Reserved, Shall be set to zero
* <2:0> :: USB highest speed
*
- * Active Cable VDO 1
+ * Active Cable VDO 1 (PD Rev3.0+)
* ---------
* <31:28> :: Cable HW version
* <27:24> :: Cable FW version
#define CABLE_VDO_VER1_0 0
#define CABLE_VDO_VER1_3 3
-/* Connector Type */
+/* Connector Type (_ATYPE and _BTYPE are for PD Rev2.0 only) */
+#define CABLE_ATYPE 0
+#define CABLE_BTYPE 1
#define CABLE_CTYPE 2
#define CABLE_CAPTIVE 3
#define CABLE_CURR_3A 1
#define CABLE_CURR_5A 2
+/* USB SuperSpeed Signaling Support (PD Rev2.0) */
+#define CABLE_USBSS_U2_ONLY 0
+#define CABLE_USBSS_U31_GEN1 1
+#define CABLE_USBSS_U31_GEN2 2
+
/* USB Highest Speed */
#define CABLE_USB2_ONLY 0
#define CABLE_USB32_GEN1 1
#define CABLE_USB32_4_GEN2 2
#define CABLE_USB4_GEN3 3
+#define VDO_CABLE(hw, fw, cbl, lat, term, tx1d, tx2d, rx1d, rx2d, cur, vps, sopp, usbss) \
+ (((hw) & 0x7) << 28 | ((fw) & 0x7) << 24 | ((cbl) & 0x3) << 18 \
+ | ((lat) & 0x7) << 13 | ((term) & 0x3) << 11 | (tx1d) << 10 \
+ | (tx2d) << 9 | (rx1d) << 8 | (rx2d) << 7 | ((cur) & 0x3) << 5 \
+ | (vps) << 4 | (sopp) << 3 | ((usbss) & 0x7))
#define VDO_PCABLE(hw, fw, ver, conn, lat, term, vbm, cur, spd) \
(((hw) & 0xf) << 28 | ((fw) & 0xf) << 24 | ((ver) & 0x7) << 21 \
| ((conn) & 0x3) << 18 | ((lat) & 0xf) << 13 | ((term) & 0x3) << 11 \
| (iso) << 2 | (gen))
/*
+ * AMA VDO (PD Rev2.0)
+ * ---------
+ * <31:28> :: Cable HW version
+ * <27:24> :: Cable FW version
+ * <23:12> :: Reserved, Shall be set to zero
+ * <11> :: SSTX1 Directionality support (0b == fixed, 1b == cfgable)
+ * <10> :: SSTX2 Directionality support
+ * <9> :: SSRX1 Directionality support
+ * <8> :: SSRX2 Directionality support
+ * <7:5> :: Vconn power
+ * <4> :: Vconn power required
+ * <3> :: Vbus power required
+ * <2:0> :: USB SS Signaling support
+ */
+#define VDO_AMA(hw, fw, tx1d, tx2d, rx1d, rx2d, vcpwr, vcr, vbr, usbss) \
+ (((hw) & 0x7) << 28 | ((fw) & 0x7) << 24 \
+ | (tx1d) << 11 | (tx2d) << 10 | (rx1d) << 9 | (rx2d) << 8 \
+ | ((vcpwr) & 0x7) << 5 | (vcr) << 4 | (vbr) << 3 \
+ | ((usbss) & 0x7))
+
+#define PD_VDO_AMA_VCONN_REQ(vdo) (((vdo) >> 4) & 1)
+#define PD_VDO_AMA_VBUS_REQ(vdo) (((vdo) >> 3) & 1)
+
+#define AMA_USBSS_U2_ONLY 0
+#define AMA_USBSS_U31_GEN1 1
+#define AMA_USBSS_U31_GEN2 2
+#define AMA_USBSS_BBONLY 3
+
+/*
* VPD VDO
* ---------
* <31:28> :: HW version
}
#endif
+int acpi_get_local_address(acpi_handle handle, u32 *addr);
+
#else /* !CONFIG_ACPI */
#define acpi_disabled 1
return NULL;
}
+static inline int acpi_get_local_address(acpi_handle handle, u32 *addr)
+{
+ return -ENODEV;
+}
+
#endif /* !CONFIG_ACPI */
#ifdef CONFIG_ACPI_HOTPLUG_IOAPIC
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * ACPI helper for the MDIO (Ethernet PHY) API
+ */
+
+#ifndef __LINUX_ACPI_MDIO_H
+#define __LINUX_ACPI_MDIO_H
+
+#include <linux/phy.h>
+
+#if IS_ENABLED(CONFIG_ACPI_MDIO)
+int acpi_mdiobus_register(struct mii_bus *mdio, struct fwnode_handle *fwnode);
+#else /* CONFIG_ACPI_MDIO */
+static inline int
+acpi_mdiobus_register(struct mii_bus *mdio, struct fwnode_handle *fwnode)
+{
+ /*
+ * Fall back to mdiobus_register() function to register a bus.
+ * This way, we don't have to keep compat bits around in drivers.
+ */
+
+ return mdiobus_register(mdio);
+}
+#endif
+
+#endif /* __LINUX_ACPI_MDIO_H */
enum scale_freq_source {
SCALE_FREQ_SOURCE_CPUFREQ = 0,
SCALE_FREQ_SOURCE_ARCH,
- SCALE_FREQ_SOURCE_CPPC,
};
struct scale_freq_data {
* PF removes the filters and returns status.
*/
+/* VIRTCHNL_ETHER_ADDR_LEGACY
+ * Prior to adding the @type member to virtchnl_ether_addr, there were 2 pad
+ * bytes. Moving forward all VF drivers should not set type to
+ * VIRTCHNL_ETHER_ADDR_LEGACY. This is only here to not break previous/legacy
+ * behavior. The control plane function (i.e. PF) can use a best effort method
+ * of tracking the primary/device unicast in this case, but there is no
+ * guarantee and functionality depends on the implementation of the PF.
+ */
+
+/* VIRTCHNL_ETHER_ADDR_PRIMARY
+ * All VF drivers should set @type to VIRTCHNL_ETHER_ADDR_PRIMARY for the
+ * primary/device unicast MAC address filter for VIRTCHNL_OP_ADD_ETH_ADDR and
+ * VIRTCHNL_OP_DEL_ETH_ADDR. This allows for the underlying control plane
+ * function (i.e. PF) to accurately track and use this MAC address for
+ * displaying on the host and for VM/function reset.
+ */
+
+/* VIRTCHNL_ETHER_ADDR_EXTRA
+ * All VF drivers should set @type to VIRTCHNL_ETHER_ADDR_EXTRA for any extra
+ * unicast and/or multicast filters that are being added/deleted via
+ * VIRTCHNL_OP_DEL_ETH_ADDR/VIRTCHNL_OP_ADD_ETH_ADDR respectively.
+ */
struct virtchnl_ether_addr {
u8 addr[ETH_ALEN];
- u8 pad[2];
+ u8 type;
+#define VIRTCHNL_ETHER_ADDR_LEGACY 0
+#define VIRTCHNL_ETHER_ADDR_PRIMARY 1
+#define VIRTCHNL_ETHER_ADDR_EXTRA 2
+#define VIRTCHNL_ETHER_ADDR_TYPE_MASK 3 /* first two bits of type are valid */
+ u8 pad;
};
VIRTCHNL_CHECK_STRUCT_LEN(8, virtchnl_ether_addr);
struct virtchnl_proto_hdrs {
u8 tunnel_level;
+ u8 pad[3];
/**
* specify where protocol header start from.
* 0 - from the outer layer
return NULL;
}
-extern unsigned int bio_max_size(struct bio *bio);
-
/**
* bio_full - check if the bio is full
* @bio: bio to check
if (bio->bi_vcnt >= bio->bi_max_vecs)
return true;
- if (bio->bi_iter.bi_size > bio_max_size(bio) - len)
+ if (bio->bi_iter.bi_size > UINT_MAX - len)
return true;
return false;
#include <linux/build_bug.h>
#define GENMASK_INPUT_CHECK(h, l) \
(BUILD_BUG_ON_ZERO(__builtin_choose_expr( \
- __builtin_constant_p((l) > (h)), (l) > (h), 0)))
+ __is_constexpr((l) > (h)), (l) > (h), 0)))
#else
/*
* BUILD_BUG_ON_ZERO is not available in h files included from asm files,
};
struct queue_limits {
- unsigned int bio_max_bytes;
-
enum blk_bounce bounce;
unsigned long seg_boundary_mask;
unsigned long virt_boundary_mask;
extern void blk_set_pm_only(struct request_queue *q);
extern void blk_clear_pm_only(struct request_queue *q);
-static inline bool blk_account_rq(struct request *rq)
-{
- return (rq->rq_flags & RQF_STARTED) && !blk_rq_is_passthrough(rq);
-}
-
#define list_entry_rq(ptr) list_entry((ptr), struct request, queuelist)
#define rq_data_dir(rq) (op_is_write(req_op(rq)) ? WRITE : READ)
#include <linux/sched/mm.h>
#include <linux/slab.h>
#include <linux/percpu-refcount.h>
+#include <linux/bpfptr.h>
struct bpf_verifier_env;
struct bpf_verifier_log;
void *(*map_lookup_elem_sys_only)(struct bpf_map *map, void *key);
int (*map_lookup_batch)(struct bpf_map *map, const union bpf_attr *attr,
union bpf_attr __user *uattr);
+ int (*map_lookup_and_delete_elem)(struct bpf_map *map, void *key,
+ void *value, u64 flags);
int (*map_lookup_and_delete_batch)(struct bpf_map *map,
const union bpf_attr *attr,
union bpf_attr __user *uattr);
int bpf_iter_reg_target(const struct bpf_iter_reg *reg_info);
void bpf_iter_unreg_target(const struct bpf_iter_reg *reg_info);
bool bpf_iter_prog_supported(struct bpf_prog *prog);
-int bpf_iter_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
+int bpf_iter_link_attach(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_prog *prog);
int bpf_iter_new_fd(struct bpf_link *link);
bool bpf_link_is_iter(struct bpf_link *link);
struct bpf_prog *bpf_iter_get_info(struct bpf_iter_meta *meta, bool in_stop);
int bpf_fd_htab_map_lookup_elem(struct bpf_map *map, void *key, u32 *value);
int bpf_get_file_flag(int flags);
-int bpf_check_uarg_tail_zero(void __user *uaddr, size_t expected_size,
+int bpf_check_uarg_tail_zero(bpfptr_t uaddr, size_t expected_size,
size_t actual_size);
/* memcpy that is used with 8-byte aligned pointers, power-of-8 size and
}
/* verify correctness of eBPF program */
-int bpf_check(struct bpf_prog **fp, union bpf_attr *attr,
- union bpf_attr __user *uattr);
+int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr);
#ifndef CONFIG_BPF_JIT_ALWAYS_ON
void bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth);
struct net_device *dev_rx);
int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp,
struct net_device *dev_rx);
+int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx,
+ struct bpf_map *map, bool exclude_ingress);
int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb,
struct bpf_prog *xdp_prog);
+int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb,
+ struct bpf_prog *xdp_prog, struct bpf_map *map,
+ bool exclude_ingress);
bool dev_map_can_have_prog(struct bpf_map *map);
void __cpu_map_flush(void);
return 0;
}
+static inline
+int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx,
+ struct bpf_map *map, bool exclude_ingress)
+{
+ return 0;
+}
+
struct sk_buff;
static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst,
return 0;
}
+static inline
+int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb,
+ struct bpf_prog *xdp_prog, struct bpf_map *map,
+ bool exclude_ingress)
+{
+ return 0;
+}
+
static inline void __cpu_map_flush(void)
{
}
struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr);
void bpf_map_offload_map_free(struct bpf_map *map);
+int bpf_prog_test_run_syscall(struct bpf_prog *prog,
+ const union bpf_attr *kattr,
+ union bpf_attr __user *uattr);
#else
static inline int bpf_prog_offload_init(struct bpf_prog *prog,
union bpf_attr *attr)
static inline void bpf_map_offload_map_free(struct bpf_map *map)
{
}
+
+static inline int bpf_prog_test_run_syscall(struct bpf_prog *prog,
+ const union bpf_attr *kattr,
+ union bpf_attr __user *uattr)
+{
+ return -ENOTSUPP;
+}
#endif /* CONFIG_NET && CONFIG_BPF_SYSCALL */
#if defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL)
extern const struct bpf_func_proto bpf_task_storage_get_proto;
extern const struct bpf_func_proto bpf_task_storage_delete_proto;
extern const struct bpf_func_proto bpf_for_each_map_elem_proto;
+extern const struct bpf_func_proto bpf_btf_find_by_name_kind_proto;
const struct bpf_func_proto *bpf_tracing_func_proto(
enum bpf_func_id func_id, const struct bpf_prog *prog);
struct sk_buff *skb;
struct sock *sk;
struct sock *selected_sk;
+ struct sock *migrating_sk;
void *data_end;
u32 hash;
u32 reuseport_id;
* from the object's bpf_local_storage.
*
* Put it in the same cacheline as the data to minimize
- * the number of cachelines access during the cache hit case.
+ * the number of cachelines accessed during the cache hit case.
*/
struct bpf_local_storage_map __rcu *smap;
u8 data[] __aligned(8);
struct bpf_local_storage __rcu *local_storage;
struct rcu_head rcu;
/* 8 bytes hole */
- /* The data is stored in aother cacheline to minimize
+ /* The data is stored in another cacheline to minimize
* the number of cachelines access during a cache hit.
*/
struct bpf_local_storage_data sdata ____cacheline_aligned;
void *, void *)
#endif /* CONFIG_BPF_LSM */
#endif
+BPF_PROG_TYPE(BPF_PROG_TYPE_SYSCALL, bpf_syscall,
+ void *, void *)
BPF_MAP_TYPE(BPF_MAP_TYPE_ARRAY, array_map_ops)
BPF_MAP_TYPE(BPF_MAP_TYPE_PERCPU_ARRAY, percpu_array_map_ops)
u32 idx;
};
+struct bpf_id_pair {
+ u32 old;
+ u32 cur;
+};
+
+/* Maximum number of register states that can exist at once */
+#define BPF_ID_MAP_SIZE (MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE)
#define MAX_CALL_FRAMES 8
struct bpf_verifier_state {
/* call stack tracking */
const struct bpf_line_info *prev_linfo;
struct bpf_verifier_log log;
struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 1];
+ struct bpf_id_pair idmap_scratch[BPF_ID_MAP_SIZE];
struct {
int *insn_state;
int *insn_stack;
u32 peak_states;
/* longest register parentage chain walked for liveness marking */
u32 longest_mark_read_walk;
+ bpfptr_t fd_array;
};
__printf(2, 0) void bpf_verifier_vlog(struct bpf_verifier_log *log,
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* A pointer that can point to either kernel or userspace memory. */
+#ifndef _LINUX_BPFPTR_H
+#define _LINUX_BPFPTR_H
+
+#include <linux/sockptr.h>
+
+typedef sockptr_t bpfptr_t;
+
+static inline bool bpfptr_is_kernel(bpfptr_t bpfptr)
+{
+ return bpfptr.is_kernel;
+}
+
+static inline bpfptr_t KERNEL_BPFPTR(void *p)
+{
+ return (bpfptr_t) { .kernel = p, .is_kernel = true };
+}
+
+static inline bpfptr_t USER_BPFPTR(void __user *p)
+{
+ return (bpfptr_t) { .user = p };
+}
+
+static inline bpfptr_t make_bpfptr(u64 addr, bool is_kernel)
+{
+ if (is_kernel)
+ return KERNEL_BPFPTR((void*) (uintptr_t) addr);
+ else
+ return USER_BPFPTR(u64_to_user_ptr(addr));
+}
+
+static inline bool bpfptr_is_null(bpfptr_t bpfptr)
+{
+ if (bpfptr_is_kernel(bpfptr))
+ return !bpfptr.kernel;
+ return !bpfptr.user;
+}
+
+static inline void bpfptr_add(bpfptr_t *bpfptr, size_t val)
+{
+ if (bpfptr_is_kernel(*bpfptr))
+ bpfptr->kernel += val;
+ else
+ bpfptr->user += val;
+}
+
+static inline int copy_from_bpfptr_offset(void *dst, bpfptr_t src,
+ size_t offset, size_t size)
+{
+ return copy_from_sockptr_offset(dst, (sockptr_t) src, offset, size);
+}
+
+static inline int copy_from_bpfptr(void *dst, bpfptr_t src, size_t size)
+{
+ return copy_from_bpfptr_offset(dst, src, 0, size);
+}
+
+static inline int copy_to_bpfptr_offset(bpfptr_t dst, size_t offset,
+ const void *src, size_t size)
+{
+ return copy_to_sockptr_offset((sockptr_t) dst, offset, src, size);
+}
+
+static inline void *memdup_bpfptr(bpfptr_t src, size_t len)
+{
+ return memdup_sockptr((sockptr_t) src, len);
+}
+
+static inline long strncpy_from_bpfptr(char *dst, bpfptr_t src, size_t count)
+{
+ return strncpy_from_sockptr(dst, (sockptr_t) src, count);
+}
+
+#endif /* _LINUX_BPFPTR_H */
void btf_get(struct btf *btf);
void btf_put(struct btf *btf);
-int btf_new_fd(const union bpf_attr *attr);
+int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr);
struct btf *btf_get_by_fd(int fd);
int btf_get_info_by_fd(const struct btf *btf,
const union bpf_attr *attr,
struct list_head task_iters;
/*
- * On the default hierarhcy, ->subsys[ssid] may point to a css
+ * On the default hierarchy, ->subsys[ssid] may point to a css
* attached to an ancestor instead of the cgroup this css_set is
* associated with. The following node is anchored at
* ->subsys[ssid]->cgroup->e_csets[ssid] and provides a way to
*/
bool threaded:1;
- /* the following two fields are initialized automtically during boot */
+ /* the following two fields are initialized automatically during boot */
int id;
const char *name;
* sock_cgroup_data overloads (prioidx, classid) and the cgroup pointer.
* On boot, sock_cgroup_data records the cgroup that the sock was created
* in so that cgroup2 matches can be made; however, once either net_prio or
- * net_cls starts being used, the area is overriden to carry prioidx and/or
+ * net_cls starts being used, the area is overridden to carry prioidx and/or
* classid. The two modes are distinguished by whether the lowest bit is
* set. Clear bit indicates cgroup pointer while set bit prioidx and
* classid.
#ifdef CONFIG_CGROUPS
/*
- * All weight knobs on the default hierarhcy should use the following min,
+ * All weight knobs on the default hierarchy should use the following min,
* default and max values. The default value is the logarithmic center of
* MIN and MAX and allows 100x to be expressed in both directions.
*/
/* SIGILL, SIGFPE, SIGSEGV, SIGBUS, SIGTRAP, SIGEMT */
struct {
compat_uptr_t _addr; /* faulting insn/memory ref. */
-#ifdef __ARCH_SI_TRAPNO
- int _trapno; /* TRAP # which caused the signal */
-#endif
#define __COMPAT_ADDR_BND_PKEY_PAD (__alignof__(compat_uptr_t) < sizeof(short) ? \
sizeof(short) : __alignof__(compat_uptr_t))
union {
+ /* used on alpha and sparc */
+ int _trapno; /* TRAP # which caused the signal */
/*
* used when si_code=BUS_MCEERR_AR or
* used when si_code=BUS_MCEERR_AO
u32 _pkey;
} _addr_pkey;
/* used when si_code=TRAP_PERF */
- compat_ulong_t _perf;
+ struct {
+ compat_ulong_t _data;
+ u32 _type;
+ } _perf;
};
} _sigfault;
* must end with any of these keywords:
* break;
* fallthrough;
+ * continue;
* goto <label>;
* return [expression];
*
unsigned int vc_rows;
unsigned int vc_size_row; /* Bytes per row */
unsigned int vc_scan_lines; /* # of scan lines */
+ unsigned int vc_cell_height; /* CRTC character cell height */
unsigned long vc_origin; /* [!] Start of real screen */
unsigned long vc_scr_end; /* [!] End of real screen */
unsigned long vc_visible_origin; /* [!] Top of visible window */
#include <vdso/const.h>
+/*
+ * This returns a constant expression while determining if an argument is
+ * a constant expression, most importantly without evaluating the argument.
+ * Glory to Martin Uecker <Martin.Uecker@med.uni-goettingen.de>
+ */
+#define __is_constexpr(x) \
+ (sizeof(int) == sizeof(*(8 ? ((void *)((long)(x) * 0l)) : (int *)8)))
+
#endif /* _LINUX_CONST_H */
}
}
+static __always_inline bool context_tracking_guest_enter(void)
+{
+ if (context_tracking_enabled())
+ __context_tracking_enter(CONTEXT_GUEST);
+
+ return context_tracking_enabled_this_cpu();
+}
+
+static __always_inline void context_tracking_guest_exit(void)
+{
+ if (context_tracking_enabled())
+ __context_tracking_exit(CONTEXT_GUEST);
+}
/**
* ct_state() - return the current context tracking state if known
static inline enum ctx_state exception_enter(void) { return 0; }
static inline void exception_exit(enum ctx_state prev_ctx) { }
static inline enum ctx_state ct_state(void) { return CONTEXT_DISABLED; }
+static inline bool context_tracking_guest_enter(void) { return false; }
+static inline void context_tracking_guest_exit(void) { }
+
#endif /* !CONFIG_CONTEXT_TRACKING */
#define CT_WARN_ON(cond) WARN_ON(context_tracking_enabled() && (cond))
static inline void context_tracking_init(void) { }
#endif /* CONFIG_CONTEXT_TRACKING_FORCE */
-
-#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN
-/* must be called with irqs disabled */
-static __always_inline void guest_enter_irqoff(void)
-{
- instrumentation_begin();
- if (vtime_accounting_enabled_this_cpu())
- vtime_guest_enter(current);
- else
- current->flags |= PF_VCPU;
- instrumentation_end();
-
- if (context_tracking_enabled())
- __context_tracking_enter(CONTEXT_GUEST);
-
- /* KVM does not hold any references to rcu protected data when it
- * switches CPU into a guest mode. In fact switching to a guest mode
- * is very similar to exiting to userspace from rcu point of view. In
- * addition CPU may stay in a guest mode for quite a long time (up to
- * one time slice). Lets treat guest mode as quiescent state, just like
- * we do with user-mode execution.
- */
- if (!context_tracking_enabled_this_cpu()) {
- instrumentation_begin();
- rcu_virt_note_context_switch(smp_processor_id());
- instrumentation_end();
- }
-}
-
-static __always_inline void guest_exit_irqoff(void)
-{
- if (context_tracking_enabled())
- __context_tracking_exit(CONTEXT_GUEST);
-
- instrumentation_begin();
- if (vtime_accounting_enabled_this_cpu())
- vtime_guest_exit(current);
- else
- current->flags &= ~PF_VCPU;
- instrumentation_end();
-}
-
-#else
-static __always_inline void guest_enter_irqoff(void)
-{
- /*
- * This is running in ioctl context so its safe
- * to assume that it's the stime pending cputime
- * to flush.
- */
- instrumentation_begin();
- vtime_account_kernel(current);
- current->flags |= PF_VCPU;
- rcu_virt_note_context_switch(smp_processor_id());
- instrumentation_end();
-}
-
-static __always_inline void guest_exit_irqoff(void)
-{
- instrumentation_begin();
- /* Flush the guest cputime we spent on the guest */
- vtime_account_kernel(current);
- current->flags &= ~PF_VCPU;
- instrumentation_end();
-}
-#endif /* CONFIG_VIRT_CPU_ACCOUNTING_GEN */
-
-static inline void guest_exit(void)
-{
- unsigned long flags;
-
- local_irq_save(flags);
- guest_exit_irqoff();
- local_irq_restore(flags);
-}
-
#endif
* @flags: Link flags.
* @rpm_active: Whether or not the consumer device is runtime-PM-active.
* @kref: Count repeated addition of the same link.
- * @rcu_head: An RCU head to use for deferred execution of SRCU callbacks.
+ * @rm_work: Work structure used for removing the link.
* @supplier_preactivated: Supplier has been made active before consumer probe.
*/
struct device_link {
u32 flags;
refcount_t rpm_active;
struct kref kref;
-#ifdef CONFIG_SRCU
- struct rcu_head rcu_head;
-#endif
+ struct work_struct rm_work;
bool supplier_preactivated; /* Owned by consumer probe. */
};
void set_primary_fwnode(struct device *dev, struct fwnode_handle *fwnode);
void set_secondary_fwnode(struct device *dev, struct fwnode_handle *fwnode);
void device_set_of_node_from_dev(struct device *dev, const struct device *dev2);
+void device_set_node(struct device *dev, struct fwnode_handle *fwnode);
static inline int dev_num_vf(struct device *dev)
{
#define DSA_8021Q_N_SUBVLAN 8
-#if IS_ENABLED(CONFIG_NET_DSA_TAG_8021Q)
-
int dsa_8021q_setup(struct dsa_8021q_context *ctx, bool enabled);
int dsa_8021q_crosschip_bridge_join(struct dsa_8021q_context *ctx, int port,
struct sk_buff *dsa_8021q_xmit(struct sk_buff *skb, struct net_device *netdev,
u16 tpid, u16 tci);
+void dsa_8021q_rcv(struct sk_buff *skb, int *source_port, int *switch_id,
+ int *subvlan);
+
u16 dsa_8021q_tx_vid(struct dsa_switch *ds, int port);
u16 dsa_8021q_rx_vid(struct dsa_switch *ds, int port);
bool vid_is_dsa_8021q(u16 vid);
-#else
-
-int dsa_8021q_setup(struct dsa_8021q_context *ctx, bool enabled)
-{
- return 0;
-}
-
-int dsa_8021q_crosschip_bridge_join(struct dsa_8021q_context *ctx, int port,
- struct dsa_8021q_context *other_ctx,
- int other_port)
-{
- return 0;
-}
-
-int dsa_8021q_crosschip_bridge_leave(struct dsa_8021q_context *ctx, int port,
- struct dsa_8021q_context *other_ctx,
- int other_port)
-{
- return 0;
-}
-
-struct sk_buff *dsa_8021q_xmit(struct sk_buff *skb, struct net_device *netdev,
- u16 tpid, u16 tci)
-{
- return NULL;
-}
-
-u16 dsa_8021q_tx_vid(struct dsa_switch *ds, int port)
-{
- return 0;
-}
-
-u16 dsa_8021q_rx_vid(struct dsa_switch *ds, int port)
-{
- return 0;
-}
-
-u16 dsa_8021q_rx_vid_subvlan(struct dsa_switch *ds, int port, u16 subvlan)
-{
- return 0;
-}
-
-int dsa_8021q_rx_switch_id(u16 vid)
-{
- return 0;
-}
-
-int dsa_8021q_rx_source_port(u16 vid)
-{
- return 0;
-}
-
-u16 dsa_8021q_rx_subvlan(u16 vid)
-{
- return 0;
-}
-
-bool vid_is_dsa_8021q_rxvlan(u16 vid)
-{
- return false;
-}
-
-bool vid_is_dsa_8021q_txvlan(u16 vid)
-{
- return false;
-}
-
-bool vid_is_dsa_8021q(u16 vid)
-{
- return false;
-}
-
-#endif /* IS_ENABLED(CONFIG_NET_DSA_TAG_8021Q) */
-
#endif /* _NET_DSA_8021Q_H */
#define ETH_P_SJA1105 ETH_P_DSA_8021Q
#define ETH_P_SJA1105_META 0x0008
+#define ETH_P_SJA1110 0xdadc
/* IEEE 802.3 Annex 57A: Slow Protocols PDUs (01:80:C2:xx:xx:xx) */
#define SJA1105_LINKLOCAL_FILTER_A 0x0180C2000000ull
*/
spinlock_t meta_lock;
unsigned long state;
+ u8 ts_id;
};
struct sja1105_skb_cb {
struct sk_buff *clone;
- u32 meta_tstamp;
+ u64 tstamp;
+ /* Only valid for packets cloned for 2-step TX timestamping */
+ u8 ts_id;
};
#define SJA1105_SKB_CB(skb) \
u16 xmit_tpid;
};
+enum sja1110_meta_tstamp {
+ SJA1110_META_TSTAMP_TX = 0,
+ SJA1110_META_TSTAMP_RX = 1,
+};
+
+#if IS_ENABLED(CONFIG_NET_DSA_SJA1105_PTP)
+
+void sja1110_process_meta_tstamp(struct dsa_switch *ds, int port, u8 ts_id,
+ enum sja1110_meta_tstamp dir, u64 tstamp);
+
+#else
+
+static inline void sja1110_process_meta_tstamp(struct dsa_switch *ds, int port,
+ u8 ts_id, enum sja1110_meta_tstamp dir,
+ u64 tstamp)
+{
+}
+
+#endif /* IS_ENABLED(CONFIG_NET_DSA_SJA1105_PTP) */
+
#endif /* _NET_DSA_SJA1105_H */
#define _DPRINTK_FLAGS_INCL_FUNCNAME (1<<2)
#define _DPRINTK_FLAGS_INCL_LINENO (1<<3)
#define _DPRINTK_FLAGS_INCL_TID (1<<4)
+
+#define _DPRINTK_FLAGS_INCL_ANY \
+ (_DPRINTK_FLAGS_INCL_MODNAME | _DPRINTK_FLAGS_INCL_FUNCNAME |\
+ _DPRINTK_FLAGS_INCL_LINENO | _DPRINTK_FLAGS_INCL_TID)
+
#if defined DEBUG
#define _DPRINTK_FLAGS_DEFAULT _DPRINTK_FLAGS_PRINT
#else
void (*depth_updated)(struct blk_mq_hw_ctx *);
bool (*allow_merge)(struct request_queue *, struct request *, struct bio *);
- bool (*bio_merge)(struct blk_mq_hw_ctx *, struct bio *, unsigned int);
+ bool (*bio_merge)(struct request_queue *, struct bio *, unsigned int);
int (*request_merge)(struct request_queue *q, struct request **, struct bio *);
void (*request_merged)(struct request_queue *, struct request *, enum elv_merge);
void (*requests_merged)(struct request_queue *, struct request *, struct request *);
#define __LINUX_ENTRYKVM_H
#include <linux/entry-common.h>
+#include <linux/tick.h>
/* Transfer to guest mode work */
#ifdef CONFIG_KVM_XFER_TO_GUEST_WORK
static inline void xfer_to_guest_mode_prepare(void)
{
lockdep_assert_irqs_disabled();
- rcu_nocb_flush_deferred_wakeup();
+ tick_nohz_user_enter_prepare();
}
/**
* required information to the driver.
*/
struct ethtool_module_eeprom {
- __u32 offset;
- __u32 length;
- __u8 page;
- __u8 bank;
- __u8 i2c_address;
- __u8 *data;
+ u32 offset;
+ u32 length;
+ u8 page;
+ u8 bank;
+ u8 i2c_address;
+ u8 *data;
};
/**
#define FANOTIFY_INIT_FLAGS (FANOTIFY_ADMIN_INIT_FLAGS | \
FANOTIFY_USER_INIT_FLAGS)
+/* Internal group flags */
+#define FANOTIFY_UNPRIV 0x80000000
+#define FANOTIFY_INTERNAL_GROUP_FLAGS (FANOTIFY_UNPRIV)
+
#define FANOTIFY_MARK_TYPE_BITS (FAN_MARK_INODE | FAN_MARK_MOUNT | \
FAN_MARK_FILESYSTEM)
/* drivers/video/fb_defio.c */
int fb_deferred_io_mmap(struct fb_info *info, struct vm_area_struct *vma);
extern void fb_deferred_io_init(struct fb_info *info);
+extern void fb_deferred_io_open(struct fb_info *info,
+ struct inode *inode,
+ struct file *file);
extern void fb_deferred_io_cleanup(struct fb_info *info);
extern int fb_deferred_io_fsync(struct file *file, loff_t start,
loff_t end, int datasync);
u32 flags;
u32 tgt_index;
void *tgt_value;
+ struct bpf_map *map;
u32 map_id;
enum bpf_map_type map_type;
u32 kern_flags;
#ifdef CONFIG_INET
struct sock *bpf_run_sk_reuseport(struct sock_reuseport *reuse, struct sock *sk,
struct bpf_prog *prog, struct sk_buff *skb,
+ struct sock *migrating_sk,
u32 hash);
#else
static inline struct sock *
bpf_run_sk_reuseport(struct sock_reuseport *reuse, struct sock *sk,
struct bpf_prog *prog, struct sk_buff *skb,
+ struct sock *migrating_sk,
u32 hash)
{
return NULL;
}
#endif /* IS_ENABLED(CONFIG_IPV6) */
-static __always_inline int __bpf_xdp_redirect_map(struct bpf_map *map, u32 ifindex, u64 flags,
+static __always_inline int __bpf_xdp_redirect_map(struct bpf_map *map, u32 ifindex,
+ u64 flags, const u64 flag_mask,
void *lookup_elem(struct bpf_map *map, u32 key))
{
struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info);
+ const u64 action_mask = XDP_ABORTED | XDP_DROP | XDP_PASS | XDP_TX;
/* Lower bits of the flags are used as return code on lookup failure */
- if (unlikely(flags > XDP_TX))
+ if (unlikely(flags & ~(action_mask | flag_mask)))
return XDP_ABORTED;
ri->tgt_value = lookup_elem(map, ifindex);
- if (unlikely(!ri->tgt_value)) {
+ if (unlikely(!ri->tgt_value) && !(flags & BPF_F_BROADCAST)) {
/* If the lookup fails we want to clear out the state in the
* redirect_info struct completely, so that if an eBPF program
* performs multiple lookups, the last one always takes
*/
ri->map_id = INT_MAX; /* Valid map id idr range: [1,INT_MAX[ */
ri->map_type = BPF_MAP_TYPE_UNSPEC;
- return flags;
+ return flags & action_mask;
}
ri->tgt_index = ifindex;
ri->map_id = map->id;
ri->map_type = map->map_type;
+ if (flags & BPF_F_BROADCAST) {
+ WRITE_ONCE(ri->map, map);
+ ri->flags = flags;
+ } else {
+ WRITE_ONCE(ri->map, NULL);
+ ri->flags = 0;
+ }
+
return XDP_REDIRECT;
}
extern bool fw_devlink_is_strict(void);
int fwnode_link_add(struct fwnode_handle *con, struct fwnode_handle *sup);
void fwnode_links_purge(struct fwnode_handle *fwnode);
+void fw_devlink_purge_absent_suppliers(struct fwnode_handle *fwnode);
#endif
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * FWNODE helper for the MDIO (Ethernet PHY) API
+ */
+
+#ifndef __LINUX_FWNODE_MDIO_H
+#define __LINUX_FWNODE_MDIO_H
+
+#include <linux/phy.h>
+
+#if IS_ENABLED(CONFIG_FWNODE_MDIO)
+int fwnode_mdiobus_phy_device_register(struct mii_bus *mdio,
+ struct phy_device *phy,
+ struct fwnode_handle *child, u32 addr);
+
+int fwnode_mdiobus_register_phy(struct mii_bus *bus,
+ struct fwnode_handle *child, u32 addr);
+
+#else /* CONFIG_FWNODE_MDIO */
+int fwnode_mdiobus_phy_device_register(struct mii_bus *mdio,
+ struct phy_device *phy,
+ struct fwnode_handle *child, u32 addr)
+{
+ return -EINVAL;
+}
+
+static inline int fwnode_mdiobus_register_phy(struct mii_bus *bus,
+ struct fwnode_handle *child,
+ u32 addr)
+{
+ return -EINVAL;
+}
+#endif
+
+#endif /* __LINUX_FWNODE_MDIO_H */
}
#endif /* CONFIG_SYSFS */
-extern struct rw_semaphore bdev_lookup_sem;
-
dev_t blk_lookup_devt(const char *name, int partno);
void blk_request_module(dev_t devt);
#ifdef CONFIG_BLOCK
*/
static inline u32 hid_report_len(struct hid_report *report)
{
- /* equivalent to DIV_ROUND_UP(report->size, 8) + !!(report->id > 0) */
- return ((report->size - 1) >> 3) + 1 + (report->id > 0);
+ return DIV_ROUND_UP(report->size, 8) + (report->id > 0);
}
int hid_report_raw_event(struct hid_device *hid, int type, u8 *data, u32 size,
int host1x_device_init(struct host1x_device *device);
int host1x_device_exit(struct host1x_device *device);
-int __host1x_client_register(struct host1x_client *client,
- struct lock_class_key *key);
-#define host1x_client_register(class) \
- ({ \
- static struct lock_class_key __key; \
- __host1x_client_register(class, &__key); \
+void __host1x_client_init(struct host1x_client *client, struct lock_class_key *key);
+void host1x_client_exit(struct host1x_client *client);
+
+#define host1x_client_init(client) \
+ ({ \
+ static struct lock_class_key __key; \
+ __host1x_client_init(client, &__key); \
+ })
+
+int __host1x_client_register(struct host1x_client *client);
+
+/*
+ * Note that this wrapper calls __host1x_client_init() for compatibility
+ * with existing callers. Callers that want to separately initialize and
+ * register a host1x client must first initialize using either of the
+ * __host1x_client_init() or host1x_client_init() functions and then use
+ * the low-level __host1x_client_register() function to avoid the client
+ * getting reinitialized.
+ */
+#define host1x_client_register(client) \
+ ({ \
+ static struct lock_class_key __key; \
+ __host1x_client_init(client, &__key); \
+ __host1x_client_register(client); \
})
int host1x_client_unregister(struct host1x_client *client);
vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd);
extern struct page *huge_zero_page;
+extern unsigned long huge_zero_pfn;
static inline bool is_huge_zero_page(struct page *page)
{
static inline bool is_huge_zero_pmd(pmd_t pmd)
{
- return is_huge_zero_page(pmd_page(pmd));
+ return READ_ONCE(huge_zero_pfn) == pmd_pfn(pmd) && pmd_present(pmd);
}
static inline bool is_huge_zero_pud(pud_t pud)
return false;
}
+static inline bool is_huge_zero_pmd(pmd_t pmd)
+{
+ return false;
+}
+
static inline bool is_huge_zero_pud(pud_t pud)
{
return false;
long hugetlb_unreserve_pages(struct inode *inode, long start, long end,
long freed);
bool isolate_huge_page(struct page *page, struct list_head *list);
+int get_hwpoison_huge_page(struct page *page, bool *hugetlb);
void putback_active_hugepage(struct page *page);
void move_hugetlb_state(struct page *oldpage, struct page *newpage, int reason);
void free_huge_page(struct page *page);
return false;
}
+static inline int get_hwpoison_huge_page(struct page *page, bool *hugetlb)
+{
+ return 0;
+}
+
static inline void putback_active_hugepage(struct page *page)
{
}
unsigned long address);
int huge_add_to_page_cache(struct page *page, struct address_space *mapping,
pgoff_t idx);
+void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma,
+ unsigned long address, struct page *page);
/* arch callback */
int __init __alloc_bootmem_huge_page(struct hstate *h);
* Copyright (c) 2006, Michael Wu <flamingice@sourmilk.net>
* Copyright (c) 2013 - 2014 Intel Mobile Communications GmbH
* Copyright (c) 2016 - 2017 Intel Deutschland GmbH
- * Copyright (c) 2018 - 2020 Intel Corporation
+ * Copyright (c) 2018 - 2021 Intel Corporation
*/
#ifndef LINUX_IEEE80211_H
#define IEEE80211_HE_PHY_CAP9_NOMIMAL_PKT_PADDING_RESERVED 0xc0
#define IEEE80211_HE_PHY_CAP9_NOMIMAL_PKT_PADDING_MASK 0xc0
+#define IEEE80211_HE_PHY_CAP10_HE_MU_M1RU_MAX_LTF 0x01
+
/* 802.11ax HE TX/RX MCS NSS Support */
#define IEEE80211_TX_RX_MCS_NSS_SUPP_HIGHEST_MCS_POS (3)
#define IEEE80211_TX_RX_MCS_NSS_SUPP_TX_BITMAP_POS (6)
WLAN_CATEGORY_BACK = 3,
WLAN_CATEGORY_PUBLIC = 4,
WLAN_CATEGORY_RADIO_MEASUREMENT = 5,
+ WLAN_CATEGORY_FAST_BBS_TRANSITION = 6,
WLAN_CATEGORY_HT = 7,
WLAN_CATEGORY_SA_QUERY = 8,
WLAN_CATEGORY_PROTECTED_DUAL_OF_ACTION = 9,
*/
#define WLAN_EXT_CAPA3_MULTI_BSSID_SUPPORT BIT(6)
+/* Timing Measurement protocol for time sync is set in the 7th bit of 3rd byte
+ * of the @WLAN_EID_EXT_CAPABILITY information element
+ */
+#define WLAN_EXT_CAPA3_TIMING_MEASUREMENT_SUPPORT BIT(7)
+
/* TDLS capabilities in the 4th byte of @WLAN_EID_EXT_CAPABILITY */
#define WLAN_EXT_CAPA4_TDLS_BUFFER_STA BIT(4)
#define WLAN_EXT_CAPA4_TDLS_PEER_PSM BIT(5)
case ARPHRD_TUNNEL6:
case ARPHRD_SIT:
case ARPHRD_IPGRE:
+ case ARPHRD_IP6GRE:
case ARPHRD_VOID:
case ARPHRD_NONE:
case ARPHRD_RAWIP:
/* SPDX-License-Identifier: GPL-2.0-only
- * Copyright (c) 2013-2019, The Linux Foundation. All rights reserved.
+ * Copyright (c) 2013-2019, 2021 The Linux Foundation. All rights reserved.
*/
#ifndef _LINUX_IF_RMNET_H_
} __aligned(1);
/* rmnet_map_header flags field:
- * PAD_LEN: number of pad bytes following packet data
- * CMD: 1 = packet contains a MAP command; 0 = packet contains data
+ * PAD_LEN: number of pad bytes following packet data
+ * CMD: 1 = packet contains a MAP command; 0 = packet contains data
+ * NEXT_HEADER: 1 = packet contains V5 CSUM header 0 = no V5 CSUM header
*/
#define MAP_PAD_LEN_MASK GENMASK(5, 0)
+#define MAP_NEXT_HEADER_FLAG BIT(6)
#define MAP_CMD_FLAG BIT(7)
struct rmnet_map_dl_csum_trailer {
u8 flags; /* MAP_CSUM_DL_VALID_FLAG */
__be16 csum_start_offset;
__be16 csum_length;
- __be16 csum_value;
+ __sum16 csum_value;
} __aligned(1);
/* rmnet_map_dl_csum_trailer flags field:
#define MAP_CSUM_UL_UDP_FLAG BIT(14)
#define MAP_CSUM_UL_ENABLED_FLAG BIT(15)
+/* MAP CSUM headers */
+struct rmnet_map_v5_csum_header {
+ u8 header_info;
+ u8 csum_info;
+ __be16 reserved;
+} __aligned(1);
+
+/* v5 header_info field
+ * NEXT_HEADER: represents whether there is any next header
+ * HEADER_TYPE: represents the type of this header
+ *
+ * csum_info field
+ * CSUM_VALID_OR_REQ:
+ * 1 = for UL, checksum computation is requested.
+ * 1 = for DL, validated the checksum and has found it valid
+ */
+
+#define MAPV5_HDRINFO_NXT_HDR_FLAG BIT(0)
+#define MAPV5_HDRINFO_HDR_TYPE_FMASK GENMASK(7, 1)
+#define MAPV5_CSUMINFO_VALID_FLAG BIT(7)
+
+#define RMNET_MAP_HEADER_TYPE_CSUM_OFFLOAD 2
#endif /* !(_LINUX_IF_RMNET_H_) */
asm(".section \"" __sec "\", \"a\" \n" \
__stringify(__name) ": \n" \
".long " __stringify(__stub) " - . \n" \
- ".previous \n");
+ ".previous \n"); \
+ static_assert(__same_type(initcall_t, &fn));
#else
#define ____define_initcall(fn, __unused, __name, __sec) \
static initcall_t __name __used \
*/
#define lower_32_bits(n) ((u32)((n) & 0xffffffff))
+/**
+ * upper_16_bits - return bits 16-31 of a number
+ * @n: the number we're accessing
+ */
+#define upper_16_bits(n) ((u16)((n) >> 16))
+
+/**
+ * lower_16_bits - return bits 0-15 of a number
+ * @n: the number we're accessing
+ */
+#define lower_16_bits(n) ((u16)((n) & 0xffff))
+
struct completion;
struct pt_regs;
struct user;
#include <linux/spinlock.h>
#include <linux/signal.h>
#include <linux/sched.h>
+#include <linux/sched/stat.h>
#include <linux/bug.h>
#include <linux/minmax.h>
#include <linux/mm.h>
*/
#define KVM_REQ_TLB_FLUSH (0 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
#define KVM_REQ_MMU_RELOAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
-#define KVM_REQ_PENDING_TIMER 2
+#define KVM_REQ_UNBLOCK 2
#define KVM_REQ_UNHALT 3
#define KVM_REQUEST_ARCH_BASE 8
return !!map->hva;
}
+static inline bool kvm_vcpu_can_poll(ktime_t cur, ktime_t stop)
+{
+ return single_task_running() && !need_resched() && ktime_before(cur, stop);
+}
+
/*
* Sometimes a large or cross-page mmio needs to be broken up into separate
* exits for userspace servicing.
struct kvm_dirty_ring dirty_ring;
};
+/* must be called with irqs disabled */
+static __always_inline void guest_enter_irqoff(void)
+{
+ /*
+ * This is running in ioctl context so its safe to assume that it's the
+ * stime pending cputime to flush.
+ */
+ instrumentation_begin();
+ vtime_account_guest_enter();
+ instrumentation_end();
+
+ /*
+ * KVM does not hold any references to rcu protected data when it
+ * switches CPU into a guest mode. In fact switching to a guest mode
+ * is very similar to exiting to userspace from rcu point of view. In
+ * addition CPU may stay in a guest mode for quite a long time (up to
+ * one time slice). Lets treat guest mode as quiescent state, just like
+ * we do with user-mode execution.
+ */
+ if (!context_tracking_guest_enter()) {
+ instrumentation_begin();
+ rcu_virt_note_context_switch(smp_processor_id());
+ instrumentation_end();
+ }
+}
+
+static __always_inline void guest_exit_irqoff(void)
+{
+ context_tracking_guest_exit();
+
+ instrumentation_begin();
+ /* Flush the guest cputime we spent on the guest */
+ vtime_account_guest_exit();
+ instrumentation_end();
+}
+
+static inline void guest_exit(void)
+{
+ unsigned long flags;
+
+ local_irq_save(flags);
+ guest_exit_irqoff();
+ local_irq_restore(flags);
+}
+
static inline int kvm_vcpu_exiting_guest_mode(struct kvm_vcpu *vcpu)
{
/*
static inline unsigned long
__gfn_to_hva_memslot(const struct kvm_memory_slot *slot, gfn_t gfn)
{
- return slot->userspace_addr + (gfn - slot->base_gfn) * PAGE_SIZE;
+ /*
+ * The index was checked originally in search_memslots. To avoid
+ * that a malicious guest builds a Spectre gadget out of e.g. page
+ * table walks, do not let the processor speculate loads outside
+ * the guest's registered memslots.
+ */
+ unsigned long offset = gfn - slot->base_gfn;
+ offset = array_index_nospec(offset, slot->npages);
+ return slot->userspace_addr + offset * PAGE_SIZE;
}
static inline int memslot_id(struct kvm *kvm, gfn_t gfn)
struct nvdimm_bus;
struct module;
-struct device;
struct nd_blk_region;
struct nd_blk_region_desc {
int (*enable)(struct nvdimm_bus *nvdimm_bus, struct device *dev);
struct mutex rtc_timer_lock;
};
-#define BD70528_BUCK_VOLTS 17
-#define BD70528_BUCK_VOLTS 17
-#define BD70528_BUCK_VOLTS 17
+#define BD70528_BUCK_VOLTS 0x10
#define BD70528_LDO_VOLTS 0x20
#define BD70528_REG_BUCK1_EN 0x0F
BD71828_REGULATOR_AMOUNT,
};
-#define BD71828_BUCK1267_VOLTS 0xEF
-#define BD71828_BUCK3_VOLTS 0x10
-#define BD71828_BUCK4_VOLTS 0x20
-#define BD71828_BUCK5_VOLTS 0x10
-#define BD71828_LDO_VOLTS 0x32
+#define BD71828_BUCK1267_VOLTS 0x100
+#define BD71828_BUCK3_VOLTS 0x20
+#define BD71828_BUCK4_VOLTS 0x40
+#define BD71828_BUCK5_VOLTS 0x20
+#define BD71828_LDO_VOLTS 0x40
/* LDO6 is fixed 1.8V voltage */
#define BD71828_LDO_6_VOLTAGE 1800000
/* struct phy_device dev_flags definitions */
#define MICREL_PHY_50MHZ_CLK 0x00000001
#define MICREL_PHY_FXEN 0x00000002
+#define MICREL_KSZ8_P1_ERRATA 0x00000003
#define MICREL_KSZ9021_EXTREG_CTRL 0xB
#define MICREL_KSZ9021_EXTREG_DATA_WRITE 0xC
#define MICREL_KSZ9021_RGMII_CLK_CTRL_PAD_SCEW 0x104
#define MICREL_KSZ9021_RGMII_RX_DATA_PAD_SCEW 0x105
+/* Device specific MII_BMCR (Reg 0) bits */
+/* 1 = HP Auto MDI/MDI-X mode, 0 = Microchip Auto MDI/MDI-X mode */
+#define KSZ886X_BMCR_HP_MDIX BIT(5)
+/* 1 = Force MDI (transmit on RXP/RXM pins), 0 = Normal operation
+ * (transmit on TXP/TXM pins)
+ */
+#define KSZ886X_BMCR_FORCE_MDI BIT(4)
+/* 1 = Disable auto MDI-X */
+#define KSZ886X_BMCR_DISABLE_AUTO_MDIX BIT(3)
+#define KSZ886X_BMCR_DISABLE_FAR_END_FAULT BIT(2)
+#define KSZ886X_BMCR_DISABLE_TRANSMIT BIT(1)
+#define KSZ886X_BMCR_DISABLE_LED BIT(0)
+
+#define KSZ886X_CTRL_MDIX_STAT BIT(4)
+
#endif /* _MICREL_PHY_H */
#ifndef _LINUX_MINMAX_H
#define _LINUX_MINMAX_H
+#include <linux/const.h>
+
/*
* min()/max()/clamp() macros must accomplish three things:
*
#define __typecheck(x, y) \
(!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
-/*
- * This returns a constant expression while determining if an argument is
- * a constant expression, most importantly without evaluating the argument.
- * Glory to Martin Uecker <Martin.Uecker@med.uni-goettingen.de>
- */
-#define __is_constexpr(x) \
- (sizeof(int) == sizeof(*(8 ? ((void *)((long)(x) * 0l)) : (int *)8)))
-
#define __no_side_effects(x, y) \
(__is_constexpr(x) && __is_constexpr(y))
bool wol_port[MLX4_MAX_PORTS + 1];
struct mlx4_rate_limit_caps rl_caps;
u32 health_buffer_addrs;
+ bool map_clock_to_user;
};
struct mlx4_buf_list {
MLX5_CAP_VDPA_EMULATION = 0x13,
MLX5_CAP_DEV_EVENT = 0x14,
MLX5_CAP_IPSEC,
+ MLX5_CAP_GENERAL_2 = 0x20,
/* NUM OF CAP Types */
MLX5_CAP_NUM
};
#define MLX5_CAP_GEN_MAX(mdev, cap) \
MLX5_GET(cmd_hca_cap, mdev->caps.hca_max[MLX5_CAP_GENERAL], cap)
+#define MLX5_CAP_GEN_2(mdev, cap) \
+ MLX5_GET(cmd_hca_cap_2, mdev->caps.hca_cur[MLX5_CAP_GENERAL_2], cap)
+
+#define MLX5_CAP_GEN_2_64(mdev, cap) \
+ MLX5_GET64(cmd_hca_cap_2, mdev->caps.hca_cur[MLX5_CAP_GENERAL_2], cap)
+
+#define MLX5_CAP_GEN_2_MAX(mdev, cap) \
+ MLX5_GET(cmd_hca_cap_2, mdev->caps.hca_max[MLX5_CAP_GENERAL_2], cap)
+
#define MLX5_CAP_ETH(mdev, cap) \
MLX5_GET(per_protocol_networking_offload_caps,\
mdev->caps.hca_cur[MLX5_CAP_ETHERNET_OFFLOADS], cap)
enum {
MLX5_PRIV_FLAGS_DISABLE_IB_ADEV = 1 << 0,
MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV = 1 << 1,
+ /* Set during device detach to block any further devices
+ * creation/deletion on drivers rescan. Unset during device attach.
+ */
+ MLX5_PRIV_FLAGS_DETACH = 1 << 2,
};
struct mlx5_adev {
int idx;
};
+struct mlx5_ft_pool;
struct mlx5_priv {
/* IRQ table valid only for real pci devices PF or VF */
struct mlx5_irq_table *irq_table;
struct mlx5_core_roce roce;
struct mlx5_fc_stats fc_stats;
struct mlx5_rl_table rl_table;
+ struct mlx5_ft_pool *ft_pool;
struct mlx5_bfreg_data bfregs;
struct mlx5_uars_page *uar;
#define MLX5_LOG_SW_ICM_BLOCK_SIZE(dev) (MLX5_CAP_DEV_MEM(dev, log_sw_icm_alloc_granularity))
#define MLX5_SW_ICM_BLOCK_SIZE(dev) (1 << MLX5_LOG_SW_ICM_BLOCK_SIZE(dev))
+enum {
+ MLX5_PROF_MASK_QP_SIZE = (u64)1 << 0,
+ MLX5_PROF_MASK_MR_CACHE = (u64)1 << 1,
+};
+
+enum {
+ MR_CACHE_LAST_STD_ENTRY = 20,
+ MLX5_IMR_MTT_CACHE_ENTRY,
+ MLX5_IMR_KSM_CACHE_ENTRY,
+ MAX_MR_CACHE_ENTRIES
+};
+
+struct mlx5_profile {
+ u64 mask;
+ u8 log_max_qp;
+ struct {
+ int size;
+ int limit;
+ } mr_cache[MAX_MR_CACHE_ENTRIES];
+};
+
struct mlx5_core_dev {
struct device *device;
enum mlx5_coredev_type coredev_type;
struct mutex intf_state_mutex;
unsigned long intf_state;
struct mlx5_priv priv;
- struct mlx5_profile *profile;
+ struct mlx5_profile profile;
u32 issi;
struct mlx5e_resources mlx5e_res;
struct mlx5_dm *dm;
return mkey & 0xff;
}
-enum {
- MLX5_PROF_MASK_QP_SIZE = (u64)1 << 0,
- MLX5_PROF_MASK_MR_CACHE = (u64)1 << 1,
-};
-
-enum {
- MR_CACHE_LAST_STD_ENTRY = 20,
- MLX5_IMR_MTT_CACHE_ENTRY,
- MLX5_IMR_KSM_CACHE_ENTRY,
- MAX_MR_CACHE_ENTRIES
-};
-
/* Async-atomic event notifier used by mlx5 core to forward FW
* evetns recived from event queue to mlx5 consumers.
* Optimise event queue dipatching.
struct ib_device *device,
struct rdma_netdev_alloc_params *params);
-struct mlx5_profile {
- u64 mask;
- u8 log_max_qp;
- struct {
- int size;
- int limit;
- } mr_cache[MAX_MR_CACHE_ENTRIES];
-};
-
enum {
MLX5_PCI_DEV_IS_VF = 1 << 0,
};
u8 irq_index;
int nent;
u64 mask[4];
+ cpumask_var_t affinity;
};
struct mlx5_eq *
u16 vport_num);
/* Reg C1 usage:
- * Reg C1 = < ESW_TUN_ID(12) | ESW_TUN_OPTS(12) | ESW_ZONE_ID(8) >
+ * Reg C1 = < Reserved(1) | ESW_TUN_ID(12) | ESW_TUN_OPTS(11) | ESW_ZONE_ID(8) >
*
- * Highest 12 bits of reg c1 is the encapsulation tunnel id, next 12 bits is
- * encapsulation tunnel options, and the lowest 8 bits are used for zone id.
+ * Highest bit is reserved for other offloads as marker bit, next 12 bits of reg c1
+ * is the encapsulation tunnel id, next 11 bits is encapsulation tunnel options,
+ * and the lowest 8 bits are used for zone id.
*
* Zone id is used to restore CT flow when packet misses on chain.
*
* on miss and to support inner header rewrite by means of implicit chain 0
* flows.
*/
+#define ESW_RESERVED_BITS 1
#define ESW_ZONE_ID_BITS 8
-#define ESW_TUN_OPTS_BITS 12
+#define ESW_TUN_OPTS_BITS 11
#define ESW_TUN_ID_BITS 12
#define ESW_TUN_OPTS_OFFSET ESW_ZONE_ID_BITS
#define ESW_TUN_OFFSET ESW_TUN_OPTS_OFFSET
#define ESW_ZONE_ID_MASK GENMASK(ESW_ZONE_ID_BITS - 1, 0)
-#define ESW_TUN_OPTS_MASK GENMASK(32 - ESW_TUN_ID_BITS - 1, ESW_TUN_OPTS_OFFSET)
-#define ESW_TUN_MASK GENMASK(31, ESW_TUN_OFFSET)
+#define ESW_TUN_OPTS_MASK GENMASK(31 - ESW_TUN_ID_BITS - ESW_RESERVED_BITS, ESW_TUN_OPTS_OFFSET)
+#define ESW_TUN_MASK GENMASK(31 - ESW_RESERVED_BITS, ESW_TUN_OFFSET)
#define ESW_TUN_ID_SLOW_TABLE_GOTO_VPORT 0 /* 0 is not a valid tunnel id */
-#define ESW_TUN_OPTS_SLOW_TABLE_GOTO_VPORT 0xFFF /* 0xFFF is a reserved mapping */
+/* 0x7FF is a reserved mapping */
+#define ESW_TUN_OPTS_SLOW_TABLE_GOTO_VPORT GENMASK(ESW_TUN_OPTS_BITS - 1, 0)
#define ESW_TUN_SLOW_TABLE_GOTO_VPORT ((ESW_TUN_ID_SLOW_TABLE_GOTO_VPORT << ESW_TUN_OPTS_BITS) | \
ESW_TUN_OPTS_SLOW_TABLE_GOTO_VPORT)
#define ESW_TUN_SLOW_TABLE_GOTO_VPORT_MARK ESW_TUN_OPTS_MASK
FDB_BYPASS_PATH,
FDB_TC_OFFLOAD,
FDB_FT_OFFLOAD,
+ FDB_TC_MISS,
+ FDB_BR_OFFLOAD,
FDB_SLOW_PATH,
FDB_PER_VPORT,
};
void mlx5_modify_header_dealloc(struct mlx5_core_dev *dev,
struct mlx5_modify_hdr *modify_hdr);
+struct mlx5_pkt_reformat_params {
+ int type;
+ u8 param_0;
+ u8 param_1;
+ size_t size;
+ void *data;
+};
+
struct mlx5_pkt_reformat *mlx5_packet_reformat_alloc(struct mlx5_core_dev *dev,
- int reformat_type,
- size_t size,
- void *reformat_data,
+ struct mlx5_pkt_reformat_params *params,
enum mlx5_flow_namespace_type ns_type);
void mlx5_packet_reformat_dealloc(struct mlx5_core_dev *dev,
struct mlx5_pkt_reformat *reformat);
u8 reserved_at_40[0x20];
- u8 reserved_at_60[0x18];
+ u8 reserved_at_60[0x2];
+ u8 reformat_insert[0x1];
+ u8 reformat_remove[0x1];
+ u8 reserver_at_64[0x14];
u8 log_max_ft_num[0x8];
u8 reserved_at_80[0x10];
#define MLX5_FC_BULK_NUM_FCS(fc_enum) (MLX5_FC_BULK_SIZE_FACTOR * (fc_enum))
+#define MLX5_FT_MAX_MULTIPATH_LEVEL 63
+
enum {
MLX5_STEERING_FORMAT_CONNECTX_5 = 0,
MLX5_STEERING_FORMAT_CONNECTX_6DX = 1,
u8 reserved_at_0[0x1f];
u8 vhca_resource_manager[0x1];
- u8 reserved_at_20[0x3];
+ u8 hca_cap_2[0x1];
+ u8 reserved_at_21[0x2];
u8 event_on_vhca_state_teardown_request[0x1];
u8 event_on_vhca_state_in_use[0x1];
u8 event_on_vhca_state_active[0x1];
u8 reserved_at_7c0[0x40];
};
+struct mlx5_ifc_cmd_hca_cap_2_bits {
+ u8 reserved_at_0[0xa0];
+
+ u8 max_reformat_insert_size[0x8];
+ u8 max_reformat_insert_offset[0x8];
+ u8 max_reformat_remove_size[0x8];
+ u8 max_reformat_remove_offset[0x8];
+
+ u8 reserved_at_c0[0x740];
+};
+
enum mlx5_flow_destination_type {
MLX5_FLOW_DESTINATION_TYPE_VPORT = 0x0,
MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE = 0x1,
union mlx5_ifc_hca_cap_union_bits {
struct mlx5_ifc_cmd_hca_cap_bits cmd_hca_cap;
+ struct mlx5_ifc_cmd_hca_cap_2_bits cmd_hca_cap_2;
struct mlx5_ifc_odp_cap_bits odp_cap;
struct mlx5_ifc_atomic_caps_bits atomic_caps;
struct mlx5_ifc_roce_cap_bits roce_cap;
u8 reserved_at_80[0x20];
- u8 reserved_at_a0[0x18];
- u8 intr[0x8];
+ u8 reserved_at_a0[0x14];
+ u8 intr[0xc];
u8 reserved_at_c0[0x3];
u8 log_page_size[0x5];
};
struct mlx5_ifc_packet_reformat_context_in_bits {
- u8 reserved_at_0[0x5];
- u8 reformat_type[0x3];
- u8 reserved_at_8[0xe];
+ u8 reformat_type[0x8];
+ u8 reserved_at_8[0x4];
+ u8 reformat_param_0[0x4];
+ u8 reserved_at_10[0x6];
u8 reformat_data_size[0xa];
- u8 reserved_at_20[0x10];
+ u8 reformat_param_1[0x8];
+ u8 reserved_at_28[0x8];
u8 reformat_data[2][0x8];
u8 more_reformat_data[][0x8];
u8 reserved_at_60[0x20];
};
+enum {
+ MLX5_REFORMAT_CONTEXT_ANCHOR_MAC_START = 0x1,
+ MLX5_REFORMAT_CONTEXT_ANCHOR_IP_START = 0x7,
+ MLX5_REFORMAT_CONTEXT_ANCHOR_TCP_UDP_START = 0x9,
+};
+
enum mlx5_reformat_ctx_type {
MLX5_REFORMAT_TYPE_L2_TO_VXLAN = 0x0,
MLX5_REFORMAT_TYPE_L2_TO_NVGRE = 0x1,
MLX5_REFORMAT_TYPE_L2_TO_L2_TUNNEL = 0x2,
MLX5_REFORMAT_TYPE_L3_TUNNEL_TO_L2 = 0x3,
MLX5_REFORMAT_TYPE_L2_TO_L3_TUNNEL = 0x4,
+ MLX5_REFORMAT_TYPE_INSERT_HDR = 0xf,
+ MLX5_REFORMAT_TYPE_REMOVE_HDR = 0x10,
};
struct mlx5_ifc_alloc_packet_reformat_context_in_bits {
MLX5_ACTION_IN_FIELD_OUT_TCP_SEQ_NUM = 0x59,
MLX5_ACTION_IN_FIELD_OUT_TCP_ACK_NUM = 0x5B,
MLX5_ACTION_IN_FIELD_IPSEC_SYNDROME = 0x5D,
+ MLX5_ACTION_IN_FIELD_OUT_EMD_47_32 = 0x6F,
+ MLX5_ACTION_IN_FIELD_OUT_EMD_31_0 = 0x70,
};
struct mlx5_ifc_alloc_modify_header_context_out_bits {
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+ * Copyright (c) 2021 Mellanox Technologies Ltd.
+ */
+
+#ifndef _MLX5_MPFS_
+#define _MLX5_MPFS_
+
+struct mlx5_core_dev;
+
+#ifdef CONFIG_MLX5_MPFS
+int mlx5_mpfs_add_mac(struct mlx5_core_dev *dev, u8 *mac);
+int mlx5_mpfs_del_mac(struct mlx5_core_dev *dev, u8 *mac);
+#else /* #ifndef CONFIG_MLX5_MPFS */
+static inline int mlx5_mpfs_add_mac(struct mlx5_core_dev *dev, u8 *mac) { return 0; }
+static inline int mlx5_mpfs_del_mac(struct mlx5_core_dev *dev, u8 *mac) { return 0; }
+#endif
+
+#endif
struct mlx5_hairpin_params *params);
void mlx5_core_hairpin_destroy(struct mlx5_hairpin *pair);
+void mlx5_core_hairpin_clear_dead_peer(struct mlx5_hairpin *hp);
#endif /* __TRANSOBJ_H__ */
static inline bool page_is_pfmemalloc(const struct page *page)
{
/*
- * Page index cannot be this large so this must be
- * a pfmemalloc page.
+ * lru.next has bit 1 set if the page is allocated from the
+ * pfmemalloc reserves. Callers may simply overwrite it if
+ * they do not need to preserve that information.
*/
- return page->index == -1UL;
+ return (uintptr_t)page->lru.next & BIT(1);
}
/*
*/
static inline void set_page_pfmemalloc(struct page *page)
{
- page->index = -1UL;
+ page->lru.next = (void *)BIT(1);
}
static inline void clear_page_pfmemalloc(struct page *page)
{
- page->index = 0;
+ page->lru.next = NULL;
}
/*
struct address_space *check_mapping; /* Check page->mapping if set */
pgoff_t first_index; /* Lowest page->index to unmap */
pgoff_t last_index; /* Highest page->index to unmap */
+ struct page *single_page; /* Locked page to be unmapped */
};
struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
extern int fixup_user_fault(struct mm_struct *mm,
unsigned long address, unsigned int fault_flags,
bool *unlocked);
+void unmap_mapping_page(struct page *page);
void unmap_mapping_pages(struct address_space *mapping,
pgoff_t start, pgoff_t nr, bool even_cows);
void unmap_mapping_range(struct address_space *mapping,
BUG();
return -EFAULT;
}
+static inline void unmap_mapping_page(struct page *page) { }
static inline void unmap_mapping_pages(struct address_space *mapping,
pgoff_t start, pgoff_t nr, bool even_cows) { }
static inline void unmap_mapping_range(struct address_space *mapping,
static inline void mem_dump_obj(void *object) {}
#endif
+/**
+ * seal_check_future_write - Check for F_SEAL_FUTURE_WRITE flag and handle it
+ * @seals: the seals to check
+ * @vma: the vma to operate on
+ *
+ * Check whether F_SEAL_FUTURE_WRITE is set; if so, do proper check/handling on
+ * the vma flags. Return 0 if check pass, or <0 for errors.
+ */
+static inline int seal_check_future_write(int seals, struct vm_area_struct *vma)
+{
+ if (seals & F_SEAL_FUTURE_WRITE) {
+ /*
+ * New PROT_WRITE and MAP_SHARED mmaps are not allowed when
+ * "future write" seal active.
+ */
+ if ((vma->vm_flags & VM_SHARED) && (vma->vm_flags & VM_WRITE))
+ return -EPERM;
+
+ /*
+ * Since an F_SEAL_FUTURE_WRITE sealed memfd can be mapped as
+ * MAP_SHARED and read-only, take care to not allow mprotect to
+ * revert protections on such mappings. Do this only for shared
+ * mappings. For private mappings, don't need to mask
+ * VM_MAYWRITE as we still want them to be COW-writable.
+ */
+ if (vma->vm_flags & VM_SHARED)
+ vma->vm_flags &= ~(VM_MAYWRITE);
+ }
+
+ return 0;
+}
+
#endif /* __KERNEL__ */
#endif /* _LINUX_MM_H */
};
struct { /* page_pool used by netstack */
/**
- * @dma_addr: might require a 64-bit value even on
+ * @pp_magic: magic value to avoid recycling non
+ * page_pool allocated pages.
+ */
+ unsigned long pp_magic;
+ struct page_pool *pp;
+ unsigned long _pp_mapping_pad;
+ /**
+ * @dma_addr: might require a 64-bit value on
* 32-bit architectures.
*/
- dma_addr_t dma_addr;
+ unsigned long dma_addr[2];
};
struct { /* slab, slob and slub */
union {
*/
atomic_t has_pinned;
- /**
- * @write_protect_seq: Locked when any thread is write
- * protecting pages mapped by this mm to enforce a later COW,
- * for instance during page table copying for fork().
- */
- seqcount_t write_protect_seq;
-
#ifdef CONFIG_MMU
atomic_long_t pgtables_bytes; /* PTE page table pages */
#endif
spinlock_t page_table_lock; /* Protects page tables and some
* counters
*/
+ /*
+ * With some kernel config, the current mmap_lock's offset
+ * inside 'mm_struct' is at 0x120, which is very optimal, as
+ * its two hot fields 'count' and 'owner' sit in 2 different
+ * cachelines, and when mmap_lock is highly contended, both
+ * of the 2 fields will be accessed frequently, current layout
+ * will help to reduce cache bouncing.
+ *
+ * So please be careful with adding new fields before
+ * mmap_lock, which can easily push the 2 fields into one
+ * cacheline.
+ */
struct rw_semaphore mmap_lock;
struct list_head mmlist; /* List of maybe swapped mm's. These
unsigned long stack_vm; /* VM_STACK */
unsigned long def_flags;
+ /**
+ * @write_protect_seq: Locked when any thread is write
+ * protecting pages mapped by this mm to enforce a later COW,
+ * for instance during page table copying for fork().
+ */
+ seqcount_t write_protect_seq;
+
spinlock_t arg_lock; /* protect the below fields */
+
unsigned long start_code, end_code, start_data, end_data;
unsigned long start_brk, brk, start_stack;
unsigned long arg_start, arg_end, env_start, env_end;
struct rpmsg_device_id {
char name[RPMSG_NAME_SIZE];
+ kernel_ulong_t driver_data;
};
/* i2c */
#ifndef _I40E_CLIENT_H_
#define _I40E_CLIENT_H_
+#include <linux/auxiliary_bus.h>
+
#define I40E_CLIENT_STR_LENGTH 10
/* Client interface version should be updated anytime there is a change in the
struct i40e_qvlist_info {
u32 num_vectors;
- struct i40e_qv_info qv_info[1];
+ struct i40e_qv_info qv_info[];
};
u8 lanmac[6];
struct net_device *netdev;
struct pci_dev *pcidev;
+ struct auxiliary_device *aux_dev;
u8 __iomem *hw_addr;
u8 fid; /* function id, PF id or VF id */
#define I40E_CLIENT_FTYPE_PF 0
u32 fw_build; /* firmware build number */
};
+struct i40e_auxiliary_device {
+ struct auxiliary_device aux_dev;
+ struct i40e_info *ldev;
+};
+
#define I40E_CLIENT_RESET_LEVEL_PF 1
#define I40E_CLIENT_RESET_LEVEL_CORE 2
#define I40E_CLIENT_VSI_FLAG_TCP_ENABLE BIT(1)
return test_bit(__I40E_CLIENT_REGISTERED, &client->state);
}
+void i40e_client_device_register(struct i40e_info *ldev, struct i40e_client *client);
+void i40e_client_device_unregister(struct i40e_info *ldev);
/* used by clients */
int i40e_register_client(struct i40e_client *client);
int i40e_unregister_client(struct i40e_client *client);
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2021, Intel Corporation. */
+
+#ifndef _IIDC_H_
+#define _IIDC_H_
+
+#include <linux/auxiliary_bus.h>
+#include <linux/dcbnl.h>
+#include <linux/device.h>
+#include <linux/if_ether.h>
+#include <linux/kernel.h>
+#include <linux/netdevice.h>
+
+enum iidc_event_type {
+ IIDC_EVENT_BEFORE_MTU_CHANGE,
+ IIDC_EVENT_AFTER_MTU_CHANGE,
+ IIDC_EVENT_BEFORE_TC_CHANGE,
+ IIDC_EVENT_AFTER_TC_CHANGE,
+ IIDC_EVENT_CRIT_ERR,
+ IIDC_EVENT_NBITS /* must be last */
+};
+
+enum iidc_reset_type {
+ IIDC_PFR,
+ IIDC_CORER,
+ IIDC_GLOBR,
+};
+
+#define IIDC_MAX_USER_PRIORITY 8
+
+/* Struct to hold per RDMA Qset info */
+struct iidc_rdma_qset_params {
+ /* Qset TEID returned to the RDMA driver in
+ * ice_add_rdma_qset and used by RDMA driver
+ * for calls to ice_del_rdma_qset
+ */
+ u32 teid; /* Qset TEID */
+ u16 qs_handle; /* RDMA driver provides this */
+ u16 vport_id; /* VSI index */
+ u8 tc; /* TC branch the Qset should belong to */
+};
+
+struct iidc_qos_info {
+ u64 tc_ctx;
+ u8 rel_bw;
+ u8 prio_type;
+ u8 egress_virt_up;
+ u8 ingress_virt_up;
+};
+
+/* Struct to pass QoS info */
+struct iidc_qos_params {
+ struct iidc_qos_info tc_info[IEEE_8021QAZ_MAX_TCS];
+ u8 up2tc[IIDC_MAX_USER_PRIORITY];
+ u8 vport_relative_bw;
+ u8 vport_priority_type;
+ u8 num_tc;
+};
+
+struct iidc_event {
+ DECLARE_BITMAP(type, IIDC_EVENT_NBITS);
+ u32 reg;
+};
+
+struct ice_pf;
+
+int ice_add_rdma_qset(struct ice_pf *pf, struct iidc_rdma_qset_params *qset);
+int ice_del_rdma_qset(struct ice_pf *pf, struct iidc_rdma_qset_params *qset);
+int ice_rdma_request_reset(struct ice_pf *pf, enum iidc_reset_type reset_type);
+int ice_rdma_update_vsi_filter(struct ice_pf *pf, u16 vsi_id, bool enable);
+void ice_get_qos_params(struct ice_pf *pf, struct iidc_qos_params *qos);
+
+#define IIDC_RDMA_ROCE_NAME "roce"
+
+/* Structure representing auxiliary driver tailored information about the core
+ * PCI dev, each auxiliary driver using the IIDC interface will have an
+ * instance of this struct dedicated to it.
+ */
+
+struct iidc_auxiliary_dev {
+ struct auxiliary_device adev;
+ struct ice_pf *pf;
+};
+
+/* structure representing the auxiliary driver. This struct is to be
+ * allocated and populated by the auxiliary driver's owner. The core PCI
+ * driver will access these ops by performing a container_of on the
+ * auxiliary_device->dev.driver.
+ */
+struct iidc_auxiliary_drv {
+ struct auxiliary_driver adrv;
+ /* This event_handler is meant to be a blocking call. For instance,
+ * when a BEFORE_MTU_CHANGE event comes in, the event_handler will not
+ * return until the auxiliary driver is ready for the MTU change to
+ * happen.
+ */
+ void (*event_handler)(struct ice_pf *pf, struct iidc_event *event);
+};
+
+#endif /* _IIDC_H_*/
/*
* Add your fresh new feature above and remember to update
- * netdev_features_strings[] in net/core/ethtool.c and maybe
+ * netdev_features_strings[] in net/ethtool/common.c and maybe
* some feature mask #defines below. Please also describe it
* in Documentation/networking/netdev-features.rst.
*/
void __netdev_watchdog_up(struct net_device *dev);
void netif_carrier_on(struct net_device *dev);
-
void netif_carrier_off(struct net_device *dev);
+void netif_carrier_event(struct net_device *dev);
/**
* netif_dormant_on - mark device as dormant.
struct sock;
struct nf_hook_state {
- unsigned int hook;
- u_int8_t pf;
+ u8 hook;
+ u8 pf;
struct net_device *in;
struct net_device *out;
struct sock *sk;
typedef unsigned int nf_hookfn(void *priv,
struct sk_buff *skb,
const struct nf_hook_state *state);
+enum nf_hook_ops_type {
+ NF_HOOK_OP_UNDEFINED,
+ NF_HOOK_OP_NF_TABLES,
+};
+
struct nf_hook_ops {
/* User fills in from here down. */
nf_hookfn *hook;
struct net_device *dev;
void *priv;
- u_int8_t pf;
+ u8 pf;
+ enum nf_hook_ops_type hook_ops_type:8;
unsigned int hooknum;
/* Hooks are ordered in ascending priority. */
int priority;
struct net *net;
struct sock *sk;
const struct nlmsghdr *nlh;
+ const struct nfgenmsg *nfmsg;
struct netlink_ext_ack *extack;
};
const void *matchinfo, *targinfo;
};
const struct nf_hook_state *state;
- int fragoff;
unsigned int thoff;
+ u16 fragoff;
bool hotdrop;
};
return mdiobus_register(mdio);
}
+static inline int devm_of_mdiobus_register(struct device *dev,
+ struct mii_bus *mdio,
+ struct device_node *np)
+{
+ return devm_mdiobus_register(dev, mdio);
+}
+
static inline struct mdio_device *of_mdio_find_device(struct device_node *np)
{
return NULL;
* readahead_length - The number of bytes in this readahead request.
* @rac: The readahead request.
*/
-static inline loff_t readahead_length(struct readahead_control *rac)
+static inline size_t readahead_length(struct readahead_control *rac)
{
- return (loff_t)rac->_nr_pages * PAGE_SIZE;
+ return rac->_nr_pages * PAGE_SIZE;
}
/**
* readahead_batch_length - The number of bytes in the current batch.
* @rac: The readahead request.
*/
-static inline loff_t readahead_batch_length(struct readahead_control *rac)
+static inline size_t readahead_batch_length(struct readahead_control *rac)
{
return rac->_batch_count * PAGE_SIZE;
}
struct device_node;
struct irq_domain;
struct irq_domain *pci_host_bridge_of_msi_domain(struct pci_bus *bus);
+bool pci_host_of_has_msi_map(struct device *dev);
/* Arch may override this (weak) */
struct device_node *pcibios_get_phb_of_node(struct pci_bus *bus);
#else /* CONFIG_OF */
static inline struct irq_domain *
pci_host_bridge_of_msi_domain(struct pci_bus *bus) { return NULL; }
+static inline bool pci_host_of_has_msi_map(struct device *dev) { return false; }
#endif /* CONFIG_OF */
static inline struct device_node *
#include <linux/phy.h>
#include <linux/phylink.h>
+#define NXP_SJA1105_XPCS_ID 0x00000010
+#define NXP_SJA1110_XPCS_ID 0x00000020
+
/* AN mode */
#define DW_AN_C73 1
#define DW_AN_C37_SGMII 2
+#define DW_2500BASEX 3
-struct mdio_xpcs_args {
- __ETHTOOL_DECLARE_LINK_MODE_MASK(supported);
- struct mii_bus *bus;
- int addr;
- int an_mode;
-};
+struct xpcs_id;
-struct mdio_xpcs_ops {
- int (*validate)(struct mdio_xpcs_args *xpcs,
- unsigned long *supported,
- struct phylink_link_state *state);
- int (*config)(struct mdio_xpcs_args *xpcs,
- const struct phylink_link_state *state);
- int (*get_state)(struct mdio_xpcs_args *xpcs,
- struct phylink_link_state *state);
- int (*link_up)(struct mdio_xpcs_args *xpcs, int speed,
- phy_interface_t interface);
- int (*probe)(struct mdio_xpcs_args *xpcs, phy_interface_t interface);
+struct dw_xpcs {
+ struct mdio_device *mdiodev;
+ const struct xpcs_id *id;
+ struct phylink_pcs pcs;
};
-#if IS_ENABLED(CONFIG_PCS_XPCS)
-struct mdio_xpcs_ops *mdio_xpcs_get_ops(void);
-#else
-static inline struct mdio_xpcs_ops *mdio_xpcs_get_ops(void)
-{
- return NULL;
-}
-#endif
+int xpcs_get_an_mode(struct dw_xpcs *xpcs, phy_interface_t interface);
+void xpcs_link_up(struct phylink_pcs *pcs, unsigned int mode,
+ phy_interface_t interface, int speed, int duplex);
+int xpcs_do_config(struct dw_xpcs *xpcs, phy_interface_t interface,
+ unsigned int mode);
+void xpcs_validate(struct dw_xpcs *xpcs, unsigned long *supported,
+ struct phylink_link_state *state);
+int xpcs_config_eee(struct dw_xpcs *xpcs, int mult_fact_100ns,
+ int enable);
+struct dw_xpcs *xpcs_create(struct mdio_device *mdiodev,
+ phy_interface_t interface);
+void xpcs_destroy(struct dw_xpcs *xpcs);
#endif /* __LINUX_PCS_XPCS_H */
* To be differentiate with macro pte_mkyoung, this macro is used on platforms
* where software maintains page access bit.
*/
+#ifndef pte_sw_mkyoung
+static inline pte_t pte_sw_mkyoung(pte_t pte)
+{
+ return pte;
+}
+#define pte_sw_mkyoung pte_sw_mkyoung
+#endif
+
#ifndef pte_savedwrite
#define pte_savedwrite pte_write
#endif
* @PHY_INTERFACE_MODE_TBI: Ten Bit Interface
* @PHY_INTERFACE_MODE_REVMII: Reverse Media Independent Interface
* @PHY_INTERFACE_MODE_RMII: Reduced Media Independent Interface
+ * @PHY_INTERFACE_MODE_REVRMII: Reduced Media Independent Interface in PHY role
* @PHY_INTERFACE_MODE_RGMII: Reduced gigabit media-independent interface
* @PHY_INTERFACE_MODE_RGMII_ID: RGMII with Internal RX+TX delay
* @PHY_INTERFACE_MODE_RGMII_RXID: RGMII with Internal RX delay
* @PHY_INTERFACE_MODE_RXAUI: Reduced XAUI
* @PHY_INTERFACE_MODE_XAUI: 10 Gigabit Attachment Unit Interface
* @PHY_INTERFACE_MODE_10GBASER: 10G BaseR
+ * @PHY_INTERFACE_MODE_25GBASER: 25G BaseR
* @PHY_INTERFACE_MODE_USXGMII: Universal Serial 10GE MII
* @PHY_INTERFACE_MODE_10GKR: 10GBASE-KR - with Clause 73 AN
* @PHY_INTERFACE_MODE_MAX: Book keeping
PHY_INTERFACE_MODE_TBI,
PHY_INTERFACE_MODE_REVMII,
PHY_INTERFACE_MODE_RMII,
+ PHY_INTERFACE_MODE_REVRMII,
PHY_INTERFACE_MODE_RGMII,
PHY_INTERFACE_MODE_RGMII_ID,
PHY_INTERFACE_MODE_RGMII_RXID,
PHY_INTERFACE_MODE_XAUI,
/* 10GBASE-R, XFI, SFI - single lane 10G Serdes */
PHY_INTERFACE_MODE_10GBASER,
+ PHY_INTERFACE_MODE_25GBASER,
PHY_INTERFACE_MODE_USXGMII,
/* 10GBASE-KR - with Clause 73 AN */
PHY_INTERFACE_MODE_10GKR,
return "rev-mii";
case PHY_INTERFACE_MODE_RMII:
return "rmii";
+ case PHY_INTERFACE_MODE_REVRMII:
+ return "rev-rmii";
case PHY_INTERFACE_MODE_RGMII:
return "rgmii";
case PHY_INTERFACE_MODE_RGMII_ID:
return "xaui";
case PHY_INTERFACE_MODE_10GBASER:
return "10gbase-r";
+ case PHY_INTERFACE_MODE_25GBASER:
+ return "25gbase-r";
case PHY_INTERFACE_MODE_USXGMII:
return "usxgmii";
case PHY_INTERFACE_MODE_10GKR:
* @mac_managed_pm: Set true if MAC driver takes of suspending/resuming PHY
* @state: State of the PHY for management purposes
* @dev_flags: Device-specific flags used by the PHY driver.
+ * Bits [15:0] are free to use by the PHY driver to communicate
+ * driver specific behavior.
+ * Bits [23:16] are currently reserved for future use.
+ * Bits [31:24] are reserved for defining generic
+ * PHY driver behavior.
* @irq: IRQ number of the PHY's interrupt (-1 if none)
* @phy_timer: The timer for handling the state machine
* @phylink: Pointer to phylink instance for this PHY
bool is_c45,
struct phy_c45_device_ids *c45_ids);
#if IS_ENABLED(CONFIG_PHYLIB)
+int fwnode_get_phy_id(struct fwnode_handle *fwnode, u32 *phy_id);
+struct mdio_device *fwnode_mdio_find_device(struct fwnode_handle *fwnode);
+struct phy_device *fwnode_phy_find_device(struct fwnode_handle *phy_fwnode);
+struct phy_device *device_phy_find_device(struct device *dev);
+struct fwnode_handle *fwnode_get_phy_node(struct fwnode_handle *fwnode);
struct phy_device *get_phy_device(struct mii_bus *bus, int addr, bool is_c45);
int phy_device_register(struct phy_device *phy);
void phy_device_free(struct phy_device *phydev);
#else
+static inline int fwnode_get_phy_id(struct fwnode_handle *fwnode, u32 *phy_id)
+{
+ return 0;
+}
+static inline
+struct mdio_device *fwnode_mdio_find_device(struct fwnode_handle *fwnode)
+{
+ return 0;
+}
+
+static inline
+struct phy_device *fwnode_phy_find_device(struct fwnode_handle *phy_fwnode)
+{
+ return NULL;
+}
+
+static inline struct phy_device *device_phy_find_device(struct device *dev)
+{
+ return NULL;
+}
+
+static inline
+struct fwnode_handle *fwnode_get_phy_node(struct fwnode_handle *fwnode)
+{
+ return NULL;
+}
+
static inline
struct phy_device *get_phy_device(struct mii_bus *bus, int addr, bool is_c45)
{
int phylink_connect_phy(struct phylink *, struct phy_device *);
int phylink_of_phy_connect(struct phylink *, struct device_node *, u32 flags);
+int phylink_fwnode_phy_connect(struct phylink *pl,
+ struct fwnode_handle *fwnode,
+ u32 flags);
void phylink_disconnect_phy(struct phylink *);
void phylink_mac_change(struct phylink *, bool up);
s8 emufree_shift;
};
+#define SYSC_QUIRK_REINIT_ON_RESUME BIT(27)
#define SYSC_QUIRK_GPMC_DEBUG BIT(26)
#define SYSC_MODULE_QUIRK_ENA_RESETDONE BIT(25)
#define SYSC_MODULE_QUIRK_PRUSS BIT(24)
unsigned int idle_notification:1;
unsigned int request_pending:1;
unsigned int deferred_resume:1;
+ unsigned int needs_force_resume:1;
unsigned int runtime_auto:1;
bool ignore_children:1;
unsigned int no_callbacks:1;
/********** security/ **********/
#define KEY_DESTROY 0xbd
+/********** net/core/page_pool.c **********/
+#define PP_SIGNATURE (0x40 + POISON_POINTER_DELTA)
+
#endif
*
* @ppm: Parts per million, but with a 16 bit binary fractional field
*/
-static inline s32 scaled_ppm_to_ppb(long ppm)
+static inline long scaled_ppm_to_ppb(long ppm)
{
/*
* The 'freq' field in the 'struct timex' is in parts per
ppb *= 125;
ppb >>= 13;
- return (s32)ppb;
+ return (long)ppb;
}
#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK)
/* Per-protocol connection types */
enum protocol_type {
- PROTOCOLID_ISCSI,
+ PROTOCOLID_TCP_ULP,
PROTOCOLID_FCOE,
PROTOCOLID_ROCE,
PROTOCOLID_CORE,
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) */
+/* Copyright 2021 Marvell. All rights reserved. */
+
+#ifndef __NVMETCP_COMMON__
+#define __NVMETCP_COMMON__
+
+#include "tcp_common.h"
+#include <linux/nvme-tcp.h>
+
+#define NVMETCP_SLOW_PATH_LAYER_CODE (6)
+#define NVMETCP_WQE_NUM_SGES_SLOWIO (0xf)
+
+/* NVMeTCP firmware function init parameters */
+struct nvmetcp_spe_func_init {
+ __le16 half_way_close_timeout;
+ u8 num_sq_pages_in_ring;
+ u8 num_r2tq_pages_in_ring;
+ u8 num_uhq_pages_in_ring;
+ u8 ll2_rx_queue_id;
+ u8 flags;
+#define NVMETCP_SPE_FUNC_INIT_COUNTERS_EN_MASK 0x1
+#define NVMETCP_SPE_FUNC_INIT_COUNTERS_EN_SHIFT 0
+#define NVMETCP_SPE_FUNC_INIT_NVMETCP_MODE_MASK 0x1
+#define NVMETCP_SPE_FUNC_INIT_NVMETCP_MODE_SHIFT 1
+#define NVMETCP_SPE_FUNC_INIT_RESERVED0_MASK 0x3F
+#define NVMETCP_SPE_FUNC_INIT_RESERVED0_SHIFT 2
+ u8 debug_flags;
+ __le16 reserved1;
+ u8 params;
+#define NVMETCP_SPE_FUNC_INIT_MAX_SYN_RT_MASK 0xF
+#define NVMETCP_SPE_FUNC_INIT_MAX_SYN_RT_SHIFT 0
+#define NVMETCP_SPE_FUNC_INIT_RESERVED1_MASK 0xF
+#define NVMETCP_SPE_FUNC_INIT_RESERVED1_SHIFT 4
+ u8 reserved2[5];
+ struct scsi_init_func_params func_params;
+ struct scsi_init_func_queues q_params;
+};
+
+/* NVMeTCP init params passed by driver to FW in NVMeTCP init ramrod. */
+struct nvmetcp_init_ramrod_params {
+ struct nvmetcp_spe_func_init nvmetcp_init_spe;
+ struct tcp_init_params tcp_init;
+};
+
+/* NVMeTCP Ramrod Command IDs */
+enum nvmetcp_ramrod_cmd_id {
+ NVMETCP_RAMROD_CMD_ID_UNUSED = 0,
+ NVMETCP_RAMROD_CMD_ID_INIT_FUNC = 1,
+ NVMETCP_RAMROD_CMD_ID_DESTROY_FUNC = 2,
+ NVMETCP_RAMROD_CMD_ID_OFFLOAD_CONN = 3,
+ NVMETCP_RAMROD_CMD_ID_UPDATE_CONN = 4,
+ NVMETCP_RAMROD_CMD_ID_TERMINATION_CONN = 5,
+ NVMETCP_RAMROD_CMD_ID_CLEAR_SQ = 6,
+ MAX_NVMETCP_RAMROD_CMD_ID
+};
+
+struct nvmetcp_glbl_queue_entry {
+ struct regpair cq_pbl_addr;
+ struct regpair reserved;
+};
+
+/* NVMeTCP conn level EQEs */
+enum nvmetcp_eqe_opcode {
+ NVMETCP_EVENT_TYPE_INIT_FUNC = 0, /* Response after init Ramrod */
+ NVMETCP_EVENT_TYPE_DESTROY_FUNC, /* Response after destroy Ramrod */
+ NVMETCP_EVENT_TYPE_OFFLOAD_CONN,/* Response after option 2 offload Ramrod */
+ NVMETCP_EVENT_TYPE_UPDATE_CONN, /* Response after update Ramrod */
+ NVMETCP_EVENT_TYPE_CLEAR_SQ, /* Response after clear sq Ramrod */
+ NVMETCP_EVENT_TYPE_TERMINATE_CONN, /* Response after termination Ramrod */
+ NVMETCP_EVENT_TYPE_RESERVED0,
+ NVMETCP_EVENT_TYPE_RESERVED1,
+ NVMETCP_EVENT_TYPE_ASYN_CONNECT_COMPLETE, /* Connect completed (A-syn EQE) */
+ NVMETCP_EVENT_TYPE_ASYN_TERMINATE_DONE, /* Termination completed (A-syn EQE) */
+ NVMETCP_EVENT_TYPE_START_OF_ERROR_TYPES = 10, /* Separate EQs from err EQs */
+ NVMETCP_EVENT_TYPE_ASYN_ABORT_RCVD, /* TCP RST packet receive (A-syn EQE) */
+ NVMETCP_EVENT_TYPE_ASYN_CLOSE_RCVD, /* TCP FIN packet receive (A-syn EQE) */
+ NVMETCP_EVENT_TYPE_ASYN_SYN_RCVD, /* TCP SYN+ACK packet receive (A-syn EQE) */
+ NVMETCP_EVENT_TYPE_ASYN_MAX_RT_TIME, /* TCP max retransmit time (A-syn EQE) */
+ NVMETCP_EVENT_TYPE_ASYN_MAX_RT_CNT, /* TCP max retransmit count (A-syn EQE) */
+ NVMETCP_EVENT_TYPE_ASYN_MAX_KA_PROBES_CNT, /* TCP ka probes count (A-syn EQE) */
+ NVMETCP_EVENT_TYPE_ASYN_FIN_WAIT2, /* TCP fin wait 2 (A-syn EQE) */
+ NVMETCP_EVENT_TYPE_NVMETCP_CONN_ERROR, /* NVMeTCP error response (A-syn EQE) */
+ NVMETCP_EVENT_TYPE_TCP_CONN_ERROR, /* NVMeTCP error - tcp error (A-syn EQE) */
+ MAX_NVMETCP_EQE_OPCODE
+};
+
+struct nvmetcp_conn_offload_section {
+ struct regpair cccid_itid_table_addr; /* CCCID to iTID table address */
+ __le16 cccid_max_range; /* CCCID max value - used for validation */
+ __le16 reserved[3];
+};
+
+/* NVMe TCP connection offload params passed by driver to FW in NVMeTCP offload ramrod */
+struct nvmetcp_conn_offload_params {
+ struct regpair sq_pbl_addr;
+ struct regpair r2tq_pbl_addr;
+ struct regpair xhq_pbl_addr;
+ struct regpair uhq_pbl_addr;
+ __le16 physical_q0;
+ __le16 physical_q1;
+ u8 flags;
+#define NVMETCP_CONN_OFFLOAD_PARAMS_TCP_ON_CHIP_1B_MASK 0x1
+#define NVMETCP_CONN_OFFLOAD_PARAMS_TCP_ON_CHIP_1B_SHIFT 0
+#define NVMETCP_CONN_OFFLOAD_PARAMS_TARGET_MODE_MASK 0x1
+#define NVMETCP_CONN_OFFLOAD_PARAMS_TARGET_MODE_SHIFT 1
+#define NVMETCP_CONN_OFFLOAD_PARAMS_RESTRICTED_MODE_MASK 0x1
+#define NVMETCP_CONN_OFFLOAD_PARAMS_RESTRICTED_MODE_SHIFT 2
+#define NVMETCP_CONN_OFFLOAD_PARAMS_NVMETCP_MODE_MASK 0x1
+#define NVMETCP_CONN_OFFLOAD_PARAMS_NVMETCP_MODE_SHIFT 3
+#define NVMETCP_CONN_OFFLOAD_PARAMS_RESERVED1_MASK 0xF
+#define NVMETCP_CONN_OFFLOAD_PARAMS_RESERVED1_SHIFT 4
+ u8 default_cq;
+ __le16 reserved0;
+ __le32 reserved1;
+ __le32 initial_ack;
+
+ struct nvmetcp_conn_offload_section nvmetcp; /* NVMe/TCP section */
+};
+
+/* NVMe TCP and TCP connection offload params passed by driver to FW in NVMeTCP offload ramrod. */
+struct nvmetcp_spe_conn_offload {
+ __le16 reserved;
+ __le16 conn_id;
+ __le32 fw_cid;
+ struct nvmetcp_conn_offload_params nvmetcp;
+ struct tcp_offload_params_opt2 tcp;
+};
+
+/* NVMeTCP connection update params passed by driver to FW in NVMETCP update ramrod. */
+struct nvmetcp_conn_update_ramrod_params {
+ __le16 reserved0;
+ __le16 conn_id;
+ __le32 reserved1;
+ u8 flags;
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_HD_EN_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_HD_EN_SHIFT 0
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_DD_EN_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_DD_EN_SHIFT 1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED0_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED0_SHIFT 2
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED1_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED1_DATA_SHIFT 3
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED2_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED2_SHIFT 4
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED3_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED3_SHIFT 5
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED4_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED4_SHIFT 6
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED5_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED5_SHIFT 7
+ u8 reserved3[3];
+ __le32 max_seq_size;
+ __le32 max_send_pdu_length;
+ __le32 max_recv_pdu_length;
+ __le32 first_seq_length;
+ __le32 reserved4[5];
+};
+
+/* NVMeTCP connection termination request */
+struct nvmetcp_spe_conn_termination {
+ __le16 reserved0;
+ __le16 conn_id;
+ __le32 reserved1;
+ u8 abortive;
+ u8 reserved2[7];
+ struct regpair reserved3;
+ struct regpair reserved4;
+};
+
+struct nvmetcp_dif_flags {
+ u8 flags;
+};
+
+enum nvmetcp_wqe_type {
+ NVMETCP_WQE_TYPE_NORMAL,
+ NVMETCP_WQE_TYPE_TASK_CLEANUP,
+ NVMETCP_WQE_TYPE_MIDDLE_PATH,
+ NVMETCP_WQE_TYPE_IC,
+ MAX_NVMETCP_WQE_TYPE
+};
+
+struct nvmetcp_wqe {
+ __le16 task_id;
+ u8 flags;
+#define NVMETCP_WQE_WQE_TYPE_MASK 0x7 /* [use nvmetcp_wqe_type] */
+#define NVMETCP_WQE_WQE_TYPE_SHIFT 0
+#define NVMETCP_WQE_NUM_SGES_MASK 0xF
+#define NVMETCP_WQE_NUM_SGES_SHIFT 3
+#define NVMETCP_WQE_RESPONSE_MASK 0x1
+#define NVMETCP_WQE_RESPONSE_SHIFT 7
+ struct nvmetcp_dif_flags prot_flags;
+ __le32 contlen_cdbsize;
+#define NVMETCP_WQE_CONT_LEN_MASK 0xFFFFFF
+#define NVMETCP_WQE_CONT_LEN_SHIFT 0
+#define NVMETCP_WQE_CDB_SIZE_OR_NVMETCP_CMD_MASK 0xFF
+#define NVMETCP_WQE_CDB_SIZE_OR_NVMETCP_CMD_SHIFT 24
+};
+
+struct nvmetcp_host_cccid_itid_entry {
+ __le16 itid;
+};
+
+struct nvmetcp_connect_done_results {
+ __le16 icid;
+ __le16 conn_id;
+ struct tcp_ulp_connect_done_params params;
+};
+
+struct nvmetcp_eqe_data {
+ __le16 icid;
+ __le16 conn_id;
+ __le16 reserved;
+ u8 error_code;
+ u8 error_pdu_opcode_reserved;
+#define NVMETCP_EQE_DATA_ERROR_PDU_OPCODE_MASK 0x3F
+#define NVMETCP_EQE_DATA_ERROR_PDU_OPCODE_SHIFT 0
+#define NVMETCP_EQE_DATA_ERROR_PDU_OPCODE_VALID_MASK 0x1
+#define NVMETCP_EQE_DATA_ERROR_PDU_OPCODE_VALID_SHIFT 6
+#define NVMETCP_EQE_DATA_RESERVED0_MASK 0x1
+#define NVMETCP_EQE_DATA_RESERVED0_SHIFT 7
+};
+
+enum nvmetcp_task_type {
+ NVMETCP_TASK_TYPE_HOST_WRITE,
+ NVMETCP_TASK_TYPE_HOST_READ,
+ NVMETCP_TASK_TYPE_INIT_CONN_REQUEST,
+ NVMETCP_TASK_TYPE_RESERVED0,
+ NVMETCP_TASK_TYPE_CLEANUP,
+ NVMETCP_TASK_TYPE_HOST_READ_NO_CQE,
+ MAX_NVMETCP_TASK_TYPE
+};
+
+struct nvmetcp_db_data {
+ u8 params;
+#define NVMETCP_DB_DATA_DEST_MASK 0x3 /* destination of doorbell (use enum db_dest) */
+#define NVMETCP_DB_DATA_DEST_SHIFT 0
+#define NVMETCP_DB_DATA_AGG_CMD_MASK 0x3 /* aggregative command to CM (use enum db_agg_cmd_sel) */
+#define NVMETCP_DB_DATA_AGG_CMD_SHIFT 2
+#define NVMETCP_DB_DATA_BYPASS_EN_MASK 0x1 /* enable QM bypass */
+#define NVMETCP_DB_DATA_BYPASS_EN_SHIFT 4
+#define NVMETCP_DB_DATA_RESERVED_MASK 0x1
+#define NVMETCP_DB_DATA_RESERVED_SHIFT 5
+#define NVMETCP_DB_DATA_AGG_VAL_SEL_MASK 0x3 /* aggregative value selection */
+#define NVMETCP_DB_DATA_AGG_VAL_SEL_SHIFT 6
+ u8 agg_flags; /* bit for every DQ counter flags in CM context that DQ can increment */
+ __le16 sq_prod;
+};
+
+struct nvmetcp_fw_nvmf_cqe {
+ __le32 reserved[4];
+};
+
+struct nvmetcp_icresp_mdata {
+ u8 digest;
+ u8 cpda;
+ __le16 pfv;
+ __le32 maxdata;
+ __le16 rsvd[4];
+};
+
+union nvmetcp_fw_cqe_data {
+ struct nvmetcp_fw_nvmf_cqe nvme_cqe;
+ struct nvmetcp_icresp_mdata icresp_mdata;
+};
+
+struct nvmetcp_fw_cqe {
+ __le16 conn_id;
+ u8 cqe_type;
+ u8 cqe_error_status_bits;
+#define CQE_ERROR_BITMAP_DIF_ERR_BITS_MASK 0x7
+#define CQE_ERROR_BITMAP_DIF_ERR_BITS_SHIFT 0
+#define CQE_ERROR_BITMAP_DATA_DIGEST_ERR_MASK 0x1
+#define CQE_ERROR_BITMAP_DATA_DIGEST_ERR_SHIFT 3
+#define CQE_ERROR_BITMAP_RCV_ON_INVALID_CONN_MASK 0x1
+#define CQE_ERROR_BITMAP_RCV_ON_INVALID_CONN_SHIFT 4
+ __le16 itid;
+ u8 task_type;
+ u8 fw_dbg_field;
+ u8 caused_conn_err;
+ u8 reserved0[3];
+ __le32 reserved1;
+ union nvmetcp_fw_cqe_data cqe_data;
+ struct regpair task_opaque;
+ __le32 reserved[6];
+};
+
+enum nvmetcp_fw_cqes_type {
+ NVMETCP_FW_CQE_TYPE_NORMAL = 1,
+ NVMETCP_FW_CQE_TYPE_RESERVED0,
+ NVMETCP_FW_CQE_TYPE_RESERVED1,
+ NVMETCP_FW_CQE_TYPE_CLEANUP,
+ NVMETCP_FW_CQE_TYPE_DUMMY,
+ MAX_NVMETCP_FW_CQES_TYPE
+};
+
+struct ystorm_nvmetcp_task_state {
+ struct scsi_cached_sges data_desc;
+ struct scsi_sgl_params sgl_params;
+ __le32 resrved0;
+ __le32 buffer_offset;
+ __le16 cccid;
+ struct nvmetcp_dif_flags dif_flags;
+ u8 flags;
+#define YSTORM_NVMETCP_TASK_STATE_LOCAL_COMP_MASK 0x1
+#define YSTORM_NVMETCP_TASK_STATE_LOCAL_COMP_SHIFT 0
+#define YSTORM_NVMETCP_TASK_STATE_SLOW_IO_MASK 0x1
+#define YSTORM_NVMETCP_TASK_STATE_SLOW_IO_SHIFT 1
+#define YSTORM_NVMETCP_TASK_STATE_SET_DIF_OFFSET_MASK 0x1
+#define YSTORM_NVMETCP_TASK_STATE_SET_DIF_OFFSET_SHIFT 2
+#define YSTORM_NVMETCP_TASK_STATE_SEND_W_RSP_MASK 0x1
+#define YSTORM_NVMETCP_TASK_STATE_SEND_W_RSP_SHIFT 3
+};
+
+struct ystorm_nvmetcp_task_rxmit_opt {
+ __le32 reserved[4];
+};
+
+struct nvmetcp_task_hdr {
+ __le32 reg[18];
+};
+
+struct nvmetcp_task_hdr_aligned {
+ struct nvmetcp_task_hdr task_hdr;
+ __le32 reserved[2]; /* HSI_COMMENT: Align to QREG */
+};
+
+struct e5_tdif_task_context {
+ __le32 reserved[16];
+};
+
+struct e5_rdif_task_context {
+ __le32 reserved[12];
+};
+
+struct ystorm_nvmetcp_task_st_ctx {
+ struct ystorm_nvmetcp_task_state state;
+ struct ystorm_nvmetcp_task_rxmit_opt rxmit_opt;
+ struct nvmetcp_task_hdr_aligned pdu_hdr;
+};
+
+struct mstorm_nvmetcp_task_st_ctx {
+ struct scsi_cached_sges data_desc;
+ struct scsi_sgl_params sgl_params;
+ __le32 rem_task_size;
+ __le32 data_buffer_offset;
+ u8 task_type;
+ struct nvmetcp_dif_flags dif_flags;
+ __le16 dif_task_icid;
+ struct regpair reserved0;
+ __le32 expected_itt;
+ __le32 reserved1;
+};
+
+struct ustorm_nvmetcp_task_st_ctx {
+ __le32 rem_rcv_len;
+ __le32 exp_data_transfer_len;
+ __le32 exp_data_sn;
+ struct regpair reserved0;
+ __le32 reg1_map;
+#define REG1_NUM_SGES_MASK 0xF
+#define REG1_NUM_SGES_SHIFT 0
+#define REG1_RESERVED1_MASK 0xFFFFFFF
+#define REG1_RESERVED1_SHIFT 4
+ u8 flags2;
+#define USTORM_NVMETCP_TASK_ST_CTX_AHS_EXIST_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_AHS_EXIST_SHIFT 0
+#define USTORM_NVMETCP_TASK_ST_CTX_RESERVED1_MASK 0x7F
+#define USTORM_NVMETCP_TASK_ST_CTX_RESERVED1_SHIFT 1
+ struct nvmetcp_dif_flags dif_flags;
+ __le16 reserved3;
+ __le16 tqe_opaque[2];
+ __le32 reserved5;
+ __le32 nvme_tcp_opaque_lo;
+ __le32 nvme_tcp_opaque_hi;
+ u8 task_type;
+ u8 error_flags;
+#define USTORM_NVMETCP_TASK_ST_CTX_DATA_DIGEST_ERROR_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_DATA_DIGEST_ERROR_SHIFT 0
+#define USTORM_NVMETCP_TASK_ST_CTX_DATA_TRUNCATED_ERROR_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_DATA_TRUNCATED_ERROR_SHIFT 1
+#define USTORM_NVMETCP_TASK_ST_CTX_UNDER_RUN_ERROR_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_UNDER_RUN_ERROR_SHIFT 2
+#define USTORM_NVMETCP_TASK_ST_CTX_NVME_TCP_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_NVME_TCP_SHIFT 3
+ u8 flags;
+#define USTORM_NVMETCP_TASK_ST_CTX_CQE_WRITE_MASK 0x3
+#define USTORM_NVMETCP_TASK_ST_CTX_CQE_WRITE_SHIFT 0
+#define USTORM_NVMETCP_TASK_ST_CTX_LOCAL_COMP_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_LOCAL_COMP_SHIFT 2
+#define USTORM_NVMETCP_TASK_ST_CTX_Q0_R2TQE_WRITE_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_Q0_R2TQE_WRITE_SHIFT 3
+#define USTORM_NVMETCP_TASK_ST_CTX_TOTAL_DATA_ACKED_DONE_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_TOTAL_DATA_ACKED_DONE_SHIFT 4
+#define USTORM_NVMETCP_TASK_ST_CTX_HQ_SCANNED_DONE_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_HQ_SCANNED_DONE_SHIFT 5
+#define USTORM_NVMETCP_TASK_ST_CTX_R2T2RECV_DONE_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_R2T2RECV_DONE_SHIFT 6
+ u8 cq_rss_number;
+};
+
+struct e5_ystorm_nvmetcp_task_ag_ctx {
+ u8 reserved /* cdu_validation */;
+ u8 byte1 /* state_and_core_id */;
+ __le16 word0 /* icid */;
+ u8 flags0;
+ u8 flags1;
+ u8 flags2;
+ u8 flags3;
+ __le32 TTT;
+ u8 byte2;
+ u8 byte3;
+ u8 byte4;
+ u8 e4_reserved7;
+};
+
+struct e5_mstorm_nvmetcp_task_ag_ctx {
+ u8 cdu_validation;
+ u8 byte1;
+ __le16 task_cid;
+ u8 flags0;
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CONNECTION_TYPE_MASK 0xF
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CONNECTION_TYPE_SHIFT 0
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_EXIST_IN_QM0_MASK 0x1
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_EXIST_IN_QM0_SHIFT 4
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CONN_CLEAR_SQ_FLAG_MASK 0x1
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CONN_CLEAR_SQ_FLAG_SHIFT 5
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_VALID_MASK 0x1
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_VALID_SHIFT 6
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_FLAG_MASK 0x1
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_FLAG_SHIFT 7
+ u8 flags1;
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_CF_MASK 0x3
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_CF_SHIFT 0
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF1_MASK 0x3
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF1_SHIFT 2
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF2_MASK 0x3
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF2_SHIFT 4
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_CF_EN_MASK 0x1
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_CF_EN_SHIFT 6
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF1EN_MASK 0x1
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF1EN_SHIFT 7
+ u8 flags2;
+ u8 flags3;
+ __le32 reg0;
+ u8 byte2;
+ u8 byte3;
+ u8 byte4;
+ u8 e4_reserved7;
+};
+
+struct e5_ustorm_nvmetcp_task_ag_ctx {
+ u8 reserved;
+ u8 state_and_core_id;
+ __le16 icid;
+ u8 flags0;
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CONNECTION_TYPE_MASK 0xF
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CONNECTION_TYPE_SHIFT 0
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_EXIST_IN_QM0_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_EXIST_IN_QM0_SHIFT 4
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CONN_CLEAR_SQ_FLAG_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CONN_CLEAR_SQ_FLAG_SHIFT 5
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_HQ_SCANNED_CF_MASK 0x3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_HQ_SCANNED_CF_SHIFT 6
+ u8 flags1;
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_RESERVED1_MASK 0x3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_RESERVED1_SHIFT 0
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_R2T2RECV_MASK 0x3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_R2T2RECV_SHIFT 2
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CF3_MASK 0x3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CF3_SHIFT 4
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_CF_MASK 0x3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_CF_SHIFT 6
+ u8 flags2;
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_HQ_SCANNED_CF_EN_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_HQ_SCANNED_CF_EN_SHIFT 0
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DISABLE_DATA_ACKED_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DISABLE_DATA_ACKED_SHIFT 1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_R2T2RECV_EN_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_R2T2RECV_EN_SHIFT 2
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CF3EN_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CF3EN_SHIFT 3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_CF_EN_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_CF_EN_SHIFT 4
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CMP_DATA_TOTAL_EXP_EN_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CMP_DATA_TOTAL_EXP_EN_SHIFT 5
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_RULE1EN_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_RULE1EN_SHIFT 6
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CMP_CONT_RCV_EXP_EN_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CMP_CONT_RCV_EXP_EN_SHIFT 7
+ u8 flags3;
+ u8 flags4;
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED5_MASK 0x3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED5_SHIFT 0
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED6_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED6_SHIFT 2
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED7_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED7_SHIFT 3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_TYPE_MASK 0xF
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_TYPE_SHIFT 4
+ u8 byte2;
+ u8 byte3;
+ u8 e4_reserved8;
+ __le32 dif_err_intervals;
+ __le32 dif_error_1st_interval;
+ __le32 rcv_cont_len;
+ __le32 exp_cont_len;
+ __le32 total_data_acked;
+ __le32 exp_data_acked;
+ __le16 word1;
+ __le16 next_tid;
+ __le32 hdr_residual_count;
+ __le32 exp_r2t_sn;
+};
+
+struct e5_nvmetcp_task_context {
+ struct ystorm_nvmetcp_task_st_ctx ystorm_st_context;
+ struct e5_ystorm_nvmetcp_task_ag_ctx ystorm_ag_context;
+ struct regpair ystorm_ag_padding[2];
+ struct e5_tdif_task_context tdif_context;
+ struct e5_mstorm_nvmetcp_task_ag_ctx mstorm_ag_context;
+ struct regpair mstorm_ag_padding[2];
+ struct e5_ustorm_nvmetcp_task_ag_ctx ustorm_ag_context;
+ struct regpair ustorm_ag_padding[2];
+ struct mstorm_nvmetcp_task_st_ctx mstorm_st_context;
+ struct regpair mstorm_st_padding[2];
+ struct ustorm_nvmetcp_task_st_ctx ustorm_st_context;
+ struct regpair ustorm_st_padding[2];
+ struct e5_rdif_task_context rdif_context;
+};
+
+#endif /* __NVMETCP_COMMON__*/
u8 bdq_pbl_num_entries[3];
};
+struct qed_nvmetcp_pf_params {
+ u64 glbl_q_params_addr;
+ u16 cq_num_entries;
+ u16 num_cons;
+ u16 num_tasks;
+ u8 num_sq_pages_in_ring;
+ u8 num_r2tq_pages_in_ring;
+ u8 num_uhq_pages_in_ring;
+ u8 num_queues;
+ u8 gl_rq_pi;
+ u8 gl_cmd_pi;
+ u8 debug_mode;
+ u8 ll2_ooo_queue_id;
+ u16 min_rto;
+};
+
struct qed_rdma_pf_params {
/* Supplied to QED during resource allocation (may affect the ILT and
* the doorbell BAR).
struct qed_eth_pf_params eth_pf_params;
struct qed_fcoe_pf_params fcoe_pf_params;
struct qed_iscsi_pf_params iscsi_pf_params;
+ struct qed_nvmetcp_pf_params nvmetcp_pf_params;
struct qed_rdma_pf_params rdma_pf_params;
};
enum qed_protocol {
QED_PROTOCOL_ETH,
QED_PROTOCOL_ISCSI,
+ QED_PROTOCOL_NVMETCP = QED_PROTOCOL_ISCSI,
QED_PROTOCOL_FCOE,
};
enum qed_ll2_conn_type {
QED_LL2_TYPE_FCOE,
- QED_LL2_TYPE_ISCSI,
+ QED_LL2_TYPE_TCP_ULP,
QED_LL2_TYPE_TEST,
QED_LL2_TYPE_OOO,
QED_LL2_TYPE_RESERVED2,
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) */
+/* Copyright 2021 Marvell. All rights reserved. */
+
+#ifndef _QED_NVMETCP_IF_H
+#define _QED_NVMETCP_IF_H
+#include <linux/types.h>
+#include <linux/qed/qed_if.h>
+#include <linux/qed/storage_common.h>
+#include <linux/qed/nvmetcp_common.h>
+
+#define QED_NVMETCP_MAX_IO_SIZE 0x800000
+#define QED_NVMETCP_CMN_HDR_SIZE (sizeof(struct nvme_tcp_hdr))
+#define QED_NVMETCP_CMD_HDR_SIZE (sizeof(struct nvme_tcp_cmd_pdu))
+#define QED_NVMETCP_NON_IO_HDR_SIZE ((QED_NVMETCP_CMN_HDR_SIZE + 16))
+
+typedef int (*nvmetcp_event_cb_t) (void *context,
+ u8 fw_event_code, void *fw_handle);
+
+struct qed_dev_nvmetcp_info {
+ struct qed_dev_info common;
+ u8 port_id; /* Physical port */
+ u8 num_cqs;
+};
+
+#define MAX_TID_BLOCKS_NVMETCP (512)
+struct qed_nvmetcp_tid {
+ u32 size; /* In bytes per task */
+ u32 num_tids_per_block;
+ u8 *blocks[MAX_TID_BLOCKS_NVMETCP];
+};
+
+struct qed_nvmetcp_id_params {
+ u8 mac[ETH_ALEN];
+ u32 ip[4];
+ u16 port;
+};
+
+struct qed_nvmetcp_params_offload {
+ /* FW initializations */
+ dma_addr_t sq_pbl_addr;
+ dma_addr_t nvmetcp_cccid_itid_table_addr;
+ u16 nvmetcp_cccid_max_range;
+ u8 default_cq;
+
+ /* Networking and TCP stack initializations */
+ struct qed_nvmetcp_id_params src;
+ struct qed_nvmetcp_id_params dst;
+ u32 ka_timeout;
+ u32 ka_interval;
+ u32 max_rt_time;
+ u32 cwnd;
+ u16 mss;
+ u16 vlan_id;
+ bool timestamp_en;
+ bool delayed_ack_en;
+ bool tcp_keep_alive_en;
+ bool ecn_en;
+ u8 ip_version;
+ u8 ka_max_probe_cnt;
+ u8 ttl;
+ u8 tos_or_tc;
+ u8 rcv_wnd_scale;
+};
+
+struct qed_nvmetcp_params_update {
+ u32 max_io_size;
+ u32 max_recv_pdu_length;
+ u32 max_send_pdu_length;
+
+ /* Placeholder: pfv, cpda, hpda */
+
+ bool hdr_digest_en;
+ bool data_digest_en;
+};
+
+struct qed_nvmetcp_cb_ops {
+ struct qed_common_cb_ops common;
+};
+
+struct nvmetcp_sge {
+ struct regpair sge_addr; /* SGE address */
+ __le32 sge_len; /* SGE length */
+ __le32 reserved;
+};
+
+/* IO path HSI function SGL params */
+struct storage_sgl_task_params {
+ struct nvmetcp_sge *sgl;
+ struct regpair sgl_phys_addr;
+ u32 total_buffer_size;
+ u16 num_sges;
+ bool small_mid_sge;
+};
+
+/* IO path HSI function FW task context params */
+struct nvmetcp_task_params {
+ void *context; /* Output parameter - set/filled by the HSI function */
+ struct nvmetcp_wqe *sqe;
+ u32 tx_io_size; /* in bytes (Without DIF, if exists) */
+ u32 rx_io_size; /* in bytes (Without DIF, if exists) */
+ u16 conn_icid;
+ u16 itid;
+ struct regpair opq; /* qedn_task_ctx address */
+ u16 host_cccid;
+ u8 cq_rss_number;
+ bool send_write_incapsule;
+};
+
+/**
+ * struct qed_nvmetcp_ops - qed NVMeTCP operations.
+ * @common: common operations pointer
+ * @ll2: light L2 operations pointer
+ * @fill_dev_info: fills NVMeTCP specific information
+ * @param cdev
+ * @param info
+ * @return 0 on success, otherwise error value.
+ * @register_ops: register nvmetcp operations
+ * @param cdev
+ * @param ops - specified using qed_nvmetcp_cb_ops
+ * @param cookie - driver private
+ * @start: nvmetcp in FW
+ * @param cdev
+ * @param tasks - qed will fill information about tasks
+ * return 0 on success, otherwise error value.
+ * @stop: nvmetcp in FW
+ * @param cdev
+ * return 0 on success, otherwise error value.
+ * @acquire_conn: acquire a new nvmetcp connection
+ * @param cdev
+ * @param handle - qed will fill handle that should be
+ * used henceforth as identifier of the
+ * connection.
+ * @param p_doorbell - qed will fill the address of the
+ * doorbell.
+ * @return 0 on sucesss, otherwise error value.
+ * @release_conn: release a previously acquired nvmetcp connection
+ * @param cdev
+ * @param handle - the connection handle.
+ * @return 0 on success, otherwise error value.
+ * @offload_conn: configures an offloaded connection
+ * @param cdev
+ * @param handle - the connection handle.
+ * @param conn_info - the configuration to use for the
+ * offload.
+ * @return 0 on success, otherwise error value.
+ * @update_conn: updates an offloaded connection
+ * @param cdev
+ * @param handle - the connection handle.
+ * @param conn_info - the configuration to use for the
+ * offload.
+ * @return 0 on success, otherwise error value.
+ * @destroy_conn: stops an offloaded connection
+ * @param cdev
+ * @param handle - the connection handle.
+ * @return 0 on success, otherwise error value.
+ * @clear_sq: clear all task in sq
+ * @param cdev
+ * @param handle - the connection handle.
+ * @return 0 on success, otherwise error value.
+ * @add_src_tcp_port_filter: Add source tcp port filter
+ * @param cdev
+ * @param src_port
+ * @remove_src_tcp_port_filter: Remove source tcp port filter
+ * @param cdev
+ * @param src_port
+ * @add_dst_tcp_port_filter: Add destination tcp port filter
+ * @param cdev
+ * @param dest_port
+ * @remove_dst_tcp_port_filter: Remove destination tcp port filter
+ * @param cdev
+ * @param dest_port
+ * @clear_all_filters: Clear all filters.
+ * @param cdev
+ */
+struct qed_nvmetcp_ops {
+ const struct qed_common_ops *common;
+
+ const struct qed_ll2_ops *ll2;
+
+ int (*fill_dev_info)(struct qed_dev *cdev,
+ struct qed_dev_nvmetcp_info *info);
+
+ void (*register_ops)(struct qed_dev *cdev,
+ struct qed_nvmetcp_cb_ops *ops, void *cookie);
+
+ int (*start)(struct qed_dev *cdev,
+ struct qed_nvmetcp_tid *tasks,
+ void *event_context, nvmetcp_event_cb_t async_event_cb);
+
+ int (*stop)(struct qed_dev *cdev);
+
+ int (*acquire_conn)(struct qed_dev *cdev,
+ u32 *handle,
+ u32 *fw_cid, void __iomem **p_doorbell);
+
+ int (*release_conn)(struct qed_dev *cdev, u32 handle);
+
+ int (*offload_conn)(struct qed_dev *cdev,
+ u32 handle,
+ struct qed_nvmetcp_params_offload *conn_info);
+
+ int (*update_conn)(struct qed_dev *cdev,
+ u32 handle,
+ struct qed_nvmetcp_params_update *conn_info);
+
+ int (*destroy_conn)(struct qed_dev *cdev, u32 handle, u8 abrt_conn);
+
+ int (*clear_sq)(struct qed_dev *cdev, u32 handle);
+
+ int (*add_src_tcp_port_filter)(struct qed_dev *cdev, u16 src_port);
+
+ void (*remove_src_tcp_port_filter)(struct qed_dev *cdev, u16 src_port);
+
+ int (*add_dst_tcp_port_filter)(struct qed_dev *cdev, u16 dest_port);
+
+ void (*remove_dst_tcp_port_filter)(struct qed_dev *cdev, u16 dest_port);
+
+ void (*clear_all_filters)(struct qed_dev *cdev);
+
+ void (*init_read_io)(struct nvmetcp_task_params *task_params,
+ struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+ struct nvme_command *nvme_cmd,
+ struct storage_sgl_task_params *sgl_task_params);
+
+ void (*init_write_io)(struct nvmetcp_task_params *task_params,
+ struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+ struct nvme_command *nvme_cmd,
+ struct storage_sgl_task_params *sgl_task_params);
+
+ void (*init_icreq_exchange)(struct nvmetcp_task_params *task_params,
+ struct nvme_tcp_icreq_pdu *init_conn_req_pdu_hdr,
+ struct storage_sgl_task_params *tx_sgl_task_params,
+ struct storage_sgl_task_params *rx_sgl_task_params);
+
+ void (*init_task_cleanup)(struct nvmetcp_task_params *task_params);
+};
+
+const struct qed_nvmetcp_ops *qed_get_nvmetcp_ops(void);
+void qed_put_nvmetcp_ops(void);
+#endif
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) */
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+#ifndef _QED_IP_SERVICES_IF_H
+#define _QED_IP_SERVICES_IF_H
+
+#include <linux/types.h>
+#include <net/route.h>
+#include <net/ip6_route.h>
+#include <linux/inetdevice.h>
+
+int qed_route_ipv4(struct sockaddr_storage *local_addr,
+ struct sockaddr_storage *remote_addr,
+ struct sockaddr *hardware_address,
+ struct net_device **ndev);
+int qed_route_ipv6(struct sockaddr_storage *local_addr,
+ struct sockaddr_storage *remote_addr,
+ struct sockaddr *hardware_address,
+ struct net_device **ndev);
+void qed_vlan_get_ndev(struct net_device **ndev, u16 *vlan_id);
+struct pci_dev *qed_validate_ndev(struct net_device *ndev);
+void qed_return_tcp_port(struct socket *sock);
+int qed_fetch_tcp_port(struct sockaddr_storage local_ip_addr,
+ struct socket **sock, u16 *port);
+__be16 qed_get_in_port(struct sockaddr_storage *sa);
+
+#endif /* _QED_IP_SERVICES_IF_H */
u32 offset = raw_cpu_read(kstack_offset); \
u8 *ptr = __builtin_alloca(KSTACK_OFFSET_MAX(offset)); \
/* Keep allocation even after "ptr" loses scope. */ \
- asm volatile("" : "=o"(*ptr) :: "memory"); \
+ asm volatile("" :: "r"(ptr) : "memory"); \
} \
} while (0)
TTU_SPLIT_HUGE_PMD = 0x4, /* split huge PMD if any */
TTU_IGNORE_MLOCK = 0x8, /* ignore mlock */
+ TTU_SYNC = 0x10, /* avoid racy checks with PVMW_SYNC */
TTU_IGNORE_HWPOISON = 0x20, /* corrupted page is recoverable */
TTU_BATCH_FLUSH = 0x40, /* Batch TLB flushes where possible
* and caller guarantees they will
};
enum PDEV_STAT {PDEV_STAT_IDLE, PDEV_STAT_RUN};
+enum ASPM_MODE {ASPM_MODE_CFG, ASPM_MODE_REG};
#define ASPM_L1_1_EN BIT(0)
#define ASPM_L1_2_EN BIT(1)
u8 card_drive_sel;
#define ASPM_L1_EN 0x02
u8 aspm_en;
+ enum ASPM_MODE aspm_mode;
bool aspm_enabled;
#define PCR_MS_PMOS (1 << 0)
* Only for tasks we track a moving average of the past instantaneous
* estimated utilization. This allows to absorb sporadic drops in utilization
* of an otherwise almost periodic task.
+ *
+ * The UTIL_AVG_UNCHANGED flag is used to synchronize util_est with util_avg
+ * updates. When a task is dequeued, its util_est should not be updated if its
+ * util_avg has not been updated in the meantime.
+ * This information is mapped into the MSB bit of util_est.enqueued at dequeue
+ * time. Since max value of util_est.enqueued for a task is 1024 (PELT util_avg
+ * for a task) it is safe to use MSB.
*/
struct util_est {
unsigned int enqueued;
unsigned int ewma;
#define UTIL_EST_WEIGHT_SHIFT 2
+#define UTIL_AVG_UNCHANGED 0x80000000
} __attribute__((__aligned__(sizeof(u64))));
/*
int force_sig_bnderr(void __user *addr, void __user *lower, void __user *upper);
int force_sig_pkuerr(void __user *addr, u32 pkey);
+int force_sig_perf(void __user *addr, u32 type, u64 sig_data);
int force_sig_ptrace_errno_trap(int errno, void __user *addr);
SCTP_CID_I_FWD_TSN = 0xC2,
SCTP_CID_ASCONF_ACK = 0x80,
SCTP_CID_RECONF = 0x82,
+ SCTP_CID_PAD = 0x84,
}; /* enum */
};
+/* PAD chunk could be bundled with heartbeat chunk to probe pmtu */
+struct sctp_pad_chunk {
+ struct sctp_chunkhdr uh;
+};
+
+
/* For the abort and shutdown ACK we must carry the init tag in the
* common header. Just the common header is all that is needed with a
* chunk descriptor.
SIL_TIMER,
SIL_POLL,
SIL_FAULT,
+ SIL_FAULT_TRAPNO,
SIL_FAULT_MCEERR,
SIL_FAULT_BNDERR,
SIL_FAULT_PKUERR,
#include <linux/in6.h>
#include <linux/if_packet.h>
#include <net/flow.h>
+#include <net/page_pool.h>
#if IS_ENABLED(CONFIG_NF_CONNTRACK)
#include <linux/netfilter/nf_conntrack_common.h>
#endif
* @head_frag: skb was allocated from page fragments,
* not allocated by kmalloc() or vmalloc().
* @pfmemalloc: skbuff was allocated from PFMEMALLOC reserves
+ * @pp_recycle: mark the packet for recycling instead of freeing (implies
+ * page_pool support on driver)
* @active_extensions: active extensions (skb_ext_id types)
* @ndisc_nodetype: router type (from link layer)
* @ooo_okay: allow the mapping of a socket to a queue to be changed
fclone:2,
peeked:1,
head_frag:1,
- pfmemalloc:1;
+ pfmemalloc:1,
+ pp_recycle:1; /* page_pool recycle indicator */
#ifdef CONFIG_SKB_EXTENSIONS
__u8 active_extensions;
#endif
+
/* fields enclosed in headers_start/headers_end are copied
* using a single memcpy() in __copy_skb_header()
*/
/**
* __skb_frag_unref - release a reference on a paged fragment.
* @frag: the paged fragment
+ * @recycle: recycle the page if allocated via page_pool
*
- * Releases a reference on the paged fragment @frag.
+ * Releases a reference on the paged fragment @frag
+ * or recycles the page via the page_pool API.
*/
-static inline void __skb_frag_unref(skb_frag_t *frag)
+static inline void __skb_frag_unref(skb_frag_t *frag, bool recycle)
{
- put_page(skb_frag_page(frag));
+ struct page *page = skb_frag_page(frag);
+
+#ifdef CONFIG_PAGE_POOL
+ if (recycle && page_pool_return_skb_page(page))
+ return;
+#endif
+ put_page(page);
}
/**
*/
static inline void skb_frag_unref(struct sk_buff *skb, int f)
{
- __skb_frag_unref(&skb_shinfo(skb)->frags[f]);
+ __skb_frag_unref(&skb_shinfo(skb)->frags[f], skb->pp_recycle);
}
/**
#endif
}
+#ifdef CONFIG_PAGE_POOL
+static inline void skb_mark_for_recycle(struct sk_buff *skb, struct page *page,
+ struct page_pool *pp)
+{
+ skb->pp_recycle = 1;
+ page_pool_store_mem_info(page, pp);
+}
+#endif
+
+static inline bool skb_pp_recycle(struct sk_buff *skb, void *data)
+{
+ if (!IS_ENABLED(CONFIG_PAGE_POOL) || !skb->pp_recycle)
+ return false;
+ return page_pool_return_skb_page(virt_to_page(data));
+}
+
#endif /* __KERNEL__ */
#endif /* _LINUX_SKBUFF_H */
struct sk_msg *msg, u32 bytes);
int sk_msg_memcopy_from_iter(struct sock *sk, struct iov_iter *from,
struct sk_msg *msg, u32 bytes);
-int sk_msg_wait_data(struct sock *sk, struct sk_psock *psock, int flags,
- long timeo, int *err);
+int sk_msg_wait_data(struct sock *sk, struct sk_psock *psock, long timeo);
int sk_msg_recvmsg(struct sock *sk, struct sk_psock *psock, struct msghdr *msg,
int len, int flags);
void on_each_cpu_cond_mask(smp_cond_func_t cond_func, smp_call_func_t func,
void *info, bool wait, const struct cpumask *mask);
-int smp_call_function_single_async(int cpu, call_single_data_t *csd);
+int smp_call_function_single_async(int cpu, struct __call_single_data *csd);
/*
* Cpus stopping functions in panic. All have default weak definitions.
int __user *usockvec);
extern int __sys_shutdown_sock(struct socket *sock, int how);
extern int __sys_shutdown(int fd, int how);
-
-extern struct ns_common *get_net_ns(struct ns_common *ns);
#endif /* _LINUX_SOCKET_H */
int *cs_gpios;
struct gpio_desc **cs_gpiods;
bool use_gpio_descriptors;
- u8 unused_native_cs;
- u8 max_native_cs;
+ s8 unused_native_cs;
+ s8 max_native_cs;
/* statistics */
struct spi_statistics statistics;
enum stmmac_fpe_state lo_fpe_state; /* Local station FPE state */
};
+struct stmmac_safety_feature_cfg {
+ u32 tsoee;
+ u32 mrxpee;
+ u32 mestee;
+ u32 mrxee;
+ u32 mtxee;
+ u32 epsi;
+ u32 edpp;
+ u32 prtyen;
+ u32 tmouten;
+};
+
struct plat_stmmacenet_data {
int bus_id;
int phy_addr;
struct stmmac_dma_cfg *dma_cfg;
struct stmmac_est *est;
struct stmmac_fpe_cfg *fpe_cfg;
+ struct stmmac_safety_feature_cfg *safety_feat_cfg;
int clk_csr;
int has_gmac;
int enh_desc;
void (*fix_mac_speed)(void *priv, unsigned int speed);
int (*serdes_powerup)(struct net_device *ndev, void *priv);
void (*serdes_powerdown)(struct net_device *ndev, void *priv);
+ void (*speed_mode_2500)(struct net_device *ndev, void *priv);
void (*ptp_clk_freq_config)(void *priv);
int (*init)(struct platform_device *pdev, void *priv);
void (*exit)(struct platform_device *pdev, void *priv);
struct clk *clk_ptp_ref;
unsigned int clk_ptp_rate;
unsigned int clk_ref_rate;
+ unsigned int mult_fact_100ns;
s32 ptp_max_adj;
struct reset_control *stmmac_rst;
+ struct reset_control *stmmac_ahb_rst;
struct stmmac_axi *axi;
int has_gmac4;
bool has_sun8i;
unsigned int num_prealloc,
unsigned int max_req);
void xprt_free(struct rpc_xprt *);
+void xprt_add_backlog(struct rpc_xprt *xprt, struct rpc_task *task);
+bool xprt_wake_up_backlog(struct rpc_xprt *xprt, struct rpc_rqst *req);
static inline int
xprt_enable_swap(struct rpc_xprt *xprt)
| (((fun) != SSAM_ANY_FUN) ? SSAM_MATCH_FUNCTION : 0), \
.domain = d, \
.category = cat, \
- .target = ((tid) != SSAM_ANY_TID) ? (tid) : 0, \
- .instance = ((iid) != SSAM_ANY_IID) ? (iid) : 0, \
- .function = ((fun) != SSAM_ANY_FUN) ? (fun) : 0 \
+ .target = __builtin_choose_expr((tid) != SSAM_ANY_TID, (tid), 0), \
+ .instance = __builtin_choose_expr((iid) != SSAM_ANY_IID, (iid), 0), \
+ .function = __builtin_choose_expr((fun) != SSAM_ANY_FUN, (fun), 0)
/**
* SSAM_VDEV() - Initialize a &struct ssam_device_id as virtual device with
#define SWP_TYPE_SHIFT (BITS_PER_XA_VALUE - MAX_SWAPFILES_SHIFT)
#define SWP_OFFSET_MASK ((1UL << SWP_TYPE_SHIFT) - 1)
+/* Clear all flags but only keep swp_entry_t related information */
+static inline pte_t pte_swp_clear_flags(pte_t pte)
+{
+ if (pte_swp_soft_dirty(pte))
+ pte = pte_swp_clear_soft_dirty(pte);
+ if (pte_swp_uffd_wp(pte))
+ pte = pte_swp_clear_uffd_wp(pte);
+ return pte;
+}
+
/*
* Store a type+offset into a swp_entry_t in an arch-independent format
*/
{
swp_entry_t arch_entry;
- if (pte_swp_soft_dirty(pte))
- pte = pte_swp_clear_soft_dirty(pte);
- if (pte_swp_uffd_wp(pte))
- pte = pte_swp_clear_uffd_wp(pte);
+ pte = pte_swp_clear_flags(pte);
arch_entry = __pte_to_swp_entry(pte);
return swp_entry(__swp_type(arch_entry), __swp_offset(arch_entry));
}
#include <linux/context_tracking_state.h>
#include <linux/cpumask.h>
#include <linux/sched.h>
+#include <linux/rcupdate.h>
#ifdef CONFIG_GENERIC_CLOCKEVENTS
extern void __init tick_init(void);
__tick_nohz_task_switch();
}
+static inline void tick_nohz_user_enter_prepare(void)
+{
+ if (tick_nohz_full_cpu(smp_processor_id()))
+ rcu_nocb_flush_deferred_wakeup();
+}
+
#endif
#define PD_T_RECEIVER_RESPONSE 15 /* 15ms max */
#define PD_T_SOURCE_ACTIVITY 45
#define PD_T_SINK_ACTIVITY 135
-#define PD_T_SINK_WAIT_CAP 240
+#define PD_T_SINK_WAIT_CAP 310 /* 310 - 620 ms */
#define PD_T_PS_TRANSITION 500
#define PD_T_SRC_TRANSITION 35
#define PD_T_DRP_SNK 40
#define USB_PD_EXT_SDB_EVENT_OVP BIT(3)
#define USB_PD_EXT_SDB_EVENT_CF_CV_MODE BIT(4)
-#define USB_PD_EXT_SDB_PPS_EVENTS (USB_PD_EXT_SDB_EVENT_OCP | \
- USB_PD_EXT_SDB_EVENT_OTP | \
- USB_PD_EXT_SDB_EVENT_OVP)
-
#endif /* __LINUX_USB_PD_EXT_SDB_H */
u32 rx_bytes;
u32 buf_alloc;
struct list_head rx_queue;
+ u32 msg_count;
};
struct virtio_vsock_pkt {
struct msghdr *msg,
size_t len, int flags);
+int
+virtio_transport_seqpacket_enqueue(struct vsock_sock *vsk,
+ struct msghdr *msg,
+ size_t len);
+ssize_t
+virtio_transport_seqpacket_dequeue(struct vsock_sock *vsk,
+ struct msghdr *msg,
+ int flags);
s64 virtio_transport_stream_has_data(struct vsock_sock *vsk);
s64 virtio_transport_stream_has_space(struct vsock_sock *vsk);
+u32 virtio_transport_seqpacket_has_data(struct vsock_sock *vsk);
int virtio_transport_do_socket_init(struct vsock_sock *vsk,
struct vsock_sock *psk);
#define _LINUX_KERNEL_VTIME_H
#include <linux/context_tracking_state.h>
+#include <linux/sched.h>
+
#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE
#include <asm/vtime.h>
#endif
+/*
+ * Common vtime APIs
+ */
+#ifdef CONFIG_VIRT_CPU_ACCOUNTING
+extern void vtime_account_kernel(struct task_struct *tsk);
+extern void vtime_account_idle(struct task_struct *tsk);
+#endif /* !CONFIG_VIRT_CPU_ACCOUNTING */
-struct task_struct;
+#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN
+extern void arch_vtime_task_switch(struct task_struct *tsk);
+extern void vtime_user_enter(struct task_struct *tsk);
+extern void vtime_user_exit(struct task_struct *tsk);
+extern void vtime_guest_enter(struct task_struct *tsk);
+extern void vtime_guest_exit(struct task_struct *tsk);
+extern void vtime_init_idle(struct task_struct *tsk, int cpu);
+#else /* !CONFIG_VIRT_CPU_ACCOUNTING_GEN */
+static inline void vtime_user_enter(struct task_struct *tsk) { }
+static inline void vtime_user_exit(struct task_struct *tsk) { }
+static inline void vtime_guest_enter(struct task_struct *tsk) { }
+static inline void vtime_guest_exit(struct task_struct *tsk) { }
+static inline void vtime_init_idle(struct task_struct *tsk, int cpu) { }
+#endif
+
+#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE
+extern void vtime_account_irq(struct task_struct *tsk, unsigned int offset);
+extern void vtime_account_softirq(struct task_struct *tsk);
+extern void vtime_account_hardirq(struct task_struct *tsk);
+extern void vtime_flush(struct task_struct *tsk);
+#else /* !CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */
+static inline void vtime_account_irq(struct task_struct *tsk, unsigned int offset) { }
+static inline void vtime_account_softirq(struct task_struct *tsk) { }
+static inline void vtime_account_hardirq(struct task_struct *tsk) { }
+static inline void vtime_flush(struct task_struct *tsk) { }
+#endif
/*
* vtime_accounting_enabled_this_cpu() definitions/declarations
static inline bool vtime_accounting_enabled_this_cpu(void) { return true; }
extern void vtime_task_switch(struct task_struct *prev);
+static __always_inline void vtime_account_guest_enter(void)
+{
+ vtime_account_kernel(current);
+ current->flags |= PF_VCPU;
+}
+
+static __always_inline void vtime_account_guest_exit(void)
+{
+ vtime_account_kernel(current);
+ current->flags &= ~PF_VCPU;
+}
+
#elif defined(CONFIG_VIRT_CPU_ACCOUNTING_GEN)
/*
vtime_task_switch_generic(prev);
}
+static __always_inline void vtime_account_guest_enter(void)
+{
+ if (vtime_accounting_enabled_this_cpu())
+ vtime_guest_enter(current);
+ else
+ current->flags |= PF_VCPU;
+}
+
+static __always_inline void vtime_account_guest_exit(void)
+{
+ if (vtime_accounting_enabled_this_cpu())
+ vtime_guest_exit(current);
+ else
+ current->flags &= ~PF_VCPU;
+}
+
#else /* !CONFIG_VIRT_CPU_ACCOUNTING */
-static inline bool vtime_accounting_enabled_cpu(int cpu) {return false; }
static inline bool vtime_accounting_enabled_this_cpu(void) { return false; }
static inline void vtime_task_switch(struct task_struct *prev) { }
-#endif
-
-/*
- * Common vtime APIs
- */
-#ifdef CONFIG_VIRT_CPU_ACCOUNTING
-extern void vtime_account_kernel(struct task_struct *tsk);
-extern void vtime_account_idle(struct task_struct *tsk);
-#else /* !CONFIG_VIRT_CPU_ACCOUNTING */
-static inline void vtime_account_kernel(struct task_struct *tsk) { }
-#endif /* !CONFIG_VIRT_CPU_ACCOUNTING */
+static __always_inline void vtime_account_guest_enter(void)
+{
+ current->flags |= PF_VCPU;
+}
-#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN
-extern void arch_vtime_task_switch(struct task_struct *tsk);
-extern void vtime_user_enter(struct task_struct *tsk);
-extern void vtime_user_exit(struct task_struct *tsk);
-extern void vtime_guest_enter(struct task_struct *tsk);
-extern void vtime_guest_exit(struct task_struct *tsk);
-extern void vtime_init_idle(struct task_struct *tsk, int cpu);
-#else /* !CONFIG_VIRT_CPU_ACCOUNTING_GEN */
-static inline void vtime_user_enter(struct task_struct *tsk) { }
-static inline void vtime_user_exit(struct task_struct *tsk) { }
-static inline void vtime_guest_enter(struct task_struct *tsk) { }
-static inline void vtime_guest_exit(struct task_struct *tsk) { }
-static inline void vtime_init_idle(struct task_struct *tsk, int cpu) { }
-#endif
+static __always_inline void vtime_account_guest_exit(void)
+{
+ current->flags &= ~PF_VCPU;
+}
-#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE
-extern void vtime_account_irq(struct task_struct *tsk, unsigned int offset);
-extern void vtime_account_softirq(struct task_struct *tsk);
-extern void vtime_account_hardirq(struct task_struct *tsk);
-extern void vtime_flush(struct task_struct *tsk);
-#else /* !CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */
-static inline void vtime_account_irq(struct task_struct *tsk, unsigned int offset) { }
-static inline void vtime_account_softirq(struct task_struct *tsk) { }
-static inline void vtime_account_hardirq(struct task_struct *tsk) { }
-static inline void vtime_flush(struct task_struct *tsk) { }
#endif
#include <linux/device.h>
#include <linux/kernel.h>
+#include <linux/poll.h>
#include <linux/skbuff.h>
+#include <linux/netlink.h>
+#include <linux/netdevice.h>
/**
* enum wwan_port_type - WWAN port types
* @WWAN_PORT_QMI: Qcom modem/MSM interface for modem control
* @WWAN_PORT_QCDM: Qcom Modem diagnostic interface
* @WWAN_PORT_FIREHOSE: XML based command protocol
- * @WWAN_PORT_UNKNOWN: Unknown port type
- * @WWAN_PORT_MAX: Number of supported port types
+ *
+ * @WWAN_PORT_MAX: Highest supported port types
+ * @WWAN_PORT_UNKNOWN: Special value to indicate an unknown port type
+ * @__WWAN_PORT_MAX: Internal use
*/
enum wwan_port_type {
WWAN_PORT_AT,
WWAN_PORT_QMI,
WWAN_PORT_QCDM,
WWAN_PORT_FIREHOSE,
+
+ /* Add new port types above this line */
+
+ __WWAN_PORT_MAX,
+ WWAN_PORT_MAX = __WWAN_PORT_MAX - 1,
WWAN_PORT_UNKNOWN,
- WWAN_PORT_MAX = WWAN_PORT_UNKNOWN,
};
struct wwan_port;
/** struct wwan_port_ops - The WWAN port operations
* @start: The routine for starting the WWAN port device.
* @stop: The routine for stopping the WWAN port device.
- * @tx: The routine that sends WWAN port protocol data to the device.
+ * @tx: Non-blocking routine that sends WWAN port protocol data to the device.
+ * @tx_blocking: Optional blocking routine that sends WWAN port protocol data
+ * to the device.
+ * @tx_poll: Optional routine that sets additional TX poll flags.
*
* The wwan_port_ops structure contains a list of low-level operations
- * that control a WWAN port device. All functions are mandatory.
+ * that control a WWAN port device. All functions are mandatory unless specified.
*/
struct wwan_port_ops {
int (*start)(struct wwan_port *port);
void (*stop)(struct wwan_port *port);
int (*tx)(struct wwan_port *port, struct sk_buff *skb);
+
+ /* Optional operations */
+ int (*tx_blocking)(struct wwan_port *port, struct sk_buff *skb);
+ __poll_t (*tx_poll)(struct wwan_port *port, struct file *filp,
+ poll_table *wait);
};
/**
*/
void *wwan_port_get_drvdata(struct wwan_port *port);
+/**
+ * struct wwan_netdev_priv - WWAN core network device private data
+ * @link_id: WWAN device data link id
+ * @drv_priv: driver private data area, size is determined in &wwan_ops
+ */
+struct wwan_netdev_priv {
+ u32 link_id;
+
+ /* must be last */
+ u8 drv_priv[] __aligned(sizeof(void *));
+};
+
+static inline void *wwan_netdev_drvpriv(struct net_device *dev)
+{
+ return ((struct wwan_netdev_priv *)netdev_priv(dev))->drv_priv;
+}
+
+/*
+ * Used to indicate that the WWAN core should not create a default network
+ * link.
+ */
+#define WWAN_NO_DEFAULT_LINK U32_MAX
+
+/**
+ * struct wwan_ops - WWAN device ops
+ * @priv_size: size of private netdev data area
+ * @setup: set up a new netdev
+ * @newlink: register the new netdev
+ * @dellink: remove the given netdev
+ */
+struct wwan_ops {
+ unsigned int priv_size;
+ void (*setup)(struct net_device *dev);
+ int (*newlink)(void *ctxt, struct net_device *dev,
+ u32 if_id, struct netlink_ext_ack *extack);
+ void (*dellink)(void *ctxt, struct net_device *dev,
+ struct list_head *head);
+};
+
+int wwan_register_ops(struct device *parent, const struct wwan_ops *ops,
+ void *ctxt, u32 def_link_id);
+
+void wwan_unregister_ops(struct device *parent);
+
#endif /* __WWAN_H */
bool (*stream_is_active)(struct vsock_sock *);
bool (*stream_allow)(u32 cid, u32 port);
+ /* SEQ_PACKET. */
+ ssize_t (*seqpacket_dequeue)(struct vsock_sock *vsk, struct msghdr *msg,
+ int flags);
+ int (*seqpacket_enqueue)(struct vsock_sock *vsk, struct msghdr *msg,
+ size_t len);
+ bool (*seqpacket_allow)(u32 remote_cid);
+ u32 (*seqpacket_has_data)(struct vsock_sock *vsk);
+
/* Notification. */
int (*notify_poll_in)(struct vsock_sock *, size_t, bool *);
int (*notify_poll_out)(struct vsock_sock *, size_t, bool *);
char proc_file_name[IFNAMSIZ];
#endif /* CONFIG_PROC_FS */
struct list_head bond_list;
- u32 rr_tx_counter;
+ u32 __percpu *rr_tx_counter;
struct ad_bond_info ad_info;
struct alb_bond_info alb_info;
struct bond_params params;
* The link_support layer is used to add any Link Layer specific
* framing.
*/
-void caif_enroll_dev(struct net_device *dev, struct caif_dev_common *caifdev,
+int caif_enroll_dev(struct net_device *dev, struct caif_dev_common *caifdev,
struct cflayer *link_support, int head_room,
struct cflayer **layer, int (**rcv_func)(
struct sk_buff *, struct net_device *,
* @fcs: Specify if checksum is used in CAIF Framing Layer.
* @head_room: Head space needed by link specific protocol.
*/
-void
+int
cfcnfg_add_phy_layer(struct cfcnfg *cnfg,
struct net_device *dev, struct cflayer *phy_layer,
enum cfcnfg_phy_preference pref,
#include <net/caif/caif_layer.h>
struct cflayer *cfserl_create(int instance, bool use_stx);
+void cfserl_release(struct cflayer *layer);
#endif
* Copyright 2006-2010 Johannes Berg <johannes@sipsolutions.net>
* Copyright 2013-2014 Intel Mobile Communications GmbH
* Copyright 2015-2017 Intel Deutschland GmbH
- * Copyright (C) 2018-2020 Intel Corporation
+ * Copyright (C) 2018-2021 Intel Corporation
*/
#include <linux/ethtool.h>
#include <linux/if_ether.h>
#include <linux/ieee80211.h>
#include <linux/net.h>
+#include <linux/rfkill.h>
#include <net/regulatory.h>
/**
* @he_cap: holds the HE capabilities
* @he_6ghz_capa: HE 6 GHz capabilities, must be filled in for a
* 6 GHz band channel (and 0 may be valid value).
+ * @vendor_elems: vendor element(s) to advertise
+ * @vendor_elems.data: vendor element(s) data
+ * @vendor_elems.len: vendor element(s) length
*/
struct ieee80211_sband_iftype_data {
u16 types_mask;
struct ieee80211_sta_he_cap he_cap;
struct ieee80211_he_6ghz_capa he_6ghz_capa;
+ struct {
+ const u8 *data;
+ unsigned int len;
+ } vendor_elems;
};
/**
}
/**
- * ieee80211_get_he_sta_cap - return HE capabilities for an sband's STA
- * @sband: the sband to search for the STA on
- *
- * Return: pointer to the struct ieee80211_sta_he_cap, or NULL is none found
- */
-static inline const struct ieee80211_sta_he_cap *
-ieee80211_get_he_sta_cap(const struct ieee80211_supported_band *sband)
-{
- return ieee80211_get_he_iftype_cap(sband, NL80211_IFTYPE_STATION);
-}
-
-/**
* ieee80211_get_he_6ghz_capa - return HE 6 GHz capabilities
* @sband: the sband to search for the STA on
* @iftype: the iftype to search for
}
/**
+ * cfg80211_any_usable_channels - check for usable channels
+ * @wiphy: the wiphy to check for
+ * @band_mask: which bands to check on
+ * @prohibited_flags: which channels to not consider usable,
+ * %IEEE80211_CHAN_DISABLED is always taken into account
+ */
+bool cfg80211_any_usable_channels(struct wiphy *wiphy,
+ unsigned long band_mask,
+ u32 prohibited_flags);
+
+/**
* enum survey_info_flags - survey information flags
*
* @SURVEY_INFO_NOISE_DBM: noise (in dBm) was filled in
u8 count;
};
-#define CFG80211_MAX_NUM_DIFFERENT_CHANNELS 10
-
/**
* struct iface_combination_params - input parameters for interface combinations
*
* If neither @trigger_based nor @non_trigger_based is set,
* EDCA based ranging will be used.
* @lmr_feedback: negotiate for I2R LMR feedback. Only valid if either
- * @trigger_based or @non_trigger_based is set.
+ * @trigger_based or @non_trigger_based is set.
+ * @bss_color: the bss color of the responder. Optional. Set to zero to
+ * indicate the driver should set the BSS color. Only valid if
+ * @non_trigger_based or @trigger_based is set.
*
* See also nl80211 for the respective attribute documentation.
*/
u8 burst_duration;
u8 ftms_per_burst;
u8 ftmr_retries;
+ u8 bss_color;
};
/**
* configuration through the %NL80211_TID_CONFIG_ATTR_RETRY_SHORT and
* %NL80211_TID_CONFIG_ATTR_RETRY_LONG attributes
* @sar_capa: SAR control capabilities
+ * @rfkill: a pointer to the rfkill structure
*/
struct wiphy {
struct mutex mtx;
const struct cfg80211_sar_capa *sar_capa;
+ struct rfkill *rfkill;
+
char priv[] __aligned(NETDEV_ALIGN);
};
*/
int ieee80211_data_to_8023_exthdr(struct sk_buff *skb, struct ethhdr *ehdr,
const u8 *addr, enum nl80211_iftype iftype,
- u8 data_offset);
+ u8 data_offset, bool is_amsdu);
/**
* ieee80211_data_to_8023 - convert an 802.11 data frame to 802.3
static inline int ieee80211_data_to_8023(struct sk_buff *skb, const u8 *addr,
enum nl80211_iftype iftype)
{
- return ieee80211_data_to_8023_exthdr(skb, NULL, addr, iftype, 0);
+ return ieee80211_data_to_8023_exthdr(skb, NULL, addr, iftype, 0, false);
}
/**
* wiphy_rfkill_stop_polling - stop polling rfkill
* @wiphy: the wiphy
*/
-void wiphy_rfkill_stop_polling(struct wiphy *wiphy);
+static inline void wiphy_rfkill_stop_polling(struct wiphy *wiphy)
+{
+ rfkill_pause_polling(wiphy->rfkill);
+}
/**
* DOC: Vendor commands
dev_notice(&(wiphy)->dev, format, ##args)
#define wiphy_info(wiphy, format, args...) \
dev_info(&(wiphy)->dev, format, ##args)
+#define wiphy_info_once(wiphy, format, args...) \
+ dev_info_once(&(wiphy)->dev, format, ##args)
#define wiphy_err_ratelimited(wiphy, format, args...) \
dev_err_ratelimited(&(wiphy)->dev, format, ##args)
struct devlink {
struct list_head list;
struct list_head port_list;
+ struct list_head rate_list;
struct list_head sb_list;
struct list_head dpipe_table_list;
struct list_head resource_list;
};
};
+struct devlink_rate {
+ struct list_head list;
+ enum devlink_rate_type type;
+ struct devlink *devlink;
+ void *priv;
+ u64 tx_share;
+ u64 tx_max;
+
+ struct devlink_rate *parent;
+ union {
+ struct devlink_port *devlink_port;
+ struct {
+ char *name;
+ refcount_t refcnt;
+ };
+ };
+};
+
struct devlink_port {
struct list_head list;
struct list_head param_list;
struct delayed_work type_warn_dw;
struct list_head reporter_list;
struct mutex reporters_lock; /* Protects reporter_list */
+
+ struct devlink_rate *devlink_rate;
};
struct devlink_port_new_attrs {
enum devlink_trap_action action,
struct netlink_ext_ack *extack);
/**
+ * @trap_drop_counter_get: Trap drop counter get function.
+ *
+ * Should be used by device drivers to report number of packets
+ * that have been dropped, and cannot be passed to the devlink
+ * subsystem by the underlying device.
+ */
+ int (*trap_drop_counter_get)(struct devlink *devlink,
+ const struct devlink_trap *trap,
+ u64 *p_drops);
+ /**
* @trap_policer_init: Trap policer initialization function.
*
* Should be used by device drivers to initialize the trap policer in
struct devlink_port *port,
enum devlink_port_fn_state state,
struct netlink_ext_ack *extack);
+
+ /**
+ * Rate control callbacks.
+ */
+ int (*rate_leaf_tx_share_set)(struct devlink_rate *devlink_rate, void *priv,
+ u64 tx_share, struct netlink_ext_ack *extack);
+ int (*rate_leaf_tx_max_set)(struct devlink_rate *devlink_rate, void *priv,
+ u64 tx_max, struct netlink_ext_ack *extack);
+ int (*rate_node_tx_share_set)(struct devlink_rate *devlink_rate, void *priv,
+ u64 tx_share, struct netlink_ext_ack *extack);
+ int (*rate_node_tx_max_set)(struct devlink_rate *devlink_rate, void *priv,
+ u64 tx_max, struct netlink_ext_ack *extack);
+ int (*rate_node_new)(struct devlink_rate *rate_node, void **priv,
+ struct netlink_ext_ack *extack);
+ int (*rate_node_del)(struct devlink_rate *rate_node, void *priv,
+ struct netlink_ext_ack *extack);
+ int (*rate_leaf_parent_set)(struct devlink_rate *child,
+ struct devlink_rate *parent,
+ void *priv_child, void *priv_parent,
+ struct netlink_ext_ack *extack);
+ int (*rate_node_parent_set)(struct devlink_rate *child,
+ struct devlink_rate *parent,
+ void *priv_child, void *priv_parent,
+ struct netlink_ext_ack *extack);
};
static inline void *devlink_priv(struct devlink *devlink)
void devlink_port_attrs_pci_sf_set(struct devlink_port *devlink_port,
u32 controller, u16 pf, u32 sf,
bool external);
+int devlink_rate_leaf_create(struct devlink_port *port, void *priv);
+void devlink_rate_leaf_destroy(struct devlink_port *devlink_port);
+void devlink_rate_nodes_destroy(struct devlink *devlink);
int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
u32 size, u16 ingress_pools_count,
u16 egress_pools_count, u16 ingress_tc_count,
#define DSA_TAG_PROTO_OCELOT_8021Q_VALUE 20
#define DSA_TAG_PROTO_SEVILLE_VALUE 21
#define DSA_TAG_PROTO_BRCM_LEGACY_VALUE 22
+#define DSA_TAG_PROTO_SJA1110_VALUE 23
enum dsa_tag_protocol {
DSA_TAG_PROTO_NONE = DSA_TAG_PROTO_NONE_VALUE,
DSA_TAG_PROTO_XRS700X = DSA_TAG_PROTO_XRS700X_VALUE,
DSA_TAG_PROTO_OCELOT_8021Q = DSA_TAG_PROTO_OCELOT_8021Q_VALUE,
DSA_TAG_PROTO_SEVILLE = DSA_TAG_PROTO_SEVILLE_VALUE,
+ DSA_TAG_PROTO_SJA1110 = DSA_TAG_PROTO_SJA1110_VALUE,
};
struct packet_type;
* as regular on the master net device.
*/
bool (*filter)(const struct sk_buff *skb, struct net_device *dev);
- unsigned int overhead;
+ unsigned int needed_headroom;
+ unsigned int needed_tailroom;
const char *name;
enum dsa_tag_protocol proto;
/* Some tagging protocols either mangle or shift the destination MAC
* its RX filter.
*/
bool promisc_on_master;
- bool tail_tag;
};
/* This structure defines the control interfaces that are overlayed by the
return NULL;
}
+static inline bool dsa_port_is_dsa(struct dsa_port *port)
+{
+ return port->type == DSA_PORT_TYPE_DSA;
+}
+
+static inline bool dsa_port_is_cpu(struct dsa_port *port)
+{
+ return port->type == DSA_PORT_TYPE_CPU;
+}
+
+static inline bool dsa_port_is_user(struct dsa_port *dp)
+{
+ return dp->type == DSA_PORT_TYPE_USER;
+}
+
static inline bool dsa_is_unused_port(struct dsa_switch *ds, int p)
{
return dsa_to_port(ds, p)->type == DSA_PORT_TYPE_UNUSED;
{
#if IS_ENABLED(CONFIG_NET_DSA)
const struct dsa_device_ops *ops = skb->dev->dsa_ptr->tag_ops;
- int tag_len = ops->overhead;
+ int tag_len = ops->needed_headroom;
*offset = tag_len;
*proto = ((__be16 *)skb->data)[(tag_len / 2) - 1];
void fib_sync_mtu(struct net_device *dev, u32 orig_mtu);
void fib_nhc_update_mtu(struct fib_nh_common *nhc, u32 new, u32 orig);
+/* Fields used for sysctl_fib_multipath_hash_fields.
+ * Common to IPv4 and IPv6.
+ *
+ * Add new fields at the end. This is user API.
+ */
+#define FIB_MULTIPATH_HASH_FIELD_SRC_IP BIT(0)
+#define FIB_MULTIPATH_HASH_FIELD_DST_IP BIT(1)
+#define FIB_MULTIPATH_HASH_FIELD_IP_PROTO BIT(2)
+#define FIB_MULTIPATH_HASH_FIELD_FLOWLABEL BIT(3)
+#define FIB_MULTIPATH_HASH_FIELD_SRC_PORT BIT(4)
+#define FIB_MULTIPATH_HASH_FIELD_DST_PORT BIT(5)
+#define FIB_MULTIPATH_HASH_FIELD_INNER_SRC_IP BIT(6)
+#define FIB_MULTIPATH_HASH_FIELD_INNER_DST_IP BIT(7)
+#define FIB_MULTIPATH_HASH_FIELD_INNER_IP_PROTO BIT(8)
+#define FIB_MULTIPATH_HASH_FIELD_INNER_FLOWLABEL BIT(9)
+#define FIB_MULTIPATH_HASH_FIELD_INNER_SRC_PORT BIT(10)
+#define FIB_MULTIPATH_HASH_FIELD_INNER_DST_PORT BIT(11)
+
+#define FIB_MULTIPATH_HASH_FIELD_OUTER_MASK \
+ (FIB_MULTIPATH_HASH_FIELD_SRC_IP | \
+ FIB_MULTIPATH_HASH_FIELD_DST_IP | \
+ FIB_MULTIPATH_HASH_FIELD_IP_PROTO | \
+ FIB_MULTIPATH_HASH_FIELD_FLOWLABEL | \
+ FIB_MULTIPATH_HASH_FIELD_SRC_PORT | \
+ FIB_MULTIPATH_HASH_FIELD_DST_PORT)
+
+#define FIB_MULTIPATH_HASH_FIELD_INNER_MASK \
+ (FIB_MULTIPATH_HASH_FIELD_INNER_SRC_IP | \
+ FIB_MULTIPATH_HASH_FIELD_INNER_DST_IP | \
+ FIB_MULTIPATH_HASH_FIELD_INNER_IP_PROTO | \
+ FIB_MULTIPATH_HASH_FIELD_INNER_FLOWLABEL | \
+ FIB_MULTIPATH_HASH_FIELD_INNER_SRC_PORT | \
+ FIB_MULTIPATH_HASH_FIELD_INNER_DST_PORT)
+
+#define FIB_MULTIPATH_HASH_FIELD_ALL_MASK \
+ (FIB_MULTIPATH_HASH_FIELD_OUTER_MASK | \
+ FIB_MULTIPATH_HASH_FIELD_INNER_MASK)
+
+#define FIB_MULTIPATH_HASH_FIELD_DEFAULT_MASK \
+ (FIB_MULTIPATH_HASH_FIELD_SRC_IP | \
+ FIB_MULTIPATH_HASH_FIELD_DST_IP | \
+ FIB_MULTIPATH_HASH_FIELD_IP_PROTO)
+
#ifdef CONFIG_IP_ROUTE_MULTIPATH
int fib_multipath_hash(const struct net *net, const struct flowi4 *fl4,
const struct sk_buff *skb, struct flow_keys *flkeys);
{
return net->ipv6.sysctl.multipath_hash_policy;
}
+static inline u32 ip6_multipath_hash_fields(const struct net *net)
+{
+ return net->ipv6.sysctl.multipath_hash_fields;
+}
#else
static inline int ip6_multipath_hash_policy(const struct net *net)
{
return 0;
}
+static inline u32 ip6_multipath_hash_fields(const struct net *net)
+{
+ return 0;
+}
#endif
/*
* Copyright 2007-2010 Johannes Berg <johannes@sipsolutions.net>
* Copyright 2013-2014 Intel Mobile Communications GmbH
* Copyright (C) 2015 - 2017 Intel Deutschland GmbH
- * Copyright (C) 2018 - 2020 Intel Corporation
+ * Copyright (C) 2018 - 2021 Intel Corporation
*/
#ifndef MAC80211_H
* @twt_responder: does this BSS support TWT requester (relevant for managed
* mode only, set if the AP advertises TWT responder role)
* @twt_protected: does this BSS support protected TWT frames
+ * @twt_broadcast: does this BSS support broadcast TWT
* @assoc: association status
* @ibss_joined: indicates whether this station is part of an IBSS
* or not
bool twt_requester;
bool twt_responder;
bool twt_protected;
+ bool twt_broadcast;
/* association related data */
bool assoc, ibss_joined;
bool ibss_creator;
};
/**
+ * struct ieee80211_prep_tx_info - prepare TX information
+ * @duration: if non-zero, hint about the required duration,
+ * only used with the mgd_prepare_tx() method.
+ * @subtype: frame subtype (auth, (re)assoc, deauth, disassoc)
+ * @success: whether the frame exchange was successful, only
+ * used with the mgd_complete_tx() method, and then only
+ * valid for auth and (re)assoc.
+ */
+struct ieee80211_prep_tx_info {
+ u16 duration;
+ u16 subtype;
+ u8 success:1;
+};
+
+/**
* struct ieee80211_ops - callbacks from mac80211 to the driver
*
* This structure contains various callbacks that the driver may
* frame in case that no beacon was heard from the AP/P2P GO.
* The callback will be called before each transmission and upon return
* mac80211 will transmit the frame right away.
- * If duration is greater than zero, mac80211 hints to the driver the
- * duration for which the operation is requested.
+ * Additional information is passed in the &struct ieee80211_prep_tx_info
+ * data. If duration there is greater than zero, mac80211 hints to the
+ * driver the duration for which the operation is requested.
* The callback is optional and can (should!) sleep.
+ * @mgd_complete_tx: Notify the driver that the response frame for a previously
+ * transmitted frame announced with @mgd_prepare_tx was received, the data
+ * is filled similarly to @mgd_prepare_tx though the duration is not used.
*
* @mgd_protect_tdls_discover: Protect a TDLS discovery session. After sending
* a TDLS discovery-request, we expect a reply to arrive on the AP's
void (*mgd_prepare_tx)(struct ieee80211_hw *hw,
struct ieee80211_vif *vif,
- u16 duration);
+ struct ieee80211_prep_tx_info *info);
+ void (*mgd_complete_tx)(struct ieee80211_hw *hw,
+ struct ieee80211_vif *vif,
+ struct ieee80211_prep_tx_info *info);
void (*mgd_protect_tdls_discover)(struct ieee80211_hw *hw,
struct ieee80211_vif *vif);
*
* This function iterates over the interfaces associated with a given
* hardware that are currently active and calls the callback for them.
- * This version can only be used while holding the RTNL.
+ * This version can only be used while holding the wiphy mutex.
*
* @hw: the hardware struct of which the interfaces should be iterated over
* @iter_flags: iteration flags, see &enum ieee80211_interface_iteration_flags
* otherwise the NSS difference doesn't bother us.
*/
RATE_CTRL_CAPA_VHT_EXT_NSS_BW = BIT(0),
+ /**
+ * @RATE_CTRL_CAPA_AMPDU_TRIGGER:
+ * mac80211 should start A-MPDU sessions on tx
+ */
+ RATE_CTRL_CAPA_AMPDU_TRIGGER = BIT(1),
};
struct rate_control_ops {
/**
* ieee80211_parse_tx_radiotap - Sanity-check and parse the radiotap header
- * of injected frames
+ * of injected frames.
+ *
+ * To accurately parse and take into account rate and retransmission fields,
+ * you must initialize the chandef field in the ieee80211_tx_info structure
+ * of the skb before calling this function.
+ *
* @skb: packet injected by userspace
* @dev: the &struct device of this 802.11 device
*/
{
}
-void __ieee80211_schedule_txq(struct ieee80211_hw *hw,
- struct ieee80211_txq *txq, bool force);
-
/**
* ieee80211_schedule_txq - schedule a TXQ for transmission
*
* The driver may call this function if it has buffered packets for
* this TXQ internally.
*/
-static inline void
-ieee80211_schedule_txq(struct ieee80211_hw *hw, struct ieee80211_txq *txq)
-{
- __ieee80211_schedule_txq(hw, txq, true);
-}
+void ieee80211_schedule_txq(struct ieee80211_hw *hw, struct ieee80211_txq *txq);
/**
* ieee80211_return_txq - return a TXQ previously acquired by ieee80211_next_txq()
* The driver may set force=true if it has buffered packets for this TXQ
* internally.
*/
-static inline void
-ieee80211_return_txq(struct ieee80211_hw *hw, struct ieee80211_txq *txq,
- bool force)
-{
- __ieee80211_schedule_txq(hw, txq, force);
-}
+void ieee80211_return_txq(struct ieee80211_hw *hw, struct ieee80211_txq *txq,
+ bool force);
/**
* ieee80211_txq_may_transmit - check whether TXQ is allowed to transmit
struct sk_buff *
ieee80211_get_unsol_bcast_probe_resp_tmpl(struct ieee80211_hw *hw,
struct ieee80211_vif *vif);
+
+/**
+ * ieee80211_is_tx_data - check if frame is a data frame
+ *
+ * The function is used to check if a frame is a data frame. Frames with
+ * hardware encapsulation enabled are data frames.
+ *
+ * @skb: the frame to be transmitted.
+ */
+static inline bool ieee80211_is_tx_data(struct sk_buff *skb)
+{
+ struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
+ struct ieee80211_hdr *hdr = (void *) skb->data;
+
+ return info->flags & IEEE80211_TX_CTL_HW_80211_ENCAP ||
+ ieee80211_is_data(hdr->frame_control);
+}
+
#endif /* MAC80211_H */
u64 data_seq;
u32 subflow_seq;
u16 data_len;
+ __sum16 csum;
u8 use_map:1,
dsn64:1,
data_fin:1,
mpc_map:1,
frozen:1,
reset_transient:1;
- u8 reset_reason:4;
+ u8 reset_reason:4,
+ csum_reqd:1;
};
#define MPTCP_RM_IDS_MAX 8
struct mptcp_rm_list rm_list;
u8 join_id;
u8 backup;
- u8 reset_reason:4;
- u8 reset_transient:1;
+ u8 reset_reason:4,
+ reset_transient:1,
+ csum_reqd:1,
+ allow_join_id0:1;
u32 nonce;
u64 thmac;
u32 token;
#include <net/netns/mpls.h>
#include <net/netns/can.h>
#include <net/netns/xdp.h>
+#include <net/netns/smc.h>
#include <net/netns/bpf.h>
#include <linux/ns_common.h>
#include <linux/idr.h>
struct sock *crypto_nlsk;
#endif
struct sock *diag_nlsk;
+#if IS_ENABLED(CONFIG_SMC)
+ struct netns_smc smc;
+#endif
} __randomize_layout;
#include <linux/seq_file_net.h>
void net_ns_get_ownership(const struct net *net, kuid_t *uid, kgid_t *gid);
void net_ns_barrier(void);
+
+struct ns_common *get_net_ns(struct ns_common *ns);
+struct net *get_net_ns_by_fd(int fd);
#else /* CONFIG_NET_NS */
#include <linux/sched.h>
#include <linux/nsproxy.h>
}
static inline void net_ns_barrier(void) {}
+
+static inline struct ns_common *get_net_ns(struct ns_common *ns)
+{
+ return ERR_PTR(-EINVAL);
+}
+
+static inline struct net *get_net_ns_by_fd(int fd)
+{
+ return ERR_PTR(-EINVAL);
+}
#endif /* CONFIG_NET_NS */
extern struct list_head net_namespace_list;
struct net *get_net_ns_by_pid(pid_t pid);
-struct net *get_net_ns_by_fd(int fd);
#ifdef CONFIG_SYSCTL
void ipx_register_sysctl(void);
skb_set_nfct(skb, (unsigned long)ct | info);
}
+extern unsigned int nf_conntrack_net_id;
+
+static inline struct nf_conntrack_net *nf_ct_pernet(const struct net *net)
+{
+ return net_generic(net, nf_conntrack_net_id);
+}
+
#define NF_CT_STAT_INC(net, count) __this_cpu_inc((net)->ct.stat->count)
#define NF_CT_STAT_INC_ATOMIC(net, count) this_cpu_inc((net)->ct.stat->count)
#define NF_CT_STAT_ADD_ATOMIC(net, count, v) this_cpu_add((net)->ct.stat->count, (v))
extern const struct nla_policy nf_ct_port_nla_policy[];
#ifdef CONFIG_SYSCTL
-__printf(3, 4) __cold
+__printf(4, 5) __cold
void nf_ct_l4proto_log_invalid(const struct sk_buff *skb,
const struct nf_conn *ct,
+ const struct nf_hook_state *state,
const char *fmt, ...);
-__printf(5, 6) __cold
+__printf(4, 5) __cold
void nf_l4proto_log_invalid(const struct sk_buff *skb,
- struct net *net,
- u16 pf, u8 protonum,
+ const struct nf_hook_state *state,
+ u8 protonum,
const char *fmt, ...);
#else
-static inline __printf(5, 6) __cold
-void nf_l4proto_log_invalid(const struct sk_buff *skb, struct net *net,
- u16 pf, u8 protonum, const char *fmt, ...) {}
-static inline __printf(3, 4) __cold
+static inline __printf(4, 5) __cold
+void nf_l4proto_log_invalid(const struct sk_buff *skb,
+ const struct nf_hook_state *state,
+ u8 protonum,
+ const char *fmt, ...) {}
+static inline __printf(4, 5) __cold
void nf_ct_l4proto_log_invalid(const struct sk_buff *skb,
const struct nf_conn *ct,
+ const struct nf_hook_state *state,
const char *fmt, ...) { }
#endif /* CONFIG_SYSCTL */
NF_FLOW_HW,
NF_FLOW_HW_DYING,
NF_FLOW_HW_DEAD,
- NF_FLOW_HW_REFRESH,
NF_FLOW_HW_PENDING,
};
#define NF_FLOW_TIMEOUT (30 * HZ)
#define nf_flowtable_time_stamp (u32)jiffies
+unsigned long flow_offload_get_timeout(struct flow_offload *flow);
+
static inline __s32 nf_flow_timeout_delta(unsigned int timeout)
{
return (__s32)(timeout - nf_flowtable_time_stamp);
struct nft_pktinfo {
struct sk_buff *skb;
+ const struct nf_hook_state *state;
bool tprot_set;
u8 tprot;
- /* for x_tables compatibility */
- struct xt_action_param xt;
+ u16 fragoff;
+ unsigned int thoff;
};
+static inline struct sock *nft_sk(const struct nft_pktinfo *pkt)
+{
+ return pkt->state->sk;
+}
+
+static inline unsigned int nft_thoff(const struct nft_pktinfo *pkt)
+{
+ return pkt->thoff;
+}
+
static inline struct net *nft_net(const struct nft_pktinfo *pkt)
{
- return pkt->xt.state->net;
+ return pkt->state->net;
}
static inline unsigned int nft_hook(const struct nft_pktinfo *pkt)
{
- return pkt->xt.state->hook;
+ return pkt->state->hook;
}
static inline u8 nft_pf(const struct nft_pktinfo *pkt)
{
- return pkt->xt.state->pf;
+ return pkt->state->pf;
}
static inline const struct net_device *nft_in(const struct nft_pktinfo *pkt)
{
- return pkt->xt.state->in;
+ return pkt->state->in;
}
static inline const struct net_device *nft_out(const struct nft_pktinfo *pkt)
{
- return pkt->xt.state->out;
+ return pkt->state->out;
}
static inline void nft_set_pktinfo(struct nft_pktinfo *pkt,
const struct nf_hook_state *state)
{
pkt->skb = skb;
- pkt->xt.state = state;
+ pkt->state = state;
}
-static inline void nft_set_pktinfo_unspec(struct nft_pktinfo *pkt,
- struct sk_buff *skb)
+static inline void nft_set_pktinfo_unspec(struct nft_pktinfo *pkt)
{
pkt->tprot_set = false;
pkt->tprot = 0;
- pkt->xt.thoff = 0;
- pkt->xt.fragoff = 0;
+ pkt->thoff = 0;
+ pkt->fragoff = 0;
}
/**
struct nft_trans_table {
bool update;
- u8 state;
- u32 flags;
};
#define nft_trans_table_update(trans) \
(((struct nft_trans_table *)trans->data)->update)
-#define nft_trans_table_state(trans) \
- (((struct nft_trans_table *)trans->data)->state)
-#define nft_trans_table_flags(trans) \
- (((struct nft_trans_table *)trans->data)->flags)
struct nft_trans_elem {
struct nft_set *set;
#define _NET_NF_TABLES_CORE_H
#include <net/netfilter/nf_tables.h>
+#include <linux/indirect_call_wrapper.h>
extern struct nft_expr_type nft_imm_type;
extern struct nft_expr_type nft_cmp_type;
extern struct nft_expr_type nft_meta_type;
extern struct nft_expr_type nft_rt_type;
extern struct nft_expr_type nft_exthdr_type;
+extern struct nft_expr_type nft_last_type;
#ifdef CONFIG_NETWORK_SECMARK
extern struct nft_object_type nft_secmark_obj_type;
extern const struct nft_set_type nft_set_pipapo_type;
extern const struct nft_set_type nft_set_pipapo_avx2_type;
+#ifdef CONFIG_RETPOLINE
+bool nft_rhash_lookup(const struct net *net, const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext);
+bool nft_rbtree_lookup(const struct net *net, const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext);
+bool nft_bitmap_lookup(const struct net *net, const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext);
+bool nft_hash_lookup_fast(const struct net *net,
+ const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext);
+bool nft_hash_lookup(const struct net *net, const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext);
+bool nft_set_do_lookup(const struct net *net, const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext);
+#else
+static inline bool
+nft_set_do_lookup(const struct net *net, const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext)
+{
+ return set->ops->lookup(net, set, key, ext);
+}
+#endif
+
+/* called from nft_pipapo_avx2.c */
+bool nft_pipapo_lookup(const struct net *net, const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext);
+/* called from nft_set_pipapo.c */
+bool nft_pipapo_avx2_lookup(const struct net *net, const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext);
+
struct nft_expr;
struct nft_regs;
struct nft_pktinfo;
#include <net/netfilter/nf_tables.h>
#include <net/ip.h>
-static inline void nft_set_pktinfo_ipv4(struct nft_pktinfo *pkt,
- struct sk_buff *skb)
+static inline void nft_set_pktinfo_ipv4(struct nft_pktinfo *pkt)
{
struct iphdr *ip;
ip = ip_hdr(pkt->skb);
pkt->tprot_set = true;
pkt->tprot = ip->protocol;
- pkt->xt.thoff = ip_hdrlen(pkt->skb);
- pkt->xt.fragoff = ntohs(ip->frag_off) & IP_OFFSET;
+ pkt->thoff = ip_hdrlen(pkt->skb);
+ pkt->fragoff = ntohs(ip->frag_off) & IP_OFFSET;
}
-static inline int __nft_set_pktinfo_ipv4_validate(struct nft_pktinfo *pkt,
- struct sk_buff *skb)
+static inline int __nft_set_pktinfo_ipv4_validate(struct nft_pktinfo *pkt)
{
struct iphdr *iph, _iph;
u32 len, thoff;
- iph = skb_header_pointer(skb, skb_network_offset(skb), sizeof(*iph),
- &_iph);
+ iph = skb_header_pointer(pkt->skb, skb_network_offset(pkt->skb),
+ sizeof(*iph), &_iph);
if (!iph)
return -1;
len = ntohs(iph->tot_len);
thoff = iph->ihl * 4;
- if (skb->len < len)
+ if (pkt->skb->len < len)
return -1;
else if (len < thoff)
return -1;
pkt->tprot_set = true;
pkt->tprot = iph->protocol;
- pkt->xt.thoff = thoff;
- pkt->xt.fragoff = ntohs(iph->frag_off) & IP_OFFSET;
+ pkt->thoff = thoff;
+ pkt->fragoff = ntohs(iph->frag_off) & IP_OFFSET;
return 0;
}
-static inline void nft_set_pktinfo_ipv4_validate(struct nft_pktinfo *pkt,
- struct sk_buff *skb)
+static inline void nft_set_pktinfo_ipv4_validate(struct nft_pktinfo *pkt)
{
- if (__nft_set_pktinfo_ipv4_validate(pkt, skb) < 0)
- nft_set_pktinfo_unspec(pkt, skb);
+ if (__nft_set_pktinfo_ipv4_validate(pkt) < 0)
+ nft_set_pktinfo_unspec(pkt);
}
-static inline int nft_set_pktinfo_ipv4_ingress(struct nft_pktinfo *pkt,
- struct sk_buff *skb)
+static inline int nft_set_pktinfo_ipv4_ingress(struct nft_pktinfo *pkt)
{
struct iphdr *iph;
u32 len, thoff;
- if (!pskb_may_pull(skb, sizeof(*iph)))
+ if (!pskb_may_pull(pkt->skb, sizeof(*iph)))
return -1;
- iph = ip_hdr(skb);
+ iph = ip_hdr(pkt->skb);
if (iph->ihl < 5 || iph->version != 4)
goto inhdr_error;
len = ntohs(iph->tot_len);
thoff = iph->ihl * 4;
- if (skb->len < len) {
+ if (pkt->skb->len < len) {
__IP_INC_STATS(nft_net(pkt), IPSTATS_MIB_INTRUNCATEDPKTS);
return -1;
} else if (len < thoff) {
pkt->tprot_set = true;
pkt->tprot = iph->protocol;
- pkt->xt.thoff = thoff;
- pkt->xt.fragoff = ntohs(iph->frag_off) & IP_OFFSET;
+ pkt->thoff = thoff;
+ pkt->fragoff = ntohs(iph->frag_off) & IP_OFFSET;
return 0;
#include <net/ipv6.h>
#include <net/netfilter/nf_tables.h>
-static inline void nft_set_pktinfo_ipv6(struct nft_pktinfo *pkt,
- struct sk_buff *skb)
+static inline void nft_set_pktinfo_ipv6(struct nft_pktinfo *pkt)
{
unsigned int flags = IP6_FH_F_AUTH;
int protohdr, thoff = 0;
protohdr = ipv6_find_hdr(pkt->skb, &thoff, -1, &frag_off, &flags);
if (protohdr < 0) {
- nft_set_pktinfo_unspec(pkt, skb);
+ nft_set_pktinfo_unspec(pkt);
return;
}
pkt->tprot_set = true;
pkt->tprot = protohdr;
- pkt->xt.thoff = thoff;
- pkt->xt.fragoff = frag_off;
+ pkt->thoff = thoff;
+ pkt->fragoff = frag_off;
}
-static inline int __nft_set_pktinfo_ipv6_validate(struct nft_pktinfo *pkt,
- struct sk_buff *skb)
+static inline int __nft_set_pktinfo_ipv6_validate(struct nft_pktinfo *pkt)
{
#if IS_ENABLED(CONFIG_IPV6)
unsigned int flags = IP6_FH_F_AUTH;
int protohdr;
u32 pkt_len;
- ip6h = skb_header_pointer(skb, skb_network_offset(skb), sizeof(*ip6h),
- &_ip6h);
+ ip6h = skb_header_pointer(pkt->skb, skb_network_offset(pkt->skb),
+ sizeof(*ip6h), &_ip6h);
if (!ip6h)
return -1;
return -1;
pkt_len = ntohs(ip6h->payload_len);
- if (pkt_len + sizeof(*ip6h) > skb->len)
+ if (pkt_len + sizeof(*ip6h) > pkt->skb->len)
return -1;
protohdr = ipv6_find_hdr(pkt->skb, &thoff, -1, &frag_off, &flags);
pkt->tprot_set = true;
pkt->tprot = protohdr;
- pkt->xt.thoff = thoff;
- pkt->xt.fragoff = frag_off;
+ pkt->thoff = thoff;
+ pkt->fragoff = frag_off;
return 0;
#else
#endif
}
-static inline void nft_set_pktinfo_ipv6_validate(struct nft_pktinfo *pkt,
- struct sk_buff *skb)
+static inline void nft_set_pktinfo_ipv6_validate(struct nft_pktinfo *pkt)
{
- if (__nft_set_pktinfo_ipv6_validate(pkt, skb) < 0)
- nft_set_pktinfo_unspec(pkt, skb);
+ if (__nft_set_pktinfo_ipv6_validate(pkt) < 0)
+ nft_set_pktinfo_unspec(pkt);
}
-static inline int nft_set_pktinfo_ipv6_ingress(struct nft_pktinfo *pkt,
- struct sk_buff *skb)
+static inline int nft_set_pktinfo_ipv6_ingress(struct nft_pktinfo *pkt)
{
#if IS_ENABLED(CONFIG_IPV6)
unsigned int flags = IP6_FH_F_AUTH;
int protohdr;
u32 pkt_len;
- if (!pskb_may_pull(skb, sizeof(*ip6h)))
+ if (!pskb_may_pull(pkt->skb, sizeof(*ip6h)))
return -1;
- ip6h = ipv6_hdr(skb);
+ ip6h = ipv6_hdr(pkt->skb);
if (ip6h->version != 6)
goto inhdr_error;
pkt_len = ntohs(ip6h->payload_len);
- if (pkt_len + sizeof(*ip6h) > skb->len) {
+ if (pkt_len + sizeof(*ip6h) > pkt->skb->len) {
idev = __in6_dev_get(nft_in(pkt));
__IP6_INC_STATS(nft_net(pkt), idev, IPSTATS_MIB_INTRUNCATEDPKTS);
return -1;
pkt->tprot_set = true;
pkt->tprot = protohdr;
- pkt->xt.thoff = thoff;
- pkt->xt.fragoff = frag_off;
+ pkt->thoff = thoff;
+ pkt->fragoff = frag_off;
return 0;
u8 tcp_loose;
u8 tcp_be_liberal;
u8 tcp_max_retrans;
+#if IS_ENABLED(CONFIG_NF_FLOW_TABLE)
+ unsigned int offload_timeout;
+ unsigned int offload_pickup;
+#endif
};
enum udp_conntrack {
struct nf_udp_net {
unsigned int timeouts[UDP_CT_MAX];
+#if IS_ENABLED(CONFIG_NF_FLOW_TABLE)
+ unsigned int offload_timeout;
+ unsigned int offload_pickup;
+#endif
};
struct nf_icmp_net {
u8 sysctl_tcp_syn_retries;
u8 sysctl_tcp_synack_retries;
u8 sysctl_tcp_syncookies;
+ u8 sysctl_tcp_migrate_req;
int sysctl_tcp_reordering;
u8 sysctl_tcp_retries1;
u8 sysctl_tcp_retries2;
#endif
#endif
#ifdef CONFIG_IP_ROUTE_MULTIPATH
+ u32 sysctl_fib_multipath_hash_fields;
u8 sysctl_fib_multipath_use_neigh;
u8 sysctl_fib_multipath_hash_policy;
#endif
int ip6_rt_gc_elasticity;
int ip6_rt_mtu_expires;
int ip6_rt_min_advmss;
- u8 bindv6only;
+ u32 multipath_hash_fields;
u8 multipath_hash_policy;
+ u8 bindv6only;
u8 flowlabel_consistency;
u8 auto_flowlabels;
int icmpv6_time;
/* HB.interval - 30 seconds */
unsigned int hb_interval;
+ /* The interval for PLPMTUD probe timer */
+ unsigned int probe_interval;
+
/* Association.Max.Retrans - 10 attempts
* Path.Max.Retrans - 5 attempts (per destination address)
* Max.Init.Retransmits - 8 attempts
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __NETNS_SMC_H__
+#define __NETNS_SMC_H__
+#include <linux/mutex.h>
+#include <linux/percpu.h>
+
+struct smc_stats_rsn;
+struct smc_stats;
+struct netns_smc {
+ /* per cpu counters for SMC */
+ struct smc_stats __percpu *smc_stats;
+ /* protect fback_rsn */
+ struct mutex mutex_fback_rsn;
+ struct smc_stats_rsn *fback_rsn;
+};
+#endif
struct sk_buff **resp);
struct nci_hci_dev *nci_hci_allocate(struct nci_dev *ndev);
+void nci_hci_deallocate(struct nci_dev *ndev);
int nci_hci_send_event(struct nci_dev *ndev, u8 gate, u8 event,
const u8 *param, size_t param_len);
int nci_hci_send_cmd(struct nci_dev *ndev, u8 gate,
return pool->p.dma_dir;
}
+bool page_pool_return_skb_page(struct page *page);
+
struct page_pool *page_pool_create(const struct page_pool_params *params);
#ifdef CONFIG_PAGE_POOL
static inline dma_addr_t page_pool_get_dma_addr(struct page *page)
{
- return page->dma_addr;
+ dma_addr_t ret = page->dma_addr[0];
+ if (sizeof(dma_addr_t) > sizeof(unsigned long))
+ ret |= (dma_addr_t)page->dma_addr[1] << 16 << 16;
+ return ret;
+}
+
+static inline void page_pool_set_dma_addr(struct page *page, dma_addr_t addr)
+{
+ page->dma_addr[0] = addr;
+ if (sizeof(dma_addr_t) > sizeof(unsigned long))
+ page->dma_addr[1] = upper_32_bits(addr);
}
static inline bool is_page_pool_compiled_in(void)
spin_unlock_bh(&pool->ring.producer_lock);
}
+/* Store mem_info on struct page and use it while recycling skb frags */
+static inline
+void page_pool_store_mem_info(struct page *page, struct page_pool *pp)
+{
+ page->pp = pp;
+}
+
#endif /* _NET_PAGE_POOL_H */
cls_common->extack = extack;
}
+#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
+static inline struct tc_skb_ext *tc_skb_ext_alloc(struct sk_buff *skb)
+{
+ struct tc_skb_ext *tc_skb_ext = skb_ext_add(skb, TC_SKB_EXT);
+
+ if (tc_skb_ext)
+ memset(tc_skb_ext, 0, sizeof(*tc_skb_ext));
+ return tc_skb_ext;
+}
+#endif
+
enum tc_matchall_command {
TC_CLSMATCHALL_REPLACE,
TC_CLSMATCHALL_DESTROY,
static inline void qdisc_run(struct Qdisc *q)
{
if (qdisc_run_begin(q)) {
- /* NOLOCK qdisc must check 'state' under the qdisc seqlock
- * to avoid racing with dev_qdisc_reset()
- */
- if (!(q->flags & TCQ_F_NOLOCK) ||
- likely(!test_bit(__QDISC_STATE_DEACTIVATED, &q->state)))
- __qdisc_run(q);
+ __qdisc_run(q);
qdisc_run_end(q);
}
}
int (*err_handler)(struct sk_buff *skb, u32 info);
unsigned int no_policy:1,
- netns_ok:1,
/* does the protocol do more stringent
* icmp tag validation than simple
* socket lookup?
* @maxtype: Highest device specific netlink attribute number
* @policy: Netlink policy for device specific attribute validation
* @validate: Optional validation function for netlink/changelink parameters
+ * @alloc: netdev allocation function, can be %NULL and is then used
+ * in place of alloc_netdev_mqs(), in this case @priv_size
+ * and @setup are unused. Returns a netdev or ERR_PTR().
* @priv_size: sizeof net_device private space
* @setup: net_device setup function
* @newlink: Function for configuring and registering a new device
const char *kind;
size_t priv_size;
+ struct net_device *(*alloc)(struct nlattr *tb[],
+ const char *ifname,
+ unsigned char name_assign_type,
+ unsigned int num_tx_queues,
+ unsigned int num_rx_queues);
void (*setup)(struct net_device *dev);
bool netns_refund;
enum qdisc_state_t {
__QDISC_STATE_SCHED,
__QDISC_STATE_DEACTIVATED,
+ __QDISC_STATE_MISSED,
+ __QDISC_STATE_DRAINING,
};
+#define QDISC_STATE_MISSED BIT(__QDISC_STATE_MISSED)
+#define QDISC_STATE_DRAINING BIT(__QDISC_STATE_DRAINING)
+
+#define QDISC_STATE_NON_EMPTY (QDISC_STATE_MISSED | \
+ QDISC_STATE_DRAINING)
+
struct qdisc_size_table {
struct rcu_head rcu;
struct list_head list;
spinlock_t busylock ____cacheline_aligned_in_smp;
spinlock_t seqlock;
- /* for NOLOCK qdisc, true if there are no enqueued skbs */
- bool empty;
struct rcu_head rcu;
/* private data */
return (raw_read_seqcount(&qdisc->running) & 1) ? true : false;
}
+static inline bool nolock_qdisc_is_empty(const struct Qdisc *qdisc)
+{
+ return !(READ_ONCE(qdisc->state) & QDISC_STATE_NON_EMPTY);
+}
+
static inline bool qdisc_is_percpu_stats(const struct Qdisc *q)
{
return q->flags & TCQ_F_CPUSTATS;
static inline bool qdisc_is_empty(const struct Qdisc *qdisc)
{
if (qdisc_is_percpu_stats(qdisc))
- return READ_ONCE(qdisc->empty);
+ return nolock_qdisc_is_empty(qdisc);
return !READ_ONCE(qdisc->q.qlen);
}
static inline bool qdisc_run_begin(struct Qdisc *qdisc)
{
if (qdisc->flags & TCQ_F_NOLOCK) {
- if (!spin_trylock(&qdisc->seqlock))
+ if (spin_trylock(&qdisc->seqlock))
+ return true;
+
+ /* If the MISSED flag is set, it means other thread has
+ * set the MISSED flag before second spin_trylock(), so
+ * we can return false here to avoid multi cpus doing
+ * the set_bit() and second spin_trylock() concurrently.
+ */
+ if (test_bit(__QDISC_STATE_MISSED, &qdisc->state))
return false;
- WRITE_ONCE(qdisc->empty, false);
+
+ /* Set the MISSED flag before the second spin_trylock(),
+ * if the second spin_trylock() return false, it means
+ * other cpu holding the lock will do dequeuing for us
+ * or it will see the MISSED flag set after releasing
+ * lock and reschedule the net_tx_action() to do the
+ * dequeuing.
+ */
+ set_bit(__QDISC_STATE_MISSED, &qdisc->state);
+
+ /* Retry again in case other CPU may not see the new flag
+ * after it releases the lock at the end of qdisc_run_end().
+ */
+ return spin_trylock(&qdisc->seqlock);
} else if (qdisc_is_running(qdisc)) {
return false;
}
static inline void qdisc_run_end(struct Qdisc *qdisc)
{
- write_seqcount_end(&qdisc->running);
- if (qdisc->flags & TCQ_F_NOLOCK)
+ if (qdisc->flags & TCQ_F_NOLOCK) {
spin_unlock(&qdisc->seqlock);
+
+ if (unlikely(test_bit(__QDISC_STATE_MISSED,
+ &qdisc->state)))
+ __netif_schedule(qdisc);
+ } else {
+ write_seqcount_end(&qdisc->running);
+ }
}
static inline bool qdisc_may_bulk(const struct Qdisc *qdisc)
SCTP_CMD_HB_TIMERS_START, /* Start the heartbeat timers. */
SCTP_CMD_HB_TIMER_UPDATE, /* Update a heartbeat timers. */
SCTP_CMD_HB_TIMERS_STOP, /* Stop the heartbeat timers. */
+ SCTP_CMD_PROBE_TIMER_UPDATE, /* Update a probe timer. */
SCTP_CMD_TRANSPORT_HB_SENT, /* Reset the status of a transport. */
SCTP_CMD_TRANSPORT_IDLE, /* Do manipulations on idle transport */
SCTP_CMD_TRANSPORT_ON, /* Mark the transport as active. */
SCTP_EVENT_TIMEOUT_T5_SHUTDOWN_GUARD,
SCTP_EVENT_TIMEOUT_HEARTBEAT,
SCTP_EVENT_TIMEOUT_RECONF,
+ SCTP_EVENT_TIMEOUT_PROBE,
SCTP_EVENT_TIMEOUT_SACK,
SCTP_EVENT_TIMEOUT_AUTOCLOSE,
};
SCTP_SS_CLOSING = TCP_CLOSE_WAIT,
};
+enum sctp_plpmtud_state {
+ SCTP_PL_DISABLED,
+ SCTP_PL_BASE,
+ SCTP_PL_SEARCH,
+ SCTP_PL_COMPLETE,
+ SCTP_PL_ERROR,
+};
+
+#define SCTP_BASE_PLPMTU 1200
+#define SCTP_MAX_PLPMTU 9000
+#define SCTP_MIN_PLPMTU 512
+
+#define SCTP_MAX_PROBES 3
+
+#define SCTP_PL_BIG_STEP 32
+#define SCTP_PL_MIN_STEP 4
+
/* These functions map various type to printable names. */
const char *sctp_cname(const union sctp_subtype id); /* chunk types */
const char *sctp_oname(const union sctp_subtype id); /* other events */
*/
#define SCTP_AUTH_RANDOM_LENGTH 32
+#define SCTP_PROBE_TIMER_MIN 5000
+
#endif /* __sctp_constants_h__ */
struct sctphdr *, struct sctp_association **,
struct sctp_transport **);
void sctp_err_finish(struct sock *, struct sctp_transport *);
+int sctp_udp_v4_err(struct sock *sk, struct sk_buff *skb);
+int sctp_udp_v6_err(struct sock *sk, struct sk_buff *skb);
void sctp_icmp_frag_needed(struct sock *, struct sctp_association *,
struct sctp_transport *t, __u32 pmtu);
void sctp_icmp_redirect(struct sock *, struct sctp_transport *,
/* Calculate max payload size given a MTU, or the total overhead if
* given MTU is zero
*/
-static inline __u32 sctp_mtu_payload(const struct sctp_sock *sp,
- __u32 mtu, __u32 extra)
+static inline __u32 __sctp_mtu_payload(const struct sctp_sock *sp,
+ const struct sctp_transport *t,
+ __u32 mtu, __u32 extra)
{
__u32 overhead = sizeof(struct sctphdr) + extra;
if (sp) {
overhead += sp->pf->af->net_header_len;
- if (sp->udp_port)
+ if (sp->udp_port && (!t || t->encap_port))
overhead += sizeof(struct udphdr);
} else {
overhead += sizeof(struct ipv6hdr);
return mtu ? mtu - overhead : overhead;
}
+static inline __u32 sctp_mtu_payload(const struct sctp_sock *sp,
+ __u32 mtu, __u32 extra)
+{
+ return __sctp_mtu_payload(sp, NULL, mtu, extra);
+}
+
static inline __u32 sctp_dst_mtu(const struct dst_entry *dst)
{
return SCTP_TRUNC4(max_t(__u32, dst_mtu(dst),
return sctp_mtu_payload(sp, SCTP_DEFAULT_MINSEGMENT, datasize);
}
+static inline int sctp_transport_pl_hlen(struct sctp_transport *t)
+{
+ return __sctp_mtu_payload(sctp_sk(t->asoc->base.sk), t, 0, 0);
+}
+
+static inline void sctp_transport_pl_reset(struct sctp_transport *t)
+{
+ if (t->probe_interval && (t->param_flags & SPP_PMTUD_ENABLE) &&
+ (t->state == SCTP_ACTIVE || t->state == SCTP_UNKNOWN)) {
+ if (t->pl.state == SCTP_PL_DISABLED) {
+ t->pl.state = SCTP_PL_BASE;
+ t->pl.pmtu = SCTP_BASE_PLPMTU;
+ t->pl.probe_size = SCTP_BASE_PLPMTU;
+ sctp_transport_reset_probe_timer(t);
+ }
+ } else {
+ if (t->pl.state != SCTP_PL_DISABLED) {
+ if (del_timer(&t->probe_timer))
+ sctp_transport_put(t);
+ t->pl.state = SCTP_PL_DISABLED;
+ }
+ }
+}
+
+static inline void sctp_transport_pl_update(struct sctp_transport *t)
+{
+ if (t->pl.state == SCTP_PL_DISABLED)
+ return;
+
+ if (del_timer(&t->probe_timer))
+ sctp_transport_put(t);
+
+ t->pl.state = SCTP_PL_BASE;
+ t->pl.pmtu = SCTP_BASE_PLPMTU;
+ t->pl.probe_size = SCTP_BASE_PLPMTU;
+}
+
+static inline bool sctp_transport_pl_enabled(struct sctp_transport *t)
+{
+ return t->pl.state != SCTP_PL_DISABLED;
+}
+
static inline bool sctp_newsk_ready(const struct sock *sk)
{
return sock_flag(sk, SOCK_DEAD) || sk->sk_socket;
/* Prototypes for timeout event state functions. */
sctp_state_fn_t sctp_sf_do_6_3_3_rtx;
sctp_state_fn_t sctp_sf_send_reconf;
+sctp_state_fn_t sctp_sf_send_probe;
sctp_state_fn_t sctp_sf_do_6_2_sack;
sctp_state_fn_t sctp_sf_autoclose_timer_expire;
const struct sctp_association *asoc,
const struct sctp_chunk *chunk);
struct sctp_chunk *sctp_make_heartbeat(const struct sctp_association *asoc,
- const struct sctp_transport *transport);
+ const struct sctp_transport *transport,
+ __u32 probe_size);
struct sctp_chunk *sctp_make_heartbeat_ack(const struct sctp_association *asoc,
const struct sctp_chunk *chunk,
const void *payload,
const size_t paylen);
+struct sctp_chunk *sctp_make_pad(const struct sctp_association *asoc, int len);
struct sctp_chunk *sctp_make_op_error(const struct sctp_association *asoc,
const struct sctp_chunk *chunk,
__be16 cause_code, const void *payload,
void sctp_generate_t3_rtx_event(struct timer_list *t);
void sctp_generate_heartbeat_event(struct timer_list *t);
void sctp_generate_reconf_event(struct timer_list *t);
+void sctp_generate_probe_event(struct timer_list *t);
void sctp_generate_proto_unreach_event(struct timer_list *t);
void sctp_ootb_pkt_free(struct sctp_packet *packet);
* will be inherited by all new associations.
*/
__u32 hbinterval;
+ __u32 probe_interval;
__be16 udp_port;
__be16 encap_port;
union sctp_addr daddr;
unsigned long sent_at;
__u64 hb_nonce;
+ __u32 probe_size;
};
int sctp_stream_init(struct sctp_stream *stream, __u16 outcnt, __u16 incnt,
data_accepted:1, /* At least 1 chunk accepted */
auth:1, /* IN: was auth'ed | OUT: needs auth */
has_asconf:1, /* IN: have seen an asconf before */
+ pmtu_probe:1, /* Used by PLPMTUD, can be set in s HB chunk */
tsn_missing_report:2, /* Data chunk missing counter. */
fast_retransmit:2; /* Is this chunk fast retransmitted? */
};
* the destination address every heartbeat interval.
*/
unsigned long hbinterval;
+ unsigned long probe_interval;
/* SACK delay timeout */
unsigned long sackdelay;
/* Timer to handler reconf chunk rtx */
struct timer_list reconf_timer;
+ /* Timer to send a probe HB packet for PLPMTUD */
+ struct timer_list probe_timer;
+
/* Since we're using per-destination retransmission timers
* (see above), we're also using per-destination "transmitted"
* queues. This probably ought to be a private struct
char cacc_saw_newack;
} cacc;
+ struct {
+ __u16 pmtu;
+ __u16 probe_size;
+ __u16 probe_high;
+ __u8 probe_count:3;
+ __u8 raise_count:5;
+ __u8 state;
+ } pl; /* plpmtud related */
+
/* 64-bit random number sent with heartbeat. */
__u64 hb_nonce;
void sctp_transport_reset_t3_rtx(struct sctp_transport *);
void sctp_transport_reset_hb_timer(struct sctp_transport *);
void sctp_transport_reset_reconf_timer(struct sctp_transport *transport);
+void sctp_transport_reset_probe_timer(struct sctp_transport *transport);
int sctp_transport_hold(struct sctp_transport *);
void sctp_transport_put(struct sctp_transport *);
void sctp_transport_update_rto(struct sctp_transport *, __u32);
void sctp_transport_immediate_rtx(struct sctp_transport *);
void sctp_transport_dst_release(struct sctp_transport *t);
void sctp_transport_dst_confirm(struct sctp_transport *t);
+void sctp_transport_pl_send(struct sctp_transport *t);
+void sctp_transport_pl_recv(struct sctp_transport *t);
/* This is the structure we use to queue packets as they come into
* will be inherited by all new transports.
*/
unsigned long hbinterval;
+ unsigned long probe_interval;
__be16 encap_port;
static inline void sk_set_txhash(struct sock *sk)
{
- sk->sk_txhash = net_tx_rndhash();
+ /* This pairs with READ_ONCE() in skb_set_hash_from_sk() */
+ WRITE_ONCE(sk->sk_txhash, net_tx_rndhash());
}
static inline bool sk_rethink_txhash(struct sock *sk)
static inline void skb_set_hash_from_sk(struct sk_buff *skb, struct sock *sk)
{
- if (sk->sk_txhash) {
+ /* This pairs with WRITE_ONCE() in sk_set_txhash() */
+ u32 txhash = READ_ONCE(sk->sk_txhash);
+
+ if (txhash) {
skb->l4_hash = 1;
- skb->hash = sk->sk_txhash;
+ skb->hash = txhash;
}
}
sk_mem_charge(sk, skb->truesize);
}
-static inline void skb_set_owner_sk_safe(struct sk_buff *skb, struct sock *sk)
+static inline __must_check bool skb_set_owner_sk_safe(struct sk_buff *skb, struct sock *sk)
{
if (sk && refcount_inc_not_zero(&sk->sk_refcnt)) {
skb_orphan(skb);
skb->destructor = sock_efree;
skb->sk = sk;
+ return true;
}
+ return false;
}
void sk_reset_timer(struct sock *sk, struct timer_list *timer,
static inline int sock_error(struct sock *sk)
{
int err;
- if (likely(!sk->sk_err))
+
+ /* Avoid an atomic operation for the common case.
+ * This is racy since another cpu/thread can change sk_err under us.
+ */
+ if (likely(data_race(!sk->sk_err)))
return 0;
+
err = xchg(&sk->sk_err, 0);
return -err;
}
void sock_def_readable(struct sock *sk);
int sock_bindtoindex(struct sock *sk, int ifindex, bool lock_sk);
+void sock_set_timestamp(struct sock *sk, int optname, bool valbool);
+int sock_set_timestamping(struct sock *sk, int optname, int val);
+
void sock_enable_timestamps(struct sock *sk);
void sock_no_linger(struct sock *sk);
void sock_set_keepalive(struct sock *sk);
struct sock_reuseport {
struct rcu_head rcu;
- u16 max_socks; /* length of socks */
- u16 num_socks; /* elements in socks */
+ u16 max_socks; /* length of socks */
+ u16 num_socks; /* elements in socks */
+ u16 num_closed_socks; /* closed elements in socks */
/* The last synq overflow event timestamp of this
* reuse->socks[] group.
*/
extern int reuseport_add_sock(struct sock *sk, struct sock *sk2,
bool bind_inany);
extern void reuseport_detach_sock(struct sock *sk);
+void reuseport_stop_listen_sock(struct sock *sk);
extern struct sock *reuseport_select_sock(struct sock *sk,
u32 hash,
struct sk_buff *skb,
int hdr_len);
+struct sock *reuseport_migrate_sock(struct sock *sk,
+ struct sock *migrating_sk,
+ struct sk_buff *skb);
extern int reuseport_attach_prog(struct sock *sk, struct bpf_prog *prog);
extern int reuseport_detach_prog(struct sock *sk);
u16 tcfv_push_vid;
__be16 tcfv_push_proto;
u8 tcfv_push_prio;
+ bool tcfv_push_prio_exists;
struct rcu_head rcu;
};
int flags, int *addr_len);
int tcp_set_rcvlowat(struct sock *sk, int val);
int tcp_set_window_clamp(struct sock *sk, int val);
+void tcp_update_recv_tstamps(struct sk_buff *skb,
+ struct scm_timestamping_internal *tss);
+void tcp_recv_timestamp(struct msghdr *msg, const struct sock *sk,
+ struct scm_timestamping_internal *tss);
void tcp_data_ready(struct sock *sk);
#ifdef CONFIG_MMU
int tcp_mmap(struct file *file, struct socket *sock,
__SNMP_INC_STATS((net)->mib.tls_statistics, field)
#define TLS_INC_STATS(net, field) \
SNMP_INC_STATS((net)->mib.tls_statistics, field)
-#define __TLS_DEC_STATS(net, field) \
- __SNMP_DEC_STATS((net)->mib.tls_statistics, field)
#define TLS_DEC_STATS(net, field) \
SNMP_DEC_STATS((net)->mib.tls_statistics, field)
(sizeof(struct tls_offload_context_tx) + TLS_DRIVER_STATE_SIZE_TX)
enum tls_context_flags {
- TLS_RX_SYNC_RUNNING = 0,
+ /* tls_device_down was called after the netdev went down, device state
+ * was released, and kTLS works in software, even though rx_conf is
+ * still TLS_HW (needed for transition).
+ */
+ TLS_RX_DEV_DEGRADED = 0,
/* Unlike RX where resync is driven entirely by the core in TX only
* the driver knows when things went out of sync, so we need the flag
* to be atomic.
/* cache cold stuff */
struct proto *sk_proto;
+ struct sock *sk;
void (*sk_destruct)(struct sock *sk);
struct sk_buff *
tls_validate_xmit_skb(struct sock *sk, struct net_device *dev,
struct sk_buff *skb);
+struct sk_buff *
+tls_validate_xmit_skb_sw(struct sock *sk, struct net_device *dev,
+ struct sk_buff *skb);
static inline bool tls_is_sk_tx_device_offloaded(struct sock *sk)
{
struct sk_buff *xdp_build_skb_from_frame(struct xdp_frame *xdpf,
struct net_device *dev);
int xdp_alloc_skb_bulk(void **skbs, int n_skb, gfp_t gfp);
+struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf);
static inline
void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp)
#define CRYPTO_INVALID_PROTOCOL 128
__u8 proto;
+ __u8 inner_ipproto;
};
struct sec_path {
#define SND_SOC_DAIFMT_CBP_CFP (1 << 12) /* codec clk provider & frame provider */
#define SND_SOC_DAIFMT_CBC_CFP (2 << 12) /* codec clk consumer & frame provider */
#define SND_SOC_DAIFMT_CBP_CFC (3 << 12) /* codec clk provider & frame consumer */
-#define SND_SOC_DAIFMT_CBC_CFC (4 << 12) /* codec clk consumer & frame follower */
+#define SND_SOC_DAIFMT_CBC_CFC (4 << 12) /* codec clk consumer & frame consumer */
/* previous definitions kept for backwards-compatibility, do not use in new contributions */
#define SND_SOC_DAIFMT_CBM_CFM SND_SOC_DAIFMT_CBP_CFP
__field(u64, data_seq)
__field(u32, subflow_seq)
__field(u16, data_len)
+ __field(u16, csum)
__field(u8, use_map)
__field(u8, dsn64)
__field(u8, data_fin)
__field(u8, frozen)
__field(u8, reset_transient)
__field(u8, reset_reason)
+ __field(u8, csum_reqd)
),
TP_fast_assign(
__entry->data_seq = mpext->data_seq;
__entry->subflow_seq = mpext->subflow_seq;
__entry->data_len = mpext->data_len;
+ __entry->csum = (__force u16)mpext->csum;
__entry->use_map = mpext->use_map;
__entry->dsn64 = mpext->dsn64;
__entry->data_fin = mpext->data_fin;
__entry->frozen = mpext->frozen;
__entry->reset_transient = mpext->reset_transient;
__entry->reset_reason = mpext->reset_reason;
+ __entry->csum_reqd = mpext->csum_reqd;
),
- TP_printk("data_ack=%llu data_seq=%llu subflow_seq=%u data_len=%u use_map=%u dsn64=%u data_fin=%u use_ack=%u ack64=%u mpc_map=%u frozen=%u reset_transient=%u reset_reason=%u",
+ TP_printk("data_ack=%llu data_seq=%llu subflow_seq=%u data_len=%u csum=%x use_map=%u dsn64=%u data_fin=%u use_ack=%u ack64=%u mpc_map=%u frozen=%u reset_transient=%u reset_reason=%u csum_reqd=%u",
__entry->data_ack, __entry->data_seq,
__entry->subflow_seq, __entry->data_len,
- __entry->use_map, __entry->dsn64,
- __entry->data_fin, __entry->use_ack,
- __entry->ack64, __entry->mpc_map,
- __entry->frozen, __entry->reset_transient,
- __entry->reset_reason)
+ __entry->csum, __entry->use_map,
+ __entry->dsn64, __entry->data_fin,
+ __entry->use_ack, __entry->ack64,
+ __entry->mpc_map, __entry->frozen,
+ __entry->reset_transient, __entry->reset_reason,
+ __entry->csum_reqd)
);
DEFINE_EVENT(mptcp_dump_mpext, get_mapping_status,
__entry->srtt, __entry->rcv_wnd, __entry->sock_cookie)
);
+#define TP_STORE_ADDR_PORTS_SKB_V4(__entry, skb) \
+ do { \
+ const struct tcphdr *th = (const struct tcphdr *)skb->data; \
+ struct sockaddr_in *v4 = (void *)__entry->saddr; \
+ \
+ v4->sin_family = AF_INET; \
+ v4->sin_port = th->source; \
+ v4->sin_addr.s_addr = ip_hdr(skb)->saddr; \
+ v4 = (void *)__entry->daddr; \
+ v4->sin_family = AF_INET; \
+ v4->sin_port = th->dest; \
+ v4->sin_addr.s_addr = ip_hdr(skb)->daddr; \
+ } while (0)
+
+#if IS_ENABLED(CONFIG_IPV6)
+
+#define TP_STORE_ADDR_PORTS_SKB(__entry, skb) \
+ do { \
+ const struct iphdr *iph = ip_hdr(skb); \
+ \
+ if (iph->version == 6) { \
+ const struct tcphdr *th = (const struct tcphdr *)skb->data; \
+ struct sockaddr_in6 *v6 = (void *)__entry->saddr; \
+ \
+ v6->sin6_family = AF_INET6; \
+ v6->sin6_port = th->source; \
+ v6->sin6_addr = ipv6_hdr(skb)->saddr; \
+ v6 = (void *)__entry->daddr; \
+ v6->sin6_family = AF_INET6; \
+ v6->sin6_port = th->dest; \
+ v6->sin6_addr = ipv6_hdr(skb)->daddr; \
+ } else \
+ TP_STORE_ADDR_PORTS_SKB_V4(__entry, skb); \
+ } while (0)
+
+#else
+
+#define TP_STORE_ADDR_PORTS_SKB(__entry, skb) \
+ TP_STORE_ADDR_PORTS_SKB_V4(__entry, skb)
+
+#endif
+
+/*
+ * tcp event with only skb
+ */
+DECLARE_EVENT_CLASS(tcp_event_skb,
+
+ TP_PROTO(const struct sk_buff *skb),
+
+ TP_ARGS(skb),
+
+ TP_STRUCT__entry(
+ __field(const void *, skbaddr)
+ __array(__u8, saddr, sizeof(struct sockaddr_in6))
+ __array(__u8, daddr, sizeof(struct sockaddr_in6))
+ ),
+
+ TP_fast_assign(
+ __entry->skbaddr = skb;
+
+ memset(__entry->saddr, 0, sizeof(struct sockaddr_in6));
+ memset(__entry->daddr, 0, sizeof(struct sockaddr_in6));
+
+ TP_STORE_ADDR_PORTS_SKB(__entry, skb);
+ ),
+
+ TP_printk("src=%pISpc dest=%pISpc", __entry->saddr, __entry->daddr)
+);
+
+DEFINE_EVENT(tcp_event_skb, tcp_bad_csum,
+
+ TP_PROTO(const struct sk_buff *skb),
+
+ TP_ARGS(skb)
+);
+
#endif /* _TRACE_TCP_H */
/* This part must be outside protection */
#include <linux/tracepoint.h>
TRACE_DEFINE_ENUM(VIRTIO_VSOCK_TYPE_STREAM);
+TRACE_DEFINE_ENUM(VIRTIO_VSOCK_TYPE_SEQPACKET);
#define show_type(val) \
- __print_symbolic(val, { VIRTIO_VSOCK_TYPE_STREAM, "STREAM" })
+ __print_symbolic(val, \
+ { VIRTIO_VSOCK_TYPE_STREAM, "STREAM" }, \
+ { VIRTIO_VSOCK_TYPE_SEQPACKET, "SEQPACKET" })
TRACE_DEFINE_ENUM(VIRTIO_VSOCK_OP_INVALID);
TRACE_DEFINE_ENUM(VIRTIO_VSOCK_OP_REQUEST);
u32 ifindex = 0, map_index = index;
if (map_type == BPF_MAP_TYPE_DEVMAP || map_type == BPF_MAP_TYPE_DEVMAP_HASH) {
- ifindex = ((struct _bpf_dtab_netdev *)tgt)->dev->ifindex;
+ /* Just leave to_ifindex to 0 if do broadcast redirect,
+ * as tgt will be NULL.
+ */
+ if (tgt)
+ ifindex = ((struct _bpf_dtab_netdev *)tgt)->dev->ifindex;
} else if (map_type == BPF_MAP_TYPE_UNSPEC && map_id == INT_MAX) {
ifindex = index;
map_index = 0;
/* SIGILL, SIGFPE, SIGSEGV, SIGBUS, SIGTRAP, SIGEMT */
struct {
void __user *_addr; /* faulting insn/memory ref. */
-#ifdef __ARCH_SI_TRAPNO
- int _trapno; /* TRAP # which caused the signal */
-#endif
#ifdef __ia64__
int _imm; /* immediate value for "break" */
unsigned int _flags; /* see ia64 si_flags */
#define __ADDR_BND_PKEY_PAD (__alignof__(void *) < sizeof(short) ? \
sizeof(short) : __alignof__(void *))
union {
+ /* used on alpha and sparc */
+ int _trapno; /* TRAP # which caused the signal */
/*
* used when si_code=BUS_MCEERR_AR or
* used when si_code=BUS_MCEERR_AO
__u32 _pkey;
} _addr_pkey;
/* used when si_code=TRAP_PERF */
- unsigned long _perf;
+ struct {
+ unsigned long _data;
+ __u32 _type;
+ } _perf;
};
} _sigfault;
#define si_int _sifields._rt._sigval.sival_int
#define si_ptr _sifields._rt._sigval.sival_ptr
#define si_addr _sifields._sigfault._addr
-#ifdef __ARCH_SI_TRAPNO
#define si_trapno _sifields._sigfault._trapno
-#endif
#define si_addr_lsb _sifields._sigfault._addr_lsb
#define si_lower _sifields._sigfault._addr_bnd._lower
#define si_upper _sifields._sigfault._addr_bnd._upper
#define si_pkey _sifields._sigfault._addr_pkey._pkey
-#define si_perf _sifields._sigfault._perf
+#define si_perf_data _sifields._sigfault._perf._data
+#define si_perf_type _sifields._sigfault._perf._type
#define si_band _sifields._sigpoll._band
#define si_fd _sifields._sigpoll._fd
#define si_call_addr _sifields._sigsys._call_addr
#define SO_PREFER_BUSY_POLL 69
#define SO_BUSY_POLL_BUDGET 70
+#define SO_NETNS_COOKIE 71
+
#if !defined(__KERNEL__)
#if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__))
__SC_COMP(__NR_epoll_pwait2, sys_epoll_pwait2, compat_sys_epoll_pwait2)
#define __NR_mount_setattr 442
__SYSCALL(__NR_mount_setattr, sys_mount_setattr)
-#define __NR_quotactl_path 443
-__SYSCALL(__NR_quotactl_path, sys_quotactl_path)
+/* 443 is reserved for quotactl_path */
#define __NR_landlock_create_ruleset 444
__SYSCALL(__NR_landlock_create_ruleset, sys_landlock_create_ruleset)
* Look up an element with the given *key* in the map referred to
* by the file descriptor *fd*, and if found, delete the element.
*
+ * For **BPF_MAP_TYPE_QUEUE** and **BPF_MAP_TYPE_STACK** map
+ * types, the *flags* argument needs to be set to 0, but for other
+ * map types, it may be specified as:
+ *
+ * **BPF_F_LOCK**
+ * Look up and delete the value of a spin-locked map
+ * without returning the lock. This must be specified if
+ * the elements contain a spinlock.
+ *
* The **BPF_MAP_TYPE_QUEUE** and **BPF_MAP_TYPE_STACK** map types
* implement this command as a "pop" operation, deleting the top
* element rather than one corresponding to *key*.
* This command is only valid for the following map types:
* * **BPF_MAP_TYPE_QUEUE**
* * **BPF_MAP_TYPE_STACK**
+ * * **BPF_MAP_TYPE_HASH**
+ * * **BPF_MAP_TYPE_PERCPU_HASH**
+ * * **BPF_MAP_TYPE_LRU_HASH**
+ * * **BPF_MAP_TYPE_LRU_PERCPU_HASH**
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
BPF_PROG_ATTACH,
BPF_PROG_DETACH,
BPF_PROG_TEST_RUN,
+ BPF_PROG_RUN = BPF_PROG_TEST_RUN,
BPF_PROG_GET_NEXT_ID,
BPF_MAP_GET_NEXT_ID,
BPF_PROG_GET_FD_BY_ID,
BPF_PROG_TYPE_EXT,
BPF_PROG_TYPE_LSM,
BPF_PROG_TYPE_SK_LOOKUP,
+ BPF_PROG_TYPE_SYSCALL, /* a program that can execute syscalls */
};
enum bpf_attach_type {
BPF_SK_LOOKUP,
BPF_XDP,
BPF_SK_SKB_VERDICT,
+ BPF_SK_REUSEPORT_SELECT,
+ BPF_SK_REUSEPORT_SELECT_OR_MIGRATE,
__MAX_BPF_ATTACH_TYPE
};
/* When BPF ldimm64's insn[0].src_reg != 0 then this can have
* the following extensions:
*
- * insn[0].src_reg: BPF_PSEUDO_MAP_FD
- * insn[0].imm: map fd
+ * insn[0].src_reg: BPF_PSEUDO_MAP_[FD|IDX]
+ * insn[0].imm: map fd or fd_idx
* insn[1].imm: 0
* insn[0].off: 0
* insn[1].off: 0
* verifier type: CONST_PTR_TO_MAP
*/
#define BPF_PSEUDO_MAP_FD 1
-/* insn[0].src_reg: BPF_PSEUDO_MAP_VALUE
- * insn[0].imm: map fd
+#define BPF_PSEUDO_MAP_IDX 5
+
+/* insn[0].src_reg: BPF_PSEUDO_MAP_[IDX_]VALUE
+ * insn[0].imm: map fd or fd_idx
* insn[1].imm: offset into value
* insn[0].off: 0
* insn[1].off: 0
* ldimm64 rewrite: address of map[0]+offset
* verifier type: PTR_TO_MAP_VALUE
*/
-#define BPF_PSEUDO_MAP_VALUE 2
+#define BPF_PSEUDO_MAP_VALUE 2
+#define BPF_PSEUDO_MAP_IDX_VALUE 6
+
/* insn[0].src_reg: BPF_PSEUDO_BTF_ID
* insn[0].imm: kernel btd id of VAR
* insn[1].imm: 0
/* or valid module BTF object fd or 0 to attach to vmlinux */
__u32 attach_btf_obj_fd;
};
+ __u32 :32; /* pad */
+ __aligned_u64 fd_array; /* array of FDs */
};
struct { /* anonymous struct used by BPF_OBJ_* commands */
* The lower two bits of *flags* are used as the return code if
* the map lookup fails. This is so that the return value can be
* one of the XDP program return codes up to **XDP_TX**, as chosen
- * by the caller. Any higher bits in the *flags* argument must be
- * unset.
+ * by the caller. The higher bits of *flags* can be set to
+ * BPF_F_BROADCAST or BPF_F_EXCLUDE_INGRESS as defined below.
+ *
+ * With BPF_F_BROADCAST the packet will be broadcasted to all the
+ * interfaces in the map, with BPF_F_EXCLUDE_INGRESS the ingress
+ * interface will be excluded when do broadcasting.
*
* See also **bpf_redirect**\ (), which only supports redirecting
* to an ifindex, but doesn't require a map to do so.
* be zero-terminated except when **str_size** is 0.
*
* Or **-EBUSY** if the per-CPU memory copy buffer is busy.
+ *
+ * long bpf_sys_bpf(u32 cmd, void *attr, u32 attr_size)
+ * Description
+ * Execute bpf syscall with given arguments.
+ * Return
+ * A syscall result.
+ *
+ * long bpf_btf_find_by_name_kind(char *name, int name_sz, u32 kind, int flags)
+ * Description
+ * Find BTF type with given name and kind in vmlinux BTF or in module's BTFs.
+ * Return
+ * Returns btf_id and btf_obj_fd in lower and upper 32 bits.
+ *
+ * long bpf_sys_close(u32 fd)
+ * Description
+ * Execute close syscall for given FD.
+ * Return
+ * A syscall result.
*/
#define __BPF_FUNC_MAPPER(FN) \
FN(unspec), \
FN(check_mtu), \
FN(for_each_map_elem), \
FN(snprintf), \
+ FN(sys_bpf), \
+ FN(btf_find_by_name_kind), \
+ FN(sys_close), \
/* */
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
BPF_F_BPRM_SECUREEXEC = (1ULL << 0),
};
+/* Flags for bpf_redirect_map helper */
+enum {
+ BPF_F_BROADCAST = (1ULL << 3),
+ BPF_F_EXCLUDE_INGRESS = (1ULL << 4),
+};
+
#define __bpf_md_ptr(type, name) \
union { \
type name; \
__u32 ip_protocol; /* IP protocol. e.g. IPPROTO_TCP, IPPROTO_UDP */
__u32 bind_inany; /* Is sock bound to an INANY address? */
__u32 hash; /* A hash of the packet 4 tuples */
+ /* When reuse->migrating_sk is NULL, it is selecting a sk for the
+ * new incoming connection request (e.g. selecting a listen sk for
+ * the received SYN in the TCP case). reuse->sk is one of the sk
+ * in the reuseport group. The bpf prog can use reuse->sk to learn
+ * the local listening ip/port without looking into the skb.
+ *
+ * When reuse->migrating_sk is not NULL, reuse->sk is closed and
+ * reuse->migrating_sk is the socket that needs to be migrated
+ * to another listening socket. migrating_sk could be a fullsock
+ * sk that is fully established or a reqsk that is in-the-middle
+ * of 3-way handshake.
+ */
+ __bpf_md_ptr(struct bpf_sock *, sk);
+ __bpf_md_ptr(struct bpf_sock *, migrating_sk);
};
#define BPF_TAG_SIZE 8
/*
* defined bits for canfd_frame.flags
*
- * The use of struct canfd_frame implies the Extended Data Length (EDL) bit to
- * be set in the CAN frame bitstream on the wire. The EDL bit switch turns
+ * The use of struct canfd_frame implies the FD Frame (FDF) bit to
+ * be set in the CAN frame bitstream on the wire. The FDF bit switch turns
* the CAN controllers bitstream processor into the CAN FD mode which creates
* two new options within the CAN FD frame specification:
*
* controller only the CANFD_BRS bit is relevant for real CAN controllers when
* building a CAN FD frame for transmission. Setting the CANFD_ESI bit can make
* sense for virtual CAN interfaces to test applications with echoed frames.
+ *
+ * The struct can_frame and struct canfd_frame intentionally share the same
+ * layout to be able to write CAN frame content into a CAN FD frame structure.
+ * When this is done the former differentiation via CAN_MTU / CANFD_MTU gets
+ * lost. CANFD_FDF allows programmers to mark CAN FD frames in the case of
+ * using struct canfd_frame for mixed CAN / CAN FD content (dual use).
+ * N.B. the Kernel APIs do NOT provide mixed CAN / CAN FD content inside of
+ * struct canfd_frame therefore the CANFD_FDF flag is disregarded by Linux.
*/
#define CANFD_BRS 0x01 /* bit rate switch (second bitrate for payload data) */
#define CANFD_ESI 0x02 /* error state indicator of the transmitting node */
+#define CANFD_FDF 0x04 /* mark CAN FD for dual use of struct canfd_frame */
/**
* struct canfd_frame - CAN flexible data rate frame structure
DEVLINK_CMD_HEALTH_REPORTER_TEST,
+ DEVLINK_CMD_RATE_GET, /* can dump */
+ DEVLINK_CMD_RATE_SET,
+ DEVLINK_CMD_RATE_NEW,
+ DEVLINK_CMD_RATE_DEL,
+
/* add new commands above here */
__DEVLINK_CMD_MAX,
DEVLINK_CMD_MAX = __DEVLINK_CMD_MAX - 1
*/
};
+enum devlink_rate_type {
+ DEVLINK_RATE_TYPE_LEAF,
+ DEVLINK_RATE_TYPE_NODE,
+};
+
enum devlink_param_cmode {
DEVLINK_PARAM_CMODE_RUNTIME,
DEVLINK_PARAM_CMODE_DRIVERINIT,
DEVLINK_ATTR_RELOAD_ACTION_STATS, /* nested */
DEVLINK_ATTR_PORT_PCI_SF_NUMBER, /* u32 */
+
+ DEVLINK_ATTR_RATE_TYPE, /* u16 */
+ DEVLINK_ATTR_RATE_TX_SHARE, /* u64 */
+ DEVLINK_ATTR_RATE_TX_MAX, /* u64 */
+ DEVLINK_ATTR_RATE_NODE_NAME, /* string */
+ DEVLINK_ATTR_RATE_PARENT_NODE_NAME, /* string */
+
/* add new attributes above here, update the policy in devlink.c */
__DEVLINK_ATTR_MAX,
ETHTOOL_PFC_PREVENTION_TOUT, /* timeout in msecs */
/*
* Add your fresh new tunable attribute above and remember to update
- * tunable_strings[] in net/core/ethtool.c
+ * tunable_strings[] in net/ethtool/common.c
*/
__ETHTOOL_TUNABLE_COUNT,
};
ETHTOOL_PHY_EDPD,
/*
* Add your fresh new phy tunable attribute above and remember to update
- * phy_tunable_strings[] in net/core/ethtool.c
+ * phy_tunable_strings[] in net/ethtool/common.c
*/
__ETHTOOL_PHY_TUNABLE_COUNT,
};
ETHTOOL_A_MODULE_EEPROM_PAGE, /* u8 */
ETHTOOL_A_MODULE_EEPROM_BANK, /* u8 */
ETHTOOL_A_MODULE_EEPROM_I2C_ADDRESS, /* u8 */
- ETHTOOL_A_MODULE_EEPROM_DATA, /* nested */
+ ETHTOOL_A_MODULE_EEPROM_DATA, /* binary */
__ETHTOOL_A_MODULE_EEPROM_CNT,
ETHTOOL_A_MODULE_EEPROM_MAX = (__ETHTOOL_A_MODULE_EEPROM_CNT - 1)
#define BLKROTATIONAL _IO(0x12,126)
#define BLKZEROOUT _IO(0x12,127)
/*
- * A jump here: 130-131 are reserved for zoned block devices
+ * A jump here: 130-136 are reserved for zoned block devices
* (see uapi/linux/blkzoned.h)
*/
#include <linux/types.h>
#include <asm/byteorder.h>
-#include <linux/in.h>
#include <linux/if.h>
#include <linux/in6.h>
struct {
struct icmp_ext_echo_ctype3_hdr ctype3_hdr;
union {
- struct in_addr ipv4_addr;
+ __be32 ipv4_addr;
struct in6_addr ipv6_addr;
} ip_addr;
} addr;
IFLA_ALT_IFNAME, /* Alternative ifname */
IFLA_PERM_ADDRESS,
IFLA_PROTO_DOWN_REASON,
+
+ /* device (sysfs) name as parent, used instead
+ * of IFLA_LINK where there's no parent netdev
+ */
+ IFLA_PARENT_DEV_NAME,
+ IFLA_PARENT_DEV_BUS_NAME,
+
__IFLA_MAX
};
#define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1)
#define RMNET_FLAGS_INGRESS_MAP_CKSUMV4 (1U << 2)
#define RMNET_FLAGS_EGRESS_MAP_CKSUMV4 (1U << 3)
+#define RMNET_FLAGS_INGRESS_MAP_CKSUMV5 (1U << 4)
+#define RMNET_FLAGS_EGRESS_MAP_CKSUMV5 (1U << 5)
enum {
IFLA_RMNET_UNSPEC,
/* Address indicating an error return. */
#define INADDR_NONE ((unsigned long int) 0xffffffff)
+/* Dummy address for src of ICMP replies if no real address is set (RFC7600). */
+#define INADDR_DUMMY ((unsigned long int) 0xc0000008)
+
/* Network number for local host loopback. */
#define IN_LOOPBACKNET 127
#define KEY_VOICECOMMAND 0x246 /* Listening Voice Command */
#define KEY_ASSISTANT 0x247 /* AL Context-aware desktop assistant */
#define KEY_KBD_LAYOUT_NEXT 0x248 /* AC Next Keyboard Layout Select */
+#define KEY_EMOJI_PICKER 0x249 /* Show/hide emoji picker (HUTRR101) */
#define KEY_BRIGHTNESS_MIN 0x250 /* Set Brightness to Minimum */
#define KEY_BRIGHTNESS_MAX 0x251 /* Set Brightness to Maximum */
#define IORING_FEAT_SQPOLL_NONFIXED (1U << 7)
#define IORING_FEAT_EXT_ARG (1U << 8)
#define IORING_FEAT_NATIVE_WORKERS (1U << 9)
+#define IORING_FEAT_RSRC_TAGS (1U << 10)
/*
* io_uring_register(2) opcodes and arguments
IORING_UNREGISTER_PERSONALITY = 10,
IORING_REGISTER_RESTRICTIONS = 11,
IORING_REGISTER_ENABLE_RINGS = 12,
- IORING_REGISTER_RSRC = 13,
- IORING_REGISTER_RSRC_UPDATE = 14,
+
+ /* extended with tagging */
+ IORING_REGISTER_FILES2 = 13,
+ IORING_REGISTER_FILES_UPDATE2 = 14,
+ IORING_REGISTER_BUFFERS2 = 15,
+ IORING_REGISTER_BUFFERS_UPDATE = 16,
/* this goes last */
IORING_REGISTER_LAST
__aligned_u64 /* __s32 * */ fds;
};
-enum {
- IORING_RSRC_FILE = 0,
- IORING_RSRC_BUFFER = 1,
-};
-
struct io_uring_rsrc_register {
- __u32 type;
__u32 nr;
+ __u32 resv;
+ __u64 resv2;
__aligned_u64 data;
__aligned_u64 tags;
};
__u32 resv;
__aligned_u64 data;
__aligned_u64 tags;
- __u32 type;
__u32 nr;
+ __u32 resv2;
};
/* Skip updating fd indexes set to this value in the fd table */
* Note: you must update KVM_API_VERSION if you change this interface.
*/
+#include <linux/const.h>
#include <linux/types.h>
#include <linux/compiler.h>
#include <linux/ioctl.h>
* conversion after harvesting an entry. Also, it must not skip any
* dirty bits, so that dirty bits are always harvested in sequence.
*/
-#define KVM_DIRTY_GFN_F_DIRTY BIT(0)
-#define KVM_DIRTY_GFN_F_RESET BIT(1)
+#define KVM_DIRTY_GFN_F_DIRTY _BITUL(0)
+#define KVM_DIRTY_GFN_F_RESET _BITUL(1)
#define KVM_DIRTY_GFN_F_MASK 0x3
/*
__u64 mptcpi_rcv_nxt;
__u8 mptcpi_local_addr_used;
__u8 mptcpi_local_addr_max;
+ __u8 mptcpi_csum_enabled;
};
/*
* @NFT_EXTHDR_OP_IPV6: match against ipv6 extension headers
* @NFT_EXTHDR_OP_TCP: match against tcp options
* @NFT_EXTHDR_OP_IPV4: match against ipv4 options
+ * @NFT_EXTHDR_OP_SCTP: match against sctp chunks
*/
enum nft_exthdr_op {
NFT_EXTHDR_OP_IPV6,
NFT_EXTHDR_OP_TCPOPT,
NFT_EXTHDR_OP_IPV4,
+ NFT_EXTHDR_OP_SCTP,
__NFT_EXTHDR_OP_MAX
};
#define NFT_EXTHDR_OP_MAX (__NFT_EXTHDR_OP_MAX - 1)
#define NFTA_COUNTER_MAX (__NFTA_COUNTER_MAX - 1)
/**
+ * enum nft_last_attributes - nf_tables last expression netlink attributes
+ *
+ * @NFTA_LAST_SET: last update has been set, zero means never updated (NLA_U32)
+ * @NFTA_LAST_MSECS: milliseconds since last update (NLA_U64)
+ */
+enum nft_last_attributes {
+ NFTA_LAST_UNSPEC,
+ NFTA_LAST_SET,
+ NFTA_LAST_MSECS,
+ NFTA_LAST_PAD,
+ __NFTA_LAST_MAX
+};
+#define NFTA_LAST_MAX (__NFTA_LAST_MAX - 1)
+
+/**
* enum nft_log_attributes - nf_tables log expression netlink attributes
*
* @NFTA_LOG_GROUP: netlink group to send messages to (NLA_U32)
#define NFNL_SUBSYS_CTHELPER 9
#define NFNL_SUBSYS_NFTABLES 10
#define NFNL_SUBSYS_NFT_COMPAT 11
-#define NFNL_SUBSYS_COUNT 12
+#define NFNL_SUBSYS_HOOK 12
+#define NFNL_SUBSYS_COUNT 13
/* Reserved control nfnetlink messages */
#define NFNL_MSG_BATCH_BEGIN NLMSG_MIN_TYPE
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef _NFNL_HOOK_H_
+#define _NFNL_HOOK_H_
+
+enum nfnl_hook_msg_types {
+ NFNL_MSG_HOOK_GET,
+ NFNL_MSG_HOOK_MAX,
+};
+
+/**
+ * enum nfnl_hook_attributes - netfilter hook netlink attributes
+ *
+ * @NFNLA_HOOK_HOOKNUM: netfilter hook number (NLA_U32)
+ * @NFNLA_HOOK_PRIORITY: netfilter hook priority (NLA_U32)
+ * @NFNLA_HOOK_DEV: netdevice name (NLA_STRING)
+ * @NFNLA_HOOK_FUNCTION_NAME: hook function name (NLA_STRING)
+ * @NFNLA_HOOK_MODULE_NAME: kernel module that registered this hook (NLA_STRING)
+ * @NFNLA_HOOK_CHAIN_INFO: basechain hook metadata (NLA_NESTED)
+ */
+enum nfnl_hook_attributes {
+ NFNLA_HOOK_UNSPEC,
+ NFNLA_HOOK_HOOKNUM,
+ NFNLA_HOOK_PRIORITY,
+ NFNLA_HOOK_DEV,
+ NFNLA_HOOK_FUNCTION_NAME,
+ NFNLA_HOOK_MODULE_NAME,
+ NFNLA_HOOK_CHAIN_INFO,
+ __NFNLA_HOOK_MAX
+};
+#define NFNLA_HOOK_MAX (__NFNLA_HOOK_MAX - 1)
+
+/**
+ * enum nfnl_hook_chain_info_attributes - chain description
+ *
+ * NFNLA_HOOK_INFO_DESC: nft chain and table name (enum nft_table_attributes) (NLA_NESTED)
+ * NFNLA_HOOK_INFO_TYPE: chain type (enum nfnl_hook_chaintype) (NLA_U32)
+ */
+enum nfnl_hook_chain_info_attributes {
+ NFNLA_HOOK_INFO_UNSPEC,
+ NFNLA_HOOK_INFO_DESC,
+ NFNLA_HOOK_INFO_TYPE,
+ __NFNLA_HOOK_INFO_MAX,
+};
+#define NFNLA_HOOK_INFO_MAX (__NFNLA_HOOK_INFO_MAX - 1)
+
+/**
+ * enum nfnl_hook_chaintype - chain type
+ *
+ * @NFNL_HOOK_TYPE_NFTABLES nf_tables base chain
+ */
+enum nfnl_hook_chaintype {
+ NFNL_HOOK_TYPE_NFTABLES = 0x1,
+};
+
+#endif /* _NFNL_HOOK_H */
#define NLMSG_HDRLEN ((int) NLMSG_ALIGN(sizeof(struct nlmsghdr)))
#define NLMSG_LENGTH(len) ((len) + NLMSG_HDRLEN)
#define NLMSG_SPACE(len) NLMSG_ALIGN(NLMSG_LENGTH(len))
-#define NLMSG_DATA(nlh) ((void*)(((char*)nlh) + NLMSG_LENGTH(0)))
+#define NLMSG_DATA(nlh) ((void *)(((char *)nlh) + NLMSG_HDRLEN))
#define NLMSG_NEXT(nlh,len) ((len) -= NLMSG_ALIGN((nlh)->nlmsg_len), \
- (struct nlmsghdr*)(((char*)(nlh)) + NLMSG_ALIGN((nlh)->nlmsg_len)))
+ (struct nlmsghdr *)(((char *)(nlh)) + \
+ NLMSG_ALIGN((nlh)->nlmsg_len)))
#define NLMSG_OK(nlh,len) ((len) >= (int)sizeof(struct nlmsghdr) && \
(nlh)->nlmsg_len >= sizeof(struct nlmsghdr) && \
(nlh)->nlmsg_len <= (len))
* Copyright 2008 Jouni Malinen <jouni.malinen@atheros.com>
* Copyright 2008 Colin McCabe <colin@cozybit.com>
* Copyright 2015-2017 Intel Deutschland GmbH
- * Copyright (C) 2018-2020 Intel Corporation
+ * Copyright (C) 2018-2021 Intel Corporation
*
* Permission to use, copy, modify, and/or distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* defined
* @NL80211_BAND_IFTYPE_ATTR_HE_6GHZ_CAPA: HE 6GHz band capabilities (__le16),
* given for all 6 GHz band channels
+ * @NL80211_BAND_IFTYPE_ATTR_VENDOR_ELEMS: vendor element capabilities that are
+ * advertised on this band/for this iftype (binary)
* @__NL80211_BAND_IFTYPE_ATTR_AFTER_LAST: internal use
*/
enum nl80211_band_iftype_attr {
NL80211_BAND_IFTYPE_ATTR_HE_CAP_MCS_SET,
NL80211_BAND_IFTYPE_ATTR_HE_CAP_PPE,
NL80211_BAND_IFTYPE_ATTR_HE_6GHZ_CAPA,
+ NL80211_BAND_IFTYPE_ATTR_VENDOR_ELEMS,
/* keep last */
__NL80211_BAND_IFTYPE_ATTR_AFTER_LAST,
* @NL80211_PMSR_FTM_REQ_ATTR_LMR_FEEDBACK: negotiate for LMR feedback. Only
* valid if either %NL80211_PMSR_FTM_REQ_ATTR_TRIGGER_BASED or
* %NL80211_PMSR_FTM_REQ_ATTR_NON_TRIGGER_BASED is set.
+ * @NL80211_PMSR_FTM_REQ_ATTR_BSS_COLOR: optional. The BSS color of the
+ * responder. Only valid if %NL80211_PMSR_FTM_REQ_ATTR_NON_TRIGGER_BASED
+ * or %NL80211_PMSR_FTM_REQ_ATTR_TRIGGER_BASED is set.
*
* @NUM_NL80211_PMSR_FTM_REQ_ATTR: internal
* @NL80211_PMSR_FTM_REQ_ATTR_MAX: highest attribute number
NL80211_PMSR_FTM_REQ_ATTR_TRIGGER_BASED,
NL80211_PMSR_FTM_REQ_ATTR_NON_TRIGGER_BASED,
NL80211_PMSR_FTM_REQ_ATTR_LMR_FEEDBACK,
+ NL80211_PMSR_FTM_REQ_ATTR_BSS_COLOR,
/* keep last */
NUM_NL80211_PMSR_FTM_REQ_ATTR,
/*
* User provided data if sigtrap=1, passed back to user via
- * siginfo_t::si_perf, e.g. to permit user to identify the event.
+ * siginfo_t::si_perf_data, e.g. to permit user to identify the event.
*/
__u64 sig_data;
};
#define SCTP_EXPOSE_POTENTIALLY_FAILED_STATE 131
#define SCTP_EXPOSE_PF_STATE SCTP_EXPOSE_POTENTIALLY_FAILED_STATE
#define SCTP_REMOTE_UDP_ENCAPS_PORT 132
+#define SCTP_PLPMTUD_PROBE_INTERVAL 133
/* PR-SCTP policies */
#define SCTP_PR_SCTP_NONE 0x0000
SCTP_SS_MAX = SCTP_SS_RR
};
+/* Probe Interval socket option */
+struct sctp_probeinterval {
+ sctp_assoc_t spi_assoc_id;
+ struct sockaddr_storage spi_address;
+ __u32 spi_interval;
+};
+
#endif /* _UAPI_SCTP_H */
SEG6_LOCAL_ACTION_END_AM = 14,
/* custom BPF action */
SEG6_LOCAL_ACTION_END_BPF = 15,
+ /* decap and lookup of DA in v4 or v6 table */
+ SEG6_LOCAL_ACTION_END_DT46 = 16,
__SEG6_LOCAL_ACTION_MAX,
};
__s32 ssi_syscall;
__u64 ssi_call_addr;
__u32 ssi_arch;
- __u32 __pad3;
- __u64 ssi_perf;
/*
* Pad strcture to 128 bytes. Remember to update the
* comes out of a read(2) and we really don't want to have
* a compat on read(2).
*/
- __u8 __pad[16];
+ __u8 __pad[28];
};
SMC_NETLINK_GET_LGR_SMCD,
SMC_NETLINK_GET_DEV_SMCD,
SMC_NETLINK_GET_DEV_SMCR,
+ SMC_NETLINK_GET_STATS,
+ SMC_NETLINK_GET_FBACK_STATS,
};
/* SMC_GENL_FAMILY top level attributes */
SMC_GEN_LGR_SMCD, /* nest */
SMC_GEN_DEV_SMCD, /* nest */
SMC_GEN_DEV_SMCR, /* nest */
+ SMC_GEN_STATS, /* nest */
+ SMC_GEN_FBACK_STATS, /* nest */
__SMC_GEN_MAX,
SMC_GEN_MAX = __SMC_GEN_MAX - 1
};
SMC_NLA_DEV_MAX = __SMC_NLA_DEV_MAX - 1
};
+/* SMC_NLA_STATS_T_TX(RX)_RMB_SIZE nested attributes */
+/* SMC_NLA_STATS_TX(RX)PLOAD_SIZE nested attributes */
+enum {
+ SMC_NLA_STATS_PLOAD_PAD,
+ SMC_NLA_STATS_PLOAD_8K, /* u64 */
+ SMC_NLA_STATS_PLOAD_16K, /* u64 */
+ SMC_NLA_STATS_PLOAD_32K, /* u64 */
+ SMC_NLA_STATS_PLOAD_64K, /* u64 */
+ SMC_NLA_STATS_PLOAD_128K, /* u64 */
+ SMC_NLA_STATS_PLOAD_256K, /* u64 */
+ SMC_NLA_STATS_PLOAD_512K, /* u64 */
+ SMC_NLA_STATS_PLOAD_1024K, /* u64 */
+ SMC_NLA_STATS_PLOAD_G_1024K, /* u64 */
+ __SMC_NLA_STATS_PLOAD_MAX,
+ SMC_NLA_STATS_PLOAD_MAX = __SMC_NLA_STATS_PLOAD_MAX - 1
+};
+
+/* SMC_NLA_STATS_T_TX(RX)_RMB_STATS nested attributes */
+enum {
+ SMC_NLA_STATS_RMB_PAD,
+ SMC_NLA_STATS_RMB_SIZE_SM_PEER_CNT, /* u64 */
+ SMC_NLA_STATS_RMB_SIZE_SM_CNT, /* u64 */
+ SMC_NLA_STATS_RMB_FULL_PEER_CNT, /* u64 */
+ SMC_NLA_STATS_RMB_FULL_CNT, /* u64 */
+ SMC_NLA_STATS_RMB_REUSE_CNT, /* u64 */
+ SMC_NLA_STATS_RMB_ALLOC_CNT, /* u64 */
+ SMC_NLA_STATS_RMB_DGRADE_CNT, /* u64 */
+ __SMC_NLA_STATS_RMB_MAX,
+ SMC_NLA_STATS_RMB_MAX = __SMC_NLA_STATS_RMB_MAX - 1
+};
+
+/* SMC_NLA_STATS_SMCD_TECH and _SMCR_TECH nested attributes */
+enum {
+ SMC_NLA_STATS_T_PAD,
+ SMC_NLA_STATS_T_TX_RMB_SIZE, /* nest */
+ SMC_NLA_STATS_T_RX_RMB_SIZE, /* nest */
+ SMC_NLA_STATS_T_TXPLOAD_SIZE, /* nest */
+ SMC_NLA_STATS_T_RXPLOAD_SIZE, /* nest */
+ SMC_NLA_STATS_T_TX_RMB_STATS, /* nest */
+ SMC_NLA_STATS_T_RX_RMB_STATS, /* nest */
+ SMC_NLA_STATS_T_CLNT_V1_SUCC, /* u64 */
+ SMC_NLA_STATS_T_CLNT_V2_SUCC, /* u64 */
+ SMC_NLA_STATS_T_SRV_V1_SUCC, /* u64 */
+ SMC_NLA_STATS_T_SRV_V2_SUCC, /* u64 */
+ SMC_NLA_STATS_T_SENDPAGE_CNT, /* u64 */
+ SMC_NLA_STATS_T_SPLICE_CNT, /* u64 */
+ SMC_NLA_STATS_T_CORK_CNT, /* u64 */
+ SMC_NLA_STATS_T_NDLY_CNT, /* u64 */
+ SMC_NLA_STATS_T_URG_DATA_CNT, /* u64 */
+ SMC_NLA_STATS_T_RX_BYTES, /* u64 */
+ SMC_NLA_STATS_T_TX_BYTES, /* u64 */
+ SMC_NLA_STATS_T_RX_CNT, /* u64 */
+ SMC_NLA_STATS_T_TX_CNT, /* u64 */
+ __SMC_NLA_STATS_T_MAX,
+ SMC_NLA_STATS_T_MAX = __SMC_NLA_STATS_T_MAX - 1
+};
+
+/* SMC_GEN_STATS attributes */
+enum {
+ SMC_NLA_STATS_PAD,
+ SMC_NLA_STATS_SMCD_TECH, /* nest */
+ SMC_NLA_STATS_SMCR_TECH, /* nest */
+ SMC_NLA_STATS_CLNT_HS_ERR_CNT, /* u64 */
+ SMC_NLA_STATS_SRV_HS_ERR_CNT, /* u64 */
+ __SMC_NLA_STATS_MAX,
+ SMC_NLA_STATS_MAX = __SMC_NLA_STATS_MAX - 1
+};
+
+/* SMC_GEN_FBACK_STATS attributes */
+enum {
+ SMC_NLA_FBACK_STATS_PAD,
+ SMC_NLA_FBACK_STATS_TYPE, /* u8 */
+ SMC_NLA_FBACK_STATS_SRV_CNT, /* u64 */
+ SMC_NLA_FBACK_STATS_CLNT_CNT, /* u64 */
+ SMC_NLA_FBACK_STATS_RSN_CODE, /* u32 */
+ SMC_NLA_FBACK_STATS_RSN_CNT, /* u16 */
+ __SMC_NLA_FBACK_STATS_MAX,
+ SMC_NLA_FBACK_STATS_MAX = __SMC_NLA_FBACK_STATS_MAX - 1
+};
#endif /* _UAPI_LINUX_SMC_H */
LINUX_MIB_TCPDUPLICATEDATAREHASH, /* TCPDuplicateDataRehash */
LINUX_MIB_TCPDSACKRECVSEGS, /* TCPDSACKRecvSegs */
LINUX_MIB_TCPDSACKIGNOREDDUBIOUS, /* TCPDSACKIgnoredDubious */
+ LINUX_MIB_TCPMIGRATEREQSUCCESS, /* TCPMigrateReqSuccess */
+ LINUX_MIB_TCPMIGRATEREQFAILURE, /* TCPMigrateReqFailure */
__LINUX_MIB_MAX
};
#define VIRTIO_ID_SOUND 25 /* virtio sound */
#define VIRTIO_ID_FS 26 /* virtio filesystem */
#define VIRTIO_ID_PMEM 27 /* virtio pmem */
-#define VIRTIO_ID_BT 28 /* virtio bluetooth */
#define VIRTIO_ID_MAC80211_HWSIM 29 /* virtio mac80211-hwsim */
+#define VIRTIO_ID_BT 40 /* virtio bluetooth */
#endif /* _LINUX_VIRTIO_IDS_H */
#include <linux/virtio_ids.h>
#include <linux/virtio_config.h>
+/* The feature bitmap for virtio vsock */
+#define VIRTIO_VSOCK_F_SEQPACKET 1 /* SOCK_SEQPACKET supported */
+
struct virtio_vsock_config {
__le64 guest_cid;
} __attribute__((packed));
enum virtio_vsock_type {
VIRTIO_VSOCK_TYPE_STREAM = 1,
+ VIRTIO_VSOCK_TYPE_SEQPACKET = 2,
};
enum virtio_vsock_op {
VIRTIO_VSOCK_SHUTDOWN_SEND = 2,
};
+/* VIRTIO_VSOCK_OP_RW flags values */
+enum virtio_vsock_rw {
+ VIRTIO_VSOCK_SEQ_EOR = 1,
+};
+
#endif /* _UAPI_LINUX_VIRTIO_VSOCK_H */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */
+/*
+ * Copyright (C) 2021 Intel Corporation.
+ */
+#ifndef _UAPI_WWAN_H_
+#define _UAPI_WWAN_H_
+
+enum {
+ IFLA_WWAN_UNSPEC,
+ IFLA_WWAN_LINK_ID, /* u32 */
+
+ __IFLA_WWAN_MAX
+};
+#define IFLA_WWAN_MAX (__IFLA_WWAN_MAX - 1)
+
+#endif /* _UAPI_WWAN_H_ */
GAUDI_ENGINE_ID_SIZE
};
+/*
+ * ASIC specific PLL index
+ *
+ * Used to retrieve in frequency info of different IPs via
+ * HL_INFO_PLL_FREQUENCY under HL_IOCTL_INFO IOCTL. The enums need to be
+ * used as an index in struct hl_pll_frequency_info
+ */
+
+enum hl_goya_pll_index {
+ HL_GOYA_CPU_PLL = 0,
+ HL_GOYA_IC_PLL,
+ HL_GOYA_MC_PLL,
+ HL_GOYA_MME_PLL,
+ HL_GOYA_PCI_PLL,
+ HL_GOYA_EMMC_PLL,
+ HL_GOYA_TPC_PLL,
+ HL_GOYA_PLL_MAX
+};
+
+enum hl_gaudi_pll_index {
+ HL_GAUDI_CPU_PLL = 0,
+ HL_GAUDI_PCI_PLL,
+ HL_GAUDI_SRAM_PLL,
+ HL_GAUDI_HBM_PLL,
+ HL_GAUDI_NIC_PLL,
+ HL_GAUDI_DMA_PLL,
+ HL_GAUDI_MESH_PLL,
+ HL_GAUDI_MME_PLL,
+ HL_GAUDI_TPC_PLL,
+ HL_GAUDI_IF_PLL,
+ HL_GAUDI_PLL_MAX
+};
+
enum hl_device_status {
HL_DEVICE_STATUS_OPERATIONAL,
HL_DEVICE_STATUS_IN_RESET,
#ifndef _ASM_ARM_SWIOTLB_XEN_H
#define _ASM_ARM_SWIOTLB_XEN_H
-extern int xen_swiotlb_detect(void);
+#include <xen/features.h>
+#include <xen/xen.h>
+
+static inline int xen_swiotlb_detect(void)
+{
+ if (!xen_domain())
+ return 0;
+ if (xen_feature(XENFEAT_direct_mapped))
+ return 1;
+ /* legacy case */
+ if (!xen_feature(XENFEAT_not_direct_mapped) && xen_initial_domain())
+ return 1;
+ return 0;
+}
#endif /* _ASM_ARM_SWIOTLB_XEN_H */
source "kernel/irq/Kconfig"
source "kernel/time/Kconfig"
+source "kernel/bpf/Kconfig"
source "kernel/Kconfig.preempt"
menu "CPU/Task time and stats accounting"
# syscall, maps, verifier
-config BPF_LSM
- bool "LSM Instrumentation with BPF"
- depends on BPF_EVENTS
- depends on BPF_SYSCALL
- depends on SECURITY
- depends on BPF_JIT
- help
- Enables instrumentation of the security hooks with eBPF programs for
- implementing dynamic MAC and Audit Policies.
-
- If you are unsure how to answer this question, answer N.
-
-config BPF_SYSCALL
- bool "Enable bpf() system call"
- select BPF
- select IRQ_WORK
- select TASKS_TRACE_RCU
- select BINARY_PRINTF
- select NET_SOCK_MSG if INET
- default n
- help
- Enable the bpf() system call that allows to manipulate eBPF
- programs and maps via file descriptors.
-
-config ARCH_WANT_DEFAULT_BPF_JIT
- bool
-
-config BPF_JIT_ALWAYS_ON
- bool "Permanently enable BPF JIT and remove BPF interpreter"
- depends on BPF_SYSCALL && HAVE_EBPF_JIT && BPF_JIT
- help
- Enables BPF JIT and removes BPF interpreter to avoid
- speculative execution of BPF instructions by the interpreter
-
-config BPF_JIT_DEFAULT_ON
- def_bool ARCH_WANT_DEFAULT_BPF_JIT || BPF_JIT_ALWAYS_ON
- depends on HAVE_EBPF_JIT && BPF_JIT
-
-source "kernel/bpf/preload/Kconfig"
-
config USERFAULTFD
bool "Enable userfaultfd() system call"
depends on MMU
*/
set_mems_allowed(node_states[N_MEMORY]);
- cad_pid = task_pid(current);
+ cad_pid = get_pid(task_pid(current));
smp_prepare_cpus(setup_max_cpus);
struct mqueue_inode_info *info,
struct ext_wait_queue *this)
{
+ struct task_struct *task;
+
list_del(&this->list);
- get_task_struct(this->task);
+ task = get_task_struct(this->task);
/* see MQ_BARRIER for purpose/pairing */
smp_store_release(&this->state, STATE_READY);
- wake_q_add_safe(wake_q, this->task);
+ wake_q_add_safe(wake_q, task);
}
/* pipelined_send() - send a message directly to the task waiting in
struct msg_receiver *msr, *t;
list_for_each_entry_safe(msr, t, &msq->q_receivers, r_list) {
- get_task_struct(msr->r_tsk);
+ struct task_struct *r_tsk;
+
+ r_tsk = get_task_struct(msr->r_tsk);
/* see MSG_BARRIER for purpose/pairing */
smp_store_release(&msr->r_msg, ERR_PTR(res));
- wake_q_add_safe(wake_q, msr->r_tsk);
+ wake_q_add_safe(wake_q, r_tsk);
}
}
static inline void wake_up_sem_queue_prepare(struct sem_queue *q, int error,
struct wake_q_head *wake_q)
{
- get_task_struct(q->sleeper);
+ struct task_struct *sleeper;
+
+ sleeper = get_task_struct(q->sleeper);
/* see SEM_BARRIER_2 for purpose/pairing */
smp_store_release(&q->status, error);
- wake_q_add_safe(wake_q, q->sleeper);
+ wake_q_add_safe(wake_q, sleeper);
}
static void unlink_queue(struct sem_array *sma, struct sem_queue *q)
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0-only
+
+# BPF interpreter that, for example, classic socket filters depend on.
+config BPF
+ bool
+
+# Used by archs to tell that they support BPF JIT compiler plus which
+# flavour. Only one of the two can be selected for a specific arch since
+# eBPF JIT supersedes the cBPF JIT.
+
+# Classic BPF JIT (cBPF)
+config HAVE_CBPF_JIT
+ bool
+
+# Extended BPF JIT (eBPF)
+config HAVE_EBPF_JIT
+ bool
+
+# Used by archs to tell that they want the BPF JIT compiler enabled by
+# default for kernels that were compiled with BPF JIT support.
+config ARCH_WANT_DEFAULT_BPF_JIT
+ bool
+
+menu "BPF subsystem"
+
+config BPF_SYSCALL
+ bool "Enable bpf() system call"
+ select BPF
+ select IRQ_WORK
+ select TASKS_TRACE_RCU
+ select BINARY_PRINTF
+ select NET_SOCK_MSG if INET
+ default n
+ help
+ Enable the bpf() system call that allows to manipulate BPF programs
+ and maps via file descriptors.
+
+config BPF_JIT
+ bool "Enable BPF Just In Time compiler"
+ depends on BPF
+ depends on HAVE_CBPF_JIT || HAVE_EBPF_JIT
+ depends on MODULES
+ help
+ BPF programs are normally handled by a BPF interpreter. This option
+ allows the kernel to generate native code when a program is loaded
+ into the kernel. This will significantly speed-up processing of BPF
+ programs.
+
+ Note, an admin should enable this feature changing:
+ /proc/sys/net/core/bpf_jit_enable
+ /proc/sys/net/core/bpf_jit_harden (optional)
+ /proc/sys/net/core/bpf_jit_kallsyms (optional)
+
+config BPF_JIT_ALWAYS_ON
+ bool "Permanently enable BPF JIT and remove BPF interpreter"
+ depends on BPF_SYSCALL && HAVE_EBPF_JIT && BPF_JIT
+ help
+ Enables BPF JIT and removes BPF interpreter to avoid speculative
+ execution of BPF instructions by the interpreter.
+
+config BPF_JIT_DEFAULT_ON
+ def_bool ARCH_WANT_DEFAULT_BPF_JIT || BPF_JIT_ALWAYS_ON
+ depends on HAVE_EBPF_JIT && BPF_JIT
+
+config BPF_UNPRIV_DEFAULT_OFF
+ bool "Disable unprivileged BPF by default"
+ depends on BPF_SYSCALL
+ help
+ Disables unprivileged BPF by default by setting the corresponding
+ /proc/sys/kernel/unprivileged_bpf_disabled knob to 2. An admin can
+ still reenable it by setting it to 0 later on, or permanently
+ disable it by setting it to 1 (from which no other transition to
+ 0 is possible anymore).
+
+source "kernel/bpf/preload/Kconfig"
+
+config BPF_LSM
+ bool "Enable BPF LSM Instrumentation"
+ depends on BPF_EVENTS
+ depends on BPF_SYSCALL
+ depends on SECURITY
+ depends on BPF_JIT
+ help
+ Enables instrumentation of the security hooks with BPF programs for
+ implementing dynamic MAC and Audit Policies.
+
+ If you are unsure how to answer this question, answer N.
+
+endmenu # "BPF subsystem"
return;
}
- /* Netiher the bpf_prog nor the bpf-map's syscall
+ /* Neither the bpf_prog nor the bpf-map's syscall
* could be modifying the local_storage->list now.
* Thus, no elem can be added-to or deleted-from the
* local_storage->list by the bpf_prog or by the bpf-map's syscall.
return link->ops == &bpf_iter_link_lops;
}
-int bpf_iter_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
+int bpf_iter_link_attach(const union bpf_attr *attr, bpfptr_t uattr,
+ struct bpf_prog *prog)
{
- union bpf_iter_link_info __user *ulinfo;
struct bpf_link_primer link_primer;
struct bpf_iter_target_info *tinfo;
union bpf_iter_link_info linfo;
struct bpf_iter_link *link;
u32 prog_btf_id, linfo_len;
bool existed = false;
+ bpfptr_t ulinfo;
int err;
if (attr->link_create.target_fd || attr->link_create.flags)
memset(&linfo, 0, sizeof(union bpf_iter_link_info));
- ulinfo = u64_to_user_ptr(attr->link_create.iter_info);
+ ulinfo = make_bpfptr(attr->link_create.iter_info, uattr.is_kernel);
linfo_len = attr->link_create.iter_info_len;
- if (!ulinfo ^ !linfo_len)
+ if (bpfptr_is_null(ulinfo) ^ !linfo_len)
return -EINVAL;
- if (ulinfo) {
+ if (!bpfptr_is_null(ulinfo)) {
err = bpf_check_uarg_tail_zero(ulinfo, sizeof(linfo),
linfo_len);
if (err)
return err;
linfo_len = min_t(u32, linfo_len, sizeof(linfo));
- if (copy_from_user(&linfo, ulinfo, linfo_len))
+ if (copy_from_bpfptr(&linfo, ulinfo, linfo_len))
return -EFAULT;
}
return &bpf_inode_storage_get_proto;
case BPF_FUNC_inode_storage_delete:
return &bpf_inode_storage_delete_proto;
+#ifdef CONFIG_NET
case BPF_FUNC_sk_storage_get:
return &bpf_sk_storage_get_proto;
case BPF_FUNC_sk_storage_delete:
return &bpf_sk_storage_delete_proto;
+#endif /* CONFIG_NET */
case BPF_FUNC_spin_lock:
return &bpf_spin_lock_proto;
case BPF_FUNC_spin_unlock:
}
/* The set of hooks which are called without pagefaults disabled and are allowed
- * to "sleep" and thus can be used for sleeable BPF programs.
+ * to "sleep" and thus can be used for sleepable BPF programs.
*/
BTF_SET_START(sleepable_lsm_hooks)
BTF_ID(func, bpf_lsm_bpf)
* The BTF type section contains a list of 'struct btf_type' objects.
* Each one describes a C type. Recall from the above section
* that a 'struct btf_type' object could be immediately followed by extra
- * data in order to desribe some particular C types.
+ * data in order to describe some particular C types.
*
* type_id:
* ~~~~~~~
/*
* We need a new copy to our safe object, either because we haven't
- * yet copied and are intializing safe data, or because the data
+ * yet copied and are initializing safe data, or because the data
* we want falls outside the boundaries of the safe object.
*/
if (!safe) {
* BTF_KIND_FUNC_PROTO cannot be directly referred by
* a struct's member.
*
- * It should be a funciton pointer instead.
+ * It should be a function pointer instead.
* (i.e. struct's member -> BTF_KIND_PTR -> BTF_KIND_FUNC_PROTO)
*
* Hence, there is no btf_func_check_member().
return 0;
}
-static struct btf *btf_parse(void __user *btf_data, u32 btf_data_size,
+static struct btf *btf_parse(bpfptr_t btf_data, u32 btf_data_size,
u32 log_level, char __user *log_ubuf, u32 log_size)
{
struct btf_verifier_env *env = NULL;
btf->data = data;
btf->data_size = btf_data_size;
- if (copy_from_user(data, btf_data, btf_data_size)) {
+ if (copy_from_bpfptr(data, btf_data, btf_data_size)) {
err = -EFAULT;
goto errout;
}
m->ret_size = ret;
for (i = 0; i < nargs; i++) {
+ if (i == nargs - 1 && args[i].type == 0) {
+ bpf_log(log,
+ "The function %s with variable args is unsupported.\n",
+ tname);
+ return -EINVAL;
+ }
ret = __get_type_size(btf, args[i].type, &t);
if (ret < 0) {
bpf_log(log,
tname, i, btf_kind_str[BTF_INFO_KIND(t->info)]);
return -EINVAL;
}
+ if (ret == 0) {
+ bpf_log(log,
+ "The function %s has malformed void argument.\n",
+ tname);
+ return -EINVAL;
+ }
m->arg_size[i] = ret;
}
m->nr_args = nargs;
return anon_inode_getfd("btf", &btf_fops, btf, O_RDONLY | O_CLOEXEC);
}
-int btf_new_fd(const union bpf_attr *attr)
+int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr)
{
struct btf *btf;
int ret;
- btf = btf_parse(u64_to_user_ptr(attr->btf),
+ btf = btf_parse(make_bpfptr(attr->btf, uattr.is_kernel),
attr->btf_size, attr->btf_log_level,
u64_to_user_ptr(attr->btf_log_buf),
attr->btf_log_size);
return res;
}
+
+BPF_CALL_4(bpf_btf_find_by_name_kind, char *, name, int, name_sz, u32, kind, int, flags)
+{
+ struct btf *btf;
+ long ret;
+
+ if (flags)
+ return -EINVAL;
+
+ if (name_sz <= 1 || name[name_sz - 1])
+ return -EINVAL;
+
+ btf = bpf_get_btf_vmlinux();
+ if (IS_ERR(btf))
+ return PTR_ERR(btf);
+
+ ret = btf_find_by_name_kind(btf, name, kind);
+ /* ret is never zero, since btf_find_by_name_kind returns
+ * positive btf_id or negative error.
+ */
+ if (ret < 0) {
+ struct btf *mod_btf;
+ int id;
+
+ /* If name is not found in vmlinux's BTF then search in module's BTFs */
+ spin_lock_bh(&btf_idr_lock);
+ idr_for_each_entry(&btf_idr, mod_btf, id) {
+ if (!btf_is_module(mod_btf))
+ continue;
+ /* linear search could be slow hence unlock/lock
+ * the IDR to avoiding holding it for too long
+ */
+ btf_get(mod_btf);
+ spin_unlock_bh(&btf_idr_lock);
+ ret = btf_find_by_name_kind(mod_btf, name, kind);
+ if (ret > 0) {
+ int btf_obj_fd;
+
+ btf_obj_fd = __btf_new_fd(mod_btf);
+ if (btf_obj_fd < 0) {
+ btf_put(mod_btf);
+ return btf_obj_fd;
+ }
+ return ret | (((u64)btf_obj_fd) << 32);
+ }
+ spin_lock_bh(&btf_idr_lock);
+ btf_put(mod_btf);
+ }
+ spin_unlock_bh(&btf_idr_lock);
+ }
+ return ret;
+}
+
+const struct bpf_func_proto bpf_btf_find_by_name_kind_proto = {
+ .func = bpf_btf_find_by_name_kind,
+ .gpl_only = false,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_PTR_TO_MEM,
+ .arg2_type = ARG_CONST_SIZE,
+ .arg3_type = ARG_ANYTHING,
+ .arg4_type = ARG_ANYTHING,
+};
select_insn:
goto *jumptable[insn->code];
- /* ALU */
-#define ALU(OPCODE, OP) \
- ALU64_##OPCODE##_X: \
- DST = DST OP SRC; \
- CONT; \
- ALU_##OPCODE##_X: \
- DST = (u32) DST OP (u32) SRC; \
- CONT; \
- ALU64_##OPCODE##_K: \
- DST = DST OP IMM; \
- CONT; \
- ALU_##OPCODE##_K: \
- DST = (u32) DST OP (u32) IMM; \
+ /* Explicitly mask the register-based shift amounts with 63 or 31
+ * to avoid undefined behavior. Normally this won't affect the
+ * generated code, for example, in case of native 64 bit archs such
+ * as x86-64 or arm64, the compiler is optimizing the AND away for
+ * the interpreter. In case of JITs, each of the JIT backends compiles
+ * the BPF shift operations to machine instructions which produce
+ * implementation-defined results in such a case; the resulting
+ * contents of the register may be arbitrary, but program behaviour
+ * as a whole remains defined. In other words, in case of JIT backends,
+ * the AND must /not/ be added to the emitted LSH/RSH/ARSH translation.
+ */
+ /* ALU (shifts) */
+#define SHT(OPCODE, OP) \
+ ALU64_##OPCODE##_X: \
+ DST = DST OP (SRC & 63); \
+ CONT; \
+ ALU_##OPCODE##_X: \
+ DST = (u32) DST OP ((u32) SRC & 31); \
+ CONT; \
+ ALU64_##OPCODE##_K: \
+ DST = DST OP IMM; \
+ CONT; \
+ ALU_##OPCODE##_K: \
+ DST = (u32) DST OP (u32) IMM; \
+ CONT;
+ /* ALU (rest) */
+#define ALU(OPCODE, OP) \
+ ALU64_##OPCODE##_X: \
+ DST = DST OP SRC; \
+ CONT; \
+ ALU_##OPCODE##_X: \
+ DST = (u32) DST OP (u32) SRC; \
+ CONT; \
+ ALU64_##OPCODE##_K: \
+ DST = DST OP IMM; \
+ CONT; \
+ ALU_##OPCODE##_K: \
+ DST = (u32) DST OP (u32) IMM; \
CONT;
-
ALU(ADD, +)
ALU(SUB, -)
ALU(AND, &)
ALU(OR, |)
- ALU(LSH, <<)
- ALU(RSH, >>)
ALU(XOR, ^)
ALU(MUL, *)
+ SHT(LSH, <<)
+ SHT(RSH, >>)
+#undef SHT
#undef ALU
ALU_NEG:
DST = (u32) -DST;
insn++;
CONT;
ALU_ARSH_X:
- DST = (u64) (u32) (((s32) DST) >> SRC);
+ DST = (u64) (u32) (((s32) DST) >> (SRC & 31));
CONT;
ALU_ARSH_K:
DST = (u64) (u32) (((s32) DST) >> IMM);
CONT;
ALU64_ARSH_X:
- (*(s64 *) &DST) >>= SRC;
+ (*(s64 *) &DST) >>= (SRC & 63);
CONT;
ALU64_ARSH_K:
(*(s64 *) &DST) >>= IMM;
static int cpu_map_redirect(struct bpf_map *map, u32 ifindex, u64 flags)
{
- return __bpf_xdp_redirect_map(map, ifindex, flags, __cpu_map_lookup_elem);
+ return __bpf_xdp_redirect_map(map, ifindex, flags, 0,
+ __cpu_map_lookup_elem);
}
static int cpu_map_btf_id;
struct list_head flush_node;
struct net_device *dev;
struct net_device *dev_rx;
+ struct bpf_prog *xdp_prog;
unsigned int count;
};
list_del_rcu(&dtab->list);
spin_unlock(&dev_map_lock);
+ bpf_clear_redirect_map(map);
synchronize_rcu();
/* Make sure prior __dev_map_entry_free() have completed. */
return false;
}
+static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog,
+ struct xdp_frame **frames, int n,
+ struct net_device *dev)
+{
+ struct xdp_txq_info txq = { .dev = dev };
+ struct xdp_buff xdp;
+ int i, nframes = 0;
+
+ for (i = 0; i < n; i++) {
+ struct xdp_frame *xdpf = frames[i];
+ u32 act;
+ int err;
+
+ xdp_convert_frame_to_buff(xdpf, &xdp);
+ xdp.txq = &txq;
+
+ act = bpf_prog_run_xdp(xdp_prog, &xdp);
+ switch (act) {
+ case XDP_PASS:
+ err = xdp_update_frame_from_buff(&xdp, xdpf);
+ if (unlikely(err < 0))
+ xdp_return_frame_rx_napi(xdpf);
+ else
+ frames[nframes++] = xdpf;
+ break;
+ default:
+ bpf_warn_invalid_xdp_action(act);
+ fallthrough;
+ case XDP_ABORTED:
+ trace_xdp_exception(dev, xdp_prog, act);
+ fallthrough;
+ case XDP_DROP:
+ xdp_return_frame_rx_napi(xdpf);
+ break;
+ }
+ }
+ return nframes; /* sent frames count */
+}
+
static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
{
struct net_device *dev = bq->dev;
+ unsigned int cnt = bq->count;
int sent = 0, err = 0;
+ int to_send = cnt;
int i;
- if (unlikely(!bq->count))
+ if (unlikely(!cnt))
return;
- for (i = 0; i < bq->count; i++) {
+ for (i = 0; i < cnt; i++) {
struct xdp_frame *xdpf = bq->q[i];
prefetch(xdpf);
}
- sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags);
+ if (bq->xdp_prog) {
+ to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev);
+ if (!to_send)
+ goto out;
+ }
+
+ sent = dev->netdev_ops->ndo_xdp_xmit(dev, to_send, bq->q, flags);
if (sent < 0) {
/* If ndo_xdp_xmit fails with an errno, no frames have
* been xmit'ed.
/* If not all frames have been transmitted, it is our
* responsibility to free them
*/
- for (i = sent; unlikely(i < bq->count); i++)
+ for (i = sent; unlikely(i < to_send); i++)
xdp_return_frame_rx_napi(bq->q[i]);
- trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, bq->count - sent, err);
- bq->dev_rx = NULL;
+out:
bq->count = 0;
- __list_del_clearprev(&bq->flush_node);
+ trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, cnt - sent, err);
}
/* __dev_flush is called from xdp_do_flush() which _must_ be signaled
struct list_head *flush_list = this_cpu_ptr(&dev_flush_list);
struct xdp_dev_bulk_queue *bq, *tmp;
- list_for_each_entry_safe(bq, tmp, flush_list, flush_node)
+ list_for_each_entry_safe(bq, tmp, flush_list, flush_node) {
bq_xmit_all(bq, XDP_XMIT_FLUSH);
+ bq->dev_rx = NULL;
+ bq->xdp_prog = NULL;
+ __list_del_clearprev(&bq->flush_node);
+ }
}
/* rcu_read_lock (from syscall and BPF contexts) ensures that if a delete and/or
- * update happens in parallel here a dev_put wont happen until after reading the
- * ifindex.
+ * update happens in parallel here a dev_put won't happen until after reading
+ * the ifindex.
*/
static void *__dev_map_lookup_elem(struct bpf_map *map, u32 key)
{
* Thus, safe percpu variable access.
*/
static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf,
- struct net_device *dev_rx)
+ struct net_device *dev_rx, struct bpf_prog *xdp_prog)
{
struct list_head *flush_list = this_cpu_ptr(&dev_flush_list);
struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq);
/* Ingress dev_rx will be the same for all xdp_frame's in
* bulk_queue, because bq stored per-CPU and must be flushed
* from net_device drivers NAPI func end.
+ *
+ * Do the same with xdp_prog and flush_list since these fields
+ * are only ever modified together.
*/
- if (!bq->dev_rx)
+ if (!bq->dev_rx) {
bq->dev_rx = dev_rx;
+ bq->xdp_prog = xdp_prog;
+ list_add(&bq->flush_node, flush_list);
+ }
bq->q[bq->count++] = xdpf;
-
- if (!bq->flush_node.prev)
- list_add(&bq->flush_node, flush_list);
}
static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp,
- struct net_device *dev_rx)
+ struct net_device *dev_rx,
+ struct bpf_prog *xdp_prog)
{
struct xdp_frame *xdpf;
int err;
if (unlikely(!xdpf))
return -EOVERFLOW;
- bq_enqueue(dev, xdpf, dev_rx);
+ bq_enqueue(dev, xdpf, dev_rx, xdp_prog);
return 0;
}
-static struct xdp_buff *dev_map_run_prog(struct net_device *dev,
- struct xdp_buff *xdp,
- struct bpf_prog *xdp_prog)
+int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp,
+ struct net_device *dev_rx)
{
- struct xdp_txq_info txq = { .dev = dev };
- u32 act;
+ return __xdp_enqueue(dev, xdp, dev_rx, NULL);
+}
- xdp_set_data_meta_invalid(xdp);
- xdp->txq = &txq;
+int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp,
+ struct net_device *dev_rx)
+{
+ struct net_device *dev = dst->dev;
- act = bpf_prog_run_xdp(xdp_prog, xdp);
- switch (act) {
- case XDP_PASS:
- return xdp;
- case XDP_DROP:
- break;
- default:
- bpf_warn_invalid_xdp_action(act);
- fallthrough;
- case XDP_ABORTED:
- trace_xdp_exception(dev, xdp_prog, act);
- break;
- }
+ return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog);
+}
- xdp_return_buff(xdp);
- return NULL;
+static bool is_valid_dst(struct bpf_dtab_netdev *obj, struct xdp_buff *xdp,
+ int exclude_ifindex)
+{
+ if (!obj || obj->dev->ifindex == exclude_ifindex ||
+ !obj->dev->netdev_ops->ndo_xdp_xmit)
+ return false;
+
+ if (xdp_ok_fwd_dev(obj->dev, xdp->data_end - xdp->data))
+ return false;
+
+ return true;
}
-int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp,
- struct net_device *dev_rx)
+static int dev_map_enqueue_clone(struct bpf_dtab_netdev *obj,
+ struct net_device *dev_rx,
+ struct xdp_frame *xdpf)
{
- return __xdp_enqueue(dev, xdp, dev_rx);
+ struct xdp_frame *nxdpf;
+
+ nxdpf = xdpf_clone(xdpf);
+ if (!nxdpf)
+ return -ENOMEM;
+
+ bq_enqueue(obj->dev, nxdpf, dev_rx, obj->xdp_prog);
+
+ return 0;
}
-int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp,
- struct net_device *dev_rx)
+int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx,
+ struct bpf_map *map, bool exclude_ingress)
{
- struct net_device *dev = dst->dev;
+ struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
+ int exclude_ifindex = exclude_ingress ? dev_rx->ifindex : 0;
+ struct bpf_dtab_netdev *dst, *last_dst = NULL;
+ struct hlist_head *head;
+ struct xdp_frame *xdpf;
+ unsigned int i;
+ int err;
- if (dst->xdp_prog) {
- xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog);
- if (!xdp)
- return 0;
+ xdpf = xdp_convert_buff_to_frame(xdp);
+ if (unlikely(!xdpf))
+ return -EOVERFLOW;
+
+ if (map->map_type == BPF_MAP_TYPE_DEVMAP) {
+ for (i = 0; i < map->max_entries; i++) {
+ dst = READ_ONCE(dtab->netdev_map[i]);
+ if (!is_valid_dst(dst, xdp, exclude_ifindex))
+ continue;
+
+ /* we only need n-1 clones; last_dst enqueued below */
+ if (!last_dst) {
+ last_dst = dst;
+ continue;
+ }
+
+ err = dev_map_enqueue_clone(last_dst, dev_rx, xdpf);
+ if (err)
+ return err;
+
+ last_dst = dst;
+ }
+ } else { /* BPF_MAP_TYPE_DEVMAP_HASH */
+ for (i = 0; i < dtab->n_buckets; i++) {
+ head = dev_map_index_hash(dtab, i);
+ hlist_for_each_entry_rcu(dst, head, index_hlist,
+ lockdep_is_held(&dtab->index_lock)) {
+ if (!is_valid_dst(dst, xdp, exclude_ifindex))
+ continue;
+
+ /* we only need n-1 clones; last_dst enqueued below */
+ if (!last_dst) {
+ last_dst = dst;
+ continue;
+ }
+
+ err = dev_map_enqueue_clone(last_dst, dev_rx, xdpf);
+ if (err)
+ return err;
+
+ last_dst = dst;
+ }
+ }
}
- return __xdp_enqueue(dev, xdp, dev_rx);
+
+ /* consume the last copy of the frame */
+ if (last_dst)
+ bq_enqueue(last_dst->dev, xdpf, dev_rx, last_dst->xdp_prog);
+ else
+ xdp_return_frame_rx_napi(xdpf); /* dtab is empty */
+
+ return 0;
}
int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb,
return 0;
}
+static int dev_map_redirect_clone(struct bpf_dtab_netdev *dst,
+ struct sk_buff *skb,
+ struct bpf_prog *xdp_prog)
+{
+ struct sk_buff *nskb;
+ int err;
+
+ nskb = skb_clone(skb, GFP_ATOMIC);
+ if (!nskb)
+ return -ENOMEM;
+
+ err = dev_map_generic_redirect(dst, nskb, xdp_prog);
+ if (unlikely(err)) {
+ consume_skb(nskb);
+ return err;
+ }
+
+ return 0;
+}
+
+int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb,
+ struct bpf_prog *xdp_prog, struct bpf_map *map,
+ bool exclude_ingress)
+{
+ struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
+ int exclude_ifindex = exclude_ingress ? dev->ifindex : 0;
+ struct bpf_dtab_netdev *dst, *last_dst = NULL;
+ struct hlist_head *head;
+ struct hlist_node *next;
+ unsigned int i;
+ int err;
+
+ if (map->map_type == BPF_MAP_TYPE_DEVMAP) {
+ for (i = 0; i < map->max_entries; i++) {
+ dst = READ_ONCE(dtab->netdev_map[i]);
+ if (!dst || dst->dev->ifindex == exclude_ifindex)
+ continue;
+
+ /* we only need n-1 clones; last_dst enqueued below */
+ if (!last_dst) {
+ last_dst = dst;
+ continue;
+ }
+
+ err = dev_map_redirect_clone(last_dst, skb, xdp_prog);
+ if (err)
+ return err;
+
+ last_dst = dst;
+ }
+ } else { /* BPF_MAP_TYPE_DEVMAP_HASH */
+ for (i = 0; i < dtab->n_buckets; i++) {
+ head = dev_map_index_hash(dtab, i);
+ hlist_for_each_entry_safe(dst, next, head, index_hlist) {
+ if (!dst || dst->dev->ifindex == exclude_ifindex)
+ continue;
+
+ /* we only need n-1 clones; last_dst enqueued below */
+ if (!last_dst) {
+ last_dst = dst;
+ continue;
+ }
+
+ err = dev_map_redirect_clone(last_dst, skb, xdp_prog);
+ if (err)
+ return err;
+
+ last_dst = dst;
+ }
+ }
+ }
+
+ /* consume the first skb and return */
+ if (last_dst)
+ return dev_map_generic_redirect(last_dst, skb, xdp_prog);
+
+ /* dtab is empty */
+ consume_skb(skb);
+ return 0;
+}
+
static void *dev_map_lookup_elem(struct bpf_map *map, void *key)
{
struct bpf_dtab_netdev *obj = __dev_map_lookup_elem(map, *(u32 *)key);
static int dev_map_redirect(struct bpf_map *map, u32 ifindex, u64 flags)
{
- return __bpf_xdp_redirect_map(map, ifindex, flags, __dev_map_lookup_elem);
+ return __bpf_xdp_redirect_map(map, ifindex, flags,
+ BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS,
+ __dev_map_lookup_elem);
}
static int dev_hash_map_redirect(struct bpf_map *map, u32 ifindex, u64 flags)
{
- return __bpf_xdp_redirect_map(map, ifindex, flags, __dev_map_hash_lookup_elem);
+ return __bpf_xdp_redirect_map(map, ifindex, flags,
+ BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS,
+ __dev_map_hash_lookup_elem);
}
static int dev_map_btf_id;
* events, kprobes and tracing to be invoked before the prior invocation
* from one of these contexts completed. sys_bpf() uses the same mechanism
* by pinning the task to the current CPU and incrementing the recursion
- * protection accross the map operation.
+ * protection across the map operation.
*
* This has subtle implications on PREEMPT_RT. PREEMPT_RT forbids certain
* operations like memory allocations (even with GFP_ATOMIC) from atomic
* contexts. This is required because even with GFP_ATOMIC the memory
- * allocator calls into code pathes which acquire locks with long held lock
+ * allocator calls into code paths which acquire locks with long held lock
* sections. To ensure the deterministic behaviour these locks are regular
* spinlocks, which are converted to 'sleepable' spinlocks on RT. The only
* true atomic contexts on an RT kernel are the low level hardware
rcu_read_unlock();
}
+static int __htab_map_lookup_and_delete_elem(struct bpf_map *map, void *key,
+ void *value, bool is_lru_map,
+ bool is_percpu, u64 flags)
+{
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
+ struct hlist_nulls_head *head;
+ unsigned long bflags;
+ struct htab_elem *l;
+ u32 hash, key_size;
+ struct bucket *b;
+ int ret;
+
+ key_size = map->key_size;
+
+ hash = htab_map_hash(key, key_size, htab->hashrnd);
+ b = __select_bucket(htab, hash);
+ head = &b->head;
+
+ ret = htab_lock_bucket(htab, b, hash, &bflags);
+ if (ret)
+ return ret;
+
+ l = lookup_elem_raw(head, hash, key, key_size);
+ if (!l) {
+ ret = -ENOENT;
+ } else {
+ if (is_percpu) {
+ u32 roundup_value_size = round_up(map->value_size, 8);
+ void __percpu *pptr;
+ int off = 0, cpu;
+
+ pptr = htab_elem_get_ptr(l, key_size);
+ for_each_possible_cpu(cpu) {
+ bpf_long_memcpy(value + off,
+ per_cpu_ptr(pptr, cpu),
+ roundup_value_size);
+ off += roundup_value_size;
+ }
+ } else {
+ u32 roundup_key_size = round_up(map->key_size, 8);
+
+ if (flags & BPF_F_LOCK)
+ copy_map_value_locked(map, value, l->key +
+ roundup_key_size,
+ true);
+ else
+ copy_map_value(map, value, l->key +
+ roundup_key_size);
+ check_and_init_map_lock(map, value);
+ }
+
+ hlist_nulls_del_rcu(&l->hash_node);
+ if (!is_lru_map)
+ free_htab_elem(htab, l);
+ }
+
+ htab_unlock_bucket(htab, b, hash, bflags);
+
+ if (is_lru_map && l)
+ bpf_lru_push_free(&htab->lru, &l->lru_node);
+
+ return ret;
+}
+
+static int htab_map_lookup_and_delete_elem(struct bpf_map *map, void *key,
+ void *value, u64 flags)
+{
+ return __htab_map_lookup_and_delete_elem(map, key, value, false, false,
+ flags);
+}
+
+static int htab_percpu_map_lookup_and_delete_elem(struct bpf_map *map,
+ void *key, void *value,
+ u64 flags)
+{
+ return __htab_map_lookup_and_delete_elem(map, key, value, false, true,
+ flags);
+}
+
+static int htab_lru_map_lookup_and_delete_elem(struct bpf_map *map, void *key,
+ void *value, u64 flags)
+{
+ return __htab_map_lookup_and_delete_elem(map, key, value, true, false,
+ flags);
+}
+
+static int htab_lru_percpu_map_lookup_and_delete_elem(struct bpf_map *map,
+ void *key, void *value,
+ u64 flags)
+{
+ return __htab_map_lookup_and_delete_elem(map, key, value, true, true,
+ flags);
+}
+
static int
__htab_map_lookup_and_delete_batch(struct bpf_map *map,
const union bpf_attr *attr,
.map_free = htab_map_free,
.map_get_next_key = htab_map_get_next_key,
.map_lookup_elem = htab_map_lookup_elem,
+ .map_lookup_and_delete_elem = htab_map_lookup_and_delete_elem,
.map_update_elem = htab_map_update_elem,
.map_delete_elem = htab_map_delete_elem,
.map_gen_lookup = htab_map_gen_lookup,
.map_free = htab_map_free,
.map_get_next_key = htab_map_get_next_key,
.map_lookup_elem = htab_lru_map_lookup_elem,
+ .map_lookup_and_delete_elem = htab_lru_map_lookup_and_delete_elem,
.map_lookup_elem_sys_only = htab_lru_map_lookup_elem_sys,
.map_update_elem = htab_lru_map_update_elem,
.map_delete_elem = htab_lru_map_delete_elem,
.map_free = htab_map_free,
.map_get_next_key = htab_map_get_next_key,
.map_lookup_elem = htab_percpu_map_lookup_elem,
+ .map_lookup_and_delete_elem = htab_percpu_map_lookup_and_delete_elem,
.map_update_elem = htab_percpu_map_update_elem,
.map_delete_elem = htab_map_delete_elem,
.map_seq_show_elem = htab_percpu_map_seq_show_elem,
.map_free = htab_map_free,
.map_get_next_key = htab_map_get_next_key,
.map_lookup_elem = htab_lru_percpu_map_lookup_elem,
+ .map_lookup_and_delete_elem = htab_lru_percpu_map_lookup_and_delete_elem,
.map_update_elem = htab_lru_percpu_map_update_elem,
.map_delete_elem = htab_lru_map_delete_elem,
.map_seq_show_elem = htab_percpu_map_seq_show_elem,
#include <linux/jiffies.h>
#include <linux/pid_namespace.h>
#include <linux/proc_ns.h>
+#include <linux/security.h>
#include "../../lib/kstrtox.h"
return -EINVAL;
}
-/* Per-cpu temp buffers which can be used by printf-like helpers for %s or %p
+/* Per-cpu temp buffers used by printf-like helpers to store the bprintf binary
+ * arguments representation.
*/
-#define MAX_PRINTF_BUF_LEN 512
+#define MAX_BPRINTF_BUF_LEN 512
-struct bpf_printf_buf {
- char tmp_buf[MAX_PRINTF_BUF_LEN];
+/* Support executing three nested bprintf helper calls on a given CPU */
+#define MAX_BPRINTF_NEST_LEVEL 3
+struct bpf_bprintf_buffers {
+ char tmp_bufs[MAX_BPRINTF_NEST_LEVEL][MAX_BPRINTF_BUF_LEN];
};
-static DEFINE_PER_CPU(struct bpf_printf_buf, bpf_printf_buf);
-static DEFINE_PER_CPU(int, bpf_printf_buf_used);
+static DEFINE_PER_CPU(struct bpf_bprintf_buffers, bpf_bprintf_bufs);
+static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);
static int try_get_fmt_tmp_buf(char **tmp_buf)
{
- struct bpf_printf_buf *bufs;
- int used;
+ struct bpf_bprintf_buffers *bufs;
+ int nest_level;
preempt_disable();
- used = this_cpu_inc_return(bpf_printf_buf_used);
- if (WARN_ON_ONCE(used > 1)) {
- this_cpu_dec(bpf_printf_buf_used);
+ nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
+ if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
+ this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
return -EBUSY;
}
- bufs = this_cpu_ptr(&bpf_printf_buf);
- *tmp_buf = bufs->tmp_buf;
+ bufs = this_cpu_ptr(&bpf_bprintf_bufs);
+ *tmp_buf = bufs->tmp_bufs[nest_level - 1];
return 0;
}
void bpf_bprintf_cleanup(void)
{
- if (this_cpu_read(bpf_printf_buf_used)) {
- this_cpu_dec(bpf_printf_buf_used);
+ if (this_cpu_read(bpf_bprintf_nest_level)) {
+ this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
}
}
if (num_args && try_get_fmt_tmp_buf(&tmp_buf))
return -EBUSY;
- tmp_buf_end = tmp_buf + MAX_PRINTF_BUF_LEN;
+ tmp_buf_end = tmp_buf + MAX_BPRINTF_BUF_LEN;
*bin_args = (u32 *)tmp_buf;
}
case BPF_FUNC_probe_read_user:
return &bpf_probe_read_user_proto;
case BPF_FUNC_probe_read_kernel:
- return &bpf_probe_read_kernel_proto;
+ return security_locked_down(LOCKDOWN_BPF_READ) < 0 ?
+ NULL : &bpf_probe_read_kernel_proto;
case BPF_FUNC_probe_read_user_str:
return &bpf_probe_read_user_str_proto;
case BPF_FUNC_probe_read_kernel_str:
- return &bpf_probe_read_kernel_str_proto;
+ return security_locked_down(LOCKDOWN_BPF_READ) < 0 ?
+ NULL : &bpf_probe_read_kernel_str_proto;
case BPF_FUNC_snprintf_btf:
return &bpf_snprintf_btf_proto;
case BPF_FUNC_snprintf:
/* Copyright (c) 2020 Facebook */
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
/*
* ops->map_*_elem() will not be able to access this
* array now. Hence, this function only races with
- * bpf_sk_reuseport_detach() which was triggerred by
+ * bpf_sk_reuseport_detach() which was triggered by
* close() or disconnect().
*
* This function and bpf_sk_reuseport_detach() are
return -ENOTSUPP;
}
-static size_t bpf_ringbuf_mmap_page_cnt(const struct bpf_ringbuf *rb)
-{
- size_t data_pages = (rb->mask + 1) >> PAGE_SHIFT;
-
- /* consumer page + producer page + 2 x data pages */
- return RINGBUF_POS_PAGES + 2 * data_pages;
-}
-
static int ringbuf_map_mmap(struct bpf_map *map, struct vm_area_struct *vma)
{
struct bpf_ringbuf_map *rb_map;
- size_t mmap_sz;
rb_map = container_of(map, struct bpf_ringbuf_map, map);
- mmap_sz = bpf_ringbuf_mmap_page_cnt(rb_map->rb) << PAGE_SHIFT;
-
- if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) > mmap_sz)
- return -EINVAL;
+ if (vma->vm_flags & VM_WRITE) {
+ /* allow writable mapping for the consumer_pos only */
+ if (vma->vm_pgoff != 0 || vma->vm_end - vma->vm_start != PAGE_SIZE)
+ return -EPERM;
+ } else {
+ vma->vm_flags &= ~VM_MAYWRITE;
+ }
+ /* remap_vmalloc_range() checks size and offset constraints */
return remap_vmalloc_range(vma, rb_map->rb,
vma->vm_pgoff + RINGBUF_PGOFF);
}
return NULL;
len = round_up(size + BPF_RINGBUF_HDR_SZ, 8);
+ if (len > rb->mask + 1)
+ return NULL;
+
cons_pos = smp_load_acquire(&rb->consumer_pos);
if (in_nmi()) {
static DEFINE_IDR(link_idr);
static DEFINE_SPINLOCK(link_idr_lock);
-int sysctl_unprivileged_bpf_disabled __read_mostly;
+int sysctl_unprivileged_bpf_disabled __read_mostly =
+ IS_BUILTIN(CONFIG_BPF_UNPRIV_DEFAULT_OFF) ? 2 : 0;
static const struct bpf_map_ops * const bpf_map_types[] = {
#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type)
* copy_from_user() call. However, this is not a concern since this function is
* meant to be a future-proofing of bits.
*/
-int bpf_check_uarg_tail_zero(void __user *uaddr,
+int bpf_check_uarg_tail_zero(bpfptr_t uaddr,
size_t expected_size,
size_t actual_size)
{
- unsigned char __user *addr = uaddr + expected_size;
int res;
if (unlikely(actual_size > PAGE_SIZE)) /* silly large */
if (actual_size <= expected_size)
return 0;
- res = check_zeroed_user(addr, actual_size - expected_size);
+ if (uaddr.is_kernel)
+ res = memchr_inv(uaddr.kernel + expected_size, 0,
+ actual_size - expected_size) == NULL;
+ else
+ res = check_zeroed_user(uaddr.user + expected_size,
+ actual_size - expected_size);
if (res < 0)
return res;
return res ? 0 : -E2BIG;
return NULL;
}
+static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size)
+{
+ if (key_size)
+ return memdup_bpfptr(ukey, key_size);
+
+ if (!bpfptr_is_null(ukey))
+ return ERR_PTR(-EINVAL);
+
+ return NULL;
+}
+
/* last field in 'union bpf_attr' used by this command */
#define BPF_MAP_LOOKUP_ELEM_LAST_FIELD flags
#define BPF_MAP_UPDATE_ELEM_LAST_FIELD flags
-static int map_update_elem(union bpf_attr *attr)
+static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)
{
- void __user *ukey = u64_to_user_ptr(attr->key);
- void __user *uvalue = u64_to_user_ptr(attr->value);
+ bpfptr_t ukey = make_bpfptr(attr->key, uattr.is_kernel);
+ bpfptr_t uvalue = make_bpfptr(attr->value, uattr.is_kernel);
int ufd = attr->map_fd;
struct bpf_map *map;
void *key, *value;
goto err_put;
}
- key = __bpf_copy_key(ukey, map->key_size);
+ key = ___bpf_copy_key(ukey, map->key_size);
if (IS_ERR(key)) {
err = PTR_ERR(key);
goto err_put;
goto free_key;
err = -EFAULT;
- if (copy_from_user(value, uvalue, value_size) != 0)
+ if (copy_from_bpfptr(value, uvalue, value_size) != 0)
goto free_value;
err = bpf_map_update_value(map, f, key, value, attr->flags);
return err;
}
-#define BPF_MAP_LOOKUP_AND_DELETE_ELEM_LAST_FIELD value
+#define BPF_MAP_LOOKUP_AND_DELETE_ELEM_LAST_FIELD flags
static int map_lookup_and_delete_elem(union bpf_attr *attr)
{
if (CHECK_ATTR(BPF_MAP_LOOKUP_AND_DELETE_ELEM))
return -EINVAL;
+ if (attr->flags & ~BPF_F_LOCK)
+ return -EINVAL;
+
f = fdget(ufd);
map = __bpf_map_get(f);
if (IS_ERR(map))
goto err_put;
}
+ if (attr->flags &&
+ (map->map_type == BPF_MAP_TYPE_QUEUE ||
+ map->map_type == BPF_MAP_TYPE_STACK)) {
+ err = -EINVAL;
+ goto err_put;
+ }
+
+ if ((attr->flags & BPF_F_LOCK) &&
+ !map_value_has_spin_lock(map)) {
+ err = -EINVAL;
+ goto err_put;
+ }
+
key = __bpf_copy_key(ukey, map->key_size);
if (IS_ERR(key)) {
err = PTR_ERR(key);
goto err_put;
}
- value_size = map->value_size;
+ value_size = bpf_map_value_size(map);
err = -ENOMEM;
value = kmalloc(value_size, GFP_USER | __GFP_NOWARN);
if (!value)
goto free_key;
+ err = -ENOTSUPP;
if (map->map_type == BPF_MAP_TYPE_QUEUE ||
map->map_type == BPF_MAP_TYPE_STACK) {
err = map->ops->map_pop_elem(map, value);
- } else {
- err = -ENOTSUPP;
+ } else if (map->map_type == BPF_MAP_TYPE_HASH ||
+ map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
+ map->map_type == BPF_MAP_TYPE_LRU_HASH ||
+ map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) {
+ if (!bpf_map_is_dev_bound(map)) {
+ bpf_disable_instrumentation();
+ rcu_read_lock();
+ err = map->ops->map_lookup_and_delete_elem(map, key, value, attr->flags);
+ rcu_read_unlock();
+ bpf_enable_instrumentation();
+ }
}
if (err)
attr->expected_attach_type =
BPF_CGROUP_INET_SOCK_CREATE;
break;
+ case BPF_PROG_TYPE_SK_REUSEPORT:
+ if (!attr->expected_attach_type)
+ attr->expected_attach_type =
+ BPF_SK_REUSEPORT_SELECT;
+ break;
}
}
if (expected_attach_type == BPF_SK_LOOKUP)
return 0;
return -EINVAL;
+ case BPF_PROG_TYPE_SK_REUSEPORT:
+ switch (expected_attach_type) {
+ case BPF_SK_REUSEPORT_SELECT:
+ case BPF_SK_REUSEPORT_SELECT_OR_MIGRATE:
+ return 0;
+ default:
+ return -EINVAL;
+ }
+ case BPF_PROG_TYPE_SYSCALL:
case BPF_PROG_TYPE_EXT:
if (expected_attach_type)
return -EINVAL;
}
/* last field in 'union bpf_attr' used by this command */
-#define BPF_PROG_LOAD_LAST_FIELD attach_prog_fd
+#define BPF_PROG_LOAD_LAST_FIELD fd_array
-static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr)
+static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr)
{
enum bpf_prog_type type = attr->prog_type;
struct bpf_prog *prog, *dst_prog = NULL;
return -EPERM;
/* copy eBPF program license from user space */
- if (strncpy_from_user(license, u64_to_user_ptr(attr->license),
- sizeof(license) - 1) < 0)
+ if (strncpy_from_bpfptr(license,
+ make_bpfptr(attr->license, uattr.is_kernel),
+ sizeof(license) - 1) < 0)
return -EFAULT;
license[sizeof(license) - 1] = 0;
prog->len = attr->insn_cnt;
err = -EFAULT;
- if (copy_from_user(prog->insns, u64_to_user_ptr(attr->insns),
- bpf_prog_insn_size(prog)) != 0)
+ if (copy_from_bpfptr(prog->insns,
+ make_bpfptr(attr->insns, uattr.is_kernel),
+ bpf_prog_insn_size(prog)) != 0)
goto free_prog_sec;
prog->orig_prog = NULL;
u32 ulen;
int err;
- err = bpf_check_uarg_tail_zero(uinfo, sizeof(info), info_len);
+ err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len);
if (err)
return err;
info_len = min_t(u32, sizeof(info), info_len);
u32 info_len = attr->info.info_len;
int err;
- err = bpf_check_uarg_tail_zero(uinfo, sizeof(info), info_len);
+ err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len);
if (err)
return err;
info_len = min_t(u32, sizeof(info), info_len);
u32 info_len = attr->info.info_len;
int err;
- err = bpf_check_uarg_tail_zero(uinfo, sizeof(*uinfo), info_len);
+ err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(*uinfo), info_len);
if (err)
return err;
u32 info_len = attr->info.info_len;
int err;
- err = bpf_check_uarg_tail_zero(uinfo, sizeof(info), info_len);
+ err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len);
if (err)
return err;
info_len = min_t(u32, sizeof(info), info_len);
#define BPF_BTF_LOAD_LAST_FIELD btf_log_level
-static int bpf_btf_load(const union bpf_attr *attr)
+static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr)
{
if (CHECK_ATTR(BPF_BTF_LOAD))
return -EINVAL;
if (!bpf_capable())
return -EPERM;
- return btf_new_fd(attr);
+ return btf_new_fd(attr, uattr);
}
#define BPF_BTF_GET_FD_BY_ID_LAST_FIELD btf_id
return err;
}
-static int tracing_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
+static int tracing_bpf_link_attach(const union bpf_attr *attr, bpfptr_t uattr,
+ struct bpf_prog *prog)
{
if (attr->link_create.attach_type != prog->expected_attach_type)
return -EINVAL;
if (prog->expected_attach_type == BPF_TRACE_ITER)
- return bpf_iter_link_attach(attr, prog);
+ return bpf_iter_link_attach(attr, uattr, prog);
else if (prog->type == BPF_PROG_TYPE_EXT)
return bpf_tracing_prog_attach(prog,
attr->link_create.target_fd,
}
#define BPF_LINK_CREATE_LAST_FIELD link_create.iter_info_len
-static int link_create(union bpf_attr *attr)
+static int link_create(union bpf_attr *attr, bpfptr_t uattr)
{
enum bpf_prog_type ptype;
struct bpf_prog *prog;
goto out;
if (prog->type == BPF_PROG_TYPE_EXT) {
- ret = tracing_bpf_link_attach(attr, prog);
+ ret = tracing_bpf_link_attach(attr, uattr, prog);
goto out;
}
ret = cgroup_bpf_link_attach(attr, prog);
break;
case BPF_PROG_TYPE_TRACING:
- ret = tracing_bpf_link_attach(attr, prog);
+ ret = tracing_bpf_link_attach(attr, uattr, prog);
break;
case BPF_PROG_TYPE_FLOW_DISSECTOR:
case BPF_PROG_TYPE_SK_LOOKUP:
return ret;
}
-SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
+static int __sys_bpf(int cmd, bpfptr_t uattr, unsigned int size)
{
union bpf_attr attr;
int err;
/* copy attributes from user space, may be less than sizeof(bpf_attr) */
memset(&attr, 0, sizeof(attr));
- if (copy_from_user(&attr, uattr, size) != 0)
+ if (copy_from_bpfptr(&attr, uattr, size) != 0)
return -EFAULT;
err = security_bpf(cmd, &attr, size);
err = map_lookup_elem(&attr);
break;
case BPF_MAP_UPDATE_ELEM:
- err = map_update_elem(&attr);
+ err = map_update_elem(&attr, uattr);
break;
case BPF_MAP_DELETE_ELEM:
err = map_delete_elem(&attr);
err = bpf_prog_detach(&attr);
break;
case BPF_PROG_QUERY:
- err = bpf_prog_query(&attr, uattr);
+ err = bpf_prog_query(&attr, uattr.user);
break;
case BPF_PROG_TEST_RUN:
- err = bpf_prog_test_run(&attr, uattr);
+ err = bpf_prog_test_run(&attr, uattr.user);
break;
case BPF_PROG_GET_NEXT_ID:
- err = bpf_obj_get_next_id(&attr, uattr,
+ err = bpf_obj_get_next_id(&attr, uattr.user,
&prog_idr, &prog_idr_lock);
break;
case BPF_MAP_GET_NEXT_ID:
- err = bpf_obj_get_next_id(&attr, uattr,
+ err = bpf_obj_get_next_id(&attr, uattr.user,
&map_idr, &map_idr_lock);
break;
case BPF_BTF_GET_NEXT_ID:
- err = bpf_obj_get_next_id(&attr, uattr,
+ err = bpf_obj_get_next_id(&attr, uattr.user,
&btf_idr, &btf_idr_lock);
break;
case BPF_PROG_GET_FD_BY_ID:
err = bpf_map_get_fd_by_id(&attr);
break;
case BPF_OBJ_GET_INFO_BY_FD:
- err = bpf_obj_get_info_by_fd(&attr, uattr);
+ err = bpf_obj_get_info_by_fd(&attr, uattr.user);
break;
case BPF_RAW_TRACEPOINT_OPEN:
err = bpf_raw_tracepoint_open(&attr);
break;
case BPF_BTF_LOAD:
- err = bpf_btf_load(&attr);
+ err = bpf_btf_load(&attr, uattr);
break;
case BPF_BTF_GET_FD_BY_ID:
err = bpf_btf_get_fd_by_id(&attr);
break;
case BPF_TASK_FD_QUERY:
- err = bpf_task_fd_query(&attr, uattr);
+ err = bpf_task_fd_query(&attr, uattr.user);
break;
case BPF_MAP_LOOKUP_AND_DELETE_ELEM:
err = map_lookup_and_delete_elem(&attr);
break;
case BPF_MAP_LOOKUP_BATCH:
- err = bpf_map_do_batch(&attr, uattr, BPF_MAP_LOOKUP_BATCH);
+ err = bpf_map_do_batch(&attr, uattr.user, BPF_MAP_LOOKUP_BATCH);
break;
case BPF_MAP_LOOKUP_AND_DELETE_BATCH:
- err = bpf_map_do_batch(&attr, uattr,
+ err = bpf_map_do_batch(&attr, uattr.user,
BPF_MAP_LOOKUP_AND_DELETE_BATCH);
break;
case BPF_MAP_UPDATE_BATCH:
- err = bpf_map_do_batch(&attr, uattr, BPF_MAP_UPDATE_BATCH);
+ err = bpf_map_do_batch(&attr, uattr.user, BPF_MAP_UPDATE_BATCH);
break;
case BPF_MAP_DELETE_BATCH:
- err = bpf_map_do_batch(&attr, uattr, BPF_MAP_DELETE_BATCH);
+ err = bpf_map_do_batch(&attr, uattr.user, BPF_MAP_DELETE_BATCH);
break;
case BPF_LINK_CREATE:
- err = link_create(&attr);
+ err = link_create(&attr, uattr);
break;
case BPF_LINK_UPDATE:
err = link_update(&attr);
err = bpf_link_get_fd_by_id(&attr);
break;
case BPF_LINK_GET_NEXT_ID:
- err = bpf_obj_get_next_id(&attr, uattr,
+ err = bpf_obj_get_next_id(&attr, uattr.user,
&link_idr, &link_idr_lock);
break;
case BPF_ENABLE_STATS:
return err;
}
+
+SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
+{
+ return __sys_bpf(cmd, USER_BPFPTR(uattr), size);
+}
+
+static bool syscall_prog_is_valid_access(int off, int size,
+ enum bpf_access_type type,
+ const struct bpf_prog *prog,
+ struct bpf_insn_access_aux *info)
+{
+ if (off < 0 || off >= U16_MAX)
+ return false;
+ if (off % size != 0)
+ return false;
+ return true;
+}
+
+BPF_CALL_3(bpf_sys_bpf, int, cmd, void *, attr, u32, attr_size)
+{
+ switch (cmd) {
+ case BPF_MAP_CREATE:
+ case BPF_MAP_UPDATE_ELEM:
+ case BPF_MAP_FREEZE:
+ case BPF_PROG_LOAD:
+ case BPF_BTF_LOAD:
+ break;
+ /* case BPF_PROG_TEST_RUN:
+ * is not part of this list to prevent recursive test_run
+ */
+ default:
+ return -EINVAL;
+ }
+ return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size);
+}
+
+static const struct bpf_func_proto bpf_sys_bpf_proto = {
+ .func = bpf_sys_bpf,
+ .gpl_only = false,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_ANYTHING,
+ .arg2_type = ARG_PTR_TO_MEM,
+ .arg3_type = ARG_CONST_SIZE,
+};
+
+const struct bpf_func_proto * __weak
+tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
+{
+ return bpf_base_func_proto(func_id);
+}
+
+BPF_CALL_1(bpf_sys_close, u32, fd)
+{
+ /* When bpf program calls this helper there should not be
+ * an fdget() without matching completed fdput().
+ * This helper is allowed in the following callchain only:
+ * sys_bpf->prog_test_run->bpf_prog->bpf_sys_close
+ */
+ return close_fd(fd);
+}
+
+static const struct bpf_func_proto bpf_sys_close_proto = {
+ .func = bpf_sys_close,
+ .gpl_only = false,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_ANYTHING,
+};
+
+static const struct bpf_func_proto *
+syscall_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
+{
+ switch (func_id) {
+ case BPF_FUNC_sys_bpf:
+ return &bpf_sys_bpf_proto;
+ case BPF_FUNC_btf_find_by_name_kind:
+ return &bpf_btf_find_by_name_kind_proto;
+ case BPF_FUNC_sys_close:
+ return &bpf_sys_close_proto;
+ default:
+ return tracing_prog_func_proto(func_id, prog);
+ }
+}
+
+const struct bpf_verifier_ops bpf_syscall_verifier_ops = {
+ .get_func_proto = syscall_prog_func_proto,
+ .is_valid_access = syscall_prog_is_valid_access,
+};
+
+const struct bpf_prog_ops bpf_syscall_prog_ops = {
+ .test_run = bpf_prog_test_run_syscall,
+};
return TNUM(v & ~mu, mu);
}
-/* half-multiply add: acc += (unknown * mask * value).
- * An intermediate step in the multiply algorithm.
+/* Generate partial products by multiplying each bit in the multiplier (tnum a)
+ * with the multiplicand (tnum b), and add the partial products after
+ * appropriately bit-shifting them. Instead of directly performing tnum addition
+ * on the generated partial products, equivalenty, decompose each partial
+ * product into two tnums, consisting of the value-sum (acc_v) and the
+ * mask-sum (acc_m) and then perform tnum addition on them. The following paper
+ * explains the algorithm in more detail: https://arxiv.org/abs/2105.05398.
*/
-static struct tnum hma(struct tnum acc, u64 value, u64 mask)
-{
- while (mask) {
- if (mask & 1)
- acc = tnum_add(acc, TNUM(0, value));
- mask >>= 1;
- value <<= 1;
- }
- return acc;
-}
-
struct tnum tnum_mul(struct tnum a, struct tnum b)
{
- struct tnum acc;
- u64 pi;
-
- pi = a.value * b.value;
- acc = hma(TNUM(pi, 0), a.mask, b.mask | b.value);
- return hma(acc, b.mask, a.value);
+ u64 acc_v = a.value * b.value;
+ struct tnum acc_m = TNUM(0, 0);
+
+ while (a.value || a.mask) {
+ /* LSB of tnum a is a certain 1 */
+ if (a.value & 1)
+ acc_m = tnum_add(acc_m, TNUM(0, b.mask));
+ /* LSB of tnum a is uncertain */
+ else if (a.mask & 1)
+ acc_m = tnum_add(acc_m, TNUM(0, b.value | b.mask));
+ /* Note: no case for LSB is certain 0 */
+ a = tnum_rshift(a, 1);
+ b = tnum_lshift(b, 1);
+ }
+ return tnum_add(TNUM(acc_v, 0), acc_m);
}
/* Note that if a and b disagree - i.e. one has a 'known 1' where the other has
* __bpf_prog_enter returns:
* 0 - skip execution of the bpf prog
* 1 - execute bpf prog
- * [2..MAX_U64] - excute bpf prog and record execution time.
+ * [2..MAX_U64] - execute bpf prog and record execution time.
* This is start time.
*/
u64 notrace __bpf_prog_enter(struct bpf_prog *prog)
* - unreachable insns exist (shouldn't be a forest. program = one function)
* - out of bounds or malformed jumps
* The second pass is all possible path descent from the 1st insn.
- * Since it's analyzing all pathes through the program, the length of the
+ * Since it's analyzing all paths through the program, the length of the
* analysis is limited to 64k insn, which may be hit even if total number of
* insn is less then 4K, but there are too many branches that change stack/regs.
* Number of 'branches to be analyzed' is limited to 1k
* If it's ok, then verifier allows this BPF_CALL insn and looks at
* .ret_type which is RET_PTR_TO_MAP_VALUE_OR_NULL, so it sets
* R0->type = PTR_TO_MAP_VALUE_OR_NULL which means bpf_map_lookup_elem() function
- * returns ether pointer to map value or NULL.
+ * returns either pointer to map value or NULL.
*
* When type PTR_TO_MAP_VALUE_OR_NULL passes through 'if (reg != 0) goto +off'
* insn, the register holding that pointer in the true branch changes state to
verbose(env, "\n");
}
-#define COPY_STATE_FN(NAME, COUNT, FIELD, SIZE) \
-static int copy_##NAME##_state(struct bpf_func_state *dst, \
- const struct bpf_func_state *src) \
-{ \
- if (!src->FIELD) \
- return 0; \
- if (WARN_ON_ONCE(dst->COUNT < src->COUNT)) { \
- /* internal bug, make state invalid to reject the program */ \
- memset(dst, 0, sizeof(*dst)); \
- return -EFAULT; \
- } \
- memcpy(dst->FIELD, src->FIELD, \
- sizeof(*src->FIELD) * (src->COUNT / SIZE)); \
- return 0; \
-}
-/* copy_reference_state() */
-COPY_STATE_FN(reference, acquired_refs, refs, 1)
-/* copy_stack_state() */
-COPY_STATE_FN(stack, allocated_stack, stack, BPF_REG_SIZE)
-#undef COPY_STATE_FN
-
-#define REALLOC_STATE_FN(NAME, COUNT, FIELD, SIZE) \
-static int realloc_##NAME##_state(struct bpf_func_state *state, int size, \
- bool copy_old) \
-{ \
- u32 old_size = state->COUNT; \
- struct bpf_##NAME##_state *new_##FIELD; \
- int slot = size / SIZE; \
- \
- if (size <= old_size || !size) { \
- if (copy_old) \
- return 0; \
- state->COUNT = slot * SIZE; \
- if (!size && old_size) { \
- kfree(state->FIELD); \
- state->FIELD = NULL; \
- } \
- return 0; \
- } \
- new_##FIELD = kmalloc_array(slot, sizeof(struct bpf_##NAME##_state), \
- GFP_KERNEL); \
- if (!new_##FIELD) \
- return -ENOMEM; \
- if (copy_old) { \
- if (state->FIELD) \
- memcpy(new_##FIELD, state->FIELD, \
- sizeof(*new_##FIELD) * (old_size / SIZE)); \
- memset(new_##FIELD + old_size / SIZE, 0, \
- sizeof(*new_##FIELD) * (size - old_size) / SIZE); \
- } \
- state->COUNT = slot * SIZE; \
- kfree(state->FIELD); \
- state->FIELD = new_##FIELD; \
- return 0; \
-}
-/* realloc_reference_state() */
-REALLOC_STATE_FN(reference, acquired_refs, refs, 1)
-/* realloc_stack_state() */
-REALLOC_STATE_FN(stack, allocated_stack, stack, BPF_REG_SIZE)
-#undef REALLOC_STATE_FN
-
-/* do_check() starts with zero-sized stack in struct bpf_verifier_state to
- * make it consume minimal amount of memory. check_stack_write() access from
- * the program calls into realloc_func_state() to grow the stack size.
- * Note there is a non-zero 'parent' pointer inside bpf_verifier_state
- * which realloc_stack_state() copies over. It points to previous
- * bpf_verifier_state which is never reallocated.
+/* copy array src of length n * size bytes to dst. dst is reallocated if it's too
+ * small to hold src. This is different from krealloc since we don't want to preserve
+ * the contents of dst.
+ *
+ * Leaves dst untouched if src is NULL or length is zero. Returns NULL if memory could
+ * not be allocated.
*/
-static int realloc_func_state(struct bpf_func_state *state, int stack_size,
- int refs_size, bool copy_old)
+static void *copy_array(void *dst, const void *src, size_t n, size_t size, gfp_t flags)
{
- int err = realloc_reference_state(state, refs_size, copy_old);
- if (err)
- return err;
- return realloc_stack_state(state, stack_size, copy_old);
+ size_t bytes;
+
+ if (ZERO_OR_NULL_PTR(src))
+ goto out;
+
+ if (unlikely(check_mul_overflow(n, size, &bytes)))
+ return NULL;
+
+ if (ksize(dst) < bytes) {
+ kfree(dst);
+ dst = kmalloc_track_caller(bytes, flags);
+ if (!dst)
+ return NULL;
+ }
+
+ memcpy(dst, src, bytes);
+out:
+ return dst ? dst : ZERO_SIZE_PTR;
+}
+
+/* resize an array from old_n items to new_n items. the array is reallocated if it's too
+ * small to hold new_n items. new items are zeroed out if the array grows.
+ *
+ * Contrary to krealloc_array, does not free arr if new_n is zero.
+ */
+static void *realloc_array(void *arr, size_t old_n, size_t new_n, size_t size)
+{
+ if (!new_n || old_n == new_n)
+ goto out;
+
+ arr = krealloc_array(arr, new_n, size, GFP_KERNEL);
+ if (!arr)
+ return NULL;
+
+ if (new_n > old_n)
+ memset(arr + old_n * size, 0, (new_n - old_n) * size);
+
+out:
+ return arr ? arr : ZERO_SIZE_PTR;
+}
+
+static int copy_reference_state(struct bpf_func_state *dst, const struct bpf_func_state *src)
+{
+ dst->refs = copy_array(dst->refs, src->refs, src->acquired_refs,
+ sizeof(struct bpf_reference_state), GFP_KERNEL);
+ if (!dst->refs)
+ return -ENOMEM;
+
+ dst->acquired_refs = src->acquired_refs;
+ return 0;
+}
+
+static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_state *src)
+{
+ size_t n = src->allocated_stack / BPF_REG_SIZE;
+
+ dst->stack = copy_array(dst->stack, src->stack, n, sizeof(struct bpf_stack_state),
+ GFP_KERNEL);
+ if (!dst->stack)
+ return -ENOMEM;
+
+ dst->allocated_stack = src->allocated_stack;
+ return 0;
+}
+
+static int resize_reference_state(struct bpf_func_state *state, size_t n)
+{
+ state->refs = realloc_array(state->refs, state->acquired_refs, n,
+ sizeof(struct bpf_reference_state));
+ if (!state->refs)
+ return -ENOMEM;
+
+ state->acquired_refs = n;
+ return 0;
+}
+
+static int grow_stack_state(struct bpf_func_state *state, int size)
+{
+ size_t old_n = state->allocated_stack / BPF_REG_SIZE, n = size / BPF_REG_SIZE;
+
+ if (old_n >= n)
+ return 0;
+
+ state->stack = realloc_array(state->stack, old_n, n, sizeof(struct bpf_stack_state));
+ if (!state->stack)
+ return -ENOMEM;
+
+ state->allocated_stack = size;
+ return 0;
}
/* Acquire a pointer id from the env and update the state->refs to include
int new_ofs = state->acquired_refs;
int id, err;
- err = realloc_reference_state(state, state->acquired_refs + 1, true);
+ err = resize_reference_state(state, state->acquired_refs + 1);
if (err)
return err;
id = ++env->id_gen;
return -EINVAL;
}
-static int transfer_reference_state(struct bpf_func_state *dst,
- struct bpf_func_state *src)
-{
- int err = realloc_reference_state(dst, src->acquired_refs, false);
- if (err)
- return err;
- err = copy_reference_state(dst, src);
- if (err)
- return err;
- return 0;
-}
-
static void free_func_state(struct bpf_func_state *state)
{
if (!state)
{
int err;
- err = realloc_func_state(dst, src->allocated_stack, src->acquired_refs,
- false);
- if (err)
- return err;
memcpy(dst, src, offsetof(struct bpf_func_state, acquired_refs));
err = copy_reference_state(dst, src);
if (err)
const struct bpf_verifier_state *src)
{
struct bpf_func_state *dst;
- u32 jmp_sz = sizeof(struct bpf_idx_pair) * src->jmp_history_cnt;
int i, err;
- if (dst_state->jmp_history_cnt < src->jmp_history_cnt) {
- kfree(dst_state->jmp_history);
- dst_state->jmp_history = kmalloc(jmp_sz, GFP_USER);
- if (!dst_state->jmp_history)
- return -ENOMEM;
- }
- memcpy(dst_state->jmp_history, src->jmp_history, jmp_sz);
+ dst_state->jmp_history = copy_array(dst_state->jmp_history, src->jmp_history,
+ src->jmp_history_cnt, sizeof(struct bpf_idx_pair),
+ GFP_USER);
+ if (!dst_state->jmp_history)
+ return -ENOMEM;
dst_state->jmp_history_cnt = src->jmp_history_cnt;
/* if dst has more stack frames then src frame, free them */
u32 dst_reg = env->prog->insnsi[insn_idx].dst_reg;
struct bpf_reg_state *reg = NULL;
- err = realloc_func_state(state, round_up(slot + 1, BPF_REG_SIZE),
- state->acquired_refs, true);
+ err = grow_stack_state(state, round_up(slot + 1, BPF_REG_SIZE));
if (err)
return err;
/* caller checked that off % size == 0 and -MAX_BPF_STACK <= off < 0,
if (dst_reg != BPF_REG_FP) {
/* The backtracking logic can only recognize explicit
* stack slot address like [fp - 8]. Other spill of
- * scalar via different register has to be conervative.
+ * scalar via different register has to be conservative.
* Backtrack from here and mark all registers as precise
* that contributed into 'reg' being a constant.
*/
if (value_reg && register_is_null(value_reg))
writing_zero = true;
- err = realloc_func_state(state, round_up(-min_off, BPF_REG_SIZE),
- state->acquired_refs, true);
+ err = grow_stack_state(state, round_up(-min_off, BPF_REG_SIZE));
if (err)
return err;
subprog /* subprog number within this prog */);
/* Transfer references to the callee */
- err = transfer_reference_state(callee, caller);
+ err = copy_reference_state(callee, caller);
if (err)
return err;
}
/* Transfer references to the caller */
- err = transfer_reference_state(caller, callee);
+ err = copy_reference_state(caller, callee);
if (err)
return err;
};
static int retrieve_ptr_limit(const struct bpf_reg_state *ptr_reg,
- const struct bpf_reg_state *off_reg,
- u32 *alu_limit, u8 opcode)
+ u32 *alu_limit, bool mask_to_left)
{
- bool off_is_neg = off_reg->smin_value < 0;
- bool mask_to_left = (opcode == BPF_ADD && off_is_neg) ||
- (opcode == BPF_SUB && !off_is_neg);
u32 max = 0, ptr_limit = 0;
- if (!tnum_is_const(off_reg->var_off) &&
- (off_reg->smin_value < 0) != (off_reg->smax_value < 0))
- return REASON_BOUNDS;
-
switch (ptr_reg->type) {
case PTR_TO_STACK:
/* Offset 0 is out-of-bounds, but acceptable start for the
return opcode == BPF_ADD || opcode == BPF_SUB;
}
+struct bpf_sanitize_info {
+ struct bpf_insn_aux_data aux;
+ bool mask_to_left;
+};
+
+static struct bpf_verifier_state *
+sanitize_speculative_path(struct bpf_verifier_env *env,
+ const struct bpf_insn *insn,
+ u32 next_idx, u32 curr_idx)
+{
+ struct bpf_verifier_state *branch;
+ struct bpf_reg_state *regs;
+
+ branch = push_stack(env, next_idx, curr_idx, true);
+ if (branch && insn) {
+ regs = branch->frame[branch->curframe]->regs;
+ if (BPF_SRC(insn->code) == BPF_K) {
+ mark_reg_unknown(env, regs, insn->dst_reg);
+ } else if (BPF_SRC(insn->code) == BPF_X) {
+ mark_reg_unknown(env, regs, insn->dst_reg);
+ mark_reg_unknown(env, regs, insn->src_reg);
+ }
+ }
+ return branch;
+}
+
static int sanitize_ptr_alu(struct bpf_verifier_env *env,
struct bpf_insn *insn,
const struct bpf_reg_state *ptr_reg,
const struct bpf_reg_state *off_reg,
struct bpf_reg_state *dst_reg,
- struct bpf_insn_aux_data *tmp_aux,
+ struct bpf_sanitize_info *info,
const bool commit_window)
{
- struct bpf_insn_aux_data *aux = commit_window ? cur_aux(env) : tmp_aux;
+ struct bpf_insn_aux_data *aux = commit_window ? cur_aux(env) : &info->aux;
struct bpf_verifier_state *vstate = env->cur_state;
bool off_is_imm = tnum_is_const(off_reg->var_off);
bool off_is_neg = off_reg->smin_value < 0;
if (vstate->speculative)
goto do_sim;
- err = retrieve_ptr_limit(ptr_reg, off_reg, &alu_limit, opcode);
+ if (!commit_window) {
+ if (!tnum_is_const(off_reg->var_off) &&
+ (off_reg->smin_value < 0) != (off_reg->smax_value < 0))
+ return REASON_BOUNDS;
+
+ info->mask_to_left = (opcode == BPF_ADD && off_is_neg) ||
+ (opcode == BPF_SUB && !off_is_neg);
+ }
+
+ err = retrieve_ptr_limit(ptr_reg, &alu_limit, info->mask_to_left);
if (err < 0)
return err;
/* In commit phase we narrow the masking window based on
* the observed pointer move after the simulated operation.
*/
- alu_state = tmp_aux->alu_state;
- alu_limit = abs(tmp_aux->alu_limit - alu_limit);
+ alu_state = info->aux.alu_state;
+ alu_limit = abs(info->aux.alu_limit - alu_limit);
} else {
alu_state = off_is_neg ? BPF_ALU_NEG_VALUE : 0;
alu_state |= off_is_imm ? BPF_ALU_IMMEDIATE : 0;
/* If we're in commit phase, we're done here given we already
* pushed the truncated dst_reg into the speculative verification
* stack.
+ *
+ * Also, when register is a known constant, we rewrite register-based
+ * operation to immediate-based, and thus do not need masking (and as
+ * a consequence, do not need to simulate the zero-truncation either).
*/
- if (commit_window)
+ if (commit_window || off_is_imm)
return 0;
/* Simulate and find potential out-of-bounds access under
tmp = *dst_reg;
*dst_reg = *ptr_reg;
}
- ret = push_stack(env, env->insn_idx + 1, env->insn_idx, true);
+ ret = sanitize_speculative_path(env, NULL, env->insn_idx + 1,
+ env->insn_idx);
if (!ptr_is_dst_reg && ret)
*dst_reg = tmp;
return !ret ? REASON_STACK : 0;
}
+static void sanitize_mark_insn_seen(struct bpf_verifier_env *env)
+{
+ struct bpf_verifier_state *vstate = env->cur_state;
+
+ /* If we simulate paths under speculation, we don't update the
+ * insn as 'seen' such that when we verify unreachable paths in
+ * the non-speculative domain, sanitize_dead_code() can still
+ * rewrite/sanitize them.
+ */
+ if (!vstate->speculative)
+ env->insn_aux_data[env->insn_idx].seen = env->pass_cnt;
+}
+
static int sanitize_err(struct bpf_verifier_env *env,
const struct bpf_insn *insn, int reason,
const struct bpf_reg_state *off_reg,
smin_ptr = ptr_reg->smin_value, smax_ptr = ptr_reg->smax_value;
u64 umin_val = off_reg->umin_value, umax_val = off_reg->umax_value,
umin_ptr = ptr_reg->umin_value, umax_ptr = ptr_reg->umax_value;
- struct bpf_insn_aux_data tmp_aux = {};
+ struct bpf_sanitize_info info = {};
u8 opcode = BPF_OP(insn->code);
u32 dst = insn->dst_reg;
int ret;
if (sanitize_needed(opcode)) {
ret = sanitize_ptr_alu(env, insn, ptr_reg, off_reg, dst_reg,
- &tmp_aux, false);
+ &info, false);
if (ret < 0)
return sanitize_err(env, insn, ret, off_reg, dst_reg);
}
return -EACCES;
if (sanitize_needed(opcode)) {
ret = sanitize_ptr_alu(env, insn, dst_reg, off_reg, dst_reg,
- &tmp_aux, true);
+ &info, true);
if (ret < 0)
return sanitize_err(env, insn, ret, off_reg, dst_reg);
}
s32 smin_val = src_reg->s32_min_value;
u32 umax_val = src_reg->u32_max_value;
- /* Assuming scalar64_min_max_and will be called so its safe
- * to skip updating register for known 32-bit case.
- */
- if (src_known && dst_known)
+ if (src_known && dst_known) {
+ __mark_reg32_known(dst_reg, var32_off.value);
return;
+ }
/* We get our minimum from the var_off, since that's inherently
* bitwise. Our maximum is the minimum of the operands' maxima.
dst_reg->s32_min_value = dst_reg->u32_min_value;
dst_reg->s32_max_value = dst_reg->u32_max_value;
}
-
}
static void scalar_min_max_and(struct bpf_reg_state *dst_reg,
s32 smin_val = src_reg->s32_min_value;
u32 umin_val = src_reg->u32_min_value;
- /* Assuming scalar64_min_max_or will be called so it is safe
- * to skip updating register for known case.
- */
- if (src_known && dst_known)
+ if (src_known && dst_known) {
+ __mark_reg32_known(dst_reg, var32_off.value);
return;
+ }
/* We get our maximum from the var_off, and our minimum is the
* maximum of the operands' minima
struct tnum var32_off = tnum_subreg(dst_reg->var_off);
s32 smin_val = src_reg->s32_min_value;
- /* Assuming scalar64_min_max_xor will be called so it is safe
- * to skip updating register for known case.
- */
- if (src_known && dst_known)
+ if (src_known && dst_known) {
+ __mark_reg32_known(dst_reg, var32_off.value);
return;
+ }
/* We get both minimum and maximum from the var32_off. */
dst_reg->u32_min_value = var32_off.value;
if (err)
return err;
}
+
if (pred == 1) {
- /* only follow the goto, ignore fall-through */
+ /* Only follow the goto, ignore fall-through. If needed, push
+ * the fall-through branch for simulation under speculative
+ * execution.
+ */
+ if (!env->bypass_spec_v1 &&
+ !sanitize_speculative_path(env, insn, *insn_idx + 1,
+ *insn_idx))
+ return -EFAULT;
*insn_idx += insn->off;
return 0;
} else if (pred == 0) {
- /* only follow fall-through branch, since
- * that's where the program will go
+ /* Only follow the fall-through branch, since that's where the
+ * program will go. If needed, push the goto branch for
+ * simulation under speculative execution.
*/
+ if (!env->bypass_spec_v1 &&
+ !sanitize_speculative_path(env, insn,
+ *insn_idx + insn->off + 1,
+ *insn_idx))
+ return -EFAULT;
return 0;
}
mark_reg_known_zero(env, regs, insn->dst_reg);
dst_reg->map_ptr = map;
- if (insn->src_reg == BPF_PSEUDO_MAP_VALUE) {
+ if (insn->src_reg == BPF_PSEUDO_MAP_VALUE ||
+ insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE) {
dst_reg->type = PTR_TO_MAP_VALUE;
dst_reg->off = aux->map_off;
if (map_value_has_spin_lock(map))
dst_reg->id = ++env->id_gen;
- } else if (insn->src_reg == BPF_PSEUDO_MAP_FD) {
+ } else if (insn->src_reg == BPF_PSEUDO_MAP_FD ||
+ insn->src_reg == BPF_PSEUDO_MAP_IDX) {
dst_reg->type = CONST_PTR_TO_MAP;
} else {
verbose(env, "bpf verifier is misconfigured\n");
!prog->aux->attach_func_proto->type)
return 0;
- /* eBPF calling convetion is such that R0 is used
+ /* eBPF calling convention is such that R0 is used
* to return the value from eBPF program.
* Make sure that it's readable at this time
* of bpf_exit, which means that program wrote
static int check_btf_func(struct bpf_verifier_env *env,
const union bpf_attr *attr,
- union bpf_attr __user *uattr)
+ bpfptr_t uattr)
{
const struct btf_type *type, *func_proto, *ret_type;
u32 i, nfuncs, urec_size, min_size;
struct bpf_func_info_aux *info_aux = NULL;
struct bpf_prog *prog;
const struct btf *btf;
- void __user *urecord;
+ bpfptr_t urecord;
u32 prev_offset = 0;
bool scalar_return;
int ret = -ENOMEM;
prog = env->prog;
btf = prog->aux->btf;
- urecord = u64_to_user_ptr(attr->func_info);
+ urecord = make_bpfptr(attr->func_info, uattr.is_kernel);
min_size = min_t(u32, krec_size, urec_size);
krecord = kvcalloc(nfuncs, krec_size, GFP_KERNEL | __GFP_NOWARN);
/* set the size kernel expects so loader can zero
* out the rest of the record.
*/
- if (put_user(min_size, &uattr->func_info_rec_size))
+ if (copy_to_bpfptr_offset(uattr,
+ offsetof(union bpf_attr, func_info_rec_size),
+ &min_size, sizeof(min_size)))
ret = -EFAULT;
}
goto err_free;
}
- if (copy_from_user(&krecord[i], urecord, min_size)) {
+ if (copy_from_bpfptr(&krecord[i], urecord, min_size)) {
ret = -EFAULT;
goto err_free;
}
}
prev_offset = krecord[i].insn_off;
- urecord += urec_size;
+ bpfptr_add(&urecord, urec_size);
}
prog->aux->func_info = krecord;
static int check_btf_line(struct bpf_verifier_env *env,
const union bpf_attr *attr,
- union bpf_attr __user *uattr)
+ bpfptr_t uattr)
{
u32 i, s, nr_linfo, ncopy, expected_size, rec_size, prev_offset = 0;
struct bpf_subprog_info *sub;
struct bpf_line_info *linfo;
struct bpf_prog *prog;
const struct btf *btf;
- void __user *ulinfo;
+ bpfptr_t ulinfo;
int err;
nr_linfo = attr->line_info_cnt;
s = 0;
sub = env->subprog_info;
- ulinfo = u64_to_user_ptr(attr->line_info);
+ ulinfo = make_bpfptr(attr->line_info, uattr.is_kernel);
expected_size = sizeof(struct bpf_line_info);
ncopy = min_t(u32, expected_size, rec_size);
for (i = 0; i < nr_linfo; i++) {
if (err) {
if (err == -E2BIG) {
verbose(env, "nonzero tailing record in line_info");
- if (put_user(expected_size,
- &uattr->line_info_rec_size))
+ if (copy_to_bpfptr_offset(uattr,
+ offsetof(union bpf_attr, line_info_rec_size),
+ &expected_size, sizeof(expected_size)))
err = -EFAULT;
}
goto err_free;
}
- if (copy_from_user(&linfo[i], ulinfo, ncopy)) {
+ if (copy_from_bpfptr(&linfo[i], ulinfo, ncopy)) {
err = -EFAULT;
goto err_free;
}
}
prev_offset = linfo[i].insn_off;
- ulinfo += rec_size;
+ bpfptr_add(&ulinfo, rec_size);
}
if (s != env->subprog_cnt) {
static int check_btf_info(struct bpf_verifier_env *env,
const union bpf_attr *attr,
- union bpf_attr __user *uattr)
+ bpfptr_t uattr)
{
struct btf *btf;
int err;
old->s32_max_value >= cur->s32_max_value;
}
-/* Maximum number of register states that can exist at once */
-#define ID_MAP_SIZE (MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE)
-struct idpair {
- u32 old;
- u32 cur;
-};
-
/* If in the old state two registers had the same id, then they need to have
* the same id in the new state as well. But that id could be different from
* the old state, so we need to track the mapping from old to new ids.
* So we look through our idmap to see if this old id has been seen before. If
* so, we require the new id to match; otherwise, we add the id pair to the map.
*/
-static bool check_ids(u32 old_id, u32 cur_id, struct idpair *idmap)
+static bool check_ids(u32 old_id, u32 cur_id, struct bpf_id_pair *idmap)
{
unsigned int i;
- for (i = 0; i < ID_MAP_SIZE; i++) {
+ for (i = 0; i < BPF_ID_MAP_SIZE; i++) {
if (!idmap[i].old) {
/* Reached an empty slot; haven't seen this id before */
idmap[i].old = old_id;
* Since the verifier pushes the branch states as it sees them while exploring
* the program the condition of walking the branch instruction for the second
* time means that all states below this branch were already explored and
- * their final liveness markes are already propagated.
+ * their final liveness marks are already propagated.
* Hence when the verifier completes the search of state list in is_state_visited()
* we can call this clean_live_states() function to mark all liveness states
* as REG_LIVE_DONE to indicate that 'parent' pointers of 'struct bpf_reg_state'
/* Returns true if (rold safe implies rcur safe) */
static bool regsafe(struct bpf_reg_state *rold, struct bpf_reg_state *rcur,
- struct idpair *idmap)
+ struct bpf_id_pair *idmap)
{
bool equal;
static bool stacksafe(struct bpf_func_state *old,
struct bpf_func_state *cur,
- struct idpair *idmap)
+ struct bpf_id_pair *idmap)
{
int i, spi;
* whereas register type in current state is meaningful, it means that
* the current state will reach 'bpf_exit' instruction safely
*/
-static bool func_states_equal(struct bpf_func_state *old,
+static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_state *old,
struct bpf_func_state *cur)
{
- struct idpair *idmap;
- bool ret = false;
int i;
- idmap = kcalloc(ID_MAP_SIZE, sizeof(struct idpair), GFP_KERNEL);
- /* If we failed to allocate the idmap, just say it's not safe */
- if (!idmap)
- return false;
-
- for (i = 0; i < MAX_BPF_REG; i++) {
- if (!regsafe(&old->regs[i], &cur->regs[i], idmap))
- goto out_free;
- }
+ memset(env->idmap_scratch, 0, sizeof(env->idmap_scratch));
+ for (i = 0; i < MAX_BPF_REG; i++)
+ if (!regsafe(&old->regs[i], &cur->regs[i], env->idmap_scratch))
+ return false;
- if (!stacksafe(old, cur, idmap))
- goto out_free;
+ if (!stacksafe(old, cur, env->idmap_scratch))
+ return false;
if (!refsafe(old, cur))
- goto out_free;
- ret = true;
-out_free:
- kfree(idmap);
- return ret;
+ return false;
+
+ return true;
}
static bool states_equal(struct bpf_verifier_env *env,
for (i = 0; i <= old->curframe; i++) {
if (old->frame[i]->callsite != cur->frame[i]->callsite)
return false;
- if (!func_states_equal(old->frame[i], cur->frame[i]))
+ if (!func_states_equal(env, old->frame[i], cur->frame[i]))
return false;
}
return true;
}
regs = cur_regs(env);
- env->insn_aux_data[env->insn_idx].seen = env->pass_cnt;
+ sanitize_mark_insn_seen(env);
prev_insn_idx = env->insn_idx;
if (class == BPF_ALU || class == BPF_ALU64) {
return err;
env->insn_idx++;
- env->insn_aux_data[env->insn_idx].seen = env->pass_cnt;
+ sanitize_mark_insn_seen(env);
} else {
verbose(env, "invalid BPF_LD mode\n");
return -EINVAL;
struct bpf_map *map;
struct fd f;
u64 addr;
+ u32 fd;
if (i == insn_cnt - 1 || insn[1].code != 0 ||
insn[1].dst_reg != 0 || insn[1].src_reg != 0 ||
/* In final convert_pseudo_ld_imm64() step, this is
* converted into regular 64-bit imm load insn.
*/
- if ((insn[0].src_reg != BPF_PSEUDO_MAP_FD &&
- insn[0].src_reg != BPF_PSEUDO_MAP_VALUE) ||
- (insn[0].src_reg == BPF_PSEUDO_MAP_FD &&
- insn[1].imm != 0)) {
- verbose(env,
- "unrecognized bpf_ld_imm64 insn\n");
+ switch (insn[0].src_reg) {
+ case BPF_PSEUDO_MAP_VALUE:
+ case BPF_PSEUDO_MAP_IDX_VALUE:
+ break;
+ case BPF_PSEUDO_MAP_FD:
+ case BPF_PSEUDO_MAP_IDX:
+ if (insn[1].imm == 0)
+ break;
+ fallthrough;
+ default:
+ verbose(env, "unrecognized bpf_ld_imm64 insn\n");
return -EINVAL;
}
- f = fdget(insn[0].imm);
+ switch (insn[0].src_reg) {
+ case BPF_PSEUDO_MAP_IDX_VALUE:
+ case BPF_PSEUDO_MAP_IDX:
+ if (bpfptr_is_null(env->fd_array)) {
+ verbose(env, "fd_idx without fd_array is invalid\n");
+ return -EPROTO;
+ }
+ if (copy_from_bpfptr_offset(&fd, env->fd_array,
+ insn[0].imm * sizeof(fd),
+ sizeof(fd)))
+ return -EFAULT;
+ break;
+ default:
+ fd = insn[0].imm;
+ break;
+ }
+
+ f = fdget(fd);
map = __bpf_map_get(f);
if (IS_ERR(map)) {
verbose(env, "fd %d is not pointing to valid bpf_map\n",
}
aux = &env->insn_aux_data[i];
- if (insn->src_reg == BPF_PSEUDO_MAP_FD) {
+ if (insn[0].src_reg == BPF_PSEUDO_MAP_FD ||
+ insn[0].src_reg == BPF_PSEUDO_MAP_IDX) {
addr = (unsigned long)map;
} else {
u32 off = insn[1].imm;
{
struct bpf_insn_aux_data *new_data, *old_data = env->insn_aux_data;
struct bpf_insn *insn = new_prog->insnsi;
+ u32 old_seen = old_data[off].seen;
u32 prog_len;
int i;
memcpy(new_data + off + cnt - 1, old_data + off,
sizeof(struct bpf_insn_aux_data) * (prog_len - off - cnt + 1));
for (i = off; i < off + cnt - 1; i++) {
- new_data[i].seen = env->pass_cnt;
+ /* Expand insni[off]'s seen count to the patched range. */
+ new_data[i].seen = old_seen;
new_data[i].zext_dst = insn_has_def32(env, insn + i);
}
env->insn_aux_data = new_data;
prog->aux->max_pkt_offset = MAX_PACKET_OFF;
/* mark bpf_tail_call as different opcode to avoid
- * conditional branch in the interpeter for every normal
+ * conditional branch in the interpreter for every normal
* call and to prevent accidental JITing by JIT compiler
* that doesn't support bpf_tail_call yet
*/
* insn_aux_data was touched. These variables are compared to clear temporary
* data from failed pass. For testing and experiments do_check_common() can be
* run multiple times even when prior attempt to verify is unsuccessful.
+ *
+ * Note that special handling is needed on !env->bypass_spec_v1 if this is
+ * ever called outside of error path with subsequent program rejection.
*/
static void sanitize_insn_aux_data(struct bpf_verifier_env *env)
{
return 0;
}
+BTF_SET_START(btf_id_deny)
+BTF_ID_UNUSED
+#ifdef CONFIG_SMP
+BTF_ID(func, migrate_disable)
+BTF_ID(func, migrate_enable)
+#endif
+#if !defined CONFIG_PREEMPT_RCU && !defined CONFIG_TINY_RCU
+BTF_ID(func, rcu_read_unlock_strict)
+#endif
+BTF_SET_END(btf_id_deny)
+
static int check_attach_btf_id(struct bpf_verifier_env *env)
{
struct bpf_prog *prog = env->prog;
int ret;
u64 key;
+ if (prog->type == BPF_PROG_TYPE_SYSCALL) {
+ if (prog->aux->sleepable)
+ /* attach_btf_id checked to be zero already */
+ return 0;
+ verbose(env, "Syscall programs can only be sleepable\n");
+ return -EINVAL;
+ }
+
if (prog->aux->sleepable && prog->type != BPF_PROG_TYPE_TRACING &&
prog->type != BPF_PROG_TYPE_LSM) {
verbose(env, "Only fentry/fexit/fmod_ret and lsm programs can be sleepable\n");
ret = bpf_lsm_verify_prog(&env->log, prog);
if (ret < 0)
return ret;
+ } else if (prog->type == BPF_PROG_TYPE_TRACING &&
+ btf_id_set_contains(&btf_id_deny, btf_id)) {
+ return -EINVAL;
}
key = bpf_trampoline_compute_key(tgt_prog, prog->aux->attach_btf, btf_id);
return btf_vmlinux;
}
-int bpf_check(struct bpf_prog **prog, union bpf_attr *attr,
- union bpf_attr __user *uattr)
+int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr)
{
u64 start_time = ktime_get_ns();
struct bpf_verifier_env *env;
env->insn_aux_data[i].orig_idx = i;
env->prog = *prog;
env->ops = bpf_verifier_ops[env->prog->type];
+ env->fd_array = make_bpfptr(attr->fd_array, uattr.is_kernel);
is_priv = bpf_capable();
bpf_get_btf_vmlinux();
if (is_priv)
env->test_state_freq = attr->prog_flags & BPF_F_TEST_STATE_FREQ;
- if (bpf_prog_is_dev_bound(env->prog->aux)) {
- ret = bpf_prog_offload_verifier_prep(env->prog);
- if (ret)
- goto skip_full_check;
- }
-
env->explored_states = kvcalloc(state_htab_size(env),
sizeof(struct bpf_verifier_state_list *),
GFP_USER);
if (ret < 0)
goto skip_full_check;
+ if (bpf_prog_is_dev_bound(env->prog->aux)) {
+ ret = bpf_prog_offload_verifier_prep(env->prog);
+ if (ret)
+ goto skip_full_check;
+ }
+
ret = check_cfg(env);
if (ret < 0)
goto skip_full_check;
struct cgroup *cgrp = kn->priv;
int ret;
+ /* do not accept '\n' to prevent making /proc/<pid>/cgroup unparsable */
+ if (strchr(new_name_str, '\n'))
+ return -EINVAL;
+
if (kernfs_type(kn) != KERNFS_DIR)
return -ENOTDIR;
if (kn->parent != new_parent)
ctx->subsys_mask &= enabled;
/*
- * In absense of 'none', 'name=' or subsystem name options,
+ * In absence of 'none', 'name=' and subsystem name options,
* let's default to 'all'.
*/
if (!ctx->subsys_mask && !ctx->none && !ctx->name)
* @cgrp: the cgroup of interest
* @ss: the subsystem of interest
*
- * Find and get @cgrp's css assocaited with @ss. If the css doesn't exist
+ * Find and get @cgrp's css associated with @ss. If the css doesn't exist
* or is offline, %NULL is returned.
*/
static struct cgroup_subsys_state *cgroup_tryget_css(struct cgroup *cgrp,
/**
* css_clear_dir - remove subsys files in a cgroup directory
- * @css: taget css
+ * @css: target css
*/
static void css_clear_dir(struct cgroup_subsys_state *css)
{
/*
* This is called when the refcnt of a css is confirmed to be killed.
* css_tryget_online() is now guaranteed to fail. Tell the subsystem to
- * initate destruction and put the css ref from kill_css().
+ * initiate destruction and put the css ref from kill_css().
*/
static void css_killed_work_fn(struct work_struct *work)
{
return 0;
}
-static u16 cgroup_disable_mask __initdata;
-
/**
* cgroup_init - cgroup initialization
*
* disabled flag and cftype registration needs kmalloc,
* both of which aren't available during early_init.
*/
- if (cgroup_disable_mask & (1 << ssid)) {
- static_branch_disable(cgroup_subsys_enabled_key[ssid]);
- printk(KERN_INFO "Disabling %s control group subsystem\n",
- ss->name);
+ if (!cgroup_ssid_enabled(ssid))
continue;
- }
if (cgroup1_ssid_disabled(ssid))
printk(KERN_INFO "Disabling %s control group subsystem in v1 mounts\n",
* @kargs: the arguments passed to create the child process
*
* This calls the cancel_fork() callbacks if a fork failed *after*
- * cgroup_can_fork() succeded and cleans up references we took to
+ * cgroup_can_fork() succeeded and cleans up references we took to
* prepare a new css_set for the child process in cgroup_can_fork().
*/
void cgroup_cancel_fork(struct task_struct *child,
if (strcmp(token, ss->name) &&
strcmp(token, ss->legacy_name))
continue;
- cgroup_disable_mask |= 1 << i;
+
+ static_branch_disable(cgroup_subsys_enabled_key[i]);
+ pr_info("Disabling %s control group subsystem\n",
+ ss->name);
}
}
return 1;
}
/**
- * cpuset_nodemask_valid_mems_allowed - check nodemask vs. curremt mems_allowed
+ * cpuset_nodemask_valid_mems_allowed - check nodemask vs. current mems_allowed
* @nodemask: the nodemask to be checked
*
* Are any of the nodes in the nodemask allowed in current->mems_allowed?
* This function follows charging resource in hierarchical way.
* It will fail if the charge would cause the new value to exceed the
* hierarchical limit.
- * Returns 0 if the charge succeded, otherwise -EAGAIN, -ENOMEM or -EINVAL.
+ * Returns 0 if the charge succeeded, otherwise -EAGAIN, -ENOMEM or -EINVAL.
* Returns pointer to rdmacg for this resource when charging is successful.
*
* Charger needs to account resources on two criteria.
* @root: root of the tree to traversal
* @cpu: target cpu
*
- * Walks the udpated rstat_cpu tree on @cpu from @root. %NULL @pos starts
+ * Walks the updated rstat_cpu tree on @cpu from @root. %NULL @pos starts
* the traversal and %NULL return indicates the end. During traversal,
* each returned cgroup is unlinked from the tree. Must be called with the
* matching cgroup_rstat_cpu_lock held.
VMCOREINFO_LENGTH(mem_section, NR_SECTION_ROOTS);
VMCOREINFO_STRUCT_SIZE(mem_section);
VMCOREINFO_OFFSET(mem_section, section_mem_map);
+ VMCOREINFO_NUMBER(SECTION_SIZE_BITS);
VMCOREINFO_NUMBER(MAX_PHYSMEM_BITS);
#endif
VMCOREINFO_STRUCT_SIZE(page);
#include <linux/highmem.h>
#include <linux/livepatch.h>
#include <linux/audit.h>
+#include <linux/tick.h>
#include "common.h"
local_irq_disable_exit_to_user();
/* Check if any of the above work has queued a deferred wakeup */
- rcu_nocb_flush_deferred_wakeup();
+ tick_nohz_user_enter_prepare();
ti_work = READ_ONCE(current_thread_info()->flags);
}
lockdep_assert_irqs_disabled();
/* Flush pending rcuog wakeup before the last need_resched() check */
- rcu_nocb_flush_deferred_wakeup();
+ tick_nohz_user_enter_prepare();
if (unlikely(ti_work & EXIT_TO_USER_MODE_WORK))
ti_work = exit_to_user_mode_loop(regs, ti_work);
cpuctx = per_cpu_ptr(pmu->pmu_cpu_context, cpu);
ctx = &cpuctx->ctx;
get_ctx(ctx);
+ raw_spin_lock_irqsave(&ctx->lock, flags);
++ctx->pin_count;
+ raw_spin_unlock_irqrestore(&ctx->lock, flags);
return ctx;
}
static void perf_sigtrap(struct perf_event *event)
{
- struct kernel_siginfo info;
-
/*
* We'd expect this to only occur if the irq_work is delayed and either
* ctx->task or current has changed in the meantime. This can be the
if (current->flags & PF_EXITING)
return;
- clear_siginfo(&info);
- info.si_signo = SIGTRAP;
- info.si_code = TRAP_PERF;
- info.si_errno = event->attr.type;
- info.si_perf = event->attr.sig_data;
- info.si_addr = (void __user *)event->pending_addr;
- force_sig_info(&info);
+ force_sig_perf((void __user *)event->pending_addr,
+ event->attr.type, event->attr.sig_data);
}
static void perf_pending_event_disable(struct perf_event *event)
if (op & FUTEX_CLOCK_REALTIME) {
flags |= FLAGS_CLOCKRT;
- if (cmd != FUTEX_WAIT && cmd != FUTEX_WAIT_BITSET && \
- cmd != FUTEX_WAIT_REQUEUE_PI)
+ if (cmd != FUTEX_WAIT_BITSET && cmd != FUTEX_WAIT_REQUEUE_PI)
return -ENOSYS;
}
return -ENOSYS;
}
+static __always_inline bool futex_cmd_has_timeout(u32 cmd)
+{
+ switch (cmd) {
+ case FUTEX_WAIT:
+ case FUTEX_LOCK_PI:
+ case FUTEX_WAIT_BITSET:
+ case FUTEX_WAIT_REQUEUE_PI:
+ return true;
+ }
+ return false;
+}
+
+static __always_inline int
+futex_init_timeout(u32 cmd, u32 op, struct timespec64 *ts, ktime_t *t)
+{
+ if (!timespec64_valid(ts))
+ return -EINVAL;
+
+ *t = timespec64_to_ktime(*ts);
+ if (cmd == FUTEX_WAIT)
+ *t = ktime_add_safe(ktime_get(), *t);
+ else if (cmd != FUTEX_LOCK_PI && !(op & FUTEX_CLOCK_REALTIME))
+ *t = timens_ktime_to_host(CLOCK_MONOTONIC, *t);
+ return 0;
+}
SYSCALL_DEFINE6(futex, u32 __user *, uaddr, int, op, u32, val,
const struct __kernel_timespec __user *, utime,
u32 __user *, uaddr2, u32, val3)
{
- struct timespec64 ts;
+ int ret, cmd = op & FUTEX_CMD_MASK;
ktime_t t, *tp = NULL;
- u32 val2 = 0;
- int cmd = op & FUTEX_CMD_MASK;
+ struct timespec64 ts;
- if (utime && (cmd == FUTEX_WAIT || cmd == FUTEX_LOCK_PI ||
- cmd == FUTEX_WAIT_BITSET ||
- cmd == FUTEX_WAIT_REQUEUE_PI)) {
+ if (utime && futex_cmd_has_timeout(cmd)) {
if (unlikely(should_fail_futex(!(op & FUTEX_PRIVATE_FLAG))))
return -EFAULT;
if (get_timespec64(&ts, utime))
return -EFAULT;
- if (!timespec64_valid(&ts))
- return -EINVAL;
-
- t = timespec64_to_ktime(ts);
- if (cmd == FUTEX_WAIT)
- t = ktime_add_safe(ktime_get(), t);
- else if (!(op & FUTEX_CLOCK_REALTIME))
- t = timens_ktime_to_host(CLOCK_MONOTONIC, t);
+ ret = futex_init_timeout(cmd, op, &ts, &t);
+ if (ret)
+ return ret;
tp = &t;
}
- /*
- * requeue parameter in 'utime' if cmd == FUTEX_*_REQUEUE_*.
- * number of waiters to wake in 'utime' if cmd == FUTEX_WAKE_OP.
- */
- if (cmd == FUTEX_REQUEUE || cmd == FUTEX_CMP_REQUEUE ||
- cmd == FUTEX_CMP_REQUEUE_PI || cmd == FUTEX_WAKE_OP)
- val2 = (u32) (unsigned long) utime;
- return do_futex(uaddr, op, val, tp, uaddr2, val2, val3);
+ return do_futex(uaddr, op, val, tp, uaddr2, (unsigned long)utime, val3);
}
#ifdef CONFIG_COMPAT
const struct old_timespec32 __user *, utime, u32 __user *, uaddr2,
u32, val3)
{
- struct timespec64 ts;
+ int ret, cmd = op & FUTEX_CMD_MASK;
ktime_t t, *tp = NULL;
- int val2 = 0;
- int cmd = op & FUTEX_CMD_MASK;
+ struct timespec64 ts;
- if (utime && (cmd == FUTEX_WAIT || cmd == FUTEX_LOCK_PI ||
- cmd == FUTEX_WAIT_BITSET ||
- cmd == FUTEX_WAIT_REQUEUE_PI)) {
+ if (utime && futex_cmd_has_timeout(cmd)) {
if (get_old_timespec32(&ts, utime))
return -EFAULT;
- if (!timespec64_valid(&ts))
- return -EINVAL;
-
- t = timespec64_to_ktime(ts);
- if (cmd == FUTEX_WAIT)
- t = ktime_add_safe(ktime_get(), t);
- else if (!(op & FUTEX_CLOCK_REALTIME))
- t = timens_ktime_to_host(CLOCK_MONOTONIC, t);
+ ret = futex_init_timeout(cmd, op, &ts, &t);
+ if (ret)
+ return ret;
tp = &t;
}
- if (cmd == FUTEX_REQUEUE || cmd == FUTEX_CMP_REQUEUE ||
- cmd == FUTEX_CMP_REQUEUE_PI || cmd == FUTEX_WAKE_OP)
- val2 = (int) (unsigned long) utime;
- return do_futex(uaddr, op, val, tp, uaddr2, val2, val3);
+ return do_futex(uaddr, op, val, tp, uaddr2, (unsigned long)utime, val3);
}
#endif /* CONFIG_COMPAT_32BIT_TIME */
if (!irq_work_claim(work))
return false;
- /*record irq_work call stack in order to print it in KASAN reports*/
- kasan_record_aux_stack(work);
-
/* Queue the entry and raise the IPI if needed. */
preempt_disable();
__irq_work_queue_local(work);
.release = single_release
};
-static void __init kcsan_debugfs_init(void)
+static int __init kcsan_debugfs_init(void)
{
debugfs_create_file("kcsan", 0644, NULL, NULL, &debugfs_ops);
+ return 0;
}
late_initcall(kcsan_debugfs_init);
{
unsigned long flags;
- trace_lock_acquired(lock, ip);
+ trace_lock_contended(lock, ip);
if (unlikely(!lock_stat || !lockdep_enabled()))
return;
{
unsigned long flags;
- trace_lock_contended(lock, ip);
+ trace_lock_acquired(lock, ip);
if (unlikely(!lock_stat || !lockdep_enabled()))
return;
task->blocked_on = waiter;
}
-void mutex_remove_waiter(struct mutex *lock, struct mutex_waiter *waiter,
+void debug_mutex_remove_waiter(struct mutex *lock, struct mutex_waiter *waiter,
struct task_struct *task)
{
DEBUG_LOCKS_WARN_ON(list_empty(&waiter->list));
DEBUG_LOCKS_WARN_ON(task->blocked_on != waiter);
task->blocked_on = NULL;
- list_del_init(&waiter->list);
+ INIT_LIST_HEAD(&waiter->list);
waiter->task = NULL;
}
extern void debug_mutex_add_waiter(struct mutex *lock,
struct mutex_waiter *waiter,
struct task_struct *task);
-extern void mutex_remove_waiter(struct mutex *lock, struct mutex_waiter *waiter,
+extern void debug_mutex_remove_waiter(struct mutex *lock, struct mutex_waiter *waiter,
struct task_struct *task);
extern void debug_mutex_unlock(struct mutex *lock);
extern void debug_mutex_init(struct mutex *lock, const char *name,
* Add @waiter to a given location in the lock wait_list and set the
* FLAG_WAITERS flag if it's the first waiter.
*/
-static void __sched
+static void
__mutex_add_waiter(struct mutex *lock, struct mutex_waiter *waiter,
struct list_head *list)
{
__mutex_set_flag(lock, MUTEX_FLAG_WAITERS);
}
+static void
+__mutex_remove_waiter(struct mutex *lock, struct mutex_waiter *waiter)
+{
+ list_del(&waiter->list);
+ if (likely(list_empty(&lock->wait_list)))
+ __mutex_clear_flag(lock, MUTEX_FLAGS);
+
+ debug_mutex_remove_waiter(lock, waiter, current);
+}
+
/*
* Give up ownership to a specific task, when @task = NULL, this is equivalent
* to a regular unlock. Sets PICKUP on a handoff, clears HANDOFF, preserves
__ww_mutex_check_waiters(lock, ww_ctx);
}
- mutex_remove_waiter(lock, &waiter, current);
- if (likely(list_empty(&lock->wait_list)))
- __mutex_clear_flag(lock, MUTEX_FLAGS);
+ __mutex_remove_waiter(lock, &waiter);
debug_mutex_free_waiter(&waiter);
err:
__set_current_state(TASK_RUNNING);
- mutex_remove_waiter(lock, &waiter, current);
+ __mutex_remove_waiter(lock, &waiter);
err_early_kill:
spin_unlock(&lock->wait_lock);
debug_mutex_free_waiter(&waiter);
* !CONFIG_DEBUG_MUTEXES case. Most of them are NOPs:
*/
-#define mutex_remove_waiter(lock, waiter, task) \
- __list_del((waiter)->list.prev, (waiter)->list.next)
-
#define debug_mutex_wake_waiter(lock, waiter) do { } while (0)
#define debug_mutex_free_waiter(waiter) do { } while (0)
#define debug_mutex_add_waiter(lock, waiter, ti) do { } while (0)
+#define debug_mutex_remove_waiter(lock, waiter, ti) do { } while (0)
#define debug_mutex_unlock(lock) do { } while (0)
#define debug_mutex_init(lock, name, key) do { } while (0)
arch_spin_lock(&lock->wait_lock);
/* Try to acquire the lock directly if no reader is present */
- if (!atomic_read(&lock->cnts) &&
- (atomic_cmpxchg_acquire(&lock->cnts, 0, _QW_LOCKED) == 0))
+ if (!(cnts = atomic_read(&lock->cnts)) &&
+ atomic_try_cmpxchg_acquire(&lock->cnts, &cnts, _QW_LOCKED))
goto unlock;
/* Set the waiting flag to notify readers that a writer is pending */
- atomic_add(_QW_WAITING, &lock->cnts);
+ atomic_or(_QW_WAITING, &lock->cnts);
/* When no more readers or writers, set the locked flag */
do {
return ret;
}
+static bool module_init_layout_section(const char *sname)
+{
+#ifndef CONFIG_MODULE_UNLOAD
+ if (module_exit_section(sname))
+ return true;
+#endif
+ return module_init_section(sname);
+}
+
/*
* Lay out the SHF_ALLOC sections in a way not dissimilar to how ld
* might -- code, read-only data, read-write data, small data. Tally
if ((s->sh_flags & masks[m][0]) != masks[m][0]
|| (s->sh_flags & masks[m][1])
|| s->sh_entsize != ~0UL
- || module_init_section(sname))
+ || module_init_layout_section(sname))
continue;
s->sh_entsize = get_offset(mod, &mod->core_layout.size, s, i);
pr_debug("\t%s\n", sname);
if ((s->sh_flags & masks[m][0]) != masks[m][0]
|| (s->sh_flags & masks[m][1])
|| s->sh_entsize != ~0UL
- || !module_init_section(sname))
+ || !module_init_layout_section(sname))
continue;
s->sh_entsize = (get_offset(mod, &mod->init_layout.size, s, i)
| INIT_OFFSET_MASK);
bool __weak module_init_section(const char *name)
{
-#ifndef CONFIG_MODULE_UNLOAD
- return strstarts(name, ".init") || module_exit_section(name);
-#else
return strstarts(name, ".init");
-#endif
}
bool __weak module_exit_section(const char *name)
/* No obstacles. */
return vprintk_default(fmt, args);
}
+EXPORT_SYMBOL(vprintk);
void __init printk_safe_init(void)
{
/* Flush pending messages that did not have scheduled IRQ works. */
printk_safe_flush();
}
-EXPORT_SYMBOL(vprintk);
spin_unlock(&child->sighand->siglock);
}
+static bool looks_like_a_spurious_pid(struct task_struct *task)
+{
+ if (task->exit_code != ((PTRACE_EVENT_EXEC << 8) | SIGTRAP))
+ return false;
+
+ if (task_pid_vnr(task) == task->ptrace_message)
+ return false;
+ /*
+ * The tracee changed its pid but the PTRACE_EVENT_EXEC event
+ * was not wait()'ed, most probably debugger targets the old
+ * leader which was destroyed in de_thread().
+ */
+ return true;
+}
+
/* Ensure that nothing can wake it up, even SIGKILL */
static bool ptrace_freeze_traced(struct task_struct *task)
{
return ret;
spin_lock_irq(&task->sighand->siglock);
- if (task_is_traced(task) && !__fatal_signal_pending(task)) {
+ if (task_is_traced(task) && !looks_like_a_spurious_pid(task) &&
+ !__fatal_signal_pending(task)) {
task->state = __TASK_TRACED;
ret = true;
}
REGION_DISJOINT)
continue;
- if (!__request_region_locked(res, &iomem_resource, addr, size,
+ if (__request_region_locked(res, &iomem_resource, addr, size,
name, 0))
break;
static inline unsigned int uclamp_bucket_id(unsigned int clamp_value)
{
- return clamp_value / UCLAMP_BUCKET_DELTA;
+ return min_t(unsigned int, clamp_value / UCLAMP_BUCKET_DELTA, UCLAMP_BUCKETS - 1);
}
static inline unsigned int uclamp_none(enum uclamp_id clamp_id)
{
return __sched_setscheduler(p, attr, false, true);
}
-EXPORT_SYMBOL_GPL(sched_setattr_nocheck);
/**
* sched_setscheduler_nocheck - change the scheduling policy and/or RT priority of a thread from kernelspace.
#define __PS(S, F) SEQ_printf(m, "%-45s:%21Ld\n", S, (long long)(F))
#define __P(F) __PS(#F, F)
#define P(F) __PS(#F, p->F)
+#define PM(F, M) __PS(#F, p->F & (M))
#define __PSN(S, F) SEQ_printf(m, "%-45s:%14Ld.%06ld\n", S, SPLIT_NS((long long)(F)))
#define __PN(F) __PSN(#F, F)
#define PN(F) __PSN(#F, p->F)
P(se.avg.util_avg);
P(se.avg.last_update_time);
P(se.avg.util_est.ewma);
- P(se.avg.util_est.enqueued);
+ PM(se.avg.util_est.enqueued, ~UTIL_AVG_UNCHANGED);
#endif
#ifdef CONFIG_UCLAMP_TASK
__PS("uclamp.min", p->uclamp_req[UCLAMP_MIN].value);
static inline void
update_tg_cfs_load(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cfs_rq *gcfs_rq)
{
- long delta_avg, running_sum, runnable_sum = gcfs_rq->prop_runnable_sum;
+ long delta, running_sum, runnable_sum = gcfs_rq->prop_runnable_sum;
unsigned long load_avg;
u64 load_sum = 0;
- s64 delta_sum;
u32 divider;
if (!runnable_sum)
load_sum = (s64)se_weight(se) * runnable_sum;
load_avg = div_s64(load_sum, divider);
- delta_sum = load_sum - (s64)se_weight(se) * se->avg.load_sum;
- delta_avg = load_avg - se->avg.load_avg;
+ delta = load_avg - se->avg.load_avg;
se->avg.load_sum = runnable_sum;
se->avg.load_avg = load_avg;
- add_positive(&cfs_rq->avg.load_avg, delta_avg);
- add_positive(&cfs_rq->avg.load_sum, delta_sum);
+
+ add_positive(&cfs_rq->avg.load_avg, delta);
+ cfs_rq->avg.load_sum = cfs_rq->avg.load_avg * divider;
}
static inline void add_tg_cfs_propagate(struct cfs_rq *cfs_rq, long runnable_sum)
*/
static void detach_entity_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
+ /*
+ * cfs_rq->avg.period_contrib can be used for both cfs_rq and se.
+ * See ___update_load_avg() for details.
+ */
+ u32 divider = get_pelt_divider(&cfs_rq->avg);
+
dequeue_load_avg(cfs_rq, se);
sub_positive(&cfs_rq->avg.util_avg, se->avg.util_avg);
- sub_positive(&cfs_rq->avg.util_sum, se->avg.util_sum);
+ cfs_rq->avg.util_sum = cfs_rq->avg.util_avg * divider;
sub_positive(&cfs_rq->avg.runnable_avg, se->avg.runnable_avg);
- sub_positive(&cfs_rq->avg.runnable_sum, se->avg.runnable_sum);
+ cfs_rq->avg.runnable_sum = cfs_rq->avg.runnable_avg * divider;
add_tg_cfs_propagate(cfs_rq, -se->avg.load_sum);
{
struct util_est ue = READ_ONCE(p->se.avg.util_est);
- return (max(ue.ewma, ue.enqueued) | UTIL_AVG_UNCHANGED);
+ return max(ue.ewma, (ue.enqueued & ~UTIL_AVG_UNCHANGED));
}
static inline unsigned long task_util_est(struct task_struct *p)
* Reset EWMA on utilization increases, the moving average is used only
* to smooth utilization decreases.
*/
- ue.enqueued = (task_util(p) | UTIL_AVG_UNCHANGED);
+ ue.enqueued = task_util(p);
if (sched_feat(UTIL_EST_FASTUP)) {
if (ue.ewma < ue.enqueued) {
ue.ewma = ue.enqueued;
ue.ewma += last_ewma_diff;
ue.ewma >>= UTIL_EST_WEIGHT_SHIFT;
done:
+ ue.enqueued |= UTIL_AVG_UNCHANGED;
WRITE_ONCE(p->se.avg.util_est, ue);
trace_sched_util_est_se_tp(&p->se);
}
if (has_idle_core)
- set_idle_cores(this, false);
+ set_idle_cores(target, false);
if (sched_feat(SIS_PROP) && !has_idle_core) {
time = cpu_clock(this) - time;
/* Propagate pending load changes to the parent, if any: */
se = cfs_rq->tg->se[cpu];
if (se && !skip_blocked_update(se))
- update_load_avg(cfs_rq_of(se), se, 0);
+ update_load_avg(cfs_rq_of(se), se, UPDATE_TG);
/*
* There can be a lot of idle CPU cgroups. Don't let fully
{
struct cfs_rq *cfs_rq;
+ list_add_leaf_cfs_rq(cfs_rq_of(se));
+
/* Start to propagate at parent */
se = se->parent;
for_each_sched_entity(se) {
cfs_rq = cfs_rq_of(se);
- if (cfs_rq_throttled(cfs_rq))
- break;
+ if (!cfs_rq_throttled(cfs_rq)){
+ update_load_avg(cfs_rq, se, UPDATE_TG);
+ list_add_leaf_cfs_rq(cfs_rq);
+ continue;
+ }
- update_load_avg(cfs_rq, se, UPDATE_TG);
+ if (list_add_leaf_cfs_rq(cfs_rq))
+ break;
}
}
#else
return LOAD_AVG_MAX - 1024 + avg->period_contrib;
}
-/*
- * When a task is dequeued, its estimated utilization should not be update if
- * its util_avg has not been updated at least once.
- * This flag is used to synchronize util_avg updates with util_est updates.
- * We map this information into the LSB bit of the utilization saved at
- * dequeue time (i.e. util_est.dequeued).
- */
-#define UTIL_AVG_UNCHANGED 0x1
-
static inline void cfs_se_util_change(struct sched_avg *avg)
{
unsigned int enqueued;
if (!sched_feat(UTIL_EST))
return;
- /* Avoid store if the flag has been already set */
+ /* Avoid store if the flag has been already reset */
enqueued = avg->util_est.enqueued;
if (!(enqueued & UTIL_AVG_UNCHANGED))
return;
*/
void cgroup_move_task(struct task_struct *task, struct css_set *to)
{
- unsigned int task_flags = 0;
+ unsigned int task_flags;
struct rq_flags rf;
struct rq *rq;
rq = task_rq_lock(task, &rf);
- if (task_on_rq_queued(task)) {
- task_flags = TSK_RUNNING;
- if (task_current(rq, task))
- task_flags |= TSK_ONCPU;
- } else if (task->in_iowait)
- task_flags = TSK_IOWAIT;
-
- if (task->in_memstall)
- task_flags |= TSK_MEMSTALL;
+ /*
+ * We may race with schedule() dropping the rq lock between
+ * deactivating prev and switching to next. Because the psi
+ * updates from the deactivation are deferred to the switch
+ * callback to save cgroup tree updates, the task's scheduling
+ * state here is not coherent with its psi state:
+ *
+ * schedule() cgroup_move_task()
+ * rq_lock()
+ * deactivate_task()
+ * p->on_rq = 0
+ * psi_dequeue() // defers TSK_RUNNING & TSK_IOWAIT updates
+ * pick_next_task()
+ * rq_unlock()
+ * rq_lock()
+ * psi_task_change() // old cgroup
+ * task->cgroups = to
+ * psi_task_change() // new cgroup
+ * rq_unlock()
+ * rq_lock()
+ * psi_sched_switch() // does deferred updates in new cgroup
+ *
+ * Don't rely on the scheduling state. Use psi_flags instead.
+ */
+ task_flags = task->psi_flags;
if (task_flags)
psi_task_change(task, task_flags, 0);
up(&match->notif->request);
wake_up_poll(&match->wqh, EPOLLIN | EPOLLRDNORM);
- mutex_unlock(&match->notify_lock);
/*
* This is where we wait for a reply from userspace.
*/
-wait:
- err = wait_for_completion_interruptible(&n.ready);
- mutex_lock(&match->notify_lock);
- if (err == 0) {
- /* Check if we were woken up by a addfd message */
+ do {
+ mutex_unlock(&match->notify_lock);
+ err = wait_for_completion_interruptible(&n.ready);
+ mutex_lock(&match->notify_lock);
+ if (err != 0)
+ goto interrupted;
+
addfd = list_first_entry_or_null(&n.addfd,
struct seccomp_kaddfd, list);
- if (addfd && n.state != SECCOMP_NOTIFY_REPLIED) {
+ /* Check if we were woken up by a addfd message */
+ if (addfd)
seccomp_handle_addfd(addfd);
- mutex_unlock(&match->notify_lock);
- goto wait;
- }
- ret = n.val;
- err = n.error;
- flags = n.flags;
- }
+ } while (n.state != SECCOMP_NOTIFY_REPLIED);
+
+ ret = n.val;
+ err = n.error;
+ flags = n.flags;
+
+interrupted:
/* If there were any pending addfd calls, clear them out */
list_for_each_entry_safe(addfd, tmp, &n.addfd, list) {
/* The process went away before we got a chance to handle it */
case SIL_TIMER:
case SIL_POLL:
case SIL_FAULT:
+ case SIL_FAULT_TRAPNO:
case SIL_FAULT_MCEERR:
case SIL_FAULT_BNDERR:
case SIL_FAULT_PKUERR:
}
#endif
+int force_sig_perf(void __user *addr, u32 type, u64 sig_data)
+{
+ struct kernel_siginfo info;
+
+ clear_siginfo(&info);
+ info.si_signo = SIGTRAP;
+ info.si_errno = 0;
+ info.si_code = TRAP_PERF;
+ info.si_addr = addr;
+ info.si_perf_data = sig_data;
+ info.si_perf_type = type;
+
+ return force_sig_info(&info);
+}
+
/* For the crazy architectures that include trap information in
* the errno field, instead of an actual errno value.
*/
{
switch (siginfo_layout(ksig->sig, ksig->info.si_code)) {
case SIL_FAULT:
+ case SIL_FAULT_TRAPNO:
case SIL_FAULT_MCEERR:
case SIL_FAULT_BNDERR:
case SIL_FAULT_PKUERR:
#endif
else if ((sig == SIGTRAP) && (si_code == TRAP_PERF))
layout = SIL_PERF_EVENT;
+#ifdef __ARCH_SI_TRAPNO
+ else if (layout == SIL_FAULT)
+ layout = SIL_FAULT_TRAPNO;
+#endif
}
else if (si_code <= NSIGPOLL)
layout = SIL_POLL;
break;
case SIL_FAULT:
to->si_addr = ptr_to_compat(from->si_addr);
-#ifdef __ARCH_SI_TRAPNO
+ break;
+ case SIL_FAULT_TRAPNO:
+ to->si_addr = ptr_to_compat(from->si_addr);
to->si_trapno = from->si_trapno;
-#endif
break;
case SIL_FAULT_MCEERR:
to->si_addr = ptr_to_compat(from->si_addr);
-#ifdef __ARCH_SI_TRAPNO
- to->si_trapno = from->si_trapno;
-#endif
to->si_addr_lsb = from->si_addr_lsb;
break;
case SIL_FAULT_BNDERR:
to->si_addr = ptr_to_compat(from->si_addr);
-#ifdef __ARCH_SI_TRAPNO
- to->si_trapno = from->si_trapno;
-#endif
to->si_lower = ptr_to_compat(from->si_lower);
to->si_upper = ptr_to_compat(from->si_upper);
break;
case SIL_FAULT_PKUERR:
to->si_addr = ptr_to_compat(from->si_addr);
-#ifdef __ARCH_SI_TRAPNO
- to->si_trapno = from->si_trapno;
-#endif
to->si_pkey = from->si_pkey;
break;
case SIL_PERF_EVENT:
to->si_addr = ptr_to_compat(from->si_addr);
- to->si_perf = from->si_perf;
+ to->si_perf_data = from->si_perf_data;
+ to->si_perf_type = from->si_perf_type;
break;
case SIL_CHLD:
to->si_pid = from->si_pid;
break;
case SIL_FAULT:
to->si_addr = compat_ptr(from->si_addr);
-#ifdef __ARCH_SI_TRAPNO
+ break;
+ case SIL_FAULT_TRAPNO:
+ to->si_addr = compat_ptr(from->si_addr);
to->si_trapno = from->si_trapno;
-#endif
break;
case SIL_FAULT_MCEERR:
to->si_addr = compat_ptr(from->si_addr);
-#ifdef __ARCH_SI_TRAPNO
- to->si_trapno = from->si_trapno;
-#endif
to->si_addr_lsb = from->si_addr_lsb;
break;
case SIL_FAULT_BNDERR:
to->si_addr = compat_ptr(from->si_addr);
-#ifdef __ARCH_SI_TRAPNO
- to->si_trapno = from->si_trapno;
-#endif
to->si_lower = compat_ptr(from->si_lower);
to->si_upper = compat_ptr(from->si_upper);
break;
case SIL_FAULT_PKUERR:
to->si_addr = compat_ptr(from->si_addr);
-#ifdef __ARCH_SI_TRAPNO
- to->si_trapno = from->si_trapno;
-#endif
to->si_pkey = from->si_pkey;
break;
case SIL_PERF_EVENT:
to->si_addr = compat_ptr(from->si_addr);
- to->si_perf = from->si_perf;
+ to->si_perf_data = from->si_perf_data;
+ to->si_perf_type = from->si_perf_type;
break;
case SIL_CHLD:
to->si_pid = from->si_pid;
/* sigfault */
CHECK_OFFSET(si_addr);
+ CHECK_OFFSET(si_trapno);
CHECK_OFFSET(si_addr_lsb);
CHECK_OFFSET(si_lower);
CHECK_OFFSET(si_upper);
CHECK_OFFSET(si_pkey);
- CHECK_OFFSET(si_perf);
+ CHECK_OFFSET(si_perf_data);
+ CHECK_OFFSET(si_perf_type);
/* sigpoll */
CHECK_OFFSET(si_band);
} while (0)
/* Record current CSD work for current CPU, NULL to erase. */
-static void __csd_lock_record(call_single_data_t *csd)
+static void __csd_lock_record(struct __call_single_data *csd)
{
if (!csd) {
smp_mb(); /* NULL cur_csd after unlock. */
/* Or before unlock, as the case may be. */
}
-static __always_inline void csd_lock_record(call_single_data_t *csd)
+static __always_inline void csd_lock_record(struct __call_single_data *csd)
{
if (static_branch_unlikely(&csdlock_debug_enabled))
__csd_lock_record(csd);
}
-static int csd_lock_wait_getcpu(call_single_data_t *csd)
+static int csd_lock_wait_getcpu(struct __call_single_data *csd)
{
unsigned int csd_type;
return (type >= ARRAY_SIZE(seq_type)) ? "?" : seq_type[type];
}
-static void csd_lock_print_extended(call_single_data_t *csd, int cpu)
+static void csd_lock_print_extended(struct __call_single_data *csd, int cpu)
{
struct cfd_seq_local *seq = &per_cpu(cfd_seq_local, cpu);
unsigned int srccpu = csd->node.src;
* the CSD_TYPE_SYNC/ASYNC types provide the destination CPU,
* so waiting on other types gets much less information.
*/
-static bool csd_lock_wait_toolong(call_single_data_t *csd, u64 ts0, u64 *ts1, int *bug_id)
+static bool csd_lock_wait_toolong(struct __call_single_data *csd, u64 ts0, u64 *ts1, int *bug_id)
{
int cpu = -1;
int cpux;
* previous function call. For multi-cpu calls its even more interesting
* as we'll have to ensure no other cpu is observing our csd.
*/
-static void __csd_lock_wait(call_single_data_t *csd)
+static void __csd_lock_wait(struct __call_single_data *csd)
{
int bug_id = 0;
u64 ts0, ts1;
smp_acquire__after_ctrl_dep();
}
-static __always_inline void csd_lock_wait(call_single_data_t *csd)
+static __always_inline void csd_lock_wait(struct __call_single_data *csd)
{
if (static_branch_unlikely(&csdlock_debug_enabled)) {
__csd_lock_wait(csd);
#else
#define cfd_seq_store(var, src, dst, type)
-static void csd_lock_record(call_single_data_t *csd)
+static void csd_lock_record(struct __call_single_data *csd)
{
}
-static __always_inline void csd_lock_wait(call_single_data_t *csd)
+static __always_inline void csd_lock_wait(struct __call_single_data *csd)
{
smp_cond_load_acquire(&csd->node.u_flags, !(VAL & CSD_FLAG_LOCK));
}
#endif
-static __always_inline void csd_lock(call_single_data_t *csd)
+static __always_inline void csd_lock(struct __call_single_data *csd)
{
csd_lock_wait(csd);
csd->node.u_flags |= CSD_FLAG_LOCK;
smp_wmb();
}
-static __always_inline void csd_unlock(call_single_data_t *csd)
+static __always_inline void csd_unlock(struct __call_single_data *csd)
{
WARN_ON(!(csd->node.u_flags & CSD_FLAG_LOCK));
* for execution on the given CPU. data must already have
* ->func, ->info, and ->flags set.
*/
-static int generic_exec_single(int cpu, call_single_data_t *csd)
+static int generic_exec_single(int cpu, struct __call_single_data *csd)
{
if (cpu == smp_processor_id()) {
smp_call_func_t func = csd->func;
* NOTE: Be careful, there is unfortunately no current debugging facility to
* validate the correctness of this serialization.
*/
-int smp_call_function_single_async(int cpu, call_single_data_t *csd)
+int smp_call_function_single_async(int cpu, struct __call_single_data *csd)
{
int err = 0;
mutex_unlock(&bpf_stats_enabled_mutex);
return ret;
}
-#endif
+
+static int bpf_unpriv_handler(struct ctl_table *table, int write,
+ void *buffer, size_t *lenp, loff_t *ppos)
+{
+ int ret, unpriv_enable = *(int *)table->data;
+ bool locked_state = unpriv_enable == 1;
+ struct ctl_table tmp = *table;
+
+ if (write && !capable(CAP_SYS_ADMIN))
+ return -EPERM;
+
+ tmp.data = &unpriv_enable;
+ ret = proc_dointvec_minmax(&tmp, write, buffer, lenp, ppos);
+ if (write && !ret) {
+ if (locked_state && unpriv_enable != 1)
+ return -EPERM;
+ *(int *)table->data = unpriv_enable;
+ }
+ return ret;
+}
+#endif /* CONFIG_BPF_SYSCALL && CONFIG_SYSCTL */
/*
* /proc/sys support
.data = &sysctl_unprivileged_bpf_disabled,
.maxlen = sizeof(sysctl_unprivileged_bpf_disabled),
.mode = 0644,
- /* only handle a transition from default "0" to "1" */
- .proc_handler = proc_dointvec_minmax,
- .extra1 = SYSCTL_ONE,
- .extra2 = SYSCTL_ONE,
+ .proc_handler = bpf_unpriv_handler,
+ .extra1 = SYSCTL_ZERO,
+ .extra2 = &two,
},
{
.procname = "bpf_stats_enabled",
if (rtcdev)
return -EBUSY;
- if (!rtc->ops->set_alarm)
+ if (!test_bit(RTC_FEATURE_ALARM, rtc->features))
return -1;
if (!device_may_wakeup(rtc->dev.parent))
return -1;
#ifdef CONFIG_NO_HZ_FULL
cpumask_var_t tick_nohz_full_mask;
+EXPORT_SYMBOL_GPL(tick_nohz_full_mask);
bool tick_nohz_full_running;
EXPORT_SYMBOL_GPL(tick_nohz_full_running);
static atomic_t tick_dep_mask;
static __always_inline int
bpf_probe_read_kernel_common(void *dst, u32 size, const void *unsafe_ptr)
{
- int ret = security_locked_down(LOCKDOWN_BPF_READ);
+ int ret;
- if (unlikely(ret < 0))
- goto fail;
ret = copy_from_kernel_nofault(dst, unsafe_ptr, size);
if (unlikely(ret < 0))
- goto fail;
- return ret;
-fail:
- memset(dst, 0, size);
+ memset(dst, 0, size);
return ret;
}
static __always_inline int
bpf_probe_read_kernel_str_common(void *dst, u32 size, const void *unsafe_ptr)
{
- int ret = security_locked_down(LOCKDOWN_BPF_READ);
-
- if (unlikely(ret < 0))
- goto fail;
+ int ret;
/*
* The strncpy_from_kernel_nofault() call will likely not fill the
*/
ret = strncpy_from_kernel_nofault(dst, unsafe_ptr, size);
if (unlikely(ret < 0))
- goto fail;
-
- return ret;
-fail:
- memset(dst, 0, size);
+ memset(dst, 0, size);
return ret;
}
case BPF_FUNC_probe_read_user:
return &bpf_probe_read_user_proto;
case BPF_FUNC_probe_read_kernel:
- return &bpf_probe_read_kernel_proto;
+ return security_locked_down(LOCKDOWN_BPF_READ) < 0 ?
+ NULL : &bpf_probe_read_kernel_proto;
case BPF_FUNC_probe_read_user_str:
return &bpf_probe_read_user_str_proto;
case BPF_FUNC_probe_read_kernel_str:
- return &bpf_probe_read_kernel_str_proto;
+ return security_locked_down(LOCKDOWN_BPF_READ) < 0 ?
+ NULL : &bpf_probe_read_kernel_str_proto;
#ifdef CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
case BPF_FUNC_probe_read:
- return &bpf_probe_read_compat_proto;
+ return security_locked_down(LOCKDOWN_BPF_READ) < 0 ?
+ NULL : &bpf_probe_read_compat_proto;
case BPF_FUNC_probe_read_str:
- return &bpf_probe_read_compat_str_proto;
+ return security_locked_down(LOCKDOWN_BPF_READ) < 0 ?
+ NULL : &bpf_probe_read_compat_str_proto;
#endif
#ifdef CONFIG_CGROUPS
case BPF_FUNC_get_current_cgroup_id:
static void print_ip_ins(const char *fmt, const unsigned char *p)
{
+ char ins[MCOUNT_INSN_SIZE];
int i;
+ if (copy_from_kernel_nofault(ins, p, MCOUNT_INSN_SIZE)) {
+ printk(KERN_CONT "%s[FAULT] %px\n", fmt, p);
+ return;
+ }
+
printk(KERN_CONT "%s", fmt);
for (i = 0; i < MCOUNT_INSN_SIZE; i++)
- printk(KERN_CONT "%s%02x", i ? ":" : "", p[i]);
+ printk(KERN_CONT "%s%02x", i ? ":" : "", ins[i]);
}
enum ftrace_bug_type ftrace_bug_type;
};
static struct saved_cmdlines_buffer *savedcmd;
-/* temporary disable recording */
-static atomic_t trace_record_taskinfo_disabled __read_mostly;
-
static inline char *get_saved_cmdlines(int idx)
{
return &savedcmd->saved_cmdlines[idx * TASK_COMM_LEN];
{
if (unlikely(!(flags & (TRACE_RECORD_CMDLINE | TRACE_RECORD_TGID))))
return true;
- if (atomic_read(&trace_record_taskinfo_disabled) || !tracing_is_on())
- return true;
if (!__this_cpu_read(trace_taskinfo_save))
return true;
return false;
(entry = this_cpu_read(trace_buffered_event))) {
/* Try to use the per cpu buffer first */
val = this_cpu_inc_return(trace_buffered_event_cnt);
- if ((len < (PAGE_SIZE - sizeof(*entry))) && val == 1) {
+ if ((len < (PAGE_SIZE - sizeof(*entry) - sizeof(entry->array[0]))) && val == 1) {
trace_event_setup(entry, type, trace_ctx);
entry->array[0] = len;
return entry;
goto print;
while (*p) {
+ bool star = false;
+ int len = 0;
+
j = 0;
/* We only care about %s and variants */
/* Need to test cases like %08.*s */
for (j = 1; p[i+j]; j++) {
if (isdigit(p[i+j]) ||
- p[i+j] == '*' ||
p[i+j] == '.')
continue;
+ if (p[i+j] == '*') {
+ star = true;
+ continue;
+ }
break;
}
if (p[i+j] == 's')
break;
+ star = false;
}
j = 0;
}
iter->fmt[i] = '\0';
trace_seq_vprintf(&iter->seq, iter->fmt, ap);
+ if (star)
+ len = va_arg(ap, int);
+
/* The ap now points to the string data of the %s */
str = va_arg(ap, const char *);
int ret;
/* Try to safely read the string */
- ret = strncpy_from_kernel_nofault(iter->fmt, str,
- iter->fmt_size);
+ if (star) {
+ if (len + 1 > iter->fmt_size)
+ len = iter->fmt_size - 1;
+ if (len < 0)
+ len = 0;
+ ret = copy_from_kernel_nofault(iter->fmt, str, len);
+ iter->fmt[len] = 0;
+ star = false;
+ } else {
+ ret = strncpy_from_kernel_nofault(iter->fmt, str,
+ iter->fmt_size);
+ }
if (ret < 0)
trace_seq_printf(&iter->seq, "(0x%px)", str);
else
strncpy(iter->fmt, p + i, j + 1);
iter->fmt[j+1] = '\0';
}
- trace_seq_printf(&iter->seq, iter->fmt, str);
+ if (star)
+ trace_seq_printf(&iter->seq, iter->fmt, len, str);
+ else
+ trace_seq_printf(&iter->seq, iter->fmt, str);
p += i + j + 1;
}
return ERR_PTR(-EBUSY);
#endif
- if (!iter->snapshot)
- atomic_inc(&trace_record_taskinfo_disabled);
-
if (*pos != iter->pos) {
iter->ent = NULL;
iter->cpu = 0;
return;
#endif
- if (!iter->snapshot)
- atomic_dec(&trace_record_taskinfo_disabled);
-
trace_access_unlock(iter->cpu_file);
trace_event_read_unlock();
}
prev_time = READ_ONCE(trace_clock_struct.prev_time);
now = sched_clock_cpu(this_cpu);
- /* Make sure that now is always greater than prev_time */
+ /* Make sure that now is always greater than or equal to prev_time */
if ((s64)(now - prev_time) < 0)
- now = prev_time + 1;
+ now = prev_time;
/*
* If in an NMI context then dont risk lockups and simply return
/* Reread prev_time in case it was already updated */
prev_time = READ_ONCE(trace_clock_struct.prev_time);
if ((s64)(now - prev_time) < 0)
- now = prev_time + 1;
+ now = prev_time;
trace_clock_struct.prev_time = now;
}
EXPORT_SYMBOL(smp_call_function_single);
-int smp_call_function_single_async(int cpu, call_single_data_t *csd)
+int smp_call_function_single_async(int cpu, struct __call_single_data *csd)
{
unsigned long flags;
__this_cpu_write(watchdog_report_ts, SOFTLOCKUP_DELAY_REPORT);
}
-static int is_softlockup(unsigned long touch_ts, unsigned long period_ts)
+static int is_softlockup(unsigned long touch_ts,
+ unsigned long period_ts,
+ unsigned long now)
{
- unsigned long now = get_timestamp();
-
if ((watchdog_enabled & SOFT_WATCHDOG_ENABLED) && watchdog_thresh){
/* Warn about unreasonable delays. */
if (time_after(now, period_ts + get_softlockup_thresh()))
/* watchdog kicker functions */
static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
{
- unsigned long touch_ts = __this_cpu_read(watchdog_touch_ts);
- unsigned long period_ts = __this_cpu_read(watchdog_report_ts);
+ unsigned long touch_ts, period_ts, now;
struct pt_regs *regs = get_irq_regs();
int duration;
int softlockup_all_cpu_backtrace = sysctl_softlockup_all_cpu_backtrace;
hrtimer_forward_now(hrtimer, ns_to_ktime(sample_period));
/*
+ * Read the current timestamp first. It might become invalid anytime
+ * when a virtual machine is stopped by the host or when the watchog
+ * is touched from NMI.
+ */
+ now = get_timestamp();
+ /*
* If a virtual machine is stopped by the host it can look to
- * the watchdog like a soft lockup. Check to see if the host
- * stopped the vm before we process the timestamps.
+ * the watchdog like a soft lockup. This function touches the watchdog.
*/
kvm_check_and_clear_guest_paused();
+ /*
+ * The stored timestamp is comparable with @now only when not touched.
+ * It might get touched anytime from NMI. Make sure that is_softlockup()
+ * uses the same (valid) value.
+ */
+ period_ts = READ_ONCE(*this_cpu_ptr(&watchdog_report_ts));
/* Reset the interval when touched by known problematic code. */
if (period_ts == SOFTLOCKUP_DELAY_REPORT) {
return HRTIMER_RESTART;
}
- /* check for a softlockup
- * This is done by making sure a high priority task is
- * being scheduled. The task touches the watchdog to
- * indicate it is getting cpu time. If it hasn't then
- * this is a good indication some task is hogging the cpu
- */
- duration = is_softlockup(touch_ts, period_ts);
+ /* Check for a softlockup. */
+ touch_ts = __this_cpu_read(watchdog_touch_ts);
+ duration = is_softlockup(touch_ts, period_ts, now);
if (unlikely(duration)) {
/*
* Prevent multiple soft-lockup reports if one cpu is already
#include <linux/uaccess.h>
#include <linux/sched/isolation.h>
#include <linux/nmi.h>
+#include <linux/kvm_para.h>
#include "workqueue_internal.h"
{
unsigned long thresh = READ_ONCE(wq_watchdog_thresh) * HZ;
bool lockup_detected = false;
+ unsigned long now = jiffies;
struct worker_pool *pool;
int pi;
if (list_empty(&pool->worklist))
continue;
+ /*
+ * If a virtual machine is stopped by the host it can look to
+ * the watchdog like a stall.
+ */
+ kvm_check_and_clear_guest_paused();
+
/* get the latest of pool and touched timestamps */
if (pool->cpu >= 0)
touched = READ_ONCE(per_cpu(wq_watchdog_touched_cpu, pool->cpu));
ts = touched;
/* did we stall? */
- if (time_after(jiffies, ts + thresh)) {
+ if (time_after(now, ts + thresh)) {
lockup_detected = true;
pr_emerg("BUG: workqueue lockup - pool");
pr_cont_pool_info(pool);
pr_cont(" stuck for %us!\n",
- jiffies_to_msecs(jiffies - pool_ts) / 1000);
+ jiffies_to_msecs(now - pool_ts) / 1000);
}
}
obj-$(CONFIG_PLDMFW) += pldmfw/
# KUnit tests
+CFLAGS_bitfield_kunit.o := $(call cc-option,-Wframe-larger-than=10240)
obj-$(CONFIG_BITFIELD_KUNIT) += bitfield_kunit.o
obj-$(CONFIG_LIST_KUNIT_TEST) += list-test.o
obj-$(CONFIG_LINEAR_RANGES_TEST) += test_linear_ranges.o
/**
* crc64_be - Calculate bitwise big-endian ECMA-182 CRC64
* @crc: seed value for computation. 0 or (u64)~0 for a new CRC calculation,
- or the previous crc64 value if computing incrementally.
+ * or the previous crc64 value if computing incrementally.
* @p: pointer to buffer over which CRC64 is run
* @len: length of buffer @p
*/
return 0;
}
-static char *dynamic_emit_prefix(const struct _ddebug *desc, char *buf)
+static char *__dynamic_emit_prefix(const struct _ddebug *desc, char *buf)
{
int pos_after_tid;
int pos = 0;
- *buf = '\0';
-
if (desc->flags & _DPRINTK_FLAGS_INCL_TID) {
if (in_interrupt())
pos += snprintf(buf + pos, remaining(pos), "<intr> ");
return buf;
}
+static inline char *dynamic_emit_prefix(struct _ddebug *desc, char *buf)
+{
+ if (unlikely(desc->flags & _DPRINTK_FLAGS_INCL_ANY))
+ return __dynamic_emit_prefix(desc, buf);
+ return buf;
+}
+
void __dynamic_pr_debug(struct _ddebug *descriptor, const char *fmt, ...)
{
va_list args;
struct va_format vaf;
- char buf[PREFIX_SIZE];
+ char buf[PREFIX_SIZE] = "";
BUG_ON(!descriptor);
BUG_ON(!fmt);
if (!dev) {
printk(KERN_DEBUG "(NULL device *): %pV", &vaf);
} else {
- char buf[PREFIX_SIZE];
+ char buf[PREFIX_SIZE] = "";
dev_printk_emit(LOGLEVEL_DEBUG, dev, "%s%s %s: %pV",
dynamic_emit_prefix(descriptor, buf),
vaf.va = &args;
if (dev && dev->dev.parent) {
- char buf[PREFIX_SIZE];
+ char buf[PREFIX_SIZE] = "";
dev_printk_emit(LOGLEVEL_DEBUG, dev->dev.parent,
"%s%s %s %s%s: %pV",
vaf.va = &args;
if (ibdev && ibdev->dev.parent) {
- char buf[PREFIX_SIZE];
+ char buf[PREFIX_SIZE] = "";
dev_printk_emit(LOGLEVEL_DEBUG, ibdev->dev.parent,
"%s%s %s %s: %pV",
static int ddebug_proc_open(struct inode *inode, struct file *file)
{
- vpr_info("called\n");
return seq_open_private(file, &ddebug_proc_seqops,
sizeof(struct ddebug_iter));
}
wait_event_lock_irq(percpu_ref_switch_waitq, !data->confirm_switch,
percpu_ref_switch_lock);
- if (data->force_atomic || (ref->percpu_count_ptr & __PERCPU_REF_DEAD))
+ if (data->force_atomic || percpu_ref_is_dying(ref))
__percpu_ref_switch_to_atomic(ref, confirm_switch);
else
__percpu_ref_switch_to_percpu(ref);
spin_lock_irqsave(&percpu_ref_switch_lock, flags);
- WARN_ONCE(ref->percpu_count_ptr & __PERCPU_REF_DEAD,
+ WARN_ONCE(percpu_ref_is_dying(ref),
"%s called more than once on %ps!", __func__,
ref->data->release);
spin_lock_irqsave(&percpu_ref_switch_lock, flags);
- WARN_ON_ONCE(!(ref->percpu_count_ptr & __PERCPU_REF_DEAD));
+ WARN_ON_ONCE(!percpu_ref_is_dying(ref));
WARN_ON_ONCE(__ref_is_percpu(ref, &percpu_count));
ref->percpu_count_ptr &= ~__PERCPU_REF_DEAD;
static void kasan_global_oob(struct kunit *test)
{
- volatile int i = 3;
- char *p = &global_array[ARRAY_SIZE(global_array) + i];
+ /*
+ * Deliberate out-of-bounds access. To prevent CONFIG_UBSAN_LOCAL_BOUNDS
+ * from failing here and panicing the kernel, access the array via a
+ * volatile pointer, which will prevent the compiler from being able to
+ * determine the array bounds.
+ *
+ * This access uses a volatile pointer to char (char *volatile) rather
+ * than the more conventional pointer to volatile char (volatile char *)
+ * because we want to prevent the compiler from making inferences about
+ * the pointer itself (i.e. its array bounds), not the data that it
+ * refers to.
+ */
+ char *volatile array = global_array;
+ char *p = &array[ARRAY_SIZE(global_array) + 3];
/* Only generic mode instruments globals. */
KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_KASAN_GENERIC);
static void kasan_stack_oob(struct kunit *test)
{
char stack_array[10];
- volatile int i = OOB_TAG_OFF;
- char *p = &stack_array[ARRAY_SIZE(stack_array) + i];
+ /* See comment in kasan_global_oob. */
+ char *volatile array = stack_array;
+ char *p = &array[ARRAY_SIZE(stack_array) + OOB_TAG_OFF];
KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_KASAN_STACK);
{
volatile int i = 10;
char alloca_array[i];
- char *p = alloca_array - 1;
+ /* See comment in kasan_global_oob. */
+ char *volatile array = alloca_array;
+ char *p = array - 1;
/* Only generic mode instruments dynamic allocas. */
KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_KASAN_GENERIC);
{
volatile int i = 10;
char alloca_array[i];
- char *p = alloca_array + i;
+ /* See comment in kasan_global_oob. */
+ char *volatile array = alloca_array;
+ char *p = array + i;
/* Only generic mode instruments dynamic allocas. */
KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_KASAN_GENERIC);
pr_debug("Validating PMD advanced\n");
/* Align the address wrt HPAGE_PMD_SIZE */
- vaddr = (vaddr & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE;
+ vaddr &= HPAGE_PMD_MASK;
pgtable_trans_huge_deposit(mm, pmdp, pgtable);
pr_debug("Validating PUD advanced\n");
/* Align the address wrt HPAGE_PUD_SIZE */
- vaddr = (vaddr & HPAGE_PUD_MASK) + HPAGE_PUD_SIZE;
+ vaddr &= HPAGE_PUD_MASK;
set_pud_at(mm, vaddr, pudp, pud);
pudp_set_wrprotect(mm, vaddr, pudp);
FOLL_FORCE | FOLL_DUMP | FOLL_GET);
if (locked)
mmap_read_unlock(mm);
-
- if (ret == 1 && is_page_poisoned(page))
- return NULL;
-
return (ret == 1) ? page : NULL;
}
#endif /* CONFIG_ELF_CORE */
static atomic_t huge_zero_refcount;
struct page *huge_zero_page __read_mostly;
+unsigned long huge_zero_pfn __read_mostly = ~0UL;
bool transparent_hugepage_enabled(struct vm_area_struct *vma)
{
__free_pages(zero_page, compound_order(zero_page));
goto retry;
}
+ WRITE_ONCE(huge_zero_pfn, page_to_pfn(zero_page));
/* We take additional reference here. It will be put back by shrinker */
atomic_set(&huge_zero_refcount, 2);
if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) == 1) {
struct page *zero_page = xchg(&huge_zero_page, NULL);
BUG_ON(zero_page == NULL);
+ WRITE_ONCE(huge_zero_pfn, ~0UL);
__free_pages(zero_page, compound_order(zero_page));
return HPAGE_PMD_NR;
}
count_vm_event(THP_SPLIT_PMD);
if (!vma_is_anonymous(vma)) {
- _pmd = pmdp_huge_clear_flush_notify(vma, haddr, pmd);
+ old_pmd = pmdp_huge_clear_flush_notify(vma, haddr, pmd);
/*
* We are going to unmap this huge page. So
* just go ahead and zap it
zap_deposited_table(mm, pmd);
if (vma_is_special_huge(vma))
return;
- page = pmd_page(_pmd);
- if (!PageDirty(page) && pmd_dirty(_pmd))
- set_page_dirty(page);
- if (!PageReferenced(page) && pmd_young(_pmd))
- SetPageReferenced(page);
- page_remove_rmap(page, true);
- put_page(page);
+ if (unlikely(is_pmd_migration_entry(old_pmd))) {
+ swp_entry_t entry;
+
+ entry = pmd_to_swp_entry(old_pmd);
+ page = migration_entry_to_page(entry);
+ } else {
+ page = pmd_page(old_pmd);
+ if (!PageDirty(page) && pmd_dirty(old_pmd))
+ set_page_dirty(page);
+ if (!PageReferenced(page) && pmd_young(old_pmd))
+ SetPageReferenced(page);
+ page_remove_rmap(page, true);
+ put_page(page);
+ }
add_mm_counter(mm, mm_counter_file(page), -HPAGE_PMD_NR);
return;
- } else if (pmd_trans_huge(*pmd) && is_huge_zero_pmd(*pmd)) {
+ }
+
+ if (is_huge_zero_pmd(*pmd)) {
/*
* FIXME: Do we want to invalidate secondary mmu by calling
* mmu_notifier_invalidate_range() see comments below inside
static void unmap_page(struct page *page)
{
- enum ttu_flags ttu_flags = TTU_IGNORE_MLOCK |
+ enum ttu_flags ttu_flags = TTU_IGNORE_MLOCK | TTU_SYNC |
TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD;
- bool unmap_success;
VM_BUG_ON_PAGE(!PageHead(page), page);
if (PageAnon(page))
ttu_flags |= TTU_SPLIT_FREEZE;
- unmap_success = try_to_unmap(page, ttu_flags);
- VM_BUG_ON_PAGE(!unmap_success, page);
+ try_to_unmap(page, ttu_flags);
+
+ VM_WARN_ON_ONCE_PAGE(page_mapped(page), page);
}
static void remap_page(struct page *page, unsigned int nr)
struct deferred_split *ds_queue = get_deferred_split_queue(head);
struct anon_vma *anon_vma = NULL;
struct address_space *mapping = NULL;
- int count, mapcount, extra_pins, ret;
+ int extra_pins, ret;
pgoff_t end;
VM_BUG_ON_PAGE(is_huge_zero_page(head), head);
}
unmap_page(head);
- VM_BUG_ON_PAGE(compound_mapcount(head), head);
/* block interrupt reentry in xa_lock and spinlock */
local_irq_disable();
/* Prevent deferred_split_scan() touching ->_refcount */
spin_lock(&ds_queue->split_queue_lock);
- count = page_count(head);
- mapcount = total_mapcount(head);
- if (!mapcount && page_ref_freeze(head, 1 + extra_pins)) {
+ if (page_ref_freeze(head, 1 + extra_pins)) {
if (!list_empty(page_deferred_list(head))) {
ds_queue->split_queue_len--;
list_del(page_deferred_list(head));
__split_huge_page(page, list, end);
ret = 0;
} else {
- if (IS_ENABLED(CONFIG_DEBUG_VM) && mapcount) {
- pr_alert("total_mapcount: %u, page_count(): %u\n",
- mapcount, count);
- if (PageTail(page))
- dump_page(head, NULL);
- dump_page(page, "total_mapcount(head) > 0");
- BUG();
- }
spin_unlock(&ds_queue->split_queue_lock);
-fail: if (mapping)
+fail:
+ if (mapping)
xa_unlock(&mapping->i_pages);
local_irq_enable();
remap_page(head, thp_nr_pages(head));
SetPageHWPoison(page);
ClearPageHWPoison(head);
}
- remove_hugetlb_page(h, page, false);
+ remove_hugetlb_page(h, head, false);
h->max_huge_pages--;
spin_unlock_irq(&hugetlb_lock);
update_and_free_page(h, head);
* be restored when a newly allocated huge page must be freed. It is
* to be called after calling vma_needs_reservation to determine if a
* reservation exists.
+ *
+ * vma_del_reservation is used in error paths where an entry in the reserve
+ * map was created during huge page allocation and must be removed. It is to
+ * be called after calling vma_needs_reservation to determine if a reservation
+ * exists.
*/
enum vma_resv_mode {
VMA_NEEDS_RESV,
VMA_COMMIT_RESV,
VMA_END_RESV,
VMA_ADD_RESV,
+ VMA_DEL_RESV,
};
static long __vma_reservation_common(struct hstate *h,
struct vm_area_struct *vma, unsigned long addr,
ret = region_del(resv, idx, idx + 1);
}
break;
+ case VMA_DEL_RESV:
+ if (vma->vm_flags & VM_MAYSHARE) {
+ region_abort(resv, idx, idx + 1, 1);
+ ret = region_del(resv, idx, idx + 1);
+ } else {
+ ret = region_add(resv, idx, idx + 1, 1, NULL, NULL);
+ /* region_add calls of range 1 should never fail. */
+ VM_BUG_ON(ret < 0);
+ }
+ break;
default:
BUG();
}
- if (vma->vm_flags & VM_MAYSHARE)
+ if (vma->vm_flags & VM_MAYSHARE || mode == VMA_DEL_RESV)
return ret;
/*
* We know private mapping must have HPAGE_RESV_OWNER set.
return __vma_reservation_common(h, vma, addr, VMA_ADD_RESV);
}
+static long vma_del_reservation(struct hstate *h,
+ struct vm_area_struct *vma, unsigned long addr)
+{
+ return __vma_reservation_common(h, vma, addr, VMA_DEL_RESV);
+}
+
/*
- * This routine is called to restore a reservation on error paths. In the
- * specific error paths, a huge page was allocated (via alloc_huge_page)
- * and is about to be freed. If a reservation for the page existed,
- * alloc_huge_page would have consumed the reservation and set
- * HPageRestoreReserve in the newly allocated page. When the page is freed
- * via free_huge_page, the global reservation count will be incremented if
- * HPageRestoreReserve is set. However, free_huge_page can not adjust the
- * reserve map. Adjust the reserve map here to be consistent with global
- * reserve count adjustments to be made by free_huge_page.
+ * This routine is called to restore reservation information on error paths.
+ * It should ONLY be called for pages allocated via alloc_huge_page(), and
+ * the hugetlb mutex should remain held when calling this routine.
+ *
+ * It handles two specific cases:
+ * 1) A reservation was in place and the page consumed the reservation.
+ * HPageRestoreReserve is set in the page.
+ * 2) No reservation was in place for the page, so HPageRestoreReserve is
+ * not set. However, alloc_huge_page always updates the reserve map.
+ *
+ * In case 1, free_huge_page later in the error path will increment the
+ * global reserve count. But, free_huge_page does not have enough context
+ * to adjust the reservation map. This case deals primarily with private
+ * mappings. Adjust the reserve map here to be consistent with global
+ * reserve count adjustments to be made by free_huge_page. Make sure the
+ * reserve map indicates there is a reservation present.
+ *
+ * In case 2, simply undo reserve map modifications done by alloc_huge_page.
*/
-static void restore_reserve_on_error(struct hstate *h,
- struct vm_area_struct *vma, unsigned long address,
- struct page *page)
+void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma,
+ unsigned long address, struct page *page)
{
- if (unlikely(HPageRestoreReserve(page))) {
- long rc = vma_needs_reservation(h, vma, address);
+ long rc = vma_needs_reservation(h, vma, address);
- if (unlikely(rc < 0)) {
+ if (HPageRestoreReserve(page)) {
+ if (unlikely(rc < 0))
/*
* Rare out of memory condition in reserve map
* manipulation. Clear HPageRestoreReserve so that
* accounting of reserve counts.
*/
ClearHPageRestoreReserve(page);
- } else if (rc) {
- rc = vma_add_reservation(h, vma, address);
- if (unlikely(rc < 0))
+ else if (rc)
+ (void)vma_add_reservation(h, vma, address);
+ else
+ vma_end_reservation(h, vma, address);
+ } else {
+ if (!rc) {
+ /*
+ * This indicates there is an entry in the reserve map
+ * added by alloc_huge_page. We know it was added
+ * before the alloc_huge_page call, otherwise
+ * HPageRestoreReserve would be set on the page.
+ * Remove the entry so that a subsequent allocation
+ * does not consume a reservation.
+ */
+ rc = vma_del_reservation(h, vma, address);
+ if (rc < 0)
/*
- * See above comment about rare out of
- * memory condition.
+ * VERY rare out of memory condition. Since
+ * we can not delete the entry, set
+ * HPageRestoreReserve so that the reserve
+ * count will be incremented when the page
+ * is freed. This reserve will be consumed
+ * on a subsequent allocation.
*/
- ClearHPageRestoreReserve(page);
+ SetHPageRestoreReserve(page);
+ } else if (rc < 0) {
+ /*
+ * Rare out of memory condition from
+ * vma_needs_reservation call. Memory allocation is
+ * only attempted if a new entry is needed. Therefore,
+ * this implies there is not an entry in the
+ * reserve map.
+ *
+ * For shared mappings, no entry in the map indicates
+ * no reservation. We are done.
+ */
+ if (!(vma->vm_flags & VM_MAYSHARE))
+ /*
+ * For private mappings, no entry indicates
+ * a reservation is present. Since we can
+ * not add an entry, set SetHPageRestoreReserve
+ * on the page so reserve count will be
+ * incremented when freed. This reserve will
+ * be consumed on a subsequent allocation.
+ */
+ SetHPageRestoreReserve(page);
} else
- vma_end_reservation(h, vma, address);
+ /*
+ * No reservation present, do nothing
+ */
+ vma_end_reservation(h, vma, address);
}
}
spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
entry = huge_ptep_get(src_pte);
if (!pte_same(src_pte_old, entry)) {
+ restore_reserve_on_error(h, vma, addr,
+ new);
put_page(new);
/* dst_entry won't change as in child */
goto again;
* See Documentation/vm/mmu_notifier.rst
*/
huge_ptep_set_wrprotect(src, addr, src_pte);
+ entry = huge_pte_wrprotect(entry);
}
page_dup_rmap(ptepage, true);
if (!page)
goto out;
} else if (!*pagep) {
- ret = -ENOMEM;
+ /* If a page already exists, then it's UFFDIO_COPY for
+ * a non-missing case. Return -EEXIST.
+ */
+ if (vm_shared &&
+ hugetlbfs_pagecache_present(h, dst_vma, dst_addr)) {
+ ret = -EEXIST;
+ goto out;
+ }
+
page = alloc_huge_page(dst_vma, dst_addr, 0);
- if (IS_ERR(page))
+ if (IS_ERR(page)) {
+ ret = -ENOMEM;
goto out;
+ }
ret = copy_huge_page_from_user(page,
(const void __user *) src_addr,
if (vm_shared || is_continue)
unlock_page(page);
out_release_nounlock:
+ restore_reserve_on_error(h, dst_vma, dst_addr, page);
put_page(page);
goto out;
}
return ret;
}
+int get_hwpoison_huge_page(struct page *page, bool *hugetlb)
+{
+ int ret = 0;
+
+ *hugetlb = false;
+ spin_lock_irq(&hugetlb_lock);
+ if (PageHeadHuge(page)) {
+ *hugetlb = true;
+ if (HPageFreed(page) || HPageMigratable(page))
+ ret = get_page_unless_zero(page);
+ }
+ spin_unlock_irq(&hugetlb_lock);
+ return ret;
+}
+
void putback_active_hugepage(struct page *page)
{
spin_lock_irq(&hugetlb_lock);
set_page_count(page, 1);
}
-/*
- * When kernel touch the user page, the user page may be have been marked
- * poison but still mapped in user space, if without this page, the kernel
- * can guarantee the data integrity and operation success, the kernel is
- * better to check the posion status and avoid touching it, be good not to
- * panic, coredump for process fatal signal is a sample case matching this
- * scenario. Or if kernel can't guarantee the data integrity, it's better
- * not to call this function, let kernel touch the poison page and get to
- * panic.
- */
-static inline bool is_page_poisoned(struct page *page)
-{
- if (PageHWPoison(page))
- return true;
- else if (PageHuge(page) && PageHWPoison(compound_head(page)))
- return true;
-
- return false;
-}
-
extern unsigned long highest_memmap_pfn;
/*
extern pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma);
/*
- * At what user virtual address is page expected in @vma?
+ * At what user virtual address is page expected in vma?
+ * Returns -EFAULT if all of the page is outside the range of vma.
+ * If page is a compound head, the entire compound page is considered.
*/
static inline unsigned long
-__vma_address(struct page *page, struct vm_area_struct *vma)
+vma_address(struct page *page, struct vm_area_struct *vma)
{
- pgoff_t pgoff = page_to_pgoff(page);
- return vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
+ pgoff_t pgoff;
+ unsigned long address;
+
+ VM_BUG_ON_PAGE(PageKsm(page), page); /* KSM page->index unusable */
+ pgoff = page_to_pgoff(page);
+ if (pgoff >= vma->vm_pgoff) {
+ address = vma->vm_start +
+ ((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
+ /* Check for address beyond vma (or wrapped through 0?) */
+ if (address < vma->vm_start || address >= vma->vm_end)
+ address = -EFAULT;
+ } else if (PageHead(page) &&
+ pgoff + compound_nr(page) - 1 >= vma->vm_pgoff) {
+ /* Test above avoids possibility of wrap to 0 on 32-bit */
+ address = vma->vm_start;
+ } else {
+ address = -EFAULT;
+ }
+ return address;
}
+/*
+ * Then at what user virtual address will none of the page be found in vma?
+ * Assumes that vma_address() already returned a good starting address.
+ * If page is a compound head, the entire compound page is considered.
+ */
static inline unsigned long
-vma_address(struct page *page, struct vm_area_struct *vma)
+vma_address_end(struct page *page, struct vm_area_struct *vma)
{
- unsigned long start, end;
-
- start = __vma_address(page, vma);
- end = start + thp_size(page) - PAGE_SIZE;
-
- /* page should be within @vma mapping range */
- VM_BUG_ON_VMA(end < vma->vm_start || start >= vma->vm_end, vma);
-
- return max(start, vma->vm_start);
+ pgoff_t pgoff;
+ unsigned long address;
+
+ VM_BUG_ON_PAGE(PageKsm(page), page); /* KSM page->index unusable */
+ pgoff = page_to_pgoff(page) + compound_nr(page);
+ address = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
+ /* Check for address beyond vma (or wrapped through 0?) */
+ if (address < vma->vm_start || address > vma->vm_end)
+ address = vma->vm_end;
+ return address;
}
static inline struct file *maybe_unlock_mmap_for_io(struct vm_fault *vmf,
#include "pgalloc-track.h"
#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP
-static bool __ro_after_init iomap_max_page_shift = PAGE_SHIFT;
+static unsigned int __ro_after_init iomap_max_page_shift = BITS_PER_LONG - 1;
static int __init set_nohugeiomap(char *str)
{
- iomap_max_page_shift = P4D_SHIFT;
+ iomap_max_page_shift = PAGE_SHIFT;
return 0;
}
early_param("nohugeiomap", set_nohugeiomap);
#else /* CONFIG_HAVE_ARCH_HUGE_VMAP */
-static const bool iomap_max_page_shift = PAGE_SHIFT;
+static const unsigned int iomap_max_page_shift = PAGE_SHIFT;
#endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */
int ioremap_page_range(unsigned long addr,
/**
* kasan_populate_early_shadow - populate shadow memory region with
* kasan_early_shadow_page
- * @shadow_start - start of the memory range to populate
- * @shadow_end - end of the memory range to populate
+ * @shadow_start: start of the memory range to populate
+ * @shadow_end: end of the memory range to populate
*/
int __ref kasan_populate_early_shadow(const void *shadow_start,
const void *shadow_end)
* During low activity with no allocations we might wait a
* while; let's avoid the hung task warning.
*/
- wait_event_timeout(allocation_wait, atomic_read(&kfence_allocation_gate),
- sysctl_hung_task_timeout_secs * HZ / 2);
+ wait_event_idle_timeout(allocation_wait, atomic_read(&kfence_allocation_gate),
+ sysctl_hung_task_timeout_secs * HZ / 2);
} else {
- wait_event(allocation_wait, atomic_read(&kfence_allocation_gate));
+ wait_event_idle(allocation_wait, atomic_read(&kfence_allocation_gate));
}
/* Disable static key and reset timer. */
struct page *page;
stable_node = rmap_item->head;
- page = get_ksm_page(stable_node, GET_KSM_PAGE_NOLOCK);
+ page = get_ksm_page(stable_node, GET_KSM_PAGE_LOCK);
if (!page)
goto out;
hlist_del(&rmap_item->hlist);
+ unlock_page(page);
put_page(page);
if (!hlist_empty(&stable_node->hlist))
return (result == MF_RECOVERED || result == MF_DELAYED) ? 0 : -EBUSY;
}
+/*
+ * Return true if a page type of a given page is supported by hwpoison
+ * mechanism (while handling could fail), otherwise false. This function
+ * does not return true for hugetlb or device memory pages, so it's assumed
+ * to be called only in the context where we never have such pages.
+ */
+static inline bool HWPoisonHandlable(struct page *page)
+{
+ return PageLRU(page) || __PageMovable(page);
+}
+
/**
* __get_hwpoison_page() - Get refcount for memory error handling:
* @page: raw error page (hit by memory error)
static int __get_hwpoison_page(struct page *page)
{
struct page *head = compound_head(page);
+ int ret = 0;
+ bool hugetlb = false;
+
+ ret = get_hwpoison_huge_page(head, &hugetlb);
+ if (hugetlb)
+ return ret;
- if (!PageHuge(head) && PageTransHuge(head)) {
+ /*
+ * This check prevents from calling get_hwpoison_unless_zero()
+ * for any unsupported type of page in order to reduce the risk of
+ * unexpected races caused by taking a page refcount.
+ */
+ if (!HWPoisonHandlable(head))
+ return 0;
+
+ if (PageTransHuge(head)) {
/*
* Non anonymous thp exists only in allocation/free time. We
* can't handle such a case correctly, so let's give it up.
ret = -EIO;
}
} else {
- if (PageHuge(p) || PageLRU(p) || __PageMovable(p)) {
+ if (PageHuge(p) || HWPoisonHandlable(p)) {
ret = 1;
} else {
/*
return 0;
}
- if (!PageTransTail(p) && !PageLRU(p))
+ /*
+ * __munlock_pagevec may clear a writeback page's LRU flag without
+ * page_lock. We need wait writeback completion for this page or it
+ * may trigger vfs BUG while evict inode.
+ */
+ if (!PageTransTail(p) && !PageLRU(p) && !PageWriteback(p))
goto identify_page_state;
/*
else if (zap_huge_pmd(tlb, vma, pmd, addr))
goto next;
/* fall through */
+ } else if (details && details->single_page &&
+ PageTransCompound(details->single_page) &&
+ next - addr == HPAGE_PMD_SIZE && pmd_none(*pmd)) {
+ spinlock_t *ptl = pmd_lock(tlb->mm, pmd);
+ /*
+ * Take and drop THP pmd lock so that we cannot return
+ * prematurely, while zap_huge_pmd() has cleared *pmd,
+ * but not yet decremented compound_mapcount().
+ */
+ spin_unlock(ptl);
}
+
/*
* Here there can be other concurrent MADV_DONTNEED or
* trans huge page faults running, and if the pmd is
}
flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
entry = mk_pte(new_page, vma->vm_page_prot);
+ entry = pte_sw_mkyoung(entry);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
/*
}
/**
+ * unmap_mapping_page() - Unmap single page from processes.
+ * @page: The locked page to be unmapped.
+ *
+ * Unmap this page from any userspace process which still has it mmaped.
+ * Typically, for efficiency, the range of nearby pages has already been
+ * unmapped by unmap_mapping_pages() or unmap_mapping_range(). But once
+ * truncation or invalidation holds the lock on a page, it may find that
+ * the page has been remapped again: and then uses unmap_mapping_page()
+ * to unmap it finally.
+ */
+void unmap_mapping_page(struct page *page)
+{
+ struct address_space *mapping = page->mapping;
+ struct zap_details details = { };
+
+ VM_BUG_ON(!PageLocked(page));
+ VM_BUG_ON(PageTail(page));
+
+ details.check_mapping = mapping;
+ details.first_index = page->index;
+ details.last_index = page->index + thp_nr_pages(page) - 1;
+ details.single_page = page;
+
+ i_mmap_lock_write(mapping);
+ if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)))
+ unmap_mapping_range_tree(&mapping->i_mmap, &details);
+ i_mmap_unlock_write(mapping);
+}
+
+/**
* unmap_mapping_pages() - Unmap pages from processes.
* @mapping: The address space containing pages to be unmapped.
* @start: Index of first page to be unmapped.
__SetPageUptodate(page);
entry = mk_pte(page, vma->vm_page_prot);
+ entry = pte_sw_mkyoung(entry);
if (vma->vm_flags & VM_WRITE)
entry = pte_mkwrite(pte_mkdirty(entry));
if (prefault && arch_wants_old_prefaulted_pte())
entry = pte_mkold(entry);
+ else
+ entry = pte_sw_mkyoung(entry);
if (write)
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
goto out;
page = migration_entry_to_page(entry);
+ page = compound_head(page);
/*
* Once page cache replacement of page migration started, page_count
del_page_from_free_list(page_head, zone, page_order);
break_down_buddy_pages(zone, page_head, page, 0,
page_order, migratetype);
+ if (!is_migrate_isolate(migratetype))
+ __mod_zone_freepage_state(zone, -1, migratetype);
ret = true;
break;
}
pvmw->ptl = NULL;
}
} else if (!pmd_present(pmde)) {
+ /*
+ * If PVMW_SYNC, take and drop THP pmd lock so that we
+ * cannot return prematurely, while zap_huge_pmd() has
+ * cleared *pmd but not decremented compound_mapcount().
+ */
+ if ((pvmw->flags & PVMW_SYNC) &&
+ PageTransCompound(pvmw->page)) {
+ spinlock_t *ptl = pmd_lock(mm, pvmw->pmd);
+
+ spin_unlock(ptl);
+ }
return false;
}
if (!map_pte(pvmw))
goto next_pte;
while (1) {
+ unsigned long end;
+
if (check_pte(pvmw))
return true;
next_pte:
/* Seek to next pte only makes sense for THP */
if (!PageTransHuge(pvmw->page) || PageHuge(pvmw->page))
return not_found(pvmw);
+ end = vma_address_end(pvmw->page, pvmw->vma);
do {
pvmw->address += PAGE_SIZE;
- if (pvmw->address >= pvmw->vma->vm_end ||
- pvmw->address >=
- __vma_address(pvmw->page, pvmw->vma) +
- thp_size(pvmw->page))
+ if (pvmw->address >= end)
return not_found(pvmw);
/* Did we cross page table boundary? */
if (pvmw->address % PMD_SIZE == 0) {
.vma = vma,
.flags = PVMW_SYNC,
};
- unsigned long start, end;
-
- start = __vma_address(page, vma);
- end = start + thp_size(page) - PAGE_SIZE;
- if (unlikely(end < vma->vm_start || start >= vma->vm_end))
+ pvmw.address = vma_address(page, vma);
+ if (pvmw.address == -EFAULT)
return 0;
- pvmw.address = max(start, vma->vm_start);
if (!page_vma_mapped_walk(&pvmw))
return 0;
page_vma_mapped_walk_done(&pvmw);
{
pmd_t pmd;
VM_BUG_ON(address & ~HPAGE_PMD_MASK);
- VM_BUG_ON(!pmd_present(*pmdp));
- /* Below assumes pmd_present() is true */
- VM_BUG_ON(!pmd_trans_huge(*pmdp) && !pmd_devmap(*pmdp));
+ VM_BUG_ON(pmd_present(*pmdp) && !pmd_trans_huge(*pmdp) &&
+ !pmd_devmap(*pmdp));
pmd = pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp);
flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
return pmd;
*/
unsigned long page_address_in_vma(struct page *page, struct vm_area_struct *vma)
{
- unsigned long address;
if (PageAnon(page)) {
struct anon_vma *page__anon_vma = page_anon_vma(page);
/*
if (!vma->anon_vma || !page__anon_vma ||
vma->anon_vma->root != page__anon_vma->root)
return -EFAULT;
- } else if (page->mapping) {
- if (!vma->vm_file || vma->vm_file->f_mapping != page->mapping)
- return -EFAULT;
- } else
+ } else if (!vma->vm_file) {
return -EFAULT;
- address = __vma_address(page, vma);
- if (unlikely(address < vma->vm_start || address >= vma->vm_end))
+ } else if (vma->vm_file->f_mapping != compound_head(page)->mapping) {
return -EFAULT;
- return address;
+ }
+
+ return vma_address(page, vma);
}
pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address)
*/
mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_PAGE,
0, vma, vma->vm_mm, address,
- min(vma->vm_end, address + page_size(page)));
+ vma_address_end(page, vma));
mmu_notifier_invalidate_range_start(&range);
while (page_vma_mapped_walk(&pvmw)) {
struct mmu_notifier_range range;
enum ttu_flags flags = (enum ttu_flags)(long)arg;
+ /*
+ * When racing against e.g. zap_pte_range() on another cpu,
+ * in between its ptep_get_and_clear_full() and page_remove_rmap(),
+ * try_to_unmap() may return false when it is about to become true,
+ * if page table locking is skipped: use TTU_SYNC to wait for that.
+ */
+ if (flags & TTU_SYNC)
+ pvmw.flags = PVMW_SYNC;
+
/* munlock has nothing to gain from examining un-locked vmas */
if ((flags & TTU_MUNLOCK) && !(vma->vm_flags & VM_LOCKED))
return true;
* Note that the page can not be free in this function as call of
* try_to_unmap() must hold a reference on the page.
*/
+ range.end = PageKsm(page) ?
+ address + PAGE_SIZE : vma_address_end(page, vma);
mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, vma->vm_mm,
- address,
- min(vma->vm_end, address + page_size(page)));
+ address, range.end);
if (PageHuge(page)) {
/*
* If sharing is possible, start and end will be adjusted
else
rmap_walk(page, &rwc);
- return !page_mapcount(page) ? true : false;
+ /*
+ * When racing against e.g. zap_pte_range() on another cpu,
+ * in between its ptep_get_and_clear_full() and page_remove_rmap(),
+ * try_to_unmap() may return false when it is about to become true,
+ * if page table locking is skipped: use TTU_SYNC to wait for that.
+ */
+ return !page_mapcount(page);
}
/**
struct vm_area_struct *vma = avc->vma;
unsigned long address = vma_address(page, vma);
+ VM_BUG_ON_VMA(address == -EFAULT, vma);
cond_resched();
if (rwc->invalid_vma && rwc->invalid_vma(vma, rwc->arg))
pgoff_start, pgoff_end) {
unsigned long address = vma_address(page, vma);
+ VM_BUG_ON_VMA(address == -EFAULT, vma);
cond_resched();
if (rwc->invalid_vma && rwc->invalid_vma(vma, rwc->arg))
static int shmem_mmap(struct file *file, struct vm_area_struct *vma)
{
struct shmem_inode_info *info = SHMEM_I(file_inode(file));
+ int ret;
- if (info->seals & F_SEAL_FUTURE_WRITE) {
- /*
- * New PROT_WRITE and MAP_SHARED mmaps are not allowed when
- * "future write" seal active.
- */
- if ((vma->vm_flags & VM_SHARED) && (vma->vm_flags & VM_WRITE))
- return -EPERM;
-
- /*
- * Since an F_SEAL_FUTURE_WRITE sealed memfd can be mapped as
- * MAP_SHARED and read-only, take care to not allow mprotect to
- * revert protections on such mappings. Do this only for shared
- * mappings. For private mappings, don't need to mask
- * VM_MAYWRITE as we still want them to be COW-writable.
- */
- if (vma->vm_flags & VM_SHARED)
- vma->vm_flags &= ~(VM_MAYWRITE);
- }
+ ret = seal_check_future_write(info->seals, vma);
+ if (ret)
+ return ret;
/* arm64 - allow memory tagging on RAM-based files */
vma->vm_flags |= VM_MTE_ALLOWED;
pgoff_t offset, max_off;
ret = -ENOMEM;
- if (!shmem_inode_acct_block(inode, 1))
+ if (!shmem_inode_acct_block(inode, 1)) {
+ /*
+ * We may have got a page, returned -ENOENT triggering a retry,
+ * and now we find ourselves with -ENOMEM. Release the page, to
+ * avoid a BUG_ON in our caller.
+ */
+ if (unlikely(*pagep)) {
+ put_page(*pagep);
+ *pagep = NULL;
+ }
goto out;
+ }
if (!*pagep) {
page = shmem_alloc_page(gfp, info, pgoff);
DECLARE_STATIC_KEY_FALSE(page_alloc_shuffle_key);
extern void __shuffle_free_memory(pg_data_t *pgdat);
extern bool shuffle_pick_tail(void);
-static inline void shuffle_free_memory(pg_data_t *pgdat)
+static inline void __meminit shuffle_free_memory(pg_data_t *pgdat)
{
if (!static_branch_unlikely(&page_alloc_shuffle_key))
return;
}
extern void __shuffle_zone(struct zone *z);
-static inline void shuffle_zone(struct zone *z)
+static inline void __meminit shuffle_zone(struct zone *z)
{
if (!static_branch_unlikely(&page_alloc_shuffle_key))
return;
#ifdef CONFIG_DEBUG_VM
static int kmem_cache_sanity_check(const char *name, unsigned int size)
{
- if (!name || in_interrupt() || size < sizeof(void *) ||
- size > KMALLOC_MAX_SIZE) {
+ if (!name || in_interrupt() || size > KMALLOC_MAX_SIZE) {
pr_err("kmem_cache_create(%s) integrity check failed\n", name);
return -EINVAL;
}
const char *cache_name;
int err;
+#ifdef CONFIG_SLUB_DEBUG
+ /*
+ * If no slub_debug was enabled globally, the static key is not yet
+ * enabled by setup_slub_debug(). Enable it if the cache is being
+ * created with any of the debugging flags passed explicitly.
+ */
+ if (flags & SLAB_DEBUG_FLAGS)
+ static_branch_enable(&slub_debug_enabled);
+#endif
+
mutex_lock(&slab_mutex);
err = kmem_cache_sanity_check(name, size);
#include <linux/module.h>
#include <linux/bit_spinlock.h>
#include <linux/interrupt.h>
+#include <linux/swab.h>
#include <linux/bitops.h>
#include <linux/slab.h>
#include "slab.h"
if (!debug_pagealloc_enabled_static())
return get_freepointer(s, object);
+ object = kasan_reset_tag(object);
freepointer_addr = (unsigned long)object + s->offset;
copy_from_kernel_nofault(&p, (void **)freepointer_addr, sizeof(p));
return freelist_ptr(s, p, freepointer_addr);
p, p - addr, get_freepointer(s, p));
if (s->flags & SLAB_RED_ZONE)
- print_section(KERN_ERR, "Redzone ", p - s->red_left_pad,
+ print_section(KERN_ERR, "Redzone ", p - s->red_left_pad,
s->red_left_pad);
else if (p > addr + 16)
print_section(KERN_ERR, "Bytes b4 ", p - 16, 16);
- print_section(KERN_ERR, "Object ", p,
+ print_section(KERN_ERR, "Object ", p,
min_t(unsigned int, s->object_size, PAGE_SIZE));
if (s->flags & SLAB_RED_ZONE)
- print_section(KERN_ERR, "Redzone ", p + s->object_size,
+ print_section(KERN_ERR, "Redzone ", p + s->object_size,
s->inuse - s->object_size);
off = get_info_end(s);
if (off != size_from_object(s))
/* Beginning of the filler is the free pointer */
- print_section(KERN_ERR, "Padding ", p + off,
+ print_section(KERN_ERR, "Padding ", p + off,
size_from_object(s) - off);
dump_stack();
u8 *endobject = object + s->object_size;
if (s->flags & SLAB_RED_ZONE) {
- if (!check_bytes_and_report(s, page, object, "Redzone",
+ if (!check_bytes_and_report(s, page, object, "Left Redzone",
object - s->red_left_pad, val, s->red_left_pad))
return 0;
- if (!check_bytes_and_report(s, page, object, "Redzone",
+ if (!check_bytes_and_report(s, page, object, "Right Redzone",
endobject, val, s->inuse - s->object_size))
return 0;
} else {
if (val != SLUB_RED_ACTIVE && (s->flags & __OBJECT_POISON) &&
(!check_bytes_and_report(s, page, p, "Poison", p,
POISON_FREE, s->object_size - 1) ||
- !check_bytes_and_report(s, page, p, "Poison",
+ !check_bytes_and_report(s, page, p, "End Poison",
p + s->object_size - 1, POISON_END, 1)))
return 0;
/*
{
slab_flags_t flags = s->flags;
unsigned int size = s->object_size;
- unsigned int freepointer_area;
unsigned int order;
/*
* the possible location of the free pointer.
*/
size = ALIGN(size, sizeof(void *));
- /*
- * This is the area of the object where a freepointer can be
- * safely written. If redzoning adds more to the inuse size, we
- * can't use that portion for writing the freepointer, so
- * s->offset must be limited within this for the general case.
- */
- freepointer_area = size;
#ifdef CONFIG_SLUB_DEBUG
/*
/*
* With that we have determined the number of bytes in actual use
- * by the object. This is the potential offset to the free pointer.
+ * by the object and redzoning.
*/
s->inuse = size;
- if (((flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON)) ||
- s->ctor)) {
+ if ((flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON)) ||
+ ((flags & SLAB_RED_ZONE) && s->object_size < sizeof(void *)) ||
+ s->ctor) {
/*
* Relocate free pointer after the object if it is not
* permitted to overwrite the first word of the object on
* kmem_cache_free.
*
* This is the case if we do RCU, have a constructor or
- * destructor or are poisoning the objects.
+ * destructor, are poisoning the objects, or are
+ * redzoning an object smaller than sizeof(void *).
*
* The assumption that s->offset >= s->inuse means free
* pointer is outside of the object is used in the
*/
s->offset = size;
size += sizeof(void *);
- } else if (freepointer_area > sizeof(void *)) {
+ } else {
/*
* Store freelist pointer near middle of object to keep
* it away from the edges of the object to avoid small
* sized over/underflows from neighboring allocations.
*/
- s->offset = ALIGN(freepointer_area / 2, sizeof(void *));
+ s->offset = ALIGN_DOWN(s->object_size / 2, sizeof(void *));
}
#ifdef CONFIG_SLUB_DEBUG
static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags)
{
-#ifdef CONFIG_SLUB_DEBUG
- /*
- * If no slub_debug was enabled globally, the static key is not yet
- * enabled by setup_slub_debug(). Enable it if the cache is being
- * created with any of the debugging flags passed explicitly.
- */
- if (flags & SLAB_DEBUG_FLAGS)
- static_branch_enable(&slub_debug_enabled);
-#endif
s->flags = kmem_cache_flags(s->size, flags, s->name);
#ifdef CONFIG_SLAB_FREELIST_HARDENED
s->random = get_random_long();
return sizeof(struct mem_section_usage) + usemap_size();
}
+static inline phys_addr_t pgdat_to_phys(struct pglist_data *pgdat)
+{
+#ifndef CONFIG_NEED_MULTIPLE_NODES
+ return __pa_symbol(pgdat);
+#else
+ return __pa(pgdat);
+#endif
+}
+
#ifdef CONFIG_MEMORY_HOTREMOVE
static struct mem_section_usage * __init
sparse_early_usemaps_alloc_pgdat_section(struct pglist_data *pgdat,
* from the same section as the pgdat where possible to avoid
* this problem.
*/
- goal = __pa(pgdat) & (PAGE_SECTION_MASK << PAGE_SHIFT);
+ goal = pgdat_to_phys(pgdat) & (PAGE_SECTION_MASK << PAGE_SHIFT);
limit = goal + (1UL << PA_SECTION_SHIFT);
nid = early_pfn_to_nid(goal >> PAGE_SHIFT);
again:
}
usemap_snr = pfn_to_section_nr(__pa(usage) >> PAGE_SHIFT);
- pgdat_snr = pfn_to_section_nr(__pa(pgdat) >> PAGE_SHIFT);
+ pgdat_snr = pfn_to_section_nr(pgdat_to_phys(pgdat) >> PAGE_SHIFT);
if (usemap_snr == pgdat_snr)
return;
static inline int pte_same_as_swp(pte_t pte, pte_t swp_pte)
{
- return pte_same(pte_swp_clear_soft_dirty(pte), swp_pte);
+ return pte_same(pte_swp_clear_flags(pte), swp_pte);
}
/*
* its lock, b) when a concurrent invalidate_mapping_pages got there first and
* c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
*/
-static void
-truncate_cleanup_page(struct address_space *mapping, struct page *page)
+static void truncate_cleanup_page(struct page *page)
{
- if (page_mapped(page)) {
- unsigned int nr = thp_nr_pages(page);
- unmap_mapping_pages(mapping, page->index, nr, false);
- }
+ if (page_mapped(page))
+ unmap_mapping_page(page);
if (page_has_private(page))
do_invalidatepage(page, 0, thp_size(page));
if (page->mapping != mapping)
return -EIO;
- truncate_cleanup_page(mapping, page);
+ truncate_cleanup_page(page);
delete_from_page_cache(page);
return 0;
}
index = indices[pagevec_count(&pvec) - 1] + 1;
truncate_exceptional_pvec_entries(mapping, &pvec, indices);
for (i = 0; i < pagevec_count(&pvec); i++)
- truncate_cleanup_page(mapping, pvec.pages[i]);
+ truncate_cleanup_page(pvec.pages[i]);
delete_from_page_cache_batch(mapping, &pvec);
for (i = 0; i < pagevec_count(&pvec); i++)
unlock_page(pvec.pages[i]);
continue;
}
+ if (!did_range_unmap && page_mapped(page)) {
+ /*
+ * If page is mapped, before taking its lock,
+ * zap the rest of the file in one hit.
+ */
+ unmap_mapping_pages(mapping, index,
+ (1 + end - index), false);
+ did_range_unmap = 1;
+ }
+
lock_page(page);
WARN_ON(page_to_index(page) != index);
if (page->mapping != mapping) {
continue;
}
wait_on_page_writeback(page);
- if (page_mapped(page)) {
- if (!did_range_unmap) {
- /*
- * Zap the rest of the file in one hit.
- */
- unmap_mapping_pages(mapping, index,
- (1 + end - index), false);
- did_range_unmap = 1;
- } else {
- /*
- * Just zap this page
- */
- unmap_mapping_pages(mapping, index,
- 1, false);
- }
- }
+
+ if (page_mapped(page))
+ unmap_mapping_page(page);
BUG_ON(page_mapped(page));
+
ret2 = do_launder_page(mapping, page);
if (ret2 == 0) {
if (!invalidate_complete_page2(mapping, page))
* If a reservation for the page existed in the reservation
* map of a private mapping, the map was modified to indicate
* the reservation was consumed when the page was allocated.
- * We clear the PagePrivate flag now so that the global
+ * We clear the HPageRestoreReserve flag now so that the global
* reserve count will not be incremented in free_huge_page.
* The reservation map will still indicate the reservation
* was consumed and possibly prevent later page allocation.
* This is better than leaking a global reservation. If no
- * reservation existed, it is still safe to clear PagePrivate
- * as no adjustments to reservation counts were made during
- * allocation.
+ * reservation existed, it is still safe to clear
+ * HPageRestoreReserve as no adjustments to reservation counts
+ * were made during allocation.
*
* The reservation map for shared mappings indicates which
* pages have reservations. When a huge page is allocated
* for an address with a reservation, no change is made to
- * the reserve map. In this case PagePrivate will be set
- * to indicate that the global reservation count should be
+ * the reserve map. In this case HPageRestoreReserve will be
+ * set to indicate that the global reservation count should be
* incremented when the page is freed. This is the desired
* behavior. However, when a huge page is allocated for an
* address without a reservation a reservation entry is added
- * to the reservation map, and PagePrivate will not be set.
- * When the page is freed, the global reserve count will NOT
- * be incremented and it will appear as though we have leaked
- * reserved page. In this case, set PagePrivate so that the
- * global reserve count will be incremented to match the
- * reservation map entry which was created.
+ * to the reservation map, and HPageRestoreReserve will not be
+ * set. When the page is freed, the global reserve count will
+ * NOT be incremented and it will appear as though we have
+ * leaked reserved page. In this case, set HPageRestoreReserve
+ * so that the global reserve count will be incremented to
+ * match the reservation map entry which was created.
*
* Note that vm_alloc_shared is based on the flags of the vma
* for which the page was originally allocated. dst_vma could
* be different or NULL on error.
*/
if (vm_alloc_shared)
- SetPagePrivate(page);
+ SetHPageRestoreReserve(page);
else
- ClearPagePrivate(page);
+ ClearHPageRestoreReserve(page);
put_page(page);
}
BUG_ON(copied < 0);
case GET_VLAN_REALDEV_NAME_CMD:
err = 0;
- vlan_dev_get_realdev_name(dev, args.u.device2);
+ vlan_dev_get_realdev_name(dev, args.u.device2,
+ sizeof(args.u.device2));
if (copy_to_user(arg, &args,
sizeof(struct vlan_ioctl_args)))
err = -EFAULT;
netdev_features_t ret;
ret = real_dev->hw_enc_features &
- (NETIF_F_CSUM_MASK | NETIF_F_ALL_TSO | NETIF_F_GSO_ENCAP_ALL);
+ (NETIF_F_CSUM_MASK | NETIF_F_GSO_SOFTWARE |
+ NETIF_F_GSO_ENCAP_ALL);
if ((ret & NETIF_F_GSO_ENCAP_ALL) && (ret & NETIF_F_CSUM_MASK))
return (ret & ~NETIF_F_CSUM_MASK) | NETIF_F_HW_CSUM;
int vlan_dev_set_egress_priority(const struct net_device *dev,
u32 skb_prio, u16 vlan_prio);
int vlan_dev_change_flags(const struct net_device *dev, u32 flag, u32 mask);
-void vlan_dev_get_realdev_name(const struct net_device *dev, char *result);
+void vlan_dev_get_realdev_name(const struct net_device *dev, char *result,
+ size_t size);
int vlan_check_real_dev(struct net_device *real_dev,
__be16 protocol, u16 vlan_id,
return 0;
}
-void vlan_dev_get_realdev_name(const struct net_device *dev, char *result)
+void vlan_dev_get_realdev_name(const struct net_device *dev, char *result, size_t size)
{
- strncpy(result, vlan_dev_priv(dev)->real_dev->name, 23);
+ strscpy_pad(result, vlan_dev_priv(dev)->real_dev->name, size);
}
bool vlan_dev_inherit_address(struct net_device *dev,
struct ifreq ifrr;
int err = -EOPNOTSUPP;
- strncpy(ifrr.ifr_name, real_dev->name, IFNAMSIZ);
+ strscpy_pad(ifrr.ifr_name, real_dev->name, IFNAMSIZ);
ifrr.ifr_ifru = ifr->ifr_ifru;
switch (cmd) {
* @client: client instance
*
* This reclaims a channel by freeing its resources and
- * reseting its inuse flag.
+ * resetting its inuse flag.
*
*/
* For example TREAD have 11.
* 11 is the read/write header = PDU Header(7) + IO Size (4).
* Arrange in such a way that server places header in the
- * alloced memory and payload onto the user buffer.
+ * allocated memory and payload onto the user buffer.
*/
in = pack_sg_list(chan->sg, out,
VIRTQUEUE_NUM, req->rc.sdata, in_hdr_len);
.cancelled = p9_virtio_cancelled,
/*
* We leave one entry for input and one entry for response
- * headers. We also skip one more entry to accomodate, address
+ * headers. We also skip one more entry to accommodate, address
* that are not at page boundary, that can result in an extra
* page in zero copy.
*/
select DQL
default y
-config BPF_JIT
- bool "enable BPF Just In Time compiler"
- depends on HAVE_CBPF_JIT || HAVE_EBPF_JIT
- depends on MODULES
- help
- Berkeley Packet Filter filtering capabilities are normally handled
- by an interpreter. This option allows kernel to generate a native
- code when filter is loaded in memory. This should speedup
- packet sniffing (libpcap/tcpdump).
-
- Note, admin should enable this feature changing:
- /proc/sys/net/core/bpf_jit_enable
- /proc/sys/net/core/bpf_jit_harden (optional)
- /proc/sys/net/core/bpf_jit_kallsyms (optional)
-
config BPF_STREAM_PARSER
bool "enable BPF STREAM_PARSER"
depends on INET
e.g. notification messages.
endif # if NET
-
-# Used by archs to tell that they support BPF JIT compiler plus which flavour.
-# Only one of the two can be selected for a specific arch since eBPF JIT supersedes
-# the cBPF JIT.
-
-# Classic BPF JIT (cBPF)
-config HAVE_CBPF_JIT
- bool
-
-# Extended BPF JIT (eBPF)
-config HAVE_EBPF_JIT
- bool
if (a && a->status & ATIF_PROBE) {
a->status |= ATIF_PROBE_FAIL;
/*
- * we do not respond to probe or request packets for
+ * we do not respond to probe or request packets of
* this address while we are probing this address
*/
goto unlock;
/*
* Phase 1 is fine on LocalTalk but we don't do
- * EtherTalk phase 1. Anyone wanting to add it go ahead.
+ * EtherTalk phase 1. Anyone wanting to add it, go ahead.
*/
if (dev->type == ARPHRD_ETHER && nr->nr_phase != 2)
return -EPROTONOSUPPORT;
nr = (struct atalk_netrange *)&(atif->nets);
/*
* Phase 1 is fine on Localtalk but we don't do
- * Ethertalk phase 1. Anyone wanting to add it go ahead.
+ * Ethertalk phase 1. Anyone wanting to add it, go ahead.
*/
if (dev->type == ARPHRD_ETHER && nr->nr_phase != 2)
return -EPROTONOSUPPORT;
* by the network device layer.
*
* Ergo, before the AppleTalk module can be removed, all AppleTalk
- * sockets be closed from user space.
+ * sockets should be closed from user space.
*/
static void __exit atalk_exit(void)
{
#define to_atm_dev(cldev) container_of(cldev, struct atm_dev, class_dev)
-static ssize_t show_type(struct device *cdev,
+static ssize_t type_show(struct device *cdev,
struct device_attribute *attr, char *buf)
{
struct atm_dev *adev = to_atm_dev(cdev);
return scnprintf(buf, PAGE_SIZE, "%s\n", adev->type);
}
-static ssize_t show_address(struct device *cdev,
+static ssize_t address_show(struct device *cdev,
struct device_attribute *attr, char *buf)
{
struct atm_dev *adev = to_atm_dev(cdev);
return scnprintf(buf, PAGE_SIZE, "%pM\n", adev->esi);
}
-static ssize_t show_atmaddress(struct device *cdev,
+static ssize_t atmaddress_show(struct device *cdev,
struct device_attribute *attr, char *buf)
{
unsigned long flags;
return count;
}
-static ssize_t show_atmindex(struct device *cdev,
+static ssize_t atmindex_show(struct device *cdev,
struct device_attribute *attr, char *buf)
{
struct atm_dev *adev = to_atm_dev(cdev);
return scnprintf(buf, PAGE_SIZE, "%d\n", adev->number);
}
-static ssize_t show_carrier(struct device *cdev,
+static ssize_t carrier_show(struct device *cdev,
struct device_attribute *attr, char *buf)
{
struct atm_dev *adev = to_atm_dev(cdev);
adev->signal == ATM_PHY_SIG_LOST ? 0 : 1);
}
-static ssize_t show_link_rate(struct device *cdev,
+static ssize_t link_rate_show(struct device *cdev,
struct device_attribute *attr, char *buf)
{
struct atm_dev *adev = to_atm_dev(cdev);
return scnprintf(buf, PAGE_SIZE, "%d\n", link_rate);
}
-static DEVICE_ATTR(address, 0444, show_address, NULL);
-static DEVICE_ATTR(atmaddress, 0444, show_atmaddress, NULL);
-static DEVICE_ATTR(atmindex, 0444, show_atmindex, NULL);
-static DEVICE_ATTR(carrier, 0444, show_carrier, NULL);
-static DEVICE_ATTR(type, 0444, show_type, NULL);
-static DEVICE_ATTR(link_rate, 0444, show_link_rate, NULL);
+static DEVICE_ATTR_RO(address);
+static DEVICE_ATTR_RO(atmaddress);
+static DEVICE_ATTR_RO(atmindex);
+static DEVICE_ATTR_RO(carrier);
+static DEVICE_ATTR_RO(type);
+static DEVICE_ATTR_RO(link_rate);
static struct device_attribute *atm_attrs[] = {
&dev_attr_atmaddress,
* This lock should be held for writing any time the list of devices or
* their attached vcc's could be altered. It should be held for reading
* any time these are being queried. Note that we sometimes need to
- * do read-locking under interrupt context, so write locking must block
- * the current CPU's interrupts
+ * do read-locking under interrupting context, so write locking must block
+ * the current CPU's interrupts.
*/
static DEFINE_RWLOCK(devs_lock);
static struct atm_dev *__atm_dev_lookup(int number)
{
struct atm_dev *dev;
- struct list_head *p;
- list_for_each(p, &atm_devs) {
- dev = list_entry(p, struct atm_dev, dev_list);
+ list_for_each_entry(dev, &atm_devs, dev_list) {
if (dev->number == number) {
atm_dev_hold(dev);
return dev;
return -ENOMEM;
}
tmp_p = tmp_buf;
- list_for_each(p, &atm_devs) {
- dev = list_entry(p, struct atm_dev, dev_list);
+ list_for_each_entry(dev, &atm_devs, dev_list) {
*tmp_p++ = dev->number;
}
mutex_unlock(&atm_dev_mutex);
if (WARN_ON(!forw_packet->if_outgoing))
return;
- if (WARN_ON(forw_packet->if_outgoing->soft_iface != soft_iface))
+ if (forw_packet->if_outgoing->soft_iface != soft_iface) {
+ pr_warn("%s: soft interface switch for queued OGM\n", __func__);
return;
+ }
if (forw_packet->if_incoming->if_status != BATADV_IF_ACTIVE)
return;
orig_node->orig) ||
nla_put(msg, BATADV_ATTR_NEIGH_ADDRESS, ETH_ALEN,
neigh_node->addr) ||
+ nla_put_string(msg, BATADV_ATTR_HARD_IFNAME,
+ neigh_node->if_incoming->net_dev->name) ||
nla_put_u32(msg, BATADV_ATTR_HARD_IFINDEX,
neigh_node->if_incoming->net_dev->ifindex) ||
nla_put_u8(msg, BATADV_ATTR_TQ, tq_avg) ||
if (nla_put(msg, BATADV_ATTR_NEIGH_ADDRESS, ETH_ALEN,
hardif_neigh->addr) ||
+ nla_put_string(msg, BATADV_ATTR_HARD_IFNAME,
+ hardif_neigh->if_incoming->net_dev->name) ||
nla_put_u32(msg, BATADV_ATTR_HARD_IFINDEX,
hardif_neigh->if_incoming->net_dev->ifindex) ||
nla_put_u32(msg, BATADV_ATTR_LAST_SEEN_MSECS,
router->addr) ||
nla_put_string(msg, BATADV_ATTR_HARD_IFNAME,
router->if_incoming->net_dev->name) ||
+ nla_put_u32(msg, BATADV_ATTR_HARD_IFINDEX,
+ router->if_incoming->net_dev->ifindex) ||
nla_put_u32(msg, BATADV_ATTR_BANDWIDTH_DOWN,
gw_node->bandwidth_down) ||
nla_put_u32(msg, BATADV_ATTR_BANDWIDTH_UP,
if (nla_put(msg, BATADV_ATTR_NEIGH_ADDRESS, ETH_ALEN,
hardif_neigh->addr) ||
+ nla_put_string(msg, BATADV_ATTR_HARD_IFNAME,
+ hardif_neigh->if_incoming->net_dev->name) ||
nla_put_u32(msg, BATADV_ATTR_HARD_IFINDEX,
hardif_neigh->if_incoming->net_dev->ifindex) ||
nla_put_u32(msg, BATADV_ATTR_LAST_SEEN_MSECS,
if (nla_put(msg, BATADV_ATTR_ORIG_ADDRESS, ETH_ALEN, orig_node->orig) ||
nla_put(msg, BATADV_ATTR_NEIGH_ADDRESS, ETH_ALEN,
neigh_node->addr) ||
+ nla_put_string(msg, BATADV_ATTR_HARD_IFNAME,
+ neigh_node->if_incoming->net_dev->name) ||
nla_put_u32(msg, BATADV_ATTR_HARD_IFINDEX,
neigh_node->if_incoming->net_dev->ifindex) ||
nla_put_u32(msg, BATADV_ATTR_THROUGHPUT, throughput) ||
goto out;
}
+ if (nla_put_u32(msg, BATADV_ATTR_HARD_IFINDEX,
+ router->if_incoming->net_dev->ifindex)) {
+ genlmsg_cancel(msg, hdr);
+ goto out;
+ }
+
if (nla_put_u32(msg, BATADV_ATTR_BANDWIDTH_DOWN,
gw_node->bandwidth_down)) {
genlmsg_cancel(msg, hdr);
break;
case BATADV_CLAIM_TYPE_ANNOUNCE:
/* announcement frame
- * set HW SRC to the special mac containg the crc
+ * set HW SRC to the special mac containing the crc
*/
ether_addr_copy(hw_src, mac);
batadv_dbg(BATADV_DBG_BLA, bat_priv,
/* lets see if this originator is in our mesh */
orig_node = batadv_orig_hash_find(bat_priv, backbone_addr);
- /* dont accept claims from gateways which are not in
+ /* don't accept claims from gateways which are not in
* the same mesh or group.
*/
if (!orig_node)
void batadv_bla_status_update(struct net_device *net_dev);
int batadv_bla_init(struct batadv_priv *bat_priv);
void batadv_bla_free(struct batadv_priv *bat_priv);
-int batadv_bla_claim_dump(struct sk_buff *msg, struct netlink_callback *cb);
#ifdef CONFIG_BATMAN_ADV_DAT
bool batadv_bla_check_claim(struct batadv_priv *bat_priv, u8 *addr,
unsigned short vid);
#include <linux/atomic.h>
#include <linux/byteorder/generic.h>
-#include <linux/errno.h>
#include <linux/gfp.h>
#include <linux/if.h>
#include <linux/if_arp.h>
goto out;
}
- /* >1 neighbors -> (re)brodcast */
+ /* >1 neighbors -> (re)broadcast */
if (rcu_dereference(hlist_next_rcu(first)))
goto out;
}
/**
- * batadv_master_del_slave() - remove hard_iface from the current master iface
- * @slave: the interface enslaved in another master
- * @master: the master from which slave has to be removed
- *
- * Invoke ndo_del_slave on master passing slave as argument. In this way the
- * slave is free'd and the master can correctly change its internal state.
- *
- * Return: 0 on success, a negative value representing the error otherwise
- */
-static int batadv_master_del_slave(struct batadv_hard_iface *slave,
- struct net_device *master)
-{
- int ret;
-
- if (!master)
- return 0;
-
- ret = -EBUSY;
- if (master->netdev_ops->ndo_del_slave)
- ret = master->netdev_ops->ndo_del_slave(master, slave->net_dev);
-
- return ret;
-}
-
-/**
* batadv_hardif_enable_interface() - Enslave hard interface to soft interface
* @hard_iface: hard interface to add to soft interface
- * @net: the applicable net namespace
- * @iface_name: name of the soft interface
+ * @soft_iface: netdev struct of the mesh interface
*
* Return: 0 on success or negative error number in case of failure
*/
int batadv_hardif_enable_interface(struct batadv_hard_iface *hard_iface,
- struct net *net, const char *iface_name)
+ struct net_device *soft_iface)
{
struct batadv_priv *bat_priv;
- struct net_device *soft_iface, *master;
__be16 ethertype = htons(ETH_P_BATMAN);
int max_header_len = batadv_max_header_len();
int ret;
kref_get(&hard_iface->refcount);
- soft_iface = dev_get_by_name(net, iface_name);
-
- if (!soft_iface) {
- soft_iface = batadv_softif_create(net, iface_name);
-
- if (!soft_iface) {
- ret = -ENOMEM;
- goto err;
- }
-
- /* dev_get_by_name() increases the reference counter for us */
- dev_hold(soft_iface);
- }
-
- if (!batadv_softif_is_valid(soft_iface)) {
- pr_err("Can't create batman mesh interface %s: already exists as regular interface\n",
- soft_iface->name);
- ret = -EINVAL;
- goto err_dev;
- }
-
- /* check if the interface is enslaved in another virtual one and
- * in that case unlink it first
- */
- master = netdev_master_upper_dev_get(hard_iface->net_dev);
- ret = batadv_master_del_slave(hard_iface, master);
- if (ret)
- goto err_dev;
-
+ dev_hold(soft_iface);
hard_iface->soft_iface = soft_iface;
bat_priv = netdev_priv(hard_iface->soft_iface);
err_dev:
hard_iface->soft_iface = NULL;
dev_put(soft_iface);
-err:
batadv_hardif_put(hard_iface);
return ret;
}
#include <linux/rcupdate.h>
#include <linux/stddef.h>
#include <linux/types.h>
-#include <net/net_namespace.h>
/**
* enum batadv_hard_if_state - State of a hard interface
struct batadv_hard_iface*
batadv_hardif_get_by_netdev(const struct net_device *net_dev);
int batadv_hardif_enable_interface(struct batadv_hard_iface *hard_iface,
- struct net *net, const char *iface_name);
+ struct net_device *soft_iface);
void batadv_hardif_disable_interface(struct batadv_hard_iface *hard_iface);
int batadv_hardif_min_mtu(struct net_device *soft_iface);
void batadv_update_min_mtu(struct net_device *soft_iface);
#include <linux/stddef.h>
#include <linux/types.h>
-/* callback to a compare function. should compare 2 element datas for their
+/* callback to a compare function. should compare 2 element data for their
* keys
*
* Return: true if same and false if not same
#define BATADV_DRIVER_DEVICE "batman-adv"
#ifndef BATADV_SOURCE_VERSION
-#define BATADV_SOURCE_VERSION "2021.1"
+#define BATADV_SOURCE_VERSION "2021.2"
#endif
/* B.A.T.M.A.N. parameters */
/* number of packets to send for broadcasts on different interface types */
#define BATADV_NUM_BCASTS_DEFAULT 1
#define BATADV_NUM_BCASTS_WIRELESS 3
-#define BATADV_NUM_BCASTS_MAX 3
/* length of the single packet used by the TP meter */
#define BATADV_TP_PACKET_LEN ETH_DATA_LEN
* BATADV_MCAST_WANT_NO_RTR6: No IPv6 multicast router is present
* The former two OR'd: no multicast router is present
*/
-#if IS_ENABLED(CONFIG_IPV6)
static u8 batadv_mcast_mla_rtr_flags_bridge_get(struct batadv_priv *bat_priv,
struct net_device *bridge)
{
- struct list_head bridge_mcast_list = LIST_HEAD_INIT(bridge_mcast_list);
struct net_device *dev = bat_priv->soft_iface;
- struct br_ip_list *br_ip_entry, *tmp;
- u8 flags = BATADV_MCAST_WANT_NO_RTR6;
- int ret;
+ u8 flags = BATADV_NO_FLAGS;
if (!bridge)
return BATADV_MCAST_WANT_NO_RTR4 | BATADV_MCAST_WANT_NO_RTR6;
- /* TODO: ask the bridge if a multicast router is present (the bridge
- * is capable of performing proper RFC4286 multicast router
- * discovery) instead of searching for a ff02::2 listener here
- */
- ret = br_multicast_list_adjacent(dev, &bridge_mcast_list);
- if (ret < 0)
- return BATADV_NO_FLAGS;
-
- list_for_each_entry_safe(br_ip_entry, tmp, &bridge_mcast_list, list) {
- /* the bridge snooping does not maintain IPv4 link-local
- * addresses - therefore we won't find any IPv4 multicast router
- * address here, only IPv6 ones
- */
- if (br_ip_entry->addr.proto == htons(ETH_P_IPV6) &&
- ipv6_addr_is_ll_all_routers(&br_ip_entry->addr.dst.ip6))
- flags &= ~BATADV_MCAST_WANT_NO_RTR6;
-
- list_del(&br_ip_entry->list);
- kfree(br_ip_entry);
- }
+ if (!br_multicast_has_router_adjacent(dev, ETH_P_IP))
+ flags |= BATADV_MCAST_WANT_NO_RTR4;
+ if (!br_multicast_has_router_adjacent(dev, ETH_P_IPV6))
+ flags |= BATADV_MCAST_WANT_NO_RTR6;
return flags;
}
-#else
-static inline u8
-batadv_mcast_mla_rtr_flags_bridge_get(struct batadv_priv *bat_priv,
- struct net_device *bridge)
-{
- if (bridge)
- return BATADV_NO_FLAGS;
- else
- return BATADV_MCAST_WANT_NO_RTR4 | BATADV_MCAST_WANT_NO_RTR6;
-}
-#endif
/**
* batadv_mcast_mla_rtr_flags_get() - get multicast router flags
bat_priv->soft_iface->ifindex))
goto nla_put_failure;
+ if (nla_put_string(msg, BATADV_ATTR_MESH_IFNAME,
+ bat_priv->soft_iface->name))
+ goto nla_put_failure;
+
if (nla_put_u32(msg, BATADV_ATTR_HARD_IFINDEX,
net_dev->ifindex) ||
nla_put_string(msg, BATADV_ATTR_HARD_IFNAME,
bat_priv->soft_iface->ifindex))
goto nla_put_failure;
+ if (nla_put_string(msg, BATADV_ATTR_MESH_IFNAME,
+ bat_priv->soft_iface->name))
+ goto nla_put_failure;
+
if (nla_put_u32(msg, BATADV_ATTR_VLANID, vlan->vid & VLAN_VID_MASK))
goto nla_put_failure;
struct batadv_bcast_packet *bcast_packet;
struct ethhdr *ethhdr;
int hdr_size = sizeof(*bcast_packet);
- int ret = NET_RX_DROP;
s32 seq_diff;
u32 seqno;
+ int ret;
/* drop packet if it has not necessary minimum size */
if (unlikely(!pskb_may_pull(skb, hdr_size)))
if (batadv_is_my_mac(bat_priv, bcast_packet->orig))
goto free_skb;
- if (bcast_packet->ttl < 2)
+ if (bcast_packet->ttl-- < 2)
goto free_skb;
orig_node = batadv_orig_hash_find(bat_priv, bcast_packet->orig);
batadv_skb_set_priority(skb, sizeof(struct batadv_bcast_packet));
/* rebroadcast packet */
- batadv_add_bcast_packet_to_list(bat_priv, skb, 1, false);
+ ret = batadv_forw_bcast_packet(bat_priv, skb, 0, false);
+ if (ret == NETDEV_TX_BUSY)
+ goto free_skb;
/* don't hand the broadcast up if it is from an originator
* from the same backbone.
spin_unlock_bh(&orig_node->bcast_seqno_lock);
free_skb:
kfree_skb(skb);
+ ret = NET_RX_DROP;
out:
if (orig_node)
batadv_orig_node_put(orig_node);
}
/**
- * batadv_add_bcast_packet_to_list() - queue broadcast packet for multiple sends
+ * batadv_forw_bcast_packet_to_list() - queue broadcast packet for transmissions
* @bat_priv: the bat priv with all the soft interface information
* @skb: broadcast packet to add
* @delay: number of jiffies to wait before sending
* @own_packet: true if it is a self-generated broadcast packet
+ * @if_in: the interface where the packet was received on
+ * @if_out: the outgoing interface to queue on
*
- * add a broadcast packet to the queue and setup timers. broadcast packets
+ * Adds a broadcast packet to the queue and sets up timers. Broadcast packets
* are sent multiple times to increase probability for being received.
*
- * The skb is not consumed, so the caller should make sure that the
- * skb is freed.
- *
* Return: NETDEV_TX_OK on success and NETDEV_TX_BUSY on errors.
*/
-int batadv_add_bcast_packet_to_list(struct batadv_priv *bat_priv,
- const struct sk_buff *skb,
- unsigned long delay,
- bool own_packet)
+static int batadv_forw_bcast_packet_to_list(struct batadv_priv *bat_priv,
+ struct sk_buff *skb,
+ unsigned long delay,
+ bool own_packet,
+ struct batadv_hard_iface *if_in,
+ struct batadv_hard_iface *if_out)
{
- struct batadv_hard_iface *primary_if;
struct batadv_forw_packet *forw_packet;
- struct batadv_bcast_packet *bcast_packet;
+ unsigned long send_time = jiffies;
struct sk_buff *newskb;
- primary_if = batadv_primary_if_get_selected(bat_priv);
- if (!primary_if)
- goto err;
-
newskb = skb_copy(skb, GFP_ATOMIC);
- if (!newskb) {
- batadv_hardif_put(primary_if);
+ if (!newskb)
goto err;
- }
- forw_packet = batadv_forw_packet_alloc(primary_if, NULL,
+ forw_packet = batadv_forw_packet_alloc(if_in, if_out,
&bat_priv->bcast_queue_left,
bat_priv, newskb);
- batadv_hardif_put(primary_if);
if (!forw_packet)
goto err_packet_free;
- /* as we have a copy now, it is safe to decrease the TTL */
- bcast_packet = (struct batadv_bcast_packet *)newskb->data;
- bcast_packet->ttl--;
-
forw_packet->own = own_packet;
INIT_DELAYED_WORK(&forw_packet->delayed_work,
batadv_send_outstanding_bcast_packet);
- batadv_forw_packet_bcast_queue(bat_priv, forw_packet, jiffies + delay);
+ send_time += delay ? delay : msecs_to_jiffies(5);
+
+ batadv_forw_packet_bcast_queue(bat_priv, forw_packet, send_time);
return NETDEV_TX_OK;
err_packet_free:
}
/**
+ * batadv_forw_bcast_packet_if() - forward and queue a broadcast packet
+ * @bat_priv: the bat priv with all the soft interface information
+ * @skb: broadcast packet to add
+ * @delay: number of jiffies to wait before sending
+ * @own_packet: true if it is a self-generated broadcast packet
+ * @if_in: the interface where the packet was received on
+ * @if_out: the outgoing interface to forward to
+ *
+ * Transmits a broadcast packet on the specified interface either immediately
+ * or if a delay is given after that. Furthermore, queues additional
+ * retransmissions if this interface is a wireless one.
+ *
+ * Return: NETDEV_TX_OK on success and NETDEV_TX_BUSY on errors.
+ */
+static int batadv_forw_bcast_packet_if(struct batadv_priv *bat_priv,
+ struct sk_buff *skb,
+ unsigned long delay,
+ bool own_packet,
+ struct batadv_hard_iface *if_in,
+ struct batadv_hard_iface *if_out)
+{
+ unsigned int num_bcasts = if_out->num_bcasts;
+ struct sk_buff *newskb;
+ int ret = NETDEV_TX_OK;
+
+ if (!delay) {
+ newskb = skb_copy(skb, GFP_ATOMIC);
+ if (!newskb)
+ return NETDEV_TX_BUSY;
+
+ batadv_send_broadcast_skb(newskb, if_out);
+ num_bcasts--;
+ }
+
+ /* delayed broadcast or rebroadcasts? */
+ if (num_bcasts >= 1) {
+ BATADV_SKB_CB(skb)->num_bcasts = num_bcasts;
+
+ ret = batadv_forw_bcast_packet_to_list(bat_priv, skb, delay,
+ own_packet, if_in,
+ if_out);
+ }
+
+ return ret;
+}
+
+/**
+ * batadv_send_no_broadcast() - check whether (re)broadcast is necessary
+ * @bat_priv: the bat priv with all the soft interface information
+ * @skb: broadcast packet to check
+ * @own_packet: true if it is a self-generated broadcast packet
+ * @if_out: the outgoing interface checked and considered for (re)broadcast
+ *
+ * Return: False if a packet needs to be (re)broadcasted on the given interface,
+ * true otherwise.
+ */
+static bool batadv_send_no_broadcast(struct batadv_priv *bat_priv,
+ struct sk_buff *skb, bool own_packet,
+ struct batadv_hard_iface *if_out)
+{
+ struct batadv_hardif_neigh_node *neigh_node = NULL;
+ struct batadv_bcast_packet *bcast_packet;
+ u8 *orig_neigh;
+ u8 *neigh_addr;
+ char *type;
+ int ret;
+
+ if (!own_packet) {
+ neigh_addr = eth_hdr(skb)->h_source;
+ neigh_node = batadv_hardif_neigh_get(if_out,
+ neigh_addr);
+ }
+
+ bcast_packet = (struct batadv_bcast_packet *)skb->data;
+ orig_neigh = neigh_node ? neigh_node->orig : NULL;
+
+ ret = batadv_hardif_no_broadcast(if_out, bcast_packet->orig,
+ orig_neigh);
+
+ if (neigh_node)
+ batadv_hardif_neigh_put(neigh_node);
+
+ /* ok, may broadcast */
+ if (!ret)
+ return false;
+
+ /* no broadcast */
+ switch (ret) {
+ case BATADV_HARDIF_BCAST_NORECIPIENT:
+ type = "no neighbor";
+ break;
+ case BATADV_HARDIF_BCAST_DUPFWD:
+ type = "single neighbor is source";
+ break;
+ case BATADV_HARDIF_BCAST_DUPORIG:
+ type = "single neighbor is originator";
+ break;
+ default:
+ type = "unknown";
+ }
+
+ batadv_dbg(BATADV_DBG_BATMAN, bat_priv,
+ "BCAST packet from orig %pM on %s suppressed: %s\n",
+ bcast_packet->orig,
+ if_out->net_dev->name, type);
+
+ return true;
+}
+
+/**
+ * __batadv_forw_bcast_packet() - forward and queue a broadcast packet
+ * @bat_priv: the bat priv with all the soft interface information
+ * @skb: broadcast packet to add
+ * @delay: number of jiffies to wait before sending
+ * @own_packet: true if it is a self-generated broadcast packet
+ *
+ * Transmits a broadcast packet either immediately or if a delay is given
+ * after that. Furthermore, queues additional retransmissions on wireless
+ * interfaces.
+ *
+ * This call clones the given skb, hence the caller needs to take into
+ * account that the data segment of the given skb might not be
+ * modifiable anymore.
+ *
+ * Return: NETDEV_TX_OK on success and NETDEV_TX_BUSY on errors.
+ */
+static int __batadv_forw_bcast_packet(struct batadv_priv *bat_priv,
+ struct sk_buff *skb,
+ unsigned long delay,
+ bool own_packet)
+{
+ struct batadv_hard_iface *hard_iface;
+ struct batadv_hard_iface *primary_if;
+ int ret = NETDEV_TX_OK;
+
+ primary_if = batadv_primary_if_get_selected(bat_priv);
+ if (!primary_if)
+ return NETDEV_TX_BUSY;
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(hard_iface, &batadv_hardif_list, list) {
+ if (hard_iface->soft_iface != bat_priv->soft_iface)
+ continue;
+
+ if (!kref_get_unless_zero(&hard_iface->refcount))
+ continue;
+
+ if (batadv_send_no_broadcast(bat_priv, skb, own_packet,
+ hard_iface)) {
+ batadv_hardif_put(hard_iface);
+ continue;
+ }
+
+ ret = batadv_forw_bcast_packet_if(bat_priv, skb, delay,
+ own_packet, primary_if,
+ hard_iface);
+ batadv_hardif_put(hard_iface);
+
+ if (ret == NETDEV_TX_BUSY)
+ break;
+ }
+ rcu_read_unlock();
+
+ batadv_hardif_put(primary_if);
+ return ret;
+}
+
+/**
+ * batadv_forw_bcast_packet() - forward and queue a broadcast packet
+ * @bat_priv: the bat priv with all the soft interface information
+ * @skb: broadcast packet to add
+ * @delay: number of jiffies to wait before sending
+ * @own_packet: true if it is a self-generated broadcast packet
+ *
+ * Transmits a broadcast packet either immediately or if a delay is given
+ * after that. Furthermore, queues additional retransmissions on wireless
+ * interfaces.
+ *
+ * Return: NETDEV_TX_OK on success and NETDEV_TX_BUSY on errors.
+ */
+int batadv_forw_bcast_packet(struct batadv_priv *bat_priv,
+ struct sk_buff *skb,
+ unsigned long delay,
+ bool own_packet)
+{
+ return __batadv_forw_bcast_packet(bat_priv, skb, delay, own_packet);
+}
+
+/**
+ * batadv_send_bcast_packet() - send and queue a broadcast packet
+ * @bat_priv: the bat priv with all the soft interface information
+ * @skb: broadcast packet to add
+ * @delay: number of jiffies to wait before sending
+ * @own_packet: true if it is a self-generated broadcast packet
+ *
+ * Transmits a broadcast packet either immediately or if a delay is given
+ * after that. Furthermore, queues additional retransmissions on wireless
+ * interfaces.
+ *
+ * Consumes the provided skb.
+ */
+void batadv_send_bcast_packet(struct batadv_priv *bat_priv,
+ struct sk_buff *skb,
+ unsigned long delay,
+ bool own_packet)
+{
+ __batadv_forw_bcast_packet(bat_priv, skb, delay, own_packet);
+ consume_skb(skb);
+}
+
+/**
* batadv_forw_packet_bcasts_left() - check if a retransmission is necessary
* @forw_packet: the forwarding packet to check
- * @hard_iface: the interface to check on
*
* Checks whether a given packet has any (re)transmissions left on the provided
* interface.
* Return: True if (re)transmissions are left, false otherwise.
*/
static bool
-batadv_forw_packet_bcasts_left(struct batadv_forw_packet *forw_packet,
- struct batadv_hard_iface *hard_iface)
+batadv_forw_packet_bcasts_left(struct batadv_forw_packet *forw_packet)
{
- unsigned int max;
-
- if (hard_iface)
- max = hard_iface->num_bcasts;
- else
- max = BATADV_NUM_BCASTS_MAX;
-
- return BATADV_SKB_CB(forw_packet->skb)->num_bcasts < max;
+ return BATADV_SKB_CB(forw_packet->skb)->num_bcasts;
}
/**
- * batadv_forw_packet_bcasts_inc() - increment retransmission counter of a
+ * batadv_forw_packet_bcasts_dec() - decrement retransmission counter of a
* packet
- * @forw_packet: the packet to increase the counter for
+ * @forw_packet: the packet to decrease the counter for
*/
static void
-batadv_forw_packet_bcasts_inc(struct batadv_forw_packet *forw_packet)
+batadv_forw_packet_bcasts_dec(struct batadv_forw_packet *forw_packet)
{
- BATADV_SKB_CB(forw_packet->skb)->num_bcasts++;
+ BATADV_SKB_CB(forw_packet->skb)->num_bcasts--;
}
/**
*/
bool batadv_forw_packet_is_rebroadcast(struct batadv_forw_packet *forw_packet)
{
- return BATADV_SKB_CB(forw_packet->skb)->num_bcasts > 0;
+ unsigned char num_bcasts = BATADV_SKB_CB(forw_packet->skb)->num_bcasts;
+
+ return num_bcasts != forw_packet->if_outgoing->num_bcasts;
}
+/**
+ * batadv_send_outstanding_bcast_packet() - transmit a queued broadcast packet
+ * @work: work queue item
+ *
+ * Transmits a queued broadcast packet and if necessary reschedules it.
+ */
static void batadv_send_outstanding_bcast_packet(struct work_struct *work)
{
- struct batadv_hard_iface *hard_iface;
- struct batadv_hardif_neigh_node *neigh_node;
- struct delayed_work *delayed_work;
+ unsigned long send_time = jiffies + msecs_to_jiffies(5);
struct batadv_forw_packet *forw_packet;
- struct batadv_bcast_packet *bcast_packet;
- struct sk_buff *skb1;
- struct net_device *soft_iface;
+ struct delayed_work *delayed_work;
struct batadv_priv *bat_priv;
- unsigned long send_time = jiffies + msecs_to_jiffies(5);
+ struct sk_buff *skb1;
bool dropped = false;
- u8 *neigh_addr;
- u8 *orig_neigh;
- int ret = 0;
delayed_work = to_delayed_work(work);
forw_packet = container_of(delayed_work, struct batadv_forw_packet,
delayed_work);
- soft_iface = forw_packet->if_incoming->soft_iface;
- bat_priv = netdev_priv(soft_iface);
+ bat_priv = netdev_priv(forw_packet->if_incoming->soft_iface);
if (atomic_read(&bat_priv->mesh_state) == BATADV_MESH_DEACTIVATING) {
dropped = true;
goto out;
}
- bcast_packet = (struct batadv_bcast_packet *)forw_packet->skb->data;
-
- /* rebroadcast packet */
- rcu_read_lock();
- list_for_each_entry_rcu(hard_iface, &batadv_hardif_list, list) {
- if (hard_iface->soft_iface != soft_iface)
- continue;
-
- if (!batadv_forw_packet_bcasts_left(forw_packet, hard_iface))
- continue;
-
- if (forw_packet->own) {
- neigh_node = NULL;
- } else {
- neigh_addr = eth_hdr(forw_packet->skb)->h_source;
- neigh_node = batadv_hardif_neigh_get(hard_iface,
- neigh_addr);
- }
-
- orig_neigh = neigh_node ? neigh_node->orig : NULL;
-
- ret = batadv_hardif_no_broadcast(hard_iface, bcast_packet->orig,
- orig_neigh);
-
- if (ret) {
- char *type;
-
- switch (ret) {
- case BATADV_HARDIF_BCAST_NORECIPIENT:
- type = "no neighbor";
- break;
- case BATADV_HARDIF_BCAST_DUPFWD:
- type = "single neighbor is source";
- break;
- case BATADV_HARDIF_BCAST_DUPORIG:
- type = "single neighbor is originator";
- break;
- default:
- type = "unknown";
- }
-
- batadv_dbg(BATADV_DBG_BATMAN, bat_priv, "BCAST packet from orig %pM on %s suppressed: %s\n",
- bcast_packet->orig,
- hard_iface->net_dev->name, type);
-
- if (neigh_node)
- batadv_hardif_neigh_put(neigh_node);
-
- continue;
- }
-
- if (neigh_node)
- batadv_hardif_neigh_put(neigh_node);
-
- if (!kref_get_unless_zero(&hard_iface->refcount))
- continue;
-
- /* send a copy of the saved skb */
- skb1 = skb_clone(forw_packet->skb, GFP_ATOMIC);
- if (skb1)
- batadv_send_broadcast_skb(skb1, hard_iface);
-
- batadv_hardif_put(hard_iface);
- }
- rcu_read_unlock();
+ /* send a copy of the saved skb */
+ skb1 = skb_clone(forw_packet->skb, GFP_ATOMIC);
+ if (!skb1)
+ goto out;
- batadv_forw_packet_bcasts_inc(forw_packet);
+ batadv_send_broadcast_skb(skb1, forw_packet->if_outgoing);
+ batadv_forw_packet_bcasts_dec(forw_packet);
- /* if we still have some more bcasts to send */
- if (batadv_forw_packet_bcasts_left(forw_packet, NULL)) {
+ if (batadv_forw_packet_bcasts_left(forw_packet)) {
batadv_forw_packet_bcast_queue(bat_priv, forw_packet,
send_time);
return;
struct batadv_hard_iface *hard_iface);
int batadv_send_unicast_skb(struct sk_buff *skb,
struct batadv_neigh_node *neigh_node);
-int batadv_add_bcast_packet_to_list(struct batadv_priv *bat_priv,
- const struct sk_buff *skb,
- unsigned long delay,
- bool own_packet);
+int batadv_forw_bcast_packet(struct batadv_priv *bat_priv,
+ struct sk_buff *skb,
+ unsigned long delay,
+ bool own_packet);
+void batadv_send_bcast_packet(struct batadv_priv *bat_priv,
+ struct sk_buff *skb,
+ unsigned long delay,
+ bool own_packet);
void
batadv_purge_outstanding_packets(struct batadv_priv *bat_priv,
const struct batadv_hard_iface *hard_iface);
#include <linux/netdevice.h>
#include <linux/netlink.h>
#include <linux/percpu.h>
-#include <linux/printk.h>
#include <linux/random.h>
#include <linux/rculist.h>
#include <linux/rcupdate.h>
#include <linux/stddef.h>
#include <linux/string.h>
#include <linux/types.h>
+#include <net/net_namespace.h>
#include <net/netlink.h>
#include <uapi/linux/batadv_packet.h>
#include <uapi/linux/batman_adv.h>
struct vlan_ethhdr *vhdr;
unsigned int header_len = 0;
int data_len = skb->len, ret;
- unsigned long brd_delay = 1;
+ unsigned long brd_delay = 0;
bool do_bcast = false, client_added;
unsigned short vid;
u32 seqno;
bcast_packet = (struct batadv_bcast_packet *)skb->data;
bcast_packet->version = BATADV_COMPAT_VERSION;
- bcast_packet->ttl = BATADV_TTL;
+ bcast_packet->ttl = BATADV_TTL - 1;
/* batman packet type: broadcast */
bcast_packet->packet_type = BATADV_BCAST;
seqno = atomic_inc_return(&bat_priv->bcast_seqno);
bcast_packet->seqno = htonl(seqno);
- batadv_add_bcast_packet_to_list(bat_priv, skb, brd_delay, true);
-
- /* a copy is stored in the bcast list, therefore removing
- * the original skb.
- */
- consume_skb(skb);
-
+ batadv_send_bcast_packet(bat_priv, skb, brd_delay, true);
/* unicast packet */
} else {
/* DHCP packets going to a server will use the GW feature */
struct netlink_ext_ack *extack)
{
struct batadv_hard_iface *hard_iface;
- struct net *net = dev_net(dev);
int ret = -EINVAL;
hard_iface = batadv_hardif_get_by_netdev(slave_dev);
if (!hard_iface || hard_iface->soft_iface)
goto out;
- ret = batadv_hardif_enable_interface(hard_iface, net, dev->name);
+ ret = batadv_hardif_enable_interface(hard_iface, dev);
out:
if (hard_iface)
}
/**
- * batadv_softif_create() - Create and register soft interface
- * @net: the applicable net namespace
- * @name: name of the new soft interface
- *
- * Return: newly allocated soft_interface, NULL on errors
- */
-struct net_device *batadv_softif_create(struct net *net, const char *name)
-{
- struct net_device *soft_iface;
- int ret;
-
- soft_iface = alloc_netdev(sizeof(struct batadv_priv), name,
- NET_NAME_UNKNOWN, batadv_softif_init_early);
- if (!soft_iface)
- return NULL;
-
- dev_net_set(soft_iface, net);
-
- soft_iface->rtnl_link_ops = &batadv_link_ops;
-
- ret = register_netdevice(soft_iface);
- if (ret < 0) {
- pr_err("Unable to register the batman interface '%s': %i\n",
- name, ret);
- free_netdev(soft_iface);
- return NULL;
- }
-
- return soft_iface;
-}
-
-/**
* batadv_softif_destroy_netlink() - deletion of batadv_soft_interface via
* netlink
* @soft_iface: the to-be-removed batman-adv interface
#include <linux/netdevice.h>
#include <linux/skbuff.h>
#include <linux/types.h>
-#include <net/net_namespace.h>
#include <net/rtnetlink.h>
int batadv_skb_head_push(struct sk_buff *skb, unsigned int len);
void batadv_interface_rx(struct net_device *soft_iface,
struct sk_buff *skb, int hdr_size,
struct batadv_orig_node *orig_node);
-struct net_device *batadv_softif_create(struct net *net, const char *name);
bool batadv_softif_is_valid(const struct net_device *net_dev);
extern struct rtnl_link_ops batadv_link_ops;
int batadv_softif_create_vlan(struct batadv_priv *bat_priv, unsigned short vid);
conn->state = BT_CLOSED;
/* If the status indicates successful cancellation of
- * the attempt (i.e. Unkown Connection Id) there's no point of
+ * the attempt (i.e. Unknown Connection Id) there's no point of
* notifying failure since we'll go back to keep trying to
* connect. The only exception is explicit connect requests
* where a timeout + cancel does indicate an actual failure.
*/
/* If the controller supports Extended Scanner Filter
- * Policies, enable the correspondig event.
+ * Policies, enable the corresponding event.
*/
if (hdev->le_features[0] & HCI_LE_EXT_SCAN_POLICY)
events[1] |= 0x04; /* LE Direct Advertising
}
/* Check for valid public address or a configured static
- * random adddress, but let the HCI setup proceed to
+ * random address, but let the HCI setup proceed to
* be able to determine if there is a public address
* or not.
*
} else {
/* Init failed, cleanup */
flush_work(&hdev->tx_work);
- flush_work(&hdev->cmd_work);
+
+ /* Since hci_rx_work() is possible to awake new cmd_work
+ * it should be flushed first to avoid unexpected call of
+ * hci_cmd_work()
+ */
flush_work(&hdev->rx_work);
+ flush_work(&hdev->cmd_work);
skb_queue_purge(&hdev->cmd_q);
skb_queue_purge(&hdev->rx_q);
if (params->auto_connect != HCI_AUTO_CONN_DISABLED)
continue;
- /* If trying to estabilish one time connection to disabled
+ /* If trying to establish one time connection to disabled
* device, leave the params, but mark them as just once.
*/
if (params->explicit_connect) {
return hdev->sent_cmd->data + HCI_COMMAND_HDR_SIZE;
}
-/* Send HCI command and wait for command commplete event */
+/* Send HCI command and wait for command complete event */
struct sk_buff *hci_cmd_sync(struct hci_dev *hdev, u16 opcode, u32 plen,
const void *param, u32 timeout)
{
return true;
}
- /* Check if request ended in Command Status - no way to retreive
+ /* Check if request ended in Command Status - no way to retrieve
* any extra parameters in this case.
*/
if (hdr->evt == HCI_EV_CMD_STATUS)
/* Detach sockets from device */
read_lock(&hci_sk_list.lock);
sk_for_each(sk, &hci_sk_list.head) {
- bh_lock_sock_nested(sk);
+ lock_sock(sk);
if (hci_pi(sk)->hdev == hdev) {
hci_pi(sk)->hdev = NULL;
sk->sk_err = EPIPE;
hci_dev_put(hdev);
}
- bh_unlock_sock(sk);
+ release_sock(sk);
}
read_unlock(&hci_sk_list.lock);
}
if (!hci_sock_gen_cookie(sk)) {
/* In the case when a cookie has already been assigned,
* then there has been already an ioctl issued against
- * an unbound socket and with that triggerd an open
+ * an unbound socket and with that triggered an open
* notification. Send a close notification first to
* allow the state transition to bounded.
*/
if (hci_pi(sk)->channel == HCI_CHANNEL_CONTROL) {
if (!hci_sock_gen_cookie(sk)) {
/* In the case when a cookie has already been
- * assigned, this socket will transtion from
+ * assigned, this socket will transition from
* a raw socket into a control socket. To
- * allow for a clean transtion, send the
+ * allow for a clean transition, send the
* close notification first.
*/
skb = create_monitor_ctrl_close(sk);
}
/* The name is stored in the scan response data and so
- * no need to udpate the advertising data here.
+ * no need to update the advertising data here.
*/
if (lmp_le_capable(hdev) && hci_dev_test_flag(hdev, HCI_ADVERTISING))
__hci_req_update_scan_rsp_data(&req, hdev->cur_adv_instance);
((struct smp_dev *)((struct l2cap_chan *)((hdev)->smp_data))->data)
/* Low-level debug macros to be used for stuff that we don't want
- * accidentially in dmesg, i.e. the values of the various crypto keys
+ * accidentally in dmesg, i.e. the values of the various crypto keys
* and the inputs & outputs of crypto functions.
*/
#ifdef DEBUG
return err;
/* This is unlikely, but we need to check that
- * we didn't accidentially generate a debug key.
+ * we didn't accidentally generate a debug key.
*/
if (crypto_memneq(smp->local_pk, debug_pk, 64))
break;
return SMP_UNSPECIFIED;
/* This is unlikely, but we need to check that
- * we didn't accidentially generate a debug key.
+ * we didn't accidentally generate a debug key.
*/
if (crypto_memneq(smp->local_pk, debug_pk, 64))
break;
{
struct l2cap_chan *chan;
- bt_dev_dbg(pchan->conn->hcon->hdev, "pchan %p", pchan);
+ BT_DBG("pchan %p", pchan);
chan = l2cap_chan_create();
if (!chan)
*/
atomic_set(&chan->nesting, L2CAP_NESTING_SMP);
- bt_dev_dbg(pchan->conn->hcon->hdev, "created chan %p", chan);
+ BT_DBG("created chan %p", chan);
return chan;
}
{
struct smp_dev *smp;
- bt_dev_dbg(chan->conn->hcon->hdev, "chan %p", chan);
+ BT_DBG("chan %p", chan);
smp = chan->data;
if (smp) {
return ERR_PTR(-ENOMEM);
if (data_in) {
- err = bpf_check_uarg_tail_zero(data_in, max_size, size);
+ err = bpf_check_uarg_tail_zero(USER_BPFPTR(data_in), max_size, size);
if (err) {
kfree(data);
return ERR_PTR(err);
kfree(user_ctx);
return ret;
}
+
+int bpf_prog_test_run_syscall(struct bpf_prog *prog,
+ const union bpf_attr *kattr,
+ union bpf_attr __user *uattr)
+{
+ void __user *ctx_in = u64_to_user_ptr(kattr->test.ctx_in);
+ __u32 ctx_size_in = kattr->test.ctx_size_in;
+ void *ctx = NULL;
+ u32 retval;
+ int err = 0;
+
+ /* doesn't support data_in/out, ctx_out, duration, or repeat or flags */
+ if (kattr->test.data_in || kattr->test.data_out ||
+ kattr->test.ctx_out || kattr->test.duration ||
+ kattr->test.repeat || kattr->test.flags)
+ return -EINVAL;
+
+ if (ctx_size_in < prog->aux->max_ctx_offset ||
+ ctx_size_in > U16_MAX)
+ return -EINVAL;
+
+ if (ctx_size_in) {
+ ctx = kzalloc(ctx_size_in, GFP_USER);
+ if (!ctx)
+ return -ENOMEM;
+ if (copy_from_user(ctx, ctx_in, ctx_size_in)) {
+ err = -EFAULT;
+ goto out;
+ }
+ }
+ retval = bpf_prog_run_pin_on_cpu(prog, ctx);
+
+ if (copy_to_user(&uattr->test.retval, &retval, sizeof(u32))) {
+ err = -EFAULT;
+ goto out;
+ }
+ if (ctx_size_in)
+ if (copy_to_user(ctx_in, ctx, ctx_size_in))
+ err = -EFAULT;
+out:
+ kfree(ctx);
+ return err;
+}
{
u32 filter = RTEXT_FILTER_CFM_STATUS;
- return br_info_notify(event, port->br, NULL, filter);
+ br_info_notify(event, port->br, NULL, filter);
}
static void cc_peer_enable(struct br_cfm_peer_mep *peer_mep)
if (!mrp)
return -EINVAL;
- if (mrp->ring_state == BR_MRP_RING_STATE_CLOSED &&
- state->ring_state != BR_MRP_RING_STATE_CLOSED)
+ if (mrp->ring_state != state->ring_state)
mrp->ring_transitions++;
mrp->ring_state = state->ring_state;
if (!mrp)
return -EINVAL;
- if (mrp->in_state == BR_MRP_IN_STATE_CLOSED &&
- state->in_state != BR_MRP_IN_STATE_CLOSED)
+ if (mrp->in_state != state->in_state)
mrp->in_transitions++;
mrp->in_state = state->in_state;
static void
br_multicast_rport_del_notify(struct net_bridge_port *p, bool deleted);
-#if IS_ENABLED(CONFIG_IPV6)
static void br_ip6_multicast_add_router(struct net_bridge *br,
struct net_bridge_port *port);
+#if IS_ENABLED(CONFIG_IPV6)
static void br_ip6_multicast_leave_group(struct net_bridge *br,
struct net_bridge_port *port,
const struct in6_addr *group,
p = br_port_get_rtnl(dev);
if (!p)
return 0;
- br = p->br;
vg = nbp_vlan_group(p);
break;
default:
#endif
struct br_tunnel_info {
- __be64 tunnel_id;
- struct metadata_dst *tunnel_dst;
+ __be64 tunnel_id;
+ struct metadata_dst __rcu *tunnel_dst;
};
/* private vlan flags */
{
}
-static inline bool br_multicast_is_router(struct net_bridge *br)
+static inline bool br_multicast_is_router(struct net_bridge *br,
+ struct sk_buff *skb)
{
return false;
}
headp = &vg->vlan_list;
list_for_each_prev(hpos, headp) {
vent = list_entry(hpos, struct net_bridge_vlan, vlist);
- if (v->vid < vent->vid)
- continue;
- else
+ if (v->vid >= vent->vid)
break;
}
list_add_rcu(&v->vlist, hpos);
br_vlan_tunnel_rht_params);
}
+static void vlan_tunnel_info_release(struct net_bridge_vlan *vlan)
+{
+ struct metadata_dst *tdst = rtnl_dereference(vlan->tinfo.tunnel_dst);
+
+ WRITE_ONCE(vlan->tinfo.tunnel_id, 0);
+ RCU_INIT_POINTER(vlan->tinfo.tunnel_dst, NULL);
+ dst_release(&tdst->dst);
+}
+
void vlan_tunnel_info_del(struct net_bridge_vlan_group *vg,
struct net_bridge_vlan *vlan)
{
- if (!vlan->tinfo.tunnel_dst)
+ if (!rcu_access_pointer(vlan->tinfo.tunnel_dst))
return;
rhashtable_remove_fast(&vg->tunnel_hash, &vlan->tnode,
br_vlan_tunnel_rht_params);
- vlan->tinfo.tunnel_id = 0;
- dst_release(&vlan->tinfo.tunnel_dst->dst);
- vlan->tinfo.tunnel_dst = NULL;
+ vlan_tunnel_info_release(vlan);
}
static int __vlan_tunnel_info_add(struct net_bridge_vlan_group *vg,
struct net_bridge_vlan *vlan, u32 tun_id)
{
- struct metadata_dst *metadata = NULL;
+ struct metadata_dst *metadata = rtnl_dereference(vlan->tinfo.tunnel_dst);
__be64 key = key32_to_tunnel_id(cpu_to_be32(tun_id));
int err;
- if (vlan->tinfo.tunnel_dst)
+ if (metadata)
return -EEXIST;
metadata = __ip_tun_set_dst(0, 0, 0, 0, 0, TUNNEL_KEY,
return -EINVAL;
metadata->u.tun_info.mode |= IP_TUNNEL_INFO_TX | IP_TUNNEL_INFO_BRIDGE;
- vlan->tinfo.tunnel_dst = metadata;
- vlan->tinfo.tunnel_id = key;
+ rcu_assign_pointer(vlan->tinfo.tunnel_dst, metadata);
+ WRITE_ONCE(vlan->tinfo.tunnel_id, key);
err = rhashtable_lookup_insert_fast(&vg->tunnel_hash, &vlan->tnode,
br_vlan_tunnel_rht_params);
return 0;
out:
- dst_release(&vlan->tinfo.tunnel_dst->dst);
- vlan->tinfo.tunnel_dst = NULL;
- vlan->tinfo.tunnel_id = 0;
+ vlan_tunnel_info_release(vlan);
return err;
}
int br_handle_egress_vlan_tunnel(struct sk_buff *skb,
struct net_bridge_vlan *vlan)
{
+ struct metadata_dst *tunnel_dst;
+ __be64 tunnel_id;
int err;
- if (!vlan || !vlan->tinfo.tunnel_id)
+ if (!vlan)
return 0;
- if (unlikely(!skb_vlan_tag_present(skb)))
+ tunnel_id = READ_ONCE(vlan->tinfo.tunnel_id);
+ if (!tunnel_id || unlikely(!skb_vlan_tag_present(skb)))
return 0;
skb_dst_drop(skb);
if (err)
return err;
- skb_dst_set(skb, dst_clone(&vlan->tinfo.tunnel_dst->dst));
+ tunnel_dst = rcu_dereference(vlan->tinfo.tunnel_dst);
+ if (tunnel_dst && dst_hold_safe(&tunnel_dst->dst))
+ skb_dst_set(skb, &tunnel_dst->dst);
return 0;
}
caifd_put(caifd);
}
-void caif_enroll_dev(struct net_device *dev, struct caif_dev_common *caifdev,
+int caif_enroll_dev(struct net_device *dev, struct caif_dev_common *caifdev,
struct cflayer *link_support, int head_room,
struct cflayer **layer,
int (**rcv_func)(struct sk_buff *, struct net_device *,
enum cfcnfg_phy_preference pref;
struct cfcnfg *cfg = get_cfcnfg(dev_net(dev));
struct caif_device_entry_list *caifdevs;
+ int res;
caifdevs = caif_device_list(dev_net(dev));
caifd = caif_device_alloc(dev);
if (!caifd)
- return;
+ return -ENOMEM;
*layer = &caifd->layer;
spin_lock_init(&caifd->flow_lock);
strlcpy(caifd->layer.name, dev->name,
sizeof(caifd->layer.name));
caifd->layer.transmit = transmit;
- cfcnfg_add_phy_layer(cfg,
+ res = cfcnfg_add_phy_layer(cfg,
dev,
&caifd->layer,
pref,
mutex_unlock(&caifdevs->lock);
if (rcv_func)
*rcv_func = receive;
+ return res;
}
EXPORT_SYMBOL(caif_enroll_dev);
struct cflayer *layer, *link_support;
int head_room = 0;
struct caif_device_entry_list *caifdevs;
+ int res;
cfg = get_cfcnfg(dev_net(dev));
caifdevs = caif_device_list(dev_net(dev));
break;
}
}
- caif_enroll_dev(dev, caifdev, link_support, head_room,
+ res = caif_enroll_dev(dev, caifdev, link_support, head_room,
&layer, NULL);
+ if (res)
+ cfserl_release(link_support);
caifdev->flowctrl = dev_flowctrl;
break;
return (struct cflayer *) this;
}
+static void cfusbl_release(struct cflayer *layer)
+{
+ kfree(layer);
+}
+
static struct packet_type caif_usb_type __read_mostly = {
.type = cpu_to_be16(ETH_P_802_EX1),
};
struct cflayer *layer, *link_support;
struct usbnet *usbnet;
struct usb_device *usbdev;
+ int res;
/* Check whether we have a NCM device, and find its VID/PID. */
if (!(dev->dev.parent && dev->dev.parent->driver &&
if (dev->num_tx_queues > 1)
pr_warn("USB device uses more than one tx queue\n");
- caif_enroll_dev(dev, &common, link_support, CFUSB_MAX_HEADLEN,
+ res = caif_enroll_dev(dev, &common, link_support, CFUSB_MAX_HEADLEN,
&layer, &caif_usb_type.func);
+ if (res)
+ goto err;
+
if (!pack_added)
dev_add_pack(&caif_usb_type);
pack_added = true;
strlcpy(layer->name, dev->name, sizeof(layer->name));
return 0;
+err:
+ cfusbl_release(link_support);
+ return res;
}
static struct notifier_block caif_device_notifier = {
rcu_read_unlock();
}
-void
+int
cfcnfg_add_phy_layer(struct cfcnfg *cnfg,
struct net_device *dev, struct cflayer *phy_layer,
enum cfcnfg_phy_preference pref,
{
struct cflayer *frml;
struct cfcnfg_phyinfo *phyinfo = NULL;
- int i;
+ int i, res = 0;
u8 phyid;
mutex_lock(&cnfg->lock);
goto got_phyid;
}
pr_warn("Too many CAIF Link Layers (max 6)\n");
+ res = -EEXIST;
goto out;
got_phyid:
phyinfo = kzalloc(sizeof(struct cfcnfg_phyinfo), GFP_ATOMIC);
- if (!phyinfo)
- goto out_err;
+ if (!phyinfo) {
+ res = -ENOMEM;
+ goto out;
+ }
phy_layer->id = phyid;
phyinfo->pref = pref;
frml = cffrml_create(phyid, fcs);
- if (!frml)
+ if (!frml) {
+ res = -ENOMEM;
goto out_err;
+ }
phyinfo->frm_layer = frml;
layer_set_up(frml, cnfg->mux);
list_add_rcu(&phyinfo->node, &cnfg->phys);
out:
mutex_unlock(&cnfg->lock);
- return;
+ return res;
out_err:
kfree(phyinfo);
mutex_unlock(&cnfg->lock);
+ return res;
}
EXPORT_SYMBOL(cfcnfg_add_phy_layer);
static void cfserl_ctrlcmd(struct cflayer *layr, enum caif_ctrlcmd ctrl,
int phyid);
+void cfserl_release(struct cflayer *layer)
+{
+ kfree(layer);
+}
+
struct cflayer *cfserl_create(int instance, bool use_stx)
{
struct cfserl *this = kzalloc(sizeof(struct cfserl), GFP_ATOMIC);
struct sock sk;
int bound;
int ifindex;
- struct notifier_block notifier;
+ struct list_head notifier;
struct list_head rx_ops;
struct list_head tx_ops;
unsigned long dropped_usr_msgs;
char procname [32]; /* inode number in decimal with \0 */
};
+static LIST_HEAD(bcm_notifier_list);
+static DEFINE_SPINLOCK(bcm_notifier_lock);
+static struct bcm_sock *bcm_busy_notifier;
+
static inline struct bcm_sock *bcm_sk(const struct sock *sk)
{
return (struct bcm_sock *)sk;
if (!op->count && (op->flags & TX_COUNTEVT)) {
/* create notification to user */
+ memset(&msg_head, 0, sizeof(msg_head));
msg_head.opcode = TX_EXPIRED;
msg_head.flags = op->flags;
msg_head.count = op->count;
/* this element is not throttled anymore */
data->flags &= (BCM_CAN_FLAGS_MASK|RX_RECV);
+ memset(&head, 0, sizeof(head));
head.opcode = RX_CHANGED;
head.flags = op->flags;
head.count = op->count;
}
/* create notification to user */
+ memset(&msg_head, 0, sizeof(msg_head));
msg_head.opcode = RX_TIMEOUT;
msg_head.flags = op->flags;
msg_head.count = op->count;
/*
* notification handler for netdevice status changes
*/
-static int bcm_notifier(struct notifier_block *nb, unsigned long msg,
- void *ptr)
+static void bcm_notify(struct bcm_sock *bo, unsigned long msg,
+ struct net_device *dev)
{
- struct net_device *dev = netdev_notifier_info_to_dev(ptr);
- struct bcm_sock *bo = container_of(nb, struct bcm_sock, notifier);
struct sock *sk = &bo->sk;
struct bcm_op *op;
int notify_enodev = 0;
if (!net_eq(dev_net(dev), sock_net(sk)))
- return NOTIFY_DONE;
-
- if (dev->type != ARPHRD_CAN)
- return NOTIFY_DONE;
+ return;
switch (msg) {
sk->sk_error_report(sk);
}
}
+}
+static int bcm_notifier(struct notifier_block *nb, unsigned long msg,
+ void *ptr)
+{
+ struct net_device *dev = netdev_notifier_info_to_dev(ptr);
+
+ if (dev->type != ARPHRD_CAN)
+ return NOTIFY_DONE;
+ if (msg != NETDEV_UNREGISTER && msg != NETDEV_DOWN)
+ return NOTIFY_DONE;
+ if (unlikely(bcm_busy_notifier)) /* Check for reentrant bug. */
+ return NOTIFY_DONE;
+
+ spin_lock(&bcm_notifier_lock);
+ list_for_each_entry(bcm_busy_notifier, &bcm_notifier_list, notifier) {
+ spin_unlock(&bcm_notifier_lock);
+ bcm_notify(bcm_busy_notifier, msg, dev);
+ spin_lock(&bcm_notifier_lock);
+ }
+ bcm_busy_notifier = NULL;
+ spin_unlock(&bcm_notifier_lock);
return NOTIFY_DONE;
}
INIT_LIST_HEAD(&bo->rx_ops);
/* set notifier */
- bo->notifier.notifier_call = bcm_notifier;
-
- register_netdevice_notifier(&bo->notifier);
+ spin_lock(&bcm_notifier_lock);
+ list_add_tail(&bo->notifier, &bcm_notifier_list);
+ spin_unlock(&bcm_notifier_lock);
return 0;
}
/* remove bcm_ops, timer, rx_unregister(), etc. */
- unregister_netdevice_notifier(&bo->notifier);
+ spin_lock(&bcm_notifier_lock);
+ while (bcm_busy_notifier == bo) {
+ spin_unlock(&bcm_notifier_lock);
+ schedule_timeout_uninterruptible(1);
+ spin_lock(&bcm_notifier_lock);
+ }
+ list_del(&bo->notifier);
+ spin_unlock(&bcm_notifier_lock);
lock_sock(sk);
.exit = canbcm_pernet_exit,
};
+static struct notifier_block canbcm_notifier = {
+ .notifier_call = bcm_notifier
+};
+
static int __init bcm_module_init(void)
{
int err;
}
register_pernet_subsys(&canbcm_pernet_ops);
+ register_netdevice_notifier(&canbcm_notifier);
return 0;
}
static void __exit bcm_module_exit(void)
{
can_proto_unregister(&bcm_can_proto);
+ unregister_netdevice_notifier(&canbcm_notifier);
unregister_pernet_subsys(&canbcm_pernet_ops);
}
u32 force_tx_stmin;
u32 force_rx_stmin;
struct tpcon rx, tx;
- struct notifier_block notifier;
+ struct list_head notifier;
wait_queue_head_t wait;
};
+static LIST_HEAD(isotp_notifier_list);
+static DEFINE_SPINLOCK(isotp_notifier_lock);
+static struct isotp_sock *isotp_busy_notifier;
+
static inline struct isotp_sock *isotp_sk(const struct sock *sk)
{
return (struct isotp_sock *)sk;
can_send_ret = can_send(nskb, 1);
if (can_send_ret)
- pr_notice_once("can-isotp: %s: can_send_ret %d\n",
- __func__, can_send_ret);
+ pr_notice_once("can-isotp: %s: can_send_ret %pe\n",
+ __func__, ERR_PTR(can_send_ret));
dev_put(dev);
can_skb_set_owner(skb, sk);
can_send_ret = can_send(skb, 1);
- if (can_send_ret)
- pr_notice_once("can-isotp: %s: can_send_ret %d\n",
- __func__, can_send_ret);
-
+ if (can_send_ret) {
+ pr_notice_once("can-isotp: %s: can_send_ret %pe\n",
+ __func__, ERR_PTR(can_send_ret));
+ if (can_send_ret == -ENOBUFS)
+ pr_notice_once("can-isotp: tx queue is full, increasing txqueuelen may prevent this error\n");
+ }
if (so->tx.idx >= so->tx.len) {
/* we are done */
so->tx.state = ISOTP_IDLE;
err = can_send(skb, 1);
dev_put(dev);
if (err) {
- pr_notice_once("can-isotp: %s: can_send_ret %d\n",
- __func__, err);
+ pr_notice_once("can-isotp: %s: can_send_ret %pe\n",
+ __func__, ERR_PTR(err));
return err;
}
/* wait for complete transmission of current pdu */
wait_event_interruptible(so->wait, so->tx.state == ISOTP_IDLE);
- unregister_netdevice_notifier(&so->notifier);
+ spin_lock(&isotp_notifier_lock);
+ while (isotp_busy_notifier == so) {
+ spin_unlock(&isotp_notifier_lock);
+ schedule_timeout_uninterruptible(1);
+ spin_lock(&isotp_notifier_lock);
+ }
+ list_del(&so->notifier);
+ spin_unlock(&isotp_notifier_lock);
lock_sock(sk);
if (len < ISOTP_MIN_NAMELEN)
return -EINVAL;
+ if (addr->can_addr.tp.tx_id & (CAN_ERR_FLAG | CAN_RTR_FLAG))
+ return -EADDRNOTAVAIL;
+
+ if (!addr->can_ifindex)
+ return -ENODEV;
+
+ lock_sock(sk);
+
/* do not register frame reception for functional addressing */
if (so->opt.flags & CAN_ISOTP_SF_BROADCAST)
do_rx_reg = 0;
/* do not validate rx address for functional addressing */
if (do_rx_reg) {
- if (addr->can_addr.tp.rx_id == addr->can_addr.tp.tx_id)
- return -EADDRNOTAVAIL;
+ if (addr->can_addr.tp.rx_id == addr->can_addr.tp.tx_id) {
+ err = -EADDRNOTAVAIL;
+ goto out;
+ }
- if (addr->can_addr.tp.rx_id & (CAN_ERR_FLAG | CAN_RTR_FLAG))
- return -EADDRNOTAVAIL;
+ if (addr->can_addr.tp.rx_id & (CAN_ERR_FLAG | CAN_RTR_FLAG)) {
+ err = -EADDRNOTAVAIL;
+ goto out;
+ }
}
- if (addr->can_addr.tp.tx_id & (CAN_ERR_FLAG | CAN_RTR_FLAG))
- return -EADDRNOTAVAIL;
-
- if (!addr->can_ifindex)
- return -ENODEV;
-
- lock_sock(sk);
-
if (so->bound && addr->can_ifindex == so->ifindex &&
addr->can_addr.tp.rx_id == so->rxid &&
addr->can_addr.tp.tx_id == so->txid)
return ISOTP_MIN_NAMELEN;
}
-static int isotp_setsockopt(struct socket *sock, int level, int optname,
+static int isotp_setsockopt_locked(struct socket *sock, int level, int optname,
sockptr_t optval, unsigned int optlen)
{
struct sock *sk = sock->sk;
struct isotp_sock *so = isotp_sk(sk);
int ret = 0;
- if (level != SOL_CAN_ISOTP)
- return -EINVAL;
-
if (so->bound)
return -EISCONN;
return ret;
}
+static int isotp_setsockopt(struct socket *sock, int level, int optname,
+ sockptr_t optval, unsigned int optlen)
+
+{
+ struct sock *sk = sock->sk;
+ int ret;
+
+ if (level != SOL_CAN_ISOTP)
+ return -EINVAL;
+
+ lock_sock(sk);
+ ret = isotp_setsockopt_locked(sock, level, optname, optval, optlen);
+ release_sock(sk);
+ return ret;
+}
+
static int isotp_getsockopt(struct socket *sock, int level, int optname,
char __user *optval, int __user *optlen)
{
return 0;
}
-static int isotp_notifier(struct notifier_block *nb, unsigned long msg,
- void *ptr)
+static void isotp_notify(struct isotp_sock *so, unsigned long msg,
+ struct net_device *dev)
{
- struct net_device *dev = netdev_notifier_info_to_dev(ptr);
- struct isotp_sock *so = container_of(nb, struct isotp_sock, notifier);
struct sock *sk = &so->sk;
if (!net_eq(dev_net(dev), sock_net(sk)))
- return NOTIFY_DONE;
-
- if (dev->type != ARPHRD_CAN)
- return NOTIFY_DONE;
+ return;
if (so->ifindex != dev->ifindex)
- return NOTIFY_DONE;
+ return;
switch (msg) {
case NETDEV_UNREGISTER:
sk->sk_error_report(sk);
break;
}
+}
+
+static int isotp_notifier(struct notifier_block *nb, unsigned long msg,
+ void *ptr)
+{
+ struct net_device *dev = netdev_notifier_info_to_dev(ptr);
+
+ if (dev->type != ARPHRD_CAN)
+ return NOTIFY_DONE;
+ if (msg != NETDEV_UNREGISTER && msg != NETDEV_DOWN)
+ return NOTIFY_DONE;
+ if (unlikely(isotp_busy_notifier)) /* Check for reentrant bug. */
+ return NOTIFY_DONE;
+ spin_lock(&isotp_notifier_lock);
+ list_for_each_entry(isotp_busy_notifier, &isotp_notifier_list, notifier) {
+ spin_unlock(&isotp_notifier_lock);
+ isotp_notify(isotp_busy_notifier, msg, dev);
+ spin_lock(&isotp_notifier_lock);
+ }
+ isotp_busy_notifier = NULL;
+ spin_unlock(&isotp_notifier_lock);
return NOTIFY_DONE;
}
init_waitqueue_head(&so->wait);
- so->notifier.notifier_call = isotp_notifier;
- register_netdevice_notifier(&so->notifier);
+ spin_lock(&isotp_notifier_lock);
+ list_add_tail(&so->notifier, &isotp_notifier_list);
+ spin_unlock(&isotp_notifier_lock);
return 0;
}
.prot = &isotp_proto,
};
+static struct notifier_block canisotp_notifier = {
+ .notifier_call = isotp_notifier
+};
+
static __init int isotp_module_init(void)
{
int err;
err = can_proto_register(&isotp_can_proto);
if (err < 0)
- pr_err("can: registration of isotp protocol failed\n");
+ pr_err("can: registration of isotp protocol failed %pe\n", ERR_PTR(err));
+ else
+ register_netdevice_notifier(&canisotp_notifier);
return err;
}
static __exit void isotp_module_exit(void)
{
can_proto_unregister(&isotp_can_proto);
+ unregister_netdevice_notifier(&canisotp_notifier);
}
module_init(isotp_module_init);
if ((do_skcb->offset + do_skb->len) < offset_start) {
__skb_unlink(do_skb, &session->skb_queue);
+ /* drop ref taken in j1939_session_skb_queue() */
+ skb_unref(do_skb);
+
kfree_skb(do_skb);
}
spin_unlock_irqrestore(&session->skb_queue.lock, flags);
skcb->flags |= J1939_ECU_LOCAL_SRC;
+ skb_get(skb);
skb_queue_tail(&session->skb_queue, skb);
}
static struct
-sk_buff *j1939_session_skb_find_by_offset(struct j1939_session *session,
- unsigned int offset_start)
+sk_buff *j1939_session_skb_get_by_offset(struct j1939_session *session,
+ unsigned int offset_start)
{
struct j1939_priv *priv = session->priv;
struct j1939_sk_buff_cb *do_skcb;
skb = do_skb;
}
}
+
+ if (skb)
+ skb_get(skb);
+
spin_unlock_irqrestore(&session->skb_queue.lock, flags);
if (!skb)
return skb;
}
-static struct sk_buff *j1939_session_skb_find(struct j1939_session *session)
+static struct sk_buff *j1939_session_skb_get(struct j1939_session *session)
{
unsigned int offset_start;
offset_start = session->pkt.dpo * 7;
- return j1939_session_skb_find_by_offset(session, offset_start);
+ return j1939_session_skb_get_by_offset(session, offset_start);
}
/* see if we are receiver
int ret = 0;
u8 dat[8];
- se_skb = j1939_session_skb_find_by_offset(session, session->pkt.tx * 7);
+ se_skb = j1939_session_skb_get_by_offset(session, session->pkt.tx * 7);
if (!se_skb)
return -ENOBUFS;
netdev_err_once(priv->ndev,
"%s: 0x%p: requested data outside of queued buffer: offset %i, len %i, pkt.tx: %i\n",
__func__, session, skcb->offset, se_skb->len , session->pkt.tx);
- return -EOVERFLOW;
+ ret = -EOVERFLOW;
+ goto out_free;
}
if (!len) {
if (pkt_done)
j1939_tp_set_rxtimeout(session, 250);
+ out_free:
+ if (ret)
+ kfree_skb(se_skb);
+ else
+ consume_skb(se_skb);
+
return ret;
}
static int j1939_simple_txnext(struct j1939_session *session)
{
struct j1939_priv *priv = session->priv;
- struct sk_buff *se_skb = j1939_session_skb_find(session);
+ struct sk_buff *se_skb = j1939_session_skb_get(session);
struct sk_buff *skb;
int ret;
return 0;
skb = skb_clone(se_skb, GFP_ATOMIC);
- if (!skb)
- return -ENOMEM;
+ if (!skb) {
+ ret = -ENOMEM;
+ goto out_free;
+ }
can_skb_set_owner(skb, se_skb->sk);
ret = j1939_send_one(priv, skb);
if (ret)
- return ret;
+ goto out_free;
j1939_sk_errqueue(session, J1939_ERRQUEUE_SCHED);
j1939_sk_queue_activate_next(session);
- return 0;
+ out_free:
+ if (ret)
+ kfree_skb(se_skb);
+ else
+ consume_skb(se_skb);
+
+ return ret;
}
static bool j1939_session_deactivate_locked(struct j1939_session *session)
struct sk_buff *skb;
if (!session->transmission) {
- skb = j1939_session_skb_find(session);
+ skb = j1939_session_skb_get(session);
/* distribute among j1939 receivers */
j1939_sk_recv(session->priv, skb);
+ consume_skb(skb);
}
j1939_session_deactivate_activate_next(session);
{
struct j1939_priv *priv = session->priv;
struct j1939_sk_buff_cb *skcb;
- struct sk_buff *se_skb;
+ struct sk_buff *se_skb = NULL;
const u8 *dat;
u8 *tpdat;
int offset;
goto out_session_cancel;
}
- se_skb = j1939_session_skb_find_by_offset(session, packet * 7);
+ se_skb = j1939_session_skb_get_by_offset(session, packet * 7);
if (!se_skb) {
netdev_warn(priv->ndev, "%s: 0x%p: no skb found\n", __func__,
session);
j1939_tp_set_rxtimeout(session, 250);
}
session->last_cmd = 0xff;
+ consume_skb(se_skb);
j1939_session_put(session);
return;
out_session_cancel:
+ kfree_skb(se_skb);
j1939_session_timers_cancel(session);
j1939_session_cancel(session, J1939_XTP_ABORT_FAULT);
j1939_session_put(session);
static unsigned long calc_rate(unsigned long oldjif, unsigned long newjif,
unsigned long count)
{
- unsigned long rate;
-
if (oldjif == newjif)
return 0;
return 99999999;
}
- rate = (count * HZ) / (newjif - oldjif);
-
- return rate;
+ return (count * HZ) / (newjif - oldjif);
}
void can_stat_update(struct timer_list *t)
struct sock sk;
int bound;
int ifindex;
- struct notifier_block notifier;
+ struct list_head notifier;
int loopback;
int recv_own_msgs;
int fd_frames;
struct uniqframe __percpu *uniq;
};
+static LIST_HEAD(raw_notifier_list);
+static DEFINE_SPINLOCK(raw_notifier_lock);
+static struct raw_sock *raw_busy_notifier;
+
/* Return pointer to store the extra msg flags for raw_recvmsg().
* We use the space of one unsigned int beyond the 'struct sockaddr_can'
* in skb->cb.
return err;
}
-static int raw_notifier(struct notifier_block *nb,
- unsigned long msg, void *ptr)
+static void raw_notify(struct raw_sock *ro, unsigned long msg,
+ struct net_device *dev)
{
- struct net_device *dev = netdev_notifier_info_to_dev(ptr);
- struct raw_sock *ro = container_of(nb, struct raw_sock, notifier);
struct sock *sk = &ro->sk;
if (!net_eq(dev_net(dev), sock_net(sk)))
- return NOTIFY_DONE;
-
- if (dev->type != ARPHRD_CAN)
- return NOTIFY_DONE;
+ return;
if (ro->ifindex != dev->ifindex)
- return NOTIFY_DONE;
+ return;
switch (msg) {
case NETDEV_UNREGISTER:
sk->sk_error_report(sk);
break;
}
+}
+
+static int raw_notifier(struct notifier_block *nb, unsigned long msg,
+ void *ptr)
+{
+ struct net_device *dev = netdev_notifier_info_to_dev(ptr);
+
+ if (dev->type != ARPHRD_CAN)
+ return NOTIFY_DONE;
+ if (msg != NETDEV_UNREGISTER && msg != NETDEV_DOWN)
+ return NOTIFY_DONE;
+ if (unlikely(raw_busy_notifier)) /* Check for reentrant bug. */
+ return NOTIFY_DONE;
+ spin_lock(&raw_notifier_lock);
+ list_for_each_entry(raw_busy_notifier, &raw_notifier_list, notifier) {
+ spin_unlock(&raw_notifier_lock);
+ raw_notify(raw_busy_notifier, msg, dev);
+ spin_lock(&raw_notifier_lock);
+ }
+ raw_busy_notifier = NULL;
+ spin_unlock(&raw_notifier_lock);
return NOTIFY_DONE;
}
return -ENOMEM;
/* set notifier */
- ro->notifier.notifier_call = raw_notifier;
-
- register_netdevice_notifier(&ro->notifier);
+ spin_lock(&raw_notifier_lock);
+ list_add_tail(&ro->notifier, &raw_notifier_list);
+ spin_unlock(&raw_notifier_lock);
return 0;
}
ro = raw_sk(sk);
- unregister_netdevice_notifier(&ro->notifier);
+ spin_lock(&raw_notifier_lock);
+ while (raw_busy_notifier == ro) {
+ spin_unlock(&raw_notifier_lock);
+ schedule_timeout_uninterruptible(1);
+ spin_lock(&raw_notifier_lock);
+ }
+ list_del(&ro->notifier);
+ spin_unlock(&raw_notifier_lock);
lock_sock(sk);
.prot = &raw_proto,
};
+static struct notifier_block canraw_notifier = {
+ .notifier_call = raw_notifier
+};
+
static __init int raw_module_init(void)
{
int err;
err = can_proto_register(&raw_can_proto);
if (err < 0)
pr_err("can: registration of raw protocol failed\n");
+ else
+ register_netdevice_notifier(&canraw_notifier);
return err;
}
static __exit void raw_module_exit(void)
{
can_proto_unregister(&raw_can_proto);
+ unregister_netdevice_notifier(&canraw_notifier);
}
module_init(raw_module_init);
/*
- * encyption bundle
+ * encryption bundle
*/
#define CEPHX_ENC_MAGIC 0xff009cad8826aa55ull
return get_generic_reply(con, hdr, skip);
/*
- * Older OSDs don't set reply tid even if the orignal
+ * Older OSDs don't set reply tid even if the original
* request had a non-zero tid. Work around this weirdness
* by allocating a new message.
*/
return -EINVAL;
}
- /* old osdmap enconding */
+ /* old osdmap encoding */
struct_v = 0;
}
* parent, returns 0.
*
* Does a linear search, as there are no parent pointers of any
- * kind. Note that the result is ambigous for items that occur
+ * kind. Note that the result is ambiguous for items that occur
* multiple times in the map.
*/
static int get_immediate_parent(struct crush_map *c, int id,
if (kcmlen > stackbuf_size)
kcmsg_base = kcmsg = sock_kmalloc(sk, kcmlen, GFP_KERNEL);
if (kcmsg == NULL)
- return -ENOBUFS;
+ return -ENOMEM;
/* Now copy them over neatly. */
memset(kcmsg, 0, kcmlen);
qdisc_calculate_pkt_len(skb, q);
if (q->flags & TCQ_F_NOLOCK) {
+ if (q->flags & TCQ_F_CAN_BYPASS && nolock_qdisc_is_empty(q) &&
+ qdisc_run_begin(q)) {
+ /* Retest nolock_qdisc_is_empty() within the protection
+ * of q->seqlock to protect from racing with requeuing.
+ */
+ if (unlikely(!nolock_qdisc_is_empty(q))) {
+ rc = q->enqueue(skb, q, &to_free) &
+ NET_XMIT_MASK;
+ __qdisc_run(q);
+ qdisc_run_end(q);
+
+ goto no_lock_out;
+ }
+
+ qdisc_bstats_cpu_update(q, skb);
+ if (sch_direct_xmit(skb, q, dev, txq, NULL, true) &&
+ !nolock_qdisc_is_empty(q))
+ __qdisc_run(q);
+
+ qdisc_run_end(q);
+ return NET_XMIT_SUCCESS;
+ }
+
rc = q->enqueue(skb, q, &to_free) & NET_XMIT_MASK;
qdisc_run(q);
+no_lock_out:
if (unlikely(to_free))
kfree_skb_list(to_free);
return rc;
sd->output_queue_tailp = &sd->output_queue;
local_irq_enable();
+ rcu_read_lock();
+
while (head) {
struct Qdisc *q = head;
spinlock_t *root_lock = NULL;
head = head->next_sched;
- if (!(q->flags & TCQ_F_NOLOCK)) {
- root_lock = qdisc_lock(q);
- spin_lock(root_lock);
- }
/* We need to make sure head->next_sched is read
* before clearing __QDISC_STATE_SCHED
*/
smp_mb__before_atomic();
+
+ if (!(q->flags & TCQ_F_NOLOCK)) {
+ root_lock = qdisc_lock(q);
+ spin_lock(root_lock);
+ } else if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED,
+ &q->state))) {
+ /* There is a synchronize_net() between
+ * STATE_DEACTIVATED flag being set and
+ * qdisc_reset()/some_qdisc_is_busy() in
+ * dev_deactivate(), so we can safely bail out
+ * early here to avoid data race between
+ * qdisc_deactivate() and some_qdisc_is_busy()
+ * for lockless qdisc.
+ */
+ clear_bit(__QDISC_STATE_SCHED, &q->state);
+ continue;
+ }
+
clear_bit(__QDISC_STATE_SCHED, &q->state);
qdisc_run(q);
if (root_lock)
spin_unlock(root_lock);
}
+
+ rcu_read_unlock();
}
xfrm_dev_backlog(sd);
return devlink_port_get_from_attrs(devlink, info->attrs);
}
+static inline bool
+devlink_rate_is_leaf(struct devlink_rate *devlink_rate)
+{
+ return devlink_rate->type == DEVLINK_RATE_TYPE_LEAF;
+}
+
+static inline bool
+devlink_rate_is_node(struct devlink_rate *devlink_rate)
+{
+ return devlink_rate->type == DEVLINK_RATE_TYPE_NODE;
+}
+
+static struct devlink_rate *
+devlink_rate_leaf_get_from_info(struct devlink *devlink, struct genl_info *info)
+{
+ struct devlink_rate *devlink_rate;
+ struct devlink_port *devlink_port;
+
+ devlink_port = devlink_port_get_from_attrs(devlink, info->attrs);
+ if (IS_ERR(devlink_port))
+ return ERR_CAST(devlink_port);
+ devlink_rate = devlink_port->devlink_rate;
+ return devlink_rate ?: ERR_PTR(-ENODEV);
+}
+
+static struct devlink_rate *
+devlink_rate_node_get_by_name(struct devlink *devlink, const char *node_name)
+{
+ static struct devlink_rate *devlink_rate;
+
+ list_for_each_entry(devlink_rate, &devlink->rate_list, list) {
+ if (devlink_rate_is_node(devlink_rate) &&
+ !strcmp(node_name, devlink_rate->name))
+ return devlink_rate;
+ }
+ return ERR_PTR(-ENODEV);
+}
+
+static struct devlink_rate *
+devlink_rate_node_get_from_attrs(struct devlink *devlink, struct nlattr **attrs)
+{
+ const char *rate_node_name;
+ size_t len;
+
+ if (!attrs[DEVLINK_ATTR_RATE_NODE_NAME])
+ return ERR_PTR(-EINVAL);
+ rate_node_name = nla_data(attrs[DEVLINK_ATTR_RATE_NODE_NAME]);
+ len = strlen(rate_node_name);
+ /* Name cannot be empty or decimal number */
+ if (!len || strspn(rate_node_name, "0123456789") == len)
+ return ERR_PTR(-EINVAL);
+
+ return devlink_rate_node_get_by_name(devlink, rate_node_name);
+}
+
+static struct devlink_rate *
+devlink_rate_node_get_from_info(struct devlink *devlink, struct genl_info *info)
+{
+ return devlink_rate_node_get_from_attrs(devlink, info->attrs);
+}
+
+static struct devlink_rate *
+devlink_rate_get_from_info(struct devlink *devlink, struct genl_info *info)
+{
+ struct nlattr **attrs = info->attrs;
+
+ if (attrs[DEVLINK_ATTR_PORT_INDEX])
+ return devlink_rate_leaf_get_from_info(devlink, info);
+ else if (attrs[DEVLINK_ATTR_RATE_NODE_NAME])
+ return devlink_rate_node_get_from_info(devlink, info);
+ else
+ return ERR_PTR(-EINVAL);
+}
+
struct devlink_sb {
struct list_head list;
unsigned int index;
#define DEVLINK_NL_FLAG_NEED_PORT BIT(0)
#define DEVLINK_NL_FLAG_NEED_DEVLINK_OR_PORT BIT(1)
+#define DEVLINK_NL_FLAG_NEED_RATE BIT(2)
+#define DEVLINK_NL_FLAG_NEED_RATE_NODE BIT(3)
/* The per devlink instance lock is taken by default in the pre-doit
* operation, yet several commands do not require this. The global
* devlink lock is taken and protects from disruption by user-calls.
*/
-#define DEVLINK_NL_FLAG_NO_LOCK BIT(2)
+#define DEVLINK_NL_FLAG_NO_LOCK BIT(4)
static int devlink_nl_pre_doit(const struct genl_ops *ops,
struct sk_buff *skb, struct genl_info *info)
devlink_port = devlink_port_get_from_info(devlink, info);
if (!IS_ERR(devlink_port))
info->user_ptr[1] = devlink_port;
+ } else if (ops->internal_flags & DEVLINK_NL_FLAG_NEED_RATE) {
+ struct devlink_rate *devlink_rate;
+
+ devlink_rate = devlink_rate_get_from_info(devlink, info);
+ if (IS_ERR(devlink_rate)) {
+ err = PTR_ERR(devlink_rate);
+ goto unlock;
+ }
+ info->user_ptr[1] = devlink_rate;
+ } else if (ops->internal_flags & DEVLINK_NL_FLAG_NEED_RATE_NODE) {
+ struct devlink_rate *rate_node;
+
+ rate_node = devlink_rate_node_get_from_info(devlink, info);
+ if (IS_ERR(rate_node)) {
+ err = PTR_ERR(rate_node);
+ goto unlock;
+ }
+ info->user_ptr[1] = rate_node;
}
return 0;
case DEVLINK_PORT_FLAVOUR_PHYSICAL:
case DEVLINK_PORT_FLAVOUR_CPU:
case DEVLINK_PORT_FLAVOUR_DSA:
- case DEVLINK_PORT_FLAVOUR_VIRTUAL:
if (nla_put_u32(msg, DEVLINK_ATTR_PORT_NUMBER,
attrs->phys.port_number))
return -EMSGSIZE;
return 0;
}
+static int devlink_nl_rate_fill(struct sk_buff *msg,
+ struct devlink *devlink,
+ struct devlink_rate *devlink_rate,
+ enum devlink_command cmd, u32 portid,
+ u32 seq, int flags,
+ struct netlink_ext_ack *extack)
+{
+ void *hdr;
+
+ hdr = genlmsg_put(msg, portid, seq, &devlink_nl_family, flags, cmd);
+ if (!hdr)
+ return -EMSGSIZE;
+
+ if (devlink_nl_put_handle(msg, devlink))
+ goto nla_put_failure;
+
+ if (nla_put_u16(msg, DEVLINK_ATTR_RATE_TYPE, devlink_rate->type))
+ goto nla_put_failure;
+
+ if (devlink_rate_is_leaf(devlink_rate)) {
+ if (nla_put_u32(msg, DEVLINK_ATTR_PORT_INDEX,
+ devlink_rate->devlink_port->index))
+ goto nla_put_failure;
+ } else if (devlink_rate_is_node(devlink_rate)) {
+ if (nla_put_string(msg, DEVLINK_ATTR_RATE_NODE_NAME,
+ devlink_rate->name))
+ goto nla_put_failure;
+ }
+
+ if (nla_put_u64_64bit(msg, DEVLINK_ATTR_RATE_TX_SHARE,
+ devlink_rate->tx_share, DEVLINK_ATTR_PAD))
+ goto nla_put_failure;
+
+ if (nla_put_u64_64bit(msg, DEVLINK_ATTR_RATE_TX_MAX,
+ devlink_rate->tx_max, DEVLINK_ATTR_PAD))
+ goto nla_put_failure;
+
+ if (devlink_rate->parent)
+ if (nla_put_string(msg, DEVLINK_ATTR_RATE_PARENT_NODE_NAME,
+ devlink_rate->parent->name))
+ goto nla_put_failure;
+
+ genlmsg_end(msg, hdr);
+ return 0;
+
+nla_put_failure:
+ genlmsg_cancel(msg, hdr);
+ return -EMSGSIZE;
+}
+
static bool
devlink_port_fn_state_valid(enum devlink_port_fn_state state)
{
msg, 0, DEVLINK_MCGRP_CONFIG, GFP_KERNEL);
}
+static void devlink_rate_notify(struct devlink_rate *devlink_rate,
+ enum devlink_command cmd)
+{
+ struct devlink *devlink = devlink_rate->devlink;
+ struct sk_buff *msg;
+ int err;
+
+ WARN_ON(cmd != DEVLINK_CMD_RATE_NEW &&
+ cmd != DEVLINK_CMD_RATE_DEL);
+
+ msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
+ if (!msg)
+ return;
+
+ err = devlink_nl_rate_fill(msg, devlink, devlink_rate,
+ cmd, 0, 0, 0, NULL);
+ if (err) {
+ nlmsg_free(msg);
+ return;
+ }
+
+ genlmsg_multicast_netns(&devlink_nl_family, devlink_net(devlink),
+ msg, 0, DEVLINK_MCGRP_CONFIG, GFP_KERNEL);
+}
+
+static int devlink_nl_cmd_rate_get_dumpit(struct sk_buff *msg,
+ struct netlink_callback *cb)
+{
+ struct devlink_rate *devlink_rate;
+ struct devlink *devlink;
+ int start = cb->args[0];
+ int idx = 0;
+ int err = 0;
+
+ mutex_lock(&devlink_mutex);
+ list_for_each_entry(devlink, &devlink_list, list) {
+ if (!net_eq(devlink_net(devlink), sock_net(msg->sk)))
+ continue;
+ mutex_lock(&devlink->lock);
+ list_for_each_entry(devlink_rate, &devlink->rate_list, list) {
+ enum devlink_command cmd = DEVLINK_CMD_RATE_NEW;
+ u32 id = NETLINK_CB(cb->skb).portid;
+
+ if (idx < start) {
+ idx++;
+ continue;
+ }
+ err = devlink_nl_rate_fill(msg, devlink,
+ devlink_rate,
+ cmd, id,
+ cb->nlh->nlmsg_seq,
+ NLM_F_MULTI, NULL);
+ if (err) {
+ mutex_unlock(&devlink->lock);
+ goto out;
+ }
+ idx++;
+ }
+ mutex_unlock(&devlink->lock);
+ }
+out:
+ mutex_unlock(&devlink_mutex);
+ if (err != -EMSGSIZE)
+ return err;
+
+ cb->args[0] = idx;
+ return msg->len;
+}
+
+static int devlink_nl_cmd_rate_get_doit(struct sk_buff *skb,
+ struct genl_info *info)
+{
+ struct devlink_rate *devlink_rate = info->user_ptr[1];
+ struct devlink *devlink = devlink_rate->devlink;
+ struct sk_buff *msg;
+ int err;
+
+ msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
+ if (!msg)
+ return -ENOMEM;
+
+ err = devlink_nl_rate_fill(msg, devlink, devlink_rate,
+ DEVLINK_CMD_RATE_NEW,
+ info->snd_portid, info->snd_seq, 0,
+ info->extack);
+ if (err) {
+ nlmsg_free(msg);
+ return err;
+ }
+
+ return genlmsg_reply(msg, info);
+}
+
+static bool
+devlink_rate_is_parent_node(struct devlink_rate *devlink_rate,
+ struct devlink_rate *parent)
+{
+ while (parent) {
+ if (parent == devlink_rate)
+ return true;
+ parent = parent->parent;
+ }
+ return false;
+}
+
static int devlink_nl_cmd_get_doit(struct sk_buff *skb, struct genl_info *info)
{
struct devlink *devlink = info->user_ptr[0];
return devlink->ops->port_del(devlink, port_index, extack);
}
+static int
+devlink_nl_rate_parent_node_set(struct devlink_rate *devlink_rate,
+ struct genl_info *info,
+ struct nlattr *nla_parent)
+{
+ struct devlink *devlink = devlink_rate->devlink;
+ const char *parent_name = nla_data(nla_parent);
+ const struct devlink_ops *ops = devlink->ops;
+ size_t len = strlen(parent_name);
+ struct devlink_rate *parent;
+ int err = -EOPNOTSUPP;
+
+ parent = devlink_rate->parent;
+ if (parent && len) {
+ NL_SET_ERR_MSG_MOD(info->extack, "Rate object already has parent.");
+ return -EBUSY;
+ } else if (parent && !len) {
+ if (devlink_rate_is_leaf(devlink_rate))
+ err = ops->rate_leaf_parent_set(devlink_rate, NULL,
+ devlink_rate->priv, NULL,
+ info->extack);
+ else if (devlink_rate_is_node(devlink_rate))
+ err = ops->rate_node_parent_set(devlink_rate, NULL,
+ devlink_rate->priv, NULL,
+ info->extack);
+ if (err)
+ return err;
+
+ refcount_dec(&parent->refcnt);
+ devlink_rate->parent = NULL;
+ } else if (!parent && len) {
+ parent = devlink_rate_node_get_by_name(devlink, parent_name);
+ if (IS_ERR(parent))
+ return -ENODEV;
+
+ if (parent == devlink_rate) {
+ NL_SET_ERR_MSG_MOD(info->extack, "Parent to self is not allowed");
+ return -EINVAL;
+ }
+
+ if (devlink_rate_is_node(devlink_rate) &&
+ devlink_rate_is_parent_node(devlink_rate, parent->parent)) {
+ NL_SET_ERR_MSG_MOD(info->extack, "Node is already a parent of parent node.");
+ return -EEXIST;
+ }
+
+ if (devlink_rate_is_leaf(devlink_rate))
+ err = ops->rate_leaf_parent_set(devlink_rate, parent,
+ devlink_rate->priv, parent->priv,
+ info->extack);
+ else if (devlink_rate_is_node(devlink_rate))
+ err = ops->rate_node_parent_set(devlink_rate, parent,
+ devlink_rate->priv, parent->priv,
+ info->extack);
+ if (err)
+ return err;
+
+ refcount_inc(&parent->refcnt);
+ devlink_rate->parent = parent;
+ }
+
+ return 0;
+}
+
+static int devlink_nl_rate_set(struct devlink_rate *devlink_rate,
+ const struct devlink_ops *ops,
+ struct genl_info *info)
+{
+ struct nlattr *nla_parent, **attrs = info->attrs;
+ int err = -EOPNOTSUPP;
+ u64 rate;
+
+ if (attrs[DEVLINK_ATTR_RATE_TX_SHARE]) {
+ rate = nla_get_u64(attrs[DEVLINK_ATTR_RATE_TX_SHARE]);
+ if (devlink_rate_is_leaf(devlink_rate))
+ err = ops->rate_leaf_tx_share_set(devlink_rate, devlink_rate->priv,
+ rate, info->extack);
+ else if (devlink_rate_is_node(devlink_rate))
+ err = ops->rate_node_tx_share_set(devlink_rate, devlink_rate->priv,
+ rate, info->extack);
+ if (err)
+ return err;
+ devlink_rate->tx_share = rate;
+ }
+
+ if (attrs[DEVLINK_ATTR_RATE_TX_MAX]) {
+ rate = nla_get_u64(attrs[DEVLINK_ATTR_RATE_TX_MAX]);
+ if (devlink_rate_is_leaf(devlink_rate))
+ err = ops->rate_leaf_tx_max_set(devlink_rate, devlink_rate->priv,
+ rate, info->extack);
+ else if (devlink_rate_is_node(devlink_rate))
+ err = ops->rate_node_tx_max_set(devlink_rate, devlink_rate->priv,
+ rate, info->extack);
+ if (err)
+ return err;
+ devlink_rate->tx_max = rate;
+ }
+
+ nla_parent = attrs[DEVLINK_ATTR_RATE_PARENT_NODE_NAME];
+ if (nla_parent) {
+ err = devlink_nl_rate_parent_node_set(devlink_rate, info,
+ nla_parent);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+static bool devlink_rate_set_ops_supported(const struct devlink_ops *ops,
+ struct genl_info *info,
+ enum devlink_rate_type type)
+{
+ struct nlattr **attrs = info->attrs;
+
+ if (type == DEVLINK_RATE_TYPE_LEAF) {
+ if (attrs[DEVLINK_ATTR_RATE_TX_SHARE] && !ops->rate_leaf_tx_share_set) {
+ NL_SET_ERR_MSG_MOD(info->extack, "TX share set isn't supported for the leafs");
+ return false;
+ }
+ if (attrs[DEVLINK_ATTR_RATE_TX_MAX] && !ops->rate_leaf_tx_max_set) {
+ NL_SET_ERR_MSG_MOD(info->extack, "TX max set isn't supported for the leafs");
+ return false;
+ }
+ if (attrs[DEVLINK_ATTR_RATE_PARENT_NODE_NAME] &&
+ !ops->rate_leaf_parent_set) {
+ NL_SET_ERR_MSG_MOD(info->extack, "Parent set isn't supported for the leafs");
+ return false;
+ }
+ } else if (type == DEVLINK_RATE_TYPE_NODE) {
+ if (attrs[DEVLINK_ATTR_RATE_TX_SHARE] && !ops->rate_node_tx_share_set) {
+ NL_SET_ERR_MSG_MOD(info->extack, "TX share set isn't supported for the nodes");
+ return false;
+ }
+ if (attrs[DEVLINK_ATTR_RATE_TX_MAX] && !ops->rate_node_tx_max_set) {
+ NL_SET_ERR_MSG_MOD(info->extack, "TX max set isn't supported for the nodes");
+ return false;
+ }
+ if (attrs[DEVLINK_ATTR_RATE_PARENT_NODE_NAME] &&
+ !ops->rate_node_parent_set) {
+ NL_SET_ERR_MSG_MOD(info->extack, "Parent set isn't supported for the nodes");
+ return false;
+ }
+ } else {
+ WARN(1, "Unknown type of rate object");
+ return false;
+ }
+
+ return true;
+}
+
+static int devlink_nl_cmd_rate_set_doit(struct sk_buff *skb,
+ struct genl_info *info)
+{
+ struct devlink_rate *devlink_rate = info->user_ptr[1];
+ struct devlink *devlink = devlink_rate->devlink;
+ const struct devlink_ops *ops = devlink->ops;
+ int err;
+
+ if (!ops || !devlink_rate_set_ops_supported(ops, info, devlink_rate->type))
+ return -EOPNOTSUPP;
+
+ err = devlink_nl_rate_set(devlink_rate, ops, info);
+
+ if (!err)
+ devlink_rate_notify(devlink_rate, DEVLINK_CMD_RATE_NEW);
+ return err;
+}
+
+static int devlink_nl_cmd_rate_new_doit(struct sk_buff *skb,
+ struct genl_info *info)
+{
+ struct devlink *devlink = info->user_ptr[0];
+ struct devlink_rate *rate_node;
+ const struct devlink_ops *ops;
+ int err;
+
+ ops = devlink->ops;
+ if (!ops || !ops->rate_node_new || !ops->rate_node_del) {
+ NL_SET_ERR_MSG_MOD(info->extack, "Rate nodes aren't supported");
+ return -EOPNOTSUPP;
+ }
+
+ if (!devlink_rate_set_ops_supported(ops, info, DEVLINK_RATE_TYPE_NODE))
+ return -EOPNOTSUPP;
+
+ rate_node = devlink_rate_node_get_from_attrs(devlink, info->attrs);
+ if (!IS_ERR(rate_node))
+ return -EEXIST;
+ else if (rate_node == ERR_PTR(-EINVAL))
+ return -EINVAL;
+
+ rate_node = kzalloc(sizeof(*rate_node), GFP_KERNEL);
+ if (!rate_node)
+ return -ENOMEM;
+
+ rate_node->devlink = devlink;
+ rate_node->type = DEVLINK_RATE_TYPE_NODE;
+ rate_node->name = nla_strdup(info->attrs[DEVLINK_ATTR_RATE_NODE_NAME], GFP_KERNEL);
+ if (!rate_node->name) {
+ err = -ENOMEM;
+ goto err_strdup;
+ }
+
+ err = ops->rate_node_new(rate_node, &rate_node->priv, info->extack);
+ if (err)
+ goto err_node_new;
+
+ err = devlink_nl_rate_set(rate_node, ops, info);
+ if (err)
+ goto err_rate_set;
+
+ refcount_set(&rate_node->refcnt, 1);
+ list_add(&rate_node->list, &devlink->rate_list);
+ devlink_rate_notify(rate_node, DEVLINK_CMD_RATE_NEW);
+ return 0;
+
+err_rate_set:
+ ops->rate_node_del(rate_node, rate_node->priv, info->extack);
+err_node_new:
+ kfree(rate_node->name);
+err_strdup:
+ kfree(rate_node);
+ return err;
+}
+
+static int devlink_nl_cmd_rate_del_doit(struct sk_buff *skb,
+ struct genl_info *info)
+{
+ struct devlink_rate *rate_node = info->user_ptr[1];
+ struct devlink *devlink = rate_node->devlink;
+ const struct devlink_ops *ops = devlink->ops;
+ int err;
+
+ if (refcount_read(&rate_node->refcnt) > 1) {
+ NL_SET_ERR_MSG_MOD(info->extack, "Node has children. Cannot delete node.");
+ return -EBUSY;
+ }
+
+ devlink_rate_notify(rate_node, DEVLINK_CMD_RATE_DEL);
+ err = ops->rate_node_del(rate_node, rate_node->priv, info->extack);
+ if (rate_node->parent)
+ refcount_dec(&rate_node->parent->refcnt);
+ list_del(&rate_node->list);
+ kfree(rate_node->name);
+ kfree(rate_node);
+ return err;
+}
+
static int devlink_nl_sb_fill(struct sk_buff *msg, struct devlink *devlink,
struct devlink_sb *devlink_sb,
enum devlink_command cmd, u32 portid,
return genlmsg_reply(msg, info);
}
+static int devlink_rate_nodes_check(struct devlink *devlink, u16 mode,
+ struct netlink_ext_ack *extack)
+{
+ struct devlink_rate *devlink_rate;
+
+ /* Take the lock to sync with devlink_rate_nodes_destroy() */
+ mutex_lock(&devlink->lock);
+ list_for_each_entry(devlink_rate, &devlink->rate_list, list)
+ if (devlink_rate_is_node(devlink_rate)) {
+ mutex_unlock(&devlink->lock);
+ NL_SET_ERR_MSG_MOD(extack, "Rate node(s) exists.");
+ return -EBUSY;
+ }
+ mutex_unlock(&devlink->lock);
+ return 0;
+}
+
static int devlink_nl_cmd_eswitch_set_doit(struct sk_buff *skb,
struct genl_info *info)
{
if (!ops->eswitch_mode_set)
return -EOPNOTSUPP;
mode = nla_get_u16(info->attrs[DEVLINK_ATTR_ESWITCH_MODE]);
+ err = devlink_rate_nodes_check(devlink, mode, info->extack);
+ if (err)
+ return err;
err = ops->eswitch_mode_set(devlink, mode, info->extack);
if (err)
return err;
}
}
-static int devlink_trap_stats_put(struct sk_buff *msg,
- struct devlink_stats __percpu *trap_stats)
+static int
+devlink_trap_group_stats_put(struct sk_buff *msg,
+ struct devlink_stats __percpu *trap_stats)
{
struct devlink_stats stats;
struct nlattr *attr;
return -EMSGSIZE;
}
+static int devlink_trap_stats_put(struct sk_buff *msg, struct devlink *devlink,
+ const struct devlink_trap_item *trap_item)
+{
+ struct devlink_stats stats;
+ struct nlattr *attr;
+ u64 drops = 0;
+ int err;
+
+ if (devlink->ops->trap_drop_counter_get) {
+ err = devlink->ops->trap_drop_counter_get(devlink,
+ trap_item->trap,
+ &drops);
+ if (err)
+ return err;
+ }
+
+ devlink_trap_stats_read(trap_item->stats, &stats);
+
+ attr = nla_nest_start(msg, DEVLINK_ATTR_STATS);
+ if (!attr)
+ return -EMSGSIZE;
+
+ if (devlink->ops->trap_drop_counter_get &&
+ nla_put_u64_64bit(msg, DEVLINK_ATTR_STATS_RX_DROPPED, drops,
+ DEVLINK_ATTR_PAD))
+ goto nla_put_failure;
+
+ if (nla_put_u64_64bit(msg, DEVLINK_ATTR_STATS_RX_PACKETS,
+ stats.rx_packets, DEVLINK_ATTR_PAD))
+ goto nla_put_failure;
+
+ if (nla_put_u64_64bit(msg, DEVLINK_ATTR_STATS_RX_BYTES,
+ stats.rx_bytes, DEVLINK_ATTR_PAD))
+ goto nla_put_failure;
+
+ nla_nest_end(msg, attr);
+
+ return 0;
+
+nla_put_failure:
+ nla_nest_cancel(msg, attr);
+ return -EMSGSIZE;
+}
+
static int devlink_nl_trap_fill(struct sk_buff *msg, struct devlink *devlink,
const struct devlink_trap_item *trap_item,
enum devlink_command cmd, u32 portid, u32 seq,
if (err)
goto nla_put_failure;
- err = devlink_trap_stats_put(msg, trap_item->stats);
+ err = devlink_trap_stats_put(msg, devlink, trap_item);
if (err)
goto nla_put_failure;
group_item->policer_item->policer->id))
goto nla_put_failure;
- err = devlink_trap_stats_put(msg, group_item->stats);
+ err = devlink_trap_group_stats_put(msg, group_item->stats);
if (err)
goto nla_put_failure;
[DEVLINK_ATTR_PORT_PCI_PF_NUMBER] = { .type = NLA_U16 },
[DEVLINK_ATTR_PORT_PCI_SF_NUMBER] = { .type = NLA_U32 },
[DEVLINK_ATTR_PORT_CONTROLLER_NUMBER] = { .type = NLA_U32 },
+ [DEVLINK_ATTR_RATE_TYPE] = { .type = NLA_U16 },
+ [DEVLINK_ATTR_RATE_TX_SHARE] = { .type = NLA_U64 },
+ [DEVLINK_ATTR_RATE_TX_MAX] = { .type = NLA_U64 },
+ [DEVLINK_ATTR_RATE_NODE_NAME] = { .type = NLA_NUL_STRING },
+ [DEVLINK_ATTR_RATE_PARENT_NODE_NAME] = { .type = NLA_NUL_STRING },
};
static const struct genl_small_ops devlink_nl_ops[] = {
.internal_flags = DEVLINK_NL_FLAG_NEED_PORT,
},
{
+ .cmd = DEVLINK_CMD_RATE_GET,
+ .doit = devlink_nl_cmd_rate_get_doit,
+ .dumpit = devlink_nl_cmd_rate_get_dumpit,
+ .internal_flags = DEVLINK_NL_FLAG_NEED_RATE,
+ /* can be retrieved by unprivileged users */
+ },
+ {
+ .cmd = DEVLINK_CMD_RATE_SET,
+ .doit = devlink_nl_cmd_rate_set_doit,
+ .flags = GENL_ADMIN_PERM,
+ .internal_flags = DEVLINK_NL_FLAG_NEED_RATE,
+ },
+ {
+ .cmd = DEVLINK_CMD_RATE_NEW,
+ .doit = devlink_nl_cmd_rate_new_doit,
+ .flags = GENL_ADMIN_PERM,
+ },
+ {
+ .cmd = DEVLINK_CMD_RATE_DEL,
+ .doit = devlink_nl_cmd_rate_del_doit,
+ .flags = GENL_ADMIN_PERM,
+ .internal_flags = DEVLINK_NL_FLAG_NEED_RATE_NODE,
+ },
+ {
.cmd = DEVLINK_CMD_PORT_SPLIT,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = devlink_nl_cmd_port_split_doit,
xa_init_flags(&devlink->snapshot_ids, XA_FLAGS_ALLOC);
__devlink_net_set(devlink, &init_net);
INIT_LIST_HEAD(&devlink->port_list);
+ INIT_LIST_HEAD(&devlink->rate_list);
INIT_LIST_HEAD(&devlink->sb_list);
INIT_LIST_HEAD_RCU(&devlink->dpipe_table_list);
INIT_LIST_HEAD(&devlink->resource_list);
WARN_ON(!list_empty(&devlink->resource_list));
WARN_ON(!list_empty(&devlink->dpipe_table_list));
WARN_ON(!list_empty(&devlink->sb_list));
+ WARN_ON(!list_empty(&devlink->rate_list));
WARN_ON(!list_empty(&devlink->port_list));
xa_destroy(&devlink->snapshot_ids);
}
EXPORT_SYMBOL_GPL(devlink_port_attrs_pci_sf_set);
+/**
+ * devlink_rate_leaf_create - create devlink rate leaf
+ *
+ * @devlink_port: devlink port object to create rate object on
+ * @priv: driver private data
+ *
+ * Create devlink rate object of type leaf on provided @devlink_port.
+ * Throws call trace if @devlink_port already has a devlink rate object.
+ *
+ * Context: Takes and release devlink->lock <mutex>.
+ *
+ * Return: -ENOMEM if failed to allocate rate object, 0 otherwise.
+ */
+int
+devlink_rate_leaf_create(struct devlink_port *devlink_port, void *priv)
+{
+ struct devlink *devlink = devlink_port->devlink;
+ struct devlink_rate *devlink_rate;
+
+ devlink_rate = kzalloc(sizeof(*devlink_rate), GFP_KERNEL);
+ if (!devlink_rate)
+ return -ENOMEM;
+
+ mutex_lock(&devlink->lock);
+ WARN_ON(devlink_port->devlink_rate);
+ devlink_rate->type = DEVLINK_RATE_TYPE_LEAF;
+ devlink_rate->devlink = devlink;
+ devlink_rate->devlink_port = devlink_port;
+ devlink_rate->priv = priv;
+ list_add_tail(&devlink_rate->list, &devlink->rate_list);
+ devlink_port->devlink_rate = devlink_rate;
+ devlink_rate_notify(devlink_rate, DEVLINK_CMD_RATE_NEW);
+ mutex_unlock(&devlink->lock);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(devlink_rate_leaf_create);
+
+/**
+ * devlink_rate_leaf_destroy - destroy devlink rate leaf
+ *
+ * @devlink_port: devlink port linked to the rate object
+ *
+ * Context: Takes and release devlink->lock <mutex>.
+ */
+void devlink_rate_leaf_destroy(struct devlink_port *devlink_port)
+{
+ struct devlink_rate *devlink_rate = devlink_port->devlink_rate;
+ struct devlink *devlink = devlink_port->devlink;
+
+ if (!devlink_rate)
+ return;
+
+ mutex_lock(&devlink->lock);
+ devlink_rate_notify(devlink_rate, DEVLINK_CMD_RATE_DEL);
+ if (devlink_rate->parent)
+ refcount_dec(&devlink_rate->parent->refcnt);
+ list_del(&devlink_rate->list);
+ devlink_port->devlink_rate = NULL;
+ mutex_unlock(&devlink->lock);
+ kfree(devlink_rate);
+}
+EXPORT_SYMBOL_GPL(devlink_rate_leaf_destroy);
+
+/**
+ * devlink_rate_nodes_destroy - destroy all devlink rate nodes on device
+ *
+ * @devlink: devlink instance
+ *
+ * Unset parent for all rate objects and destroy all rate nodes
+ * on specified device.
+ *
+ * Context: Takes and release devlink->lock <mutex>.
+ */
+void devlink_rate_nodes_destroy(struct devlink *devlink)
+{
+ static struct devlink_rate *devlink_rate, *tmp;
+ const struct devlink_ops *ops = devlink->ops;
+
+ mutex_lock(&devlink->lock);
+ list_for_each_entry(devlink_rate, &devlink->rate_list, list) {
+ if (!devlink_rate->parent)
+ continue;
+
+ refcount_dec(&devlink_rate->parent->refcnt);
+ if (devlink_rate_is_leaf(devlink_rate))
+ ops->rate_leaf_parent_set(devlink_rate, NULL, devlink_rate->priv,
+ NULL, NULL);
+ else if (devlink_rate_is_node(devlink_rate))
+ ops->rate_node_parent_set(devlink_rate, NULL, devlink_rate->priv,
+ NULL, NULL);
+ }
+ list_for_each_entry_safe(devlink_rate, tmp, &devlink->rate_list, list) {
+ if (devlink_rate_is_node(devlink_rate)) {
+ ops->rate_node_del(devlink_rate, devlink_rate->priv, NULL);
+ list_del(&devlink_rate->list);
+ kfree(devlink_rate->name);
+ kfree(devlink_rate);
+ }
+ }
+ mutex_unlock(&devlink->lock);
+}
+EXPORT_SYMBOL_GPL(devlink_rate_nodes_destroy);
+
static int __devlink_port_phys_port_name_get(struct devlink_port *devlink_port,
char *name, size_t len)
{
switch (attrs->flavour) {
case DEVLINK_PORT_FLAVOUR_PHYSICAL:
case DEVLINK_PORT_FLAVOUR_VIRTUAL:
+ n = snprintf(name, len, "p%u", attrs->phys.port_number);
+ if (n < len && attrs->split)
+ n += snprintf(name + n, len - n, "s%u",
+ attrs->phys.split_subport_number);
if (!attrs->split)
n = snprintf(name, len, "p%u", attrs->phys.port_number);
else
n = snprintf(name, len, "p%us%u",
attrs->phys.port_number,
attrs->phys.split_subport_number);
+
break;
case DEVLINK_PORT_FLAVOUR_CPU:
case DEVLINK_PORT_FLAVOUR_DSA:
{
struct net *net;
struct sk_buff *skb;
- int err = -ENOBUFS;
+ int err = -ENOMEM;
net = ops->fro_net;
skb = nlmsg_new(fib_rule_nlmsg_size(ops, rule), GFP_KERNEL);
return ret;
}
-static int bpf_skb_proto_4_to_6(struct sk_buff *skb)
+static int bpf_skb_proto_4_to_6(struct sk_buff *skb, u64 flags)
{
const u32 len_diff = sizeof(struct ipv6hdr) - sizeof(struct iphdr);
u32 off = skb_mac_header_len(skb);
}
/* Due to IPv6 header, MSS needs to be downgraded. */
- skb_decrease_gso_size(shinfo, len_diff);
+ if (!(flags & BPF_F_ADJ_ROOM_FIXED_GSO))
+ skb_decrease_gso_size(shinfo, len_diff);
+
/* Header must be checked, and gso_segs recomputed. */
shinfo->gso_type |= SKB_GSO_DODGY;
shinfo->gso_segs = 0;
return 0;
}
-static int bpf_skb_proto_6_to_4(struct sk_buff *skb)
+static int bpf_skb_proto_6_to_4(struct sk_buff *skb, u64 flags)
{
const u32 len_diff = sizeof(struct ipv6hdr) - sizeof(struct iphdr);
u32 off = skb_mac_header_len(skb);
}
/* Due to IPv4 header, MSS can be upgraded. */
- skb_increase_gso_size(shinfo, len_diff);
+ if (!(flags & BPF_F_ADJ_ROOM_FIXED_GSO))
+ skb_increase_gso_size(shinfo, len_diff);
+
/* Header must be checked, and gso_segs recomputed. */
shinfo->gso_type |= SKB_GSO_DODGY;
shinfo->gso_segs = 0;
return 0;
}
-static int bpf_skb_proto_xlat(struct sk_buff *skb, __be16 to_proto)
+static int bpf_skb_proto_xlat(struct sk_buff *skb, __be16 to_proto, u64 flags)
{
__be16 from_proto = skb->protocol;
if (from_proto == htons(ETH_P_IP) &&
to_proto == htons(ETH_P_IPV6))
- return bpf_skb_proto_4_to_6(skb);
+ return bpf_skb_proto_4_to_6(skb, flags);
if (from_proto == htons(ETH_P_IPV6) &&
to_proto == htons(ETH_P_IP))
- return bpf_skb_proto_6_to_4(skb);
+ return bpf_skb_proto_6_to_4(skb, flags);
return -ENOTSUPP;
}
{
int ret;
- if (unlikely(flags))
+ if (unlikely(flags & ~(BPF_F_ADJ_ROOM_FIXED_GSO)))
return -EINVAL;
/* General idea is that this helper does the basic groundwork
* that. For offloads, we mark packet as dodgy, so that headers
* need to be verified first.
*/
- ret = bpf_skb_proto_xlat(skb, proto);
+ ret = bpf_skb_proto_xlat(skb, proto, flags);
bpf_compute_data_pointers(skb);
return ret;
}
__skb_push(skb, head_room);
memset(skb->data, 0, head_room);
skb_reset_mac_header(skb);
+ skb_reset_mac_len(skb);
}
return ret;
}
EXPORT_SYMBOL_GPL(xdp_do_flush);
+void bpf_clear_redirect_map(struct bpf_map *map)
+{
+ struct bpf_redirect_info *ri;
+ int cpu;
+
+ for_each_possible_cpu(cpu) {
+ ri = per_cpu_ptr(&bpf_redirect_info, cpu);
+ /* Avoid polluting remote cacheline due to writes if
+ * not needed. Once we pass this test, we need the
+ * cmpxchg() to make sure it hasn't been changed in
+ * the meantime by remote CPU.
+ */
+ if (unlikely(READ_ONCE(ri->map) == map))
+ cmpxchg(&ri->map, map, NULL);
+ }
+}
+
int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp,
struct bpf_prog *xdp_prog)
{
enum bpf_map_type map_type = ri->map_type;
void *fwd = ri->tgt_value;
u32 map_id = ri->map_id;
+ struct bpf_map *map;
int err;
ri->map_id = 0; /* Valid map id idr range: [1,INT_MAX[ */
case BPF_MAP_TYPE_DEVMAP:
fallthrough;
case BPF_MAP_TYPE_DEVMAP_HASH:
- err = dev_map_enqueue(fwd, xdp, dev);
+ map = READ_ONCE(ri->map);
+ if (unlikely(map)) {
+ WRITE_ONCE(ri->map, NULL);
+ err = dev_map_enqueue_multi(xdp, dev, map,
+ ri->flags & BPF_F_EXCLUDE_INGRESS);
+ } else {
+ err = dev_map_enqueue(fwd, xdp, dev);
+ }
break;
case BPF_MAP_TYPE_CPUMAP:
err = cpu_map_enqueue(fwd, xdp, dev);
enum bpf_map_type map_type, u32 map_id)
{
struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info);
+ struct bpf_map *map;
int err;
switch (map_type) {
case BPF_MAP_TYPE_DEVMAP:
fallthrough;
case BPF_MAP_TYPE_DEVMAP_HASH:
- err = dev_map_generic_redirect(fwd, skb, xdp_prog);
+ map = READ_ONCE(ri->map);
+ if (unlikely(map)) {
+ WRITE_ONCE(ri->map, NULL);
+ err = dev_map_redirect_multi(dev, skb, xdp_prog, map,
+ ri->flags & BPF_F_EXCLUDE_INGRESS);
+ } else {
+ err = dev_map_generic_redirect(fwd, skb, xdp_prog);
+ }
if (unlikely(err))
goto err;
break;
static void bpf_init_reuseport_kern(struct sk_reuseport_kern *reuse_kern,
struct sock_reuseport *reuse,
struct sock *sk, struct sk_buff *skb,
+ struct sock *migrating_sk,
u32 hash)
{
reuse_kern->skb = skb;
reuse_kern->sk = sk;
reuse_kern->selected_sk = NULL;
+ reuse_kern->migrating_sk = migrating_sk;
reuse_kern->data_end = skb->data + skb_headlen(skb);
reuse_kern->hash = hash;
reuse_kern->reuseport_id = reuse->reuseport_id;
struct sock *bpf_run_sk_reuseport(struct sock_reuseport *reuse, struct sock *sk,
struct bpf_prog *prog, struct sk_buff *skb,
+ struct sock *migrating_sk,
u32 hash)
{
struct sk_reuseport_kern reuse_kern;
enum sk_action action;
- bpf_init_reuseport_kern(&reuse_kern, reuse, sk, skb, hash);
+ bpf_init_reuseport_kern(&reuse_kern, reuse, sk, skb, migrating_sk, hash);
action = BPF_PROG_RUN(prog, &reuse_kern);
if (action == SK_PASS)
return &sk_reuseport_load_bytes_proto;
case BPF_FUNC_skb_load_bytes_relative:
return &sk_reuseport_load_bytes_relative_proto;
+ case BPF_FUNC_get_socket_cookie:
+ return &bpf_get_socket_ptr_cookie_proto;
default:
return bpf_base_func_proto(func_id);
}
case offsetof(struct sk_reuseport_md, hash):
return size == size_default;
+ case offsetof(struct sk_reuseport_md, sk):
+ info->reg_type = PTR_TO_SOCKET;
+ return size == sizeof(__u64);
+
+ case offsetof(struct sk_reuseport_md, migrating_sk):
+ info->reg_type = PTR_TO_SOCK_COMMON_OR_NULL;
+ return size == sizeof(__u64);
+
/* Fields that allow narrowing */
case bpf_ctx_range(struct sk_reuseport_md, eth_protocol):
if (size < sizeof_field(struct sk_buff, protocol))
case offsetof(struct sk_reuseport_md, bind_inany):
SK_REUSEPORT_LOAD_FIELD(bind_inany);
break;
+
+ case offsetof(struct sk_reuseport_md, sk):
+ SK_REUSEPORT_LOAD_FIELD(sk);
+ break;
+
+ case offsetof(struct sk_reuseport_md, migrating_sk):
+ SK_REUSEPORT_LOAD_FIELD(migrating_sk);
+ break;
}
return insn - insn_buf;
int offset = 0;
ops = skb->dev->dsa_ptr->tag_ops;
- /* Tail taggers don't break flow dissection */
- if (!ops->tail_tag) {
+ /* Only DSA header taggers break flow dissection */
+ if (ops->needed_headroom) {
if (ops->flow_dissect)
ops->flow_dissect(skb, &proto, &offset);
else
write_lock(&n->lock);
if ((n->nud_state == NUD_FAILED) ||
+ (n->nud_state == NUD_NOARP) ||
(tbl->is_multicast &&
tbl->is_multicast(n->primary_key)) ||
time_after(tref, n->updated))
EXPORT_TRACEPOINT_SYMBOL_GPL(napi_poll);
EXPORT_TRACEPOINT_SYMBOL_GPL(tcp_send_reset);
+EXPORT_TRACEPOINT_SYMBOL_GPL(tcp_bad_csum);
}
EXPORT_SYMBOL_GPL(__put_net);
+/**
+ * get_net_ns - increment the refcount of the network namespace
+ * @ns: common namespace (net)
+ *
+ * Returns the net's common namespace.
+ */
+struct ns_common *get_net_ns(struct ns_common *ns)
+{
+ return &get_net(container_of(ns, struct net, ns))->ns;
+}
+EXPORT_SYMBOL_GPL(get_net_ns);
+
struct net *get_net_ns_by_fd(int fd)
{
struct file *file;
fput(file);
return net;
}
-
-#else
-struct net *get_net_ns_by_fd(int fd)
-{
- return ERR_PTR(-EINVAL);
-}
-#endif
EXPORT_SYMBOL_GPL(get_net_ns_by_fd);
+#endif
struct net *get_net_ns_by_pid(pid_t pid)
{
#include <net/ip6_checksum.h>
#include <asm/unaligned.h>
#include <trace/events/napi.h>
+#include <linux/kconfig.h>
/*
* We maintain a small pool of fully-sized skbs, to make sure the
static atomic_t ip_ident;
struct ipv6hdr *ip6h;
- WARN_ON_ONCE(!irqs_disabled());
+ if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+ WARN_ON_ONCE(!irqs_disabled());
udp_len = len + sizeof(*udph);
if (np->ipv6)
#include <linux/dma-mapping.h>
#include <linux/page-flags.h>
#include <linux/mm.h> /* for __put_page() */
+#include <linux/poison.h>
#include <trace/events/page_pool.h>
struct page *page,
unsigned int dma_sync_size)
{
+ dma_addr_t dma_addr = page_pool_get_dma_addr(page);
+
dma_sync_size = min(dma_sync_size, pool->p.max_len);
- dma_sync_single_range_for_device(pool->p.dev, page->dma_addr,
+ dma_sync_single_range_for_device(pool->p.dev, dma_addr,
pool->p.offset, dma_sync_size,
pool->p.dma_dir);
}
if (dma_mapping_error(pool->p.dev, dma))
return false;
- page->dma_addr = dma;
+ page_pool_set_dma_addr(page, dma);
if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV)
page_pool_dma_sync_for_device(pool, page, pool->p.max_len);
return NULL;
}
+ page->pp_magic |= PP_SIGNATURE;
+
/* Track how many pages are held 'in-flight' */
pool->pages_state_hold_cnt++;
trace_page_pool_state_hold(pool, page, pool->pages_state_hold_cnt);
put_page(page);
continue;
}
+ page->pp_magic |= PP_SIGNATURE;
pool->alloc.cache[pool->alloc.count++] = page;
/* Track how many pages are held 'in-flight' */
pool->pages_state_hold_cnt++;
*/
goto skip_dma_unmap;
- dma = page->dma_addr;
+ dma = page_pool_get_dma_addr(page);
- /* When page is unmapped, it cannot be returned our pool */
+ /* When page is unmapped, it cannot be returned to our pool */
dma_unmap_page_attrs(pool->p.dev, dma,
PAGE_SIZE << pool->p.order, pool->p.dma_dir,
DMA_ATTR_SKIP_CPU_SYNC);
- page->dma_addr = 0;
+ page_pool_set_dma_addr(page, 0);
skip_dma_unmap:
+ page->pp_magic = 0;
+
/* This may be the last page returned, releasing the pool, so
* it is not safe to reference pool afterwards.
*/
}
}
EXPORT_SYMBOL(page_pool_update_nid);
+
+bool page_pool_return_skb_page(struct page *page)
+{
+ struct page_pool *pp;
+
+ page = compound_head(page);
+ if (unlikely(page->pp_magic != PP_SIGNATURE))
+ return false;
+
+ pp = page->pp;
+
+ /* Driver set this to memory recycling info. Reset it on recycle.
+ * This will *not* work for NIC using a split-page memory model.
+ * The page will be returned to the pool here regardless of the
+ * 'flipped' fragment being in use or not.
+ */
+ page->pp = NULL;
+ page_pool_put_full_page(pp, page, false);
+
+ return true;
+}
+EXPORT_SYMBOL(page_pool_return_skb_page);
static int pktgen_device_event(struct notifier_block *, unsigned long, void *);
static void pktgen_run_all_threads(struct pktgen_net *pn);
static void pktgen_reset_all_threads(struct pktgen_net *pn);
-static void pktgen_stop_all_threads_ifs(struct pktgen_net *pn);
+static void pktgen_stop_all_threads(struct pktgen_net *pn);
static void pktgen_stop(struct pktgen_thread *t);
static void pktgen_clear_counters(struct pktgen_dev *pkt_dev);
data[count - 1] = 0; /* Strip trailing '\n' and terminate string */
if (!strcmp(data, "stop"))
- pktgen_stop_all_threads_ifs(pn);
-
+ pktgen_stop_all_threads(pn);
else if (!strcmp(data, "start"))
pktgen_run_all_threads(pn);
-
else if (!strcmp(data, "reset"))
pktgen_reset_all_threads(pn);
-
else
return -EINVAL;
t->control &= ~(T_STOP);
}
-static void pktgen_stop_all_threads_ifs(struct pktgen_net *pn)
+static void pktgen_handle_all_threads(struct pktgen_net *pn, u32 flags)
{
struct pktgen_thread *t;
- func_enter();
-
mutex_lock(&pktgen_thread_lock);
list_for_each_entry(t, &pn->pktgen_threads, th_list)
- t->control |= T_STOP;
+ t->control |= (flags);
mutex_unlock(&pktgen_thread_lock);
}
+static void pktgen_stop_all_threads(struct pktgen_net *pn)
+{
+ func_enter();
+
+ pktgen_handle_all_threads(pn, T_STOP);
+}
+
static int thread_is_running(const struct pktgen_thread *t)
{
const struct pktgen_dev *pkt_dev;
static void pktgen_run_all_threads(struct pktgen_net *pn)
{
- struct pktgen_thread *t;
-
func_enter();
- mutex_lock(&pktgen_thread_lock);
-
- list_for_each_entry(t, &pn->pktgen_threads, th_list)
- t->control |= (T_RUN);
-
- mutex_unlock(&pktgen_thread_lock);
+ pktgen_handle_all_threads(pn, T_RUN);
/* Propagate thread->control */
schedule_timeout_interruptible(msecs_to_jiffies(125));
static void pktgen_reset_all_threads(struct pktgen_net *pn)
{
- struct pktgen_thread *t;
-
func_enter();
- mutex_lock(&pktgen_thread_lock);
-
- list_for_each_entry(t, &pn->pktgen_threads, th_list)
- t->control |= (T_REMDEVALL);
-
- mutex_unlock(&pktgen_thread_lock);
+ pktgen_handle_all_threads(pn, T_REMDEVALL);
/* Propagate thread->control */
schedule_timeout_interruptible(msecs_to_jiffies(125));
* Authors: Alexey Kuznetsov, <kuznet@ms2.inr.ac.ru>
*
* Fixes:
- * Vitaly E. Lavrov RTA_OK arithmetics was wrong.
+ * Vitaly E. Lavrov RTA_OK arithmetic was wrong.
*/
#include <linux/bitops.h>
* @msgtype: rtnetlink message type
* @doit: Function pointer called for each request message
* @dumpit: Function pointer called for each dump request (NLM_F_DUMP) message
- * @flags: rtnl_link_flags to modifiy behaviour of doit/dumpit functions
+ * @flags: rtnl_link_flags to modify behaviour of doit/dumpit functions
*
* Like rtnl_register, but for use by removable modules.
*/
* @msgtype: rtnetlink message type
* @doit: Function pointer called for each request message
* @dumpit: Function pointer called for each dump request (NLM_F_DUMP) message
- * @flags: rtnl_link_flags to modifiy behaviour of doit/dumpit functions
+ * @flags: rtnl_link_flags to modify behaviour of doit/dumpit functions
*
* Registers the specified function pointers (at least one of them has
* to be non-NULL) to be called whenever a request message for the
if (rtnl_link_ops_get(ops->kind))
return -EEXIST;
- /* The check for setup is here because if ops
+ /* The check for alloc/setup is here because if ops
* does not have that filled up, it is not possible
* to use the ops for creating device. So do not
* fill up dellink as well. That disables rtnl_dellink.
*/
- if (ops->setup && !ops->dellink)
+ if ((ops->alloc || ops->setup) && !ops->dellink)
ops->dellink = unregister_netdevice_queue;
list_add_tail(&ops->list, &link_ops);
if (rtnl_fill_prop_list(skb, dev))
goto nla_put_failure;
+ if (dev->dev.parent &&
+ nla_put_string(skb, IFLA_PARENT_DEV_NAME,
+ dev_name(dev->dev.parent)))
+ goto nla_put_failure;
+
+ if (dev->dev.parent && dev->dev.parent->bus &&
+ nla_put_string(skb, IFLA_PARENT_DEV_BUS_NAME,
+ dev->dev.parent->bus->name))
+ goto nla_put_failure;
+
nlmsg_end(skb, nlh);
return 0;
[IFLA_PERM_ADDRESS] = { .type = NLA_REJECT },
[IFLA_PROTO_DOWN_REASON] = { .type = NLA_NESTED },
[IFLA_NEW_IFINDEX] = NLA_POLICY_MIN(NLA_S32, 1),
+ [IFLA_PARENT_DEV_NAME] = { .type = NLA_NUL_STRING },
};
static const struct nla_policy ifla_info_policy[IFLA_INFO_MAX+1] = {
if (nl_proto_down) {
proto_down = nla_get_u8(nl_proto_down);
- /* Dont turn off protodown if there are active reasons */
+ /* Don't turn off protodown if there are active reasons */
if (!proto_down && dev->proto_down_reason) {
NL_SET_ERR_MSG(extack, "Cannot clear protodown, active reasons");
return -EBUSY;
return ERR_PTR(-EINVAL);
}
- dev = alloc_netdev_mqs(ops->priv_size, ifname, name_assign_type,
- ops->setup, num_tx_queues, num_rx_queues);
+ if (ops->alloc) {
+ dev = ops->alloc(tb, ifname, name_assign_type,
+ num_tx_queues, num_rx_queues);
+ if (IS_ERR(dev))
+ return dev;
+ } else {
+ dev = alloc_netdev_mqs(ops->priv_size, ifname,
+ name_assign_type, ops->setup,
+ num_tx_queues, num_rx_queues);
+ }
+
if (!dev)
return ERR_PTR(-ENOMEM);
return -EOPNOTSUPP;
}
- if (!ops->setup)
+ if (!ops->alloc && !ops->setup)
return -EOPNOTSUPP;
if (!ifname[0]) {
if (err < 0)
goto errout;
+ /* Notification info is only filled for bridge ports, not the bridge
+ * device itself. Therefore, a zero notification length is valid and
+ * should not result in an error.
+ */
if (!skb->len)
goto errout;
#include <net/xfrm.h>
#include <net/mpls.h>
#include <net/mptcp.h>
+#include <net/page_pool.h>
#include <linux/uaccess.h>
#include <trace/events/skb.h>
{
unsigned char *head = skb->head;
- if (skb->head_frag)
+ if (skb->head_frag) {
+ if (skb_pp_recycle(skb, head))
+ return;
skb_free_frag(head);
- else
+ } else {
kfree(head);
+ }
}
static void skb_release_data(struct sk_buff *skb)
skb_zcopy_clear(skb, true);
for (i = 0; i < shinfo->nr_frags; i++)
- __skb_frag_unref(&shinfo->frags[i]);
+ __skb_frag_unref(&shinfo->frags[i], skb->pp_recycle);
if (shinfo->frag_list)
kfree_skb_list(shinfo->frag_list);
n->nohdr = 0;
n->peeked = 0;
C(pfmemalloc);
+ C(pp_recycle);
n->destructor = NULL;
C(tail);
C(end);
struct sock *sk = skb->sk;
struct sk_buff_head *q;
unsigned long flags;
+ bool is_zerocopy;
u32 lo, hi;
u16 len;
len = uarg->len;
lo = uarg->id;
hi = uarg->id + len - 1;
+ is_zerocopy = uarg->zerocopy;
serr = SKB_EXT_ERR(skb);
memset(serr, 0, sizeof(*serr));
serr->ee.ee_origin = SO_EE_ORIGIN_ZEROCOPY;
serr->ee.ee_data = hi;
serr->ee.ee_info = lo;
- if (!uarg->zerocopy)
+ if (!is_zerocopy)
serr->ee.ee_code |= SO_EE_CODE_ZEROCOPY_COPIED;
q = &sk->sk_error_queue;
fragto = &skb_shinfo(tgt)->frags[merge];
skb_frag_size_add(fragto, skb_frag_size(fragfrom));
- __skb_frag_unref(fragfrom);
+ __skb_frag_unref(fragfrom, skb->pp_recycle);
}
/* Reposition in the original skb */
if (skb_cloned(to))
return false;
+ /* The page pool signature of struct page will eventually figure out
+ * which pages can be recycled or not but for now let's prohibit slab
+ * allocated and page_pool allocated SKBs from being coalesced.
+ */
+ if (to->pp_recycle != from->pp_recycle)
+ return false;
+
if (len <= skb_tailroom(to)) {
if (len)
BUG_ON(skb_copy_bits(from, 0, skb_put(to, len), len));
}
EXPORT_SYMBOL_GPL(sk_msg_memcopy_from_iter);
-int sk_msg_wait_data(struct sock *sk, struct sk_psock *psock, int flags,
- long timeo, int *err)
+int sk_msg_wait_data(struct sock *sk, struct sk_psock *psock, long timeo)
{
DEFINE_WAIT_FUNC(wait, woken_wake_function);
int ret = 0;
}
EXPORT_SYMBOL(sock_enable_timestamps);
+void sock_set_timestamp(struct sock *sk, int optname, bool valbool)
+{
+ switch (optname) {
+ case SO_TIMESTAMP_OLD:
+ __sock_set_timestamps(sk, valbool, false, false);
+ break;
+ case SO_TIMESTAMP_NEW:
+ __sock_set_timestamps(sk, valbool, true, false);
+ break;
+ case SO_TIMESTAMPNS_OLD:
+ __sock_set_timestamps(sk, valbool, false, true);
+ break;
+ case SO_TIMESTAMPNS_NEW:
+ __sock_set_timestamps(sk, valbool, true, true);
+ break;
+ }
+}
+
+int sock_set_timestamping(struct sock *sk, int optname, int val)
+{
+ if (val & ~SOF_TIMESTAMPING_MASK)
+ return -EINVAL;
+
+ if (val & SOF_TIMESTAMPING_OPT_ID &&
+ !(sk->sk_tsflags & SOF_TIMESTAMPING_OPT_ID)) {
+ if (sk->sk_protocol == IPPROTO_TCP &&
+ sk->sk_type == SOCK_STREAM) {
+ if ((1 << sk->sk_state) &
+ (TCPF_CLOSE | TCPF_LISTEN))
+ return -EINVAL;
+ sk->sk_tskey = tcp_sk(sk)->snd_una;
+ } else {
+ sk->sk_tskey = 0;
+ }
+ }
+
+ if (val & SOF_TIMESTAMPING_OPT_STATS &&
+ !(val & SOF_TIMESTAMPING_OPT_TSONLY))
+ return -EINVAL;
+
+ sk->sk_tsflags = val;
+ sock_valbool_flag(sk, SOCK_TSTAMP_NEW, optname == SO_TIMESTAMPING_NEW);
+
+ if (val & SOF_TIMESTAMPING_RX_SOFTWARE)
+ sock_enable_timestamp(sk,
+ SOCK_TIMESTAMPING_RX_SOFTWARE);
+ else
+ sock_disable_timestamp(sk,
+ (1UL << SOCK_TIMESTAMPING_RX_SOFTWARE));
+ return 0;
+}
+
void sock_set_keepalive(struct sock *sk)
{
lock_sock(sk);
}
EXPORT_SYMBOL(sock_set_rcvbuf);
+static void __sock_set_mark(struct sock *sk, u32 val)
+{
+ if (val != sk->sk_mark) {
+ sk->sk_mark = val;
+ sk_dst_reset(sk);
+ }
+}
+
void sock_set_mark(struct sock *sk, u32 val)
{
lock_sock(sk);
- sk->sk_mark = val;
+ __sock_set_mark(sk, val);
release_sock(sk);
}
EXPORT_SYMBOL(sock_set_mark);
break;
case SO_TIMESTAMP_OLD:
- __sock_set_timestamps(sk, valbool, false, false);
- break;
case SO_TIMESTAMP_NEW:
- __sock_set_timestamps(sk, valbool, true, false);
- break;
case SO_TIMESTAMPNS_OLD:
- __sock_set_timestamps(sk, valbool, false, true);
- break;
case SO_TIMESTAMPNS_NEW:
- __sock_set_timestamps(sk, valbool, true, true);
+ sock_set_timestamp(sk, valbool, optname);
break;
+
case SO_TIMESTAMPING_NEW:
case SO_TIMESTAMPING_OLD:
- if (val & ~SOF_TIMESTAMPING_MASK) {
- ret = -EINVAL;
- break;
- }
-
- if (val & SOF_TIMESTAMPING_OPT_ID &&
- !(sk->sk_tsflags & SOF_TIMESTAMPING_OPT_ID)) {
- if (sk->sk_protocol == IPPROTO_TCP &&
- sk->sk_type == SOCK_STREAM) {
- if ((1 << sk->sk_state) &
- (TCPF_CLOSE | TCPF_LISTEN)) {
- ret = -EINVAL;
- break;
- }
- sk->sk_tskey = tcp_sk(sk)->snd_una;
- } else {
- sk->sk_tskey = 0;
- }
- }
-
- if (val & SOF_TIMESTAMPING_OPT_STATS &&
- !(val & SOF_TIMESTAMPING_OPT_TSONLY)) {
- ret = -EINVAL;
- break;
- }
-
- sk->sk_tsflags = val;
- sock_valbool_flag(sk, SOCK_TSTAMP_NEW, optname == SO_TIMESTAMPING_NEW);
-
- if (val & SOF_TIMESTAMPING_RX_SOFTWARE)
- sock_enable_timestamp(sk,
- SOCK_TIMESTAMPING_RX_SOFTWARE);
- else
- sock_disable_timestamp(sk,
- (1UL << SOCK_TIMESTAMPING_RX_SOFTWARE));
+ ret = sock_set_timestamping(sk, optname, val);
break;
case SO_RCVLOWAT:
case SO_MARK:
if (!ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN)) {
ret = -EPERM;
- } else if (val != sk->sk_mark) {
- sk->sk_mark = val;
- sk_dst_reset(sk);
+ break;
}
+
+ __sock_set_mark(sk, val);
break;
case SO_RXQ_OVFL:
v.val = sk->sk_bound_dev_if;
break;
+ case SO_NETNS_COOKIE:
+ lv = sizeof(u64);
+ if (len != lv)
+ return -EINVAL;
+ v.val64 = sock_net(sk)->net_cookie;
+ break;
+
default:
/* We implement the SO_SNDLOWAT etc to not be settable
* (1003.1g 7).
if (skb_is_tcp_pure_ack(skb))
return;
- if (can_skb_orphan_partial(skb))
- skb_set_owner_sk_safe(skb, skb->sk);
- else
- skb_orphan(skb);
+ if (can_skb_orphan_partial(skb) && skb_set_owner_sk_safe(skb, skb->sk))
+ return;
+
+ skb_orphan(skb);
}
EXPORT_SYMBOL(skb_orphan_partial);
* selecting the socket index from the array of available sockets.
*/
+#include <net/ip.h>
#include <net/sock_reuseport.h>
#include <linux/bpf.h>
#include <linux/idr.h>
DEFINE_SPINLOCK(reuseport_lock);
static DEFINE_IDA(reuseport_ida);
+static int reuseport_resurrect(struct sock *sk, struct sock_reuseport *old_reuse,
+ struct sock_reuseport *reuse, bool bind_inany);
+
+static int reuseport_sock_index(struct sock *sk,
+ const struct sock_reuseport *reuse,
+ bool closed)
+{
+ int left, right;
+
+ if (!closed) {
+ left = 0;
+ right = reuse->num_socks;
+ } else {
+ left = reuse->max_socks - reuse->num_closed_socks;
+ right = reuse->max_socks;
+ }
+
+ for (; left < right; left++)
+ if (reuse->socks[left] == sk)
+ return left;
+ return -1;
+}
+
+static void __reuseport_add_sock(struct sock *sk,
+ struct sock_reuseport *reuse)
+{
+ reuse->socks[reuse->num_socks] = sk;
+ /* paired with smp_rmb() in reuseport_(select|migrate)_sock() */
+ smp_wmb();
+ reuse->num_socks++;
+}
+
+static bool __reuseport_detach_sock(struct sock *sk,
+ struct sock_reuseport *reuse)
+{
+ int i = reuseport_sock_index(sk, reuse, false);
+
+ if (i == -1)
+ return false;
+
+ reuse->socks[i] = reuse->socks[reuse->num_socks - 1];
+ reuse->num_socks--;
+
+ return true;
+}
+
+static void __reuseport_add_closed_sock(struct sock *sk,
+ struct sock_reuseport *reuse)
+{
+ reuse->socks[reuse->max_socks - reuse->num_closed_socks - 1] = sk;
+ /* paired with READ_ONCE() in inet_csk_bind_conflict() */
+ WRITE_ONCE(reuse->num_closed_socks, reuse->num_closed_socks + 1);
+}
+
+static bool __reuseport_detach_closed_sock(struct sock *sk,
+ struct sock_reuseport *reuse)
+{
+ int i = reuseport_sock_index(sk, reuse, true);
+
+ if (i == -1)
+ return false;
+
+ reuse->socks[i] = reuse->socks[reuse->max_socks - reuse->num_closed_socks];
+ /* paired with READ_ONCE() in inet_csk_bind_conflict() */
+ WRITE_ONCE(reuse->num_closed_socks, reuse->num_closed_socks - 1);
+
+ return true;
+}
static struct sock_reuseport *__reuseport_alloc(unsigned int max_socks)
{
reuse = rcu_dereference_protected(sk->sk_reuseport_cb,
lockdep_is_held(&reuseport_lock));
if (reuse) {
+ if (reuse->num_closed_socks) {
+ /* sk was shutdown()ed before */
+ ret = reuseport_resurrect(sk, reuse, NULL, bind_inany);
+ goto out;
+ }
+
/* Only set reuse->bind_inany if the bind_inany is true.
* Otherwise, it will overwrite the reuse->bind_inany
* which was set by the bind/hash path.
}
reuse->reuseport_id = id;
+ reuse->bind_inany = bind_inany;
reuse->socks[0] = sk;
reuse->num_socks = 1;
- reuse->bind_inany = bind_inany;
rcu_assign_pointer(sk->sk_reuseport_cb, reuse);
out:
u32 more_socks_size, i;
more_socks_size = reuse->max_socks * 2U;
- if (more_socks_size > U16_MAX)
+ if (more_socks_size > U16_MAX) {
+ if (reuse->num_closed_socks) {
+ /* Make room by removing a closed sk.
+ * The child has already been migrated.
+ * Only reqsk left at this point.
+ */
+ struct sock *sk;
+
+ sk = reuse->socks[reuse->max_socks - reuse->num_closed_socks];
+ RCU_INIT_POINTER(sk->sk_reuseport_cb, NULL);
+ __reuseport_detach_closed_sock(sk, reuse);
+
+ return reuse;
+ }
+
return NULL;
+ }
more_reuse = __reuseport_alloc(more_socks_size);
if (!more_reuse)
return NULL;
more_reuse->num_socks = reuse->num_socks;
+ more_reuse->num_closed_socks = reuse->num_closed_socks;
more_reuse->prog = reuse->prog;
more_reuse->reuseport_id = reuse->reuseport_id;
more_reuse->bind_inany = reuse->bind_inany;
memcpy(more_reuse->socks, reuse->socks,
reuse->num_socks * sizeof(struct sock *));
+ memcpy(more_reuse->socks +
+ (more_reuse->max_socks - more_reuse->num_closed_socks),
+ reuse->socks + (reuse->max_socks - reuse->num_closed_socks),
+ reuse->num_closed_socks * sizeof(struct sock *));
more_reuse->synq_overflow_ts = READ_ONCE(reuse->synq_overflow_ts);
- for (i = 0; i < reuse->num_socks; ++i)
+ for (i = 0; i < reuse->max_socks; ++i)
rcu_assign_pointer(reuse->socks[i]->sk_reuseport_cb,
more_reuse);
reuse = rcu_dereference_protected(sk2->sk_reuseport_cb,
lockdep_is_held(&reuseport_lock));
old_reuse = rcu_dereference_protected(sk->sk_reuseport_cb,
- lockdep_is_held(&reuseport_lock));
+ lockdep_is_held(&reuseport_lock));
+ if (old_reuse && old_reuse->num_closed_socks) {
+ /* sk was shutdown()ed before */
+ int err = reuseport_resurrect(sk, old_reuse, reuse, reuse->bind_inany);
+
+ spin_unlock_bh(&reuseport_lock);
+ return err;
+ }
+
if (old_reuse && old_reuse->num_socks != 1) {
spin_unlock_bh(&reuseport_lock);
return -EBUSY;
}
- if (reuse->num_socks == reuse->max_socks) {
+ if (reuse->num_socks + reuse->num_closed_socks == reuse->max_socks) {
reuse = reuseport_grow(reuse);
if (!reuse) {
spin_unlock_bh(&reuseport_lock);
}
}
- reuse->socks[reuse->num_socks] = sk;
- /* paired with smp_rmb() in reuseport_select_sock() */
- smp_wmb();
- reuse->num_socks++;
+ __reuseport_add_sock(sk, reuse);
rcu_assign_pointer(sk->sk_reuseport_cb, reuse);
spin_unlock_bh(&reuseport_lock);
}
EXPORT_SYMBOL(reuseport_add_sock);
+static int reuseport_resurrect(struct sock *sk, struct sock_reuseport *old_reuse,
+ struct sock_reuseport *reuse, bool bind_inany)
+{
+ if (old_reuse == reuse) {
+ /* If sk was in the same reuseport group, just pop sk out of
+ * the closed section and push sk into the listening section.
+ */
+ __reuseport_detach_closed_sock(sk, old_reuse);
+ __reuseport_add_sock(sk, old_reuse);
+ return 0;
+ }
+
+ if (!reuse) {
+ /* In bind()/listen() path, we cannot carry over the eBPF prog
+ * for the shutdown()ed socket. In setsockopt() path, we should
+ * not change the eBPF prog of listening sockets by attaching a
+ * prog to the shutdown()ed socket. Thus, we will allocate a new
+ * reuseport group and detach sk from the old group.
+ */
+ int id;
+
+ reuse = __reuseport_alloc(INIT_SOCKS);
+ if (!reuse)
+ return -ENOMEM;
+
+ id = ida_alloc(&reuseport_ida, GFP_ATOMIC);
+ if (id < 0) {
+ kfree(reuse);
+ return id;
+ }
+
+ reuse->reuseport_id = id;
+ reuse->bind_inany = bind_inany;
+ } else {
+ /* Move sk from the old group to the new one if
+ * - all the other listeners in the old group were close()d or
+ * shutdown()ed, and then sk2 has listen()ed on the same port
+ * OR
+ * - sk listen()ed without bind() (or with autobind), was
+ * shutdown()ed, and then listen()s on another port which
+ * sk2 listen()s on.
+ */
+ if (reuse->num_socks + reuse->num_closed_socks == reuse->max_socks) {
+ reuse = reuseport_grow(reuse);
+ if (!reuse)
+ return -ENOMEM;
+ }
+ }
+
+ __reuseport_detach_closed_sock(sk, old_reuse);
+ __reuseport_add_sock(sk, reuse);
+ rcu_assign_pointer(sk->sk_reuseport_cb, reuse);
+
+ if (old_reuse->num_socks + old_reuse->num_closed_socks == 0)
+ call_rcu(&old_reuse->rcu, reuseport_free_rcu);
+
+ return 0;
+}
+
void reuseport_detach_sock(struct sock *sk)
{
struct sock_reuseport *reuse;
- int i;
spin_lock_bh(&reuseport_lock);
reuse = rcu_dereference_protected(sk->sk_reuseport_cb,
lockdep_is_held(&reuseport_lock));
+ /* reuseport_grow() has detached a closed sk */
+ if (!reuse)
+ goto out;
+
/* Notify the bpf side. The sk may be added to a sockarray
* map. If so, sockarray logic will remove it from the map.
*
rcu_assign_pointer(sk->sk_reuseport_cb, NULL);
- for (i = 0; i < reuse->num_socks; i++) {
- if (reuse->socks[i] == sk) {
- reuse->socks[i] = reuse->socks[reuse->num_socks - 1];
- reuse->num_socks--;
- if (reuse->num_socks == 0)
- call_rcu(&reuse->rcu, reuseport_free_rcu);
- break;
- }
- }
+ if (!__reuseport_detach_closed_sock(sk, reuse))
+ __reuseport_detach_sock(sk, reuse);
+
+ if (reuse->num_socks + reuse->num_closed_socks == 0)
+ call_rcu(&reuse->rcu, reuseport_free_rcu);
+
+out:
spin_unlock_bh(&reuseport_lock);
}
EXPORT_SYMBOL(reuseport_detach_sock);
+void reuseport_stop_listen_sock(struct sock *sk)
+{
+ if (sk->sk_protocol == IPPROTO_TCP) {
+ struct sock_reuseport *reuse;
+ struct bpf_prog *prog;
+
+ spin_lock_bh(&reuseport_lock);
+
+ reuse = rcu_dereference_protected(sk->sk_reuseport_cb,
+ lockdep_is_held(&reuseport_lock));
+ prog = rcu_dereference_protected(reuse->prog,
+ lockdep_is_held(&reuseport_lock));
+
+ if (sock_net(sk)->ipv4.sysctl_tcp_migrate_req ||
+ (prog && prog->expected_attach_type == BPF_SK_REUSEPORT_SELECT_OR_MIGRATE)) {
+ /* Migration capable, move sk from the listening section
+ * to the closed section.
+ */
+ bpf_sk_reuseport_detach(sk);
+
+ __reuseport_detach_sock(sk, reuse);
+ __reuseport_add_closed_sock(sk, reuse);
+
+ spin_unlock_bh(&reuseport_lock);
+ return;
+ }
+
+ spin_unlock_bh(&reuseport_lock);
+ }
+
+ /* Not capable to do migration, detach immediately */
+ reuseport_detach_sock(sk);
+}
+EXPORT_SYMBOL(reuseport_stop_listen_sock);
+
static struct sock *run_bpf_filter(struct sock_reuseport *reuse, u16 socks,
struct bpf_prog *prog, struct sk_buff *skb,
int hdr_len)
return reuse->socks[index];
}
+static struct sock *reuseport_select_sock_by_hash(struct sock_reuseport *reuse,
+ u32 hash, u16 num_socks)
+{
+ int i, j;
+
+ i = j = reciprocal_scale(hash, num_socks);
+ while (reuse->socks[i]->sk_state == TCP_ESTABLISHED) {
+ i++;
+ if (i >= num_socks)
+ i = 0;
+ if (i == j)
+ return NULL;
+ }
+
+ return reuse->socks[i];
+}
+
/**
* reuseport_select_sock - Select a socket from an SO_REUSEPORT group.
* @sk: First socket in the group.
prog = rcu_dereference(reuse->prog);
socks = READ_ONCE(reuse->num_socks);
if (likely(socks)) {
- /* paired with smp_wmb() in reuseport_add_sock() */
+ /* paired with smp_wmb() in __reuseport_add_sock() */
smp_rmb();
if (!prog || !skb)
goto select_by_hash;
if (prog->type == BPF_PROG_TYPE_SK_REUSEPORT)
- sk2 = bpf_run_sk_reuseport(reuse, sk, prog, skb, hash);
+ sk2 = bpf_run_sk_reuseport(reuse, sk, prog, skb, NULL, hash);
else
sk2 = run_bpf_filter(reuse, socks, prog, skb, hdr_len);
select_by_hash:
/* no bpf or invalid bpf result: fall back to hash usage */
- if (!sk2) {
- int i, j;
-
- i = j = reciprocal_scale(hash, socks);
- while (reuse->socks[i]->sk_state == TCP_ESTABLISHED) {
- i++;
- if (i >= socks)
- i = 0;
- if (i == j)
- goto out;
- }
- sk2 = reuse->socks[i];
- }
+ if (!sk2)
+ sk2 = reuseport_select_sock_by_hash(reuse, hash, socks);
}
out:
}
EXPORT_SYMBOL(reuseport_select_sock);
+/**
+ * reuseport_migrate_sock - Select a socket from an SO_REUSEPORT group.
+ * @sk: close()ed or shutdown()ed socket in the group.
+ * @migrating_sk: ESTABLISHED/SYN_RECV full socket in the accept queue or
+ * NEW_SYN_RECV request socket during 3WHS.
+ * @skb: skb to run through BPF filter.
+ * Returns a socket (with sk_refcnt +1) that should accept the child socket
+ * (or NULL on error).
+ */
+struct sock *reuseport_migrate_sock(struct sock *sk,
+ struct sock *migrating_sk,
+ struct sk_buff *skb)
+{
+ struct sock_reuseport *reuse;
+ struct sock *nsk = NULL;
+ bool allocated = false;
+ struct bpf_prog *prog;
+ u16 socks;
+ u32 hash;
+
+ rcu_read_lock();
+
+ reuse = rcu_dereference(sk->sk_reuseport_cb);
+ if (!reuse)
+ goto out;
+
+ socks = READ_ONCE(reuse->num_socks);
+ if (unlikely(!socks))
+ goto failure;
+
+ /* paired with smp_wmb() in __reuseport_add_sock() */
+ smp_rmb();
+
+ hash = migrating_sk->sk_hash;
+ prog = rcu_dereference(reuse->prog);
+ if (!prog || prog->expected_attach_type != BPF_SK_REUSEPORT_SELECT_OR_MIGRATE) {
+ if (sock_net(sk)->ipv4.sysctl_tcp_migrate_req)
+ goto select_by_hash;
+ goto failure;
+ }
+
+ if (!skb) {
+ skb = alloc_skb(0, GFP_ATOMIC);
+ if (!skb)
+ goto failure;
+ allocated = true;
+ }
+
+ nsk = bpf_run_sk_reuseport(reuse, sk, prog, skb, migrating_sk, hash);
+
+ if (allocated)
+ kfree_skb(skb);
+
+select_by_hash:
+ if (!nsk)
+ nsk = reuseport_select_sock_by_hash(reuse, hash, socks);
+
+ if (IS_ERR_OR_NULL(nsk) || unlikely(!refcount_inc_not_zero(&nsk->sk_refcnt))) {
+ nsk = NULL;
+ goto failure;
+ }
+
+out:
+ rcu_read_unlock();
+ return nsk;
+
+failure:
+ __NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMIGRATEREQFAILURE);
+ goto out;
+}
+EXPORT_SYMBOL(reuseport_migrate_sock);
+
int reuseport_attach_prog(struct sock *sk, struct bpf_prog *prog)
{
struct sock_reuseport *reuse;
struct bpf_prog *old_prog;
- if (sk_unhashed(sk) && sk->sk_reuseport) {
- int err = reuseport_alloc(sk, false);
+ if (sk_unhashed(sk)) {
+ int err;
+ if (!sk->sk_reuseport)
+ return -EINVAL;
+
+ err = reuseport_alloc(sk, false);
if (err)
return err;
} else if (!rcu_access_pointer(sk->sk_reuseport_cb)) {
struct sock_reuseport *reuse;
struct bpf_prog *old_prog;
- if (!rcu_access_pointer(sk->sk_reuseport_cb))
- return sk->sk_reuseport ? -ENOENT : -EINVAL;
-
old_prog = NULL;
spin_lock_bh(&reuseport_lock);
reuse = rcu_dereference_protected(sk->sk_reuseport_cb,
lockdep_is_held(&reuseport_lock));
+
+ /* reuse must be checked after acquiring the reuseport_lock
+ * because reuseport_grow() can detach a closed sk.
+ */
+ if (!reuse) {
+ spin_unlock_bh(&reuseport_lock);
+ return sk->sk_reuseport ? -ENOENT : -EINVAL;
+ }
+
+ if (sk_unhashed(sk) && reuse->num_closed_socks) {
+ spin_unlock_bh(&reuseport_lock);
+ return -ENOENT;
+ }
+
old_prog = rcu_replace_pointer(reuse->prog, old_prog,
lockdep_is_held(&reuseport_lock));
spin_unlock_bh(&reuseport_lock);
return __xdp_build_skb_from_frame(xdpf, skb, dev);
}
EXPORT_SYMBOL_GPL(xdp_build_skb_from_frame);
+
+struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf)
+{
+ unsigned int headroom, totalsize;
+ struct xdp_frame *nxdpf;
+ struct page *page;
+ void *addr;
+
+ headroom = xdpf->headroom + sizeof(*xdpf);
+ totalsize = headroom + xdpf->len;
+
+ if (unlikely(totalsize > PAGE_SIZE))
+ return NULL;
+ page = dev_alloc_page();
+ if (!page)
+ return NULL;
+ addr = page_to_virt(page);
+
+ memcpy(addr, xdpf, totalsize);
+
+ nxdpf = addr;
+ nxdpf->data = addr + headroom;
+ nxdpf->frame_sz = PAGE_SIZE;
+ nxdpf->mem.type = MEM_TYPE_PAGE_ORDER0;
+ nxdpf->mem.id = 0;
+
+ return nxdpf;
+}
skb = dcbnl_newmsg(event, cmd, portid, seq, 0, &nlh);
if (!skb)
- return -ENOBUFS;
+ return -ENOMEM;
if (dcbx_ver == DCB_CAP_DCBX_VER_IEEE)
err = dcbnl_ieee_fill(skb, dev);
reply_skb = dcbnl_newmsg(fn->type, dcb->cmd, portid, nlh->nlmsg_seq,
nlh->nlmsg_flags, &reply_nlh);
if (!reply_skb)
- return -ENOBUFS;
+ return -ENOMEM;
ret = fn->cb(netdev, nlh, nlh->nlmsg_seq, tb, reply_skb);
if (ret < 0) {
static int __init dcbnl_init(void)
{
- INIT_LIST_HEAD(&dcb_app_list);
-
rtnl_register(PF_UNSPEC, RTM_GETDCB, dcb_doit, NULL, 0);
rtnl_register(PF_UNSPEC, RTM_SETDCB, dcb_doit, NULL, 0);
/**
* tfrc_invert_loss_event_rate - Compute p so that 10^6 corresponds to 100%
+ * @loss_event_rate: loss event rate to invert
* When @loss_event_rate is large, there is a chance that p is truncated to 0.
* To avoid re-entering slow-start in that case, we set p = TFRC_SMALLEST_P > 0.
*/
.handler = dccp_v4_rcv,
.err_handler = dccp_v4_err,
.no_policy = 1,
- .netns_ok = 1,
.icmp_strict_tag_validation = 1,
};
/*
* Read out ack data here, this applies equally
- * to data, other data, link serivce and both
+ * to data, other data, link service and both
* ack data and ack otherdata.
*/
dn_process_ack(sk, skb, other);
scp->nsp_srtt = 1;
/*
- * Add new rtt varience to smoothed varience
+ * Add new rtt variance to smoothed varience
*/
delta >>= 1;
rttvar += ((((delta>0)?(delta):(-delta)) - rttvar) >> 2);
static int dn_route_discard(struct net *net, struct sock *sk, struct sk_buff *skb)
{
/*
- * I know we drop the packet here, but thats considered success in
+ * I know we drop the packet here, but that's considered success in
* this case
*/
kfree_skb(skb);
* @ndev: device to register
*
* This is a devres variant of register_netdev() for which the unregister
- * function will be call automatically when the managing device is
+ * function will be called automatically when the managing device is
* detached. Note: the net_device used must also be resource managed by
* the same struct device.
*/
kref_put(&dst->refcount, dsa_tree_release);
}
-static bool dsa_port_is_dsa(struct dsa_port *port)
-{
- return port->type == DSA_PORT_TYPE_DSA;
-}
-
-static bool dsa_port_is_cpu(struct dsa_port *port)
-{
- return port->type == DSA_PORT_TYPE_CPU;
-}
-
-static bool dsa_port_is_user(struct dsa_port *dp)
-{
- return dp->type == DSA_PORT_TYPE_USER;
-}
-
static struct dsa_port *dsa_tree_find_port_by_node(struct dsa_switch_tree *dst,
struct device_node *dn)
{
if (!ds->dst)
return -ENOMEM;
+ if (dsa_switch_find(ds->dst->index, ds->index)) {
+ dev_err(ds->dev,
+ "A DSA switch with index %d already exists in tree %d\n",
+ ds->index, ds->dst->index);
+ return -EEXIST;
+ }
+
return 0;
}
/* DSA_NOTIFIER_MTU */
struct dsa_notifier_mtu_info {
- bool propagate_upstream;
+ bool targeted_match;
int sw_index;
int port;
int mtu;
bool dsa_schedule_work(struct work_struct *work);
const char *dsa_tag_protocol_to_str(const struct dsa_device_ops *ops);
+static inline int dsa_tag_protocol_overhead(const struct dsa_device_ops *ops)
+{
+ return ops->needed_headroom + ops->needed_tailroom;
+}
+
/* master.c */
int dsa_master_setup(struct net_device *dev, struct dsa_port *cpu_dp);
void dsa_master_teardown(struct net_device *dev);
bool dsa_port_skip_vlan_configuration(struct dsa_port *dp);
int dsa_port_ageing_time(struct dsa_port *dp, clock_t ageing_clock);
int dsa_port_mtu_change(struct dsa_port *dp, int new_mtu,
- bool propagate_upstream);
+ bool targeted_match);
int dsa_port_fdb_add(struct dsa_port *dp, const unsigned char *addr,
u16 vid);
int dsa_port_fdb_del(struct dsa_port *dp, const unsigned char *addr,
struct dsa_switch *ds = cpu_dp->ds;
int port = cpu_dp->index;
int len = ETH_GSTRING_LEN;
- int mcount = 0, count;
- unsigned int i;
+ int mcount = 0, count, i;
uint8_t pfx[4];
uint8_t *ndata;
*/
ds->ops->get_strings(ds, port, stringset, ndata);
count = ds->ops->get_sset_count(ds, port, stringset);
+ if (count < 0)
+ return;
for (i = 0; i < count; i++) {
memmove(ndata + (i * len + sizeof(pfx)),
ndata + i * len, len - sizeof(pfx));
int dsa_master_setup(struct net_device *dev, struct dsa_port *cpu_dp)
{
- int mtu = ETH_DATA_LEN + cpu_dp->tag_ops->overhead;
+ const struct dsa_device_ops *tag_ops = cpu_dp->tag_ops;
struct dsa_switch *ds = cpu_dp->ds;
struct device_link *consumer_link;
- int ret;
+ int mtu, ret;
+
+ mtu = ETH_DATA_LEN + dsa_tag_protocol_overhead(tag_ops);
/* The DSA master must use SET_NETDEV_DEV for this to work. */
consumer_link = device_link_add(ds->dev, dev->dev.parent,
}
int dsa_port_mtu_change(struct dsa_port *dp, int new_mtu,
- bool propagate_upstream)
+ bool targeted_match)
{
struct dsa_notifier_mtu_info info = {
.sw_index = dp->ds->index,
- .propagate_upstream = propagate_upstream,
+ .targeted_match = targeted_match,
.port = dp->index,
.mtu = new_mtu,
};
struct dsa_switch *ds = dp->ds;
if (sset == ETH_SS_STATS) {
- int count;
+ int count = 0;
- count = 4;
- if (ds->ops->get_sset_count)
- count += ds->ops->get_sset_count(ds, dp->index, sset);
+ if (ds->ops->get_sset_count) {
+ count = ds->ops->get_sset_count(ds, dp->index, sset);
+ if (count < 0)
+ return count;
+ }
- return count;
+ return count + 4;
} else if (sset == ETH_SS_TEST) {
return net_selftest_get_count();
}
struct dsa_port *dp = dsa_slave_to_port(dev);
struct dsa_slave_priv *p = netdev_priv(dev);
struct dsa_switch *ds = p->dp->ds;
+ struct dsa_port *dp_iter;
struct dsa_port *cpu_dp;
int port = p->dp->index;
int largest_mtu = 0;
int old_master_mtu;
int mtu_limit;
int cpu_mtu;
- int err, i;
+ int err;
if (!ds->ops->port_change_mtu)
return -EOPNOTSUPP;
- for (i = 0; i < ds->num_ports; i++) {
+ list_for_each_entry(dp_iter, &ds->dst->ports, list) {
int slave_mtu;
- if (!dsa_is_user_port(ds, i))
+ if (!dsa_port_is_user(dp_iter))
continue;
/* During probe, this function will be called for each slave
* device, while not all of them have been allocated. That's
* ok, it doesn't change what the maximum is, so ignore it.
*/
- if (!dsa_to_port(ds, i)->slave)
+ if (!dp_iter->slave)
continue;
/* Pretend that we already applied the setting, which we
* actually haven't (still haven't done all integrity checks)
*/
- if (i == port)
+ if (dp_iter == dp)
slave_mtu = new_mtu;
else
- slave_mtu = dsa_to_port(ds, i)->slave->mtu;
+ slave_mtu = dp_iter->slave->mtu;
if (largest_mtu < slave_mtu)
largest_mtu = slave_mtu;
mtu_limit = min_t(int, master->max_mtu, dev->max_mtu);
old_master_mtu = master->mtu;
- new_master_mtu = largest_mtu + cpu_dp->tag_ops->overhead;
+ new_master_mtu = largest_mtu + dsa_tag_protocol_overhead(cpu_dp->tag_ops);
if (new_master_mtu > mtu_limit)
return -ERANGE;
goto out_master_failed;
/* We only need to propagate the MTU of the CPU port to
- * upstream switches.
+ * upstream switches, so create a non-targeted notifier which
+ * updates all switches.
*/
- err = dsa_port_mtu_change(cpu_dp, cpu_mtu, true);
+ err = dsa_port_mtu_change(cpu_dp, cpu_mtu, false);
if (err)
goto out_cpu_failed;
}
- err = dsa_port_mtu_change(dp, new_mtu, false);
+ err = dsa_port_mtu_change(dp, new_mtu, true);
if (err)
goto out_port_failed;
out_port_failed:
if (new_master_mtu != old_master_mtu)
dsa_port_mtu_change(cpu_dp, old_master_mtu -
- cpu_dp->tag_ops->overhead,
- true);
+ dsa_tag_protocol_overhead(cpu_dp->tag_ops),
+ false);
out_cpu_failed:
if (new_master_mtu != old_master_mtu)
dev_set_mtu(master, old_master_mtu);
}
/* slave device setup *******************************************************/
-static int dsa_slave_phy_connect(struct net_device *slave_dev, int addr)
+static int dsa_slave_phy_connect(struct net_device *slave_dev, int addr,
+ u32 flags)
{
struct dsa_port *dp = dsa_slave_to_port(slave_dev);
struct dsa_switch *ds = dp->ds;
return -ENODEV;
}
+ slave_dev->phydev->dev_flags |= flags;
+
return phylink_connect_phy(dp->pl, slave_dev->phydev);
}
/* We could not connect to a designated PHY or SFP, so try to
* use the switch internal MDIO bus instead
*/
- ret = dsa_slave_phy_connect(slave_dev, dp->index);
+ ret = dsa_slave_phy_connect(slave_dev, dp->index, phy_flags);
if (ret) {
netdev_err(slave_dev,
"failed to connect to port %d: %d\n",
const struct dsa_port *cpu_dp = dp->cpu_dp;
struct net_device *master = cpu_dp->master;
- if (cpu_dp->tag_ops->tail_tag)
- slave->needed_tailroom = cpu_dp->tag_ops->overhead;
- else
- slave->needed_headroom = cpu_dp->tag_ops->overhead;
+ slave->needed_headroom = cpu_dp->tag_ops->needed_headroom;
+ slave->needed_tailroom = cpu_dp->tag_ops->needed_tailroom;
/* Try to save one extra realloc later in the TX path (in the master)
* by also inheriting the master's needed headroom and tailroom.
* The 8021q driver also does this.
static bool dsa_switch_mtu_match(struct dsa_switch *ds, int port,
struct dsa_notifier_mtu_info *info)
{
- if (ds->index == info->sw_index)
- return (port == info->port) || dsa_is_dsa_port(ds, port);
+ if (ds->index == info->sw_index && port == info->port)
+ return true;
- if (!info->propagate_upstream)
+ /* Do not propagate to other switches in the tree if the notifier was
+ * targeted for a single switch.
+ */
+ if (info->targeted_match)
return false;
if (dsa_is_dsa_port(ds, port) || dsa_is_cpu_port(ds, port))
return 0;
}
-static bool dsa_switch_mdb_match(struct dsa_switch *ds, int port,
- struct dsa_notifier_mdb_info *info)
-{
- if (ds->index == info->sw_index && port == info->port)
- return true;
-
- if (dsa_is_dsa_port(ds, port))
- return true;
-
- return false;
-}
-
static int dsa_switch_mdb_add(struct dsa_switch *ds,
struct dsa_notifier_mdb_info *info)
{
- int err = 0;
- int port;
+ int port = dsa_towards_port(ds, info->sw_index, info->port);
if (!ds->ops->port_mdb_add)
return -EOPNOTSUPP;
- for (port = 0; port < ds->num_ports; port++) {
- if (dsa_switch_mdb_match(ds, port, info)) {
- err = ds->ops->port_mdb_add(ds, port, info->mdb);
- if (err)
- break;
- }
- }
-
- return err;
+ return ds->ops->port_mdb_add(ds, port, info->mdb);
}
static int dsa_switch_mdb_del(struct dsa_switch *ds,
return 0;
}
-static bool dsa_switch_mrp_match(struct dsa_switch *ds, int port,
- struct dsa_notifier_mrp_info *info)
-{
- if (ds->index == info->sw_index && port == info->port)
- return true;
-
- if (dsa_is_dsa_port(ds, port))
- return true;
-
- return false;
-}
-
static int dsa_switch_mrp_add(struct dsa_switch *ds,
struct dsa_notifier_mrp_info *info)
{
- int err = 0;
- int port;
-
if (!ds->ops->port_mrp_add)
return -EOPNOTSUPP;
- for (port = 0; port < ds->num_ports; port++) {
- if (dsa_switch_mrp_match(ds, port, info)) {
- err = ds->ops->port_mrp_add(ds, port, info->mrp);
- if (err)
- break;
- }
- }
+ if (ds->index == info->sw_index)
+ return ds->ops->port_mrp_add(ds, info->port, info->mrp);
- return err;
+ return 0;
}
static int dsa_switch_mrp_del(struct dsa_switch *ds,
return 0;
}
-static bool
-dsa_switch_mrp_ring_role_match(struct dsa_switch *ds, int port,
- struct dsa_notifier_mrp_ring_role_info *info)
-{
- if (ds->index == info->sw_index && port == info->port)
- return true;
-
- if (dsa_is_dsa_port(ds, port))
- return true;
-
- return false;
-}
-
static int
dsa_switch_mrp_add_ring_role(struct dsa_switch *ds,
struct dsa_notifier_mrp_ring_role_info *info)
{
- int err = 0;
- int port;
-
if (!ds->ops->port_mrp_add)
return -EOPNOTSUPP;
- for (port = 0; port < ds->num_ports; port++) {
- if (dsa_switch_mrp_ring_role_match(ds, port, info)) {
- err = ds->ops->port_mrp_add_ring_role(ds, port,
- info->mrp);
- if (err)
- break;
- }
- }
+ if (ds->index == info->sw_index)
+ return ds->ops->port_mrp_add_ring_role(ds, info->port,
+ info->mrp);
- return err;
+ return 0;
}
static int
#define DSA_8021Q_SUBVLAN_HI_SHIFT 9
#define DSA_8021Q_SUBVLAN_HI_MASK GENMASK(9, 9)
#define DSA_8021Q_SUBVLAN_LO_SHIFT 4
-#define DSA_8021Q_SUBVLAN_LO_MASK GENMASK(4, 3)
+#define DSA_8021Q_SUBVLAN_LO_MASK GENMASK(5, 4)
#define DSA_8021Q_SUBVLAN_HI(x) (((x) & GENMASK(2, 2)) >> 2)
#define DSA_8021Q_SUBVLAN_LO(x) ((x) & GENMASK(1, 0))
#define DSA_8021Q_SUBVLAN(x) \
}
EXPORT_SYMBOL_GPL(dsa_8021q_xmit);
+void dsa_8021q_rcv(struct sk_buff *skb, int *source_port, int *switch_id,
+ int *subvlan)
+{
+ u16 vid, tci;
+
+ skb_push_rcsum(skb, ETH_HLEN);
+ if (skb_vlan_tag_present(skb)) {
+ tci = skb_vlan_tag_get(skb);
+ __vlan_hwaccel_clear_tag(skb);
+ } else {
+ __skb_vlan_pop(skb, &tci);
+ }
+ skb_pull_rcsum(skb, ETH_HLEN);
+
+ vid = tci & VLAN_VID_MASK;
+
+ *source_port = dsa_8021q_rx_source_port(vid);
+ *switch_id = dsa_8021q_rx_switch_id(vid);
+ *subvlan = dsa_8021q_rx_subvlan(vid);
+ skb->priority = (tci & VLAN_PRIO_MASK) >> VLAN_PRIO_SHIFT;
+}
+EXPORT_SYMBOL_GPL(dsa_8021q_rcv);
+
MODULE_LICENSE("GPL v2");
.proto = DSA_TAG_PROTO_AR9331,
.xmit = ar9331_tag_xmit,
.rcv = ar9331_tag_rcv,
- .overhead = AR9331_HDR_LEN,
+ .needed_headroom = AR9331_HDR_LEN,
};
MODULE_LICENSE("GPL v2");
.proto = DSA_TAG_PROTO_BRCM,
.xmit = brcm_tag_xmit,
.rcv = brcm_tag_rcv,
- .overhead = BRCM_TAG_LEN,
+ .needed_headroom = BRCM_TAG_LEN,
};
DSA_TAG_DRIVER(brcm_netdev_ops);
.proto = DSA_TAG_PROTO_BRCM_LEGACY,
.xmit = brcm_leg_tag_xmit,
.rcv = brcm_leg_tag_rcv,
- .overhead = BRCM_LEG_TAG_LEN,
+ .needed_headroom = BRCM_LEG_TAG_LEN,
};
DSA_TAG_DRIVER(brcm_legacy_netdev_ops);
.proto = DSA_TAG_PROTO_BRCM_PREPEND,
.xmit = brcm_tag_xmit_prepend,
.rcv = brcm_tag_rcv_prepend,
- .overhead = BRCM_TAG_LEN,
+ .needed_headroom = BRCM_TAG_LEN,
};
DSA_TAG_DRIVER(brcm_prepend_netdev_ops);
.proto = DSA_TAG_PROTO_DSA,
.xmit = dsa_xmit,
.rcv = dsa_rcv,
- .overhead = DSA_HLEN,
+ .needed_headroom = DSA_HLEN,
};
DSA_TAG_DRIVER(dsa_netdev_ops);
.proto = DSA_TAG_PROTO_EDSA,
.xmit = edsa_xmit,
.rcv = edsa_rcv,
- .overhead = EDSA_HLEN,
+ .needed_headroom = EDSA_HLEN,
};
DSA_TAG_DRIVER(edsa_netdev_ops);
.proto = DSA_TAG_PROTO_GSWIP,
.xmit = gswip_tag_xmit,
.rcv = gswip_tag_rcv,
- .overhead = GSWIP_RX_HEADER_LEN,
+ .needed_headroom = GSWIP_RX_HEADER_LEN,
};
MODULE_LICENSE("GPL");
.proto = DSA_TAG_PROTO_HELLCREEK,
.xmit = hellcreek_xmit,
.rcv = hellcreek_rcv,
- .overhead = HELLCREEK_TAG_LEN,
- .tail_tag = true,
+ .needed_tailroom = HELLCREEK_TAG_LEN,
};
MODULE_LICENSE("Dual MIT/GPL");
.proto = DSA_TAG_PROTO_KSZ8795,
.xmit = ksz8795_xmit,
.rcv = ksz8795_rcv,
- .overhead = KSZ_INGRESS_TAG_LEN,
- .tail_tag = true,
+ .needed_tailroom = KSZ_INGRESS_TAG_LEN,
};
DSA_TAG_DRIVER(ksz8795_netdev_ops);
.proto = DSA_TAG_PROTO_KSZ9477,
.xmit = ksz9477_xmit,
.rcv = ksz9477_rcv,
- .overhead = KSZ9477_INGRESS_TAG_LEN,
- .tail_tag = true,
+ .needed_tailroom = KSZ9477_INGRESS_TAG_LEN,
};
DSA_TAG_DRIVER(ksz9477_netdev_ops);
.proto = DSA_TAG_PROTO_KSZ9893,
.xmit = ksz9893_xmit,
.rcv = ksz9477_rcv,
- .overhead = KSZ_INGRESS_TAG_LEN,
- .tail_tag = true,
+ .needed_tailroom = KSZ_INGRESS_TAG_LEN,
};
DSA_TAG_DRIVER(ksz9893_netdev_ops);
.proto = DSA_TAG_PROTO_LAN9303,
.xmit = lan9303_xmit,
.rcv = lan9303_rcv,
- .overhead = LAN9303_TAG_LEN,
+ .needed_headroom = LAN9303_TAG_LEN,
};
MODULE_LICENSE("GPL");
.proto = DSA_TAG_PROTO_MTK,
.xmit = mtk_tag_xmit,
.rcv = mtk_tag_rcv,
- .overhead = MTK_HDR_LEN,
+ .needed_headroom = MTK_HDR_LEN,
};
MODULE_LICENSE("GPL");
.proto = DSA_TAG_PROTO_OCELOT,
.xmit = ocelot_xmit,
.rcv = ocelot_rcv,
- .overhead = OCELOT_TOTAL_TAG_LEN,
+ .needed_headroom = OCELOT_TOTAL_TAG_LEN,
.promisc_on_master = true,
};
.proto = DSA_TAG_PROTO_SEVILLE,
.xmit = seville_xmit,
.rcv = ocelot_rcv,
- .overhead = OCELOT_TOTAL_TAG_LEN,
+ .needed_headroom = OCELOT_TOTAL_TAG_LEN,
.promisc_on_master = true,
};
struct net_device *netdev,
struct packet_type *pt)
{
- int src_port, switch_id, qos_class;
- u16 vid, tci;
+ int src_port, switch_id, subvlan;
- skb_push_rcsum(skb, ETH_HLEN);
- if (skb_vlan_tag_present(skb)) {
- tci = skb_vlan_tag_get(skb);
- __vlan_hwaccel_clear_tag(skb);
- } else {
- __skb_vlan_pop(skb, &tci);
- }
- skb_pull_rcsum(skb, ETH_HLEN);
-
- vid = tci & VLAN_VID_MASK;
- src_port = dsa_8021q_rx_source_port(vid);
- switch_id = dsa_8021q_rx_switch_id(vid);
- qos_class = (tci & VLAN_PRIO_MASK) >> VLAN_PRIO_SHIFT;
+ dsa_8021q_rcv(skb, &src_port, &switch_id, &subvlan);
skb->dev = dsa_master_find_slave(netdev, switch_id, src_port);
if (!skb->dev)
return NULL;
skb->offload_fwd_mark = 1;
- skb->priority = qos_class;
return skb;
}
.proto = DSA_TAG_PROTO_OCELOT_8021Q,
.xmit = ocelot_xmit,
.rcv = ocelot_rcv,
- .overhead = VLAN_HLEN,
+ .needed_headroom = VLAN_HLEN,
.promisc_on_master = true,
};
.proto = DSA_TAG_PROTO_QCA,
.xmit = qca_tag_xmit,
.rcv = qca_tag_rcv,
- .overhead = QCA_HDR_LEN,
+ .needed_headroom = QCA_HDR_LEN,
};
MODULE_LICENSE("GPL");
.proto = DSA_TAG_PROTO_RTL4_A,
.xmit = rtl4a_tag_xmit,
.rcv = rtl4a_tag_rcv,
- .overhead = RTL4_A_HDR_LEN,
+ .needed_headroom = RTL4_A_HDR_LEN,
};
module_dsa_tag_driver(rtl4a_netdev_ops);
#include <linux/packing.h>
#include "dsa_priv.h"
+/* Is this a TX or an RX header? */
+#define SJA1110_HEADER_HOST_TO_SWITCH BIT(15)
+
+/* RX header */
+#define SJA1110_RX_HEADER_IS_METADATA BIT(14)
+#define SJA1110_RX_HEADER_HOST_ONLY BIT(13)
+#define SJA1110_RX_HEADER_HAS_TRAILER BIT(12)
+
+/* Trap-to-host format (no trailer present) */
+#define SJA1110_RX_HEADER_SRC_PORT(x) (((x) & GENMASK(7, 4)) >> 4)
+#define SJA1110_RX_HEADER_SWITCH_ID(x) ((x) & GENMASK(3, 0))
+
+/* Timestamp format (trailer present) */
+#define SJA1110_RX_HEADER_TRAILER_POS(x) ((x) & GENMASK(11, 0))
+
+#define SJA1110_RX_TRAILER_SWITCH_ID(x) (((x) & GENMASK(7, 4)) >> 4)
+#define SJA1110_RX_TRAILER_SRC_PORT(x) ((x) & GENMASK(3, 0))
+
+/* Meta frame format (for 2-step TX timestamps) */
+#define SJA1110_RX_HEADER_N_TS(x) (((x) & GENMASK(8, 4)) >> 4)
+
+/* TX header */
+#define SJA1110_TX_HEADER_UPDATE_TC BIT(14)
+#define SJA1110_TX_HEADER_TAKE_TS BIT(13)
+#define SJA1110_TX_HEADER_TAKE_TS_CASC BIT(12)
+#define SJA1110_TX_HEADER_HAS_TRAILER BIT(11)
+
+/* Only valid if SJA1110_TX_HEADER_HAS_TRAILER is false */
+#define SJA1110_TX_HEADER_PRIO(x) (((x) << 7) & GENMASK(10, 7))
+#define SJA1110_TX_HEADER_TSTAMP_ID(x) ((x) & GENMASK(7, 0))
+
+/* Only valid if SJA1110_TX_HEADER_HAS_TRAILER is true */
+#define SJA1110_TX_HEADER_TRAILER_POS(x) ((x) & GENMASK(10, 0))
+
+#define SJA1110_TX_TRAILER_TSTAMP_ID(x) (((x) << 24) & GENMASK(31, 24))
+#define SJA1110_TX_TRAILER_PRIO(x) (((x) << 21) & GENMASK(23, 21))
+#define SJA1110_TX_TRAILER_SWITCHID(x) (((x) << 12) & GENMASK(15, 12))
+#define SJA1110_TX_TRAILER_DESTPORTS(x) (((x) << 1) & GENMASK(11, 1))
+
+#define SJA1110_META_TSTAMP_SIZE 10
+
+#define SJA1110_HEADER_LEN 4
+#define SJA1110_RX_TRAILER_LEN 13
+#define SJA1110_TX_TRAILER_LEN 4
+#define SJA1110_MAX_PADDING_LEN 15
+
/* Similar to is_link_local_ether_addr(hdr->h_dest) but also covers PTP */
static inline bool sja1105_is_link_local(const struct sk_buff *skb)
{
((pcp << VLAN_PRIO_SHIFT) | tx_vid));
}
+static struct sk_buff *sja1110_xmit(struct sk_buff *skb,
+ struct net_device *netdev)
+{
+ struct sk_buff *clone = SJA1105_SKB_CB(skb)->clone;
+ struct dsa_port *dp = dsa_slave_to_port(netdev);
+ u16 tx_vid = dsa_8021q_tx_vid(dp->ds, dp->index);
+ u16 queue_mapping = skb_get_queue_mapping(skb);
+ u8 pcp = netdev_txq_to_tc(netdev, queue_mapping);
+ struct ethhdr *eth_hdr;
+ __be32 *tx_trailer;
+ __be16 *tx_header;
+ int trailer_pos;
+
+ /* Transmitting control packets is done using in-band control
+ * extensions, while data packets are transmitted using
+ * tag_8021q TX VLANs.
+ */
+ if (likely(!sja1105_is_link_local(skb)))
+ return dsa_8021q_xmit(skb, netdev, sja1105_xmit_tpid(dp->priv),
+ ((pcp << VLAN_PRIO_SHIFT) | tx_vid));
+
+ skb_push(skb, SJA1110_HEADER_LEN);
+
+ /* Move Ethernet header to the left, making space for DSA tag */
+ memmove(skb->data, skb->data + SJA1110_HEADER_LEN, 2 * ETH_ALEN);
+
+ trailer_pos = skb->len;
+
+ /* On TX, skb->data points to skb_mac_header(skb) */
+ eth_hdr = (struct ethhdr *)skb->data;
+ tx_header = (__be16 *)(eth_hdr + 1);
+ tx_trailer = skb_put(skb, SJA1110_TX_TRAILER_LEN);
+
+ eth_hdr->h_proto = htons(ETH_P_SJA1110);
+
+ *tx_header = htons(SJA1110_HEADER_HOST_TO_SWITCH |
+ SJA1110_TX_HEADER_HAS_TRAILER |
+ SJA1110_TX_HEADER_TRAILER_POS(trailer_pos));
+ *tx_trailer = cpu_to_be32(SJA1110_TX_TRAILER_PRIO(pcp) |
+ SJA1110_TX_TRAILER_SWITCHID(dp->ds->index) |
+ SJA1110_TX_TRAILER_DESTPORTS(BIT(dp->index)));
+ if (clone) {
+ u8 ts_id = SJA1105_SKB_CB(clone)->ts_id;
+
+ *tx_header |= htons(SJA1110_TX_HEADER_TAKE_TS);
+ *tx_trailer |= cpu_to_be32(SJA1110_TX_TRAILER_TSTAMP_ID(ts_id));
+ }
+
+ return skb;
+}
+
static void sja1105_transfer_meta(struct sk_buff *skb,
const struct sja1105_meta *meta)
{
hdr->h_dest[3] = meta->dmac_byte_3;
hdr->h_dest[4] = meta->dmac_byte_4;
- SJA1105_SKB_CB(skb)->meta_tstamp = meta->tstamp;
+ SJA1105_SKB_CB(skb)->tstamp = meta->tstamp;
}
/* This is a simple state machine which follows the hardware mechanism of
__vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), vlan_tci);
}
+static bool sja1105_skb_has_tag_8021q(const struct sk_buff *skb)
+{
+ u16 tpid = ntohs(eth_hdr(skb)->h_proto);
+
+ return tpid == ETH_P_SJA1105 || tpid == ETH_P_8021Q ||
+ skb_vlan_tag_present(skb);
+}
+
+static bool sja1110_skb_has_inband_control_extension(const struct sk_buff *skb)
+{
+ return ntohs(eth_hdr(skb)->h_proto) == ETH_P_SJA1110;
+}
+
static struct sk_buff *sja1105_rcv(struct sk_buff *skb,
struct net_device *netdev,
struct packet_type *pt)
{
+ int source_port, switch_id, subvlan = 0;
struct sja1105_meta meta = {0};
- int source_port, switch_id;
struct ethhdr *hdr;
- u16 tpid, vid, tci;
bool is_link_local;
- u16 subvlan = 0;
- bool is_tagged;
bool is_meta;
hdr = eth_hdr(skb);
- tpid = ntohs(hdr->h_proto);
- is_tagged = (tpid == ETH_P_SJA1105 || tpid == ETH_P_8021Q ||
- skb_vlan_tag_present(skb));
is_link_local = sja1105_is_link_local(skb);
is_meta = sja1105_is_meta_frame(skb);
skb->offload_fwd_mark = 1;
- if (is_tagged) {
+ if (sja1105_skb_has_tag_8021q(skb)) {
/* Normal traffic path. */
- skb_push_rcsum(skb, ETH_HLEN);
- if (skb_vlan_tag_present(skb)) {
- tci = skb_vlan_tag_get(skb);
- __vlan_hwaccel_clear_tag(skb);
- } else {
- __skb_vlan_pop(skb, &tci);
- }
- skb_pull_rcsum(skb, ETH_HLEN);
- skb_reset_network_header(skb);
- skb_reset_transport_header(skb);
-
- vid = tci & VLAN_VID_MASK;
- source_port = dsa_8021q_rx_source_port(vid);
- switch_id = dsa_8021q_rx_switch_id(vid);
- skb->priority = (tci & VLAN_PRIO_MASK) >> VLAN_PRIO_SHIFT;
- subvlan = dsa_8021q_rx_subvlan(vid);
+ dsa_8021q_rcv(skb, &source_port, &switch_id, &subvlan);
} else if (is_link_local) {
/* Management traffic path. Switch embeds the switch ID and
* port ID into bytes of the destination MAC, courtesy of
is_meta);
}
+static struct sk_buff *sja1110_rcv_meta(struct sk_buff *skb, u16 rx_header)
+{
+ int switch_id = SJA1110_RX_HEADER_SWITCH_ID(rx_header);
+ int n_ts = SJA1110_RX_HEADER_N_TS(rx_header);
+ struct net_device *master = skb->dev;
+ struct dsa_port *cpu_dp;
+ u8 *buf = skb->data + 2;
+ struct dsa_switch *ds;
+ int i;
+
+ cpu_dp = master->dsa_ptr;
+ ds = dsa_switch_find(cpu_dp->dst->index, switch_id);
+ if (!ds) {
+ net_err_ratelimited("%s: cannot find switch id %d\n",
+ master->name, switch_id);
+ return NULL;
+ }
+
+ for (i = 0; i <= n_ts; i++) {
+ u8 ts_id, source_port, dir;
+ u64 tstamp;
+
+ ts_id = buf[0];
+ source_port = (buf[1] & GENMASK(7, 4)) >> 4;
+ dir = (buf[1] & BIT(3)) >> 3;
+ tstamp = be64_to_cpu(*(__be64 *)(buf + 2));
+
+ sja1110_process_meta_tstamp(ds, source_port, ts_id, dir,
+ tstamp);
+
+ buf += SJA1110_META_TSTAMP_SIZE;
+ }
+
+ /* Discard the meta frame, we've consumed the timestamps it contained */
+ return NULL;
+}
+
+static struct sk_buff *sja1110_rcv_inband_control_extension(struct sk_buff *skb,
+ int *source_port,
+ int *switch_id)
+{
+ u16 rx_header;
+
+ if (unlikely(!pskb_may_pull(skb, SJA1110_HEADER_LEN)))
+ return NULL;
+
+ /* skb->data points to skb_mac_header(skb) + ETH_HLEN, which is exactly
+ * what we need because the caller has checked the EtherType (which is
+ * located 2 bytes back) and we just need a pointer to the header that
+ * comes afterwards.
+ */
+ rx_header = ntohs(*(__be16 *)skb->data);
+
+ if (rx_header & SJA1110_RX_HEADER_IS_METADATA)
+ return sja1110_rcv_meta(skb, rx_header);
+
+ /* Timestamp frame, we have a trailer */
+ if (rx_header & SJA1110_RX_HEADER_HAS_TRAILER) {
+ int start_of_padding = SJA1110_RX_HEADER_TRAILER_POS(rx_header);
+ u8 *rx_trailer = skb_tail_pointer(skb) - SJA1110_RX_TRAILER_LEN;
+ u64 *tstamp = &SJA1105_SKB_CB(skb)->tstamp;
+ u8 last_byte = rx_trailer[12];
+
+ /* The timestamp is unaligned, so we need to use packing()
+ * to get it
+ */
+ packing(rx_trailer, tstamp, 63, 0, 8, UNPACK, 0);
+
+ *source_port = SJA1110_RX_TRAILER_SRC_PORT(last_byte);
+ *switch_id = SJA1110_RX_TRAILER_SWITCH_ID(last_byte);
+
+ /* skb->len counts from skb->data, while start_of_padding
+ * counts from the destination MAC address. Right now skb->data
+ * is still as set by the DSA master, so to trim away the
+ * padding and trailer we need to account for the fact that
+ * skb->data points to skb_mac_header(skb) + ETH_HLEN.
+ */
+ pskb_trim_rcsum(skb, start_of_padding - ETH_HLEN);
+ /* Trap-to-host frame, no timestamp trailer */
+ } else {
+ *source_port = SJA1110_RX_HEADER_SRC_PORT(rx_header);
+ *switch_id = SJA1110_RX_HEADER_SWITCH_ID(rx_header);
+ }
+
+ /* Advance skb->data past the DSA header */
+ skb_pull_rcsum(skb, SJA1110_HEADER_LEN);
+
+ /* Remove the DSA header */
+ memmove(skb->data - ETH_HLEN, skb->data - ETH_HLEN - SJA1110_HEADER_LEN,
+ 2 * ETH_ALEN);
+
+ /* With skb->data in its final place, update the MAC header
+ * so that eth_hdr() continues to works properly.
+ */
+ skb_set_mac_header(skb, -ETH_HLEN);
+
+ return skb;
+}
+
+static struct sk_buff *sja1110_rcv(struct sk_buff *skb,
+ struct net_device *netdev,
+ struct packet_type *pt)
+{
+ int source_port = -1, switch_id = -1, subvlan = 0;
+
+ skb->offload_fwd_mark = 1;
+
+ if (sja1110_skb_has_inband_control_extension(skb)) {
+ skb = sja1110_rcv_inband_control_extension(skb, &source_port,
+ &switch_id);
+ if (!skb)
+ return NULL;
+ }
+
+ /* Packets with in-band control extensions might still have RX VLANs */
+ if (likely(sja1105_skb_has_tag_8021q(skb)))
+ dsa_8021q_rcv(skb, &source_port, &switch_id, &subvlan);
+
+ skb->dev = dsa_master_find_slave(netdev, switch_id, source_port);
+ if (!skb->dev) {
+ netdev_warn(netdev,
+ "Couldn't decode source port %d and switch id %d\n",
+ source_port, switch_id);
+ return NULL;
+ }
+
+ if (subvlan)
+ sja1105_decode_subvlan(skb, subvlan);
+
+ return skb;
+}
+
static void sja1105_flow_dissect(const struct sk_buff *skb, __be16 *proto,
int *offset)
{
dsa_tag_generic_flow_dissect(skb, proto, offset);
}
+static void sja1110_flow_dissect(const struct sk_buff *skb, __be16 *proto,
+ int *offset)
+{
+ /* Management frames have 2 DSA tags on RX, so the needed_headroom we
+ * declared is fine for the generic dissector adjustment procedure.
+ */
+ if (unlikely(sja1105_is_link_local(skb)))
+ return dsa_tag_generic_flow_dissect(skb, proto, offset);
+
+ /* For the rest, there is a single DSA tag, the tag_8021q one */
+ *offset = VLAN_HLEN;
+ *proto = ((__be16 *)skb->data)[(VLAN_HLEN / 2) - 1];
+}
+
static const struct dsa_device_ops sja1105_netdev_ops = {
.name = "sja1105",
.proto = DSA_TAG_PROTO_SJA1105,
.xmit = sja1105_xmit,
.rcv = sja1105_rcv,
.filter = sja1105_filter,
- .overhead = VLAN_HLEN,
+ .needed_headroom = VLAN_HLEN,
.flow_dissect = sja1105_flow_dissect,
.promisc_on_master = true,
};
-MODULE_LICENSE("GPL v2");
+DSA_TAG_DRIVER(sja1105_netdev_ops);
MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_SJA1105);
-module_dsa_tag_driver(sja1105_netdev_ops);
+static const struct dsa_device_ops sja1110_netdev_ops = {
+ .name = "sja1110",
+ .proto = DSA_TAG_PROTO_SJA1110,
+ .xmit = sja1110_xmit,
+ .rcv = sja1110_rcv,
+ .filter = sja1105_filter,
+ .flow_dissect = sja1110_flow_dissect,
+ .needed_headroom = SJA1110_HEADER_LEN + VLAN_HLEN,
+ .needed_tailroom = SJA1110_RX_TRAILER_LEN + SJA1110_MAX_PADDING_LEN,
+};
+
+DSA_TAG_DRIVER(sja1110_netdev_ops);
+MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_SJA1110);
+
+static struct dsa_tag_driver *sja1105_tag_driver_array[] = {
+ &DSA_TAG_DRIVER_NAME(sja1105_netdev_ops),
+ &DSA_TAG_DRIVER_NAME(sja1110_netdev_ops),
+};
+
+module_dsa_tag_drivers(sja1105_tag_driver_array);
+
+MODULE_LICENSE("GPL v2");
.proto = DSA_TAG_PROTO_TRAILER,
.xmit = trailer_xmit,
.rcv = trailer_rcv,
- .overhead = 4,
- .tail_tag = true,
+ .needed_tailroom = 4,
};
MODULE_LICENSE("GPL");
.proto = DSA_TAG_PROTO_XRS700X,
.xmit = xrs700x_xmit,
.rcv = xrs700x_rcv,
- .overhead = 1,
- .tail_tag = true,
+ .needed_tailroom = 1,
};
MODULE_LICENSE("GPL");
if (dev->sfp_bus)
return sfp_get_module_eeprom_by_page(dev->sfp_bus, page_data, extack);
- if (ops->get_module_info)
+ if (ops->get_module_eeprom_by_page)
return ops->get_module_eeprom_by_page(dev, page_data, extack);
return -EOPNOTSUPP;
request->offset = nla_get_u32(tb[ETHTOOL_A_MODULE_EEPROM_OFFSET]);
request->length = nla_get_u32(tb[ETHTOOL_A_MODULE_EEPROM_LENGTH]);
- if (!request->length)
- return -EINVAL;
-
/* The following set of conditions limit the API to only dump 1/2
* EEPROM page without crossing low page boundary located at offset 128.
* This means user may only request dumps of length limited to 128 from
NL_SET_ERR_MSG_ATTR(extack, tb[ETHTOOL_A_MODULE_EEPROM_LENGTH],
"reading cross half page boundary is illegal");
return -EINVAL;
- } else if (request->offset >= ETH_MODULE_EEPROM_PAGE_LEN * 2) {
- NL_SET_ERR_MSG_ATTR(extack, tb[ETHTOOL_A_MODULE_EEPROM_OFFSET],
- "offset is out of bounds");
- return -EINVAL;
} else if (request->offset + request->length > ETH_MODULE_EEPROM_PAGE_LEN * 2) {
NL_SET_ERR_MSG_ATTR(extack, tb[ETHTOOL_A_MODULE_EEPROM_LENGTH],
"reading cross page boundary is illegal");
const struct nla_policy ethnl_module_eeprom_get_policy[] = {
[ETHTOOL_A_MODULE_EEPROM_HEADER] = NLA_POLICY_NESTED(ethnl_header_policy),
- [ETHTOOL_A_MODULE_EEPROM_OFFSET] = { .type = NLA_U32 },
- [ETHTOOL_A_MODULE_EEPROM_LENGTH] = { .type = NLA_U32 },
+ [ETHTOOL_A_MODULE_EEPROM_OFFSET] =
+ NLA_POLICY_MAX(NLA_U32, ETH_MODULE_EEPROM_PAGE_LEN * 2 - 1),
+ [ETHTOOL_A_MODULE_EEPROM_LENGTH] =
+ NLA_POLICY_RANGE(NLA_U32, 1, ETH_MODULE_EEPROM_PAGE_LEN),
[ETHTOOL_A_MODULE_EEPROM_PAGE] = { .type = NLA_U8 },
[ETHTOOL_A_MODULE_EEPROM_BANK] = { .type = NLA_U8 },
[ETHTOOL_A_MODULE_EEPROM_I2C_ADDRESS] =
if (eeprom.offset + eeprom.len > total_len)
return -EINVAL;
- data = kmalloc(PAGE_SIZE, GFP_USER);
+ data = kzalloc(PAGE_SIZE, GFP_USER);
if (!data)
return -ENOMEM;
if (eeprom.offset + eeprom.len > ops->get_eeprom_len(dev))
return -EINVAL;
- data = kmalloc(PAGE_SIZE, GFP_USER);
+ data = kzalloc(PAGE_SIZE, GFP_USER);
if (!data)
return -ENOMEM;
return -EFAULT;
test.len = test_len;
- data = kmalloc_array(test_len, sizeof(u64), GFP_USER);
+ data = kcalloc(test_len, sizeof(u64), GFP_USER);
if (!data)
return -ENOMEM;
ret = ethtool_tunable_valid(&tuna);
if (ret)
return ret;
- data = kmalloc(tuna.len, GFP_USER);
+ data = kzalloc(tuna.len, GFP_USER);
if (!data)
return -ENOMEM;
ret = ops->get_tunable(dev, &tuna, data);
ret = ethtool_phy_tunable_valid(&tuna);
if (ret)
return ret;
- data = kmalloc(tuna.len, GFP_USER);
+ data = kzalloc(tuna.len, GFP_USER);
if (!data)
return -ENOMEM;
if (phy_drv_tunable) {
struct ethnl_req_info *req_info = NULL;
const u8 cmd = info->genlhdr->cmd;
const struct ethnl_request_ops *ops;
+ int hdr_len, reply_len;
struct sk_buff *rskb;
void *reply_payload;
- int reply_len;
int ret;
ops = ethnl_default_requests[cmd];
ret = ops->reply_size(req_info, reply_data);
if (ret < 0)
goto err_cleanup;
- reply_len = ret + ethnl_reply_header_size();
+ reply_len = ret;
ret = -ENOMEM;
- rskb = ethnl_reply_init(reply_len, req_info->dev, ops->reply_cmd,
+ rskb = ethnl_reply_init(reply_len + ethnl_reply_header_size(),
+ req_info->dev, ops->reply_cmd,
ops->hdr_attr, info, &reply_payload);
if (!rskb)
goto err_cleanup;
+ hdr_len = rskb->len;
ret = ops->fill_reply(rskb, req_info, reply_data);
if (ret < 0)
goto err_msg;
+ WARN_ONCE(rskb->len - hdr_len > reply_len,
+ "ethnl cmd %d: calculated reply length %d, but consumed %d\n",
+ cmd, reply_len, rskb->len - hdr_len);
if (ops->cleanup_data)
ops->cleanup_data(reply_data);
}
/**
- * ethnl_update_binary() - update binary data from NLA_BINARY atribute
+ * ethnl_update_binary() - update binary data from NLA_BINARY attribute
* @dst: value to update
* @len: destination buffer length
* @attr: netlink attribute with new value or null
extern const struct nla_policy ethnl_tunnel_info_get_policy[ETHTOOL_A_TUNNEL_INFO_HEADER + 1];
extern const struct nla_policy ethnl_fec_get_policy[ETHTOOL_A_FEC_HEADER + 1];
extern const struct nla_policy ethnl_fec_set_policy[ETHTOOL_A_FEC_AUTO + 1];
-extern const struct nla_policy ethnl_module_eeprom_get_policy[ETHTOOL_A_MODULE_EEPROM_DATA + 1];
+extern const struct nla_policy ethnl_module_eeprom_get_policy[ETHTOOL_A_MODULE_EEPROM_I2C_ADDRESS + 1];
extern const struct nla_policy ethnl_stats_get_policy[ETHTOOL_A_STATS_GROUPS + 1];
int ethnl_set_linkinfo(struct sk_buff *skb, struct genl_info *info);
*/
memset(&data->phy_stats, 0xff, sizeof(data->phy_stats));
memset(&data->mac_stats, 0xff, sizeof(data->mac_stats));
- memset(&data->ctrl_stats, 0xff, sizeof(data->mac_stats));
+ memset(&data->ctrl_stats, 0xff, sizeof(data->ctrl_stats));
memset(&data->rmon_stats, 0xff, sizeof(data->rmon_stats));
if (test_bit(ETHTOOL_STATS_ETH_PHY, req_info->stat_mask) &&
int len = 0;
int ret;
+ len += nla_total_size(0); /* ETHTOOL_A_STRSET_STRINGSETS */
+
for (i = 0; i < ETH_SS_COUNT; i++) {
const struct strset_info *set_info = &data->sets[i];
if (master) {
skb->dev = master->dev;
skb_reset_mac_header(skb);
+ skb_reset_mac_len(skb);
hsr_forward_skb(skb, master);
} else {
atomic_long_inc(&dev->tx_dropped);
goto out;
skb_reset_mac_header(skb);
+ skb_reset_mac_len(skb);
skb_reset_network_header(skb);
skb_reset_transport_header(skb);
}
}
-void hsr_fill_frame_info(__be16 proto, struct sk_buff *skb,
- struct hsr_frame_info *frame)
+int hsr_fill_frame_info(__be16 proto, struct sk_buff *skb,
+ struct hsr_frame_info *frame)
{
struct hsr_port *port = frame->port_rcv;
struct hsr_priv *hsr = port->hsr;
/* HSRv0 supervisory frames double as a tag so treat them as tagged. */
if ((!hsr->prot_version && proto == htons(ETH_P_PRP)) ||
proto == htons(ETH_P_HSR)) {
+ /* Check if skb contains hsr_ethhdr */
+ if (skb->mac_len < sizeof(struct hsr_ethhdr))
+ return -EINVAL;
+
/* HSR tagged frame :- Data or Supervision */
frame->skb_std = NULL;
frame->skb_prp = NULL;
frame->skb_hsr = skb;
frame->sequence_nr = hsr_get_skb_sequence_nr(skb);
- return;
+ return 0;
}
/* Standard frame or PRP from master port */
handle_std_frame(skb, frame);
+
+ return 0;
}
-void prp_fill_frame_info(__be16 proto, struct sk_buff *skb,
- struct hsr_frame_info *frame)
+int prp_fill_frame_info(__be16 proto, struct sk_buff *skb,
+ struct hsr_frame_info *frame)
{
/* Supervision frame */
struct prp_rct *rct = skb_get_PRP_rct(skb);
frame->skb_std = NULL;
frame->skb_prp = skb;
frame->sequence_nr = prp_get_skb_sequence_nr(rct);
- return;
+ return 0;
}
handle_std_frame(skb, frame);
+
+ return 0;
}
static int fill_frame_info(struct hsr_frame_info *frame,
struct hsr_vlan_ethhdr *vlan_hdr;
struct ethhdr *ethhdr;
__be16 proto;
+ int ret;
- /* Check if skb contains hsr_ethhdr */
- if (skb->mac_len < sizeof(struct hsr_ethhdr))
+ /* Check if skb contains ethhdr */
+ if (skb->mac_len < sizeof(struct ethhdr))
return -EINVAL;
memset(frame, 0, sizeof(*frame));
frame->is_from_san = false;
frame->port_rcv = port;
- hsr->proto_ops->fill_frame_info(proto, skb, frame);
+ ret = hsr->proto_ops->fill_frame_info(proto, skb, frame);
+ if (ret)
+ return ret;
+
check_local_dest(port->hsr, skb, frame);
return 0;
struct hsr_port *port);
bool prp_drop_frame(struct hsr_frame_info *frame, struct hsr_port *port);
bool hsr_drop_frame(struct hsr_frame_info *frame, struct hsr_port *port);
-void prp_fill_frame_info(__be16 proto, struct sk_buff *skb,
- struct hsr_frame_info *frame);
-void hsr_fill_frame_info(__be16 proto, struct sk_buff *skb,
- struct hsr_frame_info *frame);
+int prp_fill_frame_info(__be16 proto, struct sk_buff *skb,
+ struct hsr_frame_info *frame);
+int hsr_fill_frame_info(__be16 proto, struct sk_buff *skb,
+ struct hsr_frame_info *frame);
#endif /* __HSR_FORWARD_H */
* ensures entries of restarted nodes gets pruned so that they can
* re-register and resume communications.
*/
- if (seq_nr_before(sequence_nr, node->seq_out[port->type]))
+ if (!(port->dev->features & NETIF_F_HW_HSR_TAG_RM) &&
+ seq_nr_before(sequence_nr, node->seq_out[port->type]))
return;
node->time_in[port->type] = jiffies;
struct hsr_port *port);
struct sk_buff * (*create_tagged_frame)(struct hsr_frame_info *frame,
struct hsr_port *port);
- void (*fill_frame_info)(__be16 proto, struct sk_buff *skb,
- struct hsr_frame_info *frame);
+ int (*fill_frame_info)(__be16 proto, struct sk_buff *skb,
+ struct hsr_frame_info *frame);
bool (*invalid_dan_ingress_frame)(__be16 protocol);
void (*update_san_info)(struct hsr_node *node, bool is_sup);
};
goto finish_pass;
skb_push(skb, ETH_HLEN);
-
- if (skb_mac_header(skb) != skb->data) {
- WARN_ONCE(1, "%s:%d: Malformed frame at source port %s)\n",
- __func__, __LINE__, port->dev->name);
- goto finish_consume;
- }
+ skb_reset_mac_header(skb);
+ if ((!hsr->prot_version && protocol == htons(ETH_P_PRP)) ||
+ protocol == htons(ETH_P_HSR))
+ skb_set_network_header(skb, ETH_HLEN + HSR_HLEN);
+ skb_reset_mac_len(skb);
hsr_forward_skb(skb, port);
nla_put_u8(msg, IEEE802154_ATTR_LLSEC_SECLEVEL, params.out_level) ||
nla_put_u32(msg, IEEE802154_ATTR_LLSEC_FRAME_COUNTER,
be32_to_cpu(params.frame_counter)) ||
- ieee802154_llsec_fill_key_id(msg, ¶ms.out_key))
+ ieee802154_llsec_fill_key_id(msg, ¶ms.out_key)) {
+ rc = -ENOBUFS;
goto out_free;
+ }
dev_put(dev);
{
struct ieee802154_llsec_device *dpos;
struct ieee802154_llsec_device_key *kpos;
- int rc = 0, idx = 0, idx2;
+ int idx = 0, idx2;
list_for_each_entry(dpos, &data->table->devices, list) {
if (idx++ < data->s_idx)
data->nlmsg_seq,
dpos->hwaddr, kpos,
data->dev)) {
- return rc = -EMSGSIZE;
+ return -EMSGSIZE;
}
data->s_idx2++;
data->s_idx++;
}
- return rc;
+ return 0;
}
int ieee802154_llsec_dump_devkeys(struct sk_buff *skb,
}
if (nla_put_string(msg, IEEE802154_ATTR_PHY_NAME, wpan_phy_name(phy)) ||
- nla_put_string(msg, IEEE802154_ATTR_DEV_NAME, dev->name))
+ nla_put_string(msg, IEEE802154_ATTR_DEV_NAME, dev->name)) {
+ rc = -EMSGSIZE;
goto nla_put_failure;
+ }
dev_put(dev);
wpan_phy_put(phy);
if (!nla || nla_parse_nested_deprecated(attrs, NL802154_DEV_ADDR_ATTR_MAX, nla, nl802154_dev_addr_policy, NULL))
return -EINVAL;
- if (!attrs[NL802154_DEV_ADDR_ATTR_PAN_ID] ||
- !attrs[NL802154_DEV_ADDR_ATTR_MODE] ||
- !(attrs[NL802154_DEV_ADDR_ATTR_SHORT] ||
- attrs[NL802154_DEV_ADDR_ATTR_EXTENDED]))
+ if (!attrs[NL802154_DEV_ADDR_ATTR_PAN_ID] || !attrs[NL802154_DEV_ADDR_ATTR_MODE])
return -EINVAL;
addr->pan_id = nla_get_le16(attrs[NL802154_DEV_ADDR_ATTR_PAN_ID]);
addr->mode = nla_get_u32(attrs[NL802154_DEV_ADDR_ATTR_MODE]);
switch (addr->mode) {
case NL802154_DEV_ADDR_SHORT:
+ if (!attrs[NL802154_DEV_ADDR_ATTR_SHORT])
+ return -EINVAL;
addr->short_addr = nla_get_le16(attrs[NL802154_DEV_ADDR_ATTR_SHORT]);
break;
case NL802154_DEV_ADDR_EXTENDED:
+ if (!attrs[NL802154_DEV_ADDR_ATTR_EXTENDED])
+ return -EINVAL;
addr->extended_addr = nla_get_le64(attrs[NL802154_DEV_ADDR_ATTR_EXTENDED]);
break;
default:
WARN_ON(!answer_prot->slab);
- err = -ENOBUFS;
+ err = -ENOMEM;
sk = sk_alloc(net, PF_INET, GFP_KERNEL, answer_prot, kern);
if (!sk)
goto out;
return err;
}
- if (!inet_sk(sk)->inet_num && inet_autobind(sk))
+ if (data_race(!inet_sk(sk)->inet_num) && inet_autobind(sk))
return -EAGAIN;
return sk->sk_prot->connect(sk, uaddr, addr_len);
}
sock_rps_record_flow(sk);
/* We may need to bind the socket. */
- if (!inet_sk(sk)->inet_num && !sk->sk_prot->no_autobind &&
+ if (data_race(!inet_sk(sk)->inet_num) && !sk->sk_prot->no_autobind &&
inet_autobind(sk))
return -EAGAIN;
#ifdef CONFIG_IP_MULTICAST
static const struct net_protocol igmp_protocol = {
.handler = igmp_rcv,
- .netns_ok = 1,
};
#endif
.handler = tcp_v4_rcv,
.err_handler = tcp_v4_err,
.no_policy = 1,
- .netns_ok = 1,
.icmp_strict_tag_validation = 1,
};
.handler = udp_rcv,
.err_handler = udp_err,
.no_policy = 1,
- .netns_ok = 1,
};
static const struct net_protocol icmp_protocol = {
.handler = icmp_rcv,
.err_handler = icmp_err,
.no_policy = 1,
- .netns_ok = 1,
};
static __net_init int ipv4_mib_init_net(struct net *net)
BTF_ID(func, tcp_reno_undo_cwnd)
BTF_ID(func, tcp_slow_start)
BTF_ID(func, tcp_cong_avoid_ai)
+#ifdef CONFIG_X86
#ifdef CONFIG_DYNAMIC_FTRACE
#if IS_BUILTIN(CONFIG_TCP_CONG_CUBIC)
BTF_ID(func, cubictcp_init)
BTF_ID(func, bbr_set_state)
#endif
#endif /* CONFIG_DYNAMIC_FTRACE */
+#endif /* CONFIG_X86 */
BTF_SET_END(bpf_tcp_ca_kfunc_ids)
static bool bpf_tcp_ca_check_kfunc_call(u32 kfunc_btf_id)
* cipso_v4_cache_invalidate - Invalidates the current CIPSO cache
*
* Description:
- * Invalidates and frees any entries in the CIPSO cache. Returns zero on
- * success and negative values on failure.
+ * Invalidates and frees any entries in the CIPSO cache.
*
*/
void cipso_v4_cache_invalidate(void)
kfree(doi_def->map.std->lvl.local);
kfree(doi_def->map.std->cat.cipso);
kfree(doi_def->map.std->cat.local);
+ kfree(doi_def->map.std);
break;
}
kfree(doi_def);
return -EAFNOSUPPORT;
if (nla_parse_nested_deprecated(tb, IFLA_INET_MAX, nla, NULL, NULL) < 0)
- BUG();
+ return -EINVAL;
if (tb[IFLA_INET_CONF]) {
nla_for_each_nested(a, tb[IFLA_INET_CONF], rem)
prefix, ifa->ifa_prefixlen, prim,
ifa->ifa_rt_priority);
- /* Add network specific broadcasts, when it takes a sense */
+ /* Add the network broadcast address, when it makes sense */
if (ifa->ifa_prefixlen < 31) {
- fib_magic(RTM_NEWROUTE, RTN_BROADCAST, prefix, 32,
- prim, 0);
fib_magic(RTM_NEWROUTE, RTN_BROADCAST, prefix | ~mask,
32, prim, 0);
}
if (err)
return err;
+#ifdef CONFIG_IP_ROUTE_MULTIPATH
+ /* Default to 3-tuple */
+ net->ipv4.sysctl_fib_multipath_hash_fields =
+ FIB_MULTIPATH_HASH_FIELD_DEFAULT_MASK;
+#endif
+
/* Avoid false sharing : Use at least a full cache line */
size = max_t(size_t, size, L1_CACHE_BYTES);
#define FA_S_ACCESSED 0x01
-/* Dont write on fa_state unless needed, to keep it shared on all cpus */
+/* Don't write on fa_state unless needed, to keep it shared on all cpus */
static inline void fib_alias_accessed(struct fib_alias *fa)
{
if (!(fa->fa_state & FA_S_ACCESSED))
static const struct net_protocol net_gre_protocol = {
.handler = gre_rcv,
.err_handler = gre_err,
- .netns_ok = 1,
};
static int __init gre_init(void)
icmp_param.data_len = room;
icmp_param.head_len = sizeof(struct icmphdr);
+ /* if we don't have a source address at this point, fall back to the
+ * dummy address instead of sending out a packet with a source address
+ * of 0.0.0.0
+ */
+ if (!fl4.saddr)
+ fl4.saddr = htonl(INADDR_DUMMY);
+
icmp_push_reply(&icmp_param, &fl4, &ipc, &rt);
ende:
ip_rt_put(rt);
if (ident_len != sizeof(iio->ident.addr.ctype3_hdr) +
sizeof(struct in_addr))
goto send_mal_query;
- dev = ip_dev_find(net, iio->ident.addr.ip_addr.ipv4_addr.s_addr);
+ dev = ip_dev_find(net, iio->ident.addr.ip_addr.ipv4_addr);
break;
#if IS_ENABLED(CONFIG_IPV6)
case ICMP_AFI_IP6:
while ((i = rtnl_dereference(in_dev->mc_list)) != NULL) {
in_dev->mc_list = i->next_rcu;
in_dev->mc_count--;
+ ip_mc_clear_src(i);
ip_ma_put(i);
}
}
bool relax, bool reuseport_ok)
{
struct sock *sk2;
+ bool reuseport_cb_ok;
bool reuse = sk->sk_reuse;
bool reuseport = !!sk->sk_reuseport;
+ struct sock_reuseport *reuseport_cb;
kuid_t uid = sock_i_uid((struct sock *)sk);
+ rcu_read_lock();
+ reuseport_cb = rcu_dereference(sk->sk_reuseport_cb);
+ /* paired with WRITE_ONCE() in __reuseport_(add|detach)_closed_sock */
+ reuseport_cb_ok = !reuseport_cb || READ_ONCE(reuseport_cb->num_closed_socks);
+ rcu_read_unlock();
+
/*
* Unlike other sk lookup places we do not check
* for sk_net here, since _all_ the socks listed
if ((!relax ||
(!reuseport_ok &&
reuseport && sk2->sk_reuseport &&
- !rcu_access_pointer(sk->sk_reuseport_cb) &&
+ reuseport_cb_ok &&
(sk2->sk_state == TCP_TIME_WAIT ||
uid_eq(uid, sock_i_uid(sk2))))) &&
inet_rcv_saddr_equal(sk, sk2, true))
break;
} else if (!reuseport_ok ||
!reuseport || !sk2->sk_reuseport ||
- rcu_access_pointer(sk->sk_reuseport_cb) ||
+ !reuseport_cb_ok ||
(sk2->sk_state != TCP_TIME_WAIT &&
!uid_eq(uid, sock_i_uid(sk2)))) {
if (inet_rcv_saddr_equal(sk, sk2, true))
}
EXPORT_SYMBOL(inet_rtx_syn_ack);
+static struct request_sock *inet_reqsk_clone(struct request_sock *req,
+ struct sock *sk)
+{
+ struct sock *req_sk, *nreq_sk;
+ struct request_sock *nreq;
+
+ nreq = kmem_cache_alloc(req->rsk_ops->slab, GFP_ATOMIC | __GFP_NOWARN);
+ if (!nreq) {
+ __NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMIGRATEREQFAILURE);
+
+ /* paired with refcount_inc_not_zero() in reuseport_migrate_sock() */
+ sock_put(sk);
+ return NULL;
+ }
+
+ req_sk = req_to_sk(req);
+ nreq_sk = req_to_sk(nreq);
+
+ memcpy(nreq_sk, req_sk,
+ offsetof(struct sock, sk_dontcopy_begin));
+ memcpy(&nreq_sk->sk_dontcopy_end, &req_sk->sk_dontcopy_end,
+ req->rsk_ops->obj_size - offsetof(struct sock, sk_dontcopy_end));
+
+ sk_node_init(&nreq_sk->sk_node);
+ nreq_sk->sk_tx_queue_mapping = req_sk->sk_tx_queue_mapping;
+#ifdef CONFIG_XPS
+ nreq_sk->sk_rx_queue_mapping = req_sk->sk_rx_queue_mapping;
+#endif
+ nreq_sk->sk_incoming_cpu = req_sk->sk_incoming_cpu;
+
+ nreq->rsk_listener = sk;
+
+ /* We need not acquire fastopenq->lock
+ * because the child socket is locked in inet_csk_listen_stop().
+ */
+ if (sk->sk_protocol == IPPROTO_TCP && tcp_rsk(nreq)->tfo_listener)
+ rcu_assign_pointer(tcp_sk(nreq->sk)->fastopen_rsk, nreq);
+
+ return nreq;
+}
+
+static void reqsk_queue_migrated(struct request_sock_queue *queue,
+ const struct request_sock *req)
+{
+ if (req->num_timeout == 0)
+ atomic_inc(&queue->young);
+ atomic_inc(&queue->qlen);
+}
+
+static void reqsk_migrate_reset(struct request_sock *req)
+{
+ req->saved_syn = NULL;
+#if IS_ENABLED(CONFIG_IPV6)
+ inet_rsk(req)->ipv6_opt = NULL;
+ inet_rsk(req)->pktopts = NULL;
+#else
+ inet_rsk(req)->ireq_opt = NULL;
+#endif
+}
+
/* return true if req was found in the ehash table */
static bool reqsk_queue_unlink(struct request_sock *req)
{
static void reqsk_timer_handler(struct timer_list *t)
{
struct request_sock *req = from_timer(req, t, rsk_timer);
+ struct request_sock *nreq = NULL, *oreq = req;
struct sock *sk_listener = req->rsk_listener;
- struct net *net = sock_net(sk_listener);
- struct inet_connection_sock *icsk = inet_csk(sk_listener);
- struct request_sock_queue *queue = &icsk->icsk_accept_queue;
+ struct inet_connection_sock *icsk;
+ struct request_sock_queue *queue;
+ struct net *net;
int max_syn_ack_retries, qlen, expire = 0, resend = 0;
- if (inet_sk_state_load(sk_listener) != TCP_LISTEN)
- goto drop;
+ if (inet_sk_state_load(sk_listener) != TCP_LISTEN) {
+ struct sock *nsk;
+
+ nsk = reuseport_migrate_sock(sk_listener, req_to_sk(req), NULL);
+ if (!nsk)
+ goto drop;
+ nreq = inet_reqsk_clone(req, nsk);
+ if (!nreq)
+ goto drop;
+
+ /* The new timer for the cloned req can decrease the 2
+ * by calling inet_csk_reqsk_queue_drop_and_put(), so
+ * hold another count to prevent use-after-free and
+ * call reqsk_put() just before return.
+ */
+ refcount_set(&nreq->rsk_refcnt, 2 + 1);
+ timer_setup(&nreq->rsk_timer, reqsk_timer_handler, TIMER_PINNED);
+ reqsk_queue_migrated(&inet_csk(nsk)->icsk_accept_queue, req);
+
+ req = nreq;
+ sk_listener = nsk;
+ }
+
+ icsk = inet_csk(sk_listener);
+ net = sock_net(sk_listener);
max_syn_ack_retries = icsk->icsk_syn_retries ? : net->ipv4.sysctl_tcp_synack_retries;
/* Normally all the openreqs are young and become mature
* (i.e. converted to established socket) for first timeout.
* embrions; and abort old ones without pity, if old
* ones are about to clog our table.
*/
+ queue = &icsk->icsk_accept_queue;
qlen = reqsk_queue_len(queue);
if ((qlen << 1) > max(8U, READ_ONCE(sk_listener->sk_max_ack_backlog))) {
int young = reqsk_queue_len_young(queue) << 1;
atomic_dec(&queue->young);
timeo = min(TCP_TIMEOUT_INIT << req->num_timeout, TCP_RTO_MAX);
mod_timer(&req->rsk_timer, jiffies + timeo);
+
+ if (!nreq)
+ return;
+
+ if (!inet_ehash_insert(req_to_sk(nreq), req_to_sk(oreq), NULL)) {
+ /* delete timer */
+ inet_csk_reqsk_queue_drop(sk_listener, nreq);
+ goto no_ownership;
+ }
+
+ __NET_INC_STATS(net, LINUX_MIB_TCPMIGRATEREQSUCCESS);
+ reqsk_migrate_reset(oreq);
+ reqsk_queue_removed(&inet_csk(oreq->rsk_listener)->icsk_accept_queue, oreq);
+ reqsk_put(oreq);
+
+ reqsk_put(nreq);
return;
}
+
+ /* Even if we can clone the req, we may need not retransmit any more
+ * SYN+ACKs (nreq->num_timeout > max_syn_ack_retries, etc), or another
+ * CPU may win the "own_req" race so that inet_ehash_insert() fails.
+ */
+ if (nreq) {
+ __NET_INC_STATS(net, LINUX_MIB_TCPMIGRATEREQFAILURE);
+no_ownership:
+ reqsk_migrate_reset(nreq);
+ reqsk_queue_removed(queue, nreq);
+ __reqsk_free(nreq);
+ }
+
drop:
- inet_csk_reqsk_queue_drop_and_put(sk_listener, req);
+ inet_csk_reqsk_queue_drop_and_put(oreq->rsk_listener, oreq);
}
static void reqsk_queue_hash_req(struct request_sock *req,
struct request_sock *req, bool own_req)
{
if (own_req) {
- inet_csk_reqsk_queue_drop(sk, req);
- reqsk_queue_removed(&inet_csk(sk)->icsk_accept_queue, req);
- if (inet_csk_reqsk_queue_add(sk, req, child))
+ inet_csk_reqsk_queue_drop(req->rsk_listener, req);
+ reqsk_queue_removed(&inet_csk(req->rsk_listener)->icsk_accept_queue, req);
+
+ if (sk != req->rsk_listener) {
+ /* another listening sk has been selected,
+ * migrate the req to it.
+ */
+ struct request_sock *nreq;
+
+ /* hold a refcnt for the nreq->rsk_listener
+ * which is assigned in inet_reqsk_clone()
+ */
+ sock_hold(sk);
+ nreq = inet_reqsk_clone(req, sk);
+ if (!nreq) {
+ inet_child_forget(sk, req, child);
+ goto child_put;
+ }
+
+ refcount_set(&nreq->rsk_refcnt, 1);
+ if (inet_csk_reqsk_queue_add(sk, nreq, child)) {
+ __NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMIGRATEREQSUCCESS);
+ reqsk_migrate_reset(req);
+ reqsk_put(req);
+ return child;
+ }
+
+ __NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMIGRATEREQFAILURE);
+ reqsk_migrate_reset(nreq);
+ __reqsk_free(nreq);
+ } else if (inet_csk_reqsk_queue_add(sk, req, child)) {
return child;
+ }
}
/* Too bad, another child took ownership of the request, undo. */
+child_put:
bh_unlock_sock(child);
sock_put(child);
return NULL;
* of the variants now. --ANK
*/
while ((req = reqsk_queue_remove(queue, sk)) != NULL) {
- struct sock *child = req->sk;
+ struct sock *child = req->sk, *nsk;
+ struct request_sock *nreq;
local_bh_disable();
bh_lock_sock(child);
WARN_ON(sock_owned_by_user(child));
sock_hold(child);
+ nsk = reuseport_migrate_sock(sk, child, NULL);
+ if (nsk) {
+ nreq = inet_reqsk_clone(req, nsk);
+ if (nreq) {
+ refcount_set(&nreq->rsk_refcnt, 1);
+
+ if (inet_csk_reqsk_queue_add(nsk, nreq, child)) {
+ __NET_INC_STATS(sock_net(nsk),
+ LINUX_MIB_TCPMIGRATEREQSUCCESS);
+ reqsk_migrate_reset(req);
+ } else {
+ __NET_INC_STATS(sock_net(nsk),
+ LINUX_MIB_TCPMIGRATEREQFAILURE);
+ reqsk_migrate_reset(nreq);
+ __reqsk_free(nreq);
+ }
+
+ /* inet_csk_reqsk_queue_add() has already
+ * called inet_child_forget() on failure case.
+ */
+ goto skip_child_forget;
+ }
+ }
+
inet_child_forget(sk, req, child);
+skip_child_forget:
reqsk_put(req);
bh_unlock_sock(child);
local_bh_enable();
static int inet_twsk_diag_fill(struct sock *sk,
struct sk_buff *skb,
struct netlink_callback *cb,
- u16 nlmsg_flags)
+ u16 nlmsg_flags, bool net_admin)
{
struct inet_timewait_sock *tw = inet_twsk(sk);
struct inet_diag_msg *r;
r->idiag_uid = 0;
r->idiag_inode = 0;
+ if (net_admin && nla_put_u32(skb, INET_DIAG_MARK,
+ tw->tw_mark)) {
+ nlmsg_cancel(skb, nlh);
+ return -EMSGSIZE;
+ }
+
nlmsg_end(skb, nlh);
return 0;
}
u16 nlmsg_flags, bool net_admin)
{
if (sk->sk_state == TCP_TIME_WAIT)
- return inet_twsk_diag_fill(sk, skb, cb, nlmsg_flags);
+ return inet_twsk_diag_fill(sk, skb, cb, nlmsg_flags, net_admin);
if (sk->sk_state == TCP_NEW_SYN_RECV)
return inet_req_diag_fill(sk, skb, cb, nlmsg_flags, net_admin);
entry.mark = sk->sk_mark;
else if (sk->sk_state == TCP_NEW_SYN_RECV)
entry.mark = inet_rsk(inet_reqsk(sk))->ir_mark;
+ else if (sk->sk_state == TCP_TIME_WAIT)
+ entry.mark = inet_twsk(sk)->tw_mark;
else
entry.mark = 0;
#ifdef CONFIG_SOCK_CGROUP_DATA
goto unlock;
if (rcu_access_pointer(sk->sk_reuseport_cb))
- reuseport_detach_sock(sk);
+ reuseport_stop_listen_sock(sk);
if (ilb) {
inet_unhash2(hashinfo, sk);
ilb->count--;
MODULE_PARM_DESC(log_ecn_error, "Log packets received with corrupted ECN");
static struct rtnl_link_ops ipgre_link_ops __read_mostly;
+static const struct header_ops ipgre_header_ops;
+
static int ipgre_tunnel_init(struct net_device *dev);
static void erspan_build_header(struct sk_buff *skb,
u32 id, u32 index,
raw_proto, false) < 0)
goto drop;
- if (tunnel->dev->type != ARPHRD_NONE)
+ /* Special case for ipgre_header_parse(), which expects the
+ * mac_header to point to the outer IP header.
+ */
+ if (tunnel->dev->header_ops == &ipgre_header_ops)
skb_pop_mac_header(skb);
else
skb_reset_mac_header(skb);
unsigned int datalen;
unsigned int fraglen;
unsigned int fraggap;
- unsigned int alloclen;
+ unsigned int alloclen, alloc_extra;
unsigned int pagedlen;
struct sk_buff *skb_prev;
alloc_new_skb:
fraglen = datalen + fragheaderlen;
pagedlen = 0;
+ alloc_extra = hh_len + 15;
+ alloc_extra += exthdrlen;
+
+ /* The last fragment gets additional space at tail.
+ * Note, with MSG_MORE we overallocate on fragments,
+ * because we have no idea what fragment will be
+ * the last.
+ */
+ if (datalen == length + fraggap)
+ alloc_extra += rt->dst.trailer_len;
+
if ((flags & MSG_MORE) &&
!(rt->dst.dev->features&NETIF_F_SG))
alloclen = mtu;
- else if (!paged)
+ else if (!paged &&
+ (fraglen + alloc_extra < SKB_MAX_ALLOC ||
+ !(rt->dst.dev->features & NETIF_F_SG)))
alloclen = fraglen;
else {
alloclen = min_t(int, fraglen, MAX_HEADER);
pagedlen = fraglen - alloclen;
}
- alloclen += exthdrlen;
-
- /* The last fragment gets additional space at tail.
- * Note, with MSG_MORE we overallocate on fragments,
- * because we have no idea what fragment will be
- * the last.
- */
- if (datalen == length + fraggap)
- alloclen += rt->dst.trailer_len;
+ alloclen += alloc_extra;
if (transhdrlen) {
- skb = sock_alloc_send_skb(sk,
- alloclen + hh_len + 15,
+ skb = sock_alloc_send_skb(sk, alloclen,
(flags & MSG_DONTWAIT), &err);
} else {
skb = NULL;
if (refcount_read(&sk->sk_wmem_alloc) + wmem_alloc_delta <=
2 * sk->sk_sndbuf)
- skb = alloc_skb(alloclen + hh_len + 15,
+ skb = alloc_skb(alloclen,
sk->sk_allocation);
if (unlikely(!skb))
err = -ENOBUFS;
/*
- * Copy BOOTP-supplied string if not already set.
+ * Copy BOOTP-supplied string
*/
static int __init ic_bootp_string(char *dest, char *src, int len, int max)
{
}
break;
case 12: /* Host name */
- ic_bootp_string(utsname()->nodename, ext+1, *ext,
- __NEW_UTS_LEN);
- ic_host_name_set = 1;
+ if (!ic_host_name_set) {
+ ic_bootp_string(utsname()->nodename, ext+1, *ext,
+ __NEW_UTS_LEN);
+ ic_host_name_set = 1;
+ }
break;
case 15: /* Domain name (DNS) */
- ic_bootp_string(ic_domain, ext+1, *ext, sizeof(ic_domain));
+ if (!ic_domain[0])
+ ic_bootp_string(ic_domain, ext+1, *ext, sizeof(ic_domain));
break;
case 17: /* Root path */
if (!root_server_path[0])
if (!tun_dst)
return 0;
}
+ skb_reset_mac_header(skb);
+
return ip_tunnel_rcv(tunnel, skb, tpi, tun_dst, log_ecn_error);
}
}
/* called from ip_ra_control(), before an RCU grace period,
- * we dont need to call synchronize_rcu() here
+ * we don't need to call synchronize_rcu() here
*/
static void mrtsock_destruct(struct sock *sk)
{
if (c->mfc_origin == htonl(INADDR_ANY) && true_vifi >= 0) {
struct mfc_cache *cache_proxy;
- /* For an (*,G) entry, we only check that the incomming
+ /* For an (*,G) entry, we only check that the incoming
* interface is part of the static tree.
*/
cache_proxy = mr_mfc_find_any_parent(mrt, vif);
#ifdef CONFIG_IP_PIMSM_V2
static const struct net_protocol pim_protocol = {
.handler = pim_rcv,
- .netns_ok = 1,
};
#endif
nf_send_unreach(pkt->skb, priv->icmp_code, nft_hook(pkt));
break;
case NFT_REJECT_TCP_RST:
- nf_send_reset(nft_net(pkt), pkt->xt.state->sk, pkt->skb,
+ nf_send_reset(nft_net(pkt), nft_sk(pkt), pkt->skb,
nft_hook(pkt));
break;
default:
struct sock *sk;
struct net *net = dev_net(skb->dev);
struct icmphdr *icmph = icmp_hdr(skb);
+ bool rc = false;
/* We assume the packet has already been checked by icmp_rcv */
struct sk_buff *skb2 = skb_clone(skb, GFP_ATOMIC);
pr_debug("rcv on socket %p\n", sk);
- if (skb2)
- ping_queue_rcv_skb(sk, skb2);
+ if (skb2 && !ping_queue_rcv_skb(sk, skb2))
+ rc = true;
sock_put(sk);
- return true;
}
- pr_debug("no socket, dropping\n");
- return false;
+ if (!rc)
+ pr_debug("no socket, dropping\n");
+
+ return rc;
}
EXPORT_SYMBOL_GPL(ping_rcv);
SNMP_MIB_ITEM("TcpDuplicateDataRehash", LINUX_MIB_TCPDUPLICATEDATAREHASH),
SNMP_MIB_ITEM("TCPDSACKRecvSegs", LINUX_MIB_TCPDSACKRECVSEGS),
SNMP_MIB_ITEM("TCPDSACKIgnoredDubious", LINUX_MIB_TCPDSACKIGNOREDDUBIOUS),
+ SNMP_MIB_ITEM("TCPMigrateReqSuccess", LINUX_MIB_TCPMIGRATEREQSUCCESS),
+ SNMP_MIB_ITEM("TCPMigrateReqFailure", LINUX_MIB_TCPMIGRATEREQFAILURE),
SNMP_MIB_SENTINEL
};
int inet_add_protocol(const struct net_protocol *prot, unsigned char protocol)
{
- if (!prot->netns_ok) {
- pr_err("Protocol %u is not namespace aware, cannot register.\n",
- protocol);
- return -EINVAL;
- }
-
return !cmpxchg((const struct net_protocol **)&inet_protos[protocol],
NULL, prot) ? 0 : -1;
}
hash_keys->addrs.v4addrs.dst = key_iph->daddr;
}
+static u32 fib_multipath_custom_hash_outer(const struct net *net,
+ const struct sk_buff *skb,
+ bool *p_has_inner)
+{
+ u32 hash_fields = net->ipv4.sysctl_fib_multipath_hash_fields;
+ struct flow_keys keys, hash_keys;
+
+ if (!(hash_fields & FIB_MULTIPATH_HASH_FIELD_OUTER_MASK))
+ return 0;
+
+ memset(&hash_keys, 0, sizeof(hash_keys));
+ skb_flow_dissect_flow_keys(skb, &keys, FLOW_DISSECTOR_F_STOP_AT_ENCAP);
+
+ hash_keys.control.addr_type = FLOW_DISSECTOR_KEY_IPV4_ADDRS;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_SRC_IP)
+ hash_keys.addrs.v4addrs.src = keys.addrs.v4addrs.src;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_DST_IP)
+ hash_keys.addrs.v4addrs.dst = keys.addrs.v4addrs.dst;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_IP_PROTO)
+ hash_keys.basic.ip_proto = keys.basic.ip_proto;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_SRC_PORT)
+ hash_keys.ports.src = keys.ports.src;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_DST_PORT)
+ hash_keys.ports.dst = keys.ports.dst;
+
+ *p_has_inner = !!(keys.control.flags & FLOW_DIS_ENCAPSULATION);
+ return flow_hash_from_keys(&hash_keys);
+}
+
+static u32 fib_multipath_custom_hash_inner(const struct net *net,
+ const struct sk_buff *skb,
+ bool has_inner)
+{
+ u32 hash_fields = net->ipv4.sysctl_fib_multipath_hash_fields;
+ struct flow_keys keys, hash_keys;
+
+ /* We assume the packet carries an encapsulation, but if none was
+ * encountered during dissection of the outer flow, then there is no
+ * point in calling the flow dissector again.
+ */
+ if (!has_inner)
+ return 0;
+
+ if (!(hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_MASK))
+ return 0;
+
+ memset(&hash_keys, 0, sizeof(hash_keys));
+ skb_flow_dissect_flow_keys(skb, &keys, 0);
+
+ if (!(keys.control.flags & FLOW_DIS_ENCAPSULATION))
+ return 0;
+
+ if (keys.control.addr_type == FLOW_DISSECTOR_KEY_IPV4_ADDRS) {
+ hash_keys.control.addr_type = FLOW_DISSECTOR_KEY_IPV4_ADDRS;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_SRC_IP)
+ hash_keys.addrs.v4addrs.src = keys.addrs.v4addrs.src;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_DST_IP)
+ hash_keys.addrs.v4addrs.dst = keys.addrs.v4addrs.dst;
+ } else if (keys.control.addr_type == FLOW_DISSECTOR_KEY_IPV6_ADDRS) {
+ hash_keys.control.addr_type = FLOW_DISSECTOR_KEY_IPV6_ADDRS;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_SRC_IP)
+ hash_keys.addrs.v6addrs.src = keys.addrs.v6addrs.src;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_DST_IP)
+ hash_keys.addrs.v6addrs.dst = keys.addrs.v6addrs.dst;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_FLOWLABEL)
+ hash_keys.tags.flow_label = keys.tags.flow_label;
+ }
+
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_IP_PROTO)
+ hash_keys.basic.ip_proto = keys.basic.ip_proto;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_SRC_PORT)
+ hash_keys.ports.src = keys.ports.src;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_DST_PORT)
+ hash_keys.ports.dst = keys.ports.dst;
+
+ return flow_hash_from_keys(&hash_keys);
+}
+
+static u32 fib_multipath_custom_hash_skb(const struct net *net,
+ const struct sk_buff *skb)
+{
+ u32 mhash, mhash_inner;
+ bool has_inner = true;
+
+ mhash = fib_multipath_custom_hash_outer(net, skb, &has_inner);
+ mhash_inner = fib_multipath_custom_hash_inner(net, skb, has_inner);
+
+ return jhash_2words(mhash, mhash_inner, 0);
+}
+
+static u32 fib_multipath_custom_hash_fl4(const struct net *net,
+ const struct flowi4 *fl4)
+{
+ u32 hash_fields = net->ipv4.sysctl_fib_multipath_hash_fields;
+ struct flow_keys hash_keys;
+
+ if (!(hash_fields & FIB_MULTIPATH_HASH_FIELD_OUTER_MASK))
+ return 0;
+
+ memset(&hash_keys, 0, sizeof(hash_keys));
+ hash_keys.control.addr_type = FLOW_DISSECTOR_KEY_IPV4_ADDRS;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_SRC_IP)
+ hash_keys.addrs.v4addrs.src = fl4->saddr;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_DST_IP)
+ hash_keys.addrs.v4addrs.dst = fl4->daddr;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_IP_PROTO)
+ hash_keys.basic.ip_proto = fl4->flowi4_proto;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_SRC_PORT)
+ hash_keys.ports.src = fl4->fl4_sport;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_DST_PORT)
+ hash_keys.ports.dst = fl4->fl4_dport;
+
+ return flow_hash_from_keys(&hash_keys);
+}
+
/* if skb is set it will be used and fl4 can be NULL */
int fib_multipath_hash(const struct net *net, const struct flowi4 *fl4,
const struct sk_buff *skb, struct flow_keys *flkeys)
{
u32 multipath_hash = fl4 ? fl4->flowi4_multipath_hash : 0;
struct flow_keys hash_keys;
- u32 mhash;
+ u32 mhash = 0;
switch (net->ipv4.sysctl_fib_multipath_hash_policy) {
case 0:
hash_keys.addrs.v4addrs.src = fl4->saddr;
hash_keys.addrs.v4addrs.dst = fl4->daddr;
}
+ mhash = flow_hash_from_keys(&hash_keys);
break;
case 1:
/* skb is currently provided only when forwarding */
hash_keys.ports.dst = fl4->fl4_dport;
hash_keys.basic.ip_proto = fl4->flowi4_proto;
}
+ mhash = flow_hash_from_keys(&hash_keys);
break;
case 2:
memset(&hash_keys, 0, sizeof(hash_keys));
hash_keys.addrs.v4addrs.src = fl4->saddr;
hash_keys.addrs.v4addrs.dst = fl4->daddr;
}
+ mhash = flow_hash_from_keys(&hash_keys);
+ break;
+ case 3:
+ if (skb)
+ mhash = fib_multipath_custom_hash_skb(net, skb);
+ else
+ mhash = fib_multipath_custom_hash_fl4(net, fl4);
break;
}
- mhash = flow_hash_from_keys(&hash_keys);
if (multipath_hash)
mhash = jhash_2words(mhash, multipath_hash, 0);
return err;
}
+/* get device for dst_alloc with local routes */
+static struct net_device *ip_rt_get_dev(struct net *net,
+ const struct fib_result *res)
+{
+ struct fib_nh_common *nhc = res->fi ? res->nhc : NULL;
+ struct net_device *dev = NULL;
+
+ if (nhc)
+ dev = l3mdev_master_dev_rcu(nhc->nhc_dev);
+
+ return dev ? : net->loopback_dev;
+}
+
/*
* NOTE. We drop all the packets that has local source
* addresses, because every properly looped back packet
}
}
- rth = rt_dst_alloc(l3mdev_master_dev_rcu(dev) ? : net->loopback_dev,
+ rth = rt_dst_alloc(ip_rt_get_dev(net, res),
flags | RTCF_LOCAL, res->type,
IN_DEV_ORCONF(in_dev, NOPOLICY), false);
if (!rth)
#include <net/snmp.h>
#include <net/icmp.h>
#include <net/ip.h>
+#include <net/ip_fib.h>
#include <net/route.h>
#include <net/tcp.h>
#include <net/udp.h>
#include <net/netevent.h>
static int two = 2;
+static int three __maybe_unused = 3;
static int four = 4;
static int thousand = 1000;
static int tcp_retr1_max = 255;
static int ip_ping_group_range_max[] = { GID_T_MAX, GID_T_MAX };
static u32 u32_max_div_HZ = UINT_MAX / HZ;
static int one_day_secs = 24 * 3600;
+static u32 fib_multipath_hash_fields_all_mask __maybe_unused =
+ FIB_MULTIPATH_HASH_FIELD_ALL_MASK;
/* obsolete */
static int sysctl_tcp_low_latency __read_mostly;
return ret;
}
+
+static int proc_fib_multipath_hash_fields(struct ctl_table *table, int write,
+ void *buffer, size_t *lenp,
+ loff_t *ppos)
+{
+ struct net *net;
+ int ret;
+
+ net = container_of(table->data, struct net,
+ ipv4.sysctl_fib_multipath_hash_fields);
+ ret = proc_douintvec_minmax(table, write, buffer, lenp, ppos);
+ if (write && ret == 0)
+ call_netevent_notifiers(NETEVENT_IPV4_MPATH_HASH_UPDATE, net);
+
+ return ret;
+}
#endif
static struct ctl_table ipv4_table[] = {
},
#endif
{
+ .procname = "tcp_migrate_req",
+ .data = &init_net.ipv4.sysctl_tcp_migrate_req,
+ .maxlen = sizeof(u8),
+ .mode = 0644,
+ .proc_handler = proc_dou8vec_minmax,
+ .extra1 = SYSCTL_ZERO,
+ .extra2 = SYSCTL_ONE
+ },
+ {
.procname = "tcp_reordering",
.data = &init_net.ipv4.sysctl_tcp_reordering,
.maxlen = sizeof(int),
.mode = 0644,
.proc_handler = proc_fib_multipath_hash_policy,
.extra1 = SYSCTL_ZERO,
- .extra2 = &two,
+ .extra2 = &three,
+ },
+ {
+ .procname = "fib_multipath_hash_fields",
+ .data = &init_net.ipv4.sysctl_fib_multipath_hash_fields,
+ .maxlen = sizeof(u32),
+ .mode = 0644,
+ .proc_handler = proc_fib_multipath_hash_fields,
+ .extra1 = SYSCTL_ONE,
+ .extra2 = &fib_multipath_hash_fields_all_mask,
},
#endif
{
}
EXPORT_SYMBOL(tcp_set_rcvlowat);
-static void tcp_update_recv_tstamps(struct sk_buff *skb,
- struct scm_timestamping_internal *tss)
+void tcp_update_recv_tstamps(struct sk_buff *skb,
+ struct scm_timestamping_internal *tss)
{
if (skb->tstamp)
tss->ts[0] = ktime_to_timespec64(skb->tstamp);
}
#define TCP_VALID_ZC_MSG_FLAGS (TCP_CMSG_TS)
-static void tcp_recv_timestamp(struct msghdr *msg, const struct sock *sk,
- struct scm_timestamping_internal *tss);
static void tcp_zc_finalize_rx_tstamp(struct sock *sk,
struct tcp_zerocopy_receive *zc,
struct scm_timestamping_internal *tss)
#endif
/* Similar to __sock_recv_timestamp, but does not require an skb */
-static void tcp_recv_timestamp(struct msghdr *msg, const struct sock *sk,
- struct scm_timestamping_internal *tss)
+void tcp_recv_timestamp(struct msghdr *msg, const struct sock *sk,
+ struct scm_timestamping_internal *tss)
{
int new_tstamp = sock_flag(sk, SOCK_TSTAMP_NEW);
bool has_timestamping = false;
msg_bytes_ready:
copied = sk_msg_recvmsg(sk, psock, msg, len, flags);
if (!copied) {
- int data, err = 0;
long timeo;
+ int data;
timeo = sock_rcvtimeo(sk, nonblock);
- data = sk_msg_wait_data(sk, psock, flags, timeo, &err);
+ data = sk_msg_wait_data(sk, psock, timeo);
if (data) {
if (!sk_psock_queue_empty(psock))
goto msg_bytes_ready;
sk_psock_put(sk, psock);
return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
}
- if (err) {
- ret = err;
- goto out;
- }
copied = -EAGAIN;
}
ret = copied;
-out:
release_sock(sk);
sk_psock_put(sk, psock);
return ret;
if (!tfo_da_times)
return false;
- /* Limit timout to max: 2^6 * initial timeout */
+ /* Limit timeout to max: 2^6 * initial timeout */
multiplier = 1 << min(tfo_da_times - 1, 6);
timeout = multiplier * tfo_bh_timeout * HZ;
if (time_before(jiffies, sock_net(sk)->ipv4.tfo_active_disable_stamp + timeout))
*rexmit = REXMIT_LOST;
}
+static bool tcp_force_fast_retransmit(struct sock *sk)
+{
+ struct tcp_sock *tp = tcp_sk(sk);
+
+ return after(tcp_highest_sack_seq(tp),
+ tp->snd_una + tp->reordering * tp->mss_cache);
+}
+
/* Undo during fast recovery after partial ACK. */
-static bool tcp_try_undo_partial(struct sock *sk, u32 prior_snd_una)
+static bool tcp_try_undo_partial(struct sock *sk, u32 prior_snd_una,
+ bool *do_lost)
{
struct tcp_sock *tp = tcp_sk(sk);
tcp_undo_cwnd_reduction(sk, true);
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPPARTIALUNDO);
tcp_try_keep_open(sk);
- return true;
+ } else {
+ /* Partial ACK arrived. Force fast retransmit. */
+ *do_lost = tcp_force_fast_retransmit(sk);
}
return false;
}
}
}
-static bool tcp_force_fast_retransmit(struct sock *sk)
-{
- struct tcp_sock *tp = tcp_sk(sk);
-
- return after(tcp_highest_sack_seq(tp),
- tp->snd_una + tp->reordering * tp->mss_cache);
-}
-
/* Process an event, which can update packets-in-flight not trivially.
* Main goal of this function is to calculate new estimate for left_out,
* taking into account both packets sitting in receiver's buffer and
if (!(flag & FLAG_SND_UNA_ADVANCED)) {
if (tcp_is_reno(tp))
tcp_add_reno_sack(sk, num_dupack, ece_ack);
- } else {
- if (tcp_try_undo_partial(sk, prior_snd_una))
- return;
- /* Partial ACK arrived. Force fast retransmit. */
- do_lost = tcp_force_fast_retransmit(sk);
- }
- if (tcp_try_undo_dsack(sk)) {
- tcp_try_keep_open(sk);
+ } else if (tcp_try_undo_partial(sk, prior_snd_una, &do_lost))
return;
- }
+
+ if (tcp_try_undo_dsack(sk))
+ tcp_try_keep_open(sk);
+
tcp_identify_packet_loss(sk, ack_flag);
+ if (icsk->icsk_ca_state != TCP_CA_Recovery) {
+ if (!tcp_time_to_recover(sk, flag))
+ return;
+ /* Undo reverts the recovery state. If loss is evident,
+ * starts a new recovery (e.g. reordering then loss);
+ */
+ tcp_enter_recovery(sk, ece_ack);
+ }
break;
case TCP_CA_Loss:
tcp_process_loss(sk, flag, num_dupack, rexmit);
return;
csum_error:
+ trace_tcp_bad_csum(skb);
TCP_INC_STATS(sock_net(sk), TCP_MIB_CSUMERRORS);
TCP_INC_STATS(sock_net(sk), TCP_MIB_INERRS);
return 0;
csum_err:
+ trace_tcp_bad_csum(skb);
TCP_INC_STATS(sock_net(sk), TCP_MIB_CSUMERRORS);
TCP_INC_STATS(sock_net(sk), TCP_MIB_INERRS);
goto discard;
if (unlikely(tcp_checksum_complete(skb))) {
bh_unlock_sock(sk);
+ trace_tcp_bad_csum(skb);
__TCP_INC_STATS(sock_net(sk), TCP_MIB_CSUMERRORS);
__TCP_INC_STATS(sock_net(sk), TCP_MIB_INERRS);
return true;
goto csum_error;
}
if (unlikely(sk->sk_state != TCP_LISTEN)) {
- inet_csk_reqsk_queue_drop_and_put(sk, req);
- goto lookup;
+ nsk = reuseport_migrate_sock(sk, req_to_sk(req), skb);
+ if (!nsk) {
+ inet_csk_reqsk_queue_drop_and_put(sk, req);
+ goto lookup;
+ }
+ sk = nsk;
+ /* reuseport_migrate_sock() has already held one sk_refcnt
+ * before returning.
+ */
+ } else {
+ /* We own a reference on the listener, increase it again
+ * as we might lose it too soon.
+ */
+ sock_hold(sk);
}
- /* We own a reference on the listener, increase it again
- * as we might lose it too soon.
- */
- sock_hold(sk);
refcounted = true;
nsk = NULL;
if (!tcp_filter(sk, skb)) {
if (tcp_checksum_complete(skb)) {
csum_error:
+ trace_tcp_bad_csum(skb);
__TCP_INC_STATS(net, TCP_MIB_CSUMERRORS);
bad_packet:
__TCP_INC_STATS(net, TCP_MIB_INERRS);
goto listen_overflow;
if (own_req && rsk_drop_req(req)) {
- reqsk_queue_removed(&inet_csk(sk)->icsk_accept_queue, req);
- inet_csk_reqsk_queue_drop_and_put(sk, req);
+ reqsk_queue_removed(&inet_csk(req->rsk_listener)->icsk_accept_queue, req);
+ inet_csk_reqsk_queue_drop_and_put(req->rsk_listener, req);
return child;
}
return inet_csk_complete_hashdance(sk, child, req, own_req);
listen_overflow:
+ if (sk != req->rsk_listener)
+ __NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMIGRATEREQFAILURE);
+
if (!sock_net(sk)->ipv4.sysctl_tcp_abort_on_overflow) {
inet_rsk(req)->acked = 1;
return NULL;
* This function gets called when the kernel timer for a TCP packet
* of this socket expires.
*
- * It handles retransmission, timer adjustment and other necesarry measures.
+ * It handles retransmission, timer adjustment and other necessary measures.
*
* Returns: Nothing (void)
*/
if (!sock_owned_by_user(sk)) {
if (tp->compressed_ack) {
/* Since we have to send one ack finally,
- * substract one from tp->compressed_ack to keep
+ * subtract one from tp->compressed_ack to keep
* LINUX_MIB_TCPACKCOMPRESSED accurate.
*/
tp->compressed_ack--;
.handler = tunnel4_rcv,
.err_handler = tunnel4_err,
.no_policy = 1,
- .netns_ok = 1,
};
#if IS_ENABLED(CONFIG_IPV6)
.handler = tunnel64_rcv,
.err_handler = tunnel64_err,
.no_policy = 1,
- .netns_ok = 1,
};
#endif
.handler = tunnelmpls4_rcv,
.err_handler = tunnelmpls4_err,
.no_policy = 1,
- .netns_ok = 1,
};
#endif
{
struct udp_sock *up = udp_sk(sk);
bool slow = lock_sock_fast(sk);
+
+ /* protects from races with udp_abort() */
+ sock_set_flag(sk, SOCK_DEAD);
udp_flush_pending_frames(sk);
unlock_sock_fast(sk, slow);
if (static_branch_unlikely(&udp_encap_needed_key)) {
{
lock_sock(sk);
+ /* udp{v6}_destroy_sock() sets it under the sk lock, avoid racing
+ * with close()
+ */
+ if (sock_flag(sk, SOCK_DEAD))
+ goto out;
+
sk->sk_err = err;
sk->sk_error_report(sk);
__udp_disconnect(sk, 0);
+out:
release_sock(sk);
return 0;
msg_bytes_ready:
copied = sk_msg_recvmsg(sk, psock, msg, len, flags);
if (!copied) {
- int data, err = 0;
long timeo;
+ int data;
timeo = sock_rcvtimeo(sk, nonblock);
- data = sk_msg_wait_data(sk, psock, flags, timeo, &err);
+ data = sk_msg_wait_data(sk, psock, timeo);
if (data) {
if (!sk_psock_queue_empty(psock))
goto msg_bytes_ready;
ret = sk_udp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
goto out;
}
- if (err) {
- ret = err;
- goto out;
- }
copied = -EAGAIN;
}
ret = copied;
.handler = udplite_rcv,
.err_handler = udplite_err,
.no_policy = 1,
- .netns_ok = 1,
};
struct proto udplite_prot = {
.handler = xfrm4_esp_rcv,
.err_handler = xfrm4_esp_err,
.no_policy = 1,
- .netns_ok = 1,
};
static const struct net_protocol ah4_protocol = {
.handler = xfrm4_ah_rcv,
.err_handler = xfrm4_ah_err,
.no_policy = 1,
- .netns_ok = 1,
};
static const struct net_protocol ipcomp4_protocol = {
.handler = xfrm4_ipcomp_rcv,
.err_handler = xfrm4_ipcomp_err,
.no_policy = 1,
- .netns_ok = 1,
};
static const struct xfrm_input_afinfo xfrm4_input_afinfo = {
return -EAFNOSUPPORT;
if (nla_parse_nested_deprecated(tb, IFLA_INET6_MAX, nla, NULL, NULL) < 0)
- BUG();
+ return -EINVAL;
if (tb[IFLA_INET6_TOKEN]) {
err = inet6_set_iftoken(idev, nla_data(tb[IFLA_INET6_TOKEN]),
.proc_handler = proc_dointvec,
},
{
- .procname = "addr_gen_mode",
- .data = &ipv6_devconf.addr_gen_mode,
- .maxlen = sizeof(int),
- .mode = 0644,
+ .procname = "addr_gen_mode",
+ .data = &ipv6_devconf.addr_gen_mode,
+ .maxlen = sizeof(int),
+ .mode = 0644,
.proc_handler = addrconf_sysctl_addr_gen_mode,
},
{
static int __net_init fib6_rules_net_init(struct net *net)
{
struct fib_rules_ops *ops;
- int err = -ENOMEM;
+ int err;
ops = fib_rules_register(&fib6_rules_ops_template, net);
if (IS_ERR(ops))
#include <net/lwtunnel.h>
#include <net/fib_notifier.h>
+#include <net/ip_fib.h>
#include <net/ip6_fib.h>
#include <net/ip6_route.h>
if (err)
return err;
+ /* Default to 3-tuple */
+ net->ipv6.sysctl.multipath_hash_fields =
+ FIB_MULTIPATH_HASH_FIELD_DEFAULT_MASK;
+
spin_lock_init(&net->ipv6.fib6_gc_lock);
rwlock_init(&net->ipv6.fib6_walker_lock);
INIT_LIST_HEAD(&net->ipv6.fib6_walkers);
net->ipv6.rt6_stats = kzalloc(sizeof(*net->ipv6.rt6_stats), GFP_KERNEL);
if (!net->ipv6.rt6_stats)
- goto out_timer;
+ goto out_notifier;
/* Avoid false sharing : Use at least a full cache line */
size = max_t(size_t, size, L1_CACHE_BYTES);
kfree(net->ipv6.fib_table_hash);
out_rt6_stats:
kfree(net->ipv6.rt6_stats);
-out_timer:
+out_notifier:
fib6_notifier_exit(net);
return -ENOMEM;
}
* ip6_route_output will fail given src=any saddr, though, so
* that's why we try it again later.
*/
- if (ipv6_addr_any(&fl6->saddr) && (!*dst || !(*dst)->error)) {
+ if (ipv6_addr_any(&fl6->saddr)) {
struct fib6_info *from;
struct rt6_info *rt;
- bool had_dst = *dst != NULL;
- if (!had_dst)
- *dst = ip6_route_output(net, sk, fl6);
+ *dst = ip6_route_output(net, sk, fl6);
rt = (*dst)->error ? NULL : (struct rt6_info *)*dst;
rcu_read_lock();
* never existed and let the SA-enabled version take
* over.
*/
- if (!had_dst && (*dst)->error) {
+ if ((*dst)->error) {
dst_release(*dst);
*dst = NULL;
}
unsigned int datalen;
unsigned int fraglen;
unsigned int fraggap;
- unsigned int alloclen;
+ unsigned int alloclen, alloc_extra;
unsigned int pagedlen;
alloc_new_skb:
/* There's no room in the current skb */
fraglen = datalen + fragheaderlen;
pagedlen = 0;
+ alloc_extra = hh_len;
+ alloc_extra += dst_exthdrlen;
+ alloc_extra += rt->dst.trailer_len;
+
+ /* We just reserve space for fragment header.
+ * Note: this may be overallocation if the message
+ * (without MSG_MORE) fits into the MTU.
+ */
+ alloc_extra += sizeof(struct frag_hdr);
+
if ((flags & MSG_MORE) &&
!(rt->dst.dev->features&NETIF_F_SG))
alloclen = mtu;
- else if (!paged)
+ else if (!paged &&
+ (fraglen + alloc_extra < SKB_MAX_ALLOC ||
+ !(rt->dst.dev->features & NETIF_F_SG)))
alloclen = fraglen;
else {
alloclen = min_t(int, fraglen, MAX_HEADER);
pagedlen = fraglen - alloclen;
}
-
- alloclen += dst_exthdrlen;
+ alloclen += alloc_extra;
if (datalen != length + fraggap) {
/*
datalen += rt->dst.trailer_len;
}
- alloclen += rt->dst.trailer_len;
fraglen = datalen + fragheaderlen;
- /*
- * We just reserve space for fragment header.
- * Note: this may be overallocation if the message
- * (without MSG_MORE) fits into the MTU.
- */
- alloclen += sizeof(struct frag_hdr);
-
copy = datalen - transhdrlen - fraggap - pagedlen;
if (copy < 0) {
err = -EINVAL;
goto error;
}
if (transhdrlen) {
- skb = sock_alloc_send_skb(sk,
- alloclen + hh_len,
+ skb = sock_alloc_send_skb(sk, alloclen,
(flags & MSG_DONTWAIT), &err);
} else {
skb = NULL;
if (refcount_read(&sk->sk_wmem_alloc) + wmem_alloc_delta <=
2 * sk->sk_sndbuf)
- skb = alloc_skb(alloclen + hh_len,
+ skb = alloc_skb(alloclen,
sk->sk_allocation);
if (unlikely(!skb))
err = -ENOBUFS;
skb_postpull_rcsum(skb, eth_hdr(skb), ETH_HLEN);
} else {
skb->dev = tunnel->dev;
+ skb_reset_mac_header(skb);
}
skb_reset_network_header(skb);
static struct sk_buff *mld_newpack(struct inet6_dev *idev, unsigned int mtu)
{
+ u8 ra[8] = { IPPROTO_ICMPV6, 0, IPV6_TLV_ROUTERALERT,
+ 2, 0, 0, IPV6_TLV_PADN, 0 };
struct net_device *dev = idev->dev;
- struct net *net = dev_net(dev);
- struct sock *sk = net->ipv6.igmp_sk;
- struct sk_buff *skb;
- struct mld2_report *pmr;
- struct in6_addr addr_buf;
- const struct in6_addr *saddr;
int hlen = LL_RESERVED_SPACE(dev);
int tlen = dev->needed_tailroom;
- unsigned int size = mtu + hlen + tlen;
+ struct net *net = dev_net(dev);
+ const struct in6_addr *saddr;
+ struct in6_addr addr_buf;
+ struct mld2_report *pmr;
+ struct sk_buff *skb;
+ unsigned int size;
+ struct sock *sk;
int err;
- u8 ra[8] = { IPPROTO_ICMPV6, 0,
- IPV6_TLV_ROUTERALERT, 2, 0, 0,
- IPV6_TLV_PADN, 0 };
- /* we assume size > sizeof(ra) here */
- /* limit our allocations to order-0 page */
- size = min_t(int, size, SKB_MAX_ORDER(0, 0));
+ sk = net->ipv6.igmp_sk;
+ /* we assume size > sizeof(ra) here
+ * Also try to not allocate high-order pages for big MTU
+ */
+ size = min_t(int, mtu, PAGE_SIZE / 2) + hlen + tlen;
skb = sock_alloc_send_skb(sk, size, 1, &err);
-
if (!skb)
return NULL;
const char *outdev,
const struct ip6t_ip6 *ip6info,
unsigned int *protoff,
- int *fragoff, bool *hotdrop)
+ u16 *fragoff, bool *hotdrop)
{
unsigned long ret;
const struct ipv6hdr *ipv6 = ipv6_hdr(skb);
}
EXPORT_SYMBOL_GPL(nft_fib6_eval_type);
+static bool nft_fib_v6_skip_icmpv6(const struct sk_buff *skb, u8 next, const struct ipv6hdr *iph)
+{
+ if (likely(next != IPPROTO_ICMPV6))
+ return false;
+
+ if (ipv6_addr_type(&iph->saddr) != IPV6_ADDR_ANY)
+ return false;
+
+ return ipv6_addr_type(&iph->daddr) & IPV6_ADDR_LINKLOCAL;
+}
+
void nft_fib6_eval(const struct nft_expr *expr, struct nft_regs *regs,
const struct nft_pktinfo *pkt)
{
lookup_flags = nft_fib6_flowi_init(&fl6, priv, pkt, oif, iph);
- if (nft_hook(pkt) == NF_INET_PRE_ROUTING &&
- nft_fib_is_loopback(pkt->skb, nft_in(pkt))) {
- nft_fib_store_result(dest, priv, nft_in(pkt));
- return;
+ if (nft_hook(pkt) == NF_INET_PRE_ROUTING ||
+ nft_hook(pkt) == NF_INET_INGRESS) {
+ if (nft_fib_is_loopback(pkt->skb, nft_in(pkt)) ||
+ nft_fib_v6_skip_icmpv6(pkt->skb, pkt->tprot, iph)) {
+ nft_fib_store_result(dest, priv, nft_in(pkt));
+ return;
+ }
}
*dest = 0;
nft_hook(pkt));
break;
case NFT_REJECT_TCP_RST:
- nf_send_reset6(nft_net(pkt), pkt->xt.state->sk, pkt->skb,
+ nf_send_reset6(nft_net(pkt), nft_sk(pkt), pkt->skb,
nft_hook(pkt));
break;
default:
const struct in6_addr *dst,
const struct in6_addr *src)
{
- const struct {
- struct in6_addr dst;
- struct in6_addr src;
- } __aligned(SIPHASH_ALIGNMENT) combined = {
- .dst = *dst,
- .src = *src,
- };
- u32 hash, id;
-
- /* Note the following code is not safe, but this is okay. */
- if (unlikely(siphash_key_is_zero(&net->ipv4.ip_id_key)))
- get_random_bytes(&net->ipv4.ip_id_key,
- sizeof(net->ipv4.ip_id_key));
-
- hash = siphash(&combined, sizeof(combined), &net->ipv4.ip_id_key);
-
- /* Treat id of 0 as unset and if we get 0 back from ip_idents_reserve,
- * set the hight order instead thus minimizing possible future
- * collisions.
- */
- id = ip_idents_reserve(hash, 1);
- if (unlikely(!id))
- id = 1 << 31;
+ u32 id;
+
+ do {
+ id = prandom_u32();
+ } while (!id);
return id;
}
hdr = ipv6_hdr(skb);
fhdr = (struct frag_hdr *)skb_transport_header(skb);
- if (!(fhdr->frag_off & htons(0xFFF9))) {
+ if (!(fhdr->frag_off & htons(IP6_OFFSET | IP6_MF))) {
/* It is not a fragmented frame */
skb->transport_header += sizeof(struct frag_hdr);
__IP6_INC_STATS(net,
IP6CB(skb)->nhoff = (u8 *)fhdr - skb_network_header(skb);
IP6CB(skb)->flags |= IP6SKB_FRAGMENTED;
+ IP6CB(skb)->frag_max_size = ntohs(hdr->payload_len) +
+ sizeof(struct ipv6hdr);
return 1;
}
}
}
+static u32 rt6_multipath_custom_hash_outer(const struct net *net,
+ const struct sk_buff *skb,
+ bool *p_has_inner)
+{
+ u32 hash_fields = ip6_multipath_hash_fields(net);
+ struct flow_keys keys, hash_keys;
+
+ if (!(hash_fields & FIB_MULTIPATH_HASH_FIELD_OUTER_MASK))
+ return 0;
+
+ memset(&hash_keys, 0, sizeof(hash_keys));
+ skb_flow_dissect_flow_keys(skb, &keys, FLOW_DISSECTOR_F_STOP_AT_ENCAP);
+
+ hash_keys.control.addr_type = FLOW_DISSECTOR_KEY_IPV6_ADDRS;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_SRC_IP)
+ hash_keys.addrs.v6addrs.src = keys.addrs.v6addrs.src;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_DST_IP)
+ hash_keys.addrs.v6addrs.dst = keys.addrs.v6addrs.dst;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_IP_PROTO)
+ hash_keys.basic.ip_proto = keys.basic.ip_proto;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_FLOWLABEL)
+ hash_keys.tags.flow_label = keys.tags.flow_label;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_SRC_PORT)
+ hash_keys.ports.src = keys.ports.src;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_DST_PORT)
+ hash_keys.ports.dst = keys.ports.dst;
+
+ *p_has_inner = !!(keys.control.flags & FLOW_DIS_ENCAPSULATION);
+ return flow_hash_from_keys(&hash_keys);
+}
+
+static u32 rt6_multipath_custom_hash_inner(const struct net *net,
+ const struct sk_buff *skb,
+ bool has_inner)
+{
+ u32 hash_fields = ip6_multipath_hash_fields(net);
+ struct flow_keys keys, hash_keys;
+
+ /* We assume the packet carries an encapsulation, but if none was
+ * encountered during dissection of the outer flow, then there is no
+ * point in calling the flow dissector again.
+ */
+ if (!has_inner)
+ return 0;
+
+ if (!(hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_MASK))
+ return 0;
+
+ memset(&hash_keys, 0, sizeof(hash_keys));
+ skb_flow_dissect_flow_keys(skb, &keys, 0);
+
+ if (!(keys.control.flags & FLOW_DIS_ENCAPSULATION))
+ return 0;
+
+ if (keys.control.addr_type == FLOW_DISSECTOR_KEY_IPV4_ADDRS) {
+ hash_keys.control.addr_type = FLOW_DISSECTOR_KEY_IPV4_ADDRS;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_SRC_IP)
+ hash_keys.addrs.v4addrs.src = keys.addrs.v4addrs.src;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_DST_IP)
+ hash_keys.addrs.v4addrs.dst = keys.addrs.v4addrs.dst;
+ } else if (keys.control.addr_type == FLOW_DISSECTOR_KEY_IPV6_ADDRS) {
+ hash_keys.control.addr_type = FLOW_DISSECTOR_KEY_IPV6_ADDRS;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_SRC_IP)
+ hash_keys.addrs.v6addrs.src = keys.addrs.v6addrs.src;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_DST_IP)
+ hash_keys.addrs.v6addrs.dst = keys.addrs.v6addrs.dst;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_FLOWLABEL)
+ hash_keys.tags.flow_label = keys.tags.flow_label;
+ }
+
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_IP_PROTO)
+ hash_keys.basic.ip_proto = keys.basic.ip_proto;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_SRC_PORT)
+ hash_keys.ports.src = keys.ports.src;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_INNER_DST_PORT)
+ hash_keys.ports.dst = keys.ports.dst;
+
+ return flow_hash_from_keys(&hash_keys);
+}
+
+static u32 rt6_multipath_custom_hash_skb(const struct net *net,
+ const struct sk_buff *skb)
+{
+ u32 mhash, mhash_inner;
+ bool has_inner = true;
+
+ mhash = rt6_multipath_custom_hash_outer(net, skb, &has_inner);
+ mhash_inner = rt6_multipath_custom_hash_inner(net, skb, has_inner);
+
+ return jhash_2words(mhash, mhash_inner, 0);
+}
+
+static u32 rt6_multipath_custom_hash_fl6(const struct net *net,
+ const struct flowi6 *fl6)
+{
+ u32 hash_fields = ip6_multipath_hash_fields(net);
+ struct flow_keys hash_keys;
+
+ if (!(hash_fields & FIB_MULTIPATH_HASH_FIELD_OUTER_MASK))
+ return 0;
+
+ memset(&hash_keys, 0, sizeof(hash_keys));
+ hash_keys.control.addr_type = FLOW_DISSECTOR_KEY_IPV6_ADDRS;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_SRC_IP)
+ hash_keys.addrs.v6addrs.src = fl6->saddr;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_DST_IP)
+ hash_keys.addrs.v6addrs.dst = fl6->daddr;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_IP_PROTO)
+ hash_keys.basic.ip_proto = fl6->flowi6_proto;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_FLOWLABEL)
+ hash_keys.tags.flow_label = (__force u32)flowi6_get_flowlabel(fl6);
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_SRC_PORT)
+ hash_keys.ports.src = fl6->fl6_sport;
+ if (hash_fields & FIB_MULTIPATH_HASH_FIELD_DST_PORT)
+ hash_keys.ports.dst = fl6->fl6_dport;
+
+ return flow_hash_from_keys(&hash_keys);
+}
+
/* if skb is set it will be used and fl6 can be NULL */
u32 rt6_multipath_hash(const struct net *net, const struct flowi6 *fl6,
const struct sk_buff *skb, struct flow_keys *flkeys)
{
struct flow_keys hash_keys;
- u32 mhash;
+ u32 mhash = 0;
switch (ip6_multipath_hash_policy(net)) {
case 0:
hash_keys.tags.flow_label = (__force u32)flowi6_get_flowlabel(fl6);
hash_keys.basic.ip_proto = fl6->flowi6_proto;
}
+ mhash = flow_hash_from_keys(&hash_keys);
break;
case 1:
if (skb) {
hash_keys.ports.dst = fl6->fl6_dport;
hash_keys.basic.ip_proto = fl6->flowi6_proto;
}
+ mhash = flow_hash_from_keys(&hash_keys);
break;
case 2:
memset(&hash_keys, 0, sizeof(hash_keys));
hash_keys.tags.flow_label = (__force u32)flowi6_get_flowlabel(fl6);
hash_keys.basic.ip_proto = fl6->flowi6_proto;
}
+ mhash = flow_hash_from_keys(&hash_keys);
+ break;
+ case 3:
+ if (skb)
+ mhash = rt6_multipath_custom_hash_skb(net, skb);
+ else
+ mhash = rt6_multipath_custom_hash_fl6(net, fl6);
break;
}
- mhash = flow_hash_from_keys(&hash_keys);
return mhash >> 1;
}
if (nh) {
if (rt->fib6_src.plen) {
NL_SET_ERR_MSG(extack, "Nexthops can not be used with source routing");
- goto out;
+ goto out_free;
}
if (!nexthop_get(nh)) {
NL_SET_ERR_MSG(extack, "Nexthop has been deleted");
- goto out;
+ goto out_free;
}
rt->nh = nh;
fib6_nh = nexthop_fib6_nh(rt->nh);
out:
fib6_info_release(rt);
return ERR_PTR(err);
+out_free:
+ ip_fib_metrics_put(rt->fib6_metrics);
+ kfree(rt);
+ return ERR_PTR(err);
}
int ip6_route_add(struct fib6_config *cfg, gfp_t gfp_flags,
int vrf_ifindex;
int vrf_table;
- /* tunneled packet proto and family (IPv4 or IPv6) */
- __be16 proto;
+ /* tunneled packet family (IPv4 or IPv6).
+ * Protocol and header length are inferred from family.
+ */
u16 family;
- int hdrlen;
};
struct pcpu_seg6_local_counters {
info->net = net;
info->vrf_ifindex = vrf_ifindex;
- switch (family) {
- case AF_INET:
- info->proto = htons(ETH_P_IP);
- info->hdrlen = sizeof(struct iphdr);
- break;
- case AF_INET6:
- info->proto = htons(ETH_P_IPV6);
- info->hdrlen = sizeof(struct ipv6hdr);
- break;
- default:
- return -EINVAL;
- }
-
info->family = family;
info->mode = DT_VRF_MODE;
}
static struct sk_buff *end_dt_vrf_core(struct sk_buff *skb,
- struct seg6_local_lwt *slwt)
+ struct seg6_local_lwt *slwt, u16 family)
{
struct seg6_end_dt_info *info = &slwt->dt_info;
struct net_device *vrf;
+ __be16 protocol;
+ int hdrlen;
vrf = end_dt_get_vrf_rcu(skb, info);
if (unlikely(!vrf))
goto drop;
- skb->protocol = info->proto;
+ switch (family) {
+ case AF_INET:
+ protocol = htons(ETH_P_IP);
+ hdrlen = sizeof(struct iphdr);
+ break;
+ case AF_INET6:
+ protocol = htons(ETH_P_IPV6);
+ hdrlen = sizeof(struct ipv6hdr);
+ break;
+ case AF_UNSPEC:
+ fallthrough;
+ default:
+ goto drop;
+ }
+
+ if (unlikely(info->family != AF_UNSPEC && info->family != family)) {
+ pr_warn_once("seg6local: SRv6 End.DT* family mismatch");
+ goto drop;
+ }
+
+ skb->protocol = protocol;
skb_dst_drop(skb);
- skb_set_transport_header(skb, info->hdrlen);
+ skb_set_transport_header(skb, hdrlen);
- return end_dt_vrf_rcv(skb, info->family, vrf);
+ return end_dt_vrf_rcv(skb, family, vrf);
drop:
kfree_skb(skb);
if (!pskb_may_pull(skb, sizeof(struct iphdr)))
goto drop;
- skb = end_dt_vrf_core(skb, slwt);
+ skb = end_dt_vrf_core(skb, slwt, AF_INET);
if (!skb)
/* packet has been processed and consumed by the VRF */
return 0;
goto legacy_mode;
/* DT6_VRF_MODE */
- skb = end_dt_vrf_core(skb, slwt);
+ skb = end_dt_vrf_core(skb, slwt, AF_INET6);
if (!skb)
/* packet has been processed and consumed by the VRF */
return 0;
return -EINVAL;
}
+#ifdef CONFIG_NET_L3_MASTER_DEV
+static int seg6_end_dt46_build(struct seg6_local_lwt *slwt, const void *cfg,
+ struct netlink_ext_ack *extack)
+{
+ return __seg6_end_dt_vrf_build(slwt, cfg, AF_UNSPEC, extack);
+}
+
+static int input_action_end_dt46(struct sk_buff *skb,
+ struct seg6_local_lwt *slwt)
+{
+ unsigned int off = 0;
+ int nexthdr;
+
+ nexthdr = ipv6_find_hdr(skb, &off, -1, NULL, NULL);
+ if (unlikely(nexthdr < 0))
+ goto drop;
+
+ switch (nexthdr) {
+ case IPPROTO_IPIP:
+ return input_action_end_dt4(skb, slwt);
+ case IPPROTO_IPV6:
+ return input_action_end_dt6(skb, slwt);
+ }
+
+drop:
+ kfree_skb(skb);
+ return -EINVAL;
+}
+#endif
+
/* push an SRH on top of the current one */
static int input_action_end_b6(struct sk_buff *skb, struct seg6_local_lwt *slwt)
{
.input = input_action_end_dt6,
},
{
+ .action = SEG6_LOCAL_ACTION_END_DT46,
+ .attrs = SEG6_F_ATTR(SEG6_LOCAL_VRFTABLE),
+ .optattrs = SEG6_F_LOCAL_COUNTERS,
+#ifdef CONFIG_NET_L3_MASTER_DEV
+ .input = input_action_end_dt46,
+ .slwt_ops = {
+ .build_state = seg6_end_dt46_build,
+ },
+#endif
+ },
+ {
.action = SEG6_LOCAL_ACTION_END_B6,
.attrs = SEG6_F_ATTR(SEG6_LOCAL_SRH),
.optattrs = SEG6_F_LOCAL_COUNTERS,
if (ipip6_tunnel_create(dev) < 0)
goto failed_free;
+ if (!parms->name[0])
+ strcpy(parms->name, dev->name);
+
return nt;
failed_free:
* old iph is no longer valid
*/
iph = (const struct iphdr *)skb_mac_header(skb);
+ skb_reset_mac_header(skb);
+
err = IP_ECN_decapsulate(iph, skb);
if (unlikely(err)) {
if (log_ecn_error)
tpi = &ipip_tpi;
if (iptunnel_pull_header(skb, 0, tpi->proto, false))
goto drop;
+ skb_reset_mac_header(skb);
+
return ip_tunnel_rcv(tunnel, skb, tpi, NULL, log_ecn_error);
}
if (df) {
mtu = dst_mtu(&rt->dst) - t_hlen;
- if (mtu < 68) {
+ if (mtu < IPV4_MIN_MTU) {
dev->stats.collisions++;
ip_rt_put(rt);
goto tx_error;
#include <net/addrconf.h>
#include <net/inet_frag.h>
#include <net/netevent.h>
+#include <net/ip_fib.h>
#ifdef CONFIG_NETLABEL
#include <net/calipso.h>
#endif
static int two = 2;
+static int three = 3;
static int flowlabel_reflect_max = 0x7;
static int auto_flowlabels_max = IP6_AUTO_FLOW_LABEL_MAX;
+static u32 rt6_multipath_hash_fields_all_mask =
+ FIB_MULTIPATH_HASH_FIELD_ALL_MASK;
static int proc_rt6_multipath_hash_policy(struct ctl_table *table, int write,
void *buffer, size_t *lenp, loff_t *ppos)
return ret;
}
+static int
+proc_rt6_multipath_hash_fields(struct ctl_table *table, int write, void *buffer,
+ size_t *lenp, loff_t *ppos)
+{
+ struct net *net;
+ int ret;
+
+ net = container_of(table->data, struct net,
+ ipv6.sysctl.multipath_hash_fields);
+ ret = proc_douintvec_minmax(table, write, buffer, lenp, ppos);
+ if (write && ret == 0)
+ call_netevent_notifiers(NETEVENT_IPV6_MPATH_HASH_UPDATE, net);
+
+ return ret;
+}
+
static struct ctl_table ipv6_table_template[] = {
{
.procname = "bindv6only",
.mode = 0644,
.proc_handler = proc_rt6_multipath_hash_policy,
.extra1 = SYSCTL_ZERO,
- .extra2 = &two,
+ .extra2 = &three,
+ },
+ {
+ .procname = "fib_multipath_hash_fields",
+ .data = &init_net.ipv6.sysctl.multipath_hash_fields,
+ .maxlen = sizeof(u32),
+ .mode = 0644,
+ .proc_handler = proc_rt6_multipath_hash_fields,
+ .extra1 = SYSCTL_ONE,
+ .extra2 = &rt6_multipath_hash_fields_all_mask,
},
{
.procname = "seg6_flowlabel",
kfree_skb(skb);
return 0;
csum_err:
+ trace_tcp_bad_csum(skb);
TCP_INC_STATS(sock_net(sk), TCP_MIB_CSUMERRORS);
TCP_INC_STATS(sock_net(sk), TCP_MIB_INERRS);
goto discard;
goto csum_error;
}
if (unlikely(sk->sk_state != TCP_LISTEN)) {
- inet_csk_reqsk_queue_drop_and_put(sk, req);
- goto lookup;
+ nsk = reuseport_migrate_sock(sk, req_to_sk(req), skb);
+ if (!nsk) {
+ inet_csk_reqsk_queue_drop_and_put(sk, req);
+ goto lookup;
+ }
+ sk = nsk;
+ /* reuseport_migrate_sock() has already held one sk_refcnt
+ * before returning.
+ */
+ } else {
+ sock_hold(sk);
}
- sock_hold(sk);
refcounted = true;
nsk = NULL;
if (!tcp_filter(sk, skb)) {
if (tcp_checksum_complete(skb)) {
csum_error:
+ trace_tcp_bad_csum(skb);
__TCP_INC_STATS(net, TCP_MIB_CSUMERRORS);
bad_packet:
__TCP_INC_STATS(net, TCP_MIB_INERRS);
{
struct udp_sock *up = udp_sk(sk);
lock_sock(sk);
+
+ /* protects from races with udp_abort() */
+ sock_set_flag(sk, SOCK_DEAD);
udp_v6_flush_pending_frames(sk);
release_sock(sk);
};
static struct iucv_interface *pr_iucv;
+static struct iucv_handler af_iucv_handler;
/* special AF_IUCV IPRM messages */
static const u8 iprm_shutdown[8] =
static void afiucv_hs_callback_txnotify(struct sock *sk, enum iucv_tx_notify);
-/* Call Back functions */
-static void iucv_callback_rx(struct iucv_path *, struct iucv_message *);
-static void iucv_callback_txdone(struct iucv_path *, struct iucv_message *);
-static void iucv_callback_connack(struct iucv_path *, u8 *);
-static int iucv_callback_connreq(struct iucv_path *, u8 *, u8 *);
-static void iucv_callback_connrej(struct iucv_path *, u8 *);
-static void iucv_callback_shutdown(struct iucv_path *, u8 *);
-
static struct iucv_sock_list iucv_sk_list = {
.lock = __RW_LOCK_UNLOCKED(iucv_sk_list.lock),
.autobind_name = ATOMIC_INIT(0)
};
-static struct iucv_handler af_iucv_handler = {
- .path_pending = iucv_callback_connreq,
- .path_complete = iucv_callback_connack,
- .path_severed = iucv_callback_connrej,
- .message_pending = iucv_callback_rx,
- .message_complete = iucv_callback_txdone,
- .path_quiesced = iucv_callback_shutdown,
-};
-
static inline void high_nmcpy(unsigned char *dst, char *src)
{
memcpy(dst, src, 8);
bh_unlock_sock(sk);
}
+static struct iucv_handler af_iucv_handler = {
+ .path_pending = iucv_callback_connreq,
+ .path_complete = iucv_callback_connack,
+ .path_severed = iucv_callback_connrej,
+ .message_pending = iucv_callback_rx,
+ .message_complete = iucv_callback_txdone,
+ .path_quiesced = iucv_callback_shutdown,
+};
+
/***************** HiperSockets transport callbacks ********************/
static void afiucv_swap_src_dest(struct sk_buff *skb)
{
}
}
- /* We dont need to clone dst here, it is guaranteed to not disappear.
+ /* We don't need to clone dst here, it is guaranteed to not disappear.
* __dev_xmit_skb() might force a refcount if needed.
*/
skb_dst_set_noref(skb, &rt->dst);
static struct net_protocol l2tp_ip_protocol __read_mostly = {
.handler = l2tp_ip_recv,
- .netns_ok = 1,
};
static int __init l2tp_ip_init(void)
/* If the first two bytes are 0xFF03, consider that it is the PPP's
* Address and Control fields and skip them. The L2TP module has always
* worked this way, although, in theory, the use of these fields should
- * be negociated and handled at the PPP layer. These fields are
+ * be negotiated and handled at the PPP layer. These fields are
* constant: 0xFF is the All-Stations Address and 0x03 the Unnumbered
* Information command with Poll/Final bit set to zero (RFC 1662).
*/
static struct lapb_cb *__lapb_devtostruct(struct net_device *dev)
{
- struct list_head *entry;
struct lapb_cb *lapb, *use = NULL;
- list_for_each(entry, &lapb_list) {
- lapb = list_entry(entry, struct lapb_cb, node);
+ list_for_each_entry(lapb, &lapb_list, node) {
if (lapb->dev == dev) {
use = lapb;
break;
#endif
}
+static void sta_apply_airtime_params(struct ieee80211_local *local,
+ struct sta_info *sta,
+ struct station_parameters *params)
+{
+ u8 ac;
+
+ for (ac = 0; ac < IEEE80211_NUM_ACS; ac++) {
+ struct airtime_sched_info *air_sched = &local->airtime[ac];
+ struct airtime_info *air_info = &sta->airtime[ac];
+ struct txq_info *txqi;
+ u8 tid;
+
+ spin_lock_bh(&air_sched->lock);
+ for (tid = 0; tid < IEEE80211_NUM_TIDS + 1; tid++) {
+ if (air_info->weight == params->airtime_weight ||
+ !sta->sta.txq[tid] ||
+ ac != ieee80211_ac_from_tid(tid))
+ continue;
+
+ airtime_weight_set(air_info, params->airtime_weight);
+
+ txqi = to_txq_info(sta->sta.txq[tid]);
+ if (RB_EMPTY_NODE(&txqi->schedule_order))
+ continue;
+
+ ieee80211_update_airtime_weight(local, air_sched,
+ 0, true);
+ }
+ spin_unlock_bh(&air_sched->lock);
+ }
+}
+
static int sta_apply_parameters(struct ieee80211_local *local,
struct sta_info *sta,
struct station_parameters *params)
sta_apply_mesh_params(local, sta, params);
if (params->airtime_weight)
- sta->airtime_weight = params->airtime_weight;
+ sta_apply_airtime_params(local, sta, params);
+
/* set the STA state after all sta info from usermode has been set */
if (test_sta_flag(sta, WLAN_STA_TDLS_PEER) ||
test_sta_flag(sta, WLAN_STA_ASSOC))
rate_control_rate_init(sta);
- err = sta_info_insert_rcu(sta);
- if (err) {
- rcu_read_unlock();
- return err;
- }
-
- rcu_read_unlock();
-
- return 0;
+ return sta_info_insert(sta);
}
static int ieee80211_del_station(struct wiphy *wiphy, struct net_device *dev,
// SPDX-License-Identifier: GPL-2.0-only
/*
* mac80211 - channel management
+ * Copyright 2020 - 2021 Intel Corporation
*/
#include <linux/nl80211.h>
* the max of min required widths of all the interfaces bound to this
* channel context.
*/
-void ieee80211_recalc_chanctx_min_def(struct ieee80211_local *local,
- struct ieee80211_chanctx *ctx)
+static u32 _ieee80211_recalc_chanctx_min_def(struct ieee80211_local *local,
+ struct ieee80211_chanctx *ctx)
{
enum nl80211_chan_width max_bw;
struct cfg80211_chan_def min_def;
ctx->conf.def.width == NL80211_CHAN_WIDTH_16 ||
ctx->conf.radar_enabled) {
ctx->conf.min_def = ctx->conf.def;
- return;
+ return 0;
}
max_bw = ieee80211_get_chanctx_max_required_bw(local, &ctx->conf);
ieee80211_chandef_downgrade(&min_def);
if (cfg80211_chandef_identical(&ctx->conf.min_def, &min_def))
- return;
+ return 0;
ctx->conf.min_def = min_def;
if (!ctx->driver_present)
- return;
+ return 0;
- drv_change_chanctx(local, ctx, IEEE80211_CHANCTX_CHANGE_MIN_WIDTH);
+ return IEEE80211_CHANCTX_CHANGE_MIN_WIDTH;
}
+/* calling this function is assuming that station vif is updated to
+ * lates changes by calling ieee80211_vif_update_chandef
+ */
static void ieee80211_chan_bw_change(struct ieee80211_local *local,
- struct ieee80211_chanctx *ctx)
+ struct ieee80211_chanctx *ctx,
+ bool narrowed)
{
struct sta_info *sta;
struct ieee80211_supported_band *sband =
continue;
new_sta_bw = ieee80211_sta_cur_vht_bw(sta);
+
+ /* nothing change */
if (new_sta_bw == sta->sta.bandwidth)
continue;
+ /* vif changed to narrow BW and narrow BW for station wasn't
+ * requested or vise versa */
+ if ((new_sta_bw < sta->sta.bandwidth) == !narrowed)
+ continue;
+
sta->sta.bandwidth = new_sta_bw;
rate_control_rate_update(local, sband, sta,
IEEE80211_RC_BW_CHANGED);
rcu_read_unlock();
}
-static void ieee80211_change_chanctx(struct ieee80211_local *local,
- struct ieee80211_chanctx *ctx,
- const struct cfg80211_chan_def *chandef)
+/*
+ * recalc the min required chan width of the channel context, which is
+ * the max of min required widths of all the interfaces bound to this
+ * channel context.
+ */
+void ieee80211_recalc_chanctx_min_def(struct ieee80211_local *local,
+ struct ieee80211_chanctx *ctx)
{
- enum nl80211_chan_width width;
+ u32 changed = _ieee80211_recalc_chanctx_min_def(local, ctx);
- if (cfg80211_chandef_identical(&ctx->conf.def, chandef)) {
- ieee80211_recalc_chanctx_min_def(local, ctx);
+ if (!changed)
return;
- }
- WARN_ON(!cfg80211_chandef_compatible(&ctx->conf.def, chandef));
+ /* check is BW narrowed */
+ ieee80211_chan_bw_change(local, ctx, true);
- width = ctx->conf.def.width;
- ctx->conf.def = *chandef;
+ drv_change_chanctx(local, ctx, changed);
+
+ /* check is BW wider */
+ ieee80211_chan_bw_change(local, ctx, false);
+}
+
+static void ieee80211_change_chanctx(struct ieee80211_local *local,
+ struct ieee80211_chanctx *ctx,
+ struct ieee80211_chanctx *old_ctx,
+ const struct cfg80211_chan_def *chandef)
+{
+ u32 changed;
/* expected to handle only 20/40/80/160 channel widths */
switch (chandef->width) {
WARN_ON(1);
}
- if (chandef->width < width)
- ieee80211_chan_bw_change(local, ctx);
+ /* Check maybe BW narrowed - we do this _before_ calling recalc_chanctx_min_def
+ * due to maybe not returning from it, e.g in case new context was added
+ * first time with all parameters up to date.
+ */
+ ieee80211_chan_bw_change(local, old_ctx, true);
+
+ if (cfg80211_chandef_identical(&ctx->conf.def, chandef)) {
+ ieee80211_recalc_chanctx_min_def(local, ctx);
+ return;
+ }
- drv_change_chanctx(local, ctx, IEEE80211_CHANCTX_CHANGE_WIDTH);
- ieee80211_recalc_chanctx_min_def(local, ctx);
+ WARN_ON(!cfg80211_chandef_compatible(&ctx->conf.def, chandef));
+
+ ctx->conf.def = *chandef;
+
+ /* check if min chanctx also changed */
+ changed = IEEE80211_CHANCTX_CHANGE_WIDTH |
+ _ieee80211_recalc_chanctx_min_def(local, ctx);
+ drv_change_chanctx(local, ctx, changed);
if (!local->use_chanctx) {
local->_oper_chandef = *chandef;
ieee80211_hw_config(local, 0);
}
- if (chandef->width > width)
- ieee80211_chan_bw_change(local, ctx);
+ /* check is BW wider */
+ ieee80211_chan_bw_change(local, old_ctx, false);
}
static struct ieee80211_chanctx *
if (!compat)
continue;
- ieee80211_change_chanctx(local, ctx, compat);
+ ieee80211_change_chanctx(local, ctx, ctx, compat);
return ctx;
}
if (!compat)
return;
- ieee80211_change_chanctx(local, ctx, compat);
+ ieee80211_change_chanctx(local, ctx, ctx, compat);
}
static void ieee80211_recalc_radar_chanctx(struct ieee80211_local *local,
if (WARN_ON(!chandef))
return -EINVAL;
- if (old_ctx->conf.def.width > new_ctx->conf.def.width)
- ieee80211_chan_bw_change(local, new_ctx);
+ if (sdata->vif.bss_conf.chandef.width != sdata->reserved_chandef.width)
+ changed = BSS_CHANGED_BANDWIDTH;
- ieee80211_change_chanctx(local, new_ctx, chandef);
+ ieee80211_vif_update_chandef(sdata, &sdata->reserved_chandef);
- if (old_ctx->conf.def.width < new_ctx->conf.def.width)
- ieee80211_chan_bw_change(local, new_ctx);
+ ieee80211_change_chanctx(local, new_ctx, old_ctx, chandef);
vif_chsw[0].vif = &sdata->vif;
vif_chsw[0].old_ctx = &old_ctx->conf;
if (ieee80211_chanctx_refcount(local, old_ctx) == 0)
ieee80211_free_chanctx(local, old_ctx);
- if (sdata->vif.bss_conf.chandef.width != sdata->reserved_chandef.width)
- changed = BSS_CHANGED_BANDWIDTH;
-
- ieee80211_vif_update_chandef(sdata, &sdata->reserved_chandef);
-
+ ieee80211_recalc_chanctx_min_def(local, new_ctx);
ieee80211_recalc_smps_chanctx(local, new_ctx);
ieee80211_recalc_radar_chanctx(local, new_ctx);
- ieee80211_recalc_chanctx_min_def(local, new_ctx);
if (changed)
ieee80211_bss_info_change_notify(sdata, changed);
if (WARN_ON(!chandef))
return -EINVAL;
- ieee80211_change_chanctx(local, new_ctx, chandef);
+ ieee80211_change_chanctx(local, new_ctx, new_ctx, chandef);
list_del(&sdata->reserved_chanctx_list);
sdata->reserved_chanctx = NULL;
ieee80211_recalc_smps_chanctx(local, ctx);
ieee80211_recalc_radar_chanctx(local, ctx);
ieee80211_recalc_chanctx_min_def(local, ctx);
- ieee80211_chan_bw_change(local, ctx);
list_for_each_entry_safe(sdata, sdata_tmp, &ctx->reserved_vifs,
reserved_chanctx_list) {
*
* Copyright 2007 Johannes Berg <johannes@sipsolutions.net>
* Copyright 2013-2014 Intel Mobile Communications GmbH
- * Copyright (C) 2018 - 2019 Intel Corporation
+ * Copyright (C) 2018 - 2019, 2021 Intel Corporation
*/
#include <linux/debugfs.h>
"VI %u %u\n"
"BE %u %u\n"
"BK %u %u\n",
- local->aql_txq_limit_low[IEEE80211_AC_VO],
- local->aql_txq_limit_high[IEEE80211_AC_VO],
- local->aql_txq_limit_low[IEEE80211_AC_VI],
- local->aql_txq_limit_high[IEEE80211_AC_VI],
- local->aql_txq_limit_low[IEEE80211_AC_BE],
- local->aql_txq_limit_high[IEEE80211_AC_BE],
- local->aql_txq_limit_low[IEEE80211_AC_BK],
- local->aql_txq_limit_high[IEEE80211_AC_BK]);
+ local->airtime[IEEE80211_AC_VO].aql_txq_limit_low,
+ local->airtime[IEEE80211_AC_VO].aql_txq_limit_high,
+ local->airtime[IEEE80211_AC_VI].aql_txq_limit_low,
+ local->airtime[IEEE80211_AC_VI].aql_txq_limit_high,
+ local->airtime[IEEE80211_AC_BE].aql_txq_limit_low,
+ local->airtime[IEEE80211_AC_BE].aql_txq_limit_high,
+ local->airtime[IEEE80211_AC_BK].aql_txq_limit_low,
+ local->airtime[IEEE80211_AC_BK].aql_txq_limit_high);
return simple_read_from_buffer(user_buf, count, ppos,
buf, len);
}
if (ac >= IEEE80211_NUM_ACS)
return -EINVAL;
- q_limit_low_old = local->aql_txq_limit_low[ac];
- q_limit_high_old = local->aql_txq_limit_high[ac];
+ q_limit_low_old = local->airtime[ac].aql_txq_limit_low;
+ q_limit_high_old = local->airtime[ac].aql_txq_limit_high;
- local->aql_txq_limit_low[ac] = q_limit_low;
- local->aql_txq_limit_high[ac] = q_limit_high;
+ local->airtime[ac].aql_txq_limit_low = q_limit_low;
+ local->airtime[ac].aql_txq_limit_high = q_limit_high;
mutex_lock(&local->sta_mtx);
list_for_each_entry(sta, &local->sta_list, list) {
.llseek = default_llseek,
};
+static ssize_t airtime_read(struct file *file,
+ char __user *user_buf,
+ size_t count,
+ loff_t *ppos)
+{
+ struct ieee80211_local *local = file->private_data;
+ char buf[200];
+ u64 v_t[IEEE80211_NUM_ACS];
+ u64 wt[IEEE80211_NUM_ACS];
+ int len = 0, ac;
+
+ for (ac = 0; ac < IEEE80211_NUM_ACS; ac++) {
+ spin_lock_bh(&local->airtime[ac].lock);
+ v_t[ac] = local->airtime[ac].v_t;
+ wt[ac] = local->airtime[ac].weight_sum;
+ spin_unlock_bh(&local->airtime[ac].lock);
+ }
+ len = scnprintf(buf, sizeof(buf),
+ "\tVO VI BE BK\n"
+ "Virt-t\t%-10llu %-10llu %-10llu %-10llu\n"
+ "Weight\t%-10llu %-10llu %-10llu %-10llu\n",
+ v_t[0],
+ v_t[1],
+ v_t[2],
+ v_t[3],
+ wt[0],
+ wt[1],
+ wt[2],
+ wt[3]);
+
+ return simple_read_from_buffer(user_buf, count, ppos,
+ buf, len);
+}
+
+static const struct file_operations airtime_ops = {
+ .read = airtime_read,
+ .open = simple_open,
+ .llseek = default_llseek,
+};
+
#ifdef CONFIG_PM
static ssize_t reset_write(struct file *file, const char __user *user_buf,
size_t count, loff_t *ppos)
{
struct ieee80211_local *local = file->private_data;
+ int ret;
rtnl_lock();
+ wiphy_lock(local->hw.wiphy);
__ieee80211_suspend(&local->hw, NULL);
- __ieee80211_resume(&local->hw);
+ ret = __ieee80211_resume(&local->hw);
+ wiphy_unlock(local->hw.wiphy);
+
+ if (ret)
+ cfg80211_shutdown_all_interfaces(local->hw.wiphy);
+
rtnl_unlock();
return count;
if (local->ops->wake_tx_queue)
DEBUGFS_ADD_MODE(aqm, 0600);
- DEBUGFS_ADD_MODE(airtime_flags, 0600);
+ if (wiphy_ext_feature_isset(local->hw.wiphy,
+ NL80211_EXT_FEATURE_AIRTIME_FAIRNESS)) {
+ DEBUGFS_ADD_MODE(airtime, 0600);
+ DEBUGFS_ADD_MODE(airtime_flags, 0600);
+ }
DEBUGFS_ADD(aql_txq_limit);
debugfs_create_u32("aql_threshold", 0600,
return -EFAULT;
buf[count] = '\0';
- ret = -ENODEV;
rtnl_lock();
ret = (*write)(sdata, buf, count);
rtnl_unlock();
}
IEEE80211_IF_FILE_R(aqm);
+static ssize_t ieee80211_if_fmt_airtime(
+ const struct ieee80211_sub_if_data *sdata, char *buf, int buflen)
+{
+ struct ieee80211_local *local = sdata->local;
+ struct ieee80211_txq *txq = sdata->vif.txq;
+ struct airtime_info *air_info;
+ int len;
+
+ if (!txq)
+ return 0;
+
+ spin_lock_bh(&local->airtime[txq->ac].lock);
+ air_info = to_airtime_info(txq);
+ len = scnprintf(buf,
+ buflen,
+ "RX: %llu us\nTX: %llu us\nWeight: %u\n"
+ "Virt-T: %lld us\n",
+ air_info->rx_airtime,
+ air_info->tx_airtime,
+ air_info->weight,
+ air_info->v_t);
+ spin_unlock_bh(&local->airtime[txq->ac].lock);
+
+ return len;
+}
+
+IEEE80211_IF_FILE_R(airtime);
+
IEEE80211_IF_FILE(multicast_to_unicast, u.ap.multicast_to_unicast, HEX);
/* IBSS attributes */
if (sdata->local->ops->wake_tx_queue &&
sdata->vif.type != NL80211_IFTYPE_P2P_DEVICE &&
- sdata->vif.type != NL80211_IFTYPE_NAN)
+ sdata->vif.type != NL80211_IFTYPE_NAN) {
DEBUGFS_ADD(aqm);
+ DEBUGFS_ADD(airtime);
+ }
}
static void add_sta_files(struct ieee80211_sub_if_data *sdata)
size_t bufsz = 400;
char *buf = kzalloc(bufsz, GFP_KERNEL), *p = buf;
u64 rx_airtime = 0, tx_airtime = 0;
- s64 deficit[IEEE80211_NUM_ACS];
+ u64 v_t[IEEE80211_NUM_ACS];
ssize_t rv;
int ac;
return -ENOMEM;
for (ac = 0; ac < IEEE80211_NUM_ACS; ac++) {
- spin_lock_bh(&local->active_txq_lock[ac]);
+ spin_lock_bh(&local->airtime[ac].lock);
rx_airtime += sta->airtime[ac].rx_airtime;
tx_airtime += sta->airtime[ac].tx_airtime;
- deficit[ac] = sta->airtime[ac].deficit;
- spin_unlock_bh(&local->active_txq_lock[ac]);
+ v_t[ac] = sta->airtime[ac].v_t;
+ spin_unlock_bh(&local->airtime[ac].lock);
}
p += scnprintf(p, bufsz + buf - p,
"RX: %llu us\nTX: %llu us\nWeight: %u\n"
- "Deficit: VO: %lld us VI: %lld us BE: %lld us BK: %lld us\n",
- rx_airtime, tx_airtime, sta->airtime_weight,
- deficit[0], deficit[1], deficit[2], deficit[3]);
+ "Virt-T: VO: %lld us VI: %lld us BE: %lld us BK: %lld us\n",
+ rx_airtime, tx_airtime, sta->airtime[0].weight,
+ v_t[0], v_t[1], v_t[2], v_t[3]);
rv = simple_read_from_buffer(userbuf, count, ppos, buf, p - buf);
kfree(buf);
int ac;
for (ac = 0; ac < IEEE80211_NUM_ACS; ac++) {
- spin_lock_bh(&local->active_txq_lock[ac]);
+ spin_lock_bh(&local->airtime[ac].lock);
sta->airtime[ac].rx_airtime = 0;
sta->airtime[ac].tx_airtime = 0;
- sta->airtime[ac].deficit = sta->airtime_weight;
- spin_unlock_bh(&local->active_txq_lock[ac]);
+ sta->airtime[ac].v_t = 0;
+ spin_unlock_bh(&local->airtime[ac].lock);
}
return count;
return -ENOMEM;
for (ac = 0; ac < IEEE80211_NUM_ACS; ac++) {
- spin_lock_bh(&local->active_txq_lock[ac]);
+ spin_lock_bh(&local->airtime[ac].lock);
q_limit_l[ac] = sta->airtime[ac].aql_limit_low;
q_limit_h[ac] = sta->airtime[ac].aql_limit_high;
- spin_unlock_bh(&local->active_txq_lock[ac]);
+ spin_unlock_bh(&local->airtime[ac].lock);
q_depth[ac] = atomic_read(&sta->airtime[ac].aql_tx_pending);
}
/*
* Portions of this file
* Copyright(c) 2016 Intel Deutschland GmbH
-* Copyright (C) 2018 - 2019 Intel Corporation
+* Copyright (C) 2018 - 2019, 2021 Intel Corporation
*/
#ifndef __MAC80211_DRIVER_OPS
static inline void drv_mgd_prepare_tx(struct ieee80211_local *local,
struct ieee80211_sub_if_data *sdata,
- u16 duration)
+ struct ieee80211_prep_tx_info *info)
{
might_sleep();
return;
WARN_ON_ONCE(sdata->vif.type != NL80211_IFTYPE_STATION);
- trace_drv_mgd_prepare_tx(local, sdata, duration);
+ trace_drv_mgd_prepare_tx(local, sdata, info->duration,
+ info->subtype, info->success);
if (local->ops->mgd_prepare_tx)
- local->ops->mgd_prepare_tx(&local->hw, &sdata->vif, duration);
+ local->ops->mgd_prepare_tx(&local->hw, &sdata->vif, info);
+ trace_drv_return_void(local);
+}
+
+static inline void drv_mgd_complete_tx(struct ieee80211_local *local,
+ struct ieee80211_sub_if_data *sdata,
+ struct ieee80211_prep_tx_info *info)
+{
+ might_sleep();
+
+ if (!check_sdata_in_driver(sdata))
+ return;
+ WARN_ON_ONCE(sdata->vif.type != NL80211_IFTYPE_STATION);
+
+ trace_drv_mgd_complete_tx(local, sdata, info->duration,
+ info->subtype, info->success);
+ if (local->ops->mgd_complete_tx)
+ local->ops->mgd_complete_tx(&local->hw, &sdata->vif, info);
trace_drv_return_void(local);
}
struct sta_info *sta)
{
struct ieee80211_sta_he_cap *he_cap = &sta->sta.he_cap;
- struct ieee80211_sta_he_cap own_he_cap = sband->iftype_data->he_cap;
+ struct ieee80211_sta_he_cap own_he_cap;
struct ieee80211_he_cap_elem *he_cap_ie_elem = (void *)he_cap_ie;
u8 he_ppe_size;
u8 mcs_nss_size;
memset(he_cap, 0, sizeof(*he_cap));
- if (!he_cap_ie || !ieee80211_get_he_sta_cap(sband))
+ if (!he_cap_ie ||
+ !ieee80211_get_he_iftype_cap(sband,
+ ieee80211_vif_type_p2p(&sdata->vif)))
return;
+ own_he_cap = sband->iftype_data->he_cap;
+
/* Make sure size is OK */
mcs_nss_size = ieee80211_he_mcs_nss_size(he_cap_ie_elem);
he_ppe_size =
* Copyright 2007, Michael Wu <flamingice@sourmilk.net>
* Copyright 2007-2010, Intel Corporation
* Copyright 2017 Intel Deutschland GmbH
- * Copyright(c) 2020 Intel Corporation
+ * Copyright(c) 2020-2021 Intel Corporation
*/
#include <linux/ieee80211.h>
{
struct ieee80211_sub_if_data *sdata = vif_to_sdata(vif);
- if (WARN_ON_ONCE(vif->type != NL80211_IFTYPE_STATION &&
- vif->type != NL80211_IFTYPE_AP))
+ if (WARN_ON_ONCE(vif->type != NL80211_IFTYPE_STATION))
return;
- if (vif->type == NL80211_IFTYPE_STATION) {
- if (sdata->u.mgd.driver_smps_mode == smps_mode)
- return;
- sdata->u.mgd.driver_smps_mode = smps_mode;
- ieee80211_queue_work(&sdata->local->hw,
- &sdata->u.mgd.request_smps_work);
- }
+ if (sdata->u.mgd.driver_smps_mode == smps_mode)
+ return;
+
+ sdata->u.mgd.driver_smps_mode = smps_mode;
+ ieee80211_queue_work(&sdata->local->hw,
+ &sdata->u.mgd.request_smps_work);
}
/* this might change ... don't want non-open drivers using it */
EXPORT_SYMBOL_GPL(ieee80211_request_smps);
* Copyright 2006-2007 Jiri Benc <jbenc@suse.cz>
* Copyright 2007-2010 Johannes Berg <johannes@sipsolutions.net>
* Copyright 2013-2015 Intel Mobile Communications GmbH
- * Copyright (C) 2018-2020 Intel Corporation
+ * Copyright (C) 2018-2021 Intel Corporation
*/
#ifndef IEEE80211_I_H
#define IEEE80211_ENCRYPT_HEADROOM 8
#define IEEE80211_ENCRYPT_TAILROOM 18
-/* IEEE 802.11 (Ch. 9.5 Defragmentation) requires support for concurrent
- * reception of at least three fragmented frames. This limit can be increased
- * by changing this define, at the cost of slower frame reassembly and
- * increased memory use (about 2 kB of RAM per entry). */
-#define IEEE80211_FRAGMENT_MAX 4
-
/* power level hasn't been configured (or set to automatic) */
#define IEEE80211_UNSET_POWER_LEVEL INT_MIN
#define IEEE80211_MAX_NAN_INSTANCE_ID 255
-struct ieee80211_fragment_entry {
- struct sk_buff_head skb_list;
- unsigned long first_frag_time;
- u16 seq;
- u16 extra_len;
- u16 last_frag;
- u8 rx_queue;
- bool check_sequential_pn; /* needed for CCMP/GCMP */
- u8 last_pn[6]; /* PN of the last fragment if CCMP was used */
-};
-
-
struct ieee80211_bss {
u32 device_ts_beacon, device_ts_presp;
*/
int security_idx;
- u32 tkip_iv32;
- u16 tkip_iv16;
+ union {
+ struct {
+ u32 iv32;
+ u16 iv16;
+ } tkip;
+ struct {
+ u8 pn[IEEE80211_CCMP_PN_LEN];
+ } ccm_gcm;
+ };
};
struct ieee80211_csa_settings {
* @def_flow: used as a fallback flow when a packet destined to @tin hashes to
* a fq_flow which is already owned by a different tin
* @def_cvars: codel vars for @def_flow
- * @frags: used to keep fragments created after dequeue
* @schedule_order: used with ieee80211_local->active_txqs
- * @schedule_round: counter to prevent infinite loops on TXQ scheduling
+ * @frags: used to keep fragments created after dequeue
*/
struct txq_info {
struct fq_tin tin;
struct codel_vars def_cvars;
struct codel_stats cstats;
+ struct rb_node schedule_order;
+
struct sk_buff_head frags;
- struct list_head schedule_order;
- u16 schedule_round;
unsigned long flags;
/* keep last! */
char name[IFNAMSIZ];
- /* Fragment table for host-based reassembly */
- struct ieee80211_fragment_entry fragments[IEEE80211_FRAGMENT_MAX];
- unsigned int fragment_next;
+ struct ieee80211_fragment_cache frags;
/* TID bitmap for NoAck policy */
u16 noack_map;
struct ieee80211_tx_queue_params tx_conf[IEEE80211_NUM_ACS];
struct mac80211_qos_map __rcu *qos_map;
+ struct airtime_info airtime[IEEE80211_NUM_ACS];
+
struct work_struct csa_finalize_work;
bool csa_block_tx; /* write-protected by sdata_lock and local->mtx */
struct cfg80211_chan_def csa_chandef;
SCAN_ABORT,
};
+/**
+ * struct airtime_sched_info - state used for airtime scheduling and AQL
+ *
+ * @lock: spinlock that protects all the fields in this struct
+ * @active_txqs: rbtree of currently backlogged queues, sorted by virtual time
+ * @schedule_pos: the current position maintained while a driver walks the tree
+ * with ieee80211_next_txq()
+ * @active_list: list of struct airtime_info structs that were active within
+ * the last AIRTIME_ACTIVE_DURATION (100 ms), used to compute
+ * weight_sum
+ * @last_weight_update: used for rate limiting walking active_list
+ * @last_schedule_time: tracks the last time a transmission was scheduled; used
+ * for catching up v_t if no stations are eligible for
+ * transmission.
+ * @v_t: global virtual time; queues with v_t < this are eligible for
+ * transmission
+ * @weight_sum: total sum of all active stations used for dividing airtime
+ * @weight_sum_reciprocal: reciprocal of weight_sum (to avoid divisions in fast
+ * path - see comment above
+ * IEEE80211_RECIPROCAL_DIVISOR_64)
+ * @aql_txq_limit_low: AQL limit when total outstanding airtime
+ * is < IEEE80211_AQL_THRESHOLD
+ * @aql_txq_limit_high: AQL limit when total outstanding airtime
+ * is > IEEE80211_AQL_THRESHOLD
+ */
+struct airtime_sched_info {
+ spinlock_t lock;
+ struct rb_root_cached active_txqs;
+ struct rb_node *schedule_pos;
+ struct list_head active_list;
+ u64 last_weight_update;
+ u64 last_schedule_activity;
+ u64 v_t;
+ u64 weight_sum;
+ u64 weight_sum_reciprocal;
+ u32 aql_txq_limit_low;
+ u32 aql_txq_limit_high;
+};
DECLARE_STATIC_KEY_FALSE(aql_disable);
struct ieee80211_local {
struct codel_params cparams;
/* protects active_txqs and txqi->schedule_order */
- spinlock_t active_txq_lock[IEEE80211_NUM_ACS];
- struct list_head active_txqs[IEEE80211_NUM_ACS];
- u16 schedule_round[IEEE80211_NUM_ACS];
-
+ struct airtime_sched_info airtime[IEEE80211_NUM_ACS];
u16 airtime_flags;
- u32 aql_txq_limit_low[IEEE80211_NUM_ACS];
- u32 aql_txq_limit_high[IEEE80211_NUM_ACS];
u32 aql_threshold;
atomic_t aql_total_pending_airtime;
/* extended capabilities provided by mac80211 */
u8 ext_capa[8];
-
- /* TDLS channel switch */
- struct work_struct tdls_chsw_work;
- struct sk_buff_head skb_queue_tdls_chsw;
};
static inline struct ieee80211_sub_if_data *
rcu_read_lock();
chanctx_conf = rcu_dereference(sdata->vif.chanctx_conf);
- if (WARN_ON_ONCE(!chanctx_conf)) {
+ if (!chanctx_conf) {
rcu_read_unlock();
return NULL;
}
return !(skb_queue_empty(&txqi->frags) && !txqi->tin.backlog_packets);
}
+static inline struct airtime_info *to_airtime_info(struct ieee80211_txq *txq)
+{
+ struct ieee80211_sub_if_data *sdata;
+ struct sta_info *sta;
+
+ if (txq->sta) {
+ sta = container_of(txq->sta, struct sta_info, sta);
+ return &sta->airtime[txq->ac];
+ }
+
+ sdata = vif_to_sdata(txq->vif);
+ return &sdata->airtime[txq->ac];
+}
+
+/* To avoid divisions in the fast path, we keep pre-computed reciprocals for
+ * airtime weight calculations. There are two different weights to keep track
+ * of: The per-station weight and the sum of weights per phy.
+ *
+ * For the per-station weights (kept in airtime_info below), we use 32-bit
+ * reciprocals with a devisor of 2^19. This lets us keep the multiplications and
+ * divisions for the station weights as 32-bit operations at the cost of a bit
+ * of rounding error for high weights; but the choice of divisor keeps rounding
+ * errors <10% for weights <2^15, assuming no more than 8ms of airtime is
+ * reported at a time.
+ *
+ * For the per-phy sum of weights the values can get higher, so we use 64-bit
+ * operations for those with a 32-bit divisor, which should avoid any
+ * significant rounding errors.
+ */
+#define IEEE80211_RECIPROCAL_DIVISOR_64 0x100000000ULL
+#define IEEE80211_RECIPROCAL_SHIFT_64 32
+#define IEEE80211_RECIPROCAL_DIVISOR_32 0x80000U
+#define IEEE80211_RECIPROCAL_SHIFT_32 19
+
+static inline void airtime_weight_set(struct airtime_info *air_info, u16 weight)
+{
+ if (air_info->weight == weight)
+ return;
+
+ air_info->weight = weight;
+ if (weight) {
+ air_info->weight_reciprocal =
+ IEEE80211_RECIPROCAL_DIVISOR_32 / weight;
+ } else {
+ air_info->weight_reciprocal = 0;
+ }
+}
+
+static inline void airtime_weight_sum_set(struct airtime_sched_info *air_sched,
+ int weight_sum)
+{
+ if (air_sched->weight_sum == weight_sum)
+ return;
+
+ air_sched->weight_sum = weight_sum;
+ if (air_sched->weight_sum) {
+ air_sched->weight_sum_reciprocal = IEEE80211_RECIPROCAL_DIVISOR_64;
+ do_div(air_sched->weight_sum_reciprocal, air_sched->weight_sum);
+ } else {
+ air_sched->weight_sum_reciprocal = 0;
+ }
+}
+
+/* A problem when trying to enforce airtime fairness is that we want to divide
+ * the airtime between the currently *active* stations. However, basing this on
+ * the instantaneous queue state of stations doesn't work, as queues tend to
+ * oscillate very quickly between empty and occupied, leading to the scheduler
+ * thinking only a single station is active when deciding whether to allow
+ * transmission (and thus not throttling correctly).
+ *
+ * To fix this we use a timer-based notion of activity: a station is considered
+ * active if it has been scheduled within the last 100 ms; we keep a separate
+ * list of all the stations considered active in this manner, and lazily update
+ * the total weight of active stations from this list (filtering the stations in
+ * the list by their 'last active' time).
+ *
+ * We add one additional safeguard to guard against stations that manage to get
+ * scheduled every 100 ms but don't transmit a lot of data, and thus don't use
+ * up any airtime. Such stations would be able to get priority for an extended
+ * period of time if they do start transmitting at full capacity again, and so
+ * we add an explicit maximum for how far behind a station is allowed to fall in
+ * the virtual airtime domain. This limit is set to a relatively high value of
+ * 20 ms because the main mechanism for catching up idle stations is the active
+ * state as described above; i.e., the hard limit should only be hit in
+ * pathological cases.
+ */
+#define AIRTIME_ACTIVE_DURATION (100 * NSEC_PER_MSEC)
+#define AIRTIME_MAX_BEHIND 20000 /* 20 ms */
+
+static inline bool airtime_is_active(struct airtime_info *air_info, u64 now)
+{
+ return air_info->last_scheduled >= now - AIRTIME_ACTIVE_DURATION;
+}
+
+static inline void airtime_set_active(struct airtime_sched_info *air_sched,
+ struct airtime_info *air_info, u64 now)
+{
+ air_info->last_scheduled = now;
+ air_sched->last_schedule_activity = now;
+ list_move_tail(&air_info->list, &air_sched->active_list);
+}
+
+static inline bool airtime_catchup_v_t(struct airtime_sched_info *air_sched,
+ u64 v_t, u64 now)
+{
+ air_sched->v_t = v_t;
+ return true;
+}
+
+static inline void init_airtime_info(struct airtime_info *air_info,
+ struct airtime_sched_info *air_sched)
+{
+ atomic_set(&air_info->aql_tx_pending, 0);
+ air_info->aql_limit_low = air_sched->aql_txq_limit_low;
+ air_info->aql_limit_high = air_sched->aql_txq_limit_high;
+ airtime_weight_set(air_info, IEEE80211_DEFAULT_AIRTIME_WEIGHT);
+ INIT_LIST_HEAD(&air_info->list);
+}
+
static inline int ieee80211_bssid_match(const u8 *raddr, const u8 *addr)
{
return ether_addr_equal(raddr, addr) ||
u64 *cookie);
int ieee80211_probe_mesh_link(struct wiphy *wiphy, struct net_device *dev,
const u8 *buf, size_t len);
+void ieee80211_resort_txq(struct ieee80211_hw *hw,
+ struct ieee80211_txq *txq);
+void ieee80211_unschedule_txq(struct ieee80211_hw *hw,
+ struct ieee80211_txq *txq,
+ bool purge);
+void ieee80211_update_airtime_weight(struct ieee80211_local *local,
+ struct airtime_sched_info *air_sched,
+ u64 now, bool force);
/* HT */
void ieee80211_apply_htcap_overrides(struct ieee80211_sub_if_data *sdata,
enum ieee80211_sta_rx_bandwidth
ieee80211_chan_width_to_rx_bw(enum nl80211_chan_width width);
enum nl80211_chan_width ieee80211_sta_cap_chan_bw(struct sta_info *sta);
-void ieee80211_sta_set_rx_nss(struct sta_info *sta);
void ieee80211_process_mu_groups(struct ieee80211_sub_if_data *sdata,
struct ieee80211_mgmt *mgmt);
u32 __ieee80211_vht_handle_opmode(struct ieee80211_sub_if_data *sdata,
struct net_device *dev,
const u8 *addr);
void ieee80211_teardown_tdls_peers(struct ieee80211_sub_if_data *sdata);
-void ieee80211_tdls_chsw_work(struct work_struct *wk);
void ieee80211_tdls_handle_disconnect(struct ieee80211_sub_if_data *sdata,
const u8 *peer, u16 reason);
+void
+ieee80211_process_tdls_channel_switch(struct ieee80211_sub_if_data *sdata,
+ struct sk_buff *skb);
+
+
const char *ieee80211_get_reason_code_string(u16 reason_code);
u16 ieee80211_encode_usf(int val);
u8 *ieee80211_get_bssid(struct ieee80211_hdr *hdr, size_t len,
#define debug_noinline
#endif
+void ieee80211_init_frag_cache(struct ieee80211_fragment_cache *cache);
+void ieee80211_destroy_frag_cache(struct ieee80211_fragment_cache *cache);
+
#endif /* IEEE80211_I_H */
* Copyright 2008, Johannes Berg <johannes@sipsolutions.net>
* Copyright 2013-2014 Intel Mobile Communications GmbH
* Copyright (c) 2016 Intel Deutschland GmbH
- * Copyright (C) 2018-2020 Intel Corporation
+ * Copyright (C) 2018-2021 Intel Corporation
*/
#include <linux/slab.h>
#include <linux/kernel.h>
GFP_KERNEL);
}
- /* APs need special treatment */
if (sdata->vif.type == NL80211_IFTYPE_AP) {
- struct ieee80211_sub_if_data *vlan, *tmpsdata;
-
- /* down all dependent devices, that is VLANs */
- list_for_each_entry_safe(vlan, tmpsdata, &sdata->u.ap.vlans,
- u.vlan.list)
- dev_close(vlan->dev);
WARN_ON(!list_empty(&sdata->u.ap.vlans));
} else if (sdata->vif.type == NL80211_IFTYPE_AP_VLAN) {
/* remove all packets in parent bc_buf pointing to this dev */
{
struct ieee80211_sub_if_data *sdata = IEEE80211_DEV_TO_SUB_IF(dev);
+ /* close all dependent VLAN interfaces before locking wiphy */
+ if (sdata->vif.type == NL80211_IFTYPE_AP) {
+ struct ieee80211_sub_if_data *vlan, *tmpsdata;
+
+ list_for_each_entry_safe(vlan, tmpsdata, &sdata->u.ap.vlans,
+ u.vlan.list)
+ dev_close(vlan->dev);
+ }
+
wiphy_lock(sdata->local->hw.wiphy);
ieee80211_do_stop(sdata, true);
wiphy_unlock(sdata->local->hw.wiphy);
*/
static void ieee80211_teardown_sdata(struct ieee80211_sub_if_data *sdata)
{
- int i;
-
/* free extra data */
ieee80211_free_keys(sdata, false);
ieee80211_debugfs_remove_netdev(sdata);
- for (i = 0; i < IEEE80211_FRAGMENT_MAX; i++)
- __skb_queue_purge(&sdata->fragments[i].skb_list);
- sdata->fragment_next = 0;
+ ieee80211_destroy_frag_cache(&sdata->frags);
if (ieee80211_vif_is_mesh(&sdata->vif))
ieee80211_mesh_teardown_sdata(sdata);
dev->priv_flags |= IFF_NO_QUEUE;
}
+static void ieee80211_iface_process_skb(struct ieee80211_local *local,
+ struct ieee80211_sub_if_data *sdata,
+ struct sk_buff *skb)
+{
+ struct ieee80211_mgmt *mgmt = (void *)skb->data;
+
+ if (ieee80211_is_action(mgmt->frame_control) &&
+ mgmt->u.action.category == WLAN_CATEGORY_BACK) {
+ struct sta_info *sta;
+ int len = skb->len;
+
+ mutex_lock(&local->sta_mtx);
+ sta = sta_info_get_bss(sdata, mgmt->sa);
+ if (sta) {
+ switch (mgmt->u.action.u.addba_req.action_code) {
+ case WLAN_ACTION_ADDBA_REQ:
+ ieee80211_process_addba_request(local, sta,
+ mgmt, len);
+ break;
+ case WLAN_ACTION_ADDBA_RESP:
+ ieee80211_process_addba_resp(local, sta,
+ mgmt, len);
+ break;
+ case WLAN_ACTION_DELBA:
+ ieee80211_process_delba(sdata, sta,
+ mgmt, len);
+ break;
+ default:
+ WARN_ON(1);
+ break;
+ }
+ }
+ mutex_unlock(&local->sta_mtx);
+ } else if (ieee80211_is_action(mgmt->frame_control) &&
+ mgmt->u.action.category == WLAN_CATEGORY_VHT) {
+ switch (mgmt->u.action.u.vht_group_notif.action_code) {
+ case WLAN_VHT_ACTION_OPMODE_NOTIF: {
+ struct ieee80211_rx_status *status;
+ enum nl80211_band band;
+ struct sta_info *sta;
+ u8 opmode;
+
+ status = IEEE80211_SKB_RXCB(skb);
+ band = status->band;
+ opmode = mgmt->u.action.u.vht_opmode_notif.operating_mode;
+
+ mutex_lock(&local->sta_mtx);
+ sta = sta_info_get_bss(sdata, mgmt->sa);
+
+ if (sta)
+ ieee80211_vht_handle_opmode(sdata, sta, opmode,
+ band);
+
+ mutex_unlock(&local->sta_mtx);
+ break;
+ }
+ case WLAN_VHT_ACTION_GROUPID_MGMT:
+ ieee80211_process_mu_groups(sdata, mgmt);
+ break;
+ default:
+ WARN_ON(1);
+ break;
+ }
+ } else if (ieee80211_is_ext(mgmt->frame_control)) {
+ if (sdata->vif.type == NL80211_IFTYPE_STATION)
+ ieee80211_sta_rx_queued_ext(sdata, skb);
+ else
+ WARN_ON(1);
+ } else if (ieee80211_is_data_qos(mgmt->frame_control)) {
+ struct ieee80211_hdr *hdr = (void *)mgmt;
+ struct sta_info *sta;
+
+ /*
+ * So the frame isn't mgmt, but frame_control
+ * is at the right place anyway, of course, so
+ * the if statement is correct.
+ *
+ * Warn if we have other data frame types here,
+ * they must not get here.
+ */
+ WARN_ON(hdr->frame_control &
+ cpu_to_le16(IEEE80211_STYPE_NULLFUNC));
+ WARN_ON(!(hdr->seq_ctrl &
+ cpu_to_le16(IEEE80211_SCTL_FRAG)));
+ /*
+ * This was a fragment of a frame, received while
+ * a block-ack session was active. That cannot be
+ * right, so terminate the session.
+ */
+ mutex_lock(&local->sta_mtx);
+ sta = sta_info_get_bss(sdata, mgmt->sa);
+ if (sta) {
+ u16 tid = ieee80211_get_tid(hdr);
+
+ __ieee80211_stop_rx_ba_session(
+ sta, tid, WLAN_BACK_RECIPIENT,
+ WLAN_REASON_QSTA_REQUIRE_SETUP,
+ true);
+ }
+ mutex_unlock(&local->sta_mtx);
+ } else switch (sdata->vif.type) {
+ case NL80211_IFTYPE_STATION:
+ ieee80211_sta_rx_queued_mgmt(sdata, skb);
+ break;
+ case NL80211_IFTYPE_ADHOC:
+ ieee80211_ibss_rx_queued_mgmt(sdata, skb);
+ break;
+ case NL80211_IFTYPE_MESH_POINT:
+ if (!ieee80211_vif_is_mesh(&sdata->vif))
+ break;
+ ieee80211_mesh_rx_queued_mgmt(sdata, skb);
+ break;
+ default:
+ WARN(1, "frame for unexpected interface type");
+ break;
+ }
+}
+
static void ieee80211_iface_work(struct work_struct *work)
{
struct ieee80211_sub_if_data *sdata =
container_of(work, struct ieee80211_sub_if_data, work);
struct ieee80211_local *local = sdata->local;
struct sk_buff *skb;
- struct sta_info *sta;
if (!ieee80211_sdata_running(sdata))
return;
/* first process frames */
while ((skb = skb_dequeue(&sdata->skb_queue))) {
- struct ieee80211_mgmt *mgmt = (void *)skb->data;
-
kcov_remote_start_common(skb_get_kcov_handle(skb));
- if (ieee80211_is_action(mgmt->frame_control) &&
- mgmt->u.action.category == WLAN_CATEGORY_BACK) {
- int len = skb->len;
- mutex_lock(&local->sta_mtx);
- sta = sta_info_get_bss(sdata, mgmt->sa);
- if (sta) {
- switch (mgmt->u.action.u.addba_req.action_code) {
- case WLAN_ACTION_ADDBA_REQ:
- ieee80211_process_addba_request(
- local, sta, mgmt, len);
- break;
- case WLAN_ACTION_ADDBA_RESP:
- ieee80211_process_addba_resp(local, sta,
- mgmt, len);
- break;
- case WLAN_ACTION_DELBA:
- ieee80211_process_delba(sdata, sta,
- mgmt, len);
- break;
- default:
- WARN_ON(1);
- break;
- }
- }
- mutex_unlock(&local->sta_mtx);
- } else if (ieee80211_is_action(mgmt->frame_control) &&
- mgmt->u.action.category == WLAN_CATEGORY_VHT) {
- switch (mgmt->u.action.u.vht_group_notif.action_code) {
- case WLAN_VHT_ACTION_OPMODE_NOTIF: {
- struct ieee80211_rx_status *status;
- enum nl80211_band band;
- u8 opmode;
-
- status = IEEE80211_SKB_RXCB(skb);
- band = status->band;
- opmode = mgmt->u.action.u.vht_opmode_notif.operating_mode;
-
- mutex_lock(&local->sta_mtx);
- sta = sta_info_get_bss(sdata, mgmt->sa);
-
- if (sta)
- ieee80211_vht_handle_opmode(sdata, sta,
- opmode,
- band);
-
- mutex_unlock(&local->sta_mtx);
- break;
- }
- case WLAN_VHT_ACTION_GROUPID_MGMT:
- ieee80211_process_mu_groups(sdata, mgmt);
- break;
- default:
- WARN_ON(1);
- break;
- }
- } else if (ieee80211_is_ext(mgmt->frame_control)) {
- if (sdata->vif.type == NL80211_IFTYPE_STATION)
- ieee80211_sta_rx_queued_ext(sdata, skb);
- else
- WARN_ON(1);
- } else if (ieee80211_is_data_qos(mgmt->frame_control)) {
- struct ieee80211_hdr *hdr = (void *)mgmt;
- /*
- * So the frame isn't mgmt, but frame_control
- * is at the right place anyway, of course, so
- * the if statement is correct.
- *
- * Warn if we have other data frame types here,
- * they must not get here.
- */
- WARN_ON(hdr->frame_control &
- cpu_to_le16(IEEE80211_STYPE_NULLFUNC));
- WARN_ON(!(hdr->seq_ctrl &
- cpu_to_le16(IEEE80211_SCTL_FRAG)));
- /*
- * This was a fragment of a frame, received while
- * a block-ack session was active. That cannot be
- * right, so terminate the session.
- */
- mutex_lock(&local->sta_mtx);
- sta = sta_info_get_bss(sdata, mgmt->sa);
- if (sta) {
- u16 tid = ieee80211_get_tid(hdr);
-
- __ieee80211_stop_rx_ba_session(
- sta, tid, WLAN_BACK_RECIPIENT,
- WLAN_REASON_QSTA_REQUIRE_SETUP,
- true);
- }
- mutex_unlock(&local->sta_mtx);
- } else switch (sdata->vif.type) {
- case NL80211_IFTYPE_STATION:
- ieee80211_sta_rx_queued_mgmt(sdata, skb);
- break;
- case NL80211_IFTYPE_ADHOC:
- ieee80211_ibss_rx_queued_mgmt(sdata, skb);
- break;
- case NL80211_IFTYPE_MESH_POINT:
- if (!ieee80211_vif_is_mesh(&sdata->vif))
- break;
- ieee80211_mesh_rx_queued_mgmt(sdata, skb);
- break;
- default:
- WARN(1, "frame for unexpected interface type");
- break;
- }
+ if (skb->protocol == cpu_to_be16(ETH_P_TDLS))
+ ieee80211_process_tdls_channel_switch(sdata, skb);
+ else
+ ieee80211_iface_process_skb(local, sdata, skb);
kfree_skb(skb);
kcov_remote_stop();
switch (sdata->vif.type) {
case NL80211_IFTYPE_AP:
+ if (!list_empty(&sdata->u.ap.vlans))
+ return -EBUSY;
+ break;
case NL80211_IFTYPE_STATION:
case NL80211_IFTYPE_ADHOC:
case NL80211_IFTYPE_OCB:
sdata->wdev.wiphy = local->hw.wiphy;
sdata->local = local;
- for (i = 0; i < IEEE80211_FRAGMENT_MAX; i++)
- skb_queue_head_init(&sdata->fragments[i].skb_list);
+ ieee80211_init_frag_cache(&sdata->frags);
INIT_LIST_HEAD(&sdata->key_list);
}
}
+ for (i = 0; i < IEEE80211_NUM_ACS; i++)
+ init_airtime_info(&sdata->airtime[i], &local->airtime[i]);
+
ieee80211_set_default_queues(sdata);
sdata->ap_power_level = IEEE80211_UNSET_POWER_LEVEL;
struct ieee80211_sub_if_data *sdata,
struct sta_info *sta)
{
+ static atomic_t key_color = ATOMIC_INIT(0);
struct ieee80211_key *old_key;
int idx = key->conf.keyidx;
bool pairwise = key->conf.flags & IEEE80211_KEY_FLAG_PAIRWISE;
key->sdata = sdata;
key->sta = sta;
+ /*
+ * Assign a unique ID to every key so we can easily prevent mixed
+ * key and fragment cache attacks.
+ */
+ key->color = atomic_inc_return(&key_color);
+
increment_tailroom_need_count(sdata);
ret = ieee80211_key_replace(sdata, sta, pairwise, old_key, key);
} debugfs;
#endif
+ unsigned int color;
+
/*
* key config, must be last because it contains key
* material as variable length member
{
struct tpt_led_trigger *tpt_trig = from_timer(tpt_trig, t, timer);
struct ieee80211_local *local = tpt_trig->local;
- struct led_classdev *led_cdev;
unsigned long on, off, tpt;
int i;
}
}
- read_lock(&local->tpt_led.leddev_list_lock);
- list_for_each_entry(led_cdev, &local->tpt_led.led_cdevs, trig_list)
- led_blink_set(led_cdev, &on, &off);
- read_unlock(&local->tpt_led.leddev_list_lock);
+ led_trigger_blink(&local->tpt_led, &on, &off);
}
const char *
static void ieee80211_stop_tpt_led_trig(struct ieee80211_local *local)
{
struct tpt_led_trigger *tpt_trig = local->tpt_led_trigger;
- struct led_classdev *led_cdev;
if (!tpt_trig->running)
return;
tpt_trig->running = false;
del_timer_sync(&tpt_trig->timer);
- read_lock(&local->tpt_led.leddev_list_lock);
- list_for_each_entry(led_cdev, &local->tpt_led.led_cdevs, trig_list)
- led_set_brightness(led_cdev, LED_OFF);
- read_unlock(&local->tpt_led.leddev_list_lock);
+ led_trigger_event(&local->tpt_led, LED_OFF);
}
void ieee80211_mod_tpt_led_trig(struct ieee80211_local *local,
struct ieee80211_local *local =
container_of(work, struct ieee80211_local, restart_work);
struct ieee80211_sub_if_data *sdata;
+ int ret;
/* wait for scan work complete */
flush_workqueue(local->workqueue);
flush_work(&local->sched_scan_stopped_work);
+ flush_work(&local->radar_detected_work);
+
+ rtnl_lock();
WARN(test_bit(SCAN_HW_SCANNING, &local->scanning),
"%s called with hardware scan in progress\n", __func__);
- flush_work(&local->radar_detected_work);
- /* we might do interface manipulations, so need both */
- rtnl_lock();
- wiphy_lock(local->hw.wiphy);
list_for_each_entry(sdata, &local->interfaces, list) {
/*
* XXX: there may be more work for other vif types and even
/* wait for all packet processing to be done */
synchronize_net();
- ieee80211_reconfig(local);
+ ret = ieee80211_reconfig(local);
wiphy_unlock(local->hw.wiphy);
+
+ if (ret)
+ cfg80211_shutdown_all_interfaces(local->hw.wiphy);
+
rtnl_unlock();
}
spin_lock_init(&local->queue_stop_reason_lock);
for (i = 0; i < IEEE80211_NUM_ACS; i++) {
- INIT_LIST_HEAD(&local->active_txqs[i]);
- spin_lock_init(&local->active_txq_lock[i]);
- local->aql_txq_limit_low[i] = IEEE80211_DEFAULT_AQL_TXQ_LIMIT_L;
- local->aql_txq_limit_high[i] =
+ struct airtime_sched_info *air_sched = &local->airtime[i];
+
+ air_sched->active_txqs = RB_ROOT_CACHED;
+ INIT_LIST_HEAD(&air_sched->active_list);
+ spin_lock_init(&air_sched->lock);
+ air_sched->aql_txq_limit_low = IEEE80211_DEFAULT_AQL_TXQ_LIMIT_L;
+ air_sched->aql_txq_limit_high =
IEEE80211_DEFAULT_AQL_TXQ_LIMIT_H;
}
INIT_WORK(&local->sched_scan_stopped_work,
ieee80211_sched_scan_stopped_work);
- INIT_WORK(&local->tdls_chsw_work, ieee80211_tdls_chsw_work);
-
spin_lock_init(&local->ack_status_lock);
idr_init(&local->ack_status_frames);
skb_queue_head_init(&local->skb_queue);
skb_queue_head_init(&local->skb_queue_unreliable);
- skb_queue_head_init(&local->skb_queue_tdls_chsw);
ieee80211_alloc_led_names(local);
supp_ht = supp_ht || sband->ht_cap.ht_supported;
supp_vht = supp_vht || sband->vht_cap.vht_supported;
- if (!supp_he)
- supp_he = !!ieee80211_get_he_sta_cap(sband);
+ for (i = 0; i < sband->n_iftype_data; i++) {
+ const struct ieee80211_sband_iftype_data *iftd;
+
+ iftd = &sband->iftype_data[i];
+
+ supp_he = supp_he || (iftd && iftd->he_cap.has_he);
+ }
/* HT, VHT, HE require QoS, thus >= 4 queues */
if (WARN_ON(local->hw.queues < IEEE80211_NUM_ACS &&
cancel_delayed_work_sync(&local->roc_work);
cancel_work_sync(&local->restart_work);
cancel_work_sync(&local->reconfig_filter);
- cancel_work_sync(&local->tdls_chsw_work);
flush_work(&local->sched_scan_stopped_work);
flush_work(&local->radar_detected_work);
wiphy_warn(local->hw.wiphy, "skb_queue not empty\n");
skb_queue_purge(&local->skb_queue);
skb_queue_purge(&local->skb_queue_unreliable);
- skb_queue_purge(&local->skb_queue_tdls_chsw);
wiphy_unregister(local->hw.wiphy);
destroy_workqueue(local->workqueue);
* gate's mpath may or may not be resolved and active.
* @gates_lock: protects updates to known_gates
* @rhead: the rhashtable containing struct mesh_paths, keyed by dest addr
- * @walk_head: linked list containging all mesh_path objects
+ * @walk_head: linked list containing all mesh_path objects
* @walk_lock: lock protecting walk_head
* @entries: number of entries in the table
*/
* forwarding information is found.
*
* Returns: 0 if the next hop was found and -ENOENT if the frame was queued.
- * skb is freeed here if no mpath could be allocated.
+ * skb is freed here if no mpath could be allocated.
*/
int mesh_nexthop_resolve(struct ieee80211_sub_if_data *sdata,
struct sk_buff *skb)
hdr = (struct ieee80211_hdr *) skb->data;
/* we preserve the previous mesh header and only add
- * the new addreses */
+ * the new addresses */
mshdr = (struct ieee80211s_hdr *) (skb->data + hdrlen);
mshdr->flags = MESH_FLAGS_AE_A5_A6;
memcpy(mshdr->eaddr1, hdr->addr3, ETH_ALEN);
* mesh STA in a MBSS. Three HT protection modes are supported for now, non-HT
* mixed mode, 20MHz-protection and no-protection mode. non-HT mixed mode is
* selected if any non-HT peers are present in our MBSS. 20MHz-protection mode
- * is selected if all peers in our 20/40MHz MBSS support HT and atleast one
+ * is selected if all peers in our 20/40MHz MBSS support HT and at least one
* HT20 peer is present. Otherwise no-protection mode is selected.
*/
static u32 mesh_set_ht_prot_mode(struct ieee80211_sub_if_data *sdata)
* Copyright 2007, Michael Wu <flamingice@sourmilk.net>
* Copyright 2013-2014 Intel Mobile Communications GmbH
* Copyright (C) 2015 - 2017 Intel Deutschland GmbH
- * Copyright (C) 2018 - 2020 Intel Corporation
+ * Copyright (C) 2018 - 2021 Intel Corporation
*/
#include <linux/delay.h>
struct cfg80211_chan_def chandef;
u16 ht_opmode;
u32 flags;
- enum ieee80211_sta_rx_bandwidth new_sta_bw;
u32 vht_cap_info = 0;
int ret;
/* don't check HE if we associated as non-HE station */
if (ifmgd->flags & IEEE80211_STA_DISABLE_HE ||
- !ieee80211_get_he_sta_cap(sband))
+ !ieee80211_get_he_iftype_cap(sband,
+ ieee80211_vif_type_p2p(&sdata->vif)))
+
he_oper = NULL;
if (WARN_ON_ONCE(!sta))
IEEE80211_STA_DISABLE_160MHZ)) ||
!cfg80211_chandef_valid(&chandef)) {
sdata_info(sdata,
- "AP %pM changed bandwidth in a way we can't support - disconnect\n",
- ifmgd->bssid);
+ "AP %pM changed caps/bw in a way we can't support (0x%x/0x%x) - disconnect\n",
+ ifmgd->bssid, flags, ifmgd->flags);
return -EINVAL;
}
- switch (chandef.width) {
- case NL80211_CHAN_WIDTH_20_NOHT:
- case NL80211_CHAN_WIDTH_20:
- new_sta_bw = IEEE80211_STA_RX_BW_20;
- break;
- case NL80211_CHAN_WIDTH_40:
- new_sta_bw = IEEE80211_STA_RX_BW_40;
- break;
- case NL80211_CHAN_WIDTH_80:
- new_sta_bw = IEEE80211_STA_RX_BW_80;
- break;
- case NL80211_CHAN_WIDTH_80P80:
- case NL80211_CHAN_WIDTH_160:
- new_sta_bw = IEEE80211_STA_RX_BW_160;
- break;
- default:
- return -EINVAL;
- }
-
- if (new_sta_bw > sta->cur_max_bandwidth)
- new_sta_bw = sta->cur_max_bandwidth;
-
- if (new_sta_bw < sta->sta.bandwidth) {
- sta->sta.bandwidth = new_sta_bw;
- rate_control_rate_update(local, sband, sta,
- IEEE80211_RC_BW_CHANGED);
- }
-
ret = ieee80211_vif_change_bandwidth(sdata, &chandef, changed);
+
if (ret) {
sdata_info(sdata,
"AP %pM changed bandwidth to incompatible one - disconnect\n",
return ret;
}
- if (new_sta_bw > sta->sta.bandwidth) {
- sta->sta.bandwidth = new_sta_bw;
- rate_control_rate_update(local, sband, sta,
- IEEE80211_RC_BW_CHANGED);
- }
-
return 0;
}
cap &= ~IEEE80211_VHT_CAP_MU_BEAMFORMEE_CAPABLE;
/*
- * If some other vif is using the MU-MIMO capablity we cannot associate
+ * If some other vif is using the MU-MIMO capability we cannot associate
* using MU-MIMO - this will lead to contradictions in the group-id
* mechanism.
* Ownership is defined since association request, in order to avoid
rcu_read_unlock();
- he_cap = ieee80211_get_he_sta_cap(sband);
+ he_cap = ieee80211_get_he_iftype_cap(sband,
+ ieee80211_vif_type_p2p(&sdata->vif));
if (!he_cap || !reg_cap)
return;
u32 rates = 0;
__le16 listen_int;
struct element *ext_capa = NULL;
+ enum nl80211_iftype iftype = ieee80211_vif_type_p2p(&sdata->vif);
+ const struct ieee80211_sband_iftype_data *iftd;
+ struct ieee80211_prep_tx_info info = {};
/* we know it's writable, cast away the const */
if (assoc_data->ie_len)
}
}
+ iftd = ieee80211_get_sband_iftype_data(sband, iftype);
+
skb = alloc_skb(local->hw.extra_tx_headroom +
sizeof(*mgmt) + /* bit too much but doesn't matter */
2 + assoc_data->ssid_len + /* SSID */
2 + 1 + sizeof(struct ieee80211_he_6ghz_capa) +
assoc_data->ie_len + /* extra IEs */
(assoc_data->fils_kek_len ? 16 /* AES-SIV */ : 0) +
- 9, /* WMM */
+ 9 + /* WMM */
+ (iftd ? iftd->vendor_elems.len : 0),
GFP_KERNEL);
if (!skb)
return;
mgmt->u.reassoc_req.listen_interval = listen_int;
memcpy(mgmt->u.reassoc_req.current_ap, assoc_data->prev_bssid,
ETH_ALEN);
+ info.subtype = IEEE80211_STYPE_REASSOC_REQ;
} else {
skb_put(skb, 4);
mgmt->frame_control = cpu_to_le16(IEEE80211_FTYPE_MGMT |
IEEE80211_STYPE_ASSOC_REQ);
mgmt->u.assoc_req.capab_info = cpu_to_le16(capab);
mgmt->u.assoc_req.listen_interval = listen_int;
+ info.subtype = IEEE80211_STYPE_ASSOC_REQ;
}
/* SSID */
ieee80211_add_s1g_capab_ie(sdata, &sband->s1g_cap, skb);
}
+ if (iftd && iftd->vendor_elems.data && iftd->vendor_elems.len)
+ skb_put_data(skb, iftd->vendor_elems.data, iftd->vendor_elems.len);
+
/* add any remaining custom (i.e. vendor specific here) IEs */
if (assoc_data->ie_len) {
noffset = assoc_data->ie_len;
ifmgd->assoc_req_ies = kmemdup(ie_start, pos - ie_start, GFP_ATOMIC);
ifmgd->assoc_req_ies_len = pos - ie_start;
- drv_mgd_prepare_tx(local, sdata, 0);
+ drv_mgd_prepare_tx(local, sdata, &info);
IEEE80211_SKB_CB(skb)->flags |= IEEE80211_TX_INTFL_DONT_ENCRYPT;
if (ieee80211_hw_check(&local->hw, REPORTS_TX_ACK_STATUS))
struct ieee80211_hdr_3addr *nullfunc;
struct ieee80211_if_managed *ifmgd = &sdata->u.mgd;
- /* Don't send NDPs when STA is connected HE */
- if (sdata->vif.type == NL80211_IFTYPE_STATION &&
- !(ifmgd->flags & IEEE80211_STA_DISABLE_HE))
- return;
-
skb = ieee80211_nullfunc_get(&local->hw, &sdata->vif,
!ieee80211_hw_check(&local->hw, DOESNT_SUPPORT_QOS_NDP));
if (!skb)
if (WARN_ON(sdata->vif.type != NL80211_IFTYPE_STATION))
return;
- /* Don't send NDPs when connected HE */
- if (!(sdata->u.mgd.flags & IEEE80211_STA_DISABLE_HE))
- return;
-
skb = dev_alloc_skb(local->hw.extra_tx_headroom + 30);
if (!skb)
return;
*/
if (sdata->reserved_chanctx) {
- struct ieee80211_supported_band *sband = NULL;
- struct sta_info *mgd_sta = NULL;
- enum ieee80211_sta_rx_bandwidth bw = IEEE80211_STA_RX_BW_20;
-
/*
* with multi-vif csa driver may call ieee80211_csa_finish()
* many times while waiting for other interfaces to use their
if (sdata->reserved_ready)
goto out;
- if (sdata->vif.bss_conf.chandef.width !=
- sdata->csa_chandef.width) {
- /*
- * For managed interface, we need to also update the AP
- * station bandwidth and align the rate scale algorithm
- * on the bandwidth change. Here we only consider the
- * bandwidth of the new channel definition (as channel
- * switch flow does not have the full HT/VHT/HE
- * information), assuming that if additional changes are
- * required they would be done as part of the processing
- * of the next beacon from the AP.
- */
- switch (sdata->csa_chandef.width) {
- case NL80211_CHAN_WIDTH_20_NOHT:
- case NL80211_CHAN_WIDTH_20:
- default:
- bw = IEEE80211_STA_RX_BW_20;
- break;
- case NL80211_CHAN_WIDTH_40:
- bw = IEEE80211_STA_RX_BW_40;
- break;
- case NL80211_CHAN_WIDTH_80:
- bw = IEEE80211_STA_RX_BW_80;
- break;
- case NL80211_CHAN_WIDTH_80P80:
- case NL80211_CHAN_WIDTH_160:
- bw = IEEE80211_STA_RX_BW_160;
- break;
- }
-
- mgd_sta = sta_info_get(sdata, ifmgd->bssid);
- sband =
- local->hw.wiphy->bands[sdata->csa_chandef.chan->band];
- }
-
- if (sdata->vif.bss_conf.chandef.width >
- sdata->csa_chandef.width) {
- mgd_sta->sta.bandwidth = bw;
- rate_control_rate_update(local, sband, mgd_sta,
- IEEE80211_RC_BW_CHANGED);
- }
-
ret = ieee80211_vif_use_reserved_context(sdata);
if (ret) {
sdata_info(sdata,
goto out;
}
- if (sdata->vif.bss_conf.chandef.width <
- sdata->csa_chandef.width) {
- mgd_sta->sta.bandwidth = bw;
- rate_control_rate_update(local, sband, mgd_sta,
- IEEE80211_RC_BW_CHANGED);
- }
-
goto out;
}
struct ieee80211_if_managed *ifmgd = &sdata->u.mgd;
struct ieee80211_local *local = sdata->local;
u32 changed = 0;
+ struct ieee80211_prep_tx_info info = {
+ .subtype = stype,
+ };
sdata_assert_lock(sdata);
* driver requested so.
*/
if (ieee80211_hw_check(&local->hw, DEAUTH_NEED_MGD_TX_PREP) &&
- !ifmgd->have_beacon)
- drv_mgd_prepare_tx(sdata->local, sdata, 0);
+ !ifmgd->have_beacon) {
+ drv_mgd_prepare_tx(sdata->local, sdata, &info);
+ }
ieee80211_send_deauth_disassoc(sdata, ifmgd->bssid,
ifmgd->bssid, stype, reason,
if (tx)
ieee80211_flush_queues(local, sdata, false);
+ drv_mgd_complete_tx(sdata->local, sdata, &info);
+
/* clear bssid only after building the needed mgmt frames */
eth_zero_addr(ifmgd->bssid);
if (ieee80211_hw_check(&sdata->local->hw, REPORTS_TX_ACK_STATUS)) {
ifmgd->nullfunc_failed = false;
- if (!(ifmgd->flags & IEEE80211_STA_DISABLE_HE))
- ifmgd->probe_send_count--;
- else
- ieee80211_send_nullfunc(sdata->local, sdata, false);
+ ieee80211_send_nullfunc(sdata->local, sdata, false);
} else {
int ssid_len;
u8 *pos;
struct ieee802_11_elems elems;
u32 tx_flags = 0;
+ struct ieee80211_prep_tx_info info = {
+ .subtype = IEEE80211_STYPE_AUTH,
+ };
pos = mgmt->u.auth.variable;
ieee802_11_parse_elems(pos, len - (pos - (u8 *)mgmt), false, &elems,
if (!elems.challenge)
return;
auth_data->expected_transaction = 4;
- drv_mgd_prepare_tx(sdata->local, sdata, 0);
+ drv_mgd_prepare_tx(sdata->local, sdata, &info);
if (ieee80211_hw_check(&local->hw, REPORTS_TX_ACK_STATUS))
tx_flags = IEEE80211_TX_CTL_REQ_TX_STATUS |
IEEE80211_TX_INTFL_MLME_CONN_TX;
.type = MLME_EVENT,
.u.mlme.data = AUTH_EVENT,
};
+ struct ieee80211_prep_tx_info info = {
+ .subtype = IEEE80211_STYPE_AUTH,
+ };
sdata_assert_lock(sdata);
mgmt->sa, auth_alg, ifmgd->auth_data->algorithm,
auth_transaction,
ifmgd->auth_data->expected_transaction);
- return;
+ goto notify_driver;
}
if (status_code != WLAN_STATUS_SUCCESS) {
(auth_transaction == 1 &&
(status_code == WLAN_STATUS_SAE_HASH_TO_ELEMENT ||
status_code == WLAN_STATUS_SAE_PK))))
- return;
+ goto notify_driver;
sdata_info(sdata, "%pM denied authentication (status %d)\n",
mgmt->sa, status_code);
event.u.mlme.status = MLME_DENIED;
event.u.mlme.reason = status_code;
drv_event_callback(sdata->local, sdata, &event);
- return;
+ goto notify_driver;
}
switch (ifmgd->auth_data->algorithm) {
default:
WARN_ONCE(1, "invalid auth alg %d",
ifmgd->auth_data->algorithm);
- return;
+ goto notify_driver;
}
event.u.mlme.status = MLME_SUCCESS;
+ info.success = 1;
drv_event_callback(sdata->local, sdata, &event);
if (ifmgd->auth_data->algorithm != WLAN_AUTH_SAE ||
(auth_transaction == 2 &&
}
cfg80211_rx_mlme_mgmt(sdata->dev, (u8 *)mgmt, len);
+notify_driver:
+ drv_mgd_complete_tx(sdata->local, sdata, &info);
}
#define case_WLAN(type) \
return 0;
}
+static bool ieee80211_twt_bcast_support(struct ieee80211_sub_if_data *sdata,
+ struct ieee80211_bss_conf *bss_conf,
+ struct ieee80211_supported_band *sband,
+ struct sta_info *sta)
+{
+ const struct ieee80211_sta_he_cap *own_he_cap =
+ ieee80211_get_he_iftype_cap(sband,
+ ieee80211_vif_type_p2p(&sdata->vif));
+
+ return bss_conf->he_support &&
+ (sta->sta.he_cap.he_cap_elem.mac_cap_info[2] &
+ IEEE80211_HE_MAC_CAP2_BCAST_TWT) &&
+ own_he_cap &&
+ (own_he_cap->he_cap_elem.mac_cap_info[2] &
+ IEEE80211_HE_MAC_CAP2_BCAST_TWT);
+}
+
static bool ieee80211_assoc_success(struct ieee80211_sub_if_data *sdata,
struct cfg80211_bss *cbss,
struct ieee80211_mgmt *mgmt, size_t len,
bss_conf->twt_protected = false;
}
+ bss_conf->twt_broadcast =
+ ieee80211_twt_bcast_support(sdata, bss_conf, sband, sta);
+
if (bss_conf->he_support) {
bss_conf->he_bss_color.color =
le32_get_bits(elems->he_operation->he_oper_params,
.type = MLME_EVENT,
.u.mlme.data = ASSOC_EVENT,
};
+ struct ieee80211_prep_tx_info info = {};
sdata_assert_lock(sdata);
aid = 0; /* TODO */
}
+ /*
+ * Note: this may not be perfect, AP might misbehave - if
+ * anyone needs to rely on perfect complete notification
+ * with the exact right subtype, then we need to track what
+ * we actually transmitted.
+ */
+ info.subtype = reassoc ? IEEE80211_STYPE_REASSOC_REQ :
+ IEEE80211_STYPE_ASSOC_REQ;
+
sdata_info(sdata,
"RX %sssocResp from %pM (capab=0x%x status=%d aid=%d)\n",
reassoc ? "Rea" : "A", mgmt->sa,
assoc_data->timeout_started = true;
if (ms > IEEE80211_ASSOC_TIMEOUT)
run_again(sdata, assoc_data->timeout);
- return;
+ goto notify_driver;
}
if (status_code != WLAN_STATUS_SUCCESS) {
/* oops -- internal error -- send timeout for now */
ieee80211_destroy_assoc_data(sdata, false, false);
cfg80211_assoc_timeout(sdata->dev, cbss);
- return;
+ goto notify_driver;
}
event.u.mlme.status = MLME_SUCCESS;
drv_event_callback(sdata->local, sdata, &event);
for (ac = 0; ac < IEEE80211_NUM_ACS; ac++)
if (sdata->tx_conf[ac].uapsd)
uapsd_queues |= ieee80211_ac_to_qos_mask[ac];
+
+ info.success = 1;
}
cfg80211_rx_assoc_resp(sdata->dev, cbss, (u8 *)mgmt, len, uapsd_queues,
ifmgd->assoc_req_ies, ifmgd->assoc_req_ies_len);
+notify_driver:
+ drv_mgd_complete_tx(sdata->local, sdata, &info);
}
static void ieee80211_rx_bss_info(struct ieee80211_sub_if_data *sdata,
if (elems.mbssid_config_ie)
bss_conf->profile_periodicity =
elems.mbssid_config_ie->profile_periodicity;
+ else
+ bss_conf->profile_periodicity = 0;
if (elems.ext_capab_len >= 11 &&
(elems.ext_capab[10] & WLAN_EXT_CAPA11_EMA_SUPPORT))
bss_conf->ema_ap = true;
+ else
+ bss_conf->ema_ap = false;
/* continue assoc process */
ifmgd->assoc_data->timeout = jiffies;
u32 tx_flags = 0;
u16 trans = 1;
u16 status = 0;
- u16 prepare_tx_duration = 0;
+ struct ieee80211_prep_tx_info info = {
+ .subtype = IEEE80211_STYPE_AUTH,
+ };
sdata_assert_lock(sdata);
}
if (auth_data->algorithm == WLAN_AUTH_SAE)
- prepare_tx_duration =
- jiffies_to_msecs(IEEE80211_AUTH_TIMEOUT_SAE);
+ info.duration = jiffies_to_msecs(IEEE80211_AUTH_TIMEOUT_SAE);
- drv_mgd_prepare_tx(local, sdata, prepare_tx_duration);
+ drv_mgd_prepare_tx(local, sdata, &info);
sdata_info(sdata, "send auth to %pM (try %d/%d)\n",
auth_data->bss->bssid, auth_data->tries,
}
static bool
-ieee80211_verify_sta_he_mcs_support(struct ieee80211_supported_band *sband,
+ieee80211_verify_sta_he_mcs_support(struct ieee80211_sub_if_data *sdata,
+ struct ieee80211_supported_band *sband,
const struct ieee80211_he_operation *he_op)
{
const struct ieee80211_sta_he_cap *sta_he_cap =
- ieee80211_get_he_sta_cap(sband);
+ ieee80211_get_he_iftype_cap(sband,
+ ieee80211_vif_type_p2p(&sdata->vif));
u16 ap_min_req_set;
int i;
ifmgd->flags |= IEEE80211_STA_DISABLE_HE;
}
- if (!ieee80211_get_he_sta_cap(sband))
+ if (!ieee80211_get_he_iftype_cap(sband,
+ ieee80211_vif_type_p2p(&sdata->vif)))
ifmgd->flags |= IEEE80211_STA_DISABLE_HE;
rcu_read_lock();
else
he_oper = NULL;
- if (!ieee80211_verify_sta_he_mcs_support(sband, he_oper))
+ if (!ieee80211_verify_sta_he_mcs_support(sdata, sband, he_oper))
ifmgd->flags |= IEEE80211_STA_DISABLE_HE;
}
2 * FILS_NONCE_LEN);
assoc_data->bss = req->bss;
-
- if (ifmgd->req_smps == IEEE80211_SMPS_AUTOMATIC) {
- if (ifmgd->powersave)
- sdata->smps_mode = IEEE80211_SMPS_DYNAMIC;
- else
- sdata->smps_mode = IEEE80211_SMPS_OFF;
- } else
- sdata->smps_mode = ifmgd->req_smps;
-
assoc_data->capability = req->bss->capability;
assoc_data->supp_rates = bss->supp_rates;
assoc_data->supp_rates_len = bss->supp_rates_len;
if (err)
goto err_clear;
+ if (ifmgd->req_smps == IEEE80211_SMPS_AUTOMATIC) {
+ if (ifmgd->powersave)
+ sdata->smps_mode = IEEE80211_SMPS_DYNAMIC;
+ else
+ sdata->smps_mode = IEEE80211_SMPS_OFF;
+ } else {
+ sdata->smps_mode = ifmgd->req_smps;
+ }
+
rcu_read_lock();
beacon_ies = rcu_dereference(req->bss->beacon_ies);
beacon_ies->data, beacon_ies->len);
if (elem && elem->datalen >= 3)
sdata->vif.bss_conf.profile_periodicity = elem->data[2];
+ else
+ sdata->vif.bss_conf.profile_periodicity = 0;
elem = cfg80211_find_elem(WLAN_EID_EXT_CAPABILITY,
beacon_ies->data, beacon_ies->len);
if (elem && elem->datalen >= 11 &&
(elem->data[10] & WLAN_EXT_CAPA11_EMA_SUPPORT))
sdata->vif.bss_conf.ema_ap = true;
+ else
+ sdata->vif.bss_conf.ema_ap = false;
} else {
assoc_data->timeout = jiffies;
assoc_data->timeout_started = true;
struct ieee80211_if_managed *ifmgd = &sdata->u.mgd;
u8 frame_buf[IEEE80211_DEAUTH_FRAME_LEN];
bool tx = !req->local_state_change;
+ struct ieee80211_prep_tx_info info = {
+ .subtype = IEEE80211_STYPE_DEAUTH,
+ };
if (ifmgd->auth_data &&
ether_addr_equal(ifmgd->auth_data->bss->bssid, req->bssid)) {
req->bssid, req->reason_code,
ieee80211_get_reason_code_string(req->reason_code));
- drv_mgd_prepare_tx(sdata->local, sdata, 0);
+ drv_mgd_prepare_tx(sdata->local, sdata, &info);
ieee80211_send_deauth_disassoc(sdata, req->bssid, req->bssid,
IEEE80211_STYPE_DEAUTH,
req->reason_code, tx,
ieee80211_report_disconnect(sdata, frame_buf,
sizeof(frame_buf), true,
req->reason_code, false);
-
+ drv_mgd_complete_tx(sdata->local, sdata, &info);
return 0;
}
req->bssid, req->reason_code,
ieee80211_get_reason_code_string(req->reason_code));
- drv_mgd_prepare_tx(sdata->local, sdata, 0);
+ drv_mgd_prepare_tx(sdata->local, sdata, &info);
ieee80211_send_deauth_disassoc(sdata, req->bssid, req->bssid,
IEEE80211_STYPE_DEAUTH,
req->reason_code, tx,
ieee80211_report_disconnect(sdata, frame_buf,
sizeof(frame_buf), true,
req->reason_code, false);
+ drv_mgd_complete_tx(sdata->local, sdata, &info);
return 0;
}
static bool rc_no_data_or_no_ack_use_min(struct ieee80211_tx_rate_control *txrc)
{
struct sk_buff *skb = txrc->skb;
- struct ieee80211_hdr *hdr = (struct ieee80211_hdr *) skb->data;
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
- __le16 fc;
-
- fc = hdr->frame_control;
return (info->flags & (IEEE80211_TX_CTL_NO_ACK |
IEEE80211_TX_CTL_USE_MINRATE)) ||
- !ieee80211_is_data(fc);
+ !ieee80211_is_tx_data(skb);
}
static void rc_send_low_basicrate(struct ieee80211_tx_rate *rate,
int mcast_rate;
bool use_basicrate = false;
+ if (ieee80211_is_tx_data(txrc->skb) &&
+ info->flags & IEEE80211_TX_CTL_NO_ACK)
+ return false;
+
if (!pubsta || rc_no_data_or_no_ack_use_min(txrc)) {
__rate_control_send_low(txrc->hw, sband, pubsta, info,
txrc->rate_idx_mask);
int max_rates)
{
struct ieee80211_sub_if_data *sdata;
- struct ieee80211_hdr *hdr = (struct ieee80211_hdr *) skb->data;
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
struct ieee80211_supported_band *sband;
sdata = vif_to_sdata(vif);
sband = sdata->local->hw.wiphy->bands[info->band];
- if (ieee80211_is_data(hdr->frame_control))
+ if (ieee80211_is_tx_data(skb))
rate_control_apply_mask(sdata, sta, sband, dest, max_rates);
if (dest[0].idx < 0)
unsigned int nsecs = 0, overhead = mi->overhead;
unsigned int ampdu_len = 1;
- /* do not account throughput if sucess prob is below 10% */
+ /* do not account throughput if success prob is below 10% */
if (prob_avg < MINSTREL_FRAC(10, 100))
return 0;
}
static void
-minstrel_aggr_check(struct ieee80211_sta *pubsta, struct sk_buff *skb)
-{
- struct ieee80211_hdr *hdr = (struct ieee80211_hdr *) skb->data;
- struct sta_info *sta = container_of(pubsta, struct sta_info, sta);
- u16 tid;
-
- if (skb_get_queue_mapping(skb) == IEEE80211_AC_VO)
- return;
-
- if (unlikely(!ieee80211_is_data_qos(hdr->frame_control)))
- return;
-
- if (unlikely(skb->protocol == cpu_to_be16(ETH_P_PAE)))
- return;
-
- tid = ieee80211_get_tid(hdr);
- if (likely(sta->ampdu_mlme.tid_tx[tid]))
- return;
-
- ieee80211_start_tx_ba_session(pubsta, tid, 0);
-}
-
-static void
minstrel_ht_tx_status(void *priv, struct ieee80211_supported_band *sband,
void *priv_sta, struct ieee80211_tx_status *st)
{
bool last, update = false;
int i;
+ /* Ignore packet that was sent with noAck flag */
+ if (info->flags & IEEE80211_TX_CTL_NO_ACK)
+ return;
+
/* This packet was aggregated but doesn't carry status info */
if ((info->flags & IEEE80211_TX_CTL_AMPDU) &&
!(info->flags & IEEE80211_TX_STAT_AMPDU))
struct minstrel_priv *mp = priv;
u16 sample_idx;
- if (!(info->flags & IEEE80211_TX_CTL_AMPDU) &&
- !minstrel_ht_is_legacy_group(MI_RATE_GROUP(mi->max_prob_rate)))
- minstrel_aggr_check(sta, txrc->skb);
-
info->flags |= mi->tx_flags;
#ifdef CONFIG_MAC80211_DEBUGFS
(info->control.flags & IEEE80211_TX_CTRL_PORT_CTRL_PROTO))
return;
- if (time_is_before_jiffies(mi->sample_time))
+ if (time_is_after_jiffies(mi->sample_time))
return;
mi->sample_time = jiffies + MINSTREL_SAMPLE_INTERVAL;
static const struct rate_control_ops mac80211_minstrel_ht = {
.name = "minstrel_ht",
+ .capa = RATE_CTRL_CAPA_AMPDU_TRIGGER,
.tx_status_ext = minstrel_ht_tx_status,
.get_rate = minstrel_ht_get_rate,
.rate_init = minstrel_ht_rate_init,
* Copyright 2007-2010 Johannes Berg <johannes@sipsolutions.net>
* Copyright 2013-2014 Intel Mobile Communications GmbH
* Copyright(c) 2015 - 2017 Intel Deutschland GmbH
- * Copyright (C) 2018-2020 Intel Corporation
+ * Copyright (C) 2018-2021 Intel Corporation
*/
#include <linux/jiffies.h>
return len;
}
+static void __ieee80211_queue_skb_to_iface(struct ieee80211_sub_if_data *sdata,
+ struct sta_info *sta,
+ struct sk_buff *skb)
+{
+ skb_queue_tail(&sdata->skb_queue, skb);
+ ieee80211_queue_work(&sdata->local->hw, &sdata->work);
+ if (sta)
+ sta->rx_stats.packets++;
+}
+
+static void ieee80211_queue_skb_to_iface(struct ieee80211_sub_if_data *sdata,
+ struct sta_info *sta,
+ struct sk_buff *skb)
+{
+ skb->protocol = 0;
+ __ieee80211_queue_skb_to_iface(sdata, sta, skb);
+}
+
static void ieee80211_handle_mu_mimo_mon(struct ieee80211_sub_if_data *sdata,
struct sk_buff *skb,
int rtap_space)
if (!skb)
return;
- skb_queue_tail(&sdata->skb_queue, skb);
- ieee80211_queue_work(&sdata->local->hw, &sdata->work);
+ ieee80211_queue_skb_to_iface(sdata, NULL, skb);
}
/*
struct sk_buff_head *frames)
{
struct sk_buff *skb = rx->skb;
- struct ieee80211_local *local = rx->local;
struct ieee80211_hdr *hdr = (struct ieee80211_hdr *) skb->data;
struct sta_info *sta = rx->sta;
struct tid_ampdu_rx *tid_agg_rx;
/* if this mpdu is fragmented - terminate rx aggregation session */
sc = le16_to_cpu(hdr->seq_ctrl);
if (sc & IEEE80211_SCTL_FRAG) {
- skb_queue_tail(&rx->sdata->skb_queue, skb);
- ieee80211_queue_work(&local->hw, &rx->sdata->work);
+ ieee80211_queue_skb_to_iface(rx->sdata, NULL, skb);
return;
}
for (tid = 0; tid < IEEE80211_NUM_TIDS; tid++) {
struct ieee80211_txq *txq = sta->sta.txq[tid];
- struct txq_info *txqi = to_txq_info(txq);
- spin_lock(&local->active_txq_lock[txq->ac]);
- if (!list_empty(&txqi->schedule_order))
- list_del_init(&txqi->schedule_order);
- spin_unlock(&local->active_txq_lock[txq->ac]);
+ ieee80211_unschedule_txq(&local->hw, txq, false);
if (txq_has_queue(txq))
set_bit(tid, &sta->txq_buffered_tids);
return result;
}
+void ieee80211_init_frag_cache(struct ieee80211_fragment_cache *cache)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(cache->entries); i++)
+ skb_queue_head_init(&cache->entries[i].skb_list);
+}
+
+void ieee80211_destroy_frag_cache(struct ieee80211_fragment_cache *cache)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(cache->entries); i++)
+ __skb_queue_purge(&cache->entries[i].skb_list);
+}
+
static inline struct ieee80211_fragment_entry *
-ieee80211_reassemble_add(struct ieee80211_sub_if_data *sdata,
+ieee80211_reassemble_add(struct ieee80211_fragment_cache *cache,
unsigned int frag, unsigned int seq, int rx_queue,
struct sk_buff **skb)
{
struct ieee80211_fragment_entry *entry;
- entry = &sdata->fragments[sdata->fragment_next++];
- if (sdata->fragment_next >= IEEE80211_FRAGMENT_MAX)
- sdata->fragment_next = 0;
+ entry = &cache->entries[cache->next++];
+ if (cache->next >= IEEE80211_FRAGMENT_MAX)
+ cache->next = 0;
- if (!skb_queue_empty(&entry->skb_list))
- __skb_queue_purge(&entry->skb_list);
+ __skb_queue_purge(&entry->skb_list);
__skb_queue_tail(&entry->skb_list, *skb); /* no need for locking */
*skb = NULL;
}
static inline struct ieee80211_fragment_entry *
-ieee80211_reassemble_find(struct ieee80211_sub_if_data *sdata,
+ieee80211_reassemble_find(struct ieee80211_fragment_cache *cache,
unsigned int frag, unsigned int seq,
int rx_queue, struct ieee80211_hdr *hdr)
{
struct ieee80211_fragment_entry *entry;
int i, idx;
- idx = sdata->fragment_next;
+ idx = cache->next;
for (i = 0; i < IEEE80211_FRAGMENT_MAX; i++) {
struct ieee80211_hdr *f_hdr;
struct sk_buff *f_skb;
if (idx < 0)
idx = IEEE80211_FRAGMENT_MAX - 1;
- entry = &sdata->fragments[idx];
+ entry = &cache->entries[idx];
if (skb_queue_empty(&entry->skb_list) || entry->seq != seq ||
entry->rx_queue != rx_queue ||
entry->last_frag + 1 != frag)
return NULL;
}
+static bool requires_sequential_pn(struct ieee80211_rx_data *rx, __le16 fc)
+{
+ return rx->key &&
+ (rx->key->conf.cipher == WLAN_CIPHER_SUITE_CCMP ||
+ rx->key->conf.cipher == WLAN_CIPHER_SUITE_CCMP_256 ||
+ rx->key->conf.cipher == WLAN_CIPHER_SUITE_GCMP ||
+ rx->key->conf.cipher == WLAN_CIPHER_SUITE_GCMP_256) &&
+ ieee80211_has_protected(fc);
+}
+
static ieee80211_rx_result debug_noinline
ieee80211_rx_h_defragment(struct ieee80211_rx_data *rx)
{
+ struct ieee80211_fragment_cache *cache = &rx->sdata->frags;
struct ieee80211_hdr *hdr;
u16 sc;
__le16 fc;
unsigned int frag, seq;
struct ieee80211_fragment_entry *entry;
struct sk_buff *skb;
+ struct ieee80211_rx_status *status = IEEE80211_SKB_RXCB(rx->skb);
hdr = (struct ieee80211_hdr *)rx->skb->data;
fc = hdr->frame_control;
sc = le16_to_cpu(hdr->seq_ctrl);
frag = sc & IEEE80211_SCTL_FRAG;
- if (is_multicast_ether_addr(hdr->addr1)) {
- I802_DEBUG_INC(rx->local->dot11MulticastReceivedFrameCount);
- goto out_no_led;
- }
+ if (rx->sta)
+ cache = &rx->sta->frags;
if (likely(!ieee80211_has_morefrags(fc) && frag == 0))
goto out;
+ if (is_multicast_ether_addr(hdr->addr1))
+ return RX_DROP_MONITOR;
+
I802_DEBUG_INC(rx->local->rx_handlers_fragments);
if (skb_linearize(rx->skb))
if (frag == 0) {
/* This is the first fragment of a new frame. */
- entry = ieee80211_reassemble_add(rx->sdata, frag, seq,
+ entry = ieee80211_reassemble_add(cache, frag, seq,
rx->seqno_idx, &(rx->skb));
- if (rx->key &&
- (rx->key->conf.cipher == WLAN_CIPHER_SUITE_CCMP ||
- rx->key->conf.cipher == WLAN_CIPHER_SUITE_CCMP_256 ||
- rx->key->conf.cipher == WLAN_CIPHER_SUITE_GCMP ||
- rx->key->conf.cipher == WLAN_CIPHER_SUITE_GCMP_256) &&
- ieee80211_has_protected(fc)) {
+ if (requires_sequential_pn(rx, fc)) {
int queue = rx->security_idx;
/* Store CCMP/GCMP PN so that we can verify that the
* next fragment has a sequential PN value.
*/
entry->check_sequential_pn = true;
+ entry->is_protected = true;
+ entry->key_color = rx->key->color;
memcpy(entry->last_pn,
rx->key->u.ccmp.rx_pn[queue],
IEEE80211_CCMP_PN_LEN);
sizeof(rx->key->u.gcmp.rx_pn[queue]));
BUILD_BUG_ON(IEEE80211_CCMP_PN_LEN !=
IEEE80211_GCMP_PN_LEN);
+ } else if (rx->key &&
+ (ieee80211_has_protected(fc) ||
+ (status->flag & RX_FLAG_DECRYPTED))) {
+ entry->is_protected = true;
+ entry->key_color = rx->key->color;
}
return RX_QUEUED;
}
/* This is a fragment for a frame that should already be pending in
* fragment cache. Add this fragment to the end of the pending entry.
*/
- entry = ieee80211_reassemble_find(rx->sdata, frag, seq,
+ entry = ieee80211_reassemble_find(cache, frag, seq,
rx->seqno_idx, hdr);
if (!entry) {
I802_DEBUG_INC(rx->local->rx_handlers_drop_defrag);
if (entry->check_sequential_pn) {
int i;
u8 pn[IEEE80211_CCMP_PN_LEN], *rpn;
- int queue;
- if (!rx->key ||
- (rx->key->conf.cipher != WLAN_CIPHER_SUITE_CCMP &&
- rx->key->conf.cipher != WLAN_CIPHER_SUITE_CCMP_256 &&
- rx->key->conf.cipher != WLAN_CIPHER_SUITE_GCMP &&
- rx->key->conf.cipher != WLAN_CIPHER_SUITE_GCMP_256))
+ if (!requires_sequential_pn(rx, fc))
+ return RX_DROP_UNUSABLE;
+
+ /* Prevent mixed key and fragment cache attacks */
+ if (entry->key_color != rx->key->color)
return RX_DROP_UNUSABLE;
+
memcpy(pn, entry->last_pn, IEEE80211_CCMP_PN_LEN);
for (i = IEEE80211_CCMP_PN_LEN - 1; i >= 0; i--) {
pn[i]++;
if (pn[i])
break;
}
- queue = rx->security_idx;
- rpn = rx->key->u.ccmp.rx_pn[queue];
+
+ rpn = rx->ccm_gcm.pn;
if (memcmp(pn, rpn, IEEE80211_CCMP_PN_LEN))
return RX_DROP_UNUSABLE;
memcpy(entry->last_pn, pn, IEEE80211_CCMP_PN_LEN);
+ } else if (entry->is_protected &&
+ (!rx->key ||
+ (!ieee80211_has_protected(fc) &&
+ !(status->flag & RX_FLAG_DECRYPTED)) ||
+ rx->key->color != entry->key_color)) {
+ /* Drop this as a mixed key or fragment cache attack, even
+ * if for TKIP Michael MIC should protect us, and WEP is a
+ * lost cause anyway.
+ */
+ return RX_DROP_UNUSABLE;
+ } else if (entry->is_protected && rx->key &&
+ entry->key_color != rx->key->color &&
+ (status->flag & RX_FLAG_DECRYPTED)) {
+ return RX_DROP_UNUSABLE;
}
skb_pull(rx->skb, ieee80211_hdrlen(fc));
out:
ieee80211_led_rx(rx->local);
- out_no_led:
if (rx->sta)
rx->sta->rx_stats.packets++;
return RX_CONTINUE;
struct ethhdr *ehdr = (struct ethhdr *) rx->skb->data;
/*
- * Allow EAPOL frames to us/the PAE group address regardless
- * of whether the frame was encrypted or not.
+ * Allow EAPOL frames to us/the PAE group address regardless of
+ * whether the frame was encrypted or not, and always disallow
+ * all other destination addresses for them.
*/
- if (ehdr->h_proto == rx->sdata->control_port_protocol &&
- (ether_addr_equal(ehdr->h_dest, rx->sdata->vif.addr) ||
- ether_addr_equal(ehdr->h_dest, pae_group_addr)))
- return true;
+ if (unlikely(ehdr->h_proto == rx->sdata->control_port_protocol))
+ return ether_addr_equal(ehdr->h_dest, rx->sdata->vif.addr) ||
+ ether_addr_equal(ehdr->h_dest, pae_group_addr);
if (ieee80211_802_1x_port_control(rx) ||
ieee80211_drop_unencrypted(rx, fc))
cfg80211_rx_control_port(dev, skb, noencrypt);
dev_kfree_skb(skb);
} else {
+ struct ethhdr *ehdr = (void *)skb_mac_header(skb);
+
memset(skb->cb, 0, sizeof(skb->cb));
+ /*
+ * 802.1X over 802.11 requires that the authenticator address
+ * be used for EAPOL frames. However, 802.1X allows the use of
+ * the PAE group address instead. If the interface is part of
+ * a bridge and we pass the frame with the PAE group address,
+ * then the bridge will forward it to the network (even if the
+ * client was not associated yet), which isn't supposed to
+ * happen.
+ * To avoid that, rewrite the destination address to our own
+ * address, so that the authenticator (e.g. hostapd) will see
+ * the frame, but bridge won't forward it anywhere else. Note
+ * that due to earlier filtering, the only other address can
+ * be the PAE group address.
+ */
+ if (unlikely(skb->protocol == sdata->control_port_protocol &&
+ !ether_addr_equal(ehdr->h_dest, sdata->vif.addr)))
+ ether_addr_copy(ehdr->h_dest, sdata->vif.addr);
+
/* deliver to local stack */
if (rx->list)
list_add_tail(&skb->list, rx->list);
if ((sdata->vif.type == NL80211_IFTYPE_AP ||
sdata->vif.type == NL80211_IFTYPE_AP_VLAN) &&
!(sdata->flags & IEEE80211_SDATA_DONT_BRIDGE_PACKETS) &&
+ ehdr->h_proto != rx->sdata->control_port_protocol &&
(sdata->vif.type != NL80211_IFTYPE_AP_VLAN || !sdata->u.vlan.sta)) {
if (is_multicast_ether_addr(ehdr->h_dest) &&
ieee80211_vif_get_num_mcast_if(sdata) != 0) {
if (ieee80211_data_to_8023_exthdr(skb, ðhdr,
rx->sdata->vif.addr,
rx->sdata->vif.type,
- data_offset))
+ data_offset, true))
return RX_DROP_UNUSABLE;
ieee80211_amsdu_to_8023s(skb, &frame_list, dev->dev_addr,
if (is_multicast_ether_addr(hdr->addr1))
return RX_DROP_UNUSABLE;
+ if (rx->key) {
+ /*
+ * We should not receive A-MSDUs on pre-HT connections,
+ * and HT connections cannot use old ciphers. Thus drop
+ * them, as in those cases we couldn't even have SPP
+ * A-MSDUs or such.
+ */
+ switch (rx->key->conf.cipher) {
+ case WLAN_CIPHER_SUITE_WEP40:
+ case WLAN_CIPHER_SUITE_WEP104:
+ case WLAN_CIPHER_SUITE_TKIP:
+ return RX_DROP_UNUSABLE;
+ default:
+ break;
+ }
+ }
+
return __ieee80211_rx_h_amsdu(rx, 0);
}
tf->category == WLAN_CATEGORY_TDLS &&
(tf->action_code == WLAN_TDLS_CHANNEL_SWITCH_REQUEST ||
tf->action_code == WLAN_TDLS_CHANNEL_SWITCH_RESPONSE)) {
- skb_queue_tail(&local->skb_queue_tdls_chsw, rx->skb);
- schedule_work(&local->tdls_chsw_work);
- if (rx->sta)
- rx->sta->rx_stats.packets++;
-
+ rx->skb->protocol = cpu_to_be16(ETH_P_TDLS);
+ __ieee80211_queue_skb_to_iface(sdata, rx->sta, rx->skb);
return RX_QUEUED;
}
}
return RX_QUEUED;
queue:
- skb_queue_tail(&sdata->skb_queue, rx->skb);
- ieee80211_queue_work(&local->hw, &sdata->work);
- if (rx->sta)
- rx->sta->rx_stats.packets++;
+ ieee80211_queue_skb_to_iface(sdata, rx->sta, rx->skb);
return RX_QUEUED;
}
return RX_DROP_MONITOR;
/* for now only beacons are ext, so queue them */
- skb_queue_tail(&sdata->skb_queue, rx->skb);
- ieee80211_queue_work(&rx->local->hw, &sdata->work);
- if (rx->sta)
- rx->sta->rx_stats.packets++;
+ ieee80211_queue_skb_to_iface(sdata, rx->sta, rx->skb);
return RX_QUEUED;
}
return RX_DROP_MONITOR;
}
- /* queue up frame and kick off work to process it */
- skb_queue_tail(&sdata->skb_queue, rx->skb);
- ieee80211_queue_work(&rx->local->hw, &sdata->work);
- if (rx->sta)
- rx->sta->rx_stats.packets++;
+ ieee80211_queue_skb_to_iface(sdata, rx->sta, rx->skb);
return RX_QUEUED;
}
struct ieee80211_mgmt *mgmt = (void *)skb->data;
struct ieee80211_bss *bss;
struct ieee80211_channel *channel;
+ size_t min_hdr_len = offsetof(struct ieee80211_mgmt,
+ u.probe_resp.variable);
+
+ if (!ieee80211_is_probe_resp(mgmt->frame_control) &&
+ !ieee80211_is_beacon(mgmt->frame_control) &&
+ !ieee80211_is_s1g_beacon(mgmt->frame_control))
+ return;
if (ieee80211_is_s1g_beacon(mgmt->frame_control)) {
- if (skb->len < 15)
- return;
- } else if (skb->len < 24 ||
- (!ieee80211_is_probe_resp(mgmt->frame_control) &&
- !ieee80211_is_beacon(mgmt->frame_control)))
+ if (ieee80211_is_s1g_short_beacon(mgmt->frame_control))
+ min_hdr_len = offsetof(struct ieee80211_ext,
+ u.s1g_short_beacon.variable);
+ else
+ min_hdr_len = offsetof(struct ieee80211_ext,
+ u.s1g_beacon);
+ }
+
+ if (skb->len < min_hdr_len)
return;
sdata1 = rcu_dereference(local->scan_sdata);
* Copyright 2006-2007 Jiri Benc <jbenc@suse.cz>
* Copyright 2013-2014 Intel Mobile Communications GmbH
* Copyright (C) 2015 - 2017 Intel Deutschland GmbH
- * Copyright (C) 2018-2020 Intel Corporation
+ * Copyright (C) 2018-2021 Intel Corporation
*/
#include <linux/module.h>
u64_stats_init(&sta->rx_stats.syncp);
+ ieee80211_init_frag_cache(&sta->frags);
+
sta->sta_state = IEEE80211_STA_NONE;
/* Mark TID as unreserved */
if (sta_prepare_rate_control(local, sta, gfp))
goto free_txq;
- sta->airtime_weight = IEEE80211_DEFAULT_AIRTIME_WEIGHT;
for (i = 0; i < IEEE80211_NUM_ACS; i++) {
skb_queue_head_init(&sta->ps_tx_buf[i]);
skb_queue_head_init(&sta->tx_filtered[i]);
- sta->airtime[i].deficit = sta->airtime_weight;
- atomic_set(&sta->airtime[i].aql_tx_pending, 0);
- sta->airtime[i].aql_limit_low = local->aql_txq_limit_low[i];
- sta->airtime[i].aql_limit_high = local->aql_txq_limit_high[i];
+ init_airtime_info(&sta->airtime[i], &local->airtime[i]);
}
for (i = 0; i < IEEE80211_NUM_TIDS; i++)
ieee80211_sta_debugfs_remove(sta);
+ ieee80211_destroy_frag_cache(&sta->frags);
+
cleanup_single_sta(sta);
}
struct ieee80211_tx_info *info;
struct ieee80211_chanctx_conf *chanctx_conf;
- /* Don't send NDPs when STA is connected HE */
- if (sdata->vif.type == NL80211_IFTYPE_STATION &&
- !(sdata->u.mgd.flags & IEEE80211_STA_DISABLE_HE))
- return;
-
if (qos) {
fc = cpu_to_le16(IEEE80211_FTYPE_DATA |
IEEE80211_STYPE_QOS_NULLFUNC |
}
EXPORT_SYMBOL(ieee80211_sta_set_buffered);
-void ieee80211_sta_register_airtime(struct ieee80211_sta *pubsta, u8 tid,
- u32 tx_airtime, u32 rx_airtime)
+void ieee80211_register_airtime(struct ieee80211_txq *txq,
+ u32 tx_airtime, u32 rx_airtime)
{
- struct sta_info *sta = container_of(pubsta, struct sta_info, sta);
- struct ieee80211_local *local = sta->sdata->local;
- u8 ac = ieee80211_ac_from_tid(tid);
+ struct ieee80211_sub_if_data *sdata = vif_to_sdata(txq->vif);
+ struct ieee80211_local *local = sdata->local;
+ u64 weight_sum, weight_sum_reciprocal;
+ struct airtime_sched_info *air_sched;
+ struct airtime_info *air_info;
u32 airtime = 0;
- if (sta->local->airtime_flags & AIRTIME_USE_TX)
+ air_sched = &local->airtime[txq->ac];
+ air_info = to_airtime_info(txq);
+
+ if (local->airtime_flags & AIRTIME_USE_TX)
airtime += tx_airtime;
- if (sta->local->airtime_flags & AIRTIME_USE_RX)
+ if (local->airtime_flags & AIRTIME_USE_RX)
airtime += rx_airtime;
- spin_lock_bh(&local->active_txq_lock[ac]);
- sta->airtime[ac].tx_airtime += tx_airtime;
- sta->airtime[ac].rx_airtime += rx_airtime;
- sta->airtime[ac].deficit -= airtime;
- spin_unlock_bh(&local->active_txq_lock[ac]);
+ /* Weights scale so the unit weight is 256 */
+ airtime <<= 8;
+
+ spin_lock_bh(&air_sched->lock);
+
+ air_info->tx_airtime += tx_airtime;
+ air_info->rx_airtime += rx_airtime;
+
+ if (air_sched->weight_sum) {
+ weight_sum = air_sched->weight_sum;
+ weight_sum_reciprocal = air_sched->weight_sum_reciprocal;
+ } else {
+ weight_sum = air_info->weight;
+ weight_sum_reciprocal = air_info->weight_reciprocal;
+ }
+
+ /* Round the calculation of global vt */
+ air_sched->v_t += (u64)((airtime + (weight_sum >> 1)) *
+ weight_sum_reciprocal) >> IEEE80211_RECIPROCAL_SHIFT_64;
+ air_info->v_t += (u32)((airtime + (air_info->weight >> 1)) *
+ air_info->weight_reciprocal) >> IEEE80211_RECIPROCAL_SHIFT_32;
+ ieee80211_resort_txq(&local->hw, txq);
+
+ spin_unlock_bh(&air_sched->lock);
+}
+
+void ieee80211_sta_register_airtime(struct ieee80211_sta *pubsta, u8 tid,
+ u32 tx_airtime, u32 rx_airtime)
+{
+ struct ieee80211_txq *txq = pubsta->txq[tid];
+
+ if (!txq)
+ return;
+
+ ieee80211_register_airtime(txq, tx_airtime, rx_airtime);
}
EXPORT_SYMBOL(ieee80211_sta_register_airtime);
sta_get_last_rx_stats(struct sta_info *sta)
{
struct ieee80211_sta_rx_stats *stats = &sta->rx_stats;
- struct ieee80211_local *local = sta->local;
int cpu;
- if (!ieee80211_hw_check(&local->hw, USES_RSS))
+ if (!sta->pcpu_rx_stats)
return stats;
for_each_possible_cpu(cpu) {
int cpu;
if (!(tidstats->filled & BIT(NL80211_TID_STATS_RX_MSDU))) {
- if (!ieee80211_hw_check(&local->hw, USES_RSS))
- tidstats->rx_msdu +=
- sta_get_tidstats_msdu(&sta->rx_stats, tid);
+ tidstats->rx_msdu += sta_get_tidstats_msdu(&sta->rx_stats, tid);
if (sta->pcpu_rx_stats) {
for_each_possible_cpu(cpu) {
sinfo->rx_beacon = sdata->u.mgd.count_beacon_signal;
drv_sta_statistics(local, sdata, &sta->sta, sinfo);
-
sinfo->filled |= BIT_ULL(NL80211_STA_INFO_INACTIVE_TIME) |
BIT_ULL(NL80211_STA_INFO_STA_FLAGS) |
BIT_ULL(NL80211_STA_INFO_BSS_PARAM) |
if (!(sinfo->filled & (BIT_ULL(NL80211_STA_INFO_RX_BYTES64) |
BIT_ULL(NL80211_STA_INFO_RX_BYTES)))) {
- if (!ieee80211_hw_check(&local->hw, USES_RSS))
- sinfo->rx_bytes += sta_get_stats_bytes(&sta->rx_stats);
+ sinfo->rx_bytes += sta_get_stats_bytes(&sta->rx_stats);
if (sta->pcpu_rx_stats) {
for_each_possible_cpu(cpu) {
}
if (!(sinfo->filled & BIT_ULL(NL80211_STA_INFO_AIRTIME_WEIGHT))) {
- sinfo->airtime_weight = sta->airtime_weight;
+ sinfo->airtime_weight = sta->airtime[0].weight;
sinfo->filled |= BIT_ULL(NL80211_STA_INFO_AIRTIME_WEIGHT);
}
* Copyright 2002-2005, Devicescape Software, Inc.
* Copyright 2013-2014 Intel Mobile Communications GmbH
* Copyright(c) 2015-2017 Intel Deutschland GmbH
- * Copyright(c) 2020 Intel Corporation
+ * Copyright(c) 2020-2021 Intel Corporation
*/
#ifndef STA_INFO_H
#define AIRTIME_USE_TX BIT(0)
#define AIRTIME_USE_RX BIT(1)
+
struct airtime_info {
u64 rx_airtime;
u64 tx_airtime;
- s64 deficit;
+ u64 v_t;
+ u64 last_scheduled;
+ struct list_head list;
atomic_t aql_tx_pending; /* Estimated airtime for frames pending */
u32 aql_limit_low;
u32 aql_limit_high;
+ u32 weight_reciprocal;
+ u16 weight;
};
void ieee80211_sta_update_pending_airtime(struct ieee80211_local *local,
struct sta_info *sta, u8 ac,
u16 tx_airtime, bool tx_completed);
+void ieee80211_register_airtime(struct ieee80211_txq *txq,
+ u32 tx_airtime, u32 rx_airtime);
struct sta_info;
};
/*
+ * IEEE 802.11-2016 (10.6 "Defragmentation") recommends support for "concurrent
+ * reception of at least one MSDU per access category per associated STA"
+ * on APs, or "at least one MSDU per access category" on other interface types.
+ *
+ * This limit can be increased by changing this define, at the cost of slower
+ * frame reassembly and increased memory use while fragments are pending.
+ */
+#define IEEE80211_FRAGMENT_MAX 4
+
+struct ieee80211_fragment_entry {
+ struct sk_buff_head skb_list;
+ unsigned long first_frag_time;
+ u16 seq;
+ u16 extra_len;
+ u16 last_frag;
+ u8 rx_queue;
+ u8 check_sequential_pn:1, /* needed for CCMP/GCMP */
+ is_protected:1;
+ u8 last_pn[6]; /* PN of the last fragment if CCMP was used */
+ unsigned int key_color;
+};
+
+struct ieee80211_fragment_cache {
+ struct ieee80211_fragment_entry entries[IEEE80211_FRAGMENT_MAX];
+ unsigned int next;
+};
+
+/*
* The bandwidth threshold below which the per-station CoDel parameters will be
* scaled to be more lenient (to prevent starvation of slow stations). This
* value will be scaled by the number of active stations when it is being
* @tid_seq: per-TID sequence numbers for sending to this STA
* @airtime: per-AC struct airtime_info describing airtime statistics for this
* station
- * @airtime_weight: station weight for airtime fairness calculation purposes
* @ampdu_mlme: A-MPDU state machine state
* @mesh: mesh STA information
* @debugfs_dir: debug filesystem directory dentry
* @status_stats.last_ack_signal: last ACK signal
* @status_stats.ack_signal_filled: last ACK signal validity
* @status_stats.avg_ack_signal: average ACK signal
+ * @frags: fragment cache
*/
struct sta_info {
/* General information, mostly static */
u16 tid_seq[IEEE80211_QOS_CTL_TID_MASK + 1];
struct airtime_info airtime[IEEE80211_NUM_ACS];
- u16 airtime_weight;
/*
* Aggregation information, locked with lock.
struct cfg80211_chan_def tdls_chandef;
+ struct ieee80211_fragment_cache frags;
+
/* keep last! */
struct ieee80211_sta sta;
};
if (!(info->flags & IEEE80211_TX_CTL_INJECTED) && acked)
ieee80211_frame_acked(sta, skb);
+ } else if (wiphy_ext_feature_isset(local->hw.wiphy,
+ NL80211_EXT_FEATURE_AIRTIME_FAIRNESS)) {
+ struct ieee80211_sub_if_data *sdata;
+ struct ieee80211_txq *txq;
+ u32 airtime;
+
+ /* Account airtime to multicast queue */
+ sdata = ieee80211_sdata_from_skb(local, skb);
+
+ if (sdata && (txq = sdata->vif.txq)) {
+ airtime = info->status.tx_time ?:
+ ieee80211_calc_expected_tx_airtime(hw,
+ &sdata->vif,
+ NULL,
+ skb->len,
+ false);
+
+ ieee80211_register_airtime(txq, airtime, 0);
+ }
}
/* SNMP counters
ieee80211_hw_check(&local->hw, REPORTS_TX_ACK_STATUS) &&
!(info->flags & IEEE80211_TX_CTL_INJECTED) &&
local->ps_sdata && !(local->scanning)) {
- if (info->flags & IEEE80211_TX_STAT_ACK) {
+ if (info->flags & IEEE80211_TX_STAT_ACK)
local->ps_sdata->u.mgd.flags |=
IEEE80211_STA_NULLFUNC_ACKED;
- } else
- mod_timer(&local->dynamic_ps_timer, jiffies +
- msecs_to_jiffies(10));
+ mod_timer(&local->dynamic_ps_timer,
+ jiffies + msecs_to_jiffies(10));
}
ieee80211_report_used_skb(local, skb, false);
return ret;
}
-static void
+void
ieee80211_process_tdls_channel_switch(struct ieee80211_sub_if_data *sdata,
struct sk_buff *skb)
{
rcu_read_unlock();
}
-void ieee80211_tdls_chsw_work(struct work_struct *wk)
-{
- struct ieee80211_local *local =
- container_of(wk, struct ieee80211_local, tdls_chsw_work);
- struct ieee80211_sub_if_data *sdata;
- struct sk_buff *skb;
- struct ieee80211_tdls_data *tf;
-
- wiphy_lock(local->hw.wiphy);
- while ((skb = skb_dequeue(&local->skb_queue_tdls_chsw))) {
- tf = (struct ieee80211_tdls_data *)skb->data;
- list_for_each_entry(sdata, &local->interfaces, list) {
- if (!ieee80211_sdata_running(sdata) ||
- sdata->vif.type != NL80211_IFTYPE_STATION ||
- !ether_addr_equal(tf->da, sdata->vif.addr))
- continue;
-
- ieee80211_process_tdls_channel_switch(sdata, skb);
- break;
- }
-
- kfree_skb(skb);
- }
- wiphy_unlock(local->hw.wiphy);
-}
-
void ieee80211_tdls_handle_disconnect(struct ieee80211_sub_if_data *sdata,
const u8 *peer, u16 reason)
{
/*
* Portions of this file
* Copyright(c) 2016-2017 Intel Deutschland GmbH
-* Copyright (C) 2018 - 2020 Intel Corporation
+* Copyright (C) 2018 - 2021 Intel Corporation
*/
#if !defined(__MAC80211_DRIVER_TRACE) || defined(TRACE_HEADER_MULTI_READ)
TP_ARGS(local, sta, tids, num_frames, reason, more_data)
);
-TRACE_EVENT(drv_mgd_prepare_tx,
+DECLARE_EVENT_CLASS(mgd_prepare_complete_tx_evt,
TP_PROTO(struct ieee80211_local *local,
struct ieee80211_sub_if_data *sdata,
- u16 duration),
+ u16 duration, u16 subtype, bool success),
- TP_ARGS(local, sdata, duration),
+ TP_ARGS(local, sdata, duration, subtype, success),
TP_STRUCT__entry(
LOCAL_ENTRY
VIF_ENTRY
__field(u32, duration)
+ __field(u16, subtype)
+ __field(u8, success)
),
TP_fast_assign(
LOCAL_ASSIGN;
VIF_ASSIGN;
__entry->duration = duration;
+ __entry->subtype = subtype;
+ __entry->success = success;
),
TP_printk(
- LOCAL_PR_FMT VIF_PR_FMT " duration: %u",
- LOCAL_PR_ARG, VIF_PR_ARG, __entry->duration
+ LOCAL_PR_FMT VIF_PR_FMT " duration: %u, subtype:0x%x, success:%d",
+ LOCAL_PR_ARG, VIF_PR_ARG, __entry->duration,
+ __entry->subtype, __entry->success
)
);
+DEFINE_EVENT(mgd_prepare_complete_tx_evt, drv_mgd_prepare_tx,
+ TP_PROTO(struct ieee80211_local *local,
+ struct ieee80211_sub_if_data *sdata,
+ u16 duration, u16 subtype, bool success),
+
+ TP_ARGS(local, sdata, duration, subtype, success)
+);
+
+DEFINE_EVENT(mgd_prepare_complete_tx_evt, drv_mgd_complete_tx,
+ TP_PROTO(struct ieee80211_local *local,
+ struct ieee80211_sub_if_data *sdata,
+ u16 duration, u16 subtype, bool success),
+
+ TP_ARGS(local, sdata, duration, subtype, success)
+);
+
DEFINE_EVENT(local_sdata_evt, drv_mgd_protect_tdls_discover,
TP_PROTO(struct ieee80211_local *local,
struct ieee80211_sub_if_data *sdata),
#include <linux/bitmap.h>
#include <linux/rcupdate.h>
#include <linux/export.h>
+#include <linux/timekeeping.h>
#include <net/net_namespace.h>
#include <net/ieee80211_radiotap.h>
#include <net/cfg80211.h>
u32 len;
struct ieee80211_tx_rate_control txrc;
struct ieee80211_sta_rates *ratetbl = NULL;
+ bool encap = info->flags & IEEE80211_TX_CTL_HW_80211_ENCAP;
bool assoc = false;
memset(&txrc, 0, sizeof(txrc));
* just wants a probe response.
*/
if (tx->sdata->vif.bss_conf.use_short_preamble &&
- (ieee80211_is_data(hdr->frame_control) ||
+ (ieee80211_is_tx_data(tx->skb) ||
(tx->sta && test_sta_flag(tx->sta, WLAN_STA_SHORT_PREAMBLE))))
txrc.short_preamble = true;
"%s: Dropped data frame as no usable bitrate found while "
"scanning and associated. Target station: "
"%pM on %d GHz band\n",
- tx->sdata->name, hdr->addr1,
+ tx->sdata->name,
+ encap ? ((struct ethhdr *)hdr)->h_dest : hdr->addr1,
info->band ? 5 : 2))
return TX_DROP;
if (txrc.reported_rate.idx < 0) {
txrc.reported_rate = tx->rate;
- if (tx->sta && ieee80211_is_data(hdr->frame_control))
+ if (tx->sta && ieee80211_is_tx_data(tx->skb))
tx->sta->tx_stats.last_rate = txrc.reported_rate;
} else if (tx->sta)
tx->sta->tx_stats.last_rate = txrc.reported_rate;
codel_vars_init(&txqi->def_cvars);
codel_stats_init(&txqi->cstats);
__skb_queue_head_init(&txqi->frags);
- INIT_LIST_HEAD(&txqi->schedule_order);
+ RB_CLEAR_NODE(&txqi->schedule_order);
txqi->txq.vif = &sdata->vif;
ieee80211_purge_tx_queue(&local->hw, &txqi->frags);
spin_unlock_bh(&fq->lock);
- spin_lock_bh(&local->active_txq_lock[txqi->txq.ac]);
- list_del_init(&txqi->schedule_order);
- spin_unlock_bh(&local->active_txq_lock[txqi->txq.ac]);
+ ieee80211_unschedule_txq(&local->hw, &txqi->txq, true);
}
void ieee80211_txq_set_params(struct ieee80211_local *local)
CALL_TXH(ieee80211_tx_h_ps_buf);
CALL_TXH(ieee80211_tx_h_check_control_port_protocol);
CALL_TXH(ieee80211_tx_h_select_key);
- if (!ieee80211_hw_check(&tx->local->hw, HAS_RATE_CONTROL))
- CALL_TXH(ieee80211_tx_h_rate_ctrl);
txh_done:
if (unlikely(res == TX_DROP)) {
goto txh_done;
}
+ if (!ieee80211_hw_check(&tx->local->hw, HAS_RATE_CONTROL))
+ CALL_TXH(ieee80211_tx_h_rate_ctrl);
+
CALL_TXH(ieee80211_tx_h_michael_mic_add);
CALL_TXH(ieee80211_tx_h_sequence);
CALL_TXH(ieee80211_tx_h_fragment);
ieee80211_tx(sdata, sta, skb, false);
}
+static bool ieee80211_validate_radiotap_len(struct sk_buff *skb)
+{
+ struct ieee80211_radiotap_header *rthdr =
+ (struct ieee80211_radiotap_header *)skb->data;
+
+ /* check for not even having the fixed radiotap header part */
+ if (unlikely(skb->len < sizeof(struct ieee80211_radiotap_header)))
+ return false; /* too short to be possibly valid */
+
+ /* is it a header version we can trust to find length from? */
+ if (unlikely(rthdr->it_version))
+ return false; /* only version 0 is supported */
+
+ /* does the skb contain enough to deliver on the alleged length? */
+ if (unlikely(skb->len < ieee80211_get_radiotap_len(skb->data)))
+ return false; /* skb too short for claimed rt header extent */
+
+ return true;
+}
+
bool ieee80211_parse_tx_radiotap(struct sk_buff *skb,
struct net_device *dev)
{
struct ieee80211_radiotap_header *rthdr =
(struct ieee80211_radiotap_header *) skb->data;
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
- struct ieee80211_supported_band *sband =
- local->hw.wiphy->bands[info->band];
int ret = ieee80211_radiotap_iterator_init(&iterator, rthdr, skb->len,
NULL);
u16 txflags;
u8 vht_mcs = 0, vht_nss = 0;
int i;
- /* check for not even having the fixed radiotap header part */
- if (unlikely(skb->len < sizeof(struct ieee80211_radiotap_header)))
- return false; /* too short to be possibly valid */
-
- /* is it a header version we can trust to find length from? */
- if (unlikely(rthdr->it_version))
- return false; /* only version 0 is supported */
-
- /* does the skb contain enough to deliver on the alleged length? */
- if (unlikely(skb->len < ieee80211_get_radiotap_len(skb->data)))
- return false; /* skb too short for claimed rt header extent */
+ if (!ieee80211_validate_radiotap_len(skb))
+ return false;
info->flags |= IEEE80211_TX_INTFL_DONT_ENCRYPT |
IEEE80211_TX_CTL_DONTFRAG;
return false;
if (rate_found) {
+ struct ieee80211_supported_band *sband =
+ local->hw.wiphy->bands[info->band];
+
info->control.flags |= IEEE80211_TX_CTRL_RATE_INJECT;
for (i = 0; i < IEEE80211_TX_MAX_RATES; i++) {
} else if (rate_flags & IEEE80211_TX_RC_VHT_MCS) {
ieee80211_rate_set_vht(info->control.rates, vht_mcs,
vht_nss);
- } else {
+ } else if (sband) {
for (i = 0; i < sband->n_bitrates; i++) {
if (rate * 5 != sband->bitrates[i].bitrate)
continue;
info->flags = IEEE80211_TX_CTL_REQ_TX_STATUS |
IEEE80211_TX_CTL_INJECTED;
- /* Sanity-check and process the injection radiotap header */
- if (!ieee80211_parse_tx_radiotap(skb, dev))
+ /* Sanity-check the length of the radiotap header */
+ if (!ieee80211_validate_radiotap_len(skb))
goto fail;
/* we now know there is a radiotap header with a length we can use */
ieee80211_select_queue_80211(sdata, skb, hdr);
skb_set_queue_mapping(skb, ieee80211_ac_from_tid(skb->priority));
+ /*
+ * Process the radiotap header. This will now take into account the
+ * selected chandef above to accurately set injection rates and
+ * retransmissions.
+ */
+ if (!ieee80211_parse_tx_radiotap(skb, dev))
+ goto fail_rcu;
+
/* remove the injection radiotap header */
skb_pull(skb, len_rthdr);
if (!ieee80211_hw_check(&local->hw, TX_AMSDU))
return false;
+ if (sdata->vif.offload_flags & IEEE80211_OFFLOAD_ENCAP_ENABLED)
+ return false;
+
if (skb_is_gso(skb))
return false;
* Can be called while the sta lock is held. Anything that can cause packets to
* be generated will cause deadlock!
*/
-static void ieee80211_xmit_fast_finish(struct ieee80211_sub_if_data *sdata,
- struct sta_info *sta, u8 pn_offs,
- struct ieee80211_key *key,
- struct sk_buff *skb)
+static ieee80211_tx_result
+ieee80211_xmit_fast_finish(struct ieee80211_sub_if_data *sdata,
+ struct sta_info *sta, u8 pn_offs,
+ struct ieee80211_key *key,
+ struct ieee80211_tx_data *tx)
{
+ struct sk_buff *skb = tx->skb;
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
struct ieee80211_hdr *hdr = (void *)skb->data;
u8 tid = IEEE80211_NUM_TIDS;
+ if (!ieee80211_hw_check(&tx->local->hw, HAS_RATE_CONTROL) &&
+ ieee80211_tx_h_rate_ctrl(tx) != TX_CONTINUE)
+ return TX_DROP;
+
if (key)
info->control.hw_key = &key->conf;
break;
}
}
+
+ return TX_CONTINUE;
}
static bool ieee80211_xmit_fast(struct ieee80211_sub_if_data *sdata,
tx.sta = sta;
tx.key = fast_tx->key;
- if (!ieee80211_hw_check(&local->hw, HAS_RATE_CONTROL)) {
- tx.skb = skb;
- r = ieee80211_tx_h_rate_ctrl(&tx);
- skb = tx.skb;
- tx.skb = NULL;
-
- if (r != TX_CONTINUE) {
- if (r != TX_QUEUED)
- kfree_skb(skb);
- return true;
- }
- }
-
if (ieee80211_queue_skb(local, sdata, sta, skb))
return true;
- ieee80211_xmit_fast_finish(sdata, sta, fast_tx->pn_offs,
- fast_tx->key, skb);
+ tx.skb = skb;
+ r = ieee80211_xmit_fast_finish(sdata, sta, fast_tx->pn_offs,
+ fast_tx->key, &tx);
+ tx.skb = NULL;
+ if (r == TX_DROP) {
+ kfree_skb(skb);
+ return true;
+ }
if (sdata->vif.type == NL80211_IFTYPE_AP_VLAN)
sdata = container_of(sdata->bss,
else
info->flags &= ~IEEE80211_TX_CTL_AMPDU;
- if (info->flags & IEEE80211_TX_CTL_HW_80211_ENCAP)
+ if (info->flags & IEEE80211_TX_CTL_HW_80211_ENCAP) {
+ if (!ieee80211_hw_check(&local->hw, HAS_RATE_CONTROL)) {
+ r = ieee80211_tx_h_rate_ctrl(&tx);
+ if (r != TX_CONTINUE) {
+ ieee80211_free_txskb(&local->hw, skb);
+ goto begin;
+ }
+ }
goto encap_out;
+ }
if (info->control.flags & IEEE80211_TX_CTRL_FAST_XMIT) {
struct sta_info *sta = container_of(txq->sta, struct sta_info,
(tx.key->conf.flags & IEEE80211_KEY_FLAG_GENERATE_IV))
pn_offs = ieee80211_hdrlen(hdr->frame_control);
- ieee80211_xmit_fast_finish(sta->sdata, sta, pn_offs,
- tx.key, skb);
+ r = ieee80211_xmit_fast_finish(sta->sdata, sta, pn_offs,
+ tx.key, &tx);
+ if (r != TX_CONTINUE) {
+ ieee80211_free_txskb(&local->hw, skb);
+ goto begin;
+ }
} else {
if (invoke_tx_handlers_late(&tx))
goto begin;
struct ieee80211_txq *ieee80211_next_txq(struct ieee80211_hw *hw, u8 ac)
{
struct ieee80211_local *local = hw_to_local(hw);
+ struct airtime_sched_info *air_sched;
+ u64 now = ktime_get_boottime_ns();
struct ieee80211_txq *ret = NULL;
- struct txq_info *txqi = NULL, *head = NULL;
- bool found_eligible_txq = false;
+ struct airtime_info *air_info;
+ struct txq_info *txqi = NULL;
+ struct rb_node *node;
+ bool first = false;
- spin_lock_bh(&local->active_txq_lock[ac]);
+ air_sched = &local->airtime[ac];
+ spin_lock_bh(&air_sched->lock);
- begin:
- txqi = list_first_entry_or_null(&local->active_txqs[ac],
- struct txq_info,
- schedule_order);
- if (!txqi)
+ node = air_sched->schedule_pos;
+
+begin:
+ if (!node) {
+ node = rb_first_cached(&air_sched->active_txqs);
+ first = true;
+ } else {
+ node = rb_next(node);
+ }
+
+ if (!node)
goto out;
- if (txqi == head) {
- if (!found_eligible_txq)
- goto out;
- else
- found_eligible_txq = false;
+ txqi = container_of(node, struct txq_info, schedule_order);
+ air_info = to_airtime_info(&txqi->txq);
+
+ if (air_info->v_t > air_sched->v_t &&
+ (!first || !airtime_catchup_v_t(air_sched, air_info->v_t, now)))
+ goto out;
+
+ if (!ieee80211_txq_airtime_check(hw, &txqi->txq)) {
+ first = false;
+ goto begin;
}
- if (!head)
- head = txqi;
+ air_sched->schedule_pos = node;
+ air_sched->last_schedule_activity = now;
+ ret = &txqi->txq;
+out:
+ spin_unlock_bh(&air_sched->lock);
+ return ret;
+}
+EXPORT_SYMBOL(ieee80211_next_txq);
- if (txqi->txq.sta) {
- struct sta_info *sta = container_of(txqi->txq.sta,
- struct sta_info, sta);
- bool aql_check = ieee80211_txq_airtime_check(hw, &txqi->txq);
- s64 deficit = sta->airtime[txqi->txq.ac].deficit;
+static void __ieee80211_insert_txq(struct rb_root_cached *root,
+ struct txq_info *txqi)
+{
+ struct rb_node **new = &root->rb_root.rb_node;
+ struct airtime_info *old_air, *new_air;
+ struct rb_node *parent = NULL;
+ struct txq_info *__txqi;
+ bool leftmost = true;
+
+ while (*new) {
+ parent = *new;
+ __txqi = rb_entry(parent, struct txq_info, schedule_order);
+ old_air = to_airtime_info(&__txqi->txq);
+ new_air = to_airtime_info(&txqi->txq);
+
+ if (new_air->v_t <= old_air->v_t) {
+ new = &parent->rb_left;
+ } else {
+ new = &parent->rb_right;
+ leftmost = false;
+ }
+ }
- if (aql_check)
- found_eligible_txq = true;
+ rb_link_node(&txqi->schedule_order, parent, new);
+ rb_insert_color_cached(&txqi->schedule_order, root, leftmost);
+}
- if (deficit < 0)
- sta->airtime[txqi->txq.ac].deficit +=
- sta->airtime_weight;
+void ieee80211_resort_txq(struct ieee80211_hw *hw,
+ struct ieee80211_txq *txq)
+{
+ struct airtime_info *air_info = to_airtime_info(txq);
+ struct ieee80211_local *local = hw_to_local(hw);
+ struct txq_info *txqi = to_txq_info(txq);
+ struct airtime_sched_info *air_sched;
- if (deficit < 0 || !aql_check) {
- list_move_tail(&txqi->schedule_order,
- &local->active_txqs[txqi->txq.ac]);
- goto begin;
+ air_sched = &local->airtime[txq->ac];
+
+ lockdep_assert_held(&air_sched->lock);
+
+ if (!RB_EMPTY_NODE(&txqi->schedule_order)) {
+ struct airtime_info *a_prev = NULL, *a_next = NULL;
+ struct txq_info *t_prev, *t_next;
+ struct rb_node *n_prev, *n_next;
+
+ /* Erasing a node can cause an expensive rebalancing operation,
+ * so we check the previous and next nodes first and only remove
+ * and re-insert if the current node is not already in the
+ * correct position.
+ */
+ if ((n_prev = rb_prev(&txqi->schedule_order)) != NULL) {
+ t_prev = container_of(n_prev, struct txq_info,
+ schedule_order);
+ a_prev = to_airtime_info(&t_prev->txq);
+ }
+
+ if ((n_next = rb_next(&txqi->schedule_order)) != NULL) {
+ t_next = container_of(n_next, struct txq_info,
+ schedule_order);
+ a_next = to_airtime_info(&t_next->txq);
}
+
+ if ((!a_prev || a_prev->v_t <= air_info->v_t) &&
+ (!a_next || a_next->v_t > air_info->v_t))
+ return;
+
+ if (air_sched->schedule_pos == &txqi->schedule_order)
+ air_sched->schedule_pos = n_prev;
+
+ rb_erase_cached(&txqi->schedule_order,
+ &air_sched->active_txqs);
+ RB_CLEAR_NODE(&txqi->schedule_order);
+ __ieee80211_insert_txq(&air_sched->active_txqs, txqi);
}
+}
+
+void ieee80211_update_airtime_weight(struct ieee80211_local *local,
+ struct airtime_sched_info *air_sched,
+ u64 now, bool force)
+{
+ struct airtime_info *air_info, *tmp;
+ u64 weight_sum = 0;
+
+ if (unlikely(!now))
+ now = ktime_get_boottime_ns();
+ lockdep_assert_held(&air_sched->lock);
+
+ if (!force && (air_sched->last_weight_update <
+ now - AIRTIME_ACTIVE_DURATION))
+ return;
+
+ list_for_each_entry_safe(air_info, tmp,
+ &air_sched->active_list, list) {
+ if (airtime_is_active(air_info, now))
+ weight_sum += air_info->weight;
+ else
+ list_del_init(&air_info->list);
+ }
+ airtime_weight_sum_set(air_sched, weight_sum);
+ air_sched->last_weight_update = now;
+}
- if (txqi->schedule_round == local->schedule_round[ac])
+void ieee80211_schedule_txq(struct ieee80211_hw *hw,
+ struct ieee80211_txq *txq)
+ __acquires(txq_lock) __releases(txq_lock)
+{
+ struct ieee80211_local *local = hw_to_local(hw);
+ struct txq_info *txqi = to_txq_info(txq);
+ struct airtime_sched_info *air_sched;
+ u64 now = ktime_get_boottime_ns();
+ struct airtime_info *air_info;
+ u8 ac = txq->ac;
+ bool was_active;
+
+ air_sched = &local->airtime[ac];
+ air_info = to_airtime_info(txq);
+
+ spin_lock_bh(&air_sched->lock);
+ was_active = airtime_is_active(air_info, now);
+ airtime_set_active(air_sched, air_info, now);
+
+ if (!RB_EMPTY_NODE(&txqi->schedule_order))
goto out;
- list_del_init(&txqi->schedule_order);
- txqi->schedule_round = local->schedule_round[ac];
- ret = &txqi->txq;
+ /* If the station has been inactive for a while, catch up its v_t so it
+ * doesn't get indefinite priority; see comment above the definition of
+ * AIRTIME_MAX_BEHIND.
+ */
+ if ((!was_active && air_info->v_t < air_sched->v_t) ||
+ air_info->v_t < air_sched->v_t - AIRTIME_MAX_BEHIND)
+ air_info->v_t = air_sched->v_t;
+
+ ieee80211_update_airtime_weight(local, air_sched, now, !was_active);
+ __ieee80211_insert_txq(&air_sched->active_txqs, txqi);
out:
- spin_unlock_bh(&local->active_txq_lock[ac]);
- return ret;
+ spin_unlock_bh(&air_sched->lock);
}
-EXPORT_SYMBOL(ieee80211_next_txq);
+EXPORT_SYMBOL(ieee80211_schedule_txq);
-void __ieee80211_schedule_txq(struct ieee80211_hw *hw,
- struct ieee80211_txq *txq,
- bool force)
+static void __ieee80211_unschedule_txq(struct ieee80211_hw *hw,
+ struct ieee80211_txq *txq,
+ bool purge)
{
struct ieee80211_local *local = hw_to_local(hw);
struct txq_info *txqi = to_txq_info(txq);
+ struct airtime_sched_info *air_sched;
+ struct airtime_info *air_info;
- spin_lock_bh(&local->active_txq_lock[txq->ac]);
-
- if (list_empty(&txqi->schedule_order) &&
- (force || !skb_queue_empty(&txqi->frags) ||
- txqi->tin.backlog_packets)) {
- /* If airtime accounting is active, always enqueue STAs at the
- * head of the list to ensure that they only get moved to the
- * back by the airtime DRR scheduler once they have a negative
- * deficit. A station that already has a negative deficit will
- * get immediately moved to the back of the list on the next
- * call to ieee80211_next_txq().
- */
- if (txqi->txq.sta && local->airtime_flags &&
- wiphy_ext_feature_isset(local->hw.wiphy,
- NL80211_EXT_FEATURE_AIRTIME_FAIRNESS))
- list_add(&txqi->schedule_order,
- &local->active_txqs[txq->ac]);
- else
- list_add_tail(&txqi->schedule_order,
- &local->active_txqs[txq->ac]);
+ air_sched = &local->airtime[txq->ac];
+ air_info = to_airtime_info(&txqi->txq);
+
+ lockdep_assert_held(&air_sched->lock);
+
+ if (purge) {
+ list_del_init(&air_info->list);
+ ieee80211_update_airtime_weight(local, air_sched, 0, true);
}
- spin_unlock_bh(&local->active_txq_lock[txq->ac]);
+ if (RB_EMPTY_NODE(&txqi->schedule_order))
+ return;
+
+ if (air_sched->schedule_pos == &txqi->schedule_order)
+ air_sched->schedule_pos = rb_prev(&txqi->schedule_order);
+
+ if (!purge)
+ airtime_set_active(air_sched, air_info,
+ ktime_get_boottime_ns());
+
+ rb_erase_cached(&txqi->schedule_order,
+ &air_sched->active_txqs);
+ RB_CLEAR_NODE(&txqi->schedule_order);
}
-EXPORT_SYMBOL(__ieee80211_schedule_txq);
+
+void ieee80211_unschedule_txq(struct ieee80211_hw *hw,
+ struct ieee80211_txq *txq,
+ bool purge)
+ __acquires(txq_lock) __releases(txq_lock)
+{
+ struct ieee80211_local *local = hw_to_local(hw);
+
+ spin_lock_bh(&local->airtime[txq->ac].lock);
+ __ieee80211_unschedule_txq(hw, txq, purge);
+ spin_unlock_bh(&local->airtime[txq->ac].lock);
+}
+
+void ieee80211_return_txq(struct ieee80211_hw *hw,
+ struct ieee80211_txq *txq, bool force)
+{
+ struct ieee80211_local *local = hw_to_local(hw);
+ struct txq_info *txqi = to_txq_info(txq);
+
+ spin_lock_bh(&local->airtime[txq->ac].lock);
+
+ if (!RB_EMPTY_NODE(&txqi->schedule_order) && !force &&
+ !txq_has_queue(txq))
+ __ieee80211_unschedule_txq(hw, txq, false);
+
+ spin_unlock_bh(&local->airtime[txq->ac].lock);
+}
+EXPORT_SYMBOL(ieee80211_return_txq);
DEFINE_STATIC_KEY_FALSE(aql_disable);
bool ieee80211_txq_airtime_check(struct ieee80211_hw *hw,
struct ieee80211_txq *txq)
{
- struct sta_info *sta;
+ struct airtime_info *air_info = to_airtime_info(txq);
struct ieee80211_local *local = hw_to_local(hw);
if (!wiphy_ext_feature_isset(local->hw.wiphy, NL80211_EXT_FEATURE_AQL))
if (unlikely(txq->tid == IEEE80211_NUM_TIDS))
return true;
- sta = container_of(txq->sta, struct sta_info, sta);
- if (atomic_read(&sta->airtime[txq->ac].aql_tx_pending) <
- sta->airtime[txq->ac].aql_limit_low)
+ if (atomic_read(&air_info->aql_tx_pending) < air_info->aql_limit_low)
return true;
if (atomic_read(&local->aql_total_pending_airtime) <
local->aql_threshold &&
- atomic_read(&sta->airtime[txq->ac].aql_tx_pending) <
- sta->airtime[txq->ac].aql_limit_high)
+ atomic_read(&air_info->aql_tx_pending) < air_info->aql_limit_high)
return true;
return false;
bool ieee80211_txq_may_transmit(struct ieee80211_hw *hw,
struct ieee80211_txq *txq)
{
+ struct txq_info *first_txqi = NULL, *txqi = to_txq_info(txq);
struct ieee80211_local *local = hw_to_local(hw);
- struct txq_info *iter, *tmp, *txqi = to_txq_info(txq);
- struct sta_info *sta;
- u8 ac = txq->ac;
+ struct airtime_sched_info *air_sched;
+ struct airtime_info *air_info;
+ struct rb_node *node = NULL;
+ bool ret = false;
+ u64 now;
- spin_lock_bh(&local->active_txq_lock[ac]);
- if (!txqi->txq.sta)
- goto out;
+ if (!ieee80211_txq_airtime_check(hw, txq))
+ return false;
+
+ air_sched = &local->airtime[txq->ac];
+ spin_lock_bh(&air_sched->lock);
- if (list_empty(&txqi->schedule_order))
+ if (RB_EMPTY_NODE(&txqi->schedule_order))
goto out;
- list_for_each_entry_safe(iter, tmp, &local->active_txqs[ac],
- schedule_order) {
- if (iter == txqi)
- break;
+ now = ktime_get_boottime_ns();
- if (!iter->txq.sta) {
- list_move_tail(&iter->schedule_order,
- &local->active_txqs[ac]);
- continue;
- }
- sta = container_of(iter->txq.sta, struct sta_info, sta);
- if (sta->airtime[ac].deficit < 0)
- sta->airtime[ac].deficit += sta->airtime_weight;
- list_move_tail(&iter->schedule_order, &local->active_txqs[ac]);
- }
+ /* Like in ieee80211_next_txq(), make sure the first station in the
+ * scheduling order is eligible for transmission to avoid starvation.
+ */
+ node = rb_first_cached(&air_sched->active_txqs);
+ if (node) {
+ first_txqi = container_of(node, struct txq_info,
+ schedule_order);
+ air_info = to_airtime_info(&first_txqi->txq);
- sta = container_of(txqi->txq.sta, struct sta_info, sta);
- if (sta->airtime[ac].deficit >= 0)
- goto out;
+ if (air_sched->v_t < air_info->v_t)
+ airtime_catchup_v_t(air_sched, air_info->v_t, now);
+ }
- sta->airtime[ac].deficit += sta->airtime_weight;
- list_move_tail(&txqi->schedule_order, &local->active_txqs[ac]);
- spin_unlock_bh(&local->active_txq_lock[ac]);
+ air_info = to_airtime_info(&txqi->txq);
+ if (air_info->v_t <= air_sched->v_t) {
+ air_sched->last_schedule_activity = now;
+ ret = true;
+ }
- return false;
out:
- if (!list_empty(&txqi->schedule_order))
- list_del_init(&txqi->schedule_order);
- spin_unlock_bh(&local->active_txq_lock[ac]);
-
- return true;
+ spin_unlock_bh(&air_sched->lock);
+ return ret;
}
EXPORT_SYMBOL(ieee80211_txq_may_transmit);
void ieee80211_txq_schedule_start(struct ieee80211_hw *hw, u8 ac)
{
struct ieee80211_local *local = hw_to_local(hw);
+ struct airtime_sched_info *air_sched = &local->airtime[ac];
- spin_lock_bh(&local->active_txq_lock[ac]);
- local->schedule_round[ac]++;
- spin_unlock_bh(&local->active_txq_lock[ac]);
+ spin_lock_bh(&air_sched->lock);
+ air_sched->schedule_pos = NULL;
+ spin_unlock_bh(&air_sched->lock);
}
EXPORT_SYMBOL(ieee80211_txq_schedule_start);
+static void
+ieee80211_aggr_check(struct ieee80211_sub_if_data *sdata,
+ struct sta_info *sta,
+ struct sk_buff *skb)
+{
+ struct rate_control_ref *ref = sdata->local->rate_ctrl;
+ u16 tid;
+
+ if (!ref || !(ref->ops->capa & RATE_CTRL_CAPA_AMPDU_TRIGGER))
+ return;
+
+ if (!sta || !sta->sta.ht_cap.ht_supported ||
+ !sta->sta.wme || skb_get_queue_mapping(skb) == IEEE80211_AC_VO ||
+ skb->protocol == sdata->control_port_protocol)
+ return;
+
+ tid = skb->priority & IEEE80211_QOS_CTL_TID_MASK;
+ if (likely(sta->ampdu_mlme.tid_tx[tid]))
+ return;
+
+ ieee80211_start_tx_ba_session(&sta->sta, tid, 0);
+}
+
void __ieee80211_subif_start_xmit(struct sk_buff *skb,
struct net_device *dev,
u32 info_flags,
skb_get_hash(skb);
}
+ ieee80211_aggr_check(sdata, sta, skb);
+
if (sta) {
struct ieee80211_fast_tx *fast_tx;
memset(info, 0, sizeof(*info));
+ ieee80211_aggr_check(sdata, sta, skb);
+
tid = skb->priority & IEEE80211_QOS_CTL_TAG1D_MASK;
tid_tx = rcu_dereference(sta->ampdu_mlme.tid_tx[tid]);
if (tid_tx) {
* Copyright 2007 Johannes Berg <johannes@sipsolutions.net>
* Copyright 2013-2014 Intel Mobile Communications GmbH
* Copyright (C) 2015-2017 Intel Deutschland GmbH
- * Copyright (C) 2018-2020 Intel Corporation
+ * Copyright (C) 2018-2021 Intel Corporation
*
* utilities for mac80211
*/
switch (elem->data[0]) {
case WLAN_EID_EXT_HE_MU_EDCA:
- if (len == sizeof(*elems->mu_edca_param_set)) {
+ if (len >= sizeof(*elems->mu_edca_param_set)) {
elems->mu_edca_param_set = data;
if (crc)
*crc = crc32_be(*crc, (void *)elem,
}
break;
case WLAN_EID_EXT_UORA:
- if (len == 1)
+ if (len >= 1)
elems->uora_element = data;
break;
case WLAN_EID_EXT_MAX_CHANNEL_SWITCH_TIME:
elems->max_channel_switch_time = data;
break;
case WLAN_EID_EXT_MULTIPLE_BSSID_CONFIGURATION:
- if (len == sizeof(*elems->mbssid_config_ie))
+ if (len >= sizeof(*elems->mbssid_config_ie))
elems->mbssid_config_ie = data;
break;
case WLAN_EID_EXT_HE_SPR:
elems->he_spr = data;
break;
case WLAN_EID_EXT_HE_6GHZ_CAPA:
- if (len == sizeof(*elems->he_6ghz_capa))
+ if (len >= sizeof(*elems->he_6ghz_capa))
elems->he_6ghz_capa = data;
break;
}
switch (id) {
case WLAN_EID_LINK_ID:
- if (elen + 2 != sizeof(struct ieee80211_tdls_lnkie)) {
+ if (elen + 2 < sizeof(struct ieee80211_tdls_lnkie)) {
elem_parse_failed = true;
break;
}
elems->lnk_id = (void *)(pos - 2);
break;
case WLAN_EID_CHAN_SWITCH_TIMING:
- if (elen != sizeof(struct ieee80211_ch_switch_timing)) {
+ if (elen < sizeof(struct ieee80211_ch_switch_timing)) {
elem_parse_failed = true;
break;
}
elems->sec_chan_offs = (void *)pos;
break;
case WLAN_EID_CHAN_SWITCH_PARAM:
- if (elen !=
+ if (elen <
sizeof(*elems->mesh_chansw_params_ie)) {
elem_parse_failed = true;
break;
break;
case WLAN_EID_WIDE_BW_CHANNEL_SWITCH:
if (!action ||
- elen != sizeof(*elems->wide_bw_chansw_ie)) {
+ elen < sizeof(*elems->wide_bw_chansw_ie)) {
elem_parse_failed = true;
break;
}
ie = cfg80211_find_ie(WLAN_EID_WIDE_BW_CHANNEL_SWITCH,
pos, elen);
if (ie) {
- if (ie[1] == sizeof(*elems->wide_bw_chansw_ie))
+ if (ie[1] >= sizeof(*elems->wide_bw_chansw_ie))
elems->wide_bw_chansw_ie =
(void *)(ie + 2);
else
elems->cisco_dtpc_elem = pos;
break;
case WLAN_EID_ADDBA_EXT:
- if (elen != sizeof(struct ieee80211_addba_ext_ie)) {
+ if (elen < sizeof(struct ieee80211_addba_ext_ie)) {
elem_parse_failed = true;
break;
}
elem, elems);
break;
case WLAN_EID_S1G_CAPABILITIES:
- if (elen == sizeof(*elems->s1g_capab))
+ if (elen >= sizeof(*elems->s1g_capab))
elems->s1g_capab = (void *)pos;
else
elem_parse_failed = true;
if (auth_alg == WLAN_AUTH_SHARED_KEY && transaction == 3) {
mgmt->frame_control |= cpu_to_le16(IEEE80211_FCTL_PROTECTED);
err = ieee80211_wep_encrypt(local, skb, key, key_len, key_idx);
- WARN_ON(err);
+ if (WARN_ON(err)) {
+ kfree_skb(skb);
+ return;
+ }
}
IEEE80211_SKB_CB(skb)->flags |= IEEE80211_TX_INTFL_DONT_ENCRYPT |
*offset = noffset;
}
- he_cap = ieee80211_get_he_sta_cap(sband);
- if (he_cap) {
+ he_cap = ieee80211_get_he_iftype_cap(sband,
+ ieee80211_vif_type_p2p(&sdata->vif));
+ if (he_cap &&
+ cfg80211_any_usable_channels(local->hw.wiphy, BIT(sband->band),
+ IEEE80211_CHAN_NO_HE)) {
pos = ieee80211_ie_build_he_cap(pos, he_cap, end);
if (!pos)
goto out_err;
+ }
+
+ if (cfg80211_any_usable_channels(local->hw.wiphy,
+ BIT(NL80211_BAND_6GHZ),
+ IEEE80211_CHAN_NO_HE)) {
+ struct ieee80211_supported_band *sband6;
+
+ sband6 = local->hw.wiphy->bands[NL80211_BAND_6GHZ];
+ he_cap = ieee80211_get_he_iftype_cap(sband6,
+ ieee80211_vif_type_p2p(&sdata->vif));
- if (sband->band == NL80211_BAND_6GHZ) {
+ if (he_cap) {
enum nl80211_iftype iftype =
ieee80211_vif_type_p2p(&sdata->vif);
__le16 cap = ieee80211_get_he_6ghz_capa(sband, iftype);
list_for_each_entry(ctx, &local->chanctx_list, list)
ctx->driver_present = false;
mutex_unlock(&local->chanctx_mtx);
-
- cfg80211_shutdown_all_interfaces(local->hw.wiphy);
}
static void ieee80211_assign_chanctx(struct ieee80211_local *local,
u8 *pos;
u16 cap;
- sband = ieee80211_get_sband(sdata);
- if (!sband)
+ if (!cfg80211_any_usable_channels(sdata->local->hw.wiphy,
+ BIT(NL80211_BAND_6GHZ),
+ IEEE80211_CHAN_NO_HE))
return;
+ sband = sdata->local->hw.wiphy->bands[NL80211_BAND_6GHZ];
+
iftd = ieee80211_get_sband_iftype_data(sband, iftype);
- if (WARN_ON(!iftd))
+ if (!iftd)
return;
/* Check for device HE 6 GHz capability before adding element */
* Copyright 2002-2004, Instant802 Networks, Inc.
* Copyright 2008, Jouni Malinen <j@w1.fi>
* Copyright (C) 2016-2017 Intel Deutschland GmbH
+ * Copyright (C) 2020-2021 Intel Corporation
*/
#include <linux/netdevice.h>
update_iv:
/* update IV in key information to be able to detect replays */
- rx->key->u.tkip.rx[rx->security_idx].iv32 = rx->tkip_iv32;
- rx->key->u.tkip.rx[rx->security_idx].iv16 = rx->tkip_iv16;
+ rx->key->u.tkip.rx[rx->security_idx].iv32 = rx->tkip.iv32;
+ rx->key->u.tkip.rx[rx->security_idx].iv16 = rx->tkip.iv16;
return RX_CONTINUE;
key, skb->data + hdrlen,
skb->len - hdrlen, rx->sta->sta.addr,
hdr->addr1, hwaccel, rx->security_idx,
- &rx->tkip_iv32,
- &rx->tkip_iv16);
+ &rx->tkip.iv32,
+ &rx->tkip.iv16);
if (res != TKIP_DECRYPT_OK)
return RX_DROP_UNUSABLE;
}
memcpy(key->u.ccmp.rx_pn[queue], pn, IEEE80211_CCMP_PN_LEN);
+ if (unlikely(ieee80211_is_frag(hdr)))
+ memcpy(rx->ccm_gcm.pn, pn, IEEE80211_CCMP_PN_LEN);
}
/* Remove CCMP header and MIC */
}
memcpy(key->u.gcmp.rx_pn[queue], pn, IEEE80211_GCMP_PN_LEN);
+ if (unlikely(ieee80211_is_frag(hdr)))
+ memcpy(rx->ccm_gcm.pn, pn, IEEE80211_CCMP_PN_LEN);
}
/* Remove GCMP header and MIC */
* Copyright (c) 2019, Tessares SA.
*/
+#ifdef CONFIG_SYSCTL
#include <linux/sysctl.h>
+#endif
#include <net/net_namespace.h>
#include <net/netns/generic.h>
static int mptcp_pernet_id;
struct mptcp_pernet {
+#ifdef CONFIG_SYSCTL
struct ctl_table_header *ctl_table_hdr;
+#endif
- int mptcp_enabled;
+ u8 mptcp_enabled;
unsigned int add_addr_timeout;
+ u8 checksum_enabled;
+ u8 allow_join_initial_addr_port;
};
static struct mptcp_pernet *mptcp_get_pernet(struct net *net)
return mptcp_get_pernet(net)->add_addr_timeout;
}
+int mptcp_is_checksum_enabled(struct net *net)
+{
+ return mptcp_get_pernet(net)->checksum_enabled;
+}
+
+int mptcp_allow_join_id0(struct net *net)
+{
+ return mptcp_get_pernet(net)->allow_join_initial_addr_port;
+}
+
+static void mptcp_pernet_set_defaults(struct mptcp_pernet *pernet)
+{
+ pernet->mptcp_enabled = 1;
+ pernet->add_addr_timeout = TCP_RTO_MAX;
+ pernet->checksum_enabled = 0;
+ pernet->allow_join_initial_addr_port = 1;
+}
+
+#ifdef CONFIG_SYSCTL
static struct ctl_table mptcp_sysctl_table[] = {
{
.procname = "enabled",
- .maxlen = sizeof(int),
+ .maxlen = sizeof(u8),
.mode = 0644,
/* users with CAP_NET_ADMIN or root (not and) can change this
* value, same as other sysctl or the 'net' tree.
*/
- .proc_handler = proc_dointvec,
+ .proc_handler = proc_dou8vec_minmax,
+ .extra1 = SYSCTL_ZERO,
+ .extra2 = SYSCTL_ONE
},
{
.procname = "add_addr_timeout",
.mode = 0644,
.proc_handler = proc_dointvec_jiffies,
},
+ {
+ .procname = "checksum_enabled",
+ .maxlen = sizeof(u8),
+ .mode = 0644,
+ .proc_handler = proc_dou8vec_minmax,
+ .extra1 = SYSCTL_ZERO,
+ .extra2 = SYSCTL_ONE
+ },
+ {
+ .procname = "allow_join_initial_addr_port",
+ .maxlen = sizeof(u8),
+ .mode = 0644,
+ .proc_handler = proc_dou8vec_minmax,
+ .extra1 = SYSCTL_ZERO,
+ .extra2 = SYSCTL_ONE
+ },
{}
};
-static void mptcp_pernet_set_defaults(struct mptcp_pernet *pernet)
-{
- pernet->mptcp_enabled = 1;
- pernet->add_addr_timeout = TCP_RTO_MAX;
-}
-
static int mptcp_pernet_new_table(struct net *net, struct mptcp_pernet *pernet)
{
struct ctl_table_header *hdr;
table[0].data = &pernet->mptcp_enabled;
table[1].data = &pernet->add_addr_timeout;
+ table[2].data = &pernet->checksum_enabled;
+ table[3].data = &pernet->allow_join_initial_addr_port;
hdr = register_net_sysctl(net, MPTCP_SYSCTL_PATH, table);
if (!hdr)
kfree(table);
}
+#else
+
+static int mptcp_pernet_new_table(struct net *net, struct mptcp_pernet *pernet)
+{
+ return 0;
+}
+
+static void mptcp_pernet_del_table(struct mptcp_pernet *pernet) {}
+
+#endif /* CONFIG_SYSCTL */
+
static int __net_init mptcp_net_init(struct net *net)
{
struct mptcp_pernet *pernet = mptcp_get_pernet(net);
SNMP_MIB_ITEM("MPJoinAckHMacFailure", MPTCP_MIB_JOINACKMAC),
SNMP_MIB_ITEM("DSSNotMatching", MPTCP_MIB_DSSNOMATCH),
SNMP_MIB_ITEM("InfiniteMapRx", MPTCP_MIB_INFINITEMAPRX),
+ SNMP_MIB_ITEM("DSSNoMatchTCP", MPTCP_MIB_DSSTCPMISMATCH),
+ SNMP_MIB_ITEM("DataCsumErr", MPTCP_MIB_DATACSUMERR),
SNMP_MIB_ITEM("OFOQueueTail", MPTCP_MIB_OFOQUEUETAIL),
SNMP_MIB_ITEM("OFOQueue", MPTCP_MIB_OFOQUEUE),
SNMP_MIB_ITEM("OFOMerge", MPTCP_MIB_OFOMERGE),
MPTCP_MIB_JOINACKMAC, /* HMAC was wrong on ACK + MP_JOIN */
MPTCP_MIB_DSSNOMATCH, /* Received a new mapping that did not match the previous one */
MPTCP_MIB_INFINITEMAPRX, /* Received an infinite mapping */
+ MPTCP_MIB_DSSTCPMISMATCH, /* DSS-mapping did not map with TCP's sequence numbers */
+ MPTCP_MIB_DATACSUMERR, /* The data checksum fail */
MPTCP_MIB_OFOQUEUETAIL, /* Segments inserted into OoO queue tail */
MPTCP_MIB_OFOQUEUE, /* Segments inserted into OoO queue */
MPTCP_MIB_OFOMERGE, /* Segments merged in OoO queue */
info->mptcpi_write_seq = READ_ONCE(msk->write_seq);
info->mptcpi_snd_una = READ_ONCE(msk->snd_una);
info->mptcpi_rcv_nxt = READ_ONCE(msk->ack_seq);
+ info->mptcpi_csum_enabled = READ_ONCE(msk->csum_enabled);
unlock_sock_fast(sk, slow);
}
else
expected_opsize = TCPOLEN_MPTCP_MPC_SYN;
}
- if (opsize != expected_opsize)
+
+ /* Cfr RFC 8684 Section 3.3.0:
+ * If a checksum is present but its use had
+ * not been negotiated in the MP_CAPABLE handshake, the receiver MUST
+ * close the subflow with a RST, as it is not behaving as negotiated.
+ * If a checksum is not present when its use has been negotiated, the
+ * receiver MUST close the subflow with a RST, as it is considered
+ * broken
+ * We parse even option with mismatching csum presence, so that
+ * later in subflow_data_ready we can trigger the reset.
+ */
+ if (opsize != expected_opsize &&
+ (expected_opsize != TCPOLEN_MPTCP_MPC_ACK_DATA ||
+ opsize != TCPOLEN_MPTCP_MPC_ACK_DATA_CSUM))
break;
/* try to be gentle vs future versions on the initial syn */
* host requires the use of checksums, checksums MUST be used.
* In other words, the only way for checksums not to be used
* is if both hosts in their SYNs set A=0."
- *
- * Section 3.3.0:
- * "If a checksum is not present when its use has been
- * negotiated, the receiver MUST close the subflow with a RST as
- * it is considered broken."
- *
- * We don't implement DSS checksum - fall back to TCP.
*/
if (flags & MPTCP_CAP_CHECKSUM_REQD)
- break;
+ mp_opt->csum_reqd = 1;
+
+ if (flags & MPTCP_CAP_DENY_JOIN_ID0)
+ mp_opt->deny_join_id0 = 1;
mp_opt->mp_capable = 1;
if (opsize >= TCPOLEN_MPTCP_MPC_SYNACK) {
mp_opt->rcvr_key = get_unaligned_be64(ptr);
ptr += 8;
}
- if (opsize == TCPOLEN_MPTCP_MPC_ACK_DATA) {
+ if (opsize >= TCPOLEN_MPTCP_MPC_ACK_DATA) {
/* Section 3.1.:
* "the data parameters in a MP_CAPABLE are semantically
* equivalent to those in a DSS option and can be used
mp_opt->data_len = get_unaligned_be16(ptr);
ptr += 2;
}
- pr_debug("MP_CAPABLE version=%x, flags=%x, optlen=%d sndr=%llu, rcvr=%llu len=%d",
+ if (opsize == TCPOLEN_MPTCP_MPC_ACK_DATA_CSUM) {
+ mp_opt->csum = (__force __sum16)get_unaligned_be16(ptr);
+ mp_opt->csum_reqd = 1;
+ ptr += 2;
+ }
+ pr_debug("MP_CAPABLE version=%x, flags=%x, optlen=%d sndr=%llu, rcvr=%llu len=%d csum=%u",
version, flags, opsize, mp_opt->sndr_key,
- mp_opt->rcvr_key, mp_opt->data_len);
+ mp_opt->rcvr_key, mp_opt->data_len, mp_opt->csum);
break;
case MPTCPOPT_MP_JOIN:
memcpy(mp_opt->hmac, ptr, MPTCPOPT_HMAC_LEN);
pr_debug("MP_JOIN hmac");
} else {
- pr_warn("MP_JOIN bad option size");
mp_opt->mp_join = 0;
}
break;
expected_opsize += TCPOLEN_MPTCP_DSS_MAP32;
}
- /* RFC 6824, Section 3.3:
- * If a checksum is present, but its use had
- * not been negotiated in the MP_CAPABLE handshake,
- * the checksum field MUST be ignored.
+ /* Always parse any csum presence combination, we will enforce
+ * RFC 8684 Section 3.3.0 checks later in subflow_data_ready
*/
if (opsize != expected_opsize &&
opsize != expected_opsize + TCPOLEN_MPTCP_DSS_CHECKSUM)
mp_opt->data_len = get_unaligned_be16(ptr);
ptr += 2;
- pr_debug("data_seq=%llu subflow_seq=%u data_len=%u",
+ if (opsize == expected_opsize + TCPOLEN_MPTCP_DSS_CHECKSUM) {
+ mp_opt->csum_reqd = 1;
+ mp_opt->csum = (__force __sum16)get_unaligned_be16(ptr);
+ ptr += 2;
+ }
+
+ pr_debug("data_seq=%llu subflow_seq=%u data_len=%u csum=%d:%u",
mp_opt->data_seq, mp_opt->subflow_seq,
- mp_opt->data_len);
+ mp_opt->data_len, mp_opt->csum_reqd, mp_opt->csum);
}
break;
}
}
-void mptcp_get_options(const struct sk_buff *skb,
+void mptcp_get_options(const struct sock *sk,
+ const struct sk_buff *skb,
struct mptcp_options_received *mp_opt)
{
+ struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(sk);
+ struct mptcp_sock *msk = mptcp_sk(subflow->conn);
const struct tcphdr *th = tcp_hdr(skb);
const unsigned char *ptr;
int length;
mp_opt->dss = 0;
mp_opt->mp_prio = 0;
mp_opt->reset = 0;
+ mp_opt->csum_reqd = READ_ONCE(msk->csum_enabled);
+ mp_opt->deny_join_id0 = 0;
length = (th->doff * 4) - sizeof(struct tcphdr);
ptr = (const unsigned char *)(th + 1);
length--;
continue;
default:
+ if (length < 2)
+ return;
opsize = *ptr++;
if (opsize < 2) /* "silly options" */
return;
subflow->snd_isn = TCP_SKB_CB(skb)->end_seq;
if (subflow->request_mptcp) {
opts->suboptions = OPTION_MPTCP_MPC_SYN;
+ opts->csum_reqd = mptcp_is_checksum_enabled(sock_net(sk));
+ opts->allow_join_id0 = mptcp_allow_join_id0(sock_net(sk));
*size = TCPOLEN_MPTCP_MPC_SYN;
return true;
} else if (subflow->request_join) {
struct mptcp_out_options *opts)
{
struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(sk);
+ struct mptcp_sock *msk = mptcp_sk(subflow->conn);
struct mptcp_ext *mpext;
unsigned int data_len;
+ u8 len;
/* When skb is not available, we better over-estimate the emitted
* options len. A full DSS option (28 bytes) is longer than
opts->suboptions = OPTION_MPTCP_MPC_ACK;
opts->sndr_key = subflow->local_key;
opts->rcvr_key = subflow->remote_key;
+ opts->csum_reqd = READ_ONCE(msk->csum_enabled);
+ opts->allow_join_id0 = mptcp_allow_join_id0(sock_net(sk));
/* Section 3.1.
* The MP_CAPABLE option is carried on the SYN, SYN/ACK, and ACK
* packets that start the first subflow of an MPTCP connection,
* as well as the first packet that carries data
*/
- if (data_len > 0)
- *size = ALIGN(TCPOLEN_MPTCP_MPC_ACK_DATA, 4);
- else
+ if (data_len > 0) {
+ len = TCPOLEN_MPTCP_MPC_ACK_DATA;
+ if (opts->csum_reqd) {
+ /* we need to propagate more info to csum the pseudo hdr */
+ opts->ext_copy.data_seq = mpext->data_seq;
+ opts->ext_copy.subflow_seq = mpext->subflow_seq;
+ opts->ext_copy.csum = mpext->csum;
+ len += TCPOLEN_MPTCP_DSS_CHECKSUM;
+ }
+ *size = ALIGN(len, 4);
+ } else {
*size = TCPOLEN_MPTCP_MPC_ACK;
+ }
pr_debug("subflow=%p, local_key=%llu, remote_key=%llu map_len=%d",
subflow, subflow->local_key, subflow->remote_key,
bool ret = false;
u64 ack_seq;
+ opts->csum_reqd = READ_ONCE(msk->csum_enabled);
mpext = skb ? mptcp_get_ext(skb) : NULL;
if (!skb || (mpext && mpext->use_map) || snd_data_fin_enable) {
- unsigned int map_size;
+ unsigned int map_size = TCPOLEN_MPTCP_DSS_BASE + TCPOLEN_MPTCP_DSS_MAP64;
- map_size = TCPOLEN_MPTCP_DSS_BASE + TCPOLEN_MPTCP_DSS_MAP64;
+ if (mpext) {
+ if (opts->csum_reqd)
+ map_size += TCPOLEN_MPTCP_DSS_CHECKSUM;
- remaining -= map_size;
- dss_size = map_size;
- if (mpext)
opts->ext_copy = *mpext;
+ }
+ remaining -= map_size;
+ dss_size = map_size;
if (skb && snd_data_fin_enable)
mptcp_write_data_fin(subflow, skb, &opts->ext_copy);
ret = true;
if (subflow_req->mp_capable) {
opts->suboptions = OPTION_MPTCP_MPC_SYNACK;
opts->sndr_key = subflow_req->local_key;
+ opts->csum_reqd = subflow_req->csum_reqd;
+ opts->allow_join_id0 = subflow_req->allow_join_id0;
*size = TCPOLEN_MPTCP_MPC_SYNACK;
pr_debug("subflow_req=%p, local_key=%llu",
subflow_req, subflow_req->local_key);
return false;
}
+ if (mp_opt->deny_join_id0)
+ WRITE_ONCE(msk->pm.remote_deny_join_id0, true);
+
if (unlikely(!READ_ONCE(msk->pm.server_side)))
pr_warn_once("bogus mpc option on established client sk");
mptcp_subflow_fully_established(subflow, mp_opt);
return;
}
- mptcp_get_options(skb, &mp_opt);
+ mptcp_get_options(sk, skb, &mp_opt);
if (!check_fully_established(msk, sk, subflow, skb, &mp_opt))
return;
MPTCP_INC_STATS(sock_net(sk), MPTCP_MIB_ADDADDR);
} else {
mptcp_pm_add_addr_echoed(msk, &mp_opt.addr);
- mptcp_pm_del_add_timer(msk, &mp_opt.addr);
+ mptcp_pm_del_add_timer(msk, &mp_opt.addr, true);
MPTCP_INC_STATS(sock_net(sk), MPTCP_MIB_ECHOADD);
}
}
mpext->data_len = mp_opt.data_len;
mpext->use_map = 1;
+ mpext->csum_reqd = mp_opt.csum_reqd;
+
+ if (mpext->csum_reqd)
+ mpext->csum = mp_opt.csum;
}
}
WRITE_ONCE(msk->rcv_wnd_sent, ack_seq);
}
+static u16 mptcp_make_csum(const struct mptcp_ext *mpext)
+{
+ struct csum_pseudo_header header;
+ __wsum csum;
+
+ /* cfr RFC 8684 3.3.1.:
+ * the data sequence number used in the pseudo-header is
+ * always the 64-bit value, irrespective of what length is used in the
+ * DSS option itself.
+ */
+ header.data_seq = cpu_to_be64(mpext->data_seq);
+ header.subflow_seq = htonl(mpext->subflow_seq);
+ header.data_len = htons(mpext->data_len);
+ header.csum = 0;
+
+ csum = csum_partial(&header, sizeof(header), ~csum_unfold(mpext->csum));
+ return (__force u16)csum_fold(csum);
+}
+
void mptcp_write_options(__be32 *ptr, const struct tcp_sock *tp,
struct mptcp_out_options *opts)
{
if ((OPTION_MPTCP_MPC_SYN | OPTION_MPTCP_MPC_SYNACK |
OPTION_MPTCP_MPC_ACK) & opts->suboptions) {
- u8 len;
+ u8 len, flag = MPTCP_CAP_HMAC_SHA256;
- if (OPTION_MPTCP_MPC_SYN & opts->suboptions)
+ if (OPTION_MPTCP_MPC_SYN & opts->suboptions) {
len = TCPOLEN_MPTCP_MPC_SYN;
- else if (OPTION_MPTCP_MPC_SYNACK & opts->suboptions)
+ } else if (OPTION_MPTCP_MPC_SYNACK & opts->suboptions) {
len = TCPOLEN_MPTCP_MPC_SYNACK;
- else if (opts->ext_copy.data_len)
+ } else if (opts->ext_copy.data_len) {
len = TCPOLEN_MPTCP_MPC_ACK_DATA;
- else
+ if (opts->csum_reqd)
+ len += TCPOLEN_MPTCP_DSS_CHECKSUM;
+ } else {
len = TCPOLEN_MPTCP_MPC_ACK;
+ }
+
+ if (opts->csum_reqd)
+ flag |= MPTCP_CAP_CHECKSUM_REQD;
+
+ if (!opts->allow_join_id0)
+ flag |= MPTCP_CAP_DENY_JOIN_ID0;
*ptr++ = mptcp_option(MPTCPOPT_MP_CAPABLE, len,
MPTCP_SUPPORTED_VERSION,
- MPTCP_CAP_HMAC_SHA256);
+ flag);
if (!((OPTION_MPTCP_MPC_SYNACK | OPTION_MPTCP_MPC_ACK) &
opts->suboptions))
if (!opts->ext_copy.data_len)
goto mp_capable_done;
- put_unaligned_be32(opts->ext_copy.data_len << 16 |
- TCPOPT_NOP << 8 | TCPOPT_NOP, ptr);
+ if (opts->csum_reqd) {
+ put_unaligned_be32(opts->ext_copy.data_len << 16 |
+ mptcp_make_csum(&opts->ext_copy), ptr);
+ } else {
+ put_unaligned_be32(opts->ext_copy.data_len << 16 |
+ TCPOPT_NOP << 8 | TCPOPT_NOP, ptr);
+ }
ptr += 1;
}
flags |= MPTCP_DSS_HAS_MAP | MPTCP_DSS_DSN64;
if (mpext->data_fin)
flags |= MPTCP_DSS_DATA_FIN;
+
+ if (opts->csum_reqd)
+ len += TCPOLEN_MPTCP_DSS_CHECKSUM;
}
*ptr++ = mptcp_option(MPTCPOPT_DSS, len, 0, flags);
ptr += 2;
put_unaligned_be32(mpext->subflow_seq, ptr);
ptr += 1;
- put_unaligned_be32(mpext->data_len << 16 |
- TCPOPT_NOP << 8 | TCPOPT_NOP, ptr);
+ if (opts->csum_reqd) {
+ put_unaligned_be32(mpext->data_len << 16 |
+ mptcp_make_csum(mpext), ptr);
+ } else {
+ put_unaligned_be32(mpext->data_len << 16 |
+ TCPOPT_NOP << 8 | TCPOPT_NOP, ptr);
+ }
}
}
WRITE_ONCE(msk->pm.addr_signal, 0);
WRITE_ONCE(msk->pm.accept_addr, false);
WRITE_ONCE(msk->pm.accept_subflow, false);
+ WRITE_ONCE(msk->pm.remote_deny_join_id0, false);
msk->pm.status = 0;
spin_lock_init(&msk->pm.lock);
struct mptcp_pm_add_entry *
mptcp_pm_del_add_timer(struct mptcp_sock *msk,
- struct mptcp_addr_info *addr)
+ struct mptcp_addr_info *addr, bool check_id)
{
struct mptcp_pm_add_entry *entry;
struct sock *sk = (struct sock *)msk;
spin_lock_bh(&msk->pm.lock);
entry = mptcp_lookup_anno_list_by_saddr(msk, addr);
- if (entry)
+ if (entry && (!check_id || entry->addr.id == addr->id))
entry->retrans_times = ADD_ADDR_RETRANS_MAX;
spin_unlock_bh(&msk->pm.lock);
- if (entry)
+ if (entry && (!check_id || entry->addr.id == addr->id))
sk_stop_timer_sync(sk, &entry->add_timer);
return entry;
/* check if should create a new subflow */
if (msk->pm.local_addr_used < local_addr_max &&
- msk->pm.subflows < subflows_max) {
+ msk->pm.subflows < subflows_max &&
+ !READ_ONCE(msk->pm.remote_deny_join_id0)) {
local = select_local_address(pernet, msk);
if (local) {
struct mptcp_addr_info remote = { 0 };
subflow = list_first_entry_or_null(&msk->conn_list, typeof(*subflow), node);
if (subflow) {
struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
+ bool slow;
spin_unlock_bh(&msk->pm.lock);
pr_debug("send ack for %s%s%s",
mptcp_pm_should_add_signal_ipv6(msk) ? " [ipv6]" : "",
mptcp_pm_should_add_signal_port(msk) ? " [port]" : "");
- lock_sock(ssk);
+ slow = lock_sock_fast(ssk);
tcp_send_ack(ssk);
- release_sock(ssk);
+ unlock_sock_fast(ssk, slow);
spin_lock_bh(&msk->pm.lock);
}
}
struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
struct sock *sk = (struct sock *)msk;
struct mptcp_addr_info local;
+ bool slow;
local_address((struct sock_common *)ssk, &local);
if (!addresses_equal(&local, addr, addr->port))
spin_unlock_bh(&msk->pm.lock);
pr_debug("send ack for mp_prio");
- lock_sock(ssk);
+ slow = lock_sock_fast(ssk);
tcp_send_ack(ssk);
- release_sock(ssk);
+ unlock_sock_fast(ssk, slow);
spin_lock_bh(&msk->pm.lock);
return 0;
if (tb[MPTCP_PM_ADDR_ATTR_FLAGS])
entry->flags = nla_get_u32(tb[MPTCP_PM_ADDR_ATTR_FLAGS]);
- if (tb[MPTCP_PM_ADDR_ATTR_PORT])
+ if (tb[MPTCP_PM_ADDR_ATTR_PORT]) {
+ if (!(entry->flags & MPTCP_PM_ADDR_FLAG_SIGNAL)) {
+ NL_SET_ERR_MSG_ATTR(info->extack, attr,
+ "flags must have signal when using port");
+ return -EINVAL;
+ }
entry->addr.port = htons(nla_get_u16(tb[MPTCP_PM_ADDR_ATTR_PORT]));
+ }
return 0;
}
{
struct mptcp_pm_add_entry *entry;
- entry = mptcp_pm_del_add_timer(msk, addr);
+ entry = mptcp_pm_del_add_timer(msk, addr, false);
if (entry) {
list_del(&entry->list);
kfree(entry);
struct pm_nl_pernet *pernet = net_generic(net, pm_nl_pernet_id);
INIT_LIST_HEAD_RCU(&pernet->local_addr_list);
- __reset_counters(pernet);
pernet->next_id = 1;
- bitmap_zero(pernet->id_bitmap, MAX_ADDR_ID + 1);
spin_lock_init(&pernet->lock);
+
+ /* No need to initialize other pernet fields, the struct is zeroed at
+ * allocation time.
+ */
+
return 0;
}
u64 map_seq;
u64 end_seq;
u32 offset;
+ u8 has_rxtstamp:1;
};
#define MPTCP_SKB_CB(__skb) ((struct mptcp_skb_cb *)&((__skb)->cb[0]))
+enum {
+ MPTCP_CMSG_TS = BIT(0),
+};
+
static struct percpu_counter mptcp_sockets_allocated;
static void __mptcp_destroy_sock(struct sock *sk);
struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(ssk);
struct sock *sk = (struct sock *)msk;
struct sk_buff *tail;
+ bool has_rxtstamp;
__skb_unlink(skb, &ssk->sk_receive_queue);
/* try to fetch required memory from subflow */
if (!sk_rmem_schedule(sk, skb, skb->truesize)) {
- if (ssk->sk_forward_alloc < skb->truesize)
- goto drop;
- __sk_mem_reclaim(ssk, skb->truesize);
- if (!sk_rmem_schedule(sk, skb, skb->truesize))
+ int amount = sk_mem_pages(skb->truesize) << SK_MEM_QUANTUM_SHIFT;
+
+ if (ssk->sk_forward_alloc < amount)
goto drop;
+
+ ssk->sk_forward_alloc -= amount;
+ sk->sk_forward_alloc += amount;
}
+ has_rxtstamp = TCP_SKB_CB(skb)->has_rxtstamp;
+
/* the skb map_seq accounts for the skb offset:
* mptcp_subflow_get_mapped_dsn() is based on the current tp->copied_seq
* value
MPTCP_SKB_CB(skb)->map_seq = mptcp_subflow_get_mapped_dsn(subflow);
MPTCP_SKB_CB(skb)->end_seq = MPTCP_SKB_CB(skb)->map_seq + copy_len;
MPTCP_SKB_CB(skb)->offset = offset;
+ MPTCP_SKB_CB(skb)->has_rxtstamp = has_rxtstamp;
if (MPTCP_SKB_CB(skb)->map_seq == msk->ack_seq) {
/* in sequence */
mptcp_for_each_subflow(msk, subflow) {
struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
+ bool slow;
- lock_sock(ssk);
+ slow = lock_sock_fast(ssk);
if (tcp_can_send_ack(ssk))
tcp_send_ack(ssk);
- release_sock(ssk);
+ unlock_sock_fast(ssk, slow);
}
}
-static bool mptcp_subflow_cleanup_rbuf(struct sock *ssk)
+static void mptcp_subflow_cleanup_rbuf(struct sock *ssk)
{
- int ret;
+ bool slow;
- lock_sock(ssk);
- ret = tcp_can_send_ack(ssk);
- if (ret)
+ slow = lock_sock_fast(ssk);
+ if (tcp_can_send_ack(ssk))
tcp_cleanup_rbuf(ssk, 1);
- release_sock(ssk);
- return ret;
+ unlock_sock_fast(ssk, slow);
+}
+
+static bool mptcp_subflow_could_cleanup(const struct sock *ssk, bool rx_empty)
+{
+ const struct inet_connection_sock *icsk = inet_csk(ssk);
+ u8 ack_pending = READ_ONCE(icsk->icsk_ack.pending);
+ const struct tcp_sock *tp = tcp_sk(ssk);
+
+ return (ack_pending & ICSK_ACK_SCHED) &&
+ ((READ_ONCE(tp->rcv_nxt) - READ_ONCE(tp->rcv_wup) >
+ READ_ONCE(icsk->icsk_ack.rcv_mss)) ||
+ (rx_empty && ack_pending &
+ (ICSK_ACK_PUSHED2 | ICSK_ACK_PUSHED)));
}
static void mptcp_cleanup_rbuf(struct mptcp_sock *msk)
{
- struct sock *ack_hint = READ_ONCE(msk->ack_hint);
int old_space = READ_ONCE(msk->old_wspace);
struct mptcp_subflow_context *subflow;
struct sock *sk = (struct sock *)msk;
- bool cleanup;
+ int space = __mptcp_space(sk);
+ bool cleanup, rx_empty;
- /* this is a simple superset of what tcp_cleanup_rbuf() implements
- * so that we don't have to acquire the ssk socket lock most of the time
- * to do actually nothing
- */
- cleanup = __mptcp_space(sk) - old_space >= max(0, old_space);
- if (!cleanup)
- return;
+ cleanup = (space > 0) && (space >= (old_space << 1));
+ rx_empty = !atomic_read(&sk->sk_rmem_alloc);
- /* if the hinted ssk is still active, try to use it */
- if (likely(ack_hint)) {
- mptcp_for_each_subflow(msk, subflow) {
- struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
+ mptcp_for_each_subflow(msk, subflow) {
+ struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
- if (ack_hint == ssk && mptcp_subflow_cleanup_rbuf(ssk))
- return;
- }
+ if (cleanup || mptcp_subflow_could_cleanup(ssk, rx_empty))
+ mptcp_subflow_cleanup_rbuf(ssk);
}
-
- /* otherwise pick the first active subflow */
- mptcp_for_each_subflow(msk, subflow)
- if (mptcp_subflow_cleanup_rbuf(mptcp_subflow_tcp_sock(subflow)))
- return;
}
static bool mptcp_check_data_fin(struct sock *sk)
break;
}
} while (more_data_avail);
- WRITE_ONCE(msk->ack_hint, ssk);
*bytes += moved;
return done;
/* In most cases we will be able to lock the mptcp socket. If its already
* owned, we need to defer to the work queue to avoid ABBA deadlock.
*/
-static void move_skbs_to_msk(struct mptcp_sock *msk, struct sock *ssk)
+static bool move_skbs_to_msk(struct mptcp_sock *msk, struct sock *ssk)
{
struct sock *sk = (struct sock *)msk;
unsigned int moved = 0;
- if (inet_sk_state_load(sk) == TCP_CLOSE)
- return;
-
- mptcp_data_lock(sk);
-
__mptcp_move_skbs_from_subflow(msk, ssk, &moved);
__mptcp_ofo_queue(msk);
+ if (unlikely(ssk->sk_err)) {
+ if (!sock_owned_by_user(sk))
+ __mptcp_error_report(sk);
+ else
+ set_bit(MPTCP_ERROR_REPORT, &msk->flags);
+ }
/* If the moves have caught up with the DATA_FIN sequence number
* it's time to ack the DATA_FIN and change socket state, but
*/
if (mptcp_pending_data_fin(sk, NULL))
mptcp_schedule_work(sk);
- mptcp_data_unlock(sk);
+ return moved > 0;
}
void mptcp_data_ready(struct sock *sk, struct sock *ssk)
struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(ssk);
struct mptcp_sock *msk = mptcp_sk(sk);
int sk_rbuf, ssk_rbuf;
- bool wake;
/* The peer can send data while we are shutting down this
* subflow at msk destruction time, but we must avoid enqueuing
if (unlikely(subflow->disposable))
return;
- /* move_skbs_to_msk below can legitly clear the data_avail flag,
- * but we will need later to properly woke the reader, cache its
- * value
- */
- wake = subflow->data_avail == MPTCP_SUBFLOW_DATA_AVAIL;
- if (wake)
- set_bit(MPTCP_DATA_READY, &msk->flags);
-
ssk_rbuf = READ_ONCE(ssk->sk_rcvbuf);
sk_rbuf = READ_ONCE(sk->sk_rcvbuf);
if (unlikely(ssk_rbuf > sk_rbuf))
sk_rbuf = ssk_rbuf;
- /* over limit? can't append more skbs to msk */
+ /* over limit? can't append more skbs to msk, Also, no need to wake-up*/
if (atomic_read(&sk->sk_rmem_alloc) > sk_rbuf)
- goto wake;
-
- move_skbs_to_msk(msk, ssk);
+ return;
-wake:
- if (wake)
+ /* Wake-up the reader only for in-sequence data */
+ mptcp_data_lock(sk);
+ if (move_skbs_to_msk(msk, ssk)) {
+ set_bit(MPTCP_DATA_READY, &msk->flags);
sk->sk_data_ready(sk);
+ }
+ mptcp_data_unlock(sk);
}
static bool mptcp_do_flush_join_list(struct mptcp_sock *msk)
sock_owned_by_me(sk);
mptcp_for_each_subflow(msk, subflow) {
- if (subflow->data_avail)
+ if (READ_ONCE(subflow->data_avail))
return mptcp_subflow_tcp_sock(subflow);
}
!mpext->frozen;
}
+/* we can append data to the given data frag if:
+ * - there is space available in the backing page_frag
+ * - the data frag tail matches the current page_frag free offset
+ * - the data frag end sequence number matches the current write seq
+ */
static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk,
const struct page_frag *pfrag,
const struct mptcp_data_frag *df)
{
return df && pfrag->page == df->page &&
pfrag->size - pfrag->offset > 0 &&
+ pfrag->offset == (df->offset + df->data_len) &&
df->data_seq + df->data_len == msk->write_seq;
}
-static int mptcp_wmem_with_overhead(struct sock *sk, int size)
+static int mptcp_wmem_with_overhead(int size)
{
- struct mptcp_sock *msk = mptcp_sk(sk);
- int ret, skbs;
-
- ret = size + ((sizeof(struct mptcp_data_frag) * size) >> PAGE_SHIFT);
- skbs = (msk->tx_pending_data + size) / msk->size_goal_cache;
- if (skbs < msk->skb_tx_cache.qlen)
- return ret;
-
- return ret + (skbs - msk->skb_tx_cache.qlen) * SKB_TRUESIZE(MAX_TCP_HEADER);
+ return size + ((sizeof(struct mptcp_data_frag) * size) >> PAGE_SHIFT);
}
static void __mptcp_wmem_reserve(struct sock *sk, int size)
{
- int amount = mptcp_wmem_with_overhead(sk, size);
+ int amount = mptcp_wmem_with_overhead(size);
struct mptcp_sock *msk = mptcp_sk(sk);
WARN_ON_ONCE(msk->wmem_reserved);
{
struct mptcp_sock *msk = mptcp_sk(sk);
+#ifdef CONFIG_LOCKDEP
+ WARN_ON_ONCE(!lockdep_is_held(&sk->sk_lock.slock));
+#endif
+
if (!msk->wmem_reserved)
return;
static void __mptcp_clean_una_wakeup(struct sock *sk)
{
+#ifdef CONFIG_LOCKDEP
+ WARN_ON_ONCE(!lockdep_is_held(&sk->sk_lock.slock));
+#endif
__mptcp_clean_una(sk);
mptcp_write_space(sk);
}
+static void mptcp_clean_una_wakeup(struct sock *sk)
+{
+ mptcp_data_lock(sk);
+ __mptcp_clean_una_wakeup(sk);
+ mptcp_data_unlock(sk);
+}
+
static void mptcp_enter_memory_pressure(struct sock *sk)
{
struct mptcp_subflow_context *subflow;
return NULL;
}
-static bool mptcp_tx_cache_refill(struct sock *sk, int size,
- struct sk_buff_head *skbs, int *total_ts)
-{
- struct mptcp_sock *msk = mptcp_sk(sk);
- struct sk_buff *skb;
- int space_needed;
-
- if (unlikely(tcp_under_memory_pressure(sk))) {
- mptcp_mem_reclaim_partial(sk);
-
- /* under pressure pre-allocate at most a single skb */
- if (msk->skb_tx_cache.qlen)
- return true;
- space_needed = msk->size_goal_cache;
- } else {
- space_needed = msk->tx_pending_data + size -
- msk->skb_tx_cache.qlen * msk->size_goal_cache;
- }
-
- while (space_needed > 0) {
- skb = __mptcp_do_alloc_tx_skb(sk, sk->sk_allocation);
- if (unlikely(!skb)) {
- /* under memory pressure, try to pass the caller a
- * single skb to allow forward progress
- */
- while (skbs->qlen > 1) {
- skb = __skb_dequeue_tail(skbs);
- *total_ts -= skb->truesize;
- __kfree_skb(skb);
- }
- return skbs->qlen > 0;
- }
-
- *total_ts += skb->truesize;
- __skb_queue_tail(skbs, skb);
- space_needed -= msk->size_goal_cache;
- }
- return true;
-}
-
static bool __mptcp_alloc_tx_skb(struct sock *sk, struct sock *ssk, gfp_t gfp)
{
- struct mptcp_sock *msk = mptcp_sk(sk);
struct sk_buff *skb;
if (ssk->sk_tx_skb_cache) {
return true;
}
- skb = skb_peek(&msk->skb_tx_cache);
- if (skb) {
- if (likely(sk_wmem_schedule(ssk, skb->truesize))) {
- skb = __skb_dequeue(&msk->skb_tx_cache);
- if (WARN_ON_ONCE(!skb))
- return false;
-
- mptcp_wmem_uncharge(sk, skb->truesize);
- ssk->sk_tx_skb_cache = skb;
- return true;
- }
-
- /* over memory limit, no point to try to allocate a new skb */
- return false;
- }
-
skb = __mptcp_do_alloc_tx_skb(sk, gfp);
if (!skb)
return false;
static bool mptcp_must_reclaim_memory(struct sock *sk, struct sock *ssk)
{
return !ssk->sk_tx_skb_cache &&
- !skb_peek(&mptcp_sk(sk)->skb_tx_cache) &&
tcp_under_memory_pressure(sk);
}
return __mptcp_alloc_tx_skb(sk, ssk, sk->sk_allocation);
}
+/* note: this always recompute the csum on the whole skb, even
+ * if we just appended a single frag. More status info needed
+ */
+static void mptcp_update_data_checksum(struct sk_buff *skb, int added)
+{
+ struct mptcp_ext *mpext = mptcp_get_ext(skb);
+ __wsum csum = ~csum_unfold(mpext->csum);
+ int offset = skb->len - added;
+
+ mpext->csum = csum_fold(csum_block_add(csum, skb_checksum(skb, offset, added, 0), offset));
+}
+
static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
struct mptcp_data_frag *dfrag,
struct mptcp_sendmsg_info *info)
/* compute send limit */
info->mss_now = tcp_send_mss(ssk, &info->size_goal, info->flags);
avail_size = info->size_goal;
- msk->size_goal_cache = info->size_goal;
skb = tcp_write_queue_tail(ssk);
if (skb) {
/* Limit the write to the size available in the
if (zero_window_probe) {
mptcp_subflow_ctx(ssk)->rel_write_seq += ret;
mpext->frozen = 1;
- ret = 0;
+ if (READ_ONCE(msk->csum_enabled))
+ mptcp_update_data_checksum(tail, ret);
tcp_push_pending_frames(ssk);
+ return 0;
}
out:
+ if (READ_ONCE(msk->csum_enabled))
+ mptcp_update_data_checksum(tail, ret);
mptcp_subflow_ctx(ssk)->rel_write_seq += ret;
return ret;
}
while (msg_data_left(msg)) {
int total_ts, frag_truesize = 0;
struct mptcp_data_frag *dfrag;
- struct sk_buff_head skbs;
bool dfrag_collapsed;
size_t psize, offset;
psize = pfrag->size - offset;
psize = min_t(size_t, psize, msg_data_left(msg));
total_ts = psize + frag_truesize;
- __skb_queue_head_init(&skbs);
- if (!mptcp_tx_cache_refill(sk, psize, &skbs, &total_ts))
- goto wait_for_memory;
- if (!mptcp_wmem_alloc(sk, total_ts)) {
- __skb_queue_purge(&skbs);
+ if (!mptcp_wmem_alloc(sk, total_ts))
goto wait_for_memory;
- }
- skb_queue_splice_tail(&skbs, &msk->skb_tx_cache);
if (copy_page_from_iter(dfrag->page, offset, psize,
&msg->msg_iter) != psize) {
mptcp_wmem_uncharge(sk, psize + frag_truesize);
sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
sk_wait_event(sk, timeo,
- test_and_clear_bit(MPTCP_DATA_READY, &msk->flags), &wait);
+ test_bit(MPTCP_DATA_READY, &msk->flags), &wait);
sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
remove_wait_queue(sk_sleep(sk), &wait);
static int __mptcp_recvmsg_mskq(struct mptcp_sock *msk,
struct msghdr *msg,
- size_t len, int flags)
+ size_t len, int flags,
+ struct scm_timestamping_internal *tss,
+ int *cmsg_flags)
{
struct sk_buff *skb, *tmp;
int copied = 0;
}
}
+ if (MPTCP_SKB_CB(skb)->has_rxtstamp) {
+ tcp_update_recv_tstamps(skb, tss);
+ *cmsg_flags |= MPTCP_CMSG_TS;
+ }
+
copied += count;
if (count < data_len) {
__mptcp_update_rmem(sk);
done = __mptcp_move_skbs_from_subflow(msk, ssk, &moved);
mptcp_data_unlock(sk);
- tcp_cleanup_rbuf(ssk, moved);
+
+ if (unlikely(ssk->sk_err))
+ __mptcp_error_report(sk);
unlock_sock_fast(ssk, slowpath);
} while (!done);
ret |= __mptcp_ofo_queue(msk);
__mptcp_splice_receive_queue(sk);
mptcp_data_unlock(sk);
- mptcp_cleanup_rbuf(msk);
}
if (ret)
mptcp_check_data_fin((struct sock *)msk);
int nonblock, int flags, int *addr_len)
{
struct mptcp_sock *msk = mptcp_sk(sk);
- int copied = 0;
+ struct scm_timestamping_internal tss;
+ int copied = 0, cmsg_flags = 0;
int target;
long timeo;
while (copied < len) {
int bytes_read;
- bytes_read = __mptcp_recvmsg_mskq(msk, msg, len - copied, flags);
+ bytes_read = __mptcp_recvmsg_mskq(msk, msg, len - copied, flags, &tss, &cmsg_flags);
if (unlikely(bytes_read < 0)) {
if (!copied)
copied = bytes_read;
*/
if (unlikely(__mptcp_move_skbs(msk)))
set_bit(MPTCP_DATA_READY, &msk->flags);
- } else if (unlikely(!test_bit(MPTCP_DATA_READY, &msk->flags))) {
- /* data to read but mptcp_wait_data() cleared DATA_READY */
- set_bit(MPTCP_DATA_READY, &msk->flags);
}
+
out_err:
+ if (cmsg_flags && copied >= 0) {
+ if (cmsg_flags & MPTCP_CMSG_TS)
+ tcp_recv_timestamp(msg, sk, &tss);
+ }
+
pr_debug("msk=%p data_ready=%d rx queue empty=%d copied=%d",
msk, test_bit(MPTCP_DATA_READY, &msk->flags),
skb_queue_empty_lockless(&sk->sk_receive_queue), copied);
if (ssk == msk->last_snd)
msk->last_snd = NULL;
- if (ssk == msk->ack_hint)
- msk->ack_hint = NULL;
-
if (ssk == msk->first)
msk->first = NULL;
list_for_each_entry_safe(subflow, tmp, &msk->conn_list, node) {
struct sock *tcp_sk = mptcp_subflow_tcp_sock(subflow);
+ bool slow;
- lock_sock(tcp_sk);
+ slow = lock_sock_fast(tcp_sk);
if (tcp_sk->sk_state != TCP_CLOSE) {
tcp_send_active_reset(tcp_sk, GFP_ATOMIC);
tcp_set_state(tcp_sk, TCP_CLOSE);
}
- release_sock(tcp_sk);
+ unlock_sock_fast(tcp_sk, slow);
}
inet_sk_state_store(sk, TCP_CLOSE);
struct sock *ssk;
int ret;
- __mptcp_clean_una_wakeup(sk);
+ mptcp_clean_una_wakeup(sk);
dfrag = mptcp_rtx_head(sk);
if (!dfrag) {
if (mptcp_data_fin_enabled(msk)) {
/* limit retransmission to the bytes already sent on some subflows */
info.sent = 0;
- info.limit = dfrag->already_sent;
- while (info.sent < dfrag->already_sent) {
+ info.limit = READ_ONCE(msk->csum_enabled) ? dfrag->data_len : dfrag->already_sent;
+ while (info.sent < info.limit) {
if (!mptcp_alloc_tx_skb(sk, ssk))
break;
copied += ret;
info.sent += ret;
}
- if (copied)
+ if (copied) {
+ dfrag->already_sent = max(dfrag->already_sent, info.sent);
tcp_push(ssk, 0, info.mss_now, tcp_sk(ssk)->nonagle,
info.size_goal);
+ }
mptcp_set_timeout(sk, ssk);
release_sock(ssk);
INIT_LIST_HEAD(&msk->rtx_queue);
INIT_WORK(&msk->work, mptcp_worker);
__skb_queue_head_init(&msk->receive_queue);
- __skb_queue_head_init(&msk->skb_tx_cache);
msk->out_of_order_queue = RB_ROOT;
msk->first_pending = NULL;
msk->wmem_reserved = 0;
msk->rmem_released = 0;
msk->tx_pending_data = 0;
- msk->size_goal_cache = TCP_BASE_MSS;
- msk->ack_hint = NULL;
msk->first = NULL;
inet_csk(sk)->icsk_sync_mss = mptcp_sync_mss;
+ WRITE_ONCE(msk->csum_enabled, mptcp_is_checksum_enabled(sock_net(sk)));
mptcp_pm_data_init(msk);
timer_setup(&msk->sk.icsk_retransmit_timer, mptcp_retransmit_timer, 0);
timer_setup(&sk->sk_timer, mptcp_timeout_timer, 0);
- tcp_assign_congestion_control(sk);
-
return 0;
}
static int mptcp_init_sock(struct sock *sk)
{
+ struct inet_connection_sock *icsk = inet_csk(sk);
struct net *net = sock_net(sk);
int ret;
if (ret)
return ret;
+ /* fetch the ca name; do it outside __mptcp_init_sock(), so that clone will
+ * propagate the correct value
+ */
+ tcp_assign_congestion_control(sk);
+ strcpy(mptcp_sk(sk)->ca_name, icsk->icsk_ca_ops->name);
+
+ /* no need to keep a reference to the ops, the name will suffice */
+ tcp_cleanup_congestion_control(sk);
+ icsk->icsk_ca_ops = NULL;
+
sk_sockets_allocated_inc(sk);
sk->sk_rcvbuf = sock_net(sk)->ipv4.sysctl_tcp_rmem[1];
sk->sk_sndbuf = sock_net(sk)->ipv4.sysctl_tcp_wmem[1];
{
struct mptcp_sock *msk = mptcp_sk(sk);
struct mptcp_data_frag *dtmp, *dfrag;
- struct sk_buff *skb;
WRITE_ONCE(msk->first_pending, NULL);
list_for_each_entry_safe(dfrag, dtmp, &msk->rtx_queue, list)
dfrag_clear(sk, dfrag);
- while ((skb = __skb_dequeue(&msk->skb_tx_cache)) != NULL) {
- sk->sk_forward_alloc += skb->truesize;
- kfree_skb(skb);
- }
}
static void mptcp_cancel_work(struct sock *sk)
sk_stream_kill_queues(sk);
xfrm_sk_free_policy(sk);
- tcp_cleanup_congestion_control(sk);
sk_refcnt_debug_release(sk);
mptcp_dispose_initial_subflow(msk);
sock_put(sk);
msk->token = subflow_req->token;
msk->subflow = NULL;
WRITE_ONCE(msk->fully_established, false);
+ if (mp_opt->csum_reqd)
+ WRITE_ONCE(msk->csum_enabled, true);
msk->write_seq = subflow_req->idsn + 1;
msk->snd_nxt = msk->write_seq;
#define TCPOLEN_MPTCP_FASTCLOSE 12
#define TCPOLEN_MPTCP_RST 4
+#define TCPOLEN_MPTCP_MPC_ACK_DATA_CSUM (TCPOLEN_MPTCP_DSS_CHECKSUM + TCPOLEN_MPTCP_MPC_ACK_DATA)
+
/* MPTCP MP_JOIN flags */
#define MPTCPOPT_BACKUP BIT(0)
#define MPTCPOPT_HMAC_LEN 20
#define MPTCP_VERSION_MASK (0x0F)
#define MPTCP_CAP_CHECKSUM_REQD BIT(7)
#define MPTCP_CAP_EXTENSIBILITY BIT(6)
+#define MPTCP_CAP_DENY_JOIN_ID0 BIT(5)
#define MPTCP_CAP_HMAC_SHA256 BIT(0)
-#define MPTCP_CAP_FLAG_MASK (0x3F)
+#define MPTCP_CAP_FLAG_MASK (0x1F)
/* MPTCP DSS flags */
#define MPTCP_DSS_DATA_FIN BIT(4)
u64 data_seq;
u32 subflow_seq;
u16 data_len;
+ __sum16 csum;
u16 mp_capable : 1,
mp_join : 1,
fastclose : 1,
rm_addr : 1,
mp_prio : 1,
echo : 1,
- backup : 1;
+ csum_reqd : 1,
+ backup : 1,
+ deny_join_id0 : 1;
u32 token;
u32 nonce;
u64 thmac;
bool work_pending;
bool accept_addr;
bool accept_subflow;
+ bool remote_deny_join_id0;
u8 add_addr_signaled;
u8 add_addr_accepted;
u8 local_addr_used;
bool snd_data_fin_enable;
bool rcv_fastclose;
bool use_64bit_ack; /* Set when we received a 64-bit DSN */
+ bool csum_enabled;
spinlock_t join_list_lock;
- struct sock *ack_hint;
struct work_struct work;
struct sk_buff *ooo_last_skb;
struct rb_root out_of_order_queue;
struct sk_buff_head receive_queue;
- struct sk_buff_head skb_tx_cache; /* this is wmem accounted */
int tx_pending_data;
- int size_goal_cache;
struct list_head conn_list;
struct list_head rtx_queue;
struct mptcp_data_frag *first_pending;
} rcvq_space;
u32 setsockopt_seq;
+ char ca_name[TCP_CA_NAME_MAX];
};
#define mptcp_lock_sock(___sk, cb) do { \
return list_first_entry_or_null(&msk->rtx_queue, struct mptcp_data_frag, list);
}
+struct csum_pseudo_header {
+ __be64 data_seq;
+ __be32 subflow_seq;
+ __be16 data_len;
+ __sum16 csum;
+};
+
struct mptcp_subflow_request_sock {
struct tcp_request_sock sk;
u16 mp_capable : 1,
mp_join : 1,
- backup : 1;
+ backup : 1,
+ csum_reqd : 1,
+ allow_join_id0 : 1;
u8 local_id;
u8 remote_id;
u64 local_key;
enum mptcp_data_avail {
MPTCP_SUBFLOW_NODATA,
MPTCP_SUBFLOW_DATA_AVAIL,
- MPTCP_SUBFLOW_OOO_DATA
};
struct mptcp_delegated_action {
u32 map_subflow_seq;
u32 ssn_offset;
u32 map_data_len;
+ __wsum map_data_csum;
+ u32 map_csum_len;
u32 request_mptcp : 1, /* send MP_CAPABLE */
request_join : 1, /* send MP_JOIN */
request_bkup : 1,
pm_notified : 1, /* PM hook called for established status */
conn_finished : 1,
map_valid : 1,
+ map_csum_reqd : 1,
+ map_data_fin : 1,
mpc_map : 1,
backup : 1,
send_mp_prio : 1,
int mptcp_is_enabled(struct net *net);
unsigned int mptcp_get_add_addr_timeout(struct net *net);
+int mptcp_is_checksum_enabled(struct net *net);
+int mptcp_allow_join_id0(struct net *net);
void mptcp_subflow_fully_established(struct mptcp_subflow_context *subflow,
struct mptcp_options_received *mp_opt);
bool mptcp_subflow_data_available(struct sock *sk);
struct sock *mptcp_sk_clone(const struct sock *sk,
const struct mptcp_options_received *mp_opt,
struct request_sock *req);
-void mptcp_get_options(const struct sk_buff *skb,
+void mptcp_get_options(const struct sock *sk,
+ const struct sk_buff *skb,
struct mptcp_options_received *mp_opt);
void mptcp_finish_connect(struct sock *sk);
void mptcp_destroy_common(struct mptcp_sock *msk);
+#define MPTCP_TOKEN_MAX_RETRIES 4
+
void __init mptcp_token_init(void);
static inline void mptcp_token_init_request(struct request_sock *req)
{
bool mptcp_pm_sport_in_anno_list(struct mptcp_sock *msk, const struct sock *sk);
struct mptcp_pm_add_entry *
mptcp_pm_del_add_timer(struct mptcp_sock *msk,
- struct mptcp_addr_info *addr);
+ struct mptcp_addr_info *addr, bool check_id);
struct mptcp_pm_add_entry *
mptcp_lookup_anno_list_by_saddr(struct mptcp_sock *msk,
struct mptcp_addr_info *addr);
mptcp_sol_socket_sync_intval(msk, SO_INCOMING_CPU, val);
}
+static int mptcp_setsockopt_sol_socket_tstamp(struct mptcp_sock *msk, int optname, int val)
+{
+ sockptr_t optval = KERNEL_SOCKPTR(&val);
+ struct mptcp_subflow_context *subflow;
+ struct sock *sk = (struct sock *)msk;
+ int ret;
+
+ ret = sock_setsockopt(sk->sk_socket, SOL_SOCKET, optname,
+ optval, sizeof(val));
+ if (ret)
+ return ret;
+
+ lock_sock(sk);
+ mptcp_for_each_subflow(msk, subflow) {
+ struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
+ bool slow = lock_sock_fast(ssk);
+
+ switch (optname) {
+ case SO_TIMESTAMP_OLD:
+ case SO_TIMESTAMP_NEW:
+ case SO_TIMESTAMPNS_OLD:
+ case SO_TIMESTAMPNS_NEW:
+ sock_set_timestamp(sk, optname, !!val);
+ break;
+ case SO_TIMESTAMPING_NEW:
+ case SO_TIMESTAMPING_OLD:
+ sock_set_timestamping(sk, optname, val);
+ break;
+ }
+
+ unlock_sock_fast(ssk, slow);
+ }
+
+ release_sock(sk);
+ return 0;
+}
+
static int mptcp_setsockopt_sol_socket_int(struct mptcp_sock *msk, int optname,
sockptr_t optval, unsigned int optlen)
{
case SO_INCOMING_CPU:
mptcp_so_incoming_cpu(msk, val);
return 0;
+ case SO_TIMESTAMP_OLD:
+ case SO_TIMESTAMP_NEW:
+ case SO_TIMESTAMPNS_OLD:
+ case SO_TIMESTAMPNS_NEW:
+ case SO_TIMESTAMPING_OLD:
+ case SO_TIMESTAMPING_NEW:
+ return mptcp_setsockopt_sol_socket_tstamp(msk, optname, val);
}
return -ENOPROTOOPT;
case SO_MARK:
case SO_INCOMING_CPU:
case SO_DEBUG:
+ case SO_TIMESTAMP_OLD:
+ case SO_TIMESTAMP_NEW:
+ case SO_TIMESTAMPNS_OLD:
+ case SO_TIMESTAMPNS_NEW:
+ case SO_TIMESTAMPING_OLD:
+ case SO_TIMESTAMPING_NEW:
return mptcp_setsockopt_sol_socket_int(msk, optname, optval, optlen);
case SO_LINGER:
return mptcp_setsockopt_sol_socket_linger(msk, optval, optlen);
+ case SO_RCVLOWAT:
+ case SO_RCVTIMEO_OLD:
+ case SO_RCVTIMEO_NEW:
+ case SO_BUSY_POLL:
+ case SO_PREFER_BUSY_POLL:
+ case SO_BUSY_POLL_BUDGET:
+ /* No need to copy: only relevant for msk */
+ return sock_setsockopt(sk->sk_socket, SOL_SOCKET, optname, optval, optlen);
case SO_NO_CHECK:
case SO_DONTROUTE:
case SO_BROADCAST:
return 0;
}
- return sock_setsockopt(sk->sk_socket, SOL_SOCKET, optname, optval, optlen);
+ /* SO_OOBINLINE is not supported, let's avoid the related mess
+ * SO_ATTACH_FILTER, SO_ATTACH_BPF, SO_ATTACH_REUSEPORT_CBPF,
+ * SO_DETACH_REUSEPORT_BPF, SO_DETACH_FILTER, SO_LOCK_FILTER,
+ * we must be careful with subflows
+ *
+ * SO_ATTACH_REUSEPORT_EBPF is not supported, at it checks
+ * explicitly the sk_protocol field
+ *
+ * SO_PEEK_OFF is unsupported, as it is for plain TCP
+ * SO_MAX_PACING_RATE is unsupported, we must be careful with subflows
+ * SO_CNX_ADVICE is currently unsupported, could possibly be relevant,
+ * but likely needs careful design
+ *
+ * SO_ZEROCOPY is currently unsupported, TODO in sndmsg
+ * SO_TXTIME is currently unsupported
+ */
+
+ return -EOPNOTSUPP;
}
static int mptcp_setsockopt_v6(struct mptcp_sock *msk, int optname,
static bool mptcp_supported_sockopt(int level, int optname)
{
- if (level == SOL_SOCKET) {
- switch (optname) {
- case SO_DEBUG:
- case SO_REUSEPORT:
- case SO_REUSEADDR:
-
- /* the following ones need a better implementation,
- * but are quite common we want to preserve them
- */
- case SO_BINDTODEVICE:
- case SO_SNDBUF:
- case SO_SNDBUFFORCE:
- case SO_RCVBUF:
- case SO_RCVBUFFORCE:
- case SO_KEEPALIVE:
- case SO_PRIORITY:
- case SO_LINGER:
- case SO_TIMESTAMP_OLD:
- case SO_TIMESTAMP_NEW:
- case SO_TIMESTAMPNS_OLD:
- case SO_TIMESTAMPNS_NEW:
- case SO_TIMESTAMPING_OLD:
- case SO_TIMESTAMPING_NEW:
- case SO_RCVLOWAT:
- case SO_RCVTIMEO_OLD:
- case SO_RCVTIMEO_NEW:
- case SO_SNDTIMEO_OLD:
- case SO_SNDTIMEO_NEW:
- case SO_MARK:
- case SO_INCOMING_CPU:
- case SO_BINDTOIFINDEX:
- case SO_BUSY_POLL:
- case SO_PREFER_BUSY_POLL:
- case SO_BUSY_POLL_BUDGET:
-
- /* next ones are no-op for plain TCP */
- case SO_NO_CHECK:
- case SO_DONTROUTE:
- case SO_BROADCAST:
- case SO_BSDCOMPAT:
- case SO_PASSCRED:
- case SO_PASSSEC:
- case SO_RXQ_OVFL:
- case SO_WIFI_STATUS:
- case SO_NOFCS:
- case SO_SELECT_ERR_QUEUE:
- return true;
- }
-
- /* SO_OOBINLINE is not supported, let's avoid the related mess */
- /* SO_ATTACH_FILTER, SO_ATTACH_BPF, SO_ATTACH_REUSEPORT_CBPF,
- * SO_DETACH_REUSEPORT_BPF, SO_DETACH_FILTER, SO_LOCK_FILTER,
- * we must be careful with subflows
- */
- /* SO_ATTACH_REUSEPORT_EBPF is not supported, at it checks
- * explicitly the sk_protocol field
- */
- /* SO_PEEK_OFF is unsupported, as it is for plain TCP */
- /* SO_MAX_PACING_RATE is unsupported, we must be careful with subflows */
- /* SO_CNX_ADVICE is currently unsupported, could possibly be relevant,
- * but likely needs careful design
- */
- /* SO_ZEROCOPY is currently unsupported, TODO in sndmsg */
- /* SO_TXTIME is currently unsupported */
- return false;
- }
if (level == SOL_IP) {
switch (optname) {
/* should work fine */
}
if (ret == 0)
- tcp_set_congestion_control(sk, name, false, cap_net_admin);
+ strcpy(msk->ca_name, name);
release_sock(sk);
return ret;
pr_debug("msk=%p", msk);
- if (!mptcp_supported_sockopt(level, optname))
- return -ENOPROTOOPT;
-
if (level == SOL_SOCKET)
return mptcp_setsockopt_sol_socket(msk, optname, optval, optlen);
+ if (!mptcp_supported_sockopt(level, optname))
+ return -ENOPROTOOPT;
+
/* @@ the meaning of setsockopt() when the socket is connected and
* there are multiple subflows is not yet defined. It is up to the
* MPTCP-level socket to configure the subflows until the subflow
sock_valbool_flag(ssk, SOCK_DBG, sock_flag(sk, SOCK_DBG));
if (inet_csk(sk)->icsk_ca_ops != inet_csk(ssk)->icsk_ca_ops)
- tcp_set_congestion_control(ssk, inet_csk(sk)->icsk_ca_ops->name, false, true);
+ tcp_set_congestion_control(ssk, msk->ca_name, false, true);
}
static void __mptcp_sockopt_sync(struct mptcp_sock *msk, struct sock *ssk)
subflow_req->mp_capable = 0;
subflow_req->mp_join = 0;
+ subflow_req->csum_reqd = mptcp_is_checksum_enabled(sock_net(sk_listener));
+ subflow_req->allow_join_id0 = mptcp_allow_join_id0(sock_net(sk_listener));
subflow_req->msk = NULL;
mptcp_token_init_request(req);
}
return -EINVAL;
#endif
- mptcp_get_options(skb, &mp_opt);
+ mptcp_get_options(sk_listener, skb, &mp_opt);
if (mp_opt.mp_capable) {
SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_MPCAPABLEPASSIVE);
}
if (mp_opt.mp_capable && listener->request_mptcp) {
- int err, retries = 4;
+ int err, retries = MPTCP_TOKEN_MAX_RETRIES;
subflow_req->ssn_offset = TCP_SKB_CB(skb)->seq;
again:
int err;
subflow_init_req(req, sk_listener);
- mptcp_get_options(skb, &mp_opt);
+ mptcp_get_options(sk_listener, skb, &mp_opt);
if (mp_opt.mp_capable && mp_opt.mp_join)
return -EINVAL;
subflow->ssn_offset = TCP_SKB_CB(skb)->seq;
pr_debug("subflow=%p synack seq=%x", subflow, subflow->ssn_offset);
- mptcp_get_options(skb, &mp_opt);
+ mptcp_get_options(sk, skb, &mp_opt);
if (subflow->request_mptcp) {
if (!mp_opt.mp_capable) {
MPTCP_INC_STATS(sock_net(sk),
goto fallback;
}
+ if (mp_opt.csum_reqd)
+ WRITE_ONCE(mptcp_sk(parent)->csum_enabled, true);
+ if (mp_opt.deny_join_id0)
+ WRITE_ONCE(mptcp_sk(parent)->pm.remote_deny_join_id0, true);
subflow->mp_capable = 1;
subflow->can_ack = 1;
subflow->remote_key = mp_opt.sndr_key;
goto do_reset;
}
+ if (!mptcp_finish_join(sk))
+ goto do_reset;
+
subflow_generate_hmac(subflow->local_key, subflow->remote_key,
subflow->local_nonce,
subflow->remote_nonce,
hmac);
memcpy(subflow->hmac, hmac, MPTCPOPT_HMAC_LEN);
- if (!mptcp_finish_join(sk))
- goto do_reset;
-
subflow->mp_join = 1;
MPTCP_INC_STATS(sock_net(sk), MPTCP_MIB_JOINSYNACKRX);
/* if the sk is MP_CAPABLE, we try to fetch the client key */
if (subflow_req->mp_capable) {
- if (TCP_SKB_CB(skb)->seq != subflow_req->ssn_offset + 1) {
- /* here we can receive and accept an in-window,
- * out-of-order pkt, which will not carry the MP_CAPABLE
- * opt even on mptcp enabled paths
- */
- goto create_msk;
- }
-
- mptcp_get_options(skb, &mp_opt);
+ /* we can receive and accept an in-window, out-of-order pkt,
+ * which may not carry the MP_CAPABLE opt even on mptcp enabled
+ * paths: always try to extract the peer key, and fallback
+ * for packets missing it.
+ * Even OoO DSS packets coming legitly after dropped or
+ * reordered MPC will cause fallback, but we don't have other
+ * options.
+ */
+ mptcp_get_options(sk, skb, &mp_opt);
if (!mp_opt.mp_capable) {
fallback = true;
goto create_child;
}
-create_msk:
new_msk = mptcp_sk_clone(listener->conn, &mp_opt, req);
if (!new_msk)
fallback = true;
} else if (subflow_req->mp_join) {
- mptcp_get_options(skb, &mp_opt);
+ mptcp_get_options(sk, skb, &mp_opt);
if (!mp_opt.mp_join || !subflow_hmac_valid(req, &mp_opt) ||
!mptcp_can_accept_new_subflow(subflow_req->msk)) {
SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_JOINACKMAC);
return seq | ((old_seq + old_data_len + 1) & GENMASK_ULL(63, 32));
}
-static void warn_bad_map(struct mptcp_subflow_context *subflow, u32 ssn)
+static void dbg_bad_map(struct mptcp_subflow_context *subflow, u32 ssn)
{
- WARN_ONCE(1, "Bad mapping: ssn=%d map_seq=%d map_data_len=%d",
- ssn, subflow->map_subflow_seq, subflow->map_data_len);
+ pr_debug("Bad mapping: ssn=%d map_seq=%d map_data_len=%d",
+ ssn, subflow->map_subflow_seq, subflow->map_data_len);
}
static bool skb_is_fully_mapped(struct sock *ssk, struct sk_buff *skb)
/* Mapping covers data later in the subflow stream,
* currently unsupported.
*/
- warn_bad_map(subflow, ssn);
+ dbg_bad_map(subflow, ssn);
return false;
}
if (unlikely(!before(ssn, subflow->map_subflow_seq +
subflow->map_data_len))) {
/* Mapping does covers past subflow data, invalid */
- warn_bad_map(subflow, ssn + skb->len);
+ dbg_bad_map(subflow, ssn);
return false;
}
return true;
}
+static enum mapping_status validate_data_csum(struct sock *ssk, struct sk_buff *skb,
+ bool csum_reqd)
+{
+ struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(ssk);
+ struct csum_pseudo_header header;
+ u32 offset, seq, delta;
+ __wsum csum;
+ int len;
+
+ if (!csum_reqd)
+ return MAPPING_OK;
+
+ /* mapping already validated on previous traversal */
+ if (subflow->map_csum_len == subflow->map_data_len)
+ return MAPPING_OK;
+
+ /* traverse the receive queue, ensuring it contains a full
+ * DSS mapping and accumulating the related csum.
+ * Preserve the accoumlate csum across multiple calls, to compute
+ * the csum only once
+ */
+ delta = subflow->map_data_len - subflow->map_csum_len;
+ for (;;) {
+ seq = tcp_sk(ssk)->copied_seq + subflow->map_csum_len;
+ offset = seq - TCP_SKB_CB(skb)->seq;
+
+ /* if the current skb has not been accounted yet, csum its contents
+ * up to the amount covered by the current DSS
+ */
+ if (offset < skb->len) {
+ __wsum csum;
+
+ len = min(skb->len - offset, delta);
+ csum = skb_checksum(skb, offset, len, 0);
+ subflow->map_data_csum = csum_block_add(subflow->map_data_csum, csum,
+ subflow->map_csum_len);
+
+ delta -= len;
+ subflow->map_csum_len += len;
+ }
+ if (delta == 0)
+ break;
+
+ if (skb_queue_is_last(&ssk->sk_receive_queue, skb)) {
+ /* if this subflow is closed, the partial mapping
+ * will be never completed; flush the pending skbs, so
+ * that subflow_sched_work_if_closed() can kick in
+ */
+ if (unlikely(ssk->sk_state == TCP_CLOSE))
+ while ((skb = skb_peek(&ssk->sk_receive_queue)))
+ sk_eat_skb(ssk, skb);
+
+ /* not enough data to validate the csum */
+ return MAPPING_EMPTY;
+ }
+
+ /* the DSS mapping for next skbs will be validated later,
+ * when a get_mapping_status call will process such skb
+ */
+ skb = skb->next;
+ }
+
+ /* note that 'map_data_len' accounts only for the carried data, does
+ * not include the eventual seq increment due to the data fin,
+ * while the pseudo header requires the original DSS data len,
+ * including that
+ */
+ header.data_seq = cpu_to_be64(subflow->map_seq);
+ header.subflow_seq = htonl(subflow->map_subflow_seq);
+ header.data_len = htons(subflow->map_data_len + subflow->map_data_fin);
+ header.csum = 0;
+
+ csum = csum_partial(&header, sizeof(header), subflow->map_data_csum);
+ if (unlikely(csum_fold(csum))) {
+ MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_DATACSUMERR);
+ return subflow->mp_join ? MAPPING_INVALID : MAPPING_DUMMY;
+ }
+
+ return MAPPING_OK;
+}
+
static enum mapping_status get_mapping_status(struct sock *ssk,
struct mptcp_sock *msk)
{
struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(ssk);
+ bool csum_reqd = READ_ONCE(msk->csum_enabled);
struct mptcp_ext *mpext;
struct sk_buff *skb;
u16 data_len;
data_len = mpext->data_len;
if (data_len == 0) {
- pr_err("Infinite mapping not handled");
MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_INFINITEMAPRX);
return MAPPING_INVALID;
}
/* Allow replacing only with an identical map */
if (subflow->map_seq == map_seq &&
subflow->map_subflow_seq == mpext->subflow_seq &&
- subflow->map_data_len == data_len) {
+ subflow->map_data_len == data_len &&
+ subflow->map_csum_reqd == mpext->csum_reqd) {
skb_ext_del(skb, SKB_EXT_MPTCP);
- return MAPPING_OK;
+ goto validate_csum;
}
/* If this skb data are fully covered by the current mapping,
}
/* will validate the next map after consuming the current one */
- return MAPPING_OK;
+ goto validate_csum;
}
subflow->map_seq = map_seq;
subflow->map_subflow_seq = mpext->subflow_seq;
subflow->map_data_len = data_len;
subflow->map_valid = 1;
+ subflow->map_data_fin = mpext->data_fin;
subflow->mpc_map = mpext->mpc_map;
- pr_debug("new map seq=%llu subflow_seq=%u data_len=%u",
+ subflow->map_csum_reqd = mpext->csum_reqd;
+ subflow->map_csum_len = 0;
+ subflow->map_data_csum = csum_unfold(mpext->csum);
+
+ /* Cfr RFC 8684 Section 3.3.0 */
+ if (unlikely(subflow->map_csum_reqd != csum_reqd))
+ return MAPPING_INVALID;
+
+ pr_debug("new map seq=%llu subflow_seq=%u data_len=%u csum=%d:%u",
subflow->map_seq, subflow->map_subflow_seq,
- subflow->map_data_len);
+ subflow->map_data_len, subflow->map_csum_reqd,
+ subflow->map_data_csum);
validate_seq:
/* we revalidate valid mapping on new skb, because we must ensure
* the current skb is completely covered by the available mapping
*/
- if (!validate_mapping(ssk, skb))
+ if (!validate_mapping(ssk, skb)) {
+ MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_DSSTCPMISMATCH);
return MAPPING_INVALID;
+ }
skb_ext_del(skb, SKB_EXT_MPTCP);
- return MAPPING_OK;
+
+validate_csum:
+ return validate_data_csum(ssk, skb, csum_reqd);
}
static void mptcp_subflow_discard_data(struct sock *ssk, struct sk_buff *skb,
struct sk_buff *skb;
if (!skb_peek(&ssk->sk_receive_queue))
- subflow->data_avail = 0;
+ WRITE_ONCE(subflow->data_avail, 0);
if (subflow->data_avail)
return true;
status = get_mapping_status(ssk, msk);
trace_subflow_check_data_avail(status, skb_peek(&ssk->sk_receive_queue));
- if (status == MAPPING_INVALID) {
- ssk->sk_err = EBADMSG;
- goto fatal;
- }
- if (status == MAPPING_DUMMY) {
- __mptcp_do_fallback(msk);
- skb = skb_peek(&ssk->sk_receive_queue);
- subflow->map_valid = 1;
- subflow->map_seq = READ_ONCE(msk->ack_seq);
- subflow->map_data_len = skb->len;
- subflow->map_subflow_seq = tcp_sk(ssk)->copied_seq -
- subflow->ssn_offset;
- subflow->data_avail = MPTCP_SUBFLOW_DATA_AVAIL;
- return true;
- }
+ if (unlikely(status == MAPPING_INVALID))
+ goto fallback;
+
+ if (unlikely(status == MAPPING_DUMMY))
+ goto fallback;
if (status != MAPPING_OK)
goto no_data;
* MP_CAPABLE-based mapping
*/
if (unlikely(!READ_ONCE(msk->can_ack))) {
- if (!subflow->mpc_map) {
- ssk->sk_err = EBADMSG;
- goto fatal;
- }
+ if (!subflow->mpc_map)
+ goto fallback;
WRITE_ONCE(msk->remote_key, subflow->remote_key);
WRITE_ONCE(msk->ack_seq, subflow->map_seq);
WRITE_ONCE(msk->can_ack, true);
ack_seq = mptcp_subflow_get_mapped_dsn(subflow);
pr_debug("msk ack_seq=%llx subflow ack_seq=%llx", old_ack,
ack_seq);
- if (ack_seq == old_ack) {
- subflow->data_avail = MPTCP_SUBFLOW_DATA_AVAIL;
- break;
- } else if (after64(ack_seq, old_ack)) {
- subflow->data_avail = MPTCP_SUBFLOW_OOO_DATA;
- break;
+ if (unlikely(before64(ack_seq, old_ack))) {
+ mptcp_subflow_discard_data(ssk, skb, old_ack - ack_seq);
+ continue;
}
- /* only accept in-sequence mapping. Old values are spurious
- * retransmission
- */
- mptcp_subflow_discard_data(ssk, skb, old_ack - ack_seq);
+ WRITE_ONCE(subflow->data_avail, MPTCP_SUBFLOW_DATA_AVAIL);
+ break;
}
return true;
no_data:
subflow_sched_work_if_closed(msk, ssk);
return false;
-fatal:
- /* fatal protocol error, close the socket */
- /* This barrier is coupled with smp_rmb() in tcp_poll() */
- smp_wmb();
- ssk->sk_error_report(ssk);
- tcp_set_state(ssk, TCP_CLOSE);
- subflow->reset_transient = 0;
- subflow->reset_reason = MPTCP_RST_EMPTCP;
- tcp_send_active_reset(ssk, GFP_ATOMIC);
- subflow->data_avail = 0;
- return false;
+
+fallback:
+ /* RFC 8684 section 3.7. */
+ if (subflow->mp_join || subflow->fully_established) {
+ /* fatal protocol error, close the socket.
+ * subflow_error_report() will introduce the appropriate barriers
+ */
+ ssk->sk_err = EBADMSG;
+ tcp_set_state(ssk, TCP_CLOSE);
+ subflow->reset_transient = 0;
+ subflow->reset_reason = MPTCP_RST_EMPTCP;
+ tcp_send_active_reset(ssk, GFP_ATOMIC);
+ WRITE_ONCE(subflow->data_avail, 0);
+ return false;
+ }
+
+ __mptcp_do_fallback(msk);
+ skb = skb_peek(&ssk->sk_receive_queue);
+ subflow->map_valid = 1;
+ subflow->map_seq = READ_ONCE(msk->ack_seq);
+ subflow->map_data_len = skb->len;
+ subflow->map_subflow_seq = tcp_sk(ssk)->copied_seq - subflow->ssn_offset;
+ WRITE_ONCE(subflow->data_avail, MPTCP_SUBFLOW_DATA_AVAIL);
+ return true;
}
bool mptcp_subflow_data_available(struct sock *sk)
if (subflow->map_valid &&
mptcp_subflow_get_map_offset(subflow) >= subflow->map_data_len) {
subflow->map_valid = 0;
- subflow->data_avail = 0;
+ WRITE_ONCE(subflow->data_avail, 0);
pr_debug("Done with mapping: seq=%u data_len=%u",
subflow->map_subflow_seq,
*full_space = tcp_full_space(sk);
}
-static void subflow_data_ready(struct sock *sk)
-{
- struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(sk);
- u16 state = 1 << inet_sk_state_load(sk);
- struct sock *parent = subflow->conn;
- struct mptcp_sock *msk;
-
- msk = mptcp_sk(parent);
- if (state & TCPF_LISTEN) {
- /* MPJ subflow are removed from accept queue before reaching here,
- * avoid stray wakeups
- */
- if (reqsk_queue_empty(&inet_csk(sk)->icsk_accept_queue))
- return;
-
- set_bit(MPTCP_DATA_READY, &msk->flags);
- parent->sk_data_ready(parent);
- return;
- }
-
- WARN_ON_ONCE(!__mptcp_check_fallback(msk) && !subflow->mp_capable &&
- !subflow->mp_join && !(state & TCPF_CLOSE));
-
- if (mptcp_subflow_data_available(sk))
- mptcp_data_ready(parent, sk);
-}
-
-static void subflow_write_space(struct sock *ssk)
-{
- struct sock *sk = mptcp_subflow_ctx(ssk)->conn;
-
- mptcp_propagate_sndbuf(sk, ssk);
- mptcp_write_space(sk);
-}
-
void __mptcp_error_report(struct sock *sk)
{
struct mptcp_subflow_context *subflow;
mptcp_data_unlock(sk);
}
+static void subflow_data_ready(struct sock *sk)
+{
+ struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(sk);
+ u16 state = 1 << inet_sk_state_load(sk);
+ struct sock *parent = subflow->conn;
+ struct mptcp_sock *msk;
+
+ msk = mptcp_sk(parent);
+ if (state & TCPF_LISTEN) {
+ /* MPJ subflow are removed from accept queue before reaching here,
+ * avoid stray wakeups
+ */
+ if (reqsk_queue_empty(&inet_csk(sk)->icsk_accept_queue))
+ return;
+
+ set_bit(MPTCP_DATA_READY, &msk->flags);
+ parent->sk_data_ready(parent);
+ return;
+ }
+
+ WARN_ON_ONCE(!__mptcp_check_fallback(msk) && !subflow->mp_capable &&
+ !subflow->mp_join && !(state & TCPF_CLOSE));
+
+ if (mptcp_subflow_data_available(sk))
+ mptcp_data_ready(parent, sk);
+ else if (unlikely(sk->sk_err))
+ subflow_error_report(sk);
+}
+
+static void subflow_write_space(struct sock *ssk)
+{
+ struct sock *sk = mptcp_subflow_ctx(ssk)->conn;
+
+ mptcp_propagate_sndbuf(sk, ssk);
+ mptcp_write_space(sk);
+}
+
static struct inet_connection_sock_af_ops *
subflow_default_af_ops(struct sock *sk)
{
*/
if (mptcp_subflow_data_available(sk))
mptcp_data_ready(parent, sk);
+ else if (unlikely(sk->sk_err))
+ subflow_error_report(sk);
subflow_sched_work_if_closed(mptcp_sk(parent), sk);
#include <net/mptcp.h>
#include "protocol.h"
-#define TOKEN_MAX_RETRIES 4
#define TOKEN_MAX_CHAIN_LEN 4
struct token_bucket {
{
struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(sk);
struct mptcp_sock *msk = mptcp_sk(subflow->conn);
- int retries = TOKEN_MAX_RETRIES;
+ int retries = MPTCP_TOKEN_MAX_RETRIES;
struct token_bucket *bucket;
- pr_debug("ssk=%p, local_key=%llu, token=%u, idsn=%llu\n",
- sk, subflow->local_key, subflow->token, subflow->idsn);
-
again:
mptcp_crypto_key_gen_sha(&subflow->local_key, &subflow->token,
&subflow->idsn);
goto again;
}
+ pr_debug("ssk=%p, local_key=%llu, token=%u, idsn=%llu\n",
+ sk, subflow->local_key, subflow->token, subflow->idsn);
+
WRITE_ONCE(msk->token, subflow->token);
__sk_nulls_add_node_rcu((struct sock *)msk, &bucket->msk_chain);
bucket->chain_len++;
struct ncsi_dev_priv *ndp; /* NCSI device */
spinlock_t lock; /* Protect the package */
unsigned int channel_num; /* Number of channels */
- struct list_head channels; /* List of chanels */
+ struct list_head channels; /* List of channels */
struct list_head node; /* Form list of packages */
bool multi_channel; /* Enable multiple channels */
unsigned char type; /* Command in the NCSI packet */
unsigned char id; /* Request ID (sequence number) */
unsigned char package; /* Destination package ID */
- unsigned char channel; /* Detination channel ID or 0x1f */
+ unsigned char channel; /* Destination channel ID or 0x1f */
unsigned short payload; /* Command packet payload length */
unsigned int req_flags; /* NCSI request properties */
union {
return 0;
}
-/* Find an outstanding VLAN tag and constuct a "Set VLAN Filter - Enable"
+/* Find an outstanding VLAN tag and construct a "Set VLAN Filter - Enable"
* packet.
*/
static int set_one_vid(struct ncsi_dev_priv *ndp, struct ncsi_channel *nc,
config NETFILTER_FAMILY_ARP
bool
+config NETFILTER_NETLINK_HOOK
+ tristate "Netfilter base hook dump support"
+ depends on NETFILTER_ADVANCED
+ depends on NF_TABLES
+ select NETFILTER_NETLINK
+ help
+ If this option is enabled, the kernel will include support
+ to list the base netfilter hooks via NFNETLINK.
+ This is helpful for debugging.
+
config NETFILTER_NETLINK_ACCT
tristate "Netfilter NFACCT over NFNETLINK interface"
depends on NETFILTER_ADVANCED
the priority of a packet. Some qdiscs can use this value for
classification, among these are:
- atm, cbq, dsmark, pfifo_fast, htb, prio
+ atm, cbq, dsmark, pfifo_fast, htb, prio
To compile it as a module, choose M here. If unsure, say N.
obj-$(CONFIG_NETFILTER_NETLINK_QUEUE) += nfnetlink_queue.o
obj-$(CONFIG_NETFILTER_NETLINK_LOG) += nfnetlink_log.o
obj-$(CONFIG_NETFILTER_NETLINK_OSF) += nfnetlink_osf.o
+obj-$(CONFIG_NETFILTER_NETLINK_HOOK) += nfnetlink_hook.o
# connection tracking
obj-$(CONFIG_NF_CONNTRACK) += nf_conntrack.o
nf_tables-objs := nf_tables_core.o nf_tables_api.o nft_chain_filter.o \
nf_tables_trace.o nft_immediate.o nft_cmp.o nft_range.o \
nft_bitwise.o nft_byteorder.o nft_payload.o nft_lookup.o \
- nft_dynset.o nft_meta.o nft_rt.o nft_exthdr.o \
+ nft_dynset.o nft_meta.o nft_rt.o nft_exthdr.o nft_last.o \
nft_chain_route.o nf_tables_offload.o \
nft_set_hash.o nft_set_bitmap.o nft_set_rbtree.o \
nft_set_pipapo.o
};
static int
-call_ad(struct sock *ctnl, struct sk_buff *skb, struct ip_set *set,
- struct nlattr *tb[], enum ipset_adt adt,
+call_ad(struct net *net, struct sock *ctnl, struct sk_buff *skb,
+ struct ip_set *set, struct nlattr *tb[], enum ipset_adt adt,
u32 flags, bool use_lineno)
{
int ret;
*errline = lineno;
- netlink_unicast(ctnl, skb2, NETLINK_CB(skb).portid,
- MSG_DONTWAIT);
+ nfnetlink_unicast(skb2, net, NETLINK_CB(skb).portid);
/* Signal netlink not to send its ACK/errmsg. */
return -EINTR;
}
attr[IPSET_ATTR_DATA],
set->type->adt_policy, NULL))
return -IPSET_ERR_PROTOCOL;
- ret = call_ad(ctnl, skb, set, tb, adt, flags,
+ ret = call_ad(net, ctnl, skb, set, tb, adt, flags,
use_lineno);
} else {
int nla_rem;
nla_parse_nested(tb, IPSET_ATTR_ADT_MAX, nla,
set->type->adt_policy, NULL))
return -IPSET_ERR_PROTOCOL;
- ret = call_ad(ctnl, skb, set, tb, adt,
+ ret = call_ad(net, ctnl, skb, set, tb, adt,
flags, use_lineno);
if (ret < 0)
return ret;
const struct ip_set *set;
struct sk_buff *skb2;
struct nlmsghdr *nlh2;
- int ret = 0;
if (unlikely(protocol_min_failed(attr) ||
!attr[IPSET_ATTR_SETNAME]))
goto nla_put_failure;
nlmsg_end(skb2, nlh2);
- ret = netlink_unicast(info->sk, skb2, NETLINK_CB(skb).portid,
- MSG_DONTWAIT);
- if (ret < 0)
- return ret;
-
- return 0;
+ return nfnetlink_unicast(skb2, info->net, NETLINK_CB(skb).portid);
nla_put_failure:
nlmsg_cancel(skb2, nlh2);
nlmsg_end(skb2, nlh2);
pr_debug("Send TYPE, nlmsg_len: %u\n", nlh2->nlmsg_len);
- ret = netlink_unicast(info->sk, skb2, NETLINK_CB(skb).portid,
- MSG_DONTWAIT);
- if (ret < 0)
- return ret;
-
- return 0;
+ return nfnetlink_unicast(skb2, info->net, NETLINK_CB(skb).portid);
nla_put_failure:
nlmsg_cancel(skb2, nlh2);
{
struct sk_buff *skb2;
struct nlmsghdr *nlh2;
- int ret = 0;
if (unlikely(!attr[IPSET_ATTR_PROTOCOL]))
return -IPSET_ERR_PROTOCOL;
goto nla_put_failure;
nlmsg_end(skb2, nlh2);
- ret = netlink_unicast(info->sk, skb2, NETLINK_CB(skb).portid,
- MSG_DONTWAIT);
- if (ret < 0)
- return ret;
-
- return 0;
+ return nfnetlink_unicast(skb2, info->net, NETLINK_CB(skb).portid);
nla_put_failure:
nlmsg_cancel(skb2, nlh2);
struct nlmsghdr *nlh2;
ip_set_id_t id = IPSET_INVALID_ID;
const struct ip_set *set;
- int ret = 0;
if (unlikely(protocol_failed(attr) ||
!attr[IPSET_ATTR_SETNAME]))
goto nla_put_failure;
nlmsg_end(skb2, nlh2);
- ret = netlink_unicast(info->sk, skb2, NETLINK_CB(skb).portid,
- MSG_DONTWAIT);
- if (ret < 0)
- return ret;
-
- return 0;
+ return nfnetlink_unicast(skb2, info->net, NETLINK_CB(skb).portid);
nla_put_failure:
nlmsg_cancel(skb2, nlh2);
struct nlmsghdr *nlh2;
ip_set_id_t id = IPSET_INVALID_ID;
const struct ip_set *set;
- int ret = 0;
if (unlikely(protocol_failed(attr) ||
!attr[IPSET_ATTR_INDEX]))
goto nla_put_failure;
nlmsg_end(skb2, nlh2);
- ret = netlink_unicast(info->sk, skb2, NETLINK_CB(skb).portid,
- MSG_DONTWAIT);
- if (ret < 0)
- return ret;
-
- return 0;
+ return nfnetlink_unicast(skb2, info->net, NETLINK_CB(skb).portid);
nla_put_failure:
nlmsg_cancel(skb2, nlh2);
comment 'IPVS application helper'
config IP_VS_FTP
- tristate "FTP protocol helper"
+ tristate "FTP protocol helper"
depends on IP_VS_PROTO_TCP && NF_CONNTRACK && NF_NAT && \
NF_CONNTRACK_FTP
select IP_VS_NFCT
ip_vs_addr_copy(svc->af, &svc->addr, &u->addr);
svc->port = u->port;
svc->fwmark = u->fwmark;
- svc->flags = u->flags;
+ svc->flags = u->flags & ~IP_VS_SVC_F_HASHED;
svc->timeout = u->timeout * HZ;
svc->netmask = u->netmask;
svc->ipvs = ipvs;
#include "nf_internals.h"
-extern unsigned int nf_conntrack_net_id;
-
__cacheline_aligned_in_smp spinlock_t nf_conntrack_locks[CONNTRACK_LOCKS];
EXPORT_SYMBOL_GPL(nf_conntrack_locks);
static struct conntrack_gc_work conntrack_gc_work;
-extern unsigned int nf_conntrack_net_id;
-
void nf_conntrack_lock(spinlock_t *lock) __acquires(lock)
{
/* 1) Acquire the lock */
continue;
net = nf_ct_net(tmp);
- cnet = net_generic(net, nf_conntrack_net_id);
+ cnet = nf_ct_pernet(net);
if (atomic_read(&cnet->count) < nf_conntrack_max95)
continue;
const struct nf_conntrack_tuple *repl,
gfp_t gfp, u32 hash)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
unsigned int ct_count;
struct nf_conn *ct;
nf_ct_ext_destroy(ct);
kmem_cache_free(nf_conntrack_cachep, ct);
- cnet = net_generic(net, nf_conntrack_net_id);
+ cnet = nf_ct_pernet(net);
smp_mb__before_atomic();
atomic_dec(&cnet->count);
GFP_ATOMIC);
local_bh_disable();
- cnet = net_generic(net, nf_conntrack_net_id);
+ cnet = nf_ct_pernet(net);
if (cnet->expect_count) {
spin_lock(&nf_conntrack_expect_lock);
exp = nf_ct_find_expectation(net, zone, tuple);
void nf_ct_unconfirmed_destroy(struct net *net)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
might_sleep();
int (*iter)(struct nf_conn *i, void *data),
void *data, u32 portid, int report)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
struct iter_data d;
might_sleep();
down_read(&net_rwsem);
for_each_net(net) {
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
if (atomic_read(&cnet->count) == 0)
continue;
i_see_dead_people:
busy = 0;
list_for_each_entry(net, net_exit_list, exit_list) {
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
nf_ct_iterate_cleanup(kill_all, net, 0, 0);
if (atomic_read(&cnet->count) != 0)
int nf_conntrack_init_net(struct net *net)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
int ret = -ENOMEM;
int cpu;
#include <net/netfilter/nf_conntrack_ecache.h>
#include <net/netfilter/nf_conntrack_extend.h>
-extern unsigned int nf_conntrack_net_id;
-
static DEFINE_MUTEX(nf_ct_ecache_mutex);
#define ECACHE_RETRY_WAIT (HZ/10)
void nf_conntrack_ecache_work(struct net *net, enum nf_ct_ecache_state state)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
if (state == NFCT_ECACHE_DESTROY_FAIL &&
!delayed_work_pending(&cnet->ecache_dwork)) {
void nf_conntrack_ecache_pernet_init(struct net *net)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
net->ct.sysctl_events = nf_ct_events;
cnet->ct_net = &net->ct;
void nf_conntrack_ecache_pernet_fini(struct net *net)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
cancel_delayed_work_sync(&cnet->ecache_dwork);
}
static struct kmem_cache *nf_ct_expect_cachep __read_mostly;
static unsigned int nf_ct_expect_hashrnd __read_mostly;
-extern unsigned int nf_conntrack_net_id;
-
/* nf_conntrack_expect helper functions */
void nf_ct_unlink_expect_report(struct nf_conntrack_expect *exp,
u32 portid, int report)
hlist_del_rcu(&exp->hnode);
- cnet = net_generic(net, nf_conntrack_net_id);
+ cnet = nf_ct_pernet(net);
cnet->expect_count--;
hlist_del_rcu(&exp->lnode);
const struct nf_conntrack_zone *zone,
const struct nf_conntrack_tuple *tuple)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
struct nf_conntrack_expect *i;
unsigned int h;
const struct nf_conntrack_zone *zone,
const struct nf_conntrack_tuple *tuple)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
struct nf_conntrack_expect *i, *exp = NULL;
unsigned int h;
master_help->expecting[exp->class]++;
hlist_add_head_rcu(&exp->hnode, &nf_ct_expect_hash[h]);
- cnet = net_generic(net, nf_conntrack_net_id);
+ cnet = nf_ct_pernet(net);
cnet->expect_count++;
NF_CT_STAT_INC(net, expect_create);
}
}
- cnet = net_generic(net, nf_conntrack_net_id);
+ cnet = nf_ct_pernet(net);
if (cnet->expect_count >= nf_ct_expect_max) {
net_warn_ratelimited("nf_conntrack: expectation table full\n");
ret = -EMFILE;
if (tcpdatalen == 4) { /* Separate TPKT header */
/* Netmeeting sends TPKT header and data separately */
pr_debug("nf_ct_h323: separate TPKT header indicates "
- "there will be TPKT data of %hu bytes\n",
+ "there will be TPKT data of %d bytes\n",
tpktlen - 4);
info->tpkt_len[dir] = tpktlen - 4;
return 0;
static DEFINE_MUTEX(nf_ct_nat_helpers_mutex);
static struct list_head nf_ct_nat_helpers __read_mostly;
-extern unsigned int nf_conntrack_net_id;
-
/* Stupid hash, but collision free for the default registrations of the
* helpers currently in the kernel. */
static unsigned int helper_hash(const struct nf_conntrack_tuple *tuple)
static struct nf_conntrack_helper *
nf_ct_lookup_helper(struct nf_conn *ct, struct net *net)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
if (!cnet->sysctl_auto_assign_helper) {
if (cnet->auto_assign_helper_warned)
void nf_conntrack_helper_pernet_init(struct net *net)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
cnet->sysctl_auto_assign_helper = nf_ct_auto_assign_helper;
}
const struct nfnl_info *info,
const struct nlattr * const cda[])
{
- struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
+ u8 family = info->nfmsg->nfgen_family;
struct nf_conntrack_tuple_hash *h;
struct nf_conntrack_tuple tuple;
struct nf_conntrack_zone zone;
if (cda[CTA_TUPLE_ORIG])
err = ctnetlink_parse_tuple(cda, &tuple, CTA_TUPLE_ORIG,
- nfmsg->nfgen_family, &zone);
+ family, &zone);
else if (cda[CTA_TUPLE_REPLY])
err = ctnetlink_parse_tuple(cda, &tuple, CTA_TUPLE_REPLY,
- nfmsg->nfgen_family, &zone);
+ family, &zone);
else {
- u_int8_t u3 = nfmsg->version ? nfmsg->nfgen_family : AF_UNSPEC;
+ u_int8_t u3 = info->nfmsg->version ? family : AF_UNSPEC;
return ctnetlink_flush_conntrack(info->net, cda,
NETLINK_CB(skb).portid,
const struct nfnl_info *info,
const struct nlattr * const cda[])
{
- struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
- u_int8_t u3 = nfmsg->nfgen_family;
+ u_int8_t u3 = info->nfmsg->nfgen_family;
struct nf_conntrack_tuple_hash *h;
struct nf_conntrack_tuple tuple;
struct nf_conntrack_zone zone;
ct = nf_ct_tuplehash_to_ctrack(h);
- err = -ENOMEM;
skb2 = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
- if (skb2 == NULL) {
+ if (!skb2) {
nf_ct_put(ct);
return -ENOMEM;
}
NFNL_MSG_TYPE(info->nlh->nlmsg_type), ct,
true, 0);
nf_ct_put(ct);
- if (err <= 0)
- goto free;
-
- err = netlink_unicast(info->sk, skb2, NETLINK_CB(skb).portid,
- MSG_DONTWAIT);
- if (err < 0)
- goto out;
-
- return 0;
+ if (err <= 0) {
+ kfree_skb(skb2);
+ return -ENOMEM;
+ }
-free:
- kfree_skb(skb2);
-out:
- /* this avoids a loop in nfnetlink. */
- return err == -EAGAIN ? -ENOBUFS : err;
+ return nfnetlink_unicast(skb2, info->net, NETLINK_CB(skb).portid);
}
static int ctnetlink_done_list(struct netlink_callback *cb)
const struct nfnl_info *info,
const struct nlattr * const cda[])
{
- struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct nf_conntrack_tuple otuple, rtuple;
struct nf_conntrack_tuple_hash *h = NULL;
- u_int8_t u3 = nfmsg->nfgen_family;
+ u_int8_t u3 = info->nfmsg->nfgen_family;
struct nf_conntrack_zone zone;
struct nf_conn *ct;
int err;
info->nlh->nlmsg_seq,
NFNL_MSG_TYPE(info->nlh->nlmsg_type),
sock_net(skb->sk));
- if (err <= 0)
- goto free;
-
- err = netlink_unicast(info->sk, skb2, NETLINK_CB(skb).portid,
- MSG_DONTWAIT);
- if (err < 0)
- goto out;
-
- return 0;
+ if (err <= 0) {
+ kfree_skb(skb2);
+ return -ENOMEM;
+ }
-free:
- kfree_skb(skb2);
-out:
- /* this avoids a loop in nfnetlink. */
- return err == -EAGAIN ? -ENOBUFS : err;
+ return nfnetlink_unicast(skb2, info->net, NETLINK_CB(skb).portid);
}
static const struct nla_policy exp_nla_policy[CTA_EXPECT_MAX+1] = {
const struct nfnl_info *info,
const struct nlattr * const cda[])
{
- struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
- u_int8_t u3 = nfmsg->nfgen_family;
+ u_int8_t u3 = info->nfmsg->nfgen_family;
struct nf_conntrack_tuple tuple;
struct nf_conntrack_expect *exp;
struct nf_conntrack_zone zone;
}
}
- err = -ENOMEM;
skb2 = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
- if (skb2 == NULL) {
+ if (!skb2) {
nf_ct_expect_put(exp);
- goto out;
+ return -ENOMEM;
}
rcu_read_lock();
exp);
rcu_read_unlock();
nf_ct_expect_put(exp);
- if (err <= 0)
- goto free;
-
- err = netlink_unicast(info->sk, skb2, NETLINK_CB(skb).portid,
- MSG_DONTWAIT);
- if (err < 0)
- goto out;
-
- return 0;
+ if (err <= 0) {
+ kfree_skb(skb2);
+ return -ENOMEM;
+ }
-free:
- kfree_skb(skb2);
-out:
- /* this avoids a loop in nfnetlink. */
- return err == -EAGAIN ? -ENOBUFS : err;
+ return nfnetlink_unicast(skb2, info->net, NETLINK_CB(skb).portid);
}
static bool expect_iter_name(struct nf_conntrack_expect *exp, void *data)
const struct nfnl_info *info,
const struct nlattr * const cda[])
{
- struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
- u_int8_t u3 = nfmsg->nfgen_family;
+ u_int8_t u3 = info->nfmsg->nfgen_family;
struct nf_conntrack_expect *exp;
struct nf_conntrack_tuple tuple;
struct nf_conntrack_zone zone;
const struct nfnl_info *info,
const struct nlattr * const cda[])
{
- struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
- u_int8_t u3 = nfmsg->nfgen_family;
+ u_int8_t u3 = info->nfmsg->nfgen_family;
struct nf_conntrack_tuple tuple;
struct nf_conntrack_expect *exp;
struct nf_conntrack_zone zone;
#include <net/ipv6.h>
#include <net/inet_frag.h>
-extern unsigned int nf_conntrack_net_id;
-
static DEFINE_MUTEX(nf_ct_proto_mutex);
#ifdef CONFIG_SYSCTL
-__printf(5, 6)
+__printf(4, 5)
void nf_l4proto_log_invalid(const struct sk_buff *skb,
- struct net *net,
- u16 pf, u8 protonum,
+ const struct nf_hook_state *state,
+ u8 protonum,
const char *fmt, ...)
{
+ struct net *net = state->net;
struct va_format vaf;
va_list args;
vaf.fmt = fmt;
vaf.va = &args;
- nf_log_packet(net, pf, 0, skb, NULL, NULL, NULL,
- "nf_ct_proto_%d: %pV ", protonum, &vaf);
+ nf_log_packet(net, state->pf, 0, skb, state->in, state->out,
+ NULL, "nf_ct_proto_%d: %pV ", protonum, &vaf);
va_end(args);
}
EXPORT_SYMBOL_GPL(nf_l4proto_log_invalid);
-__printf(3, 4)
+__printf(4, 5)
void nf_ct_l4proto_log_invalid(const struct sk_buff *skb,
const struct nf_conn *ct,
+ const struct nf_hook_state *state,
const char *fmt, ...)
{
struct va_format vaf;
vaf.fmt = fmt;
vaf.va = &args;
- nf_l4proto_log_invalid(skb, net, nf_ct_l3num(ct),
+ nf_l4proto_log_invalid(skb, state,
nf_ct_protonum(ct), "%pV", &vaf);
va_end(args);
}
static int nf_ct_netns_do_get(struct net *net, u8 nfproto)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
bool fixup_needed = false, retry = true;
int err = 0;
retry:
static void nf_ct_netns_do_put(struct net *net, u8 nfproto)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
mutex_lock(&nf_ct_proto_mutex);
switch (nfproto) {
#if IS_ENABLED(CONFIG_IPV6)
cleanup_sockopt:
- nf_unregister_sockopt(&so_getorigdst6);
+ nf_unregister_sockopt(&so_getorigdst);
#endif
return ret;
}
static noinline bool
dccp_new(struct nf_conn *ct, const struct sk_buff *skb,
- const struct dccp_hdr *dh)
+ const struct dccp_hdr *dh,
+ const struct nf_hook_state *hook_state)
{
struct net *net = nf_ct_net(ct);
struct nf_dccp_net *dn;
return true;
out_invalid:
- nf_ct_l4proto_log_invalid(skb, ct, "%s", msg);
+ nf_ct_l4proto_log_invalid(skb, ct, hook_state, "%s", msg);
return false;
}
}
return false;
out_invalid:
- nf_l4proto_log_invalid(skb, state->net, state->pf,
- IPPROTO_DCCP, "%s", msg);
+ nf_l4proto_log_invalid(skb, state, IPPROTO_DCCP, "%s", msg);
return true;
}
return -NF_ACCEPT;
type = dh->dccph_type;
- if (!nf_ct_is_confirmed(ct) && !dccp_new(ct, skb, dh))
+ if (!nf_ct_is_confirmed(ct) && !dccp_new(ct, skb, dh, state))
return -NF_ACCEPT;
if (type == DCCP_PKT_RESET &&
ct->proto.dccp.last_pkt = type;
spin_unlock_bh(&ct->lock);
- nf_ct_l4proto_log_invalid(skb, ct, "%s", "invalid packet");
+ nf_ct_l4proto_log_invalid(skb, ct, state, "%s", "invalid packet");
return NF_ACCEPT;
case CT_DCCP_INVALID:
spin_unlock_bh(&ct->lock);
- nf_ct_l4proto_log_invalid(skb, ct, "%s", "invalid state transition");
+ nf_ct_l4proto_log_invalid(skb, ct, state, "%s", "invalid state transition");
return -NF_ACCEPT;
}
ct_daddr = &ct->tuplehash[dir].tuple.dst.u3;
if (!nf_inet_addr_cmp(outer_daddr, ct_daddr)) {
if (state->pf == AF_INET) {
- nf_l4proto_log_invalid(skb, state->net, state->pf,
+ nf_l4proto_log_invalid(skb, state,
l4proto,
"outer daddr %pI4 != inner %pI4",
&outer_daddr->ip, &ct_daddr->ip);
} else if (state->pf == AF_INET6) {
- nf_l4proto_log_invalid(skb, state->net, state->pf,
+ nf_l4proto_log_invalid(skb, state,
l4proto,
"outer daddr %pI6 != inner %pI6",
&outer_daddr->ip6, &ct_daddr->ip6);
const struct nf_hook_state *state,
const char *msg)
{
- nf_l4proto_log_invalid(skb, state->net, state->pf,
- IPPROTO_ICMP, "%s", msg);
+ nf_l4proto_log_invalid(skb, state, IPPROTO_ICMP, "%s", msg);
}
/* Small and modified version of icmp_rcv */
const struct nf_hook_state *state,
const char *msg)
{
- nf_l4proto_log_invalid(skb, state->net, state->pf,
- IPPROTO_ICMPV6, "%s", msg);
+ nf_l4proto_log_invalid(skb, state, IPPROTO_ICMPV6, "%s", msg);
}
int nf_conntrack_icmpv6_error(struct nf_conn *tmpl,
}
return false;
out_invalid:
- nf_l4proto_log_invalid(skb, state->net, state->pf, IPPROTO_SCTP, "%s", logmsg);
+ nf_l4proto_log_invalid(skb, state, IPPROTO_SCTP, "%s", logmsg);
return true;
}
}
}
-static bool tcp_in_window(const struct nf_conn *ct,
- struct ip_ct_tcp *state,
+static bool tcp_in_window(struct nf_conn *ct,
enum ip_conntrack_dir dir,
unsigned int index,
const struct sk_buff *skb,
unsigned int dataoff,
- const struct tcphdr *tcph)
+ const struct tcphdr *tcph,
+ const struct nf_hook_state *hook_state)
{
+ struct ip_ct_tcp *state = &ct->proto.tcp;
struct net *net = nf_ct_net(ct);
struct nf_tcp_net *tn = nf_tcp_pernet(net);
struct ip_ct_tcp_state *sender = &state->seen[dir];
tn->tcp_be_liberal)
res = true;
if (!res) {
- nf_ct_l4proto_log_invalid(skb, ct,
+ nf_ct_l4proto_log_invalid(skb, ct, hook_state,
"%s",
before(seq, sender->td_maxend + 1) ?
in_recv_win ?
const struct nf_hook_state *state,
const char *msg)
{
- nf_l4proto_log_invalid(skb, state->net, state->pf, IPPROTO_TCP, "%s", msg);
+ nf_l4proto_log_invalid(skb, state, IPPROTO_TCP, "%s", msg);
}
/* Protect conntrack agaist broken packets. Code taken from ipt_unclean.c. */
IP_CT_EXP_CHALLENGE_ACK;
}
spin_unlock_bh(&ct->lock);
- nf_ct_l4proto_log_invalid(skb, ct,
+ nf_ct_l4proto_log_invalid(skb, ct, state,
"packet (index %d) in dir %d ignored, state %s",
index, dir,
tcp_conntrack_names[old_state]);
pr_debug("nf_ct_tcp: Invalid dir=%i index=%u ostate=%u\n",
dir, get_conntrack_index(th), old_state);
spin_unlock_bh(&ct->lock);
- nf_ct_l4proto_log_invalid(skb, ct, "invalid state");
+ nf_ct_l4proto_log_invalid(skb, ct, state, "invalid state");
return -NF_ACCEPT;
case TCP_CONNTRACK_TIME_WAIT:
/* RFC5961 compliance cause stack to send "challenge-ACK"
/* Detected RFC5961 challenge ACK */
ct->proto.tcp.last_flags &= ~IP_CT_EXP_CHALLENGE_ACK;
spin_unlock_bh(&ct->lock);
- nf_ct_l4proto_log_invalid(skb, ct, "challenge-ack ignored");
+ nf_ct_l4proto_log_invalid(skb, ct, state, "challenge-ack ignored");
return NF_ACCEPT; /* Don't change state */
}
break;
if (before(seq, ct->proto.tcp.seen[!dir].td_maxack)) {
/* Invalid RST */
spin_unlock_bh(&ct->lock);
- nf_ct_l4proto_log_invalid(skb, ct, "invalid rst");
+ nf_ct_l4proto_log_invalid(skb, ct, state, "invalid rst");
return -NF_ACCEPT;
}
break;
}
- if (!tcp_in_window(ct, &ct->proto.tcp, dir, index,
- skb, dataoff, th)) {
+ if (!tcp_in_window(ct, dir, index,
+ skb, dataoff, th, state)) {
spin_unlock_bh(&ct->lock);
return -NF_ACCEPT;
}
* will be started.
*/
tn->tcp_max_retrans = 3;
+
+#if IS_ENABLED(CONFIG_NF_FLOW_TABLE)
+ tn->offload_timeout = 30 * HZ;
+ tn->offload_pickup = 120 * HZ;
+#endif
}
const struct nf_conntrack_l4proto nf_conntrack_l4proto_tcp =
const struct nf_hook_state *state,
const char *msg)
{
- nf_l4proto_log_invalid(skb, state->net, state->pf,
- IPPROTO_UDP, "%s", msg);
+ nf_l4proto_log_invalid(skb, state, IPPROTO_UDP, "%s", msg);
}
static bool udp_error(struct sk_buff *skb,
const struct nf_hook_state *state,
const char *msg)
{
- nf_l4proto_log_invalid(skb, state->net, state->pf,
- IPPROTO_UDPLITE, "%s", msg);
+ nf_l4proto_log_invalid(skb, state, IPPROTO_UDPLITE, "%s", msg);
}
static bool udplite_error(struct sk_buff *skb,
for (i = 0; i < UDP_CT_MAX; i++)
un->timeouts[i] = udp_timeouts[i];
+
+#if IS_ENABLED(CONFIG_NF_FLOW_TABLE)
+ un->offload_timeout = 30 * HZ;
+ un->offload_pickup = 30 * HZ;
+#endif
}
const struct nf_conntrack_l4proto nf_conntrack_l4proto_udp =
u32 nf_conntrack_count(const struct net *net)
{
- const struct nf_conntrack_net *cnet;
-
- cnet = net_generic(net, nf_conntrack_net_id);
+ const struct nf_conntrack_net *cnet = nf_ct_pernet(net);
return atomic_read(&cnet->count);
}
NF_SYSCTL_CT_PROTO_TIMEOUT_TCP_CLOSE,
NF_SYSCTL_CT_PROTO_TIMEOUT_TCP_RETRANS,
NF_SYSCTL_CT_PROTO_TIMEOUT_TCP_UNACK,
+#if IS_ENABLED(CONFIG_NF_FLOW_TABLE)
+ NF_SYSCTL_CT_PROTO_TIMEOUT_TCP_OFFLOAD,
+ NF_SYSCTL_CT_PROTO_TIMEOUT_TCP_OFFLOAD_PICKUP,
+#endif
NF_SYSCTL_CT_PROTO_TCP_LOOSE,
NF_SYSCTL_CT_PROTO_TCP_LIBERAL,
NF_SYSCTL_CT_PROTO_TCP_MAX_RETRANS,
NF_SYSCTL_CT_PROTO_TIMEOUT_UDP,
NF_SYSCTL_CT_PROTO_TIMEOUT_UDP_STREAM,
+#if IS_ENABLED(CONFIG_NF_FLOW_TABLE)
+ NF_SYSCTL_CT_PROTO_TIMEOUT_UDP_OFFLOAD,
+ NF_SYSCTL_CT_PROTO_TIMEOUT_UDP_OFFLOAD_PICKUP,
+#endif
NF_SYSCTL_CT_PROTO_TIMEOUT_ICMP,
NF_SYSCTL_CT_PROTO_TIMEOUT_ICMPV6,
#ifdef CONFIG_NF_CT_PROTO_SCTP
.mode = 0644,
.proc_handler = proc_dointvec_jiffies,
},
+#if IS_ENABLED(CONFIG_NF_FLOW_TABLE)
+ [NF_SYSCTL_CT_PROTO_TIMEOUT_TCP_OFFLOAD] = {
+ .procname = "nf_flowtable_tcp_timeout",
+ .maxlen = sizeof(unsigned int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_jiffies,
+ },
+ [NF_SYSCTL_CT_PROTO_TIMEOUT_TCP_OFFLOAD_PICKUP] = {
+ .procname = "nf_flowtable_tcp_pickup",
+ .maxlen = sizeof(unsigned int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_jiffies,
+ },
+#endif
[NF_SYSCTL_CT_PROTO_TCP_LOOSE] = {
.procname = "nf_conntrack_tcp_loose",
.maxlen = sizeof(u8),
.mode = 0644,
.proc_handler = proc_dointvec_jiffies,
},
+#if IS_ENABLED(CONFIG_NFT_FLOW_OFFLOAD)
+ [NF_SYSCTL_CT_PROTO_TIMEOUT_UDP_OFFLOAD] = {
+ .procname = "nf_flowtable_udp_timeout",
+ .maxlen = sizeof(unsigned int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_jiffies,
+ },
+ [NF_SYSCTL_CT_PROTO_TIMEOUT_UDP_OFFLOAD_PICKUP] = {
+ .procname = "nf_flowtable_udp_pickup",
+ .maxlen = sizeof(unsigned int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_jiffies,
+ },
+#endif
[NF_SYSCTL_CT_PROTO_TIMEOUT_ICMP] = {
.procname = "nf_conntrack_icmp_timeout",
.maxlen = sizeof(unsigned int),
XASSIGN(LIBERAL, &tn->tcp_be_liberal);
XASSIGN(MAX_RETRANS, &tn->tcp_max_retrans);
#undef XASSIGN
+
+#if IS_ENABLED(CONFIG_NF_FLOW_TABLE)
+ table[NF_SYSCTL_CT_PROTO_TIMEOUT_TCP_OFFLOAD].data = &tn->offload_timeout;
+ table[NF_SYSCTL_CT_PROTO_TIMEOUT_TCP_OFFLOAD_PICKUP].data = &tn->offload_pickup;
+#endif
+
}
static void nf_conntrack_standalone_init_sctp_sysctl(struct net *net,
static int nf_conntrack_standalone_init_sysctl(struct net *net)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
struct nf_udp_net *un = nf_udp_pernet(net);
struct ctl_table *table;
table[NF_SYSCTL_CT_PROTO_TIMEOUT_ICMPV6].data = &nf_icmpv6_pernet(net)->timeout;
table[NF_SYSCTL_CT_PROTO_TIMEOUT_UDP].data = &un->timeouts[UDP_CT_UNREPLIED];
table[NF_SYSCTL_CT_PROTO_TIMEOUT_UDP_STREAM].data = &un->timeouts[UDP_CT_REPLIED];
+#if IS_ENABLED(CONFIG_NF_FLOW_TABLE)
+ table[NF_SYSCTL_CT_PROTO_TIMEOUT_UDP_OFFLOAD].data = &un->offload_timeout;
+ table[NF_SYSCTL_CT_PROTO_TIMEOUT_UDP_OFFLOAD_PICKUP].data = &un->offload_pickup;
+#endif
nf_conntrack_standalone_init_tcp_sysctl(net, table);
nf_conntrack_standalone_init_sctp_sysctl(net, table);
static void nf_conntrack_standalone_fini_sysctl(struct net *net)
{
- struct nf_conntrack_net *cnet = net_generic(net, nf_conntrack_net_id);
+ struct nf_conntrack_net *cnet = nf_ct_pernet(net);
struct ctl_table *table;
table = cnet->sysctl_header->ctl_table_arg;
tcp->seen[1].td_maxwin = 0;
}
-#define NF_FLOWTABLE_TCP_PICKUP_TIMEOUT (120 * HZ)
-#define NF_FLOWTABLE_UDP_PICKUP_TIMEOUT (30 * HZ)
-
static void flow_offload_fixup_ct_timeout(struct nf_conn *ct)
{
const struct nf_conntrack_l4proto *l4proto;
+ struct net *net = nf_ct_net(ct);
int l4num = nf_ct_protonum(ct);
unsigned int timeout;
if (!l4proto)
return;
- if (l4num == IPPROTO_TCP)
- timeout = NF_FLOWTABLE_TCP_PICKUP_TIMEOUT;
- else if (l4num == IPPROTO_UDP)
- timeout = NF_FLOWTABLE_UDP_PICKUP_TIMEOUT;
- else
+ if (l4num == IPPROTO_TCP) {
+ struct nf_tcp_net *tn = nf_tcp_pernet(net);
+
+ timeout = tn->offload_pickup;
+ } else if (l4num == IPPROTO_UDP) {
+ struct nf_udp_net *tn = nf_udp_pernet(net);
+
+ timeout = tn->offload_pickup;
+ } else {
return;
+ }
if (nf_flow_timeout_delta(ct->timeout) > (__s32)timeout)
ct->timeout = nfct_time_stamp + timeout;
.automatic_shrinking = true,
};
+unsigned long flow_offload_get_timeout(struct flow_offload *flow)
+{
+ const struct nf_conntrack_l4proto *l4proto;
+ unsigned long timeout = NF_FLOW_TIMEOUT;
+ struct net *net = nf_ct_net(flow->ct);
+ int l4num = nf_ct_protonum(flow->ct);
+
+ l4proto = nf_ct_l4proto_find(l4num);
+ if (!l4proto)
+ return timeout;
+
+ if (l4num == IPPROTO_TCP) {
+ struct nf_tcp_net *tn = nf_tcp_pernet(net);
+
+ timeout = tn->offload_timeout;
+ } else if (l4num == IPPROTO_UDP) {
+ struct nf_udp_net *tn = nf_udp_pernet(net);
+
+ timeout = tn->offload_timeout;
+ }
+
+ return timeout;
+}
+
int flow_offload_add(struct nf_flowtable *flow_table, struct flow_offload *flow)
{
int err;
- flow->timeout = nf_flowtable_time_stamp + NF_FLOW_TIMEOUT;
+ flow->timeout = nf_flowtable_time_stamp + flow_offload_get_timeout(flow);
err = rhashtable_insert_fast(&flow_table->rhashtable,
&flow->tuplehash[0].node,
void flow_offload_refresh(struct nf_flowtable *flow_table,
struct flow_offload *flow)
{
- flow->timeout = nf_flowtable_time_stamp + NF_FLOW_TIMEOUT;
+ flow->timeout = nf_flowtable_time_stamp + flow_offload_get_timeout(flow);
- if (likely(!nf_flowtable_hw_offload(flow_table) ||
- !test_and_clear_bit(NF_FLOW_HW_REFRESH, &flow->flags)))
+ if (likely(!nf_flowtable_hw_offload(flow_table)))
return;
nf_flow_offload_add(flow_table, flow);
err = flow_offload_rule_add(offload, flow_rule);
if (err < 0)
- set_bit(NF_FLOW_HW_REFRESH, &offload->flow->flags);
- else
- set_bit(IPS_HW_OFFLOAD_BIT, &offload->flow->ct->status);
+ goto out;
+
+ set_bit(IPS_HW_OFFLOAD_BIT, &offload->flow->ct->status);
+out:
nf_flow_offload_destroy(flow_rule);
}
lastused = max_t(u64, stats[0].lastused, stats[1].lastused);
offload->flow->timeout = max_t(u64, offload->flow->timeout,
- lastused + NF_FLOW_TIMEOUT);
+ lastused + flow_offload_get_timeout(offload->flow));
if (offload->flowtable->flags & NF_FLOWTABLE_COUNTER) {
if (stats[0].pkts)
__s32 delta;
delta = nf_flow_timeout_delta(flow->timeout);
- if ((delta >= (9 * NF_FLOW_TIMEOUT) / 10))
+ if ((delta >= (9 * flow_offload_get_timeout(flow)) / 10))
return;
offload = nf_flow_offload_work_alloc(flowtable, flow, FLOW_CLS_STATS);
int length = (th->doff * 4) - sizeof(*th);
u8 buf[40], *ptr;
+ if (unlikely(length < 0))
+ return false;
+
ptr = skb_header_pointer(skb, doff + sizeof(*th), length, buf);
if (ptr == NULL)
return false;
length--;
continue;
default:
+ if (length < 2)
+ return true;
opsize = *ptr++;
if (opsize < 2)
return true;
goto nla_put_failure;
if (nla_put_string(skb, NFTA_TABLE_NAME, table->name) ||
- nla_put_be32(skb, NFTA_TABLE_FLAGS, htonl(table->flags)) ||
+ nla_put_be32(skb, NFTA_TABLE_FLAGS,
+ htonl(table->flags & NFT_TABLE_F_MASK)) ||
nla_put_be32(skb, NFTA_TABLE_USE, htonl(table->use)) ||
nla_put_be64(skb, NFTA_TABLE_HANDLE, cpu_to_be64(table->handle),
NFTA_TABLE_PAD))
static int nf_tables_gettable(struct sk_buff *skb, const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_cur(info->net);
- int family = nfmsg->nfgen_family;
+ u8 family = info->nfmsg->nfgen_family;
const struct nft_table *table;
struct net *net = info->net;
struct sk_buff *skb2;
static void nf_tables_table_disable(struct net *net, struct nft_table *table)
{
+ table->flags &= ~NFT_TABLE_F_DORMANT;
nft_table_disable(net, table, 0);
+ table->flags |= NFT_TABLE_F_DORMANT;
}
-enum {
- NFT_TABLE_STATE_UNCHANGED = 0,
- NFT_TABLE_STATE_DORMANT,
- NFT_TABLE_STATE_WAKEUP
-};
+#define __NFT_TABLE_F_INTERNAL (NFT_TABLE_F_MASK + 1)
+#define __NFT_TABLE_F_WAS_DORMANT (__NFT_TABLE_F_INTERNAL << 0)
+#define __NFT_TABLE_F_WAS_AWAKEN (__NFT_TABLE_F_INTERNAL << 1)
+#define __NFT_TABLE_F_UPDATE (__NFT_TABLE_F_WAS_DORMANT | \
+ __NFT_TABLE_F_WAS_AWAKEN)
static int nf_tables_updtable(struct nft_ctx *ctx)
{
struct nft_trans *trans;
u32 flags;
- int ret = 0;
+ int ret;
if (!ctx->nla[NFTA_TABLE_FLAGS])
return 0;
if ((flags & NFT_TABLE_F_DORMANT) &&
!(ctx->table->flags & NFT_TABLE_F_DORMANT)) {
- nft_trans_table_state(trans) = NFT_TABLE_STATE_DORMANT;
+ ctx->table->flags |= NFT_TABLE_F_DORMANT;
+ if (!(ctx->table->flags & __NFT_TABLE_F_UPDATE))
+ ctx->table->flags |= __NFT_TABLE_F_WAS_AWAKEN;
} else if (!(flags & NFT_TABLE_F_DORMANT) &&
ctx->table->flags & NFT_TABLE_F_DORMANT) {
- ret = nf_tables_table_enable(ctx->net, ctx->table);
- if (ret >= 0)
- nft_trans_table_state(trans) = NFT_TABLE_STATE_WAKEUP;
+ ctx->table->flags &= ~NFT_TABLE_F_DORMANT;
+ if (!(ctx->table->flags & __NFT_TABLE_F_UPDATE)) {
+ ret = nf_tables_table_enable(ctx->net, ctx->table);
+ if (ret < 0)
+ goto err_register_hooks;
+
+ ctx->table->flags |= __NFT_TABLE_F_WAS_DORMANT;
+ }
}
- if (ret < 0)
- goto err;
- nft_trans_table_flags(trans) = flags;
nft_trans_table_update(trans) = true;
nft_trans_commit_list_add_tail(ctx->net, trans);
+
return 0;
-err:
+
+err_register_hooks:
nft_trans_destroy(trans);
return ret;
}
const struct nlattr * const nla[])
{
struct nftables_pernet *nft_net = nft_pernet(info->net);
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_next(info->net);
- int family = nfmsg->nfgen_family;
+ u8 family = info->nfmsg->nfgen_family;
struct net *net = info->net;
const struct nlattr *attr;
struct nft_table *table;
static int nf_tables_deltable(struct sk_buff *skb, const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_next(info->net);
- int family = nfmsg->nfgen_family;
+ u8 family = info->nfmsg->nfgen_family;
struct net *net = info->net;
const struct nlattr *attr;
struct nft_table *table;
static int nf_tables_getchain(struct sk_buff *skb, const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_cur(info->net);
- int family = nfmsg->nfgen_family;
+ u8 family = info->nfmsg->nfgen_family;
const struct nft_chain *chain;
struct net *net = info->net;
struct nft_table *table;
static int nft_chain_parse_hook(struct net *net,
const struct nlattr * const nla[],
struct nft_chain_hook *hook, u8 family,
- bool autoload)
+ struct netlink_ext_ack *extack, bool autoload)
{
struct nftables_pernet *nft_net = nft_pernet(net);
struct nlattr *ha[NFTA_HOOK_MAX + 1];
if (nla[NFTA_CHAIN_TYPE]) {
type = nf_tables_chain_type_lookup(net, nla[NFTA_CHAIN_TYPE],
family, autoload);
- if (IS_ERR(type))
+ if (IS_ERR(type)) {
+ NL_SET_BAD_ATTR(extack, nla[NFTA_CHAIN_TYPE]);
return PTR_ERR(type);
+ }
}
if (hook->num >= NFT_MAX_HOOKS || !(type->hook_mask & (1 << hook->num)))
return -EOPNOTSUPP;
hook->priority <= NF_IP_PRI_CONNTRACK)
return -EOPNOTSUPP;
- if (!try_module_get(type->owner))
+ if (!try_module_get(type->owner)) {
+ if (nla[NFTA_CHAIN_TYPE])
+ NL_SET_BAD_ATTR(extack, nla[NFTA_CHAIN_TYPE]);
return -ENOENT;
+ }
hook->type = type;
const struct nft_chain_hook *hook,
struct nft_chain *chain)
{
- ops->pf = family;
- ops->hooknum = hook->num;
- ops->priority = hook->priority;
- ops->priv = chain;
- ops->hook = hook->type->hooks[ops->hooknum];
+ ops->pf = family;
+ ops->hooknum = hook->num;
+ ops->priority = hook->priority;
+ ops->priv = chain;
+ ops->hook = hook->type->hooks[ops->hooknum];
+ ops->hook_ops_type = NF_HOOK_OP_NF_TABLES;
}
static int nft_basechain_init(struct nft_base_chain *basechain, u8 family,
static u64 chain_id;
static int nf_tables_addchain(struct nft_ctx *ctx, u8 family, u8 genmask,
- u8 policy, u32 flags)
+ u8 policy, u32 flags,
+ struct netlink_ext_ack *extack)
{
const struct nlattr * const *nla = ctx->nla;
struct nft_table *table = ctx->table;
if (flags & NFT_CHAIN_BINDING)
return -EOPNOTSUPP;
- err = nft_chain_parse_hook(net, nla, &hook, family, true);
+ err = nft_chain_parse_hook(net, nla, &hook, family, extack,
+ true);
if (err < 0)
return err;
return -EEXIST;
}
err = nft_chain_parse_hook(ctx->net, nla, &hook, ctx->family,
- false);
+ extack, false);
if (err < 0)
return err;
const struct nlattr * const nla[])
{
struct nftables_pernet *nft_net = nft_pernet(info->net);
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_next(info->net);
- int family = nfmsg->nfgen_family;
+ u8 family = info->nfmsg->nfgen_family;
struct nft_chain *chain = NULL;
struct net *net = info->net;
const struct nlattr *attr;
extack);
}
- return nf_tables_addchain(&ctx, family, genmask, policy, flags);
+ return nf_tables_addchain(&ctx, family, genmask, policy, flags, extack);
}
static int nf_tables_delchain(struct sk_buff *skb, const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_next(info->net);
- int family = nfmsg->nfgen_family;
+ u8 family = info->nfmsg->nfgen_family;
struct net *net = info->net;
const struct nlattr *attr;
struct nft_table *table;
static int nf_tables_getrule(struct sk_buff *skb, const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_cur(info->net);
- int family = nfmsg->nfgen_family;
+ u8 family = info->nfmsg->nfgen_family;
const struct nft_chain *chain;
const struct nft_rule *rule;
struct net *net = info->net;
const struct nlattr * const nla[])
{
struct nftables_pernet *nft_net = nft_pernet(info->net);
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
unsigned int size, i, n, ulen = 0, usize = 0;
u8 genmask = nft_genmask_next(info->net);
struct nft_rule *rule, *old_rule = NULL;
struct nft_expr_info *expr_info = NULL;
- int family = nfmsg->nfgen_family;
+ u8 family = info->nfmsg->nfgen_family;
struct net *net = info->net;
struct nft_flow_rule *flow;
struct nft_userdata *udata;
if (n == NFT_RULE_MAXEXPRS)
goto err1;
err = nf_tables_expr_parse(&ctx, tmp, &expr_info[n]);
- if (err < 0)
+ if (err < 0) {
+ NL_SET_BAD_ATTR(extack, tmp);
goto err1;
+ }
size += expr_info[n].ops->size;
n++;
}
static int nf_tables_delrule(struct sk_buff *skb, const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
- int family = nfmsg->nfgen_family, err = 0;
u8 genmask = nft_genmask_next(info->net);
+ u8 family = info->nfmsg->nfgen_family;
struct nft_chain *chain = NULL;
struct net *net = info->net;
struct nft_table *table;
struct nft_rule *rule;
struct nft_ctx ctx;
+ int err = 0;
table = nft_table_lookup(net, nla[NFTA_RULE_TABLE], family, genmask,
NETLINK_CB(skb).portid);
[NFTA_SET_DESC_CONCAT] = { .type = NLA_NESTED },
};
-static int nft_ctx_init_from_setattr(struct nft_ctx *ctx, struct net *net,
- const struct sk_buff *skb,
- const struct nlmsghdr *nlh,
- const struct nlattr * const nla[],
- struct netlink_ext_ack *extack,
- u8 genmask, u32 nlpid)
-{
- const struct nfgenmsg *nfmsg = nlmsg_data(nlh);
- int family = nfmsg->nfgen_family;
- struct nft_table *table = NULL;
-
- if (nla[NFTA_SET_TABLE] != NULL) {
- table = nft_table_lookup(net, nla[NFTA_SET_TABLE], family,
- genmask, nlpid);
- if (IS_ERR(table)) {
- NL_SET_BAD_ATTR(extack, nla[NFTA_SET_TABLE]);
- return PTR_ERR(table);
- }
- }
-
- nft_ctx_init(ctx, net, skb, nlh, family, table, NULL, nla);
- return 0;
-}
-
static struct nft_set *nft_set_lookup(const struct nft_table *table,
const struct nlattr *nla, u8 genmask)
{
static int nf_tables_getset(struct sk_buff *skb, const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_cur(info->net);
+ u8 family = info->nfmsg->nfgen_family;
+ struct nft_table *table = NULL;
struct net *net = info->net;
const struct nft_set *set;
struct sk_buff *skb2;
struct nft_ctx ctx;
int err;
- /* Verify existence before starting dump */
- err = nft_ctx_init_from_setattr(&ctx, net, skb, info->nlh, nla, extack,
- genmask, 0);
- if (err < 0)
- return err;
+ if (nla[NFTA_SET_TABLE]) {
+ table = nft_table_lookup(net, nla[NFTA_SET_TABLE], family,
+ genmask, 0);
+ if (IS_ERR(table)) {
+ NL_SET_BAD_ATTR(extack, nla[NFTA_SET_TABLE]);
+ return PTR_ERR(table);
+ }
+ }
+
+ nft_ctx_init(&ctx, net, skb, info->nlh, family, table, NULL, nla);
if (info->nlh->nlmsg_flags & NLM_F_DUMP) {
struct netlink_dump_control c = {
}
/* Only accept unspec with dump */
- if (nfmsg->nfgen_family == NFPROTO_UNSPEC)
+ if (info->nfmsg->nfgen_family == NFPROTO_UNSPEC)
return -EAFNOSUPPORT;
if (!nla[NFTA_SET_TABLE])
return -EINVAL;
- set = nft_set_lookup(ctx.table, nla[NFTA_SET_NAME], genmask);
+ set = nft_set_lookup(table, nla[NFTA_SET_NAME], genmask);
if (IS_ERR(set))
return PTR_ERR(set);
static int nf_tables_newset(struct sk_buff *skb, const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
u32 ktype, dtype, flags, policy, gc_int, objtype;
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_next(info->net);
- int family = nfmsg->nfgen_family;
+ u8 family = info->nfmsg->nfgen_family;
const struct nft_set_ops *ops;
struct nft_expr *expr = NULL;
struct net *net = info->net;
err = nf_tables_set_alloc_name(&ctx, set, name);
kfree(name);
if (err < 0)
- goto err_set_alloc_name;
+ goto err_set_name;
+
+ udata = NULL;
+ if (udlen) {
+ udata = set->data + size;
+ nla_memcpy(udata, nla[NFTA_SET_USERDATA], udlen);
+ }
+
+ INIT_LIST_HEAD(&set->bindings);
+ INIT_LIST_HEAD(&set->catchall_list);
+ set->table = table;
+ write_pnet(&set->net, net);
+ set->ops = ops;
+ set->ktype = ktype;
+ set->klen = desc.klen;
+ set->dtype = dtype;
+ set->objtype = objtype;
+ set->dlen = desc.dlen;
+ set->flags = flags;
+ set->size = desc.size;
+ set->policy = policy;
+ set->udlen = udlen;
+ set->udata = udata;
+ set->timeout = timeout;
+ set->gc_int = gc_int;
+
+ set->field_count = desc.field_count;
+ for (i = 0; i < desc.field_count; i++)
+ set->field_len[i] = desc.field_len[i];
+
+ err = ops->init(set, &desc, nla);
+ if (err < 0)
+ goto err_set_init;
if (nla[NFTA_SET_EXPR]) {
expr = nft_set_elem_expr_alloc(&ctx, set, nla[NFTA_SET_EXPR]);
if (IS_ERR(expr)) {
err = PTR_ERR(expr);
- goto err_set_alloc_name;
+ goto err_set_expr_alloc;
}
set->exprs[0] = expr;
set->num_exprs++;
if (!(flags & NFT_SET_EXPR)) {
err = -EINVAL;
- goto err_set_alloc_name;
+ goto err_set_expr_alloc;
}
i = 0;
nla_for_each_nested(tmp, nla[NFTA_SET_EXPRESSIONS], left) {
if (i == NFT_SET_EXPR_MAX) {
err = -E2BIG;
- goto err_set_init;
+ goto err_set_expr_alloc;
}
if (nla_type(tmp) != NFTA_LIST_ELEM) {
err = -EINVAL;
- goto err_set_init;
+ goto err_set_expr_alloc;
}
expr = nft_set_elem_expr_alloc(&ctx, set, tmp);
if (IS_ERR(expr)) {
err = PTR_ERR(expr);
- goto err_set_init;
+ goto err_set_expr_alloc;
}
set->exprs[i++] = expr;
set->num_exprs++;
}
}
- udata = NULL;
- if (udlen) {
- udata = set->data + size;
- nla_memcpy(udata, nla[NFTA_SET_USERDATA], udlen);
- }
-
- INIT_LIST_HEAD(&set->bindings);
- INIT_LIST_HEAD(&set->catchall_list);
- set->table = table;
- write_pnet(&set->net, net);
- set->ops = ops;
- set->ktype = ktype;
- set->klen = desc.klen;
- set->dtype = dtype;
- set->objtype = objtype;
- set->dlen = desc.dlen;
- set->flags = flags;
- set->size = desc.size;
- set->policy = policy;
- set->udlen = udlen;
- set->udata = udata;
- set->timeout = timeout;
- set->gc_int = gc_int;
set->handle = nf_tables_alloc_handle(table);
- set->field_count = desc.field_count;
- for (i = 0; i < desc.field_count; i++)
- set->field_len[i] = desc.field_len[i];
-
- err = ops->init(set, &desc, nla);
- if (err < 0)
- goto err_set_init;
-
err = nft_trans_set_add(&ctx, NFT_MSG_NEWSET, set);
if (err < 0)
- goto err_set_trans;
+ goto err_set_expr_alloc;
list_add_tail_rcu(&set->list, &table->sets);
table->use++;
return 0;
-err_set_trans:
- ops->destroy(set);
-err_set_init:
+err_set_expr_alloc:
for (i = 0; i < set->num_exprs; i++)
nft_expr_destroy(&ctx, set->exprs[i]);
-err_set_alloc_name:
+
+ ops->destroy(set);
+err_set_init:
kfree(set->name);
err_set_name:
kvfree(set);
static int nf_tables_delset(struct sk_buff *skb, const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_next(info->net);
+ u8 family = info->nfmsg->nfgen_family;
struct net *net = info->net;
const struct nlattr *attr;
+ struct nft_table *table;
struct nft_set *set;
struct nft_ctx ctx;
- int err;
- if (nfmsg->nfgen_family == NFPROTO_UNSPEC)
+ if (info->nfmsg->nfgen_family == NFPROTO_UNSPEC)
return -EAFNOSUPPORT;
- if (nla[NFTA_SET_TABLE] == NULL)
- return -EINVAL;
- err = nft_ctx_init_from_setattr(&ctx, net, skb, info->nlh, nla, extack,
- genmask, NETLINK_CB(skb).portid);
- if (err < 0)
- return err;
+ table = nft_table_lookup(net, nla[NFTA_SET_TABLE], family,
+ genmask, NETLINK_CB(skb).portid);
+ if (IS_ERR(table)) {
+ NL_SET_BAD_ATTR(extack, nla[NFTA_SET_TABLE]);
+ return PTR_ERR(table);
+ }
if (nla[NFTA_SET_HANDLE]) {
attr = nla[NFTA_SET_HANDLE];
- set = nft_set_lookup_byhandle(ctx.table, attr, genmask);
+ set = nft_set_lookup_byhandle(table, attr, genmask);
} else {
attr = nla[NFTA_SET_NAME];
- set = nft_set_lookup(ctx.table, attr, genmask);
+ set = nft_set_lookup(table, attr, genmask);
}
if (IS_ERR(set)) {
return -EBUSY;
}
+ nft_ctx_init(&ctx, net, skb, info->nlh, family, table, NULL, nla);
+
return nft_delset(&ctx, set);
}
[NFTA_SET_ELEM_LIST_SET_ID] = { .type = NLA_U32 },
};
-static int nft_ctx_init_from_elemattr(struct nft_ctx *ctx, struct net *net,
- const struct sk_buff *skb,
- const struct nlmsghdr *nlh,
- const struct nlattr * const nla[],
- struct netlink_ext_ack *extack,
- u8 genmask, u32 nlpid)
-{
- const struct nfgenmsg *nfmsg = nlmsg_data(nlh);
- int family = nfmsg->nfgen_family;
- struct nft_table *table;
-
- table = nft_table_lookup(net, nla[NFTA_SET_ELEM_LIST_TABLE], family,
- genmask, nlpid);
- if (IS_ERR(table)) {
- NL_SET_BAD_ATTR(extack, nla[NFTA_SET_ELEM_LIST_TABLE]);
- return PTR_ERR(table);
- }
-
- nft_ctx_init(ctx, net, skb, nlh, family, table, NULL, nla);
- return 0;
-}
-
static int nft_set_elem_expr_dump(struct sk_buff *skb,
const struct nft_set *set,
const struct nft_set_ext *ext)
{
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_cur(info->net);
+ u8 family = info->nfmsg->nfgen_family;
struct net *net = info->net;
+ struct nft_table *table;
struct nft_set *set;
struct nlattr *attr;
struct nft_ctx ctx;
int rem, err = 0;
- err = nft_ctx_init_from_elemattr(&ctx, net, skb, info->nlh, nla, extack,
- genmask, NETLINK_CB(skb).portid);
- if (err < 0)
- return err;
+ table = nft_table_lookup(net, nla[NFTA_SET_ELEM_LIST_TABLE], family,
+ genmask, NETLINK_CB(skb).portid);
+ if (IS_ERR(table)) {
+ NL_SET_BAD_ATTR(extack, nla[NFTA_SET_ELEM_LIST_TABLE]);
+ return PTR_ERR(table);
+ }
- set = nft_set_lookup(ctx.table, nla[NFTA_SET_ELEM_LIST_SET], genmask);
+ set = nft_set_lookup(table, nla[NFTA_SET_ELEM_LIST_SET], genmask);
if (IS_ERR(set))
return PTR_ERR(set);
+ nft_ctx_init(&ctx, net, skb, info->nlh, family, table, NULL, nla);
+
if (info->nlh->nlmsg_flags & NLM_F_DUMP) {
struct netlink_dump_control c = {
.start = nf_tables_dump_set_start,
struct nftables_pernet *nft_net = nft_pernet(info->net);
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_next(info->net);
+ u8 family = info->nfmsg->nfgen_family;
struct net *net = info->net;
const struct nlattr *attr;
+ struct nft_table *table;
struct nft_set *set;
struct nft_ctx ctx;
int rem, err;
if (nla[NFTA_SET_ELEM_LIST_ELEMENTS] == NULL)
return -EINVAL;
- err = nft_ctx_init_from_elemattr(&ctx, net, skb, info->nlh, nla, extack,
- genmask, NETLINK_CB(skb).portid);
- if (err < 0)
- return err;
+ table = nft_table_lookup(net, nla[NFTA_SET_ELEM_LIST_TABLE], family,
+ genmask, NETLINK_CB(skb).portid);
+ if (IS_ERR(table)) {
+ NL_SET_BAD_ATTR(extack, nla[NFTA_SET_ELEM_LIST_TABLE]);
+ return PTR_ERR(table);
+ }
- set = nft_set_lookup_global(net, ctx.table, nla[NFTA_SET_ELEM_LIST_SET],
+ set = nft_set_lookup_global(net, table, nla[NFTA_SET_ELEM_LIST_SET],
nla[NFTA_SET_ELEM_LIST_SET_ID], genmask);
if (IS_ERR(set))
return PTR_ERR(set);
if (!list_empty(&set->bindings) && set->flags & NFT_SET_CONSTANT)
return -EBUSY;
+ nft_ctx_init(&ctx, net, skb, info->nlh, family, table, NULL, nla);
+
nla_for_each_nested(attr, nla[NFTA_SET_ELEM_LIST_ELEMENTS], rem) {
err = nft_add_set_elem(&ctx, set, attr, info->nlh->nlmsg_flags);
if (err < 0)
}
if (nft_net->validate_state == NFT_VALIDATE_DO)
- return nft_table_validate(net, ctx.table);
+ return nft_table_validate(net, table);
return 0;
}
{
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_next(info->net);
+ u8 family = info->nfmsg->nfgen_family;
struct net *net = info->net;
const struct nlattr *attr;
+ struct nft_table *table;
struct nft_set *set;
struct nft_ctx ctx;
int rem, err = 0;
- err = nft_ctx_init_from_elemattr(&ctx, net, skb, info->nlh, nla, extack,
- genmask, NETLINK_CB(skb).portid);
- if (err < 0)
- return err;
+ table = nft_table_lookup(net, nla[NFTA_SET_ELEM_LIST_TABLE], family,
+ genmask, NETLINK_CB(skb).portid);
+ if (IS_ERR(table)) {
+ NL_SET_BAD_ATTR(extack, nla[NFTA_SET_ELEM_LIST_TABLE]);
+ return PTR_ERR(table);
+ }
- set = nft_set_lookup(ctx.table, nla[NFTA_SET_ELEM_LIST_SET], genmask);
+ set = nft_set_lookup(table, nla[NFTA_SET_ELEM_LIST_SET], genmask);
if (IS_ERR(set))
return PTR_ERR(set);
if (!list_empty(&set->bindings) && set->flags & NFT_SET_CONSTANT)
return -EBUSY;
+ nft_ctx_init(&ctx, net, skb, info->nlh, family, table, NULL, nla);
+
if (!nla[NFTA_SET_ELEM_LIST_ELEMENTS])
return nft_set_flush(&ctx, set, genmask);
static int nf_tables_newobj(struct sk_buff *skb, const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_next(info->net);
+ u8 family = info->nfmsg->nfgen_family;
const struct nft_object_type *type;
- int family = nfmsg->nfgen_family;
struct net *net = info->net;
struct nft_table *table;
struct nft_object *obj;
static int nf_tables_getobj(struct sk_buff *skb, const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_cur(info->net);
- int family = nfmsg->nfgen_family;
+ u8 family = info->nfmsg->nfgen_family;
const struct nft_table *table;
struct net *net = info->net;
struct nft_object *obj;
static int nf_tables_delobj(struct sk_buff *skb, const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_next(info->net);
- int family = nfmsg->nfgen_family;
+ u8 family = info->nfmsg->nfgen_family;
struct net *net = info->net;
const struct nlattr *attr;
struct nft_table *table;
const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
struct nft_flowtable_hook flowtable_hook;
u8 genmask = nft_genmask_next(info->net);
+ u8 family = info->nfmsg->nfgen_family;
const struct nf_flowtable_type *type;
- int family = nfmsg->nfgen_family;
struct nft_flowtable *flowtable;
struct nft_hook *hook, *next;
struct net *net = info->net;
const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
struct netlink_ext_ack *extack = info->extack;
u8 genmask = nft_genmask_next(info->net);
- int family = nfmsg->nfgen_family;
+ u8 family = info->nfmsg->nfgen_family;
struct nft_flowtable *flowtable;
struct net *net = info->net;
const struct nlattr *attr;
const struct nfnl_info *info,
const struct nlattr * const nla[])
{
- const struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
u8 genmask = nft_genmask_cur(info->net);
- int family = nfmsg->nfgen_family;
+ u8 family = info->nfmsg->nfgen_family;
struct nft_flowtable *flowtable;
const struct nft_table *table;
struct net *net = info->net;
switch (trans->msg_type) {
case NFT_MSG_NEWTABLE:
if (nft_trans_table_update(trans)) {
- if (nft_trans_table_state(trans) == NFT_TABLE_STATE_DORMANT)
+ if (!(trans->ctx.table->flags & __NFT_TABLE_F_UPDATE)) {
+ nft_trans_destroy(trans);
+ break;
+ }
+ if (trans->ctx.table->flags & NFT_TABLE_F_DORMANT)
nf_tables_table_disable(net, trans->ctx.table);
- trans->ctx.table->flags = nft_trans_table_flags(trans);
+ trans->ctx.table->flags &= ~__NFT_TABLE_F_UPDATE;
} else {
nft_clear(net, trans->ctx.table);
}
switch (trans->msg_type) {
case NFT_MSG_NEWTABLE:
if (nft_trans_table_update(trans)) {
- if (nft_trans_table_state(trans) == NFT_TABLE_STATE_WAKEUP)
+ if (!(trans->ctx.table->flags & __NFT_TABLE_F_UPDATE)) {
+ nft_trans_destroy(trans);
+ break;
+ }
+ if (trans->ctx.table->flags & __NFT_TABLE_F_WAS_DORMANT) {
nf_tables_table_disable(net, trans->ctx.table);
-
+ trans->ctx.table->flags |= NFT_TABLE_F_DORMANT;
+ } else if (trans->ctx.table->flags & __NFT_TABLE_F_WAS_AWAKEN) {
+ trans->ctx.table->flags &= ~NFT_TABLE_F_DORMANT;
+ }
+ trans->ctx.table->flags &= ~__NFT_TABLE_F_UPDATE;
nft_trans_destroy(trans);
} else {
list_del_rcu(&trans->ctx.table->list);
else {
if (!pkt->tprot_set)
return false;
- ptr = skb_network_header(skb) + pkt->xt.thoff;
+ ptr = skb_network_header(skb) + nft_thoff(pkt);
}
ptr += priv->offset;
&nft_meta_type,
&nft_rt_type,
&nft_exthdr_type,
+ &nft_last_type,
};
static struct nft_object_type *nft_basic_objects[] = {
int off = skb_network_offset(skb);
unsigned int len, nh_end;
- nh_end = pkt->tprot_set ? pkt->xt.thoff : skb->len;
+ nh_end = pkt->tprot_set ? nft_thoff(pkt) : skb->len;
len = min_t(unsigned int, nh_end - skb_network_offset(skb),
NFT_TRACETYPE_NETWORK_HSIZE);
if (trace_fill_header(nlskb, NFTA_TRACE_NETWORK_HEADER, skb, off, len))
return -1;
if (pkt->tprot_set) {
- len = min_t(unsigned int, skb->len - pkt->xt.thoff,
+ len = min_t(unsigned int, skb->len - nft_thoff(pkt),
NFT_TRACETYPE_TRANSPORT_HSIZE);
if (trace_fill_header(nlskb, NFTA_TRACE_TRANSPORT_HEADER, skb,
- pkt->xt.thoff, len))
+ nft_thoff(pkt), len))
return -1;
}
[NFNL_SUBSYS_CTHELPER] = "nfnl_subsys_cthelper",
[NFNL_SUBSYS_NFTABLES] = "nfnl_subsys_nftables",
[NFNL_SUBSYS_NFT_COMPAT] = "nfnl_subsys_nftcompat",
+ [NFNL_SUBSYS_HOOK] = "nfnl_subsys_hook",
};
static const int nfnl_group2type[NFNLGRP_MAX+1] = {
.net = net,
.sk = nfnlnet->nfnl,
.nlh = nlh,
+ .nfmsg = nlmsg_data(nlh),
.extack = extack,
};
.net = net,
.sk = nfnlnet->nfnl,
.nlh = nlh,
+ .nfmsg = nlmsg_data(nlh),
.extack = &extack,
};
kfree_skb(skb2);
break;
}
- ret = netlink_unicast(info->sk, skb2, NETLINK_CB(skb).portid,
- MSG_DONTWAIT);
- if (ret > 0)
- ret = 0;
- /* this avoids a loop in nfnetlink. */
- return ret == -EAGAIN ? -ENOBUFS : ret;
+ ret = nfnetlink_unicast(skb2, info->net, NETLINK_CB(skb).portid);
+ break;
}
+
return ret;
}
nfnl_cthelper_update(const struct nlattr * const tb[],
struct nf_conntrack_helper *helper)
{
+ u32 size;
int ret;
- if (tb[NFCTH_PRIV_DATA_LEN])
- return -EBUSY;
+ if (tb[NFCTH_PRIV_DATA_LEN]) {
+ size = ntohl(nla_get_be32(tb[NFCTH_PRIV_DATA_LEN]));
+ if (size != helper->data_len)
+ return -EBUSY;
+ }
if (tb[NFCTH_POLICY]) {
ret = nfnl_cthelper_update_policy(helper, tb[NFCTH_POLICY]);
break;
}
- ret = netlink_unicast(info->sk, skb2, NETLINK_CB(skb).portid,
- MSG_DONTWAIT);
- if (ret > 0)
- ret = 0;
-
- /* this avoids a loop in nfnetlink. */
- return ret == -EAGAIN ? -ENOBUFS : ret;
+ ret = nfnetlink_unicast(skb2, info->net, NETLINK_CB(skb).portid);
+ break;
}
+
return ret;
}
kfree_skb(skb2);
break;
}
- ret = netlink_unicast(info->sk, skb2, NETLINK_CB(skb).portid,
- MSG_DONTWAIT);
- if (ret > 0)
- ret = 0;
- /* this avoids a loop in nfnetlink. */
- return ret == -EAGAIN ? -ENOBUFS : ret;
+ ret = nfnetlink_unicast(skb2, info->net, NETLINK_CB(skb).portid);
+ break;
}
+
return ret;
}
const struct nf_conntrack_l4proto *l4proto;
unsigned int *timeouts = NULL;
struct sk_buff *skb2;
- int ret, err;
__u16 l3num;
__u8 l4num;
+ int ret;
if (!cda[CTA_TIMEOUT_L3PROTO] || !cda[CTA_TIMEOUT_L4PROTO])
return -EINVAL;
l4num = nla_get_u8(cda[CTA_TIMEOUT_L4PROTO]);
l4proto = nf_ct_l4proto_find(l4num);
- err = -EOPNOTSUPP;
if (l4proto->l4proto != l4num)
- goto err;
+ return -EOPNOTSUPP;
switch (l4proto->l4proto) {
case IPPROTO_ICMP:
}
if (!timeouts)
- goto err;
+ return -EOPNOTSUPP;
skb2 = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
- if (skb2 == NULL) {
- err = -ENOMEM;
- goto err;
- }
+ if (!skb2)
+ return -ENOMEM;
ret = cttimeout_default_fill_info(info->net, skb2,
NETLINK_CB(skb).portid,
l3num, l4proto, timeouts);
if (ret <= 0) {
kfree_skb(skb2);
- err = -ENOMEM;
- goto err;
+ return -ENOMEM;
}
- ret = netlink_unicast(info->sk, skb2, NETLINK_CB(skb).portid,
- MSG_DONTWAIT);
- if (ret > 0)
- ret = 0;
- /* this avoids a loop in nfnetlink. */
- return ret == -EAGAIN ? -ENOBUFS : ret;
-err:
- return err;
+ return nfnetlink_unicast(skb2, info->net, NETLINK_CB(skb).portid);
}
static struct nf_ct_timeout *ctnl_timeout_find_get(struct net *net,
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (c) 2021 Red Hat GmbH
+ *
+ * Author: Florian Westphal <fw@strlen.de>
+ */
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/skbuff.h>
+#include <linux/errno.h>
+#include <linux/netlink.h>
+#include <linux/slab.h>
+
+#include <linux/netfilter.h>
+
+#include <linux/netfilter/nfnetlink.h>
+#include <linux/netfilter/nfnetlink_hook.h>
+
+#include <net/netfilter/nf_tables.h>
+#include <net/sock.h>
+
+static const struct nla_policy nfnl_hook_nla_policy[NFNLA_HOOK_MAX + 1] = {
+ [NFNLA_HOOK_HOOKNUM] = { .type = NLA_U32 },
+ [NFNLA_HOOK_PRIORITY] = { .type = NLA_U32 },
+ [NFNLA_HOOK_DEV] = { .type = NLA_STRING,
+ .len = IFNAMSIZ - 1 },
+ [NFNLA_HOOK_FUNCTION_NAME] = { .type = NLA_NUL_STRING,
+ .len = KSYM_NAME_LEN, },
+ [NFNLA_HOOK_MODULE_NAME] = { .type = NLA_NUL_STRING,
+ .len = MODULE_NAME_LEN, },
+ [NFNLA_HOOK_CHAIN_INFO] = { .type = NLA_NESTED, },
+};
+
+static int nf_netlink_dump_start_rcu(struct sock *nlsk, struct sk_buff *skb,
+ const struct nlmsghdr *nlh,
+ struct netlink_dump_control *c)
+{
+ int err;
+
+ if (!try_module_get(THIS_MODULE))
+ return -EINVAL;
+
+ rcu_read_unlock();
+ err = netlink_dump_start(nlsk, skb, nlh, c);
+ rcu_read_lock();
+ module_put(THIS_MODULE);
+
+ return err;
+}
+
+struct nfnl_dump_hook_data {
+ char devname[IFNAMSIZ];
+ unsigned long headv;
+ u8 hook;
+};
+
+static int nfnl_hook_put_nft_chain_info(struct sk_buff *nlskb,
+ const struct nfnl_dump_hook_data *ctx,
+ unsigned int seq,
+ const struct nf_hook_ops *ops)
+{
+ struct net *net = sock_net(nlskb->sk);
+ struct nlattr *nest, *nest2;
+ struct nft_chain *chain;
+ int ret = 0;
+
+ if (ops->hook_ops_type != NF_HOOK_OP_NF_TABLES)
+ return 0;
+
+ chain = ops->priv;
+ if (WARN_ON_ONCE(!chain))
+ return 0;
+
+ if (!nft_is_active(net, chain))
+ return 0;
+
+ nest = nla_nest_start(nlskb, NFNLA_HOOK_CHAIN_INFO);
+ if (!nest)
+ return -EMSGSIZE;
+
+ ret = nla_put_be32(nlskb, NFNLA_HOOK_INFO_TYPE,
+ htonl(NFNL_HOOK_TYPE_NFTABLES));
+ if (ret)
+ goto cancel_nest;
+
+ nest2 = nla_nest_start(nlskb, NFNLA_HOOK_INFO_DESC);
+ if (!nest2)
+ goto cancel_nest;
+
+ ret = nla_put_string(nlskb, NFTA_CHAIN_TABLE, chain->table->name);
+ if (ret)
+ goto cancel_nest;
+
+ ret = nla_put_string(nlskb, NFTA_CHAIN_NAME, chain->name);
+ if (ret)
+ goto cancel_nest;
+
+ nla_nest_end(nlskb, nest2);
+ nla_nest_end(nlskb, nest);
+ return ret;
+
+cancel_nest:
+ nla_nest_cancel(nlskb, nest);
+ return -EMSGSIZE;
+}
+
+static int nfnl_hook_dump_one(struct sk_buff *nlskb,
+ const struct nfnl_dump_hook_data *ctx,
+ const struct nf_hook_ops *ops,
+ unsigned int seq)
+{
+ u16 event = nfnl_msg_type(NFNL_SUBSYS_HOOK, NFNL_MSG_HOOK_GET);
+ unsigned int portid = NETLINK_CB(nlskb).portid;
+ struct nlmsghdr *nlh;
+ int ret = -EMSGSIZE;
+#ifdef CONFIG_KALLSYMS
+ char sym[KSYM_SYMBOL_LEN];
+ char *module_name;
+#endif
+ nlh = nfnl_msg_put(nlskb, portid, seq, event,
+ NLM_F_MULTI, ops->pf, NFNETLINK_V0, 0);
+ if (!nlh)
+ goto nla_put_failure;
+
+#ifdef CONFIG_KALLSYMS
+ ret = snprintf(sym, sizeof(sym), "%ps", ops->hook);
+ if (ret >= sizeof(sym)) {
+ ret = -EINVAL;
+ goto nla_put_failure;
+ }
+
+ module_name = strstr(sym, " [");
+ if (module_name) {
+ char *end;
+
+ module_name += 2;
+ end = strchr(module_name, ']');
+ if (end) {
+ *end = 0;
+
+ ret = nla_put_string(nlskb, NFNLA_HOOK_MODULE_NAME, module_name);
+ if (ret)
+ goto nla_put_failure;
+ }
+ }
+
+ ret = nla_put_string(nlskb, NFNLA_HOOK_FUNCTION_NAME, sym);
+ if (ret)
+ goto nla_put_failure;
+#endif
+
+ ret = nla_put_be32(nlskb, NFNLA_HOOK_HOOKNUM, htonl(ops->hooknum));
+ if (ret)
+ goto nla_put_failure;
+
+ ret = nla_put_be32(nlskb, NFNLA_HOOK_PRIORITY, htonl(ops->priority));
+ if (ret)
+ goto nla_put_failure;
+
+ ret = nfnl_hook_put_nft_chain_info(nlskb, ctx, seq, ops);
+ if (ret)
+ goto nla_put_failure;
+
+ nlmsg_end(nlskb, nlh);
+ return 0;
+nla_put_failure:
+ nlmsg_trim(nlskb, nlh);
+ return ret;
+}
+
+static const struct nf_hook_entries *
+nfnl_hook_entries_head(u8 pf, unsigned int hook, struct net *net, const char *dev)
+{
+ const struct nf_hook_entries *hook_head = NULL;
+ struct net_device *netdev;
+
+ switch (pf) {
+ case NFPROTO_IPV4:
+ if (hook >= ARRAY_SIZE(net->nf.hooks_ipv4))
+ return ERR_PTR(-EINVAL);
+ hook_head = rcu_dereference(net->nf.hooks_ipv4[hook]);
+ break;
+ case NFPROTO_IPV6:
+ if (hook >= ARRAY_SIZE(net->nf.hooks_ipv6))
+ return ERR_PTR(-EINVAL);
+ hook_head = rcu_dereference(net->nf.hooks_ipv6[hook]);
+ break;
+ case NFPROTO_ARP:
+#ifdef CONFIG_NETFILTER_FAMILY_ARP
+ if (hook >= ARRAY_SIZE(net->nf.hooks_arp))
+ return ERR_PTR(-EINVAL);
+ hook_head = rcu_dereference(net->nf.hooks_arp[hook]);
+#endif
+ break;
+ case NFPROTO_BRIDGE:
+#ifdef CONFIG_NETFILTER_FAMILY_BRIDGE
+ if (hook >= ARRAY_SIZE(net->nf.hooks_bridge))
+ return ERR_PTR(-EINVAL);
+ hook_head = rcu_dereference(net->nf.hooks_bridge[hook]);
+#endif
+ break;
+#if IS_ENABLED(CONFIG_DECNET)
+ case NFPROTO_DECNET:
+ if (hook >= ARRAY_SIZE(net->nf.hooks_decnet))
+ return ERR_PTR(-EINVAL);
+ hook_head = rcu_dereference(net->nf.hooks_decnet[hook]);
+ break;
+#endif
+#ifdef CONFIG_NETFILTER_INGRESS
+ case NFPROTO_NETDEV:
+ if (hook != NF_NETDEV_INGRESS)
+ return ERR_PTR(-EOPNOTSUPP);
+
+ if (!dev)
+ return ERR_PTR(-ENODEV);
+
+ netdev = dev_get_by_name_rcu(net, dev);
+ if (!netdev)
+ return ERR_PTR(-ENODEV);
+
+ return rcu_dereference(netdev->nf_hooks_ingress);
+#endif
+ default:
+ return ERR_PTR(-EPROTONOSUPPORT);
+ }
+
+ return hook_head;
+}
+
+static int nfnl_hook_dump(struct sk_buff *nlskb,
+ struct netlink_callback *cb)
+{
+ struct nfgenmsg *nfmsg = nlmsg_data(cb->nlh);
+ struct nfnl_dump_hook_data *ctx = cb->data;
+ int err, family = nfmsg->nfgen_family;
+ struct net *net = sock_net(nlskb->sk);
+ struct nf_hook_ops * const *ops;
+ const struct nf_hook_entries *e;
+ unsigned int i = cb->args[0];
+
+ rcu_read_lock();
+
+ e = nfnl_hook_entries_head(family, ctx->hook, net, ctx->devname);
+ if (!e)
+ goto done;
+
+ if (IS_ERR(e)) {
+ cb->seq++;
+ goto done;
+ }
+
+ if ((unsigned long)e != ctx->headv || i >= e->num_hook_entries)
+ cb->seq++;
+
+ ops = nf_hook_entries_get_hook_ops(e);
+
+ for (; i < e->num_hook_entries; i++) {
+ err = nfnl_hook_dump_one(nlskb, ctx, ops[i], cb->seq);
+ if (err)
+ break;
+ }
+
+done:
+ nl_dump_check_consistent(cb, nlmsg_hdr(nlskb));
+ rcu_read_unlock();
+ cb->args[0] = i;
+ return nlskb->len;
+}
+
+static int nfnl_hook_dump_start(struct netlink_callback *cb)
+{
+ const struct nfgenmsg *nfmsg = nlmsg_data(cb->nlh);
+ const struct nlattr * const *nla = cb->data;
+ struct nfnl_dump_hook_data *ctx = NULL;
+ struct net *net = sock_net(cb->skb->sk);
+ u8 family = nfmsg->nfgen_family;
+ char name[IFNAMSIZ] = "";
+ const void *head;
+ u32 hooknum;
+
+ hooknum = ntohl(nla_get_be32(nla[NFNLA_HOOK_HOOKNUM]));
+ if (hooknum > 255)
+ return -EINVAL;
+
+ if (family == NFPROTO_NETDEV) {
+ if (!nla[NFNLA_HOOK_DEV])
+ return -EINVAL;
+
+ nla_strscpy(name, nla[NFNLA_HOOK_DEV], sizeof(name));
+ }
+
+ rcu_read_lock();
+ /* Not dereferenced; for consistency check only */
+ head = nfnl_hook_entries_head(family, hooknum, net, name);
+ rcu_read_unlock();
+
+ if (head && IS_ERR(head))
+ return PTR_ERR(head);
+
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return -ENOMEM;
+
+ strscpy(ctx->devname, name, sizeof(ctx->devname));
+ ctx->headv = (unsigned long)head;
+ ctx->hook = hooknum;
+
+ cb->seq = 1;
+ cb->data = ctx;
+
+ return 0;
+}
+
+static int nfnl_hook_dump_stop(struct netlink_callback *cb)
+{
+ kfree(cb->data);
+ return 0;
+}
+
+static int nfnl_hook_get(struct sk_buff *skb,
+ const struct nfnl_info *info,
+ const struct nlattr * const nla[])
+{
+ if (!nla[NFNLA_HOOK_HOOKNUM])
+ return -EINVAL;
+
+ if (info->nlh->nlmsg_flags & NLM_F_DUMP) {
+ struct netlink_dump_control c = {
+ .start = nfnl_hook_dump_start,
+ .done = nfnl_hook_dump_stop,
+ .dump = nfnl_hook_dump,
+ .module = THIS_MODULE,
+ .data = (void *)nla,
+ };
+
+ return nf_netlink_dump_start_rcu(info->sk, skb, info->nlh, &c);
+ }
+
+ return -EOPNOTSUPP;
+}
+
+static const struct nfnl_callback nfnl_hook_cb[NFNL_MSG_HOOK_MAX] = {
+ [NFNL_MSG_HOOK_GET] = {
+ .call = nfnl_hook_get,
+ .type = NFNL_CB_RCU,
+ .attr_count = NFNLA_HOOK_MAX,
+ .policy = nfnl_hook_nla_policy
+ },
+};
+
+static const struct nfnetlink_subsystem nfhook_subsys = {
+ .name = "nfhook",
+ .subsys_id = NFNL_SUBSYS_HOOK,
+ .cb_count = NFNL_MSG_HOOK_MAX,
+ .cb = nfnl_hook_cb,
+};
+
+MODULE_ALIAS_NFNL_SUBSYS(NFNL_SUBSYS_HOOK);
+
+static int __init nfnetlink_hook_init(void)
+{
+ return nfnetlink_subsys_register(&nfhook_subsys);
+}
+
+static void __exit nfnetlink_hook_exit(void)
+{
+ nfnetlink_subsys_unregister(&nfhook_subsys);
+}
+
+module_init(nfnetlink_hook_init);
+module_exit(nfnetlink_hook_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Florian Westphal <fw@strlen.de>");
+MODULE_DESCRIPTION("nfnetlink_hook: list registered netfilter hooks");
const struct nlattr * const nfula[])
{
struct nfnl_log_net *log = nfnl_log_pernet(info->net);
- struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
- u_int16_t group_num = ntohs(nfmsg->res_id);
+ u_int16_t group_num = ntohs(info->nfmsg->res_id);
struct nfulnl_msg_config_cmd *cmd = NULL;
struct nfulnl_instance *inst;
u16 flags = 0;
int ret = 0;
if (nfula[NFULA_CFG_CMD]) {
- u_int8_t pf = nfmsg->nfgen_family;
+ u_int8_t pf = info->nfmsg->nfgen_family;
cmd = nla_data(nfula[NFULA_CFG_CMD]);
/* Commands without queue context */
const struct nlattr * const nfqa[])
{
struct nfnl_queue_net *q = nfnl_queue_pernet(info->net);
- struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
- u16 queue_num = ntohs(nfmsg->res_id);
+ u16 queue_num = ntohs(info->nfmsg->res_id);
struct nf_queue_entry *entry, *tmp;
struct nfqnl_msg_verdict_hdr *vhdr;
struct nfqnl_instance *queue;
const struct nlattr * const nfqa[])
{
struct nfnl_queue_net *q = nfnl_queue_pernet(info->net);
- struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
- u_int16_t queue_num = ntohs(nfmsg->res_id);
+ u_int16_t queue_num = ntohs(info->nfmsg->res_id);
struct nfqnl_msg_verdict_hdr *vhdr;
enum ip_conntrack_info ctinfo;
struct nfqnl_instance *queue;
const struct nlattr * const nfqa[])
{
struct nfnl_queue_net *q = nfnl_queue_pernet(info->net);
- struct nfgenmsg *nfmsg = nlmsg_data(info->nlh);
- u_int16_t queue_num = ntohs(nfmsg->res_id);
+ u_int16_t queue_num = ntohs(info->nfmsg->res_id);
struct nfqnl_msg_config_cmd *cmd = NULL;
struct nfqnl_instance *queue;
__u32 flags = 0, mask = 0;
struct nft_pktinfo pkt;
nft_set_pktinfo(&pkt, skb, state);
- nft_set_pktinfo_ipv4(&pkt, skb);
+ nft_set_pktinfo_ipv4(&pkt);
return nft_do_chain(&pkt, priv);
}
struct nft_pktinfo pkt;
nft_set_pktinfo(&pkt, skb, state);
- nft_set_pktinfo_unspec(&pkt, skb);
+ nft_set_pktinfo_unspec(&pkt);
return nft_do_chain(&pkt, priv);
}
struct nft_pktinfo pkt;
nft_set_pktinfo(&pkt, skb, state);
- nft_set_pktinfo_ipv6(&pkt, skb);
+ nft_set_pktinfo_ipv6(&pkt);
return nft_do_chain(&pkt, priv);
}
switch (state->pf) {
case NFPROTO_IPV4:
- nft_set_pktinfo_ipv4(&pkt, skb);
+ nft_set_pktinfo_ipv4(&pkt);
break;
case NFPROTO_IPV6:
- nft_set_pktinfo_ipv6(&pkt, skb);
+ nft_set_pktinfo_ipv6(&pkt);
break;
default:
break;
ingress_state.hook = NF_INET_INGRESS;
nft_set_pktinfo(&pkt, skb, &ingress_state);
- if (nft_set_pktinfo_ipv4_ingress(&pkt, skb) < 0)
+ if (nft_set_pktinfo_ipv4_ingress(&pkt) < 0)
return NF_DROP;
break;
case htons(ETH_P_IPV6):
ingress_state.hook = NF_INET_INGRESS;
nft_set_pktinfo(&pkt, skb, &ingress_state);
- if (nft_set_pktinfo_ipv6_ingress(&pkt, skb) < 0)
+ if (nft_set_pktinfo_ipv6_ingress(&pkt) < 0)
return NF_DROP;
break;
default:
switch (eth_hdr(skb)->h_proto) {
case htons(ETH_P_IP):
- nft_set_pktinfo_ipv4_validate(&pkt, skb);
+ nft_set_pktinfo_ipv4_validate(&pkt);
break;
case htons(ETH_P_IPV6):
- nft_set_pktinfo_ipv6_validate(&pkt, skb);
+ nft_set_pktinfo_ipv6_validate(&pkt);
break;
default:
- nft_set_pktinfo_unspec(&pkt, skb);
+ nft_set_pktinfo_unspec(&pkt);
break;
}
switch (skb->protocol) {
case htons(ETH_P_IP):
- nft_set_pktinfo_ipv4_validate(&pkt, skb);
+ nft_set_pktinfo_ipv4_validate(&pkt);
break;
case htons(ETH_P_IPV6):
- nft_set_pktinfo_ipv6_validate(&pkt, skb);
+ nft_set_pktinfo_ipv6_validate(&pkt);
break;
default:
- nft_set_pktinfo_unspec(&pkt, skb);
+ nft_set_pktinfo_unspec(&pkt);
break;
}
switch (state->pf) {
#ifdef CONFIG_NF_TABLES_IPV4
case NFPROTO_IPV4:
- nft_set_pktinfo_ipv4(&pkt, skb);
+ nft_set_pktinfo_ipv4(&pkt);
break;
#endif
#ifdef CONFIG_NF_TABLES_IPV6
case NFPROTO_IPV6:
- nft_set_pktinfo_ipv6(&pkt, skb);
+ nft_set_pktinfo_ipv6(&pkt);
break;
#endif
default:
u8 tos;
nft_set_pktinfo(&pkt, skb, state);
- nft_set_pktinfo_ipv4(&pkt, skb);
+ nft_set_pktinfo_ipv4(&pkt);
mark = skb->mark;
iph = ip_hdr(skb);
int err;
nft_set_pktinfo(&pkt, skb, state);
- nft_set_pktinfo_ipv6(&pkt, skb);
+ nft_set_pktinfo_ipv6(&pkt);
/* save source/dest address, mark, hoplimit, flowlabel, priority */
memcpy(&saddr, &ipv6_hdr(skb)->saddr, sizeof(saddr));
};
static inline void
-nft_compat_set_par(struct xt_action_param *par, void *xt, const void *xt_info)
+nft_compat_set_par(struct xt_action_param *par,
+ const struct nft_pktinfo *pkt,
+ const void *xt, const void *xt_info)
{
+ par->state = pkt->state;
+ par->thoff = nft_thoff(pkt);
+ par->fragoff = pkt->fragoff;
par->target = xt;
par->targinfo = xt_info;
par->hotdrop = false;
void *info = nft_expr_priv(expr);
struct xt_target *target = expr->ops->data;
struct sk_buff *skb = pkt->skb;
+ struct xt_action_param xt;
int ret;
- nft_compat_set_par((struct xt_action_param *)&pkt->xt, target, info);
+ nft_compat_set_par(&xt, pkt, target, info);
- ret = target->target(skb, &pkt->xt);
+ ret = target->target(skb, &xt);
- if (pkt->xt.hotdrop)
+ if (xt.hotdrop)
ret = NF_DROP;
switch (ret) {
void *info = nft_expr_priv(expr);
struct xt_target *target = expr->ops->data;
struct sk_buff *skb = pkt->skb;
+ struct xt_action_param xt;
int ret;
- nft_compat_set_par((struct xt_action_param *)&pkt->xt, target, info);
+ nft_compat_set_par(&xt, pkt, target, info);
- ret = target->target(skb, &pkt->xt);
+ ret = target->target(skb, &xt);
- if (pkt->xt.hotdrop)
+ if (xt.hotdrop)
ret = NF_DROP;
switch (ret) {
{
struct xt_match *match = expr->ops->data;
struct sk_buff *skb = pkt->skb;
+ struct xt_action_param xt;
bool ret;
- nft_compat_set_par((struct xt_action_param *)&pkt->xt, match, info);
+ nft_compat_set_par(&xt, pkt, match, info);
- ret = match->match(skb, (struct xt_action_param *)&pkt->xt);
+ ret = match->match(skb, &xt);
- if (pkt->xt.hotdrop) {
+ if (xt.hotdrop) {
regs->verdict.code = NF_DROP;
return;
}
const struct nfnl_info *info,
const struct nlattr * const tb[])
{
- struct nfgenmsg *nfmsg;
+ u8 family = info->nfmsg->nfgen_family;
const char *name, *fmt;
struct sk_buff *skb2;
int ret = 0, target;
rev = ntohl(nla_get_be32(tb[NFTA_COMPAT_REV]));
target = ntohl(nla_get_be32(tb[NFTA_COMPAT_TYPE]));
- nfmsg = nlmsg_data(info->nlh);
-
- switch(nfmsg->nfgen_family) {
+ switch(family) {
case AF_INET:
fmt = "ipt_%s";
break;
fmt = "arpt_%s";
break;
default:
- pr_err("nft_compat: unsupported protocol %d\n",
- nfmsg->nfgen_family);
+ pr_err("nft_compat: unsupported protocol %d\n", family);
return -EINVAL;
}
return -EINVAL;
rcu_read_unlock();
- try_then_request_module(xt_find_revision(nfmsg->nfgen_family, name,
- rev, target, &ret),
- fmt, name);
+ try_then_request_module(xt_find_revision(family, name, rev, target, &ret),
+ fmt, name);
if (ret < 0)
goto out_put;
info->nlh->nlmsg_seq,
NFNL_MSG_TYPE(info->nlh->nlmsg_type),
NFNL_MSG_COMPAT_GET,
- nfmsg->nfgen_family,
- name, ret, target) <= 0) {
+ family, name, ret, target) <= 0) {
kfree_skb(skb2);
goto out_put;
}
struct nf_conn *ct;
ct = nf_ct_get(pkt->skb, &ctinfo);
- if (!ct || ctinfo == IP_CT_UNTRACKED) {
+ if (!ct || nf_ct_is_confirmed(ct) || nf_ct_is_template(ct)) {
regs->verdict.code = NFT_BREAK;
return;
}
#include <linux/netlink.h>
#include <linux/netfilter.h>
#include <linux/netfilter/nf_tables.h>
+#include <linux/sctp.h>
#include <net/netfilter/nf_tables_core.h>
#include <net/netfilter/nf_tables.h>
+#include <net/sctp/sctp.h>
#include <net/tcp.h>
struct nft_exthdr {
{
struct tcphdr *tcph;
- if (!pkt->tprot_set || pkt->tprot != IPPROTO_TCP)
+ if (pkt->tprot != IPPROTO_TCP)
return NULL;
- tcph = skb_header_pointer(pkt->skb, pkt->xt.thoff, sizeof(*tcph), buffer);
+ tcph = skb_header_pointer(pkt->skb, nft_thoff(pkt), sizeof(*tcph), buffer);
if (!tcph)
return NULL;
if (*tcphdr_len < sizeof(*tcph) || *tcphdr_len > len)
return NULL;
- return skb_header_pointer(pkt->skb, pkt->xt.thoff, *tcphdr_len, buffer);
+ return skb_header_pointer(pkt->skb, nft_thoff(pkt), *tcphdr_len, buffer);
}
static void nft_exthdr_tcp_eval(const struct nft_expr *expr,
return;
if (skb_ensure_writable(pkt->skb,
- pkt->xt.thoff + i + priv->len))
+ nft_thoff(pkt) + i + priv->len))
return;
tcph = nft_tcp_header_pointer(pkt, sizeof(buff), buff,
}
}
+static void nft_exthdr_sctp_eval(const struct nft_expr *expr,
+ struct nft_regs *regs,
+ const struct nft_pktinfo *pkt)
+{
+ unsigned int offset = nft_thoff(pkt) + sizeof(struct sctphdr);
+ struct nft_exthdr *priv = nft_expr_priv(expr);
+ u32 *dest = ®s->data[priv->dreg];
+ const struct sctp_chunkhdr *sch;
+ struct sctp_chunkhdr _sch;
+
+ if (pkt->tprot != IPPROTO_SCTP)
+ goto err;
+
+ do {
+ sch = skb_header_pointer(pkt->skb, offset, sizeof(_sch), &_sch);
+ if (!sch || !sch->length)
+ break;
+
+ if (sch->type == priv->type) {
+ if (priv->flags & NFT_EXTHDR_F_PRESENT) {
+ nft_reg_store8(dest, true);
+ return;
+ }
+ if (priv->offset + priv->len > ntohs(sch->length) ||
+ offset + ntohs(sch->length) > pkt->skb->len)
+ break;
+
+ dest[priv->len / NFT_REG32_SIZE] = 0;
+ if (skb_copy_bits(pkt->skb, offset + priv->offset,
+ dest, priv->len) < 0)
+ break;
+ return;
+ }
+ offset += SCTP_PAD4(ntohs(sch->length));
+ } while (offset < pkt->skb->len);
+err:
+ if (priv->flags & NFT_EXTHDR_F_PRESENT)
+ nft_reg_store8(dest, false);
+ else
+ regs->verdict.code = NFT_BREAK;
+}
+
static const struct nla_policy nft_exthdr_policy[NFTA_EXTHDR_MAX + 1] = {
[NFTA_EXTHDR_DREG] = { .type = NLA_U32 },
[NFTA_EXTHDR_TYPE] = { .type = NLA_U8 },
.dump = nft_exthdr_dump_set,
};
+static const struct nft_expr_ops nft_exthdr_sctp_ops = {
+ .type = &nft_exthdr_type,
+ .size = NFT_EXPR_SIZE(sizeof(struct nft_exthdr)),
+ .eval = nft_exthdr_sctp_eval,
+ .init = nft_exthdr_init,
+ .dump = nft_exthdr_dump,
+};
+
static const struct nft_expr_ops *
nft_exthdr_select_ops(const struct nft_ctx *ctx,
const struct nlattr * const tb[])
return &nft_exthdr_ipv4_ops;
}
break;
+ case NFT_EXTHDR_OP_SCTP:
+ if (tb[NFTA_EXTHDR_DREG])
+ return &nft_exthdr_sctp_ops;
+ break;
}
return ERR_PTR(-EOPNOTSUPP);
switch (ct->tuplehash[IP_CT_DIR_ORIGINAL].tuple.dst.protonum) {
case IPPROTO_TCP:
- tcph = skb_header_pointer(pkt->skb, pkt->xt.thoff,
+ tcph = skb_header_pointer(pkt->skb, nft_thoff(pkt),
sizeof(_tcph), &_tcph);
if (unlikely(!tcph || tcph->fin || tcph->rst))
goto out;
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/netlink.h>
+#include <linux/netfilter.h>
+#include <linux/netfilter/nf_tables.h>
+#include <net/netfilter/nf_tables_core.h>
+#include <net/netfilter/nf_tables.h>
+
+struct nft_last_priv {
+ unsigned long last_jiffies;
+ unsigned int last_set;
+};
+
+static const struct nla_policy nft_last_policy[NFTA_LAST_MAX + 1] = {
+ [NFTA_LAST_SET] = { .type = NLA_U32 },
+ [NFTA_LAST_MSECS] = { .type = NLA_U64 },
+};
+
+static int nft_last_init(const struct nft_ctx *ctx, const struct nft_expr *expr,
+ const struct nlattr * const tb[])
+{
+ struct nft_last_priv *priv = nft_expr_priv(expr);
+ u64 last_jiffies;
+ int err;
+
+ if (tb[NFTA_LAST_MSECS]) {
+ err = nf_msecs_to_jiffies64(tb[NFTA_LAST_MSECS], &last_jiffies);
+ if (err < 0)
+ return err;
+
+ priv->last_jiffies = jiffies + (unsigned long)last_jiffies;
+ priv->last_set = 1;
+ }
+
+ return 0;
+}
+
+static void nft_last_eval(const struct nft_expr *expr,
+ struct nft_regs *regs, const struct nft_pktinfo *pkt)
+{
+ struct nft_last_priv *priv = nft_expr_priv(expr);
+
+ priv->last_jiffies = jiffies;
+ priv->last_set = 1;
+}
+
+static int nft_last_dump(struct sk_buff *skb, const struct nft_expr *expr)
+{
+ struct nft_last_priv *priv = nft_expr_priv(expr);
+ __be64 msecs;
+
+ if (time_before(jiffies, priv->last_jiffies))
+ priv->last_set = 0;
+
+ if (priv->last_set)
+ msecs = nf_jiffies64_to_msecs(jiffies - priv->last_jiffies);
+ else
+ msecs = 0;
+
+ if (nla_put_be32(skb, NFTA_LAST_SET, htonl(priv->last_set)) ||
+ nla_put_be64(skb, NFTA_LAST_MSECS, msecs, NFTA_LAST_PAD))
+ goto nla_put_failure;
+
+ return 0;
+
+nla_put_failure:
+ return -1;
+}
+
+static const struct nft_expr_ops nft_last_ops = {
+ .type = &nft_last_type,
+ .size = NFT_EXPR_SIZE(sizeof(struct nft_last_priv)),
+ .eval = nft_last_eval,
+ .init = nft_last_init,
+ .dump = nft_last_dump,
+};
+
+struct nft_expr_type nft_last_type __read_mostly = {
+ .name = "last",
+ .ops = &nft_last_ops,
+ .policy = nft_last_policy,
+ .maxattr = NFTA_LAST_MAX,
+ .flags = NFT_EXPR_STATEFUL,
+ .owner = THIS_MODULE,
+};
struct nft_set_binding binding;
};
+#ifdef CONFIG_RETPOLINE
+bool nft_set_do_lookup(const struct net *net, const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext)
+{
+ if (set->ops == &nft_set_hash_fast_type.ops)
+ return nft_hash_lookup_fast(net, set, key, ext);
+ if (set->ops == &nft_set_hash_type.ops)
+ return nft_hash_lookup(net, set, key, ext);
+
+ if (set->ops == &nft_set_rhash_type.ops)
+ return nft_rhash_lookup(net, set, key, ext);
+
+ if (set->ops == &nft_set_bitmap_type.ops)
+ return nft_bitmap_lookup(net, set, key, ext);
+
+ if (set->ops == &nft_set_pipapo_type.ops)
+ return nft_pipapo_lookup(net, set, key, ext);
+#if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+ if (set->ops == &nft_set_pipapo_avx2_type.ops)
+ return nft_pipapo_avx2_lookup(net, set, key, ext);
+#endif
+
+ if (set->ops == &nft_set_rbtree_type.ops)
+ return nft_rbtree_lookup(net, set, key, ext);
+
+ WARN_ON_ONCE(1);
+ return set->ops->lookup(net, set, key, ext);
+}
+EXPORT_SYMBOL_GPL(nft_set_do_lookup);
+#endif
+
void nft_lookup_eval(const struct nft_expr *expr,
struct nft_regs *regs,
const struct nft_pktinfo *pkt)
const struct net *net = nft_net(pkt);
bool found;
- found = set->ops->lookup(net, set, ®s->data[priv->sreg], &ext) ^
- priv->invert;
+ found = nft_set_do_lookup(net, set, ®s->data[priv->sreg], &ext) ^
+ priv->invert;
if (!found) {
ext = nft_set_catchall_lookup(net, set);
if (!ext) {
#include <linux/netlink.h>
#include <linux/netfilter.h>
#include <linux/netfilter/nf_tables.h>
-#include <net/netfilter/nf_tables.h>
+#include <net/netfilter/nf_tables_core.h>
#define nft_objref_priv(expr) *((struct nft_object **)nft_expr_priv(expr))
struct nft_object *obj;
bool found;
- found = set->ops->lookup(net, set, ®s->data[priv->sreg], &ext);
+ found = nft_set_do_lookup(net, set, ®s->data[priv->sreg], &ext);
if (!found) {
ext = nft_set_catchall_lookup(net, set);
if (!ext) {
case NFT_PAYLOAD_TRANSPORT_HEADER:
if (!pkt->tprot_set)
goto err;
- offset = pkt->xt.thoff;
+ offset = nft_thoff(pkt);
break;
default:
BUG();
*l4csum_offset = offsetof(struct tcphdr, check);
break;
case IPPROTO_UDP:
- if (!nft_payload_udp_checksum(skb, pkt->xt.thoff))
+ if (!nft_payload_udp_checksum(skb, nft_thoff(pkt)))
return -1;
fallthrough;
case IPPROTO_UDPLITE:
return -1;
}
- *l4csum_offset += pkt->xt.thoff;
+ *l4csum_offset += nft_thoff(pkt);
return 0;
}
case NFT_PAYLOAD_TRANSPORT_HEADER:
if (!pkt->tprot_set)
goto err;
- offset = pkt->xt.thoff;
+ offset = nft_thoff(pkt);
break;
default:
BUG();
if (priv->csum_type == NFT_PAYLOAD_CSUM_SCTP &&
pkt->tprot == IPPROTO_SCTP &&
skb->ip_summed != CHECKSUM_PARTIAL) {
- if (nft_payload_csum_sctp(skb, pkt->xt.thoff))
+ if (nft_payload_csum_sctp(skb, nft_thoff(pkt)))
goto err;
}
nft_hook(pkt));
break;
case NFT_REJECT_TCP_RST:
- nf_send_reset(nft_net(pkt), pkt->xt.state->sk,
+ nf_send_reset(nft_net(pkt), nft_sk(pkt),
pkt->skb, nft_hook(pkt));
break;
case NFT_REJECT_ICMPX_UNREACH:
priv->icmp_code, nft_hook(pkt));
break;
case NFT_REJECT_TCP_RST:
- nf_send_reset6(nft_net(pkt), pkt->xt.state->sk,
+ nf_send_reset6(nft_net(pkt), nft_sk(pkt),
pkt->skb, nft_hook(pkt));
break;
case NFT_REJECT_ICMPX_UNREACH:
return (bitmap[idx] & (0x3 << off)) & (genmask << off);
}
-static bool nft_bitmap_lookup(const struct net *net, const struct nft_set *set,
- const u32 *key, const struct nft_set_ext **ext)
+INDIRECT_CALLABLE_SCOPE
+bool nft_bitmap_lookup(const struct net *net, const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext)
{
const struct nft_bitmap *priv = nft_set_priv(set);
u8 genmask = nft_genmask_cur(net);
.automatic_shrinking = true,
};
-static bool nft_rhash_lookup(const struct net *net, const struct nft_set *set,
- const u32 *key, const struct nft_set_ext **ext)
+INDIRECT_CALLABLE_SCOPE
+bool nft_rhash_lookup(const struct net *net, const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext)
{
struct nft_rhash *priv = nft_set_priv(set);
const struct nft_rhash_elem *he;
struct nft_set_ext ext;
};
-static bool nft_hash_lookup(const struct net *net, const struct nft_set *set,
- const u32 *key, const struct nft_set_ext **ext)
+INDIRECT_CALLABLE_SCOPE
+bool nft_hash_lookup(const struct net *net, const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext)
{
struct nft_hash *priv = nft_set_priv(set);
u8 genmask = nft_genmask_cur(net);
return ERR_PTR(-ENOENT);
}
-static bool nft_hash_lookup_fast(const struct net *net,
- const struct nft_set *set,
- const u32 *key, const struct nft_set_ext **ext)
+INDIRECT_CALLABLE_SCOPE
+bool nft_hash_lookup_fast(const struct net *net,
+ const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext)
{
struct nft_hash *priv = nft_set_priv(set);
u8 genmask = nft_genmask_cur(net);
*
* Return: true on match, false otherwise.
*/
-static bool nft_pipapo_lookup(const struct net *net, const struct nft_set *set,
- const u32 *key, const struct nft_set_ext **ext)
+bool nft_pipapo_lookup(const struct net *net, const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext)
{
struct nft_pipapo *priv = nft_set_priv(set);
unsigned long *res_map, *fill_map;
* @map: Bitmap to be scanned for set bits
* @dst: Destination bitmap
* @mt: Mapping table containing bit set specifiers
- * @len: Length of bitmap in longs
* @last: Return index of first set bit, if this is the last field
*
* This is an alternative implementation of pipapo_refill() suitable for usage
* nft_pipapo_avx2_lookup() - Lookup function for AVX2 implementation
* @net: Network namespace
* @set: nftables API set representation
- * @elem: nftables API element representation containing key data
+ * @key: nftables API element representation containing key data
* @ext: nftables API extension pointer, filled with matching reference
*
* For more details, see DOC: Theory of Operation in nft_set_pipapo.c.
bool map_index;
int i, ret = 0;
+ if (unlikely(!irq_fpu_usable()))
+ return nft_pipapo_lookup(net, set, key, ext);
+
m = rcu_dereference(priv->match);
- /* This also protects access to all data related to scratch maps */
- kernel_fpu_begin();
+ /* This also protects access to all data related to scratch maps.
+ *
+ * Note that we don't need a valid MXCSR state for any of the
+ * operations we use here, so pass 0 as mask and spare a LDMXCSR
+ * instruction.
+ */
+ kernel_fpu_begin_mask(0);
scratch = *raw_cpu_ptr(m->scratch_aligned);
if (unlikely(!scratch)) {
#include <asm/fpu/xstate.h>
#define NFT_PIPAPO_ALIGN (XSAVE_YMM_SIZE / BITS_PER_BYTE)
-bool nft_pipapo_avx2_lookup(const struct net *net, const struct nft_set *set,
- const u32 *key, const struct nft_set_ext **ext);
bool nft_pipapo_avx2_estimate(const struct nft_set_desc *desc, u32 features,
struct nft_set_estimate *est);
#endif /* defined(CONFIG_X86_64) && !defined(CONFIG_UML) */
return false;
}
-static bool nft_rbtree_lookup(const struct net *net, const struct nft_set *set,
- const u32 *key, const struct nft_set_ext **ext)
+INDIRECT_CALLABLE_SCOPE
+bool nft_rbtree_lookup(const struct net *net, const struct nft_set *set,
+ const u32 *key, const struct nft_set_ext **ext)
{
struct nft_rbtree *priv = nft_set_priv(set);
unsigned int seq = read_seqcount_begin(&priv->count);
{
struct synproxy_options opts = {};
struct sk_buff *skb = pkt->skb;
- int thoff = pkt->xt.thoff;
+ int thoff = nft_thoff(pkt);
const struct tcphdr *tcp;
struct tcphdr _tcph;
return;
}
- tcp = skb_header_pointer(skb, pkt->xt.thoff,
+ tcp = skb_header_pointer(skb, thoff,
sizeof(struct tcphdr),
&_tcph);
if (!tcp) {
const struct nft_tproxy *priv = nft_expr_priv(expr);
struct sk_buff *skb = pkt->skb;
const struct ipv6hdr *iph = ipv6_hdr(skb);
- struct in6_addr taddr;
- int thoff = pkt->xt.thoff;
+ int thoff = nft_thoff(pkt);
struct udphdr _hdr, *hp;
+ struct in6_addr taddr;
__be16 tport = 0;
struct sock *sk;
int l4proto;
const struct xt_audit_info *info = par->targinfo;
if (info->type > XT_AUDIT_TYPE_MAX) {
- pr_info_ratelimited("Audit type out of range (valid range: 0..%hhu)\n",
+ pr_info_ratelimited("Audit type out of range (valid range: 0..%u)\n",
XT_AUDIT_TYPE_MAX);
return -ERANGE;
}
goto err2;
}
- ret = 0;
if ((info->ct_events || info->exp_events) &&
!nf_ct_ecache_ext_add(ct, info->ct_events, info->exp_events,
GFP_KERNEL)) {
#include <linux/slab.h>
#include <linux/module.h>
#include <linux/skbuff.h>
-#include <linux/spinlock.h>
#include <linux/interrupt.h>
#include <linux/netfilter/x_tables.h>
#include <linux/netfilter/xt_limit.h>
struct xt_limit_priv {
- spinlock_t lock;
unsigned long prev;
- uint32_t credit;
+ u32 credit;
};
MODULE_LICENSE("GPL");
{
const struct xt_rateinfo *r = par->matchinfo;
struct xt_limit_priv *priv = r->master;
- unsigned long now = jiffies;
-
- spin_lock_bh(&priv->lock);
- priv->credit += (now - xchg(&priv->prev, now)) * CREDITS_PER_JIFFY;
- if (priv->credit > r->credit_cap)
- priv->credit = r->credit_cap;
-
- if (priv->credit >= r->cost) {
- /* We're not limited. */
- priv->credit -= r->cost;
- spin_unlock_bh(&priv->lock);
- return true;
- }
-
- spin_unlock_bh(&priv->lock);
- return false;
+ unsigned long now;
+ u32 old_credit, new_credit, credit_increase = 0;
+ bool ret;
+
+ /* fastpath if there is nothing to update */
+ if ((READ_ONCE(priv->credit) < r->cost) && (READ_ONCE(priv->prev) == jiffies))
+ return false;
+
+ do {
+ now = jiffies;
+ credit_increase += (now - xchg(&priv->prev, now)) * CREDITS_PER_JIFFY;
+ old_credit = READ_ONCE(priv->credit);
+ new_credit = old_credit;
+ new_credit += credit_increase;
+ if (new_credit > r->credit_cap)
+ new_credit = r->credit_cap;
+ if (new_credit >= r->cost) {
+ ret = true;
+ new_credit -= r->cost;
+ } else {
+ ret = false;
+ }
+ } while (cmpxchg(&priv->credit, old_credit, new_credit) != old_credit);
+
+ return ret;
}
/* Precision saver. */
r->credit_cap = priv->credit; /* Credits full. */
r->cost = user2credits(r->avg);
}
- spin_lock_init(&priv->lock);
return 0;
}
!info->attrs[NLBL_CALIPSO_A_MTYPE])
return -EINVAL;
- netlbl_netlink_auditinfo(skb, &audit_info);
+ netlbl_netlink_auditinfo(&audit_info);
switch (nla_get_u32(info->attrs[NLBL_CALIPSO_A_MTYPE])) {
case CALIPSO_MAP_PASS:
ret_val = netlbl_calipso_add_pass(info, &audit_info);
if (!info->attrs[NLBL_CALIPSO_A_DOI])
return -EINVAL;
- netlbl_netlink_auditinfo(skb, &audit_info);
+ netlbl_netlink_auditinfo(&audit_info);
cb_arg.doi = nla_get_u32(info->attrs[NLBL_CALIPSO_A_DOI]);
cb_arg.audit_info = &audit_info;
ret_val = netlbl_domhsh_walk(&skip_bkt, &skip_chain,
!info->attrs[NLBL_CIPSOV4_A_MTYPE])
return -EINVAL;
- netlbl_netlink_auditinfo(skb, &audit_info);
+ netlbl_netlink_auditinfo(&audit_info);
switch (nla_get_u32(info->attrs[NLBL_CIPSOV4_A_MTYPE])) {
case CIPSO_V4_MAP_TRANS:
ret_val = netlbl_cipsov4_add_std(info, &audit_info);
if (!info->attrs[NLBL_CIPSOV4_A_DOI])
return -EINVAL;
- netlbl_netlink_auditinfo(skb, &audit_info);
+ netlbl_netlink_auditinfo(&audit_info);
cb_arg.doi = nla_get_u32(info->attrs[NLBL_CIPSOV4_A_DOI]);
cb_arg.audit_info = &audit_info;
ret_val = netlbl_domhsh_walk(&skip_bkt, &skip_chain,
* @cb_arg: argument for the callback function
*
* Description:
- * Interate over the domain mapping hash table, skipping the first @skip_bkt
+ * Iterate over the domain mapping hash table, skipping the first @skip_bkt
* buckets and @skip_chain entries. For each entry in the table call
* @callback, if @callback returns a negative value stop 'walking' through the
* table and return. Updates the values in @skip_bkt and @skip_chain on
* it in @bitmap. The @offset must be aligned to an unsigned long and will be
* updated on return if different from what was requested; if the catmap is
* empty at the requested offset and beyond, the @offset is set to (u32)-1.
- * Returns zero on sucess, negative values on failure.
+ * Returns zero on success, negative values on failure.
*
*/
int netlbl_catmap_getlong(struct netlbl_lsm_catmap *catmap,
static int netlbl_mgmt_add_common(struct genl_info *info,
struct netlbl_audit *audit_info)
{
+ void *pmap = NULL;
int ret_val = -EINVAL;
struct netlbl_domaddr_map *addrmap = NULL;
struct cipso_v4_doi *cipsov4 = NULL;
ret_val = -ENOMEM;
goto add_free_addrmap;
}
+ pmap = map;
map->list.addr = addr->s_addr & mask->s_addr;
map->list.mask = mask->s_addr;
map->list.valid = 1;
map->def.cipso = cipsov4;
ret_val = netlbl_af4list_add(&map->list, &addrmap->list4);
- if (ret_val != 0) {
- kfree(map);
- goto add_free_addrmap;
- }
+ if (ret_val != 0)
+ goto add_free_map;
entry->family = AF_INET;
entry->def.type = NETLBL_NLTYPE_ADDRSELECT;
ret_val = -ENOMEM;
goto add_free_addrmap;
}
+ pmap = map;
map->list.addr = *addr;
map->list.addr.s6_addr32[0] &= mask->s6_addr32[0];
map->list.addr.s6_addr32[1] &= mask->s6_addr32[1];
map->def.calipso = calipso;
ret_val = netlbl_af6list_add(&map->list, &addrmap->list6);
- if (ret_val != 0) {
- kfree(map);
- goto add_free_addrmap;
- }
+ if (ret_val != 0)
+ goto add_free_map;
entry->family = AF_INET6;
entry->def.type = NETLBL_NLTYPE_ADDRSELECT;
ret_val = netlbl_domhsh_add(entry, audit_info);
if (ret_val != 0)
- goto add_free_addrmap;
+ goto add_free_map;
return 0;
+add_free_map:
+ kfree(pmap);
add_free_addrmap:
kfree(addrmap);
add_doi_put_def:
(info->attrs[NLBL_MGMT_A_IPV6MASK] != NULL)))
return -EINVAL;
- netlbl_netlink_auditinfo(skb, &audit_info);
+ netlbl_netlink_auditinfo(&audit_info);
return netlbl_mgmt_add_common(info, &audit_info);
}
if (!info->attrs[NLBL_MGMT_A_DOMAIN])
return -EINVAL;
- netlbl_netlink_auditinfo(skb, &audit_info);
+ netlbl_netlink_auditinfo(&audit_info);
domain = nla_data(info->attrs[NLBL_MGMT_A_DOMAIN]);
return netlbl_domhsh_remove(domain, AF_UNSPEC, &audit_info);
(info->attrs[NLBL_MGMT_A_IPV6MASK] != NULL)))
return -EINVAL;
- netlbl_netlink_auditinfo(skb, &audit_info);
+ netlbl_netlink_auditinfo(&audit_info);
return netlbl_mgmt_add_common(info, &audit_info);
}
{
struct netlbl_audit audit_info;
- netlbl_netlink_auditinfo(skb, &audit_info);
+ netlbl_netlink_auditinfo(&audit_info);
return netlbl_domhsh_remove_default(AF_UNSPEC, &audit_info);
}
if (info->attrs[NLBL_UNLABEL_A_ACPTFLG]) {
value = nla_get_u8(info->attrs[NLBL_UNLABEL_A_ACPTFLG]);
if (value == 1 || value == 0) {
- netlbl_netlink_auditinfo(skb, &audit_info);
+ netlbl_netlink_auditinfo(&audit_info);
netlbl_unlabel_acceptflg_set(value, &audit_info);
return 0;
}
!info->attrs[NLBL_UNLABEL_A_IPV6MASK])))
return -EINVAL;
- netlbl_netlink_auditinfo(skb, &audit_info);
+ netlbl_netlink_auditinfo(&audit_info);
ret_val = netlbl_unlabel_addrinfo_get(info, &addr, &mask, &addr_len);
if (ret_val != 0)
!info->attrs[NLBL_UNLABEL_A_IPV6MASK])))
return -EINVAL;
- netlbl_netlink_auditinfo(skb, &audit_info);
+ netlbl_netlink_auditinfo(&audit_info);
ret_val = netlbl_unlabel_addrinfo_get(info, &addr, &mask, &addr_len);
if (ret_val != 0)
!info->attrs[NLBL_UNLABEL_A_IPV6MASK])))
return -EINVAL;
- netlbl_netlink_auditinfo(skb, &audit_info);
+ netlbl_netlink_auditinfo(&audit_info);
ret_val = netlbl_unlabel_addrinfo_get(info, &addr, &mask, &addr_len);
if (ret_val != 0)
!info->attrs[NLBL_UNLABEL_A_IPV6MASK])))
return -EINVAL;
- netlbl_netlink_auditinfo(skb, &audit_info);
+ netlbl_netlink_auditinfo(&audit_info);
ret_val = netlbl_unlabel_addrinfo_get(info, &addr, &mask, &addr_len);
if (ret_val != 0)
/**
* netlbl_netlink_auditinfo - Fetch the audit information from a NETLINK msg
- * @skb: the packet
* @audit_info: NetLabel audit information
*/
-static inline void netlbl_netlink_auditinfo(struct sk_buff *skb,
- struct netlbl_audit *audit_info)
+static inline void netlbl_netlink_auditinfo(struct netlbl_audit *audit_info)
{
security_task_getsecid_subj(current, &audit_info->secid);
audit_info->loginuid = audit_get_loginuid(current);
static inline void
netlink_lock_table(void)
{
+ unsigned long flags;
+
/* read_lock() synchronizes us to netlink_table_grab */
- read_lock(&nl_table_lock);
+ read_lock_irqsave(&nl_table_lock, flags);
atomic_inc(&nl_table_users);
- read_unlock(&nl_table_lock);
+ read_unlock_irqrestore(&nl_table_lock, flags);
}
static inline void
* HCI command execution completion callback.
* err will be a standard linux error (may be converted from HCI response)
* skb contains the response data and must be disposed, or may be NULL if
- * an error occured
+ * an error occurred
*/
static void nfc_hci_execute_cb(void *context, struct sk_buff *skb, int err)
{
/*
* TODO: Check RF Error indicator to make sure data is valid.
* It seems that HCI cmd can complete without error, but data
- * can be invalid if an RF error occured? Ignore for now.
+ * can be invalid if an RF error occurred? Ignore for now.
*/
if (err == 0)
skb_trim(skb, skb->len - 1); /* RF Err ind */
case SHDLC_NEGOTIATING:
case SHDLC_CONNECTING:
/*
- * We sent RSET, but chip wants to negociate or we
+ * We sent RSET, but chip wants to negotiate or we
* got RSET before we managed to send out our.
*/
if (skb->len > 0)
if (!llcp_sock->service_name) {
nfc_llcp_local_put(llcp_sock->local);
llcp_sock->local = NULL;
+ llcp_sock->dev = NULL;
ret = -ENOMEM;
goto put_dev;
}
llcp_sock->local = NULL;
kfree(llcp_sock->service_name);
llcp_sock->service_name = NULL;
+ llcp_sock->dev = NULL;
ret = -EADDRINUSE;
goto put_dev;
}
void nci_free_device(struct nci_dev *ndev)
{
nfc_free_device(ndev->nfc_dev);
+ nci_hci_deallocate(ndev);
kfree(ndev);
}
EXPORT_SYMBOL(nci_free_device);
*(u8 *)skb_push(skb, 1) = data_type;
do {
- len = conn_info->max_pkt_payload_len;
-
/* If last packet add NCI_HFP_NO_CHAINING */
if (i + conn_info->max_pkt_payload_len -
(skb->len + 1) >= data_len) {
return hdev;
}
+
+void nci_hci_deallocate(struct nci_dev *ndev)
+{
+ kfree(ndev->hci_dev);
+}
return -ESOCKTNOSUPPORT;
if (sock->type == SOCK_RAW) {
- if (!capable(CAP_NET_RAW))
+ if (!ns_capable(net->user_ns, CAP_NET_RAW))
return -EPERM;
sock->ops = &rawsock_raw_ops;
} else {
flow_netlink.o \
flow_table.o \
meter.o \
+ openvswitch_trace.o \
vport.o \
vport-internal_dev.o \
vport-netdev.o
obj-$(CONFIG_OPENVSWITCH_VXLAN)+= vport-vxlan.o
obj-$(CONFIG_OPENVSWITCH_GENEVE)+= vport-geneve.o
obj-$(CONFIG_OPENVSWITCH_GRE) += vport-gre.o
+
+CFLAGS_openvswitch_trace.o = -I$(src)
#include "conntrack.h"
#include "vport.h"
#include "flow_netlink.h"
+#include "openvswitch_trace.h"
struct deferred_action {
struct sk_buff *skb;
a = nla_next(a, &rem)) {
int err = 0;
+ if (trace_ovs_do_execute_action_enabled())
+ trace_ovs_do_execute_action(dp, skb, key, a, rem);
+
switch (nla_type(a)) {
case OVS_ACTION_ATTR_OUTPUT: {
int port = nla_get_u32(a);
#include "flow_table.h"
#include "flow_netlink.h"
#include "meter.h"
+#include "openvswitch_trace.h"
#include "vport-internal_dev.h"
#include "vport-netdev.h"
struct dp_stats_percpu *stats;
int err;
+ if (trace_ovs_dp_upcall_enabled())
+ trace_ovs_dp_upcall(dp, skb, key, upcall_info);
+
if (upcall_info->portid == 0) {
err = -ENOTCONN;
goto err;
spin_lock(&meter->lock);
long_delta_ms = (now_ms - meter->used); /* ms */
+ if (long_delta_ms < 0) {
+ /* This condition means that we have several threads fighting
+ * for a meter lock, and the one who received the packets a
+ * bit later wins. Assuming that all racing threads received
+ * packets at the same time to avoid overflow.
+ */
+ long_delta_ms = 0;
+ }
/* Make sure delta_ms will not be too large, so that bucket will not
* wrap around below.
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* bug in tracepoint.h, it should include this */
+#include <linux/module.h>
+
+/* sparse isn't too happy with all macros... */
+#ifndef __CHECKER__
+#define CREATE_TRACE_POINTS
+#include "openvswitch_trace.h"
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM openvswitch
+
+#if !defined(_TRACE_OPENVSWITCH_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_OPENVSWITCH_H
+
+#include <linux/tracepoint.h>
+
+#include "datapath.h"
+
+TRACE_EVENT(ovs_do_execute_action,
+
+ TP_PROTO(struct datapath *dp, struct sk_buff *skb,
+ struct sw_flow_key *key, const struct nlattr *a, int rem),
+
+ TP_ARGS(dp, skb, key, a, rem),
+
+ TP_STRUCT__entry(
+ __field( void *, dpaddr )
+ __string( dp_name, ovs_dp_name(dp) )
+ __string( dev_name, skb->dev->name )
+ __field( void *, skbaddr )
+ __field( unsigned int, len )
+ __field( unsigned int, data_len )
+ __field( unsigned int, truesize )
+ __field( u8, nr_frags )
+ __field( u16, gso_size )
+ __field( u16, gso_type )
+ __field( u32, ovs_flow_hash )
+ __field( u32, recirc_id )
+ __field( void *, keyaddr )
+ __field( u16, key_eth_type )
+ __field( u8, key_ct_state )
+ __field( u8, key_ct_orig_proto )
+ __field( u16, key_ct_zone )
+ __field( unsigned int, flow_key_valid )
+ __field( u8, action_type )
+ __field( unsigned int, action_len )
+ __field( void *, action_data )
+ __field( u8, is_last )
+ ),
+
+ TP_fast_assign(
+ __entry->dpaddr = dp;
+ __assign_str(dp_name, ovs_dp_name(dp));
+ __assign_str(dev_name, skb->dev->name);
+ __entry->skbaddr = skb;
+ __entry->len = skb->len;
+ __entry->data_len = skb->data_len;
+ __entry->truesize = skb->truesize;
+ __entry->nr_frags = skb_shinfo(skb)->nr_frags;
+ __entry->gso_size = skb_shinfo(skb)->gso_size;
+ __entry->gso_type = skb_shinfo(skb)->gso_type;
+ __entry->ovs_flow_hash = key->ovs_flow_hash;
+ __entry->recirc_id = key->recirc_id;
+ __entry->keyaddr = key;
+ __entry->key_eth_type = key->eth.type;
+ __entry->key_ct_state = key->ct_state;
+ __entry->key_ct_orig_proto = key->ct_orig_proto;
+ __entry->key_ct_zone = key->ct_zone;
+ __entry->flow_key_valid = !(key->mac_proto & SW_FLOW_KEY_INVALID);
+ __entry->action_type = nla_type(a);
+ __entry->action_len = nla_len(a);
+ __entry->action_data = nla_data(a);
+ __entry->is_last = nla_is_last(a, rem);
+ ),
+
+ TP_printk("dpaddr=%p dp_name=%s dev=%s skbaddr=%p len=%u data_len=%u truesize=%u nr_frags=%d gso_size=%d gso_type=%#x ovs_flow_hash=0x%08x recirc_id=0x%08x keyaddr=%p eth_type=0x%04x ct_state=%02x ct_orig_proto=%02x ct_Zone=%04x flow_key_valid=%d action_type=%u action_len=%u action_data=%p is_last=%d",
+ __entry->dpaddr, __get_str(dp_name), __get_str(dev_name),
+ __entry->skbaddr, __entry->len, __entry->data_len,
+ __entry->truesize, __entry->nr_frags, __entry->gso_size,
+ __entry->gso_type, __entry->ovs_flow_hash,
+ __entry->recirc_id, __entry->keyaddr, __entry->key_eth_type,
+ __entry->key_ct_state, __entry->key_ct_orig_proto,
+ __entry->key_ct_zone,
+ __entry->flow_key_valid,
+ __entry->action_type, __entry->action_len,
+ __entry->action_data, __entry->is_last)
+);
+
+TRACE_EVENT(ovs_dp_upcall,
+
+ TP_PROTO(struct datapath *dp, struct sk_buff *skb,
+ const struct sw_flow_key *key,
+ const struct dp_upcall_info *upcall_info),
+
+ TP_ARGS(dp, skb, key, upcall_info),
+
+ TP_STRUCT__entry(
+ __field( void *, dpaddr )
+ __string( dp_name, ovs_dp_name(dp) )
+ __string( dev_name, skb->dev->name )
+ __field( void *, skbaddr )
+ __field( unsigned int, len )
+ __field( unsigned int, data_len )
+ __field( unsigned int, truesize )
+ __field( u8, nr_frags )
+ __field( u16, gso_size )
+ __field( u16, gso_type )
+ __field( u32, ovs_flow_hash )
+ __field( u32, recirc_id )
+ __field( const void *, keyaddr )
+ __field( u16, key_eth_type )
+ __field( u8, key_ct_state )
+ __field( u8, key_ct_orig_proto )
+ __field( u16, key_ct_zone )
+ __field( unsigned int, flow_key_valid )
+ __field( u8, upcall_cmd )
+ __field( u32, upcall_port )
+ __field( u16, upcall_mru )
+ ),
+
+ TP_fast_assign(
+ __entry->dpaddr = dp;
+ __assign_str(dp_name, ovs_dp_name(dp));
+ __assign_str(dev_name, skb->dev->name);
+ __entry->skbaddr = skb;
+ __entry->len = skb->len;
+ __entry->data_len = skb->data_len;
+ __entry->truesize = skb->truesize;
+ __entry->nr_frags = skb_shinfo(skb)->nr_frags;
+ __entry->gso_size = skb_shinfo(skb)->gso_size;
+ __entry->gso_type = skb_shinfo(skb)->gso_type;
+ __entry->ovs_flow_hash = key->ovs_flow_hash;
+ __entry->recirc_id = key->recirc_id;
+ __entry->keyaddr = key;
+ __entry->key_eth_type = key->eth.type;
+ __entry->key_ct_state = key->ct_state;
+ __entry->key_ct_orig_proto = key->ct_orig_proto;
+ __entry->key_ct_zone = key->ct_zone;
+ __entry->flow_key_valid = !(key->mac_proto & SW_FLOW_KEY_INVALID);
+ __entry->upcall_cmd = upcall_info->cmd;
+ __entry->upcall_port = upcall_info->portid;
+ __entry->upcall_mru = upcall_info->mru;
+ ),
+
+ TP_printk("dpaddr=%p dp_name=%s dev=%s skbaddr=%p len=%u data_len=%u truesize=%u nr_frags=%d gso_size=%d gso_type=%#x ovs_flow_hash=0x%08x recirc_id=0x%08x keyaddr=%p eth_type=0x%04x ct_state=%02x ct_orig_proto=%02x ct_zone=%04x flow_key_valid=%d upcall_cmd=%u upcall_port=%u upcall_mru=%u",
+ __entry->dpaddr, __get_str(dp_name), __get_str(dev_name),
+ __entry->skbaddr, __entry->len, __entry->data_len,
+ __entry->truesize, __entry->nr_frags, __entry->gso_size,
+ __entry->gso_type, __entry->ovs_flow_hash,
+ __entry->recirc_id, __entry->keyaddr, __entry->key_eth_type,
+ __entry->key_ct_state, __entry->key_ct_orig_proto,
+ __entry->key_ct_zone,
+ __entry->flow_key_valid,
+ __entry->upcall_cmd, __entry->upcall_port,
+ __entry->upcall_mru)
+);
+
+#endif /* _TRACE_OPENVSWITCH_H */
+
+/* This part must be outside protection */
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE openvswitch_trace
+#include <trace/define_trace.h>
ktime_to_timespec64_cond(shhwtstamps->hwtstamp, ts))
return TP_STATUS_TS_RAW_HARDWARE;
- if (ktime_to_timespec64_cond(skb->tstamp, ts))
+ if ((flags & SOF_TIMESTAMPING_SOFTWARE) &&
+ ktime_to_timespec64_cond(skb->tstamp, ts))
return TP_STATUS_TS_SOFTWARE;
return 0;
skb_copy_bits(skb, 0, h.raw + macoff, snaplen);
- if (!(ts_status = tpacket_get_timestamp(skb, &ts, po->tp_tstamp)))
+ /* Always timestamp; prefer an existing software timestamp taken
+ * closer to the time of capture.
+ */
+ ts_status = tpacket_get_timestamp(skb, &ts,
+ po->tp_tstamp | SOF_TIMESTAMPING_SOFTWARE);
+ if (!ts_status)
ktime_get_real_ts64(&ts);
status |= ts_status;
}
if (likely(saddr == NULL)) {
dev = packet_cached_dev_get(po);
- proto = po->num;
+ proto = READ_ONCE(po->num);
} else {
err = -EINVAL;
if (msg->msg_namelen < sizeof(struct sockaddr_ll))
if (likely(saddr == NULL)) {
dev = packet_cached_dev_get(po);
- proto = po->num;
+ proto = READ_ONCE(po->num);
} else {
err = -EINVAL;
if (msg->msg_namelen < sizeof(struct sockaddr_ll))
struct sock *sk = sock->sk;
struct packet_sock *po = pkt_sk(sk);
- if (po->tx_ring.pg_vec)
+ /* Reading tx_ring.pg_vec without holding pg_vec_lock is racy.
+ * tpacket_snd() will redo the check safely.
+ */
+ if (data_race(po->tx_ring.pg_vec))
return tpacket_snd(po, msg);
- else
- return packet_snd(sock, msg, len);
+
+ return packet_snd(sock, msg, len);
}
/*
/* prevents packet_notifier() from calling
* register_prot_hook()
*/
- po->num = 0;
+ WRITE_ONCE(po->num, 0);
__unregister_prot_hook(sk, true);
rcu_read_lock();
dev_curr = po->prot_hook.dev;
}
BUG_ON(po->running);
- po->num = proto;
+ WRITE_ONCE(po->num, proto);
po->prot_hook.type = proto;
if (unlikely(unlisted)) {
dev_put(dev);
po->prot_hook.dev = NULL;
- po->ifindex = -1;
+ WRITE_ONCE(po->ifindex, -1);
packet_cached_dev_reset(po);
} else {
po->prot_hook.dev = dev;
- po->ifindex = dev ? dev->ifindex : 0;
+ WRITE_ONCE(po->ifindex, dev ? dev->ifindex : 0);
packet_cached_dev_assign(po, dev);
}
}
uaddr->sa_family = AF_PACKET;
memset(uaddr->sa_data, 0, sizeof(uaddr->sa_data));
rcu_read_lock();
- dev = dev_get_by_index_rcu(sock_net(sk), pkt_sk(sk)->ifindex);
+ dev = dev_get_by_index_rcu(sock_net(sk), READ_ONCE(pkt_sk(sk)->ifindex));
if (dev)
strlcpy(uaddr->sa_data, dev->name, sizeof(uaddr->sa_data));
rcu_read_unlock();
struct sock *sk = sock->sk;
struct packet_sock *po = pkt_sk(sk);
DECLARE_SOCKADDR(struct sockaddr_ll *, sll, uaddr);
+ int ifindex;
if (peer)
return -EOPNOTSUPP;
+ ifindex = READ_ONCE(po->ifindex);
sll->sll_family = AF_PACKET;
- sll->sll_ifindex = po->ifindex;
- sll->sll_protocol = po->num;
+ sll->sll_ifindex = ifindex;
+ sll->sll_protocol = READ_ONCE(po->num);
sll->sll_pkttype = 0;
rcu_read_lock();
- dev = dev_get_by_index_rcu(sock_net(sk), po->ifindex);
+ dev = dev_get_by_index_rcu(sock_net(sk), ifindex);
if (dev) {
sll->sll_hatype = dev->type;
sll->sll_halen = dev->addr_len;
return -EFAULT;
lock_sock(sk);
- if (po->rx_ring.pg_vec || po->tx_ring.pg_vec) {
- ret = -EBUSY;
- } else {
+ if (!po->rx_ring.pg_vec && !po->tx_ring.pg_vec)
po->tp_tx_has_off = !!val;
- ret = 0;
- }
+
release_sock(sk);
return 0;
}
}
if (msg == NETDEV_UNREGISTER) {
packet_cached_dev_reset(po);
- po->ifindex = -1;
+ WRITE_ONCE(po->ifindex, -1);
if (po->prot_hook.dev)
dev_put(po->prot_hook.dev);
po->prot_hook.dev = NULL;
was_running = po->running;
num = po->num;
if (was_running) {
- po->num = 0;
+ WRITE_ONCE(po->num, 0);
__unregister_prot_hook(sk, false);
}
spin_unlock(&po->bind_lock);
spin_lock(&po->bind_lock);
if (was_running) {
- po->num = num;
+ WRITE_ONCE(po->num, num);
register_prot_hook(sk);
}
spin_unlock(&po->bind_lock);
s,
refcount_read(&s->sk_refcnt),
s->sk_type,
- ntohs(po->num),
- po->ifindex,
+ ntohs(READ_ONCE(po->num)),
+ READ_ONCE(po->ifindex),
po->running,
atomic_read(&s->sk_rmem_alloc),
from_kuid_munged(seq_user_ns(seq), sock_i_uid(s)),
}
qrtr_ns.workqueue = alloc_workqueue("qrtr_ns_handler", WQ_UNBOUND, 1);
- if (!qrtr_ns.workqueue)
+ if (!qrtr_ns.workqueue) {
+ ret = -ENOMEM;
goto err_sock;
+ }
qrtr_ns.sock->sk->sk_data_ready = qrtr_ns_data_ready;
struct qrtr_sock *ipc;
struct sk_buff *skb;
struct qrtr_cb *cb;
- unsigned int size;
+ size_t size;
unsigned int ver;
size_t hdrlen;
if (loop_trans) {
rds_trans_put(loop_trans);
conn->c_loopback = 1;
- if (is_outgoing && trans->t_prefer_loopback) {
- /* "outgoing" connection - and the transport
- * says it wants the connection handled by the
- * loopback transport. This is what TCP does.
- */
- trans = &rds_loop_transport;
+ if (trans->t_prefer_loopback) {
+ if (likely(is_outgoing)) {
+ /* "outgoing" connection to local address.
+ * Protocol says it wants the connection
+ * handled by the loopback transport.
+ * This is what TCP does.
+ */
+ trans = &rds_loop_transport;
+ } else {
+ /* No transport currently in use
+ * should end up here, but if it
+ * does, reset/destroy the connection.
+ */
+ kmem_cache_free(rds_conn_slab, conn);
+ conn = ERR_PTR(-EOPNOTSUPP);
+ goto out;
+ }
}
}
}
/*
- * returns the oldest alloced ring entry. This will be the next one
+ * returns the oldest allocated ring entry. This will be the next one
* freed. This can't be called if there are none allocated.
*/
u32 rds_ib_ring_oldest(struct rds_ib_work_ring *ring)
if (rds_cmsg_recv(inc, msg, rs)) {
ret = -EFAULT;
- goto out;
+ break;
}
rds_recvmsg_zcookie(rs, msg);
}
#endif
-static int rds_tcp_laddr_check(struct net *net, const struct in6_addr *addr,
- __u32 scope_id)
+int rds_tcp_laddr_check(struct net *net, const struct in6_addr *addr,
+ __u32 scope_id)
{
struct net_device *dev = NULL;
#if IS_ENABLED(CONFIG_IPV6)
u64 rds_tcp_map_seq(struct rds_tcp_connection *tc, u32 seq);
extern struct rds_transport rds_tcp_transport;
void rds_tcp_accept_work(struct sock *sk);
-
+int rds_tcp_laddr_check(struct net *net, const struct in6_addr *addr,
+ __u32 scope_id);
/* tcp_connect.c */
int rds_tcp_conn_path_connect(struct rds_conn_path *cp);
void rds_tcp_conn_path_shutdown(struct rds_conn_path *conn);
}
#endif
+ if (!rds_tcp_laddr_check(sock_net(sock->sk), peer_addr, dev_if)) {
+ /* local address connection is only allowed via loopback */
+ ret = -EOPNOTSUPP;
+ goto out;
+ }
+
conn = rds_conn_create(sock_net(sock->sk),
my_addr, peer_addr,
&rds_tcp_transport, 0, GFP_KERNEL, dev_if);
goto out;
}
tc->t_tinc = tinc;
- rdsdebug("alloced tinc %p\n", tinc);
+ rdsdebug("allocated tinc %p\n", tinc);
rds_inc_path_init(&tinc->ti_inc, cp,
&cp->cp_conn->c_faddr);
tinc->ti_inc.i_rx_lat_trace[RDS_MSG_RX_HDR] =
}
/*
- * Process event packets targetted at a local endpoint.
+ * Process event packets targeted at a local endpoint.
*/
void rxrpc_process_local_events(struct rxrpc_local *local)
{
}
mutex_unlock(&idrinfo->lock);
- if (nla_put_u32(skb, TCA_FCNT, n_i))
+ ret = nla_put_u32(skb, TCA_FCNT, n_i);
+ if (ret)
goto nla_put_failure;
nla_nest_end(skb, nest);
}
err = ct_nat_execute(skb, ct, ctinfo, range, maniptype);
- if (err == NF_ACCEPT &&
- ct->status & IPS_SRC_NAT && ct->status & IPS_DST_NAT) {
- if (maniptype == NF_NAT_MANIP_SRC)
- maniptype = NF_NAT_MANIP_DST;
- else
- maniptype = NF_NAT_MANIP_SRC;
-
- err = ct_nat_execute(skb, ct, ctinfo, range, maniptype);
+ if (err == NF_ACCEPT && ct->status & IPS_DST_NAT) {
+ if (ct->status & IPS_SRC_NAT) {
+ if (maniptype == NF_NAT_MANIP_SRC)
+ maniptype = NF_NAT_MANIP_DST;
+ else
+ maniptype = NF_NAT_MANIP_SRC;
+
+ err = ct_nat_execute(skb, ct, ctinfo, range,
+ maniptype);
+ } else if (CTINFO2DIR(ctinfo) == IP_CT_DIR_ORIGINAL) {
+ err = ct_nat_execute(skb, ct, ctinfo, NULL,
+ NF_NAT_MANIP_SRC);
+ }
}
return err;
#else
*/
cached = tcf_ct_skb_nfct_cached(net, skb, p->zone, force);
if (!cached) {
- if (!commit && tcf_ct_flow_table_lookup(p, skb, family)) {
+ if (tcf_ct_flow_table_lookup(p, skb, family)) {
skip_add = true;
goto do_nat;
}
* even if the connection is already confirmed.
*/
nf_conntrack_confirm(skb);
- } else if (!skip_add) {
- tcf_ct_flow_table_process_conn(p->ct_ft, ct, ctinfo);
}
+ if (!skip_add)
+ tcf_ct_flow_table_process_conn(p->ct_ft, ct, ctinfo);
+
out_push:
skb_push_rcsum(skb, nh_ofs);
sizeof(p->zone));
}
- if (p->zone == NF_CT_DEFAULT_ZONE_ID)
- return 0;
-
nf_ct_zone_init(&zone, p->zone, NF_CT_DEFAULT_ZONE_DIR, 0);
tmpl = nf_ct_tmpl_alloc(net, &zone, GFP_KERNEL);
if (!tmpl) {
/* replace the vid */
tci = (tci & ~VLAN_VID_MASK) | p->tcfv_push_vid;
/* replace prio bits, if tcfv_push_prio specified */
- if (p->tcfv_push_prio) {
+ if (p->tcfv_push_prio_exists) {
tci &= ~VLAN_PRIO_MASK;
tci |= p->tcfv_push_prio << VLAN_PRIO_SHIFT;
}
struct tc_action_net *tn = net_generic(net, vlan_net_id);
struct nlattr *tb[TCA_VLAN_MAX + 1];
struct tcf_chain *goto_ch = NULL;
+ bool push_prio_exists = false;
struct tcf_vlan_params *p;
struct tc_vlan *parm;
struct tcf_vlan *v;
push_proto = htons(ETH_P_8021Q);
}
- if (tb[TCA_VLAN_PUSH_VLAN_PRIORITY])
+ push_prio_exists = !!tb[TCA_VLAN_PUSH_VLAN_PRIORITY];
+ if (push_prio_exists)
push_prio = nla_get_u8(tb[TCA_VLAN_PUSH_VLAN_PRIORITY]);
break;
case TCA_VLAN_ACT_POP_ETH:
p->tcfv_action = action;
p->tcfv_push_vid = push_vid;
p->tcfv_push_prio = push_prio;
+ p->tcfv_push_prio_exists = push_prio_exists || action == TCA_VLAN_ACT_PUSH;
p->tcfv_push_proto = push_proto;
if (action == TCA_VLAN_ACT_PUSH_ETH) {
(nla_put_u16(skb, TCA_VLAN_PUSH_VLAN_ID, p->tcfv_push_vid) ||
nla_put_be16(skb, TCA_VLAN_PUSH_VLAN_PROTOCOL,
p->tcfv_push_proto) ||
- (nla_put_u8(skb, TCA_VLAN_PUSH_VLAN_PRIORITY,
- p->tcfv_push_prio))))
+ (p->tcfv_push_prio_exists &&
+ nla_put_u8(skb, TCA_VLAN_PUSH_VLAN_PRIORITY, p->tcfv_push_prio))))
goto nla_put_failure;
if (p->tcfv_action == TCA_VLAN_ACT_PUSH_ETH) {
u32 *last_executed_chain)
{
#ifdef CONFIG_NET_CLS_ACT
- const int max_reclassify_loop = 4;
+ const int max_reclassify_loop = 16;
const struct tcf_proto *first_tp;
int limit = 0;
/* If we missed on some chain */
if (ret == TC_ACT_UNSPEC && last_executed_chain) {
- ext = skb_ext_add(skb, TC_SKB_EXT);
+ ext = tc_skb_ext_alloc(skb);
if (WARN_ON_ONCE(!ext))
return TC_ACT_SHOT;
ext->chain = last_executed_chain;
/*
Comparing to general packet classification problem,
- RSVP needs only sevaral relatively simple rules:
+ RSVP needs only several relatively simple rules:
* (dst, protocol) are always specified,
so that we are able to hash them.
EXPORT_SYMBOL(tcf_em_register);
/**
- * tcf_em_unregister - unregster and extended match
+ * tcf_em_unregister - unregister and extended match
*
* @ops: ematch operations lookup table
*
}
tcph = skb_header_pointer(skb, offset, sizeof(_tcph), &_tcph);
- if (!tcph)
+ if (!tcph || tcph->doff < 5)
return NULL;
return skb_header_pointer(skb, offset,
length--;
continue;
}
+ if (length < 2)
+ break;
opsize = *ptr++;
if (opsize < 2 || opsize > length)
break;
length--;
continue;
}
+ if (length < 2)
+ break;
opsize = *ptr++;
if (opsize < 2 || opsize > length)
break;
/* List of known Diffserv codepoints:
*
- * Least Effort (CS1)
+ * Least Effort (CS1, LE)
* Best Effort (CS0)
* Max Reliability & LLT "Lo" (TOS1)
* Max Throughput (TOS2)
* Total 25 codepoints.
*/
-/* List of traffic classes in RFC 4594:
+/* List of traffic classes in RFC 4594, updated by RFC 8622:
* (roughly descending order of contended priority)
* (roughly ascending order of uncontended throughput)
*
* Ops, Admin, Management (CS2,TOS1) - eg. ssh
* Standard Service (CS0 & unrecognised codepoints)
* High Throughput Data (AF1x,TOS2) - eg. web traffic
- * Low Priority Data (CS1) - eg. BitTorrent
+ * Low Priority Data (CS1,LE) - eg. BitTorrent
* Total 12 traffic classes.
*/
* Video Streaming (AF4x, AF3x, CS3)
* Bog Standard (CS0 etc.)
* High Throughput (AF1x, TOS2)
- * Background Traffic (CS1)
+ * Background Traffic (CS1, LE)
*
* Total 8 traffic classes.
*/
* Latency Sensitive (CS7, CS6, EF, VA, CS5, CS4)
* Streaming Media (AF4x, AF3x, CS3, AF2x, TOS4, CS2, TOS1)
* Best Effort (CS0, AF1x, TOS2, and those not specified)
- * Background Traffic (CS1)
+ * Background Traffic (CS1, LE)
*
* Total 4 traffic classes.
*/
static int cake_config_diffserv3(struct Qdisc *sch)
{
/* Simplified Diffserv structure with 3 tins.
- * Low Priority (CS1)
+ * Low Priority (CS1, LE)
* Best Effort
* Latency Sensitive (TOS4, VA, EF, CS6, CS7)
*/
struct dsmark_qdisc_data *p = qdisc_priv(sch);
pr_debug("%s(sch %p,[qdisc %p])\n", __func__, sch, p);
- qdisc_reset(p->q);
+ if (p->q)
+ qdisc_reset(p->q);
sch->qstats.backlog = 0;
sch->q.qlen = 0;
}
/* Classifies packet into corresponding flow */
idx = fq_pie_classify(skb, sch, &ret);
- sel_flow = &q->flows[idx];
+ if (idx == 0) {
+ if (ret & __NET_XMIT_BYPASS)
+ qdisc_qstats_drop(sch);
+ __qdisc_drop(skb, to_free);
+ return ret;
+ }
+ idx--;
+ sel_flow = &q->flows[idx];
/* Checks whether adding a new packet would exceed memory limit */
get_pie_cb(skb)->mem_usage = skb->truesize;
memory_limited = q->memory_usage > q->memory_limit + skb->truesize;
goto flow_error;
}
q->flows_cnt = nla_get_u32(tb[TCA_FQ_PIE_FLOWS]);
- if (!q->flows_cnt || q->flows_cnt >= 65536) {
+ if (!q->flows_cnt || q->flows_cnt > 65536) {
NL_SET_ERR_MSG_MOD(extack,
- "Number of flows must range in [1..65535]");
+ "Number of flows must range in [1..65536]");
goto flow_error;
}
}
struct fq_pie_sched_data *q = from_timer(q, t, adapt_timer);
struct Qdisc *sch = q->sch;
spinlock_t *root_lock; /* to lock qdisc for probability calculations */
- u16 idx;
+ u32 idx;
root_lock = qdisc_lock(qdisc_root_sleeping(sch));
spin_lock(root_lock);
{
struct fq_pie_sched_data *q = qdisc_priv(sch);
int err;
- u16 idx;
+ u32 idx;
pie_params_init(&q->p_params);
sch->limit = 10 * 1024;
static void fq_pie_reset(struct Qdisc *sch)
{
struct fq_pie_sched_data *q = qdisc_priv(sch);
- u16 idx;
+ u32 idx;
INIT_LIST_HEAD(&q->new_flows);
INIT_LIST_HEAD(&q->old_flows);
const struct Qdisc_ops *default_qdisc_ops = &pfifo_fast_ops;
EXPORT_SYMBOL(default_qdisc_ops);
+static void qdisc_maybe_clear_missed(struct Qdisc *q,
+ const struct netdev_queue *txq)
+{
+ clear_bit(__QDISC_STATE_MISSED, &q->state);
+
+ /* Make sure the below netif_xmit_frozen_or_stopped()
+ * checking happens after clearing STATE_MISSED.
+ */
+ smp_mb__after_atomic();
+
+ /* Checking netif_xmit_frozen_or_stopped() again to
+ * make sure STATE_MISSED is set if the STATE_MISSED
+ * set by netif_tx_wake_queue()'s rescheduling of
+ * net_tx_action() is cleared by the above clear_bit().
+ */
+ if (!netif_xmit_frozen_or_stopped(txq))
+ set_bit(__QDISC_STATE_MISSED, &q->state);
+ else
+ set_bit(__QDISC_STATE_DRAINING, &q->state);
+}
+
/* Main transmission queue. */
/* Modifications to data participating in scheduling must be protected with
}
} else {
skb = SKB_XOFF_MAGIC;
+ qdisc_maybe_clear_missed(q, txq);
}
}
skb = next;
}
- if (lock)
+
+ if (lock) {
spin_unlock(lock);
- __netif_schedule(q);
+ set_bit(__QDISC_STATE_MISSED, &q->state);
+ } else {
+ __netif_schedule(q);
+ }
}
static void try_bulk_dequeue_skb(struct Qdisc *q,
}
} else {
skb = NULL;
+ qdisc_maybe_clear_missed(q, txq);
}
if (lock)
spin_unlock(lock);
*validate = true;
if ((q->flags & TCQ_F_ONETXQUEUE) &&
- netif_xmit_frozen_or_stopped(txq))
+ netif_xmit_frozen_or_stopped(txq)) {
+ qdisc_maybe_clear_missed(q, txq);
return skb;
+ }
skb = qdisc_dequeue_skb_bad_txq(q);
if (unlikely(skb)) {
HARD_TX_LOCK(dev, txq, smp_processor_id());
if (!netif_xmit_frozen_or_stopped(txq))
skb = dev_hard_start_xmit(skb, dev, txq, &ret);
+ else
+ qdisc_maybe_clear_missed(q, txq);
HARD_TX_UNLOCK(dev, txq);
} else {
while (qdisc_restart(q, &packets)) {
quota -= packets;
if (quota <= 0) {
- __netif_schedule(q);
+ if (q->flags & TCQ_F_NOLOCK)
+ set_bit(__QDISC_STATE_MISSED, &q->state);
+ else
+ __netif_schedule(q);
+
break;
}
}
}
EXPORT_SYMBOL(netif_carrier_off);
+/**
+ * netif_carrier_event - report carrier state event
+ * @dev: network device
+ *
+ * Device has detected a carrier event but the carrier state wasn't changed.
+ * Use in drivers when querying carrier state asynchronously, to avoid missing
+ * events (link flaps) if link recovers before it's queried.
+ */
+void netif_carrier_event(struct net_device *dev)
+{
+ if (dev->reg_state == NETREG_UNINITIALIZED)
+ return;
+ atomic_inc(&dev->carrier_up_count);
+ atomic_inc(&dev->carrier_down_count);
+ linkwatch_fire_event(dev);
+}
+EXPORT_SYMBOL_GPL(netif_carrier_event);
+
/* "NOOP" scheduler: the best scheduler, recommended for all interfaces
under all circumstances. It is difficult to invent anything faster or
cheaper.
{
struct pfifo_fast_priv *priv = qdisc_priv(qdisc);
struct sk_buff *skb = NULL;
+ bool need_retry = true;
int band;
+retry:
for (band = 0; band < PFIFO_FAST_BANDS && !skb; band++) {
struct skb_array *q = band2list(priv, band);
}
if (likely(skb)) {
qdisc_update_stats_at_dequeue(qdisc, skb);
- } else {
- WRITE_ONCE(qdisc->empty, true);
+ } else if (need_retry &&
+ READ_ONCE(qdisc->state) & QDISC_STATE_NON_EMPTY) {
+ /* Delay clearing the STATE_MISSED here to reduce
+ * the overhead of the second spin_trylock() in
+ * qdisc_run_begin() and __netif_schedule() calling
+ * in qdisc_run_end().
+ */
+ clear_bit(__QDISC_STATE_MISSED, &qdisc->state);
+ clear_bit(__QDISC_STATE_DRAINING, &qdisc->state);
+
+ /* Make sure dequeuing happens after clearing
+ * STATE_MISSED.
+ */
+ smp_mb__after_atomic();
+
+ need_retry = false;
+
+ goto retry;
}
return skb;
sch->enqueue = ops->enqueue;
sch->dequeue = ops->dequeue;
sch->dev_queue = dev_queue;
- sch->empty = true;
dev_hold(dev);
refcount_set(&sch->refcnt, 1);
qdisc_reset(qdisc);
spin_unlock_bh(qdisc_lock(qdisc));
- if (nolock)
+ if (nolock) {
+ clear_bit(__QDISC_STATE_MISSED, &qdisc->state);
+ clear_bit(__QDISC_STATE_DRAINING, &qdisc->state);
spin_unlock_bh(&qdisc->seqlock);
+ }
}
static bool some_qdisc_is_busy(struct net_device *dev)
*
* 991129: - Bug fix with grio mode
* - a better sing. AvgQ mode with Grio(WRED)
- * - A finer grained VQ dequeue based on sugestion
+ * - A finer grained VQ dequeue based on suggestion
* from Ren Liu
* - More error checks
*
*/
static int htb_hysteresis __read_mostly = 0; /* whether to use mode hysteresis for speedup */
-#define HTB_VER 0x30011 /* major must be matched with number suplied by TC as version */
+#define HTB_VER 0x30011 /* major must be matched with number supplied by TC as version */
#if HTB_VER >> 16 != TC_HTB_PROTOVER
#error "Mismatched sch_htb.c and pkt_sch.h"
/**
* htb_add_to_id_tree - adds class to the round robin list
+ * @root: the root of the tree
+ * @cl: the class to add
+ * @prio: the give prio in class
*
* Routine adds class to the list (actually tree) sorted by classid.
* Make sure that class is not already on such list for given prio.
/**
* htb_add_to_wait_tree - adds class to the event queue with delay
+ * @q: the priority event queue
+ * @cl: the class to add
+ * @delay: delay in microseconds
*
* The class is added to priority event queue to indicate that class will
* change its mode in cl->pq_key microseconds. Make sure that class is not
/**
* htb_next_rb_node - finds next node in binary tree
+ * @n: the current node in binary tree
*
* When we are past last key we return NULL.
* Average complexity is 2 steps per call.
/**
* htb_add_class_to_row - add class to its row
+ * @q: the priority event queue
+ * @cl: the class to add
+ * @mask: the given priorities in class in bitmap
*
* The class is added to row at priorities marked in mask.
* It does nothing if mask == 0.
/**
* htb_remove_class_from_row - removes class from its row
+ * @q: the priority event queue
+ * @cl: the class to add
+ * @mask: the given priorities in class in bitmap
*
* The class is removed from row at priorities marked in mask.
* It does nothing if mask == 0.
/**
* htb_activate_prios - creates active classe's feed chain
+ * @q: the priority event queue
+ * @cl: the class to activate
*
* The class is connected to ancestors and/or appropriate rows
* for priorities it is participating on. cl->cmode must be new
/**
* htb_deactivate_prios - remove class from feed chain
+ * @q: the priority event queue
+ * @cl: the class to deactivate
*
* cl->cmode must represent old mode (before deactivation). It does
* nothing if cl->prio_activity == 0. Class is removed from all feed
/**
* htb_class_mode - computes and returns current class mode
+ * @cl: the target class
+ * @diff: diff time in microseconds
*
* It computes cl's mode at time cl->t_c+diff and returns it. If mode
* is not HTB_CAN_SEND then cl->pq_key is updated to time difference
/**
* htb_change_class_mode - changes classe's mode
+ * @q: the priority event queue
+ * @cl: the target class
+ * @diff: diff time in microseconds
*
* This should be the only way how to change classe's mode under normal
- * cirsumstances. Routine will update feed lists linkage, change mode
+ * circumstances. Routine will update feed lists linkage, change mode
* and add class to the wait event queue if appropriate. New mode should
* be different from old one and cl->pq_key has to be valid if changing
* to mode other than HTB_CAN_SEND (see htb_add_to_wait_tree).
/**
* htb_activate - inserts leaf cl into appropriate active feeds
+ * @q: the priority event queue
+ * @cl: the target class
*
* Routine learns (new) priority of leaf and activates feed chain
* for the prio. It can be called on already active leaf safely.
/**
* htb_deactivate - remove leaf cl from active feeds
+ * @q: the priority event queue
+ * @cl: the target class
*
* Make sure that leaf is active. In the other words it can't be called
* with non-active leaf. It also removes class from the drop list.
/**
* htb_charge_class - charges amount "bytes" to leaf and ancestors
+ * @q: the priority event queue
+ * @cl: the class to start iterate
+ * @level: the minimum level to account
+ * @skb: the socket buffer
*
* Routine assumes that packet "bytes" long was dequeued from leaf cl
* borrowing from "level". It accounts bytes to ceil leaky bucket for
/**
* htb_do_events - make mode changes to classes at the level
+ * @q: the priority event queue
+ * @level: which wait_pq in 'q->hlevel'
+ * @start: start jiffies
*
* Scans event queue for pending events and applies them. Returns time of
* next pending event (0 for no event in pq, q->now for too many events).
/**
* htb_lookup_leaf - returns next leaf class in DRR order
+ * @hprio: the current one
+ * @prio: which prio in class
*
* Find leaf where current feed pointers points to.
*/
struct Qdisc *old_q;
/* One ref for cl->leaf.q, the other for dev_queue->qdisc. */
- qdisc_refcount_inc(new_q);
+ if (new_q)
+ qdisc_refcount_inc(new_q);
old_q = htb_graft_helper(dev_queue, new_q);
WARN_ON(!(old_q->flags & TCQ_F_BUILTIN));
}
cl->parent->common.classid,
NULL);
if (q->offload) {
- if (new_q) {
+ if (new_q)
htb_set_lockdep_class_child(new_q);
- htb_parent_to_leaf_offload(sch, dev_queue, new_q);
- }
+ htb_parent_to_leaf_offload(sch, dev_queue, new_q);
}
}
* sock configured value.
*/
asoc->hbinterval = msecs_to_jiffies(sp->hbinterval);
+ asoc->probe_interval = msecs_to_jiffies(sp->probe_interval);
asoc->encap_port = sp->encap_port;
* association configured value.
*/
peer->hbinterval = asoc->hbinterval;
+ peer->probe_interval = asoc->probe_interval;
peer->encap_port = asoc->encap_port;
return NULL;
}
+ sctp_transport_pl_reset(peer);
+
/* Attach the remote transport to our asoc. */
list_add_tail_rcu(&peer->transports, &asoc->peer.transport_addr_list);
asoc->peer.transport_count++;
spc_state = SCTP_ADDR_CONFIRMED;
transport->state = SCTP_ACTIVE;
+ sctp_transport_pl_reset(transport);
break;
case SCTP_TRANSPORT_DOWN:
*/
if (transport->state != SCTP_UNCONFIRMED) {
transport->state = SCTP_INACTIVE;
+ sctp_transport_pl_reset(transport);
spc_state = SCTP_ADDR_UNREACHABLE;
} else {
sctp_transport_dst_release(transport);
"TIMEOUT_T5_SHUTDOWN_GUARD",
"TIMEOUT_HEARTBEAT",
"TIMEOUT_RECONF",
+ "TIMEOUT_PROBE",
"TIMEOUT_SACK",
"TIMEOUT_AUTOCLOSE",
};
void sctp_icmp_frag_needed(struct sock *sk, struct sctp_association *asoc,
struct sctp_transport *t, __u32 pmtu)
{
- if (!t || (t->pathmtu <= pmtu))
+ if (!t ||
+ (t->pathmtu <= pmtu &&
+ t->pl.probe_size + sctp_transport_pl_hlen(t) <= pmtu))
return;
if (sock_owned_by_user(sk)) {
sctp_transport_put(t);
}
+static void sctp_v4_err_handle(struct sctp_transport *t, struct sk_buff *skb,
+ __u8 type, __u8 code, __u32 info)
+{
+ struct sctp_association *asoc = t->asoc;
+ struct sock *sk = asoc->base.sk;
+ int err = 0;
+
+ switch (type) {
+ case ICMP_PARAMETERPROB:
+ err = EPROTO;
+ break;
+ case ICMP_DEST_UNREACH:
+ if (code > NR_ICMP_UNREACH)
+ return;
+ if (code == ICMP_FRAG_NEEDED) {
+ sctp_icmp_frag_needed(sk, asoc, t, SCTP_TRUNC4(info));
+ return;
+ }
+ if (code == ICMP_PROT_UNREACH) {
+ sctp_icmp_proto_unreachable(sk, asoc, t);
+ return;
+ }
+ err = icmp_err_convert[code].errno;
+ break;
+ case ICMP_TIME_EXCEEDED:
+ if (code == ICMP_EXC_FRAGTIME)
+ return;
+
+ err = EHOSTUNREACH;
+ break;
+ case ICMP_REDIRECT:
+ sctp_icmp_redirect(sk, t, skb);
+ default:
+ return;
+ }
+ if (!sock_owned_by_user(sk) && inet_sk(sk)->recverr) {
+ sk->sk_err = err;
+ sk->sk_error_report(sk);
+ } else { /* Only an error on timeout */
+ sk->sk_err_soft = err;
+ }
+}
+
/*
* This routine is called by the ICMP module when it gets some
* sort of error condition. If err < 0 then the socket should
int sctp_v4_err(struct sk_buff *skb, __u32 info)
{
const struct iphdr *iph = (const struct iphdr *)skb->data;
- const int ihlen = iph->ihl * 4;
const int type = icmp_hdr(skb)->type;
const int code = icmp_hdr(skb)->code;
- struct sock *sk;
- struct sctp_association *asoc = NULL;
+ struct net *net = dev_net(skb->dev);
struct sctp_transport *transport;
- struct inet_sock *inet;
+ struct sctp_association *asoc;
__u16 saveip, savesctp;
- int err;
- struct net *net = dev_net(skb->dev);
+ struct sock *sk;
/* Fix up skb to look at the embedded net header. */
saveip = skb->network_header;
savesctp = skb->transport_header;
skb_reset_network_header(skb);
- skb_set_transport_header(skb, ihlen);
+ skb_set_transport_header(skb, iph->ihl * 4);
sk = sctp_err_lookup(net, AF_INET, skb, sctp_hdr(skb), &asoc, &transport);
/* Put back, the original values. */
skb->network_header = saveip;
__ICMP_INC_STATS(net, ICMP_MIB_INERRORS);
return -ENOENT;
}
- /* Warning: The sock lock is held. Remember to call
- * sctp_err_finish!
- */
- switch (type) {
- case ICMP_PARAMETERPROB:
- err = EPROTO;
- break;
- case ICMP_DEST_UNREACH:
- if (code > NR_ICMP_UNREACH)
- goto out_unlock;
+ sctp_v4_err_handle(transport, skb, type, code, info);
+ sctp_err_finish(sk, transport);
- /* PMTU discovery (RFC1191) */
- if (ICMP_FRAG_NEEDED == code) {
- sctp_icmp_frag_needed(sk, asoc, transport,
- SCTP_TRUNC4(info));
- goto out_unlock;
- } else {
- if (ICMP_PROT_UNREACH == code) {
- sctp_icmp_proto_unreachable(sk, asoc,
- transport);
- goto out_unlock;
- }
- }
- err = icmp_err_convert[code].errno;
- break;
- case ICMP_TIME_EXCEEDED:
- /* Ignore any time exceeded errors due to fragment reassembly
- * timeouts.
- */
- if (ICMP_EXC_FRAGTIME == code)
- goto out_unlock;
+ return 0;
+}
- err = EHOSTUNREACH;
- break;
- case ICMP_REDIRECT:
- sctp_icmp_redirect(sk, transport, skb);
- /* Fall through to out_unlock. */
- default:
- goto out_unlock;
+int sctp_udp_v4_err(struct sock *sk, struct sk_buff *skb)
+{
+ struct net *net = dev_net(skb->dev);
+ struct sctp_association *asoc;
+ struct sctp_transport *t;
+ struct icmphdr *hdr;
+ __u32 info = 0;
+
+ skb->transport_header += sizeof(struct udphdr);
+ sk = sctp_err_lookup(net, AF_INET, skb, sctp_hdr(skb), &asoc, &t);
+ if (!sk) {
+ __ICMP_INC_STATS(net, ICMP_MIB_INERRORS);
+ return -ENOENT;
}
- inet = inet_sk(sk);
- if (!sock_owned_by_user(sk) && inet->recverr) {
- sk->sk_err = err;
- sk->sk_error_report(sk);
- } else { /* Only an error on timeout */
- sk->sk_err_soft = err;
+ skb->transport_header -= sizeof(struct udphdr);
+ hdr = (struct icmphdr *)(skb_network_header(skb) - sizeof(struct icmphdr));
+ if (hdr->type == ICMP_REDIRECT) {
+ /* can't be handled without outer iphdr known, leave it to udp_err */
+ sctp_err_finish(sk, t);
+ return 0;
}
+ if (hdr->type == ICMP_DEST_UNREACH && hdr->code == ICMP_FRAG_NEEDED)
+ info = ntohs(hdr->un.frag.mtu);
+ sctp_v4_err_handle(t, skb, hdr->type, hdr->code, info);
-out_unlock:
- sctp_err_finish(sk, transport);
- return 0;
+ sctp_err_finish(sk, t);
+ return 1;
}
/*
.notifier_call = sctp_inet6addr_event,
};
-/* ICMP error handler. */
-static int sctp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt,
- u8 type, u8 code, int offset, __be32 info)
+static void sctp_v6_err_handle(struct sctp_transport *t, struct sk_buff *skb,
+ __u8 type, __u8 code, __u32 info)
{
- struct inet6_dev *idev;
- struct sock *sk;
- struct sctp_association *asoc;
- struct sctp_transport *transport;
+ struct sctp_association *asoc = t->asoc;
+ struct sock *sk = asoc->base.sk;
struct ipv6_pinfo *np;
- __u16 saveip, savesctp;
- int err, ret = 0;
- struct net *net = dev_net(skb->dev);
-
- idev = in6_dev_get(skb->dev);
-
- /* Fix up skb to look at the embedded net header. */
- saveip = skb->network_header;
- savesctp = skb->transport_header;
- skb_reset_network_header(skb);
- skb_set_transport_header(skb, offset);
- sk = sctp_err_lookup(net, AF_INET6, skb, sctp_hdr(skb), &asoc, &transport);
- /* Put back, the original pointers. */
- skb->network_header = saveip;
- skb->transport_header = savesctp;
- if (!sk) {
- __ICMP6_INC_STATS(net, idev, ICMP6_MIB_INERRORS);
- ret = -ENOENT;
- goto out;
- }
-
- /* Warning: The sock lock is held. Remember to call
- * sctp_err_finish!
- */
+ int err = 0;
switch (type) {
case ICMPV6_PKT_TOOBIG:
if (ip6_sk_accept_pmtu(sk))
- sctp_icmp_frag_needed(sk, asoc, transport, ntohl(info));
- goto out_unlock;
+ sctp_icmp_frag_needed(sk, asoc, t, info);
+ return;
case ICMPV6_PARAMPROB:
if (ICMPV6_UNK_NEXTHDR == code) {
- sctp_icmp_proto_unreachable(sk, asoc, transport);
- goto out_unlock;
+ sctp_icmp_proto_unreachable(sk, asoc, t);
+ return;
}
break;
case NDISC_REDIRECT:
- sctp_icmp_redirect(sk, transport, skb);
- goto out_unlock;
+ sctp_icmp_redirect(sk, t, skb);
+ return;
default:
break;
}
if (!sock_owned_by_user(sk) && np->recverr) {
sk->sk_err = err;
sk->sk_error_report(sk);
- } else { /* Only an error on timeout */
+ } else {
sk->sk_err_soft = err;
}
+}
+
+/* ICMP error handler. */
+static int sctp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt,
+ u8 type, u8 code, int offset, __be32 info)
+{
+ struct net *net = dev_net(skb->dev);
+ struct sctp_transport *transport;
+ struct sctp_association *asoc;
+ __u16 saveip, savesctp;
+ struct sock *sk;
+
+ /* Fix up skb to look at the embedded net header. */
+ saveip = skb->network_header;
+ savesctp = skb->transport_header;
+ skb_reset_network_header(skb);
+ skb_set_transport_header(skb, offset);
+ sk = sctp_err_lookup(net, AF_INET6, skb, sctp_hdr(skb), &asoc, &transport);
+ /* Put back, the original pointers. */
+ skb->network_header = saveip;
+ skb->transport_header = savesctp;
+ if (!sk) {
+ __ICMP6_INC_STATS(net, __in6_dev_get(skb->dev), ICMP6_MIB_INERRORS);
+ return -ENOENT;
+ }
-out_unlock:
+ sctp_v6_err_handle(transport, skb, type, code, ntohl(info));
sctp_err_finish(sk, transport);
-out:
- if (likely(idev != NULL))
- in6_dev_put(idev);
- return ret;
+ return 0;
+}
+
+int sctp_udp_v6_err(struct sock *sk, struct sk_buff *skb)
+{
+ struct net *net = dev_net(skb->dev);
+ struct sctp_association *asoc;
+ struct sctp_transport *t;
+ struct icmp6hdr *hdr;
+ __u32 info = 0;
+
+ skb->transport_header += sizeof(struct udphdr);
+ sk = sctp_err_lookup(net, AF_INET6, skb, sctp_hdr(skb), &asoc, &t);
+ if (!sk) {
+ __ICMP6_INC_STATS(net, __in6_dev_get(skb->dev), ICMP6_MIB_INERRORS);
+ return -ENOENT;
+ }
+
+ skb->transport_header -= sizeof(struct udphdr);
+ hdr = (struct icmp6hdr *)(skb_network_header(skb) - sizeof(struct icmp6hdr));
+ if (hdr->icmp6_type == NDISC_REDIRECT) {
+ /* can't be handled without outer ip6hdr known, leave it to udpv6_err */
+ sctp_err_finish(sk, t);
+ return 0;
+ }
+ if (hdr->icmp6_type == ICMPV6_PKT_TOOBIG)
+ info = ntohl(hdr->icmp6_mtu);
+ sctp_v6_err_handle(t, skb, hdr->icmp6_type, hdr->icmp6_code, info);
+
+ sctp_err_finish(sk, t);
+ return 1;
}
static int sctp_v6_xmit(struct sk_buff *skb, struct sctp_transport *t)
sctp_transport_route(tp, NULL, sp);
if (asoc->param_flags & SPP_PMTUD_ENABLE)
sctp_assoc_sync_pmtu(asoc);
- } else if (!sctp_transport_pmtu_check(tp)) {
+ } else if (!sctp_transport_pl_enabled(tp) &&
+ !sctp_transport_pmtu_check(tp)) {
if (asoc->param_flags & SPP_PMTUD_ENABLE)
sctp_assoc_sync_pmtu(asoc);
}
return retval;
}
+/* Try to bundle a pad chunk into a packet with a heartbeat chunk for PLPMTUTD probe */
+static enum sctp_xmit sctp_packet_bundle_pad(struct sctp_packet *pkt, struct sctp_chunk *chunk)
+{
+ struct sctp_transport *t = pkt->transport;
+ struct sctp_chunk *pad;
+ int overhead = 0;
+
+ if (!chunk->pmtu_probe)
+ return SCTP_XMIT_OK;
+
+ /* calculate the Padding Data size for the pad chunk */
+ overhead += sizeof(struct sctphdr) + sizeof(struct sctp_chunkhdr);
+ overhead += sizeof(struct sctp_sender_hb_info) + sizeof(struct sctp_pad_chunk);
+ pad = sctp_make_pad(t->asoc, t->pl.probe_size - overhead);
+ if (!pad)
+ return SCTP_XMIT_DELAY;
+
+ list_add_tail(&pad->list, &pkt->chunk_list);
+ pkt->size += SCTP_PAD4(ntohs(pad->chunk_hdr->length));
+ chunk->transport = t;
+
+ return SCTP_XMIT_OK;
+}
+
/* Try to bundle an auth chunk into the packet. */
static enum sctp_xmit sctp_packet_bundle_auth(struct sctp_packet *pkt,
struct sctp_chunk *chunk)
goto finish;
retval = __sctp_packet_append_chunk(packet, chunk);
+ if (retval != SCTP_XMIT_OK)
+ goto finish;
+
+ retval = sctp_packet_bundle_pad(packet, chunk);
finish:
return retval;
sk = chunk->skb->sk;
/* check gso */
- if (packet->size > tp->pathmtu && !packet->ipfragok) {
+ if (packet->size > tp->pathmtu && !packet->ipfragok && !chunk->pmtu_probe) {
if (!sk_can_gso(sk)) {
pr_err_once("Trying to GSO but underlying device doesn't support it.");
goto out;
sctp_packet_init(&singleton, transport, sport, dport);
sctp_packet_config(&singleton, vtag, 0);
- sctp_packet_append_chunk(&singleton, chunk);
+ if (sctp_packet_append_chunk(&singleton, chunk) != SCTP_XMIT_OK) {
+ list_del_init(&chunk->list);
+ sctp_chunk_free(chunk);
+ return -ENOMEM;
+ }
return sctp_packet_transmit(&singleton, gfp);
}
one_packet = 1;
fallthrough;
- case SCTP_CID_SACK:
case SCTP_CID_HEARTBEAT:
+ if (chunk->pmtu_probe) {
+ sctp_packet_singleton(ctx->transport, chunk, ctx->gfp);
+ break;
+ }
+ fallthrough;
+ case SCTP_CID_SACK:
case SCTP_CID_SHUTDOWN:
case SCTP_CID_ECN_ECNE:
case SCTP_CID_ASCONF:
return 0;
}
-static int sctp_udp_err_lookup(struct sock *sk, struct sk_buff *skb)
-{
- struct sctp_association *asoc;
- struct sctp_transport *t;
- int family;
-
- skb->transport_header += sizeof(struct udphdr);
- family = (ip_hdr(skb)->version == 4) ? AF_INET : AF_INET6;
- sk = sctp_err_lookup(dev_net(skb->dev), family, skb, sctp_hdr(skb),
- &asoc, &t);
- if (!sk)
- return -ENOENT;
-
- sctp_err_finish(sk, t);
- return 0;
-}
-
int sctp_udp_sock_start(struct net *net)
{
struct udp_tunnel_sock_cfg tuncfg = {NULL};
tuncfg.encap_type = 1;
tuncfg.encap_rcv = sctp_udp_rcv;
- tuncfg.encap_err_lookup = sctp_udp_err_lookup;
+ tuncfg.encap_err_lookup = sctp_udp_v4_err;
setup_udp_tunnel_sock(net, sock, &tuncfg);
net->sctp.udp4_sock = sock->sk;
tuncfg.encap_type = 1;
tuncfg.encap_rcv = sctp_udp_rcv;
- tuncfg.encap_err_lookup = sctp_udp_err_lookup;
+ tuncfg.encap_err_lookup = sctp_udp_v6_err;
setup_udp_tunnel_sock(net, sock, &tuncfg);
net->sctp.udp6_sock = sock->sk;
#endif
.handler = sctp4_rcv,
.err_handler = sctp_v4_err,
.no_policy = 1,
- .netns_ok = 1,
.icmp_strict_tag_validation = 1,
};
/* Make a HEARTBEAT chunk. */
struct sctp_chunk *sctp_make_heartbeat(const struct sctp_association *asoc,
- const struct sctp_transport *transport)
+ const struct sctp_transport *transport,
+ __u32 probe_size)
{
struct sctp_sender_hb_info hbinfo;
struct sctp_chunk *retval;
hbinfo.daddr = transport->ipaddr;
hbinfo.sent_at = jiffies;
hbinfo.hb_nonce = transport->hb_nonce;
+ hbinfo.probe_size = probe_size;
/* Cast away the 'const', as this is just telling the chunk
* what transport it belongs to.
retval->transport = (struct sctp_transport *) transport;
retval->subh.hbs_hdr = sctp_addto_chunk(retval, sizeof(hbinfo),
&hbinfo);
+ retval->pmtu_probe = !!probe_size;
nodata:
return retval;
return retval;
}
+/* RFC4820 3. Padding Chunk (PAD)
+ * 0 1 2 3
+ * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+ * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ * | Type = 0x84 | Flags=0 | Length |
+ * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ * | |
+ * \ Padding Data /
+ * / \
+ * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ */
+struct sctp_chunk *sctp_make_pad(const struct sctp_association *asoc, int len)
+{
+ struct sctp_chunk *retval;
+
+ retval = sctp_make_control(asoc, SCTP_CID_PAD, 0, len, GFP_ATOMIC);
+ if (!retval)
+ return NULL;
+
+ skb_put_zero(retval->skb, len);
+ retval->chunk_hdr->length = htons(ntohs(retval->chunk_hdr->length) + len);
+ retval->chunk_end = skb_tail_pointer(retval->skb);
+
+ return retval;
+}
+
/* Create an Operation Error chunk with the specified space reserved.
* This routine can be used for containing multiple causes in the chunk.
*/
sctp_transport_put(transport);
}
+/* Handle the timeout of the probe timer. */
+void sctp_generate_probe_event(struct timer_list *t)
+{
+ struct sctp_transport *transport = from_timer(transport, t, probe_timer);
+ struct sctp_association *asoc = transport->asoc;
+ struct sock *sk = asoc->base.sk;
+ struct net *net = sock_net(sk);
+ int error = 0;
+
+ bh_lock_sock(sk);
+ if (sock_owned_by_user(sk)) {
+ pr_debug("%s: sock is busy\n", __func__);
+
+ /* Try again later. */
+ if (!mod_timer(&transport->probe_timer, jiffies + (HZ / 20)))
+ sctp_transport_hold(transport);
+ goto out_unlock;
+ }
+
+ error = sctp_do_sm(net, SCTP_EVENT_T_TIMEOUT,
+ SCTP_ST_TIMEOUT(SCTP_EVENT_TIMEOUT_PROBE),
+ asoc->state, asoc->ep, asoc,
+ transport, GFP_ATOMIC);
+
+ if (error)
+ sk->sk_err = -error;
+
+out_unlock:
+ bh_unlock_sock(sk);
+ sctp_transport_put(transport);
+}
+
/* Inject a SACK Timeout event into the state machine. */
static void sctp_generate_sack_event(struct timer_list *t)
{
sctp_cmd_hb_timers_stop(commands, asoc);
break;
+ case SCTP_CMD_PROBE_TIMER_UPDATE:
+ t = cmd->obj.transport;
+ sctp_transport_reset_probe_timer(t);
+ break;
+
case SCTP_CMD_REPORT_ERROR:
error = cmd->obj.error;
break;
/* If the INIT is coming toward a closing socket, we'll send back
* and ABORT. Essentially, this catches the race of INIT being
- * backloged to the socket at the same time as the user isses close().
+ * backloged to the socket at the same time as the user issues close().
* Since the socket and all its associations are going away, we
* can treat this OOTB
*/
sctp_add_cmd_sf(commands, SCTP_CMD_NEW_STATE,
SCTP_STATE(SCTP_STATE_COOKIE_ECHOED));
- /* SCTP-AUTH: genereate the assocition shared keys so that
- * we can potentially signe the COOKIE-ECHO.
+ /* SCTP-AUTH: generate the association shared keys so that
+ * we can potentially sign the COOKIE-ECHO.
*/
sctp_add_cmd_sf(commands, SCTP_CMD_ASSOC_SHKEY, SCTP_NULL());
goto nomem_init;
/* SCTP-AUTH: Now that we've populate required fields in
- * sctp_process_init, set up the assocaition shared keys as
+ * sctp_process_init, set up the association shared keys as
* necessary so that we can potentially authenticate the ACK
*/
error = sctp_auth_asoc_init_active_key(new_asoc, GFP_ATOMIC);
/* Add all the state machine commands now since we've created
* everything. This way we don't introduce memory corruptions
- * during side-effect processing and correclty count established
+ * during side-effect processing and correctly count established
* associations.
*/
sctp_add_cmd_sf(commands, SCTP_CMD_NEW_ASOC, SCTP_ASOC(new_asoc));
commands);
/* Reset init error count upon receipt of COOKIE-ACK,
- * to avoid problems with the managemement of this
+ * to avoid problems with the management of this
* counter in stale cookie situations when a transition back
* from the COOKIE-ECHOED state to the COOKIE-WAIT
* state is performed.
struct sctp_chunk *reply;
/* Send a heartbeat to our peer. */
- reply = sctp_make_heartbeat(asoc, transport);
+ reply = sctp_make_heartbeat(asoc, transport, 0);
if (!reply)
return SCTP_DISPOSITION_NOMEM;
return SCTP_DISPOSITION_CONSUME;
}
+/* send hb chunk with padding for PLPMUTD. */
+enum sctp_disposition sctp_sf_send_probe(struct net *net,
+ const struct sctp_endpoint *ep,
+ const struct sctp_association *asoc,
+ const union sctp_subtype type,
+ void *arg,
+ struct sctp_cmd_seq *commands)
+{
+ struct sctp_transport *transport = (struct sctp_transport *)arg;
+ struct sctp_chunk *reply;
+
+ if (!sctp_transport_pl_enabled(transport))
+ return SCTP_DISPOSITION_CONSUME;
+
+ sctp_transport_pl_send(transport);
+
+ reply = sctp_make_heartbeat(asoc, transport, transport->pl.probe_size);
+ if (!reply)
+ return SCTP_DISPOSITION_NOMEM;
+ sctp_add_cmd_sf(commands, SCTP_CMD_REPLY, SCTP_CHUNK(reply));
+ sctp_add_cmd_sf(commands, SCTP_CMD_PROBE_TIMER_UPDATE,
+ SCTP_TRANSPORT(transport));
+
+ return SCTP_DISPOSITION_CONSUME;
+}
+
/*
* Process an heartbeat request.
*
if (hbinfo->hb_nonce != link->hb_nonce)
return SCTP_DISPOSITION_DISCARD;
+ if (hbinfo->probe_size) {
+ if (hbinfo->probe_size != link->pl.probe_size ||
+ !sctp_transport_pl_enabled(link))
+ return SCTP_DISPOSITION_DISCARD;
+
+ sctp_transport_pl_recv(link);
+ if (link->pl.state == SCTP_PL_COMPLETE)
+ return SCTP_DISPOSITION_CONSUME;
+
+ return sctp_sf_send_probe(net, ep, asoc, type, link, commands);
+ }
+
max_interval = link->hbinterval + link->rto;
/* Check if the timestamp looks valid. */
commands);
/* Since we are not going to really process this INIT, there
- * is no point in verifying chunk boundries. Just generate
+ * is no point in verifying chunk boundaries. Just generate
* the SHUTDOWN ACK.
*/
reply = sctp_make_shutdown_ack(asoc, chunk);
goto nomem_chunk;
/* Do all the commands now (after allocation), so that we
- * have consistent state if memory allocation failes
+ * have consistent state if memory allocation fails
*/
sctp_add_cmd_sf(commands, SCTP_CMD_EVENT_ULP, SCTP_ULPEVENT(ev));
return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
/* We need to discard the rest of the packet to prevent
- * potential bomming attacks from additional bundled chunks.
+ * potential boomming attacks from additional bundled chunks.
* This is documented in SCTP Threats ID.
*/
return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
}
/*
- * SCTP-AUTH Section 6.3 Receiving authenticated chukns
+ * SCTP-AUTH Section 6.3 Receiving authenticated chunks
*
* The receiver MUST use the HMAC algorithm indicated in the HMAC
* Identifier field. If this algorithm was not specified by the
/* Handle protocol violation of an invalid chunk bundling. For example,
* when we have an association and we receive bundled INIT-ACK, or
- * SHUDOWN-COMPLETE, our peer is clearly violationg the "MUST NOT bundle"
+ * SHUTDOWN-COMPLETE, our peer is clearly violating the "MUST NOT bundle"
* statement from the specs. Additionally, there might be an attacker
* on the path and we may not want to continue this communication.
*/
* Inputs
* (endpoint, asoc)
*
- * The RFC does not explcitly address this issue, but is the route through the
+ * The RFC does not explicitly address this issue, but is the route through the
* state table when someone issues a shutdown while in COOKIE_ECHOED state.
*
* Outputs
/* RFC2960 9.2 If the timer expires, the endpoint must re-send the SHUTDOWN
* with the updated last sequential TSN received from its peer.
*
- * An endpoint should limit the number of retransmissions of the
+ * An endpoint should limit the number of retransmission of the
* SHUTDOWN chunk to the protocol parameter 'Association.Max.Retrans'.
* If this threshold is exceeded the endpoint should destroy the TCB and
* MUST report the peer endpoint unreachable to the upper layer (and
}
/*
- * ADDIP Section 4.1 ASCONF CHunk Procedures
+ * ADDIP Section 4.1 ASCONF Chunk Procedures
* If the T4 RTO timer expires the endpoint should do B1 to B5
*/
enum sctp_disposition sctp_sf_t4_timer_expire(
chunk->ecn_ce_done = 1;
if (af->is_ce(sctp_gso_headskb(chunk->skb))) {
- /* Do real work as sideffect. */
+ /* Do real work as side effect. */
sctp_add_cmd_sf(commands, SCTP_CMD_ECN_CE,
SCTP_U32(tsn));
}
}; /*state_fn_t auth_chunk_event_table[][] */
static const struct sctp_sm_table_entry
+pad_chunk_event_table[SCTP_STATE_NUM_STATES] = {
+ /* SCTP_STATE_CLOSED */
+ TYPE_SCTP_FUNC(sctp_sf_discard_chunk),
+ /* SCTP_STATE_COOKIE_WAIT */
+ TYPE_SCTP_FUNC(sctp_sf_discard_chunk),
+ /* SCTP_STATE_COOKIE_ECHOED */
+ TYPE_SCTP_FUNC(sctp_sf_discard_chunk),
+ /* SCTP_STATE_ESTABLISHED */
+ TYPE_SCTP_FUNC(sctp_sf_discard_chunk),
+ /* SCTP_STATE_SHUTDOWN_PENDING */
+ TYPE_SCTP_FUNC(sctp_sf_discard_chunk),
+ /* SCTP_STATE_SHUTDOWN_SENT */
+ TYPE_SCTP_FUNC(sctp_sf_discard_chunk),
+ /* SCTP_STATE_SHUTDOWN_RECEIVED */
+ TYPE_SCTP_FUNC(sctp_sf_discard_chunk),
+ /* SCTP_STATE_SHUTDOWN_ACK_SENT */
+ TYPE_SCTP_FUNC(sctp_sf_discard_chunk),
+}; /* chunk pad */
+
+static const struct sctp_sm_table_entry
chunk_event_table_unknown[SCTP_STATE_NUM_STATES] = {
/* SCTP_STATE_CLOSED */
TYPE_SCTP_FUNC(sctp_sf_ootb),
TYPE_SCTP_FUNC(sctp_sf_timer_ignore), \
}
+#define TYPE_SCTP_EVENT_TIMEOUT_PROBE { \
+ /* SCTP_STATE_CLOSED */ \
+ TYPE_SCTP_FUNC(sctp_sf_timer_ignore), \
+ /* SCTP_STATE_COOKIE_WAIT */ \
+ TYPE_SCTP_FUNC(sctp_sf_timer_ignore), \
+ /* SCTP_STATE_COOKIE_ECHOED */ \
+ TYPE_SCTP_FUNC(sctp_sf_timer_ignore), \
+ /* SCTP_STATE_ESTABLISHED */ \
+ TYPE_SCTP_FUNC(sctp_sf_send_probe), \
+ /* SCTP_STATE_SHUTDOWN_PENDING */ \
+ TYPE_SCTP_FUNC(sctp_sf_timer_ignore), \
+ /* SCTP_STATE_SHUTDOWN_SENT */ \
+ TYPE_SCTP_FUNC(sctp_sf_timer_ignore), \
+ /* SCTP_STATE_SHUTDOWN_RECEIVED */ \
+ TYPE_SCTP_FUNC(sctp_sf_timer_ignore), \
+ /* SCTP_STATE_SHUTDOWN_ACK_SENT */ \
+ TYPE_SCTP_FUNC(sctp_sf_timer_ignore), \
+}
+
static const struct sctp_sm_table_entry
timeout_event_table[SCTP_NUM_TIMEOUT_TYPES][SCTP_STATE_NUM_STATES] = {
TYPE_SCTP_EVENT_TIMEOUT_NONE,
TYPE_SCTP_EVENT_TIMEOUT_T5_SHUTDOWN_GUARD,
TYPE_SCTP_EVENT_TIMEOUT_HEARTBEAT,
TYPE_SCTP_EVENT_TIMEOUT_RECONF,
+ TYPE_SCTP_EVENT_TIMEOUT_PROBE,
TYPE_SCTP_EVENT_TIMEOUT_SACK,
TYPE_SCTP_EVENT_TIMEOUT_AUTOCLOSE,
};
case SCTP_CID_AUTH:
return &auth_chunk_event_table[0][state];
+
+ case SCTP_CID_PAD:
+ return &pad_chunk_event_table[state];
}
return &chunk_event_table_unknown[state];
sctp_transport_pmtu(trans, sctp_opt2sk(sp));
sctp_assoc_sync_pmtu(asoc);
}
+ sctp_transport_pl_reset(trans);
} else if (asoc) {
asoc->param_flags =
(asoc->param_flags & ~SPP_PMTUD) | pmtud_change;
transports)
t->encap_port = encap_port;
+ asoc->encap_port = encap_port;
return 0;
}
return 0;
}
+static int sctp_setsockopt_probe_interval(struct sock *sk,
+ struct sctp_probeinterval *params,
+ unsigned int optlen)
+{
+ struct sctp_association *asoc;
+ struct sctp_transport *t;
+ __u32 probe_interval;
+
+ if (optlen != sizeof(*params))
+ return -EINVAL;
+
+ probe_interval = params->spi_interval;
+ if (probe_interval && probe_interval < SCTP_PROBE_TIMER_MIN)
+ return -EINVAL;
+
+ /* If an address other than INADDR_ANY is specified, and
+ * no transport is found, then the request is invalid.
+ */
+ if (!sctp_is_any(sk, (union sctp_addr *)¶ms->spi_address)) {
+ t = sctp_addr_id2transport(sk, ¶ms->spi_address,
+ params->spi_assoc_id);
+ if (!t)
+ return -EINVAL;
+
+ t->probe_interval = msecs_to_jiffies(probe_interval);
+ sctp_transport_pl_reset(t);
+ return 0;
+ }
+
+ /* Get association, if assoc_id != SCTP_FUTURE_ASSOC and the
+ * socket is a one to many style socket, and an association
+ * was not found, then the id was invalid.
+ */
+ asoc = sctp_id2assoc(sk, params->spi_assoc_id);
+ if (!asoc && params->spi_assoc_id != SCTP_FUTURE_ASSOC &&
+ sctp_style(sk, UDP))
+ return -EINVAL;
+
+ /* If changes are for association, also apply probe_interval to
+ * each transport.
+ */
+ if (asoc) {
+ list_for_each_entry(t, &asoc->peer.transport_addr_list, transports) {
+ t->probe_interval = msecs_to_jiffies(probe_interval);
+ sctp_transport_pl_reset(t);
+ }
+
+ asoc->probe_interval = msecs_to_jiffies(probe_interval);
+ return 0;
+ }
+
+ sctp_sk(sk)->probe_interval = probe_interval;
+ return 0;
+}
+
/* API 6.2 setsockopt(), getsockopt()
*
* Applications use setsockopt() and getsockopt() to set or retrieve
case SCTP_REMOTE_UDP_ENCAPS_PORT:
retval = sctp_setsockopt_encap_port(sk, kopt, optlen);
break;
+ case SCTP_PLPMTUD_PROBE_INTERVAL:
+ retval = sctp_setsockopt_probe_interval(sk, kopt, optlen);
+ break;
default:
retval = -ENOPROTOOPT;
break;
atomic_set(&sp->pd_mode, 0);
skb_queue_head_init(&sp->pd_lobby);
sp->frag_interleave = 0;
+ sp->probe_interval = net->sctp.probe_interval;
/* Create a per socket endpoint structure. Even if we
* change the data structure relationships, this may still
return 0;
}
+static int sctp_getsockopt_probe_interval(struct sock *sk, int len,
+ char __user *optval,
+ int __user *optlen)
+{
+ struct sctp_probeinterval params;
+ struct sctp_association *asoc;
+ struct sctp_transport *t;
+ __u32 probe_interval;
+
+ if (len < sizeof(params))
+ return -EINVAL;
+
+ len = sizeof(params);
+ if (copy_from_user(¶ms, optval, len))
+ return -EFAULT;
+
+ /* If an address other than INADDR_ANY is specified, and
+ * no transport is found, then the request is invalid.
+ */
+ if (!sctp_is_any(sk, (union sctp_addr *)¶ms.spi_address)) {
+ t = sctp_addr_id2transport(sk, ¶ms.spi_address,
+ params.spi_assoc_id);
+ if (!t) {
+ pr_debug("%s: failed no transport\n", __func__);
+ return -EINVAL;
+ }
+
+ probe_interval = jiffies_to_msecs(t->probe_interval);
+ goto out;
+ }
+
+ /* Get association, if assoc_id != SCTP_FUTURE_ASSOC and the
+ * socket is a one to many style socket, and an association
+ * was not found, then the id was invalid.
+ */
+ asoc = sctp_id2assoc(sk, params.spi_assoc_id);
+ if (!asoc && params.spi_assoc_id != SCTP_FUTURE_ASSOC &&
+ sctp_style(sk, UDP)) {
+ pr_debug("%s: failed no association\n", __func__);
+ return -EINVAL;
+ }
+
+ if (asoc) {
+ probe_interval = jiffies_to_msecs(asoc->probe_interval);
+ goto out;
+ }
+
+ probe_interval = sctp_sk(sk)->probe_interval;
+
+out:
+ params.spi_interval = probe_interval;
+ if (copy_to_user(optval, ¶ms, len))
+ return -EFAULT;
+
+ if (put_user(len, optlen))
+ return -EFAULT;
+
+ return 0;
+}
+
static int sctp_getsockopt(struct sock *sk, int level, int optname,
char __user *optval, int __user *optlen)
{
case SCTP_REMOTE_UDP_ENCAPS_PORT:
retval = sctp_getsockopt_encap_port(sk, len, optval, optlen);
break;
+ case SCTP_PLPMTUD_PROBE_INTERVAL:
+ retval = sctp_getsockopt_probe_interval(sk, len, optval, optlen);
+ break;
default:
retval = -ENOPROTOOPT;
break;
void *buffer, size_t *lenp, loff_t *ppos);
static int proc_sctp_do_auth(struct ctl_table *ctl, int write,
void *buffer, size_t *lenp, loff_t *ppos);
+static int proc_sctp_do_probe_interval(struct ctl_table *ctl, int write,
+ void *buffer, size_t *lenp, loff_t *ppos);
static struct ctl_table sctp_table[] = {
{
.proc_handler = proc_dointvec,
},
{
+ .procname = "plpmtud_probe_interval",
+ .data = &init_net.sctp.probe_interval,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_sctp_do_probe_interval,
+ },
+ {
.procname = "udp_port",
.data = &init_net.sctp.udp_port,
.maxlen = sizeof(int),
.data = &init_net.sctp.encap_port,
.maxlen = sizeof(int),
.mode = 0644,
- .proc_handler = proc_dointvec,
+ .proc_handler = proc_dointvec_minmax,
.extra1 = SYSCTL_ZERO,
.extra2 = &udp_port_max,
},
return ret;
}
+static int proc_sctp_do_probe_interval(struct ctl_table *ctl, int write,
+ void *buffer, size_t *lenp, loff_t *ppos)
+{
+ struct net *net = current->nsproxy->net_ns;
+ struct ctl_table tbl;
+ int ret, new_value;
+
+ memset(&tbl, 0, sizeof(struct ctl_table));
+ tbl.maxlen = sizeof(unsigned int);
+
+ if (write)
+ tbl.data = &new_value;
+ else
+ tbl.data = &net->sctp.probe_interval;
+
+ ret = proc_dointvec(&tbl, write, buffer, lenp, ppos);
+ if (write && ret == 0) {
+ if (new_value && new_value < SCTP_PROBE_TIMER_MIN)
+ return -EINVAL;
+
+ net->sctp.probe_interval = new_value;
+ }
+
+ return ret;
+}
+
int sctp_sysctl_net_register(struct net *net)
{
struct ctl_table *table;
timer_setup(&peer->T3_rtx_timer, sctp_generate_t3_rtx_event, 0);
timer_setup(&peer->hb_timer, sctp_generate_heartbeat_event, 0);
timer_setup(&peer->reconf_timer, sctp_generate_reconf_event, 0);
+ timer_setup(&peer->probe_timer, sctp_generate_probe_event, 0);
timer_setup(&peer->proto_unreach_timer,
sctp_generate_proto_unreach_event, 0);
if (del_timer(&transport->reconf_timer))
sctp_transport_put(transport);
+ if (del_timer(&transport->probe_timer))
+ sctp_transport_put(transport);
+
/* Delete the ICMP proto unreachable timer if it's active. */
if (del_timer(&transport->proto_unreach_timer))
sctp_transport_put(transport);
sctp_transport_hold(transport);
}
+void sctp_transport_reset_probe_timer(struct sctp_transport *transport)
+{
+ if (timer_pending(&transport->probe_timer))
+ return;
+ if (!mod_timer(&transport->probe_timer,
+ jiffies + transport->probe_interval))
+ sctp_transport_hold(transport);
+}
+
/* This transport has been assigned to an association.
* Initialize fields from the association or from the sock itself.
* Register the reference count in the association.
transport->pathmtu = sctp_dst_mtu(transport->dst);
else
transport->pathmtu = SCTP_DEFAULT_MAXSEGMENT;
+
+ sctp_transport_pl_update(transport);
+}
+
+void sctp_transport_pl_send(struct sctp_transport *t)
+{
+ pr_debug("%s: PLPMTUD: transport: %p, state: %d, pmtu: %d, size: %d, high: %d\n",
+ __func__, t, t->pl.state, t->pl.pmtu, t->pl.probe_size, t->pl.probe_high);
+
+ if (t->pl.probe_count < SCTP_MAX_PROBES) {
+ t->pl.probe_count++;
+ return;
+ }
+
+ if (t->pl.state == SCTP_PL_BASE) {
+ if (t->pl.probe_size == SCTP_BASE_PLPMTU) { /* BASE_PLPMTU Confirmation Failed */
+ t->pl.state = SCTP_PL_ERROR; /* Base -> Error */
+
+ t->pl.pmtu = SCTP_MIN_PLPMTU;
+ t->pathmtu = t->pl.pmtu + sctp_transport_pl_hlen(t);
+ sctp_assoc_sync_pmtu(t->asoc);
+ }
+ } else if (t->pl.state == SCTP_PL_SEARCH) {
+ if (t->pl.pmtu == t->pl.probe_size) { /* Black Hole Detected */
+ t->pl.state = SCTP_PL_BASE; /* Search -> Base */
+ t->pl.probe_size = SCTP_BASE_PLPMTU;
+ t->pl.probe_high = 0;
+
+ t->pl.pmtu = SCTP_BASE_PLPMTU;
+ t->pathmtu = t->pl.pmtu + sctp_transport_pl_hlen(t);
+ sctp_assoc_sync_pmtu(t->asoc);
+ } else { /* Normal probe failure. */
+ t->pl.probe_high = t->pl.probe_size;
+ t->pl.probe_size = t->pl.pmtu;
+ }
+ } else if (t->pl.state == SCTP_PL_COMPLETE) {
+ if (t->pl.pmtu == t->pl.probe_size) { /* Black Hole Detected */
+ t->pl.state = SCTP_PL_BASE; /* Search Complete -> Base */
+ t->pl.probe_size = SCTP_BASE_PLPMTU;
+
+ t->pl.pmtu = SCTP_BASE_PLPMTU;
+ t->pathmtu = t->pl.pmtu + sctp_transport_pl_hlen(t);
+ sctp_assoc_sync_pmtu(t->asoc);
+ }
+ }
+ t->pl.probe_count = 1;
+}
+
+void sctp_transport_pl_recv(struct sctp_transport *t)
+{
+ pr_debug("%s: PLPMTUD: transport: %p, state: %d, pmtu: %d, size: %d, high: %d\n",
+ __func__, t, t->pl.state, t->pl.pmtu, t->pl.probe_size, t->pl.probe_high);
+
+ t->pl.pmtu = t->pl.probe_size;
+ t->pl.probe_count = 0;
+ if (t->pl.state == SCTP_PL_BASE) {
+ t->pl.state = SCTP_PL_SEARCH; /* Base -> Search */
+ t->pl.probe_size += SCTP_PL_BIG_STEP;
+ } else if (t->pl.state == SCTP_PL_ERROR) {
+ t->pl.state = SCTP_PL_SEARCH; /* Error -> Search */
+
+ t->pl.pmtu = t->pl.probe_size;
+ t->pathmtu = t->pl.pmtu + sctp_transport_pl_hlen(t);
+ sctp_assoc_sync_pmtu(t->asoc);
+ t->pl.probe_size += SCTP_PL_BIG_STEP;
+ } else if (t->pl.state == SCTP_PL_SEARCH) {
+ if (!t->pl.probe_high) {
+ t->pl.probe_size = min(t->pl.probe_size + SCTP_PL_BIG_STEP,
+ SCTP_MAX_PLPMTU);
+ return;
+ }
+ t->pl.probe_size += SCTP_PL_MIN_STEP;
+ if (t->pl.probe_size >= t->pl.probe_high) {
+ t->pl.probe_high = 0;
+ t->pl.raise_count = 0;
+ t->pl.state = SCTP_PL_COMPLETE; /* Search -> Search Complete */
+
+ t->pl.probe_size = t->pl.pmtu;
+ t->pathmtu = t->pl.pmtu + sctp_transport_pl_hlen(t);
+ sctp_assoc_sync_pmtu(t->asoc);
+ }
+ } else if (t->pl.state == SCTP_PL_COMPLETE && ++t->pl.raise_count == 30) {
+ /* Raise probe_size again after 30 * interval in Search Complete */
+ t->pl.state = SCTP_PL_SEARCH; /* Search Complete -> Search */
+ t->pl.probe_size += SCTP_PL_MIN_STEP;
+ }
+}
+
+static bool sctp_transport_pl_toobig(struct sctp_transport *t, u32 pmtu)
+{
+ pr_debug("%s: PLPMTUD: transport: %p, state: %d, pmtu: %d, size: %d, ptb: %d\n",
+ __func__, t, t->pl.state, t->pl.pmtu, t->pl.probe_size, pmtu);
+
+ if (pmtu < SCTP_MIN_PLPMTU || pmtu >= t->pl.probe_size)
+ return false;
+
+ if (t->pl.state == SCTP_PL_BASE) {
+ if (pmtu >= SCTP_MIN_PLPMTU && pmtu < SCTP_BASE_PLPMTU) {
+ t->pl.state = SCTP_PL_ERROR; /* Base -> Error */
+
+ t->pl.pmtu = SCTP_MIN_PLPMTU;
+ t->pathmtu = t->pl.pmtu + sctp_transport_pl_hlen(t);
+ }
+ } else if (t->pl.state == SCTP_PL_SEARCH) {
+ if (pmtu >= SCTP_BASE_PLPMTU && pmtu < t->pl.pmtu) {
+ t->pl.state = SCTP_PL_BASE; /* Search -> Base */
+ t->pl.probe_size = SCTP_BASE_PLPMTU;
+ t->pl.probe_count = 0;
+
+ t->pl.probe_high = 0;
+ t->pl.pmtu = SCTP_BASE_PLPMTU;
+ t->pathmtu = t->pl.pmtu + sctp_transport_pl_hlen(t);
+ } else if (pmtu > t->pl.pmtu && pmtu < t->pl.probe_size) {
+ t->pl.probe_size = pmtu;
+ t->pl.probe_count = 0;
+
+ return false;
+ }
+ } else if (t->pl.state == SCTP_PL_COMPLETE) {
+ if (pmtu >= SCTP_BASE_PLPMTU && pmtu < t->pl.pmtu) {
+ t->pl.state = SCTP_PL_BASE; /* Complete -> Base */
+ t->pl.probe_size = SCTP_BASE_PLPMTU;
+ t->pl.probe_count = 0;
+
+ t->pl.probe_high = 0;
+ t->pl.pmtu = SCTP_BASE_PLPMTU;
+ t->pathmtu = t->pl.pmtu + sctp_transport_pl_hlen(t);
+ }
+ }
+
+ return true;
}
bool sctp_transport_update_pmtu(struct sctp_transport *t, u32 pmtu)
{
- struct dst_entry *dst = sctp_transport_dst_check(t);
struct sock *sk = t->asoc->base.sk;
+ struct dst_entry *dst;
bool change = true;
if (unlikely(pmtu < SCTP_DEFAULT_MINSEGMENT)) {
}
pmtu = SCTP_TRUNC4(pmtu);
+ if (sctp_transport_pl_enabled(t))
+ return sctp_transport_pl_toobig(t, pmtu - sctp_transport_pl_hlen(t));
+
+ dst = sctp_transport_dst_check(t);
if (dst) {
struct sctp_pf *pf = sctp_get_pf_specific(dst->ops->family);
union sctp_addr addr;
obj-$(CONFIG_SMC) += smc.o
obj-$(CONFIG_SMC_DIAG) += smc_diag.o
smc-y := af_smc.o smc_pnet.o smc_ib.o smc_clc.o smc_core.o smc_wr.o smc_llc.o
-smc-y += smc_cdc.o smc_tx.o smc_rx.o smc_close.o smc_ism.o smc_netlink.o
+smc-y += smc_cdc.o smc_tx.o smc_rx.o smc_close.o smc_ism.o smc_netlink.o smc_stats.o
#include "smc_tx.h"
#include "smc_rx.h"
#include "smc_close.h"
+#include "smc_stats.h"
static DEFINE_MUTEX(smc_server_lgr_pending); /* serialize link group
* creation on server
link->peer_mtu = clc->r0.qp_mtu;
}
-static void smc_switch_to_fallback(struct smc_sock *smc)
+static void smc_stat_inc_fback_rsn_cnt(struct smc_sock *smc,
+ struct smc_stats_fback *fback_arr)
+{
+ int cnt;
+
+ for (cnt = 0; cnt < SMC_MAX_FBACK_RSN_CNT; cnt++) {
+ if (fback_arr[cnt].fback_code == smc->fallback_rsn) {
+ fback_arr[cnt].count++;
+ break;
+ }
+ if (!fback_arr[cnt].fback_code) {
+ fback_arr[cnt].fback_code = smc->fallback_rsn;
+ fback_arr[cnt].count++;
+ break;
+ }
+ }
+}
+
+static void smc_stat_fallback(struct smc_sock *smc)
+{
+ struct net *net = sock_net(&smc->sk);
+
+ mutex_lock(&net->smc.mutex_fback_rsn);
+ if (smc->listen_smc) {
+ smc_stat_inc_fback_rsn_cnt(smc, net->smc.fback_rsn->srv);
+ net->smc.fback_rsn->srv_fback_cnt++;
+ } else {
+ smc_stat_inc_fback_rsn_cnt(smc, net->smc.fback_rsn->clnt);
+ net->smc.fback_rsn->clnt_fback_cnt++;
+ }
+ mutex_unlock(&net->smc.mutex_fback_rsn);
+}
+
+static void smc_switch_to_fallback(struct smc_sock *smc, int reason_code)
{
smc->use_fallback = true;
+ smc->fallback_rsn = reason_code;
+ smc_stat_fallback(smc);
if (smc->sk.sk_socket && smc->sk.sk_socket->file) {
smc->clcsock->file = smc->sk.sk_socket->file;
smc->clcsock->file->private_data = smc->clcsock;
/* fall back during connect */
static int smc_connect_fallback(struct smc_sock *smc, int reason_code)
{
- smc_switch_to_fallback(smc);
- smc->fallback_rsn = reason_code;
+ smc_switch_to_fallback(smc, reason_code);
smc_copy_sock_settings_to_clc(smc);
smc->connect_nonblock = 0;
if (smc->sk.sk_state == SMC_INIT)
static int smc_connect_decline_fallback(struct smc_sock *smc, int reason_code,
u8 version)
{
+ struct net *net = sock_net(&smc->sk);
int rc;
if (reason_code < 0) { /* error, fallback is not possible */
+ this_cpu_inc(net->smc.smc_stats->clnt_hshake_err_cnt);
if (smc->sk.sk_state == SMC_INIT)
sock_put(&smc->sk); /* passive closing */
return reason_code;
if (reason_code != SMC_CLC_DECL_PEERDECL) {
rc = smc_clc_send_decline(smc, reason_code, version);
if (rc < 0) {
+ this_cpu_inc(net->smc.smc_stats->clnt_hshake_err_cnt);
if (smc->sk.sk_state == SMC_INIT)
sock_put(&smc->sk); /* passive closing */
return rc;
if (rc)
goto vlan_cleanup;
+ SMC_STAT_CLNT_SUCC_INC(sock_net(smc->clcsock->sk), aclc);
smc_connect_ism_vlan_cleanup(smc, ini);
kfree(buf);
kfree(ini);
static void smc_listen_out_err(struct smc_sock *new_smc)
{
struct sock *newsmcsk = &new_smc->sk;
+ struct net *net = sock_net(newsmcsk);
+ this_cpu_inc(net->smc.smc_stats->srv_hshake_err_cnt);
if (newsmcsk->sk_state == SMC_INIT)
sock_put(&new_smc->sk); /* passive closing */
newsmcsk->sk_state = SMC_CLOSED;
smc_listen_out_err(new_smc);
return;
}
- smc_switch_to_fallback(new_smc);
- new_smc->fallback_rsn = reason_code;
+ smc_switch_to_fallback(new_smc, reason_code);
if (reason_code && reason_code != SMC_CLC_DECL_PEERDECL) {
if (smc_clc_send_decline(new_smc, reason_code, version) < 0) {
smc_listen_out_err(new_smc);
/* check if peer is smc capable */
if (!tcp_sk(newclcsock->sk)->syn_smc) {
- smc_switch_to_fallback(new_smc);
- new_smc->fallback_rsn = SMC_CLC_DECL_PEERNOSMC;
+ smc_switch_to_fallback(new_smc, SMC_CLC_DECL_PEERNOSMC);
smc_listen_out_connected(new_smc);
return;
}
}
smc_conn_save_peer_info(new_smc, cclc);
smc_listen_out_connected(new_smc);
+ SMC_STAT_SERV_SUCC_INC(sock_net(newclcsock->sk), ini);
goto out_free;
out_unlock:
if (msg->msg_flags & MSG_FASTOPEN) {
if (sk->sk_state == SMC_INIT && !smc->connect_nonblock) {
- smc_switch_to_fallback(smc);
- smc->fallback_rsn = SMC_CLC_DECL_OPTUNSUPP;
+ smc_switch_to_fallback(smc, SMC_CLC_DECL_OPTUNSUPP);
} else {
rc = -EINVAL;
goto out;
}
}
- if (smc->use_fallback)
+ if (smc->use_fallback) {
rc = smc->clcsock->ops->sendmsg(smc->clcsock, msg, len);
- else
+ } else {
rc = smc_tx_sendmsg(smc, msg, len);
+ SMC_STAT_TX_PAYLOAD(smc, len, rc);
+ }
out:
release_sock(sk);
return rc;
} else {
msg->msg_namelen = 0;
rc = smc_rx_recvmsg(smc, msg, NULL, len, flags);
+ SMC_STAT_RX_PAYLOAD(smc, rc, rc);
}
out:
case TCP_FASTOPEN_NO_COOKIE:
/* option not supported by SMC */
if (sk->sk_state == SMC_INIT && !smc->connect_nonblock) {
- smc_switch_to_fallback(smc);
- smc->fallback_rsn = SMC_CLC_DECL_OPTUNSUPP;
+ smc_switch_to_fallback(smc, SMC_CLC_DECL_OPTUNSUPP);
} else {
rc = -EINVAL;
}
if (sk->sk_state != SMC_INIT &&
sk->sk_state != SMC_LISTEN &&
sk->sk_state != SMC_CLOSED) {
- if (val)
+ if (val) {
+ SMC_STAT_INC(smc, ndly_cnt);
mod_delayed_work(smc->conn.lgr->tx_wq,
&smc->conn.tx_work, 0);
+ }
}
break;
case TCP_CORK:
if (sk->sk_state != SMC_INIT &&
sk->sk_state != SMC_LISTEN &&
sk->sk_state != SMC_CLOSED) {
- if (!val)
+ if (!val) {
+ SMC_STAT_INC(smc, cork_cnt);
mod_delayed_work(smc->conn.lgr->tx_wq,
&smc->conn.tx_work, 0);
+ }
}
break;
case TCP_DEFER_ACCEPT:
goto out;
}
release_sock(sk);
- if (smc->use_fallback)
+ if (smc->use_fallback) {
rc = kernel_sendpage(smc->clcsock, page, offset,
size, flags);
- else
+ } else {
+ SMC_STAT_INC(smc, sendpage_cnt);
rc = sock_no_sendpage(sock, page, offset, size, flags);
+ }
out:
return rc;
flags = MSG_DONTWAIT;
else
flags = 0;
+ SMC_STAT_INC(smc, splice_cnt);
rc = smc_rx_recvmsg(smc, NULL, pipe, len, flags);
}
out:
smc_pnet_net_exit(net);
}
+static __net_init int smc_net_stat_init(struct net *net)
+{
+ return smc_stats_init(net);
+}
+
+static void __net_exit smc_net_stat_exit(struct net *net)
+{
+ smc_stats_exit(net);
+}
+
static struct pernet_operations smc_net_ops = {
.init = smc_net_init,
.exit = smc_net_exit,
.size = sizeof(struct smc_net),
};
+static struct pernet_operations smc_net_stat_ops = {
+ .init = smc_net_stat_init,
+ .exit = smc_net_stat_exit,
+};
+
static int __init smc_init(void)
{
int rc;
if (rc)
return rc;
+ rc = register_pernet_subsys(&smc_net_stat_ops);
+ if (rc)
+ return rc;
+
smc_ism_init();
smc_clc_init();
proto_unregister(&smc_proto);
smc_pnet_exit();
smc_nl_exit();
+ unregister_pernet_subsys(&smc_net_stat_ops);
unregister_pernet_subsys(&smc_net_ops);
rcu_barrier();
}
#include "smc_close.h"
#include "smc_ism.h"
#include "smc_netlink.h"
+#include "smc_stats.h"
#define SMC_LGR_NUM_INCR 256
#define SMC_LGR_FREE_DELAY_SERV (600 * HZ)
kfree(lgr);
}
-static void smcd_unregister_all_dmbs(struct smc_link_group *lgr)
-{
- int i;
-
- for (i = 0; i < SMC_RMBE_SIZES; i++) {
- struct smc_buf_desc *buf_desc;
-
- list_for_each_entry(buf_desc, &lgr->rmbs[i], list) {
- buf_desc->len += sizeof(struct smcd_cdc_msg);
- smc_ism_unregister_dmb(lgr->smcd, buf_desc);
- }
- }
-}
-
static void smc_sk_wake_ups(struct smc_sock *smc)
{
smc->sk.sk_write_space(&smc->sk);
{
if (lgr->is_smcd) {
smc_ism_signal_shutdown(lgr);
- smcd_unregister_all_dmbs(lgr);
} else {
u32 rsn = lgr->llc_termination_rsn;
struct smc_link_group *lgr = conn->lgr;
struct list_head *buf_list;
int bufsize, bufsize_short;
+ bool is_dgraded = false;
struct mutex *lock; /* lock buffer list */
int sk_buf_size;
/* check for reusable slot in the link group */
buf_desc = smc_buf_get_slot(bufsize_short, lock, buf_list);
if (buf_desc) {
+ SMC_STAT_RMB_SIZE(smc, is_smcd, is_rmb, bufsize);
+ SMC_STAT_BUF_REUSE(smc, is_smcd, is_rmb);
memset(buf_desc->cpu_addr, 0, bufsize);
break; /* found reusable slot */
}
if (PTR_ERR(buf_desc) == -ENOMEM)
break;
- if (IS_ERR(buf_desc))
+ if (IS_ERR(buf_desc)) {
+ if (!is_dgraded) {
+ is_dgraded = true;
+ SMC_STAT_RMB_DOWNGRADED(smc, is_smcd, is_rmb);
+ }
continue;
+ }
+ SMC_STAT_RMB_ALLOC(smc, is_smcd, is_rmb);
+ SMC_STAT_RMB_SIZE(smc, is_smcd, is_rmb, bufsize);
buf_desc->used = 1;
mutex_lock(lock);
list_add(&buf_desc->list, buf_list);
return NULL;
}
+ smcd->event_wq = alloc_ordered_workqueue("ism_evt_wq-%s)",
+ WQ_MEM_RECLAIM, name);
+ if (!smcd->event_wq) {
+ kfree(smcd->conn);
+ kfree(smcd);
+ return NULL;
+ }
+
smcd->dev.parent = parent;
smcd->dev.release = smcd_release;
device_initialize(&smcd->dev);
INIT_LIST_HEAD(&smcd->vlan);
INIT_LIST_HEAD(&smcd->lgr_list);
init_waitqueue_head(&smcd->lgrs_deleted);
- smcd->event_wq = alloc_ordered_workqueue("ism_evt_wq-%s)",
- WQ_MEM_RECLAIM, name);
- if (!smcd->event_wq) {
- kfree(smcd->conn);
- kfree(smcd);
- return NULL;
- }
return smcd;
}
EXPORT_SYMBOL_GPL(smcd_alloc_dev);
int smcd_register_dev(struct smcd_dev *smcd)
{
+ int rc;
+
mutex_lock(&smcd_dev_list.mutex);
if (list_empty(&smcd_dev_list.list)) {
u8 *system_eid = NULL;
dev_name(&smcd->dev), smcd->pnetid,
smcd->pnetid_by_user ? " (user defined)" : "");
- return device_add(&smcd->dev);
+ rc = device_add(&smcd->dev);
+ if (rc) {
+ mutex_lock(&smcd_dev_list.mutex);
+ list_del(&smcd->list);
+ mutex_unlock(&smcd_dev_list.mutex);
+ }
+
+ return rc;
}
EXPORT_SYMBOL_GPL(smcd_register_dev);
mutex_unlock(&smcd_dev_list.mutex);
smcd->going_away = 1;
smc_smcd_terminate_all(smcd);
- flush_workqueue(smcd->event_wq);
destroy_workqueue(smcd->event_wq);
device_del(&smcd->dev);
#include "smc_core.h"
#include "smc_ism.h"
#include "smc_ib.h"
+#include "smc_stats.h"
#include "smc_netlink.h"
#define SMC_CMD_MAX_ATTR 1
/* can be retrieved by unprivileged users */
.dumpit = smcr_nl_get_device,
},
+ {
+ .cmd = SMC_NETLINK_GET_STATS,
+ /* can be retrieved by unprivileged users */
+ .dumpit = smc_nl_get_stats,
+ },
+ {
+ .cmd = SMC_NETLINK_GET_FBACK_STATS,
+ /* can be retrieved by unprivileged users */
+ .dumpit = smc_nl_get_fback_stats,
+ },
};
static const struct nla_policy smc_gen_nl_policy[2] = {
extern struct genl_family smc_gen_nl_family;
struct smc_nl_dmp_ctx {
- int pos[2];
+ int pos[3];
};
static inline struct smc_nl_dmp_ctx *smc_nl_dmp_ctx(struct netlink_callback *c)
#include "smc_cdc.h"
#include "smc_tx.h" /* smc_tx_consumer_update() */
#include "smc_rx.h"
+#include "smc_stats.h"
/* callback implementation to wakeup consumers blocked with smc_rx_wait().
* indirectly called by smc_cdc_msg_recv_action().
conn->urg_state == SMC_URG_READ)
return -EINVAL;
+ SMC_STAT_INC(smc, urg_data_cnt);
if (conn->urg_state == SMC_URG_VALID) {
if (!(flags & MSG_PEEK))
smc->conn.urg_state = SMC_URG_READ;
timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
target = sock_rcvlowat(sk, flags & MSG_WAITALL, len);
+ readable = atomic_read(&conn->bytes_to_rcv);
+ if (readable >= conn->rmb_desc->len)
+ SMC_STAT_RMB_RX_FULL(smc, !conn->lnk);
+
+ if (len < readable)
+ SMC_STAT_RMB_RX_SIZE_SMALL(smc, !conn->lnk);
/* we currently use 1 RMBE per RMB, so RMBE == RMB base addr */
rcvbuf_base = conn->rx_off + conn->rmb_desc->cpu_addr;
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Shared Memory Communications over RDMA (SMC-R) and RoCE
+ *
+ * SMC statistics netlink routines
+ *
+ * Copyright IBM Corp. 2021
+ *
+ * Author(s): Guvenc Gulce
+ */
+#include <linux/init.h>
+#include <linux/mutex.h>
+#include <linux/percpu.h>
+#include <linux/ctype.h>
+#include <linux/smc.h>
+#include <net/genetlink.h>
+#include <net/sock.h>
+#include "smc_netlink.h"
+#include "smc_stats.h"
+
+int smc_stats_init(struct net *net)
+{
+ net->smc.fback_rsn = kzalloc(sizeof(*net->smc.fback_rsn), GFP_KERNEL);
+ if (!net->smc.fback_rsn)
+ goto err_fback;
+ net->smc.smc_stats = alloc_percpu(struct smc_stats);
+ if (!net->smc.smc_stats)
+ goto err_stats;
+ mutex_init(&net->smc.mutex_fback_rsn);
+ return 0;
+
+err_stats:
+ kfree(net->smc.fback_rsn);
+err_fback:
+ return -ENOMEM;
+}
+
+void smc_stats_exit(struct net *net)
+{
+ kfree(net->smc.fback_rsn);
+ if (net->smc.smc_stats)
+ free_percpu(net->smc.smc_stats);
+}
+
+static int smc_nl_fill_stats_rmb_data(struct sk_buff *skb,
+ struct smc_stats *stats, int tech,
+ int type)
+{
+ struct smc_stats_rmbcnt *stats_rmb_cnt;
+ struct nlattr *attrs;
+
+ if (type == SMC_NLA_STATS_T_TX_RMB_STATS)
+ stats_rmb_cnt = &stats->smc[tech].rmb_tx;
+ else
+ stats_rmb_cnt = &stats->smc[tech].rmb_rx;
+
+ attrs = nla_nest_start(skb, type);
+ if (!attrs)
+ goto errout;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_RMB_REUSE_CNT,
+ stats_rmb_cnt->reuse_cnt,
+ SMC_NLA_STATS_RMB_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_RMB_SIZE_SM_PEER_CNT,
+ stats_rmb_cnt->buf_size_small_peer_cnt,
+ SMC_NLA_STATS_RMB_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_RMB_SIZE_SM_CNT,
+ stats_rmb_cnt->buf_size_small_cnt,
+ SMC_NLA_STATS_RMB_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_RMB_FULL_PEER_CNT,
+ stats_rmb_cnt->buf_full_peer_cnt,
+ SMC_NLA_STATS_RMB_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_RMB_FULL_CNT,
+ stats_rmb_cnt->buf_full_cnt,
+ SMC_NLA_STATS_RMB_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_RMB_ALLOC_CNT,
+ stats_rmb_cnt->alloc_cnt,
+ SMC_NLA_STATS_RMB_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_RMB_DGRADE_CNT,
+ stats_rmb_cnt->dgrade_cnt,
+ SMC_NLA_STATS_RMB_PAD))
+ goto errattr;
+
+ nla_nest_end(skb, attrs);
+ return 0;
+
+errattr:
+ nla_nest_cancel(skb, attrs);
+errout:
+ return -EMSGSIZE;
+}
+
+static int smc_nl_fill_stats_bufsize_data(struct sk_buff *skb,
+ struct smc_stats *stats, int tech,
+ int type)
+{
+ struct smc_stats_memsize *stats_pload;
+ struct nlattr *attrs;
+
+ if (type == SMC_NLA_STATS_T_TXPLOAD_SIZE)
+ stats_pload = &stats->smc[tech].tx_pd;
+ else if (type == SMC_NLA_STATS_T_RXPLOAD_SIZE)
+ stats_pload = &stats->smc[tech].rx_pd;
+ else if (type == SMC_NLA_STATS_T_TX_RMB_SIZE)
+ stats_pload = &stats->smc[tech].tx_rmbsize;
+ else if (type == SMC_NLA_STATS_T_RX_RMB_SIZE)
+ stats_pload = &stats->smc[tech].rx_rmbsize;
+ else
+ goto errout;
+
+ attrs = nla_nest_start(skb, type);
+ if (!attrs)
+ goto errout;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_PLOAD_8K,
+ stats_pload->buf[SMC_BUF_8K],
+ SMC_NLA_STATS_PLOAD_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_PLOAD_16K,
+ stats_pload->buf[SMC_BUF_16K],
+ SMC_NLA_STATS_PLOAD_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_PLOAD_32K,
+ stats_pload->buf[SMC_BUF_32K],
+ SMC_NLA_STATS_PLOAD_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_PLOAD_64K,
+ stats_pload->buf[SMC_BUF_64K],
+ SMC_NLA_STATS_PLOAD_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_PLOAD_128K,
+ stats_pload->buf[SMC_BUF_128K],
+ SMC_NLA_STATS_PLOAD_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_PLOAD_256K,
+ stats_pload->buf[SMC_BUF_256K],
+ SMC_NLA_STATS_PLOAD_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_PLOAD_512K,
+ stats_pload->buf[SMC_BUF_512K],
+ SMC_NLA_STATS_PLOAD_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_PLOAD_1024K,
+ stats_pload->buf[SMC_BUF_1024K],
+ SMC_NLA_STATS_PLOAD_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_PLOAD_G_1024K,
+ stats_pload->buf[SMC_BUF_G_1024K],
+ SMC_NLA_STATS_PLOAD_PAD))
+ goto errattr;
+
+ nla_nest_end(skb, attrs);
+ return 0;
+
+errattr:
+ nla_nest_cancel(skb, attrs);
+errout:
+ return -EMSGSIZE;
+}
+
+static int smc_nl_fill_stats_tech_data(struct sk_buff *skb,
+ struct smc_stats *stats, int tech)
+{
+ struct smc_stats_tech *smc_tech;
+ struct nlattr *attrs;
+
+ smc_tech = &stats->smc[tech];
+ if (tech == SMC_TYPE_D)
+ attrs = nla_nest_start(skb, SMC_NLA_STATS_SMCD_TECH);
+ else
+ attrs = nla_nest_start(skb, SMC_NLA_STATS_SMCR_TECH);
+
+ if (!attrs)
+ goto errout;
+ if (smc_nl_fill_stats_rmb_data(skb, stats, tech,
+ SMC_NLA_STATS_T_TX_RMB_STATS))
+ goto errattr;
+ if (smc_nl_fill_stats_rmb_data(skb, stats, tech,
+ SMC_NLA_STATS_T_RX_RMB_STATS))
+ goto errattr;
+ if (smc_nl_fill_stats_bufsize_data(skb, stats, tech,
+ SMC_NLA_STATS_T_TXPLOAD_SIZE))
+ goto errattr;
+ if (smc_nl_fill_stats_bufsize_data(skb, stats, tech,
+ SMC_NLA_STATS_T_RXPLOAD_SIZE))
+ goto errattr;
+ if (smc_nl_fill_stats_bufsize_data(skb, stats, tech,
+ SMC_NLA_STATS_T_TX_RMB_SIZE))
+ goto errattr;
+ if (smc_nl_fill_stats_bufsize_data(skb, stats, tech,
+ SMC_NLA_STATS_T_RX_RMB_SIZE))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_CLNT_V1_SUCC,
+ smc_tech->clnt_v1_succ_cnt,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_CLNT_V2_SUCC,
+ smc_tech->clnt_v2_succ_cnt,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_SRV_V1_SUCC,
+ smc_tech->srv_v1_succ_cnt,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_SRV_V2_SUCC,
+ smc_tech->srv_v2_succ_cnt,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_RX_BYTES,
+ smc_tech->rx_bytes,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_TX_BYTES,
+ smc_tech->tx_bytes,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_RX_CNT,
+ smc_tech->rx_cnt,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_TX_CNT,
+ smc_tech->tx_cnt,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_SENDPAGE_CNT,
+ smc_tech->sendpage_cnt,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_CORK_CNT,
+ smc_tech->cork_cnt,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_NDLY_CNT,
+ smc_tech->ndly_cnt,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_SPLICE_CNT,
+ smc_tech->splice_cnt,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_URG_DATA_CNT,
+ smc_tech->urg_data_cnt,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+
+ nla_nest_end(skb, attrs);
+ return 0;
+
+errattr:
+ nla_nest_cancel(skb, attrs);
+errout:
+ return -EMSGSIZE;
+}
+
+int smc_nl_get_stats(struct sk_buff *skb,
+ struct netlink_callback *cb)
+{
+ struct smc_nl_dmp_ctx *cb_ctx = smc_nl_dmp_ctx(cb);
+ struct net *net = sock_net(skb->sk);
+ struct smc_stats *stats;
+ struct nlattr *attrs;
+ int cpu, i, size;
+ void *nlh;
+ u64 *src;
+ u64 *sum;
+
+ if (cb_ctx->pos[0])
+ goto errmsg;
+ nlh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, cb->nlh->nlmsg_seq,
+ &smc_gen_nl_family, NLM_F_MULTI,
+ SMC_NETLINK_GET_STATS);
+ if (!nlh)
+ goto errmsg;
+
+ attrs = nla_nest_start(skb, SMC_GEN_STATS);
+ if (!attrs)
+ goto errnest;
+ stats = kzalloc(sizeof(*stats), GFP_KERNEL);
+ if (!stats)
+ goto erralloc;
+ size = sizeof(*stats) / sizeof(u64);
+ for_each_possible_cpu(cpu) {
+ src = (u64 *)per_cpu_ptr(net->smc.smc_stats, cpu);
+ sum = (u64 *)stats;
+ for (i = 0; i < size; i++)
+ *(sum++) += *(src++);
+ }
+ if (smc_nl_fill_stats_tech_data(skb, stats, SMC_TYPE_D))
+ goto errattr;
+ if (smc_nl_fill_stats_tech_data(skb, stats, SMC_TYPE_R))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_CLNT_HS_ERR_CNT,
+ stats->clnt_hshake_err_cnt,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_STATS_SRV_HS_ERR_CNT,
+ stats->srv_hshake_err_cnt,
+ SMC_NLA_STATS_PAD))
+ goto errattr;
+
+ nla_nest_end(skb, attrs);
+ genlmsg_end(skb, nlh);
+ cb_ctx->pos[0] = 1;
+ kfree(stats);
+ return skb->len;
+
+errattr:
+ kfree(stats);
+erralloc:
+ nla_nest_cancel(skb, attrs);
+errnest:
+ genlmsg_cancel(skb, nlh);
+errmsg:
+ return skb->len;
+}
+
+static int smc_nl_get_fback_details(struct sk_buff *skb,
+ struct netlink_callback *cb, int pos,
+ bool is_srv)
+{
+ struct smc_nl_dmp_ctx *cb_ctx = smc_nl_dmp_ctx(cb);
+ struct net *net = sock_net(skb->sk);
+ int cnt_reported = cb_ctx->pos[2];
+ struct smc_stats_fback *trgt_arr;
+ struct nlattr *attrs;
+ int rc = 0;
+ void *nlh;
+
+ if (is_srv)
+ trgt_arr = &net->smc.fback_rsn->srv[0];
+ else
+ trgt_arr = &net->smc.fback_rsn->clnt[0];
+ if (!trgt_arr[pos].fback_code)
+ return -ENODATA;
+ nlh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, cb->nlh->nlmsg_seq,
+ &smc_gen_nl_family, NLM_F_MULTI,
+ SMC_NETLINK_GET_FBACK_STATS);
+ if (!nlh)
+ goto errmsg;
+ attrs = nla_nest_start(skb, SMC_GEN_FBACK_STATS);
+ if (!attrs)
+ goto errout;
+ if (nla_put_u8(skb, SMC_NLA_FBACK_STATS_TYPE, is_srv))
+ goto errattr;
+ if (!cnt_reported) {
+ if (nla_put_u64_64bit(skb, SMC_NLA_FBACK_STATS_SRV_CNT,
+ net->smc.fback_rsn->srv_fback_cnt,
+ SMC_NLA_FBACK_STATS_PAD))
+ goto errattr;
+ if (nla_put_u64_64bit(skb, SMC_NLA_FBACK_STATS_CLNT_CNT,
+ net->smc.fback_rsn->clnt_fback_cnt,
+ SMC_NLA_FBACK_STATS_PAD))
+ goto errattr;
+ cnt_reported = 1;
+ }
+
+ if (nla_put_u32(skb, SMC_NLA_FBACK_STATS_RSN_CODE,
+ trgt_arr[pos].fback_code))
+ goto errattr;
+ if (nla_put_u16(skb, SMC_NLA_FBACK_STATS_RSN_CNT,
+ trgt_arr[pos].count))
+ goto errattr;
+
+ cb_ctx->pos[2] = cnt_reported;
+ nla_nest_end(skb, attrs);
+ genlmsg_end(skb, nlh);
+ return rc;
+
+errattr:
+ nla_nest_cancel(skb, attrs);
+errout:
+ genlmsg_cancel(skb, nlh);
+errmsg:
+ return -EMSGSIZE;
+}
+
+int smc_nl_get_fback_stats(struct sk_buff *skb, struct netlink_callback *cb)
+{
+ struct smc_nl_dmp_ctx *cb_ctx = smc_nl_dmp_ctx(cb);
+ struct net *net = sock_net(skb->sk);
+ int rc_srv = 0, rc_clnt = 0, k;
+ int skip_serv = cb_ctx->pos[1];
+ int snum = cb_ctx->pos[0];
+ bool is_srv = true;
+
+ mutex_lock(&net->smc.mutex_fback_rsn);
+ for (k = 0; k < SMC_MAX_FBACK_RSN_CNT; k++) {
+ if (k < snum)
+ continue;
+ if (!skip_serv) {
+ rc_srv = smc_nl_get_fback_details(skb, cb, k, is_srv);
+ if (rc_srv && rc_srv != -ENODATA)
+ break;
+ } else {
+ skip_serv = 0;
+ }
+ rc_clnt = smc_nl_get_fback_details(skb, cb, k, !is_srv);
+ if (rc_clnt && rc_clnt != -ENODATA) {
+ skip_serv = 1;
+ break;
+ }
+ if (rc_clnt == -ENODATA && rc_srv == -ENODATA)
+ break;
+ }
+ mutex_unlock(&net->smc.mutex_fback_rsn);
+ cb_ctx->pos[1] = skip_serv;
+ cb_ctx->pos[0] = k;
+ return skb->len;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Shared Memory Communications over RDMA (SMC-R) and RoCE
+ *
+ * Macros for SMC statistics
+ *
+ * Copyright IBM Corp. 2021
+ *
+ * Author(s): Guvenc Gulce
+ */
+
+#ifndef NET_SMC_SMC_STATS_H_
+#define NET_SMC_SMC_STATS_H_
+#include <linux/init.h>
+#include <linux/mutex.h>
+#include <linux/percpu.h>
+#include <linux/ctype.h>
+#include <linux/smc.h>
+
+#include "smc_clc.h"
+
+#define SMC_MAX_FBACK_RSN_CNT 30
+
+enum {
+ SMC_BUF_8K,
+ SMC_BUF_16K,
+ SMC_BUF_32K,
+ SMC_BUF_64K,
+ SMC_BUF_128K,
+ SMC_BUF_256K,
+ SMC_BUF_512K,
+ SMC_BUF_1024K,
+ SMC_BUF_G_1024K,
+ SMC_BUF_MAX,
+};
+
+struct smc_stats_fback {
+ int fback_code;
+ u16 count;
+};
+
+struct smc_stats_rsn {
+ struct smc_stats_fback srv[SMC_MAX_FBACK_RSN_CNT];
+ struct smc_stats_fback clnt[SMC_MAX_FBACK_RSN_CNT];
+ u64 srv_fback_cnt;
+ u64 clnt_fback_cnt;
+};
+
+struct smc_stats_rmbcnt {
+ u64 buf_size_small_peer_cnt;
+ u64 buf_size_small_cnt;
+ u64 buf_full_peer_cnt;
+ u64 buf_full_cnt;
+ u64 reuse_cnt;
+ u64 alloc_cnt;
+ u64 dgrade_cnt;
+};
+
+struct smc_stats_memsize {
+ u64 buf[SMC_BUF_MAX];
+};
+
+struct smc_stats_tech {
+ struct smc_stats_memsize tx_rmbsize;
+ struct smc_stats_memsize rx_rmbsize;
+ struct smc_stats_memsize tx_pd;
+ struct smc_stats_memsize rx_pd;
+ struct smc_stats_rmbcnt rmb_tx;
+ struct smc_stats_rmbcnt rmb_rx;
+ u64 clnt_v1_succ_cnt;
+ u64 clnt_v2_succ_cnt;
+ u64 srv_v1_succ_cnt;
+ u64 srv_v2_succ_cnt;
+ u64 sendpage_cnt;
+ u64 urg_data_cnt;
+ u64 splice_cnt;
+ u64 cork_cnt;
+ u64 ndly_cnt;
+ u64 rx_bytes;
+ u64 tx_bytes;
+ u64 rx_cnt;
+ u64 tx_cnt;
+};
+
+struct smc_stats {
+ struct smc_stats_tech smc[2];
+ u64 clnt_hshake_err_cnt;
+ u64 srv_hshake_err_cnt;
+};
+
+#define SMC_STAT_PAYLOAD_SUB(_smc_stats, _tech, key, _len, _rc) \
+do { \
+ typeof(_smc_stats) stats = (_smc_stats); \
+ typeof(_tech) t = (_tech); \
+ typeof(_len) l = (_len); \
+ int _pos = fls64((l) >> 13); \
+ typeof(_rc) r = (_rc); \
+ int m = SMC_BUF_MAX - 1; \
+ this_cpu_inc((*stats).smc[t].key ## _cnt); \
+ if (r <= 0) \
+ break; \
+ _pos = (_pos < m) ? ((l == 1 << (_pos + 12)) ? _pos - 1 : _pos) : m; \
+ this_cpu_inc((*stats).smc[t].key ## _pd.buf[_pos]); \
+ this_cpu_add((*stats).smc[t].key ## _bytes, r); \
+} \
+while (0)
+
+#define SMC_STAT_TX_PAYLOAD(_smc, length, rcode) \
+do { \
+ typeof(_smc) __smc = _smc; \
+ struct net *_net = sock_net(&__smc->sk); \
+ struct smc_stats __percpu *_smc_stats = _net->smc.smc_stats; \
+ typeof(length) _len = (length); \
+ typeof(rcode) _rc = (rcode); \
+ bool is_smcd = !__smc->conn.lnk; \
+ if (is_smcd) \
+ SMC_STAT_PAYLOAD_SUB(_smc_stats, SMC_TYPE_D, tx, _len, _rc); \
+ else \
+ SMC_STAT_PAYLOAD_SUB(_smc_stats, SMC_TYPE_R, tx, _len, _rc); \
+} \
+while (0)
+
+#define SMC_STAT_RX_PAYLOAD(_smc, length, rcode) \
+do { \
+ typeof(_smc) __smc = _smc; \
+ struct net *_net = sock_net(&__smc->sk); \
+ struct smc_stats __percpu *_smc_stats = _net->smc.smc_stats; \
+ typeof(length) _len = (length); \
+ typeof(rcode) _rc = (rcode); \
+ bool is_smcd = !__smc->conn.lnk; \
+ if (is_smcd) \
+ SMC_STAT_PAYLOAD_SUB(_smc_stats, SMC_TYPE_D, rx, _len, _rc); \
+ else \
+ SMC_STAT_PAYLOAD_SUB(_smc_stats, SMC_TYPE_R, rx, _len, _rc); \
+} \
+while (0)
+
+#define SMC_STAT_RMB_SIZE_SUB(_smc_stats, _tech, k, _len) \
+do { \
+ typeof(_len) _l = (_len); \
+ typeof(_tech) t = (_tech); \
+ int _pos = fls((_l) >> 13); \
+ int m = SMC_BUF_MAX - 1; \
+ _pos = (_pos < m) ? ((_l == 1 << (_pos + 12)) ? _pos - 1 : _pos) : m; \
+ this_cpu_inc((*(_smc_stats)).smc[t].k ## _rmbsize.buf[_pos]); \
+} \
+while (0)
+
+#define SMC_STAT_RMB_SUB(_smc_stats, type, t, key) \
+ this_cpu_inc((*(_smc_stats)).smc[t].rmb ## _ ## key.type ## _cnt)
+
+#define SMC_STAT_RMB_SIZE(_smc, _is_smcd, _is_rx, _len) \
+do { \
+ struct net *_net = sock_net(&(_smc)->sk); \
+ struct smc_stats __percpu *_smc_stats = _net->smc.smc_stats; \
+ typeof(_is_smcd) is_d = (_is_smcd); \
+ typeof(_is_rx) is_r = (_is_rx); \
+ typeof(_len) l = (_len); \
+ if ((is_d) && (is_r)) \
+ SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_D, rx, l); \
+ if ((is_d) && !(is_r)) \
+ SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_D, tx, l); \
+ if (!(is_d) && (is_r)) \
+ SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_R, rx, l); \
+ if (!(is_d) && !(is_r)) \
+ SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_R, tx, l); \
+} \
+while (0)
+
+#define SMC_STAT_RMB(_smc, type, _is_smcd, _is_rx) \
+do { \
+ struct net *net = sock_net(&(_smc)->sk); \
+ struct smc_stats __percpu *_smc_stats = net->smc.smc_stats; \
+ typeof(_is_smcd) is_d = (_is_smcd); \
+ typeof(_is_rx) is_r = (_is_rx); \
+ if ((is_d) && (is_r)) \
+ SMC_STAT_RMB_SUB(_smc_stats, type, SMC_TYPE_D, rx); \
+ if ((is_d) && !(is_r)) \
+ SMC_STAT_RMB_SUB(_smc_stats, type, SMC_TYPE_D, tx); \
+ if (!(is_d) && (is_r)) \
+ SMC_STAT_RMB_SUB(_smc_stats, type, SMC_TYPE_R, rx); \
+ if (!(is_d) && !(is_r)) \
+ SMC_STAT_RMB_SUB(_smc_stats, type, SMC_TYPE_R, tx); \
+} \
+while (0)
+
+#define SMC_STAT_BUF_REUSE(smc, is_smcd, is_rx) \
+ SMC_STAT_RMB(smc, reuse, is_smcd, is_rx)
+
+#define SMC_STAT_RMB_ALLOC(smc, is_smcd, is_rx) \
+ SMC_STAT_RMB(smc, alloc, is_smcd, is_rx)
+
+#define SMC_STAT_RMB_DOWNGRADED(smc, is_smcd, is_rx) \
+ SMC_STAT_RMB(smc, dgrade, is_smcd, is_rx)
+
+#define SMC_STAT_RMB_TX_PEER_FULL(smc, is_smcd) \
+ SMC_STAT_RMB(smc, buf_full_peer, is_smcd, false)
+
+#define SMC_STAT_RMB_TX_FULL(smc, is_smcd) \
+ SMC_STAT_RMB(smc, buf_full, is_smcd, false)
+
+#define SMC_STAT_RMB_TX_PEER_SIZE_SMALL(smc, is_smcd) \
+ SMC_STAT_RMB(smc, buf_size_small_peer, is_smcd, false)
+
+#define SMC_STAT_RMB_TX_SIZE_SMALL(smc, is_smcd) \
+ SMC_STAT_RMB(smc, buf_size_small, is_smcd, false)
+
+#define SMC_STAT_RMB_RX_SIZE_SMALL(smc, is_smcd) \
+ SMC_STAT_RMB(smc, buf_size_small, is_smcd, true)
+
+#define SMC_STAT_RMB_RX_FULL(smc, is_smcd) \
+ SMC_STAT_RMB(smc, buf_full, is_smcd, true)
+
+#define SMC_STAT_INC(_smc, type) \
+do { \
+ typeof(_smc) __smc = _smc; \
+ bool is_smcd = !(__smc)->conn.lnk; \
+ struct net *net = sock_net(&(__smc)->sk); \
+ struct smc_stats __percpu *smc_stats = net->smc.smc_stats; \
+ if ((is_smcd)) \
+ this_cpu_inc(smc_stats->smc[SMC_TYPE_D].type); \
+ else \
+ this_cpu_inc(smc_stats->smc[SMC_TYPE_R].type); \
+} \
+while (0)
+
+#define SMC_STAT_CLNT_SUCC_INC(net, _aclc) \
+do { \
+ typeof(_aclc) acl = (_aclc); \
+ bool is_v2 = (acl->hdr.version == SMC_V2); \
+ bool is_smcd = (acl->hdr.typev1 == SMC_TYPE_D); \
+ struct smc_stats __percpu *smc_stats = (net)->smc.smc_stats; \
+ if (is_v2 && is_smcd) \
+ this_cpu_inc(smc_stats->smc[SMC_TYPE_D].clnt_v2_succ_cnt); \
+ else if (is_v2 && !is_smcd) \
+ this_cpu_inc(smc_stats->smc[SMC_TYPE_R].clnt_v2_succ_cnt); \
+ else if (!is_v2 && is_smcd) \
+ this_cpu_inc(smc_stats->smc[SMC_TYPE_D].clnt_v1_succ_cnt); \
+ else if (!is_v2 && !is_smcd) \
+ this_cpu_inc(smc_stats->smc[SMC_TYPE_R].clnt_v1_succ_cnt); \
+} \
+while (0)
+
+#define SMC_STAT_SERV_SUCC_INC(net, _ini) \
+do { \
+ typeof(_ini) i = (_ini); \
+ bool is_v2 = (i->smcd_version & SMC_V2); \
+ bool is_smcd = (i->is_smcd); \
+ typeof(net->smc.smc_stats) smc_stats = (net)->smc.smc_stats; \
+ if (is_v2 && is_smcd) \
+ this_cpu_inc(smc_stats->smc[SMC_TYPE_D].srv_v2_succ_cnt); \
+ else if (is_v2 && !is_smcd) \
+ this_cpu_inc(smc_stats->smc[SMC_TYPE_R].srv_v2_succ_cnt); \
+ else if (!is_v2 && is_smcd) \
+ this_cpu_inc(smc_stats->smc[SMC_TYPE_D].srv_v1_succ_cnt); \
+ else if (!is_v2 && !is_smcd) \
+ this_cpu_inc(smc_stats->smc[SMC_TYPE_R].srv_v1_succ_cnt); \
+} \
+while (0)
+
+int smc_nl_get_stats(struct sk_buff *skb, struct netlink_callback *cb);
+int smc_nl_get_fback_stats(struct sk_buff *skb, struct netlink_callback *cb);
+int smc_stats_init(struct net *net);
+void smc_stats_exit(struct net *net);
+
+#endif /* NET_SMC_SMC_STATS_H_ */
#include "smc_close.h"
#include "smc_ism.h"
#include "smc_tx.h"
+#include "smc_stats.h"
#define SMC_TX_WORK_DELAY 0
#define SMC_TX_CORK_DELAY (HZ >> 2) /* 250 ms */
/* similar to sk_stream_write_space */
if (atomic_read(&smc->conn.sndbuf_space) && sock) {
+ if (test_bit(SOCK_NOSPACE, &sock->flags))
+ SMC_STAT_RMB_TX_FULL(smc, !smc->conn.lnk);
clear_bit(SOCK_NOSPACE, &sock->flags);
rcu_read_lock();
wq = rcu_dereference(sk->sk_wq);
goto out_err;
}
+ if (sk->sk_state == SMC_INIT)
+ return -ENOTCONN;
+
+ if (len > conn->sndbuf_desc->len)
+ SMC_STAT_RMB_TX_SIZE_SMALL(smc, !conn->lnk);
+
+ if (len > conn->peer_rmbe_size)
+ SMC_STAT_RMB_TX_PEER_SIZE_SMALL(smc, !conn->lnk);
+
+ if (msg->msg_flags & MSG_OOB)
+ SMC_STAT_INC(smc, urg_data_cnt);
+
while (msg_data_left(msg)) {
- if (sk->sk_state == SMC_INIT)
- return -ENOTCONN;
if (smc->sk.sk_shutdown & SEND_SHUTDOWN ||
(smc->sk.sk_err == ECONNABORTED) ||
conn->killed)
/* destination: RMBE */
/* cf. snd_wnd */
rmbespace = atomic_read(&conn->peer_rmbe_space);
- if (rmbespace <= 0)
+ if (rmbespace <= 0) {
+ struct smc_sock *smc = container_of(conn, struct smc_sock,
+ conn);
+ SMC_STAT_RMB_TX_PEER_FULL(smc, !conn->lnk);
return 0;
+ }
smc_curs_copy(&prod, &conn->local_tx_ctrl.prod, conn);
smc_curs_copy(&cons, &conn->local_rx_ctrl.cons, conn);
.show_fdinfo = sock_show_fdinfo,
};
+static const char * const pf_family_names[] = {
+ [PF_UNSPEC] = "PF_UNSPEC",
+ [PF_UNIX] = "PF_UNIX/PF_LOCAL",
+ [PF_INET] = "PF_INET",
+ [PF_AX25] = "PF_AX25",
+ [PF_IPX] = "PF_IPX",
+ [PF_APPLETALK] = "PF_APPLETALK",
+ [PF_NETROM] = "PF_NETROM",
+ [PF_BRIDGE] = "PF_BRIDGE",
+ [PF_ATMPVC] = "PF_ATMPVC",
+ [PF_X25] = "PF_X25",
+ [PF_INET6] = "PF_INET6",
+ [PF_ROSE] = "PF_ROSE",
+ [PF_DECnet] = "PF_DECnet",
+ [PF_NETBEUI] = "PF_NETBEUI",
+ [PF_SECURITY] = "PF_SECURITY",
+ [PF_KEY] = "PF_KEY",
+ [PF_NETLINK] = "PF_NETLINK/PF_ROUTE",
+ [PF_PACKET] = "PF_PACKET",
+ [PF_ASH] = "PF_ASH",
+ [PF_ECONET] = "PF_ECONET",
+ [PF_ATMSVC] = "PF_ATMSVC",
+ [PF_RDS] = "PF_RDS",
+ [PF_SNA] = "PF_SNA",
+ [PF_IRDA] = "PF_IRDA",
+ [PF_PPPOX] = "PF_PPPOX",
+ [PF_WANPIPE] = "PF_WANPIPE",
+ [PF_LLC] = "PF_LLC",
+ [PF_IB] = "PF_IB",
+ [PF_MPLS] = "PF_MPLS",
+ [PF_CAN] = "PF_CAN",
+ [PF_TIPC] = "PF_TIPC",
+ [PF_BLUETOOTH] = "PF_BLUETOOTH",
+ [PF_IUCV] = "PF_IUCV",
+ [PF_RXRPC] = "PF_RXRPC",
+ [PF_ISDN] = "PF_ISDN",
+ [PF_PHONET] = "PF_PHONET",
+ [PF_IEEE802154] = "PF_IEEE802154",
+ [PF_CAIF] = "PF_CAIF",
+ [PF_ALG] = "PF_ALG",
+ [PF_NFC] = "PF_NFC",
+ [PF_VSOCK] = "PF_VSOCK",
+ [PF_KCM] = "PF_KCM",
+ [PF_QIPCRTR] = "PF_QIPCRTR",
+ [PF_SMC] = "PF_SMC",
+ [PF_XDP] = "PF_XDP",
+};
+
/*
* The protocol list. Each protocol is registered in here.
*/
* what to do with it - that's up to the protocol still.
*/
-/**
- * get_net_ns - increment the refcount of the network namespace
- * @ns: common namespace (net)
- *
- * Returns the net's common namespace.
- */
-
-struct ns_common *get_net_ns(struct ns_common *ns)
-{
- return &get_net(container_of(ns, struct net, ns))->ns;
-}
-EXPORT_SYMBOL_GPL(get_net_ns);
-
static long sock_ioctl(struct file *file, unsigned cmd, unsigned long arg)
{
struct socket *sock;
}
spin_unlock(&net_family_lock);
- pr_info("NET: Registered protocol family %d\n", ops->family);
+ pr_info("NET: Registered %s protocol family\n", pf_family_names[ops->family]);
return err;
}
EXPORT_SYMBOL(sock_register);
synchronize_rcu();
- pr_info("NET: Unregistered protocol family %d\n", family);
+ pr_info("NET: Unregistered %s protocol family\n", pf_family_names[family]);
}
EXPORT_SYMBOL(sock_unregister);
return;
}
- /*
- * Even though there was an error, we may have acquired
- * a request slot somehow. Make sure not to leak it.
- */
- if (task->tk_rqstp)
- xprt_release(task);
-
switch (status) {
case -ENOMEM:
rpc_delay(task, HZ >> 2);
static void xprt_init(struct rpc_xprt *xprt, struct net *net);
static __be32 xprt_alloc_xid(struct rpc_xprt *xprt);
static void xprt_destroy(struct rpc_xprt *xprt);
+static void xprt_request_init(struct rpc_task *task);
static DEFINE_SPINLOCK(xprt_list_lock);
static LIST_HEAD(xprt_list);
spin_unlock(&xprt->queue_lock);
}
-static void xprt_add_backlog(struct rpc_xprt *xprt, struct rpc_task *task)
+static void xprt_complete_request_init(struct rpc_task *task)
+{
+ if (task->tk_rqstp)
+ xprt_request_init(task);
+}
+
+void xprt_add_backlog(struct rpc_xprt *xprt, struct rpc_task *task)
{
set_bit(XPRT_CONGESTED, &xprt->state);
- rpc_sleep_on(&xprt->backlog, task, NULL);
+ rpc_sleep_on(&xprt->backlog, task, xprt_complete_request_init);
+}
+EXPORT_SYMBOL_GPL(xprt_add_backlog);
+
+static bool __xprt_set_rq(struct rpc_task *task, void *data)
+{
+ struct rpc_rqst *req = data;
+
+ if (task->tk_rqstp == NULL) {
+ memset(req, 0, sizeof(*req)); /* mark unused */
+ task->tk_rqstp = req;
+ return true;
+ }
+ return false;
}
-static void xprt_wake_up_backlog(struct rpc_xprt *xprt)
+bool xprt_wake_up_backlog(struct rpc_xprt *xprt, struct rpc_rqst *req)
{
- if (rpc_wake_up_next(&xprt->backlog) == NULL)
+ if (rpc_wake_up_first(&xprt->backlog, __xprt_set_rq, req) == NULL) {
clear_bit(XPRT_CONGESTED, &xprt->state);
+ return false;
+ }
+ return true;
}
+EXPORT_SYMBOL_GPL(xprt_wake_up_backlog);
static bool xprt_throttle_congested(struct rpc_xprt *xprt, struct rpc_task *task)
{
goto out;
spin_lock(&xprt->reserve_lock);
if (test_bit(XPRT_CONGESTED, &xprt->state)) {
- rpc_sleep_on(&xprt->backlog, task, NULL);
+ xprt_add_backlog(xprt, task);
ret = true;
}
spin_unlock(&xprt->reserve_lock);
void xprt_free_slot(struct rpc_xprt *xprt, struct rpc_rqst *req)
{
spin_lock(&xprt->reserve_lock);
- if (!xprt_dynamic_free_slot(xprt, req)) {
+ if (!xprt_wake_up_backlog(xprt, req) &&
+ !xprt_dynamic_free_slot(xprt, req)) {
memset(req, 0, sizeof(*req)); /* mark unused */
list_add(&req->rq_list, &xprt->free);
}
- xprt_wake_up_backlog(xprt);
spin_unlock(&xprt->reserve_lock);
}
EXPORT_SYMBOL_GPL(xprt_free_slot);
xdr_free_bvec(&req->rq_snd_buf);
if (req->rq_cred != NULL)
put_rpccred(req->rq_cred);
- task->tk_rqstp = NULL;
if (req->rq_release_snd_buf)
req->rq_release_snd_buf(req);
+ task->tk_rqstp = NULL;
if (likely(!bc_prealloc(req)))
xprt->ops->free_slot(xprt, req);
else
return false;
}
-/* The tail iovec might not reside in the same page as the
- * head iovec.
+/* The tail iovec may include an XDR pad for the page list,
+ * as well as additional content, and may not reside in the
+ * same page as the head iovec.
*/
static bool rpcrdma_prepare_tail_iov(struct rpcrdma_req *req,
struct xdr_buf *xdr,
struct rpcrdma_req *req,
struct xdr_buf *xdr)
{
- struct kvec *tail = &xdr->tail[0];
-
if (!rpcrdma_prepare_head_iov(r_xprt, req, xdr->head[0].iov_len))
return false;
- /* If there is a Read chunk, the page list is handled
+ /* If there is a Read chunk, the page list is being handled
* via explicit RDMA, and thus is skipped here.
*/
- if (tail->iov_len) {
- if (!rpcrdma_prepare_tail_iov(req, xdr,
- offset_in_page(tail->iov_base),
- tail->iov_len))
+ /* Do not include the tail if it is only an XDR pad */
+ if (xdr->tail[0].iov_len > 3) {
+ unsigned int page_base, len;
+
+ /* If the content in the page list is an odd length,
+ * xdr_write_pages() adds a pad at the beginning of
+ * the tail iovec. Force the tail's non-pad content to
+ * land at the next XDR position in the Send message.
+ */
+ page_base = offset_in_page(xdr->tail[0].iov_base);
+ len = xdr->tail[0].iov_len;
+ page_base += len & 3;
+ len -= len & 3;
+ if (!rpcrdma_prepare_tail_iov(req, xdr, page_base, len))
return false;
kref_get(&req->rl_kref);
}
return;
out_sleep:
- set_bit(XPRT_CONGESTED, &xprt->state);
- rpc_sleep_on(&xprt->backlog, task, NULL);
task->tk_status = -EAGAIN;
+ xprt_add_backlog(xprt, task);
}
/**
struct rpcrdma_xprt *r_xprt =
container_of(xprt, struct rpcrdma_xprt, rx_xprt);
- memset(rqst, 0, sizeof(*rqst));
- rpcrdma_buffer_put(&r_xprt->rx_buf, rpcr_to_rdmar(rqst));
- if (unlikely(!rpc_wake_up_next(&xprt->backlog)))
- clear_bit(XPRT_CONGESTED, &xprt->state);
+ rpcrdma_reply_put(&r_xprt->rx_buf, rpcr_to_rdmar(rqst));
+ if (!xprt_wake_up_backlog(xprt, rqst)) {
+ memset(rqst, 0, sizeof(*rqst));
+ rpcrdma_buffer_put(&r_xprt->rx_buf, rpcr_to_rdmar(rqst));
+ }
}
static bool rpcrdma_check_regbuf(struct rpcrdma_xprt *r_xprt,
}
/**
+ * rpcrdma_reply_put - Put reply buffers back into pool
+ * @buffers: buffer pool
+ * @req: object to return
+ *
+ */
+void rpcrdma_reply_put(struct rpcrdma_buffer *buffers, struct rpcrdma_req *req)
+{
+ if (req->rl_reply) {
+ rpcrdma_rep_put(buffers, req->rl_reply);
+ req->rl_reply = NULL;
+ }
+}
+
+/**
* rpcrdma_buffer_get - Get a request buffer
* @buffers: Buffer pool from which to obtain a buffer
*
*/
void rpcrdma_buffer_put(struct rpcrdma_buffer *buffers, struct rpcrdma_req *req)
{
- if (req->rl_reply)
- rpcrdma_rep_put(buffers, req->rl_reply);
- req->rl_reply = NULL;
+ rpcrdma_reply_put(buffers, req);
spin_lock(&buffers->rb_lock);
list_add(&req->rl_list, &buffers->rb_send_bufs);
void rpcrdma_buffer_put(struct rpcrdma_buffer *buffers,
struct rpcrdma_req *req);
void rpcrdma_rep_put(struct rpcrdma_buffer *buf, struct rpcrdma_rep *rep);
+void rpcrdma_reply_put(struct rpcrdma_buffer *buffers, struct rpcrdma_req *req);
bool rpcrdma_regbuf_realloc(struct rpcrdma_regbuf *rb, size_t size,
gfp_t flags);
kernel_sock_shutdown(transport->sock, SHUT_RDWR);
return -ENOTCONN;
}
+ if (!transport->inet)
+ return -ENOTCONN;
xs_pktdump("packet data:",
req->rq_svec->iov_base,
tn->trial_addr = 0;
tn->addr_trial_end = 0;
tn->capabilities = TIPC_NODE_CAPABILITIES;
- INIT_WORK(&tn->final_work.work, tipc_net_finalize_work);
+ INIT_WORK(&tn->work, tipc_net_finalize_work);
memset(tn->node_id, 0, sizeof(tn->node_id));
memset(tn->node_id_string, 0, sizeof(tn->node_id_string));
tn->mon_threshold = TIPC_DEF_MON_THRESHOLD;
tipc_detach_loopback(net);
/* Make sure the tipc_net_finalize_work() finished */
- cancel_work_sync(&tn->final_work.work);
+ cancel_work_sync(&tn->work);
tipc_net_stop(net);
tipc_bcast_stop(net);
#ifdef CONFIG_TIPC_CRYPTO
tipc_crypto_stop(&tipc_net(net)->crypto_tx);
#endif
+ while (atomic_read(&tn->wq_count))
+ cond_resched();
}
static void __net_exit tipc_pernet_pre_exit(struct net *net)
extern int sysctl_tipc_rmem[3] __read_mostly;
extern int sysctl_tipc_named_timeout __read_mostly;
-struct tipc_net_work {
- struct work_struct work;
- struct net *net;
- u32 addr;
-};
-
struct tipc_net {
u8 node_id[NODE_ID_LEN];
u32 node_addr;
struct tipc_crypto *crypto_tx;
#endif
/* Work item for net finalize */
- struct tipc_net_work final_work;
+ struct work_struct work;
+ /* The numbers of work queues in schedule */
+ atomic_t wq_count;
};
static inline struct tipc_net *tipc_net(struct net *net)
/* Apply trial address if we just left trial period */
if (!trial && !self) {
- tipc_sched_net_finalize(net, tn->trial_addr);
+ schedule_work(&tn->work);
msg_set_prevnode(buf_msg(d->skb), tn->trial_addr);
msg_set_type(buf_msg(d->skb), DSC_REQ_MSG);
}
if (!time_before(jiffies, tn->addr_trial_end) && !tipc_own_addr(net)) {
mod_timer(&d->timer, jiffies + TIPC_DISC_INIT);
spin_unlock_bh(&d->lock);
- tipc_sched_net_finalize(net, tn->trial_addr);
+ schedule_work(&tn->work);
return;
}
return l->net_plane;
}
+struct net *tipc_link_net(struct tipc_link *l)
+{
+ return l->net;
+}
+
void tipc_link_update_caps(struct tipc_link *l, u16 capabilities)
{
l->peer_caps = capabilities;
skb = tipc_msg_create(SOCK_WAKEUP, 0, INT_H_SIZE, 0,
dnode, l->addr, dport, 0, 0);
if (!skb)
- return -ENOBUFS;
+ return -ENOMEM;
msg_set_dest_droppable(buf_msg(skb), true);
TIPC_SKB_CB(skb)->chain_imp = msg_importance(hdr);
skb_queue_tail(&l->wakeupq, skb);
*
* Consumes the buffer chain.
* Messages at TIPC_SYSTEM_IMPORTANCE are always accepted
- * Return: 0 if success, or errno: -ELINKCONG, -EMSGSIZE or -ENOBUFS
+ * Return: 0 if success, or errno: -ELINKCONG, -EMSGSIZE or -ENOBUFS or -ENOMEM
*/
int tipc_link_xmit(struct tipc_link *l, struct sk_buff_head *list,
struct sk_buff_head *xmitq)
if (!_skb) {
kfree_skb(skb);
__skb_queue_purge(list);
- return -ENOBUFS;
+ return -ENOMEM;
}
__skb_queue_tail(transmq, skb);
tipc_link_set_skb_retransmit_time(skb, l);
int tipc_link_bc_nack_rcv(struct tipc_link *l, struct sk_buff *skb,
struct sk_buff_head *xmitq);
bool tipc_link_too_silent(struct tipc_link *l);
+struct net *tipc_link_net(struct tipc_link *l);
#endif
if (unlikely(head))
goto err;
*buf = NULL;
+ if (skb_has_frag_list(frag) && __skb_linearize(frag))
+ goto err;
frag = skb_unshare(frag, GFP_ATOMIC);
if (unlikely(!frag))
goto err;
head = *headbuf = frag;
TIPC_SKB_CB(head)->tail = NULL;
- if (skb_is_nonlinear(head)) {
- skb_walk_frags(head, tail) {
- TIPC_SKB_CB(head)->tail = tail;
- }
- } else {
- skb_frag_list_init(head);
- }
return 0;
}
* Returns a list of local sockets
*/
void tipc_nametbl_lookup_mcast_sockets(struct net *net, struct tipc_uaddr *ua,
- bool exact, struct list_head *dports)
+ struct list_head *dports)
{
struct service_range *sr;
struct tipc_service *sc;
struct publication *p;
- u32 scope = ua->scope;
+ u8 scope = ua->scope;
rcu_read_lock();
sc = tipc_service_find(net, ua);
spin_lock_bh(&sc->lock);
service_range_foreach_match(sr, sc, ua->sr.lower, ua->sr.upper) {
list_for_each_entry(p, &sr->local_publ, local_publ) {
- if (p->scope == scope || (!exact && p->scope < scope))
+ if (scope == p->scope || scope == TIPC_ANY_SCOPE)
tipc_dest_push(dports, 0, p->sk.ref);
}
}
#define TIPC_PUBL_SCOPE_NUM (TIPC_NODE_SCOPE + 1)
#define TIPC_NAMETBL_SIZE 1024 /* must be a power of 2 */
+#define TIPC_ANY_SCOPE 10 /* Both node and cluster scope will match */
+
/**
* struct publication - info about a published service address or range
* @sr: service range represented by this publication
bool tipc_nametbl_lookup_anycast(struct net *net, struct tipc_uaddr *ua,
struct tipc_socket_addr *sk);
void tipc_nametbl_lookup_mcast_sockets(struct net *net, struct tipc_uaddr *ua,
- bool exact, struct list_head *dports);
+ struct list_head *dports);
void tipc_nametbl_lookup_mcast_nodes(struct net *net, struct tipc_uaddr *ua,
struct tipc_nlist *nodes);
bool tipc_nametbl_lookup_group(struct net *net, struct tipc_uaddr *ua,
#include "socket.h"
#include "node.h"
#include "bcast.h"
+#include "link.h"
#include "netlink.h"
#include "monitor.h"
void tipc_net_finalize_work(struct work_struct *work)
{
- struct tipc_net_work *fwork;
+ struct tipc_net *tn = container_of(work, struct tipc_net, work);
- fwork = container_of(work, struct tipc_net_work, work);
- tipc_net_finalize(fwork->net, fwork->addr);
-}
-
-void tipc_sched_net_finalize(struct net *net, u32 addr)
-{
- struct tipc_net *tn = tipc_net(net);
-
- tn->final_work.net = net;
- tn->final_work.addr = addr;
- schedule_work(&tn->final_work.work);
+ tipc_net_finalize(tipc_link_net(tn->bcl), tn->trial_addr);
}
void tipc_net_stop(struct net *net)
write_unlock_bh(&n->lock);
if (flags & TIPC_NOTIFY_NODE_DOWN)
- tipc_publ_notify(net, publ_list, n->addr, n->capabilities);
+ tipc_publ_notify(net, publ_list, sk.node, n->capabilities);
if (flags & TIPC_NOTIFY_NODE_UP)
- tipc_named_node_up(net, n->addr, n->capabilities);
+ tipc_named_node_up(net, sk.node, n->capabilities);
if (flags & TIPC_NOTIFY_LINK_UP) {
- tipc_mon_peer_up(net, n->addr, bearer_id);
- tipc_nametbl_publish(net, &ua, &sk, n->link_id);
+ tipc_mon_peer_up(net, sk.node, bearer_id);
+ tipc_nametbl_publish(net, &ua, &sk, sk.ref);
}
if (flags & TIPC_NOTIFY_LINK_DOWN) {
- tipc_mon_peer_down(net, n->addr, bearer_id);
- tipc_nametbl_withdraw(net, &ua, &sk, n->link_id);
+ tipc_mon_peer_down(net, sk.node, bearer_id);
+ tipc_nametbl_withdraw(net, &ua, &sk, sk.ref);
}
}
/* Peer has changed i/f address without rebooting.
* If so, the link will reset soon, and the next
* discovery will be accepted. So we can ignore it.
- * It may also be an cloned or malicious peer having
+ * It may also be a cloned or malicious peer having
* chosen the same node address and signature as an
* existing one.
* Ignore requests until the link goes down, if ever.
/**
* struct tipc_sock - TIPC socket structure
* @sk: socket - interacts with 'port' and with user via the socket API
- * @conn_type: TIPC type used when connection was established
- * @conn_instance: TIPC instance used when connection was established
- * @published: non-zero if port has one or more associated names
* @max_pkt: maximum packet size "hint" used when building messages sent by port
* @maxnagle: maximum size of msg which can be subject to nagle
* @portid: unique port identity in TIPC socket hash table
* @expect_ack: whether this TIPC socket is expecting an ack
* @nodelay: setsockopt() TIPC_NODELAY setting
* @group_is_open: TIPC socket group is fully open (FIXME)
+ * @published: true if port has one or more associated names
+ * @conn_addrtype: address type used when establishing connection
*/
struct tipc_sock {
struct sock sk;
- u32 conn_type;
- u32 conn_instance;
u32 max_pkt;
u32 maxnagle;
u32 portid;
bool nodelay;
bool group_is_open;
bool published;
+ u8 conn_addrtype;
};
static int tipc_sk_backlog_rcv(struct sock *sk, struct sk_buff *skb);
* @skaddr: socket address describing name(s) and desired operation
* @alen: size of socket address data structure
*
- * Name and name sequence binding is indicated using a positive scope value;
+ * Name and name sequence binding are indicated using a positive scope value;
* a negative scope value unbinds the specified name. Specifying no name
* (i.e. a socket address length of 0) unbinds all names from the socket.
*
struct tipc_msg *hdr;
struct tipc_uaddr ua;
int user, mtyp, hlen;
- bool exact;
__skb_queue_head_init(&tmpq);
INIT_LIST_HEAD(&dports);
ua.addrtype = TIPC_SERVICE_RANGE;
+ /* tipc_skb_peek() increments the head skb's reference counter */
skb = tipc_skb_peek(arrvq, &inputq->lock);
for (; skb; skb = tipc_skb_peek(arrvq, &inputq->lock)) {
hdr = buf_msg(skb);
hlen = skb_headroom(skb) + msg_hdr_sz(hdr);
onode = msg_orignode(hdr);
ua.sr.type = msg_nametype(hdr);
+ ua.sr.lower = msg_namelower(hdr);
+ ua.sr.upper = msg_nameupper(hdr);
+ if (onode == self)
+ ua.scope = TIPC_ANY_SCOPE;
+ else
+ ua.scope = TIPC_CLUSTER_SCOPE;
if (mtyp == TIPC_GRP_UCAST_MSG || user == GROUP_PROTOCOL) {
spin_lock_bh(&inputq->lock);
ua.sr.lower = 0;
ua.sr.upper = ~0;
ua.scope = msg_lookup_scope(hdr);
- exact = true;
- } else {
- /* TIPC_NODE_SCOPE means "any scope" in this context */
- if (onode == self)
- ua.scope = TIPC_NODE_SCOPE;
- else
- ua.scope = TIPC_CLUSTER_SCOPE;
- exact = false;
- ua.sr.lower = msg_namelower(hdr);
- ua.sr.upper = msg_nameupper(hdr);
}
/* Create destination port list: */
- tipc_nametbl_lookup_mcast_sockets(net, &ua, exact, &dports);
+ tipc_nametbl_lookup_mcast_sockets(net, &ua, &dports);
/* Clone message per destination */
while (tipc_dest_pop(&dports, NULL, &portid)) {
}
pr_warn("Failed to clone mcast rcv buffer\n");
}
- /* Append to inputq if not already done by other thread */
+ /* Append clones to inputq only if skb is still head of arrvq */
spin_lock_bh(&inputq->lock);
if (skb_peek(arrvq) == skb) {
skb_queue_splice_tail_init(&tmpq, inputq);
- __skb_dequeue(arrvq);
+ /* Decrement the skb's refcnt */
+ kfree_skb(__skb_dequeue(arrvq));
}
spin_unlock_bh(&inputq->lock);
__skb_queue_purge(&tmpq);
return -EISCONN;
if (tsk->published)
return -EOPNOTSUPP;
- if (atype == TIPC_SERVICE_ADDR) {
- tsk->conn_type = ua->sa.type;
- tsk->conn_instance = ua->sa.instance;
- }
+ if (atype == TIPC_SERVICE_ADDR)
+ tsk->conn_addrtype = atype;
msg_set_syn(hdr, 1);
}
static int tipc_sk_anc_data_recv(struct msghdr *m, struct sk_buff *skb,
struct tipc_sock *tsk)
{
- struct tipc_msg *msg;
- u32 anc_data[3];
- u32 err;
- u32 dest_type;
- int has_name;
- int res;
+ struct tipc_msg *hdr;
+ u32 data[3] = {0,};
+ bool has_addr;
+ int dlen, rc;
if (likely(m->msg_controllen == 0))
return 0;
- msg = buf_msg(skb);
- /* Optionally capture errored message object(s) */
- err = msg ? msg_errcode(msg) : 0;
- if (unlikely(err)) {
- anc_data[0] = err;
- anc_data[1] = msg_data_sz(msg);
- res = put_cmsg(m, SOL_TIPC, TIPC_ERRINFO, 8, anc_data);
- if (res)
- return res;
- if (anc_data[1]) {
- if (skb_linearize(skb))
- return -ENOMEM;
- msg = buf_msg(skb);
- res = put_cmsg(m, SOL_TIPC, TIPC_RETDATA, anc_data[1],
- msg_data(msg));
- if (res)
- return res;
- }
+ hdr = buf_msg(skb);
+ dlen = msg_data_sz(hdr);
+
+ /* Capture errored message object, if any */
+ if (msg_errcode(hdr)) {
+ if (skb_linearize(skb))
+ return -ENOMEM;
+ hdr = buf_msg(skb);
+ data[0] = msg_errcode(hdr);
+ data[1] = dlen;
+ rc = put_cmsg(m, SOL_TIPC, TIPC_ERRINFO, 8, data);
+ if (rc || !dlen)
+ return rc;
+ rc = put_cmsg(m, SOL_TIPC, TIPC_RETDATA, dlen, msg_data(hdr));
+ if (rc)
+ return rc;
}
- /* Optionally capture message destination object */
- dest_type = msg ? msg_type(msg) : TIPC_DIRECT_MSG;
- switch (dest_type) {
+ /* Capture TIPC_SERVICE_ADDR/RANGE destination address, if any */
+ switch (msg_type(hdr)) {
case TIPC_NAMED_MSG:
- has_name = 1;
- anc_data[0] = msg_nametype(msg);
- anc_data[1] = msg_namelower(msg);
- anc_data[2] = msg_namelower(msg);
+ has_addr = true;
+ data[0] = msg_nametype(hdr);
+ data[1] = msg_namelower(hdr);
+ data[2] = data[1];
break;
case TIPC_MCAST_MSG:
- has_name = 1;
- anc_data[0] = msg_nametype(msg);
- anc_data[1] = msg_namelower(msg);
- anc_data[2] = msg_nameupper(msg);
+ has_addr = true;
+ data[0] = msg_nametype(hdr);
+ data[1] = msg_namelower(hdr);
+ data[2] = msg_nameupper(hdr);
break;
case TIPC_CONN_MSG:
- has_name = (tsk->conn_type != 0);
- anc_data[0] = tsk->conn_type;
- anc_data[1] = tsk->conn_instance;
- anc_data[2] = tsk->conn_instance;
+ has_addr = !!tsk->conn_addrtype;
+ data[0] = msg_nametype(&tsk->phdr);
+ data[1] = msg_nameinst(&tsk->phdr);
+ data[2] = data[1];
break;
default:
- has_name = 0;
+ has_addr = false;
}
- if (has_name) {
- res = put_cmsg(m, SOL_TIPC, TIPC_DESTNAME, 12, anc_data);
- if (res)
- return res;
- }
-
- return 0;
+ if (!has_addr)
+ return 0;
+ return put_cmsg(m, SOL_TIPC, TIPC_DESTNAME, 12, data);
}
static struct sk_buff *tipc_sk_build_ack(struct tipc_sock *tsk)
tsk_set_importance(new_sk, msg_importance(msg));
if (msg_named(msg)) {
- new_tsock->conn_type = msg_nametype(msg);
- new_tsock->conn_instance = msg_nameinst(msg);
+ new_tsock->conn_addrtype = TIPC_SERVICE_ADDR;
+ msg_set_nametype(&new_tsock->phdr, msg_nametype(msg));
+ msg_set_nameinst(&new_tsock->phdr, msg_nameinst(msg));
}
/*
/* Caller should hold socket lock for the passed tipc socket. */
static int __tipc_nl_add_sk_con(struct sk_buff *skb, struct tipc_sock *tsk)
{
- u32 peer_node;
- u32 peer_port;
+ u32 peer_node, peer_port;
+ u32 conn_type, conn_instance;
struct nlattr *nest;
peer_node = tsk_peer_node(tsk);
peer_port = tsk_peer_port(tsk);
-
+ conn_type = msg_nametype(&tsk->phdr);
+ conn_instance = msg_nameinst(&tsk->phdr);
nest = nla_nest_start_noflag(skb, TIPC_NLA_SOCK_CON);
if (!nest)
return -EMSGSIZE;
if (nla_put_u32(skb, TIPC_NLA_CON_SOCK, peer_port))
goto msg_full;
- if (tsk->conn_type != 0) {
+ if (tsk->conn_addrtype != 0) {
if (nla_put_flag(skb, TIPC_NLA_CON_FLAG))
goto msg_full;
- if (nla_put_u32(skb, TIPC_NLA_CON_TYPE, tsk->conn_type))
+ if (nla_put_u32(skb, TIPC_NLA_CON_TYPE, conn_type))
goto msg_full;
- if (nla_put_u32(skb, TIPC_NLA_CON_INST, tsk->conn_instance))
+ if (nla_put_u32(skb, TIPC_NLA_CON_INST, conn_instance))
goto msg_full;
}
nla_nest_end(skb, nest);
}
if (!tipc_sk_type_connectionless(sk)) {
- type = tsk->conn_type;
- lower = tsk->conn_instance;
- upper = tsk->conn_instance;
+ type = msg_nametype(&tsk->phdr);
+ lower = msg_nameinst(&tsk->phdr);
+ upper = lower;
}
if ((_type && _type != type) || (_lower && _lower != lower) ||
{
int i = 0;
size_t sz = (dqueues) ? SK_LMAX : SK_LMIN;
+ u32 conn_type, conn_instance;
struct tipc_sock *tsk;
struct publication *p;
bool tsk_connected;
if (tsk_connected) {
i += scnprintf(buf + i, sz - i, " %x", tsk_peer_node(tsk));
i += scnprintf(buf + i, sz - i, " %u", tsk_peer_port(tsk));
- i += scnprintf(buf + i, sz - i, " %u", tsk->conn_type);
- i += scnprintf(buf + i, sz - i, " %u", tsk->conn_instance);
+ conn_type = msg_nametype(&tsk->phdr);
+ conn_instance = msg_nameinst(&tsk->phdr);
+ i += scnprintf(buf + i, sz - i, " %u", conn_type);
+ i += scnprintf(buf + i, sz - i, " %u", conn_instance);
}
i += scnprintf(buf + i, sz - i, " | %u", tsk->published);
if (tsk->published) {
/**
* tipc_sub_check_overlap - test for subscription overlap with the given values
* @subscribed: the service range subscribed for
- * @found: the service range we are checning for match
+ * @found: the service range we are checking for match
*
* Returns true if there is overlap, otherwise false.
*/
kfree_rcu(rcast, rcu);
}
+ atomic_dec(&tipc_net(sock_net(ub->ubsock->sk))->wq_count);
dst_cache_destroy(&ub->rcast.dst_cache);
udp_tunnel_sock_release(ub->ubsock);
synchronize_net();
RCU_INIT_POINTER(ub->bearer, NULL);
/* sock_release need to be done outside of rtnl lock */
+ atomic_inc(&tipc_net(sock_net(ub->ubsock->sk))->wq_count);
INIT_WORK(&ub->work, cleanup_bearer);
schedule_work(&ub->work);
}
static DECLARE_WORK(tls_device_gc_work, tls_device_gc_task);
static LIST_HEAD(tls_device_gc_list);
static LIST_HEAD(tls_device_list);
+static LIST_HEAD(tls_device_down_list);
static DEFINE_SPINLOCK(tls_device_lock);
static void tls_device_free_ctx(struct tls_context *ctx)
int i;
for (i = 0; i < record->num_frags; i++)
- __skb_frag_unref(&record->frags[i]);
+ __skb_frag_unref(&record->frags[i], false);
kfree(record);
}
struct tls_offload_context_rx *rx_ctx = tls_offload_ctx_rx(tls_ctx);
struct net_device *netdev;
- if (WARN_ON(test_and_set_bit(TLS_RX_SYNC_RUNNING, &tls_ctx->flags)))
- return;
-
trace_tls_device_rx_resync_send(sk, seq, rcd_sn, rx_ctx->resync_type);
+ rcu_read_lock();
netdev = READ_ONCE(tls_ctx->netdev);
if (netdev)
netdev->tlsdev_ops->tls_dev_resync(netdev, sk, seq, rcd_sn,
TLS_OFFLOAD_CTX_DIR_RX);
- clear_bit_unlock(TLS_RX_SYNC_RUNNING, &tls_ctx->flags);
+ rcu_read_unlock();
TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXDEVICERESYNC);
}
if (tls_ctx->rx_conf != TLS_HW)
return;
+ if (unlikely(test_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags)))
+ return;
prot = &tls_ctx->prot_info;
rx_ctx = tls_offload_ctx_rx(tls_ctx);
ctx->sw.decrypted |= is_decrypted;
+ if (unlikely(test_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags))) {
+ if (likely(is_encrypted || is_decrypted))
+ return 0;
+
+ /* After tls_device_down disables the offload, the next SKB will
+ * likely have initial fragments decrypted, and final ones not
+ * decrypted. We need to reencrypt that single SKB.
+ */
+ return tls_device_reencrypt(sk, skb);
+ }
+
/* Return immediately if the record is either entirely plaintext or
* entirely ciphertext. Otherwise handle reencrypt partially decrypted
* record.
spin_unlock_irqrestore(&tls_device_lock, flags);
list_for_each_entry_safe(ctx, tmp, &list, list) {
+ /* Stop offloaded TX and switch to the fallback.
+ * tls_is_sk_tx_device_offloaded will return false.
+ */
+ WRITE_ONCE(ctx->sk->sk_validate_xmit_skb, tls_validate_xmit_skb_sw);
+
+ /* Stop the RX and TX resync.
+ * tls_dev_resync must not be called after tls_dev_del.
+ */
+ WRITE_ONCE(ctx->netdev, NULL);
+
+ /* Start skipping the RX resync logic completely. */
+ set_bit(TLS_RX_DEV_DEGRADED, &ctx->flags);
+
+ /* Sync with inflight packets. After this point:
+ * TX: no non-encrypted packets will be passed to the driver.
+ * RX: resync requests from the driver will be ignored.
+ */
+ synchronize_net();
+
+ /* Release the offload context on the driver side. */
if (ctx->tx_conf == TLS_HW)
netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
TLS_OFFLOAD_CTX_DIR_TX);
!test_bit(TLS_RX_DEV_CLOSED, &ctx->flags))
netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
TLS_OFFLOAD_CTX_DIR_RX);
- WRITE_ONCE(ctx->netdev, NULL);
- smp_mb__before_atomic(); /* pairs with test_and_set_bit() */
- while (test_bit(TLS_RX_SYNC_RUNNING, &ctx->flags))
- usleep_range(10, 200);
+
dev_put(netdev);
- list_del_init(&ctx->list);
- if (refcount_dec_and_test(&ctx->refcount))
- tls_device_free_ctx(ctx);
+ /* Move the context to a separate list for two reasons:
+ * 1. When the context is deallocated, list_del is called.
+ * 2. It's no longer an offloaded context, so we don't want to
+ * run offload-specific code on this context.
+ */
+ spin_lock_irqsave(&tls_device_lock, flags);
+ list_move_tail(&ctx->list, &tls_device_down_list);
+ spin_unlock_irqrestore(&tls_device_lock, flags);
+
+ /* Device contexts for RX and TX will be freed in on sk_destruct
+ * by tls_device_free_ctx. rx_conf and tx_conf stay in TLS_HW.
+ */
}
up_write(&device_offload_lock);
}
EXPORT_SYMBOL_GPL(tls_validate_xmit_skb);
+struct sk_buff *tls_validate_xmit_skb_sw(struct sock *sk,
+ struct net_device *dev,
+ struct sk_buff *skb)
+{
+ return tls_sw_fallback(sk, skb);
+}
+
struct sk_buff *tls_encrypt_skb(struct sk_buff *skb)
{
return tls_sw_fallback(skb->sk, skb);
mutex_init(&ctx->tx_lock);
rcu_assign_pointer(icsk->icsk_ulp_data, ctx);
ctx->sk_proto = READ_ONCE(sk->sk_prot);
+ ctx->sk = sk;
return ctx;
}
#include <linux/sched/signal.h>
#include <linux/module.h>
+#include <linux/splice.h>
#include <crypto/aead.h>
#include <net/strparser.h>
}
static struct sk_buff *tls_wait_data(struct sock *sk, struct sk_psock *psock,
- int flags, long timeo, int *err)
+ bool nonblock, long timeo, int *err)
{
struct tls_context *tls_ctx = tls_get_ctx(sk);
struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
if (sock_flag(sk, SOCK_DONE))
return NULL;
- if ((flags & MSG_DONTWAIT) || !timeo) {
+ if (nonblock || !timeo) {
*err = -EAGAIN;
return NULL;
}
bool async_capable;
bool async = false;
- skb = tls_wait_data(sk, psock, flags, timeo, &err);
+ skb = tls_wait_data(sk, psock, flags & MSG_DONTWAIT, timeo, &err);
if (!skb) {
if (psock) {
int ret = sk_msg_recvmsg(sk, psock, msg, len,
lock_sock(sk);
- timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
+ timeo = sock_rcvtimeo(sk, flags & SPLICE_F_NONBLOCK);
- skb = tls_wait_data(sk, NULL, flags, timeo, &err);
+ skb = tls_wait_data(sk, NULL, flags & SPLICE_F_NONBLOCK, timeo, &err);
if (!skb)
goto splice_read_end;
sk_add_node(sk, list);
}
+static void __unix_set_addr(struct sock *sk, struct unix_address *addr,
+ unsigned hash)
+{
+ __unix_remove_socket(sk);
+ smp_store_release(&unix_sk(sk)->addr, addr);
+ __unix_insert_socket(&unix_socket_table[hash], sk);
+}
+
static inline void unix_remove_socket(struct sock *sk)
{
spin_lock(&unix_table_lock);
static struct sock *__unix_find_socket_byname(struct net *net,
struct sockaddr_un *sunname,
- int len, int type, unsigned int hash)
+ int len, unsigned int hash)
{
struct sock *s;
- sk_for_each(s, &unix_socket_table[hash ^ type]) {
+ sk_for_each(s, &unix_socket_table[hash]) {
struct unix_sock *u = unix_sk(s);
if (!net_eq(sock_net(s), net))
static inline struct sock *unix_find_socket_byname(struct net *net,
struct sockaddr_un *sunname,
- int len, int type,
- unsigned int hash)
+ int len, unsigned int hash)
{
struct sock *s;
spin_lock(&unix_table_lock);
- s = __unix_find_socket_byname(net, sunname, len, type, hash);
+ s = __unix_find_socket_byname(net, sunname, len, hash);
if (s)
sock_hold(s);
spin_unlock(&unix_table_lock);
u->path.mnt = NULL;
state = sk->sk_state;
sk->sk_state = TCP_CLOSE;
+
+ skpair = unix_peer(sk);
+ unix_peer(sk) = NULL;
+
unix_state_unlock(sk);
wake_up_interruptible_all(&u->peer_wait);
- skpair = unix_peer(sk);
-
if (skpair != NULL) {
if (sk->sk_type == SOCK_STREAM || sk->sk_type == SOCK_SEQPACKET) {
unix_state_lock(skpair);
unix_dgram_peer_wake_disconnect(sk, skpair);
sock_put(skpair); /* It may now die */
- unix_peer(sk) = NULL;
}
/* Try to flush out this socket. Throw out buffers at least */
retry:
addr->len = sprintf(addr->name->sun_path+1, "%05x", ordernum) + 1 + sizeof(short);
addr->hash = unix_hash_fold(csum_partial(addr->name, addr->len, 0));
+ addr->hash ^= sk->sk_type;
spin_lock(&unix_table_lock);
ordernum = (ordernum+1)&0xFFFFF;
- if (__unix_find_socket_byname(net, addr->name, addr->len, sock->type,
- addr->hash)) {
+ if (__unix_find_socket_byname(net, addr->name, addr->len, addr->hash)) {
spin_unlock(&unix_table_lock);
/*
* __unix_find_socket_byname() may take long time if many names
}
goto retry;
}
- addr->hash ^= sk->sk_type;
- __unix_remove_socket(sk);
- smp_store_release(&u->addr, addr);
- __unix_insert_socket(&unix_socket_table[addr->hash], sk);
+ __unix_set_addr(sk, addr, addr->hash);
spin_unlock(&unix_table_lock);
err = 0;
}
} else {
err = -ECONNREFUSED;
- u = unix_find_socket_byname(net, sunname, len, type, hash);
+ u = unix_find_socket_byname(net, sunname, len, type ^ hash);
if (u) {
struct dentry *dentry;
dentry = unix_sk(u)->path.dentry;
return NULL;
}
-static int unix_mknod(const char *sun_path, umode_t mode, struct path *res)
+static int unix_bind_bsd(struct sock *sk, struct unix_address *addr)
{
+ struct unix_sock *u = unix_sk(sk);
+ umode_t mode = S_IFSOCK |
+ (SOCK_INODE(sk->sk_socket)->i_mode & ~current_umask());
+ struct user_namespace *ns; // barf...
+ struct path parent;
struct dentry *dentry;
- struct path path;
- int err = 0;
+ unsigned int hash;
+ int err;
+
/*
* Get the parent directory, calculate the hash for last
* component.
*/
- dentry = kern_path_create(AT_FDCWD, sun_path, &path, 0);
- err = PTR_ERR(dentry);
+ dentry = kern_path_create(AT_FDCWD, addr->name->sun_path, &parent, 0);
if (IS_ERR(dentry))
- return err;
+ return PTR_ERR(dentry);
+ ns = mnt_user_ns(parent.mnt);
/*
* All right, let's create it.
*/
- err = security_path_mknod(&path, dentry, mode, 0);
- if (!err) {
- err = vfs_mknod(mnt_user_ns(path.mnt), d_inode(path.dentry),
- dentry, mode, 0);
- if (!err) {
- res->mnt = mntget(path.mnt);
- res->dentry = dget(dentry);
- }
- }
- done_path_create(&path, dentry);
+ err = security_path_mknod(&parent, dentry, mode, 0);
+ if (!err)
+ err = vfs_mknod(ns, d_inode(parent.dentry), dentry, mode, 0);
+ if (err)
+ goto out;
+ err = mutex_lock_interruptible(&u->bindlock);
+ if (err)
+ goto out_unlink;
+ if (u->addr)
+ goto out_unlock;
+
+ addr->hash = UNIX_HASH_SIZE;
+ hash = d_backing_inode(dentry)->i_ino & (UNIX_HASH_SIZE - 1);
+ spin_lock(&unix_table_lock);
+ u->path.mnt = mntget(parent.mnt);
+ u->path.dentry = dget(dentry);
+ __unix_set_addr(sk, addr, hash);
+ spin_unlock(&unix_table_lock);
+ mutex_unlock(&u->bindlock);
+ done_path_create(&parent, dentry);
+ return 0;
+
+out_unlock:
+ mutex_unlock(&u->bindlock);
+ err = -EINVAL;
+out_unlink:
+ /* failed after successful mknod? unlink what we'd created... */
+ vfs_unlink(ns, d_inode(parent.dentry), dentry, NULL);
+out:
+ done_path_create(&parent, dentry);
return err;
}
+static int unix_bind_abstract(struct sock *sk, struct unix_address *addr)
+{
+ struct unix_sock *u = unix_sk(sk);
+ int err;
+
+ err = mutex_lock_interruptible(&u->bindlock);
+ if (err)
+ return err;
+
+ if (u->addr) {
+ mutex_unlock(&u->bindlock);
+ return -EINVAL;
+ }
+
+ spin_lock(&unix_table_lock);
+ if (__unix_find_socket_byname(sock_net(sk), addr->name, addr->len,
+ addr->hash)) {
+ spin_unlock(&unix_table_lock);
+ mutex_unlock(&u->bindlock);
+ return -EADDRINUSE;
+ }
+ __unix_set_addr(sk, addr, addr->hash);
+ spin_unlock(&unix_table_lock);
+ mutex_unlock(&u->bindlock);
+ return 0;
+}
+
static int unix_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
{
struct sock *sk = sock->sk;
- struct net *net = sock_net(sk);
- struct unix_sock *u = unix_sk(sk);
struct sockaddr_un *sunaddr = (struct sockaddr_un *)uaddr;
char *sun_path = sunaddr->sun_path;
int err;
unsigned int hash;
struct unix_address *addr;
- struct hlist_head *list;
- struct path path = { };
- err = -EINVAL;
if (addr_len < offsetofend(struct sockaddr_un, sun_family) ||
sunaddr->sun_family != AF_UNIX)
- goto out;
+ return -EINVAL;
- if (addr_len == sizeof(short)) {
- err = unix_autobind(sock);
- goto out;
- }
+ if (addr_len == sizeof(short))
+ return unix_autobind(sock);
err = unix_mkname(sunaddr, addr_len, &hash);
if (err < 0)
- goto out;
+ return err;
addr_len = err;
-
- if (sun_path[0]) {
- umode_t mode = S_IFSOCK |
- (SOCK_INODE(sock)->i_mode & ~current_umask());
- err = unix_mknod(sun_path, mode, &path);
- if (err) {
- if (err == -EEXIST)
- err = -EADDRINUSE;
- goto out;
- }
- }
-
- err = mutex_lock_interruptible(&u->bindlock);
- if (err)
- goto out_put;
-
- err = -EINVAL;
- if (u->addr)
- goto out_up;
-
- err = -ENOMEM;
addr = kmalloc(sizeof(*addr)+addr_len, GFP_KERNEL);
if (!addr)
- goto out_up;
+ return -ENOMEM;
memcpy(addr->name, sunaddr, addr_len);
addr->len = addr_len;
addr->hash = hash ^ sk->sk_type;
refcount_set(&addr->refcnt, 1);
- if (sun_path[0]) {
- addr->hash = UNIX_HASH_SIZE;
- hash = d_backing_inode(path.dentry)->i_ino & (UNIX_HASH_SIZE - 1);
- spin_lock(&unix_table_lock);
- u->path = path;
- list = &unix_socket_table[hash];
- } else {
- spin_lock(&unix_table_lock);
- err = -EADDRINUSE;
- if (__unix_find_socket_byname(net, sunaddr, addr_len,
- sk->sk_type, hash)) {
- unix_release_addr(addr);
- goto out_unlock;
- }
-
- list = &unix_socket_table[addr->hash];
- }
-
- err = 0;
- __unix_remove_socket(sk);
- smp_store_release(&u->addr, addr);
- __unix_insert_socket(list, sk);
-
-out_unlock:
- spin_unlock(&unix_table_lock);
-out_up:
- mutex_unlock(&u->bindlock);
-out_put:
+ if (sun_path[0])
+ err = unix_bind_bsd(sk, addr);
+ else
+ err = unix_bind_abstract(sk, addr);
if (err)
- path_put(&path);
-out:
- return err;
+ unix_release_addr(addr);
+ return err == -EEXIST ? -EADDRINUSE : err;
}
static void unix_state_double_lock(struct sock *sk1, struct sock *sk2)
unix_state_unlock(sk);
- /* take ten and and send info to listening sock */
+ /* take ten and send info to listening sock */
spin_lock(&other->sk_receive_queue.lock);
__skb_queue_tail(&other->sk_receive_queue, skb);
spin_unlock(&other->sk_receive_queue.lock);
/* Assign a transport to a socket and call the .init transport callback.
*
- * Note: for stream socket this must be called when vsk->remote_addr is set
- * (e.g. during the connect() or when a connection request on a listener
+ * Note: for connection oriented socket this must be called when vsk->remote_addr
+ * is set (e.g. during the connect() or when a connection request on a listener
* socket is received).
* The vsk->remote_addr is used to decide which transport to use:
* - remote CID == VMADDR_CID_LOCAL or g2h->local_cid or VMADDR_CID_HOST if
new_transport = transport_dgram;
break;
case SOCK_STREAM:
+ case SOCK_SEQPACKET:
if (vsock_use_local_transport(remote_cid))
new_transport = transport_local;
else if (remote_cid <= VMADDR_CID_HOST || !transport_h2g ||
return 0;
/* transport->release() must be called with sock lock acquired.
- * This path can only be taken during vsock_stream_connect(),
- * where we have already held the sock lock.
- * In the other cases, this function is called on a new socket
- * which is not assigned to any transport.
+ * This path can only be taken during vsock_connect(), where we
+ * have already held the sock lock. In the other cases, this
+ * function is called on a new socket which is not assigned to
+ * any transport.
*/
vsk->transport->release(vsk);
vsock_deassign_transport(vsk);
if (!new_transport || !try_module_get(new_transport->module))
return -ENODEV;
+ if (sk->sk_type == SOCK_SEQPACKET) {
+ if (!new_transport->seqpacket_allow ||
+ !new_transport->seqpacket_allow(remote_cid)) {
+ module_put(new_transport->module);
+ return -ESOCKTNOSUPPORT;
+ }
+ }
+
ret = new_transport->init(vsk, psk);
if (ret) {
module_put(new_transport->module);
/**** SOCKET OPERATIONS ****/
-static int __vsock_bind_stream(struct vsock_sock *vsk,
- struct sockaddr_vm *addr)
+static int __vsock_bind_connectible(struct vsock_sock *vsk,
+ struct sockaddr_vm *addr)
{
static u32 port;
struct sockaddr_vm new_addr;
vsock_addr_init(&vsk->local_addr, new_addr.svm_cid, new_addr.svm_port);
- /* Remove stream sockets from the unbound list and add them to the hash
- * table for easy lookup by its address. The unbound list is simply an
- * extra entry at the end of the hash table, a trick used by AF_UNIX.
+ /* Remove connection oriented sockets from the unbound list and add them
+ * to the hash table for easy lookup by its address. The unbound list
+ * is simply an extra entry at the end of the hash table, a trick used
+ * by AF_UNIX.
*/
__vsock_remove_bound(vsk);
__vsock_insert_bound(vsock_bound_sockets(&vsk->local_addr), vsk);
switch (sk->sk_socket->type) {
case SOCK_STREAM:
+ case SOCK_SEQPACKET:
spin_lock_bh(&vsock_table_lock);
- retval = __vsock_bind_stream(vsk, addr);
+ retval = __vsock_bind_connectible(vsk, addr);
spin_unlock_bh(&vsock_table_lock);
break;
return sk;
}
+static bool sock_type_connectible(u16 type)
+{
+ return (type == SOCK_STREAM) || (type == SOCK_SEQPACKET);
+}
+
static void __vsock_release(struct sock *sk, int level)
{
if (sk) {
if (vsk->transport)
vsk->transport->release(vsk);
- else if (sk->sk_type == SOCK_STREAM)
+ else if (sock_type_connectible(sk->sk_type))
vsock_remove_sock(vsk);
sock_orphan(sk);
}
EXPORT_SYMBOL_GPL(vsock_stream_has_data);
+static s64 vsock_connectible_has_data(struct vsock_sock *vsk)
+{
+ struct sock *sk = sk_vsock(vsk);
+
+ if (sk->sk_type == SOCK_SEQPACKET)
+ return vsk->transport->seqpacket_has_data(vsk);
+ else
+ return vsock_stream_has_data(vsk);
+}
+
s64 vsock_stream_has_space(struct vsock_sock *vsk)
{
return vsk->transport->stream_has_space(vsk);
if ((mode & ~SHUTDOWN_MASK) || !mode)
return -EINVAL;
- /* If this is a STREAM socket and it is not connected then bail out
- * immediately. If it is a DGRAM socket then we must first kick the
- * socket so that it wakes up from any sleeping calls, for example
- * recv(), and then afterwards return the error.
+ /* If this is a connection oriented socket and it is not connected then
+ * bail out immediately. If it is a DGRAM socket then we must first
+ * kick the socket so that it wakes up from any sleeping calls, for
+ * example recv(), and then afterwards return the error.
*/
sk = sock->sk;
lock_sock(sk);
if (sock->state == SS_UNCONNECTED) {
err = -ENOTCONN;
- if (sk->sk_type == SOCK_STREAM)
+ if (sock_type_connectible(sk->sk_type))
goto out;
} else {
sock->state = SS_DISCONNECTING;
sk->sk_shutdown |= mode;
sk->sk_state_change(sk);
- if (sk->sk_type == SOCK_STREAM) {
+ if (sock_type_connectible(sk->sk_type)) {
sock_reset_flag(sk, SOCK_DONE);
vsock_send_shutdown(sk, mode);
}
if (!(sk->sk_shutdown & SEND_SHUTDOWN))
mask |= EPOLLOUT | EPOLLWRNORM | EPOLLWRBAND;
- } else if (sock->type == SOCK_STREAM) {
+ } else if (sock_type_connectible(sk->sk_type)) {
const struct vsock_transport *transport;
lock_sock(sk);
sock_put(sk);
}
-static int vsock_stream_connect(struct socket *sock, struct sockaddr *addr,
- int addr_len, int flags)
+static int vsock_connect(struct socket *sock, struct sockaddr *addr,
+ int addr_len, int flags)
{
int err;
struct sock *sk;
lock_sock(listener);
- if (sock->type != SOCK_STREAM) {
+ if (!sock_type_connectible(sock->type)) {
err = -EOPNOTSUPP;
goto out;
}
lock_sock(sk);
- if (sock->type != SOCK_STREAM) {
+ if (!sock_type_connectible(sk->sk_type)) {
err = -EOPNOTSUPP;
goto out;
}
vsk->buffer_size = val;
}
-static int vsock_stream_setsockopt(struct socket *sock,
- int level,
- int optname,
- sockptr_t optval,
- unsigned int optlen)
+static int vsock_connectible_setsockopt(struct socket *sock,
+ int level,
+ int optname,
+ sockptr_t optval,
+ unsigned int optlen)
{
int err;
struct sock *sk;
return err;
}
-static int vsock_stream_getsockopt(struct socket *sock,
- int level, int optname,
- char __user *optval,
- int __user *optlen)
+static int vsock_connectible_getsockopt(struct socket *sock,
+ int level, int optname,
+ char __user *optval,
+ int __user *optlen)
{
int err;
int len;
return 0;
}
-static int vsock_stream_sendmsg(struct socket *sock, struct msghdr *msg,
- size_t len)
+static int vsock_connectible_sendmsg(struct socket *sock, struct msghdr *msg,
+ size_t len)
{
struct sock *sk;
struct vsock_sock *vsk;
transport = vsk->transport;
- /* Callers should not provide a destination with stream sockets. */
+ /* Callers should not provide a destination with connection oriented
+ * sockets.
+ */
if (msg->msg_namelen) {
err = sk->sk_state == TCP_ESTABLISHED ? -EISCONN : -EOPNOTSUPP;
goto out;
* responsibility to check how many bytes we were able to send.
*/
- written = transport->stream_enqueue(
- vsk, msg,
- len - total_written);
+ if (sk->sk_type == SOCK_SEQPACKET) {
+ written = transport->seqpacket_enqueue(vsk,
+ msg, len - total_written);
+ } else {
+ written = transport->stream_enqueue(vsk,
+ msg, len - total_written);
+ }
if (written < 0) {
err = -ENOMEM;
goto out_err;
}
out_err:
- if (total_written > 0)
- err = total_written;
+ if (total_written > 0) {
+ /* Return number of written bytes only if:
+ * 1) SOCK_STREAM socket.
+ * 2) SOCK_SEQPACKET socket when whole buffer is sent.
+ */
+ if (sk->sk_type == SOCK_STREAM || total_written == len)
+ err = total_written;
+ }
out:
release_sock(sk);
return err;
}
-
-static int
-vsock_stream_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
- int flags)
+static int vsock_connectible_wait_data(struct sock *sk,
+ struct wait_queue_entry *wait,
+ long timeout,
+ struct vsock_transport_recv_notify_data *recv_data,
+ size_t target)
{
- struct sock *sk;
- struct vsock_sock *vsk;
const struct vsock_transport *transport;
+ struct vsock_sock *vsk;
+ s64 data;
int err;
- size_t target;
- ssize_t copied;
- long timeout;
- struct vsock_transport_recv_notify_data recv_data;
-
- DEFINE_WAIT(wait);
- sk = sock->sk;
vsk = vsock_sk(sk);
err = 0;
+ transport = vsk->transport;
- lock_sock(sk);
+ while ((data = vsock_connectible_has_data(vsk)) == 0) {
+ prepare_to_wait(sk_sleep(sk), wait, TASK_INTERRUPTIBLE);
- transport = vsk->transport;
+ if (sk->sk_err != 0 ||
+ (sk->sk_shutdown & RCV_SHUTDOWN) ||
+ (vsk->peer_shutdown & SEND_SHUTDOWN)) {
+ break;
+ }
- if (!transport || sk->sk_state != TCP_ESTABLISHED) {
- /* Recvmsg is supposed to return 0 if a peer performs an
- * orderly shutdown. Differentiate between that case and when a
- * peer has not connected or a local shutdown occurred with the
- * SOCK_DONE flag.
- */
- if (sock_flag(sk, SOCK_DONE))
- err = 0;
- else
- err = -ENOTCONN;
+ /* Don't wait for non-blocking sockets. */
+ if (timeout == 0) {
+ err = -EAGAIN;
+ break;
+ }
- goto out;
- }
+ if (recv_data) {
+ err = transport->notify_recv_pre_block(vsk, target, recv_data);
+ if (err < 0)
+ break;
+ }
- if (flags & MSG_OOB) {
- err = -EOPNOTSUPP;
- goto out;
- }
+ release_sock(sk);
+ timeout = schedule_timeout(timeout);
+ lock_sock(sk);
- /* We don't check peer_shutdown flag here since peer may actually shut
- * down, but there can be data in the queue that a local socket can
- * receive.
- */
- if (sk->sk_shutdown & RCV_SHUTDOWN) {
- err = 0;
- goto out;
+ if (signal_pending(current)) {
+ err = sock_intr_errno(timeout);
+ break;
+ } else if (timeout == 0) {
+ err = -EAGAIN;
+ break;
+ }
}
- /* It is valid on Linux to pass in a zero-length receive buffer. This
- * is not an error. We may as well bail out now.
+ finish_wait(sk_sleep(sk), wait);
+
+ if (err)
+ return err;
+
+ /* Internal transport error when checking for available
+ * data. XXX This should be changed to a connection
+ * reset in a later change.
*/
- if (!len) {
- err = 0;
- goto out;
- }
+ if (data < 0)
+ return -ENOMEM;
+
+ return data;
+}
+
+static int __vsock_stream_recvmsg(struct sock *sk, struct msghdr *msg,
+ size_t len, int flags)
+{
+ struct vsock_transport_recv_notify_data recv_data;
+ const struct vsock_transport *transport;
+ struct vsock_sock *vsk;
+ ssize_t copied;
+ size_t target;
+ long timeout;
+ int err;
+
+ DEFINE_WAIT(wait);
+
+ vsk = vsock_sk(sk);
+ transport = vsk->transport;
/* We must not copy less than target bytes into the user's buffer
* before returning successfully, so we wait for the consume queue to
while (1) {
- s64 ready;
+ ssize_t read;
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
- ready = vsock_stream_has_data(vsk);
+ err = vsock_connectible_wait_data(sk, &wait, timeout,
+ &recv_data, target);
+ if (err <= 0)
+ break;
- if (ready == 0) {
- if (sk->sk_err != 0 ||
- (sk->sk_shutdown & RCV_SHUTDOWN) ||
- (vsk->peer_shutdown & SEND_SHUTDOWN)) {
- finish_wait(sk_sleep(sk), &wait);
- break;
- }
- /* Don't wait for non-blocking sockets. */
- if (timeout == 0) {
- err = -EAGAIN;
- finish_wait(sk_sleep(sk), &wait);
- break;
- }
+ err = transport->notify_recv_pre_dequeue(vsk, target,
+ &recv_data);
+ if (err < 0)
+ break;
- err = transport->notify_recv_pre_block(
- vsk, target, &recv_data);
- if (err < 0) {
- finish_wait(sk_sleep(sk), &wait);
- break;
- }
- release_sock(sk);
- timeout = schedule_timeout(timeout);
- lock_sock(sk);
+ read = transport->stream_dequeue(vsk, msg, len - copied, flags);
+ if (read < 0) {
+ err = -ENOMEM;
+ break;
+ }
- if (signal_pending(current)) {
- err = sock_intr_errno(timeout);
- finish_wait(sk_sleep(sk), &wait);
- break;
- } else if (timeout == 0) {
- err = -EAGAIN;
- finish_wait(sk_sleep(sk), &wait);
- break;
- }
- } else {
- ssize_t read;
+ copied += read;
- finish_wait(sk_sleep(sk), &wait);
+ err = transport->notify_recv_post_dequeue(vsk, target, read,
+ !(flags & MSG_PEEK), &recv_data);
+ if (err < 0)
+ goto out;
- if (ready < 0) {
- /* Invalid queue pair content. XXX This should
- * be changed to a connection reset in a later
- * change.
- */
+ if (read >= target || flags & MSG_PEEK)
+ break;
- err = -ENOMEM;
- goto out;
- }
+ target -= read;
+ }
- err = transport->notify_recv_pre_dequeue(
- vsk, target, &recv_data);
- if (err < 0)
- break;
+ if (sk->sk_err)
+ err = -sk->sk_err;
+ else if (sk->sk_shutdown & RCV_SHUTDOWN)
+ err = 0;
- read = transport->stream_dequeue(
- vsk, msg,
- len - copied, flags);
- if (read < 0) {
- err = -ENOMEM;
- break;
- }
+ if (copied > 0)
+ err = copied;
+
+out:
+ return err;
+}
- copied += read;
+static int __vsock_seqpacket_recvmsg(struct sock *sk, struct msghdr *msg,
+ size_t len, int flags)
+{
+ const struct vsock_transport *transport;
+ struct vsock_sock *vsk;
+ ssize_t record_len;
+ long timeout;
+ int err = 0;
+ DEFINE_WAIT(wait);
- err = transport->notify_recv_post_dequeue(
- vsk, target, read,
- !(flags & MSG_PEEK), &recv_data);
- if (err < 0)
- goto out;
+ vsk = vsock_sk(sk);
+ transport = vsk->transport;
- if (read >= target || flags & MSG_PEEK)
- break;
+ timeout = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
- target -= read;
- }
+ err = vsock_connectible_wait_data(sk, &wait, timeout, NULL, 0);
+ if (err <= 0)
+ goto out;
+
+ record_len = transport->seqpacket_dequeue(vsk, msg, flags);
+
+ if (record_len < 0) {
+ err = -ENOMEM;
+ goto out;
}
- if (sk->sk_err)
+ if (sk->sk_err) {
err = -sk->sk_err;
- else if (sk->sk_shutdown & RCV_SHUTDOWN)
+ } else if (sk->sk_shutdown & RCV_SHUTDOWN) {
err = 0;
+ } else {
+ /* User sets MSG_TRUNC, so return real length of
+ * packet.
+ */
+ if (flags & MSG_TRUNC)
+ err = record_len;
+ else
+ err = len - msg_data_left(msg);
- if (copied > 0)
- err = copied;
+ /* Always set MSG_TRUNC if real length of packet is
+ * bigger than user's buffer.
+ */
+ if (record_len > len)
+ msg->msg_flags |= MSG_TRUNC;
+ }
+
+out:
+ return err;
+}
+
+static int
+vsock_connectible_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
+ int flags)
+{
+ struct sock *sk;
+ struct vsock_sock *vsk;
+ const struct vsock_transport *transport;
+ int err;
+
+ DEFINE_WAIT(wait);
+
+ sk = sock->sk;
+ vsk = vsock_sk(sk);
+ err = 0;
+
+ lock_sock(sk);
+
+ transport = vsk->transport;
+
+ if (!transport || sk->sk_state != TCP_ESTABLISHED) {
+ /* Recvmsg is supposed to return 0 if a peer performs an
+ * orderly shutdown. Differentiate between that case and when a
+ * peer has not connected or a local shutdown occurred with the
+ * SOCK_DONE flag.
+ */
+ if (sock_flag(sk, SOCK_DONE))
+ err = 0;
+ else
+ err = -ENOTCONN;
+
+ goto out;
+ }
+
+ if (flags & MSG_OOB) {
+ err = -EOPNOTSUPP;
+ goto out;
+ }
+
+ /* We don't check peer_shutdown flag here since peer may actually shut
+ * down, but there can be data in the queue that a local socket can
+ * receive.
+ */
+ if (sk->sk_shutdown & RCV_SHUTDOWN) {
+ err = 0;
+ goto out;
+ }
+
+ /* It is valid on Linux to pass in a zero-length receive buffer. This
+ * is not an error. We may as well bail out now.
+ */
+ if (!len) {
+ err = 0;
+ goto out;
+ }
+
+ if (sk->sk_type == SOCK_STREAM)
+ err = __vsock_stream_recvmsg(sk, msg, len, flags);
+ else
+ err = __vsock_seqpacket_recvmsg(sk, msg, len, flags);
out:
release_sock(sk);
.owner = THIS_MODULE,
.release = vsock_release,
.bind = vsock_bind,
- .connect = vsock_stream_connect,
+ .connect = vsock_connect,
.socketpair = sock_no_socketpair,
.accept = vsock_accept,
.getname = vsock_getname,
.ioctl = sock_no_ioctl,
.listen = vsock_listen,
.shutdown = vsock_shutdown,
- .setsockopt = vsock_stream_setsockopt,
- .getsockopt = vsock_stream_getsockopt,
- .sendmsg = vsock_stream_sendmsg,
- .recvmsg = vsock_stream_recvmsg,
+ .setsockopt = vsock_connectible_setsockopt,
+ .getsockopt = vsock_connectible_getsockopt,
+ .sendmsg = vsock_connectible_sendmsg,
+ .recvmsg = vsock_connectible_recvmsg,
+ .mmap = sock_no_mmap,
+ .sendpage = sock_no_sendpage,
+};
+
+static const struct proto_ops vsock_seqpacket_ops = {
+ .family = PF_VSOCK,
+ .owner = THIS_MODULE,
+ .release = vsock_release,
+ .bind = vsock_bind,
+ .connect = vsock_connect,
+ .socketpair = sock_no_socketpair,
+ .accept = vsock_accept,
+ .getname = vsock_getname,
+ .poll = vsock_poll,
+ .ioctl = sock_no_ioctl,
+ .listen = vsock_listen,
+ .shutdown = vsock_shutdown,
+ .setsockopt = vsock_connectible_setsockopt,
+ .getsockopt = vsock_connectible_getsockopt,
+ .sendmsg = vsock_connectible_sendmsg,
+ .recvmsg = vsock_connectible_recvmsg,
.mmap = sock_no_mmap,
.sendpage = sock_no_sendpage,
};
case SOCK_STREAM:
sock->ops = &vsock_stream_ops;
break;
+ case SOCK_SEQPACKET:
+ sock->ops = &vsock_seqpacket_ops;
+ break;
default:
return -ESOCKTNOSUPPORT;
}
struct virtio_vsock_event event_list[8];
u32 guest_cid;
+ bool seqpacket_allow;
};
static u32 virtio_transport_get_local_cid(void)
queue_work(virtio_vsock_workqueue, &vsock->rx_work);
}
+static bool virtio_transport_seqpacket_allow(u32 remote_cid);
+
static struct virtio_transport virtio_transport = {
.transport = {
.module = THIS_MODULE,
.stream_is_active = virtio_transport_stream_is_active,
.stream_allow = virtio_transport_stream_allow,
+ .seqpacket_dequeue = virtio_transport_seqpacket_dequeue,
+ .seqpacket_enqueue = virtio_transport_seqpacket_enqueue,
+ .seqpacket_allow = virtio_transport_seqpacket_allow,
+ .seqpacket_has_data = virtio_transport_seqpacket_has_data,
+
.notify_poll_in = virtio_transport_notify_poll_in,
.notify_poll_out = virtio_transport_notify_poll_out,
.notify_recv_init = virtio_transport_notify_recv_init,
.send_pkt = virtio_transport_send_pkt,
};
+static bool virtio_transport_seqpacket_allow(u32 remote_cid)
+{
+ struct virtio_vsock *vsock;
+ bool seqpacket_allow;
+
+ seqpacket_allow = false;
+ rcu_read_lock();
+ vsock = rcu_dereference(the_virtio_vsock);
+ if (vsock)
+ seqpacket_allow = vsock->seqpacket_allow;
+ rcu_read_unlock();
+
+ return seqpacket_allow;
+}
+
static void virtio_transport_rx_work(struct work_struct *work)
{
struct virtio_vsock *vsock =
vsock->event_run = true;
mutex_unlock(&vsock->event_lock);
+ if (virtio_has_feature(vdev, VIRTIO_VSOCK_F_SEQPACKET))
+ vsock->seqpacket_allow = true;
+
vdev->priv = vsock;
rcu_assign_pointer(the_virtio_vsock, vsock);
mutex_unlock(&the_virtio_vsock_mutex);
+
return 0;
out:
};
static unsigned int features[] = {
+ VIRTIO_VSOCK_F_SEQPACKET
};
static struct virtio_driver virtio_vsock_driver = {
err = memcpy_from_msg(pkt->buf, info->msg, len);
if (err)
goto out;
+
+ if (msg_data_left(info->msg) == 0 &&
+ info->type == VIRTIO_VSOCK_TYPE_SEQPACKET)
+ pkt->hdr.flags |= cpu_to_le32(VIRTIO_VSOCK_SEQ_EOR);
}
trace_virtio_transport_alloc_pkt(src_cid, src_port,
}
EXPORT_SYMBOL_GPL(virtio_transport_deliver_tap_pkt);
+static u16 virtio_transport_get_type(struct sock *sk)
+{
+ if (sk->sk_type == SOCK_STREAM)
+ return VIRTIO_VSOCK_TYPE_STREAM;
+ else
+ return VIRTIO_VSOCK_TYPE_SEQPACKET;
+}
+
/* This function can only be used on connecting/connected sockets,
* since a socket assigned to a transport is required.
*
struct virtio_vsock_pkt *pkt;
u32 pkt_len = info->pkt_len;
+ info->type = virtio_transport_get_type(sk_vsock(vsk));
+
t_ops = virtio_transport_get_ops(vsk);
if (unlikely(!t_ops))
return -EFAULT;
}
EXPORT_SYMBOL_GPL(virtio_transport_put_credit);
-static int virtio_transport_send_credit_update(struct vsock_sock *vsk,
- int type,
- struct virtio_vsock_hdr *hdr)
+static int virtio_transport_send_credit_update(struct vsock_sock *vsk)
{
struct virtio_vsock_pkt_info info = {
.op = VIRTIO_VSOCK_OP_CREDIT_UPDATE,
- .type = type,
.vsk = vsk,
};
* messages, we set the limit to a high value. TODO: experiment
* with different values.
*/
- if (free_space < VIRTIO_VSOCK_MAX_PKT_BUF_SIZE) {
- virtio_transport_send_credit_update(vsk,
- VIRTIO_VSOCK_TYPE_STREAM,
- NULL);
- }
+ if (free_space < VIRTIO_VSOCK_MAX_PKT_BUF_SIZE)
+ virtio_transport_send_credit_update(vsk);
return total;
return err;
}
+static int virtio_transport_seqpacket_do_dequeue(struct vsock_sock *vsk,
+ struct msghdr *msg,
+ int flags)
+{
+ struct virtio_vsock_sock *vvs = vsk->trans;
+ struct virtio_vsock_pkt *pkt;
+ int dequeued_len = 0;
+ size_t user_buf_len = msg_data_left(msg);
+ bool msg_ready = false;
+
+ spin_lock_bh(&vvs->rx_lock);
+
+ if (vvs->msg_count == 0) {
+ spin_unlock_bh(&vvs->rx_lock);
+ return 0;
+ }
+
+ while (!msg_ready) {
+ pkt = list_first_entry(&vvs->rx_queue, struct virtio_vsock_pkt, list);
+
+ if (dequeued_len >= 0) {
+ size_t pkt_len;
+ size_t bytes_to_copy;
+
+ pkt_len = (size_t)le32_to_cpu(pkt->hdr.len);
+ bytes_to_copy = min(user_buf_len, pkt_len);
+
+ if (bytes_to_copy) {
+ int err;
+
+ /* sk_lock is held by caller so no one else can dequeue.
+ * Unlock rx_lock since memcpy_to_msg() may sleep.
+ */
+ spin_unlock_bh(&vvs->rx_lock);
+
+ err = memcpy_to_msg(msg, pkt->buf, bytes_to_copy);
+ if (err) {
+ /* Copy of message failed. Rest of
+ * fragments will be freed without copy.
+ */
+ dequeued_len = err;
+ } else {
+ user_buf_len -= bytes_to_copy;
+ }
+
+ spin_lock_bh(&vvs->rx_lock);
+ }
+
+ if (dequeued_len >= 0)
+ dequeued_len += pkt_len;
+ }
+
+ if (le32_to_cpu(pkt->hdr.flags) & VIRTIO_VSOCK_SEQ_EOR) {
+ msg_ready = true;
+ vvs->msg_count--;
+ }
+
+ virtio_transport_dec_rx_pkt(vvs, pkt);
+ list_del(&pkt->list);
+ virtio_transport_free_pkt(pkt);
+ }
+
+ spin_unlock_bh(&vvs->rx_lock);
+
+ virtio_transport_send_credit_update(vsk);
+
+ return dequeued_len;
+}
+
ssize_t
virtio_transport_stream_dequeue(struct vsock_sock *vsk,
struct msghdr *msg,
}
EXPORT_SYMBOL_GPL(virtio_transport_stream_dequeue);
+ssize_t
+virtio_transport_seqpacket_dequeue(struct vsock_sock *vsk,
+ struct msghdr *msg,
+ int flags)
+{
+ if (flags & MSG_PEEK)
+ return -EOPNOTSUPP;
+
+ return virtio_transport_seqpacket_do_dequeue(vsk, msg, flags);
+}
+EXPORT_SYMBOL_GPL(virtio_transport_seqpacket_dequeue);
+
+int
+virtio_transport_seqpacket_enqueue(struct vsock_sock *vsk,
+ struct msghdr *msg,
+ size_t len)
+{
+ struct virtio_vsock_sock *vvs = vsk->trans;
+
+ spin_lock_bh(&vvs->tx_lock);
+
+ if (len > vvs->peer_buf_alloc) {
+ spin_unlock_bh(&vvs->tx_lock);
+ return -EMSGSIZE;
+ }
+
+ spin_unlock_bh(&vvs->tx_lock);
+
+ return virtio_transport_stream_enqueue(vsk, msg, len);
+}
+EXPORT_SYMBOL_GPL(virtio_transport_seqpacket_enqueue);
+
int
virtio_transport_dgram_dequeue(struct vsock_sock *vsk,
struct msghdr *msg,
}
EXPORT_SYMBOL_GPL(virtio_transport_stream_has_data);
+u32 virtio_transport_seqpacket_has_data(struct vsock_sock *vsk)
+{
+ struct virtio_vsock_sock *vvs = vsk->trans;
+ u32 msg_count;
+
+ spin_lock_bh(&vvs->rx_lock);
+ msg_count = vvs->msg_count;
+ spin_unlock_bh(&vvs->rx_lock);
+
+ return msg_count;
+}
+EXPORT_SYMBOL_GPL(virtio_transport_seqpacket_has_data);
+
static s64 virtio_transport_has_space(struct vsock_sock *vsk)
{
struct virtio_vsock_sock *vvs = vsk->trans;
vvs->buf_alloc = *val;
- virtio_transport_send_credit_update(vsk, VIRTIO_VSOCK_TYPE_STREAM,
- NULL);
+ virtio_transport_send_credit_update(vsk);
}
EXPORT_SYMBOL_GPL(virtio_transport_notify_buffer_size);
{
struct virtio_vsock_pkt_info info = {
.op = VIRTIO_VSOCK_OP_REQUEST,
- .type = VIRTIO_VSOCK_TYPE_STREAM,
.vsk = vsk,
};
{
struct virtio_vsock_pkt_info info = {
.op = VIRTIO_VSOCK_OP_SHUTDOWN,
- .type = VIRTIO_VSOCK_TYPE_STREAM,
.flags = (mode & RCV_SHUTDOWN ?
VIRTIO_VSOCK_SHUTDOWN_RCV : 0) |
(mode & SEND_SHUTDOWN ?
{
struct virtio_vsock_pkt_info info = {
.op = VIRTIO_VSOCK_OP_RW,
- .type = VIRTIO_VSOCK_TYPE_STREAM,
.msg = msg,
.pkt_len = len,
.vsk = vsk,
{
struct virtio_vsock_pkt_info info = {
.op = VIRTIO_VSOCK_OP_RST,
- .type = VIRTIO_VSOCK_TYPE_STREAM,
.reply = !!pkt,
.vsk = vsk,
};
struct sock *sk = &vsk->sk;
bool remove_sock = true;
- if (sk->sk_type == SOCK_STREAM)
+ if (sk->sk_type == SOCK_STREAM || sk->sk_type == SOCK_SEQPACKET)
remove_sock = virtio_transport_close(vsk);
if (remove_sock) {
goto out;
}
+ if (le32_to_cpu(pkt->hdr.flags) & VIRTIO_VSOCK_SEQ_EOR)
+ vvs->msg_count++;
+
/* Try to copy small packets into the buffer of last packet queued,
* to avoid wasting memory queueing the entire buffer with a small
* payload.
struct virtio_vsock_pkt, list);
/* If there is space in the last packet queued, we copy the
- * new packet in its buffer.
+ * new packet in its buffer. We avoid this if the last packet
+ * queued has VIRTIO_VSOCK_SEQ_EOR set, because this is
+ * delimiter of SEQPACKET record, so 'pkt' is the first packet
+ * of a new record.
*/
- if (pkt->len <= last_pkt->buf_len - last_pkt->len) {
+ if ((pkt->len <= last_pkt->buf_len - last_pkt->len) &&
+ !(le32_to_cpu(last_pkt->hdr.flags) & VIRTIO_VSOCK_SEQ_EOR)) {
memcpy(last_pkt->buf + last_pkt->len, pkt->buf,
pkt->len);
last_pkt->len += pkt->len;
free_pkt = true;
+ last_pkt->hdr.flags |= pkt->hdr.flags;
goto out;
}
}
{
struct virtio_vsock_pkt_info info = {
.op = VIRTIO_VSOCK_OP_RESPONSE,
- .type = VIRTIO_VSOCK_TYPE_STREAM,
.remote_cid = le64_to_cpu(pkt->hdr.src_cid),
.remote_port = le32_to_cpu(pkt->hdr.src_port),
.reply = true,
return 0;
}
+static bool virtio_transport_valid_type(u16 type)
+{
+ return (type == VIRTIO_VSOCK_TYPE_STREAM) ||
+ (type == VIRTIO_VSOCK_TYPE_SEQPACKET);
+}
+
/* We are under the virtio-vsock's vsock->rx_lock or vhost-vsock's vq->mutex
* lock.
*/
le32_to_cpu(pkt->hdr.buf_alloc),
le32_to_cpu(pkt->hdr.fwd_cnt));
- if (le16_to_cpu(pkt->hdr.type) != VIRTIO_VSOCK_TYPE_STREAM) {
+ if (!virtio_transport_valid_type(le16_to_cpu(pkt->hdr.type))) {
(void)virtio_transport_reset_no_sock(t, pkt);
goto free_pkt;
}
}
}
+ if (virtio_transport_get_type(sk) != le16_to_cpu(pkt->hdr.type)) {
+ (void)virtio_transport_reset_no_sock(t, pkt);
+ sock_put(sk);
+ goto free_pkt;
+ }
+
vsk = vsock_sk(sk);
lock_sock(sk);
vsock_remove_pending(listener, pending);
vsock_enqueue_accept(listener, pending);
- /* Callers of accept() will be be waiting on the listening socket, not
+ /* Callers of accept() will be waiting on the listening socket, not
* the pending socket.
*/
listener->sk_data_ready(listener);
return 0;
}
+static bool vsock_loopback_seqpacket_allow(u32 remote_cid);
+
static struct virtio_transport loopback_transport = {
.transport = {
.module = THIS_MODULE,
.stream_is_active = virtio_transport_stream_is_active,
.stream_allow = virtio_transport_stream_allow,
+ .seqpacket_dequeue = virtio_transport_seqpacket_dequeue,
+ .seqpacket_enqueue = virtio_transport_seqpacket_enqueue,
+ .seqpacket_allow = vsock_loopback_seqpacket_allow,
+ .seqpacket_has_data = virtio_transport_seqpacket_has_data,
+
.notify_poll_in = virtio_transport_notify_poll_in,
.notify_poll_out = virtio_transport_notify_poll_out,
.notify_recv_init = virtio_transport_notify_recv_init,
.send_pkt = vsock_loopback_send_pkt,
};
+static bool vsock_loopback_seqpacket_allow(u32 remote_cid)
+{
+ return true;
+}
+
static void vsock_loopback_work(struct work_struct *work)
{
struct vsock_loopback *vsock =
@$(kecho) " GEN $@"
@(echo '#include "reg.h"'; \
echo 'const u8 shipped_regdb_certs[] = {'; \
- cat $^ ; \
+ echo | cat - $^ ; \
echo '};'; \
echo 'unsigned int shipped_regdb_certs_len = sizeof(shipped_regdb_certs);'; \
) > $@
*
* Copyright 2009 Johannes Berg <johannes@sipsolutions.net>
* Copyright 2013-2014 Intel Mobile Communications GmbH
- * Copyright 2018-2020 Intel Corporation
+ * Copyright 2018-2021 Intel Corporation
*/
#include <linux/export.h>
struct ieee80211_sta_vht_cap *vht_cap;
struct ieee80211_edmg *edmg_cap;
u32 width, control_freq, cap;
- bool support_80_80 = false;
+ bool ext_nss_cap, support_80_80 = false;
if (WARN_ON(!cfg80211_chandef_valid(chandef)))
return false;
ht_cap = &wiphy->bands[chandef->chan->band]->ht_cap;
vht_cap = &wiphy->bands[chandef->chan->band]->vht_cap;
edmg_cap = &wiphy->bands[chandef->chan->band]->edmg_cap;
+ ext_nss_cap = __le16_to_cpu(vht_cap->vht_mcs.tx_highest) &
+ IEEE80211_VHT_EXT_NSS_BW_CAPABLE;
if (edmg_cap->channels &&
!cfg80211_edmg_usable(wiphy,
(cap & IEEE80211_VHT_CAP_SUPP_CHAN_WIDTH_160_80PLUS80MHZ) ||
(cap & IEEE80211_VHT_CAP_SUPP_CHAN_WIDTH_160MHZ &&
cap & IEEE80211_VHT_CAP_EXT_NSS_BW_MASK) ||
- u32_get_bits(cap, IEEE80211_VHT_CAP_EXT_NSS_BW_MASK) > 1;
+ (ext_nss_cap &&
+ u32_get_bits(cap, IEEE80211_VHT_CAP_EXT_NSS_BW_MASK) > 1);
if (chandef->chan->band != NL80211_BAND_6GHZ && !support_80_80)
return false;
fallthrough;
cap = vht_cap->cap & IEEE80211_VHT_CAP_SUPP_CHAN_WIDTH_MASK;
if (cap != IEEE80211_VHT_CAP_SUPP_CHAN_WIDTH_160MHZ &&
cap != IEEE80211_VHT_CAP_SUPP_CHAN_WIDTH_160_80PLUS80MHZ &&
- !(vht_cap->cap & IEEE80211_VHT_CAP_EXT_NSS_BW_MASK))
+ !(ext_nss_cap &&
+ (vht_cap->cap & IEEE80211_VHT_CAP_EXT_NSS_BW_MASK)))
return false;
break;
default:
WARN_ON(1);
}
}
+
+bool cfg80211_any_usable_channels(struct wiphy *wiphy,
+ unsigned long sband_mask,
+ u32 prohibited_flags)
+{
+ int idx;
+
+ prohibited_flags |= IEEE80211_CHAN_DISABLED;
+
+ for_each_set_bit(idx, &sband_mask, NUM_NL80211_BANDS) {
+ struct ieee80211_supported_band *sband = wiphy->bands[idx];
+ int chanidx;
+
+ if (!sband)
+ continue;
+
+ for (chanidx = 0; chanidx < sband->n_channels; chanidx++) {
+ struct ieee80211_channel *chan;
+
+ chan = &sband->channels[chanidx];
+
+ if (chan->flags & prohibited_flags)
+ continue;
+
+ return true;
+ }
+ }
+
+ return false;
+}
+EXPORT_SYMBOL(cfg80211_any_usable_channels);
* Copyright 2006-2010 Johannes Berg <johannes@sipsolutions.net>
* Copyright 2013-2014 Intel Mobile Communications GmbH
* Copyright 2015-2017 Intel Deutschland GmbH
- * Copyright (C) 2018-2020 Intel Corporation
+ * Copyright (C) 2018-2021 Intel Corporation
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
wiphy_net_set(&rdev->wiphy, &init_net);
rdev->rfkill_ops.set_block = cfg80211_rfkill_set_block;
- rdev->rfkill = rfkill_alloc(dev_name(&rdev->wiphy.dev),
- &rdev->wiphy.dev, RFKILL_TYPE_WLAN,
- &rdev->rfkill_ops, rdev);
+ rdev->wiphy.rfkill = rfkill_alloc(dev_name(&rdev->wiphy.dev),
+ &rdev->wiphy.dev, RFKILL_TYPE_WLAN,
+ &rdev->rfkill_ops, rdev);
- if (!rdev->rfkill) {
+ if (!rdev->wiphy.rfkill) {
wiphy_free(&rdev->wiphy);
return NULL;
}
if (WARN_ON(!c->num_different_channels))
return -EINVAL;
- /*
- * Put a sane limit on maximum number of different
- * channels to simplify channel accounting code.
- */
- if (WARN_ON(c->num_different_channels >
- CFG80211_MAX_NUM_DIFFERENT_CHANNELS))
- return -EINVAL;
-
/* DFS only works on one channel. */
if (WARN_ON(c->radar_detect_widths &&
(c->num_different_channels > 1)))
return res;
}
- /* set up regulatory info */
- wiphy_regulatory_register(wiphy);
-
list_add_rcu(&rdev->list, &cfg80211_rdev_list);
cfg80211_rdev_list_generation++;
cfg80211_debugfs_rdev_add(rdev);
nl80211_notify_wiphy(rdev, NL80211_CMD_NEW_WIPHY);
+ /* set up regulatory info */
+ wiphy_regulatory_register(wiphy);
+
if (wiphy->regulatory_flags & REGULATORY_CUSTOM_REG) {
struct regulatory_request request;
rdev->wiphy.registered = true;
rtnl_unlock();
- res = rfkill_register(rdev->rfkill);
+ res = rfkill_register(rdev->wiphy.rfkill);
if (res) {
- rfkill_destroy(rdev->rfkill);
- rdev->rfkill = NULL;
+ rfkill_destroy(rdev->wiphy.rfkill);
+ rdev->wiphy.rfkill = NULL;
wiphy_unregister(&rdev->wiphy);
return res;
}
if (!rdev->ops->rfkill_poll)
return;
rdev->rfkill_ops.poll = cfg80211_rfkill_poll;
- rfkill_resume_polling(rdev->rfkill);
+ rfkill_resume_polling(wiphy->rfkill);
}
EXPORT_SYMBOL(wiphy_rfkill_start_polling);
-void wiphy_rfkill_stop_polling(struct wiphy *wiphy)
-{
- struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy);
-
- rfkill_pause_polling(rdev->rfkill);
-}
-EXPORT_SYMBOL(wiphy_rfkill_stop_polling);
-
void wiphy_unregister(struct wiphy *wiphy)
{
struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy);
wiphy_unlock(&rdev->wiphy);
__count == 0; }));
- if (rdev->rfkill)
- rfkill_unregister(rdev->rfkill);
+ if (rdev->wiphy.rfkill)
+ rfkill_unregister(rdev->wiphy.rfkill);
rtnl_lock();
wiphy_lock(&rdev->wiphy);
{
struct cfg80211_internal_bss *scan, *tmp;
struct cfg80211_beacon_registration *reg, *treg;
- rfkill_destroy(rdev->rfkill);
+ rfkill_destroy(rdev->wiphy.rfkill);
list_for_each_entry_safe(reg, treg, &rdev->beacon_registrations, list) {
list_del(®->list);
kfree(reg);
{
struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy);
- if (rfkill_set_hw_state_reason(rdev->rfkill, blocked, reason))
+ if (rfkill_set_hw_state_reason(wiphy->rfkill, blocked, reason))
schedule_work(&rdev->rfkill_block);
}
EXPORT_SYMBOL(wiphy_rfkill_set_hw_state_reason);
rdev->devlist_generation++;
wdev->registered = true;
+ if (wdev->netdev &&
+ sysfs_create_link(&wdev->netdev->dev.kobj, &rdev->wiphy.dev.kobj,
+ "phy80211"))
+ pr_err("failed to add phy80211 symlink to netdev!\n");
+
nl80211_notify_iface(rdev, wdev, NL80211_CMD_NEW_INTERFACE);
}
if (ret)
goto out;
- if (sysfs_create_link(&dev->dev.kobj, &rdev->wiphy.dev.kobj,
- "phy80211")) {
- pr_err("failed to add phy80211 symlink to netdev!\n");
- unregister_netdevice(dev);
- ret = -EINVAL;
- goto out;
- }
-
cfg80211_register_wdev(rdev, wdev);
ret = 0;
out:
wdev->use_4addr, 0))
return notifier_from_errno(-EOPNOTSUPP);
- if (rfkill_blocked(rdev->rfkill))
+ if (rfkill_blocked(rdev->wiphy.rfkill))
return notifier_from_errno(-ERFKILL);
break;
default:
* Wireless configuration interface internals.
*
* Copyright 2006-2010 Johannes Berg <johannes@sipsolutions.net>
- * Copyright (C) 2018-2020 Intel Corporation
+ * Copyright (C) 2018-2021 Intel Corporation
*/
#ifndef __NET_WIRELESS_CORE_H
#define __NET_WIRELESS_CORE_H
/* rfkill support */
struct rfkill_ops rfkill_ops;
- struct rfkill *rfkill;
struct work_struct rfkill_block;
/* ISO / IEC 3166 alpha2 for which this device is receiving
};
static const struct nla_policy
-nl80211_psmr_peer_attr_policy[NL80211_PMSR_PEER_ATTR_MAX + 1] = {
+nl80211_pmsr_peer_attr_policy[NL80211_PMSR_PEER_ATTR_MAX + 1] = {
[NL80211_PMSR_PEER_ATTR_ADDR] = NLA_POLICY_ETH_ADDR,
[NL80211_PMSR_PEER_ATTR_CHAN] = NLA_POLICY_NESTED(nl80211_policy),
[NL80211_PMSR_PEER_ATTR_REQ] =
[NL80211_PMSR_ATTR_RANDOMIZE_MAC_ADDR] = { .type = NLA_REJECT },
[NL80211_PMSR_ATTR_TYPE_CAPA] = { .type = NLA_REJECT },
[NL80211_PMSR_ATTR_PEERS] =
- NLA_POLICY_NESTED_ARRAY(nl80211_psmr_peer_attr_policy),
+ NLA_POLICY_NESTED_ARRAY(nl80211_pmsr_peer_attr_policy),
};
static const struct nla_policy
&iftdata->he_6ghz_capa))
return -ENOBUFS;
+ if (iftdata->vendor_elems.data && iftdata->vendor_elems.len &&
+ nla_put(msg, NL80211_BAND_IFTYPE_ATTR_VENDOR_ELEMS,
+ iftdata->vendor_elems.len, iftdata->vendor_elems.data))
+ return -ENOBUFS;
+
return 0;
}
sband->ht_cap.mcs.rx_mask,
sizeof(mask->control[i].ht_mcs));
- if (!sband->vht_cap.vht_supported)
- continue;
-
- vht_tx_mcs_map = le16_to_cpu(sband->vht_cap.vht_mcs.tx_mcs_map);
- vht_build_mcs_mask(vht_tx_mcs_map, mask->control[i].vht_mcs);
+ if (sband->vht_cap.vht_supported) {
+ vht_tx_mcs_map = le16_to_cpu(sband->vht_cap.vht_mcs.tx_mcs_map);
+ vht_build_mcs_mask(vht_tx_mcs_map, mask->control[i].vht_mcs);
+ }
he_cap = ieee80211_get_he_iftype_cap(sband, wdev->iftype);
if (!he_cap)
if (wdev_running(wdev))
return 0;
- if (rfkill_blocked(rdev->rfkill))
+ if (rfkill_blocked(rdev->wiphy.rfkill))
return -ERFKILL;
err = rdev_start_p2p_device(rdev, wdev);
if (wdev_running(wdev))
return -EEXIST;
- if (rfkill_blocked(rdev->rfkill))
+ if (rfkill_blocked(rdev->wiphy.rfkill))
return -ERFKILL;
if (!info->attrs[NL80211_ATTR_NAN_MASTER_PREF])
return -EINVAL;
}
+ if (tb[NL80211_PMSR_FTM_REQ_ATTR_BSS_COLOR]) {
+ if (!out->ftm.non_trigger_based && !out->ftm.trigger_based) {
+ NL_SET_ERR_MSG_ATTR(info->extack,
+ tb[NL80211_PMSR_FTM_REQ_ATTR_BSS_COLOR],
+ "FTM: BSS color set for EDCA based ranging");
+ return -EINVAL;
+ }
+
+ out->ftm.bss_color =
+ nla_get_u8(tb[NL80211_PMSR_FTM_REQ_ATTR_BSS_COLOR]);
+ }
+
return 0;
}
gfp_t gfp)
{
struct cfg80211_registered_device *rdev = wiphy_to_rdev(wdev->wiphy);
+ struct cfg80211_pmsr_request *tmp, *prev, *to_free = NULL;
struct sk_buff *msg;
void *hdr;
nlmsg_free(msg);
free_request:
spin_lock_bh(&wdev->pmsr_lock);
- list_del(&req->list);
+ /*
+ * cfg80211_pmsr_process_abort() may have already moved this request
+ * to the free list, and will free it later. In this case, don't free
+ * it here.
+ */
+ list_for_each_entry_safe(tmp, prev, &wdev->pmsr_list, list) {
+ if (tmp == req) {
+ list_del(&req->list);
+ to_free = req;
+ break;
+ }
+ }
spin_unlock_bh(&wdev->pmsr_lock);
- kfree(req);
+ kfree(to_free);
}
EXPORT_SYMBOL_GPL(cfg80211_pmsr_complete);
struct net_device *dev,
struct cfg80211_assoc_request *req)
{
+ const struct cfg80211_bss_ies *bss_ies;
int ret;
- trace_rdev_assoc(&rdev->wiphy, dev, req);
+
+ /*
+ * Note: we might trace not exactly the data that's processed,
+ * due to races and the driver/mac80211 getting a newer copy.
+ */
+ rcu_read_lock();
+ bss_ies = rcu_dereference(req->bss->ies);
+ trace_rdev_assoc(&rdev->wiphy, dev, req, bss_ies);
+ rcu_read_unlock();
+
ret = rdev->ops->assoc(&rdev->wiphy, dev, req);
trace_rdev_return_int(&rdev->wiphy, ret);
return ret;
"wiphy should have REGULATORY_WIPHY_SELF_MANAGED\n"))
return -EPERM;
- if (WARN(!is_valid_rd(rd), "Invalid regulatory domain detected\n")) {
+ if (WARN(!is_valid_rd(rd),
+ "Invalid regulatory domain detected: %c%c\n",
+ rd->alpha2[0], rd->alpha2[1])) {
print_regdomain_info(rd);
return -EINVAL;
}
wiphy_update_regulatory(wiphy, lr->initiator);
wiphy_all_share_dfs_chan_state(wiphy);
+ reg_process_self_managed_hints();
}
void wiphy_regulatory_deregister(struct wiphy *wiphy)
* Copyright 2008 Johannes Berg <johannes@sipsolutions.net>
* Copyright 2013-2014 Intel Mobile Communications GmbH
* Copyright 2016 Intel Deutschland GmbH
- * Copyright (C) 2018-2020 Intel Corporation
+ * Copyright (C) 2018-2021 Intel Corporation
*/
#include <linux/kernel.h>
#include <linux/slab.h>
freq = ieee80211_channel_to_frequency(ap_info->channel, band);
- if (end - pos < count * ap_info->tbtt_info_len)
+ if (end - pos < count * length)
break;
/*
if (band != NL80211_BAND_6GHZ ||
(length != IEEE80211_TBTT_INFO_OFFSET_BSSID_BSS_PARAM &&
length < IEEE80211_TBTT_INFO_OFFSET_BSSID_SSSID_BSS_PARAM)) {
- pos += count * ap_info->tbtt_info_len;
+ pos += count * length;
continue;
}
kfree(entry);
}
- pos += ap_info->tbtt_info_len;
+ pos += length;
}
}
}
request = kzalloc(struct_size(request, channels, n_channels) +
- sizeof(*request->scan_6ghz_params) * count,
+ sizeof(*request->scan_6ghz_params) * count +
+ sizeof(*request->ssids) * rdev_req->n_ssids,
GFP_KERNEL);
if (!request) {
cfg80211_free_coloc_ap_list(&coloc_ap_list);
if (request->n_channels) {
struct cfg80211_scan_request *old = rdev->int_scan_req;
-
rdev->int_scan_req = request;
/*
+ * Add the ssids from the parent scan request to the new scan
+ * request, so the driver would be able to use them in its
+ * probe requests to discover hidden APs on PSC channels.
+ */
+ request->ssids = (void *)&request->channels[request->n_channels];
+ request->n_ssids = rdev_req->n_ssids;
+ memcpy(request->ssids, rdev_req->ssids, sizeof(*request->ssids) *
+ request->n_ssids);
+
+ /*
* If this scan follows a previous scan, save the scan start
* info from the first part of the scan
*/
if (rdev->wiphy.registered && rdev->ops->resume)
ret = rdev_resume(rdev);
wiphy_unlock(&rdev->wiphy);
+
+ if (ret)
+ cfg80211_shutdown_all_interfaces(&rdev->wiphy);
+
rtnl_unlock();
return ret;
TRACE_EVENT(rdev_assoc,
TP_PROTO(struct wiphy *wiphy, struct net_device *netdev,
- struct cfg80211_assoc_request *req),
- TP_ARGS(wiphy, netdev, req),
+ struct cfg80211_assoc_request *req,
+ const struct cfg80211_bss_ies *bss_ies),
+ TP_ARGS(wiphy, netdev, req, bss_ies),
TP_STRUCT__entry(
WIPHY_ENTRY
NETDEV_ENTRY
MAC_ENTRY(prev_bssid)
__field(bool, use_mfp)
__field(u32, flags)
+ __dynamic_array(u8, bss_elements, bss_ies->len)
+ __field(bool, bss_elements_bcon)
+ __field(u64, bss_elements_tsf)
+ __dynamic_array(u8, elements, req->ie_len)
+ __array(u8, ht_capa, sizeof(struct ieee80211_ht_cap))
+ __array(u8, ht_capa_mask, sizeof(struct ieee80211_ht_cap))
+ __array(u8, vht_capa, sizeof(struct ieee80211_vht_cap))
+ __array(u8, vht_capa_mask, sizeof(struct ieee80211_vht_cap))
+ __dynamic_array(u8, fils_kek, req->fils_kek_len)
+ __dynamic_array(u8, fils_nonces,
+ req->fils_nonces ? 2 * FILS_NONCE_LEN : 0)
),
TP_fast_assign(
WIPHY_ASSIGN;
MAC_ASSIGN(prev_bssid, req->prev_bssid);
__entry->use_mfp = req->use_mfp;
__entry->flags = req->flags;
+ if (bss_ies->len)
+ memcpy(__get_dynamic_array(bss_elements),
+ bss_ies->data, bss_ies->len);
+ __entry->bss_elements_bcon = bss_ies->from_beacon;
+ __entry->bss_elements_tsf = bss_ies->tsf;
+ if (req->ie)
+ memcpy(__get_dynamic_array(elements),
+ req->ie, req->ie_len);
+ memcpy(__entry->ht_capa, &req->ht_capa, sizeof(req->ht_capa));
+ memcpy(__entry->ht_capa_mask, &req->ht_capa_mask,
+ sizeof(req->ht_capa_mask));
+ memcpy(__entry->vht_capa, &req->vht_capa, sizeof(req->vht_capa));
+ memcpy(__entry->vht_capa_mask, &req->vht_capa_mask,
+ sizeof(req->vht_capa_mask));
+ if (req->fils_kek)
+ memcpy(__get_dynamic_array(fils_kek),
+ req->fils_kek, req->fils_kek_len);
+ if (req->fils_nonces)
+ memcpy(__get_dynamic_array(fils_nonces),
+ req->fils_nonces, 2 * FILS_NONCE_LEN);
),
TP_printk(WIPHY_PR_FMT ", " NETDEV_PR_FMT ", bssid: " MAC_PR_FMT
", previous bssid: " MAC_PR_FMT ", use mfp: %s, flags: %u",
int ieee80211_data_to_8023_exthdr(struct sk_buff *skb, struct ethhdr *ehdr,
const u8 *addr, enum nl80211_iftype iftype,
- u8 data_offset)
+ u8 data_offset, bool is_amsdu)
{
struct ieee80211_hdr *hdr = (struct ieee80211_hdr *) skb->data;
struct {
skb_copy_bits(skb, hdrlen, &payload, sizeof(payload));
tmp.h_proto = payload.proto;
- if (likely((ether_addr_equal(payload.hdr, rfc1042_header) &&
+ if (likely((!is_amsdu && ether_addr_equal(payload.hdr, rfc1042_header) &&
tmp.h_proto != htons(ETH_P_AARP) &&
tmp.h_proto != htons(ETH_P_IPX)) ||
ether_addr_equal(payload.hdr, bridge_tunnel_header)))
remaining = skb->len - offset;
if (subframe_len > remaining)
goto purge;
+ /* mitigate A-MSDU aggregation injection attacks */
+ if (ether_addr_equal(eth.h_dest, rfc1042_header))
+ goto purge;
offset += sizeof(struct ethhdr);
last = remaining <= subframe_len + padding;
case NL80211_IFTYPE_MESH_POINT:
/* mesh should be handled? */
break;
+ case NL80211_IFTYPE_OCB:
+ cfg80211_leave_ocb(rdev, dev);
+ break;
default:
break;
}
/* only change when not disabling */
if (!data->txpower.disabled) {
- rfkill_set_sw_state(rdev->rfkill, false);
+ rfkill_set_sw_state(rdev->wiphy.rfkill, false);
if (data->txpower.fixed) {
/*
}
}
} else {
- if (rfkill_set_sw_state(rdev->rfkill, true))
+ if (rfkill_set_sw_state(rdev->wiphy.rfkill, true))
schedule_work(&rdev->rfkill_block);
return 0;
}
/* well... oh well */
data->txpower.fixed = 1;
- data->txpower.disabled = rfkill_blocked(rdev->rfkill);
+ data->txpower.disabled = rfkill_blocked(rdev->wiphy.rfkill);
data->txpower.value = val;
data->txpower.flags = IW_TXPOW_DBM;
{
struct wireless_dev *wdev = dev->ieee80211_ptr;
struct cfg80211_registered_device *rdev = wiphy_to_rdev(wdev->wiphy);
- bool ps = wdev->ps;
+ bool ps;
int timeout = wdev->ps_timeout;
int err;
return -EOPNOTSUPP;
/* Just do it */
- memcpy(&(spydata->spy_thr_low), &(threshold->low),
- 2 * sizeof(struct iw_quality));
+ spydata->spy_thr_low = threshold->low;
+ spydata->spy_thr_high = threshold->high;
/* Clear flag */
memset(spydata->spy_thr_under, '\0', sizeof(spydata->spy_thr_under));
return -EOPNOTSUPP;
/* Just do it */
- memcpy(&(threshold->low), &(spydata->spy_thr_low),
- 2 * sizeof(struct iw_quality));
+ threshold->low = spydata->spy_thr_low;
+ threshold->high = spydata->spy_thr_high;
return 0;
}
memcpy(threshold.addr.sa_data, address, ETH_ALEN);
threshold.addr.sa_family = ARPHRD_ETHER;
/* Copy stats */
- memcpy(&(threshold.qual), wstats, sizeof(struct iw_quality));
+ threshold.qual = *wstats;
/* Copy also thresholds */
- memcpy(&(threshold.low), &(spydata->spy_thr_low),
- 2 * sizeof(struct iw_quality));
+ threshold.low = spydata->spy_thr_low;
+ threshold.high = spydata->spy_thr_high;
/* Send event to user space */
wireless_send_event(dev, SIOCGIWTHRSPY, &wrqu, (char *) &threshold);
/*
* This is called from user mode and the timers. Thus it protects itself
- * against interrupt users but doesn't worry about being called during
+ * against interrupting users but doesn't worry about being called during
* work. Once it is removed from the queue no interrupt or bottom half
* will touch it and we are (fairly 8-) ) safe.
* Not static as it's used by the timer
if (protocol)
goto out;
- rc = -ENOBUFS;
+ rc = -ENOMEM;
if ((sk = x25_alloc_socket(net, kern)) == NULL)
goto out;
{
struct x25_route *rt;
struct x25_neigh *neigh_new = NULL;
- struct list_head *entry;
struct x25_forward *x25_frwd, *new_frwd;
struct sk_buff *skbn;
short same_lci = 0;
* established LCI? It shouldn't happen, just in case..
*/
read_lock_bh(&x25_forward_list_lock);
- list_for_each(entry, &x25_forward_list) {
- x25_frwd = list_entry(entry, struct x25_forward, node);
+ list_for_each_entry(x25_frwd, &x25_forward_list, node) {
if (x25_frwd->lci == lci) {
pr_warn("call request for lci which is already registered!, transmitting but not registering new pair\n");
same_lci = 1;
int x25_forward_data(int lci, struct x25_neigh *from, struct sk_buff *skb) {
struct x25_forward *frwd;
- struct list_head *entry;
struct net_device *peer = NULL;
struct x25_neigh *nb;
struct sk_buff *skbn;
int rc = 0;
read_lock_bh(&x25_forward_list_lock);
- list_for_each(entry, &x25_forward_list) {
- frwd = list_entry(entry, struct x25_forward, node);
+ list_for_each_entry(frwd, &x25_forward_list, node) {
if (frwd->lci == lci) {
/* The call is established, either side can send */
if (from->dev == frwd->dev1) {
struct x25_neigh *x25_get_neigh(struct net_device *dev)
{
struct x25_neigh *nb, *use = NULL;
- struct list_head *entry;
read_lock_bh(&x25_neigh_list_lock);
- list_for_each(entry, &x25_neigh_list) {
- nb = list_entry(entry, struct x25_neigh, node);
-
+ list_for_each_entry(nb, &x25_neigh_list, node) {
if (nb->dev == dev) {
use = nb;
break;
struct net_device *dev)
{
struct x25_route *rt;
- struct list_head *entry;
int rc = -EINVAL;
write_lock_bh(&x25_route_list_lock);
- list_for_each(entry, &x25_route_list) {
- rt = list_entry(entry, struct x25_route, node);
-
+ list_for_each_entry(rt, &x25_route_list, node) {
if (!memcmp(&rt->address, address, sigdigits) &&
rt->sigdigits == sigdigits)
goto out;
struct net_device *dev)
{
struct x25_route *rt;
- struct list_head *entry;
int rc = -EINVAL;
write_lock_bh(&x25_route_list_lock);
- list_for_each(entry, &x25_route_list) {
- rt = list_entry(entry, struct x25_route, node);
-
+ list_for_each_entry(rt, &x25_route_list, node) {
if (!memcmp(&rt->address, address, sigdigits) &&
rt->sigdigits == sigdigits && rt->dev == dev) {
__x25_remove_route(rt);
struct x25_route *x25_get_route(struct x25_address *addr)
{
struct x25_route *rt, *use = NULL;
- struct list_head *entry;
read_lock_bh(&x25_route_list_lock);
- list_for_each(entry, &x25_route_list) {
- rt = list_entry(entry, struct x25_route, node);
-
+ list_for_each_entry(rt, &x25_route_list, node) {
if (!memcmp(&rt->address, addr, rt->sigdigits)) {
if (!use)
use = rt;
{
unpin_user_pages_dirty_lock(umem->pgs, umem->npgs, true);
- kfree(umem->pgs);
+ kvfree(umem->pgs);
umem->pgs = NULL;
}
long npgs;
int err;
- umem->pgs = kcalloc(umem->npgs, sizeof(*umem->pgs),
- GFP_KERNEL | __GFP_NOWARN);
+ umem->pgs = kvcalloc(umem->npgs, sizeof(*umem->pgs), GFP_KERNEL | __GFP_NOWARN);
if (!umem->pgs)
return -ENOMEM;
out_pin:
xdp_umem_unpin_pages(umem);
out_pgs:
- kfree(umem->pgs);
+ kvfree(umem->pgs);
umem->pgs = NULL;
return err;
}
static int xsk_map_redirect(struct bpf_map *map, u32 ifindex, u64 flags)
{
- return __bpf_xdp_redirect_map(map, ifindex, flags, __xsk_map_lookup_elem);
+ return __bpf_xdp_redirect_map(map, ifindex, flags, 0,
+ __xsk_map_lookup_elem);
}
void xsk_map_try_sock_delete(struct xsk_map *map, struct xdp_sock *xs,
return 0;
}
+/* For partial checksum offload, the outer header checksum is calculated
+ * by software and the inner header checksum is calculated by hardware.
+ * This requires hardware to know the inner packet type to calculate
+ * the inner header checksum. Save inner ip protocol here to avoid
+ * traversing the packet in the vendor's xmit code.
+ * If the encap type is IPIP, just save skb->inner_ipproto. Otherwise,
+ * get the ip protocol from the IP header.
+ */
+static void xfrm_get_inner_ipproto(struct sk_buff *skb)
+{
+ struct xfrm_offload *xo = xfrm_offload(skb);
+ const struct ethhdr *eth;
+
+ if (!xo)
+ return;
+
+ if (skb->inner_protocol_type == ENCAP_TYPE_IPPROTO) {
+ xo->inner_ipproto = skb->inner_ipproto;
+ return;
+ }
+
+ if (skb->inner_protocol_type != ENCAP_TYPE_ETHER)
+ return;
+
+ eth = (struct ethhdr *)skb_inner_mac_header(skb);
+
+ switch (ntohs(eth->h_proto)) {
+ case ETH_P_IPV6:
+ xo->inner_ipproto = inner_ipv6_hdr(skb)->nexthdr;
+ break;
+ case ETH_P_IP:
+ xo->inner_ipproto = inner_ip_hdr(skb)->protocol;
+ break;
+ }
+}
+
int xfrm_output(struct sock *sk, struct sk_buff *skb)
{
struct net *net = dev_net(skb_dst(skb)->dev);
kfree_skb(skb);
return -ENOMEM;
}
- skb->encapsulation = 1;
sp->olen++;
sp->xvec[sp->len++] = x;
xfrm_state_hold(x);
+ if (skb->encapsulation)
+ xfrm_get_inner_ipproto(skb);
+ skb->encapsulation = 1;
+
if (skb_is_gso(skb)) {
if (skb->inner_protocol)
return xfrm_output_gso(net, sk, skb);
tprogs-y += per_socket_stats_example
tprogs-y += xdp_redirect
tprogs-y += xdp_redirect_map
+tprogs-y += xdp_redirect_map_multi
tprogs-y += xdp_redirect_cpu
tprogs-y += xdp_monitor
tprogs-y += xdp_rxq_info
per_socket_stats_example-objs := cookie_uid_helper_example.o
xdp_redirect-objs := xdp_redirect_user.o
xdp_redirect_map-objs := xdp_redirect_map_user.o
+xdp_redirect_map_multi-objs := xdp_redirect_map_multi_user.o
xdp_redirect_cpu-objs := xdp_redirect_cpu_user.o
xdp_monitor-objs := xdp_monitor_user.o
xdp_rxq_info-objs := xdp_rxq_info_user.o
always-y += tcp_dumpstats_kern.o
always-y += xdp_redirect_kern.o
always-y += xdp_redirect_map_kern.o
+always-y += xdp_redirect_map_multi_kern.o
always-y += xdp_redirect_cpu_kern.o
always-y += xdp_monitor_kern.o
always-y += xdp_rxq_info_kern.o
// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
-/**
+/*
* ibumad BPF sample kernel side
*
* This program is free software; you can redistribute it and/or
// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
-/**
+/*
* ibumad BPF sample user side
*
* This program is free software; you can redistribute it and/or
* on different systems with different compilers. The right way is
* to parse the ELF file. We took a shortcut here.
*/
- uprobe_file_offset = (__u64)main - (__u64)&__executable_start;
+ uprobe_file_offset = (unsigned long)main - (unsigned long)&__executable_start;
CHECK_AND_RET(test_nondebug_fs_probe("uprobe", (char *)argv[0],
uprobe_file_offset, 0x0, false,
BPF_FD_TYPE_UPROBE,
"usage: %s [OPTS] interface-list\n"
"\nOPTS:\n"
" -d detach program\n"
+ " -S use skb-mode\n"
+ " -F force loading prog\n"
" -D direct table lookups (skip fib rules)\n",
prog);
}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+#define KBUILD_MODNAME "foo"
+#include <uapi/linux/bpf.h>
+#include <linux/in.h>
+#include <linux/if_ether.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include <bpf/bpf_helpers.h>
+
+struct {
+ __uint(type, BPF_MAP_TYPE_DEVMAP_HASH);
+ __uint(key_size, sizeof(int));
+ __uint(value_size, sizeof(int));
+ __uint(max_entries, 32);
+} forward_map_general SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_DEVMAP_HASH);
+ __uint(key_size, sizeof(int));
+ __uint(value_size, sizeof(struct bpf_devmap_val));
+ __uint(max_entries, 32);
+} forward_map_native SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+ __type(key, u32);
+ __type(value, long);
+ __uint(max_entries, 1);
+} rxcnt SEC(".maps");
+
+/* map to store egress interfaces mac addresses, set the
+ * max_entries to 1 and extend it in user sapce prog.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __type(key, u32);
+ __type(value, __be64);
+ __uint(max_entries, 1);
+} mac_map SEC(".maps");
+
+static int xdp_redirect_map(struct xdp_md *ctx, void *forward_map)
+{
+ long *value;
+ u32 key = 0;
+
+ /* count packet in global counter */
+ value = bpf_map_lookup_elem(&rxcnt, &key);
+ if (value)
+ *value += 1;
+
+ return bpf_redirect_map(forward_map, key,
+ BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS);
+}
+
+SEC("xdp_redirect_general")
+int xdp_redirect_map_general(struct xdp_md *ctx)
+{
+ return xdp_redirect_map(ctx, &forward_map_general);
+}
+
+SEC("xdp_redirect_native")
+int xdp_redirect_map_native(struct xdp_md *ctx)
+{
+ return xdp_redirect_map(ctx, &forward_map_native);
+}
+
+SEC("xdp_devmap/map_prog")
+int xdp_devmap_prog(struct xdp_md *ctx)
+{
+ void *data_end = (void *)(long)ctx->data_end;
+ void *data = (void *)(long)ctx->data;
+ u32 key = ctx->egress_ifindex;
+ struct ethhdr *eth = data;
+ __be64 *mac;
+ u64 nh_off;
+
+ nh_off = sizeof(*eth);
+ if (data + nh_off > data_end)
+ return XDP_DROP;
+
+ mac = bpf_map_lookup_elem(&mac_map, &key);
+ if (mac)
+ __builtin_memcpy(eth->h_source, mac, ETH_ALEN);
+
+ return XDP_PASS;
+}
+
+char _license[] SEC("license") = "GPL";
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <linux/if_link.h>
+#include <assert.h>
+#include <errno.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <net/if.h>
+#include <unistd.h>
+#include <libgen.h>
+#include <sys/resource.h>
+#include <sys/ioctl.h>
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
+
+#include "bpf_util.h"
+#include <bpf/bpf.h>
+#include <bpf/libbpf.h>
+
+#define MAX_IFACE_NUM 32
+
+static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST;
+static int ifaces[MAX_IFACE_NUM] = {};
+static int rxcnt_map_fd;
+
+static void int_exit(int sig)
+{
+ __u32 prog_id = 0;
+ int i;
+
+ for (i = 0; ifaces[i] > 0; i++) {
+ if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) {
+ printf("bpf_get_link_xdp_id failed\n");
+ exit(1);
+ }
+ if (prog_id)
+ bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags);
+ }
+
+ exit(0);
+}
+
+static void poll_stats(int interval)
+{
+ unsigned int nr_cpus = bpf_num_possible_cpus();
+ __u64 values[nr_cpus], prev[nr_cpus];
+
+ memset(prev, 0, sizeof(prev));
+
+ while (1) {
+ __u64 sum = 0;
+ __u32 key = 0;
+ int i;
+
+ sleep(interval);
+ assert(bpf_map_lookup_elem(rxcnt_map_fd, &key, values) == 0);
+ for (i = 0; i < nr_cpus; i++)
+ sum += (values[i] - prev[i]);
+ if (sum)
+ printf("Forwarding %10llu pkt/s\n", sum / interval);
+ memcpy(prev, values, sizeof(values));
+ }
+}
+
+static int get_mac_addr(unsigned int ifindex, void *mac_addr)
+{
+ char ifname[IF_NAMESIZE];
+ struct ifreq ifr;
+ int fd, ret = -1;
+
+ fd = socket(AF_INET, SOCK_DGRAM, 0);
+ if (fd < 0)
+ return ret;
+
+ if (!if_indextoname(ifindex, ifname))
+ goto err_out;
+
+ strcpy(ifr.ifr_name, ifname);
+
+ if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0)
+ goto err_out;
+
+ memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char));
+ ret = 0;
+
+err_out:
+ close(fd);
+ return ret;
+}
+
+static int update_mac_map(struct bpf_object *obj)
+{
+ int i, ret = -1, mac_map_fd;
+ unsigned char mac_addr[6];
+ unsigned int ifindex;
+
+ mac_map_fd = bpf_object__find_map_fd_by_name(obj, "mac_map");
+ if (mac_map_fd < 0) {
+ printf("find mac map fd failed\n");
+ return ret;
+ }
+
+ for (i = 0; ifaces[i] > 0; i++) {
+ ifindex = ifaces[i];
+
+ ret = get_mac_addr(ifindex, mac_addr);
+ if (ret < 0) {
+ printf("get interface %d mac failed\n", ifindex);
+ return ret;
+ }
+
+ ret = bpf_map_update_elem(mac_map_fd, &ifindex, mac_addr, 0);
+ if (ret) {
+ perror("bpf_update_elem mac_map_fd");
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static void usage(const char *prog)
+{
+ fprintf(stderr,
+ "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n"
+ "OPTS:\n"
+ " -S use skb-mode\n"
+ " -N enforce native mode\n"
+ " -F force loading prog\n"
+ " -X load xdp program on egress\n",
+ prog);
+}
+
+int main(int argc, char **argv)
+{
+ int i, ret, opt, forward_map_fd, max_ifindex = 0;
+ struct bpf_program *ingress_prog, *egress_prog;
+ int ingress_prog_fd, egress_prog_fd = 0;
+ struct bpf_devmap_val devmap_val;
+ bool attach_egress_prog = false;
+ char ifname[IF_NAMESIZE];
+ struct bpf_map *mac_map;
+ struct bpf_object *obj;
+ unsigned int ifindex;
+ char filename[256];
+
+ while ((opt = getopt(argc, argv, "SNFX")) != -1) {
+ switch (opt) {
+ case 'S':
+ xdp_flags |= XDP_FLAGS_SKB_MODE;
+ break;
+ case 'N':
+ /* default, set below */
+ break;
+ case 'F':
+ xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST;
+ break;
+ case 'X':
+ attach_egress_prog = true;
+ break;
+ default:
+ usage(basename(argv[0]));
+ return 1;
+ }
+ }
+
+ if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) {
+ xdp_flags |= XDP_FLAGS_DRV_MODE;
+ } else if (attach_egress_prog) {
+ printf("Load xdp program on egress with SKB mode not supported yet\n");
+ return 1;
+ }
+
+ if (optind == argc) {
+ printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]);
+ return 1;
+ }
+
+ printf("Get interfaces");
+ for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) {
+ ifaces[i] = if_nametoindex(argv[optind + i]);
+ if (!ifaces[i])
+ ifaces[i] = strtoul(argv[optind + i], NULL, 0);
+ if (!if_indextoname(ifaces[i], ifname)) {
+ perror("Invalid interface name or i");
+ return 1;
+ }
+
+ /* Find the largest index number */
+ if (ifaces[i] > max_ifindex)
+ max_ifindex = ifaces[i];
+
+ printf(" %d", ifaces[i]);
+ }
+ printf("\n");
+
+ snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+
+ obj = bpf_object__open(filename);
+ if (libbpf_get_error(obj)) {
+ printf("ERROR: opening BPF object file failed\n");
+ obj = NULL;
+ goto err_out;
+ }
+
+ /* Reset the map size to max ifindex + 1 */
+ if (attach_egress_prog) {
+ mac_map = bpf_object__find_map_by_name(obj, "mac_map");
+ ret = bpf_map__resize(mac_map, max_ifindex + 1);
+ if (ret < 0) {
+ printf("ERROR: reset mac map size failed\n");
+ goto err_out;
+ }
+ }
+
+ /* load BPF program */
+ if (bpf_object__load(obj)) {
+ printf("ERROR: loading BPF object file failed\n");
+ goto err_out;
+ }
+
+ if (xdp_flags & XDP_FLAGS_SKB_MODE) {
+ ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_general");
+ forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_general");
+ } else {
+ ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_native");
+ forward_map_fd = bpf_object__find_map_fd_by_name(obj, "forward_map_native");
+ }
+ if (!ingress_prog || forward_map_fd < 0) {
+ printf("finding ingress_prog/forward_map in obj file failed\n");
+ goto err_out;
+ }
+
+ ingress_prog_fd = bpf_program__fd(ingress_prog);
+ if (ingress_prog_fd < 0) {
+ printf("find ingress_prog fd failed\n");
+ goto err_out;
+ }
+
+ rxcnt_map_fd = bpf_object__find_map_fd_by_name(obj, "rxcnt");
+ if (rxcnt_map_fd < 0) {
+ printf("bpf_object__find_map_fd_by_name failed\n");
+ goto err_out;
+ }
+
+ if (attach_egress_prog) {
+ /* Update mac_map with all egress interfaces' mac addr */
+ if (update_mac_map(obj) < 0) {
+ printf("Error: update mac map failed");
+ goto err_out;
+ }
+
+ /* Find egress prog fd */
+ egress_prog = bpf_object__find_program_by_name(obj, "xdp_devmap_prog");
+ if (!egress_prog) {
+ printf("finding egress_prog in obj file failed\n");
+ goto err_out;
+ }
+ egress_prog_fd = bpf_program__fd(egress_prog);
+ if (egress_prog_fd < 0) {
+ printf("find egress_prog fd failed\n");
+ goto err_out;
+ }
+ }
+
+ /* Remove attached program when program is interrupted or killed */
+ signal(SIGINT, int_exit);
+ signal(SIGTERM, int_exit);
+
+ /* Init forward multicast groups */
+ for (i = 0; ifaces[i] > 0; i++) {
+ ifindex = ifaces[i];
+
+ /* bind prog_fd to each interface */
+ ret = bpf_set_link_xdp_fd(ifindex, ingress_prog_fd, xdp_flags);
+ if (ret) {
+ printf("Set xdp fd failed on %d\n", ifindex);
+ goto err_out;
+ }
+
+ /* Add all the interfaces to forward group and attach
+ * egress devmap programe if exist
+ */
+ devmap_val.ifindex = ifindex;
+ devmap_val.bpf_prog.fd = egress_prog_fd;
+ ret = bpf_map_update_elem(forward_map_fd, &ifindex, &devmap_val, 0);
+ if (ret) {
+ perror("bpf_map_update_elem forward_map");
+ goto err_out;
+ }
+ }
+
+ poll_stats(2);
+
+ return 0;
+
+err_out:
+ return 1;
+}
fprintf(stderr,
"%s: %s [OPTS] <ifname|ifindex>\n\n"
"OPTS:\n"
- " -F force loading prog\n",
+ " -F force loading prog\n"
+ " -S use skb-mode\n",
__func__, prog);
}
for (i = 0; i < batch_size; i++) {
struct xdp_desc *tx_desc = xsk_ring_prod__tx_desc(&xsk->tx,
idx + i);
- tx_desc->addr = (*frame_nb + i) << XSK_UMEM__DEFAULT_FRAME_SHIFT;
+ tx_desc->addr = (*frame_nb + i) * opt_xsk_frame_size;
tx_desc->len = PKT_SIZE;
}
echo " -d : (\$DEST_IP) destination IP. CIDR (e.g. 198.18.0.0/15) is also allowed"
echo " -m : (\$DST_MAC) destination MAC-addr"
echo " -p : (\$DST_PORT) destination PORT range (e.g. 433-444) is also allowed"
+ echo " -k : (\$UDP_CSUM) enable UDP tx checksum"
echo " -t : (\$THREADS) threads to start"
echo " -f : (\$F_THREAD) index of first thread (zero indexed CPU number)"
echo " -c : (\$SKB_CLONE) SKB clones send before alloc new SKB"
## --- Parse command line arguments / parameters ---
## echo "Commandline options:"
-while getopts "s:i:d:m:p:f:t:c:n:b:w:vxh6a" option; do
+while getopts "s:i:d:m:p:f:t:c:n:b:w:vxh6ak" option; do
case $option in
i) # interface
export DEV=$OPTARG
export APPEND=yes
info "Append mode: APPEND=$APPEND"
;;
+ k)
+ export UDP_CSUM=yes
+ info "UDP tx checksum: UDP_CSUM=$UDP_CSUM"
+ ;;
h|?|*)
usage;
err 2 "[ERROR] Unknown parameters!!!"
pg_set $DEV "udp_dst_max $UDP_DST_MAX"
fi
+[ ! -z "$UDP_CSUM" ] && pg_set $dev "flag UDPCSUM"
+
# Setup random UDP port src range
pg_set $DEV "flag UDPSRC_RND"
pg_set $DEV "udp_src_min $UDP_SRC_MIN"
pg_set $dev "udp_dst_max $UDP_DST_MAX"
fi
+ [ ! -z "$UDP_CSUM" ] && pg_set $dev "flag UDPCSUM"
+
# Setup random UDP port src range
pg_set $dev "flag UDPSRC_RND"
pg_set $dev "udp_src_min $UDP_SRC_MIN"
pg_set $dev "udp_dst_max $UDP_DST_MAX"
fi
+ [ ! -z "$UDP_CSUM" ] && pg_set $dev "flag UDPCSUM"
+
# Setup burst, for easy testing -b 0 disable bursting
# (internally in pktgen default and minimum burst=1)
if [[ ${BURST} -ne 0 ]]; then
pg_set $dev "udp_dst_max $UDP_DST_MAX"
fi
+ [ ! -z "$UDP_CSUM" ] && pg_set $dev "flag UDPCSUM"
+
# Randomize source IP-addresses
pg_set $dev "flag IPSRC_RND"
pg_set $dev "src_min $SRC_MIN"
pg_set $dev "udp_dst_max $UDP_DST_MAX"
fi
+ [ ! -z "$UDP_CSUM" ] && pg_set $dev "flag UDPCSUM"
+
# Setup source IP-addresses based on thread number
pg_set $dev "src_min 198.18.$((thread+1)).1"
pg_set $dev "src_max 198.18.$((thread+1)).1"
pg_set $dev "udp_dst_max $UDP_DST_MAX"
fi
+ [ ! -z "$UDP_CSUM" ] && pg_set $dev "flag UDPCSUM"
+
# Setup random UDP port src range
pg_set $dev "flag UDPSRC_RND"
pg_set $dev "udp_src_min $UDP_SRC_MIN"
if (format != DRM_FORMAT_XRGB8888) {
pci_err(pdev, "format mismatch (0x%x != 0x%x)\n",
format, DRM_FORMAT_XRGB8888);
- return -EINVAL;
+ ret = -EINVAL;
+ goto err_release_regions;
}
if (width < 100 || width > 10000) {
pci_err(pdev, "width (%d) out of range\n", width);
- return -EINVAL;
+ ret = -EINVAL;
+ goto err_release_regions;
}
if (height < 100 || height > 10000) {
pci_err(pdev, "height (%d) out of range\n", height);
- return -EINVAL;
+ ret = -EINVAL;
+ goto err_release_regions;
}
pci_info(pdev, "mdpy found: %dx%d framebuffer\n",
width, height);
info = framebuffer_alloc(sizeof(struct mdpy_fb_par), &pdev->dev);
- if (!info)
+ if (!info) {
+ ret = -ENOMEM;
goto err_release_regions;
+ }
pci_set_drvdata(pdev, info);
par = info->par;
quiet_cmd_btf_ko = BTF [M] $@
cmd_btf_ko = \
if [ -f vmlinux ]; then \
- LLVM_OBJCOPY=$(OBJCOPY) $(PAHOLE) -J --btf_base vmlinux $@; \
+ LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J --btf_base vmlinux $@; \
else \
printf "Skipping BTF generation for %s due to unavailability of vmlinux\n" $@ 1>&2; \
fi;
if arg_contain -S "$@"; then
# For scripts/gcc-x86-*-has-stack-protector.sh
if arg_contain -fstack-protector "$@"; then
- echo "%gs"
+ if arg_contain -mstack-protector-guard-reg=fs "$@"; then
+ echo "%fs"
+ else
+ echo "%gs"
+ fi
exit 0
fi
import os, sys, errno
import subprocess
-# Extract and prepare jobserver file descriptors from envirnoment.
+# Extract and prepare jobserver file descriptors from environment.
claim = 0
jobs = b""
try:
fi
info "BTF" ${2}
- LLVM_OBJCOPY=${OBJCOPY} ${PAHOLE} -J ${extra_paholeopt} ${1}
+ LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${extra_paholeopt} ${1}
# Create ${2} which contains just .BTF section but no symbols. Add
# SHF_ALLOC because .BTF will be part of the vmlinux image. --strip-all
Elf32_Word const *symtab_shndx)
{
unsigned long offset;
+ unsigned short shndx = w2(sym->st_shndx);
int index;
- if (sym->st_shndx != SHN_XINDEX)
- return w2(sym->st_shndx);
+ if (shndx > SHN_UNDEF && shndx < SHN_LORESERVE)
+ return shndx;
- offset = (unsigned long)sym - (unsigned long)symtab;
- index = offset / sizeof(*sym);
+ if (shndx == SHN_XINDEX) {
+ offset = (unsigned long)sym - (unsigned long)symtab;
+ index = offset / sizeof(*sym);
- return w(symtab_shndx[index]);
+ return w(symtab_shndx[index]);
+ }
+
+ return 0;
}
static unsigned int get_shnum(Elf_Ehdr const *ehdr, Elf_Shdr const *shdr0)
ret = tpm_get_random(chip, td->nonceodd, TPM_NONCE_SIZE);
if (ret < 0)
- return ret;
+ goto out;
- if (ret != TPM_NONCE_SIZE)
- return -EIO;
+ if (ret != TPM_NONCE_SIZE) {
+ ret = -EIO;
+ goto out;
+ }
ordinal = htonl(TPM_ORD_SEAL);
datsize = htonl(datalen);
rc = -EPERM;
}
if (blob_len < 0)
- return blob_len;
-
- payload->blob_len = blob_len;
+ rc = blob_len;
+ else
+ payload->blob_len = blob_len;
tpm_put_ops(chip);
return rc;
#define MAX_LED (((SNDRV_CTL_ELEM_ACCESS_MIC_LED - SNDRV_CTL_ELEM_ACCESS_SPK_LED) \
>> SNDRV_CTL_ELEM_ACCESS_LED_SHIFT) + 1)
+#define to_led_card_dev(_dev) \
+ container_of(_dev, struct snd_ctl_led_card, dev)
+
enum snd_ctl_led_mode {
MODE_FOLLOW_MUTE = 0,
MODE_FOLLOW_ROUTE,
snd_ctl_led_refresh();
}
+static void snd_ctl_led_card_release(struct device *dev)
+{
+ struct snd_ctl_led_card *led_card = to_led_card_dev(dev);
+
+ kfree(led_card);
+}
+
+static void snd_ctl_led_release(struct device *dev)
+{
+}
+
+static void snd_ctl_led_dev_release(struct device *dev)
+{
+}
+
/*
* sysfs
*/
led_card->number = card->number;
led_card->led = led;
device_initialize(&led_card->dev);
+ led_card->dev.release = snd_ctl_led_card_release;
if (dev_set_name(&led_card->dev, "card%d", card->number) < 0)
goto cerr;
led_card->dev.parent = &led->dev;
put_device(&led_card->dev);
cerr2:
printk(KERN_ERR "snd_ctl_led: unable to add card%d", card->number);
- kfree(led_card);
}
}
snprintf(link_name, sizeof(link_name), "led-%s", led->name);
sysfs_remove_link(&card->ctl_dev.kobj, link_name);
sysfs_remove_link(&led_card->dev.kobj, "card");
- device_del(&led_card->dev);
- kfree(led_card);
+ device_unregister(&led_card->dev);
led->cards[card->number] = NULL;
}
}
device_initialize(&snd_ctl_led_dev);
snd_ctl_led_dev.class = sound_class;
+ snd_ctl_led_dev.release = snd_ctl_led_dev_release;
dev_set_name(&snd_ctl_led_dev, "ctl-led");
if (device_add(&snd_ctl_led_dev)) {
put_device(&snd_ctl_led_dev);
INIT_LIST_HEAD(&led->controls);
device_initialize(&led->dev);
led->dev.parent = &snd_ctl_led_dev;
+ led->dev.release = snd_ctl_led_release;
led->dev.groups = snd_ctl_led_dev_attr_groups;
dev_set_name(&led->dev, led->name);
if (device_add(&led->dev)) {
put_device(&led->dev);
for (; group > 0; group--) {
led = &snd_ctl_leds[group - 1];
- device_del(&led->dev);
+ device_unregister(&led->dev);
}
- device_del(&snd_ctl_led_dev);
+ device_unregister(&snd_ctl_led_dev);
return -ENOMEM;
}
}
}
for (group = 0; group < MAX_LED; group++) {
led = &snd_ctl_leds[group];
- device_del(&led->dev);
+ device_unregister(&led->dev);
}
- device_del(&snd_ctl_led_dev);
+ device_unregister(&snd_ctl_led_dev);
snd_ctl_led_clean(NULL);
}
return err;
}
spin_lock_irq(&tmr->lock);
- tmr->timeri = t;
+ if (tmr->timeri)
+ err = -EBUSY;
+ else
+ tmr->timeri = t;
spin_unlock_irq(&tmr->lock);
+ if (err < 0) {
+ snd_timer_close(t);
+ snd_timer_instance_free(t);
+ return err;
+ }
return 0;
}
return;
if (timer->hw.flags & SNDRV_TIMER_HW_SLAVE)
return;
+ event += 10; /* convert to SNDRV_TIMER_EVENT_MXXX */
list_for_each_entry(ts, &ti->slave_active_head, active_list)
if (ts->ccallback)
- ts->ccallback(ts, event + 100, &tstamp, resolution);
+ ts->ccallback(ts, event, &tstamp, resolution);
}
/* start/continue a master timer */
* Mackie(Loud) Onyx 1640i (former model)
* Mackie(Loud) Onyx Satellite
* Mackie(Loud) Tapco Link.Firewire
- * Mackie(Loud) d.2 pro/d.4 pro
+ * Mackie(Loud) d.4 pro
* Mackie(Loud) U.420/U.420d
* TASCAM FireOne
* Stanton Controllers & Systems 1 Deck/Mixer
* PreSonus FIREBOX/FIREPOD/FP10/Inspire1394
* BridgeCo RDAudio1/Audio5
* Mackie Onyx 1220/1620/1640 (FireWire I/O Card)
- * Mackie d.2 (FireWire Option)
+ * Mackie d.2 (FireWire Option) and d.2 Pro
* Stanton FinalScratch 2 (ScratchAmp)
* Tascam IF-FW/DM
* Behringer XENIX UFX 1204/1604
#include <linux/tracepoint.h>
TRACE_EVENT(amdtp_packet,
- TP_PROTO(const struct amdtp_stream *s, u32 cycles, const __be32 *cip_header, unsigned int payload_length, unsigned int data_blocks, unsigned int data_block_counter, unsigned int index),
- TP_ARGS(s, cycles, cip_header, payload_length, data_blocks, data_block_counter, index),
+ TP_PROTO(const struct amdtp_stream *s, u32 cycles, const __be32 *cip_header, unsigned int payload_length, unsigned int data_blocks, unsigned int data_block_counter, unsigned int packet_index, unsigned int index),
+ TP_ARGS(s, cycles, cip_header, payload_length, data_blocks, data_block_counter, packet_index, index),
TP_STRUCT__entry(
__field(unsigned int, second)
__field(unsigned int, cycle)
__entry->payload_quadlets = payload_length / sizeof(__be32);
__entry->data_blocks = data_blocks;
__entry->data_block_counter = data_block_counter,
- __entry->packet_index = s->packet_index;
+ __entry->packet_index = packet_index;
__entry->irq = !!in_interrupt();
__entry->index = index;
),
}
trace_amdtp_packet(s, cycle, cip_header, payload_length, data_blocks,
- data_block_counter, index);
+ data_block_counter, s->packet_index, index);
}
static int check_cip_header(struct amdtp_stream *s, const __be32 *buf,
unsigned int *payload_length,
unsigned int *data_blocks,
unsigned int *data_block_counter,
- unsigned int *syt, unsigned int index)
+ unsigned int *syt, unsigned int packet_index, unsigned int index)
{
const __be32 *cip_header;
+ unsigned int cip_header_size;
int err;
*payload_length = be32_to_cpu(ctx_header[0]) >> ISO_DATA_LENGTH_SHIFT;
- if (*payload_length > s->ctx_data.tx.ctx_header_size +
- s->ctx_data.tx.max_ctx_payload_length) {
+
+ if (!(s->flags & CIP_NO_HEADER))
+ cip_header_size = 8;
+ else
+ cip_header_size = 0;
+
+ if (*payload_length > cip_header_size + s->ctx_data.tx.max_ctx_payload_length) {
dev_err(&s->unit->device,
"Detect jumbo payload: %04x %04x\n",
- *payload_length, s->ctx_data.tx.max_ctx_payload_length);
+ *payload_length, cip_header_size + s->ctx_data.tx.max_ctx_payload_length);
return -EIO;
}
- if (!(s->flags & CIP_NO_HEADER)) {
+ if (cip_header_size > 0) {
cip_header = ctx_header + 2;
err = check_cip_header(s, cip_header, *payload_length,
data_blocks, data_block_counter, syt);
}
trace_amdtp_packet(s, cycle, cip_header, *payload_length, *data_blocks,
- *data_block_counter, index);
+ *data_block_counter, packet_index, index);
return err;
}
unsigned int packets)
{
unsigned int dbc = s->data_block_counter;
+ unsigned int packet_index = s->packet_index;
+ unsigned int queue_size = s->queue_size;
int i;
int err;
for (i = 0; i < packets; ++i) {
struct pkt_desc *desc = descs + i;
- unsigned int index = (s->packet_index + i) % s->queue_size;
unsigned int cycle;
unsigned int payload_length;
unsigned int data_blocks;
cycle = compute_cycle_count(ctx_header[1]);
err = parse_ir_ctx_header(s, cycle, ctx_header, &payload_length,
- &data_blocks, &dbc, &syt, i);
+ &data_blocks, &dbc, &syt, packet_index, i);
if (err < 0)
return err;
desc->syt = syt;
desc->data_blocks = data_blocks;
desc->data_block_counter = dbc;
- desc->ctx_payload = s->buffer.packets[index].buffer;
+ desc->ctx_payload = s->buffer.packets[packet_index].buffer;
if (!(s->flags & CIP_DBC_IS_END_EVENT))
dbc = (dbc + desc->data_blocks) & 0xff;
ctx_header +=
s->ctx_data.tx.ctx_header_size / sizeof(*ctx_header);
+
+ packet_index = (packet_index + 1) % queue_size;
}
s->data_block_counter = dbc;
static inline void cancel_stream(struct amdtp_stream *s)
{
s->packet_index = -1;
- if (current_work() == &s->period_work)
+ if (in_interrupt())
amdtp_stream_pcm_abort(s);
WRITE_ONCE(s->pcm_buffer_pointer, SNDRV_PCM_POS_XRUN);
}
s->data_block_counter = 0;
}
- /* initialize packet buffer */
+ // initialize packet buffer.
+ max_ctx_payload_size = amdtp_stream_get_max_payload(s);
if (s->direction == AMDTP_IN_STREAM) {
dir = DMA_FROM_DEVICE;
type = FW_ISO_CONTEXT_RECEIVE;
- if (!(s->flags & CIP_NO_HEADER))
+ if (!(s->flags & CIP_NO_HEADER)) {
+ max_ctx_payload_size -= 8;
ctx_header_size = IR_CTX_HEADER_SIZE_CIP;
- else
+ } else {
ctx_header_size = IR_CTX_HEADER_SIZE_NO_CIP;
-
- max_ctx_payload_size = amdtp_stream_get_max_payload(s) -
- ctx_header_size;
+ }
} else {
dir = DMA_TO_DEVICE;
type = FW_ISO_CONTEXT_TRANSMIT;
ctx_header_size = 0; // No effect for IT context.
- max_ctx_payload_size = amdtp_stream_get_max_payload(s);
if (!(s->flags & CIP_NO_HEADER))
max_ctx_payload_size -= IT_PKT_HEADER_SIZE_CIP;
}
SND_BEBOB_DEV_ENTRY(VEN_BRIDGECO, 0x00010049, &spec_normal),
/* Mackie, Onyx 1220/1620/1640 (Firewire I/O Card) */
SND_BEBOB_DEV_ENTRY(VEN_MACKIE2, 0x00010065, &spec_normal),
- /* Mackie, d.2 (Firewire Option) */
+ // Mackie, d.2 (Firewire option card) and d.2 Pro (the card is built-in).
SND_BEBOB_DEV_ENTRY(VEN_MACKIE1, 0x00010067, &spec_normal),
/* Stanton, ScratchAmp */
SND_BEBOB_DEV_ENTRY(VEN_STANTON, 0x00000001, &spec_normal),
static const unsigned int
alesis_io26_tx_pcm_chs[MAX_STREAMS][SND_DICE_RATE_MODE_COUNT] = {
{10, 10, 4}, /* Tx0 = Analog + S/PDIF. */
- {16, 8, 0}, /* Tx1 = ADAT1 + ADAT2. */
+ {16, 4, 0}, /* Tx1 = ADAT1 + ADAT2 (available at low rate). */
};
int snd_dice_detect_alesis_formats(struct snd_dice *dice)
if (frames_per_period > 0) {
// For double_pcm_frame quirk.
- if (rate > 96000) {
+ if (rate > 96000 && !dice->disable_double_pcm_frames) {
frames_per_period *= 2;
frames_per_buffer *= 2;
}
mutex_lock(&dice->mutex);
// For double_pcm_frame quirk.
- if (rate > 96000) {
+ if (rate > 96000 && !dice->disable_double_pcm_frames) {
events_per_period /= 2;
events_per_buffer /= 2;
}
// as 'Dual Wire'.
// For this quirk, blocking mode is required and PCM buffer size should
// be aligned to SYT_INTERVAL.
- double_pcm_frames = rate > 96000;
+ double_pcm_frames = (rate > 96000 && !dice->disable_double_pcm_frames);
if (double_pcm_frames) {
rate /= 2;
pcm_chs *= 2;
};
static const struct dice_tc_spec konnekt_live = {
- .tx_pcm_chs = {{16, 16, 16}, {0, 0, 0} },
- .rx_pcm_chs = {{16, 16, 16}, {0, 0, 0} },
+ .tx_pcm_chs = {{16, 16, 6}, {0, 0, 0} },
+ .rx_pcm_chs = {{16, 16, 6}, {0, 0, 0} },
.has_midi = true,
};
#define OUI_SSL 0x0050c2 // Actually ID reserved by IEEE.
#define OUI_PRESONUS 0x000a92
#define OUI_HARMAN 0x000fd7
+#define OUI_AVID 0x00a07e
#define DICE_CATEGORY_ID 0x04
#define WEISS_CATEGORY_ID 0x00
(snd_dice_detect_formats_t)entry->driver_data;
}
+ // Below models are compliant to IEC 61883-1/6 and have no quirk at high sampling transfer
+ // frequency.
+ // * Avid M-Box 3 Pro
+ // * M-Audio Profire 610
+ // * M-Audio Profire 2626
+ if (entry->vendor_id == OUI_MAUDIO || entry->vendor_id == OUI_AVID)
+ dice->disable_double_pcm_frames = true;
+
spin_lock_init(&dice->lock);
mutex_init(&dice->mutex);
init_completion(&dice->clock_accepted);
#define DICE_INTERFACE 0x000001
+#define DICE_DEV_ENTRY_TYPICAL(vendor, model, data) \
+ { \
+ .match_flags = IEEE1394_MATCH_VENDOR_ID | \
+ IEEE1394_MATCH_MODEL_ID | \
+ IEEE1394_MATCH_SPECIFIER_ID | \
+ IEEE1394_MATCH_VERSION, \
+ .vendor_id = (vendor), \
+ .model_id = (model), \
+ .specifier_id = (vendor), \
+ .version = DICE_INTERFACE, \
+ .driver_data = (kernel_ulong_t)(data), \
+ }
+
static const struct ieee1394_device_id dice_id_table[] = {
+ // Avid M-Box 3 Pro. To match in probe function.
+ DICE_DEV_ENTRY_TYPICAL(OUI_AVID, 0x000004, snd_dice_detect_extension_formats),
/* M-Audio Profire 2626 has a different value in version field. */
{
.match_flags = IEEE1394_MATCH_VENDOR_ID |
struct fw_iso_resources rx_resources[MAX_STREAMS];
struct amdtp_stream tx_stream[MAX_STREAMS];
struct amdtp_stream rx_stream[MAX_STREAMS];
- bool global_enabled;
+ bool global_enabled:1;
+ bool disable_double_pcm_frames:1;
struct completion clock_accepted;
unsigned int substreams_counter;
* Onyx-i series (former models): 0x081216
* Mackie Onyx Satellite: 0x00200f
* Tapco LINK.firewire 4x6: 0x000460
- * d.2 pro: Unknown
* d.4 pro: Unknown
* U.420: Unknown
* U.420d: Unknown
.flags = FLAG_SOF | FLAG_SOF_ONLY_IF_DMIC_OR_SOUNDWIRE,
.device = 0x51c8,
},
+ {
+ .flags = FLAG_SOF | FLAG_SOF_ONLY_IF_DMIC_OR_SOUNDWIRE,
+ .device = 0x51cc,
+ },
#endif
};
static void snd_gus_init_control(struct snd_gus_card *gus)
{
- int ret;
-
- if (!gus->ace_flag) {
- ret =
- snd_ctl_add(gus->card,
- snd_ctl_new1(&snd_gus_joystick_control,
- gus));
- if (ret)
- snd_printk(KERN_ERR "gus: snd_ctl_add failed: %d\n",
- ret);
- }
+ if (!gus->ace_flag)
+ snd_ctl_add(gus->card, snd_ctl_new1(&snd_gus_joystick_control, gus));
}
/*
snd_pcm_set_ops(pcm, SNDRV_PCM_STREAM_PLAYBACK, &snd_sb16_playback_ops);
snd_pcm_set_ops(pcm, SNDRV_PCM_STREAM_CAPTURE, &snd_sb16_capture_ops);
- if (chip->dma16 >= 0 && chip->dma8 != chip->dma16) {
- err = snd_ctl_add(card, snd_ctl_new1(
- &snd_sb16_dma_control, chip));
- if (err)
- return err;
- } else {
+ if (chip->dma16 >= 0 && chip->dma8 != chip->dma16)
+ snd_ctl_add(card, snd_ctl_new1(&snd_sb16_dma_control, chip));
+ else
pcm->info_flags = SNDRV_PCM_INFO_HALF_DUPLEX;
- }
snd_pcm_set_managed_buffer_all(pcm, SNDRV_DMA_TYPE_DEV,
card->dev, 64*1024, 128*1024);
acard = card->private_data;
card->private_free = snd_sb8_free;
- /* block the 0x388 port to avoid PnP conflicts */
+ /*
+ * Block the 0x388 port to avoid PnP conflicts.
+ * No need to check this value after request_region,
+ * as we never do anything with it.
+ */
acard->fm_res = request_region(0x388, 4, "SoundBlaster FM");
- if (!acard->fm_res) {
- err = -EBUSY;
- goto _err;
- }
if (port[dev] != SNDRV_AUTO_PORT) {
if ((err = snd_sbdsp_create(card, port[dev], irq[dev],
#ifdef CONFIG_PM_SLEEP
static int hda_codec_pm_prepare(struct device *dev)
{
+ dev->power.power_state = PMSG_SUSPEND;
return pm_runtime_suspended(dev);
}
{
struct hda_codec *codec = dev_to_hda_codec(dev);
+ /* If no other pm-functions are called between prepare() and complete() */
+ if (dev->power.power_state.event == PM_EVENT_SUSPEND)
+ dev->power.power_state = PMSG_RESUME;
+
if (pm_runtime_suspended(dev) && (codec->jackpoll_interval ||
hda_codec_need_resume(codec) || codec->forced_resume))
pm_request_resume(dev);
static const struct snd_kcontrol_new cap_sw_temp = {
.iface = SNDRV_CTL_ELEM_IFACE_MIXER,
.name = "Capture Switch",
+ .access = SNDRV_CTL_ELEM_ACCESS_READWRITE,
.info = cap_sw_info,
.get = cap_sw_get,
.put = cap_sw_put,
/* Alderlake-P */
{ PCI_DEVICE(0x8086, 0x51c8),
.driver_data = AZX_DRIVER_SKL | AZX_DCAPS_INTEL_SKYLAKE},
+ /* Alderlake-M */
+ { PCI_DEVICE(0x8086, 0x51cc),
+ .driver_data = AZX_DRIVER_SKL | AZX_DCAPS_INTEL_SKYLAKE},
/* Elkhart Lake */
{ PCI_DEVICE(0x8086, 0x4b55),
.driver_data = AZX_DRIVER_SKL | AZX_DCAPS_INTEL_SKYLAKE},
break;
case HDA_FIXUP_ACT_PROBE:
- /* Set initial volume on Bullseye to -26 dB */
- if (codec->fixup_id == CS8409_BULLSEYE)
- snd_hda_codec_amp_init_stereo(codec, CS8409_CS42L42_DMIC_ADC_PIN_NID,
- HDA_INPUT, 0, 0xff, 0x19);
+ /* Set initial DMIC volume to -26 dB */
+ snd_hda_codec_amp_init_stereo(codec, CS8409_CS42L42_DMIC_ADC_PIN_NID,
+ HDA_INPUT, 0, 0xff, 0x19);
snd_hda_gen_add_kctl(&spec->gen,
NULL, &cs8409_cs42l42_hp_volume_mixer);
snd_hda_gen_add_kctl(&spec->gen,
case 0x10ec0282:
case 0x10ec0283:
case 0x10ec0286:
- case 0x10ec0287:
case 0x10ec0288:
case 0x10ec0285:
case 0x10ec0298:
case 0x10ec0275:
alc_update_coef_idx(codec, 0xe, 0, 1<<0);
break;
+ case 0x10ec0287:
+ alc_update_coef_idx(codec, 0x10, 1<<9, 0);
+ alc_write_coef_idx(codec, 0x8, 0x4ab7);
+ break;
case 0x10ec0293:
alc_update_coef_idx(codec, 0xa, 1<<13, 0);
break;
{}
};
+static const struct snd_hda_pin_quirk alc882_pin_fixup_tbl[] = {
+ SND_HDA_PIN_QUIRK(0x10ec1220, 0x1043, "ASUS", ALC1220_FIXUP_CLEVO_P950,
+ {0x14, 0x01014010},
+ {0x15, 0x01011012},
+ {0x16, 0x01016011},
+ {0x18, 0x01a19040},
+ {0x19, 0x02a19050},
+ {0x1a, 0x0181304f},
+ {0x1b, 0x0221401f},
+ {0x1e, 0x01456130}),
+ SND_HDA_PIN_QUIRK(0x10ec1220, 0x1462, "MS-7C35", ALC1220_FIXUP_CLEVO_P950,
+ {0x14, 0x01015010},
+ {0x15, 0x01011012},
+ {0x16, 0x01011011},
+ {0x18, 0x01a11040},
+ {0x19, 0x02a19050},
+ {0x1a, 0x0181104f},
+ {0x1b, 0x0221401f},
+ {0x1e, 0x01451130}),
+ {}
+};
+
/*
* BIOS auto configuration
*/
snd_hda_pick_fixup(codec, alc882_fixup_models, alc882_fixup_tbl,
alc882_fixups);
+ snd_hda_pick_pin_fixup(codec, alc882_pin_fixup_tbl, alc882_fixups, true);
snd_hda_apply_fixup(codec, HDA_FIXUP_ACT_PRE_PROBE);
alc_auto_parse_customize_define(codec);
}
}
+static void alc294_gu502_toggle_output(struct hda_codec *codec,
+ struct hda_jack_callback *cb)
+{
+ /* Windows sets 0x10 to 0x8420 for Node 0x20 which is
+ * responsible from changes between speakers and headphones
+ */
+ if (snd_hda_jack_detect_state(codec, 0x21) == HDA_JACK_PRESENT)
+ alc_write_coef_idx(codec, 0x10, 0x8420);
+ else
+ alc_write_coef_idx(codec, 0x10, 0x0a20);
+}
+
+static void alc294_fixup_gu502_hp(struct hda_codec *codec,
+ const struct hda_fixup *fix, int action)
+{
+ if (!is_jack_detectable(codec, 0x21))
+ return;
+
+ switch (action) {
+ case HDA_FIXUP_ACT_PRE_PROBE:
+ snd_hda_jack_detect_enable_callback(codec, 0x21,
+ alc294_gu502_toggle_output);
+ break;
+ case HDA_FIXUP_ACT_INIT:
+ alc294_gu502_toggle_output(codec, NULL);
+ break;
+ }
+}
+
static void alc285_fixup_hp_gpio_amp_init(struct hda_codec *codec,
const struct hda_fixup *fix, int action)
{
ALC294_FIXUP_ASUS_GX502_HP,
ALC294_FIXUP_ASUS_GX502_PINS,
ALC294_FIXUP_ASUS_GX502_VERBS,
+ ALC294_FIXUP_ASUS_GU502_HP,
+ ALC294_FIXUP_ASUS_GU502_PINS,
+ ALC294_FIXUP_ASUS_GU502_VERBS,
ALC285_FIXUP_HP_GPIO_LED,
ALC285_FIXUP_HP_MUTE_LED,
ALC236_FIXUP_HP_GPIO_LED,
ALC285_FIXUP_HP_LIMIT_INT_MIC_BOOST,
ALC295_FIXUP_ASUS_DACS,
ALC295_FIXUP_HP_OMEN,
+ ALC285_FIXUP_HP_SPECTRE_X360,
+ ALC287_FIXUP_IDEAPAD_BASS_SPK_AMP,
+ ALC623_FIXUP_LENOVO_THINKSTATION_P340,
+ ALC255_FIXUP_ACER_HEADPHONE_AND_MIC,
};
static const struct hda_fixup alc269_fixups[] = {
.type = HDA_FIXUP_FUNC,
.v.func = alc294_fixup_gx502_hp,
},
+ [ALC294_FIXUP_ASUS_GU502_PINS] = {
+ .type = HDA_FIXUP_PINS,
+ .v.pins = (const struct hda_pintbl[]) {
+ { 0x19, 0x01a11050 }, /* rear HP mic */
+ { 0x1a, 0x01a11830 }, /* rear external mic */
+ { 0x21, 0x012110f0 }, /* rear HP out */
+ { }
+ },
+ .chained = true,
+ .chain_id = ALC294_FIXUP_ASUS_GU502_VERBS
+ },
+ [ALC294_FIXUP_ASUS_GU502_VERBS] = {
+ .type = HDA_FIXUP_VERBS,
+ .v.verbs = (const struct hda_verb[]) {
+ /* set 0x15 to HP-OUT ctrl */
+ { 0x15, AC_VERB_SET_PIN_WIDGET_CONTROL, 0xc0 },
+ /* unmute the 0x15 amp */
+ { 0x15, AC_VERB_SET_AMP_GAIN_MUTE, 0xb000 },
+ /* set 0x1b to HP-OUT */
+ { 0x1b, AC_VERB_SET_PIN_WIDGET_CONTROL, 0x24 },
+ { }
+ },
+ .chained = true,
+ .chain_id = ALC294_FIXUP_ASUS_GU502_HP
+ },
+ [ALC294_FIXUP_ASUS_GU502_HP] = {
+ .type = HDA_FIXUP_FUNC,
+ .v.func = alc294_fixup_gu502_hp,
+ },
[ALC294_FIXUP_ASUS_COEF_1B] = {
.type = HDA_FIXUP_VERBS,
.v.verbs = (const struct hda_verb[]) {
.chained = true,
.chain_id = ALC269_FIXUP_HP_LINE1_MIC1_LED,
},
+ [ALC285_FIXUP_HP_SPECTRE_X360] = {
+ .type = HDA_FIXUP_PINS,
+ .v.pins = (const struct hda_pintbl[]) {
+ { 0x14, 0x90170110 }, /* enable top speaker */
+ {}
+ },
+ .chained = true,
+ .chain_id = ALC285_FIXUP_SPEAKER2_TO_DAC1,
+ },
+ [ALC287_FIXUP_IDEAPAD_BASS_SPK_AMP] = {
+ .type = HDA_FIXUP_FUNC,
+ .v.func = alc285_fixup_ideapad_s740_coef,
+ .chained = true,
+ .chain_id = ALC285_FIXUP_THINKPAD_HEADSET_JACK,
+ },
+ [ALC623_FIXUP_LENOVO_THINKSTATION_P340] = {
+ .type = HDA_FIXUP_FUNC,
+ .v.func = alc_fixup_no_shutup,
+ .chained = true,
+ .chain_id = ALC283_FIXUP_HEADSET_MIC,
+ },
+ [ALC255_FIXUP_ACER_HEADPHONE_AND_MIC] = {
+ .type = HDA_FIXUP_PINS,
+ .v.pins = (const struct hda_pintbl[]) {
+ { 0x21, 0x03211030 }, /* Change the Headphone location to Left */
+ { }
+ },
+ .chained = true,
+ .chain_id = ALC255_FIXUP_XIAOMI_HEADSET_MIC
+ },
};
static const struct snd_pci_quirk alc269_fixup_tbl[] = {
SND_PCI_QUIRK(0x1025, 0x132a, "Acer TravelMate B114-21", ALC233_FIXUP_ACER_HEADSET_MIC),
SND_PCI_QUIRK(0x1025, 0x1330, "Acer TravelMate X514-51T", ALC255_FIXUP_ACER_HEADSET_MIC),
SND_PCI_QUIRK(0x1025, 0x1430, "Acer TravelMate B311R-31", ALC256_FIXUP_ACER_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1025, 0x1466, "Acer Aspire A515-56", ALC255_FIXUP_ACER_HEADPHONE_AND_MIC),
SND_PCI_QUIRK(0x1028, 0x0470, "Dell M101z", ALC269_FIXUP_DELL_M101Z),
SND_PCI_QUIRK(0x1028, 0x054b, "Dell XPS one 2710", ALC275_FIXUP_DELL_XPS),
SND_PCI_QUIRK(0x1028, 0x05bd, "Dell Latitude E6440", ALC292_FIXUP_DELL_E7X),
SND_PCI_QUIRK(0x103c, 0x82bf, "HP G3 mini", ALC221_FIXUP_HP_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x103c, 0x82c0, "HP G3 mini premium", ALC221_FIXUP_HP_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x103c, 0x83b9, "HP Spectre x360", ALC269_FIXUP_HP_MUTE_LED_MIC3),
+ SND_PCI_QUIRK(0x103c, 0x841c, "HP Pavilion 15-CK0xx", ALC269_FIXUP_HP_MUTE_LED_MIC3),
SND_PCI_QUIRK(0x103c, 0x8497, "HP Envy x360", ALC269_FIXUP_HP_MUTE_LED_MIC3),
SND_PCI_QUIRK(0x103c, 0x84da, "HP OMEN dc0019-ur", ALC295_FIXUP_HP_OMEN),
SND_PCI_QUIRK(0x103c, 0x84e7, "HP Pavilion 15", ALC269_FIXUP_HP_MUTE_LED_MIC3),
+ SND_PCI_QUIRK(0x103c, 0x8519, "HP Spectre x360 15-df0xxx", ALC285_FIXUP_HP_SPECTRE_X360),
SND_PCI_QUIRK(0x103c, 0x869d, "HP", ALC236_FIXUP_HP_MUTE_LED),
SND_PCI_QUIRK(0x103c, 0x86c7, "HP Envy AiO 32", ALC274_FIXUP_HP_ENVY_GPIO),
+ SND_PCI_QUIRK(0x103c, 0x8716, "HP Elite Dragonfly G2 Notebook PC", ALC285_FIXUP_HP_GPIO_AMP_INIT),
+ SND_PCI_QUIRK(0x103c, 0x8720, "HP EliteBook x360 1040 G8 Notebook PC", ALC285_FIXUP_HP_GPIO_AMP_INIT),
SND_PCI_QUIRK(0x103c, 0x8724, "HP EliteBook 850 G7", ALC285_FIXUP_HP_GPIO_LED),
SND_PCI_QUIRK(0x103c, 0x8729, "HP", ALC285_FIXUP_HP_GPIO_LED),
SND_PCI_QUIRK(0x103c, 0x8730, "HP ProBook 445 G7", ALC236_FIXUP_HP_MUTE_LED_MICMUTE_VREF),
SND_PCI_QUIRK(0x103c, 0x87f5, "HP", ALC287_FIXUP_HP_GPIO_LED),
SND_PCI_QUIRK(0x103c, 0x87f7, "HP Spectre x360 14", ALC245_FIXUP_HP_X360_AMP),
SND_PCI_QUIRK(0x103c, 0x8846, "HP EliteBook 850 G8 Notebook PC", ALC285_FIXUP_HP_GPIO_LED),
+ SND_PCI_QUIRK(0x103c, 0x884b, "HP EliteBook 840 Aero G8 Notebook PC", ALC285_FIXUP_HP_GPIO_LED),
SND_PCI_QUIRK(0x103c, 0x884c, "HP EliteBook 840 G8 Notebook PC", ALC285_FIXUP_HP_GPIO_LED),
+ SND_PCI_QUIRK(0x103c, 0x886d, "HP ZBook Fury 17.3 Inch G8 Mobile Workstation PC", ALC285_FIXUP_HP_GPIO_AMP_INIT),
+ SND_PCI_QUIRK(0x103c, 0x8870, "HP ZBook Fury 15.6 Inch G8 Mobile Workstation PC", ALC285_FIXUP_HP_GPIO_AMP_INIT),
+ SND_PCI_QUIRK(0x103c, 0x8873, "HP ZBook Studio 15.6 Inch G8 Mobile Workstation PC", ALC285_FIXUP_HP_GPIO_AMP_INIT),
+ SND_PCI_QUIRK(0x103c, 0x888d, "HP ZBook Power 15.6 inch G8 Mobile Workstation PC", ALC236_FIXUP_HP_GPIO_LED),
+ SND_PCI_QUIRK(0x103c, 0x8896, "HP EliteBook 855 G8 Notebook PC", ALC285_FIXUP_HP_MUTE_LED),
SND_PCI_QUIRK(0x103c, 0x8898, "HP EliteBook 845 G8 Notebook PC", ALC285_FIXUP_HP_LIMIT_INT_MIC_BOOST),
SND_PCI_QUIRK(0x1043, 0x103e, "ASUS X540SA", ALC256_FIXUP_ASUS_MIC),
SND_PCI_QUIRK(0x1043, 0x103f, "ASUS TX300", ALC282_FIXUP_ASUS_TX300),
SND_PCI_QUIRK(0x1043, 0x1ccd, "ASUS X555UB", ALC256_FIXUP_ASUS_MIC),
SND_PCI_QUIRK(0x1043, 0x1d4e, "ASUS TM420", ALC256_FIXUP_ASUS_HPE),
SND_PCI_QUIRK(0x1043, 0x1e11, "ASUS Zephyrus G15", ALC289_FIXUP_ASUS_GA502),
+ SND_PCI_QUIRK(0x1043, 0x1e51, "ASUS Zephyrus M15", ALC294_FIXUP_ASUS_GU502_PINS),
SND_PCI_QUIRK(0x1043, 0x1e8e, "ASUS Zephyrus G15", ALC289_FIXUP_ASUS_GA401),
SND_PCI_QUIRK(0x1043, 0x1f11, "ASUS Zephyrus G14", ALC289_FIXUP_ASUS_GA401),
SND_PCI_QUIRK(0x1043, 0x3030, "ASUS ZN270IE", ALC256_FIXUP_ASUS_AIO_GPIO2),
SND_PCI_QUIRK(0x1558, 0x50b8, "Clevo NK50SZ", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x50d5, "Clevo NP50D5", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x50f0, "Clevo NH50A[CDF]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0x50f2, "Clevo NH50E[PR]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x50f3, "Clevo NH58DPQ", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0x50f5, "Clevo NH55EPY", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0x50f6, "Clevo NH55DPQ", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x5101, "Clevo S510WU", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x5157, "Clevo W517GU1", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x51a1, "Clevo NS50MU", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x70a1, "Clevo NB70T[HJK]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x70b3, "Clevo NK70SB", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0x70f2, "Clevo NH79EPY", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0x70f3, "Clevo NH77DPQ", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0x70f4, "Clevo NH77EPY", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0x70f6, "Clevo NH77DPQ-Y", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x8228, "Clevo NR40BU", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x8520, "Clevo NH50D[CD]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x8521, "Clevo NH77D[CD]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x8a51, "Clevo NH70RCQ-Y", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x8d50, "Clevo NH55RCQ-M", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x951d, "Clevo N950T[CDF]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0x9600, "Clevo N960K[PR]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x961d, "Clevo N960S[CDF]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x971d, "Clevo N970T[CDF]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0xa500, "Clevo NL53RU", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0xa600, "Clevo NL5XNU", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0xb018, "Clevo NP50D[BE]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0xb019, "Clevo NH77D[BE]Q", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0xb022, "Clevo NH77D[DC][QW]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0xc018, "Clevo NP50D[BE]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0xc019, "Clevo NH77D[BE]Q", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0xc022, "Clevo NH77[DC][QW]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x17aa, 0x1036, "Lenovo P520", ALC233_FIXUP_LENOVO_MULTI_CODECS),
- SND_PCI_QUIRK(0x17aa, 0x1048, "ThinkCentre Station", ALC283_FIXUP_HEADSET_MIC),
+ SND_PCI_QUIRK(0x17aa, 0x1048, "ThinkCentre Station", ALC623_FIXUP_LENOVO_THINKSTATION_P340),
SND_PCI_QUIRK(0x17aa, 0x20f2, "Thinkpad SL410/510", ALC269_FIXUP_SKU_IGNORE),
SND_PCI_QUIRK(0x17aa, 0x215e, "Thinkpad L512", ALC269_FIXUP_SKU_IGNORE),
SND_PCI_QUIRK(0x17aa, 0x21b8, "Thinkpad Edge 14", ALC269_FIXUP_SKU_IGNORE),
SND_PCI_QUIRK(0x17aa, 0x3178, "ThinkCentre Station", ALC283_FIXUP_HEADSET_MIC),
SND_PCI_QUIRK(0x17aa, 0x3818, "Lenovo C940", ALC298_FIXUP_LENOVO_SPK_VOLUME),
SND_PCI_QUIRK(0x17aa, 0x3827, "Ideapad S740", ALC285_FIXUP_IDEAPAD_S740_COEF),
+ SND_PCI_QUIRK(0x17aa, 0x3843, "Yoga 9i", ALC287_FIXUP_IDEAPAD_BASS_SPK_AMP),
SND_PCI_QUIRK(0x17aa, 0x3902, "Lenovo E50-80", ALC269_FIXUP_DMIC_THINKPAD_ACPI),
SND_PCI_QUIRK(0x17aa, 0x3977, "IdeaPad S210", ALC283_FIXUP_INT_MIC),
SND_PCI_QUIRK(0x17aa, 0x3978, "Lenovo B50-70", ALC269_FIXUP_DMIC_THINKPAD_ACPI),
{.id = ALC274_FIXUP_HP_MIC, .name = "alc274-hp-mic-detect"},
{.id = ALC245_FIXUP_HP_X360_AMP, .name = "alc245-hp-x360-amp"},
{.id = ALC295_FIXUP_HP_OMEN, .name = "alc295-hp-omen"},
+ {.id = ALC285_FIXUP_HP_SPECTRE_X360, .name = "alc285-hp-spectre-x360"},
+ {.id = ALC287_FIXUP_IDEAPAD_BASS_SPK_AMP, .name = "alc287-ideapad-bass-spk-amp"},
+ {.id = ALC623_FIXUP_LENOVO_THINKSTATION_P340, .name = "alc623-lenovo-thinkstation-p340"},
+ {.id = ALC255_FIXUP_ACER_HEADPHONE_AND_MIC, .name = "alc255-acer-headphone-and-mic"},
{}
};
#define ALC225_STANDARD_PINS \
unsigned int ali_slot; /* ALI DMA slot */
struct ac97_pcm *pcm;
int pcm_open_flag;
+ unsigned int prepared:1;
unsigned int suspended: 1;
};
int status, civ, i, step;
int ack = 0;
+ if (!ichdev->prepared || ichdev->suspended)
+ return;
+
spin_lock_irqsave(&chip->reg_lock, flags);
status = igetbyte(chip, port + ichdev->roff_sr);
civ = igetbyte(chip, port + ICH_REG_OFF_CIV);
if (ichdev->pcm_open_flag) {
snd_ac97_pcm_close(ichdev->pcm);
ichdev->pcm_open_flag = 0;
+ ichdev->prepared = 0;
}
err = snd_ac97_pcm_open(ichdev->pcm, params_rate(hw_params),
params_channels(hw_params),
if (ichdev->pcm_open_flag) {
snd_ac97_pcm_close(ichdev->pcm);
ichdev->pcm_open_flag = 0;
+ ichdev->prepared = 0;
}
return 0;
}
ichdev->pos_shift = (runtime->sample_bits > 16) ? 2 : 1;
}
snd_intel8x0_setup_periods(chip, ichdev);
+ ichdev->prepared = 1;
return 0;
}
return ret;
}
- if (!adata->play_stream && !adata->capture_stream &&
- !adata->i2ssp_play_stream && !adata->i2ssp_capture_stream)
- rv_writel(1, adata->acp3x_base + mmACP_EXTERNAL_INTR_ENB);
-
i2s_data->acp3x_base = adata->acp3x_base;
runtime->private_data = i2s_data;
return ret;
}
}
- /* Disable ACP irq, when the current stream is being closed and
- * another stream is also not active.
- */
- if (!adata->play_stream && !adata->capture_stream &&
- !adata->i2ssp_play_stream && !adata->i2ssp_capture_stream)
- rv_writel(0, adata->acp3x_base + mmACP_EXTERNAL_INTR_ENB);
return 0;
}
#define ACP_POWER_OFF_IN_PROGRESS 0x03
#define ACP3x_ITER_IRER_SAMP_LEN_MASK 0x38
+#define ACP_EXT_INTR_STAT_CLEAR_MASK 0xFFFFFFFF
struct acp3x_platform_info {
u16 play_i2s_instance;
return -ETIMEDOUT;
}
+static void acp3x_enable_interrupts(void __iomem *acp_base)
+{
+ rv_writel(0x01, acp_base + mmACP_EXTERNAL_INTR_ENB);
+}
+
+static void acp3x_disable_interrupts(void __iomem *acp_base)
+{
+ rv_writel(ACP_EXT_INTR_STAT_CLEAR_MASK, acp_base +
+ mmACP_EXTERNAL_INTR_STAT);
+ rv_writel(0x00, acp_base + mmACP_EXTERNAL_INTR_CNTL);
+ rv_writel(0x00, acp_base + mmACP_EXTERNAL_INTR_ENB);
+}
+
static int acp3x_init(struct acp3x_dev_data *adata)
{
void __iomem *acp3x_base = adata->acp3x_base;
pr_err("ACP3x reset failed\n");
return ret;
}
+ acp3x_enable_interrupts(acp3x_base);
return 0;
}
{
int ret;
+ acp3x_disable_interrupts(acp3x_base);
/* Reset */
ret = acp3x_reset(acp3x_base);
if (ret) {
};
static struct snd_soc_dai_driver ak5552_dai = {
- .name = "ak5558-aif",
+ .name = "ak5552-aif",
.capture = {
.stream_name = "Capture",
.channels_min = 1,
.readable_reg = cs35l32_readable_register,
.precious_reg = cs35l32_precious_register,
.cache_type = REGCACHE_RBTREE,
+
+ .use_single_read = true,
+ .use_single_write = true,
};
static int cs35l32_handle_of_data(struct i2c_client *i2c_client,
dev_err(&i2c_client->dev,
"CS35L33 Device ID (%X). Expected ID %X\n",
devid, CS35L33_CHIP_ID);
+ ret = -EINVAL;
goto err_enable;
}
.readable_reg = cs35l34_readable_register,
.precious_reg = cs35l34_precious_register,
.cache_type = REGCACHE_RBTREE,
+
+ .use_single_read = true,
+ .use_single_write = true,
};
static int cs35l34_handle_of_data(struct i2c_client *i2c_client,
.reg_defaults = cs42l42_reg_defaults,
.num_reg_defaults = ARRAY_SIZE(cs42l42_reg_defaults),
.cache_type = REGCACHE_RBTREE,
+
+ .use_single_read = true,
+ .use_single_write = true,
};
static DECLARE_TLV_DB_SCALE(adc_tlv, -9600, 100, false);
struct cs42l56_platform_data *pdata =
dev_get_platdata(&i2c_client->dev);
int ret, i;
- unsigned int devid = 0;
+ unsigned int devid;
unsigned int alpha_rev, metal_rev;
unsigned int reg;
}
ret = regmap_read(cs42l56->regmap, CS42L56_CHIP_ID_1, ®);
+ if (ret) {
+ dev_err(&i2c_client->dev, "Failed to read chip ID: %d\n", ret);
+ return ret;
+ }
+
devid = reg & CS42L56_CHIP_ID_MASK;
if (devid != CS42L56_DEVID) {
dev_err(&i2c_client->dev,
.volatile_reg = cs42l73_volatile_register,
.readable_reg = cs42l73_readable_register,
.cache_type = REGCACHE_RBTREE,
+
+ .use_single_read = true,
+ .use_single_write = true,
};
static int cs42l73_i2c_probe(struct i2c_client *i2c_client,
static DEVICE_ATTR(hpload_ac_l, 0444, cs43130_show_ac_l, NULL);
static DEVICE_ATTR(hpload_ac_r, 0444, cs43130_show_ac_r, NULL);
+static struct attribute *hpload_attrs[] = {
+ &dev_attr_hpload_dc_l.attr,
+ &dev_attr_hpload_dc_r.attr,
+ &dev_attr_hpload_ac_l.attr,
+ &dev_attr_hpload_ac_r.attr,
+};
+ATTRIBUTE_GROUPS(hpload);
+
static struct reg_sequence hp_en_cal_seq[] = {
{CS43130_INT_MASK_4, CS43130_INT_MASK_ALL},
{CS43130_HP_MEAS_LOAD_1, 0},
cs43130->hpload_done = false;
if (cs43130->dc_meas) {
- ret = device_create_file(component->dev, &dev_attr_hpload_dc_l);
- if (ret < 0)
- return ret;
-
- ret = device_create_file(component->dev, &dev_attr_hpload_dc_r);
- if (ret < 0)
- return ret;
-
- ret = device_create_file(component->dev, &dev_attr_hpload_ac_l);
- if (ret < 0)
- return ret;
-
- ret = device_create_file(component->dev, &dev_attr_hpload_ac_r);
- if (ret < 0)
+ ret = sysfs_create_groups(&component->dev->kobj, hpload_groups);
+ if (ret)
return ret;
cs43130->wq = create_singlethread_workqueue("cs43130_hp");
- if (!cs43130->wq)
+ if (!cs43130->wq) {
+ sysfs_remove_groups(&component->dev->kobj, hpload_groups);
return -ENOMEM;
+ }
INIT_WORK(&cs43130->work, cs43130_imp_meas);
}
.writeable_reg = cs53l30_writeable_register,
.readable_reg = cs53l30_readable_register,
.cache_type = REGCACHE_RBTREE,
+
+ .use_single_read = true,
+ .use_single_write = true,
};
static int cs53l30_i2c_probe(struct i2c_client *client,
ret);
goto err;
}
-
- da7219->dai_clks[i] = devm_clk_hw_get_clk(dev, dai_clk_hw, NULL);
- if (IS_ERR(da7219->dai_clks[i]))
- return PTR_ERR(da7219->dai_clks[i]);
+ da7219->dai_clks[i] = dai_clk_hw->clk;
/* For DT setup onecell data, otherwise create lookup */
if (np) {
{ .compatible = "qcom,sm8250-lpass-rx-macro" },
{ }
};
+MODULE_DEVICE_TABLE(of, rx_macro_dt_match);
static struct platform_driver rx_macro_driver = {
.driver = {
{ .compatible = "qcom,sm8250-lpass-tx-macro" },
{ }
};
+MODULE_DEVICE_TABLE(of, tx_macro_dt_match);
static struct platform_driver tx_macro_driver = {
.driver = {
.name = "tx_macro",
enum max98088_type devtype;
struct max98088_pdata *pdata;
struct clk *mclk;
+ unsigned char mclk_prescaler;
unsigned int sysclk;
struct max98088_cdata dai[2];
int eq_textcnt;
/* Configure NI when operating as master */
if (snd_soc_component_read(component, M98088_REG_14_DAI1_FORMAT)
& M98088_DAI_MAS) {
+ unsigned long pclk;
+
if (max98088->sysclk == 0) {
dev_err(component->dev, "Invalid system clock frequency\n");
return -EINVAL;
}
ni = 65536ULL * (rate < 50000 ? 96ULL : 48ULL)
* (unsigned long long int)rate;
- do_div(ni, (unsigned long long int)max98088->sysclk);
+ pclk = DIV_ROUND_CLOSEST(max98088->sysclk, max98088->mclk_prescaler);
+ ni = DIV_ROUND_CLOSEST_ULL(ni, pclk);
snd_soc_component_write(component, M98088_REG_12_DAI1_CLKCFG_HI,
(ni >> 8) & 0x7F);
snd_soc_component_write(component, M98088_REG_13_DAI1_CLKCFG_LO,
/* Configure NI when operating as master */
if (snd_soc_component_read(component, M98088_REG_1C_DAI2_FORMAT)
& M98088_DAI_MAS) {
+ unsigned long pclk;
+
if (max98088->sysclk == 0) {
dev_err(component->dev, "Invalid system clock frequency\n");
return -EINVAL;
}
ni = 65536ULL * (rate < 50000 ? 96ULL : 48ULL)
* (unsigned long long int)rate;
- do_div(ni, (unsigned long long int)max98088->sysclk);
+ pclk = DIV_ROUND_CLOSEST(max98088->sysclk, max98088->mclk_prescaler);
+ ni = DIV_ROUND_CLOSEST_ULL(ni, pclk);
snd_soc_component_write(component, M98088_REG_1A_DAI2_CLKCFG_HI,
(ni >> 8) & 0x7F);
snd_soc_component_write(component, M98088_REG_1B_DAI2_CLKCFG_LO,
*/
if ((freq >= 10000000) && (freq < 20000000)) {
snd_soc_component_write(component, M98088_REG_10_SYS_CLK, 0x10);
+ max98088->mclk_prescaler = 1;
} else if ((freq >= 20000000) && (freq < 30000000)) {
snd_soc_component_write(component, M98088_REG_10_SYS_CLK, 0x20);
+ max98088->mclk_prescaler = 2;
} else {
dev_err(component->dev, "Invalid master clock frequency\n");
return -EINVAL;
{
struct snd_soc_dapm_context *dapm = snd_soc_component_get_dapm(component);
struct rt5645_priv *rt5645 = snd_soc_component_get_drvdata(component);
+ int ret = 0;
rt5645->component = component;
switch (rt5645->codec_type) {
case CODEC_TYPE_RT5645:
- snd_soc_dapm_new_controls(dapm,
+ ret = snd_soc_dapm_new_controls(dapm,
rt5645_specific_dapm_widgets,
ARRAY_SIZE(rt5645_specific_dapm_widgets));
- snd_soc_dapm_add_routes(dapm,
+ if (ret < 0)
+ goto exit;
+
+ ret = snd_soc_dapm_add_routes(dapm,
rt5645_specific_dapm_routes,
ARRAY_SIZE(rt5645_specific_dapm_routes));
+ if (ret < 0)
+ goto exit;
+
if (rt5645->v_id < 3) {
- snd_soc_dapm_add_routes(dapm,
+ ret = snd_soc_dapm_add_routes(dapm,
rt5645_old_dapm_routes,
ARRAY_SIZE(rt5645_old_dapm_routes));
+ if (ret < 0)
+ goto exit;
}
break;
case CODEC_TYPE_RT5650:
- snd_soc_dapm_new_controls(dapm,
+ ret = snd_soc_dapm_new_controls(dapm,
rt5650_specific_dapm_widgets,
ARRAY_SIZE(rt5650_specific_dapm_widgets));
- snd_soc_dapm_add_routes(dapm,
+ if (ret < 0)
+ goto exit;
+
+ ret = snd_soc_dapm_add_routes(dapm,
rt5650_specific_dapm_routes,
ARRAY_SIZE(rt5650_specific_dapm_routes));
+ if (ret < 0)
+ goto exit;
break;
}
/* for JD function */
if (rt5645->pdata.jd_mode) {
- snd_soc_dapm_force_enable_pin(dapm, "JD Power");
- snd_soc_dapm_force_enable_pin(dapm, "LDO2");
- snd_soc_dapm_sync(dapm);
+ ret = snd_soc_dapm_force_enable_pin(dapm, "JD Power");
+ if (ret < 0)
+ goto exit;
+
+ ret = snd_soc_dapm_force_enable_pin(dapm, "LDO2");
+ if (ret < 0)
+ goto exit;
+
+ ret = snd_soc_dapm_sync(dapm);
+ if (ret < 0)
+ goto exit;
}
if (rt5645->pdata.long_name)
GFP_KERNEL);
if (!rt5645->eq_param)
- return -ENOMEM;
-
- return 0;
+ ret = -ENOMEM;
+exit:
+ /*
+ * If there was an error above, everything will be cleaned up by the
+ * caller if we return an error here. This will be done with a later
+ * call to rt5645_remove().
+ */
+ return ret;
}
static void rt5645_remove(struct snd_soc_component *component)
return 0;
}
-static const struct snd_soc_dapm_widget rt5659_dapm_widgets[] = {
+static const struct snd_soc_dapm_widget rt5659_particular_dapm_widgets[] = {
SND_SOC_DAPM_SUPPLY("LDO2", RT5659_PWR_ANLG_3, RT5659_PWR_LDO2_BIT, 0,
NULL, 0),
- SND_SOC_DAPM_SUPPLY("PLL", RT5659_PWR_ANLG_3, RT5659_PWR_PLL_BIT, 0,
- NULL, 0),
+ SND_SOC_DAPM_SUPPLY("MICBIAS1", RT5659_PWR_ANLG_2, RT5659_PWR_MB1_BIT,
+ 0, NULL, 0),
SND_SOC_DAPM_SUPPLY("Mic Det Power", RT5659_PWR_VOL,
RT5659_PWR_MIC_DET_BIT, 0, NULL, 0),
+};
+
+static const struct snd_soc_dapm_widget rt5659_dapm_widgets[] = {
+ SND_SOC_DAPM_SUPPLY("PLL", RT5659_PWR_ANLG_3, RT5659_PWR_PLL_BIT, 0,
+ NULL, 0),
SND_SOC_DAPM_SUPPLY("Mono Vref", RT5659_PWR_ANLG_1,
RT5659_PWR_VREF3_BIT, 0, NULL, 0),
RT5659_ADC_MONO_R_ASRC_SFT, 0, NULL, 0),
/* Input Side */
- SND_SOC_DAPM_SUPPLY("MICBIAS1", RT5659_PWR_ANLG_2, RT5659_PWR_MB1_BIT,
- 0, NULL, 0),
SND_SOC_DAPM_SUPPLY("MICBIAS2", RT5659_PWR_ANLG_2, RT5659_PWR_MB2_BIT,
0, NULL, 0),
SND_SOC_DAPM_SUPPLY("MICBIAS3", RT5659_PWR_ANLG_2, RT5659_PWR_MB3_BIT,
static int rt5659_probe(struct snd_soc_component *component)
{
+ struct snd_soc_dapm_context *dapm =
+ snd_soc_component_get_dapm(component);
struct rt5659_priv *rt5659 = snd_soc_component_get_drvdata(component);
rt5659->component = component;
+ switch (rt5659->pdata.jd_src) {
+ case RT5659_JD_HDA_HEADER:
+ break;
+
+ default:
+ snd_soc_dapm_new_controls(dapm,
+ rt5659_particular_dapm_widgets,
+ ARRAY_SIZE(rt5659_particular_dapm_widgets));
+ break;
+ }
+
return 0;
}
regmap_update_bits(rt5682->regmap, RT5682_CBJ_CTRL_2,
RT5682_EXT_JD_SRC, RT5682_EXT_JD_SRC_MANUAL);
- regmap_write(rt5682->regmap, RT5682_CBJ_CTRL_1, 0xd042);
+ regmap_write(rt5682->regmap, RT5682_CBJ_CTRL_1, 0xd142);
+ regmap_update_bits(rt5682->regmap, RT5682_CBJ_CTRL_5, 0x0700, 0x0600);
regmap_update_bits(rt5682->regmap, RT5682_CBJ_CTRL_3,
RT5682_CBJ_IN_BUF_EN, RT5682_CBJ_IN_BUF_EN);
regmap_update_bits(rt5682->regmap, RT5682_SAR_IL_CMD_1,
ch_r = (rt711->fu1e_dapm_mute || rt711->fu1e_mixer_r_mute) ? 0x01 : 0x00;
err = regmap_write(rt711->regmap,
- SDW_SDCA_CTL(FUNC_NUM_JACK_CODEC, RT711_SDCA_ENT_USER_FU1E,
+ SDW_SDCA_CTL(FUNC_NUM_MIC_ARRAY, RT711_SDCA_ENT_USER_FU1E,
RT711_SDCA_CTL_FU_MUTE, CH_L), ch_l);
if (err < 0)
return err;
err = regmap_write(rt711->regmap,
- SDW_SDCA_CTL(FUNC_NUM_JACK_CODEC, RT711_SDCA_ENT_USER_FU1E,
+ SDW_SDCA_CTL(FUNC_NUM_MIC_ARRAY, RT711_SDCA_ENT_USER_FU1E,
RT711_SDCA_CTL_FU_MUTE, CH_R), ch_r);
if (err < 0)
return err;
},
{},
};
+MODULE_DEVICE_TABLE(of, sti_sas_dev_match);
static int sti_sas_driver_probe(struct platform_device *pdev)
{
#define TAS2562_TDM_CFG0_RAMPRATE_MASK BIT(5)
#define TAS2562_TDM_CFG0_RAMPRATE_44_1 BIT(5)
#define TAS2562_TDM_CFG0_SAMPRATE_MASK GENMASK(3, 1)
-#define TAS2562_TDM_CFG0_SAMPRATE_7305_8KHZ 0x0
-#define TAS2562_TDM_CFG0_SAMPRATE_14_7_16KHZ 0x1
-#define TAS2562_TDM_CFG0_SAMPRATE_22_05_24KHZ 0x2
-#define TAS2562_TDM_CFG0_SAMPRATE_29_4_32KHZ 0x3
-#define TAS2562_TDM_CFG0_SAMPRATE_44_1_48KHZ 0x4
-#define TAS2562_TDM_CFG0_SAMPRATE_88_2_96KHZ 0x5
-#define TAS2562_TDM_CFG0_SAMPRATE_176_4_192KHZ 0x6
+#define TAS2562_TDM_CFG0_SAMPRATE_7305_8KHZ (0x0 << 1)
+#define TAS2562_TDM_CFG0_SAMPRATE_14_7_16KHZ (0x1 << 1)
+#define TAS2562_TDM_CFG0_SAMPRATE_22_05_24KHZ (0x2 << 1)
+#define TAS2562_TDM_CFG0_SAMPRATE_29_4_32KHZ (0x3 << 1)
+#define TAS2562_TDM_CFG0_SAMPRATE_44_1_48KHZ (0x4 << 1)
+#define TAS2562_TDM_CFG0_SAMPRATE_88_2_96KHZ (0x5 << 1)
+#define TAS2562_TDM_CFG0_SAMPRATE_176_4_192KHZ (0x6 << 1)
#define TAS2562_TDM_CFG2_RIGHT_JUSTIFY BIT(6)
tristate "NXP Audio Base On RPMSG support"
depends on COMMON_CLK
depends on RPMSG
+ depends on SND_IMX_SOC || SND_IMX_SOC = n
select SND_SOC_IMX_RPMSG if SND_IMX_SOC != n
help
Say Y if you want to add rpmsg audio support for the Freescale CPUs.
/* Initialize sound card */
priv->pdev = pdev;
priv->card.dev = &pdev->dev;
+ priv->card.owner = THIS_MODULE;
ret = snd_soc_of_parse_card_name(&priv->card, "model");
if (ret) {
snprintf(priv->name, sizeof(priv->name), "%s-audio",
static int graph_parse_node(struct asoc_simple_priv *priv,
struct device_node *ep,
struct link_info *li,
- int is_cpu)
+ int *cpu)
{
struct device *dev = simple_priv_to_dev(priv);
struct device_node *top = dev->of_node;
struct simple_dai_props *dai_props = simple_priv_to_props(priv, li->link);
struct snd_soc_dai_link_component *dlc;
struct asoc_simple_dai *dai;
- int ret, single = 0;
+ int ret;
- if (is_cpu) {
+ if (cpu) {
dlc = asoc_link_to_cpu(dai_link, 0);
dai = simple_props_to_dai_cpu(dai_props, 0);
} else {
graph_parse_mclk_fs(top, ep, dai_props);
- ret = asoc_simple_parse_dai(ep, dlc, &single);
+ ret = asoc_simple_parse_dai(ep, dlc, cpu);
if (ret < 0)
return ret;
if (ret < 0)
return ret;
- if (is_cpu)
- asoc_simple_canonicalize_cpu(dlc, single);
-
return 0;
}
struct link_info *li)
{
struct device *dev = simple_priv_to_dev(priv);
- struct snd_soc_card *card = simple_priv_to_card(priv);
struct snd_soc_dai_link *dai_link = simple_priv_to_link(priv, li->link);
struct simple_dai_props *dai_props = simple_priv_to_props(priv, li->link);
struct device_node *top = dev->of_node;
struct device_node *ep = li->cpu ? cpu_ep : codec_ep;
- struct device_node *port;
- struct device_node *ports;
- struct snd_soc_dai_link_component *cpus = asoc_link_to_cpu(dai_link, 0);
- struct snd_soc_dai_link_component *codecs = asoc_link_to_codec(dai_link, 0);
char dai_name[64];
int ret;
- port = of_get_parent(ep);
- ports = of_get_parent(port);
-
dev_dbg(dev, "link_of DPCM (%pOF)\n", ep);
if (li->cpu) {
+ struct snd_soc_card *card = simple_priv_to_card(priv);
+ struct snd_soc_dai_link_component *cpus = asoc_link_to_cpu(dai_link, 0);
+ int is_single_links = 0;
+
/* Codec is dummy */
/* FE settings */
dai_link->dynamic = 1;
dai_link->dpcm_merged_format = 1;
- ret = graph_parse_node(priv, cpu_ep, li, 1);
+ ret = graph_parse_node(priv, cpu_ep, li, &is_single_links);
if (ret)
- goto out_put_node;
+ return ret;
snprintf(dai_name, sizeof(dai_name),
"fe.%pOFP.%s", cpus->of_node, cpus->dai_name);
*/
if (card->component_chaining && !soc_component_is_pcm(cpus))
dai_link->no_pcm = 1;
+
+ asoc_simple_canonicalize_cpu(cpus, is_single_links);
} else {
- struct snd_soc_codec_conf *cconf;
+ struct snd_soc_codec_conf *cconf = simple_props_to_codec_conf(dai_props, 0);
+ struct snd_soc_dai_link_component *codecs = asoc_link_to_codec(dai_link, 0);
+ struct device_node *port;
+ struct device_node *ports;
/* CPU is dummy */
dai_link->no_pcm = 1;
dai_link->be_hw_params_fixup = asoc_simple_be_hw_params_fixup;
- cconf = simple_props_to_codec_conf(dai_props, 0);
-
- ret = graph_parse_node(priv, codec_ep, li, 0);
+ ret = graph_parse_node(priv, codec_ep, li, NULL);
if (ret < 0)
- goto out_put_node;
+ return ret;
snprintf(dai_name, sizeof(dai_name),
"be.%pOFP.%s", codecs->of_node, codecs->dai_name);
/* check "prefix" from top node */
+ port = of_get_parent(ep);
+ ports = of_get_parent(port);
snd_soc_of_parse_node_prefix(top, cconf, codecs->of_node,
"prefix");
if (of_node_name_eq(ports, "ports"))
snd_soc_of_parse_node_prefix(ports, cconf, codecs->of_node, "prefix");
snd_soc_of_parse_node_prefix(port, cconf, codecs->of_node,
"prefix");
+
+ of_node_put(ports);
+ of_node_put(port);
}
graph_parse_convert(dev, ep, &dai_props->adata);
ret = graph_link_init(priv, cpu_ep, codec_ep, li, dai_name);
-out_put_node:
li->link++;
- of_node_put(ports);
- of_node_put(port);
return ret;
}
struct snd_soc_dai_link_component *cpus = asoc_link_to_cpu(dai_link, 0);
struct snd_soc_dai_link_component *codecs = asoc_link_to_codec(dai_link, 0);
char dai_name[64];
- int ret;
+ int ret, is_single_links = 0;
dev_dbg(dev, "link_of (%pOF)\n", cpu_ep);
- ret = graph_parse_node(priv, cpu_ep, li, 1);
+ ret = graph_parse_node(priv, cpu_ep, li, &is_single_links);
if (ret < 0)
return ret;
- ret = graph_parse_node(priv, codec_ep, li, 0);
+ ret = graph_parse_node(priv, codec_ep, li, NULL);
if (ret < 0)
return ret;
snprintf(dai_name, sizeof(dai_name),
"%s-%s", cpus->dai_name, codecs->dai_name);
+
+ asoc_simple_canonicalize_cpu(cpus, is_single_links);
+
ret = graph_link_init(priv, cpu_ep, codec_ep, li, dai_name);
if (ret < 0)
return ret;
}
static void simple_parse_mclk_fs(struct device_node *top,
- struct device_node *cpu,
- struct device_node *codec,
+ struct device_node *np,
struct simple_dai_props *props,
char *prefix)
{
- struct device_node *node = of_get_parent(cpu);
+ struct device_node *node = of_get_parent(np);
char prop[128];
snprintf(prop, sizeof(prop), "%smclk-fs", PREFIX);
snprintf(prop, sizeof(prop), "%smclk-fs", prefix);
of_property_read_u32(node, prop, &props->mclk_fs);
- of_property_read_u32(cpu, prop, &props->mclk_fs);
- of_property_read_u32(codec, prop, &props->mclk_fs);
+ of_property_read_u32(np, prop, &props->mclk_fs);
of_node_put(node);
}
+static int simple_parse_node(struct asoc_simple_priv *priv,
+ struct device_node *np,
+ struct link_info *li,
+ char *prefix,
+ int *cpu)
+{
+ struct device *dev = simple_priv_to_dev(priv);
+ struct device_node *top = dev->of_node;
+ struct snd_soc_dai_link *dai_link = simple_priv_to_link(priv, li->link);
+ struct simple_dai_props *dai_props = simple_priv_to_props(priv, li->link);
+ struct snd_soc_dai_link_component *dlc;
+ struct asoc_simple_dai *dai;
+ int ret;
+
+ if (cpu) {
+ dlc = asoc_link_to_cpu(dai_link, 0);
+ dai = simple_props_to_dai_cpu(dai_props, 0);
+ } else {
+ dlc = asoc_link_to_codec(dai_link, 0);
+ dai = simple_props_to_dai_codec(dai_props, 0);
+ }
+
+ simple_parse_mclk_fs(top, np, dai_props, prefix);
+
+ ret = asoc_simple_parse_dai(np, dlc, cpu);
+ if (ret)
+ return ret;
+
+ ret = asoc_simple_parse_clk(dev, np, dai, dlc);
+ if (ret)
+ return ret;
+
+ ret = asoc_simple_parse_tdm(np, dai);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static int simple_link_init(struct asoc_simple_priv *priv,
+ struct device_node *node,
+ struct device_node *codec,
+ struct link_info *li,
+ char *prefix, char *name)
+{
+ struct device *dev = simple_priv_to_dev(priv);
+ struct snd_soc_dai_link *dai_link = simple_priv_to_link(priv, li->link);
+ int ret;
+
+ ret = asoc_simple_parse_daifmt(dev, node, codec,
+ prefix, &dai_link->dai_fmt);
+ if (ret < 0)
+ return 0;
+
+ dai_link->init = asoc_simple_dai_init;
+ dai_link->ops = &simple_ops;
+
+ return asoc_simple_set_dailink_name(dev, dai_link, name);
+}
+
static int simple_dai_link_of_dpcm(struct asoc_simple_priv *priv,
struct device_node *np,
struct device_node *codec,
struct device *dev = simple_priv_to_dev(priv);
struct snd_soc_dai_link *dai_link = simple_priv_to_link(priv, li->link);
struct simple_dai_props *dai_props = simple_priv_to_props(priv, li->link);
- struct asoc_simple_dai *dai;
- struct snd_soc_dai_link_component *cpus = asoc_link_to_cpu(dai_link, 0);
- struct snd_soc_dai_link_component *codecs = asoc_link_to_codec(dai_link, 0);
- struct snd_soc_dai_link_component *platforms = asoc_link_to_platform(dai_link, 0);
struct device_node *top = dev->of_node;
struct device_node *node = of_get_parent(np);
char *prefix = "";
+ char dai_name[64];
int ret;
dev_dbg(dev, "link_of DPCM (%pOF)\n", np);
- li->link++;
-
/* For single DAI link & old style of DT node */
if (is_top)
prefix = PREFIX;
if (li->cpu) {
+ struct snd_soc_dai_link_component *cpus = asoc_link_to_cpu(dai_link, 0);
+ struct snd_soc_dai_link_component *platforms = asoc_link_to_platform(dai_link, 0);
int is_single_links = 0;
/* Codec is dummy */
dai_link->dynamic = 1;
dai_link->dpcm_merged_format = 1;
- dai = simple_props_to_dai_cpu(dai_props, 0);
-
- ret = asoc_simple_parse_dai(np, cpus, &is_single_links);
- if (ret)
- goto out_put_node;
-
- ret = asoc_simple_parse_clk(dev, np, dai, cpus);
+ ret = simple_parse_node(priv, np, li, prefix, &is_single_links);
if (ret < 0)
goto out_put_node;
- ret = asoc_simple_set_dailink_name(dev, dai_link,
- "fe.%s",
- cpus->dai_name);
- if (ret < 0)
- goto out_put_node;
+ snprintf(dai_name, sizeof(dai_name), "fe.%s", cpus->dai_name);
asoc_simple_canonicalize_cpu(cpus, is_single_links);
asoc_simple_canonicalize_platform(platforms, cpus);
} else {
+ struct snd_soc_dai_link_component *codecs = asoc_link_to_codec(dai_link, 0);
struct snd_soc_codec_conf *cconf;
/* CPU is dummy */
dai_link->no_pcm = 1;
dai_link->be_hw_params_fixup = asoc_simple_be_hw_params_fixup;
- dai = simple_props_to_dai_codec(dai_props, 0);
cconf = simple_props_to_codec_conf(dai_props, 0);
- ret = asoc_simple_parse_dai(np, codecs, NULL);
+ ret = simple_parse_node(priv, np, li, prefix, NULL);
if (ret < 0)
goto out_put_node;
- ret = asoc_simple_parse_clk(dev, np, dai, codecs);
- if (ret < 0)
- goto out_put_node;
-
- ret = asoc_simple_set_dailink_name(dev, dai_link,
- "be.%s",
- codecs->dai_name);
- if (ret < 0)
- goto out_put_node;
+ snprintf(dai_name, sizeof(dai_name), "be.%s", codecs->dai_name);
/* check "prefix" from top node */
snd_soc_of_parse_node_prefix(top, cconf, codecs->of_node,
}
simple_parse_convert(dev, np, &dai_props->adata);
- simple_parse_mclk_fs(top, np, codec, dai_props, prefix);
-
- ret = asoc_simple_parse_tdm(np, dai);
- if (ret)
- goto out_put_node;
-
- ret = asoc_simple_parse_daifmt(dev, node, codec,
- prefix, &dai_link->dai_fmt);
- if (ret < 0)
- goto out_put_node;
snd_soc_dai_link_set_capabilities(dai_link);
- dai_link->ops = &simple_ops;
- dai_link->init = asoc_simple_dai_init;
+ ret = simple_link_init(priv, node, codec, li, prefix, dai_name);
out_put_node:
+ li->link++;
+
of_node_put(node);
return ret;
}
{
struct device *dev = simple_priv_to_dev(priv);
struct snd_soc_dai_link *dai_link = simple_priv_to_link(priv, li->link);
- struct simple_dai_props *dai_props = simple_priv_to_props(priv, li->link);
- struct asoc_simple_dai *cpu_dai = simple_props_to_dai_cpu(dai_props, 0);
- struct asoc_simple_dai *codec_dai = simple_props_to_dai_codec(dai_props, 0);
struct snd_soc_dai_link_component *cpus = asoc_link_to_cpu(dai_link, 0);
struct snd_soc_dai_link_component *codecs = asoc_link_to_codec(dai_link, 0);
struct snd_soc_dai_link_component *platforms = asoc_link_to_platform(dai_link, 0);
- struct device_node *top = dev->of_node;
struct device_node *cpu = NULL;
struct device_node *node = NULL;
struct device_node *plat = NULL;
+ char dai_name[64];
char prop[128];
char *prefix = "";
int ret, single_cpu = 0;
cpu = np;
node = of_get_parent(np);
- li->link++;
dev_dbg(dev, "link_of (%pOF)\n", node);
snprintf(prop, sizeof(prop), "%splat", prefix);
plat = of_get_child_by_name(node, prop);
- ret = asoc_simple_parse_daifmt(dev, node, codec,
- prefix, &dai_link->dai_fmt);
- if (ret < 0)
- goto dai_link_of_err;
-
- simple_parse_mclk_fs(top, cpu, codec, dai_props, prefix);
-
- ret = asoc_simple_parse_dai(cpu, cpus, &single_cpu);
+ ret = simple_parse_node(priv, cpu, li, prefix, &single_cpu);
if (ret < 0)
goto dai_link_of_err;
- ret = asoc_simple_parse_dai(codec, codecs, NULL);
+ ret = simple_parse_node(priv, codec, li, prefix, NULL);
if (ret < 0)
goto dai_link_of_err;
if (ret < 0)
goto dai_link_of_err;
- ret = asoc_simple_parse_tdm(cpu, cpu_dai);
- if (ret < 0)
- goto dai_link_of_err;
-
- ret = asoc_simple_parse_tdm(codec, codec_dai);
- if (ret < 0)
- goto dai_link_of_err;
-
- ret = asoc_simple_parse_clk(dev, cpu, cpu_dai, cpus);
- if (ret < 0)
- goto dai_link_of_err;
-
- ret = asoc_simple_parse_clk(dev, codec, codec_dai, codecs);
- if (ret < 0)
- goto dai_link_of_err;
-
- ret = asoc_simple_set_dailink_name(dev, dai_link,
- "%s-%s",
- cpus->dai_name,
- codecs->dai_name);
- if (ret < 0)
- goto dai_link_of_err;
-
- dai_link->ops = &simple_ops;
- dai_link->init = asoc_simple_dai_init;
+ snprintf(dai_name, sizeof(dai_name),
+ "%s-%s", cpus->dai_name, codecs->dai_name);
asoc_simple_canonicalize_cpu(cpus, single_cpu);
asoc_simple_canonicalize_platform(platforms, cpus);
+ ret = simple_link_init(priv, node, codec, li, prefix, dai_name);
+
dai_link_of_err:
of_node_put(plat);
of_node_put(node);
+ li->link++;
+
return ret;
}
BYT_RT5640_SSP0_AIF1 |
BYT_RT5640_MCLK_EN),
},
+ { /* Glavey TM800A550L */
+ .matches = {
+ DMI_MATCH(DMI_BOARD_VENDOR, "AMI Corporation"),
+ DMI_MATCH(DMI_BOARD_NAME, "Aptio CRB"),
+ /* Above strings are too generic, also match on BIOS version */
+ DMI_MATCH(DMI_BIOS_VERSION, "ZY-8-BI-PX4S70VTR400-X423B-005-D"),
+ },
+ .driver_data = (void *)(BYTCR_INPUT_DEFAULTS |
+ BYT_RT5640_SSP0_AIF1 |
+ BYT_RT5640_MCLK_EN),
+ },
{
.matches = {
DMI_EXACT_MATCH(DMI_SYS_VENDOR, "Hewlett-Packard"),
BYT_RT5640_MONO_SPEAKER |
BYT_RT5640_MCLK_EN),
},
+ { /* Lenovo Miix 3-830 */
+ .matches = {
+ DMI_EXACT_MATCH(DMI_SYS_VENDOR, "LENOVO"),
+ DMI_EXACT_MATCH(DMI_PRODUCT_VERSION, "Lenovo MIIX 3-830"),
+ },
+ .driver_data = (void *)(BYT_RT5640_IN1_MAP |
+ BYT_RT5640_JD_SRC_JD2_IN4N |
+ BYT_RT5640_OVCD_TH_2000UA |
+ BYT_RT5640_OVCD_SF_0P75 |
+ BYT_RT5640_MONO_SPEAKER |
+ BYT_RT5640_DIFF_MIC |
+ BYT_RT5640_SSP0_AIF1 |
+ BYT_RT5640_MCLK_EN),
+ },
{ /* Linx Linx7 tablet */
.matches = {
DMI_EXACT_MATCH(DMI_SYS_VENDOR, "LINX"),
struct snd_soc_dai *dai)
{
struct lpass_data *drvdata = snd_soc_dai_get_drvdata(dai);
+ struct lpaif_i2sctl *i2sctl = drvdata->i2sctl;
+ unsigned int id = dai->driver->id;
clk_disable_unprepare(drvdata->mi2s_osr_clk[dai->driver->id]);
+ /*
+ * Ensure LRCLK is disabled even in device node validation.
+ * Will not impact if disabled in lpass_cpu_daiops_trigger()
+ * suspend.
+ */
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
+ regmap_fields_write(i2sctl->spken, id, LPAIF_I2SCTL_SPKEN_DISABLE);
+ else
+ regmap_fields_write(i2sctl->micen, id, LPAIF_I2SCTL_MICEN_DISABLE);
+
+ /*
+ * BCLK may not be enabled if lpass_cpu_daiops_prepare is called before
+ * lpass_cpu_daiops_shutdown. It's paired with the clk_enable in
+ * lpass_cpu_daiops_prepare.
+ */
+ if (drvdata->mi2s_was_prepared[dai->driver->id]) {
+ drvdata->mi2s_was_prepared[dai->driver->id] = false;
+ clk_disable(drvdata->mi2s_bit_clk[dai->driver->id]);
+ }
+
clk_unprepare(drvdata->mi2s_bit_clk[dai->driver->id]);
}
case SNDRV_PCM_TRIGGER_START:
case SNDRV_PCM_TRIGGER_RESUME:
case SNDRV_PCM_TRIGGER_PAUSE_RELEASE:
+ /*
+ * Ensure lpass BCLK/LRCLK is enabled during
+ * device resume as lpass_cpu_daiops_prepare() is not called
+ * after the device resumes. We don't check mi2s_was_prepared before
+ * enable/disable BCLK in trigger events because:
+ * 1. These trigger events are paired, so the BCLK
+ * enable_count is balanced.
+ * 2. the BCLK can be shared (ex: headset and headset mic),
+ * we need to increase the enable_count so that we don't
+ * turn off the shared BCLK while other devices are using
+ * it.
+ */
if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) {
ret = regmap_fields_write(i2sctl->spken, id,
LPAIF_I2SCTL_SPKEN_ENABLE);
case SNDRV_PCM_TRIGGER_STOP:
case SNDRV_PCM_TRIGGER_SUSPEND:
case SNDRV_PCM_TRIGGER_PAUSE_PUSH:
+ /*
+ * To ensure lpass BCLK/LRCLK is disabled during
+ * device suspend.
+ */
if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) {
ret = regmap_fields_write(i2sctl->spken, id,
LPAIF_I2SCTL_SPKEN_DISABLE);
return ret;
}
+static int lpass_cpu_daiops_prepare(struct snd_pcm_substream *substream,
+ struct snd_soc_dai *dai)
+{
+ struct lpass_data *drvdata = snd_soc_dai_get_drvdata(dai);
+ struct lpaif_i2sctl *i2sctl = drvdata->i2sctl;
+ unsigned int id = dai->driver->id;
+ int ret;
+
+ /*
+ * Ensure lpass BCLK/LRCLK is enabled bit before playback/capture
+ * data flow starts. This allows other codec to have some delay before
+ * the data flow.
+ * (ex: to drop start up pop noise before capture starts).
+ */
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
+ ret = regmap_fields_write(i2sctl->spken, id, LPAIF_I2SCTL_SPKEN_ENABLE);
+ else
+ ret = regmap_fields_write(i2sctl->micen, id, LPAIF_I2SCTL_MICEN_ENABLE);
+
+ if (ret) {
+ dev_err(dai->dev, "error writing to i2sctl reg: %d\n", ret);
+ return ret;
+ }
+
+ /*
+ * Check mi2s_was_prepared before enabling BCLK as lpass_cpu_daiops_prepare can
+ * be called multiple times. It's paired with the clk_disable in
+ * lpass_cpu_daiops_shutdown.
+ */
+ if (!drvdata->mi2s_was_prepared[dai->driver->id]) {
+ ret = clk_enable(drvdata->mi2s_bit_clk[id]);
+ if (ret) {
+ dev_err(dai->dev, "error in enabling mi2s bit clk: %d\n", ret);
+ return ret;
+ }
+ drvdata->mi2s_was_prepared[dai->driver->id] = true;
+ }
+ return 0;
+}
+
const struct snd_soc_dai_ops asoc_qcom_lpass_cpu_dai_ops = {
.set_sysclk = lpass_cpu_daiops_set_sysclk,
.startup = lpass_cpu_daiops_startup,
.shutdown = lpass_cpu_daiops_shutdown,
.hw_params = lpass_cpu_daiops_hw_params,
.trigger = lpass_cpu_daiops_trigger,
+ .prepare = lpass_cpu_daiops_prepare,
};
EXPORT_SYMBOL_GPL(asoc_qcom_lpass_cpu_dai_ops);
if (dai_id == LPASS_DP_RX)
continue;
- drvdata->mi2s_osr_clk[dai_id] = devm_clk_get(dev,
+ drvdata->mi2s_osr_clk[dai_id] = devm_clk_get_optional(dev,
variant->dai_osr_clk_names[i]);
- if (IS_ERR(drvdata->mi2s_osr_clk[dai_id])) {
- dev_warn(dev,
- "%s() error getting optional %s: %ld\n",
- __func__,
- variant->dai_osr_clk_names[i],
- PTR_ERR(drvdata->mi2s_osr_clk[dai_id]));
-
- drvdata->mi2s_osr_clk[dai_id] = NULL;
- }
-
drvdata->mi2s_bit_clk[dai_id] = devm_clk_get(dev,
variant->dai_bit_clk_names[i]);
if (IS_ERR(drvdata->mi2s_bit_clk[dai_id])) {
/* MI2S SD lines to use for playback/capture */
unsigned int mi2s_playback_sd_mode[LPASS_MAX_MI2S_PORTS];
unsigned int mi2s_capture_sd_mode[LPASS_MAX_MI2S_PORTS];
+
+ /* The state of MI2S prepare dai_ops was called */
+ bool mi2s_was_prepared[LPASS_MAX_MI2S_PORTS];
+
int hdmi_port_enable;
/* low-power audio interface (LPAIF) registers */
return NULL;
name = devm_kstrdup(dev, devname, GFP_KERNEL);
+ if (!name)
+ return NULL;
/* are we a "%s.%d" name (platform and SPI components) */
found = strstr(name, dev->driver->name);
* @src: older version of pcm as a source
* @pcm: latest version of pcm created from the source
*
- * Support from vesion 4. User should free the returned pcm manually.
+ * Support from version 4. User should free the returned pcm manually.
*/
static int pcm_new_ver(struct soc_tplg *tplg,
struct snd_soc_tplg_pcm *src,
* @src: old version of phyical link config as a source
* @link: latest version of physical link config created from the source
*
- * Support from vesion 4. User need free the returned link config manually.
+ * Support from version 4. User need free the returned link config manually.
*/
static int link_new_ver(struct soc_tplg *tplg,
struct snd_soc_tplg_link_config *src,
* @src: old version of manifest as a source
* @manifest: latest version of manifest created from the source
*
- * Support from vesion 4. Users need free the returned manifest manually.
+ * Support from version 4. Users need free the returned manifest manually.
*/
static int manifest_new_ver(struct soc_tplg *tplg,
struct snd_soc_tplg_manifest *src,
struct snd_soc_pcm_runtime *rtd = asoc_substream_to_rtd(substream);
struct snd_soc_component *component = snd_soc_rtdcom_lookup(rtd, SOF_AUDIO_PCM_DRV_NAME);
struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(component);
+ struct sof_ipc_fw_version *v = &sdev->fw_ready.version;
struct sof_ipc_dai_config *config;
struct snd_sof_dai *sof_dai;
struct sof_ipc_reply reply;
int ret;
+ /* DAI_CONFIG IPC during hw_params is not supported in older firmware */
+ if (v->abi_version < SOF_ABI_VER(3, 18, 0))
+ return 0;
+
list_for_each_entry(sof_dai, &sdev->dai_list, list) {
if (!sof_dai->cpu_dai_name || !sof_dai->dai_config)
continue;
/* reset FW state */
sdev->fw_state = SOF_FW_BOOT_NOT_STARTED;
+ sdev->enabled_cores_mask = 0;
return ret;
}
dev_err(dev, "mclk register returned %d\n", ret);
return ret;
}
-
- sai->sai_mclk = devm_clk_hw_get_clk(dev, hw, NULL);
- if (IS_ERR(sai->sai_mclk))
- return PTR_ERR(sai->sai_mclk);
+ sai->sai_mclk = hw->clk;
/* register mclk provider */
return devm_of_clk_add_hw_provider(dev, of_clk_hw_simple_get, hw);
if (snd_BUG_ON(altsetting >= 64 - 8))
return false;
- err = snd_usb_ctl_msg(dev, usb_sndctrlpipe(dev, 0), UAC2_CS_CUR,
+ err = snd_usb_ctl_msg(dev, usb_rcvctrlpipe(dev, 0), UAC2_CS_CUR,
USB_TYPE_CLASS | USB_RECIP_INTERFACE | USB_DIR_IN,
UAC2_AS_VAL_ALT_SETTINGS << 8,
iface, &raw_data, sizeof(raw_data));
line6->buffer_message = kmalloc(LINE6_MIDI_MESSAGE_MAXLEN, GFP_KERNEL);
if (!line6->buffer_message)
return -ENOMEM;
+
+ ret = line6_init_midi(line6);
+ if (ret < 0)
+ return ret;
} else {
ret = line6_hwdep_init(line6);
if (ret < 0)
if (err < 0)
return err;
- /* initialize MIDI subsystem: */
- err = line6_init_midi(line6);
- if (err < 0)
- return err;
-
/* initialize PCM subsystem: */
err = line6_init_pcm(line6, &pod_pcm_properties);
if (err < 0)
const struct usb_device_id *id)
{
struct usb_line6_variax *variax = line6_to_variax(line6);
- int err;
line6->process_message = line6_variax_process_message;
line6->disconnect = line6_variax_disconnect;
if (variax->buffer_activate == NULL)
return -ENOMEM;
- /* initialize MIDI subsystem: */
- err = line6_init_midi(&variax->line6);
- if (err < 0)
- return err;
-
/* initiate startup procedure: */
schedule_delayed_work(&line6->startup_work,
msecs_to_jiffies(VARIAX_STARTUP_DELAY1));
struct usb_midi_in_jack_descriptor *injd =
(struct usb_midi_in_jack_descriptor *)extra;
- if (injd->bLength > 4 &&
+ if (injd->bLength >= sizeof(*injd) &&
injd->bDescriptorType == USB_DT_CS_INTERFACE &&
injd->bDescriptorSubtype == UAC_MIDI_IN_JACK &&
injd->bJackID == jack_id)
struct usb_midi_out_jack_descriptor *outjd =
(struct usb_midi_out_jack_descriptor *)extra;
- if (outjd->bLength > 4 &&
+ if (outjd->bLength >= sizeof(*outjd) &&
outjd->bDescriptorType == USB_DT_CS_INTERFACE &&
outjd->bDescriptorSubtype == UAC_MIDI_OUT_JACK &&
outjd->bJackID == jack_id)
outjd = find_usb_out_jack_descriptor(hostif, jack_id);
if (outjd) {
sz = USB_DT_MIDI_OUT_SIZE(outjd->bNrInputPins);
- iJack = *(((uint8_t *) outjd) + sz - sizeof(uint8_t));
+ if (outjd->bLength >= sz)
+ iJack = *(((uint8_t *) outjd) + sz - sizeof(uint8_t));
}
} else {
/* and out jacks connect to ins */
ms_ep = find_usb_ms_endpoint_descriptor(hostep);
if (!ms_ep)
continue;
+ if (ms_ep->bLength <= sizeof(*ms_ep))
+ continue;
if (ms_ep->bNumEmbMIDIJack > 0x10)
continue;
+ if (ms_ep->bLength < sizeof(*ms_ep) + ms_ep->bNumEmbMIDIJack)
+ continue;
if (usb_endpoint_dir_out(ep)) {
if (endpoints[epidx].out_ep) {
if (++epidx >= MIDI_MAX_ENDPOINTS) {
case USB_ID(0x1235, 0x8203): /* Focusrite Scarlett 6i6 2nd Gen */
case USB_ID(0x1235, 0x8204): /* Focusrite Scarlett 18i8 2nd Gen */
case USB_ID(0x1235, 0x8201): /* Focusrite Scarlett 18i20 2nd Gen */
- err = snd_scarlett_gen2_controls_create(mixer);
+ err = snd_scarlett_gen2_init(mixer);
break;
case USB_ID(0x041e, 0x323b): /* Creative Sound Blaster E1 */
/* send a second message to get the response */
err = snd_usb_ctl_msg(mixer->chip->dev,
- usb_sndctrlpipe(mixer->chip->dev, 0),
+ usb_rcvctrlpipe(mixer->chip->dev, 0),
SCARLETT2_USB_VENDOR_SPECIFIC_CMD_RESP,
USB_RECIP_INTERFACE | USB_TYPE_CLASS | USB_DIR_IN,
0,
return usb_submit_urb(mixer->urb, GFP_KERNEL);
}
-/* Entry point */
-int snd_scarlett_gen2_controls_create(struct usb_mixer_interface *mixer)
+static int snd_scarlett_gen2_controls_create(struct usb_mixer_interface *mixer,
+ const struct scarlett2_device_info *info)
{
- const struct scarlett2_device_info *info;
int err;
- /* only use UAC_VERSION_2 */
- if (!mixer->protocol)
- return 0;
-
- switch (mixer->chip->usb_id) {
- case USB_ID(0x1235, 0x8203):
- info = &s6i6_gen2_info;
- break;
- case USB_ID(0x1235, 0x8204):
- info = &s18i8_gen2_info;
- break;
- case USB_ID(0x1235, 0x8201):
- info = &s18i20_gen2_info;
- break;
- default: /* device not (yet) supported */
- return -EINVAL;
- }
-
- if (!(mixer->chip->setup & SCARLETT2_ENABLE)) {
- usb_audio_err(mixer->chip,
- "Focusrite Scarlett Gen 2 Mixer Driver disabled; "
- "use options snd_usb_audio device_setup=1 "
- "to enable and report any issues to g@b4.vu");
- return 0;
- }
-
/* Initialise private data, routing, sequence number */
err = scarlett2_init_private(mixer, info);
if (err < 0)
return 0;
}
+
+int snd_scarlett_gen2_init(struct usb_mixer_interface *mixer)
+{
+ struct snd_usb_audio *chip = mixer->chip;
+ const struct scarlett2_device_info *info;
+ int err;
+
+ /* only use UAC_VERSION_2 */
+ if (!mixer->protocol)
+ return 0;
+
+ switch (chip->usb_id) {
+ case USB_ID(0x1235, 0x8203):
+ info = &s6i6_gen2_info;
+ break;
+ case USB_ID(0x1235, 0x8204):
+ info = &s18i8_gen2_info;
+ break;
+ case USB_ID(0x1235, 0x8201):
+ info = &s18i20_gen2_info;
+ break;
+ default: /* device not (yet) supported */
+ return -EINVAL;
+ }
+
+ if (!(chip->setup & SCARLETT2_ENABLE)) {
+ usb_audio_info(chip,
+ "Focusrite Scarlett Gen 2 Mixer Driver disabled; "
+ "use options snd_usb_audio vid=0x%04x pid=0x%04x "
+ "device_setup=1 to enable and report any issues "
+ "to g@b4.vu",
+ USB_ID_VENDOR(chip->usb_id),
+ USB_ID_PRODUCT(chip->usb_id));
+ return 0;
+ }
+
+ usb_audio_info(chip,
+ "Focusrite Scarlett Gen 2 Mixer Driver enabled pid=0x%04x",
+ USB_ID_PRODUCT(chip->usb_id));
+
+ err = snd_scarlett_gen2_controls_create(mixer, info);
+ if (err < 0)
+ usb_audio_err(mixer->chip,
+ "Error initialising Scarlett Mixer Driver: %d",
+ err);
+
+ return err;
+}
#ifndef __USB_MIXER_SCARLETT_GEN2_H
#define __USB_MIXER_SCARLETT_GEN2_H
-int snd_scarlett_gen2_controls_create(struct usb_mixer_interface *mixer);
+int snd_scarlett_gen2_init(struct usb_mixer_interface *mixer);
#endif /* __USB_MIXER_SCARLETT_GEN2_H */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef _ASM_MIPS_PERF_REGS_H
+#define _ASM_MIPS_PERF_REGS_H
+
+enum perf_event_mips_regs {
+ PERF_REG_MIPS_PC,
+ PERF_REG_MIPS_R1,
+ PERF_REG_MIPS_R2,
+ PERF_REG_MIPS_R3,
+ PERF_REG_MIPS_R4,
+ PERF_REG_MIPS_R5,
+ PERF_REG_MIPS_R6,
+ PERF_REG_MIPS_R7,
+ PERF_REG_MIPS_R8,
+ PERF_REG_MIPS_R9,
+ PERF_REG_MIPS_R10,
+ PERF_REG_MIPS_R11,
+ PERF_REG_MIPS_R12,
+ PERF_REG_MIPS_R13,
+ PERF_REG_MIPS_R14,
+ PERF_REG_MIPS_R15,
+ PERF_REG_MIPS_R16,
+ PERF_REG_MIPS_R17,
+ PERF_REG_MIPS_R18,
+ PERF_REG_MIPS_R19,
+ PERF_REG_MIPS_R20,
+ PERF_REG_MIPS_R21,
+ PERF_REG_MIPS_R22,
+ PERF_REG_MIPS_R23,
+ PERF_REG_MIPS_R24,
+ PERF_REG_MIPS_R25,
+ PERF_REG_MIPS_R26,
+ PERF_REG_MIPS_R27,
+ PERF_REG_MIPS_R28,
+ PERF_REG_MIPS_R29,
+ PERF_REG_MIPS_R30,
+ PERF_REG_MIPS_R31,
+ PERF_REG_MIPS_MAX = PERF_REG_MIPS_R31 + 1,
+};
+#endif /* _ASM_MIPS_PERF_REGS_H */
#ifndef _ASM_POWERPC_ERRNO_H
#define _ASM_POWERPC_ERRNO_H
+#undef EDEADLOCK
#include <asm-generic/errno.h>
#undef EDEADLOCK
/* CPU types for specific tunings: */
#define X86_FEATURE_K8 ( 3*32+ 4) /* "" Opteron, Athlon64 */
-#define X86_FEATURE_K7 ( 3*32+ 5) /* "" Athlon */
+/* FREE, was #define X86_FEATURE_K7 ( 3*32+ 5) "" Athlon */
#define X86_FEATURE_P3 ( 3*32+ 6) /* "" P3 */
#define X86_FEATURE_P4 ( 3*32+ 7) /* "" P4 */
#define X86_FEATURE_CONSTANT_TSC ( 3*32+ 8) /* TSC ticks at a constant rate */
#define X86_FEATURE_EPT_AD ( 8*32+17) /* Intel Extended Page Table access-dirty bit */
#define X86_FEATURE_VMCALL ( 8*32+18) /* "" Hypervisor supports the VMCALL instruction */
#define X86_FEATURE_VMW_VMMCALL ( 8*32+19) /* "" VMware prefers VMMCALL hypercall instruction */
+#define X86_FEATURE_PVUNLOCK ( 8*32+20) /* "" PV unlock function */
+#define X86_FEATURE_VCPUPREEMPT ( 8*32+21) /* "" PV vcpu_is_preempted function */
/* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */
#define X86_FEATURE_FSGSBASE ( 9*32+ 0) /* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/
#define X86_FEATURE_FENCE_SWAPGS_KERNEL (11*32+ 5) /* "" LFENCE in kernel entry SWAPGS path */
#define X86_FEATURE_SPLIT_LOCK_DETECT (11*32+ 6) /* #AC for split lock */
#define X86_FEATURE_PER_THREAD_MBA (11*32+ 7) /* "" Per-thread Memory Bandwidth Allocation */
+#define X86_FEATURE_SGX1 (11*32+ 8) /* "" Basic SGX */
+#define X86_FEATURE_SGX2 (11*32+ 9) /* "" SGX Enclave Dynamic Memory Management (EDMM) */
/* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */
#define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */
#define X86_FEATURE_AVIC (15*32+13) /* Virtual Interrupt Controller */
#define X86_FEATURE_V_VMSAVE_VMLOAD (15*32+15) /* Virtual VMSAVE VMLOAD */
#define X86_FEATURE_VGIF (15*32+16) /* Virtual GIF */
+#define X86_FEATURE_V_SPEC_CTRL (15*32+20) /* Virtual SPEC_CTRL */
#define X86_FEATURE_SVME_ADDR_CHK (15*32+28) /* "" SVME addr check */
/* Intel-defined CPU features, CPUID level 0x00000007:0 (ECX), word 16 */
#define X86_FEATURE_AVX512_VPOPCNTDQ (16*32+14) /* POPCNT for vectors of DW/QW */
#define X86_FEATURE_LA57 (16*32+16) /* 5-level page tables */
#define X86_FEATURE_RDPID (16*32+22) /* RDPID instruction */
+#define X86_FEATURE_BUS_LOCK_DETECT (16*32+24) /* Bus Lock detect */
#define X86_FEATURE_CLDEMOTE (16*32+25) /* CLDEMOTE instruction */
#define X86_FEATURE_MOVDIRI (16*32+27) /* MOVDIRI instruction */
#define X86_FEATURE_MOVDIR64B (16*32+28) /* MOVDIR64B instruction */
#define X86_FEATURE_MD_CLEAR (18*32+10) /* VERW clears CPU buffers */
#define X86_FEATURE_TSX_FORCE_ABORT (18*32+13) /* "" TSX_FORCE_ABORT */
#define X86_FEATURE_SERIALIZE (18*32+14) /* SERIALIZE instruction */
+#define X86_FEATURE_HYBRID_CPU (18*32+15) /* "" This part has CPUs of more than one type */
#define X86_FEATURE_TSXLDTRK (18*32+16) /* TSX Suspend Load Address Tracking */
#define X86_FEATURE_PCONFIG (18*32+18) /* Intel PCONFIG */
#define X86_FEATURE_ARCH_LBR (18*32+19) /* Intel ARCH LBR */
# define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31))
#endif
-#ifdef CONFIG_IOMMU_SUPPORT
-# define DISABLE_ENQCMD 0
-#else
-# define DISABLE_ENQCMD (1 << (X86_FEATURE_ENQCMD & 31))
-#endif
+/* Force disable because it's broken beyond repair */
+#define DISABLE_ENQCMD (1 << (X86_FEATURE_ENQCMD & 31))
#ifdef CONFIG_X86_SGX
# define DISABLE_SGX 0
#define MSR_PEBS_DATA_CFG 0x000003f2
#define MSR_IA32_DS_AREA 0x00000600
#define MSR_IA32_PERF_CAPABILITIES 0x00000345
+#define PERF_CAP_METRICS_IDX 15
+#define PERF_CAP_PT_IDX 16
+
#define MSR_PEBS_LD_LAT_THRESHOLD 0x000003f6
#define MSR_IA32_RTIT_CTL 0x00000570
#define DEBUGCTLMSR_LBR (1UL << 0) /* last branch recording */
#define DEBUGCTLMSR_BTF_SHIFT 1
#define DEBUGCTLMSR_BTF (1UL << 1) /* single-step on branches */
+#define DEBUGCTLMSR_BUS_LOCK_DETECT (1UL << 2)
#define DEBUGCTLMSR_TR (1UL << 6)
#define DEBUGCTLMSR_BTS (1UL << 7)
#define DEBUGCTLMSR_BTINT (1UL << 8)
/* K8 MSRs */
#define MSR_K8_TOP_MEM1 0xc001001a
#define MSR_K8_TOP_MEM2 0xc001001d
-#define MSR_K8_SYSCFG 0xc0010010
-#define MSR_K8_SYSCFG_MEM_ENCRYPT_BIT 23
-#define MSR_K8_SYSCFG_MEM_ENCRYPT BIT_ULL(MSR_K8_SYSCFG_MEM_ENCRYPT_BIT)
+#define MSR_AMD64_SYSCFG 0xc0010010
+#define MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT 23
+#define MSR_AMD64_SYSCFG_MEM_ENCRYPT BIT_ULL(MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT)
#define MSR_K8_INT_PENDING_MSG 0xc0010055
/* C1E active bits in int pending message */
#define K8_INTP_C1E_ACTIVE_MASK 0x18000000
__u16 flags;
} smm;
+ __u16 pad;
+
__u32 flags;
__u64 preemption_timer_deadline;
};
#define VMX_EXIT_REASONS_FAILED_VMENTRY 0x80000000
+#define VMX_EXIT_REASONS_SGX_ENCLAVE_MODE 0x08000000
#define EXIT_REASON_EXCEPTION_NMI 0
#define EXIT_REASON_EXTERNAL_INTERRUPT 1
#include <linux/linkage.h>
#include <asm/errno.h>
#include <asm/cpufeatures.h>
-#include <asm/alternative-asm.h>
+#include <asm/alternative.h>
#include <asm/export.h>
.pushsection .noinstr.text, "ax"
#include <linux/linkage.h>
#include <asm/cpufeatures.h>
-#include <asm/alternative-asm.h>
+#include <asm/alternative.h>
#include <asm/export.h>
/*
#include "../../../../include/linux/bootconfig.h"
+#ifndef fallthrough
+# define fallthrough
+#endif
+
#endif
}
/* TODO: Ensure the @path is initramfs/initrd image */
if (fstat(fd, &stat) < 0) {
+ ret = -errno;
pr_err("Failed to get the size of %s\n", path);
goto out;
}
| *ATTACH_TYPE* := { **ingress** | **egress** | **sock_create** | **sock_ops** | **device** |
| **bind4** | **bind6** | **post_bind4** | **post_bind6** | **connect4** | **connect6** |
| **getpeername4** | **getpeername6** | **getsockname4** | **getsockname6** | **sendmsg4** |
-| **sendmsg6** | **recvmsg4** | **recvmsg6** | **sysctl** | **getsockopt** | **setsockopt** }
+| **sendmsg6** | **recvmsg4** | **recvmsg6** | **sysctl** | **getsockopt** | **setsockopt** |
+| **sock_release** }
| *ATTACH_FLAGS* := { **multi** | **override** }
DESCRIPTION
**getpeername6** call to getpeername(2) for an inet6 socket (since 5.8);
**getsockname4** call to getsockname(2) for an inet4 socket (since 5.8);
**getsockname6** call to getsockname(2) for an inet6 socket (since 5.8).
+ **sock_release** closing an userspace inet socket (since 5.9).
**bpftool cgroup detach** *CGROUP* *ATTACH_TYPE* *PROG*
Detach *PROG* from the cgroup *CGROUP* and attach type
| **cgroup/connect4** | **cgroup/connect6** | **cgroup/getpeername4** | **cgroup/getpeername6** |
| **cgroup/getsockname4** | **cgroup/getsockname6** | **cgroup/sendmsg4** | **cgroup/sendmsg6** |
| **cgroup/recvmsg4** | **cgroup/recvmsg6** | **cgroup/sysctl** |
-| **cgroup/getsockopt** | **cgroup/setsockopt** |
+| **cgroup/getsockopt** | **cgroup/setsockopt** | **cgroup/sock_release** |
| **struct_ops** | **fentry** | **fexit** | **freplace** | **sk_lookup**
| }
| *ATTACH_TYPE* := {
BPFTOOL_BOOTSTRAP := $(BOOTSTRAP_OUTPUT)bpftool
-BOOTSTRAP_OBJS = $(addprefix $(BOOTSTRAP_OUTPUT),main.o common.o json_writer.o gen.o btf.o)
+BOOTSTRAP_OBJS = $(addprefix $(BOOTSTRAP_OUTPUT),main.o common.o json_writer.o gen.o btf.o xlated_dumper.o btf_dumper.o disasm.o)
OBJS = $(patsubst %.c,$(OUTPUT)%.o,$(SRCS)) $(OUTPUT)disasm.o
VMLINUX_BTF_PATHS ?= $(if $(O),$(O)/vmlinux) \
CFLAGS += $(if $(BUILD_BPF_SKELS),,-DBPFTOOL_WITHOUT_SKELETONS)
+$(BOOTSTRAP_OUTPUT)disasm.o: $(srctree)/kernel/bpf/disasm.c
+ $(QUIET_CC)$(HOSTCC) $(CFLAGS) -c -MMD -o $@ $<
+
$(OUTPUT)disasm.o: $(srctree)/kernel/bpf/disasm.c
$(QUIET_CC)$(CC) $(CFLAGS) -c -MMD -o $@ $<
cgroup/recvmsg4 cgroup/recvmsg6 \
cgroup/post_bind4 cgroup/post_bind6 \
cgroup/sysctl cgroup/getsockopt \
- cgroup/setsockopt struct_ops \
+ cgroup/setsockopt cgroup/sock_release struct_ops \
fentry fexit freplace sk_lookup" -- \
"$cur" ) )
return 0
device bind4 bind6 post_bind4 post_bind6 connect4 connect6 \
getpeername4 getpeername6 getsockname4 getsockname6 \
sendmsg4 sendmsg6 recvmsg4 recvmsg6 sysctl getsockopt \
- setsockopt'
+ setsockopt sock_release'
local ATTACH_FLAGS='multi override'
local PROG_TYPE='id pinned tag name'
case $prev in
ingress|egress|sock_create|sock_ops|device|bind4|bind6|\
post_bind4|post_bind6|connect4|connect6|getpeername4|\
getpeername6|getsockname4|getsockname6|sendmsg4|sendmsg6|\
- recvmsg4|recvmsg6|sysctl|getsockopt|setsockopt)
+ recvmsg4|recvmsg6|sysctl|getsockopt|setsockopt|sock_release)
COMPREPLY=( $( compgen -W "$PROG_TYPE" -- \
"$cur" ) )
return 0
" connect6 | getpeername4 | getpeername6 |\n" \
" getsockname4 | getsockname6 | sendmsg4 |\n" \
" sendmsg6 | recvmsg4 | recvmsg6 |\n" \
- " sysctl | getsockopt | setsockopt }"
+ " sysctl | getsockopt | setsockopt |\n" \
+ " sock_release }"
static unsigned int query_flags;
#include <sys/stat.h>
#include <sys/mman.h>
#include <bpf/btf.h>
+#include <bpf/bpf_gen_internal.h>
#include "json_writer.h"
#include "main.h"
if (strcmp(sec_name, ".data") == 0) {
sec_ident = "data";
+ strip_mods = true;
} else if (strcmp(sec_name, ".bss") == 0) {
sec_ident = "bss";
+ strip_mods = true;
} else if (strcmp(sec_name, ".rodata") == 0) {
sec_ident = "rodata";
strip_mods = true;
int need_off = sec_var->offset, align_off, align;
__u32 var_type_id = var->type;
+ /* static variables are not exposed through BPF skeleton */
+ if (btf_var(var)->linkage == BTF_VAR_STATIC)
+ continue;
+
if (off > need_off) {
p_err("Something is wrong for %s's variable #%d: need offset %d, already at %d.\n",
sec_name, i, need_off, off);
free(s);
}
+static void print_hex(const char *data, int data_sz)
+{
+ int i, len;
+
+ for (i = 0, len = 0; i < data_sz; i++) {
+ int w = data[i] ? 4 : 2;
+
+ len += w;
+ if (len > 78) {
+ printf("\\\n");
+ len = w;
+ }
+ if (!data[i])
+ printf("\\0");
+ else
+ printf("\\x%02x", (unsigned char)data[i]);
+ }
+}
+
+static size_t bpf_map_mmap_sz(const struct bpf_map *map)
+{
+ long page_sz = sysconf(_SC_PAGE_SIZE);
+ size_t map_sz;
+
+ map_sz = (size_t)roundup(bpf_map__value_size(map), 8) * bpf_map__max_entries(map);
+ map_sz = roundup(map_sz, page_sz);
+ return map_sz;
+}
+
+static void codegen_attach_detach(struct bpf_object *obj, const char *obj_name)
+{
+ struct bpf_program *prog;
+
+ bpf_object__for_each_program(prog, obj) {
+ const char *tp_name;
+
+ codegen("\
+ \n\
+ \n\
+ static inline int \n\
+ %1$s__%2$s__attach(struct %1$s *skel) \n\
+ { \n\
+ int prog_fd = skel->progs.%2$s.prog_fd; \n\
+ ", obj_name, bpf_program__name(prog));
+
+ switch (bpf_program__get_type(prog)) {
+ case BPF_PROG_TYPE_RAW_TRACEPOINT:
+ tp_name = strchr(bpf_program__section_name(prog), '/') + 1;
+ printf("\tint fd = bpf_raw_tracepoint_open(\"%s\", prog_fd);\n", tp_name);
+ break;
+ case BPF_PROG_TYPE_TRACING:
+ printf("\tint fd = bpf_raw_tracepoint_open(NULL, prog_fd);\n");
+ break;
+ default:
+ printf("\tint fd = ((void)prog_fd, 0); /* auto-attach not supported */\n");
+ break;
+ }
+ codegen("\
+ \n\
+ \n\
+ if (fd > 0) \n\
+ skel->links.%1$s_fd = fd; \n\
+ return fd; \n\
+ } \n\
+ ", bpf_program__name(prog));
+ }
+
+ codegen("\
+ \n\
+ \n\
+ static inline int \n\
+ %1$s__attach(struct %1$s *skel) \n\
+ { \n\
+ int ret = 0; \n\
+ \n\
+ ", obj_name);
+
+ bpf_object__for_each_program(prog, obj) {
+ codegen("\
+ \n\
+ ret = ret < 0 ? ret : %1$s__%2$s__attach(skel); \n\
+ ", obj_name, bpf_program__name(prog));
+ }
+
+ codegen("\
+ \n\
+ return ret < 0 ? ret : 0; \n\
+ } \n\
+ \n\
+ static inline void \n\
+ %1$s__detach(struct %1$s *skel) \n\
+ { \n\
+ ", obj_name);
+
+ bpf_object__for_each_program(prog, obj) {
+ codegen("\
+ \n\
+ skel_closenz(skel->links.%1$s_fd); \n\
+ ", bpf_program__name(prog));
+ }
+
+ codegen("\
+ \n\
+ } \n\
+ ");
+}
+
+static void codegen_destroy(struct bpf_object *obj, const char *obj_name)
+{
+ struct bpf_program *prog;
+ struct bpf_map *map;
+
+ codegen("\
+ \n\
+ static void \n\
+ %1$s__destroy(struct %1$s *skel) \n\
+ { \n\
+ if (!skel) \n\
+ return; \n\
+ %1$s__detach(skel); \n\
+ ",
+ obj_name);
+
+ bpf_object__for_each_program(prog, obj) {
+ codegen("\
+ \n\
+ skel_closenz(skel->progs.%1$s.prog_fd); \n\
+ ", bpf_program__name(prog));
+ }
+
+ bpf_object__for_each_map(map, obj) {
+ const char * ident;
+
+ ident = get_map_ident(map);
+ if (!ident)
+ continue;
+ if (bpf_map__is_internal(map) &&
+ (bpf_map__def(map)->map_flags & BPF_F_MMAPABLE))
+ printf("\tmunmap(skel->%1$s, %2$zd);\n",
+ ident, bpf_map_mmap_sz(map));
+ codegen("\
+ \n\
+ skel_closenz(skel->maps.%1$s.map_fd); \n\
+ ", ident);
+ }
+ codegen("\
+ \n\
+ free(skel); \n\
+ } \n\
+ ",
+ obj_name);
+}
+
+static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *header_guard)
+{
+ struct bpf_object_load_attr load_attr = {};
+ DECLARE_LIBBPF_OPTS(gen_loader_opts, opts);
+ struct bpf_map *map;
+ int err = 0;
+
+ err = bpf_object__gen_loader(obj, &opts);
+ if (err)
+ return err;
+
+ load_attr.obj = obj;
+ if (verifier_logs)
+ /* log_level1 + log_level2 + stats, but not stable UAPI */
+ load_attr.log_level = 1 + 2 + 4;
+
+ err = bpf_object__load_xattr(&load_attr);
+ if (err) {
+ p_err("failed to load object file");
+ goto out;
+ }
+ /* If there was no error during load then gen_loader_opts
+ * are populated with the loader program.
+ */
+
+ /* finish generating 'struct skel' */
+ codegen("\
+ \n\
+ }; \n\
+ ", obj_name);
+
+
+ codegen_attach_detach(obj, obj_name);
+
+ codegen_destroy(obj, obj_name);
+
+ codegen("\
+ \n\
+ static inline struct %1$s * \n\
+ %1$s__open(void) \n\
+ { \n\
+ struct %1$s *skel; \n\
+ \n\
+ skel = calloc(sizeof(*skel), 1); \n\
+ if (!skel) \n\
+ goto cleanup; \n\
+ skel->ctx.sz = (void *)&skel->links - (void *)skel; \n\
+ ",
+ obj_name, opts.data_sz);
+ bpf_object__for_each_map(map, obj) {
+ const char *ident;
+ const void *mmap_data = NULL;
+ size_t mmap_size = 0;
+
+ ident = get_map_ident(map);
+ if (!ident)
+ continue;
+
+ if (!bpf_map__is_internal(map) ||
+ !(bpf_map__def(map)->map_flags & BPF_F_MMAPABLE))
+ continue;
+
+ codegen("\
+ \n\
+ skel->%1$s = \n\
+ mmap(NULL, %2$zd, PROT_READ | PROT_WRITE,\n\
+ MAP_SHARED | MAP_ANONYMOUS, -1, 0); \n\
+ if (skel->%1$s == (void *) -1) \n\
+ goto cleanup; \n\
+ memcpy(skel->%1$s, (void *)\"\\ \n\
+ ", ident, bpf_map_mmap_sz(map));
+ mmap_data = bpf_map__initial_value(map, &mmap_size);
+ print_hex(mmap_data, mmap_size);
+ printf("\", %2$zd);\n"
+ "\tskel->maps.%1$s.initial_value = (__u64)(long)skel->%1$s;\n",
+ ident, mmap_size);
+ }
+ codegen("\
+ \n\
+ return skel; \n\
+ cleanup: \n\
+ %1$s__destroy(skel); \n\
+ return NULL; \n\
+ } \n\
+ \n\
+ static inline int \n\
+ %1$s__load(struct %1$s *skel) \n\
+ { \n\
+ struct bpf_load_and_run_opts opts = {}; \n\
+ int err; \n\
+ \n\
+ opts.ctx = (struct bpf_loader_ctx *)skel; \n\
+ opts.data_sz = %2$d; \n\
+ opts.data = (void *)\"\\ \n\
+ ",
+ obj_name, opts.data_sz);
+ print_hex(opts.data, opts.data_sz);
+ codegen("\
+ \n\
+ \"; \n\
+ ");
+
+ codegen("\
+ \n\
+ opts.insns_sz = %d; \n\
+ opts.insns = (void *)\"\\ \n\
+ ",
+ opts.insns_sz);
+ print_hex(opts.insns, opts.insns_sz);
+ codegen("\
+ \n\
+ \"; \n\
+ err = bpf_load_and_run(&opts); \n\
+ if (err < 0) \n\
+ return err; \n\
+ ", obj_name);
+ bpf_object__for_each_map(map, obj) {
+ const char *ident, *mmap_flags;
+
+ ident = get_map_ident(map);
+ if (!ident)
+ continue;
+
+ if (!bpf_map__is_internal(map) ||
+ !(bpf_map__def(map)->map_flags & BPF_F_MMAPABLE))
+ continue;
+ if (bpf_map__def(map)->map_flags & BPF_F_RDONLY_PROG)
+ mmap_flags = "PROT_READ";
+ else
+ mmap_flags = "PROT_READ | PROT_WRITE";
+
+ printf("\tskel->%1$s =\n"
+ "\t\tmmap(skel->%1$s, %2$zd, %3$s, MAP_SHARED | MAP_FIXED,\n"
+ "\t\t\tskel->maps.%1$s.map_fd, 0);\n",
+ ident, bpf_map_mmap_sz(map), mmap_flags);
+ }
+ codegen("\
+ \n\
+ return 0; \n\
+ } \n\
+ \n\
+ static inline struct %1$s * \n\
+ %1$s__open_and_load(void) \n\
+ { \n\
+ struct %1$s *skel; \n\
+ \n\
+ skel = %1$s__open(); \n\
+ if (!skel) \n\
+ return NULL; \n\
+ if (%1$s__load(skel)) { \n\
+ %1$s__destroy(skel); \n\
+ return NULL; \n\
+ } \n\
+ return skel; \n\
+ } \n\
+ ", obj_name);
+
+ codegen("\
+ \n\
+ \n\
+ #endif /* %s */ \n\
+ ",
+ header_guard);
+ err = 0;
+out:
+ return err;
+}
+
static int do_skeleton(int argc, char **argv)
{
char header_guard[MAX_OBJ_NAME_LEN + sizeof("__SKEL_H__")];
struct bpf_object *obj = NULL;
const char *file, *ident;
struct bpf_program *prog;
- int fd, len, err = -1;
+ int fd, err = -1;
struct bpf_map *map;
struct btf *btf;
struct stat st;
}
get_header_guard(header_guard, obj_name);
- codegen("\
+ if (use_loader) {
+ codegen("\
+ \n\
+ /* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */ \n\
+ /* THIS FILE IS AUTOGENERATED! */ \n\
+ #ifndef %2$s \n\
+ #define %2$s \n\
+ \n\
+ #include <stdlib.h> \n\
+ #include <bpf/bpf.h> \n\
+ #include <bpf/skel_internal.h> \n\
+ \n\
+ struct %1$s { \n\
+ struct bpf_loader_ctx ctx; \n\
+ ",
+ obj_name, header_guard
+ );
+ } else {
+ codegen("\
\n\
/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */ \n\
\n\
#ifndef %2$s \n\
#define %2$s \n\
\n\
+ #include <errno.h> \n\
#include <stdlib.h> \n\
#include <bpf/libbpf.h> \n\
\n\
struct bpf_object *obj; \n\
",
obj_name, header_guard
- );
+ );
+ }
if (map_cnt) {
printf("\tstruct {\n");
ident = get_map_ident(map);
if (!ident)
continue;
- printf("\t\tstruct bpf_map *%s;\n", ident);
+ if (use_loader)
+ printf("\t\tstruct bpf_map_desc %s;\n", ident);
+ else
+ printf("\t\tstruct bpf_map *%s;\n", ident);
}
printf("\t} maps;\n");
}
if (prog_cnt) {
printf("\tstruct {\n");
bpf_object__for_each_program(prog, obj) {
- printf("\t\tstruct bpf_program *%s;\n",
- bpf_program__name(prog));
+ if (use_loader)
+ printf("\t\tstruct bpf_prog_desc %s;\n",
+ bpf_program__name(prog));
+ else
+ printf("\t\tstruct bpf_program *%s;\n",
+ bpf_program__name(prog));
}
printf("\t} progs;\n");
printf("\tstruct {\n");
bpf_object__for_each_program(prog, obj) {
- printf("\t\tstruct bpf_link *%s;\n",
- bpf_program__name(prog));
+ if (use_loader)
+ printf("\t\tint %s_fd;\n",
+ bpf_program__name(prog));
+ else
+ printf("\t\tstruct bpf_link *%s;\n",
+ bpf_program__name(prog));
}
printf("\t} links;\n");
}
if (err)
goto out;
}
+ if (use_loader) {
+ err = gen_trace(obj, obj_name, header_guard);
+ goto out;
+ }
codegen("\
\n\
%1$s__open_opts(const struct bpf_object_open_opts *opts) \n\
{ \n\
struct %1$s *obj; \n\
+ int err; \n\
\n\
obj = (struct %1$s *)calloc(1, sizeof(*obj)); \n\
- if (!obj) \n\
+ if (!obj) { \n\
+ errno = ENOMEM; \n\
return NULL; \n\
- if (%1$s__create_skeleton(obj)) \n\
- goto err; \n\
- if (bpf_object__open_skeleton(obj->skeleton, opts)) \n\
- goto err; \n\
+ } \n\
+ \n\
+ err = %1$s__create_skeleton(obj); \n\
+ err = err ?: bpf_object__open_skeleton(obj->skeleton, opts);\n\
+ if (err) \n\
+ goto err_out; \n\
\n\
return obj; \n\
- err: \n\
+ err_out: \n\
%1$s__destroy(obj); \n\
+ errno = -err; \n\
return NULL; \n\
} \n\
\n\
%1$s__open_and_load(void) \n\
{ \n\
struct %1$s *obj; \n\
+ int err; \n\
\n\
obj = %1$s__open(); \n\
if (!obj) \n\
return NULL; \n\
- if (%1$s__load(obj)) { \n\
+ err = %1$s__load(obj); \n\
+ if (err) { \n\
%1$s__destroy(obj); \n\
+ errno = -err; \n\
return NULL; \n\
} \n\
return obj; \n\
\n\
s = (struct bpf_object_skeleton *)calloc(1, sizeof(*s));\n\
if (!s) \n\
- return -1; \n\
+ goto err; \n\
obj->skeleton = s; \n\
\n\
s->sz = sizeof(*s); \n\
file_sz);
/* embed contents of BPF object file */
- for (i = 0, len = 0; i < file_sz; i++) {
- int w = obj_data[i] ? 4 : 2;
-
- len += w;
- if (len > 78) {
- printf("\\\n");
- len = w;
- }
- if (!obj_data[i])
- printf("\\0");
- else
- printf("\\x%02x", (unsigned char)obj_data[i]);
- }
+ print_hex(obj_data, file_sz);
codegen("\
\n\
return 0; \n\
err: \n\
bpf_object__destroy_skeleton(s); \n\
- return -1; \n\
+ return -ENOMEM; \n\
} \n\
\n\
#endif /* %s */ \n\
while (argc) {
file = GET_ARG();
- err = bpf_linker__add_file(linker, file);
+ err = bpf_linker__add_file(linker, file, NULL);
if (err) {
p_err("failed to link '%s': %s (%d)", file, strerror(err), err);
goto out;
bool block_mount;
bool verifier_logs;
bool relaxed_maps;
+bool use_loader;
struct btf *base_btf;
struct pinned_obj_table prog_table;
struct pinned_obj_table map_table;
n_argc = make_args(buf, n_argv, BATCH_ARG_NB_MAX, lines);
if (!n_argc)
continue;
- if (n_argc < 0)
+ if (n_argc < 0) {
+ err = n_argc;
goto err_close;
+ }
if (json_output) {
jsonw_start_object(json_wtr);
{ "mapcompat", no_argument, NULL, 'm' },
{ "nomount", no_argument, NULL, 'n' },
{ "debug", no_argument, NULL, 'd' },
+ { "use-loader", no_argument, NULL, 'L' },
{ "base-btf", required_argument, NULL, 'B' },
{ 0 }
};
hash_init(link_table.table);
opterr = 0;
- while ((opt = getopt_long(argc, argv, "VhpjfmndB:",
+ while ((opt = getopt_long(argc, argv, "VhpjfLmndB:",
options, NULL)) >= 0) {
switch (opt) {
case 'V':
return -1;
}
break;
+ case 'L':
+ use_loader = true;
+ break;
default:
p_err("unrecognized option '%s'", argv[optind - 1]);
if (json_output)
extern bool block_mount;
extern bool verifier_logs;
extern bool relaxed_maps;
+extern bool use_loader;
extern struct btf *base_btf;
extern struct pinned_obj_table prog_table;
extern struct pinned_obj_table map_table;
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/syscall.h>
+#include <dirent.h>
#include <linux/err.h>
#include <linux/perf_event.h>
#include <bpf/bpf.h>
#include <bpf/btf.h>
#include <bpf/libbpf.h>
+#include <bpf/bpf_gen_internal.h>
+#include <bpf/skel_internal.h>
#include "cfg.h"
#include "main.h"
set_max_rlimit();
obj = bpf_object__open_file(file, &open_opts);
- if (IS_ERR_OR_NULL(obj)) {
+ if (libbpf_get_error(obj)) {
p_err("failed to open object file");
goto err_free_reuse_maps;
}
return -1;
}
+static int count_open_fds(void)
+{
+ DIR *dp = opendir("/proc/self/fd");
+ struct dirent *de;
+ int cnt = -3;
+
+ if (!dp)
+ return -1;
+
+ while ((de = readdir(dp)))
+ cnt++;
+
+ closedir(dp);
+ return cnt;
+}
+
+static int try_loader(struct gen_loader_opts *gen)
+{
+ struct bpf_load_and_run_opts opts = {};
+ struct bpf_loader_ctx *ctx;
+ int ctx_sz = sizeof(*ctx) + 64 * max(sizeof(struct bpf_map_desc),
+ sizeof(struct bpf_prog_desc));
+ int log_buf_sz = (1u << 24) - 1;
+ int err, fds_before, fd_delta;
+ char *log_buf;
+
+ ctx = alloca(ctx_sz);
+ memset(ctx, 0, ctx_sz);
+ ctx->sz = ctx_sz;
+ ctx->log_level = 1;
+ ctx->log_size = log_buf_sz;
+ log_buf = malloc(log_buf_sz);
+ if (!log_buf)
+ return -ENOMEM;
+ ctx->log_buf = (long) log_buf;
+ opts.ctx = ctx;
+ opts.data = gen->data;
+ opts.data_sz = gen->data_sz;
+ opts.insns = gen->insns;
+ opts.insns_sz = gen->insns_sz;
+ fds_before = count_open_fds();
+ err = bpf_load_and_run(&opts);
+ fd_delta = count_open_fds() - fds_before;
+ if (err < 0) {
+ fprintf(stderr, "err %d\n%s\n%s", err, opts.errstr, log_buf);
+ if (fd_delta)
+ fprintf(stderr, "loader prog leaked %d FDs\n",
+ fd_delta);
+ }
+ free(log_buf);
+ return err;
+}
+
+static int do_loader(int argc, char **argv)
+{
+ DECLARE_LIBBPF_OPTS(bpf_object_open_opts, open_opts);
+ DECLARE_LIBBPF_OPTS(gen_loader_opts, gen);
+ struct bpf_object_load_attr load_attr = {};
+ struct bpf_object *obj;
+ const char *file;
+ int err = 0;
+
+ if (!REQ_ARGS(1))
+ return -1;
+ file = GET_ARG();
+
+ obj = bpf_object__open_file(file, &open_opts);
+ if (libbpf_get_error(obj)) {
+ p_err("failed to open object file");
+ goto err_close_obj;
+ }
+
+ err = bpf_object__gen_loader(obj, &gen);
+ if (err)
+ goto err_close_obj;
+
+ load_attr.obj = obj;
+ if (verifier_logs)
+ /* log_level1 + log_level2 + stats, but not stable UAPI */
+ load_attr.log_level = 1 + 2 + 4;
+
+ err = bpf_object__load_xattr(&load_attr);
+ if (err) {
+ p_err("failed to load object file");
+ goto err_close_obj;
+ }
+
+ if (verifier_logs) {
+ struct dump_data dd = {};
+
+ kernel_syms_load(&dd);
+ dump_xlated_plain(&dd, (void *)gen.insns, gen.insns_sz, false, false);
+ kernel_syms_destroy(&dd);
+ }
+ err = try_loader(&gen);
+err_close_obj:
+ bpf_object__close(obj);
+ return err;
+}
+
static int do_load(int argc, char **argv)
{
+ if (use_loader)
+ return do_loader(argc, argv);
return load_with_options(argc, argv, true);
}
" cgroup/getpeername4 | cgroup/getpeername6 |\n"
" cgroup/getsockname4 | cgroup/getsockname6 | cgroup/sendmsg4 |\n"
" cgroup/sendmsg6 | cgroup/recvmsg4 | cgroup/recvmsg6 |\n"
- " cgroup/getsockopt | cgroup/setsockopt |\n"
+ " cgroup/getsockopt | cgroup/setsockopt | cgroup/sock_release |\n"
" struct_ops | fentry | fexit | freplace | sk_lookup }\n"
" ATTACH_TYPE := { msg_verdict | stream_verdict | stream_parser |\n"
" flow_dissector }\n"
else if (insn->src_reg == BPF_PSEUDO_MAP_VALUE)
snprintf(dd->scratch_buff, sizeof(dd->scratch_buff),
"map[id:%u][0]+%u", insn->imm, (insn + 1)->imm);
+ else if (insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE)
+ snprintf(dd->scratch_buff, sizeof(dd->scratch_buff),
+ "map[idx:%u]+%u", insn->imm, (insn + 1)->imm);
else if (insn->src_reg == BPF_PSEUDO_FUNC)
snprintf(dd->scratch_buff, sizeof(dd->scratch_buff),
"subprog[%+d]", insn->imm);
build-file := $(dir)/Build
-include $(build-file)
-quiet_cmd_flex = FLEX $@
-quiet_cmd_bison = BISON $@
+quiet_cmd_flex = FLEX $@
+quiet_cmd_bison = BISON $@
# Create directory unless it exists
-quiet_cmd_mkdir = MKDIR $(dir $@)
+quiet_cmd_mkdir = MKDIR $(dir $@)
cmd_mkdir = mkdir -p $(dir $@)
rule_mkdir = $(if $(wildcard $(dir $@)),,@$(call echo-cmd,mkdir) $(cmd_mkdir))
# Compile command
-quiet_cmd_cc_o_c = CC $@
+quiet_cmd_cc_o_c = CC $@
cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $<
-quiet_cmd_host_cc_o_c = HOSTCC $@
+quiet_cmd_host_cc_o_c = HOSTCC $@
cmd_host_cc_o_c = $(HOSTCC) $(host_c_flags) -c -o $@ $<
-quiet_cmd_cxx_o_c = CXX $@
+quiet_cmd_cxx_o_c = CXX $@
cmd_cxx_o_c = $(CXX) $(cxx_flags) -c -o $@ $<
-quiet_cmd_cpp_i_c = CPP $@
+quiet_cmd_cpp_i_c = CPP $@
cmd_cpp_i_c = $(CC) $(c_flags) -E -o $@ $<
-quiet_cmd_cc_s_c = AS $@
+quiet_cmd_cc_s_c = AS $@
cmd_cc_s_c = $(CC) $(c_flags) -S -o $@ $<
-quiet_cmd_gen = GEN $@
+quiet_cmd_gen = GEN $@
# Link agregate command
# If there's nothing to link, create empty $@ object.
-quiet_cmd_ld_multi = LD $@
+quiet_cmd_ld_multi = LD $@
cmd_ld_multi = $(if $(strip $(obj-y)),\
$(LD) -r -o $@ $(filter $(obj-y),$^),rm -f $@; $(AR) rcs $@)
-quiet_cmd_host_ld_multi = HOSTLD $@
+quiet_cmd_host_ld_multi = HOSTLD $@
cmd_host_ld_multi = $(if $(strip $(obj-y)),\
$(HOSTLD) -r -o $@ $(filter $(obj-y),$^),rm -f $@; $(HOSTAR) rcs $@)
#include <linux/build_bug.h>
#define GENMASK_INPUT_CHECK(h, l) \
(BUILD_BUG_ON_ZERO(__builtin_choose_expr( \
- __builtin_constant_p((l) > (h)), (l) > (h), 0)))
+ __is_constexpr((l) > (h)), (l) > (h), 0)))
#else
/*
* BUILD_BUG_ON_ZERO is not available in h files included from asm files,
#include <vdso/const.h>
+/*
+ * This returns a constant expression while determining if an argument is
+ * a constant expression, most importantly without evaluating the argument.
+ * Glory to Martin Uecker <Martin.Uecker@med.uni-goettingen.de>
+ */
+#define __is_constexpr(x) \
+ (sizeof(int) == sizeof(*(8 ? ((void *)((long)(x) * 0l)) : (int *)8)))
+
#endif /* _LINUX_CONST_H */
__SC_COMP(__NR_epoll_pwait2, sys_epoll_pwait2, compat_sys_epoll_pwait2)
#define __NR_mount_setattr 442
__SYSCALL(__NR_mount_setattr, sys_mount_setattr)
+#define __NR_quotactl_path 443
+__SYSCALL(__NR_quotactl_path, sys_quotactl_path)
+
+#define __NR_landlock_create_ruleset 444
+__SYSCALL(__NR_landlock_create_ruleset, sys_landlock_create_ruleset)
+#define __NR_landlock_add_rule 445
+__SYSCALL(__NR_landlock_add_rule, sys_landlock_add_rule)
+#define __NR_landlock_restrict_self 446
+__SYSCALL(__NR_landlock_restrict_self, sys_landlock_restrict_self)
#undef __NR_syscalls
-#define __NR_syscalls 443
+#define __NR_syscalls 447
/*
* 32 bit systems traditionally used different
__u64 size;
};
+/**
+ * DRM_CAP_DUMB_BUFFER
+ *
+ * If set to 1, the driver supports creating dumb buffers via the
+ * &DRM_IOCTL_MODE_CREATE_DUMB ioctl.
+ */
#define DRM_CAP_DUMB_BUFFER 0x1
+/**
+ * DRM_CAP_VBLANK_HIGH_CRTC
+ *
+ * If set to 1, the kernel supports specifying a CRTC index in the high bits of
+ * &drm_wait_vblank_request.type.
+ *
+ * Starting kernel version 2.6.39, this capability is always set to 1.
+ */
#define DRM_CAP_VBLANK_HIGH_CRTC 0x2
+/**
+ * DRM_CAP_DUMB_PREFERRED_DEPTH
+ *
+ * The preferred bit depth for dumb buffers.
+ *
+ * The bit depth is the number of bits used to indicate the color of a single
+ * pixel excluding any padding. This is different from the number of bits per
+ * pixel. For instance, XRGB8888 has a bit depth of 24 but has 32 bits per
+ * pixel.
+ *
+ * Note that this preference only applies to dumb buffers, it's irrelevant for
+ * other types of buffers.
+ */
#define DRM_CAP_DUMB_PREFERRED_DEPTH 0x3
+/**
+ * DRM_CAP_DUMB_PREFER_SHADOW
+ *
+ * If set to 1, the driver prefers userspace to render to a shadow buffer
+ * instead of directly rendering to a dumb buffer. For best speed, userspace
+ * should do streaming ordered memory copies into the dumb buffer and never
+ * read from it.
+ *
+ * Note that this preference only applies to dumb buffers, it's irrelevant for
+ * other types of buffers.
+ */
#define DRM_CAP_DUMB_PREFER_SHADOW 0x4
+/**
+ * DRM_CAP_PRIME
+ *
+ * Bitfield of supported PRIME sharing capabilities. See &DRM_PRIME_CAP_IMPORT
+ * and &DRM_PRIME_CAP_EXPORT.
+ *
+ * PRIME buffers are exposed as dma-buf file descriptors. See
+ * Documentation/gpu/drm-mm.rst, section "PRIME Buffer Sharing".
+ */
#define DRM_CAP_PRIME 0x5
+/**
+ * DRM_PRIME_CAP_IMPORT
+ *
+ * If this bit is set in &DRM_CAP_PRIME, the driver supports importing PRIME
+ * buffers via the &DRM_IOCTL_PRIME_FD_TO_HANDLE ioctl.
+ */
#define DRM_PRIME_CAP_IMPORT 0x1
+/**
+ * DRM_PRIME_CAP_EXPORT
+ *
+ * If this bit is set in &DRM_CAP_PRIME, the driver supports exporting PRIME
+ * buffers via the &DRM_IOCTL_PRIME_HANDLE_TO_FD ioctl.
+ */
#define DRM_PRIME_CAP_EXPORT 0x2
+/**
+ * DRM_CAP_TIMESTAMP_MONOTONIC
+ *
+ * If set to 0, the kernel will report timestamps with ``CLOCK_REALTIME`` in
+ * struct drm_event_vblank. If set to 1, the kernel will report timestamps with
+ * ``CLOCK_MONOTONIC``. See ``clock_gettime(2)`` for the definition of these
+ * clocks.
+ *
+ * Starting from kernel version 2.6.39, the default value for this capability
+ * is 1. Starting kernel version 4.15, this capability is always set to 1.
+ */
#define DRM_CAP_TIMESTAMP_MONOTONIC 0x6
+/**
+ * DRM_CAP_ASYNC_PAGE_FLIP
+ *
+ * If set to 1, the driver supports &DRM_MODE_PAGE_FLIP_ASYNC.
+ */
#define DRM_CAP_ASYNC_PAGE_FLIP 0x7
-/*
- * The CURSOR_WIDTH and CURSOR_HEIGHT capabilities return a valid widthxheight
- * combination for the hardware cursor. The intention is that a hardware
- * agnostic userspace can query a cursor plane size to use.
+/**
+ * DRM_CAP_CURSOR_WIDTH
+ *
+ * The ``CURSOR_WIDTH`` and ``CURSOR_HEIGHT`` capabilities return a valid
+ * width x height combination for the hardware cursor. The intention is that a
+ * hardware agnostic userspace can query a cursor plane size to use.
*
* Note that the cross-driver contract is to merely return a valid size;
* drivers are free to attach another meaning on top, eg. i915 returns the
* maximum plane size.
*/
#define DRM_CAP_CURSOR_WIDTH 0x8
+/**
+ * DRM_CAP_CURSOR_HEIGHT
+ *
+ * See &DRM_CAP_CURSOR_WIDTH.
+ */
#define DRM_CAP_CURSOR_HEIGHT 0x9
+/**
+ * DRM_CAP_ADDFB2_MODIFIERS
+ *
+ * If set to 1, the driver supports supplying modifiers in the
+ * &DRM_IOCTL_MODE_ADDFB2 ioctl.
+ */
#define DRM_CAP_ADDFB2_MODIFIERS 0x10
+/**
+ * DRM_CAP_PAGE_FLIP_TARGET
+ *
+ * If set to 1, the driver supports the &DRM_MODE_PAGE_FLIP_TARGET_ABSOLUTE and
+ * &DRM_MODE_PAGE_FLIP_TARGET_RELATIVE flags in
+ * &drm_mode_crtc_page_flip_target.flags for the &DRM_IOCTL_MODE_PAGE_FLIP
+ * ioctl.
+ */
#define DRM_CAP_PAGE_FLIP_TARGET 0x11
+/**
+ * DRM_CAP_CRTC_IN_VBLANK_EVENT
+ *
+ * If set to 1, the kernel supports reporting the CRTC ID in
+ * &drm_event_vblank.crtc_id for the &DRM_EVENT_VBLANK and
+ * &DRM_EVENT_FLIP_COMPLETE events.
+ *
+ * Starting kernel version 4.12, this capability is always set to 1.
+ */
#define DRM_CAP_CRTC_IN_VBLANK_EVENT 0x12
+/**
+ * DRM_CAP_SYNCOBJ
+ *
+ * If set to 1, the driver supports sync objects. See
+ * Documentation/gpu/drm-mm.rst, section "DRM Sync Objects".
+ */
#define DRM_CAP_SYNCOBJ 0x13
+/**
+ * DRM_CAP_SYNCOBJ_TIMELINE
+ *
+ * If set to 1, the driver supports timeline operations on sync objects. See
+ * Documentation/gpu/drm-mm.rst, section "DRM Sync Objects".
+ */
#define DRM_CAP_SYNCOBJ_TIMELINE 0x14
/* DRM_IOCTL_GET_CAP ioctl argument type */
__u64 offset;
};
+/* DRM_IOCTL_I915_GEM_EXECBUFFER was removed in Linux 5.13 */
struct drm_i915_gem_execbuffer {
/**
* List of buffers to be validated with their relocations to be
* Look up an element with the given *key* in the map referred to
* by the file descriptor *fd*, and if found, delete the element.
*
+ * For **BPF_MAP_TYPE_QUEUE** and **BPF_MAP_TYPE_STACK** map
+ * types, the *flags* argument needs to be set to 0, but for other
+ * map types, it may be specified as:
+ *
+ * **BPF_F_LOCK**
+ * Look up and delete the value of a spin-locked map
+ * without returning the lock. This must be specified if
+ * the elements contain a spinlock.
+ *
* The **BPF_MAP_TYPE_QUEUE** and **BPF_MAP_TYPE_STACK** map types
* implement this command as a "pop" operation, deleting the top
* element rather than one corresponding to *key*.
* This command is only valid for the following map types:
* * **BPF_MAP_TYPE_QUEUE**
* * **BPF_MAP_TYPE_STACK**
+ * * **BPF_MAP_TYPE_HASH**
+ * * **BPF_MAP_TYPE_PERCPU_HASH**
+ * * **BPF_MAP_TYPE_LRU_HASH**
+ * * **BPF_MAP_TYPE_LRU_PERCPU_HASH**
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
BPF_PROG_ATTACH,
BPF_PROG_DETACH,
BPF_PROG_TEST_RUN,
+ BPF_PROG_RUN = BPF_PROG_TEST_RUN,
BPF_PROG_GET_NEXT_ID,
BPF_MAP_GET_NEXT_ID,
BPF_PROG_GET_FD_BY_ID,
BPF_PROG_TYPE_EXT,
BPF_PROG_TYPE_LSM,
BPF_PROG_TYPE_SK_LOOKUP,
+ BPF_PROG_TYPE_SYSCALL, /* a program that can execute syscalls */
};
enum bpf_attach_type {
BPF_SK_LOOKUP,
BPF_XDP,
BPF_SK_SKB_VERDICT,
+ BPF_SK_REUSEPORT_SELECT,
+ BPF_SK_REUSEPORT_SELECT_OR_MIGRATE,
__MAX_BPF_ATTACH_TYPE
};
/* When BPF ldimm64's insn[0].src_reg != 0 then this can have
* the following extensions:
*
- * insn[0].src_reg: BPF_PSEUDO_MAP_FD
- * insn[0].imm: map fd
+ * insn[0].src_reg: BPF_PSEUDO_MAP_[FD|IDX]
+ * insn[0].imm: map fd or fd_idx
* insn[1].imm: 0
* insn[0].off: 0
* insn[1].off: 0
* verifier type: CONST_PTR_TO_MAP
*/
#define BPF_PSEUDO_MAP_FD 1
-/* insn[0].src_reg: BPF_PSEUDO_MAP_VALUE
- * insn[0].imm: map fd
+#define BPF_PSEUDO_MAP_IDX 5
+
+/* insn[0].src_reg: BPF_PSEUDO_MAP_[IDX_]VALUE
+ * insn[0].imm: map fd or fd_idx
* insn[1].imm: offset into value
* insn[0].off: 0
* insn[1].off: 0
* ldimm64 rewrite: address of map[0]+offset
* verifier type: PTR_TO_MAP_VALUE
*/
-#define BPF_PSEUDO_MAP_VALUE 2
+#define BPF_PSEUDO_MAP_VALUE 2
+#define BPF_PSEUDO_MAP_IDX_VALUE 6
+
/* insn[0].src_reg: BPF_PSEUDO_BTF_ID
* insn[0].imm: kernel btd id of VAR
* insn[1].imm: 0
/* or valid module BTF object fd or 0 to attach to vmlinux */
__u32 attach_btf_obj_fd;
};
+ __u32 :32; /* pad */
+ __aligned_u64 fd_array; /* array of FDs */
};
struct { /* anonymous struct used by BPF_OBJ_* commands */
* The lower two bits of *flags* are used as the return code if
* the map lookup fails. This is so that the return value can be
* one of the XDP program return codes up to **XDP_TX**, as chosen
- * by the caller. Any higher bits in the *flags* argument must be
- * unset.
+ * by the caller. The higher bits of *flags* can be set to
+ * BPF_F_BROADCAST or BPF_F_EXCLUDE_INGRESS as defined below.
+ *
+ * With BPF_F_BROADCAST the packet will be broadcasted to all the
+ * interfaces in the map, with BPF_F_EXCLUDE_INGRESS the ingress
+ * interface will be excluded when do broadcasting.
*
* See also **bpf_redirect**\ (), which only supports redirecting
* to an ifindex, but doesn't require a map to do so.
* be zero-terminated except when **str_size** is 0.
*
* Or **-EBUSY** if the per-CPU memory copy buffer is busy.
+ *
+ * long bpf_sys_bpf(u32 cmd, void *attr, u32 attr_size)
+ * Description
+ * Execute bpf syscall with given arguments.
+ * Return
+ * A syscall result.
+ *
+ * long bpf_btf_find_by_name_kind(char *name, int name_sz, u32 kind, int flags)
+ * Description
+ * Find BTF type with given name and kind in vmlinux BTF or in module's BTFs.
+ * Return
+ * Returns btf_id and btf_obj_fd in lower and upper 32 bits.
+ *
+ * long bpf_sys_close(u32 fd)
+ * Description
+ * Execute close syscall for given FD.
+ * Return
+ * A syscall result.
*/
#define __BPF_FUNC_MAPPER(FN) \
FN(unspec), \
FN(check_mtu), \
FN(for_each_map_elem), \
FN(snprintf), \
+ FN(sys_bpf), \
+ FN(btf_find_by_name_kind), \
+ FN(sys_close), \
/* */
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
BPF_F_BPRM_SECUREEXEC = (1ULL << 0),
};
+/* Flags for bpf_redirect_map helper */
+enum {
+ BPF_F_BROADCAST = (1ULL << 3),
+ BPF_F_EXCLUDE_INGRESS = (1ULL << 4),
+};
+
#define __bpf_md_ptr(type, name) \
union { \
type name; \
__u32 ip_protocol; /* IP protocol. e.g. IPPROTO_TCP, IPPROTO_UDP */
__u32 bind_inany; /* Is sock bound to an INANY address? */
__u32 hash; /* A hash of the packet 4 tuples */
+ /* When reuse->migrating_sk is NULL, it is selecting a sk for the
+ * new incoming connection request (e.g. selecting a listen sk for
+ * the received SYN in the TCP case). reuse->sk is one of the sk
+ * in the reuseport group. The bpf prog can use reuse->sk to learn
+ * the local listening ip/port without looking into the skb.
+ *
+ * When reuse->migrating_sk is not NULL, reuse->sk is closed and
+ * reuse->migrating_sk is the socket that needs to be migrated
+ * to another listening socket. migrating_sk could be a fullsock
+ * sk that is fully established or a reqsk that is in-the-middle
+ * of 3-way handshake.
+ */
+ __bpf_md_ptr(struct bpf_sock *, sk);
+ __bpf_md_ptr(struct bpf_sock *, migrating_sk);
};
#define BPF_TAG_SIZE 8
#define BLKROTATIONAL _IO(0x12,126)
#define BLKZEROOUT _IO(0x12,127)
/*
- * A jump here: 130-131 are reserved for zoned block devices
+ * A jump here: 130-136 are reserved for zoned block devices
* (see uapi/linux/blkzoned.h)
*/
* Note: you must update KVM_API_VERSION if you change this interface.
*/
+#include <linux/const.h>
#include <linux/types.h>
#include <linux/compiler.h>
#include <linux/ioctl.h>
#define KVM_CAP_DIRTY_LOG_RING 192
#define KVM_CAP_X86_BUS_LOCK_EXIT 193
#define KVM_CAP_PPC_DAWR1 194
+#define KVM_CAP_SET_GUEST_DEBUG2 195
+#define KVM_CAP_SGX_ATTRIBUTE 196
+#define KVM_CAP_VM_COPY_ENC_CONTEXT_FROM 197
+#define KVM_CAP_PTP_KVM 198
#ifdef KVM_CAP_IRQ_ROUTING
KVM_SEV_CERT_EXPORT,
/* Attestation report */
KVM_SEV_GET_ATTESTATION_REPORT,
+ /* Guest Migration Extension */
+ KVM_SEV_SEND_CANCEL,
KVM_SEV_NR_MAX,
};
__u32 len;
};
+struct kvm_sev_send_start {
+ __u32 policy;
+ __u64 pdh_cert_uaddr;
+ __u32 pdh_cert_len;
+ __u64 plat_certs_uaddr;
+ __u32 plat_certs_len;
+ __u64 amd_certs_uaddr;
+ __u32 amd_certs_len;
+ __u64 session_uaddr;
+ __u32 session_len;
+};
+
+struct kvm_sev_send_update_data {
+ __u64 hdr_uaddr;
+ __u32 hdr_len;
+ __u64 guest_uaddr;
+ __u32 guest_len;
+ __u64 trans_uaddr;
+ __u32 trans_len;
+};
+
+struct kvm_sev_receive_start {
+ __u32 handle;
+ __u32 policy;
+ __u64 pdh_uaddr;
+ __u32 pdh_len;
+ __u64 session_uaddr;
+ __u32 session_len;
+};
+
+struct kvm_sev_receive_update_data {
+ __u64 hdr_uaddr;
+ __u32 hdr_len;
+ __u64 guest_uaddr;
+ __u32 guest_len;
+ __u64 trans_uaddr;
+ __u32 trans_len;
+};
+
#define KVM_DEV_ASSIGN_ENABLE_IOMMU (1 << 0)
#define KVM_DEV_ASSIGN_PCI_2_3 (1 << 1)
#define KVM_DEV_ASSIGN_MASK_INTX (1 << 2)
* conversion after harvesting an entry. Also, it must not skip any
* dirty bits, so that dirty bits are always harvested in sequence.
*/
-#define KVM_DIRTY_GFN_F_DIRTY BIT(0)
-#define KVM_DIRTY_GFN_F_RESET BIT(1)
+#define KVM_DIRTY_GFN_F_DIRTY _BITUL(0)
+#define KVM_DIRTY_GFN_F_RESET _BITUL(1)
#define KVM_DIRTY_GFN_F_MASK 0x3
/*
PERF_COUNT_SW_EMULATION_FAULTS = 8,
PERF_COUNT_SW_DUMMY = 9,
PERF_COUNT_SW_BPF_OUTPUT = 10,
+ PERF_COUNT_SW_CGROUP_SWITCHES = 11,
PERF_COUNT_SW_MAX, /* non-ABI */
};
#define PERF_ATTR_SIZE_VER4 104 /* add: sample_regs_intr */
#define PERF_ATTR_SIZE_VER5 112 /* add: aux_watermark */
#define PERF_ATTR_SIZE_VER6 120 /* add: aux_sample_size */
+#define PERF_ATTR_SIZE_VER7 128 /* add: sig_data */
/*
* Hardware event_id to monitor via a performance monitoring event:
cgroup : 1, /* include cgroup events */
text_poke : 1, /* include text poke events */
build_id : 1, /* use build id in mmap2 events */
- __reserved_1 : 29;
+ inherit_thread : 1, /* children only inherit if cloned with CLONE_THREAD */
+ remove_on_exec : 1, /* event is removed from task on exec */
+ sigtrap : 1, /* send synchronous SIGTRAP on event */
+ __reserved_1 : 26;
union {
__u32 wakeup_events; /* wakeup every n events */
__u16 __reserved_2;
__u32 aux_sample_size;
__u32 __reserved_3;
+
+ /*
+ * User provided data if sigtrap=1, passed back to user via
+ * siginfo_t::si_perf_data, e.g. to permit user to identify the event.
+ */
+ __u64 sig_data;
};
/*
/**
* PERF_RECORD_AUX::flags bits
*/
-#define PERF_AUX_FLAG_TRUNCATED 0x01 /* record was truncated to fit */
-#define PERF_AUX_FLAG_OVERWRITE 0x02 /* snapshot from overwrite mode */
-#define PERF_AUX_FLAG_PARTIAL 0x04 /* record contains gaps */
-#define PERF_AUX_FLAG_COLLISION 0x08 /* sample collided with another */
+#define PERF_AUX_FLAG_TRUNCATED 0x01 /* record was truncated to fit */
+#define PERF_AUX_FLAG_OVERWRITE 0x02 /* snapshot from overwrite mode */
+#define PERF_AUX_FLAG_PARTIAL 0x04 /* record contains gaps */
+#define PERF_AUX_FLAG_COLLISION 0x08 /* sample collided with another */
+#define PERF_AUX_FLAG_PMU_FORMAT_TYPE_MASK 0xff00 /* PMU specific trace format type */
+
+/* CoreSight PMU AUX buffer formats */
+#define PERF_AUX_FLAG_CORESIGHT_FORMAT_CORESIGHT 0x0000 /* Default for backward compatibility */
+#define PERF_AUX_FLAG_CORESIGHT_FORMAT_RAW 0x0100 /* Raw format of the source */
#define PERF_FLAG_FD_NO_GROUP (1UL << 0)
#define PERF_FLAG_FD_OUTPUT (1UL << 1)
# define SYSCALL_DISPATCH_FILTER_ALLOW 0
# define SYSCALL_DISPATCH_FILTER_BLOCK 1
+/* Set/get enabled arm64 pointer authentication keys */
+#define PR_PAC_SET_ENABLED_KEYS 60
+#define PR_PAC_GET_ENABLED_KEYS 61
+
#endif /* _LINUX_PRCTL_H */
--tracepoints::
retrieve statistics from tracepoints
-*z*::
+-z::
--skip-zero-records::
omit records with all zeros in logging mode
libbpf-y := libbpf.o bpf.o nlattr.o btf.o libbpf_errno.o str_error.o \
netlink.o bpf_prog_linfo.o libbpf_probes.o xsk.o hashmap.o \
- btf_dump.o ringbuf.o strset.o linker.o
+ btf_dump.o ringbuf.o strset.o linker.o gen_loader.o
$(call do_install_mkdir,$(libdir_SQ)); \
cp -fpR $(LIB_FILE) $(DESTDIR)$(libdir_SQ)
+INSTALL_HEADERS = bpf.h libbpf.h btf.h libbpf_common.h libbpf_legacy.h xsk.h \
+ bpf_helpers.h $(BPF_HELPER_DEFS) bpf_tracing.h \
+ bpf_endian.h bpf_core_read.h skel_internal.h
+
install_headers: $(BPF_HELPER_DEFS)
- $(call QUIET_INSTALL, headers) \
- $(call do_install,bpf.h,$(prefix)/include/bpf,644); \
- $(call do_install,libbpf.h,$(prefix)/include/bpf,644); \
- $(call do_install,btf.h,$(prefix)/include/bpf,644); \
- $(call do_install,libbpf_common.h,$(prefix)/include/bpf,644); \
- $(call do_install,xsk.h,$(prefix)/include/bpf,644); \
- $(call do_install,bpf_helpers.h,$(prefix)/include/bpf,644); \
- $(call do_install,$(BPF_HELPER_DEFS),$(prefix)/include/bpf,644); \
- $(call do_install,bpf_tracing.h,$(prefix)/include/bpf,644); \
- $(call do_install,bpf_endian.h,$(prefix)/include/bpf,644); \
- $(call do_install,bpf_core_read.h,$(prefix)/include/bpf,644);
+ $(call QUIET_INSTALL, headers) \
+ $(foreach hdr,$(INSTALL_HEADERS), \
+ $(call do_install,$(hdr),$(prefix)/include/bpf,644);)
install_pkgconfig: $(PC_FILE)
$(call QUIET_INSTALL, $(PC_FILE)) \
int bpf_create_map_xattr(const struct bpf_create_map_attr *create_attr)
{
union bpf_attr attr;
+ int fd;
memset(&attr, '\0', sizeof(attr));
else
attr.inner_map_fd = create_attr->inner_map_fd;
- return sys_bpf(BPF_MAP_CREATE, &attr, sizeof(attr));
+ fd = sys_bpf(BPF_MAP_CREATE, &attr, sizeof(attr));
+ return libbpf_err_errno(fd);
}
int bpf_create_map_node(enum bpf_map_type map_type, const char *name,
__u32 map_flags, int node)
{
union bpf_attr attr;
+ int fd;
memset(&attr, '\0', sizeof(attr));
attr.numa_node = node;
}
- return sys_bpf(BPF_MAP_CREATE, &attr, sizeof(attr));
+ fd = sys_bpf(BPF_MAP_CREATE, &attr, sizeof(attr));
+ return libbpf_err_errno(fd);
}
int bpf_create_map_in_map(enum bpf_map_type map_type, const char *name,
int fd;
if (!load_attr->log_buf != !load_attr->log_buf_sz)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (load_attr->log_level > (4 | 2 | 1) || (load_attr->log_level && !load_attr->log_buf))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
memset(&attr, 0, sizeof(attr));
attr.prog_type = load_attr->prog_type;
load_attr->func_info_cnt,
load_attr->func_info_rec_size,
attr.func_info_rec_size);
- if (!finfo)
+ if (!finfo) {
+ errno = E2BIG;
goto done;
+ }
attr.func_info = ptr_to_u64(finfo);
attr.func_info_rec_size = load_attr->func_info_rec_size;
load_attr->line_info_cnt,
load_attr->line_info_rec_size,
attr.line_info_rec_size);
- if (!linfo)
+ if (!linfo) {
+ errno = E2BIG;
goto done;
+ }
attr.line_info = ptr_to_u64(linfo);
attr.line_info_rec_size = load_attr->line_info_rec_size;
fd = sys_bpf_prog_load(&attr, sizeof(attr));
done:
+ /* free() doesn't affect errno, so we don't need to restore it */
free(finfo);
free(linfo);
- return fd;
+ return libbpf_err_errno(fd);
}
int bpf_load_program_xattr(const struct bpf_load_program_attr *load_attr,
struct bpf_prog_load_params p = {};
if (!load_attr || !log_buf != !log_buf_sz)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
p.prog_type = load_attr->prog_type;
p.expected_attach_type = load_attr->expected_attach_type;
int log_level)
{
union bpf_attr attr;
+ int fd;
memset(&attr, 0, sizeof(attr));
attr.prog_type = type;
attr.kern_version = kern_version;
attr.prog_flags = prog_flags;
- return sys_bpf_prog_load(&attr, sizeof(attr));
+ fd = sys_bpf_prog_load(&attr, sizeof(attr));
+ return libbpf_err_errno(fd);
}
int bpf_map_update_elem(int fd, const void *key, const void *value,
__u64 flags)
{
union bpf_attr attr;
+ int ret;
memset(&attr, 0, sizeof(attr));
attr.map_fd = fd;
attr.value = ptr_to_u64(value);
attr.flags = flags;
- return sys_bpf(BPF_MAP_UPDATE_ELEM, &attr, sizeof(attr));
+ ret = sys_bpf(BPF_MAP_UPDATE_ELEM, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
}
int bpf_map_lookup_elem(int fd, const void *key, void *value)
{
union bpf_attr attr;
+ int ret;
memset(&attr, 0, sizeof(attr));
attr.map_fd = fd;
attr.key = ptr_to_u64(key);
attr.value = ptr_to_u64(value);
- return sys_bpf(BPF_MAP_LOOKUP_ELEM, &attr, sizeof(attr));
+ ret = sys_bpf(BPF_MAP_LOOKUP_ELEM, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
}
int bpf_map_lookup_elem_flags(int fd, const void *key, void *value, __u64 flags)
{
union bpf_attr attr;
+ int ret;
memset(&attr, 0, sizeof(attr));
attr.map_fd = fd;
attr.value = ptr_to_u64(value);
attr.flags = flags;
- return sys_bpf(BPF_MAP_LOOKUP_ELEM, &attr, sizeof(attr));
+ ret = sys_bpf(BPF_MAP_LOOKUP_ELEM, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
}
int bpf_map_lookup_and_delete_elem(int fd, const void *key, void *value)
{
union bpf_attr attr;
+ int ret;
+
+ memset(&attr, 0, sizeof(attr));
+ attr.map_fd = fd;
+ attr.key = ptr_to_u64(key);
+ attr.value = ptr_to_u64(value);
+
+ ret = sys_bpf(BPF_MAP_LOOKUP_AND_DELETE_ELEM, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
+}
+
+int bpf_map_lookup_and_delete_elem_flags(int fd, const void *key, void *value, __u64 flags)
+{
+ union bpf_attr attr;
memset(&attr, 0, sizeof(attr));
attr.map_fd = fd;
attr.key = ptr_to_u64(key);
attr.value = ptr_to_u64(value);
+ attr.flags = flags;
return sys_bpf(BPF_MAP_LOOKUP_AND_DELETE_ELEM, &attr, sizeof(attr));
}
int bpf_map_delete_elem(int fd, const void *key)
{
union bpf_attr attr;
+ int ret;
memset(&attr, 0, sizeof(attr));
attr.map_fd = fd;
attr.key = ptr_to_u64(key);
- return sys_bpf(BPF_MAP_DELETE_ELEM, &attr, sizeof(attr));
+ ret = sys_bpf(BPF_MAP_DELETE_ELEM, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
}
int bpf_map_get_next_key(int fd, const void *key, void *next_key)
{
union bpf_attr attr;
+ int ret;
memset(&attr, 0, sizeof(attr));
attr.map_fd = fd;
attr.key = ptr_to_u64(key);
attr.next_key = ptr_to_u64(next_key);
- return sys_bpf(BPF_MAP_GET_NEXT_KEY, &attr, sizeof(attr));
+ ret = sys_bpf(BPF_MAP_GET_NEXT_KEY, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
}
int bpf_map_freeze(int fd)
{
union bpf_attr attr;
+ int ret;
memset(&attr, 0, sizeof(attr));
attr.map_fd = fd;
- return sys_bpf(BPF_MAP_FREEZE, &attr, sizeof(attr));
+ ret = sys_bpf(BPF_MAP_FREEZE, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
}
static int bpf_map_batch_common(int cmd, int fd, void *in_batch,
int ret;
if (!OPTS_VALID(opts, bpf_map_batch_opts))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
memset(&attr, 0, sizeof(attr));
attr.batch.map_fd = fd;
ret = sys_bpf(cmd, &attr, sizeof(attr));
*count = attr.batch.count;
- return ret;
+ return libbpf_err_errno(ret);
}
int bpf_map_delete_batch(int fd, void *keys, __u32 *count,
int bpf_obj_pin(int fd, const char *pathname)
{
union bpf_attr attr;
+ int ret;
memset(&attr, 0, sizeof(attr));
attr.pathname = ptr_to_u64((void *)pathname);
attr.bpf_fd = fd;
- return sys_bpf(BPF_OBJ_PIN, &attr, sizeof(attr));
+ ret = sys_bpf(BPF_OBJ_PIN, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
}
int bpf_obj_get(const char *pathname)
{
union bpf_attr attr;
+ int fd;
memset(&attr, 0, sizeof(attr));
attr.pathname = ptr_to_u64((void *)pathname);
- return sys_bpf(BPF_OBJ_GET, &attr, sizeof(attr));
+ fd = sys_bpf(BPF_OBJ_GET, &attr, sizeof(attr));
+ return libbpf_err_errno(fd);
}
int bpf_prog_attach(int prog_fd, int target_fd, enum bpf_attach_type type,
const struct bpf_prog_attach_opts *opts)
{
union bpf_attr attr;
+ int ret;
if (!OPTS_VALID(opts, bpf_prog_attach_opts))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
memset(&attr, 0, sizeof(attr));
attr.target_fd = target_fd;
attr.attach_flags = OPTS_GET(opts, flags, 0);
attr.replace_bpf_fd = OPTS_GET(opts, replace_prog_fd, 0);
- return sys_bpf(BPF_PROG_ATTACH, &attr, sizeof(attr));
+ ret = sys_bpf(BPF_PROG_ATTACH, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
}
int bpf_prog_detach(int target_fd, enum bpf_attach_type type)
{
union bpf_attr attr;
+ int ret;
memset(&attr, 0, sizeof(attr));
attr.target_fd = target_fd;
attr.attach_type = type;
- return sys_bpf(BPF_PROG_DETACH, &attr, sizeof(attr));
+ ret = sys_bpf(BPF_PROG_DETACH, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
}
int bpf_prog_detach2(int prog_fd, int target_fd, enum bpf_attach_type type)
{
union bpf_attr attr;
+ int ret;
memset(&attr, 0, sizeof(attr));
attr.target_fd = target_fd;
attr.attach_bpf_fd = prog_fd;
attr.attach_type = type;
- return sys_bpf(BPF_PROG_DETACH, &attr, sizeof(attr));
+ ret = sys_bpf(BPF_PROG_DETACH, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
}
int bpf_link_create(int prog_fd, int target_fd,
{
__u32 target_btf_id, iter_info_len;
union bpf_attr attr;
+ int fd;
if (!OPTS_VALID(opts, bpf_link_create_opts))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
iter_info_len = OPTS_GET(opts, iter_info_len, 0);
target_btf_id = OPTS_GET(opts, target_btf_id, 0);
if (iter_info_len && target_btf_id)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
memset(&attr, 0, sizeof(attr));
attr.link_create.prog_fd = prog_fd;
attr.link_create.target_btf_id = target_btf_id;
}
- return sys_bpf(BPF_LINK_CREATE, &attr, sizeof(attr));
+ fd = sys_bpf(BPF_LINK_CREATE, &attr, sizeof(attr));
+ return libbpf_err_errno(fd);
}
int bpf_link_detach(int link_fd)
{
union bpf_attr attr;
+ int ret;
memset(&attr, 0, sizeof(attr));
attr.link_detach.link_fd = link_fd;
- return sys_bpf(BPF_LINK_DETACH, &attr, sizeof(attr));
+ ret = sys_bpf(BPF_LINK_DETACH, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
}
int bpf_link_update(int link_fd, int new_prog_fd,
const struct bpf_link_update_opts *opts)
{
union bpf_attr attr;
+ int ret;
if (!OPTS_VALID(opts, bpf_link_update_opts))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
memset(&attr, 0, sizeof(attr));
attr.link_update.link_fd = link_fd;
attr.link_update.flags = OPTS_GET(opts, flags, 0);
attr.link_update.old_prog_fd = OPTS_GET(opts, old_prog_fd, 0);
- return sys_bpf(BPF_LINK_UPDATE, &attr, sizeof(attr));
+ ret = sys_bpf(BPF_LINK_UPDATE, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
}
int bpf_iter_create(int link_fd)
{
union bpf_attr attr;
+ int fd;
memset(&attr, 0, sizeof(attr));
attr.iter_create.link_fd = link_fd;
- return sys_bpf(BPF_ITER_CREATE, &attr, sizeof(attr));
+ fd = sys_bpf(BPF_ITER_CREATE, &attr, sizeof(attr));
+ return libbpf_err_errno(fd);
}
int bpf_prog_query(int target_fd, enum bpf_attach_type type, __u32 query_flags,
attr.query.prog_ids = ptr_to_u64(prog_ids);
ret = sys_bpf(BPF_PROG_QUERY, &attr, sizeof(attr));
+
if (attach_flags)
*attach_flags = attr.query.attach_flags;
*prog_cnt = attr.query.prog_cnt;
- return ret;
+
+ return libbpf_err_errno(ret);
}
int bpf_prog_test_run(int prog_fd, int repeat, void *data, __u32 size,
attr.test.repeat = repeat;
ret = sys_bpf(BPF_PROG_TEST_RUN, &attr, sizeof(attr));
+
if (size_out)
*size_out = attr.test.data_size_out;
if (retval)
*retval = attr.test.retval;
if (duration)
*duration = attr.test.duration;
- return ret;
+
+ return libbpf_err_errno(ret);
}
int bpf_prog_test_run_xattr(struct bpf_prog_test_run_attr *test_attr)
int ret;
if (!test_attr->data_out && test_attr->data_size_out > 0)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
memset(&attr, 0, sizeof(attr));
attr.test.prog_fd = test_attr->prog_fd;
attr.test.repeat = test_attr->repeat;
ret = sys_bpf(BPF_PROG_TEST_RUN, &attr, sizeof(attr));
+
test_attr->data_size_out = attr.test.data_size_out;
test_attr->ctx_size_out = attr.test.ctx_size_out;
test_attr->retval = attr.test.retval;
test_attr->duration = attr.test.duration;
- return ret;
+
+ return libbpf_err_errno(ret);
}
int bpf_prog_test_run_opts(int prog_fd, struct bpf_test_run_opts *opts)
int ret;
if (!OPTS_VALID(opts, bpf_test_run_opts))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
memset(&attr, 0, sizeof(attr));
attr.test.prog_fd = prog_fd;
attr.test.data_out = ptr_to_u64(OPTS_GET(opts, data_out, NULL));
ret = sys_bpf(BPF_PROG_TEST_RUN, &attr, sizeof(attr));
+
OPTS_SET(opts, data_size_out, attr.test.data_size_out);
OPTS_SET(opts, ctx_size_out, attr.test.ctx_size_out);
OPTS_SET(opts, duration, attr.test.duration);
OPTS_SET(opts, retval, attr.test.retval);
- return ret;
+
+ return libbpf_err_errno(ret);
}
static int bpf_obj_get_next_id(__u32 start_id, __u32 *next_id, int cmd)
if (!err)
*next_id = attr.next_id;
- return err;
+ return libbpf_err_errno(err);
}
int bpf_prog_get_next_id(__u32 start_id, __u32 *next_id)
int bpf_prog_get_fd_by_id(__u32 id)
{
union bpf_attr attr;
+ int fd;
memset(&attr, 0, sizeof(attr));
attr.prog_id = id;
- return sys_bpf(BPF_PROG_GET_FD_BY_ID, &attr, sizeof(attr));
+ fd = sys_bpf(BPF_PROG_GET_FD_BY_ID, &attr, sizeof(attr));
+ return libbpf_err_errno(fd);
}
int bpf_map_get_fd_by_id(__u32 id)
{
union bpf_attr attr;
+ int fd;
memset(&attr, 0, sizeof(attr));
attr.map_id = id;
- return sys_bpf(BPF_MAP_GET_FD_BY_ID, &attr, sizeof(attr));
+ fd = sys_bpf(BPF_MAP_GET_FD_BY_ID, &attr, sizeof(attr));
+ return libbpf_err_errno(fd);
}
int bpf_btf_get_fd_by_id(__u32 id)
{
union bpf_attr attr;
+ int fd;
memset(&attr, 0, sizeof(attr));
attr.btf_id = id;
- return sys_bpf(BPF_BTF_GET_FD_BY_ID, &attr, sizeof(attr));
+ fd = sys_bpf(BPF_BTF_GET_FD_BY_ID, &attr, sizeof(attr));
+ return libbpf_err_errno(fd);
}
int bpf_link_get_fd_by_id(__u32 id)
{
union bpf_attr attr;
+ int fd;
memset(&attr, 0, sizeof(attr));
attr.link_id = id;
- return sys_bpf(BPF_LINK_GET_FD_BY_ID, &attr, sizeof(attr));
+ fd = sys_bpf(BPF_LINK_GET_FD_BY_ID, &attr, sizeof(attr));
+ return libbpf_err_errno(fd);
}
int bpf_obj_get_info_by_fd(int bpf_fd, void *info, __u32 *info_len)
attr.info.info = ptr_to_u64(info);
err = sys_bpf(BPF_OBJ_GET_INFO_BY_FD, &attr, sizeof(attr));
+
if (!err)
*info_len = attr.info.info_len;
- return err;
+ return libbpf_err_errno(err);
}
int bpf_raw_tracepoint_open(const char *name, int prog_fd)
{
union bpf_attr attr;
+ int fd;
memset(&attr, 0, sizeof(attr));
attr.raw_tracepoint.name = ptr_to_u64(name);
attr.raw_tracepoint.prog_fd = prog_fd;
- return sys_bpf(BPF_RAW_TRACEPOINT_OPEN, &attr, sizeof(attr));
+ fd = sys_bpf(BPF_RAW_TRACEPOINT_OPEN, &attr, sizeof(attr));
+ return libbpf_err_errno(fd);
}
int bpf_load_btf(const void *btf, __u32 btf_size, char *log_buf, __u32 log_buf_size,
}
fd = sys_bpf(BPF_BTF_LOAD, &attr, sizeof(attr));
- if (fd == -1 && !do_log && log_buf && log_buf_size) {
+
+ if (fd < 0 && !do_log && log_buf && log_buf_size) {
do_log = true;
goto retry;
}
- return fd;
+ return libbpf_err_errno(fd);
}
int bpf_task_fd_query(int pid, int fd, __u32 flags, char *buf, __u32 *buf_len,
attr.task_fd_query.buf_len = *buf_len;
err = sys_bpf(BPF_TASK_FD_QUERY, &attr, sizeof(attr));
+
*buf_len = attr.task_fd_query.buf_len;
*prog_id = attr.task_fd_query.prog_id;
*fd_type = attr.task_fd_query.fd_type;
*probe_offset = attr.task_fd_query.probe_offset;
*probe_addr = attr.task_fd_query.probe_addr;
- return err;
+ return libbpf_err_errno(err);
}
int bpf_enable_stats(enum bpf_stats_type type)
{
union bpf_attr attr;
+ int fd;
memset(&attr, 0, sizeof(attr));
attr.enable_stats.type = type;
- return sys_bpf(BPF_ENABLE_STATS, &attr, sizeof(attr));
+ fd = sys_bpf(BPF_ENABLE_STATS, &attr, sizeof(attr));
+ return libbpf_err_errno(fd);
}
int bpf_prog_bind_map(int prog_fd, int map_fd,
const struct bpf_prog_bind_opts *opts)
{
union bpf_attr attr;
+ int ret;
if (!OPTS_VALID(opts, bpf_prog_bind_opts))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
memset(&attr, 0, sizeof(attr));
attr.prog_bind_map.prog_fd = prog_fd;
attr.prog_bind_map.map_fd = map_fd;
attr.prog_bind_map.flags = OPTS_GET(opts, flags, 0);
- return sys_bpf(BPF_PROG_BIND_MAP, &attr, sizeof(attr));
+ ret = sys_bpf(BPF_PROG_BIND_MAP, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
}
__u64 flags);
LIBBPF_API int bpf_map_lookup_and_delete_elem(int fd, const void *key,
void *value);
+LIBBPF_API int bpf_map_lookup_and_delete_elem_flags(int fd, const void *key,
+ void *value, __u64 flags);
LIBBPF_API int bpf_map_delete_elem(int fd, const void *key);
LIBBPF_API int bpf_map_get_next_key(int fd, const void *key, void *next_key);
LIBBPF_API int bpf_map_freeze(int fd);
--- /dev/null
+/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
+/* Copyright (c) 2021 Facebook */
+#ifndef __BPF_GEN_INTERNAL_H
+#define __BPF_GEN_INTERNAL_H
+
+struct ksym_relo_desc {
+ const char *name;
+ int kind;
+ int insn_idx;
+};
+
+struct bpf_gen {
+ struct gen_loader_opts *opts;
+ void *data_start;
+ void *data_cur;
+ void *insn_start;
+ void *insn_cur;
+ ssize_t cleanup_label;
+ __u32 nr_progs;
+ __u32 nr_maps;
+ int log_level;
+ int error;
+ struct ksym_relo_desc *relos;
+ int relo_cnt;
+ char attach_target[128];
+ int attach_kind;
+};
+
+void bpf_gen__init(struct bpf_gen *gen, int log_level);
+int bpf_gen__finish(struct bpf_gen *gen);
+void bpf_gen__free(struct bpf_gen *gen);
+void bpf_gen__load_btf(struct bpf_gen *gen, const void *raw_data, __u32 raw_size);
+void bpf_gen__map_create(struct bpf_gen *gen, struct bpf_create_map_attr *map_attr, int map_idx);
+struct bpf_prog_load_params;
+void bpf_gen__prog_load(struct bpf_gen *gen, struct bpf_prog_load_params *load_attr, int prog_idx);
+void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size);
+void bpf_gen__map_freeze(struct bpf_gen *gen, int map_idx);
+void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *name, enum bpf_attach_type type);
+void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, int kind, int insn_idx);
+
+#endif
#define __kconfig __attribute__((section(".kconfig")))
#define __ksym __attribute__((section(".ksyms")))
+#ifndef ___bpf_concat
+#define ___bpf_concat(a, b) a ## b
+#endif
+#ifndef ___bpf_apply
+#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
+#endif
+#ifndef ___bpf_nth
+#define ___bpf_nth(_, _1, _2, _3, _4, _5, _6, _7, _8, _9, _a, _b, _c, N, ...) N
+#endif
+#ifndef ___bpf_narg
+#define ___bpf_narg(...) \
+ ___bpf_nth(_, ##__VA_ARGS__, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0)
+#endif
+
+#define ___bpf_fill0(arr, p, x) do {} while (0)
+#define ___bpf_fill1(arr, p, x) arr[p] = x
+#define ___bpf_fill2(arr, p, x, args...) arr[p] = x; ___bpf_fill1(arr, p + 1, args)
+#define ___bpf_fill3(arr, p, x, args...) arr[p] = x; ___bpf_fill2(arr, p + 1, args)
+#define ___bpf_fill4(arr, p, x, args...) arr[p] = x; ___bpf_fill3(arr, p + 1, args)
+#define ___bpf_fill5(arr, p, x, args...) arr[p] = x; ___bpf_fill4(arr, p + 1, args)
+#define ___bpf_fill6(arr, p, x, args...) arr[p] = x; ___bpf_fill5(arr, p + 1, args)
+#define ___bpf_fill7(arr, p, x, args...) arr[p] = x; ___bpf_fill6(arr, p + 1, args)
+#define ___bpf_fill8(arr, p, x, args...) arr[p] = x; ___bpf_fill7(arr, p + 1, args)
+#define ___bpf_fill9(arr, p, x, args...) arr[p] = x; ___bpf_fill8(arr, p + 1, args)
+#define ___bpf_fill10(arr, p, x, args...) arr[p] = x; ___bpf_fill9(arr, p + 1, args)
+#define ___bpf_fill11(arr, p, x, args...) arr[p] = x; ___bpf_fill10(arr, p + 1, args)
+#define ___bpf_fill12(arr, p, x, args...) arr[p] = x; ___bpf_fill11(arr, p + 1, args)
+#define ___bpf_fill(arr, args...) \
+ ___bpf_apply(___bpf_fill, ___bpf_narg(args))(arr, 0, args)
+
+/*
+ * BPF_SEQ_PRINTF to wrap bpf_seq_printf to-be-printed values
+ * in a structure.
+ */
+#define BPF_SEQ_PRINTF(seq, fmt, args...) \
+({ \
+ static const char ___fmt[] = fmt; \
+ unsigned long long ___param[___bpf_narg(args)]; \
+ \
+ _Pragma("GCC diagnostic push") \
+ _Pragma("GCC diagnostic ignored \"-Wint-conversion\"") \
+ ___bpf_fill(___param, args); \
+ _Pragma("GCC diagnostic pop") \
+ \
+ bpf_seq_printf(seq, ___fmt, sizeof(___fmt), \
+ ___param, sizeof(___param)); \
+})
+
+/*
+ * BPF_SNPRINTF wraps the bpf_snprintf helper with variadic arguments instead of
+ * an array of u64.
+ */
+#define BPF_SNPRINTF(out, out_size, fmt, args...) \
+({ \
+ static const char ___fmt[] = fmt; \
+ unsigned long long ___param[___bpf_narg(args)]; \
+ \
+ _Pragma("GCC diagnostic push") \
+ _Pragma("GCC diagnostic ignored \"-Wint-conversion\"") \
+ ___bpf_fill(___param, args); \
+ _Pragma("GCC diagnostic pop") \
+ \
+ bpf_snprintf(out, out_size, ___fmt, \
+ ___param, sizeof(___param)); \
+})
+
#endif
nr_linfo = info->nr_line_info;
if (!nr_linfo)
- return NULL;
+ return errno = EINVAL, NULL;
/*
* The min size that bpf_prog_linfo has to access for
*/
if (info->line_info_rec_size <
offsetof(struct bpf_line_info, file_name_off))
- return NULL;
+ return errno = EINVAL, NULL;
prog_linfo = calloc(1, sizeof(*prog_linfo));
if (!prog_linfo)
- return NULL;
+ return errno = ENOMEM, NULL;
/* Copy xlated line_info */
prog_linfo->nr_linfo = nr_linfo;
err_free:
bpf_prog_linfo__free(prog_linfo);
- return NULL;
+ return errno = EINVAL, NULL;
}
const struct bpf_line_info *
const __u64 *jited_linfo;
if (func_idx >= prog_linfo->nr_jited_func)
- return NULL;
+ return errno = ENOENT, NULL;
nr_linfo = prog_linfo->nr_jited_linfo_per_func[func_idx];
if (nr_skip >= nr_linfo)
- return NULL;
+ return errno = ENOENT, NULL;
start = prog_linfo->jited_linfo_func_idx[func_idx] + nr_skip;
jited_rec_size = prog_linfo->jited_rec_size;
(start * jited_rec_size);
jited_linfo = raw_jited_linfo;
if (addr < *jited_linfo)
- return NULL;
+ return errno = ENOENT, NULL;
nr_linfo -= nr_skip;
rec_size = prog_linfo->rec_size;
nr_linfo = prog_linfo->nr_linfo;
if (nr_skip >= nr_linfo)
- return NULL;
+ return errno = ENOENT, NULL;
rec_size = prog_linfo->rec_size;
raw_linfo = prog_linfo->raw_linfo + (nr_skip * rec_size);
linfo = raw_linfo;
if (insn_off < linfo->insn_off)
- return NULL;
+ return errno = ENOENT, NULL;
nr_linfo -= nr_skip;
for (i = 0; i < nr_linfo; i++) {
#define bpf_target_sparc
#define bpf_target_defined
#else
- #undef bpf_target_defined
-#endif
/* Fall back to what the compiler says */
-#ifndef bpf_target_defined
#if defined(__x86_64__)
#define bpf_target_x86
+ #define bpf_target_defined
#elif defined(__s390__)
#define bpf_target_s390
+ #define bpf_target_defined
#elif defined(__arm__)
#define bpf_target_arm
+ #define bpf_target_defined
#elif defined(__aarch64__)
#define bpf_target_arm64
+ #define bpf_target_defined
#elif defined(__mips__)
#define bpf_target_mips
+ #define bpf_target_defined
#elif defined(__powerpc__)
#define bpf_target_powerpc
+ #define bpf_target_defined
#elif defined(__sparc__)
#define bpf_target_sparc
+ #define bpf_target_defined
+#endif /* no compiler target */
+
#endif
+
+#ifndef __BPF_TARGET_MISSING
+#define __BPF_TARGET_MISSING "GCC error \"Must specify a BPF target arch via __TARGET_ARCH_xxx\""
#endif
#if defined(bpf_target_x86)
#elif defined(bpf_target_sparc)
#define BPF_KPROBE_READ_RET_IP(ip, ctx) ({ (ip) = PT_REGS_RET(ctx); })
#define BPF_KRETPROBE_READ_RET_IP BPF_KPROBE_READ_RET_IP
-#else
+#elif defined(bpf_target_defined)
#define BPF_KPROBE_READ_RET_IP(ip, ctx) \
({ bpf_probe_read_kernel(&(ip), sizeof(ip), (void *)PT_REGS_RET(ctx)); })
#define BPF_KRETPROBE_READ_RET_IP(ip, ctx) \
(void *)(PT_REGS_FP(ctx) + sizeof(ip))); })
#endif
+#if !defined(bpf_target_defined)
+
+#define PT_REGS_PARM1(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_PARM2(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_PARM3(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_PARM4(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_PARM5(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_RET(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_FP(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_RC(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_SP(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_IP(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+
+#define PT_REGS_PARM1_CORE(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_PARM2_CORE(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_PARM3_CORE(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_PARM4_CORE(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_PARM5_CORE(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_RET_CORE(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_FP_CORE(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_RC_CORE(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_SP_CORE(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define PT_REGS_IP_CORE(x) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+
+#define BPF_KPROBE_READ_RET_IP(ip, ctx) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+#define BPF_KRETPROBE_READ_RET_IP(ip, ctx) ({ _Pragma(__BPF_TARGET_MISSING); 0l; })
+
+#endif /* !defined(bpf_target_defined) */
+
+#ifndef ___bpf_concat
#define ___bpf_concat(a, b) a ## b
+#endif
+#ifndef ___bpf_apply
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
+#endif
+#ifndef ___bpf_nth
#define ___bpf_nth(_, _1, _2, _3, _4, _5, _6, _7, _8, _9, _a, _b, _c, N, ...) N
+#endif
+#ifndef ___bpf_narg
#define ___bpf_narg(...) \
___bpf_nth(_, ##__VA_ARGS__, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0)
-#define ___bpf_empty(...) \
- ___bpf_nth(_, ##__VA_ARGS__, N, N, N, N, N, N, N, N, N, N, 0)
+#endif
#define ___bpf_ctx_cast0() ctx
#define ___bpf_ctx_cast1(x) ___bpf_ctx_cast0(), (void *)ctx[0]
} \
static __always_inline typeof(name(0)) ____##name(struct pt_regs *ctx, ##args)
-#define ___bpf_fill0(arr, p, x) do {} while (0)
-#define ___bpf_fill1(arr, p, x) arr[p] = x
-#define ___bpf_fill2(arr, p, x, args...) arr[p] = x; ___bpf_fill1(arr, p + 1, args)
-#define ___bpf_fill3(arr, p, x, args...) arr[p] = x; ___bpf_fill2(arr, p + 1, args)
-#define ___bpf_fill4(arr, p, x, args...) arr[p] = x; ___bpf_fill3(arr, p + 1, args)
-#define ___bpf_fill5(arr, p, x, args...) arr[p] = x; ___bpf_fill4(arr, p + 1, args)
-#define ___bpf_fill6(arr, p, x, args...) arr[p] = x; ___bpf_fill5(arr, p + 1, args)
-#define ___bpf_fill7(arr, p, x, args...) arr[p] = x; ___bpf_fill6(arr, p + 1, args)
-#define ___bpf_fill8(arr, p, x, args...) arr[p] = x; ___bpf_fill7(arr, p + 1, args)
-#define ___bpf_fill9(arr, p, x, args...) arr[p] = x; ___bpf_fill8(arr, p + 1, args)
-#define ___bpf_fill10(arr, p, x, args...) arr[p] = x; ___bpf_fill9(arr, p + 1, args)
-#define ___bpf_fill11(arr, p, x, args...) arr[p] = x; ___bpf_fill10(arr, p + 1, args)
-#define ___bpf_fill12(arr, p, x, args...) arr[p] = x; ___bpf_fill11(arr, p + 1, args)
-#define ___bpf_fill(arr, args...) \
- ___bpf_apply(___bpf_fill, ___bpf_narg(args))(arr, 0, args)
-
-/*
- * BPF_SEQ_PRINTF to wrap bpf_seq_printf to-be-printed values
- * in a structure.
- */
-#define BPF_SEQ_PRINTF(seq, fmt, args...) \
-({ \
- static const char ___fmt[] = fmt; \
- unsigned long long ___param[___bpf_narg(args)]; \
- \
- _Pragma("GCC diagnostic push") \
- _Pragma("GCC diagnostic ignored \"-Wint-conversion\"") \
- ___bpf_fill(___param, args); \
- _Pragma("GCC diagnostic pop") \
- \
- bpf_seq_printf(seq, ___fmt, sizeof(___fmt), \
- ___param, sizeof(___param)); \
-})
-
-/*
- * BPF_SNPRINTF wraps the bpf_snprintf helper with variadic arguments instead of
- * an array of u64.
- */
-#define BPF_SNPRINTF(out, out_size, fmt, args...) \
-({ \
- static const char ___fmt[] = fmt; \
- unsigned long long ___param[___bpf_narg(args)]; \
- \
- _Pragma("GCC diagnostic push") \
- _Pragma("GCC diagnostic ignored \"-Wint-conversion\"") \
- ___bpf_fill(___param, args); \
- _Pragma("GCC diagnostic pop") \
- \
- bpf_snprintf(out, out_size, ___fmt, \
- ___param, sizeof(___param)); \
-})
-
#endif
const struct btf_type *btf__type_by_id(const struct btf *btf, __u32 type_id)
{
if (type_id >= btf->start_id + btf->nr_types)
- return NULL;
+ return errno = EINVAL, NULL;
return btf_type_by_id((struct btf *)btf, type_id);
}
int btf__set_pointer_size(struct btf *btf, size_t ptr_sz)
{
if (ptr_sz != 4 && ptr_sz != 8)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
btf->ptr_sz = ptr_sz;
return 0;
}
int btf__set_endianness(struct btf *btf, enum btf_endianness endian)
{
if (endian != BTF_LITTLE_ENDIAN && endian != BTF_BIG_ENDIAN)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
btf->swapped_endian = is_host_big_endian() != (endian == BTF_BIG_ENDIAN);
if (!btf->swapped_endian) {
int i;
t = btf__type_by_id(btf, type_id);
- for (i = 0; i < MAX_RESOLVE_DEPTH && !btf_type_is_void_or_null(t);
- i++) {
+ for (i = 0; i < MAX_RESOLVE_DEPTH && !btf_type_is_void_or_null(t); i++) {
switch (btf_kind(t)) {
case BTF_KIND_INT:
case BTF_KIND_STRUCT:
case BTF_KIND_ARRAY:
array = btf_array(t);
if (nelems && array->nelems > UINT32_MAX / nelems)
- return -E2BIG;
+ return libbpf_err(-E2BIG);
nelems *= array->nelems;
type_id = array->type;
break;
default:
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
t = btf__type_by_id(btf, type_id);
done:
if (size < 0)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (nelems && size > UINT32_MAX / nelems)
- return -E2BIG;
+ return libbpf_err(-E2BIG);
return nelems * size;
}
for (i = 0; i < vlen; i++, m++) {
align = btf__align_of(btf, m->type);
if (align <= 0)
- return align;
+ return libbpf_err(align);
max_align = max(max_align, align);
}
}
default:
pr_warn("unsupported BTF_KIND:%u\n", btf_kind(t));
- return 0;
+ return errno = EINVAL, 0;
}
}
}
if (depth == MAX_RESOLVE_DEPTH || btf_type_is_void_or_null(t))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
return type_id;
}
return i;
}
- return -ENOENT;
+ return libbpf_err(-ENOENT);
}
__s32 btf__find_by_name_kind(const struct btf *btf, const char *type_name,
return i;
}
- return -ENOENT;
+ return libbpf_err(-ENOENT);
}
static bool btf_is_modifiable(const struct btf *btf)
struct btf *btf__new_empty(void)
{
- return btf_new_empty(NULL);
+ return libbpf_ptr(btf_new_empty(NULL));
}
struct btf *btf__new_empty_split(struct btf *base_btf)
{
- return btf_new_empty(base_btf);
+ return libbpf_ptr(btf_new_empty(base_btf));
}
static struct btf *btf_new(const void *data, __u32 size, struct btf *base_btf)
struct btf *btf__new(const void *data, __u32 size)
{
- return btf_new(data, size, NULL);
+ return libbpf_ptr(btf_new(data, size, NULL));
}
static struct btf *btf_parse_elf(const char *path, struct btf *base_btf,
goto done;
}
btf = btf_new(btf_data->d_buf, btf_data->d_size, base_btf);
- if (IS_ERR(btf))
+ err = libbpf_get_error(btf);
+ if (err)
goto done;
switch (gelf_getclass(elf)) {
}
if (btf_ext && btf_ext_data) {
- *btf_ext = btf_ext__new(btf_ext_data->d_buf,
- btf_ext_data->d_size);
- if (IS_ERR(*btf_ext))
+ *btf_ext = btf_ext__new(btf_ext_data->d_buf, btf_ext_data->d_size);
+ err = libbpf_get_error(*btf_ext);
+ if (err)
goto done;
} else if (btf_ext) {
*btf_ext = NULL;
elf_end(elf);
close(fd);
- if (err)
- return ERR_PTR(err);
- /*
- * btf is always parsed before btf_ext, so no need to clean up
- * btf_ext, if btf loading failed
- */
- if (IS_ERR(btf))
+ if (!err)
return btf;
- if (btf_ext && IS_ERR(*btf_ext)) {
- btf__free(btf);
- err = PTR_ERR(*btf_ext);
- return ERR_PTR(err);
- }
- return btf;
+
+ if (btf_ext)
+ btf_ext__free(*btf_ext);
+ btf__free(btf);
+
+ return ERR_PTR(err);
}
struct btf *btf__parse_elf(const char *path, struct btf_ext **btf_ext)
{
- return btf_parse_elf(path, NULL, btf_ext);
+ return libbpf_ptr(btf_parse_elf(path, NULL, btf_ext));
}
struct btf *btf__parse_elf_split(const char *path, struct btf *base_btf)
{
- return btf_parse_elf(path, base_btf, NULL);
+ return libbpf_ptr(btf_parse_elf(path, base_btf, NULL));
}
static struct btf *btf_parse_raw(const char *path, struct btf *base_btf)
struct btf *btf__parse_raw(const char *path)
{
- return btf_parse_raw(path, NULL);
+ return libbpf_ptr(btf_parse_raw(path, NULL));
}
struct btf *btf__parse_raw_split(const char *path, struct btf *base_btf)
{
- return btf_parse_raw(path, base_btf);
+ return libbpf_ptr(btf_parse_raw(path, base_btf));
}
static struct btf *btf_parse(const char *path, struct btf *base_btf, struct btf_ext **btf_ext)
{
struct btf *btf;
+ int err;
if (btf_ext)
*btf_ext = NULL;
btf = btf_parse_raw(path, base_btf);
- if (!IS_ERR(btf) || PTR_ERR(btf) != -EPROTO)
+ err = libbpf_get_error(btf);
+ if (!err)
return btf;
-
+ if (err != -EPROTO)
+ return ERR_PTR(err);
return btf_parse_elf(path, base_btf, btf_ext);
}
struct btf *btf__parse(const char *path, struct btf_ext **btf_ext)
{
- return btf_parse(path, NULL, btf_ext);
+ return libbpf_ptr(btf_parse(path, NULL, btf_ext));
}
struct btf *btf__parse_split(const char *path, struct btf *base_btf)
{
- return btf_parse(path, base_btf, NULL);
+ return libbpf_ptr(btf_parse(path, base_btf, NULL));
}
static int compare_vsi_off(const void *_a, const void *_b)
}
}
- return err;
+ return libbpf_err(err);
}
static void *btf_get_raw_data(const struct btf *btf, __u32 *size, bool swap_endian);
int err = 0;
if (btf->fd >= 0)
- return -EEXIST;
+ return libbpf_err(-EEXIST);
retry_load:
if (log_buf_size) {
log_buf = malloc(log_buf_size);
if (!log_buf)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
*log_buf = 0;
}
done:
free(log_buf);
- return err;
+ return libbpf_err(err);
}
int btf__fd(const struct btf *btf)
data = btf_get_raw_data(btf, &data_sz, btf->swapped_endian);
if (!data)
- return NULL;
+ return errno = -ENOMEM, NULL;
btf->raw_size = data_sz;
if (btf->swapped_endian)
else if (offset - btf->start_str_off < btf->hdr->str_len)
return btf_strs_data(btf) + (offset - btf->start_str_off);
else
- return NULL;
+ return errno = EINVAL, NULL;
}
const char *btf__name_by_offset(const struct btf *btf, __u32 offset)
int btf__get_from_id(__u32 id, struct btf **btf)
{
struct btf *res;
- int btf_fd;
+ int err, btf_fd;
*btf = NULL;
btf_fd = bpf_btf_get_fd_by_id(id);
if (btf_fd < 0)
- return -errno;
+ return libbpf_err(-errno);
res = btf_get_from_fd(btf_fd, NULL);
+ err = libbpf_get_error(res);
+
close(btf_fd);
- if (IS_ERR(res))
- return PTR_ERR(res);
+
+ if (err)
+ return libbpf_err(err);
*btf = res;
return 0;
__s64 key_size, value_size;
__s32 container_id;
- if (snprintf(container_name, max_name, "____btf_map_%s", map_name) ==
- max_name) {
+ if (snprintf(container_name, max_name, "____btf_map_%s", map_name) == max_name) {
pr_warn("map:%s length of '____btf_map_%s' is too long\n",
map_name, map_name);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
container_id = btf__find_by_name(btf, container_name);
if (container_id < 0) {
pr_debug("map:%s container_name:%s cannot be found in BTF. Missing BPF_ANNOTATE_KV_PAIR?\n",
map_name, container_name);
- return container_id;
+ return libbpf_err(container_id);
}
container_type = btf__type_by_id(btf, container_id);
if (!container_type) {
pr_warn("map:%s cannot find BTF type for container_id:%u\n",
map_name, container_id);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
if (!btf_is_struct(container_type) || btf_vlen(container_type) < 2) {
pr_warn("map:%s container_name:%s is an invalid container struct\n",
map_name, container_name);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
key = btf_members(container_type);
key_size = btf__resolve_size(btf, key->type);
if (key_size < 0) {
pr_warn("map:%s invalid BTF key_type_size\n", map_name);
- return key_size;
+ return libbpf_err(key_size);
}
if (expected_key_size != key_size) {
pr_warn("map:%s btf_key_type_size:%u != map_def_key_size:%u\n",
map_name, (__u32)key_size, expected_key_size);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
value_size = btf__resolve_size(btf, value->type);
if (value_size < 0) {
pr_warn("map:%s invalid BTF value_type_size\n", map_name);
- return value_size;
+ return libbpf_err(value_size);
}
if (expected_value_size != value_size) {
pr_warn("map:%s btf_value_type_size:%u != map_def_value_size:%u\n",
map_name, (__u32)value_size, expected_value_size);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
*key_type_id = key->type;
/* BTF needs to be in a modifiable state to build string lookup index */
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
off = strset__find_str(btf->strs_set, s);
if (off < 0)
- return off;
+ return libbpf_err(off);
return btf->start_str_off + off;
}
}
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
off = strset__add_str(btf->strs_set, s);
if (off < 0)
- return off;
+ return libbpf_err(off);
btf->hdr->str_len = strset__data_size(btf->strs_set);
err = btf_add_type_idx_entry(btf, btf->hdr->type_len);
if (err)
- return err;
+ return libbpf_err(err);
btf->hdr->type_len += data_sz;
btf->hdr->str_off += data_sz;
sz = btf_type_size(src_type);
if (sz < 0)
- return sz;
+ return libbpf_err(sz);
/* deconstruct BTF, if necessary, and invalidate raw_data */
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
t = btf_add_type_mem(btf, sz);
if (!t)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
memcpy(t, src_type, sz);
err = btf_type_visit_str_offs(t, btf_rewrite_str, &p);
if (err)
- return err;
+ return libbpf_err(err);
return btf_commit_type(btf, sz);
}
/* non-empty name */
if (!name || !name[0])
- return -EINVAL;
+ return libbpf_err(-EINVAL);
/* byte_sz must be power of 2 */
if (!byte_sz || (byte_sz & (byte_sz - 1)) || byte_sz > 16)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (encoding & ~(BTF_INT_SIGNED | BTF_INT_CHAR | BTF_INT_BOOL))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
/* deconstruct BTF, if necessary, and invalidate raw_data */
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
sz = sizeof(struct btf_type) + sizeof(int);
t = btf_add_type_mem(btf, sz);
if (!t)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
/* if something goes wrong later, we might end up with an extra string,
* but that shouldn't be a problem, because BTF can't be constructed
/* non-empty name */
if (!name || !name[0])
- return -EINVAL;
+ return libbpf_err(-EINVAL);
/* byte_sz must be one of the explicitly allowed values */
if (byte_sz != 2 && byte_sz != 4 && byte_sz != 8 && byte_sz != 12 &&
byte_sz != 16)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
sz = sizeof(struct btf_type);
t = btf_add_type_mem(btf, sz);
if (!t)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
name_off = btf__add_str(btf, name);
if (name_off < 0)
int sz, name_off = 0;
if (validate_type_id(ref_type_id))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
sz = sizeof(struct btf_type);
t = btf_add_type_mem(btf, sz);
if (!t)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
if (name && name[0]) {
name_off = btf__add_str(btf, name);
int sz;
if (validate_type_id(index_type_id) || validate_type_id(elem_type_id))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
sz = sizeof(struct btf_type) + sizeof(struct btf_array);
t = btf_add_type_mem(btf, sz);
if (!t)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
t->name_off = 0;
t->info = btf_type_info(BTF_KIND_ARRAY, 0, 0);
int sz, name_off = 0;
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
sz = sizeof(struct btf_type);
t = btf_add_type_mem(btf, sz);
if (!t)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
if (name && name[0]) {
name_off = btf__add_str(btf, name);
/* last type should be union/struct */
if (btf->nr_types == 0)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
t = btf_last_type(btf);
if (!btf_is_composite(t))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (validate_type_id(type_id))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
/* best-effort bit field offset/size enforcement */
is_bitfield = bit_size || (bit_offset % 8 != 0);
if (is_bitfield && (bit_size == 0 || bit_size > 255 || bit_offset > 0xffffff))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
/* only offset 0 is allowed for unions */
if (btf_is_union(t) && bit_offset)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
/* decompose and invalidate raw data */
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
sz = sizeof(struct btf_member);
m = btf_add_type_mem(btf, sz);
if (!m)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
if (name && name[0]) {
name_off = btf__add_str(btf, name);
/* byte_sz must be power of 2 */
if (!byte_sz || (byte_sz & (byte_sz - 1)) || byte_sz > 8)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
sz = sizeof(struct btf_type);
t = btf_add_type_mem(btf, sz);
if (!t)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
if (name && name[0]) {
name_off = btf__add_str(btf, name);
/* last type should be BTF_KIND_ENUM */
if (btf->nr_types == 0)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
t = btf_last_type(btf);
if (!btf_is_enum(t))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
/* non-empty name */
if (!name || !name[0])
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (value < INT_MIN || value > UINT_MAX)
- return -E2BIG;
+ return libbpf_err(-E2BIG);
/* decompose and invalidate raw data */
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
sz = sizeof(struct btf_enum);
v = btf_add_type_mem(btf, sz);
if (!v)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
name_off = btf__add_str(btf, name);
if (name_off < 0)
int btf__add_fwd(struct btf *btf, const char *name, enum btf_fwd_kind fwd_kind)
{
if (!name || !name[0])
- return -EINVAL;
+ return libbpf_err(-EINVAL);
switch (fwd_kind) {
case BTF_FWD_STRUCT:
*/
return btf__add_enum(btf, name, sizeof(int));
default:
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
}
int btf__add_typedef(struct btf *btf, const char *name, int ref_type_id)
{
if (!name || !name[0])
- return -EINVAL;
+ return libbpf_err(-EINVAL);
return btf_add_ref_kind(btf, BTF_KIND_TYPEDEF, name, ref_type_id);
}
int id;
if (!name || !name[0])
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (linkage != BTF_FUNC_STATIC && linkage != BTF_FUNC_GLOBAL &&
linkage != BTF_FUNC_EXTERN)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
id = btf_add_ref_kind(btf, BTF_KIND_FUNC, name, proto_type_id);
if (id > 0) {
t->info = btf_type_info(BTF_KIND_FUNC, linkage, 0);
}
- return id;
+ return libbpf_err(id);
}
/*
int sz;
if (validate_type_id(ret_type_id))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
sz = sizeof(struct btf_type);
t = btf_add_type_mem(btf, sz);
if (!t)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
/* start out with vlen=0; this will be adjusted when adding enum
* values, if necessary
int sz, name_off = 0;
if (validate_type_id(type_id))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
/* last type should be BTF_KIND_FUNC_PROTO */
if (btf->nr_types == 0)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
t = btf_last_type(btf);
if (!btf_is_func_proto(t))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
/* decompose and invalidate raw data */
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
sz = sizeof(struct btf_param);
p = btf_add_type_mem(btf, sz);
if (!p)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
if (name && name[0]) {
name_off = btf__add_str(btf, name);
/* non-empty name */
if (!name || !name[0])
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (linkage != BTF_VAR_STATIC && linkage != BTF_VAR_GLOBAL_ALLOCATED &&
linkage != BTF_VAR_GLOBAL_EXTERN)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (validate_type_id(type_id))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
/* deconstruct BTF, if necessary, and invalidate raw_data */
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
sz = sizeof(struct btf_type) + sizeof(struct btf_var);
t = btf_add_type_mem(btf, sz);
if (!t)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
name_off = btf__add_str(btf, name);
if (name_off < 0)
/* non-empty name */
if (!name || !name[0])
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
sz = sizeof(struct btf_type);
t = btf_add_type_mem(btf, sz);
if (!t)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
name_off = btf__add_str(btf, name);
if (name_off < 0)
/* last type should be BTF_KIND_DATASEC */
if (btf->nr_types == 0)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
t = btf_last_type(btf);
if (!btf_is_datasec(t))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (validate_type_id(var_type_id))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
/* decompose and invalidate raw data */
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
sz = sizeof(struct btf_var_secinfo);
v = btf_add_type_mem(btf, sz);
if (!v)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
v->type = var_type_id;
v->offset = offset;
err = btf_ext_parse_hdr(data, size);
if (err)
- return ERR_PTR(err);
+ return libbpf_err_ptr(err);
btf_ext = calloc(1, sizeof(struct btf_ext));
if (!btf_ext)
- return ERR_PTR(-ENOMEM);
+ return libbpf_err_ptr(-ENOMEM);
btf_ext->data_size = size;
btf_ext->data = malloc(size);
}
memcpy(btf_ext->data, data, size);
- if (btf_ext->hdr->hdr_len <
- offsetofend(struct btf_ext_header, line_info_len))
+ if (btf_ext->hdr->hdr_len < offsetofend(struct btf_ext_header, line_info_len)) {
+ err = -EINVAL;
goto done;
+ }
+
err = btf_ext_setup_func_info(btf_ext);
if (err)
goto done;
if (err)
goto done;
- if (btf_ext->hdr->hdr_len < offsetofend(struct btf_ext_header, core_relo_len))
+ if (btf_ext->hdr->hdr_len < offsetofend(struct btf_ext_header, core_relo_len)) {
+ err = -EINVAL;
goto done;
+ }
+
err = btf_ext_setup_core_relos(btf_ext);
if (err)
goto done;
done:
if (err) {
btf_ext__free(btf_ext);
- return ERR_PTR(err);
+ return libbpf_err_ptr(err);
}
return btf_ext;
existing_len = (*cnt) * record_size;
data = realloc(*info, existing_len + records_len);
if (!data)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
memcpy(data + existing_len, sinfo->data, records_len);
/* adjust insn_off only, the rest data will be passed
__u32 *insn_off;
insn_off = data + existing_len + (i * record_size);
- *insn_off = *insn_off / sizeof(struct bpf_insn) +
- insns_cnt;
+ *insn_off = *insn_off / sizeof(struct bpf_insn) + insns_cnt;
}
*info = data;
*cnt += sinfo->num_info;
return 0;
}
- return -ENOENT;
+ return libbpf_err(-ENOENT);
}
int btf_ext__reloc_func_info(const struct btf *btf,
if (IS_ERR(d)) {
pr_debug("btf_dedup_new failed: %ld", PTR_ERR(d));
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
if (btf_ensure_modifiable(btf))
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
err = btf_dedup_prep(d);
if (err) {
done:
btf_dedup_free(d);
- return err;
+ return libbpf_err(err);
}
#define BTF_UNPROCESSED_ID ((__u32)-1)
char path[PATH_MAX + 1];
struct utsname buf;
struct btf *btf;
- int i;
+ int i, err;
uname(&buf);
btf = btf__parse_raw(path);
else
btf = btf__parse_elf(path, NULL);
-
- pr_debug("loading kernel BTF '%s': %ld\n",
- path, IS_ERR(btf) ? PTR_ERR(btf) : 0);
- if (IS_ERR(btf))
+ err = libbpf_get_error(btf);
+ pr_debug("loading kernel BTF '%s': %d\n", path, err);
+ if (err)
continue;
return btf;
}
pr_warn("failed to find valid kernel BTF\n");
- return ERR_PTR(-ESRCH);
+ return libbpf_err_ptr(-ESRCH);
}
int btf_type_visit_type_ids(struct btf_type *t, type_id_visit_fn visit, void *ctx)
d = calloc(1, sizeof(struct btf_dump));
if (!d)
- return ERR_PTR(-ENOMEM);
+ return libbpf_err_ptr(-ENOMEM);
d->btf = btf;
d->btf_ext = btf_ext;
return d;
err:
btf_dump__free(d);
- return ERR_PTR(err);
+ return libbpf_err_ptr(err);
}
static int btf_dump_resize(struct btf_dump *d)
int err, i;
if (id > btf__get_nr_types(d->btf))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
err = btf_dump_resize(d);
if (err)
- return err;
+ return libbpf_err(err);
d->emit_queue_cnt = 0;
err = btf_dump_order_type(d, id, false);
if (err < 0)
- return err;
+ return libbpf_err(err);
for (i = 0; i < d->emit_queue_cnt; i++)
btf_dump_emit_type(d, d->emit_queue[i], 0 /*top-level*/);
int lvl, err;
if (!OPTS_VALID(opts, btf_dump_emit_type_decl_opts))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
err = btf_dump_resize(d);
if (err)
- return -EINVAL;
+ return libbpf_err(err);
fname = OPTS_GET(opts, field_name, "");
lvl = OPTS_GET(opts, indent_level, 0);
--- /dev/null
+// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+/* Copyright (c) 2021 Facebook */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <linux/filter.h>
+#include "btf.h"
+#include "bpf.h"
+#include "libbpf.h"
+#include "libbpf_internal.h"
+#include "hashmap.h"
+#include "bpf_gen_internal.h"
+#include "skel_internal.h"
+
+#define MAX_USED_MAPS 64
+#define MAX_USED_PROGS 32
+
+/* The following structure describes the stack layout of the loader program.
+ * In addition R6 contains the pointer to context.
+ * R7 contains the result of the last sys_bpf command (typically error or FD).
+ * R9 contains the result of the last sys_close command.
+ *
+ * Naming convention:
+ * ctx - bpf program context
+ * stack - bpf program stack
+ * blob - bpf_attr-s, strings, insns, map data.
+ * All the bytes that loader prog will use for read/write.
+ */
+struct loader_stack {
+ __u32 btf_fd;
+ __u32 map_fd[MAX_USED_MAPS];
+ __u32 prog_fd[MAX_USED_PROGS];
+ __u32 inner_map_fd;
+};
+
+#define stack_off(field) \
+ (__s16)(-sizeof(struct loader_stack) + offsetof(struct loader_stack, field))
+
+#define attr_field(attr, field) (attr + offsetof(union bpf_attr, field))
+
+static int realloc_insn_buf(struct bpf_gen *gen, __u32 size)
+{
+ size_t off = gen->insn_cur - gen->insn_start;
+ void *insn_start;
+
+ if (gen->error)
+ return gen->error;
+ if (size > INT32_MAX || off + size > INT32_MAX) {
+ gen->error = -ERANGE;
+ return -ERANGE;
+ }
+ insn_start = realloc(gen->insn_start, off + size);
+ if (!insn_start) {
+ gen->error = -ENOMEM;
+ free(gen->insn_start);
+ gen->insn_start = NULL;
+ return -ENOMEM;
+ }
+ gen->insn_start = insn_start;
+ gen->insn_cur = insn_start + off;
+ return 0;
+}
+
+static int realloc_data_buf(struct bpf_gen *gen, __u32 size)
+{
+ size_t off = gen->data_cur - gen->data_start;
+ void *data_start;
+
+ if (gen->error)
+ return gen->error;
+ if (size > INT32_MAX || off + size > INT32_MAX) {
+ gen->error = -ERANGE;
+ return -ERANGE;
+ }
+ data_start = realloc(gen->data_start, off + size);
+ if (!data_start) {
+ gen->error = -ENOMEM;
+ free(gen->data_start);
+ gen->data_start = NULL;
+ return -ENOMEM;
+ }
+ gen->data_start = data_start;
+ gen->data_cur = data_start + off;
+ return 0;
+}
+
+static void emit(struct bpf_gen *gen, struct bpf_insn insn)
+{
+ if (realloc_insn_buf(gen, sizeof(insn)))
+ return;
+ memcpy(gen->insn_cur, &insn, sizeof(insn));
+ gen->insn_cur += sizeof(insn);
+}
+
+static void emit2(struct bpf_gen *gen, struct bpf_insn insn1, struct bpf_insn insn2)
+{
+ emit(gen, insn1);
+ emit(gen, insn2);
+}
+
+void bpf_gen__init(struct bpf_gen *gen, int log_level)
+{
+ size_t stack_sz = sizeof(struct loader_stack);
+ int i;
+
+ gen->log_level = log_level;
+ /* save ctx pointer into R6 */
+ emit(gen, BPF_MOV64_REG(BPF_REG_6, BPF_REG_1));
+
+ /* bzero stack */
+ emit(gen, BPF_MOV64_REG(BPF_REG_1, BPF_REG_10));
+ emit(gen, BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, -stack_sz));
+ emit(gen, BPF_MOV64_IMM(BPF_REG_2, stack_sz));
+ emit(gen, BPF_MOV64_IMM(BPF_REG_3, 0));
+ emit(gen, BPF_EMIT_CALL(BPF_FUNC_probe_read_kernel));
+
+ /* jump over cleanup code */
+ emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0,
+ /* size of cleanup code below */
+ (stack_sz / 4) * 3 + 2));
+
+ /* remember the label where all error branches will jump to */
+ gen->cleanup_label = gen->insn_cur - gen->insn_start;
+ /* emit cleanup code: close all temp FDs */
+ for (i = 0; i < stack_sz; i += 4) {
+ emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_10, -stack_sz + i));
+ emit(gen, BPF_JMP_IMM(BPF_JSLE, BPF_REG_1, 0, 1));
+ emit(gen, BPF_EMIT_CALL(BPF_FUNC_sys_close));
+ }
+ /* R7 contains the error code from sys_bpf. Copy it into R0 and exit. */
+ emit(gen, BPF_MOV64_REG(BPF_REG_0, BPF_REG_7));
+ emit(gen, BPF_EXIT_INSN());
+}
+
+static int add_data(struct bpf_gen *gen, const void *data, __u32 size)
+{
+ void *prev;
+
+ if (realloc_data_buf(gen, size))
+ return 0;
+ prev = gen->data_cur;
+ memcpy(gen->data_cur, data, size);
+ gen->data_cur += size;
+ return prev - gen->data_start;
+}
+
+static int insn_bytes_to_bpf_size(__u32 sz)
+{
+ switch (sz) {
+ case 8: return BPF_DW;
+ case 4: return BPF_W;
+ case 2: return BPF_H;
+ case 1: return BPF_B;
+ default: return -1;
+ }
+}
+
+/* *(u64 *)(blob + off) = (u64)(void *)(blob + data) */
+static void emit_rel_store(struct bpf_gen *gen, int off, int data)
+{
+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_0, BPF_PSEUDO_MAP_IDX_VALUE,
+ 0, 0, 0, data));
+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
+ 0, 0, 0, off));
+ emit(gen, BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 0));
+}
+
+/* *(u64 *)(blob + off) = (u64)(void *)(%sp + stack_off) */
+static void emit_rel_store_sp(struct bpf_gen *gen, int off, int stack_off)
+{
+ emit(gen, BPF_MOV64_REG(BPF_REG_0, BPF_REG_10));
+ emit(gen, BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, stack_off));
+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
+ 0, 0, 0, off));
+ emit(gen, BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 0));
+}
+
+static void move_ctx2blob(struct bpf_gen *gen, int off, int size, int ctx_off,
+ bool check_non_zero)
+{
+ emit(gen, BPF_LDX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_0, BPF_REG_6, ctx_off));
+ if (check_non_zero)
+ /* If value in ctx is zero don't update the blob.
+ * For example: when ctx->map.max_entries == 0, keep default max_entries from bpf.c
+ */
+ emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3));
+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
+ 0, 0, 0, off));
+ emit(gen, BPF_STX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_1, BPF_REG_0, 0));
+}
+
+static void move_stack2blob(struct bpf_gen *gen, int off, int size, int stack_off)
+{
+ emit(gen, BPF_LDX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_0, BPF_REG_10, stack_off));
+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
+ 0, 0, 0, off));
+ emit(gen, BPF_STX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_1, BPF_REG_0, 0));
+}
+
+static void move_stack2ctx(struct bpf_gen *gen, int ctx_off, int size, int stack_off)
+{
+ emit(gen, BPF_LDX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_0, BPF_REG_10, stack_off));
+ emit(gen, BPF_STX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_6, BPF_REG_0, ctx_off));
+}
+
+static void emit_sys_bpf(struct bpf_gen *gen, int cmd, int attr, int attr_size)
+{
+ emit(gen, BPF_MOV64_IMM(BPF_REG_1, cmd));
+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_2, BPF_PSEUDO_MAP_IDX_VALUE,
+ 0, 0, 0, attr));
+ emit(gen, BPF_MOV64_IMM(BPF_REG_3, attr_size));
+ emit(gen, BPF_EMIT_CALL(BPF_FUNC_sys_bpf));
+ /* remember the result in R7 */
+ emit(gen, BPF_MOV64_REG(BPF_REG_7, BPF_REG_0));
+}
+
+static bool is_simm16(__s64 value)
+{
+ return value == (__s64)(__s16)value;
+}
+
+static void emit_check_err(struct bpf_gen *gen)
+{
+ __s64 off = -(gen->insn_cur - gen->insn_start - gen->cleanup_label) / 8 - 1;
+
+ /* R7 contains result of last sys_bpf command.
+ * if (R7 < 0) goto cleanup;
+ */
+ if (is_simm16(off)) {
+ emit(gen, BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0, off));
+ } else {
+ gen->error = -ERANGE;
+ emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, -1));
+ }
+}
+
+/* reg1 and reg2 should not be R1 - R5. They can be R0, R6 - R10 */
+static void emit_debug(struct bpf_gen *gen, int reg1, int reg2,
+ const char *fmt, va_list args)
+{
+ char buf[1024];
+ int addr, len, ret;
+
+ if (!gen->log_level)
+ return;
+ ret = vsnprintf(buf, sizeof(buf), fmt, args);
+ if (ret < 1024 - 7 && reg1 >= 0 && reg2 < 0)
+ /* The special case to accommodate common debug_ret():
+ * to avoid specifying BPF_REG_7 and adding " r=%%d" to
+ * prints explicitly.
+ */
+ strcat(buf, " r=%d");
+ len = strlen(buf) + 1;
+ addr = add_data(gen, buf, len);
+
+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
+ 0, 0, 0, addr));
+ emit(gen, BPF_MOV64_IMM(BPF_REG_2, len));
+ if (reg1 >= 0)
+ emit(gen, BPF_MOV64_REG(BPF_REG_3, reg1));
+ if (reg2 >= 0)
+ emit(gen, BPF_MOV64_REG(BPF_REG_4, reg2));
+ emit(gen, BPF_EMIT_CALL(BPF_FUNC_trace_printk));
+}
+
+static void debug_regs(struct bpf_gen *gen, int reg1, int reg2, const char *fmt, ...)
+{
+ va_list args;
+
+ va_start(args, fmt);
+ emit_debug(gen, reg1, reg2, fmt, args);
+ va_end(args);
+}
+
+static void debug_ret(struct bpf_gen *gen, const char *fmt, ...)
+{
+ va_list args;
+
+ va_start(args, fmt);
+ emit_debug(gen, BPF_REG_7, -1, fmt, args);
+ va_end(args);
+}
+
+static void __emit_sys_close(struct bpf_gen *gen)
+{
+ emit(gen, BPF_JMP_IMM(BPF_JSLE, BPF_REG_1, 0,
+ /* 2 is the number of the following insns
+ * * 6 is additional insns in debug_regs
+ */
+ 2 + (gen->log_level ? 6 : 0)));
+ emit(gen, BPF_MOV64_REG(BPF_REG_9, BPF_REG_1));
+ emit(gen, BPF_EMIT_CALL(BPF_FUNC_sys_close));
+ debug_regs(gen, BPF_REG_9, BPF_REG_0, "close(%%d) = %%d");
+}
+
+static void emit_sys_close_stack(struct bpf_gen *gen, int stack_off)
+{
+ emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_10, stack_off));
+ __emit_sys_close(gen);
+}
+
+static void emit_sys_close_blob(struct bpf_gen *gen, int blob_off)
+{
+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_0, BPF_PSEUDO_MAP_IDX_VALUE,
+ 0, 0, 0, blob_off));
+ emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0));
+ __emit_sys_close(gen);
+}
+
+int bpf_gen__finish(struct bpf_gen *gen)
+{
+ int i;
+
+ emit_sys_close_stack(gen, stack_off(btf_fd));
+ for (i = 0; i < gen->nr_progs; i++)
+ move_stack2ctx(gen,
+ sizeof(struct bpf_loader_ctx) +
+ sizeof(struct bpf_map_desc) * gen->nr_maps +
+ sizeof(struct bpf_prog_desc) * i +
+ offsetof(struct bpf_prog_desc, prog_fd), 4,
+ stack_off(prog_fd[i]));
+ for (i = 0; i < gen->nr_maps; i++)
+ move_stack2ctx(gen,
+ sizeof(struct bpf_loader_ctx) +
+ sizeof(struct bpf_map_desc) * i +
+ offsetof(struct bpf_map_desc, map_fd), 4,
+ stack_off(map_fd[i]));
+ emit(gen, BPF_MOV64_IMM(BPF_REG_0, 0));
+ emit(gen, BPF_EXIT_INSN());
+ pr_debug("gen: finish %d\n", gen->error);
+ if (!gen->error) {
+ struct gen_loader_opts *opts = gen->opts;
+
+ opts->insns = gen->insn_start;
+ opts->insns_sz = gen->insn_cur - gen->insn_start;
+ opts->data = gen->data_start;
+ opts->data_sz = gen->data_cur - gen->data_start;
+ }
+ return gen->error;
+}
+
+void bpf_gen__free(struct bpf_gen *gen)
+{
+ if (!gen)
+ return;
+ free(gen->data_start);
+ free(gen->insn_start);
+ free(gen);
+}
+
+void bpf_gen__load_btf(struct bpf_gen *gen, const void *btf_raw_data,
+ __u32 btf_raw_size)
+{
+ int attr_size = offsetofend(union bpf_attr, btf_log_level);
+ int btf_data, btf_load_attr;
+ union bpf_attr attr;
+
+ memset(&attr, 0, attr_size);
+ pr_debug("gen: load_btf: size %d\n", btf_raw_size);
+ btf_data = add_data(gen, btf_raw_data, btf_raw_size);
+
+ attr.btf_size = btf_raw_size;
+ btf_load_attr = add_data(gen, &attr, attr_size);
+
+ /* populate union bpf_attr with user provided log details */
+ move_ctx2blob(gen, attr_field(btf_load_attr, btf_log_level), 4,
+ offsetof(struct bpf_loader_ctx, log_level), false);
+ move_ctx2blob(gen, attr_field(btf_load_attr, btf_log_size), 4,
+ offsetof(struct bpf_loader_ctx, log_size), false);
+ move_ctx2blob(gen, attr_field(btf_load_attr, btf_log_buf), 8,
+ offsetof(struct bpf_loader_ctx, log_buf), false);
+ /* populate union bpf_attr with a pointer to the BTF data */
+ emit_rel_store(gen, attr_field(btf_load_attr, btf), btf_data);
+ /* emit BTF_LOAD command */
+ emit_sys_bpf(gen, BPF_BTF_LOAD, btf_load_attr, attr_size);
+ debug_ret(gen, "btf_load size %d", btf_raw_size);
+ emit_check_err(gen);
+ /* remember btf_fd in the stack, if successful */
+ emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_7, stack_off(btf_fd)));
+}
+
+void bpf_gen__map_create(struct bpf_gen *gen,
+ struct bpf_create_map_attr *map_attr, int map_idx)
+{
+ int attr_size = offsetofend(union bpf_attr, btf_vmlinux_value_type_id);
+ bool close_inner_map_fd = false;
+ int map_create_attr;
+ union bpf_attr attr;
+
+ memset(&attr, 0, attr_size);
+ attr.map_type = map_attr->map_type;
+ attr.key_size = map_attr->key_size;
+ attr.value_size = map_attr->value_size;
+ attr.map_flags = map_attr->map_flags;
+ memcpy(attr.map_name, map_attr->name,
+ min((unsigned)strlen(map_attr->name), BPF_OBJ_NAME_LEN - 1));
+ attr.numa_node = map_attr->numa_node;
+ attr.map_ifindex = map_attr->map_ifindex;
+ attr.max_entries = map_attr->max_entries;
+ switch (attr.map_type) {
+ case BPF_MAP_TYPE_PERF_EVENT_ARRAY:
+ case BPF_MAP_TYPE_CGROUP_ARRAY:
+ case BPF_MAP_TYPE_STACK_TRACE:
+ case BPF_MAP_TYPE_ARRAY_OF_MAPS:
+ case BPF_MAP_TYPE_HASH_OF_MAPS:
+ case BPF_MAP_TYPE_DEVMAP:
+ case BPF_MAP_TYPE_DEVMAP_HASH:
+ case BPF_MAP_TYPE_CPUMAP:
+ case BPF_MAP_TYPE_XSKMAP:
+ case BPF_MAP_TYPE_SOCKMAP:
+ case BPF_MAP_TYPE_SOCKHASH:
+ case BPF_MAP_TYPE_QUEUE:
+ case BPF_MAP_TYPE_STACK:
+ case BPF_MAP_TYPE_RINGBUF:
+ break;
+ default:
+ attr.btf_key_type_id = map_attr->btf_key_type_id;
+ attr.btf_value_type_id = map_attr->btf_value_type_id;
+ }
+
+ pr_debug("gen: map_create: %s idx %d type %d value_type_id %d\n",
+ attr.map_name, map_idx, map_attr->map_type, attr.btf_value_type_id);
+
+ map_create_attr = add_data(gen, &attr, attr_size);
+ if (attr.btf_value_type_id)
+ /* populate union bpf_attr with btf_fd saved in the stack earlier */
+ move_stack2blob(gen, attr_field(map_create_attr, btf_fd), 4,
+ stack_off(btf_fd));
+ switch (attr.map_type) {
+ case BPF_MAP_TYPE_ARRAY_OF_MAPS:
+ case BPF_MAP_TYPE_HASH_OF_MAPS:
+ move_stack2blob(gen, attr_field(map_create_attr, inner_map_fd), 4,
+ stack_off(inner_map_fd));
+ close_inner_map_fd = true;
+ break;
+ default:
+ break;
+ }
+ /* conditionally update max_entries */
+ if (map_idx >= 0)
+ move_ctx2blob(gen, attr_field(map_create_attr, max_entries), 4,
+ sizeof(struct bpf_loader_ctx) +
+ sizeof(struct bpf_map_desc) * map_idx +
+ offsetof(struct bpf_map_desc, max_entries),
+ true /* check that max_entries != 0 */);
+ /* emit MAP_CREATE command */
+ emit_sys_bpf(gen, BPF_MAP_CREATE, map_create_attr, attr_size);
+ debug_ret(gen, "map_create %s idx %d type %d value_size %d value_btf_id %d",
+ attr.map_name, map_idx, map_attr->map_type, attr.value_size,
+ attr.btf_value_type_id);
+ emit_check_err(gen);
+ /* remember map_fd in the stack, if successful */
+ if (map_idx < 0) {
+ /* This bpf_gen__map_create() function is called with map_idx >= 0
+ * for all maps that libbpf loading logic tracks.
+ * It's called with -1 to create an inner map.
+ */
+ emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_7,
+ stack_off(inner_map_fd)));
+ } else if (map_idx != gen->nr_maps) {
+ gen->error = -EDOM; /* internal bug */
+ return;
+ } else {
+ emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_7,
+ stack_off(map_fd[map_idx])));
+ gen->nr_maps++;
+ }
+ if (close_inner_map_fd)
+ emit_sys_close_stack(gen, stack_off(inner_map_fd));
+}
+
+void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *attach_name,
+ enum bpf_attach_type type)
+{
+ const char *prefix;
+ int kind, ret;
+
+ btf_get_kernel_prefix_kind(type, &prefix, &kind);
+ gen->attach_kind = kind;
+ ret = snprintf(gen->attach_target, sizeof(gen->attach_target), "%s%s",
+ prefix, attach_name);
+ if (ret == sizeof(gen->attach_target))
+ gen->error = -ENOSPC;
+}
+
+static void emit_find_attach_target(struct bpf_gen *gen)
+{
+ int name, len = strlen(gen->attach_target) + 1;
+
+ pr_debug("gen: find_attach_tgt %s %d\n", gen->attach_target, gen->attach_kind);
+ name = add_data(gen, gen->attach_target, len);
+
+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
+ 0, 0, 0, name));
+ emit(gen, BPF_MOV64_IMM(BPF_REG_2, len));
+ emit(gen, BPF_MOV64_IMM(BPF_REG_3, gen->attach_kind));
+ emit(gen, BPF_MOV64_IMM(BPF_REG_4, 0));
+ emit(gen, BPF_EMIT_CALL(BPF_FUNC_btf_find_by_name_kind));
+ emit(gen, BPF_MOV64_REG(BPF_REG_7, BPF_REG_0));
+ debug_ret(gen, "find_by_name_kind(%s,%d)",
+ gen->attach_target, gen->attach_kind);
+ emit_check_err(gen);
+ /* if successful, btf_id is in lower 32-bit of R7 and
+ * btf_obj_fd is in upper 32-bit
+ */
+}
+
+void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, int kind,
+ int insn_idx)
+{
+ struct ksym_relo_desc *relo;
+
+ relo = libbpf_reallocarray(gen->relos, gen->relo_cnt + 1, sizeof(*relo));
+ if (!relo) {
+ gen->error = -ENOMEM;
+ return;
+ }
+ gen->relos = relo;
+ relo += gen->relo_cnt;
+ relo->name = name;
+ relo->kind = kind;
+ relo->insn_idx = insn_idx;
+ gen->relo_cnt++;
+}
+
+static void emit_relo(struct bpf_gen *gen, struct ksym_relo_desc *relo, int insns)
+{
+ int name, insn, len = strlen(relo->name) + 1;
+
+ pr_debug("gen: emit_relo: %s at %d\n", relo->name, relo->insn_idx);
+ name = add_data(gen, relo->name, len);
+
+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
+ 0, 0, 0, name));
+ emit(gen, BPF_MOV64_IMM(BPF_REG_2, len));
+ emit(gen, BPF_MOV64_IMM(BPF_REG_3, relo->kind));
+ emit(gen, BPF_MOV64_IMM(BPF_REG_4, 0));
+ emit(gen, BPF_EMIT_CALL(BPF_FUNC_btf_find_by_name_kind));
+ emit(gen, BPF_MOV64_REG(BPF_REG_7, BPF_REG_0));
+ debug_ret(gen, "find_by_name_kind(%s,%d)", relo->name, relo->kind);
+ emit_check_err(gen);
+ /* store btf_id into insn[insn_idx].imm */
+ insn = insns + sizeof(struct bpf_insn) * relo->insn_idx +
+ offsetof(struct bpf_insn, imm);
+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_0, BPF_PSEUDO_MAP_IDX_VALUE,
+ 0, 0, 0, insn));
+ emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_7, 0));
+ if (relo->kind == BTF_KIND_VAR) {
+ /* store btf_obj_fd into insn[insn_idx + 1].imm */
+ emit(gen, BPF_ALU64_IMM(BPF_RSH, BPF_REG_7, 32));
+ emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_7,
+ sizeof(struct bpf_insn)));
+ }
+}
+
+static void emit_relos(struct bpf_gen *gen, int insns)
+{
+ int i;
+
+ for (i = 0; i < gen->relo_cnt; i++)
+ emit_relo(gen, gen->relos + i, insns);
+}
+
+static void cleanup_relos(struct bpf_gen *gen, int insns)
+{
+ int i, insn;
+
+ for (i = 0; i < gen->relo_cnt; i++) {
+ if (gen->relos[i].kind != BTF_KIND_VAR)
+ continue;
+ /* close fd recorded in insn[insn_idx + 1].imm */
+ insn = insns +
+ sizeof(struct bpf_insn) * (gen->relos[i].insn_idx + 1) +
+ offsetof(struct bpf_insn, imm);
+ emit_sys_close_blob(gen, insn);
+ }
+ if (gen->relo_cnt) {
+ free(gen->relos);
+ gen->relo_cnt = 0;
+ gen->relos = NULL;
+ }
+}
+
+void bpf_gen__prog_load(struct bpf_gen *gen,
+ struct bpf_prog_load_params *load_attr, int prog_idx)
+{
+ int attr_size = offsetofend(union bpf_attr, fd_array);
+ int prog_load_attr, license, insns, func_info, line_info;
+ union bpf_attr attr;
+
+ memset(&attr, 0, attr_size);
+ pr_debug("gen: prog_load: type %d insns_cnt %zd\n",
+ load_attr->prog_type, load_attr->insn_cnt);
+ /* add license string to blob of bytes */
+ license = add_data(gen, load_attr->license, strlen(load_attr->license) + 1);
+ /* add insns to blob of bytes */
+ insns = add_data(gen, load_attr->insns,
+ load_attr->insn_cnt * sizeof(struct bpf_insn));
+
+ attr.prog_type = load_attr->prog_type;
+ attr.expected_attach_type = load_attr->expected_attach_type;
+ attr.attach_btf_id = load_attr->attach_btf_id;
+ attr.prog_ifindex = load_attr->prog_ifindex;
+ attr.kern_version = 0;
+ attr.insn_cnt = (__u32)load_attr->insn_cnt;
+ attr.prog_flags = load_attr->prog_flags;
+
+ attr.func_info_rec_size = load_attr->func_info_rec_size;
+ attr.func_info_cnt = load_attr->func_info_cnt;
+ func_info = add_data(gen, load_attr->func_info,
+ attr.func_info_cnt * attr.func_info_rec_size);
+
+ attr.line_info_rec_size = load_attr->line_info_rec_size;
+ attr.line_info_cnt = load_attr->line_info_cnt;
+ line_info = add_data(gen, load_attr->line_info,
+ attr.line_info_cnt * attr.line_info_rec_size);
+
+ memcpy(attr.prog_name, load_attr->name,
+ min((unsigned)strlen(load_attr->name), BPF_OBJ_NAME_LEN - 1));
+ prog_load_attr = add_data(gen, &attr, attr_size);
+
+ /* populate union bpf_attr with a pointer to license */
+ emit_rel_store(gen, attr_field(prog_load_attr, license), license);
+
+ /* populate union bpf_attr with a pointer to instructions */
+ emit_rel_store(gen, attr_field(prog_load_attr, insns), insns);
+
+ /* populate union bpf_attr with a pointer to func_info */
+ emit_rel_store(gen, attr_field(prog_load_attr, func_info), func_info);
+
+ /* populate union bpf_attr with a pointer to line_info */
+ emit_rel_store(gen, attr_field(prog_load_attr, line_info), line_info);
+
+ /* populate union bpf_attr fd_array with a pointer to stack where map_fds are saved */
+ emit_rel_store_sp(gen, attr_field(prog_load_attr, fd_array),
+ stack_off(map_fd[0]));
+
+ /* populate union bpf_attr with user provided log details */
+ move_ctx2blob(gen, attr_field(prog_load_attr, log_level), 4,
+ offsetof(struct bpf_loader_ctx, log_level), false);
+ move_ctx2blob(gen, attr_field(prog_load_attr, log_size), 4,
+ offsetof(struct bpf_loader_ctx, log_size), false);
+ move_ctx2blob(gen, attr_field(prog_load_attr, log_buf), 8,
+ offsetof(struct bpf_loader_ctx, log_buf), false);
+ /* populate union bpf_attr with btf_fd saved in the stack earlier */
+ move_stack2blob(gen, attr_field(prog_load_attr, prog_btf_fd), 4,
+ stack_off(btf_fd));
+ if (gen->attach_kind) {
+ emit_find_attach_target(gen);
+ /* populate union bpf_attr with btf_id and btf_obj_fd found by helper */
+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_0, BPF_PSEUDO_MAP_IDX_VALUE,
+ 0, 0, 0, prog_load_attr));
+ emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_7,
+ offsetof(union bpf_attr, attach_btf_id)));
+ emit(gen, BPF_ALU64_IMM(BPF_RSH, BPF_REG_7, 32));
+ emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_7,
+ offsetof(union bpf_attr, attach_btf_obj_fd)));
+ }
+ emit_relos(gen, insns);
+ /* emit PROG_LOAD command */
+ emit_sys_bpf(gen, BPF_PROG_LOAD, prog_load_attr, attr_size);
+ debug_ret(gen, "prog_load %s insn_cnt %d", attr.prog_name, attr.insn_cnt);
+ /* successful or not, close btf module FDs used in extern ksyms and attach_btf_obj_fd */
+ cleanup_relos(gen, insns);
+ if (gen->attach_kind)
+ emit_sys_close_blob(gen,
+ attr_field(prog_load_attr, attach_btf_obj_fd));
+ emit_check_err(gen);
+ /* remember prog_fd in the stack, if successful */
+ emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_7,
+ stack_off(prog_fd[gen->nr_progs])));
+ gen->nr_progs++;
+}
+
+void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
+ __u32 value_size)
+{
+ int attr_size = offsetofend(union bpf_attr, flags);
+ int map_update_attr, value, key;
+ union bpf_attr attr;
+ int zero = 0;
+
+ memset(&attr, 0, attr_size);
+ pr_debug("gen: map_update_elem: idx %d\n", map_idx);
+
+ value = add_data(gen, pvalue, value_size);
+ key = add_data(gen, &zero, sizeof(zero));
+
+ /* if (map_desc[map_idx].initial_value)
+ * copy_from_user(value, initial_value, value_size);
+ */
+ emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6,
+ sizeof(struct bpf_loader_ctx) +
+ sizeof(struct bpf_map_desc) * map_idx +
+ offsetof(struct bpf_map_desc, initial_value)));
+ emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0, 4));
+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
+ 0, 0, 0, value));
+ emit(gen, BPF_MOV64_IMM(BPF_REG_2, value_size));
+ emit(gen, BPF_EMIT_CALL(BPF_FUNC_copy_from_user));
+
+ map_update_attr = add_data(gen, &attr, attr_size);
+ move_stack2blob(gen, attr_field(map_update_attr, map_fd), 4,
+ stack_off(map_fd[map_idx]));
+ emit_rel_store(gen, attr_field(map_update_attr, key), key);
+ emit_rel_store(gen, attr_field(map_update_attr, value), value);
+ /* emit MAP_UPDATE_ELEM command */
+ emit_sys_bpf(gen, BPF_MAP_UPDATE_ELEM, map_update_attr, attr_size);
+ debug_ret(gen, "update_elem idx %d value_size %d", map_idx, value_size);
+ emit_check_err(gen);
+}
+
+void bpf_gen__map_freeze(struct bpf_gen *gen, int map_idx)
+{
+ int attr_size = offsetofend(union bpf_attr, map_fd);
+ int map_freeze_attr;
+ union bpf_attr attr;
+
+ memset(&attr, 0, attr_size);
+ pr_debug("gen: map_freeze: idx %d\n", map_idx);
+ map_freeze_attr = add_data(gen, &attr, attr_size);
+ move_stack2blob(gen, attr_field(map_freeze_attr, map_fd), 4,
+ stack_off(map_fd[map_idx]));
+ /* emit MAP_FREEZE command */
+ emit_sys_bpf(gen, BPF_MAP_FREEZE, map_freeze_attr, attr_size);
+ debug_ret(gen, "map_freeze");
+ emit_check_err(gen);
+}
#include "str_error.h"
#include "libbpf_internal.h"
#include "hashmap.h"
+#include "bpf_gen_internal.h"
#ifndef BPF_FS_MAGIC
#define BPF_FS_MAGIC 0xcafe4a11
return (__u64) (unsigned long) ptr;
}
+/* this goes away in libbpf 1.0 */
+enum libbpf_strict_mode libbpf_mode = LIBBPF_STRICT_NONE;
+
+int libbpf_set_strict_mode(enum libbpf_strict_mode mode)
+{
+ /* __LIBBPF_STRICT_LAST is the last power-of-2 value used + 1, so to
+ * get all possible values we compensate last +1, and then (2*x - 1)
+ * to get the bit mask
+ */
+ if (mode != LIBBPF_STRICT_ALL
+ && (mode & ~((__LIBBPF_STRICT_LAST - 1) * 2 - 1)))
+ return errno = EINVAL, -EINVAL;
+
+ libbpf_mode = mode;
+ return 0;
+}
+
enum kern_feature_id {
/* v4.14: kernel support for program & map names. */
FEAT_PROG_NAME,
__FEAT_CNT,
};
-static bool kernel_supports(enum kern_feature_id feat_id);
+static bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id);
enum reloc_type {
RELO_LD64,
bool loaded;
bool has_subcalls;
+ struct bpf_gen *gen_loader;
+
/*
* Information when doing elf related work. Only valid if fd
* is valid.
return -LIBBPF_ERRNO__FORMAT;
}
+ if (sec_idx != obj->efile.text_shndx && GELF_ST_BIND(sym.st_info) == STB_LOCAL) {
+ pr_warn("sec '%s': program '%s' is static and not supported\n", sec_name, name);
+ return -ENOTSUP;
+ }
+
pr_debug("sec '%s': found program '%s' at insn offset %zu (%zu bytes), code size %zu insns (%zu bytes)\n",
sec_name, name, sec_off / BPF_INSN_SZ, sec_off, prog_sz / BPF_INSN_SZ, prog_sz);
if (err)
return err;
- /* if function is a global/weak symbol, but has hidden
- * visibility (STV_HIDDEN), mark its BTF FUNC as static to
- * enable more permissive BPF verification mode with more
- * outside context available to BPF verifier
+ /* if function is a global/weak symbol, but has restricted
+ * (STV_HIDDEN or STV_INTERNAL) visibility, mark its BTF FUNC
+ * as static to enable more permissive BPF verification mode
+ * with more outside context available to BPF verifier
*/
if (GELF_ST_BIND(sym.st_info) != STB_LOCAL
- && GELF_ST_VISIBILITY(sym.st_other) == STV_HIDDEN)
+ && (GELF_ST_VISIBILITY(sym.st_other) == STV_HIDDEN
+ || GELF_ST_VISIBILITY(sym.st_other) == STV_INTERNAL))
prog->mark_btf_static = true;
nr_progs++;
if (!symbols)
return -EINVAL;
-
scn = elf_sec_by_idx(obj, obj->efile.maps_shndx);
data = elf_sec_data(obj, scn);
if (!scn || !data) {
return -LIBBPF_ERRNO__FORMAT;
}
+ if (GELF_ST_TYPE(sym.st_info) == STT_SECTION
+ || GELF_ST_BIND(sym.st_info) == STB_LOCAL) {
+ pr_warn("map '%s' (legacy): static maps are not supported\n", map_name);
+ return -ENOTSUP;
+ }
+
map->libbpf_type = LIBBPF_MAP_UNSPEC;
map->sec_idx = sym.st_shndx;
map->sec_offset = sym.st_value;
pr_debug("map '%s': found inner map definition.\n", map->name);
}
+static const char *btf_var_linkage_str(__u32 linkage)
+{
+ switch (linkage) {
+ case BTF_VAR_STATIC: return "static";
+ case BTF_VAR_GLOBAL_ALLOCATED: return "global";
+ case BTF_VAR_GLOBAL_EXTERN: return "extern";
+ default: return "unknown";
+ }
+}
+
static int bpf_object__init_user_btf_map(struct bpf_object *obj,
const struct btf_type *sec,
int var_idx, int sec_idx,
map_name, btf_kind_str(var));
return -EINVAL;
}
- if (var_extra->linkage != BTF_VAR_GLOBAL_ALLOCATED &&
- var_extra->linkage != BTF_VAR_STATIC) {
- pr_warn("map '%s': unsupported var linkage %u.\n",
- map_name, var_extra->linkage);
+ if (var_extra->linkage != BTF_VAR_GLOBAL_ALLOCATED) {
+ pr_warn("map '%s': unsupported map linkage %s.\n",
+ map_name, btf_var_linkage_str(var_extra->linkage));
return -EOPNOTSUPP;
}
err = err ?: bpf_object__init_global_data_maps(obj);
err = err ?: bpf_object__init_kconfig_map(obj);
err = err ?: bpf_object__init_struct_ops_maps(obj);
- if (err)
- return err;
- return 0;
+ return err;
}
static bool section_have_execinstr(struct bpf_object *obj, int idx)
static bool btf_needs_sanitization(struct bpf_object *obj)
{
- bool has_func_global = kernel_supports(FEAT_BTF_GLOBAL_FUNC);
- bool has_datasec = kernel_supports(FEAT_BTF_DATASEC);
- bool has_float = kernel_supports(FEAT_BTF_FLOAT);
- bool has_func = kernel_supports(FEAT_BTF_FUNC);
+ bool has_func_global = kernel_supports(obj, FEAT_BTF_GLOBAL_FUNC);
+ bool has_datasec = kernel_supports(obj, FEAT_BTF_DATASEC);
+ bool has_float = kernel_supports(obj, FEAT_BTF_FLOAT);
+ bool has_func = kernel_supports(obj, FEAT_BTF_FUNC);
return !has_func || !has_datasec || !has_func_global || !has_float;
}
static void bpf_object__sanitize_btf(struct bpf_object *obj, struct btf *btf)
{
- bool has_func_global = kernel_supports(FEAT_BTF_GLOBAL_FUNC);
- bool has_datasec = kernel_supports(FEAT_BTF_DATASEC);
- bool has_float = kernel_supports(FEAT_BTF_FLOAT);
- bool has_func = kernel_supports(FEAT_BTF_FUNC);
+ bool has_func_global = kernel_supports(obj, FEAT_BTF_GLOBAL_FUNC);
+ bool has_datasec = kernel_supports(obj, FEAT_BTF_DATASEC);
+ bool has_float = kernel_supports(obj, FEAT_BTF_FLOAT);
+ bool has_func = kernel_supports(obj, FEAT_BTF_FUNC);
struct btf_type *t;
int i, j, vlen;
if (btf_data) {
obj->btf = btf__new(btf_data->d_buf, btf_data->d_size);
- if (IS_ERR(obj->btf)) {
- err = PTR_ERR(obj->btf);
+ err = libbpf_get_error(obj->btf);
+ if (err) {
obj->btf = NULL;
- pr_warn("Error loading ELF section %s: %d.\n",
- BTF_ELF_SEC, err);
+ pr_warn("Error loading ELF section %s: %d.\n", BTF_ELF_SEC, err);
goto out;
}
/* enforce 8-byte pointers for BPF-targeted BTFs */
btf__set_pointer_size(obj->btf, 8);
- err = 0;
}
if (btf_ext_data) {
if (!obj->btf) {
BTF_EXT_ELF_SEC, BTF_ELF_SEC);
goto out;
}
- obj->btf_ext = btf_ext__new(btf_ext_data->d_buf,
- btf_ext_data->d_size);
- if (IS_ERR(obj->btf_ext)) {
- pr_warn("Error loading ELF section %s: %ld. Ignored and continue.\n",
- BTF_EXT_ELF_SEC, PTR_ERR(obj->btf_ext));
+ obj->btf_ext = btf_ext__new(btf_ext_data->d_buf, btf_ext_data->d_size);
+ err = libbpf_get_error(obj->btf_ext);
+ if (err) {
+ pr_warn("Error loading ELF section %s: %d. Ignored and continue.\n",
+ BTF_EXT_ELF_SEC, err);
obj->btf_ext = NULL;
goto out;
}
int err;
/* btf_vmlinux could be loaded earlier */
- if (obj->btf_vmlinux)
+ if (obj->btf_vmlinux || obj->gen_loader)
return 0;
if (!force && !obj_needs_vmlinux_btf(obj))
return 0;
obj->btf_vmlinux = libbpf_find_kernel_btf();
- if (IS_ERR(obj->btf_vmlinux)) {
- err = PTR_ERR(obj->btf_vmlinux);
+ err = libbpf_get_error(obj->btf_vmlinux);
+ if (err) {
pr_warn("Error loading vmlinux BTF: %d\n", err);
obj->btf_vmlinux = NULL;
return err;
if (!obj->btf)
return 0;
- if (!kernel_supports(FEAT_BTF)) {
+ if (!kernel_supports(obj, FEAT_BTF)) {
if (kernel_needs_btf(obj)) {
err = -EOPNOTSUPP;
goto report;
/* clone BTF to sanitize a copy and leave the original intact */
raw_data = btf__get_raw_data(obj->btf, &sz);
kern_btf = btf__new(raw_data, sz);
- if (IS_ERR(kern_btf))
- return PTR_ERR(kern_btf);
+ err = libbpf_get_error(kern_btf);
+ if (err)
+ return err;
/* enforce 8-byte pointers for BPF-targeted BTFs */
btf__set_pointer_size(obj->btf, 8);
bpf_object__sanitize_btf(obj, kern_btf);
}
- err = btf__load(kern_btf);
+ if (obj->gen_loader) {
+ __u32 raw_size = 0;
+ const void *raw_data = btf__get_raw_data(kern_btf, &raw_size);
+
+ if (!raw_data)
+ return -ENOMEM;
+ bpf_gen__load_btf(obj->gen_loader, raw_data, raw_size);
+ /* Pretend to have valid FD to pass various fd >= 0 checks.
+ * This fd == 0 will not be used with any syscall and will be reset to -1 eventually.
+ */
+ btf__set_fd(kern_btf, 0);
+ } else {
+ err = btf__load(kern_btf);
+ }
if (sanitize) {
if (!err) {
/* move fd to libbpf's BTF */
const struct btf_var_secinfo *vs;
const struct btf_type *sec;
+ if (!btf)
+ return 0;
+
sec_btf_id = btf__find_by_name_kind(btf, KSYMS_SEC,
BTF_KIND_DATASEC);
if (sec_btf_id < 0)
if (pos->sec_name && !strcmp(pos->sec_name, title))
return pos;
}
- return NULL;
+ return errno = ENOENT, NULL;
}
static bool prog_is_subprog(const struct bpf_object *obj,
if (!strcmp(prog->name, name))
return prog;
}
- return NULL;
+ return errno = ENOENT, NULL;
}
static bool bpf_object__shndx_is_data(const struct bpf_object *obj,
err = bpf_obj_get_info_by_fd(fd, &info, &len);
if (err)
- return err;
+ return libbpf_err(err);
new_name = strdup(info.name);
if (!new_name)
- return -errno;
+ return libbpf_err(-errno);
new_fd = open("/", O_RDONLY | O_CLOEXEC);
if (new_fd < 0) {
close(new_fd);
err_free_new_name:
free(new_name);
- return err;
+ return libbpf_err(err);
}
__u32 bpf_map__max_entries(const struct bpf_map *map)
struct bpf_map *bpf_map__inner_map(struct bpf_map *map)
{
if (!bpf_map_type__is_map_in_map(map->def.type))
- return NULL;
+ return errno = EINVAL, NULL;
return map->inner_map;
}
int bpf_map__set_max_entries(struct bpf_map *map, __u32 max_entries)
{
if (map->fd >= 0)
- return -EBUSY;
+ return libbpf_err(-EBUSY);
map->def.max_entries = max_entries;
return 0;
}
int bpf_map__resize(struct bpf_map *map, __u32 max_entries)
{
if (!map || !max_entries)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
return bpf_map__set_max_entries(map, max_entries);
}
};
int ret;
+ if (obj->gen_loader)
+ return 0;
+
/* make sure basic loading works */
memset(&attr, 0, sizeof(attr));
},
};
-static bool kernel_supports(enum kern_feature_id feat_id)
+static bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id)
{
struct kern_feature_desc *feat = &feature_probes[feat_id];
int ret;
+ if (obj->gen_loader)
+ /* To generate loader program assume the latest kernel
+ * to avoid doing extra prog_load, map_create syscalls.
+ */
+ return true;
+
if (READ_ONCE(feat->res) == FEAT_UNKNOWN) {
ret = feat->probe();
if (ret > 0) {
char *cp, errmsg[STRERR_BUFSIZE];
int err, zero = 0;
+ if (obj->gen_loader) {
+ bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps,
+ map->mmaped, map->def.value_size);
+ if (map_type == LIBBPF_MAP_RODATA || map_type == LIBBPF_MAP_KCONFIG)
+ bpf_gen__map_freeze(obj->gen_loader, map - obj->maps);
+ return 0;
+ }
err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0);
if (err) {
err = -errno;
static void bpf_map__destroy(struct bpf_map *map);
-static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map)
+static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, bool is_inner)
{
struct bpf_create_map_attr create_attr;
struct bpf_map_def *def = &map->def;
memset(&create_attr, 0, sizeof(create_attr));
- if (kernel_supports(FEAT_PROG_NAME))
+ if (kernel_supports(obj, FEAT_PROG_NAME))
create_attr.name = map->name;
create_attr.map_ifindex = map->map_ifindex;
create_attr.map_type = def->type;
if (map->inner_map) {
int err;
- err = bpf_object__create_map(obj, map->inner_map);
+ err = bpf_object__create_map(obj, map->inner_map, true);
if (err) {
pr_warn("map '%s': failed to create inner map: %d\n",
map->name, err);
create_attr.inner_map_fd = map->inner_map_fd;
}
- map->fd = bpf_create_map_xattr(&create_attr);
+ if (obj->gen_loader) {
+ bpf_gen__map_create(obj->gen_loader, &create_attr, is_inner ? -1 : map - obj->maps);
+ /* Pretend to have valid FD to pass various fd >= 0 checks.
+ * This fd == 0 will not be used with any syscall and will be reset to -1 eventually.
+ */
+ map->fd = 0;
+ } else {
+ map->fd = bpf_create_map_xattr(&create_attr);
+ }
if (map->fd < 0 && (create_attr.btf_key_type_id ||
create_attr.btf_value_type_id)) {
char *cp, errmsg[STRERR_BUFSIZE];
return -errno;
if (bpf_map_type__is_map_in_map(def->type) && map->inner_map) {
+ if (obj->gen_loader)
+ map->inner_map->fd = -1;
bpf_map__destroy(map->inner_map);
zfree(&map->inner_map);
}
return 0;
}
-static int init_map_slots(struct bpf_map *map)
+static int init_map_slots(struct bpf_object *obj, struct bpf_map *map)
{
const struct bpf_map *targ_map;
unsigned int i;
- int fd, err;
+ int fd, err = 0;
for (i = 0; i < map->init_slots_sz; i++) {
if (!map->init_slots[i])
targ_map = map->init_slots[i];
fd = bpf_map__fd(targ_map);
- err = bpf_map_update_elem(map->fd, &i, &fd, 0);
+ if (obj->gen_loader) {
+ pr_warn("// TODO map_update_elem: idx %td key %d value==map_idx %td\n",
+ map - obj->maps, i, targ_map - obj->maps);
+ return -ENOTSUP;
+ } else {
+ err = bpf_map_update_elem(map->fd, &i, &fd, 0);
+ }
if (err) {
err = -errno;
pr_warn("map '%s': failed to initialize slot [%d] to map '%s' fd=%d: %d\n",
pr_debug("map '%s': skipping creation (preset fd=%d)\n",
map->name, map->fd);
} else {
- err = bpf_object__create_map(obj, map);
+ err = bpf_object__create_map(obj, map, false);
if (err)
goto err_out;
}
if (map->init_slots_sz) {
- err = init_map_slots(map);
+ err = init_map_slots(obj, map);
if (err < 0) {
zclose(map->fd);
goto err_out;
if (obj->btf_modules_loaded)
return 0;
+ if (obj->gen_loader)
+ return 0;
+
/* don't do this again, even if we find no module BTFs */
obj->btf_modules_loaded = true;
/* kernel too old to support module BTFs */
- if (!kernel_supports(FEAT_MODULE_BTF))
+ if (!kernel_supports(obj, FEAT_MODULE_BTF))
return 0;
while (true) {
}
btf = btf_get_from_fd(fd, obj->btf_vmlinux);
- if (IS_ERR(btf)) {
- pr_warn("failed to load module [%s]'s BTF object #%d: %ld\n",
- name, id, PTR_ERR(btf));
- err = PTR_ERR(btf);
+ err = libbpf_get_error(btf);
+ if (err) {
+ pr_warn("failed to load module [%s]'s BTF object #%d: %d\n",
+ name, id, err);
goto err_out;
}
if (str_is_empty(spec_str))
return -EINVAL;
+ if (prog->obj->gen_loader) {
+ pr_warn("// TODO core_relo: prog %td insn[%d] %s %s kind %d\n",
+ prog - prog->obj->programs, relo->insn_off / 8,
+ local_name, spec_str, relo->kind);
+ return -ENOTSUP;
+ }
err = bpf_core_parse_spec(local_btf, local_id, spec_str, relo->kind, &local_spec);
if (err) {
pr_warn("prog '%s': relo #%d: parsing [%d] %s %s + %s failed: %d\n",
if (targ_btf_path) {
obj->btf_vmlinux_override = btf__parse(targ_btf_path, NULL);
- if (IS_ERR_OR_NULL(obj->btf_vmlinux_override)) {
- err = PTR_ERR(obj->btf_vmlinux_override);
+ err = libbpf_get_error(obj->btf_vmlinux_override);
+ if (err) {
pr_warn("failed to parse target BTF: %d\n", err);
return err;
}
switch (relo->type) {
case RELO_LD64:
- insn[0].src_reg = BPF_PSEUDO_MAP_FD;
- insn[0].imm = obj->maps[relo->map_idx].fd;
+ if (obj->gen_loader) {
+ insn[0].src_reg = BPF_PSEUDO_MAP_IDX;
+ insn[0].imm = relo->map_idx;
+ } else {
+ insn[0].src_reg = BPF_PSEUDO_MAP_FD;
+ insn[0].imm = obj->maps[relo->map_idx].fd;
+ }
break;
case RELO_DATA:
- insn[0].src_reg = BPF_PSEUDO_MAP_VALUE;
insn[1].imm = insn[0].imm + relo->sym_off;
- insn[0].imm = obj->maps[relo->map_idx].fd;
+ if (obj->gen_loader) {
+ insn[0].src_reg = BPF_PSEUDO_MAP_IDX_VALUE;
+ insn[0].imm = relo->map_idx;
+ } else {
+ insn[0].src_reg = BPF_PSEUDO_MAP_VALUE;
+ insn[0].imm = obj->maps[relo->map_idx].fd;
+ }
break;
case RELO_EXTERN_VAR:
ext = &obj->externs[relo->sym_off];
if (ext->type == EXT_KCFG) {
- insn[0].src_reg = BPF_PSEUDO_MAP_VALUE;
- insn[0].imm = obj->maps[obj->kconfig_map_idx].fd;
+ if (obj->gen_loader) {
+ insn[0].src_reg = BPF_PSEUDO_MAP_IDX_VALUE;
+ insn[0].imm = obj->kconfig_map_idx;
+ } else {
+ insn[0].src_reg = BPF_PSEUDO_MAP_VALUE;
+ insn[0].imm = obj->maps[obj->kconfig_map_idx].fd;
+ }
insn[1].imm = ext->kcfg.data_off;
} else /* EXT_KSYM */ {
if (ext->ksym.type_id) { /* typed ksyms */
insn[0].imm = ext->ksym.kernel_btf_id;
break;
case RELO_SUBPROG_ADDR:
- insn[0].src_reg = BPF_PSEUDO_FUNC;
- /* will be handled as a follow up pass */
+ if (insn[0].src_reg != BPF_PSEUDO_FUNC) {
+ pr_warn("prog '%s': relo #%d: bad insn\n",
+ prog->name, i);
+ return -EINVAL;
+ }
+ /* handled already */
break;
case RELO_CALL:
- /* will be handled as a follow up pass */
+ /* handled already */
break;
default:
pr_warn("prog '%s': relo #%d: bad relo type %d\n",
/* no .BTF.ext relocation if .BTF.ext is missing or kernel doesn't
* supprot func/line info
*/
- if (!obj->btf_ext || !kernel_supports(FEAT_BTF_FUNC))
+ if (!obj->btf_ext || !kernel_supports(obj, FEAT_BTF_FUNC))
return 0;
/* only attempt func info relocation if main program's func_info
sizeof(*prog->reloc_desc), cmp_relo_by_insn_idx);
}
+static int append_subprog_relos(struct bpf_program *main_prog, struct bpf_program *subprog)
+{
+ int new_cnt = main_prog->nr_reloc + subprog->nr_reloc;
+ struct reloc_desc *relos;
+ int i;
+
+ if (main_prog == subprog)
+ return 0;
+ relos = libbpf_reallocarray(main_prog->reloc_desc, new_cnt, sizeof(*relos));
+ if (!relos)
+ return -ENOMEM;
+ memcpy(relos + main_prog->nr_reloc, subprog->reloc_desc,
+ sizeof(*relos) * subprog->nr_reloc);
+
+ for (i = main_prog->nr_reloc; i < new_cnt; i++)
+ relos[i].insn_idx += subprog->sub_insn_off;
+ /* After insn_idx adjustment the 'relos' array is still sorted
+ * by insn_idx and doesn't break bsearch.
+ */
+ main_prog->reloc_desc = relos;
+ main_prog->nr_reloc = new_cnt;
+ return 0;
+}
+
static int
bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog,
struct bpf_program *prog)
continue;
relo = find_prog_insn_relo(prog, insn_idx);
+ if (relo && relo->type == RELO_EXTERN_FUNC)
+ /* kfunc relocations will be handled later
+ * in bpf_object__relocate_data()
+ */
+ continue;
if (relo && relo->type != RELO_CALL && relo->type != RELO_SUBPROG_ADDR) {
pr_warn("prog '%s': unexpected relo for insn #%zu, type %d\n",
prog->name, insn_idx, relo->type);
pr_debug("prog '%s': added %zu insns from sub-prog '%s'\n",
main_prog->name, subprog->insns_cnt, subprog->name);
+ /* The subprog insns are now appended. Append its relos too. */
+ err = append_subprog_relos(main_prog, subprog);
+ if (err)
+ return err;
err = bpf_object__reloc_code(obj, main_prog, subprog);
if (err)
return err;
return 0;
}
+static void
+bpf_object__free_relocs(struct bpf_object *obj)
+{
+ struct bpf_program *prog;
+ int i;
+
+ /* free up relocation descriptors */
+ for (i = 0; i < obj->nr_programs; i++) {
+ prog = &obj->programs[i];
+ zfree(&prog->reloc_desc);
+ prog->nr_reloc = 0;
+ }
+}
+
static int
bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_path)
{
struct bpf_program *prog;
- size_t i;
+ size_t i, j;
int err;
if (obj->btf_ext) {
return err;
}
}
- /* relocate data references first for all programs and sub-programs,
- * as they don't change relative to code locations, so subsequent
- * subprogram processing won't need to re-calculate any of them
+
+ /* Before relocating calls pre-process relocations and mark
+ * few ld_imm64 instructions that points to subprogs.
+ * Otherwise bpf_object__reloc_code() later would have to consider
+ * all ld_imm64 insns as relocation candidates. That would
+ * reduce relocation speed, since amount of find_prog_insn_relo()
+ * would increase and most of them will fail to find a relo.
*/
for (i = 0; i < obj->nr_programs; i++) {
prog = &obj->programs[i];
- err = bpf_object__relocate_data(obj, prog);
- if (err) {
- pr_warn("prog '%s': failed to relocate data references: %d\n",
- prog->name, err);
- return err;
+ for (j = 0; j < prog->nr_reloc; j++) {
+ struct reloc_desc *relo = &prog->reloc_desc[j];
+ struct bpf_insn *insn = &prog->insns[relo->insn_idx];
+
+ /* mark the insn, so it's recognized by insn_is_pseudo_func() */
+ if (relo->type == RELO_SUBPROG_ADDR)
+ insn[0].src_reg = BPF_PSEUDO_FUNC;
}
}
- /* now relocate subprogram calls and append used subprograms to main
+
+ /* relocate subprogram calls and append used subprograms to main
* programs; each copy of subprogram code needs to be relocated
* differently for each main program, because its code location might
- * have changed
+ * have changed.
+ * Append subprog relos to main programs to allow data relos to be
+ * processed after text is completely relocated.
*/
for (i = 0; i < obj->nr_programs; i++) {
prog = &obj->programs[i];
return err;
}
}
- /* free up relocation descriptors */
+ /* Process data relos for main programs */
for (i = 0; i < obj->nr_programs; i++) {
prog = &obj->programs[i];
- zfree(&prog->reloc_desc);
- prog->nr_reloc = 0;
+ if (prog_is_subprog(obj, prog))
+ continue;
+ err = bpf_object__relocate_data(obj, prog);
+ if (err) {
+ pr_warn("prog '%s': failed to relocate data references: %d\n",
+ prog->name, err);
+ return err;
+ }
}
+ if (!obj->gen_loader)
+ bpf_object__free_relocs(obj);
return 0;
}
enum bpf_func_id func_id;
int i;
+ if (obj->gen_loader)
+ return 0;
+
for (i = 0; i < prog->insns_cnt; i++, insn++) {
if (!insn_is_helper_call(insn, &func_id))
continue;
switch (func_id) {
case BPF_FUNC_probe_read_kernel:
case BPF_FUNC_probe_read_user:
- if (!kernel_supports(FEAT_PROBE_READ_KERN))
+ if (!kernel_supports(obj, FEAT_PROBE_READ_KERN))
insn->imm = BPF_FUNC_probe_read;
break;
case BPF_FUNC_probe_read_kernel_str:
case BPF_FUNC_probe_read_user_str:
- if (!kernel_supports(FEAT_PROBE_READ_KERN))
+ if (!kernel_supports(obj, FEAT_PROBE_READ_KERN))
insn->imm = BPF_FUNC_probe_read_str;
break;
default:
load_attr.prog_type = prog->type;
/* old kernels might not support specifying expected_attach_type */
- if (!kernel_supports(FEAT_EXP_ATTACH_TYPE) && prog->sec_def &&
+ if (!kernel_supports(prog->obj, FEAT_EXP_ATTACH_TYPE) && prog->sec_def &&
prog->sec_def->is_exp_attach_type_optional)
load_attr.expected_attach_type = 0;
else
load_attr.expected_attach_type = prog->expected_attach_type;
- if (kernel_supports(FEAT_PROG_NAME))
+ if (kernel_supports(prog->obj, FEAT_PROG_NAME))
load_attr.name = prog->name;
load_attr.insns = insns;
load_attr.insn_cnt = insns_cnt;
/* specify func_info/line_info only if kernel supports them */
btf_fd = bpf_object__btf_fd(prog->obj);
- if (btf_fd >= 0 && kernel_supports(FEAT_BTF_FUNC)) {
+ if (btf_fd >= 0 && kernel_supports(prog->obj, FEAT_BTF_FUNC)) {
load_attr.prog_btf_fd = btf_fd;
load_attr.func_info = prog->func_info;
load_attr.func_info_rec_size = prog->func_info_rec_size;
load_attr.log_level = prog->log_level;
load_attr.prog_flags = prog->prog_flags;
+ if (prog->obj->gen_loader) {
+ bpf_gen__prog_load(prog->obj->gen_loader, &load_attr,
+ prog - prog->obj->programs);
+ *pfd = -1;
+ return 0;
+ }
retry_load:
if (log_buf_size) {
log_buf = malloc(log_buf_size);
pr_debug("verifier log:\n%s", log_buf);
if (prog->obj->rodata_map_idx >= 0 &&
- kernel_supports(FEAT_PROG_BIND_MAP)) {
+ kernel_supports(prog->obj, FEAT_PROG_BIND_MAP)) {
struct bpf_map *rodata_map =
&prog->obj->maps[prog->obj->rodata_map_idx];
return ret;
}
+static int bpf_program__record_externs(struct bpf_program *prog)
+{
+ struct bpf_object *obj = prog->obj;
+ int i;
+
+ for (i = 0; i < prog->nr_reloc; i++) {
+ struct reloc_desc *relo = &prog->reloc_desc[i];
+ struct extern_desc *ext = &obj->externs[relo->sym_off];
+
+ switch (relo->type) {
+ case RELO_EXTERN_VAR:
+ if (ext->type != EXT_KSYM)
+ continue;
+ if (!ext->ksym.type_id) {
+ pr_warn("typeless ksym %s is not supported yet\n",
+ ext->name);
+ return -ENOTSUP;
+ }
+ bpf_gen__record_extern(obj->gen_loader, ext->name, BTF_KIND_VAR,
+ relo->insn_idx);
+ break;
+ case RELO_EXTERN_FUNC:
+ bpf_gen__record_extern(obj->gen_loader, ext->name, BTF_KIND_FUNC,
+ relo->insn_idx);
+ break;
+ default:
+ continue;
+ }
+ }
+ return 0;
+}
+
static int libbpf_find_attach_btf_id(struct bpf_program *prog, int *btf_obj_fd, int *btf_type_id);
int bpf_program__load(struct bpf_program *prog, char *license, __u32 kern_ver)
if (prog->obj->loaded) {
pr_warn("prog '%s': can't load after object was loaded\n", prog->name);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
if ((prog->type == BPF_PROG_TYPE_TRACING ||
err = libbpf_find_attach_btf_id(prog, &btf_obj_fd, &btf_type_id);
if (err)
- return err;
+ return libbpf_err(err);
prog->attach_btf_obj_fd = btf_obj_fd;
prog->attach_btf_id = btf_type_id;
if (prog->preprocessor) {
pr_warn("Internal error: can't load program '%s'\n",
prog->name);
- return -LIBBPF_ERRNO__INTERNAL;
+ return libbpf_err(-LIBBPF_ERRNO__INTERNAL);
}
prog->instances.fds = malloc(sizeof(int));
if (!prog->instances.fds) {
pr_warn("Not enough memory for BPF fds\n");
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
}
prog->instances.nr = 1;
prog->instances.fds[0] = -1;
pr_warn("prog '%s': inconsistent nr(%d) != 1\n",
prog->name, prog->instances.nr);
}
+ if (prog->obj->gen_loader)
+ bpf_program__record_externs(prog);
err = load_program(prog, prog->insns, prog->insns_cnt,
license, kern_ver, &fd);
if (!err)
pr_warn("failed to load program '%s'\n", prog->name);
zfree(&prog->insns);
prog->insns_cnt = 0;
- return err;
+ return libbpf_err(err);
}
static int
if (err)
return err;
}
+ if (obj->gen_loader)
+ bpf_object__free_relocs(obj);
return 0;
}
struct bpf_object *bpf_object__open_xattr(struct bpf_object_open_attr *attr)
{
- return __bpf_object__open_xattr(attr, 0);
+ return libbpf_ptr(__bpf_object__open_xattr(attr, 0));
}
struct bpf_object *bpf_object__open(const char *path)
.prog_type = BPF_PROG_TYPE_UNSPEC,
};
- return bpf_object__open_xattr(&attr);
+ return libbpf_ptr(__bpf_object__open_xattr(&attr, 0));
}
struct bpf_object *
bpf_object__open_file(const char *path, const struct bpf_object_open_opts *opts)
{
if (!path)
- return ERR_PTR(-EINVAL);
+ return libbpf_err_ptr(-EINVAL);
pr_debug("loading %s\n", path);
- return __bpf_object__open(path, NULL, 0, opts);
+ return libbpf_ptr(__bpf_object__open(path, NULL, 0, opts));
}
struct bpf_object *
const struct bpf_object_open_opts *opts)
{
if (!obj_buf || obj_buf_sz == 0)
- return ERR_PTR(-EINVAL);
+ return libbpf_err_ptr(-EINVAL);
- return __bpf_object__open(NULL, obj_buf, obj_buf_sz, opts);
+ return libbpf_ptr(__bpf_object__open(NULL, obj_buf, obj_buf_sz, opts));
}
struct bpf_object *
/* returning NULL is wrong, but backwards-compatible */
if (!obj_buf || obj_buf_sz == 0)
- return NULL;
+ return errno = EINVAL, NULL;
- return bpf_object__open_mem(obj_buf, obj_buf_sz, &opts);
+ return libbpf_ptr(__bpf_object__open(NULL, obj_buf, obj_buf_sz, &opts));
}
int bpf_object__unload(struct bpf_object *obj)
size_t i;
if (!obj)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
for (i = 0; i < obj->nr_maps; i++) {
zclose(obj->maps[i].fd);
bpf_object__for_each_map(m, obj) {
if (!bpf_map__is_internal(m))
continue;
- if (!kernel_supports(FEAT_GLOBAL_DATA)) {
+ if (!kernel_supports(obj, FEAT_GLOBAL_DATA)) {
pr_warn("kernel doesn't support global data\n");
return -ENOTSUP;
}
- if (!kernel_supports(FEAT_ARRAY_MMAP))
+ if (!kernel_supports(obj, FEAT_ARRAY_MMAP))
m->def.map_flags ^= BPF_F_MMAPABLE;
}
if (ext->type != EXT_KSYM || !ext->ksym.type_id)
continue;
+ if (obj->gen_loader) {
+ ext->is_set = true;
+ ext->ksym.kernel_btf_obj_fd = 0;
+ ext->ksym.kernel_btf_id = 0;
+ continue;
+ }
t = btf__type_by_id(obj->btf, ext->btf_id);
if (btf_is_var(t))
err = bpf_object__resolve_ksym_var_btf_id(obj, ext);
int err, i;
if (!attr)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
obj = attr->obj;
if (!obj)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (obj->loaded) {
pr_warn("object '%s': load can't be attempted twice\n", obj->name);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
+ if (obj->gen_loader)
+ bpf_gen__init(obj->gen_loader, attr->log_level);
+
err = bpf_object__probe_loading(obj);
err = err ? : bpf_object__load_vmlinux_btf(obj, false);
err = err ? : bpf_object__resolve_externs(obj, obj->kconfig);
err = err ? : bpf_object__relocate(obj, attr->target_btf_path);
err = err ? : bpf_object__load_progs(obj, attr->log_level);
+ if (obj->gen_loader) {
+ /* reset FDs */
+ btf__set_fd(obj->btf, -1);
+ for (i = 0; i < obj->nr_maps; i++)
+ obj->maps[i].fd = -1;
+ if (!err)
+ err = bpf_gen__finish(obj->gen_loader);
+ }
+
/* clean up module BTFs */
for (i = 0; i < obj->btf_module_cnt; i++) {
close(obj->btf_modules[i].fd);
bpf_object__unload(obj);
pr_warn("failed to load object '%s'\n", obj->path);
- return err;
+ return libbpf_err(err);
}
int bpf_object__load(struct bpf_object *obj)
err = make_parent_dir(path);
if (err)
- return err;
+ return libbpf_err(err);
err = check_path(path);
if (err)
- return err;
+ return libbpf_err(err);
if (prog == NULL) {
pr_warn("invalid program pointer\n");
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
if (instance < 0 || instance >= prog->instances.nr) {
pr_warn("invalid prog instance %d of prog %s (max %d)\n",
instance, prog->name, prog->instances.nr);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
if (bpf_obj_pin(prog->instances.fds[instance], path)) {
err = -errno;
cp = libbpf_strerror_r(err, errmsg, sizeof(errmsg));
pr_warn("failed to pin program: %s\n", cp);
- return err;
+ return libbpf_err(err);
}
pr_debug("pinned program '%s'\n", path);
err = check_path(path);
if (err)
- return err;
+ return libbpf_err(err);
if (prog == NULL) {
pr_warn("invalid program pointer\n");
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
if (instance < 0 || instance >= prog->instances.nr) {
pr_warn("invalid prog instance %d of prog %s (max %d)\n",
instance, prog->name, prog->instances.nr);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
err = unlink(path);
if (err != 0)
- return -errno;
+ return libbpf_err(-errno);
+
pr_debug("unpinned program '%s'\n", path);
return 0;
err = make_parent_dir(path);
if (err)
- return err;
+ return libbpf_err(err);
err = check_path(path);
if (err)
- return err;
+ return libbpf_err(err);
if (prog == NULL) {
pr_warn("invalid program pointer\n");
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
if (prog->instances.nr <= 0) {
pr_warn("no instances of prog %s to pin\n", prog->name);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
if (prog->instances.nr == 1) {
rmdir(path);
- return err;
+ return libbpf_err(err);
}
int bpf_program__unpin(struct bpf_program *prog, const char *path)
err = check_path(path);
if (err)
- return err;
+ return libbpf_err(err);
if (prog == NULL) {
pr_warn("invalid program pointer\n");
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
if (prog->instances.nr <= 0) {
pr_warn("no instances of prog %s to pin\n", prog->name);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
if (prog->instances.nr == 1) {
len = snprintf(buf, PATH_MAX, "%s/%d", path, i);
if (len < 0)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
else if (len >= PATH_MAX)
- return -ENAMETOOLONG;
+ return libbpf_err(-ENAMETOOLONG);
err = bpf_program__unpin_instance(prog, buf, i);
if (err)
err = rmdir(path);
if (err)
- return -errno;
+ return libbpf_err(-errno);
return 0;
}
if (map == NULL) {
pr_warn("invalid map pointer\n");
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
if (map->pin_path) {
if (path && strcmp(path, map->pin_path)) {
pr_warn("map '%s' already has pin path '%s' different from '%s'\n",
bpf_map__name(map), map->pin_path, path);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
} else if (map->pinned) {
pr_debug("map '%s' already pinned at '%s'; not re-pinning\n",
bpf_map__name(map), map->pin_path);
if (!path) {
pr_warn("missing a path to pin map '%s' at\n",
bpf_map__name(map));
- return -EINVAL;
+ return libbpf_err(-EINVAL);
} else if (map->pinned) {
pr_warn("map '%s' already pinned\n", bpf_map__name(map));
- return -EEXIST;
+ return libbpf_err(-EEXIST);
}
map->pin_path = strdup(path);
err = make_parent_dir(map->pin_path);
if (err)
- return err;
+ return libbpf_err(err);
err = check_path(map->pin_path);
if (err)
- return err;
+ return libbpf_err(err);
if (bpf_obj_pin(map->fd, map->pin_path)) {
err = -errno;
out_err:
cp = libbpf_strerror_r(-err, errmsg, sizeof(errmsg));
pr_warn("failed to pin map: %s\n", cp);
- return err;
+ return libbpf_err(err);
}
int bpf_map__unpin(struct bpf_map *map, const char *path)
if (map == NULL) {
pr_warn("invalid map pointer\n");
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
if (map->pin_path) {
if (path && strcmp(path, map->pin_path)) {
pr_warn("map '%s' already has pin path '%s' different from '%s'\n",
bpf_map__name(map), map->pin_path, path);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
path = map->pin_path;
} else if (!path) {
pr_warn("no path to unpin map '%s' from\n",
bpf_map__name(map));
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
err = check_path(path);
if (err)
- return err;
+ return libbpf_err(err);
err = unlink(path);
if (err != 0)
- return -errno;
+ return libbpf_err(-errno);
map->pinned = false;
pr_debug("unpinned map '%s' from '%s'\n", bpf_map__name(map), path);
if (path) {
new = strdup(path);
if (!new)
- return -errno;
+ return libbpf_err(-errno);
}
free(map->pin_path);
int err;
if (!obj)
- return -ENOENT;
+ return libbpf_err(-ENOENT);
if (!obj->loaded) {
pr_warn("object not yet loaded; load it first\n");
- return -ENOENT;
+ return libbpf_err(-ENOENT);
}
bpf_object__for_each_map(map, obj) {
bpf_map__unpin(map, NULL);
}
- return err;
+ return libbpf_err(err);
}
int bpf_object__unpin_maps(struct bpf_object *obj, const char *path)
int err;
if (!obj)
- return -ENOENT;
+ return libbpf_err(-ENOENT);
bpf_object__for_each_map(map, obj) {
char *pin_path = NULL;
len = snprintf(buf, PATH_MAX, "%s/%s", path,
bpf_map__name(map));
if (len < 0)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
else if (len >= PATH_MAX)
- return -ENAMETOOLONG;
+ return libbpf_err(-ENAMETOOLONG);
sanitize_pin_path(buf);
pin_path = buf;
} else if (!map->pin_path) {
err = bpf_map__unpin(map, pin_path);
if (err)
- return err;
+ return libbpf_err(err);
}
return 0;
int err;
if (!obj)
- return -ENOENT;
+ return libbpf_err(-ENOENT);
if (!obj->loaded) {
pr_warn("object not yet loaded; load it first\n");
- return -ENOENT;
+ return libbpf_err(-ENOENT);
}
bpf_object__for_each_program(prog, obj) {
bpf_program__unpin(prog, buf);
}
- return err;
+ return libbpf_err(err);
}
int bpf_object__unpin_programs(struct bpf_object *obj, const char *path)
int err;
if (!obj)
- return -ENOENT;
+ return libbpf_err(-ENOENT);
bpf_object__for_each_program(prog, obj) {
char buf[PATH_MAX];
len = snprintf(buf, PATH_MAX, "%s/%s", path,
prog->pin_name);
if (len < 0)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
else if (len >= PATH_MAX)
- return -ENAMETOOLONG;
+ return libbpf_err(-ENAMETOOLONG);
err = bpf_program__unpin(prog, buf);
if (err)
- return err;
+ return libbpf_err(err);
}
return 0;
err = bpf_object__pin_maps(obj, path);
if (err)
- return err;
+ return libbpf_err(err);
err = bpf_object__pin_programs(obj, path);
if (err) {
bpf_object__unpin_maps(obj, path);
- return err;
+ return libbpf_err(err);
}
return 0;
if (obj->clear_priv)
obj->clear_priv(obj, obj->priv);
+ bpf_gen__free(obj->gen_loader);
bpf_object__elf_finish(obj);
bpf_object__unload(obj);
btf__free(obj->btf);
const char *bpf_object__name(const struct bpf_object *obj)
{
- return obj ? obj->name : ERR_PTR(-EINVAL);
+ return obj ? obj->name : libbpf_err_ptr(-EINVAL);
}
unsigned int bpf_object__kversion(const struct bpf_object *obj)
int bpf_object__set_kversion(struct bpf_object *obj, __u32 kern_version)
{
if (obj->loaded)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
obj->kern_version = kern_version;
void *bpf_object__priv(const struct bpf_object *obj)
{
- return obj ? obj->priv : ERR_PTR(-EINVAL);
+ return obj ? obj->priv : libbpf_err_ptr(-EINVAL);
+}
+
+int bpf_object__gen_loader(struct bpf_object *obj, struct gen_loader_opts *opts)
+{
+ struct bpf_gen *gen;
+
+ if (!opts)
+ return -EFAULT;
+ if (!OPTS_VALID(opts, gen_loader_opts))
+ return -EINVAL;
+ gen = calloc(sizeof(*gen), 1);
+ if (!gen)
+ return -ENOMEM;
+ gen->opts = opts;
+ obj->gen_loader = gen;
+ return 0;
}
static struct bpf_program *
if (p->obj != obj) {
pr_warn("error: program handler doesn't match object\n");
- return NULL;
+ return errno = EINVAL, NULL;
}
idx = (p - obj->programs) + (forward ? 1 : -1);
void *bpf_program__priv(const struct bpf_program *prog)
{
- return prog ? prog->priv : ERR_PTR(-EINVAL);
+ return prog ? prog->priv : libbpf_err_ptr(-EINVAL);
}
void bpf_program__set_ifindex(struct bpf_program *prog, __u32 ifindex)
title = strdup(title);
if (!title) {
pr_warn("failed to strdup program title\n");
- return ERR_PTR(-ENOMEM);
+ return libbpf_err_ptr(-ENOMEM);
}
}
int bpf_program__set_autoload(struct bpf_program *prog, bool autoload)
{
if (prog->obj->loaded)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
prog->load = autoload;
return 0;
int *instances_fds;
if (nr_instances <= 0 || !prep)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (prog->instances.nr > 0 || prog->instances.fds) {
pr_warn("Can't set pre-processor after loading\n");
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
instances_fds = malloc(sizeof(int) * nr_instances);
if (!instances_fds) {
pr_warn("alloc memory failed for fds\n");
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
}
/* fill all fd with -1 */
int fd;
if (!prog)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (n >= prog->instances.nr || n < 0) {
pr_warn("Can't get the %dth fd from program %s: only %d instances\n",
n, prog->name, prog->instances.nr);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
fd = prog->instances.fds[n];
if (fd < 0) {
pr_warn("%dth instance of program '%s' is invalid\n",
n, prog->name);
- return -ENOENT;
+ return libbpf_err(-ENOENT);
}
return fd;
int bpf_program__set_##NAME(struct bpf_program *prog) \
{ \
if (!prog) \
- return -EINVAL; \
+ return libbpf_err(-EINVAL); \
bpf_program__set_type(prog, TYPE); \
return 0; \
} \
static const struct bpf_sec_def section_defs[] = {
BPF_PROG_SEC("socket", BPF_PROG_TYPE_SOCKET_FILTER),
- BPF_PROG_SEC("sk_reuseport", BPF_PROG_TYPE_SK_REUSEPORT),
+ BPF_EAPROG_SEC("sk_reuseport/migrate", BPF_PROG_TYPE_SK_REUSEPORT,
+ BPF_SK_REUSEPORT_SELECT_OR_MIGRATE),
+ BPF_EAPROG_SEC("sk_reuseport", BPF_PROG_TYPE_SK_REUSEPORT,
+ BPF_SK_REUSEPORT_SELECT),
SEC_DEF("kprobe/", KPROBE,
.attach_fn = attach_kprobe),
BPF_PROG_SEC("uprobe/", BPF_PROG_TYPE_KPROBE),
.expected_attach_type = BPF_TRACE_ITER,
.is_attach_btf = true,
.attach_fn = attach_iter),
+ SEC_DEF("syscall", SYSCALL,
+ .is_sleepable = true),
BPF_EAPROG_SEC("xdp_devmap/", BPF_PROG_TYPE_XDP,
BPF_XDP_DEVMAP),
BPF_EAPROG_SEC("xdp_cpumap/", BPF_PROG_TYPE_XDP,
char *type_names;
if (!name)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
sec_def = find_sec_def(name);
if (sec_def) {
free(type_names);
}
- return -ESRCH;
+ return libbpf_err(-ESRCH);
}
static struct bpf_map *find_struct_ops_map_by_offset(struct bpf_object *obj,
#define BTF_ITER_PREFIX "bpf_iter_"
#define BTF_MAX_NAME_SIZE 128
+void btf_get_kernel_prefix_kind(enum bpf_attach_type attach_type,
+ const char **prefix, int *kind)
+{
+ switch (attach_type) {
+ case BPF_TRACE_RAW_TP:
+ *prefix = BTF_TRACE_PREFIX;
+ *kind = BTF_KIND_TYPEDEF;
+ break;
+ case BPF_LSM_MAC:
+ *prefix = BTF_LSM_PREFIX;
+ *kind = BTF_KIND_FUNC;
+ break;
+ case BPF_TRACE_ITER:
+ *prefix = BTF_ITER_PREFIX;
+ *kind = BTF_KIND_FUNC;
+ break;
+ default:
+ *prefix = "";
+ *kind = BTF_KIND_FUNC;
+ }
+}
+
static int find_btf_by_prefix_kind(const struct btf *btf, const char *prefix,
const char *name, __u32 kind)
{
static inline int find_attach_btf_id(struct btf *btf, const char *name,
enum bpf_attach_type attach_type)
{
- int err;
+ const char *prefix;
+ int kind;
- if (attach_type == BPF_TRACE_RAW_TP)
- err = find_btf_by_prefix_kind(btf, BTF_TRACE_PREFIX, name,
- BTF_KIND_TYPEDEF);
- else if (attach_type == BPF_LSM_MAC)
- err = find_btf_by_prefix_kind(btf, BTF_LSM_PREFIX, name,
- BTF_KIND_FUNC);
- else if (attach_type == BPF_TRACE_ITER)
- err = find_btf_by_prefix_kind(btf, BTF_ITER_PREFIX, name,
- BTF_KIND_FUNC);
- else
- err = btf__find_by_name_kind(btf, name, BTF_KIND_FUNC);
-
- return err;
+ btf_get_kernel_prefix_kind(attach_type, &prefix, &kind);
+ return find_btf_by_prefix_kind(btf, prefix, name, kind);
}
int libbpf_find_vmlinux_btf_id(const char *name,
int err;
btf = libbpf_find_kernel_btf();
- if (IS_ERR(btf)) {
+ err = libbpf_get_error(btf);
+ if (err) {
pr_warn("vmlinux BTF is not found\n");
- return -EINVAL;
+ return libbpf_err(err);
}
err = find_attach_btf_id(btf, name, attach_type);
pr_warn("%s is not found in vmlinux BTF\n", name);
btf__free(btf);
- return err;
+ return libbpf_err(err);
}
static int libbpf_find_prog_btf_id(const char *name, __u32 attach_prog_fd)
int err = -EINVAL;
info_linear = bpf_program__get_prog_info_linear(attach_prog_fd, 0);
- if (IS_ERR_OR_NULL(info_linear)) {
+ err = libbpf_get_error(info_linear);
+ if (err) {
pr_warn("failed get_prog_info_linear for FD %d\n",
attach_prog_fd);
- return -EINVAL;
+ return err;
}
info = &info_linear->info;
if (!info->btf_id) {
__u32 attach_prog_fd = prog->attach_prog_fd;
const char *name = prog->sec_name, *attach_name;
const struct bpf_sec_def *sec = NULL;
- int i, err;
+ int i, err = 0;
if (!name)
return -EINVAL;
}
/* kernel/module BTF ID */
- err = find_kernel_btf_id(prog->obj, attach_name, attach_type, btf_obj_fd, btf_type_id);
+ if (prog->obj->gen_loader) {
+ bpf_gen__record_attach_target(prog->obj->gen_loader, attach_name, attach_type);
+ *btf_obj_fd = 0;
+ *btf_type_id = 1;
+ } else {
+ err = find_kernel_btf_id(prog->obj, attach_name, attach_type, btf_obj_fd, btf_type_id);
+ }
if (err) {
pr_warn("failed to find kernel BTF type ID of '%s': %d\n", attach_name, err);
return err;
int i;
if (!name)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
for (i = 0; i < ARRAY_SIZE(section_defs); i++) {
if (strncmp(name, section_defs[i].sec, section_defs[i].len))
continue;
if (!section_defs[i].is_attachable)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
*attach_type = section_defs[i].expected_attach_type;
return 0;
}
free(type_names);
}
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
int bpf_map__fd(const struct bpf_map *map)
{
- return map ? map->fd : -EINVAL;
+ return map ? map->fd : libbpf_err(-EINVAL);
}
const struct bpf_map_def *bpf_map__def(const struct bpf_map *map)
{
- return map ? &map->def : ERR_PTR(-EINVAL);
+ return map ? &map->def : libbpf_err_ptr(-EINVAL);
}
const char *bpf_map__name(const struct bpf_map *map)
int bpf_map__set_type(struct bpf_map *map, enum bpf_map_type type)
{
if (map->fd >= 0)
- return -EBUSY;
+ return libbpf_err(-EBUSY);
map->def.type = type;
return 0;
}
int bpf_map__set_map_flags(struct bpf_map *map, __u32 flags)
{
if (map->fd >= 0)
- return -EBUSY;
+ return libbpf_err(-EBUSY);
map->def.map_flags = flags;
return 0;
}
int bpf_map__set_numa_node(struct bpf_map *map, __u32 numa_node)
{
if (map->fd >= 0)
- return -EBUSY;
+ return libbpf_err(-EBUSY);
map->numa_node = numa_node;
return 0;
}
int bpf_map__set_key_size(struct bpf_map *map, __u32 size)
{
if (map->fd >= 0)
- return -EBUSY;
+ return libbpf_err(-EBUSY);
map->def.key_size = size;
return 0;
}
int bpf_map__set_value_size(struct bpf_map *map, __u32 size)
{
if (map->fd >= 0)
- return -EBUSY;
+ return libbpf_err(-EBUSY);
map->def.value_size = size;
return 0;
}
bpf_map_clear_priv_t clear_priv)
{
if (!map)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (map->priv) {
if (map->clear_priv)
void *bpf_map__priv(const struct bpf_map *map)
{
- return map ? map->priv : ERR_PTR(-EINVAL);
+ return map ? map->priv : libbpf_err_ptr(-EINVAL);
}
int bpf_map__set_initial_value(struct bpf_map *map,
{
if (!map->mmaped || map->libbpf_type == LIBBPF_MAP_KCONFIG ||
size != map->def.value_size || map->fd >= 0)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
memcpy(map->mmaped, data, size);
return 0;
}
+const void *bpf_map__initial_value(struct bpf_map *map, size_t *psize)
+{
+ if (!map->mmaped)
+ return NULL;
+ *psize = map->def.value_size;
+ return map->mmaped;
+}
+
bool bpf_map__is_offload_neutral(const struct bpf_map *map)
{
return map->def.type == BPF_MAP_TYPE_PERF_EVENT_ARRAY;
int bpf_map__set_ifindex(struct bpf_map *map, __u32 ifindex)
{
if (map->fd >= 0)
- return -EBUSY;
+ return libbpf_err(-EBUSY);
map->map_ifindex = ifindex;
return 0;
}
{
if (!bpf_map_type__is_map_in_map(map->def.type)) {
pr_warn("error: unsupported map type\n");
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
if (map->inner_map_fd != -1) {
pr_warn("error: inner_map_fd already specified\n");
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
zfree(&map->inner_map);
map->inner_map_fd = fd;
struct bpf_map *s, *e;
if (!obj || !obj->maps)
- return NULL;
+ return errno = EINVAL, NULL;
s = obj->maps;
e = obj->maps + obj->nr_maps;
if ((m < s) || (m >= e)) {
pr_warn("error in %s: map handler doesn't belong to object\n",
__func__);
- return NULL;
+ return errno = EINVAL, NULL;
}
idx = (m - obj->maps) + i;
if (pos->name && !strcmp(pos->name, name))
return pos;
}
- return NULL;
+ return errno = ENOENT, NULL;
}
int
struct bpf_map *
bpf_object__find_map_by_offset(struct bpf_object *obj, size_t offset)
{
- return ERR_PTR(-ENOTSUP);
+ return libbpf_err_ptr(-ENOTSUP);
}
long libbpf_get_error(const void *ptr)
{
- return PTR_ERR_OR_ZERO(ptr);
+ if (!IS_ERR_OR_NULL(ptr))
+ return 0;
+
+ if (IS_ERR(ptr))
+ errno = -PTR_ERR(ptr);
+
+ /* If ptr == NULL, then errno should be already set by the failing
+ * API, because libbpf never returns NULL on success and it now always
+ * sets errno on error. So no extra errno handling for ptr == NULL
+ * case.
+ */
+ return -errno;
}
int bpf_prog_load(const char *file, enum bpf_prog_type type,
int err;
if (!attr)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (!attr->file)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
open_attr.file = attr->file;
open_attr.prog_type = attr->prog_type;
obj = bpf_object__open_xattr(&open_attr);
- if (IS_ERR_OR_NULL(obj))
- return -ENOENT;
+ err = libbpf_get_error(obj);
+ if (err)
+ return libbpf_err(-ENOENT);
bpf_object__for_each_program(prog, obj) {
enum bpf_attach_type attach_type = attr->expected_attach_type;
* didn't provide a fallback type, too bad...
*/
bpf_object__close(obj);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
prog->prog_ifindex = attr->ifindex;
if (!first_prog) {
pr_warn("object file doesn't contain bpf program\n");
bpf_object__close(obj);
- return -ENOENT;
+ return libbpf_err(-ENOENT);
}
err = bpf_object__load(obj);
if (err) {
bpf_object__close(obj);
- return err;
+ return libbpf_err(err);
}
*pobj = obj;
/* Replace link's underlying BPF program with the new one */
int bpf_link__update_program(struct bpf_link *link, struct bpf_program *prog)
{
- return bpf_link_update(bpf_link__fd(link), bpf_program__fd(prog), NULL);
+ int ret;
+
+ ret = bpf_link_update(bpf_link__fd(link), bpf_program__fd(prog), NULL);
+ return libbpf_err_errno(ret);
}
/* Release "ownership" of underlying BPF resource (typically, BPF program
free(link->pin_path);
free(link);
- return err;
+ return libbpf_err(err);
}
int bpf_link__fd(const struct bpf_link *link)
static int bpf_link__detach_fd(struct bpf_link *link)
{
- return close(link->fd);
+ return libbpf_err_errno(close(link->fd));
}
struct bpf_link *bpf_link__open(const char *path)
if (fd < 0) {
fd = -errno;
pr_warn("failed to open link at %s: %d\n", path, fd);
- return ERR_PTR(fd);
+ return libbpf_err_ptr(fd);
}
link = calloc(1, sizeof(*link));
if (!link) {
close(fd);
- return ERR_PTR(-ENOMEM);
+ return libbpf_err_ptr(-ENOMEM);
}
link->detach = &bpf_link__detach_fd;
link->fd = fd;
link->pin_path = strdup(path);
if (!link->pin_path) {
bpf_link__destroy(link);
- return ERR_PTR(-ENOMEM);
+ return libbpf_err_ptr(-ENOMEM);
}
return link;
int err;
if (link->pin_path)
- return -EBUSY;
+ return libbpf_err(-EBUSY);
err = make_parent_dir(path);
if (err)
- return err;
+ return libbpf_err(err);
err = check_path(path);
if (err)
- return err;
+ return libbpf_err(err);
link->pin_path = strdup(path);
if (!link->pin_path)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
if (bpf_obj_pin(link->fd, link->pin_path)) {
err = -errno;
zfree(&link->pin_path);
- return err;
+ return libbpf_err(err);
}
pr_debug("link fd=%d: pinned at %s\n", link->fd, link->pin_path);
int err;
if (!link->pin_path)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
err = unlink(link->pin_path);
if (err != 0)
- return -errno;
+ return libbpf_err_errno(err);
pr_debug("link fd=%d: unpinned from %s\n", link->fd, link->pin_path);
zfree(&link->pin_path);
err = -errno;
close(link->fd);
- return err;
+ return libbpf_err(err);
}
-struct bpf_link *bpf_program__attach_perf_event(struct bpf_program *prog,
- int pfd)
+struct bpf_link *bpf_program__attach_perf_event(struct bpf_program *prog, int pfd)
{
char errmsg[STRERR_BUFSIZE];
struct bpf_link *link;
if (pfd < 0) {
pr_warn("prog '%s': invalid perf event FD %d\n",
prog->name, pfd);
- return ERR_PTR(-EINVAL);
+ return libbpf_err_ptr(-EINVAL);
}
prog_fd = bpf_program__fd(prog);
if (prog_fd < 0) {
pr_warn("prog '%s': can't attach BPF program w/o FD (did you load it?)\n",
prog->name);
- return ERR_PTR(-EINVAL);
+ return libbpf_err_ptr(-EINVAL);
}
link = calloc(1, sizeof(*link));
if (!link)
- return ERR_PTR(-ENOMEM);
+ return libbpf_err_ptr(-ENOMEM);
link->detach = &bpf_link__detach_perf_event;
link->fd = pfd;
if (err == -EPROTO)
pr_warn("prog '%s': try add PERF_SAMPLE_CALLCHAIN to or remove exclude_callchain_[kernel|user] from pfd %d\n",
prog->name, pfd);
- return ERR_PTR(err);
+ return libbpf_err_ptr(err);
}
if (ioctl(pfd, PERF_EVENT_IOC_ENABLE, 0) < 0) {
err = -errno;
free(link);
pr_warn("prog '%s': failed to enable pfd %d: %s\n",
prog->name, pfd, libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
- return ERR_PTR(err);
+ return libbpf_err_ptr(err);
}
return link;
}
pr_warn("prog '%s': failed to create %s '%s' perf event: %s\n",
prog->name, retprobe ? "kretprobe" : "kprobe", func_name,
libbpf_strerror_r(pfd, errmsg, sizeof(errmsg)));
- return ERR_PTR(pfd);
+ return libbpf_err_ptr(pfd);
}
link = bpf_program__attach_perf_event(prog, pfd);
- if (IS_ERR(link)) {
+ err = libbpf_get_error(link);
+ if (err) {
close(pfd);
- err = PTR_ERR(link);
pr_warn("prog '%s': failed to attach to %s '%s': %s\n",
prog->name, retprobe ? "kretprobe" : "kprobe", func_name,
libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
- return link;
+ return libbpf_err_ptr(err);
}
return link;
}
prog->name, retprobe ? "uretprobe" : "uprobe",
binary_path, func_offset,
libbpf_strerror_r(pfd, errmsg, sizeof(errmsg)));
- return ERR_PTR(pfd);
+ return libbpf_err_ptr(pfd);
}
link = bpf_program__attach_perf_event(prog, pfd);
- if (IS_ERR(link)) {
+ err = libbpf_get_error(link);
+ if (err) {
close(pfd);
- err = PTR_ERR(link);
pr_warn("prog '%s': failed to attach to %s '%s:0x%zx': %s\n",
prog->name, retprobe ? "uretprobe" : "uprobe",
binary_path, func_offset,
libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
- return link;
+ return libbpf_err_ptr(err);
}
return link;
}
pr_warn("prog '%s': failed to create tracepoint '%s/%s' perf event: %s\n",
prog->name, tp_category, tp_name,
libbpf_strerror_r(pfd, errmsg, sizeof(errmsg)));
- return ERR_PTR(pfd);
+ return libbpf_err_ptr(pfd);
}
link = bpf_program__attach_perf_event(prog, pfd);
- if (IS_ERR(link)) {
+ err = libbpf_get_error(link);
+ if (err) {
close(pfd);
- err = PTR_ERR(link);
pr_warn("prog '%s': failed to attach to tracepoint '%s/%s': %s\n",
prog->name, tp_category, tp_name,
libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
- return link;
+ return libbpf_err_ptr(err);
}
return link;
}
sec_name = strdup(prog->sec_name);
if (!sec_name)
- return ERR_PTR(-ENOMEM);
+ return libbpf_err_ptr(-ENOMEM);
/* extract "tp/<category>/<name>" */
tp_cat = sec_name + sec->len;
tp_name = strchr(tp_cat, '/');
if (!tp_name) {
- link = ERR_PTR(-EINVAL);
- goto out;
+ free(sec_name);
+ return libbpf_err_ptr(-EINVAL);
}
*tp_name = '\0';
tp_name++;
link = bpf_program__attach_tracepoint(prog, tp_cat, tp_name);
-out:
free(sec_name);
return link;
}
prog_fd = bpf_program__fd(prog);
if (prog_fd < 0) {
pr_warn("prog '%s': can't attach before loaded\n", prog->name);
- return ERR_PTR(-EINVAL);
+ return libbpf_err_ptr(-EINVAL);
}
link = calloc(1, sizeof(*link));
if (!link)
- return ERR_PTR(-ENOMEM);
+ return libbpf_err_ptr(-ENOMEM);
link->detach = &bpf_link__detach_fd;
pfd = bpf_raw_tracepoint_open(tp_name, prog_fd);
free(link);
pr_warn("prog '%s': failed to attach to raw tracepoint '%s': %s\n",
prog->name, tp_name, libbpf_strerror_r(pfd, errmsg, sizeof(errmsg)));
- return ERR_PTR(pfd);
+ return libbpf_err_ptr(pfd);
}
link->fd = pfd;
return link;
prog_fd = bpf_program__fd(prog);
if (prog_fd < 0) {
pr_warn("prog '%s': can't attach before loaded\n", prog->name);
- return ERR_PTR(-EINVAL);
+ return libbpf_err_ptr(-EINVAL);
}
link = calloc(1, sizeof(*link));
if (!link)
- return ERR_PTR(-ENOMEM);
+ return libbpf_err_ptr(-ENOMEM);
link->detach = &bpf_link__detach_fd;
pfd = bpf_raw_tracepoint_open(NULL, prog_fd);
free(link);
pr_warn("prog '%s': failed to attach: %s\n",
prog->name, libbpf_strerror_r(pfd, errmsg, sizeof(errmsg)));
- return ERR_PTR(pfd);
+ return libbpf_err_ptr(pfd);
}
link->fd = pfd;
return (struct bpf_link *)link;
return bpf_program__attach_lsm(prog);
}
-static struct bpf_link *attach_iter(const struct bpf_sec_def *sec,
- struct bpf_program *prog)
-{
- return bpf_program__attach_iter(prog, NULL);
-}
-
static struct bpf_link *
bpf_program__attach_fd(struct bpf_program *prog, int target_fd, int btf_id,
const char *target_name)
prog_fd = bpf_program__fd(prog);
if (prog_fd < 0) {
pr_warn("prog '%s': can't attach before loaded\n", prog->name);
- return ERR_PTR(-EINVAL);
+ return libbpf_err_ptr(-EINVAL);
}
link = calloc(1, sizeof(*link));
if (!link)
- return ERR_PTR(-ENOMEM);
+ return libbpf_err_ptr(-ENOMEM);
link->detach = &bpf_link__detach_fd;
attach_type = bpf_program__get_expected_attach_type(prog);
pr_warn("prog '%s': failed to attach to %s: %s\n",
prog->name, target_name,
libbpf_strerror_r(link_fd, errmsg, sizeof(errmsg)));
- return ERR_PTR(link_fd);
+ return libbpf_err_ptr(link_fd);
}
link->fd = link_fd;
return link;
if (!!target_fd != !!attach_func_name) {
pr_warn("prog '%s': supply none or both of target_fd and attach_func_name\n",
prog->name);
- return ERR_PTR(-EINVAL);
+ return libbpf_err_ptr(-EINVAL);
}
if (prog->type != BPF_PROG_TYPE_EXT) {
pr_warn("prog '%s': only BPF_PROG_TYPE_EXT can attach as freplace",
prog->name);
- return ERR_PTR(-EINVAL);
+ return libbpf_err_ptr(-EINVAL);
}
if (target_fd) {
btf_id = libbpf_find_prog_btf_id(attach_func_name, target_fd);
if (btf_id < 0)
- return ERR_PTR(btf_id);
+ return libbpf_err_ptr(btf_id);
return bpf_program__attach_fd(prog, target_fd, btf_id, "freplace");
} else {
__u32 target_fd = 0;
if (!OPTS_VALID(opts, bpf_iter_attach_opts))
- return ERR_PTR(-EINVAL);
+ return libbpf_err_ptr(-EINVAL);
link_create_opts.iter_info = OPTS_GET(opts, link_info, (void *)0);
link_create_opts.iter_info_len = OPTS_GET(opts, link_info_len, 0);
prog_fd = bpf_program__fd(prog);
if (prog_fd < 0) {
pr_warn("prog '%s': can't attach before loaded\n", prog->name);
- return ERR_PTR(-EINVAL);
+ return libbpf_err_ptr(-EINVAL);
}
link = calloc(1, sizeof(*link));
if (!link)
- return ERR_PTR(-ENOMEM);
+ return libbpf_err_ptr(-ENOMEM);
link->detach = &bpf_link__detach_fd;
link_fd = bpf_link_create(prog_fd, target_fd, BPF_TRACE_ITER,
free(link);
pr_warn("prog '%s': failed to attach to iterator: %s\n",
prog->name, libbpf_strerror_r(link_fd, errmsg, sizeof(errmsg)));
- return ERR_PTR(link_fd);
+ return libbpf_err_ptr(link_fd);
}
link->fd = link_fd;
return link;
}
+static struct bpf_link *attach_iter(const struct bpf_sec_def *sec,
+ struct bpf_program *prog)
+{
+ return bpf_program__attach_iter(prog, NULL);
+}
+
struct bpf_link *bpf_program__attach(struct bpf_program *prog)
{
const struct bpf_sec_def *sec_def;
sec_def = find_sec_def(prog->sec_name);
if (!sec_def || !sec_def->attach_fn)
- return ERR_PTR(-ESRCH);
+ return libbpf_err_ptr(-ESRCH);
return sec_def->attach_fn(sec_def, prog);
}
int err;
if (!bpf_map__is_struct_ops(map) || map->fd == -1)
- return ERR_PTR(-EINVAL);
+ return libbpf_err_ptr(-EINVAL);
link = calloc(1, sizeof(*link));
if (!link)
- return ERR_PTR(-EINVAL);
+ return libbpf_err_ptr(-EINVAL);
st_ops = map->st_ops;
for (i = 0; i < btf_vlen(st_ops->type); i++) {
if (err) {
err = -errno;
free(link);
- return ERR_PTR(err);
+ return libbpf_err_ptr(err);
}
link->detach = bpf_link__detach_struct_ops;
}
ring_buffer_write_tail(header, data_tail);
- return ret;
+ return libbpf_err(ret);
}
struct perf_buffer;
p.lost_cb = opts ? opts->lost_cb : NULL;
p.ctx = opts ? opts->ctx : NULL;
- return __perf_buffer__new(map_fd, page_cnt, &p);
+ return libbpf_ptr(__perf_buffer__new(map_fd, page_cnt, &p));
}
struct perf_buffer *
p.cpus = opts->cpus;
p.map_keys = opts->map_keys;
- return __perf_buffer__new(map_fd, page_cnt, &p);
+ return libbpf_ptr(__perf_buffer__new(map_fd, page_cnt, &p));
}
static struct perf_buffer *__perf_buffer__new(int map_fd, size_t page_cnt,
int i, cnt, err;
cnt = epoll_wait(pb->epoll_fd, pb->events, pb->cpu_cnt, timeout_ms);
+ if (cnt < 0)
+ return libbpf_err_errno(cnt);
+
for (i = 0; i < cnt; i++) {
struct perf_cpu_buf *cpu_buf = pb->events[i].data.ptr;
err = perf_buffer__process_records(pb, cpu_buf);
if (err) {
pr_warn("error while processing records: %d\n", err);
- return err;
+ return libbpf_err(err);
}
}
- return cnt < 0 ? -errno : cnt;
+ return cnt;
}
/* Return number of PERF_EVENT_ARRAY map slots set up by this perf_buffer
struct perf_cpu_buf *cpu_buf;
if (buf_idx >= pb->cpu_cnt)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
cpu_buf = pb->cpu_bufs[buf_idx];
if (!cpu_buf)
- return -ENOENT;
+ return libbpf_err(-ENOENT);
return cpu_buf->fd;
}
struct perf_cpu_buf *cpu_buf;
if (buf_idx >= pb->cpu_cnt)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
cpu_buf = pb->cpu_bufs[buf_idx];
if (!cpu_buf)
- return -ENOENT;
+ return libbpf_err(-ENOENT);
return perf_buffer__process_records(pb, cpu_buf);
}
err = perf_buffer__process_records(pb, cpu_buf);
if (err) {
pr_warn("perf_buffer: failed to process records in buffer #%d: %d\n", i, err);
- return err;
+ return libbpf_err(err);
}
}
return 0;
void *ptr;
if (arrays >> BPF_PROG_INFO_LAST_ARRAY)
- return ERR_PTR(-EINVAL);
+ return libbpf_err_ptr(-EINVAL);
/* step 1: get array dimensions */
err = bpf_obj_get_info_by_fd(fd, &info, &info_len);
if (err) {
pr_debug("can't get prog info: %s", strerror(errno));
- return ERR_PTR(-EFAULT);
+ return libbpf_err_ptr(-EFAULT);
}
/* step 2: calculate total size of all arrays */
data_len = roundup(data_len, sizeof(__u64));
info_linear = malloc(sizeof(struct bpf_prog_info_linear) + data_len);
if (!info_linear)
- return ERR_PTR(-ENOMEM);
+ return libbpf_err_ptr(-ENOMEM);
/* step 4: fill data to info_linear->info */
info_linear->arrays = arrays;
if (err) {
pr_debug("can't get prog info: %s", strerror(errno));
free(info_linear);
- return ERR_PTR(-EFAULT);
+ return libbpf_err_ptr(-EFAULT);
}
/* step 6: verify the data */
int btf_obj_fd = 0, btf_id = 0, err;
if (!prog || attach_prog_fd < 0 || !attach_func_name)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (prog->obj->loaded)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (attach_prog_fd) {
btf_id = libbpf_find_prog_btf_id(attach_func_name,
attach_prog_fd);
if (btf_id < 0)
- return btf_id;
+ return libbpf_err(btf_id);
} else {
/* load btf_vmlinux, if not yet */
err = bpf_object__load_vmlinux_btf(prog->obj, true);
if (err)
- return err;
+ return libbpf_err(err);
err = find_kernel_btf_id(prog->obj, attach_func_name,
prog->expected_attach_type,
&btf_obj_fd, &btf_id);
if (err)
- return err;
+ return libbpf_err(err);
}
prog->attach_btf_id = btf_id;
err = parse_cpu_mask_file(fcpu, &mask, &n);
if (err)
- return err;
+ return libbpf_err(err);
tmp_cpus = 0;
for (i = 0; i < n; i++) {
.object_name = s->name,
);
struct bpf_object *obj;
- int i;
+ int i, err;
/* Attempt to preserve opts->object_name, unless overriden by user
* explicitly. Overwriting object name for skeletons is discouraged,
}
obj = bpf_object__open_mem(s->data, s->data_sz, &skel_opts);
- if (IS_ERR(obj)) {
- pr_warn("failed to initialize skeleton BPF object '%s': %ld\n",
- s->name, PTR_ERR(obj));
- return PTR_ERR(obj);
+ err = libbpf_get_error(obj);
+ if (err) {
+ pr_warn("failed to initialize skeleton BPF object '%s': %d\n",
+ s->name, err);
+ return libbpf_err(err);
}
*s->obj = obj;
*map = bpf_object__find_map_by_name(obj, name);
if (!*map) {
pr_warn("failed to find skeleton map '%s'\n", name);
- return -ESRCH;
+ return libbpf_err(-ESRCH);
}
/* externs shouldn't be pre-setup from user code */
*prog = bpf_object__find_program_by_name(obj, name);
if (!*prog) {
pr_warn("failed to find skeleton program '%s'\n", name);
- return -ESRCH;
+ return libbpf_err(-ESRCH);
}
}
err = bpf_object__load(*s->obj);
if (err) {
pr_warn("failed to load BPF skeleton '%s': %d\n", s->name, err);
- return err;
+ return libbpf_err(err);
}
for (i = 0; i < s->map_cnt; i++) {
*mmaped = NULL;
pr_warn("failed to re-mmap() map '%s': %d\n",
bpf_map__name(map), err);
- return err;
+ return libbpf_err(err);
}
}
int bpf_object__attach_skeleton(struct bpf_object_skeleton *s)
{
- int i;
+ int i, err;
for (i = 0; i < s->prog_cnt; i++) {
struct bpf_program *prog = *s->progs[i].prog;
continue;
*link = sec_def->attach_fn(sec_def, prog);
- if (IS_ERR(*link)) {
- pr_warn("failed to auto-attach program '%s': %ld\n",
- bpf_program__name(prog), PTR_ERR(*link));
- return PTR_ERR(*link);
+ err = libbpf_get_error(*link);
+ if (err) {
+ pr_warn("failed to auto-attach program '%s': %d\n",
+ bpf_program__name(prog), err);
+ return libbpf_err(err);
}
}
#include <linux/bpf.h>
#include "libbpf_common.h"
+#include "libbpf_legacy.h"
#ifdef __cplusplus
extern "C" {
LIBBPF_API void *bpf_map__priv(const struct bpf_map *map);
LIBBPF_API int bpf_map__set_initial_value(struct bpf_map *map,
const void *data, size_t size);
+LIBBPF_API const void *bpf_map__initial_value(struct bpf_map *map, size_t *psize);
LIBBPF_API bool bpf_map__is_offload_neutral(const struct bpf_map *map);
LIBBPF_API bool bpf_map__is_internal(const struct bpf_map *map);
LIBBPF_API int bpf_map__set_pin_path(struct bpf_map *map, const char *path);
LIBBPF_API int bpf_prog_load(const char *file, enum bpf_prog_type type,
struct bpf_object **pobj, int *prog_fd);
+/* XDP related API */
struct xdp_link_info {
__u32 prog_id;
__u32 drv_prog_id;
LIBBPF_API int bpf_get_link_xdp_info(int ifindex, struct xdp_link_info *info,
size_t info_size, __u32 flags);
+/* TC related API */
+enum bpf_tc_attach_point {
+ BPF_TC_INGRESS = 1 << 0,
+ BPF_TC_EGRESS = 1 << 1,
+ BPF_TC_CUSTOM = 1 << 2,
+};
+
+#define BPF_TC_PARENT(a, b) \
+ ((((a) << 16) & 0xFFFF0000U) | ((b) & 0x0000FFFFU))
+
+enum bpf_tc_flags {
+ BPF_TC_F_REPLACE = 1 << 0,
+};
+
+struct bpf_tc_hook {
+ size_t sz;
+ int ifindex;
+ enum bpf_tc_attach_point attach_point;
+ __u32 parent;
+ size_t :0;
+};
+#define bpf_tc_hook__last_field parent
+
+struct bpf_tc_opts {
+ size_t sz;
+ int prog_fd;
+ __u32 flags;
+ __u32 prog_id;
+ __u32 handle;
+ __u32 priority;
+ size_t :0;
+};
+#define bpf_tc_opts__last_field priority
+
+LIBBPF_API int bpf_tc_hook_create(struct bpf_tc_hook *hook);
+LIBBPF_API int bpf_tc_hook_destroy(struct bpf_tc_hook *hook);
+LIBBPF_API int bpf_tc_attach(const struct bpf_tc_hook *hook,
+ struct bpf_tc_opts *opts);
+LIBBPF_API int bpf_tc_detach(const struct bpf_tc_hook *hook,
+ const struct bpf_tc_opts *opts);
+LIBBPF_API int bpf_tc_query(const struct bpf_tc_hook *hook,
+ struct bpf_tc_opts *opts);
+
/* Ring buffer APIs */
struct ring_buffer;
LIBBPF_API void bpf_object__detach_skeleton(struct bpf_object_skeleton *s);
LIBBPF_API void bpf_object__destroy_skeleton(struct bpf_object_skeleton *s);
+struct gen_loader_opts {
+ size_t sz; /* size of this struct, for forward/backward compatiblity */
+ const char *data;
+ const char *insns;
+ __u32 data_sz;
+ __u32 insns_sz;
+};
+
+#define gen_loader_opts__last_field insns_sz
+LIBBPF_API int bpf_object__gen_loader(struct bpf_object *obj,
+ struct gen_loader_opts *opts);
+
enum libbpf_tristate {
TRI_NO = 0,
TRI_YES = 1,
};
#define bpf_linker_opts__last_field sz
+struct bpf_linker_file_opts {
+ /* size of this struct, for forward/backward compatiblity */
+ size_t sz;
+};
+#define bpf_linker_file_opts__last_field sz
+
struct bpf_linker;
LIBBPF_API struct bpf_linker *bpf_linker__new(const char *filename, struct bpf_linker_opts *opts);
-LIBBPF_API int bpf_linker__add_file(struct bpf_linker *linker, const char *filename);
+LIBBPF_API int bpf_linker__add_file(struct bpf_linker *linker,
+ const char *filename,
+ const struct bpf_linker_file_opts *opts);
LIBBPF_API int bpf_linker__finalize(struct bpf_linker *linker);
LIBBPF_API void bpf_linker__free(struct bpf_linker *linker);
bpf_linker__new;
bpf_map__inner_map;
bpf_object__set_kversion;
+ bpf_tc_attach;
+ bpf_tc_detach;
+ bpf_tc_hook_create;
+ bpf_tc_hook_destroy;
+ bpf_tc_query;
} LIBBPF_0.3.0;
+
+LIBBPF_0.5.0 {
+ global:
+ bpf_map__initial_value;
+ bpf_map_lookup_and_delete_elem_flags;
+ bpf_object__gen_loader;
+ libbpf_set_strict_mode;
+} LIBBPF_0.4.0;
#include <string.h>
#include "libbpf.h"
+#include "libbpf_internal.h"
/* make sure libbpf doesn't use kernel-only integer typedefs */
#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
int libbpf_strerror(int err, char *buf, size_t size)
{
if (!buf || !size)
- return -1;
+ return libbpf_err(-EINVAL);
err = err > 0 ? err : -err;
ret = strerror_r(err, buf, size);
buf[size - 1] = '\0';
- return ret;
+ return libbpf_err_errno(ret);
}
if (err < __LIBBPF_ERRNO__END) {
snprintf(buf, size, "Unknown libbpf error %d", err);
buf[size - 1] = '\0';
- return -1;
+ return libbpf_err(-ENOENT);
}
#include <stdlib.h>
#include <limits.h>
+#include <errno.h>
+#include <linux/err.h>
+#include "libbpf_legacy.h"
/* make sure libbpf doesn't use kernel-only integer typedefs */
#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
#ifndef R_BPF_64_64
#define R_BPF_64_64 1
#endif
+#ifndef R_BPF_64_ABS64
+#define R_BPF_64_ABS64 2
+#endif
+#ifndef R_BPF_64_ABS32
+#define R_BPF_64_ABS32 3
+#endif
#ifndef R_BPF_64_32
#define R_BPF_64_32 10
#endif
#define ELF_C_READ_MMAP ELF_C_READ
#endif
+/* Older libelf all end up in this expression, for both 32 and 64 bit */
+#ifndef GELF_ST_VISIBILITY
+#define GELF_ST_VISIBILITY(o) ((o) & 0x03)
+#endif
+
#define BTF_INFO_ENC(kind, kind_flag, vlen) \
((!!(kind_flag) << 31) | ((kind) << 24) | ((vlen) & BTF_MAX_VLEN))
#define BTF_TYPE_ENC(name, info, size_or_type) (name), (info), (size_or_type)
int bpf_object__variable_offset(const struct bpf_object *obj, const char *name,
__u32 *off);
struct btf *btf_get_from_fd(int btf_fd, struct btf *base_btf);
+void btf_get_kernel_prefix_kind(enum bpf_attach_type attach_type,
+ const char **prefix, int *kind);
struct btf_ext_info {
/*
int btf_ext_visit_type_ids(struct btf_ext *btf_ext, type_id_visit_fn visit, void *ctx);
int btf_ext_visit_str_offs(struct btf_ext *btf_ext, str_off_visit_fn visit, void *ctx);
+extern enum libbpf_strict_mode libbpf_mode;
+
+/* handle direct returned errors */
+static inline int libbpf_err(int ret)
+{
+ if (ret < 0)
+ errno = -ret;
+ return ret;
+}
+
+/* handle errno-based (e.g., syscall or libc) errors according to libbpf's
+ * strict mode settings
+ */
+static inline int libbpf_err_errno(int ret)
+{
+ if (libbpf_mode & LIBBPF_STRICT_DIRECT_ERRS)
+ /* errno is already assumed to be set on error */
+ return ret < 0 ? -errno : ret;
+
+ /* legacy: on error return -1 directly and don't touch errno */
+ return ret;
+}
+
+/* handle error for pointer-returning APIs, err is assumed to be < 0 always */
+static inline void *libbpf_err_ptr(int err)
+{
+ /* set errno on error, this doesn't break anything */
+ errno = -err;
+
+ if (libbpf_mode & LIBBPF_STRICT_CLEAN_PTRS)
+ return NULL;
+
+ /* legacy: encode err as ptr */
+ return ERR_PTR(err);
+}
+
+/* handle pointer-returning APIs' error handling */
+static inline void *libbpf_ptr(void *ret)
+{
+ /* set errno on error, this doesn't break anything */
+ if (IS_ERR(ret))
+ errno = -PTR_ERR(ret);
+
+ if (libbpf_mode & LIBBPF_STRICT_CLEAN_PTRS)
+ return IS_ERR(ret) ? NULL : ret;
+
+ /* legacy: pass-through original pointer */
+ return ret;
+}
+
#endif /* __LIBBPF_LIBBPF_INTERNAL_H */
--- /dev/null
+/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
+
+/*
+ * Libbpf legacy APIs (either discouraged or deprecated, as mentioned in [0])
+ *
+ * [0] https://docs.google.com/document/d/1UyjTZuPFWiPFyKk1tV5an11_iaRuec6U-ZESZ54nNTY
+ *
+ * Copyright (C) 2021 Facebook
+ */
+#ifndef __LIBBPF_LEGACY_BPF_H
+#define __LIBBPF_LEGACY_BPF_H
+
+#include <linux/bpf.h>
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+#include "libbpf_common.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+enum libbpf_strict_mode {
+ /* Turn on all supported strict features of libbpf to simulate libbpf
+ * v1.0 behavior.
+ * This will be the default behavior in libbpf v1.0.
+ */
+ LIBBPF_STRICT_ALL = 0xffffffff,
+
+ /*
+ * Disable any libbpf 1.0 behaviors. This is the default before libbpf
+ * v1.0. It won't be supported anymore in v1.0, please update your
+ * code so that it handles LIBBPF_STRICT_ALL mode before libbpf v1.0.
+ */
+ LIBBPF_STRICT_NONE = 0x00,
+ /*
+ * Return NULL pointers on error, not ERR_PTR(err).
+ * Additionally, libbpf also always sets errno to corresponding Exx
+ * (positive) error code.
+ */
+ LIBBPF_STRICT_CLEAN_PTRS = 0x01,
+ /*
+ * Return actual error codes from low-level APIs directly, not just -1.
+ * Additionally, libbpf also always sets errno to corresponding Exx
+ * (positive) error code.
+ */
+ LIBBPF_STRICT_DIRECT_ERRS = 0x02,
+
+ __LIBBPF_STRICT_LAST,
+};
+
+LIBBPF_API int libbpf_set_strict_mode(enum libbpf_strict_mode mode);
+
+
+#ifdef __cplusplus
+} /* extern "C" */
+#endif
+
+#endif /* __LIBBPF_LEGACY_BPF_H */
static int init_output_elf(struct bpf_linker *linker, const char *file);
-static int linker_load_obj_file(struct bpf_linker *linker, const char *filename, struct src_obj *obj);
+static int linker_load_obj_file(struct bpf_linker *linker, const char *filename,
+ const struct bpf_linker_file_opts *opts,
+ struct src_obj *obj);
static int linker_sanity_check_elf(struct src_obj *obj);
static int linker_sanity_check_elf_symtab(struct src_obj *obj, struct src_sec *sec);
static int linker_sanity_check_elf_relos(struct src_obj *obj, struct src_sec *sec);
int err;
if (!OPTS_VALID(opts, bpf_linker_opts))
- return NULL;
+ return errno = EINVAL, NULL;
if (elf_version(EV_CURRENT) == EV_NONE) {
pr_warn_elf("libelf initialization failed");
- return NULL;
+ return errno = EINVAL, NULL;
}
linker = calloc(1, sizeof(*linker));
if (!linker)
- return NULL;
+ return errno = ENOMEM, NULL;
linker->fd = -1;
err_out:
bpf_linker__free(linker);
- return NULL;
+ return errno = -err, NULL;
}
static struct dst_sec *add_dst_sec(struct bpf_linker *linker, const char *sec_name)
return 0;
}
-int bpf_linker__add_file(struct bpf_linker *linker, const char *filename)
+int bpf_linker__add_file(struct bpf_linker *linker, const char *filename,
+ const struct bpf_linker_file_opts *opts)
{
struct src_obj obj = {};
int err = 0;
+ if (!OPTS_VALID(opts, bpf_linker_file_opts))
+ return libbpf_err(-EINVAL);
+
if (!linker->elf)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
- err = err ?: linker_load_obj_file(linker, filename, &obj);
+ err = err ?: linker_load_obj_file(linker, filename, opts, &obj);
err = err ?: linker_append_sec_data(linker, &obj);
err = err ?: linker_append_elf_syms(linker, &obj);
err = err ?: linker_append_elf_relos(linker, &obj);
if (obj.fd >= 0)
close(obj.fd);
- return err;
+ return libbpf_err(err);
}
static bool is_dwarf_sec_name(const char *name)
return sec;
}
-static int linker_load_obj_file(struct bpf_linker *linker, const char *filename, struct src_obj *obj)
+static int linker_load_obj_file(struct bpf_linker *linker, const char *filename,
+ const struct bpf_linker_file_opts *opts,
+ struct src_obj *obj)
{
#if __BYTE_ORDER == __LITTLE_ENDIAN
const int host_endianness = ELFDATA2LSB;
size_t sym_idx = ELF64_R_SYM(relo->r_info);
size_t sym_type = ELF64_R_TYPE(relo->r_info);
- if (sym_type != R_BPF_64_64 && sym_type != R_BPF_64_32) {
+ if (sym_type != R_BPF_64_64 && sym_type != R_BPF_64_32 &&
+ sym_type != R_BPF_64_ABS64 && sym_type != R_BPF_64_ABS32) {
pr_warn("ELF relo #%d in section #%zu has unexpected type %zu in %s\n",
i, sec->sec_idx, sym_type, obj->filename);
return -EINVAL;
/* libelf doesn't provide setters for ST_VISIBILITY,
* but it is stored in the lower 2 bits of st_other
*/
- sym->st_other &= 0x03;
+ sym->st_other &= ~0x03;
sym->st_other |= sym_vis;
}
int err, i;
if (!linker->elf)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
err = finalize_btf(linker);
if (err)
- return err;
+ return libbpf_err(err);
/* Finalize strings */
strs_sz = strset__data_size(linker->strtab_strs);
if (elf_update(linker->elf, ELF_C_NULL) < 0) {
err = -errno;
pr_warn_elf("failed to finalize ELF layout");
- return err;
+ return libbpf_err(err);
}
/* Write out final ELF contents */
if (elf_update(linker->elf, ELF_C_WRITE) < 0) {
err = -errno;
pr_warn_elf("failed to write ELF contents");
- return err;
+ return libbpf_err(err);
}
elf_end(linker->elf);
#include <stdlib.h>
#include <memory.h>
#include <unistd.h>
+#include <arpa/inet.h>
#include <linux/bpf.h>
+#include <linux/if_ether.h>
+#include <linux/pkt_cls.h>
#include <linux/rtnetlink.h>
#include <sys/socket.h>
#include <errno.h>
return ret;
}
-static int bpf_netlink_recv(int sock, __u32 nl_pid, int seq,
- __dump_nlmsg_t _fn, libbpf_dump_nlmsg_t fn,
- void *cookie)
+static void libbpf_netlink_close(int sock)
+{
+ close(sock);
+}
+
+enum {
+ NL_CONT,
+ NL_NEXT,
+ NL_DONE,
+};
+
+static int libbpf_netlink_recv(int sock, __u32 nl_pid, int seq,
+ __dump_nlmsg_t _fn, libbpf_dump_nlmsg_t fn,
+ void *cookie)
{
bool multipart = true;
struct nlmsgerr *err;
int len, ret;
while (multipart) {
+start:
multipart = false;
len = recv(sock, buf, sizeof(buf), 0);
if (len < 0) {
}
if (_fn) {
ret = _fn(nh, fn, cookie);
- if (ret)
+ switch (ret) {
+ case NL_CONT:
+ break;
+ case NL_NEXT:
+ goto start;
+ case NL_DONE:
+ return 0;
+ default:
return ret;
+ }
}
}
}
return ret;
}
+static int libbpf_netlink_send_recv(struct nlmsghdr *nh,
+ __dump_nlmsg_t parse_msg,
+ libbpf_dump_nlmsg_t parse_attr,
+ void *cookie)
+{
+ __u32 nl_pid = 0;
+ int sock, ret;
+
+ sock = libbpf_netlink_open(&nl_pid);
+ if (sock < 0)
+ return sock;
+
+ nh->nlmsg_pid = 0;
+ nh->nlmsg_seq = time(NULL);
+
+ if (send(sock, nh, nh->nlmsg_len, 0) < 0) {
+ ret = -errno;
+ goto out;
+ }
+
+ ret = libbpf_netlink_recv(sock, nl_pid, nh->nlmsg_seq,
+ parse_msg, parse_attr, cookie);
+out:
+ libbpf_netlink_close(sock);
+ return ret;
+}
+
static int __bpf_set_link_xdp_fd_replace(int ifindex, int fd, int old_fd,
__u32 flags)
{
- int sock, seq = 0, ret;
- struct nlattr *nla, *nla_xdp;
+ struct nlattr *nla;
+ int ret;
struct {
struct nlmsghdr nh;
struct ifinfomsg ifinfo;
char attrbuf[64];
} req;
- __u32 nl_pid = 0;
-
- sock = libbpf_netlink_open(&nl_pid);
- if (sock < 0)
- return sock;
memset(&req, 0, sizeof(req));
- req.nh.nlmsg_len = NLMSG_LENGTH(sizeof(struct ifinfomsg));
- req.nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK;
- req.nh.nlmsg_type = RTM_SETLINK;
- req.nh.nlmsg_pid = 0;
- req.nh.nlmsg_seq = ++seq;
+ req.nh.nlmsg_len = NLMSG_LENGTH(sizeof(struct ifinfomsg));
+ req.nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK;
+ req.nh.nlmsg_type = RTM_SETLINK;
req.ifinfo.ifi_family = AF_UNSPEC;
- req.ifinfo.ifi_index = ifindex;
-
- /* started nested attribute for XDP */
- nla = (struct nlattr *)(((char *)&req)
- + NLMSG_ALIGN(req.nh.nlmsg_len));
- nla->nla_type = NLA_F_NESTED | IFLA_XDP;
- nla->nla_len = NLA_HDRLEN;
-
- /* add XDP fd */
- nla_xdp = (struct nlattr *)((char *)nla + nla->nla_len);
- nla_xdp->nla_type = IFLA_XDP_FD;
- nla_xdp->nla_len = NLA_HDRLEN + sizeof(int);
- memcpy((char *)nla_xdp + NLA_HDRLEN, &fd, sizeof(fd));
- nla->nla_len += nla_xdp->nla_len;
-
- /* if user passed in any flags, add those too */
+ req.ifinfo.ifi_index = ifindex;
+
+ nla = nlattr_begin_nested(&req.nh, sizeof(req), IFLA_XDP);
+ if (!nla)
+ return -EMSGSIZE;
+ ret = nlattr_add(&req.nh, sizeof(req), IFLA_XDP_FD, &fd, sizeof(fd));
+ if (ret < 0)
+ return ret;
if (flags) {
- nla_xdp = (struct nlattr *)((char *)nla + nla->nla_len);
- nla_xdp->nla_type = IFLA_XDP_FLAGS;
- nla_xdp->nla_len = NLA_HDRLEN + sizeof(flags);
- memcpy((char *)nla_xdp + NLA_HDRLEN, &flags, sizeof(flags));
- nla->nla_len += nla_xdp->nla_len;
+ ret = nlattr_add(&req.nh, sizeof(req), IFLA_XDP_FLAGS, &flags,
+ sizeof(flags));
+ if (ret < 0)
+ return ret;
}
-
if (flags & XDP_FLAGS_REPLACE) {
- nla_xdp = (struct nlattr *)((char *)nla + nla->nla_len);
- nla_xdp->nla_type = IFLA_XDP_EXPECTED_FD;
- nla_xdp->nla_len = NLA_HDRLEN + sizeof(old_fd);
- memcpy((char *)nla_xdp + NLA_HDRLEN, &old_fd, sizeof(old_fd));
- nla->nla_len += nla_xdp->nla_len;
+ ret = nlattr_add(&req.nh, sizeof(req), IFLA_XDP_EXPECTED_FD,
+ &old_fd, sizeof(old_fd));
+ if (ret < 0)
+ return ret;
}
+ nlattr_end_nested(&req.nh, nla);
- req.nh.nlmsg_len += NLA_ALIGN(nla->nla_len);
-
- if (send(sock, &req, req.nh.nlmsg_len, 0) < 0) {
- ret = -errno;
- goto cleanup;
- }
- ret = bpf_netlink_recv(sock, nl_pid, seq, NULL, NULL, NULL);
-
-cleanup:
- close(sock);
- return ret;
+ return libbpf_netlink_send_recv(&req.nh, NULL, NULL, NULL);
}
int bpf_set_link_xdp_fd_opts(int ifindex, int fd, __u32 flags,
const struct bpf_xdp_set_link_opts *opts)
{
- int old_fd = -1;
+ int old_fd = -1, ret;
if (!OPTS_VALID(opts, bpf_xdp_set_link_opts))
- return -EINVAL;
+ return libbpf_err(-EINVAL);
if (OPTS_HAS(opts, old_fd)) {
old_fd = OPTS_GET(opts, old_fd, -1);
flags |= XDP_FLAGS_REPLACE;
}
- return __bpf_set_link_xdp_fd_replace(ifindex, fd,
- old_fd,
- flags);
+ ret = __bpf_set_link_xdp_fd_replace(ifindex, fd, old_fd, flags);
+ return libbpf_err(ret);
}
int bpf_set_link_xdp_fd(int ifindex, int fd, __u32 flags)
{
- return __bpf_set_link_xdp_fd_replace(ifindex, fd, 0, flags);
+ int ret;
+
+ ret = __bpf_set_link_xdp_fd_replace(ifindex, fd, 0, flags);
+ return libbpf_err(ret);
}
static int __dump_link_nlmsg(struct nlmsghdr *nlh,
len = nlh->nlmsg_len - NLMSG_LENGTH(sizeof(*ifi));
attr = (struct nlattr *) ((void *) ifi + NLMSG_ALIGN(sizeof(*ifi)));
+
if (libbpf_nla_parse(tb, IFLA_MAX, attr, len, NULL) != 0)
return -LIBBPF_ERRNO__NLPARSE;
return 0;
}
-static int libbpf_nl_get_link(int sock, unsigned int nl_pid,
- libbpf_dump_nlmsg_t dump_link_nlmsg, void *cookie);
-
int bpf_get_link_xdp_info(int ifindex, struct xdp_link_info *info,
size_t info_size, __u32 flags)
{
struct xdp_id_md xdp_id = {};
- int sock, ret;
- __u32 nl_pid = 0;
__u32 mask;
+ int ret;
+ struct {
+ struct nlmsghdr nh;
+ struct ifinfomsg ifm;
+ } req = {
+ .nh.nlmsg_len = NLMSG_LENGTH(sizeof(struct ifinfomsg)),
+ .nh.nlmsg_type = RTM_GETLINK,
+ .nh.nlmsg_flags = NLM_F_DUMP | NLM_F_REQUEST,
+ .ifm.ifi_family = AF_PACKET,
+ };
if (flags & ~XDP_FLAGS_MASK || !info_size)
- return -EINVAL;
+ return libbpf_err(-EINVAL);
/* Check whether the single {HW,DRV,SKB} mode is set */
flags &= (XDP_FLAGS_SKB_MODE | XDP_FLAGS_DRV_MODE | XDP_FLAGS_HW_MODE);
mask = flags - 1;
if (flags && flags & mask)
- return -EINVAL;
-
- sock = libbpf_netlink_open(&nl_pid);
- if (sock < 0)
- return sock;
+ return libbpf_err(-EINVAL);
xdp_id.ifindex = ifindex;
xdp_id.flags = flags;
- ret = libbpf_nl_get_link(sock, nl_pid, get_xdp_info, &xdp_id);
+ ret = libbpf_netlink_send_recv(&req.nh, __dump_link_nlmsg,
+ get_xdp_info, &xdp_id);
if (!ret) {
size_t sz = min(info_size, sizeof(xdp_id.info));
memset((void *) info + sz, 0, info_size - sz);
}
- close(sock);
- return ret;
+ return libbpf_err(ret);
}
static __u32 get_xdp_id(struct xdp_link_info *info, __u32 flags)
if (!ret)
*prog_id = get_xdp_id(&info, flags);
- return ret;
+ return libbpf_err(ret);
+}
+
+typedef int (*qdisc_config_t)(struct nlmsghdr *nh, struct tcmsg *t,
+ size_t maxsz);
+
+static int clsact_config(struct nlmsghdr *nh, struct tcmsg *t, size_t maxsz)
+{
+ t->tcm_parent = TC_H_CLSACT;
+ t->tcm_handle = TC_H_MAKE(TC_H_CLSACT, 0);
+
+ return nlattr_add(nh, maxsz, TCA_KIND, "clsact", sizeof("clsact"));
+}
+
+static int attach_point_to_config(struct bpf_tc_hook *hook,
+ qdisc_config_t *config)
+{
+ switch (OPTS_GET(hook, attach_point, 0)) {
+ case BPF_TC_INGRESS:
+ case BPF_TC_EGRESS:
+ case BPF_TC_INGRESS | BPF_TC_EGRESS:
+ if (OPTS_GET(hook, parent, 0))
+ return -EINVAL;
+ *config = &clsact_config;
+ return 0;
+ case BPF_TC_CUSTOM:
+ return -EOPNOTSUPP;
+ default:
+ return -EINVAL;
+ }
+}
+
+static int tc_get_tcm_parent(enum bpf_tc_attach_point attach_point,
+ __u32 *parent)
+{
+ switch (attach_point) {
+ case BPF_TC_INGRESS:
+ case BPF_TC_EGRESS:
+ if (*parent)
+ return -EINVAL;
+ *parent = TC_H_MAKE(TC_H_CLSACT,
+ attach_point == BPF_TC_INGRESS ?
+ TC_H_MIN_INGRESS : TC_H_MIN_EGRESS);
+ break;
+ case BPF_TC_CUSTOM:
+ if (!*parent)
+ return -EINVAL;
+ break;
+ default:
+ return -EINVAL;
+ }
+ return 0;
}
-int libbpf_nl_get_link(int sock, unsigned int nl_pid,
- libbpf_dump_nlmsg_t dump_link_nlmsg, void *cookie)
+static int tc_qdisc_modify(struct bpf_tc_hook *hook, int cmd, int flags)
{
+ qdisc_config_t config;
+ int ret;
struct {
- struct nlmsghdr nlh;
- struct ifinfomsg ifm;
- } req = {
- .nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct ifinfomsg)),
- .nlh.nlmsg_type = RTM_GETLINK,
- .nlh.nlmsg_flags = NLM_F_DUMP | NLM_F_REQUEST,
- .ifm.ifi_family = AF_PACKET,
- };
- int seq = time(NULL);
+ struct nlmsghdr nh;
+ struct tcmsg tc;
+ char buf[256];
+ } req;
+
+ ret = attach_point_to_config(hook, &config);
+ if (ret < 0)
+ return ret;
+
+ memset(&req, 0, sizeof(req));
+ req.nh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg));
+ req.nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK | flags;
+ req.nh.nlmsg_type = cmd;
+ req.tc.tcm_family = AF_UNSPEC;
+ req.tc.tcm_ifindex = OPTS_GET(hook, ifindex, 0);
+
+ ret = config(&req.nh, &req.tc, sizeof(req));
+ if (ret < 0)
+ return ret;
+
+ return libbpf_netlink_send_recv(&req.nh, NULL, NULL, NULL);
+}
- req.nlh.nlmsg_seq = seq;
- if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0)
+static int tc_qdisc_create_excl(struct bpf_tc_hook *hook)
+{
+ return tc_qdisc_modify(hook, RTM_NEWQDISC, NLM_F_CREATE | NLM_F_EXCL);
+}
+
+static int tc_qdisc_delete(struct bpf_tc_hook *hook)
+{
+ return tc_qdisc_modify(hook, RTM_DELQDISC, 0);
+}
+
+int bpf_tc_hook_create(struct bpf_tc_hook *hook)
+{
+ int ret;
+
+ if (!hook || !OPTS_VALID(hook, bpf_tc_hook) ||
+ OPTS_GET(hook, ifindex, 0) <= 0)
+ return libbpf_err(-EINVAL);
+
+ ret = tc_qdisc_create_excl(hook);
+ return libbpf_err(ret);
+}
+
+static int __bpf_tc_detach(const struct bpf_tc_hook *hook,
+ const struct bpf_tc_opts *opts,
+ const bool flush);
+
+int bpf_tc_hook_destroy(struct bpf_tc_hook *hook)
+{
+ if (!hook || !OPTS_VALID(hook, bpf_tc_hook) ||
+ OPTS_GET(hook, ifindex, 0) <= 0)
+ return libbpf_err(-EINVAL);
+
+ switch (OPTS_GET(hook, attach_point, 0)) {
+ case BPF_TC_INGRESS:
+ case BPF_TC_EGRESS:
+ return libbpf_err(__bpf_tc_detach(hook, NULL, true));
+ case BPF_TC_INGRESS | BPF_TC_EGRESS:
+ return libbpf_err(tc_qdisc_delete(hook));
+ case BPF_TC_CUSTOM:
+ return libbpf_err(-EOPNOTSUPP);
+ default:
+ return libbpf_err(-EINVAL);
+ }
+}
+
+struct bpf_cb_ctx {
+ struct bpf_tc_opts *opts;
+ bool processed;
+};
+
+static int __get_tc_info(void *cookie, struct tcmsg *tc, struct nlattr **tb,
+ bool unicast)
+{
+ struct nlattr *tbb[TCA_BPF_MAX + 1];
+ struct bpf_cb_ctx *info = cookie;
+
+ if (!info || !info->opts)
+ return -EINVAL;
+ if (unicast && info->processed)
+ return -EINVAL;
+ if (!tb[TCA_OPTIONS])
+ return NL_CONT;
+
+ libbpf_nla_parse_nested(tbb, TCA_BPF_MAX, tb[TCA_OPTIONS], NULL);
+ if (!tbb[TCA_BPF_ID])
+ return -EINVAL;
+
+ OPTS_SET(info->opts, prog_id, libbpf_nla_getattr_u32(tbb[TCA_BPF_ID]));
+ OPTS_SET(info->opts, handle, tc->tcm_handle);
+ OPTS_SET(info->opts, priority, TC_H_MAJ(tc->tcm_info) >> 16);
+
+ info->processed = true;
+ return unicast ? NL_NEXT : NL_DONE;
+}
+
+static int get_tc_info(struct nlmsghdr *nh, libbpf_dump_nlmsg_t fn,
+ void *cookie)
+{
+ struct tcmsg *tc = NLMSG_DATA(nh);
+ struct nlattr *tb[TCA_MAX + 1];
+
+ libbpf_nla_parse(tb, TCA_MAX,
+ (struct nlattr *)((char *)tc + NLMSG_ALIGN(sizeof(*tc))),
+ NLMSG_PAYLOAD(nh, sizeof(*tc)), NULL);
+ if (!tb[TCA_KIND])
+ return NL_CONT;
+ return __get_tc_info(cookie, tc, tb, nh->nlmsg_flags & NLM_F_ECHO);
+}
+
+static int tc_add_fd_and_name(struct nlmsghdr *nh, size_t maxsz, int fd)
+{
+ struct bpf_prog_info info = {};
+ __u32 info_len = sizeof(info);
+ char name[256];
+ int len, ret;
+
+ ret = bpf_obj_get_info_by_fd(fd, &info, &info_len);
+ if (ret < 0)
+ return ret;
+
+ ret = nlattr_add(nh, maxsz, TCA_BPF_FD, &fd, sizeof(fd));
+ if (ret < 0)
+ return ret;
+ len = snprintf(name, sizeof(name), "%s:[%u]", info.name, info.id);
+ if (len < 0)
return -errno;
+ if (len >= sizeof(name))
+ return -ENAMETOOLONG;
+ return nlattr_add(nh, maxsz, TCA_BPF_NAME, name, len + 1);
+}
+
+int bpf_tc_attach(const struct bpf_tc_hook *hook, struct bpf_tc_opts *opts)
+{
+ __u32 protocol, bpf_flags, handle, priority, parent, prog_id, flags;
+ int ret, ifindex, attach_point, prog_fd;
+ struct bpf_cb_ctx info = {};
+ struct nlattr *nla;
+ struct {
+ struct nlmsghdr nh;
+ struct tcmsg tc;
+ char buf[256];
+ } req;
+
+ if (!hook || !opts ||
+ !OPTS_VALID(hook, bpf_tc_hook) ||
+ !OPTS_VALID(opts, bpf_tc_opts))
+ return libbpf_err(-EINVAL);
+
+ ifindex = OPTS_GET(hook, ifindex, 0);
+ parent = OPTS_GET(hook, parent, 0);
+ attach_point = OPTS_GET(hook, attach_point, 0);
+
+ handle = OPTS_GET(opts, handle, 0);
+ priority = OPTS_GET(opts, priority, 0);
+ prog_fd = OPTS_GET(opts, prog_fd, 0);
+ prog_id = OPTS_GET(opts, prog_id, 0);
+ flags = OPTS_GET(opts, flags, 0);
- return bpf_netlink_recv(sock, nl_pid, seq, __dump_link_nlmsg,
- dump_link_nlmsg, cookie);
+ if (ifindex <= 0 || !prog_fd || prog_id)
+ return libbpf_err(-EINVAL);
+ if (priority > UINT16_MAX)
+ return libbpf_err(-EINVAL);
+ if (flags & ~BPF_TC_F_REPLACE)
+ return libbpf_err(-EINVAL);
+
+ flags = (flags & BPF_TC_F_REPLACE) ? NLM_F_REPLACE : NLM_F_EXCL;
+ protocol = ETH_P_ALL;
+
+ memset(&req, 0, sizeof(req));
+ req.nh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg));
+ req.nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK | NLM_F_CREATE |
+ NLM_F_ECHO | flags;
+ req.nh.nlmsg_type = RTM_NEWTFILTER;
+ req.tc.tcm_family = AF_UNSPEC;
+ req.tc.tcm_ifindex = ifindex;
+ req.tc.tcm_handle = handle;
+ req.tc.tcm_info = TC_H_MAKE(priority << 16, htons(protocol));
+
+ ret = tc_get_tcm_parent(attach_point, &parent);
+ if (ret < 0)
+ return libbpf_err(ret);
+ req.tc.tcm_parent = parent;
+
+ ret = nlattr_add(&req.nh, sizeof(req), TCA_KIND, "bpf", sizeof("bpf"));
+ if (ret < 0)
+ return libbpf_err(ret);
+ nla = nlattr_begin_nested(&req.nh, sizeof(req), TCA_OPTIONS);
+ if (!nla)
+ return libbpf_err(-EMSGSIZE);
+ ret = tc_add_fd_and_name(&req.nh, sizeof(req), prog_fd);
+ if (ret < 0)
+ return libbpf_err(ret);
+ bpf_flags = TCA_BPF_FLAG_ACT_DIRECT;
+ ret = nlattr_add(&req.nh, sizeof(req), TCA_BPF_FLAGS, &bpf_flags,
+ sizeof(bpf_flags));
+ if (ret < 0)
+ return libbpf_err(ret);
+ nlattr_end_nested(&req.nh, nla);
+
+ info.opts = opts;
+
+ ret = libbpf_netlink_send_recv(&req.nh, get_tc_info, NULL, &info);
+ if (ret < 0)
+ return libbpf_err(ret);
+ if (!info.processed)
+ return libbpf_err(-ENOENT);
+ return ret;
+}
+
+static int __bpf_tc_detach(const struct bpf_tc_hook *hook,
+ const struct bpf_tc_opts *opts,
+ const bool flush)
+{
+ __u32 protocol = 0, handle, priority, parent, prog_id, flags;
+ int ret, ifindex, attach_point, prog_fd;
+ struct {
+ struct nlmsghdr nh;
+ struct tcmsg tc;
+ char buf[256];
+ } req;
+
+ if (!hook ||
+ !OPTS_VALID(hook, bpf_tc_hook) ||
+ !OPTS_VALID(opts, bpf_tc_opts))
+ return -EINVAL;
+
+ ifindex = OPTS_GET(hook, ifindex, 0);
+ parent = OPTS_GET(hook, parent, 0);
+ attach_point = OPTS_GET(hook, attach_point, 0);
+
+ handle = OPTS_GET(opts, handle, 0);
+ priority = OPTS_GET(opts, priority, 0);
+ prog_fd = OPTS_GET(opts, prog_fd, 0);
+ prog_id = OPTS_GET(opts, prog_id, 0);
+ flags = OPTS_GET(opts, flags, 0);
+
+ if (ifindex <= 0 || flags || prog_fd || prog_id)
+ return -EINVAL;
+ if (priority > UINT16_MAX)
+ return -EINVAL;
+ if (!flush) {
+ if (!handle || !priority)
+ return -EINVAL;
+ protocol = ETH_P_ALL;
+ } else {
+ if (handle || priority)
+ return -EINVAL;
+ }
+
+ memset(&req, 0, sizeof(req));
+ req.nh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg));
+ req.nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK;
+ req.nh.nlmsg_type = RTM_DELTFILTER;
+ req.tc.tcm_family = AF_UNSPEC;
+ req.tc.tcm_ifindex = ifindex;
+ if (!flush) {
+ req.tc.tcm_handle = handle;
+ req.tc.tcm_info = TC_H_MAKE(priority << 16, htons(protocol));
+ }
+
+ ret = tc_get_tcm_parent(attach_point, &parent);
+ if (ret < 0)
+ return ret;
+ req.tc.tcm_parent = parent;
+
+ if (!flush) {
+ ret = nlattr_add(&req.nh, sizeof(req), TCA_KIND,
+ "bpf", sizeof("bpf"));
+ if (ret < 0)
+ return ret;
+ }
+
+ return libbpf_netlink_send_recv(&req.nh, NULL, NULL, NULL);
+}
+
+int bpf_tc_detach(const struct bpf_tc_hook *hook,
+ const struct bpf_tc_opts *opts)
+{
+ int ret;
+
+ if (!opts)
+ return libbpf_err(-EINVAL);
+
+ ret = __bpf_tc_detach(hook, opts, false);
+ return libbpf_err(ret);
+}
+
+int bpf_tc_query(const struct bpf_tc_hook *hook, struct bpf_tc_opts *opts)
+{
+ __u32 protocol, handle, priority, parent, prog_id, flags;
+ int ret, ifindex, attach_point, prog_fd;
+ struct bpf_cb_ctx info = {};
+ struct {
+ struct nlmsghdr nh;
+ struct tcmsg tc;
+ char buf[256];
+ } req;
+
+ if (!hook || !opts ||
+ !OPTS_VALID(hook, bpf_tc_hook) ||
+ !OPTS_VALID(opts, bpf_tc_opts))
+ return libbpf_err(-EINVAL);
+
+ ifindex = OPTS_GET(hook, ifindex, 0);
+ parent = OPTS_GET(hook, parent, 0);
+ attach_point = OPTS_GET(hook, attach_point, 0);
+
+ handle = OPTS_GET(opts, handle, 0);
+ priority = OPTS_GET(opts, priority, 0);
+ prog_fd = OPTS_GET(opts, prog_fd, 0);
+ prog_id = OPTS_GET(opts, prog_id, 0);
+ flags = OPTS_GET(opts, flags, 0);
+
+ if (ifindex <= 0 || flags || prog_fd || prog_id ||
+ !handle || !priority)
+ return libbpf_err(-EINVAL);
+ if (priority > UINT16_MAX)
+ return libbpf_err(-EINVAL);
+
+ protocol = ETH_P_ALL;
+
+ memset(&req, 0, sizeof(req));
+ req.nh.nlmsg_len = NLMSG_LENGTH(sizeof(struct tcmsg));
+ req.nh.nlmsg_flags = NLM_F_REQUEST;
+ req.nh.nlmsg_type = RTM_GETTFILTER;
+ req.tc.tcm_family = AF_UNSPEC;
+ req.tc.tcm_ifindex = ifindex;
+ req.tc.tcm_handle = handle;
+ req.tc.tcm_info = TC_H_MAKE(priority << 16, htons(protocol));
+
+ ret = tc_get_tcm_parent(attach_point, &parent);
+ if (ret < 0)
+ return libbpf_err(ret);
+ req.tc.tcm_parent = parent;
+
+ ret = nlattr_add(&req.nh, sizeof(req), TCA_KIND, "bpf", sizeof("bpf"));
+ if (ret < 0)
+ return libbpf_err(ret);
+
+ info.opts = opts;
+
+ ret = libbpf_netlink_send_recv(&req.nh, get_tc_info, NULL, &info);
+ if (ret < 0)
+ return libbpf_err(ret);
+ if (!info.processed)
+ return libbpf_err(-ENOENT);
+ return ret;
}
#define __LIBBPF_NLATTR_H
#include <stdint.h>
+#include <string.h>
+#include <errno.h>
#include <linux/netlink.h>
+
/* avoid multiple definition of netlink features */
#define __LINUX_NETLINK_H
int libbpf_nla_dump_errormsg(struct nlmsghdr *nlh);
+static inline struct nlattr *nla_data(struct nlattr *nla)
+{
+ return (struct nlattr *)((char *)nla + NLA_HDRLEN);
+}
+
+static inline struct nlattr *nh_tail(struct nlmsghdr *nh)
+{
+ return (struct nlattr *)((char *)nh + NLMSG_ALIGN(nh->nlmsg_len));
+}
+
+static inline int nlattr_add(struct nlmsghdr *nh, size_t maxsz, int type,
+ const void *data, int len)
+{
+ struct nlattr *nla;
+
+ if (NLMSG_ALIGN(nh->nlmsg_len) + NLA_ALIGN(NLA_HDRLEN + len) > maxsz)
+ return -EMSGSIZE;
+ if (!!data != !!len)
+ return -EINVAL;
+
+ nla = nh_tail(nh);
+ nla->nla_type = type;
+ nla->nla_len = NLA_HDRLEN + len;
+ if (data)
+ memcpy(nla_data(nla), data, len);
+ nh->nlmsg_len = NLMSG_ALIGN(nh->nlmsg_len) + NLA_ALIGN(nla->nla_len);
+ return 0;
+}
+
+static inline struct nlattr *nlattr_begin_nested(struct nlmsghdr *nh,
+ size_t maxsz, int type)
+{
+ struct nlattr *tail;
+
+ tail = nh_tail(nh);
+ if (nlattr_add(nh, maxsz, type | NLA_F_NESTED, NULL, 0))
+ return NULL;
+ return tail;
+}
+
+static inline void nlattr_end_nested(struct nlmsghdr *nh, struct nlattr *tail)
+{
+ tail->nla_len = (char *)nh_tail(nh) - (char *)tail;
+}
+
#endif /* __LIBBPF_NLATTR_H */
err = -errno;
pr_warn("ringbuf: failed to get map info for fd=%d: %d\n",
map_fd, err);
- return err;
+ return libbpf_err(err);
}
if (info.type != BPF_MAP_TYPE_RINGBUF) {
pr_warn("ringbuf: map fd=%d is not BPF_MAP_TYPE_RINGBUF\n",
map_fd);
- return -EINVAL;
+ return libbpf_err(-EINVAL);
}
tmp = libbpf_reallocarray(rb->rings, rb->ring_cnt + 1, sizeof(*rb->rings));
if (!tmp)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
rb->rings = tmp;
tmp = libbpf_reallocarray(rb->events, rb->ring_cnt + 1, sizeof(*rb->events));
if (!tmp)
- return -ENOMEM;
+ return libbpf_err(-ENOMEM);
rb->events = tmp;
r = &rb->rings[rb->ring_cnt];
err = -errno;
pr_warn("ringbuf: failed to mmap consumer page for map fd=%d: %d\n",
map_fd, err);
- return err;
+ return libbpf_err(err);
}
r->consumer_pos = tmp;
ringbuf_unmap_ring(rb, r);
pr_warn("ringbuf: failed to mmap data pages for map fd=%d: %d\n",
map_fd, err);
- return err;
+ return libbpf_err(err);
}
r->producer_pos = tmp;
r->data = tmp + rb->page_size;
ringbuf_unmap_ring(rb, r);
pr_warn("ringbuf: failed to epoll add map fd=%d: %d\n",
map_fd, err);
- return err;
+ return libbpf_err(err);
}
rb->ring_cnt++;
int err;
if (!OPTS_VALID(opts, ring_buffer_opts))
- return NULL;
+ return errno = EINVAL, NULL;
rb = calloc(1, sizeof(*rb));
if (!rb)
- return NULL;
+ return errno = ENOMEM, NULL;
rb->page_size = getpagesize();
err_out:
ring_buffer__free(rb);
- return NULL;
+ return errno = -err, NULL;
}
static inline int roundup_len(__u32 len)
err = ringbuf_process_ring(ring);
if (err < 0)
- return err;
+ return libbpf_err(err);
res += err;
}
if (res > INT_MAX)
cnt = epoll_wait(rb->epoll_fd, rb->events, rb->ring_cnt, timeout_ms);
if (cnt < 0)
- return -errno;
+ return libbpf_err(-errno);
for (i = 0; i < cnt; i++) {
__u32 ring_id = rb->events[i].data.fd;
err = ringbuf_process_ring(ring);
if (err < 0)
- return err;
+ return libbpf_err(err);
res += err;
}
if (res > INT_MAX)
--- /dev/null
+/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
+/* Copyright (c) 2021 Facebook */
+#ifndef __SKEL_INTERNAL_H
+#define __SKEL_INTERNAL_H
+
+#include <unistd.h>
+#include <sys/syscall.h>
+#include <sys/mman.h>
+
+/* This file is a base header for auto-generated *.lskel.h files.
+ * Its contents will change and may become part of auto-generation in the future.
+ *
+ * The layout of bpf_[map|prog]_desc and bpf_loader_ctx is feature dependent
+ * and will change from one version of libbpf to another and features
+ * requested during loader program generation.
+ */
+struct bpf_map_desc {
+ union {
+ /* input for the loader prog */
+ struct {
+ __aligned_u64 initial_value;
+ __u32 max_entries;
+ };
+ /* output of the loader prog */
+ struct {
+ int map_fd;
+ };
+ };
+};
+struct bpf_prog_desc {
+ int prog_fd;
+};
+
+struct bpf_loader_ctx {
+ size_t sz;
+ __u32 log_level;
+ __u32 log_size;
+ __u64 log_buf;
+};
+
+struct bpf_load_and_run_opts {
+ struct bpf_loader_ctx *ctx;
+ const void *data;
+ const void *insns;
+ __u32 data_sz;
+ __u32 insns_sz;
+ const char *errstr;
+};
+
+static inline int skel_sys_bpf(enum bpf_cmd cmd, union bpf_attr *attr,
+ unsigned int size)
+{
+ return syscall(__NR_bpf, cmd, attr, size);
+}
+
+static inline int skel_closenz(int fd)
+{
+ if (fd > 0)
+ return close(fd);
+ return -EINVAL;
+}
+
+static inline int bpf_load_and_run(struct bpf_load_and_run_opts *opts)
+{
+ int map_fd = -1, prog_fd = -1, key = 0, err;
+ union bpf_attr attr;
+
+ map_fd = bpf_create_map_name(BPF_MAP_TYPE_ARRAY, "__loader.map", 4,
+ opts->data_sz, 1, 0);
+ if (map_fd < 0) {
+ opts->errstr = "failed to create loader map";
+ err = -errno;
+ goto out;
+ }
+
+ err = bpf_map_update_elem(map_fd, &key, opts->data, 0);
+ if (err < 0) {
+ opts->errstr = "failed to update loader map";
+ err = -errno;
+ goto out;
+ }
+
+ memset(&attr, 0, sizeof(attr));
+ attr.prog_type = BPF_PROG_TYPE_SYSCALL;
+ attr.insns = (long) opts->insns;
+ attr.insn_cnt = opts->insns_sz / sizeof(struct bpf_insn);
+ attr.license = (long) "Dual BSD/GPL";
+ memcpy(attr.prog_name, "__loader.prog", sizeof("__loader.prog"));
+ attr.fd_array = (long) &map_fd;
+ attr.log_level = opts->ctx->log_level;
+ attr.log_size = opts->ctx->log_size;
+ attr.log_buf = opts->ctx->log_buf;
+ attr.prog_flags = BPF_F_SLEEPABLE;
+ prog_fd = skel_sys_bpf(BPF_PROG_LOAD, &attr, sizeof(attr));
+ if (prog_fd < 0) {
+ opts->errstr = "failed to load loader prog";
+ err = -errno;
+ goto out;
+ }
+
+ memset(&attr, 0, sizeof(attr));
+ attr.test.prog_fd = prog_fd;
+ attr.test.ctx_in = (long) opts->ctx;
+ attr.test.ctx_size_in = opts->ctx->sz;
+ err = skel_sys_bpf(BPF_PROG_RUN, &attr, sizeof(attr));
+ if (err < 0 || (int)attr.test.retval < 0) {
+ opts->errstr = "failed to execute loader prog";
+ if (err < 0)
+ err = -errno;
+ else
+ err = (int)attr.test.retval;
+ goto out;
+ }
+ err = 0;
+out:
+ if (map_fd >= 0)
+ close(map_fd);
+ if (prog_fd >= 0)
+ close(prog_fd);
+ return err;
+}
+
+#endif
goto out_put_ctx;
}
if (xsk->fd == umem->fd)
- umem->rx_ring_setup_done = true;
+ umem->tx_ring_setup_done = true;
}
err = xsk_get_mmap_offsets(xsk->fd, &off);
#include <objtool/elf.h>
#include <objtool/arch.h>
#include <objtool/warn.h>
+#include <objtool/endianness.h>
#include <arch/elf.h>
static int is_x86_64(const struct elf *elf)
return -1;
}
- alt->cpuid = cpuid;
+ alt->cpuid = bswap_if_needed(cpuid);
alt->instrlen = orig_len;
alt->replacementlen = repl_len;
list_for_each_entry(insn, &file->retpoline_call_list, call_node) {
+ if (insn->type != INSN_JUMP_DYNAMIC &&
+ insn->type != INSN_CALL_DYNAMIC)
+ continue;
+
if (!strcmp(insn->sec->name, ".text.__x86.indirect_thunk"))
continue;
struct symbol *elf_create_undef_symbol(struct elf *elf, const char *name)
{
- struct section *symtab;
+ struct section *symtab, *symtab_shndx;
struct symbol *sym;
Elf_Data *data;
Elf_Scn *s;
data->d_buf = &sym->sym;
data->d_size = sizeof(sym->sym);
data->d_align = 1;
+ data->d_type = ELF_T_SYM;
sym->idx = symtab->len / sizeof(sym->sym);
symtab->len += data->d_size;
symtab->changed = true;
+ symtab_shndx = find_section_by_name(elf, ".symtab_shndx");
+ if (symtab_shndx) {
+ s = elf_getscn(elf->elf, symtab_shndx->idx);
+ if (!s) {
+ WARN_ELF("elf_getscn");
+ return NULL;
+ }
+
+ data = elf_newdata(s);
+ if (!data) {
+ WARN_ELF("elf_newdata");
+ return NULL;
+ }
+
+ data->d_buf = &sym->sym.st_size; /* conveniently 0 */
+ data->d_size = sizeof(Elf32_Word);
+ data->d_align = 4;
+ data->d_type = ELF_T_WORD;
+
+ symtab_shndx->len += 4;
+ symtab_shndx->changed = true;
+ }
+
sym->sec = find_section_by_index(elf, 0);
elf_add_symbol(elf, sym);
perf script --itrace=ibxwpe -F+flags
-The flags are "bcrosyiABEx" which stand for branch, call, return, conditional,
-system, asynchronous, interrupt, transaction abort, trace begin, trace end, and
-in transaction, respectively.
+The flags are "bcrosyiABExgh" which stand for branch, call, return, conditional,
+system, asynchronous, interrupt, transaction abort, trace begin, trace end,
+in transaction, VM-entry, and VM-exit respectively.
perf script also supports higher level ways to dump instruction traces:
At this point usage is displayed, and perf-script exits.
The flags field is synthesized and may have a value when Instruction
- Trace decoding. The flags are "bcrosyiABEx" which stand for branch,
+ Trace decoding. The flags are "bcrosyiABExgh" which stand for branch,
call, return, conditional, system, asynchronous, interrupt,
- transaction abort, trace begin, trace end, and in transaction,
+ transaction abort, trace begin, trace end, in transaction, VM-Entry, and VM-Exit
respectively. Known combinations of flags are printed more nicely e.g.
"call" for "bc", "return" for "br", "jcc" for "bo", "jmp" for "b",
"int" for "bci", "iret" for "bri", "syscall" for "bcs", "sysret" for "brs",
"async" for "by", "hw int" for "bcyi", "tx abrt" for "bA", "tr strt" for "bB",
- "tr end" for "bE". However the "x" flag will be display separately in those
+ "tr end" for "bE", "vmentry" for "bcg", "vmexit" for "bch".
+ However the "x" flag will be displayed separately in those
cases e.g. "jcc (x)" for a condition branch within a transaction.
The callindent field is synthesized and may have a value when
ifeq ($(ARCH),mips)
NO_PERF_REGS := 0
CFLAGS += -I$(OUTPUT)arch/mips/include/generated
- CFLAGS += -I../../arch/mips/include/uapi -I../../arch/mips/include/generated/uapi
LIBUNWIND_LIBS = -lunwind -lunwind-mips
endif
ifdef LIBBPF_DYNAMIC
ifeq ($(feature-libbpf), 1)
EXTLIBS += -lbpf
+ $(call detected,CONFIG_LIBBPF_DYNAMIC)
else
dummy := $(error Error: No libbpf devel library found, please install libbpf-devel);
endif
.name = "vmexit",
.ops = &exit_events,
},
- { NULL },
+ { NULL, NULL },
};
const char * const kvm_skip_events[] = {
439 n64 faccessat2 sys_faccessat2
440 n64 process_madvise sys_process_madvise
441 n64 epoll_pwait2 sys_epoll_pwait2
+442 n64 mount_setattr sys_mount_setattr
+# 443 reserved for quotactl_path
+444 n64 landlock_create_ruleset sys_landlock_create_ruleset
+445 n64 landlock_add_rule sys_landlock_add_rule
+446 n64 landlock_restrict_self sys_landlock_restrict_self
440 common process_madvise sys_process_madvise
441 common epoll_pwait2 sys_epoll_pwait2 compat_sys_epoll_pwait2
442 common mount_setattr sys_mount_setattr
+# 443 reserved for quotactl_path
+444 common landlock_create_ruleset sys_landlock_create_ruleset
+445 common landlock_add_rule sys_landlock_add_rule
+446 common landlock_restrict_self sys_landlock_restrict_self
440 common process_madvise sys_process_madvise sys_process_madvise
441 common epoll_pwait2 sys_epoll_pwait2 compat_sys_epoll_pwait2
442 common mount_setattr sys_mount_setattr sys_mount_setattr
+# 443 reserved for quotactl_path
+444 common landlock_create_ruleset sys_landlock_create_ruleset sys_landlock_create_ruleset
+445 common landlock_add_rule sys_landlock_add_rule sys_landlock_add_rule
+446 common landlock_restrict_self sys_landlock_restrict_self sys_landlock_restrict_self
440 common process_madvise sys_process_madvise
441 common epoll_pwait2 sys_epoll_pwait2
442 common mount_setattr sys_mount_setattr
+# 443 reserved for quotactl_path
+444 common landlock_create_ruleset sys_landlock_create_ruleset
+445 common landlock_add_rule sys_landlock_add_rule
+446 common landlock_restrict_self sys_landlock_restrict_self
#
# Due to a historical design error, certain syscalls are numbered differently
if (!perf_header__has_feat(&session->header, HEADER_BUILD_ID))
with_hits = true;
+ if (zstd_init(&(session->zstd_data), 0) < 0)
+ pr_warning("Decompression initialization failed. Reported data may be incomplete.\n");
+
/*
* in pipe-mode, the only way to get the buildids is to parse
* the record stream. Buildids are stored as RECORD_HEADER_BUILD_ID
rec->no_buildid = true;
}
+ if (rec->opts.record_cgroup && !perf_can_record_cgroup()) {
+ pr_err("Kernel has no cgroup sampling support.\n");
+ err = -EINVAL;
+ goto out_opts;
+ }
+
if (rec->opts.kcore)
rec->data.is_dir = true;
* - we have initial delay configured
*/
if (!target__none(&target) || stat_config.initial_delay) {
- evlist__enable(evsel_list);
+ if (!all_counters_use_bpf)
+ evlist__enable(evsel_list);
if (stat_config.initial_delay > 0)
pr_info(EVLIST_ENABLED_MSG);
}
static void disable_counters(void)
{
+ struct evsel *counter;
+
/*
* If we don't have tracee (attaching to task or cpu), counters may
* still be running. To get accurate group ratios, we must stop groups
* from counting before reading their constituent counters.
*/
- if (!target__none(&target))
- evlist__disable(evsel_list);
+ if (!target__none(&target)) {
+ evlist__for_each_entry(evsel_list, counter)
+ bpf_counter__disable(counter);
+ if (!all_counters_use_bpf)
+ evlist__disable(evsel_list);
+ }
}
static volatile int workload_exec_errno;
arch/x86/tools/gen-insn-attr-x86.awk
arch/arm/include/uapi/asm/perf_regs.h
arch/arm64/include/uapi/asm/perf_regs.h
+arch/mips/include/uapi/asm/perf_regs.h
arch/powerpc/include/uapi/asm/perf_regs.h
arch/s390/include/uapi/asm/perf_regs.h
arch/x86/include/uapi/asm/perf_regs.h
const char *cmd;
char sbuf[STRERR_BUFSIZE];
+ perf_debug_setup();
+
/* libsubcmd init */
exec_cmd_init("perf", PREFIX, PERF_EXEC_PATH, EXEC_PATH_ENVIRONMENT);
pager_init(PERF_PAGER_ENVIRONMENT);
*/
pthread__block_sigwinch();
- perf_debug_setup();
-
while (1) {
static int done_help;
[
{
- "EventCode": "1003C",
+ "EventCode": "0x1003C",
"EventName": "PM_EXEC_STALL_DMISS_L2L3",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was waiting for a load miss to resolve from either the local L2 or local L3."
},
{
- "EventCode": "34056",
+ "EventCode": "0x1E054",
+ "EventName": "PM_EXEC_STALL_DMISS_L21_L31",
+ "BriefDescription": "Cycles in which the oldest instruction in the pipeline was waiting for a load miss to resolve from another core's L2 or L3 on the same chip."
+ },
+ {
+ "EventCode": "0x34054",
+ "EventName": "PM_EXEC_STALL_DMISS_L2L3_NOCONFLICT",
+ "BriefDescription": "Cycles in which the oldest instruction in the pipeline was waiting for a load miss to resolve from the local L2 or local L3, without a dispatch conflict."
+ },
+ {
+ "EventCode": "0x34056",
"EventName": "PM_EXEC_STALL_LOAD_FINISH",
- "BriefDescription": "Cycles in which the oldest instruction in the pipeline was finishing a load after its data was reloaded from a data source beyond the local L1; cycles in which the LSU was processing an L1-hit; cycles in which the NTF instruction merged with another load in the LMQ."
+ "BriefDescription": "Cycles in which the oldest instruction in the pipeline was finishing a load after its data was reloaded from a data source beyond the local L1; cycles in which the LSU was processing an L1-hit; cycles in which the NTF instruction merged with another load in the LMQ; cycles in which the NTF instruction is waiting for a data reload for a load miss, but the data comes back with a non-NTF instruction."
},
{
- "EventCode": "3006C",
+ "EventCode": "0x3006C",
"EventName": "PM_RUN_CYC_SMT2_MODE",
"BriefDescription": "Cycles when this thread's run latch is set and the core is in SMT2 mode."
},
{
- "EventCode": "300F4",
+ "EventCode": "0x300F4",
"EventName": "PM_RUN_INST_CMPL_CONC",
"BriefDescription": "PowerPC instructions completed by this thread when all threads in the core had the run-latch set."
},
{
- "EventCode": "4C016",
+ "EventCode": "0x4C016",
"EventName": "PM_EXEC_STALL_DMISS_L2L3_CONFLICT",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was waiting for a load miss to resolve from the local L2 or local L3, with a dispatch conflict."
},
{
- "EventCode": "4D014",
+ "EventCode": "0x4D014",
"EventName": "PM_EXEC_STALL_LOAD",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was a load instruction executing in the Load Store Unit."
},
{
- "EventCode": "4D016",
+ "EventCode": "0x4D016",
"EventName": "PM_EXEC_STALL_PTESYNC",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was a PTESYNC instruction executing in the Load Store Unit."
},
{
- "EventCode": "401EA",
+ "EventCode": "0x401EA",
"EventName": "PM_THRESH_EXC_128",
"BriefDescription": "Threshold counter exceeded a value of 128."
},
{
- "EventCode": "400F6",
+ "EventCode": "0x400F6",
"EventName": "PM_BR_MPRED_CMPL",
"BriefDescription": "A mispredicted branch completed. Includes direction and target."
}
[
{
- "EventCode": "4016E",
+ "EventCode": "0x4016E",
"EventName": "PM_THRESH_NOT_MET",
"BriefDescription": "Threshold counter did not meet threshold."
}
[
{
- "EventCode": "10004",
+ "EventCode": "0x10004",
"EventName": "PM_EXEC_STALL_TRANSLATION",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline suffered a TLB miss or ERAT miss and waited for it to resolve."
},
{
- "EventCode": "10010",
+ "EventCode": "0x10006",
+ "EventName": "PM_DISP_STALL_HELD_OTHER_CYC",
+ "BriefDescription": "Cycles in which the NTC instruction is held at dispatch for any other reason."
+ },
+ {
+ "EventCode": "0x10010",
"EventName": "PM_PMC4_OVERFLOW",
"BriefDescription": "The event selected for PMC4 caused the event counter to overflow."
},
{
- "EventCode": "10020",
+ "EventCode": "0x10020",
"EventName": "PM_PMC4_REWIND",
"BriefDescription": "The speculative event selected for PMC4 rewinds and the counter for PMC4 is not charged."
},
{
- "EventCode": "10038",
+ "EventCode": "0x10038",
"EventName": "PM_DISP_STALL_TRANSLATION",
"BriefDescription": "Cycles when dispatch was stalled for this thread because the MMU was handling a translation miss."
},
{
- "EventCode": "1003A",
+ "EventCode": "0x1003A",
"EventName": "PM_DISP_STALL_BR_MPRED_IC_L2",
"BriefDescription": "Cycles when dispatch was stalled while the instruction was fetched from the local L2 after suffering a branch mispredict."
},
{
- "EventCode": "1E050",
+ "EventCode": "0x1D05E",
+ "EventName": "PM_DISP_STALL_HELD_HALT_CYC",
+ "BriefDescription": "Cycles in which the NTC instruction is held at dispatch because of power management."
+ },
+ {
+ "EventCode": "0x1E050",
"EventName": "PM_DISP_STALL_HELD_STF_MAPPER_CYC",
"BriefDescription": "Cycles in which the NTC instruction is held at dispatch because the STF mapper/SRB was full. Includes GPR (count, link, tar), VSR, VMR, FPR."
},
{
- "EventCode": "1F054",
+ "EventCode": "0x1F054",
"EventName": "PM_DTLB_HIT",
"BriefDescription": "The PTE required by the instruction was resident in the TLB (data TLB access). When MMCR1[16]=0 this event counts only demand hits. When MMCR1[16]=1 this event includes demand and prefetch. Applies to both HPT and RPT."
},
{
- "EventCode": "101E8",
+ "EventCode": "0x10064",
+ "EventName": "PM_DISP_STALL_IC_L2",
+ "BriefDescription": "Cycles when dispatch was stalled while the instruction was fetched from the local L2."
+ },
+ {
+ "EventCode": "0x101E8",
"EventName": "PM_THRESH_EXC_256",
"BriefDescription": "Threshold counter exceeded a count of 256."
},
{
- "EventCode": "101EC",
+ "EventCode": "0x101EC",
"EventName": "PM_THRESH_MET",
"BriefDescription": "Threshold exceeded."
},
{
- "EventCode": "100F2",
+ "EventCode": "0x100F2",
"EventName": "PM_1PLUS_PPC_CMPL",
"BriefDescription": "Cycles in which at least one instruction is completed by this thread."
},
{
- "EventCode": "100F6",
+ "EventCode": "0x100F6",
"EventName": "PM_IERAT_MISS",
"BriefDescription": "IERAT Reloaded to satisfy an IERAT miss. All page sizes are counted by this event."
},
{
- "EventCode": "100F8",
+ "EventCode": "0x100F8",
"EventName": "PM_DISP_STALL_CYC",
"BriefDescription": "Cycles the ICT has no itags assigned to this thread (no instructions were dispatched during these cycles)."
},
{
- "EventCode": "20114",
+ "EventCode": "0x20006",
+ "EventName": "PM_DISP_STALL_HELD_ISSQ_FULL_CYC",
+ "BriefDescription": "Cycles in which the NTC instruction is held at dispatch due to Issue queue full. Includes issue queue and branch queue."
+ },
+ {
+ "EventCode": "0x20114",
"EventName": "PM_MRK_L2_RC_DISP",
"BriefDescription": "Marked instruction RC dispatched in L2."
},
{
- "EventCode": "2C010",
+ "EventCode": "0x2C010",
"EventName": "PM_EXEC_STALL_LSU",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was executing in the Load Store Unit. This does not include simple fixed point instructions."
},
{
- "EventCode": "2C016",
+ "EventCode": "0x2C016",
"EventName": "PM_DISP_STALL_IERAT_ONLY_MISS",
"BriefDescription": "Cycles when dispatch was stalled while waiting to resolve an instruction ERAT miss."
},
{
- "EventCode": "2C01E",
+ "EventCode": "0x2C01E",
"EventName": "PM_DISP_STALL_BR_MPRED_IC_L3",
"BriefDescription": "Cycles when dispatch was stalled while the instruction was fetched from the local L3 after suffering a branch mispredict."
},
{
- "EventCode": "2D01A",
+ "EventCode": "0x2D01A",
"EventName": "PM_DISP_STALL_IC_MISS",
"BriefDescription": "Cycles when dispatch was stalled for this thread due to an Icache Miss."
},
{
- "EventCode": "2D01C",
- "EventName": "PM_CMPL_STALL_STCX",
- "BriefDescription": "Cycles in which the oldest instruction in the pipeline was a stcx waiting for resolution from the nest before completing."
- },
- {
- "EventCode": "2E018",
+ "EventCode": "0x2E018",
"EventName": "PM_DISP_STALL_FETCH",
"BriefDescription": "Cycles when dispatch was stalled for this thread because Fetch was being held."
},
{
- "EventCode": "2E01A",
+ "EventCode": "0x2E01A",
"EventName": "PM_DISP_STALL_HELD_XVFC_MAPPER_CYC",
"BriefDescription": "Cycles in which the NTC instruction is held at dispatch because the XVFC mapper/SRB was full."
},
{
- "EventCode": "2C142",
+ "EventCode": "0x2C142",
"EventName": "PM_MRK_XFER_FROM_SRC_PMC2",
"BriefDescription": "For a marked data transfer instruction, the processor's L1 data cache was reloaded from the source specified in MMCR3[15:27]. If MMCR1[16|17] is 0 (default), this count includes only lines that were reloaded to satisfy a demand miss. If MMCR1[16|17] is 1, this count includes both demand misses and prefetch reloads."
},
{
- "EventCode": "24050",
+ "EventCode": "0x24050",
"EventName": "PM_IOPS_DISP",
"BriefDescription": "Internal Operations dispatched. PM_IOPS_DISP / PM_INST_DISP will show the average number of internal operations per PowerPC instruction."
},
{
- "EventCode": "2405E",
+ "EventCode": "0x2405E",
"EventName": "PM_ISSUE_CANCEL",
"BriefDescription": "An instruction issued and the issue was later cancelled. Only one cancel per PowerPC instruction."
},
{
- "EventCode": "200FA",
+ "EventCode": "0x200FA",
"EventName": "PM_BR_TAKEN_CMPL",
"BriefDescription": "Branch Taken instruction completed."
},
{
- "EventCode": "30012",
+ "EventCode": "0x30004",
+ "EventName": "PM_DISP_STALL_FLUSH",
+ "BriefDescription": "Cycles when dispatch was stalled because of a flush that happened to an instruction(s) that was not yet NTC. PM_EXEC_STALL_NTC_FLUSH only includes instructions that were flushed after becoming NTC."
+ },
+ {
+ "EventCode": "0x3000A",
+ "EventName": "PM_DISP_STALL_ITLB_MISS",
+ "BriefDescription": "Cycles when dispatch was stalled while waiting to resolve an instruction TLB miss."
+ },
+ {
+ "EventCode": "0x30012",
"EventName": "PM_FLUSH_COMPLETION",
"BriefDescription": "The instruction that was next to complete (oldest in the pipeline) did not complete because it suffered a flush."
},
{
- "EventCode": "30014",
+ "EventCode": "0x30014",
"EventName": "PM_EXEC_STALL_STORE",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was a store instruction executing in the Load Store Unit."
},
{
- "EventCode": "30018",
+ "EventCode": "0x30018",
"EventName": "PM_DISP_STALL_HELD_SCOREBOARD_CYC",
"BriefDescription": "Cycles in which the NTC instruction is held at dispatch while waiting on the Scoreboard. This event combines VSCR and FPSCR together."
},
{
- "EventCode": "30026",
+ "EventCode": "0x30026",
"EventName": "PM_EXEC_STALL_STORE_MISS",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was a store whose cache line was not resident in the L1 and was waiting for allocation of the missing line into the L1."
},
{
- "EventCode": "3012A",
+ "EventCode": "0x3012A",
"EventName": "PM_MRK_L2_RC_DONE",
"BriefDescription": "L2 RC machine completed the transaction for the marked instruction."
},
{
- "EventCode": "3F046",
+ "EventCode": "0x3F046",
"EventName": "PM_ITLB_HIT_1G",
"BriefDescription": "Instruction TLB hit (IERAT reload) page size 1G, which implies Radix Page Table translation is in use. When MMCR1[17]=0 this event counts only for demand misses. When MMCR1[17]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "34058",
+ "EventCode": "0x34058",
"EventName": "PM_DISP_STALL_BR_MPRED_ICMISS",
"BriefDescription": "Cycles when dispatch was stalled after a mispredicted branch resulted in an instruction cache miss."
},
{
- "EventCode": "3D05C",
+ "EventCode": "0x3D05C",
"EventName": "PM_DISP_STALL_HELD_RENAME_CYC",
"BriefDescription": "Cycles in which the NTC instruction is held at dispatch because the mapper/SRB was full. Includes GPR (count, link, tar), VSR, VMR, FPR and XVFC."
},
{
- "EventCode": "3E052",
+ "EventCode": "0x3E052",
"EventName": "PM_DISP_STALL_IC_L3",
"BriefDescription": "Cycles when dispatch was stalled while the instruction was fetched from the local L3."
},
{
- "EventCode": "3E054",
+ "EventCode": "0x3E054",
"EventName": "PM_LD_MISS_L1",
"BriefDescription": "Load Missed L1, counted at execution time (can be greater than loads finished). LMQ merges are not included in this count. i.e. if a load instruction misses on an address that is already allocated on the LMQ, this event will not increment for that load). Note that this count is per slice, so if a load spans multiple slices this event will increment multiple times for a single load."
},
{
- "EventCode": "301EA",
+ "EventCode": "0x301EA",
"EventName": "PM_THRESH_EXC_1024",
"BriefDescription": "Threshold counter exceeded a value of 1024."
},
{
- "EventCode": "300FA",
+ "EventCode": "0x300FA",
"EventName": "PM_INST_FROM_L3MISS",
"BriefDescription": "The processor's instruction cache was reloaded from a source other than the local core's L1, L2, or L3 due to a demand miss."
},
{
- "EventCode": "40006",
+ "EventCode": "0x40006",
"EventName": "PM_ISSUE_KILL",
"BriefDescription": "Cycles in which an instruction or group of instructions were cancelled after being issued. This event increments once per occurrence, regardless of how many instructions are included in the issue group."
},
{
- "EventCode": "40116",
+ "EventCode": "0x40116",
"EventName": "PM_MRK_LARX_FIN",
"BriefDescription": "Marked load and reserve instruction (LARX) finished. LARX and STCX are instructions used to acquire a lock."
},
{
- "EventCode": "4C010",
+ "EventCode": "0x4C010",
"EventName": "PM_DISP_STALL_BR_MPRED_IC_L3MISS",
"BriefDescription": "Cycles when dispatch was stalled while the instruction was fetched from sources beyond the local L3 after suffering a mispredicted branch."
},
{
- "EventCode": "4D01E",
+ "EventCode": "0x4D01E",
"EventName": "PM_DISP_STALL_BR_MPRED",
"BriefDescription": "Cycles when dispatch was stalled for this thread due to a mispredicted branch."
},
{
- "EventCode": "4E010",
+ "EventCode": "0x4E010",
"EventName": "PM_DISP_STALL_IC_L3MISS",
"BriefDescription": "Cycles when dispatch was stalled while the instruction was fetched from any source beyond the local L3."
},
{
- "EventCode": "4E01A",
+ "EventCode": "0x4E01A",
"EventName": "PM_DISP_STALL_HELD_CYC",
"BriefDescription": "Cycles in which the NTC instruction is held at dispatch for any reason."
},
{
- "EventCode": "44056",
+ "EventCode": "0x4003C",
+ "EventName": "PM_DISP_STALL_HELD_SYNC_CYC",
+ "BriefDescription": "Cycles in which the NTC instruction is held at dispatch because of a synchronizing instruction that requires the ICT to be empty before dispatch."
+ },
+ {
+ "EventCode": "0x44056",
"EventName": "PM_VECTOR_ST_CMPL",
"BriefDescription": "Vector store instructions completed."
}
[
{
- "EventCode": "1E058",
+ "EventCode": "0x1E058",
"EventName": "PM_STCX_FAIL_FIN",
"BriefDescription": "Conditional store instruction (STCX) failed. LARX and STCX are instructions used to acquire a lock."
},
{
- "EventCode": "4E050",
+ "EventCode": "0x4E050",
"EventName": "PM_STCX_PASS_FIN",
"BriefDescription": "Conditional store instruction (STCX) passed. LARX and STCX are instructions used to acquire a lock."
}
[
{
- "EventCode": "1002C",
+ "EventCode": "0x1002C",
"EventName": "PM_LD_PREFETCH_CACHE_LINE_MISS",
"BriefDescription": "The L1 cache was reloaded with a line that fulfills a prefetch request."
},
{
- "EventCode": "10132",
+ "EventCode": "0x10132",
"EventName": "PM_MRK_INST_ISSUED",
"BriefDescription": "Marked instruction issued. Note that stores always get issued twice, the address gets issued to the LSU and the data gets issued to the VSU. Also, issues can sometimes get killed/cancelled and cause multiple sequential issues for the same instruction."
},
{
- "EventCode": "101E0",
+ "EventCode": "0x101E0",
"EventName": "PM_MRK_INST_DISP",
"BriefDescription": "The thread has dispatched a randomly sampled marked instruction."
},
{
- "EventCode": "101E2",
+ "EventCode": "0x101E2",
"EventName": "PM_MRK_BR_TAKEN_CMPL",
"BriefDescription": "Marked Branch Taken instruction completed."
},
{
- "EventCode": "20112",
+ "EventCode": "0x20112",
"EventName": "PM_MRK_NTF_FIN",
"BriefDescription": "The marked instruction became the oldest in the pipeline before it finished. It excludes instructions that finish at dispatch."
},
{
- "EventCode": "2C01C",
+ "EventCode": "0x2C01C",
"EventName": "PM_EXEC_STALL_DMISS_OFF_CHIP",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was waiting for a load miss to resolve from a remote chip."
},
{
- "EventCode": "20138",
+ "EventCode": "0x20138",
"EventName": "PM_MRK_ST_NEST",
"BriefDescription": "A store has been sampled/marked and is at the point of execution where it has completed in the core and can no longer be flushed. At this point the store is sent to the L2."
},
{
- "EventCode": "2013A",
+ "EventCode": "0x2013A",
"EventName": "PM_MRK_BRU_FIN",
"BriefDescription": "Marked Branch instruction finished."
},
{
- "EventCode": "2C144",
+ "EventCode": "0x2C144",
"EventName": "PM_MRK_XFER_FROM_SRC_CYC_PMC2",
"BriefDescription": "Cycles taken for a marked demand miss to reload a line from the source specified in MMCR3[15:27]."
},
{
- "EventCode": "24156",
+ "EventCode": "0x24156",
"EventName": "PM_MRK_STCX_FIN",
"BriefDescription": "Marked conditional store instruction (STCX) finished. LARX and STCX are instructions used to acquire a lock."
},
{
- "EventCode": "24158",
+ "EventCode": "0x24158",
"EventName": "PM_MRK_INST",
"BriefDescription": "An instruction was marked. Includes both Random Instruction Sampling (RIS) at decode time and Random Event Sampling (RES) at the time the configured event happens."
},
{
- "EventCode": "2415C",
+ "EventCode": "0x2415C",
"EventName": "PM_MRK_BR_CMPL",
"BriefDescription": "A marked branch completed. All branches are included."
},
{
- "EventCode": "200FD",
+ "EventCode": "0x200FD",
"EventName": "PM_L1_ICACHE_MISS",
"BriefDescription": "Demand iCache Miss."
},
{
- "EventCode": "30130",
+ "EventCode": "0x30130",
"EventName": "PM_MRK_INST_FIN",
"BriefDescription": "marked instruction finished. Excludes instructions that finish at dispatch. Note that stores always finish twice since the address gets issued to the LSU and the data gets issued to the VSU."
},
{
- "EventCode": "34146",
+ "EventCode": "0x34146",
"EventName": "PM_MRK_LD_CMPL",
"BriefDescription": "Marked loads completed."
},
{
- "EventCode": "3E158",
+ "EventCode": "0x3E158",
"EventName": "PM_MRK_STCX_FAIL",
"BriefDescription": "Marked conditional store instruction (STCX) failed. LARX and STCX are instructions used to acquire a lock."
},
{
- "EventCode": "3E15A",
+ "EventCode": "0x3E15A",
"EventName": "PM_MRK_ST_FIN",
"BriefDescription": "The marked instruction was a store of any kind."
},
{
- "EventCode": "30068",
+ "EventCode": "0x30068",
"EventName": "PM_L1_ICACHE_RELOADED_PREF",
"BriefDescription": "Counts all Icache prefetch reloads ( includes demand turned into prefetch)."
},
{
- "EventCode": "301E4",
+ "EventCode": "0x301E4",
"EventName": "PM_MRK_BR_MPRED_CMPL",
"BriefDescription": "Marked Branch Mispredicted. Includes direction and target."
},
{
- "EventCode": "300F6",
+ "EventCode": "0x300F6",
"EventName": "PM_LD_DEMAND_MISS_L1",
"BriefDescription": "The L1 cache was reloaded with a line that fulfills a demand miss request. Counted at reload time, before finish."
},
{
- "EventCode": "300FE",
+ "EventCode": "0x300FE",
"EventName": "PM_DATA_FROM_L3MISS",
"BriefDescription": "The processor's data cache was reloaded from a source other than the local core's L1, L2, or L3 due to a demand miss."
},
{
- "EventCode": "40012",
+ "EventCode": "0x40012",
"EventName": "PM_L1_ICACHE_RELOADED_ALL",
"BriefDescription": "Counts all Icache reloads includes demand, prefetch, prefetch turned into demand and demand turned into prefetch."
},
{
- "EventCode": "40134",
+ "EventCode": "0x40134",
"EventName": "PM_MRK_INST_TIMEO",
"BriefDescription": "Marked instruction finish timeout (instruction was lost)."
},
{
- "EventCode": "4003C",
- "EventName": "PM_DISP_STALL_HELD_SYNC_CYC",
- "BriefDescription": "Cycles in which the NTC instruction is held at dispatch because of a synchronizing instruction that requires the ICT to be empty before dispatch."
- },
- {
- "EventCode": "4505A",
+ "EventCode": "0x4505A",
"EventName": "PM_SP_FLOP_CMPL",
"BriefDescription": "Single Precision floating point instructions completed."
},
{
- "EventCode": "4D058",
+ "EventCode": "0x4D058",
"EventName": "PM_VECTOR_FLOP_CMPL",
"BriefDescription": "Vector floating point instructions completed."
},
{
- "EventCode": "4D05A",
+ "EventCode": "0x4D05A",
"EventName": "PM_NON_MATH_FLOP_CMPL",
"BriefDescription": "Non Math instructions completed."
},
{
- "EventCode": "401E0",
+ "EventCode": "0x401E0",
"EventName": "PM_MRK_INST_CMPL",
"BriefDescription": "marked instruction completed."
},
{
- "EventCode": "400FE",
+ "EventCode": "0x400FE",
"EventName": "PM_DATA_FROM_MEMORY",
"BriefDescription": "The processor's data cache was reloaded from local, remote, or distant memory due to a demand miss."
}
[
{
- "EventCode": "1000A",
+ "EventCode": "0x1000A",
"EventName": "PM_PMC3_REWIND",
"BriefDescription": "The speculative event selected for PMC3 rewinds and the counter for PMC3 is not charged."
},
{
- "EventCode": "1C040",
+ "EventCode": "0x1C040",
"EventName": "PM_XFER_FROM_SRC_PMC1",
"BriefDescription": "The processor's L1 data cache was reloaded from the source specified in MMCR3[0:12]. If MMCR1[16|17] is 0 (default), this count includes only lines that were reloaded to satisfy a demand miss. If MMCR1[16|17] is 1, this count includes both demand misses and prefetch reloads."
},
{
- "EventCode": "1C142",
+ "EventCode": "0x1C142",
"EventName": "PM_MRK_XFER_FROM_SRC_PMC1",
"BriefDescription": "For a marked data transfer instruction, the processor's L1 data cache was reloaded from the source specified in MMCR3[0:12]. If MMCR1[16|17] is 0 (default), this count includes only lines that were reloaded to satisfy a demand miss. If MMCR1[16|17] is 1, this count includes both demand misses and prefetch reloads."
},
{
- "EventCode": "1C144",
+ "EventCode": "0x1C144",
"EventName": "PM_MRK_XFER_FROM_SRC_CYC_PMC1",
"BriefDescription": "Cycles taken for a marked demand miss to reload a line from the source specified in MMCR3[0:12]."
},
{
- "EventCode": "1C056",
+ "EventCode": "0x1C056",
"EventName": "PM_DERAT_MISS_4K",
"BriefDescription": "Data ERAT Miss (Data TLB Access) page size 4K. When MMCR1[16]=0 this event counts only DERAT reloads for demand misses. When MMCR1[16]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "1C058",
+ "EventCode": "0x1C058",
"EventName": "PM_DTLB_MISS_16G",
"BriefDescription": "Data TLB reload (after a miss) page size 16G. When MMCR1[16]=0 this event counts only for demand misses. When MMCR1[16]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "1C05C",
+ "EventCode": "0x1C05C",
"EventName": "PM_DTLB_MISS_2M",
"BriefDescription": "Data TLB reload (after a miss) page size 2M. Implies radix translation was used. When MMCR1[16]=0 this event counts only for demand misses. When MMCR1[16]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "1E056",
+ "EventCode": "0x1E056",
"EventName": "PM_EXEC_STALL_STORE_PIPE",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was executing in the store unit. This does not include cycles spent handling store misses, PTESYNC instructions or TLBIE instructions."
},
{
- "EventCode": "1F150",
+ "EventCode": "0x1F150",
"EventName": "PM_MRK_ST_L2_CYC",
"BriefDescription": "Cycles from L2 RC dispatch to L2 RC completion."
},
{
- "EventCode": "10062",
+ "EventCode": "0x10062",
"EventName": "PM_LD_L3MISS_PEND_CYC",
"BriefDescription": "Cycles L3 miss was pending for this thread."
},
{
- "EventCode": "20010",
+ "EventCode": "0x20010",
"EventName": "PM_PMC1_OVERFLOW",
"BriefDescription": "The event selected for PMC1 caused the event counter to overflow."
},
{
- "EventCode": "2001A",
+ "EventCode": "0x2001A",
"EventName": "PM_ITLB_HIT",
"BriefDescription": "The PTE required to translate the instruction address was resident in the TLB (instruction TLB access/IERAT reload). Applies to both HPT and RPT. When MMCR1[17]=0 this event counts only for demand misses. When MMCR1[17]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "2003E",
+ "EventCode": "0x2003E",
"EventName": "PM_PTESYNC_FIN",
"BriefDescription": "Ptesync instruction finished in the store unit. Only one ptesync can finish at a time."
},
{
- "EventCode": "2C040",
+ "EventCode": "0x2C040",
"EventName": "PM_XFER_FROM_SRC_PMC2",
"BriefDescription": "The processor's L1 data cache was reloaded from the source specified in MMCR3[15:27]. If MMCR1[16|17] is 0 (default), this count includes only lines that were reloaded to satisfy a demand miss. If MMCR1[16|17] is 1, this count includes both demand misses and prefetch reloads."
},
{
- "EventCode": "2C054",
+ "EventCode": "0x2C054",
"EventName": "PM_DERAT_MISS_64K",
"BriefDescription": "Data ERAT Miss (Data TLB Access) page size 64K. When MMCR1[16]=0 this event counts only DERAT reloads for demand misses. When MMCR1[16]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "2C056",
+ "EventCode": "0x2C056",
"EventName": "PM_DTLB_MISS_4K",
"BriefDescription": "Data TLB reload (after a miss) page size 4K. When MMCR1[16]=0 this event counts only for demand misses. When MMCR1[16]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "2D154",
+ "EventCode": "0x2D154",
"EventName": "PM_MRK_DERAT_MISS_64K",
"BriefDescription": "Data ERAT Miss (Data TLB Access) page size 64K for a marked instruction. When MMCR1[16]=0 this event counts only DERAT reloads for demand misses. When MMCR1[16]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "200F6",
+ "EventCode": "0x200F6",
"EventName": "PM_DERAT_MISS",
"BriefDescription": "DERAT Reloaded to satisfy a DERAT miss. All page sizes are counted by this event. When MMCR1[16]=0 this event counts only DERAT reloads for demand misses. When MMCR1[16]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "3000A",
- "EventName": "PM_DISP_STALL_ITLB_MISS",
- "BriefDescription": "Cycles when dispatch was stalled while waiting to resolve an instruction TLB miss."
- },
- {
- "EventCode": "30016",
+ "EventCode": "0x30016",
"EventName": "PM_EXEC_STALL_DERAT_DTLB_MISS",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline suffered a TLB miss and waited for it resolve."
},
{
- "EventCode": "3C040",
+ "EventCode": "0x3C040",
"EventName": "PM_XFER_FROM_SRC_PMC3",
"BriefDescription": "The processor's L1 data cache was reloaded from the source specified in MMCR3[30:42]. If MMCR1[16|17] is 0 (default), this count includes only lines that were reloaded to satisfy a demand miss. If MMCR1[16|17] is 1, this count includes both demand misses and prefetch reloads."
},
{
- "EventCode": "3C142",
+ "EventCode": "0x3C142",
"EventName": "PM_MRK_XFER_FROM_SRC_PMC3",
"BriefDescription": "For a marked data transfer instruction, the processor's L1 data cache was reloaded from the source specified in MMCR3[30:42]. If MMCR1[16|17] is 0 (default), this count includes only lines that were reloaded to satisfy a demand miss. If MMCR1[16|17] is 1, this count includes both demand misses and prefetch reloads."
},
{
- "EventCode": "3C144",
+ "EventCode": "0x3C144",
"EventName": "PM_MRK_XFER_FROM_SRC_CYC_PMC3",
"BriefDescription": "Cycles taken for a marked demand miss to reload a line from the source specified in MMCR3[30:42]."
},
{
- "EventCode": "3C054",
+ "EventCode": "0x3C054",
"EventName": "PM_DERAT_MISS_16M",
"BriefDescription": "Data ERAT Miss (Data TLB Access) page size 16M. When MMCR1[16]=0 this event counts only DERAT reloads for demand misses. When MMCR1[16]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "3C056",
+ "EventCode": "0x3C056",
"EventName": "PM_DTLB_MISS_64K",
"BriefDescription": "Data TLB reload (after a miss) page size 64K. When MMCR1[16]=0 this event counts only for demand misses. When MMCR1[16]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "3C058",
+ "EventCode": "0x3C058",
"EventName": "PM_LARX_FIN",
"BriefDescription": "Load and reserve instruction (LARX) finished. LARX and STCX are instructions used to acquire a lock."
},
{
- "EventCode": "301E2",
+ "EventCode": "0x301E2",
"EventName": "PM_MRK_ST_CMPL",
"BriefDescription": "Marked store completed and sent to nest. Note that this count excludes cache-inhibited stores."
},
{
- "EventCode": "300FC",
+ "EventCode": "0x300FC",
"EventName": "PM_DTLB_MISS",
"BriefDescription": "The DPTEG required for the load/store instruction in execution was missing from the TLB. It includes pages of all sizes for demand and prefetch activity."
},
{
- "EventCode": "4D02C",
+ "EventCode": "0x4D02C",
"EventName": "PM_PMC1_REWIND",
"BriefDescription": "The speculative event selected for PMC1 rewinds and the counter for PMC1 is not charged."
},
{
- "EventCode": "4003E",
+ "EventCode": "0x4003E",
"EventName": "PM_LD_CMPL",
"BriefDescription": "Loads completed."
},
{
- "EventCode": "4C040",
+ "EventCode": "0x4C040",
"EventName": "PM_XFER_FROM_SRC_PMC4",
"BriefDescription": "The processor's L1 data cache was reloaded from the source specified in MMCR3[45:57]. If MMCR1[16|17] is 0 (default), this count includes only lines that were reloaded to satisfy a demand miss. If MMCR1[16|17] is 1, this count includes both demand misses and prefetch reloads."
},
{
- "EventCode": "4C142",
+ "EventCode": "0x4C142",
"EventName": "PM_MRK_XFER_FROM_SRC_PMC4",
"BriefDescription": "For a marked data transfer instruction, the processor's L1 data cache was reloaded from the source specified in MMCR3[45:57]. If MMCR1[16|17] is 0 (default), this count includes only lines that were reloaded to satisfy a demand miss. If MMCR1[16|17] is 1, this count includes both demand misses and prefetch reloads."
},
{
- "EventCode": "4C144",
+ "EventCode": "0x4C144",
"EventName": "PM_MRK_XFER_FROM_SRC_CYC_PMC4",
"BriefDescription": "Cycles taken for a marked demand miss to reload a line from the source specified in MMCR3[45:57]."
},
{
- "EventCode": "4C056",
+ "EventCode": "0x4C056",
"EventName": "PM_DTLB_MISS_16M",
"BriefDescription": "Data TLB reload (after a miss) page size 16M. When MMCR1[16]=0 this event counts only for demand misses. When MMCR1[16]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "4C05A",
+ "EventCode": "0x4C05A",
"EventName": "PM_DTLB_MISS_1G",
"BriefDescription": "Data TLB reload (after a miss) page size 1G. Implies radix translation was used. When MMCR1[16]=0 this event counts only for demand misses. When MMCR1[16]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "4C15E",
+ "EventCode": "0x4C15E",
"EventName": "PM_MRK_DTLB_MISS_64K",
"BriefDescription": "Marked Data TLB reload (after a miss) page size 64K. When MMCR1[16]=0 this event counts only for demand misses. When MMCR1[16]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "4D056",
+ "EventCode": "0x4D056",
"EventName": "PM_NON_FMA_FLOP_CMPL",
"BriefDescription": "Non FMA instruction completed."
},
{
- "EventCode": "40164",
+ "EventCode": "0x40164",
"EventName": "PM_MRK_DERAT_MISS_2M",
"BriefDescription": "Data ERAT Miss (Data TLB Access) page size 2M for a marked instruction. When MMCR1[16]=0 this event counts only DERAT reloads for demand misses. When MMCR1[16]=1 this event includes demand misses and prefetches."
}
[
{
- "EventCode": "10016",
+ "EventCode": "0x10016",
"EventName": "PM_VSU0_ISSUE",
"BriefDescription": "VSU instructions issued to VSU pipe 0."
},
{
- "EventCode": "1001C",
+ "EventCode": "0x1001C",
"EventName": "PM_ULTRAVISOR_INST_CMPL",
"BriefDescription": "PowerPC instructions that completed while the thread was in ultravisor state."
},
{
- "EventCode": "100F0",
+ "EventCode": "0x100F0",
"EventName": "PM_CYC",
"BriefDescription": "Processor cycles."
},
{
- "EventCode": "10134",
+ "EventCode": "0x10134",
"EventName": "PM_MRK_ST_DONE_L2",
"BriefDescription": "Marked stores completed in L2 (RC machine done)."
},
{
- "EventCode": "1505E",
+ "EventCode": "0x1505E",
"EventName": "PM_LD_HIT_L1",
"BriefDescription": "Loads that finished without experiencing an L1 miss."
},
{
- "EventCode": "1D05E",
- "EventName": "PM_DISP_STALL_HELD_HALT_CYC",
- "BriefDescription": "Cycles in which the NTC instruction is held at dispatch because of power management."
- },
- {
- "EventCode": "1E054",
- "EventName": "PM_EXEC_STALL_DMISS_L21_L31",
- "BriefDescription": "Cycles in which the oldest instruction in the pipeline was waiting for a load miss to resolve from another core's L2 or L3 on the same chip."
- },
- {
- "EventCode": "1E05A",
- "EventName": "PM_CMPL_STALL_LWSYNC",
- "BriefDescription": "Cycles in which the oldest instruction in the pipeline was a lwsync waiting to complete."
- },
- {
- "EventCode": "1F056",
+ "EventCode": "0x1F056",
"EventName": "PM_DISP_SS0_2_INSTR_CYC",
"BriefDescription": "Cycles in which Superslice 0 dispatches either 1 or 2 instructions."
},
{
- "EventCode": "1F15C",
+ "EventCode": "0x1F15C",
"EventName": "PM_MRK_STCX_L2_CYC",
"BriefDescription": "Cycles spent in the nest portion of a marked Stcx instruction. It starts counting when the operation starts to drain to the L2 and it stops counting when the instruction retires from the Instruction Completion Table (ICT) in the Instruction Sequencing Unit (ISU)."
},
{
- "EventCode": "10066",
+ "EventCode": "0x10066",
"EventName": "PM_ADJUNCT_CYC",
"BriefDescription": "Cycles in which the thread is in Adjunct state. MSR[S HV PR] bits = 011."
},
{
- "EventCode": "101E4",
+ "EventCode": "0x101E4",
"EventName": "PM_MRK_L1_ICACHE_MISS",
"BriefDescription": "Marked Instruction suffered an icache Miss."
},
{
- "EventCode": "101EA",
+ "EventCode": "0x101EA",
"EventName": "PM_MRK_L1_RELOAD_VALID",
"BriefDescription": "Marked demand reload."
},
{
- "EventCode": "100F4",
+ "EventCode": "0x100F4",
"EventName": "PM_FLOP_CMPL",
"BriefDescription": "Floating Point Operations Completed. Includes any type. It counts once for each 1, 2, 4 or 8 flop instruction. Use PM_1|2|4|8_FLOP_CMPL events to count flops."
},
{
- "EventCode": "100FA",
+ "EventCode": "0x100FA",
"EventName": "PM_RUN_LATCH_ANY_THREAD_CYC",
"BriefDescription": "Cycles when at least one thread has the run latch set."
},
{
- "EventCode": "100FC",
+ "EventCode": "0x100FC",
"EventName": "PM_LD_REF_L1",
"BriefDescription": "All L1 D cache load references counted at finish, gated by reject. In P9 and earlier this event counted only cacheable loads but in P10 both cacheable and non-cacheable loads are included."
},
{
- "EventCode": "20006",
- "EventName": "PM_DISP_STALL_HELD_ISSQ_FULL_CYC",
- "BriefDescription": "Cycles in which the NTC instruction is held at dispatch due to Issue queue full. Includes issue queue and branch queue."
- },
- {
- "EventCode": "2000C",
+ "EventCode": "0x2000C",
"EventName": "PM_RUN_LATCH_ALL_THREADS_CYC",
"BriefDescription": "Cycles when the run latch is set for all threads."
},
{
- "EventCode": "2E010",
+ "EventCode": "0x2E010",
"EventName": "PM_ADJUNCT_INST_CMPL",
"BriefDescription": "PowerPC instructions that completed while the thread is in Adjunct state."
},
{
- "EventCode": "2E014",
+ "EventCode": "0x2E014",
"EventName": "PM_STCX_FIN",
"BriefDescription": "Conditional store instruction (STCX) finished. LARX and STCX are instructions used to acquire a lock."
},
{
- "EventCode": "20130",
+ "EventCode": "0x20130",
"EventName": "PM_MRK_INST_DECODED",
"BriefDescription": "An instruction was marked at decode time. Random Instruction Sampling (RIS) only."
},
{
- "EventCode": "20132",
+ "EventCode": "0x20132",
"EventName": "PM_MRK_DFU_ISSUE",
"BriefDescription": "The marked instruction was a decimal floating point operation issued to the VSU. Measured at issue time."
},
{
- "EventCode": "20134",
+ "EventCode": "0x20134",
"EventName": "PM_MRK_FXU_ISSUE",
"BriefDescription": "The marked instruction was a fixed point operation issued to the VSU. Measured at issue time."
},
{
- "EventCode": "2505C",
+ "EventCode": "0x2505C",
"EventName": "PM_VSU_ISSUE",
"BriefDescription": "At least one VSU instruction was issued to one of the VSU pipes. Up to 4 per cycle. Includes fixed point operations."
},
{
- "EventCode": "2F054",
+ "EventCode": "0x2F054",
"EventName": "PM_DISP_SS1_2_INSTR_CYC",
"BriefDescription": "Cycles in which Superslice 1 dispatches either 1 or 2 instructions."
},
{
- "EventCode": "2F056",
+ "EventCode": "0x2F056",
"EventName": "PM_DISP_SS1_4_INSTR_CYC",
"BriefDescription": "Cycles in which Superslice 1 dispatches either 3 or 4 instructions."
},
{
- "EventCode": "2006C",
+ "EventCode": "0x2006C",
"EventName": "PM_RUN_CYC_SMT4_MODE",
"BriefDescription": "Cycles when this thread's run latch is set and the core is in SMT4 mode."
},
{
- "EventCode": "201E0",
+ "EventCode": "0x201E0",
"EventName": "PM_MRK_DATA_FROM_MEMORY",
"BriefDescription": "The processor's data cache was reloaded from local, remote, or distant memory due to a demand miss for a marked load."
},
{
- "EventCode": "201E4",
+ "EventCode": "0x201E4",
"EventName": "PM_MRK_DATA_FROM_L3MISS",
"BriefDescription": "The processor's data cache was reloaded from a source other than the local core's L1, L2, or L3 due to a demand miss for a marked load."
},
{
- "EventCode": "201E8",
+ "EventCode": "0x201E8",
"EventName": "PM_THRESH_EXC_512",
"BriefDescription": "Threshold counter exceeded a value of 512."
},
{
- "EventCode": "200F2",
+ "EventCode": "0x200F2",
"EventName": "PM_INST_DISP",
"BriefDescription": "PowerPC instructions dispatched."
},
{
- "EventCode": "30132",
+ "EventCode": "0x30132",
"EventName": "PM_MRK_VSU_FIN",
"BriefDescription": "VSU marked instructions finished. Excludes simple FX instructions issued to the Store Unit."
},
{
- "EventCode": "30038",
+ "EventCode": "0x30038",
"EventName": "PM_EXEC_STALL_DMISS_LMEM",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was waiting for a load miss to resolve from the local memory, local OpenCapp cache, or local OpenCapp memory."
},
{
- "EventCode": "3F04A",
+ "EventCode": "0x3F04A",
"EventName": "PM_LSU_ST5_FIN",
"BriefDescription": "LSU Finished an internal operation in ST2 port."
},
{
- "EventCode": "34054",
- "EventName": "PM_EXEC_STALL_DMISS_L2L3_NOCONFLICT",
- "BriefDescription": "Cycles in which the oldest instruction in the pipeline was waiting for a load miss to resolve from the local L2 or local L3, without a dispatch conflict."
- },
- {
- "EventCode": "3405A",
+ "EventCode": "0x3405A",
"EventName": "PM_PRIVILEGED_INST_CMPL",
"BriefDescription": "PowerPC Instructions that completed while the thread is in Privileged state."
},
{
- "EventCode": "3F150",
+ "EventCode": "0x3F150",
"EventName": "PM_MRK_ST_DRAIN_CYC",
"BriefDescription": "cycles to drain st from core to L2."
},
{
- "EventCode": "3F054",
+ "EventCode": "0x3F054",
"EventName": "PM_DISP_SS0_4_INSTR_CYC",
"BriefDescription": "Cycles in which Superslice 0 dispatches either 3 or 4 instructions."
},
{
- "EventCode": "3F056",
+ "EventCode": "0x3F056",
"EventName": "PM_DISP_SS0_8_INSTR_CYC",
"BriefDescription": "Cycles in which Superslice 0 dispatches either 5, 6, 7 or 8 instructions."
},
{
- "EventCode": "30162",
+ "EventCode": "0x30162",
"EventName": "PM_MRK_ISSUE_DEPENDENT_LOAD",
"BriefDescription": "The marked instruction was dependent on a load. It is eligible for issue kill."
},
{
- "EventCode": "40114",
+ "EventCode": "0x40114",
"EventName": "PM_MRK_START_PROBE_NOP_DISP",
"BriefDescription": "Marked Start probe nop dispatched. Instruction AND R0,R0,R0."
},
{
- "EventCode": "4001C",
+ "EventCode": "0x4001C",
"EventName": "PM_VSU_FIN",
"BriefDescription": "VSU instructions finished."
},
{
- "EventCode": "4C01A",
+ "EventCode": "0x4C01A",
"EventName": "PM_EXEC_STALL_DMISS_OFF_NODE",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was waiting for a load miss to resolve from a distant chip."
},
{
- "EventCode": "4D012",
+ "EventCode": "0x4D012",
"EventName": "PM_PMC3_SAVED",
"BriefDescription": "The conditions for the speculative event selected for PMC3 are met and PMC3 is charged."
},
{
- "EventCode": "4D022",
+ "EventCode": "0x4D022",
"EventName": "PM_HYPERVISOR_INST_CMPL",
"BriefDescription": "PowerPC instructions that completed while the thread is in hypervisor state."
},
{
- "EventCode": "4D026",
+ "EventCode": "0x4D026",
"EventName": "PM_ULTRAVISOR_CYC",
"BriefDescription": "Cycles when the thread is in Ultravisor state. MSR[S HV PR]=110."
},
{
- "EventCode": "4D028",
+ "EventCode": "0x4D028",
"EventName": "PM_PRIVILEGED_CYC",
"BriefDescription": "Cycles when the thread is in Privileged state. MSR[S HV PR]=x00."
},
{
- "EventCode": "40030",
+ "EventCode": "0x40030",
"EventName": "PM_INST_FIN",
"BriefDescription": "Instructions finished."
},
{
- "EventCode": "44146",
+ "EventCode": "0x44146",
"EventName": "PM_MRK_STCX_CORE_CYC",
"BriefDescription": "Cycles spent in the core portion of a marked Stcx instruction. It starts counting when the instruction is decoded and stops counting when it drains into the L2."
},
{
- "EventCode": "44054",
+ "EventCode": "0x44054",
"EventName": "PM_VECTOR_LD_CMPL",
"BriefDescription": "Vector load instructions completed."
},
{
- "EventCode": "45054",
+ "EventCode": "0x45054",
"EventName": "PM_FMA_CMPL",
"BriefDescription": "Two floating point instructions completed (FMA class of instructions: fmadd, fnmadd, fmsub, fnmsub). Scalar instructions only."
},
{
- "EventCode": "45056",
+ "EventCode": "0x45056",
"EventName": "PM_SCALAR_FLOP_CMPL",
"BriefDescription": "Scalar floating point instructions completed."
},
{
- "EventCode": "4505C",
+ "EventCode": "0x4505C",
"EventName": "PM_MATH_FLOP_CMPL",
"BriefDescription": "Math floating point instructions completed."
},
{
- "EventCode": "4D05E",
+ "EventCode": "0x4D05E",
"EventName": "PM_BR_CMPL",
"BriefDescription": "A branch completed. All branches are included."
},
{
- "EventCode": "4E15E",
+ "EventCode": "0x4E15E",
"EventName": "PM_MRK_INST_FLUSHED",
"BriefDescription": "The marked instruction was flushed."
},
{
- "EventCode": "401E6",
+ "EventCode": "0x401E6",
"EventName": "PM_MRK_INST_FROM_L3MISS",
"BriefDescription": "The processor's instruction cache was reloaded from a source other than the local core's L1, L2, or L3 due to a demand miss for a marked instruction."
},
{
- "EventCode": "401E8",
+ "EventCode": "0x401E8",
"EventName": "PM_MRK_DATA_FROM_L2MISS",
"BriefDescription": "The processor's data cache was reloaded from a source other than the local core's L1 or L2 due to a demand miss for a marked load."
},
{
- "EventCode": "400F0",
+ "EventCode": "0x400F0",
"EventName": "PM_LD_DEMAND_MISS_L1_FIN",
"BriefDescription": "Load Missed L1, counted at finish time."
},
{
- "EventCode": "400FA",
+ "EventCode": "0x400FA",
"EventName": "PM_RUN_INST_CMPL",
"BriefDescription": "Completed PowerPC instructions gated by the run latch."
}
[
{
- "EventCode": "100FE",
+ "EventCode": "0x100FE",
"EventName": "PM_INST_CMPL",
"BriefDescription": "PowerPC instructions completed."
},
{
- "EventCode": "10006",
- "EventName": "PM_DISP_STALL_HELD_OTHER_CYC",
- "BriefDescription": "Cycles in which the NTC instruction is held at dispatch for any other reason."
- },
- {
- "EventCode": "1000C",
+ "EventCode": "0x1000C",
"EventName": "PM_LSU_LD0_FIN",
"BriefDescription": "LSU Finished an internal operation in LD0 port."
},
{
- "EventCode": "1000E",
+ "EventCode": "0x1000E",
"EventName": "PM_MMA_ISSUED",
"BriefDescription": "MMA instructions issued."
},
{
- "EventCode": "10012",
+ "EventCode": "0x10012",
"EventName": "PM_LSU_ST0_FIN",
"BriefDescription": "LSU Finished an internal operation in ST0 port."
},
{
- "EventCode": "10014",
+ "EventCode": "0x10014",
"EventName": "PM_LSU_ST4_FIN",
"BriefDescription": "LSU Finished an internal operation in ST4 port."
},
{
- "EventCode": "10018",
+ "EventCode": "0x10018",
"EventName": "PM_IC_DEMAND_CYC",
"BriefDescription": "Cycles in which an instruction reload is pending to satisfy a demand miss."
},
{
- "EventCode": "10022",
+ "EventCode": "0x10022",
"EventName": "PM_PMC2_SAVED",
"BriefDescription": "The conditions for the speculative event selected for PMC2 are met and PMC2 is charged."
},
{
- "EventCode": "10024",
+ "EventCode": "0x10024",
"EventName": "PM_PMC5_OVERFLOW",
"BriefDescription": "The event selected for PMC5 caused the event counter to overflow."
},
{
- "EventCode": "10058",
+ "EventCode": "0x10058",
"EventName": "PM_EXEC_STALL_FIN_AT_DISP",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline finished at dispatch and did not require execution in the LSU, BRU or VSU."
},
{
- "EventCode": "1005A",
+ "EventCode": "0x1005A",
"EventName": "PM_FLUSH_MPRED",
"BriefDescription": "A flush occurred due to a mispredicted branch. Includes target and direction."
},
{
- "EventCode": "1C05A",
+ "EventCode": "0x1C05A",
"EventName": "PM_DERAT_MISS_2M",
"BriefDescription": "Data ERAT Miss (Data TLB Access) page size 2M. Implies radix translation. When MMCR1[16]=0 this event counts only DERAT reloads for demand misses. When MMCR1[16]=1 this event includes demand misses and prefetches."
},
{
- "EventCode": "10064",
- "EventName": "PM_DISP_STALL_IC_L2",
- "BriefDescription": "Cycles when dispatch was stalled while the instruction was fetched from the local L2."
+ "EventCode": "0x1E05A",
+ "EventName": "PM_CMPL_STALL_LWSYNC",
+ "BriefDescription": "Cycles in which the oldest instruction in the pipeline was a lwsync waiting to complete."
},
{
- "EventCode": "10068",
+ "EventCode": "0x10068",
"EventName": "PM_BR_FIN",
"BriefDescription": "A branch instruction finished. Includes predicted/mispredicted/unconditional."
},
{
- "EventCode": "1006A",
+ "EventCode": "0x1006A",
"EventName": "PM_FX_LSU_FIN",
"BriefDescription": "Simple fixed point instruction issued to the store unit. Measured at finish time."
},
{
- "EventCode": "1006C",
+ "EventCode": "0x1006C",
"EventName": "PM_RUN_CYC_ST_MODE",
"BriefDescription": "Cycles when the run latch is set and the core is in ST mode."
},
{
- "EventCode": "20004",
+ "EventCode": "0x20004",
"EventName": "PM_ISSUE_STALL",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was dispatched but not issued yet."
},
{
- "EventCode": "2000A",
+ "EventCode": "0x2000A",
"EventName": "PM_HYPERVISOR_CYC",
"BriefDescription": "Cycles when the thread is in Hypervisor state. MSR[S HV PR]=010."
},
{
- "EventCode": "2000E",
+ "EventCode": "0x2000E",
"EventName": "PM_LSU_LD1_FIN",
"BriefDescription": "LSU Finished an internal operation in LD1 port."
},
{
- "EventCode": "2C014",
+ "EventCode": "0x2C014",
"EventName": "PM_CMPL_STALL_SPECIAL",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline required special handling before completing."
},
{
- "EventCode": "2C018",
+ "EventCode": "0x2C018",
"EventName": "PM_EXEC_STALL_DMISS_L3MISS",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was waiting for a load miss to resolve from a source beyond the local L2 or local L3."
},
{
- "EventCode": "2D010",
+ "EventCode": "0x2D010",
"EventName": "PM_LSU_ST1_FIN",
"BriefDescription": "LSU Finished an internal operation in ST1 port."
},
{
- "EventCode": "2D012",
+ "EventCode": "0x2D012",
"EventName": "PM_VSU1_ISSUE",
"BriefDescription": "VSU instructions issued to VSU pipe 1."
},
{
- "EventCode": "2D018",
+ "EventCode": "0x2D018",
"EventName": "PM_EXEC_STALL_VSU",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was executing in the VSU (includes FXU, VSU, CRU)."
},
{
- "EventCode": "2E01E",
+ "EventCode": "0x2D01C",
+ "EventName": "PM_CMPL_STALL_STCX",
+ "BriefDescription": "Cycles in which the oldest instruction in the pipeline was a stcx waiting for resolution from the nest before completing."
+ },
+ {
+ "EventCode": "0x2E01E",
"EventName": "PM_EXEC_STALL_NTC_FLUSH",
- "BriefDescription": "Cycles in which the oldest instruction in the pipeline was executing in any unit before it was flushed. Note that if the flush of the oldest instruction happens after finish, the cycles from dispatch to issue will be included in PM_DISP_STALL and the cycles from issue to finish will be included in PM_EXEC_STALL and its corresponding children."
+ "BriefDescription": "Cycles in which the oldest instruction in the pipeline was executing in any unit before it was flushed. Note that if the flush of the oldest instruction happens after finish, the cycles from dispatch to issue will be included in PM_DISP_STALL and the cycles from issue to finish will be included in PM_EXEC_STALL and its corresponding children. This event will also count cycles when the previous NTF instruction is still completing and the new NTF instruction is stalled at dispatch."
},
{
- "EventCode": "2013C",
+ "EventCode": "0x2013C",
"EventName": "PM_MRK_FX_LSU_FIN",
"BriefDescription": "The marked instruction was simple fixed point that was issued to the store unit. Measured at finish time."
},
{
- "EventCode": "2405A",
+ "EventCode": "0x2405A",
"EventName": "PM_NTC_FIN",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline (NTC) finishes. Note that instructions can finish out of order, therefore not all the instructions that finish have a Next-to-complete status."
},
{
- "EventCode": "201E2",
+ "EventCode": "0x201E2",
"EventName": "PM_MRK_LD_MISS_L1",
"BriefDescription": "Marked DL1 Demand Miss counted at finish time."
},
{
- "EventCode": "200F4",
+ "EventCode": "0x200F4",
"EventName": "PM_RUN_CYC",
"BriefDescription": "Processor cycles gated by the run latch."
},
{
- "EventCode": "30004",
- "EventName": "PM_DISP_STALL_FLUSH",
- "BriefDescription": "Cycles when dispatch was stalled because of a flush that happened to an instruction(s) that was not yet NTC. PM_EXEC_STALL_NTC_FLUSH only includes instructions that were flushed after becoming NTC."
- },
- {
- "EventCode": "30008",
+ "EventCode": "0x30008",
"EventName": "PM_EXEC_STALL",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was waiting to finish in one of the execution units (BRU, LSU, VSU). Only cycles between issue and finish are counted in this category."
},
{
- "EventCode": "3001A",
+ "EventCode": "0x3001A",
"EventName": "PM_LSU_ST2_FIN",
"BriefDescription": "LSU Finished an internal operation in ST2 port."
},
{
- "EventCode": "30020",
+ "EventCode": "0x30020",
"EventName": "PM_PMC2_REWIND",
"BriefDescription": "The speculative event selected for PMC2 rewinds and the counter for PMC2 is not charged."
},
{
- "EventCode": "30022",
+ "EventCode": "0x30022",
"EventName": "PM_PMC4_SAVED",
"BriefDescription": "The conditions for the speculative event selected for PMC4 are met and PMC4 is charged."
},
{
- "EventCode": "30024",
+ "EventCode": "0x30024",
"EventName": "PM_PMC6_OVERFLOW",
"BriefDescription": "The event selected for PMC6 caused the event counter to overflow."
},
{
- "EventCode": "30028",
+ "EventCode": "0x30028",
"EventName": "PM_CMPL_STALL_MEM_ECC",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was waiting for the non-speculative finish of either a stcx waiting for its result or a load waiting for non-critical sectors of data and ECC."
},
{
- "EventCode": "30036",
+ "EventCode": "0x30036",
"EventName": "PM_EXEC_STALL_SIMPLE_FX",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was a simple fixed point instruction executing in the Load Store Unit."
},
{
- "EventCode": "3003A",
+ "EventCode": "0x3003A",
"EventName": "PM_CMPL_STALL_EXCEPTION",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was not allowed to complete because it was interrupted by ANY exception, which has to be serviced before the instruction can complete."
},
{
- "EventCode": "3F044",
+ "EventCode": "0x3F044",
"EventName": "PM_VSU2_ISSUE",
"BriefDescription": "VSU instructions issued to VSU pipe 2."
},
{
- "EventCode": "30058",
+ "EventCode": "0x30058",
"EventName": "PM_TLBIE_FIN",
"BriefDescription": "TLBIE instructions finished in the LSU. Two TLBIEs can finish each cycle. All will be counted."
},
{
- "EventCode": "3D058",
+ "EventCode": "0x3D058",
"EventName": "PM_SCALAR_FSQRT_FDIV_ISSUE",
"BriefDescription": "Scalar versions of four floating point operations: fdiv,fsqrt (xvdivdp, xvdivsp, xvsqrtdp, xvsqrtsp)."
},
{
- "EventCode": "30066",
+ "EventCode": "0x30066",
"EventName": "PM_LSU_FIN",
"BriefDescription": "LSU Finished an internal operation (up to 4 per cycle)."
},
{
- "EventCode": "40004",
+ "EventCode": "0x40004",
"EventName": "PM_FXU_ISSUE",
"BriefDescription": "A fixed point instruction was issued to the VSU."
},
{
- "EventCode": "40008",
+ "EventCode": "0x40008",
"EventName": "PM_NTC_ALL_FIN",
"BriefDescription": "Cycles in which both instructions in the ICT entry pair show as finished. These are the cycles between finish and completion for the oldest pair of instructions in the pipeline."
},
{
- "EventCode": "40010",
+ "EventCode": "0x40010",
"EventName": "PM_PMC3_OVERFLOW",
"BriefDescription": "The event selected for PMC3 caused the event counter to overflow."
},
{
- "EventCode": "4C012",
+ "EventCode": "0x4C012",
"EventName": "PM_EXEC_STALL_DERAT_ONLY_MISS",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline suffered an ERAT miss and waited for it resolve."
},
{
- "EventCode": "4C018",
+ "EventCode": "0x4C018",
"EventName": "PM_CMPL_STALL",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline cannot complete because the thread was blocked for any reason."
},
{
- "EventCode": "4C01E",
+ "EventCode": "0x4C01E",
"EventName": "PM_LSU_ST3_FIN",
"BriefDescription": "LSU Finished an internal operation in ST3 port."
},
{
- "EventCode": "4D018",
+ "EventCode": "0x4D018",
"EventName": "PM_EXEC_STALL_BRU",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was executing in the Branch unit."
},
{
- "EventCode": "4D01A",
+ "EventCode": "0x4D01A",
"EventName": "PM_CMPL_STALL_HWSYNC",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was a hwsync waiting for response from L2 before completing."
},
{
- "EventCode": "4D01C",
+ "EventCode": "0x4D01C",
"EventName": "PM_EXEC_STALL_TLBIEL",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was a TLBIEL instruction executing in the Load Store Unit. TLBIEL instructions have lower overhead than TLBIE instructions because they don't get set to the nest."
},
{
- "EventCode": "4E012",
+ "EventCode": "0x4E012",
"EventName": "PM_EXEC_STALL_UNKNOWN",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline completed without an ntf_type pulse. The ntf_pulse was missed by the ISU because the NTF finishes and completions came too close together."
},
{
- "EventCode": "4D020",
+ "EventCode": "0x4D020",
"EventName": "PM_VSU3_ISSUE",
"BriefDescription": "VSU instruction was issued to VSU pipe 3."
},
{
- "EventCode": "40132",
+ "EventCode": "0x40132",
"EventName": "PM_MRK_LSU_FIN",
"BriefDescription": "LSU marked instruction finish."
},
{
- "EventCode": "45058",
+ "EventCode": "0x45058",
"EventName": "PM_IC_MISS_CMPL",
"BriefDescription": "Non-speculative icache miss, counted at completion."
},
{
- "EventCode": "4D050",
+ "EventCode": "0x4D050",
"EventName": "PM_VSU_NON_FLOP_CMPL",
"BriefDescription": "Non-floating point VSU instructions completed."
},
{
- "EventCode": "4D052",
+ "EventCode": "0x4D052",
"EventName": "PM_2FLOP_CMPL",
"BriefDescription": "Double Precision vector version of fmul, fsub, fcmp, fsel, fabs, fnabs, fres, fsqrte, fneg completed."
},
{
- "EventCode": "400F2",
+ "EventCode": "0x400F2",
"EventName": "PM_1PLUS_PPC_DISP",
"BriefDescription": "Cycles at least one Instr Dispatched."
},
{
- "EventCode": "400F8",
+ "EventCode": "0x400F8",
"EventName": "PM_FLUSH",
"BriefDescription": "Flush (any type)."
}
[
{
- "EventCode": "301E8",
+ "EventCode": "0x301E8",
"EventName": "PM_THRESH_EXC_64",
"BriefDescription": "Threshold counter exceeded a value of 64."
},
{
- "EventCode": "45050",
+ "EventCode": "0x45050",
"EventName": "PM_1FLOP_CMPL",
"BriefDescription": "One floating point instruction completed (fadd, fmul, fsub, fcmp, fsel, fabs, fnabs, fres, fsqrte, fneg)."
},
{
- "EventCode": "45052",
+ "EventCode": "0x45052",
"EventName": "PM_4FLOP_CMPL",
"BriefDescription": "Four floating point instructions completed (fadd, fmul, fsub, fcmp, fsel, fabs, fnabs, fres, fsqrte, fneg)."
},
{
- "EventCode": "4D054",
+ "EventCode": "0x4D054",
"EventName": "PM_8FLOP_CMPL",
"BriefDescription": "Four Double Precision vector instructions completed."
}
[
{
- "EventCode": "1F15E",
+ "EventCode": "0x1F15E",
"EventName": "PM_MRK_START_PROBE_NOP_CMPL",
"BriefDescription": "Marked Start probe nop (AND R0,R0,R0) completed."
},
{
- "EventCode": "20016",
+ "EventCode": "0x20016",
"EventName": "PM_ST_FIN",
"BriefDescription": "Store finish count. Includes speculative activity."
},
{
- "EventCode": "20018",
+ "EventCode": "0x20018",
"EventName": "PM_ST_FWD",
"BriefDescription": "Store forwards that finished."
},
{
- "EventCode": "2011C",
+ "EventCode": "0x2011C",
"EventName": "PM_MRK_NTF_CYC",
"BriefDescription": "Cycles during which the marked instruction is the oldest in the pipeline (NTF or NTC)."
},
{
- "EventCode": "2E01C",
+ "EventCode": "0x2E01C",
"EventName": "PM_EXEC_STALL_TLBIE",
"BriefDescription": "Cycles in which the oldest instruction in the pipeline was a TLBIE instruction executing in the Load Store Unit."
},
{
- "EventCode": "201E6",
+ "EventCode": "0x201E6",
"EventName": "PM_THRESH_EXC_32",
"BriefDescription": "Threshold counter exceeded a value of 32."
},
{
- "EventCode": "200F0",
+ "EventCode": "0x200F0",
"EventName": "PM_ST_CMPL",
"BriefDescription": "Stores completed from S2Q (2nd-level store queue). This event includes regular stores, stcx and cache inhibited stores. The following operations are excluded (pteupdate, snoop tlbie complete, store atomics, miso, load atomic payloads, tlbie, tlbsync, slbieg, isync, msgsnd, slbiag, cpabort, copy, tcheck, tend, stsync, dcbst, icbi, dcbf, hwsync, lwsync, ptesync, eieio, msgsync)."
},
{
- "EventCode": "200FE",
+ "EventCode": "0x200FE",
"EventName": "PM_DATA_FROM_L2MISS",
"BriefDescription": "The processor's data cache was reloaded from a source other than the local core's L1 or L2 due to a demand miss."
},
{
- "EventCode": "30010",
+ "EventCode": "0x30010",
"EventName": "PM_PMC2_OVERFLOW",
"BriefDescription": "The event selected for PMC2 caused the event counter to overflow."
},
{
- "EventCode": "4D010",
+ "EventCode": "0x4D010",
"EventName": "PM_PMC1_SAVED",
"BriefDescription": "The conditions for the speculative event selected for PMC1 are met and PMC1 is charged."
},
{
- "EventCode": "4D05C",
+ "EventCode": "0x4D05C",
"EventName": "PM_DPP_FLOP_CMPL",
"BriefDescription": "Double-Precision or Quad-Precision instructions completed."
}
struct rlimit rlim;
if (getrlimit(RLIMIT_NOFILE, &rlim) == 0)
- return min((int)rlim.rlim_max / 2, 512);
+ return min(rlim.rlim_max / 2, (rlim_t)512);
return 512;
}
mapfile = strdup(fpath);
return 0;
}
-
- pr_info("%s: Ignoring file %s\n", prog, fpath);
+ if (is_json_file(bname))
+ pr_debug("%s: ArchStd json is preprocessed %s\n", prog, fpath);
+ else
+ pr_info("%s: Ignoring file %s\n", prog, fpath);
return 0;
}
from __future__ import print_function
import sys
+# Only change warnings if the python -W option was not used
+if not sys.warnoptions:
+ import warnings
+ # PySide2 causes deprecation warnings, ignore them.
+ warnings.filterwarnings("ignore", category=DeprecationWarning)
import argparse
import weakref
import threading
from PySide.QtGui import *
from PySide.QtSql import *
-from decimal import *
-from ctypes import *
+from decimal import Decimal, ROUND_HALF_UP
+from ctypes import CDLL, Structure, create_string_buffer, addressof, sizeof, \
+ c_void_p, c_bool, c_byte, c_char, c_int, c_uint, c_longlong, c_ulonglong
from multiprocessing import Process, Array, Value, Event
# xrange is range in Python3
if with_hdr:
model = indexes[0].model()
for col in range(min_col, max_col + 1):
- val = model.headerData(col, Qt.Horizontal)
+ val = model.headerData(col, Qt.Horizontal, Qt.DisplayRole)
if as_csv:
text += sep + ToCSValue(val)
sep = ","
flags=0|8
cpu=*
type=0|1
-size=120
+size=128
config=0
sample_period=*
sample_type=263
exclusive=0
exclude_user=0
exclude_kernel=0|1
-exclude_hv=0
+exclude_hv=0|1
exclude_idle=0
mmap=1
comm=1
flags=0|8
cpu=*
type=0
-size=120
+size=128
config=0
sample_period=0
sample_type=65536
pid=-1
flags=8
type=1
-size=120
+size=128
config=9
sample_period=4000
sample_type=455
},
{
.events = "{},{instructions}",
- .nr_events = 0,
- .nr_groups = 0,
+ .nr_events = 1,
+ .nr_groups = 1,
},
{
.events = "{instructions},{instructions}",
perf-$(CONFIG_LIBELF) += probe-file.o
perf-$(CONFIG_LIBELF) += probe-event.o
+ifdef CONFIG_LIBBPF_DYNAMIC
+ hashmap := 1
+endif
ifndef CONFIG_LIBBPF
+ hashmap := 1
+endif
+
+ifdef hashmap
perf-y += hashmap.o
endif
goto out;
}
- err = -1;
link = bpf_program__attach(skel->progs.on_switch);
- if (!link) {
+ if (IS_ERR(link)) {
pr_err("Failed to attach leader program\n");
+ err = PTR_ERR(link);
goto out;
}
evsel->bperf_leader_link_fd = bpf_link_get_fd_by_id(entry.link_id);
if (evsel->bperf_leader_link_fd < 0 &&
- bperf_reload_leader_program(evsel, attr_map_fd, &entry))
+ bperf_reload_leader_program(evsel, attr_map_fd, &entry)) {
+ err = -1;
goto out;
-
+ }
/*
* The bpf_link holds reference to the leader program, and the
* leader program holds reference to the maps. Therefore, if
/* Step 2: load the follower skeleton */
evsel->follower_skel = bperf_follower_bpf__open();
if (!evsel->follower_skel) {
+ err = -1;
pr_err("Failed to open follower skeleton\n");
goto out;
}
if ((tag == DW_TAG_formal_parameter ||
tag == DW_TAG_variable) &&
die_compare_name(die_mem, fvp->name) &&
- /* Does the DIE have location information or external instance? */
+ /*
+ * Does the DIE have location information or const value
+ * or external instance?
+ */
(dwarf_attr(die_mem, DW_AT_external, &attr) ||
- dwarf_attr(die_mem, DW_AT_location, &attr)))
+ dwarf_attr(die_mem, DW_AT_location, &attr) ||
+ dwarf_attr(die_mem, DW_AT_const_value, &attr)))
return DIE_FIND_CB_END;
if (dwarf_haspc(die_mem, fvp->addr))
return DIE_FIND_CB_CONTINUE;
node = rb_entry(next, struct bpf_prog_info_node, rb_node);
next = rb_next(&node->rb_node);
rb_erase(&node->rb_node, root);
+ free(node->info_linear);
free(node);
}
PERF_IP_FLAG_VMEXIT = 1ULL << 12,
};
-#define PERF_IP_FLAG_CHARS "bcrosyiABEx"
+#define PERF_IP_FLAG_CHARS "bcrosyiABExgh"
#define PERF_BRANCH_MASK (\
PERF_IP_FLAG_BRANCH |\
if (affinity__setup(&affinity) < 0)
return;
- evlist__for_each_entry(evlist, pos)
- bpf_counter__disable(pos);
-
/* Disable 'immediate' events last */
for (imm = 0; imm <= 1; imm++) {
evlist__for_each_cpu(evlist, i, cpu) {
evsel->auto_merge_stats = orig->auto_merge_stats;
evsel->collect_stat = orig->collect_stat;
evsel->weak_group = orig->weak_group;
+ evsel->use_config_name = orig->use_config_name;
if (evsel__copy_config_terms(evsel, orig) < 0)
goto out_err;
bool collect_stat;
bool weak_group;
bool bpf_counter;
+ bool use_config_name;
int bpf_fd;
struct bpf_object *bpf_obj;
+ struct list_head config_terms;
};
/*
bool merged_stat;
bool reset_group;
bool errored;
- bool use_config_name;
struct hashmap *per_pkg_mask;
struct evsel *leader;
- struct list_head config_terms;
int err;
int cpu_iter;
struct {
decoder->set_fup_tx_flags = false;
decoder->tx_flags = decoder->fup_tx_flags;
decoder->state.type = INTEL_PT_TRANSACTION;
+ if (decoder->fup_tx_flags & INTEL_PT_ABORT_TX)
+ decoder->state.type |= INTEL_PT_BRANCH;
decoder->state.from_ip = decoder->ip;
decoder->state.to_ip = 0;
decoder->state.flags = decoder->fup_tx_flags;
return 0;
if (err == -EAGAIN ||
intel_pt_fup_with_nlip(decoder, &intel_pt_insn, ip, err)) {
+ bool no_tip = decoder->pkt_state != INTEL_PT_STATE_FUP;
+
decoder->pkt_state = INTEL_PT_STATE_IN_SYNC;
- if (intel_pt_fup_event(decoder))
+ if (intel_pt_fup_event(decoder) && no_tip)
return 0;
return -EAGAIN;
}
*ip += intel_pt_insn->length;
- if (to_ip && *ip == to_ip)
+ if (to_ip && *ip == to_ip) {
+ intel_pt_insn->length = 0;
goto out_no_cache;
+ }
if (*ip >= al.map->end)
break;
static void intel_pt_sample_flags(struct intel_pt_queue *ptq)
{
+ ptq->insn_len = 0;
if (ptq->state->flags & INTEL_PT_ABORT_TX) {
ptq->flags = PERF_IP_FLAG_BRANCH | PERF_IP_FLAG_TX_ABORT;
} else if (ptq->state->flags & INTEL_PT_ASYNC) {
ptq->flags = PERF_IP_FLAG_BRANCH | PERF_IP_FLAG_CALL |
PERF_IP_FLAG_ASYNC |
PERF_IP_FLAG_INTERRUPT;
- ptq->insn_len = 0;
} else {
if (ptq->state->from_ip)
ptq->flags = intel_pt_insn_type(ptq->state->insn_op);
.symbol = "bpf-output",
.alias = "",
},
+ [PERF_COUNT_SW_CGROUP_SWITCHES] = {
+ .symbol = "cgroup-switches",
+ .alias = "",
+ },
};
#define __PERF_EVENT_FIELD(config, name) \
}
for (i = 0; i < max; i++, syms++) {
+ /*
+ * New attr.config still not supported here, the latest
+ * example was PERF_COUNT_SW_CGROUP_SWITCHES
+ */
+ if (syms->symbol == NULL)
+ continue;
- if (event_glob != NULL && syms->symbol != NULL &&
- !(strglobmatch(syms->symbol, event_glob) ||
+ if (event_glob != NULL && !(strglobmatch(syms->symbol, event_glob) ||
(syms->alias && strglobmatch(syms->alias, event_glob))))
continue;
dummy { return sym(yyscanner, PERF_TYPE_SOFTWARE, PERF_COUNT_SW_DUMMY); }
duration_time { return tool(yyscanner, PERF_TOOL_DURATION_TIME); }
bpf-output { return sym(yyscanner, PERF_TYPE_SOFTWARE, PERF_COUNT_SW_BPF_OUTPUT); }
+cgroup-switches { return sym(yyscanner, PERF_TYPE_SOFTWARE, PERF_COUNT_SW_CGROUP_SWITCHES); }
/*
* We have to handle the kernel PMU event cycles-ct/cycles-t/mem-loads/mem-stores separately.
evsel->core.attr.build_id = 1;
}
+static void perf_probe_cgroup(struct evsel *evsel)
+{
+ evsel->core.attr.cgroup = 1;
+}
+
bool perf_can_sample_identifier(void)
{
return perf_probe_api(perf_probe_sample_identifier);
{
return perf_probe_api(perf_probe_build_id);
}
+
+bool perf_can_record_cgroup(void)
+{
+ return perf_probe_api(perf_probe_cgroup);
+}
bool perf_can_record_text_poke_events(void);
bool perf_can_sample_identifier(void);
bool perf_can_record_build_id(void);
+bool perf_can_record_cgroup(void);
#endif // __PERF_API_PROBE_H
}
/* no event */
- if (*q == '\0')
+ if (*q == '\0') {
+ if (*sep == '}') {
+ if (grp_evt < 0) {
+ ui__error("cannot close a non-existing event group\n");
+ goto error;
+ }
+ grp_evt--;
+ }
continue;
+ }
memset(&attr, 0, sizeof(attr));
event_attr_init(&attr);
grp_evt = -1;
}
}
+ free(p_orig);
return 0;
error:
free(p_orig);
immediate_value_is_supported()) {
Dwarf_Sword snum;
+ if (!tvar)
+ return 0;
+
dwarf_formsdata(&attr, &snum);
ret = asprintf(&tvar->value, "\\%ld", (long)snum);
static int record_opts__config_freq(struct record_opts *opts)
{
bool user_freq = opts->user_freq != UINT_MAX;
+ bool user_interval = opts->user_interval != ULLONG_MAX;
unsigned int max_rate;
- if (opts->user_interval != ULLONG_MAX)
+ if (user_interval && user_freq) {
+ pr_err("cannot set frequency and period at the same time\n");
+ return -1;
+ }
+
+ if (user_interval)
opts->default_interval = opts->user_interval;
if (user_freq)
opts->freq = opts->user_freq;
struct perf_record_record_cpu_map *mask;
unsigned i;
- data->type = bswap_64(data->type);
+ data->type = bswap_16(data->type);
switch (data->type) {
case PERF_CPU_MAP__CPUS:
{
u64 size;
- size = event->stat_config.nr * sizeof(event->stat_config.data[0]);
+ size = bswap_64(event->stat_config.nr) * sizeof(event->stat_config.data[0]);
size += 1; /* nr item itself */
mem_bswap_64(&event->stat_config.nr, size);
}
if (event->header.size < hdr_sz || event->header.size > buf_sz)
return -1;
+ buf += hdr_sz;
rest = event->header.size - hdr_sz;
if (readn(fd, buf, rest) != (ssize_t)rest)
char *config;
int ret = 0;
- if (counter->uniquified_name ||
+ if (counter->uniquified_name || counter->use_config_name ||
!counter->pmu_name || !strncmp(counter->name, counter->pmu_name,
strlen(counter->pmu_name)))
return;
}
} else {
if (perf_pmu__has_hybrid()) {
- if (!counter->use_config_name) {
- ret = asprintf(&new_name, "%s/%s/",
- counter->pmu_name, counter->name);
- }
+ ret = asprintf(&new_name, "%s/%s/",
+ counter->pmu_name, counter->name);
} else {
ret = asprintf(&new_name, "%s [%s]",
counter->name, counter->pmu_name);
list_for_each_entry_safe(pos, tmp, sdt_notes, note_list) {
list_del_init(&pos->note_list);
+ zfree(&pos->args);
zfree(&pos->name);
zfree(&pos->provider);
free(pos);
ifneq ($(silent),1)
ifneq ($(V),1)
- QUIET_CC = @echo ' CC '$@;
- QUIET_CC_FPIC = @echo ' CC FPIC '$@;
- QUIET_CLANG = @echo ' CLANG '$@;
- QUIET_AR = @echo ' AR '$@;
- QUIET_LINK = @echo ' LINK '$@;
- QUIET_MKDIR = @echo ' MKDIR '$@;
- QUIET_GEN = @echo ' GEN '$@;
+ QUIET_CC = @echo ' CC '$@;
+ QUIET_CC_FPIC = @echo ' CC FPIC '$@;
+ QUIET_CLANG = @echo ' CLANG '$@;
+ QUIET_AR = @echo ' AR '$@;
+ QUIET_LINK = @echo ' LINK '$@;
+ QUIET_MKDIR = @echo ' MKDIR '$@;
+ QUIET_GEN = @echo ' GEN '$@;
QUIET_SUBDIR0 = +@subdir=
QUIET_SUBDIR1 = ;$(NO_SUBDIR) \
- echo ' SUBDIR '$$subdir; \
+ echo ' SUBDIR '$$subdir; \
$(MAKE) $(PRINT_DIR) -C $$subdir
- QUIET_FLEX = @echo ' FLEX '$@;
- QUIET_BISON = @echo ' BISON '$@;
- QUIET_GENSKEL = @echo ' GEN-SKEL '$@;
+ QUIET_FLEX = @echo ' FLEX '$@;
+ QUIET_BISON = @echo ' BISON '$@;
+ QUIET_GENSKEL = @echo ' GENSKEL '$@;
descend = \
- +@echo ' DESCEND '$(1); \
+ +@echo ' DESCEND '$(1); \
mkdir -p $(OUTPUT)$(1) && \
$(MAKE) $(COMMAND_O) subdir=$(if $(subdir),$(subdir)/$(1),$(1)) $(PRINT_DIR) -C $(1) $(2)
- QUIET_CLEAN = @printf ' CLEAN %s\n' $1;
- QUIET_INSTALL = @printf ' INSTALL %s\n' $1;
- QUIET_UNINST = @printf ' UNINST %s\n' $1;
+ QUIET_CLEAN = @printf ' CLEAN %s\n' $1;
+ QUIET_INSTALL = @printf ' INSTALL %s\n' $1;
+ QUIET_UNINST = @printf ' UNINST %s\n' $1;
endif
endif
}
EXPORT_SYMBOL(get_nfit_res);
-void __iomem *__nfit_test_ioremap(resource_size_t offset, unsigned long size,
+static void __iomem *__nfit_test_ioremap(resource_size_t offset, unsigned long size,
void __iomem *(*fallback_fn)(resource_size_t, unsigned long))
{
struct nfit_test_resource *nfit_res = get_nfit_res(offset);
}
}
+static size_t sizeof_spa(struct acpi_nfit_system_address *spa)
+{
+ /* until spa location cookie support is added... */
+ return sizeof(*spa) - 8;
+}
+
static int nfit_test0_alloc(struct nfit_test *t)
{
- size_t nfit_size = sizeof(struct acpi_nfit_system_address) * NUM_SPA
+ struct acpi_nfit_system_address *spa = NULL;
+ size_t nfit_size = sizeof_spa(spa) * NUM_SPA
+ sizeof(struct acpi_nfit_memory_map) * NUM_MEM
+ sizeof(struct acpi_nfit_control_region) * NUM_DCR
+ offsetof(struct acpi_nfit_control_region,
static int nfit_test1_alloc(struct nfit_test *t)
{
- size_t nfit_size = sizeof(struct acpi_nfit_system_address) * 2
+ struct acpi_nfit_system_address *spa = NULL;
+ size_t nfit_size = sizeof_spa(spa) * 2
+ sizeof(struct acpi_nfit_memory_map) * 2
+ offsetof(struct acpi_nfit_control_region, window_size) * 2;
int i;
*/
spa = nfit_buf;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_PM), 16);
spa->range_index = 0+1;
spa->address = t->spa_set_dma[0];
*/
spa = nfit_buf + offset;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_PM), 16);
spa->range_index = 1+1;
spa->address = t->spa_set_dma[1];
/* spa2 (dcr0) dimm0 */
spa = nfit_buf + offset;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_DCR), 16);
spa->range_index = 2+1;
spa->address = t->dcr_dma[0];
/* spa3 (dcr1) dimm1 */
spa = nfit_buf + offset;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_DCR), 16);
spa->range_index = 3+1;
spa->address = t->dcr_dma[1];
/* spa4 (dcr2) dimm2 */
spa = nfit_buf + offset;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_DCR), 16);
spa->range_index = 4+1;
spa->address = t->dcr_dma[2];
/* spa5 (dcr3) dimm3 */
spa = nfit_buf + offset;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_DCR), 16);
spa->range_index = 5+1;
spa->address = t->dcr_dma[3];
/* spa6 (bdw for dcr0) dimm0 */
spa = nfit_buf + offset;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_BDW), 16);
spa->range_index = 6+1;
spa->address = t->dimm_dma[0];
/* spa7 (bdw for dcr1) dimm1 */
spa = nfit_buf + offset;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_BDW), 16);
spa->range_index = 7+1;
spa->address = t->dimm_dma[1];
/* spa8 (bdw for dcr2) dimm2 */
spa = nfit_buf + offset;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_BDW), 16);
spa->range_index = 8+1;
spa->address = t->dimm_dma[2];
/* spa9 (bdw for dcr3) dimm3 */
spa = nfit_buf + offset;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_BDW), 16);
spa->range_index = 9+1;
spa->address = t->dimm_dma[3];
/* spa10 (dcr4) dimm4 */
spa = nfit_buf + offset;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_DCR), 16);
spa->range_index = 10+1;
spa->address = t->dcr_dma[4];
*/
spa = nfit_buf + offset;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_PM), 16);
spa->range_index = 11+1;
spa->address = t->spa_set_dma[2];
/* spa12 (bdw for dcr4) dimm4 */
spa = nfit_buf + offset;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_BDW), 16);
spa->range_index = 12+1;
spa->address = t->dimm_dma[4];
/* spa0 (flat range with no bdw aliasing) */
spa = nfit_buf + offset;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_PM), 16);
spa->range_index = 0+1;
spa->address = t->spa_set_dma[0];
/* virtual cd region */
spa = nfit_buf + offset;
spa->header.type = ACPI_NFIT_TYPE_SYSTEM_ADDRESS;
- spa->header.length = sizeof(*spa);
+ spa->header.length = sizeof_spa(spa);
memcpy(spa->range_guid, to_nfit_uuid(NFIT_SPA_VCD), 16);
spa->range_index = 0;
spa->address = t->spa_set_dma[1];
#include "system.h"
+#include <stddef.h>
#include <linux/errno.h>
#include <linux/auxvec.h>
#include <linux/signal.h>
fixdep
test_dev_cgroup
/test_progs*
+!test_progs.h
test_verifier_log
feature
test_sock
xdping
test_cpp
*.skel.h
+*.lskel.h
/no_alu32
/bpf_gcc
/tools
/runqslower
/bench
*.ko
+*.tmp
xdpxceiver
+xdp_redirect_multi
# Order correspond to 'make run_tests' order
TEST_PROGS := test_kmod.sh \
test_xdp_redirect.sh \
+ test_xdp_redirect_multi.sh \
test_xdp_meta.sh \
test_xdp_veth.sh \
test_offload.py \
TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \
flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \
test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \
- xdpxceiver
+ xdpxceiver xdp_redirect_multi
TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read
LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \
linked_vars.skel.h linked_maps.skel.h
+LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \
+ test_ksyms_module.c test_ringbuf.c atomics.c trace_printk.c
+SKEL_BLACKLIST += $$(LSKELS)
+
test_static_linked.skel.h-deps := test_static_linked1.o test_static_linked2.o
linked_funcs.skel.h-deps := linked_funcs1.o linked_funcs2.o
linked_vars.skel.h-deps := linked_vars1.o linked_vars2.o
TRUNNER_BPF_SKELS := $$(patsubst %.c,$$(TRUNNER_OUTPUT)/%.skel.h, \
$$(filter-out $(SKEL_BLACKLIST) $(LINKED_BPF_SRCS),\
$$(TRUNNER_BPF_SRCS)))
+TRUNNER_BPF_LSKELS := $$(patsubst %.c,$$(TRUNNER_OUTPUT)/%.lskel.h, $$(LSKELS))
TRUNNER_BPF_SKELS_LINKED := $$(addprefix $$(TRUNNER_OUTPUT)/,$(LINKED_SKELS))
TEST_GEN_FILES += $$(TRUNNER_BPF_OBJS)
$(Q)diff $$(<:.o=.linked2.o) $$(<:.o=.linked3.o)
$(Q)$$(BPFTOOL) gen skeleton $$(<:.o=.linked3.o) name $$(notdir $$(<:.o=)) > $$@
+$(TRUNNER_BPF_LSKELS): %.lskel.h: %.o $(BPFTOOL) | $(TRUNNER_OUTPUT)
+ $$(call msg,GEN-SKEL,$(TRUNNER_BINARY),$$@)
+ $(Q)$$(BPFTOOL) gen object $$(<:.o=.linked1.o) $$<
+ $(Q)$$(BPFTOOL) gen object $$(<:.o=.linked2.o) $$(<:.o=.linked1.o)
+ $(Q)$$(BPFTOOL) gen object $$(<:.o=.linked3.o) $$(<:.o=.linked2.o)
+ $(Q)diff $$(<:.o=.linked2.o) $$(<:.o=.linked3.o)
+ $(Q)$$(BPFTOOL) gen skeleton -L $$(<:.o=.linked3.o) name $$(notdir $$(<:.o=)) > $$@
+
$(TRUNNER_BPF_SKELS_LINKED): $(TRUNNER_BPF_OBJS) $(BPFTOOL) | $(TRUNNER_OUTPUT)
$$(call msg,LINK-BPF,$(TRUNNER_BINARY),$$(@:.skel.h=.o))
$(Q)$$(BPFTOOL) gen object $$(@:.skel.h=.linked1.o) $$(addprefix $(TRUNNER_OUTPUT)/,$$($$(@F)-deps))
$(TRUNNER_EXTRA_HDRS) \
$(TRUNNER_BPF_OBJS) \
$(TRUNNER_BPF_SKELS) \
+ $(TRUNNER_BPF_LSKELS) \
$(TRUNNER_BPF_SKELS_LINKED) \
$$(BPFOBJ) | $(TRUNNER_OUTPUT)
$$(call msg,TEST-OBJ,$(TRUNNER_BINARY),$$@)
EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) $(HOST_SCRATCH_DIR) \
prog_tests/tests.h map_tests/tests.h verifier/tests.h \
feature \
- $(addprefix $(OUTPUT)/,*.o *.skel.h no_alu32 bpf_gcc bpf_testmod.ko)
+ $(addprefix $(OUTPUT)/,*.o *.skel.h *.lskel.h no_alu32 bpf_gcc bpf_testmod.ko)
.PHONY: docs docs-clean
ifndef RST2MAN_DEP
$$(error "rst2man not found, but required to generate man pages")
endif
- $$(QUIET_GEN)rst2man $$< > $$@
+ $$(QUIET_GEN)rst2man --exit-status=1 $$< > $$@.tmp
+ $$(QUIET_GEN)mv $$@.tmp $$@
docs-clean-$1:
$$(call QUIET_CLEAN, eBPF_$1-manpage)
Clang that contains the fix.
__ https://reviews.llvm.org/D100362
+
+Clang relocation changes
+========================
+
+Clang 13 patch `clang reloc patch`_ made some changes on relocations such
+that existing relocation types are broken into more types and
+each new type corresponds to only one way to resolve relocation.
+See `kernel llvm reloc`_ for more explanation and some examples.
+Using clang 13 to compile old libbpf which has static linker support,
+there will be a compilation failure::
+
+ libbpf: ELF relo #0 in section #6 has unexpected type 2 in .../bpf_tcp_nogpl.o
+
+Here, ``type 2`` refers to new relocation type ``R_BPF_64_ABS64``.
+To fix this issue, user newer libbpf.
+
+.. Links
+.. _clang reloc patch: https://reviews.llvm.org/D102712
+.. _kernel llvm reloc: /Documentation/bpf/llvm_reloc.rst
{
int err;
+ libbpf_set_strict_mode(LIBBPF_STRICT_ALL);
libbpf_set_print(libbpf_print_fn);
err = bump_memlock_rlimit();
struct bpf_link *link;
link = bpf_program__attach(prog);
- if (IS_ERR(link)) {
+ if (!link) {
fprintf(stderr, "failed to attach program!\n");
exit(1);
}
}
link = bpf_program__attach(ctx->skel->progs.bench_ringbuf);
- if (IS_ERR(link)) {
+ if (!link) {
fprintf(stderr, "failed to attach program!\n");
exit(1);
}
}
link = bpf_program__attach(ctx->skel->progs.bench_ringbuf);
- if (IS_ERR(link)) {
+ if (!link) {
fprintf(stderr, "failed to attach program\n");
exit(1);
}
}
link = bpf_program__attach(ctx->skel->progs.bench_perfbuf);
- if (IS_ERR(link)) {
+ if (!link) {
fprintf(stderr, "failed to attach program\n");
exit(1);
}
struct bpf_link *link;
link = bpf_program__attach(prog);
- if (IS_ERR(link)) {
+ if (!link) {
fprintf(stderr, "failed to attach program!\n");
exit(1);
}
.tcp.doff = 5,
};
-static int settimeo(int fd, int timeout_ms)
+int settimeo(int fd, int timeout_ms)
{
struct timeval timeout = { .tv_sec = 3 };
} __packed;
extern struct ipv6_packet pkt_v6;
+int settimeo(int fd, int timeout_ms);
int start_server(int family, int type, const char *addr, __u16 port,
int timeout_ms);
int connect_to_fd(int server_fd, int timeout_ms);
#include <test_progs.h>
-#include "atomics.skel.h"
+#include "atomics.lskel.h"
static void test_add(struct atomics *skel)
{
int err, prog_fd;
__u32 duration = 0, retval;
- struct bpf_link *link;
+ int link_fd;
- link = bpf_program__attach(skel->progs.add);
- if (CHECK(IS_ERR(link), "attach(add)", "err: %ld\n", PTR_ERR(link)))
+ link_fd = atomics__add__attach(skel);
+ if (!ASSERT_GT(link_fd, 0, "attach(add)"))
return;
- prog_fd = bpf_program__fd(skel->progs.add);
+ prog_fd = skel->progs.add.prog_fd;
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
NULL, NULL, &retval, &duration);
if (CHECK(err || retval, "test_run add",
ASSERT_EQ(skel->data->add_noreturn_value, 3, "add_noreturn_value");
cleanup:
- bpf_link__destroy(link);
+ close(link_fd);
}
static void test_sub(struct atomics *skel)
{
int err, prog_fd;
__u32 duration = 0, retval;
- struct bpf_link *link;
+ int link_fd;
- link = bpf_program__attach(skel->progs.sub);
- if (CHECK(IS_ERR(link), "attach(sub)", "err: %ld\n", PTR_ERR(link)))
+ link_fd = atomics__sub__attach(skel);
+ if (!ASSERT_GT(link_fd, 0, "attach(sub)"))
return;
- prog_fd = bpf_program__fd(skel->progs.sub);
+ prog_fd = skel->progs.sub.prog_fd;
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
NULL, NULL, &retval, &duration);
if (CHECK(err || retval, "test_run sub",
ASSERT_EQ(skel->data->sub_noreturn_value, -1, "sub_noreturn_value");
cleanup:
- bpf_link__destroy(link);
+ close(link_fd);
}
static void test_and(struct atomics *skel)
{
int err, prog_fd;
__u32 duration = 0, retval;
- struct bpf_link *link;
+ int link_fd;
- link = bpf_program__attach(skel->progs.and);
- if (CHECK(IS_ERR(link), "attach(and)", "err: %ld\n", PTR_ERR(link)))
+ link_fd = atomics__and__attach(skel);
+ if (!ASSERT_GT(link_fd, 0, "attach(and)"))
return;
- prog_fd = bpf_program__fd(skel->progs.and);
+ prog_fd = skel->progs.and.prog_fd;
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
NULL, NULL, &retval, &duration);
if (CHECK(err || retval, "test_run and",
ASSERT_EQ(skel->data->and_noreturn_value, 0x010ull << 32, "and_noreturn_value");
cleanup:
- bpf_link__destroy(link);
+ close(link_fd);
}
static void test_or(struct atomics *skel)
{
int err, prog_fd;
__u32 duration = 0, retval;
- struct bpf_link *link;
+ int link_fd;
- link = bpf_program__attach(skel->progs.or);
- if (CHECK(IS_ERR(link), "attach(or)", "err: %ld\n", PTR_ERR(link)))
+ link_fd = atomics__or__attach(skel);
+ if (!ASSERT_GT(link_fd, 0, "attach(or)"))
return;
- prog_fd = bpf_program__fd(skel->progs.or);
+ prog_fd = skel->progs.or.prog_fd;
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
NULL, NULL, &retval, &duration);
if (CHECK(err || retval, "test_run or",
ASSERT_EQ(skel->data->or_noreturn_value, 0x111ull << 32, "or_noreturn_value");
cleanup:
- bpf_link__destroy(link);
+ close(link_fd);
}
static void test_xor(struct atomics *skel)
{
int err, prog_fd;
__u32 duration = 0, retval;
- struct bpf_link *link;
+ int link_fd;
- link = bpf_program__attach(skel->progs.xor);
- if (CHECK(IS_ERR(link), "attach(xor)", "err: %ld\n", PTR_ERR(link)))
+ link_fd = atomics__xor__attach(skel);
+ if (!ASSERT_GT(link_fd, 0, "attach(xor)"))
return;
- prog_fd = bpf_program__fd(skel->progs.xor);
+ prog_fd = skel->progs.xor.prog_fd;
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
NULL, NULL, &retval, &duration);
if (CHECK(err || retval, "test_run xor",
ASSERT_EQ(skel->data->xor_noreturn_value, 0x101ull << 32, "xor_nxoreturn_value");
cleanup:
- bpf_link__destroy(link);
+ close(link_fd);
}
static void test_cmpxchg(struct atomics *skel)
{
int err, prog_fd;
__u32 duration = 0, retval;
- struct bpf_link *link;
+ int link_fd;
- link = bpf_program__attach(skel->progs.cmpxchg);
- if (CHECK(IS_ERR(link), "attach(cmpxchg)", "err: %ld\n", PTR_ERR(link)))
+ link_fd = atomics__cmpxchg__attach(skel);
+ if (!ASSERT_GT(link_fd, 0, "attach(cmpxchg)"))
return;
- prog_fd = bpf_program__fd(skel->progs.cmpxchg);
+ prog_fd = skel->progs.cmpxchg.prog_fd;
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
NULL, NULL, &retval, &duration);
if (CHECK(err || retval, "test_run add",
ASSERT_EQ(skel->bss->cmpxchg32_result_succeed, 1, "cmpxchg_result_succeed");
cleanup:
- bpf_link__destroy(link);
+ close(link_fd);
}
static void test_xchg(struct atomics *skel)
{
int err, prog_fd;
__u32 duration = 0, retval;
- struct bpf_link *link;
+ int link_fd;
- link = bpf_program__attach(skel->progs.xchg);
- if (CHECK(IS_ERR(link), "attach(xchg)", "err: %ld\n", PTR_ERR(link)))
+ link_fd = atomics__xchg__attach(skel);
+ if (!ASSERT_GT(link_fd, 0, "attach(xchg)"))
return;
- prog_fd = bpf_program__fd(skel->progs.xchg);
+ prog_fd = skel->progs.xchg.prog_fd;
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
NULL, NULL, &retval, &duration);
if (CHECK(err || retval, "test_run add",
ASSERT_EQ(skel->bss->xchg32_result, 1, "xchg32_result");
cleanup:
- bpf_link__destroy(link);
+ close(link_fd);
}
void test_atomics(void)
kprobe_link = bpf_program__attach_kprobe(skel->progs.handle_kprobe,
false /* retprobe */,
SYS_NANOSLEEP_KPROBE_NAME);
- if (CHECK(IS_ERR(kprobe_link), "attach_kprobe",
- "err %ld\n", PTR_ERR(kprobe_link)))
+ if (!ASSERT_OK_PTR(kprobe_link, "attach_kprobe"))
goto cleanup;
skel->links.handle_kprobe = kprobe_link;
kretprobe_link = bpf_program__attach_kprobe(skel->progs.handle_kretprobe,
true /* retprobe */,
SYS_NANOSLEEP_KPROBE_NAME);
- if (CHECK(IS_ERR(kretprobe_link), "attach_kretprobe",
- "err %ld\n", PTR_ERR(kretprobe_link)))
+ if (!ASSERT_OK_PTR(kretprobe_link, "attach_kretprobe"))
goto cleanup;
skel->links.handle_kretprobe = kretprobe_link;
0 /* self pid */,
"/proc/self/exe",
uprobe_offset);
- if (CHECK(IS_ERR(uprobe_link), "attach_uprobe",
- "err %ld\n", PTR_ERR(uprobe_link)))
+ if (!ASSERT_OK_PTR(uprobe_link, "attach_uprobe"))
goto cleanup;
skel->links.handle_uprobe = uprobe_link;
-1 /* any pid */,
"/proc/self/exe",
uprobe_offset);
- if (CHECK(IS_ERR(uretprobe_link), "attach_uretprobe",
- "err %ld\n", PTR_ERR(uretprobe_link)))
+ if (!ASSERT_OK_PTR(uretprobe_link, "attach_uretprobe"))
goto cleanup;
skel->links.handle_uretprobe = uretprobe_link;
int iter_fd, len;
link = bpf_program__attach_iter(prog, NULL);
- if (CHECK(IS_ERR(link), "attach_iter", "attach_iter failed\n"))
+ if (!ASSERT_OK_PTR(link, "attach_iter"))
return;
iter_fd = bpf_iter_create(bpf_link__fd(link));
int ret = 0;
link = bpf_program__attach_iter(prog, NULL);
- if (CHECK(IS_ERR(link), "attach_iter", "attach_iter failed\n"))
+ if (!ASSERT_OK_PTR(link, "attach_iter"))
return ret;
iter_fd = bpf_iter_create(bpf_link__fd(link));
return;
link = bpf_program__attach_iter(skel1->progs.dump_task, NULL);
- if (CHECK(IS_ERR(link), "attach_iter", "attach_iter failed\n"))
+ if (!ASSERT_OK_PTR(link, "attach_iter"))
goto out;
/* unlink this path if it exists. */
skel->bss->map2_id = map_info.id;
link = bpf_program__attach_iter(skel->progs.dump_bpf_map, NULL);
- if (CHECK(IS_ERR(link), "attach_iter", "attach_iter failed\n"))
+ if (!ASSERT_OK_PTR(link, "attach_iter"))
goto free_map2;
iter_fd = bpf_iter_create(bpf_link__fd(link));
opts.link_info = &linfo;
opts.link_info_len = sizeof(linfo);
link = bpf_program__attach_iter(skel->progs.dump_bpf_hash_map, &opts);
- if (CHECK(!IS_ERR(link), "attach_iter",
- "attach_iter for hashmap2 unexpected succeeded\n"))
+ if (!ASSERT_ERR_PTR(link, "attach_iter"))
goto out;
linfo.map.map_fd = bpf_map__fd(skel->maps.hashmap3);
link = bpf_program__attach_iter(skel->progs.dump_bpf_hash_map, &opts);
- if (CHECK(!IS_ERR(link), "attach_iter",
- "attach_iter for hashmap3 unexpected succeeded\n"))
+ if (!ASSERT_ERR_PTR(link, "attach_iter"))
goto out;
/* hashmap1 should be good, update map values here */
linfo.map.map_fd = map_fd;
link = bpf_program__attach_iter(skel->progs.dump_bpf_hash_map, &opts);
- if (CHECK(IS_ERR(link), "attach_iter", "attach_iter failed\n"))
+ if (!ASSERT_OK_PTR(link, "attach_iter"))
goto out;
iter_fd = bpf_iter_create(bpf_link__fd(link));
opts.link_info = &linfo;
opts.link_info_len = sizeof(linfo);
link = bpf_program__attach_iter(skel->progs.dump_bpf_percpu_hash_map, &opts);
- if (CHECK(IS_ERR(link), "attach_iter", "attach_iter failed\n"))
+ if (!ASSERT_OK_PTR(link, "attach_iter"))
goto out;
iter_fd = bpf_iter_create(bpf_link__fd(link));
opts.link_info = &linfo;
opts.link_info_len = sizeof(linfo);
link = bpf_program__attach_iter(skel->progs.dump_bpf_array_map, &opts);
- if (CHECK(IS_ERR(link), "attach_iter", "attach_iter failed\n"))
+ if (!ASSERT_OK_PTR(link, "attach_iter"))
goto out;
iter_fd = bpf_iter_create(bpf_link__fd(link));
opts.link_info = &linfo;
opts.link_info_len = sizeof(linfo);
link = bpf_program__attach_iter(skel->progs.dump_bpf_percpu_array_map, &opts);
- if (CHECK(IS_ERR(link), "attach_iter", "attach_iter failed\n"))
+ if (!ASSERT_OK_PTR(link, "attach_iter"))
goto out;
iter_fd = bpf_iter_create(bpf_link__fd(link));
opts.link_info_len = sizeof(linfo);
link = bpf_program__attach_iter(skel->progs.delete_bpf_sk_storage_map,
&opts);
- if (CHECK(IS_ERR(link), "attach_iter", "attach_iter failed\n"))
+ if (!ASSERT_OK_PTR(link, "attach_iter"))
goto out;
iter_fd = bpf_iter_create(bpf_link__fd(link));
opts.link_info = &linfo;
opts.link_info_len = sizeof(linfo);
link = bpf_program__attach_iter(skel->progs.dump_bpf_sk_storage_map, &opts);
- if (CHECK(IS_ERR(link), "attach_iter", "attach_iter failed\n"))
+ if (!ASSERT_OK_PTR(link, "attach_iter"))
goto out;
iter_fd = bpf_iter_create(bpf_link__fd(link));
opts.link_info = &linfo;
opts.link_info_len = sizeof(linfo);
link = bpf_program__attach_iter(skel->progs.dump_bpf_hash_map, &opts);
- if (CHECK(!IS_ERR(link), "attach_iter", "unexpected success\n"))
+ if (!ASSERT_ERR_PTR(link, "attach_iter"))
bpf_link__destroy(link);
bpf_iter_test_kern5__destroy(skel);
skel->links.proc_maps = bpf_program__attach_iter(
skel->progs.proc_maps, NULL);
- if (CHECK(IS_ERR(skel->links.proc_maps), "bpf_program__attach_iter",
- "attach iterator failed\n")) {
+ if (!ASSERT_OK_PTR(skel->links.proc_maps, "bpf_program__attach_iter")) {
skel->links.proc_maps = NULL;
goto out;
}
bytes, total_bytes, nr_sent, errno);
done:
- if (fd != -1)
+ if (fd >= 0)
close(fd);
if (err) {
WRITE_ONCE(stop, 1);
return;
link = bpf_map__attach_struct_ops(cubic_skel->maps.cubic);
- if (CHECK(IS_ERR(link), "bpf_map__attach_struct_ops", "err:%ld\n",
- PTR_ERR(link))) {
+ if (!ASSERT_OK_PTR(link, "bpf_map__attach_struct_ops")) {
bpf_cubic__destroy(cubic_skel);
return;
}
return;
link = bpf_map__attach_struct_ops(dctcp_skel->maps.dctcp);
- if (CHECK(IS_ERR(link), "bpf_map__attach_struct_ops", "err:%ld\n",
- PTR_ERR(link))) {
+ if (!ASSERT_OK_PTR(link, "bpf_map__attach_struct_ops")) {
bpf_dctcp__destroy(dctcp_skel);
return;
}
always_log);
free(raw_btf);
- err = ((btf_fd == -1) != test->btf_load_err);
+ err = ((btf_fd < 0) != test->btf_load_err);
if (CHECK(err, "btf_fd:%d test->btf_load_err:%u",
btf_fd, test->btf_load_err) ||
CHECK(test->err_str && !strstr(btf_log_buf, test->err_str),
goto done;
}
- if (err || btf_fd == -1)
+ if (err || btf_fd < 0)
goto done;
create_attr.name = test->map_name;
map_fd = bpf_create_map_xattr(&create_attr);
- err = ((map_fd == -1) != test->map_create_err);
+ err = ((map_fd < 0) != test->map_create_err);
CHECK(err, "map_fd:%d test->map_create_err:%u",
map_fd, test->map_create_err);
done:
if (*btf_log_buf && (err || always_log))
fprintf(stderr, "\n%s", btf_log_buf);
- if (btf_fd != -1)
+ if (btf_fd >= 0)
close(btf_fd);
- if (map_fd != -1)
+ if (map_fd >= 0)
close(map_fd);
}
btf_fd = bpf_load_btf(raw_btf, raw_btf_size,
btf_log_buf, BTF_LOG_BUF_SIZE,
always_log);
- if (CHECK(btf_fd == -1, "errno:%d", errno)) {
+ if (CHECK(btf_fd < 0, "errno:%d", errno)) {
err = -1;
goto done;
}
free(raw_btf);
free(user_btf);
- if (btf_fd != -1)
+ if (btf_fd >= 0)
close(btf_fd);
return err;
btf_fd[0] = bpf_load_btf(raw_btf, raw_btf_size,
btf_log_buf, BTF_LOG_BUF_SIZE,
always_log);
- if (CHECK(btf_fd[0] == -1, "errno:%d", errno)) {
+ if (CHECK(btf_fd[0] < 0, "errno:%d", errno)) {
err = -1;
goto done;
}
}
btf_fd[1] = bpf_btf_get_fd_by_id(info[0].id);
- if (CHECK(btf_fd[1] == -1, "errno:%d", errno)) {
+ if (CHECK(btf_fd[1] < 0, "errno:%d", errno)) {
err = -1;
goto done;
}
create_attr.btf_value_type_id = 2;
map_fd = bpf_create_map_xattr(&create_attr);
- if (CHECK(map_fd == -1, "errno:%d", errno)) {
+ if (CHECK(map_fd < 0, "errno:%d", errno)) {
err = -1;
goto done;
}
/* Test BTF ID is removed from the kernel */
btf_fd[0] = bpf_btf_get_fd_by_id(map_info.btf_id);
- if (CHECK(btf_fd[0] == -1, "errno:%d", errno)) {
+ if (CHECK(btf_fd[0] < 0, "errno:%d", errno)) {
err = -1;
goto done;
}
close(map_fd);
map_fd = -1;
btf_fd[0] = bpf_btf_get_fd_by_id(map_info.btf_id);
- if (CHECK(btf_fd[0] != -1, "BTF lingers")) {
+ if (CHECK(btf_fd[0] >= 0, "BTF lingers")) {
err = -1;
goto done;
}
fprintf(stderr, "\n%s", btf_log_buf);
free(raw_btf);
- if (map_fd != -1)
+ if (map_fd >= 0)
close(map_fd);
for (i = 0; i < 2; i++) {
free(user_btf[i]);
- if (btf_fd[i] != -1)
+ if (btf_fd[i] >= 0)
close(btf_fd[i]);
}
btf_fd = bpf_load_btf(raw_btf, raw_btf_size,
btf_log_buf, BTF_LOG_BUF_SIZE,
always_log);
- if (CHECK(btf_fd == -1, "errno:%d", errno)) {
+ if (CHECK(btf_fd <= 0, "errno:%d", errno)) {
err = -1;
goto done;
}
free(raw_btf);
free(user_btf);
- if (btf_fd != -1)
+ if (btf_fd >= 0)
close(btf_fd);
}
return;
btf = btf__parse_elf(test->file, &btf_ext);
- if (IS_ERR(btf)) {
- if (PTR_ERR(btf) == -ENOENT) {
+ err = libbpf_get_error(btf);
+ if (err) {
+ if (err == -ENOENT) {
printf("%s:SKIP: No ELF %s found", __func__, BTF_ELF_SEC);
test__skip();
return;
btf_ext__free(btf_ext);
obj = bpf_object__open(test->file);
- if (CHECK(IS_ERR(obj), "obj: %ld", PTR_ERR(obj)))
+ err = libbpf_get_error(obj);
+ if (CHECK(err, "obj: %d", err))
return;
prog = bpf_program__next(NULL, obj);
info_len = sizeof(struct bpf_prog_info);
err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len);
- if (CHECK(err == -1, "invalid get info (1st) errno:%d", errno)) {
+ if (CHECK(err < 0, "invalid get info (1st) errno:%d", errno)) {
fprintf(stderr, "%s\n", btf_log_buf);
err = -1;
goto done;
err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len);
- if (CHECK(err == -1, "invalid get info (2nd) errno:%d", errno)) {
+ if (CHECK(err < 0, "invalid get info (2nd) errno:%d", errno)) {
fprintf(stderr, "%s\n", btf_log_buf);
err = -1;
goto done;
always_log);
free(raw_btf);
- if (CHECK(btf_fd == -1, "errno:%d", errno)) {
+ if (CHECK(btf_fd < 0, "errno:%d", errno)) {
err = -1;
goto done;
}
create_attr.btf_value_type_id = test->value_type_id;
map_fd = bpf_create_map_xattr(&create_attr);
- if (CHECK(map_fd == -1, "errno:%d", errno)) {
+ if (CHECK(map_fd < 0, "errno:%d", errno)) {
err = -1;
goto done;
}
err = check_line(expected_line, nexpected_line,
sizeof(expected_line), line);
- if (err == -1)
+ if (err < 0)
goto done;
}
cpu, cmapv);
err = check_line(expected_line, nexpected_line,
sizeof(expected_line), line);
- if (err == -1)
+ if (err < 0)
goto done;
cmapv = cmapv + rounded_value_size;
fprintf(stderr, "OK");
if (*btf_log_buf && (err || always_log))
fprintf(stderr, "\n%s", btf_log_buf);
- if (btf_fd != -1)
+ if (btf_fd >= 0)
close(btf_fd);
- if (map_fd != -1)
+ if (map_fd >= 0)
close(map_fd);
if (pin_file)
fclose(pin_file);
/* get necessary lens */
info_len = sizeof(struct bpf_prog_info);
err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len);
- if (CHECK(err == -1, "invalid get info (1st) errno:%d", errno)) {
+ if (CHECK(err < 0, "invalid get info (1st) errno:%d", errno)) {
fprintf(stderr, "%s\n", btf_log_buf);
return -1;
}
info.func_info_rec_size = rec_size;
info.func_info = ptr_to_u64(func_info);
err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len);
- if (CHECK(err == -1, "invalid get info (2nd) errno:%d", errno)) {
+ if (CHECK(err < 0, "invalid get info (2nd) errno:%d", errno)) {
fprintf(stderr, "%s\n", btf_log_buf);
err = -1;
goto done;
info_len = sizeof(struct bpf_prog_info);
err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len);
- if (CHECK(err == -1, "err:%d errno:%d", err, errno)) {
+ if (CHECK(err < 0, "err:%d errno:%d", err, errno)) {
err = -1;
goto done;
}
* Only recheck the info.*line_info* fields.
* Other fields are not the concern of this test.
*/
- if (CHECK(err == -1 ||
+ if (CHECK(err < 0 ||
info.nr_line_info != cnt ||
(jited_cnt && !info.jited_line_info) ||
info.nr_jited_line_info != jited_cnt ||
always_log);
free(raw_btf);
- if (CHECK(btf_fd == -1, "invalid btf_fd errno:%d", errno)) {
+ if (CHECK(btf_fd < 0, "invalid btf_fd errno:%d", errno)) {
err = -1;
goto done;
}
patched_linfo = patch_name_tbd(test->line_info,
test->str_sec, linfo_str_off,
test->str_sec_size, &linfo_size);
- if (IS_ERR(patched_linfo)) {
+ err = libbpf_get_error(patched_linfo);
+ if (err) {
fprintf(stderr, "error in creating raw bpf_line_info");
err = -1;
goto done;
}
prog_fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, sizeof(attr));
- err = ((prog_fd == -1) != test->expected_prog_load_failure);
+ err = ((prog_fd < 0) != test->expected_prog_load_failure);
if (CHECK(err, "prog_fd:%d expected_prog_load_failure:%u errno:%d",
prog_fd, test->expected_prog_load_failure, errno) ||
CHECK(test->err_str && !strstr(btf_log_buf, test->err_str),
goto done;
}
- if (prog_fd == -1)
+ if (prog_fd < 0)
goto done;
err = test_get_finfo(test, prog_fd);
if (*btf_log_buf && (err || always_log))
fprintf(stderr, "\n%s", btf_log_buf);
- if (btf_fd != -1)
+ if (btf_fd >= 0)
close(btf_fd);
- if (prog_fd != -1)
+ if (prog_fd >= 0)
close(prog_fd);
- if (!IS_ERR(patched_linfo))
+ if (!libbpf_get_error(patched_linfo))
free(patched_linfo);
}
return;
test_btf = btf__new((__u8 *)raw_btf, raw_btf_size);
+ err = libbpf_get_error(test_btf);
free(raw_btf);
- if (CHECK(IS_ERR(test_btf), "invalid test_btf errno:%ld",
- PTR_ERR(test_btf))) {
+ if (CHECK(err, "invalid test_btf errno:%d", err)) {
err = -1;
goto done;
}
if (!raw_btf)
return;
expect_btf = btf__new((__u8 *)raw_btf, raw_btf_size);
+ err = libbpf_get_error(expect_btf);
free(raw_btf);
- if (CHECK(IS_ERR(expect_btf), "invalid expect_btf errno:%ld",
- PTR_ERR(expect_btf))) {
+ if (CHECK(err, "invalid expect_btf errno:%d", err)) {
err = -1;
goto done;
}
}
done:
- if (!IS_ERR(test_btf))
- btf__free(test_btf);
- if (!IS_ERR(expect_btf))
- btf__free(expect_btf);
+ btf__free(test_btf);
+ btf__free(expect_btf);
}
void test_btf(void)
int err = 0, id;
d = btf_dump__new(btf, NULL, opts, btf_dump_printf);
- if (IS_ERR(d))
- return PTR_ERR(d);
+ err = libbpf_get_error(d);
+ if (err)
+ return err;
for (id = 1; id <= type_cnt; id++) {
err = btf_dump__dump_type(d, id);
snprintf(test_file, sizeof(test_file), "%s.o", t->file);
btf = btf__parse_elf(test_file, NULL);
- if (CHECK(IS_ERR(btf), "btf_parse_elf",
- "failed to load test BTF: %ld\n", PTR_ERR(btf))) {
+ if (!ASSERT_OK_PTR(btf, "btf_parse_elf")) {
err = -PTR_ERR(btf);
btf = NULL;
goto done;
#include <bpf/btf.h>
#include "btf_helpers.h"
-static int duration = 0;
-
void test_btf_write() {
const struct btf_var_secinfo *vi;
const struct btf_type *t;
int id, err, str_off;
btf = btf__new_empty();
- if (CHECK(IS_ERR(btf), "new_empty", "failed: %ld\n", PTR_ERR(btf)))
+ if (!ASSERT_OK_PTR(btf, "new_empty"))
return;
str_off = btf__find_str(btf, "int");
*/
parent_link = bpf_program__attach_cgroup(obj->progs.egress,
parent_cgroup_fd);
- if (CHECK(IS_ERR(parent_link), "parent-cg-attach",
- "err %ld", PTR_ERR(parent_link)))
+ if (!ASSERT_OK_PTR(parent_link, "parent-cg-attach"))
goto close_bpf_object;
err = connect_send(CHILD_CGROUP);
if (CHECK(err, "first-connect-send", "errno %d", errno))
*/
child_link = bpf_program__attach_cgroup(obj->progs.egress,
child_cgroup_fd);
- if (CHECK(IS_ERR(child_link), "child-cg-attach",
- "err %ld", PTR_ERR(child_link)))
+ if (!ASSERT_OK_PTR(child_link, "child-cg-attach"))
goto close_bpf_object;
err = connect_send(CHILD_CGROUP);
if (CHECK(err, "second-connect-send", "errno %d", errno))
goto close_bpf_object;
close_bpf_object:
- if (!IS_ERR(parent_link))
- bpf_link__destroy(parent_link);
- if (!IS_ERR(child_link))
- bpf_link__destroy(child_link);
+ bpf_link__destroy(parent_link);
+ bpf_link__destroy(child_link);
cg_storage_multi_egress_only__destroy(obj);
}
*/
parent_egress1_link = bpf_program__attach_cgroup(obj->progs.egress1,
parent_cgroup_fd);
- if (CHECK(IS_ERR(parent_egress1_link), "parent-egress1-cg-attach",
- "err %ld", PTR_ERR(parent_egress1_link)))
+ if (!ASSERT_OK_PTR(parent_egress1_link, "parent-egress1-cg-attach"))
goto close_bpf_object;
parent_egress2_link = bpf_program__attach_cgroup(obj->progs.egress2,
parent_cgroup_fd);
- if (CHECK(IS_ERR(parent_egress2_link), "parent-egress2-cg-attach",
- "err %ld", PTR_ERR(parent_egress2_link)))
+ if (!ASSERT_OK_PTR(parent_egress2_link, "parent-egress2-cg-attach"))
goto close_bpf_object;
parent_ingress_link = bpf_program__attach_cgroup(obj->progs.ingress,
parent_cgroup_fd);
- if (CHECK(IS_ERR(parent_ingress_link), "parent-ingress-cg-attach",
- "err %ld", PTR_ERR(parent_ingress_link)))
+ if (!ASSERT_OK_PTR(parent_ingress_link, "parent-ingress-cg-attach"))
goto close_bpf_object;
err = connect_send(CHILD_CGROUP);
if (CHECK(err, "first-connect-send", "errno %d", errno))
*/
child_egress1_link = bpf_program__attach_cgroup(obj->progs.egress1,
child_cgroup_fd);
- if (CHECK(IS_ERR(child_egress1_link), "child-egress1-cg-attach",
- "err %ld", PTR_ERR(child_egress1_link)))
+ if (!ASSERT_OK_PTR(child_egress1_link, "child-egress1-cg-attach"))
goto close_bpf_object;
child_egress2_link = bpf_program__attach_cgroup(obj->progs.egress2,
child_cgroup_fd);
- if (CHECK(IS_ERR(child_egress2_link), "child-egress2-cg-attach",
- "err %ld", PTR_ERR(child_egress2_link)))
+ if (!ASSERT_OK_PTR(child_egress2_link, "child-egress2-cg-attach"))
goto close_bpf_object;
child_ingress_link = bpf_program__attach_cgroup(obj->progs.ingress,
child_cgroup_fd);
- if (CHECK(IS_ERR(child_ingress_link), "child-ingress-cg-attach",
- "err %ld", PTR_ERR(child_ingress_link)))
+ if (!ASSERT_OK_PTR(child_ingress_link, "child-ingress-cg-attach"))
goto close_bpf_object;
err = connect_send(CHILD_CGROUP);
if (CHECK(err, "second-connect-send", "errno %d", errno))
goto close_bpf_object;
close_bpf_object:
- if (!IS_ERR(parent_egress1_link))
- bpf_link__destroy(parent_egress1_link);
- if (!IS_ERR(parent_egress2_link))
- bpf_link__destroy(parent_egress2_link);
- if (!IS_ERR(parent_ingress_link))
- bpf_link__destroy(parent_ingress_link);
- if (!IS_ERR(child_egress1_link))
- bpf_link__destroy(child_egress1_link);
- if (!IS_ERR(child_egress2_link))
- bpf_link__destroy(child_egress2_link);
- if (!IS_ERR(child_ingress_link))
- bpf_link__destroy(child_ingress_link);
+ bpf_link__destroy(parent_egress1_link);
+ bpf_link__destroy(parent_egress2_link);
+ bpf_link__destroy(parent_ingress_link);
+ bpf_link__destroy(child_egress1_link);
+ bpf_link__destroy(child_egress2_link);
+ bpf_link__destroy(child_ingress_link);
cg_storage_multi_isolated__destroy(obj);
}
*/
parent_egress1_link = bpf_program__attach_cgroup(obj->progs.egress1,
parent_cgroup_fd);
- if (CHECK(IS_ERR(parent_egress1_link), "parent-egress1-cg-attach",
- "err %ld", PTR_ERR(parent_egress1_link)))
+ if (!ASSERT_OK_PTR(parent_egress1_link, "parent-egress1-cg-attach"))
goto close_bpf_object;
parent_egress2_link = bpf_program__attach_cgroup(obj->progs.egress2,
parent_cgroup_fd);
- if (CHECK(IS_ERR(parent_egress2_link), "parent-egress2-cg-attach",
- "err %ld", PTR_ERR(parent_egress2_link)))
+ if (!ASSERT_OK_PTR(parent_egress2_link, "parent-egress2-cg-attach"))
goto close_bpf_object;
parent_ingress_link = bpf_program__attach_cgroup(obj->progs.ingress,
parent_cgroup_fd);
- if (CHECK(IS_ERR(parent_ingress_link), "parent-ingress-cg-attach",
- "err %ld", PTR_ERR(parent_ingress_link)))
+ if (!ASSERT_OK_PTR(parent_ingress_link, "parent-ingress-cg-attach"))
goto close_bpf_object;
err = connect_send(CHILD_CGROUP);
if (CHECK(err, "first-connect-send", "errno %d", errno))
*/
child_egress1_link = bpf_program__attach_cgroup(obj->progs.egress1,
child_cgroup_fd);
- if (CHECK(IS_ERR(child_egress1_link), "child-egress1-cg-attach",
- "err %ld", PTR_ERR(child_egress1_link)))
+ if (!ASSERT_OK_PTR(child_egress1_link, "child-egress1-cg-attach"))
goto close_bpf_object;
child_egress2_link = bpf_program__attach_cgroup(obj->progs.egress2,
child_cgroup_fd);
- if (CHECK(IS_ERR(child_egress2_link), "child-egress2-cg-attach",
- "err %ld", PTR_ERR(child_egress2_link)))
+ if (!ASSERT_OK_PTR(child_egress2_link, "child-egress2-cg-attach"))
goto close_bpf_object;
child_ingress_link = bpf_program__attach_cgroup(obj->progs.ingress,
child_cgroup_fd);
- if (CHECK(IS_ERR(child_ingress_link), "child-ingress-cg-attach",
- "err %ld", PTR_ERR(child_ingress_link)))
+ if (!ASSERT_OK_PTR(child_ingress_link, "child-ingress-cg-attach"))
goto close_bpf_object;
err = connect_send(CHILD_CGROUP);
if (CHECK(err, "second-connect-send", "errno %d", errno))
goto close_bpf_object;
close_bpf_object:
- if (!IS_ERR(parent_egress1_link))
- bpf_link__destroy(parent_egress1_link);
- if (!IS_ERR(parent_egress2_link))
- bpf_link__destroy(parent_egress2_link);
- if (!IS_ERR(parent_ingress_link))
- bpf_link__destroy(parent_ingress_link);
- if (!IS_ERR(child_egress1_link))
- bpf_link__destroy(child_egress1_link);
- if (!IS_ERR(child_egress2_link))
- bpf_link__destroy(child_egress2_link);
- if (!IS_ERR(child_ingress_link))
- bpf_link__destroy(child_ingress_link);
+ bpf_link__destroy(parent_egress1_link);
+ bpf_link__destroy(parent_egress2_link);
+ bpf_link__destroy(parent_ingress_link);
+ bpf_link__destroy(child_egress1_link);
+ bpf_link__destroy(child_egress2_link);
+ bpf_link__destroy(child_ingress_link);
cg_storage_multi_shared__destroy(obj);
}
prog_cnt = 2;
CHECK_FAIL(bpf_prog_query(cg5, BPF_CGROUP_INET_EGRESS,
BPF_F_QUERY_EFFECTIVE, &attach_flags,
- prog_ids, &prog_cnt) != -1);
+ prog_ids, &prog_cnt) >= 0);
CHECK_FAIL(errno != ENOSPC);
CHECK_FAIL(prog_cnt != 4);
/* check that prog_ids are returned even when buffer is too small */
for (i = 0; i < cg_nr; i++) {
links[i] = bpf_program__attach_cgroup(skel->progs.egress,
cgs[i].fd);
- if (CHECK(IS_ERR(links[i]), "cg_attach", "i: %d, err: %ld\n",
- i, PTR_ERR(links[i])))
+ if (!ASSERT_OK_PTR(links[i], "cg_attach"))
goto cleanup;
}
links[last_cg] = bpf_program__attach_cgroup(skel->progs.egress,
cgs[last_cg].fd);
- if (CHECK(IS_ERR(links[last_cg]), "cg_attach", "err: %ld\n",
- PTR_ERR(links[last_cg])))
+ if (!ASSERT_OK_PTR(links[last_cg], "cg_attach"))
goto cleanup;
ping_and_check(cg_nr + 1, 0);
/* attempt to mix in with multi-attach bpf_link */
tmp_link = bpf_program__attach_cgroup(skel->progs.egress,
cgs[last_cg].fd);
- if (CHECK(!IS_ERR(tmp_link), "cg_attach_fail", "unexpected success!\n")) {
+ if (!ASSERT_ERR_PTR(tmp_link, "cg_attach_fail")) {
bpf_link__destroy(tmp_link);
goto cleanup;
}
/* attach back link-based one */
links[last_cg] = bpf_program__attach_cgroup(skel->progs.egress,
cgs[last_cg].fd);
- if (CHECK(IS_ERR(links[last_cg]), "cg_attach", "err: %ld\n",
- PTR_ERR(links[last_cg])))
+ if (!ASSERT_OK_PTR(links[last_cg], "cg_attach"))
goto cleanup;
ping_and_check(cg_nr, 0);
BPF_CGROUP_INET_EGRESS);
for (i = 0; i < cg_nr; i++) {
- if (!IS_ERR(links[i]))
- bpf_link__destroy(links[i]);
+ bpf_link__destroy(links[i]);
}
test_cgroup_link__destroy(skel);
goto cleanup;
link = bpf_program__attach_cgroup(skel->progs.ingress_lookup, cgfd);
- if (CHECK(IS_ERR(link), "cgroup_attach", "err: %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "cgroup_attach"))
goto cleanup;
run_lookup_test(&skel->bss->g_serv_port, out_sk);
prog = skel->progs.xdp_use_helper_basic;
link = bpf_program__attach_xdp(prog, IFINDEX_LO);
- if (CHECK(IS_ERR(link), "link_attach", "failed: %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "link_attach"))
goto out;
skel->links.xdp_use_helper_basic = link;
const char *name;
int i;
- if (CHECK(IS_ERR(local_btf), "local_btf", "failed: %ld\n", PTR_ERR(local_btf)) ||
- CHECK(IS_ERR(targ_btf), "targ_btf", "failed: %ld\n", PTR_ERR(targ_btf))) {
+ if (!ASSERT_OK_PTR(local_btf, "local_btf") || !ASSERT_OK_PTR(targ_btf, "targ_btf")) {
btf__free(local_btf);
btf__free(targ_btf);
return -EINVAL;
}
obj = bpf_object__open_file(test_case->bpf_obj_file, NULL);
- if (CHECK(IS_ERR(obj), "obj_open", "failed to open '%s': %ld\n",
- test_case->bpf_obj_file, PTR_ERR(obj)))
+ if (!ASSERT_OK_PTR(obj, "obj_open"))
continue;
probe_name = "raw_tracepoint/sys_enter";
data->my_pid_tgid = my_pid_tgid;
link = bpf_program__attach_raw_tracepoint(prog, tp_name);
- if (CHECK(IS_ERR(link), "attach_raw_tp", "err %ld\n",
- PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_raw_tp"))
goto cleanup;
/* trigger test run */
CHECK_FAIL(munmap(mmap_data, mmap_sz));
mmap_data = NULL;
}
- if (!IS_ERR_OR_NULL(link)) {
- bpf_link__destroy(link);
- link = NULL;
- }
+ bpf_link__destroy(link);
+ link = NULL;
bpf_object__close(obj);
}
}
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2019 Facebook */
#include <test_progs.h>
-#include "fentry_test.skel.h"
-#include "fexit_test.skel.h"
+#include "fentry_test.lskel.h"
+#include "fexit_test.lskel.h"
void test_fentry_fexit(void)
{
if (CHECK(err, "fexit_attach", "fexit attach failed: %d\n", err))
goto close_prog;
- prog_fd = bpf_program__fd(fexit_skel->progs.test1);
+ prog_fd = fexit_skel->progs.test1.prog_fd;
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
NULL, NULL, &retval, &duration);
CHECK(err || retval, "ipv6",
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2019 Facebook */
#include <test_progs.h>
-#include "fentry_test.skel.h"
+#include "fentry_test.lskel.h"
static int fentry_test(struct fentry_test *fentry_skel)
{
int err, prog_fd, i;
__u32 duration = 0, retval;
- struct bpf_link *link;
+ int link_fd;
__u64 *result;
err = fentry_test__attach(fentry_skel);
return err;
/* Check that already linked program can't be attached again. */
- link = bpf_program__attach(fentry_skel->progs.test1);
- if (!ASSERT_ERR_PTR(link, "fentry_attach_link"))
+ link_fd = fentry_test__test1__attach(fentry_skel);
+ if (!ASSERT_LT(link_fd, 0, "fentry_attach_link"))
return -1;
- prog_fd = bpf_program__fd(fentry_skel->progs.test1);
+ prog_fd = fentry_skel->progs.test1.prog_fd;
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
NULL, NULL, &retval, &duration);
ASSERT_OK(err, "test_run");
close_prog:
for (i = 0; i < prog_cnt; i++)
- if (!IS_ERR_OR_NULL(link[i]))
- bpf_link__destroy(link[i]);
- if (!IS_ERR_OR_NULL(obj))
- bpf_object__close(obj);
+ bpf_link__destroy(link[i]);
+ bpf_object__close(obj);
bpf_object__close(tgt_obj);
free(link);
free(prog);
return err;
link = bpf_program__attach_freplace(prog, tgt_fd, tgt_name);
- if (CHECK(IS_ERR(link), "second_link", "failed to attach second link prog_fd %d tgt_fd %d\n", bpf_program__fd(prog), tgt_fd))
+ if (!ASSERT_OK_PTR(link, "second_link"))
goto out;
err = bpf_prog_test_run(tgt_fd, 1, &pkt_v6, sizeof(pkt_v6),
opts.attach_prog_fd = pkt_fd;
freplace_obj = bpf_object__open_file(freplace_name, &opts);
- if (CHECK(IS_ERR_OR_NULL(freplace_obj), "freplace_obj_open",
- "failed to open %s: %ld\n", freplace_name,
- PTR_ERR(freplace_obj)))
+ if (!ASSERT_OK_PTR(freplace_obj, "freplace_obj_open"))
goto out;
err = bpf_object__load(freplace_obj);
prog = bpf_program__next(NULL, freplace_obj);
freplace_link = bpf_program__attach_trace(prog);
- if (CHECK(IS_ERR(freplace_link), "freplace_attach_trace", "failed to link\n"))
+ if (!ASSERT_OK_PTR(freplace_link, "freplace_attach_trace"))
goto out;
opts.attach_prog_fd = bpf_program__fd(prog);
fmod_obj = bpf_object__open_file(fmod_ret_name, &opts);
- if (CHECK(IS_ERR_OR_NULL(fmod_obj), "fmod_obj_open",
- "failed to open %s: %ld\n", fmod_ret_name,
- PTR_ERR(fmod_obj)))
+ if (!ASSERT_OK_PTR(fmod_obj, "fmod_obj_open"))
goto out;
err = bpf_object__load(fmod_obj);
);
obj = bpf_object__open_file(obj_file, &opts);
- if (CHECK(IS_ERR_OR_NULL(obj), "obj_open",
- "failed to open %s: %ld\n", obj_file,
- PTR_ERR(obj)))
+ if (!ASSERT_OK_PTR(obj, "obj_open"))
goto close_prog;
/* It should fail to load the program */
goto close_prog;
close_prog:
- if (!IS_ERR_OR_NULL(obj))
- bpf_object__close(obj);
+ bpf_object__close(obj);
bpf_object__close(pkt_obj);
}
#include <time.h>
#include <sys/mman.h>
#include <sys/syscall.h>
-#include "fexit_sleep.skel.h"
+#include "fexit_sleep.lskel.h"
static int do_sleep(void *skel)
{
* waiting for percpu_ref_kill to confirm). The other one
* will be freed quickly.
*/
- close(bpf_program__fd(fexit_skel->progs.nanosleep_fentry));
- close(bpf_program__fd(fexit_skel->progs.nanosleep_fexit));
+ close(fexit_skel->progs.nanosleep_fentry.prog_fd);
+ close(fexit_skel->progs.nanosleep_fexit.prog_fd);
fexit_sleep__detach(fexit_skel);
/* kill the thread to unwind sys_nanosleep stack through the trampoline */
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2019 Facebook */
#include <test_progs.h>
-#include "fexit_test.skel.h"
+#include "fexit_test.lskel.h"
static int fexit_test(struct fexit_test *fexit_skel)
{
int err, prog_fd, i;
__u32 duration = 0, retval;
- struct bpf_link *link;
+ int link_fd;
__u64 *result;
err = fexit_test__attach(fexit_skel);
return err;
/* Check that already linked program can't be attached again. */
- link = bpf_program__attach(fexit_skel->progs.test1);
- if (!ASSERT_ERR_PTR(link, "fexit_attach_link"))
+ link_fd = fexit_test__test1__attach(fexit_skel);
+ if (!ASSERT_LT(link_fd, 0, "fexit_attach_link"))
return -1;
- prog_fd = bpf_program__fd(fexit_skel->progs.test1);
+ prog_fd = fexit_skel->progs.test1.prog_fd;
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
NULL, NULL, &retval, &duration);
ASSERT_OK(err, "test_run");
return;
link = bpf_program__attach_netns(skel->progs._dissect, net_fd);
- if (CHECK(IS_ERR(link), "attach_netns", "err %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_netns"))
goto out_close;
run_tests_skb_less(tap_fd, skel->maps.last_dissection);
/* Expect failure creating link when another link exists */
errno = 0;
link2 = bpf_link_create(prog2, netns, BPF_FLOW_DISSECTOR, &opts);
- if (CHECK_FAIL(link2 != -1 || errno != E2BIG))
+ if (CHECK_FAIL(link2 >= 0 || errno != E2BIG))
perror("bpf_prog_attach(prog2) expected E2BIG");
- if (link2 != -1)
+ if (link2 >= 0)
close(link2);
CHECK_FAIL(query_attached_prog_id(netns) != query_prog_id(prog1));
/* Expect failure creating link when prog attached */
errno = 0;
link = bpf_link_create(prog2, netns, BPF_FLOW_DISSECTOR, &opts);
- if (CHECK_FAIL(link != -1 || errno != EEXIST))
+ if (CHECK_FAIL(link >= 0 || errno != EEXIST))
perror("bpf_link_create(prog2) expected EEXIST");
- if (link != -1)
+ if (link >= 0)
close(link);
CHECK_FAIL(query_attached_prog_id(netns) != query_prog_id(prog1));
}
out_close:
for (i = 0; i < ARRAY_SIZE(progs); i++) {
- if (progs[i] != -1)
+ if (progs[i] >= 0)
CHECK_FAIL(close(progs[i]));
}
}
goto close_prog;
link = bpf_program__attach_raw_tracepoint(prog, "sys_enter");
- if (CHECK(IS_ERR(link), "attach_raw_tp", "err %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_raw_tp"))
goto close_prog;
pb_opts.sample_cb = get_stack_print_output;
pb = perf_buffer__new(bpf_map__fd(map), 8, &pb_opts);
- if (CHECK(IS_ERR(pb), "perf_buf__new", "err %ld\n", PTR_ERR(pb)))
+ if (!ASSERT_OK_PTR(pb, "perf_buf__new"))
goto close_prog;
/* trigger some syscall action */
}
close_prog:
- if (!IS_ERR_OR_NULL(link))
- bpf_link__destroy(link);
- if (!IS_ERR_OR_NULL(pb))
- perf_buffer__free(pb);
+ bpf_link__destroy(link);
+ perf_buffer__free(pb);
bpf_object__close(obj);
}
skel->links.oncpu = bpf_program__attach_perf_event(skel->progs.oncpu,
pmu_fd);
- CHECK(!IS_ERR(skel->links.oncpu), "attach_perf_event_no_callchain",
- "should have failed\n");
+ ASSERT_ERR_PTR(skel->links.oncpu, "attach_perf_event_no_callchain");
close(pmu_fd);
/* add PERF_SAMPLE_CALLCHAIN, attach should succeed */
skel->links.oncpu = bpf_program__attach_perf_event(skel->progs.oncpu,
pmu_fd);
- CHECK(IS_ERR(skel->links.oncpu), "attach_perf_event_callchain",
- "err: %ld\n", PTR_ERR(skel->links.oncpu));
+ ASSERT_OK_PTR(skel->links.oncpu, "attach_perf_event_callchain");
close(pmu_fd);
/* add exclude_callchain_kernel, attach should fail */
skel->links.oncpu = bpf_program__attach_perf_event(skel->progs.oncpu,
pmu_fd);
- CHECK(!IS_ERR(skel->links.oncpu), "attach_perf_event_exclude_callchain_kernel",
- "should have failed\n");
+ ASSERT_ERR_PTR(skel->links.oncpu, "attach_perf_event_exclude_callchain_kernel");
close(pmu_fd);
cleanup:
struct hashmap *map;
map = hashmap__new(hash_fn, equal_fn, NULL);
- if (CHECK(IS_ERR(map), "hashmap__new",
- "failed to create map: %ld\n", PTR_ERR(map)))
+ if (!ASSERT_OK_PTR(map, "hashmap__new"))
return;
for (i = 0; i < ELEM_CNT; i++) {
/* force collisions */
map = hashmap__new(collision_hash_fn, equal_fn, NULL);
- if (CHECK(IS_ERR(map), "hashmap__new",
- "failed to create map: %ld\n", PTR_ERR(map)))
+ if (!ASSERT_OK_PTR(map, "hashmap__new"))
return;
/* set up multimap:
/* force collisions */
map = hashmap__new(hash_fn, equal_fn, NULL);
- if (CHECK(IS_ERR(map), "hashmap__new",
- "failed to create map: %ld\n", PTR_ERR(map)))
+ if (!ASSERT_OK_PTR(map, "hashmap__new"))
goto cleanup;
if (CHECK(hashmap__size(map) != 0, "hashmap__size",
goto close_prog;
link = bpf_program__attach_raw_tracepoint(prog, NULL);
- if (CHECK(IS_ERR(link), "attach_raw_tp", "err %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_raw_tp"))
goto close_prog;
link_fentry = bpf_program__attach_trace(fentry);
- if (CHECK(IS_ERR(link_fentry), "attach fentry", "err %ld\n",
- PTR_ERR(link_fentry)))
+ if (!ASSERT_OK_PTR(link_fentry, "attach fentry"))
goto close_prog;
link_fexit = bpf_program__attach_trace(fexit);
- if (CHECK(IS_ERR(link_fexit), "attach fexit", "err %ld\n",
- PTR_ERR(link_fexit)))
+ if (!ASSERT_OK_PTR(link_fexit, "attach fexit"))
goto close_prog;
perf_buf_map = bpf_object__find_map_by_name(obj2, "perf_buf_map");
pb_opts.sample_cb = on_sample;
pb_opts.ctx = &passed;
pb = perf_buffer__new(bpf_map__fd(perf_buf_map), 1, &pb_opts);
- if (CHECK(IS_ERR(pb), "perf_buf__new", "err %ld\n", PTR_ERR(pb)))
+ if (!ASSERT_OK_PTR(pb, "perf_buf__new"))
goto close_prog;
memcpy(skb.cb, &cb, sizeof(cb));
CHECK_FAIL(!test_ok[0] || !test_ok[1]);
close_prog:
perf_buffer__free(pb);
- if (!IS_ERR_OR_NULL(link))
- bpf_link__destroy(link);
- if (!IS_ERR_OR_NULL(link_fentry))
- bpf_link__destroy(link_fentry);
- if (!IS_ERR_OR_NULL(link_fexit))
- bpf_link__destroy(link_fexit);
+ bpf_link__destroy(link);
+ bpf_link__destroy(link_fentry);
+ bpf_link__destroy(link_fexit);
bpf_object__close(obj);
bpf_object__close(obj2);
}
/* Copyright (c) 2021 Facebook */
#include <test_progs.h>
#include <network_helpers.h>
-#include "kfunc_call_test.skel.h"
+#include "kfunc_call_test.lskel.h"
#include "kfunc_call_test_subprog.skel.h"
static void test_main(void)
if (!ASSERT_OK_PTR(skel, "skel"))
return;
- prog_fd = bpf_program__fd(skel->progs.kfunc_call_test1);
+ prog_fd = skel->progs.kfunc_call_test1.prog_fd;
err = bpf_prog_test_run(prog_fd, 1, &pkt_v4, sizeof(pkt_v4),
NULL, NULL, (__u32 *)&retval, NULL);
ASSERT_OK(err, "bpf_prog_test_run(test1)");
ASSERT_EQ(retval, 12, "test1-retval");
- prog_fd = bpf_program__fd(skel->progs.kfunc_call_test2);
+ prog_fd = skel->progs.kfunc_call_test2.prog_fd;
err = bpf_prog_test_run(prog_fd, 1, &pkt_v4, sizeof(pkt_v4),
NULL, NULL, (__u32 *)&retval, NULL);
ASSERT_OK(err, "bpf_prog_test_run(test2)");
struct btf *btf;
btf = libbpf_find_kernel_btf();
- if (CHECK(IS_ERR(btf), "btf_exists", "failed to load kernel BTF: %ld\n",
- PTR_ERR(btf)))
+ if (!ASSERT_OK_PTR(btf, "btf_exists"))
return;
percpu_datasec = btf__find_by_name_kind(btf, ".data..percpu",
#include <test_progs.h>
#include <bpf/libbpf.h>
#include <bpf/btf.h>
-#include "test_ksyms_module.skel.h"
+#include "test_ksyms_module.lskel.h"
static int duration;
int err, i;
link = bpf_program__attach(prog);
- if (CHECK(IS_ERR(link), "link_attach", "err: %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "link_attach"))
goto cleanup;
bss->in = 1;
/* re-open link from BPFFS */
link = bpf_link__open(link_pin_path);
- if (CHECK(IS_ERR(link), "link_open", "err: %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "link_open"))
goto cleanup;
CHECK(strcmp(link_pin_path, bpf_link__pin_path(link)), "pin_path2",
CHECK(i == 10000, "link_attached", "got to iteration #%d\n", i);
cleanup:
- if (!IS_ERR(link))
- bpf_link__destroy(link);
+ bpf_link__destroy(link);
}
void test_link_pinning(void)
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <test_progs.h>
+#include "test_lookup_and_delete.skel.h"
+
+#define START_VALUE 1234
+#define NEW_VALUE 4321
+#define MAX_ENTRIES 2
+
+static int duration;
+static int nr_cpus;
+
+static int fill_values(int map_fd)
+{
+ __u64 key, value = START_VALUE;
+ int err;
+
+ for (key = 1; key < MAX_ENTRIES + 1; key++) {
+ err = bpf_map_update_elem(map_fd, &key, &value, BPF_NOEXIST);
+ if (!ASSERT_OK(err, "bpf_map_update_elem"))
+ return -1;
+ }
+
+ return 0;
+}
+
+static int fill_values_percpu(int map_fd)
+{
+ __u64 key, value[nr_cpus];
+ int i, err;
+
+ for (i = 0; i < nr_cpus; i++)
+ value[i] = START_VALUE;
+
+ for (key = 1; key < MAX_ENTRIES + 1; key++) {
+ err = bpf_map_update_elem(map_fd, &key, value, BPF_NOEXIST);
+ if (!ASSERT_OK(err, "bpf_map_update_elem"))
+ return -1;
+ }
+
+ return 0;
+}
+
+static struct test_lookup_and_delete *setup_prog(enum bpf_map_type map_type,
+ int *map_fd)
+{
+ struct test_lookup_and_delete *skel;
+ int err;
+
+ skel = test_lookup_and_delete__open();
+ if (!ASSERT_OK_PTR(skel, "test_lookup_and_delete__open"))
+ return NULL;
+
+ err = bpf_map__set_type(skel->maps.hash_map, map_type);
+ if (!ASSERT_OK(err, "bpf_map__set_type"))
+ goto cleanup;
+
+ err = bpf_map__set_max_entries(skel->maps.hash_map, MAX_ENTRIES);
+ if (!ASSERT_OK(err, "bpf_map__set_max_entries"))
+ goto cleanup;
+
+ err = test_lookup_and_delete__load(skel);
+ if (!ASSERT_OK(err, "test_lookup_and_delete__load"))
+ goto cleanup;
+
+ *map_fd = bpf_map__fd(skel->maps.hash_map);
+ if (!ASSERT_GE(*map_fd, 0, "bpf_map__fd"))
+ goto cleanup;
+
+ return skel;
+
+cleanup:
+ test_lookup_and_delete__destroy(skel);
+ return NULL;
+}
+
+/* Triggers BPF program that updates map with given key and value */
+static int trigger_tp(struct test_lookup_and_delete *skel, __u64 key,
+ __u64 value)
+{
+ int err;
+
+ skel->bss->set_pid = getpid();
+ skel->bss->set_key = key;
+ skel->bss->set_value = value;
+
+ err = test_lookup_and_delete__attach(skel);
+ if (!ASSERT_OK(err, "test_lookup_and_delete__attach"))
+ return -1;
+
+ syscall(__NR_getpgid);
+
+ test_lookup_and_delete__detach(skel);
+
+ return 0;
+}
+
+static void test_lookup_and_delete_hash(void)
+{
+ struct test_lookup_and_delete *skel;
+ __u64 key, value;
+ int map_fd, err;
+
+ /* Setup program and fill the map. */
+ skel = setup_prog(BPF_MAP_TYPE_HASH, &map_fd);
+ if (!ASSERT_OK_PTR(skel, "setup_prog"))
+ return;
+
+ err = fill_values(map_fd);
+ if (!ASSERT_OK(err, "fill_values"))
+ goto cleanup;
+
+ /* Lookup and delete element. */
+ key = 1;
+ err = bpf_map_lookup_and_delete_elem(map_fd, &key, &value);
+ if (!ASSERT_OK(err, "bpf_map_lookup_and_delete_elem"))
+ goto cleanup;
+
+ /* Fetched value should match the initially set value. */
+ if (CHECK(value != START_VALUE, "bpf_map_lookup_and_delete_elem",
+ "unexpected value=%lld\n", value))
+ goto cleanup;
+
+ /* Check that the entry is non existent. */
+ err = bpf_map_lookup_elem(map_fd, &key, &value);
+ if (!ASSERT_ERR(err, "bpf_map_lookup_elem"))
+ goto cleanup;
+
+cleanup:
+ test_lookup_and_delete__destroy(skel);
+}
+
+static void test_lookup_and_delete_percpu_hash(void)
+{
+ struct test_lookup_and_delete *skel;
+ __u64 key, val, value[nr_cpus];
+ int map_fd, err, i;
+
+ /* Setup program and fill the map. */
+ skel = setup_prog(BPF_MAP_TYPE_PERCPU_HASH, &map_fd);
+ if (!ASSERT_OK_PTR(skel, "setup_prog"))
+ return;
+
+ err = fill_values_percpu(map_fd);
+ if (!ASSERT_OK(err, "fill_values_percpu"))
+ goto cleanup;
+
+ /* Lookup and delete element. */
+ key = 1;
+ err = bpf_map_lookup_and_delete_elem(map_fd, &key, value);
+ if (!ASSERT_OK(err, "bpf_map_lookup_and_delete_elem"))
+ goto cleanup;
+
+ for (i = 0; i < nr_cpus; i++) {
+ val = value[i];
+
+ /* Fetched value should match the initially set value. */
+ if (CHECK(val != START_VALUE, "map value",
+ "unexpected for cpu %d: %lld\n", i, val))
+ goto cleanup;
+ }
+
+ /* Check that the entry is non existent. */
+ err = bpf_map_lookup_elem(map_fd, &key, value);
+ if (!ASSERT_ERR(err, "bpf_map_lookup_elem"))
+ goto cleanup;
+
+cleanup:
+ test_lookup_and_delete__destroy(skel);
+}
+
+static void test_lookup_and_delete_lru_hash(void)
+{
+ struct test_lookup_and_delete *skel;
+ __u64 key, value;
+ int map_fd, err;
+
+ /* Setup program and fill the LRU map. */
+ skel = setup_prog(BPF_MAP_TYPE_LRU_HASH, &map_fd);
+ if (!ASSERT_OK_PTR(skel, "setup_prog"))
+ return;
+
+ err = fill_values(map_fd);
+ if (!ASSERT_OK(err, "fill_values"))
+ goto cleanup;
+
+ /* Insert new element at key=3, should reuse LRU element. */
+ key = 3;
+ err = trigger_tp(skel, key, NEW_VALUE);
+ if (!ASSERT_OK(err, "trigger_tp"))
+ goto cleanup;
+
+ /* Lookup and delete element 3. */
+ err = bpf_map_lookup_and_delete_elem(map_fd, &key, &value);
+ if (!ASSERT_OK(err, "bpf_map_lookup_and_delete_elem"))
+ goto cleanup;
+
+ /* Value should match the new value. */
+ if (CHECK(value != NEW_VALUE, "bpf_map_lookup_and_delete_elem",
+ "unexpected value=%lld\n", value))
+ goto cleanup;
+
+ /* Check that entries 3 and 1 are non existent. */
+ err = bpf_map_lookup_elem(map_fd, &key, &value);
+ if (!ASSERT_ERR(err, "bpf_map_lookup_elem"))
+ goto cleanup;
+
+ key = 1;
+ err = bpf_map_lookup_elem(map_fd, &key, &value);
+ if (!ASSERT_ERR(err, "bpf_map_lookup_elem"))
+ goto cleanup;
+
+cleanup:
+ test_lookup_and_delete__destroy(skel);
+}
+
+static void test_lookup_and_delete_lru_percpu_hash(void)
+{
+ struct test_lookup_and_delete *skel;
+ __u64 key, val, value[nr_cpus];
+ int map_fd, err, i, cpucnt = 0;
+
+ /* Setup program and fill the LRU map. */
+ skel = setup_prog(BPF_MAP_TYPE_LRU_PERCPU_HASH, &map_fd);
+ if (!ASSERT_OK_PTR(skel, "setup_prog"))
+ return;
+
+ err = fill_values_percpu(map_fd);
+ if (!ASSERT_OK(err, "fill_values_percpu"))
+ goto cleanup;
+
+ /* Insert new element at key=3, should reuse LRU element 1. */
+ key = 3;
+ err = trigger_tp(skel, key, NEW_VALUE);
+ if (!ASSERT_OK(err, "trigger_tp"))
+ goto cleanup;
+
+ /* Clean value. */
+ for (i = 0; i < nr_cpus; i++)
+ value[i] = 0;
+
+ /* Lookup and delete element 3. */
+ err = bpf_map_lookup_and_delete_elem(map_fd, &key, value);
+ if (!ASSERT_OK(err, "bpf_map_lookup_and_delete_elem")) {
+ goto cleanup;
+ }
+
+ /* Check if only one CPU has set the value. */
+ for (i = 0; i < nr_cpus; i++) {
+ val = value[i];
+ if (val) {
+ if (CHECK(val != NEW_VALUE, "map value",
+ "unexpected for cpu %d: %lld\n", i, val))
+ goto cleanup;
+ cpucnt++;
+ }
+ }
+ if (CHECK(cpucnt != 1, "map value", "set for %d CPUs instead of 1!\n",
+ cpucnt))
+ goto cleanup;
+
+ /* Check that entries 3 and 1 are non existent. */
+ err = bpf_map_lookup_elem(map_fd, &key, &value);
+ if (!ASSERT_ERR(err, "bpf_map_lookup_elem"))
+ goto cleanup;
+
+ key = 1;
+ err = bpf_map_lookup_elem(map_fd, &key, &value);
+ if (!ASSERT_ERR(err, "bpf_map_lookup_elem"))
+ goto cleanup;
+
+cleanup:
+ test_lookup_and_delete__destroy(skel);
+}
+
+void test_lookup_and_delete(void)
+{
+ nr_cpus = bpf_num_possible_cpus();
+
+ if (test__start_subtest("lookup_and_delete"))
+ test_lookup_and_delete_hash();
+ if (test__start_subtest("lookup_and_delete_percpu"))
+ test_lookup_and_delete_percpu_hash();
+ if (test__start_subtest("lookup_and_delete_lru"))
+ test_lookup_and_delete_lru_hash();
+ if (test__start_subtest("lookup_and_delete_lru_percpu"))
+ test_lookup_and_delete_lru_percpu_hash();
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Check if we can migrate child sockets.
+ *
+ * 1. call listen() for 4 server sockets.
+ * 2. call connect() for 25 client sockets.
+ * 3. call listen() for 1 server socket. (migration target)
+ * 4. update a map to migrate all child sockets
+ * to the last server socket (migrate_map[cookie] = 4)
+ * 5. call shutdown() for first 4 server sockets
+ * and migrate the requests in the accept queue
+ * to the last server socket.
+ * 6. call listen() for the second server socket.
+ * 7. call shutdown() for the last server
+ * and migrate the requests in the accept queue
+ * to the second server socket.
+ * 8. call listen() for the last server.
+ * 9. call shutdown() for the second server
+ * and migrate the requests in the accept queue
+ * to the last server socket.
+ * 10. call accept() for the last server socket.
+ *
+ * Author: Kuniyuki Iwashima <kuniyu@amazon.co.jp>
+ */
+
+#include <bpf/bpf.h>
+#include <bpf/libbpf.h>
+
+#include "test_progs.h"
+#include "test_migrate_reuseport.skel.h"
+#include "network_helpers.h"
+
+#ifndef TCP_FASTOPEN_CONNECT
+#define TCP_FASTOPEN_CONNECT 30
+#endif
+
+#define IFINDEX_LO 1
+
+#define NR_SERVERS 5
+#define NR_CLIENTS (NR_SERVERS * 5)
+#define MIGRATED_TO (NR_SERVERS - 1)
+
+/* fastopenq->max_qlen and sk->sk_max_ack_backlog */
+#define QLEN (NR_CLIENTS * 5)
+
+#define MSG "Hello World\0"
+#define MSGLEN 12
+
+static struct migrate_reuseport_test_case {
+ const char *name;
+ __s64 servers[NR_SERVERS];
+ __s64 clients[NR_CLIENTS];
+ struct sockaddr_storage addr;
+ socklen_t addrlen;
+ int family;
+ int state;
+ bool drop_ack;
+ bool expire_synack_timer;
+ bool fastopen;
+ struct bpf_link *link;
+} test_cases[] = {
+ {
+ .name = "IPv4 TCP_ESTABLISHED inet_csk_listen_stop",
+ .family = AF_INET,
+ .state = BPF_TCP_ESTABLISHED,
+ .drop_ack = false,
+ .expire_synack_timer = false,
+ .fastopen = false,
+ },
+ {
+ .name = "IPv4 TCP_SYN_RECV inet_csk_listen_stop",
+ .family = AF_INET,
+ .state = BPF_TCP_SYN_RECV,
+ .drop_ack = true,
+ .expire_synack_timer = false,
+ .fastopen = true,
+ },
+ {
+ .name = "IPv4 TCP_NEW_SYN_RECV reqsk_timer_handler",
+ .family = AF_INET,
+ .state = BPF_TCP_NEW_SYN_RECV,
+ .drop_ack = true,
+ .expire_synack_timer = true,
+ .fastopen = false,
+ },
+ {
+ .name = "IPv4 TCP_NEW_SYN_RECV inet_csk_complete_hashdance",
+ .family = AF_INET,
+ .state = BPF_TCP_NEW_SYN_RECV,
+ .drop_ack = true,
+ .expire_synack_timer = false,
+ .fastopen = false,
+ },
+ {
+ .name = "IPv6 TCP_ESTABLISHED inet_csk_listen_stop",
+ .family = AF_INET6,
+ .state = BPF_TCP_ESTABLISHED,
+ .drop_ack = false,
+ .expire_synack_timer = false,
+ .fastopen = false,
+ },
+ {
+ .name = "IPv6 TCP_SYN_RECV inet_csk_listen_stop",
+ .family = AF_INET6,
+ .state = BPF_TCP_SYN_RECV,
+ .drop_ack = true,
+ .expire_synack_timer = false,
+ .fastopen = true,
+ },
+ {
+ .name = "IPv6 TCP_NEW_SYN_RECV reqsk_timer_handler",
+ .family = AF_INET6,
+ .state = BPF_TCP_NEW_SYN_RECV,
+ .drop_ack = true,
+ .expire_synack_timer = true,
+ .fastopen = false,
+ },
+ {
+ .name = "IPv6 TCP_NEW_SYN_RECV inet_csk_complete_hashdance",
+ .family = AF_INET6,
+ .state = BPF_TCP_NEW_SYN_RECV,
+ .drop_ack = true,
+ .expire_synack_timer = false,
+ .fastopen = false,
+ }
+};
+
+static void init_fds(__s64 fds[], int len)
+{
+ int i;
+
+ for (i = 0; i < len; i++)
+ fds[i] = -1;
+}
+
+static void close_fds(__s64 fds[], int len)
+{
+ int i;
+
+ for (i = 0; i < len; i++) {
+ if (fds[i] != -1) {
+ close(fds[i]);
+ fds[i] = -1;
+ }
+ }
+}
+
+static int setup_fastopen(char *buf, int size, int *saved_len, bool restore)
+{
+ int err = 0, fd, len;
+
+ fd = open("/proc/sys/net/ipv4/tcp_fastopen", O_RDWR);
+ if (!ASSERT_NEQ(fd, -1, "open"))
+ return -1;
+
+ if (restore) {
+ len = write(fd, buf, *saved_len);
+ if (!ASSERT_EQ(len, *saved_len, "write - restore"))
+ err = -1;
+ } else {
+ *saved_len = read(fd, buf, size);
+ if (!ASSERT_GE(*saved_len, 1, "read")) {
+ err = -1;
+ goto close;
+ }
+
+ err = lseek(fd, 0, SEEK_SET);
+ if (!ASSERT_OK(err, "lseek"))
+ goto close;
+
+ /* (TFO_CLIENT_ENABLE | TFO_SERVER_ENABLE |
+ * TFO_CLIENT_NO_COOKIE | TFO_SERVER_COOKIE_NOT_REQD)
+ */
+ len = write(fd, "519", 3);
+ if (!ASSERT_EQ(len, 3, "write - setup"))
+ err = -1;
+ }
+
+close:
+ close(fd);
+
+ return err;
+}
+
+static int drop_ack(struct migrate_reuseport_test_case *test_case,
+ struct test_migrate_reuseport *skel)
+{
+ if (test_case->family == AF_INET)
+ skel->bss->server_port = ((struct sockaddr_in *)
+ &test_case->addr)->sin_port;
+ else
+ skel->bss->server_port = ((struct sockaddr_in6 *)
+ &test_case->addr)->sin6_port;
+
+ test_case->link = bpf_program__attach_xdp(skel->progs.drop_ack,
+ IFINDEX_LO);
+ if (!ASSERT_OK_PTR(test_case->link, "bpf_program__attach_xdp"))
+ return -1;
+
+ return 0;
+}
+
+static int pass_ack(struct migrate_reuseport_test_case *test_case)
+{
+ int err;
+
+ err = bpf_link__detach(test_case->link);
+ if (!ASSERT_OK(err, "bpf_link__detach"))
+ return -1;
+
+ test_case->link = NULL;
+
+ return 0;
+}
+
+static int start_servers(struct migrate_reuseport_test_case *test_case,
+ struct test_migrate_reuseport *skel)
+{
+ int i, err, prog_fd, reuseport = 1, qlen = QLEN;
+
+ prog_fd = bpf_program__fd(skel->progs.migrate_reuseport);
+
+ make_sockaddr(test_case->family,
+ test_case->family == AF_INET ? "127.0.0.1" : "::1", 0,
+ &test_case->addr, &test_case->addrlen);
+
+ for (i = 0; i < NR_SERVERS; i++) {
+ test_case->servers[i] = socket(test_case->family, SOCK_STREAM,
+ IPPROTO_TCP);
+ if (!ASSERT_NEQ(test_case->servers[i], -1, "socket"))
+ return -1;
+
+ err = setsockopt(test_case->servers[i], SOL_SOCKET,
+ SO_REUSEPORT, &reuseport, sizeof(reuseport));
+ if (!ASSERT_OK(err, "setsockopt - SO_REUSEPORT"))
+ return -1;
+
+ err = bind(test_case->servers[i],
+ (struct sockaddr *)&test_case->addr,
+ test_case->addrlen);
+ if (!ASSERT_OK(err, "bind"))
+ return -1;
+
+ if (i == 0) {
+ err = setsockopt(test_case->servers[i], SOL_SOCKET,
+ SO_ATTACH_REUSEPORT_EBPF,
+ &prog_fd, sizeof(prog_fd));
+ if (!ASSERT_OK(err,
+ "setsockopt - SO_ATTACH_REUSEPORT_EBPF"))
+ return -1;
+
+ err = getsockname(test_case->servers[i],
+ (struct sockaddr *)&test_case->addr,
+ &test_case->addrlen);
+ if (!ASSERT_OK(err, "getsockname"))
+ return -1;
+ }
+
+ if (test_case->fastopen) {
+ err = setsockopt(test_case->servers[i],
+ SOL_TCP, TCP_FASTOPEN,
+ &qlen, sizeof(qlen));
+ if (!ASSERT_OK(err, "setsockopt - TCP_FASTOPEN"))
+ return -1;
+ }
+
+ /* All requests will be tied to the first four listeners */
+ if (i != MIGRATED_TO) {
+ err = listen(test_case->servers[i], qlen);
+ if (!ASSERT_OK(err, "listen"))
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int start_clients(struct migrate_reuseport_test_case *test_case)
+{
+ char buf[MSGLEN] = MSG;
+ int i, err;
+
+ for (i = 0; i < NR_CLIENTS; i++) {
+ test_case->clients[i] = socket(test_case->family, SOCK_STREAM,
+ IPPROTO_TCP);
+ if (!ASSERT_NEQ(test_case->clients[i], -1, "socket"))
+ return -1;
+
+ /* The attached XDP program drops only the final ACK, so
+ * clients will transition to TCP_ESTABLISHED immediately.
+ */
+ err = settimeo(test_case->clients[i], 100);
+ if (!ASSERT_OK(err, "settimeo"))
+ return -1;
+
+ if (test_case->fastopen) {
+ int fastopen = 1;
+
+ err = setsockopt(test_case->clients[i], IPPROTO_TCP,
+ TCP_FASTOPEN_CONNECT, &fastopen,
+ sizeof(fastopen));
+ if (!ASSERT_OK(err,
+ "setsockopt - TCP_FASTOPEN_CONNECT"))
+ return -1;
+ }
+
+ err = connect(test_case->clients[i],
+ (struct sockaddr *)&test_case->addr,
+ test_case->addrlen);
+ if (!ASSERT_OK(err, "connect"))
+ return -1;
+
+ err = write(test_case->clients[i], buf, MSGLEN);
+ if (!ASSERT_EQ(err, MSGLEN, "write"))
+ return -1;
+ }
+
+ return 0;
+}
+
+static int update_maps(struct migrate_reuseport_test_case *test_case,
+ struct test_migrate_reuseport *skel)
+{
+ int i, err, migrated_to = MIGRATED_TO;
+ int reuseport_map_fd, migrate_map_fd;
+ __u64 value;
+
+ reuseport_map_fd = bpf_map__fd(skel->maps.reuseport_map);
+ migrate_map_fd = bpf_map__fd(skel->maps.migrate_map);
+
+ for (i = 0; i < NR_SERVERS; i++) {
+ value = (__u64)test_case->servers[i];
+ err = bpf_map_update_elem(reuseport_map_fd, &i, &value,
+ BPF_NOEXIST);
+ if (!ASSERT_OK(err, "bpf_map_update_elem - reuseport_map"))
+ return -1;
+
+ err = bpf_map_lookup_elem(reuseport_map_fd, &i, &value);
+ if (!ASSERT_OK(err, "bpf_map_lookup_elem - reuseport_map"))
+ return -1;
+
+ err = bpf_map_update_elem(migrate_map_fd, &value, &migrated_to,
+ BPF_NOEXIST);
+ if (!ASSERT_OK(err, "bpf_map_update_elem - migrate_map"))
+ return -1;
+ }
+
+ return 0;
+}
+
+static int migrate_dance(struct migrate_reuseport_test_case *test_case)
+{
+ int i, err;
+
+ /* Migrate TCP_ESTABLISHED and TCP_SYN_RECV requests
+ * to the last listener based on eBPF.
+ */
+ for (i = 0; i < MIGRATED_TO; i++) {
+ err = shutdown(test_case->servers[i], SHUT_RDWR);
+ if (!ASSERT_OK(err, "shutdown"))
+ return -1;
+ }
+
+ /* No dance for TCP_NEW_SYN_RECV to migrate based on eBPF */
+ if (test_case->state == BPF_TCP_NEW_SYN_RECV)
+ return 0;
+
+ /* Note that we use the second listener instead of the
+ * first one here.
+ *
+ * The fist listener is bind()ed with port 0 and,
+ * SOCK_BINDPORT_LOCK is not set to sk_userlocks, so
+ * calling listen() again will bind() the first listener
+ * on a new ephemeral port and detach it from the existing
+ * reuseport group. (See: __inet_bind(), tcp_set_state())
+ *
+ * OTOH, the second one is bind()ed with a specific port,
+ * and SOCK_BINDPORT_LOCK is set. Thus, re-listen() will
+ * resurrect the listener on the existing reuseport group.
+ */
+ err = listen(test_case->servers[1], QLEN);
+ if (!ASSERT_OK(err, "listen"))
+ return -1;
+
+ /* Migrate from the last listener to the second one.
+ *
+ * All listeners were detached out of the reuseport_map,
+ * so migration will be done by kernel random pick from here.
+ */
+ err = shutdown(test_case->servers[MIGRATED_TO], SHUT_RDWR);
+ if (!ASSERT_OK(err, "shutdown"))
+ return -1;
+
+ /* Back to the existing reuseport group */
+ err = listen(test_case->servers[MIGRATED_TO], QLEN);
+ if (!ASSERT_OK(err, "listen"))
+ return -1;
+
+ /* Migrate back to the last one from the second one */
+ err = shutdown(test_case->servers[1], SHUT_RDWR);
+ if (!ASSERT_OK(err, "shutdown"))
+ return -1;
+
+ return 0;
+}
+
+static void count_requests(struct migrate_reuseport_test_case *test_case,
+ struct test_migrate_reuseport *skel)
+{
+ struct sockaddr_storage addr;
+ socklen_t len = sizeof(addr);
+ int err, cnt = 0, client;
+ char buf[MSGLEN];
+
+ err = settimeo(test_case->servers[MIGRATED_TO], 4000);
+ if (!ASSERT_OK(err, "settimeo"))
+ goto out;
+
+ for (; cnt < NR_CLIENTS; cnt++) {
+ client = accept(test_case->servers[MIGRATED_TO],
+ (struct sockaddr *)&addr, &len);
+ if (!ASSERT_NEQ(client, -1, "accept"))
+ goto out;
+
+ memset(buf, 0, MSGLEN);
+ read(client, &buf, MSGLEN);
+ close(client);
+
+ if (!ASSERT_STREQ(buf, MSG, "read"))
+ goto out;
+ }
+
+out:
+ ASSERT_EQ(cnt, NR_CLIENTS, "count in userspace");
+
+ switch (test_case->state) {
+ case BPF_TCP_ESTABLISHED:
+ cnt = skel->bss->migrated_at_close;
+ break;
+ case BPF_TCP_SYN_RECV:
+ cnt = skel->bss->migrated_at_close_fastopen;
+ break;
+ case BPF_TCP_NEW_SYN_RECV:
+ if (test_case->expire_synack_timer)
+ cnt = skel->bss->migrated_at_send_synack;
+ else
+ cnt = skel->bss->migrated_at_recv_ack;
+ break;
+ default:
+ cnt = 0;
+ }
+
+ ASSERT_EQ(cnt, NR_CLIENTS, "count in BPF prog");
+}
+
+static void run_test(struct migrate_reuseport_test_case *test_case,
+ struct test_migrate_reuseport *skel)
+{
+ int err, saved_len;
+ char buf[16];
+
+ skel->bss->migrated_at_close = 0;
+ skel->bss->migrated_at_close_fastopen = 0;
+ skel->bss->migrated_at_send_synack = 0;
+ skel->bss->migrated_at_recv_ack = 0;
+
+ init_fds(test_case->servers, NR_SERVERS);
+ init_fds(test_case->clients, NR_CLIENTS);
+
+ if (test_case->fastopen) {
+ memset(buf, 0, sizeof(buf));
+
+ err = setup_fastopen(buf, sizeof(buf), &saved_len, false);
+ if (!ASSERT_OK(err, "setup_fastopen - setup"))
+ return;
+ }
+
+ err = start_servers(test_case, skel);
+ if (!ASSERT_OK(err, "start_servers"))
+ goto close_servers;
+
+ if (test_case->drop_ack) {
+ /* Drop the final ACK of the 3-way handshake and stick the
+ * in-flight requests on TCP_SYN_RECV or TCP_NEW_SYN_RECV.
+ */
+ err = drop_ack(test_case, skel);
+ if (!ASSERT_OK(err, "drop_ack"))
+ goto close_servers;
+ }
+
+ /* Tie requests to the first four listners */
+ err = start_clients(test_case);
+ if (!ASSERT_OK(err, "start_clients"))
+ goto close_clients;
+
+ err = listen(test_case->servers[MIGRATED_TO], QLEN);
+ if (!ASSERT_OK(err, "listen"))
+ goto close_clients;
+
+ err = update_maps(test_case, skel);
+ if (!ASSERT_OK(err, "fill_maps"))
+ goto close_clients;
+
+ /* Migrate the requests in the accept queue only.
+ * TCP_NEW_SYN_RECV requests are not migrated at this point.
+ */
+ err = migrate_dance(test_case);
+ if (!ASSERT_OK(err, "migrate_dance"))
+ goto close_clients;
+
+ if (test_case->expire_synack_timer) {
+ /* Wait for SYN+ACK timers to expire so that
+ * reqsk_timer_handler() migrates TCP_NEW_SYN_RECV requests.
+ */
+ sleep(1);
+ }
+
+ if (test_case->link) {
+ /* Resume 3WHS and migrate TCP_NEW_SYN_RECV requests */
+ err = pass_ack(test_case);
+ if (!ASSERT_OK(err, "pass_ack"))
+ goto close_clients;
+ }
+
+ count_requests(test_case, skel);
+
+close_clients:
+ close_fds(test_case->clients, NR_CLIENTS);
+
+ if (test_case->link) {
+ err = pass_ack(test_case);
+ ASSERT_OK(err, "pass_ack - clean up");
+ }
+
+close_servers:
+ close_fds(test_case->servers, NR_SERVERS);
+
+ if (test_case->fastopen) {
+ err = setup_fastopen(buf, sizeof(buf), &saved_len, true);
+ ASSERT_OK(err, "setup_fastopen - restore");
+ }
+}
+
+void test_migrate_reuseport(void)
+{
+ struct test_migrate_reuseport *skel;
+ int i;
+
+ skel = test_migrate_reuseport__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+
+ for (i = 0; i < ARRAY_SIZE(test_cases); i++) {
+ test__start_subtest(test_cases[i].name);
+ run_test(&test_cases[i], skel);
+ }
+
+ test_migrate_reuseport__destroy(skel);
+}
fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, sizeof(attr));
CHECK((tests[i].success && fd < 0) ||
- (!tests[i].success && fd != -1) ||
+ (!tests[i].success && fd >= 0) ||
(!tests[i].success && errno != tests[i].expected_errno),
"check-bpf-prog-name",
"fd %d(%d) errno %d(%d)\n",
fd, tests[i].success, errno, tests[i].expected_errno);
- if (fd != -1)
+ if (fd >= 0)
close(fd);
/* test different attr.map_name during BPF_MAP_CREATE */
memcpy(attr.map_name, tests[i].name, ncopy);
fd = syscall(__NR_bpf, BPF_MAP_CREATE, &attr, sizeof(attr));
CHECK((tests[i].success && fd < 0) ||
- (!tests[i].success && fd != -1) ||
+ (!tests[i].success && fd >= 0) ||
(!tests[i].success && errno != tests[i].expected_errno),
"check-bpf-map-name",
"fd %d(%d) errno %d(%d)\n",
fd, tests[i].success, errno, tests[i].expected_errno);
- if (fd != -1)
+ if (fd >= 0)
close(fd);
}
}
/* attach perf_event */
link = bpf_program__attach_perf_event(skel->progs.perf_branches, perf_fd);
- if (CHECK(IS_ERR(link), "attach_perf_event", "err %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_perf_event"))
goto out_destroy_skel;
/* generate some branches on cpu 0 */
* Some setups don't support branch records (virtual machines, !x86),
* so skip test in this case.
*/
- if (pfd == -1) {
+ if (pfd < 0) {
if (errno == ENOENT || errno == EOPNOTSUPP) {
printf("%s:SKIP:no PERF_SAMPLE_BRANCH_STACK\n",
__func__);
pb_opts.sample_cb = on_sample;
pb_opts.ctx = &cpu_seen;
pb = perf_buffer__new(bpf_map__fd(skel->maps.perf_buf_map), 1, &pb_opts);
- if (CHECK(IS_ERR(pb), "perf_buf__new", "err %ld\n", PTR_ERR(pb)))
+ if (!ASSERT_OK_PTR(pb, "perf_buf__new"))
goto out_close;
CHECK(perf_buffer__epoll_fd(pb) < 0, "epoll_fd",
skel->links.oncpu = bpf_program__attach_perf_event(skel->progs.oncpu,
pmu_fd);
- if (CHECK(IS_ERR(skel->links.oncpu), "attach_perf_event",
- "err %ld\n", PTR_ERR(skel->links.oncpu))) {
+ if (!ASSERT_OK_PTR(skel->links.oncpu, "attach_perf_event")) {
close(pmu_fd);
goto cleanup;
}
static const int zero = 0;
obj = bpf_object__open_file(obj_file, &opts);
- if (CHECK(IS_ERR(obj), "obj_open_file", "err %ld\n", PTR_ERR(obj)))
+ if (!ASSERT_OK_PTR(obj, "obj_open_file"))
return;
kprobe_prog = bpf_object__find_program_by_title(obj, prog_name);
goto cleanup;
kprobe_link = bpf_program__attach(kprobe_prog);
- if (CHECK(IS_ERR(kprobe_link), "attach_kprobe",
- "err %ld\n", PTR_ERR(kprobe_link))) {
- kprobe_link = NULL;
+ if (!ASSERT_OK_PTR(kprobe_link, "attach_kprobe"))
goto cleanup;
- }
memset(&curr, 0, sizeof(curr));
in->sin_family = AF_INET;
tattr.prog_fd = bpf_program__fd(skel->progs.test_pkt_access);
err = bpf_prog_test_run_xattr(&tattr);
- CHECK_ATTR(err != -1 || errno != ENOSPC || tattr.retval, "run",
+ CHECK_ATTR(err >= 0 || errno != ENOSPC || tattr.retval, "run",
"err %d errno %d retval %d\n", err, errno, tattr.retval);
CHECK_ATTR(tattr.data_size_out != sizeof(pkt_v4), "data_size_out",
cleanup:
if (skel)
test_pkt_access__destroy(skel);
- if (stats_fd != -1)
+ if (stats_fd >= 0)
close(stats_fd);
}
/* invalid cpu ID should fail with ENXIO */
opts.cpu = 0xffffffff;
err = bpf_prog_test_run_opts(prog_fd, &opts);
- CHECK(err != -1 || errno != ENXIO,
+ CHECK(err >= 0 || errno != ENXIO,
"test_run_opts_fail",
"should failed with ENXIO\n");
opts.cpu = 1;
opts.flags = 0;
err = bpf_prog_test_run_opts(prog_fd, &opts);
- CHECK(err != -1 || errno != EINVAL,
+ CHECK(err >= 0 || errno != EINVAL,
"test_run_opts_fail",
"should failed with EINVAL\n");
struct bss bss;
obj = bpf_object__open_file(file, NULL);
- if (CHECK(IS_ERR(obj), "obj_open", "err %ld\n", PTR_ERR(obj)))
+ if (!ASSERT_OK_PTR(obj, "obj_open"))
return;
err = bpf_object__load(obj);
goto cleanup;
link = bpf_program__attach_raw_tracepoint(prog, "sys_enter");
- if (CHECK(IS_ERR(link), "attach_prog", "prog '%s', err %ld\n",
- t->prog_name, PTR_ERR(link))) {
- link = NULL;
+ if (!ASSERT_OK_PTR(link, "attach_prog"))
goto cleanup;
- }
/* trigger probe */
usleep(1);
int err = 0;
obj = bpf_object__open_file(file, &open_opts);
- if (CHECK_FAIL(IS_ERR(obj)))
+ if (!ASSERT_OK_PTR(obj, "obj_open_file"))
return;
if (CHECK(strcmp(bpf_object__name(obj), obj_name), "obj_name",
}
for (i = 0; i < ARRAY_SIZE(test_symbols); i++) {
- if (test_symbols[i].id != -1)
+ if (test_symbols[i].id >= 0)
continue;
if (BTF_INFO_KIND(type->info) != test_symbols[i].type)
#include <sys/sysinfo.h>
#include <linux/perf_event.h>
#include <linux/ring_buffer.h>
-#include "test_ringbuf.skel.h"
+#include "test_ringbuf.lskel.h"
#define EDONE 7777
const size_t rec_sz = BPF_RINGBUF_HDR_SZ + sizeof(struct sample);
pthread_t thread;
long bg_ret = -1;
- int err, cnt;
+ int err, cnt, rb_fd;
int page_size = getpagesize();
+ void *mmap_ptr, *tmp_ptr;
skel = test_ringbuf__open();
if (CHECK(!skel, "skel_open", "skeleton open failed\n"))
return;
- err = bpf_map__set_max_entries(skel->maps.ringbuf, page_size);
- if (CHECK(err != 0, "bpf_map__set_max_entries", "bpf_map__set_max_entries failed\n"))
- goto cleanup;
+ skel->maps.ringbuf.max_entries = page_size;
err = test_ringbuf__load(skel);
if (CHECK(err != 0, "skel_load", "skeleton load failed\n"))
goto cleanup;
+ rb_fd = bpf_map__fd(skel->maps.ringbuf);
+ /* good read/write cons_pos */
+ mmap_ptr = mmap(NULL, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, rb_fd, 0);
+ ASSERT_OK_PTR(mmap_ptr, "rw_cons_pos");
+ tmp_ptr = mremap(mmap_ptr, page_size, 2 * page_size, MREMAP_MAYMOVE);
+ if (!ASSERT_ERR_PTR(tmp_ptr, "rw_extend"))
+ goto cleanup;
+ ASSERT_ERR(mprotect(mmap_ptr, page_size, PROT_EXEC), "exec_cons_pos_protect");
+ ASSERT_OK(munmap(mmap_ptr, page_size), "unmap_rw");
+
+ /* bad writeable prod_pos */
+ mmap_ptr = mmap(NULL, page_size, PROT_WRITE, MAP_SHARED, rb_fd, page_size);
+ err = -errno;
+ ASSERT_ERR_PTR(mmap_ptr, "wr_prod_pos");
+ ASSERT_EQ(err, -EPERM, "wr_prod_pos_err");
+
+ /* bad writeable data pages */
+ mmap_ptr = mmap(NULL, page_size, PROT_WRITE, MAP_SHARED, rb_fd, 2 * page_size);
+ err = -errno;
+ ASSERT_ERR_PTR(mmap_ptr, "wr_data_page_one");
+ ASSERT_EQ(err, -EPERM, "wr_data_page_one_err");
+ mmap_ptr = mmap(NULL, page_size, PROT_WRITE, MAP_SHARED, rb_fd, 3 * page_size);
+ ASSERT_ERR_PTR(mmap_ptr, "wr_data_page_two");
+ mmap_ptr = mmap(NULL, 2 * page_size, PROT_WRITE, MAP_SHARED, rb_fd, 2 * page_size);
+ ASSERT_ERR_PTR(mmap_ptr, "wr_data_page_all");
+
+ /* good read-only pages */
+ mmap_ptr = mmap(NULL, 4 * page_size, PROT_READ, MAP_SHARED, rb_fd, 0);
+ if (!ASSERT_OK_PTR(mmap_ptr, "ro_prod_pos"))
+ goto cleanup;
+
+ ASSERT_ERR(mprotect(mmap_ptr, 4 * page_size, PROT_WRITE), "write_protect");
+ ASSERT_ERR(mprotect(mmap_ptr, 4 * page_size, PROT_EXEC), "exec_protect");
+ ASSERT_ERR_PTR(mremap(mmap_ptr, 0, 4 * page_size, MREMAP_MAYMOVE), "ro_remap");
+ ASSERT_OK(munmap(mmap_ptr, 4 * page_size), "unmap_ro");
+
+ /* good read-only pages with initial offset */
+ mmap_ptr = mmap(NULL, page_size, PROT_READ, MAP_SHARED, rb_fd, page_size);
+ if (!ASSERT_OK_PTR(mmap_ptr, "ro_prod_pos"))
+ goto cleanup;
+
+ ASSERT_ERR(mprotect(mmap_ptr, page_size, PROT_WRITE), "write_protect");
+ ASSERT_ERR(mprotect(mmap_ptr, page_size, PROT_EXEC), "exec_protect");
+ ASSERT_ERR_PTR(mremap(mmap_ptr, 0, 3 * page_size, MREMAP_MAYMOVE), "ro_remap");
+ ASSERT_OK(munmap(mmap_ptr, page_size), "unmap_ro");
+
/* only trigger BPF program for current process */
skel->bss->pid = getpid();
- ringbuf = ring_buffer__new(bpf_map__fd(skel->maps.ringbuf),
+ ringbuf = ring_buffer__new(skel->maps.ringbuf.map_fd,
process_sample, NULL, NULL);
if (CHECK(!ringbuf, "ringbuf_create", "failed to create ringbuf\n"))
goto cleanup;
goto cleanup;
proto_fd = bpf_create_map(BPF_MAP_TYPE_RINGBUF, 0, 0, page_size, 0);
- if (CHECK(proto_fd == -1, "bpf_create_map", "bpf_create_map failed\n"))
+ if (CHECK(proto_fd < 0, "bpf_create_map", "bpf_create_map failed\n"))
goto cleanup;
err = bpf_map__set_inner_map_fd(skel->maps.ringbuf_hash, proto_fd);
attr.max_entries = REUSEPORT_ARRAY_SIZE;
reuseport_array = bpf_create_map_xattr(&attr);
- RET_ERR(reuseport_array == -1, "creating reuseport_array",
+ RET_ERR(reuseport_array < 0, "creating reuseport_array",
"reuseport_array:%d errno:%d\n", reuseport_array, errno);
/* Creating outer_map */
attr.max_entries = 1;
attr.inner_map_fd = reuseport_array;
outer_map = bpf_create_map_xattr(&attr);
- RET_ERR(outer_map == -1, "creating outer_map",
+ RET_ERR(outer_map < 0, "creating outer_map",
"outer_map:%d errno:%d\n", outer_map, errno);
return 0;
int err;
obj = bpf_object__open("test_select_reuseport_kern.o");
- RET_ERR(IS_ERR_OR_NULL(obj), "open test_select_reuseport_kern.o",
- "obj:%p PTR_ERR(obj):%ld\n", obj, PTR_ERR(obj));
+ err = libbpf_get_error(obj);
+ RET_ERR(err, "open test_select_reuseport_kern.o",
+ "obj:%p PTR_ERR(obj):%d\n", obj, err);
map = bpf_object__find_map_by_name(obj, "outer_map");
RET_ERR(!map, "find outer_map", "!map\n");
prog = bpf_program__next(NULL, obj);
RET_ERR(!prog, "get first bpf_program", "!prog\n");
select_by_skb_data_prog = bpf_program__fd(prog);
- RET_ERR(select_by_skb_data_prog == -1, "get prog fd",
+ RET_ERR(select_by_skb_data_prog < 0, "get prog fd",
"select_by_skb_data_prog:%d\n", select_by_skb_data_prog);
map = bpf_object__find_map_by_name(obj, "result_map");
RET_ERR(!map, "find result_map", "!map\n");
result_map = bpf_map__fd(map);
- RET_ERR(result_map == -1, "get result_map fd",
+ RET_ERR(result_map < 0, "get result_map fd",
"result_map:%d\n", result_map);
map = bpf_object__find_map_by_name(obj, "tmp_index_ovr_map");
RET_ERR(!map, "find tmp_index_ovr_map\n", "!map");
tmp_index_ovr_map = bpf_map__fd(map);
- RET_ERR(tmp_index_ovr_map == -1, "get tmp_index_ovr_map fd",
+ RET_ERR(tmp_index_ovr_map < 0, "get tmp_index_ovr_map fd",
"tmp_index_ovr_map:%d\n", tmp_index_ovr_map);
map = bpf_object__find_map_by_name(obj, "linum_map");
RET_ERR(!map, "find linum_map", "!map\n");
linum_map = bpf_map__fd(map);
- RET_ERR(linum_map == -1, "get linum_map fd",
+ RET_ERR(linum_map < 0, "get linum_map fd",
"linum_map:%d\n", linum_map);
map = bpf_object__find_map_by_name(obj, "data_check_map");
RET_ERR(!map, "find data_check_map", "!map\n");
data_check_map = bpf_map__fd(map);
- RET_ERR(data_check_map == -1, "get data_check_map fd",
+ RET_ERR(data_check_map < 0, "get data_check_map fd",
"data_check_map:%d\n", data_check_map);
return 0;
int err;
err = bpf_map_lookup_elem(linum_map, &index_zero, &linum);
- RET_ERR(err == -1, "lookup_elem(linum_map)", "err:%d errno:%d\n",
+ RET_ERR(err < 0, "lookup_elem(linum_map)", "err:%d errno:%d\n",
err, errno);
return linum;
addrlen = sizeof(cli_sa);
err = getsockname(cli_fd, (struct sockaddr *)&cli_sa,
&addrlen);
- RET_IF(err == -1, "getsockname(cli_fd)", "err:%d errno:%d\n",
+ RET_IF(err < 0, "getsockname(cli_fd)", "err:%d errno:%d\n",
err, errno);
err = bpf_map_lookup_elem(data_check_map, &index_zero, &result);
- RET_IF(err == -1, "lookup_elem(data_check_map)", "err:%d errno:%d\n",
+ RET_IF(err < 0, "lookup_elem(data_check_map)", "err:%d errno:%d\n",
err, errno);
if (type == SOCK_STREAM) {
for (i = 0; i < NR_RESULTS; i++) {
err = bpf_map_lookup_elem(result_map, &i, &results[i]);
- RET_IF(err == -1, "lookup_elem(result_map)",
+ RET_IF(err < 0, "lookup_elem(result_map)",
"i:%u err:%d errno:%d\n", i, err, errno);
}
*/
err = bpf_map_update_elem(tmp_index_ovr_map, &index_zero,
&tmp_index, BPF_ANY);
- RET_IF(err == -1, "update_elem(tmp_index_ovr_map, 0, 1)",
+ RET_IF(err < 0, "update_elem(tmp_index_ovr_map, 0, 1)",
"err:%d errno:%d\n", err, errno);
do_test(type, family, &cmd, PASS);
err = bpf_map_lookup_elem(tmp_index_ovr_map, &index_zero,
&tmp_index);
- RET_IF(err == -1 || tmp_index != -1,
+ RET_IF(err < 0 || tmp_index >= 0,
"lookup_elem(tmp_index_ovr_map)",
"err:%d errno:%d tmp_index:%d\n",
err, errno, tmp_index);
for (i = 0; i < NR_RESULTS; i++) {
err = bpf_map_lookup_elem(result_map, &i, &tmp);
- RET_IF(err == -1, "lookup_elem(result_map)",
+ RET_IF(err < 0, "lookup_elem(result_map)",
"i:%u err:%d errno:%d\n", i, err, errno);
nr_run_before += tmp;
}
for (i = 0; i < NR_RESULTS; i++) {
err = bpf_map_lookup_elem(result_map, &i, &tmp);
- RET_IF(err == -1, "lookup_elem(result_map)",
+ RET_IF(err < 0, "lookup_elem(result_map)",
"i:%u err:%d errno:%d\n", i, err, errno);
nr_run_after += tmp;
}
SO_ATTACH_REUSEPORT_EBPF,
&select_by_skb_data_prog,
sizeof(select_by_skb_data_prog));
- RET_IF(err == -1, "setsockopt(SO_ATTACH_REUEPORT_EBPF)",
+ RET_IF(err < 0, "setsockopt(SO_ATTACH_REUEPORT_EBPF)",
"err:%d errno:%d\n", err, errno);
}
err = bind(sk_fds[i], (struct sockaddr *)&srv_sa, addrlen);
- RET_IF(err == -1, "bind()", "sk_fds[%d] err:%d errno:%d\n",
+ RET_IF(err < 0, "bind()", "sk_fds[%d] err:%d errno:%d\n",
i, err, errno);
if (type == SOCK_STREAM) {
err = listen(sk_fds[i], 10);
- RET_IF(err == -1, "listen()",
+ RET_IF(err < 0, "listen()",
"sk_fds[%d] err:%d errno:%d\n",
i, err, errno);
}
err = bpf_map_update_elem(reuseport_array, &i, &sk_fds[i],
BPF_NOEXIST);
- RET_IF(err == -1, "update_elem(reuseport_array)",
+ RET_IF(err < 0, "update_elem(reuseport_array)",
"sk_fds[%d] err:%d errno:%d\n", i, err, errno);
if (i == first) {
prepare_sk_fds(type, family, inany);
err = bpf_map_update_elem(tmp_index_ovr_map, &index_zero, &ovr,
BPF_ANY);
- RET_IF(err == -1, "update_elem(tmp_index_ovr_map, 0, -1)",
+ RET_IF(err < 0, "update_elem(tmp_index_ovr_map, 0, -1)",
"err:%d errno:%d\n", err, errno);
/* Install reuseport_array to outer_map? */
err = bpf_map_update_elem(outer_map, &index_zero, &reuseport_array,
BPF_ANY);
- RET_IF(err == -1, "update_elem(outer_map, 0, reuseport_array)",
+ RET_IF(err < 0, "update_elem(outer_map, 0, reuseport_array)",
"err:%d errno:%d\n", err, errno);
}
return;
err = bpf_map_delete_elem(outer_map, &index_zero);
- RET_IF(err == -1, "delete_elem(outer_map)",
+ RET_IF(err < 0, "delete_elem(outer_map)",
"err:%d errno:%d\n", err, errno);
}
static void cleanup(void)
{
- if (outer_map != -1) {
+ if (outer_map >= 0) {
close(outer_map);
outer_map = -1;
}
- if (reuseport_array != -1) {
+ if (reuseport_array >= 0) {
close(reuseport_array);
reuseport_array = -1;
}
#include <test_progs.h>
#include "test_send_signal_kern.skel.h"
-static volatile int sigusr1_received = 0;
+int sigusr1_received = 0;
static void sigusr1_handler(int signum)
{
skel->links.send_signal_perf =
bpf_program__attach_perf_event(skel->progs.send_signal_perf, pmu_fd);
- if (CHECK(IS_ERR(skel->links.send_signal_perf), "attach_perf_event",
- "err %ld\n", PTR_ERR(skel->links.send_signal_perf)))
+ if (!ASSERT_OK_PTR(skel->links.send_signal_perf, "attach_perf_event"))
goto disable_pmu;
}
}
link = bpf_program__attach_netns(prog, net_fd);
- if (CHECK(IS_ERR(link), "bpf_program__attach_netns", "failed\n")) {
+ if (!ASSERT_OK_PTR(link, "bpf_program__attach_netns")) {
errno = -PTR_ERR(link);
log_err("failed to attach program '%s' to netns",
bpf_program__name(prog));
CHECK(data->out2 != 2, "res2", "got %lld != exp %d\n", data->out2, 2);
CHECK(bss->out3 != 3, "res3", "got %d != exp %d\n", (int)bss->out3, 3);
CHECK(bss->out4 != 4, "res4", "got %lld != exp %d\n", bss->out4, 4);
- CHECK(bss->handler_out5.a != 5, "res5", "got %d != exp %d\n",
- bss->handler_out5.a, 5);
- CHECK(bss->handler_out5.b != 6, "res6", "got %lld != exp %d\n",
- bss->handler_out5.b, 6);
+ CHECK(bss->out5.a != 5, "res5", "got %d != exp %d\n", bss->out5.a, 5);
+ CHECK(bss->out5.b != 6, "res6", "got %lld != exp %d\n", bss->out5.b, 6);
CHECK(bss->out6 != 14, "res7", "got %d != exp %d\n", bss->out6, 14);
CHECK(bss->bpf_syscall != kcfg->CONFIG_BPF_SYSCALL, "ext1",
err = bpf_map_lookup_elem(linum_map_fd, &egress_linum_idx,
&egress_linum);
- CHECK(err == -1, "bpf_map_lookup_elem(linum_map_fd)",
+ CHECK(err < 0, "bpf_map_lookup_elem(linum_map_fd)",
"err:%d errno:%d\n", err, errno);
err = bpf_map_lookup_elem(linum_map_fd, &ingress_linum_idx,
&ingress_linum);
- CHECK(err == -1, "bpf_map_lookup_elem(linum_map_fd)",
+ CHECK(err < 0, "bpf_map_lookup_elem(linum_map_fd)",
"err:%d errno:%d\n", err, errno);
memcpy(&srv_sk, &skel->bss->srv_sk, sizeof(srv_sk));
egress_link = bpf_program__attach_cgroup(skel->progs.egress_read_sock_fields,
child_cg_fd);
- if (CHECK(IS_ERR(egress_link), "attach_cgroup(egress)", "err:%ld\n",
- PTR_ERR(egress_link)))
+ if (!ASSERT_OK_PTR(egress_link, "attach_cgroup(egress)"))
goto done;
ingress_link = bpf_program__attach_cgroup(skel->progs.ingress_read_sock_fields,
child_cg_fd);
- if (CHECK(IS_ERR(ingress_link), "attach_cgroup(ingress)", "err:%ld\n",
- PTR_ERR(ingress_link)))
+ if (!ASSERT_OK_PTR(ingress_link, "attach_cgroup(ingress)"))
goto done;
linum_map_fd = bpf_map__fd(skel->maps.linum_map);
bpf_link__destroy(egress_link);
bpf_link__destroy(ingress_link);
test_sock_fields__destroy(skel);
- if (child_cg_fd != -1)
+ if (child_cg_fd >= 0)
close(child_cg_fd);
- if (parent_cg_fd != -1)
+ if (parent_cg_fd >= 0)
close(parent_cg_fd);
}
int s, map, err;
s = connected_socket_v4();
- if (CHECK_FAIL(s == -1))
+ if (CHECK_FAIL(s < 0))
return;
map = bpf_create_map(map_type, sizeof(int), sizeof(int), 1, 0);
- if (CHECK_FAIL(map == -1)) {
+ if (CHECK_FAIL(map < 0)) {
perror("bpf_create_map");
goto out;
}
opts.link_info = &linfo;
opts.link_info_len = sizeof(linfo);
link = bpf_program__attach_iter(skel->progs.copy, &opts);
- if (CHECK(IS_ERR(link), "attach_iter", "attach_iter failed\n"))
+ if (!ASSERT_OK_PTR(link, "attach_iter"))
goto out;
iter_fd = bpf_iter_create(bpf_link__fd(link));
}
err = bpf_prog_attach(verdict, map, second, 0);
- assert(err == -1 && errno == EBUSY);
+ ASSERT_EQ(err, -EBUSY, "prog_attach_fail");
err = bpf_prog_detach2(verdict, map, first);
if (CHECK_FAIL(err)) {
int map;
map = bpf_create_map(map_type, sizeof(int), sizeof(int), 1, 0);
- if (CHECK_FAIL(map == -1)) {
+ if (CHECK_FAIL(map < 0)) {
perror("bpf_map_create");
return;
}
#define xbpf_map_delete_elem(fd, key) \
({ \
int __ret = bpf_map_delete_elem((fd), (key)); \
- if (__ret == -1) \
+ if (__ret < 0) \
FAIL_ERRNO("map_delete"); \
__ret; \
})
#define xbpf_map_lookup_elem(fd, key, val) \
({ \
int __ret = bpf_map_lookup_elem((fd), (key), (val)); \
- if (__ret == -1) \
+ if (__ret < 0) \
FAIL_ERRNO("map_lookup"); \
__ret; \
})
#define xbpf_map_update_elem(fd, key, val, flags) \
({ \
int __ret = bpf_map_update_elem((fd), (key), (val), (flags)); \
- if (__ret == -1) \
+ if (__ret < 0) \
FAIL_ERRNO("map_update"); \
__ret; \
})
({ \
int __ret = \
bpf_prog_attach((prog), (target), (type), (flags)); \
- if (__ret == -1) \
+ if (__ret < 0) \
FAIL_ERRNO("prog_attach(" #type ")"); \
__ret; \
})
#define xbpf_prog_detach2(prog, target, type) \
({ \
int __ret = bpf_prog_detach2((prog), (target), (type)); \
- if (__ret == -1) \
+ if (__ret < 0) \
FAIL_ERRNO("prog_detach2(" #type ")"); \
__ret; \
})
skel->links.oncpu = bpf_program__attach_perf_event(skel->progs.oncpu,
pmu_fd);
- if (CHECK(IS_ERR(skel->links.oncpu), "attach_perf_event",
- "err %ld\n", PTR_ERR(skel->links.oncpu))) {
+ if (!ASSERT_OK_PTR(skel->links.oncpu, "attach_perf_event")) {
close(pmu_fd);
goto cleanup;
}
goto close_prog;
link = bpf_program__attach_tracepoint(prog, "sched", "sched_switch");
- if (CHECK(IS_ERR(link), "attach_tp", "err %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_tp"))
goto close_prog;
/* find map fds */
goto close_prog;
link = bpf_program__attach_raw_tracepoint(prog, "sched_switch");
- if (CHECK(IS_ERR(link), "attach_raw_tp", "err %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_raw_tp"))
goto close_prog;
/* find map fds */
goto close_prog;
close_prog:
- if (!IS_ERR_OR_NULL(link))
- bpf_link__destroy(link);
+ bpf_link__destroy(link);
bpf_object__close(obj);
}
return;
skel->rodata->rovar1 = 1;
- skel->bss->static_var1 = 2;
- skel->bss->static_var11 = 3;
-
skel->rodata->rovar2 = 4;
- skel->bss->static_var2 = 5;
- skel->bss->static_var22 = 6;
err = test_static_linked__load(skel);
if (!ASSERT_OK(err, "skel_load"))
/* trigger */
usleep(1);
- ASSERT_EQ(skel->bss->var1, 1 * 2 + 2 + 3, "var1");
- ASSERT_EQ(skel->bss->var2, 4 * 3 + 5 + 6, "var2");
+ ASSERT_EQ(skel->data->var1, 1 * 2 + 2 + 3, "var1");
+ ASSERT_EQ(skel->data->var2, 4 * 3 + 5 + 6, "var2");
cleanup:
test_static_linked__destroy(skel);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2021 Facebook */
+#include <test_progs.h>
+#include "syscall.skel.h"
+
+struct args {
+ __u64 log_buf;
+ __u32 log_size;
+ int max_entries;
+ int map_fd;
+ int prog_fd;
+ int btf_fd;
+};
+
+void test_syscall(void)
+{
+ static char verifier_log[8192];
+ struct args ctx = {
+ .max_entries = 1024,
+ .log_buf = (uintptr_t) verifier_log,
+ .log_size = sizeof(verifier_log),
+ };
+ struct bpf_prog_test_run_attr tattr = {
+ .ctx_in = &ctx,
+ .ctx_size_in = sizeof(ctx),
+ };
+ struct syscall *skel = NULL;
+ __u64 key = 12, value = 0;
+ int err;
+
+ skel = syscall__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "skel_load"))
+ goto cleanup;
+
+ tattr.prog_fd = bpf_program__fd(skel->progs.bpf_prog);
+ err = bpf_prog_test_run_xattr(&tattr);
+ ASSERT_EQ(err, 0, "err");
+ ASSERT_EQ(tattr.retval, 1, "retval");
+ ASSERT_GT(ctx.map_fd, 0, "ctx.map_fd");
+ ASSERT_GT(ctx.prog_fd, 0, "ctx.prog_fd");
+ ASSERT_OK(memcmp(verifier_log, "processed", sizeof("processed") - 1),
+ "verifier_log");
+
+ err = bpf_map_lookup_elem(ctx.map_fd, &key, &value);
+ ASSERT_EQ(err, 0, "map_lookup");
+ ASSERT_EQ(value, 34, "map lookup value");
+cleanup:
+ syscall__destroy(skel);
+ if (ctx.prog_fd > 0)
+ close(ctx.prog_fd);
+ if (ctx.map_fd > 0)
+ close(ctx.map_fd);
+ if (ctx.btf_fd > 0)
+ close(ctx.btf_fd);
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+
+#include <test_progs.h>
+#include <linux/pkt_cls.h>
+
+#include "test_tc_bpf.skel.h"
+
+#define LO_IFINDEX 1
+
+#define TEST_DECLARE_OPTS(__fd) \
+ DECLARE_LIBBPF_OPTS(bpf_tc_opts, opts_h, .handle = 1); \
+ DECLARE_LIBBPF_OPTS(bpf_tc_opts, opts_p, .priority = 1); \
+ DECLARE_LIBBPF_OPTS(bpf_tc_opts, opts_f, .prog_fd = __fd); \
+ DECLARE_LIBBPF_OPTS(bpf_tc_opts, opts_hp, .handle = 1, .priority = 1); \
+ DECLARE_LIBBPF_OPTS(bpf_tc_opts, opts_hf, .handle = 1, .prog_fd = __fd); \
+ DECLARE_LIBBPF_OPTS(bpf_tc_opts, opts_pf, .priority = 1, .prog_fd = __fd); \
+ DECLARE_LIBBPF_OPTS(bpf_tc_opts, opts_hpf, .handle = 1, .priority = 1, .prog_fd = __fd); \
+ DECLARE_LIBBPF_OPTS(bpf_tc_opts, opts_hpi, .handle = 1, .priority = 1, .prog_id = 42); \
+ DECLARE_LIBBPF_OPTS(bpf_tc_opts, opts_hpr, .handle = 1, .priority = 1, \
+ .flags = BPF_TC_F_REPLACE); \
+ DECLARE_LIBBPF_OPTS(bpf_tc_opts, opts_hpfi, .handle = 1, .priority = 1, .prog_fd = __fd, \
+ .prog_id = 42); \
+ DECLARE_LIBBPF_OPTS(bpf_tc_opts, opts_prio_max, .handle = 1, .priority = UINT16_MAX + 1);
+
+static int test_tc_bpf_basic(const struct bpf_tc_hook *hook, int fd)
+{
+ DECLARE_LIBBPF_OPTS(bpf_tc_opts, opts, .handle = 1, .priority = 1, .prog_fd = fd);
+ struct bpf_prog_info info = {};
+ __u32 info_len = sizeof(info);
+ int ret;
+
+ ret = bpf_obj_get_info_by_fd(fd, &info, &info_len);
+ if (!ASSERT_OK(ret, "bpf_obj_get_info_by_fd"))
+ return ret;
+
+ ret = bpf_tc_attach(hook, &opts);
+ if (!ASSERT_OK(ret, "bpf_tc_attach"))
+ return ret;
+
+ if (!ASSERT_EQ(opts.handle, 1, "handle set") ||
+ !ASSERT_EQ(opts.priority, 1, "priority set") ||
+ !ASSERT_EQ(opts.prog_id, info.id, "prog_id set"))
+ goto end;
+
+ opts.prog_id = 0;
+ opts.flags = BPF_TC_F_REPLACE;
+ ret = bpf_tc_attach(hook, &opts);
+ if (!ASSERT_OK(ret, "bpf_tc_attach replace mode"))
+ goto end;
+
+ opts.flags = opts.prog_fd = opts.prog_id = 0;
+ ret = bpf_tc_query(hook, &opts);
+ if (!ASSERT_OK(ret, "bpf_tc_query"))
+ goto end;
+
+ if (!ASSERT_EQ(opts.handle, 1, "handle set") ||
+ !ASSERT_EQ(opts.priority, 1, "priority set") ||
+ !ASSERT_EQ(opts.prog_id, info.id, "prog_id set"))
+ goto end;
+
+end:
+ opts.flags = opts.prog_fd = opts.prog_id = 0;
+ ret = bpf_tc_detach(hook, &opts);
+ ASSERT_OK(ret, "bpf_tc_detach");
+ return ret;
+}
+
+static int test_tc_bpf_api(struct bpf_tc_hook *hook, int fd)
+{
+ DECLARE_LIBBPF_OPTS(bpf_tc_opts, attach_opts, .handle = 1, .priority = 1, .prog_fd = fd);
+ DECLARE_LIBBPF_OPTS(bpf_tc_hook, inv_hook, .attach_point = BPF_TC_INGRESS);
+ DECLARE_LIBBPF_OPTS(bpf_tc_opts, opts, .handle = 1, .priority = 1);
+ int ret;
+
+ ret = bpf_tc_hook_create(NULL);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_hook_create invalid hook = NULL"))
+ return -EINVAL;
+
+ /* hook ifindex = 0 */
+ ret = bpf_tc_hook_create(&inv_hook);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_hook_create invalid hook ifindex == 0"))
+ return -EINVAL;
+
+ ret = bpf_tc_hook_destroy(&inv_hook);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_hook_destroy invalid hook ifindex == 0"))
+ return -EINVAL;
+
+ ret = bpf_tc_attach(&inv_hook, &attach_opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_attach invalid hook ifindex == 0"))
+ return -EINVAL;
+ attach_opts.prog_id = 0;
+
+ ret = bpf_tc_detach(&inv_hook, &opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_detach invalid hook ifindex == 0"))
+ return -EINVAL;
+
+ ret = bpf_tc_query(&inv_hook, &opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_query invalid hook ifindex == 0"))
+ return -EINVAL;
+
+ /* hook ifindex < 0 */
+ inv_hook.ifindex = -1;
+
+ ret = bpf_tc_hook_create(&inv_hook);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_hook_create invalid hook ifindex < 0"))
+ return -EINVAL;
+
+ ret = bpf_tc_hook_destroy(&inv_hook);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_hook_destroy invalid hook ifindex < 0"))
+ return -EINVAL;
+
+ ret = bpf_tc_attach(&inv_hook, &attach_opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_attach invalid hook ifindex < 0"))
+ return -EINVAL;
+ attach_opts.prog_id = 0;
+
+ ret = bpf_tc_detach(&inv_hook, &opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_detach invalid hook ifindex < 0"))
+ return -EINVAL;
+
+ ret = bpf_tc_query(&inv_hook, &opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_query invalid hook ifindex < 0"))
+ return -EINVAL;
+
+ inv_hook.ifindex = LO_IFINDEX;
+
+ /* hook.attach_point invalid */
+ inv_hook.attach_point = 0xabcd;
+ ret = bpf_tc_hook_create(&inv_hook);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_hook_create invalid hook.attach_point"))
+ return -EINVAL;
+
+ ret = bpf_tc_hook_destroy(&inv_hook);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_hook_destroy invalid hook.attach_point"))
+ return -EINVAL;
+
+ ret = bpf_tc_attach(&inv_hook, &attach_opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_attach invalid hook.attach_point"))
+ return -EINVAL;
+
+ ret = bpf_tc_detach(&inv_hook, &opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_detach invalid hook.attach_point"))
+ return -EINVAL;
+
+ ret = bpf_tc_query(&inv_hook, &opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_query invalid hook.attach_point"))
+ return -EINVAL;
+
+ inv_hook.attach_point = BPF_TC_INGRESS;
+
+ /* hook.attach_point valid, but parent invalid */
+ inv_hook.parent = TC_H_MAKE(1UL << 16, 10);
+ ret = bpf_tc_hook_create(&inv_hook);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_hook_create invalid hook parent"))
+ return -EINVAL;
+
+ ret = bpf_tc_hook_destroy(&inv_hook);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_hook_destroy invalid hook parent"))
+ return -EINVAL;
+
+ ret = bpf_tc_attach(&inv_hook, &attach_opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_attach invalid hook parent"))
+ return -EINVAL;
+
+ ret = bpf_tc_detach(&inv_hook, &opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_detach invalid hook parent"))
+ return -EINVAL;
+
+ ret = bpf_tc_query(&inv_hook, &opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_query invalid hook parent"))
+ return -EINVAL;
+
+ inv_hook.attach_point = BPF_TC_CUSTOM;
+ inv_hook.parent = 0;
+ /* These return EOPNOTSUPP instead of EINVAL as parent is checked after
+ * attach_point of the hook.
+ */
+ ret = bpf_tc_hook_create(&inv_hook);
+ if (!ASSERT_EQ(ret, -EOPNOTSUPP, "bpf_tc_hook_create invalid hook parent"))
+ return -EINVAL;
+
+ ret = bpf_tc_hook_destroy(&inv_hook);
+ if (!ASSERT_EQ(ret, -EOPNOTSUPP, "bpf_tc_hook_destroy invalid hook parent"))
+ return -EINVAL;
+
+ ret = bpf_tc_attach(&inv_hook, &attach_opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_attach invalid hook parent"))
+ return -EINVAL;
+
+ ret = bpf_tc_detach(&inv_hook, &opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_detach invalid hook parent"))
+ return -EINVAL;
+
+ ret = bpf_tc_query(&inv_hook, &opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_query invalid hook parent"))
+ return -EINVAL;
+
+ inv_hook.attach_point = BPF_TC_INGRESS;
+
+ /* detach */
+ {
+ TEST_DECLARE_OPTS(fd);
+
+ ret = bpf_tc_detach(NULL, &opts_hp);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_detach invalid hook = NULL"))
+ return -EINVAL;
+
+ ret = bpf_tc_detach(hook, NULL);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_detach invalid opts = NULL"))
+ return -EINVAL;
+
+ ret = bpf_tc_detach(hook, &opts_hpr);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_detach invalid flags set"))
+ return -EINVAL;
+
+ ret = bpf_tc_detach(hook, &opts_hpf);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_detach invalid prog_fd set"))
+ return -EINVAL;
+
+ ret = bpf_tc_detach(hook, &opts_hpi);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_detach invalid prog_id set"))
+ return -EINVAL;
+
+ ret = bpf_tc_detach(hook, &opts_p);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_detach invalid handle unset"))
+ return -EINVAL;
+
+ ret = bpf_tc_detach(hook, &opts_h);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_detach invalid priority unset"))
+ return -EINVAL;
+
+ ret = bpf_tc_detach(hook, &opts_prio_max);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_detach invalid priority > UINT16_MAX"))
+ return -EINVAL;
+ }
+
+ /* query */
+ {
+ TEST_DECLARE_OPTS(fd);
+
+ ret = bpf_tc_query(NULL, &opts);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_query invalid hook = NULL"))
+ return -EINVAL;
+
+ ret = bpf_tc_query(hook, NULL);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_query invalid opts = NULL"))
+ return -EINVAL;
+
+ ret = bpf_tc_query(hook, &opts_hpr);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_query invalid flags set"))
+ return -EINVAL;
+
+ ret = bpf_tc_query(hook, &opts_hpf);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_query invalid prog_fd set"))
+ return -EINVAL;
+
+ ret = bpf_tc_query(hook, &opts_hpi);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_query invalid prog_id set"))
+ return -EINVAL;
+
+ ret = bpf_tc_query(hook, &opts_p);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_query invalid handle unset"))
+ return -EINVAL;
+
+ ret = bpf_tc_query(hook, &opts_h);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_query invalid priority unset"))
+ return -EINVAL;
+
+ ret = bpf_tc_query(hook, &opts_prio_max);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_query invalid priority > UINT16_MAX"))
+ return -EINVAL;
+
+ /* when chain is not present, kernel returns -EINVAL */
+ ret = bpf_tc_query(hook, &opts_hp);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_query valid handle, priority set"))
+ return -EINVAL;
+ }
+
+ /* attach */
+ {
+ TEST_DECLARE_OPTS(fd);
+
+ ret = bpf_tc_attach(NULL, &opts_hp);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_attach invalid hook = NULL"))
+ return -EINVAL;
+
+ ret = bpf_tc_attach(hook, NULL);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_attach invalid opts = NULL"))
+ return -EINVAL;
+
+ opts_hp.flags = 42;
+ ret = bpf_tc_attach(hook, &opts_hp);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_attach invalid flags"))
+ return -EINVAL;
+
+ ret = bpf_tc_attach(hook, NULL);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_attach invalid prog_fd unset"))
+ return -EINVAL;
+
+ ret = bpf_tc_attach(hook, &opts_hpi);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_attach invalid prog_id set"))
+ return -EINVAL;
+
+ ret = bpf_tc_attach(hook, &opts_pf);
+ if (!ASSERT_OK(ret, "bpf_tc_attach valid handle unset"))
+ return -EINVAL;
+ opts_pf.prog_fd = opts_pf.prog_id = 0;
+ ASSERT_OK(bpf_tc_detach(hook, &opts_pf), "bpf_tc_detach");
+
+ ret = bpf_tc_attach(hook, &opts_hf);
+ if (!ASSERT_OK(ret, "bpf_tc_attach valid priority unset"))
+ return -EINVAL;
+ opts_hf.prog_fd = opts_hf.prog_id = 0;
+ ASSERT_OK(bpf_tc_detach(hook, &opts_hf), "bpf_tc_detach");
+
+ ret = bpf_tc_attach(hook, &opts_prio_max);
+ if (!ASSERT_EQ(ret, -EINVAL, "bpf_tc_attach invalid priority > UINT16_MAX"))
+ return -EINVAL;
+
+ ret = bpf_tc_attach(hook, &opts_f);
+ if (!ASSERT_OK(ret, "bpf_tc_attach valid both handle and priority unset"))
+ return -EINVAL;
+ opts_f.prog_fd = opts_f.prog_id = 0;
+ ASSERT_OK(bpf_tc_detach(hook, &opts_f), "bpf_tc_detach");
+ }
+
+ return 0;
+}
+
+void test_tc_bpf(void)
+{
+ DECLARE_LIBBPF_OPTS(bpf_tc_hook, hook, .ifindex = LO_IFINDEX,
+ .attach_point = BPF_TC_INGRESS);
+ struct test_tc_bpf *skel = NULL;
+ bool hook_created = false;
+ int cls_fd, ret;
+
+ skel = test_tc_bpf__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "test_tc_bpf__open_and_load"))
+ return;
+
+ cls_fd = bpf_program__fd(skel->progs.cls);
+
+ ret = bpf_tc_hook_create(&hook);
+ if (ret == 0)
+ hook_created = true;
+
+ ret = ret == -EEXIST ? 0 : ret;
+ if (!ASSERT_OK(ret, "bpf_tc_hook_create(BPF_TC_INGRESS)"))
+ goto end;
+
+ hook.attach_point = BPF_TC_CUSTOM;
+ hook.parent = TC_H_MAKE(TC_H_CLSACT, TC_H_MIN_INGRESS);
+ ret = bpf_tc_hook_create(&hook);
+ if (!ASSERT_EQ(ret, -EOPNOTSUPP, "bpf_tc_hook_create invalid hook.attach_point"))
+ goto end;
+
+ ret = test_tc_bpf_basic(&hook, cls_fd);
+ if (!ASSERT_OK(ret, "test_tc_internal ingress"))
+ goto end;
+
+ ret = bpf_tc_hook_destroy(&hook);
+ if (!ASSERT_EQ(ret, -EOPNOTSUPP, "bpf_tc_hook_destroy invalid hook.attach_point"))
+ goto end;
+
+ hook.attach_point = BPF_TC_INGRESS;
+ hook.parent = 0;
+ bpf_tc_hook_destroy(&hook);
+
+ ret = test_tc_bpf_basic(&hook, cls_fd);
+ if (!ASSERT_OK(ret, "test_tc_internal ingress"))
+ goto end;
+
+ bpf_tc_hook_destroy(&hook);
+
+ hook.attach_point = BPF_TC_EGRESS;
+ ret = test_tc_bpf_basic(&hook, cls_fd);
+ if (!ASSERT_OK(ret, "test_tc_internal egress"))
+ goto end;
+
+ bpf_tc_hook_destroy(&hook);
+
+ ret = test_tc_bpf_api(&hook, cls_fd);
+ if (!ASSERT_OK(ret, "test_tc_bpf_api"))
+ goto end;
+
+ bpf_tc_hook_destroy(&hook);
+
+end:
+ if (hook_created) {
+ hook.attach_point = BPF_TC_INGRESS | BPF_TC_EGRESS;
+ bpf_tc_hook_destroy(&hook);
+ }
+ test_tc_bpf__destroy(skel);
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+
+/*
+ * This test sets up 3 netns (src <-> fwd <-> dst). There is no direct veth link
+ * between src and dst. The netns fwd has veth links to each src and dst. The
+ * client is in src and server in dst. The test installs a TC BPF program to each
+ * host facing veth in fwd which calls into i) bpf_redirect_neigh() to perform the
+ * neigh addr population and redirect or ii) bpf_redirect_peer() for namespace
+ * switch from ingress side; it also installs a checker prog on the egress side
+ * to drop unexpected traffic.
+ */
+
+#define _GNU_SOURCE
+
+#include <arpa/inet.h>
+#include <linux/limits.h>
+#include <linux/sysctl.h>
+#include <linux/if_tun.h>
+#include <linux/if.h>
+#include <sched.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/stat.h>
+#include <sys/mount.h>
+
+#include "test_progs.h"
+#include "network_helpers.h"
+#include "test_tc_neigh_fib.skel.h"
+#include "test_tc_neigh.skel.h"
+#include "test_tc_peer.skel.h"
+
+#define NS_SRC "ns_src"
+#define NS_FWD "ns_fwd"
+#define NS_DST "ns_dst"
+
+#define IP4_SRC "172.16.1.100"
+#define IP4_DST "172.16.2.100"
+#define IP4_TUN_SRC "172.17.1.100"
+#define IP4_TUN_FWD "172.17.1.200"
+#define IP4_PORT 9004
+
+#define IP6_SRC "0::1:dead:beef:cafe"
+#define IP6_DST "0::2:dead:beef:cafe"
+#define IP6_TUN_SRC "1::1:dead:beef:cafe"
+#define IP6_TUN_FWD "1::2:dead:beef:cafe"
+#define IP6_PORT 9006
+
+#define IP4_SLL "169.254.0.1"
+#define IP4_DLL "169.254.0.2"
+#define IP4_NET "169.254.0.0"
+
+#define MAC_DST_FWD "00:11:22:33:44:55"
+#define MAC_DST "00:22:33:44:55:66"
+
+#define IFADDR_STR_LEN 18
+#define PING_ARGS "-i 0.2 -c 3 -w 10 -q"
+
+#define SRC_PROG_PIN_FILE "/sys/fs/bpf/test_tc_src"
+#define DST_PROG_PIN_FILE "/sys/fs/bpf/test_tc_dst"
+#define CHK_PROG_PIN_FILE "/sys/fs/bpf/test_tc_chk"
+
+#define TIMEOUT_MILLIS 10000
+
+#define log_err(MSG, ...) \
+ fprintf(stderr, "(%s:%d: errno: %s) " MSG "\n", \
+ __FILE__, __LINE__, strerror(errno), ##__VA_ARGS__)
+
+static const char * const namespaces[] = {NS_SRC, NS_FWD, NS_DST, NULL};
+
+static int write_file(const char *path, const char *newval)
+{
+ FILE *f;
+
+ f = fopen(path, "r+");
+ if (!f)
+ return -1;
+ if (fwrite(newval, strlen(newval), 1, f) != 1) {
+ log_err("writing to %s failed", path);
+ fclose(f);
+ return -1;
+ }
+ fclose(f);
+ return 0;
+}
+
+struct nstoken {
+ int orig_netns_fd;
+};
+
+static int setns_by_fd(int nsfd)
+{
+ int err;
+
+ err = setns(nsfd, CLONE_NEWNET);
+ close(nsfd);
+
+ if (!ASSERT_OK(err, "setns"))
+ return err;
+
+ /* Switch /sys to the new namespace so that e.g. /sys/class/net
+ * reflects the devices in the new namespace.
+ */
+ err = unshare(CLONE_NEWNS);
+ if (!ASSERT_OK(err, "unshare"))
+ return err;
+
+ err = umount2("/sys", MNT_DETACH);
+ if (!ASSERT_OK(err, "umount2 /sys"))
+ return err;
+
+ err = mount("sysfs", "/sys", "sysfs", 0, NULL);
+ if (!ASSERT_OK(err, "mount /sys"))
+ return err;
+
+ err = mount("bpffs", "/sys/fs/bpf", "bpf", 0, NULL);
+ if (!ASSERT_OK(err, "mount /sys/fs/bpf"))
+ return err;
+
+ return 0;
+}
+
+/**
+ * open_netns() - Switch to specified network namespace by name.
+ *
+ * Returns token with which to restore the original namespace
+ * using close_netns().
+ */
+static struct nstoken *open_netns(const char *name)
+{
+ int nsfd;
+ char nspath[PATH_MAX];
+ int err;
+ struct nstoken *token;
+
+ token = malloc(sizeof(struct nstoken));
+ if (!ASSERT_OK_PTR(token, "malloc token"))
+ return NULL;
+
+ token->orig_netns_fd = open("/proc/self/ns/net", O_RDONLY);
+ if (!ASSERT_GE(token->orig_netns_fd, 0, "open /proc/self/ns/net"))
+ goto fail;
+
+ snprintf(nspath, sizeof(nspath), "%s/%s", "/var/run/netns", name);
+ nsfd = open(nspath, O_RDONLY | O_CLOEXEC);
+ if (!ASSERT_GE(nsfd, 0, "open netns fd"))
+ goto fail;
+
+ err = setns_by_fd(nsfd);
+ if (!ASSERT_OK(err, "setns_by_fd"))
+ goto fail;
+
+ return token;
+fail:
+ free(token);
+ return NULL;
+}
+
+static void close_netns(struct nstoken *token)
+{
+ ASSERT_OK(setns_by_fd(token->orig_netns_fd), "setns_by_fd");
+ free(token);
+}
+
+static int netns_setup_namespaces(const char *verb)
+{
+ const char * const *ns = namespaces;
+ char cmd[128];
+
+ while (*ns) {
+ snprintf(cmd, sizeof(cmd), "ip netns %s %s", verb, *ns);
+ if (!ASSERT_OK(system(cmd), cmd))
+ return -1;
+ ns++;
+ }
+ return 0;
+}
+
+struct netns_setup_result {
+ int ifindex_veth_src_fwd;
+ int ifindex_veth_dst_fwd;
+};
+
+static int get_ifaddr(const char *name, char *ifaddr)
+{
+ char path[PATH_MAX];
+ FILE *f;
+ int ret;
+
+ snprintf(path, PATH_MAX, "/sys/class/net/%s/address", name);
+ f = fopen(path, "r");
+ if (!ASSERT_OK_PTR(f, path))
+ return -1;
+
+ ret = fread(ifaddr, 1, IFADDR_STR_LEN, f);
+ if (!ASSERT_EQ(ret, IFADDR_STR_LEN, "fread ifaddr")) {
+ fclose(f);
+ return -1;
+ }
+ fclose(f);
+ return 0;
+}
+
+static int get_ifindex(const char *name)
+{
+ char path[PATH_MAX];
+ char buf[32];
+ FILE *f;
+ int ret;
+
+ snprintf(path, PATH_MAX, "/sys/class/net/%s/ifindex", name);
+ f = fopen(path, "r");
+ if (!ASSERT_OK_PTR(f, path))
+ return -1;
+
+ ret = fread(buf, 1, sizeof(buf), f);
+ if (!ASSERT_GT(ret, 0, "fread ifindex")) {
+ fclose(f);
+ return -1;
+ }
+ fclose(f);
+ return atoi(buf);
+}
+
+#define SYS(fmt, ...) \
+ ({ \
+ char cmd[1024]; \
+ snprintf(cmd, sizeof(cmd), fmt, ##__VA_ARGS__); \
+ if (!ASSERT_OK(system(cmd), cmd)) \
+ goto fail; \
+ })
+
+static int netns_setup_links_and_routes(struct netns_setup_result *result)
+{
+ struct nstoken *nstoken = NULL;
+ char veth_src_fwd_addr[IFADDR_STR_LEN+1] = {};
+
+ SYS("ip link add veth_src type veth peer name veth_src_fwd");
+ SYS("ip link add veth_dst type veth peer name veth_dst_fwd");
+
+ SYS("ip link set veth_dst_fwd address " MAC_DST_FWD);
+ SYS("ip link set veth_dst address " MAC_DST);
+
+ if (get_ifaddr("veth_src_fwd", veth_src_fwd_addr))
+ goto fail;
+
+ result->ifindex_veth_src_fwd = get_ifindex("veth_src_fwd");
+ if (result->ifindex_veth_src_fwd < 0)
+ goto fail;
+ result->ifindex_veth_dst_fwd = get_ifindex("veth_dst_fwd");
+ if (result->ifindex_veth_dst_fwd < 0)
+ goto fail;
+
+ SYS("ip link set veth_src netns " NS_SRC);
+ SYS("ip link set veth_src_fwd netns " NS_FWD);
+ SYS("ip link set veth_dst_fwd netns " NS_FWD);
+ SYS("ip link set veth_dst netns " NS_DST);
+
+ /** setup in 'src' namespace */
+ nstoken = open_netns(NS_SRC);
+ if (!ASSERT_OK_PTR(nstoken, "setns src"))
+ goto fail;
+
+ SYS("ip addr add " IP4_SRC "/32 dev veth_src");
+ SYS("ip addr add " IP6_SRC "/128 dev veth_src nodad");
+ SYS("ip link set dev veth_src up");
+
+ SYS("ip route add " IP4_DST "/32 dev veth_src scope global");
+ SYS("ip route add " IP4_NET "/16 dev veth_src scope global");
+ SYS("ip route add " IP6_DST "/128 dev veth_src scope global");
+
+ SYS("ip neigh add " IP4_DST " dev veth_src lladdr %s",
+ veth_src_fwd_addr);
+ SYS("ip neigh add " IP6_DST " dev veth_src lladdr %s",
+ veth_src_fwd_addr);
+
+ close_netns(nstoken);
+
+ /** setup in 'fwd' namespace */
+ nstoken = open_netns(NS_FWD);
+ if (!ASSERT_OK_PTR(nstoken, "setns fwd"))
+ goto fail;
+
+ /* The fwd netns automatically gets a v6 LL address / routes, but also
+ * needs v4 one in order to start ARP probing. IP4_NET route is added
+ * to the endpoints so that the ARP processing will reply.
+ */
+ SYS("ip addr add " IP4_SLL "/32 dev veth_src_fwd");
+ SYS("ip addr add " IP4_DLL "/32 dev veth_dst_fwd");
+ SYS("ip link set dev veth_src_fwd up");
+ SYS("ip link set dev veth_dst_fwd up");
+
+ SYS("ip route add " IP4_SRC "/32 dev veth_src_fwd scope global");
+ SYS("ip route add " IP6_SRC "/128 dev veth_src_fwd scope global");
+ SYS("ip route add " IP4_DST "/32 dev veth_dst_fwd scope global");
+ SYS("ip route add " IP6_DST "/128 dev veth_dst_fwd scope global");
+
+ close_netns(nstoken);
+
+ /** setup in 'dst' namespace */
+ nstoken = open_netns(NS_DST);
+ if (!ASSERT_OK_PTR(nstoken, "setns dst"))
+ goto fail;
+
+ SYS("ip addr add " IP4_DST "/32 dev veth_dst");
+ SYS("ip addr add " IP6_DST "/128 dev veth_dst nodad");
+ SYS("ip link set dev veth_dst up");
+
+ SYS("ip route add " IP4_SRC "/32 dev veth_dst scope global");
+ SYS("ip route add " IP4_NET "/16 dev veth_dst scope global");
+ SYS("ip route add " IP6_SRC "/128 dev veth_dst scope global");
+
+ SYS("ip neigh add " IP4_SRC " dev veth_dst lladdr " MAC_DST_FWD);
+ SYS("ip neigh add " IP6_SRC " dev veth_dst lladdr " MAC_DST_FWD);
+
+ close_netns(nstoken);
+
+ return 0;
+fail:
+ if (nstoken)
+ close_netns(nstoken);
+ return -1;
+}
+
+static int netns_load_bpf(void)
+{
+ SYS("tc qdisc add dev veth_src_fwd clsact");
+ SYS("tc filter add dev veth_src_fwd ingress bpf da object-pinned "
+ SRC_PROG_PIN_FILE);
+ SYS("tc filter add dev veth_src_fwd egress bpf da object-pinned "
+ CHK_PROG_PIN_FILE);
+
+ SYS("tc qdisc add dev veth_dst_fwd clsact");
+ SYS("tc filter add dev veth_dst_fwd ingress bpf da object-pinned "
+ DST_PROG_PIN_FILE);
+ SYS("tc filter add dev veth_dst_fwd egress bpf da object-pinned "
+ CHK_PROG_PIN_FILE);
+
+ return 0;
+fail:
+ return -1;
+}
+
+static void test_tcp(int family, const char *addr, __u16 port)
+{
+ int listen_fd = -1, accept_fd = -1, client_fd = -1;
+ char buf[] = "testing testing";
+ int n;
+ struct nstoken *nstoken;
+
+ nstoken = open_netns(NS_DST);
+ if (!ASSERT_OK_PTR(nstoken, "setns dst"))
+ return;
+
+ listen_fd = start_server(family, SOCK_STREAM, addr, port, 0);
+ if (!ASSERT_GE(listen_fd, 0, "listen"))
+ goto done;
+
+ close_netns(nstoken);
+ nstoken = open_netns(NS_SRC);
+ if (!ASSERT_OK_PTR(nstoken, "setns src"))
+ goto done;
+
+ client_fd = connect_to_fd(listen_fd, TIMEOUT_MILLIS);
+ if (!ASSERT_GE(client_fd, 0, "connect_to_fd"))
+ goto done;
+
+ accept_fd = accept(listen_fd, NULL, NULL);
+ if (!ASSERT_GE(accept_fd, 0, "accept"))
+ goto done;
+
+ if (!ASSERT_OK(settimeo(accept_fd, TIMEOUT_MILLIS), "settimeo"))
+ goto done;
+
+ n = write(client_fd, buf, sizeof(buf));
+ if (!ASSERT_EQ(n, sizeof(buf), "send to server"))
+ goto done;
+
+ n = read(accept_fd, buf, sizeof(buf));
+ ASSERT_EQ(n, sizeof(buf), "recv from server");
+
+done:
+ if (nstoken)
+ close_netns(nstoken);
+ if (listen_fd >= 0)
+ close(listen_fd);
+ if (accept_fd >= 0)
+ close(accept_fd);
+ if (client_fd >= 0)
+ close(client_fd);
+}
+
+static int test_ping(int family, const char *addr)
+{
+ const char *ping = family == AF_INET6 ? "ping6" : "ping";
+
+ SYS("ip netns exec " NS_SRC " %s " PING_ARGS " %s > /dev/null", ping, addr);
+ return 0;
+fail:
+ return -1;
+}
+
+static void test_connectivity(void)
+{
+ test_tcp(AF_INET, IP4_DST, IP4_PORT);
+ test_ping(AF_INET, IP4_DST);
+ test_tcp(AF_INET6, IP6_DST, IP6_PORT);
+ test_ping(AF_INET6, IP6_DST);
+}
+
+static int set_forwarding(bool enable)
+{
+ int err;
+
+ err = write_file("/proc/sys/net/ipv4/ip_forward", enable ? "1" : "0");
+ if (!ASSERT_OK(err, "set ipv4.ip_forward=0"))
+ return err;
+
+ err = write_file("/proc/sys/net/ipv6/conf/all/forwarding", enable ? "1" : "0");
+ if (!ASSERT_OK(err, "set ipv6.forwarding=0"))
+ return err;
+
+ return 0;
+}
+
+static void test_tc_redirect_neigh_fib(struct netns_setup_result *setup_result)
+{
+ struct nstoken *nstoken = NULL;
+ struct test_tc_neigh_fib *skel = NULL;
+ int err;
+
+ nstoken = open_netns(NS_FWD);
+ if (!ASSERT_OK_PTR(nstoken, "setns fwd"))
+ return;
+
+ skel = test_tc_neigh_fib__open();
+ if (!ASSERT_OK_PTR(skel, "test_tc_neigh_fib__open"))
+ goto done;
+
+ if (!ASSERT_OK(test_tc_neigh_fib__load(skel), "test_tc_neigh_fib__load"))
+ goto done;
+
+ err = bpf_program__pin(skel->progs.tc_src, SRC_PROG_PIN_FILE);
+ if (!ASSERT_OK(err, "pin " SRC_PROG_PIN_FILE))
+ goto done;
+
+ err = bpf_program__pin(skel->progs.tc_chk, CHK_PROG_PIN_FILE);
+ if (!ASSERT_OK(err, "pin " CHK_PROG_PIN_FILE))
+ goto done;
+
+ err = bpf_program__pin(skel->progs.tc_dst, DST_PROG_PIN_FILE);
+ if (!ASSERT_OK(err, "pin " DST_PROG_PIN_FILE))
+ goto done;
+
+ if (netns_load_bpf())
+ goto done;
+
+ /* bpf_fib_lookup() checks if forwarding is enabled */
+ if (!ASSERT_OK(set_forwarding(true), "enable forwarding"))
+ goto done;
+
+ test_connectivity();
+
+done:
+ if (skel)
+ test_tc_neigh_fib__destroy(skel);
+ close_netns(nstoken);
+}
+
+static void test_tc_redirect_neigh(struct netns_setup_result *setup_result)
+{
+ struct nstoken *nstoken = NULL;
+ struct test_tc_neigh *skel = NULL;
+ int err;
+
+ nstoken = open_netns(NS_FWD);
+ if (!ASSERT_OK_PTR(nstoken, "setns fwd"))
+ return;
+
+ skel = test_tc_neigh__open();
+ if (!ASSERT_OK_PTR(skel, "test_tc_neigh__open"))
+ goto done;
+
+ skel->rodata->IFINDEX_SRC = setup_result->ifindex_veth_src_fwd;
+ skel->rodata->IFINDEX_DST = setup_result->ifindex_veth_dst_fwd;
+
+ err = test_tc_neigh__load(skel);
+ if (!ASSERT_OK(err, "test_tc_neigh__load"))
+ goto done;
+
+ err = bpf_program__pin(skel->progs.tc_src, SRC_PROG_PIN_FILE);
+ if (!ASSERT_OK(err, "pin " SRC_PROG_PIN_FILE))
+ goto done;
+
+ err = bpf_program__pin(skel->progs.tc_chk, CHK_PROG_PIN_FILE);
+ if (!ASSERT_OK(err, "pin " CHK_PROG_PIN_FILE))
+ goto done;
+
+ err = bpf_program__pin(skel->progs.tc_dst, DST_PROG_PIN_FILE);
+ if (!ASSERT_OK(err, "pin " DST_PROG_PIN_FILE))
+ goto done;
+
+ if (netns_load_bpf())
+ goto done;
+
+ if (!ASSERT_OK(set_forwarding(false), "disable forwarding"))
+ goto done;
+
+ test_connectivity();
+
+done:
+ if (skel)
+ test_tc_neigh__destroy(skel);
+ close_netns(nstoken);
+}
+
+static void test_tc_redirect_peer(struct netns_setup_result *setup_result)
+{
+ struct nstoken *nstoken;
+ struct test_tc_peer *skel;
+ int err;
+
+ nstoken = open_netns(NS_FWD);
+ if (!ASSERT_OK_PTR(nstoken, "setns fwd"))
+ return;
+
+ skel = test_tc_peer__open();
+ if (!ASSERT_OK_PTR(skel, "test_tc_peer__open"))
+ goto done;
+
+ skel->rodata->IFINDEX_SRC = setup_result->ifindex_veth_src_fwd;
+ skel->rodata->IFINDEX_DST = setup_result->ifindex_veth_dst_fwd;
+
+ err = test_tc_peer__load(skel);
+ if (!ASSERT_OK(err, "test_tc_peer__load"))
+ goto done;
+
+ err = bpf_program__pin(skel->progs.tc_src, SRC_PROG_PIN_FILE);
+ if (!ASSERT_OK(err, "pin " SRC_PROG_PIN_FILE))
+ goto done;
+
+ err = bpf_program__pin(skel->progs.tc_chk, CHK_PROG_PIN_FILE);
+ if (!ASSERT_OK(err, "pin " CHK_PROG_PIN_FILE))
+ goto done;
+
+ err = bpf_program__pin(skel->progs.tc_dst, DST_PROG_PIN_FILE);
+ if (!ASSERT_OK(err, "pin " DST_PROG_PIN_FILE))
+ goto done;
+
+ if (netns_load_bpf())
+ goto done;
+
+ if (!ASSERT_OK(set_forwarding(false), "disable forwarding"))
+ goto done;
+
+ test_connectivity();
+
+done:
+ if (skel)
+ test_tc_peer__destroy(skel);
+ close_netns(nstoken);
+}
+
+static int tun_open(char *name)
+{
+ struct ifreq ifr;
+ int fd, err;
+
+ fd = open("/dev/net/tun", O_RDWR);
+ if (!ASSERT_GE(fd, 0, "open /dev/net/tun"))
+ return -1;
+
+ memset(&ifr, 0, sizeof(ifr));
+
+ ifr.ifr_flags = IFF_TUN | IFF_NO_PI;
+ if (*name)
+ strncpy(ifr.ifr_name, name, IFNAMSIZ);
+
+ err = ioctl(fd, TUNSETIFF, &ifr);
+ if (!ASSERT_OK(err, "ioctl TUNSETIFF"))
+ goto fail;
+
+ SYS("ip link set dev %s up", name);
+
+ return fd;
+fail:
+ close(fd);
+ return -1;
+}
+
+#define MAX(a, b) ((a) > (b) ? (a) : (b))
+enum {
+ SRC_TO_TARGET = 0,
+ TARGET_TO_SRC = 1,
+};
+
+static int tun_relay_loop(int src_fd, int target_fd)
+{
+ fd_set rfds, wfds;
+
+ FD_ZERO(&rfds);
+ FD_ZERO(&wfds);
+
+ for (;;) {
+ char buf[1500];
+ int direction, nread, nwrite;
+
+ FD_SET(src_fd, &rfds);
+ FD_SET(target_fd, &rfds);
+
+ if (select(1 + MAX(src_fd, target_fd), &rfds, NULL, NULL, NULL) < 0) {
+ log_err("select failed");
+ return 1;
+ }
+
+ direction = FD_ISSET(src_fd, &rfds) ? SRC_TO_TARGET : TARGET_TO_SRC;
+
+ nread = read(direction == SRC_TO_TARGET ? src_fd : target_fd, buf, sizeof(buf));
+ if (nread < 0) {
+ log_err("read failed");
+ return 1;
+ }
+
+ nwrite = write(direction == SRC_TO_TARGET ? target_fd : src_fd, buf, nread);
+ if (nwrite != nread) {
+ log_err("write failed");
+ return 1;
+ }
+ }
+}
+
+static void test_tc_redirect_peer_l3(struct netns_setup_result *setup_result)
+{
+ struct test_tc_peer *skel = NULL;
+ struct nstoken *nstoken = NULL;
+ int err;
+ int tunnel_pid = -1;
+ int src_fd, target_fd;
+ int ifindex;
+
+ /* Start a L3 TUN/TAP tunnel between the src and dst namespaces.
+ * This test is using TUN/TAP instead of e.g. IPIP or GRE tunnel as those
+ * expose the L2 headers encapsulating the IP packet to BPF and hence
+ * don't have skb in suitable state for this test. Alternative to TUN/TAP
+ * would be e.g. Wireguard which would appear as a pure L3 device to BPF,
+ * but that requires much more complicated setup.
+ */
+ nstoken = open_netns(NS_SRC);
+ if (!ASSERT_OK_PTR(nstoken, "setns " NS_SRC))
+ return;
+
+ src_fd = tun_open("tun_src");
+ if (!ASSERT_GE(src_fd, 0, "tun_open tun_src"))
+ goto fail;
+
+ close_netns(nstoken);
+
+ nstoken = open_netns(NS_FWD);
+ if (!ASSERT_OK_PTR(nstoken, "setns " NS_FWD))
+ goto fail;
+
+ target_fd = tun_open("tun_fwd");
+ if (!ASSERT_GE(target_fd, 0, "tun_open tun_fwd"))
+ goto fail;
+
+ tunnel_pid = fork();
+ if (!ASSERT_GE(tunnel_pid, 0, "fork tun_relay_loop"))
+ goto fail;
+
+ if (tunnel_pid == 0)
+ exit(tun_relay_loop(src_fd, target_fd));
+
+ skel = test_tc_peer__open();
+ if (!ASSERT_OK_PTR(skel, "test_tc_peer__open"))
+ goto fail;
+
+ ifindex = get_ifindex("tun_fwd");
+ if (!ASSERT_GE(ifindex, 0, "get_ifindex tun_fwd"))
+ goto fail;
+
+ skel->rodata->IFINDEX_SRC = ifindex;
+ skel->rodata->IFINDEX_DST = setup_result->ifindex_veth_dst_fwd;
+
+ err = test_tc_peer__load(skel);
+ if (!ASSERT_OK(err, "test_tc_peer__load"))
+ goto fail;
+
+ err = bpf_program__pin(skel->progs.tc_src_l3, SRC_PROG_PIN_FILE);
+ if (!ASSERT_OK(err, "pin " SRC_PROG_PIN_FILE))
+ goto fail;
+
+ err = bpf_program__pin(skel->progs.tc_dst_l3, DST_PROG_PIN_FILE);
+ if (!ASSERT_OK(err, "pin " DST_PROG_PIN_FILE))
+ goto fail;
+
+ err = bpf_program__pin(skel->progs.tc_chk, CHK_PROG_PIN_FILE);
+ if (!ASSERT_OK(err, "pin " CHK_PROG_PIN_FILE))
+ goto fail;
+
+ /* Load "tc_src_l3" to the tun_fwd interface to redirect packets
+ * towards dst, and "tc_dst" to redirect packets
+ * and "tc_chk" on veth_dst_fwd to drop non-redirected packets.
+ */
+ SYS("tc qdisc add dev tun_fwd clsact");
+ SYS("tc filter add dev tun_fwd ingress bpf da object-pinned "
+ SRC_PROG_PIN_FILE);
+
+ SYS("tc qdisc add dev veth_dst_fwd clsact");
+ SYS("tc filter add dev veth_dst_fwd ingress bpf da object-pinned "
+ DST_PROG_PIN_FILE);
+ SYS("tc filter add dev veth_dst_fwd egress bpf da object-pinned "
+ CHK_PROG_PIN_FILE);
+
+ /* Setup route and neigh tables */
+ SYS("ip -netns " NS_SRC " addr add dev tun_src " IP4_TUN_SRC "/24");
+ SYS("ip -netns " NS_FWD " addr add dev tun_fwd " IP4_TUN_FWD "/24");
+
+ SYS("ip -netns " NS_SRC " addr add dev tun_src " IP6_TUN_SRC "/64 nodad");
+ SYS("ip -netns " NS_FWD " addr add dev tun_fwd " IP6_TUN_FWD "/64 nodad");
+
+ SYS("ip -netns " NS_SRC " route del " IP4_DST "/32 dev veth_src scope global");
+ SYS("ip -netns " NS_SRC " route add " IP4_DST "/32 via " IP4_TUN_FWD
+ " dev tun_src scope global");
+ SYS("ip -netns " NS_DST " route add " IP4_TUN_SRC "/32 dev veth_dst scope global");
+ SYS("ip -netns " NS_SRC " route del " IP6_DST "/128 dev veth_src scope global");
+ SYS("ip -netns " NS_SRC " route add " IP6_DST "/128 via " IP6_TUN_FWD
+ " dev tun_src scope global");
+ SYS("ip -netns " NS_DST " route add " IP6_TUN_SRC "/128 dev veth_dst scope global");
+
+ SYS("ip -netns " NS_DST " neigh add " IP4_TUN_SRC " dev veth_dst lladdr " MAC_DST_FWD);
+ SYS("ip -netns " NS_DST " neigh add " IP6_TUN_SRC " dev veth_dst lladdr " MAC_DST_FWD);
+
+ if (!ASSERT_OK(set_forwarding(false), "disable forwarding"))
+ goto fail;
+
+ test_connectivity();
+
+fail:
+ if (tunnel_pid > 0) {
+ kill(tunnel_pid, SIGTERM);
+ waitpid(tunnel_pid, NULL, 0);
+ }
+ if (src_fd >= 0)
+ close(src_fd);
+ if (target_fd >= 0)
+ close(target_fd);
+ if (skel)
+ test_tc_peer__destroy(skel);
+ if (nstoken)
+ close_netns(nstoken);
+}
+
+#define RUN_TEST(name) \
+ ({ \
+ struct netns_setup_result setup_result; \
+ if (test__start_subtest(#name)) \
+ if (ASSERT_OK(netns_setup_namespaces("add"), "setup namespaces")) { \
+ if (ASSERT_OK(netns_setup_links_and_routes(&setup_result), \
+ "setup links and routes")) \
+ test_ ## name(&setup_result); \
+ netns_setup_namespaces("delete"); \
+ } \
+ })
+
+static void *test_tc_redirect_run_tests(void *arg)
+{
+ RUN_TEST(tc_redirect_peer);
+ RUN_TEST(tc_redirect_peer_l3);
+ RUN_TEST(tc_redirect_neigh);
+ RUN_TEST(tc_redirect_neigh_fib);
+ return NULL;
+}
+
+void test_tc_redirect(void)
+{
+ pthread_t test_thread;
+ int err;
+
+ /* Run the tests in their own thread to isolate the namespace changes
+ * so they do not affect the environment of other tests.
+ * (specifically needed because of unshare(CLONE_NEWNS) in open_netns())
+ */
+ err = pthread_create(&test_thread, NULL, &test_tc_redirect_run_tests, NULL);
+ if (ASSERT_OK(err, "pthread_create"))
+ ASSERT_OK(pthread_join(test_thread, NULL), "pthread_join");
+}
return;
link = bpf_program__attach_cgroup(skel->progs.estab, cg_fd);
- if (CHECK(IS_ERR(link), "attach_cgroup(estab)", "err: %ld\n",
- PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_cgroup(estab)"))
return;
if (sk_fds_connect(&sk_fds, true)) {
return;
link = bpf_program__attach_cgroup(skel->progs.estab, cg_fd);
- if (CHECK(IS_ERR(link), "attach_cgroup(estab)", "err: %ld\n",
- PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_cgroup(estab)"))
return;
if (sk_fds_connect(&sk_fds, false)) {
return;
link = bpf_program__attach_cgroup(skel->progs.estab, cg_fd);
- if (CHECK(IS_ERR(link), "attach_cgroup(estab)", "err: %ld\n",
- PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_cgroup(estab)"))
return;
if (sk_fds_connect(&sk_fds, false)) {
return;
link = bpf_program__attach_cgroup(skel->progs.estab, cg_fd);
- if (CHECK(IS_ERR(link), "attach_cgroup(estab)", "err: %ld\n",
- PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_cgroup(estab)"))
return;
if (sk_fds_connect(&sk_fds, false)) {
return;
link = bpf_program__attach_cgroup(misc_skel->progs.misc_estab, cg_fd);
- if (CHECK(IS_ERR(link), "attach_cgroup(misc_estab)", "err: %ld\n",
- PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_cgroup(misc_estab)"))
return;
if (sk_fds_connect(&sk_fds, false)) {
return;
obj = bpf_object__open_file("./test_overhead.o", NULL);
- if (CHECK(IS_ERR(obj), "obj_open_file", "err %ld\n", PTR_ERR(obj)))
+ if (!ASSERT_OK_PTR(obj, "obj_open_file"))
return;
kprobe_prog = bpf_object__find_program_by_title(obj, kprobe_name);
/* attach kprobe */
link = bpf_program__attach_kprobe(kprobe_prog, false /* retprobe */,
kprobe_func);
- if (CHECK(IS_ERR(link), "attach_kprobe", "err %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_kprobe"))
goto cleanup;
test_run("kprobe");
bpf_link__destroy(link);
/* attach kretprobe */
link = bpf_program__attach_kprobe(kretprobe_prog, true /* retprobe */,
kprobe_func);
- if (CHECK(IS_ERR(link), "attach kretprobe", "err %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_kretprobe"))
goto cleanup;
test_run("kretprobe");
bpf_link__destroy(link);
/* attach raw_tp */
link = bpf_program__attach_raw_tracepoint(raw_tp_prog, "task_rename");
- if (CHECK(IS_ERR(link), "attach fentry", "err %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_raw_tp"))
goto cleanup;
test_run("raw_tp");
bpf_link__destroy(link);
/* attach fentry */
link = bpf_program__attach_trace(fentry_prog);
- if (CHECK(IS_ERR(link), "attach fentry", "err %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_fentry"))
goto cleanup;
test_run("fentry");
bpf_link__destroy(link);
/* attach fexit */
link = bpf_program__attach_trace(fexit_prog);
- if (CHECK(IS_ERR(link), "attach fexit", "err %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "attach_fexit"))
goto cleanup;
test_run("fexit");
bpf_link__destroy(link);
#include <test_progs.h>
-#include "trace_printk.skel.h"
+#include "trace_printk.lskel.h"
#define TRACEBUF "/sys/kernel/debug/tracing/trace_pipe"
#define SEARCHMSG "testing,testing"
if (CHECK(!skel, "skel_open", "failed to open skeleton\n"))
return;
+ ASSERT_EQ(skel->rodata->fmt[0], 'T', "invalid printk fmt string");
+ skel->rodata->fmt[0] = 't';
+
err = trace_printk__load(skel);
if (CHECK(err, "skel_load", "failed to load skeleton: %d\n", err))
goto cleanup;
/* attach 'allowed' trampoline programs */
for (i = 0; i < MAX_TRAMP_PROGS; i++) {
obj = bpf_object__open_file(object, NULL);
- if (CHECK(IS_ERR(obj), "obj_open_file", "err %ld\n", PTR_ERR(obj))) {
+ if (!ASSERT_OK_PTR(obj, "obj_open_file")) {
obj = NULL;
goto cleanup;
}
if (rand() % 2) {
link = load(inst[i].obj, fentry_name);
- if (CHECK(IS_ERR(link), "attach prog", "err %ld\n", PTR_ERR(link))) {
+ if (!ASSERT_OK_PTR(link, "attach_prog")) {
link = NULL;
goto cleanup;
}
inst[i].link_fentry = link;
} else {
link = load(inst[i].obj, fexit_name);
- if (CHECK(IS_ERR(link), "attach prog", "err %ld\n", PTR_ERR(link))) {
+ if (!ASSERT_OK_PTR(link, "attach_prog")) {
link = NULL;
goto cleanup;
}
/* and try 1 extra.. */
obj = bpf_object__open_file(object, NULL);
- if (CHECK(IS_ERR(obj), "obj_open_file", "err %ld\n", PTR_ERR(obj))) {
+ if (!ASSERT_OK_PTR(obj, "obj_open_file")) {
obj = NULL;
goto cleanup;
}
/* ..that needs to fail */
link = load(obj, fentry_name);
- if (CHECK(!IS_ERR(link), "cannot attach over the limit", "err %ld\n", PTR_ERR(link))) {
+ err = libbpf_get_error(link);
+ if (!ASSERT_ERR_PTR(link, "cannot attach over the limit")) {
bpf_link__destroy(link);
goto cleanup_extra;
}
/* with E2BIG error */
- CHECK(PTR_ERR(link) != -E2BIG, "proper error check", "err %ld\n", PTR_ERR(link));
+ ASSERT_EQ(err, -E2BIG, "proper error check");
+ ASSERT_EQ(link, NULL, "ptr_is_null");
/* and finaly execute the probe */
if (CHECK_FAIL(prctl(PR_GET_NAME, comm, 0L, 0L, 0L)))
goto close_cgroup_fd;
skel->links.sock = bpf_program__attach_cgroup(skel->progs.sock, cgroup_fd);
+ if (!ASSERT_OK_PTR(skel->links.sock, "cg_attach_sock"))
+ goto close_skeleton;
skel->links.sock_release = bpf_program__attach_cgroup(skel->progs.sock_release, cgroup_fd);
- if (CHECK(IS_ERR(skel->links.sock) || IS_ERR(skel->links.sock_release),
- "cg-attach", "sock %ld sock_release %ld",
- PTR_ERR(skel->links.sock),
- PTR_ERR(skel->links.sock_release)))
+ if (!ASSERT_OK_PTR(skel->links.sock_release, "cg_attach_sock_release"))
goto close_skeleton;
/* BPF program enforces a single UDP socket per cgroup,
pb_opts.ctx = &passed;
pb = perf_buffer__new(bpf_map__fd(ftrace_skel->maps.perf_buf_map),
1, &pb_opts);
- if (CHECK(IS_ERR(pb), "perf_buf__new", "err %ld\n", PTR_ERR(pb)))
+ if (!ASSERT_OK_PTR(pb, "perf_buf__new"))
goto out;
/* Run test program */
/* BPF link is not allowed to replace prog attachment */
link = bpf_program__attach_xdp(skel1->progs.xdp_handler, IFINDEX_LO);
- if (CHECK(!IS_ERR(link), "link_attach_fail", "unexpected success\n")) {
+ if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
bpf_link__destroy(link);
/* best-effort detach prog */
opts.old_fd = prog_fd1;
/* now BPF link should attach successfully */
link = bpf_program__attach_xdp(skel1->progs.xdp_handler, IFINDEX_LO);
- if (CHECK(IS_ERR(link), "link_attach", "failed: %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "link_attach"))
goto cleanup;
skel1->links.xdp_handler = link;
/* BPF link is not allowed to replace another BPF link */
link = bpf_program__attach_xdp(skel2->progs.xdp_handler, IFINDEX_LO);
- if (CHECK(!IS_ERR(link), "link_attach_fail", "unexpected success\n")) {
+ if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
bpf_link__destroy(link);
goto cleanup;
}
/* new link attach should succeed */
link = bpf_program__attach_xdp(skel2->progs.xdp_handler, IFINDEX_LO);
- if (CHECK(IS_ERR(link), "link_attach", "failed: %ld\n", PTR_ERR(link)))
+ if (!ASSERT_OK_PTR(link, "link_attach"))
goto cleanup;
skel2->links.xdp_handler = link;
/* Copyright (c) 2020 Facebook */
#include "bpf_iter.h"
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
/* Copyright (c) 2020 Facebook */
#include "bpf_iter.h"
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
#include "bpf_iter.h"
#include "bpf_tracing_net.h"
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
#include "bpf_iter.h"
#include "bpf_tracing_net.h"
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
/* Copyright (c) 2020 Facebook */
#include "bpf_iter.h"
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
/* Copyright (c) 2020, Oracle and/or its affiliates. */
#include "bpf_iter.h"
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
#include <errno.h>
/* Copyright (c) 2020 Facebook */
#include "bpf_iter.h"
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
/* Copyright (c) 2020 Facebook */
#include "bpf_iter.h"
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
/* Copyright (c) 2020 Facebook */
#include "bpf_iter.h"
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
char _license[] SEC("license") = "GPL";
#include "bpf_iter.h"
#include "bpf_tracing_net.h"
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
#include <bpf/bpf_endian.h>
char _license[] SEC("license") = "GPL";
#include "bpf_iter.h"
#include "bpf_tracing_net.h"
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
#include <bpf/bpf_endian.h>
char _license[] SEC("license") = "GPL";
__u32 map1_accessed = 0, map2_accessed = 0;
__u64 map1_seqnum = 0, map2_seqnum1 = 0, map2_seqnum2 = 0;
-static volatile const __u32 print_len;
-static volatile const __u32 ret1;
+volatile const __u32 print_len;
+volatile const __u32 ret1;
SEC("iter/bpf_map")
int dump_bpf_map(struct bpf_iter__bpf_map *ctx)
#include "bpf_iter.h"
#include "bpf_tracing_net.h"
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
#include <bpf/bpf_endian.h>
char _license[] SEC("license") = "GPL";
#include "bpf_iter.h"
#include "bpf_tracing_net.h"
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
#include <bpf/bpf_endian.h>
char _license[] SEC("license") = "GPL";
return 0;
}
-static volatile struct {
+struct {
bool fentry_test_ok;
bool fexit_test_ok;
-} result;
+} result = {};
SEC("fentry/eth_type_trans")
int BPF_PROG(fentry_eth_type_trans, struct sk_buff *skb, struct net_device *dev,
val = bpf_map_lookup_elem(&map_weak, &key);
if (val)
output_weak1 = *val;
-
+
return 0;
}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2021 Facebook */
+#include <linux/stddef.h>
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <../../../tools/include/linux/filter.h>
+#include <linux/btf.h>
+
+char _license[] SEC("license") = "GPL";
+
+struct args {
+ __u64 log_buf;
+ __u32 log_size;
+ int max_entries;
+ int map_fd;
+ int prog_fd;
+ int btf_fd;
+};
+
+#define BTF_INFO_ENC(kind, kind_flag, vlen) \
+ ((!!(kind_flag) << 31) | ((kind) << 24) | ((vlen) & BTF_MAX_VLEN))
+#define BTF_TYPE_ENC(name, info, size_or_type) (name), (info), (size_or_type)
+#define BTF_INT_ENC(encoding, bits_offset, nr_bits) \
+ ((encoding) << 24 | (bits_offset) << 16 | (nr_bits))
+#define BTF_TYPE_INT_ENC(name, encoding, bits_offset, bits, sz) \
+ BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_INT, 0, 0), sz), \
+ BTF_INT_ENC(encoding, bits_offset, bits)
+
+static int btf_load(void)
+{
+ struct btf_blob {
+ struct btf_header btf_hdr;
+ __u32 types[8];
+ __u32 str;
+ } raw_btf = {
+ .btf_hdr = {
+ .magic = BTF_MAGIC,
+ .version = BTF_VERSION,
+ .hdr_len = sizeof(struct btf_header),
+ .type_len = sizeof(__u32) * 8,
+ .str_off = sizeof(__u32) * 8,
+ .str_len = sizeof(__u32),
+ },
+ .types = {
+ /* long */
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 64, 8), /* [1] */
+ /* unsigned long */
+ BTF_TYPE_INT_ENC(0, 0, 0, 64, 8), /* [2] */
+ },
+ };
+ static union bpf_attr btf_load_attr = {
+ .btf_size = sizeof(raw_btf),
+ };
+
+ btf_load_attr.btf = (long)&raw_btf;
+ return bpf_sys_bpf(BPF_BTF_LOAD, &btf_load_attr, sizeof(btf_load_attr));
+}
+
+SEC("syscall")
+int bpf_prog(struct args *ctx)
+{
+ static char license[] = "GPL";
+ static struct bpf_insn insns[] = {
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ };
+ static union bpf_attr map_create_attr = {
+ .map_type = BPF_MAP_TYPE_HASH,
+ .key_size = 8,
+ .value_size = 8,
+ .btf_key_type_id = 1,
+ .btf_value_type_id = 2,
+ };
+ static union bpf_attr map_update_attr = { .map_fd = 1, };
+ static __u64 key = 12;
+ static __u64 value = 34;
+ static union bpf_attr prog_load_attr = {
+ .prog_type = BPF_PROG_TYPE_XDP,
+ .insn_cnt = sizeof(insns) / sizeof(insns[0]),
+ };
+ int ret;
+
+ ret = btf_load();
+ if (ret <= 0)
+ return ret;
+
+ ctx->btf_fd = ret;
+ map_create_attr.max_entries = ctx->max_entries;
+ map_create_attr.btf_fd = ret;
+
+ prog_load_attr.license = (long) license;
+ prog_load_attr.insns = (long) insns;
+ prog_load_attr.log_buf = ctx->log_buf;
+ prog_load_attr.log_size = ctx->log_size;
+ prog_load_attr.log_level = 1;
+
+ ret = bpf_sys_bpf(BPF_MAP_CREATE, &map_create_attr, sizeof(map_create_attr));
+ if (ret <= 0)
+ return ret;
+ ctx->map_fd = ret;
+ insns[3].imm = ret;
+
+ map_update_attr.map_fd = ret;
+ map_update_attr.key = (long) &key;
+ map_update_attr.value = (long) &value;
+ ret = bpf_sys_bpf(BPF_MAP_UPDATE_ELEM, &map_update_attr, sizeof(map_update_attr));
+ if (ret < 0)
+ return ret;
+
+ ret = bpf_sys_bpf(BPF_PROG_LOAD, &prog_load_attr, sizeof(prog_load_attr));
+ if (ret <= 0)
+ return ret;
+ ctx->prog_fd = ret;
+ return 1;
+}
__uint(value_size, sizeof(__u32));
} jmp_table SEC(".maps");
-static volatile int count;
+int count = 0;
SEC("classifier/0")
int bpf_func_0(struct __sk_buff *skb)
__uint(value_size, sizeof(__u32));
} jmp_table SEC(".maps");
-static volatile int selector;
+int selector = 0;
#define TAIL_FUNC(x) \
SEC("classifier/" #x) \
__uint(value_size, sizeof(__u32));
} jmp_table SEC(".maps");
-static volatile int selector;
+int selector = 0;
#define TAIL_FUNC(x) \
SEC("classifier/" #x) \
return 1;
}
-static volatile int count;
+int count = 0;
SEC("classifier/0")
int bpf_func_0(struct __sk_buff *skb)
__uint(value_size, sizeof(__u32));
} jmp_table SEC(".maps");
-static volatile int count;
+int count = 0;
__noinline
int subprog_tail_2(struct __sk_buff *skb)
char _license[] SEC("license") = "GPL";
/* Userspace will update with MTU it can see on device */
-static volatile const int GLOBAL_USER_MTU;
-static volatile const __u32 GLOBAL_USER_IFINDEX;
+volatile const int GLOBAL_USER_MTU;
+volatile const __u32 GLOBAL_USER_IFINDEX;
/* BPF-prog will update these with MTU values it can see */
__u32 global_bpf_mtu_xdp = 0;
/**
* Destination port and IP used for UDP encapsulation.
*/
-static volatile const __be16 ENCAPSULATION_PORT;
-static volatile const __be32 ENCAPSULATION_IP;
+volatile const __be16 ENCAPSULATION_PORT;
+volatile const __be32 ENCAPSULATION_IP;
typedef struct {
uint64_t processed_packets_total;
int v;
};
-static volatile struct S global_variable;
+struct S global_variable = {};
struct {
__uint(type, BPF_MAP_TYPE_ARRAY);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+
+__u32 set_pid = 0;
+__u64 set_key = 0;
+__u64 set_value = 0;
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(max_entries, 2);
+ __type(key, __u64);
+ __type(value, __u64);
+} hash_map SEC(".maps");
+
+SEC("tp/syscalls/sys_enter_getpgid")
+int bpf_lookup_and_delete_test(const void *ctx)
+{
+ if (set_pid == bpf_get_current_pid_tgid() >> 32)
+ bpf_map_update_elem(&hash_map, &set_key, &set_value, BPF_NOEXIST);
+
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Check if we can migrate child sockets.
+ *
+ * 1. If reuse_md->migrating_sk is NULL (SYN packet),
+ * return SK_PASS without selecting a listener.
+ * 2. If reuse_md->migrating_sk is not NULL (socket migration),
+ * select a listener (reuseport_map[migrate_map[cookie]])
+ *
+ * Author: Kuniyuki Iwashima <kuniyu@amazon.co.jp>
+ */
+
+#include <stddef.h>
+#include <string.h>
+#include <linux/bpf.h>
+#include <linux/if_ether.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include <linux/tcp.h>
+#include <linux/in.h>
+#include <bpf/bpf_endian.h>
+#include <bpf/bpf_helpers.h>
+
+struct {
+ __uint(type, BPF_MAP_TYPE_REUSEPORT_SOCKARRAY);
+ __uint(max_entries, 256);
+ __type(key, int);
+ __type(value, __u64);
+} reuseport_map SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(max_entries, 256);
+ __type(key, __u64);
+ __type(value, int);
+} migrate_map SEC(".maps");
+
+int migrated_at_close = 0;
+int migrated_at_close_fastopen = 0;
+int migrated_at_send_synack = 0;
+int migrated_at_recv_ack = 0;
+__be16 server_port;
+
+SEC("xdp")
+int drop_ack(struct xdp_md *xdp)
+{
+ void *data_end = (void *)(long)xdp->data_end;
+ void *data = (void *)(long)xdp->data;
+ struct ethhdr *eth = data;
+ struct tcphdr *tcp = NULL;
+
+ if (eth + 1 > data_end)
+ goto pass;
+
+ switch (bpf_ntohs(eth->h_proto)) {
+ case ETH_P_IP: {
+ struct iphdr *ip = (struct iphdr *)(eth + 1);
+
+ if (ip + 1 > data_end)
+ goto pass;
+
+ if (ip->protocol != IPPROTO_TCP)
+ goto pass;
+
+ tcp = (struct tcphdr *)((void *)ip + ip->ihl * 4);
+ break;
+ }
+ case ETH_P_IPV6: {
+ struct ipv6hdr *ipv6 = (struct ipv6hdr *)(eth + 1);
+
+ if (ipv6 + 1 > data_end)
+ goto pass;
+
+ if (ipv6->nexthdr != IPPROTO_TCP)
+ goto pass;
+
+ tcp = (struct tcphdr *)(ipv6 + 1);
+ break;
+ }
+ default:
+ goto pass;
+ }
+
+ if (tcp + 1 > data_end)
+ goto pass;
+
+ if (tcp->dest != server_port)
+ goto pass;
+
+ if (!tcp->syn && tcp->ack)
+ return XDP_DROP;
+
+pass:
+ return XDP_PASS;
+}
+
+SEC("sk_reuseport/migrate")
+int migrate_reuseport(struct sk_reuseport_md *reuse_md)
+{
+ int *key, flags = 0, state, err;
+ __u64 cookie;
+
+ if (!reuse_md->migrating_sk)
+ return SK_PASS;
+
+ state = reuse_md->migrating_sk->state;
+ cookie = bpf_get_socket_cookie(reuse_md->sk);
+
+ key = bpf_map_lookup_elem(&migrate_map, &cookie);
+ if (!key)
+ return SK_DROP;
+
+ err = bpf_sk_select_reuseport(reuse_md, &reuseport_map, key, flags);
+ if (err)
+ return SK_PASS;
+
+ switch (state) {
+ case BPF_TCP_ESTABLISHED:
+ __sync_fetch_and_add(&migrated_at_close, 1);
+ break;
+ case BPF_TCP_SYN_RECV:
+ __sync_fetch_and_add(&migrated_at_close_fastopen, 1);
+ break;
+ case BPF_TCP_NEW_SYN_RECV:
+ if (!reuse_md->len)
+ __sync_fetch_and_add(&migrated_at_send_synack, 1);
+ else
+ __sync_fetch_and_add(&migrated_at_recv_ack, 1);
+ break;
+ }
+
+ return SK_PASS;
+}
+
+char _license[] SEC("license") = "GPL";
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
-static volatile const struct {
+const struct {
unsigned a[4];
/*
* if the struct's size is multiple of 16, compiler will put it into
char _y;
} rdonly_values = { .a = {2, 3, 4, 5} };
-static volatile struct {
+struct {
unsigned did_run;
unsigned iters;
unsigned sum;
-} res;
+} res = {};
SEC("raw_tracepoint/sys_enter:skip_loop")
int skip_loop(struct pt_regs *ctx)
/* inner state */
long seq = 0;
-SEC("tp/syscalls/sys_enter_getpgid")
+SEC("fentry/__x64_sys_getpgid")
int test_ringbuf(void *ctx)
{
int cur_pid = bpf_get_current_pid_tgid() >> 32;
sample = bpf_ringbuf_reserve(&ringbuf, sizeof(*sample), 0);
if (!sample) {
__sync_fetch_and_add(&dropped, 1);
- return 1;
+ return 0;
}
sample->pid = pid;
bool bpf_syscall = 0;
int kern_ver = 0;
+struct s out5 = {};
+
SEC("raw_tp/sys_enter")
int handler(const void *ctx)
{
- static volatile struct s out5;
-
out1 = in1;
out2 = in2;
out3 = in3;
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
-#include <bpf/bpf_tracing.h>
__u32 pid = 0;
#include <bpf/bpf_helpers.h>
/* The format string is filled from the userspace such that loading fails */
-static const char fmt[10];
+const char fmt[10];
SEC("raw_tp/sys_enter")
int handler(const void *ctx)
__type(value, unsigned int);
} verdict_map SEC(".maps");
-static volatile bool test_sockmap; /* toggled by user-space */
-static volatile bool test_ingress; /* toggled by user-space */
+bool test_sockmap = false; /* toggled by user-space */
+bool test_ingress = false; /* toggled by user-space */
SEC("sk_skb/stream_parser")
int prog_stream_parser(struct __sk_buff *skb)
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
-/* 8-byte aligned .bss */
-static volatile long static_var1;
-static volatile int static_var11;
-int var1 = 0;
+/* 8-byte aligned .data */
+static volatile long static_var1 = 2;
+static volatile int static_var2 = 3;
+int var1 = -1;
/* 4-byte aligned .rodata */
const volatile int rovar1;
SEC("raw_tp/sys_enter")
int handler1(const void *ctx)
{
- var1 = subprog(rovar1) + static_var1 + static_var11;
+ var1 = subprog(rovar1) + static_var1 + static_var2;
return 0;
}
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
-/* 4-byte aligned .bss */
-static volatile int static_var2;
-static volatile int static_var22;
-int var2 = 0;
+/* 4-byte aligned .data */
+static volatile int static_var1 = 5;
+static volatile int static_var2 = 6;
+int var2 = -1;
/* 8-byte aligned .rodata */
const volatile long rovar2;
SEC("raw_tp/sys_enter")
int handler2(const void *ctx)
{
- var2 = subprog(rovar2) + static_var2 + static_var22;
+ var2 = subprog(rovar2) + static_var1 + static_var2;
return 0;
}
const char LICENSE[] SEC("license") = "GPL";
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, __u32);
+ __type(value, __u64);
+} array SEC(".maps");
+
__noinline int sub1(int x)
{
+ int key = 0;
+
+ bpf_map_lookup_elem(&array, &key);
return x + 1;
}
static __noinline int sub4(int w)
{
+ int key = 0;
+
+ bpf_map_lookup_elem(&array, &key);
return w + sub3(5) + sub1(6);
}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+/* Dummy prog to test TC-BPF API */
+
+SEC("classifier")
+int cls(struct __sk_buff *skb)
+{
+ return 0;
+}
a.s6_addr32[3] == b.s6_addr32[3])
#endif
-enum {
- dev_src,
- dev_dst,
-};
-
-struct bpf_map_def SEC("maps") ifindex_map = {
- .type = BPF_MAP_TYPE_ARRAY,
- .key_size = sizeof(int),
- .value_size = sizeof(int),
- .max_entries = 2,
-};
+volatile const __u32 IFINDEX_SRC;
+volatile const __u32 IFINDEX_DST;
static __always_inline bool is_remote_ep_v4(struct __sk_buff *skb,
__be32 addr)
return v6_equal(ip6h->daddr, addr);
}
-static __always_inline int get_dev_ifindex(int which)
-{
- int *ifindex = bpf_map_lookup_elem(&ifindex_map, &which);
-
- return ifindex ? *ifindex : 0;
-}
-
-SEC("chk_egress") int tc_chk(struct __sk_buff *skb)
+SEC("classifier/chk_egress")
+int tc_chk(struct __sk_buff *skb)
{
void *data_end = ctx_ptr(skb->data_end);
void *data = ctx_ptr(skb->data);
return !raw[0] && !raw[1] && !raw[2] ? TC_ACT_SHOT : TC_ACT_OK;
}
-SEC("dst_ingress") int tc_dst(struct __sk_buff *skb)
+SEC("classifier/dst_ingress")
+int tc_dst(struct __sk_buff *skb)
{
__u8 zero[ETH_ALEN * 2];
bool redirect = false;
if (bpf_skb_store_bytes(skb, 0, &zero, sizeof(zero), 0) < 0)
return TC_ACT_SHOT;
- return bpf_redirect_neigh(get_dev_ifindex(dev_src), NULL, 0, 0);
+ return bpf_redirect_neigh(IFINDEX_SRC, NULL, 0, 0);
}
-SEC("src_ingress") int tc_src(struct __sk_buff *skb)
+SEC("classifier/src_ingress")
+int tc_src(struct __sk_buff *skb)
{
__u8 zero[ETH_ALEN * 2];
bool redirect = false;
if (bpf_skb_store_bytes(skb, 0, &zero, sizeof(zero), 0) < 0)
return TC_ACT_SHOT;
- return bpf_redirect_neigh(get_dev_ifindex(dev_dst), NULL, 0, 0);
+ return bpf_redirect_neigh(IFINDEX_DST, NULL, 0, 0);
}
char __license[] SEC("license") = "GPL";
return 0;
}
-SEC("chk_egress") int tc_chk(struct __sk_buff *skb)
+SEC("classifier/chk_egress")
+int tc_chk(struct __sk_buff *skb)
{
void *data_end = ctx_ptr(skb->data_end);
void *data = ctx_ptr(skb->data);
/* these are identical, but keep them separate for compatibility with the
* section names expected by test_tc_redirect.sh
*/
-SEC("dst_ingress") int tc_dst(struct __sk_buff *skb)
+SEC("classifier/dst_ingress")
+int tc_dst(struct __sk_buff *skb)
{
return tc_redir(skb);
}
-SEC("src_ingress") int tc_src(struct __sk_buff *skb)
+SEC("classifier/src_ingress")
+int tc_src(struct __sk_buff *skb)
{
return tc_redir(skb);
}
#include <linux/bpf.h>
#include <linux/stddef.h>
#include <linux/pkt_cls.h>
+#include <linux/if_ether.h>
+#include <linux/ip.h>
#include <bpf/bpf_helpers.h>
-enum {
- dev_src,
- dev_dst,
-};
+volatile const __u32 IFINDEX_SRC;
+volatile const __u32 IFINDEX_DST;
-struct bpf_map_def SEC("maps") ifindex_map = {
- .type = BPF_MAP_TYPE_ARRAY,
- .key_size = sizeof(int),
- .value_size = sizeof(int),
- .max_entries = 2,
-};
+static const __u8 src_mac[] = {0x00, 0x11, 0x22, 0x33, 0x44, 0x55};
+static const __u8 dst_mac[] = {0x00, 0x22, 0x33, 0x44, 0x55, 0x66};
-static __always_inline int get_dev_ifindex(int which)
+SEC("classifier/chk_egress")
+int tc_chk(struct __sk_buff *skb)
{
- int *ifindex = bpf_map_lookup_elem(&ifindex_map, &which);
+ return TC_ACT_SHOT;
+}
- return ifindex ? *ifindex : 0;
+SEC("classifier/dst_ingress")
+int tc_dst(struct __sk_buff *skb)
+{
+ return bpf_redirect_peer(IFINDEX_SRC, 0);
}
-SEC("chk_egress") int tc_chk(struct __sk_buff *skb)
+SEC("classifier/src_ingress")
+int tc_src(struct __sk_buff *skb)
{
- return TC_ACT_SHOT;
+ return bpf_redirect_peer(IFINDEX_DST, 0);
}
-SEC("dst_ingress") int tc_dst(struct __sk_buff *skb)
+SEC("classifier/dst_ingress_l3")
+int tc_dst_l3(struct __sk_buff *skb)
{
- return bpf_redirect_peer(get_dev_ifindex(dev_src), 0);
+ return bpf_redirect(IFINDEX_SRC, 0);
}
-SEC("src_ingress") int tc_src(struct __sk_buff *skb)
+SEC("classifier/src_ingress_l3")
+int tc_src_l3(struct __sk_buff *skb)
{
- return bpf_redirect_peer(get_dev_ifindex(dev_dst), 0);
+ __u16 proto = skb->protocol;
+
+ if (bpf_skb_change_head(skb, ETH_HLEN, 0) != 0)
+ return TC_ACT_SHOT;
+
+ if (bpf_skb_store_bytes(skb, 0, &src_mac, ETH_ALEN, 0) != 0)
+ return TC_ACT_SHOT;
+
+ if (bpf_skb_store_bytes(skb, ETH_ALEN, &dst_mac, ETH_ALEN, 0) != 0)
+ return TC_ACT_SHOT;
+
+ if (bpf_skb_store_bytes(skb, ETH_ALEN + ETH_ALEN, &proto, sizeof(__u16), 0) != 0)
+ return TC_ACT_SHOT;
+
+ return bpf_redirect_peer(IFINDEX_DST, 0);
}
char __license[] SEC("license") = "GPL";
int trace_printk_ret = 0;
int trace_printk_ran = 0;
-SEC("tp/raw_syscalls/sys_enter")
+const char fmt[] = "Testing,testing %d\n";
+
+SEC("fentry/__x64_sys_nanosleep")
int sys_enter(void *ctx)
{
- static const char fmt[] = "testing,testing %d\n";
-
trace_printk_ret = bpf_trace_printk(fmt, sizeof(fmt),
++trace_printk_ran);
return 0;
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+#define KBUILD_MODNAME "foo"
+#include <string.h>
+#include <linux/in.h>
+#include <linux/if_ether.h>
+#include <linux/if_packet.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_endian.h>
+
+/* One map use devmap, another one use devmap_hash for testing */
+struct {
+ __uint(type, BPF_MAP_TYPE_DEVMAP);
+ __uint(key_size, sizeof(int));
+ __uint(value_size, sizeof(int));
+ __uint(max_entries, 1024);
+} map_all SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_DEVMAP_HASH);
+ __uint(key_size, sizeof(int));
+ __uint(value_size, sizeof(struct bpf_devmap_val));
+ __uint(max_entries, 128);
+} map_egress SEC(".maps");
+
+/* map to store egress interfaces mac addresses */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, __u32);
+ __type(value, __be64);
+ __uint(max_entries, 128);
+} mac_map SEC(".maps");
+
+SEC("xdp_redirect_map_multi")
+int xdp_redirect_map_multi_prog(struct xdp_md *ctx)
+{
+ void *data_end = (void *)(long)ctx->data_end;
+ void *data = (void *)(long)ctx->data;
+ int if_index = ctx->ingress_ifindex;
+ struct ethhdr *eth = data;
+ __u16 h_proto;
+ __u64 nh_off;
+
+ nh_off = sizeof(*eth);
+ if (data + nh_off > data_end)
+ return XDP_DROP;
+
+ h_proto = eth->h_proto;
+
+ /* Using IPv4 for (BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS) testing */
+ if (h_proto == bpf_htons(ETH_P_IP))
+ return bpf_redirect_map(&map_all, 0,
+ BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS);
+ /* Using IPv6 for none flag testing */
+ else if (h_proto == bpf_htons(ETH_P_IPV6))
+ return bpf_redirect_map(&map_all, if_index, 0);
+ /* All others for BPF_F_BROADCAST testing */
+ else
+ return bpf_redirect_map(&map_all, 0, BPF_F_BROADCAST);
+}
+
+/* The following 2 progs are for 2nd devmap prog testing */
+SEC("xdp_redirect_map_ingress")
+int xdp_redirect_map_all_prog(struct xdp_md *ctx)
+{
+ return bpf_redirect_map(&map_egress, 0,
+ BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS);
+}
+
+SEC("xdp_devmap/map_prog")
+int xdp_devmap_prog(struct xdp_md *ctx)
+{
+ void *data_end = (void *)(long)ctx->data_end;
+ void *data = (void *)(long)ctx->data;
+ __u32 key = ctx->egress_ifindex;
+ struct ethhdr *eth = data;
+ __u64 nh_off;
+ __be64 *mac;
+
+ nh_off = sizeof(*eth);
+ if (data + nh_off > data_end)
+ return XDP_DROP;
+
+ mac = bpf_map_lookup_elem(&mac_map, &key);
+ if (mac)
+ __builtin_memcpy(eth->h_source, mac, ETH_ALEN);
+
+ return XDP_PASS;
+}
+
+char _license[] SEC("license") = "GPL";
#!/bin/bash
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+set -e
# Assume script is located under tools/testing/selftests/bpf/. We want to start
# build attempts from the top of kernel repository.
assert(bpf_map_lookup_elem(lru_map_fd, &key, value) == -1 &&
errno == ENOENT);
+ /* lookup elem key=1 and delete it, then check it doesn't exist */
+ key = 1;
+ assert(!bpf_map_lookup_and_delete_elem(lru_map_fd, &key, &value));
+ assert(value[0] == 1234);
+
+ /* remove the same element from the expected map */
+ assert(!bpf_map_delete_elem(expected_map_fd, &key));
+
assert(map_equal(lru_map_fd, expected_map_fd));
close(expected_map_fd);
value = 0;
/* BPF_NOEXIST means add new element if it doesn't exist. */
- assert(bpf_map_update_elem(fd, &key, &value, BPF_NOEXIST) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, &value, BPF_NOEXIST) < 0 &&
/* key=1 already exists. */
errno == EEXIST);
/* -1 is an invalid flag. */
- assert(bpf_map_update_elem(fd, &key, &value, -1) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, &value, -1) < 0 &&
errno == EINVAL);
/* Check that key=1 can be found. */
assert(bpf_map_lookup_elem(fd, &key, &value) == 0 && value == 1234);
key = 2;
+ value = 1234;
+ /* Insert key=2 element. */
+ assert(bpf_map_update_elem(fd, &key, &value, BPF_ANY) == 0);
+
+ /* Check that key=2 matches the value and delete it */
+ assert(bpf_map_lookup_and_delete_elem(fd, &key, &value) == 0 && value == 1234);
+
/* Check that key=2 is not found. */
- assert(bpf_map_lookup_elem(fd, &key, &value) == -1 && errno == ENOENT);
+ assert(bpf_map_lookup_elem(fd, &key, &value) < 0 && errno == ENOENT);
/* BPF_EXIST means update existing element. */
- assert(bpf_map_update_elem(fd, &key, &value, BPF_EXIST) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, &value, BPF_EXIST) < 0 &&
/* key=2 is not there. */
errno == ENOENT);
* inserted due to max_entries limit.
*/
key = 0;
- assert(bpf_map_update_elem(fd, &key, &value, BPF_NOEXIST) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, &value, BPF_NOEXIST) < 0 &&
errno == E2BIG);
/* Update existing element, though the map is full. */
key = 2;
assert(bpf_map_update_elem(fd, &key, &value, BPF_ANY) == 0);
key = 3;
- assert(bpf_map_update_elem(fd, &key, &value, BPF_NOEXIST) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, &value, BPF_NOEXIST) < 0 &&
errno == E2BIG);
/* Check that key = 0 doesn't exist. */
key = 0;
- assert(bpf_map_delete_elem(fd, &key) == -1 && errno == ENOENT);
+ assert(bpf_map_delete_elem(fd, &key) < 0 && errno == ENOENT);
/* Iterate over two elements. */
assert(bpf_map_get_next_key(fd, NULL, &first_key) == 0 &&
assert(bpf_map_get_next_key(fd, &next_key, &next_key) == 0 &&
(next_key == 1 || next_key == 2) &&
(next_key != first_key));
- assert(bpf_map_get_next_key(fd, &next_key, &next_key) == -1 &&
+ assert(bpf_map_get_next_key(fd, &next_key, &next_key) < 0 &&
errno == ENOENT);
/* Delete both elements. */
assert(bpf_map_delete_elem(fd, &key) == 0);
key = 2;
assert(bpf_map_delete_elem(fd, &key) == 0);
- assert(bpf_map_delete_elem(fd, &key) == -1 && errno == ENOENT);
+ assert(bpf_map_delete_elem(fd, &key) < 0 && errno == ENOENT);
key = 0;
/* Check that map is empty. */
- assert(bpf_map_get_next_key(fd, NULL, &next_key) == -1 &&
+ assert(bpf_map_get_next_key(fd, NULL, &next_key) < 0 &&
errno == ENOENT);
- assert(bpf_map_get_next_key(fd, &key, &next_key) == -1 &&
+ assert(bpf_map_get_next_key(fd, &key, &next_key) < 0 &&
errno == ENOENT);
close(fd);
/* Insert key=1 element. */
assert(!(expected_key_mask & key));
assert(bpf_map_update_elem(fd, &key, value, BPF_ANY) == 0);
+
+ /* Lookup and delete elem key=1 and check value. */
+ assert(bpf_map_lookup_and_delete_elem(fd, &key, value) == 0 &&
+ bpf_percpu(value,0) == 100);
+
+ for (i = 0; i < nr_cpus; i++)
+ bpf_percpu(value,i) = i + 100;
+
+ /* Insert key=1 element which should not exist. */
+ assert(bpf_map_update_elem(fd, &key, value, BPF_NOEXIST) == 0);
expected_key_mask |= key;
/* BPF_NOEXIST means add new element if it doesn't exist. */
- assert(bpf_map_update_elem(fd, &key, value, BPF_NOEXIST) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, value, BPF_NOEXIST) < 0 &&
/* key=1 already exists. */
errno == EEXIST);
/* -1 is an invalid flag. */
- assert(bpf_map_update_elem(fd, &key, value, -1) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, value, -1) < 0 &&
errno == EINVAL);
/* Check that key=1 can be found. Value could be 0 if the lookup
key = 2;
/* Check that key=2 is not found. */
- assert(bpf_map_lookup_elem(fd, &key, value) == -1 && errno == ENOENT);
+ assert(bpf_map_lookup_elem(fd, &key, value) < 0 && errno == ENOENT);
/* BPF_EXIST means update existing element. */
- assert(bpf_map_update_elem(fd, &key, value, BPF_EXIST) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, value, BPF_EXIST) < 0 &&
/* key=2 is not there. */
errno == ENOENT);
* inserted due to max_entries limit.
*/
key = 0;
- assert(bpf_map_update_elem(fd, &key, value, BPF_NOEXIST) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, value, BPF_NOEXIST) < 0 &&
errno == E2BIG);
/* Check that key = 0 doesn't exist. */
- assert(bpf_map_delete_elem(fd, &key) == -1 && errno == ENOENT);
+ assert(bpf_map_delete_elem(fd, &key) < 0 && errno == ENOENT);
/* Iterate over two elements. */
assert(bpf_map_get_next_key(fd, NULL, &first_key) == 0 &&
assert(bpf_map_delete_elem(fd, &key) == 0);
key = 2;
assert(bpf_map_delete_elem(fd, &key) == 0);
- assert(bpf_map_delete_elem(fd, &key) == -1 && errno == ENOENT);
+ assert(bpf_map_delete_elem(fd, &key) < 0 && errno == ENOENT);
key = 0;
/* Check that map is empty. */
- assert(bpf_map_get_next_key(fd, NULL, &next_key) == -1 &&
+ assert(bpf_map_get_next_key(fd, NULL, &next_key) < 0 &&
errno == ENOENT);
- assert(bpf_map_get_next_key(fd, &key, &next_key) == -1 &&
+ assert(bpf_map_get_next_key(fd, &key, &next_key) < 0 &&
errno == ENOENT);
close(fd);
assert(bpf_map_update_elem(fd, &key, &value, BPF_ANY) == 0);
value = 0;
- assert(bpf_map_update_elem(fd, &key, &value, BPF_NOEXIST) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, &value, BPF_NOEXIST) < 0 &&
errno == EEXIST);
/* Check that key=1 can be found. */
* due to max_entries limit.
*/
key = 2;
- assert(bpf_map_update_elem(fd, &key, &value, BPF_EXIST) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, &value, BPF_EXIST) < 0 &&
errno == E2BIG);
/* Check that key = 2 doesn't exist. */
- assert(bpf_map_lookup_elem(fd, &key, &value) == -1 && errno == ENOENT);
+ assert(bpf_map_lookup_elem(fd, &key, &value) < 0 && errno == ENOENT);
/* Iterate over two elements. */
assert(bpf_map_get_next_key(fd, NULL, &next_key) == 0 &&
next_key == 0);
assert(bpf_map_get_next_key(fd, &next_key, &next_key) == 0 &&
next_key == 1);
- assert(bpf_map_get_next_key(fd, &next_key, &next_key) == -1 &&
+ assert(bpf_map_get_next_key(fd, &next_key, &next_key) < 0 &&
errno == ENOENT);
/* Delete shouldn't succeed. */
key = 1;
- assert(bpf_map_delete_elem(fd, &key) == -1 && errno == EINVAL);
+ assert(bpf_map_delete_elem(fd, &key) < 0 && errno == EINVAL);
close(fd);
}
assert(bpf_map_update_elem(fd, &key, values, BPF_ANY) == 0);
bpf_percpu(values, 0) = 0;
- assert(bpf_map_update_elem(fd, &key, values, BPF_NOEXIST) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, values, BPF_NOEXIST) < 0 &&
errno == EEXIST);
/* Check that key=1 can be found. */
/* Check that key=2 cannot be inserted due to max_entries limit. */
key = 2;
- assert(bpf_map_update_elem(fd, &key, values, BPF_EXIST) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, values, BPF_EXIST) < 0 &&
errno == E2BIG);
/* Check that key = 2 doesn't exist. */
- assert(bpf_map_lookup_elem(fd, &key, values) == -1 && errno == ENOENT);
+ assert(bpf_map_lookup_elem(fd, &key, values) < 0 && errno == ENOENT);
/* Iterate over two elements. */
assert(bpf_map_get_next_key(fd, NULL, &next_key) == 0 &&
next_key == 0);
assert(bpf_map_get_next_key(fd, &next_key, &next_key) == 0 &&
next_key == 1);
- assert(bpf_map_get_next_key(fd, &next_key, &next_key) == -1 &&
+ assert(bpf_map_get_next_key(fd, &next_key, &next_key) < 0 &&
errno == ENOENT);
/* Delete shouldn't succeed. */
key = 1;
- assert(bpf_map_delete_elem(fd, &key) == -1 && errno == EINVAL);
+ assert(bpf_map_delete_elem(fd, &key) < 0 && errno == EINVAL);
close(fd);
}
assert(bpf_map_update_elem(fd, NULL, &vals[i], 0) == 0);
/* Check that element cannot be pushed due to max_entries limit */
- assert(bpf_map_update_elem(fd, NULL, &val, 0) == -1 &&
+ assert(bpf_map_update_elem(fd, NULL, &val, 0) < 0 &&
errno == E2BIG);
/* Peek element */
val == vals[i]);
/* Check that there are not elements left */
- assert(bpf_map_lookup_and_delete_elem(fd, NULL, &val) == -1 &&
+ assert(bpf_map_lookup_and_delete_elem(fd, NULL, &val) < 0 &&
errno == ENOENT);
/* Check that non supported functions set errno to EINVAL */
- assert(bpf_map_delete_elem(fd, NULL) == -1 && errno == EINVAL);
- assert(bpf_map_get_next_key(fd, NULL, NULL) == -1 && errno == EINVAL);
+ assert(bpf_map_delete_elem(fd, NULL) < 0 && errno == EINVAL);
+ assert(bpf_map_get_next_key(fd, NULL, NULL) < 0 && errno == EINVAL);
close(fd);
}
assert(bpf_map_update_elem(fd, NULL, &vals[i], 0) == 0);
/* Check that element cannot be pushed due to max_entries limit */
- assert(bpf_map_update_elem(fd, NULL, &val, 0) == -1 &&
+ assert(bpf_map_update_elem(fd, NULL, &val, 0) < 0 &&
errno == E2BIG);
/* Peek element */
val == vals[i]);
/* Check that there are not elements left */
- assert(bpf_map_lookup_and_delete_elem(fd, NULL, &val) == -1 &&
+ assert(bpf_map_lookup_and_delete_elem(fd, NULL, &val) < 0 &&
errno == ENOENT);
/* Check that non supported functions set errno to EINVAL */
- assert(bpf_map_delete_elem(fd, NULL) == -1 && errno == EINVAL);
- assert(bpf_map_get_next_key(fd, NULL, NULL) == -1 && errno == EINVAL);
+ assert(bpf_map_delete_elem(fd, NULL) < 0 && errno == EINVAL);
+ assert(bpf_map_get_next_key(fd, NULL, NULL) < 0 && errno == EINVAL);
close(fd);
}
}
bpf_map_rx = bpf_object__find_map_by_name(obj, "sock_map_rx");
- if (IS_ERR(bpf_map_rx)) {
+ if (!bpf_map_rx) {
printf("Failed to load map rx from verdict prog\n");
goto out_sockmap;
}
}
bpf_map_tx = bpf_object__find_map_by_name(obj, "sock_map_tx");
- if (IS_ERR(bpf_map_tx)) {
+ if (!bpf_map_tx) {
printf("Failed to load map tx from verdict prog\n");
goto out_sockmap;
}
}
bpf_map_msg = bpf_object__find_map_by_name(obj, "sock_map_msg");
- if (IS_ERR(bpf_map_msg)) {
+ if (!bpf_map_msg) {
printf("Failed to load map msg from msg_verdict prog\n");
goto out_sockmap;
}
}
bpf_map_break = bpf_object__find_map_by_name(obj, "sock_map_break");
- if (IS_ERR(bpf_map_break)) {
+ if (!bpf_map_break) {
printf("Failed to load map tx from verdict prog\n");
goto out_sockmap;
}
}
map = bpf_object__find_map_by_name(obj, "mim_array");
- if (IS_ERR(map)) {
+ if (!map) {
printf("Failed to load array of maps from test prog\n");
goto out_map_in_map;
}
}
map = bpf_object__find_map_by_name(obj, "mim_hash");
- if (IS_ERR(map)) {
+ if (!map) {
printf("Failed to load hash of maps from test prog\n");
goto out_map_in_map;
}
bpf_object__load(obj);
map = bpf_object__find_map_by_name(obj, "mim_array");
- if (IS_ERR(map)) {
+ if (!map) {
printf("Failed to load array of maps from test prog\n");
goto out_map_in_map;
}
}
map = bpf_object__find_map_by_name(obj, "mim_hash");
- if (IS_ERR(map)) {
+ if (!map) {
printf("Failed to load hash of maps from test prog\n");
goto out_map_in_map;
}
}
key.c = -1;
- assert(bpf_map_update_elem(fd, &key, &value, BPF_NOEXIST) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, &value, BPF_NOEXIST) < 0 &&
errno == E2BIG);
/* Iterate through all elements. */
key.c = -1;
for (i = 0; i < MAP_SIZE; i++)
assert(bpf_map_get_next_key(fd, &key, &key) == 0);
- assert(bpf_map_get_next_key(fd, &key, &key) == -1 && errno == ENOENT);
+ assert(bpf_map_get_next_key(fd, &key, &key) < 0 && errno == ENOENT);
key.c = 0;
assert(bpf_map_lookup_elem(fd, &key, &value) == 0 && value == 0);
key.a = 1;
- assert(bpf_map_lookup_elem(fd, &key, &value) == -1 && errno == ENOENT);
+ assert(bpf_map_lookup_elem(fd, &key, &value) < 0 && errno == ENOENT);
close(fd);
}
run_parallel(TASKS, test_update_delete, data);
/* Check that key=0 is already there. */
- assert(bpf_map_update_elem(fd, &key, &value, BPF_NOEXIST) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, &value, BPF_NOEXIST) < 0 &&
errno == EEXIST);
/* Check that all elements were inserted. */
key = -1;
for (i = 0; i < MAP_SIZE; i++)
assert(bpf_map_get_next_key(fd, &key, &key) == 0);
- assert(bpf_map_get_next_key(fd, &key, &key) == -1 && errno == ENOENT);
+ assert(bpf_map_get_next_key(fd, &key, &key) < 0 && errno == ENOENT);
/* Another check for all elements */
for (i = 0; i < MAP_SIZE; i++) {
/* Nothing should be left. */
key = -1;
- assert(bpf_map_get_next_key(fd, NULL, &key) == -1 && errno == ENOENT);
- assert(bpf_map_get_next_key(fd, &key, &key) == -1 && errno == ENOENT);
+ assert(bpf_map_get_next_key(fd, NULL, &key) < 0 && errno == ENOENT);
+ assert(bpf_map_get_next_key(fd, &key, &key) < 0 && errno == ENOENT);
}
static void test_map_rdonly(void)
key = 1;
value = 1234;
/* Try to insert key=1 element. */
- assert(bpf_map_update_elem(fd, &key, &value, BPF_ANY) == -1 &&
+ assert(bpf_map_update_elem(fd, &key, &value, BPF_ANY) < 0 &&
errno == EPERM);
/* Check that key=1 is not found. */
- assert(bpf_map_lookup_elem(fd, &key, &value) == -1 && errno == ENOENT);
- assert(bpf_map_get_next_key(fd, &key, &value) == -1 && errno == ENOENT);
+ assert(bpf_map_lookup_elem(fd, &key, &value) < 0 && errno == ENOENT);
+ assert(bpf_map_get_next_key(fd, &key, &value) < 0 && errno == ENOENT);
close(fd);
}
assert(bpf_map_update_elem(fd, &key, &value, BPF_ANY) == 0);
/* Check that reading elements and keys from the map is not allowed. */
- assert(bpf_map_lookup_elem(fd, &key, &value) == -1 && errno == EPERM);
- assert(bpf_map_get_next_key(fd, &key, &value) == -1 && errno == EPERM);
+ assert(bpf_map_lookup_elem(fd, &key, &value) < 0 && errno == EPERM);
+ assert(bpf_map_get_next_key(fd, &key, &value) < 0 && errno == EPERM);
close(fd);
}
assert(bpf_map_update_elem(fd, NULL, &value, BPF_ANY) == 0);
/* Peek element should fail */
- assert(bpf_map_lookup_elem(fd, NULL, &value) == -1 && errno == EPERM);
+ assert(bpf_map_lookup_elem(fd, NULL, &value) < 0 && errno == EPERM);
/* Pop element should fail */
- assert(bpf_map_lookup_and_delete_elem(fd, NULL, &value) == -1 &&
+ assert(bpf_map_lookup_and_delete_elem(fd, NULL, &value) < 0 &&
errno == EPERM);
close(fd);
value = &fd32;
}
err = bpf_map_update_elem(map_fd, &index0, value, BPF_ANY);
- CHECK(err != -1 || errno != EINVAL,
+ CHECK(err >= 0 || errno != EINVAL,
"reuseport array update unbound sk",
"sock_type:%d err:%d errno:%d\n",
type, err, errno);
*/
err = bpf_map_update_elem(map_fd, &index0, value,
BPF_ANY);
- CHECK(err != -1 || errno != EINVAL,
+ CHECK(err >= 0 || errno != EINVAL,
"reuseport array update non-listening sk",
"sock_type:%d err:%d errno:%d\n",
type, err, errno);
map_fd = bpf_create_map(BPF_MAP_TYPE_REUSEPORT_SOCKARRAY,
sizeof(__u32), sizeof(__u64), array_size, 0);
- CHECK(map_fd == -1, "reuseport array create",
+ CHECK(map_fd < 0, "reuseport array create",
"map_fd:%d, errno:%d\n", map_fd, errno);
/* Test lookup/update/delete with invalid index */
err = bpf_map_delete_elem(map_fd, &bad_index);
- CHECK(err != -1 || errno != E2BIG, "reuseport array del >=max_entries",
+ CHECK(err >= 0 || errno != E2BIG, "reuseport array del >=max_entries",
"err:%d errno:%d\n", err, errno);
err = bpf_map_update_elem(map_fd, &bad_index, &fd64, BPF_ANY);
- CHECK(err != -1 || errno != E2BIG,
+ CHECK(err >= 0 || errno != E2BIG,
"reuseport array update >=max_entries",
"err:%d errno:%d\n", err, errno);
err = bpf_map_lookup_elem(map_fd, &bad_index, &map_cookie);
- CHECK(err != -1 || errno != ENOENT,
+ CHECK(err >= 0 || errno != ENOENT,
"reuseport array update >=max_entries",
"err:%d errno:%d\n", err, errno);
/* Test lookup/delete non existence elem */
err = bpf_map_lookup_elem(map_fd, &index3, &map_cookie);
- CHECK(err != -1 || errno != ENOENT,
+ CHECK(err >= 0 || errno != ENOENT,
"reuseport array lookup not-exist elem",
"err:%d errno:%d\n", err, errno);
err = bpf_map_delete_elem(map_fd, &index3);
- CHECK(err != -1 || errno != ENOENT,
+ CHECK(err >= 0 || errno != ENOENT,
"reuseport array del not-exist elem",
"err:%d errno:%d\n", err, errno);
/* BPF_EXIST failure case */
err = bpf_map_update_elem(map_fd, &index3, &grpa_fds64[fds_idx],
BPF_EXIST);
- CHECK(err != -1 || errno != ENOENT,
+ CHECK(err >= 0 || errno != ENOENT,
"reuseport array update empty elem BPF_EXIST",
"sock_type:%d err:%d errno:%d\n",
type, err, errno);
/* BPF_NOEXIST success case */
err = bpf_map_update_elem(map_fd, &index3, &grpa_fds64[fds_idx],
BPF_NOEXIST);
- CHECK(err == -1,
+ CHECK(err < 0,
"reuseport array update empty elem BPF_NOEXIST",
"sock_type:%d err:%d errno:%d\n",
type, err, errno);
/* BPF_EXIST success case. */
err = bpf_map_update_elem(map_fd, &index3, &grpa_fds64[fds_idx],
BPF_EXIST);
- CHECK(err == -1,
+ CHECK(err < 0,
"reuseport array update same elem BPF_EXIST",
"sock_type:%d err:%d errno:%d\n", type, err, errno);
fds_idx = REUSEPORT_FD_IDX(err, fds_idx);
/* BPF_NOEXIST failure case */
err = bpf_map_update_elem(map_fd, &index3, &grpa_fds64[fds_idx],
BPF_NOEXIST);
- CHECK(err != -1 || errno != EEXIST,
+ CHECK(err >= 0 || errno != EEXIST,
"reuseport array update non-empty elem BPF_NOEXIST",
"sock_type:%d err:%d errno:%d\n",
type, err, errno);
/* BPF_ANY case (always succeed) */
err = bpf_map_update_elem(map_fd, &index3, &grpa_fds64[fds_idx],
BPF_ANY);
- CHECK(err == -1,
+ CHECK(err < 0,
"reuseport array update same sk with BPF_ANY",
"sock_type:%d err:%d errno:%d\n", type, err, errno);
/* The same sk cannot be added to reuseport_array twice */
err = bpf_map_update_elem(map_fd, &index3, &fd64, BPF_ANY);
- CHECK(err != -1 || errno != EBUSY,
+ CHECK(err >= 0 || errno != EBUSY,
"reuseport array update same sk with same index",
"sock_type:%d err:%d errno:%d\n",
type, err, errno);
err = bpf_map_update_elem(map_fd, &index0, &fd64, BPF_ANY);
- CHECK(err != -1 || errno != EBUSY,
+ CHECK(err >= 0 || errno != EBUSY,
"reuseport array update same sk with different index",
"sock_type:%d err:%d errno:%d\n",
type, err, errno);
/* Test delete elem */
err = bpf_map_delete_elem(map_fd, &index3);
- CHECK(err == -1, "reuseport array delete sk",
+ CHECK(err < 0, "reuseport array delete sk",
"sock_type:%d err:%d errno:%d\n",
type, err, errno);
/* Add it back with BPF_NOEXIST */
err = bpf_map_update_elem(map_fd, &index3, &fd64, BPF_NOEXIST);
- CHECK(err == -1,
+ CHECK(err < 0,
"reuseport array re-add with BPF_NOEXIST after del",
"sock_type:%d err:%d errno:%d\n", type, err, errno);
/* Test cookie */
err = bpf_map_lookup_elem(map_fd, &index3, &map_cookie);
- CHECK(err == -1 || sk_cookie != map_cookie,
+ CHECK(err < 0 || sk_cookie != map_cookie,
"reuseport array lookup re-added sk",
"sock_type:%d err:%d errno:%d sk_cookie:0x%llx map_cookie:0x%llxn",
type, err, errno, sk_cookie, map_cookie);
for (f = 0; f < ARRAY_SIZE(grpa_fds64); f++)
close(grpa_fds64[f]);
err = bpf_map_lookup_elem(map_fd, &index3, &map_cookie);
- CHECK(err != -1 || errno != ENOENT,
+ CHECK(err >= 0 || errno != ENOENT,
"reuseport array lookup after close()",
"sock_type:%d err:%d errno:%d\n",
type, err, errno);
CHECK(fd64 == -1, "socket(SOCK_RAW)", "err:%d errno:%d\n",
err, errno);
err = bpf_map_update_elem(map_fd, &index3, &fd64, BPF_NOEXIST);
- CHECK(err != -1 || errno != ENOTSUPP, "reuseport array update SOCK_RAW",
+ CHECK(err >= 0 || errno != ENOTSUPP, "reuseport array update SOCK_RAW",
"err:%d errno:%d\n", err, errno);
close(fd64);
/* Test 32 bit fd */
map_fd = bpf_create_map(BPF_MAP_TYPE_REUSEPORT_SOCKARRAY,
sizeof(__u32), sizeof(__u32), array_size, 0);
- CHECK(map_fd == -1, "reuseport array create",
+ CHECK(map_fd < 0, "reuseport array create",
"map_fd:%d, errno:%d\n", map_fd, errno);
prepare_reuseport_grp(SOCK_STREAM, map_fd, sizeof(__u32), &fd64,
&sk_cookie, 1);
fd = fd64;
err = bpf_map_update_elem(map_fd, &index3, &fd, BPF_NOEXIST);
- CHECK(err == -1, "reuseport array update 32 bit fd",
+ CHECK(err < 0, "reuseport array update 32 bit fd",
"err:%d errno:%d\n", err, errno);
err = bpf_map_lookup_elem(map_fd, &index3, &map_cookie);
- CHECK(err != -1 || errno != ENOSPC,
+ CHECK(err >= 0 || errno != ENOSPC,
"reuseport array lookup 32 bit fd",
"err:%d errno:%d\n", err, errno);
close(fd);
{
srand(time(NULL));
+ libbpf_set_strict_mode(LIBBPF_STRICT_ALL);
+
map_flags = 0;
run_all_tests();
if (err)
return err;
+ /* Use libbpf 1.0 API mode */
+ libbpf_set_strict_mode(LIBBPF_STRICT_ALL);
+
libbpf_set_print(libbpf_print_fn);
srand(time(NULL));
#define ASSERT_OK_PTR(ptr, name) ({ \
static int duration = 0; \
const void *___res = (ptr); \
- bool ___ok = !IS_ERR_OR_NULL(___res); \
- CHECK(!___ok, (name), \
- "unexpected error: %ld\n", PTR_ERR(___res)); \
+ int ___err = libbpf_get_error(___res); \
+ bool ___ok = ___err == 0; \
+ CHECK(!___ok, (name), "unexpected error: %d\n", ___err); \
___ok; \
})
#define ASSERT_ERR_PTR(ptr, name) ({ \
static int duration = 0; \
const void *___res = (ptr); \
- bool ___ok = IS_ERR(___res); \
+ int ___err = libbpf_get_error(___res); \
+ bool ___ok = ___err != 0; \
CHECK(!___ok, (name), "unexpected pointer: %p\n", ___res); \
___ok; \
})
+++ /dev/null
-#!/bin/bash
-# SPDX-License-Identifier: GPL-2.0
-#
-# This test sets up 3 netns (src <-> fwd <-> dst). There is no direct veth link
-# between src and dst. The netns fwd has veth links to each src and dst. The
-# client is in src and server in dst. The test installs a TC BPF program to each
-# host facing veth in fwd which calls into i) bpf_redirect_neigh() to perform the
-# neigh addr population and redirect or ii) bpf_redirect_peer() for namespace
-# switch from ingress side; it also installs a checker prog on the egress side
-# to drop unexpected traffic.
-
-if [[ $EUID -ne 0 ]]; then
- echo "This script must be run as root"
- echo "FAIL"
- exit 1
-fi
-
-# check that needed tools are present
-command -v nc >/dev/null 2>&1 || \
- { echo >&2 "nc is not available"; exit 1; }
-command -v dd >/dev/null 2>&1 || \
- { echo >&2 "dd is not available"; exit 1; }
-command -v timeout >/dev/null 2>&1 || \
- { echo >&2 "timeout is not available"; exit 1; }
-command -v ping >/dev/null 2>&1 || \
- { echo >&2 "ping is not available"; exit 1; }
-if command -v ping6 >/dev/null 2>&1; then PING6=ping6; else PING6=ping; fi
-command -v perl >/dev/null 2>&1 || \
- { echo >&2 "perl is not available"; exit 1; }
-command -v jq >/dev/null 2>&1 || \
- { echo >&2 "jq is not available"; exit 1; }
-command -v bpftool >/dev/null 2>&1 || \
- { echo >&2 "bpftool is not available"; exit 1; }
-
-readonly GREEN='\033[0;92m'
-readonly RED='\033[0;31m'
-readonly NC='\033[0m' # No Color
-
-readonly PING_ARG="-c 3 -w 10 -q"
-
-readonly TIMEOUT=10
-
-readonly NS_SRC="ns-src-$(mktemp -u XXXXXX)"
-readonly NS_FWD="ns-fwd-$(mktemp -u XXXXXX)"
-readonly NS_DST="ns-dst-$(mktemp -u XXXXXX)"
-
-readonly IP4_SRC="172.16.1.100"
-readonly IP4_DST="172.16.2.100"
-
-readonly IP6_SRC="::1:dead:beef:cafe"
-readonly IP6_DST="::2:dead:beef:cafe"
-
-readonly IP4_SLL="169.254.0.1"
-readonly IP4_DLL="169.254.0.2"
-readonly IP4_NET="169.254.0.0"
-
-netns_cleanup()
-{
- ip netns del ${NS_SRC}
- ip netns del ${NS_FWD}
- ip netns del ${NS_DST}
-}
-
-netns_setup()
-{
- ip netns add "${NS_SRC}"
- ip netns add "${NS_FWD}"
- ip netns add "${NS_DST}"
-
- ip link add veth_src type veth peer name veth_src_fwd
- ip link add veth_dst type veth peer name veth_dst_fwd
-
- ip link set veth_src netns ${NS_SRC}
- ip link set veth_src_fwd netns ${NS_FWD}
-
- ip link set veth_dst netns ${NS_DST}
- ip link set veth_dst_fwd netns ${NS_FWD}
-
- ip -netns ${NS_SRC} addr add ${IP4_SRC}/32 dev veth_src
- ip -netns ${NS_DST} addr add ${IP4_DST}/32 dev veth_dst
-
- # The fwd netns automatically get a v6 LL address / routes, but also
- # needs v4 one in order to start ARP probing. IP4_NET route is added
- # to the endpoints so that the ARP processing will reply.
-
- ip -netns ${NS_FWD} addr add ${IP4_SLL}/32 dev veth_src_fwd
- ip -netns ${NS_FWD} addr add ${IP4_DLL}/32 dev veth_dst_fwd
-
- ip -netns ${NS_SRC} addr add ${IP6_SRC}/128 dev veth_src nodad
- ip -netns ${NS_DST} addr add ${IP6_DST}/128 dev veth_dst nodad
-
- ip -netns ${NS_SRC} link set dev veth_src up
- ip -netns ${NS_FWD} link set dev veth_src_fwd up
-
- ip -netns ${NS_DST} link set dev veth_dst up
- ip -netns ${NS_FWD} link set dev veth_dst_fwd up
-
- ip -netns ${NS_SRC} route add ${IP4_DST}/32 dev veth_src scope global
- ip -netns ${NS_SRC} route add ${IP4_NET}/16 dev veth_src scope global
- ip -netns ${NS_FWD} route add ${IP4_SRC}/32 dev veth_src_fwd scope global
-
- ip -netns ${NS_SRC} route add ${IP6_DST}/128 dev veth_src scope global
- ip -netns ${NS_FWD} route add ${IP6_SRC}/128 dev veth_src_fwd scope global
-
- ip -netns ${NS_DST} route add ${IP4_SRC}/32 dev veth_dst scope global
- ip -netns ${NS_DST} route add ${IP4_NET}/16 dev veth_dst scope global
- ip -netns ${NS_FWD} route add ${IP4_DST}/32 dev veth_dst_fwd scope global
-
- ip -netns ${NS_DST} route add ${IP6_SRC}/128 dev veth_dst scope global
- ip -netns ${NS_FWD} route add ${IP6_DST}/128 dev veth_dst_fwd scope global
-
- fmac_src=$(ip netns exec ${NS_FWD} cat /sys/class/net/veth_src_fwd/address)
- fmac_dst=$(ip netns exec ${NS_FWD} cat /sys/class/net/veth_dst_fwd/address)
-
- ip -netns ${NS_SRC} neigh add ${IP4_DST} dev veth_src lladdr $fmac_src
- ip -netns ${NS_DST} neigh add ${IP4_SRC} dev veth_dst lladdr $fmac_dst
-
- ip -netns ${NS_SRC} neigh add ${IP6_DST} dev veth_src lladdr $fmac_src
- ip -netns ${NS_DST} neigh add ${IP6_SRC} dev veth_dst lladdr $fmac_dst
-}
-
-netns_test_connectivity()
-{
- set +e
-
- ip netns exec ${NS_DST} bash -c "nc -4 -l -p 9004 &"
- ip netns exec ${NS_DST} bash -c "nc -6 -l -p 9006 &"
-
- TEST="TCPv4 connectivity test"
- ip netns exec ${NS_SRC} bash -c "timeout ${TIMEOUT} dd if=/dev/zero bs=1000 count=100 > /dev/tcp/${IP4_DST}/9004"
- if [ $? -ne 0 ]; then
- echo -e "${TEST}: ${RED}FAIL${NC}"
- exit 1
- fi
- echo -e "${TEST}: ${GREEN}PASS${NC}"
-
- TEST="TCPv6 connectivity test"
- ip netns exec ${NS_SRC} bash -c "timeout ${TIMEOUT} dd if=/dev/zero bs=1000 count=100 > /dev/tcp/${IP6_DST}/9006"
- if [ $? -ne 0 ]; then
- echo -e "${TEST}: ${RED}FAIL${NC}"
- exit 1
- fi
- echo -e "${TEST}: ${GREEN}PASS${NC}"
-
- TEST="ICMPv4 connectivity test"
- ip netns exec ${NS_SRC} ping $PING_ARG ${IP4_DST}
- if [ $? -ne 0 ]; then
- echo -e "${TEST}: ${RED}FAIL${NC}"
- exit 1
- fi
- echo -e "${TEST}: ${GREEN}PASS${NC}"
-
- TEST="ICMPv6 connectivity test"
- ip netns exec ${NS_SRC} $PING6 $PING_ARG ${IP6_DST}
- if [ $? -ne 0 ]; then
- echo -e "${TEST}: ${RED}FAIL${NC}"
- exit 1
- fi
- echo -e "${TEST}: ${GREEN}PASS${NC}"
-
- set -e
-}
-
-hex_mem_str()
-{
- perl -e 'print join(" ", unpack("(H2)8", pack("L", @ARGV)))' $1
-}
-
-netns_setup_bpf()
-{
- local obj=$1
- local use_forwarding=${2:-0}
-
- ip netns exec ${NS_FWD} tc qdisc add dev veth_src_fwd clsact
- ip netns exec ${NS_FWD} tc filter add dev veth_src_fwd ingress bpf da obj $obj sec src_ingress
- ip netns exec ${NS_FWD} tc filter add dev veth_src_fwd egress bpf da obj $obj sec chk_egress
-
- ip netns exec ${NS_FWD} tc qdisc add dev veth_dst_fwd clsact
- ip netns exec ${NS_FWD} tc filter add dev veth_dst_fwd ingress bpf da obj $obj sec dst_ingress
- ip netns exec ${NS_FWD} tc filter add dev veth_dst_fwd egress bpf da obj $obj sec chk_egress
-
- if [ "$use_forwarding" -eq "1" ]; then
- # bpf_fib_lookup() checks if forwarding is enabled
- ip netns exec ${NS_FWD} sysctl -w net.ipv4.ip_forward=1
- ip netns exec ${NS_FWD} sysctl -w net.ipv6.conf.veth_dst_fwd.forwarding=1
- ip netns exec ${NS_FWD} sysctl -w net.ipv6.conf.veth_src_fwd.forwarding=1
- return 0
- fi
-
- veth_src=$(ip netns exec ${NS_FWD} cat /sys/class/net/veth_src_fwd/ifindex)
- veth_dst=$(ip netns exec ${NS_FWD} cat /sys/class/net/veth_dst_fwd/ifindex)
-
- progs=$(ip netns exec ${NS_FWD} bpftool net --json | jq -r '.[] | .tc | map(.id) | .[]')
- for prog in $progs; do
- map=$(bpftool prog show id $prog --json | jq -r '.map_ids | .? | .[]')
- if [ ! -z "$map" ]; then
- bpftool map update id $map key hex $(hex_mem_str 0) value hex $(hex_mem_str $veth_src)
- bpftool map update id $map key hex $(hex_mem_str 1) value hex $(hex_mem_str $veth_dst)
- fi
- done
-}
-
-trap netns_cleanup EXIT
-set -e
-
-netns_setup
-netns_setup_bpf test_tc_neigh.o
-netns_test_connectivity
-netns_cleanup
-netns_setup
-netns_setup_bpf test_tc_neigh_fib.o 1
-netns_test_connectivity
-netns_cleanup
-netns_setup
-netns_setup_bpf test_tc_peer.o
-netns_test_connectivity
cpu_set_t cpuset;
__u32 key = 0;
+ libbpf_set_strict_mode(LIBBPF_STRICT_ALL);
+
CPU_ZERO(&cpuset);
CPU_SET(0, &cpuset);
pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpuset);
pb_opts.sample_cb = dummyfn;
pb = perf_buffer__new(bpf_map__fd(perf_map), 8, &pb_opts);
- if (IS_ERR(pb))
+ if (!pb)
goto err;
pthread_create(&tid, NULL, poller_thread, pb);
bpf_prog_detach(cg_fd, BPF_CGROUP_SOCK_OPS);
close(cg_fd);
cleanup_cgroup_environment();
- if (!IS_ERR_OR_NULL(pb))
- perf_buffer__free(pb);
+ perf_buffer__free(pb);
return error;
}
}
}
- if (test->insn_processed) {
+ if (!unpriv && test->insn_processed) {
uint32_t insn_processed;
char *proc;
--- /dev/null
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# Test topology:
+# - - - - - - - - - - - - - - - - - - - - - - - - -
+# | veth1 veth2 veth3 | ... init net
+# - -| - - - - - - | - - - - - - | - -
+# --------- --------- ---------
+# | veth0 | | veth0 | | veth0 | ...
+# --------- --------- ---------
+# ns1 ns2 ns3
+#
+# Test modules:
+# XDP modes: generic, native, native + egress_prog
+#
+# Test cases:
+# ARP: Testing BPF_F_BROADCAST, the ingress interface also should receive
+# the redirects.
+# ns1 -> gw: ns1, ns2, ns3, should receive the arp request
+# IPv4: Testing BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS, the ingress
+# interface should not receive the redirects.
+# ns1 -> gw: ns1 should not receive, ns2, ns3 should receive redirects.
+# IPv6: Testing none flag, all the pkts should be redirected back
+# ping test: ns1 -> ns2 (block), echo requests will be redirect back
+# egress_prog:
+# all src mac should be egress interface's mac
+
+# netns numbers
+NUM=3
+IFACES=""
+DRV_MODE="xdpgeneric xdpdrv xdpegress"
+PASS=0
+FAIL=0
+
+test_pass()
+{
+ echo "Pass: $@"
+ PASS=$((PASS + 1))
+}
+
+test_fail()
+{
+ echo "fail: $@"
+ FAIL=$((FAIL + 1))
+}
+
+clean_up()
+{
+ for i in $(seq $NUM); do
+ ip link del veth$i 2> /dev/null
+ ip netns del ns$i 2> /dev/null
+ done
+}
+
+# Kselftest framework requirement - SKIP code is 4.
+check_env()
+{
+ ip link set dev lo xdpgeneric off &>/dev/null
+ if [ $? -ne 0 ];then
+ echo "selftests: [SKIP] Could not run test without the ip xdpgeneric support"
+ exit 4
+ fi
+
+ which tcpdump &>/dev/null
+ if [ $? -ne 0 ];then
+ echo "selftests: [SKIP] Could not run test without tcpdump"
+ exit 4
+ fi
+}
+
+setup_ns()
+{
+ local mode=$1
+ IFACES=""
+
+ if [ "$mode" = "xdpegress" ]; then
+ mode="xdpdrv"
+ fi
+
+ for i in $(seq $NUM); do
+ ip netns add ns$i
+ ip link add veth$i type veth peer name veth0 netns ns$i
+ ip link set veth$i up
+ ip -n ns$i link set veth0 up
+
+ ip -n ns$i addr add 192.0.2.$i/24 dev veth0
+ ip -n ns$i addr add 2001:db8::$i/64 dev veth0
+ # Add a neigh entry for IPv4 ping test
+ ip -n ns$i neigh add 192.0.2.253 lladdr 00:00:00:00:00:01 dev veth0
+ ip -n ns$i link set veth0 $mode obj \
+ xdp_dummy.o sec xdp_dummy &> /dev/null || \
+ { test_fail "Unable to load dummy xdp" && exit 1; }
+ IFACES="$IFACES veth$i"
+ veth_mac[$i]=$(ip link show veth$i | awk '/link\/ether/ {print $2}')
+ done
+}
+
+do_egress_tests()
+{
+ local mode=$1
+
+ # mac test
+ ip netns exec ns2 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-2_${mode}.log &
+ ip netns exec ns3 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-3_${mode}.log &
+ sleep 0.5
+ ip netns exec ns1 ping 192.0.2.254 -i 0.1 -c 4 &> /dev/null
+ sleep 0.5
+ pkill -9 tcpdump
+
+ # mac check
+ grep -q "${veth_mac[2]} > ff:ff:ff:ff:ff:ff" mac_ns1-2_${mode}.log && \
+ test_pass "$mode mac ns1-2" || test_fail "$mode mac ns1-2"
+ grep -q "${veth_mac[3]} > ff:ff:ff:ff:ff:ff" mac_ns1-3_${mode}.log && \
+ test_pass "$mode mac ns1-3" || test_fail "$mode mac ns1-3"
+}
+
+do_ping_tests()
+{
+ local mode=$1
+
+ # ping6 test: echo request should be redirect back to itself, not others
+ ip netns exec ns1 ip neigh add 2001:db8::2 dev veth0 lladdr 00:00:00:00:00:02
+
+ ip netns exec ns1 tcpdump -i veth0 -nn -l -e &> ns1-1_${mode}.log &
+ ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> ns1-2_${mode}.log &
+ ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> ns1-3_${mode}.log &
+ sleep 0.5
+ # ARP test
+ ip netns exec ns1 ping 192.0.2.254 -i 0.1 -c 4 &> /dev/null
+ # IPv4 test
+ ip netns exec ns1 ping 192.0.2.253 -i 0.1 -c 4 &> /dev/null
+ # IPv6 test
+ ip netns exec ns1 ping6 2001:db8::2 -i 0.1 -c 2 &> /dev/null
+ sleep 0.5
+ pkill -9 tcpdump
+
+ # All netns should receive the redirect arp requests
+ [ $(grep -c "who-has 192.0.2.254" ns1-1_${mode}.log) -gt 4 ] && \
+ test_pass "$mode arp(F_BROADCAST) ns1-1" || \
+ test_fail "$mode arp(F_BROADCAST) ns1-1"
+ [ $(grep -c "who-has 192.0.2.254" ns1-2_${mode}.log) -le 4 ] && \
+ test_pass "$mode arp(F_BROADCAST) ns1-2" || \
+ test_fail "$mode arp(F_BROADCAST) ns1-2"
+ [ $(grep -c "who-has 192.0.2.254" ns1-3_${mode}.log) -le 4 ] && \
+ test_pass "$mode arp(F_BROADCAST) ns1-3" || \
+ test_fail "$mode arp(F_BROADCAST) ns1-3"
+
+ # ns1 should not receive the redirect echo request, others should
+ [ $(grep -c "ICMP echo request" ns1-1_${mode}.log) -eq 4 ] && \
+ test_pass "$mode IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-1" || \
+ test_fail "$mode IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-1"
+ [ $(grep -c "ICMP echo request" ns1-2_${mode}.log) -eq 4 ] && \
+ test_pass "$mode IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-2" || \
+ test_fail "$mode IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-2"
+ [ $(grep -c "ICMP echo request" ns1-3_${mode}.log) -eq 4 ] && \
+ test_pass "$mode IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-3" || \
+ test_fail "$mode IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-3"
+
+ # ns1 should receive the echo request, ns2 should not
+ [ $(grep -c "ICMP6, echo request" ns1-1_${mode}.log) -eq 4 ] && \
+ test_pass "$mode IPv6 (no flags) ns1-1" || \
+ test_fail "$mode IPv6 (no flags) ns1-1"
+ [ $(grep -c "ICMP6, echo request" ns1-2_${mode}.log) -eq 0 ] && \
+ test_pass "$mode IPv6 (no flags) ns1-2" || \
+ test_fail "$mode IPv6 (no flags) ns1-2"
+}
+
+do_tests()
+{
+ local mode=$1
+ local drv_p
+
+ case ${mode} in
+ xdpdrv) drv_p="-N";;
+ xdpegress) drv_p="-X";;
+ xdpgeneric) drv_p="-S";;
+ esac
+
+ ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log &
+ xdp_pid=$!
+ sleep 1
+
+ if [ "$mode" = "xdpegress" ]; then
+ do_egress_tests $mode
+ else
+ do_ping_tests $mode
+ fi
+
+ kill $xdp_pid
+}
+
+trap clean_up 0 2 3 6 9
+
+check_env
+rm -f xdp_redirect_*.log ns*.log mac_ns*.log
+
+for mode in ${DRV_MODE}; do
+ setup_ns $mode
+ do_tests $mode
+ clean_up
+done
+
+echo "Summary: PASS $PASS, FAIL $FAIL"
+[ $FAIL -eq 0 ] && exit 0 || exit 1
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R1 !read_ok",
+ .result_unpriv = REJECT,
.result = ACCEPT,
.retval = 0
},
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, -1),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 invalid mem access 'inv'",
+ .result_unpriv = REJECT,
.result = ACCEPT
},
{
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, -1),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 invalid mem access 'inv'",
+ .result_unpriv = REJECT,
.result = ACCEPT
},
{
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 min value is outside of the allowed memory range",
+ .result_unpriv = REJECT,
.fixup_map_hash_8b = { 3 },
.result = ACCEPT,
},
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 min value is outside of the allowed memory range",
+ .result_unpriv = REJECT,
.fixup_map_hash_8b = { 3 },
.result = ACCEPT,
},
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 min value is outside of the allowed memory range",
+ .result_unpriv = REJECT,
.fixup_map_hash_8b = { 3 },
.result = ACCEPT,
},
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 min value is outside of the allowed memory range",
+ .result_unpriv = REJECT,
.fixup_map_hash_8b = { 3 },
.result = ACCEPT,
},
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 min value is outside of the allowed memory range",
+ .result_unpriv = REJECT,
.fixup_map_hash_8b = { 3 },
.result = ACCEPT,
},
BPF_JMP_IMM(BPF_JGE, BPF_REG_0, 10, -4),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R9 !read_ok",
+ .result_unpriv = REJECT,
.result = ACCEPT,
.retval = 7,
},
BPF_LDX_MEM(BPF_B, BPF_REG_8, BPF_REG_9, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R9 !read_ok",
+ .result_unpriv = REJECT,
.result = ACCEPT,
},
{
BPF_LDX_MEM(BPF_B, BPF_REG_8, BPF_REG_9, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R9 !read_ok",
+ .result_unpriv = REJECT,
.result = ACCEPT,
},
{
BPF_LDX_MEM(BPF_B, BPF_REG_8, BPF_REG_9, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R9 !read_ok",
+ .result_unpriv = REJECT,
.result = ACCEPT,
},
{
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 invalid mem access 'inv'",
+ .result_unpriv = REJECT,
.result = ACCEPT,
.retval = 2,
},
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 invalid mem access 'inv'",
+ .result_unpriv = REJECT,
.result = ACCEPT,
.retval = 2,
},
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 invalid mem access 'inv'",
+ .result_unpriv = REJECT,
.result = ACCEPT,
.retval = 2,
},
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 invalid mem access 'inv'",
+ .result_unpriv = REJECT,
.result = ACCEPT,
.retval = 2,
},
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 invalid mem access 'inv'",
+ .result_unpriv = REJECT,
.result = ACCEPT,
.retval = 2,
},
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 invalid mem access 'inv'",
+ .result_unpriv = REJECT,
.result = ACCEPT,
.retval = 2,
},
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 invalid mem access 'inv'",
+ .result_unpriv = REJECT,
.result = ACCEPT,
.retval = 2,
},
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R0 invalid mem access 'inv'",
+ .result_unpriv = REJECT,
.result = ACCEPT,
.retval = 2,
},
BPF_EXIT_INSN(),
},
.prog_type = BPF_PROG_TYPE_SOCKET_FILTER,
- .retval_unpriv = 1,
- .result_unpriv = ACCEPT,
+ .errstr_unpriv = "R9 !read_ok",
+ .result_unpriv = REJECT,
.retval = 1,
.result = ACCEPT,
},
BPF_EXIT_INSN(),
},
.prog_type = BPF_PROG_TYPE_SOCKET_FILTER,
- .result_unpriv = ACCEPT,
+ .errstr_unpriv = "R9 !read_ok",
+ .result_unpriv = REJECT,
.result = ACCEPT,
},
{
BPF_EXIT_INSN(),
},
.prog_type = BPF_PROG_TYPE_SOCKET_FILTER,
- .result_unpriv = ACCEPT,
+ .errstr_unpriv = "R9 !read_ok",
+ .result_unpriv = REJECT,
.result = ACCEPT,
},
BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0),
BPF_EXIT_INSN(),
},
- .result_unpriv = REJECT,
- .errstr_unpriv = "invalid write to stack R1 off=0 size=1",
.result = ACCEPT,
.retval = 42,
},
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_7, 0),
BPF_EXIT_INSN(),
},
+ .errstr_unpriv = "R7 invalid mem access 'inv'",
+ .result_unpriv = REJECT,
.result = ACCEPT,
.retval = 0,
},
.fixup_map_array_48b = { 1 },
.result = ACCEPT,
.result_unpriv = REJECT,
- .errstr_unpriv = "R2 tried to add from different maps, paths or scalars",
+ .errstr_unpriv = "R2 pointer comparison prohibited",
.retval = 0,
},
{
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
// fake-dead code; targeted from branch A to
- // prevent dead code sanitization
+ // prevent dead code sanitization, rejected
+ // via branch B however
BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0),
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
.fixup_map_array_48b = { 1 },
.result = ACCEPT,
.result_unpriv = REJECT,
- .errstr_unpriv = "R2 tried to add from different maps, paths or scalars",
+ .errstr_unpriv = "R0 invalid mem access 'inv'",
.retval = 0,
},
{
},
.fixup_map_array_48b = { 3 },
.result = ACCEPT,
- .result_unpriv = REJECT,
- .errstr_unpriv = "R0 pointer arithmetic of map value goes out of range",
.retval = 1,
},
{
},
.fixup_map_array_48b = { 3 },
.result = ACCEPT,
- .result_unpriv = REJECT,
- .errstr_unpriv = "R0 pointer arithmetic of map value goes out of range",
.retval = 1,
},
{
},
.fixup_map_array_48b = { 3 },
.result = ACCEPT,
- .result_unpriv = REJECT,
- .errstr_unpriv = "R0 pointer arithmetic of map value goes out of range",
.retval = 1,
},
{
},
.fixup_map_array_48b = { 3 },
.result = ACCEPT,
- .result_unpriv = REJECT,
- .errstr_unpriv = "R0 pointer arithmetic of map value goes out of range",
.retval = 1,
},
{
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <linux/if_link.h>
+#include <assert.h>
+#include <errno.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <net/if.h>
+#include <unistd.h>
+#include <libgen.h>
+#include <sys/resource.h>
+#include <sys/ioctl.h>
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
+
+#include "bpf_util.h"
+#include <bpf/bpf.h>
+#include <bpf/libbpf.h>
+
+#define MAX_IFACE_NUM 32
+#define MAX_INDEX_NUM 1024
+
+static __u32 xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST;
+static int ifaces[MAX_IFACE_NUM] = {};
+
+static void int_exit(int sig)
+{
+ __u32 prog_id = 0;
+ int i;
+
+ for (i = 0; ifaces[i] > 0; i++) {
+ if (bpf_get_link_xdp_id(ifaces[i], &prog_id, xdp_flags)) {
+ printf("bpf_get_link_xdp_id failed\n");
+ exit(1);
+ }
+ if (prog_id)
+ bpf_set_link_xdp_fd(ifaces[i], -1, xdp_flags);
+ }
+
+ exit(0);
+}
+
+static int get_mac_addr(unsigned int ifindex, void *mac_addr)
+{
+ char ifname[IF_NAMESIZE];
+ struct ifreq ifr;
+ int fd, ret = -1;
+
+ fd = socket(AF_INET, SOCK_DGRAM, 0);
+ if (fd < 0)
+ return ret;
+
+ if (!if_indextoname(ifindex, ifname))
+ goto err_out;
+
+ strcpy(ifr.ifr_name, ifname);
+
+ if (ioctl(fd, SIOCGIFHWADDR, &ifr) != 0)
+ goto err_out;
+
+ memcpy(mac_addr, ifr.ifr_hwaddr.sa_data, 6 * sizeof(char));
+ ret = 0;
+
+err_out:
+ close(fd);
+ return ret;
+}
+
+static void usage(const char *prog)
+{
+ fprintf(stderr,
+ "usage: %s [OPTS] <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n"
+ "OPTS:\n"
+ " -S use skb-mode\n"
+ " -N enforce native mode\n"
+ " -F force loading prog\n"
+ " -X load xdp program on egress\n",
+ prog);
+}
+
+int main(int argc, char **argv)
+{
+ int prog_fd, group_all, mac_map;
+ struct bpf_program *ingress_prog, *egress_prog;
+ struct bpf_prog_load_attr prog_load_attr = {
+ .prog_type = BPF_PROG_TYPE_UNSPEC,
+ };
+ int i, ret, opt, egress_prog_fd = 0;
+ struct bpf_devmap_val devmap_val;
+ bool attach_egress_prog = false;
+ unsigned char mac_addr[6];
+ char ifname[IF_NAMESIZE];
+ struct bpf_object *obj;
+ unsigned int ifindex;
+ char filename[256];
+
+ while ((opt = getopt(argc, argv, "SNFX")) != -1) {
+ switch (opt) {
+ case 'S':
+ xdp_flags |= XDP_FLAGS_SKB_MODE;
+ break;
+ case 'N':
+ /* default, set below */
+ break;
+ case 'F':
+ xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST;
+ break;
+ case 'X':
+ attach_egress_prog = true;
+ break;
+ default:
+ usage(basename(argv[0]));
+ return 1;
+ }
+ }
+
+ if (!(xdp_flags & XDP_FLAGS_SKB_MODE)) {
+ xdp_flags |= XDP_FLAGS_DRV_MODE;
+ } else if (attach_egress_prog) {
+ printf("Load xdp program on egress with SKB mode not supported yet\n");
+ goto err_out;
+ }
+
+ if (optind == argc) {
+ printf("usage: %s <IFNAME|IFINDEX> <IFNAME|IFINDEX> ...\n", argv[0]);
+ goto err_out;
+ }
+
+ printf("Get interfaces");
+ for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) {
+ ifaces[i] = if_nametoindex(argv[optind + i]);
+ if (!ifaces[i])
+ ifaces[i] = strtoul(argv[optind + i], NULL, 0);
+ if (!if_indextoname(ifaces[i], ifname)) {
+ perror("Invalid interface name or i");
+ goto err_out;
+ }
+ if (ifaces[i] > MAX_INDEX_NUM) {
+ printf("Interface index to large\n");
+ goto err_out;
+ }
+ printf(" %d", ifaces[i]);
+ }
+ printf("\n");
+
+ snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+ prog_load_attr.file = filename;
+
+ if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd))
+ goto err_out;
+
+ if (attach_egress_prog)
+ group_all = bpf_object__find_map_fd_by_name(obj, "map_egress");
+ else
+ group_all = bpf_object__find_map_fd_by_name(obj, "map_all");
+ mac_map = bpf_object__find_map_fd_by_name(obj, "mac_map");
+
+ if (group_all < 0 || mac_map < 0) {
+ printf("bpf_object__find_map_fd_by_name failed\n");
+ goto err_out;
+ }
+
+ if (attach_egress_prog) {
+ /* Find ingress/egress prog for 2nd xdp prog */
+ ingress_prog = bpf_object__find_program_by_name(obj, "xdp_redirect_map_all_prog");
+ egress_prog = bpf_object__find_program_by_name(obj, "xdp_devmap_prog");
+ if (!ingress_prog || !egress_prog) {
+ printf("finding ingress/egress_prog in obj file failed\n");
+ goto err_out;
+ }
+ prog_fd = bpf_program__fd(ingress_prog);
+ egress_prog_fd = bpf_program__fd(egress_prog);
+ if (prog_fd < 0 || egress_prog_fd < 0) {
+ printf("find egress_prog fd failed\n");
+ goto err_out;
+ }
+ }
+
+ signal(SIGINT, int_exit);
+ signal(SIGTERM, int_exit);
+
+ /* Init forward multicast groups and exclude group */
+ for (i = 0; ifaces[i] > 0; i++) {
+ ifindex = ifaces[i];
+
+ if (attach_egress_prog) {
+ ret = get_mac_addr(ifindex, mac_addr);
+ if (ret < 0) {
+ printf("get interface %d mac failed\n", ifindex);
+ goto err_out;
+ }
+ ret = bpf_map_update_elem(mac_map, &ifindex, mac_addr, 0);
+ if (ret) {
+ perror("bpf_update_elem mac_map failed\n");
+ goto err_out;
+ }
+ }
+
+ /* Add all the interfaces to group all */
+ devmap_val.ifindex = ifindex;
+ devmap_val.bpf_prog.fd = egress_prog_fd;
+ ret = bpf_map_update_elem(group_all, &ifindex, &devmap_val, 0);
+ if (ret) {
+ perror("bpf_map_update_elem");
+ goto err_out;
+ }
+
+ /* bind prog_fd to each interface */
+ ret = bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags);
+ if (ret) {
+ printf("Set xdp fd failed on %d\n", ifindex);
+ goto err_out;
+ }
+ }
+
+ /* sleep some time for testing */
+ sleep(999);
+
+ return 0;
+
+err_out:
+ return 1;
+}
__addr_add_del $rp1 del 192.0.2.2/24 2001:db8:1::2/64
tc qdisc del dev $rp2 clsact
+
+ ip link set dev $rp2 down
+ ip link set dev $rp1 down
}
setup_prepare()
__addr_add_del $rp1 del 192.0.2.2/24 2001:db8:1::2/64
tc qdisc del dev $rp2 clsact
+
+ ip link set dev $rp2 down
+ ip link set dev $rp1 down
}
setup_prepare()
PORT_NUM_NETIFS=0
+declare -a unsplit
+
port_setup_prepare()
{
:
devlink port unsplit $port
check_err $? "Did not unsplit $netdev"
done
+ unsplit=()
}
split_all_ports()
{
local should_fail=$1; shift
- local -a unsplit
# Loop over the splittable netdevs and create tuples of netdev along
# with its width. For example:
lldptool -T -i $swp1 -V APP -d $(dscp_map 10) >/dev/null
lldpad_app_wait_del
+ ip link set dev $swp2 down
ip link set dev $swp2 nomaster
+ ip link set dev $swp1 down
ip link set dev $swp1 nomaster
ip link del dev br1
}
get_prio_pg()
{
- __mlnx_qos -i $swp | sed -n '/^PFC/,/^[^[:space:]]/p' |
- grep buffer | sed 's/ \+/ /g' | cut -d' ' -f 2-
+ # Produces a string of numbers "<B0> <B1> ... <B7> ", where BX is number
+ # of buffer that priority X is mapped to.
+ dcb -j buffer show dev $swp |
+ jq -r '[.prio_buffer | .[] | tostring + " "] | add'
}
get_prio_pfc()
{
- __mlnx_qos -i $swp | sed -n '/^PFC/,/^[^[:space:]]/p' |
- grep enabled | sed 's/ \+/ /g' | cut -d' ' -f 2-
+ # Produces a string of numbers "<P0> <P1> ... <P7> ", where PX denotes
+ # whether priority X has PFC enabled (the value is 1) or disabled (0).
+ dcb -j pfc show dev $swp |
+ jq -r '[.prio_pfc | .[] | if . then "1 " else "0 " end] | add'
}
get_prio_tc()
{
- __mlnx_qos -i $swp | sed -n '/^tc/,$p' |
- awk '/^tc/ { TC = $2 }
- /priority:/ { PRIO[$2]=TC }
- END {
- for (i in PRIO)
- printf("%d ", PRIO[i])
- }'
+ # Produces a string of numbers "<T0> <T1> ... <T7> ", where TC is number
+ # of TC that priority X is mapped to.
+ dcb -j ets show dev $swp |
+ jq -r '[.prio_tc | .[] | tostring + " "] | add'
}
get_buf_size()
{
local idx=$1; shift
- __mlnx_qos -i $swp | grep Receive | sed 's/.*: //' | cut -d, -f $((idx + 1))
+ dcb -j buffer show dev $swp | jq ".buffer_size[$idx]"
}
get_tot_size()
{
- __mlnx_qos -i $swp | grep Receive | sed 's/.*total_size=//'
+ dcb -j buffer show dev $swp | jq '.total_size'
}
check_prio_pg()
{
RET=0
- __mlnx_qos -i $swp --prio_tc=0,2,4,6,1,3,5,7 > /dev/null
+ dcb ets set dev $swp prio-tc 0:0 1:2 2:4 3:6 4:1 5:3 6:5 7:7
check_prio_pg "0 2 4 6 1 3 5 7 "
check_prio_tc "0 2 4 6 1 3 5 7 "
check_prio_pfc "0 0 0 0 0 0 0 0 "
- __mlnx_qos -i $swp --prio_tc=0,0,0,0,0,0,0,0 > /dev/null
+ dcb ets set dev $swp prio-tc all:0
check_prio_pg "0 0 0 0 0 0 0 0 "
check_prio_tc "0 0 0 0 0 0 0 0 "
- __mlnx_qos -i $swp --prio2buffer=1,3,5,7,0,2,4,6 &> /dev/null
+ dcb buffer set dev $swp prio-buffer 0:1 1:3 2:5 3:7 4:0 5:2 6:4 7:6 2>/dev/null
check_fail $? "prio2buffer accepted in DCB mode"
log_test "Configuring headroom through ETS"
{
RET=0
- __mlnx_qos -i $swp --prio_tc=0,0,0,0,0,1,2,3 > /dev/null
+ dcb ets set dev $swp prio-tc all:0 5:1 6:2 7:3
local buf0size=$(get_buf_size 0)
local buf1size=$(get_buf_size 1)
RET=0
- __mlnx_qos -i $swp --pfc=0,0,0,0,0,1,1,1 --cable_len=0 > /dev/null
+ dcb pfc set dev $swp prio-pfc all:off 5:on 6:on 7:on delay 0
check_prio_pg "0 0 0 0 0 1 2 3 "
check_prio_pfc "0 0 0 0 0 1 1 1 "
RET=0
- __mlnx_qos -i $swp --pfc=0,0,0,0,0,1,1,1 --cable_len=1000 > /dev/null
+ dcb pfc set dev $swp delay 1000
check_buf_size 0 "== $buf0size"
check_buf_size 1 "> $buf1size"
RET=0
- __mlnx_qos -i $swp --pfc=0,0,0,0,0,0,0,0 --cable_len=0 > /dev/null
- __mlnx_qos -i $swp --prio_tc=0,0,0,0,0,0,0,0 > /dev/null
+ dcb pfc set dev $swp prio-pfc all:off delay 0
+ dcb ets set dev $swp prio-tc all:0
check_prio_pg "0 0 0 0 0 0 0 0 "
check_prio_tc "0 0 0 0 0 0 0 0 "
{
RET=0
- __mlnx_qos -i $swp --prio_tc=0,1,2,3,4,5,6,7 > /dev/null
+ dcb ets set dev $swp prio-tc 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
check_prio_pg "0 1 2 3 4 5 6 7 "
tc qdisc replace dev $swp root handle 1: bfifo limit 1.5M
check_prio_pg "0 0 0 0 0 0 0 0 "
- __mlnx_qos -i $swp --prio2buffer=1,3,5,7,0,2,4,6 > /dev/null
+ dcb buffer set dev $swp prio-buffer 0:1 1:3 2:5 3:7 4:0 5:2 6:4 7:6
check_prio_pg "1 3 5 7 0 2 4 6 "
tc qdisc delete dev $swp root
# Clean up.
tc qdisc replace dev $swp root handle 1: bfifo limit 1.5M
- __mlnx_qos -i $swp --prio2buffer=0,0,0,0,0,0,0,0 > /dev/null
+ dcb buffer set dev $swp prio-buffer all:0
tc qdisc delete dev $swp root
- __mlnx_qos -i $swp --prio_tc=0,0,0,0,0,0,0,0 > /dev/null
+ dcb ets set dev $swp prio-tc all:0
log_test "TC: priomap"
}
RET=0
- __mlnx_qos -i $swp --buffer_size=$size,0,0,0,0,0,0,0 &> /dev/null
+ dcb buffer set dev $swp buffer-size all:0 0:$size 2>/dev/null
check_fail $? "buffer_size should fail before qdisc is added"
tc qdisc replace dev $swp root handle 1: bfifo limit 1.5M
- __mlnx_qos -i $swp --buffer_size=$size,0,0,0,0,0,0,0 > /dev/null
+ dcb buffer set dev $swp buffer-size all:0 0:$size
check_err $? "buffer_size should pass after qdisc is added"
check_buf_size 0 "== $size" "set size: "
check_buf_size 0 "== $size" "set MTU: "
mtu_restore $swp
- __mlnx_qos -i $swp --buffer_size=0,0,0,0,0,0,0,0 > /dev/null
+ dcb buffer set dev $swp buffer-size all:0
# After replacing the qdisc for the same kind, buffer_size still has to
# work.
tc qdisc replace dev $swp root handle 1: bfifo limit 1M
- __mlnx_qos -i $swp --buffer_size=$size,0,0,0,0,0,0,0 > /dev/null
+ dcb buffer set dev $swp buffer-size all:0 0:$size
check_buf_size 0 "== $size" "post replace, set size: "
- __mlnx_qos -i $swp --buffer_size=0,0,0,0,0,0,0,0 > /dev/null
+ dcb buffer set dev $swp buffer-size all:0
# Likewise after replacing for a different kind.
tc qdisc replace dev $swp root handle 2: prio bands 8
- __mlnx_qos -i $swp --buffer_size=$size,0,0,0,0,0,0,0 > /dev/null
+ dcb buffer set dev $swp buffer-size all:0 0:$size
check_buf_size 0 "== $size" "post replace different kind, set size: "
tc qdisc delete dev $swp root
- __mlnx_qos -i $swp --buffer_size=$size,0,0,0,0,0,0,0 &> /dev/null
+ dcb buffer set dev $swp buffer-size all:0 0:$size 2>/dev/null
check_fail $? "buffer_size should fail after qdisc is deleted"
log_test "TC: buffer size"
tc qdisc replace dev $swp root handle 1: bfifo limit 1.5M
test_int_buf "TC: "
- __mlnx_qos -i $swp --buffer_size=$size,0,0,0,0,0,0,0 > /dev/null
+ dcb buffer set dev $swp buffer-size all:0 0:$size
test_int_buf "TC+buffsize: "
- __mlnx_qos -i $swp --buffer_size=0,0,0,0,0,0,0,0 > /dev/null
+ dcb buffer set dev $swp buffer-size all:0
tc qdisc delete dev $swp root
}
fi
fi
}
-
-__mlnx_qos()
-{
- local err
-
- mlnx_qos "$@" 2>/dev/null
- err=$?
-
- if ((err)); then
- echo "Error ($err) in mlnx_qos $@" >/dev/stderr
- fi
-
- return $err
-}
# assignment.
tc qdisc replace dev $swp1 root handle 1: \
ets bands 8 strict 8 priomap 7 6
- __mlnx_qos -i $swp1 --prio2buffer=0,1,0,0,0,0,0,0 >/dev/null
+ dcb buffer set dev $swp1 prio-buffer all:0 1:1
# $swp2
# -----
# the lossless prio into a buffer of its own. Don't bother with buffer
# sizes though, there is not going to be any pressure in the "backward"
# direction.
- __mlnx_qos -i $swp3 --prio2buffer=0,1,0,0,0,0,0,0 >/dev/null
- __mlnx_qos -i $swp3 --pfc=0,1,0,0,0,0,0,0 >/dev/null
+ dcb buffer set dev $swp3 prio-buffer all:0 1:1
+ dcb pfc set dev $swp3 prio-pfc all:off 1:on
# $swp4
# -----
# Configure qdisc so that we can hand-tune headroom.
tc qdisc replace dev $swp4 root handle 1: \
ets bands 8 strict 8 priomap 7 6
- __mlnx_qos -i $swp4 --prio2buffer=0,1,0,0,0,0,0,0 >/dev/null
- __mlnx_qos -i $swp4 --pfc=0,1,0,0,0,0,0,0 >/dev/null
+ dcb buffer set dev $swp4 prio-buffer all:0 1:1
+ dcb pfc set dev $swp4 prio-pfc all:off 1:on
# PG0 will get autoconfigured to Xoff, give PG1 arbitrarily 100K, which
# is (-2*MTU) about 80K of delay provision.
- __mlnx_qos -i $swp4 --buffer_size=0,$_100KB,0,0,0,0,0,0 >/dev/null
+ dcb buffer set dev $swp4 buffer-size all:0 1:$_100KB
# bridges
# -------
# $swp4
# -----
- __mlnx_qos -i $swp4 --buffer_size=0,0,0,0,0,0,0,0 >/dev/null
- __mlnx_qos -i $swp4 --pfc=0,0,0,0,0,0,0,0 >/dev/null
- __mlnx_qos -i $swp4 --prio2buffer=0,0,0,0,0,0,0,0 >/dev/null
+ dcb buffer set dev $swp4 buffer-size all:0
+ dcb pfc set dev $swp4 prio-pfc all:off
+ dcb buffer set dev $swp4 prio-buffer all:0
tc qdisc del dev $swp4 root
devlink_tc_bind_pool_th_restore $swp4 1 ingress
# $swp3
# -----
- __mlnx_qos -i $swp3 --pfc=0,0,0,0,0,0,0,0 >/dev/null
- __mlnx_qos -i $swp3 --prio2buffer=0,0,0,0,0,0,0,0 >/dev/null
+ dcb pfc set dev $swp3 prio-pfc all:off
+ dcb buffer set dev $swp3 prio-buffer all:0
tc qdisc del dev $swp3 root
devlink_tc_bind_pool_th_restore $swp3 1 egress
# $swp1
# -----
- __mlnx_qos -i $swp1 --prio2buffer=0,0,0,0,0,0,0,0 >/dev/null
+ dcb buffer set dev $swp1 prio-buffer all:0
tc qdisc del dev $swp1 root
devlink_tc_bind_pool_th_restore $swp1 1 ingress
local t0=$1; shift
local route_count=$1; shift
- local t1=$(ip route | grep -o 'offload' | wc -l)
+ local t1=$(ip route | grep 'offload' | grep -v 'offload_failed' | wc -l)
local delta=$((t1 - t0))
echo $delta
[[ $delta -ge $route_count ]]
psample_capture_start
- ip vrf exec v$h1 $MZ $h1 -c 3200 -d 1msec -p 64 -A 192.0.2.1 \
+ ip vrf exec v$h1 $MZ $h1 -c 320000 -d 100usec -p 64 -A 192.0.2.1 \
-B $dip -t udp dp=52768,sp=42768 -q
psample_capture_stop
pkts=$(grep -e "group 1 " $CAPTURE_FILE | wc -l)
- pct=$((100 * (pkts - 100) / 100))
+ pct=$((100 * (pkts - 10000) / 10000))
(( -25 <= pct && pct <= 25))
- check_err $? "Expected 100 packets, got $pkts packets, which is $pct% off. Required accuracy is +-25%"
+ check_err $? "Expected 10000 packets, got $pkts packets, which is $pct% off. Required accuracy is +-25%"
log_test "tc sample rate ($desc)"
psample_capture_start
- ip vrf exec v$h1 $MZ $h1 -c 3200 -d 1msec -p 64 -A 192.0.2.1 \
+ ip vrf exec v$h1 $MZ $h1 -c 320000 -d 100usec -p 64 -A 192.0.2.1 \
-B 198.51.100.1 -t udp dp=52768,sp=42768 -q
psample_capture_stop
pkts=$(grep -e "group 1 " $CAPTURE_FILE | wc -l)
- pct=$((100 * (pkts - 100) / 100))
+ pct=$((100 * (pkts - 10000) / 10000))
(( -25 <= pct && pct <= 25))
- check_err $? "Expected 100 packets, got $pkts packets, which is $pct% off. Required accuracy is +-25%"
+ check_err $? "Expected 10000 packets, got $pkts packets, which is $pct% off. Required accuracy is +-25%"
# Setup a filter that should not match any packet and make sure packets
# are not sampled.
ALL_TESTS="fw_flash_test params_test regions_test reload_test \
netns_reload_test resource_test dev_info_test \
- empty_reporter_test dummy_reporter_test"
+ empty_reporter_test dummy_reporter_test rate_test"
NUM_NETIFS=0
source $lib_dir/lib.sh
BUS_ADDR=10
PORT_COUNT=4
+VF_COUNT=4
DEV_NAME=netdevsim$BUS_ADDR
SYSFS_NET_DIR=/sys/bus/netdevsim/devices/$DEV_NAME/net/
DEBUGFS_DIR=/sys/kernel/debug/netdevsim/$DEV_NAME/
log_test "dummy reporter test"
}
+rate_leafs_get()
+{
+ local handle=$1
+
+ cmd_jq "devlink port function rate show -j" \
+ '.[] | to_entries | .[] | select(.value.type == "leaf") | .key | select(contains("'$handle'"))'
+}
+
+rate_nodes_get()
+{
+ local handle=$1
+
+ cmd_jq "devlink port function rate show -j" \
+ '.[] | to_entries | .[] | select(.value.type == "node") | .key | select(contains("'$handle'"))'
+}
+
+rate_attr_set()
+{
+ local handle=$1
+ local name=$2
+ local value=$3
+ local units=$4
+
+ devlink port function rate set $handle $name $value$units
+}
+
+rate_attr_get()
+{
+ local handle=$1
+ local name=$2
+
+ cmd_jq "devlink port function rate show $handle -j" '.[][].'$name
+}
+
+rate_attr_tx_rate_check()
+{
+ local handle=$1
+ local name=$2
+ local rate=$3
+ local debug_file=$4
+
+ rate_attr_set $handle $name $rate mbit
+ check_err $? "Failed to set $name value"
+
+ local debug_value=$(cat $debug_file)
+ check_err $? "Failed to read $name value from debugfs"
+ [ "$debug_value" == "$rate" ]
+ check_err $? "Unexpected $name debug value $debug_value != $rate"
+
+ local api_value=$(( $(rate_attr_get $handle $name) * 8 / 1000000 ))
+ check_err $? "Failed to get $name attr value"
+ [ "$api_value" == "$rate" ]
+ check_err $? "Unexpected $name attr value $api_value != $rate"
+}
+
+rate_attr_parent_check()
+{
+ local handle=$1
+ local parent=$2
+ local debug_file=$3
+
+ rate_attr_set $handle parent $parent
+ check_err $? "Failed to set parent"
+
+ debug_value=$(cat $debug_file)
+ check_err $? "Failed to get parent debugfs value"
+ [ "$debug_value" == "$parent" ]
+ check_err $? "Unexpected parent debug value $debug_value != $parent"
+
+ api_value=$(rate_attr_get $r_obj parent)
+ check_err $? "Failed to get parent attr value"
+ [ "$api_value" == "$parent" ]
+ check_err $? "Unexpected parent attr value $api_value != $parent"
+}
+
+rate_node_add()
+{
+ local handle=$1
+
+ devlink port function rate add $handle
+}
+
+rate_node_del()
+{
+ local handle=$1
+
+ devlink port function rate del $handle
+}
+
+rate_test()
+{
+ RET=0
+
+ echo $VF_COUNT > /sys/bus/netdevsim/devices/$DEV_NAME/sriov_numvfs
+ devlink dev eswitch set $DL_HANDLE mode switchdev
+ local leafs=`rate_leafs_get $DL_HANDLE`
+ local num_leafs=`echo $leafs | wc -w`
+ [ "$num_leafs" == "$VF_COUNT" ]
+ check_err $? "Expected $VF_COUNT rate leafs but got $num_leafs"
+
+ rate=10
+ for r_obj in $leafs
+ do
+ rate_attr_tx_rate_check $r_obj tx_share $rate \
+ $DEBUGFS_DIR/ports/${r_obj##*/}/tx_share
+ rate=$(($rate+10))
+ done
+
+ rate=100
+ for r_obj in $leafs
+ do
+ rate_attr_tx_rate_check $r_obj tx_max $rate \
+ $DEBUGFS_DIR/ports/${r_obj##*/}/tx_max
+ rate=$(($rate+100))
+ done
+
+ local node1_name='group1'
+ local node1="$DL_HANDLE/$node1_name"
+ rate_node_add "$node1"
+ check_err $? "Failed to add node $node1"
+
+ local num_nodes=`rate_nodes_get $DL_HANDLE | wc -w`
+ [ $num_nodes == 1 ]
+ check_err $? "Expected 1 rate node in output but got $num_nodes"
+
+ local node_tx_share=10
+ rate_attr_tx_rate_check $node1 tx_share $node_tx_share \
+ $DEBUGFS_DIR/rate_nodes/${node1##*/}/tx_share
+
+ local node_tx_max=100
+ rate_attr_tx_rate_check $node1 tx_max $node_tx_max \
+ $DEBUGFS_DIR/rate_nodes/${node1##*/}/tx_max
+
+ rate_node_del "$node1"
+ check_err $? "Failed to delete node $node1"
+ local num_nodes=`rate_nodes_get $DL_HANDLE | wc -w`
+ [ $num_nodes == 0 ]
+ check_err $? "Expected 0 rate node but got $num_nodes"
+
+ local node1_name='group1'
+ local node1="$DL_HANDLE/$node1_name"
+ rate_node_add "$node1"
+ check_err $? "Failed to add node $node1"
+
+ rate_attr_parent_check $r_obj $node1_name \
+ $DEBUGFS_DIR/ports/${r_obj##*/}/rate_parent
+
+ local node2_name='group2'
+ local node2="$DL_HANDLE/$node2_name"
+ rate_node_add "$node2"
+ check_err $? "Failed to add node $node2"
+
+ rate_attr_parent_check $node2 $node1_name \
+ $DEBUGFS_DIR/rate_nodes/$node2_name/rate_parent
+ rate_node_del "$node2"
+ check_err $? "Failed to delete node $node2"
+ rate_attr_set "$r_obj" noparent
+ check_err $? "Failed to unset $r_obj parent node"
+ rate_node_del "$node1"
+ check_err $? "Failed to delete node $node1"
+
+ log_test "rate test"
+}
+
setup_prepare()
{
modprobe netdevsim
NETDEVSIM_PATH=/sys/bus/netdevsim/
DEV_ADDR=1337
DEV=netdevsim${DEV_ADDR}
-DEVLINK_DEV=netdevsim/${DEV}
DEBUGFS_DIR=/sys/kernel/debug/netdevsim/$DEV/
SLEEP_TIME=1
NETDEV=""
NUM_NETIFS=0
source $lib_dir/lib.sh
+
+DEVLINK_DEV=
source $lib_dir/devlink_lib.sh
+DEVLINK_DEV=netdevsim/${DEV}
require_command udevadm
devlink_trap_action_set $trap_name "drop"
devlink_trap_stats_idle_test $trap_name
check_err $? "Stats of trap $trap_name not idle when action is drop"
+
+ echo "y"> $DEBUGFS_DIR/fail_trap_drop_counter_get
+ devlink -s trap show $DEVLINK_DEV trap $trap_name &> /dev/null
+ check_fail $? "Managed to read trap (hard dropped) statistics when should not"
+ echo "n"> $DEBUGFS_DIR/fail_trap_drop_counter_get
+ devlink -s trap show $DEVLINK_DEV trap $trap_name &> /dev/null
+ check_err $? "Did not manage to read trap (hard dropped) statistics when should"
+
+ devlink_trap_drop_stats_idle_test $trap_name
+ check_fail $? "Drop stats of trap $trap_name idle when should not"
else
devlink_trap_stats_idle_test $trap_name
check_fail $? "Stats of non-drop trap $trap_name idle when should not"
NETDEVSIM_PATH=/sys/bus/netdevsim/
DEV_ADDR=1337
DEV=netdevsim${DEV_ADDR}
-DEVLINK_DEV=netdevsim/${DEV}
SYSFS_NET_DIR=/sys/bus/netdevsim/devices/$DEV/net/
NUM_NETIFS=0
source $lib_dir/lib.sh
-source $lib_dir/devlink_lib.sh
source $lib_dir/fib_offload_lib.sh
+DEVLINK_DEV=
+source $lib_dir/devlink_lib.sh
+DEVLINK_DEV=netdevsim/${DEV}
+
ipv4_identical_routes()
{
fib_ipv4_identical_routes_test "testns1"
NETDEVSIM_PATH=/sys/bus/netdevsim/
DEV_ADDR=1337
DEV=netdevsim${DEV_ADDR}
-DEVLINK_DEV=netdevsim/${DEV}
SYSFS_NET_DIR=/sys/bus/netdevsim/devices/$DEV/net/
DEBUGFS_NET_DIR=/sys/kernel/debug/netdevsim/$DEV/
NUM_NETIFS=0
source $lib_dir/lib.sh
+
+DEVLINK_DEV=
source $lib_dir/devlink_lib.sh
+DEVLINK_DEV=netdevsim/${DEV}
nexthop_check()
{
NETDEVSIM_PATH=/sys/bus/netdevsim/
DEV_ADDR=1337
DEV=netdevsim${DEV_ADDR}
-DEVLINK_DEV=netdevsim/${DEV}
SYSFS_NET_DIR=/sys/bus/netdevsim/devices/$DEV/net/
PSAMPLE_DIR=/sys/kernel/debug/netdevsim/$DEV/psample/
CAPTURE_FILE=$(mktemp)
NUM_NETIFS=0
source $lib_dir/lib.sh
+
+DEVLINK_DEV=
source $lib_dir/devlink_lib.sh
+DEVLINK_DEV=netdevsim/${DEV}
# Available at https://github.com/Mellanox/libpsample
require_command psample
cp $< $@
chmod -x $@
$(OUTPUT)/load_address_4096: load_address.c
- $(CC) $(CFLAGS) $(LDFLAGS) -Wl,-z,max-page-size=0x1000 -pie $< -o $@
+ $(CC) $(CFLAGS) $(LDFLAGS) -Wl,-z,max-page-size=0x1000 -pie -static $< -o $@
$(OUTPUT)/load_address_2097152: load_address.c
- $(CC) $(CFLAGS) $(LDFLAGS) -Wl,-z,max-page-size=0x200000 -pie $< -o $@
+ $(CC) $(CFLAGS) $(LDFLAGS) -Wl,-z,max-page-size=0x200000 -pie -static $< -o $@
$(OUTPUT)/load_address_16777216: load_address.c
- $(CC) $(CFLAGS) $(LDFLAGS) -Wl,-z,max-page-size=0x1000000 -pie $< -o $@
+ $(CC) $(CFLAGS) $(LDFLAGS) -Wl,-z,max-page-size=0x1000000 -pie -static $< -o $@
/kvm_create_max_vcpus
/kvm_page_table_test
/memslot_modification_stress_test
+/memslot_perf_test
/set_memory_region_test
/steal_time
UNAME_M := s390x
endif
-LIBKVM = lib/assert.c lib/elf.c lib/io.c lib/kvm_util.c lib/sparsebit.c lib/test_util.c lib/guest_modes.c lib/perf_test_util.c
+LIBKVM = lib/assert.c lib/elf.c lib/io.c lib/kvm_util.c lib/rbtree.c lib/sparsebit.c lib/test_util.c lib/guest_modes.c lib/perf_test_util.c
LIBKVM_x86_64 = lib/x86_64/processor.c lib/x86_64/vmx.c lib/x86_64/svm.c lib/x86_64/ucall.c lib/x86_64/handlers.S
LIBKVM_aarch64 = lib/aarch64/processor.c lib/aarch64/ucall.c
LIBKVM_s390x = lib/s390x/processor.c lib/s390x/ucall.c lib/s390x/diag318_test_handler.c
TEST_GEN_PROGS_x86_64 += kvm_create_max_vcpus
TEST_GEN_PROGS_x86_64 += kvm_page_table_test
TEST_GEN_PROGS_x86_64 += memslot_modification_stress_test
+TEST_GEN_PROGS_x86_64 += memslot_perf_test
TEST_GEN_PROGS_x86_64 += set_memory_region_test
TEST_GEN_PROGS_x86_64 += steal_time
#define _GNU_SOURCE /* for pipe2 */
+#include <inttypes.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
static int nr_vcpus = 1;
static uint64_t guest_percpu_mem_size = DEFAULT_PER_VCPU_MEM_SIZE;
+static size_t demand_paging_size;
static char *guest_data_prototype;
static void *vcpu_worker(void *data)
return NULL;
}
-static int handle_uffd_page_request(int uffd, uint64_t addr)
+static int handle_uffd_page_request(int uffd_mode, int uffd, uint64_t addr)
{
- pid_t tid;
+ pid_t tid = syscall(__NR_gettid);
struct timespec start;
struct timespec ts_diff;
- struct uffdio_copy copy;
int r;
- tid = syscall(__NR_gettid);
+ clock_gettime(CLOCK_MONOTONIC, &start);
- copy.src = (uint64_t)guest_data_prototype;
- copy.dst = addr;
- copy.len = perf_test_args.host_page_size;
- copy.mode = 0;
+ if (uffd_mode == UFFDIO_REGISTER_MODE_MISSING) {
+ struct uffdio_copy copy;
- clock_gettime(CLOCK_MONOTONIC, &start);
+ copy.src = (uint64_t)guest_data_prototype;
+ copy.dst = addr;
+ copy.len = demand_paging_size;
+ copy.mode = 0;
- r = ioctl(uffd, UFFDIO_COPY, ©);
- if (r == -1) {
- pr_info("Failed Paged in 0x%lx from thread %d with errno: %d\n",
- addr, tid, errno);
- return r;
+ r = ioctl(uffd, UFFDIO_COPY, ©);
+ if (r == -1) {
+ pr_info("Failed UFFDIO_COPY in 0x%lx from thread %d with errno: %d\n",
+ addr, tid, errno);
+ return r;
+ }
+ } else if (uffd_mode == UFFDIO_REGISTER_MODE_MINOR) {
+ struct uffdio_continue cont = {0};
+
+ cont.range.start = addr;
+ cont.range.len = demand_paging_size;
+
+ r = ioctl(uffd, UFFDIO_CONTINUE, &cont);
+ if (r == -1) {
+ pr_info("Failed UFFDIO_CONTINUE in 0x%lx from thread %d with errno: %d\n",
+ addr, tid, errno);
+ return r;
+ }
+ } else {
+ TEST_FAIL("Invalid uffd mode %d", uffd_mode);
}
ts_diff = timespec_elapsed(start);
- PER_PAGE_DEBUG("UFFDIO_COPY %d \t%ld ns\n", tid,
+ PER_PAGE_DEBUG("UFFD page-in %d \t%ld ns\n", tid,
timespec_to_ns(ts_diff));
PER_PAGE_DEBUG("Paged in %ld bytes at 0x%lx from thread %d\n",
- perf_test_args.host_page_size, addr, tid);
+ demand_paging_size, addr, tid);
return 0;
}
bool quit_uffd_thread;
struct uffd_handler_args {
+ int uffd_mode;
int uffd;
int pipefd;
useconds_t delay;
if (r == -1) {
if (errno == EAGAIN)
continue;
- pr_info("Read of uffd gor errno %d", errno);
+ pr_info("Read of uffd got errno %d\n", errno);
return NULL;
}
if (delay)
usleep(delay);
addr = msg.arg.pagefault.address;
- r = handle_uffd_page_request(uffd, addr);
+ r = handle_uffd_page_request(uffd_args->uffd_mode, uffd, addr);
if (r < 0)
return NULL;
pages++;
return NULL;
}
-static int setup_demand_paging(struct kvm_vm *vm,
- pthread_t *uffd_handler_thread, int pipefd,
- useconds_t uffd_delay,
- struct uffd_handler_args *uffd_args,
- void *hva, uint64_t len)
+static void setup_demand_paging(struct kvm_vm *vm,
+ pthread_t *uffd_handler_thread, int pipefd,
+ int uffd_mode, useconds_t uffd_delay,
+ struct uffd_handler_args *uffd_args,
+ void *hva, void *alias, uint64_t len)
{
+ bool is_minor = (uffd_mode == UFFDIO_REGISTER_MODE_MINOR);
int uffd;
struct uffdio_api uffdio_api;
struct uffdio_register uffdio_register;
+ uint64_t expected_ioctls = ((uint64_t) 1) << _UFFDIO_COPY;
- uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
- if (uffd == -1) {
- pr_info("uffd creation failed\n");
- return -1;
+ PER_PAGE_DEBUG("Userfaultfd %s mode, faults resolved with %s\n",
+ is_minor ? "MINOR" : "MISSING",
+ is_minor ? "UFFDIO_CONINUE" : "UFFDIO_COPY");
+
+ /* In order to get minor faults, prefault via the alias. */
+ if (is_minor) {
+ size_t p;
+
+ expected_ioctls = ((uint64_t) 1) << _UFFDIO_CONTINUE;
+
+ TEST_ASSERT(alias != NULL, "Alias required for minor faults");
+ for (p = 0; p < (len / demand_paging_size); ++p) {
+ memcpy(alias + (p * demand_paging_size),
+ guest_data_prototype, demand_paging_size);
+ }
}
+ uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
+ TEST_ASSERT(uffd >= 0, "uffd creation failed, errno: %d", errno);
+
uffdio_api.api = UFFD_API;
uffdio_api.features = 0;
- if (ioctl(uffd, UFFDIO_API, &uffdio_api) == -1) {
- pr_info("ioctl uffdio_api failed\n");
- return -1;
- }
+ TEST_ASSERT(ioctl(uffd, UFFDIO_API, &uffdio_api) != -1,
+ "ioctl UFFDIO_API failed: %" PRIu64,
+ (uint64_t)uffdio_api.api);
uffdio_register.range.start = (uint64_t)hva;
uffdio_register.range.len = len;
- uffdio_register.mode = UFFDIO_REGISTER_MODE_MISSING;
- if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register) == -1) {
- pr_info("ioctl uffdio_register failed\n");
- return -1;
- }
-
- if ((uffdio_register.ioctls & UFFD_API_RANGE_IOCTLS) !=
- UFFD_API_RANGE_IOCTLS) {
- pr_info("unexpected userfaultfd ioctl set\n");
- return -1;
- }
+ uffdio_register.mode = uffd_mode;
+ TEST_ASSERT(ioctl(uffd, UFFDIO_REGISTER, &uffdio_register) != -1,
+ "ioctl UFFDIO_REGISTER failed");
+ TEST_ASSERT((uffdio_register.ioctls & expected_ioctls) ==
+ expected_ioctls, "missing userfaultfd ioctls");
+ uffd_args->uffd_mode = uffd_mode;
uffd_args->uffd = uffd;
uffd_args->pipefd = pipefd;
uffd_args->delay = uffd_delay;
PER_VCPU_DEBUG("Created uffd thread for HVA range [%p, %p)\n",
hva, hva + len);
-
- return 0;
}
struct test_params {
- bool use_uffd;
+ int uffd_mode;
useconds_t uffd_delay;
+ enum vm_mem_backing_src_type src_type;
bool partition_vcpu_memory_access;
};
int r;
vm = perf_test_create_vm(mode, nr_vcpus, guest_percpu_mem_size,
- VM_MEM_SRC_ANONYMOUS);
+ p->src_type);
perf_test_args.wr_fract = 1;
- guest_data_prototype = malloc(perf_test_args.host_page_size);
+ demand_paging_size = get_backing_src_pagesz(p->src_type);
+
+ guest_data_prototype = malloc(demand_paging_size);
TEST_ASSERT(guest_data_prototype,
"Failed to allocate buffer for guest data pattern");
- memset(guest_data_prototype, 0xAB, perf_test_args.host_page_size);
+ memset(guest_data_prototype, 0xAB, demand_paging_size);
vcpu_threads = malloc(nr_vcpus * sizeof(*vcpu_threads));
TEST_ASSERT(vcpu_threads, "Memory allocation failed");
perf_test_setup_vcpus(vm, nr_vcpus, guest_percpu_mem_size,
p->partition_vcpu_memory_access);
- if (p->use_uffd) {
+ if (p->uffd_mode) {
uffd_handler_threads =
malloc(nr_vcpus * sizeof(*uffd_handler_threads));
TEST_ASSERT(uffd_handler_threads, "Memory allocation failed");
for (vcpu_id = 0; vcpu_id < nr_vcpus; vcpu_id++) {
vm_paddr_t vcpu_gpa;
void *vcpu_hva;
+ void *vcpu_alias;
uint64_t vcpu_mem_size;
PER_VCPU_DEBUG("Added VCPU %d with test mem gpa [%lx, %lx)\n",
vcpu_id, vcpu_gpa, vcpu_gpa + vcpu_mem_size);
- /* Cache the HVA pointer of the region */
+ /* Cache the host addresses of the region */
vcpu_hva = addr_gpa2hva(vm, vcpu_gpa);
+ vcpu_alias = addr_gpa2alias(vm, vcpu_gpa);
/*
* Set up user fault fd to handle demand paging
O_CLOEXEC | O_NONBLOCK);
TEST_ASSERT(!r, "Failed to set up pipefd");
- r = setup_demand_paging(vm,
- &uffd_handler_threads[vcpu_id],
- pipefds[vcpu_id * 2],
- p->uffd_delay, &uffd_args[vcpu_id],
- vcpu_hva, vcpu_mem_size);
- if (r < 0)
- exit(-r);
+ setup_demand_paging(vm, &uffd_handler_threads[vcpu_id],
+ pipefds[vcpu_id * 2], p->uffd_mode,
+ p->uffd_delay, &uffd_args[vcpu_id],
+ vcpu_hva, vcpu_alias,
+ vcpu_mem_size);
}
}
pr_info("All vCPU threads joined\n");
- if (p->use_uffd) {
+ if (p->uffd_mode) {
char c;
/* Tell the user fault fd handler threads to quit */
free(guest_data_prototype);
free(vcpu_threads);
- if (p->use_uffd) {
+ if (p->uffd_mode) {
free(uffd_handler_threads);
free(uffd_args);
free(pipefds);
static void help(char *name)
{
puts("");
- printf("usage: %s [-h] [-m mode] [-u] [-d uffd_delay_usec]\n"
- " [-b memory] [-v vcpus] [-o]\n", name);
+ printf("usage: %s [-h] [-m vm_mode] [-u uffd_mode] [-d uffd_delay_usec]\n"
+ " [-b memory] [-t type] [-v vcpus] [-o]\n", name);
guest_modes_help();
- printf(" -u: use User Fault FD to handle vCPU page\n"
- " faults.\n");
+ printf(" -u: use userfaultfd to handle vCPU page faults. Mode is a\n"
+ " UFFD registration mode: 'MISSING' or 'MINOR'.\n");
printf(" -d: add a delay in usec to the User Fault\n"
" FD handler to simulate demand paging\n"
" overheads. Ignored without -u.\n");
printf(" -b: specify the size of the memory region which should be\n"
" demand paged by each vCPU. e.g. 10M or 3G.\n"
" Default: 1G\n");
+ printf(" -t: The type of backing memory to use. Default: anonymous\n");
+ backing_src_help();
printf(" -v: specify the number of vCPUs to run.\n");
printf(" -o: Overlap guest memory accesses instead of partitioning\n"
" them into a separate region of memory for each vCPU.\n");
{
int max_vcpus = kvm_check_cap(KVM_CAP_MAX_VCPUS);
struct test_params p = {
+ .src_type = VM_MEM_SRC_ANONYMOUS,
.partition_vcpu_memory_access = true,
};
int opt;
guest_modes_append_default();
- while ((opt = getopt(argc, argv, "hm:ud:b:v:o")) != -1) {
+ while ((opt = getopt(argc, argv, "hm:u:d:b:t:v:o")) != -1) {
switch (opt) {
case 'm':
guest_modes_cmdline(optarg);
break;
case 'u':
- p.use_uffd = true;
+ if (!strcmp("MISSING", optarg))
+ p.uffd_mode = UFFDIO_REGISTER_MODE_MISSING;
+ else if (!strcmp("MINOR", optarg))
+ p.uffd_mode = UFFDIO_REGISTER_MODE_MINOR;
+ TEST_ASSERT(p.uffd_mode, "UFFD mode must be 'MISSING' or 'MINOR'.");
break;
case 'd':
p.uffd_delay = strtoul(optarg, NULL, 0);
case 'b':
guest_percpu_mem_size = parse_size(optarg);
break;
+ case 't':
+ p.src_type = parse_backing_src_type(optarg);
+ break;
case 'v':
nr_vcpus = atoi(optarg);
TEST_ASSERT(nr_vcpus > 0 && nr_vcpus <= max_vcpus,
}
}
+ if (p.uffd_mode == UFFDIO_REGISTER_MODE_MINOR &&
+ !backing_src_is_shared(p.src_type)) {
+ TEST_FAIL("userfaultfd MINOR mode requires shared memory; pick a different -t");
+ }
+
for_each_guest_mode(run_test, &p);
return 0;
TEST_ASSERT(false, "%s: [%d] child escaped the ninja\n", __func__, run);
}
+void wait_for_child_setup(pid_t pid)
+{
+ /*
+ * Wait for the child to post to the semaphore, but wake up periodically
+ * to check if the child exited prematurely.
+ */
+ for (;;) {
+ const struct timespec wait_period = { .tv_sec = 1 };
+ int status;
+
+ if (!sem_timedwait(sem, &wait_period))
+ return;
+
+ /* Child is still running, keep waiting. */
+ if (pid != waitpid(pid, &status, WNOHANG))
+ continue;
+
+ /*
+ * Child is no longer running, which is not expected.
+ *
+ * If it exited with a non-zero status, we explicitly forward
+ * the child's status in case it exited with KSFT_SKIP.
+ */
+ if (WIFEXITED(status))
+ exit(WEXITSTATUS(status));
+ else
+ TEST_ASSERT(false, "Child exited unexpectedly");
+ }
+}
+
int main(int argc, char **argv)
{
uint32_t i;
run_test(i); /* This function always exits */
pr_debug("%s: [%d] waiting semaphore\n", __func__, i);
- sem_wait(sem);
+ wait_for_child_setup(pid);
r = (rand() % DELAY_US_MAX) + 1;
pr_debug("%s: [%d] waiting %dus\n", __func__, i, r);
usleep(r);
VM_MODE_P40V48_4K,
VM_MODE_P40V48_64K,
VM_MODE_PXXV48_4K, /* For 48bits VA but ANY bits PA */
+ VM_MODE_P47V64_4K,
NUM_VM_MODES,
};
#elif defined(__s390x__)
-#define VM_MODE_DEFAULT VM_MODE_P52V48_4K
+#define VM_MODE_DEFAULT VM_MODE_P47V64_4K
#define MIN_PAGE_SHIFT 12U
#define ptes_per_page(page_size) ((page_size) / 16)
};
extern const struct vm_guest_mode_params vm_guest_mode_params[];
+int open_kvm_dev_path_or_exit(void);
int kvm_check_cap(long cap);
int vm_enable_cap(struct kvm_vm *vm, struct kvm_enable_cap *cap);
int vcpu_enable_cap(struct kvm_vm *vm, uint32_t vcpu_id,
void *addr_gpa2hva(struct kvm_vm *vm, vm_paddr_t gpa);
void *addr_gva2hva(struct kvm_vm *vm, vm_vaddr_t gva);
vm_paddr_t addr_hva2gpa(struct kvm_vm *vm, void *hva);
+void *addr_gpa2alias(struct kvm_vm *vm, vm_paddr_t gpa);
/*
* Address Guest Virtual to Guest Physical
uint32_t num_percpu_pages, void *guest_code,
uint32_t vcpuids[]);
-/* Like vm_create_default_with_vcpus, but accepts mode as a parameter */
+/* Like vm_create_default_with_vcpus, but accepts mode and slot0 memory as a parameter */
struct kvm_vm *vm_create_with_vcpus(enum vm_guest_mode mode, uint32_t nr_vcpus,
- uint64_t extra_mem_pages, uint32_t num_percpu_pages,
- void *guest_code, uint32_t vcpuids[]);
+ uint64_t slot0_mem_pages, uint64_t extra_mem_pages,
+ uint32_t num_percpu_pages, void *guest_code,
+ uint32_t vcpuids[]);
/*
* Adds a vCPU with reasonable defaults (e.g. a stack)
unsigned int vm_get_page_size(struct kvm_vm *vm);
unsigned int vm_get_page_shift(struct kvm_vm *vm);
-unsigned int vm_get_max_gfn(struct kvm_vm *vm);
+uint64_t vm_get_max_gfn(struct kvm_vm *vm);
int vm_get_fd(struct kvm_vm *vm);
unsigned int vm_calc_num_guest_pages(enum vm_guest_mode mode, size_t size);
#include <errno.h>
#include <unistd.h>
#include <fcntl.h>
+#include <sys/mman.h>
#include "kselftest.h"
static inline int _no_printf(const char *format, ...) { return 0; }
VM_MEM_SRC_ANONYMOUS_HUGETLB_1GB,
VM_MEM_SRC_ANONYMOUS_HUGETLB_2GB,
VM_MEM_SRC_ANONYMOUS_HUGETLB_16GB,
+ VM_MEM_SRC_SHMEM,
+ VM_MEM_SRC_SHARED_HUGETLB,
NUM_SRC_TYPES,
};
void backing_src_help(void);
enum vm_mem_backing_src_type parse_backing_src_type(const char *type_name);
+/*
+ * Whether or not the given source type is shared memory (as opposed to
+ * anonymous).
+ */
+static inline bool backing_src_is_shared(enum vm_mem_backing_src_type t)
+{
+ return vm_mem_backing_src_alias(t)->flag & MAP_SHARED;
+}
+
#endif /* SELFTEST_KVM_TEST_UTIL_H */
/* Create a VM with enough guest pages */
guest_num_pages = test_mem_size / guest_page_size;
- vm = vm_create_with_vcpus(mode, nr_vcpus,
+ vm = vm_create_with_vcpus(mode, nr_vcpus, DEFAULT_GUEST_PHY_PAGES,
guest_num_pages, 0, guest_code, NULL);
/* Align down GPA of the testing memslot */
}
/*
+ * Open KVM_DEV_PATH if available, otherwise exit the entire program.
+ *
+ * Input Args:
+ * flags - The flags to pass when opening KVM_DEV_PATH.
+ *
+ * Return:
+ * The opened file descriptor of /dev/kvm.
+ */
+static int _open_kvm_dev_path_or_exit(int flags)
+{
+ int fd;
+
+ fd = open(KVM_DEV_PATH, flags);
+ if (fd < 0) {
+ print_skip("%s not available, is KVM loaded? (errno: %d)",
+ KVM_DEV_PATH, errno);
+ exit(KSFT_SKIP);
+ }
+
+ return fd;
+}
+
+int open_kvm_dev_path_or_exit(void)
+{
+ return _open_kvm_dev_path_or_exit(O_RDONLY);
+}
+
+/*
* Capability
*
* Input Args:
int ret;
int kvm_fd;
- kvm_fd = open(KVM_DEV_PATH, O_RDONLY);
- if (kvm_fd < 0)
- exit(KSFT_SKIP);
-
+ kvm_fd = open_kvm_dev_path_or_exit();
ret = ioctl(kvm_fd, KVM_CHECK_EXTENSION, cap);
- TEST_ASSERT(ret != -1, "KVM_CHECK_EXTENSION IOCTL failed,\n"
+ TEST_ASSERT(ret >= 0, "KVM_CHECK_EXTENSION IOCTL failed,\n"
" rc: %i errno: %i", ret, errno);
close(kvm_fd);
static void vm_open(struct kvm_vm *vm, int perm)
{
- vm->kvm_fd = open(KVM_DEV_PATH, perm);
- if (vm->kvm_fd < 0)
- exit(KSFT_SKIP);
+ vm->kvm_fd = _open_kvm_dev_path_or_exit(perm);
if (!kvm_check_cap(KVM_CAP_IMMEDIATE_EXIT)) {
print_skip("immediate_exit not available");
[VM_MODE_P40V48_4K] = "PA-bits:40, VA-bits:48, 4K pages",
[VM_MODE_P40V48_64K] = "PA-bits:40, VA-bits:48, 64K pages",
[VM_MODE_PXXV48_4K] = "PA-bits:ANY, VA-bits:48, 4K pages",
+ [VM_MODE_P47V64_4K] = "PA-bits:47, VA-bits:64, 4K pages",
};
_Static_assert(sizeof(strings)/sizeof(char *) == NUM_VM_MODES,
"Missing new mode strings?");
{ 40, 48, 0x1000, 12 },
{ 40, 48, 0x10000, 16 },
{ 0, 0, 0x1000, 12 },
+ { 47, 64, 0x1000, 12 },
};
_Static_assert(sizeof(vm_guest_mode_params)/sizeof(struct vm_guest_mode_params) == NUM_VM_MODES,
"Missing new mode params?");
TEST_ASSERT(vm != NULL, "Insufficient Memory");
INIT_LIST_HEAD(&vm->vcpus);
- INIT_LIST_HEAD(&vm->userspace_mem_regions);
+ vm->regions.gpa_tree = RB_ROOT;
+ vm->regions.hva_tree = RB_ROOT;
+ hash_init(vm->regions.slot_hash);
vm->mode = mode;
vm->type = 0;
TEST_FAIL("VM_MODE_PXXV48_4K not supported on non-x86 platforms");
#endif
break;
+ case VM_MODE_P47V64_4K:
+ vm->pgtable_levels = 5;
+ break;
default:
TEST_FAIL("Unknown guest mode, mode: 0x%x", mode);
}
return vm;
}
+/*
+ * VM Create with customized parameters
+ *
+ * Input Args:
+ * mode - VM Mode (e.g. VM_MODE_P52V48_4K)
+ * nr_vcpus - VCPU count
+ * slot0_mem_pages - Slot0 physical memory size
+ * extra_mem_pages - Non-slot0 physical memory total size
+ * num_percpu_pages - Per-cpu physical memory pages
+ * guest_code - Guest entry point
+ * vcpuids - VCPU IDs
+ *
+ * Output Args: None
+ *
+ * Return:
+ * Pointer to opaque structure that describes the created VM.
+ *
+ * Creates a VM with the mode specified by mode (e.g. VM_MODE_P52V48_4K),
+ * with customized slot0 memory size, at least 512 pages currently.
+ * extra_mem_pages is only used to calculate the maximum page table size,
+ * no real memory allocation for non-slot0 memory in this function.
+ */
struct kvm_vm *vm_create_with_vcpus(enum vm_guest_mode mode, uint32_t nr_vcpus,
- uint64_t extra_mem_pages, uint32_t num_percpu_pages,
- void *guest_code, uint32_t vcpuids[])
+ uint64_t slot0_mem_pages, uint64_t extra_mem_pages,
+ uint32_t num_percpu_pages, void *guest_code,
+ uint32_t vcpuids[])
{
+ uint64_t vcpu_pages, extra_pg_pages, pages;
+ struct kvm_vm *vm;
+ int i;
+
+ /* Force slot0 memory size not small than DEFAULT_GUEST_PHY_PAGES */
+ if (slot0_mem_pages < DEFAULT_GUEST_PHY_PAGES)
+ slot0_mem_pages = DEFAULT_GUEST_PHY_PAGES;
+
/* The maximum page table size for a memory region will be when the
* smallest pages are used. Considering each page contains x page
* table descriptors, the total extra size for page tables (for extra
* N pages) will be: N/x+N/x^2+N/x^3+... which is definitely smaller
* than N/x*2.
*/
- uint64_t vcpu_pages = (DEFAULT_STACK_PGS + num_percpu_pages) * nr_vcpus;
- uint64_t extra_pg_pages = (extra_mem_pages + vcpu_pages) / PTES_PER_MIN_PAGE * 2;
- uint64_t pages = DEFAULT_GUEST_PHY_PAGES + vcpu_pages + extra_pg_pages;
- struct kvm_vm *vm;
- int i;
+ vcpu_pages = (DEFAULT_STACK_PGS + num_percpu_pages) * nr_vcpus;
+ extra_pg_pages = (slot0_mem_pages + extra_mem_pages + vcpu_pages) / PTES_PER_MIN_PAGE * 2;
+ pages = slot0_mem_pages + vcpu_pages + extra_pg_pages;
TEST_ASSERT(nr_vcpus <= kvm_check_cap(KVM_CAP_MAX_VCPUS),
"nr_vcpus = %d too large for host, max-vcpus = %d",
uint32_t num_percpu_pages, void *guest_code,
uint32_t vcpuids[])
{
- return vm_create_with_vcpus(VM_MODE_DEFAULT, nr_vcpus, extra_mem_pages,
- num_percpu_pages, guest_code, vcpuids);
+ return vm_create_with_vcpus(VM_MODE_DEFAULT, nr_vcpus, DEFAULT_GUEST_PHY_PAGES,
+ extra_mem_pages, num_percpu_pages, guest_code, vcpuids);
}
struct kvm_vm *vm_create_default(uint32_t vcpuid, uint64_t extra_mem_pages,
*/
void kvm_vm_restart(struct kvm_vm *vmp, int perm)
{
+ int ctr;
struct userspace_mem_region *region;
vm_open(vmp, perm);
if (vmp->has_irqchip)
vm_create_irqchip(vmp);
- list_for_each_entry(region, &vmp->userspace_mem_regions, list) {
+ hash_for_each(vmp->regions.slot_hash, ctr, region, slot_node) {
int ret = ioctl(vmp->fd, KVM_SET_USER_MEMORY_REGION, ®ion->region);
TEST_ASSERT(ret == 0, "KVM_SET_USER_MEMORY_REGION IOCTL failed,\n"
" rc: %i errno: %i\n"
static struct userspace_mem_region *
userspace_mem_region_find(struct kvm_vm *vm, uint64_t start, uint64_t end)
{
- struct userspace_mem_region *region;
+ struct rb_node *node;
- list_for_each_entry(region, &vm->userspace_mem_regions, list) {
+ for (node = vm->regions.gpa_tree.rb_node; node; ) {
+ struct userspace_mem_region *region =
+ container_of(node, struct userspace_mem_region, gpa_node);
uint64_t existing_start = region->region.guest_phys_addr;
uint64_t existing_end = region->region.guest_phys_addr
+ region->region.memory_size - 1;
if (start <= existing_end && end >= existing_start)
return region;
+
+ if (start < existing_start)
+ node = node->rb_left;
+ else
+ node = node->rb_right;
}
return NULL;
}
static void __vm_mem_region_delete(struct kvm_vm *vm,
- struct userspace_mem_region *region)
+ struct userspace_mem_region *region,
+ bool unlink)
{
int ret;
- list_del(®ion->list);
+ if (unlink) {
+ rb_erase(®ion->gpa_node, &vm->regions.gpa_tree);
+ rb_erase(®ion->hva_node, &vm->regions.hva_tree);
+ hash_del(®ion->slot_node);
+ }
region->region.memory_size = 0;
ret = ioctl(vm->fd, KVM_SET_USER_MEMORY_REGION, ®ion->region);
*/
void kvm_vm_free(struct kvm_vm *vmp)
{
- struct userspace_mem_region *region, *tmp;
+ int ctr;
+ struct hlist_node *node;
+ struct userspace_mem_region *region;
if (vmp == NULL)
return;
/* Free userspace_mem_regions. */
- list_for_each_entry_safe(region, tmp, &vmp->userspace_mem_regions, list)
- __vm_mem_region_delete(vmp, region);
+ hash_for_each_safe(vmp->regions.slot_hash, ctr, node, region, slot_node)
+ __vm_mem_region_delete(vmp, region, false);
/* Free sparsebit arrays. */
sparsebit_free(&vmp->vpages_valid);
return 0;
}
+static void vm_userspace_mem_region_gpa_insert(struct rb_root *gpa_tree,
+ struct userspace_mem_region *region)
+{
+ struct rb_node **cur, *parent;
+
+ for (cur = &gpa_tree->rb_node, parent = NULL; *cur; ) {
+ struct userspace_mem_region *cregion;
+
+ cregion = container_of(*cur, typeof(*cregion), gpa_node);
+ parent = *cur;
+ if (region->region.guest_phys_addr <
+ cregion->region.guest_phys_addr)
+ cur = &(*cur)->rb_left;
+ else {
+ TEST_ASSERT(region->region.guest_phys_addr !=
+ cregion->region.guest_phys_addr,
+ "Duplicate GPA in region tree");
+
+ cur = &(*cur)->rb_right;
+ }
+ }
+
+ rb_link_node(®ion->gpa_node, parent, cur);
+ rb_insert_color(®ion->gpa_node, gpa_tree);
+}
+
+static void vm_userspace_mem_region_hva_insert(struct rb_root *hva_tree,
+ struct userspace_mem_region *region)
+{
+ struct rb_node **cur, *parent;
+
+ for (cur = &hva_tree->rb_node, parent = NULL; *cur; ) {
+ struct userspace_mem_region *cregion;
+
+ cregion = container_of(*cur, typeof(*cregion), hva_node);
+ parent = *cur;
+ if (region->host_mem < cregion->host_mem)
+ cur = &(*cur)->rb_left;
+ else {
+ TEST_ASSERT(region->host_mem !=
+ cregion->host_mem,
+ "Duplicate HVA in region tree");
+
+ cur = &(*cur)->rb_right;
+ }
+ }
+
+ rb_link_node(®ion->hva_node, parent, cur);
+ rb_insert_color(®ion->hva_node, hva_tree);
+}
+
/*
* VM Userspace Memory Region Add
*
* Input Args:
* vm - Virtual Machine
- * backing_src - Storage source for this region.
- * NULL to use anonymous memory.
+ * src_type - Storage source for this region.
+ * NULL to use anonymous memory.
* guest_paddr - Starting guest physical address
* slot - KVM region slot
* npages - Number of physical pages
(uint64_t) region->region.memory_size);
/* Confirm no region with the requested slot already exists. */
- list_for_each_entry(region, &vm->userspace_mem_regions, list) {
+ hash_for_each_possible(vm->regions.slot_hash, region, slot_node,
+ slot) {
if (region->region.slot != slot)
continue;
if (alignment > 1)
region->mmap_size += alignment;
+ region->fd = -1;
+ if (backing_src_is_shared(src_type)) {
+ int memfd_flags = MFD_CLOEXEC;
+
+ if (src_type == VM_MEM_SRC_SHARED_HUGETLB)
+ memfd_flags |= MFD_HUGETLB;
+
+ region->fd = memfd_create("kvm_selftest", memfd_flags);
+ TEST_ASSERT(region->fd != -1,
+ "memfd_create failed, errno: %i", errno);
+
+ ret = ftruncate(region->fd, region->mmap_size);
+ TEST_ASSERT(ret == 0, "ftruncate failed, errno: %i", errno);
+
+ ret = fallocate(region->fd,
+ FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0,
+ region->mmap_size);
+ TEST_ASSERT(ret == 0, "fallocate failed, errno: %i", errno);
+ }
+
region->mmap_start = mmap(NULL, region->mmap_size,
PROT_READ | PROT_WRITE,
- MAP_PRIVATE | MAP_ANONYMOUS
- | vm_mem_backing_src_alias(src_type)->flag,
- -1, 0);
+ vm_mem_backing_src_alias(src_type)->flag,
+ region->fd, 0);
TEST_ASSERT(region->mmap_start != MAP_FAILED,
"test_malloc failed, mmap_start: %p errno: %i",
region->mmap_start, errno);
ret, errno, slot, flags,
guest_paddr, (uint64_t) region->region.memory_size);
- /* Add to linked-list of memory regions. */
- list_add(®ion->list, &vm->userspace_mem_regions);
+ /* Add to quick lookup data structures */
+ vm_userspace_mem_region_gpa_insert(&vm->regions.gpa_tree, region);
+ vm_userspace_mem_region_hva_insert(&vm->regions.hva_tree, region);
+ hash_add(vm->regions.slot_hash, ®ion->slot_node, slot);
+
+ /* If shared memory, create an alias. */
+ if (region->fd >= 0) {
+ region->mmap_alias = mmap(NULL, region->mmap_size,
+ PROT_READ | PROT_WRITE,
+ vm_mem_backing_src_alias(src_type)->flag,
+ region->fd, 0);
+ TEST_ASSERT(region->mmap_alias != MAP_FAILED,
+ "mmap of alias failed, errno: %i", errno);
+
+ /* Align host alias address */
+ region->host_alias = align(region->mmap_alias, alignment);
+ }
}
/*
{
struct userspace_mem_region *region;
- list_for_each_entry(region, &vm->userspace_mem_regions, list) {
+ hash_for_each_possible(vm->regions.slot_hash, region, slot_node,
+ memslot)
if (region->region.slot == memslot)
return region;
- }
fprintf(stderr, "No mem region with the requested slot found,\n"
" requested slot: %u\n", memslot);
*/
void vm_mem_region_delete(struct kvm_vm *vm, uint32_t slot)
{
- __vm_mem_region_delete(vm, memslot2region(vm, slot));
+ __vm_mem_region_delete(vm, memslot2region(vm, slot), true);
}
/*
{
int dev_fd, ret;
- dev_fd = open(KVM_DEV_PATH, O_RDONLY);
- if (dev_fd < 0)
- exit(KSFT_SKIP);
+ dev_fd = open_kvm_dev_path_or_exit();
ret = ioctl(dev_fd, KVM_GET_VCPU_MMAP_SIZE, NULL);
TEST_ASSERT(ret >= sizeof(struct kvm_run),
uint64_t pages = (sz >> vm->page_shift) + ((sz % vm->page_size) != 0);
virt_pgd_alloc(vm, pgd_memslot);
+ vm_paddr_t paddr = vm_phy_pages_alloc(vm, pages,
+ KVM_UTIL_MIN_PFN * vm->page_size,
+ data_memslot);
/*
* Find an unused range of virtual page addresses of at least
/* Map the virtual pages. */
for (vm_vaddr_t vaddr = vaddr_start; pages > 0;
- pages--, vaddr += vm->page_size) {
- vm_paddr_t paddr;
-
- paddr = vm_phy_page_alloc(vm,
- KVM_UTIL_MIN_PFN * vm->page_size, data_memslot);
+ pages--, vaddr += vm->page_size, paddr += vm->page_size) {
virt_pg_map(vm, vaddr, paddr, pgd_memslot);
{
struct userspace_mem_region *region;
- list_for_each_entry(region, &vm->userspace_mem_regions, list) {
- if ((gpa >= region->region.guest_phys_addr)
- && (gpa <= (region->region.guest_phys_addr
- + region->region.memory_size - 1)))
- return (void *) ((uintptr_t) region->host_mem
- + (gpa - region->region.guest_phys_addr));
+ region = userspace_mem_region_find(vm, gpa, gpa);
+ if (!region) {
+ TEST_FAIL("No vm physical memory at 0x%lx", gpa);
+ return NULL;
}
- TEST_FAIL("No vm physical memory at 0x%lx", gpa);
- return NULL;
+ return (void *)((uintptr_t)region->host_mem
+ + (gpa - region->region.guest_phys_addr));
}
/*
*/
vm_paddr_t addr_hva2gpa(struct kvm_vm *vm, void *hva)
{
- struct userspace_mem_region *region;
+ struct rb_node *node;
- list_for_each_entry(region, &vm->userspace_mem_regions, list) {
- if ((hva >= region->host_mem)
- && (hva <= (region->host_mem
- + region->region.memory_size - 1)))
- return (vm_paddr_t) ((uintptr_t)
- region->region.guest_phys_addr
- + (hva - (uintptr_t) region->host_mem));
+ for (node = vm->regions.hva_tree.rb_node; node; ) {
+ struct userspace_mem_region *region =
+ container_of(node, struct userspace_mem_region, hva_node);
+
+ if (hva >= region->host_mem) {
+ if (hva <= (region->host_mem
+ + region->region.memory_size - 1))
+ return (vm_paddr_t)((uintptr_t)
+ region->region.guest_phys_addr
+ + (hva - (uintptr_t)region->host_mem));
+
+ node = node->rb_right;
+ } else
+ node = node->rb_left;
}
TEST_FAIL("No mapping to a guest physical address, hva: %p", hva);
}
/*
+ * Address VM physical to Host Virtual *alias*.
+ *
+ * Input Args:
+ * vm - Virtual Machine
+ * gpa - VM physical address
+ *
+ * Output Args: None
+ *
+ * Return:
+ * Equivalent address within the host virtual *alias* area, or NULL
+ * (without failing the test) if the guest memory is not shared (so
+ * no alias exists).
+ *
+ * When vm_create() and related functions are called with a shared memory
+ * src_type, we also create a writable, shared alias mapping of the
+ * underlying guest memory. This allows the host to manipulate guest memory
+ * without mapping that memory in the guest's address space. And, for
+ * userfaultfd-based demand paging, we can do so without triggering userfaults.
+ */
+void *addr_gpa2alias(struct kvm_vm *vm, vm_paddr_t gpa)
+{
+ struct userspace_mem_region *region;
+ uintptr_t offset;
+
+ region = userspace_mem_region_find(vm, gpa, gpa);
+ if (!region)
+ return NULL;
+
+ if (!region->host_alias)
+ return NULL;
+
+ offset = gpa - region->region.guest_phys_addr;
+ return (void *) ((uintptr_t) region->host_alias + offset);
+}
+
+/*
* VM Create IRQ Chip
*
* Input Args:
*/
void vm_dump(FILE *stream, struct kvm_vm *vm, uint8_t indent)
{
+ int ctr;
struct userspace_mem_region *region;
struct vcpu *vcpu;
fprintf(stream, "%*sfd: %i\n", indent, "", vm->fd);
fprintf(stream, "%*spage_size: 0x%x\n", indent, "", vm->page_size);
fprintf(stream, "%*sMem Regions:\n", indent, "");
- list_for_each_entry(region, &vm->userspace_mem_regions, list) {
+ hash_for_each(vm->regions.slot_hash, ctr, region, slot_node) {
fprintf(stream, "%*sguest_phys: 0x%lx size: 0x%lx "
"host_virt: %p\n", indent + 2, "",
(uint64_t) region->region.guest_phys_addr,
if (vm == NULL) {
/* Ensure that the KVM vendor-specific module is loaded. */
- f = fopen(KVM_DEV_PATH, "r");
- TEST_ASSERT(f != NULL, "Error in opening KVM dev file: %d",
- errno);
- fclose(f);
+ close(open_kvm_dev_path_or_exit());
}
f = fopen("/sys/module/kvm_intel/parameters/unrestricted_guest", "r");
return vm->page_shift;
}
-unsigned int vm_get_max_gfn(struct kvm_vm *vm)
+uint64_t vm_get_max_gfn(struct kvm_vm *vm)
{
return vm->max_gfn;
}
#ifndef SELFTEST_KVM_UTIL_INTERNAL_H
#define SELFTEST_KVM_UTIL_INTERNAL_H
+#include "linux/hashtable.h"
+#include "linux/rbtree.h"
+
#include "sparsebit.h"
struct userspace_mem_region {
int fd;
off_t offset;
void *host_mem;
+ void *host_alias;
void *mmap_start;
+ void *mmap_alias;
size_t mmap_size;
- struct list_head list;
+ struct rb_node gpa_node;
+ struct rb_node hva_node;
+ struct hlist_node slot_node;
};
struct vcpu {
uint32_t dirty_gfns_count;
};
+struct userspace_mem_regions {
+ struct rb_root gpa_tree;
+ struct rb_root hva_tree;
+ DECLARE_HASHTABLE(slot_hash, 9);
+};
+
struct kvm_vm {
int mode;
unsigned long type;
unsigned int va_bits;
uint64_t max_gfn;
struct list_head vcpus;
- struct list_head userspace_mem_regions;
+ struct userspace_mem_regions regions;
struct sparsebit *vpages_valid;
struct sparsebit *vpages_mapped;
bool has_irqchip;
/*
* Copyright (C) 2020, Google LLC.
*/
+#include <inttypes.h>
#include "kvm_util.h"
#include "perf_test_util.h"
TEST_ASSERT(vcpu_memory_bytes % perf_test_args.guest_page_size == 0,
"Guest memory size is not guest page size aligned.");
- vm = vm_create_with_vcpus(mode, vcpus,
+ vm = vm_create_with_vcpus(mode, vcpus, DEFAULT_GUEST_PHY_PAGES,
(vcpus * vcpu_memory_bytes) / perf_test_args.guest_page_size,
0, guest_code, NULL);
*/
TEST_ASSERT(guest_num_pages < vm_get_max_gfn(vm),
"Requested more guest memory than address space allows.\n"
- " guest pages: %lx max gfn: %x vcpus: %d wss: %lx]\n",
+ " guest pages: %" PRIx64 " max gfn: %" PRIx64
+ " vcpus: %d wss: %" PRIx64 "]\n",
guest_num_pages, vm_get_max_gfn(vm), vcpus,
vcpu_memory_bytes);
--- /dev/null
+#include "../../../../lib/rbtree.c"
return 0;
}
+#define ANON_FLAGS (MAP_PRIVATE | MAP_ANONYMOUS)
+#define ANON_HUGE_FLAGS (ANON_FLAGS | MAP_HUGETLB)
+
const struct vm_mem_backing_src_alias *vm_mem_backing_src_alias(uint32_t i)
{
static const struct vm_mem_backing_src_alias aliases[] = {
[VM_MEM_SRC_ANONYMOUS] = {
.name = "anonymous",
- .flag = 0,
+ .flag = ANON_FLAGS,
},
[VM_MEM_SRC_ANONYMOUS_THP] = {
.name = "anonymous_thp",
- .flag = 0,
+ .flag = ANON_FLAGS,
},
[VM_MEM_SRC_ANONYMOUS_HUGETLB] = {
.name = "anonymous_hugetlb",
- .flag = MAP_HUGETLB,
+ .flag = ANON_HUGE_FLAGS,
},
[VM_MEM_SRC_ANONYMOUS_HUGETLB_16KB] = {
.name = "anonymous_hugetlb_16kb",
- .flag = MAP_HUGETLB | MAP_HUGE_16KB,
+ .flag = ANON_HUGE_FLAGS | MAP_HUGE_16KB,
},
[VM_MEM_SRC_ANONYMOUS_HUGETLB_64KB] = {
.name = "anonymous_hugetlb_64kb",
- .flag = MAP_HUGETLB | MAP_HUGE_64KB,
+ .flag = ANON_HUGE_FLAGS | MAP_HUGE_64KB,
},
[VM_MEM_SRC_ANONYMOUS_HUGETLB_512KB] = {
.name = "anonymous_hugetlb_512kb",
- .flag = MAP_HUGETLB | MAP_HUGE_512KB,
+ .flag = ANON_HUGE_FLAGS | MAP_HUGE_512KB,
},
[VM_MEM_SRC_ANONYMOUS_HUGETLB_1MB] = {
.name = "anonymous_hugetlb_1mb",
- .flag = MAP_HUGETLB | MAP_HUGE_1MB,
+ .flag = ANON_HUGE_FLAGS | MAP_HUGE_1MB,
},
[VM_MEM_SRC_ANONYMOUS_HUGETLB_2MB] = {
.name = "anonymous_hugetlb_2mb",
- .flag = MAP_HUGETLB | MAP_HUGE_2MB,
+ .flag = ANON_HUGE_FLAGS | MAP_HUGE_2MB,
},
[VM_MEM_SRC_ANONYMOUS_HUGETLB_8MB] = {
.name = "anonymous_hugetlb_8mb",
- .flag = MAP_HUGETLB | MAP_HUGE_8MB,
+ .flag = ANON_HUGE_FLAGS | MAP_HUGE_8MB,
},
[VM_MEM_SRC_ANONYMOUS_HUGETLB_16MB] = {
.name = "anonymous_hugetlb_16mb",
- .flag = MAP_HUGETLB | MAP_HUGE_16MB,
+ .flag = ANON_HUGE_FLAGS | MAP_HUGE_16MB,
},
[VM_MEM_SRC_ANONYMOUS_HUGETLB_32MB] = {
.name = "anonymous_hugetlb_32mb",
- .flag = MAP_HUGETLB | MAP_HUGE_32MB,
+ .flag = ANON_HUGE_FLAGS | MAP_HUGE_32MB,
},
[VM_MEM_SRC_ANONYMOUS_HUGETLB_256MB] = {
.name = "anonymous_hugetlb_256mb",
- .flag = MAP_HUGETLB | MAP_HUGE_256MB,
+ .flag = ANON_HUGE_FLAGS | MAP_HUGE_256MB,
},
[VM_MEM_SRC_ANONYMOUS_HUGETLB_512MB] = {
.name = "anonymous_hugetlb_512mb",
- .flag = MAP_HUGETLB | MAP_HUGE_512MB,
+ .flag = ANON_HUGE_FLAGS | MAP_HUGE_512MB,
},
[VM_MEM_SRC_ANONYMOUS_HUGETLB_1GB] = {
.name = "anonymous_hugetlb_1gb",
- .flag = MAP_HUGETLB | MAP_HUGE_1GB,
+ .flag = ANON_HUGE_FLAGS | MAP_HUGE_1GB,
},
[VM_MEM_SRC_ANONYMOUS_HUGETLB_2GB] = {
.name = "anonymous_hugetlb_2gb",
- .flag = MAP_HUGETLB | MAP_HUGE_2GB,
+ .flag = ANON_HUGE_FLAGS | MAP_HUGE_2GB,
},
[VM_MEM_SRC_ANONYMOUS_HUGETLB_16GB] = {
.name = "anonymous_hugetlb_16gb",
- .flag = MAP_HUGETLB | MAP_HUGE_16GB,
+ .flag = ANON_HUGE_FLAGS | MAP_HUGE_16GB,
+ },
+ [VM_MEM_SRC_SHMEM] = {
+ .name = "shmem",
+ .flag = MAP_SHARED,
+ },
+ [VM_MEM_SRC_SHARED_HUGETLB] = {
+ .name = "shared_hugetlb",
+ /*
+ * No MAP_HUGETLB, we use MFD_HUGETLB instead. Since
+ * we're using "file backed" memory, we need to specify
+ * this when the FD is created, not when the area is
+ * mapped.
+ */
+ .flag = MAP_SHARED,
},
};
_Static_assert(ARRAY_SIZE(aliases) == NUM_SRC_TYPES,
switch (i) {
case VM_MEM_SRC_ANONYMOUS:
+ case VM_MEM_SRC_SHMEM:
return getpagesize();
case VM_MEM_SRC_ANONYMOUS_THP:
return get_trans_hugepagesz();
case VM_MEM_SRC_ANONYMOUS_HUGETLB:
+ case VM_MEM_SRC_SHARED_HUGETLB:
return get_def_hugetlb_pagesz();
default:
return MAP_HUGE_PAGE_SIZE(flag);
.align 8
/* Fetch current address and append it to idt_handlers. */
- current_handler = .
+666 :
.pushsection .rodata
-.quad current_handler
+ .quad 666b
.popsection
.if ! \has_error
return cpuid;
cpuid = allocate_kvm_cpuid2();
- kvm_fd = open(KVM_DEV_PATH, O_RDONLY);
- if (kvm_fd < 0)
- exit(KSFT_SKIP);
+ kvm_fd = open_kvm_dev_path_or_exit();
ret = ioctl(kvm_fd, KVM_GET_SUPPORTED_CPUID, cpuid);
TEST_ASSERT(ret == 0, "KVM_GET_SUPPORTED_CPUID failed %d %d\n",
buffer.header.nmsrs = 1;
buffer.entry.index = msr_index;
- kvm_fd = open(KVM_DEV_PATH, O_RDONLY);
- if (kvm_fd < 0)
- exit(KSFT_SKIP);
+ kvm_fd = open_kvm_dev_path_or_exit();
r = ioctl(kvm_fd, KVM_GET_MSRS, &buffer.header);
TEST_ASSERT(r == 1, "KVM_GET_MSRS IOCTL failed,\n"
struct kvm_msr_list *list;
int nmsrs, r, kvm_fd;
- kvm_fd = open(KVM_DEV_PATH, O_RDONLY);
- if (kvm_fd < 0)
- exit(KSFT_SKIP);
+ kvm_fd = open_kvm_dev_path_or_exit();
nmsrs = kvm_get_num_msrs_fd(kvm_fd);
list = malloc(sizeof(*list) + nmsrs * sizeof(list->indices[0]));
return cpuid;
cpuid = allocate_kvm_cpuid2();
- kvm_fd = open(KVM_DEV_PATH, O_RDONLY);
- if (kvm_fd < 0)
- exit(KSFT_SKIP);
+ kvm_fd = open_kvm_dev_path_or_exit();
ret = ioctl(kvm_fd, KVM_GET_SUPPORTED_HV_CPUID, cpuid);
TEST_ASSERT(ret == 0, "KVM_GET_SUPPORTED_HV_CPUID failed %d %d\n",
};
static void add_remove_memslot(struct kvm_vm *vm, useconds_t delay,
- uint64_t nr_modifications, uint64_t gpa)
+ uint64_t nr_modifications)
{
+ const uint64_t pages = 1;
+ uint64_t gpa;
int i;
+ /*
+ * Add the dummy memslot just below the perf_test_util memslot, which is
+ * at the top of the guest physical address space.
+ */
+ gpa = guest_test_phys_mem - pages * vm_get_page_size(vm);
+
for (i = 0; i < nr_modifications; i++) {
usleep(delay);
vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, gpa,
- DUMMY_MEMSLOT_INDEX, 1, 0);
+ DUMMY_MEMSLOT_INDEX, pages, 0);
vm_mem_region_delete(vm, DUMMY_MEMSLOT_INDEX);
}
pr_info("Started all vCPUs\n");
add_remove_memslot(vm, p->memslot_modification_delay,
- p->nr_memslot_modifications,
- guest_test_phys_mem +
- (guest_percpu_mem_size * nr_vcpus) +
- perf_test_args.host_page_size +
- perf_test_args.guest_page_size);
+ p->nr_memslot_modifications);
run_vcpus = false;
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * A memslot-related performance benchmark.
+ *
+ * Copyright (C) 2021 Oracle and/or its affiliates.
+ *
+ * Basic guest setup / host vCPU thread code lifted from set_memory_region_test.
+ */
+#include <pthread.h>
+#include <sched.h>
+#include <semaphore.h>
+#include <stdatomic.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <time.h>
+#include <unistd.h>
+
+#include <linux/compiler.h>
+
+#include <test_util.h>
+#include <kvm_util.h>
+#include <processor.h>
+
+#define VCPU_ID 0
+
+#define MEM_SIZE ((512U << 20) + 4096)
+#define MEM_SIZE_PAGES (MEM_SIZE / 4096)
+#define MEM_GPA 0x10000000UL
+#define MEM_AUX_GPA MEM_GPA
+#define MEM_SYNC_GPA MEM_AUX_GPA
+#define MEM_TEST_GPA (MEM_AUX_GPA + 4096)
+#define MEM_TEST_SIZE (MEM_SIZE - 4096)
+static_assert(MEM_SIZE % 4096 == 0, "invalid mem size");
+static_assert(MEM_TEST_SIZE % 4096 == 0, "invalid mem test size");
+
+/*
+ * 32 MiB is max size that gets well over 100 iterations on 509 slots.
+ * Considering that each slot needs to have at least one page up to
+ * 8194 slots in use can then be tested (although with slightly
+ * limited resolution).
+ */
+#define MEM_SIZE_MAP ((32U << 20) + 4096)
+#define MEM_SIZE_MAP_PAGES (MEM_SIZE_MAP / 4096)
+#define MEM_TEST_MAP_SIZE (MEM_SIZE_MAP - 4096)
+#define MEM_TEST_MAP_SIZE_PAGES (MEM_TEST_MAP_SIZE / 4096)
+static_assert(MEM_SIZE_MAP % 4096 == 0, "invalid map test region size");
+static_assert(MEM_TEST_MAP_SIZE % 4096 == 0, "invalid map test region size");
+static_assert(MEM_TEST_MAP_SIZE_PAGES % 2 == 0, "invalid map test region size");
+static_assert(MEM_TEST_MAP_SIZE_PAGES > 2, "invalid map test region size");
+
+/*
+ * 128 MiB is min size that fills 32k slots with at least one page in each
+ * while at the same time gets 100+ iterations in such test
+ */
+#define MEM_TEST_UNMAP_SIZE (128U << 20)
+#define MEM_TEST_UNMAP_SIZE_PAGES (MEM_TEST_UNMAP_SIZE / 4096)
+/* 2 MiB chunk size like a typical huge page */
+#define MEM_TEST_UNMAP_CHUNK_PAGES (2U << (20 - 12))
+static_assert(MEM_TEST_UNMAP_SIZE <= MEM_TEST_SIZE,
+ "invalid unmap test region size");
+static_assert(MEM_TEST_UNMAP_SIZE % 4096 == 0,
+ "invalid unmap test region size");
+static_assert(MEM_TEST_UNMAP_SIZE_PAGES %
+ (2 * MEM_TEST_UNMAP_CHUNK_PAGES) == 0,
+ "invalid unmap test region size");
+
+/*
+ * For the move active test the middle of the test area is placed on
+ * a memslot boundary: half lies in the memslot being moved, half in
+ * other memslot(s).
+ *
+ * When running this test with 32k memslots (32764, really) each memslot
+ * contains 4 pages.
+ * The last one additionally contains the remaining 21 pages of memory,
+ * for the total size of 25 pages.
+ * Hence, the maximum size here is 50 pages.
+ */
+#define MEM_TEST_MOVE_SIZE_PAGES (50)
+#define MEM_TEST_MOVE_SIZE (MEM_TEST_MOVE_SIZE_PAGES * 4096)
+#define MEM_TEST_MOVE_GPA_DEST (MEM_GPA + MEM_SIZE)
+static_assert(MEM_TEST_MOVE_SIZE <= MEM_TEST_SIZE,
+ "invalid move test region size");
+
+#define MEM_TEST_VAL_1 0x1122334455667788
+#define MEM_TEST_VAL_2 0x99AABBCCDDEEFF00
+
+struct vm_data {
+ struct kvm_vm *vm;
+ pthread_t vcpu_thread;
+ uint32_t nslots;
+ uint64_t npages;
+ uint64_t pages_per_slot;
+ void **hva_slots;
+ bool mmio_ok;
+ uint64_t mmio_gpa_min;
+ uint64_t mmio_gpa_max;
+};
+
+struct sync_area {
+ atomic_bool start_flag;
+ atomic_bool exit_flag;
+ atomic_bool sync_flag;
+ void *move_area_ptr;
+};
+
+/*
+ * Technically, we need also for the atomic bool to be address-free, which
+ * is recommended, but not strictly required, by C11 for lockless
+ * implementations.
+ * However, in practice both GCC and Clang fulfill this requirement on
+ * all KVM-supported platforms.
+ */
+static_assert(ATOMIC_BOOL_LOCK_FREE == 2, "atomic bool is not lockless");
+
+static sem_t vcpu_ready;
+
+static bool map_unmap_verify;
+
+static bool verbose;
+#define pr_info_v(...) \
+ do { \
+ if (verbose) \
+ pr_info(__VA_ARGS__); \
+ } while (0)
+
+static void *vcpu_worker(void *data)
+{
+ struct vm_data *vm = data;
+ struct kvm_run *run;
+ struct ucall uc;
+ uint64_t cmd;
+
+ run = vcpu_state(vm->vm, VCPU_ID);
+ while (1) {
+ vcpu_run(vm->vm, VCPU_ID);
+
+ if (run->exit_reason == KVM_EXIT_IO) {
+ cmd = get_ucall(vm->vm, VCPU_ID, &uc);
+ if (cmd != UCALL_SYNC)
+ break;
+
+ sem_post(&vcpu_ready);
+ continue;
+ }
+
+ if (run->exit_reason != KVM_EXIT_MMIO)
+ break;
+
+ TEST_ASSERT(vm->mmio_ok, "Unexpected mmio exit");
+ TEST_ASSERT(run->mmio.is_write, "Unexpected mmio read");
+ TEST_ASSERT(run->mmio.len == 8,
+ "Unexpected exit mmio size = %u", run->mmio.len);
+ TEST_ASSERT(run->mmio.phys_addr >= vm->mmio_gpa_min &&
+ run->mmio.phys_addr <= vm->mmio_gpa_max,
+ "Unexpected exit mmio address = 0x%llx",
+ run->mmio.phys_addr);
+ }
+
+ if (run->exit_reason == KVM_EXIT_IO && cmd == UCALL_ABORT)
+ TEST_FAIL("%s at %s:%ld, val = %lu", (const char *)uc.args[0],
+ __FILE__, uc.args[1], uc.args[2]);
+
+ return NULL;
+}
+
+static void wait_for_vcpu(void)
+{
+ struct timespec ts;
+
+ TEST_ASSERT(!clock_gettime(CLOCK_REALTIME, &ts),
+ "clock_gettime() failed: %d\n", errno);
+
+ ts.tv_sec += 2;
+ TEST_ASSERT(!sem_timedwait(&vcpu_ready, &ts),
+ "sem_timedwait() failed: %d\n", errno);
+}
+
+static void *vm_gpa2hva(struct vm_data *data, uint64_t gpa, uint64_t *rempages)
+{
+ uint64_t gpage, pgoffs;
+ uint32_t slot, slotoffs;
+ void *base;
+
+ TEST_ASSERT(gpa >= MEM_GPA, "Too low gpa to translate");
+ TEST_ASSERT(gpa < MEM_GPA + data->npages * 4096,
+ "Too high gpa to translate");
+ gpa -= MEM_GPA;
+
+ gpage = gpa / 4096;
+ pgoffs = gpa % 4096;
+ slot = min(gpage / data->pages_per_slot, (uint64_t)data->nslots - 1);
+ slotoffs = gpage - (slot * data->pages_per_slot);
+
+ if (rempages) {
+ uint64_t slotpages;
+
+ if (slot == data->nslots - 1)
+ slotpages = data->npages - slot * data->pages_per_slot;
+ else
+ slotpages = data->pages_per_slot;
+
+ TEST_ASSERT(!pgoffs,
+ "Asking for remaining pages in slot but gpa not page aligned");
+ *rempages = slotpages - slotoffs;
+ }
+
+ base = data->hva_slots[slot];
+ return (uint8_t *)base + slotoffs * 4096 + pgoffs;
+}
+
+static uint64_t vm_slot2gpa(struct vm_data *data, uint32_t slot)
+{
+ TEST_ASSERT(slot < data->nslots, "Too high slot number");
+
+ return MEM_GPA + slot * data->pages_per_slot * 4096;
+}
+
+static struct vm_data *alloc_vm(void)
+{
+ struct vm_data *data;
+
+ data = malloc(sizeof(*data));
+ TEST_ASSERT(data, "malloc(vmdata) failed");
+
+ data->vm = NULL;
+ data->hva_slots = NULL;
+
+ return data;
+}
+
+static bool prepare_vm(struct vm_data *data, int nslots, uint64_t *maxslots,
+ void *guest_code, uint64_t mempages,
+ struct timespec *slot_runtime)
+{
+ uint32_t max_mem_slots;
+ uint64_t rempages;
+ uint64_t guest_addr;
+ uint32_t slot;
+ struct timespec tstart;
+ struct sync_area *sync;
+
+ max_mem_slots = kvm_check_cap(KVM_CAP_NR_MEMSLOTS);
+ TEST_ASSERT(max_mem_slots > 1,
+ "KVM_CAP_NR_MEMSLOTS should be greater than 1");
+ TEST_ASSERT(nslots > 1 || nslots == -1,
+ "Slot count cap should be greater than 1");
+ if (nslots != -1)
+ max_mem_slots = min(max_mem_slots, (uint32_t)nslots);
+ pr_info_v("Allowed number of memory slots: %"PRIu32"\n", max_mem_slots);
+
+ TEST_ASSERT(mempages > 1,
+ "Can't test without any memory");
+
+ data->npages = mempages;
+ data->nslots = max_mem_slots - 1;
+ data->pages_per_slot = mempages / data->nslots;
+ if (!data->pages_per_slot) {
+ *maxslots = mempages + 1;
+ return false;
+ }
+
+ rempages = mempages % data->nslots;
+ data->hva_slots = malloc(sizeof(*data->hva_slots) * data->nslots);
+ TEST_ASSERT(data->hva_slots, "malloc() fail");
+
+ data->vm = vm_create_default(VCPU_ID, mempages, guest_code);
+
+ pr_info_v("Adding slots 1..%i, each slot with %"PRIu64" pages + %"PRIu64" extra pages last\n",
+ max_mem_slots - 1, data->pages_per_slot, rempages);
+
+ clock_gettime(CLOCK_MONOTONIC, &tstart);
+ for (slot = 1, guest_addr = MEM_GPA; slot < max_mem_slots; slot++) {
+ uint64_t npages;
+
+ npages = data->pages_per_slot;
+ if (slot == max_mem_slots - 1)
+ npages += rempages;
+
+ vm_userspace_mem_region_add(data->vm, VM_MEM_SRC_ANONYMOUS,
+ guest_addr, slot, npages,
+ 0);
+ guest_addr += npages * 4096;
+ }
+ *slot_runtime = timespec_elapsed(tstart);
+
+ for (slot = 0, guest_addr = MEM_GPA; slot < max_mem_slots - 1; slot++) {
+ uint64_t npages;
+ uint64_t gpa;
+
+ npages = data->pages_per_slot;
+ if (slot == max_mem_slots - 2)
+ npages += rempages;
+
+ gpa = vm_phy_pages_alloc(data->vm, npages, guest_addr,
+ slot + 1);
+ TEST_ASSERT(gpa == guest_addr,
+ "vm_phy_pages_alloc() failed\n");
+
+ data->hva_slots[slot] = addr_gpa2hva(data->vm, guest_addr);
+ memset(data->hva_slots[slot], 0, npages * 4096);
+
+ guest_addr += npages * 4096;
+ }
+
+ virt_map(data->vm, MEM_GPA, MEM_GPA, mempages, 0);
+
+ sync = (typeof(sync))vm_gpa2hva(data, MEM_SYNC_GPA, NULL);
+ atomic_init(&sync->start_flag, false);
+ atomic_init(&sync->exit_flag, false);
+ atomic_init(&sync->sync_flag, false);
+
+ data->mmio_ok = false;
+
+ return true;
+}
+
+static void launch_vm(struct vm_data *data)
+{
+ pr_info_v("Launching the test VM\n");
+
+ pthread_create(&data->vcpu_thread, NULL, vcpu_worker, data);
+
+ /* Ensure the guest thread is spun up. */
+ wait_for_vcpu();
+}
+
+static void free_vm(struct vm_data *data)
+{
+ kvm_vm_free(data->vm);
+ free(data->hva_slots);
+ free(data);
+}
+
+static void wait_guest_exit(struct vm_data *data)
+{
+ pthread_join(data->vcpu_thread, NULL);
+}
+
+static void let_guest_run(struct sync_area *sync)
+{
+ atomic_store_explicit(&sync->start_flag, true, memory_order_release);
+}
+
+static void guest_spin_until_start(void)
+{
+ struct sync_area *sync = (typeof(sync))MEM_SYNC_GPA;
+
+ while (!atomic_load_explicit(&sync->start_flag, memory_order_acquire))
+ ;
+}
+
+static void make_guest_exit(struct sync_area *sync)
+{
+ atomic_store_explicit(&sync->exit_flag, true, memory_order_release);
+}
+
+static bool _guest_should_exit(void)
+{
+ struct sync_area *sync = (typeof(sync))MEM_SYNC_GPA;
+
+ return atomic_load_explicit(&sync->exit_flag, memory_order_acquire);
+}
+
+#define guest_should_exit() unlikely(_guest_should_exit())
+
+/*
+ * noinline so we can easily see how much time the host spends waiting
+ * for the guest.
+ * For the same reason use alarm() instead of polling clock_gettime()
+ * to implement a wait timeout.
+ */
+static noinline void host_perform_sync(struct sync_area *sync)
+{
+ alarm(2);
+
+ atomic_store_explicit(&sync->sync_flag, true, memory_order_release);
+ while (atomic_load_explicit(&sync->sync_flag, memory_order_acquire))
+ ;
+
+ alarm(0);
+}
+
+static bool guest_perform_sync(void)
+{
+ struct sync_area *sync = (typeof(sync))MEM_SYNC_GPA;
+ bool expected;
+
+ do {
+ if (guest_should_exit())
+ return false;
+
+ expected = true;
+ } while (!atomic_compare_exchange_weak_explicit(&sync->sync_flag,
+ &expected, false,
+ memory_order_acq_rel,
+ memory_order_relaxed));
+
+ return true;
+}
+
+static void guest_code_test_memslot_move(void)
+{
+ struct sync_area *sync = (typeof(sync))MEM_SYNC_GPA;
+ uintptr_t base = (typeof(base))READ_ONCE(sync->move_area_ptr);
+
+ GUEST_SYNC(0);
+
+ guest_spin_until_start();
+
+ while (!guest_should_exit()) {
+ uintptr_t ptr;
+
+ for (ptr = base; ptr < base + MEM_TEST_MOVE_SIZE;
+ ptr += 4096)
+ *(uint64_t *)ptr = MEM_TEST_VAL_1;
+
+ /*
+ * No host sync here since the MMIO exits are so expensive
+ * that the host would spend most of its time waiting for
+ * the guest and so instead of measuring memslot move
+ * performance we would measure the performance and
+ * likelihood of MMIO exits
+ */
+ }
+
+ GUEST_DONE();
+}
+
+static void guest_code_test_memslot_map(void)
+{
+ struct sync_area *sync = (typeof(sync))MEM_SYNC_GPA;
+
+ GUEST_SYNC(0);
+
+ guest_spin_until_start();
+
+ while (1) {
+ uintptr_t ptr;
+
+ for (ptr = MEM_TEST_GPA;
+ ptr < MEM_TEST_GPA + MEM_TEST_MAP_SIZE / 2; ptr += 4096)
+ *(uint64_t *)ptr = MEM_TEST_VAL_1;
+
+ if (!guest_perform_sync())
+ break;
+
+ for (ptr = MEM_TEST_GPA + MEM_TEST_MAP_SIZE / 2;
+ ptr < MEM_TEST_GPA + MEM_TEST_MAP_SIZE; ptr += 4096)
+ *(uint64_t *)ptr = MEM_TEST_VAL_2;
+
+ if (!guest_perform_sync())
+ break;
+ }
+
+ GUEST_DONE();
+}
+
+static void guest_code_test_memslot_unmap(void)
+{
+ struct sync_area *sync = (typeof(sync))MEM_SYNC_GPA;
+
+ GUEST_SYNC(0);
+
+ guest_spin_until_start();
+
+ while (1) {
+ uintptr_t ptr = MEM_TEST_GPA;
+
+ /*
+ * We can afford to access (map) just a small number of pages
+ * per host sync as otherwise the host will spend
+ * a significant amount of its time waiting for the guest
+ * (instead of doing unmap operations), so this will
+ * effectively turn this test into a map performance test.
+ *
+ * Just access a single page to be on the safe side.
+ */
+ *(uint64_t *)ptr = MEM_TEST_VAL_1;
+
+ if (!guest_perform_sync())
+ break;
+
+ ptr += MEM_TEST_UNMAP_SIZE / 2;
+ *(uint64_t *)ptr = MEM_TEST_VAL_2;
+
+ if (!guest_perform_sync())
+ break;
+ }
+
+ GUEST_DONE();
+}
+
+static void guest_code_test_memslot_rw(void)
+{
+ GUEST_SYNC(0);
+
+ guest_spin_until_start();
+
+ while (1) {
+ uintptr_t ptr;
+
+ for (ptr = MEM_TEST_GPA;
+ ptr < MEM_TEST_GPA + MEM_TEST_SIZE; ptr += 4096)
+ *(uint64_t *)ptr = MEM_TEST_VAL_1;
+
+ if (!guest_perform_sync())
+ break;
+
+ for (ptr = MEM_TEST_GPA + 4096 / 2;
+ ptr < MEM_TEST_GPA + MEM_TEST_SIZE; ptr += 4096) {
+ uint64_t val = *(uint64_t *)ptr;
+
+ GUEST_ASSERT_1(val == MEM_TEST_VAL_2, val);
+ *(uint64_t *)ptr = 0;
+ }
+
+ if (!guest_perform_sync())
+ break;
+ }
+
+ GUEST_DONE();
+}
+
+static bool test_memslot_move_prepare(struct vm_data *data,
+ struct sync_area *sync,
+ uint64_t *maxslots, bool isactive)
+{
+ uint64_t movesrcgpa, movetestgpa;
+
+ movesrcgpa = vm_slot2gpa(data, data->nslots - 1);
+
+ if (isactive) {
+ uint64_t lastpages;
+
+ vm_gpa2hva(data, movesrcgpa, &lastpages);
+ if (lastpages < MEM_TEST_MOVE_SIZE_PAGES / 2) {
+ *maxslots = 0;
+ return false;
+ }
+ }
+
+ movetestgpa = movesrcgpa - (MEM_TEST_MOVE_SIZE / (isactive ? 2 : 1));
+ sync->move_area_ptr = (void *)movetestgpa;
+
+ if (isactive) {
+ data->mmio_ok = true;
+ data->mmio_gpa_min = movesrcgpa;
+ data->mmio_gpa_max = movesrcgpa + MEM_TEST_MOVE_SIZE / 2 - 1;
+ }
+
+ return true;
+}
+
+static bool test_memslot_move_prepare_active(struct vm_data *data,
+ struct sync_area *sync,
+ uint64_t *maxslots)
+{
+ return test_memslot_move_prepare(data, sync, maxslots, true);
+}
+
+static bool test_memslot_move_prepare_inactive(struct vm_data *data,
+ struct sync_area *sync,
+ uint64_t *maxslots)
+{
+ return test_memslot_move_prepare(data, sync, maxslots, false);
+}
+
+static void test_memslot_move_loop(struct vm_data *data, struct sync_area *sync)
+{
+ uint64_t movesrcgpa;
+
+ movesrcgpa = vm_slot2gpa(data, data->nslots - 1);
+ vm_mem_region_move(data->vm, data->nslots - 1 + 1,
+ MEM_TEST_MOVE_GPA_DEST);
+ vm_mem_region_move(data->vm, data->nslots - 1 + 1, movesrcgpa);
+}
+
+static void test_memslot_do_unmap(struct vm_data *data,
+ uint64_t offsp, uint64_t count)
+{
+ uint64_t gpa, ctr;
+
+ for (gpa = MEM_TEST_GPA + offsp * 4096, ctr = 0; ctr < count; ) {
+ uint64_t npages;
+ void *hva;
+ int ret;
+
+ hva = vm_gpa2hva(data, gpa, &npages);
+ TEST_ASSERT(npages, "Empty memory slot at gptr 0x%"PRIx64, gpa);
+ npages = min(npages, count - ctr);
+ ret = madvise(hva, npages * 4096, MADV_DONTNEED);
+ TEST_ASSERT(!ret,
+ "madvise(%p, MADV_DONTNEED) on VM memory should not fail for gptr 0x%"PRIx64,
+ hva, gpa);
+ ctr += npages;
+ gpa += npages * 4096;
+ }
+ TEST_ASSERT(ctr == count,
+ "madvise(MADV_DONTNEED) should exactly cover all of the requested area");
+}
+
+static void test_memslot_map_unmap_check(struct vm_data *data,
+ uint64_t offsp, uint64_t valexp)
+{
+ uint64_t gpa;
+ uint64_t *val;
+
+ if (!map_unmap_verify)
+ return;
+
+ gpa = MEM_TEST_GPA + offsp * 4096;
+ val = (typeof(val))vm_gpa2hva(data, gpa, NULL);
+ TEST_ASSERT(*val == valexp,
+ "Guest written values should read back correctly before unmap (%"PRIu64" vs %"PRIu64" @ %"PRIx64")",
+ *val, valexp, gpa);
+ *val = 0;
+}
+
+static void test_memslot_map_loop(struct vm_data *data, struct sync_area *sync)
+{
+ /*
+ * Unmap the second half of the test area while guest writes to (maps)
+ * the first half.
+ */
+ test_memslot_do_unmap(data, MEM_TEST_MAP_SIZE_PAGES / 2,
+ MEM_TEST_MAP_SIZE_PAGES / 2);
+
+ /*
+ * Wait for the guest to finish writing the first half of the test
+ * area, verify the written value on the first and the last page of
+ * this area and then unmap it.
+ * Meanwhile, the guest is writing to (mapping) the second half of
+ * the test area.
+ */
+ host_perform_sync(sync);
+ test_memslot_map_unmap_check(data, 0, MEM_TEST_VAL_1);
+ test_memslot_map_unmap_check(data,
+ MEM_TEST_MAP_SIZE_PAGES / 2 - 1,
+ MEM_TEST_VAL_1);
+ test_memslot_do_unmap(data, 0, MEM_TEST_MAP_SIZE_PAGES / 2);
+
+
+ /*
+ * Wait for the guest to finish writing the second half of the test
+ * area and verify the written value on the first and the last page
+ * of this area.
+ * The area will be unmapped at the beginning of the next loop
+ * iteration.
+ * Meanwhile, the guest is writing to (mapping) the first half of
+ * the test area.
+ */
+ host_perform_sync(sync);
+ test_memslot_map_unmap_check(data, MEM_TEST_MAP_SIZE_PAGES / 2,
+ MEM_TEST_VAL_2);
+ test_memslot_map_unmap_check(data, MEM_TEST_MAP_SIZE_PAGES - 1,
+ MEM_TEST_VAL_2);
+}
+
+static void test_memslot_unmap_loop_common(struct vm_data *data,
+ struct sync_area *sync,
+ uint64_t chunk)
+{
+ uint64_t ctr;
+
+ /*
+ * Wait for the guest to finish mapping page(s) in the first half
+ * of the test area, verify the written value and then perform unmap
+ * of this area.
+ * Meanwhile, the guest is writing to (mapping) page(s) in the second
+ * half of the test area.
+ */
+ host_perform_sync(sync);
+ test_memslot_map_unmap_check(data, 0, MEM_TEST_VAL_1);
+ for (ctr = 0; ctr < MEM_TEST_UNMAP_SIZE_PAGES / 2; ctr += chunk)
+ test_memslot_do_unmap(data, ctr, chunk);
+
+ /* Likewise, but for the opposite host / guest areas */
+ host_perform_sync(sync);
+ test_memslot_map_unmap_check(data, MEM_TEST_UNMAP_SIZE_PAGES / 2,
+ MEM_TEST_VAL_2);
+ for (ctr = MEM_TEST_UNMAP_SIZE_PAGES / 2;
+ ctr < MEM_TEST_UNMAP_SIZE_PAGES; ctr += chunk)
+ test_memslot_do_unmap(data, ctr, chunk);
+}
+
+static void test_memslot_unmap_loop(struct vm_data *data,
+ struct sync_area *sync)
+{
+ test_memslot_unmap_loop_common(data, sync, 1);
+}
+
+static void test_memslot_unmap_loop_chunked(struct vm_data *data,
+ struct sync_area *sync)
+{
+ test_memslot_unmap_loop_common(data, sync, MEM_TEST_UNMAP_CHUNK_PAGES);
+}
+
+static void test_memslot_rw_loop(struct vm_data *data, struct sync_area *sync)
+{
+ uint64_t gptr;
+
+ for (gptr = MEM_TEST_GPA + 4096 / 2;
+ gptr < MEM_TEST_GPA + MEM_TEST_SIZE; gptr += 4096)
+ *(uint64_t *)vm_gpa2hva(data, gptr, NULL) = MEM_TEST_VAL_2;
+
+ host_perform_sync(sync);
+
+ for (gptr = MEM_TEST_GPA;
+ gptr < MEM_TEST_GPA + MEM_TEST_SIZE; gptr += 4096) {
+ uint64_t *vptr = (typeof(vptr))vm_gpa2hva(data, gptr, NULL);
+ uint64_t val = *vptr;
+
+ TEST_ASSERT(val == MEM_TEST_VAL_1,
+ "Guest written values should read back correctly (is %"PRIu64" @ %"PRIx64")",
+ val, gptr);
+ *vptr = 0;
+ }
+
+ host_perform_sync(sync);
+}
+
+struct test_data {
+ const char *name;
+ uint64_t mem_size;
+ void (*guest_code)(void);
+ bool (*prepare)(struct vm_data *data, struct sync_area *sync,
+ uint64_t *maxslots);
+ void (*loop)(struct vm_data *data, struct sync_area *sync);
+};
+
+static bool test_execute(int nslots, uint64_t *maxslots,
+ unsigned int maxtime,
+ const struct test_data *tdata,
+ uint64_t *nloops,
+ struct timespec *slot_runtime,
+ struct timespec *guest_runtime)
+{
+ uint64_t mem_size = tdata->mem_size ? : MEM_SIZE_PAGES;
+ struct vm_data *data;
+ struct sync_area *sync;
+ struct timespec tstart;
+ bool ret = true;
+
+ data = alloc_vm();
+ if (!prepare_vm(data, nslots, maxslots, tdata->guest_code,
+ mem_size, slot_runtime)) {
+ ret = false;
+ goto exit_free;
+ }
+
+ sync = (typeof(sync))vm_gpa2hva(data, MEM_SYNC_GPA, NULL);
+
+ if (tdata->prepare &&
+ !tdata->prepare(data, sync, maxslots)) {
+ ret = false;
+ goto exit_free;
+ }
+
+ launch_vm(data);
+
+ clock_gettime(CLOCK_MONOTONIC, &tstart);
+ let_guest_run(sync);
+
+ while (1) {
+ *guest_runtime = timespec_elapsed(tstart);
+ if (guest_runtime->tv_sec >= maxtime)
+ break;
+
+ tdata->loop(data, sync);
+
+ (*nloops)++;
+ }
+
+ make_guest_exit(sync);
+ wait_guest_exit(data);
+
+exit_free:
+ free_vm(data);
+
+ return ret;
+}
+
+static const struct test_data tests[] = {
+ {
+ .name = "map",
+ .mem_size = MEM_SIZE_MAP_PAGES,
+ .guest_code = guest_code_test_memslot_map,
+ .loop = test_memslot_map_loop,
+ },
+ {
+ .name = "unmap",
+ .mem_size = MEM_TEST_UNMAP_SIZE_PAGES + 1,
+ .guest_code = guest_code_test_memslot_unmap,
+ .loop = test_memslot_unmap_loop,
+ },
+ {
+ .name = "unmap chunked",
+ .mem_size = MEM_TEST_UNMAP_SIZE_PAGES + 1,
+ .guest_code = guest_code_test_memslot_unmap,
+ .loop = test_memslot_unmap_loop_chunked,
+ },
+ {
+ .name = "move active area",
+ .guest_code = guest_code_test_memslot_move,
+ .prepare = test_memslot_move_prepare_active,
+ .loop = test_memslot_move_loop,
+ },
+ {
+ .name = "move inactive area",
+ .guest_code = guest_code_test_memslot_move,
+ .prepare = test_memslot_move_prepare_inactive,
+ .loop = test_memslot_move_loop,
+ },
+ {
+ .name = "RW",
+ .guest_code = guest_code_test_memslot_rw,
+ .loop = test_memslot_rw_loop
+ },
+};
+
+#define NTESTS ARRAY_SIZE(tests)
+
+struct test_args {
+ int tfirst;
+ int tlast;
+ int nslots;
+ int seconds;
+ int runs;
+};
+
+static void help(char *name, struct test_args *targs)
+{
+ int ctr;
+
+ pr_info("usage: %s [-h] [-v] [-d] [-s slots] [-f first_test] [-e last_test] [-l test_length] [-r run_count]\n",
+ name);
+ pr_info(" -h: print this help screen.\n");
+ pr_info(" -v: enable verbose mode (not for benchmarking).\n");
+ pr_info(" -d: enable extra debug checks.\n");
+ pr_info(" -s: specify memslot count cap (-1 means no cap; currently: %i)\n",
+ targs->nslots);
+ pr_info(" -f: specify the first test to run (currently: %i; max %zu)\n",
+ targs->tfirst, NTESTS - 1);
+ pr_info(" -e: specify the last test to run (currently: %i; max %zu)\n",
+ targs->tlast, NTESTS - 1);
+ pr_info(" -l: specify the test length in seconds (currently: %i)\n",
+ targs->seconds);
+ pr_info(" -r: specify the number of runs per test (currently: %i)\n",
+ targs->runs);
+
+ pr_info("\nAvailable tests:\n");
+ for (ctr = 0; ctr < NTESTS; ctr++)
+ pr_info("%d: %s\n", ctr, tests[ctr].name);
+}
+
+static bool parse_args(int argc, char *argv[],
+ struct test_args *targs)
+{
+ int opt;
+
+ while ((opt = getopt(argc, argv, "hvds:f:e:l:r:")) != -1) {
+ switch (opt) {
+ case 'h':
+ default:
+ help(argv[0], targs);
+ return false;
+ case 'v':
+ verbose = true;
+ break;
+ case 'd':
+ map_unmap_verify = true;
+ break;
+ case 's':
+ targs->nslots = atoi(optarg);
+ if (targs->nslots <= 0 && targs->nslots != -1) {
+ pr_info("Slot count cap has to be positive or -1 for no cap\n");
+ return false;
+ }
+ break;
+ case 'f':
+ targs->tfirst = atoi(optarg);
+ if (targs->tfirst < 0) {
+ pr_info("First test to run has to be non-negative\n");
+ return false;
+ }
+ break;
+ case 'e':
+ targs->tlast = atoi(optarg);
+ if (targs->tlast < 0 || targs->tlast >= NTESTS) {
+ pr_info("Last test to run has to be non-negative and less than %zu\n",
+ NTESTS);
+ return false;
+ }
+ break;
+ case 'l':
+ targs->seconds = atoi(optarg);
+ if (targs->seconds < 0) {
+ pr_info("Test length in seconds has to be non-negative\n");
+ return false;
+ }
+ break;
+ case 'r':
+ targs->runs = atoi(optarg);
+ if (targs->runs <= 0) {
+ pr_info("Runs per test has to be positive\n");
+ return false;
+ }
+ break;
+ }
+ }
+
+ if (optind < argc) {
+ help(argv[0], targs);
+ return false;
+ }
+
+ if (targs->tfirst > targs->tlast) {
+ pr_info("First test to run cannot be greater than the last test to run\n");
+ return false;
+ }
+
+ return true;
+}
+
+struct test_result {
+ struct timespec slot_runtime, guest_runtime, iter_runtime;
+ int64_t slottimens, runtimens;
+ uint64_t nloops;
+};
+
+static bool test_loop(const struct test_data *data,
+ const struct test_args *targs,
+ struct test_result *rbestslottime,
+ struct test_result *rbestruntime)
+{
+ uint64_t maxslots;
+ struct test_result result;
+
+ result.nloops = 0;
+ if (!test_execute(targs->nslots, &maxslots, targs->seconds, data,
+ &result.nloops,
+ &result.slot_runtime, &result.guest_runtime)) {
+ if (maxslots)
+ pr_info("Memslot count too high for this test, decrease the cap (max is %"PRIu64")\n",
+ maxslots);
+ else
+ pr_info("Memslot count may be too high for this test, try adjusting the cap\n");
+
+ return false;
+ }
+
+ pr_info("Test took %ld.%.9lds for slot setup + %ld.%.9lds all iterations\n",
+ result.slot_runtime.tv_sec, result.slot_runtime.tv_nsec,
+ result.guest_runtime.tv_sec, result.guest_runtime.tv_nsec);
+ if (!result.nloops) {
+ pr_info("No full loops done - too short test time or system too loaded?\n");
+ return true;
+ }
+
+ result.iter_runtime = timespec_div(result.guest_runtime,
+ result.nloops);
+ pr_info("Done %"PRIu64" iterations, avg %ld.%.9lds each\n",
+ result.nloops,
+ result.iter_runtime.tv_sec,
+ result.iter_runtime.tv_nsec);
+ result.slottimens = timespec_to_ns(result.slot_runtime);
+ result.runtimens = timespec_to_ns(result.iter_runtime);
+
+ /*
+ * Only rank the slot setup time for tests using the whole test memory
+ * area so they are comparable
+ */
+ if (!data->mem_size &&
+ (!rbestslottime->slottimens ||
+ result.slottimens < rbestslottime->slottimens))
+ *rbestslottime = result;
+ if (!rbestruntime->runtimens ||
+ result.runtimens < rbestruntime->runtimens)
+ *rbestruntime = result;
+
+ return true;
+}
+
+int main(int argc, char *argv[])
+{
+ struct test_args targs = {
+ .tfirst = 0,
+ .tlast = NTESTS - 1,
+ .nslots = -1,
+ .seconds = 5,
+ .runs = 1,
+ };
+ struct test_result rbestslottime;
+ int tctr;
+
+ /* Tell stdout not to buffer its content */
+ setbuf(stdout, NULL);
+
+ if (!parse_args(argc, argv, &targs))
+ return -1;
+
+ rbestslottime.slottimens = 0;
+ for (tctr = targs.tfirst; tctr <= targs.tlast; tctr++) {
+ const struct test_data *data = &tests[tctr];
+ unsigned int runctr;
+ struct test_result rbestruntime;
+
+ if (tctr > targs.tfirst)
+ pr_info("\n");
+
+ pr_info("Testing %s performance with %i runs, %d seconds each\n",
+ data->name, targs.runs, targs.seconds);
+
+ rbestruntime.runtimens = 0;
+ for (runctr = 0; runctr < targs.runs; runctr++)
+ if (!test_loop(data, &targs,
+ &rbestslottime, &rbestruntime))
+ break;
+
+ if (rbestruntime.runtimens)
+ pr_info("Best runtime result was %ld.%.9lds per iteration (with %"PRIu64" iterations)\n",
+ rbestruntime.iter_runtime.tv_sec,
+ rbestruntime.iter_runtime.tv_nsec,
+ rbestruntime.nloops);
+ }
+
+ if (rbestslottime.slottimens)
+ pr_info("Best slot setup time for the whole test area was %ld.%.9lds\n",
+ rbestslottime.slot_runtime.tv_sec,
+ rbestslottime.slot_runtime.tv_nsec);
+
+ return 0;
+}
#include "vmx.h"
#define VCPU_ID 5
+#define NMI_VECTOR 2
+
+static int ud_count;
+
+void enable_x2apic(void)
+{
+ uint32_t spiv_reg = APIC_BASE_MSR + (APIC_SPIV >> 4);
+
+ wrmsr(MSR_IA32_APICBASE, rdmsr(MSR_IA32_APICBASE) |
+ MSR_IA32_APICBASE_ENABLE | MSR_IA32_APICBASE_EXTD);
+ wrmsr(spiv_reg, rdmsr(spiv_reg) | APIC_SPIV_APIC_ENABLED);
+}
+
+static void guest_ud_handler(struct ex_regs *regs)
+{
+ ud_count++;
+ regs->rip += 3; /* VMLAUNCH */
+}
+
+static void guest_nmi_handler(struct ex_regs *regs)
+{
+}
void l2_guest_code(void)
{
GUEST_SYNC(8);
+ /* Forced exit to L1 upon restore */
+ GUEST_SYNC(9);
+
/* Done, exit to L1 and never come back. */
vmcall();
}
-void l1_guest_code(struct vmx_pages *vmx_pages)
+void guest_code(struct vmx_pages *vmx_pages)
{
#define L2_GUEST_STACK_SIZE 64
unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE];
+ enable_x2apic();
+
+ GUEST_SYNC(1);
+ GUEST_SYNC(2);
+
enable_vp_assist(vmx_pages->vp_assist_gpa, vmx_pages->vp_assist);
GUEST_ASSERT(vmx_pages->vmcs_gpa);
current_evmcs->revision_id = EVMCS_VERSION;
GUEST_SYNC(6);
+ current_evmcs->pin_based_vm_exec_control |=
+ PIN_BASED_NMI_EXITING;
GUEST_ASSERT(!vmlaunch());
GUEST_ASSERT(vmptrstz() == vmx_pages->enlightened_vmcs_gpa);
- GUEST_SYNC(9);
+
+ /*
+ * NMI forces L2->L1 exit, resuming L2 and hope that EVMCS is
+ * up-to-date (RIP points where it should and not at the beginning
+ * of l2_guest_code(). GUEST_SYNC(9) checkes that.
+ */
GUEST_ASSERT(!vmresume());
- GUEST_ASSERT(vmreadz(VM_EXIT_REASON) == EXIT_REASON_VMCALL);
+
GUEST_SYNC(10);
+
+ GUEST_ASSERT(vmreadz(VM_EXIT_REASON) == EXIT_REASON_VMCALL);
+ GUEST_SYNC(11);
+
+ /* Try enlightened vmptrld with an incorrect GPA */
+ evmcs_vmptrld(0xdeadbeef, vmx_pages->enlightened_vmcs);
+ GUEST_ASSERT(vmlaunch());
+ GUEST_ASSERT(ud_count == 1);
+ GUEST_DONE();
}
-void guest_code(struct vmx_pages *vmx_pages)
+void inject_nmi(struct kvm_vm *vm)
{
- GUEST_SYNC(1);
- GUEST_SYNC(2);
+ struct kvm_vcpu_events events;
- if (vmx_pages)
- l1_guest_code(vmx_pages);
+ vcpu_events_get(vm, VCPU_ID, &events);
- GUEST_DONE();
+ events.nmi.pending = 1;
+ events.flags |= KVM_VCPUEVENT_VALID_NMI_PENDING;
- /* Try enlightened vmptrld with an incorrect GPA */
- evmcs_vmptrld(0xdeadbeef, vmx_pages->enlightened_vmcs);
- GUEST_ASSERT(vmlaunch());
+ vcpu_events_set(vm, VCPU_ID, &events);
}
int main(int argc, char *argv[])
vcpu_alloc_vmx(vm, &vmx_pages_gva);
vcpu_args_set(vm, VCPU_ID, 1, vmx_pages_gva);
+ vm_init_descriptor_tables(vm);
+ vcpu_init_descriptor_tables(vm, VCPU_ID);
+ vm_handle_exception(vm, UD_VECTOR, guest_ud_handler);
+ vm_handle_exception(vm, NMI_VECTOR, guest_nmi_handler);
+
+ pr_info("Running L1 which uses EVMCS to run L2\n");
+
for (stage = 1;; stage++) {
_vcpu_run(vm, VCPU_ID);
TEST_ASSERT(run->exit_reason == KVM_EXIT_IO,
case UCALL_SYNC:
break;
case UCALL_DONE:
- goto part1_done;
+ goto done;
default:
TEST_FAIL("Unknown ucall %lu", uc.cmd);
}
TEST_ASSERT(!memcmp(®s1, ®s2, sizeof(regs2)),
"Unexpected register values after vcpu_load_state; rdi: %lx rsi: %lx",
(ulong) regs2.rdi, (ulong) regs2.rsi);
- }
-part1_done:
- _vcpu_run(vm, VCPU_ID);
- TEST_ASSERT(run->exit_reason == KVM_EXIT_SHUTDOWN,
- "Unexpected successful VMEnter with invalid eVMCS pointer!");
+ /* Force immediate L2->L1 exit before resuming */
+ if (stage == 8) {
+ pr_info("Injecting NMI into L1 before L2 had a chance to run after restore\n");
+ inject_nmi(vm);
+ }
+ }
+done:
kvm_vm_free(vm);
}
u32 function;
u32 index;
} mangled_cpuids[] = {
+ /*
+ * These entries depend on the vCPU's XCR0 register and IA32_XSS MSR,
+ * which are not controlled for by this test.
+ */
{.function = 0xd, .index = 0},
+ {.function = 0xd, .index = 1},
};
static void test_guest_cpuids(struct kvm_cpuid2 *guest_cpuid)
int old_res, res, kvm_fd, r;
struct kvm_msr_list *list;
- kvm_fd = open(KVM_DEV_PATH, O_RDONLY);
- if (kvm_fd < 0)
- exit(KSFT_SKIP);
+ kvm_fd = open_kvm_dev_path_or_exit();
old_res = kvm_num_index_msrs(kvm_fd, 0);
TEST_ASSERT(old_res != 0, "Expecting nmsrs to be > 0");
int res, old_res, i, kvm_fd;
struct kvm_msr_list *feature_list;
- kvm_fd = open(KVM_DEV_PATH, O_RDONLY);
- if (kvm_fd < 0)
- exit(KSFT_SKIP);
+ kvm_fd = open_kvm_dev_path_or_exit();
old_res = kvm_num_feature_msrs(kvm_fd, 0);
TEST_ASSERT(old_res != 0, "Expecting nmsrs to be > 0");
rxtimestamp
timestamping
txtimestamp
+so_netns_cookie
TEST_GEN_FILES += psock_fanout psock_tpacket msg_zerocopy reuseport_addr_any
TEST_GEN_FILES += tcp_mmap tcp_inq psock_snd txring_overwrite
TEST_GEN_FILES += udpgso udpgso_bench_tx udpgso_bench_rx ip_defrag
-TEST_GEN_FILES += so_txtime ipv6_flowlabel ipv6_flowlabel_mgr
+TEST_GEN_FILES += so_txtime ipv6_flowlabel ipv6_flowlabel_mgr so_netns_cookie
TEST_GEN_FILES += tcp_fastopen_backup_key
TEST_GEN_FILES += fin_ack_lat
TEST_GEN_FILES += reuseaddr_ports_exhausted
CONFIG_USER_NS=y
+CONFIG_NET_NS=y
CONFIG_BPF_SYSCALL=y
CONFIG_TEST_BPF=m
CONFIG_NUMA=y
#
+# Kselftest framework requirement - SKIP code is 4
+KSFT_SKIP=4
Port = collections.namedtuple('Port', 'bus_info name')
assert stderr == ""
devs = json.loads(stdout)['dev']
- dev = list(devs.keys())[0]
+ if devs:
+ dev = list(devs.keys())[0]
+ else:
+ print("no devlink device was found, test skipped")
+ sys.exit(KSFT_SKIP)
cmd = "devlink dev show %s" % dev
stdout, stderr = run_command(cmd)
run_cmd "$IP nexthop add id 86 via 2001:db8:91::2 dev veth1"
run_cmd "$IP ro add 2001:db8:101::1/128 nhid 81"
+ # route can not use prefsrc with nexthops
+ run_cmd "$IP ro add 2001:db8:101::2/128 nhid 86 from 2001:db8:91::1"
+ log_test $? 2 "IPv6 route can not use src routing with external nexthop"
+
+ # check cleanup path on invalid metric
+ run_cmd "$IP ro add 2001:db8:101::2/128 nhid 86 congctl lock foo"
+ log_test $? 2 "IPv6 route with invalid metric"
+
# rpfilter and default route
$IP nexthop flush >/dev/null 2>&1
run_cmd "ip netns exec me ip6tables -t mangle -I PREROUTING 1 -m rpfilter --invert -j DROP"
run_cmd "$IP nexthop replace id 22 via 172.16.2.2 dev veth3"
log_test $? 2 "Nexthop replace with invalid scope for existing route"
+ # check cleanup path on invalid metric
+ run_cmd "$IP ro add 172.16.101.2/32 nhid 22 congctl lock foo"
+ log_test $? 2 "IPv4 route with invalid metric"
+
#
# add route with nexthop and check traffic
#
ipv4_rt_replace_mpath
}
+# checks that cached input route on VRF port is deleted
+# when VRF is deleted
+ipv4_local_rt_cache()
+{
+ run_cmd "ip addr add 10.0.0.1/32 dev lo"
+ run_cmd "ip netns add test-ns"
+ run_cmd "ip link add veth-outside type veth peer name veth-inside"
+ run_cmd "ip link add vrf-100 type vrf table 1100"
+ run_cmd "ip link set veth-outside master vrf-100"
+ run_cmd "ip link set veth-inside netns test-ns"
+ run_cmd "ip link set veth-outside up"
+ run_cmd "ip link set vrf-100 up"
+ run_cmd "ip route add 10.1.1.1/32 dev veth-outside table 1100"
+ run_cmd "ip netns exec test-ns ip link set veth-inside up"
+ run_cmd "ip netns exec test-ns ip addr add 10.1.1.1/32 dev veth-inside"
+ run_cmd "ip netns exec test-ns ip route add 10.0.0.1/32 dev veth-inside"
+ run_cmd "ip netns exec test-ns ip route add default via 10.0.0.1"
+ run_cmd "ip netns exec test-ns ping 10.0.0.1 -c 1 -i 1"
+ run_cmd "ip link delete vrf-100"
+
+ # if we do not hang test is a success
+ log_test $? 0 "Cached route removed from VRF port device"
+}
+
ipv4_route_test()
{
route_setup
ipv4_rt_add
ipv4_rt_replace
+ ipv4_local_rt_cache
route_cleanup
}
--- /dev/null
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# Test traffic distribution between two paths when using custom hash policy.
+#
+# +--------------------------------+
+# | H1 |
+# | $h1 + |
+# | 198.51.100.{2-253}/24 | |
+# | 2001:db8:1::{2-fd}/64 | |
+# +-------------------------|------+
+# |
+# +-------------------------|-------------------------+
+# | SW1 | |
+# | $rp1 + |
+# | 198.51.100.1/24 |
+# | 2001:db8:1::1/64 |
+# | |
+# | |
+# | $rp11 + + $rp12 |
+# | 192.0.2.1/28 | | 192.0.2.17/28 |
+# | 2001:db8:2::1/64 | | 2001:db8:3::1/64 |
+# +------------------|-------------|------------------+
+# | |
+# +------------------|-------------|------------------+
+# | SW2 | | |
+# | | | |
+# | $rp21 + + $rp22 |
+# | 192.0.2.2/28 192.0.2.18/28 |
+# | 2001:db8:2::2/64 2001:db8:3::2/64 |
+# | |
+# | |
+# | $rp2 + |
+# | 203.0.113.1/24 | |
+# | 2001:db8:4::1/64 | |
+# +-------------------------|-------------------------+
+# |
+# +-------------------------|------+
+# | H2 | |
+# | $h2 + |
+# | 203.0.113.{2-253}/24 |
+# | 2001:db8:4::{2-fd}/64 |
+# +--------------------------------+
+
+ALL_TESTS="
+ ping_ipv4
+ ping_ipv6
+ custom_hash
+"
+
+NUM_NETIFS=8
+source lib.sh
+
+h1_create()
+{
+ simple_if_init $h1 198.51.100.2/24 2001:db8:1::2/64
+ ip route add vrf v$h1 default via 198.51.100.1 dev $h1
+ ip -6 route add vrf v$h1 default via 2001:db8:1::1 dev $h1
+}
+
+h1_destroy()
+{
+ ip -6 route del vrf v$h1 default
+ ip route del vrf v$h1 default
+ simple_if_fini $h1 198.51.100.2/24 2001:db8:1::2/64
+}
+
+sw1_create()
+{
+ simple_if_init $rp1 198.51.100.1/24 2001:db8:1::1/64
+ __simple_if_init $rp11 v$rp1 192.0.2.1/28 2001:db8:2::1/64
+ __simple_if_init $rp12 v$rp1 192.0.2.17/28 2001:db8:3::1/64
+
+ ip route add vrf v$rp1 203.0.113.0/24 \
+ nexthop via 192.0.2.2 dev $rp11 \
+ nexthop via 192.0.2.18 dev $rp12
+
+ ip -6 route add vrf v$rp1 2001:db8:4::/64 \
+ nexthop via 2001:db8:2::2 dev $rp11 \
+ nexthop via 2001:db8:3::2 dev $rp12
+}
+
+sw1_destroy()
+{
+ ip -6 route del vrf v$rp1 2001:db8:4::/64
+
+ ip route del vrf v$rp1 203.0.113.0/24
+
+ __simple_if_fini $rp12 192.0.2.17/28 2001:db8:3::1/64
+ __simple_if_fini $rp11 192.0.2.1/28 2001:db8:2::1/64
+ simple_if_fini $rp1 198.51.100.1/24 2001:db8:1::1/64
+}
+
+sw2_create()
+{
+ simple_if_init $rp2 203.0.113.1/24 2001:db8:4::1/64
+ __simple_if_init $rp21 v$rp2 192.0.2.2/28 2001:db8:2::2/64
+ __simple_if_init $rp22 v$rp2 192.0.2.18/28 2001:db8:3::2/64
+
+ ip route add vrf v$rp2 198.51.100.0/24 \
+ nexthop via 192.0.2.1 dev $rp21 \
+ nexthop via 192.0.2.17 dev $rp22
+
+ ip -6 route add vrf v$rp2 2001:db8:1::/64 \
+ nexthop via 2001:db8:2::1 dev $rp21 \
+ nexthop via 2001:db8:3::1 dev $rp22
+}
+
+sw2_destroy()
+{
+ ip -6 route del vrf v$rp2 2001:db8:1::/64
+
+ ip route del vrf v$rp2 198.51.100.0/24
+
+ __simple_if_fini $rp22 192.0.2.18/28 2001:db8:3::2/64
+ __simple_if_fini $rp21 192.0.2.2/28 2001:db8:2::2/64
+ simple_if_fini $rp2 203.0.113.1/24 2001:db8:4::1/64
+}
+
+h2_create()
+{
+ simple_if_init $h2 203.0.113.2/24 2001:db8:4::2/64
+ ip route add vrf v$h2 default via 203.0.113.1 dev $h2
+ ip -6 route add vrf v$h2 default via 2001:db8:4::1 dev $h2
+}
+
+h2_destroy()
+{
+ ip -6 route del vrf v$h2 default
+ ip route del vrf v$h2 default
+ simple_if_fini $h2 203.0.113.2/24 2001:db8:4::2/64
+}
+
+setup_prepare()
+{
+ h1=${NETIFS[p1]}
+
+ rp1=${NETIFS[p2]}
+
+ rp11=${NETIFS[p3]}
+ rp21=${NETIFS[p4]}
+
+ rp12=${NETIFS[p5]}
+ rp22=${NETIFS[p6]}
+
+ rp2=${NETIFS[p7]}
+
+ h2=${NETIFS[p8]}
+
+ vrf_prepare
+ h1_create
+ sw1_create
+ sw2_create
+ h2_create
+
+ forwarding_enable
+}
+
+cleanup()
+{
+ pre_cleanup
+
+ forwarding_restore
+
+ h2_destroy
+ sw2_destroy
+ sw1_destroy
+ h1_destroy
+ vrf_cleanup
+}
+
+ping_ipv4()
+{
+ ping_test $h1 203.0.113.2
+}
+
+ping_ipv6()
+{
+ ping6_test $h1 2001:db8:4::2
+}
+
+send_src_ipv4()
+{
+ $MZ $h1 -q -p 64 -A "198.51.100.2-198.51.100.253" -B 203.0.113.2 \
+ -d 1msec -c 50 -t udp "sp=20000,dp=30000"
+}
+
+send_dst_ipv4()
+{
+ $MZ $h1 -q -p 64 -A 198.51.100.2 -B "203.0.113.2-203.0.113.253" \
+ -d 1msec -c 50 -t udp "sp=20000,dp=30000"
+}
+
+send_src_udp4()
+{
+ $MZ $h1 -q -p 64 -A 198.51.100.2 -B 203.0.113.2 \
+ -d 1msec -t udp "sp=0-32768,dp=30000"
+}
+
+send_dst_udp4()
+{
+ $MZ $h1 -q -p 64 -A 198.51.100.2 -B 203.0.113.2 \
+ -d 1msec -t udp "sp=20000,dp=0-32768"
+}
+
+send_src_ipv6()
+{
+ $MZ -6 $h1 -q -p 64 -A "2001:db8:1::2-2001:db8:1::fd" -B 2001:db8:4::2 \
+ -d 1msec -c 50 -t udp "sp=20000,dp=30000"
+}
+
+send_dst_ipv6()
+{
+ $MZ -6 $h1 -q -p 64 -A 2001:db8:1::2 -B "2001:db8:4::2-2001:db8:4::fd" \
+ -d 1msec -c 50 -t udp "sp=20000,dp=30000"
+}
+
+send_flowlabel()
+{
+ # Generate 16384 echo requests, each with a random flow label.
+ for _ in $(seq 1 16384); do
+ ip vrf exec v$h1 \
+ $PING6 2001:db8:4::2 -F 0 -c 1 -q >/dev/null 2>&1
+ done
+}
+
+send_src_udp6()
+{
+ $MZ -6 $h1 -q -p 64 -A 2001:db8:1::2 -B 2001:db8:4::2 \
+ -d 1msec -t udp "sp=0-32768,dp=30000"
+}
+
+send_dst_udp6()
+{
+ $MZ -6 $h1 -q -p 64 -A 2001:db8:1::2 -B 2001:db8:4::2 \
+ -d 1msec -t udp "sp=20000,dp=0-32768"
+}
+
+custom_hash_test()
+{
+ local field="$1"; shift
+ local balanced="$1"; shift
+ local send_flows="$@"
+
+ RET=0
+
+ local t0_rp11=$(link_stats_tx_packets_get $rp11)
+ local t0_rp12=$(link_stats_tx_packets_get $rp12)
+
+ $send_flows
+
+ local t1_rp11=$(link_stats_tx_packets_get $rp11)
+ local t1_rp12=$(link_stats_tx_packets_get $rp12)
+
+ local d_rp11=$((t1_rp11 - t0_rp11))
+ local d_rp12=$((t1_rp12 - t0_rp12))
+
+ local diff=$((d_rp12 - d_rp11))
+ local sum=$((d_rp11 + d_rp12))
+
+ local pct=$(echo "$diff / $sum * 100" | bc -l)
+ local is_balanced=$(echo "-20 <= $pct && $pct <= 20" | bc)
+
+ [[ ( $is_balanced -eq 1 && $balanced == "balanced" ) ||
+ ( $is_balanced -eq 0 && $balanced == "unbalanced" ) ]]
+ check_err $? "Expected traffic to be $balanced, but it is not"
+
+ log_test "Multipath hash field: $field ($balanced)"
+ log_info "Packets sent on path1 / path2: $d_rp11 / $d_rp12"
+}
+
+custom_hash_v4()
+{
+ log_info "Running IPv4 custom multipath hash tests"
+
+ sysctl_set net.ipv4.fib_multipath_hash_policy 3
+
+ # Prevent the neighbour table from overflowing, as different neighbour
+ # entries will be created on $ol4 when using different destination IPs.
+ sysctl_set net.ipv4.neigh.default.gc_thresh1 1024
+ sysctl_set net.ipv4.neigh.default.gc_thresh2 1024
+ sysctl_set net.ipv4.neigh.default.gc_thresh3 1024
+
+ sysctl_set net.ipv4.fib_multipath_hash_fields 0x0001
+ custom_hash_test "Source IP" "balanced" send_src_ipv4
+ custom_hash_test "Source IP" "unbalanced" send_dst_ipv4
+
+ sysctl_set net.ipv4.fib_multipath_hash_fields 0x0002
+ custom_hash_test "Destination IP" "balanced" send_dst_ipv4
+ custom_hash_test "Destination IP" "unbalanced" send_src_ipv4
+
+ sysctl_set net.ipv4.fib_multipath_hash_fields 0x0010
+ custom_hash_test "Source port" "balanced" send_src_udp4
+ custom_hash_test "Source port" "unbalanced" send_dst_udp4
+
+ sysctl_set net.ipv4.fib_multipath_hash_fields 0x0020
+ custom_hash_test "Destination port" "balanced" send_dst_udp4
+ custom_hash_test "Destination port" "unbalanced" send_src_udp4
+
+ sysctl_restore net.ipv4.neigh.default.gc_thresh3
+ sysctl_restore net.ipv4.neigh.default.gc_thresh2
+ sysctl_restore net.ipv4.neigh.default.gc_thresh1
+
+ sysctl_restore net.ipv4.fib_multipath_hash_policy
+}
+
+custom_hash_v6()
+{
+ log_info "Running IPv6 custom multipath hash tests"
+
+ sysctl_set net.ipv6.fib_multipath_hash_policy 3
+
+ # Prevent the neighbour table from overflowing, as different neighbour
+ # entries will be created on $ol4 when using different destination IPs.
+ sysctl_set net.ipv6.neigh.default.gc_thresh1 1024
+ sysctl_set net.ipv6.neigh.default.gc_thresh2 1024
+ sysctl_set net.ipv6.neigh.default.gc_thresh3 1024
+
+ sysctl_set net.ipv6.fib_multipath_hash_fields 0x0001
+ custom_hash_test "Source IP" "balanced" send_src_ipv6
+ custom_hash_test "Source IP" "unbalanced" send_dst_ipv6
+
+ sysctl_set net.ipv6.fib_multipath_hash_fields 0x0002
+ custom_hash_test "Destination IP" "balanced" send_dst_ipv6
+ custom_hash_test "Destination IP" "unbalanced" send_src_ipv6
+
+ sysctl_set net.ipv6.fib_multipath_hash_fields 0x0008
+ custom_hash_test "Flowlabel" "balanced" send_flowlabel
+ custom_hash_test "Flowlabel" "unbalanced" send_src_ipv6
+
+ sysctl_set net.ipv6.fib_multipath_hash_fields 0x0010
+ custom_hash_test "Source port" "balanced" send_src_udp6
+ custom_hash_test "Source port" "unbalanced" send_dst_udp6
+
+ sysctl_set net.ipv6.fib_multipath_hash_fields 0x0020
+ custom_hash_test "Destination port" "balanced" send_dst_udp6
+ custom_hash_test "Destination port" "unbalanced" send_src_udp6
+
+ sysctl_restore net.ipv6.neigh.default.gc_thresh3
+ sysctl_restore net.ipv6.neigh.default.gc_thresh2
+ sysctl_restore net.ipv6.neigh.default.gc_thresh1
+
+ sysctl_restore net.ipv6.fib_multipath_hash_policy
+}
+
+custom_hash()
+{
+ # Test that when the hash policy is set to custom, traffic is
+ # distributed only according to the fields set in the
+ # fib_multipath_hash_fields sysctl.
+ #
+ # Each time set a different field and make sure traffic is only
+ # distributed when the field is changed in the packet stream.
+ custom_hash_v4
+ custom_hash_v6
+}
+
+trap cleanup EXIT
+
+setup_prepare
+setup_wait
+tests_run
+
+exit $EXIT_STATUS
DEVLINK_VIDDID=$(lspci -s $(echo $DEVLINK_DEV | cut -d"/" -f2) \
-n | cut -d" " -f3)
+elif [[ ! -z "$DEVLINK_DEV" ]]; then
+ devlink dev show $DEVLINK_DEV &> /dev/null
+ if [ $? -ne 0 ]; then
+ echo "SKIP: devlink device \"$DEVLINK_DEV\" not found"
+ exit 1
+ fi
fi
##############################################################################
| jq '.[][][]["stats"]["rx"]["bytes"]'
}
+devlink_trap_drop_packets_get()
+{
+ local trap_name=$1; shift
+
+ devlink -js trap show $DEVLINK_DEV trap $trap_name \
+ | jq '.[][][]["stats"]["rx"]["dropped"]'
+}
+
devlink_trap_stats_idle_test()
{
local trap_name=$1; shift
fi
}
+devlink_trap_drop_stats_idle_test()
+{
+ local trap_name=$1; shift
+ local t0_packets t0_bytes
+
+ t0_packets=$(devlink_trap_drop_packets_get $trap_name)
+
+ sleep 1
+
+ t1_packets=$(devlink_trap_drop_packets_get $trap_name)
+
+ if [[ $t0_packets -eq $t1_packets ]]; then
+ return 0
+ else
+ return 1
+ fi
+}
+
devlink_traps_enable_all()
{
local trap_name
--- /dev/null
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# Test traffic distribution when there are multiple paths between an IPv4 GRE
+# tunnel. The tunnel carries IPv4 and IPv6 traffic between multiple hosts.
+# Multiple routes are in the underlay network. With the default multipath
+# policy, SW2 will only look at the outer IP addresses, hence only a single
+# route would be used.
+#
+# +--------------------------------+
+# | H1 |
+# | $h1 + |
+# | 198.51.100.{2-253}/24 | |
+# | 2001:db8:1::{2-fd}/64 | |
+# +-------------------------|------+
+# |
+# +-------------------------|------------------+
+# | SW1 | |
+# | $ol1 + |
+# | 198.51.100.1/24 |
+# | 2001:db8:1::1/64 |
+# | |
+# | + g1 (gre) |
+# | loc=192.0.2.1 |
+# | rem=192.0.2.2 --. |
+# | tos=inherit | |
+# | v |
+# | + $ul1 |
+# | | 192.0.2.17/28 |
+# +---------------------|----------------------+
+# |
+# +---------------------|----------------------+
+# | SW2 | |
+# | $ul21 + |
+# | 192.0.2.18/28 | |
+# | | |
+# ! __________________+___ |
+# | / \ |
+# | | | |
+# | + $ul22.111 (vlan) + $ul22.222 (vlan) |
+# | | 192.0.2.33/28 | 192.0.2.49/28 |
+# | | | |
+# +--|----------------------|------------------+
+# | |
+# +--|----------------------|------------------+
+# | | | |
+# | + $ul32.111 (vlan) + $ul32.222 (vlan) |
+# | | 192.0.2.34/28 | 192.0.2.50/28 |
+# | | | |
+# | \__________________+___/ |
+# | | |
+# | | |
+# | $ul31 + |
+# | 192.0.2.65/28 | SW3 |
+# +---------------------|----------------------+
+# |
+# +---------------------|----------------------+
+# | + $ul4 |
+# | ^ 192.0.2.66/28 |
+# | | |
+# | + g2 (gre) | |
+# | loc=192.0.2.2 | |
+# | rem=192.0.2.1 --' |
+# | tos=inherit |
+# | |
+# | $ol4 + |
+# | 203.0.113.1/24 | |
+# | 2001:db8:2::1/64 | SW4 |
+# +-------------------------|------------------+
+# |
+# +-------------------------|------+
+# | | |
+# | $h2 + |
+# | 203.0.113.{2-253}/24 |
+# | 2001:db8:2::{2-fd}/64 H2 |
+# +--------------------------------+
+
+ALL_TESTS="
+ ping_ipv4
+ ping_ipv6
+ custom_hash
+"
+
+NUM_NETIFS=10
+source lib.sh
+
+h1_create()
+{
+ simple_if_init $h1 198.51.100.2/24 2001:db8:1::2/64
+ ip route add vrf v$h1 default via 198.51.100.1 dev $h1
+ ip -6 route add vrf v$h1 default via 2001:db8:1::1 dev $h1
+}
+
+h1_destroy()
+{
+ ip -6 route del vrf v$h1 default
+ ip route del vrf v$h1 default
+ simple_if_fini $h1 198.51.100.2/24 2001:db8:1::2/64
+}
+
+sw1_create()
+{
+ simple_if_init $ol1 198.51.100.1/24 2001:db8:1::1/64
+ __simple_if_init $ul1 v$ol1 192.0.2.17/28
+
+ tunnel_create g1 gre 192.0.2.1 192.0.2.2 tos inherit dev v$ol1
+ __simple_if_init g1 v$ol1 192.0.2.1/32
+ ip route add vrf v$ol1 192.0.2.2/32 via 192.0.2.18
+
+ ip route add vrf v$ol1 203.0.113.0/24 dev g1
+ ip -6 route add vrf v$ol1 2001:db8:2::/64 dev g1
+}
+
+sw1_destroy()
+{
+ ip -6 route del vrf v$ol1 2001:db8:2::/64
+ ip route del vrf v$ol1 203.0.113.0/24
+
+ ip route del vrf v$ol1 192.0.2.2/32
+ __simple_if_fini g1 192.0.2.1/32
+ tunnel_destroy g1
+
+ __simple_if_fini $ul1 192.0.2.17/28
+ simple_if_fini $ol1 198.51.100.1/24 2001:db8:1::1/64
+}
+
+sw2_create()
+{
+ simple_if_init $ul21 192.0.2.18/28
+ __simple_if_init $ul22 v$ul21
+ vlan_create $ul22 111 v$ul21 192.0.2.33/28
+ vlan_create $ul22 222 v$ul21 192.0.2.49/28
+
+ ip route add vrf v$ul21 192.0.2.1/32 via 192.0.2.17
+ ip route add vrf v$ul21 192.0.2.2/32 \
+ nexthop via 192.0.2.34 \
+ nexthop via 192.0.2.50
+}
+
+sw2_destroy()
+{
+ ip route del vrf v$ul21 192.0.2.2/32
+ ip route del vrf v$ul21 192.0.2.1/32
+
+ vlan_destroy $ul22 222
+ vlan_destroy $ul22 111
+ __simple_if_fini $ul22
+ simple_if_fini $ul21 192.0.2.18/28
+}
+
+sw3_create()
+{
+ simple_if_init $ul31 192.0.2.65/28
+ __simple_if_init $ul32 v$ul31
+ vlan_create $ul32 111 v$ul31 192.0.2.34/28
+ vlan_create $ul32 222 v$ul31 192.0.2.50/28
+
+ ip route add vrf v$ul31 192.0.2.2/32 via 192.0.2.66
+ ip route add vrf v$ul31 192.0.2.1/32 \
+ nexthop via 192.0.2.33 \
+ nexthop via 192.0.2.49
+
+ tc qdisc add dev $ul32 clsact
+ tc filter add dev $ul32 ingress pref 111 prot 802.1Q \
+ flower vlan_id 111 action pass
+ tc filter add dev $ul32 ingress pref 222 prot 802.1Q \
+ flower vlan_id 222 action pass
+}
+
+sw3_destroy()
+{
+ tc qdisc del dev $ul32 clsact
+
+ ip route del vrf v$ul31 192.0.2.1/32
+ ip route del vrf v$ul31 192.0.2.2/32
+
+ vlan_destroy $ul32 222
+ vlan_destroy $ul32 111
+ __simple_if_fini $ul32
+ simple_if_fini $ul31 192.0.2.65/28
+}
+
+sw4_create()
+{
+ simple_if_init $ol4 203.0.113.1/24 2001:db8:2::1/64
+ __simple_if_init $ul4 v$ol4 192.0.2.66/28
+
+ tunnel_create g2 gre 192.0.2.2 192.0.2.1 tos inherit dev v$ol4
+ __simple_if_init g2 v$ol4 192.0.2.2/32
+ ip route add vrf v$ol4 192.0.2.1/32 via 192.0.2.65
+
+ ip route add vrf v$ol4 198.51.100.0/24 dev g2
+ ip -6 route add vrf v$ol4 2001:db8:1::/64 dev g2
+}
+
+sw4_destroy()
+{
+ ip -6 route del vrf v$ol4 2001:db8:1::/64
+ ip route del vrf v$ol4 198.51.100.0/24
+
+ ip route del vrf v$ol4 192.0.2.1/32
+ __simple_if_fini g2 192.0.2.2/32
+ tunnel_destroy g2
+
+ __simple_if_fini $ul4 192.0.2.66/28
+ simple_if_fini $ol4 203.0.113.1/24 2001:db8:2::1/64
+}
+
+h2_create()
+{
+ simple_if_init $h2 203.0.113.2/24 2001:db8:2::2/64
+ ip route add vrf v$h2 default via 203.0.113.1 dev $h2
+ ip -6 route add vrf v$h2 default via 2001:db8:2::1 dev $h2
+}
+
+h2_destroy()
+{
+ ip -6 route del vrf v$h2 default
+ ip route del vrf v$h2 default
+ simple_if_fini $h2 203.0.113.2/24 2001:db8:2::2/64
+}
+
+setup_prepare()
+{
+ h1=${NETIFS[p1]}
+
+ ol1=${NETIFS[p2]}
+ ul1=${NETIFS[p3]}
+
+ ul21=${NETIFS[p4]}
+ ul22=${NETIFS[p5]}
+
+ ul32=${NETIFS[p6]}
+ ul31=${NETIFS[p7]}
+
+ ul4=${NETIFS[p8]}
+ ol4=${NETIFS[p9]}
+
+ h2=${NETIFS[p10]}
+
+ vrf_prepare
+ h1_create
+ sw1_create
+ sw2_create
+ sw3_create
+ sw4_create
+ h2_create
+
+ forwarding_enable
+}
+
+cleanup()
+{
+ pre_cleanup
+
+ forwarding_restore
+
+ h2_destroy
+ sw4_destroy
+ sw3_destroy
+ sw2_destroy
+ sw1_destroy
+ h1_destroy
+ vrf_cleanup
+}
+
+ping_ipv4()
+{
+ ping_test $h1 203.0.113.2
+}
+
+ping_ipv6()
+{
+ ping6_test $h1 2001:db8:2::2
+}
+
+send_src_ipv4()
+{
+ $MZ $h1 -q -p 64 -A "198.51.100.2-198.51.100.253" -B 203.0.113.2 \
+ -d 1msec -c 50 -t udp "sp=20000,dp=30000"
+}
+
+send_dst_ipv4()
+{
+ $MZ $h1 -q -p 64 -A 198.51.100.2 -B "203.0.113.2-203.0.113.253" \
+ -d 1msec -c 50 -t udp "sp=20000,dp=30000"
+}
+
+send_src_udp4()
+{
+ $MZ $h1 -q -p 64 -A 198.51.100.2 -B 203.0.113.2 \
+ -d 1msec -t udp "sp=0-32768,dp=30000"
+}
+
+send_dst_udp4()
+{
+ $MZ $h1 -q -p 64 -A 198.51.100.2 -B 203.0.113.2 \
+ -d 1msec -t udp "sp=20000,dp=0-32768"
+}
+
+send_src_ipv6()
+{
+ $MZ -6 $h1 -q -p 64 -A "2001:db8:1::2-2001:db8:1::fd" -B 2001:db8:2::2 \
+ -d 1msec -c 50 -t udp "sp=20000,dp=30000"
+}
+
+send_dst_ipv6()
+{
+ $MZ -6 $h1 -q -p 64 -A 2001:db8:1::2 -B "2001:db8:2::2-2001:db8:2::fd" \
+ -d 1msec -c 50 -t udp "sp=20000,dp=30000"
+}
+
+send_flowlabel()
+{
+ # Generate 16384 echo requests, each with a random flow label.
+ for _ in $(seq 1 16384); do
+ ip vrf exec v$h1 \
+ $PING6 2001:db8:2::2 -F 0 -c 1 -q >/dev/null 2>&1
+ done
+}
+
+send_src_udp6()
+{
+ $MZ -6 $h1 -q -p 64 -A 2001:db8:1::2 -B 2001:db8:2::2 \
+ -d 1msec -t udp "sp=0-32768,dp=30000"
+}
+
+send_dst_udp6()
+{
+ $MZ -6 $h1 -q -p 64 -A 2001:db8:1::2 -B 2001:db8:2::2 \
+ -d 1msec -t udp "sp=20000,dp=0-32768"
+}
+
+custom_hash_test()
+{
+ local field="$1"; shift
+ local balanced="$1"; shift
+ local send_flows="$@"
+
+ RET=0
+
+ local t0_111=$(tc_rule_stats_get $ul32 111 ingress)
+ local t0_222=$(tc_rule_stats_get $ul32 222 ingress)
+
+ $send_flows
+
+ local t1_111=$(tc_rule_stats_get $ul32 111 ingress)
+ local t1_222=$(tc_rule_stats_get $ul32 222 ingress)
+
+ local d111=$((t1_111 - t0_111))
+ local d222=$((t1_222 - t0_222))
+
+ local diff=$((d222 - d111))
+ local sum=$((d111 + d222))
+
+ local pct=$(echo "$diff / $sum * 100" | bc -l)
+ local is_balanced=$(echo "-20 <= $pct && $pct <= 20" | bc)
+
+ [[ ( $is_balanced -eq 1 && $balanced == "balanced" ) ||
+ ( $is_balanced -eq 0 && $balanced == "unbalanced" ) ]]
+ check_err $? "Expected traffic to be $balanced, but it is not"
+
+ log_test "Multipath hash field: $field ($balanced)"
+ log_info "Packets sent on path1 / path2: $d111 / $d222"
+}
+
+custom_hash_v4()
+{
+ log_info "Running IPv4 overlay custom multipath hash tests"
+
+ # Prevent the neighbour table from overflowing, as different neighbour
+ # entries will be created on $ol4 when using different destination IPs.
+ sysctl_set net.ipv4.neigh.default.gc_thresh1 1024
+ sysctl_set net.ipv4.neigh.default.gc_thresh2 1024
+ sysctl_set net.ipv4.neigh.default.gc_thresh3 1024
+
+ sysctl_set net.ipv4.fib_multipath_hash_fields 0x0040
+ custom_hash_test "Inner source IP" "balanced" send_src_ipv4
+ custom_hash_test "Inner source IP" "unbalanced" send_dst_ipv4
+
+ sysctl_set net.ipv4.fib_multipath_hash_fields 0x0080
+ custom_hash_test "Inner destination IP" "balanced" send_dst_ipv4
+ custom_hash_test "Inner destination IP" "unbalanced" send_src_ipv4
+
+ sysctl_set net.ipv4.fib_multipath_hash_fields 0x0400
+ custom_hash_test "Inner source port" "balanced" send_src_udp4
+ custom_hash_test "Inner source port" "unbalanced" send_dst_udp4
+
+ sysctl_set net.ipv4.fib_multipath_hash_fields 0x0800
+ custom_hash_test "Inner destination port" "balanced" send_dst_udp4
+ custom_hash_test "Inner destination port" "unbalanced" send_src_udp4
+
+ sysctl_restore net.ipv4.neigh.default.gc_thresh3
+ sysctl_restore net.ipv4.neigh.default.gc_thresh2
+ sysctl_restore net.ipv4.neigh.default.gc_thresh1
+}
+
+custom_hash_v6()
+{
+ log_info "Running IPv6 overlay custom multipath hash tests"
+
+ # Prevent the neighbour table from overflowing, as different neighbour
+ # entries will be created on $ol4 when using different destination IPs.
+ sysctl_set net.ipv6.neigh.default.gc_thresh1 1024
+ sysctl_set net.ipv6.neigh.default.gc_thresh2 1024
+ sysctl_set net.ipv6.neigh.default.gc_thresh3 1024
+
+ sysctl_set net.ipv4.fib_multipath_hash_fields 0x0040
+ custom_hash_test "Inner source IP" "balanced" send_src_ipv6
+ custom_hash_test "Inner source IP" "unbalanced" send_dst_ipv6
+
+ sysctl_set net.ipv4.fib_multipath_hash_fields 0x0080
+ custom_hash_test "Inner destination IP" "balanced" send_dst_ipv6
+ custom_hash_test "Inner destination IP" "unbalanced" send_src_ipv6
+
+ sysctl_set net.ipv4.fib_multipath_hash_fields 0x0200
+ custom_hash_test "Inner flowlabel" "balanced" send_flowlabel
+ custom_hash_test "Inner flowlabel" "unbalanced" send_src_ipv6
+
+ sysctl_set net.ipv4.fib_multipath_hash_fields 0x0400
+ custom_hash_test "Inner source port" "balanced" send_src_udp6
+ custom_hash_test "Inner source port" "unbalanced" send_dst_udp6
+
+ sysctl_set net.ipv4.fib_multipath_hash_fields 0x0800
+ custom_hash_test "Inner destination port" "balanced" send_dst_udp6
+ custom_hash_test "Inner destination port" "unbalanced" send_src_udp6
+
+ sysctl_restore net.ipv6.neigh.default.gc_thresh3
+ sysctl_restore net.ipv6.neigh.default.gc_thresh2
+ sysctl_restore net.ipv6.neigh.default.gc_thresh1
+}
+
+custom_hash()
+{
+ # Test that when the hash policy is set to custom, traffic is
+ # distributed only according to the fields set in the
+ # fib_multipath_hash_fields sysctl.
+ #
+ # Each time set a different field and make sure traffic is only
+ # distributed when the field is changed in the packet stream.
+
+ sysctl_set net.ipv4.fib_multipath_hash_policy 3
+
+ custom_hash_v4
+ custom_hash_v6
+
+ sysctl_restore net.ipv4.fib_multipath_hash_policy
+}
+
+trap cleanup EXIT
+
+setup_prepare
+setup_wait
+tests_run
+
+exit $EXIT_STATUS
--- /dev/null
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# Test traffic distribution when there are multiple paths between an IPv6 GRE
+# tunnel. The tunnel carries IPv4 and IPv6 traffic between multiple hosts.
+# Multiple routes are in the underlay network. With the default multipath
+# policy, SW2 will only look at the outer IP addresses, hence only a single
+# route would be used.
+#
+# +--------------------------------+
+# | H1 |
+# | $h1 + |
+# | 198.51.100.{2-253}/24 | |
+# | 2001:db8:1::{2-fd}/64 | |
+# +-------------------------|------+
+# |
+# +-------------------------|-------------------+
+# | SW1 | |
+# | $ol1 + |
+# | 198.51.100.1/24 |
+# | 2001:db8:1::1/64 |
+# | |
+# |+ g1 (ip6gre) |
+# | loc=2001:db8:3::1 |
+# | rem=2001:db8:3::2 -. |
+# | tos=inherit | |
+# | v |
+# | + $ul1 |
+# | | 2001:db8:10::1/64 |
+# +---------------------|-----------------------+
+# |
+# +---------------------|-----------------------+
+# | SW2 | |
+# | $ul21 + |
+# | 2001:db8:10::2/64 | |
+# | | |
+# ! __________________+___ |
+# | / \ |
+# | | | |
+# | + $ul22.111 (vlan) + $ul22.222 (vlan) |
+# | | 2001:db8:11::1/64 | 2001:db8:12::1/64 |
+# | | | |
+# +--|----------------------|-------------------+
+# | |
+# +--|----------------------|-------------------+
+# | | | |
+# | + $ul32.111 (vlan) + $ul32.222 (vlan) |
+# | | 2001:db8:11::2/64 | 2001:db8:12::2/64 |
+# | | | |
+# | \__________________+___/ |
+# | | |
+# | | |
+# | $ul31 + |
+# | 2001:db8:13::1/64 | SW3 |
+# +---------------------|-----------------------+
+# |
+# +---------------------|-----------------------+
+# | + $ul4 |
+# | ^ 2001:db8:13::2/64 |
+# | | |
+# |+ g2 (ip6gre) | |
+# | loc=2001:db8:3::2 | |
+# | rem=2001:db8:3::1 -' |
+# | tos=inherit |
+# | |
+# | $ol4 + |
+# | 203.0.113.1/24 | |
+# | 2001:db8:2::1/64 | SW4 |
+# +-------------------------|-------------------+
+# |
+# +-------------------------|------+
+# | | |
+# | $h2 + |
+# | 203.0.113.{2-253}/24 |
+# | 2001:db8:2::{2-fd}/64 H2 |
+# +--------------------------------+
+
+ALL_TESTS="
+ ping_ipv4
+ ping_ipv6
+ custom_hash
+"
+
+NUM_NETIFS=10
+source lib.sh
+
+h1_create()
+{
+ simple_if_init $h1 198.51.100.2/24 2001:db8:1::2/64
+ ip route add vrf v$h1 default via 198.51.100.1 dev $h1
+ ip -6 route add vrf v$h1 default via 2001:db8:1::1 dev $h1
+}
+
+h1_destroy()
+{
+ ip -6 route del vrf v$h1 default
+ ip route del vrf v$h1 default
+ simple_if_fini $h1 198.51.100.2/24 2001:db8:1::2/64
+}
+
+sw1_create()
+{
+ simple_if_init $ol1 198.51.100.1/24 2001:db8:1::1/64
+ __simple_if_init $ul1 v$ol1 2001:db8:10::1/64
+
+ tunnel_create g1 ip6gre 2001:db8:3::1 2001:db8:3::2 tos inherit \
+ dev v$ol1
+ __simple_if_init g1 v$ol1 2001:db8:3::1/128
+ ip route add vrf v$ol1 2001:db8:3::2/128 via 2001:db8:10::2
+
+ ip route add vrf v$ol1 203.0.113.0/24 dev g1
+ ip -6 route add vrf v$ol1 2001:db8:2::/64 dev g1
+}
+
+sw1_destroy()
+{
+ ip -6 route del vrf v$ol1 2001:db8:2::/64
+ ip route del vrf v$ol1 203.0.113.0/24
+
+ ip route del vrf v$ol1 2001:db8:3::2/128
+ __simple_if_fini g1 2001:db8:3::1/128
+ tunnel_destroy g1
+
+ __simple_if_fini $ul1 2001:db8:10::1/64
+ simple_if_fini $ol1 198.51.100.1/24 2001:db8:1::1/64
+}
+
+sw2_create()
+{
+ simple_if_init $ul21 2001:db8:10::2/64
+ __simple_if_init $ul22 v$ul21
+ vlan_create $ul22 111 v$ul21 2001:db8:11::1/64
+ vlan_create $ul22 222 v$ul21 2001:db8:12::1/64
+
+ ip -6 route add vrf v$ul21 2001:db8:3::1/128 via 2001:db8:10::1
+ ip -6 route add vrf v$ul21 2001:db8:3::2/128 \
+ nexthop via 2001:db8:11::2 \
+ nexthop via 2001:db8:12::2
+}
+
+sw2_destroy()
+{
+ ip -6 route del vrf v$ul21 2001:db8:3::2/128
+ ip -6 route del vrf v$ul21 2001:db8:3::1/128
+
+ vlan_destroy $ul22 222
+ vlan_destroy $ul22 111
+ __simple_if_fini $ul22
+ simple_if_fini $ul21 2001:db8:10::2/64
+}
+
+sw3_create()
+{
+ simple_if_init $ul31 2001:db8:13::1/64
+ __simple_if_init $ul32 v$ul31
+ vlan_create $ul32 111 v$ul31 2001:db8:11::2/64
+ vlan_create $ul32 222 v$ul31 2001:db8:12::2/64
+
+ ip -6 route add vrf v$ul31 2001:db8:3::2/128 via 2001:db8:13::2
+ ip -6 route add vrf v$ul31 2001:db8:3::1/128 \
+ nexthop via 2001:db8:11::1 \
+ nexthop via 2001:db8:12::1
+
+ tc qdisc add dev $ul32 clsact
+ tc filter add dev $ul32 ingress pref 111 prot 802.1Q \
+ flower vlan_id 111 action pass
+ tc filter add dev $ul32 ingress pref 222 prot 802.1Q \
+ flower vlan_id 222 action pass
+}
+
+sw3_destroy()
+{
+ tc qdisc del dev $ul32 clsact
+
+ ip -6 route del vrf v$ul31 2001:db8:3::1/128
+ ip -6 route del vrf v$ul31 2001:db8:3::2/128
+
+ vlan_destroy $ul32 222
+ vlan_destroy $ul32 111
+ __simple_if_fini $ul32
+ simple_if_fini $ul31 2001:db8:13::1/64
+}
+
+sw4_create()
+{
+ simple_if_init $ol4 203.0.113.1/24 2001:db8:2::1/64
+ __simple_if_init $ul4 v$ol4 2001:db8:13::2/64
+
+ tunnel_create g2 ip6gre 2001:db8:3::2 2001:db8:3::1 tos inherit \
+ dev v$ol4
+ __simple_if_init g2 v$ol4 2001:db8:3::2/128
+ ip -6 route add vrf v$ol4 2001:db8:3::1/128 via 2001:db8:13::1
+
+ ip route add vrf v$ol4 198.51.100.0/24 dev g2
+ ip -6 route add vrf v$ol4 2001:db8:1::/64 dev g2
+}
+
+sw4_destroy()
+{
+ ip -6 route del vrf v$ol4 2001:db8:1::/64
+ ip route del vrf v$ol4 198.51.100.0/24
+
+ ip -6 route del vrf v$ol4 2001:db8:3::1/128
+ __simple_if_fini g2 2001:db8:3::2/128
+ tunnel_destroy g2
+
+ __simple_if_fini $ul4 2001:db8:13::2/64
+ simple_if_fini $ol4 203.0.113.1/24 2001:db8:2::1/64
+}
+
+h2_create()
+{
+ simple_if_init $h2 203.0.113.2/24 2001:db8:2::2/64
+ ip route add vrf v$h2 default via 203.0.113.1 dev $h2
+ ip -6 route add vrf v$h2 default via 2001:db8:2::1 dev $h2
+}
+
+h2_destroy()
+{
+ ip -6 route del vrf v$h2 default
+ ip route del vrf v$h2 default
+ simple_if_fini $h2 203.0.113.2/24 2001:db8:2::2/64
+}
+
+setup_prepare()
+{
+ h1=${NETIFS[p1]}
+
+ ol1=${NETIFS[p2]}
+ ul1=${NETIFS[p3]}
+
+ ul21=${NETIFS[p4]}
+ ul22=${NETIFS[p5]}
+
+ ul32=${NETIFS[p6]}
+ ul31=${NETIFS[p7]}
+
+ ul4=${NETIFS[p8]}
+ ol4=${NETIFS[p9]}
+
+ h2=${NETIFS[p10]}
+
+ vrf_prepare
+ h1_create
+ sw1_create
+ sw2_create
+ sw3_create
+ sw4_create
+ h2_create
+
+ forwarding_enable
+}
+
+cleanup()
+{
+ pre_cleanup
+
+ forwarding_restore
+
+ h2_destroy
+ sw4_destroy
+ sw3_destroy
+ sw2_destroy
+ sw1_destroy
+ h1_destroy
+ vrf_cleanup
+}
+
+ping_ipv4()
+{
+ ping_test $h1 203.0.113.2
+}
+
+ping_ipv6()
+{
+ ping6_test $h1 2001:db8:2::2
+}
+
+send_src_ipv4()
+{
+ $MZ $h1 -q -p 64 -A "198.51.100.2-198.51.100.253" -B 203.0.113.2 \
+ -d 1msec -c 50 -t udp "sp=20000,dp=30000"
+}
+
+send_dst_ipv4()
+{
+ $MZ $h1 -q -p 64 -A 198.51.100.2 -B "203.0.113.2-203.0.113.253" \
+ -d 1msec -c 50 -t udp "sp=20000,dp=30000"
+}
+
+send_src_udp4()
+{
+ $MZ $h1 -q -p 64 -A 198.51.100.2 -B 203.0.113.2 \
+ -d 1msec -t udp "sp=0-32768,dp=30000"
+}
+
+send_dst_udp4()
+{
+ $MZ $h1 -q -p 64 -A 198.51.100.2 -B 203.0.113.2 \
+ -d 1msec -t udp "sp=20000,dp=0-32768"
+}
+
+send_src_ipv6()
+{
+ $MZ -6 $h1 -q -p 64 -A "2001:db8:1::2-2001:db8:1::fd" -B 2001:db8:2::2 \
+ -d 1msec -c 50 -t udp "sp=20000,dp=30000"
+}
+
+send_dst_ipv6()
+{
+ $MZ -6 $h1 -q -p 64 -A 2001:db8:1::2 -B "2001:db8:2::2-2001:db8:2::fd" \
+ -d 1msec -c 50 -t udp "sp=20000,dp=30000"
+}
+
+send_flowlabel()
+{
+ # Generate 16384 echo requests, each with a random flow label.
+ for _ in $(seq 1 16384); do
+ ip vrf exec v$h1 \
+ $PING6 2001:db8:2::2 -F 0 -c 1 -q >/dev/null 2>&1
+ done
+}
+
+send_src_udp6()
+{
+ $MZ -6 $h1 -q -p 64 -A 2001:db8:1::2 -B 2001:db8:2::2 \
+ -d 1msec -t udp "sp=0-32768,dp=30000"
+}
+
+send_dst_udp6()
+{
+ $MZ -6 $h1 -q -p 64 -A 2001:db8:1::2 -B 2001:db8:2::2 \
+ -d 1msec -t udp "sp=20000,dp=0-32768"
+}
+
+custom_hash_test()
+{
+ local field="$1"; shift
+ local balanced="$1"; shift
+ local send_flows="$@"
+
+ RET=0
+
+ local t0_111=$(tc_rule_stats_get $ul32 111 ingress)
+ local t0_222=$(tc_rule_stats_get $ul32 222 ingress)
+
+ $send_flows
+
+ local t1_111=$(tc_rule_stats_get $ul32 111 ingress)
+ local t1_222=$(tc_rule_stats_get $ul32 222 ingress)
+
+ local d111=$((t1_111 - t0_111))
+ local d222=$((t1_222 - t0_222))
+
+ local diff=$((d222 - d111))
+ local sum=$((d111 + d222))
+
+ local pct=$(echo "$diff / $sum * 100" | bc -l)
+ local is_balanced=$(echo "-20 <= $pct && $pct <= 20" | bc)
+
+ [[ ( $is_balanced -eq 1 && $balanced == "balanced" ) ||
+ ( $is_balanced -eq 0 && $balanced == "unbalanced" ) ]]
+ check_err $? "Expected traffic to be $balanced, but it is not"
+
+ log_test "Multipath hash field: $field ($balanced)"
+ log_info "Packets sent on path1 / path2: $d111 / $d222"
+}
+
+custom_hash_v4()
+{
+ log_info "Running IPv4 overlay custom multipath hash tests"
+
+ # Prevent the neighbour table from overflowing, as different neighbour
+ # entries will be created on $ol4 when using different destination IPs.
+ sysctl_set net.ipv4.neigh.default.gc_thresh1 1024
+ sysctl_set net.ipv4.neigh.default.gc_thresh2 1024
+ sysctl_set net.ipv4.neigh.default.gc_thresh3 1024
+
+ sysctl_set net.ipv6.fib_multipath_hash_fields 0x0040
+ custom_hash_test "Inner source IP" "balanced" send_src_ipv4
+ custom_hash_test "Inner source IP" "unbalanced" send_dst_ipv4
+
+ sysctl_set net.ipv6.fib_multipath_hash_fields 0x0080
+ custom_hash_test "Inner destination IP" "balanced" send_dst_ipv4
+ custom_hash_test "Inner destination IP" "unbalanced" send_src_ipv4
+
+ sysctl_set net.ipv6.fib_multipath_hash_fields 0x0400
+ custom_hash_test "Inner source port" "balanced" send_src_udp4
+ custom_hash_test "Inner source port" "unbalanced" send_dst_udp4
+
+ sysctl_set net.ipv6.fib_multipath_hash_fields 0x0800
+ custom_hash_test "Inner destination port" "balanced" send_dst_udp4
+ custom_hash_test "Inner destination port" "unbalanced" send_src_udp4
+
+ sysctl_restore net.ipv4.neigh.default.gc_thresh3
+ sysctl_restore net.ipv4.neigh.default.gc_thresh2
+ sysctl_restore net.ipv4.neigh.default.gc_thresh1
+}
+
+custom_hash_v6()
+{
+ log_info "Running IPv6 overlay custom multipath hash tests"
+
+ # Prevent the neighbour table from overflowing, as different neighbour
+ # entries will be created on $ol4 when using different destination IPs.
+ sysctl_set net.ipv6.neigh.default.gc_thresh1 1024
+ sysctl_set net.ipv6.neigh.default.gc_thresh2 1024
+ sysctl_set net.ipv6.neigh.default.gc_thresh3 1024
+
+ sysctl_set net.ipv6.fib_multipath_hash_fields 0x0040
+ custom_hash_test "Inner source IP" "balanced" send_src_ipv6
+ custom_hash_test "Inner source IP" "unbalanced" send_dst_ipv6
+
+ sysctl_set net.ipv6.fib_multipath_hash_fields 0x0080
+ custom_hash_test "Inner destination IP" "balanced" send_dst_ipv6
+ custom_hash_test "Inner destination IP" "unbalanced" send_src_ipv6
+
+ sysctl_set net.ipv6.fib_multipath_hash_fields 0x0200
+ custom_hash_test "Inner flowlabel" "balanced" send_flowlabel
+ custom_hash_test "Inner flowlabel" "unbalanced" send_src_ipv6
+
+ sysctl_set net.ipv6.fib_multipath_hash_fields 0x0400
+ custom_hash_test "Inner source port" "balanced" send_src_udp6
+ custom_hash_test "Inner source port" "unbalanced" send_dst_udp6
+
+ sysctl_set net.ipv6.fib_multipath_hash_fields 0x0800
+ custom_hash_test "Inner destination port" "balanced" send_dst_udp6
+ custom_hash_test "Inner destination port" "unbalanced" send_src_udp6
+
+ sysctl_restore net.ipv6.neigh.default.gc_thresh3
+ sysctl_restore net.ipv6.neigh.default.gc_thresh2
+ sysctl_restore net.ipv6.neigh.default.gc_thresh1
+}
+
+custom_hash()
+{
+ # Test that when the hash policy is set to custom, traffic is
+ # distributed only according to the fields set in the
+ # fib_multipath_hash_fields sysctl.
+ #
+ # Each time set a different field and make sure traffic is only
+ # distributed when the field is changed in the packet stream.
+
+ sysctl_set net.ipv6.fib_multipath_hash_policy 3
+
+ custom_hash_v4
+ custom_hash_v6
+
+ sysctl_restore net.ipv6.fib_multipath_hash_policy
+}
+
+trap cleanup EXIT
+
+setup_prepare
+setup_wait
+tests_run
+
+exit $EXIT_STATUS
tc qdisc del dev $swp2 clsact
tc qdisc del dev $swp1 clsact
+ ip link set dev $swp2 down
ip link set dev $swp2 nomaster
+ ip link set dev $swp1 down
ip link set dev $swp1 nomaster
ip link del dev br1
}
tc qdisc del dev $swp2 clsact
tc qdisc del dev $swp1 clsact
+ ip link set dev $swp2 down
ip link set dev $swp2 nomaster
+ ip link set dev $swp1 down
ip link set dev $swp1 nomaster
ip link del dev br1
}
tc qdisc del dev $swp2 clsact
tc qdisc del dev $swp1 clsact
+ ip link set dev $swp2 down
ip link set dev $swp2 nomaster
+ ip link set dev $swp1 down
ip link set dev $swp1 nomaster
ip link del dev br1
}
--- /dev/null
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+# Test for checking ICMP response with dummy address instead of 0.0.0.0.
+# Sets up two namespaces like:
+# +----------------------+ +--------------------+
+# | ns1 | v4-via-v6 routes: | ns2 |
+# | | ' | |
+# | +--------+ -> 172.16.1.0/24 -> +--------+ |
+# | | veth0 +--------------------------+ veth0 | |
+# | +--------+ <- 172.16.0.0/24 <- +--------+ |
+# | 172.16.0.1 | | 2001:db8:1::2/64 |
+# | 2001:db8:1::2/64 | | |
+# +----------------------+ +--------------------+
+#
+# And then tries to ping 172.16.1.1 from ns1. This results in a "net
+# unreachable" message being sent from ns2, but there is no IPv4 address set in
+# that address space, so the kernel should substitute the dummy address
+# 192.0.0.8 defined in RFC7600.
+
+NS1=ns1
+NS2=ns2
+H1_IP=172.16.0.1/32
+H1_IP6=2001:db8:1::1
+RT1=172.16.1.0/24
+PINGADDR=172.16.1.1
+RT2=172.16.0.0/24
+H2_IP6=2001:db8:1::2
+
+TMPFILE=$(mktemp)
+
+cleanup()
+{
+ rm -f "$TMPFILE"
+ ip netns del $NS1
+ ip netns del $NS2
+}
+
+trap cleanup EXIT
+
+# Namespaces
+ip netns add $NS1
+ip netns add $NS2
+
+# Connectivity
+ip -netns $NS1 link add veth0 type veth peer name veth0 netns $NS2
+ip -netns $NS1 link set dev veth0 up
+ip -netns $NS2 link set dev veth0 up
+ip -netns $NS1 addr add $H1_IP dev veth0
+ip -netns $NS1 addr add $H1_IP6/64 dev veth0 nodad
+ip -netns $NS2 addr add $H2_IP6/64 dev veth0 nodad
+ip -netns $NS1 route add $RT1 via inet6 $H2_IP6
+ip -netns $NS2 route add $RT2 via inet6 $H1_IP6
+
+# Make sure ns2 will respond with ICMP unreachable
+ip netns exec $NS2 sysctl -qw net.ipv4.icmp_ratelimit=0 net.ipv4.ip_forward=1
+
+# Run the test - a ping runs in the background, and we capture ICMP responses
+# with tcpdump; -c 1 means it should exit on the first ping, but add a timeout
+# in case something goes wrong
+ip netns exec $NS1 ping -w 3 -i 0.5 $PINGADDR >/dev/null &
+ip netns exec $NS1 timeout 10 tcpdump -tpni veth0 -c 1 'icmp and icmp[icmptype] != icmp-echo' > $TMPFILE 2>/dev/null
+
+# Parse response and check for dummy address
+# tcpdump output looks like:
+# IP 192.0.0.8 > 172.16.0.1: ICMP net 172.16.1.1 unreachable, length 92
+RESP_IP=$(awk '{print $2}' < $TMPFILE)
+if [[ "$RESP_IP" != "192.0.0.8" ]]; then
+ echo "FAIL - got ICMP response from $RESP_IP, should be 192.0.0.8"
+ exit 1
+else
+ echo "OK"
+ exit 0
+fi
local rc=$1
local expected=$2
local msg="$3"
+ local xfail=$4
if [ ${rc} -eq ${expected} ]; then
printf "TEST: %-60s [ OK ]\n" "${msg}"
nsuccess=$((nsuccess+1))
+ elif [ ${rc} -eq ${xfail} ]; then
+ printf "TEST: %-60s [XFAIL]\n" "${msg}"
+ nxfail=$((nxfail+1))
else
ret=1
nfail=$((nfail+1))
ip -netns h1 -6 ro get ${H1_VRF_ARG} ${H2_N2_IP6} | \
grep -v "mtu" | grep -q "${R1_LLADDR}"
fi
- log_test $? 0 "IPv6: ${desc}"
+ log_test $? 0 "IPv6: ${desc}" 1
}
run_ping()
ret=0
nsuccess=0
nfail=0
+nxfail=0
while getopts :pv o
do
printf "\nTests passed: %3d\n" ${nsuccess}
printf "Tests failed: %3d\n" ${nfail}
+printf "Tests xfailed: %3d\n" ${nxfail}
exit $ret
#include <limits.h>
#include <fcntl.h>
#include <string.h>
+#include <stdarg.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <netinet/in.h>
#include <linux/tcp.h>
+#include <linux/time_types.h>
extern int optind;
static int cfg_wait;
static uint32_t cfg_mark;
+struct cfg_cmsg_types {
+ unsigned int cmsg_enabled:1;
+ unsigned int timestampns:1;
+};
+
+static struct cfg_cmsg_types cfg_cmsg_types;
+
static void die_usage(void)
{
fprintf(stderr, "Usage: mptcp_connect [-6] [-u] [-s MPTCP|TCP] [-p port] [-m mode]"
fprintf(stderr, "\t-M mark -- set socket packet mark\n");
fprintf(stderr, "\t-u -- check mptcp ulp\n");
fprintf(stderr, "\t-w num -- wait num sec before closing the socket\n");
+ fprintf(stderr, "\t-c cmsg -- test cmsg type <cmsg>\n");
fprintf(stderr,
"\t-P [saveWithPeek|saveAfterPeek] -- save data with/after MSG_PEEK form tcp socket\n");
exit(1);
}
+static void xerror(const char *fmt, ...)
+{
+ va_list ap;
+
+ va_start(ap, fmt);
+ vfprintf(stderr, fmt, ap);
+ va_end(ap);
+ exit(1);
+}
+
static void handle_signal(int nr)
{
quit = true;
return offset;
}
+static void process_cmsg(struct msghdr *msgh)
+{
+ struct __kernel_timespec ts;
+ bool ts_found = false;
+ struct cmsghdr *cmsg;
+
+ for (cmsg = CMSG_FIRSTHDR(msgh); cmsg ; cmsg = CMSG_NXTHDR(msgh, cmsg)) {
+ if (cmsg->cmsg_level == SOL_SOCKET && cmsg->cmsg_type == SO_TIMESTAMPNS_NEW) {
+ memcpy(&ts, CMSG_DATA(cmsg), sizeof(ts));
+ ts_found = true;
+ continue;
+ }
+ }
+
+ if (cfg_cmsg_types.timestampns) {
+ if (!ts_found)
+ xerror("TIMESTAMPNS not present\n");
+ }
+}
+
+static ssize_t do_recvmsg_cmsg(const int fd, char *buf, const size_t len)
+{
+ char msg_buf[8192];
+ struct iovec iov = {
+ .iov_base = buf,
+ .iov_len = len,
+ };
+ struct msghdr msg = {
+ .msg_iov = &iov,
+ .msg_iovlen = 1,
+ .msg_control = msg_buf,
+ .msg_controllen = sizeof(msg_buf),
+ };
+ int flags = 0;
+ int ret = recvmsg(fd, &msg, flags);
+
+ if (ret <= 0)
+ return ret;
+
+ if (msg.msg_controllen && !cfg_cmsg_types.cmsg_enabled)
+ xerror("got %lu bytes of cmsg data, expected 0\n",
+ (unsigned long)msg.msg_controllen);
+
+ if (msg.msg_controllen == 0 && cfg_cmsg_types.cmsg_enabled)
+ xerror("%s\n", "got no cmsg data");
+
+ if (msg.msg_controllen)
+ process_cmsg(&msg);
+
+ return ret;
+}
+
static ssize_t do_rnd_read(const int fd, char *buf, const size_t len)
{
int ret = 0;
} else if (cfg_peek == CFG_AFTER_PEEK) {
ret = recv(fd, buf, cap, MSG_PEEK);
ret = (ret < 0) ? ret : read(fd, buf, cap);
+ } else if (cfg_cmsg_types.cmsg_enabled) {
+ ret = do_recvmsg_cmsg(fd, buf, cap);
} else {
ret = read(fd, buf, cap);
}
srand(foo);
}
+static void xsetsockopt(int fd, int level, int optname, const void *optval, socklen_t optlen)
+{
+ int err;
+
+ err = setsockopt(fd, level, optname, optval, optlen);
+ if (err) {
+ perror("setsockopt");
+ exit(1);
+ }
+}
+
+static void apply_cmsg_types(int fd, const struct cfg_cmsg_types *cmsg)
+{
+ static const unsigned int on = 1;
+
+ if (cmsg->timestampns)
+ xsetsockopt(fd, SOL_SOCKET, SO_TIMESTAMPNS_NEW, &on, sizeof(on));
+}
+
+static void parse_cmsg_types(const char *type)
+{
+ char *next = strchr(type, ',');
+ unsigned int len = 0;
+
+ cfg_cmsg_types.cmsg_enabled = 1;
+
+ if (next) {
+ parse_cmsg_types(next + 1);
+ len = next - type;
+ } else {
+ len = strlen(type);
+ }
+
+ if (strncmp(type, "TIMESTAMPNS", len) == 0) {
+ cfg_cmsg_types.timestampns = 1;
+ return;
+ }
+
+ fprintf(stderr, "Unrecognized cmsg option %s\n", type);
+ exit(1);
+}
+
int main_loop(void)
{
int fd;
set_rcvbuf(fd, cfg_rcvbuf);
if (cfg_sndbuf)
set_sndbuf(fd, cfg_sndbuf);
+ if (cfg_cmsg_types.cmsg_enabled)
+ apply_cmsg_types(fd, &cfg_cmsg_types);
return copyfd_io(0, fd, 1);
}
{
int c;
- while ((c = getopt(argc, argv, "6jr:lp:s:hut:m:S:R:w:M:P:")) != -1) {
+ while ((c = getopt(argc, argv, "6jr:lp:s:hut:m:S:R:w:M:P:c:")) != -1) {
switch (c) {
case 'j':
cfg_join = true;
case 'P':
cfg_peek = parse_peek(optarg);
break;
+ case 'c':
+ parse_cmsg_types(optarg);
+ break;
}
}
set_sndbuf(fd, cfg_sndbuf);
if (cfg_mark)
set_mark(fd, cfg_mark);
+ if (cfg_cmsg_types.cmsg_enabled)
+ apply_cmsg_types(fd, &cfg_cmsg_types);
return main_loop_s(fd);
}
time_start=$(date +%s)
-optstring="S:R:d:e:l:r:h4cm:f:t"
+optstring="S:R:d:e:l:r:h4cm:f:tC"
ret=0
sin=""
sout=""
rcvbuf=0
options_log=true
do_tcp=0
+checksum=false
filesize=0
if [ $tc_loss -eq 100 ];then
echo -e "\t-R: set rcvbuf value (default: use kernel default)"
echo -e "\t-m: test mode (poll, sendfile; default: poll)"
echo -e "\t-t: also run tests with TCP (use twice to non-fallback tcp)"
+ echo -e "\t-C: enable the MPTCP data checksum"
}
while getopts "$optstring" option;do
"t")
do_tcp=$((do_tcp+1))
;;
+ "C")
+ checksum=true
+ ;;
"?")
usage $0
exit 1
ip -net "$ns4" route add default via 10.0.3.2
ip -net "$ns4" route add default via dead:beef:3::2
-# use TCP syn cookies, even if no flooding was detected.
-ip netns exec "$ns2" sysctl -q net.ipv4.tcp_syncookies=2
+if $checksum; then
+ for i in "$ns1" "$ns2" "$ns3" "$ns4";do
+ ip netns exec $i sysctl -q net.mptcp.checksum_enabled=1
+ done
+fi
set_ethtool_flags() {
local ns="$1"
local stat_ackrx_now_l=$(get_mib_counter "${listener_ns}" "MPTcpExtMPCapableACKRX")
local stat_cookietx_now=$(get_mib_counter "${listener_ns}" "TcpExtSyncookiesSent")
local stat_cookierx_now=$(get_mib_counter "${listener_ns}" "TcpExtSyncookiesRecv")
+ local stat_ooo_now=$(get_mib_counter "${listener_ns}" "TcpExtTCPOFOQueue")
expect_synrx=$((stat_synrx_last_l))
expect_ackrx=$((stat_ackrx_last_l))
"${stat_synrx_now_l}" "${expect_synrx}" 1>&2
retc=1
fi
- if [ ${stat_ackrx_now_l} -lt ${expect_ackrx} ]; then
- printf "[ FAIL ] lower MPC ACK rx (%d) than expected (%d)\n" \
- "${stat_ackrx_now_l}" "${expect_ackrx}" 1>&2
- rets=1
+ if [ ${stat_ackrx_now_l} -lt ${expect_ackrx} -a ${stat_ooo_now} -eq 0 ]; then
+ if [ ${stat_ooo_now} -eq 0 ]; then
+ printf "[ FAIL ] lower MPC ACK rx (%d) than expected (%d)\n" \
+ "${stat_ackrx_now_l}" "${expect_ackrx}" 1>&2
+ rets=1
+ else
+ printf "[ Note ] fallback due to TCP OoO"
+ fi
fi
if [ $retc -eq 0 ] && [ $rets -eq 0 ]; then
run_tests_lo "$ns1" "$ns1" dead:beef:1::1 1 "-P ${peekmode}"
}
+display_time()
+{
+ time_end=$(date +%s)
+ time_run=$((time_end-time_start))
+
+ echo "Time: ${time_run} seconds"
+}
+
+stop_if_error()
+{
+ local msg="$1"
+
+ if [ ${ret} -ne 0 ]; then
+ echo "FAIL: ${msg}" 1>&2
+ display_time
+ exit ${ret}
+ fi
+}
+
make_file "$cin" "client"
make_file "$sin" "server"
check_mptcp_ulp_setsockopt
+stop_if_error "The kernel configuration is not valid for MPTCP"
+
echo "INFO: validating network environment with pings"
for sender in "$ns1" "$ns2" "$ns3" "$ns4";do
do_ping "$ns1" $sender 10.0.1.1
do_ping "$ns4" $sender dead:beef:3::1
done
+stop_if_error "Could not even run ping tests"
+
[ -n "$tc_loss" ] && tc -net "$ns2" qdisc add dev ns2eth3 root netem loss random $tc_loss delay ${tc_delay}ms
echo -n "INFO: Using loss of $tc_loss "
test "$tc_delay" -gt 0 && echo -n "delay $tc_delay ms "
tc -net "$ns3" qdisc add dev ns3eth4 root netem delay ${reorder_delay}ms $tc_reorder
+run_tests_lo "$ns1" "$ns1" 10.0.1.1 1
+stop_if_error "Could not even run loopback test"
+
+run_tests_lo "$ns1" "$ns1" dead:beef:1::1 1
+stop_if_error "Could not even run loopback v6 test"
+
for sender in $ns1 $ns2 $ns3 $ns4;do
- run_tests_lo "$ns1" "$sender" 10.0.1.1 1
- if [ $ret -ne 0 ] ;then
- echo "FAIL: Could not even run loopback test" 1>&2
- exit $ret
- fi
- run_tests_lo "$ns1" $sender dead:beef:1::1 1
- if [ $ret -ne 0 ] ;then
- echo "FAIL: Could not even run loopback v6 test" 2>&1
- exit $ret
+ # ns1<->ns2 is not subject to reordering/tc delays. Use it to test
+ # mptcp syncookie support.
+ if [ $sender = $ns1 ]; then
+ ip netns exec "$ns2" sysctl -q net.ipv4.tcp_syncookies=2
+ else
+ ip netns exec "$ns2" sysctl -q net.ipv4.tcp_syncookies=1
fi
+ run_tests "$ns1" $sender 10.0.1.1
+ run_tests "$ns1" $sender dead:beef:1::1
+
run_tests "$ns2" $sender 10.0.1.2
run_tests "$ns2" $sender dead:beef:1::2
run_tests "$ns2" $sender 10.0.2.1
run_tests "$ns4" $sender 10.0.3.1
run_tests "$ns4" $sender dead:beef:3::1
+
+ stop_if_error "Tests with $sender as a sender have failed"
done
run_tests_peekmode "saveWithPeek"
run_tests_peekmode "saveAfterPeek"
+stop_if_error "Tests with peek mode have failed"
-time_end=$(date +%s)
-time_run=$((time_end-time_start))
-
-echo "Time: ${time_run} seconds"
-
+display_time
exit $ret
timeout_test=$((timeout_poll * 2 + 1))
mptcp_connect=""
capture=0
+checksum=0
do_all_tests=1
TEST_COUNT=0
ip netns exec $netns sysctl -q net.mptcp.enabled=1
ip netns exec $netns sysctl -q net.ipv4.conf.all.rp_filter=0
ip netns exec $netns sysctl -q net.ipv4.conf.default.rp_filter=0
+ if [ $checksum -eq 1 ]; then
+ ip netns exec $netns sysctl -q net.mptcp.checksum_enabled=1
+ fi
done
# ns1 ns2
-j DROP
}
+reset_with_checksum()
+{
+ local ns1_enable=$1
+ local ns2_enable=$2
+
+ reset
+
+ ip netns exec $ns1 sysctl -q net.mptcp.checksum_enabled=$ns1_enable
+ ip netns exec $ns2 sysctl -q net.mptcp.checksum_enabled=$ns2_enable
+}
+
+reset_with_allow_join_id0()
+{
+ local ns1_enable=$1
+ local ns2_enable=$2
+
+ reset
+
+ ip netns exec $ns1 sysctl -q net.mptcp.allow_join_initial_addr_port=$ns1_enable
+ ip netns exec $ns2 sysctl -q net.mptcp.allow_join_initial_addr_port=$ns2_enable
+}
+
ip -Version > /dev/null 2>&1
if [ $? -ne 0 ];then
echo "SKIP: Could not run test without ip tool"
fi
}
+chk_csum_nr()
+{
+ local msg=${1:-""}
+ local count
+ local dump_stats
+
+ if [ ! -z "$msg" ]; then
+ printf "%02u" "$TEST_COUNT"
+ else
+ echo -n " "
+ fi
+ printf " %-36s %s" "$msg" "sum"
+ count=`ip netns exec $ns1 nstat -as | grep MPTcpExtDataCsumErr | awk '{print $2}'`
+ [ -z "$count" ] && count=0
+ if [ "$count" != 0 ]; then
+ echo "[fail] got $count data checksum error[s] expected 0"
+ ret=1
+ dump_stats=1
+ else
+ echo -n "[ ok ]"
+ fi
+ echo -n " - csum "
+ count=`ip netns exec $ns2 nstat -as | grep MPTcpExtDataCsumErr | awk '{print $2}'`
+ [ -z "$count" ] && count=0
+ if [ "$count" != 0 ]; then
+ echo "[fail] got $count data checksum error[s] expected 0"
+ ret=1
+ dump_stats=1
+ else
+ echo "[ ok ]"
+ fi
+ if [ "${dump_stats}" = 1 ]; then
+ echo Server ns stats
+ ip netns exec $ns1 nstat -as | grep MPTcp
+ echo Client ns stats
+ ip netns exec $ns2 nstat -as | grep MPTcp
+ fi
+}
+
chk_join_nr()
{
local msg="$1"
echo Client ns stats
ip netns exec $ns2 nstat -as | grep MPTcp
fi
+ if [ $checksum -eq 1 ]; then
+ chk_csum_nr
+ fi
}
chk_add_nr()
chk_add_nr 1 1
}
+checksum_tests()
+{
+ # checksum test 0 0
+ reset_with_checksum 0 0
+ ip netns exec $ns1 ./pm_nl_ctl limits 0 1
+ ip netns exec $ns2 ./pm_nl_ctl limits 0 1
+ run_tests $ns1 $ns2 10.0.1.1
+ chk_csum_nr "checksum test 0 0"
+
+ # checksum test 1 1
+ reset_with_checksum 1 1
+ ip netns exec $ns1 ./pm_nl_ctl limits 0 1
+ ip netns exec $ns2 ./pm_nl_ctl limits 0 1
+ run_tests $ns1 $ns2 10.0.1.1
+ chk_csum_nr "checksum test 1 1"
+
+ # checksum test 0 1
+ reset_with_checksum 0 1
+ ip netns exec $ns1 ./pm_nl_ctl limits 0 1
+ ip netns exec $ns2 ./pm_nl_ctl limits 0 1
+ run_tests $ns1 $ns2 10.0.1.1
+ chk_csum_nr "checksum test 0 1"
+
+ # checksum test 1 0
+ reset_with_checksum 1 0
+ ip netns exec $ns1 ./pm_nl_ctl limits 0 1
+ ip netns exec $ns2 ./pm_nl_ctl limits 0 1
+ run_tests $ns1 $ns2 10.0.1.1
+ chk_csum_nr "checksum test 1 0"
+}
+
+deny_join_id0_tests()
+{
+ # subflow allow join id0 ns1
+ reset_with_allow_join_id0 1 0
+ ip netns exec $ns1 ./pm_nl_ctl limits 1 1
+ ip netns exec $ns2 ./pm_nl_ctl limits 1 1
+ ip netns exec $ns2 ./pm_nl_ctl add 10.0.3.2 flags subflow
+ run_tests $ns1 $ns2 10.0.1.1
+ chk_join_nr "single subflow allow join id0 ns1" 1 1 1
+
+ # subflow allow join id0 ns2
+ reset_with_allow_join_id0 0 1
+ ip netns exec $ns1 ./pm_nl_ctl limits 1 1
+ ip netns exec $ns2 ./pm_nl_ctl limits 1 1
+ ip netns exec $ns2 ./pm_nl_ctl add 10.0.3.2 flags subflow
+ run_tests $ns1 $ns2 10.0.1.1
+ chk_join_nr "single subflow allow join id0 ns2" 0 0 0
+
+ # signal address allow join id0 ns1
+ # ADD_ADDRs are not affected by allow_join_id0 value.
+ reset_with_allow_join_id0 1 0
+ ip netns exec $ns1 ./pm_nl_ctl limits 1 1
+ ip netns exec $ns2 ./pm_nl_ctl limits 1 1
+ ip netns exec $ns1 ./pm_nl_ctl add 10.0.2.1 flags signal
+ run_tests $ns1 $ns2 10.0.1.1
+ chk_join_nr "signal address allow join id0 ns1" 1 1 1
+ chk_add_nr 1 1
+
+ # signal address allow join id0 ns2
+ # ADD_ADDRs are not affected by allow_join_id0 value.
+ reset_with_allow_join_id0 0 1
+ ip netns exec $ns1 ./pm_nl_ctl limits 1 1
+ ip netns exec $ns2 ./pm_nl_ctl limits 1 1
+ ip netns exec $ns1 ./pm_nl_ctl add 10.0.2.1 flags signal
+ run_tests $ns1 $ns2 10.0.1.1
+ chk_join_nr "signal address allow join id0 ns2" 1 1 1
+ chk_add_nr 1 1
+
+ # subflow and address allow join id0 ns1
+ reset_with_allow_join_id0 1 0
+ ip netns exec $ns1 ./pm_nl_ctl limits 2 2
+ ip netns exec $ns2 ./pm_nl_ctl limits 2 2
+ ip netns exec $ns1 ./pm_nl_ctl add 10.0.2.1 flags signal
+ ip netns exec $ns2 ./pm_nl_ctl add 10.0.3.2 flags subflow
+ run_tests $ns1 $ns2 10.0.1.1
+ chk_join_nr "subflow and address allow join id0 1" 2 2 2
+
+ # subflow and address allow join id0 ns2
+ reset_with_allow_join_id0 0 1
+ ip netns exec $ns1 ./pm_nl_ctl limits 2 2
+ ip netns exec $ns2 ./pm_nl_ctl limits 2 2
+ ip netns exec $ns1 ./pm_nl_ctl add 10.0.2.1 flags signal
+ ip netns exec $ns2 ./pm_nl_ctl add 10.0.3.2 flags subflow
+ run_tests $ns1 $ns2 10.0.1.1
+ chk_join_nr "subflow and address allow join id0 2" 1 1 1
+}
+
all_tests()
{
subflows_tests
backup_tests
add_addr_ports_tests
syncookies_tests
+ checksum_tests
+ deny_join_id0_tests
}
usage()
echo " -b backup_tests"
echo " -p add_addr_ports_tests"
echo " -k syncookies_tests"
+ echo " -S checksum_tests"
+ echo " -d deny_join_id0_tests"
echo " -c capture pcap files"
+ echo " -C enable data checksum"
echo " -h help"
}
trap cleanup EXIT
for arg in "$@"; do
- # check for "capture" arg before launching tests
+ # check for "capture/checksum" args before launching tests
if [[ "${arg}" =~ ^"-"[0-9a-zA-Z]*"c"[0-9a-zA-Z]*$ ]]; then
capture=1
fi
+ if [[ "${arg}" =~ ^"-"[0-9a-zA-Z]*"C"[0-9a-zA-Z]*$ ]]; then
+ checksum=1
+ fi
- # exception for the capture option, the rest means: a part of the tests
- if [ "${arg}" != "-c" ]; then
+ # exception for the capture/checksum options, the rest means: a part of the tests
+ if [ "${arg}" != "-c" ] && [ "${arg}" != "-C" ]; then
do_all_tests=0
fi
done
exit $ret
fi
-while getopts 'fsltra64bpkch' opt; do
+while getopts 'fsltra64bpkdchCS' opt; do
case $opt in
f)
subflows_tests
k)
syncookies_tests
;;
+ S)
+ checksum_tests
+ ;;
+ d)
+ deny_join_id0_tests
+ ;;
c)
;;
+ C)
+ ;;
h | *)
usage
;;
timeout ${timeout_test} \
ip netns exec ${listener_ns} \
- $mptcp_connect -t ${timeout_poll} -l -M 1 -p $port -s ${srv_proto} \
+ $mptcp_connect -t ${timeout_poll} -l -M 1 -p $port -s ${srv_proto} -c TIMESTAMPNS \
${local_addr} < "$sin" > "$sout" &
spid=$!
timeout ${timeout_test} \
ip netns exec ${connector_ns} \
- $mptcp_connect -t ${timeout_poll} -M 2 -p $port -s ${cl_proto} \
+ $mptcp_connect -t ${timeout_poll} -M 2 -p $port -s ${cl_proto} -c TIMESTAMPNS \
$connect_addr < "$cin" > "$cout" &
cpid=$!
for i in "$ns1" "$ns2" "$ns3";do
ip netns add $i || exit $ksft_skip
ip -net $i link set lo up
+ ip netns exec $i sysctl -q net.ipv4.conf.all.rp_filter=0
+ ip netns exec $i sysctl -q net.ipv4.conf.default.rp_filter=0
done
ip link add ns1eth1 netns "$ns1" type veth peer name ns2eth1 netns "$ns2"
ip netns exec "$ns1" ./pm_nl_ctl limits 1 1
ip netns exec "$ns1" ./pm_nl_ctl add 10.0.2.1 dev ns1eth2 flags subflow
- ip netns exec "$ns1" sysctl -q net.ipv4.conf.all.rp_filter=0
ip -net "$ns2" addr add 10.0.1.2/24 dev ns2eth1
ip -net "$ns2" addr add dead:beef:1::2/64 dev ns2eth1 nodad
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+#include <sched.h>
+#include <unistd.h>
+#include <stdio.h>
+#include <errno.h>
+#include <string.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <sys/types.h>
+#include <sys/socket.h>
+
+#ifndef SO_NETNS_COOKIE
+#define SO_NETNS_COOKIE 71
+#endif
+
+#define pr_err(fmt, ...) \
+ ({ \
+ fprintf(stderr, "%s:%d:" fmt ": %m\n", \
+ __func__, __LINE__, ##__VA_ARGS__); \
+ 1; \
+ })
+
+int main(int argc, char *argvp[])
+{
+ uint64_t cookie1, cookie2;
+ socklen_t vallen;
+ int sock1, sock2;
+
+ sock1 = socket(AF_INET, SOCK_STREAM, 0);
+ if (sock1 < 0)
+ return pr_err("Unable to create TCP socket");
+
+ vallen = sizeof(cookie1);
+ if (getsockopt(sock1, SOL_SOCKET, SO_NETNS_COOKIE, &cookie1, &vallen) != 0)
+ return pr_err("getsockopt(SOL_SOCKET, SO_NETNS_COOKIE)");
+
+ if (!cookie1)
+ return pr_err("SO_NETNS_COOKIE returned zero cookie");
+
+ if (unshare(CLONE_NEWNET))
+ return pr_err("unshare");
+
+ sock2 = socket(AF_INET, SOCK_STREAM, 0);
+ if (sock2 < 0)
+ return pr_err("Unable to create TCP socket");
+
+ vallen = sizeof(cookie2);
+ if (getsockopt(sock2, SOL_SOCKET, SO_NETNS_COOKIE, &cookie2, &vallen) != 0)
+ return pr_err("getsockopt(SOL_SOCKET, SO_NETNS_COOKIE)");
+
+ if (!cookie2)
+ return pr_err("SO_NETNS_COOKIE returned zero cookie");
+
+ if (cookie1 == cookie2)
+ return pr_err("SO_NETNS_COOKIE returned identical cookies for distinct ns");
+
+ close(sock1);
+ close(sock2);
+ return 0;
+}
--- /dev/null
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# author: Andrea Mayer <andrea.mayer@uniroma2.it>
+# author: Paolo Lungaroni <paolo.lungaroni@uniroma2.it>
+
+# This test is designed for evaluating the new SRv6 End.DT46 Behavior used for
+# implementing IPv4/IPv6 L3 VPN use cases.
+#
+# The current SRv6 code in the Linux kernel only implements SRv6 End.DT4 and
+# End.DT6 Behaviors which can be used respectively to support IPv4-in-IPv6 and
+# IPv6-in-IPv6 VPNs. With End.DT4 and End.DT6 it is not possible to create a
+# single SRv6 VPN tunnel to carry both IPv4 and IPv6 traffic.
+# The SRv6 End.DT46 Behavior implementation is meant to support the
+# decapsulation of IPv4 and IPv6 traffic coming from a single SRv6 tunnel.
+# Therefore, the SRv6 End.DT46 Behavior in the Linux kernel greatly simplifies
+# the setup and operations of SRv6 VPNs.
+#
+# Hereafter a network diagram is shown, where two different tenants (named 100
+# and 200) offer IPv4/IPv6 L3 VPN services allowing hosts to communicate with
+# each other across an IPv6 network.
+#
+# Only hosts belonging to the same tenant (and to the same VPN) can communicate
+# with each other. Instead, the communication among hosts of different tenants
+# is forbidden.
+# In other words, hosts hs-t100-1 and hs-t100-2 are connected through the
+# IPv4/IPv6 L3 VPN of tenant 100 while hs-t200-3 and hs-t200-4 are connected
+# using the IPv4/IPv6 L3 VPN of tenant 200. Cross connection between tenant 100
+# and tenant 200 is forbidden and thus, for example, hs-t100-1 cannot reach
+# hs-t200-3 and vice versa.
+#
+# Routers rt-1 and rt-2 implement IPv4/IPv6 L3 VPN services leveraging the SRv6
+# architecture. The key components for such VPNs are: a) SRv6 Encap behavior,
+# b) SRv6 End.DT46 Behavior and c) VRF.
+#
+# To explain how an IPv4/IPv6 L3 VPN based on SRv6 works, let us briefly
+# consider an example where, within the same domain of tenant 100, the host
+# hs-t100-1 pings the host hs-t100-2.
+#
+# First of all, L2 reachability of the host hs-t100-2 is taken into account by
+# the router rt-1 which acts as a arp/ndp proxy.
+#
+# When the host hs-t100-1 sends an IPv6 or IPv4 packet destined to hs-t100-2,
+# the router rt-1 receives the packet on the internal veth-t100 interface. Such
+# interface is enslaved to the VRF vrf-100 whose associated table contains the
+# SRv6 Encap route for encapsulating any IPv6 or IPv4 packet in a IPv6 plus the
+# Segment Routing Header (SRH) packet. This packet is sent through the (IPv6)
+# core network up to the router rt-2 that receives it on veth0 interface.
+#
+# The rt-2 router uses the 'localsid' routing table to process incoming
+# IPv6+SRH packets which belong to the VPN of the tenant 100. For each of these
+# packets, the SRv6 End.DT46 Behavior removes the outer IPv6+SRH headers and
+# performs the lookup on the vrf-100 table using the destination address of
+# the decapsulated IPv6 or IPv4 packet. Afterwards, the packet is sent to the
+# host hs-t100-2 through the veth-t100 interface.
+#
+# The ping response follows the same processing but this time the roles of rt-1
+# and rt-2 are swapped.
+#
+# Of course, the IPv4/IPv6 L3 VPN for tenant 200 works exactly as the IPv4/IPv6
+# L3 VPN for tenant 100. In this case, only hosts hs-t200-3 and hs-t200-4 are
+# able to connect with each other.
+#
+#
+# +-------------------+ +-------------------+
+# | | | |
+# | hs-t100-1 netns | | hs-t100-2 netns |
+# | | | |
+# | +-------------+ | | +-------------+ |
+# | | veth0 | | | | veth0 | |
+# | | cafe::1/64 | | | | cafe::2/64 | |
+# | | 10.0.0.1/24 | | | | 10.0.0.2/24 | |
+# | +-------------+ | | +-------------+ |
+# | . | | . |
+# +-------------------+ +-------------------+
+# . .
+# . .
+# . .
+# +-----------------------------------+ +-----------------------------------+
+# | . | | . |
+# | +---------------+ | | +---------------- |
+# | | veth-t100 | | | | veth-t100 | |
+# | | cafe::254/64 | | | | cafe::254/64 | |
+# | | 10.0.0.254/24 | +----------+ | | +----------+ | 10.0.0.254/24 | |
+# | +-------+-------+ | localsid | | | | localsid | +-------+-------- |
+# | | | table | | | | table | | |
+# | +----+----+ +----------+ | | +----------+ +----+----+ |
+# | | vrf-100 | | | | vrf-100 | |
+# | +---------+ +------------+ | | +------------+ +---------+ |
+# | | veth0 | | | | veth0 | |
+# | | fd00::1/64 |.|...|.| fd00::2/64 | |
+# | +---------+ +------------+ | | +------------+ +---------+ |
+# | | vrf-200 | | | | vrf-200 | |
+# | +----+----+ | | +----+----+ |
+# | | | | | |
+# | +-------+-------+ | | +-------+-------- |
+# | | veth-t200 | | | | veth-t200 | |
+# | | cafe::254/64 | | | | cafe::254/64 | |
+# | | 10.0.0.254/24 | | | | 10.0.0.254/24 | |
+# | +---------------+ rt-1 netns | | rt-2 netns +---------------- |
+# | . | | . |
+# +-----------------------------------+ +-----------------------------------+
+# . .
+# . .
+# . .
+# . .
+# +-------------------+ +-------------------+
+# | . | | . |
+# | +-------------+ | | +-------------+ |
+# | | veth0 | | | | veth0 | |
+# | | cafe::3/64 | | | | cafe::4/64 | |
+# | | 10.0.0.3/24 | | | | 10.0.0.4/24 | |
+# | +-------------+ | | +-------------+ |
+# | | | |
+# | hs-t200-3 netns | | hs-t200-4 netns |
+# | | | |
+# +-------------------+ +-------------------+
+#
+#
+# ~~~~~~~~~~~~~~~~~~~~~~~~~
+# | Network configuration |
+# ~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# rt-1: localsid table (table 90)
+# +--------------------------------------------------+
+# |SID |Action |
+# +--------------------------------------------------+
+# |fc00:21:100::6046|apply SRv6 End.DT46 vrftable 100|
+# +--------------------------------------------------+
+# |fc00:21:200::6046|apply SRv6 End.DT46 vrftable 200|
+# +--------------------------------------------------+
+#
+# rt-1: VRF tenant 100 (table 100)
+# +---------------------------------------------------+
+# |host |Action |
+# +---------------------------------------------------+
+# |cafe::2 |apply seg6 encap segs fc00:12:100::6046|
+# +---------------------------------------------------+
+# |cafe::/64 |forward to dev veth-t100 |
+# +---------------------------------------------------+
+# |10.0.0.2 |apply seg6 encap segs fc00:12:100::6046|
+# +---------------------------------------------------+
+# |10.0.0.0/24|forward to dev veth-t100 |
+# +---------------------------------------------------+
+#
+# rt-1: VRF tenant 200 (table 200)
+# +---------------------------------------------------+
+# |host |Action |
+# +---------------------------------------------------+
+# |cafe::4 |apply seg6 encap segs fc00:12:200::6046|
+# +---------------------------------------------------+
+# |cafe::/64 |forward to dev veth-t200 |
+# +---------------------------------------------------+
+# |10.0.0.4 |apply seg6 encap segs fc00:12:200::6046|
+# +---------------------------------------------------+
+# |10.0.0.0/24|forward to dev veth-t200 |
+# +---------------------------------------------------+
+#
+#
+# rt-2: localsid table (table 90)
+# +--------------------------------------------------+
+# |SID |Action |
+# +--------------------------------------------------+
+# |fc00:12:100::6046|apply SRv6 End.DT46 vrftable 100|
+# +--------------------------------------------------+
+# |fc00:12:200::6046|apply SRv6 End.DT46 vrftable 200|
+# +--------------------------------------------------+
+#
+# rt-2: VRF tenant 100 (table 100)
+# +---------------------------------------------------+
+# |host |Action |
+# +---------------------------------------------------+
+# |cafe::1 |apply seg6 encap segs fc00:21:100::6046|
+# +---------------------------------------------------+
+# |cafe::/64 |forward to dev veth-t100 |
+# +---------------------------------------------------+
+# |10.0.0.1 |apply seg6 encap segs fc00:21:100::6046|
+# +---------------------------------------------------+
+# |10.0.0.0/24|forward to dev veth-t100 |
+# +---------------------------------------------------+
+#
+# rt-2: VRF tenant 200 (table 200)
+# +---------------------------------------------------+
+# |host |Action |
+# +---------------------------------------------------+
+# |cafe::3 |apply seg6 encap segs fc00:21:200::6046|
+# +---------------------------------------------------+
+# |cafe::/64 |forward to dev veth-t200 |
+# +---------------------------------------------------+
+# |10.0.0.3 |apply seg6 encap segs fc00:21:200::6046|
+# +---------------------------------------------------+
+# |10.0.0.0/24|forward to dev veth-t200 |
+# +---------------------------------------------------+
+#
+
+readonly LOCALSID_TABLE_ID=90
+readonly IPv6_RT_NETWORK=fd00
+readonly IPv6_HS_NETWORK=cafe
+readonly IPv4_HS_NETWORK=10.0.0
+readonly VPN_LOCATOR_SERVICE=fc00
+PING_TIMEOUT_SEC=4
+
+ret=0
+
+PAUSE_ON_FAIL=${PAUSE_ON_FAIL:=no}
+
+log_test()
+{
+ local rc=$1
+ local expected=$2
+ local msg="$3"
+
+ if [ ${rc} -eq ${expected} ]; then
+ nsuccess=$((nsuccess+1))
+ printf "\n TEST: %-60s [ OK ]\n" "${msg}"
+ else
+ ret=1
+ nfail=$((nfail+1))
+ printf "\n TEST: %-60s [FAIL]\n" "${msg}"
+ if [ "${PAUSE_ON_FAIL}" = "yes" ]; then
+ echo
+ echo "hit enter to continue, 'q' to quit"
+ read a
+ [ "$a" = "q" ] && exit 1
+ fi
+ fi
+}
+
+print_log_test_results()
+{
+ if [ "$TESTS" != "none" ]; then
+ printf "\nTests passed: %3d\n" ${nsuccess}
+ printf "Tests failed: %3d\n" ${nfail}
+ fi
+}
+
+log_section()
+{
+ echo
+ echo "################################################################################"
+ echo "TEST SECTION: $*"
+ echo "################################################################################"
+}
+
+cleanup()
+{
+ ip link del veth-rt-1 2>/dev/null || true
+ ip link del veth-rt-2 2>/dev/null || true
+
+ # destroy routers rt-* and hosts hs-*
+ for ns in $(ip netns show | grep -E 'rt-*|hs-*'); do
+ ip netns del ${ns} || true
+ done
+}
+
+# Setup the basic networking for the routers
+setup_rt_networking()
+{
+ local rt=$1
+ local nsname=rt-${rt}
+
+ ip netns add ${nsname}
+ ip link set veth-rt-${rt} netns ${nsname}
+ ip -netns ${nsname} link set veth-rt-${rt} name veth0
+
+ ip netns exec ${nsname} sysctl -wq net.ipv6.conf.all.accept_dad=0
+ ip netns exec ${nsname} sysctl -wq net.ipv6.conf.default.accept_dad=0
+
+ ip -netns ${nsname} addr add ${IPv6_RT_NETWORK}::${rt}/64 dev veth0 nodad
+ ip -netns ${nsname} link set veth0 up
+ ip -netns ${nsname} link set lo up
+
+ ip netns exec ${nsname} sysctl -wq net.ipv4.ip_forward=1
+ ip netns exec ${nsname} sysctl -wq net.ipv6.conf.all.forwarding=1
+}
+
+setup_hs()
+{
+ local hs=$1
+ local rt=$2
+ local tid=$3
+ local hsname=hs-t${tid}-${hs}
+ local rtname=rt-${rt}
+ local rtveth=veth-t${tid}
+
+ # set the networking for the host
+ ip netns add ${hsname}
+
+ ip netns exec ${hsname} sysctl -wq net.ipv6.conf.all.accept_dad=0
+ ip netns exec ${hsname} sysctl -wq net.ipv6.conf.default.accept_dad=0
+
+ ip -netns ${hsname} link add veth0 type veth peer name ${rtveth}
+ ip -netns ${hsname} link set ${rtveth} netns ${rtname}
+ ip -netns ${hsname} addr add ${IPv6_HS_NETWORK}::${hs}/64 dev veth0 nodad
+ ip -netns ${hsname} addr add ${IPv4_HS_NETWORK}.${hs}/24 dev veth0
+ ip -netns ${hsname} link set veth0 up
+ ip -netns ${hsname} link set lo up
+
+ # configure the VRF for the tenant X on the router which is directly
+ # connected to the source host.
+ ip -netns ${rtname} link add vrf-${tid} type vrf table ${tid}
+ ip -netns ${rtname} link set vrf-${tid} up
+
+ ip netns exec ${rtname} sysctl -wq net.ipv6.conf.all.accept_dad=0
+ ip netns exec ${rtname} sysctl -wq net.ipv6.conf.default.accept_dad=0
+
+ # enslave the veth-tX interface to the vrf-X in the access router
+ ip -netns ${rtname} link set ${rtveth} master vrf-${tid}
+ ip -netns ${rtname} addr add ${IPv6_HS_NETWORK}::254/64 dev ${rtveth} nodad
+ ip -netns ${rtname} addr add ${IPv4_HS_NETWORK}.254/24 dev ${rtveth}
+ ip -netns ${rtname} link set ${rtveth} up
+
+ ip netns exec ${rtname} sysctl -wq net.ipv6.conf.${rtveth}.proxy_ndp=1
+ ip netns exec ${rtname} sysctl -wq net.ipv4.conf.${rtveth}.proxy_arp=1
+
+ # disable the rp_filter otherwise the kernel gets confused about how
+ # to route decap ipv4 packets.
+ ip netns exec ${rtname} sysctl -wq net.ipv4.conf.all.rp_filter=0
+ ip netns exec ${rtname} sysctl -wq net.ipv4.conf.${rtveth}.rp_filter=0
+
+ ip netns exec ${rtname} sh -c "echo 1 > /proc/sys/net/vrf/strict_mode"
+}
+
+setup_vpn_config()
+{
+ local hssrc=$1
+ local rtsrc=$2
+ local hsdst=$3
+ local rtdst=$4
+ local tid=$5
+
+ local hssrc_name=hs-t${tid}-${hssrc}
+ local hsdst_name=hs-t${tid}-${hsdst}
+ local rtsrc_name=rt-${rtsrc}
+ local rtdst_name=rt-${rtdst}
+ local rtveth=veth-t${tid}
+ local vpn_sid=${VPN_LOCATOR_SERVICE}:${hssrc}${hsdst}:${tid}::6046
+
+ ip -netns ${rtsrc_name} -6 neigh add proxy ${IPv6_HS_NETWORK}::${hsdst} dev ${rtveth}
+
+ # set the encap route for encapsulating packets which arrive from the
+ # host hssrc and destined to the access router rtsrc.
+ ip -netns ${rtsrc_name} -6 route add ${IPv6_HS_NETWORK}::${hsdst}/128 vrf vrf-${tid} \
+ encap seg6 mode encap segs ${vpn_sid} dev veth0
+ ip -netns ${rtsrc_name} -4 route add ${IPv4_HS_NETWORK}.${hsdst}/32 vrf vrf-${tid} \
+ encap seg6 mode encap segs ${vpn_sid} dev veth0
+ ip -netns ${rtsrc_name} -6 route add ${vpn_sid}/128 vrf vrf-${tid} \
+ via fd00::${rtdst} dev veth0
+
+ # set the decap route for decapsulating packets which arrive from
+ # the rtdst router and destined to the hsdst host.
+ ip -netns ${rtdst_name} -6 route add ${vpn_sid}/128 table ${LOCALSID_TABLE_ID} \
+ encap seg6local action End.DT46 vrftable ${tid} dev vrf-${tid}
+
+ # all sids for VPNs start with a common locator which is fc00::/16.
+ # Routes for handling the SRv6 End.DT46 behavior instances are grouped
+ # together in the 'localsid' table.
+ #
+ # NOTE: added only once
+ if [ -z "$(ip -netns ${rtdst_name} -6 rule show | \
+ grep "to ${VPN_LOCATOR_SERVICE}::/16 lookup ${LOCALSID_TABLE_ID}")" ]; then
+ ip -netns ${rtdst_name} -6 rule add \
+ to ${VPN_LOCATOR_SERVICE}::/16 \
+ lookup ${LOCALSID_TABLE_ID} prio 999
+ fi
+
+ # set default routes to unreachable for both ipv4 and ipv6
+ ip -netns ${rtsrc_name} -6 route add unreachable default metric 4278198272 \
+ vrf vrf-${tid}
+
+ ip -netns ${rtsrc_name} -4 route add unreachable default metric 4278198272 \
+ vrf vrf-${tid}
+}
+
+setup()
+{
+ ip link add veth-rt-1 type veth peer name veth-rt-2
+ # setup the networking for router rt-1 and router rt-2
+ setup_rt_networking 1
+ setup_rt_networking 2
+
+ # setup two hosts for the tenant 100.
+ # - host hs-1 is directly connected to the router rt-1;
+ # - host hs-2 is directly connected to the router rt-2.
+ setup_hs 1 1 100 #args: host router tenant
+ setup_hs 2 2 100
+
+ # setup two hosts for the tenant 200
+ # - host hs-3 is directly connected to the router rt-1;
+ # - host hs-4 is directly connected to the router rt-2.
+ setup_hs 3 1 200
+ setup_hs 4 2 200
+
+ # setup the IPv4/IPv6 L3 VPN which connects the host hs-t100-1 and host
+ # hs-t100-2 within the same tenant 100.
+ setup_vpn_config 1 1 2 2 100 #args: src_host src_router dst_host dst_router tenant
+ setup_vpn_config 2 2 1 1 100
+
+ # setup the IPv4/IPv6 L3 VPN which connects the host hs-t200-3 and host
+ # hs-t200-4 within the same tenant 200.
+ setup_vpn_config 3 1 4 2 200
+ setup_vpn_config 4 2 3 1 200
+}
+
+check_rt_connectivity()
+{
+ local rtsrc=$1
+ local rtdst=$2
+
+ ip netns exec rt-${rtsrc} ping -c 1 -W 1 ${IPv6_RT_NETWORK}::${rtdst} \
+ >/dev/null 2>&1
+}
+
+check_and_log_rt_connectivity()
+{
+ local rtsrc=$1
+ local rtdst=$2
+
+ check_rt_connectivity ${rtsrc} ${rtdst}
+ log_test $? 0 "Routers connectivity: rt-${rtsrc} -> rt-${rtdst}"
+}
+
+check_hs_ipv6_connectivity()
+{
+ local hssrc=$1
+ local hsdst=$2
+ local tid=$3
+
+ ip netns exec hs-t${tid}-${hssrc} ping -c 1 -W ${PING_TIMEOUT_SEC} \
+ ${IPv6_HS_NETWORK}::${hsdst} >/dev/null 2>&1
+}
+
+check_hs_ipv4_connectivity()
+{
+ local hssrc=$1
+ local hsdst=$2
+ local tid=$3
+
+ ip netns exec hs-t${tid}-${hssrc} ping -c 1 -W ${PING_TIMEOUT_SEC} \
+ ${IPv4_HS_NETWORK}.${hsdst} >/dev/null 2>&1
+}
+
+check_and_log_hs_connectivity()
+{
+ local hssrc=$1
+ local hsdst=$2
+ local tid=$3
+
+ check_hs_ipv6_connectivity ${hssrc} ${hsdst} ${tid}
+ log_test $? 0 "IPv6 Hosts connectivity: hs-t${tid}-${hssrc} -> hs-t${tid}-${hsdst} (tenant ${tid})"
+
+ check_hs_ipv4_connectivity ${hssrc} ${hsdst} ${tid}
+ log_test $? 0 "IPv4 Hosts connectivity: hs-t${tid}-${hssrc} -> hs-t${tid}-${hsdst} (tenant ${tid})"
+
+}
+
+check_and_log_hs_isolation()
+{
+ local hssrc=$1
+ local tidsrc=$2
+ local hsdst=$3
+ local tiddst=$4
+
+ check_hs_ipv6_connectivity ${hssrc} ${hsdst} ${tidsrc}
+ # NOTE: ping should fail
+ log_test $? 1 "IPv6 Hosts isolation: hs-t${tidsrc}-${hssrc} -X-> hs-t${tiddst}-${hsdst}"
+
+ check_hs_ipv4_connectivity ${hssrc} ${hsdst} ${tidsrc}
+ # NOTE: ping should fail
+ log_test $? 1 "IPv4 Hosts isolation: hs-t${tidsrc}-${hssrc} -X-> hs-t${tiddst}-${hsdst}"
+
+}
+
+
+check_and_log_hs2gw_connectivity()
+{
+ local hssrc=$1
+ local tid=$2
+
+ check_hs_ipv6_connectivity ${hssrc} 254 ${tid}
+ log_test $? 0 "IPv6 Hosts connectivity: hs-t${tid}-${hssrc} -> gw (tenant ${tid})"
+
+ check_hs_ipv4_connectivity ${hssrc} 254 ${tid}
+ log_test $? 0 "IPv4 Hosts connectivity: hs-t${tid}-${hssrc} -> gw (tenant ${tid})"
+
+}
+
+router_tests()
+{
+ log_section "IPv6 routers connectivity test"
+
+ check_and_log_rt_connectivity 1 2
+ check_and_log_rt_connectivity 2 1
+}
+
+host2gateway_tests()
+{
+ log_section "IPv4/IPv6 connectivity test among hosts and gateway"
+
+ check_and_log_hs2gw_connectivity 1 100
+ check_and_log_hs2gw_connectivity 2 100
+
+ check_and_log_hs2gw_connectivity 3 200
+ check_and_log_hs2gw_connectivity 4 200
+}
+
+host_vpn_tests()
+{
+ log_section "SRv6 VPN connectivity test among hosts in the same tenant"
+
+ check_and_log_hs_connectivity 1 2 100
+ check_and_log_hs_connectivity 2 1 100
+
+ check_and_log_hs_connectivity 3 4 200
+ check_and_log_hs_connectivity 4 3 200
+}
+
+host_vpn_isolation_tests()
+{
+ local i
+ local j
+ local k
+ local tmp
+ local l1="1 2"
+ local l2="3 4"
+ local t1=100
+ local t2=200
+
+ log_section "SRv6 VPN isolation test among hosts in different tentants"
+
+ for k in 0 1; do
+ for i in ${l1}; do
+ for j in ${l2}; do
+ check_and_log_hs_isolation ${i} ${t1} ${j} ${t2}
+ done
+ done
+
+ # let us test the reverse path
+ tmp="${l1}"; l1="${l2}"; l2="${tmp}"
+ tmp=${t1}; t1=${t2}; t2=${tmp}
+ done
+}
+
+if [ "$(id -u)" -ne 0 ];then
+ echo "SKIP: Need root privileges"
+ exit 0
+fi
+
+if [ ! -x "$(command -v ip)" ]; then
+ echo "SKIP: Could not run test without ip tool"
+ exit 0
+fi
+
+modprobe vrf &>/dev/null
+if [ ! -e /proc/sys/net/vrf/strict_mode ]; then
+ echo "SKIP: vrf sysctl does not exist"
+ exit 0
+fi
+
+cleanup &>/dev/null
+
+setup
+
+router_tests
+host2gateway_tests
+host_vpn_tests
+host_vpn_isolation_tests
+
+print_log_test_results
+
+cleanup &>/dev/null
+
+exit ${ret}
#define TLS_PAYLOAD_MAX_LEN 16384
#define SOL_TLS 282
+struct tls_crypto_info_keys {
+ union {
+ struct tls12_crypto_info_aes_gcm_128 aes128;
+ struct tls12_crypto_info_chacha20_poly1305 chacha20;
+ };
+ size_t len;
+};
+
+static void tls_crypto_info_init(uint16_t tls_version, uint16_t cipher_type,
+ struct tls_crypto_info_keys *tls12)
+{
+ memset(tls12, 0, sizeof(*tls12));
+
+ switch (cipher_type) {
+ case TLS_CIPHER_CHACHA20_POLY1305:
+ tls12->len = sizeof(struct tls12_crypto_info_chacha20_poly1305);
+ tls12->chacha20.info.version = tls_version;
+ tls12->chacha20.info.cipher_type = cipher_type;
+ break;
+ case TLS_CIPHER_AES_GCM_128:
+ tls12->len = sizeof(struct tls12_crypto_info_aes_gcm_128);
+ tls12->aes128.info.version = tls_version;
+ tls12->aes128.info.cipher_type = cipher_type;
+ break;
+ default:
+ break;
+ }
+}
+
+static void memrnd(void *s, size_t n)
+{
+ int *dword = s;
+ char *byte;
+
+ for (; n >= 4; n -= 4)
+ *dword++ = rand();
+ byte = (void *)dword;
+ while (n--)
+ *byte++ = rand();
+}
+
FIXTURE(tls_basic)
{
int fd, cfd;
FIXTURE_SETUP(tls)
{
- union {
- struct tls12_crypto_info_aes_gcm_128 aes128;
- struct tls12_crypto_info_chacha20_poly1305 chacha20;
- } tls12;
+ struct tls_crypto_info_keys tls12;
struct sockaddr_in addr;
socklen_t len;
int sfd, ret;
- size_t tls12_sz;
self->notls = false;
len = sizeof(addr);
- memset(&tls12, 0, sizeof(tls12));
- switch (variant->cipher_type) {
- case TLS_CIPHER_CHACHA20_POLY1305:
- tls12_sz = sizeof(struct tls12_crypto_info_chacha20_poly1305);
- tls12.chacha20.info.version = variant->tls_version;
- tls12.chacha20.info.cipher_type = variant->cipher_type;
- break;
- case TLS_CIPHER_AES_GCM_128:
- tls12_sz = sizeof(struct tls12_crypto_info_aes_gcm_128);
- tls12.aes128.info.version = variant->tls_version;
- tls12.aes128.info.cipher_type = variant->cipher_type;
- break;
- default:
- tls12_sz = 0;
- }
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type,
+ &tls12);
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = htonl(INADDR_ANY);
if (!self->notls) {
ret = setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12,
- tls12_sz);
+ tls12.len);
ASSERT_EQ(ret, 0);
}
ASSERT_EQ(ret, 0);
ret = setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12,
- tls12_sz);
+ tls12.len);
ASSERT_EQ(ret, 0);
}
char recv_mem[TLS_PAYLOAD_MAX_LEN];
char buf[TLS_PAYLOAD_MAX_LEN];
+ memrnd(buf, sizeof(buf));
+
EXPECT_GE(send(self->fd, buf, send_len, 0), 0);
EXPECT_NE(recv(self->cfd, recv_mem, send_len, 0), -1);
EXPECT_EQ(memcmp(buf, recv_mem, send_len), 0);
struct iovec vec;
struct msghdr hdr;
+ memrnd(send_mem, sizeof(send_mem));
+
EXPECT_EQ(send(self->fd, send_mem, send_len, 0), send_len);
vec.iov_base = (char *)recv_mem;
vec.iov_len = TLS_PAYLOAD_MAX_LEN;
struct msghdr hdr;
int i;
+ memrnd(buf, sizeof(buf));
+
EXPECT_EQ(send(self->fd, buf, send_len, 0), send_len);
for (i = 0; i < msg_iovlen; i++) {
iov_base[i] = (char *)malloc(iov_len);
char send_mem[TLS_PAYLOAD_MAX_LEN * 2];
char recv_mem[TLS_PAYLOAD_MAX_LEN * 2];
+ memrnd(send_mem, sizeof(send_mem));
+
EXPECT_GE(send(self->fd, send_mem, total_len, 0), 0);
memset(recv_mem, 0, total_len);
int ret;
if (!self->notls) {
- struct tls12_crypto_info_aes_gcm_128 tls12;
+ struct tls_crypto_info_keys tls12;
- memset(&tls12, 0, sizeof(tls12));
- tls12.info.version = variant->tls_version;
- tls12.info.cipher_type = TLS_CIPHER_AES_GCM_128;
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type,
+ &tls12);
ret = setsockopt(self->fd, SOL_TLS, TLS_RX, &tls12,
- sizeof(tls12));
+ tls12.len);
ASSERT_EQ(ret, 0);
ret = setsockopt(self->cfd, SOL_TLS, TLS_TX, &tls12,
- sizeof(tls12));
+ tls12.len);
ASSERT_EQ(ret, 0);
}
-#!/bin/sh
+#!/bin/bash
# SPDX-License-Identifier: GPL-2.0
readonly BASE="ns-$(mktemp -u XXXXXX)"
route_test 240.5.6.7 240.5.6.1 255.1.2.1 255.1.2.3 24 "route between 240.5.6/24 and 255.1.2/24 (is allowed)"
route_test 0.200.6.7 0.200.38.1 245.99.101.1 245.99.200.111 16 "route between 0.200/16 and 245.99/16 (is allowed)"
#
+# Test support for lowest address ending in .0
+segmenttest 5.10.15.20 5.10.15.0 24 "assign and ping lowest address (/24)"
+#
+# Test support for lowest address not ending in .0
+segmenttest 192.168.101.192 192.168.101.193 26 "assign and ping lowest address (/26)"
+#
+# Routing using lowest address as a gateway/endpoint
+route_test 192.168.42.1 192.168.42.0 9.8.7.6 9.8.7.0 24 "routing using lowest address"
+#
# ==============================================
# ==== TESTS THAT CURRENTLY EXPECT FAILURE =====
# ==============================================
# Currently Linux does not allow this, so this should fail too
segmenttest 127.99.4.5 127.99.4.6 16 "assign and ping inside 127/8 (is forbidden)"
#
-# Test support for lowest address
-# Currently Linux does not allow this, so this should fail too
-segmenttest 5.10.15.20 5.10.15.0 24 "assign and ping lowest address (is forbidden)"
-#
-# Routing using lowest address as a gateway/endpoint
-# Currently Linux does not allow this, so this should fail too
-route_test 192.168.42.1 192.168.42.0 9.8.7.6 9.8.7.0 24 "routing using lowest address (is forbidden)"
-#
# Test support for unicast use of class D
# Currently Linux does not allow this, so this should fail too
segmenttest 225.1.2.3 225.1.2.200 24 "assign and ping class D address (is forbidden)"
cleanup() {
local ns
- local -r jobs="$(jobs -p)"
+ local jobs
+ readonly jobs="$(jobs -p)"
[ -n "${jobs}" ] && kill -1 ${jobs} 2>/dev/null
rm -f $STATS
if [ ! -f ../bpf/xdp_dummy.o ]; then
echo "Missing xdp_dummy helper. Build bpf selftest first"
- exit -1
+ exit 1
fi
create_ns
# SPDX-License-Identifier: GPL-2.0
# Makefile for netfilter selftests
-TEST_PROGS := nft_trans_stress.sh nft_nat.sh bridge_brouter.sh \
+TEST_PROGS := nft_trans_stress.sh nft_fib.sh nft_nat.sh bridge_brouter.sh \
conntrack_icmp_related.sh nft_flowtable.sh ipvs.sh \
nft_concat_range.sh nft_conntrack_helper.sh \
nft_queue.sh nft_meta.sh nf_nat_edemux.sh \
--- /dev/null
+#!/bin/bash
+#
+# This tests the fib expression.
+#
+# Kselftest framework requirement - SKIP code is 4.
+ksft_skip=4
+ret=0
+
+sfx=$(mktemp -u "XXXXXXXX")
+ns1="ns1-$sfx"
+ns2="ns2-$sfx"
+nsrouter="nsrouter-$sfx"
+timeout=4
+
+log_netns=$(sysctl -n net.netfilter.nf_log_all_netns)
+
+cleanup()
+{
+ ip netns del ${ns1}
+ ip netns del ${ns2}
+ ip netns del ${nsrouter}
+
+ [ $log_netns -eq 0 ] && sysctl -q net.netfilter.nf_log_all_netns=$log_netns
+}
+
+nft --version > /dev/null 2>&1
+if [ $? -ne 0 ];then
+ echo "SKIP: Could not run test without nft tool"
+ exit $ksft_skip
+fi
+
+ip -Version > /dev/null 2>&1
+if [ $? -ne 0 ];then
+ echo "SKIP: Could not run test without ip tool"
+ exit $ksft_skip
+fi
+
+ip netns add ${nsrouter}
+if [ $? -ne 0 ];then
+ echo "SKIP: Could not create net namespace"
+ exit $ksft_skip
+fi
+
+trap cleanup EXIT
+
+dmesg | grep -q ' nft_rpfilter: '
+if [ $? -eq 0 ]; then
+ dmesg -c | grep ' nft_rpfilter: '
+ echo "WARN: a previous test run has failed" 1>&2
+fi
+
+sysctl -q net.netfilter.nf_log_all_netns=1
+ip netns add ${ns1}
+ip netns add ${ns2}
+
+load_ruleset() {
+ local netns=$1
+
+ip netns exec ${netns} nft -f /dev/stdin <<EOF
+table inet filter {
+ chain prerouting {
+ type filter hook prerouting priority 0; policy accept;
+ fib saddr . iif oif missing counter log prefix "$netns nft_rpfilter: " drop
+ }
+}
+EOF
+}
+
+load_ruleset_count() {
+ local netns=$1
+
+ip netns exec ${netns} nft -f /dev/stdin <<EOF
+table inet filter {
+ chain prerouting {
+ type filter hook prerouting priority 0; policy accept;
+ ip daddr 1.1.1.1 fib saddr . iif oif missing counter drop
+ ip6 daddr 1c3::c01d fib saddr . iif oif missing counter drop
+ }
+}
+EOF
+}
+
+check_drops() {
+ dmesg | grep -q ' nft_rpfilter: '
+ if [ $? -eq 0 ]; then
+ dmesg | grep ' nft_rpfilter: '
+ echo "FAIL: rpfilter did drop packets"
+ return 1
+ fi
+
+ return 0
+}
+
+check_fib_counter() {
+ local want=$1
+ local ns=$2
+ local address=$3
+
+ line=$(ip netns exec ${ns} nft list table inet filter | grep 'fib saddr . iif' | grep $address | grep "packets $want" )
+ ret=$?
+
+ if [ $ret -ne 0 ];then
+ echo "Netns $ns fib counter doesn't match expected packet count of $want for $address" 1>&2
+ ip netns exec ${ns} nft list table inet filter
+ return 1
+ fi
+
+ if [ $want -gt 0 ]; then
+ echo "PASS: fib expression did drop packets for $address"
+ fi
+
+ return 0
+}
+
+load_ruleset ${nsrouter}
+load_ruleset ${ns1}
+load_ruleset ${ns2}
+
+ip link add veth0 netns ${nsrouter} type veth peer name eth0 netns ${ns1} > /dev/null 2>&1
+if [ $? -ne 0 ];then
+ echo "SKIP: No virtual ethernet pair device support in kernel"
+ exit $ksft_skip
+fi
+ip link add veth1 netns ${nsrouter} type veth peer name eth0 netns ${ns2}
+
+ip -net ${nsrouter} link set lo up
+ip -net ${nsrouter} link set veth0 up
+ip -net ${nsrouter} addr add 10.0.1.1/24 dev veth0
+ip -net ${nsrouter} addr add dead:1::1/64 dev veth0
+
+ip -net ${nsrouter} link set veth1 up
+ip -net ${nsrouter} addr add 10.0.2.1/24 dev veth1
+ip -net ${nsrouter} addr add dead:2::1/64 dev veth1
+
+ip -net ${ns1} link set lo up
+ip -net ${ns1} link set eth0 up
+
+ip -net ${ns2} link set lo up
+ip -net ${ns2} link set eth0 up
+
+ip -net ${ns1} addr add 10.0.1.99/24 dev eth0
+ip -net ${ns1} addr add dead:1::99/64 dev eth0
+ip -net ${ns1} route add default via 10.0.1.1
+ip -net ${ns1} route add default via dead:1::1
+
+ip -net ${ns2} addr add 10.0.2.99/24 dev eth0
+ip -net ${ns2} addr add dead:2::99/64 dev eth0
+ip -net ${ns2} route add default via 10.0.2.1
+ip -net ${ns2} route add default via dead:2::1
+
+test_ping() {
+ local daddr4=$1
+ local daddr6=$2
+
+ ip netns exec ${ns1} ping -c 1 -q $daddr4 > /dev/null
+ ret=$?
+ if [ $ret -ne 0 ];then
+ check_drops
+ echo "FAIL: ${ns1} cannot reach $daddr4, ret $ret" 1>&2
+ return 1
+ fi
+
+ ip netns exec ${ns1} ping -c 3 -q $daddr6 > /dev/null
+ ret=$?
+ if [ $ret -ne 0 ];then
+ check_drops
+ echo "FAIL: ${ns1} cannot reach $daddr6, ret $ret" 1>&2
+ return 1
+ fi
+
+ return 0
+}
+
+ip netns exec ${nsrouter} sysctl net.ipv6.conf.all.forwarding=1 > /dev/null
+ip netns exec ${nsrouter} sysctl net.ipv4.conf.veth0.forwarding=1 > /dev/null
+ip netns exec ${nsrouter} sysctl net.ipv4.conf.veth1.forwarding=1 > /dev/null
+
+sleep 3
+
+test_ping 10.0.2.1 dead:2::1 || exit 1
+check_drops || exit 1
+
+test_ping 10.0.2.99 dead:2::99 || exit 1
+check_drops || exit 1
+
+echo "PASS: fib expression did not cause unwanted packet drops"
+
+ip netns exec ${nsrouter} nft flush table inet filter
+
+ip -net ${ns1} route del default
+ip -net ${ns1} -6 route del default
+
+ip -net ${ns1} addr del 10.0.1.99/24 dev eth0
+ip -net ${ns1} addr del dead:1::99/64 dev eth0
+
+ip -net ${ns1} addr add 10.0.2.99/24 dev eth0
+ip -net ${ns1} addr add dead:2::99/64 dev eth0
+
+ip -net ${ns1} route add default via 10.0.2.1
+ip -net ${ns1} -6 route add default via dead:2::1
+
+ip -net ${nsrouter} addr add dead:2::1/64 dev veth0
+
+# switch to ruleset that doesn't log, this time
+# its expected that this does drop the packets.
+load_ruleset_count ${nsrouter}
+
+# ns1 has a default route, but nsrouter does not.
+# must not check return value, ping to 1.1.1.1 will
+# fail.
+check_fib_counter 0 ${nsrouter} 1.1.1.1 || exit 1
+check_fib_counter 0 ${nsrouter} 1c3::c01d || exit 1
+
+ip netns exec ${ns1} ping -c 1 -W 1 -q 1.1.1.1 > /dev/null
+check_fib_counter 1 ${nsrouter} 1.1.1.1 || exit 1
+
+sleep 2
+ip netns exec ${ns1} ping -c 3 -q 1c3::c01d > /dev/null
+check_fib_counter 3 ${nsrouter} 1c3::c01d || exit 1
+
+exit 0
siginfo_t first_siginfo; /* First observed siginfo_t. */
} ctx;
-/* Unique value to check si_perf is correctly set from perf_event_attr::sig_data. */
+/* Unique value to check si_perf_data is correctly set from perf_event_attr::sig_data. */
#define TEST_SIG_DATA(addr) (~(unsigned long)(addr))
static struct perf_event_attr make_event_attr(bool enabled, volatile void *addr)
EXPECT_EQ(ctx.signal_count, NUM_THREADS);
EXPECT_EQ(ctx.tids_want_signal, 0);
EXPECT_EQ(ctx.first_siginfo.si_addr, &ctx.iterate_on);
- EXPECT_EQ(ctx.first_siginfo.si_errno, PERF_TYPE_BREAKPOINT);
- EXPECT_EQ(ctx.first_siginfo.si_perf, TEST_SIG_DATA(&ctx.iterate_on));
+ EXPECT_EQ(ctx.first_siginfo.si_perf_type, PERF_TYPE_BREAKPOINT);
+ EXPECT_EQ(ctx.first_siginfo.si_perf_data, TEST_SIG_DATA(&ctx.iterate_on));
/* Check enabled for parent. */
ctx.iterate_on = 0;
EXPECT_EQ(ctx.signal_count, NUM_THREADS);
EXPECT_EQ(ctx.tids_want_signal, 0);
EXPECT_EQ(ctx.first_siginfo.si_addr, &ctx.iterate_on);
- EXPECT_EQ(ctx.first_siginfo.si_errno, PERF_TYPE_BREAKPOINT);
- EXPECT_EQ(ctx.first_siginfo.si_perf, TEST_SIG_DATA(&ctx.iterate_on));
+ EXPECT_EQ(ctx.first_siginfo.si_perf_type, PERF_TYPE_BREAKPOINT);
+ EXPECT_EQ(ctx.first_siginfo.si_perf_data, TEST_SIG_DATA(&ctx.iterate_on));
/* Check enabled for parent. */
ctx.iterate_on = 0;
EXPECT_EQ(ctx.signal_count, NUM_THREADS * ctx.iterate_on);
EXPECT_EQ(ctx.tids_want_signal, 0);
EXPECT_EQ(ctx.first_siginfo.si_addr, &ctx.iterate_on);
- EXPECT_EQ(ctx.first_siginfo.si_errno, PERF_TYPE_BREAKPOINT);
- EXPECT_EQ(ctx.first_siginfo.si_perf, TEST_SIG_DATA(&ctx.iterate_on));
+ EXPECT_EQ(ctx.first_siginfo.si_perf_type, PERF_TYPE_BREAKPOINT);
+ EXPECT_EQ(ctx.first_siginfo.si_perf_data, TEST_SIG_DATA(&ctx.iterate_on));
}
TEST_HARNESS_MAIN
/proc-self-map-files-002
/proc-self-syscall
/proc-self-wchan
+/proc-subset-pid
/proc-uptime-001
/proc-uptime-002
/read
# define SYSCALL_RET_SET(_regs, _val) \
do { \
typeof(_val) _result = (_val); \
- /* \
- * A syscall error is signaled by CR0 SO bit \
- * and the code is stored as a positive value. \
- */ \
- if (_result < 0) { \
- SYSCALL_RET(_regs) = -_result; \
- (_regs).ccr |= 0x10000000; \
- } else { \
+ if ((_regs.trap & 0xfff0) == 0x3000) { \
+ /* \
+ * scv 0 system call uses -ve result \
+ * for error, so no need to adjust. \
+ */ \
SYSCALL_RET(_regs) = _result; \
- (_regs).ccr &= ~0x10000000; \
+ } else { \
+ /* \
+ * A syscall error is signaled by the \
+ * CR0 SO bit and the code is stored as \
+ * a positive value. \
+ */ \
+ if (_result < 0) { \
+ SYSCALL_RET(_regs) = -_result; \
+ (_regs).ccr |= 0x10000000; \
+ } else { \
+ SYSCALL_RET(_regs) = _result; \
+ (_regs).ccr &= ~0x10000000; \
+ } \
} \
} while (0)
# define SYSCALL_RET_SET_ON_PTRACE_EXIT
return
# Check for required fields
- scapyinfo = self.args.caseinfo['scapy']
- scapy_keys = ['iface', 'count', 'packet']
- missing_keys = []
- keyfail = False
- for k in scapy_keys:
- if k not in scapyinfo:
- keyfail = True
- missing_keys.add(k)
- if keyfail:
- print('{}: Scapy block present in the test, but is missing info:'
- .format(self.sub_class))
- print('{}'.format(missing_keys))
-
- pkt = eval(scapyinfo['packet'])
- if '$' in scapyinfo['iface']:
- tpl = Template(scapyinfo['iface'])
- scapyinfo['iface'] = tpl.safe_substitute(NAMES)
- for count in range(scapyinfo['count']):
- sendp(pkt, iface=scapyinfo['iface'])
+ lscapyinfo = self.args.caseinfo['scapy']
+ if type(lscapyinfo) != list:
+ lscapyinfo = [ lscapyinfo, ]
+
+ for scapyinfo in lscapyinfo:
+ scapy_keys = ['iface', 'count', 'packet']
+ missing_keys = []
+ keyfail = False
+ for k in scapy_keys:
+ if k not in scapyinfo:
+ keyfail = True
+ missing_keys.append(k)
+ if keyfail:
+ print('{}: Scapy block present in the test, but is missing info:'
+ .format(self.sub_class))
+ print('{}'.format(missing_keys))
+
+ pkt = eval(scapyinfo['packet'])
+ if '$' in scapyinfo['iface']:
+ tpl = Template(scapyinfo['iface'])
+ scapyinfo['iface'] = tpl.safe_substitute(NAMES)
+ for count in range(scapyinfo['count']):
+ sendp(pkt, iface=scapyinfo['iface'])
"teardown": [
"$TC actions flush action ct"
]
+ },
+ {
+ "id": "3992",
+ "name": "Add ct action triggering DNAT tuple conflict",
+ "category": [
+ "actions",
+ "ct",
+ "scapy"
+ ],
+ "plugins": {
+ "requires": [
+ "nsPlugin",
+ "scapyPlugin"
+ ]
+ },
+ "setup": [
+ [
+ "$TC qdisc del dev $DEV1 ingress",
+ 0,
+ 1,
+ 2,
+ 255
+ ],
+ "$TC qdisc add dev $DEV1 ingress"
+ ],
+ "cmdUnderTest": "$TC filter add dev $DEV1 ingress protocol ip prio 1 flower ct_state -trk action ct commit nat dst addr 20.0.0.1 port 10 pipe action drop",
+ "scapy": [
+ {
+ "iface": "$DEV0",
+ "count": 1,
+ "packet": "Ether(type=0x800)/IP(src='10.0.0.10',dst='10.0.0.10')/TCP(sport=5000,dport=10)"
+ },
+ {
+ "iface": "$DEV0",
+ "count": 1,
+ "packet": "Ether(type=0x800)/IP(src='10.0.0.10',dst='10.0.0.20')/TCP(sport=5000,dport=10)"
+ }
+ ],
+ "expExitCode": "0",
+ "verifyCmd": "cat /proc/net/nf_conntrack",
+ "matchPattern": "dst=10.0.0.20",
+ "matchCount": "1",
+ "teardown": [
+ "$TC qdisc del dev $DEV1 ingress"
+ ]
}
]
"teardown": []
},
{
+ "id": "ba5b",
+ "name": "Add vlan modify action for protocol 802.1Q setting priority 0",
+ "category": [
+ "actions",
+ "vlan"
+ ],
+ "setup": [
+ [
+ "$TC actions flush action vlan",
+ 0,
+ 1,
+ 255
+ ]
+ ],
+ "cmdUnderTest": "$TC actions add action vlan modify protocol 802.1Q id 5 priority 0 index 100",
+ "expExitCode": "0",
+ "verifyCmd": "$TC actions get action vlan index 100",
+ "matchPattern": "action order [0-9]+: vlan.*modify id 100 priority 0 protocol 802.1Q pipe.*index 100 ref",
+ "matchCount": "0",
+ "teardown": [
+ "$TC actions flush action vlan"
+ ]
+ },
+ {
"id": "6812",
"name": "Add vlan modify action for protocol 802.1Q",
"category": [
"cmdUnderTest": "$TC actions add action vlan modify protocol 802.1Q id 5 index 100",
"expExitCode": "0",
"verifyCmd": "$TC actions get action vlan index 100",
- "matchPattern": "action order [0-9]+: vlan.*modify id 100 protocol 802.1Q priority 0 pipe.*index 100 ref",
+ "matchPattern": "action order [0-9]+: vlan.*modify id 100 protocol 802.1Q pipe.*index 100 ref",
"matchCount": "0",
"teardown": [
"$TC actions flush action vlan"
"cmdUnderTest": "$TC actions add action vlan modify protocol 802.1ad id 500 reclassify index 12",
"expExitCode": "0",
"verifyCmd": "$TC actions get action vlan index 12",
- "matchPattern": "action order [0-9]+: vlan.*modify id 500 protocol 802.1ad priority 0 reclassify.*index 12 ref",
+ "matchPattern": "action order [0-9]+: vlan.*modify id 500 protocol 802.1ad reclassify.*index 12 ref",
"matchCount": "1",
"teardown": [
"$TC actions flush action vlan"
"setup": [
"$IP link add dev $DUMMY type dummy || /bin/true"
],
- "cmdUnderTest": "$TC qdisc add dev $DUMMY root fq_pie flows 65536",
- "expExitCode": "2",
+ "cmdUnderTest": "$TC qdisc add dev $DUMMY handle 1: root fq_pie flows 65536",
+ "expExitCode": "0",
"verifyCmd": "$TC qdisc show dev $DUMMY",
- "matchPattern": "qdisc",
- "matchCount": "0",
+ "matchPattern": "qdisc fq_pie 1: root refcnt 2 limit 10240p flows 65536",
+ "matchCount": "1",
"teardown": [
"$IP link del dev $DUMMY"
]
ip1 -4 route add default dev wg0 table 51820
ip1 -4 rule add not fwmark 51820 table 51820
ip1 -4 rule add table main suppress_prefixlength 0
+n1 bash -c 'printf 0 > /proc/sys/net/ipv4/conf/vethc/rp_filter'
# Flood the pings instead of sending just one, to trigger routing table reference counting bugs.
n1 ping -W 1 -c 100 -f 192.168.99.7
n1 ping -W 1 -c 100 -f abab::1111
CONFIG_NETFILTER_XT_NAT=y
CONFIG_NETFILTER_XT_MATCH_LENGTH=y
CONFIG_NETFILTER_XT_MARK=y
-CONFIG_NF_CONNTRACK_IPV4=y
CONFIG_NF_NAT_IPV4=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_FILTER=y
}
/* Connect to <cid, port> and return the file descriptor. */
-int vsock_stream_connect(unsigned int cid, unsigned int port)
+static int vsock_connect(unsigned int cid, unsigned int port, int type)
{
union {
struct sockaddr sa;
control_expectln("LISTENING");
- fd = socket(AF_VSOCK, SOCK_STREAM, 0);
+ fd = socket(AF_VSOCK, type, 0);
timeout_begin(TIMEOUT);
do {
return fd;
}
+int vsock_stream_connect(unsigned int cid, unsigned int port)
+{
+ return vsock_connect(cid, port, SOCK_STREAM);
+}
+
+int vsock_seqpacket_connect(unsigned int cid, unsigned int port)
+{
+ return vsock_connect(cid, port, SOCK_SEQPACKET);
+}
+
/* Listen on <cid, port> and return the first incoming connection. The remote
* address is stored to clientaddrp. clientaddrp may be NULL.
*/
-int vsock_stream_accept(unsigned int cid, unsigned int port,
- struct sockaddr_vm *clientaddrp)
+static int vsock_accept(unsigned int cid, unsigned int port,
+ struct sockaddr_vm *clientaddrp, int type)
{
union {
struct sockaddr sa;
int client_fd;
int old_errno;
- fd = socket(AF_VSOCK, SOCK_STREAM, 0);
+ fd = socket(AF_VSOCK, type, 0);
if (bind(fd, &addr.sa, sizeof(addr.svm)) < 0) {
perror("bind");
return client_fd;
}
+int vsock_stream_accept(unsigned int cid, unsigned int port,
+ struct sockaddr_vm *clientaddrp)
+{
+ return vsock_accept(cid, port, clientaddrp, SOCK_STREAM);
+}
+
+int vsock_seqpacket_accept(unsigned int cid, unsigned int port,
+ struct sockaddr_vm *clientaddrp)
+{
+ return vsock_accept(cid, port, clientaddrp, SOCK_SEQPACKET);
+}
+
/* Transmit one byte and check the return value.
*
* expected_ret:
void init_signals(void);
unsigned int parse_cid(const char *str);
int vsock_stream_connect(unsigned int cid, unsigned int port);
+int vsock_seqpacket_connect(unsigned int cid, unsigned int port);
int vsock_stream_accept(unsigned int cid, unsigned int port,
struct sockaddr_vm *clientaddrp);
+int vsock_seqpacket_accept(unsigned int cid, unsigned int port,
+ struct sockaddr_vm *clientaddrp);
void vsock_wait_remote_close(int fd);
void send_byte(int fd, int expected_ret, int flags);
void recv_byte(int fd, int expected_ret, int flags);
#include <errno.h>
#include <unistd.h>
#include <linux/kernel.h>
+#include <sys/types.h>
+#include <sys/socket.h>
#include "timeout.h"
#include "control.h"
close(fd);
}
+#define MESSAGES_CNT 7
+static void test_seqpacket_msg_bounds_client(const struct test_opts *opts)
+{
+ int fd;
+
+ fd = vsock_seqpacket_connect(opts->peer_cid, 1234);
+ if (fd < 0) {
+ perror("connect");
+ exit(EXIT_FAILURE);
+ }
+
+ /* Send several messages, one with MSG_EOR flag */
+ for (int i = 0; i < MESSAGES_CNT; i++)
+ send_byte(fd, 1, 0);
+
+ control_writeln("SENDDONE");
+ close(fd);
+}
+
+static void test_seqpacket_msg_bounds_server(const struct test_opts *opts)
+{
+ int fd;
+ char buf[16];
+ struct msghdr msg = {0};
+ struct iovec iov = {0};
+
+ fd = vsock_seqpacket_accept(VMADDR_CID_ANY, 1234, NULL);
+ if (fd < 0) {
+ perror("accept");
+ exit(EXIT_FAILURE);
+ }
+
+ control_expectln("SENDDONE");
+ iov.iov_base = buf;
+ iov.iov_len = sizeof(buf);
+ msg.msg_iov = &iov;
+ msg.msg_iovlen = 1;
+
+ for (int i = 0; i < MESSAGES_CNT; i++) {
+ if (recvmsg(fd, &msg, 0) != 1) {
+ perror("message bound violated");
+ exit(EXIT_FAILURE);
+ }
+ }
+
+ close(fd);
+}
+
+#define MESSAGE_TRUNC_SZ 32
+static void test_seqpacket_msg_trunc_client(const struct test_opts *opts)
+{
+ int fd;
+ char buf[MESSAGE_TRUNC_SZ];
+
+ fd = vsock_seqpacket_connect(opts->peer_cid, 1234);
+ if (fd < 0) {
+ perror("connect");
+ exit(EXIT_FAILURE);
+ }
+
+ if (send(fd, buf, sizeof(buf), 0) != sizeof(buf)) {
+ perror("send failed");
+ exit(EXIT_FAILURE);
+ }
+
+ control_writeln("SENDDONE");
+ close(fd);
+}
+
+static void test_seqpacket_msg_trunc_server(const struct test_opts *opts)
+{
+ int fd;
+ char buf[MESSAGE_TRUNC_SZ / 2];
+ struct msghdr msg = {0};
+ struct iovec iov = {0};
+
+ fd = vsock_seqpacket_accept(VMADDR_CID_ANY, 1234, NULL);
+ if (fd < 0) {
+ perror("accept");
+ exit(EXIT_FAILURE);
+ }
+
+ control_expectln("SENDDONE");
+ iov.iov_base = buf;
+ iov.iov_len = sizeof(buf);
+ msg.msg_iov = &iov;
+ msg.msg_iovlen = 1;
+
+ ssize_t ret = recvmsg(fd, &msg, MSG_TRUNC);
+
+ if (ret != MESSAGE_TRUNC_SZ) {
+ printf("%zi\n", ret);
+ perror("MSG_TRUNC doesn't work");
+ exit(EXIT_FAILURE);
+ }
+
+ if (!(msg.msg_flags & MSG_TRUNC)) {
+ fprintf(stderr, "MSG_TRUNC expected\n");
+ exit(EXIT_FAILURE);
+ }
+
+ close(fd);
+}
+
static struct test_case test_cases[] = {
{
.name = "SOCK_STREAM connection reset",
.run_client = test_stream_msg_peek_client,
.run_server = test_stream_msg_peek_server,
},
+ {
+ .name = "SOCK_SEQPACKET msg bounds",
+ .run_client = test_seqpacket_msg_bounds_client,
+ .run_server = test_seqpacket_msg_bounds_server,
+ },
+ {
+ .name = "SOCK_SEQPACKET MSG_TRUNC flag",
+ .run_client = test_seqpacket_msg_trunc_client,
+ .run_server = test_seqpacket_msg_trunc_server,
+ },
{},
};
{
return kvm_make_all_cpus_request_except(kvm, req, NULL);
}
+EXPORT_SYMBOL_GPL(kvm_make_all_cpus_request);
#ifndef CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL
void kvm_flush_remote_tlbs(struct kvm *kvm)
if (val < grow_start)
val = grow_start;
- if (val > halt_poll_ns)
- val = halt_poll_ns;
+ if (val > vcpu->kvm->max_halt_poll_ns)
+ val = vcpu->kvm->max_halt_poll_ns;
vcpu->halt_poll_ns = val;
out:
goto out;
if (signal_pending(current))
goto out;
+ if (kvm_check_request(KVM_REQ_UNBLOCK, vcpu))
+ goto out;
ret = 0;
out:
goto out;
}
poll_end = cur = ktime_get();
- } while (single_task_running() && ktime_before(cur, stop));
+ } while (kvm_vcpu_can_poll(cur, stop));
}
prepare_to_rcuwait(&vcpu->wait);
if (prod->add_consumer)
ret = prod->add_consumer(prod, cons);
- if (ret)
- goto err_add_consumer;
-
- ret = cons->add_producer(cons, prod);
- if (ret)
- goto err_add_producer;
+ if (!ret) {
+ ret = cons->add_producer(cons, prod);
+ if (ret && prod->del_consumer)
+ prod->del_consumer(prod, cons);
+ }
if (cons->start)
cons->start(cons);
if (prod->start)
prod->start(prod);
-err_add_producer:
- if (prod->del_consumer)
- prod->del_consumer(prod, cons);
-err_add_consumer:
+
return ret;
}
projects / platform / kernel / linux-rpi.git / commitdiff